<div align="center">

<img src="resources/mmpt-logo.png" width="600"/>
  <div>&nbsp;</div>
  <div align="center">
    <b><font size="5">OpenMMLab website</font></b>
    <sup>
      <a href="https://openmmlab.com">
        <i><font size="4">HOT</font></i>
      </a>
    </sup>
    &nbsp;&nbsp;&nbsp;&nbsp;
    <b><font size="5">OpenMMLab platform</font></b>
    <sup>
      <a href="https://platform.openmmlab.com">
        <i><font size="4">TRY IT OUT</font></i>
      </a>
    </sup>
  </div>

[![PyPI](https://img.shields.io/pypi/v/mmpretrain)](https://pypi.org/project/mmpretrain)
[![Docs](https://img.shields.io/badge/docs-latest-blue)](https://mmpretrain.readthedocs.io/en/latest/)
[![Build Status](https://github.com/open-mmlab/mmpretrain/workflows/build/badge.svg)](https://github.com/open-mmlab/mmpretrain/actions)
[![codecov](https://codecov.io/gh/open-mmlab/mmpretrain/branch/main/graph/badge.svg)](https://codecov.io/gh/open-mmlab/mmpretrain)

[🆕 Update News](https://mmpretrain.readthedocs.io/en/latest/notes/changelog.html) |
[🤔 Reporting Issues](https://github.com/open-mmlab/mmpretrain/issues/new/choose)

<img src="https://user-images.githubusercontent.com/36138628/230307505-4727ad0a-7d71-4069-939d-b499c7e272b7.png" width="400"/>

English | [简体中文](/README_zh-CN.md)

</div>

</div>

  <a href="https://openmmlab.medium.com/" style="text-decoration:none;">
    <img src="https://user-images.githubusercontent.com/25839884/219255827-67c1a27f-f8c5-46a9-811d-5e57448c61d1.png" width="3%" alt="" /></a>
  <img src="https://user-images.githubusercontent.com/25839884/218346358-56cc8e2f-a2b8-487f-9088-32480cceabcf.png" width="3%" alt="" />
  <a href="https://discord.gg/raweFPmdzG" style="text-decoration:none;">
    <img src="https://user-images.githubusercontent.com/25839884/218347213-c080267f-cbb6-443e-8532-8e1ed9a58ea9.png" width="3%" alt="" /></a>

<img src="https://user-images.githubusercontent.com/25839884/218347213-c080267f-cbb6-443e-8532-8e1ed9a58ea9.png" width="3%" alt="" /></a>
  <a href="https://twitter.com/OpenMMLab" style="text-decoration:none;">
    <img src="https://user-images.githubusercontent.com/25839884/218346637-d30c8a0f-3eba-4699-8131-512fb06d46db.png" width="3%" alt="" /></a>
  <a href="https://www.youtube.com/openmmlab" style="text-decoration:none;">
    <img src="https://user-images.githubusercontent.com/25839884/218346691-ceb2116a-465a-40af-8424-9f30d2348ca9.png" width="3%" alt="" /></a>

<img src="https://user-images.githubusercontent.com/25839884/218346358-56cc8e2f-a2b8-487f-9088-32480cceabcf.png" width="3%" alt="" />
  <a href="https://space.bilibili.com/1293512903" style="text-decoration:none;">
    <img src="https://user-images.githubusercontent.com/25839884/219026751-d7d14cce-a7c9-4e82-9942-8375fca65b99.png" width="3%" alt="" /></a>
  <a href="https://www.zhihu.com/people/openmmlab" style="text-decoration:none;">
    <img src="https://user-images.githubusercontent.com/25839884/219026120-ba71e48b-6e94-4bd4-b4e9-b7d175b5e362.png" width="3%" alt="" /></a>
</div>

## Introduction

MMPreTrain is an open source pre-training toolbox based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com/) project.

The `main` branch works with **PyTorch 1.8+**.

### Major features

- Various backbones and pretrained models

- Rich training strategies (supervised learning, self-supervised learning, multi-modality learning etc.)
- Bag of training tricks
- Large-scale training configs
- High efficiency and extensibility
- Powerful toolkits for model analysis and experiments
- Various out-of-box inference tasks.
  - Image Classification
  - Image Caption
  - Visual Question Answering
  - Visual Grounding
  - Retrieval (Image-To-Image, Text-To-Image, Image-To-Text)

https://github.com/open-mmlab/mmpretrain/assets/26739999/e4dcd3a2-f895-4d1b-a351-fbc74a04e904

## What's new

🌟 v1.0.1 was released in 28/07/2023

Fix some bugs and enhance the codebase. Please refer to [changelog](https://mmpretrain.readthedocs.io/en/latest/notes/changelog.html) for more details.

🌟 v1.0.0 was released in 04/07/2023

- Support inference of more **multi-modal** algorithms, such as [**LLaVA**](./configs/llava/), [**MiniGPT-4**](./configs/minigpt4), [**Otter**](./configs/otter/), etc.
- Support around **10 multi-modal** datasets!
- Add [**iTPN**](./configs/itpn/), [**SparK**](./configs/spark/) self-supervised learning algorithms.
- Provide examples of [New Config](./mmpretrain/configs/) and [DeepSpeed/FSDP with FlexibleRunner](./configs/mae/benchmarks/). Here are the documentation links of [New Config](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#a-pure-python-style-configuration-file-beta) and [DeepSpeed/FSDP with FlexibleRunner](https://mmengine.readthedocs.io/en/latest/api/generated/mmengine.runner.FlexibleRunner.html#mmengine.runner.FlexibleRunner).

🌟 v1.0.0rc8 was released in 22/05/2023

- Support multiple **multi-modal** algorithms and inferencers. You can explore these features by the [gradio demo](https://github.com/open-mmlab/mmpretrain/tree/main/projects/gradio_demo)!
- Add EVA-02, Dino-V2, ViT-SAM and GLIP backbones.
- Register torchvision transforms into MMPretrain, you can now easily integrate torchvision's data augmentations in MMPretrain. See [the doc](https://mmpretrain.readthedocs.io/en/latest/api/data_process.html#torchvision-transforms)

Update of previous versions

- Integrated Self-supervised learning algorithms from **MMSelfSup**, such as **MAE**, **BEiT**, etc.
- Support **RIFormer**, a simple but effective vision backbone by removing token mixer.
- Refactor dataset pipeline visualization.
- Support **LeViT**, **XCiT**, **ViG**, **ConvNeXt-V2**, **EVA**, **RevViT**, **EfficientnetV2**, **CLIP**, **TinyViT** and **MixMIM** backbones.

This release introduced a brand new and flexible training & test engine, but it's still in progress. Welcome
to try according to [the documentation](https://mmpretrain.readthedocs.io/en/latest/).

And there are some BC-breaking changes. Please check [the migration tutorial](https://mmpretrain.readthedocs.io/en/latest/migration.html).

Please refer to [changelog](https://mmpretrain.readthedocs.io/en/latest/notes/changelog.html) for more details and other release history.

## Installation

Below are quick steps for installation:

```shell
conda create -n open-mmlab python=3.8 pytorch==1.10.1 torchvision==0.11.2 cudatoolkit=11.3 -c pytorch -y
conda activate open-mmlab
pip install openmim
git clone https://github.com/open-mmlab/mmpretrain.git
cd mmpretrain
mim install -e .
```

```

Please refer to [installation documentation](https://mmpretrain.readthedocs.io/en/latest/get_started.html) for more detailed installation and dataset preparation.

For multi-modality models support, please install the extra dependencies by:

```shell
mim install -e ".[multimodal]"
```

## User Guides

We provided a series of tutorials about the basic usage of MMPreTrain for new users:

- [Learn about Configs](https://mmpretrain.readthedocs.io/en/latest/user_guides/config.html)
- [Prepare Dataset](https://mmpretrain.readthedocs.io/en/latest/user_guides/dataset_prepare.html)
- [Inference with existing models](https://mmpretrain.readthedocs.io/en/latest/user_guides/inference.html)
- [Train](https://mmpretrain.readthedocs.io/en/latest/user_guides/train.html)
- [Test](https://mmpretrain.readthedocs.io/en/latest/user_guides/test.html)

- [Downstream tasks](https://mmpretrain.readthedocs.io/en/latest/user_guides/downstream.html)

For more information, please refer to [our documentation](https://mmpretrain.readthedocs.io/en/latest/).

## Model zoo

Results and models are available in the [model zoo](https://mmpretrain.readthedocs.io/en/latest/modelzoo_statistics.html).

  <b>Overview</b>
</div>
<table align="center">
  <tbody>
    <tr align="center" valign="bottom">
      <td>
        <b>Supported Backbones</b>
      </td>
      <td>
        <b>Self-supervised Learning</b>
      </td>
      <td>
        <b>Multi-Modality Algorithms</b>
      </td>
      <td>
        <b>Others</b>
      </td>
    </tr>
    <tr valign="top">
      <td>
        <ul>
        <li><a href="configs/vgg">VGG</a></li>
        <li><a href="configs/resnet">ResNet</a></li>

<td>
      </td>
    </tr>
      <td>
        <ul>
        <li><a href="configs/resnext">ResNeXt</a></li>
        <li><a href="configs/seresnet">SE-ResNet</a></li>
        <li><a href="configs/seresnet">SE-ResNeXt</a></li>
        <li><a href="configs/regnet">RegNet</a></li>
        <li><a href="configs/shufflenet_v1">ShuffleNet V1</a></li>
        <li><a href="configs/shufflenet_v2">ShuffleNet V2</a></li>
        <li><a href="configs/mobilenet_v2">MobileNet V2</a></li>
        <li><a href="configs/mobilenet_v3">MobileNet V3</a></li>
        <li><a href="configs/swin_transformer">Swin-Transformer</a></li>
        <li><a href="configs/swin_transformer_v2">Swin-Transformer V2</a></li>
        <li><a href="configs/repvgg">RepVGG</a></li>
        <li><a href="configs/vision_transformer">Vision-Transformer</a></li>

<li><a href="configs/repvgg">RepVGG</a></li>
        <li><a href="configs/tnt">Transformer-in-Transformer</a></li>
        <li><a href="configs/res2net">Res2Net</a></li>
        <li><a href="configs/mlp_mixer">MLP-Mixer</a></li>
        <li><a href="configs/deit">DeiT</a></li>
        <li><a href="configs/deit3">DeiT-3</a></li>
        <li><a href="configs/conformer">Conformer</a></li>
        <li><a href="configs/t2t_vit">T2T-ViT</a></li>
        <li><a href="configs/twins">Twins</a></li>
        <li><a href="configs/efficientnet">EfficientNet</a></li>
        <li><a href="configs/edgenext">EdgeNeXt</a></li>
        <li><a href="configs/convnext">ConvNeXt</a></li>
        <li><a href="configs/hrnet">HRNet</a></li>
        <li><a href="configs/van">VAN</a></li>
        <li><a href="configs/convmixer">ConvMixer</a></li>
        <li><a href="configs/cspnet">CSPNet</a></li>

<li><a href="configs/van">VAN</a></li>
        <li><a href="configs/poolformer">PoolFormer</a></li>
        <li><a href="configs/inception_v3">Inception V3</a></li>
        <li><a href="configs/mobileone">MobileOne</a></li>
        <li><a href="configs/efficientformer">EfficientFormer</a></li>
        <li><a href="configs/mvit">MViT</a></li>
        <li><a href="configs/hornet">HorNet</a></li>
        <li><a href="configs/mobilevit">MobileViT</a></li>
        <li><a href="configs/davit">DaViT</a></li>
        <li><a href="configs/replknet">RepLKNet</a></li>
        <li><a href="configs/beit">BEiT</a></li>
        <li><a href="configs/mixmim">MixMIM</a></li>
        <li><a href="configs/efficientnet_v2">EfficientNet V2</a></li>
        <li><a href="configs/revvit">RevViT</a></li>
        <li><a href="configs/convnext_v2">ConvNeXt V2</a></li>

<li><a href="configs/efficientnet_v2">EfficientNet V2</a></li>
        <li><a href="configs/vig">ViG</a></li>
        <li><a href="configs/xcit">XCiT</a></li>
        <li><a href="configs/levit">LeViT</a></li>
        <li><a href="configs/riformer">RIFormer</a></li>
        <li><a href="configs/glip">GLIP</a></li>
        <li><a href="configs/sam">ViT SAM</a></li>
        <li><a href="configs/eva02">EVA02</a></li>
        <li><a href="configs/dinov2">DINO V2</a></li>
        <li><a href="configs/hivit">HiViT</a></li>
        </ul>
      </td>
      <td>
        <ul>
        <li><a href="configs/mocov2">MoCo V1 (CVPR'2020)</a></li>
        <li><a href="configs/simclr">SimCLR (ICML'2020)</a></li>
        <li><a href="configs/mocov2">MoCo V2 (arXiv'2020)</a></li>
        <li><a href="configs/byol">BYOL (NeurIPS'2020)</a></li>

<li><a href="configs/simclr">SimCLR (ICML'2020)</a></li>
        <li><a href="configs/swav">SwAV (NeurIPS'2020)</a></li>
        <li><a href="configs/densecl">DenseCL (CVPR'2021)</a></li>
        <li><a href="configs/simsiam">SimSiam (CVPR'2021)</a></li>
        <li><a href="configs/barlowtwins">Barlow Twins (ICML'2021)</a></li>
        <li><a href="configs/mocov3">MoCo V3 (ICCV'2021)</a></li>
        <li><a href="configs/beit">BEiT (ICLR'2022)</a></li>
        <li><a href="configs/mae">MAE (CVPR'2022)</a></li>
        <li><a href="configs/simmim">SimMIM (CVPR'2022)</a></li>
        <li><a href="configs/maskfeat">MaskFeat (CVPR'2022)</a></li>
        <li><a href="configs/cae">CAE (arXiv'2022)</a></li>
        <li><a href="configs/milan">MILAN (arXiv'2022)</a></li>
        <li><a href="configs/beitv2">BEiT V2 (arXiv'2022)</a></li>

<li><a href="configs/cae">CAE (arXiv'2022)</a></li>
        <li><a href="configs/eva">EVA (CVPR'2023)</a></li>
        <li><a href="configs/mixmim">MixMIM (arXiv'2022)</a></li>
        <li><a href="configs/itpn">iTPN (CVPR'2023)</a></li>
        <li><a href="configs/spark">SparK (ICLR'2023)</a></li>
        </ul>
      </td>
      <td>
        <ul>
        <li><a href="configs/blip">BLIP (arxiv'2022)</a></li>
        <li><a href="configs/blip2">BLIP-2 (arxiv'2023)</a></li>
        <li><a href="configs/ofa">OFA (CoRR'2022)</a></li>
        <li><a href="configs/flamingo">Flamingo (NeurIPS'2022)</a></li>
        <li><a href="configs/chinese_clip">Chinese CLIP (arxiv'2022)</a></li>
        <li><a href="configs/minigpt4">MiniGPT-4 (arxiv'2023)</a></li>
        <li><a href="configs/llava">LLaVA (arxiv'2023)</a></li>

<li><a href="configs/minigpt4">MiniGPT-4 (arxiv'2023)</a></li>
        <li><a href="configs/otter">Otter (arxiv'2023)</a></li>
        </ul>
      </td>
      <td>
      Image Retrieval Task:
        <ul>
        <li><a href="configs/arcface">ArcFace (CVPR'2019)</a></li>
        </ul>
      Training&Test Tips:
        <ul>
        <li><a href="https://arxiv.org/abs/1909.13719">RandAug</a></li>
        <li><a href="https://arxiv.org/abs/1805.09501">AutoAug</a></li>
        <li><a href="mmpretrain/datasets/samplers/repeat_aug.py">RepeatAugSampler</a></li>
        <li><a href="mmpretrain/models/tta/score_tta.py">TTA</a></li>
        <li>...</li>
        </ul>
      </td>
  </tbody>
</table>

## Contributing

We appreciate all contributions to improve MMPreTrain.
Please refer to [CONTRUBUTING](https://mmpretrain.readthedocs.io/en/latest/notes/contribution_guide.html) for the contributing guideline.

## Acknowledgement

MMPreTrain is an open source project that is contributed by researchers and engineers from various colleges and companies. We appreciate all the contributors who implement their methods or add new features, as well as users who give valuable feedbacks.
We wish that the toolbox and benchmark could serve the growing research community by providing a flexible toolkit to reimplement existing methods and supporting their own academic research.

## Citation

If you find this project useful in your research, please consider cite:

```BibTeX
@misc{2023mmpretrain,
    title={OpenMMLab's Pre-training Toolbox and Benchmark},
    author={MMPreTrain Contributors},
    howpublished = {\url{https://github.com/open-mmlab/mmpretrain}},
    year={2023}
}
```

## License

This project is released under the [Apache 2.0 license](LICENSE).

howpublished = {\url{https://github.com/open-mmlab/mmpretrain}},
}
```

## License

## Projects in OpenMMLab

- [MMEngine](https://github.com/open-mmlab/mmengine): OpenMMLab foundational library for training deep learning models.
- [MMCV](https://github.com/open-mmlab/mmcv): OpenMMLab foundational library for computer vision.
- [MIM](https://github.com/open-mmlab/mim): MIM installs OpenMMLab packages.
- [MMEval](https://github.com/open-mmlab/mmeval): A unified evaluation library for multiple machine learning libraries.
- [MMPreTrain](https://github.com/open-mmlab/mmpretrain): OpenMMLab pre-training toolbox and benchmark.
- [MMDetection](https://github.com/open-mmlab/mmdetection): OpenMMLab detection toolbox and benchmark.
- [MMDetection3D](https://github.com/open-mmlab/mmdetection3d): OpenMMLab's next-generation platform for general 3D object detection.

- [MMRotate](https://github.com/open-mmlab/mmrotate): OpenMMLab rotated object detection toolbox and benchmark.
- [MMYOLO](https://github.com/open-mmlab/mmyolo): OpenMMLab YOLO series toolbox and benchmark.
- [MMSegmentation](https://github.com/open-mmlab/mmsegmentation): OpenMMLab semantic segmentation toolbox and benchmark.
- [MMOCR](https://github.com/open-mmlab/mmocr): OpenMMLab text detection, recognition, and understanding toolbox.
- [MMPose](https://github.com/open-mmlab/mmpose): OpenMMLab pose estimation toolbox and benchmark.
- [MMHuman3D](https://github.com/open-mmlab/mmhuman3d): OpenMMLab 3D human parametric model toolbox and benchmark.
- [MMSelfSup](https://github.com/open-mmlab/mmselfsup): OpenMMLab self-supervised learning toolbox and benchmark.

- [MMRazor](https://github.com/open-mmlab/mmrazor): OpenMMLab model compression toolbox and benchmark.
- [MMFewShot](https://github.com/open-mmlab/mmfewshot): OpenMMLab fewshot learning toolbox and benchmark.
- [MMAction2](https://github.com/open-mmlab/mmaction2): OpenMMLab's next-generation action understanding toolbox and benchmark.
- [MMTracking](https://github.com/open-mmlab/mmtracking): OpenMMLab video perception toolbox and benchmark.
- [MMFlow](https://github.com/open-mmlab/mmflow): OpenMMLab optical flow toolbox and benchmark.
- [MMagic](https://github.com/open-mmlab/mmagic): Open**MM**Lab **A**dvanced, **G**enerative and **I**ntelligent **C**reation toolbox.
- [MMGeneration](https://github.com/open-mmlab/mmgeneration): OpenMMLab image and video generative models toolbox.
- [MMDeploy](https://github.com/open-mmlab/mmdeploy): OpenMMLab model deployment framework.

- [Playground](https://github.com/open-mmlab/playground): A central hub for gathering and showcasing amazing projects built upon OpenMMLab.

<div id="top" align="center">
  <img src="docs/en/_static/image/mmagic-logo.png" width="500px"/>
    <font size="10"><b>M</b>ultimodal <b>A</b>dvanced, <b>G</b>enerative, and <b>I</b>ntelligent <b>C</b>reation (MMagic [em'mædʒɪk])</font>
  </div>
    <sup>
      </a>
    </sup>
    <sup>
      </a>
    </sup>
  </div>

[![PyPI](https://badge.fury.io/py/mmagic.svg)](https://pypi.org/project/mmagic/)
[![docs](https://img.shields.io/badge/docs-latest-blue)](https://mmagic.readthedocs.io/en/latest/)

[📘Documentation](https://mmagic.readthedocs.io/en/latest/) |
[🛠️Installation](https://mmagic.readthedocs.io/en/latest/get_started/install.html) |
[📊Model Zoo](https://mmagic.readthedocs.io/en/latest/model_zoo/overview.html) |
[🆕Update News](https://mmagic.readthedocs.io/en/latest/changelog.html) |
[🚀Ongoing Projects](https://github.com/open-mmlab/mmagic/projects) |
[🤔Reporting Issues](https://github.com/open-mmlab/mmagic/issues)

English | [简体中文](README_zh-CN.md)

</div>

    <img src="https://user-images.githubusercontent.com/25839884/218352562-cdded397-b0f3-4ca1-b8dd-a60df8dca75b.png" width="3%" alt="" /></a>

<a href="https://discord.gg/raweFPmdzG" style="text-decoration:none;">
</div>

<img src="https://user-images.githubusercontent.com/25839884/218346691-ceb2116a-465a-40af-8424-9f30d2348ca9.png" width="3%" alt="" /></a>
</div>

## 🚀 What's New <a><img width="35" height="20" src="https://user-images.githubusercontent.com/12782558/212848161-5e783dd6-11e8-4fe0-bbba-39ffb77730be.png"></a>

### New release [**MMagic v1.0.1**](https://github.com/open-mmlab/mmagic/releases/tag/v1.0.1) \[26/05/2023\]:

- Support tomesd for StableDiffusion speed-up.
- Support all inpainting/matting/image restoration models inferencer.
- Support animated drawings.
- Support Style-Based Global Appearance Flow for Virtual Try-On.
- Fix inferencer in pip-install.

We are excited to announce the release of MMagic v1.0.0 that inherits from [MMEditing](https://github.com/open-mmlab/mmediting) and [MMGeneration](https://github.com/open-mmlab/mmgeneration).

After iterative updates with OpenMMLab 2.0 framework and merged with MMGeneration, MMEditing has become a powerful tool that supports low-level algorithms based on both GAN and CNN. Today, MMEditing embraces Generative AI and transforms into a more advanced and comprehensive AIGC toolkit: **MMagic** (**M**ultimodal **A**dvanced, **G**enerative, and **I**ntelligent **C**reation). MMagic will provide more agile and flexible experimental support for researchers and AIGC enthusiasts, and help you on your AIGC exploration journey.

We highlight the following new features.

**1. New Models**

We support 11 new models in 4 new tasks.

- Text2Image / Diffusion
  - ControlNet
  - DreamBooth
  - Stable Diffusion
  - Disco Diffusion
  - GLIDE
  - Guided Diffusion
- 3D-aware Generation
  - EG3D

  - GLIDE
  - EG3D
- Image Restoration
  - NAFNet
  - Restormer
  - SwinIR
- Image Colorization
  - InstColorization

**2. Magic Diffusion Model**

For the Diffusion Model, we provide the following "magic" :

- Support image generation based on Stable Diffusion and Disco Diffusion.
- Support Finetune methods such as Dreambooth and DreamBooth LoRA.
- Support controllability in text-to-image generation using ControlNet.
- Support acceleration and optimization strategies based on xFormers to improve training and inference efficiency.
- Support video generation based on MultiFrame Render.
- Support calling basic models and sampling strategies through DiffuserWrapper.

**3. Upgraded Framework**

By using MMEngine and MMCV of OpenMMLab 2.0 framework, MMagic has upgraded in the following new features:

- Refactor DataSample to support the combination and splitting of batch dimensions.
- Refactor DataPreprocessor and unify the data format for various tasks during training and inference.
- Refactor MultiValLoop and MultiTestLoop, supporting the evaluation of both generation-type metrics (e.g. FID) and reconstruction-type metrics (e.g. SSIM), and supporting the evaluation of multiple datasets at once.
- Support visualization on local files or using tensorboard and wandb.
- Support for 33+ algorithms accelerated by Pytorch 2.0.

**MMagic** has supported all the tasks, models, metrics, and losses in [MMEditing](https://github.com/open-mmlab/mmediting) and [MMGeneration](https://github.com/open-mmlab/mmgeneration) and unifies interfaces of all components based on [MMEngine](https://github.com/open-mmlab/mmengine) 😍.

Please refer to [changelog.md](docs/en/changelog.md) for details and release history.

Please refer to [migration documents](docs/en/migration/overview.md) to migrate from [old version](https://github.com/open-mmlab/mmagic/tree/0.x) MMEditing 0.x to new version MMagic 1.x .

## 📄 Table of Contents

- [📖 Introduction](#-introduction)
- [🙌 Contributing](#-contributing)
- [🛠️ Installation](#%EF%B8%8F-installation)
- [📊 Model Zoo](#-model-zoo)
- [🤝 Acknowledgement](#-acknowledgement)
- [🖊️ Citation](#%EF%B8%8F-citation)
- [🎫 License](#-license)
- [🏗️ ️OpenMMLab Family](#%EF%B8%8F-️openmmlab-family)

<p align="right"><a href="#top">🔝Back to top</a></p>

## 📖 Introduction

MMagic (**M**ultimodal **A**dvanced, **G**enerative, and **I**ntelligent **C**reation) is an advanced and comprehensive AIGC toolkit that inherits from [MMEditing](https://github.com/open-mmlab/mmediting) and [MMGeneration](https://github.com/open-mmlab/mmgeneration). It is an open-source image and video editing&generating toolbox based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com/) project.

Currently, MMagic support multiple image and video generation/editing tasks.

https://user-images.githubusercontent.com/49083766/233564593-7d3d48ed-e843-4432-b610-35e3d257765c.mp4

The best practice on our main branch works with **Python 3.8+** and **PyTorch 1.9+**.

### ✨ Major features

- **State of the Art Models**

  MMagic provides state-of-the-art generative models to process, edit and synthesize images and videos.

- **Powerful and Popular Applications**

  MMagic supports popular and contemporary image restoration, text-to-image, 3D-aware generation, inpainting, matting, super-resolution and generation applications. Specifically, MMagic supports fine-tuning for stable diffusion and many exciting diffusion's application such as ControlNet Animation with SAM. MMagic also supports GAN interpolation, GAN projection, GAN manipulations and many other popular GAN’s applications. It’s time to begin your AIGC exploration journey!

- **Efficient Framework**

  By using MMEngine and MMCV of OpenMMLab 2.0 framework, MMagic decompose the editing framework into different modules and one can easily construct a customized editor framework by combining different modules. We can define the training process just like playing with Legos and provide rich components and strategies. In MMagic, you can complete controls on the training process with different levels of APIs. With the support of [MMSeparateDistributedDataParallel](https://github.com/open-mmlab/mmengine/blob/main/mmengine/model/wrappers/seperate_distributed.py), distributed training for dynamic architectures can be easily implemented.

## 🙌 Contributing

More and more community contributors are joining us to make our repo better. Some recent projects are contributed by the community including:

- [GLIDE](projects/glide/configs/README.md) is contributed by @Taited.

- [Restormer](configs/restormer/README.md) is contributed by @AlexZou14.
- [SwinIR](configs/swinir/README.md) is contributed by @Zdafeng.

[Projects](projects/README.md) is opened to make it easier for everyone to add projects to MMagic.

We appreciate all contributions to improve MMagic. Please refer to [CONTRIBUTING.md](https://github.com/open-mmlab/mmcv/blob/main/CONTRIBUTING.md) in MMCV and [CONTRIBUTING.md](https://github.com/open-mmlab/mmengine/blob/main/CONTRIBUTING.md) in MMEngine for more details about the contributing guideline.

## 🛠️ Installation

MMagic depends on [PyTorch](https://pytorch.org/), [MMEngine](https://github.com/open-mmlab/mmengine) and [MMCV](https://github.com/open-mmlab/mmcv).
Below are quick steps for installation.

**Step 1.**
Install PyTorch following [official instructions](https://pytorch.org/get-started/locally/).

**Step 2.**

Install MMCV, MMEngine and MMagic with [MIM](https://github.com/open-mmlab/mim).

```shell
pip3 install openmim
mim install 'mmcv>=2.0.0'
mim install 'mmengine'
mim install 'mmagic'
```

**Step 3.**
Verify MMagic has been successfully installed.

```shell
cd ~
python -c "import mmagic; print(mmagic.__version__)"
# Example output: 1.0.0
```

**Getting Started**

After installing MMagic successfully, now you are able to play with MMagic! To generate an image from text, you only need several lines of codes by MMagic!

```python
from mmagic.apis import MMagicInferencer
sd_inferencer = MMagicInferencer(model_name='stable_diffusion')
text_prompts = 'A panda is having dinner at KFC'
result_out_dir = 'output/sd_res.png'
sd_inferencer.infer(text=text_prompts, result_out_dir=result_out_dir)
```

```

Please see [quick run](docs/en/get_started/quick_run.md) and [inference](docs/en/user_guides/inference.md) for the basic usage of MMagic.

**Install MMagic from source**

You can also experiment on the latest developed version rather than the stable release by installing MMagic from source with the following commands:

```shell
git clone https://github.com/open-mmlab/mmagic.git
cd mmagic
pip3 install -e .
```

Please refer to [installation](docs/en/get_started/install.md) for more detailed instruction.

## 📊 Model Zoo

  <b>Supported algorithms</b>
</div>
  <tbody>
      <td>
        <b>Conditional GANs</b>
      </td>
      <td>
        <b>Unconditional GANs</b>
      </td>
      <td>

  <tbody>
      <td>
      </td>
      <td>
      </td>
      <td>
        <b>Image Restoration</b>
      </td>
      <td>
        <b>Image Super-Resolution</b>
      </td>
    </tr>
      <td>
        <ul>
            <li><a href="configs/sngan_proj/README.md">SNGAN/Projection GAN (ICLR'2018)</a></li>
            <li><a href="configs/sagan/README.md">SAGAN (ICML'2019)</a></li>
            <li><a href="configs/biggan/README.md">BIGGAN/BIGGAN-DEEP (ICLR'2018)</a></li>
      </ul>
      </td>
      <td>
        <ul>
          <li><a href="configs/dcgan/README.md">DCGAN (ICLR'2016)</a></li>
          <li><a href="configs/wgan-gp/README.md">WGAN-GP (NeurIPS'2017)</a></li>
          <li><a href="configs/lsgan/README.md">LSGAN (ICCV'2017)</a></li>
          <li><a href="configs/ggan/README.md">GGAN (ArXiv'2017)</a></li>

<li><a href="configs/lsgan/README.md">LSGAN (ICCV'2017)</a></li>
          <li><a href="configs/pggan/README.md">PGGAN (ICLR'2018)</a></li>
          <li><a href="configs/singan/README.md">SinGAN (ICCV'2019)</a></li>
          <li><a href="configs/styleganv1/README.md">StyleGANV1 (CVPR'2019)</a></li>
          <li><a href="configs/styleganv2/README.md">StyleGANV2 (CVPR'2019)</a></li>
          <li><a href="configs/styleganv3/README.md">StyleGANV3 (NeurIPS'2021)</a></li>
        </ul>
      </td>
      <td>
        <ul>
          <li><a href="configs/swinir/README.md">SwinIR (ICCVW'2021)</a></li>
          <li><a href="configs/nafnet/README.md">NAFNet (ECCV'2022)</a></li>
          <li><a href="configs/restormer/README.md">Restormer (CVPR'2022)</a></li>
        </ul>
      </td>
      <td>
        <ul>
          <li><a href="configs/srcnn/README.md">SRCNN (TPAMI'2015)</a></li>

</ul>
      </td>
      <td>
        <ul>
          <li><a href="configs/srgan_resnet/README.md">SRResNet&SRGAN (CVPR'2016)</a></li>
          <li><a href="configs/edsr/README.md">EDSR (CVPR'2017)</a></li>
          <li><a href="configs/esrgan/README.md">ESRGAN (ECCV'2018)</a></li>
          <li><a href="configs/rdn/README.md">RDN (CVPR'2018)</a></li>
          <li><a href="configs/dic/README.md">DIC (CVPR'2020)</a></li>
          <li><a href="configs/ttsr/README.md">TTSR (CVPR'2020)</a></li>
          <li><a href="configs/glean/README.md">GLEAN (CVPR'2021)</a></li>
          <li><a href="configs/liif/README.md">LIIF (CVPR'2021)</a></li>
          <li><a href="configs/real_esrgan/README.md">Real-ESRGAN (ICCVW'2021)</a></li>
        </ul>
      </td>
    </tr>
</td>
    </tr>
  </tbody>
<tbody>
      <td>
        <b>Video Super-Resolution</b>
      </td>
      <td>

</ul>
      </td>
    </tr>
</td>
    </tr>
  </tbody>
<tbody>
      <td>
      </td>
      <td>
        <b>Video Interpolation</b>
      </td>
      <td>
        <b>Image Colorization</b>
      </td>
      <td>
        <b>Image Translation</b>
      </td>
    </tr>
      <td>
        <ul>
            <li><a href="configs/edvr/README.md">EDVR (CVPR'2018)</a></li>
            <li><a href="configs/tof/README.md">TOF (IJCV'2019)</a></li>
            <li><a href="configs/tdan/README.md">TDAN (CVPR'2020)</a></li>
            <li><a href="configs/basicvsr/README.md">BasicVSR (CVPR'2021)</a></li>
            <li><a href="configs/iconvsr/README.md">IconVSR (CVPR'2021)</a></li>
            <li><a href="configs/basicvsr_pp/README.md">BasicVSR++ (CVPR'2022)</a></li>
            <li><a href="configs/real_basicvsr/README.md">RealBasicVSR (CVPR'2022)</a></li>
      </ul>
      </td>
      <td>

<li><a href="configs/real_basicvsr/README.md">RealBasicVSR (CVPR'2022)</a></li>
      </ul>
      </td>
      <td>
        <ul>
          <li><a href="configs/tof/README.md">TOFlow (IJCV'2019)</a></li>
          <li><a href="configs/cain/README.md">CAIN (AAAI'2020)</a></li>
          <li><a href="configs/flavr/README.md">FLAVR (CVPR'2021)</a></li>
        </ul>
      </td>
      <td>
        <ul>
          <li><a href="configs/inst_colorization/README.md">InstColorization (CVPR'2020)</a></li>
        </ul>
      </td>
      <td>
        <ul>
          <li><a href="configs/pix2pix/README.md">Pix2Pix (CVPR'2017)</a></li>
          <li><a href="configs/cyclegan/README.md">CycleGAN (ICCV'2017)</a></li>
        </ul>
      </td>
    </tr>
</td>
    </tr>
  </tbody>
<tbody>
      <td>
        <b>Inpainting</b>
      </td>
      <td>
        <b>Matting</b>
      </td>
      <td>
        <b>Text-to-Image</b>
      </td>
      <td>

<tr align="center" valign="bottom">
      <td>
      </td>
      <td>
      </td>
      <td>
      </td>
      <td>
        <b>3D-aware Generation</b>
      </td>
    </tr>
      <td>
        <ul>
          <li><a href="configs/global_local/README.md">Global&Local (ToG'2017)</a></li>
          <li><a href="configs/deepfillv1/README.md">DeepFillv1 (CVPR'2018)</a></li>
          <li><a href="configs/partial_conv/README.md">PConv (ECCV'2018)</a></li>
          <li><a href="configs/deepfillv2/README.md">DeepFillv2 (CVPR'2019)</a></li>
          <li><a href="configs/aot_gan/README.md">AOT-GAN (TVCG'2019)</a></li>
        </ul>
      </td>
      <td>
        <ul>
          <li><a href="configs/dim/README.md">DIM (CVPR'2017)</a></li>
          <li><a href="configs/indexnet/README.md">IndexNet (ICCV'2019)</a></li>
          <li><a href="configs/mask2former">GCA (AAAI'2020)</a></li>

<li><a href="configs/indexnet/README.md">IndexNet (ICCV'2019)</a></li>
        </ul>
      </td>
      <td>
        <ul>
          <li><a href="projects/glide/configs/README.md">GLIDE (NeurIPS'2021)</a></li>
          <li><a href="configs/guided_diffusion/README.md">Guided Diffusion (NeurIPS'2021)</a></li>
          <li><a href="configs/disco_diffusion/README.md">Disco-Diffusion (2022)</a></li>
          <li><a href="configs/stable_diffusion/README.md">Stable-Diffusion (2022)</a></li>
          <li><a href="configs/dreambooth/README.md">DreamBooth (2022)</a></li>
          <li><a href="configs/controlnet/README.md">ControlNet (2023)</a></li>
          <li><a href="configs/controlnet_animation/README.md">ControlNet Animation (2023)</a></li>
        </ul>
      </td>
      <td>
        <ul>
          <li><a href="configs/eg3d/README.md">EG3D (CVPR'2022)</a></li>
        </ul>
      </td>
    </tr>
</td>

</ul>
      </td>
      <td>
        <ul>
        </ul>
      </td>
    </tr>
</td>
    </tr>
  </tbody>
</table>

Please refer to [model_zoo](https://mmagic.readthedocs.io/en/latest/model_zoo/overview.html) for more details.

## 🤝 Acknowledgement

MMagic is an open source project that is contributed by researchers and engineers from various colleges and companies. We wish that the toolbox and benchmark could serve the growing research community by providing a flexible toolkit to reimplement existing methods and develop their own new methods.

We appreciate all the contributors who implement their methods or add new features, as well as users who give valuable feedbacks. Thank you all!

<a href="https://github.com/open-mmlab/mmagic/graphs/contributors">
  <img src="https://contrib.rocks/image?repo=open-mmlab/mmagic" />
</a>

</a>

## 🖊️ Citation

If MMagic is helpful to your research, please cite it as below.

```bibtex
@misc{mmagic2023,
    title = {{MMagic}: {OpenMMLab} Multimodal Advanced, Generative, and Intelligent Creation Toolbox},
    author = {{MMagic Contributors}},
    howpublished = {\url{https://github.com/open-mmlab/mmagic}},
    year = {2023}
}
```

```bibtex
@misc{mmediting2022,
    title = {{MMEditing}: {OpenMMLab} Image and Video Editing Toolbox},
    author = {{MMEditing Contributors}},
    howpublished = {\url{https://github.com/open-mmlab/mmediting}},
    year = {2022}
}
```

## 🎫 License

year = {2022}
}
```

Please refer to [LICENSES](LICENSE) for the careful check, if you are using our code for commercial matters.

## 🏗️ ️OpenMMLab Family

- [MMPreTrain](https://github.com/open-mmlab/mmpretrain): OpenMMLab Pre-training Toolbox and Benchmark.

- [MMagic](https://github.com/open-mmlab/mmagic): OpenMMLab Multimodal Advanced, Generative, and Intelligent Creation Toolbox.

<!-- [BACKBONE] -->

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_cvpr_2018/html/Sandler_MobileNetV2_Inverted_Residuals_CVPR_2018_paper.html">MobilenetV2 (CVPR'2018)</a></summary>

```bibtex
@inproceedings{sandler2018mobilenetv2,
  title={Mobilenetv2: Inverted residuals and linear bottlenecks},
  author={Sandler, Mark and Howard, Andrew and Zhu, Menglong and Zhmoginov, Andrey and Chen, Liang-Chieh},
  booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
  pages={4510--4520},
  year={2018}
}
```

</details>

<!-- [DATASET] -->

<details>
<summary align="right"><a href="https://link.springer.com/chapter/10.1007/978-3-030-58545-7_12">COCO-WholeBody-Hand (ECCV'2020)</a></summary>

```bibtex
@inproceedings{jin2020whole,
  title={Whole-Body Human Pose Estimation in the Wild},
  author={Jin, Sheng and Xu, Lumin and Xu, Jin and Wang, Can and Liu, Wentao and Qian, Chen and Ouyang, Wanli and Luo, Ping},

title={Whole-Body Human Pose Estimation in the Wild},
  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
  year={2020}
}
```

Results on COCO-WholeBody-Hand val set

|                            Arch                            | Input Size | PCK@0.2 |  AUC  | EPE  |                            ckpt                            |                            log                             |
| :--------------------------------------------------------: | :--------: | :-----: | :---: | :--: | :--------------------------------------------------------: | :--------------------------------------------------------: |

| [pose_mobilenetv2](/configs/hand_2d_keypoint/topdown_heatmap/coco_wholebody_hand/td-hm_mobilenetv2_8xb32-210e_coco-wholebody-hand-256x256.py) |  256x256   |  0.795  | 0.829 | 4.77 | [ckpt](https://download.openmmlab.com/mmpose/hand/mobilenetv2/mobilenetv2_coco_wholebody_hand_256x256-06b8c877_20210909.pth) | [log](https://download.openmmlab.com/mmpose/hand/mobilenetv2/mobilenetv2_coco_wholebody_hand_256x256_20210909.log.json) |

# How to Fine-tune with Custom Dataset

In most scenarios, we want to apply a pre-trained model without training from scratch, which might possibly introduce extra uncertainties about the model convergency and therefore, is time-consuming.
The common sense is to learn from previous models trained on large dataset, which can hopefully provide better knowledge than a random beginner. Roughly speaking, this process is as known as fine-tuning.

Models pre-trained on the ImageNet dataset have been demonstrated to be effective for other datasets and other downstream tasks.
Hence, this tutorial provides instructions for users to use the models provided in the [Model Zoo](../modelzoo_statistics.md) for other datasets to obtain better performance.

In this tutorial, we provide a practice example and some tips on how to fine-tune a model on your own dataset.

## Step-1: Prepare your dataset

Prepare your dataset following [Prepare Dataset](../user_guides/dataset_prepare.md).

And the root folder of the dataset can be like `data/custom_dataset/`.

Here, we assume you want to do supervised image-classification training, and use the sub-folder format
`CustomDataset` to organize your dataset as:

```text
data/custom_dataset/
├── train
│   ├── class_x
│   │   ├── x_1.png
│   │   ├── x_2.png
│   │   ├── x_3.png
│   │   └── ...
│   ├── class_y
│   └── ...
└── test
    ├── class_x
    │   ├── test_x_1.png
    │   ├── test_x_2.png
    │   ├── test_x_3.png
    │   └── ...
    ├── class_y
    └── ...
```

## Step-2: Choose one config as template

Here, we would like to use `configs/resnet/resnet50_8xb32_in1k.py` as the example. We first copy this config
file to the same folder and rename it as `resnet50_8xb32-ft_custom.py`.

```{tip}

```{tip}
As a convention, the last field of the config name is the dataset, e.g.,`in1k` for ImageNet dataset, `coco` for COCO dataset
```

The content of this config is:

```python
_base_ = [
    '../_base_/models/resnet50.py',           # model settings
    '../_base_/datasets/imagenet_bs32.py',    # data settings
    '../_base_/schedules/imagenet_bs256.py',  # schedule settings
    '../_base_/default_runtime.py',           # runtime settings
]
```

## Step-3: Edit the model settings

When fine-tuning a model, usually we want to load the pre-trained backbone
weights and train a new classification head from scratch.

To load the pre-trained backbone, we need to change the initialization config
of the backbone and use `Pretrained` initialization function. Besides, in the

`init_cfg`, we use `prefix='backbone'` to tell the initialization function
the prefix of the submodule that needs to be loaded in the checkpoint.

For example, `backbone` here means to load the backbone submodule. And here we
use an online checkpoint, it will be downloaded automatically during training,
you can also download the model manually and use a local path.
And then we need to modify the head according to the class numbers of the new
datasets by just changing `num_classes` in the head.

When new dataset is small and shares the domain with the pre-trained dataset,
we might want to freeze the first several stages' parameters of the
backbone, that will help the network to keep ability to extract low-level
information learnt from pre-trained model. In MMPretrain, you can simply

specify how many stages to freeze by `frozen_stages` argument. For example, to
freeze the first two stages' parameters, just use the following configs:

```{note}
Not all backbones support the `frozen_stages` argument by now. Please check
[the docs](https://mmpretrain.readthedocs.io/en/latest/api.html#module-mmpretrain.models.backbones)
to confirm if your backbone supports it.
```

```python
_base_ = [
]

# >>>>>>>>>>>>>>> Override model settings here >>>>>>>>>>>>>>>>>>>
model = dict(
    backbone=dict(
        frozen_stages=2,
        init_cfg=dict(
            type='Pretrained',

]

            checkpoint='https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_8xb32_in1k_20210831-ea4938fc.pth',
            prefix='backbone',
        )),
    head=dict(num_classes=10),
)
# <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
```

```{tip}
Here we only need to set the part of configs we want to modify, because the
inherited configs will be merged and get the entire configs.
```

## Step-4: Edit the dataset settings

To fine-tuning on a new dataset, we need to override some dataset settings, like the type of dataset, data
pipeline, etc.

```python
_base_ = [

'../_base_/models/resnet50.py',           # model settings
]

# model settings
        )),
)

# >>>>>>>>>>>>>>> Override data settings here >>>>>>>>>>>>>>>>>>>
data_root = 'data/custom_dataset'
train_dataloader = dict(
    dataset=dict(
        type='CustomDataset',
        data_root=data_root,
        ann_file='',       # We assume you are using the sub-folder format without ann_file
        data_prefix='train',
    ))
val_dataloader = dict(

ann_file='',       # We assume you are using the sub-folder format without ann_file
    ))
        data_prefix='test',
    ))
test_dataloader = val_dataloader
# <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
```

## Step-5: Edit the schedule settings (optional)

The fine-tuning hyper parameters vary from the default schedule. It usually
requires smaller learning rate and quicker decaying scheduler epochs.

```python
_base_ = [
]

'../_base_/schedules/imagenet_bs256.py',  # schedule settings
]

        )),
)

# data settings
    ))
    ))

type='CustomDataset',
    ))

# >>>>>>>>>>>>>>> Override schedule settings here >>>>>>>>>>>>>>>>>>>
# optimizer hyper-parameters
optim_wrapper = dict(
    optimizer=dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001))
# learning policy
param_scheduler = dict(
    type='MultiStepLR', by_epoch=True, milestones=[15], gamma=0.1)
# <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
```

```{tip}
Refers to [Learn about Configs](../user_guides/config.md) for more detailed configurations.
```

## Start Training

Now, we have finished the fine-tuning config file as following:

```python
_base_ = [

```python
_base_ = [
]

        )),
)

    ))

data_prefix='train',
    ))
    ))

# schedule settings
```

Here we use 8 GPUs on your computer to train the model with the following command:

```shell
bash tools/dist_train.sh configs/resnet/resnet50_8xb32-ft_custom.py 8
```

Also, you can use only one GPU to train the model with the following command:

```shell
python tools/train.py configs/resnet/resnet50_8xb32-ft_custom.py
```

But wait, an important config need to be changed if using one GPU. We need to
change the dataset config as following:

```python

```

```python
    batch_size=256,
    ))
    ))
```

It's because our training schedule is for a batch size of 256. If using 8 GPUs,
just use `batch_size=32` config in the base config file for every GPU, and the total batch
size will be 256. But if using one GPU, you need to change it to 256 manually to
match the training schedule.

However, a larger batch size requires a larger GPU memory, and here are several simple tricks to save the GPU
memory:

1. Enable Automatic-Mixed-Precision training.

   ```shell
   python tools/train.py configs/resnet/resnet50_8xb32-ft_custom.py --amp
   ```

2. Use a smaller batch size, like `batch_size=32` instead of 256, and enable the auto learning rate scaling.

   ```shell
   python tools/train.py configs/resnet/resnet50_8xb32-ft_custom.py --auto-scale-lr
   ```

   The auto learning rate scaling will adjust the learning rate according to the actual batch size and the
   `auto_scale_lr.base_batch_size` (You can find it in the base config
   `configs/_base_/schedules/imagenet_bs256.py`)

```{note}
Most of these tricks may influence the training performance slightly.
```

### Apply pre-trained model with command line

`configs/_base_/schedules/imagenet_bs256.py`)

```{note}
```

If you don't want to modify the configs, you could use `--cfg-options` to add your pre-trained model path to `init_cfg`.

For example, the command below will also load pre-trained model.

```shell
bash tools/dist_train.sh configs/resnet/resnet50_8xb32-ft_custom.py 8 \
    --cfg-options model.backbone.init_cfg.type='Pretrained' \
    model.backbone.init_cfg.checkpoint='https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_resnet50_8xb512-amp-coslr-100e_in1k/mocov3_resnet50_8xb512-amp-coslr-100e_in1k_20220927-f1144efa.pth' \
    model.backbone.init_cfg.prefix='backbone' \
```

# Migrate Configuration File from MMDetection 2.x to 3.x

The configuration file of MMDetection 3.x has undergone significant changes in comparison to the 2.x version. This document explains how to migrate 2.x configuration files to 3.x.

In the previous tutorial [Learn about Configs](../user_guides/config.md), we used Mask R-CNN as an example to introduce the configuration file structure of MMDetection 3.x. Here, we will follow the same structure to demonstrate how to migrate 2.x configuration files to 3.x.

## Model Configuration

There have been no major changes to the model configuration in 3.x compared to 2.x. For the model's backbone, neck, head, as well as train_cfg and test_cfg, the parameters remain the same as in version 2.x.

On the other hand, we have added the `DataPreprocessor` module in MMDetection 3.x. The configuration for the `DataPreprocessor` module is located in `model.data_preprocessor`. It is used to preprocess the input data, such as normalizing input images and padding images of different sizes into batches, and loading images from memory to VRAM. This configuration replaces the `Normalize` and `Pad` modules in `train_pipeline` and `test_pipeline` of the earlier version.

<table class="docutils">
<tr>
<td>2.x Config</td>
<td>

```python
# Image normalization parameters
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53],
    std=[58.395, 57.12, 57.375],
    to_rgb=True)
pipeline=[
    ...,
    dict(type='Normalize', **img_norm_cfg),
    dict(type='Pad', size_divisor=32),  # Padding the image to multiples of 32
    ...
]
```

</td>
<tr>
<td>3.x Config</td>
<td>

```python
    data_preprocessor=dict(
        type='DetDataPreprocessor',
        # Image normalization parameters

...
]
```

</td>
<tr>
<td>

```python
        mean=[123.675, 116.28, 103.53],
        std=[58.395, 57.12, 57.375],
        bgr_to_rgb=True,
        # Image padding parameters
        pad_mask=True,  # In instance segmentation, the mask needs to be padded
        pad_size_divisor=32)  # Padding the image to multiples of 32
)

```

</td>
</tr>
</table>

## Dataset and Evaluator Configuration

The dataset and evaluator configurations have undergone major changes compared to version 2.x. We will introduce how to migrate from version 2.x to version 3.x from three aspects: Dataloader and Dataset, Data transform pipeline, and Evaluator configuration.

### Dataloader and Dataset Configuration

In the new version, we set the data loading settings consistent with PyTorch's official DataLoader,
making it easier for users to understand and get started with.

We put the data loading settings for training, validation, and testing separately in `train_dataloader`, `val_dataloader`, and `test_dataloader`.
Users can set different parameters for these dataloaders.
The input parameters are basically the same as those required by [PyTorch DataLoader](https://pytorch.org/docs/stable/data.html?highlight=dataloader#torch.utils.data.DataLoader).

This way, we put the unconfigurable parameters in version 2.x, such as `sampler`, `batch_sampler`, and `persistent_workers`, in the configuration file, so that users can set dataloader parameters more flexibly.

Users can set the dataset configuration through `train_dataloader.dataset`, `val_dataloader.dataset`, and `test_dataloader.dataset`, which correspond to `data.train`, `data.val`, and `data.test` in version 2.x.

<tr>
<td>

```python
data = dict(
    samples_per_gpu=2,
    workers_per_gpu=2,
    train=dict(
        type=dataset_type,
        ann_file=data_root + 'annotations/instances_train2017.json',
        img_prefix=data_root + 'train2017/',
        pipeline=train_pipeline),
    val=dict(
        ann_file=data_root + 'annotations/instances_val2017.json',
        img_prefix=data_root + 'val2017/',
        pipeline=test_pipeline),
    test=dict(
        pipeline=test_pipeline))
```

</td>
<tr>
<td>

```python
    batch_size=2,
    num_workers=2,
    persistent_workers=True,  # Avoid recreating subprocesses after each iteration

```

</td>
<tr>
<td>

```python
    sampler=dict(type='DefaultSampler', shuffle=True),  # Default sampler, supports both distributed and non-distributed training
    batch_sampler=dict(type='AspectRatioBatchSampler'),  # Default batch_sampler, used to ensure that images in the batch have similar aspect ratios, so as to better utilize graphics memory
        ann_file='annotations/instances_train2017.json',
        data_prefix=dict(img='train2017/'),
        filter_cfg=dict(filter_empty_gt=True, min_size=32),
        pipeline=train_pipeline))
# In version 3.x, validation and test dataloaders can be configured independently
    batch_size=1,
    persistent_workers=True,
    drop_last=False,

    sampler=dict(type='DefaultSampler', shuffle=False),
        ann_file='annotations/instances_val2017.json',
        data_prefix=dict(img='val2017/'),
        test_mode=True,
test_dataloader = val_dataloader  # The configuration of the testing dataloader is the same as that of the validation dataloader, which is omitted here

```

</td>
</tr>
</table>

### Data Transform Pipeline Configuration

```

</td>
</tr>
</table>

As mentioned earlier, we have separated the normalization and padding configurations for images from the `train_pipeline` and `test_pipeline`, and have placed them in `model.data_preprocessor` instead. Hence, in the 3.x version of the pipeline, we no longer require the `Normalize` and `Pad` transforms.

At the same time, we have also refactored the transform responsible for packing the data format, and have merged the `Collect` and `DefaultFormatBundle` transforms into `PackDetInputs`. This transform is responsible for packing the data from the data pipeline into the input format of the model. For more details on the input format conversion, please refer to the [data flow documentation](../advanced_guides/data_flow.md).

Below, we will use the `train_pipeline` of Mask R-CNN as an example, to demonstrate how to migrate from the 2.x configuration to the 3.x configuration:

<tr>

<tr>
<td>

```python
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='LoadAnnotations', with_bbox=True),
    dict(type='Resize', img_scale=(1333, 800), keep_ratio=True),
    dict(type='RandomFlip', flip_ratio=0.5),
    dict(type='Pad', size_divisor=32),
    dict(type='DefaultFormatBundle'),
    dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels']),
]
```

</td>
<tr>
<td>

```python
    dict(type='Resize', scale=(1333, 800), keep_ratio=True),
    dict(type='RandomFlip', prob=0.5),

dict(type='LoadImageFromFile'),
    dict(type='PackDetInputs')
]
```

</td>
</tr>
</table>

For the `test_pipeline`, apart from removing the `Normalize` and `Pad` transforms, we have also separated the data augmentation for testing (TTA) from the normal testing process, and have removed `MultiScaleFlipAug`. For more information on how to use the new TTA version, please refer to the [TTA documentation](../advanced_guides/tta.md).

Below, we will again use the `test_pipeline` of Mask R-CNN as an example, to demonstrate how to migrate from the 2.x configuration to the 3.x configuration:

<tr>
<td>

```python
test_pipeline = [
    dict(
        type='MultiScaleFlipAug',
        img_scale=(1333, 800),
        flip=False,
        transforms=[

<td>

```python
    dict(
            dict(type='Resize', keep_ratio=True),
            dict(type='RandomFlip'),
            dict(type='Normalize', **img_norm_cfg),
            dict(type='Pad', size_divisor=32),
            dict(type='ImageToTensor', keys=['img']),
            dict(type='Collect', keys=['img']),
        ])
]
```

</td>
<tr>
<td>

```python
    dict(
        type='PackDetInputs',
        meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',
                   'scale_factor'))
]
```

</td>
</tr>
</table>

dict(
]
```

</td>
</tr>
</table>

In addition, we have also refactored some data augmentation transforms. The following table lists the mapping between the transforms used in the 2.x version and the 3.x version:

<thead>
  <tr>
    <th>Name</th>
    <th>2.x Config</th>
    <th>3.x Config</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>Resize</td>
<td>

```python
dict(type='Resize',
     img_scale=(1333, 800),
     keep_ratio=True)
```

</td>

<td>

```python
     scale=(1333, 800),
```

</td>
  </tr>
  <tr>
    <td>RandomResize</td>
<td>

```python
dict(
    type='Resize',
    img_scale=[
        (1333, 640), (1333, 800)],
    multiscale_mode='range',
    keep_ratio=True)
```

</td>
<td>

```python
dict(
    type='RandomResize',
    scale=[

(1333, 640), (1333, 800)],
```

</td>
<td>

```python
dict(
    scale=[
```

</td>
  </tr>
  <tr>
    <td>RandomChoiceResize</td>
<td>

```python
dict(
        (1333, 640), (1333, 672),
        (1333, 704), (1333, 736),
        (1333, 768), (1333, 800)],
    multiscale_mode='value',
```

</td>
<td>

```python
dict(
    type='RandomChoiceResize',
    scales=[
```

</td>
  </tr>

<tr>
    <td>RandomFlip</td>
<td>

```python
dict(type='RandomFlip', flip_ratio=0.5)
```

</td>
<td>

```python
dict(type='RandomFlip', prob=0.5)
```

</td>
  </tr>

</tbody>
</table>

### 评测器配置

<tr>
<td>

```python
```

</td>
<td>

```python
```

</td>
  </tr>

</tbody>
</table>

### 评测器配置

In version 3.x, model accuracy evaluation is no longer tied to the dataset, but is instead accomplished through the use of an Evaluator.
The Evaluator configuration is divided into two parts: `val_evaluator` and `test_evaluator`. The `val_evaluator` is used for validation dataset evaluation, while the `test_evaluator` is used for testing dataset evaluation.
This corresponds to the `evaluation` field in version 2.x.

The following table shows the corresponding relationship between Evaluators in version 2.x and 3.x.

<thead>
  <tr>
    <th>Metric Name</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>COCO</td>
<td>

```python
    val=dict(
        type='CocoDataset',

<tr>
  </tr>
</thead>
<tbody>
  <tr>
<td>

```python
    val=dict(
        ann_file=data_root + 'annotations/instances_val2017.json'))
evaluation = dict(metric=['bbox', 'segm'])
```

</td>

<td>

```python
val_evaluator = dict(
    type='CocoMetric',
    ann_file=data_root + 'annotations/instances_val2017.json',
    metric=['bbox', 'segm'],
    format_only=False)
```

</td>
  </tr>
  <tr>
    <td>Pascal VOC</td>
<td>

```python
    val=dict(
        ann_file=data_root + 'VOC2007/ImageSets/Main/test.txt'))
evaluation = dict(metric='mAP')
```

</td>
<td>

```python
    type='VOCMetric',
    metric='mAP',
    eval_mode='11points')
```

</td>
  </tr>
  <tr>
    <td>OpenImages</td>
<td>

```python
    val=dict(
        type='OpenImagesDataset',

type='VOCMetric',
```

</td>
  </tr>
  <tr>
<td>

```python
    val=dict(
        ann_file=data_root + 'annotations/validation-annotations-bbox.csv',
        img_prefix=data_root + 'OpenImages/validation/',
        label_file=data_root + 'annotations/class-descriptions-boxable.csv',
        hierarchy_file=data_root +
        'annotations/bbox_labels_600_hierarchy.json',
        meta_file=data_root + 'annotations/validation-image-metas.pkl',
        image_level_ann_file=data_root +
        'annotations/validation-annotations-human-imagelabels-boxable.csv'))
evaluation = dict(interval=1, metric='mAP')
```

</td>
<td>

```python
    type='OpenImagesMetric',
    iou_thrs=0.5,
    ioa_thrs=0.5,
    use_group_of=True,
    get_supercategory=True)
```

</td>
  </tr>

<tr>
    <td>CityScapes</td>
<td>

```python
    val=dict(

iou_thrs=0.5,
```

</td>
  </tr>

<tr>
<td>

```python
    val=dict(
        type='CityScapesDataset',
        ann_file=data_root +
        'annotations/instancesonly_filtered_gtFine_val.json',
        img_prefix=data_root + 'leftImg8bit/val/',
```

</td>
<td>

```python
val_evaluator = [
    dict(
        type='CocoMetric',
        metric=['bbox', 'segm']),
    dict(
        type='CityScapesMetric',
        seg_prefix=data_root + '/gtFine/val',
        outfile_prefix='./work_dirs/cityscapes_metric/instance')
]
```

</td>
  </tr>

</tbody>
</table>

## Configuration for Training and Testing

<tr>

outfile_prefix='./work_dirs/cityscapes_metric/instance')
]
```

</td>
  </tr>

</tbody>
</table>

<tr>
<td>

```python
runner = dict(
    type='EpochBasedRunner',  # Type of training loop
    max_epochs=12)  # Maximum number of training epochs
evaluation = dict(interval=2)  # Interval for evaluation, check the performance every 2 epochs
```

</td>
<tr>
<td>

```python
train_cfg = dict(
    type='EpochBasedTrainLoop',  # Type of training loop, please refer to https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py
    max_epochs=12,  # Maximum number of training epochs
    val_interval=2)  # Interval for validation, check the performance every 2 epochs
val_cfg = dict(type='ValLoop')  # Type of validation loop
test_cfg = dict(type='TestLoop')  # Type of testing loop
```

</td>
</tr>
</table>

## Optimization Configuration

```

</td>
</tr>
</table>

The configuration for optimizer and gradient clipping is moved to the `optim_wrapper` field.
The following table shows the correspondences for optimizer configuration between 2.x version and 3.x version:

<tr>
<td>

```python
optimizer = dict(
    type='SGD',  # Optimizer: Stochastic Gradient Descent
    lr=0.02,  # Base learning rate
    momentum=0.9,  # SGD with momentum
    weight_decay=0.0001)  # Weight decay
optimizer_config = dict(grad_clip=None)  # Configuration for gradient clipping, set to None to disable
```

</td>
<tr>
<td>

```python
optim_wrapper = dict(  # Configuration for the optimizer wrapper
    type='OptimWrapper',  # Type of optimizer wrapper, you can switch to AmpOptimWrapper to enable mixed precision training

```python
    optimizer=dict(  # Optimizer configuration, supports various PyTorch optimizers, please refer to https://pytorch.org/docs/stable/optim.html#algorithms
        type='SGD',  # SGD
        lr=0.02,  # Base learning rate
        momentum=0.9,  # SGD with momentum
        weight_decay=0.0001),  # Weight decay
    clip_grad=None,  # Configuration for gradient clipping, set to None to disable. For usage, please see https://mmengine.readthedocs.io/en/latest/tutorials/optimizer.html
    )
```

</td>
</tr>
</table>

)
```

</td>
</tr>
</table>

The configuration for learning rate is also moved from the `lr_config` field to the `param_scheduler` field. The `param_scheduler` configuration is more similar to PyTorch's learning rate scheduler and more flexible. The following table shows the correspondences for learning rate configuration between 2.x version and 3.x version:

<tr>
<td>

```python
lr_config = dict(
    policy='step',  # Use multi-step learning rate strategy during training
    warmup='linear',  # Use linear learning rate warmup
    warmup_iters=500,  # End warmup at iteration 500
    warmup_ratio=0.001,  # Coefficient for learning rate warmup
    step=[8, 11],  # Learning rate decay at which epochs
    gamma=0.1)  # Learning rate decay coefficient

```

</td>
<tr>
<td>

```python
param_scheduler = [
    dict(
        type='LinearLR',  # Use linear learning rate warmup

gamma=0.1)  # Learning rate decay coefficient

```

</td>
<tr>
<td>

```python
    dict(
        start_factor=0.001, # Coefficient for learning rate warmup
        by_epoch=False,  # Update the learning rate during warmup at each iteration
        begin=0,  # Starting from the first iteration
        end=500),  # End at the 500th iteration
    dict(
        type='MultiStepLR',  # Use multi-step learning rate strategy during training
        by_epoch=True,  # Update the learning rate at each epoch
        begin=0,   # Starting from the first epoch
        end=12,  # Ending at the 12th epoch
        milestones=[8, 11],  # Learning rate decay at which epochs
        gamma=0.1)  # Learning rate decay coefficient
]

```

</td>
</tr>
</table>

end=12,  # Ending at the 12th epoch
]

```

</td>
</tr>
</table>

For information on how to migrate other learning rate adjustment policies, please refer to the [learning rate migration document of MMEngine](https://mmengine.readthedocs.io/zh_CN/latest/migration/param_scheduler.html).

## Migration of Other Configurations

### Configuration for Saving Checkpoints

<thead>
  <tr>
    <th>Function</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>Set Save Interval</td>
<td>

```python
checkpoint_config = dict(
    interval=1)
```

</td>

<td>

```python
default_hooks = dict(
    checkpoint=dict(
        type='CheckpointHook',
        interval=1))
```

</td>
  </tr>

<tr>
    <td>Save Best Model</td>
<td>

```python
evaluation = dict(
    save_best='auto')
```

</td>
<td>

```python

type='CheckpointHook',
```

</td>
  </tr>

<tr>
<td>

```python
```

</td>
<td>

```python
        save_best='auto'))
```

</td>
  </tr>

<tr>
    <td>Keep Latest Model</td>
<td>

```python
    max_keep_ckpts=3)
```

</td>
<td>

```python
        max_keep_ckpts=3))
```

</td>
  </tr>

</tbody>
</table>

### Logging Configuration

In MMDetection 3.x, the logging and visualization of the log are carried out respectively by the logger and visualizer in MMEngine. The following table shows the comparison between the configuration of printing logs and visualizing logs in MMDetection 2.x and 3.x.

<thead>
  <tr>
  </tr>
</thead>

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
    <td>Set Log Printing Interval</td>
<td>

```python
log_config = dict(interval=50)
```

</td>

<td>

```python
    logger=dict(type='LoggerHook', interval=50))
# Optional： set moving average window size
log_processor = dict(
    type='LogProcessor', window_size=50)
```

</td>
  </tr>

<tr>
    <td>Use TensorBoard or WandB to visualize logs</td>
<td>

```python
log_config = dict(
    interval=50,
    hooks=[
        dict(type='TextLoggerHook'),
        dict(type='TensorboardLoggerHook'),
        dict(type='MMDetWandbHook',
             init_kwargs={
                'project': 'mmdetection',
                'group': 'maskrcnn-r50-fpn-1x-coco'
             },
             interval=50,
             log_checkpoint=True,
             log_checkpoint_metadata=True,
             num_eval_images=100)
    ])
```

</td>
<td>

},
    ])
```

</td>
<td>

```python
vis_backends = [
    dict(type='LocalVisBackend'),
    dict(type='TensorboardVisBackend'),
    dict(type='WandbVisBackend',
         init_kwargs={
            'project': 'mmdetection',
            'group': 'maskrcnn-r50-fpn-1x-coco'
         })
]
visualizer = dict(
    type='DetLocalVisualizer',
    vis_backends=vis_backends,
    name='visualizer')
```

</td>
  </tr>

</tbody>
</table>

For visualization-related tutorials, please refer to [Visualization Tutorial](../user_guides/visualization.md) of MMDetection.

### Runtime Configuration

The runtime configuration fields in version 3.x have been adjusted, and the specific correspondence is as follows:

<thead>
  <tr>
  </tr>
</thead>
<tbody>

<tr>
<td>

```python

<thead>
  <tr>
  </tr>
</thead>
<tbody>

<tr>
<td>

```python
cudnn_benchmark = False
opencv_num_threads = 0
mp_start_method = 'fork'
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = None
resume_from = None

```

</td>
<td>

```python
env_cfg = dict(
    cudnn_benchmark=False,
    mp_cfg=dict(mp_start_method='fork',
                opencv_num_threads=0),
    dist_cfg=dict(backend='nccl'))
resume = False
```

</td>
  </tr>

</tbody>
</table>

# 准备 Deraining 数据集

```bibtex
@inproceedings{Zamir2021Restormer,
  title={Restormer: Efficient Transformer for High-Resolution Image Restoration},
  author={Syed Waqas Zamir and Aditya Arora and Salman Khan and Munawar Hayat and Fahad Shahbaz Khan and Ming-Hsuan Yang},
  booktitle={CVPR},
  year={2022}
}
```

测试数据集(Rain100H, Rain100L, Test100, Test1200, Test2800)可以从 [此处](https://drive.google.com/file/d/1P_-RAvltEoEhfT-9GrWRdpEi6NSswTs8/) 下载。

文件目录结构应如下所示：

```text
mmagic
├── mmagic
├── tools
├── configs
├── data
|   ├── Rain100H
|   |   ├── input
|   |   ├── target
|   ├── Rain100L
|   ├── Test100
|   ├── Test1200
|   ├── Test2800
```

Apart from training/testing scripts, We provide lots of useful tools under the
`tools/` directory.

## Log Analysis

`tools/analysis_tools/analyze_logs.py` plots loss/mAP curves given a training
log file. Run `pip install seaborn` first to install the dependency.

```shell
python tools/analysis_tools/analyze_logs.py plot_curve [--keys ${KEYS}] [--eval-interval ${EVALUATION_INTERVAL}] [--title ${TITLE}] [--legend ${LEGEND}] [--backend ${BACKEND}] [--style ${STYLE}] [--out ${OUT_FILE}]
```

![loss curve image](../../../resources/loss_curve.png)

Examples:

- Plot the classification loss of some run.

  ```shell
  python tools/analysis_tools/analyze_logs.py plot_curve log.json --keys loss_cls --legend loss_cls
  ```

- Plot the classification and regression loss of some run, and save the figure to a pdf.

  ```shell
  python tools/analysis_tools/analyze_logs.py plot_curve log.json --keys loss_cls loss_bbox --out losses.pdf
  ```

- Compare the bbox mAP of two runs in the same figure.

```shell
  ```

  ```shell
  python tools/analysis_tools/analyze_logs.py plot_curve log1.json log2.json --keys bbox_mAP --legend run1 run2
  ```

- Compute the average training speed.

  ```shell
  python tools/analysis_tools/analyze_logs.py cal_train_time log.json [--include-outliers]
  ```

  The output is expected to be like the following.

  ```text
  -----Analyze train time of work_dirs/some_exp/20190611_192040.log.json-----
  slowest epoch 11, average time is 1.2024
  fastest epoch 1, average time is 1.1909
  time std over epochs is 0.0028
  average iter time: 1.1959 s/iter
  ```

## Result Analysis

`tools/analysis_tools/analyze_results.py` calculates single image mAP and saves or shows the topk images with the highest and lowest scores based on prediction results.

**Usage**

```shell

**Usage**

```shell
python tools/analysis_tools/analyze_results.py \
      ${CONFIG} \
      ${PREDICTION_PATH} \
      ${SHOW_DIR} \
      [--show] \
      [--wait-time ${WAIT_TIME}] \
      [--topk ${TOPK}] \
      [--show-score-thr ${SHOW_SCORE_THR}] \
      [--cfg-options ${CFG_OPTIONS}]
```

Description of all arguments:

- `config` : The path of a model config file.
- `prediction_path`:  Output result file in pickle format from `tools/test.py`
- `show_dir`: Directory where painted GT and detection images will be saved
- `--show`：Determines whether to show painted images, If not specified, it will be set to `False`
- `--wait-time`: The interval of show (s), 0 is block
- `--topk`: The number of saved images that have the highest and lowest `topk` scores after sorting. If not specified, it will be set to `20`.

- `--show-score-thr`:  Show score threshold. If not specified, it will be set to `0`.
- `--cfg-options`: If specified, the key-value pair optional cfg will be merged into config file

**Examples**:

Assume that you have got result file in pickle format from `tools/test.py`  in the path './result.pkl'.

1. Test Faster R-CNN and visualize the results, save images to the directory `results/`

```shell
       configs/faster_rcnn/faster-rcnn_r50_fpn_1x_coco.py \
       result.pkl \
       results \
       --show
```

2. Test Faster R-CNN and specified topk to 50, save images to the directory `results/`

```shell

```shell
       results \
       --topk 50
```

3. If you want to filter the low score prediction results, you can specify the `show-score-thr` parameter

```shell
       results \
       --show-score-thr 0.3
```

## Visualization

### Visualize Datasets

`tools/analysis_tools/browse_dataset.py` helps the user to browse a detection dataset (both
images and bounding box annotations) visually, or save the image to a
designated directory.

```shell
python tools/misc/browse_dataset.py ${CONFIG} [-h] [--skip-type ${SKIP_TYPE[SKIP_TYPE...]}] [--output-dir ${OUTPUT_DIR}] [--not-show] [--show-interval ${SHOW_INTERVAL}]
```

### Visualize Models

First, convert the model to ONNX as described

```

[here](#convert-mmdetection-model-to-onnx-experimental).
Note that currently only RetinaNet is supported, support for other models
will be coming in later versions.
The converted model could be visualized by tools like [Netron](https://github.com/lutzroeder/netron).

### Visualize Predictions

If you need a lightweight GUI for visualizing the detection results, you can refer [DetVisGUI project](https://github.com/Chien-Hung/DetVisGUI/tree/mmdetection).

## Error Analysis

`tools/analysis_tools/coco_error_analysis.py` analyzes COCO results per category and by
different criterion. It can also make a plot to provide useful information.

```shell
python tools/analysis_tools/coco_error_analysis.py ${RESULT} ${OUT_DIR} [-h] [--ann ${ANN}] [--types ${TYPES[TYPES...]}]
```

Example:

```shell
```

Example:

Assume that you have got [Mask R-CNN checkpoint file](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_fpn_1x_coco/mask_rcnn_r50_fpn_1x_coco_20200205-d4b0c5d6.pth) in the path 'checkpoint'. For other checkpoints, please refer to our [model zoo](./model_zoo.md).

You can modify the test_evaluator to save the results bbox by:

1. Find which dataset in 'configs/base/datasets' the current config corresponds to.
2. Replace the original test_evaluator and test_dataloader with test_evaluator and test_dataloader in the comment in dataset config.
3. Use the following command to get the results bbox and segmentation json file.

```shell
python tools/test.py \
       configs/mask_rcnn/mask-rcnn_r50_fpn_1x_coco.py \
       checkpoint/mask_rcnn_r50_fpn_1x_coco_20200205-d4b0c5d6.pth \
```

```shell
```

1. Get COCO bbox error results per category , save analyze result images to the directory(In  [config](../../../configs/_base_/datasets/coco_instance.py) the default directory is './work_dirs/coco_instance/test')

```shell
python tools/analysis_tools/coco_error_analysis.py \
       results.bbox.json \
       results \
       --ann=data/coco/annotations/instances_val2017.json \
```

2. Get COCO segmentation error results per category , save analyze result images to the directory

```shell
       results.segm.json \
       results \
       --types='segm'
```

## Model Serving

In order to serve an `MMDetection` model with [`TorchServe`](https://pytorch.org/serve/), you can follow the steps:

### 1. Install TorchServe

--types='segm'
```

Suppose you have a `Python` environment with `PyTorch` and `MMDetection` successfully installed,
then you could run the following command to install `TorchServe` and its dependencies.
For more other installation options, please refer to the [quick start](https://github.com/pytorch/serve/blob/master/README.md#serve-a-model).

```shell
python -m pip install torchserve torch-model-archiver torch-workflow-archiver nvgpu
```

**Note**: Please refer to [torchserve docker](https://github.com/pytorch/serve/blob/master/docker/README.md) if you want to use `TorchServe` in docker.

### 2. Convert model from MMDetection to TorchServe

```shell
python tools/deployment/mmdet2torchserve.py ${CONFIG_FILE} ${CHECKPOINT_FILE} \
--output-folder ${MODEL_STORE} \
--model-name ${MODEL_NAME}
```

### 3. Start `TorchServe`

```shell
```

```shell
torchserve --start --ncs \
  --model-store ${MODEL_STORE} \
  --models  ${MODEL_NAME}.mar
```

### 4. Test deployment

```shell
curl -O curl -O https://raw.githubusercontent.com/pytorch/serve/master/docs/images/3dogs.jpg
curl http://127.0.0.1:8080/predictions/${MODEL_NAME} -T 3dogs.jpg
```

You should obtain a response similar to:

```json
[
  {
    "class_label": 16,
    "class_name": "dog",
    "bbox": [
      294.63409423828125,
      203.99111938476562,
      417.048583984375,
      281.62744140625
    ],
    "score": 0.9987992644309998
  },
  {
    "bbox": [
      404.26019287109375,
      126.0080795288086,
      574.5091552734375,
      293.6662292480469
    ],
    "score": 0.9979367256164551
  },
  {

"bbox": [
    ],
  },
  {
    "bbox": [
      197.2144775390625,
      93.3067855834961,
      307.8505554199219,
      276.7560119628906
    ],
    "score": 0.993338406085968
  }
]
```

#### Compare results

And you can use `test_torchserver.py` to compare result of `TorchServe` and `PyTorch`, and visualize them.

```shell
python tools/deployment/test_torchserver.py ${IMAGE_FILE} ${CONFIG_FILE} ${CHECKPOINT_FILE} ${MODEL_NAME}
[--inference-addr ${INFERENCE_ADDR}] [--device ${DEVICE}] [--score-thr ${SCORE_THR}] [--work-dir ${WORK_DIR}]
```

Example:

```shell
python tools/deployment/test_torchserver.py \
demo/demo.jpg \
configs/yolo/yolov3_d53_8xb8-320-273e_coco.py \
checkpoint/yolov3_d53_320_273e_coco-421362b6.pth \
yolov3 \
--work-dir ./work-dir
```

### 5. Stop `TorchServe`

```shell
torchserve --stop

yolov3 \
```

```shell
```

## Model Complexity

`tools/analysis_tools/get_flops.py` is a script adapted from [flops-counter.pytorch](https://github.com/sovrasov/flops-counter.pytorch) to compute the FLOPs and params of a given model.

```shell
python tools/analysis_tools/get_flops.py ${CONFIG_FILE} [--shape ${INPUT_SHAPE}]
```

You will get the results like this.

```text
==============================
Input shape: (3, 1280, 800)
Flops: 239.32 GFLOPs
Params: 37.74 M
```

**Note**: This tool is still experimental and we do not guarantee that the
number is absolutely correct. You may well use the result for simple
comparisons, but double check it before you adopt it in technical reports or papers.

1. FLOPs are related to the input shape while parameters are not. The default

   input shape is (1, 3, 1280, 800).
2. Some operators are not counted into FLOPs like GN and custom operators. Refer to [`mmcv.cnn.get_model_complexity_info()`](https://github.com/open-mmlab/mmcv/blob/2.x/mmcv/cnn/utils/flops_counter.py) for details.
3. The FLOPs of two-stage detectors is dependent on the number of proposals.

## Model conversion

### MMDetection model to ONNX

We provide a script to convert model to [ONNX](https://github.com/onnx/onnx) format. We also support comparing the output results between Pytorch and ONNX model for verification. More details can refer to [mmdeploy](https://github.com/open-mmlab/mmdeploy)

### MMDetection 1.x model to MMDetection 2.x

`tools/model_converters/upgrade_model_version.py` upgrades a previous MMDetection checkpoint

to the new version. Note that this script is not guaranteed to work as some
breaking changes are introduced in the new version. It is recommended to
directly use the new checkpoints.

```shell
python tools/model_converters/upgrade_model_version.py ${IN_FILE} ${OUT_FILE} [-h] [--num-classes NUM_CLASSES]
```

### RegNet model to MMDetection

`tools/model_converters/regnet2mmdet.py` convert keys in pycls pretrained RegNet models to
MMDetection style.

```shell
python tools/model_converters/regnet2mmdet.py ${SRC} ${DST} [-h]
```

### Detectron ResNet to Pytorch

`tools/model_converters/detectron2pytorch.py` converts keys in the original detectron pretrained
ResNet models to PyTorch style.

```shell
python tools/model_converters/detectron2pytorch.py ${SRC} ${DST} ${DEPTH} [-h]
```

### Prepare a model for publishing

```shell
```

`tools/model_converters/publish_model.py` helps users to prepare their model for publishing.

Before you upload a model to AWS, you may want to

1. convert model weights to CPU tensors
2. delete the optimizer states and
3. compute the hash of the checkpoint file and append the hash id to the
   filename.

```shell
python tools/model_converters/publish_model.py ${INPUT_FILENAME} ${OUTPUT_FILENAME}
```

E.g.,

```shell
python tools/model_converters/publish_model.py work_dirs/faster_rcnn/latest.pth faster_rcnn_r50_fpn_1x_20190801.pth
```

The final output filename will be `faster_rcnn_r50_fpn_1x_20190801-{hash id}.pth`.

## Dataset Conversion

`tools/data_converters/` contains tools to convert the Cityscapes dataset
and Pascal VOC dataset to the COCO format.

```shell

```shell
python tools/dataset_converters/cityscapes.py ${CITYSCAPES_PATH} [-h] [--img-dir ${IMG_DIR}] [--gt-dir ${GT_DIR}] [-o ${OUT_DIR}] [--nproc ${NPROC}]
python tools/dataset_converters/pascal_voc.py ${DEVKIT_PATH} [-h] [-o ${OUT_DIR}]
```

## Dataset Download

`tools/misc/download_dataset.py` supports downloading datasets such as COCO, VOC, and LVIS.

```shell
python tools/misc/download_dataset.py --dataset-name coco2017
python tools/misc/download_dataset.py --dataset-name voc2007
python tools/misc/download_dataset.py --dataset-name lvis
```

For users in China, these datasets can also be downloaded from [OpenDataLab](https://opendatalab.com/?source=OpenMMLab%20GitHub) with high speed:

- [COCO2017](https://opendatalab.com/COCO_2017/download?source=OpenMMLab%20GitHub)

- [VOC2007](https://opendatalab.com/PASCAL_VOC2007/download?source=OpenMMLab%20GitHub)
- [VOC2012](https://opendatalab.com/PASCAL_VOC2012/download?source=OpenMMLab%20GitHub)
- [LVIS](https://opendatalab.com/LVIS/download?source=OpenMMLab%20GitHub)

## Benchmark

### Robust Detection Benchmark

`tools/analysis_tools/test_robustness.py` and`tools/analysis_tools/robustness_eval.py`  helps users to evaluate model robustness. The core idea comes from [Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming](https://arxiv.org/abs/1907.07484). For more information how to evaluate models on corrupted images and results for a set of standard models please refer to [robustness_benchmarking.md](robustness_benchmarking.md).

### FPS Benchmark

`tools/analysis_tools/benchmark.py` helps users to calculate FPS. The FPS value includes model forward and post-processing. In order to get a more accurate value, currently only supports single GPU distributed startup mode.

```shell
python -m torch.distributed.launch --nproc_per_node=1 --master_port=${PORT} tools/analysis_tools/benchmark.py \
    ${CONFIG} \
    [--checkpoint ${CHECKPOINT}] \
    [--repeat-num ${REPEAT_NUM}] \
    [--max-iter ${MAX_ITER}] \
    [--log-interval ${LOG_INTERVAL}] \
    --launcher pytorch
```

Examples: Assuming that you have already downloaded the `Faster R-CNN` model checkpoint to the directory `checkpoints/`.

```shell
python -m torch.distributed.launch --nproc_per_node=1 --master_port=29500 tools/analysis_tools/benchmark.py \
       checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
       --launcher pytorch
```

## Miscellaneous

### Evaluating a metric

checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
```

`tools/analysis_tools/eval_metric.py` evaluates certain metrics of a pkl result file
according to a config file.

```shell
python tools/analysis_tools/eval_metric.py ${CONFIG} ${PKL_RESULTS} [-h] [--format-only] [--eval ${EVAL[EVAL ...]}]
                      [--cfg-options ${CFG_OPTIONS [CFG_OPTIONS ...]}]
                      [--eval-options ${EVAL_OPTIONS [EVAL_OPTIONS ...]}]
```

### Print the entire config

`tools/misc/print_config.py` prints the whole config verbatim, expanding all its
imports.

```shell
python tools/misc/print_config.py ${CONFIG} [-h] [--options ${OPTIONS [OPTIONS...]}]
```

## Hyper-parameter Optimization

### YOLO Anchor Optimization

`tools/analysis_tools/optimize_anchors.py` provides two method to optimize YOLO anchors.

```

One is k-means anchor cluster which refers from [darknet](https://github.com/AlexeyAB/darknet/blob/master/src/detector.c#L1421).

```shell
python tools/analysis_tools/optimize_anchors.py ${CONFIG} --algorithm k-means --input-shape ${INPUT_SHAPE [WIDTH HEIGHT]} --output-dir ${OUTPUT_DIR}
```

Another is using differential evolution to optimize anchors.

```shell
python tools/analysis_tools/optimize_anchors.py ${CONFIG} --algorithm differential_evolution --input-shape ${INPUT_SHAPE [WIDTH HEIGHT]} --output-dir ${OUTPUT_DIR}
```

E.g.,

```shell
python tools/analysis_tools/optimize_anchors.py configs/yolo/yolov3_d53_8xb8-320-273e_coco.py --algorithm differential_evolution --input-shape 608 608 --device cuda --output-dir work_dirs
```

You will get:

```
loading annotations into memory...
Done (t=9.70s)
creating index...

```

```
index created!
2021-07-19 19:37:20,951 - mmdet - INFO - Collecting bboxes from annotation...
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 117266/117266, 15874.5 task/s, elapsed: 7s, ETA:     0s

2021-07-19 19:37:28,753 - mmdet - INFO - Collected 849902 bboxes.
differential_evolution step 1: f(x)= 0.506055
differential_evolution step 2: f(x)= 0.506055
......

differential_evolution step 489: f(x)= 0.386625
2021-07-19 19:46:40,775 - mmdet - INFO Anchor evolution finish. Average IOU: 0.6133754253387451
2021-07-19 19:46:40,776 - mmdet - INFO Anchor differential evolution result:[[10, 12], [15, 30], [32, 22], [29, 59], [61, 46], [57, 116], [112, 89], [154, 198], [349, 336]]
2021-07-19 19:46:40,798 - mmdet - INFO Result saved in work_dirs/anchor_optimize_result.json
```

## Confusion Matrix

A confusion matrix is a summary of prediction results.

```

`tools/analysis_tools/confusion_matrix.py` can analyze the prediction results and plot a confusion matrix table.

First, run `tools/test.py` to save the `.pkl` detection results.

Then, run

```
python tools/analysis_tools/confusion_matrix.py ${CONFIG}  ${DETECTION_RESULTS}  ${SAVE_DIR} --show
```

And you will get a confusion matrix like this:

![confusion_matrix_example](https://user-images.githubusercontent.com/12907710/140513068-994cdbf4-3a4a-48f0-8fd8-2830d93fd963.png)

## COCO Separated & Occluded Mask Metric

Detecting occluded objects still remains a challenge for state-of-the-art object detectors.
We implemented the metric presented in paper [A Tri-Layer Plugin to Improve Occluded Detection](https://arxiv.org/abs/2210.10046) to calculate the recall of separated and occluded masks.

There are two ways to use this metric:

### Offline evaluation

We provide a script to calculate the metric with a dumped prediction file.

First, use the `tools/test.py` script to dump the detection results:

```shell
python tools/test.py ${CONFIG} ${MODEL_PATH} --out results.pkl
```

Then, run the `tools/analysis_tools/coco_occluded_separated_recall.py` script to get the recall of separated and occluded masks:

```shell
python tools/analysis_tools/coco_occluded_separated_recall.py results.pkl --out occluded_separated_recall.json
```

The output should be like this:

```
Done (t=0.51s)
processing detection results...
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 5000/5000, 109.3 task/s, elapsed: 46s, ETA:     0s

computing occluded mask recall...
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 5550/5550, 780.5 task/s, elapsed: 7s, ETA:     0s
COCO occluded mask recall: 58.79%
COCO occluded mask success num: 3263
computing separated mask recall...
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 3522/3522, 778.3 task/s, elapsed: 5s, ETA:     0s
COCO separated mask recall: 31.94%
COCO separated mask success num: 1125

+-----------+--------+-------------+
| mask type | recall | num correct |
| occluded  | 58.79% | 3263        |
| separated | 31.94% | 1125        |
Evaluation results have been saved to occluded_separated_recall.json.
```

### Online evaluation

```

We implement `CocoOccludedSeparatedMetric` which inherits from the `CocoMetic`.
To evaluate the recall of separated and occluded masks during training, just replace the evaluator metric type with `'CocoOccludedSeparatedMetric'` in your config:

```python
    type='CocoOccludedSeparatedMetric',  # modify this
test_evaluator = val_evaluator
```

Please cite the paper if you use this metric:

```latex
@article{zhan2022triocc,
    title={A Tri-Layer Plugin to Improve Occluded Detection},
    author={Zhan, Guanqi and Xie, Weidi and Zisserman, Andrew},
    journal={British Machine Vision Conference},
    year={2022}
}
```

# A 20-minute Tour to MMPose

MMPose 1.0 is built upon a brand-new framework. For developers with basic knowledge of deep learning, this tutorial provides a overview of MMPose 1.0 framework design. Whether you are **a user of the previous version of MMPose**, or **a beginner of MMPose wishing to start with v1.0**, this tutorial will show you how to build a project based on MMPose 1.0.

```{note}
This  tutorial covers what developers will concern when using MMPose 1.0:

- Overall code architecture

- How to manage modules with configs

- How to use my own custom datasets

- How to add new modules(backbone, head, loss function, etc.)
```

The content of this tutorial is organized as follows:

- [A 20 Minute Guide to MMPose Framework](#a-20-minute-guide-to-mmpose-framework)
  - [Overview](#overview)
  - [Step1: Configs](#step1-configs)
  - [Step2: Data](#step2-data)
    - [Dataset Meta Information](#dataset-meta-information)
    - [Dataset](#dataset)
    - [Pipeline](#pipeline)

- [Step1: Configs](#step1-configs)
      - [i. Augmentation](#i-augmentation)
      - [ii. Transformation](#ii-transformation)
      - [iii. Encoding](#iii-encoding)
      - [iv. Packing](#iv-packing)
  - [Step3: Model](#step3-model)
    - [Data Preprocessor](#data-preprocessor)
    - [Backbone](#backbone)
    - [Neck](#neck)
    - [Head](#head)

## Overview

![overall-en](https://user-images.githubusercontent.com/13503330/187372008-2a94bad5-5252-4155-9ae3-3da1c426f569.png)

Generally speaking, there are **five parts** developers will use during project development:

- **General:** Environment, Hook, Checkpoint, Logger, etc.

- **Data:** Dataset, Dataloader, Data Augmentation, etc.

- **Training:** Optimizer, Learning Rate Scheduler, etc.

- **Model:** Backbone, Neck, Head, Loss function, etc.

- **Evaluation:** Metric, Evaluator, etc.

Among them, modules related to **General**, **Training** and **Evaluation** are often provided by the training framework [MMEngine](https://github.com/open-mmlab/mmengine), and developers only need to call APIs and adjust the parameters.  Developers mainly focus on implementing the **Data** and **Model** parts.

## Step1: Configs

In MMPose, we use a Python file as config for the definition and parameter management of the whole project. Therefore, we strongly recommend the developers who use MMPose for the first time to refer to [Configs](./user_guides/configs.md).

Note that all new modules need to be registered using `Registry` and imported in `__init__.py` in the corresponding directory before we can create their instances from configs.

## Step2: Data

The organization of data in MMPose contains:

- Dataset Meta Information

- Dataset

- Pipeline

### Dataset Meta Information

The meta information of a pose dataset usually includes the definition of keypoints and skeleton, symmetrical characteristic, and keypoint properties (e.g. belonging to upper or lower body, weights and sigmas). These information is important in data preprocessing, model training and evaluation. In MMpose, the dataset meta information is stored in configs files under `$MMPOSE/configs/_base_/datasets/`.

To use a custom dataset in MMPose, you need to add a new config file of the dataset meta information. Take the MPII dataset (`$MMPOSE/configs/_base_/datasets/mpii.py`) as an example. Here is its dataset information:

```Python
dataset_info = dict(
    dataset_name='mpii',
    paper_info=dict(
        author='Mykhaylo Andriluka and Leonid Pishchulin and '
        'Peter Gehler and Schiele, Bernt',

```Python
        title='2D Human Pose Estimation: New Benchmark and '
        'State of the Art Analysis',
        container='IEEE Conference on Computer Vision and '
        'Pattern Recognition (CVPR)',
        year='2014',
        homepage='http://human-pose.mpi-inf.mpg.de/',
    ),
    keypoint_info={
        0:
        dict(
            name='right_ankle',
            id=0,
            color=[255, 128, 0],
            type='lower',
            swap='left_ankle'),
        ## omitted
    },
    skeleton_info={
        0:
        dict(link=('right_ankle', 'right_knee'), id=0, color=[255, 128, 0]),
        ## omitted
    },
    joint_weights=[
        1.5, 1.2, 1., 1., 1.2, 1.5, 1., 1., 1., 1., 1.5, 1.2, 1., 1., 1.2, 1.5
    ],
    # Adapted from COCO dataset.
    sigmas=[

## omitted
    },
    ],
    sigmas=[
        0.089, 0.083, 0.107, 0.107, 0.083, 0.089, 0.026, 0.026, 0.026, 0.026,
        0.062, 0.072, 0.179, 0.179, 0.072, 0.062
    ])
```

In the model config, the user needs to specify the metainfo path of the custom dataset (e.g. `$MMPOSE/configs/_base_/datasets/custom.py`) as follows:\`\`\`

```python
# dataset and dataloader settings
dataset_type = 'MyCustomDataset' # or 'CocoDataset'

        data_root='root/of/your/train/data',
        ann_file='path/to/your/train/json',
        data_prefix=dict(img='path/to/your/train/img'),
        # specify the new dataset meta information config file
        metainfo=dict(from_file='configs/_base_/datasets/custom.py'),
        ...),
    )

# specify the new dataset meta information config file
        ...),
    )

        data_root='root/of/your/val/data',
        ann_file='path/to/your/val/json',
        data_prefix=dict(img='path/to/your/val/img'),
        ...),
    )

```

### Dataset

To use custom dataset in MMPose, we recommend converting the annotations into a supported format (e.g. COCO or MPII) and directly using our implementation of the corresponding dataset. If this is not applicable, you may need to implement your own dataset class.

Most 2D keypoint datasets in MMPose **organize the annotations in a COCO-like style**. Thus we provide a base class [BaseCocoStyleDataset](mmpose/datasets/datasets/base/base_coco_style_dataset.py) for these datasets. We recommend that users subclass `BaseCocoStyleDataset` and override the methods as needed (usually `__init__()` and `_load_annotations()`) to extend to a new custom 2D keypoint dataset.

```{note}
Please refer to [COCO](./dataset_zoo/2d_body_keypoint.md) for more details about the COCO data format.
```

```{note}
The bbox format in MMPose is in `xyxy` instead of `xywh`, which is consistent with the format used in other OpenMMLab projects like [MMDetection](https://github.com/open-mmlab/mmdetection).  We provide useful utils for bbox format conversion, such as `bbox_xyxy2xywh`, `bbox_xywh2xyxy`, `bbox_xyxy2cs`, etc., which are defined in `$MMPOSE/mmpose/structures/bbox/transforms.py`.
```

```

Let's take the implementation of the MPII dataset (`$MMPOSE/mmpose/datasets/datasets/body/mpii_dataset.py`) as an example.

```Python
@DATASETS.register_module()
class MpiiDataset(BaseCocoStyleDataset):
    METAINFO: dict = dict(from_file='configs/_base_/datasets/mpii.py')

    def __init__(self,
                 ## omitted
                 headbox_file: Optional[str] = None,
                 ## omitted
                ):

        if headbox_file:
            if data_mode != 'topdown':
                raise ValueError(
                    f'{self.__class__.__name__} is set to {data_mode}: '
                    'mode, while "headbox_file" is only '
                    'supported in topdown mode.')

            if not test_mode:
                    f'{self.__class__.__name__} has `test_mode==False` '
                    'while "headbox_file" is only '
                    'supported when `test_mode==True`.')

f'{self.__class__.__name__} has `test_mode==False` '

            headbox_file_type = headbox_file[-3:]
            allow_headbox_file_type = ['mat']
            if headbox_file_type not in allow_headbox_file_type:
                raise KeyError(
                    f'The head boxes file type {headbox_file_type} is not '
                    f'supported. Should be `mat` but got {headbox_file_type}.')
        self.headbox_file = headbox_file

        super().__init__(
            ## omitted
            )

    def _load_annotations(self) -> List[dict]:
        """Load data from annotations in MPII format."""
        check_file_exist(self.ann_file)
        with open(self.ann_file) as anno_file:
            anns = json.load(anno_file)

        if self.headbox_file:
            check_file_exist(self.headbox_file)

with open(self.ann_file) as anno_file:

            headbox_dict = loadmat(self.headbox_file)
            headboxes_src = np.transpose(headbox_dict['headboxes_src'],
                                         [2, 0, 1])
            SC_BIAS = 0.6

        data_list = []
        ann_id = 0

        # mpii bbox scales are normalized with factor 200.
        pixel_std = 200.

        for idx, ann in enumerate(anns):
            center = np.array(ann['center'], dtype=np.float32)
            scale = np.array([ann['scale'], ann['scale']],
                             dtype=np.float32) * pixel_std

            # Adjust center/scale slightly to avoid cropping limbs
            if center[0] != -1:
                center[1] = center[1] + 15. / pixel_std * scale[1]

            # MPII uses matlab format, index is 1-based,
            # we should first convert to 0-based index

center[1] = center[1] + 15. / pixel_std * scale[1]

            center = center - 1

            # unify shape with coco datasets
            center = center.reshape(1, -1)
            scale = scale.reshape(1, -1)
            bbox = bbox_cs2xyxy(center, scale)

            # load keypoints in shape [1, K, 2] and keypoints_visible in [1, K]
            keypoints = np.array(ann['joints']).reshape(1, -1, 2)
            keypoints_visible = np.array(ann['joints_vis']).reshape(1, -1)

            data_info = {
                'id': ann_id,
                'img_id': int(ann['image'].split('.')[0]),
                'img_path': osp.join(self.data_prefix['img'], ann['image']),
                'bbox_center': center,
                'bbox_scale': scale,
                'bbox': bbox,
                'bbox_score': np.ones(1, dtype=np.float32),

'bbox_center': center,
                'keypoints': keypoints,
                'keypoints_visible': keypoints_visible,
            }

            if self.headbox_file:
                # calculate the diagonal length of head box as norm_factor
                headbox = headboxes_src[idx]
                head_size = np.linalg.norm(headbox[1] - headbox[0], axis=0)
                head_size *= SC_BIAS
                data_info['head_size'] = head_size.reshape(1, -1)

            data_list.append(data_info)
            ann_id = ann_id + 1

        return data_list
```

When supporting MPII dataset, since we need to use `head_size` to calculate `PCKh`, we add `headbox_file` to `__init__()` and override`_load_annotations()`.

return data_list
```

To support a dataset that is beyond the scope of `BaseCocoStyleDataset`, you may need to subclass from the `BaseDataset` provided by [MMEngine](https://github.com/open-mmlab/mmengine). Please refer to the [documents](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/basedataset.html) for details.

### Pipeline

Data augmentations and transformations during pre-processing are organized as a pipeline. Here is an example of typical pipelines：

```Python
# pipelines
    dict(type='LoadImage'),
    dict(type='GetBBoxCenterScale'),
    dict(type='RandomFlip', direction='horizontal'),
    dict(type='RandomHalfBody'),
    dict(type='RandomBBoxTransform'),
    dict(type='TopdownAffine', input_size=codec['input_size']),
    dict(type='GenerateTarget', encoder=codec),

dict(type='RandomHalfBody'),
    dict(type='PackPoseInputs')
]
]
```

In a keypoint detection task, data will be transformed among three scale spaces:

- **Original Image Space**: the space where the images are stored. The sizes of different images are not necessarily the same

- **Input Image Space**: the image space used for model input. All **images** and **annotations** will be transformed into this space, such as `256x256`, `256x192`, etc.

- **Output Space**: the scale space where model outputs are located, such as `64x64(Heatmap)`，`1x1(Regression)`, etc. The supervision signal is also in this space during training

Here is a diagram to show the workflow of data transformation among the three scale spaces:

![migration-en](https://user-images.githubusercontent.com/13503330/187190213-cad87b5f-0a95-4f1f-b722-15896914ded4.png)

In MMPose, the modules used for data transformation are under `$MMPOSE/mmpose/datasets/transforms`, and their workflow is shown as follows:

![transforms-en](https://user-images.githubusercontent.com/13503330/187190352-a7662346-b8da-4256-9192-c7a84b15cbb5.png)

#### i. Augmentation

Commonly used transforms are defined in `$MMPOSE/mmpose/datasets/transforms/common_transforms.py`, such as `RandomFlip`, `RandomHalfBody`, etc.

For top-down methods, `Shift`, `Rotate`and `Resize` are implemented by `RandomBBoxTransform`**.** For bottom-up methods, `BottomupRandomAffine` is used.

```{note}

```{note}
Most data transforms depend on `bbox_center` and `bbox_scale`, which can be obtained by `GetBBoxCenterScale`.
```

#### ii. Transformation

Affine transformation is used to convert images and annotations from the original image space to the input space. This is done by `TopdownAffine` for top-down methods and `BottomupRandomAffine` for bottom-up methods.

#### iii. Encoding

In training phase, after the data is transformed from the original image space into the input space, it is necessary to use `GenerateTarget` to obtain the training target(e.g. Gaussian Heatmaps). We name this process **Encoding**. Conversely, the process of getting the corresponding coordinates from Gaussian Heatmaps is called **Decoding**.

In MMPose, we collect Encoding and Decoding processes into a **Codec**, in which `encode()` and `decode()` are implemented.

Currently we support the following types of Targets.

- `heatmap`: Gaussian heatmaps
- `keypoint_label`: keypoint representation (e.g. normalized coordinates)
- `keypoint_xy_label`: axis-wise keypoint representation
- `heatmap+keypoint_label`: Gaussian heatmaps and keypoint representation
- `multiscale_heatmap`: multi-scale Gaussian heatmaps

and the generated targets will be packed as follows.

- `heatmaps`: Gaussian heatmaps
- `keypoint_labels`: keypoint representation (e.g. normalized coordinates)
- `keypoint_x_labels`: keypoint x-axis representation
- `keypoint_y_labels`: keypoint y-axis representation
- `keypoint_weights`: keypoint visibility and weights

Note that we unify the data format of top-down and bottom-up methods, which means that a new dimension is added to represent different instances from the same image, in shape:

```Python

```Python
[batch_size, num_instances, num_keypoints, dim_coordinates]
```

- top-down: `[B, 1, K, D]`

- Bottom-up: `[B, N, K, D]`

The provided codecs are stored under `$MMPOSE/mmpose/codecs`.

```{note}
If you wish to customize a new codec, you can refer to [Codec](./user_guides/codecs.md) for more details.
```

#### iv. Packing

After the data is transformed, you need to pack it using `PackPoseInputs`.

This method converts the data stored in the dictionary `results` into standard data structures in MMPose, such as `InstanceData`, `PixelData`, `PoseDataSample`, etc.

Specifically, we divide the data into `gt` (ground-truth) and `pred` (prediction), each of which has the following types:

- **instances**(numpy.array): instance-level raw annotations or predictions in the original scale space

- **instance_labels**(torch.tensor): instance-level training labels (e.g. normalized coordinates, keypoint visibility) in the output scale space
- **fields**(torch.tensor): pixel-level training labels or predictions (e.g. Gaussian Heatmaps) in the output scale space

The following is an example of the implementation of `PoseDataSample` under the hood:

```Python
def get_pose_data_sample(self):
    # meta
    pose_meta = dict(
        img_shape=(600, 900),   # [h, w, c]
        crop_size=(256, 192),   # [h, w]
        heatmap_size=(64, 48),  # [h, w]
    )

    # gt_instances
    gt_instances = InstanceData()
    gt_instances.bboxes = np.random.rand(1, 4)
    gt_instances.keypoints = np.random.rand(1, 17, 2)

    # gt_instance_labels
    gt_instance_labels = InstanceData()
    gt_instance_labels.keypoint_labels = torch.rand(1, 17, 2)

gt_instances.keypoints = np.random.rand(1, 17, 2)

    gt_instance_labels.keypoint_weights = torch.rand(1, 17)

    # pred_instances
    pred_instances = InstanceData()
    pred_instances.keypoints = np.random.rand(1, 17, 2)
    pred_instances.keypoint_scores = np.random.rand(1, 17)

    # gt_fields
    gt_fields = PixelData()
    gt_fields.heatmaps = torch.rand(17, 64, 48)

    # pred_fields
    pred_fields = PixelData()
    pred_fields.heatmaps = torch.rand(17, 64, 48)
    data_sample = PoseDataSample(
        gt_instances=gt_instances,
        pred_instances=pred_instances,
        gt_fields=gt_fields,
        pred_fields=pred_fields,
        metainfo=pose_meta)

    return data_sample
```

## Step3: Model

In MMPose 1.0, the model consists of the following components:

- **Data Preprocessor**: perform data normalization and channel transposition

return data_sample
```

- **Backbone**: used for feature extraction

- **Neck**: GAP，FPN, etc. are optional

- **Head**: used to implement the core algorithm and loss function

We define a base class `BasePoseEstimator` for the model in `$MMPOSE/models/pose_estimators/base.py`. All models, e.g. `TopdownPoseEstimator`, should inherit from this base class and override the corresponding methods.

Three modes are provided in `forward()` of the estimator:

- `mode == 'loss'`: return the result of loss function for model training

- `mode == 'predict'`: return the prediction result in the input space, used for model inference

- `mode == 'tensor'`: return the model output in the output space, i.e. model forward propagation only, for model export

Developers should build the components by calling the corresponding registry. Taking the top-down model as an example:

```Python
@MODELS.register_module()
class TopdownPoseEstimator(BasePoseEstimator):
                 backbone: ConfigType,
                 neck: OptConfigType = None,
                 head: OptConfigType = None,
                 train_cfg: OptConfigType = None,
                 test_cfg: OptConfigType = None,
                 data_preprocessor: OptConfigType = None,
                 init_cfg: OptMultiConfig = None):
        super().__init__(data_preprocessor, init_cfg)

        self.backbone = MODELS.build(backbone)

        if neck is not None:
            self.neck = MODELS.build(neck)

        if head is not None:
            self.head = MODELS.build(head)
```

### Data Preprocessor

if neck is not None:

```

Starting from MMPose 1.0, we have added a new module to the model called data preprocessor, which performs data preprocessings like image normalization and channel transposition. It can benefit from the high computing power of devices like GPU, and improve the integrity in model export and deployment.

A typical `data_preprocessor` in the config is as follows:

```Python
data_preprocessor=dict(
    type='PoseDataPreprocessor',
    bgr_to_rgb=True),
```

It will transpose the channel order of the input image from `bgr` to `rgb` and normalize the data according to `mean` and `std`.

### Backbone

MMPose provides some commonly used backbones under `$MMPOSE/mmpose/models/backbones`.

In practice, developers often use pre-trained backbone weights for transfer learning, which can improve the performance of the model on small datasets.

In MMPose, you can use the pre-trained weights by setting `init_cfg` in config:

```Python
init_cfg=dict(
    type='Pretrained',
    checkpoint='PATH/TO/YOUR_MODEL_WEIGHTS.pth'),
```

If you want to load a checkpoint to your backbone, you should specify the `prefix`:

```Python
    prefix='backbone.',
    checkpoint='PATH/TO/YOUR_CHECKPOINT.pth'),
```

`checkpoint` can be either a local path or a download link. Thus, if you wish to use a pre-trained model provided by Torchvision(e.g. ResNet50), you can simply use:

```Python
    checkpoint='torchvision://resnet50')
```

```Python
```

In addition to these commonly used backbones, you can easily use backbones from other repositories in the OpenMMLab family such as MMClassification, which all share the same config system and provide pre-trained weights.

It should be emphasized that if you add a new backbone, you need to register it by doing:

```Python
class YourBackbone(BaseBackbone):
```

Besides, import it in `$MMPOSE/mmpose/models/backbones/__init__.py`, and add it to `__all__`.

### Neck

Neck is usually a module between Backbone and Head, which is used in some algorithms. Here are some commonly used Neck:

- Global Average Pooling (GAP)

- Feature Pyramid Networks (FPN)

- Feature Map Processor (FMP)

  The `FeatureMapProcessor` is a flexible PyTorch module designed to transform the feature outputs generated by backbones into a format suitable for heads. It achieves this by utilizing non-parametric operations such as selecting, concatenating, and rescaling. Below are some examples along with their corresponding configurations:

  - Select operation

    ```python
    neck=dict(type='FeatureMapProcessor', select_index=0)
    ```

    <img src="https://user-images.githubusercontent.com/26127467/227108468-b44c9c13-9e51-403c-a035-b17b5268acc3.png" height="100px" alt><br>

  - Concatenate operation

    ```python
    neck=dict(type='FeatureMapProcessor', concat=True)
    ```

    <img src="https://user-images.githubusercontent.com/26127467/227108705-4d197c71-4019-42cb-abdb-ba159111abb4.png" height="85px" alt><br>

```

    Note that all feature maps will be resized to match the shape of the first feature map (index 0) prior to concatenation.

  - rescale operation

    ```python
    neck=dict(type='FeatureMapProcessor', scale_factor=2.0)
    ```

    <img src="https://user-images.githubusercontent.com/26127467/227109402-94106e4b-b941-4ce9-8201-c64920d82ed1.png" height="120px" alt><br>

### Head

Generally speaking, Head is often the core of an algorithm, which is used to make predictions and perform loss calculation.

Modules related to Head in MMPose are defined under `$MMPOSE/mmpose/models/heads`, and developers need to inherit the base class `BaseHead` when customizing Head and override the following methods:

- forward()

- predict()

- loss()

- loss()

Specifically, `predict()` method needs to return pose predictions in the image space, which is obtained from the model output though the decoding function provided by the codec. We implement this process in `BaseHead.decode()`.

On the other hand, we will perform test-time augmentation(TTA) in `predict()`.

A commonly used TTA is `flip_test`, namely, an image and its flipped version are sent into the model to inference, and the output of the flipped version will be flipped back, then average them to stabilize the prediction.

Here is an example of `predict()` in `RegressionHead`:

```Python
def predict(self,
            feats: Tuple[Tensor],
            batch_data_samples: OptSampleList,
            test_cfg: ConfigType = {}) -> Predictions:
    """Predict results from outputs."""

    if test_cfg.get('flip_test', False):
        # TTA: flip test -> feats = [orig, flipped]
        assert isinstance(feats, list) and len(feats) == 2

"""Predict results from outputs."""

        flip_indices = batch_data_samples[0].metainfo['flip_indices']
        input_size = batch_data_samples[0].metainfo['input_size']
        _feats, _feats_flip = feats
        _batch_coords = self.forward(_feats)
        _batch_coords_flip = flip_coordinates(
            self.forward(_feats_flip),
            flip_indices=flip_indices,
            shift_coords=test_cfg.get('shift_coords', True),
            input_size=input_size)
        batch_coords = (_batch_coords + _batch_coords_flip) * 0.5
    else:
        batch_coords = self.forward(feats)  # (B, K, D)

    batch_coords.unsqueeze_(dim=1)  # (B, N, K, D)
    preds = self.decode(batch_coords)
```

else:

```

The `loss()` not only performs the calculation of loss functions, but also the calculation of training-time metrics such as pose accuracy. The results are carried by a dictionary `losses`:

```Python
 # calculate accuracy
_, avg_acc, _ = keypoint_pck_accuracy(
    pred=to_numpy(pred_coords),
    gt=to_numpy(keypoint_labels),
    mask=to_numpy(keypoint_weights) > 0,
    thr=0.05,
    norm_factor=np.ones((pred_coords.size(0), 2), dtype=np.float32))

acc_pose = torch.tensor(avg_acc, device=keypoint_labels.device)
losses.update(acc_pose=acc_pose)
```

The data of each batch is packaged into `batch_data_samples`. Taking the Regression-based method as an example, the normalized coordinates and keypoint weights can be obtained as follows:

```Python
keypoint_labels = torch.cat(

```Python
    [d.gt_instance_labels.keypoint_labels for d in batch_data_samples])
keypoint_weights = torch.cat([
    d.gt_instance_labels.keypoint_weights for d in batch_data_samples
])
```

Here is the complete implementation of `loss()` in `RegressionHead`:

```Python
def loss(self,
         inputs: Tuple[Tensor],
         batch_data_samples: OptSampleList,
         train_cfg: ConfigType = {}) -> dict:
    """Calculate losses from a batch of inputs and data samples."""

    pred_outputs = self.forward(inputs)

    keypoint_labels = torch.cat(
        [d.gt_instance_labels.keypoint_labels for d in batch_data_samples])
    keypoint_weights = torch.cat([
        d.gt_instance_labels.keypoint_weights for d in batch_data_samples
    ])

    # calculate losses
    losses = dict()
    loss = self.loss_module(pred_outputs, keypoint_labels,
                            keypoint_weights.unsqueeze(-1))

    if isinstance(loss, dict):
        losses.update(loss)

losses = dict()

    else:
        losses.update(loss_kpt=loss)

    # calculate accuracy
    _, avg_acc, _ = keypoint_pck_accuracy(
        pred=to_numpy(pred_outputs),
        gt=to_numpy(keypoint_labels),
        mask=to_numpy(keypoint_weights) > 0,
        thr=0.05,
        norm_factor=np.ones((pred_outputs.size(0), 2), dtype=np.float32))
    acc_pose = torch.tensor(avg_acc, device=keypoint_labels.device)
    losses.update(acc_pose=acc_pose)

    return losses
```

# Learning to Estimate 3D Hand Pose from Single RGB Images

<details>
<summary align="right"><a href="https://lmb.informatik.uni-freiburg.de/projects/hand3d/">RHD (ICCV'2017)</a></summary>

```bibtex
@TechReport{zb2017hand,
  author={Christian Zimmermann and Thomas Brox},
  title={Learning to Estimate 3D Hand Pose from Single RGB Images},
  institution={arXiv:1705.01389},
  year={2017},
  note="https://arxiv.org/abs/1705.01389",
  url="https://lmb.informatik.uni-freiburg.de/projects/hand3d/"
}
```

<!-- [ALGORITHM] -->

<details>
<summary align="right"><a href="https://link.springer.com/chapter/10.1007/978-3-030-58580-8_27">RTMPose (arXiv'2023)</a></summary>

```bibtex
@misc{https://doi.org/10.48550/arxiv.2303.07399,
  doi = {10.48550/ARXIV.2303.07399},
  url = {https://arxiv.org/abs/2303.07399},
  author = {Jiang, Tao and Lu, Peng and Zhang, Li and Ma, Ningsheng and Han, Rui and Lyu, Chengqi and Li, Yining and Chen, Kai},
  keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose},
  publisher = {arXiv},
  year = {2023},
  copyright = {Creative Commons Attribution 4.0 International}
}

```

<details>
<summary align="right"><a href="https://arxiv.org/abs/2212.07784">RTMDet (arXiv'2022)</a></summary>

```bibtex
@misc{lyu2022rtmdet,

}

```

<details>

```bibtex
      title={RTMDet: An Empirical Study of Designing Real-Time Object Detectors},
      author={Chengqi Lyu and Wenwei Zhang and Haian Huang and Yue Zhou and Yudong Wang and Yanyi Liu and Shilong Zhang and Kai Chen},
      year={2022},
      eprint={2212.07784},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
```

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_CVPR_2019/html/Li_CrowdPose_Efficient_Crowded_Scenes_Pose_Estimation_and_a_New_Benchmark_CVPR_2019_paper.html">CrowdPose (CVPR'2019)</a></summary>

```bibtex
@article{li2018crowdpose,
  title={CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmark},
  author={Li, Jiefeng and Wang, Can and Zhu, Hao and Mao, Yihuan and Fang, Hao-Shu and Lu, Cewu},

title={CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmark},
  journal={arXiv preprint arXiv:1812.00324},
}
```

Results on CrowdPose test with [YOLOv3](https://github.com/eriklindernoren/PyTorch-YOLOv3) human detector

| Arch                                           | Input Size |  AP   | AP<sup>50</sup> | AP<sup>75</sup> | AP (E) | AP (M) | AP (H) |                      ckpt                      |                      log                      |
| :--------------------------------------------- | :--------: | :---: | :-------------: | :-------------: | :----: | :----: | :----: | :--------------------------------------------: | :-------------------------------------------: |

| [rtmpose-m](/configs/body_2d_keypoint/rtmpose/crowdpose/rtmpose-m_8xb64-210e_crowdpose-256x192.py) |  256x192   | 0.706 |      0.841      |      0.765      | 0.799  | 0.719  | 0.582  | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-crowdpose_pt-aic-coco_210e-256x192-e6192cac_20230224.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-crowdpose_pt-aic-coco_210e-256x192-e6192cac_20230224.json) |

# 自定义数据预处理流程

## 数据预处理流程的设计

遵循一般惯例，我们使用 `Dataset` 和 `DataLoader` 来调用多个进程进行数据的加载。`Dataset` 将会返回与模型前向传播的参数所对应的数据项构成的字典。因为目标检测中的数据的尺寸可能无法保持一致（如点云中点的数量、真实标注框的尺寸等），我们在 MMCV 中引入一个 `DataContainer` 类型，来帮助收集和分发不同尺寸的数据。请参考[此处](https://github.com/open-mmlab/mmcv/blob/master/mmcv/parallel/data_container.py)获取更多细节。

数据预处理流程和数据集之间是互相分离的两个部分，通常数据集定义了如何处理标注信息，而数据预处理流程定义了准备数据项字典的所有步骤。数据集预处理流程包含一系列的操作，每个操作将一个字典作为输入，并输出应用于下一个转换的一个新的字典。

我们将在下图中展示一个最经典的数据集预处理流程，其中蓝色框表示预处理流程中的各项操作。随着预处理的进行，每一个操作都会添加新的键值（图中标记为绿色）到输出字典中，或者更新当前存在的键值（图中标记为橙色）。

![](../../../resources/data_pipeline.png)

预处理流程中的各项操作主要分为数据加载、预处理、格式化、测试时的数据增强。

接下来将展示一个用于 PointPillars 模型的数据集预处理流程的例子。

```python
    dict(
        type='LoadPointsFromFile',
        load_dim=5,
        use_dim=5,
        backend_args=backend_args),
    dict(
        type='LoadPointsFromMultiSweeps',
        sweeps_num=10,
    dict(type='LoadAnnotations3D', with_bbox_3d=True, with_label_3d=True),
    dict(

dict(
    dict(
        type='GlobalRotScaleTrans',
        rot_range=[-0.3925, 0.3925],
        scale_ratio_range=[0.95, 1.05],
        translation_std=[0, 0, 0]),
    dict(type='RandomFlip3D', flip_ratio_bev_horizontal=0.5),
    dict(type='PointsRangeFilter', point_cloud_range=point_cloud_range),
    dict(type='ObjectRangeFilter', point_cloud_range=point_cloud_range),
    dict(type='ObjectNameFilter', classes=class_names),
    dict(type='PointShuffle'),
    dict(type='DefaultFormatBundle3D', class_names=class_names),
    dict(type='Collect3D', keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
]
    dict(
    dict(

dict(
    dict(
    dict(
        pts_scale_ratio=1.0,
        pcd_horizontal_flip=False,
        pcd_vertical_flip=False,
            dict(
                type='GlobalRotScaleTrans',
                rot_range=[0, 0],
                scale_ratio_range=[1., 1.],
                translation_std=[0, 0, 0]),
            dict(type='RandomFlip3D'),
            dict(
                type='PointsRangeFilter', point_cloud_range=point_cloud_range),
            dict(
                type='DefaultFormatBundle3D',
                class_names=class_names,
                with_label=False),
            dict(type='Collect3D', keys=['points'])
        ])
]
```

type='DefaultFormatBundle3D',
        ])
]
```

对于每项操作，我们将列出相关的被添加/更新/移除的字典项。

### 数据加载

`LoadPointsFromFile`

- 添加：points

`LoadPointsFromMultiSweeps`

- 更新：points

`LoadAnnotations3D`

- 添加：gt_bboxes_3d, gt_labels_3d, gt_bboxes, gt_labels, pts_instance_mask, pts_semantic_mask, bbox3d_fields, pts_mask_fields, pts_seg_fields

### 预处理

`GlobalRotScaleTrans`

- 添加：pcd_trans, pcd_rotation, pcd_scale_factor
- 更新：points, \*bbox3d_fields

`RandomFlip3D`

- 添加：flip, pcd_horizontal_flip, pcd_vertical_flip

`PointsRangeFilter`

`ObjectRangeFilter`

- 更新：gt_bboxes_3d, gt_labels_3d

`ObjectNameFilter`

`PointShuffle`

### 格式化

`DefaultFormatBundle3D`

- 更新：points, gt_bboxes_3d, gt_labels_3d, gt_bboxes, gt_labels

`Collect3D`

### 格式化

- 添加：img_meta （由 `meta_keys` 指定的键值构成的 img_meta）
- 移除：所有除 `keys` 指定的键值以外的其他键值

### 测试时的数据增强

`MultiScaleFlipAug`

- 更新: scale, pcd_scale_factor, flip, flip_direction, pcd_horizontal_flip, pcd_vertical_flip （与这些指定的参数对应的增强后的数据列表）

## 扩展并使用自定义数据集预处理方法

1. 在任意文件中写入新的数据集预处理方法，如 `my_pipeline.py`，该预处理方法的输入和输出均为字典

   ```python
   from mmdet.datasets import PIPELINES

   @PIPELINES.register_module()
   class MyTransform:

       def __call__(self, results):
           results['dummy'] = True
           return results
   ```

2. 导入新的预处理方法类

   ```python
   from .my_pipeline import MyTransform
   ```

3. 在配置文件中使用该数据集预处理方法

   ```python
   train_pipeline = [
       dict(
           type='LoadPointsFromFile',
           load_dim=5,
           use_dim=5,
           backend_args=backend_args),
       dict(

```python
       dict(
       dict(
           type='LoadPointsFromMultiSweeps',
           sweeps_num=10,
       dict(type='LoadAnnotations3D', with_bbox_3d=True, with_label_3d=True),
       dict(
           type='GlobalRotScaleTrans',
           rot_range=[-0.3925, 0.3925],
           scale_ratio_range=[0.95, 1.05],
           translation_std=[0, 0, 0]),
       dict(type='RandomFlip3D', flip_ratio_bev_horizontal=0.5),
       dict(type='PointsRangeFilter', point_cloud_range=point_cloud_range),
       dict(type='ObjectRangeFilter', point_cloud_range=point_cloud_range),
       dict(type='ObjectNameFilter', classes=class_names),
       dict(type='MyTransform'),
       dict(type='PointShuffle'),
       dict(type='DefaultFormatBundle3D', class_names=class_names),

dict(type='ObjectNameFilter', classes=class_names),
       dict(type='Collect3D', keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
   ]
   ```

# DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning

<details>
<summary align="right"><a href="https://elifesciences.org/articles/47994">Grévy’s Zebra (Elife'2019)</a></summary>

```bibtex
@article{graving2019deepposekit,
  title={DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning},
  author={Graving, Jacob M and Chae, Daniel and Naik, Hemal and Li, Liang and Koger, Benjamin and Costelloe, Blair R and Couzin, Iain D},
  journal={Elife},
  volume={8},
  pages={e47994},
  year={2019},
  publisher={eLife Sciences Publications Limited}
}
```

# Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking

## Abstract

<!-- [ABSTRACT] -->

Multi-Object Tracking (MOT) has rapidly progressed with the development of object detection and re-identification. However, motion modeling, which facilitates object association by forecasting short-term trajec- tories with past observations, has been relatively under-explored in recent years. Current motion models in MOT typically assume that the object motion is linear in a small time window and needs continuous observations, so these methods are sensitive to occlusions and non-linear motion and require high frame-rate videos. In this work, we show that a simple motion model can obtain state-of-the-art tracking performance without other cues like appearance. We emphasize the role of “observation” when recovering tracks from being lost and reducing the error accumulated by linear motion models during the lost period. We thus name the proposed method as Observation-Centric SORT, OC-SORT for short. It remains simple, online, and real-time but improves robustness over occlusion and non-linear motion. It achieves 63.2 and 62.1 HOTA on MOT17 and MOT20, respectively, surpassing all published methods. It also sets new states of the art on KITTI Pedestrian Tracking and DanceTrack where the object motion is highly non-linear

<!-- [IMAGE] -->

  <img src="https://user-images.githubusercontent.com/17743251/168193097-b3ad1a94-b18c-4b14-b7b1-5f8c6ed842f0.png"/>
</div>

```latex
@article{cao2022observation,
  title={Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking},
  author={Cao, Jinkun and Weng, Xinshuo and Khirodkar, Rawal and Pang, Jiangmiao and Kitani, Kris},
  journal={arXiv preprint arXiv:2203.14360},
}
```

## Results and models on MOT17

The performance on `MOT17-half-val` is comparable with the performance from [the OC-SORT official implementation](https://github.com/noahcao/OC_SORT). We use the same YOLO-X detector weights as in [ByteTrack](https://github.com/open-mmlab/mmtracking/tree/master/configs/mot/bytetrack).

| Method  | Detector |        Train Set        | Test Set | Public | Inf time (fps) | HOTA | MOTA | IDF1 |  FP   |  FN   | IDSw. |                          Config                           |                                                                                                                                               Download                                                                                                                                               |
| :-----: | :------: | :---------------------: | :------: | :----: | :------------: | :--: | :--: | :--: | :---: | :---: | :---: | :-------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

| OC-SORT | YOLOX-X  | CrowdHuman + half-train | half-val |   N    |       -        | 67.5 | 77.5 | 78.2 | 15987 | 19590 |  855  | [config](ocsort_yolox_x_crowdhuman_mot17-private-half.py) | [model](https://download.openmmlab.com/mmtracking/mot/ocsort/mot_dataset/ocsort_yolox_x_crowdhuman_mot17-private-half_20220813_101618-fe150582.pth) \| [log](https://download.openmmlab.com/mmtracking/mot/ocsort/mot_dataset/ocsort_yolox_x_crowdhuman_mot17-private-half_20220813_101618.log.json) |

## Get started

### 1. Development Environment Setup

Tracking Development Environment Setup can refer to this [document](../../docs/en/get_started.md).

### 2. Dataset Prepare

Tracking Dataset Prepare can refer to this [document](../../docs/en/user_guides/tracking_dataset_prepare.md).

### 3. Training

OCSORT training is same as Bytetrack, please refer to [document](../../configs/bytetrack/README.md).

### 4. Testing and evaluation

OCSORT evaluation and test are same as Bytetrack, please refer to [document](../../configs/bytetrack/README.md).

### 5.Inference

OCSORT inference is same as Bytetrack, please refer to [document](../../configs/bytetrack/README.md).

## <a href='https://mmrotate.readthedocs.io/en/latest/'>English</a>

## <a href='https://mmrotate.readthedocs.io/zh_CN/latest/'>简体中文</a>

# 自定义训练设置 (待更新)

## 自定义优化设置

### 自定义 Pytorch 支持的优化器

我们已经支持了全部 Pytorch 自带的优化器，唯一需要修改的就是配置文件中 `optimizer` 部分。
例如，如果您想使用 `ADAM` (注意如下操作可能会让模型表现下降)，可以使用如下修改：

```python
optimizer = dict(type='Adam', lr=0.0003, weight_decay=0.0001)
```

为了修改模型训练的学习率，使用者仅需修改配置文件里 `optimizer` 的 `lr` 即可。
使用者可以参考 PyTorch 的 [API doc](https://pytorch.org/docs/stable/optim.html?highlight=optim#module-torch.optim) 直接设置参数。

### 自定义用户自己实现的优化器

#### 1. 定义一个新的优化器

一个自定义的优化器可以这样定义：

假如您想增加一个叫做 `MyOptimizer` 的优化器，它的参数分别有 `a`, `b`, 和 `c`。
您需要创建一个名为 `mmrotate/core/optimizer` 的新文件夹；然后参考如下代码段在 `mmrotate/core/optimizer/my_optimizer.py` 文件中实现新的优化器:

```python
from mmdet.core.optimizer.registry import OPTIMIZERS
from torch.optim import Optimizer

@OPTIMIZERS.register_module()
class MyOptimizer(Optimizer):

    def __init__(self, a, b, c)

```

#### 2. 增加优化器到注册表 (registry)

为了能够使得上述添加的模块被 `mmrotate` 发现，需要先将该模块添加到主命名空间（main namespace）。

- 修改 `mmrotate/core/optimizer/__init__.py` 文件来导入该模块。

def __init__(self, a, b, c)

```

  新的被定义的模块应该被导入到 `mmrotate/core/optimizer/__init__.py` 中，这样注册表才会发现新的模块并添加它：

```python
from .my_optimizer import MyOptimizer
```

- 在配置文件中使用 `custom_imports` 来手动添加该模块

```python
custom_imports = dict(imports=['mmrotate.core.optimizer.my_optimizer'], allow_failed_imports=False)
```

`mmrotate.core.optimizer.my_optimizer` 模块将会在程序开始被导入，并且 `MyOptimizer` 类将会自动注册。
需要注意只有包含 `MyOptimizer` 类的包 (package) 应当被导入。
而 `mmrotate.core.optimizer.my_optimizer.MyOptimizer` **不能** 被直接导入。

事实上，在这种导入方式下用户可以用完全不同的文件夹结构，只要这一模块的根目录已经被添加到 `PYTHONPATH` 里面。

#### 3. 在配置文件中指定优化器

之后您可以在配置文件的 `optimizer` 部分里面使用 `MyOptimizer`。
在配置文件里，优化器按照如下形式被定义在 `optimizer` 部分里：

```python
optimizer = dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001)
```

要使用用户自定义的优化器，这部分应该改成：

```python

```python
```

```python
optimizer = dict(type='MyOptimizer', a=a_value, b=b_value, c=c_value)
```

### 自定义优化器的构造函数 (constructor)

有些模型的优化器可能有一些特别参数配置，例如批归一化层 (BatchNorm layers) 的权重衰减系数 (weight decay)。
用户可以通过自定义优化器的构造函数去微调这些细粒度参数。

```python
from mmcv.utils import build_from_cfg

from mmcv.runner.optimizer import OPTIMIZER_BUILDERS, OPTIMIZERS
from mmrotate.utils import get_root_logger

@OPTIMIZER_BUILDERS.register_module()
class MyOptimizerConstructor(object):

    def __init__(self, optimizer_cfg, paramwise_cfg=None):

    def __call__(self, model):

        return my_optimizer

```

`mmcv` 默认的优化器构造函数实现可以参考 [这里](https://github.com/open-mmlab/mmcv/blob/9ecd6b0d5ff9d2172c49a182eaa669e9f27bb8e7/mmcv/runner/optimizer/default_constructor.py#L11) ，这也可以作为新的优化器构造函数的模板。

### 其他配置

```

### 其他配置

优化器未实现的技巧应该通过修改优化器构造函数（如设置基于参数的学习率）或者钩子（hooks）去实现。我们列出一些常见的设置，它们可以稳定或加速模型的训练。
如果您有更多的设置，欢迎在 PR 和 issue 里面提出。

- __使用梯度裁剪 (gradient clip) 来稳定训练__:
  一些模型需要梯度裁剪来稳定训练过程。使用方式如下：

  ```python
  optimizer_config = dict(
      _delete_=True, grad_clip=dict(max_norm=35, norm_type=2))
  ```

  如果您的配置继承了已经设置了 `optimizer_config` 的基础配置（base config），你可能需要设置 `_delete_=True` 来覆盖不必要的配置参数。请参考 [配置文档](https://mmdetection.readthedocs.io/en/latest/tutorials/config.html) 了解更多细节。

- __使用动量调度加速模型收敛__:
  我们支持动量规划器（Momentum scheduler），以实现根据学习率调节模型优化过程中的动量设置，这可以使模型以更快速度收敛。

  动量规划器经常与学习率规划器（LR scheduler）一起使用，例如下面的配置经常被用于 3D 检测模型训练中以加速收敛。更多细节请参考 [CyclicLrUpdater](https://github.com/open-mmlab/mmcv/blob/f48241a65aebfe07db122e9db320c31b685dc674/mmcv/runner/hooks/lr_updater.py#L327) 和 [CyclicMomentumUpdater](https://github.com/open-mmlab/mmcv/blob/f48241a65aebfe07db122e9db320c31b685dc674/mmcv/runner/hooks/momentum_updater.py#L130)。

  ```python
  lr_config = dict(
      policy='cyclic',
      target_ratio=(10, 1e-4),
      cyclic_times=1,
      step_ratio_up=0.4,
  )
  momentum_config = dict(
      target_ratio=(0.85 / 0.95, 1),
  )
  ```

## 自定义训练计划

默认地，我们使用 1x 计划（1x schedule）的步进学习率（step learning rate），这在 MMCV 中被称为 [`StepLRHook`](https://github.com/open-mmlab/mmcv/blob/f48241a65aebfe07db122e9db320c31b685dc674/mmcv/runner/hooks/lr_updater.py#L153)。

我们支持很多其他的学习率规划器，参考 [这里](https://github.com/open-mmlab/mmcv/blob/master/mmcv/runner/hooks/lr_updater.py) ，例如 `CosineAnnealing` 和 `Poly`。下面是一些例子：

- `Poly` :

  ```python
  lr_config = dict(policy='poly', power=0.9, min_lr=1e-4, by_epoch=False)
  ```

- `ConsineAnnealing` :

  ```python
      policy='CosineAnnealing',
      warmup='linear',
      warmup_iters=1000,
      warmup_ratio=1.0 / 10,
      min_lr_ratio=1e-5)
  ```

## 自定义工作流 (workflow)

工作流是一个专门定义运行顺序和轮数（epochs）的列表。
默认情况下它设置成：

```python
workflow = [('train', 1)]
```

这是指训练 1 个 epoch。
有时候用户可能想检查一些模型在验证集上的指标，如损失函数值（Loss）和准确性（Accuracy）。
在这种情况下，我们可以将工作流设置为：

```python
[('train', 1), ('val', 1)]
```

这样以来， 1 个 epoch 训练，1 个 epoch 验证将交替运行。

**注意**:

1. 模型参数在验证的阶段不会被自动更新。
2. 配置文件里的键值 `total_epochs` 仅控制训练的 epochs 数目，而不会影响验证工作流。
3. 工作流 `[('train', 1), ('val', 1)]` 和 `[('train', 1)]` 将不会改变 `EvalHook` 的行为，因为 `EvalHook` 被 `after_train_epoch`

**注意**:

   调用而且验证的工作流仅仅影响通过调用 `after_val_epoch` 的钩子 (hooks)。因此， `[('train', 1), ('val', 1)]` 和 `[('train', 1)]`
   的区别仅在于 runner 将在每次训练阶段（training epoch）结束后计算在验证集上的损失。

## 自定义钩 (hooks)

### 自定义用户自己实现的钩子（hooks）

#### 1. 实现一个新的钩子（hook）

在某些情况下，用户可能需要实现一个新的钩子。 MMRotate 支持训练中的自定义钩子。 因此，用户可以直接在 mmrotate 或其基于 mmdet 的代码库中实现钩子，并通过仅在训练中修改配置来使用钩子。
这里我们举一个例子：在 mmrotate 中创建一个新的钩子并在训练中使用它。

```python
from mmcv.runner import HOOKS, Hook

@HOOKS.register_module()
class MyHook(Hook):

    def __init__(self, a, b):
        pass

    def before_run(self, runner):
        pass

    def after_run(self, runner):
        pass

    def before_epoch(self, runner):
        pass

    def after_epoch(self, runner):
        pass

    def before_iter(self, runner):
        pass

    def after_iter(self, runner):
        pass

def before_epoch(self, runner):
        pass

        pass

        pass

        pass
```

用户需要根据钩子的功能指定钩子在训练各阶段中（ `before_run` , `after_run` , `before_epoch` , `after_epoch` , `before_iter` , `after_iter`）做什么。

#### 2. 注册新的钩子（hook）

接下来我们需要导入 `MyHook`。如果文件的路径是 `mmrotate/core/utils/my_hook.py` ，有两种方式导入：

- 修改 `mmrotate/core/utils/__init__.py` 文件来导入

  新定义的模块需要在 `mmrotate/core/utils/__init__.py` 导入，注册表才会发现并添加该模块：

```python
from .my_hook import MyHook
```

- 在配置文件中使用 `custom_imports` 来手动导入

```python
custom_imports = dict(imports=['mmrotate.core.utils.my_hook'], allow_failed_imports=False)
```

#### 3. 修改配置

```python
custom_hooks = [
    dict(type='MyHook', a=a_value, b=b_value)
]
```

您也可以通过配置键值 `priority` 为 `'NORMAL'` 或 `'HIGHEST'` 来设置钩子的优先级：

```python
    dict(type='MyHook', a=a_value, b=b_value, priority='NORMAL')
]
```

默认地，钩子的优先级在注册时被设置为 `NORMAL`。

]
```

```python
]
```

### 使用 MMCV 中实现的钩子 (hooks)

如果钩子已经在 MMCV 里实现了，您可以直接修改配置文件来使用钩子。

#### 4. 示例: `NumClassCheckHook`

我们实现了一个自定义的钩子 [NumClassCheckHook](https://github.com/open-mmlab/mmdetection/blob/master/mmdet/datasets/utils.py) ，用来检验 head 中的 `num_classes` 是否与 `dataset` 中的 `CLASSSES` 长度匹配。

我们在 [default_runtime.py](https://github.com/open-mmlab/mmdetection/blob/master/configs/_base_/default_runtime.py) 中对其进行设置。

```python
custom_hooks = [dict(type='NumClassCheckHook')]
```

### 修改默认运行挂钩

有一些常见的钩子并不通过 `custom_hooks` 注册，这些钩子包括：

- log_config
- checkpoint_config
- evaluation
- lr_config
- optimizer_config
- momentum_config

这些钩子中，只有记录器钩子（logger hook）是 `VERY_LOW` 优先级，其他钩子的优先级为 `NORMAL`。
前面提到的教程已经介绍了如何修改 `optimizer_config` , `momentum_config` 以及 `lr_config`。

这里我们介绍一下如何处理 `log_config` , `checkpoint_config` 以及 `evaluation`。

#### Checkpoint config

MMCV runner 将使用 `checkpoint_config` 来初始化 [`CheckpointHook`](https://github.com/open-mmlab/mmcv/blob/9ecd6b0d5ff9d2172c49a182eaa669e9f27bb8e7/mmcv/runner/hooks/checkpoint.py#L9)。

```python
checkpoint_config = dict(interval=1)
```

用户可以设置 `max_keep_ckpts` 来仅保存一小部分检查点（checkpoint）或者通过设置 `save_optimizer` 来决定是否保存优化器的状态字典 (state dict of optimizer)。更多使用参数的细节请参考 [这里](https://mmcv.readthedocs.io/en/latest/api.html#mmcv.runner.CheckpointHook)。

#### Log config

`log_config` 包裹了许多日志钩 (logger hooks) 而且能去设置间隔 (intervals)。现在 MMCV 支持 `WandbLoggerHook` ， `MlflowLoggerHook` 和 `TensorboardLoggerHook`。
详细的使用请参照 [文档](https://mmcv.readthedocs.io/en/latest/api.html#mmcv.runner.LoggerHook)。

```python

```python
    hooks=[
        dict(type='TensorboardLoggerHook')
    ])
```

#### Evaluation config

`evaluation` 的配置文件将被用来初始化 [`EvalHook`](https://github.com/open-mmlab/mmdetection/blob/7a404a2c000620d52156774a5025070d9e00d918/mmdet/core/evaluation/eval_hooks.py#L8)。
除了 `interval` 键，其他的像 `metric` 这样的参数将被传递给 `dataset.evaluate()`。

```python
evaluation = dict(interval=1, metric='bbox')
```

# Covid-19 CT Chest X-ray Dataset

## Description

This project supports **`Covid-19 CT Chest X-ray Dataset`**, which can be downloaded from [here](https://github.com/ieee8023/covid-chestxray-dataset).

### Dataset Overview

In the context of a COVID-19 pandemic, we want to improve prognostic predictions to triage and manage patient care. Data is the first step to developing any diagnostic/prognostic tool. While there exist large public datasets of more typical chest X-rays from the NIH \[Wang 2017\], Spain \[Bustos 2019\], Stanford \[Irvin 2019\], MIT \[Johnson 2019\] and Indiana University \[Demner-Fushman 2016\], there is no collection of COVID-19 chest X-rays or CT scans designed to be used for computational analysis.

The 2019 novel coronavirus (COVID-19) presents several unique features [Fang, 2020](https://pubs.rsna.org/doi/10.1148/radiol.2020200432) and [Ai 2020](https://pubs.rsna.org/doi/10.1148/radiol.2020200642). While the diagnosis is confirmed using polymerase chain reaction (PCR), infected patients with pneumonia may present on chest X-ray and computed tomography (CT) images with a pattern that is only moderately characteristic for the human eye [Ng, 2020](https://pubs.rsna.org/doi/10.1148/ryct.2020200034). In late January, a Chinese team published a paper detailing the clinical and paraclinical features of COVID-19. They reported that patients present abnormalities in chest CT images with most having bilateral involvement [Huang 2020](<https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(20)30183-5/fulltext>). Bilateral multiple lobular and subsegmental areas of consolidation constitute the typical findings in chest CT images of intensive care unit (ICU) patients on admission [Huang 2020](<https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(20)30183-5/fulltext>). In comparison, non-ICU patients show bilateral ground-glass opacity and subsegmental areas of consolidation in their chest CT images [Huang 2020](<https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(20)30183-5/fulltext>). In these patients, later chest CT images display bilateral ground-glass opacity with resolved consolidation [Huang 2020](<https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(20)30183-5/fulltext>).

### Statistic Information

| Dataset Name                                                           | Anatomical Region | Task Type    | Modality | Nnum. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release date | License                                                               |
| ---------------------------------------------------------------------- | ----------------- | ------------ | -------- | ------------- | --------------------- | ---------------------- | ------------ | --------------------------------------------------------------------- |
| [Covid-19-ct-cxr](https://github.com/ieee8023/covid-chestxray-dataset) | thorax            | segmentation | x_ray    | 2             | 205/-/714             | yes/-/no               | 2021         | [CC-BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) |

| Class Name | Num. Train | Pct. Train | Num. Val | Pct. Val | Num. Test | Pct. Test |

| :--------: | :--------: | :--------: | :------: | :------: | :-------: | :-------: |
| background |    205     |   72.84    |    -     |    -     |     -     |     -     |
|    lung    |    205     |   27.16    |    -     |    -     |     -     |     -     |

Note:

- `Pct` means percentage of pixels in this category in all pixels.

### Visualization

![cov19ctcxr](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/x_ray/covid_19_ct_cxr/covid_19_ct_cxr_dataset.png?raw=true)

### Dataset Citation

```
@article{cohen2020covidProspective,
  title={{COVID-19} Image Data Collection: Prospective Predictions Are the Future},
  author={Joseph Paul Cohen and Paul Morrison and Lan Dao and Karsten Roth and Tim Q Duong and Marzyeh Ghassemi},
  journal={arXiv 2006.11988},
}

@article{cohen2020covid,
  title={COVID-19 image data collection},

journal={arXiv 2006.11988},
}

  author={Joseph Paul Cohen and Paul Morrison and Lan Dao},
  journal={arXiv 2003.11597},
}
```

### Prerequisites

- Python v3.8
- PyTorch v1.10.0
- pillow(PIL) v9.3.0 9.3.0
- scikit-learn(sklearn) v1.2.0 1.2.0
- [MIM](https://github.com/open-mmlab/mim) v0.3.4
- [MMCV](https://github.com/open-mmlab/mmcv) v2.0.0rc4
- [MMEngine](https://github.com/open-mmlab/mmengine) v0.2.0 or higher
- [MMSegmentation](https://github.com/open-mmlab/mmsegmentation) v1.0.0rc5

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `covid_19_ct_cxr/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
export PYTHONPATH=`pwd`:$PYTHONPATH
```

### Dataset Preparing

```shell
```

- download dataset from [here](https://github.com/ieee8023/covid-chestxray-dataset) and decompress data to path `'data/'`.
- run script `"python tools/prepare_dataset.py"` to format data and change folder structure as below.
- run script `"python ../../tools/split_seg_dataset.py"` to split dataset and generate `train.txt`, `val.txt` and `test.txt`. If the label of official validation set and test set cannot be obtained, we generate `train.txt` and `val.txt` from the training set randomly.

```shell
mkdir data && cd data
git clone git@github.com:ieee8023/covid-chestxray-dataset.git
cd ..
python tools/prepare_dataset.py
python ../../tools/split_seg_dataset.py
```

```none
  mmsegmentation
  ├── mmseg
  ├── projects
  │   ├── medical
  │   │   ├── 2d_image
  │   │   │   ├── x_ray
  │   │   │   │   ├── covid_19_ct_cxr
  │   │   │   │   │   ├── configs
  │   │   │   │   │   ├── datasets
  │   │   │   │   │   ├── tools

│   │   ├── 2d_image
  │   │   │   │   │   ├── data
  │   │   │   │   │   │   ├── train.txt
  │   │   │   │   │   │   ├── val.txt
  │   │   │   │   │   │   ├── images
  │   │   │   │   │   │   │   ├── train
  │   │   │   │   |   │   │   │   ├── xxx.png
  │   │   │   │   |   │   │   │   ├── ...
  │   │   │   │   |   │   │   │   └── xxx.png
  │   │   │   │   │   │   ├── masks
```

### Divided Dataset Information

***Note: The table information below is divided by ourselves.***

| background |    164     |   72.88    |    41    |  72.69   |     -     |     -     |
|    lung    |    164     |   27.12    |    41    |  27.31   |     -     |     -     |

### Training commands

To train models on a single server with one GPU. (default)

```shell
mim train mmseg ./configs/${CONFIG_FILE}
```

### Testing commands

To test models on a single server with one GPU. (default)

```shell
mim test mmseg ./configs/${CONFIG_FILE}  --checkpoint ${CHECKPOINT_PATH}
```

<!-- List the results as usually done in other model's README. [Example](https://github.com/open-mmlab/mmsegmentation/tree/dev-1.x/configs/fcn#results-and-models)

```

You should claim whether this is based on the pre-trained weights, which are converted from the official release; or it's a reproduced result obtained from retraining the model in this project. -->

## Checklist

- [x] Milestone 1: PR-ready, and acceptable to be one of the `projects/`.

  - [x] Finish the code
  - [x] Basic docstrings & proper citation
  - [x] Test-time correctness
  - [x] A full README

- [x] Milestone 2: Indicates a successful model implementation.

  - [x] Training-time correctness

- [ ] Milestone 3: Good to be a part of our core package!

  - [ ] Type hints and docstrings
  - [ ] Unit tests
  - [ ] Code polishing
  - [ ] Metafile.yml

- [ ] Move your modules into the core package following the codebase's file hierarchy structure.

- [ ] Type hints and docstrings

- [ ] Refactor your modules into the core package following the codebase's file hierarchy structure.

# SwinIR (ICCVW'2021)

> [SwinIR: Image Restoration Using Swin Transformer](https://arxiv.org/abs/2108.10257)

> **Task**: Image Super-Resolution, Image denoising, JPEG compression artifact reduction

Image restoration is a long-standing low-level vision problem that aims to restore high-quality images from low-quality images (e.g., downscaled, noisy and compressed images). While state-of-the-art image restoration methods are based on convolutional neural networks, few attempts have been made with Transformers which show impressive performance on high-level vision tasks. In this paper, we propose a strong baseline model SwinIR for image restoration based on the Swin Transformer. SwinIR consists of three parts: shallow feature extraction, deep feature extraction and high-quality image reconstruction. In particular, the deep feature extraction module is composed of several residual Swin Transformer blocks (RSTB), each of which has several Swin Transformer layers together with a residual connection. We conduct experiments on three representative tasks: image super-resolution (including classical, lightweight and real-world image super-resolution), image denoising (including grayscale and color image denoising) and JPEG compression artifact reduction. Experimental results demonstrate that SwinIR outperforms state-of-the-art methods on different tasks by up to 0.14~0.45dB, while the total number of parameters can be reduced by up to 67%.

<div align=center >
 <img src="https://user-images.githubusercontent.com/40970489/204525707-272fb8c6-1bb3-41f2-9a9b-612c48ddd9b4.png" width="800"/>
</div >

## Results and models

### **Classical Image Super-Resolution**

Evaluated on Y channels, `scale` pixels in each border are cropped before evaluation.
The metrics are `PSNR / SSIM` .

|                                     Model                                     | Dataset | Scale |  PSNR   |  SSIM  | Training Resources |                                     Download                                     |
| :---------------------------------------------------------------------------: | :-----: | :---: | :-----: | :----: | :----------------: | :------------------------------------------------------------------------------: |

| [swinir_x2s48w8d6e180_8xb4-lr2e-4-500k_div2k](/configs/swinir/swinir_x2s48w8d6e180_8xb4-lr2e-4-500k_div2k.py) |  Set5   |  x2   | 38.3240 | 0.9626 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x2s48w8d6e180_8xb4-lr2e-4-500k_div2k-ed2d419e.pth) \| log |
| [swinir_x2s48w8d6e180_8xb4-lr2e-4-500k_div2k](/configs/swinir/swinir_x2s48w8d6e180_8xb4-lr2e-4-500k_div2k.py) |  Set14  |  x2   | 34.1174 | 0.9230 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x2s48w8d6e180_8xb4-lr2e-4-500k_div2k-ed2d419e.pth) \| log |
| [swinir_x2s48w8d6e180_8xb4-lr2e-4-500k_div2k](/configs/swinir/swinir_x2s48w8d6e180_8xb4-lr2e-4-500k_div2k.py) |  DIV2K  |  x2   | 37.8921 | 0.9481 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x2s48w8d6e180_8xb4-lr2e-4-500k_div2k-ed2d419e.pth) \| log |

| [swinir_x3s48w8d6e180_8xb4-lr2e-4-500k_div2k](/configs/swinir/swinir_x3s48w8d6e180_8xb4-lr2e-4-500k_div2k.py) |  Set5   |  x3   | 34.8640 | 0.9317 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x3s48w8d6e180_8xb4-lr2e-4-500k_div2k-926950f1.pth) \| log |
| [swinir_x3s48w8d6e180_8xb4-lr2e-4-500k_div2k](/configs/swinir/swinir_x3s48w8d6e180_8xb4-lr2e-4-500k_div2k.py) |  Set14  |  x3   | 30.7669 | 0.8508 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x3s48w8d6e180_8xb4-lr2e-4-500k_div2k-926950f1.pth) \| log |
| [swinir_x3s48w8d6e180_8xb4-lr2e-4-500k_div2k](/configs/swinir/swinir_x3s48w8d6e180_8xb4-lr2e-4-500k_div2k.py) |  DIV2K  |  x3   | 34.1397 | 0.8917 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x3s48w8d6e180_8xb4-lr2e-4-500k_div2k-926950f1.pth) \| log |

| [swinir_x4s48w8d6e180_8xb4-lr2e-4-500k_div2k](/configs/swinir/swinir_x4s48w8d6e180_8xb4-lr2e-4-500k_div2k.py) |  Set5   |  x4   | 32.7315 | 0.9029 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x4s48w8d6e180_8xb4-lr2e-4-500k_div2k-88e4903d.pth) \| log |
| [swinir_x4s48w8d6e180_8xb4-lr2e-4-500k_div2k](/configs/swinir/swinir_x4s48w8d6e180_8xb4-lr2e-4-500k_div2k.py) |  Set14  |  x4   | 28.9065 | 0.7915 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x4s48w8d6e180_8xb4-lr2e-4-500k_div2k-88e4903d.pth) \| log |
| [swinir_x4s48w8d6e180_8xb4-lr2e-4-500k_div2k](/configs/swinir/swinir_x4s48w8d6e180_8xb4-lr2e-4-500k_div2k.py) |  DIV2K  |  x4   | 32.0953 | 0.8418 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x4s48w8d6e180_8xb4-lr2e-4-500k_div2k-88e4903d.pth) \| log |

| [swinir_x2s64w8d6e180_8xb4-lr2e-4-500k_df2k](/configs/swinir/swinir_x2s64w8d6e180_8xb4-lr2e-4-500k_df2k.py) |  Set5   |  x2   | 38.3971 | 0.9629 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x2s64w8d6e180_8xb4-lr2e-4-500k_df2k-69e15fb6.pth) \| log |
| [swinir_x2s64w8d6e180_8xb4-lr2e-4-500k_df2k](/configs/swinir/swinir_x2s64w8d6e180_8xb4-lr2e-4-500k_df2k.py) |  Set14  |  x2   | 34.4149 | 0.9252 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x2s64w8d6e180_8xb4-lr2e-4-500k_df2k-69e15fb6.pth) \| log |
| [swinir_x2s64w8d6e180_8xb4-lr2e-4-500k_df2k](/configs/swinir/swinir_x2s64w8d6e180_8xb4-lr2e-4-500k_df2k.py) |  DIV2K  |  x2   | 37.9473 | 0.9488 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x2s64w8d6e180_8xb4-lr2e-4-500k_df2k-69e15fb6.pth) \| log |

| [swinir_x3s64w8d6e180_8xb4-lr2e-4-500k_df2k](/configs/swinir/swinir_x3s64w8d6e180_8xb4-lr2e-4-500k_df2k.py) |  Set5   |  x3   | 34.9335 | 0.9323 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x3s64w8d6e180_8xb4-lr2e-4-500k_df2k-d6982f7b.pth) \| log |
| [swinir_x3s64w8d6e180_8xb4-lr2e-4-500k_df2k](/configs/swinir/swinir_x3s64w8d6e180_8xb4-lr2e-4-500k_df2k.py) |  Set14  |  x3   | 30.9258 | 0.8540 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x3s64w8d6e180_8xb4-lr2e-4-500k_df2k-d6982f7b.pth) \| log |
| [swinir_x3s64w8d6e180_8xb4-lr2e-4-500k_df2k](/configs/swinir/swinir_x3s64w8d6e180_8xb4-lr2e-4-500k_df2k.py) |  DIV2K  |  x3   | 34.2830 | 0.8939 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x3s64w8d6e180_8xb4-lr2e-4-500k_df2k-d6982f7b.pth) \| log |

| [swinir_x4s64w8d6e180_8xb4-lr2e-4-500k_df2k](/configs/swinir/swinir_x4s64w8d6e180_8xb4-lr2e-4-500k_df2k.py) |  Set5   |  x4   | 32.9214 | 0.9053 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x4s64w8d6e180_8xb4-lr2e-4-500k_df2k-0502d775.pth) \| log |
| [swinir_x4s64w8d6e180_8xb4-lr2e-4-500k_df2k](/configs/swinir/swinir_x4s64w8d6e180_8xb4-lr2e-4-500k_df2k.py) |  Set14  |  x4   | 29.0792 | 0.7953 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x4s64w8d6e180_8xb4-lr2e-4-500k_df2k-0502d775.pth) \| log |
| [swinir_x4s64w8d6e180_8xb4-lr2e-4-500k_df2k](/configs/swinir/swinir_x4s64w8d6e180_8xb4-lr2e-4-500k_df2k.py) |  DIV2K  |  x4   | 32.3021 | 0.8451 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x4s64w8d6e180_8xb4-lr2e-4-500k_df2k-0502d775.pth) \| log |

### **Lightweight Image Super-Resolution**

| [swinir_x2s64w8d4e60_8xb4-lr2e-4-500k_div2k](/configs/swinir/swinir_x2s64w8d4e60_8xb4-lr2e-4-500k_div2k.py) |  Set5   |  x2   | 38.1289 | 0.9617 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x2s64w8d4e60_8xb4-lr2e-4-500k_div2k-131d3f64.pth) \| log |

| [swinir_x2s64w8d4e60_8xb4-lr2e-4-500k_div2k](/configs/swinir/swinir_x2s64w8d4e60_8xb4-lr2e-4-500k_div2k.py) |  Set14  |  x2   | 33.8404 | 0.9207 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x2s64w8d4e60_8xb4-lr2e-4-500k_div2k-131d3f64.pth) \| log |
| [swinir_x2s64w8d4e60_8xb4-lr2e-4-500k_div2k](/configs/swinir/swinir_x2s64w8d4e60_8xb4-lr2e-4-500k_div2k.py) |  DIV2K  |  x2   | 37.5844 | 0.9459 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x2s64w8d4e60_8xb4-lr2e-4-500k_div2k-131d3f64.pth) \| log |
| [swinir_x3s64w8d4e60_8xb4-lr2e-4-500k_div2k](/configs/swinir/swinir_x3s64w8d4e60_8xb4-lr2e-4-500k_div2k.py) |  Set5   |  x3   | 34.6037 | 0.9293 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x3s64w8d4e60_8xb4-lr2e-4-500k_div2k-309cb239.pth) \| log |

| [swinir_x3s64w8d4e60_8xb4-lr2e-4-500k_div2k](/configs/swinir/swinir_x3s64w8d4e60_8xb4-lr2e-4-500k_div2k.py) |  Set14  |  x3   | 30.5340 | 0.8468 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x3s64w8d4e60_8xb4-lr2e-4-500k_div2k-309cb239.pth) \| log |
| [swinir_x3s64w8d4e60_8xb4-lr2e-4-500k_div2k](/configs/swinir/swinir_x3s64w8d4e60_8xb4-lr2e-4-500k_div2k.py) |  DIV2K  |  x3   | 33.8394 | 0.8867 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x3s64w8d4e60_8xb4-lr2e-4-500k_div2k-309cb239.pth) \| log |
| [swinir_x4s64w8d4e60_8xb4-lr2e-4-500k_div2k](/configs/swinir/swinir_x4s64w8d4e60_8xb4-lr2e-4-500k_div2k.py) |  Set5   |  x4   | 32.4343 | 0.8984 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x4s64w8d4e60_8xb4-lr2e-4-500k_div2k-d6622d03.pth) \| log |

| [swinir_x4s64w8d4e60_8xb4-lr2e-4-500k_div2k](/configs/swinir/swinir_x4s64w8d4e60_8xb4-lr2e-4-500k_div2k.py) |  Set14  |  x4   | 28.7441 | 0.7861 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x4s64w8d4e60_8xb4-lr2e-4-500k_div2k-d6622d03.pth) \| log |
| [swinir_x4s64w8d4e60_8xb4-lr2e-4-500k_div2k](/configs/swinir/swinir_x4s64w8d4e60_8xb4-lr2e-4-500k_div2k.py) |  DIV2K  |  x4   | 31.8636 | 0.8353 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x4s64w8d4e60_8xb4-lr2e-4-500k_div2k-d6622d03.pth) \| log |

### **Real-World Image Super-Resolution**

Evaluated on Y channels.
The metrics are `NIQE` .

|                                     Model                                      |      Dataset      |  NIQE  | Training Resources |                                     Download                                      |

| :----------------------------------------------------------------------------: | :---------------: | :----: | :----------------: | :-------------------------------------------------------------------------------: |
| [swinir_gan-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost](/configs/swinir/swinir_gan-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py) | RealSRSet+5images | 5.7975 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_gan-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-os-c6425057.pth) \| log |
| [swinir_psnr-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost](/configs/swinir/swinir_psnr-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py) | RealSRSet+5images | 7.2738 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_psnr-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-os-6f0c425f.pth) \| log |

| [swinir_gan-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost](/configs/swinir/swinir_gan-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py) | RealSRSet+5images | 5.2329 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_gan-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-os-36960d18.pth) \| log |
| [swinir_psnr-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost](/configs/swinir/swinir_psnr-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py) | RealSRSet+5images | 7.7460 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_psnr-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-os-a016a72f.pth) \| log |
| [swinir_gan-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-ost](/configs/swinir/swinir_gan-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-ost.py) | RealSRSet+5images | 5.1464 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_gan-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-os-9f1599b5.pth) \| log |

| [swinir_psnr-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-ost](/configs/swinir/swinir_psnr-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-ost.py) | RealSRSet+5images | 7.6378 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_gan-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-os-9f1599b5.pth) \| log |

### **Grayscale Image Deoising**

Evaluated on grayscale images.
The metrics are `PSNR` .

|                                       Model                                        | Dataset  |  PSNR   | Training Resources |                                       Download                                        |
| :--------------------------------------------------------------------------------: | :------: | :-----: | :----------------: | :-----------------------------------------------------------------------------------: |

| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN15](/configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN15.py) |  Set12   | 33.9731 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN15-6782691b.pth) \| log |
| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN15](/configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN15.py) |  BSD68   | 32.5203 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN15-6782691b.pth) \| log |
| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN15](/configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN15.py) | Urban100 | 34.3424 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN15-6782691b.pth) \| log |

| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN25](/configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN25.py) |  Set12   | 31.6434 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN25-d0d8d4da.pth) \| log |
| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN25](/configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN25.py) |  BSD68   | 30.1377 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN25-d0d8d4da.pth) \| log |
| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN25](/configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN25.py) | Urban100 | 31.9493 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN25-d0d8d4da.pth) \| log |

| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN50](/configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN50.py) |  Set12   | 28.5651 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN50-54c9968a.pth) \| log |
| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN50](/configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN50.py) |  BSD68   | 27.3157 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN50-54c9968a.pth) \| log |
| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN50](/configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN50.py) | Urban100 | 28.6626 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN50-54c9968a.pth) \| log |

### **Color Image Deoising**

Evaluated on RGB channels.

| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15](/configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15.py) |  CBSD68  | 34.4136 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15-c74a2cee.pth) \| log |

| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15](/configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15.py) | Kodak24  | 35.3555 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15-c74a2cee.pth) \| log |
| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15](/configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15.py) | McMaster | 35.6205 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15-c74a2cee.pth) \| log |
| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15](/configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15.py) | Urban100 | 35.1836 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15-c74a2cee.pth) \| log |

| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25](/configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25.py) |  CBSD68  | 31.7626 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25-df2b1c0c.pth) \| log |
| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25](/configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25.py) | Kodak24  | 32.9003 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25-df2b1c0c.pth) \| log |
| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25](/configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25.py) | McMaster | 33.3198 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25-df2b1c0c.pth) \| log |

| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25](/configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25.py) | Urban100 | 32.9458 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25-df2b1c0c.pth) \| log |
| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50](/configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50.py) |  CBSD68  | 28.5346 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50-e369874c.pth) \| log |
| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50](/configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50.py) | Kodak24  | 29.8058 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50-e369874c.pth) \| log |

| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50](/configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50.py) | McMaster | 30.2027 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50-e369874c.pth) \| log |
| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50](/configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50.py) | Urban100 | 29.8832 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50-e369874c.pth) \| log |

### **JPEG Compression Artifact Reduction (grayscale)**

|                                      Model                                      | Dataset  |  PSNR   |  SSIM  | Training Resources |                                      Download                                      |

| :-----------------------------------------------------------------------------: | :------: | :-----: | :----: | :----------------: | :--------------------------------------------------------------------------------: |
| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR10](/configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR10.py) | Classic5 | 30.2746 | 0.8254 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR10-da93c8e9.pth) \| log |
| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR10](/configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR10.py) |  LIVE1   | 29.8611 | 0.8292 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR10-da93c8e9.pth) \| log |

| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR20](/configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR20.py) | Classic5 | 32.5331 | 0.8753 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR20-d47367b1.pth) \| log |
| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR20](/configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR20.py) |  LIVE1   | 32.2667 | 0.8914 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR20-d47367b1.pth) \| log |
| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR30](/configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR30.py) | Classic5 | 33.7504 | 0.8966 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR30-52c083cf.pth) \| log |

| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR30](/configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR30.py) |  LIVE1   | 33.7001 | 0.9179 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR30-52c083cf.pth) \| log |
| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR40](/configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR40.py) | Classic5 | 34.5377 | 0.9087 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR40-803e8d9b.pth) \| log |
| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR40](/configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR40.py) |  LIVE1   | 34.6846 | 0.9322 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR40-803e8d9b.pth) \| log |

### **JPEG Compression Artifact Reduction (color)**

| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR10](/configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR10.py) | Classic5 | 30.1019 | 0.8217 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR10-09aafadc.pth) \| log |

| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR10](/configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR10.py) |  LIVE1   | 28.0676 | 0.8094 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR10-09aafadc.pth) \| log |
| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR20](/configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR20.py) | Classic5 | 32.3489 | 0.8727 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR20-b8a42b5e.pth) \| log |
| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR20](/configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR20.py) |  LIVE1   | 30.4514 | 0.8745 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR20-b8a42b5e.pth) \| log |

| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR30](/configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR30.py) | Classic5 | 33.6028 | 0.8949 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR30-e9fe6859.pth) \| log |
| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR30](/configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR30.py) |  LIVE1   | 31.8235 | 0.9023 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR30-e9fe6859.pth) \| log |
| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR40](/configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR40.py) | Classic5 | 34.4344 | 0.9076 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR40-5b77a6e6.pth) \| log |

| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR40](/configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR40.py) |  LIVE1   | 32.7610 | 0.9179 |         8          | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR40-5b77a6e6.pth) \| log |

## Quick Start

**Train**

<details>
<summary>Train Instructions</summary>

You can use the following commands to train a model with cpu or single/multiple GPUs.

```shell
# cpu train
# 001 Classical Image Super-Resolution (middle size)
# (setting1: when model is trained on DIV2K and with training_patch_size=48)
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_x2s48w8d6e180_8xb4-lr2e-4-500k_div2k.py
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_x3s48w8d6e180_8xb4-lr2e-4-500k_div2k.py
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_x4s48w8d6e180_8xb4-lr2e-4-500k_div2k.py

# (setting2: when model is trained on DIV2K+Flickr2K and with training_patch_size=64)
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_x2s64w8d6e180_8xb4-lr2e-4-500k_df2k.py
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_x3s64w8d6e180_8xb4-lr2e-4-500k_df2k.py
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_x4s64w8d6e180_8xb4-lr2e-4-500k_df2k.py

# 002 Lightweight Image Super-Resolution (small size)
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_x2s64w8d4e60_8xb4-lr2e-4-500k_div2k.py
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_x3s64w8d4e60_8xb4-lr2e-4-500k_div2k.py
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_x4s64w8d4e60_8xb4-lr2e-4-500k_div2k.py

# 003 Real-World Image Super-Resolution

CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_gan-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_psnr-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_gan-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_psnr-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_gan-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-ost.py
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_psnr-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-ost.py

# 004 Grayscale Image Deoising (middle size)

CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN15.py
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN25.py
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN50.py

# 005 Color Image Deoising (middle size)
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15.py
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25.py
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50.py

# 006 JPEG Compression Artifact Reduction (middle size, using window_size=7 because JPEG encoding uses 8x8 blocks)
# grayscale
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR10.py
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR20.py
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR30.py
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR40.py

# color
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR10.py
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR20.py

CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR30.py
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR40.py

# single-gpu train
python tools/train.py configs/swinir/swinir_x2s48w8d6e180_8xb4-lr2e-4-500k_div2k.py
python tools/train.py configs/swinir/swinir_x3s48w8d6e180_8xb4-lr2e-4-500k_div2k.py
python tools/train.py configs/swinir/swinir_x4s48w8d6e180_8xb4-lr2e-4-500k_div2k.py

python tools/train.py configs/swinir/swinir_x2s64w8d6e180_8xb4-lr2e-4-500k_df2k.py

python tools/train.py configs/swinir/swinir_x3s64w8d6e180_8xb4-lr2e-4-500k_df2k.py
python tools/train.py configs/swinir/swinir_x4s64w8d6e180_8xb4-lr2e-4-500k_df2k.py

python tools/train.py configs/swinir/swinir_x2s64w8d4e60_8xb4-lr2e-4-500k_div2k.py
python tools/train.py configs/swinir/swinir_x3s64w8d4e60_8xb4-lr2e-4-500k_div2k.py
python tools/train.py configs/swinir/swinir_x4s64w8d4e60_8xb4-lr2e-4-500k_div2k.py

python tools/train.py configs/swinir/swinir_gan-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py
python tools/train.py configs/swinir/swinir_psnr-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py
python tools/train.py configs/swinir/swinir_gan-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py

python tools/train.py configs/swinir/swinir_psnr-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py
python tools/train.py configs/swinir/swinir_gan-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-ost.py
python tools/train.py configs/swinir/swinir_psnr-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-ost.py

python tools/train.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN15.py
python tools/train.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN25.py
python tools/train.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN50.py

python tools/train.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15.py

python tools/train.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25.py
python tools/train.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50.py

python tools/train.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR10.py
python tools/train.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR20.py
python tools/train.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR30.py
python tools/train.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR40.py

# color
python tools/train.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR10.py

# color
python tools/train.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR20.py
python tools/train.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR30.py
python tools/train.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR40.py

# multi-gpu train
./tools/dist_train.sh configs/swinir/swinir_x2s48w8d6e180_8xb4-lr2e-4-500k_div2k.py 8
./tools/dist_train.sh configs/swinir/swinir_x3s48w8d6e180_8xb4-lr2e-4-500k_div2k.py 8
./tools/dist_train.sh configs/swinir/swinir_x4s48w8d6e180_8xb4-lr2e-4-500k_div2k.py 8

./tools/dist_train.sh configs/swinir/swinir_x2s64w8d6e180_8xb4-lr2e-4-500k_df2k.py 8
./tools/dist_train.sh configs/swinir/swinir_x3s64w8d6e180_8xb4-lr2e-4-500k_df2k.py 8
./tools/dist_train.sh configs/swinir/swinir_x4s64w8d6e180_8xb4-lr2e-4-500k_df2k.py 8

./tools/dist_train.sh configs/swinir/swinir_x2s64w8d4e60_8xb4-lr2e-4-500k_div2k.py 8
./tools/dist_train.sh configs/swinir/swinir_x3s64w8d4e60_8xb4-lr2e-4-500k_div2k.py 8
./tools/dist_train.sh configs/swinir/swinir_x4s64w8d4e60_8xb4-lr2e-4-500k_div2k.py 8

./tools/dist_train.sh configs/swinir/swinir_gan-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py 8
./tools/dist_train.sh configs/swinir/swinir_psnr-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py 8

./tools/dist_train.sh configs/swinir/swinir_gan-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py 8
./tools/dist_train.sh configs/swinir/swinir_psnr-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py 8
./tools/dist_train.sh configs/swinir/swinir_gan-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-ost.py 8
./tools/dist_train.sh configs/swinir/swinir_psnr-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-ost.py 8

./tools/dist_train.sh configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN15.py 8
./tools/dist_train.sh configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN25.py 8
./tools/dist_train.sh configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN50.py 8

./tools/dist_train.sh configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15.py 8
./tools/dist_train.sh configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25.py 8
./tools/dist_train.sh configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50.py 8

./tools/dist_train.sh configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR10.py 8
./tools/dist_train.sh configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR20.py 8
./tools/dist_train.sh configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR30.py 8
./tools/dist_train.sh configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR40.py 8

# color

# color
./tools/dist_train.sh configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR10.py 8
./tools/dist_train.sh configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR20.py 8
./tools/dist_train.sh configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR30.py 8
./tools/dist_train.sh configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR40.py 8
```

For more details, you can refer to **Train a model** part in [train_test.md](/docs/en/user_guides/train_test.md#Train-a-model-in-MMagic).

**Test**

<details>
<summary>Test Instructions</summary>

You can use the following commands to test a model with cpu or single/multiple GPUs.

```shell
# cpu test

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_x2s48w8d6e180_8xb4-lr2e-4-500k_div2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x2s48w8d6e180_8xb4-lr2e-4-500k_div2k-ed2d419e.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_x3s48w8d6e180_8xb4-lr2e-4-500k_div2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x3s48w8d6e180_8xb4-lr2e-4-500k_div2k-926950f1.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_x4s48w8d6e180_8xb4-lr2e-4-500k_div2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x4s48w8d6e180_8xb4-lr2e-4-500k_div2k-88e4903d.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_x2s64w8d6e180_8xb4-lr2e-4-500k_df2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x2s64w8d6e180_8xb4-lr2e-4-500k_df2k-69e15fb6.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_x3s64w8d6e180_8xb4-lr2e-4-500k_df2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x3s64w8d6e180_8xb4-lr2e-4-500k_df2k-d6982f7b.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_x4s64w8d6e180_8xb4-lr2e-4-500k_df2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x4s64w8d6e180_8xb4-lr2e-4-500k_df2k-0502d775.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_x2s64w8d4e60_8xb4-lr2e-4-500k_div2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x2s64w8d4e60_8xb4-lr2e-4-500k_div2k-131d3f64.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_x3s64w8d4e60_8xb4-lr2e-4-500k_div2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x3s64w8d4e60_8xb4-lr2e-4-500k_div2k-309cb239.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_x4s64w8d4e60_8xb4-lr2e-4-500k_div2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x4s64w8d4e60_8xb4-lr2e-4-500k_div2k-d6622d03.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_gan-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py https://download.openmmlab.com/mmediting/swinir/swinir_gan-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-os-c6425057.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_psnr-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py https://download.openmmlab.com/mmediting/swinir/swinir_psnr-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-os-6f0c425f.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_gan-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py https://download.openmmlab.com/mmediting/swinir/swinir_gan-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-os-36960d18.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_psnr-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py https://download.openmmlab.com/mmediting/swinir/swinir_psnr-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-os-a016a72f.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_gan-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-ost.py https://download.openmmlab.com/mmediting/swinir/swinir_gan-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-os-9f1599b5.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_psnr-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-ost.py https://download.openmmlab.com/mmediting/swinir/swinir_psnr-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-os-25f1722a.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN15.py https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN15-6782691b.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN25.py https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN25-d0d8d4da.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN50.py https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN50-54c9968a.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15.py https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15-c74a2cee.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25.py https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25-df2b1c0c.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50.py https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50-e369874c.pth

# 006 JPEG Compression Artifact Reduction (middle size, using window_size=7 because JPEG encoding usesx8 blocks)

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR10.py https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR10-da93c8e9.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR20.py https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR20-d47367b1.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR30.py https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR30-52c083cf.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR40.py https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR40-803e8d9b.pth

# color
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR10.py https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR10-09aafadc.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR20.py https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR20-b8a42b5e.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR30.py https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR30-e9fe6859.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR40.py https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR40-5b77a6e6.pth

# single-gpu test
python tools/test.py configs/swinir/swinir_x2s48w8d6e180_8xb4-lr2e-4-500k_div2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x2s48w8d6e180_8xb4-lr2e-4-500k_div2k-ed2d419e.pth

python tools/test.py configs/swinir/swinir_x3s48w8d6e180_8xb4-lr2e-4-500k_div2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x3s48w8d6e180_8xb4-lr2e-4-500k_div2k-926950f1.pth

python tools/test.py configs/swinir/swinir_x4s48w8d6e180_8xb4-lr2e-4-500k_div2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x4s48w8d6e180_8xb4-lr2e-4-500k_div2k-88e4903d.pth

python tools/test.py configs/swinir/swinir_x2s64w8d6e180_8xb4-lr2e-4-500k_df2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x2s64w8d6e180_8xb4-lr2e-4-500k_df2k-69e15fb6.pth

python tools/test.py configs/swinir/swinir_x3s64w8d6e180_8xb4-lr2e-4-500k_df2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x3s64w8d6e180_8xb4-lr2e-4-500k_df2k-d6982f7b.pth

python tools/test.py configs/swinir/swinir_x4s64w8d6e180_8xb4-lr2e-4-500k_df2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x4s64w8d6e180_8xb4-lr2e-4-500k_df2k-0502d775.pth

python tools/test.py configs/swinir/swinir_x2s64w8d4e60_8xb4-lr2e-4-500k_div2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x2s64w8d4e60_8xb4-lr2e-4-500k_div2k-131d3f64.pth

python tools/test.py configs/swinir/swinir_x3s64w8d4e60_8xb4-lr2e-4-500k_div2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x3s64w8d4e60_8xb4-lr2e-4-500k_div2k-309cb239.pth

python tools/test.py configs/swinir/swinir_x4s64w8d4e60_8xb4-lr2e-4-500k_div2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x4s64w8d4e60_8xb4-lr2e-4-500k_div2k-d6622d03.pth

python tools/test.py configs/swinir/swinir_gan-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py https://download.openmmlab.com/mmediting/swinir/swinir_gan-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-os-c6425057.pth

python tools/test.py configs/swinir/swinir_psnr-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-ost.py https://download.openmmlab.com/mmediting/swinir/swinir_psnr-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-os-25f1722a.pth

python tools/test.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN15.py https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN15-6782691b.pth

python tools/test.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN25.py https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN25-d0d8d4da.pth

python tools/test.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN50.py https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN50-54c9968a.pth

python tools/test.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15.py https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15-c74a2cee.pth

python tools/test.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25.py https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25-df2b1c0c.pth

python tools/test.py configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50.py https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50-e369874c.pth

python tools/test.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR10.py https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR10-da93c8e9.pth

python tools/test.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR20.py https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR20-d47367b1.pth

python tools/test.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR30.py https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR30-52c083cf.pth

python tools/test.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR40.py https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR40-803e8d9b.pth

# color
python tools/test.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR10.py https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR10-09aafadc.pth

python tools/test.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR20.py https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR20-b8a42b5e.pth

python tools/test.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR30.py https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR30-e9fe6859.pth

python tools/test.py configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR40.py https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR40-5b77a6e6.pth

# multi-gpu test
./tools/dist_test.sh configs/swinir/swinir_x2s48w8d6e180_8xb4-lr2e-4-500k_div2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x2s48w8d6e180_8xb4-lr2e-4-500k_div2k-ed2d419e.pth

./tools/dist_test.sh configs/swinir/swinir_x3s48w8d6e180_8xb4-lr2e-4-500k_div2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x3s48w8d6e180_8xb4-lr2e-4-500k_div2k-926950f1.pth

./tools/dist_test.sh configs/swinir/swinir_x4s48w8d6e180_8xb4-lr2e-4-500k_div2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x4s48w8d6e180_8xb4-lr2e-4-500k_div2k-88e4903d.pth

./tools/dist_test.sh configs/swinir/swinir_x2s64w8d6e180_8xb4-lr2e-4-500k_df2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x2s64w8d6e180_8xb4-lr2e-4-500k_df2k-69e15fb6.pth

./tools/dist_test.sh configs/swinir/swinir_x3s64w8d6e180_8xb4-lr2e-4-500k_df2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x3s64w8d6e180_8xb4-lr2e-4-500k_df2k-d6982f7b.pth

./tools/dist_test.sh configs/swinir/swinir_x4s64w8d6e180_8xb4-lr2e-4-500k_df2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x4s64w8d6e180_8xb4-lr2e-4-500k_df2k-0502d775.pth

./tools/dist_test.sh configs/swinir/swinir_x2s64w8d4e60_8xb4-lr2e-4-500k_div2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x2s64w8d4e60_8xb4-lr2e-4-500k_div2k-131d3f64.pth

./tools/dist_test.sh configs/swinir/swinir_x3s64w8d4e60_8xb4-lr2e-4-500k_div2k.py https://download.openmmlab.com/mmediting/swinir/swinir_x3s64w8d4e60_8xb4-lr2e-4-500k_div2k-309cb239.pth

./tools/dist_test.sh configs/swinir/swinir_gan-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py https://download.openmmlab.com/mmediting/swinir/swinir_gan-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-os-36960d18.pth

./tools/dist_test.sh configs/swinir/swinir_psnr-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py https://download.openmmlab.com/mmediting/swinir/swinir_psnr-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-os-a016a72f.pth

./tools/dist_test.sh configs/swinir/swinir_gan-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-ost.py https://download.openmmlab.com/mmediting/swinir/swinir_gan-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-os-9f1599b5.pth

./tools/dist_test.sh configs/swinir/swinir_psnr-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-ost.py https://download.openmmlab.com/mmediting/swinir/swinir_psnr-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-os-25f1722a.pth

./tools/dist_test.sh configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN15.py https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN15-6782691b.pth

./tools/dist_test.sh configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN25.py https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN25-d0d8d4da.pth

./tools/dist_test.sh configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN50.py https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN50-54c9968a.pth

./tools/dist_test.sh configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15.py https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15-c74a2cee.pth

./tools/dist_test.sh configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25.py https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25-df2b1c0c.pth

./tools/dist_test.sh configs/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50.py https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50-e369874c.pth

./tools/dist_test.sh configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR10.py https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR10-da93c8e9.pth

./tools/dist_test.sh configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR20.py https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR20-d47367b1.pth

./tools/dist_test.sh configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR30.py https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR30-e9fe6859.pth

./tools/dist_test.sh configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR40.py https://download.openmmlab.com/mmediting/swinir/

```

For more details, you can refer to **Test a pre-trained model** part in [train_test.md](/docs/en/user_guides/train_test.md#Test-a-pre-trained-model-in-MMagic).

```bibtex
@inproceedings{liang2021swinir,
  title={Swinir: Image restoration using swin transformer},
  author={Liang, Jingyun and Cao, Jiezhang and Sun, Guolei and Zhang, Kai and Van Gool, Luc and Timofte, Radu},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={1833--1844},
  year={2021}
}
```

# 训练

在本教程中，我们将介绍如何使用 MMPretrain 中提供的脚本启动训练任务。
如果你需要了解一些具体的训练例子，可以查阅 [如何在自定义数据集上进行模型预训练](../notes/pretrain_custom_dataset.md) 和 [如何在自定义数据集上微调模型](../notes/finetune_custom_dataset.md).

## 单机单卡训练

你可以使用 `tools/train.py` 在电脑上用 CPU 或是 GPU 进行模型的训练。

以下是训练脚本的完整用法：

```shell
python tools/train.py ${CONFIG_FILE} [ARGS]
```

````{note}
默认情况下，MMPretrain 会自动调用你的 GPU 进行训练。如果你有 GPU 但仍想使用 CPU 进行训练，请设置环境变量 `CUDA_VISIBLE_DEVICES` 为空或者 -1 来对禁用 GPU。

```bash
CUDA_VISIBLE_DEVICES=-1 python tools/train.py ${CONFIG_FILE} [ARGS]
```
````

| 参数                                  | 描述                                                                                                                                                                |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

| `CONFIG_FILE`                         | 配置文件的路径。                                                                                                                                                    |
| `--work-dir WORK_DIR`                 | 用来保存训练日志和权重文件的文件夹，默认是 `./work_dirs` 目录下，与配置文件同名的文件夹。                                                                           |
| `--resume [RESUME]`                   | 恢复训练。如果指定了权重文件路径，则从指定的权重文件恢复；如果没有指定，则尝试从最新的权重文件进行恢复。                                                            |
| `--amp`                               | 启用混合精度训练。                                                                                                                                                  |
| `--no-validate`                       | **不建议** 在训练过程中不进行验证集上的精度验证。                                                                                                                   |

| `--auto-scale-lr`                     | 自动根据实际的批次大小（batch size）和预设的批次大小对学习率进行缩放。                                                                                              |
| `--no-pin-memory`                     | 是否在 dataloaders 中关闭 `pin_memory` 选项                                                                                                                         |
| `--no-persistent-workers`             | 是否在 dataloaders 中关闭 `persistent_workers` 选项                                                                                                                 |
| `--cfg-options CFG_OPTIONS`           | 重载配置文件中的一些设置。使用类似 `xxx=yyy` 的键值对形式指定，这些设置会被融合入从配置文件读取的配置。你可以使用 `key="[a,b]"` 或者 `key=a,b` 的格式来指定列表格式的值，且支持嵌套，例如 \`key="[(a,b),(c,d)]"，这里的引号是不可省略的。另外每个重载项内部不可出现空格。 |

| `--launcher {none,pytorch,slurm,mpi}` | 启动器，默认为 "none"。                                                                                                                                             |

## 单机多卡训练

我们提供了一个 shell 脚本，可以使用 `torch.distributed.launch` 启动多 GPU 任务。

```shell
bash ./tools/dist_train.sh ${CONFIG_FILE} ${GPU_NUM} [PY_ARGS]
```

| 参数          | 描述                                                             |
| ------------- | ---------------------------------------------------------------- |
| `CONFIG_FILE` | 配置文件的路径。                                                 |
| `GPU_NUM`     | 使用的 GPU 数量。                                                |
| `[PY_ARGS]`   | `tools/train.py` 支持的其他可选参数，参见[上文](#单机单卡训练)。 |

你还可以使用环境变量来指定启动器的额外参数，比如用如下命令将启动器的通讯端口变更为 29666：

```shell
PORT=29666 bash ./tools/dist_train.sh ${CONFIG_FILE} ${GPU_NUM} [PY_ARGS]
```

如果你希望使用不同的 GPU 进行多项训练任务，可以在启动时指定不同的通讯端口和不同的可用设备。

```shell

```shell
```

```shell
CUDA_VISIBLE_DEVICES=0,1,2,3 PORT=29500 bash ./tools/dist_train.sh ${CONFIG_FILE1} 4 [PY_ARGS]
CUDA_VISIBLE_DEVICES=4,5,6,7 PORT=29501 bash ./tools/dist_train.sh ${CONFIG_FILE2} 4 [PY_ARGS]
```

## 多机训练

### 同一网络下的多机

如果你希望使用同一局域网下连接的多台电脑进行一个训练任务，可以使用如下命令：

在第一台机器上：

```shell
NNODES=2 NODE_RANK=0 PORT=$MASTER_PORT MASTER_ADDR=$MASTER_ADDR bash tools/dist_train.sh $CONFIG $GPUS
```

在第二台机器上：

```shell
NNODES=2 NODE_RANK=1 PORT=$MASTER_PORT MASTER_ADDR=$MASTER_ADDR bash tools/dist_train.sh $CONFIG $GPUS
```

和单机多卡相比，你需要指定一些额外的环境变量：

| 环境变量      | 描述                                           |
| ------------- | ---------------------------------------------- |
| `NNODES`      | 机器总数。                                     |
| `NODE_RANK`   | 本机的序号                                     |

| `PORT`        | 通讯端口，它在所有机器上都应当是一致的。       |
| `MASTER_ADDR` | 主机的 IP 地址，它在所有机器上都应当是一致的。 |

通常来说，如果这几台机器之间不是高速网络连接，训练速度会非常慢。

### Slurm 管理下的多机集群

如果你在 [slurm](https://slurm.schedmd.com/) 集群上，可以使用 `tools/slurm_train.sh` 脚本启动任务。

```shell
[ENV_VARS] ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} ${CONFIG_FILE} ${WORK_DIR} [PY_ARGS]
```

这里是该脚本的一些参数：

| 参数          | 描述                                                             |
| `PARTITION`   | 使用的集群分区。                                                 |
| `JOB_NAME`    | 任务的名称，你可以随意起一个名字。                               |
| `CONFIG_FILE` | 配置文件路径。                                                   |

| `WORK_DIR`    | 用以保存日志和权重文件的文件夹。                                 |

这里是一些你可以用来配置 slurm 任务的环境变量：

| 环境变量        | 描述                                                                                       |
| --------------- | ------------------------------------------------------------------------------------------ |
| `GPUS`          | 使用的 GPU 总数，默认为 8。                                                                |
| `GPUS_PER_NODE` | 每个节点分配的 GPU 数，你可以根据节点情况指定。默认为 8。                                  |
| `CPUS_PER_TASK` | 每个任务分配的 CPU 数（通常一个 GPU 对应一个任务）。默认为 5。                             |
| `SRUN_ARGS`     | `srun` 命令支持的其他参数。可用的选项参见[官方文档](https://slurm.schedmd.com/srun.html)。 |

# BasicVSR++ (CVPR'2022)

> [BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment](https://arxiv.org/abs/2104.13371)

> **Task**: Video Super-Resolution

A recurrent structure is a popular framework choice for the task of video super-resolution. The state-of-the-art method BasicVSR adopts bidirectional propagation with feature alignment to effectively exploit information from the entire input video. In this study, we redesign BasicVSR by proposing second-order grid propagation and flow-guided deformable alignment. We show that by empowering the recurrent framework with the enhanced propagation and alignment, one can exploit spatiotemporal information across misaligned video frames more effectively. The new components lead to an improved performance under a similar computational constraint. In particular, our model BasicVSR++ surpasses BasicVSR by 0.82 dB in PSNR with similar number of parameters. In addition to video super-resolution, BasicVSR++ generalizes well to other video restoration tasks such as compressed video enhancement. In NTIRE 2021, BasicVSR++ obtains three champions and one runner-up in the Video Super-Resolution and Compressed Video Enhancement Challenges. Codes and models will be released to MMagic.

 <img src="https://user-images.githubusercontent.com/7676947/144017685-9354df55-aa6d-445f-a946-116f0d6c38d7.png" width="400"/>
</div >

The pretrained weights of SPyNet can be found [here](https://download.openmmlab.com/mmediting/restorers/basicvsr/spynet_20210409-c6c1bd09.pth).

|                          Model                           |      Dataset       | PSNR (RGB)  | SSIM (RGB) |  PSNR (Y)   |  SSIM (Y)  |    Training Resources    |                           Download                           |
| :------------------------------------------------------: | :----------------: | :---------: | :--------: | :---------: | :--------: | :----------------------: | :----------------------------------------------------------: |

| [basicvsr_plusplus_c64n7_8x1_600k_reds4](./basicvsr-pp_c64n7_8xb1-600k_reds4.py) |    REDS4 (BIx4)    | **32.3855** | **0.9069** |      -      |     -      | 8 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_600k_reds4_20210217-db622b2f.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_600k_reds4_20210217_113115.log.json) |
| [basicvsr_plusplus_c64n7_8x1_600k_reds4](./basicvsr-pp_c64n7_8xb1-600k_reds4.py) |    UDM10 (BDx4)    |      -      |     -      |   34.6868   |   0.9417   | 8 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_600k_reds4_20210217-db622b2f.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_600k_reds4_20210217_113115.log.json) |

| [basicvsr_plusplus_c64n7_8x1_600k_reds4](./basicvsr-pp_c64n7_8xb1-600k_reds4.py) |    Vid4 (BIx4)     |      -      |     -      |   27.7674   |   0.8444   | 8 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_600k_reds4_20210217-db622b2f.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_600k_reds4_20210217_113115.log.json) |
| [basicvsr_plusplus_c64n7_8x1_600k_reds4](./basicvsr-pp_c64n7_8xb1-600k_reds4.py) |    Vid4 (BDx4)     |      -      |     -      |   24.6209   |   0.7540   | 8 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_600k_reds4_20210217-db622b2f.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_600k_reds4_20210217_113115.log.json) |

| [basicvsr_plusplus_c64n7_8x1_600k_reds4](./basicvsr-pp_c64n7_8xb1-600k_reds4.py) | Vimeo-90K-T (BIx4) |      -      |     -      |   36.4445   |   0.9411   | 8 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_600k_reds4_20210217-db622b2f.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_600k_reds4_20210217_113115.log.json) |
| [basicvsr_plusplus_c64n7_8x1_600k_reds4](./basicvsr-pp_c64n7_8xb1-600k_reds4.py) | Vimeo-90K-T (BDx4) |      -      |     -      |   34.0372   |   0.9244   | 8 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_600k_reds4_20210217-db622b2f.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_600k_reds4_20210217_113115.log.json) |

| [basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bi](./basicvsr-pp_c64n7_4xb2-300k_vimeo90k-bi.py) |    REDS4 (BIx4)    |   31.0126   |   0.8804   |      -      |     -      | 4 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bi_20210305-4ef437e2.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bi_20210305_141254.log.json) |
| [basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bi](./basicvsr-pp_c64n7_4xb2-300k_vimeo90k-bi.py) |    UDM10 (BDx4)    |      -      |     -      |   33.1211   |   0.9270   | 4 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bi_20210305-4ef437e2.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bi_20210305_141254.log.json) |

| [basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bi](./basicvsr-pp_c64n7_4xb2-300k_vimeo90k-bi.py) |    Vid4 (BIx4)     |      -      |     -      | **27.7882** | **0.8401** | 4 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bi_20210305-4ef437e2.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bi_20210305_141254.log.json) |
| [basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bi](./basicvsr-pp_c64n7_4xb2-300k_vimeo90k-bi.py) |    Vid4 (BDx4)     |      -      |     -      |   23.6086   |   0.7033   | 4 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bi_20210305-4ef437e2.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bi_20210305_141254.log.json) |

| [basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bi](./basicvsr-pp_c64n7_4xb2-300k_vimeo90k-bi.py) | Vimeo-90K-T (BIx4) |      -      |     -      | **37.7864** | **0.9500** | 4 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bi_20210305-4ef437e2.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bi_20210305_141254.log.json) |
| [basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bi](./basicvsr-pp_c64n7_4xb2-300k_vimeo90k-bi.py) | Vimeo-90K-T (BDx4) |      -      |     -      |   33.8972   |   0.9195   | 4 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bi_20210305-4ef437e2.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bi_20210305_141254.log.json) |

| [basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bd](./basicvsr-pp_c64n7_4xb2-300k_vimeo90k-bd.py) |    REDS4 (BIx4)    |   29.2041   |   0.8528   |      -      |     -      | 4 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bd_20210305-ab315ab1.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bd_20210305_140921.log.json) |
| [basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bd](./basicvsr-pp_c64n7_4xb2-300k_vimeo90k-bd.py) |    UDM10 (BDx4)    |      -      |     -      | **40.7216** | **0.9722** | 4 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bd_20210305-ab315ab1.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bd_20210305_140921.log.json) |

| [basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bd](./basicvsr-pp_c64n7_4xb2-300k_vimeo90k-bd.py) |    Vid4 (BIx4)     |      -      |     -      |   26.4377   |   0.8074   | 4 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bd_20210305-ab315ab1.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bd_20210305_140921.log.json) |
| [basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bd](./basicvsr-pp_c64n7_4xb2-300k_vimeo90k-bd.py) |    Vid4 (BDx4)     |      -      |     -      | **29.0400** | **0.8753** | 4 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bd_20210305-ab315ab1.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bd_20210305_140921.log.json) |

| [basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bd](./basicvsr-pp_c64n7_4xb2-300k_vimeo90k-bd.py) | Vimeo-90K-T (BIx4) |      -      |     -      |   34.7248   |   0.9351   | 4 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bd_20210305-ab315ab1.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bd_20210305_140921.log.json) |
| [basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bd](./basicvsr-pp_c64n7_4xb2-300k_vimeo90k-bd.py) | Vimeo-90K-T (BDx4) |      -      |     -      | **38.2054** | **0.9550** | 4 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bd_20210305-ab315ab1.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bd_20210305_140921.log.json) |

**NTIRE 2021 checkpoints**

Note that the following models are finetuned from smaller models. The training schemes of these models will be released when MMagic reaches 5k stars. We provide the pre-trained models here.

|                                Model                                 |                           Dataset                            |                                 Download                                 |
| :------------------------------------------------------------------: | :----------------------------------------------------------: | :----------------------------------------------------------------------: |
| [basicvsr-pp_c128n25_600k_ntire-vsr](./basicvsr-pp_c128n25_600k_ntire-vsr.py) |         NTIRE 2021 Video Super-Resolution - Track 1          | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c128n25_ntire_vsr_20210311-1ff35292.pth) |

| [basicvsr-pp_c128n25_600k_ntire-decompress-track1](./basicvsr-pp_c128n25_600k_ntire-decompress-track1.py) | NTIRE 2021 Quality Enhancement of Compressed Video - Track 1 | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c128n25_ntire_decompress_track1_20210223-7b2eba02.pth) |
| [basicvsr-pp_c128n25_600k_ntire-decompress-track2](./basicvsr-pp_c128n25_600k_ntire-decompress-track2.py) | NTIRE 2021 Quality Enhancement of Compressed Video - Track 2 | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c128n25_ntire_decompress_track2_20210314-eeae05e6.pth) |
| [basicvsr-pp_c128n25_600k_ntire-decompress-track3](./basicvsr-pp_c128n25_600k_ntire-decompress-track3.py) | NTIRE 2021 Quality Enhancement of Compressed Video - Track 3 | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c128n25_ntire_decompress_track3_20210304-6daf4a40.pth) |

**Train**

<details>

```shell
# cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/basicvsr_pp/basicvsr-pp_c64n7_8xb1-600k_reds4.py

python tools/train.py configs/basicvsr_pp/basicvsr-pp_c64n7_8xb1-600k_reds4.py

./tools/dist_train.sh configs/basicvsr_pp/basicvsr-pp_c64n7_8xb1-600k_reds4.py 8
```

**Test**

<details>

```shell
# cpu test

**Test**

<details>

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/basicvsr_pp/basicvsr-pp_c64n7_8xb1-600k_reds4.py https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_600k_reds4_20210217-db622b2f.pth

python tools/test.py configs/basicvsr_pp/basicvsr-pp_c64n7_8xb1-600k_reds4.py https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_600k_reds4_20210217-db622b2f.pth

./tools/dist_test.sh configs/basicvsr_pp/basicvsr-pp_c64n7_8xb1-600k_reds4.py https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_600k_reds4_20210217-db622b2f.pth 8
```

```

```bibtex
@InProceedings{chan2022basicvsrplusplus,
  author = {Chan, Kelvin C.K. and Zhou, Shangchen and Xu, Xiangyu and Loy, Chen Change},
  title = {BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment},
  booktitle = {Proceedings of the IEEE conference on computer vision and pattern recognition},
  year = {2022}
}
```

# Gesture Recognition

In this project, we present a skeleton based pipeline for gesture recognition. The pipeline is three-stage. The first stage consists of a hand detection module that outputs bounding boxes of human hands from video frames. Afterwards, the second stage employs a pose estimation module to generate keypoints of the detected hands. Finally, the third stage utilizes a skeleton-based gesture recognition module to classify hand actions based on the provided hand skeleton. The three-stage pipeline is lightweight and can achieve real-time on CPU devices. In this README, we provide the models and the inference demo for the project. Training data preparation and training scripts are described in [TRAINING.md](/projects/gesture_recognition/TRAINING.md).

## Hand detection stage

Hand detection results on OneHand10K validation dataset

| Config                                                  | Input Size | bbox mAP | bbox mAP 50 | bbox mAP 75 |                         ckpt                          |                         log                          |
| :------------------------------------------------------ | :--------: | :------: | :---------: | :---------: | :---------------------------------------------------: | :--------------------------------------------------: |
| [rtmpose_nano](/projects/gesture_recognition/configs/rtmdet-nano_8xb32-300e_multi-dataset-hand-320x320.py) |  320x320   |  0.8100  |   0.9870    |   0.9190    | [ckpt](https://download.openmmlab.com/mmaction/v1.0/projects/gesture_recognition/rtmdet-nano_8xb32-300e_multi-dataset-hand-320x320_20230524-f6ffed6a.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/projects/gesture_recognition/rtmdet-nano_8xb32-300e_multi-dataset-hand-320x320.log) |

## Pose estimation stage

Pose estimation results on COCO-WholeBody-Hand validation set

| Config                                                                                                 | Input Size | PCK@0.2 |  AUC  | EPE  |                  ckpt                   |
| :----------------------------------------------------------------------------------------------------- | :--------: | :-----: | :---: | :--: | :-------------------------------------: |
| [rtmpose_m](/projects/gesture_recognition/configs/rtmpose-m_8xb32-210e_coco-wholebody-hand-256x256.py) |  256x256   |  0.815  | 0.837 | 4.51 | [ckpt](https://download.openmmlab.com/) |

## Gesture recognition stage

Skeleton base gesture recognition results on Jester validation

| Config                                                  | Input Size | Top 1 accuracy | Top 5 accuracy |                          ckpt                          |                          log                          |

| :------------------------------------------------------ | :--------: | :------------: | :------------: | :----------------------------------------------------: | :---------------------------------------------------: |
| [STGCNPP](/projects/gesture_recognition/configs/stgcnpp_8xb16-joint-u100-16e_jester-keypoint-2d.py) |  100x17x3  |     89.22      |     97.52      | [ckpt](https://download.openmmlab.com/mmaction/v1.0/projects/gesture_recognition/stgcnpp_8xb16-joint-u100-16e_jester-keypoint-2d_20230524-fffa7ff0.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/projects/gesture_recognition/stgcnpp_8xb16-joint-u100-16e_jester-keypoint-2d.log) |

# CentripetalNet

> [CentripetalNet: Pursuing High-quality Keypoint Pairs for Object Detection](https://arxiv.org/abs/2003.09119)

Keypoint-based detectors have achieved pretty-well performance. However, incorrect keypoint matching is still widespread and greatly affects the performance of the detector. In this paper, we propose CentripetalNet which uses centripetal shift to pair corner keypoints from the same instance. CentripetalNet predicts the position and the centripetal shift of the corner points and matches corners whose shifted results are aligned. Combining position information, our approach matches corner points more accurately than the conventional embedding approaches do. Corner pooling extracts information inside the bounding boxes onto the border. To make this information more aware at the corners, we design a cross-star deformable convolution network to conduct feature adaption. Furthermore, we explore instance segmentation on anchor-free detectors by equipping our CentripetalNet with a mask prediction module. On MS-COCO test-dev, our CentripetalNet not only outperforms all existing anchor-free detectors with an AP of 48.0% but also achieves comparable performance to the state-of-the-art instance segmentation approaches with a 40.2% MaskAP.

<div align=center>
<img src="https://user-images.githubusercontent.com/40661020/143873955-42804e0e-3638-4c5b-8bf4-ac8133bbcdc8.png"/>
</div>

## Results and Models

|     Backbone     |                                Batch Size                                 | Step/Total Epochs | Mem (GB) | Inf time (fps) | box AP |                                  Config                                   |                                                                                                                                                                                                    Download                                                                                                                                                                                                    |

| HourglassNet-104 | [16 x 6](./centripetalnet_hourglass104_16xb6-crop511-210e-mstest_coco.py) |      190/210      |   16.7   |      3.7       |  44.8  | [config](./centripetalnet_hourglass104_16xb6-crop511-210e-mstest_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/centripetalnet/centripetalnet_hourglass104_mstest_16x6_210e_coco/centripetalnet_hourglass104_mstest_16x6_210e_coco_20200915_204804-3ccc61e5.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/centripetalnet/centripetalnet_hourglass104_mstest_16x6_210e_coco/centripetalnet_hourglass104_mstest_16x6_210e_coco_20200915_204804.log.json) |

Note:

- TTA setting is single-scale and `flip=True`. If you want to reproduce the TTA performance, please add `--tta` in the test command.
- The model we released is the best checkpoint rather than the latest checkpoint (box AP 44.8 vs 44.6 in our experiment).

```latex
@InProceedings{Dong_2020_CVPR,

```latex
author = {Dong, Zhiwei and Li, Guoxuan and Liao, Yue and Wang, Fei and Ren, Pengju and Qian, Chen},
title = {CentripetalNet: Pursuing High-Quality Keypoint Pairs for Object Detection},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}
```

# Structure-Coherent Deep Feature Learning for Robust Face Alignment

<details>
<summary align="right"><a href="https://ieeexplore.ieee.org/document/9442331/">SoftWingloss (TIP'2021)</a></summary>

```bibtex
@article{lin2021structure,
  title={Structure-Coherent Deep Feature Learning for Robust Face Alignment},
  author={Lin, Chunze and Zhu, Beier and Wang, Quan and Liao, Renjie and Qian, Chen and Lu, Jiwen and Zhou, Jie},
  journal={IEEE Transactions on Image Processing},
  year={2021},
  publisher={IEEE}
}
```

In this paper, we propose a structure-coherent deep feature learning method for face alignment. Unlike most existing face alignment methods which overlook the facial structure cues, we explicitly exploit the relation among facial landmarks to make the detector robust to hard cases such as occlusion and large pose. Specifically, we leverage a landmark-graph relational network to enforce the structural relationships among landmarks. We consider the facial landmarks as structural graph nodes and carefully design the neighborhood to passing features among the most related nodes. Our method dynamically adapts the weights of node neighborhood to eliminate distracted information from noisy nodes, such as occluded landmark point. Moreover, different from most previous works which only tend to penalize the landmarks absolute position during the training, we propose a relative location loss to enhance the information of relative location of landmarks. This relative location supervision further regularizes the facial structure. Our approach considers the interactions among facial landmarks and can be easily implemented on top of any convolutional backbone to boost the performance. Extensive experiments on three popular benchmarks, including WFLW, COFW and 300W, demonstrate the effectiveness of the proposed method. In particular, due to explicit structure modeling, our approach is especially robust to challenging cases resulting in impressive low failure rate on COFW and WFLW datasets.

<img src="https://user-images.githubusercontent.com/15977946/148036579-b128fb21-3a2d-4894-bb54-4223a4cdf856.png">
</div>

# NuScenes Dataset

This page provides specific tutorials about the usage of MMDetection3D for nuScenes dataset.

## Before Preparation

You can download nuScenes 3D detection `Full dataset (v1.0)` [HERE](https://www.nuscenes.org/download) and unzip all zip files.

If you want to implement 3D semantic segmentation task, you need to additionally download the `nuScenes-lidarseg` data annotation and place the extracted files in the nuScenes corresponding folder.

**Note**: `v1.0trainval(test)/categroy.json` in nuScenes-lidarseg will replace the original `v1.0trainval(test)/categroy.json` of the Full dataset (v1.0), but will not affect the 3D object detection task.

Like the general way to prepare dataset, it is recommended to symlink the dataset root to `$MMDETECTION3D/data`.

The folder structure should be organized as follows before our processing.

```
mmdetection3d
├── mmdet3d
├── tools
├── data
│   ├── nuscenes
│   │   ├── maps
│   │   ├── samples
│   │   ├── sweeps

```
├── tools
├── data
│   │   ├── maps
│   │   ├── lidarseg (optional)
│   │   ├── v1.0-test
|   |   ├── v1.0-trainval
```

## Dataset Preparation

We typically need to organize the useful data information with a `.pkl` file in a specific style.
To prepare these files for nuScenes, run the following command:

```bash
python tools/create_data.py nuscenes --root-path ./data/nuscenes --out-dir ./data/nuscenes --extra-tag nuscenes
```

The folder structure after processing should be as below.

```
├── tools
├── data
│   │   ├── maps
│   │   ├── nuscenes_database
│   │   ├── nuscenes_infos_train.pkl
│   │   ├── nuscenes_infos_val.pkl
│   │   ├── nuscenes_infos_test.pkl
│   │   ├── nuscenes_dbinfos_train.pkl
```

```

- `nuscenes_database/xxxxx.bin`: point cloud data included in each 3D bounding box of the training dataset
- `nuscenes_infos_train.pkl`: training dataset, a dict contains two keys: `metainfo` and `data_list`.
  `metainfo` contains the basic information for the dataset itself, such as `categories`, `dataset` and `info_version`, while `data_list` is a list of dict, each dict (hereinafter referred to as `info`) contains all the detailed information of single sample as follows:
  - info\['sample_idx'\]: The index of this sample in the whole dataset.
  - info\['token'\]: Sample data token.
  - info\['timestamp'\]: Timestamp of the sample data.
  - info\['ego2global'\]: The transformation matrix from the ego vehicle to global coordinates. (4x4 list)

- info\['timestamp'\]: Timestamp of the sample data.
  - info\['lidar_points'\]: A dict containing all the information related to the lidar points.
    - info\['lidar_points'\]\['lidar_path'\]: The filename of the lidar point cloud data.
    - info\['lidar_points'\]\['num_pts_feats'\]: The feature dimension of point.
    - info\['lidar_points'\]\['lidar2ego'\]: The transformation matrix from this lidar sensor to ego vehicle. (4x4 list)
  - info\['lidar_sweeps'\]: A list contains sweeps information (The intermediate lidar frames without annotations)
    - info\['lidar_sweeps'\]\[i\]\['lidar_points'\]\['data_path'\]: The lidar data path of i-th sweep.
    - info\['lidar_sweeps'\]\[i\]\['lidar_points'\]\['lidar2ego'\]: The transformation matrix from this lidar sensor to ego vehicle. (4x4 list)

- info\['lidar_sweeps'\]\[i\]\['lidar_points'\]\['lidar2ego'\]: The transformation matrix from this lidar sensor to ego vehicle. (4x4 list)
    - info\['lidar_sweeps'\]\[i\]\['lidar_points'\]\['ego2global'\]: The transformation matrix from the ego vehicle to global coordinates. (4x4 list)
    - info\['lidar_sweeps'\]\[i\]\['lidar2sensor'\]: The transformation matrix from the main lidar sensor to the current sensor (for collecting the sweep data). (4x4 list)
    - info\['lidar_sweeps'\]\[i\]\['timestamp'\]: Timestamp of the sweep data.
    - info\['lidar_sweeps'\]\[i\]\['sample_data_token'\]: The sweep sample data token.
  - info\['images'\]: A dict contains six keys corresponding to each camera: `'CAM_FRONT'`, `'CAM_FRONT_RIGHT'`, `'CAM_FRONT_LEFT'`, `'CAM_BACK'`, `'CAM_BACK_LEFT'`, `'CAM_BACK_RIGHT'`. Each dict contains all data information related to  corresponding camera.
    - info\['images'\]\['CAM_XXX'\]\['img_path'\]: The filename of the image.

- info\['images'\]\['CAM_XXX'\]\['img_path'\]: The filename of the image.
    - info\['images'\]\['CAM_XXX'\]\['cam2img'\]: The transformation matrix recording the intrinsic parameters when projecting 3D points to each image plane. (3x3 list)
    - info\['images'\]\['CAM_XXX'\]\['sample_data_token'\]: Sample data token of image.
    - info\['images'\]\['CAM_XXX'\]\['timestamp'\]: Timestamp of the image.
    - info\['images'\]\['CAM_XXX'\]\['cam2ego'\]: The transformation matrix from this camera sensor to ego vehicle. (4x4 list)
    - info\['images'\]\['CAM_XXX'\]\['lidar2cam'\]: The transformation matrix from lidar sensor to this camera. (4x4 list)
  - info\['instances'\]: It is a list of dict. Each dict contains all annotation information of single instance. For the i-th instance:
    - info\['instances'\]\[i\]\['bbox_3d'\]: List of 7 numbers representing the 3D bounding box of the instance, in (x, y, z, l, w, h, yaw) order.

- info\['instances'\]\[i\]\['bbox_3d'\]: List of 7 numbers representing the 3D bounding box of the instance, in (x, y, z, l, w, h, yaw) order.
    - info\['instances'\]\[i\]\['bbox_label_3d'\]: A int indicate the label of instance and the -1 indicate ignore.
    - info\['instances'\]\[i\]\['velocity'\]: Velocities of 3D bounding boxes (no vertical measurements due to inaccuracy), a list has shape (2.).
    - info\['instances'\]\[i\]\['num_lidar_pts'\]: Number of lidar points included in each 3D bounding box.
    - info\['instances'\]\[i\]\['num_radar_pts'\]: Number of radar points included in each 3D bounding box.
    - info\['instances'\]\[i\]\['bbox_3d_isvalid'\]: Whether each bounding box is valid. In general, we only take the 3D boxes that include at least one lidar or radar point as valid boxes.

- info\['instances'\]\[i\]\['bbox_3d_isvalid'\]: Whether each bounding box is valid. In general, we only take the 3D boxes that include at least one lidar or radar point as valid boxes.
  - info\['cam_instances'\]: It is a dict containing keys `'CAM_FRONT'`, `'CAM_FRONT_RIGHT'`, `'CAM_FRONT_LEFT'`, `'CAM_BACK'`, `'CAM_BACK_LEFT'`, `'CAM_BACK_RIGHT'`. For vision-based 3D object detection task, we split 3D annotations of the whole scenes according to the camera they belong to. For the i-th instance:
    - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['bbox_label'\]: Label of instance.
    - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['bbox_label_3d'\]: Label of instance.
    - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['bbox'\]: 2D bounding box annotation (exterior rectangle of the projected 3D box), a list arrange as \[x1, y1, x2, y2\].
    - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['center_2d'\]: Projected center location on the image, a list has shape (2,), .

- info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['center_2d'\]: Projected center location on the image, a list has shape (2,), .
    - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['depth'\]: The depth of projected center.
    - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['velocity'\]: Velocities of 3D bounding boxes (no vertical measurements due to inaccuracy), a list has shape (2,).
    - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['attr_label'\]: The attr label of instance. We maintain a default attribute collection and mapping for attribute classification.
    - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['bbox_3d'\]: List of 7 numbers representing the 3D bounding box of the instance, in (x, y, z, l, h, w, yaw) order.
  - info\['pts_semantic_mask_path'\]：The filename of the lidar point cloud semantic segmentation annotation.

Note:

1. The differences between `bbox_3d` in `instances` and that in `cam_instances`.

- info\['pts_semantic_mask_path'\]：The filename of the lidar point cloud semantic segmentation annotation.

Note:

   Both `bbox_3d` have been converted to MMDet3D coordinate system, but `bboxes_3d` in `instances` is in LiDAR coordinate format and `bboxes_3d` in `cam_instances` is in Camera coordinate format. Mind the difference between them in 3D Box representation ('l, w, h' and 'l, h, w').

2. Here we only explain the data recorded in the training info files. The same applies to validation and testing set (the `.pkl` file of test set does not contains `instances` and `cam_instances`).

The core function to get `nuscenes_infos_xxx.pkl` is  [\_fill_trainval_infos](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/nuscenes_converter.py#L146).

Please refer to [nuscenes_converter.py](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/nuscenes_converter.py) for more details.

## Training pipeline

### LiDAR-Based Methods

A typical training pipeline of LiDAR-based 3D detection (including multi-modality methods) on nuScenes is as below.

```python
    dict(
        coord_type='LIDAR',
        use_dim=5),
    dict(
        sweeps_num=10),
    dict(

dict(
    dict(
        type='Pack3DDetInputs',
        keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
]
```

Compared to general cases, nuScenes has a specific `'LoadPointsFromMultiSweeps'` pipeline to load point clouds from consecutive frames. This is a common practice used in this setting.
Please refer to the nuScenes [original paper](https://arxiv.org/abs/1903.11027) for more details.

The default `use_dim` in `'LoadPointsFromMultiSweeps'` is `[0, 1, 2, 4]`, where the first 3 dimensions refer to point coordinates and the last refers to timestamp differences.
Intensity is not used by default due to its yielded noise when concatenating the points from different frames.

### Vision-Based Methods

A typical training pipeline of image-based 3D detection on nuScenes is as below.

```python
    dict(type='LoadImageFromFileMono3D'),
    dict(
        type='LoadAnnotations3D',
        with_bbox=True,
        with_label=True,
        with_attr_label=True,
        with_bbox_3d=True,
        with_label_3d=True,
        with_bbox_depth=True),
    dict(type='mmdet.Resize', scale=(1600, 900), keep_ratio=True),
    dict(
        keys=[

dict(type='mmdet.Resize', scale=(1600, 900), keep_ratio=True),
    dict(
        keys=[
            'img', 'gt_bboxes', 'gt_bboxes_labels', 'attr_labels', 'gt_bboxes_3d',
            'gt_labels_3d', 'centers_2d', 'depths'
        ]),
]
```

It follows the general pipeline of 2D detection while differs in some details:

- It uses monocular pipelines to load images, which includes additional required information like camera intrinsics.
- It needs to load 3D annotations.
- Some data augmentation techniques need to be adjusted, such as `RandomFlip3D`.
  Currently we do not support more augmentation methods, because how to transfer and apply other techniques is still under explored.

## Evaluation

An example to evaluate PointPillars with 8 GPUs with nuScenes metrics is as follows.

```shell

```shell
bash ./tools/dist_test.sh configs/pointpillars/pointpillars_hv_fpn_sbn-all_8xb4-2x_nus-3d.py checkpoints/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d_20200620_230405-2fa62f3d.pth 8
```

## Metrics

NuScenes proposes a comprehensive metric, namely nuScenes detection score (NDS), to evaluate different methods and set up the benchmark.
It consists of mean Average Precision (mAP), Average Translation Error (ATE), Average Scale Error (ASE), Average Orientation Error (AOE), Average Velocity Error (AVE) and Average Attribute Error (AAE).
Please refer to its [official website](https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Any) for more details.

We also adopt this approach for evaluation on nuScenes. An example of printed evaluation results is as follows:

```
mAP: 0.3197
mATE: 0.7595
mASE: 0.2700
mAOE: 0.4918
mAVE: 1.3307
mAAE: 0.1724
NDS: 0.3905

```
Eval time: 170.8s

Per-class results:
Object Class    AP      ATE     ASE     AOE     AVE     AAE
car     0.503   0.577   0.152   0.111   2.096   0.136
truck   0.223   0.857   0.224   0.220   1.389   0.179
bus     0.294   0.855   0.204   0.190   2.689   0.283
trailer 0.081   1.094   0.243   0.553   0.742   0.167
construction_vehicle    0.058   1.017   0.450   1.019   0.137   0.341
pedestrian      0.392   0.687   0.284   0.694   0.876   0.158
motorcycle      0.317   0.737   0.265   0.580   2.033   0.104
bicycle 0.308   0.704   0.299   0.892   0.683   0.010
traffic_cone    0.555   0.486   0.309   nan     nan     nan
barrier 0.466   0.581   0.269   0.169   nan     nan
```

## Testing and make a submission

An example to test PointPillars on nuScenes with 8 GPUs and generate a submission to the leaderboard is as follows.

```

You should modify the `jsonfile_prefix` in the `test_evaluator` of corresponding configuration. For example, adding `test_evaluator = dict(type='NuScenesMetric', jsonfile_prefix='work_dirs/pp-nus/results_eval.json')` or using `--cfg-options "test_evaluator.jsonfile_prefix=work_dirs/pp-nus/results_eval.json)` after the test command.

```shell
./tools/dist_test.sh configs/pointpillars/pointpillars_hv_fpn_sbn-all_8xb4-2x_nus-3d.py work_dirs/pp-nus/latest.pth 8 --cfg-options 'test_evaluator.jsonfile_prefix=work_dirs/pp-nus/results_eval'
```

Note that the testing info should be changed to that for testing set instead of validation set [here](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/datasets/nus-3d.py#L132).

After generating the `work_dirs/pp-nus/results_eval.json`, you can compress it and submit it to nuScenes benchmark. Please refer to the [nuScenes official website](https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Any) for more information.

We can also visualize the prediction results with our developed visualization tools. Please refer to the [visualization doc](https://mmdetection3d.readthedocs.io/en/latest/useful_tools.html#visualization) for more details.

## Notes

### Transformation between `NuScenesBox` and our `CameraInstanceBoxes`.

## Notes

In general, the main difference of `NuScenesBox` and our `CameraInstanceBoxes` is mainly reflected in the yaw definition. `NuScenesBox` defines the rotation with a quaternion or three Euler angles while ours only defines one yaw angle due to the practical scenario. It requires us to add some additional rotations manually in the pre-processing and post-processing, such as [here](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/datasets/nuscenes_mono_dataset.py#L673).

In addition, please note that the definition of corners and locations are detached in the `NuScenesBox`. For example, in monocular 3D detection, the definition of the box location is in its camera coordinate (see its official [illustration](https://www.nuscenes.org/nuscenes#data-collection) for car setup), which is consistent with [ours](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/core/bbox/structures/cam_box3d.py). In contrast, its corners are defined with the [convention](https://github.com/nutonomy/nuscenes-devkit/blob/02e9200218977193a1058dd7234f935834378319/python-sdk/nuscenes/utils/data_classes.py#L527) "x points forward, y to the left, z up". It results in different philosophy of dimension and rotation definitions from our `CameraInstanceBoxes`. An example to remove similar hacks is PR [#744](https://github.com/open-mmlab/mmdetection3d/pull/744). The same problem also exists in the LiDAR system. To deal with them, we typically add some transformation in the pre-processing and post-processing to guarantee the box will be in our coordinate system during the entire training and inference procedure.

# Improving Convolutional Networks with Self-Calibrated Convolutions

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_CVPR_2020/html/Liu_Improving_Convolutional_Networks_With_Self-Calibrated_Convolutions_CVPR_2020_paper.html">SCNet (CVPR'2020)</a></summary>

```bibtex
@inproceedings{liu2020improving,
  title={Improving Convolutional Networks with Self-Calibrated Convolutions},
  author={Liu, Jiang-Jiang and Hou, Qibin and Cheng, Ming-Ming and Wang, Changhu and Feng, Jiashi},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={10096--10105},
}
```

Recent advances on CNNs are mostly devoted to designing more complex architectures to enhance their representation learning capacity. In this paper, we consider how to improve the basic convolutional feature transformation process of CNNs without tuning the model architectures. To this end, we present a novel self-calibrated convolutions that explicitly expand fields-of-view of each convolutional layers through internal communications and hence enrich the output features. In particular, unlike the standard convolutions that fuse spatial and channel-wise information using small kernels (e.g., 3x3), self-calibrated convolutions adaptively build long-range spatial and inter-channel dependencies around each spatial location through a novel self-calibration operation. Thus, it can help CNNs generate more discriminative representations by explicitly incorporating richer information. Our self-calibrated convolution design is simple and generic, and can be easily applied to augment standard convolutional layers without introducing extra parameters and complexity. Extensive experiments demonstrate that when applying self-calibrated convolutions into different backbones, our networks can significantly improve the baseline models in a variety of vision tasks, including image recognition, object detection, instance segmentation, and keypoint detection, with no need to change the network architectures. We hope this work could provide a promising way for future research in designing novel convolutional feature transformations for improving convolutional networks. Code is available on the project page.

<img src="https://user-images.githubusercontent.com/15977946/146522623-fdc88996-1a8f-4b0a-bdee-fc284dfd7348.png">
</div>

1. 在任意文件里写一个新的流程，例如在 `my_pipeline.py`，它以一个字典作为输入并且输出一个字典：

   ```python
   import random
   from mmcv.transforms import BaseTransform
   from mmdet.registry import TRANSFORMS

   @TRANSFORMS.register_module()
   class MyTransform(BaseTransform):
       """Add your transform

       Args:
           p (float): Probability of shifts. Default 0.5.
       """

       def __init__(self, prob=0.5):
           self.prob = prob

       def transform(self, results):
           if random.random() > self.prob:
               results['dummy'] = True
   ```

2. 在配置文件里调用并使用你写的数据处理流程，需要确保你的训练脚本能够正确导入新增模块：

   ```python
   custom_imports = dict(imports=['path.to.my_pipeline'], allow_failed_imports=False)

       dict(type='LoadImageFromFile'),
       dict(type='LoadAnnotations', with_bbox=True),
       dict(type='Resize', scale=(1333, 800), keep_ratio=True),
       dict(type='RandomFlip', prob=0.5),

       dict(type='MyTransform', prob=0.2),
       dict(type='PackDetInputs')
   ]
   ```

3. 可视化数据增强处理流程的结果

   如果想要可视化数据增强处理流程的结果，可以使用 `tools/misc/browse_dataset.py` 直观
   地浏览检测数据集（图像和标注信息），或将图像保存到指定目录。
   使用方法请参考[可视化文档](../user_guides/visualization.md)

# HSSN

Author: AI-Tianlong

This project implements `Deep Hierarchical Semantic Segmentation`  inference on `cityscapes` dataset

## Usage

- Python 3.8
- PyTorch 1.6 or higher
- mmcv v2.0.0rc4
- mmengine >=0.4.0

### Dataset preparing

Preparing `cityscapes` dataset following this [Dataset Preparing Guide](https://github.com/open-mmlab/mmsegmentation/blob/master/docs/en/dataset_prepare.md#prepare-datasets)

Please put [`hieraseg_deeplabv3plus_r101-d8_4xb2-80k_cityscapes-512x1024_20230112_125023-bc59a3d1.pth`](https://download.openmmlab.com/mmsegmentation/v0.5/hieraseg/hieraseg_deeplabv3plus_r101-d8_4xb2-80k_cityscapes-512x1024_20230112_125023-bc59a3d1.pth) to `mmsegmentation/checkpoints`

#### Multi-GPUs Test

```bash
# --tta optional, multi-scale test, need mmengine >=0.4.0
bash tools/dist_test.sh [configs] [model weights] [number of gpu]  --tta
```

```bash
```

#### Example

```shell
bash tools/dist_test.sh projects/hssn/configs/hssn/hieraseg_deeplabv3plus_r101-d8_4xb2-80l_cityscapes-512x1024.py checkpoints/hieraseg_deeplabv3plus_r101-d8_4xb2-80k_cityscapes-512x1024_20230112_125023-bc59a3d1.pth 2 --tta
```

## Results

### Cityscapes

|   Method   | Backbone | Crop Size | mIoU  | mIoU (ms+flip) |                                                                               config                                                                               |                                                                             model                                                                             |

| :--------: | :------: | :-------: | :---: | :------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------: |
| DeeplabV3+ | R-101-D8 | 512x1024  | 81.61 |     82.71      | [config](https://github.com/open-mmlab/mmsegmentation/tree/main/projects/HieraSeg/configs/hieraseg/hieraseg_deeplabv3plus_r101-d8_4xb2-80l_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hieraseg/hieraseg_deeplabv3plus_r101-d8_4xb2-80k_cityscapes-512x1024_20230112_125023-bc59a3d1.pth) |

<img src="https://user-images.githubusercontent.com/50650583/210488953-e3e35ade-1132-47e1-9dfd-cf12b357ae80.png" width="50%"><img src="https://user-images.githubusercontent.com/50650583/210489746-e35ee229-3234-4292-a649-a8cd85f312ad.png" width="50%">

This project is modified from [qhanghu/HSSN_pytorch](https://github.com/qhanghu/HSSN_pytorch)

```bibtex
@article{li2022deep,
  title={Deep Hierarchical Semantic Segmentation},
  author={Li, Liulei and Zhou, Tianfei and Wang, Wenguan and Li, Jianwu and Yang, Yi},
  journal={CVPR},
}
```

- [ ] Milestone 2: Indicates a successful model implementation.

  - [ ] Training-time correctness

- [ ] Training-time correctness

# C2D

[Non-local Neural Networks](https://arxiv.org/abs/1711.07971)

Both convolutional and recurrent operations are building blocks that process one local neighborhood at a time. In this paper, we present non-local operations as a generic family of building blocks for capturing long-range dependencies. Inspired by the classical non-local means method in computer vision, our non-local operation computes the response at a position as a weighted sum of the features at all positions. This building block can be plugged into many computer vision architectures. On the task of video classification, even without any bells and whistles, our non-local models can compete or outperform current competition winners on both Kinetics and Charades datasets. In static image recognition, our non-local models improve object detection/segmentation and pose estimation on the COCO suite of tasks.

<img src="https://user-images.githubusercontent.com/37479394/195281946-b70c76fd-d424-4371-95cf-09f51f20eda0.jpg" width="800"/>

</div>

### Kinetics-400

| frame sampling strategy | scheduler | resolution | gpus |    backbone     | pretrain | top1 acc | top5 acc |    reference top1 acc     |    reference top5 acc     | testing protocol  | FLOPs | params |     config     |     ckpt     |     log     |
| :---------------------: | :-------: | :--------: | :--: | :-------------: | :------: | :------: | :------: | :-----------------------: | :-----------------------: | :---------------: | :---: | :----: | :------------: | :----------: | :---------: |

|          8x8x1          | MultiStep |  224x224   |  8   |  ResNet50<br>   | ImageNet |  73.44   |  91.00   | 67.2<br>[\[PySlowFast\]](https://github.com/facebookresearch/SlowFast/blob/main/MODEL_ZOO.md#kinetics-400-and-600) | 87.8<br>[\[PySlowFast\]](https://github.com/facebookresearch/SlowFast/blob/main/MODEL_ZOO.md#kinetics-400-and-600) | 10 clips x 3 crop |  33G  | 24.3M  | [config](/configs/recognition/c2d/c2d_r50-in1k-pre-nopool_8xb32-8x8x1-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/c2d/c2d_r50-in1k-pre-nopool_8xb32-8x8x1-100e_kinetics400-rgb/c2d_r50-in1k-pre-nopool_8xb32-8x8x1-100e_kinetics400-rgb_20221027-e0227b22.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/c2d/c2d_r50-in1k-pre-nopool_8xb32-8x8x1-100e_kinetics400-rgb/c2d_r50-in1k-pre-nopool_8xb32-8x8x1-100e_kinetics400-rgb.log) |

|          8x8x1          | MultiStep |  224x224   |  8   |  ResNet101<br>  | ImageNet |  74.97   |  91.77   |             x             |             x             | 10 clips x 3 crop |  63G  | 43.3M  | [config](/configs/recognition/c2d/c2d_r101-in1k-pre-nopool_8xb32-8x8x1-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/c2d/c2d_r101-in1k-pre-nopool_8xb32-8x8x1-100e_kinetics400-rgb/c2d_r101-in1k-pre-nopool_8xb32-8x8x1-100e_kinetics400-rgb_20221027-557bd8bc.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/c2d/c2d_r101-in1k-pre-nopool_8xb32-8x8x1-100e_kinetics400-rgb/c2d_r101-in1k-pre-nopool_8xb32-8x8x1-100e_kinetics400-rgb.log) |

|          8x8x1          | MultiStep |  224x224   |  8   | ResNet50<br>(TemporalPool) | ImageNet |  73.89   |  91.21   | 71.9<br>[\[Non-Local\]](https://github.com/facebookresearch/video-nonlocal-net#modifications-for-improving-speed) | 90.0<br>[\[Non-Local\]](https://github.com/facebookresearch/video-nonlocal-net#modifications-for-improving-speed) | 10 clips x 3 crop |  19G  | 24.3M  | [config](/configs/recognition/c2d/c2d_r50-in1k-pre_8xb32-8x8x1-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/c2d/c2d_r50-in1k-pre_8xb32-8x8x1-100e_kinetics400-rgb/c2d_r50-in1k-pre_8xb32-8x8x1-100e_kinetics400-rgb_20221027-3ca304fa.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/c2d/c2d_r50-in1k-pre_8xb32-8x8x1-100e_kinetics400-rgb/c2d_r50-in1k-pre_8xb32-8x8x1-100e_kinetics400-rgb.log) |

|         16x4x1          | MultiStep |  224x224   |  8   | ResNet50<br>(TemporalPool) | ImageNet |  74.97   |  91.91   |             x             |             x             | 10 clips x 3 crop |  39G  | 24.3M  | [config](/configs/recognition/c2d/c2d_r50-in1k-pre_8xb32-16x4x1-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/c2d/c2d_r50-in1k-pre_8xb32-16x4x1-100e_kinetics400-rgb/c2d_r50-in1k-pre_8xb32-16x4x1-100e_kinetics400-rgb_20221027-5f382a43.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/c2d/c2d_r50-in1k-pre_8xb32-16x4x1-100e_kinetics400-rgb/c2d_r50-in1k-pre_8xb32-16x4x1-100e_kinetics400-rgb.log) |

1. The values in columns named after "reference" are the results reported in the original repo, using the same model settings.

2. The **gpus** indicates the number of gpus we used to get the checkpoint. If you want to use a different number of gpus or videos per gpu, the best way is to set `--auto-scale-lr` when calling `tools/train.py`, this parameter will auto-scale the learning rate according to the actual batch size and the original batch size.

3. The validation set of Kinetics400 we used consists of 19796 videos. These videos are available at [Kinetics400-Validation](https://mycuhk-my.sharepoint.com/:u:/g/personal/1155136485_link_cuhk_edu_hk/EbXw2WX94J1Hunyt3MWNDJUBz-nHvQYhO9pvKqm6g39PMA?e=a9QldB). The corresponding [data list](https://download.openmmlab.com/mmaction/dataset/k400_val/kinetics_val_list.txt) (each line is of the format 'video_id, num_frames, label_index') and the [label map](https://download.openmmlab.com/mmaction/dataset/k400_val/kinetics_class2ind.txt) are also available.

For more details on data preparation, you can refer to [Kinetics400](/tools/data/kinetics/README.md).

## Train

You can use the following command to train a model.

```shell
python tools/train.py ${CONFIG_FILE} [optional arguments]
```

Example: train C2D model on Kinetics-400 dataset in a deterministic option with periodic validation.

```shell

```shell
```

```shell
python tools/train.py configs/recognition/c2d/c2d_r50-in1k-pre_8xb32-8x8x1-100e_kinetics400-rgb.py  \
    --seed 0 --deterministic
```

For more details, you can refer to the **Training** part in the [Training and Test Tutorial](/docs/en/user_guides/train_test.md).

## Test

You can use the following command to test a model.

```shell
python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [optional arguments]
```

Example: test C2D model on Kinetics-400 dataset and dump the result to a pkl file.

```shell
python tools/test.py configs/recognition/c2d/c2d_r50-in1k-pre_8xb32-8x8x1-100e_kinetics400-rgb.py \
    checkpoints/SOME_CHECKPOINT.pth --dump result.pkl
```

For more details, you can refer to the **Test** part in the [Training and Test Tutorial](/docs/en/user_guides/train_test.md).

checkpoints/SOME_CHECKPOINT.pth --dump result.pkl
```

```BibTeX
@article{XiaolongWang2017NonlocalNN,
  title={Non-local Neural Networks},
  author={Xiaolong Wang and Ross Girshick and Abhinav Gupta and Kaiming He},
  journal={arXiv: Computer Vision and Pattern Recognition},
  year={2017}
}
```

# Fuse Transform（Experimental）

MMDeploy provides ability to fuse transform for acceleration in some cases.

When make inference with SDK, one can edit the pipeline.json to turn on the fuse option.

To bring the ability of fuse transform to MMDeploy, you can refer to the use of CVFusion.

## 1. Use CVFusion

There are two ways to use CVFusion, one is to use the pre-generated kernel code, the other is to generate the code yourself.

A）Use pre-generated kernel code

i) Download the kernel code from here，unzip it and copy the csrc folder to the mmdeploy root folder.

[elena_kernel-20220823.tar.gz](https://github.com/open-mmlab/mmdeploy/files/9399795/elena_kernel-20220823.tar.gz)

ii) Add option `-DMMDEPLOY_ELENA_FUSION=ON` when compile MMDeploy.

B) Generate kernel code by yourself

i) Compile CVFusion

```bash
$ git clone --recursive https://github.com/OpenComputeLab/CVFusion.git
$ cd CVFusion
$ bash build.sh
```

```
# add OpFuse to PATH
$ export PATH=`pwd`/build/examples/MMDeploy:$PATH

```bash
```

```
```

ii) Download algorithm codebase

```bash
$ tree -L 1 .
├── mmdeploy
├── mmpretrain
├── mmdetection
├── mmsegmentation
├── ...
```

iii) Generate kernel code

```bash
python tools/elena/extract_transform.py ..
# The generated code will be saved to csrc/preprocess/elena/{cpu_kernel}/{cuda_kernel}
```

iv) Add option `-DMMDEPLOY_ELENA_FUSION=ON` when compile MMDeploy.

## 2. Model conversion

Add `--dump-info` argument when convert a model, this will generate files that SDK needs.

```bash
$ export MODEL_CONFIG=/path/to/mmpretrain/configs/resnet/resnet18_8xb32_in1k.py
$ export MODEL_PATH=https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_8xb32_in1k_20210831-fbbb1da6.pth

$ python tools/deploy.py \
    configs/mmpretrain/classification_onnxruntime_static.py \
    $MODEL_CONFIG \

    $MODEL_PATH \
    tests/data/tiger.jpeg \
    --work-dir resnet18 \
    --device cpu \
    --dump-info
```

## 3. Model Inference

If the model preprocess supports fusion, there will be a filed named `fuse_transform` in `pipeline.json`. It represents fusion switch and the default value `false` stands for off. One need to edit this filed to `true` to use the fuse option.

# VACC Backend

- cmake 3.10.0+
- gcc/g++ 7.5.0
- llvm 9.0.1
- ubuntu 18.04

## PCIE

### 1.package

- dkms (>=1.95)
- linux-headers
- dpkg (Ubuntu)
- rpm  (CentOS)
- python2
- python3

Check if there is a vacc card：`lspci -d:0100`

1. Requirements

   ```bash
   sudo apt-get install dkms dpkg python2 python3
   ```

2. install driver

   ```bash
   sudo dpkg -i vastai-pci_xx.xx.xx.xx_xx.deb
   ```

3. Verify installation

   ```bash
   dpkg --status vastai-pci-xxx

   #output
   Package: vastai-pci-dkms
   Status: install ok installed
   ……
   Version: xx.xx.xx.xx
   Provides: vastai-pci-modules (= xx.xx.xx.xx)
   Depends: dkms (>= 1.95)
   Description: vastai-pci driver in DKMS format.

   lsmod | grep vastai_pci

   #output
   vastai_pci        xxx  x
   ```

4. Upgrade driver

   ```bash
   sudo dpkg -i vastai-pci_dkms_xx.xx.xx.xx_xx.deb
   ```

5. Uninstall driver

   ```bash
   sudo dpkg -r vastai-pci_dkms_xx.xx.xx.xx_xx
   ```

### 2.reboot pcie

```bash

```bash
   ```

   ```bash
   ```

```bash
sudo chmod 666 /dev/kchar:0 && sudo echo reboot > /dev/kchar:0
```

## SDK

### step.1

```bash
pip install torch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0
pip install onnx==1.10.0 tqdm==4.64.1
pip install h5py==3.8.0
pip install decorator==5.1.1 scipy==1.7.3
```

### step.2

```bash
sudo vi ~/.bashrc

export VASTSTREAM_PIPELINE=true
export VACC_IRTEXT_ENABLE=1
export TVM_HOME="/opt/vastai/vaststream/tvm"
export VASTSTREAM_HOME="/opt/vastai/vaststream/vacl"
export LD_LIBRARY_PATH=$TVM_HOME/lib:$VASTSTREAM_HOME/lib
export PYTHONPATH=$TVM_HOME/python:$TVM_HOME/vacc/python:$TVM_HOME/topi/python:${PYTHONPATH}:$VASTSTREAM_HOME/python

source ~/.bashrc
```

# SRCNN (TPAMI'2015)

> **任务**: 图像超分辨率

<details>
<summary align="right">SRCNN (TPAMI'2015)</summary>

```bibtex
@article{dong2015image,
  title={Image super-resolution using deep convolutional networks},
  author={Dong, Chao and Loy, Chen Change and He, Kaiming and Tang, Xiaoou},
  journal={IEEE transactions on pattern analysis and machine intelligence},
  volume={38},
  number={2},
  pages={295--307},
  year={2015},
}
```

<br/>

在 RGB 通道上进行评估，在评估之前裁剪每个边界中的 `scale` 像素。
我们使用 `PSNR` 和 `SSIM` 作为指标。

|                                 算法                                 |       Set5       |       Set14       |      DIV2K       | GPU 信息 |                                    下载                                     |
| :------------------------------------------------------------------: | :--------------: | :---------------: | :--------------: | :------: | :-------------------------------------------------------------------------: |

| [srcnn_x4k915_1x16_1000k_div2k](./srcnn_x4k915_1xb16-1000k_div2k.py) | 28.4316 / 0.8099 | 25.6486 /  0.7014 | 27.7460 / 0.7854 |    1     | [模型](https://download.openmmlab.com/mmediting/restorers/srcnn/srcnn_x4k915_1x16_1000k_div2k_20200608-4186f232.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/srcnn/srcnn_x4k915_1x16_1000k_div2k_20200608_120159.log.json) |

## 快速开始

**训练**

<details>
<summary>训练说明</summary>

您可以使用以下命令来训练模型。

```shell
# CPU上训练
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/srcnn/srcnn_x4k915_1xb16-1000k_div2k.py

# 单个GPU上训练
python tools/train.py configs/srcnn/srcnn_x4k915_1xb16-1000k_div2k.py

# 多个GPU上训练
./tools/dist_train.sh configs/srcnn/srcnn_x4k915_1xb16-1000k_div2k.py 8
```

更多细节可以参考 [train_test.md](/docs/zh_cn/user_guides/train_test.md) 中的 **Train a model** 部分。

**测试**

<details>
<summary>测试说明</summary>

您可以使用以下命令来测试模型。

```shell
# CPU上测试

```

**测试**

<details>

```shell
# CPU上测试
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/srcnn/srcnn_x4k915_1xb16-1000k_div2k.py https://download.openmmlab.com/mmediting/restorers/srcnn/srcnn_x4k915_1x16_1000k_div2k_20200608-4186f232.pth

# 单个GPU上测试
python tools/test.py configs/srcnn/srcnn_x4k915_1xb16-1000k_div2k.py https://download.openmmlab.com/mmediting/restorers/srcnn/srcnn_x4k915_1x16_1000k_div2k_20200608-4186f232.pth

# 多个GPU上测试
./tools/dist_test.sh configs/srcnn/srcnn_x4k915_1xb16-1000k_div2k.py https://download.openmmlab.com/mmediting/restorers/srcnn/srcnn_x4k915_1x16_1000k_div2k_20200608-4186f232.pth 8
```

更多细节可以参考 [train_test.md](/docs/zh_cn/user_guides/train_test.md) 中的 **Test a pre-trained model** 部分。

# Training Engine

MMEngine defined some [basic loop controllers](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py) such as epoch-based training loop (`EpochBasedTrainLoop`), iteration-based training loop (`IterBasedTrainLoop`), standard validation loop (`ValLoop`), and standard testing loop (`TestLoop`).

OpenMMLab's algorithm libraries like MMSegmentation abstract model training, testing, and inference as `Runner` to handle. Users can use the default `Runner` in MMEngine directly or modify the `Runner` to meet customized needs. This document mainly introduces how users can configure existing running settings, hooks, and optimizers' basic concepts and usage methods.

## Configuring Runtime Settings

### Configuring Training Iterations

Loop controllers refer to the execution process during training, validation, and testing. `train_cfg`, `val_cfg`, and `test_cfg` are used to build these processes in the configuration file. MMSegmentation sets commonly used training iterations in `train_cfg` under the `configs/_base_/schedules` folder.
For example, to train for 80,000 iterations using the iteration-based training loop (`IterBasedTrainLoop`) and perform validation every 8,000 iterations, you can set it as follows:

```python
train_cfg = dict(type='IterBasedTrainLoop', max_iters=80000, val_interval=8000)
```

### Configuring Training Optimizers

Here's an example of a SGD optimizer:

```python
    type='OptimWrapper',
    optimizer=dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0005),
    clip_grad=None)
```

```python
```

OpenMMLab supports all optimizers in PyTorch. For more details, please refer to the [MMEngine optimizer documentation](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/optim_wrapper.md).

It is worth emphasizing that `optim_wrapper` is a variable of `runner`, so when configuring the optimizer, the field to configure is the `optim_wrapper` field. For more information on using optimizers, see the [Optimizer](#Optimizer) section below.

### Configuring Training Parameter Schedulers

Before configuring the training parameter scheduler, it is recommended to first understand the basic concepts of parameter schedulers in the [MMEngine documentation](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md).

Here's an example of a parameter scheduler. During training, a linearly changing learning rate strategy is used for warm-up in the first 1,000 iterations. After the first 1,000 iterations until the 16,000 iterations in the end, the default polynomial learning rate decay is used:

```python
    dict(type='LinearLR', by_epoch=False, start_factor=0.1, begin=0, end=1000),
    dict(
        type='PolyLR',
        eta_min=1e-4,
        power=0.9,
        begin=1000,
        end=160000,
        by_epoch=False,
    )
]
```

Note: When modifying the `max_iters` in `train_cfg`, make sure the parameters in the parameter scheduler `param_scheduler` are also modified accordingly.

## Hook

### Introduction

)
]
```

## Hook

OpenMMLab abstracts the model training and testing process as `Runner`. Inserting hooks can implement the corresponding functionality needed at different training and testing stages (such as "before and after each training iter", "before and after each validation iter", etc.) in `Runner`. For more introduction on hook mechanisms, please refer to [here](https://www.calltutors.com/blog/what-is-hook).

Hooks used in `Runner` are divided into two categories:

- Default hooks:

They implement essential functions during training and are defined in the configuration file by `default_hooks` and passed to `Runner`. `Runner` registers them through the [`register_default_hooks`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/runner.py#L1780) method.

Hooks have corresponding priorities; the higher the priority, the earlier the runner calls them. If the priorities are the same, the calling order is consistent with the hook registration order.

It is not recommended for users to modify the default hook priorities. Please refer to the [MMEngine hooks documentation](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/hook.md) to understand the hook priority definitions.

The following are the default hooks used in MMSegmentation:

|                                                          Hook                                                          |                                                          Function                                                          |     Priority      |

| :--------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------: | :---------------: |
|          [IterTimerHook](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/iter_timer_hook.py)           |                                          Record the time spent on each iteration.                                          |    NORMAL (50)    |
|              [LoggerHook](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/logger_hook.py)              | Collect log records from different components in `Runner` and output them to terminal, JSON file, tensorboard, wandb, etc. | BELOW_NORMAL (60) |

|     [ParamSchedulerHook](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/param_scheduler_hook.py)      |                       Update some hyperparameters in the optimizer, such as learning rate momentum.                        |     LOW (70)      |
|          [CheckpointHook](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py)          |                                              Regularly save checkpoint files.                                              |   VERY_LOW (90)   |
|      [DistSamplerSeedHook](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/sampler_seed_hook.py)       |                                     Ensure the distributed sampler shuffle is enabled.                                     |    NORMAL (50)    |

| [SegVisualizationHook](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/visualization/local_visualizer.py) |                                Visualize prediction results during validation and testing.                                 |    NORMAL (50)    |

MMSegmentation registers some hooks with essential training functions in `default_hooks`:

```python
    timer=dict(type='IterTimerHook'),
    logger=dict(type='LoggerHook', interval=50, log_metric_by_epoch=False),
    param_scheduler=dict(type='ParamSchedulerHook'),
    checkpoint=dict(type='CheckpointHook', by_epoch=False, interval=32000),
    sampler_seed=dict(type='DistSamplerSeedHook'),
    visualization=dict(type='SegVisualizationHook'))
```

All the default hooks mentioned above, except for `SegVisualizationHook`, are implemented in MMEngine. The `SegVisualizationHook` is a hook implemented in MMSegmentation, which will be introduced later.

- Modifying default hooks

We will use the `logger` and `checkpoint` in `default_hooks` as examples to demonstrate how to modify the default hooks in `default_hooks`.

(1) Model saving configuration

`default_hooks` uses the `checkpoint` field to initialize the [model saving hook (CheckpointHook)](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L19).

```python
checkpoint = dict(type='CheckpointHook', interval=1)
```

Users can set `max_keep_ckpts` to save only a small number of checkpoints or use `save_optimizer` to determine whether to save optimizer information. More details on related parameters can be found [here](https://mmengine.readthedocs.io/en/latest/api/generated/mmengine.hooks.CheckpointHook.html#checkpointhook).

(2) Logging configuration

The `LoggerHook` is used to collect log information from different components in `Runner` and write it to terminal, JSON files, tensorboard, wandb, etc.

```python

```python
logger=dict(type='LoggerHook', interval=10)
```

In the latest 1.x version of MMSegmentation, some logger hooks (LoggerHook) such as `TextLoggerHook`, `WandbLoggerHook`, and `TensorboardLoggerHook` will no longer be used. Instead, MMEngine uses `LogProcessor` to handle the information processed by the aforementioned hooks, which are now in [`MessageHub`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/logging/message_hub.py#L17), [`WandbVisBackend`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/visualization/vis_backend.py#L324), and [`TensorboardVisBackend`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/visualization/vis_backend.py#L472).

Detailed usage is as follows, configuring the visualizer and specifying the visualization backend at the same time, here using Tensorboard as the visualizer's backend:

```python
# TensorboardVisBackend
    type='SegLocalVisualizer', vis_backends=[dict(type='TensorboardVisBackend')], name='visualizer')
```

For more related usage, please refer to [MMEngine Visualization Backend User Tutorial](https://github.com/open-mmlab/mmengine/blob/main/docs/en/advanced_tutorials/visualization.md).

- Custom hooks

Custom hooks are defined in the configuration through `custom_hooks`, and `Runner` registers them using the [`register_custom_hooks`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/runner.py#L1820) method.

The priority of custom hooks needs to be set in the configuration file; if not, it will be set to `NORMAL` by default. The following are some custom hooks implemented in MMEngine:

|                                                  Hook                                                  |                                                               Usage                                                                |
| :----------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------: |
|         [EMAHook](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/ema_hook.py)         |                                    Use Exponential Moving Average (EMA) during model training.                                     |

| [EmptyCacheHook](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/empty_cache_hook.py)  |                                  Release all GPU memory not occupied by the cache during training                                  |
| [SyncBuffersHook](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/sync_buffer_hook.py) | Synchronize the parameters in the model buffer, such as `running_mean` and `running_var` in BN, at the end of each training epoch. |

The following is a use case for `EMAHook`, where the config file includes the configuration of the implemented custom hooks as members of the `custom_hooks` list.

```python
    dict(type='EMAHook', start_iters=500, priority='NORMAL')
]
```

### SegVisualizationHook

MMSegmentation implemented [`SegVisualizationHook`](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/engine/hooks/visualization_hook.py#L17), which is used to visualize prediction results during validation and testing.

`SegVisualizationHook` overrides the `_after_iter` method in the base class `Hook`. During validation or testing, it calls the `add_datasample` method of `visualizer` to draw semantic segmentation results according to the specified iteration interval. The specific implementation is as follows:

```python
...
class SegVisualizationHook(Hook):
...
    def _after_iter(self,
                    runner: Runner,
                    batch_idx: int,
                    data_batch: dict,
                    outputs: Sequence[SegDataSample],
                    mode: str = 'val') -> None:
...
        # If it's a training phase or self.draw is False, then skip it
        if self.draw is False or mode == 'train':
            return
...
        if self.every_n_inner_iters(batch_idx, self.interval):
            for output in outputs:
                img_path = output.img_path
                img_bytes = self.file_client.get(img_path)

if self.every_n_inner_iters(batch_idx, self.interval):
                img = mmcv.imfrombytes(img_bytes, channel_order='rgb')
                window_name = f'{mode}_{osp.basename(img_path)}'

                self._visualizer.add_datasample(
                    window_name,
                    img,
                    data_sample=output,
                    show=self.show,
                    wait_time=self.wait_time,
                    step=runner.iter)

```

For more details about visualization, you can check [here](../user_guides/visualization.md).

## Optimizer

In the previous configuration and runtime settings, we provided a simple example of configuring the training optimizer. This section will further detailly introduce  how to configure optimizers in MMSegmentation.

## Optimizer Wrapper

OpenMMLab 2.0 introduces an optimizer wrapper that supports different training strategies, including mixed-precision training, gradient accumulation, and gradient clipping. Users can choose the appropriate training strategy according to their needs. The optimizer wrapper also defines a standard parameter update process, allowing users to switch between different training strategies within the same code. For more information, please refer to the [MMEngine optimizer wrapper documentation](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/optim_wrapper.md).

Here are some common usage methods in MMSegmentation:

#### Configuring PyTorch Supported Optimizers

OpenMMLab 2.0 supports all native PyTorch optimizers, as referenced [here](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/optim_wrapper.md).

To set the optimizer used by the `Runner` during training in the configuration file, you need to define `optim_wrapper` instead of `optimizer`. Below is an example of configuring an optimizer during training:

```python
```

#### Configuring Gradient Clipping

When the model training requires gradient clipping, you can configure it as shown in the following example:

```python
optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001)
optim_wrapper = dict(type='OptimWrapper', optimizer=optimizer,
                        clip_grad=dict(max_norm=0.01, norm_type=2))
```

```

Here, `max_norm` refers to the maximum value of the gradient after clipping, and `norm_type` refers to the norm used when clipping the gradient. Related methods can be found in [torch.nn.utils.clip_grad_norm\_](https://pytorch.org/docs/stable/generated/torch.nn.utils.clip_grad_norm_.html).

#### Configuring Mixed Precision Training

When mixed precision training is needed to reduce memory usage, you can use `AmpOptimWrapper`. The specific configuration is as follows:

```python
optim_wrapper = dict(type='AmpOptimWrapper', optimizer=optimizer)
```

The default setting for `loss_scale` in [`AmpOptimWrapper`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/optimizer/amp_optimizer_wrapper.py#L20) is `dynamic`.

```

#### Configuring Hyperparameters for Different Layers of the Model Network

In model training, if you want to set different optimization strategies for different parameters in the optimizer, such as setting different learning rates, weight decay, and other hyperparameters, you can achieve this by setting `paramwise_cfg` in the `optim_wrapper` of the configuration file.

The following config file uses the [ViT `optim_wrapper`](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/vit/vit_vit-b16-ln_mln_upernet_8xb2-160k_ade20k-512x512.py#L15-L27) as an example to introduce the use of `paramwise_cfg` parameters. During training, the weight decay parameter coefficients for the `pos_embed`, `mask_token`, and `norm` modules are set to 0. That is, during training, the weight decay for these modules will be changed to `weight_decay * decay_mult`=0.

```python
        type='AdamW', lr=0.00006, betas=(0.9, 0.999), weight_decay=0.01)
    optimizer=optimizer,
    paramwise_cfg=dict(
        custom_keys={
            'pos_embed': dict(decay_mult=0.),
            'cls_token': dict(decay_mult=0.),
            'norm': dict(decay_mult=0.)
        }))
```

paramwise_cfg=dict(
        }))
```

Here, `decay_mult` refers to the weight decay coefficient for the corresponding parameters. For more information on the usage of `paramwise_cfg`, please refer to the [MMEngine optimizer wrapper documentation](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/optim_wrapper.md).

### Optimizer Wrapper Constructor

The default optimizer wrapper constructor [`DefaultOptimWrapperConstructor`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/optimizer/default_constructor.py#L19) builds the optimizer used in training based on the input `optim_wrapper` and `paramwise_cfg` defined in the `optim_wrapper`. When the functionality of [`DefaultOptimWrapperConstructor`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/optimizer/default_constructor.py#L19) does not meet the requirements, you can customize the optimizer wrapper constructor to implement the configuration of hyperparameters.

MMSegmentation has implemented the [`LearningRateDecayOptimizerConstructor`](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/engine/optimizers/layer_decay_optimizer_constructor.py#L104), which can decay the learning rate of model parameters in the backbone networks of ConvNeXt, BEiT, and MAE models during training according to the defined decay ratio (`decay_rate`). The configuration in the configuration file is as follows:

```python
    _delete_=True,
    type='AmpOptimWrapper',
    optimizer=dict(
        type='AdamW', lr=0.0001, betas=(0.9, 0.999), weight_decay=0.05),
    paramwise_cfg={
        'decay_rate': 0.9,
        'decay_type': 'stage_wise',
        'num_layers': 12
    },
    constructor='LearningRateDecayOptimizerConstructor',
    loss_scale='dynamic')
```

'decay_rate': 0.9,
    },
```

The purpose of `_delete_=True` is to ignore the inherited configuration in the OpenMMLab Config. In this code snippet, the inherited `optim_wrapper` configuration is ignored. For more information on `_delete_` fields, please refer to the [MMEngine documentation](https://github.com/open-mmlab/mmengine/blob/main/docs/en/advanced_tutorials/config.md#delete-key-in-dict).

# 如何转换模型

<!-- TOC -->

- [如何转换模型](#如何转换模型)
  - [如何将模型从pytorch形式转换成其他后端形式](#如何将模型从pytorch形式转换成其他后端形式)
    - [准备工作](#准备工作)
    - [使用方法](#使用方法)
    - [参数描述](#参数描述)
    - [如何查找pytorch模型对应的部署配置文件](#如何查找pytorch模型对应的部署配置文件)
    - [示例](#示例)
  - [如何评测模型](#如何评测模型)
  - [各后端已支持导出的模型列表](#各后端已支持导出的模型列表)

这篇教程介绍了如何使用 MMDeploy 的工具将一个 OpenMMlab 模型转换成某个后端的模型文件。

注意:

- 现在已支持的后端包括 [ONNXRuntime](../05-supported-backends/onnxruntime.md) ，[TensorRT](../05-supported-backends/tensorrt.md) ，[ncnn](../05-supported-backends/ncnn.md) ，[PPLNN](../05-supported-backends/pplnn.md) ，[OpenVINO](../05-supported-backends/openvino.md)。
- 现在已支持的代码库包括 [MMPretrain](../04-supported-codebases/mmpretrain.md) ，[MMDetection](../04-supported-codebases/mmdet.md) ，[MMSegmentation](../04-supported-codebases/mmseg.md) ，[MMOCR](../04-supported-codebases/mmocr.md) ，[MMagic](../04-supported-codebases/mmagic.md)。

## 如何将模型从pytorch形式转换成其他后端形式

### 准备工作

### 准备工作

1. 安装您的目标后端。 您可以参考 [ONNXRuntime-install](../05-supported-backends/onnxruntime.md) ，[TensorRT-install](../05-supported-backends/tensorrt.md) ，[ncnn-install](../05-supported-backends/ncnn.md) ，[PPLNN-install](../05-supported-backends/pplnn.md), [OpenVINO-install](../05-supported-backends/openvino.md)。
2. 安装您的目标代码库。 您可以参考 [MMPretrain-install](https://mmpretrain.readthedocs.io/en/latest/get_started.html#installation)，[MMDetection-install](https://mmdetection.readthedocs.io/en/latest/get_started.html#installation)，[MMSegmentation-install](https://mmsegmentation.readthedocs.io/en/latest/get_started.html#installation)，[MMOCR-install](https://mmocr.readthedocs.io/en/latest/get_started/install.html#installation-steps)，[MMagic-install](https://mmagic.readthedocs.io/en/latest/get_started/install.html#installation)。

### 使用方法

```bash
python ./tools/deploy.py \
    ${DEPLOY_CFG_PATH} \
    ${MODEL_CFG_PATH} \
    ${MODEL_CHECKPOINT_PATH} \
    ${INPUT_IMG} \

### 使用方法

```bash
    --test-img ${TEST_IMG} \
    --work-dir ${WORK_DIR} \
    --calib-dataset-cfg ${CALIB_DATA_CFG} \
    --device ${DEVICE} \
    --log-level INFO \
    --show \
```

### 参数描述

- `deploy_cfg` : mmdeploy 针对此模型的部署配置，包含推理框架类型、是否量化、输入 shape 是否动态等。配置文件之间可能有引用关系，`configs/mmpretrain/classification_ncnn_static.py` 是一个示例。
- `model_cfg` : mm 算法库的模型配置，例如 `mmpretrain/configs/vision_transformer/vit-base-p32_ft-64xb64_in1k-384.py`，与 mmdeploy 的路径无关。
- `checkpoint` : torch 模型路径。可以 http/https 开头，详见 `mmcv.FileClient`   的实现。
- `img` : 模型转换时，用做测试的图像或点云文件路径。
- `--test-img` : 用于测试模型的图像文件路径。默认设置成`None`。
- `--work-dir` : 工作目录，用来保存日志和模型文件。
- `--calib-dataset-cfg` : 此参数只有int8模式下生效，用于校准数据集配置文件。若在int8模式下未传入参数，则会自动使用模型配置文件中的'val'数据集进行校准。
- `--device` : 用于模型转换的设备。 默认是`cpu`，对于 trt 可使用 `cuda:0` 这种形式。

- `--log-level` : 设置日记的等级，选项包括`'CRITICAL'， 'FATAL'， 'ERROR'， 'WARN'， 'WARNING'， 'INFO'， 'DEBUG'， 'NOTSET'`。 默认是`INFO`。
- `--show` : 是否显示检测的结果。
- `--dump-info` : 是否输出 SDK 信息。

### 如何查找pytorch模型对应的部署配置文件

1. 在 `configs/` 文件夹中找到模型对应的代码库文件夹。 例如，转换一个yolov3模型您可以查找到 `configs/mmdet` 文件夹。
2. 根据模型的任务类型在 `configs/codebase_folder/` 下查找对应的文件夹。 例如yolov3模型，您可以查找到 `configs/mmdet/detection` 文件夹。
3. 在 `configs/codebase_folder/task_folder/` 下找到模型的部署配置文件。 例如部署yolov3您可以使用 `configs/mmdet/detection/detection_onnxruntime_dynamic.py`。

### 示例

```bash
    configs/mmdet/detection/detection_tensorrt_dynamic-320x320-1344x1344.py \
    $PATH_TO_MMDET/configs/yolo/yolov3_d53_8xb8-ms-608-273e_coco.py \
    $PATH_TO_MMDET/checkpoints/yolo/yolov3_d53_mstrain-608_273e_coco_20210518_115020-a2c3acb8.pth \

$PATH_TO_MMDET/configs/yolo/yolov3_d53_8xb8-ms-608-273e_coco.py \
    $PATH_TO_MMDET/demo/demo.jpg \
    --work-dir work_dir \
    --show \
    --device cuda:0
```

## 如何评测模型

您可以尝试去评测转换出来的模型 ，参考 [profile 模型](profile_model.md)。

## 各后端已支持导出的模型列表

参考[已支持的模型列表](../03-benchmark/supported_models.md)。

# A 20-Minute Guide to MMAction2 FrameWork

In this tutorial, we will demonstrate the overall architecture of our `MMACTION2 1.0` through a step-by-step example of video action recognition.

The structure of this tutorial is as follows:

- [A 20-Minute Guide to MMAction2 FrameWork](#a-20-minute-guide-to-mmaction2-framework)
  - [Step0: Prepare Data](#step0-prepare-data)
  - [Step1: Build a Pipeline](#step1-build-a-pipeline)
  - [Step2: Build a Dataset and DataLoader](#step2-build-a-dataset-and-dataloader)
  - [Step3: Build a Recognizer](#step3-build-a-recognizer)
  - [Step4: Build a Evaluation Metric](#step4-build-a-evaluation-metric)
  - [Step5: Train and Test with Native PyTorch](#step5-train-and-test-with-native-pytorch)
  - [Step6: Train and Test with MMEngine (Recommended)](#step6-train-and-test-with-mmengine-recommended)

- [Step5: Train and Test with Native PyTorch](#step5-train-and-test-with-native-pytorch)

First, we need to initialize the `scope` for registry, to ensure that each module is registered under the scope of `mmaction`. For more detailed information about registry, please refer to [MMEngine Tutorial](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/registry.html).

```python
from mmaction.utils import register_all_modules

register_all_modules(init_default_scope=True)
```

## Step0: Prepare Data

Please download our self-made [kinetics400_tiny](https://download.openmmlab.com/mmaction/kinetics400_tiny.zip) dataset and extract it to the `$MMACTION2/data` directory.
The directory structure after extraction should be as follows:

```
mmaction2
├── data
│   ├── kinetics400_tiny
│   │    ├── kinetics_tiny_train_video.txt
│   │    ├── kinetics_tiny_val_video.txt

```
mmaction2
├── data
│   │    ├── train
│   │    │   ├── 27_CSXByd3s.mp4
│   │    │   ├── 34XczvTaRiI.mp4
│   │    │   ├── A-wiliK50Zw.mp4
│   │    │   ├── ...
│   │    └── val
│   │       ├── 0pVGiAU6XEA.mp4
│   │       ├── AQrbRSnRt8M.mp4
│   │       ├── ...
```

Here are some examples from the annotation file `kinetics_tiny_train_video.txt`:

```
D32_1gwq35E.mp4 0
iRuyZSKhHRg.mp4 1
oXy-e_P_cAI.mp4 0
34XczvTaRiI.mp4 1
h2YqqUhnR34.mp4 0
```

Each line in the file represents the annotation of a video, where the first item denotes the video filename (e.g., `D32_1gwq35E.mp4`), and the second item represents the corresponding label (e.g., label `0` for `D32_1gwq35E.mp4`). In this dataset, there are only `two` categories.

## Step1: Build a Pipeline

In order to `decode`, `sample`, `resize`, `crop`, `format`, and `pack` the input video and corresponding annotation, we need to design a pipeline to handle these processes. Specifically, we design seven `Transform` classes to build this video processing pipeline. Note that all `Transform` classes in OpenMMLab must inherit from the `BaseTransform` class in `mmcv`, implement the abstract method `transform`, and be registered to the `TRANSFORMS` registry. For more detailed information about data transform, please refer to [MMEngine Tutorial](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/data_transform.html).

```python
import mmcv
import decord
import numpy as np
from mmcv.transforms import TRANSFORMS, BaseTransform, to_tensor
from mmaction.structures import ActionDataSample

@TRANSFORMS.register_module()
class VideoInit(BaseTransform):
    def transform(self, results):
        container = decord.VideoReader(results['filename'])

        results['total_frames'] = len(container)
        results['video_reader'] = container
        return results

class VideoSample(BaseTransform):
    def __init__(self, clip_len, num_clips, test_mode=False):
        self.clip_len = clip_len
        self.num_clips = num_clips
        self.test_mode = test_mode

        total_frames = results['total_frames']
        interval = total_frames // self.clip_len

        if self.test_mode:
            # Make the sampling during testing deterministic
            np.random.seed(42)

        inds_of_all_clips = []
        for i in range(self.num_clips):
            bids = np.arange(self.clip_len) * interval
            offset = np.random.randint(interval, size=bids.shape)
            inds = bids + offset

for i in range(self.num_clips):
            inds_of_all_clips.append(inds)

        results['frame_inds'] = np.concatenate(inds_of_all_clips)
        results['clip_len'] = self.clip_len
        results['num_clips'] = self.num_clips

class VideoDecode(BaseTransform):
        frame_inds = results['frame_inds']
        container = results['video_reader']

        imgs = container.get_batch(frame_inds).asnumpy()
        imgs = list(imgs)

        results['video_reader'] = None
        del container

        results['imgs'] = imgs
        results['img_shape'] = imgs[0].shape[:2]

class VideoResize(BaseTransform):
    def __init__(self, r_size):
        self.r_size = (np.inf, r_size)

return results

        img_h, img_w = results['img_shape']
        new_w, new_h = mmcv.rescale_size((img_w, img_h), self.r_size)

        imgs = [mmcv.imresize(img, (new_w, new_h))
                for img in results['imgs']]

class VideoCrop(BaseTransform):
    def __init__(self, c_size):
        self.c_size = c_size

        center_x, center_y = img_w // 2, img_h // 2
        x1, x2 = center_x - self.c_size // 2, center_x + self.c_size // 2
        y1, y2 = center_y - self.c_size // 2, center_y + self.c_size // 2
        imgs = [img[y1:y2, x1:x2] for img in results['imgs']]

y1, y2 = center_y - self.c_size // 2, center_y + self.c_size // 2

class VideoFormat(BaseTransform):
        num_clips = results['num_clips']
        clip_len = results['clip_len']
        imgs = results['imgs']

        # [num_clips*clip_len, H, W, C]
        imgs = np.array(imgs)
        # [num_clips, clip_len, H, W, C]
        imgs = imgs.reshape((num_clips, clip_len) + imgs.shape[1:])
        # [num_clips, C, clip_len, H, W]
        imgs = imgs.transpose(0, 4, 1, 2, 3)

class VideoPack(BaseTransform):
    def __init__(self, meta_keys=('img_shape', 'num_clips', 'clip_len')):
        self.meta_keys = meta_keys

        packed_results = dict()
        inputs = to_tensor(results['imgs'])
        data_sample = ActionDataSample().set_gt_labels(results['label'])
        metainfo = {k: results[k] for k in self.meta_keys if k in results}
        data_sample.set_metainfo(metainfo)
        packed_results['inputs'] = inputs
        packed_results['data_samples'] = data_sample
        return packed_results
```

Below, we provide a code snippet (using `D32_1gwq35E.mp4 0` from the annotation file) to demonstrate how to use the pipeline.

```python
import os.path as osp
from mmengine.dataset import Compose

pipeline_cfg = [
    dict(type='VideoInit'),
    dict(type='VideoSample', clip_len=16, num_clips=1, test_mode=False),
    dict(type='VideoDecode'),
    dict(type='VideoResize', r_size=256),
    dict(type='VideoCrop', c_size=224),

dict(type='VideoSample', clip_len=16, num_clips=1, test_mode=False),
    dict(type='VideoFormat'),
    dict(type='VideoPack')
]

pipeline = Compose(pipeline_cfg)
data_prefix = 'data/kinetics400_tiny/train'
results = dict(filename=osp.join(data_prefix, 'D32_1gwq35E.mp4'), label=0)
packed_results = pipeline(results)

inputs = packed_results['inputs']
data_sample = packed_results['data_samples']

print('shape of the inputs: ', inputs.shape)

# Get metainfo of the inputs
print('image_shape: ', data_sample.img_shape)
print('num_clips: ', data_sample.num_clips)
print('clip_len: ', data_sample.clip_len)

# Get label of the inputs
print('label: ', data_sample.gt_labels.item)
```

```
shape of the inputs:  torch.Size([1, 3, 16, 224, 224])
image_shape:  (224, 224)
num_clips:  1
clip_len:  16
label:  tensor([0])
```

## Step2: Build a Dataset and DataLoader

```

```
```

All `Dataset` classes in OpenMMLab must inherit from the `BaseDataset` class in `mmengine`. We can customize annotation loading process by overriding the `load_data_list` method. Additionally, we can add more information to the `results` dict that is passed as input to the `pipeline` by overriding the `get_data_info` method. For more detailed information about `BaseDataset` class, please refer to [MMEngine Tutorial](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/basedataset.html).

```python
from mmengine.fileio import list_from_file
from mmengine.dataset import BaseDataset
from mmaction.registry import DATASETS

class DatasetZelda(BaseDataset):
    def __init__(self, ann_file, pipeline, data_root, data_prefix=dict(video=''),

                 test_mode=False, modality='RGB', **kwargs):
        self.modality = modality
        super(DatasetZelda, self).__init__(ann_file=ann_file, pipeline=pipeline, data_root=data_root,
                                           data_prefix=data_prefix, test_mode=test_mode,
                                           **kwargs)

    def load_data_list(self):
        fin = list_from_file(self.ann_file)
        for line in fin:
            line_split = line.strip().split()
            filename, label = line_split
            label = int(label)
            filename = osp.join(self.data_prefix['video'], filename)
            data_list.append(dict(filename=filename, label=label))

    def get_data_info(self, idx: int) -> dict:

data_list.append(dict(filename=filename, label=label))

        data_info = super().get_data_info(idx)
        data_info['modality'] = self.modality
        return data_info
```

Next, we will demonstrate how to use dataset and dataloader to index data. We will use the `Runner.build_dataloader` method to construct the dataloader. For more detailed information about dataloader, please refer to [MMEngine Tutorial](https://mmengine.readthedocs.io/en/latest/tutorials/dataset.html#details-on-dataloader).

```python

train_pipeline_cfg = [
]

val_pipeline_cfg = [

dict(type='VideoResize', r_size=256),
]

    dict(type='VideoSample', clip_len=16, num_clips=5, test_mode=True),
]

train_dataset_cfg = dict(
    type='DatasetZelda',
    ann_file='kinetics_tiny_train_video.txt',
    pipeline=train_pipeline_cfg,
    data_root='data/kinetics400_tiny/',
    data_prefix=dict(video='train'))

val_dataset_cfg = dict(
    ann_file='kinetics_tiny_val_video.txt',
    pipeline=val_pipeline_cfg,
    data_prefix=dict(video='val'))

train_dataset = DATASETS.build(train_dataset_cfg)

packed_results = train_dataset[0]

data_root='data/kinetics400_tiny/',

from mmengine.runner import Runner

BATCH_SIZE = 2

train_dataloader_cfg = dict(
    batch_size=BATCH_SIZE,
    num_workers=0,
    persistent_workers=False,
    sampler=dict(type='DefaultSampler', shuffle=True),
    dataset=train_dataset_cfg)

val_dataloader_cfg = dict(
    dataset=val_dataset_cfg)

train_data_loader = Runner.build_dataloader(dataloader=train_dataloader_cfg)
val_data_loader = Runner.build_dataloader(dataloader=val_dataloader_cfg)

batched_packed_results = next(iter(train_data_loader))

batched_inputs = batched_packed_results['inputs']
batched_data_sample = batched_packed_results['data_samples']

assert len(batched_inputs) == BATCH_SIZE
assert len(batched_data_sample) == BATCH_SIZE
```

The terminal output should be the same as the one shown in the [Step1: Build a Pipeline](#step1-build-a-pipeline).

## Step3: Build a Recognizer

Next, we will construct the `recognizer`, which mainly consists of three parts: `data preprocessor` for batching and normalizing the data, `backbone` for feature extraction, and `cls_head` for classification.

The implementation of `data_preprocessor` is as follows:

```python
import torch
from mmengine.model import BaseDataPreprocessor, stack_batch
from mmaction.registry import MODELS

class DataPreprocessorZelda(BaseDataPreprocessor):
    def __init__(self, mean, std):
        super().__init__()

        self.register_buffer(
            'mean',
            torch.tensor(mean, dtype=torch.float32).view(-1, 1, 1, 1),
            False)
            'std',
            torch.tensor(std, dtype=torch.float32).view(-1, 1, 1, 1),
            False)

    def forward(self, data, training=False):
        data = self.cast_data(data)
        inputs = data['inputs']
        batch_inputs = stack_batch(inputs)  # Batching
        batch_inputs = (batch_inputs - self.mean) / self.std  # Normalization
        data['inputs'] = batch_inputs
        return data
```

batch_inputs = stack_batch(inputs)  # Batching
```

Here is the usage of data_preprocessor: feed the `batched_packed_results` obtained from the [Step2: Build a Dataset and DataLoader](#step2-build-a-dataset-and-dataloader) into the `data_preprocessor` for batching and normalization.

```python

data_preprocessor_cfg = dict(
    type='DataPreprocessorZelda',
    std=[58.395, 57.12, 57.375])

data_preprocessor = MODELS.build(data_preprocessor_cfg)

preprocessed_inputs = data_preprocessor(batched_packed_results)
print(preprocessed_inputs['inputs'].shape)
```

```
torch.Size([2, 1, 3, 16, 224, 224])
```

The implementations of `backbone`, `cls_head` and `recognizer` are as follows:

```python
import torch.nn as nn
import torch.nn.functional as F

```

```python
from mmengine.model import BaseModel, BaseModule, Sequential
from mmengine.structures import LabelData

class BackBoneZelda(BaseModule):
    def __init__(self, init_cfg=None):
        if init_cfg is None:
            init_cfg = [dict(type='Kaiming', layer='Conv3d', mode='fan_out', nonlinearity="relu"),
                        dict(type='Constant', layer='BatchNorm3d', val=1, bias=0)]

        super(BackBoneZelda, self).__init__(init_cfg=init_cfg)

        self.conv1 = Sequential(nn.Conv3d(3, 64, kernel_size=(3, 7, 7),
                                          stride=(1, 2, 2), padding=(1, 3, 3)),
                                nn.BatchNorm3d(64), nn.ReLU())

stride=(1, 2, 2), padding=(1, 3, 3)),
        self.maxpool = nn.MaxPool3d(kernel_size=(1, 3, 3), stride=(1, 2, 2),
                                    padding=(0, 1, 1))

        self.conv = Sequential(nn.Conv3d(64, 128, kernel_size=3, stride=2, padding=1),
                               nn.BatchNorm3d(128), nn.ReLU())

    def forward(self, imgs):
        # imgs: [batch_size*num_views, 3, T, H, W]
        # features: [batch_size*num_views, 128, T/2, H//8, W//8]
        features = self.conv(self.maxpool(self.conv1(imgs)))
        return features

class ClsHeadZelda(BaseModule):
    def __init__(self, num_classes, in_channels, dropout=0.5, average_clips='prob', init_cfg=None):
            init_cfg = dict(type='Normal', layer='Linear', std=0.01)

        super(ClsHeadZelda, self).__init__(init_cfg=init_cfg)

if init_cfg is None:

        self.num_classes = num_classes
        self.in_channels = in_channels
        self.average_clips = average_clips

        if dropout != 0:
            self.dropout = nn.Dropout(dropout)
        else:
            self.dropout = None

        self.fc = nn.Linear(self.in_channels, self.num_classes)
        self.pool = nn.AdaptiveAvgPool3d(1)
        self.loss_fn = nn.CrossEntropyLoss()

    def forward(self, x):
        N, C, T, H, W = x.shape
        x = self.pool(x)
        x = x.view(N, C)
        assert x.shape[1] == self.in_channels

        if self.dropout is not None:
            x = self.dropout(x)

        cls_scores = self.fc(x)
        return cls_scores

    def loss(self, feats, data_samples):
        cls_scores = self(feats)
        labels = torch.stack([x.gt_labels.item for x in data_samples])

return cls_scores

        labels = labels.squeeze()

        if labels.shape == torch.Size([]):
            labels = labels.unsqueeze(0)

        loss_cls = self.loss_fn(cls_scores, labels)
        return dict(loss_cls=loss_cls)

    def predict(self, feats, data_samples):
        num_views = cls_scores.shape[0] // len(data_samples)
        # assert num_views == data_samples[0].num_clips
        cls_scores = self.average_clip(cls_scores, num_views)

        for ds, sc in zip(data_samples, cls_scores):
            pred = LabelData(item=sc)
            ds.pred_scores = pred
        return data_samples

    def average_clip(self, cls_scores, num_views):
          if self.average_clips not in ['score', 'prob', None]:
            raise ValueError(f'{self.average_clips} is not supported. '

def average_clip(self, cls_scores, num_views):
                             f'Currently supported ones are '
                             f'["score", "prob", None]')

          total_views = cls_scores.shape[0]
          cls_scores = cls_scores.view(total_views // num_views, num_views, -1)

          if self.average_clips is None:
              return cls_scores
          elif self.average_clips == 'prob':
              cls_scores = F.softmax(cls_scores, dim=2).mean(dim=1)
          elif self.average_clips == 'score':
              cls_scores = cls_scores.mean(dim=1)

          return cls_scores

class RecognizerZelda(BaseModel):
    def __init__(self, backbone, cls_head, data_preprocessor):
        super().__init__(data_preprocessor=data_preprocessor)

def __init__(self, backbone, cls_head, data_preprocessor):

        self.cls_head = MODELS.build(cls_head)

    def extract_feat(self, inputs):
        inputs = inputs.view((-1, ) + inputs.shape[2:])
        return self.backbone(inputs)

    def loss(self, inputs, data_samples):
        feats = self.extract_feat(inputs)
        loss = self.cls_head.loss(feats, data_samples)
        return loss

    def predict(self, inputs, data_samples):
        predictions = self.cls_head.predict(feats, data_samples)
        return predictions

    def forward(self, inputs, data_samples=None, mode='tensor'):
        if mode == 'tensor':
            return self.extract_feat(inputs)
        elif mode == 'loss':
            return self.loss(inputs, data_samples)
        elif mode == 'predict':
            return self.predict(inputs, data_samples)

elif mode == 'loss':
        else:
            raise RuntimeError(f'Invalid mode: {mode}')
```

The `init_cfg` is used for model weight initialization. For more information on model weight initialization, please refer to [MMEngine Tutorial](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/initialize.html). The usage of the above modules is as follows:

```python
import copy

model_cfg = dict(
    type='RecognizerZelda',
    backbone=dict(type='BackBoneZelda'),
    cls_head=dict(
        type='ClsHeadZelda',
        num_classes=2,
        in_channels=128,
        average_clips='prob'),
    data_preprocessor = dict(
        type='DataPreprocessorZelda',
        std=[58.395, 57.12, 57.375]))

model = MODELS.build(model_cfg)

# Train
model.train()

type='DataPreprocessorZelda',

# Train
model.init_weights()
data_batch_train = copy.deepcopy(batched_packed_results)
data = model.data_preprocessor(data_batch_train, training=True)
loss = model(**data, mode='loss')
print('loss dict: ', loss)

# Test
with torch.no_grad():
    model.eval()
    data_batch_test = copy.deepcopy(batched_packed_results)
    data = model.data_preprocessor(data_batch_test, training=False)
    predictions = model(**data, mode='predict')
print('Label of Sample[0]', predictions[0].gt_labels.item)
print('Scores of Sample[0]', predictions[0].pred_scores.item)
```

```shell
04/03 23:28:01 - mmengine - INFO -
backbone.conv1.0.weight - torch.Size([64, 3, 3, 7, 7]):
KaimingInit: a=0, mode=fan_out, nonlinearity=relu, distribution =normal, bias=0

backbone.conv1.0.bias - torch.Size([64]):

backbone.conv1.1.weight - torch.Size([64]):
The value is the same before and after calling `init_weights` of RecognizerZelda

backbone.conv1.1.bias - torch.Size([64]):

backbone.conv.0.weight - torch.Size([128, 64, 3, 3, 3]):

backbone.conv.0.bias - torch.Size([128]):

backbone.conv.1.weight - torch.Size([128]):

backbone.conv.1.bias - torch.Size([128]):

cls_head.fc.weight - torch.Size([2, 128]):
NormalInit: mean=0, std=0.01, bias=0

cls_head.fc.bias - torch.Size([2]):

loss dict:  {'loss_cls': tensor(0.6853, grad_fn=<NllLossBackward0>)}
Label of Sample[0] tensor([0])
Scores of Sample[0] tensor([0.5240, 0.4760])
```

## Step4: Build a Evaluation Metric

```

Note that all `Metric` classes in `OpenMMLab` must inherit from the `BaseMetric` class in `mmengine` and  implement the abstract methods, `process` and `compute_metrics`. For more information on evaluation, please refer to [MMEngine Tutorial](https://mmengine.readthedocs.io/en/latest/tutorials/evaluation.html).

```python
from collections import OrderedDict
from mmengine.evaluator import BaseMetric
from mmaction.evaluation import top_k_accuracy
from mmaction.registry import METRICS

@METRICS.register_module()
class AccuracyMetric(BaseMetric):
    def __init__(self, topk=(1, 5), collect_device='cpu', prefix='acc'):
        super().__init__(collect_device=collect_device, prefix=prefix)
        self.topk = topk

    def process(self, data_batch, data_samples):

super().__init__(collect_device=collect_device, prefix=prefix)

        data_samples = copy.deepcopy(data_samples)
        for data_sample in data_samples:
            result = dict()
            scores = data_sample['pred_scores']['item'].cpu().numpy()
            label = data_sample['gt_labels']['item'].item()
            result['scores'] = scores
            result['label'] = label
            self.results.append(result)

    def compute_metrics(self, results: list) -> dict:
        eval_results = OrderedDict()
        labels = [res['label'] for res in results]
        scores = [res['scores'] for res in results]
        topk_acc = top_k_accuracy(scores, labels, self.topk)
        for k, acc in zip(self.topk, topk_acc):
            eval_results[f'topk{k}'] = acc
        return eval_results
```

```python

metric_cfg = dict(type='AccuracyMetric', topk=(1, 5))

eval_results[f'topk{k}'] = acc
```

```python

metric = METRICS.build(metric_cfg)

data_samples = [d.to_dict() for d in predictions]

metric.process(batched_packed_results, data_samples)
acc = metric.compute_metrics(metric.results)
print(acc)
```

```shell
OrderedDict([('topk1', 0.5), ('topk5', 1.0)])
```

## Step5: Train and Test with Native PyTorch

```python
import torch.optim as optim
from mmengine import track_iter_progress

device = 'cuda' # or 'cpu'
max_epochs = 10

optimizer = optim.Adam(model.parameters(), lr=0.01)

for epoch in range(max_epochs):
    model.train()
    losses = []
    for data_batch in track_iter_progress(train_data_loader):
        data = model.data_preprocessor(data_batch, training=True)
        loss_dict = model(**data, mode='loss')
        loss = loss_dict['loss_cls']

        optimizer.zero_grad()
        loss.backward()

loss_dict = model(**data, mode='loss')

        optimizer.step()

        losses.append(loss.item())

    print(f'Epoch[{epoch}]: loss ', sum(losses) / len(train_data_loader))

    with torch.no_grad():
        model.eval()
        for data_batch in track_iter_progress(val_data_loader):
            data = model.data_preprocessor(data_batch, training=False)
            predictions = model(**data, mode='predict')
            data_samples = [d.to_dict() for d in predictions]
            metric.process(data_batch, data_samples)

        acc = metric.acc = metric.compute_metrics(metric.results)
        for name, topk in acc.items():
            print(f'{name}: ', topk)
```

## Step6: Train and Test with MMEngine (Recommended)

acc = metric.acc = metric.compute_metrics(metric.results)
```

For more details on training and testing, you can refer to [MMAction2 Tutorial](https://mmaction2.readthedocs.io/en/latest/user_guides/train_test.html). For more information on `Runner`, please refer to [MMEngine Tutorial](https://mmengine.readthedocs.io/en/latest/tutorials/runner.html).

```python

train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=10, val_interval=1)
val_cfg = dict(type='ValLoop')

optim_wrapper = dict(optimizer=dict(type='Adam', lr=0.01))

runner = Runner(model=model_cfg, work_dir='./work_dirs/guide',
                train_dataloader=train_dataloader_cfg,
                train_cfg=train_cfg,
                val_dataloader=val_dataloader_cfg,
                val_cfg=val_cfg,
                optim_wrapper=optim_wrapper,

train_cfg=train_cfg,
                val_evaluator=[metric_cfg],
                default_scope='mmaction')
runner.train()
```

# VideoMAE V2

[VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://arxiv.org/abs/2303.16727)

Scale is the primary factor for building a powerful foundation model that could well generalize to a variety of downstream tasks. However, it is still challenging to train video foundation models with billions of parameters. This paper shows that video masked autoencoder (VideoMAE) is a scalable and general self-supervised pre-trainer for building video foundation models. We scale the VideoMAE in both model and data with a core design. Specifically, we present a dual masking strategy for efficient pre-training, with an encoder operating on a subset of video tokens and a decoder processing another subset of video tokens. Although VideoMAE is very efficient due to high masking ratio in encoder, masking decoder can still further reduce the overall computational cost. This enables the efficient pre-training of billion-level models in video. We also use a progressive training paradigm that involves an initial pre-training on a diverse multi-sourced unlabeled dataset, followed by a post-pre-training on a mixed labeled dataset. Finally, we successfully train a video ViT model with a billion parameters, which achieves a new state-of-the-art performance on the datasets of Kinetics (90.0% on K400 and 89.9% on K600) and Something-Something (68.7% on V1 and 77.0% on V2). In addition, we extensively verify the pre-trained video ViT models on a variety of downstream tasks, demonstrating its effectiveness as a general video representation learner.

<img src="https://user-images.githubusercontent.com/35596075/237352561-6204d743-8705-43f5-817f-0bc4907b88d0.png" width="800"/>
</div>

| frame sampling strategy |   resolution   | backbone | top1 acc | top5 acc |         reference top1 acc         |         reference top5 acc         | testing protocol  | FLOPs | params |         config         |         ckpt          |
| :---------------------: | :------------: | :------: | :------: | :------: | :--------------------------------: | :--------------------------------: | :---------------: | :---: | :----: | :--------------------: | :-------------------: |

|         16x4x1          | short-side 320 |  ViT-S   |   83.6   |   96.3   | 83.7 \[[VideoMAE V2](https://github.com/OpenGVLab/VideoMAEv2/blob/master/docs/MODEL_ZOO.md)\] | 96.2 \[[VideoMAE V2](https://github.com/OpenGVLab/VideoMAEv2/blob/master/docs/MODEL_ZOO.md)\] | 5 clips x 3 crops |  57G  |  22M   | [config](/configs/recognition/videomaev2/vit-small-p16_videomaev2-vit-g-dist-k710-pre_16x4x1_kinetics-400.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/videomaev2/vit-small-p16_videomaev2-vit-g-dist-k710-pre_16x4x1_kinetics-400/vit-small-p16_videomaev2-vit-g-dist-k710-pre_16x4x1_kinetics-400_20230510-25c748fd.pth) \[1\] |

|         16x4x1          | short-side 320 |  ViT-B   |   86.6   |   97.3   | 86.6 \[[VideoMAE V2](https://github.com/OpenGVLab/VideoMAEv2/blob/master/docs/MODEL_ZOO.md)\] | 97.3 \[[VideoMAE V2](https://github.com/OpenGVLab/VideoMAEv2/blob/master/docs/MODEL_ZOO.md)\] | 5 clips x 3 crops | 180G  |  87M   | [config](/configs/recognition/videomaev2/vit-base-p16_videomaev2-vit-g-dist-k710-pre_16x4x1_kinetics-400.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/videomaev2/vit-base-p16_videomaev2-vit-g-dist-k710-pre_16x4x1_kinetics-400/vit-base-p16_videomaev2-vit-g-dist-k710-pre_16x4x1_kinetics-400_20230510-3e7f93b2.pth) \[1\] |

\[1\] The models were distilled from the VideoMAE V2-g model. Specifically, models are initialized with VideoMAE V2 pretraining, then distilled on Kinetics 710 dataset. They are ported from the repo [VideoMAE V2](https://github.com/OpenGVLab/VideoMAEv2) and tested on our data. The VideoMAE V2-g model can be obtained from the original repository. Currently, we only support the testing of VideoMAE V2 models.

1. The values in columns named after "reference" are the results of the original repo.

2. The validation set of Kinetics400 we used consists of 19796 videos. These videos are available at [Kinetics400-Validation](https://mycuhk-my.sharepoint.com/:u:/g/personal/1155136485_link_cuhk_edu_hk/EbXw2WX94J1Hunyt3MWNDJUBz-nHvQYhO9pvKqm6g39PMA?e=a9QldB). The corresponding [data list](https://download.openmmlab.com/mmaction/dataset/k400_val/kinetics_val_list.txt) (each line is of the format 'video_id, num_frames, label_index') and the [label map](https://download.openmmlab.com/mmaction/dataset/k400_val/kinetics_class2ind.txt) are also available.

For more details on data preparation, you can refer to [preparing_kinetics](/tools/data/kinetics/README.md).

## Test

```shell
```

Example: test ViT-base model on Kinetics-400 dataset and dump the result to a pkl file.

```shell
```

```shell
python tools/test.py configs/recognition/videomaev2/vit-base-p16_videomaev2-vit-g-dist-k710-pre_16x4x1_kinetics-400.py \
```

```BibTeX
@misc{wang2023videomaev2,
      title={VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking},
      author={Limin Wang and Bingkun Huang and Zhiyu Zhao and Zhan Tong and Yinan He and Yi Wang and Yali Wang and Yu Qiao},
      year={2023},
      eprint={2303.16727},
}
```

# Compatibility of MMDetection 2.x

## MMDetection 2.25.0

In order to support Mask2Former for instance segmentation, the original config files of Mask2Former for panpotic segmentation need to be renamed [PR #7571](https://github.com/open-mmlab/mmdetection/pull/7571).

    <thead>
        <tr align='center'>
            <td>before v2.25.0</td>
            <td>after v2.25.0</td>
        </tr>
    </thead>
    <tbody><tr valign='top'>
    <th>

```
'mask2former_xxx_coco.py' represents config files for **panoptic segmentation**.
```

</th>
    <th>

```
'mask2former_xxx_coco.py' represents config files for **instance segmentation**.
'mask2former_xxx_coco-panoptic.py' represents config files for **panoptic segmentation**.
```

</th></tr>
  </tbody></table>

## MMDetection 2.21.0

In order to support CPU training, the logic of scatter in batch collating has been changed. We recommend to use

```

MMCV v1.4.4 or higher. For more details, please refer to [MMCV PR #1621](https://github.com/open-mmlab/mmcv/pull/1621).

## MMDetection 2.18.1

### MMCV compatibility

In order to fix the wrong weight reference bug in BaseTransformerLayer, the logic in batch first mode of MultiheadAttention has been changed.
We recommend to use MMCV v1.3.17 or higher. For more details, please refer to [MMCV PR #1418](https://github.com/open-mmlab/mmcv/pull/1418).

## MMDetection 2.18.0

### DIIHead compatibility

In order to support QueryInst, attn_feats is added into the returned tuple of DIIHead.

## MMDetection 2.14.0

### MMCV Version

In order to fix the problem that the priority of EvalHook is too low, all hook priorities have been re-adjusted in 1.3.8, so MMDetection 2.14.0 needs to rely on the latest MMCV 1.3.8 version. For related information, please refer to [#1120](https://github.com/open-mmlab/mmcv/pull/1120), for related issues, please refer to [#5343](https://github.com/open-mmlab/mmdetection/issues/5343).

### SSD compatibility

In v2.14.0, to make SSD more flexible to use, [PR5291](https://github.com/open-mmlab/mmdetection/pull/5291) refactored its backbone, neck and head. The users can use the script `tools/model_converters/upgrade_ssd_version.py` to convert their models.

```bash
python tools/model_converters/upgrade_ssd_version.py ${OLD_MODEL_PATH} ${NEW_MODEL_PATH}
```

```bash
```

- OLD_MODEL_PATH: the path to load the old version SSD model.
- NEW_MODEL_PATH: the path to save the converted model weights.

## MMDetection 2.12.0

MMDetection is going through big refactoring for more general and convenient usages during the releases from v2.12.0 to v2.18.0 (maybe longer).
In v2.12.0 MMDetection inevitably brings some BC-breakings, including the MMCV dependency, model initialization, model registry, and mask AP evaluation.

MMDetection v2.12.0 relies on the newest features in MMCV 1.3.3, including `BaseModule` for unified parameter initialization, model registry, and the CUDA operator `MultiScaleDeformableAttn` for [Deformable DETR](https://arxiv.org/abs/2010.04159). Note that MMCV 1.3.2 already contains all the features used by MMDet but has known issues. Therefore, we recommend users to skip MMCV v1.3.2 and use v1.3.2, though v1.3.2 might work for most of the cases.

### Unified model initialization

To unify the parameter initialization in OpenMMLab projects, MMCV supports `BaseModule` that accepts `init_cfg` to allow the modules' parameters initialized in a flexible and unified manner. Now the users need to explicitly call `model.init_weights()` in the training script to initialize the model (as in [here](https://github.com/open-mmlab/mmdetection/blob/main/tools/train.py#L162), previously this was handled by the detector. **The downstream projects must update their model initialization accordingly to use MMDetection v2.12.0**. Please refer to PR #4750 for details.

### Unified model registry

To easily use backbones implemented in other OpenMMLab projects, MMDetection v2.12.0 inherits the model registry created in MMCV (#760). In this way, as long as the backbone is supported in an OpenMMLab project and that project also uses the registry in MMCV, users can use that backbone in MMDetection by simply modifying the config without copying the code of that backbone into MMDetection. Please refer to PR #5059 for more details.

### Mask AP evaluation

Before [PR 4898](https://github.com/open-mmlab/mmdetection/pull/4898) and V2.12.0, the mask AP of small, medium, and large instances is calculated based on the bounding box area rather than the real mask area. This leads to higher `APs` and `APm` but lower `APl` but will not affect the overall mask AP. [PR 4898](https://github.com/open-mmlab/mmdetection/pull/4898) change it to use mask areas by deleting `bbox` in mask AP calculation.

The new calculation does not affect the overall mask AP evaluation and is consistent with [Detectron2](https://github.com/facebookresearch/detectron2/).

## Compatibility with MMDetection 1.x

MMDetection 2.0 goes through a big refactoring and addresses many legacy issues. It is not compatible with the 1.x version, i.e., running inference with the same model weights in these two versions will produce different results. Thus, MMDetection 2.0 re-benchmarks all the models and provides their links and logs in the model zoo.

The major differences are in four folds: coordinate system, codebase conventions, training hyperparameters, and modular design.

### Coordinate System

The new coordinate system is consistent with [Detectron2](https://github.com/facebookresearch/detectron2/) and treats the center of the most left-top pixel as (0, 0) rather than the left-top corner of that pixel.

Accordingly, the system interprets the coordinates in COCO bounding box and segmentation annotations as coordinates in range `[0, width]` or `[0, height]`.
This modification affects all the computation related to the bbox and pixel selection,
which is more natural and accurate.

- The height and width of a box with corners (x1, y1) and (x2, y2) in the new coordinate system is computed as `width = x2 - x1` and `height = y2 - y1`.
  In MMDetection 1.x and previous version, a "+ 1" was added both height and width.
  This modification are in three folds:

  1. Box transformation and encoding/decoding in regression.
  2. IoU calculation. This affects the matching process between ground truth and bounding box and the NMS process. The effect to compatibility is very negligible, though.

2. IoU calculation. This affects the matching process between ground truth and bounding box and the NMS process. The effect to compatibility is very negligible, though.
  3. The corners of bounding box is in float type and no longer quantized. This should provide more accurate bounding box results. This also makes the bounding box and RoIs not required to have minimum size of 1, whose effect is small, though.

- The anchors are center-aligned to feature grid points and in float type.
  In MMDetection 1.x and previous version, the anchors are in `int` type and not center-aligned.
  This affects the anchor generation in RPN and all the anchor-based methods.

- ROIAlign is better aligned with the image coordinate system. The new implementation is adopted from [Detectron2](https://github.com/facebookresearch/detectron2/tree/master/detectron2/layers/csrc/ROIAlign).
  The RoIs are shifted by half a pixel by default when they are used to cropping RoI features, compared to MMDetection 1.x.

The RoIs are shifted by half a pixel by default when they are used to cropping RoI features, compared to MMDetection 1.x.
  The old behavior is still available by setting `aligned=False` instead of `aligned=True`.

- Mask cropping and pasting are more accurate.

  1. We use the new RoIAlign to crop mask targets. In MMDetection 1.x, the bounding box is quantized before it is used to crop mask target, and the crop process is implemented by numpy. In new implementation, the bounding box for crop is not quantized and sent to RoIAlign. This implementation accelerates the training speed by a large margin (~0.1s per iter, ~2 hour when training Mask R50 for 1x schedule) and should be more accurate.

2. In MMDetection 2.0, the "`paste_mask()`" function is different and should be more accurate than those in previous versions. This change follows the modification in [Detectron2](https://github.com/facebookresearch/detectron2/blob/master/detectron2/structures/masks.py) and can improve mask AP on COCO by ~0.5% absolute.

### Codebase Conventions

- MMDetection 2.0 changes the order of class labels to reduce unused parameters in regression and mask branch more naturally (without +1 and -1).
  This effect all the classification layers of the model to have a different ordering of class labels. The final layers of regression branch and mask head no longer keep K+1 channels for K categories, and their class orders are consistent with the classification branch.

  - In MMDetection 2.0, label "K" means background, and labels \[0, K-1\] correspond to the K = num_categories object categories.

- In MMDetection 2.0, label "K" means background, and labels \[0, K-1\] correspond to the K = num_categories object categories.

  - In MMDetection 1.x and previous version, label "0" means background, and labels \[1, K\] correspond to the K categories.

  - **Note**: The class order of softmax RPN is still the same as that in 1.x in versions\<=2.4.0 while sigmoid RPN is not affected. The class orders in all heads are unified since MMDetection v2.5.0.

- Low quality matching in R-CNN is not used. In MMDetection 1.x and previous versions, the `max_iou_assigner` will match low quality boxes for each ground truth box in both RPN and R-CNN training. We observe this sometimes does not assign the most perfect GT box to some bounding boxes,
  thus MMDetection 2.0 do not allow low quality matching by default in R-CNN training in the new system. This sometimes may slightly improve the box AP (~0.1% absolute).

thus MMDetection 2.0 do not allow low quality matching by default in R-CNN training in the new system. This sometimes may slightly improve the box AP (~0.1% absolute).

- Separate scale factors for width and height. In MMDetection 1.x and previous versions, the scale factor is a single float in mode `keep_ratio=True`. This is slightly inaccurate because the scale factors for width and height have slight difference. MMDetection 2.0 adopts separate scale factors for width and height, the improvement on AP ~0.1% absolute.

- Configs name conventions are changed. MMDetection V2.0 adopts the new name convention to maintain the gradually growing model zoo as the following:

  ```shell
  [model]_(model setting)_[backbone]_[neck]_(norm setting)_(misc)_(gpu x batch)_[schedule]_[dataset].py,
  ```

  where the (`misc`) includes DCN and GCBlock, etc. More details are illustrated in the [documentation for config](tutorials/config)

```

- MMDetection V2.0 uses new ResNet Caffe backbones to reduce warnings when loading pre-trained models. Most of the new backbones' weights are the same as the former ones but do not have `conv.bias`, except that they use a different `img_norm_cfg`. Thus, the new backbone will not cause warning of unexpected keys.

### Training Hyperparameters

The change in training hyperparameters does not affect
model-level compatibility but slightly improves the performance. The major ones are:

- The number of proposals after nms is changed from 2000 to 1000 by setting `nms_post=1000` and `max_num=1000`.
  This slightly improves both mask AP and bbox AP by ~0.2% absolute.

- The default box regression losses for Mask R-CNN, Faster R-CNN and RetinaNet are changed from smooth L1 Loss to L1 loss. This leads to an overall improvement in box AP (~0.6% absolute). However, using L1-loss for other methods such as Cascade R-CNN and HTC does not improve the performance, so we keep the original settings for these methods.

- The sample num of RoIAlign layer is set to be 0 for simplicity. This leads to slightly improvement on mask AP (~0.2% absolute).

- The default setting does not use gradient clipping anymore during training for faster training speed. This does not degrade performance of the most of models. For some models such as RepPoints we keep using gradient clipping to stabilize the training process and to obtain better performance.

- The default warmup ratio is changed from 1/3 to 0.001 for a more smooth warming up process since the gradient clipping is usually not used. The effect is found negligible during our re-benchmarking, though.

### Upgrade Models from 1.x to 2.0

To convert the models trained by MMDetection V1.x to MMDetection V2.0, the users can use the script `tools/model_converters/upgrade_model_version.py` to convert
their models. The converted models can be run in MMDetection V2.0 with slightly dropped performance (less than 1% AP absolute).
Details can be found in `configs/legacy`.

## pycocotools compatibility

`mmpycocotools` is the OpenMMlab's fork of official `pycocotools`, which works for both MMDetection and Detectron2.

Before [PR 4939](https://github.com/open-mmlab/mmdetection/pull/4939), since `pycocotools` and `mmpycocotool` have the same package name, if users already installed `pycocotools` (installed Detectron2 first under the same environment), then the setup of MMDetection will skip installing `mmpycocotool`. Thus MMDetection fails due to the missing `mmpycocotools`.
If MMDetection is installed before Detectron2, they could work under the same environment.
[PR 4939](https://github.com/open-mmlab/mmdetection/pull/4939) deprecates mmpycocotools in favor of official pycocotools.
Users may install MMDetection and Detectron2 under the same environment after [PR 4939](https://github.com/open-mmlab/mmdetection/pull/4939), no matter what the installation order is.

# DAB-DETR

> [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://arxiv.org/abs/2201.12329)

We present in this paper a novel query formulation using dynamic anchor boxes for DETR (DEtection TRansformer) and offer a deeper understanding of the role of queries in DETR. This new formulation directly uses box coordinates as queries in Transformer decoders and dynamically updates them layer-by-layer. Using box coordinates not only helps using explicit positional priors to improve the query-to-feature similarity and eliminate the slow training convergence issue in DETR, but also allows us to modulate the positional attention map using the box width and height information. Such a design makes it clear that queries in DETR can be implemented as performing soft ROI pooling layer-by-layer in a cascade manner. As a result, it leads to the best performance on MS-COCO benchmark among the DETR-like detection models under the same setting, e.g., AP 45.7% using ResNet50-DC5 as backbone trained in 50 epochs. We also conducted extensive experiments to confirm our analysis and verify the effectiveness of our methods.

<img src="https://github.com/IDEA-Research/DAB-DETR/blob/main/figure/arch.png?raw=true"/>
</div>
<img src="https://github.com/IDEA-Research/DAB-DETR/blob/main/figure/model.png?raw=true"/>
</div>
<img src="https://github.com/IDEA-Research/DAB-DETR/blob/main/figure/results.png?raw=true"/>
</div>

We provide the config files and models for DAB-DETR: [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://arxiv.org/abs/2201.12329).

| Backbone |  Model   | Lr schd | Mem (GB) | Inf time (fps) | box AP |                  Config                   |                                                                                                                                                Download                                                                                                                                                |

| :------: | :------: | :-----: | :------: | :------------: | :----: | :---------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|   R-50   | DAB-DETR |   50e   |          |                |  42.3  | [config](./dab-detr_r50_8xb2-50e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/dab_detr/dab-detr_r50_8xb2-50e_coco/dab-detr_r50_8xb2-50e_coco_20221122_120837-c1035c8c.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/dab_detr/dab-detr_r50_8xb2-50e_coco/dab-detr_r50_8xb2-50e_coco_20221122_120837.log.json) |

```latex
@inproceedings{
  liu2022dabdetr,
  title={{DAB}-{DETR}: Dynamic Anchor Boxes are Better Queries for {DETR}},

```latex
  author={Shilong Liu and Feng Li and Hao Zhang and Xiao Yang and Xianbiao Qi and Hang Su and Jun Zhu and Lei Zhang},
  booktitle={International Conference on Learning Representations},
  year={2022},
  url={https://openreview.net/forum?id=oMI9PjOb9Jl}
}
```

# HRNet

> [Deep High-Resolution Representation Learning for Human Pose Estimation](https://arxiv.org/abs/1902.09212)

This is an official pytorch implementation of Deep High-Resolution Representation Learning for Human Pose Estimation. In this work, we are interested in the human pose estimation problem with a focus on learning reliable high-resolution representations. Most existing methods recover high-resolution representations from low-resolution representations produced by a high-to-low resolution network. Instead, our proposed network maintains high-resolution representations through the whole process. We start from a high-resolution subnetwork as the first stage, gradually add high-to-low resolution subnetworks one by one to form more stages, and connect the mutli-resolution subnetworks in parallel. We conduct repeated multi-scale fusions such that each of the high-to-low resolution representations receives information from other parallel representations over and over, leading to rich high-resolution representations. As a result, the predicted keypoint heatmap is potentially more accurate and spatially more precise. We empirically demonstrate the effectiveness of our network through the superior pose estimation results over two benchmark datasets: the COCO keypoint detection dataset and the MPII Human Pose dataset.

High-resolution representation learning plays an essential role in many vision problems, e.g., pose estimation and semantic segmentation. The high-resolution network (HRNet), recently developed for human pose estimation, maintains high-resolution representations through the whole process by connecting high-to-low resolution convolutions in parallel and produces strong high-resolution representations by repeatedly conducting fusions across parallel convolutions.

In this paper, we conduct a further study on high-resolution representations by introducing a simple yet effective modification and apply it to a wide range of vision tasks. We augment the high-resolution representation by aggregating the (upsampled) representations from all the parallel convolutions rather than only the representation from the high-resolution convolution as done in HRNet. This simple modification leads to stronger representations, evidenced by superior results. We show top results in semantic segmentation on Cityscapes, LIP, and PASCAL Context, and facial landmark detection on AFLW, COFW, 300W, and WFLW. In addition, we build a multi-level representation from the high-resolution representation and apply it to the Faster R-CNN object detection framework and the extended frameworks. The proposed approach achieves superior results to existing single-model networks on COCO object detection.

<img src="https://user-images.githubusercontent.com/40661020/143892740-a4e9743e-a323-4ace-8025-50e251ef43ff.png"/>
</div>

### Faster R-CNN

|   Backbone   |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                     Config                      |                                                                                                                                                         Download                                                                                                                                                         |

| :----------: | :-----: | :-----: | :------: | :------------: | :----: | :---------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| HRNetV2p-W18 | pytorch |   1x    |   6.6    |      13.4      |  36.9  | [config](./faster-rcnn_hrnetv2p-w18-1x_coco.py) |    [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/faster_rcnn_hrnetv2p_w18_1x_coco/faster_rcnn_hrnetv2p_w18_1x_coco_20200130-56651a6d.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/faster_rcnn_hrnetv2p_w18_1x_coco/faster_rcnn_hrnetv2p_w18_1x_coco_20200130_211246.log.json)     |

| HRNetV2p-W18 | pytorch |   2x    |   6.6    |       -        |  38.9  | [config](./faster-rcnn_hrnetv2p-w18-2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/faster_rcnn_hrnetv2p_w18_2x_coco/faster_rcnn_hrnetv2p_w18_2x_coco_20200702_085731-a4ec0611.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/faster_rcnn_hrnetv2p_w18_2x_coco/faster_rcnn_hrnetv2p_w18_2x_coco_20200702_085731.log.json) |
| HRNetV2p-W32 | pytorch |   1x    |   9.0    |      12.4      |  40.2  | [config](./faster-rcnn_hrnetv2p-w32-1x_coco.py) |    [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/faster_rcnn_hrnetv2p_w32_1x_coco/faster_rcnn_hrnetv2p_w32_1x_coco_20200130-6e286425.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/faster_rcnn_hrnetv2p_w32_1x_coco/faster_rcnn_hrnetv2p_w32_1x_coco_20200130_204442.log.json)     |

| HRNetV2p-W32 | pytorch |   2x    |   9.0    |       -        |  41.4  | [config](./faster-rcnn_hrnetv2p-w32_2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/faster_rcnn_hrnetv2p_w32_2x_coco/faster_rcnn_hrnetv2p_w32_2x_coco_20200529_015927-976a9c15.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/faster_rcnn_hrnetv2p_w32_2x_coco/faster_rcnn_hrnetv2p_w32_2x_coco_20200529_015927.log.json) |
| HRNetV2p-W40 | pytorch |   1x    |   10.4   |      10.5      |  41.2  | [config](./faster-rcnn_hrnetv2p-w40-1x_coco.py) |    [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/faster_rcnn_hrnetv2p_w40_1x_coco/faster_rcnn_hrnetv2p_w40_1x_coco_20200210-95c1f5ce.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/faster_rcnn_hrnetv2p_w40_1x_coco/faster_rcnn_hrnetv2p_w40_1x_coco_20200210_125315.log.json)     |

| HRNetV2p-W40 | pytorch |   2x    |   10.4   |       -        |  42.1  | [config](./faster-rcnn_hrnetv2p-w40_2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/faster_rcnn_hrnetv2p_w40_2x_coco/faster_rcnn_hrnetv2p_w40_2x_coco_20200512_161033-0f236ef4.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/faster_rcnn_hrnetv2p_w40_2x_coco/faster_rcnn_hrnetv2p_w40_2x_coco_20200512_161033.log.json) |

### Mask R-CNN

|   Backbone   |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                    Config                     |                                                                                                                                                     Download                                                                                                                                                     |

| :----------: | :-----: | :-----: | :------: | :------------: | :----: | :-----: | :-------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| HRNetV2p-W18 | pytorch |   1x    |   7.0    |      11.7      |  37.7  |  34.2   | [config](./mask-rcnn_hrnetv2p-w18-1x_coco.py) |    [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/mask_rcnn_hrnetv2p_w18_1x_coco/mask_rcnn_hrnetv2p_w18_1x_coco_20200205-1c3d78ed.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/mask_rcnn_hrnetv2p_w18_1x_coco/mask_rcnn_hrnetv2p_w18_1x_coco_20200205_232523.log.json)     |

| HRNetV2p-W18 | pytorch |   2x    |   7.0    |       -        |  39.8  |  36.0   | [config](./mask-rcnn_hrnetv2p-w18-2x_coco.py) |    [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/mask_rcnn_hrnetv2p_w18_2x_coco/mask_rcnn_hrnetv2p_w18_2x_coco_20200212-b3c825b1.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/mask_rcnn_hrnetv2p_w18_2x_coco/mask_rcnn_hrnetv2p_w18_2x_coco_20200212_134222.log.json)     |
| HRNetV2p-W32 | pytorch |   1x    |   9.4    |      11.3      |  41.2  |  37.1   | [config](./mask-rcnn_hrnetv2p-w32-1x_coco.py) |    [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/mask_rcnn_hrnetv2p_w32_1x_coco/mask_rcnn_hrnetv2p_w32_1x_coco_20200207-b29f616e.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/mask_rcnn_hrnetv2p_w32_1x_coco/mask_rcnn_hrnetv2p_w32_1x_coco_20200207_055017.log.json)     |

| HRNetV2p-W32 | pytorch |   2x    |   9.4    |       -        |  42.5  |  37.8   | [config](./mask-rcnn_hrnetv2p-w32-2x_coco.py) |    [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/mask_rcnn_hrnetv2p_w32_2x_coco/mask_rcnn_hrnetv2p_w32_2x_coco_20200213-45b75b4d.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/mask_rcnn_hrnetv2p_w32_2x_coco/mask_rcnn_hrnetv2p_w32_2x_coco_20200213_150518.log.json)     |
| HRNetV2p-W40 | pytorch |   1x    |   10.9   |                |  42.1  |  37.5   | [config](./mask-rcnn_hrnetv2p-w40_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/mask_rcnn_hrnetv2p_w40_1x_coco/mask_rcnn_hrnetv2p_w40_1x_coco_20200511_015646-66738b35.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/mask_rcnn_hrnetv2p_w40_1x_coco/mask_rcnn_hrnetv2p_w40_1x_coco_20200511_015646.log.json) |

| HRNetV2p-W40 | pytorch |   2x    |   10.9   |                |  42.8  |  38.2   | [config](./mask-rcnn_hrnetv2p-w40-2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/mask_rcnn_hrnetv2p_w40_2x_coco/mask_rcnn_hrnetv2p_w40_2x_coco_20200512_163732-aed5e4ab.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/mask_rcnn_hrnetv2p_w40_2x_coco/mask_rcnn_hrnetv2p_w40_2x_coco_20200512_163732.log.json) |

### Cascade R-CNN

|   Backbone   |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                      Config                       |                                                                                                                                                             Download                                                                                                                                                             |

| :----------: | :-----: | :-----: | :------: | :------------: | :----: | :-----------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| HRNetV2p-W18 | pytorch |   20e   |   7.0    |      11.0      |  41.2  | [config](./cascade-rcnn_hrnetv2p-w18-20e_coco.py) |    [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/cascade_rcnn_hrnetv2p_w18_20e_coco/cascade_rcnn_hrnetv2p_w18_20e_coco_20200210-434be9d7.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/cascade_rcnn_hrnetv2p_w18_20e_coco/cascade_rcnn_hrnetv2p_w18_20e_coco_20200210_105632.log.json)     |

| HRNetV2p-W32 | pytorch |   20e   |   9.4    |      11.0      |  43.3  | [config](./cascade-rcnn_hrnetv2p-w32-20e_coco.py) |    [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/cascade_rcnn_hrnetv2p_w32_20e_coco/cascade_rcnn_hrnetv2p_w32_20e_coco_20200208-928455a4.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/cascade_rcnn_hrnetv2p_w32_20e_coco/cascade_rcnn_hrnetv2p_w32_20e_coco_20200208_160511.log.json)     |
| HRNetV2p-W40 | pytorch |   20e   |   10.8   |                |  43.8  | [config](./cascade-rcnn_hrnetv2p-w40-20e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/cascade_rcnn_hrnetv2p_w40_20e_coco/cascade_rcnn_hrnetv2p_w40_20e_coco_20200512_161112-75e47b04.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/cascade_rcnn_hrnetv2p_w40_20e_coco/cascade_rcnn_hrnetv2p_w40_20e_coco_20200512_161112.log.json) |

### Cascade Mask R-CNN

|   Backbone   |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                         Config                         |                                                                                                                                                                       Download                                                                                                                                                                       |
| :----------: | :-----: | :-----: | :------: | :------------: | :----: | :-----: | :----------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

| HRNetV2p-W18 | pytorch |   20e   |   8.5    |      8.5       |  41.6  |  36.4   | [config](./cascade-mask-rcnn_hrnetv2p-w18_20e_coco.py) |    [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/cascade_mask_rcnn_hrnetv2p_w18_20e_coco/cascade_mask_rcnn_hrnetv2p_w18_20e_coco_20200210-b543cd2b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/cascade_mask_rcnn_hrnetv2p_w18_20e_coco/cascade_mask_rcnn_hrnetv2p_w18_20e_coco_20200210_093149.log.json)     |
| HRNetV2p-W32 | pytorch |   20e   |          |      8.3       |  44.3  |  38.6   | [config](./cascade-mask-rcnn_hrnetv2p-w32_20e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/cascade_mask_rcnn_hrnetv2p_w32_20e_coco/cascade_mask_rcnn_hrnetv2p_w32_20e_coco_20200512_154043-39d9cf7b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/cascade_mask_rcnn_hrnetv2p_w32_20e_coco/cascade_mask_rcnn_hrnetv2p_w32_20e_coco_20200512_154043.log.json) |

| HRNetV2p-W40 | pytorch |   20e   |   12.5   |                |  45.1  |  39.3   | [config](./cascade-mask-rcnn_hrnetv2p-w40-20e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/cascade_mask_rcnn_hrnetv2p_w40_20e_coco/cascade_mask_rcnn_hrnetv2p_w40_20e_coco_20200527_204922-969c4610.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/cascade_mask_rcnn_hrnetv2p_w40_20e_coco/cascade_mask_rcnn_hrnetv2p_w40_20e_coco_20200527_204922.log.json) |

### Hybrid Task Cascade (HTC)

|   Backbone   |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                  Config                  |                                                                                                                                           Download                                                                                                                                           |

| :----------: | :-----: | :-----: | :------: | :------------: | :----: | :-----: | :--------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| HRNetV2p-W18 | pytorch |   20e   |   10.8   |      4.7       |  42.8  |  37.9   | [config](./htc_hrnetv2p-w18_20e_coco.py) |    [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/htc_hrnetv2p_w18_20e_coco/htc_hrnetv2p_w18_20e_coco_20200210-b266988c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/htc_hrnetv2p_w18_20e_coco/htc_hrnetv2p_w18_20e_coco_20200210_182735.log.json)     |

| HRNetV2p-W32 | pytorch |   20e   |   13.1   |      4.9       |  45.4  |  39.9   | [config](./htc_hrnetv2p-w32_20e_coco.py) |    [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/htc_hrnetv2p_w32_20e_coco/htc_hrnetv2p_w32_20e_coco_20200207-7639fa12.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/htc_hrnetv2p_w32_20e_coco/htc_hrnetv2p_w32_20e_coco_20200207_193153.log.json)     |
| HRNetV2p-W40 | pytorch |   20e   |   14.6   |                |  46.4  |  40.8   | [config](./htc_hrnetv2p-w40_20e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/htc_hrnetv2p_w40_20e_coco/htc_hrnetv2p_w40_20e_coco_20200529_183411-417c4d5b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/htc_hrnetv2p_w40_20e_coco/htc_hrnetv2p_w40_20e_coco_20200529_183411.log.json) |

### FCOS

### FCOS

|   Backbone   |  Style  | GN  | MS train | Lr schd | Mem (GB) | Inf time (fps) | box AP |                              Config                              |                                                                                                                                                                                                   Download                                                                                                                                                                                                   |

| HRNetV2p-W18 | pytorch |  Y  |    N     |   1x    |   13.0   |      12.9      |  35.3  |      [config](./fcos_hrnetv2p-w18-gn-head_4xb4-1x_coco.py)       |                                 [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/fcos_hrnetv2p_w18_gn-head_4x4_1x_coco/fcos_hrnetv2p_w18_gn-head_4x4_1x_coco_20201212_100710-4ad151de.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/fcos_hrnetv2p_w18_gn-head_4x4_1x_coco/fcos_hrnetv2p_w18_gn-head_4x4_1x_coco_20201212_100710.log.json)                                 |

| HRNetV2p-W18 | pytorch |  Y  |    N     |   2x    |   13.0   |       -        |  38.2  |      [config](./fcos_hrnetv2p-w18-gn-head_4xb4-2x_coco.py)       |                                 [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/fcos_hrnetv2p_w18_gn-head_4x4_2x_coco/fcos_hrnetv2p_w18_gn-head_4x4_2x_coco_20201212_101110-5c575fa5.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/fcos_hrnetv2p_w18_gn-head_4x4_2x_coco/fcos_hrnetv2p_w18_gn-head_4x4_2x_coco_20201212_101110.log.json)                                 |

| HRNetV2p-W32 | pytorch |  Y  |    N     |   1x    |   17.5   |      12.9      |  39.5  |      [config](./fcos_hrnetv2p-w32-gn-head_4xb4-1x_coco.py)       |                                 [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/fcos_hrnetv2p_w32_gn-head_4x4_1x_coco/fcos_hrnetv2p_w32_gn-head_4x4_1x_coco_20201211_134730-cb8055c0.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/fcos_hrnetv2p_w32_gn-head_4x4_1x_coco/fcos_hrnetv2p_w32_gn-head_4x4_1x_coco_20201211_134730.log.json)                                 |

| HRNetV2p-W32 | pytorch |  Y  |    N     |   2x    |   17.5   |       -        |  40.8  |      [config](./fcos_hrnetv2p-w32-gn-head_4xb4-2x_coco.py)       |                                 [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/fcos_hrnetv2p_w32_gn-head_4x4_2x_coco/fcos_hrnetv2p_w32_gn-head_4x4_2x_coco_20201212_112133-77b6b9bb.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/fcos_hrnetv2p_w32_gn-head_4x4_2x_coco/fcos_hrnetv2p_w32_gn-head_4x4_2x_coco_20201212_112133.log.json)                                 |

| HRNetV2p-W18 | pytorch |  Y  |    Y     |   2x    |   13.0   |      12.9      |  38.3  | [config](./fcos_hrnetv2p-w18-gn-head_ms-640-800-4xb4-2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/fcos_hrnetv2p_w18_gn-head_mstrain_640-800_4x4_2x_coco/fcos_hrnetv2p_w18_gn-head_mstrain_640-800_4x4_2x_coco_20201212_111651-441e9d9f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/fcos_hrnetv2p_w18_gn-head_mstrain_640-800_4x4_2x_coco/fcos_hrnetv2p_w18_gn-head_mstrain_640-800_4x4_2x_coco_20201212_111651.log.json) |

| HRNetV2p-W32 | pytorch |  Y  |    Y     |   2x    |   17.5   |      12.4      |  41.9  | [config](./fcos_hrnetv2p-w32-gn-head_ms-640-800-4xb4-2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/fcos_hrnetv2p_w32_gn-head_mstrain_640-800_4x4_2x_coco/fcos_hrnetv2p_w32_gn-head_mstrain_640-800_4x4_2x_coco_20201212_090846-b6f2b49f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/fcos_hrnetv2p_w32_gn-head_mstrain_640-800_4x4_2x_coco/fcos_hrnetv2p_w32_gn-head_mstrain_640-800_4x4_2x_coco_20201212_090846.log.json) |

| HRNetV2p-W48 | pytorch |  Y  |    Y     |   2x    |   20.3   |      10.8      |  42.7  | [config](./fcos_hrnetv2p-w40-gn-head_ms-640-800-4xb4-2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/hrnet/fcos_hrnetv2p_w40_gn-head_mstrain_640-800_4x4_2x_coco/fcos_hrnetv2p_w40_gn-head_mstrain_640-800_4x4_2x_coco_20201212_124752-f22d2ce5.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/hrnet/fcos_hrnetv2p_w40_gn-head_mstrain_640-800_4x4_2x_coco/fcos_hrnetv2p_w40_gn-head_mstrain_640-800_4x4_2x_coco_20201212_124752.log.json) |

**Note:**

- The `28e` schedule in HTC indicates decreasing the lr at 24 and 27 epochs, with a total of 28 epochs.
- HRNetV2 ImageNet pretrained models are in [HRNets for Image Classification](https://github.com/HRNet/HRNet-Image-Classification).

```latex
@inproceedings{SunXLW19,
  title={Deep High-Resolution Representation Learning for Human Pose Estimation},
  author={Ke Sun and Bin Xiao and Dong Liu and Jingdong Wang},

```latex
  year={2019}
}

@article{SunZJCXLMWLW19,
  title={High-Resolution Representations for Labeling Pixels and Regions},
  author={Ke Sun and Yang Zhao and Borui Jiang and Tianheng Cheng and Bin Xiao
  and Dong Liu and Yadong Mu and Xinggang Wang and Wenyu Liu and Jingdong Wang},
  journal   = {CoRR},
  volume    = {abs/1904.04514},
}
```

## Demo

We provide a demo showing what our mmdeploy can do for general model deployment.

In `demo_rewrite.py`, a resnet18 model from `torchvision` is rewritten through mmdeploy tool. In our rewritten model, the forward function of resnet gets modified to only down sample the original input to 4x. Original onnx model of resnet18 and its rewritten are visualized through [netron](https://netron.app/).

### Prerequisite

Before we run `demp_rewrite.py`, we need to install `pyppeteer` through:

```
pip install pyppeteer
```

### Demo results

The original resnet18 model and its modified one are visualized as follows. The left model is the original resnet18 while the right model is exported after rewritten.

|      Original resnet18      |       Rewritten model        |
| :-------------------------: | :--------------------------: |
| ![](resources/original.png) | ![](resources/rewritten.png) |

# 第五章：ONNX 模型的修改与调试

在前两期教程中，我们学习了 PyTorch 模型转 ONNX 模型的方法，了解了如何在原生算子表达能力不足时，为 PyTorch 或 ONNX 自定义算子。一直以来，我们都是通过 PyTorch 来导出 ONNX 模型的，基本没有单独探究过 ONNX 模型的构造知识。

不知道大家会不会有这样一些疑问：ONNX 模型在底层是用什么格式存储的？如何不依赖深度学习框架，只用 ONNX 的 API 来构造一个 ONNX 模型？如果没有源代码，只有一个 ONNX 模型，该如何对这个模型进行调试？这篇教程可以解答大家的这些问题。

在这期教程里，我们将围绕 ONNX 这一套神经网络定义标准本身，探究 ONNX 模型的构造、读取、子模型提取、调试。首先，我们会学习 ONNX 的底层表示方式。之后，我们会用 ONNX API 构造和读取模型。最后，我们会利用 ONNX 提供的子模型提取功能，学习如何调试 ONNX 模型。

## ONNX 的底层实现

### ONNX 的存储格式

ONNX 在底层是用 **Protobuf** 定义的。Protobuf，全称 Protocol Buffer，是 Google 提出的一套表示和序列化数据的机制。使用 Protobuf 时，用户需要先写一份数据定义文件，再根据这份定义文件把数据存储进一份二进制文件。可以说，数据定义文件就是数据类，二进制文件就是数据类的实例。
这里给出一个 Protobuf 数据定义文件的例子：

```protobuf
message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;
}
```

这段定义表示在 `Person` 这种数据类型中，必须包含 `name`、`id` 这两个字段，选择性包含 `email` 字段。根据这份定义文件，用户就可以选择一种编程语言，定义一个含有成员变量 `name`、`id`、`email` 的 `Person` 类，把这个类的某个实例用 Protobuf 存储成二进制文件；反之，用户也可以用二进制文件和对应的数据定义文件，读取出一个 `Person` 类的实例。

而对于 ONNX ，它的 Protobuf 数据定义文件在其[开源库](https://github.com/onnx/onnx/tree/main/onnx)中，这些文件定义了神经网络中模型、节点、张量的数据类型规范；而数据定义文件对应的二进制文件就是我们熟悉的“.onnx"文件，每一个 ".onnx" 文件按照数据定义规范，存储了一个神经网络的所有相关数据。直接用 Protobuf 生成 ONNX 模型还是比较麻烦的。幸运的是，ONNX 提供了很多实用 API，我们可以在完全不了解 Protobuf 的前提下，构造和读取 ONNX 模型。

### ONNX 的结构定义

在用 API 对 ONNX 模型进行操作之前，我们还需要先了解一下 ONNX 的结构定义规则，学习一下 ONNX 在 Protobuf 定义文件里是怎样描述一个神经网络的。

回想一下，神经网络本质上是一个计算图。计算图的节点是算子，边是参与运算的张量。而通过可视化 ONNX 模型，我们知道 ONNX 记录了所有算子节点的属性信息，并把参与运算的张量信息存储在算子节点的输入输出信息中。事实上，ONNX 模型的结构可以用类图大致表示如下：

![](https://user-images.githubusercontent.com/47652064/170020689-9a069a63-a4b7-44c0-8833-59e07c52fd5e.jpg)

如图所示，一个 ONNX 模型可以用 `ModelProto` 类表示。`ModelProto` 包含了版本、创建者等日志信息，还包含了存储计算图结构的 `graph`。`GraphProto` 类则由输入张量信息、输出张量信息、节点信息组成。张量信息 `ValueInfoProto` 类包括张量名、基本数据类型、形状。节点信息 `NodeProto` 类包含了算子名、算子输入张量名、算子输出张量名。
让我们来看一个具体的例子。假如我们有一个描述 `output=a*x+b` 的 ONNX 模型 `model`，用 `print(model)` 可以输出以下内容：

```python
ir_version: 8
graph {
  node {
    input: "a"
    input: "x"
    output: "c"

```python
graph {
  node {
    input: "a"
    input: "x"
    op_type: "Mul"
  }
  node {
    input: "c"
    input: "b"
    output: "output"
    op_type: "Add"
  }
  name: "linear_func"
  input {
    name: "a"
    type {
      tensor_type {
        elem_type: 1
        shape {
          dim {dim_value: 10}
        }
      }
    }
  }
  input {
    name: "x"
    type {
        shape {
        }
      }
    }
  }
  input {
    name: "b"
    type {
        shape {
        }
      }
    }
  }
  output {
    name: "output"
    type {
        shape {
          dim { dim_value: 10}

}
      }
    }
  }
  output {
    type {
        shape {
        }
      }
    }
  }
}
opset_import {version: 15}
```

对应上文中的类图，这个模型的信息由 `ir_version`，`opset_import` 等全局信息和 `graph` 图信息组成。而 `graph` 包含一个乘法节点、一个加法节点、三个输入张量 `a, x, b` 以及一个输出张量 `output`。在下一节里，我们会用 API 构造出这个模型，并输出这段结果。

## 读写 ONNX 模型

### 构造 ONNX 模型

在上一小节中，我们知道了 ONNX 模型是按以下的结构组织起来的：

- ModelProto
  - GraphProto
    - NodeProto
    - ValueInfoProto

现在，让我们抛开 PyTorch，尝试完全用 ONNX 的 Python API 构造一个描述线性函数 `output=a*x+b` 的 ONNX 模型。我们将根据上面的结构，自底向上地构造这个模型。

首先，我们可以用 `helper.make_tensor_value_info` 构造出一个描述张量信息的 `ValueInfoProto` 对象。如前面的类图所示，我们要传入张量名、张量的基本数据类型、张量形状这三个信息。在 ONNX 中，不管是输入张量还是输出张量，它们的表示方式都是一样的。因此，这里我们用类似的方式为三个输入 `a, x, b` 和一个输出 `output` 构造 `ValueInfoProto` 对象。如下面的代码所示：

```python
import onnx
from onnx import helper
from onnx import TensorProto

```python

a = helper.make_tensor_value_info('a', TensorProto.FLOAT, [10, 10])
x = helper.make_tensor_value_info('x', TensorProto.FLOAT, [10, 10])
b = helper.make_tensor_value_info('b', TensorProto.FLOAT, [10, 10])
output = helper.make_tensor_value_info('output', TensorProto.FLOAT, [10, 10])
```

之后，我们要构造算子节点信息 `NodeProto`，这可以通过在 `helper.make_node` 中传入算子类型、输入张量名、输出张量名这三个信息来实现。我们这里先构造了描述 `c=a*x` 的乘法节点，再构造了 `output=c+b` 的加法节点。如下面的代码所示：

```python
mul = helper.make_node('Mul', ['a', 'x'], ['c'])
add = helper.make_node('Add', ['c', 'b'], ['output'])
```

在计算机中，图一般是用一个节点集和一个边集表示的。而 ONNX 巧妙地把边的信息保存在了节点信息里，省去了保存边集的步骤。在 ONNX 中，如果某节点的输入名和之前某节点的输出名相同，就默认这两个节点是相连的。如上面的例子所示：`Mul` 节点定义了输出 `c`，`Add` 节点定义了输入 `c`，则 `Mul` 节点和 `Add` 节点是相连的。

正是因为有这种边的隐式定义规则，所以 ONNX 对节点的输入有一定的要求：一个节点的输入，要么是整个模型的输入，要么是之前某个节点的输出。如果我们把 `a, x, b` 中的某个输入节点从计算图中拿出（这个操作会在之后的代码中介绍），或者把 `Mul` 的输出从 `c` 改成 `d`，则最终的 ONNX 模型都是不满足标准的。

> 一个不满足标准的 ONNX 模型可能无法被推理引擎正确识别。ONNX 提供了 API `onnx.checker.check_model` 来判断一个 ONNX 模型是否满足标准。

接下来，我们用 `helper.make_graph` 来构造计算图 `GraphProto`。`helper.make_graph` 函数需要传入节点、图名称、输入张量信息、输出张量信息这 4 个参数。如下面的代码所示，我们把之前构造出来的 `NodeProto` 对象和 `ValueInfoProto` 对象按照顺序传入即可。

```python
graph = helper.make_graph([mul, add], 'linear_func', [a, x, b], [output])
```

这里 `make_graph` 的节点参数有一个要求：计算图的节点必须以拓扑序给出。

> 拓扑序是与有向图的相关的数学概念。如果按拓扑序遍历所有节点的话，能保证每个节点的输入都能在之前节点的输出里找到（对于 ONNX 模型，我们把计算图的输入张量也看成“之前的输出”）。

如果对这个概念不熟也没有关系，我们以刚刚构造出来的这个计算图为研究对象，通过下图展示的两个例子来直观理解拓扑序。

![](https://user-images.githubusercontent.com/47652064/170644483-160313b4-b000-4ad1-85b5-816278c7df80.png)

这里我们只关注 `Mul` 和 `Add` 节点以及它们之间的边 `c`。在情况 1 中：如果我们的节点以 `[Mul, Add]` 顺序给出，那么遍历到 `Add` 时，它的输入 `c` 可以在之前的 `Mul` 的输出中找到。但是，如情况 2 所示：如果我们的节点以 `[Add, Mul]` 的顺序给出，那么 `Add` 就找不到输入边，计算图也无法成功构造出来了。这里的 `[Mul, Add]` 就是符合有向图的拓扑序的，而 `[Add, Mul]` 则不满足。

最后，我们用 `helper.make_model` 把计算图 `GraphProto` 封装进模型 `ModelProto` 里，一个 ONNX 模型就构造完成了。`make_model` 函数中还可以添加模型制作者、版本等信息，为了简单起见，我们没有添加额外的信息。如下面的代码所示：

```python
model = helper.make_model(graph)
```

构造完模型之后，我们用下面这三行代码来检查模型正确性、把模型以文本形式输出、存储到一个 ".onnx" 文件里。这里用 `onnx.checker.check_model` 来检查模型是否满足 ONNX 标准是必要的，因为无论模型是否满足标准，ONNX 都允许我们用 onnx.save 存储模型。我们肯定不希望生成一个不满足标准的模型。

```python
onnx.checker.check_model(model)
print(model)
onnx.save(model, 'linear_func.onnx')
```

成功执行这些代码的话，程序会以文本格式输出模型的信息，其内容应该和我们在上一节展示的输出一样。

整理一下，用 ONNX Python API 构造模型的代码如下：

```python

```

```python

# input and output

# Mul

# Add

# graph and model

# save model
```

老规矩，我们可以用 ONNX Runtime 运行模型，来看看模型是否正确：

```python
import onnxruntime

```

```python

sess = onnxruntime.InferenceSession('linear_func.onnx')
a = np.random.rand(10, 10).astype(np.float32)
b = np.random.rand(10, 10).astype(np.float32)
x = np.random.rand(10, 10).astype(np.float32)

output = sess.run(['output'], {'a': a, 'b': b, 'x': x})[0]

assert np.allclose(output, a * x + b)
```

一切顺利的话，这段代码不会有任何报错信息。这说明我们的模型等价于执行 a * x + b 这个计算。

### 读取并修改 ONNX 模型

通过用 API 构造 ONNX 模型，我们已经彻底搞懂了 ONNX 由哪些模块组成。现在，让我们看看该如何读取现有的".onnx"文件并从中提取模型信息。

首先，我们可以用下面的代码读取一个 ONNX 模型：

```python
model = onnx.load('linear_func.onnx')
```

之前在输出模型时，我们传给 `onnx.save` 的是一个 `ModelProto` 的对象。同理，用上面的 `onnx.load` 读取 ONNX 模型时，我们收获的也是一个 `ModelProto` 的对象。输出这个对象后，我们应该得到和之前完全相同的输出。
接下来，我们来看看怎么把图 `GraphProto`、节点 `NodeProto`、张量信息 `ValueInfoProto` 读取出来：

```python
graph = model.graph
node = graph.node

```python
input = graph.input
output = graph.output
print(node)
print(input)
print(output)
```

使用如上这些代码，我们可以分别访问模型的图、节点、张量信息。这里大家或许会有疑问：该怎样找出 `graph.node,graph.input` 中 `node, input` 这些属性名称呢？其实，属性的名称就写在每个对象的输出里。我们以 `print(node)` 的输出为例：

```python
[input: "a"
input: "x"
output: "c"
op_type: "Mul"
, input: "c"
input: "b"
output: "output"
op_type: "Add"
]
```

在这段输出中，我们能看出 `node` 其实就是一个列表，列表中的对象有属性 `input, output, op_type`（这里 `input` 也是一个列表，它包含的两个元素都显示出来了）。我们可以用下面的代码来获取 `node` 里第一个节点 `Mul` 的属性：

```python
node_0 = node[0]
node_0_inputs = node_0.input
node_0_outputs = node_0.output
input_0 = node_0_inputs[0]
input_1 = node_0_inputs[1]
output = node_0_outputs[0]
op_type = node_0.op_type

print(input_0)
print(input_1)
print(op_type)

# Output
"""
a
x
c
Mul
"""
```

当我们想知道 ONNX 模型某数据对象有哪些属性时，我们不必去翻 ONNX 文档，只需要先把数据对象输出一下，然后在输出结果找出属性名即可。

# Output
"""
a
x
c
Mul
"""
```

读取完 ONNX 模型的信息后，修改 ONNX 模型就是一件很轻松的事了。我们既可以按照上一小节的模型构造方法，新建节点和张量信息，与原有模型组合成一个新的模型，也可以在不违反 ONNX 规范的前提下直接修改某个数据对象的属性。

这里我们来看一个直接修改模型属性的例子：

```python

node = model.graph.node
node[1].op_type = 'Sub'

onnx.save(model, 'linear_func_2.onnx')
```

在读入之前的 `linear_func.onnx` 模型后，我们可以直接修改第二个节点的类型 `node[1].op_type`，把加法变成减法。这样，我们的模型描述的是 `a * x - b` 这个线性函数。大家感兴趣的话，可以用 ONNX Runtime 运行新模型 `linear_func_2.onnx`，来验证一下它和 `a * x - b` 是否等价。

## 调试 ONNX 模型

在实际部署中，如果用深度学习框架导出的 ONNX 模型出了问题，一般要通过修改框架的代码来解决，而不会从 ONNX 入手，我们把 ONNX 模型当成一个不可修改的黑盒看待。
现在，我们已经深入学习了 ONNX 的原理，可以尝试对 ONNX 模型本身进行调试了。在这一节里，让我们看看该如何巧妙利用 ONNX 提供的子模型提取功能，对 ONNX 模型进行调试。

### 子模型提取

### 子模型提取

ONNX 官方为开发者提供了子模型提取（extract）的功能。子模型提取，顾名思义，就是从一个给定的 ONNX 模型中，拿出一个子模型。这个子模型的节点集、边集都是原模型中对应集合的子集。让我们来用 PyTorch 导出一个复杂一点的 ONNX 模型，并在它的基础上执行提取操作：

```python

class Model(torch.nn.Module):

    def __init__(self):
        self.convs1 = torch.nn.Sequential(torch.nn.Conv2d(3, 3, 3),
                                          torch.nn.Conv2d(3, 3, 3),
                                          torch.nn.Conv2d(3, 3, 3))
        self.convs2 = torch.nn.Sequential(torch.nn.Conv2d(3, 3, 3),
        self.convs3 = torch.nn.Sequential(torch.nn.Conv2d(3, 3, 3),
        self.convs4 = torch.nn.Sequential(torch.nn.Conv2d(3, 3, 3),

torch.nn.Conv2d(3, 3, 3))
        x = self.convs1(x)
        x1 = self.convs2(x)
        x2 = self.convs3(x)
        x = x1 + x2
        x = self.convs4(x)
        return x

model = Model()
input = torch.randn(1, 3, 20, 20)

torch.onnx.export(model, input, 'whole_model.onnx')
```

这个模型的可视化结果如下图所示（提取子模型需要输入边的序号，为了大家方面阅读，这幅图标出了之后要用到的边的序号）：

![](https://user-images.githubusercontent.com/47652064/170644578-bcaaa2aa-bdd4-4cb3-856b-c6d621273357.png)

> 在前面的章节中，我们学过，ONNX 的边用同名张量表示的。也就是说，这里的边序号，实际上是前一个节点的输出张量序号和后一个节点的输入张量序号。由于这个模型是用 PyTorch 导出的，这些张量序号都是 PyTorch 自动生成的。

接着，我们可以下面的代码提取出一个子模型：

```python

onnx.utils.extract_model('whole_model.onnx', 'partial_model.onnx', ['22'], ['28'])
```

子模型的可视化结果如下图所示：

```python

```

![](https://user-images.githubusercontent.com/47652064/170644616-42cd9d11-1525-49b2-b302-b96e985c5e79.png)

通过观察代码和输出图，应该不难猜出这段代码的作用是把原计算图从边 22 到边 28 的子图提取出来，并组成一个子模型。`onnx.utils.extract_model` 就是完成子模型提取的函数，它的参数分别是原模型路径、输出模型路径、子模型的输入边（输入张量）、子模型的输出边（输出张量）。

直观地来看，子模型提取就是把输入边到输出边之间的全部节点都取出来。那么，这个功能在使用上有什么限制呢？基于 `whole_model.onnx`, 我们来看一看三个子模型提取的示例。

#### 添加额外输出

我们在提取时新设定了一个输出张量，如下面的代码所示：

```python
onnx.utils.extract_model('whole_model.onnx', 'submodel_1.onnx', ['22'], ['27', '31'])
```

我们可以看到子模型会添加一条把张量输出的新边，如下图所示：

![](https://user-images.githubusercontent.com/47652064/170644722-d63156e5-cd74-4faa-ac0a-ce408be949eb.png)

#### 添加冗余输入

如果我们还是像开始一样提取边 22 到边 28 之间的子模型，但是多添加了一个输入 input.1，那么提取出的子模型会有一个冗余的输入 input.1，如下面的代码所示：

```python
onnx.utils.extract_model('whole_model.onnx', 'submodel_2.onnx', ['22', 'input.1'], ['28'])
```

```python
```

从下图中可以看出：无论给这个输入传入什么值，都不会影响子模型的输出。可以认为如果只用子模型的部分输入就能得到输出，那么那些”较早“的多出来的输入就是冗余的。

![](https://user-images.githubusercontent.com/47652064/170644751-c8100d04-585b-4f93-9ed0-7a77dca88c16.png)

#### 输入信息不足

这次，我们尝试提取的子模型输入是边 24，输出是边 28。如下面的代码和图所示：

```python
# Error
onnx.utils.extract_model('whole_model.onnx', 'submodel_3.onnx', ['24'], ['28'])
```

![](https://user-images.githubusercontent.com/47652064/170644773-627af9d0-8c3f-447c-9fbf-dc63a31c40ab.png)

从图中可以看出，想通过边 24 计算边 28 的结果，至少还需要输入边 26，或者更上面的边。仅凭借边 24 是无法计算出边 28 的结果的，因此这样提取子模型会报错。

通过上面几个使用示例，我们可以整理出子模型提取的实现原理：新建一个模型，把给定的输入和输出填入。之后把图的所有有向边反向，从输出边开始遍历节点，碰到输入边则停止，把这样遍历得到的节点做为子模型的节点。

如果还没有彻底弄懂这个提取原理，没关系，我们只要尽量保证在填写子模型的输入输出时，让输出恰好可以由输入决定即可。

### 输出 ONNX 中间节点的值

在使用 ONNX 模型时，最常见的一个需求是能够用推理引擎输出中间节点的值。这多见于深度学习框架模型和 ONNX 模型的精度对齐中，因为只要能够输出中间节点的值，就能定位到精度出现偏差的算子。我们来看看如何用子模型提取实现这一任务。

在刚刚的第一个子模型提取示例中，我们添加了一条原来模型中不存在的输出边。用同样的原理，我们可以在保持原有输入输出不变的同时，新增加一些输出，提取出一个能输出中间节点的”子模型“。例如：

```python
 onnx.utils.extract_model('whole_model.onnx', 'more_output_model.onnx', ['input.1'], ['31', '23', '25', '27'])
```

在这个子模型中，我们在保持原有的输入 `input.1`，输出 `31` 的同时，把其他几个边加入了输出中。如下图所示：

![](https://user-images.githubusercontent.com/47652064/170020845-6e1cb45b-962a-40ba-a17b-e47b0bdcd3bf.png)

这样，用 ONNX Runtime 运行 `more_output_model.onnx` 这个模型时，我们就能得到更多的输出了。
为了方便调试，我们还可以把原模型拆分成多个互不相交的子模型。这样，在每次调试时，可以只对原模型的部分子模块调试。比如：

```python
onnx.utils.extract_model('whole_model.onnx', 'debug_model_1.onnx', ['input.1'], ['23'])
onnx.utils.extract_model('whole_model.onnx', 'debug_model_2.onnx', ['23'], ['25'])

```python
onnx.utils.extract_model('whole_model.onnx', 'debug_model_3.onnx', ['23'], ['27'])
onnx.utils.extract_model('whole_model.onnx', 'debug_model_4.onnx', ['25', '27'], ['31'])
```

在这个例子中，我们把原来较为复杂的模型拆成了四个较为简单的子模型，如下图所示。在调试时，我们可以先调试顶层的子模型，确认顶层子模型无误后，把它的输出做为后面子模型的输入。

比如对于这些子模型，我们可以先调试第一个子模型，并存储输出 23。之后把张量 23 做为第二个和第三个子模型的输入，调试这两个模型。最后用同样方法调试第四个子模型。可以说，有了子模型提取功能，哪怕是面对一个庞大的模型，我们也能够从中提取出有问题的子模块，细致地只对这个子模块调试。

![](https://user-images.githubusercontent.com/47652064/170020865-e4d59a4f-7c57-4a12-b300-b7f5da0e1b80.png)

______________________________________________________________________

子模型提取固然是一个便利的 ONNX 调试工具。但是，在实际的情况中，我们一般是用 PyTorch 等框架导出 ONNX 模型。这里有两个问题：

1. 一旦 PyTorch 模型改变，ONNX 模型的边序号也会改变。这样每次提取同样的子模块时都要重新去 ONNX 模型里查序号，如此繁琐的调试方法是不会在实践中采用的。

2. 即使我们能保证 ONNX 的边序号不发生改变，我们也难以把 PyTorch 代码和 ONNX 节点对应起来——当模型结构变得十分复杂时，要识别 ONNX 中每个节点的含义是不可能的。

MMDeploy 为 PyTorch 模型添加了模型分块功能。使用这个功能，我们可以通过只修改 PyTorch 模型的实现代码来把原模型导出成多个互不相交的子 ONNX 模型。我们会在后续教程中对其介绍。

## 总结

在这篇教程中，我们抛开了 PyTorch，学习了 ONNX 模型本身的知识。老规矩，我们来总结一下这篇教程的知识点：

- ONNX 使用 Protobuf 定义规范和序列化模型。
- 一个 ONNX 模型主要由 `ModelProto`,`GraphProto`,`NodeProto`,`ValueInfoProto` 这几个数据类的对象组成。
- 使用 `onnx.helper.make_xxx`，我们可以构造 ONNX 模型的数据对象。
- `onnx.save()` 可以保存模型，`onnx.load()` 可以读取模型，`onnx.checker.check_model()` 可以检查模型是否符合规范。
- `onnx.utils.extract_model()` 可以从原模型中取出部分节点，和新定义的输入、输出边构成一个新的子模型。
- 利用子模型提取功能，我们可以输出原 ONNX 模型的中间结果，实现对 ONNX 模型的调试。

至此，我们对 ONNX 相关知识的学习就告一段落了。回顾一下，我们先学习了 PyTorch 转 ONNX 有关 API 的用法；接着，我们学习了如何用自定义算子解决 PyTorch 和 ONNX 表达能力不足的问题；最后我们单独学习了 ONNX 模型的调试方法。通过对 ONNX 由浅入深的学习，我们基本可以应对模型部署中和 ONNX 有关的绝大多数问题了。

如果大家想了解更多有关 ONNX API 的知识，可以去阅读 ONNX 的[官方 Python API 文档](https://github.com/onnx/onnx/blob/main/docs/PythonAPIOverview.md)。

# X-Decoder

> [X-Decoder: Generalized Decoding for Pixel, Image, and Language](https://arxiv.org/pdf/2212.11270.pdf)

We present X-Decoder, a generalized decoding model that can predict pixel-level segmentation and language tokens seamlessly. X-Decodert takes as input two types of queries: (i) generic non-semantic queries and (ii) semantic queries induced from text inputs, to decode different pixel-level and token-level outputs in the same semantic space. With such a novel design, X-Decoder is the first work that provides a unified way to support all types of image segmentation and a variety of vision-language (VL) tasks. Further, our design enables seamless interactions across tasks at different granularities and brings mutual benefits by learning a common and rich pixel-level visual-semantic understanding space, without any pseudo-labeling. After pretraining on a mixed set of a limited amount of segmentation data and millions of image-text pairs, X-Decoder exhibits strong transferability to a wide range of downstream tasks in both zero-shot and finetuning settings. Notably, it achieves (1) state-of-the-art results on open-vocabulary segmentation and referring segmentation on eight datasets; (2) better or competitive finetuned performance to other generalist and specialist models on segmentation and VL tasks; and (3) flexibility for efficient finetuning and novel task composition (e.g., referring captioning and image editing).

<img src="https://github.com/open-mmlab/mmdetection/assets/17425982/cb126615-9402-4c19-8ea9-133722d7519c" width="70%"/>
</div>

```shell
# if source
pip install -r requirements/multimodal.txt

# if wheel
mim install mmdet[multimodal]
```

## How to use it?

For convenience, you can download the weights to the `mmdetection` root dir

```shell
wget https://download.openmmlab.com/mmdetection/v3.0/xdecoder/xdecoder_focalt_last_novg.pt
wget https://download.openmmlab.com/mmdetection/v3.0/xdecoder/xdecoder_focalt_best_openseg.pt
```

The above two weights are directly copied from the official website without any modification. The specific source is https://github.com/microsoft/X-Decoder

For convenience of demonstration, please download [the folder](https://github.com/microsoft/X-Decoder/tree/main/images) and place it in the root directory of mmdetection.

**(1) Open Vocabulary Semantic Segmentation**

```shell
cd projects/XDecoder

```shell
python demo.py ../../images/animals.png configs/xdecoder-tiny_zeroshot_open-vocab-semseg_coco.py --weights ../../xdecoder_focalt_last_novg.pt --texts zebra.giraffe
```

<img src="https://github.com/open-mmlab/mmdetection/assets/17425982/c397c0ed-859a-4004-8725-78a591742bc8" width="70%"/>
</div>

**(2) Open Vocabulary Instance Segmentation**

```shell
python demo.py ../../images/owls.jpeg configs/xdecoder-tiny_zeroshot_open-vocab-instance_coco.py --weights ../../xdecoder_focalt_last_novg.pt --texts owl
```

<img src="https://github.com/open-mmlab/mmdetection/assets/17425982/494b0b1c-4a42-4019-97ae-d33ee68af3d2" width="70%"/>
</div>

**(3) Open Vocabulary Panoptic Segmentation**

```shell

</div>

```shell
python demo.py ../../images/street.jpg configs/xdecoder-tiny_zeroshot_open-vocab-panoptic_coco.py --weights ../../xdecoder_focalt_last_novg.pt  --text car.person --stuff-text tree.sky
```

<img src="https://github.com/open-mmlab/mmdetection/assets/17425982/9ad1e0f4-75ce-4e37-a5cc-83e0e8a722ed" width="70%"/>
</div>

**(4) Referring Expression Segmentation**

```shell
python demo.py ../../images/fruit.jpg configs/xdecoder-tiny_zeroshot_open-vocab-ref-seg_refcocog.py --weights ../../xdecoder_focalt_last_novg.pt  --text "The larger watermelon. The front white flower. White tea pot."
```

<img src="https://github.com/open-mmlab/mmdetection/assets/17425982/f3ecdb50-20f0-4dc4-aa9c-90995ae04893" width="70%"/>
</div>

**(5) Image Caption**

```shell

</div>

```shell
python demo.py ../../images/penguin.jpeg configs/xdecoder-tiny_zeroshot_caption_coco2014.py --weights ../../xdecoder_focalt_last_novg.pt
```

<img src="https://github.com/open-mmlab/mmdetection/assets/17425982/7690ab79-791e-4011-ab0c-01f46c4a3d80" width="70%"/>
</div>

**(6) Referring Expression Image Caption**

```shell
python demo.py ../../images/fruit.jpg configs/xdecoder-tiny_zeroshot_ref-caption.py --weights ../../xdecoder_focalt_last_novg.pt --text 'White tea pot'
```

<img src="https://github.com/open-mmlab/mmdetection/assets/17425982/bae2fdba-0172-4fc8-8ad1-73b54c64ec30" width="70%"/>
</div>

**(7) Text Image Region Retrieval**

```shell

</div>

```shell
python demo.py ../../images/coco configs/xdecoder-tiny_zeroshot_text-image-retrieval.py --weights ../../xdecoder_focalt_last_novg.pt --text 'pizza on the plate'
```

```text
The image that best matches the given text is ../../images/coco/000.jpg and probability is 0.998
```

<img src="https://github.com/open-mmlab/mmdetection/assets/17425982/479de6b2-88e7-41f0-8228-4b9a48f52954" width="70%"/>
</div>

We have also prepared a gradio program in the `projects/gradio_demo` directory, which you can run interactively all the inference supported by mmdetection in your browser.

## Models and results

### Semantic segmentation on ADE20K

Prepare your dataset according to the [docs](../../docs/en/user_guides/dataset_prepare.md#ade20k-2016-dataset-preparation).

**Test Command**

Since semantic segmentation is a pixel-level task, we don't need to use a threshold to filter out low-confidence predictions. So we set `model.test_cfg.use_thr_for_mc=False` in the test command.

```shell
./tools/dist_test.sh projects/XDecoder/configs/xdecoder-tiny_zeroshot_open-vocab-semseg_ade20k.py xdecoder_focalt_best_openseg.pt 8 --cfg-options model.test_cfg.use_thr_for_mc=False
```

| Model                             | mIoU  | mIOU(official) |                                Config                                |
| :-------------------------------- | :---: | :------------: | :------------------------------------------------------------------: |
| `xdecoder_focalt_best_openseg.pt` | 25.24 |     25.13      | [config](configs/xdecoder-tiny_zeroshot_open-vocab-semseg_ade20k.py) |

### Instance segmentation on ADE20K

```shell
./tools/dist_test.sh projects/XDecoder/configs/xdecoder-tiny_zeroshot_open-vocab-instance_ade20k.py xdecoder_focalt_best_openseg.pt 8
```

| Model                             | mIoU | mIOU(official) |                                 Config                                 |
| :-------------------------------- | :--: | :------------: | :--------------------------------------------------------------------: |
| `xdecoder_focalt_best_openseg.pt` | 10.1 |      10.1      | [config](configs/xdecoder-tiny_zeroshot_open-vocab-instance_ade20k.py) |

### Panoptic segmentation on ADE20K

```shell
./tools/dist_test.sh projects/XDecoder/configs/xdecoder-tiny_zeroshot_open-vocab-panoptic_ade20k.py xdecoder_focalt_best_openseg.pt 8
```

| Model                             | mIoU  | mIOU(official) |                                 Config                                 |
| :-------------------------------- | :---: | :------------: | :--------------------------------------------------------------------: |
| `xdecoder_focalt_best_openseg.pt` | 19.11 |     18.97      | [config](configs/xdecoder-tiny_zeroshot_open-vocab-panoptic_ade20k.py) |

### Semantic segmentation on COCO2017

Prepare your dataset according to the [docs](../../docs/en/user_guides/dataset_prepare.md#coco-semantic-dataset-preparation) of `(2) use panoptic dataset` part.

```shell
./tools/dist_test.sh projects/XDecoder/configs/xdecoder-tiny_zeroshot_open-vocab-semseg_coco.py xdecoder_focalt_last_novg.pt 8 --cfg-options model.test_cfg.use_thr_for_mc=False
```

| Model                                           | mIOU | mIOU(official) |                               Config                               |
| :---------------------------------------------- | :--: | :------------: | :----------------------------------------------------------------: |
| `xdecoder-tiny_zeroshot_open-vocab-semseg_coco` | 62.1 |      62.1      | [config](configs/xdecoder-tiny_zeroshot_open-vocab-semseg_coco.py) |

### Instance segmentation on COCO2017

Prepare your dataset according to the [docs](../../docs/en/user_guides/dataset_prepare.md#basic-detection-dataset-preparation).

```shell
./tools/dist_test.sh projects/XDecoder/configs/xdecoder-tiny_zeroshot_open-vocab-instance_coco.py xdecoder_focalt_last_novg.pt 8
```

| Model                                             | Mask mAP | Mask mAP(official) |                                Config                                |
| :------------------------------------------------ | :------: | :----------------: | :------------------------------------------------------------------: |
| `xdecoder-tiny_zeroshot_open-vocab-instance_coco` |   39.8   |        39.7        | [config](configs/xdecoder-tiny_zeroshot_open-vocab-instance_coco.py) |

### Panoptic segmentation on COCO2017

```shell
./tools/dist_test.sh projects/XDecoder/configs/xdecoder-tiny_zeroshot_open-vocab-panoptic_coco.py xdecoder_focalt_last_novg.pt 8
```

| Model                                             |  PQ   | PQ(official) |                                Config                                |
| :------------------------------------------------ | :---: | :----------: | :------------------------------------------------------------------: |
| `xdecoder-tiny_zeroshot_open-vocab-panoptic_coco` | 51.42 |    51.16     | [config](configs/xdecoder-tiny_zeroshot_open-vocab-panoptic_coco.py) |

### Referring segmentation on RefCOCO

Prepare your dataset according to the [docs](../../docs/en/user_guides/dataset_prepare.md#refcoco-dataset-preparation).

```shell
./tools/dist_test.sh  projects/XDecoder/configs/xdecoder-tiny_zeroshot_open-vocab-ref-seg_refcocog.py xdecoder_focalt_last_novg.pt 8  --cfg-options test_dataloader.dataset.split='val'
```

| Model                          |  text mode   |  cIoU   | cIOU(official) |                                 Config                                  |
| :----------------------------- | :----------: | :-----: | :------------: | :---------------------------------------------------------------------: |
| `xdecoder_focalt_last_novg.pt` | select first | 58.8415 |     57.85      | [config](configs/xdecoder-tiny_zeroshot_open-vocab-ref-seg_refcocog.py) |

| `xdecoder_focalt_last_novg.pt` |   original   | 60.0321 |       -        | [config](configs/xdecoder-tiny_zeroshot_open-vocab-ref-seg_refcocog.py) |
| `xdecoder_focalt_last_novg.pt` |    concat    | 60.3551 |       -        | [config](configs/xdecoder-tiny_zeroshot_open-vocab-ref-seg_refcocog.py) |

**Note:**

1. If you set the scale of `Resize` to (1024, 512), the result will be `57.69`.
2. `text mode` is the `RefCoCoDataset` parameter in MMDetection, it determines the texts loaded to the data list. It can be set to `select_first`, `original`, `concat` and `random`.
   - `select_first`: select the first text in the text list as the description to an instance.
   - `original`: use all texts in the text list as the description to an instance.
   - `concat`: concatenate all texts in the text list as the description to an instance.

- `original`: use all texts in the text list as the description to an instance.
   - `random`: randomly select one text in the text list as the description to an instance, usually used for training.

### Image Caption on COCO2014

Prepare your dataset according to the [docs](../../docs/en/user_guides/dataset_prepare.md#coco-caption-dataset-preparation).

Before testing, you need to install jdk 1.8, otherwise it will prompt that java does not exist during the evaluation process

```
./tools/dist_test.sh projects/XDecoder/configs/xdecoder-tiny_zeroshot_caption_coco2014.py xdecoder_focalt_last_novg.pt 8
```

| Model                                     | BLEU-4 | CIDER  |                            Config                            |
| :---------------------------------------- | :----: | :----: | :----------------------------------------------------------: |

| `xdecoder-tiny_zeroshot_caption_coco2014` | 35.26  | 116.81 | [config](configs/xdecoder-tiny_zeroshot_caption_coco2014.py) |

```latex
@article{zou2022xdecoder,
  author      = {Zou*, Xueyan and Dou*, Zi-Yi and Yang*, Jianwei and Gan, Zhe and Li, Linjie and Li, Chunyuan and Dai, Xiyang and Wang, Jianfeng and Yuan, Lu and Peng, Nanyun and Wang, Lijuan and Lee*, Yong Jae and Gao*, Jianfeng},
  title       = {Generalized Decoding for Pixel, Image and Language},
  publisher   = {arXiv},
  year        = {2022},
}
```

# Restful API

### Launch Service

```shell
python3 -m lmdeploy.serve.openai.api_server ./workspace 0.0.0.0 server_port --instance_num 32 --tp 1
```

Then, the user can open the swagger UI: `http://{server_ip}:{server_port}` for the detailed api usage.
We provide four restful api in total. Three of them are in OpenAI format. However, we recommend users try
our own api which provides more arguments for users to modify. The performance is comparatively better.

### python

Here is an example for our own api `generate`.

```python
import json
import requests
from typing import Iterable, List

def get_streaming_response(prompt: str,
                           api_url: str,
                           instance_id: int,
                           request_output_len: int,
                           stream: bool = True,
                           sequence_start: bool = True,
                           sequence_end: bool = True,

stream: bool = True,
                           ignore_eos: bool = False) -> Iterable[List[str]]:
    headers = {'User-Agent': 'Test Client'}
    pload = {
        'prompt': prompt,
        'stream': stream,
        'instance_id': instance_id,
        'request_output_len': request_output_len,
        'sequence_start': sequence_start,
        'sequence_end': sequence_end,
        'ignore_eos': ignore_eos
    }
    response = requests.post(
        api_url, headers=headers, json=pload, stream=stream)
    for chunk in response.iter_lines(
            chunk_size=8192, decode_unicode=False, delimiter=b'\0'):
        if chunk:
            data = json.loads(chunk.decode('utf-8'))
            output = data['text']
            tokens = data['tokens']
            yield output, tokens

for output, tokens in get_streaming_response(

output = data['text']

        "Hi, how are you?", "http://{server_ip}:{server_port}/generate", 0,
        512):
    print(output, end='')
```

### Java/Golang/Rust

May use [openapi-generator-cli](https://github.com/OpenAPITools/openapi-generator-cli) to convert `http://{server_ip}:{server_port}/openapi.json` to java/rust/golang client.
Here is an example:

```shell
$ docker run -it --rm -v ${PWD}:/local openapitools/openapi-generator-cli generate -i /local/openapi.json -g rust -o /local/rust

$ ls rust/*
rust/Cargo.toml  rust/git_push.sh  rust/README.md

rust/docs:
ChatCompletionRequest.md  EmbeddingsRequest.md  HttpValidationError.md  LocationInner.md  Prompt.md
DefaultApi.md             GenerateRequest.md    Input.md                Messages.md       ValidationError.md

rust/src:
apis  lib.rs  models
```

### cURL

rust/src:
```

### cURL

cURL is a tool for observing the output of the api.

List Models:

```bash
curl http://{server_ip}:{server_port}/v1/models
```

Generate:

```bash
curl http://{server_ip}:{server_port}/generate \
  -H "Content-Type: application/json" \
  -d '{
    "model": "internlm-chat-7b",
    "prompt": "Hello! Ho are you?",
    "sequence_start": true,
    "sequence_end": true
  }'
```

Chat Completions:

```bash
curl http://{server_ip}:{server_port}/v1/chat/completions \
  -d '{
    "messages": [{"role": "user", "content": "Hello! Ho are you?"}]
  }'
```

Embeddings:

```bash
curl http://{server_ip}:{server_port}/v1/embeddings \
  -d '{
    "input": "Hello world!"
  }'
```

### CLI client

```bash
  -d '{
  }'
```

There is a client script for restful api server.

```shell
# restful_api_url is what printed in api_server.py, e.g. http://localhost:23333
python -m lmdeploy.serve.openai.api_client restful_api_url
```

### webui

You can also test restful-api through webui.

```shell
# server_ip and server_port here are for gradio ui
# example: python -m lmdeploy.serve.gradio.app http://localhost:23333 localhost 6006 --restful_api True
python -m lmdeploy.serve.gradio.app restful_api_url server_ip --restful_api True
```

### FAQ

1. When user got `"finish_reason":"length"` which means the session is too long to be continued.
   Please add `"renew_session": true` into the next request.

```

### FAQ

2. When OOM appeared at the server side, please reduce the number of `instance_num` when lanching the service.

3. When the request with the same `instance_id` to `generate` got a empty return value and a negative `tokens`, please consider setting `sequence_start=false` for the second question and the same for the afterwards.

# Gliding Vertex

> [Gliding Vertex on the Horizontal Bounding Box for Multi-Oriented Object Detection](https://arxiv.org/pdf/1911.09358.pdf)

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/gv.png" width="800"/>
</div>

Object detection has recently experienced substantial progress. Yet, the widely adopted horizontal bounding box representation is not appropriate for ubiquitous oriented objects such as objects in aerial images and scene texts. In this paper, we propose a simple yet effective framework to detect multi-oriented objects. Instead of directly regressing the four vertices, we glide the vertex of the horizontal bounding box on each corresponding side to accurately describe a multi-oriented object. Specifically, We regress four length ratios characterizing the relative gliding offset on each corresponding side. This may facilitate the offset learning and avoid the confusion issue of sequential label points for oriented objects. To further remedy the confusion issue for nearly horizontal objects, we also introduce an obliquity factor based on area ratio between the object and its horizontal bounding box, guiding the selection of horizontal or oriented detection for each object. We add these five extra target variables to the regression head of rotated faster R-CNN, which requires ignorable extra computation time. Extensive experimental results demonstrate that without bells and whistles, the proposed method achieves superior performances on multiple multi-oriented object detection benchmarks including object detection in aerial images, scene text detection, pedestrian detection in fisheye images.

DOTA1.0

|         Backbone         |  mAP  | Angle | lr schd | Mem (GB) | Inf Time (fps) | Aug | Batch Size |                                                    Configs                                                     |                                                                                                                                                                              Download                                                                                                                                                                              |

| ResNet50 (1024,1024,200) | 73.40 | le90  |   1x    |   8.46   |      16.5      |  -  |     2      | [rotated-faster-rcnn-le90_r50_fpn_1x_dota](../rotated_faster_rcnn/rotated-faster-rcnn-le90_r50_fpn_1x_dota.py) | [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_faster_rcnn/rotated_faster_rcnn_r50_fpn_1x_dota_le90/rotated_faster_rcnn_r50_fpn_1x_dota_le90-0393aa5c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_faster_rcnn/rotated_faster_rcnn_r50_fpn_1x_dota_le90/rotated_faster_rcnn_r50_fpn_1x_dota_le90_20220131_082156.log.json) |

| ResNet50 (1024,1024,200) | 73.23 | le90  |   1x    |   8.45   |      16.4      |  -  |     2      |                [gliding-vertex-rbox_r50_fpn_1x_dota](./gliding-vertex-rbox_r50_fpn_1x_dota.py)                 |                [model](https://download.openmmlab.com/mmrotate/v0.1.0/gliding_vertex/gliding_vertex_r50_fpn_1x_dota_le90/gliding_vertex_r50_fpn_1x_dota_le90-12e7423c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/gliding_vertex/gliding_vertex_r50_fpn_1x_dota_le90/gliding_vertex_r50_fpn_1x_dota_le90_20220129_085529.log.json)                |

```
@article{RN57,
	author = {Xu, Yongchao and Fu, Mingtao and Wang, Qimeng and Wang, Yukang and Chen, Kai and Xia, Gui-Song and Bai, Xiang},
	title = {Gliding Vertex on the Horizontal Bounding Box for Multi-Oriented Object Detection},
	journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
	volume = {43},
	number = {4},
	pages = {1452-1459},
	ISSN = {0162-8828},

journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
	DOI = {10.1109/tpami.2020.2974745},
	year = {2021},
	type = {Journal Article}
}
```

# Sparse R-CNN

> [Sparse R-CNN: End-to-End Object Detection with Learnable Proposals](https://arxiv.org/abs/2011.12450)

We present Sparse R-CNN, a purely sparse method for object detection in images. Existing works on object detection heavily rely on dense object candidates, such as k anchor boxes pre-defined on all grids of image feature map of size H×W. In our method, however, a fixed sparse set of learned object proposals, total length of N, are provided to object recognition head to perform classification and location. By eliminating HWk (up to hundreds of thousands) hand-designed object candidates to N (e.g. 100) learnable proposals, Sparse R-CNN completely avoids all efforts related to object candidates design and many-to-one label assignment. More importantly, final predictions are directly output without non-maximum suppression post-procedure. Sparse R-CNN demonstrates accuracy, run-time and training convergence performance on par with the well-established detector baselines on the challenging COCO dataset, e.g., achieving 45.0 AP in standard 3× training schedule and running at 22 fps using ResNet-50 FPN model. We hope our work could inspire re-thinking the convention of dense prior in object detectors.

<img src="https://user-images.githubusercontent.com/40661020/143998489-8a5a687d-ceec-4590-8347-708e427e7dfe.png" height="300"/>
</div>

|    Model     | Backbone  |  Style  | Lr schd | Number of Proposals | Multi-Scale | RandomCrop | box AP |                                  Config                                   |                                                                                                                                                                                                                                 Download                                                                                                                                                                                                                                  |

| Sparse R-CNN | R-50-FPN  | pytorch |   1x    |         100         |    False    |   False    |  37.9  |                [config](./sparse-rcnn_r50_fpn_1x_coco.py)                 |                                                                         [model](https://download.openmmlab.com/mmdetection/v2.0/sparse_rcnn/sparse_rcnn_r50_fpn_1x_coco/sparse_rcnn_r50_fpn_1x_coco_20201222_214453-dc79b137.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/sparse_rcnn/sparse_rcnn_r50_fpn_1x_coco/sparse_rcnn_r50_fpn_1x_coco_20201222_214453-dc79b137.log.json)                                                                         |

| Sparse R-CNN | R-50-FPN  | pytorch |   3x    |         100         |    True     |   False    |  42.8  |           [config](./sparse-rcnn_r50_fpn_ms-480-800-3x_coco.py)           |                                         [model](https://download.openmmlab.com/mmdetection/v2.0/sparse_rcnn/sparse_rcnn_r50_fpn_mstrain_480-800_3x_coco/sparse_rcnn_r50_fpn_mstrain_480-800_3x_coco_20201218_154234-7bc5c054.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/sparse_rcnn/sparse_rcnn_r50_fpn_mstrain_480-800_3x_coco/sparse_rcnn_r50_fpn_mstrain_480-800_3x_coco_20201218_154234-7bc5c054.log.json)                                         |

| Sparse R-CNN | R-50-FPN  | pytorch |   3x    |         300         |    True     |    True    |  45.0  | [config](./sparse-rcnn_r50_fpn_300-proposals_crop-ms-480-800-3x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/sparse_rcnn/sparse_rcnn_r50_fpn_300_proposals_crop_mstrain_480-800_3x_coco/sparse_rcnn_r50_fpn_300_proposals_crop_mstrain_480-800_3x_coco_20201223_024605-9fe92701.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/sparse_rcnn/sparse_rcnn_r50_fpn_300_proposals_crop_mstrain_480-800_3x_coco/sparse_rcnn_r50_fpn_300_proposals_crop_mstrain_480-800_3x_coco_20201223_024605-9fe92701.log.json)   |

| Sparse R-CNN | R-101-FPN | pytorch |   3x    |         100         |    True     |   False    |  44.2  |          [config](./sparse-rcnn_r101_fpn_ms-480-800-3x_coco.py)           |                                       [model](https://download.openmmlab.com/mmdetection/v2.0/sparse_rcnn/sparse_rcnn_r101_fpn_mstrain_480-800_3x_coco/sparse_rcnn_r101_fpn_mstrain_480-800_3x_coco_20201223_121552-6c46c9d6.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/sparse_rcnn/sparse_rcnn_r101_fpn_mstrain_480-800_3x_coco/sparse_rcnn_r101_fpn_mstrain_480-800_3x_coco_20201223_121552-6c46c9d6.log.json)                                       |

| Sparse R-CNN | R-101-FPN | pytorch |   3x    |         300         |    True     |    True    |  46.2  | [config](./sparse-rcnn_r101_fpn_300-proposals_crop-ms-480-800-3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/sparse_rcnn/sparse_rcnn_r101_fpn_300_proposals_crop_mstrain_480-800_3x_coco/sparse_rcnn_r101_fpn_300_proposals_crop_mstrain_480-800_3x_coco_20201223_023452-c23c3564.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/sparse_rcnn/sparse_rcnn_r101_fpn_300_proposals_crop_mstrain_480-800_3x_coco/sparse_rcnn_r101_fpn_300_proposals_crop_mstrain_480-800_3x_coco_20201223_023452-c23c3564.log.json) |

### Notes

We observe about 0.3 AP noise especially when using ResNet-101 as the backbone.

```latex
@article{peize2020sparse,
  title   =  {{SparseR-CNN}: End-to-End Object Detection with Learnable Proposals},

```latex
  author  =  {Peize Sun and Rufeng Zhang and Yi Jiang and Tao Kong and Chenfeng Xu and Wei Zhan and Masayoshi Tomizuka and Lei Li and Zehuan Yuan and Changhu Wang and Ping Luo},
  journal =  {arXiv preprint arXiv:2011.12450},
  year    =  {2020}
}
```

Along with the release of OpenMMLab 2.0, MMDetection3D (namely MMDet3D) 1.1 made many significant changes, resulting in less redundant, more efficient code and a more consistent overall design. These changes break backward compatibility. Therefore, we prepared this migration guide to make the transition as smooth as possible so that all users can enjoy the productivity benefits of the new MMDet3D and the entire OpenMMLab 2.0 ecosystem.

## Environment

MMDet3D 1.1 depends on the new foundational library [MMEngine](https://github.com/open-mmlab/mmengine) for training deep learning models, and therefore has an entirely different dependency chain compared with MMDet3D 1.0. Even if you have a well-rounded MMDet3D 1.0 / 0.x environment before, you still need to create a new Python environment for MMDet3D 1.1. We provide a detailed [installation guide](./get_started.md) for reference.

The configuration files in our new version have a lot of modifications because of the differences between MMCV 1.x and MMEngine. The guides for migration from MMCV to MMEngine can be seen [here](https://github.com/open-mmlab/mmengine/tree/main/docs/en/migration).

We have renamed the names of the remote branches in MMDet3D 1.1 (renaming 1.1 to main, master to 1.0, and dev to dev-1.0). If your local branches in the git system are not aligned with branches of the remote repo, you can use the following commands to resolve it:

```
git fetch origin
git checkout main
git branch main_backup  # backup your main branch
git reset --hard origin/main
```

## Dataset

You should update the annotation files generated in the 1.0 version since some key words and structures of annotation in MMDet3D 1.1 have changed. Taking KITTI as an example, the update script is as follows:

```python
python tools/dataset_converters/update_infos_to_v2.py
        --dataset kitti

```python
        --pkl-path ./data/kitti/kitti_infos_train.pkl
        --out-dir ./kitti_v2/
```

If your annotation files are generated in the 0.x version, you should first update them to 1.0 version using this [script](../../tools/update_data_coords.py). Alternatively, you can re-generate annotation files from scratch using this [script](../../tools/create_data.py).

## Model

## Model

MMDet3D 1.1 supports loading weights trained on the old version (1.0 version). For models that are important or frequently used, we have thoroughly verified their precisions in the 1.1 version. Especially for some models that may experience potential performance drop or training bugs in the old version, such as [centerpoint](https://github.com/open-mmlab/mmdetection3d/issues/2390), we have checked them and ensured the right precision in the new version. If you encounter any problem, please feel free to raise an [issue](https://github.com/open-mmlab/mmdetection3d/issues). Additionally, we have added some of the latest SOTA methods in our [package](../../configs/) and [projects](../../projects/), making MMDet3D 1.1 a highly recommended choice for implementing your project.

# IconVSR (CVPR'2021)

> [BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond](https://arxiv.org/abs/2012.02181)

Video super-resolution (VSR) approaches tend to have more components than the image counterparts as they need to exploit the additional temporal dimension. Complex designs are not uncommon. In this study, we wish to untangle the knots and reconsider some most essential components for VSR guided by four basic functionalities, i.e., Propagation, Alignment, Aggregation, and Upsampling. By reusing some existing components added with minimal redesigns, we show a succinct pipeline, BasicVSR, that achieves appealing improvements in terms of speed and restoration quality in comparison to many state-of-the-art algorithms. We conduct systematic analysis to explain how such gain can be obtained and discuss the pitfalls. We further show the extensibility of BasicVSR by presenting an information-refill mechanism and a coupled propagation scheme to facilitate information aggregation. The BasicVSR and its extension, IconVSR, can serve as strong baselines for future VSR approaches.

 <img src="https://user-images.githubusercontent.com/7676947/144011348-c58101d4-5f69-4e58-be2b-7accd07b06fa.png" width="400"/>
</div >

Evaluated on RGB channels for REDS4 and Y channel for others. The metrics are `PSNR` / `SSIM` .
The pretrained weights of the IconVSR components can be found here: [SPyNet](https://download.openmmlab.com/mmediting/restorers/basicvsr/spynet_20210409-c6c1bd09.pth), [EDVR-M for REDS](https://download.openmmlab.com/mmediting/restorers/iconvsr/edvrm_reds_20210413-3867262f.pth), and [EDVR-M for Vimeo-90K](https://download.openmmlab.com/mmediting/restorers/iconvsr/edvrm_vimeo90k_20210413-e40e99a8.pth).

|       Model       | Dataset |  PSNR (RGB) | SSIM (RGB) |  PSNR (Y) | SSIM (Y) |   Training Resources    |       Download        |

| :---------------: | :---------------------: | :-------------------------: | :------------------: | :-------------------: | :-------------------------: | :------------------: | :----------------------: | :-------------------: |
| [iconvsr_reds4](./iconvsr_2xb4_reds4.py) | REDS4 (BIx4) | **31.6926** | **0.8951**  |-|-| 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_reds4_20210413-9e09d621.pth) | [log](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_reds4_20210413_222735.log.json) |

| [iconvsr_reds4](./iconvsr_2xb4_reds4.py) | UDM10 (BDx4)   | -|-|35.3377| 0.9471| 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_reds4_20210413-9e09d621.pth) | [log](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_reds4_20210413_222735.log.json) |
| [iconvsr_reds4](./iconvsr_2xb4_reds4.py) | Vid4 (BIx4)   | -|-|**27.4809**| **0.8354** | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_reds4_20210413-9e09d621.pth) | [log](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_reds4_20210413_222735.log.json) |
| [iconvsr_reds4](./iconvsr_2xb4_reds4.py) | Vid4 (BDx4)   | -|-|  25.2110  |  0.7732 | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_reds4_20210413-9e09d621.pth) | [log](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_reds4_20210413_222735.log.json) |

| [iconvsr_reds4](./iconvsr_2xb4_reds4.py) | Vimeo-90K-T (BIx4)   | -|-|36.4983  | 0.9416  | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_reds4_20210413-9e09d621.pth) | [log](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_reds4_20210413_222735.log.json) |
| [iconvsr_reds4](./iconvsr_2xb4_reds4.py) | Vimeo-90K-T (BDx4)   | -|-| 34.4299 | 0.9287   | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_reds4_20210413-9e09d621.pth) | [log](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_reds4_20210413_222735.log.json) |

| [iconvsr_vimeo90k_bi](./iconvsr_2xb4_vimeo90k-bi.py) | REDS4 (BIx4) | 30.3452 |0.8659 |-|-| 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bi_20210413-7c7418dc.pth) | [log](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bi_20210413_222757.log.json) |
| [iconvsr_vimeo90k_bi](./iconvsr_2xb4_vimeo90k-bi.py) | UDM10 (BDx4)   |-|-|34.2595 |0.9398 | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bi_20210413-7c7418dc.pth) | [log](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bi_20210413_222757.log.json) |

| [iconvsr_vimeo90k_bi](./iconvsr_2xb4_vimeo90k-bi.py) |  Vid4 (BIx4)   |-|-|27.4238  |0.8297   | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bi_20210413-7c7418dc.pth) | [log](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bi_20210413_222757.log.json) |
| [iconvsr_vimeo90k_bi](./iconvsr_2xb4_vimeo90k-bi.py) | Vid4 (BDx4)   |-|-|24.6666|0.7491 | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bi_20210413-7c7418dc.pth) | [log](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bi_20210413_222757.log.json) |

| [iconvsr_vimeo90k_bi](./iconvsr_2xb4_vimeo90k-bi.py) | Vimeo-90K-T (BIx4)   |-|-|**37.3729**| **0.9467** | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bi_20210413-7c7418dc.pth) | [log](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bi_20210413_222757.log.json) |
| [iconvsr_vimeo90k_bi](./iconvsr_2xb4_vimeo90k-bi.py) | Vimeo-90K-T (BDx4)   |-|-|34.5548 | 0.9295 | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bi_20210413-7c7418dc.pth) | [log](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bi_20210413_222757.log.json) |

| [iconvsr_vimeo90k_bd](./iconvsr_2xb4_vimeo90k-bd.py) | REDS4 (BIx4) |29.0150 | 0.8465 |-|-| 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bd_20210414-5f38cb34.pth) | [log](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bd_20210414_084128.log.json) |
| [iconvsr_vimeo90k_bd](./iconvsr_2xb4_vimeo90k-bd.py) | UDM10 (BDx4)   |-|-| **40.0640** | **0.9697** | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bd_20210414-5f38cb34.pth) | [log](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bd_20210414_084128.log.json) |

| [iconvsr_vimeo90k_bd](./iconvsr_2xb4_vimeo90k-bd.py) | Vid4 (BIx4)   |-|-|26.3109| 0.8028 | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bd_20210414-5f38cb34.pth) | [log](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bd_20210414_084128.log.json) |
| [iconvsr_vimeo90k_bd](./iconvsr_2xb4_vimeo90k-bd.py) | Vid4 (BDx4)   | -|-|   **28.2464** | **0.8612** | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bd_20210414-5f38cb34.pth) | [log](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bd_20210414_084128.log.json) |

| [iconvsr_vimeo90k_bd](./iconvsr_2xb4_vimeo90k-bd.py) | Vimeo-90K-T (BIx4)   | -|-|34.6780| 0.9339| 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bd_20210414-5f38cb34.pth) | [log](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bd_20210414_084128.log.json) |
| [iconvsr_vimeo90k_bd](./iconvsr_2xb4_vimeo90k-bd.py) | Vimeo-90K-T (BDx4)   |-|-|**37.7573** | **0.9517** | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bd_20210414-5f38cb34.pth) | [log](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bd_20210414_084128.log.json) |

**Train**

<details>

```shell
# cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/iconvsr/iconvsr_2xb4_reds4.py

```shell
# cpu train

python tools/train.py configs/iconvsr/iconvsr_2xb4_reds4.py

./tools/dist_train.sh configs/iconvsr/iconvsr_2xb4_reds4.py 8
```

**Test**

<details>

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/iconvsr/iconvsr_2xb4_reds4.py https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_reds4_20210413-9e09d621.pth

python tools/test.py configs/iconvsr/iconvsr_2xb4_reds4.py https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_reds4_20210413-9e09d621.pth

./tools/dist_test.sh configs/iconvsr/iconvsr_2xb4_reds4.py https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_reds4_20210413-9e09d621.pth 8
```

```bibtex
@InProceedings{chan2021basicvsr,
  author = {Chan, Kelvin CK and Wang, Xintao and Yu, Ke and Dong, Chao and Loy, Chen Change},
  title = {BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond},

author = {Chan, Kelvin CK and Wang, Xintao and Yu, Ke and Dong, Chao and Loy, Chen Change},
  year = {2021}
}
```

# Customize Optimizer

In this tutorial, we will introduce some methods about how to build the optimizer and learning rate scheduler for your tasks.

- [Customize Optimizer](#customize-optimizer)
  - [Build optimizers using optim_wrapper](#build-optimizers-using-optim_wrapper)
    - [Use optimizers supported by PyTorch](#use-optimizers-supported-by-pytorch)
    - [Parameter-wise finely configuration](#parameter-wise-finely-configuration)
    - [Gradient clipping](#gradient-clipping)
    - [Gradient accumulation](#gradient-accumulation)
  - [Customize parameter schedules](#customize-parameter-schedules)
    - [Customize learning rate schedules](#customize-learning-rate-schedules)
    - [Customize momentum schedules](#customize-momentum-schedules)
  - [Add new optimizers or constructors](#add-new-optimizers-or-constructors)
    - [Add new optimizers](#add-new-optimizers)
      - [1. Implement a new optimizer](#1-implement-a-new-optimizer)

- [Add new optimizers or constructors](#add-new-optimizers-or-constructors)
      - [2. Import the optimizer](#2-import-the-optimizer)
      - [3. Specify the optimizer in the config file](#3-specify-the-optimizer-in-the-config-file)
    - [Add new optimizer constructors](#add-new-optimizer-constructors)

## Build optimizers using optim_wrapper

We use the `optim_wrapper` field to configure the strategies of optimization, which includes choices of the optimizer, parameter-wise configurations, gradient clipping and accumulation. A simple example can be:

```python
    optimizer=dict(type='SGD', lr=0.0003, weight_decay=0.0001)
)
```

In the above example, a SGD optimizer with learning rate 0.0003 and weight decay 0.0001 is built.

### Use optimizers supported by PyTorch

)
```

We support all the optimizers implemented by PyTorch. To use a different optimizer, just need to change the `optimizer` field of config files. For example, if you want to use `torch.optim.Adam`, the modification in the config file could be as the following.

```python
    optimizer = dict(
        type='Adam',
        lr=0.001,
        betas=(0.9, 0.999),
        eps=1e-08,
        weight_decay=0,
        amsgrad=False),
)
```

First we need to change the value of `type` to the desired optimizer name supported in `torch.optim`. Next we add necessary arguments of this optimizer to the `optimizer` field. The above config will build the following optimizer:

```python
torch.optim.Adam(lr=0.001,
                 betas=(0.9, 0.999),
                 eps=1e-08,
                 weight_decay=0,

```python
                 amsgrad=False)
```

### Parameter-wise finely configuration

Some models may have parameter-specific settings for optimization, for example, no weight decay to the BatchNorm layers or using different learning rates for different network layers.
To finely configure them, we can use the `paramwise_cfg` argument in `optim_wrapper`.

- **Set different hyper-parameter multipliers for different types of parameters.**

  For instance, we can set `norm_decay_mult=0.` in `paramwise_cfg` to change the weight decay of weight and bias of normalization layers to zero.

  ```python
  optim_wrapper = dict(
      optimizer=dict(type='SGD', lr=0.8, weight_decay=1e-4),
      paramwise_cfg=dict(norm_decay_mult=0.))
  ```

  More types of parameters are supported to configured, list as follow:

optimizer=dict(type='SGD', lr=0.8, weight_decay=1e-4),
  ```

  - `lr_mult`: Multiplier for learning rate of all parameters.
  - `decay_mult`: Multiplier for weight decay of all parameters.
  - `bias_lr_mult`: Multiplier for learning rate of bias (Not include normalization layers' biases and deformable convolution layers' offsets). Defaults to 1.
  - `bias_decay_mult`: Multiplier for weight decay of bias (Not include normalization layers' biases and deformable convolution layers' offsets). Defaults to 1.
  - `norm_decay_mult`: Multiplier for weight decay of weigh and bias of normalization layers. Defaults to 1.
  - `dwconv_decay_mult`: Multiplier for weight decay of depth-wise convolution layers. Defaults to 1.
  - `bypass_duplicate`: Whether to bypass duplicated parameters. Defaults to `False`.

- `dwconv_decay_mult`: Multiplier for weight decay of depth-wise convolution layers. Defaults to 1.
  - `dcn_offset_lr_mult`: Multiplier for learning rate of deformable convolution layers. Defaults to 1.

- **Set different hyper-parameter multipliers for specific parameters.**

  MMAction2 can use `custom_keys` in `paramwise_cfg` to specify different parameters to use different learning rates or weight decay.

  For example, to set all learning rates and weight decays of `backbone.layer0` to 0, the rest of `backbone` remains the same as the optimizer and the learning rate of `head` to 0.001, use the configs below.

  ```python
      optimizer=dict(type='SGD', lr=0.01, weight_decay=0.0001),
      paramwise_cfg=dict(
          custom_keys={
              'backbone.layer0': dict(lr_mult=0, decay_mult=0),
              'backbone': dict(lr_mult=1),

              'head': dict(lr_mult=0.1)
          }))
  ```

### Gradient clipping

During the training process, the loss function may get close to a cliffy region and cause gradient explosion. And gradient clipping is helpful to stabilize the training process. More introduction can be found in [this page](https://paperswithcode.com/method/gradient-clipping).

Currently we support `clip_grad` option in `optim_wrapper` for gradient clipping, refers to [PyTorch Documentation](torch.nn.utils.clip_grad_norm_).

```python
    optimizer=dict(type='SGD', lr=0.01, weight_decay=0.0001),
    # norm_type: type of the used p-norm, here norm_type is 2.
    clip_grad=dict(max_norm=35, norm_type=2))
```

### Gradient accumulation

# norm_type: type of the used p-norm, here norm_type is 2.
```

When computing resources are lacking, the batch size can only be set to a small value, which may affect the performance of models. Gradient accumulation can be used to solve this problem. We support `accumulative_counts` option in `optim_wrapper` for gradient accumulation.

```python
train_dataloader = dict(batch_size=64)
    accumulative_counts=4)
```

Indicates that during training, back-propagation is performed every 4 iters. And the above is equivalent to:

```python
train_dataloader = dict(batch_size=256)
    optimizer=dict(type='SGD', lr=0.01, weight_decay=0.0001))
```

## Customize parameter schedules

```python
```

In training, the optimzation parameters such as learing rate, momentum, are usually not fixed but changing through iterations or epochs. PyTorch supports several learning rate schedulers, which are not sufficient for complex strategies. In MMAction2, we provide `param_scheduler` for better controls of different parameter schedules.

### Customize learning rate schedules

Learning rate schedulers are widely used to improve performance. We support most of the PyTorch schedulers, including `ExponentialLR`, `LinearLR`, `StepLR`, `MultiStepLR`, etc.

All available learning rate scheduler can be found {external+mmengine:ref}`here <scheduler>`, and the
names of learning rate schedulers end with `LR`.

- **Single learning rate schedule**

  In most cases, we use only one learning rate schedule for simplicity. For instance, [`MultiStepLR`](mmengine.optim.MultiStepLR) is used as the default learning rate schedule for ResNet. Here, `param_scheduler` is a dictionary.

  ```python
  param_scheduler = dict(
      type='MultiStepLR',
      by_epoch=True,
      milestones=[100, 150],
      gamma=0.1)
  ```

  Or, we want to use the [`CosineAnnealingLR`](mmengine.optim.CosineAnnealingLR) scheduler to decay the learning rate:

  ```python
      type='CosineAnnealingLR',
      T_max=num_epochs)
  ```

- **Multiple learning rate schedules**

```python
  ```

  In some of the training cases, multiple learning rate schedules are applied for higher accuracy. For example ,in the early stage, training is easy to be volatile, and warmup is a technique to reduce volatility.
  The learning rate will increase gradually from a minor value to the expected value by warmup and decay afterwards by other schedules.

  In MMAction2, simply combines desired schedules in `param_scheduler` as a list can achieve the warmup strategy.

  Here are some examples:

  1. linear warmup during the first 50 iters.

  ```python
    param_scheduler = [
        # linear warm-up by iters
        dict(type='LinearLR',
            start_factor=0.001,
            by_epoch=False,  # by iters
            end=50),  # only warm up for first 50 iters
        # main learing rate schedule
        dict(type='MultiStepLR',

start_factor=0.001,
            by_epoch=True,
            milestones=[8, 11],
            gamma=0.1)
    ]
  ```

  2. linear warmup and update lr by iter during the first 10 epochs.

  ```python
        # linear warm-up by epochs in [0, 10) epochs
            end=10,
            convert_to_iter_based=True,  # Update learning rate by iter.
        ),
        # use CosineAnnealing schedule after 10 epochs
        dict(type='CosineAnnealingLR', by_epoch=True, begin=10)
    ]
  ```

),
    ]
  ```

  Notice that, we use `begin` and `end` arguments here to assign the valid range, which is \[`begin`, `end`) for this schedule. And the range unit is defined by `by_epoch` argument. If not specified, the `begin` is 0 and the `end` is the max epochs or iterations.

  If the ranges for all schedules are not continuous, the learning rate will stay constant in ignored range, otherwise all valid schedulers will be executed in order in a specific stage, which behaves the same as PyTorch [`ChainedScheduler`](torch.optim.lr_scheduler.ChainedScheduler).

### Customize momentum schedules

We support using momentum schedulers to modify the optimizer's momentum according to learning rate, which could make the loss converge in a faster way. The usage is the same as learning rate schedulers.

names of momentum rate schedulers end with `Momentum`.

```python
    # the lr scheduler
    dict(type='LinearLR', ...),
    # the momentum scheduler
    dict(type='LinearMomentum',
         start_factor=0.001,
         by_epoch=False,
         begin=0,
         end=1000)
]
```

## Add new optimizers or constructors

This part will modify the MMAction2 source code or add code to the MMAction2 framework, beginners can skip it.

### Add new optimizers

In academic research and industrial practice, it may be necessary to use optimization methods not implemented by MMAction2, and you can add them through the following methods.

#### 1. Implement a new optimizer

Assume you want to add an optimizer named `MyOptimizer`, which has arguments `a`, `b`, and `c`.

You need to create a new file under `mmaction/engine/optimizers`, and implement the new optimizer in the file, for example, in `mmaction/engine/optimizers/my_optimizer.py`:

```python
from mmaction.registry import OPTIMIZERS

    def __init__(self, a, b, c):
        ...

    def step(self, closure=None):
        ...
```

#### 2. Import the optimizer

To find the above module defined above, this module should be imported during the running. First import it in the `mmaction/engine/optimizers/__init__.py` to add it into the `mmaction.engine` package.

```python
# In mmaction/engine/optimizers/__init__.py
...
from .my_optimizer import MyOptimizer # MyOptimizer maybe other class name

__all__ = [..., 'MyOptimizer']
```

```python
...

```

During running, we will automatically import the `mmaction.engine` package and register the `MyOptimizer` at the same time.

#### 3. Specify the optimizer in the config file

Then you can use `MyOptimizer` in the `optim_wrapper.optimizer` field of config files.

```python
    optimizer=dict(type='MyOptimizer', a=a_value, b=b_value, c=c_value))
```

### Add new optimizer constructors

Some models may have some parameter-specific settings for optimization, like different weight decay rate for all `BatchNorm` layers.

Although we already can use [the `optim_wrapper.paramwise_cfg` field](#parameter-wise-finely-configuration) to
configure various parameter-specific optimizer settings. It may still not cover your need.

Of course, you can modify it. By default, we use the [`DefaultOptimWrapperConstructor`](mmengine.optim.DefaultOptimWrapperConstructor)
class to deal with the construction of optimizer. And during the construction, it fine-grainedly configures the optimizer settings of
different parameters according to the `paramwise_cfg`，which could also serve as a template for new optimizer constructor.

You can overwrite these behaviors by add new optimizer constructors.

```python
# In mmaction/engine/optimizers/my_optim_constructor.py
from mmengine.optim import DefaultOptimWrapperConstructor
from mmaction.registry import OPTIM_WRAPPER_CONSTRUCTORS

@OPTIM_WRAPPER_CONSTRUCTORS.register_module()
class MyOptimWrapperConstructor:

    def __init__(self, optim_wrapper_cfg, paramwise_cfg=None):
        ...

        ...
```

        ...

        ...
```

And then, import it and use it almost like [the optimizer tutorial](#add-new-optimizers).

1. Import it in the `mmaction/engine/optimizers/__init__.py` to add it into the `mmaction.engine` package.

   ```python
   # In mmaction/engine/optimizers/__init__.py
   ...
   from .my_optim_constructor import MyOptimWrapperConstructor

   __all__ = [..., 'MyOptimWrapperConstructor']
   ```

2. Use `MyOptimWrapperConstructor` in the `optim_wrapper.constructor` field of config files.

   ```python
   optim_wrapper = dict(
       constructor=dict(type='MyOptimWrapperConstructor'),
       optimizer=...,
       paramwise_cfg=...,
   )
   ```

# SAGAN (ICML'2019)

> [Self-attention generative adversarial networks](https://proceedings.mlr.press/v97/zhang19d.html)

> **Task**: Conditional GANs

In this paper, we propose the Self-Attention Generative Adversarial Network (SAGAN) which allows attention-driven, long-range dependency modeling for image generation tasks. Traditional convolutional GANs generate high-resolution details as a function of only spatially local points in lower-resolution feature maps. In SAGAN, details can be generated using cues from all feature locations. Moreover, the discriminator can check that highly detailed features in distant portions of the image are consistent with each other. Furthermore, recent work has shown that generator conditioning affects GAN performance. Leveraging this insight, we apply spectral normalization to the GAN generator and find that this improves training dynamics. The proposed SAGAN performs better than prior work, boosting the best published Inception score from 36.8 to 52.52 and reducing Fréchet Inception distance from 27.62 to 18.65 on the challenging ImageNet dataset. Visualization of the attention layers shows that the generator leverages neighborhoods that correspond to object shapes rather than local regions of fixed shape.

<img src="https://user-images.githubusercontent.com/28132635/143054130-8cc1d9b7-df13-4fdb-8dbf-af4b2c15ed28.JPG"/>
</div>

  <b> Results from our SAGAN trained in CIFAR10</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/28132635/127619657-67f2e62d-52e4-43d2-931f-6d0e6e019813.png" width="400"/>
</div>

|                       Model                        | Dataset  | Inplace ReLU | dist_step | Total Batchsize (BZ_PER_GPU * NGPU) | Total Iters\* |  Iter  |   IS    |   FID   |                       Download                        |
| :------------------------------------------------: | :------: | :----------: | :-------: | :---------------------------------: | :-----------: | :----: | :-----: | :-----: | :---------------------------------------------------: |

| [SAGAN-32x32-woInplaceReLU Best IS](./sagan_woReLUinplace_lr2e-4-ndisc5-1xb64_cifar10-32x32.py) | CIFAR10  |     w/o      |     5     |                64x1                 |    500000     | 400000 | 9.3217  | 10.5030 | [model](https://download.openmmlab.com/mmediting/sagan/sagan_cifar10_32_lr2e-4_ndisc5_b64x1_woReUinplace_is-iter400000_20210730_125743-4008a9ca.pth) \| [Log](https://download.openmmlab.com/mmediting/sagan/sagan_cifar10_32_lr2e-4_ndisc5_b64x1_woReUinplace_20210730_125449_fid-d50568a4_is-04008a9ca.json) |

| [SAGAN-32x32-woInplaceReLU Best FID](./sagan_woReLUinplace_lr2e-4-ndisc5-1xb64_cifar10-32x32.py) | CIFAR10  |     w/o      |     5     |                64x1                 |    500000     | 480000 | 9.3174  | 9.4252  | [model](https://download.openmmlab.com/mmediting/sagan/sagan_cifar10_32_lr2e-4_ndisc5_b64x1_woReUinplace_fid-iter480000_20210730_125449-d50568a4.pth) \| [Log](https://download.openmmlab.com/mmediting/sagan/sagan_cifar10_32_lr2e-4_ndisc5_b64x1_woReUinplace_20210730_125449_fid-d50568a4_is-04008a9ca.json) |

| [SAGAN-32x32-wInplaceReLU Best IS](./sagan_wReLUinplace_lr2e-4-ndisc5-1xb64_cifar10-32x32.py) | CIFAR10  |      w       |     5     |                64x1                 |    500000     | 380000 | 9.2286  | 11.7760 | [model](https://download.openmmlab.com/mmediting/sagan/sagan_cifar10_32_lr2e-4_ndisc5_b64x1_wReLUinplace_is-iter380000_20210730_124937-c77b4d25.pth) \| [Log](https://download.openmmlab.com/mmediting/sagan/sagan_cifar10_32_lr2e-4_ndisc5_b64x1_wReLUinplace_20210730_125155_fid-cbefb354_is-c77b4d25.json) |

| [SAGAN-32x32-wInplaceReLU Best FID](./sagan_wReLUinplace_lr2e-4-ndisc5-1xb64_cifar10-32x32.py) | CIFAR10  |      w       |     5     |                64x1                 |    500000     | 460000 | 9.2061  | 10.7781 | [model](https://download.openmmlab.com/mmediting/sagan/sagan_cifar10_32_lr2e-4_ndisc5_b64x1_wReLUinplace_fid-iter460000_20210730_125155-cbefb354.pth) \| [Log](https://download.openmmlab.com/mmediting/sagan/sagan_cifar10_32_lr2e-4_ndisc5_b64x1_wReLUinplace_20210730_125155_fid-cbefb354_is-c77b4d25.json) |

| [SAGAN-128x128-woInplaceReLU Best IS](./sagan_woReLUinplace_Glr1e-4_Dlr4e-4_ndisc1-4xb64_imagenet1k-128x128.py) | ImageNet |     w/o      |     1     |                64x4                 |    1000000    | 980000 | 31.5938 | 36.7712 | [model](https://download.openmmlab.com/mmediting/sagan/sagan_imagenet1k_128_Glr1e-4_Dlr4e-4_ndisc1_b32x4_woReLUinplace_is-iter980000_20210730_163140-cfbebfc6.pth) \| [Log](https://download.openmmlab.com/mmediting/sagan/sagan_imagenet1k_128_Glr1e-4_Dlr4e-4_ndisc1_b32x4_woReLUinplace_20210730_163431_fid-d7916963_is-cfbebfc6.json) |

| [SAGAN-128x128-woInplaceReLU Best FID](./sagan_woReLUinplace_Glr1e-4_Dlr4e-4_ndisc1-4xb64_imagenet1k-128x128.py) | ImageNet |     w/o      |     1     |                64x4                 |    1000000    | 950000 | 28.4936 | 34.7838 | [model](https://download.openmmlab.com/mmediting/sagan/sagan_imagenet1k_128_Glr1e-4_Dlr4e-4_ndisc1_b32x4_woReLUinplace_fid-iter950000_20210730_163431-d7916963.pth) \| [Log](https://download.openmmlab.com/mmediting/sagan/sagan_imagenet1k_128_Glr1e-4_Dlr4e-4_ndisc1_b32x4_woReLUinplace_20210730_163431_fid-d7916963_is-cfbebfc6.json) |

| [SAGAN-128x128-BigGAN Schedule Best IS](./sagan_woReLUinplace-Glr1e-4_Dlr4e-4_noaug-ndisc1-8xb32-bigGAN-sch_imagenet1k-128x128.py) | ImageNet |     w/o      |     1     |                32x8                 |    1000000    | 826000 | 69.5350 | 12.8295 | [model](https://download.openmmlab.com/mmediting/sagan/sagan_128_woReLUinplace_noaug_bigGAN_imagenet1k_b32x8_Glr1e-4_Dlr-4e-4_ndisc1_20210818_210232-3f5686af.pth) \| [Log](https://download.openmmlab.com/mmediting/sagan/sagan_128_woReLUinplace_noaug_bigGAN_imagenet1k_b32x8_Glr1e-4_Dlr-4e-4_ndisc1_20210818_210232-3f5686af.json) |

| [SAGAN-128x128-BigGAN Schedule Best FID](./sagan_woReLUinplace-Glr1e-4_Dlr4e-4_noaug-ndisc1-8xb32-bigGAN-sch_imagenet1k-128x128.py) | ImageNet |     w/o      |     1     |                32x8                 |    1000000    | 826000 | 69.5350 | 12.8295 | [model](https://download.openmmlab.com/mmediting/sagan/sagan_128_woReLUinplace_noaug_bigGAN_imagenet1k_b32x8_Glr1e-4_Dlr-4e-4_ndisc1_20210818_210232-3f5686af.pth) \| [Log](https://download.openmmlab.com/mmediting/sagan/sagan_128_woReLUinplace_noaug_bigGAN_imagenet1k_b32x8_Glr1e-4_Dlr-4e-4_ndisc1_20210818_210232-3f5686af.json) |

'\*' Iteration counting rule in our implementation is different from others. If you want to align with other codebases, you can use the following conversion formula:

```
total_iters (biggan/pytorch studio gan) = our_total_iters / dist_step
```

We also provide converted pre-train models from [Pytorch-StudioGAN](https://github.com/POSTECH-CVLab/PyTorch-StudioGAN).

```
```

To be noted that, in Pytorch Studio GAN, **inplace ReLU** is used in generator and discriminator.

|           Model            | Dataset  | Inplace ReLU | n_disc | Total Iters | IS (Our Pipeline) | FID (Our Pipeline) | IS (StudioGAN) | FID (StudioGAN) |           Download            |            Original Download link            |
| :------------------------: | :------: | :----------: | :----: | :---------: | :---------------: | :----------------: | :------------: | :-------------: | :---------------------------: | :------------------------------------------: |

| [SAGAN-32x32 StudioGAN](./sagan_cvt-studioGAN_cifar10-32x32.py) | CIFAR10  |      w       |   5    |   100000    |       9.116       |      10.2011       |     8.680      |     14.009      | [model](https://download.openmmlab.com/mmediting/sagan/sagan_32_cifar10_convert-studio-rgb_20210730_153321-080da7e2.pth) | [model](https://drive.google.com/drive/folders/1FA8hcz4MB8-hgTwLuDA0ZUfr8slud5P_) |
| [SAGAN0-128x128 StudioGAN](./sagan_128_cvt_studioGAN.py) | ImageNet |      w       |   1    |   1000000   |      27.367       |      40.1162       |     29.848     |     34.726      | [model](https://download.openmmlab.com/mmediting/sagan/sagan_128_imagenet1k_convert-studio-rgb_20210730_153357-eddb0d1d.pth) | [model](https://drive.google.com/drive/folders/1ZYaqeeumDgxOPDhRR5QLeLFIpgBJ9S6B) |

- `Our Pipeline` denote results evaluated with our pipeline.
- `StudioGAN` denote results released by Pytorch-StudioGAN.

For IS metric, our implementation is different from PyTorch-Studio GAN in the following aspects:

1. We use [Tero's Inception](https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/pretrained/metrics/inception-2015-12-05.pt) for feature extraction.
2. We use bicubic interpolation with PIL backend to resize image before feed them to Inception.

For FID evaluation, we follow the pipeline of [BigGAN](https://github.com/ajbrock/BigGAN-PyTorch/blob/98459431a5d618d644d54cd1e9fceb1e5045648d/calculate_inception_moments.py#L52), where the whole training set is adopted to extract inception statistics, and Pytorch Studio GAN uses 50000 randomly selected samples. Besides, we also use [Tero's Inception](https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/pretrained/metrics/inception-2015-12-05.pt) for feature extraction.

You can download the preprocessed inception state by the following url: [CIFAR10](https://download.openmmlab.com/mmediting/evaluation/fid_inception_pkl/cifar10.pkl) and [ImageNet1k](https://download.openmmlab.com/mmediting/evaluation/fid_inception_pkl/imagenet.pkl).

You can use following commands to extract those inception states by yourself.

```
# For CIFAR10
python tools/utils/inception_stat.py --data-cfg configs/_base_/datasets/cifar10_inception_stat.py --pklname cifar10.pkl --no-shuffle --inception-style stylegan --num-samples -1 --subset train

# For ImageNet1k
python tools/utils/inception_stat.py --data-cfg configs/_base_/datasets/imagenet_128x128_inception_stat.py --pklname imagenet.pkl --no-shuffle --inception-style stylegan --num-samples -1 --subset train
```

```latex
@inproceedings{zhang2019self,
  title={Self-attention generative adversarial networks},
  author={Zhang, Han and Goodfellow, Ian and Metaxas, Dimitris and Odena, Augustus},

```

```latex
  booktitle={International conference on machine learning},
  pages={7354--7363},
  organization={PMLR},
  url={https://proceedings.mlr.press/v97/zhang19d.html},
}
```

# 自定义评估指标

## 使用 MMPretrain 中的指标

在 MMPretrain 中，我们为单标签分类和多标签分类提供了多种指标：

**单标签分类**:

- [`Accuracy`](mmpretrain.evaluation.Accuracy)
- [`SingleLabelMetric`](mmpretrain.evaluation.SingleLabelMetric)，包括精度、召回率、f1-score 和支持度。

**多标签分类**:

- [`AveragePrecision`](mmpretrain.evaluation.AveragePrecision)， 或 AP (mAP)。
- [`MultiLabelMetric`](mmpretrain.evaluation.MultiLabelMetric)，包括精度、召回率、f1-score 和支持度。

要在验证和测试期间使用这些指标，我们需要修改配置文件中的 `val_evaluator` 和 `test_evaluator` 字段。

以下为几个例子：

1. 在验证和测试期间计算 top-1 和 top-5 准确率。

   ```python
   val_evaluator = dict(type='Accuracy', topk=(1, 5))
   test_evaluator = val_evaluator
   ```

2. 在验证和测试期间计算 top-1 准确率、top-5 准确度、精确度和召回率。

   ```python
   val_evaluator = [
     dict(type='Accuracy', topk=(1, 5)),
     dict(type='SingleLabelMetric', items=['precision', 'recall']),
   ]
   ```

3. 计算 mAP（平均平均精度）、CP（类别平均精度）、CR（类别平均召回率）、CF（类别平均 F1 分数）、OP（总体平均精度）、OR（总体平均召回率）和 OF1（总体平均 F1 分数）。

   ```python

]
   ```

   ```python
     dict(type='AveragePrecision'),
     dict(type='MultiLabelMetric', average='macro'),  # class-wise mean
     dict(type='MultiLabelMetric', average='micro'),  # overall mean
   ]
   ```

## 添加新的指标

MMPretrain 支持为追求更高定制化的用户实现定制化的评估指标。

您需要在 `mmpretrain/evaluation/metrics` 下创建一个新文件，并在该文件中实现新的指标，例如，在 `mmpretrain/evaluation/metrics/my_metric.py` 中。并创建一个自定义的评估指标类 `MyMetric` 继承 [MMEngine 中的 BaseMetric](mmengine.evaluator.BaseMetric)。

需要分别覆盖数据格式处理方法`process`和度量计算方法`compute_metrics`。 将其添加到“METRICS”注册表以实施任何自定义评估指标。

```python
from mmpretrain.registry import METRICS

class MyMetric(BaseMetric):

    def process(self, data_batch: Sequence[Dict], data_samples: Sequence[Dict]):

    """ The processed results should be stored in ``self.results``, which will
        be used to computed the metrics when all batches have been processed.
        `data_batch` stores the batch data from dataloader,
        and `data_samples` stores the batch outputs from model.
    """
        ...

    def compute_metrics(self, results: List):
    """ Compute the metrics from processed results and returns the evaluation results.
    """
        ...
```

然后，将其导入 `mmpretrain/evaluation/metrics/__init__.py` 以将其添加到 `mmpretrain.evaluation` 包中。

```python
# In mmpretrain/evaluation/metrics/__init__.py
...
from .my_metric import MyMetric

__all__ = [..., 'MyMetric']
```

最后，在配置文件的 `val_evaluator` 和 `test_evaluator` 字段中使用 `MyMetric`。

```python
val_evaluator = dict(type='MyMetric', ...)

```

```python
```

```{note}
更多的细节可以参考 {external+mmengine:doc}`MMEngine 文档: Evaluation <design/evaluation>`.
```

# Changelog

## 1.1.0 (7/3/2023)

**Highlights**

- Support HACS-segments dataset(ICCV'2019), MultiSports dataset(ICCV'2021), Kinetics-710 dataset(Arxiv'2022)
- Support rich projects: gesture recognition, spatio-temporal action detection tutorial, and knowledge distillation
- Support TCANet(CVPR'2021)
- Support VideoMAE V2(CVPR'2023), and VideoMAE(NeurIPS'2022) on action detection
- Support CLIP-based multi-modality models: ActionCLIP(Arxiv'2021) and CLIP4clip(ArXiv'2022)
- Support [Pure Python style Configuration File](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#a-pure-python-style-configuration-file-beta) and downloading datasets by MIM

**New Features**

- Support HACS-segments dataset ([2224](https://github.com/open-mmlab/mmaction2/pull/2224))
- Support TCANet ([2271](https://github.com/open-mmlab/mmaction2/pull/2271))
- Support MultiSports dataset ([2280](https://github.com/open-mmlab/mmaction2/pull/2280))

- Support spatio-temporal action detection tutorial ([2428](https://github.com/open-mmlab/mmaction2/pull/2428))
- Support knowledge distillation based on MMRazor ([2458](https://github.com/open-mmlab/mmaction2/pull/2458))
- Support VideoMAE V2 ([2460](https://github.com/open-mmlab/mmaction2/pull/2460))
- Support ActionCLIP ([2470](https://github.com/open-mmlab/mmaction2/pull/2470))
- Support CLIP4clip ([2489](https://github.com/open-mmlab/mmaction2/pull/2489))
- Support Kinetics-710 dataset ([2534](https://github.com/open-mmlab/mmaction2/pull/2534))
- Support gesture recognition project ([2539](https://github.com/open-mmlab/mmaction2/pull/2539))
- Support VideoMAE on action detection ([2547](https://github.com/open-mmlab/mmaction2/pull/2547))

- Support downloading datasets by MIM ([2465](https://github.com/open-mmlab/mmaction2/pull/1465))
- Support new config ([2542](https://github.com/open-mmlab/mmaction2/pull/2542))

**Improvements**

- Refactor TSM init_weights ([2396](https://github.com/open-mmlab/mmaction2/pull/2396))
- Add unit test for Recognizer 2D ([2432](https://github.com/open-mmlab/mmaction2/pull/2432))
- Enhance inference APIs ([2472](https://github.com/open-mmlab/mmaction2/pull/2472))
- Support converting ST-GCN and PoseC3D to ONNX ([2543](https://github.com/open-mmlab/mmaction2/pull/2543))
- Support feature extraction head ([2525](https://github.com/open-mmlab/mmaction2/pull/2525))

**Bug Fixes**

- Fix CircleCI ([2351](https://github.com/open-mmlab/mmaction2/pull/2351))

- Fix demo skeleton script ([2380](https://github.com/open-mmlab/mmaction2/pull/2380))
- Fix docker file branch ([2397](https://github.com/open-mmlab/mmaction2/pull/2397))
- Fix NTU pose extraction script ([2402](https://github.com/open-mmlab/mmaction2/pull/2402))
- Rename typing and enhance collect_env script ([2420](https://github.com/open-mmlab/mmaction2/pull/2420))
- Fix multi-label classification ([2425](https://github.com/open-mmlab/mmaction2/pull/2425), [2466](https://github.com/open-mmlab/mmaction2/pull/2466), [2532](https://github.com/open-mmlab/mmaction2/pull/2532))
- Fix lfb configs ([2426](https://github.com/open-mmlab/mmaction2/pull/2426))
- Fix a warning caused by `torch.div` ([2449](https://github.com/open-mmlab/mmaction2/pull/2449))

- Fix incompatibility of ImgAug and latest Numpy ([2451](https://github.com/open-mmlab/mmaction2/pull/2451))
- Fix MViT with_cls_token argument ([2480](https://github.com/open-mmlab/mmaction2/pull/2480))
- Fix timm BC-breaking for TSN ([2497](https://github.com/open-mmlab/mmaction2/pull/2497))
- Close FileHandler in Windows to make the temporary directory can be deleted  ([2565](https://github.com/open-mmlab/mmaction2/pull/2565))
- Update minimum PyTorch version to 1.8.1 ([2568](https://github.com/open-mmlab/mmaction2/pull/2568))

**Documentation**

- Fix document links in README ([2358](https://github.com/open-mmlab/mmaction2/pull/2358), [2372](https://github.com/open-mmlab/mmaction2/pull/2372), [2376](https://github.com/open-mmlab/mmaction2/pull/2376), [2382](https://github.com/open-mmlab/mmaction2/pull/2382))

- Update installation document ([2362](https://github.com/open-mmlab/mmaction2/pull/2362))
- Update upstream library version requirement ([2383](https://github.com/open-mmlab/mmaction2/pull/2383))
- Fix Colab tutorial ([2384](https://github.com/open-mmlab/mmaction2/pull/2384), [2391](https://github.com/open-mmlab/mmaction2/pull/2391), [2475](https://github.com/open-mmlab/mmaction2/pull/2475))
- Refine documents ([2404](https://github.com/open-mmlab/mmaction2/pull/2404))
- Update outdated config in readme ([2419](https://github.com/open-mmlab/mmaction2/pull/2419))
- Update OpenMMLab related repo list ([2429](https://github.com/open-mmlab/mmaction2/pull/2429))
- Fix UniFormer README and metafile ([2450](https://github.com/open-mmlab/mmaction2/pull/2450))
- Add finetune document ([2457](https://github.com/open-mmlab/mmaction2/pull/2457))
- Update FAQ document ([2476](https://github.com/open-mmlab/mmaction2/pull/2476), [2482](https://github.com/open-mmlab/mmaction2/pull/2482)

- Update download datasets document ([2495](https://github.com/open-mmlab/mmaction2/pull/2495))
- Translate Chinese document ([2516](https://github.com/open-mmlab/mmaction2/pull/2516), [2506](https://github.com/open-mmlab/mmaction2/pull/2506), [2499](https://github.com/open-mmlab/mmaction2/pull/2499))
- Refactor model zoo and dataset zoo ([2552](https://github.com/open-mmlab/mmaction2/pull/2552))
- Refactor Chinese document ([2567](https://github.com/open-mmlab/mmaction2/pull/2567))

## 1.0.0 (4/6/2023)

- Support RGB-PoseC3D(CVPR'2022).
- Support training UniFormer V2(Arxiv'2022).
- Support MSG3D(CVPR'2020) and CTRGCN(CVPR'2021) in projects.
- Refactor and provide more user-friendly documentation.

- Support RGB-PoseC3D ([2182](https://github.com/open-mmlab/mmaction2/pull/2182))

- Support training UniFormer V2 ([2221](https://github.com/open-mmlab/mmaction2/pull/2221))
- Support MSG3D and CTRGCN in projects. ([2269](https://github.com/open-mmlab/mmaction2/pull/2269), [2291](https://github.com/open-mmlab/mmaction2/pull/2291))

- Use MMEngine to calculate FLOPs ([2300](https://github.com/open-mmlab/mmaction2/pull/2300))
- Speed up LFB training ([2294](https://github.com/open-mmlab/mmaction2/pull/2294))
- Support multiprocessing on AVA evaluation ([2146](https://github.com/open-mmlab/mmaction2/pull/2146))
- Add a demo for exporting spatial-temporal detection model to ONNX ([2225](https://github.com/open-mmlab/mmaction2/pull/2225))
- Update spatial-temporal detection related folders ([2262](https://github.com/open-mmlab/mmaction2/pull/2262))

- Fix flip config of TSM for sth v1/v2 dataset ([#2247](https://github.com/open-mmlab/mmaction2/pull/2247))
- Fix circle ci ([2336](https://github.com/open-mmlab/mmaction2/pull/2336), [2334](https://github.com/open-mmlab/mmaction2/pull/2334))
- Fix accepting an unexpected argument local-rank in PyTorch 2.0 ([2320](https://github.com/open-mmlab/mmaction2/pull/2320))
- Fix TSM config link ([2315](https://github.com/open-mmlab/mmaction2/pull/2315))
- Fix numpy version requirement in CI ([2284](https://github.com/open-mmlab/mmaction2/pull/2284))
- Fix NTU pose extraction script ([2246](https://github.com/open-mmlab/mmaction2/pull/2246))
- Fix TSM-MobileNet V2 ([2332](https://github.com/open-mmlab/mmaction2/pull/2332))
- Fix command bugs in localization tasks' README ([2244](https://github.com/open-mmlab/mmaction2/pull/2244))

- Fix duplicate name in DecordInit and SampleAVAFrame ([2251](https://github.com/open-mmlab/mmaction2/pull/2251))
- Fix channel order when showing video ([2308](https://github.com/open-mmlab/mmaction2/pull/2308))
- Specify map_location to cpu when using \_load_checkpoint ([2252](https://github.com/open-mmlab/mmaction2/pull/2254))

- Refactor and provide more user-friendly documentation ([2341](https://github.com/open-mmlab/mmaction2/pull/2341), [2312](https://github.com/open-mmlab/mmaction2/pull/2312), [2325](https://github.com/open-mmlab/mmaction2/pull/2325))
- Add README_zh-CN ([2252](https://github.com/open-mmlab/mmaction2/pull/2252))
- Add social networking links ([2294](https://github.com/open-mmlab/mmaction2/pull/2294))

- Fix sthv2 dataset annotations preparation document ([2248](https://github.com/open-mmlab/mmaction2/pull/2248))

## 1.0.0rc3 (2/10/2023)

- Support Action Recognition model UniFormer V1(ICLR'2022), UniFormer V2(Arxiv'2022).
- Support training MViT V2(CVPR'2022), and MaskFeat(CVPR'2022) fine-tuning.

- Support UniFormer V1/V2 ([#2153](https://github.com/open-mmlab/mmaction2/pull/2153))
- Support training MViT, and MaskFeat fine-tuning ([#2186](https://github.com/open-mmlab/mmaction2/pull/2186))
- Support a unified inference interface: Inferencer ([#2164](https://github.com/open-mmlab/mmaction2/pull/2164))

- Support load data list from multi-backends ([#2176](https://github.com/open-mmlab/mmaction2/pull/2176))

- Upgrade isort to fix CI ([#2198](https://github.com/open-mmlab/mmaction2/pull/2198))
- Fix bug in skeleton demo ([#2214](https://github.com/open-mmlab/mmaction2/pull/2214))

- Add Chinese documentation for config.md ([#2188](https://github.com/open-mmlab/mmaction2/pull/2188))
- Add readme for Omnisource ([#2205](https://github.com/open-mmlab/mmaction2/pull/2205))

## 1.0.0rc2 (1/10/2023)

- Support Action Recognition model VideoMAE(NeurIPS'2022), MViT V2(CVPR'2022), C2D and skeleton-based action recognition model STGCN++
- Support Omni-Source training on ImageNet and Kinetics datasets
- Support exporting spatial-temporal detection models to ONNX

- Support VideoMAE ([#1942](https://github.com/open-mmlab/mmaction2/pull/1942))

- Support MViT V2 ([#2007](https://github.com/open-mmlab/mmaction2/pull/2007))
- Support C2D ([#2022](https://github.com/open-mmlab/mmaction2/pull/2022))
- Support AVA-Kinetics dataset ([#2080](https://github.com/open-mmlab/mmaction2/pull/2080))
- Support STGCN++ ([#2156](https://github.com/open-mmlab/mmaction2/pull/2156))
- Support exporting spatial-temporal detection models to ONNX ([#2148](https://github.com/open-mmlab/mmaction2/pull/2148))
- Support Omni-Source training on ImageNet and Kinetics datasets ([#2143](https://github.com/open-mmlab/mmaction2/pull/2143))

- Support repeat batch data augmentation ([#2170](https://github.com/open-mmlab/mmaction2/pull/2170))
- Support calculating FLOPs tool powered by fvcore ([#1997](https://github.com/open-mmlab/mmaction2/pull/1997))

- Support Spatial-temporal detection demo ([#2019](https://github.com/open-mmlab/mmaction2/pull/2019))
- Add SyncBufferHook and add randomness config in train.py ([#2044](https://github.com/open-mmlab/mmaction2/pull/2044))
- Refactor gradcam ([#2049](https://github.com/open-mmlab/mmaction2/pull/2049))
- Support init_cfg in Swin and ViTMAE ([#2055](https://github.com/open-mmlab/mmaction2/pull/2055))
- Refactor STGCN and related pipelines ([#2087](https://github.com/open-mmlab/mmaction2/pull/2087))
- Refactor visualization tools ([#2092](https://github.com/open-mmlab/mmaction2/pull/2092))
- Update `SampleFrames` transform and improve most models' performance ([#1942](https://github.com/open-mmlab/mmaction2/pull/1942))
- Support real-time webcam demo ([#2152](https://github.com/open-mmlab/mmaction2/pull/2152))

- Refactor and enhance 2s-AGCN ([#2130](https://github.com/open-mmlab/mmaction2/pull/2130))
- Support adjusting fps in `SampleFrame` ([#2157](https://github.com/open-mmlab/mmaction2/pull/2157))

- Fix CI upstream library dependency ([#2000](https://github.com/open-mmlab/mmaction2/pull/2000))
- Fix SlowOnly readme typos and results ([#2006](https://github.com/open-mmlab/mmaction2/pull/2006))
- Fix VideoSwin readme ([#2010](https://github.com/open-mmlab/mmaction2/pull/2010))
- Fix tools and mim error ([#2028](https://github.com/open-mmlab/mmaction2/pull/2028))
- Fix Imgaug wrapper ([#2024](https://github.com/open-mmlab/mmaction2/pull/2024))
- Remove useless scripts ([#2032](https://github.com/open-mmlab/mmaction2/pull/2032))
- Fix multi-view inference ([#2045](https://github.com/open-mmlab/mmaction2/pull/2045))

- Update mmcv maximum version to 1.8.0 ([#2047](https://github.com/open-mmlab/mmaction2/pull/2047))
- Fix torchserver dependency ([#2053](https://github.com/open-mmlab/mmaction2/pull/2053))
- Fix `gen_ntu_rgbd_raw` script ([#2076](https://github.com/open-mmlab/mmaction2/pull/2076))
- Update AVA-Kinetics experiment configs and results ([#2099](https://github.com/open-mmlab/mmaction2/pull/2099))
- Add `joint.pkl` and `bone.pkl` used in multi-stream fusion tool ([#2106](https://github.com/open-mmlab/mmaction2/pull/2106))
- Fix lint CI config ([#2110](https://github.com/open-mmlab/mmaction2/pull/2110))

- Update testing accuracy for modified `SampleFrames` ([#2117](https://github.com/open-mmlab/mmaction2/pull/2117)), ([#2121](https://github.com/open-mmlab/mmaction2/pull/2121)), ([#2122](https://github.com/open-mmlab/mmaction2/pull/2122)), ([#2124](https://github.com/open-mmlab/mmaction2/pull/2124)), ([#2125](https://github.com/open-mmlab/mmaction2/pull/2125)), ([#2126](https://github.com/open-mmlab/mmaction2/pull/2126)), ([#2129](https://github.com/open-mmlab/mmaction2/pull/2129)), ([#2128](https://github.com/open-mmlab/mmaction2/pull/2128))
- Fix timm related bug ([#1976](https://github.com/open-mmlab/mmaction2/pull/1976))
- Fix `check_videos.py` script ([#2134](https://github.com/open-mmlab/mmaction2/pull/2134))
- Update CI maximum torch version to 1.13.0 ([#2118](https://github.com/open-mmlab/mmaction2/pull/2118))

- Add MMYOLO description in README ([#2011](https://github.com/open-mmlab/mmaction2/pull/2011))
- Add v1.x introduction in README ([#2023](https://github.com/open-mmlab/mmaction2/pull/2023))
- Fix link in README ([#2035](https://github.com/open-mmlab/mmaction2/pull/2035))
- Refine some docs ([#2038](https://github.com/open-mmlab/mmaction2/pull/2038)), ([#2040](https://github.com/open-mmlab/mmaction2/pull/2040)), ([#2058](https://github.com/open-mmlab/mmaction2/pull/2058))
- Update TSN/TSM Readme ([#2082](https://github.com/open-mmlab/mmaction2/pull/2082))
- Add chinese document ([#2083](https://github.com/open-mmlab/mmaction2/pull/2083))
- Adjust document structure ([#2088](https://github.com/open-mmlab/mmaction2/pull/2088))
- Fix Sth-Sth and Jester dataset links ([#2103](https://github.com/open-mmlab/mmaction2/pull/2103))

- Fix doc link ([#2131](https://github.com/open-mmlab/mmaction2/pull/2131))

## 1.0.0rc1 (10/14/2022)

- Support Video Swin Transformer

- Support Video Swin Transformer ([#1939](https://github.com/open-mmlab/mmaction2/pull/1939))

- Add colab tutorial for 1.x ([#1956](https://github.com/open-mmlab/mmaction2/pull/1956))
- Support skeleton-based action recognition demo ([#1920](https://github.com/open-mmlab/mmaction2/pull/1920))

- Fix link in doc ([#1986](https://github.com/open-mmlab/mmaction2/pull/1986), [#1967](https://github.com/open-mmlab/mmaction2/pull/1967), [#1951](https://github.com/open-mmlab/mmaction2/pull/1951), [#1926](https://github.com/open-mmlab/mmaction2/pull/1926),[#1944](https://github.com/open-mmlab/mmaction2/pull/1944), [#1944](https://github.com/open-mmlab/mmaction2/pull/1944), [#1927](https://github.com/open-mmlab/mmaction2/pull/1927), [#1925](https://github.com/open-mmlab/mmaction2/pull/1925))
- Fix CI ([#1987](https://github.com/open-mmlab/mmaction2/pull/1987), [#1930](https://github.com/open-mmlab/mmaction2/pull/1930), [#1923](https://github.com/open-mmlab/mmaction2/pull/1923))
- Fix pre-commit hook config ([#1971](https://github.com/open-mmlab/mmaction2/pull/1971))
- Fix TIN config ([#1912](https://github.com/open-mmlab/mmaction2/pull/1912))

- Fix UT for BMN and BSN ([#1966](https://github.com/open-mmlab/mmaction2/pull/1966))
- Fix UT for Recognizer2D ([#1937](https://github.com/open-mmlab/mmaction2/pull/1937))
- Fix BSN and BMN configs for localization ([#1913](https://github.com/open-mmlab/mmaction2/pull/1913))
- Modeify ST-GCN configs ([#1913](https://github.com/open-mmlab/mmaction2/pull/1914))
- Fix typo in migration doc ([#1931](https://github.com/open-mmlab/mmaction2/pull/1931))
- Remove Onnx related tools ([#1928](https://github.com/open-mmlab/mmaction2/pull/1928))
- Update TANet readme ([#1916](https://github.com/open-mmlab/mmaction2/pull/1916), [#1890](https://github.com/open-mmlab/mmaction2/pull/1890))
- Update 2S-AGCN readme ([#1915](https://github.com/open-mmlab/mmaction2/pull/1915))

- Fix TSN configs ([#1905](https://github.com/open-mmlab/mmaction2/pull/1905))
- Fix configs for detection ([#1903](https://github.com/open-mmlab/mmaction2/pull/1903))
- Fix typo in TIN config ([#1904](https://github.com/open-mmlab/mmaction2/pull/1904))
- Fix PoseC3D readme ([#1899](https://github.com/open-mmlab/mmaction2/pull/1899))
- Fix ST-GCN configs ([#1891](https://github.com/open-mmlab/mmaction2/pull/1891))
- Fix audio recognition readme ([#1898](https://github.com/open-mmlab/mmaction2/pull/1898))
- Fix TSM readme ([#1887](https://github.com/open-mmlab/mmaction2/pull/1887))
- Fix SlowOnly readme ([#1889](https://github.com/open-mmlab/mmaction2/pull/1889))
- Fix TRN readme ([#1888](https://github.com/open-mmlab/mmaction2/pull/1888))
- Fix typo in get_started doc ([#1895](https://github.com/open-mmlab/mmaction2/pull/1895))

## 1.0.0rc0 (09/01/2022)

We are excited to announce the release of MMAction2 v1.0.0rc0.
MMAction2 1.0.0beta is the first version of MMAction2 1.x, a part of the OpenMMLab 2.0 projects.
Built upon the new [training engine](https://github.com/open-mmlab/mmengine).

- **New engines**. MMAction2 1.x is based on MMEngine\](https://github.com/open-mmlab/mmengine), which provides a general and powerful runner that allows more flexible customizations and significantly simplifies the entrypoints of high-level interfaces.

- **Unified interfaces**. As a part of the OpenMMLab 2.0 projects, MMAction2 1.x unifies and refactors the interfaces and internal logics of train, testing, datasets, models, evaluation, and visualization. All the OpenMMLab 2.0 projects share the same design in those interfaces and logics to allow the emergence of multi-task/modality algorithms.

- **More documentation and tutorials**. We add a bunch of documentation and tutorials to help users get started more smoothly. Read it [here](https://github.com/open-mmlab/mmaction2/blob/main/docs/en/migration.md).

**Breaking Changes**

In this release, we made lots of major refactoring and modifications. Please refer to the [migration guide](../migration.md) for details and migration instructions.

## 0.24.0 (05/05/2022)

- Support different seeds

- Add lateral norm in multigrid config ([#1567](https://github.com/open-mmlab/mmaction2/pull/1567))

- Add openpose 25 joints in graph config ([#1578](https://github.com/open-mmlab/mmaction2/pull/1578))
- Support MLU Backend ([#1608](https://github.com/open-mmlab/mmaction2/pull/1608))

**Bug and Typo Fixes**

- Fix local_rank ([#1558](https://github.com/open-mmlab/mmaction2/pull/1558))
- Fix install typo ([#1571](https://github.com/open-mmlab/mmaction2/pull/1571))
- Fix the inference API doc ([#1580](https://github.com/open-mmlab/mmaction2/pull/1580))
- Fix zh-CN demo.md and getting_started.md ([#1587](https://github.com/open-mmlab/mmaction2/pull/1587))
- Remove Recommonmark ([#1595](https://github.com/open-mmlab/mmaction2/pull/1595))
- Fix inference with ndarray ([#1603](https://github.com/open-mmlab/mmaction2/pull/1603))

- Fix the log error when `IterBasedRunner` is used ([#1606](https://github.com/open-mmlab/mmaction2/pull/1606))

## 0.23.0 (04/01/2022)

- Provide multi-node training & testing script
- Update error log

- Support different seeds([#1502](https://github.com/open-mmlab/mmaction2/pull/1502))
- Provide multi-node training & testing script([#1521](https://github.com/open-mmlab/mmaction2/pull/1521))
- Update error log([#1546](https://github.com/open-mmlab/mmaction2/pull/1546))

**Documentations**

- Update gpus in Slowfast readme([#1497](https://github.com/open-mmlab/mmaction2/pull/1497))
- Fix work_dir in multigrid config([#1498](https://github.com/open-mmlab/mmaction2/pull/1498))
- Add sub bn docs([#1503](https://github.com/open-mmlab/mmaction2/pull/1503))

- Add shortcycle sampler docs([#1513](https://github.com/open-mmlab/mmaction2/pull/1513))
- Update Windows Declaration([#1520](https://github.com/open-mmlab/mmaction2/pull/1520))
- Update the link for ST-GCN([#1544](https://github.com/open-mmlab/mmaction2/pull/1544))
- Update install commands([#1549](https://github.com/open-mmlab/mmaction2/pull/1549))

- Update colab tutorial install cmds([#1522](https://github.com/open-mmlab/mmaction2/pull/1522))
- Fix num_iters_per_epoch in analyze_logs.py([#1530](https://github.com/open-mmlab/mmaction2/pull/1530))
- Fix distributed_sampler([#1532](https://github.com/open-mmlab/mmaction2/pull/1532))
- Fix cd dir error([#1545](https://github.com/open-mmlab/mmaction2/pull/1545))
- Update arg names([#1548](https://github.com/open-mmlab/mmaction2/pull/1548))

**ModelZoo**

## 0.22.0 (03/05/2022)

- Support Multigrid training strategy
- Support CPU training
- Support audio demo
- Support topk customizing in models/heads/base.py

- Support Multigrid training strategy([#1378](https://github.com/open-mmlab/mmaction2/pull/1378))
- Support STGCN in demo_skeleton.py([#1391](https://github.com/open-mmlab/mmaction2/pull/1391))
- Support CPU training([#1407](https://github.com/open-mmlab/mmaction2/pull/1407))
- Support audio demo([#1425](https://github.com/open-mmlab/mmaction2/pull/1425))
- Support topk customizing in models/heads/base.py([#1452](https://github.com/open-mmlab/mmaction2/pull/1452))

- Add OpenMMLab platform([#1393](https://github.com/open-mmlab/mmaction2/pull/1393))

- Update links([#1394](https://github.com/open-mmlab/mmaction2/pull/1394))
- Update readme in configs([#1404](https://github.com/open-mmlab/mmaction2/pull/1404))
- Update instructions to install mmcv-full([#1426](https://github.com/open-mmlab/mmaction2/pull/1426))
- Add shortcut([#1433](https://github.com/open-mmlab/mmaction2/pull/1433))
- Update modelzoo([#1439](https://github.com/open-mmlab/mmaction2/pull/1439))
- add video_structuralize in readme([#1455](https://github.com/open-mmlab/mmaction2/pull/1455))
- Update OpenMMLab repo information([#1482](https://github.com/open-mmlab/mmaction2/pull/1482))

- Update train.py([#1375](https://github.com/open-mmlab/mmaction2/pull/1375))
- Fix printout bug([#1382](<(https://github.com/open-mmlab/mmaction2/pull/1382)>))
- Update multi processing setting([#1395](https://github.com/open-mmlab/mmaction2/pull/1395))

- Setup multi processing both in train and test([#1405](https://github.com/open-mmlab/mmaction2/pull/1405))
- Fix bug in nondistributed multi-gpu training([#1406](https://github.com/open-mmlab/mmaction2/pull/1406))
- Add variable fps in  ava_dataset.py([#1409](https://github.com/open-mmlab/mmaction2/pull/1409))
- Only support distributed training([#1414](https://github.com/open-mmlab/mmaction2/pull/1414))
- Set test_mode for AVA configs([#1432](https://github.com/open-mmlab/mmaction2/pull/1432))
- Support single label([#1434](https://github.com/open-mmlab/mmaction2/pull/1434))
- Add check copyright([#1447](https://github.com/open-mmlab/mmaction2/pull/1447))
- Support Windows CI([#1448](https://github.com/open-mmlab/mmaction2/pull/1448))

- Fix wrong device of class_weight in models/losses/cross_entropy_loss.py([#1457](https://github.com/open-mmlab/mmaction2/pull/1457))
- Fix bug caused by distributed([#1459](https://github.com/open-mmlab/mmaction2/pull/1459))
- Update readme([#1460](https://github.com/open-mmlab/mmaction2/pull/1460))
- Fix lint caused by colab automatic upload([#1461](https://github.com/open-mmlab/mmaction2/pull/1461))
- Refine CI([#1471](https://github.com/open-mmlab/mmaction2/pull/1471))
- Update pre-commit([#1474](https://github.com/open-mmlab/mmaction2/pull/1474))
- Add deprecation message for deploy tool([#1483](https://github.com/open-mmlab/mmaction2/pull/1483))

- Support slowfast_steplr([#1421](https://github.com/open-mmlab/mmaction2/pull/1421))

## 0.21.0 (31/12/2021)

- Support 2s-AGCN

- Support publish models in Windows
- Improve some sthv1 related models
- Support BABEL

- Support 2s-AGCN([#1248](https://github.com/open-mmlab/mmaction2/pull/1248))
- Support skip postproc in ntu_pose_extraction([#1295](https://github.com/open-mmlab/mmaction2/pull/1295))
- Support publish models in Windows([#1325](https://github.com/open-mmlab/mmaction2/pull/1325))
- Add copyright checkhook in pre-commit-config([#1344](https://github.com/open-mmlab/mmaction2/pull/1344))

- Add MMFlow ([#1273](https://github.com/open-mmlab/mmaction2/pull/1273))
- Revise README.md and add projects.md ([#1286](https://github.com/open-mmlab/mmaction2/pull/1286))
- Add 2s-AGCN in Updates([#1289](https://github.com/open-mmlab/mmaction2/pull/1289))

- Add MMFewShot([#1300](https://github.com/open-mmlab/mmaction2/pull/1300))
- Add MMHuman3d([#1304](https://github.com/open-mmlab/mmaction2/pull/1304))
- Update pre-commit([#1313](https://github.com/open-mmlab/mmaction2/pull/1313))
- Use share menu from the theme instead([#1328](https://github.com/open-mmlab/mmaction2/pull/1328))
- Update installation command([#1340](https://github.com/open-mmlab/mmaction2/pull/1340))

- Update the inference part in notebooks([#1256](https://github.com/open-mmlab/mmaction2/pull/1256))
- Update the map_location([#1262](<(https://github.com/open-mmlab/mmaction2/pull/1262)>))
- Fix bug that start_index is not used in RawFrameDecode([#1278](https://github.com/open-mmlab/mmaction2/pull/1278))

- Fix bug in init_random_seed([#1282](https://github.com/open-mmlab/mmaction2/pull/1282))
- Fix bug in setup.py([#1303](https://github.com/open-mmlab/mmaction2/pull/1303))
- Fix interrogate error in workflows([#1305](https://github.com/open-mmlab/mmaction2/pull/1305))
- Fix typo in slowfast config([#1309](https://github.com/open-mmlab/mmaction2/pull/1309))
- Cancel previous runs that are not completed([#1327](https://github.com/open-mmlab/mmaction2/pull/1327))
- Fix missing skip_postproc parameter([#1347](https://github.com/open-mmlab/mmaction2/pull/1347))
- Update ssn.py([#1355](https://github.com/open-mmlab/mmaction2/pull/1355))
- Use latest youtube-dl([#1357](https://github.com/open-mmlab/mmaction2/pull/1357))
- Fix test-best([#1362](https://github.com/open-mmlab/mmaction2/pull/1362))

- Improve some sthv1 related models([#1306](https://github.com/open-mmlab/mmaction2/pull/1306))
- Support BABEL([#1332](https://github.com/open-mmlab/mmaction2/pull/1332))

## 0.20.0 (07/10/2021)

- Support TorchServe
- Add video structuralize demo
- Support using 3D skeletons for skeleton-based action recognition
- Benchmark PoseC3D on UCF and HMDB

- Support TorchServe ([#1212](https://github.com/open-mmlab/mmaction2/pull/1212))
- Support 3D skeletons pre-processing ([#1218](https://github.com/open-mmlab/mmaction2/pull/1218))
- Support video structuralize demo ([#1197](https://github.com/open-mmlab/mmaction2/pull/1197))

- Revise README.md and add projects.md ([#1214](https://github.com/open-mmlab/mmaction2/pull/1214))

- Add CN docs for Skeleton dataset, PoseC3D and ST-GCN ([#1228](https://github.com/open-mmlab/mmaction2/pull/1228), [#1237](https://github.com/open-mmlab/mmaction2/pull/1237), [#1236](https://github.com/open-mmlab/mmaction2/pull/1236))
- Add tutorial for custom dataset training for skeleton-based action recognition ([#1234](https://github.com/open-mmlab/mmaction2/pull/1234))

- Fix tutorial link ([#1219](https://github.com/open-mmlab/mmaction2/pull/1219))
- Fix GYM links ([#1224](https://github.com/open-mmlab/mmaction2/pull/1224))

- Benchmark PoseC3D on UCF and HMDB ([#1223](https://github.com/open-mmlab/mmaction2/pull/1223))
- Add ST-GCN + 3D skeleton model for NTU60-XSub ([#1236](https://github.com/open-mmlab/mmaction2/pull/1236))

## 0.19.0 (07/10/2021)

- Support ST-GCN
- Refactor the inference API

- Add code spell check hook

- Support ST-GCN ([#1123](https://github.com/open-mmlab/mmaction2/pull/1123))

**Improvement**

- Add label maps for every dataset ([#1127](https://github.com/open-mmlab/mmaction2/pull/1127))
- Remove useless code MultiGroupCrop ([#1180](https://github.com/open-mmlab/mmaction2/pull/1180))
- Refactor Inference API ([#1191](https://github.com/open-mmlab/mmaction2/pull/1191))
- Add code spell check hook ([#1208](https://github.com/open-mmlab/mmaction2/pull/1208))
- Use docker in CI ([#1159](https://github.com/open-mmlab/mmaction2/pull/1159))

- Update metafiles to new OpenMMLAB protocols ([#1134](https://github.com/open-mmlab/mmaction2/pull/1134))

- Switch to new doc style ([#1160](https://github.com/open-mmlab/mmaction2/pull/1160))
- Improve the ERROR message ([#1203](https://github.com/open-mmlab/mmaction2/pull/1203))
- Fix invalid URL in getting_started ([#1169](https://github.com/open-mmlab/mmaction2/pull/1169))

- Compatible with new MMClassification ([#1139](https://github.com/open-mmlab/mmaction2/pull/1139))
- Add missing runtime dependencies ([#1144](https://github.com/open-mmlab/mmaction2/pull/1144))
- Fix THUMOS tag proposals path ([#1156](https://github.com/open-mmlab/mmaction2/pull/1156))
- Fix LoadHVULabel ([#1194](https://github.com/open-mmlab/mmaction2/pull/1194))
- Switch the default value of `persistent_workers` to False ([#1202](https://github.com/open-mmlab/mmaction2/pull/1202))

- Fix `_freeze_stages` for MobileNetV2 ([#1193](https://github.com/open-mmlab/mmaction2/pull/1193))
- Fix resume when building rawframes ([#1150](https://github.com/open-mmlab/mmaction2/pull/1150))
- Fix device bug for class weight ([#1188](https://github.com/open-mmlab/mmaction2/pull/1188))
- Correct Arg names in extract_audio.py ([#1148](https://github.com/open-mmlab/mmaction2/pull/1148))

- Add TSM-MobileNetV2 ported from TSM ([#1163](https://github.com/open-mmlab/mmaction2/pull/1163))
- Add ST-GCN for NTURGB+D-XSub-60 ([#1123](https://github.com/open-mmlab/mmaction2/pull/1123))

## 0.18.0 (02/09/2021)

- Add CopyRight ([#1099](https://github.com/open-mmlab/mmaction2/pull/1099))

- Support NTU Pose Extraction ([#1076](https://github.com/open-mmlab/mmaction2/pull/1076))
- Support Caching in RawFrameDecode ([#1078](https://github.com/open-mmlab/mmaction2/pull/1078))
- Add citations & Support python3.9 CI & Use fixed-version sphinx ([#1125](https://github.com/open-mmlab/mmaction2/pull/1125))

- Add Descriptions of PoseC3D dataset ([#1053](https://github.com/open-mmlab/mmaction2/pull/1053))

- Fix SSV2 checkpoints ([#1101](https://github.com/open-mmlab/mmaction2/pull/1101))
- Fix CSN normalization ([#1116](https://github.com/open-mmlab/mmaction2/pull/1116))
- Fix typo ([#1121](https://github.com/open-mmlab/mmaction2/pull/1121))
- Fix new_crop_quadruple bug ([#1108](https://github.com/open-mmlab/mmaction2/pull/1108))

## 0.17.0 (03/08/2021)

- Support PyTorch 1.9

- Support Pytorchvideo Transforms
- Support PreciseBN

- Support Pytorchvideo Transforms ([#1008](https://github.com/open-mmlab/mmaction2/pull/1008))
- Support PreciseBN ([#1038](https://github.com/open-mmlab/mmaction2/pull/1038))

- Remove redundant augmentations in config files ([#996](https://github.com/open-mmlab/mmaction2/pull/996))
- Make resource directory to hold common resource pictures ([#1011](https://github.com/open-mmlab/mmaction2/pull/1011))
- Remove deprecated FrameSelector ([#1010](https://github.com/open-mmlab/mmaction2/pull/1010))
- Support Concat Dataset ([#1000](https://github.com/open-mmlab/mmaction2/pull/1000))
- Add `to-mp4` option to resize_videos.py ([#1021](https://github.com/open-mmlab/mmaction2/pull/1021))

- Add option to keep tail frames ([#1050](https://github.com/open-mmlab/mmaction2/pull/1050))
- Update MIM support ([#1061](https://github.com/open-mmlab/mmaction2/pull/1061))
- Calculate Top-K accurate and inaccurate classes ([#1047](https://github.com/open-mmlab/mmaction2/pull/1047))

- Fix bug in PoseC3D demo ([#1009](https://github.com/open-mmlab/mmaction2/pull/1009))
- Fix some problems in resize_videos.py ([#1012](https://github.com/open-mmlab/mmaction2/pull/1012))
- Support torch1.9 ([#1015](https://github.com/open-mmlab/mmaction2/pull/1015))
- Remove redundant code in CI ([#1046](https://github.com/open-mmlab/mmaction2/pull/1046))
- Fix bug about persistent_workers ([#1044](https://github.com/open-mmlab/mmaction2/pull/1044))

- Support TimeSformer feature extraction ([#1035](https://github.com/open-mmlab/mmaction2/pull/1035))
- Fix ColorJitter ([#1025](https://github.com/open-mmlab/mmaction2/pull/1025))

- Add TSM-R50 sthv1 models trained by PytorchVideo RandAugment and AugMix ([#1008](https://github.com/open-mmlab/mmaction2/pull/1008))
- Update SlowOnly SthV1 checkpoints ([#1034](https://github.com/open-mmlab/mmaction2/pull/1034))
- Add SlowOnly Kinetics400 checkpoints trained with Precise-BN ([#1038](https://github.com/open-mmlab/mmaction2/pull/1038))
- Add CSN-R50 from scratch checkpoints ([#1045](https://github.com/open-mmlab/mmaction2/pull/1045))
- TPN Kinetics-400 Checkpoints trained with the new ColorJitter ([#1025](https://github.com/open-mmlab/mmaction2/pull/1025))

- Add Chinese translation of feature_extraction.md ([#1020](https://github.com/open-mmlab/mmaction2/pull/1020))
- Fix the code snippet in getting_started.md ([#1023](https://github.com/open-mmlab/mmaction2/pull/1023))
- Fix TANet config table ([#1028](https://github.com/open-mmlab/mmaction2/pull/1028))
- Add description to PoseC3D dataset ([#1053](https://github.com/open-mmlab/mmaction2/pull/1053))

## 0.16.0 (01/07/2021)

- Support using backbone from pytorch-image-models(timm)
- Support PIMS Decoder
- Demo for skeleton-based action recognition
- Support Timesformer

- Support using backbones from pytorch-image-models(timm) for TSN ([#880](https://github.com/open-mmlab/mmaction2/pull/880))

- Support torchvision transformations in preprocessing pipelines ([#972](https://github.com/open-mmlab/mmaction2/pull/972))
- Demo for skeleton-based action recognition ([#972](https://github.com/open-mmlab/mmaction2/pull/972))
- Support Timesformer ([#839](https://github.com/open-mmlab/mmaction2/pull/839))

- Add a tool to find invalid videos ([#907](https://github.com/open-mmlab/mmaction2/pull/907), [#950](https://github.com/open-mmlab/mmaction2/pull/950))
- Add an option to specify spectrogram_type ([#909](https://github.com/open-mmlab/mmaction2/pull/909))
- Add json output to video demo ([#906](https://github.com/open-mmlab/mmaction2/pull/906))
- Add MIM related docs ([#918](https://github.com/open-mmlab/mmaction2/pull/918))

- Rename lr to scheduler ([#916](https://github.com/open-mmlab/mmaction2/pull/916))
- Support `--cfg-options` for demos ([#911](https://github.com/open-mmlab/mmaction2/pull/911))
- Support number counting for flow-wise filename template ([#922](https://github.com/open-mmlab/mmaction2/pull/922))
- Add Chinese tutorial ([#941](https://github.com/open-mmlab/mmaction2/pull/941))
- Change ResNet3D default values ([#939](https://github.com/open-mmlab/mmaction2/pull/939))
- Adjust script structure ([#935](https://github.com/open-mmlab/mmaction2/pull/935))
- Add font color to args in long_video_demo ([#947](https://github.com/open-mmlab/mmaction2/pull/947))
- Polish code style with Pylint ([#908](https://github.com/open-mmlab/mmaction2/pull/908))

- Support PIMS Decoder ([#946](https://github.com/open-mmlab/mmaction2/pull/946))
- Improve Metafiles ([#956](https://github.com/open-mmlab/mmaction2/pull/956), [#979](https://github.com/open-mmlab/mmaction2/pull/979), [#966](https://github.com/open-mmlab/mmaction2/pull/966))
- Add links to download Kinetics400 validation ([#920](https://github.com/open-mmlab/mmaction2/pull/920))
- Audit the usage of shutil.rmtree ([#943](https://github.com/open-mmlab/mmaction2/pull/943))
- Polish localizer related codes([#913](https://github.com/open-mmlab/mmaction2/pull/913))

- Fix spatiotemporal detection demo ([#899](https://github.com/open-mmlab/mmaction2/pull/899))
- Fix docstring for 3D inflate ([#925](https://github.com/open-mmlab/mmaction2/pull/925))

- Fix bug of writing text to video with TextClip ([#952](https://github.com/open-mmlab/mmaction2/pull/952))
- Fix mmcv install in CI ([#977](https://github.com/open-mmlab/mmaction2/pull/977))

- Add TSN with Swin Transformer backbone as an example for using pytorch-image-models(timm) backbones ([#880](https://github.com/open-mmlab/mmaction2/pull/880))
- Port CSN checkpoints from VMZ ([#945](https://github.com/open-mmlab/mmaction2/pull/945))
- Release various checkpoints for UCF101, HMDB51 and Sthv1 ([#938](https://github.com/open-mmlab/mmaction2/pull/938))
- Update TSM modelzoo ([#981](https://github.com/open-mmlab/mmaction2/pull/981))

## 0.15.0 (31/05/2021)

- Support PoseC3D
- Support ACRN

- Support MIM

- Support PoseC3D ([#786](https://github.com/open-mmlab/mmaction2/pull/786), [#890](https://github.com/open-mmlab/mmaction2/pull/890))
- Support MIM ([#870](https://github.com/open-mmlab/mmaction2/pull/870))
- Support ACRN and Focal Loss ([#891](https://github.com/open-mmlab/mmaction2/pull/891))
- Support Jester dataset ([#864](https://github.com/open-mmlab/mmaction2/pull/864))

- Add `metric_options` for evaluation to docs ([#873](https://github.com/open-mmlab/mmaction2/pull/873))
- Support creating a new label map based on custom classes for demos about spatio temporal demo ([#879](https://github.com/open-mmlab/mmaction2/pull/879))
- Improve document about AVA dataset preparation ([#878](https://github.com/open-mmlab/mmaction2/pull/878))

- Provide a script to extract clip-level feature ([#856](https://github.com/open-mmlab/mmaction2/pull/856))

- Fix issues about resume ([#877](https://github.com/open-mmlab/mmaction2/pull/877), [#878](https://github.com/open-mmlab/mmaction2/pull/878))
- Correct the key name of `eval_results` dictionary for metric 'mmit_mean_average_precision' ([#885](https://github.com/open-mmlab/mmaction2/pull/885))

## 0.14.0 (30/04/2021)

- Support TRN
- Support Diving48

- Support TRN ([#755](https://github.com/open-mmlab/mmaction2/pull/755))
- Support Diving48 ([#835](https://github.com/open-mmlab/mmaction2/pull/835))

- Support Webcam Demo for Spatio-temporal Action Detection Models ([#795](https://github.com/open-mmlab/mmaction2/pull/795))

- Add softmax option for pytorch2onnx tool ([#781](https://github.com/open-mmlab/mmaction2/pull/781))
- Test with onnx models and TensorRT engines ([#758](https://github.com/open-mmlab/mmaction2/pull/758))
- Speed up AVA Testing ([#784](https://github.com/open-mmlab/mmaction2/pull/784))
- Add `self.with_neck` attribute ([#796](https://github.com/open-mmlab/mmaction2/pull/796))
- Update installation document ([#798](https://github.com/open-mmlab/mmaction2/pull/798))
- Use a random master port ([#809](https://github.com/open-mmlab/mmaction2/pull/8098))

- Update AVA processing data document ([#801](https://github.com/open-mmlab/mmaction2/pull/801))
- Refactor spatio-temporal augmentation ([#782](https://github.com/open-mmlab/mmaction2/pull/782))
- Add QR code in CN README ([#812](https://github.com/open-mmlab/mmaction2/pull/812))
- Add Alternative way to download Kinetics ([#817](https://github.com/open-mmlab/mmaction2/pull/817), [#822](https://github.com/open-mmlab/mmaction2/pull/822))
- Refactor Sampler ([#790](https://github.com/open-mmlab/mmaction2/pull/790))
- Use EvalHook in MMCV with backward compatibility ([#793](https://github.com/open-mmlab/mmaction2/pull/793))
- Use MMCV Model Registry ([#843](https://github.com/open-mmlab/mmaction2/pull/843))

- Fix a bug in pytorch2onnx.py when `num_classes <= 4` ([#800](https://github.com/open-mmlab/mmaction2/pull/800), [#824](https://github.com/open-mmlab/mmaction2/pull/824))
- Fix `demo_spatiotemporal_det.py` error ([#803](https://github.com/open-mmlab/mmaction2/pull/803), [#805](https://github.com/open-mmlab/mmaction2/pull/805))
- Fix loading config bugs when resume ([#820](https://github.com/open-mmlab/mmaction2/pull/820))
- Make HMDB51 annotation generation more robust ([#811](https://github.com/open-mmlab/mmaction2/pull/811))

- Update checkpoint for 256 height in something-V2 ([#789](https://github.com/open-mmlab/mmaction2/pull/789))

## 0.13.0 (31/03/2021)

- Support LFB
- Support using backbone from MMCls/TorchVision
- Add Chinese documentation

- Support LFB ([#553](https://github.com/open-mmlab/mmaction2/pull/553))
- Support using backbones from MMCls for TSN ([#679](https://github.com/open-mmlab/mmaction2/pull/679))
- Support using backbones from TorchVision for TSN ([#720](https://github.com/open-mmlab/mmaction2/pull/720))
- Support Mixup and Cutmix for recognizers ([#681](https://github.com/open-mmlab/mmaction2/pull/681))

- Add slowfast config/json/log/ckpt for training custom classes of AVA ([#678](https://github.com/open-mmlab/mmaction2/pull/678))
- Set RandAugment as Imgaug default transforms ([#585](https://github.com/open-mmlab/mmaction2/pull/585))
- Add `--test-last` & `--test-best` for `tools/train.py` to test checkpoints after training ([#608](https://github.com/open-mmlab/mmaction2/pull/608))
- Add fcn_testing in TPN ([#684](https://github.com/open-mmlab/mmaction2/pull/684))
- Remove redundant recall functions ([#741](https://github.com/open-mmlab/mmaction2/pull/741))
- Recursively remove pretrained step for testing ([#695](https://github.com/open-mmlab/mmaction2/pull/695))
- Improve demo by limiting inference fps ([#668](https://github.com/open-mmlab/mmaction2/pull/668))

- Fix a bug about multi-class in VideoDataset ([#723](https://github.com/open-mmlab/mmaction2/pull/678))

- Reverse key-value in anet filelist generation ([#686](https://github.com/open-mmlab/mmaction2/pull/686))
- Fix flow norm cfg typo ([#693](https://github.com/open-mmlab/mmaction2/pull/693))

- Add LFB for AVA2.1 ([#553](https://github.com/open-mmlab/mmaction2/pull/553))
- Add TSN with ResNeXt-101-32x4d backbone as an example for using MMCls backbones ([#679](https://github.com/open-mmlab/mmaction2/pull/679))
- Add TSN with Densenet161 backbone as an example for using TorchVision backbones ([#720](https://github.com/open-mmlab/mmaction2/pull/720))
- Add slowonly_nl_embedded_gaussian_r50_4x16x1_150e_kinetics400_rgb ([#690](https://github.com/open-mmlab/mmaction2/pull/690))
- Add slowonly_nl_embedded_gaussian_r50_8x8x1_150e_kinetics400_rgb ([#704](https://github.com/open-mmlab/mmaction2/pull/704))

- Add slowonly_nl_kinetics_pretrained_r50_4x16x1(8x8x1)\_20e_ava_rgb ([#730](https://github.com/open-mmlab/mmaction2/pull/730))

## 0.12.0 (28/02/2021)

- Support TSM-MobileNetV2
- Support TANet
- Support GPU Normalize

- Support TSM-MobileNetV2 ([#415](https://github.com/open-mmlab/mmaction2/pull/415))
- Support flip with label mapping ([#591](https://github.com/open-mmlab/mmaction2/pull/591))
- Add seed option for sampler ([#642](https://github.com/open-mmlab/mmaction2/pull/642))
- Support GPU Normalize ([#586](https://github.com/open-mmlab/mmaction2/pull/586))
- Support TANet ([#595](https://github.com/open-mmlab/mmaction2/pull/595))

- Training custom classes of ava dataset ([#555](https://github.com/open-mmlab/mmaction2/pull/555))

- Add CN README in homepage ([#592](https://github.com/open-mmlab/mmaction2/pull/592), [#594](https://github.com/open-mmlab/mmaction2/pull/594))
- Support soft label for CrossEntropyLoss ([#625](https://github.com/open-mmlab/mmaction2/pull/625))
- Refactor config: Specify `train_cfg` and `test_cfg` in `model` ([#629](https://github.com/open-mmlab/mmaction2/pull/629))
- Provide an alternative way to download older kinetics annotations ([#597](https://github.com/open-mmlab/mmaction2/pull/597))
- Update FAQ for
  - 1). data pipeline about video and frames ([#598](https://github.com/open-mmlab/mmaction2/pull/598))
  - 2). how to show results ([#598](https://github.com/open-mmlab/mmaction2/pull/598))
  - 3). batch size setting for batchnorm ([#657](https://github.com/open-mmlab/mmaction2/pull/657))

- 2). how to show results ([#598](https://github.com/open-mmlab/mmaction2/pull/598))
  - 4). how to fix stages of backbone when finetuning models ([#658](https://github.com/open-mmlab/mmaction2/pull/658))
- Modify default value of `save_best` ([#600](https://github.com/open-mmlab/mmaction2/pull/600))
- Use BibTex rather than latex in markdown ([#607](https://github.com/open-mmlab/mmaction2/pull/607))
- Add warnings of uninstalling mmdet and supplementary documents ([#624](https://github.com/open-mmlab/mmaction2/pull/624))

- Fix value of `pem_low_temporal_iou_threshold` in BSN ([#556](https://github.com/open-mmlab/mmaction2/pull/556))
- Fix ActivityNet download script ([#601](https://github.com/open-mmlab/mmaction2/pull/601))

- Add TSM-MobileNetV2 for Kinetics400 ([#415](https://github.com/open-mmlab/mmaction2/pull/415))
- Add deeper SlowFast models ([#605](https://github.com/open-mmlab/mmaction2/pull/605))

## 0.11.0 (31/01/2021)

- Support imgaug
- Support spatial temporal demo
- Refactor EvalHook, config structure, unittest structure

- Support [imgaug](https://imgaug.readthedocs.io/en/latest/index.html) for augmentations in the data pipeline ([#492](https://github.com/open-mmlab/mmaction2/pull/492))
- Support setting `max_testing_views` for extremely large models to save GPU memory used ([#511](https://github.com/open-mmlab/mmaction2/pull/511))
- Add spatial temporal demo ([#547](https://github.com/open-mmlab/mmaction2/pull/547), [#566](https://github.com/open-mmlab/mmaction2/pull/566))

- Refactor EvalHook ([#395](https://github.com/open-mmlab/mmaction2/pull/395))
- Refactor AVA hook ([#567](https://github.com/open-mmlab/mmaction2/pull/567))
- Add repo citation ([#545](https://github.com/open-mmlab/mmaction2/pull/545))
- Add dataset size of Kinetics400 ([#503](https://github.com/open-mmlab/mmaction2/pull/503))
- Add lazy operation docs ([#504](https://github.com/open-mmlab/mmaction2/pull/504))
- Add class_weight for CrossEntropyLoss and BCELossWithLogits ([#509](https://github.com/open-mmlab/mmaction2/pull/509))
- add some explanation about the resampling in slowfast ([#502](https://github.com/open-mmlab/mmaction2/pull/502))
- Modify paper title in README.md ([#512](https://github.com/open-mmlab/mmaction2/pull/512))

- Add alternative ways to download Kinetics ([#521](https://github.com/open-mmlab/mmaction2/pull/521))
- Add OpenMMLab projects link in README ([#530](https://github.com/open-mmlab/mmaction2/pull/530))
- Change default preprocessing to shortedge to 256 ([#538](https://github.com/open-mmlab/mmaction2/pull/538))
- Add config tag in dataset README ([#540](https://github.com/open-mmlab/mmaction2/pull/540))
- Add solution for markdownlint installation issue ([#497](https://github.com/open-mmlab/mmaction2/pull/497))
- Add dataset overview in readthedocs ([#548](https://github.com/open-mmlab/mmaction2/pull/548))
- Modify the trigger mode of the warnings of missing mmdet ([#583](https://github.com/open-mmlab/mmaction2/pull/583))
- Refactor config structure ([#488](https://github.com/open-mmlab/mmaction2/pull/488), [#572](https://github.com/open-mmlab/mmaction2/pull/572))

- Refactor unittest structure ([#433](https://github.com/open-mmlab/mmaction2/pull/433))

- Fix a bug about ava dataset validation ([#527](https://github.com/open-mmlab/mmaction2/pull/527))
- Fix a bug about ResNet pretrain weight initialization ([#582](https://github.com/open-mmlab/mmaction2/pull/582))
- Fix a bug in CI due to MMCV index ([#495](https://github.com/open-mmlab/mmaction2/pull/495))
- Remove invalid links of MiT and MMiT ([#516](https://github.com/open-mmlab/mmaction2/pull/516))
- Fix frame rate bug for AVA preparation ([#576](https://github.com/open-mmlab/mmaction2/pull/576))

## 0.10.0 (31/12/2020)

- Support Spatio-Temporal Action Detection (AVA)
- Support precise BN

- Support precise BN ([#501](https://github.com/open-mmlab/mmaction2/pull/501/))

- Support Spatio-Temporal Action Detection (AVA) ([#351](https://github.com/open-mmlab/mmaction2/pull/351))
- Support to return feature maps in `inference_recognizer` ([#458](https://github.com/open-mmlab/mmaction2/pull/458))

- Add arg `stride` to long_video_demo.py, to make inference faster ([#468](https://github.com/open-mmlab/mmaction2/pull/468))
- Support training and testing for Spatio-Temporal Action Detection ([#351](https://github.com/open-mmlab/mmaction2/pull/351))
- Fix CI due to pip upgrade ([#454](https://github.com/open-mmlab/mmaction2/pull/454))
- Add markdown lint in pre-commit hook ([#255](https://github.com/open-mmlab/mmaction2/pull/225))
- Speed up confusion matrix calculation ([#465](https://github.com/open-mmlab/mmaction2/pull/465))

- Use title case in modelzoo statistics ([#456](https://github.com/open-mmlab/mmaction2/pull/456))
- Add FAQ documents for easy troubleshooting. ([#413](https://github.com/open-mmlab/mmaction2/pull/413), [#420](https://github.com/open-mmlab/mmaction2/pull/420), [#439](https://github.com/open-mmlab/mmaction2/pull/439))
- Support Spatio-Temporal Action Detection with context ([#471](https://github.com/open-mmlab/mmaction2/pull/471))
- Add class weight for CrossEntropyLoss and BCELossWithLogits ([#509](https://github.com/open-mmlab/mmaction2/pull/509))
- Add Lazy OPs docs ([#504](https://github.com/open-mmlab/mmaction2/pull/504))

- Fix typo in default argument of BaseHead ([#446](https://github.com/open-mmlab/mmaction2/pull/446))

- Fix potential bug about `output_config` overwrite ([#463](https://github.com/open-mmlab/mmaction2/pull/463))

- Add SlowOnly, SlowFast for AVA2.1 ([#351](https://github.com/open-mmlab/mmaction2/pull/351))

## 0.9.0 (30/11/2020)

- Support GradCAM utils for recognizers
- Support ResNet Audio model

- Automatically add modelzoo statistics to readthedocs ([#327](https://github.com/open-mmlab/mmaction2/pull/327))
- Support GYM99 ([#331](https://github.com/open-mmlab/mmaction2/pull/331), [#336](https://github.com/open-mmlab/mmaction2/pull/336))
- Add AudioOnly Pathway from AVSlowFast. ([#355](https://github.com/open-mmlab/mmaction2/pull/355))
- Add GradCAM utils for recognizer ([#324](https://github.com/open-mmlab/mmaction2/pull/324))

- Add print config script ([#345](https://github.com/open-mmlab/mmaction2/pull/345))
- Add online motion vector decoder ([#291](https://github.com/open-mmlab/mmaction2/pull/291))

- Support PyTorch 1.7 in CI ([#312](https://github.com/open-mmlab/mmaction2/pull/312))
- Support to predict different labels in a long video ([#274](https://github.com/open-mmlab/mmaction2/pull/274))
- Update docs bout test crops ([#359](https://github.com/open-mmlab/mmaction2/pull/359))
- Polish code format using pylint manually ([#338](https://github.com/open-mmlab/mmaction2/pull/338))
- Update unittest coverage ([#358](https://github.com/open-mmlab/mmaction2/pull/358), [#322](https://github.com/open-mmlab/mmaction2/pull/322), [#325](https://github.com/open-mmlab/mmaction2/pull/325))

- Add random seed for building filelists ([#323](https://github.com/open-mmlab/mmaction2/pull/323))
- Update colab tutorial ([#367](https://github.com/open-mmlab/mmaction2/pull/367))
- set default batch_size of evaluation and testing to 1 ([#250](https://github.com/open-mmlab/mmaction2/pull/250))
- Rename the preparation docs to `README.md` ([#388](https://github.com/open-mmlab/mmaction2/pull/388))
- Move docs about demo to `demo/README.md` ([#329](https://github.com/open-mmlab/mmaction2/pull/329))
- Remove redundant code in `tools/test.py` ([#310](https://github.com/open-mmlab/mmaction2/pull/310))
- Automatically calculate number of test clips for Recognizer2D ([#359](https://github.com/open-mmlab/mmaction2/pull/359))

- Fix rename Kinetics classnames bug ([#384](https://github.com/open-mmlab/mmaction2/pull/384))
- Fix a bug in BaseDataset when `data_prefix` is None ([#314](https://github.com/open-mmlab/mmaction2/pull/314))

- Fix a bug about `tmp_folder` in `OpenCVInit` ([#357](https://github.com/open-mmlab/mmaction2/pull/357))
- Fix `get_thread_id` when not using disk as backend ([#354](https://github.com/open-mmlab/mmaction2/pull/354), [#357](https://github.com/open-mmlab/mmaction2/pull/357))
- Fix the bug of HVU object `num_classes` from 1679 to 1678 ([#307](https://github.com/open-mmlab/mmaction2/pull/307))
- Fix typo in `export_model.md` ([#399](https://github.com/open-mmlab/mmaction2/pull/399))
- Fix OmniSource training configs ([#321](https://github.com/open-mmlab/mmaction2/pull/321))
- Fix Issue #306: Bug of SampleAVAFrames ([#317](https://github.com/open-mmlab/mmaction2/pull/317))

- Add SlowOnly model for GYM99, both RGB and Flow ([#336](https://github.com/open-mmlab/mmaction2/pull/336))

- Add auto modelzoo statistics in readthedocs ([#327](https://github.com/open-mmlab/mmaction2/pull/327))
- Add TSN for HMDB51 pretrained on Kinetics400, Moments in Time and ImageNet. ([#372](https://github.com/open-mmlab/mmaction2/pull/372))

## v0.8.0 (31/10/2020)

- Support [OmniSource](https://arxiv.org/abs/2003.13042)
- Support C3D
- Support video recognition with audio modality
- Support HVU
- Support X3D

- Support AVA dataset preparation ([#266](https://github.com/open-mmlab/mmaction2/pull/266))
- Support the training of video recognition dataset with multiple tag categories ([#235](https://github.com/open-mmlab/mmaction2/pull/235))
- Support joint training with multiple training datasets of multiple formats, including images, untrimmed videos, etc. ([#242](https://github.com/open-mmlab/mmaction2/pull/242))

- Support to specify a start epoch to conduct evaluation ([#216](https://github.com/open-mmlab/mmaction2/pull/216))
- Implement X3D models, support testing with model weights converted from SlowFast ([#288](https://github.com/open-mmlab/mmaction2/pull/288))
- Support specify a start epoch to conduct evaluation ([#216](https://github.com/open-mmlab/mmaction2/pull/216))

- Set default values of 'average_clips' in each config file so that there is no need to set it explicitly during testing in most cases ([#232](https://github.com/open-mmlab/mmaction2/pull/232))
- Extend HVU datatools to generate individual file list for each tag category ([#258](https://github.com/open-mmlab/mmaction2/pull/258))

- Support data preparation for Kinetics-600 and Kinetics-700 ([#254](https://github.com/open-mmlab/mmaction2/pull/254))
- Use `metric_dict` to replace hardcoded arguments in `evaluate` function ([#286](https://github.com/open-mmlab/mmaction2/pull/286))
- Add `cfg-options` in arguments to override some settings in the used config for convenience ([#212](https://github.com/open-mmlab/mmaction2/pull/212))
- Rename the old evaluating protocol `mean_average_precision` as `mmit_mean_average_precision` since it is only used on MMIT and is not the `mAP` we usually talk about. Add `mean_average_precision`, which is the real `mAP` ([#235](https://github.com/open-mmlab/mmaction2/pull/235))
- Add accurate setting (Three crop * 2 clip) and report corresponding performance for TSM model ([#241](https://github.com/open-mmlab/mmaction2/pull/241))

- Add citations in each preparing_dataset.md in `tools/data/dataset` ([#289](https://github.com/open-mmlab/mmaction2/pull/289))
- Update the performance of audio-visual fusion on Kinetics-400 ([#281](https://github.com/open-mmlab/mmaction2/pull/281))
- Support data preparation of OmniSource web datasets, including GoogleImage, InsImage, InsVideo and KineticsRawVideo ([#294](https://github.com/open-mmlab/mmaction2/pull/294))
- Use `metric_options` dict to provide metric args in `evaluate` ([#286](https://github.com/open-mmlab/mmaction2/pull/286))

- Register `FrameSelector` in `PIPELINES` ([#268](https://github.com/open-mmlab/mmaction2/pull/268))
- Fix the potential bug for default value in dataset_setting ([#245](https://github.com/open-mmlab/mmaction2/pull/245))

- Fix multi-node dist test ([#292](https://github.com/open-mmlab/mmaction2/pull/292))
- Fix the data preparation bug for `something-something` dataset ([#278](https://github.com/open-mmlab/mmaction2/pull/278))
- Fix the invalid config url in slowonly README data benchmark ([#249](https://github.com/open-mmlab/mmaction2/pull/249))
- Validate that the performance of models trained with videos have no significant difference comparing to the performance of models trained with rawframes ([#256](https://github.com/open-mmlab/mmaction2/pull/256))
- Correct the `img_norm_cfg` used by TSN-3seg-R50 UCF-101 model, improve the Top-1 accuracy by 3% ([#273](https://github.com/open-mmlab/mmaction2/pull/273))

- Add Baselines for Kinetics-600 and Kinetics-700, including TSN-R50-8seg and SlowOnly-R50-8x8 ([#259](https://github.com/open-mmlab/mmaction2/pull/259))

- Add OmniSource benchmark on MiniKineitcs ([#296](https://github.com/open-mmlab/mmaction2/pull/296))
- Add Baselines for HVU, including TSN-R18-8seg on 6 tag categories of HVU ([#287](https://github.com/open-mmlab/mmaction2/pull/287))
- Add X3D models ported from [SlowFast](https://github.com/facebookresearch/SlowFast/) ([#288](https://github.com/open-mmlab/mmaction2/pull/288))

## v0.7.0 (30/9/2020)

- Support TPN
- Support JHMDB, UCF101-24, HVU dataset preparation
- support onnx model conversion

- Support the data pre-processing pipeline for the HVU Dataset ([#277](https://github.com/open-mmlab/mmaction2/pull/227/))
- Support real-time action recognition from web camera ([#171](https://github.com/open-mmlab/mmaction2/pull/171))

- Support onnx ([#160](https://github.com/open-mmlab/mmaction2/pull/160))
- Support UCF101-24 preparation ([#219](https://github.com/open-mmlab/mmaction2/pull/219))
- Support evaluating mAP for ActivityNet with [CUHK17_activitynet_pred](http://activity-net.org/challenges/2017/evaluation.html) ([#176](https://github.com/open-mmlab/mmaction2/pull/176))
- Add the data pipeline for ActivityNet, including downloading videos, extracting RGB and Flow frames, finetuning TSN and extracting feature ([#190](https://github.com/open-mmlab/mmaction2/pull/190))
- Support JHMDB preparation ([#220](https://github.com/open-mmlab/mmaction2/pull/220))

- Add finetuning setting for SlowOnly ([#173](https://github.com/open-mmlab/mmaction2/pull/173))

- Add TSN and SlowOnly models trained with [OmniSource](https://arxiv.org/abs/2003.13042), which achieve 75.7% Top-1 with TSN-R50-3seg and 80.4% Top-1 with SlowOnly-R101-8x8 ([#215](https://github.com/open-mmlab/mmaction2/pull/215))

- Support demo with video url ([#165](https://github.com/open-mmlab/mmaction2/pull/165))
- Support multi-batch when testing ([#184](https://github.com/open-mmlab/mmaction2/pull/184))
- Add tutorial for adding a new learning rate updater ([#181](https://github.com/open-mmlab/mmaction2/pull/181))
- Add config name in meta info ([#183](https://github.com/open-mmlab/mmaction2/pull/183))
- Remove git hash in `__version__` ([#189](https://github.com/open-mmlab/mmaction2/pull/189))

- Check mmcv version ([#189](https://github.com/open-mmlab/mmaction2/pull/189))
- Update url with 'https://download.openmmlab.com' ([#208](https://github.com/open-mmlab/mmaction2/pull/208))
- Update Docker file to support PyTorch 1.6 and update `install.md` ([#209](https://github.com/open-mmlab/mmaction2/pull/209))
- Polish readsthedocs display ([#217](https://github.com/open-mmlab/mmaction2/pull/217), [#229](https://github.com/open-mmlab/mmaction2/pull/229))

- Fix the bug when using OpenCV to extract only RGB frames with original shape ([#184](https://github.com/open-mmlab/mmaction2/pull/187))
- Fix the bug of sthv2 `num_classes` from 339 to 174 ([#174](https://github.com/open-mmlab/mmaction2/pull/174), [#207](https://github.com/open-mmlab/mmaction2/pull/207))

## v0.6.0 (2/9/2020)

- Support TIN, CSN, SSN, NonLocal
- Support FP16 training

- Support NonLocal module and provide ckpt in TSM and I3D ([#41](https://github.com/open-mmlab/mmaction2/pull/41))
- Support SSN ([#33](https://github.com/open-mmlab/mmaction2/pull/33), [#37](https://github.com/open-mmlab/mmaction2/pull/37), [#52](https://github.com/open-mmlab/mmaction2/pull/52), [#55](https://github.com/open-mmlab/mmaction2/pull/55))
- Support CSN ([#87](https://github.com/open-mmlab/mmaction2/pull/87))
- Support TIN ([#53](https://github.com/open-mmlab/mmaction2/pull/53))
- Support HMDB51 dataset preparation ([#60](https://github.com/open-mmlab/mmaction2/pull/60))
- Support encoding videos from frames ([#84](https://github.com/open-mmlab/mmaction2/pull/84))

- Support FP16 training ([#25](https://github.com/open-mmlab/mmaction2/pull/25))
- Enhance demo by supporting rawframe inference ([#59](https://github.com/open-mmlab/mmaction2/pull/59)), output video/gif ([#72](https://github.com/open-mmlab/mmaction2/pull/72))

- Update Slowfast modelzoo ([#51](https://github.com/open-mmlab/mmaction2/pull/51))
- Update TSN, TSM video checkpoints ([#50](https://github.com/open-mmlab/mmaction2/pull/50))
- Add data benchmark for TSN ([#57](https://github.com/open-mmlab/mmaction2/pull/57))
- Add data benchmark for SlowOnly ([#77](https://github.com/open-mmlab/mmaction2/pull/77))
- Add BSN/BMN performance results with feature extracted by our codebase ([#99](https://github.com/open-mmlab/mmaction2/pull/99))

- Polish data preparation codes ([#70](https://github.com/open-mmlab/mmaction2/pull/70))
- Improve data preparation scripts ([#58](https://github.com/open-mmlab/mmaction2/pull/58))
- Improve unittest coverage and minor fix ([#62](https://github.com/open-mmlab/mmaction2/pull/62))
- Support PyTorch 1.6 in CI ([#117](https://github.com/open-mmlab/mmaction2/pull/117))
- Support `with_offset` for rawframe dataset ([#48](https://github.com/open-mmlab/mmaction2/pull/48))
- Support json annotation files ([#119](https://github.com/open-mmlab/mmaction2/pull/119))
- Support `multi-class` in TSMHead ([#104](https://github.com/open-mmlab/mmaction2/pull/104))
- Support using `val_step()` to validate data for each `val` workflow ([#123](https://github.com/open-mmlab/mmaction2/pull/123))

- Use `xxInit()` method to get `total_frames` and make `total_frames` a required key ([#90](https://github.com/open-mmlab/mmaction2/pull/90))
- Add paper introduction in model readme ([#140](https://github.com/open-mmlab/mmaction2/pull/140))
- Adjust the directory structure of `tools/` and rename some scripts files ([#142](https://github.com/open-mmlab/mmaction2/pull/142))

- Fix configs for localization test ([#67](https://github.com/open-mmlab/mmaction2/pull/67))
- Fix configs of SlowOnly by fixing lr to 8 gpus ([#136](https://github.com/open-mmlab/mmaction2/pull/136))
- Fix the bug in analyze_log ([#54](https://github.com/open-mmlab/mmaction2/pull/54))
- Fix the bug of generating HMDB51 class index file ([#69](https://github.com/open-mmlab/mmaction2/pull/69))

- Fix the bug of using `load_checkpoint()` in ResNet ([#93](https://github.com/open-mmlab/mmaction2/pull/93))
- Fix the bug of `--work-dir` when using slurm training script ([#110](https://github.com/open-mmlab/mmaction2/pull/110))
- Correct the sthv1/sthv2 rawframes filelist generate command ([#71](https://github.com/open-mmlab/mmaction2/pull/71))
- `CosineAnnealing` typo ([#47](https://github.com/open-mmlab/mmaction2/pull/47))

## v0.5.0 (9/7/2020)

- MMAction2 is released

- Support various datasets: UCF101, Kinetics-400, Something-Something V1&V2, Moments in Time,
  Multi-Moments in Time, THUMOS14
- Support various action recognition methods: TSN, TSM, R(2+1)D, I3D, SlowOnly, SlowFast, Non-local
- Support various action localization methods: BSN, BMN

Multi-Moments in Time, THUMOS14
- Colab demo for action recognition

# NPU (HUAWEI Ascend)

## Usage

### General Usage

Please refer to the [building documentation of MMCV](https://mmcv.readthedocs.io/en/latest/get_started/build.html#build-mmcv-full-on-ascend-npu-machine) to install MMCV and [MMEngine](https://mmengine.readthedocs.io/en/latest/get_started/installation.html#build-from-source) on NPU devices.

Here we use 8 NPUs on your computer to train the model with the following command:

```shell
bash ./tools/dist_train.sh configs/resnet/resnet50_8xb32_in1k.py 8
```

Also, you can use only one NPU to train the model with the following command:

```shell
python ./tools/train.py configs/resnet/resnet50_8xb32_in1k.py
```

## Models Results

|                            Model                            | Top-1 (%) | Top-5 (%) |                            Config                            |                            Download                             |

| :---------------------------------------------------------: | :-------: | :-------: | :----------------------------------------------------------: | :-------------------------------------------------------------: |
| [ResNet-50](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/resnet/README.md) |   76.40   |   93.21   | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/resnet/resnet50_8xb32_in1k.py) | [log](https://download.openmmlab.com/mmclassification/v1/device/npu/resnet50_8xb32_in1k.log) |
| [ResNetXt-32x4d-50](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/resnext/README.md) |   77.48   |   93.75   | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/resnext/resnext50-32x4d_8xb32_in1k.py) | [log](https://download.openmmlab.com/mmclassification/v1/device/npu/resnext50-32x4d_8xb32_in1k.log) |

| [HRNet-W18](https://github.com/open-mmlab/mmclassification/blob/master/configs/hrnet/README.md) |   77.06   |   93.57   | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/hrnet/hrnet-w18_4xb32_in1k.py) | [log](https://download.openmmlab.com/mmclassification/v1/device/npu/hrnet-w18_4xb32_in1k.log) |
| [ResNetV1D-152](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/resnet/README.md) |   79.41   |   94.48   | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/resnet/resnetv1d152_8xb32_in1k.py) | [log](https://download.openmmlab.com/mmclassification/v1/device/npu/resnetv1d152_8xb32_in1k.log) |
| [SE-ResNet-50](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/seresnet/README.md) |   77.65   |   93.74   | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/seresnet/seresnet50_8xb32_in1k.py) | [log](https://download.openmmlab.com/mmclassification/v1/device/npu/seresnet50_8xb32_in1k.log) |

| [ShuffleNetV2 1.0x](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/shufflenet_v2/README.md) |   69.52   |   88.79   | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/shufflenet_v2/shufflenet-v2-1x_16xb64_in1k.py) | [log](https://download.openmmlab.com/mmclassification/v1/device/npu/shufflenet-v2-1x_16xb64_in1k.log) |
| [MobileNetV2](https://github.com/open-mmlab/mmclassification/tree/1.x/configs/mobilenet_v2) |   71.74   |   90.28   | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/mobilenet_v2/mobilenet-v2_8xb32_in1k.py) | [log](https://download.openmmlab.com/mmclassification/v1/device/npu/mobilenet-v2_8xb32_in1k.log) |

| [MobileNetV3-Small](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/mobilenet_v3/README.md) |   67.09   |   87.17   | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/mobilenet_v3/mobilenet-v3-small_8xb128_in1k.py) | [log](https://download.openmmlab.com/mmclassification/v1/device/npu/mobilenet-v3-small.log) |
| [\*CSPResNeXt50](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/cspnet/README.md) |   77.25   |   93.46   | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/cspnet/cspresnext50_8xb32_in1k.py) | [log](https://download.openmmlab.com/mmclassification/v1/device/npu/cspresnext50_8xb32_in1k.log) |

| [\*EfficientNet-B4](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/efficientnet/README.md) |   75.73   |   92.91   | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/efficientnet/efficientnet-b4_8xb32_in1k.py) | [log](https://download.openmmlab.com/mmclassification/v1/device/npu/efficientnet-b4_8xb32_in1k.log) |
| [\*\*DenseNet121](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/densenet/README.md) |   72.53   |   90.85   | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/densenet/densenet121_4xb256_in1k.py) | [log](https://download.openmmlab.com/mmclassification/v1/device/npu/densenet121_4xb256_in1k.log) |

**Notes:**

- If not specially marked, the results are almost same between results on the NPU and results on the GPU with FP32.
- (\*) The training results of these models are lower than the results on the readme in the corresponding model, mainly

  because the results on the readme are directly the weight of the timm of the eval, and the results on this side are
  retrained according to the config with mmcls. The results of the config training on the GPU are consistent with the
  results of the NPU.
- (\*\*) The accuracy of this model is slightly lower because config is a 4-card config, we use 8 cards to run, and users
  can adjust hyperparameters to get the best accuracy results.

**All above models are provided by Huawei Ascend group.**

# GLIP: Grounded Language-Image Pre-training

> [GLIP: Grounded Language-Image Pre-training](https://arxiv.org/abs/2112.03857)

This paper presents a grounded language-image pre-training (GLIP) model for learning object-level, language-aware, and semantic-rich visual representations. GLIP unifies object detection and phrase grounding for pre-training. The unification brings two benefits: 1) it allows GLIP to learn from both detection and grounding data to improve both tasks and bootstrap a good grounding model; 2) GLIP can leverage massive image-text pairs by generating grounding boxes in a self-training fashion, making the learned representation semantic-rich. In our experiments, we pre-train GLIP on 27M grounding data, including 3M human-annotated and 24M web-crawled image-text pairs. The learned representations demonstrate strong zero-shot and few-shot transferability to various object-level recognition tasks. 1) When directly evaluated on COCO and LVIS (without seeing any images in COCO during pre-training), GLIP achieves 49.8 AP and 26.9 AP, respectively, surpassing many supervised baselines. 2) After fine-tuned on COCO, GLIP achieves 60.8 AP on val and 61.5 AP on test-dev, surpassing prior SoTA. 3) When transferred to 13 downstream object detection tasks, a 1-shot GLIP rivals with a fully-supervised Dynamic Head.

<img src="https://github.com/open-mmlab/mmyolo/assets/17425982/b87228d7-f000-4a5d-b103-fe535984417a"/>
</div>

```shell
cd $MMDETROOT

# source installation

# or mim installation
```

```shell

wget https://download.openmmlab.com/mmdetection/v3.0/glip/glip_tiny_a_mmdet-b3654169.pth

python demo/image_demo.py demo/demo.jpg \
configs/glip/glip_atss_swin-t_a_fpn_dyhead_pretrain_obj365.py \
glip_tiny_a_mmdet-b3654169.pth \
--texts 'bench . car .'
```

<img src="https://github.com/open-mmlab/mmdetection/assets/17425982/de370086-a5ae-4b77-8cbd-4592abf4afb0" width="40%"/>
</div>

|   Model    | Zero-shot or Funetune | COCO mAP |       Pre-Train Data       |                                 Config                                  |                                           Download                                           |

| :--------: | :-------------------: | :------: | :------------------------: | :---------------------------------------------------------------------: | :------------------------------------------------------------------------------------------: |
| GLIP-T (A) |       Zero-shot       |   43.0   |            O365            |       [config](glip_atss_swin-t_a_fpn_dyhead_pretrain_obj365.py)        | [model](https://download.openmmlab.com/mmdetection/v3.0/glip/glip_tiny_a_mmdet-b3654169.pth) |
| GLIP-T (B) |       Zero-shot       |   44.9   |            O365            |       [config](glip_atss_swin-t_b_fpn_dyhead_pretrain_obj365.py)        | [model](https://download.openmmlab.com/mmdetection/v3.0/glip/glip_tiny_b_mmdet-6dfbd102.pth) |
| GLIP-T (C) |       Zero-shot       |   46.7   |         O365,GoldG         |    [config](glip_atss_swin-t_c_fpn_dyhead_pretrain_obj365-goldg.py)     | [model](https://download.openmmlab.com/mmdetection/v3.0/glip/glip_tiny_c_mmdet-2fc427dd.pth) |

|   GLIP-T   |       Zero-shot       |   46.4   |    O365,GoldG,CC3M,SBU     | [config](glip_atss_swin-t_fpn_dyhead_pretrain_obj365-goldg-cc3m-sub.py) |  [model](https://download.openmmlab.com/mmdetection/v3.0/glip/glip_tiny_mmdet-c24ce662.pth)  |
|   GLIP-L   |       Zero-shot       |   51.3   | FourODs,GoldG,CC3M+12M,SBU |       [config](glip_atss_swin-l_fpn_dyhead_pretrain_mixeddata.py)       |   [model](https://download.openmmlab.com/mmdetection/v3.0/glip/glip_l_mmdet-abfe026b.pth)    |

Note:

1. The weights corresponding to the zero-shot model are adopted from the official weights and converted using the [script](../../tools/model_converters/glip_to_mmdet.py). We have not retrained the model for the time being.
2. We will soon support fine-tuning on COCO.

# 数据集可视化

## 数据集可视化工具简介

```bash
python tools/visualization/browse_dataset.py \
    ${CONFIG_FILE} \
    [-o, --output-dir ${OUTPUT_DIR}] \
    [-p, --phase ${DATASET_PHASE}] \
    [-n, --show-number ${NUMBER_IMAGES_DISPLAY}] \
    [-i, --show-interval ${SHOW_INTERRVAL}] \
    [-m, --mode ${DISPLAY_MODE}] \
    [-r, --rescale-factor ${RESCALE_FACTOR}] \
    [-c, --channel-order ${CHANNEL_ORDER}] \
    [--cfg-options ${CFG_OPTIONS}]
```

**所有参数的说明**：

- `config` : 模型配置文件的路径。
- `-o, --output-dir`: 保存图片文件夹，如果没有指定，默认为 `''`,表示不保存。
- **`-p, --phase`**: 可视化数据集的阶段，只能为 `['train', 'val', 'test']` 之一，默认为 `'train'`。
- **`-n, --show-number`**: 可视化样本数量。如果没有指定，默认展示数据集的所有图片。
- `-i, --show-interval`: 浏览时，每张图片的停留间隔，单位为秒。
- **`-m, --mode`**: 可视化的模式，只能为 `['original', 'transformed', 'concat', 'pipeline']` 之一。 默认为`'transformed'`.
- `-r, --rescale-factor`: 在 `mode='original'` 下，可视化图片的放缩倍数，在图片过大或过小时设置。
- `-c, --channel-order`: 图片的通道顺序，为  `['BGR', 'RGB']` 之一，默认为 `'BGR'`。

- `--cfg-options` : 对配置文件的修改，参考[学习配置文件](../user_guides/config.md)。

```{note}

1. `-m, --mode` 用于设置可视化的模式，默认设置为 'transformed'。
- 如果 `--mode` 设置为 'original'，则获取原始图片；
- 如果 `--mode` 设置为 'transformed'，则获取预处理后的图片；
- 如果 `--mode` 设置为 'concat'，获取原始图片和预处理后图片拼接的图片；
- 如果 `--mode` 设置为 'pipeline'，则获得数据流水线所有中间过程图片。

2. `-r, --rescale-factor` 在数据集中图片的分辨率过大或者过小时设置。比如在可视化 CIFAR 数据集时，由于图片的分辨率非常小，可将 `-r, --rescale-factor` 设置为 10。
```

## 如何可视化原始图像

使用 **'original'** 模式 ：

```shell
python ./tools/visualization/browse_dataset.py ./configs/resnet/resnet101_8xb16_cifar10.py --phase val --output-dir tmp --mode original --show-number 100 --rescale-factor 10 --channel-order RGB
```

- `--phase val`: 可视化验证集，可简化为 `-p val`;
- `--output-dir tmp`: 可视化结果保存在 "tmp" 文件夹，可简化为 `-o tmp`;
- `--mode original`: 可视化原图，可简化为 `-m original`;
- `--show-number 100`: 可视化 100 张图，可简化为 `-n 100`;

```

- `--rescale-factor`: 图像放大 10 倍，可简化为 `-r 10`;
- `--channel-order RGB`: 可视化图像的通道顺序为 "RGB", 可简化为 `-c RGB`。

<div align=center><img src="https://user-images.githubusercontent.com/18586273/190993839-216a7a1e-590e-47b9-92ae-08f87a7d58df.jpg" style=" width: auto; height: 40%; "></div>

## 如何可视化处理后图像

使用 **'transformed'** 模式：

```shell
python ./tools/visualization/browse_dataset.py ./configs/resnet/resnet50_8xb32_in1k.py -n 100
```

<div align=center><img src="https://user-images.githubusercontent.com/18586273/190994696-737b09d9-d0fb-4593-94a2-4487121e0286.JPEG" style=" width: auto; height: 40%; "></div>

## 如何同时可视化原始图像与处理后图像

使用 **'concat'** 模式：

```shell
python ./tools/visualization/browse_dataset.py configs/swin_transformer/swin-small_16xb64_in1k.py -n 10 -m concat
```

```shell
```

<div align=center><img src="https://user-images.githubusercontent.com/18586273/190995078-3872feb2-d4e2-4727-a21b-7062d52f7d3e.JPEG" style=" width: auto; height: 40%; "></div>

使用 **'pipeline'** 模式：

```shell
python ./tools/visualization/browse_dataset.py configs/swin_transformer/swin-small_16xb64_in1k.py -m pipeline
```

<div align=center><img src="https://user-images.githubusercontent.com/18586273/190995525-fac0220f-6630-4013-b94a-bc6de4fdff7a.JPEG" style=" width: auto; height: 40%; "></div>

```shell
python ./tools/visualization/browse_dataset.py configs/beit/beit_beit-base-p16_8xb256-amp-coslr-300e_in1k.py -m pipeline
```

<div align=center><img src="https://user-images.githubusercontent.com/26739999/226542300-74216187-e3d0-4a6e-8731-342abe719721.png" style=" width: auto; height: 40%; "></div>

# Learn about Configs

MMDetection and other OpenMMLab repositories use [MMEngine's config system](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html). It has a modular and inheritance design, which is convenient to conduct various experiments.

## Config file content

MMDetection uses a modular design, all modules with different functions can be configured through the config. Taking Mask R-CNN as an example, we will introduce each field in the config according to different function modules:

### Model config

In MMDetection's config, we use `model` to set up detection algorithm components. In addition to neural network components such as `backbone`, `neck`, etc, it also requires `data_preprocessor`, `train_cfg`, and `test_cfg`. `data_preprocessor` is responsible for processing a batch of data output by dataloader. `train_cfg`, and `test_cfg` in the model config are for training and testing hyperparameters of the components.

```python

```python
    type='MaskRCNN',  # The name of detector
    data_preprocessor=dict(  # The config of data preprocessor, usually includes image normalization and padding
        type='DetDataPreprocessor',  # The type of the data preprocessor, refer to https://mmdetection.readthedocs.io/en/latest/api.html#mmdet.models.data_preprocessors.DetDataPreprocessor
        mean=[123.675, 116.28, 103.53],  # Mean values used to pre-training the pre-trained backbone models, ordered in R, G, B
        std=[58.395, 57.12, 57.375],  # Standard variance used to pre-training the pre-trained backbone models, ordered in R, G, B
        bgr_to_rgb=True,  # whether to convert image from BGR to RGB
        pad_mask=True,  # whether to pad instance masks
        pad_size_divisor=32),  # The size of padded image should be divisible by ``pad_size_divisor``
    backbone=dict(  # The config of backbone

pad_size_divisor=32),  # The size of padded image should be divisible by ``pad_size_divisor``
        type='ResNet',  # The type of backbone network. Refer to https://mmdetection.readthedocs.io/en/latest/api.html#mmdet.models.backbones.ResNet
        depth=50,  # The depth of backbone, usually it is 50 or 101 for ResNet and ResNext backbones.
        num_stages=4,  # Number of stages of the backbone.
        out_indices=(0, 1, 2, 3),  # The index of output feature maps produced in each stage
        frozen_stages=1,  # The weights in the first stage are frozen
        norm_cfg=dict(  # The config of normalization layers.
            type='BN',  # Type of norm layer, usually it is BN or GN
            requires_grad=True),  # Whether to train the gamma and beta in BN
        norm_eval=True,  # Whether to freeze the statistics in BN

requires_grad=True),  # Whether to train the gamma and beta in BN
        style='pytorch', # The style of backbone, 'pytorch' means that stride 2 layers are in 3x3 Conv, 'caffe' means stride 2 layers are in 1x1 Convs.
    	init_cfg=dict(type='Pretrained', checkpoint='torchvision://resnet50')),  # The ImageNet pretrained backbone to be loaded
    neck=dict(
        type='FPN',  # The neck of detector is FPN. We also support 'NASFPN', 'PAFPN', etc. Refer to https://mmdetection.readthedocs.io/en/latest/api.html#mmdet.models.necks.FPN for more details.
        in_channels=[256, 512, 1024, 2048],  # The input channels, this is consistent with the output channels of backbone
        out_channels=256,  # The output channels of each level of the pyramid feature map
        num_outs=5),  # The number of output scales
    rpn_head=dict(

out_channels=256,  # The output channels of each level of the pyramid feature map
        type='RPNHead',  # The type of RPN head is 'RPNHead', we also support 'GARPNHead', etc. Refer to https://mmdetection.readthedocs.io/en/latest/api.html#mmdet.models.dense_heads.RPNHead for more details.
        in_channels=256,  # The input channels of each input feature map, this is consistent with the output channels of neck
        feat_channels=256,  # Feature channels of convolutional layers in the head.
        anchor_generator=dict(  # The config of anchor generator
            type='AnchorGenerator',  # Most of methods use AnchorGenerator, SSD Detectors uses `SSDAnchorGenerator`. Refer to https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/task_modules/prior_generators/anchor_generator.py#L18 for more details

scales=[8],  # Basic scale of the anchor, the area of the anchor in one position of a feature map will be scale * base_sizes
            ratios=[0.5, 1.0, 2.0],  # The ratio between height and width.
            strides=[4, 8, 16, 32, 64]),  # The strides of the anchor generator. This is consistent with the FPN feature strides. The strides will be taken as base_sizes if base_sizes is not set.
        bbox_coder=dict(  # Config of box coder to encode and decode the boxes during training and testing
            type='DeltaXYWHBBoxCoder',  # Type of box coder. 'DeltaXYWHBBoxCoder' is applied for most of the methods. Refer to https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/task_modules/coders/delta_xywh_bbox_coder.py#L13 for more details.
            target_means=[0.0, 0.0, 0.0, 0.0],  # The target means used to encode and decode boxes
            target_stds=[1.0, 1.0, 1.0, 1.0]),  # The standard variance used to encode and decode boxes

target_stds=[1.0, 1.0, 1.0, 1.0]),  # The standard variance used to encode and decode boxes
        loss_cls=dict(  # Config of loss function for the classification branch
            type='CrossEntropyLoss',  # Type of loss for classification branch, we also support FocalLoss etc. Refer to https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/losses/cross_entropy_loss.py#L201 for more details
            use_sigmoid=True,  # RPN usually performs two-class classification, so it usually uses the sigmoid function.
            loss_weight=1.0),  # Loss weight of the classification branch.
        loss_bbox=dict(  # Config of loss function for the regression branch.
            type='L1Loss',  # Type of loss, we also support many IoU Losses and smooth L1-loss, etc. Refer to https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/losses/smooth_l1_loss.py#L56 for implementation.
            loss_weight=1.0)),  # Loss weight of the regression branch.

loss_weight=1.0)),  # Loss weight of the regression branch.
    roi_head=dict(  # RoIHead encapsulates the second stage of two-stage/cascade detectors.
        type='StandardRoIHead',
        bbox_roi_extractor=dict(  # RoI feature extractor for bbox regression.
            type='SingleRoIExtractor',  # Type of the RoI feature extractor, most of methods uses SingleRoIExtractor. Refer to https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/roi_heads/roi_extractors/single_level_roi_extractor.py#L13 for details.
            roi_layer=dict(  # Config of RoI Layer
                type='RoIAlign',  # Type of RoI Layer, DeformRoIPoolingPack and ModulatedDeformRoIPoolingPack are also supported. Refer to https://mmcv.readthedocs.io/en/latest/api.html#mmcv.ops.RoIAlign for details.
                output_size=7,  # The output size of feature maps.
                sampling_ratio=0),  # Sampling ratio when extracting the RoI features. 0 means adaptive ratio.

output_size=7,  # The output size of feature maps.
            out_channels=256,  # output channels of the extracted feature.
            featmap_strides=[4, 8, 16, 32]),  # Strides of multi-scale feature maps. It should be consistent with the architecture of the backbone.
        bbox_head=dict(  # Config of box head in the RoIHead.
            type='Shared2FCBBoxHead',  # Type of the bbox head, Refer to https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/roi_heads/bbox_heads/convfc_bbox_head.py#L220 for implementation details.
            in_channels=256,  # Input channels for bbox head. This is consistent with the out_channels in roi_extractor
            fc_out_channels=1024,  # Output feature channels of FC layers.
            roi_feat_size=7,  # Size of RoI features
            num_classes=80,  # Number of classes for classification

fc_out_channels=1024,  # Output feature channels of FC layers.
            bbox_coder=dict(  # Box coder used in the second stage.
                type='DeltaXYWHBBoxCoder',  # Type of box coder. 'DeltaXYWHBBoxCoder' is applied for most of the methods.
                target_means=[0.0, 0.0, 0.0, 0.0],  # Means used to encode and decode box
                target_stds=[0.1, 0.1, 0.2, 0.2]),  # Standard variance for encoding and decoding. It is smaller since the boxes are more accurate. [0.1, 0.1, 0.2, 0.2] is a conventional setting.
            reg_class_agnostic=False,  # Whether the regression is class agnostic.
            loss_cls=dict(  # Config of loss function for the classification branch
                type='CrossEntropyLoss',  # Type of loss for classification branch, we also support FocalLoss etc.

loss_cls=dict(  # Config of loss function for the classification branch
                use_sigmoid=False,  # Whether to use sigmoid.
                loss_weight=1.0),  # Loss weight of the classification branch.
            loss_bbox=dict(  # Config of loss function for the regression branch.
                type='L1Loss',  # Type of loss, we also support many IoU Losses and smooth L1-loss, etc.
                loss_weight=1.0)),  # Loss weight of the regression branch.
        mask_roi_extractor=dict(  # RoI feature extractor for mask generation.
            type='SingleRoIExtractor',  # Type of the RoI feature extractor, most of methods uses SingleRoIExtractor.
            roi_layer=dict(  # Config of RoI Layer that extracts features for instance segmentation

roi_layer=dict(  # Config of RoI Layer that extracts features for instance segmentation
                type='RoIAlign',  # Type of RoI Layer, DeformRoIPoolingPack and ModulatedDeformRoIPoolingPack are also supported
                output_size=14,  # The output size of feature maps.
                sampling_ratio=0),  # Sampling ratio when extracting the RoI features.
            out_channels=256,  # Output channels of the extracted feature.
            featmap_strides=[4, 8, 16, 32]),  # Strides of multi-scale feature maps.
        mask_head=dict(  # Mask prediction head
            type='FCNMaskHead',  # Type of mask head, refer to https://mmdetection.readthedocs.io/en/latest/api.html#mmdet.models.roi_heads.FCNMaskHead for implementation details.
            num_convs=4,  # Number of convolutional layers in mask head.
            in_channels=256,  # Input channels, should be consistent with the output channels of mask roi extractor.

num_convs=4,  # Number of convolutional layers in mask head.
            conv_out_channels=256,  # Output channels of the convolutional layer.
            num_classes=80,  # Number of class to be segmented.
            loss_mask=dict(  # Config of loss function for the mask branch.
                type='CrossEntropyLoss',  # Type of loss used for segmentation
                use_mask=True,  # Whether to only train the mask in the correct class.
                loss_weight=1.0))),  # Loss weight of mask branch.
    train_cfg = dict(  # Config of training hyperparameters for rpn and rcnn
        rpn=dict(  # Training config of rpn
            assigner=dict(  # Config of assigner

train_cfg = dict(  # Config of training hyperparameters for rpn and rcnn
                type='MaxIoUAssigner',  # Type of assigner, MaxIoUAssigner is used for many common detectors. Refer to https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/task_modules/assigners/max_iou_assigner.py#L14 for more details.
                pos_iou_thr=0.7,  # IoU >= threshold 0.7 will be taken as positive samples
                neg_iou_thr=0.3,  # IoU < threshold 0.3 will be taken as negative samples
                min_pos_iou=0.3,  # The minimal IoU threshold to take boxes as positive samples
                match_low_quality=True,  # Whether to match the boxes under low quality (see API doc for more details).
                ignore_iof_thr=-1),  # IoF threshold for ignoring bboxes
            sampler=dict(  # Config of positive/negative sampler

ignore_iof_thr=-1),  # IoF threshold for ignoring bboxes
                type='RandomSampler',  # Type of sampler, PseudoSampler and other samplers are also supported. Refer to https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/task_modules/samplers/random_sampler.py#L14 for implementation details.
                num=256,  # Number of samples
                pos_fraction=0.5,  # The ratio of positive samples in the total samples.
                neg_pos_ub=-1,  # The upper bound of negative samples based on the number of positive samples.
                add_gt_as_proposals=False),  # Whether add GT as proposals after sampling.
            allowed_border=-1,  # The border allowed after padding for valid anchors.
            pos_weight=-1,  # The weight of positive samples during training.
            debug=False),  # Whether to set the debug mode

pos_weight=-1,  # The weight of positive samples during training.
        rpn_proposal=dict(  # The config to generate proposals during training
            nms_across_levels=False,  # Whether to do NMS for boxes across levels. Only work in `GARPNHead`, naive rpn does not support do nms cross levels.
            nms_pre=2000,  # The number of boxes before NMS
            nms_post=1000,  # The number of boxes to be kept by NMS. Only work in `GARPNHead`.
            max_per_img=1000,  # The number of boxes to be kept after NMS.
            nms=dict( # Config of NMS
                type='nms',  # Type of NMS
                iou_threshold=0.7 # NMS threshold
                ),
            min_bbox_size=0),  # The allowed minimal box size
        rcnn=dict(  # The config for the roi heads.
            assigner=dict(  # Config of assigner for second stage, this is different for that in rpn

rcnn=dict(  # The config for the roi heads.
                type='MaxIoUAssigner',  # Type of assigner, MaxIoUAssigner is used for all roi_heads for now. Refer to https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/task_modules/assigners/max_iou_assigner.py#L14 for more details.
                pos_iou_thr=0.5,  # IoU >= threshold 0.5 will be taken as positive samples
                neg_iou_thr=0.5,  # IoU < threshold 0.5 will be taken as negative samples
                min_pos_iou=0.5,  # The minimal IoU threshold to take boxes as positive samples
                match_low_quality=False,  # Whether to match the boxes under low quality (see API doc for more details).
            sampler=dict(

                num=512,  # Number of samples
                pos_fraction=0.25,  # The ratio of positive samples in the total samples.
                add_gt_as_proposals=True
            ),  # Whether add GT as proposals after sampling.
            mask_size=28,  # Size of mask
            debug=False)),  # Whether to set the debug mode
    test_cfg = dict(  # Config for testing hyperparameters for rpn and rcnn

debug=False)),  # Whether to set the debug mode
        rpn=dict(  # The config to generate proposals during testing
            nms_pre=1000,  # The number of boxes before NMS
                type='nms',  #Type of NMS
                ),
            score_thr=0.05,  # Threshold to filter out boxes
            nms=dict(  # Config of NMS in the second stage

                iou_thr=0.5),  # NMS threshold
            max_per_img=100,  # Max number of detections of each image
            mask_thr_binary=0.5)))  # Threshold of mask prediction
```

### Dataset and evaluator config

[Dataloaders](https://mmengine.readthedocs.io/en/latest/tutorials/dataset.html) are required for the training, validation, and testing of the [runner](https://mmengine.readthedocs.io/en/latest/tutorials/runner.html). Dataset and data pipeline need to be set to build the dataloader. Due to the complexity of this part, we use intermediate variables to simplify the writing of dataloader configs.

```python
dataset_type = 'CocoDataset'  # Dataset type, this will be used to define the dataset
data_root = 'data/coco/'  # Root path of data

```python
backend_args = None # Arguments to instantiate the corresponding file backend

train_pipeline = [  # Training data processing pipeline
    dict(type='LoadImageFromFile', backend_args=backend_args),  # First pipeline to load images from file path
    dict(
        type='LoadAnnotations',  # Second pipeline to load annotations for current image
        with_bbox=True,  # Whether to use bounding box, True for detection
        with_mask=True,  # Whether to use instance mask, True for instance segmentation
        poly2mask=True),  # Whether to convert the polygon mask to instance mask, set False for acceleration and to save memory
    dict(
        type='Resize',  # Pipeline that resizes the images and their annotations
        scale=(1333, 800),  # The largest scale of the images

dict(
        keep_ratio=True  # Whether to keep the ratio between height and width
        ),
    dict(
        type='RandomFlip',  # Augmentation pipeline that flips the images and their annotations
        prob=0.5),  # The probability to flip
    dict(type='PackDetInputs')  # Pipeline that formats the annotation data and decides which keys in the data should be packed into data_samples
]
test_pipeline = [  # Testing data processing pipeline
    dict(type='Resize', scale=(1333, 800), keep_ratio=True),  # Pipeline that resizes the images
    dict(
        type='PackDetInputs',  # Pipeline that formats the annotation data and decides which keys in the data should be packed into data_samples

dict(
]
train_dataloader = dict(   # Train dataloader config
    batch_size=2,  # Batch size of a single GPU
    num_workers=2,  # Worker to pre-fetch data for each single GPU
    persistent_workers=True,  # If ``True``, the dataloader will not shut down the worker processes after an epoch end, which can accelerate training speed.
    sampler=dict(  # training data sampler
        type='DefaultSampler',  # DefaultSampler which supports both distributed and non-distributed training. Refer to https://mmengine.readthedocs.io/en/latest/api/generated/mmengine.dataset.DefaultSampler.html#mmengine.dataset.DefaultSampler
        shuffle=True),  # randomly shuffle the training data in each epoch

shuffle=True),  # randomly shuffle the training data in each epoch
    batch_sampler=dict(type='AspectRatioBatchSampler'),  # Batch sampler for grouping images with similar aspect ratio into a same batch. It can reduce GPU memory cost.
    dataset=dict(  # Train dataset config
        ann_file='annotations/instances_train2017.json',  # Path of annotation file
        data_prefix=dict(img='train2017/'),  # Prefix of image path
        filter_cfg=dict(filter_empty_gt=True, min_size=32),  # Config of filtering images and annotations
        pipeline=train_pipeline,
        backend_args=backend_args))
val_dataloader = dict(  # Validation dataloader config
    batch_size=1,  # Batch size of a single GPU. If batch-size > 1, the extra padding area may influence the performance.

batch_size=1,  # Batch size of a single GPU. If batch-size > 1, the extra padding area may influence the performance.
    drop_last=False,  # Whether to drop the last incomplete batch, if the dataset size is not divisible by the batch size
    sampler=dict(
        type='DefaultSampler',
        shuffle=False),  # not shuffle during validation and testing
        test_mode=True,  # Turn on the test mode of the dataset to avoid filtering annotations or images
        pipeline=test_pipeline,
test_dataloader = val_dataloader  # Testing dataloader config
```

pipeline=test_pipeline,
```

[Evaluators](https://mmengine.readthedocs.io/en/latest/tutorials/evaluation.html) are used to compute the metrics of the trained model on the validation and testing datasets. The config of evaluators consists of one or a list of metric configs:

```python
val_evaluator = dict(  # Validation evaluator config
    type='CocoMetric',  # The coco metric used to evaluate AR, AP, and mAP for detection and instance segmentation
    ann_file=data_root + 'annotations/instances_val2017.json',  # Annotation file path
    metric=['bbox', 'segm'],  # Metrics to be evaluated, `bbox` for detection and `segm` for instance segmentation
    format_only=False,
    backend_args=backend_args)
test_evaluator = val_evaluator  # Testing evaluator config
```

format_only=False,
```

Since the test dataset has no annotation files, the test_dataloader and test_evaluator config in MMDetection are generally equal to the val's. If you want to save the detection results on the test dataset, you can write the config like this:

```python
# inference on test dataset and
# format the output results for submission.
test_dataloader = dict(
        ann_file=data_root + 'annotations/image_info_test-dev2017.json',
        data_prefix=dict(img='test2017/'),
test_evaluator = dict(
    ann_file=data_root + 'annotations/image_info_test-dev2017.json',

test_mode=True,
    metric=['bbox', 'segm'],  # Metrics to be evaluated
    format_only=True,  # Only format and save the results to coco json file
    outfile_prefix='./work_dirs/coco_detection/test')  # The prefix of output json files
```

### Training and testing config

MMEngine's runner uses Loop to control the training, validation, and testing processes.
Users can set the maximum training epochs and validation intervals with these fields.

```python
    type='EpochBasedTrainLoop',  # The training loop type. Refer to https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py
    max_epochs=12,  # Maximum training epochs
    val_interval=1)  # Validation intervals. Run validation every epoch.
val_cfg = dict(type='ValLoop')  # The validation loop type

max_epochs=12,  # Maximum training epochs
test_cfg = dict(type='TestLoop')  # The testing loop type
```

### Optimization config

`optim_wrapper` is the field to configure optimization-related settings. The optimizer wrapper not only provides the functions of the optimizer, but also supports functions such as gradient clipping, mixed precision training, etc. Find more in [optimizer wrapper tutorial](https://mmengine.readthedocs.io/en/latest/tutorials/optim_wrapper.html).

```python
optim_wrapper = dict(  # Optimizer wrapper config
    type='OptimWrapper',  # Optimizer wrapper type, switch to AmpOptimWrapper to enable mixed precision training.
    optimizer=dict(  # Optimizer config. Support all kinds of optimizers in PyTorch. Refer to https://pytorch.org/docs/stable/optim.html#algorithms
        type='SGD',  # Stochastic gradient descent optimizer

type='SGD',  # Stochastic gradient descent optimizer
        lr=0.02,  # The base learning rate
        momentum=0.9,  # Stochastic gradient descent with momentum
        weight_decay=0.0001),  # Weight decay of SGD
    clip_grad=None,  # Gradient clip option. Set None to disable gradient clip. Find usage in https://mmengine.readthedocs.io/en/latest/tutorials/optimizer.html
    )
```

`param_scheduler` is a field that configures methods of adjusting optimization hyperparameters such as learning rate and momentum. Users can combine multiple schedulers to create a desired parameter adjustment strategy. Find more in [parameter scheduler tutorial](https://mmengine.readthedocs.io/en/latest/tutorials/param_scheduler.html) and [parameter scheduler API documents](https://mmengine.readthedocs.io/en/latest/api/generated/mmengine.optim._ParamScheduler.html#mmengine.optim._ParamScheduler)

```python
    # Linear learning rate warm-up scheduler
    dict(

```python
    dict(
        type='LinearLR',  # Use linear policy to warmup learning rate
        start_factor=0.001, # The ratio of the starting learning rate used for warmup
        by_epoch=False,  # The warmup learning rate is updated by iteration
        begin=0,  # Start from the first iteration
        end=500),  # End the warmup at the 500th iteration
    # The main LRScheduler
    dict(
        type='MultiStepLR',  # Use multi-step learning rate policy during training
        by_epoch=True,  # The learning rate is updated by epoch
        begin=0,   # Start from the first epoch
        end=12,  # End at the 12th epoch
        milestones=[8, 11],  # Epochs to decay the learning rate
        gamma=0.1)  # The learning rate decay ratio
]
```

### Hook config

end=12,  # End at the 12th epoch
]
```

Users can attach Hooks to training, validation, and testing loops to insert some operations during running. There are two different hook fields, one is `default_hooks` and the other is `custom_hooks`.

`default_hooks` is a dict of hook configs, and they are the hooks must be required at the runtime. They have default priority which should not be modified. If not set, runner will use the default values. To disable a default hook, users can set its config to `None`. Find more in [HOOK](https://mmengine.readthedocs.io/en/latest/tutorials/hook.html).

```python
    timer=dict(type='IterTimerHook'),  # Update the time spent during iteration into message hub

```python
    logger=dict(type='LoggerHook', interval=50),  # Collect logs from different components of Runner and write them to terminal, JSON file, tensorboard and wandb .etc
    param_scheduler=dict(type='ParamSchedulerHook'), # update some hyper-parameters of optimizer
    checkpoint=dict(type='CheckpointHook', interval=1), # Save checkpoints periodically
    sampler_seed=dict(type='DistSamplerSeedHook'),  # Ensure distributed Sampler shuffle is active
    visualization=dict(type='DetVisualizationHook'))  # Detection Visualization Hook. Used to visualize validation and testing process prediction results
```

`custom_hooks` is a list of all other hook configs. Users can develop their own hooks and insert them in this field.

```python
custom_hooks = []
```

### Runtime config

```python

```

```python
```

```python
default_scope = 'mmdet'  # The default registry scope to find modules. Refer to https://mmengine.readthedocs.io/en/latest/advanced_tutorials/registry.html

    cudnn_benchmark=False,  # Whether to enable cudnn benchmark
    mp_cfg=dict(  # Multi-processing config
        mp_start_method='fork',  # Use fork to start multi-processing threads. 'fork' usually faster than 'spawn' but maybe unsafe. See discussion in https://github.com/pytorch/pytorch/issues/1355
        opencv_num_threads=0),  # Disable opencv multi-threads to avoid system being overloaded
    dist_cfg=dict(backend='nccl'),  # Distribution configs
)

vis_backends = [dict(type='LocalVisBackend')]  # Visualization backends. Refer to https://mmengine.readthedocs.io/en/latest/advanced_tutorials/visualization.html

)

    type='DetLocalVisualizer', vis_backends=vis_backends, name='visualizer')
    type='LogProcessor',  # Log processor to process runtime logs
    window_size=50,  # Smooth interval of log values
    by_epoch=True)  # Whether to format logs with epoch type. Should be consistent with the train loop's type.

log_level = 'INFO'  # The level of logging.
load_from = None  # Load model checkpoint as a pre-trained model from a given path. This will not resume training.
resume = False  # Whether to resume from the checkpoint defined in `load_from`. If `load_from` is None, it will resume the latest checkpoint in the `work_dir`.
```

## Iter-based config

MMEngine's Runner also provides an iter-based training loop except for epoch-based.

```

To use iter-based training, users should modify the `train_cfg`, `param_scheduler`, `train_dataloader`, `default_hooks`, and `log_processor`.
Here is an example of changing an epoch-based RetinaNet config to iter-based: `configs/retinanet/retinanet_r50_fpn_90k_coco.py`

```python
# Iter-based training config
    _delete_=True,  # Ignore the base config setting (optional)
    type='IterBasedTrainLoop',  # Use iter-based training loop
    max_iters=90000,  # Maximum iterations
    val_interval=10000)  # Validation interval

# Change the scheduler to iter-based
    dict(
        type='LinearLR', start_factor=0.001, by_epoch=False, begin=0, end=500),
    dict(
        type='MultiStepLR',
        begin=0,
        end=90000,
        milestones=[60000, 80000],
        gamma=0.1)
]

dict(
        begin=0,
]

# Switch to InfiniteSampler to avoid dataloader restart
train_dataloader = dict(sampler=dict(type='InfiniteSampler'))

# Change the checkpoint saving interval to iter-based
default_hooks = dict(checkpoint=dict(by_epoch=False, interval=10000))

# Change the log format to iter-based
log_processor = dict(by_epoch=False)
```

## Config file inheritance

There are 4 basic component types under `config/_base_`, dataset, model, schedule, default_runtime.
Many methods could be easily constructed with one of these models like Faster R-CNN, Mask R-CNN, Cascade R-CNN, RPN, SSD.
The configs that are composed by components from `_base_` are called the _primitive_.

For all configs under the same folder, it is recommended to have only **one** _primitive_ config. All other configs should inherit from the _primitive_ config. In this way, the maximum of inheritance level is 3.

For easy understanding, we recommend contributors to inherit from existing methods.
For example, if some modification is made based on Faster R-CNN, users may first inherit the basic Faster R-CNN structure by specifying `_base_ = ../faster_rcnn/faster-rcnn_r50_fpn_1x_coco.py`, then modify the necessary fields in the config files.

If you are building an entirely new method that does not share the structure with any of the existing methods, you may create a folder `xxx_rcnn` under `configs`,

Please refer to [mmengine config tutorial](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html) for detailed documentation.

By setting the `_base_` field, we can set which files the current configuration file inherits from.

When `_base_` is a string of a file path, it means inheriting the contents from one config file.

```python
_base_ = './mask-rcnn_r50_fpn_1x_coco.py'
```

When `_base_` is a list of multiple file paths, it means inheriting from multiple files.

```python
_base_ = [
    '../_base_/models/mask-rcnn_r50_fpn.py',
    '../_base_/datasets/coco_instance.py',
    '../_base_/schedules/schedule_1x.py', '../_base_/default_runtime.py'
]
```

If you wish to inspect the config file, you may run `python tools/misc/print_config.py /PATH/TO/CONFIG` to see the complete config.

### Ignore some fields in the base configs

Sometimes, you may set `_delete_=True` to ignore some of the fields in base configs.

You may refer to [mmengine config tutorial](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html) for a simple illustration.

In MMDetection, for example, to change the backbone of Mask R-CNN with the following config.

```python
    type='MaskRCNN',
        type='ResNet',
        depth=50,
        num_stages=4,
        out_indices=(0, 1, 2, 3),
        frozen_stages=1,
        norm_cfg=dict(type='BN', requires_grad=True),
        norm_eval=True,
        style='pytorch',
        init_cfg=dict(type='Pretrained', checkpoint='torchvision://resnet50')),
    neck=dict(...),
    rpn_head=dict(...),
    roi_head=dict(...))
```

`ResNet` and `HRNet` use different keywords to construct.

```python
_base_ = '../mask_rcnn/mask-rcnn_r50_fpn_1x_coco.py'
        _delete_=True,

```

```python
        type='HRNet',
        extra=dict(
            stage1=dict(
                num_modules=1,
                num_branches=1,
                block='BOTTLENECK',
                num_blocks=(4, ),
                num_channels=(64, )),
            stage2=dict(
                num_branches=2,
                block='BASIC',
                num_blocks=(4, 4),
                num_channels=(32, 64)),
            stage3=dict(
                num_modules=4,
                num_branches=3,
                num_blocks=(4, 4, 4),
                num_channels=(32, 64, 128)),
            stage4=dict(
                num_modules=3,
                num_branches=4,
                num_blocks=(4, 4, 4, 4),

stage4=dict(
                num_channels=(32, 64, 128, 256))),
        init_cfg=dict(type='Pretrained', checkpoint='open-mmlab://msra/hrnetv2_w32')),
    neck=dict(...))
```

The `_delete_=True` would replace all old keys in `backbone` field with new keys.

### Use intermediate variables in configs

Some intermediate variables are used in the configs files, like `train_pipeline`/`test_pipeline` in datasets.
It's worth noting that when modifying intermediate variables in the children configs, users need to pass the intermediate variables into corresponding fields again.
For example, we would like to use a multi-scale strategy to train a Mask R-CNN. `train_pipeline`/`test_pipeline` are intermediate variables we would like to modify.

```python

```python

    dict(type='LoadAnnotations', with_bbox=True, with_mask=True),
    dict(
        type='RandomResize', scale=[(1333, 640), (1333, 800)],
        keep_ratio=True),
]
    dict(
]
train_dataloader = dict(dataset=dict(pipeline=train_pipeline))
val_dataloader = dict(dataset=dict(pipeline=test_pipeline))
test_dataloader = dict(dataset=dict(pipeline=test_pipeline))
```

We first define the new `train_pipeline`/`test_pipeline` and pass them into dataloader fields.

```

Similarly, if we would like to switch from `SyncBN` to `BN` or `MMSyncBN`, we need to substitute every `norm_cfg` in the config.

```python
norm_cfg = dict(type='BN', requires_grad=True)
    backbone=dict(norm_cfg=norm_cfg),
    neck=dict(norm_cfg=norm_cfg),
    ...)
```

### Reuse variables in \_base\_ file

If the users want to reuse the variables in the base file, they can get a copy of the corresponding variable by using `{{_base_.xxx}}`. E.g:

```python

a = {{_base_.model}} # Variable `a` is equal to the `model` defined in `_base_`
```

## Modify config through script arguments

When submitting jobs using `tools/train.py` or `tools/test.py`, you may specify `--cfg-options` to in-place modify the config.

```

- Update config keys of dict chains.

  The config options can be specified following the order of the dict keys in the original config.
  For example, `--cfg-options model.backbone.norm_eval=False` changes the all BN modules in model backbones to `train` mode.

- Update keys inside a list of configs.

  Some config dicts are composed as a list in your config. For example, the training pipeline `train_dataloader.dataset.pipeline` is normally a list
  e.g. `[dict(type='LoadImageFromFile'), ...]`. If you want to change `'LoadImageFromFile'` to `'LoadImageFromNDArray'` in the pipeline,
  you may specify `--cfg-options data.train.pipeline.0.type=LoadImageFromNDArray`.

- Update values of list/tuples.

you may specify `--cfg-options data.train.pipeline.0.type=LoadImageFromNDArray`.

  If the value to be updated is a list or a tuple. For example, the config file normally sets `model.data_preprocessor.mean=[123.675, 116.28, 103.53]`. If you want to
  change the mean values, you may specify `--cfg-options model.data_preprocessor.mean="[127,127,127]"`. Note that the quotation mark `"` is necessary to
  support list/tuple data types, and **NO** white space is allowed inside the quotation marks in the specified value.

## Config name style

We follow the below style to name config files. Contributors are advised to follow the same style.

```
{algorithm name}_{model component names [component1]_[component2]_[...]}_{training settings}_{training dataset information}_{testing dataset information}.py
```

The file name is divided into five parts. All parts and components are connected with `_` and words of each part or component should be connected with `-`.

```

- `{algorithm name}`: The name of the algorithm. It can be a detector name such as `faster-rcnn`, `mask-rcnn`, etc. Or can be a semi-supervised or knowledge-distillation algorithm such as `soft-teacher`, `lad`. etc.
- `{model component names}`: Names of the components used in the algorithm such as backbone, neck, etc. For example, `r50-caffe_fpn_gn-head` means using caffe-style ResNet50, FPN and detection head with Group Norm in the algorithm.
- `{training settings}`: Information of training settings such as batch size, augmentations, loss trick, scheduler, and epochs/iterations. For example: `4xb4-mixup-giou-coslr-100e` means using 8-gpus x 4-images-per-gpu, mixup augmentation, GIoU loss, cosine annealing learning rate, and train 100 epochs.
  Some abbreviations:

Some abbreviations:
  - `{gpu x batch_per_gpu}`: GPUs and samples per GPU. `bN` indicates N batch size per GPU. E.g. `4xb4` is the short term of 4-GPUs x 4-images-per-GPU. And `8xb2` is used by default if not mentioned.
  - `{schedule}`: training schedule, options are `1x`, `2x`, `20e`, etc.
    `1x` and `2x` means 12 epochs and 24 epochs respectively.
    `20e` is adopted in cascade models, which denotes 20 epochs.
    For `1x`/`2x`, the initial learning rate decays by a factor of 10 at the 8/16th and 11/22th epochs.
    For `20e`, the initial learning rate decays by a factor of 10 at the 16th and 19th epochs.
- `{training dataset information}`: Training dataset names like `coco`, `coco-panoptic`, `cityscapes`, `voc-0712`, `wider-face`.
- `{testing dataset information}` (optional): Testing dataset name for models trained on one dataset but tested on another. If not mentioned, it means the model was trained and tested on the same dataset type.

---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: ''
assignees: ''
---

**Describe the feature**

**Motivation**
A clear and concise description of the motivation of the feature.
Ex1. It is inconvenient when \[....\].
Ex2. There is a recent paper \[....\], which is very helpful for \[....\].

**Related resources**
If there is an official code release or third-party implementations, please also provide the information here, which would be very helpful.

**Additional context**
Add any other context or screenshots about the feature request here.
If you would like to implement the feature and create a PR, please leave a comment here and that would be much appreciated.

# 数据元素与数据结构

MMOCR 基于 {external+mmengine:doc}`MMEngine: 抽象数据接口 <advanced_tutorials/data_element>` 将各任务所需的数据统一封装入 `data_sample` 中。MMEngine 的抽象数据接口实现了基础的增/删/改/查功能，且支持不同设备间的数据迁移，也支持了类字典和张量的操作，充分满足了数据的日常使用需求，这也使得不同算法的数据接口可以得到统一。

得益于统一的数据封装，算法库内的 [`visualizer`](./visualizers.md)，[`evaluator`](./evaluation.md)，[`dataset`](./datasets.md) 等各个模块间的数据流通都得到了极大的简化。在 MMOCR 中，我们对数据接口类型作出以下约定：

- **xxxData**: 单一粒度的数据标注或模型输出。目前 MMEngine 内置了三种粒度的{external+mmengine:doc}`数据元素 <advanced_tutorials/data_element>`，包括实例级数据（`InstanceData`），像素级数据（`PixelData`）以及图像级的标签数据（`LabelData`）。在 MMOCR 目前支持的任务中，文本检测以及关键信息抽取任务使用 `InstanceData` 来封装文本实例的检测框及对应标签，而文本识别任务则使用了 `LabelData` 来封装文本内容。

- **xxxDataSample**: 继承自 {external+mmengine:doc}`MMEngine: 数据基类 <advanced_tutorials/data_element>` `BaseDataElement`，用于保存单个任务的训练或测试样本的**所有**标注及预测信息。如文本检测任务的数据样本类 [`TextDetDataSample`](mmocr.structures.textdet_data_sample.TextDetDataSample)，文本识别任务的数据样本类 [`TextRecogDataSample`](mmocr.structures.textrecog_data_sample.TextRecogDataSample)，以及关键信息抽任务的数据样本类 [`KIEDataSample`](mmocr.structures.kie_data_sample.KIEDataSample)。

下面，我们将分别介绍数据元素 **xxxData** 与数据样本 **xxxDataSample** 在 MMOCR 中的实际应用。

## 数据元素 xxxData

`InstanceData` 和 `LabelData` 是 `MMEngine`中定义的基础数据元素，用于封装不同粒度的标注数据或模型输出。在 MMOCR 中，我们针对不同任务中实际使用的数据类型，分别采用了 `InstanceData` 与 `LabelData` 进行了封装。

### InstanceData

在**文本检测**任务中，检测器关注的是实例级别的文字样本，因此我们使用 `InstanceData` 来封装该任务所需的数据。其所需的训练标注和预测输出通常包含了矩形或多边形边界盒，以及边界盒标签。由于文本检测任务只有一种正样本类，即 “text”，在 MMOCR 中我们默认使用 `0` 来编号该类别。以下代码示例展示了如何使用 `InstanceData` 数据抽象接口来封装文本检测任务中使用的数据类型。

```python
from mmengine.structures import InstanceData

# 定义 gt_instance 用于封装边界盒的标注信息

```python

gt_instance = InstanceData()
gt_instance.bbox = torch.Tensor([[0, 0, 10, 10], [10, 10, 20, 20]])
gt_instance.polygons = torch.Tensor([[[0, 0], [10, 0], [10, 10], [0, 10]],
                                    [[10, 10], [20, 10], [20, 20], [10, 20]]])
gt_instance.label = torch.Tensor([0, 0])

# 定义 pred_instance 用于封装模型的输出信息
pred_instances = InstanceData()
pred_polygons, scores = model(input)
pred_instances.polygons = pred_polygons
pred_instances.scores = scores
```

MMOCR 中对 `InstanceData` 字段的约定如下表所示。值得注意的是，`InstanceData` 中的各字段的长度必须为与样本中的实例个数 `N` 相等。

|             |                                    |                                                                                                                                      |

|             |                                    |                                                                                                                                      |
| ----------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| 字段        | 类型                               | 说明                                                                                                                                 |
| bboxes      | `torch.FloatTensor`                | 文本边界框 `[x1, y1, x2, y2]`，形状为 `(N, 4)`。                                                                                     |
| labels      | `torch.LongTensor`                 | 实例的类别，长度为 `(N, )`。MMOCR 中默认使用 `0` 来表示正样本类，即 “text” 类。                                                      |

| polygons    | `list[np.array(dtype=np.float32)]` | 表示文本实例的多边形，列表长度为 `(N, )`。                                                                                           |
| scores      | `torch.Tensor`                     | 文本实例检测框的置信度，长度为 `(N, )`。                                                                                             |
| ignored     | `torch.BoolTensor`                 | 是否在训练中忽略当前文本实例，长度为 `(N, )`。                                                                                       |
| texts       | `list[str]`                        | 实例对应的文本，长度为 `(N, )`，用于端到端 OCR 任务和 KIE。                                                                          |

| text_scores | `torch.FloatTensor`                | 文本预测的置信度，长度为`(N, )`，用于端到端 OCR 任务。                                                                               |
| edge_labels | `torch.IntTensor`                  | 节点的邻接矩阵，形状为 `(N, N)`。在 KIE 任务中，节点之间状态的可选值为 `-1` （忽略，不参与 loss 计算），`0` （断开）和 `1`（连接）。 |
| edge_scores | `torch.FloatTensor`                | 用于 KIE 任务中每条边的预测置信度，形状为 `(N, N)`。                                                                                 |

### LabelData

对于**文字识别**任务，标注内容和预测内容都会使用 `LabelData` 进行封装。

```python
from mmengine.data import LabelData

# 定义一个 gt_text 用于封装标签文本内容
gt_text = LabelData()
gt_text.item = 'MMOCR'

# 定义一个 pred_text 对象用于封装预测文本以及置信度
pred_text = LabelData()
index, score = model(input)
text = dictionary.idx2str(index)
pred_text.score = score
pred_text.item = text

```

MMOCR 中对 `LabelData` 字段的约定如下表所示：

|                |                    |                                                                                                                            |
| -------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------- |
| 字段           | 类型               | 说明                                                                                                                       |
| item           | `str`              | 文本内容。                                                                                                                 |

| score          | `list[float]`      | 预测的文本内容的置信度。                                                                                                   |
| indexes        | `torch.LongTensor` | 文本字符经过[字典](../basic_concepts/models.md#dictionary)编码后的序列，且包含了除 `<UNK>` 以外的所有特殊字符。            |
| padded_indexes | `torch.LongTensor` | 如果 indexes 的长度小于最大序列长度，且 `pad_idx` 存在时，该字段保存了填充至最大序列长度 `max_seq_len`的编码后的文本序列。 |

## 数据样本 xxxDataSample

通过定义统一的数据结构，我们可以方便地将标注数据和预测结果进行统一封装，使代码库不同模块间的数据传递更加便捷。在 MMOCR 中，我们基于现在支持的三个任务及其所需要的数据分别封装了三种数据抽象，包括文本检测任务数据抽象 [`TextDetDataSample`](mmocr.structures.textdet_data_sample.TextDetDataSample)，文本识别任务数据抽象 [`TextRecogDataSample`](mmocr.structures.textrecog_data_sample.TextRecogDataSample)，以及关键信息抽取任务数据抽象 [`KIEDataSample`](mmocr.structures.kie_data_sample.KIEDataSample)。这些数据抽象均继承自 {external+mmengine:doc}`MMEngine: 数据基类 <advanced_tutorials/data_element>` `BaseDataElement`，用于保存单个任务的训练或测试样本的所有标注及预测信息。

### 文本检测任务数据抽象 TextDetDataSample

[TextDetDataSample](mmocr.structures.textdet_data_sample.TextDetDataSample) 用于封装文字检测任务所需的数据，其主要包含了两个字段 `gt_instances` 与 `pred_instances`，分别用于存放标注信息与预测结果。

|                |                                 |            |
| -------------- | ------------------------------- | ---------- |

|                |                                 |            |
| 字段           | 类型                            | 说明       |
| gt_instances   | [`InstanceData`](#instancedata) | 标注信息。 |
| pred_instances | [`InstanceData`](#instancedata) | 预测结果。 |

其中会用到的 [`InstanceData`](#instancedata) 约定字段有：

|          |                                    |                                                                                  |
| -------- | ---------------------------------- | -------------------------------------------------------------------------------- |
| 字段     | 类型                               | 说明                                                                             |
| bboxes   | `torch.FloatTensor`                | 文本边界框 `[x1, y1, x2, y2]`，形状为 `(N, 4)`。                                 |
| labels   | `torch.LongTensor`                 | 实例的类别，长度为 `(N, )`。在 MMOCR 中通常使用 `0` 来表示正样本类，即 “text” 类 |

| polygons | `list[np.array(dtype=np.float32)]` | 表示文本实例的多边形，列表长度为 `(N, )`。                                       |
| scores   | `torch.Tensor`                     | 文本实例任务预测的检测框的置信度，长度为 `(N, )`。                               |
| ignored  | `torch.BoolTensor`                 | 是否在训练中忽略当前文本实例，长度为 `(N, )`。                                   |

由于文本检测模型通常只会输出 bboxes/polygons 中的一项，因此我们只需确保这两项中的一个被赋值即可。

以下示例代码展示了 `TextDetDataSample` 的使用方法：

```python
from mmengine.data import TextDetDataSample

data_sample = TextDetDataSample()
# 指定当前图片的标注信息
img_meta = dict(img_shape=(800, 1196, 3), pad_shape=(800, 1216, 3))
gt_instances = InstanceData(metainfo=img_meta)
gt_instances.bboxes = torch.rand((5, 4))
gt_instances.labels = torch.zeros((5,), dtype=torch.long)
data_sample.gt_instances = gt_instances

# 指定当前图片的预测信息

pred_instances.bboxes = torch.rand((5, 4))
pred_instances.labels = torch.zeros((5,), dtype=torch.long)
data_sample.pred_instances = pred_instances
```

### 文本识别任务数据抽象 TextRecogDataSample

[`TextRecogDataSample`](mmocr.structures.textrecog_data_sample.TextRecogDataSample) 用于封装文字识别任务的数据。它有两个属性，`gt_text` 和 `pred_text` , 分别用于存放标注信息和预测结果。

|           |                           |            |
| --------- | ------------------------- | ---------- |
| 字段      | 类型                      | 说明       |
| gt_text   | [`LabelData`](#labeldata) | 标注信息。 |
| pred_text | [`LabelData`](#labeldata) | 预测结果。 |

以下示例代码展示了 [`TextRecogDataSample`](mmocr.structures.textrecog_data_sample.TextRecogDataSample) 的使用方法：

```python
from mmengine.data import TextRecogDataSample

data_sample = TextRecogDataSample()

```python

gt_text = LabelData(metainfo=img_meta)
gt_text.item = 'mmocr'
data_sample.gt_text = gt_text

# 指定当前图片的预测结果
pred_text = LabelData(metainfo=img_meta)
pred_text.item = 'mmocr'
data_sample.pred_text = pred_text
```

其中会用到的 `LabelData` 字段有：

|                |                     |                                                                                                                            |
| -------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| 字段           | 类型                | 说明                                                                                                                       |

| item           | `list[str]`         | 实例对应的文本，长度为 (N, ) ，用于端到端 OCR 任务和 KIE                                                                   |
| score          | `torch.FloatTensor` | 文本预测的置信度，长度为 (N, )，用于端到端 OCR 任务                                                                        |
| indexes        | `torch.LongTensor`  | 文本字符经过[字典](../basic_concepts/models.md#dictionary)编码后的序列，且包含了除 `<UNK>` 以外的所有特殊字符。            |
| padded_indexes | `torch.LongTensor`  | 如果 indexes 的长度小于最大序列长度，且 `pad_idx` 存在时，该字段保存了填充至最大序列长度 `max_seq_len`的编码后的文本序列。 |

### 关键信息抽取任务数据抽象 KIEDataSample

[`KIEDataSample`](mmocr.structures.kie_data_sample.KIEDataSample) 用于封装 KIE 任务所需的数据，其同样约定了两个属性，即 `gt_instances` 与 `pred_instances`，分别用于存放标注信息与预测结果。

|                |                                 |            |

|                |                                 |            |

该任务会用到的 [`InstanceData`](#instancedata) 字段如下表所示：

|             |                     |                                                                                                                                               |
| ----------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| 字段        | 类型                | 说明                                                                                                                                          |

| bboxes      | `torch.Tensor`      | 文本边界框 `[x1, y1, x2, y2]`，形状为 `(N, 4)`。                                                                                              |
| labels      | `torch.LongTensor`  | 实例的类别，长度为 `(N, )`。在 MMOCR 中通常为 0，即 “text” 类。                                                                               |
| texts       | `list[str]`         | 实例对应的文本，长度为 `(N, )` ，用于端到端 OCR 任务和 KIE 任务。                                                                             |
| edge_labels | `torch.IntTensor`   | 节点之间的邻接矩阵，形状为 `(N, N)`。在 KIE 任务中，节点之间状态的可选值为 `-1` （不关心，且不参与 loss 计算），`0` （断开）和 `1` （连接）。 |
| edge_scores | `torch.FloatTensor` | 每条边的预测置信度，形状为 `(N, N)`。                                                                                                         |

| scores      | `torch.FloatTensor` | 节点标签的预测置信度, 形状为 `(N,)`。                                                                                                         |

```{warning}
由于 KIE 任务的模型实现尚未有统一标准，该设计目前仅考虑了 [SDMGR](../../../configs/kie/sdmgr/README.md) 模型的使用场景。因此，该设计有可能在我们支持更多 KIE 模型后产生变动。
```

以下示例代码展示了 [`KIEDataSample`](mmocr.structures.kie_data_sample.KIEDataSample) 的使用方法。

```python
from mmengine.data import KIEDataSample

data_sample = KIEDataSample()
img_meta = dict(img_shape=(800, 1196, 3),pad_shape=(800, 1216, 3))
gt_instances.texts = ['text1', 'text2', 'text3', 'text4', 'text5']
gt_instances.edge_lebels = torch.randint(-1, 2, (5, 5))

pred_instances.labels = torch.rand((5,))
pred_instances.edge_labels = torch.randint(-1, 2, (10, 10))
pred_instances.edge_scores = torch.rand((10, 10))
```

# Prerequisites

In this section we demonstrate how to prepare an environment with PyTorch.

MMPretrain works on Linux, Windows and macOS. It requires Python 3.7+, CUDA 10.2+ and PyTorch 1.8+.

```{note}
If you are experienced with PyTorch and have already installed it, just skip this part and jump to the [next section](#installation). Otherwise, you can follow these steps for the preparation.
```

**Step 1.** Download and install Miniconda from the [official website](https://docs.conda.io/en/latest/miniconda.html).

**Step 2.** Create a conda environment and activate it.

```shell
conda create --name openmmlab python=3.8 -y
conda activate openmmlab
```

**Step 3.** Install PyTorch following [official instructions](https://pytorch.org/get-started/locally/), e.g.

On GPU platforms:

```shell
conda install pytorch torchvision -c pytorch
```

This command will automatically install the latest version PyTorch and cudatoolkit, please check whether they match your environment.

```

```

On CPU platforms:

```shell
conda install pytorch torchvision cpuonly -c pytorch
```

# Installation

## Best Practices

According to your needs, we support two install modes:

- [Install from source (Recommended)](#install-from-source): You want to develop your own network or new features based on MMPretrain framework. For example, adding new datasets or new backbones. And you can use all tools we provided.
- [Install as a Python package](#install-as-a-python-package): You just want to call MMPretrain's APIs or import MMPretrain's modules in your project.

### Install from source

In this case, install mmpretrain from source:

```shell
pip install -U openmim && mim install -e .
```

```{note}

```shell
```

```{note}
`"-e"` means installing a project in editable mode, thus any local modifications made to the code will take effect without reinstallation.
```

### Install as a Python package

Just install with mim.

```shell
pip install -U openmim && mim install "mmpretrain>=1.0.0rc8"
```

```{note}
`mim` is a light-weight command-line tool to setup appropriate environment for OpenMMLab repositories according to PyTorch and CUDA version. It also has some useful functions for deep-learning experiments.
```

## Install multi-modality support (Optional)

The multi-modality models in MMPretrain requires extra dependencies. To install these dependencies, you
can add `[multimodal]` during the installation. For example:

```shell
# Install from source

# Install as a Python package

```shell

mim install "mmpretrain[multimodal]>=1.0.0rc8"
```

## Verify the installation

To verify whether MMPretrain is installed correctly, we provide some sample codes to run an inference demo.

Option (a). If you install mmpretrain from the source, just run the following command:

```shell
python demo/image_demo.py demo/demo.JPEG resnet18_8xb32_in1k --device cpu
```

You will see the output result dict including `pred_label`, `pred_score` and `pred_class` in your terminal.

Option (b). If you install mmpretrain as a python package, open your python interpreter and copy&paste the following codes.

```python
from mmpretrain import get_model, inference_model

model = get_model('resnet18_8xb32_in1k', device='cpu')  # or device='cuda:0'
inference_model(model, 'demo/demo.JPEG')
```

```python

```

You will see a dict printed, including the predicted label, score and category name.

```{note}
The `resnet18_8xb32_in1k` is the model name, and you can use [`mmpretrain.list_models`](mmpretrain.apis.list_models) to
explore all models, or search them on the [Model Zoo Summary](./modelzoo_statistics.md)
```

## Customize Installation

### CUDA versions

When installing PyTorch, you need to specify the version of CUDA. If you are
not clear on which to choose, follow our recommendations:

- For Ampere-based NVIDIA GPUs, such as GeForce 30 series and NVIDIA A100, CUDA 11 is a must.
- For older NVIDIA GPUs, CUDA 11 is backward compatible, but CUDA 10.2 offers better compatibility and is more lightweight.

Please make sure the GPU driver satisfies the minimum version requirements. See [this table](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cuda-major-component-versions__table-cuda-toolkit-driver-versions) for more information.

```{note}
Installing CUDA runtime libraries is enough if you follow our best practices,
because no CUDA code will be compiled locally. However if you hope to compile
MMCV from source or develop other CUDA operators, you need to install the
complete CUDA toolkit from NVIDIA's [website](https://developer.nvidia.com/cuda-downloads),
and its version should match the CUDA version of PyTorch. i.e., the specified
version of cudatoolkit in `conda install` command.
```

### Install on CPU-only platforms

MMPretrain can be built for CPU only environment. In CPU mode you can train, test or inference a model.

```

### Install on Google Colab

See [the Colab tutorial](https://colab.research.google.com/github/mzr1996/mmclassification-tutorial/blob/master/1.x/MMClassification_tools.ipynb).

### Using MMPretrain with Docker

We provide a [Dockerfile](https://github.com/open-mmlab/mmpretrain/blob/main/docker/Dockerfile)
to build an image. Ensure that your [docker version](https://docs.docker.com/engine/install/) >=19.03.

```shell
# build an image with PyTorch 1.12.1, CUDA 11.3
# If you prefer other versions, just modified the Dockerfile
docker build -t mmpretrain docker/
```

Run it with

```shell
docker run --gpus all --shm-size=8g -it -v {DATA_DIR}:/mmpretrain/data mmpretrain
```

## Trouble shooting

If you have some issues during the installation, please first view the [FAQ](./notes/faq.md) page.

```

You may [open an issue](https://github.com/open-mmlab/mmpretrain/issues/new/choose)
on GitHub if no solution is found.

# MMDetection v2.x 兼容性说明

为了加入 Mask2Former 实例分割模型，对 Mask2Former 的配置文件进行了重命名 [PR #7571](https://github.com/open-mmlab/mmdetection/pull/7571)：

    <thead>
            <td>在 v2.25.0 之前</td>
            <td>v2.25.0 及之后</td>
        </tr>
    </thead>
    <th>

```
'mask2former_xxx_coco.py' 代表全景分割的配置文件
```

</th>
    <th>

```
'mask2former_xxx_coco.py' 代表实例分割的配置文件
'mask2former_xxx_coco-panoptic.py' 代表全景分割的配置文件
```

为了支持 CPU 训练，MMCV 中进行批处理的 scatter 的代码逻辑已经被修改。我们推荐使用 MMCV v1.4.4 或更高版本，
更多信息请参考 [MMCV PR #1621](https://github.com/open-mmlab/mmcv/pull/1621).

为了修复 BaseTransformerLayer 中的权重引用问题, MultiheadAttention 中 batch first 的逻辑有所改变。
我们推荐使用 MMCV v1.3.17 或更高版本。 更多信息请参考 [MMCV PR #1418](https://github.com/open-mmlab/mmcv/pull/1418) 。

### DIIHead 兼容性

为了支持 QueryInst，在 DIIHead 的返回元组中加入了 attn_feats。

## MMDetection v2.14.0

### MMCV 版本

为了修复 EvalHook 优先级过低的问题，MMCV v1.3.8 中所有 hook 的优先级都重新进行了调整，因此 MMDetection v2.14.0 需要依赖最新的 MMCV v1.3.8 版本。 相关信息请参考[PR #1120](https://github.com/open-mmlab/mmcv/pull/1120) ，相关问题请参考[#5343](https://github.com/open-mmlab/mmdetection/issues/5343) 。

### SSD 兼容性

在 v2.14.0 中，为了使 SSD 能够被更灵活地使用，[PR #5291](https://github.com/open-mmlab/mmdetection/pull/5291) 重构了 SSD 的 backbone、neck 和 head。用户可以使用 tools/model_converters/upgrade_ssd_version.py 转换旧版本训练的模型。

```shell

```

- OLD_MODEL_PATH：旧版 SSD 模型的路径。
- NEW_MODEL_PATH：保存转换后模型权重的路径。

## MMDetection v2.12.0

```shell

```

在 v2.12.0 到 v2.18.0（或以上）版本的这段时间，为了提升通用性和便捷性，MMDetection 正在进行大规模重构。在升级到 v2.12.0 后 MMDetection 不可避免地带来了一些 BC Breaking，包括 MMCV 的版本依赖、模型初始化方式、模型 registry 和 mask AP 的评估。

### MMCV 版本

MMDetection v2.12.0 依赖 MMCV v1.3.3 中新增加的功能，包括：使用 `BaseModule` 统一参数初始化，模型 registry，以及[Deformable DETR](https://arxiv.org/abs/2010.04159) 中的 `MultiScaleDeformableAttn` CUDA 算子。
注意，尽管 MMCV v1.3.2 已经包含了 MMDet 所需的功能，但是存在一些已知的问题。我们建议用户跳过 MMCV v1.3.2 使用 v1.3.3 版本。

### 统一模型初始化

为了统一 OpenMMLab 项目中的参数初始化方式，MMCV 新增加了 `BaseModule` 类，使用 `init_cfg` 参数对模块进行统一且灵活的初始化配置管理。
现在用户需要在训练脚本中显式调用 `model.init_weights()` 来初始化模型（例如 [这行代码](https://github.com/open-mmlab/mmdetection/blob/master/tools/train.py#L162) ，在这之前则是在 detector 中进行处理的。

**下游项目必须相应地更新模型初始化方式才能使用 MMDetection v2.12.0**。请参阅 [PR #4750](https://github.com/open-mmlab/mmdetection/pull/4750) 了解详情。

### 统一模型 registry

为了能够使用在其他 OpenMMLab 项目中实现的 backbone，MMDetection v2.12.0 继承了在 MMCV (#760) 中创建的模型 registry。
这样，只要 OpenMMLab 项目实现了某个 backbone，并且该项目也使用 MMCV 中的 registry，那么用户只需修改配置即可在 MMDetection 中使用该 backbone，不再需要将代码复制到 MMDetection 中。 更多详细信息，请参阅 [PR #5059](https://github.com/open-mmlab/mmdetection/pull/5059) 。

### Mask AP 评估

在 [PR #4898](https://github.com/open-mmlab/mmdetection/pull/4898) 和 v2.12.0 之前，对小、中、大目标的 mask AP 的评估是基于其边界框区域而不是真正的 mask 区域。
这导致 `APs` 和 `APm` 变得更高但 `APl` 变得更低，但是不会影响整体的 mask AP。 [PR #4898](https://github.com/open-mmlab/mmdetection/pull/4898) 删除了 mask AP 计算中的 `bbox` ，改为使用 mask 区域。
新的计算方式不会影响整体的 mask AP 评估，与 [Detectron2](https://github.com/facebookresearch/detectron2/)一致。

## 与 MMDetection v1.x 的兼容性

MMDetection v2.0 经过了大规模重构并解决了许多遗留问题。 MMDetection v2.0 不兼容 v1.x 版本，在这两个版本中使用相同的模型权重运行推理会产生不同的结果。 因此，MMDetection v2.0 重新对所有模型进行了 benchmark，并在 model zoo 中提供了新模型的权重和训练记录。

新旧版本的主要的区别有四方面：坐标系、代码库约定、训练超参和模块设计。

### 坐标系

新坐标系与 [Detectron2](https://github.com/facebookresearch/detectron2/) 一致，
将最左上角的像素的中心视为坐标原点 (0, 0) 而不是最左上角像素的左上角。 因此 COCO 边界框和分割标注中的坐标被解析为范围 `[0，width]` 和 `[0，height]` 中的坐标。 这个修改影响了所有与 bbox 及像素选择相关的计算，变得更加自然且更加准确。

- 在新坐标系中，左上角和右下角为 (x1, y1) (x2, y2) 的框的宽度及高度计算公式为 `width = x2 - x1` 和 `height = y2 - y1`。
  在 MMDetection v1.x 和之前的版本中，高度和宽度都多了 `+ 1` 的操作。
  本次修改包括三部分：

  1. box 回归中的检测框变换以及编码/解码。
  2. IoU 计算。这会影响 ground truth 和检测框之间的匹配以及 NMS 。但对兼容性的影响可以忽略不计。
  3. Box 的角点坐标为浮点型，不再取整。这能使得检测结果更为准确，也使得检测框和 RoI 的最小尺寸不再为 1，但影响很小。

- Anchor 的中心与特征图的网格点对齐，类型变为 float。
  在 MMDetection v1.x 和之前的版本中，anchors 是 `int` 类型且没有居中对齐。
  这会影响 RPN 中的 Anchor 生成和所有基于 Anchor 的方法。

3. Box 的角点坐标为浮点型，不再取整。这能使得检测结果更为准确，也使得检测框和 RoI 的最小尺寸不再为 1，但影响很小。

- ROIAlign 更好地与图像坐标系对齐。新的实现来自 [Detectron2](https://github.com/facebookresearch/detectron2/tree/master/detectron2/layers/csrc/ROIAlign) 。
  当 RoI 用于提取 RoI 特征时，与 MMDetection v1.x 相比默认情况下相差半个像素。
  能够通过设置 `aligned=False` 而不是 `aligned=True` 来维持旧版本的设置。

- Mask 的裁剪和粘贴更准确。

  1. 我们使用新的 RoIAlign 来提取 mask 目标。 在 MMDetection v1.x 中，bounding box 在提取 mask 目标之前被取整，裁剪过程是 numpy 实现的。 而在新版本中，裁剪的边界框不经过取整直接输入 RoIAlign。 此实现大大加快了训练速度（每次迭代约加速 0.1 秒，1x schedule 训练 Mask R50 时加速约 2 小时）并且理论上会更准确。
  2. 在 MMDetection v2.0 中，修改后的 `paste_mask()` 函数应该比之前版本更准确。 此更改参考了 [Detectron2](https://github.com/facebookresearch/detectron2/blob/master/detectron2/structures/masks.py) 中的修改，可以将 COCO 上的 mask AP 提高约 0.5%。

### 代码库约定

- MMDetection v2.0 更改了类别标签的顺序，减少了回归和 mask 分支里的无用参数并使得顺序更加自然（没有 +1 和 -1）。

### 代码库约定

  这会影响模型的所有分类层，使其输出的类别标签顺序发生改变。回归分支和 mask head 的最后一层不再为 K 个类别保留 K+1 个通道，类别顺序与分类分支一致。

  - 在 MMDetection v2.0 中，标签 “K” 表示背景，标签 \[0, K-1\] 对应于 K = num_categories 个对象类别。

  - 在 MMDetection v1.x 及之前的版本中，标签 “0” 表示背景，标签 \[1, K\] 对应 K 个类别。

  - **注意**：softmax RPN 的类顺序在 version\<=2.4.0 中仍然和 1.x 中的一样，而 sigmoid RPN 不受影响。从 MMDetection v2.5.0 开始，所有 head 中的类顺序是统一的。

- 不使用 R-CNN 中的低质量匹配。在 MMDetection v1.x 和之前的版本中，`max_iou_assigner` 会在 RPN 和 R-CNN 训练时给每个 ground truth 匹配低质量框。我们发现这会导致最佳的 GT 框不会被分配给某些边界框，
  因此，在MMDetection v2.0 的 R-CNN 训练中默认不允许低质量匹配。这有时可能会稍微改善 box AP（约为 0.1%）。

- 单独的宽高比例系数。在 MMDetection v1.x 和以前的版本中，`keep_ratio=True` 时比例系数是单个浮点数，这并不准确，因为宽度和高度的比例系数会有一定的差异。 MMDetection v2.0 对宽度和高度使用单独的比例系数，对 AP 的提升约为 0.1%。

- 修改了 config 文件名称的规范。 由于 model zoo 中模型不断增多， MMDetection v2.0 采用新的命名规则：

  ```shell
  [model]_(model setting)_[backbone]_[neck]_(norm setting)_(misc)_(gpu x batch)_[schedule]_[dataset].py
  ```

  ```shell
  ```

  其中 (`misc`) 包括 DCN 和 GCBlock 等。更多详细信息在 [配置文件说明文档](config.md) 中说明

- MMDetection v2.0 使用新的 ResNet Caffe backbone 来减少加载预训练模型时的警告。新 backbone 中的大部分权重与以前的相同，但没有 `conv.bias`，且它们使用不同的 `img_norm_cfg`。因此，新的 backbone 不会报 `unexpected keys` 的警告。

### 训练超参

训练超参的调整不会影响模型的兼容性，但会略微提高性能。主要有：

- 通过设置 `nms_post=1000` 和 `max_num=1000`，将 nms 之后的 proposal 数量从 2000 更改为 1000。使 mask AP 和 bbox AP 提高了约 0.2%。

- Mask R-CNN、Faster R-CNN 和 RetinaNet 的默认回归损失从 smooth L1 损失更改为 L1 损失，使得 box AP 整体上都有所提升（约 0.6%）。但是，将 L1-loss 用在 Cascade R-CNN 和 HTC 等其他方法上并不能提高性能，因此我们保留这些方法的原始设置。

- 为简单起见，RoIAlign 层的 `sampling_ratio` 设置为 0。略微提升了 AP（约 0.2% 绝对值）。

- 为了提升训练速度，默认设置在训练过程中不再使用梯度裁剪。大多数模型的性能不会受到影响。对于某些模型（例如 RepPoints），我们依旧使用梯度裁剪来稳定训练过程从而获得更好的性能。

- 因为不再默认使用梯度裁剪，默认 warmup 比率从 1/3 更改为 0.001，以使模型训练预热更加平缓。不过我们重新进行基准测试时发现这种影响可以忽略不计。

### 将模型从 v1.x 升级至 v2.0

用户可以使用脚本 `tools/model_converters/upgrade_model_version.py` 来将 MMDetection 1.x 训练的模型转换为 MMDetection v2.0。转换后的模型可以在 MMDetection v2.0 中运行，但性能略有下降（小于 1% AP）。
详细信息可以在 `configs/legacy` 中找到。

## pycocotools 兼容性

`mmpycocotools` 是 OpenMMLab 维护的 `pycocotools` 的复刻版，适用于 MMDetection 和 Detectron2。
在 [PR #4939](https://github.com/open-mmlab/mmdetection/pull/4939) 之前，由于 `pycocotools` 和 `mmpycocotool` 具有相同的包名，如果用户已经安装了 `pyccocotools`（在相同环境下先安装了 Detectron2 ），那么 MMDetection 的安装过程会跳过安装 `mmpycocotool`。 导致 MMDetection 缺少 `mmpycocotools` 而报错。
但如果在 Detectron2 之前安装 MMDetection，则可以在相同的环境下工作。
[PR #4939](https://github.com/open-mmlab/mmdetection/pull/4939) 弃用 mmpycocotools，使用官方 pycocotools。
在 [PR #4939](https://github.com/open-mmlab/mmdetection/pull/4939) 之后，用户能够在相同环境下安装 MMDetection 和 Detectron2，不再需要关注安装顺序。

## 只是实践经验，没有理论，不一定正确

```
prfm pldl1keep, [x0, #256]
```
* 放在 ld1 [x0] 前面 0~8 条指令
* #256 表示把 x0+256 的内容放进 L1 cache
* ldp 也适用
* (经验)不写 offset 不如写个 #128
* (经验)pldl1strm 似乎没啥意思，也没 pldl1keep 快
* (经验)x0 ~ x0+256 的内容也会进来
* (经验)load 128bit 用 #128，256bit或更多用 #256
* (经验)避免 pld a，pld b，load a，load b 顺序，可能相互干扰
* (经验)提前太多会失效
* (经验)适合连续读

```
prfm pldl2strm, [x0, #256]
```
* 放在 ld1 [x0] 前面 N 条指令，N 尽量大些
* #256 表示把 x0+256 的内容放进 L2 cache
* ldp 也适用
* (经验)pldl2strm 效果稍好于 pldl2keep
* (经验)load 128bit 用 #128，256bit 用 #256
* (经验)读很多数据，用不同 offset 连续两次 pldl2strm
* (经验)后面不要对同位置再 pldl1keep，会变慢
* (经验)适合提前准备要跳到很远的地方读，比如换 channel

# Use a single stage detector as RPN

Region proposal network (RPN) is a submodule in [Faster R-CNN](https://arxiv.org/abs/1506.01497), which generates proposals for the second stage of Faster R-CNN. Most two-stage detectors in MMDetection use [`RPNHead`](../../../mmdet/models/dense_heads/rpn_head.py) to generate proposals as RPN. However, any single-stage detector can serve as an RPN since their bounding box predictions can also be regarded as region proposals and thus be refined in the R-CNN. Therefore, MMDetection v3.0 supports that.

To illustrate the whole process, here we give an example of how to use an anchor-free single-stage model [FCOS](../../../configs/fcos/fcos_r50-caffe_fpn_gn-head_1x_coco.py) as an RPN in [Faster R-CNN](../../../configs/faster_rcnn/faster-rcnn_r50_fpn_fcos-rpn_1x_coco.py).

The outline of this tutorial is as below:

1. Use `FCOSHead` as an `RPNHead` in Faster R-CNN
2. Evaluate proposals
3. Train the customized Faster R-CNN with pre-trained FCOS

## Use `FCOSHead` as an `RPNHead` in Faster R-CNN

To set `FCOSHead` as an `RPNHead` in Faster R-CNN, we should create a new config file named `configs/faster_rcnn/faster-rcnn_r50_fpn_fcos-rpn_1x_coco.py`, and replace with the setting of `rpn_head` with the setting of `bbox_head` in `configs/fcos/fcos_r50-caffe_fpn_gn-head_1x_coco.py`. Besides, we still use the neck setting of FCOS with strides of `[8, 16, 32, 64, 128]`, and update `featmap_strides` of `bbox_roi_extractor` to `[8, 16, 32, 64, 128]`. To avoid loss goes NAN, we apply warmup during the first 1000 iterations instead of the first 500 iterations, which means that the lr increases more slowly. The config is as follows:

```python
_base_ = [
    '../_base_/models/faster-rcnn_r50_fpn.py',
    '../_base_/datasets/coco_detection.py',

```python
_base_ = [
]

    # copied from configs/fcos/fcos_r50-caffe_fpn_gn-head_1x_coco.py
        start_level=1,
        add_extra_convs='on_output',  # use P5
        relu_before_extra_convs=True),
        _delete_=True,  # ignore the unused old settings
        type='FCOSHead',
        num_classes=1,  # num_classes = 1 for rpn, if num_classes > 1, it will be set to 1 in TwoStageDetector automatically
        in_channels=256,
        stacked_convs=4,
        feat_channels=256,
        strides=[8, 16, 32, 64, 128],
        loss_cls=dict(
            type='FocalLoss',
            use_sigmoid=True,
            gamma=2.0,
            alpha=0.25,
            loss_weight=1.0),
        loss_bbox=dict(type='IoULoss', loss_weight=1.0),
        loss_centerness=dict(

use_sigmoid=True,
            type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0)),
    roi_head=dict(  # update featmap_strides due to the strides in neck
        bbox_roi_extractor=dict(featmap_strides=[8, 16, 32, 64, 128])))

# learning rate
    dict(
        type='LinearLR', start_factor=0.001, by_epoch=False, begin=0,
        end=1000),  # Slowly increase lr, otherwise loss becomes NAN
    dict(
        begin=0,
        end=12,
        by_epoch=True,
        milestones=[8, 11],
]
```

Then, we could use the following command to train our customized model. For more training commands, please refer to [here](train.md).

```python
# training with 8 GPUS
bash tools/dist_train.sh configs/faster_rcnn/faster-rcnn_r50_fpn_fcos-rpn_1x_coco.py \

```python
    8 \
    --work-dir ./work_dirs/faster-rcnn_r50_fpn_fcos-rpn_1x_coco
```

## Evaluate proposals

The quality of proposals is of great importance to the performance of detector, therefore, we also provide a way to evaluate proposals. Same as above, create a new config file named `configs/rpn/fcos-rpn_r50_fpn_1x_coco.py`, and replace with setting of `rpn_head` with the setting of `bbox_head` in `configs/fcos/fcos_r50-caffe_fpn_gn-head_1x_coco.py`.

```python
_base_ = [
    '../_base_/models/rpn_r50_fpn.py', '../_base_/datasets/coco_detection.py',
]

val_evaluator = dict(metric='proposal_fast')

        num_classes=1,  # num_classes = 1 for rpn, if num_classes > 1, it will be set to 1 in RPN automatically
            type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0)))
```

loss_weight=1.0),
```

Suppose we have the checkpoint `./work_dirs/faster-rcnn_r50_fpn_fcos-rpn_1x_coco/epoch_12.pth` after training, then we can evaluate the quality of proposals with the following command.

```python
# testing with 8 GPUs
bash tools/dist_test.sh \
    configs/rpn/fcos-rpn_r50_fpn_1x_coco.py \
    ./work_dirs/faster-rcnn_r50_fpn_fcos-rpn_1x_coco/epoch_12.pth \
    8
```

## Train the customized Faster R-CNN with pre-trained FCOS

configs/rpn/fcos-rpn_r50_fpn_1x_coco.py \
    8
```

Pre-training not only speeds up convergence of training, but also improves the performance of the detector. Therefore, here we give an example to illustrate how to do use a pre-trained FCOS as an RPN to accelerate training and improve the accuracy. Suppose we want to use `FCOSHead` as an rpn head in Faster R-CNN and train with the pre-trained [`fcos_r50-caffe_fpn_gn-head_1x_coco`](https://download.openmmlab.com/mmdetection/v2.0/fcos/fcos_r50_caffe_fpn_gn-head_1x_coco/fcos_r50_caffe_fpn_gn-head_1x_coco-821213aa.pth). The content of config file named `configs/faster_rcnn/faster-rcnn_r50-caffe_fpn_fcos-rpn_1x_coco.py` is as the following. Note that `fcos_r50-caffe_fpn_gn-head_1x_coco` uses a caffe version of ResNet50, the pixel mean and std in `data_preprocessor` thus need to be updated.

```python

```python
_base_ = [
]

        mean=[103.530, 116.280, 123.675],
        std=[1.0, 1.0, 1.0],
        bgr_to_rgb=False),
        norm_cfg=dict(type='BN', requires_grad=False),
        style='caffe',
        init_cfg=None),  # the checkpoint in ``load_from`` contains the weights of backbone

in_channels=256,

load_from = 'https://download.openmmlab.com/mmdetection/v2.0/fcos/fcos_r50_caffe_fpn_gn-head_1x_coco/fcos_r50_caffe_fpn_gn-head_1x_coco-821213aa.pth'
```

The command for training is as below.

```python
bash tools/dist_train.sh \
    configs/faster_rcnn/faster-rcnn_r50-caffe_fpn_fcos-rpn_1x_coco.py \
    8 \
    --work-dir ./work_dirs/faster-rcnn_r50-caffe_fpn_fcos-rpn_1x_coco
```

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

## Motivation

Please describe the motivation of this PR and the goal you want to achieve through this PR.

## Modification

Please briefly describe what modification is made in this PR.

## BC-breaking (Optional)

Does the modification introduce changes that break the backward-compatibility of the downstream repos?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

## Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

1. Pre-commit or other linting tools are used to fix the potential lint issues.

2. The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
3. If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMDet3D.
4. The documentation has been modified accordingly, like docstring or example tutorials.

## Contributing to InternLM

Welcome to the InternLM community, all kinds of contributions are welcomed, including but not limited to

**Fix bug**

You can directly post a Pull Request to fix typo in code or documents

The steps to fix the bug of code implementation are as follows.

1. If the modification involve significant changes, you should create an issue first and describe the error information and how to trigger the bug. Other developers will discuss with you and propose an proper solution.

2. Posting a pull request after fixing the bug and adding corresponding unit test.

**New Feature or Enhancement**

1. If the modification involve significant changes, you should create an issue to discuss with our developers to propose an proper design.
2. Post a Pull Request after implementing the new feature or enhancement and add corresponding unit test.

**Document**

You can directly post a pull request to fix documents. If you want to add a document, you should first create an issue to check if it is reasonable.

### Pull Request Workflow

If you're not familiar with Pull Request, don't worry! The following guidance will tell you how to create a Pull Request step by step. If you want to dive into the develop mode of Pull Request, you can refer to the [official documents](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests)

#### 1. Fork and clone

If you are posting a pull request for the first time, you should fork the OpenMMLab repositories by clicking the **Fork** button in the top right corner of the GitHub page, and the forked repositories will appear under your GitHub profile.

<img src="https://user-images.githubusercontent.com/57566630/167305749-43c7f4e9-449b-4e98-ade5-0c9276d5c9ce.png" width="1200">

Then, you can clone the repositories to local:

```shell
git clone git@github.com:{username}/lmdeploy.git
```

After that, you should add official repository as the upstream repository

```bash
git remote add upstream git@github.com:InternLM/lmdeploy.git
```

Check whether remote repository has been added successfully by `git remote -v`

```bash
origin	git@github.com:{username}/lmdeploy.git (fetch)
origin	git@github.com:{username}/lmdeploy.git (push)
upstream	git@github.com:InternLM/lmdeploy.git (fetch)
upstream	git@github.com:InternLM/lmdeploy.git (push)
```

```

> Here's a brief introduction to origin and upstream. When we use "git clone", we create an "origin" remote by default, which points to the repository cloned from. As for "upstream", we add it ourselves to point to the target repository. Of course, if you don't like the name "upstream", you could name it as you wish. Usually, we'll push the code to "origin". If the pushed code conflicts with the latest code in official("upstream"), we should pull the latest code from upstream to resolve the conflicts, and then push to "origin" again. The posted Pull Request will be updated automatically.

#### 2. Configure pre-commit

You should configure [pre-commit](https://pre-commit.com/#intro) in the local development environment to make sure the code style matches that of InternLM. **Note**: The following code should be executed under the lmdeploy directory.

```shell
pip install -U pre-commit
pre-commit install
```

Check that pre-commit is configured successfully, and install the hooks defined in `.pre-commit-config.yaml`.

```shell
pre-commit run --all-files
```

<img src="https://user-images.githubusercontent.com/57566630/173660750-3df20a63-cb66-4d33-a986-1f643f1d8aaf.png" width="1200">

<img src="https://user-images.githubusercontent.com/57566630/202368856-0465a90d-8fce-4345-918e-67b8b9c82614.png" width="1200">

If the installation process is interrupted, you can repeatedly run `pre-commit run ... ` to continue the installation.

If the code does not conform to the code style specification, pre-commit will raise a warning and  fixes some of the errors automatically.

<img src="https://user-images.githubusercontent.com/57566630/202369176-67642454-0025-4023-a095-263529107aa3.png" width="1200">

If we want to commit our code bypassing the pre-commit hook, we can use the `--no-verify` option(**only for temporarily commit**).

```shell
git commit -m "xxx" --no-verify
```

#### 3. Create a development branch

After configuring the pre-commit, we should create a branch based on the master branch to develop the new feature or fix the bug. The proposed branch name is `username/pr_name`

```shell
git checkout -b yhc/refactor_contributing_doc
```

In subsequent development, if the master branch of the local repository is behind the master branch of "upstream", we need to pull the upstream for synchronization, and then execute the above command:

```shell
git pull upstream master
```

```shell
```

#### 4. Commit the code and pass the unit test

- lmdeploy introduces mypy to do static type checking to increase the robustness of the code. Therefore, we need to add Type Hints to our code and pass the mypy check. If you are not familiar with Type Hints, you can refer to [this tutorial](https://docs.python.org/3/library/typing.html).

- The committed code should pass through the unit test

  ```shell
  # Pass all unit tests
  pytest tests

  # Pass the unit test of runner
  pytest tests/test_runner/test_runner.py
  ```

  If the unit test fails for lack of dependencies, you can install the dependencies referring to the [guidance](#unit-test)

- If the documents are modified/added, we should check the rendering result referring to [guidance](#document-rendering)

#### 5. Push the code to remote

We could push the local commits to remote after passing through the check of unit test and pre-commit. You can associate the local branch with remote branch by adding `-u` option.

```shell
git push -u origin {branch_name}
```

This will allow you to use the `git push` command to push code directly next time, without having to specify a branch or the remote repository.

#### 6. Create a Pull Request

(1) Create a pull request in GitHub's Pull request interface

<img src="https://user-images.githubusercontent.com/57566630/201533288-516f7ac4-0b14-4dc8-afbd-912475c368b5.png" width="1200">

(2) Modify the PR description according to the guidelines so that other developers can better understand your changes

<img src="https://user-images.githubusercontent.com/57566630/202242953-c91a18ff-e388-4ff9-8591-5fae0ead6c1e.png" width="1200">

Find more details about Pull Request description in [pull request guidelines](#pr-specs).

**note**

(a) The Pull Request description should contain the reason for the change, the content of the change, and the impact of the change, and be associated with the relevant Issue (see [documentation](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue))

(b) If it is your first contribution, please sign the CLA

<img src="https://user-images.githubusercontent.com/57566630/167307569-a794b967-6e28-4eac-a942-00deb657815f.png" width="1200">

(c) Check whether the Pull Request pass through the CI

<img src="https://user-images.githubusercontent.com/57566630/167307490-f9ebf9fa-63c0-4d83-8ba1-081ea169eb3a.png" width="1200">

IternLM will run unit test for the posted Pull Request on different platforms (Linux, Window, Mac), based on different versions of Python, PyTorch, CUDA to make sure the code is correct. We can see the specific test information by clicking `Details` in the above image so that we can modify the code.

(3) If the Pull Request passes the CI, then you can wait for the review from other developers. You'll modify the code based on the reviewer's comments, and repeat the steps [4](#4-commit-the-code-and-pass-the-unit-test)-[5](#5-push-the-code-to-remote) until all reviewers approve it. Then, we will merge it ASAP.

<img src="https://user-images.githubusercontent.com/57566630/202145400-cc2cd8c4-10b0-472f-ba37-07e6f50acc67.png" width="1200">

#### 7. Resolve conflicts

If your local branch conflicts with the latest master branch of "upstream", you'll need to resolove them. There are two ways to do this:

```shell
git fetch --all --prune
git rebase upstream/master
```

or

```shell
git merge upstream/master
```

If you are very good at handling conflicts, then you can use rebase to resolve conflicts, as this will keep your commit logs tidy. If you are not familiar with `rebase`, then you can use `merge` to resolve conflicts.

### Guidance

#### Document rendering

If the documents are modified/added, we should check the rendering result. We could install the dependencies and run the following command to render the documents and check the results:

```shell
pip install -r requirements/docs.txt
cd docs/zh_cn/
# or docs/en
make html

```shell
make html
# check file in ./docs/zh_cn/_build/html/index.html
```

### Code style

#### Python

We adopt [PEP8](https://www.python.org/dev/peps/pep-0008/) as the preferred code style.

We use the following tools for linting and formatting:

- [flake8](https://github.com/PyCQA/flake8): A wrapper around some linter tools.
- [isort](https://github.com/timothycrosley/isort): A Python utility to sort imports.
- [yapf](https://github.com/google/yapf): A formatter for Python files.
- [codespell](https://github.com/codespell-project/codespell): A Python utility to fix common misspellings in text files.
- [mdformat](https://github.com/executablebooks/mdformat): Mdformat is an opinionated Markdown formatter that can be used to enforce a consistent style in Markdown files.
- [docformatter](https://github.com/myint/docformatter): A formatter to format docstring.

We use [pre-commit hook](https://pre-commit.com/) that checks and formats for `flake8`, `yapf`, `isort`, `trailing whitespaces`, `markdown files`,
fixes `end-of-files`, `double-quoted-strings`, `python-encoding-pragma`, `mixed-line-ending`, sorts `requirments.txt` automatically on every commit.
The config for a pre-commit hook is stored in [.pre-commit-config](../.pre-commit-config.yaml).

#### C++ and CUDA

The clang-format config is stored in [.clang-format](../.clang-format).

### PR Specs

1. Use [pre-commit](https://pre-commit.com) hook to avoid issues of code style

2. One short-time branch should be matched with only one PR

3. Accomplish a detailed change in one PR. Avoid large PR

   - Bad: Support Faster R-CNN
   - Acceptable: Add a box head to Faster R-CNN
   - Good: Add a parameter to box head to support custom conv-layer number

4. Provide clear and significant commit message

- Acceptable: Add a box head to Faster R-CNN

5. Provide clear and meaningful PR description

   - Task name should be clarified in title. The general format is: \[Prefix\] Short description of the PR (Suffix)
   - Prefix: add new feature \[Feature\], fix bug \[Fix\], related to documents \[Docs\], in developing \[WIP\] (which will not be reviewed temporarily)
   - Introduce main changes, results and influences on other modules in short description
   - Associate related issues and pull requests with a milestone

# Data pre-processor \[Coming Soon!\]

We're improving this documentation. Don't hesitate to join us!

[Make a pull request](https://github.com/open-mmlab/mmagic/compare) or [discuss with us](https://github.com/open-mmlab/mmagic/discussions/1429)!

# 概览

伴随着 OpenMMLab 2.0 的发布，MMOCR 1.0 本身也作出了许多突破性的改变，使得代码的冗余度降低，代码效率提高，整体设计上也变得更为一致。然而，这些改变使得完美的后向兼容不再可能。我们也深知在这样巨大的变动之下，老用户想第一时间适应新版本也绝非易事。因此，我们推出了详细的迁移指南，旨在让老用户们尽可能平滑地过渡到全新的框架，最终能享受到全新的 MMOCR 和整个OpenMMLab 2.0 生态系统为生产力带来的巨大优势。

MMOCR 1.0 依赖于新的基础训练框架 [MMEngine](https://github.com/open-mmlab/mmengine)，因而有着与 MMOCR 0.x 完全不同的依赖链。尽管你可能已经拥有了一个可以正常运行 MMOCR 0.x 的环境，但你仍然需要创建一个新的 python 环境来安装 MMOCR 1.0 版本所需要的依赖库。我们提供了详细的[安装文档](../get_started/install.md)以供参考。
```

接下来，请根据你的实际需求，阅读需要的章节：

- 若需要了解 MMOCR 1.0 的主要变化，请阅读 [MMOCR 1.x 更新汇总](./news.md)
- 如果你需要把 0.x 版本中训练的模型直接迁移到 1.0 版本中使用，请阅读 [预训练模型迁移](./model.md)
- 如果你需要训练模型，请阅读 [数据集迁移](./dataset.md) 和 [数据增强迁移](./transforms.md)
- 如果你需要在 MMOCR 上进行开发，请阅读 [代码迁移](code.md) ，[分支迁移](branches.md) 和 [上游依赖库变更](https://github.com/open-mmlab/mmengine/tree/main/docs/zh_cn/migration)

如下图所示，MMOCR 1.x 版本的维护计划主要分为三个阶段，即“公测期”，“兼容期”以及“维护期”。对于旧版本，我们将不再增加主要新功能。因此，我们强烈建议用户尽早迁移至 MMOCR 1.x 版本。

![plan](https://user-images.githubusercontent.com/45810070/192678159-f8965f73-552b-48a1-bf65-659e8f519bdd.png)

<details>
<summary align="right"><a href="https://arxiv.org/abs/2212.07784">RTMDet (ArXiv 2022)</a></summary>

```bibtex
}
```

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_CVPR_2020/html/Huang_The_Devil_Is_in_the_Details_Delving_Into_Unbiased_Data_CVPR_2020_paper.html">UDP (CVPR'2020)</a></summary>

```bibtex
@InProceedings{Huang_2020_CVPR,
  author = {Huang, Junjie and Zhu, Zheng and Guo, Feng and Huang, Guan},
  title = {The Devil Is in the Details: Delving Into Unbiased Data Processing for Human Pose Estimation},

author = {Huang, Junjie and Zhu, Zheng and Guo, Feng and Huang, Guan},
  booktitle = {The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month = {June},
  year = {2020}
}
```

<details>
<summary align="right"><a href="https://link.springer.com/chapter/10.1007/978-3-030-58545-7_12">COCO-WholeBody (ECCV'2020)</a></summary>

```bibtex
}
```

Results on COCO-WholeBody v1.0 val with detector having human AP of 56.4 on COCO val2017 dataset

year={2020}
}
```

| Arch                                    | Input Size | Body AP | Body AR | Foot AP | Foot AR | Face AP | Face AR | Hand AP | Hand AR | Whole AP | Whole AR |                   ckpt                   |                   log                   |
| :-------------------------------------- | :--------: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :------: | :------: | :--------------------------------------: | :-------------------------------------: |

| [pose_cspnext_m_udp](/configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/cspnext-m_udp_8xb64-210e_coco-wholebody-256x192.py) |  256x192   |  0.687  |  0.735  |  0.680  |  0.763  |  0.697  |  0.755  |  0.460  |  0.543  |  0.567   |  0.641   | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-m_udp-coco-wholebody_pt-in1k_210e-256x192-320fa258_20230123.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-m_udp-coco-wholebody_pt-in1k_210e-256x192-320fa258_20230123.json) |

<details>

```bibtex
}
```

<details>
<summary align="right"><a href="https://link.springer.com/chapter/10.1007/978-3-030-58545-7_12">COCO-WholeBody-Face (ECCV'2020)</a></summary>

```bibtex
}
```

booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
}
```

Results on COCO-WholeBody-Face val set

| Arch                                                          | Input Size |  NME   |                              ckpt                              |                              log                              |
| :------------------------------------------------------------ | :--------: | :----: | :------------------------------------------------------------: | :-----------------------------------------------------------: |

| [pose_rtmpose_m](/configs/face_2d_keypoint/rtmpose/coco_wholebody_face/rtmpose-m_8xb32-60e_coco-wholebody-face-256x256.py) |  256x256   | 0.0466 | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-coco-wholebody-face_pt-aic-coco_60e-256x256-62026ef2_20230228.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-coco-wholebody-face_pt-aic-coco_60e-256x256-62026ef2_20230228.json) |

# CycleGAN (ICCV'2017)

> [CycleGAN: Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks](https://openaccess.thecvf.com/content_iccv_2017/html/Zhu_Unpaired_Image-To-Image_Translation_ICCV_2017_paper.html)

> **Task**: Image2Image

Image-to-image translation is a class of vision and graphics problems where the goal is to learn the mapping between an input image and an output image using a training set of aligned image pairs. However, for many tasks, paired training data will not be available. We present an approach for learning to translate an image from a source domain X to a target domain Y in the absence of paired examples. Our goal is to learn a mapping G: X \\rightarrow Y such that the distribution of images from G(X) is indistinguishable from the distribution Y using an adversarial loss. Because this mapping is highly under-constrained, we couple it with an inverse mapping F: Y \\rightarrow X and introduce a cycle consistency loss to push F(G(X)) \\approx X (and vice versa). Qualitative results are presented on several tasks where paired training data does not exist, including collection style transfer, object transfiguration, season transfer, photo enhancement, etc. Quantitative comparisons against several prior methods demonstrate the superiority of our approach.

<img src="https://user-images.githubusercontent.com/28132635/143049598-23c24d98-7a64-4ab3-a9ba-351db6a0a53d.JPG" />
</div>

  <b> Results from CycleGAN trained by mmagic</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/22982797/114303527-108ed200-9b01-11eb-978c-274392e4d8e0.PNG" width="800"/>
</div>

We use `FID` and `IS` metrics to evaluate the generation performance of CycleGAN.<sup>1</sup>
https://download.openmmlab.com/mmediting/cyclegan/refactor/cyclegan_lsgan_resnet_in_1x1_80k_facades_20210902_165905-5e2c0876.pth
https://download.openmmlab.com/mmediting/cyclegan/refactor/cyclegan_in_1x1_80k_facades_20210902_165905-5e2c0876.pth

|                                 Model                                  |      Dataset      |   FID    |  IS   |                                               Download                                               |

| :--------------------------------------------------------------------: | :---------------: | :------: | :---: | :--------------------------------------------------------------------------------------------------: |
|      [Ours](./cyclegan_lsgan-resnet-in_1xb1-80kiters_facades.py)       |      facades      | 124.8033 | 1.792 | [model](https://download.openmmlab.com/mmediting/cyclegan/refactor/cyclegan_lsgan_resnet_in_1x1_80k_facades_20210902_165905-5e2c0876.pth) \| [log](https://download.openmmlab.com/mmediting/cyclegan/cyclegan_lsgan_resnet_in_1x1_80k_facades_20210317_160938.log.json) <sup>2</sup> |
|    [Ours](./cyclegan_lsgan-id0-resnet-in_1xb1-80kiters_facades.py)     |    facades-id0    | 125.1694 | 1.905 | [model](https://download.openmmlab.com/mmediting/cyclegan/refactor/cyclegan_lsgan_id0_resnet_in_1x1_80k_facades_convert-bgr_20210902_164411-d8e72b45.pth) |

|   [Ours](./cyclegan_lsgan-resnet-in_1xb1-250kiters_summer2winter.py)   |   summer2winter   | 83.7177  | 2.771 | [model](https://download.openmmlab.com/mmediting/cyclegan/refactor/cyclegan_lsgan_resnet_in_1x1_246200_summer2winter_convert-bgr_20210902_165932-fcf08dc1.pth) |
| [Ours](./cyclegan_lsgan-id0-resnet-in_1xb1-250kiters_summer2winter.py) | summer2winter-id0 | 83.1418  | 2.720 | [model](https://download.openmmlab.com/mmediting/cyclegan/refactor/cyclegan_lsgan_id0_resnet_in_1x1_246200_summer2winter_convert-bgr_20210902_165640-8b825581.pth) |
|   [Ours](./cyclegan_lsgan-resnet-in_1xb1-250kiters_summer2winter.py)   |   winter2summer   | 72.8025  | 3.129 | [model](https://download.openmmlab.com/mmediting/cyclegan/refactor/cyclegan_lsgan_resnet_in_1x1_246200_summer2winter_convert-bgr_20210902_165932-fcf08dc1.pth) |

| [Ours](./cyclegan_lsgan-id0-resnet-in_1xb1-250kiters_summer2winter.py) | winter2summer-id0 | 73.5001  | 3.107 | [model](https://download.openmmlab.com/mmediting/cyclegan/refactor/cyclegan_lsgan_id0_resnet_in_1x1_246200_summer2winter_convert-bgr_20210902_165640-8b825581.pth) |
|    [Ours](./cyclegan_lsgan-resnet-in_1xb1-270kiters_horse2zebra.py)    |    horse2zebra    | 64.5225  | 1.418 | [model](https://download.openmmlab.com/mmediting/cyclegan/refactor/cyclegan_lsgan_resnet_in_1x1_266800_horse2zebra_convert-bgr_20210902_170004-a32c733a.pth) |
|  [Ours](./cyclegan_lsgan-id0-resnet-in_1xb1-270kiters_horse2zebra.py)  |  horse2zebra-id0  | 74.7770  | 1.542 | [model](https://download.openmmlab.com/mmediting/cyclegan/refactor/cyclegan_lsgan_id0_resnet_in_1x1_266800_horse2zebra_convert-bgr_20210902_165724-77c9c806.pth) |

|    [Ours](./cyclegan_lsgan-resnet-in_1xb1-270kiters_horse2zebra.py)    |    zebra2horse    | 141.1517 | 3.154 | [model](https://download.openmmlab.com/mmediting/cyclegan/refactor/cyclegan_lsgan_resnet_in_1x1_266800_horse2zebra_convert-bgr_20210902_170004-a32c733a.pth) |
|  [Ours](./cyclegan_lsgan-id0-resnet-in_1xb1-270kiters_horse2zebra.py)  |  zebra2horse-id0  | 134.3728 | 3.091 | [model](https://download.openmmlab.com/mmediting/cyclegan/refactor/cyclegan_lsgan_id0_resnet_in_1x1_266800_horse2zebra_convert-bgr_20210902_165724-77c9c806.pth) |

`FID` comparison with official:

<!-- SKIP THIS TABLE -->

| Dataset  |   facades   | facades-id0 | summer2winter | summer2winter-id0 | winter2summer | winter2summer-id0 | horse2zebra | horse2zebra-id0 | zebra2horse | zebra2horse-id0 |  average   |
| :------: | :---------: | :---------: | :-----------: | :---------------: | :-----------: | :---------------: | :---------: | :-------------: | :---------: | :-------------: | :--------: |

| official | **123.626** | **119.726** |  **77.342**   |    **76.773**     |  **72.631**   |      74.239       | **62.111**  |     77.202      | **138.646** |   **137.050**   | **95.935** |
|   ours   |  124.8033   |  125.1694   |    83.7177    |      83.1418      |    72.8025    |    **73.5001**    |   64.5225   |   **74.7770**   |  141.1571   |  **134.3728**   |   97.79    |

`IS` comparison with evaluation:

| Dataset  |  facades  | facades-id0 | summer2winter | summer2winter-id0 | winter2summer | winter2summer-id0 | horse2zebra | horse2zebra-id0 | zebra2horse | zebra2horse-id0 |  average  |

| :------: | :-------: | :---------: | :-----------: | :---------------: | :-----------: | :---------------: | :---------: | :-------------: | :---------: | :-------------: | :-------: |
| official |   1.638   |    1.697    |     2.762     |     **2.750**     |   **3.293**   |     **3.110**     |    1.375    |    **1.584**    |  **3.186**  |      3.047      |   2.444   |
|   ours   | **1.792** |  **1.905**  |   **2.771**   |       2.720       |     3.129     |       3.107       |  **1.418**  |      1.542      |    3.154    |    **3.091**    | **2.462** |

Note:

Note:

1. With a larger identity loss, the image-to-image translation becomes more conservative, which makes less changes. The original authors did not say what is the best weight for identity loss. Thus, in addition to the default setting, we also set the weight of identity loss to 0 (denoting `id0`) to make a more comprehensive comparison.
2. This is the training log before refactoring. Updated logs will be released soon.

```latex
@inproceedings{zhu2017unpaired,
  title={Unpaired image-to-image translation using cycle-consistent adversarial networks},
  author={Zhu, Jun-Yan and Park, Taesung and Isola, Phillip and Efros, Alexei A},
  booktitle={Proceedings of the IEEE international conference on computer vision},
  pages={2223--2232},

booktitle={Proceedings of the IEEE international conference on computer vision},
  url={https://openaccess.thecvf.com/content_iccv_2017/html/Zhu_Unpaired_Image-To-Image_Translation_ICCV_2017_paper.html},
}
```

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_CVPR_2019/html/Sun_Deep_High-Resolution_Representation_Learning_for_Human_Pose_Estimation_CVPR_2019_paper.html">HRNet (CVPR'2019)</a></summary>

```bibtex
@inproceedings{sun2019deep,
  title={Deep high-resolution representation learning for human pose estimation},
  author={Sun, Ke and Xiao, Bin and Liu, Dong and Wang, Jingdong},
  pages={5693--5703},
}
```

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_CVPR_2020/html/Zhang_Distribution-Aware_Coordinate_Representation_for_Human_Pose_Estimation_CVPR_2020_paper.html">DarkPose (CVPR'2020)</a></summary>

```bibtex
@inproceedings{zhang2020distribution,
  title={Distribution-aware coordinate representation for human pose estimation},

```bibtex
  author={Zhang, Feng and Zhu, Xiatian and Dai, Hanbin and Ye, Mao and Zhu, Ce},
  pages={7093--7102},
}
```

<details>
<summary align="right"><a href="https://link.springer.com/chapter/10.1007/978-3-319-10602-1_48">COCO (ECCV'2014)</a></summary>

```bibtex
@inproceedings{lin2014microsoft,
  title={Microsoft coco: Common objects in context},
  author={Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll{\'a}r, Piotr and Zitnick, C Lawrence},
  booktitle={European conference on computer vision},
  pages={740--755},
  year={2014},
  organization={Springer}
}
```

booktitle={European conference on computer vision},
}
```

Results on COCO val2017 with detector having human AP of 56.4 on COCO val2017 dataset

| Arch                                          | Input Size |  AP   | AP<sup>50</sup> | AP<sup>75</sup> |  AR   | AR<sup>50</sup> |                     ckpt                      |                      log                      |
| :-------------------------------------------- | :--------: | :---: | :-------------: | :-------------: | :---: | :-------------: | :-------------------------------------------: | :-------------------------------------------: |

| [pose_hrnet_w32_dark](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_dark-8xb64-210e_coco-256x192.py) |  256x192   | 0.757 |      0.907      |      0.825      | 0.807 |      0.943      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_dark-8xb64-210e_coco-256x192-0e00bf12_20220914.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_dark-8xb64-210e_coco-256x192_20220914.log) |

| [pose_hrnet_w32_dark](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_dark-8xb64-210e_coco-384x288.py) |  384x288   | 0.766 |      0.907      |      0.829      | 0.815 |      0.943      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_dark-8xb64-210e_coco-384x288-9bab4c9b_20220917.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_dark-8xb64-210e_coco-384x288_20220917.log) |

| [pose_hrnet_w48_dark](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_dark-8xb32-210e_coco-256x192.py) |  256x192   | 0.764 |      0.907      |      0.831      | 0.814 |      0.942      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_dark-8xb32-210e_coco-256x192-e1ebdd6f_20220913.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_dark-8xb32-210e_coco-256x192_20220913.log) |

| [pose_hrnet_w48_dark](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_dark-8xb32-210e_coco-384x288.py) |  384x288   | 0.772 |      0.911      |      0.833      | 0.821 |      0.948      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_dark-8xb32-210e_coco-384x288-39c3c381_20220916.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_dark-8xb32-210e_coco-384x288_20220916.log) |

# Label Studio Annotations to COCO Script

[Label Studio](https://labelstud.io/) is a popular deep learning annotation tool that can be used for annotating various tasks. However, for keypoint annotation, Label Studio can not directly export to the COCO format required by MMPose. This article will explain how to use Label Studio to annotate keypoint data and convert it into the required COCO format using the [labelstudio2coco.py](../../../tools/dataset_converters/labelstudio2coco.py) tool.

## Label Studio Annotation Requirements

According to the COCO format requirements, each annotated instance needs to include information about keypoints, segmentation, and bounding box (bbox). However, Label Studio scatters this information across different instances during annotation. Therefore, certain rules need to be followed during annotation to ensure proper usage with the subsequent scripts.

1. Label Interface Setup

For a newly created Label Studio project, the label interface needs to be set up. There should be three types of annotations: `KeyPointLabels`, `PolygonLabels`, and `RectangleLabels`, which correspond to `keypoints`, `segmentation`, and `bbox` in the COCO format, respectively. The following is an example of a label interface. You can find the `Labeling Interface` in the project's `Settings`, click on `Code`, and paste the following example.

```xml
<View>
  <KeyPointLabels name="kp-1" toName="img-1">
      <Label value="person" background="#D4380D"/>
  </KeyPointLabels>
  <PolygonLabels name="polygonlabel" toName="img-1">
      <Label value="person" background="#0DA39E"/>
  </PolygonLabels>
  <RectangleLabels name="label" toName="img-1">
      <Label value="person" background="#DDA0EE"/>
  </RectangleLabels>
  <Image name="img-1" value="$img"/>
</View>
```

2. Annotation Order

<RectangleLabels name="label" toName="img-1">
</View>
```

Since it is necessary to combine annotations of different types into one instance, a specific order of annotation is required to determine whether the annotations belong to the same instance. Annotations should be made in the order of `KeyPointLabels` -> `PolygonLabels`/`RectangleLabels`. The order and number of `KeyPointLabels` should match the order and number of keypoints specified in the `dataset_info` in MMPose configuration file. The annotation order of `PolygonLabels` and `RectangleLabels` can be interchangeable, and only one of them needs to be annotated. The annotation should be within one instance starts with keypoints and ends with non-keypoints. The following image shows an annotation example:

*Note: The bbox and area will be calculated based on the later PolygonLabels/RectangleLabels. If you annotate PolygonLabels first, the bbox will be based on the range of the later RectangleLabels, and the area will be equal to the area of the rectangle. Conversely, they will be based on the minimum bounding rectangle of the polygon and the area of the polygon.*

![image](https://github.com/open-mmlab/mmpose/assets/15847281/b2d004d0-8361-42c5-9180-cfbac0373a94)

3. Exporting Annotations

Once the annotations are completed as described above, they need to be exported. Select the `Export` button on the project interface, choose the `JSON` format, and click `Export` to download the JSON file containing the labels.

*Note: The exported file only contains the labels and does not include the original images. Therefore, the corresponding annotated images need to be provided separately. It is not recommended to use directly uploaded files because Label Studio truncates long filenames. Instead, use the export COCO format tool available in the `Export` functionality, which includes a folder with the image files within the downloaded compressed package.*

![image](https://github.com/open-mmlab/mmpose/assets/15847281/9f54ca3d-8cdd-4d7f-8ed6-494badcfeaf2)

## Usage of the Conversion Tool Script

The conversion tool script is located at `tools/dataset_converters/labelstudio2coco.py`and can be used as follows:

```bash
python tools/dataset_converters/labelstudio2coco.py config.xml project-1-at-2023-05-13-09-22-91b53efa.json output/result.json
```

```bash
```

Where `config.xml` contains the code from the Labeling Interface mentioned earlier, `project-1-at-2023-05-13-09-22-91b53efa.json` is the JSON file exported from Label Studio, and `output/result.json` is the path to the resulting JSON file in COCO format. If the path does not exist, the script will create it automatically.

Afterward, place the image folder in the output directory to complete the conversion of the COCO dataset. The directory structure can be as follows:

```bash
.
├── images
│   ├── 38b480f2.jpg
│   └── aeb26f04.jpg
└── result.json

```

If you want to use this dataset in MMPose, you can make modifications like the following example:

```python
dataset=dict(
    type=dataset_type,
    data_root=data_root,
    data_mode=data_mode,
    ann_file='result.json',
    data_prefix=dict(img='images/'),
    pipeline=train_pipeline,
)
```

# Useful Tools

## Prompt Viewer

This tool allows you to directly view the generated prompt without starting the full training process. If the passed configuration is only the dataset configuration (such as `configs/datasets/nq/nq_gen.py`), it will display the original prompt defined in the dataset configuration. If it is a complete evaluation configuration (including the model and the dataset), it will display the prompt received by the selected model during operation.

Running method:

```bash
python tools/prompt_viewer.py CONFIG_PATH [-n] [-a] [-p PATTERN]
```

- `-n`: Do not enter interactive mode, select the first model (if any) and dataset by default.
- `-a`: View the prompts received by all models and all dataset combinations in the configuration.
- `-p PATTERN`: Do not enter interactive mode, select all datasets that match the input regular expression.

## Case Analyzer (To be updated)

Based on existing evaluation results, this tool produces inference error samples and full samples with annotation information.

```bash
python tools/case_analyzer.py CONFIG_PATH [-w WORK_DIR]
```

- `-w`: Work path, default is `'./outputs/default'`.

## Lark Bot

Users can configure the Lark bot to implement real-time monitoring of task status. Please refer to [this document](https://open.feishu.cn/document/ukTMukTMukTM/ucTM5YjL3ETO24yNxkjN?lang=zh-CN#7a28964d) for setting up the Lark bot.

Configuration method:

- Open the `configs/secrets.py` file, and add the following line to the file:

```python
lark_bot_url = 'YOUR_WEBHOOK_URL'
```

- Normally, the Webhook URL format is like https://open.feishu.cn/open-apis/bot/v2/hook/xxxxxxxxxxxxxxxxx .

- Inherit this file in the complete evaluation configuration

```

- To avoid the bot sending messages frequently and causing disturbance, the running status will not be reported automatically by default. If necessary, you can start status reporting through `-l` or `--lark`:

```bash
python run.py configs/eval_demo.py -l
```

## API Model Tester

This tool can quickly test whether the functionality of the API model is normal.

```bash
python tools/test_api_model.py [CONFIG_PATH] -n
```

## Prediction Merger

This tool can merge patitioned predictions.

```bash
python tools/prediction_merger.py CONFIG_PATH [-w WORK_DIR]
```

# ABINet

> [Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition](https://arxiv.org/abs/2103.06495)

Linguistic knowledge is of great benefit to scene text recognition. However, how to effectively model linguistic rules in end-to-end deep networks remains a research challenge. In this paper, we argue that the limited capacity of language models comes from: 1) implicitly language modeling; 2) unidirectional feature representation; and 3) language model with noise input. Correspondingly, we propose an autonomous, bidirectional and iterative ABINet for scene text recognition. Firstly, the autonomous suggests to block gradient flow between vision and language models to enforce explicitly language modeling. Secondly, a novel bidirectional cloze network (BCN) as the language model is proposed based on bidirectional feature representation. Thirdly, we propose an execution manner of iterative correction for language model which can effectively alleviate the impact of noise input. Additionally, based on the ensemble of iterative predictions, we propose a self-training method which can learn from unlabeled images effectively. Extensive experiments indicate that ABINet has superiority on low-quality images and achieves state-of-the-art results on several mainstream benchmarks. Besides, the ABINet trained with ensemble self-training shows promising improvement in realizing human-level recognition.

<img src="https://user-images.githubusercontent.com/22607038/145804331-9ae955dc-0d3b-41eb-a6b2-dc7c9f7c1bef.png"/>
</div>

## Dataset

### Train Dataset

| trainset  | instance_num | repeat_num |     note     |
| :-------: | :----------: | :--------: | :----------: |
|  Syn90k   |   8919273    |     1      |    synth     |
| SynthText |   7239272    |     1      | alphanumeric |

### Test Dataset

| testset | instance_num |   note    |
| :-----: | :----------: | :-------: |
| IIIT5K  |     3000     |  regular  |
|   SVT   |     647      |  regular  |
|  IC13   |     1015     |  regular  |
|  IC15   |     2077     | irregular |
|  SVTP   |     645      | irregular |
|  CT80   |     288      | irregular |

|                    methods                     |                     pretrained                     |        | Regular Text |           |           | Irregular Text |        | download                                         |

| :--------------------------------------------: | :------------------------------------------------: | :----: | :----------: | :-------: | :-------: | :------------: | :----: | :----------------------------------------------- |
|                                                |                                                    | IIIT5K |     SVT      | IC13-1015 | IC15-2077 |      SVTP      |  CT80  |                                                  |
| [ABINet-Vision](/configs/textrecog/abinet/abinet-vision_20e_st-an_mj.py) |                         -                          | 0.9523 |    0.9196    |  0.9369   |  0.7896   |     0.8403     | 0.8437 | [model](https://download.openmmlab.com/mmocr/textrecog/abinet/abinet-vision_20e_st-an_mj/abinet-vision_20e_st-an_mj_20220915_152445-85cfb03d.pth) \| [log](https://download.openmmlab.com/mmocr/textrecog/abinet/abinet-vision_20e_st-an_mj/20220915_152445.log) |

| [ABINet-Vision-TTA](/configs/textrecog/abinet/abinet-vision_20e_st-an_mj.py) |                         -                          | 0.9523 |    0.9196    |  0.9360   |  0.8175   |     0.8450     | 0.8542 |                                                  |
| [ABINet](/configs/textrecog/abinet/abinet_20e_st-an_mj.py) | [Pretrained](https://download.openmmlab.com/mmocr/textrecog/abinet/abinet_pretrain-45deac15.pth) | 0.9603 |    0.9397    |  0.9557   |  0.8146   |     0.8868     | 0.8785 | [model](https://download.openmmlab.com/mmocr/textrecog/abinet/abinet_20e_st-an_mj/abinet_20e_st-an_mj_20221005_012617-ead8c139.pth) \| [log](https://download.openmmlab.com/mmocr/textrecog/abinet/abinet_20e_st-an_mj/20221005_012617.log) |

| [ABINet-TTA](/configs/textrecog/abinet/abinet_20e_st-an_mj.py) | [Pretrained](https://download.openmmlab.com/mmocr/textrecog/abinet/abinet_pretrain-45deac15.pth) | 0.9597 |    0.9397    |  0.9527   |  0.8426   |     0.8930     | 0.8854 |                                                  |

```{note}
1. ABINet allows its encoder to run and be trained without decoder and fuser. Its encoder is designed to recognize texts as a stand-alone model and therefore can work as an independent text recognizer. We release it as ABINet-Vision.
2. Facts about the pretrained model: MMOCR does not have a systematic pipeline to pretrain the language model (LM) yet, thus the weights of LM are converted from [the official pretrained model](https://github.com/FangShancheng/ABINet). The weights of ABINet-Vision are directly used as the vision model of ABINet.
```

```bibtex
@article{fang2021read,

```

```bibtex
  title={Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition},
  author={Fang, Shancheng and Xie, Hongtao and Wang, Yuxin and Mao, Zhendong and Zhang, Yongdong},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
}
```

# How to Migrate MMPose 0.x Projects to MMPose 1.0

MMPose 1.0 has been refactored extensively and addressed many legacy issues. Most of the code in MMPose 1.0 will not be compatible with 0.x version.

To try our best to help you migrate your code and model, here are some major changes:

## Data Transformation

### Translation, Rotation and Scaling

The transformation methods `TopDownRandomShiftBboxCenter` and `TopDownGetRandomScaleRotation` in old version, will be merged into `RandomBBoxTransform`.

```Python
class RandomBBoxTransform(BaseTransform):
    r"""Rnadomly shift, resize and rotate the bounding boxes.

    Required Keys:

        - bbox_center
        - bbox_scale

    Modified Keys:

    Added Keys:
        - bbox_rotation

    Args:
        shift_factor (float): Randomly shift the bbox in range
            :math:`[-dx, dx]` and :math:`[-dy, dy]` in X and Y directions,

Added Keys:

    Args:
            where :math:`dx(y) = x(y)_scale \cdot shift_factor` in pixels.
            Defaults to 0.16
        shift_prob (float): Probability of applying random shift. Defaults to
            0.3
        scale_factor (Tuple[float, float]): Randomly resize the bbox in range
            :math:`[scale_factor[0], scale_factor[1]]`. Defaults to (0.5, 1.5)
        scale_prob (float): Probability of applying random resizing. Defaults
            to 1.0
        rotate_factor (float): Randomly rotate the bbox in
            :math:`[-rotate_factor, rotate_factor]` in degrees. Defaults
            to 80.0
        rotate_prob (float): Probability of applying random rotation. Defaults
            to 0.6
    """

                 shift_factor: float = 0.16,

to 80.0
            to 0.6
    """

                 shift_prob: float = 0.3,
                 scale_factor: Tuple[float, float] = (0.5, 1.5),
                 scale_prob: float = 1.0,
                 rotate_factor: float = 80.0,
                 rotate_prob: float = 0.6) -> None:
```

### Target Generation

The old methods like:

- `TopDownGenerateTarget`
- `TopDownGenerateTargetRegression`
- `BottomUpGenerateHeatmapTarget`
- `BottomUpGenerateTarget`

will be merged in to `GenerateTarget`, and the actual generation methods are implemented in [Codec](./user_guides/codecs.md).

```Python
class GenerateTarget(BaseTransform):
    """Encode keypoints into Target.

    The generated target is usually the supervision signal of the model
    learning, e.g. heatmaps or regression labels.

"""Encode keypoints into Target.

        - keypoints
        - keypoints_visible
        - dataset_keypoint_weights

        - The keys of the encoded items from the codec will be updated into
            the results, e.g. ``'heatmaps'`` or ``'keypoint_weights'``. See
            the specific codec for more details.

    Args:
        encoder (dict | list[dict]): The codec config for keypoint encoding.
            Both single encoder and multiple encoders (given as a list) are
            supported
        multilevel (bool): Determine the method to handle multiple encoders.
            If ``multilevel==True``, generate multilevel targets from a group
            of encoders of the same type (e.g. multiple :class:`MSRAHeatmap`
            encoders with different sigma values); If ``multilevel==False``,

of encoders of the same type (e.g. multiple :class:`MSRAHeatmap`
            generate combined targets from a group of different encoders. This
            argument will have no effect in case of single encoder. Defaults
            to ``False``
        use_dataset_keypoint_weights (bool): Whether use the keypoint weights
            from the dataset meta information. Defaults to ``False``
    """

                 encoder: MultiConfig,
                 multilevel: bool = False,
                 use_dataset_keypoint_weights: bool = False) -> None:
```

### Data Normalization

The data normalization operations `NormalizeTensor` and `ToTensor` will be replaced by **DataPreprocessor** module, which will no longer be used as a preprocessing operation, but will be merged as a part of the model forward propagation.

## Compatibility of Models

We have performed compatibility with the model weights provided by model zoo to ensure that the same model weights can get a comparable accuracy in both version. But note that due to the large number of differences in processing details, the inference outputs can be slightly different(less than 0.05% difference in accuracy).

For model weights saved by training with 0.x version, we provide a `_load_state_dict_pre_hook()` method in Head to replace the old version of the `state_dict` with the new one. If you wish to make your model compatible with MMPose 1.0, you can refer to our implementation as follows.

```Python
class YourHead(BaseHead):
def __init__(self):

    ## omitted

    # Register the hook to automatically convert old version state dicts
    self._register_load_state_dict_pre_hook(self._load_state_dict_pre_hook)
```

### Heatmap-based Model

## omitted

```

For models based on `SimpleBaseline` approach, developers need to pay attention to the last convolutional layer.

```Python
def _load_state_dict_pre_hook(self, state_dict, prefix, local_meta, *args,
                              **kwargs):
    version = local_meta.get('version', None)

    if version and version >= self._version:
        return

    # convert old-version state dict
    keys = list(state_dict.keys())
    for _k in keys:
        if not _k.startswith(prefix):
            continue
        v = state_dict.pop(_k)
        k = _k[len(prefix):]
        # In old version, "final_layer" includes both intermediate
        # conv layers (new "conv_layers") and final conv layers (new
        # "final_layer").
        #
        # If there is no intermediate conv layer, old "final_layer" will

# conv layers (new "conv_layers") and final conv layers (new
        #
        # have keys like "final_layer.xxx", which should be still
        # named "final_layer.xxx";
        #
        # If there are intermediate conv layers, old "final_layer"  will
        # have keys like "final_layer.n.xxx", where the weights of the last
        # one should be renamed "final_layer.xxx", and others should be
        # renamed "conv_layers.n.xxx"
        k_parts = k.split('.')
        if k_parts[0] == 'final_layer':
            if len(k_parts) == 3:
                assert isinstance(self.conv_layers, nn.Sequential)
                idx = int(k_parts[1])
                if idx < len(self.conv_layers):
                    # final_layer.n.xxx -> conv_layers.n.xxx
                    k_new = 'conv_layers.' + '.'.join(k_parts[1:])
                else:

if idx < len(self.conv_layers):
                else:
                    # final_layer.n.xxx -> final_layer.xxx
                    k_new = 'final_layer.' + k_parts[2]
            else:
                # final_layer.xxx remains final_layer.xxx
                k_new = k
        else:
            k_new = k

        state_dict[prefix + k_new] = v
```

### RLE-based Model

For the RLE-based models, since the loss module is renamed to `loss_module` in MMPose 1.0, and the flow model is subsumed under the loss module, changes need to be made to the keys in `state_dict`:

```Python

        return

**kwargs):

        return

        k = _k.lstrip(prefix)
        # In old version, "loss" includes the instances of loss,
        # now it should be renamed "loss_module"
        if k_parts[0] == 'loss':
            # loss.xxx -> loss_module.xxx
            k_new = prefix + 'loss_module.' + '.'.join(k_parts[1:])
        else:
            k_new = _k

        state_dict[k_new] = v
```

# Glaucoma grAding from Multi-Modality imAges Task3

This project support **`Glaucoma grAding from Multi-Modality imAges Task3`**, and the dataset used in this project can be downloaded from [here](https://aistudio.baidu.com/aistudio/competition/detail/121/0/datasets).

This regular-challenge dataset was provided by Sun Yat-sen Ophthalmic Center, Sun Yat-sen University, Guangzhou, China. The dataset contains 200 fundus color images: 100 pairs in the training set and 100 pairs in the test set.

### Original Statistic Information

| Dataset name                                                                        | Anatomical region | Task type    | Modality        | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License                                                         |

| ----------------------------------------------------------------------------------- | ----------------- | ------------ | --------------- | ------------ | --------------------- | ---------------------- | ------------ | --------------------------------------------------------------- |
| [GammaTask3](https://aistudio.baidu.com/aistudio/competition/detail/121/0/datasets) | eye               | segmentation | fundus photophy | 3            | 100/-/100             | yes/-/-                | 2021         | [CC-BY-NC 4.0](https://creativecommons.org/licenses/by-sa/4.0/) |

| background |    100     |   99.02    |    -     |    -     |     -     |     -     |
| optic disc |    100     |    0.67    |    -     |    -     |     -     |     -     |

| optic cup  |    100     |    0.31    |    -     |    -     |     -     |     -     |

Note:

![bac](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/fundus_photography/gamma3/gamma3_dataset.png)

## Dataset Citation

```bibtex
@article{fu2018joint,
  title={Joint optic disc and cup segmentation based on multi-label deep network and polar transformation},
  author={Fu, Huazhu and Cheng, Jun and Xu, Yanwu and Wong, Damon Wing Kee and Liu, Jiang and Cao, Xiaochun},
  journal={IEEE transactions on medical imaging},
  volume={37},
  number={7},
  pages={1597--1605},
  year={2018},
}

@article{sevastopolsky2017optic,

journal={IEEE transactions on medical imaging},
}

  title={Optic disc and cup segmentation methods for glaucoma detection with modification of U-Net convolutional neural network},
  author={Sevastopolsky, Artem},
  journal={Pattern Recognition and Image Analysis},
  volume={27},
  pages={618--624},
  publisher={Springer}
}
```

- pillow(PIL) v9.3.0
- scikit-learn(sklearn) v1.2.0

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `gammm3/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
```

- download dataset from [here](https://aistudio.baidu.com/aistudio/competition/detail/121/0/datasets) and decompression data to path `'data/'`.
- run script `"python tools/prepare_dataset.py"` to split dataset and change folder structure as below.

- run script `"python ../../tools/split_seg_dataset.py"` to split dataset and generate `train.txt`, `val.txt` and `test.txt`. If the label of official validation set and test set can't be obtained, we generate `train.txt` and `val.txt` from the training set randomly.

```none
  ├── mmseg
  │   │   │   ├── fundus_photography
  │   │   │   │   ├── gamma3
  │   │   │   │   │   │   │   ├── test

│   │   │   │   |   │   │   │   ├── xxx.png
  │   │   │   │   |   │   │   │   ├── yyy.png
  │   │   │   │   |   │   │   │   └── yyy.png
```

| background |     80     |   99.01    |    20    |  99.07   |     -     |     -     |
| optic disc |     80     |    0.68    |    20    |   0.63   |     -     |     -     |

| optic cup  |     80     |    0.32    |    20    |   0.31   |     -     |     -     |

To train models on a single server with one GPU. (default）

```shell
mim train mmseg ./configs/${CONFIG_PATH}
```

To test models on a single server with one GPU. (default）

```shell
mim test mmseg ./configs/${CONFIG_PATH}  --checkpoint ${CHECKPOINT_PATH}
```

  - [ ] Test-time correctness

Does the modification introduce changes that break the backward-compatibility of the downstream repositories?

**Before PR**:

- [ ] I have read and followed the workflow indicated in the [CONTRIBUTING.md](https://github.com/open-mmlab/mmocr/blob/main/.github/CONTRIBUTING.md) to create this PR.
- [ ] Pre-commit or linting tools indicated in [CONTRIBUTING.md](https://github.com/open-mmlab/mmocr/blob/main/.github/CONTRIBUTING.md) are used to fix the potential lint issues.
- [ ] Bug fixes are covered by unit tests, the case that causes the bug should be added in the unit tests.
- [ ] New functionalities are covered by complete unit tests. If not, please add more unit test to ensure the correctness.
- [ ] The documentation has been modified accordingly, including docstring or example tutorials.

**After PR**:

- [ ] If the modification has potential influence on downstream or other related projects, this PR should be tested with some of those projects.
- [ ] CLA has been signed and all committers have signed the CLA in this PR.

## Gaofen Image Dataset (GID)

- GID 数据集可在[此处](https://x-ytong.github.io/project/GID.html)进行下载。
- GID 数据集包含 150 张 6800x7200 的大尺寸图像，标签为 RGB 标签。
- 根据[文献](https://ieeexplore.ieee.org/document/9343296/)，此处选择 15 张图像生成训练集和验证集，该 15 张图像包含了所有六类信息。所选的图像名称如下：

```None
  GF2_PMS1__L1A0000647767-MSS1
  GF2_PMS1__L1A0001064454-MSS1
  GF2_PMS1__L1A0001348919-MSS1
  GF2_PMS1__L1A0001680851-MSS1
  GF2_PMS1__L1A0001680853-MSS1
  GF2_PMS1__L1A0001680857-MSS1
  GF2_PMS1__L1A0001757429-MSS1
  GF2_PMS2__L1A0000607681-MSS2
  GF2_PMS2__L1A0000635115-MSS2
  GF2_PMS2__L1A0000658637-MSS2
  GF2_PMS2__L1A0001206072-MSS2
  GF2_PMS2__L1A0001471436-MSS2
  GF2_PMS2__L1A0001642620-MSS2
  GF2_PMS2__L1A0001787089-MSS2
  GF2_PMS2__L1A0001838560-MSS2
```

这里也提供了一个脚本来方便的筛选出15张图像，

```
python projects/gid_dataset/tools/dataset_converters/gid_select15imgFromAll.py {150 张图像的路径} {150 张标签的路径} {15 张图像的路径} {15 张标签的路径}
```

在选择出 15 张图像后，执行以下命令进行裁切及标签的转换，需要修改为您所存储 15 张图像及标签的路径。

```

```
```

```
python projects/gid_dataset/tools/dataset_converters/gid.py {15 张图像的路径} {15 张标签的路径}
```

完成裁切后的 GID 数据结构如下：

```none
mmsegmentation
├── mmseg
├── tools
├── data
│   ├── gid
│   │   ├── ann_dir
|   │   │   │   ├── train
|   │   │   │   ├── val
│   │   ├── img_dir

```

<details>

```bibtex
}
```

<details>

```bibtex

```bibtex
}
```

| [pose_scnet_50](/configs/hand_2d_keypoint/topdown_heatmap/coco_wholebody_hand/td-hm_scnet50_8xb32-210e_coco-wholebody-hand-256x256.py) |  256x256   |  0.803  | 0.834 | 4.55 | [ckpt](https://download.openmmlab.com/mmpose/hand/scnet/scnet50_coco_wholebody_hand_256x256-e73414c7_20210909.pth) | [log](https://download.openmmlab.com/mmpose/hand/scnet/scnet50_coco_wholebody_hand_256x256_20210909.log.json) |

# 新增模块

## 开发新组件

我们可以自定义 [模型文档](./models.md) 中介绍的所有组件，例如**主干网络（backbone）**、**头（head）**、**损失函数（loss function）**和**数据预处理器（data preprocessor）**。

### 添加新的主干网络（backbone）

在这里，我们以 MobileNet 为例展示如何开发新的主干网络。

1. 创建一个新文件 `mmseg/models/backbones/mobilenet.py`。

   ```python
   import torch.nn as nn

   from mmseg.registry import MODELS

   @MODELS.register_module()
   class MobileNet(nn.Module):

       def __init__(self, arg1, arg2):
           pass

       def forward(self, x):  # should return a tuple
           pass

       def init_weights(self, pretrained=None):
           pass
   ```

2. 在 `mmseg/models/backbones/__init__.py` 中引入模块。

   ```python
   from .mobilenet import MobileNet
   ```

3. 在配置文件中使用它。

   ```python
   model = dict(
       ...
       backbone=dict(
           type='MobileNet',
           arg1=xxx,
           arg2=xxx),
       ...
   ```

### 添加新的头（head）

```

   ```python
       ...
           arg1=xxx,
       ...
   ```

在 MMSegmentation 中，我们提供 [BaseDecodeHead](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/models/decode_heads/decode_head.py#L17) 用于开发所有分割头。
所有新实现的解码头都应该从中派生出来。
接下来我们以 [PSPNet](https://arxiv.org/abs/1612.01105) 为例说明如何开发新的头。

首先，在 `mmseg/models/decode_heads/psp_head.py` 中添加一个新的解码头。
PSPNet 实现了用于分割解码的解码头。
为了实现解码头，在新模块中我们需要执行以下三个函数。

```python
from mmseg.registry import MODELS

class PSPHead(BaseDecodeHead):

    def __init__(self, pool_scales=(1, 2, 3, 6), **kwargs):
        super(PSPHead, self).__init__(**kwargs)

    def init_weights(self):
        pass

    def forward(self, inputs):
        pass
```

接下来，用户需要在 `mmseg/models/decode_heads/__init__.py` 中添加模块，这样相应的注册器就可以找到并加载它们。

PSPNet 的配置文件如下

```python
norm_cfg = dict(type='SyncBN', requires_grad=True)

pass
```

```python
    type='EncoderDecoder',
    pretrained='pretrain_model/resnet50_v1c_trick-2cccc1ad.pth',
        type='ResNetV1c',
        depth=50,
        dilations=(1, 1, 2, 4),
        strides=(1, 2, 1, 1),
        norm_cfg=norm_cfg,
        norm_eval=False,
        contract_dilation=True),
    decode_head=dict(
        type='PSPHead',
        in_channels=2048,
        in_index=3,
        channels=512,
        pool_scales=(1, 2, 3, 6),
        dropout_ratio=0.1,
        num_classes=19,
        align_corners=False,
        loss_decode=dict(
            type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0)))

```

### 添加新的损失函数（loss）

假设您想为分割解码添加一个叫做 `MyLoss` 的新的损失函数。

align_corners=False,

```

要添加新的损失函数，用户需要在 `mmseg/models/loss/my_loss.py` 中实现它。
修饰器 `weighted_loss` 可以对损失的每个元素进行加权。

```python

from .utils import weighted_loss

@weighted_loss
def my_loss(pred, target):
    assert pred.size() == target.size() and target.numel() > 0
    loss = torch.abs(pred - target)
    return loss

class MyLoss(nn.Module):

    def __init__(self, reduction='mean', loss_weight=1.0):
        super(MyLoss, self).__init__()
        self.reduction = reduction
        self.loss_weight = loss_weight

    def forward(self,
                pred,
                target,
                weight=None,
                avg_factor=None,
                reduction_override=None):

def forward(self,
                pred,
                target,
        assert reduction_override in (None, 'none', 'mean', 'sum')
        reduction = (
            reduction_override if reduction_override else self.reduction)
        loss = self.loss_weight * my_loss(
            pred, target, weight, reduction=reduction, avg_factor=avg_factor)
```

然后，用户需要将其添加到 `mmseg/models/loss/__init__.py` 中。

```python
from .my_loss import MyLoss, my_loss

```

要使用它，请修改 `loss_xx` 字段。
然后需要修改头中的 `loss_decode` 字段。
`loss_weight` 可用于平衡多重损失。

```python
loss_decode=dict(type='MyLoss', loss_weight=1.0))
```

### 添加新的数据预处理器（data preprocessor）

在 MMSegmentation 1.x 版本中，我们使用 [SegDataPreProcessor](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/models/data_preprocessor.py#L13) 将数据复制到目标设备，并将数据预处理为默认的模型输入格式。这里我们将展示如何开发一个新的数据预处理器。

1. 创建一个新文件 `mmseg/models/my_datapreprocessor.py`。

   ```python
   from mmengine.model import BaseDataPreprocessor

   class MyDataPreProcessor(BaseDataPreprocessor):
       def __init__(self, **kwargs):
           super().__init__(**kwargs)

       def forward(self, data: dict, training: bool=False) -> Dict[str, Any]:
           # TODO Define the logic for data pre-processing in the forward method
           pass
   ```

2. 在 `mmseg/models/__init__.py` 中导入数据预处理器

   ```python
   from .my_datapreprocessor import MyDataPreProcessor
   ```

   ```python
       data_preprocessor=dict(type='MyDataPreProcessor)
       ...
   )
   ```

## 开发新的分割器（segmentor）

```

   ```python
       ...
   )
   ```

分割器是一种户可以通过添加自定义组件和定义算法执行逻辑来自定义其算法的算法架构。请参考[模型文档](./models.md)了解更多详情。

由于 MMSegmentation 中的 [BaseSegmenter](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/models/segmentors/base.py#L15) 统一了前向过程的三种模式，为了开发新的分割器，用户需要重写与 `loss`、`predict` 和 `tensor` 相对应的 `loss`、`predict` 和 `_forward` 方法。

这里我们将展示如何开发一个新的分割器。

1. 创建一个新文件 `mmseg/models/segmentors/my_segmentor.py`。

   ```python
    from typing import Dict, Optional, Union

    import torch

    from mmseg.registry import MODELS
    from mmseg.models import BaseSegmentor

    @MODELS.register_module()
    class MySegmentor(BaseSegmentor):
        def __init__(self, **kwargs):
            super().__init__(**kwargs)
            # TODO users should build components of the network here

        def loss(self, inputs: Tensor, data_samples: SampleList) -> dict:

super().__init__(**kwargs)

            """Calculate losses from a batch of inputs and data samples."""
            pass

        def predict(self, inputs: Tensor, data_samples: OptSampleList=None) -> SampleList:
            """Predict results from a batch of inputs and data samples with post-
            processing."""
            pass

       def _forward(self,
                 inputs: Tensor,
                 data_samples: OptSampleList = None) -> Tuple[List[Tensor]]:
            """Network forward process.

            Usually includes backbone, neck and head forward without any post-
            processing.
            """
            pass
   ```

2. 在 `mmseg/models/segmentors/__init__.py` 中导入分割器。

   ```python
   from .my_segmentor import MySegmentor
   ```

   ```python
       type='MySegmentor'

```

   ```python
   ```

   ```python
       ...
   )
   ```

}

```

<details>

```bibtex
}
```

<details>

```bibtex

}
```

- Results on COCO val2017 with detector having human AP of 56.4 on COCO val2017 dataset.
- `*` denotes model trained on 7 public datasets:
  - [AI Challenger](https://mmpose.readthedocs.io/en/latest/dataset_zoo/2d_body_keypoint.html#aic)
  - [MS COCO](https://mmpose.readthedocs.io/en/latest/dataset_zoo/2d_body_keypoint.html#coco)
  - [CrowdPose](https://mmpose.readthedocs.io/en/latest/dataset_zoo/2d_body_keypoint.html#crowdpose)
  - [MPII](https://mmpose.readthedocs.io/en/latest/dataset_zoo/2d_body_keypoint.html#mpii)
  - [sub-JHMDB](https://mmpose.readthedocs.io/en/latest/dataset_zoo/2d_body_keypoint.html#sub-jhmdb-dataset)
  - [Halpe](https://mmpose.readthedocs.io/en/latest/dataset_zoo/2d_wholebody_keypoint.html#halpe)
  - [PoseTrack18](https://mmpose.readthedocs.io/en/latest/dataset_zoo/2d_body_keypoint.html#posetrack18)

- [PoseTrack18](https://mmpose.readthedocs.io/en/latest/dataset_zoo/2d_body_keypoint.html#posetrack18)
- `Body8` denotes the addition of the [OCHuman](https://mmpose.readthedocs.io/en/latest/dataset_zoo/2d_body_keypoint.html#ochuman) dataset, in addition to the 7 datasets mentioned above, for evaluation.

|                     Config                     | Input Size | AP<sup><br>(COCO) | PCK@0.1<sup><br>(Body8) | AUC<sup><br>(Body8) | EPE<sup><br>(Body8) | Params(M) | FLOPS(G) |                     Download                      |
| :--------------------------------------------: | :--------: | :---------------: | :---------------------: | :-----------------: | :-----------------: | :-------: | :------: | :-----------------------------------------------: |

| [RTMPose-t\*](/configs/body_2d_keypoint/rtmpose/body8/rtmpose-t_8xb256-420e_body8-256x192.py) |  256x192   |       65.9        |          91.44          |        63.18        |        19.45        |   3.34    |   0.36   | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-t_simcc-body7_pt-body7_420e-256x192-026a1439_20230504.pth) |
| [RTMPose-s\*](/configs/body_2d_keypoint/rtmpose/body8/rtmpose-s_8xb256-420e_body8-256x192.py) |  256x192   |       69.7        |          92.45          |        65.15        |        17.85        |   5.47    |   0.68   | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_simcc-body7_pt-body7_420e-256x192-acd4a1ef_20230504.pth) |

| [RTMPose-m\*](/configs/body_2d_keypoint/rtmpose/body8/rtmpose-m_8xb256-420e_body8-256x192.py) |  256x192   |       74.9        |          94.25          |        68.59        |        15.12        |   13.59   |   1.93   | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-body7_pt-body7_420e-256x192-e48f03d0_20230504.pth) |
| [RTMPose-l\*](/configs/body_2d_keypoint/rtmpose/body8/rtmpose-l_8xb256-420e_body8-256x192.py) |  256x192   |       76.7        |          95.08          |        70.14        |        13.79        |   27.66   |   4.16   | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-body7_pt-body7_420e-256x192-4dba18fc_20230504.pth) |

| [RTMPose-m\*](/configs/body_2d_keypoint/rtmpose/body8/rtmpose-m_8xb256-420e_body8-384x288.py) |  384x288   |       76.6        |          94.64          |        70.38        |        13.98        |   13.72   |   4.33   | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-body7_pt-body7_420e-384x288-65e718c4_20230504.pth) |
| [RTMPose-l\*](/configs/body_2d_keypoint/rtmpose/body8/rtmpose-l_8xb256-420e_body8-384x288.py) |  384x288   |       78.3        |          95.36          |        71.58        |        13.08        |   27.79   |   9.35   | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-body7_pt-body7_420e-384x288-3f5a1437_20230504.pth) |

# Learn about Codecs

In the keypoint detection task, depending on the algorithm, it is often necessary to generate targets in different formats, such as normalized coordinates, vectors and heatmaps, etc. Similarly, for the model outputs, a decoding process is required to transform them into coordinates.

Encoding and decoding are closely related and inverse each other. In earlier versions of MMPose, encoding and decoding are implemented at different modules, making it less intuitive and unified.

MMPose 1.0 introduced a new module **Codec** to integrate the encoding and decoding together in a modular and user-friendly form.

Here is a diagram to show where the `Codec` is:

![codec-en](https://user-images.githubusercontent.com/13503330/187112635-c01f13d1-a07e-420f-be50-3b8818524dec.png)

A typical codec consists of two parts:

- Encoder
- Decoder

### Encoder

The encoder transforms the coordinates in the input image space into the needed target format:

- Normalized Coordinates

- Encoder
- Decoder

- One-dimensional Vectors
- Gaussian Heatmaps

For example, in the Regression-based method, the encoder will be:

```Python
def encode(self,
           keypoints: np.ndarray,
           keypoints_visible: Optional[np.ndarray] = None) -> dict:
    """Encoding keypoints from input image space to normalized space.

    Args:
        keypoints (np.ndarray): Keypoint coordinates in shape (N, K, D)
        keypoints_visible (np.ndarray): Keypoint visibilities in shape
            (N, K)

    Returns:
        dict:
        - keypoint_labels (np.ndarray): The normalized regression labels in
            shape (N, K, D) where D is 2 for 2d coordinates
        - keypoint_weights (np.ndarray): The target weights in shape
            (N, K)
    """
    if keypoints_visible is None:

shape (N, K, D) where D is 2 for 2d coordinates
            (N, K)
    """
        keypoints_visible = np.ones(keypoints.shape[:2], dtype=np.float32)

    w, h = self.input_size
    valid = ((keypoints >= 0) &
             (keypoints <= [w - 1, h - 1])).all(axis=-1) & (
                 keypoints_visible > 0.5)

    keypoint_labels = (keypoints / np.array([w, h])).astype(np.float32)
    keypoint_weights = np.where(valid, 1., 0.).astype(np.float32)

    encoded = dict(
        keypoint_labels=keypoint_labels, keypoint_weights=keypoint_weights)

    return encoded
```

The encoded data is converted to Tensor format in `PackPoseInputs` and packed in `data_sample.gt_instance_labels` for model calls, which is generally used for loss calculation, as demonstrated by `loss()` in `RegressionHead`.

```Python

```Python

    ])

    losses.update(loss_kpt=loss)
    ### Omitted ###
```

### Decoder

The decoder transforms the model outputs into coordinates in the input image space, which is the opposite processing of the encoder.

For example, in the Regression-based method, the decoder will be:

```Python
def decode(self, encoded: np.ndarray) -> Tuple[np.ndarray, np.ndarray]:

```Python
    """Decode keypoint coordinates from normalized space to input image
    space.

    Args:
        encoded (np.ndarray): Coordinates in shape (N, K, D)

    Returns:
        tuple:
        - keypoints (np.ndarray): Decoded coordinates in shape (N, K, D)
        - scores (np.ndarray): The keypoint scores in shape (N, K).
            It usually represents the confidence of the keypoint prediction

    """

    if encoded.shape[-1] == 2:
        N, K, _ = encoded.shape
        normalized_coords = encoded.copy()
        scores = np.ones((N, K), dtype=np.float32)
    elif encoded.shape[-1] == 4:
        # split coords and sigma if outputs contain output_sigma
        normalized_coords = encoded[..., :2].copy()
        output_sigma = encoded[..., 2:4].copy()
        scores = (1 - output_sigma).mean(axis=-1)
    else:

normalized_coords = encoded[..., :2].copy()
    else:
        raise ValueError(
            'Keypoint dimension should be 2 or 4 (with sigma), '
            f'but got {encoded.shape[-1]}')

    keypoints = normalized_coords * np.array([w, h])

    return keypoints, scores
```

By default, the `decode()` method only performs decoding on a single instance. You can also implement the `batch_decode()` method to boost the decoding process.

## Common Usage

The example below shows how to use a codec in your config:

- Define the Codec
- Generate Targets
- Head

### Define the Codec

Take the Regression-based method to generate normalized coordinates as an example, you can define a `codec` in your config as follows:

```Python
codec = dict(type='RegressionLabel', input_size=(192, 256))
```

### Generate Targets

```Python
```

In pipelines, A codec should be passed into `GenerateTarget` to work as the `encoder`:

```Python
dict(type='GenerateTarget', encoder=codec)
```

### Head

In MMPose workflows, we decode the model outputs in `Head`, which requires a codec to work as the `decoder`:

```Python
head=dict(
    type='RLEHead',
    in_channels=2048,
    num_joints=17,
    loss=dict(type='RLELoss', use_target_weight=True),
    decoder=codec
)
```

Here is the phase of a config file:

```Python

# codec settings
codec = dict(type='RegressionLabel', input_size=(192, 256))                     ## definition ##

    type='TopdownPoseEstimator',
        type='PoseDataPreprocessor',
        bgr_to_rgb=True),
        depth=50,

type='PoseDataPreprocessor',
        depth=50,
        init_cfg=dict(type='Pretrained', checkpoint='torchvision://resnet50'),
    ),
    neck=dict(type='GlobalAveragePooling'),
    head=dict(
        type='RLEHead',
        num_joints=17,
        loss=dict(type='RLELoss', use_target_weight=True),
        decoder=codec),                                                         ## Head ##
    test_cfg=dict(
        flip_test=True,
        shift_coords=True,
    ))

# base dataset settings
dataset_type = 'CocoDataset'
data_mode = 'topdown'
data_root = 'data/coco/'

backend_args = dict(backend='local')

    dict(type='LoadImage', backend_args=backend_args),

    dict(type='GenerateTarget', encoder=codec),   ## Generate Target ##
]
]
```

# EfficientNet

> [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://arxiv.org/abs/1905.11946v5)

EfficientNets are a family of image classification models, which achieve state-of-the-art accuracy, yet being an order-of-magnitude smaller and faster than previous models.

EfficientNets are based on AutoML and Compound Scaling. In particular, we first use [AutoML MNAS Mobile framework](https://ai.googleblog.com/2018/08/mnasnet-towards-automating-design-of.html) to develop a mobile-size baseline network, named as EfficientNet-B0; Then, we use the compound scaling method to scale up this baseline to obtain EfficientNet-B1 to B7.

<img src="https://user-images.githubusercontent.com/26739999/150078232-d28c91fc-d0e8-43e3-9d20-b5162f0fb463.png" width="60%"/>
</div>

<details>

<summary>Click to show the detailed Abstract</summary>

<br>

Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are available. In this paper, we systematically study model scaling and identify that carefully balancing network depth, width, and resolution can lead to better performance. Based on this observation, we propose a new scaling method that uniformly scales all dimensions of depth/width/resolution using a simple yet highly effective compound coefficient. We demonstrate the effectiveness of this method on scaling up MobileNets and ResNet.   To go even further, we use neural architecture search to design a new baseline network and scale it up to obtain a family of models, called EfficientNets, which achieve much better accuracy and efficiency than previous ConvNets. In particular, our EfficientNet-B7 achieves state-of-the-art 84.3% top-1 accuracy on ImageNet, while being 8.4x smaller and 6.1x faster on inference than the best existing ConvNet. Our EfficientNets also transfer well and achieve state-of-the-art accuracy on CIFAR-100 (91.7%), Flowers (98.8%), and 3 other transfer learning datasets, with an order of magnitude fewer parameters.

<!-- [TABS-BEGIN] -->

**Predict image**

```python
from mmpretrain import inference_model

predict = inference_model('efficientnet-b0_3rdparty_8xb32_in1k', 'demo/bird.JPEG')
print(predict['pred_class'])
print(predict['pred_score'])
```

**Use the model**

```python
from mmpretrain import get_model

model = get_model('efficientnet-b0_3rdparty_8xb32_in1k', pretrained=True)
inputs = torch.rand(1, 3, 224, 224)
out = model(inputs)
print(type(out))
# To extract features.
feats = model.extract_feat(inputs)
print(type(feats))
```

Prepare your dataset according to the [docs](https://mmpretrain.readthedocs.io/en/latest/user_guides/dataset_prepare.html#prepare-dataset).

Test:

```shell
python tools/test.py configs/efficientnet/efficientnet-b0_8xb32_in1k.py https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b0_3rdparty_8xb32_in1k_20220119-a7e2a0b1.pth
```

<!-- [TABS-END] -->

```

### Image Classification on ImageNet-1k

| Model                                               |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                     Config                     |                        Download                        |
| :-------------------------------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :--------------------------------------------: | :----------------------------------------------------: |
| `efficientnet-b0_3rdparty_8xb32_in1k`\*             | From scratch |    5.29    |   0.42    |   76.74   |   93.17   |    [config](efficientnet-b0_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b0_3rdparty_8xb32_in1k_20220119-a7e2a0b1.pth) |

| `efficientnet-b0_3rdparty_8xb32-aa_in1k`\*          | From scratch |    5.29    |   0.42    |   77.26   |   93.41   |    [config](efficientnet-b0_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b0_3rdparty_8xb32-aa_in1k_20220119-8d939117.pth) |
| `efficientnet-b0_3rdparty_8xb32-aa-advprop_in1k`\*  | From scratch |    5.29    |   0.42    |   77.53   |   93.61   | [config](efficientnet-b0_8xb32-01norm_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b0_3rdparty_8xb32-aa-advprop_in1k_20220119-26434485.pth) |
| `efficientnet-b0_3rdparty-ra-noisystudent_in1k`\*   | From scratch |    5.29    |   0.42    |   77.63   |   94.00   |    [config](efficientnet-b0_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b0_3rdparty-ra-noisystudent_in1k_20221103-75cd08d3.pth) |

| `efficientnet-b1_3rdparty_8xb32_in1k`\*             | From scratch |    7.79    |   0.74    |   78.68   |   94.28   |    [config](efficientnet-b1_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b1_3rdparty_8xb32_in1k_20220119-002556d9.pth) |
| `efficientnet-b1_3rdparty_8xb32-aa_in1k`\*          | From scratch |    7.79    |   0.74    |   79.20   |   94.42   |    [config](efficientnet-b1_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b1_3rdparty_8xb32-aa_in1k_20220119-619d8ae3.pth) |
| `efficientnet-b1_3rdparty_8xb32-aa-advprop_in1k`\*  | From scratch |    7.79    |   0.74    |   79.52   |   94.43   | [config](efficientnet-b1_8xb32-01norm_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b1_3rdparty_8xb32-aa-advprop_in1k_20220119-5715267d.pth) |

| `efficientnet-b1_3rdparty-ra-noisystudent_in1k`\*   | From scratch |    7.79    |   0.74    |   81.44   |   95.83   |    [config](efficientnet-b1_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b1_3rdparty-ra-noisystudent_in1k_20221103-756bcbc0.pth) |
| `efficientnet-b2_3rdparty_8xb32_in1k`\*             | From scratch |    9.11    |   1.07    |   79.64   |   94.80   |    [config](efficientnet-b2_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b2_3rdparty_8xb32_in1k_20220119-ea374a30.pth) |
| `efficientnet-b2_3rdparty_8xb32-aa_in1k`\*          | From scratch |    9.11    |   1.07    |   80.21   |   94.96   |    [config](efficientnet-b2_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b2_3rdparty_8xb32-aa_in1k_20220119-dd61e80b.pth) |

| `efficientnet-b2_3rdparty_8xb32-aa-advprop_in1k`\*  | From scratch |    9.11    |   1.07    |   80.45   |   95.07   | [config](efficientnet-b2_8xb32-01norm_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b2_3rdparty_8xb32-aa-advprop_in1k_20220119-1655338a.pth) |
| `efficientnet-b2_3rdparty-ra-noisystudent_in1k`\*   | From scratch |    9.11    |   1.07    |   82.47   |   96.23   |    [config](efficientnet-b2_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b2_3rdparty-ra-noisystudent_in1k_20221103-301ed299.pth) |
| `efficientnet-b3_3rdparty_8xb32_in1k`\*             | From scratch |   12.23    |   1.95    |   81.01   |   95.34   |    [config](efficientnet-b3_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b3_3rdparty_8xb32_in1k_20220119-4b4d7487.pth) |

| `efficientnet-b3_3rdparty_8xb32-aa_in1k`\*          | From scratch |   12.23    |   1.95    |   81.58   |   95.67   |    [config](efficientnet-b3_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b3_3rdparty_8xb32-aa_in1k_20220119-5b4887a0.pth) |
| `efficientnet-b3_3rdparty_8xb32-aa-advprop_in1k`\*  | From scratch |   12.23    |   1.95    |   81.81   |   95.69   | [config](efficientnet-b3_8xb32-01norm_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b3_3rdparty_8xb32-aa-advprop_in1k_20220119-53b41118.pth) |
| `efficientnet-b3_3rdparty-ra-noisystudent_in1k`\*   | From scratch |   12.23    |   1.95    |   84.02   |   96.89   |    [config](efficientnet-b3_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b3_3rdparty-ra-noisystudent_in1k_20221103-a4ab5fd6.pth) |

| `efficientnet-b4_3rdparty_8xb32_in1k`\*             | From scratch |   19.34    |   4.66    |   82.57   |   96.09   |    [config](efficientnet-b4_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b4_3rdparty_8xb32_in1k_20220119-81fd4077.pth) |
| `efficientnet-b4_3rdparty_8xb32-aa_in1k`\*          | From scratch |   19.34    |   4.66    |   82.95   |   96.26   |    [config](efficientnet-b4_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b4_3rdparty_8xb32-aa_in1k_20220119-45b8bd2b.pth) |
| `efficientnet-b4_3rdparty_8xb32-aa-advprop_in1k`\*  | From scratch |   19.34    |   4.66    |   83.25   |   96.44   | [config](efficientnet-b4_8xb32-01norm_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b4_3rdparty_8xb32-aa-advprop_in1k_20220119-38c2238c.pth) |

| `efficientnet-b4_3rdparty-ra-noisystudent_in1k`\*   | From scratch |   19.34    |   4.66    |   85.25   |   97.52   |    [config](efficientnet-b4_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b4_3rdparty-ra-noisystudent_in1k_20221103-16ba8a2d.pth) |
| `efficientnet-b5_3rdparty_8xb32_in1k`\*             | From scratch |   30.39    |   10.80   |   83.18   |   96.47   |    [config](efficientnet-b5_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b5_3rdparty_8xb32_in1k_20220119-e9814430.pth) |
| `efficientnet-b5_3rdparty_8xb32-aa_in1k`\*          | From scratch |   30.39    |   10.80   |   83.82   |   96.76   |    [config](efficientnet-b5_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b5_3rdparty_8xb32-aa_in1k_20220119-2cab8b78.pth) |

| `efficientnet-b5_3rdparty_8xb32-aa-advprop_in1k`\*  | From scratch |   30.39    |   10.80   |   84.21   |   96.98   | [config](efficientnet-b5_8xb32-01norm_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b5_3rdparty_8xb32-aa-advprop_in1k_20220119-f57a895a.pth) |
| `efficientnet-b5_3rdparty-ra-noisystudent_in1k`\*   | From scratch |   30.39    |   10.80   |   86.08   |   97.75   |    [config](efficientnet-b5_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b5_3rdparty-ra-noisystudent_in1k_20221103-111a185f.pth) |
| `efficientnet-b6_3rdparty_8xb32-aa_in1k`\*          | From scratch |   43.04    |   19.97   |   84.05   |   96.82   |    [config](efficientnet-b6_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b6_3rdparty_8xb32-aa_in1k_20220119-45b03310.pth) |

| `efficientnet-b6_3rdparty_8xb32-aa-advprop_in1k`\*  | From scratch |   43.04    |   19.97   |   84.74   |   97.14   | [config](efficientnet-b6_8xb32-01norm_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b6_3rdparty_8xb32-aa-advprop_in1k_20220119-bfe3485e.pth) |
| `efficientnet-b6_3rdparty-ra-noisystudent_in1k`\*   | From scratch |   43.04    |   19.97   |   86.47   |   97.87   |    [config](efficientnet-b6_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b6_3rdparty-ra-noisystudent_in1k_20221103-7de7d2cc.pth) |
| `efficientnet-b7_3rdparty_8xb32-aa_in1k`\*          | From scratch |   66.35    |   39.32   |   84.38   |   96.88   |    [config](efficientnet-b7_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b7_3rdparty_8xb32-aa_in1k_20220119-bf03951c.pth) |

| `efficientnet-b7_3rdparty_8xb32-aa-advprop_in1k`\*  | From scratch |   66.35    |   39.32   |   85.14   |   97.23   | [config](efficientnet-b7_8xb32-01norm_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b7_3rdparty_8xb32-aa-advprop_in1k_20220119-c6dbff10.pth) |
| `efficientnet-b7_3rdparty-ra-noisystudent_in1k`\*   | From scratch |   66.35    |   39.32   |   86.83   |   98.08   |    [config](efficientnet-b7_8xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b7_3rdparty-ra-noisystudent_in1k_20221103-a82894bc.pth) |
| `efficientnet-b8_3rdparty_8xb32-aa-advprop_in1k`\*  | From scratch |   87.41    |   65.00   |   85.38   |   97.28   | [config](efficientnet-b8_8xb32-01norm_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-b8_3rdparty_8xb32-aa-advprop_in1k_20220119-297ce1b7.pth) |

| `efficientnet-l2_3rdparty-ra-noisystudent_in1k-800px`\* | From scratch |   480.31   |  174.20   |   88.33   |   98.65   |  [config](efficientnet-l2_8xb8_in1k-800px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-l2_3rdparty-ra-noisystudent_in1k_20221103-be73be13.pth) |
| `efficientnet-l2_3rdparty-ra-noisystudent_in1k-475px`\* | From scratch |   480.31   |  484.98   |   88.18   |   98.55   | [config](efficientnet-l2_8xb32_in1k-475px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/efficientnet/efficientnet-l2_3rdparty-ra-noisystudent_in1k-475px_20221103-5a0d8058.pth) |

*Models with * are converted from the [official repo](https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@inproceedings{tan2019efficientnet,

```bibtex
  title={Efficientnet: Rethinking model scaling for convolutional neural networks},
  author={Tan, Mingxing and Le, Quoc},
  booktitle={International Conference on Machine Learning},
  pages={6105--6114},
  organization={PMLR}
}
```

# Fast-SCNN

> [Fast-SCNN for Semantic Segmentation](https://arxiv.org/abs/1902.04502)

<a href="">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/backbones/fast_scnn.py#L272">Code Snippet</a>

The encoder-decoder framework is state-of-the-art for offline semantic image segmentation. Since the rise in autonomous systems, real-time computation is increasingly desirable. In this paper, we introduce fast segmentation convolutional neural network (Fast-SCNN), an above real-time semantic segmentation model on high resolution image data (1024x2048px) suited to efficient computation on embedded devices with low memory. Building on existing two-branch methods for fast segmentation, we introduce our \`learning to downsample' module which computes low-level features for multiple resolution branches simultaneously. Our network combines spatial detail at high resolution with deep features extracted at lower resolution, yielding an accuracy of 68.0% mean intersection over union at 123.5 frames per second on Cityscapes. We also show that large scale pre-training is unnecessary. We thoroughly validate our metric in experiments with ImageNet pre-training and the coarse labeled data of Cityscapes. Finally, we show even faster computation with competitive results on subsampled inputs, without any network modifications.

<img src="https://user-images.githubusercontent.com/24582831/142901444-705b4ff4-6d1e-409b-899a-37bf3a6b69ce.png" width="80%"/>
</div>

| Method   | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                       | download                                                                                                                                                                                                                                                                                                                                               |

| FastSCNN | FastSCNN | 512x1024  |  160000 | 3.3      | 56.45          | V100   | 70.96 | 72.65         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fastscnn/fast_scnn_8xb4-160k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fast_scnn/fast_scnn_lr0.12_8x4_160k_cityscapes/fast_scnn_lr0.12_8x4_160k_cityscapes_20210630_164853-0cec9937.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fast_scnn/fast_scnn_lr0.12_8x4_160k_cityscapes/fast_scnn_lr0.12_8x4_160k_cityscapes_20210630_164853.log.json) |

```bibtex
@article{poudel2019fast,
  title={Fast-scnn: Fast semantic segmentation network},
  author={Poudel, Rudra PK and Liwicki, Stephan and Cipolla, Roberto},
  journal={arXiv preprint arXiv:1902.04502},
}
```

# X3D

[X3D: Expanding Architectures for Efficient Video Recognition](https://openaccess.thecvf.com/content_CVPR_2020/html/Feichtenhofer_X3D_Expanding_Architectures_for_Efficient_Video_Recognition_CVPR_2020_paper.html)

This paper presents X3D, a family of efficient video networks that progressively expand a tiny 2D image classification architecture along multiple network axes, in space, time, width and depth. Inspired by feature selection methods in machine learning, a simple stepwise network expansion approach is employed that expands a single axis in each step, such that good accuracy to complexity trade-off is achieved. To expand X3D to a specific target complexity, we perform progressive forward expansion followed by backward contraction. X3D achieves state-of-the-art performance while requiring 4.8x and 5.5x fewer multiply-adds and parameters for similar accuracy as previous work. Our most surprising finding is that networks with high spatiotemporal resolution can perform well, while being extremely light in terms of network width and parameters. We report competitive accuracy at unprecedented efficiency on video classification and detection benchmarks.

<img src="https://user-images.githubusercontent.com/34324155/143019391-6711febb-9e5d-4bec-85b9-65f5179e93a2.png" width="800"/>
</div>

| frame sampling strategy | resolution | backbone | top1 10-view | top1 30-view |           reference top1 10-view           |           reference top1 30-view           |           config           |           ckpt            |
| :---------------------: | :--------: | :------: | :----------: | :----------: | :----------------------------------------: | :----------------------------------------: | :------------------------: | :-----------------------: |

|         13x6x1          |  160x160   |  X3D_S   |     73.2     |     73.3     | 73.1 \[[SlowFast](https://github.com/facebookresearch/SlowFast/blob/master/MODEL_ZOO.md)\] | 73.5 \[[SlowFast](https://github.com/facebookresearch/SlowFast/blob/master/MODEL_ZOO.md)\] | [config](/configs/recognition/x3d/x3d_s_13x6x1_facebook-kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/x3d/facebook/x3d_s_13x6x1_facebook-kinetics400-rgb_20201027-623825a0.pth)\[1\] |
|         16x5x1          |  224x224   |  X3D_M   |     75.2     |     76.4     | 75.1 \[[SlowFast](https://github.com/facebookresearch/SlowFast/blob/master/MODEL_ZOO.md)\] | 76.2 \[[SlowFast](https://github.com/facebookresearch/SlowFast/blob/master/MODEL_ZOO.md)\] | [config](/configs/recognition/x3d/x3d_m_16x5x1_facebook-kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/x3d/facebook/x3d_m_16x5x1_facebook-kinetics400-rgb_20201027-3f42382a.pth)\[1\] |

\[1\] The models are ported from the repo [SlowFast](https://github.com/facebookresearch/SlowFast/) and tested on our data. Currently, we only support the testing of X3D models, training will be available soon.

1. The values in columns named after "reference" are the results got by testing the checkpoint released on the original repo and codes, using the same dataset with ours.
2. The validation set of Kinetics400 we used is same as the repo [SlowFast](https://github.com/facebookresearch/SlowFast/), which is available [here](https://github.com/facebookresearch/video-nonlocal-net/issues/67).

## Test

```shell
```

Example: test X3D model on Kinetics-400 dataset and dump the result to a pkl file.

```shell

```shell
```

```shell
python tools/test.py configs/recognition/x3d/x3d_s_13x6x1_facebook-kinetics400-rgb.py \
```

```BibTeX
@misc{feichtenhofer2020x3d,
      title={X3D: Expanding Architectures for Efficient Video Recognition},
      author={Christoph Feichtenhofer},
      year={2020},
      eprint={2004.04730},
}
```

# RTMPose-Deploy

[中文说明](./README_CN.md)

RTMPose-Deploy is a C ++ code example for RTMPose localized deployment.

At present, RTMPose-Deploy has completed to use ONNXRuntime-CPU and TensorRT to deploy the RTMDet and RTMPose on the Windows system.

| Deployment Framework | Repo                                                                 |
| -------------------- | -------------------------------------------------------------------- |
| ONNXRuntime-CPU      | [RTMPose-Deploy](https://github.com/HW140701/RTMPose-Deploy)         |
| TensorRT             | [rtmpose_tensorrt](https://github.com/Dominic23331/rtmpose_tensorrt) |

# Waymo Dataset

This page provides specific tutorials about the usage of MMDetection3D for Waymo dataset.

## Prepare dataset

Before preparing Waymo dataset, if you only installed requirements in `requirements/build.txt` and `requirements/runtime.txt` before, please install the official package for this dataset at first by running

```
# tf 2.1.0.
pip install waymo-open-dataset-tf-2-1-0==1.2.0
# tf 2.0.0
# pip install waymo-open-dataset-tf-2-0-0==1.2.0
# tf 1.15.0
# pip install waymo-open-dataset-tf-1-15-0==1.2.0
```

or

```
pip install -r requirements/optional.txt
```

Due to the original Waymo data format is based on `tfrecord`, we need to preprocess the raw data for convenient usage in the training and evaluation procedure. Our approach is to convert them into KITTI format.

```

```
├── tools
├── data
│   ├── waymo
│   │   ├── waymo_format
│   │   │   ├── training
│   │   │   ├── validation
│   │   │   ├── testing
│   │   │   ├── gt.bin
│   │   ├── kitti_format
│   │   │   ├── ImageSets

```

│   ├── waymo

```

You can download Waymo open dataset V1.2 [HERE](https://waymo.com/open/download/) and its data split [HERE](https://drive.google.com/drive/folders/18BVuF_RYJF0NjZpt8SnfzANiakoRMf0o?usp=sharing). Then put `tfrecord` files into corresponding folders in `data/waymo/waymo_format/` and put the data split txt files into `data/waymo/kitti_format/ImageSets`. Download ground truth bin files for validation set [HERE](https://console.cloud.google.com/storage/browser/waymo_open_dataset_v_1_2_0/validation/ground_truth_objects) and put it into `data/waymo/waymo_format/`. A tip is that you can use `gsutil` to download the large-scale dataset with commands. You can take this [tool](https://github.com/RalphMao/Waymo-Dataset-Tool) as an example for more details. Subsequently, prepare Waymo data by running

```bash
python tools/create_data.py waymo --root-path ./data/waymo/ --out-dir ./data/waymo/ --workers 128 --extra-tag waymo
```

Note that if your local disk does not have enough space for saving converted data, you can change the `--out-dir` to anywhere else. Just remember to create folders and prepare data there in advance and link them back to `data/waymo/kitti_format` after the data conversion.

After the data conversion, the folder structure and info files should be organized as below.

```
├── tools
├── data
│   ├── waymo
│   │   │   │   ├── calib
│   │   │   │   ├── image_0
│   │   │   │   ├── image_1
│   │   │   │   ├── image_2
│   │   │   │   ├── image_3
│   │   │   │   ├── image_4
│   │   │   │   ├── label_0
│   │   │   │   ├── label_1

│   │   │   │   ├── label_2
│   │   │   │   ├── label_3
│   │   │   │   ├── label_4
│   │   │   │   ├── label_all
│   │   │   │   ├── pose
│   │   │   │   ├── velodyne
│   │   │   │   ├── (the same as training)
│   │   │   ├── waymo_gt_database
│   │   │   ├── waymo_infos_trainval.pkl
│   │   │   ├── waymo_infos_train.pkl
│   │   │   ├── waymo_infos_val.pkl
│   │   │   ├── waymo_infos_test.pkl
│   │   │   ├── waymo_dbinfos_train.pkl

```

```

Here because there are several cameras, we store the corresponding image and labels that can be projected to that camera respectively and save pose for further usage of consecutive frames point clouds. We use a coding way `{a}{bbb}{ccc}` to name the data for each frame, where `a` is the prefix for different split (`0` for training, `1` for validation and `2` for testing), `bbb` for segment index and `ccc` for frame index. You can easily locate the required frame according to this naming rule. We gather the data for training and validation together as KITTI and store the indices for different set in the `ImageSet` files.

## Training

Considering there are many similar frames in the original dataset, we can basically use a subset to train our model primarily. In our preliminary baselines, we load one frame every five frames, and thanks to our hyper parameters settings and data augmentation, we obtain a better result compared with the performance given in the original dataset [paper](https://arxiv.org/pdf/1912.04838.pdf). For more details about the configuration and performance, please refer to README.md in the `configs/pointpillars/`. A more complete benchmark based on other settings and methods is coming soon.

For evaluation on Waymo, please follow the [instruction](https://github.com/waymo-research/waymo-open-dataset/blob/master/docs/quick_start.md) to build the binary file `compute_detection_metrics_main` for metrics computation and put it into `mmdet3d/core/evaluation/waymo_utils/`.  Basically, you can follow the commands below to install `bazel` and build the file.

```shell

```shell
# download the code and enter the base directory
git clone https://github.com/waymo-research/waymo-open-dataset.git waymo-od
# git clone https://github.com/Abyssaledge/waymo-open-dataset-master waymo-od # if you want to use faster multi-thread version.
cd waymo-od
git checkout remotes/origin/master

# use the Bazel build system
sudo apt-get install --assume-yes pkg-config zip g++ zlib1g-dev unzip python3 python3-pip
BAZEL_VERSION=3.1.0
wget https://github.com/bazelbuild/bazel/releases/download/${BAZEL_VERSION}/bazel-${BAZEL_VERSION}-installer-linux-x86_64.sh
sudo bash bazel-${BAZEL_VERSION}-installer-linux-x86_64.sh
sudo apt install build-essential

# configure .bazelrc
./configure.sh
# delete previous bazel outputs and reset internal caches
bazel clean

bazel build waymo_open_dataset/metrics/tools/compute_detection_metrics_main
cp bazel-bin/waymo_open_dataset/metrics/tools/compute_detection_metrics_main ../mmdetection3d/mmdet3d/evaluation/functional/waymo_utils/
```

```

Then you can evaluate your models on Waymo. An example to evaluate PointPillars on Waymo with 8 GPUs with Waymo metrics is as follows.

```shell
./tools/dist_test.sh configs/pointpillars/pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymo-3d-car.py checkpoints/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car_latest.pth
```

`pklfile_prefix` should be set in `test_evaluator` of configuration if the bin file is needed to be generated, so you can add `--cfg-options "test_evaluator.pklfile_prefix=xxxx"` in the end of command if you want do it.

**Notice**:

1. Sometimes when using `bazel` to build `compute_detection_metrics_main`, an error `'round' is not a member of 'std'` may appear. We just need to remove the `std::` before `round` in that file.

2. Considering it takes a little long time to evaluate once, we recommend to evaluate only once at the end of model training.

3. To use TensorFlow with CUDA 9, it is recommended to compile it from source. Apart from official tutorials, you can refer to this [link](https://github.com/SmileTM/Tensorflow2.X-GPU-CUDA9.0) for possibly suitable precompiled packages and useful information for compiling it from source.

An example to test PointPillars on Waymo with 8 GPUs, generate the bin files and make a submission to the leaderboard.

`submission_prefix` should be set in `test_evaluator` of configuration before you run the test command if you want to generate the bin files and make a submission to the leaderboard..

After generating the bin file, you can simply build the binary file `create_submission` and use them to create a submission file by following the [instruction](https://github.com/waymo-research/waymo-open-dataset/blob/master/docs/quick_start.md/). Basically, here are some example commands.

```shell
cd ../waymo-od/
bazel build waymo_open_dataset/metrics/tools/create_submission
cp bazel-bin/waymo_open_dataset/metrics/tools/create_submission ../mmdetection3d/mmdet3d/core/evaluation/waymo_utils/
vim waymo_open_dataset/metrics/tools/submission.txtpb  # set the metadata information
cp waymo_open_dataset/metrics/tools/submission.txtpb ../mmdetection3d/mmdet3d/evaluation/functional/waymo_utils/

cd ../mmdetection3d
# suppose the result bin is in `results/waymo-car/submission`

mmdet3d/core/evaluation/waymo_utils/create_submission  --input_filenames='results/waymo-car/kitti_results_test.bin' --output_filename='results/waymo-car/submission/model' --submission_filename='mmdet3d/evaluation/functional/waymo_utils/submission.txtpb'

tar cvf results/waymo-car/submission/my_model.tar results/waymo-car/submission/my_model/
gzip results/waymo-car/submission/my_model.tar
```

For evaluation on the validation set with the eval server, you can also use the same way to generate a submission. Make sure you change the fields in `submission.txtpb` before running the command above.

# Overview

Along with the release of OpenMMLab 2.0, MMOCR 1.0 made many significant changes, resulting in less redundant, more efficient code and a more consistent overall design. However, these changes break backward compatibility. We understand that with such huge changes, it is not easy for users familiar with the old version to adapt to the new version. Therefore, we prepared a detailed migration guide to make the transition as smooth as possible so that all users can enjoy the productivity benefits of the new MMOCR and the entire OpenMMLab 2.0 ecosystem.

MMOCR 1.0 depends on the new foundational library for training deep learning models [MMEngine](https://github.com/open-mmlab/mmengine), and therefore has an entirely different dependency chain compared with MMOCR 0.x. Even if you have a well-rounded MMOCR 0.x environment before, you still need to create a new python environment for MMOCR 1.0. We provide a detailed [installation guide](../get_started/install.md) for reference.
```

Next, please read the sections according to your requirements.

- Read [What's new in MMOCR 1.x](./news.md) to learn about the new features and changes in MMOCR 1.x.
- If you want to migrate a model trained in version 0.x to use it directly in version 1.0, please read [Pretrained Model Migration](./model.md).
- If you want to train the model, please read [Dataset Migration](./dataset.md) and [Data Transform Migration](./transforms.md).

- If you want to develop on MMOCR, please read [Code Migration](code.md), [Branch Migration](branches.md) and [Upstream Library Changes](https://github.com/open-mmlab/mmengine/tree/main/docs/en/migration).

As shown in the following figure, the maintenance plan of MMOCR 1.x version is mainly divided into three stages, namely "RC Period", "Compatibility Period" and "Maintenance Period". For old versions, we will no longer add major new features. Therefore, we strongly recommend users to migrate to MMOCR 1.x version as soon as possible.

![plan](https://user-images.githubusercontent.com/45810070/192927112-70c0108d-58ed-4c77-8a0a-9d9685a48333.png)

# Preparing Composition-1k Dataset

```bibtex
@inproceedings{xu2017deep,
  title={Deep image matting},
  author={Xu, Ning and Price, Brian and Cohen, Scott and Huang, Thomas},
  pages={2970--2979},
}
```

The Adobe Composition-1k dataset consists of foreground images and their corresponding alpha images.
To get the full dataset, one need to composite the foregrounds with selected backgrounds from the COCO dataset and the Pascal VOC dataset.

## Obtain and Extract

Please follow the instructions of [paper authors](https://sites.google.com/view/deepimagematting) to obtain the Composition-1k (comp1k) dataset.

## Composite the full dataset

The Adobe composition-1k dataset contains only `alpha` and `fg` (and `trimap` in test set).
It is needed to merge `fg` with COCO data (training) or VOC data (test) before training or evaluation.

Use the following script to perform image composition and generate annotation files for training or testing:

```shell
# The script is run under the root folder of MMagic
python tools/dataset_converters/matting/comp1k/preprocess_comp1k_dataset.py data/adobe_composition-1k data/coco data/VOCdevkit --composite
```

The generated data is stored under `adobe_composition-1k/Training_set` and `adobe_composition-1k/Test_set` respectively.
If you only want to composite test data (since compositing training data is time-consuming), you can skip compositing the training set by removing the `--composite` option:

```shell
# skip compositing training set
python tools/dataset_converters/matting/comp1k/preprocess_comp1k_dataset.py data/adobe_composition-1k data/coco data/VOCdevkit
```

```shell
```

If you only want to preprocess test data, i.e. for FBA, you can skip the train set by adding the `--skip-train` option:

```shell
# skip preprocessing training set
python tools/data/matting/comp1k/preprocess_comp1k_dataset.py data/adobe_composition-1k data/coco data/VOCdevkit --skip-train
```

> Currently, `GCA` and `FBA` support online composition of training data. But you can modify the data pipeline of other models to perform online composition instead of loading composited images (we called it `merged` in our data pipeline).

## Check Directory Structure for DIM

The result folder structure should look like:

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── adobe_composition-1k
│   │   ├── Test_set
│   │   │   ├── Adobe-licensed images
│   │   │   │   ├── alpha
│   │   │   │   ├── fg

```text
mmagic
├── mmagic
├── tools
├── data
│   │   │   │   ├── fg
│   │   │   │   ├── trimaps
│   │   │   ├── merged  (generated by tools/dataset_converters/matting/comp1k/preprocess_comp1k_dataset.py)
│   │   │   ├── bg      (generated by tools/dataset_converters/matting/comp1k/preprocess_comp1k_dataset.py)
│   │   ├── Training_set
│   │   │   │   ├── fg
│   │   │   ├── Other
│   │   │   │   ├── fg
│   │   ├── test_list.json     (generated by tools/dataset_converters/matting/comp1k/preprocess_comp1k_dataset.py)

│   │   ├── training_list.json (generated by tools/dataset_converters/matting/comp1k/preprocess_comp1k_dataset.py)
│   ├── coco
│   │   ├── train2014   (or train2017)
│   ├── VOCdevkit
│   │   ├── VOC2012
```

## Prepare the dataset for FBA

FBA adopts dynamic dataset augmentation proposed in [Learning-base Sampling for Natural Image Matting](https://openaccess.thecvf.com/content_CVPR_2019/papers/Tang_Learning-Based_Sampling_for_Natural_Image_Matting_CVPR_2019_paper.pdf).
In addition, to reduce artifacts during augmentation, it uses the extended version of foreground as foreground.
We provide scripts to estimate foregrounds.

Prepare the test set as follows:

```shell
# skip preprocessing training set, as it composites online during training

```shell
python tools/dataset_converters/matting/comp1k/preprocess_comp1k_dataset.py data/adobe_composition-1k data/coco data/VOCdevkit --skip-train
```

Extend the foreground of training set as follows:

```shell
python tools/dataset_converters/matting/comp1k/extend_fg.py data/adobe_composition-1k
```

The final folder structure should look like:

```text
mmagic
├── mmagic
├── tools
├── data
│   │   │   │   ├── fg
│   │   │   ├── merged  (generated by tools/data/matting/comp1k/preprocess_comp1k_dataset.py)
│   │   │   ├── bg      (generated by tools/data/matting/comp1k/preprocess_comp1k_dataset.py)

│   │   │   │   ├── fg
│   │   │   │   ├── fg_extended (generated by tools/data/matting/comp1k/extend_fg.py)
│   │   │   │   ├── fg
│   │   ├── test_list.json         (generated by tools/data/matting/comp1k/preprocess_comp1k_dataset.py)
│   │   ├── training_list_fba.json (generated by tools/data/matting/comp1k/extend_fg.py)
│   ├── coco
```

# 训练与测试

为了适配多样化的用户需求，MMOCR 实现了多种不同操作系统及设备上的模型训练及测试。无论是使用本地机器进行单机单卡训练测试，还是在部署了 slurm 系统的大规模集群上进行训练测试，MMOCR 都提供了便捷的解决方案。

## 单卡机器训练及测试

### 训练

`tools/train.py` 实现了基础的训练服务。MMOCR 推荐用户使用 GPU 进行模型训练和测试，但是，用户也可以通过指定 `CUDA_VISIBLE_DEVICES=-1` 来使用 CPU 设备进行模型训练及测试。例如，以下命令演示了如何使用 CPU 或单卡 GPU 来训练 DBNet 文本检测器。

```bash
# 通过调用 tools/train.py 来训练指定的 MMOCR 模型
CUDA_VISIBLE_DEVICES= python tools/train.py ${CONFIG_FILE} [PY_ARGS]

# 训练
# 示例 1：使用 CPU 训练 DBNet
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py

# 示例 2：指定使用 gpu:0 训练 DBNet，指定工作目录为 dbnet/，并打开混合精度（amp）训练
CUDA_VISIBLE_DEVICES=0 python tools/train.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py --work-dir dbnet/ --amp
```

```{note}
此外，如需使用指定编号的 GPU 进行训练或测试，例如使用3号 GPU，则可以通过设定 CUDA_VISIBLE_DEVICES=3 来实现。
```

下表列出了 `train.py` 支持的所有参数。其中，不带 `--` 前缀的参数为必须的位置参数，带 `--` 前缀的参数为可选参数。

| 参数            | 类型 | 说明                                                           |

```

| --------------- | ---- | -------------------------------------------------------------- |
| config          | str  | （必须）配置文件路径。                                         |
| --work-dir      | str  | 指定工作目录，用于存放训练日志以及模型 checkpoints。           |
| --resume        | bool | 是否从断点处恢复训练。                                         |
| --amp           | bool | 是否使用混合精度。                                             |
| --auto-scale-lr | bool | 是否使用学习率自动缩放。                                       |
| --cfg-options   | str  | 用于覆写配置文件中的指定参数。[示例](#添加示例)                |
| --launcher      | str  | 启动器选项，可选项目为 \['none', 'pytorch', 'slurm', 'mpi'\]。 |
| --local_rank    | int  | 本地机器编号，用于多机多卡分布式训练，默认为 0。               |

### 测试

`tools/test.py` 提供了基础的测试服务，其使用原理和训练脚本类似。例如，以下命令演示了 CPU 或 GPU 单卡测试 DBNet 模型。

```bash
# 通过调用 tools/test.py 来测试指定的 MMOCR 模型

### 测试

```bash
CUDA_VISIBLE_DEVICES= python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [PY_ARGS]

# 测试
# 示例 1：使用 CPU 测试 DBNet
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py dbnet_r50.pth
# 示例 2：使用 gpu:0 测试 DBNet
CUDA_VISIBLE_DEVICES=0 python tools/test.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py dbnet_r50.pth
```

下表列出了 `test.py` 支持的所有参数。其中，不带 `--` 前缀的参数为必须的位置参数，带 `--` 前缀的参数为可选参数。

| 参数          | 类型  | 说明                                                           |
| ------------- | ----- | -------------------------------------------------------------- |
| config        | str   | （必须）配置文件路径。                                         |
| checkpoint    | str   | （必须）待测试模型路径。                                       |

| --work-dir    | str   | 工作目录，用于存放训练日志以及模型 checkpoints。               |
| --save-preds  | bool  | 是否将预测结果写入 pkl 文件并保存。                            |
| --show        | bool  | 是否可视化预测结果。                                           |
| --show-dir    | str   | 将可视化的预测结果保存至指定路径。                             |
| --wait-time   | float | 可视化间隔时间（秒），默认为 2 秒。                            |
| --cfg-options | str   | 用于覆写配置文件中的指定参数。[示例](#添加示例)                |
| --launcher    | str   | 启动器选项，可选项目为 \['none', 'pytorch', 'slurm', 'mpi'\]。 |
| --local_rank  | int   | 本地机器编号，用于多机多卡分布式训练，默认为 0。               |
| --tta         | bool  | 是否使用测试时数据增强                                         |

## 多卡机器训练及测试

对于大规模模型，采用多 GPU 训练和测试可以极大地提升操作的效率。为此，MMOCR 提供了基于 [MMDistributedDataParallel](mmengine.model.wrappers.MMDistributedDataParallel) 实现的分布式脚本 `tools/dist_train.sh` 和 `tools/dist_test.sh`。

```bash
# 训练
NNODES=${NNODES} NODE_RANK=${NODE_RANK} PORT=${MASTER_PORT} MASTER_ADDR=${MASTER_ADDR} ./tools/dist_train.sh ${CONFIG_FILE} ${GPU_NUM} [PY_ARGS]
# 测试
NNODES=${NNODES} NODE_RANK=${NODE_RANK} PORT=${MASTER_PORT} MASTER_ADDR=${MASTER_ADDR} ./tools/dist_test.sh ${CONFIG_FILE} ${CHECKPOINT_FILE} ${GPU_NUM} [PY_ARGS]
```

下表列出了 `dist_*.sh` 支持的参数：

| 参数            | 类型 | 说明                                                                               |
| --------------- | ---- | ---------------------------------------------------------------------------------- |

| NNODES          | int  | 总共使用的机器节点个数，默认为 1。                                                 |
| NODE_RANK       | int  | 节点编号，默认为 0。                                                               |
| PORT            | int  | 在 RANK 0 机器上使用的 MASTER_PORT 端口号，取值范围是 0 至 65535，默认值为 29500。 |
| MASTER_ADDR     | str  | RANK 0 机器的 IP 地址，默认值为 127.0.0.1。                                        |
| CONFIG_FILE     | str  | （必须）指定配置文件的地址。                                                       |
| CHECKPOINT_FILE | str  | （必须，仅在 dist_test.sh 中适用）指定模型权重的地址。                             |
| GPU_NUM         | int  | （必须）指定 GPU 的数量。                                                          |
| \[PY_ARGS\]     | str  | 该部分一切的参数都会被直接传入 tools/train.py 或 tools/test.py 中。                |

这两个脚本可以实现**单机多卡**或**多机多卡**的训练和测试，下面演示了它们在不同场景下的用法。

### 单机多卡

以下命令演示了如何在搭载多块 GPU 的**单台机器**上使用指定数目的 GPU 进行训练及测试：

### 单机多卡

1. **训练**

   使用单台机器上的 4 块 GPU 训练 DBNet。

   ```bash
   # 单机 4 卡训练 DBNet
   tools/dist_train.sh configs/textdet/dbnet/dbnet_r50dcnv2_fpnc_1200e_icdar2015.py 4
   ```

2. **测试**

   使用单台机器上的 4 块 GPU 测试 DBNet。

   ```bash
   # 单机 4 卡测试 DBNet
   tools/dist_test.sh configs/textdet/dbnet/dbnet_r50dcnv2_fpnc_1200e_icdar2015.py dbnet_r50.pth 4
   ```

### 单机多任务训练及测试

对于搭载多块 GPU 的单台服务器而言，用户可以通过指定 GPU 的形式来同时执行不同的训练任务。例如，以下命令演示了如何在一台 8 卡 GPU 服务器上分别使用 `[0, 1, 2, 3]` 卡测试 DBNet 及 `[4, 5, 6, 7]` 卡训练 CRNN：

```bash
# 指定使用 gpu:0,1,2,3 测试 DBNet，并分配端口号 29500
CUDA_VISIBLE_DEVICES=0,1,2,3 PORT=29500 ./tools/dist_test.sh configs/textdet/dbnet/dbnet_r50dcnv2_fpnc_1200e_icdar2015.py dbnet_r50.pth 4
# 指定使用 gpu:4,5,6,7 训练 CRNN，并分配端口号 29501
CUDA_VISIBLE_DEVICES=4,5,6,7 PORT=29501 ./tools/dist_train.sh configs/textrecog/crnn/crnn_academic_dataset.py 4
```

```{note}

```

```{note}
`dist_train.sh` 默认将 `MASTER_PORT` 设置为 `29500`，当单台机器上有其它进程已占用该端口时，程序则会出现运行时错误 `RuntimeError: Address already in use`。此时，用户需要将 `MASTER_PORT` 设置为 `(0~65535)` 范围内的其它空闲端口号。
```

### 多机多卡训练及测试

MMOCR 基于[torch.distributed](https://pytorch.org/docs/stable/distributed.html#launch-utility) 提供了相同局域网下的多台机器间的多卡分布式训练。

1. **训练**

   以下命令演示了如何在两台机器上分别使用 2 张 GPU 合计 4 卡训练 DBNet：

   ```bash
   # 示例：在两台机器上分别使用 2 张 GPU 合计 4 卡训练 DBNet
   # 在 “机器1” 上运行以下命令
   NNODES=2 NODE_RANK=0 PORT=29501 MASTER_ADDR=10.140.0.169 tools/dist_train.sh configs/textdet/dbnet/dbnet_r50dcnv2_fpnc_1200e_icdar2015.py 2
   # 在 “机器2” 上运行以下命令
   NNODES=2 NODE_RANK=1 PORT=29501 MASTER_ADDR=10.140.0.169 tools/dist_train.sh configs/textdet/dbnet/dbnet_r50dcnv2_fpnc_1200e_icdar2015.py 2
   ```

2. **测试**

   以下命令演示了如何在两台机器上分别使用 2 张 GPU 合计 4 卡测试：

   ```bash

```

2. **测试**

   ```bash
   # 示例：在两台机器上分别使用 2 张 GPU 合计 4 卡测试
   NNODES=2 NODE_RANK=0 PORT=29500 MASTER_ADDR=10.140.0.169 tools/dist_test.sh configs/textdet/dbnet/dbnet_r50dcnv2_fpnc_1200e_icdar2015.py dbnet_r50.pth 2
   NNODES=2 NODE_RANK=1 PORT=29501 MASTER_ADDR=10.140.0.169 tools/dist_test.sh configs/textdet/dbnet/dbnet_r50dcnv2_fpnc_1200e_icdar2015.py dbnet_r50.pth 2
   ```

   ```{note}
   需要注意的是，采用多机多卡训练时，机器间的网络传输速度可能成为训练速度的瓶颈。
   ```

## 集群训练及测试

针对 [Slurm](https://slurm.schedmd.com/) 调度系统管理的计算集群，MMOCR 提供了对应的训练和测试任务提交脚本 `tools/slurm_train.sh` 及 `tools/slurm_test.sh`。

```bash
# tools/slurm_train.sh 提供基于 slurm 调度系统管理的计算集群上提交训练任务的脚本
GPUS=${GPUS} GPUS_PER_NODE=${GPUS_PER_NODE} CPUS_PER_TASK=${CPUS_PER_TASK} SRUN_ARGS=${SRUN_ARGS} ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} ${CONFIG_FILE} ${WORK_DIR} [PY_ARGS]

# tools/slurm_test.sh 提供基于 slurm 调度系统管理的计算集群上提交测试任务的脚本

GPUS=${GPUS} GPUS_PER_NODE=${GPUS_PER_NODE} CPUS_PER_TASK=${CPUS_PER_TASK} SRUN_ARGS=${SRUN_ARGS} ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} ${CONFIG_FILE} ${CHECKPOINT_FILE} ${WORK_DIR} [PY_ARGS]
```

| 参数            | 类型 | 说明                                                                      |
| --------------- | ---- | ------------------------------------------------------------------------- |
| GPUS            | int  | 使用的 GPU 数目，默认为8。                                                |
| GPUS_PER_NODE   | int  | 每台节点机器上搭载的 GPU 数目，默认为8。                                  |
| CPUS_PER_TASK   | int  | 任务使用的 CPU 个数，默认为5。                                            |
| SRUN_ARGS       | str  | 其他 srun 支持的参数。详见[这里](https://slurm.schedmd.com/srun.html)     |
| PARTITION       | str  | （必须）指定使用的集群分区。                                              |

| JOB_NAME        | str  | （必须）提交任务的名称。                                                  |
| WORK_DIR        | str  | （必须）任务的工作目录，训练日志以及模型的 checkpoints 将被保存至该目录。 |
| CHECKPOINT_FILE | str  | （必须，仅在 slurm_test.sh 中适用）指向模型权重的地址。                   |
| \[PY_ARGS\]     | str  | tools/train.py 以及 tools/test.py 支持的参数。                            |

这两个脚本可以实现 slurm 集群上的训练和测试，下面演示了它们在不同场景下的用法。

1. 训练

   以下示例为在 slurm 集群 dev 分区申请 1 块 GPU 进行 DBNet 训练。

```bash
# 示例：在 slurm 集群 dev 分区申请 1块 GPU 资源进行 DBNet 训练任务
GPUS=1 GPUS_PER_NODE=1 CPUS_PER_TASK=5 tools/slurm_train.sh dev db_r50 configs/textdet/dbnet/dbnet_r50dcnv2_fpnc_1200e_icdar2015.py work_dir
```

2. 测试

   同理， 则提供了测试任务提交脚本。以下示例为在 slurm 集群 dev 分区申请 1 块 GPU 资源进行 DBNet 测试。

```bash
# 示例：在 slurm 集群 dev 分区申请 1块 GPU 资源进行 DBNet 测试任务

```

2. 测试

```bash
GPUS=1 GPUS_PER_NODE=1 CPUS_PER_TASK=5 tools/slurm_test.sh dev db_r50 configs/textdet/dbnet/dbnet_r50dcnv2_fpnc_1200e_icdar2015.py dbnet_r50.pth work_dir
```

## 进阶技巧

### 从断点恢复训练

`tools/train.py` 提供了从断点恢复训练的功能，用户仅需在命令中指定 `--resume` 参数，即可自动从断点恢复训练。

```bash
# 示例：从断点恢复训练
python tools/train.py configs/textdet/dbnet/dbnet_r50dcnv2_fpnc_1200e_icdar2015.py 4 --resume
```

默认地，程序将自动从上次训练过程中最后成功保存的断点，即 `latest.pth` 处开始继续训练。如果用户希望指定从特定的断点处开始恢复训练，则可以按如下格式在模型的配置文件中设定该断点的路径。

```python
# 示例：在配置文件中设置想要加载的断点路径
load_from = 'work_dir/dbnet/models/epoch_10000.pth'
```

### 混合精度训练

混合精度训练可以在缩减内存占用的同时提升训练速度，为此，MMOCR 提供了一键式的混合精度训练方案，仅需在训练时添加 `--amp` 参数即可。

```bash
# 示例：使用自动混合精度训练
python tools/train.py configs/textdet/dbnet/dbnet_r50dcnv2_fpnc_1200e_icdar2015.py 4 --amp
```

下表列出了 MMOCR 中各算法对自动混合精度训练的支持情况：

|               | 是否支持混合精度训练 |             备注              |

```

| ------------- | :------------------: | :---------------------------: |
|               |       文本检测       |                               |
| DBNet         |          是          |                               |
| DBNetpp       |          是          |                               |
| DRRG          |          否          | roi_align_rotated 不支持 fp16 |
| FCENet        |          否          |      BCELoss 不支持 fp16      |
| Mask R-CNN    |          是          |                               |
| PANet         |          是          |                               |
| PSENet        |          是          |                               |
| TextSnake     |          否          |                               |
|               |       文本识别       |                               |

|               |       文本识别       |                               |
| ABINet        |          是          |                               |
| ASTER         |          是          |                               |
| CRNN          |          是          |                               |
| MASTER        |          是          |                               |
| NRTR          |          是          |                               |
| RobustScanner |          是          |                               |
| SAR           |          是          |                               |
| SATRN         |          是          |                               |

### 自动学习率缩放

MMOCR 在配置文件中为每一个模型设置了默认的初始学习率，然而，当用户使用的 `batch_size` 不同于我们预设的 `base_batch_size` 时，这些初始学习率可能不再完全适用。因此，我们提供了自动学习率缩放工具。当使用不同于 MMOCR 预设的 `base_batch_size` 进行训练时，用户仅需添加 `--auto-scale-lr` 参数即可自动依据新的 `batch_size` 将学习率缩放至对应尺度。

```bash
# 示例：使用自动学习率缩放

```bash
python tools/train.py configs/textdet/dbnet/dbnet_r50dcnv2_fpnc_1200e_icdar2015.py 4 --auto-scale-lr
```

### 可视化模型测试结果

`tools/test.py` 提供了可视化接口，以方便用户对模型进行定性分析。

![可视化文本检测模型](../../../demo/resources/det_vis.png)

(绿色框为真实标注，红色框为预测结果)

</div>

![可视化文本识别模型](../../../demo/resources/rec_vis.png)

（绿色字体为真实标注，红色字体为预测结果）

</div>

![可视化关键信息抽取模型结果](../../../demo/resources/kie_vis.png)

（从左至右分别为：原图，文本检测和识别结果，文本分类结果，关系图）

</div>

```bash
# 示例1：每间隔 2 秒绘制出
python tools/test.py configs/textdet/dbnet/dbnet_r50dcnv2_fpnc_1200e_icdar2015.py dbnet_r50.pth --show --wait-time 2

# 示例2：对于不支持图形化界面的系统（如计算集群等），可以将可视化结果存入指定路径
python tools/test.py configs/textdet/dbnet/dbnet_r50dcnv2_fpnc_1200e_icdar2015.py dbnet_r50.pth --show-dir ./vis_results
```

`tools/test.py` 中可视化相关参数说明：

| 参数        | 类型  | 说明                             |
| ----------- | ----- | -------------------------------- |

```

| --show      | bool  | 是否绘制可视化结果。             |
| --show-dir  | str   | 可视化图片存储路径。             |
| --wait-time | float | 可视化间隔时间（秒），默认为 2。 |

### 测试时数据增强

测试时增强，指的是在推理（预测）阶段，将原始图片进行水平翻转、垂直翻转、对角线翻转、旋转角度等数据增强操作，得到多张图，分别进行推理，再对多个结果进行综合分析，得到最终输出结果。
为此，MMOCR 提供了一键式测试时数据增强，仅需在测试时添加 `--tta` 参数即可。

```{note}
TTA 仅支持文本识别模型。
```

```bash
python tools/test.py configs/textrecog/crnn/crnn_mini-vgg_5e_mj.py checkpoints/crnn_mini-vgg_5e_mj.pth --tta
```

# Preparing DIV2K Dataset

```bibtex
@InProceedings{Agustsson_2017_CVPR_Workshops,
    author = {Agustsson, Eirikur and Timofte, Radu},
    title = {NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study},
    booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month = {July},
    year = {2017}
}
```

- Training dataset:  [DIV2K dataset](https://data.vision.ee.ethz.ch/cvl/DIV2K/).
- Validation dataset:  [Set5](https://drive.google.com/drive/folders/1B3DJGQKB6eNdwuQIhdskA64qUuVKLZ9u) and [Set14](https://drive.google.com/drive/folders/1B3DJGQKB6eNdwuQIhdskA64qUuVKLZ9u).

Note that we merge the original val dataset (image names from 0801 to 0900) to the original train dataset (image names from 0001 to 0800). The folder structure should look like:

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── DIV2K
│   │   ├── DIV2K_train_HR
│   │   │   ├── 0001.png
│   │   │   ├── 0002.png

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── DIV2K
│   │   │   ├── ...
│   │   │   ├── 0800.png
│   │   │   ├── 0801.png
│   │   │   ├── ...
│   │   │   ├── 0900.png
│   │   ├── DIV2K_train_LR_bicubic
│   │   │   ├── X2
│   │   │   ├── X3
│   │   │   ├── X4
│   │   ├── DIV2K_valid_HR
│   │   ├── DIV2K_valid_LR_bicubic
│   │   │   ├── X2
│   │   │   ├── X3
│   │   │   ├── X4
│   ├── Set5
│   │   ├── GTmod12
│   │   ├── LRbicx2
│   │   ├── LRbicx3
│   │   ├── LRbicx4
│   ├── Set14
```

## Crop sub-images

For faster IO, we recommend to crop the DIV2K images to sub-images. We provide such a script:

```shell
python tools/dataset_converters/div2k/preprocess_div2k_dataset.py --data-root ./data/DIV2K
```

```shell
```

The generated data is stored under `DIV2K` and the data structure is as follows, where `_sub` indicates the sub-images.

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── DIV2K
│   │   ├── DIV2K_train_HR_sub
│   │   │   ├── X2
│   │   │   ├── X3
│   │   │   ├── X4
│   │   │   ├── X2_sub
│   │   │   ├── X3_sub
│   │   │   ├── X4_sub
│   │   ├── ...
│   │   ├── meta_info_DIV2K800sub_GT.txt
│   │   ├── meta_info_DIV2K100sub_GT.txt
...
```

## Prepare annotation list

If you use the annotation mode for the dataset, you first need to prepare a specific `txt` file.

...
```

Each line in the annotation file contains the image names and image shape (usually for the ground-truth images), separated by a white space.

Example of an annotation file:

```text
0001_s001.png (480,480,3)
0001_s002.png (480,480,3)
```

Note that `preprocess_div2k_dataset` will generate default annotation files.

## Prepare LMDB dataset for DIV2K

If you want to use LMDB datasets for faster IO speed, you can make LMDB files by:

```shell
python tools/dataset_converters/div2k/preprocess_div2k_dataset.py --data-root ./data/DIV2K --make-lmdb
```

# 3D Body Mesh Recovery Datasets

It is recommended to symlink the dataset root to `$MMPOSE/data`.
If your folder structure is different, you may need to change the corresponding paths in config files.

To achieve high-quality human mesh estimation, we use multiple datasets for training.
The following items should be prepared for human mesh training:

- [3D Body Mesh Recovery Datasets](#3d-body-mesh-recovery-datasets)
  - [Notes](#notes)
    - [Annotation Files for Human Mesh Estimation](#annotation-files-for-human-mesh-estimation)
    - [SMPL Model](#smpl-model)
  - [COCO](#coco)
  - [Human3.6M](#human36m)
  - [MPI-INF-3DHP](#mpi-inf-3dhp)
  - [LSP](#lsp)
  - [LSPET](#lspet)
  - [CMU MoShed Data](#cmu-moshed-data)

## Notes

### Annotation Files for Human Mesh Estimation

For human mesh estimation, we use multiple datasets for training.
The annotation of different datasets are preprocessed to the same format. Please

follow the [preprocess procedure](https://github.com/nkolot/SPIN/tree/master/datasets/preprocess)
of SPIN to generate the annotation files or download the processed files from
[here](https://download.openmmlab.com/mmpose/datasets/mesh_annotation_files.zip),
and make it look like this:

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── mesh_annotation_files
        ├── coco_2014_train.npz
        ├── h36m_valid_protocol1.npz
        ├── h36m_valid_protocol2.npz
        ├── hr-lspet_train.npz
        ├── lsp_dataset_original_train.npz
        ├── mpi_inf_3dhp_train.npz
        └── mpii_train.npz
```

### SMPL Model

```bibtex
@article{loper2015smpl,
  title={SMPL: A skinned multi-person linear model},

├── mpi_inf_3dhp_train.npz
```

```bibtex
  author={Loper, Matthew and Mahmood, Naureen and Romero, Javier and Pons-Moll, Gerard and Black, Michael J},
  journal={ACM transactions on graphics (TOG)},
  volume={34},
  number={6},
  pages={1--16},
  publisher={ACM New York, NY, USA}
}
```

For human mesh estimation, SMPL model is used to generate the human mesh.
Please download the [gender neutral SMPL model](http://smplify.is.tue.mpg.de/),
[joints regressor](https://download.openmmlab.com/mmpose/datasets/joints_regressor_cmr.npy)
and [mean parameters](https://download.openmmlab.com/mmpose/datasets/smpl_mean_params.npz)
under `$MMPOSE/models/smpl`, and make it look like this:

```text
mmpose
├── mmpose
├── ...
├── models
    │── smpl
        ├── joints_regressor_cmr.npy
        ├── smpl_mean_params.npz
        └── SMPL_NEUTRAL.pkl
```

## COCO

```text
mmpose
├── mmpose
├── ...
├── models
    │── smpl
```

## COCO

<details>

```bibtex
}
```

For [COCO](http://cocodataset.org/) data, please download from [COCO download](http://cocodataset.org/#download). COCO'2014 Train is needed for human mesh estimation training.
Download and extract them under $MMPOSE/data, and make them look like this:

```text
mmpose

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── coco
        │-- train2014
        │   ├── COCO_train2014_000000000009.jpg
        │   ├── COCO_train2014_000000000025.jpg
        │   ├── COCO_train2014_000000000030.jpg
        |   │-- ...

```

## Human3.6M

<details>
<summary align="right"><a href="https://ieeexplore.ieee.org/abstract/document/6682899/">Human3.6M (TPAMI'2014)</a></summary>

```bibtex
@article{h36m_pami,
  author = {Ionescu, Catalin and Papava, Dragos and Olaru, Vlad and Sminchisescu,  Cristian},
  title = {Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  publisher = {IEEE Computer Society},
  volume = {36},
  number = {7},
  pages = {1325-1339},
  month = {jul},
  year = {2014}
}
```

publisher = {IEEE Computer Society},
}
```

For [Human3.6M](http://vision.imar.ro/human3.6m/description.php), we use the MoShed data provided in [HMR](https://github.com/akanazawa/hmr) for training.
However, due to license limitations, we are not allowed to redistribute the MoShed data.

For the evaluation on Human3.6M dataset, please follow the
[preprocess procedure](https://github.com/nkolot/SPIN/tree/master/datasets/preprocess)
of SPIN to extract test images from
[Human3.6M](http://vision.imar.ro/human3.6m/description.php) original videos,

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── Human3.6M
        ├── images
            ├── S11_Directions_1.54138969_000001.jpg
            ├── S11_Directions_1.54138969_000006.jpg
            ├── S11_Directions_1.54138969_000011.jpg
            ├── ...
```

├── S11_Directions_1.54138969_000001.jpg
            ├── ...
```

The download of Human3.6M dataset is quite difficult, you can also download the
[zip file](https://drive.google.com/file/d/1WnRJD9FS3NUf7MllwgLRJJC-JgYFr8oi/view?usp=sharing)
of the test images. However, due to the license limitations, we are not allowed to
redistribute the images either. So the users need to download the original video and
extract the images by themselves.

## MPI-INF-3DHP

```bibtex
@inproceedings{mono-3dhp2017,
 author = {Mehta, Dushyant and Rhodin, Helge and Casas, Dan and Fua, Pascal and Sotnychenko, Oleksandr and Xu, Weipeng and Theobalt, Christian},
 title = {Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision},
 booktitle = {3D Vision (3DV), 2017 Fifth International Conference on},
 url = {http://gvv.mpi-inf.mpg.de/3dhp_dataset},

booktitle = {3D Vision (3DV), 2017 Fifth International Conference on},
 year = {2017},
 organization={IEEE},
 doi={10.1109/3dv.2017.00064},
}
```

For [MPI-INF-3DHP](http://gvv.mpi-inf.mpg.de/3dhp-dataset/), please follow the
of SPIN to sample images, and make them like this:

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    ├── mpi_inf_3dhp_test_set
    │   ├── TS1
    │   ├── TS2
    │   ├── TS3
    │   ├── TS4
    │   ├── TS5
    │   └── TS6
    ├── S1
    │   ├── Seq1
    │   └── Seq2
    ├── S2
    │   ├── Seq1
    │   └── Seq2
    ├── S3
    │   ├── Seq1
    │   └── Seq2
    ├── S4
    │   ├── Seq1
    │   └── Seq2
    ├── S5
    │   ├── Seq1
    │   └── Seq2
    ├── S6
    │   ├── Seq1
    │   └── Seq2
    ├── S7
    │   ├── Seq1
    │   └── Seq2
    └── S8
        ├── Seq1
        └── Seq2
```

## LSP

├── S5
    │   ├── Seq1
    │   └── Seq2
    ├── S6
    │   ├── Seq1
    │   └── Seq2
    ├── S7
    │   ├── Seq1
    │   └── Seq2
    └── S8
        ├── Seq1
        └── Seq2
```

## LSP

```bibtex
@inproceedings{johnson2010clustered,
  title={Clustered Pose and Nonlinear Appearance Models for Human Pose Estimation.},
  author={Johnson, Sam and Everingham, Mark},
  booktitle={bmvc},
  volume={2},
  number={4},
  pages={5},
  year={2010},
  organization={Citeseer}
}
```

For [LSP](https://sam.johnson.io/research/lsp.html), please download the high resolution version
[LSP dataset original](http://sam.johnson.io/research/lsp_dataset_original.zip).
Extract them under `$MMPOSE/data`, and make them look like this:

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── lsp_dataset_original
        ├── images
            ├── im0001.jpg
            ├── im0002.jpg
            └── ...
```

## LSPET

```bibtex

├── tools
`── data
        ├── images
            └── ...
```

## LSPET

```bibtex
@inproceedings{johnson2011learning,
  title={Learning effective human pose estimation from inaccurate annotation},
  booktitle={CVPR 2011},
  pages={1465--1472},
  year={2011},
  organization={IEEE}
}
```

For [LSPET](https://sam.johnson.io/research/lspet.html), please download its high resolution form
[HR-LSPET](http://datasets.d2.mpi-inf.mpg.de/hr-lspet/hr-lspet.zip).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── lspet_dataset
        ├── images
        │   ├── im00001.jpg
        │   ├── im00002.jpg
        │   ├── im00003.jpg
        │   └── ...
        └── joints.mat
```

## CMU MoShed Data

```bibtex

├── images
        │   └── ...
```

```bibtex
@inproceedings{kanazawa2018end,
  title={End-to-end recovery of human shape and pose},
  author={Kanazawa, Angjoo and Black, Michael J and Jacobs, David W and Malik, Jitendra},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={7122--7131},
}
```

Real-world SMPL parameters are used for the adversarial training in human mesh estimation.
The MoShed data provided in [HMR](https://github.com/akanazawa/hmr) is included in this
[zip file](https://download.openmmlab.com/mmpose/datasets/mesh_annotation_files.zip).
Please download and extract it under `$MMPOSE/data`, and make it look like this:

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
        ├── CMU_mosh.npz

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
        └── ...
```

<details>

```bibtex
}
```

<details>

```bibtex

```bibtex
}
```

| [pose_hrnet_w32](/configs/body_2d_keypoint/topdown_heatmap/crowdpose/td-hm_hrnet-w32_8xb64-210e_crowdpose-256x192.py) |  256x192   | 0.675 |      0.825      |      0.729      | 0.770  | 0.687  | 0.553  | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_crowdpose_256x192-960be101_20201227.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_crowdpose_256x192_20201227.log.json) |

caffe-android-lib https://github.com/sh1r0/caffe-android-lib

mini-caffe https://github.com/luoyetx/mini-caffe

openblas-0.2.20 https://github.com/xianyi/OpenBLAS

ncnn https://github.com/Tencent/ncnn

***

squeezenet_v1.1 https://github.com/DeepScale/SqueezeNet/tree/master/SqueezeNet_v1.1

mobilenet_v1 https://github.com/shicai/MobileNet-Caffe

vgg16 https://gist.github.com/ksimonyan/211839e770f7b538e2d8

***

Host platform and compiler configuration: 

fedora 27, android-ndk-r15c, target arch = arm64-v8a

we manually update openblas package to version 0.2.20 in caffe-android-lib for better performance

***

Device: Nexus 6p

OS: LineageOS 15.1(Android 8.1.0), ROM newly flashed without any third-party APP installed

CPU: Snapdragon 810 (Cortex-A57 2.0GHz x 4 + Cortex-A53 1.55GHz x 4)

RAM: 3G

***

Benchmark method: 

Run squeezenet, mobilenet inference 23 times in a loop, discard the first three warmup records, and then calculate the average inference time

RAM: 3G

***

Run vgg169 times in a loop, discard the first warmup record, and then calculate the average inference time

Since the system may force SOC lowering its frequency when temperature goes high, sleep over 1 minute before each benchmark to prevent this issue.

fps performance: fps = 1000 / avgtime(ms)

cpu usage: take the CPU value in top utility output

memory usage: take the RES value in top utility output

the overall power consumption and performance per watt: 

Disable usb charging: adb shell echo 0 > /sys/class/power_supply/battery/charging_enabled

current(μA) = adb shell cat /sys/class/power_supply/battery/current_now (multiply -1 for 810 chip)

voltage(μV) = adb shell cat /sys/class/power_supply/battery/voltage_now

power consumption(mW) = current / 1000 * voltage / 1000 / 1000

performance per watt(1000fps/W) = fps / power consumption * 1000

***

The binary size after debug stripping

![](https://github.com/nihui/ncnn-assets/raw/master/20180413/1.jpg)

![](https://github.com/nihui/ncnn-assets/raw/master/20180413/2.jpg)

***

squeezenet

![](https://github.com/nihui/ncnn-assets/raw/master/20180413/3.jpg)

![](https://github.com/nihui/ncnn-assets/raw/master/20180413/4.jpg)

![](https://github.com/nihui/ncnn-assets/raw/master/20180413/5.jpg)

![](https://github.com/nihui/ncnn-assets/raw/master/20180413/6.jpg)

![](https://github.com/nihui/ncnn-assets/raw/master/20180413/7.jpg)

![](https://github.com/nihui/ncnn-assets/raw/master/20180413/8.jpg)
***

mobilenet

![](https://github.com/nihui/ncnn-assets/raw/master/20180413/9.jpg)

![](https://github.com/nihui/ncnn-assets/raw/master/20180413/10.jpg)

***

mobilenet

![](https://github.com/nihui/ncnn-assets/raw/master/20180413/11.jpg)

![](https://github.com/nihui/ncnn-assets/raw/master/20180413/12.jpg)

![](https://github.com/nihui/ncnn-assets/raw/master/20180413/13.jpg)

![](https://github.com/nihui/ncnn-assets/raw/master/20180413/14.jpg)
***

vgg16

![](https://github.com/nihui/ncnn-assets/raw/master/20180413/15.jpg)

![](https://github.com/nihui/ncnn-assets/raw/master/20180413/16.jpg)

![](https://github.com/nihui/ncnn-assets/raw/master/20180413/17.jpg)

![](https://github.com/nihui/ncnn-assets/raw/master/20180413/18.jpg)

![](https://github.com/nihui/ncnn-assets/raw/master/20180413/19.jpg)

![](https://github.com/nihui/ncnn-assets/raw/master/20180413/20.jpg)

# Changelog of v1.x

## v1.0.0rc1 (30/12/2022)

### Highlights

- Support [RTMDet](https://arxiv.org/abs/2212.07784) rotated object detection models. The technical report of RTMDet is on [arxiv](https://arxiv.org/abs/2212.07784) (#662)
- Support H2RBox. (#644)

### New Features

- Support PSC (#617)
- Add [`projects/`](./projects/) folder and give an [example](./projects/example_project/README.md) for communities to contribute their projects. (#627)
- Support DIOR Dataset. (#639)

### Bug Fixes

- Fix `get_flops.py` in 1.x. (#646)
- Fix Windows CI. (#621)
- Fix error in rbbox_overlaps. (#620)

### Improvements

- Deprecating old type alias due to new version of numpy (#674)
- Use iof in RRandomCrop. (#660)
- Modify keys in dataset.metainfo to lower case. (#654)
- Add torch 1.13 in CI. (#661)
- Add dockerfile in 1.x. (#631)
- Use mmengine in torchserve deployment. (#616)
- Add `.pre-commit-config-zh-cn.yaml`. (#630)

### New Contributors

- @yuyi1005 made their first contribution in <https://github.com/open-mmlab/mmrotate/pull/617>
- @yxzhao2022 made their first contribution in <https://github.com/open-mmlab/mmrotate/pull/639>
- @YanxingLiu made their first contribution in <https://github.com/open-mmlab/mmrotate/pull/631>

### Contributors

A total of 11 developers contributed to this release.

Thanks @yxzhao2022 @yuyi1005 @YanxingLiu @nijkah @RangeKing @austinmw @liuyanyi @yangxue0827 @zytx121 @RangiLyu @ZwwWayne

## v1.0.0rc0 (7/11/2022)

We are excited to announce the release of MMRotate 1.0.0rc0.
MMRotate 1.0.0rc0 is the first version of MMRotate 1.x, a part of the OpenMMLab 2.0 projects.

Built upon the new [training engine](https://github.com/open-mmlab/mmengine), MMRotate 1.x unifies the interfaces of dataset, models, evaluation, and visualization with faster training and testing speed.

1. **New engines**. MMRotate 1.x is based on [MMEngine](https://github.com/open-mmlab/mmengine), which provides a general and powerful runner that allows more flexible customizations and significantly simplifies the entrypoints of high-level interfaces.

2. **Unified interfaces**. As a part of the OpenMMLab 2.0 projects, MMRotate 1.x unifies and refactors the interfaces and internal logics of train, testing, datasets, models, evaluation, and visualization. All the OpenMMLab 2.0 projects share the same design in those interfaces and logics to allow the emergence of multi-task/modality algorithms.

3. **New BoxType design**. We support data structures RotatedBoxes and QuadriBoxes to encapsulate different kinds of bounding boxes. We are migrating to use data structures of boxes to replace the use of pure tensor boxes. This will unify the usages of different kinds of bounding boxes in MMDetection 3.x and MMRotate 1.x to simplify the implementation and reduce redundant codes.

4. **Stronger visualization**. We provide a series of useful tools which are mostly based on brand-new visualizers. As a result, it is more convenient for the users to explore the models and datasets now.

### Breaking Changes

We briefly list the major breaking changes here.
We will update the [migration guide](../migration.md) to provide complete details and migration instructions.

#### Dependencies

- MMRotate 1.x relies on MMEngine to run. MMEngine is a new foundational library for training deep learning models in OpenMMLab 2.0 models. The dependencies of file IO and training are migrated from MMCV 1.x to MMEngine.
- MMRotate 1.x relies on MMCV>=2.0.0rc2. Although MMCV no longer maintains the training functionalities since 2.0.0rc0, MMRotate 1.x relies on the data transforms, CUDA operators, and image processing interfaces in MMCV. Note that the package `mmcv` is the version that provide pre-built CUDA operators and `mmcv-lite` does not since MMCV 2.0.0rc0, while `mmcv-full` has been deprecated.
- MMRotate 1.x relies on MMDetection>=3.0.0rc2.

#### Training and testing

- MMRotate 1.x uses Runner in [MMEngine](https://github.com/open-mmlab/mmengine) rather than that in MMCV. The new Runner implements and unifies the building logic of dataset, model, evaluation, and visualizer. Therefore, MMRotate 1.x no longer maintains the building logics of those modules in `mmrotate.train.apis` and `tools/train.py`. Those code have been migrated into [MMEngine](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/runner.py). Please refer to the [migration guide of Runner in MMEngine](https://mmengine.readthedocs.io/en/latest/migration/runner.html) for more details.
- The Runner in MMEngine also supports testing and validation. The testing scripts are also simplified, which has similar logic as that in training scripts to build the runner.

- The execution points of hooks in the new Runner have been enriched to allow more flexible customization. Please refer to the [migration guide of Hook in MMEngine](https://mmengine.readthedocs.io/en/latest/migration/hook.html) for more details.
- Learning rate and momentum scheduling has been migrated from `Hook` to `Parameter Scheduler` in MMEngine. Please refer to the [migration guide of Parameter Scheduler in MMEngine](https://mmengine.readthedocs.io/en/latest/migration/param_scheduler.html) for more details.

#### Configs

- The [Runner in MMEngine](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/runner.py) uses a different config structures to ease the understanding of the components in runner. Users can refer to the [migration guide in MMEngine](https://mmengine.readthedocs.io/en/latest/migration/runner.html) for migration details.
- The file names of configs and models are also refactored to follow the new rules unified across OpenMMLab 2.0 projects.

#### Dataset

The Dataset classes implemented in MMRotate 1.x all inherits from the [BaseDataset in MMEngine](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/basedataset.html).

- All the datasets support to serialize the data list to reduce the memory when multiple workers are built to accelerate data loading.

#### Data Transforms

The data transforms in MMRotate 1.x all inherits from those in MMCV>=2.0.0rc2, which follows a new convention in OpenMMLab 2.0 projects.
The changes are listed as below:

- The interfaces are also changed. Please refer to the [API Reference](https://mmrotate.readthedocs.io/en/dev-1.x/)
- The functionality of some data transforms (e.g., `Rotate`) are decomposed into several transforms.

#### Model

The models in MMRotate 1.x all inherits from `BaseModel` in MMEngine, which defines a new convention of models in OpenMMLab 2.0 projects. Users can refer to the [tutorial of model](https://mmengine.readthedocs.io/en/latest/tutorials/model.html) in MMengine for more details. Accordingly, there are several changes as the following:

- The model interfaces, including the input and output formats, are significantly simplified and unified following the new convention in MMRotate 1.x. Specifically, all the input data in training and testing are packed into `inputs` and `data_samples`, where `inputs` contains model inputs like a list of image tensors, and `data_samples` contains other information of the current data sample such as ground truths and model predictions. In this way, different tasks in MMRotate 1.x can share the same input arguments, which makes the models more general and suitable for multi-task learning.
- The model has a data preprocessor module, which is used to pre-process the input data of model. In MMRotate 1.x, the data preprocessor usually does necessary steps to form the input images into a batch, such as padding. It can also serve as a place for some special data augmentations or more efficient data transformations like normalization.

- The internal logic of model have been changed. In MMRotate 0.x, model used `forward_train` and `simple_test` to deal with different model forward logics. In MMRotate 1.x and OpenMMLab 2.0, the forward function has three modes: `loss`, `predict`, and `tensor` for training, inference, and tracing or other purposes, respectively. The forward function calls `self.loss()`, `self.predict()`, and `self._forward()` given the modes `loss`, `predict`, and `tensor`, respectively.

#### Evaluation

MMRotate 1.x mainly implements corresponding metrics for each task, which are manipulated by [Evaluator](https://mmengine.readthedocs.io/en/latest/design/evaluator.html) to complete the evaluation.

In addition, users can build evaluator in MMRotate 1.x to conduct offline evaluation, i.e., evaluate predictions that may not produced by MMRotate, prediction follows our dataset conventions. More details can be find in the [Evaluation Tutorial](https://mmengine.readthedocs.io/en/latest/tutorials/evaluation.html) in MMEngine.

#### Visualization

The functions of visualization in MMRotate 1.x are removed. Instead, in OpenMMLab 2.0 projects, we use [Visualizer](https://mmengine.readthedocs.io/en/latest/design/visualization.html) to visualize data. MMRotate 1.x implements `RotLocalVisualizer` to allow visualization of ground truths, model predictions, and feature maps, etc., at any place. It also supports to dump the visualization data to any external visualization backends such as Tensorboard and Wandb.

- Support quadrilateral box detection (#520)
- Support RotatedCocoMetric (#557)
- Support COCO style annotations (#582)
- Support two new SAR datasets: RSDD and SRSDD (#591)

### Ongoing changes

1. Test-time augmentation: is not implemented yet in this version due to limited time slot. We will support it in the following releases with a new and simplified design.
2. Inference interfaces: a unified inference interfaces will be supported in the future to ease the use of released models.

3. Interfaces of useful tools that can be used in notebook: more useful tools that implemented in the `tools/` directory will have their python interfaces so that they can be used through notebook and in downstream libraries.
4. Documentation: we will add more design docs, tutorials, and migration guidance so that the community can deep dive into our new design, participate the future development, and smoothly migrate downstream libraries to MMRotate 1.x.

A total of 8 developers contributed to this release.
Thanks  @DonggeunYu @k-papadakis @liuyanyi  @yangxue0827 @jbwang1997 @zytx121  @RangiLyu @ZwwWayne

**Text Detection/Recognition/Spotting**

```json
{
  "imgs": {
    "OpenImages_ImageID_1": {
      "id": "OpenImages_ImageID_1",
      "width": "INT, Width of the image",
      "height": "INT, Height of the image",
      "set": "Split train|val|test",
      "filename": "train|test/OpenImages_ImageID_1.jpg"
    },
    "OpenImages_ImageID_2": {
      "...": "..."
    }
  },
  "anns": {
    "OpenImages_ImageID_1_1": {
      "id": "STR, OpenImages_ImageID_1_1, Specifies the nth annotation for an image",
      "image_id": "OpenImages_ImageID_1",
      "bbox": [
        "FLOAT x1",
        "FLOAT y1",
        "FLOAT x2",
        "FLOAT y2"
      ],
      "points": [
        "FLOAT y2",
        "...",
        "FLOAT xN",
        "FLOAT yN"
      ],
      "utf8_string": "text for this annotation",
      "area": "FLOAT, area of this box"
    },
    "OpenImages_ImageID_1_2": {
    },
    "OpenImages_ImageID_2_1": {

],
    },
    },
    }
  },
  "img2Anns": {
    "OpenImages_ImageID_1": [
      "OpenImages_ImageID_1_1",
      "OpenImages_ImageID_1_2",
      "OpenImages_ImageID_1_2"
    ],
    "OpenImages_ImageID_N": [
      "..."
    ]
  }
}
```

# LIIF (CVPR'2021)

> [Learning Continuous Image Representation with Local Implicit Image Function](https://arxiv.org/abs/2012.09161)

> **Task**: Image Super-Resolution

How to represent an image? While the visual world is presented in a continuous manner, machines store and see the images in a discrete way with 2D arrays of pixels. In this paper, we seek to learn a continuous representation for images. Inspired by the recent progress in 3D reconstruction with implicit neural representation, we propose Local Implicit Image Function (LIIF), which takes an image coordinate and the 2D deep features around the coordinate as inputs, predicts the RGB value at a given coordinate as an output. Since the coordinates are continuous, LIIF can be presented in arbitrary resolution. To generate the continuous representation for images, we train an encoder with LIIF representation via a self-supervised task with super-resolution. The learned continuous representation can be presented in arbitrary resolution even extrapolate to x30 higher resolution, where the training tasks are not provided. We further show that LIIF representation builds a bridge between discrete and continuous representation in 2D, it naturally supports the learning tasks with size-varied image ground-truths and significantly outperforms the method with resizing the ground-truths.

 <img src="https://user-images.githubusercontent.com/7676947/144032669-da59d683-9c4f-4598-a680-32770a369b74.png" width="400"/>
</div >

|                                    Model                                     | Dataset | scale |  PSNR   |  SSIM   | Training Resources |                                     Download                                     |
| :--------------------------------------------------------------------------: | :-----: | :---: | :-----: | :-----: | :----------------: | :------------------------------------------------------------------------------: |

| [liif_edsr_norm_c64b16_g1_1000k_div2k](./liif-edsr-norm_c64b16_1xb16-1000k_div2k.py) |  Set5   |  x2   | 35.7131 | 0.9366  |    1 (TITAN Xp)    | [model](https://download.openmmlab.com/mmediting/restorers/liif/liif_edsr_norm_c64b16_g1_1000k_div2k_20210715-ab7ce3fc.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/liif/liif_edsr_norm_c64b16_g1_1000k_div2k_20210715-ab7ce3fc.log.json) |
| [liif_edsr_norm_c64b16_g1_1000k_div2k](./liif-edsr-norm_c64b16_1xb16-1000k_div2k.py) |  Set14  |  x2   | 31.5579 | 0.8889  |         △          |                                        △                                         |
| [liif_edsr_norm_c64b16_g1_1000k_div2k](./liif-edsr-norm_c64b16_1xb16-1000k_div2k.py) |  DIV2K  |  x2   | 34.6647 | 0.9355  |         △          |                                        △                                         |

| [liif_edsr_norm_c64b16_g1_1000k_div2k](./liif-edsr-norm_c64b16_1xb16-1000k_div2k.py) |  Set5   |  x3   | 32.3805 | 0.8915  |         △          |                                        △                                         |
| [liif_edsr_norm_c64b16_g1_1000k_div2k](./liif-edsr-norm_c64b16_1xb16-1000k_div2k.py) |  Set14  |  x3   | 0.8039  | 30.9808 |         △          |                                        △                                         |
| [liif_edsr_norm_c64b16_g1_1000k_div2k](./liif-edsr-norm_c64b16_1xb16-1000k_div2k.py) |  DIV2K  |  x3   | 28.4605 | 0.8724  |         △          |                                        △                                         |
| [liif_edsr_norm_c64b16_g1_1000k_div2k](./liif-edsr-norm_c64b16_1xb16-1000k_div2k.py) |  Set5   |  x4   | 30.2748 | 0.8509  |         △          |                                        △                                         |

| [liif_edsr_norm_c64b16_g1_1000k_div2k](./liif-edsr-norm_c64b16_1xb16-1000k_div2k.py) |  Set14  |  x4   | 26.8415 | 0.7381  |         △          |                                        △                                         |
| [liif_edsr_norm_c64b16_g1_1000k_div2k](./liif-edsr-norm_c64b16_1xb16-1000k_div2k.py) |  DIV2K  |  x4   | 29.0245 | 0.8187  |         △          |                                        △                                         |
| [liif_edsr_norm_c64b16_g1_1000k_div2k](./liif-edsr-norm_c64b16_1xb16-1000k_div2k.py) |  Set5   |  x6   | 27.1187 | 0.7774  |         △          |                                        △                                         |
| [liif_edsr_norm_c64b16_g1_1000k_div2k](./liif-edsr-norm_c64b16_1xb16-1000k_div2k.py) |  Set14  |  x6   | 24.7461 | 0.6444  |         △          |                                        △                                         |

| [liif_edsr_norm_c64b16_g1_1000k_div2k](./liif-edsr-norm_c64b16_1xb16-1000k_div2k.py) |  DIV2K  |  x6   | 26.7770 | 0.7425  |         △          |                                        △                                         |
| [liif_edsr_norm_c64b16_g1_1000k_div2k](./liif-edsr-norm_c64b16_1xb16-1000k_div2k.py) |  Set5   |  x18  | 20.8516 | 0.5406  |         △          |                                        △                                         |
| [liif_edsr_norm_c64b16_g1_1000k_div2k](./liif-edsr-norm_c64b16_1xb16-1000k_div2k.py) |  Set14  |  x18  | 20.0096 | 0.4525  |         △          |                                        △                                         |
| [liif_edsr_norm_c64b16_g1_1000k_div2k](./liif-edsr-norm_c64b16_1xb16-1000k_div2k.py) |  DIV2K  |  x18  | 22.1987 | 0.5955  |         △          |                                        △                                         |

| [liif_edsr_norm_c64b16_g1_1000k_div2k](./liif-edsr-norm_c64b16_1xb16-1000k_div2k.py) |  Set5   |  x30  | 18.8467 | 0.5010  |         △          |                                        △                                         |
| [liif_edsr_norm_c64b16_g1_1000k_div2k](./liif-edsr-norm_c64b16_1xb16-1000k_div2k.py) |  Set14  |  x30  | 18.1321 | 0.3963  |         △          |                                        △                                         |
| [liif_edsr_norm_c64b16_g1_1000k_div2k](./liif-edsr-norm_c64b16_1xb16-1000k_div2k.py) |  DIV2K  |  x30  | 20.5050 | 0.5577  |         △          |                                        △                                         |

| [liif_rdn_norm_c64b16_g1_1000k_div2k](./liif-rdn-norm_c64b16_1xb16-1000k_div2k.py) |  Set5   |  x2   | 35.7874 | 0.9366  |    1 (TITAN Xp)    | [model](https://download.openmmlab.com/mmediting/restorers/liif/liif_rdn_norm_c64b16_g1_1000k_div2k_20210717-22d6fdc8.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/liif/liif_rdn_norm_c64b16_g1_1000k_div2k_20210717-22d6fdc8.log.json) |
| [liif_rdn_norm_c64b16_g1_1000k_div2k](./liif-rdn-norm_c64b16_1xb16-1000k_div2k.py) |  Set14  |  x2   | 31.6866 | 0.8896  |         △          |                                        △                                         |
| [liif_rdn_norm_c64b16_g1_1000k_div2k](./liif-rdn-norm_c64b16_1xb16-1000k_div2k.py) |  DIV2K  |  x2   | 34.7548 | 0.9356  |         △          |                                        △                                         |

| [liif_rdn_norm_c64b16_g1_1000k_div2k](./liif-rdn-norm_c64b16_1xb16-1000k_div2k.py) |  Set5   |  x3   | 32.4992 | 0.8923  |         △          |                                        △                                         |
| [liif_rdn_norm_c64b16_g1_1000k_div2k](./liif-rdn-norm_c64b16_1xb16-1000k_div2k.py) |  Set14  |  x3   | 28.4905 | 0.8037  |         △          |                                        △                                         |
| [liif_rdn_norm_c64b16_g1_1000k_div2k](./liif-rdn-norm_c64b16_1xb16-1000k_div2k.py) |  DIV2K  |  x3   | 31.0744 | 0.8731  |         △          |                                        △                                         |
| [liif_rdn_norm_c64b16_g1_1000k_div2k](./liif-rdn-norm_c64b16_1xb16-1000k_div2k.py) |  Set5   |  x4   | 30.3835 | 0.8513  |         △          |                                        △                                         |

| [liif_rdn_norm_c64b16_g1_1000k_div2k](./liif-rdn-norm_c64b16_1xb16-1000k_div2k.py) |  Set14  |  x4   | 26.8734 | 0.7373  |         △          |                                        △                                         |
| [liif_rdn_norm_c64b16_g1_1000k_div2k](./liif-rdn-norm_c64b16_1xb16-1000k_div2k.py) |  DIV2K  |  x4   | 29.1101 | 0.8197  |         △          |                                        △                                         |
| [liif_rdn_norm_c64b16_g1_1000k_div2k](./liif-rdn-norm_c64b16_1xb16-1000k_div2k.py) |  Set5   |  x6   | 27.1914 | 0.7751  |         △          |                                        △                                         |
| [liif_rdn_norm_c64b16_g1_1000k_div2k](./liif-rdn-norm_c64b16_1xb16-1000k_div2k.py) |  Set14  |  x6   | 24.7824 | 0.6434  |         △          |                                        △                                         |

| [liif_rdn_norm_c64b16_g1_1000k_div2k](./liif-rdn-norm_c64b16_1xb16-1000k_div2k.py) |  DIV2K  |  x6   | 26.8693 | 0.7437  |         △          |                                        △                                         |
| [liif_rdn_norm_c64b16_g1_1000k_div2k](./liif-rdn-norm_c64b16_1xb16-1000k_div2k.py) |  Set5   |  x18  | 20.8913 | 0.5329  |         △          |                                        △                                         |
| [liif_rdn_norm_c64b16_g1_1000k_div2k](./liif-rdn-norm_c64b16_1xb16-1000k_div2k.py) |  Set14  |  x18  | 20.1077 | 0.4537  |         △          |                                        △                                         |
| [liif_rdn_norm_c64b16_g1_1000k_div2k](./liif-rdn-norm_c64b16_1xb16-1000k_div2k.py) |  DIV2K  |  x18  | 22.2972 | 0.5950  |         △          |                                        △                                         |

| [liif_rdn_norm_c64b16_g1_1000k_div2k](./liif-rdn-norm_c64b16_1xb16-1000k_div2k.py) |  Set5   |  x30  | 18.9354 | 0.4864  |         △          |                                        △                                         |
| [liif_rdn_norm_c64b16_g1_1000k_div2k](./liif-rdn-norm_c64b16_1xb16-1000k_div2k.py) |  Set14  |  x30  | 18.1448 | 0.3942  |         △          |                                        △                                         |
| [liif_rdn_norm_c64b16_g1_1000k_div2k](./liif-rdn-norm_c64b16_1xb16-1000k_div2k.py) |  DIV2K  |  x30  | 20.5663 | 0.5560  |         △          |                                        △                                         |

Note:

- △ refers to ditto.
- Evaluated on RGB channels,  `scale` pixels in each border are cropped before evaluation.

**Train**

<details>

```shell
# cpu train

**Train**

<details>

```shell
# cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/liif/liif-edsr-norm_c64b16_1xb16-1000k_div2k.py

python tools/train.py configs/liif/liif-edsr-norm_c64b16_1xb16-1000k_div2k.py

./tools/dist_train.sh configs/liif/liif-edsr-norm_c64b16_1xb16-1000k_div2k.py 8
```

**Test**

<details>

```shell
# cpu test

**Test**

<details>

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/liif/liif-edsr-norm_c64b16_1xb16-1000k_div2k.py https://download.openmmlab.com/mmediting/restorers/liif/liif_edsr_norm_c64b16_g1_1000k_div2k_20210715-ab7ce3fc.pth

python tools/test.py configs/liif/liif-edsr-norm_c64b16_1xb16-1000k_div2k.py https://download.openmmlab.com/mmediting/restorers/liif/liif_edsr_norm_c64b16_g1_1000k_div2k_20210715-ab7ce3fc.pth

./tools/dist_test.sh configs/liif/liif-edsr-norm_c64b16_1xb16-1000k_div2k.py https://download.openmmlab.com/mmediting/restorers/liif/liif_edsr_norm_c64b16_g1_1000k_div2k_20210715-ab7ce3fc.pth 8
```

```

```bibtex
@inproceedings{chen2021learning,
  title={Learning continuous image representation with local implicit image function},
  author={Chen, Yinbo and Liu, Sifei and Wang, Xiaolong},
  pages={8628--8638},
}
```

# Cross compile snpe inference server on Ubuntu 18

mmdeploy has provided a prebuilt package, if you want to compile it by self, or need to modify the `.proto` file, you can refer to this document.

Note that the official gRPC documentation does not have complete support for the NDK.

## 1. Environment

| Item               | Version        | Remarks                                                   |
| ------------------ | -------------- | --------------------------------------------------------- |
| snpe               | 1.59           | 1.60 uses clang-8.0, which may cause compatibility issues |
| host OS            | ubuntu18.04    | snpe1.59 specified version                                |
| NDK                | r17c           | snpe1.59 specified version                                |
| gRPC               | commit 6f698b5 | -                                                         |

| Hardware equipment | qcom888        | qcom chip required                                        |

## 2. Cross compile gRPC with NDK

1. Pull gRPC repo, compile `protoc` and `grpc_cpp_plugin` on host

```bash
# Install dependencies
$ apt-get update && apt-get install -y libssl-dev
# Compile
$ git clone https://github.com/grpc/grpc --recursive=1 --depth=1
$ mkdir -p cmake/build
$ pushd cmake/build

$ cmake \
  -DCMAKE_BUILD_TYPE=Release \
  -DgRPC_INSTALL=ON \
  -DgRPC_BUILD_TESTS=OFF \
  -DgRPC_SSL_PROVIDER=package \
  ../..
# Install to host
$ make -j
$ sudo make install
```

2. Download the NDK and cross-compile the static libraries with android aarch64 format

```bash
$ wget https://dl.google.com/android/repository/android-ndk-r17c-linux-x86_64.zip

```

```bash
$ unzip android-ndk-r17c-linux-x86_64.zip

$ export ANDROID_NDK=/path/to/android-ndk-r17c

$ cd /path/to/grpc
$ mkdir -p cmake/build_aarch64  && pushd cmake/build_aarch64

$ cmake ../.. \
 -DCMAKE_TOOLCHAIN_FILE=${ANDROID_NDK}/build/cmake/android.toolchain.cmake \
 -DANDROID_ABI=arm64-v8a \
 -DANDROID_PLATFORM=android-26 \
 -DANDROID_TOOLCHAIN=clang \
 -DANDROID_STL=c++_shared \
 -DCMAKE_BUILD_TYPE=Release \
 -DCMAKE_INSTALL_PREFIX=/tmp/android_grpc_install_shared

$ make -j
$ make install
```

3. At this point `/tmp/android_grpc_install` should have the complete installation file

```bash
$ cd /tmp/android_grpc_install
$ tree -L 1
.
├── bin
├── include
├── lib
└── share
```

## 3. (Skipable) Self-test whether NDK gRPC is available

1. Compile the helloworld that comes with gRPC

```bash

$ tree -L 1
.
├── bin
├── lib
└── share
```

```bash
$ cd /path/to/grpc/examples/cpp/helloworld/
$ mkdir cmake/build_aarch64 -p && pushd cmake/build_aarch64

 -Dabsl_DIR=/tmp/android_grpc_install_shared/lib/cmake/absl \
 -DProtobuf_DIR=/tmp/android_grpc_install_shared/lib/cmake/protobuf \
 -DgRPC_DIR=/tmp/android_grpc_install_shared/lib/cmake/grpc

$ make -j
$ ls greeter*
greeter_async_client   greeter_async_server     greeter_callback_server  greeter_server
greeter_async_client2  greeter_callback_client  greeter_client
```

2. Turn on debug mode on your phone, push the binary to `/data/local/tmp`

```bash

```

```bash
$ adb push greeter* /data/local/tmp
```

3. `adb shell` into the phone, execute client/server

```bash
/data/local/tmp $ ./greeter_client
Greeter received: Hello world
```

## 4. Cross compile snpe inference server

1. Open the [snpe tools website](https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk/tools) and download version 1.59. Unzip and set environment variables

> Note that snpe >= 1.60 starts using `clang-8.0`, which may cause incompatibility with `libc++_shared.so` on older devices.

```bash
$ export SNPE_ROOT=/path/to/snpe-1.59.0.3230
```

2. Open the snpe server directory within mmdeploy, use the options when cross-compiling gRPC

```bash
$ cd /path/to/mmdeploy
$ cd service/snpe/server

$ mkdir -p build && cd build
$ cmake .. \

```bash

$ cmake .. \

 $ make -j
 $ file inference_server
inference_server: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /system/bin/linker64, BuildID[sha1]=252aa04e2b982681603dacb74b571be2851176d2, with debug_info, not stripped
```

Finally,  you can see `infernece_server`, `adb push` it to the device and execute.

## 5. Regenerate the proto interface

```

If you have changed `inference.proto`, you need to regenerate the .cpp and .py interfaces

```Shell
$ python3 -m pip install grpc_tools --user
$ python3 -m  grpc_tools.protoc -I./ --python_out=./client/ --grpc_python_out=./client/ inference.proto

$ ln -s `which protoc-gen-grpc`
$ protoc --cpp_out=./ --grpc_out=./  --plugin=protoc-gen-grpc=grpc_cpp_plugin  inference.proto
```

## Reference

- snpe tutorial https://developer.qualcomm.com/sites/default/files/docs/snpe/cplus_plus_tutorial.html
- gRPC cross build script https://raw.githubusercontent.com/grpc/grpc/master/test/distrib/cpp/run_distrib_test_cmake_aarch64_cross.sh
- stackoverflow https://stackoverflow.com/questions/54052229/build-grpc-c-for-android-using-ndk-arm-linux-androideabi-clang-compiler

# Benchmark and Model Zoo

## Common settings

- We use distributed training with 4 GPUs by default.

- All pytorch-style pretrained backbones on ImageNet are train by ourselves, with the same procedure in the [paper](https://arxiv.org/pdf/1812.01187.pdf).
  Our ResNet style backbone are based on ResNetV1c variant, where the 7x7 conv in the input stem is replaced with three 3x3 convs.

- For the consistency across different hardwares, we report the GPU memory as the maximum value of `torch.cuda.max_memory_allocated()` for all 4 GPUs with `torch.backends.cudnn.benchmark=False`.
  Note that this value is usually less than what `nvidia-smi` shows.

- We report the inference time as the total time of network forwarding and post-processing, excluding the data loading time.
  Results are obtained with the script `tools/benchmark.py` which computes the average time on 200 images with `torch.backends.cudnn.benchmark=False`.

- There are two inference modes in this framework.

  - `slide` mode: The `test_cfg` will be like `dict(mode='slide', crop_size=(769, 769), stride=(513, 513))`.

    In this mode, multiple patches will be cropped from input image, passed into network individually.
    The crop size and stride between patches are specified by `crop_size` and `stride`.
    The overlapping area will be merged by average

  - `whole` mode: The `test_cfg` will be like `dict(mode='whole')`.

    In this mode, the whole imaged will be passed into network directly.

    By default, we use `slide` inference for 769x769 trained model, `whole` inference for the rest.

- For input size of 8x+1 (e.g. 769), `align_corner=True` is adopted as a traditional practice.
  Otherwise, for input size of 8x (e.g. 512, 1024), `align_corner=False` is adopted.

## Baselines

### FCN

Please refer to [FCN](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn) for details.

### PSPNet

### FCN

### PSPNet

Please refer to [PSPNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet) for details.

### DeepLabV3

Please refer to [DeepLabV3](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3) for details.

### PSANet

Please refer to [PSANet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/psanet) for details.

### DeepLabV3+

Please refer to [DeepLabV3+](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus) for details.

### UPerNet

Please refer to [UPerNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/upernet) for details.

### NonLocal Net

Please refer to [NonLocal Net](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/nonlocal_net) for details.

### EncNet

### EncNet

Please refer to [EncNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/encnet) for details.

### CCNet

Please refer to [CCNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ccnet) for details.

### DANet

Please refer to [DANet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/danet) for details.

### APCNet

Please refer to [APCNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/apcnet) for details.

### HRNet

Please refer to [HRNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet) for details.

### GCNet

Please refer to [GCNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/gcnet) for details.

### DMNet

Please refer to [DMNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dmnet) for details.

### ANN

### DMNet

### ANN

Please refer to [ANN](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ann) for details.

### OCRNet

Please refer to [OCRNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet) for details.

### Fast-SCNN

Please refer to [Fast-SCNN](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fastscnn) for details.

### ResNeSt

Please refer to [ResNeSt](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/resnest) for details.

### Semantic FPN

Please refer to [Semantic FPN](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/sem_fpn) for details.

### PointRend

Please refer to [PointRend](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/point_rend) for details.

### MobileNetV2

Please refer to [MobileNetV2](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mobilenet_v2) for details.

### MobileNetV3

Please refer to [MobileNetV3](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mobilenet_v3) for details.

### EMANet

Please refer to [EMANet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/emanet) for details.

### DNLNet

Please refer to [DNLNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dnlnet) for details.

### CGNet

Please refer to [CGNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/cgnet) for details.

### Mixed Precision (FP16) Training

### CGNet

Please refer [Mixed Precision (FP16) Training on BiSeNetV2](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/bisenetv2/bisenetv2_fcn_4xb4-160k_cityscapes-1024x1024.py) for details.

### U-Net

Please refer to [U-Net](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/README.md) for details.

### ViT

Please refer to [ViT](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/vit/README.md) for details.

### Swin

Please refer to [Swin](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/swin/README.md) for details.

### SETR

Please refer to [SETR](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/setr/README.md) for details.

## Speed benchmark

### Hardware

- 8 NVIDIA Tesla V100 (32G) GPUs
- Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz

### Software environment

- Python 3.7
- PyTorch 1.5
- CUDA 10.1
- CUDNN 7.6.03
- NCCL 2.4.08

### Training speed

For fair comparison, we benchmark all implementations with ResNet-101V1c.
The input size is fixed to 1024x512 with batch size 2.

The training speed is reported as followed, in terms of second per iter (s/iter). The lower, the better.

| Implementation                                                              | PSPNet (s/iter) | DeepLabV3+ (s/iter) |
| --------------------------------------------------------------------------- | --------------- | ------------------- |
| [MMSegmentation](https://github.com/open-mmlab/mmsegmentation)              | **0.83**        | **0.85**            |
| [SegmenTron](https://github.com/LikeLy-Journey/SegmenTron)                  | 0.84            | 0.85                |

| [CASILVision](https://github.com/CSAILVision/semantic-segmentation-pytorch) | 1.15            | N/A                 |
| [vedaseg](https://github.com/Media-Smart/vedaseg)                           | 0.95            | 1.25                |

:::{note}
The output stride of DeepLabV3+ is 8.
:::

# Tutorial 1: Learn about Configs

We incorporate modular and inheritance design into our config system, which is convenient to conduct various experiments.
You may also pass `--cfg-options xxx.yyy=zzz` to see updated config.

## Config File Structure

There are 4 basic component types under `config/_base_`, datasets, models, schedules, default_runtime.
Many methods could be easily constructed with one of each like DeepLabV3, PSPNet.
The configs that are composed by components from `_base_` are called _primitive_.

For example, if some modification is made base on DeepLabV3, user may first inherit the basic DeepLabV3 structure by specifying `_base_ = ../deeplabv3/deeplabv3_r50-d8_4xb2-40k_cityscapes-512x1024.py`, then modify the necessary fields in the config files.

If you are building an entirely new method that does not share the structure with any of the existing methods, you may create a folder `xxxnet` under `configs`,

Please refer to [mmengine](https://mmengine.readthedocs.io/en/latest/tutorials/config.html) for detailed documentation.

## Config Name Style

```text
{algorithm name}_{model component names [component1]_[component2]_[...]}_{training settings}_{training dataset information}_{testing dataset information}
```

```text
```

The file name is divided to five parts. All parts and components are connected with `_` and words of each part or component should be connected with `-`.

- `{algorithm name}`: The name of the algorithm, such as `deeplabv3`, `pspnet`, etc.
- `{model component names}`: Names of the components used in the algorithm such as backbone, head, etc. For example, `r50-d8` means using ResNet50 backbone and use output of backbone is 8 times downsampling as input.
- `{training settings}`: Information of training settings such as batch size, augmentations, loss, learning rate scheduler, and epochs/iterations. For example: `4xb4-ce-linearlr-40K` means using 4-gpus x 4-images-per-gpu, CrossEntropy loss, Linear learning rate scheduler, and train 40K iterations.

  - `{gpu x batch_per_gpu}`: GPUs and samples per GPU. `bN` indicates N batch size per GPU. E.g. `8xb2` is the short term of 8-gpus x 2-images-per-gpu. And `4xb4` is used by default if not mentioned.
  - `{schedule}`: training schedule, options are `20k`, `40k`, etc. `20k` and `40k` means 20000 iterations and 40000 iterations respectively.
- `{training dataset information}`: Training dataset names like `cityscapes`, `ade20k`, etc, and input resolutions. For example: `cityscapes-768x768` means training on `cityscapes` dataset and the input shape is `768x768`.

## An Example of PSPNet

To help the users have a basic idea of a complete config and the modules in a modern semantic segmentation system,

we make brief comments on the config of PSPNet using ResNet50V1c as the following.
For more detailed usage and the corresponding alternative for each module, please refer to the API documentation.

```python
_base_ = [
    '../_base_/models/pspnet_r50-d8.py', '../_base_/datasets/cityscapes.py',
    '../_base_/default_runtime.py', '../_base_/schedules/schedule_40k.py'
] # base config file which we build new config file on.
crop_size = (512, 1024)
data_preprocessor = dict(size=crop_size)
model = dict(data_preprocessor=data_preprocessor)
```

`_base_/models/pspnet_r50-d8.py` is a basic model cfg file for PSPNet using ResNet50V1c

```python
norm_cfg = dict(type='SyncBN', requires_grad=True)  # Segmentation usually uses SyncBN

```python
data_preprocessor = dict(  # The config of data preprocessor, usually includes image normalization and augmentation.
    type='SegDataPreProcessor',  # The type of data preprocessor.
    mean=[123.675, 116.28, 103.53],  # Mean values used for normalizing the input images.
    std=[58.395, 57.12, 57.375],  # Standard variance used for normalizing the input images.
    bgr_to_rgb=True,  # Whether to convert image from BGR to RGB.
    pad_val=0,  # Padding value of image.
    seg_pad_val=255)  # Padding value of segmentation map.
    type='EncoderDecoder',  # Name of segmentor
    data_preprocessor=data_preprocessor,
    pretrained='open-mmlab://resnet50_v1c',  # The ImageNet pretrained backbone to be loaded
        type='ResNetV1c',  # The type of backbone. Please refer to mmseg/models/backbones/resnet.py for details.

backbone=dict(
        depth=50,  # Depth of backbone. Normally 50, 101 are used.
        num_stages=4,  # Number of stages of backbone.
        out_indices=(0, 1, 2, 3),  # The index of output feature maps produced in each stages.
        dilations=(1, 1, 2, 4),  # The dilation rate of each layer.
        strides=(1, 2, 1, 1),  # The stride of each layer.
        norm_cfg=norm_cfg,  # The configuration of norm layer.
        norm_eval=False,  # Whether to freeze the statistics in BN
        style='pytorch',  # The style of backbone, 'pytorch' means that stride 2 layers are in 3x3 conv, 'caffe' means stride 2 layers are in 1x1 convs.
        contract_dilation=True),  # When dilation > 1, whether contract first layer of dilation.
        type='PSPHead',  # Type of decode head. Please refer to mmseg/models/decode_heads for available options.

decode_head=dict(
        in_channels=2048,  # Input channel of decode head.
        in_index=3,  # The index of feature map to select.
        channels=512,  # The intermediate channels of decode head.
        pool_scales=(1, 2, 3, 6),  # The avg pooling scales of PSPHead. Please refer to paper for details.
        dropout_ratio=0.1,  # The dropout ratio before final classification layer.
        num_classes=19,  # Number of segmentation class. Usually 19 for cityscapes, 21 for VOC, 150 for ADE20k.
        align_corners=False,  # The align_corners argument for resize in decoding.
        loss_decode=dict(  # Config of loss function for the decode_head.
            type='CrossEntropyLoss',  # Type of loss used for segmentation.
            use_sigmoid=False,  # Whether use sigmoid activation for segmentation.

type='CrossEntropyLoss',  # Type of loss used for segmentation.
            loss_weight=1.0)),  # Loss weight of decode_head.
    auxiliary_head=dict(
        type='FCNHead',  # Type of auxiliary head. Please refer to mmseg/models/decode_heads for available options.
        in_channels=1024,  # Input channel of auxiliary head.
        in_index=2,  # The index of feature map to select.
        channels=256,  # The intermediate channels of decode head.
        num_convs=1,  # Number of convs in FCNHead. It is usually 1 in auxiliary head.
        concat_input=False,  # Whether concat output of convs with input before classification layer.

num_classes=19,  # Number of segmentation class. Usually 19 for cityscapes, 21 for VOC, 150 for ADE20k.
        loss_decode=dict(  # Config of loss function for the auxiliary_head.
            loss_weight=0.4)),  # Loss weight of auxiliary_head.
    # model training and testing settings
    train_cfg=dict(),  # train_cfg is just a place holder for now.
    test_cfg=dict(mode='whole'))  # The test mode, options are 'whole' and 'slide'. 'whole': whole image fully-convolutional test. 'slide': sliding crop window on the image.
```

`_base_/datasets/cityscapes.py` is the configuration file of the dataset

```python
# dataset settings

```

```python
dataset_type = 'CityscapesDataset'  # Dataset type, this will be used to define the dataset.
data_root = 'data/cityscapes/'  # Root path of data.
crop_size = (512, 1024)  # The crop size during training.
train_pipeline = [  # Training pipeline.
    dict(type='LoadImageFromFile'),  # First pipeline to load images from file path.
    dict(type='LoadAnnotations'),  # Second pipeline to load annotations for current image.
    dict(type='RandomResize',  # Augmentation pipeline that resize the images and their annotations.
        scale=(2048, 1024),  # The scale of image.
        ratio_range=(0.5, 2.0),  # The augmented scale range as ratio.
        keep_ratio=True),  # Whether to keep the aspect ratio when resizing the image.
    dict(type='RandomCrop',  # Augmentation pipeline that randomly crop a patch from current image.
        crop_size=crop_size,  # The crop size of patch.

dict(type='RandomCrop',  # Augmentation pipeline that randomly crop a patch from current image.
        cat_max_ratio=0.75),  # The max area ratio that could be occupied by single category.
    dict(type='RandomFlip',  # Augmentation pipeline that flip the images and their annotations
        prob=0.5),  # The ratio or probability to flip
    dict(type='PhotoMetricDistortion'),  # Augmentation pipeline that distort current image with several photo metric methods.
    dict(type='PackSegInputs')  # Pack the inputs data for the semantic segmentation.
]
    dict(type='LoadImageFromFile'),  # First pipeline to load images from file path
    dict(type='Resize',  # Use resize augmentation
        scale=(2048, 1024),  # Images scales for resizing.
    # add loading annotation after ``Resize`` because ground truth

keep_ratio=True),  # Whether to keep the aspect ratio when resizing the image.
    # does not need to do resize data transform
    dict(type='LoadAnnotations'),  # Load annotations for semantic segmentation provided by dataset.
]
train_dataloader = dict(  # Train dataloader config
    persistent_workers=True,  # Shut down the worker processes after an epoch end, which can accelerate training speed.
    sampler=dict(type='InfiniteSampler', shuffle=True),  # Randomly shuffle during training.
        type=dataset_type,  # Type of dataset, refer to mmseg/datasets/ for details.
        data_root=data_root,  # The root of dataset.
        data_prefix=dict(

type=dataset_type,  # Type of dataset, refer to mmseg/datasets/ for details.
            img_path='leftImg8bit/train', seg_map_path='gtFine/train'),  # Prefix for training data.
        pipeline=train_pipeline)) # Processing pipeline. This is passed by the train_pipeline created before.
    batch_size=1,  # Batch size of a single GPU
    num_workers=4,  # Worker to pre-fetch data for each single GPU
    persistent_workers=True,  # Shut down the worker processes after an epoch end, which can accelerate testing speed.
    sampler=dict(type='DefaultSampler', shuffle=False),  # Not shuffle during validation and testing.
    dataset=dict(  # Test dataset config

            img_path='leftImg8bit/val', seg_map_path='gtFine/val'),  # Prefix for testing data.
        pipeline=test_pipeline))  # Processing pipeline. This is passed by the test_pipeline created before.
# The metric to measure the accuracy. Here, we use IoUMetric.
val_evaluator = dict(type='IoUMetric', iou_metrics=['mIoU'])
```

`_base_/schedules/schedule_40k.py`

```python
# optimizer
optimizer = dict(type='SGD', # Type of optimizers, refer to https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/optimizer/default_constructor.py for more details
                lr=0.01,  # Learning rate of optimizers, see detail usages of the parameters in the documentation of PyTorch
                momentum=0.9,  # Momentum

lr=0.01,  # Learning rate of optimizers, see detail usages of the parameters in the documentation of PyTorch
                weight_decay=0.0005)  # Weight decay of SGD
optim_wrapper = dict(type='OptimWrapper',  # Optimizer wrapper provides a common interface for updating parameters.
                    optimizer=optimizer,  # Optimizer used to update model parameters.
                    clip_grad=None)  # If ``clip_grad`` is not None, it will be the arguments of ``torch.nn.utils.clip_grad``.
    dict(
        type='PolyLR',  # The policy of scheduler, also support Step, CosineAnnealing, Cyclic, etc. Refer to details of supported LrUpdater from https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py
        eta_min=1e-4,  # Minimum learning rate at the end of scheduling.
        power=0.9,  # The power of polynomial decay.

eta_min=1e-4,  # Minimum learning rate at the end of scheduling.
        begin=0,  # Step at which to start updating the parameters.
        end=40000,  # Step at which to stop updating the parameters.
        by_epoch=False)  # Whether count by epoch or not.
]
# training schedule for 40k iteration
train_cfg = dict(type='IterBasedTrainLoop', max_iters=40000, val_interval=4000)
test_cfg = dict(type='TestLoop')
# default hooks
    timer=dict(type='IterTimerHook'),  # Log the time spent during iteration.
    logger=dict(type='LoggerHook', interval=50, log_metric_by_epoch=False),  # Collect and write logs from different components of ``Runner``.
    param_scheduler=dict(type='ParamSchedulerHook'),  # update some hyper-parameters in optimizer, e.g., learning rate.
    checkpoint=dict(type='CheckpointHook', by_epoch=False, interval=4000),  # Save checkpoints periodically.

checkpoint=dict(type='CheckpointHook', by_epoch=False, interval=4000),  # Save checkpoints periodically.
    sampler_seed=dict(type='DistSamplerSeedHook'))  # Data-loading sampler for distributed training.
```

in `_base_/default_runtime.py`

```python
# Set the default scope of the registry to mmseg.
default_scope = 'mmseg'
# environment
    cudnn_benchmark=True,
    mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),
    dist_cfg=dict(backend='nccl'),
)
load_from = None  # Load checkpoint from file.
resume = False  # Whether to resume from existed model.
```

These are all the configs for training and testing PSPNet, to load and parse them, we can use [Config](https://mmengine.readthedocs.io/en/latest/tutorials/config.html) implemented in [MMEngine](https://github.com/open-mmlab/mmengine)

```python
from mmengine.config import Config

```python

cfg = Config.fromfile('configs/pspnet/pspnet_r50-d8_4xb2-40k_cityscapes-512x1024.py')
print(cfg.train_dataloader)
```

```shell
{'batch_size': 2,
 'num_workers': 2,
 'persistent_workers': True,
 'sampler': {'type': 'InfiniteSampler', 'shuffle': True},
 'dataset': {'type': 'CityscapesDataset',
  'data_root': 'data/cityscapes/',
  'data_prefix': {'img_path': 'leftImg8bit/train',
   'seg_map_path': 'gtFine/train'},
  'pipeline': [{'type': 'LoadImageFromFile'},
   {'type': 'LoadAnnotations'},
   {'type': 'RandomResize',
    'scale': (2048, 1024),
    'ratio_range': (0.5, 2.0),
    'keep_ratio': True},
   {'type': 'RandomCrop', 'crop_size': (512, 1024), 'cat_max_ratio': 0.75},
   {'type': 'RandomFlip', 'prob': 0.5},
   {'type': 'PhotoMetricDistortion'},
   {'type': 'PackSegInputs'}]}}
```

{'type': 'RandomCrop', 'crop_size': (512, 1024), 'cat_max_ratio': 0.75},
```

`cfg` is an instance of `mmengine.config.Config`, its interface is the same as a dict object and also allows access config values as attributes. See [config tutorial](https://mmengine.readthedocs.io/en/latest/tutorials/config.html) in [MMEngine](https://github.com/open-mmlab/mmengine) for more information.

## FAQ

See [config tutorial](https://mmengine.readthedocs.io/en/latest/tutorials/config.html) in [MMEngine](https://github.com/open-mmlab/mmengine) for simple illustration.

In MMSegmentation, for example, if you would like to modify the backbone of PSPNet with the following config file `pspnet.py`:

```python

```python
    pretrained='torchvision://resnet50',
        depth=50,
```

Load and parse the config file `pspnet.py` in the code as follows:

loss_decode=dict(
```

```python

cfg = Config.fromfile('pspnet.py')
print(cfg.model)
```

```shell
{'type': 'EncoderDecoder',
 'pretrained': 'torchvision://resnet50',
 'backbone': {'type': 'ResNetV1c',
  'depth': 50,
  'num_stages': 4,
  'out_indices': (0, 1, 2, 3),
  'dilations': (1, 1, 2, 4),
  'strides': (1, 2, 1, 1),
  'norm_cfg': {'type': 'SyncBN', 'requires_grad': True},
  'norm_eval': False,
  'style': 'pytorch',
  'contract_dilation': True},
 'decode_head': {'type': 'PSPHead',
  'in_channels': 2048,
  'in_index': 3,
  'channels': 512,
  'pool_scales': (1, 2, 3, 6),
  'dropout_ratio': 0.1,
  'num_classes': 19,
  'align_corners': False,
  'loss_decode': {'type': 'CrossEntropyLoss',
   'use_sigmoid': False,
   'loss_weight': 1.0}}}
```

'norm_cfg': {'type': 'SyncBN', 'requires_grad': True},
```

`ResNet` and `HRNet` use different keywords to construct, write a new config file `hrnet.py` as follows:

```python
_base_ = 'pspnet.py'
    pretrained='open-mmlab://msra/hrnetv2_w32',

num_branches=2,
                num_channels=(32, 64, 128, 256)))))
```

Load and parse the config file `hrnet.py` in the code as follows:

```python
cfg = Config.fromfile('hrnet.py')
```

```shell
 'pretrained': 'open-mmlab://msra/hrnetv2_w32',
 'backbone': {'type': 'HRNet',
  'extra': {'stage1': {'num_modules': 1,
    'num_branches': 1,
    'block': 'BOTTLENECK',

'backbone': {'type': 'HRNet',
    'num_blocks': (4,),
    'num_channels': (64,)},
   'stage2': {'num_modules': 1,
    'num_branches': 2,
    'block': 'BASIC',
    'num_blocks': (4, 4),
    'num_channels': (32, 64)},
   'stage3': {'num_modules': 4,
    'num_branches': 3,
    'num_blocks': (4, 4, 4),
    'num_channels': (32, 64, 128)},
   'stage4': {'num_modules': 3,
    'num_branches': 4,
    'num_blocks': (4, 4, 4, 4),
    'num_channels': (32, 64, 128, 256)}}},

'dropout_ratio': 0.1,
```

It's worth noting that when modifying intermediate variables in the children configs, user need to pass the intermediate variables into corresponding fields again.
For example, we would like to change multi scale strategy to train/test a PSPNet. `train_pipeline`/`test_pipeline` are intermediate variable we would like to modify.

```python
_base_ = '../pspnet/pspnet_r50-d8_4xb4-40k_cityscpaes-512x1024.py'

```python
    dict(type='LoadAnnotations'),
    dict(type='RandomResize',
         img_scale=(2048, 1024),
         ratio_range=(1., 2.),
         keep_ration=True),
    dict(type='RandomCrop', crop_size=crop_size, cat_max_ratio=0.75),
    dict(type='PhotoMetricDistortion'),
    dict(type='PackSegInputs'),
]
    dict(type='Resize',
        scale=(2048, 1024),
    dict(type='PackSegInputs')
]
train_dataset=dict(
            img_path='leftImg8bit/train', seg_map_path='gtFine/train'),

]
        pipeline=train_pipeline)
test_dataset=dict(
            img_path='leftImg8bit/val', seg_map_path='gtFine/val'),
        pipeline=test_pipeline)
train_dataloader = dict(dataset=train_dataset)
val_dataloader = dict(dataset=test_dataset)
```

We first define the new `train_pipeline`/`test_pipeline` and pass them into `dataset`.

```python
    decode_head=dict(norm_cfg=norm_cfg),

    auxiliary_head=dict(norm_cfg=norm_cfg))
```

In the [training script](https://github.com/open-mmlab/mmsegmentation/blob/1.x/tools/train.py) and the [testing script](https://github.com/open-mmlab/mmsegmentation/blob/1.x/tools/test.py), we support the script argument `--cfg-options`, it may help users override some settings in the used config, the key-value pair in `xxx=yyy` format will be merged into config file.

For example, this is a simplified script `demo_script.py`:

```python
import argparse

from mmengine.config import Config, DictAction

def parse_args():
    parser = argparse.ArgumentParser(description='Script Example')
    parser.add_argument('config', help='train config file path')
    parser.add_argument(
        '--cfg-options',
        nargs='+',
        action=DictAction,

parser.add_argument('config', help='train config file path')
        help='override some settings in the used config, the key-value pair '
        'in xxx=yyy format will be merged into config file. If the value to '
        'be overwritten is a list, it should be like key="[a,b]" or key=a,b '
        'It also allows nested list/tuple values, e.g. key="[(a,b),(c,d)]" '
        'Note that the quotation marks are necessary and that no white space '
        'is allowed.')
    args = parser.parse_args()
    return args

def main():
    args = parse_args()

    cfg = Config.fromfile(args.config)
    if args.cfg_options is not None:
        cfg.merge_from_dict(args.cfg_options)

    print(cfg)

if __name__ == '__main__':
    main()
```

An example config file `demo_config.py` as follows:

```python
backbone = dict(
    type='ResNetV1c',
    depth=50,
    num_stages=4,

print(cfg)

    main()
```

```python
    depth=50,
    out_indices=(0, 1, 2, 3),
    dilations=(1, 1, 2, 4),
    strides=(1, 2, 1, 1),
    norm_eval=False,
    style='pytorch',
    contract_dilation=True)
```

Run `demo_script.py`:

```shell
python demo_script.py demo_config.py
```

```shell
Config (path: demo_config.py): {'backbone': {'type': 'ResNetV1c', 'depth': 50, 'num_stages': 4, 'out_indices': (0, 1, 2, 3), 'dilations': (1, 1, 2, 4), 'strides': (1, 2, 1, 1), 'norm_eval': False, 'style': 'pytorch', 'contract_dilation': True}}
```

Modify config through script arguments:

```shell
python demo_script.py demo_config.py --cfg-options backbone.depth=101
```

```shell

```

```shell
```

```shell
Config (path: demo_config.py): {'backbone': {'type': 'ResNetV1c', 'depth': 101, 'num_stages': 4, 'out_indices': (0, 1, 2, 3), 'dilations': (1, 1, 2, 4), 'strides': (1, 2, 1, 1), 'norm_eval': False, 'style': 'pytorch', 'contract_dilation': True}}
```

  If the value to be updated is a list or a tuple. For example, the config file `demo_config.py` sets `strides=(1, 2, 1, 1)` in `backbone`.
  If you want to change this key, you may specify in two ways:

  1. `--cfg-options backbone.strides="(1, 1, 1, 1)"`. Note that the quotation mark " is necessary to support list/tuple data types.

     ```shell
     python demo_script.py demo_config.py --cfg-options backbone.strides="(1, 1, 1, 1)"
     ```

     ```shell

```shell
     ```

     ```shell
     Config (path: demo_config.py): {'backbone': {'type': 'ResNetV1c', 'depth': 50, 'num_stages': 4, 'out_indices': (0, 1, 2, 3), 'dilations': (1, 1, 2, 4), 'strides': (1, 1, 1, 1), 'norm_eval': False, 'style': 'pytorch', 'contract_dilation': True}}
     ```

  2. `--cfg-options backbone.strides=1,1,1,1`. Note that **NO** white space is allowed in the specified value.
     In addition, if the original type is tuple, it will be automatically converted to list after this way.

     ```shell
     python demo_script.py demo_config.py --cfg-options backbone.strides=1,1,1,1
     ```

     ```shell
     Config (path: demo_config.py): {'backbone': {'type': 'ResNetV1c', 'depth': 50, 'num_stages': 4, 'out_indices': (0, 1, 2, 3), 'dilations': (1, 1, 2, 4), 'strides': [1, 1, 1, 1], 'norm_eval': False, 'style': 'pytorch', 'contract_dilation': True}}
     ```

```{note}

```

```{note}
    This modification method only supports modifying configuration items of string, int, float, boolean, None, list and tuple types.
    More specifically, for list and tuple types, the elements inside them must also be one of the above seven types.
```

# Backbones Trained by Self-Supervise Algorithms

<!-- [OTHERS] -->

Unsupervised image representations have significantly reduced the gap with supervised pretraining, notably with the recent achievements of contrastive learning methods. These contrastive methods typically work online and rely on a large number of explicit pairwise feature comparisons, which is computationally challenging. In this paper, we propose an online algorithm, SwAV, that takes advantage of contrastive methods without requiring to compute pairwise comparisons. Specifically, our method simultaneously clusters the data while enforcing consistency between cluster assignments produced for different augmentations (or views) of the same image, instead of comparing features directly as in contrastive learning. Simply put, we use a swapped prediction mechanism where we predict the cluster assignment of a view from the representation of another view. Our method can be trained with large and small batches and can scale to unlimited amounts of data. Compared to previous contrastive methods, our method is more memory efficient since it does not require a large memory bank or a special momentum network. In addition, we also propose a new data augmentation strategy, multi-crop, that uses a mix of views with different resolutions in place of two full-resolution views, without increasing the memory or compute requirements much. We validate our findings by achieving 75.3% top-1 accuracy on ImageNet with ResNet-50, as well as surpassing supervised pretraining on all the considered transfer tasks.

<img src="https://user-images.githubusercontent.com/40661020/143997246-ac40fd8a-9b48-4ff5-a0d9-ba10e1e333d2.png"/>
</div>

We present Momentum Contrast (MoCo) for unsupervised visual representation learning. From a perspective on contrastive learning as dictionary look-up, we build a dynamic dictionary with a queue and a moving-averaged encoder. This enables building a large and consistent dictionary on-the-fly that facilitates contrastive unsupervised learning. MoCo provides competitive results under the common linear protocol on ImageNet classification. More importantly, the representations learned by MoCo transfer well to downstream tasks. MoCo can outperform its supervised pre-training counterpart in 7 detection/segmentation tasks on PASCAL VOC, COCO, and other datasets, sometimes surpassing it by large margins. This suggests that the gap between unsupervised and supervised representation learning has been largely closed in many vision tasks.

<img src="https://user-images.githubusercontent.com/40661020/143997315-5ff824d4-1007-4b59-8952-bc5a2c0bfd78.png" height="300"/>
</div>

## Usage

To use a self-supervisely pretrained backbone, there are two steps to do:

1. Download and convert the model to PyTorch-style supported by MMDetection
2. Modify the config and change the training setting accordingly

### Convert model

For more general usage, we also provide script `selfsup2mmdet.py` in the tools directory to convert the key of models pretrained by different self-supervised methods to PyTorch-style checkpoints used in MMDetection.

```bash
python -u tools/model_converters/selfsup2mmdet.py ${PRETRAIN_PATH} ${STORE_PATH} --selfsup ${method}
```

This script convert model from `PRETRAIN_PATH` and store the converted model in `STORE_PATH`.

```

For example, to use a ResNet-50 backbone released by MoCo, you can download it from [here](https://dl.fbaipublicfiles.com/moco/moco_checkpoints/moco_v2_800ep/moco_v2_800ep_pretrain.pth.tar) and use the following command

```bash
python -u tools/model_converters/selfsup2mmdet.py ./moco_v2_800ep_pretrain.pth.tar mocov2_r50_800ep_pretrain.pth --selfsup moco
```

To use the ResNet-50 backbone released by SwAV, you can download it from [here](https://dl.fbaipublicfiles.com/deepcluster/swav_800ep_pretrain.pth.tar)

### Modify config

The backbone requires SyncBN and the `frozen_stages` need to be changed. A config that use the moco backbone is as below

```python
_base_ = [

```python
_base_ = [
]

    pretrained='./mocov2_r50_800ep_pretrain.pth',
        frozen_stages=0,
        norm_cfg=dict(type='SyncBN', requires_grad=True),
        norm_eval=False))

```

```

|  Method   |                            Backbone                            |  Style  |    Lr schd     | Mem (GB) | Inf time (fps) | box AP | mask AP |                         Config                         |                                                                                                                                                                                            Download                                                                                                                                                                                            |

| Mask RCNN |  [R50 by MoCo v2](./mask-rcnn_r50-mocov2-pre_fpn_1x_coco.py)   | pytorch |       1x       |          |                |  38.0  |  34.3   |  [config](./mask-rcnn_r50-mocov2-pre_fpn_1x_coco.py)   |       [model](https://download.openmmlab.com/mmdetection/v2.0/selfsup_pretrain/mask_rcnn_r50_fpn_mocov2-pretrain_1x_coco/mask_rcnn_r50_fpn_mocov2-pretrain_1x_coco_20210604_114614-a8b63483.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/selfsup_pretrain/mask_rcnn_r50_fpn_mocov2-pretrain_1x_coco/mask_rcnn_r50_fpn_mocov2-pretrain_1x_coco_20210604_114614.log.json)       |

| Mask RCNN | [R50 by MoCo v2](./mask-rcnn_r50-mocov2-pre_fpn_ms-2x_coco.py) | pytorch | multi-scale 2x |          |                |  40.8  |  36.8   | [config](./mask-rcnn_r50-mocov2-pre_fpn_ms-2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/selfsup_pretrain/mask_rcnn_r50_fpn_mocov2-pretrain_ms-2x_coco/mask_rcnn_r50_fpn_mocov2-pretrain_ms-2x_coco_20210605_163717-d95df20a.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/selfsup_pretrain/mask_rcnn_r50_fpn_mocov2-pretrain_ms-2x_coco/mask_rcnn_r50_fpn_mocov2-pretrain_ms-2x_coco_20210605_163717.log.json) |

| Mask RCNN |     [R50 by SwAV](./mask-rcnn_r50-swav-pre_fpn_1x_coco.py)     | pytorch |       1x       |          |                |  39.1  |  35.7   |   [config](./mask-rcnn_r50-swav-pre_fpn_1x_coco.py)    |           [model](https://download.openmmlab.com/mmdetection/v2.0/selfsup_pretrain/mask_rcnn_r50_fpn_swav-pretrain_1x_coco/mask_rcnn_r50_fpn_swav-pretrain_1x_coco_20210604_114640-7b9baf28.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/selfsup_pretrain/mask_rcnn_r50_fpn_swav-pretrain_1x_coco/mask_rcnn_r50_fpn_swav-pretrain_1x_coco_20210604_114640.log.json)           |

| Mask RCNN |   [R50 by SwAV](./mask-rcnn_r50-swav-pre_fpn_ms-2x_coco.py)    | pytorch | multi-scale 2x |          |                |  41.3  |  37.3   |  [config](./mask-rcnn_r50-swav-pre_fpn_ms-2x_coco.py)  |     [model](https://download.openmmlab.com/mmdetection/v2.0/selfsup_pretrain/mask_rcnn_r50_fpn_swav-pretrain_ms-2x_coco/mask_rcnn_r50_fpn_swav-pretrain_ms-2x_coco_20210605_163717-08e26fca.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/selfsup_pretrain/mask_rcnn_r50_fpn_swav-pretrain_ms-2x_coco/mask_rcnn_r50_fpn_swav-pretrain_ms-2x_coco_20210605_163717.log.json)     |

### Notice

1. We only provide single-scale 1x and multi-scale 2x configs as examples to show how to use backbones trained by self-supervised algorithms. We will try to reproduce the results in their corresponding paper using the released backbone in the future. Please stay tuned.

We support to apply the backbone models pre-trained by different self-supervised methods in detection systems and provide their results on Mask R-CNN.

The pre-trained models are converted from [MoCo](https://github.com/facebookresearch/moco) and downloaded from [SwAV](https://github.com/facebookresearch/swav).

For SwAV, please cite

```latex
@article{caron2020unsupervised,
  title={Unsupervised Learning of Visual Features by Contrasting Cluster Assignments},
  author={Caron, Mathilde and Misra, Ishan and Mairal, Julien and Goyal, Priya and Bojanowski, Piotr and Joulin, Armand},
  booktitle={Proceedings of Advances in Neural Information Processing Systems (NeurIPS)},
}
```

For MoCo, please cite

```latex
@Article{he2019moco,
  author  = {Kaiming He and Haoqi Fan and Yuxin Wu and Saining Xie and Ross Girshick},
  title   = {Momentum Contrast for Unsupervised Visual Representation Learning},
  journal = {arXiv preprint arXiv:1911.05722},
  year    = {2019},

title   = {Momentum Contrast for Unsupervised Visual Representation Learning},
}
@Article{chen2020mocov2,
  author  = {Xinlei Chen and Haoqi Fan and Ross Girshick and Kaiming He},
  title   = {Improved Baselines with Momentum Contrastive Learning},
  journal = {arXiv preprint arXiv:2003.04297},
  year    = {2020},
}
```

# CenterFormer: Center-based Transformer for 3D Object Detection

> [CenterFormer: Center-based Transformer for 3D Object Detection](https://arxiv.org/abs/2209.05588)

Query-based transformer has shown great potential in con-
structing long-range attention in many image-domain tasks, but has
rarely been considered in LiDAR-based 3D object detection due to the
overwhelming size of the point cloud data. In this paper, we propose
CenterFormer, a center-based transformer network for 3D object de-
tection. CenterFormer first uses a center heatmap to select center candi-
dates on top of a standard voxel-based point cloud encoder. It then uses
the feature of the center candidate as the query embedding in the trans-
former. To further aggregate features from multiple frames, we design
an approach to fuse features through cross-attention. Lastly, regression
heads are added to predict the bounding box on the output center feature

representation. Our design reduces the convergence difficulty and compu-
tational complexity of the transformer structure. The results show signif-
icant improvements over the strong baseline of anchor-free object detec-
tion networks. CenterFormer achieves state-of-the-art performance for a
single model on the Waymo Open Dataset, with 73.7% mAPH on the val-
idation set and 75.6% mAPH on the test set, significantly outperforming
all previously published CNN and transformer-based methods. Our code
is publicly available at https://github.com/TuSimple/centerformer

<img src="https://user-images.githubusercontent.com/34888372/209500088-b707d7cd-d4d5-4f20-8fdf-a2c7ad15df34.png" width="800"/>
</div>

We implement CenterFormer and provide the results and checkpoints on Waymo dataset.

## Usage

</div>

## Usage

<!-- For a typical model, this section should contain the commands for training and testing. You are also suggested to dump your environment specification to env.yml by `conda env export > env.yml`. -->

In MMDetection3D's root directory, run the following command to train the model:

```bash
python tools/train.py projects/CenterFormer/configs/centerformer_voxel01_second-atten_secfpn-atten_4xb4-cyclic-20e_waymoD5-3d-3class.py
```

For multi-gpu training, run:

```bash
python -m torch.distributed.launch --nnodes=1 --node_rank=0 --nproc_per_node=${NUM_GPUS} --master_port=29506 --master_addr="127.0.0.1" tools/train.py projects/CenterFormer/configs/centerformer_voxel01_second-atten_secfpn-atten_4xb4-cyclic-20e_waymoD5-3d-3class.py
```

In MMDetection3D's root directory, run the following command to test the model:

```bash

```

```bash
python tools/test.py projects/CenterFormer/configs/centerformer_voxel01_second-atten_secfpn-atten_4xb4-cyclic-20e_waymoD5-3d-3class.py ${CHECKPOINT_PATH}
```

### Waymo

|                                                      Backbone                                                       | Load Interval | Voxel type (voxel size) | Multi-Class NMS | Multi-frames | Mem (GB) | Inf time (fps) | mAP@L1 | mAPH@L1 | mAP@L2 | **mAPH@L2** |                                                                                                                                 Download                                                                                                                                  |

| [SECFPN_WithAttention](./configs/centerformer_voxel01_second-attn_secfpn-attn_4xb4-cyclic-20e_waymoD5-3d-3class.py) |       5       |       voxel (0.1)       |        ✓        |      ×       |   14.8   |                |  72.2  |  69.5   |  65.9  |    63.3     | [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/centerformer/centerformer_voxel01_second-attn_secfpn-attn_4xb4-cyclic-20e_waymoD5-3d-3class/centerformer_voxel01_second-attn_secfpn-attn_4xb4-cyclic-20e_waymoD5-3d-3class_20221227_205613-70c9ad37.log) |

**Note** that `SECFPN_WithAttention` denotes both SECOND and SECONDFPN with ChannelAttention and SpatialAttention.

```latex
@InProceedings{Zhou_centerformer,
title = {CenterFormer: Center-based Transformer for 3D Object Detection},
author = {Zhou, Zixiang and Zhao, Xiangchen and Wang, Yu and Wang, Panqu and Foroosh, Hassan},
booktitle = {ECCV},
}
```

# Prepare Datasets

In this document, we will give a guide on the process of preparing datasets for the MMPose. Various aspects of dataset preparation will be discussed, including using built-in datasets, creating custom datasets, combining datasets for training, browsing and downloading the datasets.

## Use built-in datasets

**Step 1**: Prepare Data

MMPose supports multiple tasks and corresponding datasets. You can find them in [dataset zoo](https://mmpose.readthedocs.io/en/latest/dataset_zoo.html). To properly prepare your data, please follow the guidelines associated with your chosen dataset.

**Step 2**: Configure Dataset Settings in the Config File

Before training or evaluating models, you must configure the dataset settings. Take [`td-hm_hrnet-w32_8xb64-210e_coco-256x192.py`](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192.py) for example, which can be used to train or evaluate the HRNet pose estimator on COCO dataset. We will go through the dataset configuration.

- Basic Dataset Arguments

  ```python
  # base dataset settings
  dataset_type = 'CocoDataset'
  data_mode = 'topdown'
  data_root = 'data/coco/'
  ```

  - `dataset_type` specifies the class name of the dataset. Users can refer to [Datasets APIs](https://mmpose.readthedocs.io/en/latest/api.html#datasets) to find the class name of their desired dataset.

- `data_mode` determines the output format of the dataset, with two options available: `'topdown'` and `'bottomup'`. If `data_mode='topdown'`, the data element represents a single instance with its pose; otherwise, the data element is an entire image containing multiple instances and poses.
  - `data_root` designates the root directory of the dataset.

- Data Processing Pipelines

  ```python
  # pipelines
  train_pipeline = [
      dict(type='LoadImage'),
      dict(type='GetBBoxCenterScale'),
      dict(type='RandomFlip', direction='horizontal'),
      dict(type='RandomHalfBody'),
      dict(type='RandomBBoxTransform'),
      dict(type='TopdownAffine', input_size=codec['input_size']),
      dict(type='GenerateTarget', encoder=codec),
      dict(type='PackPoseInputs')
  ]
  val_pipeline = [
  ]
  ```

val_pipeline = [
  ]
  ```

  The `train_pipeline` and `val_pipeline` define the steps to process data elements during the training and evaluation phases, respectively. In addition to loading images and packing inputs, the `train_pipeline` primarily consists of data augmentation techniques and target generator, while the `val_pipeline` focuses on transforming data elements into a unified format.

- Data Loaders

  ```python
  # data loaders
  train_dataloader = dict(
      batch_size=64,
      num_workers=2,
      persistent_workers=True,
      sampler=dict(type='DefaultSampler', shuffle=True),
      dataset=dict(
          type=dataset_type,
          data_root=data_root,
          data_mode=data_mode,
          ann_file='annotations/person_keypoints_train2017.json',
          data_prefix=dict(img='train2017/'),

data_root=data_root,
          pipeline=train_pipeline,
      ))
  val_dataloader = dict(
      batch_size=32,
      drop_last=False,
      sampler=dict(type='DefaultSampler', shuffle=False, round_up=False),
          ann_file='annotations/person_keypoints_val2017.json',
          bbox_file='data/coco/person_detection_results/'
          'COCO_val2017_detections_AP_H_56_person.json',
          data_prefix=dict(img='val2017/'),
          test_mode=True,
          pipeline=val_pipeline,
      ))
  test_dataloader = val_dataloader
  ```

data_prefix=dict(img='val2017/'),
      ))
  ```

  This section is crucial for configuring the dataset in the config file. In addition to the basic dataset arguments and pipelines discussed earlier, other important parameters are defined here. The `batch_size` determines the batch size per GPU; the `ann_file` indicates the annotation file for the dataset; and `data_prefix` specifies the image folder. The `bbox_file`, which supplies detected bounding box information, is only used in the val/test data loader for top-down datasets.

We recommend copying the dataset configuration from provided config files that use the same dataset, rather than writing it from scratch, in order to minimize potential errors. By doing so, users can simply make the necessary modifications as needed, ensuring a more reliable and efficient setup process.

## Use a custom dataset

The [Customize Datasets](../advanced_guides/customize_datasets.md) guide provides detailed information on how to build a custom dataset. In this section, we will highlight some key tips for using and configuring custom datasets.

- Determine the dataset class name. If you reorganize your dataset into the COCO format, you can simply use `CocoDataset` as the value for `dataset_type`. Otherwise, you will need to use the name of the custom dataset class you added.

- Specify the meta information config file. MMPose 1.x employs a different strategy for specifying meta information compared to MMPose 0.x. In MMPose 1.x, users can specify the meta information config file as follows:

  ```python
      ...
          data_root='root/of/your/train/data',
          ann_file='path/to/your/train/json',
          data_prefix=dict(img='path/to/your/train/img'),

          # specify dataset meta information
          metainfo=dict(from_file='configs/_base_/datasets/custom.py'),
          ...),
  )
  ```

  Note that the argument `metainfo` must be specified in the val/test data loaders as well.

## Use mixed datasets for training

MMPose offers a convenient and versatile solution for training with mixed datasets. Please refer to [Use Mixed Datasets for Training](./mixed_datasets.md).

## Browse dataset

`tools/analysis_tools/browse_dataset.py` helps the user to browse a pose dataset visually, or save the image to a designated directory.

```shell
python tools/misc/browse_dataset.py ${CONFIG} [-h] [--output-dir ${OUTPUT_DIR}] [--not-show] [--phase ${PHASE}] [--mode ${MODE}] [--show-interval ${SHOW_INTERVAL}]
```

```shell
```

| ARGS                             | Description                                                                                                                                          |
| -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `CONFIG`                         | The path to the config file.                                                                                                                         |
| `--output-dir OUTPUT_DIR`        | The target folder to save visualization results. If not specified, the visualization results will not be saved.                                      |

| `--not-show`                     | Do not show the visualization results in an external window.                                                                                         |
| `--phase {train, val, test}`     | Options for dataset.                                                                                                                                 |
| `--mode {original, transformed}` | Specify the type of visualized images. `original` means to show images without pre-processing; `transformed` means to show images are pre-processed. |
| `--show-interval SHOW_INTERVAL`  | Time interval between visualizing two images.                                                                                                        |

For instance, users who want to visualize images and annotations in COCO dataset use:

```shell
python tools/misc/browse_dataset.py configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-e210_coco-256x192.py --mode original
```

The bounding boxes and keypoints will be plotted on the original image. Following is an example:
![original_coco](https://user-images.githubusercontent.com/26127467/187383698-7e518f21-b4cc-4712-9e97-99ddd8f0e437.jpg)

The original images need to be processed before being fed into models. To visualize pre-processed images and annotations, users need to modify the argument `mode`  to `transformed`. For example:

```shell
python tools/misc/browse_dataset.py configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-e210_coco-256x192.py --mode transformed

```shell
```

Here is a processed sample

![transformed_coco](https://user-images.githubusercontent.com/26127467/187386652-bd47335d-797c-4e8c-b823-2a4915f9812f.jpg)

The heatmap target will be visualized together if it is generated in the pipeline.

## Download dataset via MIM

By using [OpenDataLab](https://opendatalab.com/), you can obtain free formatted datasets in various fields. Through the search function of the platform, you may address the dataset they look for quickly and easily. Using the formatted datasets from the platform, you can efficiently conduct tasks across datasets.

If you use MIM to download, make sure that the version is greater than v0.3.8. You can use the following command to update, install, login and download the dataset:

```shell
# upgrade your MIM
pip install -U openmim

# install OpenDataLab CLI tools

```shell

pip install -U opendatalab
# log in OpenDataLab, registry
odl login

# download coco2017 and preprocess by MIM
mim download mmpose --dataset coco2017
```

### Supported datasets

Here is the list of supported datasets, we will continue to update it in the future.

#### Body

| Dataset name  | Download command                          |
| ------------- | ----------------------------------------- |
| COCO 2017     | `mim download mmpose --dataset coco2017`  |
| MPII          | `mim download mmpose --dataset mpii`      |
| AI Challenger | `mim download mmpose --dataset aic`       |
| CrowdPose     | `mim download mmpose --dataset crowdpose` |

#### Face

| Dataset name | Download command                     |
| ------------ | ------------------------------------ |
| LaPa         | `mim download mmpose --dataset lapa` |
| 300W         | `mim download mmpose --dataset 300w` |

| WFLW         | `mim download mmpose --dataset wflw` |

#### Hand

| Dataset name | Download command                           |
| ------------ | ------------------------------------------ |
| OneHand10K   | `mim download mmpose --dataset onehand10k` |
| FreiHand     | `mim download mmpose --dataset freihand`   |
| HaGRID       | `mim download mmpose --dataset hagrid`     |

#### Whole Body

| Dataset name | Download command                      |
| ------------ | ------------------------------------- |
| Halpe        | `mim download mmpose --dataset halpe` |

#### Animal

| AP-10K       | `mim download mmpose --dataset ap10k` |

#### Fashion

Coming Soon

# DANet

> [Dual Attention Network for Scene Segmentation](https://arxiv.org/abs/1809.02983)

<a href="https://github.com/junfu1115/DANet/">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/decode_heads/da_head.py#L76">Code Snippet</a>

In this paper, we address the scene segmentation task by capturing rich contextual dependencies based on the selfattention mechanism. Unlike previous works that capture contexts by multi-scale features fusion, we propose a Dual Attention Networks (DANet) to adaptively integrate local features with their global dependencies. Specifically, we append two types of attention modules on top of traditional dilated FCN, which model the semantic interdependencies in spatial and channel dimensions respectively. The position attention module selectively aggregates the features at each position by a weighted sum of the features at all positions. Similar features would be related to each other regardless of their distances. Meanwhile, the channel attention module selectively emphasizes interdependent channel maps by integrating associated features among all channel maps. We sum the outputs of the two attention modules to further improve feature representation which contributes to more precise segmentation results. We achieve new state-of-the-art segmentation performance on three challenging scene segmentation datasets, i.e., Cityscapes, PASCAL Context and COCO Stuff dataset. In particular, a Mean IoU score of 81.5% on Cityscapes test set is achieved without using coarse data. We make the code and trained model publicly available at [this https URL](https://github.com/junfu1115/DANet).

<img src="https://user-images.githubusercontent.com/24582831/142900467-f832fdb9-3b7d-47d3-8e80-e6ee9303bdfb.png" width="70%"/>
</div>

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                       | download                                                                                                                                                                                                                                                                                                                                           |

| DANet  | R-50-D8  | 512x1024  |   40000 | 7.4      | 2.66           | V100   | 78.74 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/danet/danet_r50-d8_4xb2-40k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r50-d8_512x1024_40k_cityscapes/danet_r50-d8_512x1024_40k_cityscapes_20200605_191324-c0dbfa5f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r50-d8_512x1024_40k_cityscapes/danet_r50-d8_512x1024_40k_cityscapes_20200605_191324.log.json)     |

| DANet  | R-101-D8 | 512x1024  |   40000 | 10.9     | 1.99           | V100   | 80.52 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/danet/danet_r101-d8_4xb2-40k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r101-d8_512x1024_40k_cityscapes/danet_r101-d8_512x1024_40k_cityscapes_20200605_200831-c57a7157.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r101-d8_512x1024_40k_cityscapes/danet_r101-d8_512x1024_40k_cityscapes_20200605_200831.log.json) |

| DANet  | R-50-D8  | 769x769   |   40000 | 8.8      | 1.56           | V100   | 78.88 | 80.62         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/danet/danet_r50-d8_4xb2-40k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r50-d8_769x769_40k_cityscapes/danet_r50-d8_769x769_40k_cityscapes_20200530_025703-76681c60.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r50-d8_769x769_40k_cityscapes/danet_r50-d8_769x769_40k_cityscapes_20200530_025703.log.json)         |

| DANet  | R-101-D8 | 769x769   |   40000 | 12.8     | 1.07           | V100   | 79.88 | 81.47         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/danet/danet_r101-d8_4xb2-40k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r101-d8_769x769_40k_cityscapes/danet_r101-d8_769x769_40k_cityscapes_20200530_025717-dcb7fd4e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r101-d8_769x769_40k_cityscapes/danet_r101-d8_769x769_40k_cityscapes_20200530_025717.log.json)     |

| DANet  | R-50-D8  | 512x1024  |   80000 | -        | -              | V100   | 79.34 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/danet/danet_r50-d8_4xb2-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r50-d8_512x1024_80k_cityscapes/danet_r50-d8_512x1024_80k_cityscapes_20200607_133029-2bfa2293.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r50-d8_512x1024_80k_cityscapes/danet_r50-d8_512x1024_80k_cityscapes_20200607_133029.log.json)     |

| DANet  | R-101-D8 | 512x1024  |   80000 | -        | -              | V100   | 80.41 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/danet/danet_r101-d8_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r101-d8_512x1024_80k_cityscapes/danet_r101-d8_512x1024_80k_cityscapes_20200607_132918-955e6350.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r101-d8_512x1024_80k_cityscapes/danet_r101-d8_512x1024_80k_cityscapes_20200607_132918.log.json) |

| DANet  | R-50-D8  | 769x769   |   80000 | -        | -              | V100   | 79.27 | 80.96         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/danet/danet_r50-d8_4xb2-80k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r50-d8_769x769_80k_cityscapes/danet_r50-d8_769x769_80k_cityscapes_20200607_132954-495689b4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r50-d8_769x769_80k_cityscapes/danet_r50-d8_769x769_80k_cityscapes_20200607_132954.log.json)         |

| DANet  | R-101-D8 | 769x769   |   80000 | -        | -              | V100   | 80.47 | 82.02         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/danet/danet_r101-d8_4xb2-80k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r101-d8_769x769_80k_cityscapes/danet_r101-d8_769x769_80k_cityscapes_20200607_132918-f3a929e7.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r101-d8_769x769_80k_cityscapes/danet_r101-d8_769x769_80k_cityscapes_20200607_132918.log.json)     |

### ADE20K

### ADE20K

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                   | download                                                                                                                                                                                                                                                                                                                           |

| DANet  | R-50-D8  | 512x512   |   80000 | 11.5     | 21.20          | V100   | 41.66 |         42.90 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/danet/danet_r50-d8_4xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r50-d8_512x512_80k_ade20k/danet_r50-d8_512x512_80k_ade20k_20200615_015125-edb18e08.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r50-d8_512x512_80k_ade20k/danet_r50-d8_512x512_80k_ade20k_20200615_015125.log.json)         |

| DANet  | R-101-D8 | 512x512   |   80000 | 15       | 14.18          | V100   | 43.64 |         45.19 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/danet/danet_r101-d8_4xb4-80k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r101-d8_512x512_80k_ade20k/danet_r101-d8_512x512_80k_ade20k_20200615_015126-d0357c73.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r101-d8_512x512_80k_ade20k/danet_r101-d8_512x512_80k_ade20k_20200615_015126.log.json)     |

| DANet  | R-50-D8  | 512x512   |  160000 | -        | -              | V100   | 42.45 |         43.25 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/danet/danet_r50-d8_4xb4-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r50-d8_512x512_160k_ade20k/danet_r50-d8_512x512_160k_ade20k_20200616_082340-9cb35dcd.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r50-d8_512x512_160k_ade20k/danet_r50-d8_512x512_160k_ade20k_20200616_082340.log.json)     |

| DANet  | R-101-D8 | 512x512   |  160000 | -        | -              | V100   | 44.17 |         45.02 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/danet/danet_r101-d8_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r101-d8_512x512_160k_ade20k/danet_r101-d8_512x512_160k_ade20k_20200616_082348-23bf12f9.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r101-d8_512x512_160k_ade20k/danet_r101-d8_512x512_160k_ade20k_20200616_082348.log.json) |

### Pascal VOC 2012 + Aug

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                    | download                                                                                                                                                                                                                                                                                                                               |

| DANet  | R-50-D8  | 512x512   |   20000 | 6.5      | 20.94          | V100   | 74.45 |         75.69 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/danet/danet_r50-d8_4xb4-20k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r50-d8_512x512_20k_voc12aug/danet_r50-d8_512x512_20k_voc12aug_20200618_070026-9e9e3ab3.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r50-d8_512x512_20k_voc12aug/danet_r50-d8_512x512_20k_voc12aug_20200618_070026.log.json)     |

| DANet  | R-101-D8 | 512x512   |   20000 | 9.9      | 13.76          | V100   | 76.02 |         77.23 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/danet/danet_r101-d8_4xb4-20k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r101-d8_512x512_20k_voc12aug/danet_r101-d8_512x512_20k_voc12aug_20200618_070026-d48d23b2.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r101-d8_512x512_20k_voc12aug/danet_r101-d8_512x512_20k_voc12aug_20200618_070026.log.json) |

| DANet  | R-50-D8  | 512x512   |   40000 | -        | -              | V100   | 76.37 |         77.29 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/danet/danet_r50-d8_4xb4-40k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r50-d8_512x512_40k_voc12aug/danet_r50-d8_512x512_40k_voc12aug_20200613_235526-426e3a64.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r50-d8_512x512_40k_voc12aug/danet_r50-d8_512x512_40k_voc12aug_20200613_235526.log.json)     |

| DANet  | R-101-D8 | 512x512   |   40000 | -        | -              | V100   | 76.51 |         77.32 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/danet/danet_r101-d8_4xb4-40k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r101-d8_512x512_40k_voc12aug/danet_r101-d8_512x512_40k_voc12aug_20200613_223031-788e232a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/danet/danet_r101-d8_512x512_40k_voc12aug/danet_r101-d8_512x512_40k_voc12aug_20200613_223031.log.json) |

```bibtex
@article{fu2018dual,
  title={Dual Attention Network for Scene Segmentation},
  author={Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang,and Hanqing Lu},
  booktitle={The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
}
```

# Foot Ulcer Segmentation Challenge 2021 (FUSC 2021)

This project supports **`Foot Ulcer Segmentation Challenge 2021 (FUSC 2021) `**, which can be downloaded from [here](https://fusc.grand-challenge.org/).

This chronic wound dataset was collected over 2 years from October 2019 to April 2021 at the center and contains 1,210 foot ulcer images taken from 889 patients during multiple clinical visits. The raw images were taken by Canon SX 620 HS digital camera and iPad Pro under uncontrolled illumination conditions,

with various backgrounds. The images (shown in Figure 1) are randomly split into 3 subsets: a training set with 810 images, a validation set with 200 images, and a testing set with 200 images. Of course, the annotations of the testing set are kept private. The data collected were de-identified and in accordance with relevant guidelines and regulations and the patient’s informed consent is waived by the institutional review board of the University of Wisconsin-Milwaukee.

### Information Statistics

| Dataset Name                                  | Anatomical Region | Task Type    | Modality       | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License                                                       |
| --------------------------------------------- | ----------------- | ------------ | -------------- | ------------ | --------------------- | ---------------------- | ------------ | ------------------------------------------------------------- |

| [fusc2021](https://fusc.grand-challenge.org/) | lower limb        | segmentation | histopathology | 2            | 810/200/200           | yes/yes/no             | 2021         | [CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/) |

| background |    810     |   98.71    |   200    |  98.78   |     -     |     -     |
|   wound    |    791     |    1.29    |   195    |   1.22   |     -     |     -     |

Note:

![fusc2021](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/histopathology/fusc2021/fusc2021_dataset.png?raw=true)

```
@article{s41598-020-78799-w,
	title={Fully automatic wound segmentation with deep convolutional neural networks},

```
	author={Chuanbo Wang and D. M. Anisuzzaman and Victor Williamson and Mrinal Kanti Dhar and Behrouz Rostami and Jeffrey Niezgoda and Sandeep Gopalakrishnan and Zeyun Yu},
	journal={Scientific Reports},
	volume={10},
	number={1},
	pages={21897},
	year={2020}
}
```

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `fusc2021/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
```

- download dataset from [here](https://fusc.grand-challenge.org/) and decompress data to path `'data/'`.

```none
  ├── mmseg
  │   │   │   ├── histopathology
  │   │   │   │   ├── fusc2021

```

```shell
```

```shell
```

```

# Shufflenet: An extremely efficient convolutional neural network for mobile devices

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_cvpr_2018/html/Zhang_ShuffleNet_An_Extremely_CVPR_2018_paper.html">ShufflenetV1 (CVPR'2018)</a></summary>

```bibtex
@inproceedings{zhang2018shufflenet,
  title={Shufflenet: An extremely efficient convolutional neural network for mobile devices},
  author={Zhang, Xiangyu and Zhou, Xinyu and Lin, Mengxiao and Sun, Jian},
  pages={6848--6856},
}
```

booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
}
```

We introduce an extremely computation-efficient CNN architecture named ShuffleNet, which is designed specially for mobile devices with very limited computing power (e.g., 10-150 MFLOPs). The new architecture utilizes two new operations, pointwise group convolution and channel shuffle, to greatly reduce computation cost while maintaining accuracy. Experiments on ImageNet classification and MS COCO object detection demonstrate the superior performance of ShuffleNet over other structures, e.g. lower top-1 error (absolute 7.8%) than recent MobileNet~cite{howard2017mobilenets} on ImageNet classification task, under the computation budget of 40 MFLOPs. On an ARM-based mobile device, ShuffleNet achieves $sim$13$ imes$ actual speedup over AlexNet while maintaining comparable accuracy.

<img src="https://user-images.githubusercontent.com/15977946/146540601-7ed2ac24-aa7d-44b1-a7e0-be03bde0e098.png">
</div>

# CLIP4Clip

[CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://arxiv.org/abs/2104.08860)

Video-text retrieval plays an essential role in multi-modal research and has been widely used in many real-world web applications. The CLIP (Contrastive Language-Image Pre-training), an image-language pre-training model, has demonstrated the power of visual concepts learning from web collected image-text datasets. In this paper, we propose a CLIP4Clip model to transfer the knowledge of the CLIP model to video-language retrieval in an end-to-end manner. Several questions are investigated via empirical studies: 1) Whether image feature is enough for video-text retrieval? 2) How a post-pretraining on a large-scale video-text dataset based on the CLIP affect the performance? 3) What is the practical mechanism to model temporal dependency between video frames? And 4) The Hyper-parameters sensitivity of the model on video-text retrieval task. Extensive experimental results present that the CLIP4Clip model transferred from the CLIP can achieve SOTA results on various video-text retrieval datasets, including MSR-VTT, MSVC, LSMDC, ActivityNet, and DiDeMo.

<img src="https://github.com/Dai-Wenxun/mmaction2/assets/58767402/f91fc927-d5f2-41dd-8198-def71d392991" width="800"/>
</div>

### MSRVTT-9k

| frame sampling strategy | resolution | gpus | backbone | adapter | pretrain | Recall@1 | Recall@5 | Recall@10 | MdR | MnR  | testing protocol |              config              |              ckpt              |              log               |
| :---------------------: | :--------: | :--: | :------: | :-----: | :------: | :------: | :------: | :-------: | :-: | :--: | :--------------: | :------------------------------: | :----------------------------: | :----------------------------: |

|       uniform 12        |  224x224   |  8   | ViT-B/32 |  Mean   |   clip   |   43.1   |   69.4   |   78.9    | 2.0 | 16.8 | 1 clips x 1 crop | [config](/configs/retrieval/clip4clip/clip4clip_vit-base-p32-res224-clip-pre_8xb16-u12-5e_msrvtt-9k-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/retrieval/clip4clip/clip4clip_vit-base-p32-res224-clip-pre_8xb16-u12-5e_msrvtt-9k-rgb/clip4clip_vit-base-p32-res224-clip-pre_8xb16-u12-5e_msrvtt-9k-rgb_20230612-b9706e54.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/retrieval/clip4clip/clip4clip_vit-base-p32-res224-clip-pre_8xb16-u12-5e_msrvtt-9k-rgb/clip4clip_vit-base-p32-res224-clip-pre_8xb16-u12-5e_msrvtt-9k-rgb.log) |

For more details on data preparation, you can refer to [video_retrieval](/tools/data/video_retrieval/README.md).

## Train

```shell
```

## Train

```shell
```

Example: train CLIP4Clip model on MSRVTT-9k dataset in a deterministic option with periodic validation.

```shell
python tools/train.py configs/retrieval/clip4clip/clip4clip_vit-base-p32-res224-clip-pre_8xb16-u12-5e_msrvtt-9k-rgb.py \
```

## Test

```shell
```

Example: test CLIP4Clip model on MSRVTT-9k dataset and dump the result to a pkl file.

```shell
python tools/test.py configs/retrieval/clip4clip/clip4clip_vit-base-p32-res224-clip-pre_8xb16-u12-5e_msrvtt-9k-rgb.py \
```

```shell
```

```BibTeX
@article{luo2022clip4clip,
  title={CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioning},
  author={Luo, Huaishao and Ji, Lei and Zhong, Ming and Chen, Yang and Lei, Wen and Duan, Nan and Li, Tianrui},
  journal={Neurocomputing},
  volume={508},
  pages={293--304},
}
```

# MMDetection Gradio Demo

Here is a gradio demo for MMDetection supported inference tasks.

Currently supported tasks:

- Object Detection
- Instance Segmentation
- Panoptic Segmentation
- Grounding Object Detection
- Open Vocabulary Object Detection
- Open Vocabulary Instance Segmentation
- Open Vocabulary Semantic Segmentation
- Open Vocabulary Panoptic Segmentation
- Referring Expression Segmentation
- Image Caption
- Referring Expression Image Caption
- Text-To-Image Retrieval

## Preview

<img src="https://github.com/open-mmlab/mmdetection/assets/17425982/421adfab-98cb-4b65-ab99-15d154cba95f" width="90%"/>

## Requirements

To run the demo, you need to install MMDetection at first. And please install with the extra multi-modality
dependencies to enable multi-modality tasks.

```shell
# At the MMDetection root folder
pip install -e ".[multimodal]"
```

And then install the latest gradio package.

```shell
pip install "gradio>=3.31.0"
```

## Start

```shell
```

```shell
```

## Start

Then, you can start the gradio server on the local machine by:

```shell
cd mmdetection
python projects/gradio_demo/launch.py
```

The demo will start a local server `http://127.0.0.1:7860` and you can browse it by your browser.

# Training and Testing

## Launch training

### Train with your PC

You can use `tools/train.py` to train a model on a single machine with a CPU and optionally a GPU.

Here is the full usage of the script:

```shell
```

```{note}
By default, MMPose prefers GPU to CPU. If you want to train a model on CPU, please empty `CUDA_VISIBLE_DEVICES` or set it to -1 to make GPU invisible to the program.

```

```shell
```

| ARGS                                  | Description                                                                                                                                                         |

| `CONFIG_FILE`                         | The path to the config file.                                                                                                                                        |
| `--work-dir WORK_DIR`                 | The target folder to save logs and checkpoints. Defaults to a folder with the same name as the config file under `./work_dirs`.                                     |
| `--resume [RESUME]`                   | Resume training. If specify a path, resume from it, while if not specify, try to auto resume from the latest checkpoint.                                            |
| `--amp`                               | Enable automatic-mixed-precision training.                                                                                                                          |

| `--no-validate`                       | **Not suggested**. Disable checkpoint evaluation during training.                                                                                                   |
| `--auto-scale-lr`                     | Automatically rescale the learning rate according to the actual batch size and the original batch size.                                                             |
| `--cfg-options CFG_OPTIONS`           | Override some settings in the used config, the key-value pair in xxx=yyy format will be merged into the config file. If the value to be overwritten is a list, it should be of the form of either `key="[a,b]"` or `key=a,b`. The argument also allows nested list/tuple values, e.g. `key="[(a,b),(c,d)]"`. Note that quotation marks are necessary and that **no white space is allowed**. |

| `--show-dir SHOW_DIR`                 | The directory to save the result visualization images generated during validation.                                                                                  |
| `--show`                              | Visualize the prediction result in a window.                                                                                                                        |
| `--interval INTERVAL`                 | The interval of samples to visualize.                                                                                                                               |
| `--wait-time WAIT_TIME`               | The display time of every window (in seconds). Defaults to 1.                                                                                                       |

| `--launcher {none,pytorch,slurm,mpi}` | Options for job launcher.                                                                                                                                           |

### Train with multiple GPUs

We provide a shell script to start a multi-GPUs task with `torch.distributed.launch`.

```shell
```

| ARGS          | Description                                                                        |
| ------------- | ---------------------------------------------------------------------------------- |
| `CONFIG_FILE` | The path to the config file.                                                       |
| `GPU_NUM`     | The number of GPUs to be used.                                                     |
| `[PYARGS]`    | The other optional arguments of `tools/train.py`, see [here](#train-with-your-pc). |

You can also specify extra arguments of the launcher by environment variables. For example, change the
communication port of the launcher to 29666 by the below command:

```shell
```

If you want to startup multiple training jobs and use different GPUs, you can launch them by specifying
different port and visible devices.

```shell
CUDA_VISIBLE_DEVICES=4,5,6,7 GPUS=29501 bash ./tools/dist_train.sh ${CONFIG_FILE2} 4 [PY_ARGS]
```

### Train with multiple machines

#### Multiple machines in the same network

If you launch a training job with multiple machines connected with ethernet, you can run the following commands:

On the first machine:

```shell

```shell
```

On the second machine:

```shell
```

Compared with multi-GPUs in a single machine, you need to specify some extra environment variables:

| ENV_VARS      | Description                                                                  |
| ------------- | ---------------------------------------------------------------------------- |
| `NNODES`      | The total number of machines.                                                |
| `NODE_RANK`   | The index of the local machine.                                              |

| `PORT`        | The communication port, it should be the same in all machines.               |
| `MASTER_ADDR` | The IP address of the master machine, it should be the same in all machines. |

Usually, it is slow if you do not have high-speed networking like InfiniBand.

#### Multiple machines managed with slurm

If you run MMPose on a cluster managed with [slurm](https://slurm.schedmd.com/), you can use the script `slurm_train.sh`.

```shell
```

Here are the arguments description of the script.

| `PARTITION`   | The partition to use in your cluster.                                              |
| `JOB_NAME`    | The name of your job, you can name it as you like.                                 |
| `WORK_DIR`    | The target folder to save logs and checkpoints.                                    |

Here are the environment variables that can be used to configure the slurm job.

| ENV_VARS        | Description                                                                                                |
| --------------- | ---------------------------------------------------------------------------------------------------------- |

| `GPUS`          | The total number of GPUs to be used. Defaults to 8.                                                        |
| `GPUS_PER_NODE` | The number of GPUs to be allocated per node. Defaults to 8.                                                |
| `CPUS_PER_TASK` | The number of CPUs to be allocated per task (Usually one GPU corresponds to one task). Defaults to 5.      |
| `SRUN_ARGS`     | The other arguments of `srun`. Available options can be found [here](https://slurm.schedmd.com/srun.html). |

## Resume training

Resume training means to continue training from the state saved from one of the previous trainings, where the state includes the model weights, the state of the optimizer and the optimizer parameter adjustment strategy.

### Automatically resume training

Users can add `--resume` to the end of the training command to resume training. The program will automatically load the latest weight file from `work_dirs` to resume training. If there is a latest `checkpoint` in `work_dirs` (e.g. the training was interrupted during the previous training), the training will be resumed from the `checkpoint`. Otherwise (e.g. the previous training did not save `checkpoint` in time or a new training task was started), the training will be restarted.

Here is an example of resuming training:

```shell
python tools/train.py configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_res50_8xb64-210e_coco-256x192.py --resume
```

### Specify the checkpoint to resume training

You can also specify the `checkpoint` path for `--resume`. MMPose will automatically read the `checkpoint` and resume training from it. The command is as follows:

```shell

```shell
python tools/train.py configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_res50_8xb64-210e_coco-256x192.py \
    --resume work_dirs/td-hm_res50_8xb64-210e_coco-256x192/latest.pth
```

If you hope to manually specify the `checkpoint` path in the config file, in addition to setting `resume=True`, you also need to set the `load_from`.

It should be noted that if only `load_from` is set without setting `resume=True`, only the weights in the `checkpoint` will be loaded and the training will be restarted from scratch, instead of continuing from the previous state.

The following example is equivalent to the example above that specifies the `--resume` parameter:

```python
resume = True
load_from = 'work_dirs/td-hm_res50_8xb64-210e_coco-256x192/latest.pth'
    ## omitted ##
    )
```

```python
    )
```

## Freeze partial parameters during training

In some scenarios, it might be desirable to freeze certain parameters of a model during training to fine-tune specific parts or to prevent overfitting. In MMPose, you can set different hyperparameters for any module in the model by setting custom_keys in `paramwise_cfg`. This allows you to control the learning rate and decay coefficient for specific parts of the model.

For example, if you want to freeze the parameters in `backbone.layer0` and `backbone.layer1`, you can modify the optimizer wrapper in the config file as:

```python
    optimizer=dict(...),
            'backbone.layer0': dict(lr_mult=0, decay_mult=0),
        }))
```

        }))
```

This configuration will freeze the parameters in `backbone.layer0` and `backbone.layer1` by setting their learning rate and decay coefficient to 0. By using this approach, you can effectively control the training process and fine-tune specific parts of your model as needed.

## Automatic Mixed Precision (AMP) training

Mixed precision training can reduce training time and storage requirements without changing the model or reducing the model training accuracy, thus supporting larger batch sizes, larger models, and larger input sizes.

To enable Automatic Mixing Precision (AMP) training, add `--amp` to the end of the training command, which is as follows:

```shell
python tools/train.py ${CONFIG_FILE} --amp
```

Specific examples are as follows:

```shell

```shell
```

```shell
python tools/train.py configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_res50_8xb64-210e_coco-256x192.py  --amp
```

## Set the random seed

If you want to specify the random seed during training, you can use the following command:

```shell
python ./tools/train.py \
    ${CONFIG} \                               # config file
    --cfg-options randomness.seed=2023 \      # set the random seed = 2023
    [randomness.diff_rank_seed=True] \        # Set different seeds according to rank.
    [randomness.deterministic=True]           # Set the cuDNN backend deterministic option to True
# `[]` stands for optional parameters, when actually entering the command line, you do not need to enter `[]`
```

`randomness` has three parameters that can be set, with the following meanings.

- `randomness.seed=2023`, set the random seed to `2023`.

```

- `randomness.diff_rank_seed=True`, set different seeds according to global `rank`. Defaults to `False`.

- `randomness.deterministic=True`, set the deterministic option for `cuDNN` backend, i.e., set `torch.backends.cudnn.deterministic` to `True` and `torch.backends.cudnn.benchmark` to `False`. Defaults to `False`. See [Pytorch Randomness](https://pytorch.org/docs/stable/notes/randomness.html) for more details.

## Visualize training process

Monitoring the training process is essential for understanding the performance of your model and making necessary adjustments. In this section, we will introduce two methods to visualize the training process of your MMPose model: TensorBoard and the MMEngine Visualizer.

### TensorBoard

TensorBoard is a powerful tool that allows you to visualize the changes in losses during training. To enable TensorBoard visualization, you may need to:

1. Install TensorBoard environment

   ```shell
   pip install tensorboard
   ```

2. Enable TensorBoard in the config file

   ```python
   visualizer = dict(vis_backends=[
       dict(type='LocalVisBackend'),
       dict(type='TensorboardVisBackend'),
   ])
   ```

The event file generated by TensorBoard will be save under the experiment log folder `${WORK_DIR}`, which defaults to `work_dir/${CONFIG}` or can be specified using the `--work-dir` option. To visualize the training process, use the following command:

```shell
tensorboard --logdir ${WORK_DIR}/${TIMESTAMP}/vis_data
```

### MMEngine visualizer

```shell
```

MMPose also supports visualizing model inference results during validation. To activate this function, please use the `--show` option or set `--show-dir` when launching training. This feature provides an effective way to analyze the model's performance on specific examples and make any necessary adjustments.

## Test your model

### Test with your PC

You can use `tools/test.py` to test a model on a single machine with a CPU and optionally a GPU.

```shell
python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [ARGS]
```

```{note}
By default, MMPose prefers GPU to CPU. If you want to test a model on CPU, please empty `CUDA_VISIBLE_DEVICES` or set it to -1 to make GPU invisible to the program.

```

```shell
CUDA_VISIBLE_DEVICES=-1 python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [ARGS]
```

```

```shell
```

| `CHECKPOINT_FILE`                     | The path to the checkpoint file (It can be a http link, and you can find checkpoints [here](https://MMPose.readthedocs.io/en/latest/model_zoo.html)).               |

| `--work-dir WORK_DIR`                 | The directory to save the file containing evaluation metrics.                                                                                                       |
| `--out OUT`                           | The path to save the file containing evaluation metrics.                                                                                                            |
| `--dump DUMP`                         | The path to dump all outputs of the model for offline evaluation.                                                                                                   |

| `--cfg-options CFG_OPTIONS`           | Override some settings in the used config, the key-value pair in xxx=yyy format will be merged into the config file. If the value to be overwritten is a list, it should be of the form of either `key="[a,b]"` or `key=a,b`. The argument also allows nested list/tuple values, e.g. `key="[(a,b),(c,d)]"`. Note that quotation marks are necessary and that no white space is allowed. |
| `--show-dir SHOW_DIR`                 | The directory to save the result visualization images.                                                                                                              |

### Test with multiple GPUs

```shell
bash ./tools/dist_test.sh ${CONFIG_FILE} ${CHECKPOINT_FILE} ${GPU_NUM} [PY_ARGS]
```

```shell
```

| ARGS              | Description                                                                                                                                           |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `CONFIG_FILE`     | The path to the config file.                                                                                                                          |
| `CHECKPOINT_FILE` | The path to the checkpoint file (It can be a http link, and you can find checkpoints [here](https://mmpose.readthedocs.io/en/latest/model_zoo.html)). |

| `GPU_NUM`         | The number of GPUs to be used.                                                                                                                        |
| `[PYARGS]`        | The other optional arguments of `tools/test.py`, see [here](#test-with-your-pc).                                                                      |

```shell
PORT=29666 bash ./tools/dist_test.sh ${CONFIG_FILE} ${CHECKPOINT_FILE} ${GPU_NUM} [PY_ARGS]
```

If you want to startup multiple test jobs and use different GPUs, you can launch them by specifying

```shell

```

```shell
CUDA_VISIBLE_DEVICES=0,1,2,3 PORT=29500 bash ./tools/dist_test.sh ${CONFIG_FILE1} ${CHECKPOINT_FILE} 4 [PY_ARGS]
CUDA_VISIBLE_DEVICES=4,5,6,7 PORT=29501 bash ./tools/dist_test.sh ${CONFIG_FILE2} ${CHECKPOINT_FILE} 4 [PY_ARGS]
```

### Test with multiple machines

If you launch a test job with multiple machines connected with ethernet, you can run the following commands:

```shell
NNODES=2 NODE_RANK=0 PORT=$MASTER_PORT MASTER_ADDR=$MASTER_ADDR bash tools/dist_test.sh $CONFIG $CHECKPOINT_FILE $GPUS
```

```shell
NNODES=2 NODE_RANK=1 PORT=$MASTER_PORT MASTER_ADDR=$MASTER_ADDR bash tools/dist_test.sh $CONFIG $CHECKPOINT_FILE $GPUS
```

```

If you run MMPose on a cluster managed with [slurm](https://slurm.schedmd.com/), you can use the script `slurm_test.sh`.

```shell

```shell
[ENV_VARS] ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} ${CONFIG_FILE} ${CHECKPOINT_FILE} [PY_ARGS]
```

Here are the argument descriptions of the script.

| `PARTITION`       | The partition to use in your cluster.                                                                                                                 |

| `JOB_NAME`        | The name of your job, you can name it as you like.                                                                                                    |
| `CHECKPOINT_FILE` | The path to the checkpoint file (It can be a http link, and you can find checkpoints [here](https://MMPose.readthedocs.io/en/latest/model_zoo.html)). |

# Contributor Covenant Code of Conduct

## Our Pledge

In the interest of fostering an open and welcoming environment, we as
contributors and maintainers pledge to making participation in our project and
our community a harassment-free experience for everyone, regardless of age, body
size, disability, ethnicity, sex characteristics, gender identity and expression,
level of experience, education, socio-economic status, nationality, personal
appearance, race, religion, or sexual identity and orientation.

## Our Standards

Examples of behavior that contributes to creating a positive environment
include:

- Using welcoming and inclusive language
- Being respectful of differing viewpoints and experiences
- Gracefully accepting constructive criticism
- Focusing on what is best for the community
- Showing empathy towards other community members

Examples of unacceptable behavior by participants include:

- The use of sexualized language or imagery and unwelcome sexual attention or
  advances

  advances
- Trolling, insulting/derogatory comments, and personal or political attacks
- Public or private harassment
- Publishing others' private information, such as a physical or electronic
  address, without explicit permission
- Other conduct which could reasonably be considered inappropriate in a
  professional setting

## Our Responsibilities

Project maintainers are responsible for clarifying the standards of acceptable
behavior and are expected to take appropriate and fair corrective action in
response to any instances of unacceptable behavior.

Project maintainers have the right and responsibility to remove, edit, or
reject comments, commits, code, wiki edits, issues, and other contributions
that are not aligned to this Code of Conduct, or to ban temporarily or

permanently any contributor for other behaviors that they deem inappropriate,
threatening, offensive, or harmful.

## Scope

This Code of Conduct applies both within project spaces and in public spaces
when an individual is representing the project or its community. Examples of
representing a project or community include using an official project e-mail
address, posting via an official social media account, or acting as an appointed
representative at an online or offline event. Representation of a project may be
further defined and clarified by project maintainers.

## Enforcement

Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported by contacting the project team at chenkaidev@gmail.com. All
complaints will be reviewed and investigated and will result in a response that

is deemed necessary and appropriate to the circumstances. The project team is
obligated to maintain confidentiality with regard to the reporter of an incident.
Further details of specific enforcement policies may be posted separately.

Project maintainers who do not follow or enforce the Code of Conduct in good
faith may face temporary or permanent repercussions as determined by other
members of the project's leadership.

## Attribution

This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html

For answers to common questions about this code of conduct, see
https://www.contributor-covenant.org/faq

[homepage]: https://www.contributor-covenant.org

# Pipeline 速度分析 (profiler)

sdk 提供 pipeline 各模块耗时统计功能，默认关闭，若要使用该功能，需要两个步骤：

- 生成性能数据
- 分析性能数据

## 生成性能数据

以 C 接口，分类 pipeline 为例。在创建 pipeline 时需要使用带有 context 信息的接口，并在 context 中加入 profiler 信息。 详细代码如下。 正常运行 demo 会在当前目录生成 profiler 数据 `profiler_data.txt`。

```c++
#include <fstream>
#include <opencv2/imgcodecs/imgcodecs.hpp>
#include <string>

#include "mmdeploy/classifier.h"

int main(int argc, char* argv[]) {
  if (argc != 4) {
    fprintf(stderr, "usage:\n  image_classification device_name dump_model_directory image_path\n");
    return 1;
  }
  auto device_name = argv[1];
  auto model_path = argv[2];
  auto image_path = argv[3];
  cv::Mat img = cv::imread(image_path);
  if (!img.data) {
    fprintf(stderr, "failed to load image: %s\n", image_path);
    return 1;
  }

  mmdeploy_model_t model{};
  mmdeploy_model_create_by_path(model_path, &model);

  // create profiler and add it to context
  // profiler data will save to profiler_data.txt
  mmdeploy_profiler_t profiler{};

mmdeploy_model_create_by_path(model_path, &model);

  mmdeploy_profiler_create("profiler_data.txt", &profiler);

  mmdeploy_context_t context{};
  mmdeploy_context_create_by_device(device_name, 0, &context);
  mmdeploy_context_add(context, MMDEPLOY_TYPE_PROFILER, nullptr, profiler);

  mmdeploy_classifier_t classifier{};
  int status{};
  status = mmdeploy_classifier_create_v2(model, context, &classifier);
  if (status != MMDEPLOY_SUCCESS) {
    fprintf(stderr, "failed to create classifier, code: %d\n", (int)status);
    return 1;
  }

  mmdeploy_mat_t mat{
      img.data, img.rows, img.cols, 3, MMDEPLOY_PIXEL_FORMAT_BGR, MMDEPLOY_DATA_TYPE_UINT8};

  // inference loop
  for (int i = 0; i < 100; i++) {
    mmdeploy_classification_t* res{};
    int* res_count{};
    status = mmdeploy_classifier_apply(classifier, &mat, 1, &res, &res_count);

// inference loop

    mmdeploy_classifier_release_result(res, res_count, 1);
  }

  mmdeploy_classifier_destroy(classifier);

  mmdeploy_model_destroy(model);
  mmdeploy_profiler_destroy(profiler);
  mmdeploy_context_destroy(context);

  return 0;
}

```

## 分析性能数据

使用脚本可对性能数据进行解析。

```bash
python tools/sdk_analyze.py profiler_data.txt
```

解析结果如下，其中 name 表示节点的名称，n_call表示调用的次数，t_mean 表示平均耗时，t_50% t_90% 表示耗时的百分位数。

```bash
+---------------------------+--------+-------+--------+--------+-------+-------+
|           name            | occupy | usage | n_call | t_mean | t_50% | t_90% |
+===========================+========+=======+========+========+=======+=======+
| ./Pipeline                | -      | -     | 100    | 4.831  | 1.913 | 1.946 |

|     Preprocess/Compose    | -      | -     | 100    | 0.125  | 0.118 | 0.144 |
|         LoadImageFromFile | 0.017  | 0.017 | 100    | 0.081  | 0.077 | 0.098 |
|         Resize            | 0.003  | 0.003 | 100    | 0.012  | 0.012 | 0.013 |
|         CenterCrop        | 0.002  | 0.002 | 100    | 0.008  | 0.008 | 0.008 |
|         Normalize         | 0.002  | 0.002 | 100    | 0.009  | 0.009 | 0.009 |

|         ImageToTensor     | 0.002  | 0.002 | 100    | 0.008  | 0.007 | 0.007 |
|         Collect           | 0.001  | 0.001 | 100    | 0.005  | 0.005 | 0.005 |
|     resnet                | 0.968  | 0.968 | 100    | 4.678  | 1.767 | 1.774 |
|     postprocess           | 0.003  | 0.003 | 100    | 0.015  | 0.015 | 0.017 |
```

## TensorRT Ops

- [TensorRT Ops](#tensorrt-ops)
  - [TRTBatchedNMS](#trtbatchednms)
    - [Description](#description)
    - [Parameters](#parameters)
    - [Inputs](#inputs)
    - [Outputs](#outputs)
    - [Type Constraints](#type-constraints)
  - [grid_sampler](#grid_sampler)
    - [Description](#description-1)
    - [Parameters](#parameters-1)
    - [Inputs](#inputs-1)
    - [Outputs](#outputs-1)
    - [Type Constraints](#type-constraints-1)
  - [MMCVInstanceNormalization](#mmcvinstancenormalization)
    - [Description](#description-2)
    - [Parameters](#parameters-2)
    - [Inputs](#inputs-2)
    - [Outputs](#outputs-2)
    - [Type Constraints](#type-constraints-2)
  - [MMCVModulatedDeformConv2d](#mmcvmodulateddeformconv2d)
    - [Description](#description-3)
    - [Parameters](#parameters-3)
    - [Inputs](#inputs-3)
    - [Outputs](#outputs-3)
    - [Type Constraints](#type-constraints-3)
  - [MMCVMultiLevelRoiAlign](#mmcvmultilevelroialign)

- [Parameters](#parameters-3)
    - [Description](#description-4)
    - [Parameters](#parameters-4)
    - [Inputs](#inputs-4)
    - [Outputs](#outputs-4)
    - [Type Constraints](#type-constraints-4)
  - [MMCVRoIAlign](#mmcvroialign)
    - [Description](#description-5)
    - [Parameters](#parameters-5)
    - [Inputs](#inputs-5)
    - [Outputs](#outputs-5)
    - [Type Constraints](#type-constraints-5)
  - [ScatterND](#scatternd)
    - [Description](#description-6)
    - [Parameters](#parameters-6)
    - [Inputs](#inputs-6)
    - [Outputs](#outputs-6)
    - [Type Constraints](#type-constraints-6)
  - [TRTBatchedRotatedNMS](#trtbatchedrotatednms)
    - [Description](#description-7)
    - [Parameters](#parameters-7)
    - [Inputs](#inputs-7)
    - [Outputs](#outputs-7)
    - [Type Constraints](#type-constraints-7)

- [Description](#description-7)
  - [GridPriorsTRT](#gridpriorstrt)
    - [Description](#description-8)
    - [Parameters](#parameters-8)
    - [Inputs](#inputs-8)
    - [Outputs](#outputs-8)
    - [Type Constraints](#type-constraints-8)
  - [ScaledDotProductAttentionTRT](#scaleddotproductattentiontrt)
    - [Description](#description-9)
    - [Parameters](#parameters-9)
    - [Inputs](#inputs-9)
    - [Outputs](#outputs-9)
    - [Type Constraints](#type-constraints-9)
  - [GatherTopk](#gathertopk)
    - [Description](#description-10)
    - [Parameters](#parameters-10)
    - [Inputs](#inputs-10)
    - [Outputs](#outputs-10)
    - [Type Constraints](#type-constraints-10)
  - [MMCVMultiScaleDeformableAttention](#mmcvmultiscaledeformableattention)
    - [Description](#description-11)
    - [Parameters](#parameters-11)
    - [Inputs](#inputs-11)

- [MMCVMultiScaleDeformableAttention](#mmcvmultiscaledeformableattention)
    - [Outputs](#outputs-11)
    - [Type Constraints](#type-constraints-11)

### TRTBatchedNMS

#### Description

Batched NMS with a fixed number of output bounding boxes.

#### Parameters

| Type    | Parameter             | Description                                                                                                                             |
| ------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `int`   | `background_label_id` | The label ID for the background class. If there is no background class, set it to `-1`.                                                 |

| `int`   | `num_classes`         | The number of classes.                                                                                                                  |
| `int`   | `topK`                | The number of bounding boxes to be fed into the NMS step.                                                                               |
| `int`   | `keepTopK`            | The number of total bounding boxes to be kept per-image after the NMS step. Should be less than or equal to the `topK` value.           |
| `float` | `scoreThreshold`      | The scalar threshold for score (low scoring boxes are removed).                                                                         |

| `float` | `iouThreshold`        | The scalar threshold for IoU (new boxes that have high IoU overlap with previously selected boxes are removed).                         |
| `int`   | `isNormalized`        | Set to `false` if the box coordinates are not normalized, meaning they are not in the range `[0,1]`. Defaults to `true`.                |
| `int`   | `clipBoxes`           | Forcibly restrict bounding boxes to the normalized range `[0,1]`. Only applicable if `isNormalized` is also `true`. Defaults to `true`. |

#### Inputs

<dl>
<dt><tt>inputs[0]</tt>: T</dt>
<dd>boxes; 4-D tensor of shape (N, num_boxes, num_classes, 4), where N is the batch size; `num_boxes` is the number of boxes; `num_classes` is the number of classes, which could be 1 if the boxes are shared between all classes.</dd>

<dt><tt>inputs[1]</tt>: T</dt>
<dd>scores; 4-D tensor of shape (N, num_boxes, 1, num_classes). </dd>
</dl>

#### Outputs

<dl>
<dt><tt>outputs[0]</tt>: T</dt>
<dd>dets; 3-D tensor of shape (N, valid_num_boxes, 5), `valid_num_boxes` is the number of boxes after NMS. For each row `dets[i,j,:] = [x0, y0, x1, y1, score]`</dd>
<dt><tt>outputs[1]</tt>: tensor(int32, Linear)</dt>
<dd>labels; 2-D tensor of shape (N, valid_num_boxes). </dd>
</dl>

#### Type Constraints

- T:tensor(float32, Linear)

### grid_sampler

Perform sample from `input` with pixel locations from `grid`.

| Type  | Parameter            | Description                                                                                                                                                                                                                                                                                     |

| ----- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `int` | `interpolation_mode` | Interpolation mode to calculate output values. (0: `bilinear` , 1: `nearest`)                                                                                                                                                                                                                   |
| `int` | `padding_mode`       | Padding mode for outside grid values. (0: `zeros`, 1: `border`, 2: `reflection`)                                                                                                                                                                                                                |

| `int` | `align_corners`      | If `align_corners=1`, the extrema (`-1` and `1`) are considered as referring to the center points of the input's corner pixels. If `align_corners=0`, they are instead considered as referring to the corner points of the input's corner pixels, making the sampling more resolution agnostic. |

<dl>
<dd>Input feature; 4-D tensor of shape (N, C, inH, inW), where N is the batch size, C is the numbers of channels, inH and inW are the height and width of the data.</dd>
<dd>Input offset; 4-D tensor of shape (N, outH, outW, 2), where outH and outW are the height and width of offset and output. </dd>
</dl>

<dl>
<dd>Output feature; 4-D tensor of shape (N, C, outH, outW).</dd>
</dl>

### MMCVInstanceNormalization

</dl>

Carry out instance normalization as described in the paper https://arxiv.org/abs/1607.08022.

y = scale * (x - mean) / sqrt(variance + epsilon) + B, where mean and variance are computed per instance per channel.

| Type    | Parameter | Description                                                          |
| ------- | --------- | -------------------------------------------------------------------- |
| `float` | `epsilon` | The epsilon value to use to avoid division by zero. Default is 1e-05 |

<dl>
<dt><tt>input</tt>: T</dt>

<dl>
<dd>Input data tensor from the previous operator; dimensions for image case are (N x C x H x W), where N is the batch size, C is the number of channels, and H and W are the height and the width of the data. For non image case, the dimensions are in the form of (N x C x D1 x D2 ... Dn), where N is the batch size.</dd>
<dt><tt>scale</tt>: T</dt>
<dd>The input 1-dimensional scale tensor of size C.</dd>
<dt><tt>B</tt>: T</dt>
<dd>The input 1-dimensional bias tensor of size C.</dd>
</dl>

<dl>
<dt><tt>output</tt>: T</dt>
<dd>The output tensor of the same shape as input.</dd>
</dl>

### MMCVModulatedDeformConv2d

<dl>
</dl>

Perform Modulated Deformable Convolution on input feature. Read [Deformable ConvNets v2: More Deformable, Better Results](https://arxiv.org/abs/1811.11168?from=timeline) for detail.

| Type           | Parameter          | Description                                                                           |
| -------------- | ------------------ | ------------------------------------------------------------------------------------- |
| `list of ints` | `stride`           | The stride of the convolving kernel. (sH, sW)                                         |
| `list of ints` | `padding`          | Paddings on both sides of the input. (padH, padW)                                     |

| `list of ints` | `dilation`         | The spacing between kernel elements. (dH, dW)                                         |
| `int`          | `deformable_group` | Groups of deformable offset.                                                          |
| `int`          | `group`            | Split input into groups. `input_channel` should be divisible by the number of groups. |

<dl>
<dd>Input feature; 4-D tensor of shape (N, C, inH, inW), where N is the batch size, C is the number of channels, inH and inW are the height and width of the data.</dd>
<dd>Input offset; 4-D tensor of shape (N, deformable_group* 2* kH* kW, outH, outW), where kH and kW are the height and width of weight, outH and outW are the height and width of offset and output.</dd>
<dt><tt>inputs[2]</tt>: T</dt>

<dd>Input mask; 4-D tensor of shape (N, deformable_group* kH* kW, outH, outW), where kH and kW are the height and width of weight, outH and outW are the height and width of offset and output.</dd>
<dt><tt>inputs[3]</tt>: T</dt>
<dd>Input weight; 4-D tensor of shape (output_channel, input_channel, kH, kW).</dd>
<dt><tt>inputs[4]</tt>: T, optional</dt>
<dd>Input weight; 1-D tensor of shape (output_channel).</dd>
</dl>

<dl>
<dd>Output feature; 4-D tensor of shape (N, output_channel, outH, outW).</dd>
</dl>

### MMCVMultiLevelRoiAlign

Perform RoIAlign on features from multiple levels. Used in bbox_head of most two-stage detectors.

| Type             | Parameter          | Description                                                                                                   |

| ---------------- | ------------------ | ------------------------------------------------------------------------------------------------------------- |
| `int`            | `output_height`    | height of output roi.                                                                                         |
| `int`            | `output_width`     | width of output roi.                                                                                          |
| `list of floats` | `featmap_strides`  | feature map stride of each level.                                                                             |
| `int`            | `sampling_ratio`   | number of input samples to take for each output sample. `0` means to take samples densely for current models. |

| `float`          | `roi_scale_factor` | RoIs will be scaled by this factor before RoI Align.                                                          |
| `int`            | `finest_scale`     | Scale threshold of mapping to level 0. Default: 56.                                                           |
| `int`            | `aligned`          | If `aligned=0`, use the legacy implementation in MMDetection. Else, align the results more perfectly.         |

<dd>RoIs (Regions of Interest) to pool over; 2-D tensor of shape (num_rois, 5) given as [[batch_index, x1, y1, x2, y2], ...].</dd>
<dt><tt>inputs[1~]</tt>: T</dt>
<dd>Input feature map; 4D tensor of shape (N, C, H, W), where N is the batch size, C is the numbers of channels, H and W are the height and width of the data.</dd>

<dl>
<dd>RoI pooled output, 4-D tensor of shape (num_rois, C, output_height, output_width). The r-th batch element output[0][r-1] is a pooled feature map corresponding to the r-th RoI inputs[1][r-1].<dd>
</dl>

### MMCVRoIAlign

Perform RoIAlign on output feature, used in bbox_head of most two-stage detectors.

| Type    | Parameter        | Description                                                                                                   |
| ------- | ---------------- | ------------------------------------------------------------------------------------------------------------- |

| `int`   | `output_height`  | height of output roi                                                                                          |
| `int`   | `output_width`   | width of output roi                                                                                           |
| `float` | `spatial_scale`  | used to scale the input boxes                                                                                 |
| `int`   | `sampling_ratio` | number of input samples to take for each output sample. `0` means to take samples densely for current models. |
| `str`   | `mode`           | pooling mode in each bin. `avg` or `max`                                                                      |
| `int`   | `aligned`        | If `aligned=0`, use the legacy implementation in MMDetection. Else, align the results more perfectly.         |

<dl>
<dd>RoIs (Regions of Interest) to pool over; 2-D tensor of shape (num_rois, 5) given as [[batch_index, x1, y1, x2, y2], ...]. The RoIs' coordinates are the coordinate system of inputs[0].</dd>
</dl>

<dl>
</dl>

### ScatterND

</dl>

ScatterND takes three inputs `data` tensor of rank r >= 1, `indices` tensor of rank q >= 1, and `updates` tensor of rank q + r - indices.shape[-1] - 1. The output of the operation is produced by creating a copy of the input `data`, and then updating its value to values specified by updates at specific index positions specified by `indices`. Its output shape is the same as the shape of `data`. Note that `indices` should not have duplicate entries. That is, two or more updates for the same index-location is not supported.

The `output` is calculated via the following equation:

```python
  output = np.copy(data)
  update_indices = indices.shape[:-1]
  for idx in np.ndindex(update_indices):
      output[indices[idx]] = updates[idx]
```

None

<dl>
<dd>Tensor of rank r>=1.</dd>

<dt><tt>inputs[1]</tt>: tensor(int32, Linear)</dt>

output[indices[idx]] = updates[idx]
```

None

<dl>

<dd>Tensor of rank q>=1.</dd>

<dd>Tensor of rank q + r - indices_shape[-1] - 1.</dd>
</dl>

<dl>
<dd>Tensor of rank r >= 1.</dd>
</dl>

- T:tensor(float32, Linear), tensor(int32, Linear)

### TRTBatchedRotatedNMS

Batched rotated NMS with a fixed number of output bounding boxes.

<dl>

<dl>
<dd>boxes; 4-D tensor of shape (N, num_boxes, num_classes, 5), where N is the batch size; `num_boxes` is the number of boxes; `num_classes` is the number of classes, which could be 1 if the boxes are shared between all classes.</dd>
</dl>

<dl>
<dd>dets; 3-D tensor of shape (N, valid_num_boxes, 6), `valid_num_boxes` is the number of boxes after NMS. For each row `dets[i,j,:] = [x0, y0, width, height, theta, score]`</dd>
</dl>

### GridPriorsTRT

Generate the anchors for object detection task.

| Type  | Parameter  | Description                       |

| ----- | ---------- | --------------------------------- |
| `int` | `stride_w` | The stride of the feature width.  |
| `int` | `stride_h` | The stride of the feature height. |

<dl>
<dd>The base anchors; 2-D tensor with shape [num_base_anchor, 4].</dd>
<dt><tt>inputs[1]</tt>: TAny</dt>
<dd>height provider; 1-D tensor with shape [featmap_height]. The data will never been used.</dd>
<dt><tt>inputs[2]</tt>: TAny</dt>
<dd>width provider; 1-D tensor with shape [featmap_width]. The data will never been used.</dd>
</dl>

<dl>
<dd>output anchors; 2-D tensor of shape (num_base_anchor*featmap_height*featmap_widht, 4).</dd>
</dl>

- TAny: Any

</dl>

- TAny: Any

### ScaledDotProductAttentionTRT

Dot product attention used to support multihead attention, read [Attention Is All You Need](https://arxiv.org/abs/1706.03762?context=cs) for more detail.

None

<dl>
<dd>query; 3-D tensor with shape [batch_size, sequence_length, embedding_size].</dd>
<dd>key; 3-D tensor with shape [batch_size, sequence_length, embedding_size].</dd>
<dd>value; 3-D tensor with shape [batch_size, sequence_length, embedding_size].</dd>
<dd>mask; 2-D/3-D tensor with shape [sequence_length, sequence_length] or [batch_size, sequence_length, sequence_length]. optional.</dd>
</dl>

<dl>

</dl>

<dl>
<dd>3-D tensor of shape [batch_size, sequence_length, embedding_size]. `softmax(q@k.T)@v`</dd>
<dt><tt>outputs[1]</tt>: T</dt>
<dd>3-D tensor of shape [batch_size, sequence_length, sequence_length]. `softmax(q@k.T)`</dd>
</dl>

### GatherTopk

TensorRT 8.2~8.4 would give unexpected result for multi-index gather.

```python
data[batch_index, bbox_index, ...]
```

Read [this](https://github.com/NVIDIA/TensorRT/issues/2299) for more details.

None

<dl>
<dd>Tensor to be gathered, with shape (A0, ..., An, G0, C0, ...).</dd>

<dd>Tensor of index. with shape (A0, ..., An, G1)</dd>

<dl>
<dd>Tensor of output. With shape (A0, ..., An, G1, C0, ...)</dd>
</dl>

### MMCVMultiScaleDeformableAttention

Perform attention computation over a small set of key sampling points around a reference point rather than looking over all possible spatial locations. Read [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://arxiv.org/abs/2010.04159) for detail.

None

<dl>
<dd>Input feature; 4-D tensor of shape (N, S, M, D), where N is the batch size, S is the length of feature maps, M is the number of attention heads, and D is hidden_dim.</dd>

<dd>Input offset; 2-D tensor of shape (L, 2), L is the number of feature maps, `2` is shape of feature maps.</dd>
<dd>Input mask; 1-D tensor of shape (L, ), this tensor is used to find the sampling locations for different feature levels as the input feature tensors are flattened.</dd>
<dd>Input weight; 6-D tensor of shape (N, Lq, M, L, P, 2). Lq is the length of feature maps(encoder)/length of queries(decoder), P is the number of points</dd>
<dd>Input weight; 5-D tensor of shape (N, Lq, M, L, P).</dd>
</dl>

<dl>
<dd>Output feature; 3-D tensor of shape (N, Lq, M*D).</dd>
</dl>

# 训练引擎

MMEngine 定义了一些[基础循环控制器](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py) 例如基于轮次的训练循环 (`EpochBasedTrainLoop`), 基于迭代次数的训练循环 (`IterBasedTrainLoop`), 标准的验证循环 (`ValLoop`) 和标准的测试循环 (`TestLoop`).
OpenMMLab 的算法库如 MMSegmentation 将模型训练, 测试和推理抽象为执行器(`Runner`) 来处理. 用户可以直接使用 MMEngine 中的默认执行器, 也可以对执行器进行修改以满足定制化需求. 这个文档主要介绍用户如何配置已有的运行设定, 钩子和优化器的基本概念与使用方法.

## 配置运行设定

### 配置训练长度

循环控制器指的是训练, 验证和测试时的执行流程, 在配置文件里面使用 `train_cfg`, `val_cfg` 和 `test_cfg` 来构建这些流程. MMSegmentation 在 `configs/_base_/schedules` 文件夹里面的 `train_cfg` 设置常用的训练长度.
例如, 使用基于迭代次数的训练循环 (`IterBasedTrainLoop`) 去训练 80,000 个迭代次数, 并且每 8,000 iteration 做一次验证, 可以如下设置:

```python
```

### 配置训练优化器

这里是一个 SGD 优化器的例子:

```python
```

```

```python
```

OpenMMLab 支持 PyTorch 里面所有的优化器, 更多细节可以参考 MMEngine [优化器文档](https://github.com/open-mmlab/mmengine/blob/main/docs/zh_cn/tutorials/optim_wrapper.md).

需要强调的是, `optim_wrapper` 是 `runner` 的变量, 所以需要配置优化器时配置的字段是 `optim_wrapper` 字段. 更多关于优化器的使用方法, 可以看下面优化器的章节.

### 配置训练参数调度器

在配置训练参数调度器前, 推荐先了解 [MMEngine 文档](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md) 里面关于参数调度器的基本概念.

以下是一个参数调度器的例子, 训练时前 1,000 个 iteration 时采用线性变化的学习率策略作为训练预热, 从 1,000 iteration 之后直到最后 16,000 个 iteration 时则采用默认的多项式学习率衰减:

```python
    dict(
    )
]
```

dict(
    )
]
```

注意: 当修改 `train_cfg` 里面 `max_iters` 的时候, 请确保参数调度器 `param_scheduler` 里面的参数也被同时修改.

## 钩子 (Hook)

### 介绍

OpenMMLab 将模型训练和测试过程抽象为 `Runner`, 插入钩子可以实现在 `Runner` 中不同的训练和测试节点 (例如 "每个训练 iter 前后", "每个验证 iter 前后" 等不同阶段) 所需要的相应功能. 更多钩子机制的介绍可以参考[这里](https://www.calltutors.com/blog/what-is-hook).

`Runner` 中所使用的钩子分为两类:

- 默认钩子 (default hooks)

它们实现了训练时所必需的功能, 在配置文件中用 `default_hooks` 定义传给 `Runner`, `Runner` 通过 [`register_default_hooks`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/runner.py#L1780) 方法注册.
钩子有对应的优先级, 优先级越高, 越早被执行器调用. 如果优先级一样, 被调用的顺序和钩子注册的顺序一致.
不建议用户修改默认钩子的优先级, 可以参考 [mmengine hooks 文档](https://github.com/open-mmlab/mmengine/blob/main/docs/zh_cn/tutorials/hook.md) 了解钩子优先级的定义.
下面是 MMSegmentation 中所用到的默认钩子：

|                                                          钩子                                                          |                                              功能                                               |      优先级       |
| :--------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------: | :---------------: |
|          [IterTimerHook](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/iter_timer_hook.py)           |                                   记录 iteration 花费的时间.                                    |    NORMAL (50)    |

|              [LoggerHook](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/logger_hook.py)              | 从 `Runner` 里不同的组件中收集日志记录, 并将其输出到终端, JSON 文件, tensorboard, wandb 等下游. | BELOW_NORMAL (60) |
|     [ParamSchedulerHook](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/param_scheduler_hook.py)      |                          更新优化器里面的一些超参数, 例如学习率的动量.                          |     LOW (70)      |
|          [CheckpointHook](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py)          |                                  规律性地保存 checkpoint 文件.                                  |   VERY_LOW (90)   |
|      [DistSamplerSeedHook](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/sampler_seed_hook.py)       |                               确保分布式采样器 shuffle 是打开的.                                |    NORMAL (50)    |

| [SegVisualizationHook](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/visualization/local_visualizer.py) |                                可视化验证和测试过程里的预测结果.                                |    NORMAL (50)    |

MMSegmentation 会在 [`defualt_hooks`](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/_base_/schedules/schedule_160k.py#L19-L25) 里面注册一些训练所必需功能的钩子::

```python
```

以上默认钩子除 `SegVisualizationHook` 外都是在 MMEngine 中所实现, `SegVisualizationHook` 是在 MMSegmentation 里被实现的钩子, 之后会专门介绍.

- 修改默认的钩子

以 `default_hooks` 里面的 `logger` 和 `checkpoint` 为例, 我们来介绍如何修改 `default_hooks` 中默认的钩子.

(1) 模型保存配置

- 修改默认的钩子

(1) 模型保存配置

`default_hooks` 使用 `checkpoint` 字段来初始化[模型保存钩子 (CheckpointHook)](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L19).

```python
```

用户可以设置 `max_keep_ckpts` 来只保存少量的检查点或者用 `save_optimizer` 来决定是否保存 optimizer 的信息.
更多相关参数的细节可以参考[这里](https://mmengine.readthedocs.io/zh_CN/latest/api/generated/mmengine.hooks.CheckpointHook.html#checkpointhook).

(2) 日志配置

`日志钩子 (LoggerHook)` 被用来收集 `执行器 (Runner)` 里面不同组件的日志信息然后写入终端, JSON 文件, tensorboard 和 wandb 等地方.

```python
```

在最新的 1.x 版本的 MMSegmentation 里面, 一些日志钩子 (LoggerHook) 例如 `TextLoggerHook`, `WandbLoggerHook` 和 `TensorboardLoggerHook` 将不再被使用.
作为替代, MMEngine 使用 `LogProcessor` 来处理上述钩子处理的信息, 它们现在在 [`MessageHub`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/logging/message_hub.py#L17),

[`WandbVisBackend`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/visualization/vis_backend.py#L324) 和 [`TensorboardVisBackend`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/visualization/vis_backend.py#L472) 里面.

具体使用方法如下, 配置可视化器和同时指定可视化后端, 这里使用 Tensorboard 作为可视化器的后端:

```python
```

关于更多相关用法, 可以参考 [MMEngine 可视化后端用户教程](https://github.com/open-mmlab/mmengine/blob/main/docs/zh_cn/advanced_tutorials/visualization.md).

- 自定义钩子 (custom hooks)

自定义钩子在配置通过 `custom_hooks` 定义, `Runner` 通过 [`register_custom_hooks`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/runner.py#L1820) 方法注册.
自定义钩子优先级需要在配置文件里设置, 如果没有设置, 则会被默认设置为 `NORMAL`. 下面是部分 MMEngine 中实现的自定义钩子:

|                                                  钩子                                                  |                                             用法                                             |
| :----------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------: |
|         [EMAHook](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/ema_hook.py)         |               在模型训练时使用指数滑动平均 (Exponential Moving Average, EMA).                |
| [EmptyCacheHook](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/empty_cache_hook.py)  |                          在训练时释放所有没有被缓存占用的 GPU 显存.                          |

| [SyncBuffersHook](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/sync_buffer_hook.py) | 在每个训练 Epoch 结束时同步模型 buffer 里的参数例如 BN 里的 `running_mean` 和 `running_var`. |

以下是 `EMAHook` 的用例, 配置文件中, 将已经实现的自定义钩子的配置作为 `custom_hooks` 列表中的成员.

```python
]
```

MMSegmentation 实现了 [`SegVisualizationHook`](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/engine/hooks/visualization_hook.py#L17), 用来在验证和测试时可视化预测结果.
`SegVisualizationHook` 重写了基类 `Hook` 中的 `_after_iter` 方法, 在验证或测试时, 根据指定的迭代次数间隔调用 `visualizer` 的 `add_datasample` 方法绘制语义分割结果, 具体实现如下:

```python
...
...

```python
...
...
...
        # 如果是训练阶段或者 self.draw 为 False 则直接跳出
            return
...

                    img,

self._visualizer.add_datasample(
                    img,

```

关于可视化更多的细节可以查看[这里](../user_guides/visualization.md).

## 优化器

在上面配置运行设定里, 我们给出了配置训练优化器的简单示例. 本章节将进一步详细介绍在 MMSegmentation 里如何配置优化器.

### 优化器封装

OpenMMLab 2.0 设计了优化器封装, 它支持不同的训练策略, 包括混合精度训练、梯度累加和梯度截断等, 用户可以根据需求选择合适的训练策略.
优化器封装还定义了一套标准的参数更新流程, 用户可以基于这一套流程, 在同一套代码里, 实现不同训练策略的切换. 如果想了解更多, 可以参考 [MMEngine 优化器封装文档](https://github.com/open-mmlab/mmengine/blob/main/docs/zh_cn/tutorials/optim_wrapper.md).

以下是 MMSegmentation 中常用的使用方法:

#### 配置 PyTorch 支持的优化器

OpenMMLab 2.0 支持 PyTorch 原生所有优化器, 参考[这里](https://github.com/open-mmlab/mmengine/blob/main/docs/zh_cn/tutorials/optim_wrapper.md#%E7%AE%80%E5%8D%95%E9%85%8D%E7%BD%AE).

在配置文件中设置训练时 `Runner` 所使用的优化器, 需要定义 `optim_wrapper`, 而不是 `optimizer`, 下面是一个配置训练中优化器的例子:

```python

```python
```

#### 配置梯度裁剪

当模型训练需要使用梯度裁剪的训练技巧式, 可以按照如下示例进行配置:

```python
```

这里 `max_norm` 指的是裁剪后梯度的最大值,  `norm_type` 指的是裁剪梯度时使用的范数. 相关方法可参考 [torch.nn.utils.clip_grad_norm\_](https://pytorch.org/docs/stable/generated/torch.nn.utils.clip_grad_norm_.html).

#### 配置混合精度训练

当需要使用混合精度训练降低内存时, 可以使用 `AmpOptimWrapper`, 具体配置如下:

```python
```

```python
```

[`AmpOptimWrapper`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/optimizer/amp_optimizer_wrapper.py#L20) 中 `loss_scale` 的默认设置是 `dynamic`.

#### 配置模型网络不同层的超参数

在模型训练中, 如果想在优化器里为不同参数分别设置优化策略, 例如设置不同的学习率、权重衰减等超参数, 可以通过设置配置文件里 `optim_wrapper` 中的 `paramwise_cfg` 来实现.

下面的配置文件以 [ViT `optim_wrapper`](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/vit/vit_vit-b16-ln_mln_upernet_8xb2-160k_ade20k-512x512.py#L15-L27) 为例介绍 `paramwise_cfg` 参数使用.
训练时将 `pos_embed`, `mask_token`, `norm` 模块的 weight decay 参数的系数设置成 0.
即: 在训练时, 这些模块的 weight decay 将被变为 `weight_decay * decay_mult`=0.

```python

type='AdamW', lr=0.00006, betas=(0.9, 0.999), weight_decay=0.01)
        }))
```

其中 `decay_mult` 指的是对应参数的权重衰减的系数.
关于更多 `paramwise_cfg` 的使用可以在 [MMEngine 优化器封装文档](https://github.com/open-mmlab/mmengine/blob/main/docs/zh_cn/tutorials/optim_wrapper.md) 里面查到.

### 优化器封装构造器

默认的优化器封装构造器 [`DefaultOptimWrapperConstructor`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/optimizer/default_constructor.py#L19) 根据输入的 `optim_wrapper` 和 `optim_wrapper` 中定义的 `paramwise_cfg` 来构建训练中使用的优化器. 当 [`DefaultOptimWrapperConstructor`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/optimizer/default_constructor.py#L19) 功能不能满足需求时, 可以自定义优化器封装构造器来实现超参数的配置.

MMSegmentation 中的实现了 [`LearningRateDecayOptimizerConstructor`](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/engine/optimizers/layer_decay_optimizer_constructor.py#L104), 可以对以 ConvNeXt, BEiT 和 MAE 为骨干网络的模型训练时, 骨干网络的模型参数的学习率按照定义的衰减比例（`decay_rate`）逐层递减, 在配置文件中的配置如下:

```python
    },
```

`_delete_=True` 的作用是 OpenMMLab Config 中的忽略继承的配置, 在该代码片段中忽略继承的 `optim_wrapper` 配置, 更多 `_delete_` 字段的内容可以参考 [MMEngine 文档](https://github.com/open-mmlab/mmengine/blob/main/docs/zh_cn/advanced_tutorials/config.md#%E5%88%A0%E9%99%A4%E5%AD%97%E5%85%B8%E4%B8%AD%E7%9A%84-key).

# YOLOv3

> [YOLOv3: An Incremental Improvement](https://arxiv.org/abs/1804.02767)

We present some updates to YOLO! We made a bunch of little design changes to make it better. We also trained this new network that's pretty swell. It's a little bigger than last time but more accurate. It's still fast though, don't worry. At 320x320 YOLOv3 runs in 22 ms at 28.2 mAP, as accurate as SSD but three times faster. When we look at the old .5 IOU mAP detection metric YOLOv3 is quite good. It achieves 57.9 mAP@50 in 51 ms on a Titan X, compared to 57.5 mAP@50 in 198 ms by RetinaNet, similar performance but 3.8x faster.

<img src="https://user-images.githubusercontent.com/40661020/144001433-b4f7fb5e-3b7a-414b-b949-93733213b670.png" height="300"/>
</div>

</div>

|  Backbone  | Scale | Lr schd | Mem (GB) | Inf time (fps) | box AP |                     Config                      |                                                                                                                                                        Download                                                                                                                                                        |

| :--------: | :---: | :-----: | :------: | :------------: | :----: | :---------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| DarkNet-53 |  320  |  273e   |   2.7    |      63.9      |  27.9  |  [config](./yolov3_d53_8xb8-320-273e_coco.py)   |                         [model](https://download.openmmlab.com/mmdetection/v2.0/yolo/yolov3_d53_320_273e_coco/yolov3_d53_320_273e_coco-421362b6.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/yolo/yolov3_d53_320_273e_coco/yolov3_d53_320_273e_coco-20200819_172101.log.json)                         |

| DarkNet-53 |  416  |  273e   |   3.8    |      61.2      |  30.9  | [config](./yolov3_d53_8xb8-ms-416-273e_coco.py) |         [model](https://download.openmmlab.com/mmdetection/v2.0/yolo/yolov3_d53_mstrain-416_273e_coco/yolov3_d53_mstrain-416_273e_coco-2b60fcd9.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/yolo/yolov3_d53_mstrain-416_273e_coco/yolov3_d53_mstrain-416_273e_coco-20200819_173424.log.json)         |
| DarkNet-53 |  608  |  273e   |   7.4    |      48.1      |  33.7  | [config](./yolov3_d53_8xb8-ms-608-273e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/yolo/yolov3_d53_mstrain-608_273e_coco/yolov3_d53_mstrain-608_273e_coco_20210518_115020-a2c3acb8.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/yolo/yolov3_d53_mstrain-608_273e_coco/yolov3_d53_mstrain-608_273e_coco_20210518_115020.log.json) |

## Mixed Precision Training

We also train YOLOv3 with mixed precision training.

|  Backbone  | Scale | Lr schd | Mem (GB) | Inf time (fps) | box AP |                       Config                        |                                                                                                                                                                  Download                                                                                                                                                                  |
| :--------: | :---: | :-----: | :------: | :------------: | :----: | :-------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

| DarkNet-53 |  608  |  273e   |   4.7    |      48.1      |  33.8  | [config](./yolov3_d53_8xb8-amp-ms-608-273e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/yolo/yolov3_d53_fp16_mstrain-608_273e_coco/yolov3_d53_fp16_mstrain-608_273e_coco_20210517_213542-4bc34944.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/yolo/yolov3_d53_fp16_mstrain-608_273e_coco/yolov3_d53_fp16_mstrain-608_273e_coco_20210517_213542.log.json) |

## Lightweight models

|  Backbone   | Scale | Lr schd | Mem (GB) | Inf time (fps) | box AP |                          Config                          |                                                                                                                                                                        Download                                                                                                                                                                        |

| :---------: | :---: | :-----: | :------: | :------------: | :----: | :------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| MobileNetV2 |  416  |  300e   |   5.3    |                |  23.9  | [config](./yolov3_mobilenetv2_8xb24-ms-416-300e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/yolo/yolov3_mobilenetv2_mstrain-416_300e_coco/yolov3_mobilenetv2_mstrain-416_300e_coco_20210718_010823-f68a07b3.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/yolo/yolov3_mobilenetv2_mstrain-416_300e_coco/yolov3_mobilenetv2_mstrain-416_300e_coco_20210718_010823.log.json) |

| MobileNetV2 |  320  |  300e   |   3.2    |                |  22.2  |  [config](./yolov3_mobilenetv2_8xb24-320-300e_coco.py)   |                 [model](https://download.openmmlab.com/mmdetection/v2.0/yolo/yolov3_mobilenetv2_320_300e_coco/yolov3_mobilenetv2_320_300e_coco_20210719_215349-d18dff72.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/yolo/yolov3_mobilenetv2_320_300e_coco/yolov3_mobilenetv2_320_300e_coco_20210719_215349.log.json)                 |

Notice: We reduce the number of channels to 96 in both head and neck. It can reduce the flops and parameters, which makes these models more suitable for edge devices.

## Credit

This implementation originates from the project of Haoyu Wu(@wuhy08) at Western Digital.

```latex
@misc{redmon2018yolov3,
    title={YOLOv3: An Incremental Improvement},
    author={Joseph Redmon and Ali Farhadi},
    year={2018},
    eprint={1804.02767},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}
```

# W4A16 LLM 模型部署

LMDeploy 支持 4bit 权重模型的推理，**对 NVIDIA 显卡的最低要求是 sm80**，比如A10，A100，Gerforce 30/40系列。

在推理之前，请确保安装了 lmdeploy

```shell
pip install lmdeploy
```

## 4bit 权重模型推理

你可以直接从 LMDeploy 的 [model zoo](https://huggingface.co/lmdeploy) 下载已经量化好的 4bit 权重模型，直接使用下面的命令推理。也可以根据["4bit 权重量化"](#4bit-权重量化)章节的内容，把 16bit 权重量化为 4bit 权重，然后再按下述说明推理

以 4bit 的 Llama-2-chat-7B 模型为例，可以从 model zoo 直接下载：

```shell
git-lfs install
git clone https://huggingface.co/lmdeploy/llama2-chat-7b-w4
```

执行以下命令，即可在终端与模型对话：

```shell

## 转换模型的layout，存放在默认路径 ./workspace 下
python3 -m lmdeploy.serve.turbomind.deploy \
    --model-name llama2 \
    --model-path ./llama2-chat-7b-w4 \
    --model-format awq \
    --group-size 128

## 推理
python3 -m lmdeploy.turbomind.chat ./workspace
```

## 启动 gradio 服务

如果想通过 webui 与模型对话，请执行以下命令启动 gradio 服务

```shell
python3 -m lmdeploy.serve.turbomind ./workspace --server_name {ip_addr} ----server_port {port}
```

然后，在浏览器中打开 http://{ip_addr}:{port}，即可在线对话

## 推理速度

```shell
```

## 推理速度

我们在 NVIDIA GeForce RTX 4090 上使用 [profile_generation.py](https://github.com/InternLM/lmdeploy/blob/main/benchmark/profile_generation.py)，分别测试了 4-bit Llama-2-7B-chat 和 Llama-2-13B-chat 模型的 token 生成速度。测试配置为 batch size = 1，(prompt_tokens, completion_tokens) = (1, 512)

| model            | llm-awq | mlc-llm | turbomind |
| ---------------- | ------- | ------- | --------- |
| Llama-2-7B-chat  | 112.9   | 159.4   | 206.4     |
| Llama-2-13B-chat | N/A     | 90.7    | 115.8     |

上述两个模型的16bit 和 4bit 权重，分别使用 turbomind 推理时，各自在context size 为 2048 和 4096 配置下，所占的显存对比如下：

| model            | 16bit(2048) | 4bit(2048) | 16bit(4096) | 4bit(4096) |
| ---------------- | ----------- | ---------- | ----------- | ---------- |
| Llama-2-7B-chat  | 15.1        | 6.3        | 16.2        | 7.5        |

| Llama-2-13B-chat | OOM         | 10.3       | OOM         | 12.0       |

```shell
python benchmark/profile_generation.py \
  ./workspace \
  --concurrency 1 --input_seqlen 1 --output_seqlen 512
```

## 4bit 权重量化

4bit 权重量化包括 2 步：

- 生成量化参数
- 根据量化参数，量化模型权重

### 第一步：生成量化参数

```shell
python3 -m lmdeploy.lite.apis.calibrate \
  --model $HF_MODEL \
  --calib_dataset 'c4' \             # 校准数据集，支持 c4, ptb, wikitext2, pileval
  --calib_samples 128 \              # 校准集的样本数，如果显存不够，可以适当调小
  --calib_seqlen 2048 \              # 单条的文本长度，如果显存不够，可以适当调小
  --work_dir $WORK_DIR \             # 保存 Pytorch 格式量化统计参数和量化后权重的文件夹
```

### 第二步：量化权重模型

LMDeploy 使用 AWQ 算法对模型权重进行量化。在执行下面的命令时，需要把步骤1的`$WORK_DIR`传入。量化结束后，权重文件也会存放在这个目录中。然后就可以根据 ["4bit权重模型推理"](#4bit-权重模型推理)章节的说明，进行模型推理。

```shell
python3 -m lmdeploy.lite.apis.auto_awq \

```shell
  --w_bits 4 \                       # 权重量化的 bit 数
  --w_group_size 128 \               # 权重量化分组统计尺寸
  --work_dir $WORK_DIR \             # 步骤 1 保存量化参数的目录
```

# Config

MMOCR mainly uses Python files as configuration files. The design of its configuration file system integrates the ideas of modularity and inheritance to facilitate various experiments.

```{note}
This section is recommended to be read together with the primary usage in {external+mmengine:doc}`MMEngine: Config <tutorials/config>`.
```

There are three most common operations in MMOCR: inheritance of configuration files, reference to `_base_` variables, and modification of `_base_` variables. Config provides two syntaxes for inheriting and modifying `_base_`, one for Python, Json, and Yaml, and one for Python configuration files only. In MMOCR, we **prefer the Python-only syntax**, so this will be the basis for further description.

The `configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py` is used as an example to illustrate the three common uses.

```Python
_base_ = [
    '_base_dbnet_resnet18_fpnc.py',
    '../_base_/datasets/icdar2015.py',

```Python
_base_ = [
    '../_base_/default_runtime.py',
    '../_base_/schedules/schedule_sgd_1200e.py',
]

icdar2015_textdet_train = _base_.icdar2015_textdet_train
icdar2015_textdet_train.pipeline = _base_.train_pipeline
icdar2015_textdet_test = _base_.icdar2015_textdet_test
icdar2015_textdet_test.pipeline = _base_.test_pipeline

    batch_size=16,
    num_workers=8,
    dataset=icdar2015_textdet_train)

    num_workers=4,
    dataset=icdar2015_textdet_test)
```

### Configuration Inheritance

batch_size=1,
```

There is an inheritance mechanism for configuration files, i.e. one configuration file A can use another configuration file B as its base and inherit all the fields directly from it, thus avoiding a lot of copy-pasting.

In `dbnet_resnet18_fpnc_1200e_icdar2015.py` you can see that

```Python
_base_ = [
]
```

The above statement reads all the base configuration files in the list, and all the fields in them are loaded into `dbnet_resnet18_fpnc_1200e_icdar2015.py`. We can see the structure of the configuration file after it has been parsed by running the following statement in a Python interpretation.

```Python
from mmengine import Config

```Python
db_config = Config.fromfile('configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py')
print(db_config)
```

It can be found that the parsed configuration contains all the fields and information in the base configuration.

```{note}
Variables with the same name cannot exist in each `_base_` profile.
```

### `_base_` Variable References

Sometimes we may need to reference some fields in the `_base_` configuration directly in order to avoid duplicate definitions. Suppose we want to get the variable `pseudo` in the `_base_` configuration, we can get the variable in the `_base_` configuration directly via `_base_.pseudo`.

This syntax has been used extensively in the configuration of MMOCR, and the dataset and pipeline configurations for each model in MMOCR are referenced in the *_base_* configuration. For example,

```Python
# ...
    # ...

```Python
# ...
    # ...
```

<div id="base_variable_modification"></div>

### `_base_` Variable Modification

In MMOCR, different algorithms usually have different pipelines in different datasets, so there are often scenarios to modify the `pipeline` in the dataset. There are also many scenarios where you need to modify variables in the `_base_` configuration, for example, modifying the training strategy of an algorithm, replacing some modules of an algorithm(backbone, etc.). Users can directly modify the referenced `_base_` variables using Python syntax. For dict, we also provide a method similar to class attribute modification to modify the contents of the dictionary directly.

1. Dictionary

   Here is an example of modifying `pipeline` in a dataset.

   The dictionary can be modified using Python syntax:

   ```Python
   # Get the dataset in _base_

   ```Python
   icdar2015_textdet_train = _base_.icdar2015_textdet_train
   # You can modify the variables directly with Python's update
   icdar2015_textdet_train.update(pipeline=_base_.train_pipeline)
   ```

   It can also be modified in the same way as changing Python class attributes.

   ```Python
   # The class property method is modified
   icdar2015_textdet_train.pipeline = _base_.train_pipeline
   ```

2. List

   Suppose the variable `pseudo = [1, 2, 3]` in the `_base_` configuration needs to be modified to `[1, 2, 4]`:

   ```Python
   # pseudo.py
   pseudo = [1, 2, 3]
   ```

   Can be rewritten directly as.

   ```Python
   _base_ = ['pseudo.py']
   pseudo = [1, 2, 4]
   ```

   Or modify the list using Python syntax:

# pseudo.py
   ```

   ```Python
   ```

   ```Python
   pseudo = _base_.pseudo
   pseudo[2] = 4
   ```

### Command Line Modification

Sometimes we only want to fix part of the configuration and do not want to modify the configuration file itself. For example, if you want to change the learning rate during an experiment but do not want to write a new configuration file, you can pass in parameters on the command line to override the relevant configuration.

We can pass `--cfg-options` on the command line and modify the corresponding fields directly with the arguments after it. For example, we can run the following command to modify the learning rate temporarily for this training session.

```Shell
python tools/train.py example.py --cfg-options optim_wrapper.optimizer.lr=1
```

```Shell
```

For more detailed usage, refer to {external+mmengine:doc}`MMEngine: Command Line Modification <tutorials/config>`.

## Configuration Content

With config files and Registry, MMOCR can modify the training parameters as well as the model configuration without invading the code. Specifically, users can customize the following modules in the configuration file: environment configuration, hook configuration, log configuration, training strategy configuration, data-related configuration, model-related configuration, evaluation configuration, and visualization configuration.

This document will take the text detection algorithm `DBNet` and the text recognition algorithm `CRNN` as examples to introduce the contents of Config in detail.

<div id="env_config"></div>

### Environment Configuration

```Python
default_scope = 'mmocr'

```Python
randomness = dict(seed=None)
```

There are three main components:

- Set the default `scope` of all registries to `mmocr`, ensuring that all modules are searched first from the `MMOCR` codebase. If the module does not exist, the search will continue from the upstream algorithm libraries `MMEngine` and `MMCV`, see {external+mmengine:doc}`MMEngine: Registry <tutorials/registry>` for more details.

- `env_cfg` configures the distributed environment, see {external+mmengine:doc}`MMEngine: Runner <tutorials/runner>` for more details.

- `randomness`: Some settings to make the experiment as reproducible
  as possible like seed and deterministic. See {external+mmengine:doc}`MMEngine: Runner <tutorials/runner>` for more details.

<div id="hook_config"></div>

as possible like seed and deterministic. See {external+mmengine:doc}`MMEngine: Runner <tutorials/runner>` for more details.

### Hook Configuration

Hooks are divided into two main parts, default hooks, which are required for all tasks to run, and custom hooks, which generally serve specific algorithms or specific tasks (there are no custom hooks in MMOCR so far).

```Python
    timer=dict(type='IterTimerHook'), # Time recording, including data time as well as model inference time
    logger=dict(type='LoggerHook', interval=1), # Collect logs from different components
    param_scheduler=dict(type='ParamSchedulerHook'), # Update some hyper-parameters in optimizer
    checkpoint=dict(type='CheckpointHook', interval=1),# Save checkpoint. `interval` control save interval
    sampler_seed=dict(type='DistSamplerSeedHook'), # Data-loading sampler for distributed training.

sampler_seed=dict(type='DistSamplerSeedHook'), # Data-loading sampler for distributed training.
    sync_buffer=dict(type='SyncBuffersHook'), # Synchronize buffer in case of distributed training
    visualization=dict( # Visualize the results of val and test
        type='VisualizationHook',
        interval=1,
        enable=False,
        show=False,
        draw_gt=False,
        draw_pred=False))
 custom_hooks = []
```

Here is a brief description of a few hooks whose parameters may be changed frequently. For a general modification method, refer to <a href="#base_variable_modification">Modify configuration</a>.

- `LoggerHook`: Used to configure the behavior of the logger. For example, by modifying `interval` you can control the interval of log printing, so that the log is printed once per `interval` iteration, for more settings refer to [LoggerHook API](mmengine.hooks.LoggerHook).

- `CheckpointHook`: Used to configure checkpoint-related behavior, such as saving optimal and/or latest weights. You can also modify `interval` to control the checkpoint saving interval. More settings can be found in [CheckpointHook API](mmengine.hooks.CheckpointHook)

- `VisualizationHook`: Used to configure visualization-related behavior, such as visualizing predicted results during validation or testing. **Default is off**. This Hook also depends on [Visualization Configuration](#Visualization-configuration). You can refer to [Visualizer](visualization.md) for more details. For more configuration, you can refer to [VisualizationHook API](mmocr.engine.hooks.VisualizationHook).

If you want to learn more about the configuration of the default hooks and their functions, you can refer to {external+mmengine:doc}`MMEngine: Hooks <tutorials/hook>`.

<div id="log_config"></div>

### Log Configuration

This section is mainly used to configure the log level and the log processor.

```Python

```Python
log_level = 'INFO' # Logging Level
log_processor = dict(type='LogProcessor',
                        window_size=10,
                        by_epoch=True)
```

- The logging severity level is the same as that of {external+python:doc}`Python: logging <library/logging>`

- The log processor is mainly used to control the format of the output, detailed functions can be found in {external+mmengine:doc}`MMEngine: logging <advanced_tutorials/logging>`.

  - `by_epoch=True` indicates that the logs are output in accordance to "epoch", and the log format needs to be consistent with the `type='EpochBasedTrainLoop'` parameter in `train_cfg`. For example, if you want to output logs by iteration number, you need to set ` by_epoch=False` in `log_processor` and `type='IterBasedTrainLoop'` in `train_cfg`.

- `window_size` indicates the smoothing window of the loss, i.e. the average value of the various losses for the last `window_size` iterations. the final loss value printed in logger is the average of all the losses.

  <div id="schedule_config"></div>

### Training Strategy Configuration

This section mainly contains optimizer settings, learning rate schedules and `Loop` settings.

Training strategies usually vary for different tasks (text detection, text recognition, key information extraction). Here we explain the example configuration in `CRNN`, which is a text recognition model.

```Python
    type='OptimWrapper', optimizer=dict(type='Adadelta', lr=1.0))
param_scheduler = [dict(type='ConstantLR', factor=1.0)]
train_cfg = dict(type='EpochBasedTrainLoop',
                    max_epochs=5, # train epochs
                    val_interval=1) # val interval
```

max_epochs=5, # train epochs
```

- `optim_wrapper` : It contains two main parts, OptimWrapper and Optimizer. Detailed usage information can be found in {external+mmengine:doc}`MMEngine: Optimizer Wrapper <tutorials/optim_wrapper>`.

  - The Optimizer wrapper supports different training strategies, including mixed-accuracy training (AMP), gradient accumulation, and gradient truncation.

  - All PyTorch optimizers are supported in the optimizer settings. All supported optimizers are available in {external+torch:ref}`PyTorch Optimizer List <optim:algorithms>`.

- All PyTorch optimizers are supported in the optimizer settings. All supported optimizers are available in {external+torch:ref}`PyTorch Optimizer List <optim:algorithms>`.

- `param_scheduler` : learning rate tuning strategy, supports most of the learning rate schedulers in PyTorch, such as `ExponentialLR`, `LinearLR`, `StepLR`, `MultiStepLR`, etc., and is used in much the same way, see [scheduler interface](mmengine.optim.scheduler), and more features can be found in the {external+mmengine:doc}`MMEngine: Optimizer Parameter Tuning Strategy <tutorials/param_scheduler>`.

- `train/test/val_cfg` : the execution flow of the task, MMEngine provides four kinds of flow: `EpochBasedTrainLoop`, `IterBasedTrainLoop`, `ValLoop`, `TestLoop` More can be found in {external+mmengine:doc}`MMEngine: loop controller <design/runner>`.

### Data-related Configuration

<div id="dataset_config"></id>

#### Dataset Configuration

It is mainly about two parts.

- The location of the dataset(s), including images and annotation files.

- Data augmentation related configurations. In the OCR domain, data augmentation is usually strongly associated with the model.

More parameter configurations can be found in [Data Base Class](#TODO).

The naming convention for dataset fields in MMOCR is

```Python
{dataset}_{task}_{train/val/test} = dict(...)
```

- dataset: See [dataset abbreviations](#TODO)

- task: `det`(text detection), `rec`(text recognition), `kie`(key information extraction)

- train/val/test: Dataset split.

For example, for text recognition tasks, Syn90k is used as the training set, while icdar2013 and icdar2015 serve as the test sets. These are configured as follows.

```Python
# text recognition dataset configuration
mjsynth_textrecog_train = dict(
    type='OCRDataset',
    data_root='data/rec/Syn90k/',

```Python
    data_prefix=dict(img_path='mnt/ramdisk/max/90kDICT32px'),
    ann_file='train_labels.json',
    test_mode=False,
    pipeline=None)

icdar2013_textrecog_test = dict(
    data_root='data/rec/icdar_2013/',
    data_prefix=dict(img_path='Challenge2_Test_Task3_Images/'),
    ann_file='test_labels.json',
    test_mode=True,

icdar2015_textrecog_test = dict(
    data_root='data/rec/icdar_2015/',
    data_prefix=dict(img_path='ch4_test_word_images_gt/'),
```

<div id="pipeline_config"></div>

#### Data Pipeline Configuration

ann_file='test_labels.json',
```

In MMOCR, dataset construction and data preparation are decoupled from each other. In other words, dataset classes such as `OCRDataset` are responsible for reading and parsing annotation files, while Data Transforms further implement data loading, data augmentation, data formatting and other related functions.

In general, there are different augmentation strategies for training and testing, so there are usually `training_pipeline` and `testing_pipeline`. More information can be found in [Data Transforms](../basic_concepts/transforms.md)

- The data augmentation process of the training pipeline is usually: data loading (LoadImageFromFile) -> annotation information loading (LoadXXXAnntation) -> data augmentation -> data formatting (PackXXXInputs).

- The data augmentation flow of the test pipeline is usually: Data Loading (LoadImageFromFile) -> Data Augmentation -> Annotation Loading (LoadXXXAnntation) -> Data Formatting (PackXXXInputs).

Due to the specificity of the OCR task, different models have different data augmentation techniques, and even the same model can have different data augmentation strategies for different datasets. Take `CRNN` as an example.

```Python
# Data Augmentation
    dict(
        type='LoadImageFromFile',
        color_type='grayscale',
        ignore_empty=True,
        min_size=5),
    dict(type='LoadOCRAnnotations', with_text=True),
    dict(type='Resize', scale=(100, 32), keep_ratio=False),
    dict(
        type='PackTextRecogInputs',
        meta_keys=('img_path', 'ori_shape', 'img_shape', 'valid_ratio'))
]
    dict(
        color_type='grayscale'),
    dict(
        type='RescaleToHeight',
        height=32,

]
    dict(
    dict(
        min_width=32,
        max_width=None,
        width_divisor=16),
    dict(
]
```

#### Dataloader Configuration

The main configuration information needed to construct the dataset loader (dataloader), see {external+torch:doc}`PyTorch DataLoader <data>` for more tutorials.

```Python
# Dataloader
    batch_size=64,
        type='ConcatDataset',
        datasets=[mjsynth_textrecog_train],

type='ConcatDataset',
        datasets=[icdar2013_textrecog_test, icdar2015_textrecog_test],
```

### Model-related Configuration

<div id="model_config"></div>

#### Network Configuration

This section configures the network architecture. Different algorithmic tasks use different network architectures. Find more info about network architecture in [structures](../basic_concepts/structures.md)

##### Text Detection

Text detection consists of several parts:

- `data_preprocessor`: [data_preprocessor](mmocr.models.textdet.data_preprocessors.TextDetDataPreprocessor)
- `backbone`: backbone network configuration

- `neck`: neck network configuration
- `det_head`: detection head network configuration
  - `module_loss`: module loss configuration
  - `postprocessor`: postprocessor configuration

We present the model configuration in text detection using DBNet as an example.

```Python
    type='DBNet',
        type='TextDetDataPreprocessor',
        pad_size_divisor=32)
        type='mmdet.ResNet',
        depth=18,
        frozen_stages=-1,
        init_cfg=dict(type='Pretrained', checkpoint='torchvision://resnet18'),

out_indices=(0, 1, 2, 3),
        style='caffe'),
        type='FPNC', in_channels=[64, 128, 256, 512], lateral_channels=256),
    det_head=dict(
        type='DBHead',
        module_loss=dict(type='DBModuleLoss'),
        postprocessor=dict(type='DBPostprocessor', text_repr_type='quad')))
```

##### Text Recognition

Text recognition mainly contains:

- `data_processor`: [data preprocessor configuration](mmocr.models.textrecog.data_processors.TextRecDataPreprocessor)
- `preprocessor`: network preprocessor configuration, e.g. TPS
- `backbone`: backbone configuration
- `encoder`: encoder configuration
- `decoder`: decoder configuration
  - `module_loss`: decoder module loss configuration
  - `postprocessor`: decoder postprocessor configuration

  - `dictionary`: dictionary configuration

Using CRNN as an example.

```Python
# model
   type='CRNN',
   data_preprocessor=dict(
        type='TextRecogDataPreprocessor', mean=[127], std=[127])
    preprocessor=None,
    backbone=dict(type='VeryDeepVgg', leaky_relu=False, input_channels=1),
    encoder=None,
    decoder=dict(
        type='CRNNDecoder',
        in_channels=512,
        rnn_flag=True,
        module_loss=dict(type='CTCModuleLoss', letter_case='lower'),
        postprocessor=dict(type='CTCPostProcessor'),
        dictionary=dict(
            type='Dictionary',
            dict_file='dicts/lower_english_digits.txt',
            with_padding=True)))
```

<div id="weight_config"></div>

#### Checkpoint Loading Configuration

type='Dictionary',
```

The model weights in the checkpoint file can be loaded via the `load_from` parameter, simply by setting the `load_from` parameter to the path of the checkpoint file.

You can also resume training by setting `resume=True` to load the training status information in the checkpoint. When both `load_from` and `resume=True` are set, MMEngine will load the training state from the checkpoint file at the `load_from` path.

If only `resume=True` is set, the executor will try to find and read the latest checkpoint file from the `work_dir` folder

```Python
load_from = None # Path to load checkpoint
resume = False # whether resume
```

```Python
```

More can be found in {external+mmengine:doc}`MMEngine: Load Weights or Recover Training <tutorials/runner>` and [OCR Advanced Tips - Resume Training from Checkpoints](train_test.md#resume-training-from-a-checkpoint).

<div id="eval_config"></id>

### Evaluation Configuration

In model validation and model testing, quantitative measurement of model accuracy is often required. MMOCR performs this function by means of `Metric` and `Evaluator`. For more information, please refer to {external+mmengine:doc}`MMEngine: Evaluation <tutorials/evaluation>` and [Evaluation](../basic_concepts/evaluation.md)

#### Evaluator

Evaluator is mainly used to manage multiple datasets and multiple `Metrics`. For single and multiple dataset cases, there are single and multiple dataset evaluators, both of which can manage multiple `Metrics`.

The single-dataset evaluator is configured as follows.

```Python

```Python
# Single Dataset Single Metric
    type='Evaluator',
    metrics=dict())

# Single Dataset Multiple Metric
    metrics=[...])
```

`MultiDatasetsEvaluator` differs from single-dataset evaluation in two aspects: `type` and `dataset_prefixes`. The evaluator type must be `MultiDatasetsEvaluator` and cannot be omitted. The `dataset_prefixes` is mainly used to distinguish the results of different datasets with the same evaluation metrics, see [MultiDatasetsEvaluation](../basic_concepts/evaluation.md).

Assuming that we need to test accuracy on IC13 and IC15 datasets, the configuration is as follows.

```Python
#  Multiple datasets, single Metric
    type='MultiDatasetsEvaluator',
    metrics=dict(),
    dataset_prefixes=['IC13', 'IC15'])

# Multiple datasets, multiple Metrics

type='MultiDatasetsEvaluator',

    metrics=[...],
```

#### Metric

A metric evaluates a model's performance from a specific perspective. While there is no such common metric that fits all the tasks, MMOCR provides enough flexibility such that multiple metrics serving the same task can be used simultaneously. Here we list task-specific metrics for reference.

Text detection: [`HmeanIOUMetric`](mmocr.evaluation.metrics.HmeanIOUMetric)

Text recognition: [`WordMetric`](mmocr.evaluation.metrics.WordMetric), [`CharMetric`](mmocr.evaluation.metrics.CharMetric), [`OneMinusNEDMetric`](mmocr.evaluation.metrics.OneMinusNEDMetric)

Key information extraction: [`F1Metric`](mmocr.evaluation.metrics.F1Metric)

Text detection as an example, using a single `Metric` in the case of single dataset evaluation.

```Python
val_evaluator = dict(type='HmeanIOUMetric')
```

Take text recognition as an example, multiple datasets (`IC13` and `IC15`) are evaluated using multiple `Metric`s (`WordMetric` and `CharMetric`).

```Python
    metrics=[
        dict(
            type='WordMetric',
            mode=['exact', 'ignore_case', 'ignore_case_symbol']),
        dict(type='CharMetric')
    ],
```

<div id="vis_config"></div>

### Visualization Configuration

dict(type='CharMetric')
    ],
```

Each task is bound to a task-specific visualizer. The visualizer is mainly used for visualizing or storing intermediate results of user models and visualizing val and test prediction results. The visualization results can also be stored in different backends such as WandB, TensorBoard, etc. through the corresponding visualization backend. Commonly used modification operations can be found in [visualization](visualization.md).

The default configuration of visualization for text detection is as follows.

```Python
vis_backends = [dict(type='LocalVisBackend')]
    type='TextDetLocalVisualizer',  # Different visualizers for different tasks
```

## Directory Structure

```

All configuration files of `MMOCR` are placed under the `configs` folder. To avoid config files from being too long and improve their reusability and clarity, MMOCR takes advantage of the inheritance mechanism and split config files into eight sections. Since each section is closely related to the task type, MMOCR provides a task folder for each task in `configs/`, namely `textdet` (text detection task), `textrecog` (text recognition task), and `kie` (key information extraction). Each folder is further divided into two parts: `_base_` folder and algorithm configuration folders.

1. the `_base_` folder stores some general config files unrelated to specific algorithms, and each section is divided into datasets, training strategies and runtime configurations by directory.

2. The algorithm configuration folder stores config files that are strongly related to the algorithm. The algorithm configuration folder has two kinds of config files.

   1. Config files starting with `_base_`: Configures the model and data pipeline of an algorithm. In OCR domain, data augmentation strategies are generally strongly related to the algorithm, so the model and data pipeline are usually placed in the same config file.

2. Other config files, i.e. the algorithm-specific configurations on the specific dataset(s): These are the full config files that further configure training and testing settings, aggregating `_base_` configurations that are scattered in different locations. Inside some modifications to the fields in `_base_` configs may be performed, such as data pipeline, training strategy, etc.

All these config files are distributed in different folders according to their contents as follows:

<style type="text/css">
.tg  {border-collapse:collapse;border-spacing:0;}
.tg td{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;
  overflow:hidden;padding:10px 5px;word-break:normal;}
.tg th{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;
  font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;}
.tg .tg-lboi{border-color:inherit;text-align:left;vertical-align:middle}

font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;}
.tg .tg-9wq8{border-color:inherit;text-align:center;vertical-align:middle}
.tg .tg-0pky{border-color:inherit;text-align:left;vertical-align:top}
</style>

<table class="tg">
<thead>
  <tr>
    <td class="tg-9wq8" rowspan="5">textdet<br></td>
    <td class="tg-lboi" rowspan="3">_base_</td>
    <td class="tg-9wq8">datasets</td>
    <td class="tg-0pky">icdar_datasets.py<br>ctw1500.py<br>...</td>
    <td class="tg-0pky"><a href="#dataset_config">Dataset configuration</a></td>
  </tr>
  <tr>
    <td class="tg-9wq8">schedules</td>
    <td class="tg-0pky">schedule_adam_600e.py<br>...</td>
    <td class="tg-0pky"><a href="#schedule_config">Training Strategy Configuration</a></td>
  </tr>
  <tr>
    <td class="tg-9wq8">default_runtime.py<br></td>
    <td class="tg-0pky">-</td>

<td class="tg-0pky"><a href="#schedule_config">Training Strategy Configuration</a></td>
  </tr>
  <tr>
    <td class="tg-0pky"><a href="#env_config">Environment Configuration</a><br><a href="#hook_config">Hook Configuration</a><br><a href="#log_config">Log Configuration</a> <br><a href="#weight_config">Checkpoint Loading Configuration</a> <br><a href="#eval_config">Evaluation Configuration</a> <br><a href="#vis_config">Visualization Configuration</a></td>
  </tr>
  <tr>
    <td class="tg-lboi" rowspan="2">dbnet</td>
    <td class="tg-9wq8">_base_dbnet_resnet18_fpnc.py</td>
    <td class="tg-0pky"><span style="font-weight:400;font-style:normal">-</span></td>
    <td class="tg-0pky"><a href="#model_config">Network Configuration</a> <br><a href="#pipeline_config">Data Pipeline Configuration</a></td>
  </tr>
  <tr>
    <td class="tg-9wq8">dbnet_resnet18_fpnc_1200e_icdar2015.py</td>

</tr>
  <tr>
    <td class="tg-0pky"><a href="#dataloader_config">Dataloader Configuration</a> <br><a href="#pipeline_config">Data Pipeline Configuration(Optional)</a></td>
  </tr>
</thead>
</table>

The final directory structure is as follows.

```Python
configs
├── textdet
│   ├── _base_
│   │   ├── datasets
│   │   │   ├── icdar2015.py
│   │   │   ├── icdar2017.py
│   │   │   └── totaltext.py
│   │   ├── schedules
│   │   │   └── schedule_adam_600e.py
│   │   └── default_runtime.py
│   └── dbnet
│       ├── _base_dbnet_resnet18_fpnc.py
│       └── dbnet_resnet18_fpnc_1200e_icdar2015.py
├── textrecog
│   │   │   └── schedule_adam_base.py
│   └── crnn

│   └── crnn
│       ├── _base_crnn_mini-vgg.py
│       └── crnn_mini-vgg_5e_mj.py
└── kie
    ├── _base_
    │   ├──datasets
    │   └── default_runtime.py
    └── sgdmr
        └── sdmgr_novisual_60e_wildreceipt_openset.py
```

## Naming Conventions

MMOCR has a convention to name config files, and contributors to the code base need to follow the same naming rules. The file names are divided into four sections: algorithm information, module information, training information, and data information. Words that logically belong to different sections are connected by an underscore `'_'`, and multiple words in the same section are connected by a hyphen `'-'`.

```Python
{{algorithm info}}_{{module info}}_{{training info}}_{{data info}}.py
```

- algorithm info: the name of the algorithm, such as dbnet, crnn, etc.

```Python
```

- module info: list some intermediate modules in the order of data flow. Its content depends on the algorithm, and some modules strongly related to the model will be omitted to avoid an overly long name. For example:

  - For the text detection task and the key information extraction task :

    ```Python
    {{algorithm info}}_{{backbone}}_{{neck}}_{{head}}_{{training info}}_{{data info}}.py
    ```

    `{head}` is usually omitted since it's algorithm-specific.

  - For text recognition tasks.

    ```Python
    {{algorithm info}}_{{backbone}}_{{encoder}}_{{decoder}}_{{training info}}_{{data info}}.py
    ```

    Since encoder and decoder are generally bound to the algorithm, they are usually omitted.

- training info: some settings of the training strategy, including batch size, schedule, etc.

```

- data info: dataset name, modality, input size, etc., such as icdar2015 and synthtext.

# PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud

> [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://arxiv.org/abs/1812.04244)

In this paper, we propose PointRCNN for 3D object detection from raw point cloud. The whole framework is composed of two stages: stage-1 for the bottom-up 3D proposal generation and stage-2 for refining proposals in the canonical coordinates to obtain the final detection results. Instead of generating proposals from RGB image or projecting point cloud to bird's view or voxels as previous methods do, our stage-1 sub-network directly generates a small number of high-quality 3D proposals from point cloud in a bottom-up manner via segmenting the point cloud of the whole scene into foreground points and background. The stage-2 sub-network transforms the pooled points of each proposal to canonical coordinates to learn better local spatial features, which is combined with global semantic features of each point learned in stage-1 for accurate box refinement and confidence prediction. Extensive experiments on the 3D detection benchmark of KITTI dataset show that our proposed architecture outperforms state-of-the-art methods with remarkable margins by using only point cloud as input.

<img src="https://user-images.githubusercontent.com/79644370/144959105-271038a2-4ae1-4cdb-b6a8-68c14daf83b0.png" width="800"/>
</div>

We implement PointRCNN and provide the result with checkpoints on KITTI dataset.

### KITTI

|                      Backbone                      |  Class  |  Lr schd   | Mem (GB) | Inf time (fps) |  mAP  |                                                                                                                                   Download                                                                                                                                    |

| :------------------------------------------------: | :-----: | :--------: | :------: | :------------: | :---: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| [PointNet++](./point-rcnn_8xb2_kitti-3d-3class.py) | 3 Class | cyclic 40e |   4.6    |                | 70.83 | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/point_rcnn/point_rcnn_2x8_kitti-3d-3classes_20211208_151344.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/point_rcnn/point_rcnn_2x8_kitti-3d-3classes_20211208_151344.log.json) |

Note: mAP represents AP11 results on 3 Class under the moderate setting.

Detailed performance on KITTI 3D detection (3D) is as follows, evaluated by AP11 metric:

|            | Easy  | Moderate | Hard  |

| ---------- | :---: | :------: | :---: |
| Car        | 89.13 |  78.72   | 78.24 |
| Pedestrian | 65.81 |  59.57   | 52.75 |
| Cyclist    | 93.51 |  74.19   | 70.73 |

```latex
@inproceedings{Shi_2019_CVPR,
    title = {PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud},
    author = {Shi, Shaoshuai and Wang, Xiaogang and Li, Hongsheng},
    booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    month = {June},
    year = {2019}
}
```

# 学习配置文件

MMDetection 和其他 OpenMMLab 仓库使用 [MMEngine 的配置文件系统](https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/config.html)。 配置文件使用了模块化和继承设计，以便于进行各类实验。

## 配置文件的内容

MMDetection 采用模块化设计，所有功能的模块都可以通过配置文件进行配置。 以 Mask R-CNN 为例，我们将根据不同的功能模块介绍配置文件中的各个字段：

### 模型配置

在 mmdetection 的配置中，我们使用 `model` 字段来配置检测算法的组件。 除了 `backbone`、`neck` 等神经网络组件外，还需要 `data_preprocessor`、`train_cfg` 和 `test_cfg`。 `data_preprocessor` 负责对 dataloader 输出的每一批数据进行预处理。 模型配置中的 `train_cfg` 和 `test_cfg` 用于设置训练和测试组件的超参数。

```python
    type='MaskRCNN',  # 检测器名
    data_preprocessor=dict(  # 数据预处理器的配置，通常包括图像归一化和 padding
        type='DetDataPreprocessor',  # 数据预处理器的类型，参考 https://mmdetection.readthedocs.io/en/latest/api.html#mmdet.models.data_preprocessors.DetDataPreprocessor
        mean=[123.675, 116.28, 103.53],  # 用于预训练骨干网络的图像归一化通道均值，按 R、G、B 排序
        std=[58.395, 57.12, 57.375],  # 用于预训练骨干网络的图像归一化通道标准差，按 R、G、B 排序
        bgr_to_rgb=True,  # 是否将图片通道从 BGR 转为 RGB
        pad_mask=True,  # 是否填充实例分割掩码

std=[58.395, 57.12, 57.375],  # 用于预训练骨干网络的图像归一化通道标准差，按 R、G、B 排序
        pad_size_divisor=32),  # padding 后的图像的大小应该可以被 ``pad_size_divisor`` 整除
    backbone=dict(  # 主干网络的配置文件
        type='ResNet',  # 主干网络的类别，可用选项请参考 https://mmdetection.readthedocs.io/en/latest/api.html#mmdet.models.backbones.ResNet
        depth=50,  # 主干网络的深度，对于 ResNet 和 ResNext 通常设置为 50 或 101
        num_stages=4,  # 主干网络状态(stages)的数目，这些状态产生的特征图作为后续的 head 的输入
        out_indices=(0, 1, 2, 3),  # 每个状态产生的特征图输出的索引
        frozen_stages=1,  # 第一个状态的权重被冻结
        norm_cfg=dict(  # 归一化层(norm layer)的配置项
            type='BN',  # 归一化层的类别，通常是 BN 或 GN
            requires_grad=True),  # 是否训练归一化里的 gamma 和 beta
        norm_eval=True,  # 是否冻结 BN 里的统计项
        style='pytorch',  # 主干网络的风格，'pytorch' 意思是步长为2的层为 3x3 卷积， 'caffe' 意思是步长为2的层为 1x1 卷积

requires_grad=True),  # 是否训练归一化里的 gamma 和 beta
        init_cfg=dict(type='Pretrained', checkpoint='torchvision://resnet50')),  # 加载通过 ImageNet 预训练的模型
        type='FPN',  # 检测器的 neck 是 FPN，我们同样支持 'NASFPN', 'PAFPN' 等，更多细节可以参考 https://mmdetection.readthedocs.io/en/latest/api.html#mmdet.models.necks.FPN
        in_channels=[256, 512, 1024, 2048],  # 输入通道数，这与主干网络的输出通道一致
        out_channels=256,  # 金字塔特征图每一层的输出通道
        num_outs=5),  # 输出的范围(scales)
        type='RPNHead',  # rpn_head 的类型是 'RPNHead', 我们也支持 'GARPNHead' 等，更多细节可以参考 https://mmdetection.readthedocs.io/en/latest/api.html#mmdet.models.dense_heads.RPNHead
        in_channels=256,  # 每个输入特征图的输入通道，这与 neck 的输出通道一致
        feat_channels=256,  # head 卷积层的特征通道
        anchor_generator=dict(  # 锚点(Anchor)生成器的配置

in_channels=256,  # 每个输入特征图的输入通道，这与 neck 的输出通道一致
            type='AnchorGenerator',  # 大多数方法使用 AnchorGenerator 作为锚点生成器, SSD 检测器使用 `SSDAnchorGenerator`。更多细节请参考 https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/task_modules/prior_generators/anchor_generator.py#L18
            scales=[8],  # 锚点的基本比例，特征图某一位置的锚点面积为 scale * base_sizes
            ratios=[0.5, 1.0, 2.0],  # 高度和宽度之间的比率
            strides=[4, 8, 16, 32, 64]),  # 锚生成器的步幅。这与 FPN 特征步幅一致。 如果未设置 base_sizes，则当前步幅值将被视为 base_sizes
        bbox_coder=dict(  # 在训练和测试期间对框进行编码和解码
            type='DeltaXYWHBBoxCoder',  # 框编码器的类别，'DeltaXYWHBBoxCoder' 是最常用的，更多细节请参考 https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/task_modules/coders/delta_xywh_bbox_coder.py#L13
            target_means=[0.0, 0.0, 0.0, 0.0],  # 用于编码和解码框的目标均值
            target_stds=[1.0, 1.0, 1.0, 1.0]),  # 用于编码和解码框的标准差

target_means=[0.0, 0.0, 0.0, 0.0],  # 用于编码和解码框的目标均值
        loss_cls=dict(  # 分类分支的损失函数配置
            type='CrossEntropyLoss',  # 分类分支的损失类型，我们也支持 FocalLoss 等，更多细节请参考 https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/losses/cross_entropy_loss.py#L201
            use_sigmoid=True,  # RPN 通常进行二分类，所以通常使用 sigmoid 函数
            los_weight=1.0),  # 分类分支的损失权重
        loss_bbox=dict(  # 回归分支的损失函数配置
            type='L1Loss',  # 损失类型，我们还支持许多 IoU Losses 和 Smooth L1-loss 等，更多细节请参考 https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/losses/smooth_l1_loss.py#L56
            loss_weight=1.0)),  # 回归分支的损失权重
    roi_head=dict(  # RoIHead 封装了两步(two-stage)/级联(cascade)检测器的第二步
        type='StandardRoIHead',  # RoI head 的类型，更多细节请参考 https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/roi_heads/standard_roi_head.py#L17
        bbox_roi_extractor=dict(  # 用于 bbox 回归的 RoI 特征提取器

bbox_roi_extractor=dict(  # 用于 bbox 回归的 RoI 特征提取器
            type='SingleRoIExtractor',  # RoI 特征提取器的类型，大多数方法使用 SingleRoIExtractor，更多细节请参考 https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/roi_heads/roi_extractors/single_level_roi_extractor.py#L13
            roi_layer=dict(  # RoI 层的配置
                type='RoIAlign',  # RoI 层的类别, 也支持 DeformRoIPoolingPack 和 ModulatedDeformRoIPoolingPack，更多细节请参考 https://mmcv.readthedocs.io/en/latest/api.html#mmcv.ops.RoIAlign
                output_size=7,  # 特征图的输出大小
                sampling_ratio=0),  # 提取 RoI 特征时的采样率。0 表示自适应比率
            out_channels=256,  # 提取特征的输出通道
            featmap_strides=[4, 8, 16, 32]),  # 多尺度特征图的步幅，应该与主干的架构保持一致
        bbox_head=dict(  # RoIHead 中 box head 的配置
            type='Shared2FCBBoxHead',  # bbox head 的类别，更多细节请参考 https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/roi_heads/bbox_heads/convfc_bbox_head.py#L220

type='Shared2FCBBoxHead',  # bbox head 的类别，更多细节请参考 https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/roi_heads/bbox_heads/convfc_bbox_head.py#L220
            in_channels=256,  # bbox head 的输入通道。 这与 roi_extractor 中的 out_channels 一致
            fc_out_channels=1024,  # FC 层的输出特征通道
            roi_feat_size=7,  # 候选区域(Region of Interest)特征的大小
            num_classes=80,  # 分类的类别数量
            bbox_coder=dict(  # 第二阶段使用的框编码器
                type='DeltaXYWHBBoxCoder',  # 框编码器的类别，大多数情况使用 'DeltaXYWHBBoxCoder'
                target_means=[0.0, 0.0, 0.0, 0.0],  # 用于编码和解码框的均值
                target_stds=[0.1, 0.1, 0.2, 0.2]),  # 编码和解码的标准差。因为框更准确，所以值更小，常规设置时 [0.1, 0.1, 0.2, 0.2]。
            reg_class_agnostic=False,  # 回归是否与类别无关
            loss_cls=dict(  # 分类分支的损失函数配
                type='CrossEntropyLoss',  # 分类分支的损失类型，我们也支持 FocalLoss 等
                use_sigmoid=False,  # 是否使用 sigmoid
                loss_weight=1.0),  # 分类分支的损失权重

type='CrossEntropyLoss',  # 分类分支的损失类型，我们也支持 FocalLoss 等
            loss_bbox=dict(  # 回归分支的损失函数配置
                type='L1Loss',  # 损失类型，我们还支持许多 IoU Losses 和 Smooth L1-loss 等
                loss_weight=1.0)),  # 回归分支的损失权重
        mask_roi_extractor=dict(  # 用于 mask 生成的 RoI 特征提取器
            type='SingleRoIExtractor',  # RoI 特征提取器的类型，大多数方法使用 SingleRoIExtractor
            roi_layer=dict(  # 提取实例分割特征的 RoI 层配置
                type='RoIAlign',  # RoI 层的类型，也支持 DeformRoIPoolingPack 和 ModulatedDeformRoIPoolingPack
                output_size=14,  # 特征图的输出大小
                sampling_ratio=0),  # 提取 RoI 特征时的采样率
            featmap_strides=[4, 8, 16, 32]),  # 多尺度特征图的步幅
        mask_head=dict(  # mask 预测 head 模型

sampling_ratio=0),  # 提取 RoI 特征时的采样率
            type='FCNMaskHead',  # mask head 的类型，更多细节请参考 https://mmdetection.readthedocs.io/en/latest/api.html#mmdet.models.roi_heads.FCNMaskHead
            num_convs=4,  # mask head 中的卷积层数
            in_channels=256,  # 输入通道，应与 mask roi extractor 的输出通道一致
            conv_out_channels=256,  # 卷积层的输出通道
            num_classes=80,  # 要分割的类别数
            loss_mask=dict(  # mask 分支的损失函数配置
                type='CrossEntropyLoss',  # 用于分割的损失类型
                use_mask=True,  # 是否只在正确的类中训练 mask
                loss_weight=1.0))),  # mask 分支的损失权重
    train_cfg = dict(  # rpn 和 rcnn 训练超参数的配置
        rpn=dict(  # rpn 的训练配置
            assigner=dict(  # 分配器(assigner)的配置

loss_weight=1.0))),  # mask 分支的损失权重
                type='MaxIoUAssigner',  # 分配器的类型，MaxIoUAssigner 用于许多常见的检测器，更多细节请参考 https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/task_modules/assigners/max_iou_assigner.py#L14
                pos_iou_thr=0.7,  # IoU >= 0.7(阈值) 被视为正样本
                neg_iou_thr=0.3,  # IoU < 0.3(阈值) 被视为负样本
                min_pos_iou=0.3,  # 将框作为正样本的最小 IoU 阈值
                match_low_quality=True,  # 是否匹配低质量的框(更多细节见 API 文档)
                ignore_iof_thr=-1),  # 忽略 bbox 的 IoF 阈值
            sampler=dict(  # 正/负采样器(sampler)的配置
                type='RandomSampler',  # 采样器类型，还支持 PseudoSampler 和其他采样器，更多细节请参考 https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/task_modules/samplers/random_sampler.py#L14
                num=256,  # 样本数量。
                pos_fraction=0.5,  # 正样本占总样本的比例

num=256,  # 样本数量。
                neg_pos_ub=-1,  # 基于正样本数量的负样本上限
                add_gt_as_proposals=False),  # 采样后是否添加 GT 作为 proposal
            allowed_border=-1,  # 填充有效锚点后允许的边框
            pos_weight=-1,  # 训练期间正样本的权重
            debug=False),  # 是否设置调试(debug)模式
        rpn_proposal=dict(  # 在训练期间生成 proposals 的配置
            nms_across_levels=False,  # 是否对跨层的 box 做 NMS。仅适用于 `GARPNHead` ，naive rpn 不支持 nms cross levels
            nms_pre=2000,  # NMS 前的 box 数
            nms_post=1000,  # NMS 要保留的 box 的数量，只在 GARPNHHead 中起作用
            max_per_img=1000,  # NMS 后要保留的 box 数量
            nms=dict( # NMS 的配置
                type='nms',  # NMS 的类别
                iou_threshold=0.7 # NMS 的阈值
                ),
            min_bbox_size=0),  # 允许的最小 box 尺寸
        rcnn=dict(  # roi head 的配置。
            assigner=dict(  # 第二阶段分配器的配置，这与 rpn 中的不同

iou_threshold=0.7 # NMS 的阈值
                ),
                type='MaxIoUAssigner',  # 分配器的类型，MaxIoUAssigner 目前用于所有 roi_heads。更多细节请参考 https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/task_modules/assigners/max_iou_assigner.py#L14
                pos_iou_thr=0.5,  # IoU >= 0.5(阈值)被认为是正样本
                neg_iou_thr=0.5,  # IoU < 0.5(阈值)被认为是负样本
                min_pos_iou=0.5,  # 将 box 作为正样本的最小 IoU 阈值
                match_low_quality=False,  # 是否匹配低质量下的 box(有关更多详细信息，请参阅 API 文档)
                type='RandomSampler',  # 采样器的类型，还支持 PseudoSampler 和其他采样器，更多细节请参考 https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/task_modules/samplers/random_sampler.py#L14
                num=512,  # 样本数量

num=512,  # 样本数量
                pos_fraction=0.25,  # 正样本占总样本的比例
            ),  # 采样后是否添加 GT 作为 proposal
            mask_size=28,  # mask 的大小
            debug=False)),  # 是否设置调试模式
    test_cfg = dict(  # 用于测试 rpn 和 rcnn 超参数的配置
        rpn=dict(  # 测试阶段生成 proposals 的配置
            nms_across_levels=False,  # 是否对跨层的 box 做 NMS。仅适用于 `GARPNHead`，naive rpn 不支持做 NMS cross levels
            nms_pre=1000,  # NMS 前的 box 数
            nms_post=1000,  # NMS 要保留的 box 的数量，只在 `GARPNHHead` 中起作用
                type='nms',  # NMS 的类型
                iou_threshold=0.7 # NMS 阈值
                ),
            min_bbox_size=0),  # box 允许的最小尺寸
        rcnn=dict(  # roi heads 的配置
            score_thr=0.05,  # bbox 的分数阈值
            nms=dict(  # 第二步的 NMS 配置

),
                iou_thr=0.5),  # NMS 的阈值
            max_per_img=100,  # 每张图像的最大检测次数
            mask_thr_binary=0.5)))  # mask 预处的阈值
```

### 数据集和评测器配置

在使用[执行器](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/runner.html) 进行训练、测试、验证时，我们需要配置 [Dataloader](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/dataset.html)。构建数据 dataloader 需要设置数据集（dataset）和数据处理流程（data pipeline）。 由于这部分的配置较为复杂，我们使用中间变量来简化 dataloader 配置的编写。

```python
dataset_type = 'CocoDataset'  # 数据集类型，这将被用来定义数据集。
data_root = 'data/coco/'  # 数据的根路径。

train_pipeline = [  # 训练数据处理流程
    dict(type='LoadImageFromFile'),  # 第 1 个流程，从文件路径里加载图像。
    dict(
        type='LoadAnnotations',  # 第 2 个流程，对于当前图像，加载它的注释信息。
        with_bbox=True,  # 是否使用标注框(bounding box)， 目标检测需要设置为 True。

dict(type='LoadImageFromFile'),  # 第 1 个流程，从文件路径里加载图像。
    dict(
        with_mask=True,  # 是否使用 instance mask，实例分割需要设置为 True。
        poly2mask=False),  # 是否将 polygon mask 转化为 instance mask, 设置为 False 以加速和节省内存。
    dict(
        type='Resize',  # 变化图像和其标注大小的流程。
        scale=(1333, 800),  # 图像的最大尺寸
        keep_ratio=True  # 是否保持图像的长宽比。
        ),
    dict(
        type='RandomFlip',  # 翻转图像和其标注的数据增广流程。
        prob=0.5),  # 翻转图像的概率。
    dict(type='PackDetInputs')  # 将数据转换为检测器输入格式的流程
]
test_pipeline = [  # 测试数据处理流程
    dict(type='Resize', scale=(1333, 800), keep_ratio=True),  # 变化图像大小的流程。
    dict(
        type='PackDetInputs',  # 将数据转换为检测器输入格式的流程
]
train_dataloader = dict(  # 训练 dataloader 配置

type='PackDetInputs',  # 将数据转换为检测器输入格式的流程
]
    batch_size=2,  # 单个 GPU 的 batch size
    num_workers=2,  # 单个 GPU 分配的数据加载线程数
    persistent_workers=True,  # 如果设置为 True，dataloader 在迭代完一轮之后不会关闭数据读取的子进程，可以加速训练
    sampler=dict(  # 训练数据的采样器
        type='DefaultSampler',  # 默认的采样器，同时支持分布式和非分布式训练。请参考 https://mmengine.readthedocs.io/zh_CN/latest/api/generated/mmengine.dataset.DefaultSampler.html#mmengine.dataset.DefaultSampler
        shuffle=True),  # 随机打乱每个轮次训练数据的顺序
    batch_sampler=dict(type='AspectRatioBatchSampler'),  # 批数据采样器，用于确保每一批次内的数据拥有相似的长宽比，可用于节省显存
    dataset=dict(  # 训练数据集的配置
        ann_file='annotations/instances_train2017.json',  # 标注文件路径
        data_prefix=dict(img='train2017/'),  # 图片路径前缀
        filter_cfg=dict(filter_empty_gt=True, min_size=32),  # 图片和标注的过滤配置

ann_file='annotations/instances_train2017.json',  # 标注文件路径
        pipeline=train_pipeline))  # 这是由之前创建的 train_pipeline 定义的数据处理流程。
val_dataloader = dict(  # 验证 dataloader 配置
    batch_size=1,  # 单个 GPU 的 Batch size。如果 batch-szie > 1，组成 batch 时的额外填充会影响模型推理精度
    drop_last=False,  # 是否丢弃最后未能组成一个批次的数据
        shuffle=False),  # 验证和测试时不打乱数据顺序
        test_mode=True,  # 开启测试模式，避免数据集过滤图片和标注
test_dataloader = val_dataloader  # 测试 dataloader 配置
```

```

[评测器](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/evaluation.html) 用于计算训练模型在验证和测试数据集上的指标。评测器的配置由一个或一组评价指标（Metric）配置组成：

```python
val_evaluator = dict(  # 验证过程使用的评测器
    type='CocoMetric',  # 用于评估检测和实例分割的 AR、AP 和 mAP 的 coco 评价指标
    ann_file=data_root + 'annotations/instances_val2017.json',  # 标注文件路径
    metric=['bbox', 'segm'],  # 需要计算的评价指标，`bbox` 用于检测，`segm` 用于实例分割
test_evaluator = val_evaluator  # 测试过程使用的评测器
```

由于测试数据集没有标注文件，因此 MMDetection 中的 test_dataloader 和 test_evaluator 配置通常等于val。 如果要保存在测试数据集上的检测结果，则可以像这样编写配置：

```python
# 在测试集上推理，
# 并将检测结果转换格式以用于提交结果

    format_only=True,  # 只将模型输出转换为 coco 的 JSON 格式并保存
    outfile_prefix='./work_dirs/coco_detection/test')  # 要保存的 JSON 文件的前缀
```

### 训练和测试的配置

MMEngine 的 Runner 使用 Loop 来控制训练，验证和测试过程。
用户可以使用这些字段设置最大训练轮次和验证间隔。

```python
    type='EpochBasedTrainLoop',  # 训练循环的类型，请参考 https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py
    max_epochs=12,  # 最大训练轮次
    val_interval=1)  # 验证间隔。每个 epoch 验证一次

type='EpochBasedTrainLoop',  # 训练循环的类型，请参考 https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py
val_cfg = dict(type='ValLoop')  # 验证循环的类型
test_cfg = dict(type='TestLoop')  # 测试循环的类型
```

### 优化相关配置

`optim_wrapper` 是配置优化相关设置的字段。优化器封装（OptimWrapper）不仅提供了优化器的功能，还支持梯度裁剪、混合精度训练等功能。更多内容请看[优化器封装教程](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/optim_wrapper.html) 。

```python
optim_wrapper = dict(  # 优化器封装的配置
    type='OptimWrapper',  # 优化器封装的类型。可以切换至 AmpOptimWrapper 来启用混合精度训练
    optimizer=dict(  # 优化器配置。支持 PyTorch 的各种优化器。请参考 https://pytorch.org/docs/stable/optim.html#algorithms
        type='SGD',  # 随机梯度下降优化器
        lr=0.02,  # 基础学习率
        momentum=0.9,  # 带动量的随机梯度下降
        weight_decay=0.0001),  # 权重衰减
    clip_grad=None,  # 梯度裁剪的配置，设置为 None 关闭梯度裁剪。使用方法请见 https://mmengine.readthedocs.io/en/latest/tutorials/optimizer.html
    )
```

weight_decay=0.0001),  # 权重衰减
    )
```

`param_scheduler` 字段用于配置参数调度器（Parameter Scheduler）来调整优化器的超参数（例如学习率和动量）。 用户可以组合多个调度器来创建所需的参数调整策略。 在 [参数调度器教程](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/param_scheduler.html) 和 [参数调度器 API 文档](https://mmengine.readthedocs.io/zh_CN/latest/api/generated/mmengine.optim._ParamScheduler.html#mmengine.optim._ParamScheduler) 中查找更多信息。

```python
    dict(
        type='LinearLR',  # 使用线性学习率预热
        start_factor=0.001, # 学习率预热的系数
        by_epoch=False,  # 按 iteration 更新预热学习率
        begin=0,  # 从第一个 iteration 开始
        end=500),  # 到第 500 个 iteration 结束
    dict(
        type='MultiStepLR',  # 在训练过程中使用 multi step 学习率策略
        by_epoch=True,  # 按 epoch 更新学习率
        begin=0,   # 从第一个 epoch 开始
        end=12,  # 到第 12 个 epoch 结束
        milestones=[8, 11],  # 在哪几个 epoch 进行学习率衰减

by_epoch=True,  # 按 epoch 更新学习率
        gamma=0.1)  # 学习率衰减系数
]
```

### 钩子配置

用户可以在训练、验证和测试循环上添加钩子，以便在运行期间插入一些操作。配置中有两种不同的钩子字段，一种是 `default_hooks`，另一种是 `custom_hooks`。

`default_hooks` 是一个字典，用于配置运行时必须使用的钩子。这些钩子具有默认优先级，如果未设置，runner 将使用默认值。如果要禁用默认钩子，用户可以将其配置设置为 `None`。更多内容请看 [钩子教程](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/hook.html) 。

```python
    logger=dict(type='LoggerHook', interval=50),
    checkpoint=dict(type='CheckpointHook', interval=1),
    visualization=dict(type='DetVisualizationHook'))
```

`custom_hooks` 是一个列表。用户可以在这个字段中加入自定义的钩子。

```python
```

### 运行相关配置

```python

visualization=dict(type='DetVisualizationHook'))
```

```python
```

### 运行相关配置

```python
default_scope = 'mmdet'  # 默认的注册器域名，默认从此注册器域中寻找模块。请参考 https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/registry.html

    cudnn_benchmark=False,  # 是否启用 cudnn benchmark
    mp_cfg=dict(  # 多进程设置
        mp_start_method='fork',  # 使用 fork 来启动多进程。'fork' 通常比 'spawn' 更快，但可能存在隐患。请参考 https://github.com/pytorch/pytorch/issues/1355
        opencv_num_threads=0),  # 关闭 opencv 的多线程以避免系统超负荷
    dist_cfg=dict(backend='nccl'),  # 分布式相关设置
)

vis_backends = [dict(type='LocalVisBackend')]  # 可视化后端，请参考 https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/visualization.html
    type='LogProcessor',  # 日志处理器用于处理运行时日志
    window_size=50,  # 日志数值的平滑窗口

    by_epoch=True)  # 是否使用 epoch 格式的日志。需要与训练循环的类型保存一致。

log_level = 'INFO'  # 日志等级
load_from = None  # 从给定路径加载模型检查点作为预训练模型。这不会恢复训练。
resume = False  # 是否从 `load_from` 中定义的检查点恢复。 如果 `load_from` 为 None，它将恢复 `work_dir` 中的最新检查点。
```

## Iter-based 配置

MMEngine 的 Runner 除了基于轮次的训练循环（epoch）外，还提供了基于迭代（iteration）的训练循环。
要使用基于迭代的训练，用户应该修改 `train_cfg`、`param_scheduler`、`train_dataloader`、`default_hooks` 和 `log_processor`。
以下是将基于 epoch 的 RetinaNet 配置更改为基于 iteration 的示例：configs/retinanet/retinanet_r50_fpn_90k_coco.py

```python
# iter-based 训练配置
    _delete_=True,  # 忽略继承的配置文件中的值（可选）
    type='IterBasedTrainLoop',  # iter-based 训练循环
    max_iters=90000,  # 最大迭代次数
    val_interval=10000)  # 每隔多少次进行一次验证

# 将参数调度器修改为 iter-based
    dict(

type='IterBasedTrainLoop',  # iter-based 训练循环

    dict(
    dict(
        begin=0,
]

# 切换至 InfiniteSampler 来避免 dataloader 重启

# 将模型检查点保存间隔设置为按 iter 保存

# 将日志格式修改为 iter-based
```

## 配置文件继承

在 `config/_base_` 文件夹下有 4 个基本组件类型，分别是：数据集(dataset)，模型(model)，训练策略(schedule)和运行时的默认设置(default runtime)。许多方法，例如 Faster R-CNN、Mask R-CNN、Cascade R-CNN、RPN、SSD 能够很容易地构建出来。由 `_base_` 下的组件组成的配置，被我们称为 _原始配置(primitive)_。

对于同一文件夹下的所有配置，推荐**只有一个**对应的**原始配置**文件。所有其他的配置文件都应该继承自这个**原始配置**文件。这样就能保证配置文件的最大继承深度为 3。

为了便于理解，我们建议贡献者继承现有方法。例如，如果在 Faster R-CNN 的基础上做了一些修改，用户首先可以通过指定 `_base_ = ../faster_rcnn/faster-rcnn_r50_fpn_1x_coco.py` 来继承基础的 Faster R-CNN 结构，然后修改配置文件中的必要参数以完成继承。

如果你在构建一个与任何现有方法不共享结构的全新方法，那么可以在 `configs` 文件夹下创建一个新的例如 `xxx_rcnn` 文件夹。

更多细节请参考 [MMEngine 配置文件教程](https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/config.html) 。

通过设置 `_base_` 字段，我们可以设置当前配置文件继承自哪些文件。

当 `_base_` 为文件路径字符串时，表示继承一个配置文件的内容。

```python
```

当 `_base_` 是多个文件路径的列表时，表示继承多个文件。

```python
_base_ = [
]
```

如果需要检查配置文件，可以通过运行 `python tools/misc/print_config.py /PATH/TO/CONFIG` 来查看完整的配置。

### 忽略基础配置文件里的部分内容

'../_base_/schedules/schedule_1x.py', '../_base_/default_runtime.py'
]
```

有时，您也许会设置 `_delete_=True` 去忽略基础配置文件里的一些域内容。 您也许可以参照 [MMEngine 配置文件教程](https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/config.html) 来获得一些简单的指导。

在 MMDetection 里，例如为了改变  Mask R-CNN 的主干网络的某些内容：

```python
        depth=50,
```

基础配置的 `Mask R-CNN` 使用 `ResNet-50`，在需要将主干网络改成 `HRNet` 的时候，因为 `HRNet` 和 `ResNet` 中有不同的字段，需要使用 `_delete_=True` 将新的键去替换 `backbone` 域内所有老的键。

```python

rpn_head=dict(...),
```

```python

block='BASIC',
```

### 使用配置文件里的中间变量

配置文件里会使用一些中间变量，例如数据集里的 `train_pipeline`/`test_pipeline`。我们在定义新的 `train_pipeline`/`test_pipeline` 之后，需要将它们传递到 `data` 里。例如，我们想在训练或测试时，改变 Mask R-CNN 的多尺度策略 (multi scale strategy)，`train_pipeline`/`test_pipeline` 是我们想要修改的中间变量。

```python

    dict(

dict(
]
    dict(
]
```

我们首先定义新的 `train_pipeline`/`test_pipeline` 然后传递到 `data` 里。

同样的，如果我们想从 `SyncBN` 切换到 `BN` 或者 `MMSyncBN`，我们需要修改配置文件里的每一个  `norm_cfg`。

```python
    ...)
```

### 复用 \_base\_ 文件中的变量

    ...)
```

如果用户希望在当前配置中复用 base 文件中的变量，则可以通过使用 `{{_base_.xxx}}` 的方式来获取对应变量的拷贝。例如：

```python

a = {{_base_.model}}  # 变量 a 等于 _base_ 中定义的 model
```

## 通过脚本参数修改配置

当运行 `tools/train.py` 和 `tools/test.py` 时，可以通过 `--cfg-options` 来修改配置文件。

- 更新字典链中的配置

  可以按照原始配置文件中的 dict 键顺序地指定配置预选项。例如，使用 `--cfg-options model.backbone.norm_eval=False` 将模型主干网络中的所有 BN 模块都改为 `train` 模式。

- 更新配置列表中的键

  在配置文件里，一些字典型的配置被包含在列表中。例如，数据训练流程 `data.train.pipeline` 通常是一个列表，比如 `[dict(type='LoadImageFromFile'), ...]`。如果需要将 `'LoadImageFromFile'` 改成 `'LoadImageFromWebcam'`，需要写成下述形式： `--cfg-options data.train.pipeline.0.type=LoadImageFromNDArray`.

- 更新列表或元组的值

  如果要更新的值是列表或元组。例如，配置文件通常设置 `model.data_preprocessor.mean=[123.675, 116.28, 103.53]`. 如果需要改变这个键，可以通过 `--cfg-options model.data_preprocessor.mean="[127,127,127]"` 来重新设置。需要注意，引号 " 是支持列表或元组数据类型所必需的，并且在指定值的引号内**不允许**有空格。

## 配置文件名称风格

我们遵循以下样式来命名配置文件。建议贡献者遵循相同的风格。

```
```

文件名分为五个部分。 每个部分用`_`连接，每个部分内的单词应该用`-`连接。

- `{algorithm name}`: 算法的名称。 它可以是检测器名称，例如 `faster-rcnn`、`mask-rcnn` 等。也可以是半监督或知识蒸馏算法，例如 `soft-teacher`、`lad` 等等
- `{component names}`: 算法中使用的组件名称，如 backbone、neck 等。例如 `r50-caffe_fpn_gn-head` 表示在算法中使用 caffe 版本的 ResNet50、FPN 和 使用了 Group Norm 的检测头。
- `{training settings}`: 训练设置的信息，例如 batch 大小、数据增强、损失、参数调度方式和训练最大轮次/迭代。 例如：`4xb4-mixup-giou-coslr-100e` 表示使用 8 个 gpu 每个 gpu 4 张图、mixup 数据增强、GIoU loss、余弦退火学习率，并训练 100 个 epoch。
  缩写介绍:

  缩写介绍:
  - `{gpu x batch_per_gpu}`: GPU 数和每个 GPU 的样本数。`bN` 表示每个 GPU 上的 batch 大小为 N。例如 `4x4b` 是 4 个 GPU 每个 GPU 4 张图的缩写。如果没有注明，默认为 8 卡每卡 2 张图。
  - `{schedule}`: 训练方案，选项是 `1x`、 `2x`、 `20e` 等。`1x`  和 `2x` 分别代表 12 epoch 和 24 epoch，`20e` 在级联模型中使用，表示 20 epoch。对于 `1x`/`2x`，初始学习率在第 8/16 和第 11/22 epoch 衰减 10 倍；对于 `20e` ，初始学习率在第 16 和第 19 epoch 衰减 10 倍。
- `{training dataset information}`: 训练数据集，例如 `coco`, `coco-panoptic`, `cityscapes`, `voc-0712`, `wider-face`。
- `{testing dataset information}` (可选): 测试数据集，用于训练和测试在不同数据集上的模型配置。 如果没有注明，则表示训练和测试的数据集类型相同。

# Frequently asked questions

We list some common troubles faced by many users and their corresponding
solutions here. Feel free to enrich the list if you find any frequent issues
and have ways to help others to solve them. If the contents here do not cover
your issue, please create an issue using the
[provided templates](https://github.com/open-mmlab/mmagic/issues/new/choose)
and make sure you fill in all required information in the template.

## FAQ

**Q1**: “xxx: ‘yyy is not in the zzz registry’”.

**A1**: The registry mechanism will be triggered only when the file of the module is imported. So you need to import that file somewhere.

**Q2**: What's the folder structure of xxx dataset?

**A2**: You can make sure the folder structure is correct following tutorials of [dataset preparation](user_guides/dataset_prepare.md).

**Q3**: How to use LMDB data to train the model?

**A3**:  You can use scripts in `tools/data` to make LMDB files. More details are shown in tutorials of [dataset preparation](user_guides/dataset_prepare.md).

**Q4**: Why `MMCV==xxx is used but incompatible` is raised when import I try to import `mmgen`?

**A4**:
This is because the version of MMCV and MMGeneration are incompatible. Compatible MMGeneration and MMCV versions are shown as below. Please choose the correct version of MMCV to avoid installation issues.

| MMGeneration version |   MMCV version   |
| :------------------: | :--------------: |
|        master        | mmcv-full>=2.0.0 |

Note: You need to run `pip uninstall mmcv` first if you have mmcv installed.
If mmcv and mmcv-full are both installed, there will be `ModuleNotFoundError`.

**Q5**: How can I ignore some fields in the base configs?

**A5**:
Sometimes, you may set `_delete_=True` to ignore some of fields in base configs.
You may refer to [MMEngine](https://github.com/open-mmlab/mmengine/blob/main/docs/en/advanced_tutorials/config.md#delete-key-in-dict) for simple illustration.

You may have a careful look at [this tutorial](https://github.com/open-mmlab/mmengine/blob/main/docs/en/advanced_tutorials/config.md) for better understanding of this feature.

**Q6**:: How can I use intermediate variables in configs?

**A6**:
Some intermediate variables are used in the config files, like `train_pipeline`/`test_pipeline` in datasets.

# Look at boundary: A boundary-aware face alignment algorithm

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_cvpr_2018/html/Wu_Look_at_Boundary_CVPR_2018_paper.html">WFLW (CVPR'2018)</a></summary>

```bibtex
@inproceedings{wu2018look,
  title={Look at boundary: A boundary-aware face alignment algorithm},
  author={Wu, Wayne and Qian, Chen and Yang, Shuo and Wang, Quan and Cai, Yici and Zhou, Qiang},
  pages={2129--2138},
}
```

# Preparing Denoising Dataset

```bibtex
}
```

The test datasets (Set12, BSD68, CBSD68, Kodak, McMaster, Urban100) can be download from [here](https://drive.google.com/file/d/1mwMLt-niNqcQpfN_ZduG9j4k6P_ZkOl0/).

The folder structure should look like:

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── denoising_gaussian_test
|   |   ├── Set12
|   |   ├── BSD68
|   |   ├── CBSD68
|   |   ├── Kodak
|   |   ├── McMaster
|   |   ├── Urban100
```

# <a href='https://mmagic.readthedocs.io/en/latest/'>English</a>

# <a href='https://mmagic.readthedocs.io/zh_CN/latest/'>简体中文</a>

# Squeeze-and-excitation networks

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_cvpr_2018/html/Hu_Squeeze-and-Excitation_Networks_CVPR_2018_paper">SEResNet (CVPR'2018)</a></summary>

```bibtex
@inproceedings{hu2018squeeze,
  title={Squeeze-and-excitation networks},
  author={Hu, Jie and Shen, Li and Sun, Gang},
  pages={7132--7141},
}
```

Convolutional neural networks are built upon the convolution operation, which extracts informative features by fusing spatial and channel-wise information together within local receptive fields. In order to boost the representational power of a network, several recent approaches have shown the benefit of enhancing spatial encoding. In this work, we focus on the channel relationship and propose a novel architectural unit, which we term the "Squeeze-and-Excitation" (SE) block, that adaptively recalibrates channel-wise feature responses by explicitly modelling interdependencies between channels. We demonstrate that by stacking these blocks together, we can construct SENet architectures that generalise extremely well across challenging datasets. Crucially, we find that SE blocks produce significant performance improvements for existing state-of-the-art deep architectures at minimal additional computational cost. SENets formed the foundation of our ILSVRC 2017 classification submission which won first place and significantly reduced the top-5 error to 2.251%, achieving a ∼25% relative improvement over the winning entry of 2016. Code and models are available at https: //github.com/hujie-frank/SENet.

<img src="https://user-images.githubusercontent.com/15977946/146540423-a7e8ae12-3b3e-447e-848b-682c06c1126e.png">
</div>

# 如何使用SSE来优化算子核心

## 一：准备

### 1.背景资料

​	SSE 全称Intel® Streaming SIMD Extensions (Intel® SSE),本质是Intel公司封装汇编语句提供的底层操作指令函数集。同样属于底层操作指令集的还有著名的Intel® AVX(Advanced Vector Extensions),  及 Intel® AVX2(Intel® Advanced Vector Extensions 2)。基于同样原理封装的还有Arm 对应Arm Intrinsic，MIPS中对应MIPS Intrinsic。

​	SSE的版本包含：SSE/SSE2/SSE3/SSE4.1/SSE4.2。下文中在描述CPU特性上统称为SSE系列指令集。在描述具体使用指令函数中的CPUID Flags，才会具体区分SSE不同版本。

​	自从MSVC不再支持x64的汇编指令后（虽然可以强制使用，但不推荐不安全）。SSE，AVX等成为MSVC 支持的最佳底层优化方法。

​	本文将从SSE的使用出发，以ncnn实现为例，展示如何使用SSE优化深度学习中算子。

​	优化算子工作需要三方面的准备事项：

- 测试正确的原生代码
- 快速测试验证环境
- 基准统计程序

### 2.确认硬件是否支持SSE

​	在开始SSE优化之前，首先请确保您硬件支持SSE指令集，对于大多数Intel CPU都支持SSE指令集。但在各种系统环境下，查看方式不同。我们有：

#### 1.windows环境

​	windows环境下推荐简单使用GPU-Z来检测当前处理器是否支持SSE扩展。在GPU-Z官网下载后，运行，在“处理器”-“CPU支持的特性”项目下，若包含SSE系列指令集，即当前CPU支持SSE。

#### 2.Linux环境和类Unix环境

​	Linux环境和类Unix环境下，使用查看cpuinfo文件来确认CPU特性；

```shell
cat  /proc/cpuinfo
...
flags: *** sse sse2 ***	#在cpu flags中即可检查是否支持sse扩展
```

#### 2.macOS环境：

```shell
...
```

​	macOS本质是像Unix环境，所以同样使用sysctl 来查看CPU特性.(注意Mac的 M1 M2系列芯片是arm架构，不支持SSE)

```shell
sysctl machdep.cpu		# 结果同Linux环境
```

## 二：编写原生代码

​	使用SSE来优化算法的过程本质就是代码重构的一种情况。代码重构的首要条件是完成完备的代码行为测试集合。所以，这部分将从测试代码的编写开始。

​	其次优化过程的目标是调优某些性能指标的过程。所以第二部分将讨论性能指标的选定和优先级；

### 1.编写测试代码

​	在大多数情况下，看到这篇文章的人肯定是比笔者更会写算法，所以我在这里只谈一些编写测试的注意事项（这里的测试指验证你算法满足你的要求所编写的代码行为，跟其他人无关）。

​	编写测试代码主要注意事项：

- 思考如何构造基础数据结构才能满足算法行为的输入要求。举例来说，如果你准备为ncnn贡献算法，请阅读ncnn中关于Mat结构的函数。最好编写相关测试来验证该数据结构满足你的需要。（笔者的建议是可以先从简单结构来验证，比如需要做一个支持f32任意大小的矩阵加法算子，可以先从支持固定矩阵int8类型的加法开始编写测试代码）。
- 保持结果的正确性。首先考虑，你所编写的原生代码行为上，是否满足你所需要的结果（不论这个结果是手算的，numpy算的或者pytorch算的）。其次要考虑，结果在内存上结构如何排布。以ncnn为例，思考你的结果该如何放入到一个Mat中，Mat的size该如何设定。（在后续SSE优化中，我们将多次以原生代码结果作为target结果，验证每次优化后的正确性，原生代码能够稳定输出正确结果非常重要）
- 不用过早考虑算法的完备性，应该随着每次测试结果的正确来迭代重构算法和测试代码。二者同样重要。如果能够自动化测试，请尽量让一个简单的脚本执行来完成所有你慢吞吞的命令行。

### 2.考虑性能指标

​	性能指标的主要作用是随着每次优化的迭代，告诉我们所采取的措施在什么方面取得效果，是正面优化还是负面优化。

​	性能指标很多，包括吞吐量，还有类似计算稳定度，时间延迟，视频方面还有fps 等等。无法确认有效的性能指标也是大多数优化算法的困难点之一。	

​	随着简单粗暴地叠晶体管数量来解决电脑运行问题，性能指标似乎变得越来越不重要。这是一种错误观念，如果在单核上编写非常烂的代码，增加N个核心只是把烂代码重复N次而已。另一方面，性能指标有着客观性，在开发板上和集群设备上运行同样的算法，性能指标的优先级也不一样。但是，我认为应当满足最基础性能指标有这两个：

- **吞吐量**：算法在单位时间内执行的次数，用Gflops表示，该值越大越好（也可以认为执行同样算法所平均占用的时间，时间越短越好）；
- **性能衰退**：即随着数据规模的增加，Gflops在不同数据规模下的波动情况。更低的离散程度意味着吞吐量保持在一定范围内不发生变化。

​	其余有效的性能指标应当由业务环境和任务需求决定。负责技术基础设施建设的算法工程师，一方面应该理解业务所需求中的最高优先级，另一方面也应该追求做到更好。

​	以SSE优化算法，本质上是重构的迭代过程。不用在初期就考虑如何达到最大性能指标，而是应该考虑每次迭代中带来一定量的性能优化。

## 三：理解SSE

​	SSE主要由SSE基础数据类型 及 针对性的SSE操作函数构成。前文提到，SSE是针对汇编语句的封装，所以本身不具备错误检查和错误处理（错误检查和错误处理一般由编译器完成）。使用不当的话，诸如segmentation fault之内指针指向不存在的内存错误非常常见。我在此处建议：<u>使用SSE优化之前，确保理解代码指针位置和移动原理，原生代码已经完成测试，输出结果正确。</u>

### 1.SSE数据类型

​	SSE数据类型形如：

```c++
__m<bit><type>			 //__m适用代表申请mm寄存器
    					// bit 代表数据类型的字节长度，在SSE中为128 或 64

```c++
    					// 默认type为单精度浮点（f32），其余为int 或double
// 另外要注意所有SSE的类型除__m128和__m64外，随着版本更新有不同的类型，建议根据需要且确定硬件性能后选择合适的类型
// 举例如下：
__m128					//4xf32 含有4个单精度浮点数；SSE
__m64    				//4xf32 含有2个单精度浮点数；SSE
__m128i   				//8个int类型（8x16)		 ；SSE3
__m128d					//2个double类型(2x64)
```

### 2.SSE内联函数结构

​	SSE内联函数在线查询：[Intel® Intrinsics Guide](https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html#techs=SSE,SSE2,SSE3,SSSE3,SSE4_1,SSE4_2) 在此 单个指令的结构如下：

- Synopsis ：摘要。描述指令的接口定义，需要引入的头文件，对应的指令，CPU必须支持的标志；
- Description：描述该指令的行为；
- Operation：逻辑层面描述指令行为；
- Performance：在不同架构中所需要的延迟和执行所需要的时钟周期数（CPI）。

​	值得指出的的是此处默认使用小端存储，即左边为高位，右边为低位。

​	相似的内联函数有很多，在使用时候一定要注意Operation中的逻辑满足您的要求。

​	另外，在ncnn中，ncnn已经将部分SSE内联函数以NCNN内联的方式封装。在为NCNN添加SSE优化的算法的过程中，请首先考虑搜索“NCNNINLINE”宏封装的SSE函数。

## 四：样例

### 1.一个简单的样例：4x4矩阵乘法

## 四：样例

​	矩阵乘法方面，已经有很多出色的成果。值得一读的比如[how-to-optimize-gemm](https://github.com/flame/how-to-optimize-gemm)，及 [以Arm Intrinsic优化矩阵乘法](https://github.com/tpoisonooo/how-to-optimize-gemm)。我建议感兴趣同学参考和学习这两份项目，来探究如何从0到1优化一份算法；

​	矩阵乘法原理很简单：

​	假设有A，B两个矩阵，如下：
$$
A_{[4][4]} =  
\begin{bmatrix}
	a_0 & a_4 & a_8 & a_{12} \\
	a_1 & a_5 & a_9 & a_{13} \\
	a_2 & a_6 & a_{10} & a_{14} \\
	a_3 & a_7 & a_{11} & a_{15} 
\end{bmatrix}
~~
B_{[4][4]} =  
	b_0 & b_4 & b_8 & b_{12} \\
	b_1 & b_5 & b_9 & b_{13} \\
	b_2 & b_6 & b_{10} & b_{14} \\
	b_3 & b_7 & b_{11} & b_{15} 
~~
C_{[4][4]} =  
	c_0 & c_4 & c_8 & c_{12} \\
	c_1 & c_5 & c_9 & c_{13} \\
	c_2 & c_6 & c_{10} & c_{14} \\
	c_3 & c_7 & c_{11} & c_{15} 
$$
​	对于C 矩阵的第一列，我们有：
$$
c_0 = a_0b_0 + a_4b_1 + a_8b_2 + a_{12}b_3 \\

c_1 & c_5 & c_9 & c_{13} \\
$$
$$
	c_1 = a_1b_0 + a_5b_1 + a_9b_2 + a_{13}b_3 \\
	c_2 = a_2b_0 + a_6b_1 + a_{10}b_2 + a_{14}b_3 \\
	c_3 = a_3b_0 + a_7b_1 + a_{11}b_2 + a_{15}b_3
$$

#### 1.编写测试代码和基准测量程序

​	在该样例中，测试代码很容易编写出来，我们只需要初始化4x4的二维数组，并返回指针即可。此时，可以不考虑泛用性，初始化为固定值即可。

```c
// <代码片段>
...
float A[16] = {0.0f};			// 此处已经将输入和输出的矩阵默认展开成im2col 后的单行（inch = 1） 宽度为h*w = 16的矩阵
float B[16] = {0.0f};
float C[16] = {0.0f};
matrix_init_rand(A, 4, 4);		// 随机初始化A数组
matrix_init_rand(B, 4, 4);		// 随机初始化B数组
```

​	编写验证正确性的测试代码。

```c
// <代码片段>
...
float T[16] = {...};			// Target即为预测的C的结果数组，可用numpy或者纸笔计算
...
float error = 0.0001;
bool CheckAuc(T, C, error);		
// 注意：float在计算机中不能完全表示，只能使用绝对误差的判别方法。gtest等测试框架的EXCEPT宏无法处理1.234e5这样结构的float数的对比。
```

​	同样，编写计算耗时的基准测量代码，此处使用1000次操作所占的平均时间来作为基准。

```c
// <代码片段>
...
const int loop = 1000;

```

```c
// <代码片段>
...
clock_gettime_(CLOCK_REALTIME, &time_start);
for(init i = 0; i < loop; i++)
{
	matirx_mult_native(C, A, B);
}
clock_gettime_(CLOCK_REALTIME, &time_end);
clocks_c = (time_end.tv_sec - time_start.tv_sec) * 1000000 +  (time_end.tv_sec - time_start.tv_sec) /1000;
```

#### 2.编写原生代码

​	编写原生代码，使得正确性测试能够通过。

```c
// <代码片段>
static void matirx_mult_native(float *C, float *A, float *B)
{
    for(int i_idx = 0; i_idx < 4; i_idx++)
    {
        for(int j_idx = 0; j_idx < 4; j_idx++)
        {
           for(int k_idx = 0; k_idx < 4; k_idx++)
           {
               C[4*j_idx + i_idx] += A[4*k_idx + i_idx] * B[4*j_idx + k_idx];
           }
        }
    }
}
```

#### 3.优化原生代码

​	注意到上述代码中，先取c0 - c3 的计算作为样例考虑：
$$
	c_0 = a_0b_0 + a_4b_1 + a_8b_2 + a_{12}b_3   \\
	c_1 = a_1b_0 + a_5b_1 + a_9b_2 + a_{13}b_3    \\

}
        }
    }
}
```

$$
$$

##### 1.装载寄存器

- 考虑竖排a0-a1-a2-a3 为4个f32 数据，又因为SSE可以申请mm寄存器，单次保存128bit，那么不妨把a0-a4保存在寄存器中，

- 对于b0-b3 则是，单次读取一个值，能够重复用4次，不妨考虑b0 重复4次，排满单个128bit的mm寄存器；

- 同理把c0-c3也放入寄存器，从列方向上考虑，取名为_c0 

  ```c++
  _m128 _a0 = _mm_load_ps(a_ptr);			//a0 -a1 -a2 -a3
  _m128 _a1 = _mm_load_ps(a_ptr + 4);		//a4 -a5 -a6 -a7
  _m128 _a2 = _mm_load_ps(a_ptr + 8);		//a8 -a9 -a10-a11
  _m128 _a3 = _mm_load_ps(a_ptr + 12);	//a12-a13-a14-a15
  
  _m128 _b0 = _mm_load_ps1(b_ptr);		// b0 - b0 - b0 - b0
  _m128 _b1 = _mm_load_ps1(b_ptr + 4);	// b1 - b1 - b1 - b1
  _m128 _b2 = _mm_load_ps1(b_ptr + 8);	// b2 - b2 - b2 - b2
  _m128 _b3 = _mm_load_ps1(b_ptr + 12);	// b3 - b3 - b3 - b3
  ```

##### 2.编写第一列的计算结果

​	对于_a0 -\_a3 数据与\_b0 数据相乘 ，有：

_m128 _b2 = _mm_load_ps1(b_ptr + 8);	// b2 - b2 - b2 - b2
  ```

```c++
// 保存结果新建一个_c0 作为临时变量
_m128 _c0 = _mm_set_ps1(0.0f);
_c0 = _mm_mul_ps(_a0, _b0);
_c0 = _mm_add_ps(_mm_mul_ps(_a1, _b1),_c0);
_c0 = _mm_add_ps(_mm_mul_ps(_a2, _b2),_c0);
_c0 = _mm_add_ps(_mm_mul_ps(_a3, _b3),_c0);
// 把 _sum0存会以c指针开头的内存中，完美！
_mm_store_ps(c_ptr, _c0);
```

##### 3.将单列输出扩展到所有列：

​	我们针对剩下的c中的c1 列也做相同的操作： 对于C1 列 有：
$$
	c_4 = a_0b_4 + a_4b_5 + a_8b_6 + a_{12}b_7 \\
	c_5 = a_1b_4 + a_5b_5 + a_9b_6 + a_{13}b_7 \\
	c_6 = a_2b_4 + a_6b_5 + a_{10}b_6 + a_{14}b_7 \\
	c_7 = a_3b_4 + a_7b_5 + a_{11}b_6 + a_{15}b_7
$$

```c++
// a 系列不变 b系列指针+1
_m128 _b4 = _mm_load_ps1(b_ptr + 1);		// b4 - b4 - b4 - b4
_m128 _b5 = _mm_load_ps1(b_ptr + 4 + 1);	// b5 - b5 - b5 - b5
_m128 _b6 = _mm_load_ps1(b_ptr + 8 + 1);	// b6 - b6 - b6 - b6
_m128 _b7 = _mm_load_ps1(b_ptr + 12+ 1);	// b7 - b7 - b7 - b7

_m128 _c1 = _mm_set_ps1(0.0f);
_c1 = _mm_mul_ps(_a0, _b4);
_c1 = _mm_add_ps(_mm_mul_ps(_a1, _b5),_c1);
_c1 = _mm_add_ps(_mm_mul_ps(_a2, _b6),_c1);
_c1 = _mm_add_ps(_mm_mul_ps(_a3, _b7),_c1);
_mm_store_ps(c_ptr, _c1);
```

​	此时我们发现，对于C1列的操作与C0列及其相似，只不过是b_ptr的指针发生移动，不妨将其放到同一个循环中，有：

```C++
// a 系列不变
_m128 _a0 = _mm_load_ps(a_ptr);			//a0 -a1 -a2 -a3
_m128 _a1 = _mm_load_ps(a_ptr + 4);		//a4 -a5 -a6 -a7
_m128 _a2 = _mm_load_ps(a_ptr + 8);		//a8 -a9 -a10-a11
_m128 _a3 = _mm_load_ps(a_ptr + 12);	//a12-a13-a14-a15

for(int i = 0; i < 4; i++)
{
    _m128 _b0 = _mm_load_ps1(b_ptr);		// b0 - b0 - b0 - b0
    _m128 _b1 = _mm_load_ps1(b_ptr + 4);	// b1 - b1 - b1 - b1
    _m128 _b2 = _mm_load_ps1(b_ptr + 8);	// b2 - b2 - b2 - b2

{
    _m128 _b3 = _mm_load_ps1(b_ptr + 12);	// b3 - b3 - b3 - b3
    
    _m128 _ci = _mm_set_ps1(0.0f);
    _ci = _mm_mul_ps(_a0, _b0);
    _ci = _mm_add_ps(_mm_mul_ps(_a1, _b1),_ci);
    _ci = _mm_add_ps(_mm_mul_ps(_a2, _b2),_ci);
    _ci = _mm_add_ps(_mm_mul_ps(_a3, _b3),_ci);
    
    _mm_store_ps(c_ptr, _ci);
    
    b_ptr += 1;				// 移动b_ptr
    c_ptr += 4;				// 移动保存内存的c_ptr
}
```

### 2.NCNN中以SSE优化算子的注意事项

#### 1.线程与openmp

​	以上计算Benchmark 和 SSE优化的方法大多集中在单个核心中，但是在实际使用ncnn中，ncnn使用Option opt 中提供的num_threads 给openmp赋值，以实现多线程并行化，同时运行在多个核心上。

```c++
#pragma omp parallel for num_threads(opt.num_threads)
```

​	在优化成SSE代码的初期，可以考虑锁定为单线程，或者直接不用考虑线程的影响，仅对单核以SSE优化，保证单核的结果正确后，再加上opt的多线程进行结果测试。

#### 2.展开循环

​	在实际ncnn实现的原生代码的算法中，循环是非常常见的。针对以SSE优化这类循环，遵循非常简单的原则：循环中，迭代器等于零时刻，整个输出的结果也是正确的。

```

​	那么，在我们使用SSE优化过程中，不妨以迭代器等于零的时刻，函数计算结果作为此时目标结果。在此基础上再利用SSE优化代码。与目标结果核对正确以后，再进一步去考虑迭代器等于1的情况（重复这个过程直到迭代器达到最大值）。在迭代器的每个元素下，SSE优化出的代码都与结果相等，那么我们可以说，该次优化是正确性，且完全覆盖了需执行代码。（一般来说不用考虑到最大值，根据数学归纳法，n有效，n+1有效，那么n的序列都是有效的）

## 五：总结

​	本文描述SSE的使用及以4x4矩阵乘法的样例来优化SSE代码。

​	值得注意的是，SSE只是128bit数据宽度的指令集，但是也可以用来模拟256bit 和 512bit数据宽度，来实现以pack4拼接成pack8，甚至pack16的做法，只不过在输出结果管理上更加繁琐而已。感兴趣的同学可以尝试一下。

## 六：引用

1. [SSE指令扩展快查](https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html#techs=SSE,SSE2,SSE3,SSSE3,SSE4_1,SSE4_2)；
2. 浮点性能基准计算-[浮点峰值那些事儿](https://zhuanlan.zhihu.com/p/28226956)
3. 硬件性能基准测试计算样例：[M1芯片搞数据科学好使吗？5种基准测试给你答案](https://mp.weixin.qq.com/s/2N5cl_Z1MRF8dfbRo-sb4A)
4. 讨论矩阵乘法如何优化的系列论文：[how-to-optimized-gemm](https://github.com/flame/how-to-optimize-gemm/wiki)

5. 讨论以Arm Intrinsic 优化gemm的系列文章：[OpenBLAS gemm从零入门](https://zhuanlan.zhihu.com/p/65436463)

# Inference with existing models

MMAction2 provides pre-trained models for video understanding in [Model Zoo](../modelzoo.md).
This note will show **how to use existing models to inference on given video**.

As for how to test existing models on standard datasets, please see this [guide](./train_test.md#test)

## Inference on a given video

MMAction2 provides high-level Python APIs for inference on a given video:

- [init_recognizer](mmaction.apis.init_recognizer): Initialize a recognizer with a config and checkpoint
- [inference_recognizer](mmaction.apis.inference_recognizer): Inference on a given video

Here is an example of building the model and inference on a given video by using Kinitics-400 pre-trained checkpoint.

```{note}
If you use mmaction2 as a 3rd-party package, you need to download the conifg and the demo video in the example.

Run 'mim download mmaction2 --config tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb --dest .' to download the required config.

Run 'wget https://github.com/open-mmlab/mmaction2/blob/main/demo/demo.mp4' to download the desired demo video.
```

```python
from mmaction.apis import inference_recognizer, init_recognizer

config_path = 'configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb.py'
checkpoint_path = 'https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb/tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb_20220906-2692d16c.pth' # can be a local path
img_path = 'demo/demo.mp4'   # you can specify your own picture path

# build the model from a config file and a checkpoint file
model = init_recognizer(config_path, checkpoint_path, device="cpu")  # device can be 'cuda:0'
# test a single image
result = inference_recognizer(model, img_path)
```

```

`result` is a dictionary containing `pred_scores`.

An action recognition demo can be found in [demo/demo.py](https://github.com/open-mmlab/mmaction2/blob/main/demo/demo.py).

# Top-down regression-based pose estimation

Top-down methods divide the task into two stages: object detection, followed by single-object pose estimation given object bounding boxes. At the 2nd stage, regression based methods directly regress the keypoint coordinates given the features extracted from the bounding box area, following the paradigm introduced in [Deeppose: Human pose estimation via deep neural networks](http://openaccess.thecvf.com/content_cvpr_2014/html/Toshev_DeepPose_Human_Pose_2014_CVPR_paper.html).

<img src="https://user-images.githubusercontent.com/15977946/146515040-a82a8a29-d6bc-42f1-a2ab-7dfa610ce363.png">
</div>

### WFLW Dataset

Result on WFLW test set

| Model                                                           | Input Size | NME  |                              ckpt                              |                              log                              |

| :-------------------------------------------------------------- | :--------: | :--: | :------------------------------------------------------------: | :-----------------------------------------------------------: |
| [ResNet-50](/configs/face_2d_keypoint/topdown_regression/wflw/td-reg_res50_8xb64-210e_wflw-256x256.py) |  256x256   | 4.88 | [ckpt](https://download.openmmlab.com/mmpose/face/deeppose/deeppose_res50_wflw_256x256-92d0ba7f_20210303.pth) | [log](https://download.openmmlab.com/mmpose/face/deeppose/deeppose_res50_wflw_256x256_20210303.log.json) |
| [ResNet-50+WingLoss](/configs/face_2d_keypoint/topdown_regression/wflw/td-reg_res50_wingloss_8xb64-210e_wflw-256x256.py) |  256x256   | 4.67 | [ckpt](https://download.openmmlab.com/mmpose/face/deeppose/deeppose_res50_wflw_256x256_wingloss-f82a5e53_20210303.pth) | [log](https://download.openmmlab.com/mmpose/face/deeppose/deeppose_res50_wflw_256x256_wingloss_20210303.log.json) |

| [ResNet-50+SoftWingLoss](/configs/face_2d_keypoint/topdown_regression/wflw/td-reg_res50_softwingloss_8xb64-210e_wflw-256x256.py) |  256x256   | 4.44 | [ckpt](https://download.openmmlab.com/mmpose/face/deeppose/deeppose_res50_wflw_256x256_softwingloss-4d34f22a_20211212.pth) | [log](https://download.openmmlab.com/mmpose/face/deeppose/deeppose_res50_wflw_256x256_softwingloss_20211212.log.json) |

# 推理

## 介绍

我们提供了多模态/单模态（基于激光雷达/图像）、室内/室外场景的 3D 检测和 3D 语义分割样例的脚本，预训练模型可以从 [Model Zoo](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/docs/zh_cn/model_zoo.md) 下载。我们也提供了 KITTI、SUN RGB-D、nuScenes 和 ScanNet 数据集的预处理样本数据，你可以根据我们的预处理步骤使用任何其它数据。

## 测试

### 3D 检测

#### 点云样例

在点云数据上测试 3D 检测器，运行：

```shell
python demo/pcd_demo.py ${PCD_FILE} ${CONFIG_FILE} ${CHECKPOINT_FILE} [--device ${GPU_ID}] [--score-thr ${SCORE_THR}] [--out-dir ${OUT_DIR}] [--show]
```

点云和预测 3D 框的可视化结果会被保存在 `${OUT_DIR}/PCD_NAME`，它可以使用 [MeshLab](http://www.meshlab.net/) 打开。注意如果你设置了 `--show`，通过 [Open3D](http://www.open3d.org/) 可以在线显示预测结果。

在 KITTI 数据上测试 [PointPillars 模型](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-car/hv_pointpillars_secfpn_6x8_160e_kitti-3d-car_20220331_134606-d42d15ed.pth)：

```shell
python demo/pcd_demo.py demo/data/kitti/000008.bin configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-car.py ${CHECKPOINT_FILE} --show

```shell
```

在 SUN RGB-D 数据上测试 [VoteNet 模型](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/votenet/votenet_16x8_sunrgbd-3d-10class/votenet_16x8_sunrgbd-3d-10class_20210820_162823-bf11f014.pth)：

```shell
python demo/pcd_demo.py demo/data/sunrgbd/sunrgbd_000017.bin configs/votenet/votenet_8xb16_sunrgbd-3d.py ${CHECKPOINT_FILE} --show
```

#### 单目 3D 样例

在图像数据上测试单目 3D 检测器，运行：

```shell
python demo/mono_det_demo.py ${IMAGE_FILE} ${ANNOTATION_FILE} ${CONFIG_FILE} ${CHECKPOINT_FILE} [--device ${GPU_ID}] [--out-dir ${OUT_DIR}] [--show]
```

`ANNOTATION_FILE` 需要提供 3D 到 2D 的仿射矩阵（相机内参矩阵），可视化结果会被保存在 `${OUT_DIR}/PCD_NAME`，其中包括图像以及预测 3D 框在图像上的投影。

```

在 KITTI 数据上测试 [PGD 模型](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pgd/pgd_r101_caffe_fpn_gn-head_3x4_4x_kitti-mono3d/pgd_r101_caffe_fpn_gn-head_3x4_4x_kitti-mono3d_20211022_102608-8a97533b.pth)：

```shell
python demo/mono_det_demo.py demo/data/kitti/000008.png demo/data/kitti/000008.pkl  configs/pgd/pgd_r101-caffe_fpn_head-gn_4xb3-4x_kitti-mono3d.py ${CHECKPOINT_FILE}  --show --cam-type CAM2 --score-thr 8
```

**注意**： PGD 方法的预测框分数并不是在 (0, 1) 之间

在 nuScenes 数据上测试 [FCOS3D 模型](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_finetune/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_finetune_20210717_095645-8d806dc2.pth)：

```shell

```shell
python demo/mono_det_demo.py demo/data/nuscenes/n015-2018-07-24-11-22-45+0800__CAM_BACK__1532402927637525.jpg demo/data/nuscenes/n015-2018-07-24-11-22-45+0800.pkl  configs/fcos3d/fcos3d_r101-caffe-dcn_fpn_head-gn_8xb2-1x_nus-mono3d_finetune.py ${CHECKPOINT_FILE}  --show --cam-type CAM_BACK
```

**注意**： 当对翻转图像可视化单目 3D 检测结果是，相机内参矩阵也应该相应修改。在 PR [#744](https://github.com/open-mmlab/mmdetection3d/pull/744) 中可以了解更多细节和示例。

#### 多模态样例

在多模态数据（通常是点云和图像）上测试 3D 检测器，运行：

```shell
python demo/multi_modality_demo.py ${PCD_FILE} ${IMAGE_FILE} ${ANNOTATION_FILE} ${CONFIG_FILE} ${CHECKPOINT_FILE} [--device ${GPU_ID}] [--score-thr ${SCORE_THR}] [--out-dir ${OUT_DIR}] [--show]
```

`ANNOTATION_FILE` 需要提供 3D 到 2D 的仿射矩阵，可视化结果会被保存在 `${OUT_DIR}/PCD_NAME`，其中包括点云、图像、预测的 3D 框以及它们在图像上的投影。

在 KITTI 数据上测试 [MVX-Net 模型](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/mvxnet/mvxnet_fpn_dv_second_secfpn_8xb2-80e_kitti-3d-3class/mvxnet_fpn_dv_second_secfpn_8xb2-80e_kitti-3d-3class-8963258a.pth)：

```shell
python demo/multi_modality_demo.py demo/data/kitti/000008.bin demo/data/kitti/000008.png demo/data/kitti/000008.pkl configs/mvxnet/mvxnet_fpn_dv_second_secfpn_8xb2-80e_kitti-3d-3class.py ${CHECKPOINT_FILE} --cam-type CAM2 --show
```

在 SUN RGB-D 数据上测试 [ImVoteNet 模型](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/imvotenet/imvotenet_stage2_16x8_sunrgbd-3d-10class/imvotenet_stage2_16x8_sunrgbd-3d-10class_20210819_192851-1bcd1b97.pth)：

```shell
python demo/multi_modality_demo.py demo/data/sunrgbd/000017.bin demo/data/sunrgbd/000017.jpg demo/data/sunrgbd/sunrgbd_000017_infos.pkl configs/imvotenet/imvotenet_stage2_8xb16_sunrgbd-3d.py ${CHECKPOINT_FILE} --cam-type CAM0 --show --score-thr 0.6
```

在 NuScenes 数据上测试 [BEVFusion 模型](https://drive.google.com/file/d/1QkvbYDk4G2d6SZoeJqish13qSyXA4lp3/view?usp=share_link)

```shell

```

```shell
python demo/multi_modality_demo.py demo/data/nuscenes/n015-2018-07-24-11-22-45+0800__LIDAR_TOP__1532402927647951.pcd.bin demo/data/nuscenes/ demo/data/nuscenes/n015-2018-07-24-11-22-45+0800.pkl projects/BEVFusion/configs/bevfusion_voxel0075_second_secfpn_8xb4-cyclic-20e_nus-3d.py ${CHECKPOINT_FILE} --cam-type all --score-thr 0.2 --show
```

### 3D 分割

在点云数据上测试 3D 分割器，运行：

```shell
python demo/pc_seg_demo.py ${PCD_FILE} ${CONFIG_FILE} ${CHECKPOINT_FILE} [--device ${GPU_ID}] [--out-dir ${OUT_DIR}] [--show]
```

可视化结果会被保存在 `${OUT_DIR}/PCD_NAME`，其中包括点云以及预测的 3D 分割掩码。

在 ScanNet 数据上测试 [PointNet++ (SSG) 模型](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointnet2/pointnet2_ssg_16x2_cosine_200e_scannet_seg-3d-20class/pointnet2_ssg_16x2_cosine_200e_scannet_seg-3d-20class_20210514_143644-ee73704a.pth)：

```shell

```shell
python demo/pcd_seg_demo.py demo/data/scannet/scene0000_00.bin configs/pointnet2/pointnet2_ssg_2xb16-cosine-200e_scannet-seg.py ${CHECKPOINT_FILE} --show
```

# How to test rewritten models

After you create a rewritten model using our [rewriter](support_new_model.md), it's better to write a unit test for the model to validate if the model rewrite would come into effect. Generally, we need to get outputs of the original model and rewritten model, then compare them. The outputs of the original model can be acquired directly by calling the forward function of the model, whereas the way to generate the outputs of the rewritten model depends on the complexity of the rewritten model.

## Test rewritten model with small changes

If the changes to the model are small (e.g., only change the behavior of one or two variables and don't introduce side effects), you can construct the input arguments for the rewritten functions/modules，run model's inference in `RewriteContext` and check the results.

```python
# mmpretrain.models.classfiers.base.py
class BaseClassifier(BaseModule, metaclass=ABCMeta):
    def forward(self, img, return_loss=True, **kwargs):

```python
        if return_loss:
            return self.forward_train(img, **kwargs)
        else:
            return self.forward_test(img, **kwargs)

# Custom rewritten function
@FUNCTION_REWRITER.register_rewriter(
    'mmpretrain.models.classifiers.BaseClassifier.forward', backend='default')
def forward_of_base_classifier(self, img, *args, **kwargs):
    """Rewrite `forward` for default backend."""
    return self.simple_test(img, {})
```

In the example, we only change the function that `forward` calls. We can test this rewritten function by writing the following test function:

```python
def test_baseclassfier_forward():
    input = torch.rand(1)
    from mmpretrain.models.classifiers import BaseClassifier
    class DummyClassifier(BaseClassifier):

        def __init__(self, init_cfg=None):

input = torch.rand(1)

            super().__init__(init_cfg=init_cfg)

        def extract_feat(self, imgs):
            pass

        def forward_train(self, imgs):
            return 'train'

        def simple_test(self, img, tmp, **kwargs):
            return 'simple_test'

    model = DummyClassifier().eval()

    model_output = model(input)
    with RewriterContext(cfg=dict()), torch.no_grad():
        backend_output = model(input)

    assert model_output == 'train'
    assert backend_output == 'simple_test'
```

with RewriterContext(cfg=dict()), torch.no_grad():

```

In this test function, we construct a derived class of `BaseClassifier` to test if the rewritten model would work in the rewrite context. We get outputs of the original model by directly calling `model(input)` and get the outputs of the rewritten model by calling `model(input)` in `RewriteContext`. Finally, we can check the outputs by asserting their value.

## Test rewritten model with big changes

In the first example, the output is generated in Python. Sometimes we may make big changes to original model functions (e.g., eliminate branch statements to generate correct computing graph). Even if the outputs of a rewritten model running in Python are correct, we cannot assure that the rewritten model can work as expected in the backend. Therefore, we need to test the rewritten model in the backend.

```python

```python
    func_name='mmseg.models.segmentors.BaseSegmentor.forward')
def base_segmentor__forward(self, img, img_metas=None, **kwargs):
    ctx = FUNCTION_REWRITER.get_context()
    if img_metas is None:
        img_metas = {}
    assert isinstance(img_metas, dict)
    assert isinstance(img, torch.Tensor)

    deploy_cfg = ctx.cfg
    is_dynamic_flag = is_dynamic_shape(deploy_cfg)
    img_shape = img.shape[2:]
    if not is_dynamic_flag:
        img_shape = [int(val) for val in img_shape]
    img_metas['img_shape'] = img_shape
    return self.simple_test(img, img_metas, **kwargs)

```

The behavior of this rewritten function is complex. We should test it as follows:

```python
def test_basesegmentor_forward():
    from mmdeploy.utils.test import (WrapModel, get_model_outputs,
                                    get_rewrite_outputs)

    segmentor = get_model()
    segmentor.cpu().eval()

    # Prepare data
    # ...

get_rewrite_outputs)

    # ...

    # Get the outputs of original model
    model_inputs = {
        'img': [imgs],
        'img_metas': [img_metas],
        'return_loss': False
    }
    model_outputs = get_model_outputs(segmentor, 'forward', model_inputs)

    # Get the outputs of rewritten model
    wrapped_model = WrapModel(segmentor, 'forward', img_metas = None, return_loss = False)
    rewrite_inputs = {'img': imgs}
    rewrite_outputs, is_backend_output = get_rewrite_outputs(
        wrapped_model=wrapped_model,
        model_inputs=rewrite_inputs,
        deploy_cfg=deploy_cfg)
    if is_backend_output:
        # If the backend plugins have been installed, the rewrite outputs are
        # generated by backend.
        rewrite_outputs = torch.tensor(rewrite_outputs)
        model_outputs = torch.tensor(model_outputs)

# generated by backend.
        model_outputs = model_outputs.unsqueeze(0).unsqueeze(0)
        assert torch.allclose(rewrite_outputs, model_outputs)
    else:
        # Otherwise, the outputs are generated by python.
        assert rewrite_outputs is not None
```

assert torch.allclose(rewrite_outputs, model_outputs)
    else:
```

We provide some utilities to test rewritten functions. At first, you can construct a model and call `get_model_outputs` to get outputs of the original model. Then you can wrap the rewritten function with `WrapModel`, which serves as a partial function, and get the results with `get_rewrite_outputs`. `get_rewrite_outputs` returns two values that indicate the content of outputs and whether the outputs come from the backend. Because we cannot assume that everyone has installed the backend, we should check if the results are generated by a Python or backend engine. The unit test must cover both conditions. Finally, we should compare the original and rewritten outputs, which may be done simply by calling `torch.allclose`.

## Note

To learn the complete usage of the test utilities, please refer to our apis document.

## Note

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_ECCV_2018/html/Bin_Xiao_Simple_Baselines_for_ECCV_2018_paper.html">SimpleBaseline2D (ECCV'2018)</a></summary>

```bibtex
@inproceedings{xiao2018simple,
  title={Simple baselines for human pose estimation and tracking},
  author={Xiao, Bin and Wu, Haiping and Wei, Yichen},
  booktitle={Proceedings of the European conference on computer vision (ECCV)},
  pages={466--481},
}
```

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_cvpr_2016/html/He_Deep_Residual_Learning_CVPR_2016_paper.html">ResNet (CVPR'2016)</a></summary>

```bibtex
@inproceedings{he2016deep,
  title={Deep residual learning for image recognition},
  author={He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian},
  pages={770--778},
  year={2016}
}

}
```

<details>
<summary align="right"><a href="https://arxiv.org/abs/1711.06475">AI Challenger (ArXiv'2017)</a></summary>

```bibtex
@article{wu2017ai,
  title={Ai challenger: A large-scale dataset for going deeper in image understanding},
  author={Wu, Jiahong and Zheng, He and Zhao, Bo and Li, Yixin and Yan, Baoming and Liang, Rui and Wang, Wenjia and Zhou, Shipei and Lin, Guosen and Fu, Yanwei and others},
  journal={arXiv preprint arXiv:1711.06475},
}
```

Results on AIC val set with ground-truth bounding boxes

| [pose_resnet_101](/configs/body_2d_keypoint/topdown_heatmap/aic/td-hm_res101_8xb64-210e_aic-256x192.py) |  256x192   | 0.294 |      0.736      |      0.172      | 0.337 |      0.762      | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnet/res101_aic_256x192-79b35445_20200826.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnet/res101_aic_256x192_20200826.log.json) |

Q1: Why set up `projects/` folder?

Implementing new models and features into OpenMMLab's algorithm libraries could be troublesome due to the rigorous requirements on code quality, which could hinder the fast iteration of SOTA models and might discourage our members from sharing their latest outcomes here. And that's why we have this `projects/` folder now, where some experimental features, frameworks and models are placed, only needed to satisfy the minimum requirement on the code quality, and can be used as standalone libraries. Users are welcome to use them if they [use MMSegmentation from source](https://mmsegmentation.readthedocs.io/en/latest/get_started.html#best-practices).

Q2: Why should there be a checklist for a project?

This checkelist is crucial not only for this project's developers but the entire community, since there might be some other contributors joining this project and deciding their starting point from this list. It also helps maintainers accurately estimate time and effort on further code polishing, if needed.

Q3: What kind of PR will be merged?

Reaching the first milestone means that this project suffices the minimum requirement of being merged into 'projects/'. That is, the very first PR of a project must have all the terms in the first milestone checked. We do not have any extra requirements on the project's following PRs, so they can be a minor bug fix or update, and do not have to achieve one milestone at once. But keep in mind that this project is only eligible to become a part of the core package upon attaining the last milestone.

Q4: Compared to other models in the core packages, why do the model implementations in projects have different training/testing commands?

Projects are organized independently from the core package, and therefore their modules cannot be directly imported by train.py and test.py. Each model implementation in projects should either use `mim` for training/testing as suggested in the example project or provide a custom train.py/test.py.

Q5: How to debug a project with a debugger?

Debugger makes our lives easier, but using it becomes a bit tricky if we have to train/test a model via `mim`. The way to circumvent that is that we can take advantage of relative path to import these modules. Assuming that we are developing a project X and the core modules are placed under `projects/X/modules`, then simply adding `custom_imports = dict(imports='projects.X.modules')` to the config allows us to debug from usual entrypoints (e.g. `tools/train.py`) from the root directory of the algorithm library. Just don't forget to remove 'projects.X' before project publishment.

# TridentNet

> [Scale-Aware Trident Networks for Object Detection](https://arxiv.org/abs/1901.01892)

Scale variation is one of the key challenges in object detection. In this work, we first present a controlled experiment to investigate the effect of receptive fields for scale variation in object detection. Based on the findings from the exploration experiments, we propose a novel Trident Network (TridentNet) aiming to generate scale-specific feature maps with a uniform representational power. We construct a parallel multi-branch architecture in which each branch shares the same transformation parameters but with different receptive fields. Then, we adopt a scale-aware training scheme to specialize each branch by sampling object instances of proper scales for training. As a bonus, a fast approximation version of TridentNet could achieve significant improvements without any additional parameters and computational cost compared with the vanilla detector. On the COCO dataset, our TridentNet with ResNet-101 backbone achieves state-of-the-art single-model results of 48.4 mAP.

<img src="https://user-images.githubusercontent.com/40661020/143999668-0927922e-efc2-45fa-8bfc-1e3df18720f5.png"/>
</div>

We reports the test results using only one branch for inference.

| Backbone | Style | mstrain | Lr schd | Mem (GB) | Inf time (fps) | box AP |                                                                                                                                                                      Download                                                                                                                                                                      |

| :------: | :---: | :-----: | :-----: | :------: | :------------: | :----: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|   R-50   | caffe |    N    |   1x    |          |                |  37.7  |                 [model](https://download.openmmlab.com/mmdetection/v2.0/tridentnet/tridentnet_r50_caffe_1x_coco/tridentnet_r50_caffe_1x_coco_20201230_141838-2ec0b530.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/tridentnet/tridentnet_r50_caffe_1x_coco/tridentnet_r50_caffe_1x_coco_20201230_141838.log.json)                 |

|   R-50   | caffe |    Y    |   1x    |          |                |  37.6  | [model](https://download.openmmlab.com/mmdetection/v2.0/tridentnet/tridentnet_r50_caffe_mstrain_1x_coco/tridentnet_r50_caffe_mstrain_1x_coco_20201230_141839-6ce55ccb.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/tridentnet/tridentnet_r50_caffe_mstrain_1x_coco/tridentnet_r50_caffe_mstrain_1x_coco_20201230_141839.log.json) |
|   R-50   | caffe |    Y    |   3x    |          |                |  40.3  | [model](https://download.openmmlab.com/mmdetection/v2.0/tridentnet/tridentnet_r50_caffe_mstrain_3x_coco/tridentnet_r50_caffe_mstrain_3x_coco_20201130_100539-46d227ba.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/tridentnet/tridentnet_r50_caffe_mstrain_3x_coco/tridentnet_r50_caffe_mstrain_3x_coco_20201130_100539.log.json) |

**Note**

**Note**

Similar to [Detectron2](https://github.com/facebookresearch/detectron2/tree/master/projects/TridentNet), we haven't implemented the Scale-aware Training Scheme in section 4.2 of the paper.

```latex
@InProceedings{li2019scale,
  title={Scale-Aware Trident Networks for Object Detection},
  author={Li, Yanghao and Chen, Yuntao and Wang, Naiyan and Zhang, Zhaoxiang},
  journal={The International Conference on Computer Vision (ICCV)},
}
```

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_cvpr_2017/html/Xie_Aggregated_Residual_Transformations_CVPR_2017_paper.html">ResNext (CVPR'2017)</a></summary>

```bibtex
@inproceedings{xie2017aggregated,
  title={Aggregated residual transformations for deep neural networks},
  author={Xie, Saining and Girshick, Ross and Doll{\'a}r, Piotr and Tu, Zhuowen and He, Kaiming},
  pages={1492--1500},
}
```

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_cvpr_2014/html/Andriluka_2D_Human_Pose_2014_CVPR_paper.html">MPII (CVPR'2014)</a></summary>

```bibtex
@inproceedings{andriluka14cvpr,
  author = {Mykhaylo Andriluka and Leonid Pishchulin and Peter Gehler and Schiele, Bernt},
  title = {2D Human Pose Estimation: New Benchmark and State of the Art Analysis},

author = {Mykhaylo Andriluka and Leonid Pishchulin and Peter Gehler and Schiele, Bernt},
  booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2014},
  month = {June}
}
```

Results on MPII val set

| Arch                                                        | Input Size | Mean  | Mean@0.1 |                            ckpt                             |                             log                             |
| :---------------------------------------------------------- | :--------: | :---: | :------: | :---------------------------------------------------------: | :---------------------------------------------------------: |

| [pose_resnext_152](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_resnext152_8xb32-210e_mpii-256x256.py) |  256x256   | 0.887 |  0.294   | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnext/resnext152_mpii_256x256-df302719_20200927.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnext/resnext152_mpii_256x256_20200927.log.json) |

# 如何拆分 onnx 模型

MMDeploy 支持将PyTorch模型导出到onnx模型并进行拆分得到多个onnx模型文件，用户可以自由的对模型图节点进行标记并根据这些标记的节点定制任意的onnx模型拆分策略。在这个教程中，我们将通过具体例子来展示如何进行onnx模型拆分。在这个例子中，我们的目标是将YOLOV3模型拆分成两个部分，保留不带后处理的onnx模型，丢弃包含Anchor生成，NMS的后处理部分。

## 步骤 1: 添加模型标记点

为了进行图拆分，我们定义了`Mark`类型op，标记模型导出的边界。在实现方法上，采用`mark`装饰器对函数的输入、输出`Tensor`打标记。需要注意的是，我们的标记函数需要在某个重写函数中执行才能生效。

为了对YOLOV3进行拆分，首先我们需要标记模型的输入。这里为了通用性，我们标记检测器父类`BaseDetector`的`forward`方法中的`img` `Tensor`，同时为了支持其他拆分方案，也对`forward`函数的输出进行了标记，分别是`dets`, `labels`和`masks`。下面的代码是截图[mmdeploy/codebase/mmdet/models/detectors/single_stage.py](https://github.com/open-mmlab/mmdeploy/blob/4fc8828af84281b62be143012cd9f9dafd1e7cc2/mmdeploy/codebase/mmdet/models/detectors/single_stage.py)中的一部分，可以看出我们使用`mark`装饰器标记了`__forward_impl`函数的输入输出，并在重写函数`base_detector__forward`进行了调用，从而完成了对检测器输入的标记。

```python
from mmdeploy.core import FUNCTION_REWRITER, mark

@mark(
    'detector_forward', inputs=['input'], outputs=['dets', 'labels', 'masks'])
def __forward_impl(self, img, img_metas=None, **kwargs):

@mark(
    ...

    'mmdet.models.detectors.base.BaseDetector.forward')
def base_detector__forward(self, img, img_metas=None, **kwargs):
    ...
    # call the mark function
    return __forward_impl(...)
```

接下来，我们只需要对`YOLOV3Head`中最后一层输出特征`Tensor`进行标记就可以将整个`YOLOV3`模型拆分成两部分。通过查看`mmdet`源码我们可以知道`YOLOV3Head`的`get_bboxes`方法中输入参数`pred_maps`就是我们想要的拆分点，因此可以在重写函数[`yolov3_head__get_bboxes`](https://github.com/open-mmlab/mmdeploy/blob/4fc8828af84281b62be143012cd9f9dafd1e7cc2/mmdeploy/codebase/mmdet/models/dense_heads/yolo_head.py#L16)中添加内部函数对`pred_mapes`进行标记，具体参考如下示例代码。值得注意的是，输入参数`pred_maps`是由三个`Tensor`组成的列表，所以我们在onnx模型中添加了三个`Mark`标记节点。

```python

```python

    func_name='mmdet.models.dense_heads.YOLOV3Head.get_bboxes')
def yolov3_head__get_bboxes(self,
                            pred_maps,
                            img_metas,
                            cfg=None,
                            rescale=False,
                            with_nms=True):
    # mark pred_maps
    @mark('yolo_head', inputs=['pred_maps'])
    def __mark_pred_maps(pred_maps):
        return pred_maps
    pred_maps = __mark_pred_maps(pred_maps)
    ...
```

## 步骤 2: 添加部署配置文件

在完成模型中节点标记之后，我们需要创建部署配置文件，我们假设部署后端是`onnxruntime`,并模型输入是固定尺寸`608x608`,因此添加文件`configs/mmdet/detection/yolov3_partition_onnxruntime_static.py`. 我们需要在配置文件中添加基本的配置信息如`onnx_config`，如何你还不熟悉如何添加配置文件，可以参考[write_config.md](../02-how-to-run/write_config.md).

在这个部署配置文件中, 我们需要添加一个特殊的模型分段配置字段`partition_config`. 在模型分段配置中，我们可以可以给分段策略添加一个类型名称如`yolov3_partition`，设定`apply_marks=True`。在分段方式`partition_cfg`,我们需要指定每段模型的分割起始点`start`, 终止点`end`以及保存分段onnx的文件名。需要提醒的是，各段模型起始点`start`和终止点`end`是由多个标记节点`Mark`组成，例如`'detector_forward:input'`代表`detector_forward`标记处输入所产生的标记节点。配置文件具体内容参考如下代码：

```python
_base_ = ['./detection_onnxruntime_static.py']

onnx_config = dict(input_shape=[608, 608])
partition_config = dict(
    type='yolov3_partition', # the partition policy name
    apply_marks=True, # should always be set to True
    partition_cfg=[
        dict(
            save_file='yolov3.onnx', # filename to save the partitioned onnx model
            start=['detector_forward:input'], # [mark_name:input/output, ...]
            end=['yolo_head:input'],  # [mark_name:input/output, ...]
            output_names=[f'pred_maps.{i}' for i in range(3)]) # output names
    ])

```

## 步骤 3: 拆分onnx模型

end=['yolo_head:input'],  # [mark_name:input/output, ...]
    ])

```

添加好节点标记和部署配置文件，我们可以使用`tools/torch2onnx.py`工具导出带有`Mark`标记的完成onnx模型并根据分段策略提取分段的onnx模型文件。我们可以执行如下脚本，得到不带后处理的`YOLOV3`onnx模型文件`yolov3.onnx`，同时输出文件中也包含了添加`Mark`标记的完整模型文件`end2end.onnx`。此外，用户可以使用网页版模型可视化工具[netron](https://netron.app/)来查看和验证输出onnx模型的结构是否正确。

```shell
python tools/torch2onnx.py \
configs/mmdet/detection/yolov3_partition_onnxruntime_static.py \
../mmdetection/configs/yolo/yolov3_d53_8xb8-ms-608-273e_coco.py \
https://download.openmmlab.com/mmdetection/v2.0/yolo/yolov3_d53_mstrain-608_273e_coco/yolov3_d53_mstrain-608_273e_coco_20210518_115020-a2c3acb8.pth \
../mmdetection/demo/demo.jpg \
--work-dir ./work-dirs/mmdet/yolov3/ort/partition
```

当得到分段onnx模型之后，我们可以使用mmdeploy提供的其他工具如`mmdeploy_onnx2ncnn`, `onnx2tensorrt`来进行后续的模型部署工作。

## 3D Human Pose Demo

<img  src="https://user-images.githubusercontent.com/15977946/118820606-02df2000-b8e9-11eb-9984-b9228101e780.gif"  width="600px"  alt><br>

### 3D Human Pose Two-stage Estimation Demo

#### Using mmdet for human bounding box detection and top-down model for the 1st stage (2D pose detection), and inference the 2nd stage (2D-to-3D lifting)

Assume that you have already installed [mmdet](https://github.com/open-mmlab/mmdetection).

```shell
python  demo/body3d_pose_lifter_demo.py  \
${MMDET_CONFIG_FILE} \
${MMDET_CHECKPOINT_FILE} \
${MMPOSE_CONFIG_FILE_2D} \
${MMPOSE_CHECKPOINT_FILE_2D} \
${MMPOSE_CONFIG_FILE_3D} \
${MMPOSE_CHECKPOINT_FILE_3D} \
--input ${VIDEO_PATH or IMAGE_PATH or 'webcam'} \
[--show] \
[--rebase-keypoint-height] \
[--norm-pose-2d] \
[--num-instances] \
[--output-root ${OUT_VIDEO_ROOT}] \
[--save-predictions]
[--save-predictions] \
[--device ${GPU_ID  or  CPU}] \
[--det-cat-id DET_CAT_ID] \
[--bbox-thr BBOX_THR] \
[--kpt-thr KPT_THR] \

[--use-oks-tracking] \
[--tracking-thr TRACKING_THR] \
[--show-interval INTERVAL] \
[--thickness THICKNESS] \
[--radius RADIUS] \
[--use-multi-frames] [--online]
```

Note that

1. `${VIDEO_PATH}` can be the local path or **URL** link to video file.

2. You can turn on the `[--use-multi-frames]` option to use multi frames for inference in the 2D pose detection stage.

3. If the `[--online]` option is set to **True**, future frame information can **not** be used when using multi frames for inference in the 2D pose detection stage.

Examples:

During 2D pose detection, for single-frame inference that do not rely on extra frames to get the final results of the current frame and save the prediction results, try this:

```shell

```shell
demo/mmdetection_cfg/faster_rcnn_r50_fpn_coco.py \
https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth  \
configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_8xb32-210e_coco-256x192.py \
https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_coco_256x192-b9e0b3ab_20200708.pth  \
configs/body_3d_keypoint/pose_lift/h36m/pose-lift_videopose3d-243frm-supv-cpn-ft_8xb128-200e_h36m.py \
https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_243frames_fullconv_supervised_cpn_ft-88f5abbb_20210527.pth  \
--input https://user-images.githubusercontent.com/87690686/164970135-b14e424c-765a-4180-9bc8-fa8d6abc5510.mp4 \
--output-root  vis_results  \
--rebase-keypoint-height  --save-predictions
```

During 2D pose detection, for multi-frame inference that rely on extra frames to get the final results of the current frame, try this:

```

```shell
configs/body_2d_keypoint/topdown_heatmap/posetrack18/td-hm_hrnet-w48_8xb64-20e_posetrack18-384x288.py \
https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_posetrack18_384x288-5fd6d3ff_20211130.pth  \

--rebase-keypoint-height \
--use-multi-frames  --online
```

### 3D Human Pose Demo with Inferencer

The Inferencer provides a convenient interface for inference, allowing customization using model aliases instead of configuration files and checkpoint paths. It supports various input formats, including image paths, video paths, image folder paths, and webcams. Below is an example command:

```shell
python demo/inferencer_demo.py tests/data/coco/000000000785.jpg \
    --pose3d human3d --vis-out-dir vis_results/human3d \
    --rebase-keypoint-height
```

This command infers the image and saves the visualization results in the `vis_results/human3d` directory.

<img src="https://github.com/open-mmlab/mmpose/assets/26127467/9621f51f-59e4-41e5-ab4c-3b03e97f0e9d" alt="Image 1" height="300"/>

In addition, the Inferencer supports saving predicted poses. For more information, please refer to the [inferencer document](https://mmpose.readthedocs.io/en/latest/user_guides/inference.html#inferencer-a-unified-inference-interface).

# NAS-FCOS

> [NAS-FCOS: Fast Neural Architecture Search for Object Detection](https://arxiv.org/abs/1906.04423)

The success of deep neural networks relies on significant architecture engineering. Recently neural architecture search (NAS) has emerged as a promise to greatly reduce manual effort in network design by automatically searching for optimal architectures, although typically such algorithms need an excessive amount of computational resources, e.g., a few thousand GPU-days. To date, on challenging vision tasks such as object detection, NAS, especially fast versions of NAS, is less studied. Here we propose to search for the decoder structure of object detectors with search efficiency being taken into consideration. To be more specific, we aim to efficiently search for the feature pyramid network (FPN) as well as the prediction head of a simple anchor-free object detector, namely FCOS, using a tailored reinforcement learning paradigm. With carefully designed search space, search algorithms and strategies for evaluating network quality, we are able to efficiently search a top-performing detection architecture within 4 days using 8 V100 GPUs. The discovered architecture surpasses state-of-the-art object detection models (such as Faster R-CNN, RetinaNet and FCOS) by 1.5 to 3.5 points in AP on the COCO dataset, with comparable computation complexity and memory footprint, demonstrating the efficacy of the proposed NAS for object detection.

<img src="https://user-images.githubusercontent.com/40661020/143967900-1c8a65b9-c58d-4b03-8900-96af8f9768e8.png"/>
</div>

|     Head     | Backbone | Style | GN-head | Lr schd | Mem (GB) | Inf time (fps) | box AP |                               Config                                |                                                                                                                                                                                           Download                                                                                                                                                                                           |

| NAS-FCOSHead |   R-50   | caffe |    Y    |   1x    |          |                |  39.4  | [config](./nas-fcos_r50-caffe_fpn_nashead-gn-head_4xb4-1x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/nas_fcos/nas_fcos_nashead_r50_caffe_fpn_gn-head_4x4_1x_coco/nas_fcos_nashead_r50_caffe_fpn_gn-head_4x4_1x_coco_20200520-1bdba3ce.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/nas_fcos/nas_fcos_nashead_r50_caffe_fpn_gn-head_4x4_1x_coco/nas_fcos_nashead_r50_caffe_fpn_gn-head_4x4_1x_coco_20200520.log.json)   |

|   FCOSHead   |   R-50   | caffe |    Y    |   1x    |          |                |  38.5  | [config](./nas-fcos_r50-caffe_fpn_fcoshead-gn-head_4xb4-1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/nas_fcos/nas_fcos_fcoshead_r50_caffe_fpn_gn-head_4x4_1x_coco/nas_fcos_fcoshead_r50_caffe_fpn_gn-head_4x4_1x_coco_20200521-7fdcbce0.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/nas_fcos/nas_fcos_fcoshead_r50_caffe_fpn_gn-head_4x4_1x_coco/nas_fcos_fcoshead_r50_caffe_fpn_gn-head_4x4_1x_coco_20200521.log.json) |

- To be consistent with the author's implementation, we use 4 GPUs with 4 images/GPU.

```latex
@article{wang2019fcos,
  title={Nas-fcos: Fast neural architecture search for object detection},
  author={Wang, Ning and Gao, Yang and Chen, Hao and Wang, Peng and Tian, Zhi and Shen, Chunhua},
  journal={arXiv preprint arXiv:1906.04423},
}
```

# Human Body 3D Pose Estimation

3D human body pose estimation aims at predicting the X, Y, Z coordinates of human body joints. Based on the camera number to capture the images or videos, existing works can be further divided into multi-view methods and single-view (monocular) methods.

## Data preparation

Please follow [DATA Preparation](/docs/en/dataset_zoo/3d_body_keypoint.md) to prepare data.

## Demo

Please follow [Demo](/demo/docs/en/3d_human_pose_demo.md) to run demos.

<img src="https://user-images.githubusercontent.com/15977946/118820606-02df2000-b8e9-11eb-9984-b9228101e780.gif" width="600px" alt><br>

# ONNX export Optimizer

This is a tool to optimize ONNX model when exporting from PyTorch.

Build MMDeploy with `torchscript` support:

```shell
export Torch_DIR=$(python -c "import torch;print(torch.utils.cmake_prefix_path + '/Torch')")

cmake \
    -DTorch_DIR=${Torch_DIR} \
    -DMMDEPLOY_TARGET_BACKENDS="${your_backend};torchscript" \
    .. # You can also add other build flags if you need

cmake --build . -- -j$(nproc) && cmake --install .
```

## Usage

```python
# import model_to_graph_custom_optimizer so we can hijack onnx.export
from mmdeploy.apis.onnx.optimizer import model_to_graph__custom_optimizer # noqa
from mmdeploy.core import RewriterContext
from mmdeploy.apis.onnx.passes import optimize_onnx

# load you model here
model = create_model()

# export with ONNX Optimizer
x = create_dummy_input()
with RewriterContext({}, onnx_custom_passes=optimize_onnx):
    torch.onnx.export(model, x, output_path)
```

The model would be optimized after export.

```

You can also define your own optimizer:

```python
# create the optimize callback
def _optimize_onnx(graph, params_dict, torch_out):
    from mmdeploy.backend.torchscript import ts_optimizer
    ts_optimizer.onnx._jit_pass_onnx_peephole(graph)
    return graph, params_dict, torch_out

with RewriterContext({}, onnx_custom_passes=_optimize_onnx):
    # export your model
```

In this section, you will know about:

- [Installation](#installation)
  - [Installation](#installation-1)
    - [Prerequisites](#prerequisites)
    - [Best practices](#best-practices)
    - [Customize installation](#customize-installation)
      - [CUDA Version](#cuda-version)
      - [Install MMCV without MIM](#install-mmcv-without-mim)
      - [Using MMagic with Docker](#using-mmagic-with-docker)
      - [Trouble shooting](#trouble-shooting)
    - [Developing with multiple MMagic versions](#developing-with-multiple-mmagic-versions)

We recommend that users follow our [Best practices](#best-practices) to install MMagic.
However, the whole process is highly customizable. See [Customize installation](#customize-installation) section for more information.

In this section, we demonstrate how to prepare an environment with PyTorch.

MMagic works on Linux, Windows, and macOS. It requires:

- Python >= 3.7

- [PyTorch](https://pytorch.org/) >= 1.8
- [MMCV](https://github.com/open-mmlab/mmcv) >= 2.0.0

>

If you are experienced with PyTorch and have already installed it,
just skip this part and jump to the [next section](#best-practices). Otherwise, you can follow these steps for the preparation.

**Step 0.**
Download and install Miniconda from [official website](https://docs.conda.io/en/latest/miniconda.html).

Create a [conda environment](https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/environments.html#) and activate it

```shell
conda create --name mmagic python=3.8 -y
conda activate mmagic
```

Install PyTorch following [official instructions](https://pytorch.org/get-started/locally/), e.g.

- On GPU platforms:

  ```shell

```

  ```shell
  conda install pytorch torchvision cudatoolkit=11.3 -c pytorch
  ```

- On CPU platforms:

  ```shell
  conda install pytorch=1.10 torchvision cpuonly -c pytorch
  ```

### Best practices

**Step 0.** Install [MMCV](https://github.com/open-mmlab/mmcv) using [MIM](https://github.com/open-mmlab/mim).

```shell
```

**Step 1.** Install [MMEngine](https://github.com/open-mmlab/mmengine).

```shell
```

Or

```shell
pip install mmengine
```

Or

```shell
pip install git+https://github.com/open-mmlab/mmengine.git
```

**Step 2.** Install MMagic.

```shell
```

Or

```shell
pip install mmagic
```

Or install [MMagic](https://github.com/open-mmlab/mmagic) from the source code.

```shell

```

```shell
```

Or

```shell
```

```shell
cd mmagic
pip3 install -e . -v
```

**Step 5.**

```shell
cd ~
```

The installation is successful if the version number is output correctly.

```{note}
You may be curious about what `-e .` means when supplied with `pip install`.
Here is the description:

- `-e` means [editable mode](https://pip.pypa.io/en/latest/cli/pip_install/#cmdoption-e).
  When `import mmagic`, modules under the cloned directory are imported.
  If `pip install` without `-e`, pip will copy cloned codes to somewhere like `lib/python/site-package`.
  Consequently, modified code under the cloned directory takes no effect unless `pip install` again.

Consequently, modified code under the cloned directory takes no effect unless `pip install` again.
  Thus, `pip install` with `-e` is particularly convenient for developers. If some codes are modified, new codes will be imported next time without reinstallation.
- `.` means code in this directory

You can also use `pip install -e .[all]`, which will install more dependencies, especially for pre-commit hooks and unittests.
```

### Customize installation

#### CUDA Version

When installing PyTorch, you need to specify the version of CUDA. If you are not clear on which to choose, follow our recommendations:

Please make sure the GPU driver satisfies the minimum version requirements.

See [this table](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cuda-major-component-versions__table-cuda-toolkit-driver-versions) for more information.

**note**
because no CUDA code will be compiled locally.
However, if you hope to compile MMCV from source or develop other CUDA operators,
you need to install the complete CUDA toolkit from NVIDIA's [website](https://developer.nvidia.com/cuda-downloads),
and its version should match the CUDA version of PyTorch. i.e., the specified version of cudatoolkit in `conda install` command.

#### Install MMCV without MIM

MMCV contains C++ and CUDA extensions, thus depending on PyTorch in a complex way.

MIM solves such dependencies automatically and makes the installation easier. However, it is not a must.

To install MMCV with pip instead of MIM, please follow [MMCV installation guides](https://mmcv.readthedocs.io/en/latest/get_started/installation.html).
This requires manually specifying a find-url based on PyTorch version and its CUDA version.

For example, the following command install mmcv-full built for PyTorch 1.10.x and CUDA 11.3.

```shell
pip install 'mmcv>=2.0.0' -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10/index.html
```

#### Using MMagic with Docker

We provide a [Dockerfile](https://github.com/open-mmlab/mmagic/blob/main/docker/Dockerfile) to build an image.
Ensure that your [docker version](https://docs.docker.com/engine/install/) >=19.03.

```shell
# build an image with PyTorch 1.8, CUDA 11.1

```shell
docker build -t mmagic docker/
```

Run it with

```shell
docker run --gpus all --shm-size=8g -it -v {DATA_DIR}:/mmagic/data mmagic
```

#### Trouble shooting

If you have some issues during the installation, please first view the [FAQ](../faq.md) page.
You may [open an issue](https://github.com/open-mmlab/mmagic/issues/new/choose) on GitHub if no solution is found.

### Developing with multiple MMagic versions

The train and test scripts already modify the `PYTHONPATH` to ensure the script uses the `MMagic` in the current directory.

To use the default MMagic installed in the environment rather than that you are working with, you can remove the following line in those scripts

```shell
PYTHONPATH="$(dirname $0)/..":$PYTHONPATH
```

# EfficientDet

> [**EfficientDet: Scalable and Efficient Object Detection**](https://arxiv.org/pdf/1911.09070.pdf),
> Mingxing Tan, Ruoming Pang, Quoc V. Le,
> *CVPR 2020*

This is an implementation of [EfficientDet](https://github.com/google/automl) based on [MMDetection](https://github.com/open-mmlab/mmdetection/tree/main), [MMCV](https://github.com/open-mmlab/mmcv), and [MMEngine](https://github.com/open-mmlab/mmengine).
<br>
EfficientDet a new family of object detectors, which consistently achieve much better efficiency than prior art across a wide
spectrum of resource constraints.
In particular, with single model and single-scale, EfficientDet-D7 achieves stateof-the-art 55.1 AP on COCO test-dev with 77M parameters and 410B FLOP.
<br>

<br>
BiFPN is a simple yet highly effective weighted bi-directional feature pyramid network, which introduces learnable weights to learn the importance of different input features, while repeatedly applying topdown and bottom-up multi-scale feature fusion.
<br>
In contrast to other feature pyramid network, such as FPN, FPN + PAN, NAS-FPN, BiFPN achieves  the best accuracy with fewer parameters and FLOPs.

<img src="https://github.com/zwhus/pictures/raw/main/Screenshot%20from%202023-01-31%2010-38-51.png">
</div>

## Usage

## Official TensorFlow Model

</div>

## Usage

This project also supports [official tensorflow model](https://github.com/google/automl), it uses 90 categories and yxyx box encoding in training. If you want to use the original model weight to get official results, please refer to the following steps.

### Model conversion

Firstly, download EfficientDet [weights](https://github.com/google/automl/tree/master/efficientdet) and unzip,  please use the following command

```bash
tar -xzvf {EFFICIENTDET_WEIGHT}
```

Then, install tensorflow, please use the following command

```bash
pip install tensorflow-gpu==2.6.0
```

Lastly, convert weights from tensorflow to pytorch, please use the following command

```bash
python projects/EfficientDet/convert_tf_to_pt.py --backbone {BACKBONE_NAME} --tensorflow_weight {TENSORFLOW_WEIGHT_PATH} --out_weight {OUT_PATH}
```

```bash
```

In MMDetection's root directory, run the following command to test the model:

```bash
python tools/test.py projects/EfficientDet/configs/tensorflow/efficientdet_effb0_bifpn_8xb16-crop512-300e_coco_tf.py ${CHECKPOINT_PATH}
```

## Reproduce Model

For convenience, we recommend the current implementation version, it uses 80 categories and xyxy encoding in training. On this basis, a higher result was finally achieved.

In MMDetection's root directory, run the following command to train the model:

```bash
python tools/train.py projects/EfficientDet/configs/efficientdet_effb3_bifpn_8xb16-crop896-300e_coco.py
```

```bash

```

```bash
python tools/test.py projects/EfficientDet/configs/efficientdet_effb3_bifpn_8xb16-crop896-300e_coco.py ${CHECKPOINT_PATH}
```

## Results

Based on mmdetection, this project aligns the accuracy of the [official model](https://github.com/google/automl).

|                                                        Method                                                        |    Backbone     | Pretrained Model |  Training set  |   Test set   | Epoch | Val Box AP | Official AP |                                                                                                                                                                                                Download                                                                                                                                                                                                |

| [efficientdet-d0\*](projects/EfficientDet/configs/tensorflow/efficientdet_effb0_bifpn_8xb16-crop512-300e_coco_tf.py) | efficientnet-b0 |     ImageNet     | COCO2017 Train | COCO2017 Val |  300  |    34.4    |    34.3     |                                                                                                                                                                                                                                                                                                                                                                                                        |

|         [efficientdet-d3](projects/EfficientDet/configs/efficientdet_effb3_bifpn_8xb16-crop896-300e_coco.py)         | efficientnet-b3 |     ImageNet     | COCO2017 Train | COCO2017 Val |  300  |    47.2    |    46.8     | [model](https://download.openmmlab.com/mmdetection/v3.0/efficientdet/efficientdet_effb3_bifpn_8xb16-crop896-300e_coco/efficientdet_effb3_bifpn_8xb16-crop896-300e_coco_20230223_122457-e6f7a833.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/efficientdet/efficientdet_effb3_bifpn_8xb16-crop896-300e_coco/efficientdet_effb3_bifpn_8xb16-crop896-300e_coco_20230223_122457.log.json) |

**Note**:
\*means use [official tensorflow model](https://github.com/google/automl) weights to test.

```BibTeX
@inproceedings{tan2020efficientdet,
  title={Efficientdet: Scalable and efficient object detection},
  author={Tan, Mingxing and Pang, Ruoming and Le, Quoc V},
  booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},

author={Tan, Mingxing and Pang, Ruoming and Le, Quoc V},
  pages={10781--10790},
}
```

<!-- Here is a checklist illustrating a usual development workflow of a successful project, and also serves as an overview of this project's progress. The PIC (person in charge) or contributors of this project should check all the items that they believe have been finished, which will further be verified by codebase maintainers via a PR.
OpenMMLab's maintainer will review the code to ensure the project's quality. Reaching the first milestone means that this project suffices the minimum requirement of being merged into 'projects/'. But this project is only eligible to become a part of the core package upon attaining the last milestone.

Note that keeping this section up-to-date is crucial not only for this project's developers but the entire community, since there might be some other contributors joining this project and deciding their starting point from this list. It also helps maintainers accurately estimate time and effort on further code polishing, if needed.
A project does not necessarily have to be finished in a single PR, but it's essential for the project to at least reach the first milestone in its very first PR. -->

    <!-- The code's design shall follow existing interfaces and convention. For example, each model component should be registered into `mmdet.registry.MODELS` and configurable via a config file. -->

- [x] Basic docstrings & proper citation

    <!-- Each major object should contain a docstring, describing its functionality and arguments. If you have adapted the code from other open-source projects, don't forget to cite the source project in docstring and make sure your behavior is not against its license. Typically, we do not accept any code snippet under GPL license. [A Short Guide to Open Source Licenses](https://medium.com/nationwide-technology/a-short-guide-to-open-source-licenses-cf5b1c329edd) -->

    <!-- If you are reproducing the result from a paper, make sure your model's inference-time performance matches that in the original paper. The weights usually could be obtained by simply renaming the keys in the official pre-trained weights. This test could be skipped though, if you are able to prove the training-time correctness and check the second milestone. -->

    <!-- As this template does. -->

- [x] A full README

    <!-- If you are reproducing the result from a paper, checking this item means that you should have trained your model from scratch based on the original paper's specification and verified that the final result matches the report within a minor error range. -->

    <!-- Ideally *all* the methods should have [type hints](https://www.pythontutorial.net/python-basics/python-type-hints/) and [docstrings](https://google.github.io/styleguide/pyguide.html#381-docstrings). [Example](https://github.com/open-mmlab/mmdetection/blob/5b0d5b40d5c6cfda906db7464ca22cbd4396728a/mmdet/datasets/transforms/transforms.py#L41-L169) -->

- [ ] Unit tests

    <!-- Unit tests for each module are required. [Example](https://github.com/open-mmlab/mmdetection/blob/5b0d5b40d5c6cfda906db7464ca22cbd4396728a/tests/test_datasets/test_transforms/test_transforms.py#L35-L88) -->

    <!-- Refactor your code according to reviewer's comment. -->

    <!-- It will be parsed by MIM and Inferencer. [Example](https://github.com/open-mmlab/mmdetection/blob/main/configs/faster_rcnn/metafile.yml) -->

  <!-- In particular, you may have to refactor this README into a standard one. [Example](https://github.com/open-mmlab/mmdetection/blob/main/configs/faster_rcnn/README.md) -->

# NonLocal Net

> [Non-local Neural Networks](https://arxiv.org/abs/1711.07971)

<a href="https://github.com/facebookresearch/video-nonlocal-net">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/decode_heads/nl_head.py#L10">Code Snippet</a>

Both convolutional and recurrent operations are building blocks that process one local neighborhood at a time. In this paper, we present non-local operations as a generic family of building blocks for capturing long-range dependencies. Inspired by the classical non-local means method in computer vision, our non-local operation computes the response at a position as a weighted sum of the features at all positions. This building block can be plugged into many computer vision architectures. On the task of video classification, even without any bells and whistles, our non-local models can compete or outperform current competition winners on both Kinetics and Charades datasets. In static image recognition, our non-local models improve object detection/segmentation and pose estimation on the COCO suite of tasks. Code is available at [this https URL](https://github.com/facebookresearch/video-nonlocal-net).

<img src="https://user-images.githubusercontent.com/24582831/142902128-17e29678-bf12-4ff4-b3d6-a39b47dfd253.png" width="50%"/>
</div>

| Method      | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                 | download                                                                                                                                                                                                                                                                                                                                                                     |

| NonLocalNet | R-50-D8  | 512x1024  |   40000 | 7.4      | 2.72           | V100   | 78.24 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/nonlocal_net/nonlocal_r50-d8_4xb2-40k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r50-d8_512x1024_40k_cityscapes/nonlocal_r50-d8_512x1024_40k_cityscapes_20200605_210748-c75e81e3.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r50-d8_512x1024_40k_cityscapes/nonlocal_r50-d8_512x1024_40k_cityscapes_20200605_210748.log.json)     |

| NonLocalNet | R-101-D8 | 512x1024  |   40000 | 10.9     | 1.95           | V100   | 78.66 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/nonlocal_net/nonlocal_r101-d8_4xb2-40k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r101-d8_512x1024_40k_cityscapes/nonlocal_r101-d8_512x1024_40k_cityscapes_20200605_210748-d63729fa.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r101-d8_512x1024_40k_cityscapes/nonlocal_r101-d8_512x1024_40k_cityscapes_20200605_210748.log.json) |

| NonLocalNet | R-50-D8  | 769x769   |   40000 | 8.9      | 1.52           | V100   | 78.33 | 79.92         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/nonlocal_net/nonlocal_r50-d8_4xb2-40k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r50-d8_769x769_40k_cityscapes/nonlocal_r50-d8_769x769_40k_cityscapes_20200530_045243-82ef6749.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r50-d8_769x769_40k_cityscapes/nonlocal_r50-d8_769x769_40k_cityscapes_20200530_045243.log.json)         |

| NonLocalNet | R-101-D8 | 769x769   |   40000 | 12.8     | 1.05           | V100   | 78.57 | 80.29         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/nonlocal_net/nonlocal_r101-d8_4xb2-40k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r101-d8_769x769_40k_cityscapes/nonlocal_r101-d8_769x769_40k_cityscapes_20200530_045348-8fe9a9dc.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r101-d8_769x769_40k_cityscapes/nonlocal_r101-d8_769x769_40k_cityscapes_20200530_045348.log.json)     |

| NonLocalNet | R-50-D8  | 512x1024  |   80000 | -        | -              | V100   | 78.01 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/nonlocal_net/nonlocal_r50-d8_4xb2-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r50-d8_512x1024_80k_cityscapes/nonlocal_r50-d8_512x1024_80k_cityscapes_20200607_193518-d6839fae.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r50-d8_512x1024_80k_cityscapes/nonlocal_r50-d8_512x1024_80k_cityscapes_20200607_193518.log.json)     |

| NonLocalNet | R-101-D8 | 512x1024  |   80000 | -        | -              | V100   | 78.93 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/nonlocal_net/nonlocal_r101-d8_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r101-d8_512x1024_80k_cityscapes/nonlocal_r101-d8_512x1024_80k_cityscapes_20200607_183411-32700183.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r101-d8_512x1024_80k_cityscapes/nonlocal_r101-d8_512x1024_80k_cityscapes_20200607_183411.log.json) |

| NonLocalNet | R-50-D8  | 769x769   |   80000 | -        | -              | V100   | 79.05 | 80.68         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/nonlocal_net/nonlocal_r50-d8_4xb2-80k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r50-d8_769x769_80k_cityscapes/nonlocal_r50-d8_769x769_80k_cityscapes_20200607_193506-1f9792f6.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r50-d8_769x769_80k_cityscapes/nonlocal_r50-d8_769x769_80k_cityscapes_20200607_193506.log.json)         |

| NonLocalNet | R-101-D8 | 769x769   |   80000 | -        | -              | V100   | 79.40 | 80.85         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/nonlocal_net/nonlocal_r101-d8_4xb2-80k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r101-d8_769x769_80k_cityscapes/nonlocal_r101-d8_769x769_80k_cityscapes_20200607_183428-0e1fa4f9.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r101-d8_769x769_80k_cityscapes/nonlocal_r101-d8_769x769_80k_cityscapes_20200607_183428.log.json)     |

### ADE20K

### ADE20K

| Method      | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                             | download                                                                                                                                                                                                                                                                                                                                                     |

| NonLocalNet | R-50-D8  | 512x512   |   80000 | 9.1      | 21.37          | V100   | 40.75 |         42.05 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/nonlocal_net/nonlocal_r50-d8_4xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r50-d8_512x512_80k_ade20k/nonlocal_r50-d8_512x512_80k_ade20k_20200615_015801-5ae0aa33.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r50-d8_512x512_80k_ade20k/nonlocal_r50-d8_512x512_80k_ade20k_20200615_015801.log.json)         |

| NonLocalNet | R-101-D8 | 512x512   |   80000 | 12.6     | 13.97          | V100   | 42.90 |         44.27 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/nonlocal_net/nonlocal_r101-d8_4xb4-80k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r101-d8_512x512_80k_ade20k/nonlocal_r101-d8_512x512_80k_ade20k_20200615_015758-24105919.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r101-d8_512x512_80k_ade20k/nonlocal_r101-d8_512x512_80k_ade20k_20200615_015758.log.json)     |

| NonLocalNet | R-50-D8  | 512x512   |  160000 | -        | -              | V100   | 42.03 |         43.04 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/nonlocal_net/nonlocal_r50-d8_4xb4-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r50-d8_512x512_160k_ade20k/nonlocal_r50-d8_512x512_160k_ade20k_20200616_005410-baef45e3.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r50-d8_512x512_160k_ade20k/nonlocal_r50-d8_512x512_160k_ade20k_20200616_005410.log.json)     |

| NonLocalNet | R-101-D8 | 512x512   |  160000 | -        | -              | V100   | 44.63 |         45.79 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/nonlocal_net/nonlocal_r101-d8_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r101-d8_512x512_160k_ade20k/nonlocal_r101-d8_512x512_160k_ade20k_20210827_221502-7881aa1a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r101-d8_512x512_160k_ade20k/nonlocal_r101-d8_512x512_160k_ade20k_20210827_221502.log.json) |

| Method      | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                              | download                                                                                                                                                                                                                                                                                                                                                         |

| NonLocalNet | R-50-D8  | 512x512   |   20000 | 6.4      | 21.21          | V100   | 76.20 |         77.12 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/nonlocal_net/nonlocal_r50-d8_4xb4-20k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r50-d8_512x512_20k_voc12aug/nonlocal_r50-d8_512x512_20k_voc12aug_20200617_222613-07f2a57c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r50-d8_512x512_20k_voc12aug/nonlocal_r50-d8_512x512_20k_voc12aug_20200617_222613.log.json)     |

| NonLocalNet | R-101-D8 | 512x512   |   20000 | 9.8      | 14.01          | V100   | 78.15 |         78.86 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/nonlocal_net/nonlocal_r101-d8_4xb4-20k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r101-d8_512x512_20k_voc12aug/nonlocal_r101-d8_512x512_20k_voc12aug_20200617_222615-948c68ab.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r101-d8_512x512_20k_voc12aug/nonlocal_r101-d8_512x512_20k_voc12aug_20200617_222615.log.json) |

| NonLocalNet | R-50-D8  | 512x512   |   40000 | -        | -              | V100   | 76.65 |         77.47 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/nonlocal_net/nonlocal_r50-d8_4xb4-40k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r50-d8_512x512_40k_voc12aug/nonlocal_r50-d8_512x512_40k_voc12aug_20200614_000028-0139d4a9.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r50-d8_512x512_40k_voc12aug/nonlocal_r50-d8_512x512_40k_voc12aug_20200614_000028.log.json)     |

| NonLocalNet | R-101-D8 | 512x512   |   40000 | -        | -              | V100   | 78.27 |         79.12 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/nonlocal_net/nonlocal_r101-d8_4xb4-40k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r101-d8_512x512_40k_voc12aug/nonlocal_r101-d8_512x512_40k_voc12aug_20200614_000028-7e5ff470.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/nonlocal_net/nonlocal_r101-d8_512x512_40k_voc12aug/nonlocal_r101-d8_512x512_40k_voc12aug_20200614_000028.log.json) |

```bibtex
@inproceedings{wang2018non,
  title={Non-local neural networks},
  author={Wang, Xiaolong and Girshick, Ross and Gupta, Abhinav and He, Kaiming},
  pages={7794--7803},
}
```

# CornerNet

> [Cornernet: Detecting objects as paired keypoints](https://arxiv.org/abs/1808.01244)

We propose CornerNet, a new approach to object detection where we detect an object bounding box as a pair of keypoints, the top-left corner and the bottom-right corner, using a single convolution neural network. By detecting objects as paired keypoints, we eliminate the need for designing a set of anchor boxes commonly used in prior single-stage detectors. In addition to our novel formulation, we introduce corner pooling, a new type of pooling layer that helps the network better localize corners. Experiments show that CornerNet achieves a 42.2% AP on MS COCO, outperforming all existing one-stage detectors.

<img src="https://user-images.githubusercontent.com/40661020/143876061-4de20768-c812-4b97-b089-944d8db91ca2.png"/>
</div>

</div>

|     Backbone     |                              Batch Size                              | Step/Total Epochs | Mem (GB) | Inf time (fps) | box AP |                                Config                                |                                                                                                                                                                                     Download                                                                                                                                                                                     |

| HourglassNet-104 | [10 x 5](./cornernet_hourglass104_10xb5-crop511-210e-mstest_coco.py) |      180/210      |   13.9   |      4.2       |  41.2  | [config](./cornernet_hourglass104_10xb5-crop511-210e-mstest_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/cornernet/cornernet_hourglass104_mstest_10x5_210e_coco/cornernet_hourglass104_mstest_10x5_210e_coco_20200824_185720-5fefbf1c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cornernet/cornernet_hourglass104_mstest_10x5_210e_coco/cornernet_hourglass104_mstest_10x5_210e_coco_20200824_185720.log.json) |

| HourglassNet-104 |      [8 x 6](./cornernet_hourglass104_8xb6-210e-mstest_coco.py)      |      180/210      |   15.9   |      4.2       |  41.2  |     [config](./cornernet_hourglass104_8xb6-210e-mstest_coco.py)      |   [model](https://download.openmmlab.com/mmdetection/v2.0/cornernet/cornernet_hourglass104_mstest_8x6_210e_coco/cornernet_hourglass104_mstest_8x6_210e_coco_20200825_150618-79b44c30.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cornernet/cornernet_hourglass104_mstest_8x6_210e_coco/cornernet_hourglass104_mstest_8x6_210e_coco_20200825_150618.log.json)   |

| HourglassNet-104 |     [32 x 3](./cornernet_hourglass104_32xb3-210e-mstest_coco.py)     |      180/210      |   9.5    |      3.9       |  40.4  |     [config](./cornernet_hourglass104_32xb3-210e-mstest_coco.py)     | [model](https://download.openmmlab.com/mmdetection/v2.0/cornernet/cornernet_hourglass104_mstest_32x3_210e_coco/cornernet_hourglass104_mstest_32x3_210e_coco_20200819_203110-1efaea91.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cornernet/cornernet_hourglass104_mstest_32x3_210e_coco/cornernet_hourglass104_mstest_32x3_210e_coco_20200819_203110.log.json) |

Note:

- Experiments with `images_per_gpu=6` are conducted on Tesla V100-SXM2-32GB, `images_per_gpu=3` are conducted on GeForce GTX 1080 Ti.
- Here are the descriptions of each experiment setting:

  - 10 x 5: 10 GPUs with 5 images per gpu. This is the same setting as that reported in the original paper.
  - 8 x 6: 8 GPUs with 6 images per gpu. The total batchsize is similar to paper and only need 1 node to train.
  - 32 x 3: 32 GPUs with 3 images per gpu. The default setting for 1080TI and need 4 nodes to train.

```latex
@inproceedings{law2018cornernet,
  title={Cornernet: Detecting objects as paired keypoints},
  author={Law, Hei and Deng, Jia},
  booktitle={15th European Conference on Computer Vision, ECCV 2018},
  pages={765--781},
  organization={Springer Verlag}
}
```

# DenseCL

> [Dense contrastive learning for self-supervised visual pre-training](https://arxiv.org/abs/2011.09157)

To date, most existing self-supervised learning methods are designed and optimized for image classification. These pre-trained models can be sub-optimal for dense prediction tasks due to the discrepancy between image-level prediction and pixel-level prediction. To fill this gap, we aim to design an effective, dense self-supervised learning method that directly works at the level of pixels (or local features) by taking into account the correspondence between local features. We present dense contrastive learning (DenseCL), which implements self-supervised learning by optimizing a pairwise contrastive (dis)similarity loss at the pixel level between two views of input images.

<img src="https://user-images.githubusercontent.com/36138628/149721111-bab03a6d-a30d-418e-b338-43c3689cfc65.png" width="900" />
</div>

</div>

```python

predict = inference_model('resnet50_densecl-pre_8xb32-linear-steplr-100e_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('densecl_resnet50_8xb32-coslr-200e_in1k', pretrained=True)
```

**Train/Test Command**

Train:

```shell
python tools/train.py configs/densecl/densecl_resnet50_8xb32-coslr-200e_in1k.py

Train:

```shell
```

Test:

```shell
python tools/test.py configs/densecl/benchmarks/resnet50_8xb32-linear-steplr-100e_in1k.py https://download.openmmlab.com/mmselfsup/1.x/densecl/densecl_resnet50_8xb32-coslr-200e_in1k/resnet50_linear-8xb32-steplr-100e_in1k/resnet50_linear-8xb32-steplr-100e_in1k_20220825-f0f0a579.pth
```

### Pretrained models

| Model                                    | Params (M) | Flops (G) |                       Config                        |                                          Download                                          |
| :--------------------------------------- | :--------: | :-------: | :-------------------------------------------------: | :----------------------------------------------------------------------------------------: |

| `densecl_resnet50_8xb32-coslr-200e_in1k` |   64.85    |   4.11    | [config](densecl_resnet50_8xb32-coslr-200e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/densecl/densecl_resnet50_8xb32-coslr-200e_in1k/densecl_resnet50_8xb32-coslr-200e_in1k_20220825-3078723b.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/densecl/densecl_resnet50_8xb32-coslr-200e_in1k/densecl_resnet50_8xb32-coslr-200e_in1k_20220825-3078723b.json) |

| Model                                     |                   Pretrain                   | Params (M) | Flops (G) | Top-1 (%) |                   Config                   |                   Download                    |
| :---------------------------------------- | :------------------------------------------: | :--------: | :-------: | :-------: | :----------------------------------------: | :-------------------------------------------: |

| `resnet50_densecl-pre_8xb32-linear-steplr-100e_in1k` | [DENSECL](https://download.openmmlab.com/mmselfsup/1.x/densecl/densecl_resnet50_8xb32-coslr-200e_in1k/densecl_resnet50_8xb32-coslr-200e_in1k_20220825-3078723b.pth) |   25.56    |   4.11    |   63.50   | [config](benchmarks/resnet50_8xb32-linear-steplr-100e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/densecl/densecl_resnet50_8xb32-coslr-200e_in1k/resnet50_linear-8xb32-steplr-100e_in1k/resnet50_linear-8xb32-steplr-100e_in1k_20220825-f0f0a579.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/densecl/densecl_resnet50_8xb32-coslr-200e_in1k/resnet50_linear-8xb32-steplr-100e_in1k/resnet50_linear-8xb32-steplr-100e_in1k_20220825-f0f0a579.json) |

```bibtex
@inproceedings{wang2021dense,
  title={Dense contrastive learning for self-supervised visual pre-training},
  author={Wang, Xinlong and Zhang, Rufeng and Shen, Chunhua and Kong, Tao and Li, Lei},
}
```

  <img src="resources/mmrotate-logo.png" width="450"/>
    <sup>
      </a>
    </sup>
    <sup>
      </a>
    </sup>
  </div>

[![PyPI](https://img.shields.io/pypi/v/mmrotate)](https://pypi.org/project/mmrotate)
[![docs](https://img.shields.io/badge/docs-latest-blue)](https://mmrotate.readthedocs.io/en/1.x/)
[![badge](https://github.com/open-mmlab/mmrotate/workflows/build/badge.svg)](https://github.com/open-mmlab/mmrotate/actions)
[![codecov](https://codecov.io/gh/open-mmlab/mmrotate/branch/main/graph/badge.svg)](https://codecov.io/gh/open-mmlab/mmrotate)

[![license](https://img.shields.io/github/license/open-mmlab/mmrotate.svg)](https://github.com/open-mmlab/mmrotate/blob/main/LICENSE)
[![open issues](https://isitmaintained.com/badge/open/open-mmlab/mmrotate.svg)](https://github.com/open-mmlab/mmrotate/issues)
[![issue resolution](https://isitmaintained.com/badge/resolution/open-mmlab/mmrotate.svg)](https://github.com/open-mmlab/mmrotate/issues)

[📘Documentation](https://mmrotate.readthedocs.io/en/1.x/) |
[🛠️Installation](https://mmrotate.readthedocs.io/en/1.x/install.html) |
[👀Model Zoo](https://mmrotate.readthedocs.io/en/1.x/model_zoo.html) |
[🆕Update News](https://mmrotate.readthedocs.io/en/1.x/notes/changelog.html) |
[🚀Ongoing Projects](https://github.com/open-mmlab/mmrotate/projects) |
[🤔Reporting Issues](https://github.com/open-mmlab/mmrotate/issues/new/choose)

</div>

<!--中/英 文档切换-->

</div>

</div>

MMRotate is an open-source toolbox for rotated object detection based on PyTorch.
It is a part of the [OpenMMLab project](https://github.com/open-mmlab).

The master branch works with **PyTorch 1.6+**.

https://user-images.githubusercontent.com/10410257/154433305-416d129b-60c8-44c7-9ebb-5ba106d3e9d5.MP4

<details open>
<summary><b>Major Features</b></summary>

- **Support multiple angle representations**

  MMRotate provides three mainstream angle representations to meet different paper settings.

- **Modular Design**

  We decompose the rotated object detection framework into different components,
  which makes it much easy and flexible to build a new model by combining different modules.

- **Strong baseline and State of the art**

which makes it much easy and flexible to build a new model by combining different modules.

  The toolbox provides strong baselines and state-of-the-art methods in rotated object detection.

## What's New

### Highlight

We are excited to announce our latest work on real-time object recognition tasks, **RTMDet**, a family of fully convolutional single-stage detectors. RTMDet not only achieves the best parameter-accuracy trade-off on object detection from tiny to extra-large model sizes but also obtains new state-of-the-art performance on instance segmentation and rotated object detection tasks. Details can be found in the [technical report](https://arxiv.org/abs/2212.07784). Pre-trained models are [here](configs/rotated_rtmdet).

[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/rtmdet-an-empirical-study-of-designing-real/real-time-instance-segmentation-on-mscoco)](https://paperswithcode.com/sota/real-time-instance-segmentation-on-mscoco?p=rtmdet-an-empirical-study-of-designing-real)
[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/rtmdet-an-empirical-study-of-designing-real/object-detection-in-aerial-images-on-dota-1)](https://paperswithcode.com/sota/object-detection-in-aerial-images-on-dota-1?p=rtmdet-an-empirical-study-of-designing-real)
[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/rtmdet-an-empirical-study-of-designing-real/object-detection-in-aerial-images-on-hrsc2016)](https://paperswithcode.com/sota/object-detection-in-aerial-images-on-hrsc2016?p=rtmdet-an-empirical-study-of-designing-real)

| Task                     | Dataset | AP                                   | FPS(TRT FP16 BS1 3090) |

| ------------------------ | ------- | ------------------------------------ | ---------------------- |
| Object Detection         | COCO    | 52.8                                 | 322                    |
| Instance Segmentation    | COCO    | 44.6                                 | 188                    |
| Rotated Object Detection | DOTA    | 78.9(single-scale)/81.3(multi-scale) | 121                    |

<img src="https://user-images.githubusercontent.com/12907710/208044554-1e8de6b5-48d8-44e4-a7b5-75076c7ebb71.png"/>
</div>

**v1.0.0rc1** was released in 30/12/2022:

- Support [RTMDet](configs/rotated_rtmdet) rotated object detection models. The technical report of RTMDet is on [arxiv](https://arxiv.org/abs/2212.07784)
- Support [H2RBox](configs/h2rbox) models. The technical report of H2RBox is on [arxiv](https://arxiv.org/abs/2210.06742)

Please refer to [Installation](https://mmrotate.readthedocs.io/en/1.x/get_started.html) for more detailed instruction.

## Getting Started

Please see [Overview](https://mmrotate.readthedocs.io/en/1.x/overview.html) for the general introduction of MMRotate.

For detailed user guides and advanced guides, please refer to our [documentation](https://mmrotate.readthedocs.io/en/1.x/):

- User Guides
  - [Train & Test](https://mmrotate.readthedocs.io/en/1.x/user_guides/index.html#train-test)
    - [Learn about Configs](https://mmrotate.readthedocs.io/en/1.x/user_guides/config.html)
    - [Inference with existing models](https://mmrotate.readthedocs.io/en/1.x/user_guides/inference.html)
    - [Dataset Prepare](https://mmrotate.readthedocs.io/en/1.x/user_guides/dataset_prepare.html)

- [Dataset Prepare](https://mmrotate.readthedocs.io/en/1.x/user_guides/dataset_prepare.html)
    - [Test existing models on standard datasets](https://mmrotate.readthedocs.io/en/1.x/user_guides/train_test.html)
    - [Train predefined models on standard datasets](https://mmrotate.readthedocs.io/en/1.x/user_guides/train_test.html)
    - [Test Results Submission](https://mmrotate.readthedocs.io/en/1.x/user_guides/test_results_submission.html)
  - [Useful Tools](https://mmrotate.readthedocs.io/en/1.x/user_guides/index.html#useful-tools)
- Advanced Guides
  - [Basic Concepts](https://mmrotate.readthedocs.io/en/1.x/advanced_guides/index.html#basic-concepts)
  - [Component Customization](https://mmrotate.readthedocs.io/en/1.x/advanced_guides/index.html#component-customization)
  - [How to](https://mmrotate.readthedocs.io/en/1.x/advanced_guides/index.html#how-to)

- [How to](https://mmrotate.readthedocs.io/en/1.x/advanced_guides/index.html#how-to)

We also provide colab tutorial [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](demo/MMRotate_Tutorial.ipynb).

To migrate from MMRotate 0.x, please refer to [migration](https://mmrotate.readthedocs.io/en/1.x/migration.html).

## Model Zoo

Results and models are available in the *README.md* of each method's config directory.
A summary can be found in the [Model Zoo](docs/en/model_zoo.md) page.

<summary><b>Supported algorithms:</b></summary>

- [x] [Rotated RetinaNet-OBB/HBB](configs/rotated_retinanet/README.md) (ICCV'2017)
- [x] [Rotated FasterRCNN-OBB](configs/rotated_faster_rcnn/README.md) (TPAMI'2017)
- [x] [Rotated RepPoints-OBB](configs/rotated_reppoints/README.md) (ICCV'2019)
- [x] [Rotated FCOS](configs/rotated_fcos/README.md) (ICCV'2019)
- [x] [RoI Transformer](configs/roi_trans/README.md) (CVPR'2019)

- [x] [Gliding Vertex](configs/gliding_vertex/README.md) (TPAMI'2020)
- [x] [Rotated ATSS-OBB](configs/rotated_atss/README.md) (CVPR'2020)
- [x] [CSL](configs/csl/README.md) (ECCV'2020)
- [x] [R<sup>3</sup>Det](configs/r3det/README.md) (AAAI'2021)
- [x] [S<sup>2</sup>A-Net](configs/s2anet/README.md) (TGRS'2021)
- [x] [ReDet](configs/redet/README.md) (CVPR'2021)
- [x] [Beyond Bounding-Box](configs/cfa/README.md) (CVPR'2021)
- [x] [Oriented R-CNN](configs/oriented_rcnn/README.md) (ICCV'2021)
- [x] [GWD](configs/gwd/README.md) (ICML'2021)
- [x] [KLD](configs/kld/README.md) (NeurIPS'2021)
- [x] [SASM](configs/sasm_reppoints/README.md) (AAAI'2022)
- [x] [Oriented RepPoints](configs/oriented_reppoints/README.md) (CVPR'2022)
- [x] [KFIoU](configs/kfiou/README.md) (arXiv)
- [x] [H2RBox](configs/h2rbox/README.md) (arXiv)

## Data Preparation

Please refer to [data_preparation.md](tools/data/README.md) to prepare the data.

## FAQ

Please refer to [FAQ](docs/en/notes/faq.md) for frequently asked questions.

We appreciate all contributions to improve MMRotate. Please refer to [CONTRIBUTING.md](.github/CONTRIBUTING.md) for the contributing guideline.

MMRotate is an open source project that is contributed by researchers and engineers from various colleges and companies. We appreciate all the contributors who implement their methods or add new features, as well as users who give valuable feedbacks. We appreciate the [Student Innovation Center of SJTU](https://www.si.sjtu.edu.cn/) for providing rich computing resources at the beginning of the project. We wish that the toolbox and benchmark could serve the growing research community by providing a flexible toolkit to reimplement existing methods and develop their own new methods.

If you use this toolbox or benchmark in your research, please cite this project.

```bibtex
@inproceedings{zhou2022mmrotate,
  title   = {MMRotate: A Rotated Object Detection Benchmark using PyTorch},

```bibtex
  author  = {Zhou, Yue and Yang, Xue and Zhang, Gefan and Wang, Jiabao and Liu, Yanyi and
             Hou, Liping and Jiang, Xue and Liu, Xingzhao and Yan, Junchi and Lyu, Chengqi and
             Zhang, Wenwei and Chen, Kai},
  booktitle={Proceedings of the 30th ACM International Conference on Multimedia},
  pages = {7331–7334},
  numpages = {4},
}
```

## License

- [MMClassification](https://github.com/open-mmlab/mmclassification): OpenMMLab image classification toolbox and benchmark.

- [MMEditing](https://github.com/open-mmlab/mmediting): OpenMMLab image and video editing toolbox.

# 准备 SPMCS 数据集

```bibtex
@InProceedings{tao2017spmc,
  author={Xin Tao and Hongyun Gao and Renjie Liao and Jue Wang and Jiaya Jia},
  title = {Detail-Revealing Deep Video Super-Resolution},
  booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
  month = {Oct},
  year = {2017}
}
```

数据集可以从 [此处](https://opendatalab.org.cn/SPMCS) 下载。

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── SPMCS
|   |   ├── GT
|   |   ├── BIx4
|   |   ├── BDx4
|   |   ├── meta_info_SPMCS_GT.txt
```

## <a href='https://mmocr.readthedocs.io/en/dev-1.x/'>English</a>

## <a href='https://mmocr.readthedocs.io/zh_CN/dev-1.x/'>简体中文</a>

# Changelog v0.x

## v0.3.2 (6/7/2022)

- Support Oriented Reppoints (CVPR'22) (#286)
- Support ConvNeXt backbone (CVPR'22) (#343)

- Support RMosaic. (#344)

- Fix max_coordinate in multiclass_nms_rotated. (#346)
- Fix bug in PolyRandomRotate. (#366)
- Fix memory shortage when using huge_image_demo.py. (#368)

- Update README.md and INSTALL.md. (#342)
- Fix typo in rotated_fcos_head. (#354)
- Update checkpoint and eval interval of base config. (#347)
- Fix mdformat version to support python3.6 & Add mim to extras_require in setup.py. (#359)
- Add mim test in CI. (#374)

A total of 9 developers contributed to this release.
Thanks @LiWentomng @heiyuxiaokai @JinYuannn @sltlls @liuyanyi  @yangxue0827 @jbwang1997 @zytx121 @ZwwWayne

## v0.3.1 (6/6/2022)

- Support Rotated FCOS (#223)

- Update PolyRandomRotate to support discrete angle value. (#281)

- Support RRandomCrop. (#322)
- Support mask in merge_results and huge_image_demo.py. (#280)
- Support don't filter images without ground truths. (#323)
- Add MultiImageMixDataset in build_dataset. (#331)

- Fix error in Windows CI. (#324)
- Fix data path error in config files. (#328)
- Fix bug when visualize the HRSC2016 detect results. (#329)

- Add torchserve doc in zh_cn. (#287)
- Fix doc typo in README. (#284)
- Configure Myst-parser to parse anchor tag (#305 #308)
- Replace markdownlint with mdformat for avoiding installing ruby. (#306)
- Fix typo about split gap of multi scale. (#272)

A total of 7 developers contributed to this release.
Thanks @liuyanyi @nijkah @remi-or @yangxue0827 @jbwang1997 @zytx121 @ZwwWayne

## v0.3.0 (29/4/2022)

- Support TorchServe (#160)
- Support Rotated ATSS (CVPR'20) (#179)

- Update performance of ReDet on HRSC2016. (#203)

- Upgrage visualization to custom colors of different classes. This requires mmdet>=2.22.0. (#187, #267, #270)

- Update Stable KLD, which solve the Nan issue of KLD training. (#183)

- Support setting dataloader arguments in config and add functions to handle config compatibility. (#215)
  The comparison between the old and new usages is as below.

  <table align="center">
    <thead>
            <td>Before v0.2.0</td>
            <td>Since v0.3.0 </td>
        </tr>
    </thead>
    <th>

  ```python
  data = dict(
      samples_per_gpu=2, workers_per_gpu=2,
      train=dict(type='xxx', ...),

</tr>
    </thead>
    <th>

  ```python
      val=dict(type='xxx', samples_per_gpu=4, ...),
      test=dict(type='xxx', ...),
  )
  ```

  </th>
    <th>

  ```python
  # A recommended config that is clear
      val=dict(type='xxx', ...),
      # Use different batch size during inference.
      train_dataloader=dict(samples_per_gpu=2, workers_per_gpu=2),
      val_dataloader=dict(samples_per_gpu=4, workers_per_gpu=4),
      test_dataloader=dict(samples_per_gpu=4, workers_per_gpu=4),
  )

  # Old style still works but allows to set more arguments about data loaders
      samples_per_gpu=2,  # only works for train_dataloader
      workers_per_gpu=2,  # only works for train_dataloader

samples_per_gpu=2,  # only works for train_dataloader
  )
  ```

  </th></tr>

- Add [get_flops](tools/analysis_tools/get_flops.py) tool (#176)

- Fix bug about rotated anchor inside flags. (#197)
- Fix Nan issue of GWD. (#206)
- Fix bug in eval_rbbox_map when labels_ignore is None. (#209)
- Fix bug of 'RoIAlignRotated' object has no attribute 'output_size' (#213)
- Fix bug in unit test for datasets. (#222)
- Fix bug in rotated_reppoints_head. (#246)
- Fix GPG key error in CI and docker. (#269)

- Update citation of mmrotate in README.md (#263)

- Update the introduction of SASM (AAAI'22) (#184)
- Fix doc typo in Config File and Model Zoo. (#199)
- Unified RBox definition in doc. (#234)

Thanks @nijkah @GamblerZSY @liuyanyi @yangxue0827 @jbwang1997 @zytx121 @ZwwWayne

## v0.2.0 (30/3/2022)

- Support Circular Smooth Label (CSL, ECCV'20) (#153)
- Support multiple machines dist_train (#143)
- Add [browse_dataset](tools/misc/browse_dataset.py) tool (#98)
- Add [gather_models](.dev_scripts/gather_models.py) script (#162)

- Remove in-place operations in rbbox_overlaps (#155)
- Fix bug in docstring. (#137)
- Fix bug in HRSCDataset with `clasesswise=ture` (#175)

- Add Chinese translation of `docs/zh_cn/tutorials/customize_dataset.md` (#65)

- Add different seeds to different ranks (#102)
- Update from-scratch install script in install.md (#166)
- Improve the arguments of all mmrotate scripts (#168)

A total of 6 developers contributed to this release.
Thanks @zytx121 @yangxue0827 @ZwwWayne @jbwang1997 @canoe-Z @matrixgame2018

## v0.1.1 (14/3/2022)

- Support [huge image inference](deom/huge_image_demo.py) (#34)
- Support HRSC Dataset (#96)
- Support mixed precision training (#72)
- Add [colab tutorial](demo/MMRotate_Tutorial.ipynb) for beginners (#66)
- Add inference speed statistics [tool](tools/analysis_tools/benchmark.py) (#86)
- Add confusion matrix analysis [tool](tools/analysis_tools/confusion_matrix.py) (#93)

- Fix URL error of Swin pretrained model (#111)

- Fix bug for SASM during training (#105)
- Fix rbbox_overlaps abnormal when the box is too small (#61)
- Fix bug for visualization (#12, #81)
- Fix stuck when compute mAP (#14, #52)
- Fix 'RoIAlignRotated' object has no attribute 'out_size' bug (#51)
- Add missing init_cfg in dense head (#37)
- Fix install an additional mmcv (#17)
- Fix typos in docs (#3, #11, #36)

- Move `eval_rbbox_map` from `mmrotate.datasets` to `mmrotate.core.evaluation` (#73)
- Add  Windows CI (#31)
- Add copyright commit hook (#30)
- Add Chinese translation of `docs/zh_cn/get_started.md` (#16)
- Add Chinese translation of `docs/zh_cn/tutorials/customize_runtime.md` (#22)
- Add Chinese translation of ` docs/zh_cn/tutorials/customize_config.md` (#23)
- Add Chinese translation of `docs/zh_cn/tutorials/customize_models.md` (#27)

- Add Chinese translation of `docs/zh_cn/model_zoo.md` (#28)
- Add Chinese translation of `docs/zh_cn/faq.md` (#33)

A total of 13 developers contributed to this release.
Thanks @zytx121 @yangxue0827 @jbwang1997 @liuyanyi @DangChuong-DC @RangeKing @liufeinuaa @np-csu @akmalulkhairin @SheffieldCao @BrotherHappy @Abyssaledge  @q3394101

# 打印完整配置文件

`print_config.py`脚本脚本会解析所有输入变量，并打印完整配置信息。

## 说明：

`tools/misc/print_config.py` 脚本会逐字打印整个配置文件，并展示所有导入的文件。

```shell
python tools/misc/print_config.py ${CONFIG} [--cfg-options ${CFG_OPTIONS}]
```

所有参数的说明：

- `config`：模型配置文件的路径。
- `--cfg-options`：额外的配置选项，会被合入配置文件，参考[学习配置文件](../user_guides/config.md)。

## 示例：

打印`configs/t2t_vit/t2t-vit-t-14_8xb64_in1k.py`文件的完整配置

```shell
# 打印完成的配置文件
python tools/misc/print_config.py configs/t2t_vit/t2t-vit-t-14_8xb64_in1k.py

# 将完整的配置文件保存为一个独立的配置文件
python tools/misc/print_config.py configs/t2t_vit/t2t-vit-t-14_8xb64_in1k.py > final_config.py
```

# 准备数据集

在这份文档将指导如何为 MMPose 准备数据集，包括使用内置数据集、创建自定义数据集、结合数据集进行训练、浏览和下载数据集。

## 使用内置数据集

**步骤一**: 准备数据

MMPose 支持多种任务和相应的数据集。你可以在 [数据集仓库](https://mmpose.readthedocs.io/en/latest/dataset_zoo.html) 中找到它们。为了正确准备你的数据，请按照你选择的数据集的指南进行操作。

**步骤二**: 在配置文件中进行数据集设置

在开始训练或评估模型之前，你必须配置数据集设置。以 [`td-hm_hrnet-w32_8xb64-210e_coco-256x192.py`](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192.py) 为例，它可以用于在 COCO 数据集上训练或评估 HRNet 姿态估计器。下面我们浏览一下数据集配置：

- 基础数据集参数

  ```python
  ```

  - `dataset_type` 指定数据集的类名。用户可以参考 [数据集 API](https://mmpose.readthedocs.io/en/latest/api.html#datasets) 来找到他们想要的数据集的类名。
  - `data_mode` 决定了数据集的输出格式，有两个选项可用：`'topdown'` 和 `'bottomup'`。如果 `data_mode='topdown'`，数据元素表示一个实例及其姿态；否则，一个数据元素代表一张图像，包含多个实例和姿态。
  - `data_root` 指定数据集的根目录。

- 数据处理流程

  ```python

- `data_root` 指定数据集的根目录。

- 数据处理流程

  ```python
  ]
  ]
  ```

  `train_pipeline` 和 `val_pipeline` 分别定义了训练和评估阶段处理数据元素的步骤。除了加载图像和打包输入之外，`train_pipeline` 主要包含数据增强技术和目标生成器，而 `val_pipeline` 则专注于将数据元素转换为统一的格式。

- 数据加载器

  ```python

# data loaders
      ))

bbox_file='data/coco/person_detection_results/'
      ))
  ```

  这个部分是配置数据集的关键。除了前面讨论过的基础数据集参数和数据处理流程之外，这里还定义了其他重要的参数。`batch_size` 决定了每个 GPU 的 batch size；`ann_file` 指定了数据集的注释文件；`data_prefix` 指定了图像文件夹。`bbox_file` 仅在 top-down 数据集的 val/test 数据加载器中使用，用于提供检测到的边界框信息。

我们推荐从使用相同数据集的配置文件中复制数据集配置，而不是从头开始编写，以最小化潜在的错误。通过这样做，用户可以根据需要进行必要的修改，从而确保更可靠和高效的设置过程。

## 使用自定义数据集

[自定义数据集](../advanced_guides/customize_datasets.md) 指南提供了如何构建自定义数据集的详细信息。在本节中，我们将强调一些使用和配置自定义数据集的关键技巧。

- 确定数据集类名。如果你将数据集重组为 COCO 格式，你可以简单地使用 `CocoDataset` 作为 `dataset_type` 的值。否则，你将需要使用你添加的自定义数据集类的名称。

- 指定元信息配置文件。MMPose 1.x 采用了与 MMPose 0.x 不同的策略来指定元信息。在 MMPose 1.x 中，用户可以按照以下方式指定元信息配置文件：

  ```python
      ...

  ```python
      ...
          ...),
  )
  ```

  注意，`metainfo` 参数必须在 val/test 数据加载器中指定。

## 使用混合数据集进行训练

MMPose 提供了一个方便且多功能的解决方案，用于训练混合数据集。请参考[混合数据集训练](./mixed_datasets.md)。

## 浏览数据集

`tools/analysis_tools/browse_dataset.py` 帮助用户可视化地浏览姿态数据集，或将图像保存到指定的目录。

```shell
```

```shell
```

| ARGS                             | Description                                                                                                |
| -------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `CONFIG`                         | 配置文件的路径                                                                                             |
| `--output-dir OUTPUT_DIR`        | 保存可视化结果的目标文件夹。如果不指定，可视化的结果将不会被保存                                           |
| `--not-show`                     | 不适用外部窗口显示可视化的结果                                                                             |
| `--phase {train, val, test}`     | 数据集选项                                                                                                 |

| `--mode {original, transformed}` | 指定可视化图片类型。 `original` 为不使用数据增强的原始图片及标注可视化; `transformed` 为经过增强后的可视化 |
| `--show-interval SHOW_INTERVAL`  | 显示图片的时间间隔                                                                                         |

例如，用户想要可视化 COCO 数据集中的图像和标注，可以使用：

```shell
```

检测框和关键点将被绘制在原始图像上。下面是一个例子：

原始图像在被输入模型之前需要被处理。为了可视化预处理后的图像和标注，用户需要将参数 `mode` 修改为 `transformed`。例如：

```shell
```

这是一个处理后的样本：

```shell
```

热图目标将与之一起可视化，如果它是在 pipeline 中生成的。

## 用 MIM 下载数据集

通过使用 [OpenDataLab](https://opendatalab.com/)，您可以直接下载开源数据集。通过平台的搜索功能，您可以快速轻松地找到他们正在寻找的数据集。使用平台上的格式化数据集，您可以高效地跨数据集执行任务。

如果您使用 MIM 下载，请确保版本大于 v0.3.8。您可以使用以下命令进行更新、安装、登录和数据集下载：

```shell

odl login

```

### 已支持的数据集

下面是支持的数据集列表，更多数据集将在之后持续更新：

#### 人体数据集

#### 人体数据集

#### 人脸数据集

#### 手部数据集

#### 全身数据集

#### 动物数据集

#### 服装数据集

## 2D Animal Pose Demo

We provide a demo script to test a single image or video with top-down pose estimators and animal detectors. Assume that you have already installed [mmdet](https://github.com/open-mmlab/mmdetection) with version >= 3.0.

### 2D Animal Pose Image Demo

```shell
python demo/topdown_demo_with_mmdet.py \
    ${MMDET_CONFIG_FILE} ${MMDET_CHECKPOINT_FILE} \
    ${MMPOSE_CONFIG_FILE} ${MMPOSE_CHECKPOINT_FILE} \
    --input ${INPUT_PATH} --det-cat-id ${DET_CAT_ID} \
    [--show] [--output-root ${OUTPUT_DIR}] [--save-predictions] \
    [--draw-heatmap ${DRAW_HEATMAP}] [--radius ${KPT_RADIUS}] \
    [--kpt-thr ${KPT_SCORE_THR}] [--bbox-thr ${BBOX_SCORE_THR}] \
    [--device ${GPU_ID or CPU}]
```

The pre-trained animal pose estimation model can be found from [model zoo](https://mmpose.readthedocs.io/en/latest/model_zoo/animal_2d_keypoint.html).

[--device ${GPU_ID or CPU}]
```

Take [animalpose model](https://download.openmmlab.com/mmpose/animal/hrnet/hrnet_w32_animalpose_256x256-1aa7f075_20210426.pth) as an example:

```shell
    demo/mmdetection_cfg/faster_rcnn_r50_fpn_coco.py \
    https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
    configs/animal_2d_keypoint/topdown_heatmap/animalpose/td-hm_hrnet-w32_8xb64-210e_animalpose-256x256.py \
    https://download.openmmlab.com/mmpose/animal/hrnet/hrnet_w32_animalpose_256x256-1aa7f075_20210426.pth \
    --input tests/data/animalpose/ca110.jpeg \
    --show --draw-heatmap --det-cat-id=15
```

Visualization result:

--input tests/data/animalpose/ca110.jpeg \
```

<img src="https://user-images.githubusercontent.com/26127467/187644168-5915551a-0876-4b85-9454-7f92c84ba6fb.jpeg" height="500px" alt><br>

If you use a heatmap-based model and set argument `--draw-heatmap`, the predicted heatmap will be visualized together with the keypoints.

The augement `--det-cat-id=15` selected detected bounding boxes with label 'cat'. 15 is the index of category 'cat' in COCO dataset, on which the detection model is trained.

**COCO-animals**
In COCO dataset, there are 80 object categories, including 10 common `animal` categories (14: 'bird', 15: 'cat', 16: 'dog', 17: 'horse', 18: 'sheep', 19: 'cow', 20: 'elephant', 21: 'bear', 22: 'zebra', 23: 'giraffe').

For other animals, we have also provided some pre-trained animal detection models (1-class models). Supported models can be found in [detection model zoo](/demo/docs/en/mmdet_modelzoo.md).

To save visualized results on disk:

```shell
    --output-root vis_results --draw-heatmap --det-cat-id=15
```

To save predicted results on disk:

```shell

```

```shell
    --output-root vis_results --save-predictions --draw-heatmap --det-cat-id=15
```

To run demos on CPU:

```shell

    --show --draw-heatmap --det-cat-id=15 --device cpu
```

### 2D Animal Pose Video Demo

Videos share the same interface with images. The difference is that the `${INPUT_PATH}` for videos can be the local path or **URL** link to video file.

For example,

```shell

    --input demo/resources/<demo_dog.mp4> \
    --output-root vis_results --draw-heatmap --det-cat-id=16
```

<img src="https://user-images.githubusercontent.com/26127467/187655602-907db86e-710b-447a-8ec9-5b623d43d160.gif" height="500px" alt><br>

The original video can be downloaded from [Google Drive](https://drive.google.com/file/d/18d8K3wuUpKiDFHvOx0mh1TEwYwpOc5UO/view?usp=sharing).

### 2D Animal Pose Demo with Inferencer

```shell
python demo/inferencer_demo.py tests/data/ap10k \
    --pose2d animal --vis-out-dir vis_results/ap10k
```

This command infers all images located in `tests/data/ap10k` and saves the visualization results in the `vis_results/ap10k` directory.

--pose2d animal --vis-out-dir vis_results/ap10k
```

<img src="https://user-images.githubusercontent.com/26127467/229789306-83ea56fa-12f2-4e27-9031-329d335ec26d.jpg" alt="Image 1" height="200"/> <img src="https://user-images.githubusercontent.com/26127467/229789324-7fef5688-422d-4663-a57c-d1e1d511e83c.jpg" alt="Image 2" height="200"/>

In addition, the Inferencer supports saving predicted poses. For more information, please refer to the [inferencer document](https://mmpose.readthedocs.io/en/dev-1.x/user_guides/inference.html#inferencer-a-unified-inference-interface).

### Speed Up Inference

Some tips to speed up MMPose inference:

1. set `model.test_cfg.flip_test=False` in [animalpose_hrnet-w32](../../configs/animal_2d_keypoint/topdown_heatmap/animalpose/td-hm_hrnet-w32_8xb64-210e_animalpose-256x256.py#85).

2. use faster human bounding box detector, see [MMDetection](https://mmdetection.readthedocs.io/en/3.x/model_zoo.html).

# Weight initialization

During training, a proper initialization strategy is beneficial to speeding up the training or obtaining a higher performance. [MMCV](https://github.com/open-mmlab/mmcv/blob/master/mmcv/cnn/utils/weight_init.py) provide some commonly used methods for initializing modules like `nn.Conv2d`. Model initialization in MMdetection mainly uses `init_cfg`. Users can initialize models with following two steps:

1. Define `init_cfg` for a model or its components in `model_cfg`,  but `init_cfg` of children components have higher priority and will override `init_cfg` of parents modules.
2. Build model as usual, but call `model.init_weights()` method explicitly, and model parameters will be initialized as configuration.

The high-level workflow of initialization in MMdetection is :

model_cfg(init_cfg) -> build_from_cfg -> model -> init_weight() -> initialize(self, self.init_cfg) -> children's init_weight()

### Description

It is dict or list\[dict\], and contains the following keys and values:

- `type` (str), containing the initializer name in `INTIALIZERS`, and followed by arguments of the initializer.
- `layer` (str or list\[str\]), containing the names of basic layers in Pytorch or MMCV with learnable parameters that will be initialized, e.g. `'Conv2d'`,`'DeformConv2d'`.
- `override` (dict or list\[dict\]),  containing the sub-modules that not inherit from BaseModule and whose initialization configuration is different from other layers' which are in `'layer'` key. Initializer defined in `type` will work for all layers defined in `layer`, so if sub-modules are not derived Classes of `BaseModule` but can be initialized as same ways of layers in `layer`, it does not need to use `override`. `override` contains:
  - `type` followed by arguments of initializer;

- `type` followed by arguments of initializer;
  - `name` to indicate sub-module which will be initialized.

### Initialize parameters

Inherit a new model from `mmcv.runner.BaseModule` or `mmdet.models`  Here we show an example of FooModel.

```python
from mmcv.runner import BaseModule

class FooModel(BaseModule)
	def __init__(self,
                 arg1,
                 arg2,
                 init_cfg=None):
    	super(FooModel, self).__init__(init_cfg)
		...
```

- Initialize model by using `init_cfg` directly in code

  ```python
  import torch.nn as nn
  from mmcv.runner import BaseModule
  # or directly inherit mmdet models

  class FooModel(BaseModule)
  	def __init__(self,
                  arg1,
                  arg2,
                  init_cfg=XXX):
  		super(FooModel, self).__init__(init_cfg)
  	    ...
  ```

- Initialize model by using `init_cfg` directly in `mmcv.Sequential` or `mmcv.ModuleList` code

  ```python

super(FooModel, self).__init__(init_cfg)
  	    ...
  ```

  ```python
  from mmcv.runner import BaseModule, ModuleList

              	arg1,
              	arg2,
              	init_cfg=None):
      	...
      	self.conv1 = ModuleList(init_cfg=XXX)
  ```

- Initialize model by using `init_cfg` in config file

  ```python
  model = dict(
  	...
  	model = dict(
      	type='FooModel',
      	arg1=XXX,
      	arg2=XXX,
      	init_cfg=XXX),
          ...
  ```

### Usage of init_cfg

1. Initialize model by `layer` key

   If we only define `layer`, it just initialize the layer in `layer` key.

   NOTE: Value of `layer` key is the class name with attributes weights and bias of Pytorch, (so such as  `MultiheadAttention layer` is not supported).

NOTE: Value of `layer` key is the class name with attributes weights and bias of Pytorch, (so such as  `MultiheadAttention layer` is not supported).

- Define `layer` key for initializing module with same configuration.

  ```python
  init_cfg = dict(type='Constant', layer=['Conv1d', 'Conv2d', 'Linear'], val=1)
  # initialize whole module with same configuration
  ```

- Define `layer` key for initializing layer with different configurations.

```python
init_cfg = [dict(type='Constant', layer='Conv1d', val=1),
            dict(type='Constant', layer='Conv2d', val=2),
            dict(type='Constant', layer='Linear', val=3)]
# nn.Conv1d will be initialized with dict(type='Constant', val=1)
# nn.Conv2d will be initialized with dict(type='Constant', val=2)
# nn.Linear will be initialized with dict(type='Constant', val=3)
```

2. Initialize model by `override` key

```

- When initializing some specific part with its attribute name, we can use `override` key, and the value in `override` will ignore the value in init_cfg.

  ```python
  # layers：
  # self.feat = nn.Conv1d(3, 1, 3)
  # self.reg = nn.Conv2d(3, 3, 3)
  # self.cls = nn.Linear(1,2)

  init_cfg = dict(type='Constant',
                  layer=['Conv1d','Conv2d'], val=1, bias=2,
                  override=dict(type='Constant', name='reg', val=3, bias=4))
  # self.feat and self.cls will be initialized with 	dict(type='Constant', val=1, bias=2)
  # The module called 'reg' will be initialized with dict(type='Constant', val=3, bias=4)
  ```

- If `layer` is None in init_cfg, only sub-module with the name in override will be initialized, and type and other args in override can be omitted.

  ```python
  # layers：

```

  ```python
  # layers：

  init_cfg = dict(type='Constant', val=1, bias=2, 	override=dict(name='reg'))

  # self.feat and self.cls will be initialized by Pytorch
  # The module called 'reg' will be initialized with dict(type='Constant', val=1, bias=2)
  ```

- If we don't define `layer` key or `override` key, it will not initialize anything.

- Invalid usage

  ```python
  # It is invalid that override don't have name key
  init_cfg = dict(type='Constant', layer=['Conv1d','Conv2d'], val=1, bias=2,
              	override=dict(type='Constant', val=3, bias=4))

  # It is also invalid that override has name and other args except type

# It is also invalid that override has name and other args except type
                  override=dict(name='reg', val=3, bias=4))
  ```

3. Initialize model with the pretrained model

   ```python
   init_cfg = dict(type='Pretrained',
               checkpoint='torchvision://resnet50')
   ```

More details can refer to the documentation in [MMEngine](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/initialize.html)

# Mask-pose cascaded cnn for 2d hand pose estimation from single color image

<details>
<summary align="right"><a href="https://ieeexplore.ieee.org/abstract/document/8529221/">OneHand10K (TCSVT'2019)</a></summary>

```bibtex
@article{wang2018mask,
  title={Mask-pose cascaded cnn for 2d hand pose estimation from single color image},
  author={Wang, Yangang and Peng, Cong and Liu, Yebin},
  journal={IEEE Transactions on Circuits and Systems for Video Technology},
  volume={29},
  number={11},
  pages={3258--3268},
}
```

**Text Recognition**

```txt
./3000/7/182_slinking_71711.jpg 71711
./3000/7/182_REMODELERS_64541.jpg 64541
```

<details>
<summary align="right"><a href="https://ieeexplore.ieee.org/abstract/document/9052469/">HRNetv2 (TPAMI'2019)</a></summary>

```bibtex
@article{WangSCJDZLMTWLX19,
  title={Deep High-Resolution Representation Learning for Visual Recognition},
  author={Jingdong Wang and Ke Sun and Tianheng Cheng and
          Borui Jiang and Chaorui Deng and Yang Zhao and Dong Liu and Yadong Mu and
          Mingkui Tan and Xinggang Wang and Wenyu Liu and Bin Xiao},
  journal={TPAMI},
}
```

<details>

```bibtex

}
```

<details>

```bibtex
}
```

Results on WFLW dataset

The model is trained on WFLW train.

pages={2129--2138},
}
```

| Arch       | Input Size | NME<sub>*test*</sub> | NME<sub>*pose*</sub> | NME<sub>*illumination*</sub> | NME<sub>*occlusion*</sub> | NME<sub>*blur*</sub> | NME<sub>*makeup*</sub> | NME<sub>*expression*</sub> |    ckpt    |    log    |
| :--------- | :--------: | :------------------: | :------------------: | :--------------------------: | :-----------------------: | :------------------: | :--------------------: | :------------------------: | :--------: | :-------: |

| [pose_hrnetv2_w18_dark](/configs/face_2d_keypoint/topdown_heatmap/wflw/td-hm_hrnetv2-w18_dark-8xb64-60e_wflw-256x256.py) |  256x256   |         3.98         |         6.98         |             3.96             |           4.78            |         4.56         |          3.89          |            4.29            | [ckpt](https://download.openmmlab.com/mmpose/face/darkpose/hrnetv2_w18_wflw_256x256_dark-3f8e0c2c_20210125.pth) | [log](https://download.openmmlab.com/mmpose/face/darkpose/hrnetv2_w18_wflw_256x256_dark_20210125.log.json) |

# Text Detection

```{note}
This page is a manual preparation guide for datasets not yet supported by [Dataset Preparer](./dataset_preparer.md), which all these scripts will be eventually migrated into.
```

|      Dataset      |                          Images                          |                                                    |                          Annotation Files                           |         |     |
| :---------------: | :------------------------------------------------------: | :------------------------------------------------: | :-----------------------------------------------------------------: | :-----: | :-: |
|                   |                                                          |                      training                      |                             validation                              | testing |     |

|     ICDAR2011     |         [homepage](https://rrc.cvc.uab.es/?ch=1)         |                         -                          |                                  -                                  |         |     |
|     ICDAR2017     |  [homepage](https://rrc.cvc.uab.es/?ch=8&com=downloads)  | [instances_training.json](https://download.openmmlab.com/mmocr/data/icdar2017/instances_training.json) | [instances_val.json](https://download.openmmlab.com/mmocr/data/icdar2017/instances_val.json) |    -    |     |

| CurvedSynText150k | [homepage](https://github.com/aim-uofa/AdelaiDet/blob/master/datasets/README.md) \| [Part1](https://drive.google.com/file/d/1OSJ-zId2h3t_-I7g_wUkrK-VqQy153Kj/view?usp=sharing) \| [Part2](https://drive.google.com/file/d/1EzkcOlIgEp5wmEubvHb7-J5EImHExYgY/view?usp=sharing) | [instances_training.json](https://download.openmmlab.com/mmocr/data/curvedsyntext/instances_training.json) |                                  -                                  |    -    |     |
|      DeText       |         [homepage](https://rrc.cvc.uab.es/?ch=9)         |                         -                          |                                  -                                  |    -    |     |
| Lecture Video DB  | [homepage](https://cvit.iiit.ac.in/research/projects/cvit-projects/lecturevideodb) |                         -                          |                                  -                                  |    -    |     |

|       LSVT        |        [homepage](https://rrc.cvc.uab.es/?ch=16)         |                         -                          |                                  -                                  |    -    |     |
|       IMGUR       | [homepage](https://github.com/facebookresearch/IMGUR5K-Handwriting-Dataset) |                         -                          |                                  -                                  |    -    |     |
|       KAIST       | [homepage](http://www.iapr-tc11.org/mediawiki/index.php/KAIST_Scene_Text_Database) |                         -                          |                                  -                                  |    -    |     |
|       MTWI        | [homepage](https://tianchi.aliyun.com/competition/entrance/231685/information?lang=en-us) |                         -                          |                                  -                                  |    -    |     |

|       ReCTS       |        [homepage](https://rrc.cvc.uab.es/?ch=12)         |                         -                          |                                  -                                  |    -    |     |
|     IIIT-ILST     | [homepage](http://cvit.iiit.ac.in/research/projects/cvit-projects/iiit-ilst) |                         -                          |                                  -                                  |    -    |     |
|      VinText      | [homepage](https://github.com/VinAIResearch/dict-guided) |                         -                          |                                  -                                  |    -    |     |
|        BID        | [homepage](https://github.com/ricardobnjunior/Brazilian-Identity-Document-Dataset) |                         -                          |                                  -                                  |    -    |     |

|       RCTW        |      [homepage](https://rctw.vlrlab.net/index.html)      |                         -                          |                                  -                                  |    -    |     |
|     HierText      | [homepage](https://github.com/google-research-datasets/hiertext) |                         -                          |                                  -                                  |    -    |     |
|        ArT        |        [homepage](https://rrc.cvc.uab.es/?ch=14)         |                         -                          |                                  -                                  |    -    |     |

### Install AWS CLI (optional)

- Since there are some datasets that require the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) to be installed in advance, we provide a quick installation guide here:

  ```bash

```bash
    curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
    unzip awscliv2.zip
    sudo ./aws/install
    ./aws/install -i /usr/local/aws-cli -b /usr/local/bin
    !aws configure
    # this command will require you to input keys, you can skip them except
    # for the Default region name
    # AWS Access Key ID [None]:
    # AWS Secret Access Key [None]:
    # Default region name [None]: us-east-1
    # Default output format [None]
  ```

For users in China, these datasets can also be downloaded from [OpenDataLab](https://opendatalab.com/) with high speed:

- [CTW1500](https://opendatalab.com/SCUT-CTW1500?source=OpenMMLab%20GitHub)
- [ICDAR2013](https://opendatalab.com/ICDAR_2013?source=OpenMMLab%20GitHub)
- [ICDAR2015](https://opendatalab.com/ICDAR2015?source=OpenMMLab%20GitHub)
- [Totaltext](https://opendatalab.com/TotalText?source=OpenMMLab%20GitHub)
- [MSRA-TD500](https://opendatalab.com/MSRA-TD500?source=OpenMMLab%20GitHub)

## Important Note

```{note}
**For users who want to train models on CTW1500, ICDAR 2015/2017, and Totaltext dataset,** there might be some images containing orientation info in EXIF data. The default OpenCV
backend used in MMCV would read them and apply the rotation on the images.  However, their gold annotations are made on the raw pixels, and such
inconsistency results in false examples in the training set. Therefore, users should use `dict(type='LoadImageFromFile', color_type='color_ignore_orientation')` in pipelines to change MMCV's default loading behaviour. (see [DBNet's pipeline config](https://github.com/open-mmlab/mmocr/blob/main/configs/_base_/det_pipelines/dbnet_pipeline.py) for example)
```

## ICDAR 2011 (Born-Digital Images)

```

- Step1: Download `Challenge1_Training_Task12_Images.zip`, `Challenge1_Training_Task1_GT.zip`, `Challenge1_Test_Task12_Images.zip`, and `Challenge1_Test_Task1_GT.zip` from [homepage](https://rrc.cvc.uab.es/?ch=1&com=downloads) `Task 1.1: Text Localization (2013 edition)`.

  ```bash
  mkdir icdar2011 && cd icdar2011
  mkdir imgs && mkdir annotations

  # Download ICDAR 2011
  wget https://rrc.cvc.uab.es/downloads/Challenge1_Training_Task12_Images.zip --no-check-certificate
  wget https://rrc.cvc.uab.es/downloads/Challenge1_Training_Task1_GT.zip --no-check-certificate
  wget https://rrc.cvc.uab.es/downloads/Challenge1_Test_Task12_Images.zip --no-check-certificate
  wget https://rrc.cvc.uab.es/downloads/Challenge1_Test_Task1_GT.zip --no-check-certificate

  # For images
  unzip -q Challenge1_Training_Task12_Images.zip -d imgs/training
  unzip -q Challenge1_Test_Task12_Images.zip -d imgs/test
  # For annotations

# For images
  unzip -q Challenge1_Training_Task1_GT.zip -d annotations/training
  unzip -q Challenge1_Test_Task1_GT.zip -d annotations/test

  rm Challenge1_Training_Task12_Images.zip && rm Challenge1_Test_Task12_Images.zip && rm Challenge1_Training_Task1_GT.zip && rm Challenge1_Test_Task1_GT.zip
  ```

- Step 2: Generate `instances_training.json` and `instances_test.json` with the following command:

  ```bash
  python tools/dataset_converters/textdet/ic11_converter.py PATH/TO/icdar2011 --nproc 4
  ```

- After running the above codes, the directory structure should be as follows:

  ```text
  │── icdar2011
  │   ├── imgs
  │   ├── instances_test.json
  │   └── instances_training.json
  ```

## ICDAR 2017

- Follow similar steps as [ICDAR 2015](#icdar-2015).

- The resulting directory structure looks like the following:

  ```text

│   └── instances_training.json
  ```

  ```text
  ├── icdar2017
  │   ├── imgs
  │   ├── annotations
  │   ├── instances_training.json
  │   └── instances_val.json
  ```

## CurvedSynText150k

- Step1: Download [syntext1.zip](https://drive.google.com/file/d/1OSJ-zId2h3t_-I7g_wUkrK-VqQy153Kj/view?usp=sharing) and [syntext2.zip](https://drive.google.com/file/d/1EzkcOlIgEp5wmEubvHb7-J5EImHExYgY/view?usp=sharing) to `CurvedSynText150k/`.

- Step2:

  ```bash
  unzip -q syntext1.zip
  mv train.json train1.json
  unzip images.zip
  rm images.zip

  unzip -q syntext2.zip
  mv train.json train2.json
  ```

- Step3: Download [instances_training.json](https://download.openmmlab.com/mmocr/data/curvedsyntext/instances_training.json) to `CurvedSynText150k/`

- Or, generate `instances_training.json` with following command:

  ```bash
  python tools/dataset_converters/common/curvedsyntext_converter.py PATH/TO/CurvedSynText150k --nproc 4
  ```

  ```text
  ├── CurvedSynText150k
  │   ├── syntext_word_eng
  │   ├── emcs_imgs
  │   └── instances_training.json
  ```

## DeText

- Step1: Download `ch9_training_images.zip`, `ch9_training_localization_transcription_gt.zip`, `ch9_validation_images.zip`, and `ch9_validation_localization_transcription_gt.zip` from **Task 3: End to End** on the [homepage](https://rrc.cvc.uab.es/?ch=9).

  ```bash
  mkdir detext && cd detext
  mkdir imgs && mkdir annotations && mkdir imgs/training && mkdir imgs/val && mkdir annotations/training && mkdir annotations/val

  # Download DeText
  wget https://rrc.cvc.uab.es/downloads/ch9_training_images.zip --no-check-certificate

# Download DeText
  wget https://rrc.cvc.uab.es/downloads/ch9_training_localization_transcription_gt.zip --no-check-certificate
  wget https://rrc.cvc.uab.es/downloads/ch9_validation_images.zip --no-check-certificate
  wget https://rrc.cvc.uab.es/downloads/ch9_validation_localization_transcription_gt.zip --no-check-certificate

  # Extract images and annotations
  unzip -q ch9_training_images.zip -d imgs/training && unzip -q ch9_training_localization_transcription_gt.zip -d annotations/training && unzip -q ch9_validation_images.zip -d imgs/val && unzip -q ch9_validation_localization_transcription_gt.zip -d annotations/val

  # Remove zips
  rm ch9_training_images.zip && rm ch9_training_localization_transcription_gt.zip && rm ch9_validation_images.zip && rm ch9_validation_localization_transcription_gt.zip
  ```

# Remove zips
  ```

- Step2: Generate `instances_training.json` and `instances_val.json` with following command:

  ```bash
  python tools/dataset_converters/textdet/detext_converter.py PATH/TO/detext --nproc 4
  ```

  ```text
  │── detext
  │   ├── imgs
  │   ├── instances_test.json
  ```

## Lecture Video DB

- Step1: Download [IIIT-CVid.zip](http://cdn.iiit.ac.in/cdn/preon.iiit.ac.in/~kartik/IIIT-CVid.zip) to `lv/`.

  ```bash
  mkdir lv && cd lv

  # Download LV dataset
  wget http://cdn.iiit.ac.in/cdn/preon.iiit.ac.in/~kartik/IIIT-CVid.zip
  unzip -q IIIT-CVid.zip

  mv IIIT-CVid/Frames imgs

  rm IIIT-CVid.zip
  ```

mkdir lv && cd lv

  ```

- Step2: Generate `instances_training.json`, `instances_val.json`, and `instances_test.json` with following command:

  ```bash
  python tools/dataset_converters/textdet/lv_converter.py PATH/TO/lv --nproc 4
  ```

  ```text
  │── lv
  │   ├── imgs
  ```

## LSVT

- Step1: Download [train_full_images_0.tar.gz](https://dataset-bj.cdn.bcebos.com/lsvt/train_full_images_0.tar.gz), [train_full_images_1.tar.gz](https://dataset-bj.cdn.bcebos.com/lsvt/train_full_images_1.tar.gz), and [train_full_labels.json](https://dataset-bj.cdn.bcebos.com/lsvt/train_full_labels.json) to `lsvt/`.

  ```bash
  mkdir lsvt && cd lsvt

  # Download LSVT dataset

```bash

  wget https://dataset-bj.cdn.bcebos.com/lsvt/train_full_images_0.tar.gz
  wget https://dataset-bj.cdn.bcebos.com/lsvt/train_full_images_1.tar.gz
  wget https://dataset-bj.cdn.bcebos.com/lsvt/train_full_labels.json

  mkdir annotations
  tar -xf train_full_images_0.tar.gz && tar -xf train_full_images_1.tar.gz
  mv train_full_labels.json annotations/ && mv train_full_images_1/*.jpg train_full_images_0/
  mv train_full_images_0 imgs

  rm train_full_images_0.tar.gz && rm train_full_images_1.tar.gz && rm -rf train_full_images_1
  ```

- Step2: Generate `instances_training.json` and `instances_val.json` (optional) with the following command:

  ```bash
  # Annotations of LSVT test split is not publicly available, split a validation
  # set by adding --val-ratio 0.2
  python tools/dataset_converters/textdet/lsvt_converter.py PATH/TO/lsvt
  ```

# set by adding --val-ratio 0.2
  ```

  ```text
  |── lsvt
  │   ├── imgs
  │   └── instances_val.json (optional)
  ```

## IMGUR

- Step1: Run `download_imgur5k.py` to download images. You can merge [PR#5](https://github.com/facebookresearch/IMGUR5K-Handwriting-Dataset/pull/5) in your local repository to enable a **much faster** parallel execution of image download.

  ```bash
  mkdir imgur && cd imgur

  git clone https://github.com/facebookresearch/IMGUR5K-Handwriting-Dataset.git

  # Download images from imgur.com. This may take SEVERAL HOURS!
  python ./IMGUR5K-Handwriting-Dataset/download_imgur5k.py --dataset_info_dir ./IMGUR5K-Handwriting-Dataset/dataset_info/ --output_dir ./imgs

  mv ./IMGUR5K-Handwriting-Dataset/dataset_info/*.json annotations

# For annotations

  rm -rf IMGUR5K-Handwriting-Dataset
  ```

- Step2: Generate `instances_train.json`, `instance_val.json` and `instances_test.json` with the following command:

  ```bash
  python tools/dataset_converters/textdet/imgur_converter.py PATH/TO/imgur
  ```

  ```text
  │── imgur
  │   ├── annotations
  │   ├── imgs
  │   ├── instances_training.json
  │   └── instances_val.json
  ```

## KAIST

- Step1: Complete download [KAIST_all.zip](http://www.iapr-tc11.org/mediawiki/index.php/KAIST_Scene_Text_Database) to `kaist/`.

  ```bash
  mkdir kaist && cd kaist

  # Download KAIST dataset
  wget http://www.iapr-tc11.org/dataset/KAIST_SceneText/KAIST_all.zip
  unzip -q KAIST_all.zip

  rm KAIST_all.zip
  ```

- Step2: Extract zips:

  ```bash

# Download KAIST dataset

  ```

  ```bash
  python tools/dataset_converters/common/extract_kaist.py PATH/TO/kaist
  ```

- Step3: Generate `instances_training.json` and `instances_val.json` (optional) with following command:

  ```bash
  # Since KAIST does not provide an official split, you can split the dataset by adding --val-ratio 0.2
  python tools/dataset_converters/textdet/kaist_converter.py PATH/TO/kaist --nproc 4
  ```

  ```text
  │── kaist
  │   ├── imgs
  │   └── instances_val.json (optional)
  ```

## MTWI

- Step1: Download `mtwi_2018_train.zip` from [homepage](https://tianchi.aliyun.com/competition/entrance/231685/information?lang=en-us).

  ```bash
  mkdir mtwi && cd mtwi

```

## MTWI

  ```bash

  unzip -q mtwi_2018_train.zip
  mv image_train imgs && mv txt_train annotations

  rm mtwi_2018_train.zip
  ```

- Step2: Generate `instances_training.json` and `instance_val.json` (optional) with the following command:

  ```bash
  # Annotations of MTWI test split is not publicly available, split a validation
  python tools/dataset_converters/textdet/mtwi_converter.py PATH/TO/mtwi --nproc 4
  ```

  ```text
  │── mtwi
  │   ├── imgs
  ```

## ReCTS

```text
  │── mtwi
  │   ├── imgs
  ```

## ReCTS

- Step1: Download [ReCTS.zip](https://datasets.cvc.uab.es/rrc/ReCTS.zip) to `rects/` from the [homepage](https://rrc.cvc.uab.es/?ch=12&com=downloads).

  ```bash
  mkdir rects && cd rects

  # Download ReCTS dataset
  # You can also find Google Drive link on the dataset homepage
  wget https://datasets.cvc.uab.es/rrc/ReCTS.zip --no-check-certificate
  unzip -q ReCTS.zip

  mv img imgs && mv gt_unicode annotations

  rm ReCTS.zip && rm -rf gt
  ```

- Step2: Generate `instances_training.json` and `instances_val.json` (optional) with following command:

  ```bash
  # Annotations of ReCTS test split is not publicly available, split a validation
  python tools/dataset_converters/textdet/rects_converter.py PATH/TO/rects --nproc 4 --val-ratio 0.2
  ```

  ```

  ```text
  │── rects
  │   ├── imgs
  │   ├── instances_val.json (optional)
  ```

## ILST

- Step1: Download `IIIT-ILST` from [onedrive](https://iiitaphyd-my.sharepoint.com/:f:/g/personal/minesh_mathew_research_iiit_ac_in/EtLvCozBgaBIoqglF4M-lHABMgNcCDW9rJYKKWpeSQEElQ?e=zToXZP)

- Step2: Run the following commands

  ```bash
  unzip -q IIIT-ILST.zip && rm IIIT-ILST.zip
  cd IIIT-ILST

  # rename files
  cd Devanagari && for i in `ls`; do mv -f $i `echo "devanagari_"$i`; done && cd ..
  cd Malayalam && for i in `ls`; do mv -f $i `echo "malayalam_"$i`; done && cd ..
  cd Telugu && for i in `ls`; do mv -f $i `echo "telugu_"$i`; done && cd ..

  # transfer image path

cd Telugu && for i in `ls`; do mv -f $i `echo "telugu_"$i`; done && cd ..

  mv Malayalam/{*jpg,*jpeg} imgs/ && mv Malayalam/*xml annotations/
  mv Devanagari/*jpg imgs/ && mv Devanagari/*xml annotations/
  mv Telugu/*jpeg imgs/ && mv Telugu/*xml annotations/

  # remove unnecessary files
  rm -rf Devanagari && rm -rf Malayalam && rm -rf Telugu && rm -rf README.txt
  ```

- Step3: Generate `instances_training.json` and `instances_val.json` (optional). Since the original dataset doesn't have a validation set, you may specify `--val-ratio` to split the dataset. E.g., if val-ratio is 0.2, then 20% of the data are left out as the validation set in this example.

  ```bash
  python tools/dataset_converters/textdet/ilst_converter.py    PATH/TO/IIIT-ILST --nproc 4
  ```

  ```text
  │── IIIT-ILST
  │   ├── imgs

```

  ```text
  │   ├── imgs
  │   ├── instances_val.json (optional)
  ```

## VinText

- Step1: Download [vintext.zip](https://drive.google.com/drive/my-drive) to `vintext`

  ```bash
  mkdir vintext && cd vintext

  # Download dataset from google drive
  wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1UUQhNvzgpZy7zXBFQp0Qox-BBjunZ0ml' -O- │ sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1UUQhNvzgpZy7zXBFQp0Qox-BBjunZ0ml" -O vintext.zip && rm -rf /tmp/cookies.txt

  unzip -q vintext.zip && rm vintext.zip

# Extract images and annotations
  mv vietnamese/labels ./ && mv vietnamese/test_image ./ && mv vietnamese/train_images ./ && mv vietnamese/unseen_test_images ./
  rm -rf vietnamese

  # Rename files
  mv labels annotations && mv test_image test && mv train_images  training && mv unseen_test_images  unseen_test
  mkdir imgs
  mv training imgs/ && mv test imgs/ && mv unseen_test imgs/
  ```

- Step2: Generate `instances_training.json`, `instances_test.json` and `instances_unseen_test.json`

  ```bash
  python tools/dataset_converters/textdet/vintext_converter.py PATH/TO/vintext --nproc 4
  ```

  ```text
  │── vintext
  │   ├── imgs
  │   ├── instances_unseen_test.json
  ```

## BID

```text
  │   ├── imgs
  ```

## BID

- Step1: Download [BID Dataset.zip](https://drive.google.com/file/d/1Oi88TRcpdjZmJ79WDLb9qFlBNG8q2De6/view)

- Step2: Run the following commands to preprocess the dataset

  ```bash
  # Rename
  mv BID\ Dataset.zip BID_Dataset.zip

  # Unzip and Rename
  unzip -q BID_Dataset.zip && rm BID_Dataset.zip
  mv BID\ Dataset BID

  # The BID dataset has a problem of permission, and you may
  # add permission for this file
  chmod -R 777 BID
  cd BID

  # For images and annotations
  mv CNH_Aberta/*in.jpg imgs && mv CNH_Aberta/*txt annotations && rm -rf CNH_Aberta
  mv CNH_Frente/*in.jpg imgs && mv CNH_Frente/*txt annotations && rm -rf CNH_Frente
  mv CNH_Verso/*in.jpg imgs && mv CNH_Verso/*txt annotations && rm -rf CNH_Verso

mv CNH_Frente/*in.jpg imgs && mv CNH_Frente/*txt annotations && rm -rf CNH_Frente
  mv CPF_Frente/*in.jpg imgs && mv CPF_Frente/*txt annotations && rm -rf CPF_Frente
  mv CPF_Verso/*in.jpg imgs && mv CPF_Verso/*txt annotations && rm -rf CPF_Verso
  mv RG_Aberto/*in.jpg imgs && mv RG_Aberto/*txt annotations && rm -rf RG_Aberto
  mv RG_Frente/*in.jpg imgs && mv RG_Frente/*txt annotations && rm -rf RG_Frente
  mv RG_Verso/*in.jpg imgs && mv RG_Verso/*txt annotations && rm -rf RG_Verso

  # Remove unnecessary files
  rm -rf desktop.ini
  ```

- Step3: - Step3: Generate `instances_training.json` and `instances_val.json` (optional). Since the original dataset doesn't have a validation set, you may specify `--val-ratio` to split the dataset. E.g., if val-ratio is 0.2, then 20% of the data are left out as the validation set in this example.

  ```bash

```bash
  python tools/dataset_converters/textdet/bid_converter.py PATH/TO/BID --nproc 4
  ```

  ```text
  │── BID
  │   ├── imgs
  ```

## RCTW

- Step1: Download `train_images.zip.001`, `train_images.zip.002`, and `train_gts.zip` from the [homepage](https://rctw.vlrlab.net/dataset.html), extract the zips to `rctw/imgs` and `rctw/annotations`, respectively.

- Step2: Generate `instances_training.json` and `instances_val.json` (optional). Since the test annotations are not publicly available, you may specify `--val-ratio` to split the dataset. E.g., if val-ratio is 0.2, then 20% of the data are left out as the validation set in this example.

  ```bash
  # Annotations of RCTW test split is not publicly available, split a validation set by adding --val-ratio 0.2

```bash
  python tools/dataset_converters/textdet/rctw_converter.py PATH/TO/rctw --nproc 4
  ```

  ```text
  │── rctw
  │   ├── imgs
  ```

## HierText

- Step1 (optional): Install [AWS CLI](https://mmocr.readthedocs.io/en/latest/datasets/det.html#install-aws-cli-optional).

- Step2: Clone [HierText](https://github.com/google-research-datasets/hiertext) repo to get annotations

  ```bash
  mkdir HierText
  git clone https://github.com/google-research-datasets/hiertext.git
  ```

- Step3: Download `train.tgz`, `validation.tgz` from aws

  ```bash
  aws s3 --no-sign-request cp s3://open-images-dataset/ocr/train.tgz .
  aws s3 --no-sign-request cp s3://open-images-dataset/ocr/validation.tgz .
  ```

```bash
  ```

- Step4: Process raw data

  ```bash
  # process annotations
  mv hiertext/gt ./
  rm -rf hiertext
  mv gt annotations
  gzip -d annotations/train.jsonl.gz
  gzip -d annotations/validation.jsonl.gz
  # process images
  mkdir imgs
  mv train.tgz imgs/
  mv validation.tgz imgs/
  tar -xzvf imgs/train.tgz
  tar -xzvf imgs/validation.tgz
  ```

- Step5: Generate `instances_training.json` and `instance_val.json`. HierText includes different levels of annotation, from paragraph, line, to word. Check the original [paper](https://arxiv.org/pdf/2203.15143.pdf) for details. E.g. set `--level paragraph` to get paragraph-level annotation. Set `--level line` to get line-level annotation. set `--level word` to get word-level annotation.

  ```bash
  # Collect word annotation from HierText  --level word

```bash
  python tools/dataset_converters/textdet/hiertext_converter.py PATH/TO/HierText --level word --nproc 4
  ```

  ```text
  │── HierText
  │   ├── imgs
  ```

## ArT

- Step1: Download `train_images.tar.gz`, and `train_labels.json` from the [homepage](https://rrc.cvc.uab.es/?ch=14&com=downloads) to `art/`

  ```bash
  mkdir art && cd art

  # Download ArT dataset
  wget https://dataset-bj.cdn.bcebos.com/art/train_images.tar.gz --no-check-certificate
  wget https://dataset-bj.cdn.bcebos.com/art/train_labels.json --no-check-certificate

  # Extract
  tar -xf train_images.tar.gz
  mv train_images imgs
  mv train_labels.json annotations/

  rm train_images.tar.gz
  ```

# Extract

  ```

  ```bash
  # Annotations of ArT test split is not publicly available, split a validation set by adding --val-ratio 0.2
  python tools/data/textdet/art_converter.py PATH/TO/art --nproc 4
  ```

  ```text
  │── art
  │   ├── imgs
  ```

# ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votes

> [ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votes](https://arxiv.org/abs/2001.10692)

3D object detection has seen quick progress thanks to advances in deep learning on point clouds. A few recent works have even shown state-of-the-art performance with just point clouds input (e.g. VOTENET). However, point cloud data have inherent limitations. They are sparse, lack color information and often suffer from sensor noise. Images, on the other hand, have high resolution and rich texture. Thus they can complement the 3D geometry provided by point clouds. Yet how to effectively use image information to assist point cloud based detection is still an open question. In this work, we build on top of VOTENET and propose a 3D detection architecture called IMVOTENET specialized for RGB-D scenes. IMVOTENET is based on fusing 2D votes in images and 3D votes in point clouds. Compared to prior work on multi-modal detection, we explicitly extract both geometric and semantic features from the 2D images. We leverage camera parameters to lift these features to 3D. To improve the synergy of 2D-3D feature fusion, we also propose a multi-tower training scheme. We validate our model on the challenging SUN RGB-D dataset, advancing state-of-the-art results by 5.7 mAP. We also provide rich ablation studies to analyze the contribution of each design choice.

<img src="https://user-images.githubusercontent.com/36950400/143869878-a2ae7f43-55c3-4b95-af09-8f97dfd975f4.png" width="800"/>
</div>

We implement ImVoteNet and provide the result and checkpoints on SUNRGBD.

### SUNRGBD-2D (Stage 1, image branch pre-train)

|                             Backbone                             | Lr schd | Mem (GB) | Inf time (fps) | AP@0.25 | AP@0.5 |                                                                                                                                                                                                              Download                                                                                                                                                                                                              |

| [PointNet++](./imvotenet_faster-rcnn-r50_fpn_4xb2_sunrgbd-3d.py) |         |   2.1    |                |         | 62.70  | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/imvotenet/imvotenet_faster_rcnn_r50_fpn_2x4_sunrgbd-3d-10class/imvotenet_faster_rcnn_r50_fpn_2x4_sunrgbd-3d-10class_20210819_225618-62eba6ce.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/imvotenet/imvotenet_faster_rcnn_r50_fpn_2x4_sunrgbd-3d-10class/imvotenet_faster_rcnn_r50_fpn_2x4_sunrgbd-3d-10class_20210819_225618.json) |

### SUNRGBD-3D (Stage 2)

|                       Backbone                       | Lr schd | Mem (GB) | Inf time (fps) | AP@0.25 | AP@0.5 |                                                                                                                                                                                        Download                                                                                                                                                                                        |

| :--------------------------------------------------: | :-----: | :------: | :------------: | :-----: | :----: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| [PointNet++](./imvotenet_stage2_8xb16_sunrgbd-3d.py) |   3x    |   9.4    |                |  64.48  |        | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/imvotenet/imvotenet_stage2_16x8_sunrgbd-3d-10class/imvotenet_stage2_16x8_sunrgbd-3d-10class_20210819_192851-1bcd1b97.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/imvotenet/imvotenet_stage2_16x8_sunrgbd-3d-10class/imvotenet_stage2_16x8_sunrgbd-3d-10class_20210819_192851.log.json) |

```latex
@inproceedings{qi2020imvotenet,
  title={Imvotenet: Boosting 3D object detection in point clouds with image votes},
  author={Qi, Charles R and Chen, Xinlei and Litany, Or and Guibas, Leonidas J},
  pages={4404--4413},
}
```

# 教程 4：训练与测试（待更新）

## 使用预训练模型进行推理

我们提供用于在完整数据集上进行预训练模型评估和特定任务图像演示的测试脚本。

### 测试一个预训练模型

MMagic 使用 `MMDistributedDataParallel` 实现 **分布式**测试。

#### 在单/多个 GPU 上进行测试

您可以使用以下命令在单/多个 GPU 上测试预训练模型。

```shell
# 单 GPU 测试
python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [--save-path ${IMAGE_SAVE_PATH}]

# 多 GPU 测试
./tools/dist_test.sh ${CONFIG_FILE} ${CHECKPOINT_FILE} ${GPU_NUM} [--out ${RESULT_FILE}] [--save-path ${IMAGE_SAVE_PATH}]
```

例如

```shell
# 单 GPU 测试
python tools/test.py configs/example_config.py work_dirs/example_exp/example_model_20200202.pth --out work_dirs/example_exp/results.pkl

# 多 GPU 测试
./tools/dist_test.sh configs/example_config.py work_dirs/example_exp/example_model_20200202.pth --save-path work_dirs/example_exp/results/
```

#### 在 slurm 上测试

如果您在使用 [slurm](https://slurm.schedmd.com/) 管理的集群上运行 MMagic，则可以使用脚本 `slurm_test.sh`。（此脚本也支持单机测试。）

```shell
[GPUS=${GPUS}] ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} ${CONFIG_FILE} ${CHECKPOINT_FILE}

```shell
```

以下是使用 8 个 GPU 在作业名称为 `test` 的 `dev` 分区上测试示例模型的例子。

```shell
GPUS=8 ./tools/slurm_test.sh dev test configs/example_config.py work_dirs/example_exp/example_model_20200202.pth
```

您可以查看 [slurm_test.sh](https://github.com/open-mmlab/mmagic/blob/main/tools/slurm_test.sh) 以获取完整的参数和环境变量。

#### 可选参数

- `--out`: 以 pickle 格式指定输出结果的文件名。 如果没有给出，结果将不会保存到文件中。
- `--save-path`: 指定存储编辑图像的路径。 如果没有给出，图像将不会被保存。
- `--seed`: 测试期间的随机种子。 此参数用于固定某些任务中的结果，例如*修复*。
- `--deterministic`: 与 `--seed` 相关，此参数决定是否为 CUDNN 后端设置确定性的选项。如果指定该参数，会将 `torch.backends.cudnn.deterministic` 设置为 `True`，将 `torch.backends.cudnn.benchmark` 设置为 `False`。
- `--cfg-options`:  如果指明，这里的键值对将会被合并到配置文件中。

注：目前，我们不使用像 [MMDetection](https://github.com/open-mmlab/mmdetection) 那样的 `--eval` 参数来指定评估指标。 评估指标在配置文件中给出（参见 [config.md](../user_guides/config.md)）。

## 训练一个模型

## 训练一个模型

所有输出（日志文件和模型权重文件）都将保存到工作目录中，工作目录由配置文件中的 `work_dir` 指定。

默认情况下，我们在多次迭代后评估验证集上的模型，您可以通过在训练配置中添加 `interval` 参数来更改评估间隔。

```python
evaluation = dict(interval=1e4, by_epoch=False)  # 每一万次迭代进行一次评估。
```

### 在单/多个 GPU 上训练

```shell
./tools/dist_train.sh ${CONFIG_FILE} ${GPU_NUM} [optional arguments]
```

可选参数是：

- `--no-validate` (**不建议**): 默认情况下，代码库将在训练期间每 k 次迭代执行一次评估。若要禁用此行为，请使用 `--no-validate`。
- `--work-dir ${WORK_DIR}`: 覆盖配置文件中指定的工作目录。
- `--resume-from ${CHECKPOINT_FILE}`: 从已有的模型权重文件恢复。

`resume-from` 和 `load-from` 之间的区别：
`resume-from` 加载模型权重和优化器状态，迭代也从指定的检查点继承。 它通常用于恢复意外中断的训练过程。
`load-from` 只加载模型权重，训练迭代从 0 开始，通常用于微调。

#### 使用多节点训练

如果您有多个计算节点，而且他们可以通过 IP 互相访问，可以使用以下命令启动分布式训练：

在第一个节点：

```shell
NNODES=2 NODE_RANK=0 PORT=$MASTER_PORT MASTER_ADDR=$MASTER_ADDR tools/dist_train.sh $CONFIG $GPUS
```

在第二个节点：

```shell

在第一个节点：

```shell
```

在第二个节点：

```shell
NNODES=2 NODE_RANK=1 PORT=$MASTER_PORT MASTER_ADDR=$MASTER_ADDR tools/dist_train.sh $CONFIG $GPUS
```

为提高网络通信速度，推荐使用高速网络设备，如 Infiniband 等。
更多信息可参照[PyTorch 文档](https://pytorch.org/docs/1.11/distributed.html#launch-utility).

### 在 slurm 上训练

如果您在使用 [slurm](https://slurm.schedmd.com/) 管理的集群上运行 MMagic，则可以使用脚本 `slurm_train.sh`。（此脚本也支持单机训练。）

```shell
[GPUS=${GPUS}] ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} ${CONFIG_FILE} ${WORK_DIR}
```

以下是使用 8 个 GPU 在 `dev` 分区上训练*修复*模型的示例。

```shell
GPUS=8 ./tools/slurm_train.sh dev configs/inpainting/gl_places.py /nfs/xxxx/gl_places_256
```

您可以查看 [slurm_train.sh](https://github.com/open-mmlab/mmagic/blob/main/tools/slurm_train.sh) 以获取完整的参数和环境变量。

### 在一台机器上启动多个作业

```

如果您在一台机器上启动多个作业，例如，在具有 8 个 GPU 的机器上进行 2 个 4-GPU 训练的作业，您需要为每个作业指定不同的端口（默认为 29500）以避免通信冲突。

```shell
CUDA_VISIBLE_DEVICES=0,1,2,3 PORT=29500 ./tools/dist_train.sh ${CONFIG_FILE} 4
CUDA_VISIBLE_DEVICES=4,5,6,7 PORT=29501 ./tools/dist_train.sh ${CONFIG_FILE} 4
```

如果您使用 Slurm 启动训练作业，则需要修改配置文件（通常是配置文件的倒数第 6 行）以设置不同的通信端口。

在 `config1.py` 中,

```python
dist_params = dict(backend='nccl', port=29500)
```

在 `config2.py` 中,

```python
dist_params = dict(backend='nccl', port=29501)
```

然后您可以使用 `config1.py` 和 `config2.py` 启动两个作业。

```shell
CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config1.py ${WORK_DIR}
CUDA_VISIBLE_DEVICES=4,5,6,7 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config2.py ${WORK_DIR}
```

# MAE

> [Masked Autoencoders Are Scalable Vision Learners](https://arxiv.org/abs/2111.06377)

This paper shows that masked autoencoders (MAE) are
scalable self-supervised learners for computer vision. Our
MAE approach is simple: we mask random patches of the
input image and reconstruct the missing pixels. It is based
on two core designs. First, we develop an asymmetric
encoder-decoder architecture, with an encoder that operates only on the
visible subset of patches (without mask tokens), along with a lightweight
decoder that reconstructs the original image from the latent representation
and mask tokens. Second, we find that masking a high proportion
of the input image, e.g., 75%, yields a nontrivial and
meaningful self-supervisory task. Coupling these two designs enables us to
train large models efficiently and effectively: we accelerate
training (by 3× or more) and improve accuracy. Our scalable approach allows

for learning high-capacity models that generalize well: e.g., a vanilla
ViT-Huge model achieves the best accuracy (87.8%) among
methods that use only ImageNet-1K data. Transfer performance in downstream tasks outperforms supervised pretraining and shows promising scaling behavior.

<img src="https://user-images.githubusercontent.com/30762564/150733959-2959852a-c7bd-4d3f-911f-3e8d8839fe67.png" width="80%"/>
</div>

```python

predict = inference_model('vit-base-p16_mae-300e-pre_8xb128-coslr-100e_in1k', 'demo/bird.JPEG')
```

```python

```

```python

model = get_model('mae_vit-base-p16_8xb512-amp-coslr-300e_in1k', pretrained=True)
```

Train:

```shell
python tools/train.py configs/mae/mae_vit-base-p16_8xb512-amp-coslr-300e_in1k.py
```

Test:

```shell
python tools/test.py configs/mae/benchmarks/vit-base-p16_8xb128-coslr-100e_in1k.py None
```

```

Test:

```shell
```

| Model                                           | Params (M) | Flops (G) |                           Config                           |                                   Download                                   |
| :---------------------------------------------- | :--------: | :-------: | :--------------------------------------------------------: | :--------------------------------------------------------------------------: |

| `mae_vit-base-p16_8xb512-amp-coslr-300e_in1k`   |   111.91   |   17.58   |  [config](mae_vit-base-p16_8xb512-amp-coslr-300e_in1k.py)  | [model](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-300e_in1k/mae_vit-base-p16_8xb512-coslr-300e-fp16_in1k_20220829-c2cf66ba.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-300e_in1k/mae_vit-base-p16_8xb512-coslr-300e-fp16_in1k_20220829-c2cf66ba.json) |
| `mae_vit-base-p16_8xb512-amp-coslr-400e_in1k`   |   111.91   |   17.58   |  [config](mae_vit-base-p16_8xb512-amp-coslr-400e_in1k.py)  | [model](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-400e_in1k/mae_vit-base-p16_8xb512-coslr-400e-fp16_in1k_20220825-bc79e40b.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-400e_in1k/mae_vit-base-p16_8xb512-coslr-400e-fp16_in1k_20220825-bc79e40b.json) |

| `mae_vit-base-p16_8xb512-amp-coslr-800e_in1k`   |   111.91   |   17.58   |  [config](mae_vit-base-p16_8xb512-amp-coslr-800e_in1k.py)  | [model](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-800e_in1k/mae_vit-base-p16_8xb512-coslr-800e-fp16_in1k_20220825-5d81fbc4.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-800e_in1k/mae_vit-base-p16_8xb512-coslr-800e-fp16_in1k_20220825-5d81fbc4.json) |
| `mae_vit-base-p16_8xb512-amp-coslr-1600e_in1k`  |   111.91   |   17.58   | [config](mae_vit-base-p16_8xb512-amp-coslr-1600e_in1k.py)  | [model](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-1600e_in1k/mae_vit-base-p16_8xb512-fp16-coslr-1600e_in1k_20220825-f7569ca2.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-1600e_in1k/mae_vit-base-p16_8xb512-fp16-coslr-1600e_in1k_20220825-f7569ca2.json) |

| `mae_vit-large-p16_8xb512-amp-coslr-400e_in1k`  |   329.54   |   61.60   | [config](mae_vit-large-p16_8xb512-amp-coslr-400e_in1k.py)  | [model](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-large-p16_8xb512-fp16-coslr-400e_in1k/mae_vit-large-p16_8xb512-fp16-coslr-400e_in1k_20220825-b11d0425.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-large-p16_8xb512-fp16-coslr-400e_in1k/mae_vit-large-p16_8xb512-fp16-coslr-400e_in1k_20220825-b11d0425.json) |
| `mae_vit-large-p16_8xb512-amp-coslr-800e_in1k`  |   329.54   |   61.60   | [config](mae_vit-large-p16_8xb512-amp-coslr-800e_in1k.py)  | [model](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-large-p16_8xb512-fp16-coslr-800e_in1k/mae_vit-large-p16_8xb512-fp16-coslr-800e_in1k_20220825-df72726a.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-large-p16_8xb512-fp16-coslr-800e_in1k/mae_vit-large-p16_8xb512-fp16-coslr-800e_in1k_20220825-df72726a.json) |

| `mae_vit-large-p16_8xb512-amp-coslr-1600e_in1k` |   329.54   |   61.60   | [config](mae_vit-large-p16_8xb512-amp-coslr-1600e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-large-p16_8xb512-fp16-coslr-1600e_in1k/mae_vit-large-p16_8xb512-fp16-coslr-1600e_in1k_20220825-cc7e98c9.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-large-p16_8xb512-fp16-coslr-1600e_in1k/mae_vit-large-p16_8xb512-fp16-coslr-1600e_in1k_20220825-cc7e98c9.json) |
| `mae_vit-huge-p16_8xb512-amp-coslr-1600e_in1k`  |   657.07   |  167.40   | [config](mae_vit-huge-p14_8xb512-amp-coslr-1600e_in1k.py)  | [model](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-huge-p16_8xb512-fp16-coslr-1600e_in1k/mae_vit-huge-p16_8xb512-fp16-coslr-1600e_in1k_20220916-ff848775.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-huge-p16_8xb512-fp16-coslr-1600e_in1k/mae_vit-huge-p16_8xb512-fp16-coslr-1600e_in1k_20220916-ff848775.json) |

| `vit-base-p16_mae-300e-pre_8xb128-coslr-100e_in1k` | [MAE 300-Epochs](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-300e_in1k/mae_vit-base-p16_8xb512-coslr-300e-fp16_in1k_20220829-c2cf66ba.pth) |   86.57    |   17.58   |   83.10   | [config](benchmarks/vit-base-p16_8xb128-coslr-100e_in1k.py) |                      N/A                      |

| `vit-base-p16_mae-400e-pre_8xb128-coslr-100e_in1k` | [MAE 400-Epochs](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-400e_in1k/mae_vit-base-p16_8xb512-coslr-400e-fp16_in1k_20220825-bc79e40b.pth) |   86.57    |   17.58   |   83.30   | [config](benchmarks/vit-base-p16_8xb128-coslr-100e_in1k.py) |                      N/A                      |
| `vit-base-p16_mae-800e-pre_8xb128-coslr-100e_in1k` | [MAE 800-Epochs](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-800e_in1k/mae_vit-base-p16_8xb512-coslr-800e-fp16_in1k_20220825-5d81fbc4.pth) |   86.57    |   17.58   |   83.30   | [config](benchmarks/vit-base-p16_8xb128-coslr-100e_in1k.py) |                      N/A                      |

| `vit-base-p16_mae-1600e-pre_8xb128-coslr-100e_in1k` | [MAE 1600-Epochs](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-1600e_in1k/mae_vit-base-p16_8xb512-fp16-coslr-1600e_in1k_20220825-f7569ca2.pth) |   86.57    |   17.58   |   83.50   | [config](benchmarks/vit-base-p16_8xb128-coslr-100e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-1600e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k_20220825-cf70aa21.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-1600e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k_20220825-cf70aa21.json) |

| `vit-base-p16_mae-300e-pre_8xb2048-linear-coslr-90e_in1k` | [MAE 300-Epochs](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-300e_in1k/mae_vit-base-p16_8xb512-coslr-300e-fp16_in1k_20220829-c2cf66ba.pth) |   86.57    |   17.58   |   60.80   | [config](benchmarks/vit-base-p16_8xb2048-linear-coslr-90e_in1k.py) |                      N/A                      |
| `vit-base-p16_mae-400e-pre_8xb2048-linear-coslr-90e_in1k` | [MAE 400-Epochs](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-400e_in1k/mae_vit-base-p16_8xb512-coslr-400e-fp16_in1k_20220825-bc79e40b.pth) |   86.57    |   17.58   |   62.50   | [config](benchmarks/vit-base-p16_8xb2048-linear-coslr-90e_in1k.py) |                      N/A                      |

| `vit-base-p16_mae-800e-pre_8xb2048-linear-coslr-90e_in1k` | [MAE 800-Epochs](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-800e_in1k/mae_vit-base-p16_8xb512-coslr-800e-fp16_in1k_20220825-5d81fbc4.pth) |   86.57    |   17.58   |   65.10   | [config](benchmarks/vit-base-p16_8xb2048-linear-coslr-90e_in1k.py) |                      N/A                      |
| `vit-base-p16_mae-1600e-pre_8xb2048-linear-coslr-90e_in1k` | [MAE 1600-Epochs](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-1600e_in1k/mae_vit-base-p16_8xb512-fp16-coslr-1600e_in1k_20220825-f7569ca2.pth) |   86.57    |   17.58   |   67.10   | [config](benchmarks/vit-base-p16_8xb2048-linear-coslr-90e_in1k.py) |                      N/A                      |

| `vit-large-p16_mae-400e-pre_8xb128-coslr-50e_in1k` | [MAE 400-Epochs](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-large-p16_8xb512-fp16-coslr-400e_in1k/mae_vit-large-p16_8xb512-fp16-coslr-400e_in1k_20220825-b11d0425.pth) |   304.32   |   61.60   |   85.20   | [config](benchmarks/vit-large-p16_8xb128-coslr-50e_in1k.py) |                      N/A                      |
| `vit-large-p16_mae-800e-pre_8xb128-coslr-50e_in1k` | [MAE 800-Epochs](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-large-p16_8xb512-fp16-coslr-800e_in1k/mae_vit-large-p16_8xb512-fp16-coslr-800e_in1k_20220825-df72726a.pth) |   304.32   |   61.60   |   85.40   | [config](benchmarks/vit-large-p16_8xb128-coslr-50e_in1k.py) |                      N/A                      |

| `vit-large-p16_mae-1600e-pre_8xb128-coslr-50e_in1k` | [MAE 1600-Epochs](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-large-p16_8xb512-fp16-coslr-1600e_in1k/mae_vit-large-p16_8xb512-fp16-coslr-1600e_in1k_20220825-cc7e98c9.pth) |   304.32   |   61.60   |   85.70   | [config](benchmarks/vit-large-p16_8xb128-coslr-50e_in1k.py) |                      N/A                      |
| `vit-large-p16_mae-400e-pre_8xb2048-linear-coslr-90e_in1k` | [MAE 400-Epochs](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-large-p16_8xb512-fp16-coslr-400e_in1k/mae_vit-large-p16_8xb512-fp16-coslr-400e_in1k_20220825-b11d0425.pth) |   304.33   |   61.60   |   70.70   | [config](benchmarks/vit-large-p16_8xb2048-linear-coslr-90e_in1k.py) |                      N/A                      |

| `vit-large-p16_mae-800e-pre_8xb2048-linear-coslr-90e_in1k` | [MAE 800-Epochs](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-large-p16_8xb512-fp16-coslr-800e_in1k/mae_vit-large-p16_8xb512-fp16-coslr-800e_in1k_20220825-df72726a.pth) |   304.33   |   61.60   |   73.70   | [config](benchmarks/vit-large-p16_8xb2048-linear-coslr-90e_in1k.py) |                      N/A                      |
| `vit-large-p16_mae-1600e-pre_8xb2048-linear-coslr-90e_in1k` | [MAE 1600-Epochs](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-large-p16_8xb512-fp16-coslr-1600e_in1k/mae_vit-large-p16_8xb512-fp16-coslr-1600e_in1k_20220825-cc7e98c9.pth) |   304.33   |   61.60   |   75.50   | [config](benchmarks/vit-large-p16_8xb2048-linear-coslr-90e_in1k.py) |                      N/A                      |

| `vit-huge-p14_mae-1600e-pre_8xb128-coslr-50e_in1k` | [MAE 1600-Epochs](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-huge-p16_8xb512-fp16-coslr-1600e_in1k/mae_vit-huge-p16_8xb512-fp16-coslr-1600e_in1k_20220916-ff848775.pth) |   632.04   |  167.40   |   86.90   | [config](benchmarks/vit-huge-p14_8xb128-coslr-50e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-huge-p16_8xb512-fp16-coslr-1600e_in1k/vit-huge-p16_ft-8xb128-coslr-50e_in1k/vit-huge-p16_ft-8xb128-coslr-50e_in1k_20220916-0bfc9bfd.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-huge-p16_8xb512-fp16-coslr-1600e_in1k/vit-huge-p16_ft-8xb128-coslr-50e_in1k/vit-huge-p16_ft-8xb128-coslr-50e_in1k_20220916-0bfc9bfd.json) |

| `vit-huge-p14_mae-1600e-pre_32xb8-coslr-50e_in1k-448px` | [MAE 1600-Epochs](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-huge-p16_8xb512-fp16-coslr-1600e_in1k/mae_vit-huge-p16_8xb512-fp16-coslr-1600e_in1k_20220916-ff848775.pth) |   633.03   |  732.13   |   87.30   | [config](benchmarks/vit-huge-p14_32xb8-coslr-50e_in1k-448px.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-huge-p16_8xb512-fp16-coslr-1600e_in1k/vit-huge-p16_ft-32xb8-coslr-50e_in1k-448/vit-huge-p16_ft-32xb8-coslr-50e_in1k-448_20220916-95b6a0ce.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-huge-p16_8xb512-fp16-coslr-1600e_in1k/vit-huge-p16_ft-32xb8-coslr-50e_in1k-448/vit-huge-p16_ft-32xb8-coslr-50e_in1k-448_20220916-95b6a0ce.json) |

```bibtex
@article{He2021MaskedAA,
  title={Masked Autoencoders Are Scalable Vision Learners},
  author={Kaiming He and Xinlei Chen and Saining Xie and Yanghao Li and
  Piotr Doll'ar and Ross B. Girshick},

  journal={arXiv},
}
```

---
name: "\U0001F6B8 model convert issue"
about: "Life is Short, Use pnnx and convertmodel.com"
---

## error log | 日志或报错信息 | ログ

## model | 模型 | モデル
1. original model

## how to reproduce | 复现步骤 | 再現方法
1.
2.
3.

# Adding New Dataset

You can write a new dataset class inherited from `BaseDataset`, and overwrite `load_data_list(self)`,
like [CIFAR10](https://github.com/open-mmlab/mmpretrain/blob/main/mmpretrain/datasets/cifar.py) and [ImageNet](https://github.com/open-mmlab/mmpretrain/blob/main/mmpretrain/datasets/imagenet.py).
Typically, this function returns a list, where each sample is a dict, containing necessary data information, e.g., `img` and `gt_label`.

Assume we are going to implement a `Filelist` dataset, which takes filelists for both training and testing. The format of annotation list is as follows:

```text
000001.jpg 0
000002.jpg 1
```

## 1. Create Dataset Class

We can create a new dataset in `mmpretrain/datasets/filelist.py` to load the data.

```python
from mmpretrain.registry import DATASETS
from .base_dataset import BaseDataset

class Filelist(BaseDataset):

        assert isinstance(self.ann_file, str),

        with open(self.ann_file) as f:
            samples = [x.strip().split(' ') for x in f.readlines()]
            for filename, gt_label in samples:
                img_path = add_prefix(filename, self.img_prefix)
                info = {'img_path': img_path, 'gt_label': int(gt_label)}
                data_list.append(info)
```

## 2. Add to the package

And add this dataset class in `mmpretrain/datasets/__init__.py`

```python
...
from .filelist import Filelist

__all__ = [
    'BaseDataset', ... ,'Filelist'
]
```

## 3. Modify Related Config

Then in the config, to use `Filelist` you can modify the config as the following

```python
    ...
        type='Filelist',

```python
    ...
        ann_file='image_list.txt',
    )
)
```

All the dataset classes inherit from [`BaseDataset`](https://github.com/open-mmlab/mmpretrain/blob/main/mmpretrain/datasets/base_dataset.py) have **lazy loading** and **memory saving** features, you can refer to related documents of {external+mmengine:doc}`BaseDataset <advanced_tutorials/basedataset>`.

```{note}
If the dictionary of the data sample contains 'img_path' but not 'img', then 'LoadImgFromFile' transform must be added in the pipeline.
```

# Positional Encoding in GANs (CVPR'2021)

> [Positional Encoding as Spatial Inductive Bias in GANs](https://openaccess.thecvf.com/content/CVPR2021/html/Xu_Positional_Encoding_As_Spatial_Inductive_Bias_in_GANs_CVPR_2021_paper.html)

> **Task**: Unconditional GANs

SinGAN shows impressive capability in learning internal patch distribution despite its limited effective receptive field. We are interested in knowing how such a translation-invariant convolutional generator could capture the global structure with just a spatially i.i.d. input. In this work, taking SinGAN and StyleGAN2 as examples, we show that such capability, to a large extent, is brought by the implicit positional encoding when using zero padding in the generators. Such positional encoding is indispensable for generating images with high fidelity. The same phenomenon is observed in other generative architectures such as DCGAN and PGGAN. We further show that zero padding leads to an unbalanced spatial bias with a vague relation between locations. To offer a better spatial inductive bias, we investigate alternative positional encodings and analyze their effects. Based on a more flexible positional encoding explicitly, we propose a new multi-scale training strategy and demonstrate its effectiveness in the state-of-the-art unconditional generator StyleGAN2. Besides, the explicit spatial inductive bias substantially improve SinGAN for more versatile image manipulation.

<img src="https://user-images.githubusercontent.com/28132635/143053767-c6a503b2-87ff-434a-a439-d9fb0e98d804.JPG"/>
</div>

## Results and models for MS-PIE

  <b> 896x896 results generated from a 256 generator using MS-PIE</b>
  <br/>
  <img src="https://download.openmmlab.com/mmediting/pe_in_gans/mspie_256-896_demo.png" width="800"/>
</div>

|                                 Model                                  | Dataset | Reference in Paper |     Scales     | FID50k |   P&R10k    |                                  Download                                  |
| :--------------------------------------------------------------------: | :-----: | :----------------: | :------------: | :----: | :---------: | :------------------------------------------------------------------------: |

| [stylegan2_c2_8xb3-1100kiters_ffhq-256x256](./stylegan2_c2_8xb3-1100kiters_ffhq-256x256.py) |  FFHQ   |   Tab.5 config-a   |      256       |  5.56  | 75.92/51.24 | [model](https://download.openmmlab.com/mmediting/pe_in_gans/stylegan2_c2_config-a_ffhq_256x256_b3x8_1100k_20210406_145127-71d9634b.pth) |
| [stylegan2_c2_8xb3-1100kiters_ffhq-512x512](./stylegan2_c2_8xb3-1100kiters_ffhq-512x512.py) |  FFHQ   |   Tab.5 config-b   |      512       |  4.91  | 75.65/54.58 | [model](https://download.openmmlab.com/mmediting/pe_in_gans/stylegan2_c2_config-b_ffhq_512x512_b3x8_1100k_20210406_145142-e85e5cf4.pth) |
| [mspie-stylegan2-config-c_c2_8xb3-1100kiters_ffhq-256-512](./mspie-stylegan2-config-c_c2_8xb3-1100kiters_ffhq-256-512.py) |  FFHQ   |   Tab.5 config-c   | 256, 384, 512  |  3.35  | 73.84/55.77 | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c2_config-c_ffhq_256-512_b3x8_1100k_20210406_144824-9f43b07d.pth) |

| [mspie-stylegan2-config-d_c2_8xb3-1100kiters_ffhq-256-512](./mspie-stylegan2-config-d_c2_8xb3-1100kiters_ffhq-256-512.py) |  FFHQ   |   Tab.5 config-d   | 256, 384, 512  |  3.50  | 73.28/56.16 | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c2_config-d_ffhq_256-512_b3x8_1100k_20210406_144840-dbefacf6.pth) |
| [mspie-stylegan2-config-e_c2_8xb3-1100kiters_ffhq-256-512](./mspie-stylegan2-config-e_c2_8xb3-1100kiters_ffhq-256-512.py) |  FFHQ   |   Tab.5 config-e   | 256, 384, 512  |  3.15  | 74.13/56.88 | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c2_config-e_ffhq_256-512_b3x8_1100k_20210406_144906-98d5a42a.pth) |

| [mspie-stylegan2-config-f_c2_8xb3-1100kiters_ffhq-256-512](./mspie-stylegan2-config-f_c2_8xb3-1100kiters_ffhq-256-512.py) |  FFHQ   |   Tab.5 config-f   | 256, 384, 512  |  2.93  | 73.51/57.32 | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c2_config-f_ffhq_256-512_b3x8_1100k_20210406_144927-4f4d5391.pth) |
| [mspie-stylegan2-config-g_c1_8xb3-1100kiters_ffhq-256-512](./mspie-stylegan2-config-g_c1_8xb3-1100kiters_ffhq-256-512.py) |  FFHQ   |   Tab.5 config-g   | 256, 384, 512  |  3.40  | 73.05/56.45 | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c1_config-g_ffhq_256-512_b3x8_1100k_20210406_144758-2df61752.pth) |

| [mspie-stylegan2-config-h_c2_8xb3-1100kiters_ffhq-256-512](./mspie-stylegan2-config-h_c2_8xb3-1100kiters_ffhq-256-512.py) |  FFHQ   |   Tab.5 config-h   | 256, 384, 512  |  4.01  | 72.81/54.35 | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c2_config-h_ffhq_256-512_b3x8_1100k_20210406_145006-84cf3f48.pth) |
| [mspie-stylegan2-config-i_c2_8xb3-1100kiters_ffhq-256-512](./mspie-stylegan2-config-i_c2_8xb3-1100kiters_ffhq-256-512.py) |  FFHQ   | 2  Tab.5 config-i  |  56, 384, 512  |  3.76  | 73.26/54.71 | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c2_config-i_ffhq_256-512_b3x8_1100k_20210406_145023-c2b0accf.pth) |

| [mspie-stylegan2-config-j_c2_8xb3-1100kiters_ffhq-256-512](./mspie-stylegan2-config-j_c2_8xb3-1100kiters_ffhq-256-512.py) |  FFHQ   |   Tab.5 config-j   | 256, 384, 512  |  4.23  | 73.11/54.63 | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c2_config-j_ffhq_256-512_b3x8_1100k_20210406_145044-c407481b.pth) |
| [mspie-stylegan2-config-k_c2_8xb3-1100kiters_ffhq-256-512](./mspie-stylegan2-config-k_c2_8xb3-1100kiters_ffhq-256-512.py) |  FFHQ   |   Tab.5 config-k   | 256, 384, 512  |  4.17  | 73.05/51.07 | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c2_config-k_ffhq_256-512_b3x8_1100k_20210406_145105-6d8cc39f.pth) |

| [mspie-stylegan2-config-f_c2_8xb3-1100kiters_ffhq-256-896](./mspie-stylegan2-config-f_c2_8xb3-1100kiters_ffhq-256-896.py) |  FFHQ   | higher-resolution  | 256, 512, 896  |  4.10  | 72.21/50.29 | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c2_config-f_ffhq_256-896_b3x8_1100k_20210406_144943-6c18ad5d.pth) |
| [mspie-stylegan2-config-f_c1_8xb2-1600kiters_ffhq-256-1024](./mspie-stylegan2-config-f_c1_8xb2-1600kiters_ffhq-256-1024.py) |  FFHQ   | higher-resolution  | 256, 512, 1024 |  6.24  | 71.79/49.92 | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c1_config-f_ffhq_256-1024_b2x8_1600k_20210406_144716-81cbdc96.pth) |

|                               Model                               | Dataset | Reference in Paper |     Scales     | FID50k | Precision10k | Recall10k |                               Download                                |

| :---------------------------------------------------------------: | :-----: | :----------------: | :------------: | :----: | :----------: | :-------: | :-------------------------------------------------------------------: |
| [stylegan2_c2_8xb3-1100kiters_ffhq-256x256](./stylegan2_c2_8xb3-1100kiters_ffhq-256x256.py) |  FFHQ   |   Tab.5 config-a   |      256       |  5.56  |    75.92     |   51.24   | [model](https://download.openmmlab.com/mmediting/pe_in_gans/stylegan2_c2_config-a_ffhq_256x256_b3x8_1100k_20210406_145127-71d9634b.pth) |
| [stylegan2_c2_8xb3-1100kiters_ffhq-512x512](./stylegan2_c2_8xb3-1100kiters_ffhq-512x512.py) |  FFHQ   |   Tab.5 config-b   |      512       |  4.91  |    75.65     |   54.58   | [model](https://download.openmmlab.com/mmediting/pe_in_gans/stylegan2_c2_config-b_ffhq_512x512_b3x8_1100k_20210406_145142-e85e5cf4.pth) |

| [mspie-stylegan2-config-c_c2_8xb3-1100kiters_ffhq-256-512](./mspie-stylegan2-config-c_c2_8xb3-1100kiters_ffhq-256-512.py) |  FFHQ   |   Tab.5 config-c   | 256, 384, 512  |  3.35  |    73.84     |   55.77   | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c2_config-c_ffhq_256-512_b3x8_1100k_20210406_144824-9f43b07d.pth) |
| [mspie-stylegan2-config-d_c2_8xb3-1100kiters_ffhq-256-512](./mspie-stylegan2-config-d_c2_8xb3-1100kiters_ffhq-256-512.py) |  FFHQ   |   Tab.5 config-d   | 256, 384, 512  |  3.50  |    73.28     |   56.16   | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c2_config-d_ffhq_256-512_b3x8_1100k_20210406_144840-dbefacf6.pth) |

| [mspie-stylegan2-config-e_c2_8xb3-1100kiters_ffhq-256-512](./mspie-stylegan2-config-e_c2_8xb3-1100kiters_ffhq-256-512.py) |  FFHQ   |   Tab.5 config-e   | 256, 384, 512  |  3.15  |    74.13     |   56.88   | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c2_config-e_ffhq_256-512_b3x8_1100k_20210406_144906-98d5a42a.pth) |
| [mspie-stylegan2-config-f_c2_8xb3-1100kiters_ffhq-256-512](./mspie-stylegan2-config-f_c2_8xb3-1100kiters_ffhq-256-512.py) |  FFHQ   |   Tab.5 config-f   | 256, 384, 512  |  2.93  |    73.51     |   57.32   | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c2_config-f_ffhq_256-512_b3x8_1100k_20210406_144927-4f4d5391.pth) |

| [mspie-stylegan2-config-g_c1_8xb3-1100kiters_ffhq-256-512](./mspie-stylegan2-config-g_c1_8xb3-1100kiters_ffhq-256-512.py) |  FFHQ   |   Tab.5 config-g   | 256, 384, 512  |  3.40  |    73.05     |   56.45   | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c1_config-g_ffhq_256-512_b3x8_1100k_20210406_144758-2df61752.pth) |
| [mspie-stylegan2-config-h_c2_8xb3-1100kiters_ffhq-256-512](./mspie-stylegan2-config-h_c2_8xb3-1100kiters_ffhq-256-512.py) |  FFHQ   |   Tab.5 config-h   | 256, 384, 512  |  4.01  |    72.81     |   54.35   | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c2_config-h_ffhq_256-512_b3x8_1100k_20210406_145006-84cf3f48.pth) |

| [mspie-stylegan2-config-i_c2_8xb3-1100kiters_ffhq-256-512](./mspie-stylegan2-config-i_c2_8xb3-1100kiters_ffhq-256-512.py) |  FFHQ   |   Tab.5 config-i   | 256, 384, 512  |  3.76  |    73.26     |   54.71   | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c2_config-i_ffhq_256-512_b3x8_1100k_20210406_145023-c2b0accf.pth) |
| [mspie-stylegan2-config-j_c2_8xb3-1100kiters_ffhq-256-512](./mspie-stylegan2-config-j_c2_8xb3-1100kiters_ffhq-256-512.py) |  FFHQ   |   Tab.5 config-j   | 256, 384, 512  |  4.23  |    73.11     |   54.63   | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c2_config-j_ffhq_256-512_b3x8_1100k_20210406_145044-c407481b.pth) |

| [mspie-stylegan2-config-k_c2_8xb3-1100kiters_ffhq-256-512](./mspie-stylegan2-config-k_c2_8xb3-1100kiters_ffhq-256-512.py) |  FFHQ   |   Tab.5 config-k   | 256, 384, 512  |  4.17  |    73.05     |   51.07   | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c2_config-k_ffhq_256-512_b3x8_1100k_20210406_145105-6d8cc39f.pth) |
| [mspie-stylegan2-config-f_c2_8xb3-1100kiters_ffhq-256-896](./mspie-stylegan2-config-f_c2_8xb3-1100kiters_ffhq-256-896.py) |  FFHQ   | higher-resolution  | 256, 512, 896  |  4.10  |    72.21     |   50.29   | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c2_config-f_ffhq_256-896_b3x8_1100k_20210406_144943-6c18ad5d.pth) |

| [mspie-stylegan2-config-f_c1_8xb2-1600kiters_ffhq-256-1024](./mspie-stylegan2-config-f_c1_8xb2-1600kiters_ffhq-256-1024.py) |  FFHQ   | higher-resolution  | 256, 512, 1024 |  6.24  |    71.79     |   49.92   | [model](https://download.openmmlab.com/mmediting/pe_in_gans/mspie-stylegan2_c1_config-f_ffhq_256-1024_b2x8_1600k_20210406_144716-81cbdc96.pth) |

Note that we report the FID and P&R metric (FFHQ dataset) in the largest scale.

## Results and Models for SinGAN

  <b> Positional Encoding in SinGAN</b>
  <br/>
  <img src="https://nbei.github.io/gan-pos-encoding/teaser-web-singan.png" width="800"/>
</div>

|                             Model                             |                             Dataset                              | Num Scales |                             Download                              |

| :-----------------------------------------------------------: | :--------------------------------------------------------------: | :--------: | :---------------------------------------------------------------: |
| [singan_interp-pad_balloons](./singan_interp-pad_balloons.py) | [balloons.png](https://download.openmmlab.com/mmediting/dataset/singan/balloons.png) |     8      | [ckpt](https://download.openmmlab.com/mmediting/pe_in_gans/singan_interp-pad_balloons_20210406_180014-96f51555.pth) \| [pkl](https://download.openmmlab.com/mmediting/pe_in_gans/singan_interp-pad_balloons_20210406_180014-96f51555.pkl) |

| [singan_interp-pad_disc-nobn_balloons](./singan_interp-pad_disc-nobn_balloons.py) | [balloons.png](https://download.openmmlab.com/mmediting/dataset/singan/balloons.png) |     8      | [ckpt](https://download.openmmlab.com/mmediting/pe_in_gans/singan_interp-pad_disc-nobn_balloons_20210406_180059-7d63e65d.pth) \| [pkl](https://download.openmmlab.com/mmediting/pe_in_gans/singan_interp-pad_disc-nobn_balloons_20210406_180059-7d63e65d.pkl) |
| [singan_interp-pad_disc-nobn_fish](./singan_interp-pad_disc-nobn_fish.py) | [fish.jpg](https://download.openmmlab.com/mmediting/dataset/singan/fish-crop.jpg) |     10     | [ckpt](https://download.openmmlab.com/mmediting/pe_in_gans/singan_interp-pad_disc-nobn_fis_20210406_175720-9428517a.pth) \| [pkl](https://download.openmmlab.com/mmediting/pe_in_gans/singan_interp-pad_disc-nobn_fis_20210406_175720-9428517a.pkl) |

|            [singan-csg_fish](./singan-csg_fish.py)            | [fish.jpg](https://download.openmmlab.com/mmediting/dataset/singan/fish-crop.jpg) |     10     | [ckpt](https://download.openmmlab.com/mmediting/pe_in_gans/singan_csg_fis_20210406_175532-f0ec7b61.pth) \| [pkl](https://download.openmmlab.com/mmediting/pe_in_gans/singan_csg_fis_20210406_175532-f0ec7b61.pkl) |
|        [singan-csg_bohemian](./singan-csg_bohemian.py)        | [bohemian.png](https://download.openmmlab.com/mmediting/dataset/singan/bohemian.png) |     10     | [ckpt](https://download.openmmlab.com/mmediting/pe_in_gans/singan_csg_bohemian_20210407_195455-5ed56db2.pth) \| [pkl](https://download.openmmlab.com/mmediting/pe_in_gans/singan_csg_bohemian_20210407_195455-5ed56db2.pkl) |

|       [singan_spe-dim4_fish](./singan_spe-dim4_fish.py)       | [fish.jpg](https://download.openmmlab.com/mmediting/dataset/singan/fish-crop.jpg) |     10     | [ckpt](https://download.openmmlab.com/mmediting/pe_in_gans/singan_spe-dim4_fish_20210406_175933-f483a7e3.pth) \| [pkl](https://download.openmmlab.com/mmediting/pe_in_gans/singan_spe-dim4_fish_20210406_175933-f483a7e3.pkl) |
|   [singan_spe-dim4_bohemian](./singan_spe-dim4_bohemian.py)   | [bohemian.png](https://download.openmmlab.com/mmediting/dataset/singan/bohemian.png) |     10     | [ckpt](https://download.openmmlab.com/mmediting/pe_in_gans/singan_spe-dim4_bohemian_20210406_175820-6e484a35.pth) \| [pkl](https://download.openmmlab.com/mmediting/pe_in_gans/singan_spe-dim4_bohemian_20210406_175820-6e484a35.pkl) |

|   [singan_spe-dim8_bohemian](./singan_spe-dim8_bohemian.py)   | [bohemian.png](https://download.openmmlab.com/mmediting/dataset/singan/bohemian.png) |     10     | [ckpt](https://download.openmmlab.com/mmediting/pe_in_gans/singan_spe-dim8_bohemian_20210406_175858-7faa50f3.pth) \| [pkl](https://download.openmmlab.com/mmediting/pe_in_gans/singan_spe-dim8_bohemian_20210406_175858-7faa50f3.pkl) |

```latex
@article{xu2020positional,
  title={Positional Encoding as Spatial Inductive Bias in GANs},
  author={Xu, Rui and Wang, Xintao and Chen, Kai and Zhou, Bolei and Loy, Chen Change},
  journal={arXiv preprint arXiv:2012.05217},
  year={2020},
  url={https://openaccess.thecvf.com/content/CVPR2021/html/Xu_Positional_Encoding_As_Spatial_Inductive_Bias_in_GANs_CVPR_2021_paper.html},
}
```

# 准备 Composition-1k 数据集

## 介绍

```bibtex
}
```

Adobe Composition-1k 数据集由前景图像及其相应的 alpha 图像组成。要获得完整的数据集，需要将前景与来自 COCO 数据集和 Pascal VOC 数据集的选定背景进行合成。

## 获取和提取

请按照 [论文作者](https://sites.google.com/view/deepimagematting) 的说明获取 Composition-1k (comp1k) 数据集。

## 合成完整数据集

Adobe composition-1k 数据集仅包含 `alpha` 和 `fg`（以及测试集中的 `trimap`）。在训练或评估之前，需要将 `fg` 与 COCO 数据（训练）或 VOC 数据（测试）合并。使用以下脚本执行图像合成并生成用于训练或测试的注释文件：

```shell
# 在 MMagic 的根文件夹下运行脚本
```

生成的数据分别存储在 “adobe_composition-1k/Training_set” 和 “adobe_composition-1k/Test_set” 下。如果你只想合成测试数据（因为合成训练数据很耗时），你可以通过删除 `--composite` 选项来跳过合成训练集：

```

```shell
# 跳过合成训练集
```

如果您只想预处理测试数据，即对于 FBA，您可以通过添加 `--skip-train` 选项来跳过训练集：

```shell
# 跳过预处理训练集
```

> 目前，`GCA` 和 `FBA` 支持在线合成训练数据。但是你可以修改其他模型的数据管道来执行在线合成，而不是加载合成图像（我们在数据管道中称之为“合并”）。

## 检查 DIM 的目录结构

最终的文件夹结构应如下所示：

```text
mmagic
├── mmagic
├── tools
├── data
│   │   │   │   ├── fg

│   │   │   │   ├── fg
│   │   │   │   ├── fg
│   │   │   │   ├── fg
│   ├── coco

│   ├── coco
```

## 为 FBA 准备数据集

FBA采用 [Learning-base Sampling for Natural Image Matting](https://openaccess.thecvf.com/content_CVPR_2019/papers/Tang_Learning-Based_Sampling_for_Natural_Image_Matting_CVPR_2019_paper.pdf) 中提出的动态数据集增强。此外，为了减少增强过程中的伪影，它使用前景的扩展版本作为前景。我们提供脚本来估计前景。

准备测试集：

```shell
# 跳过预处理训练集，因为它在训练期间在线合成
```

扩展训练集的前景：

```shell
```

```text
mmagic
├── mmagic
├── tools
├── data
│   │   │   │   ├── fg

```text
mmagic
├── mmagic
├── tools
├── data
│   │   │   │   ├── fg
│   │   │   │   ├── fg
│   │   │   │   ├── fg
│   ├── coco

│   ├── coco
```

# Libra R-CNN

> [Libra R-CNN: Towards Balanced Learning for Object Detection](https://arxiv.org/abs/1904.02701)

Compared with model architectures, the training process, which is also crucial to the success of detectors, has received relatively less attention in object detection. In this work, we carefully revisit the standard training practice of detectors, and find that the detection performance is often limited by the imbalance during the training process, which generally consists in three levels - sample level, feature level, and objective level. To mitigate the adverse effects caused thereby, we propose Libra R-CNN, a simple but effective framework towards balanced learning for object detection. It integrates three novel components: IoU-balanced sampling, balanced feature pyramid, and balanced L1 loss, respectively for reducing the imbalance at sample, feature, and objective level. Benefitted from the overall balanced design, Libra R-CNN significantly improves the detection performance. Without bells and whistles, it achieves 2.5 points and 2.0 points higher Average Precision (AP) than FPN Faster R-CNN and RetinaNet respectively on MSCOCO.

Instance recognition is rapidly advanced along with the developments of various deep convolutional neural networks. Compared to the architectures of networks, the training process, which is also crucial to the success of detectors, has received relatively less attention. In this work, we carefully revisit the standard training practice of detectors, and find that the detection performance is often limited by the imbalance during the training process, which generally consists in three levels - sample level, feature level, and objective level. To mitigate the adverse effects caused thereby, we propose Libra R-CNN, a simple yet effective framework towards balanced learning for instance recognition. It integrates IoU-balanced sampling, balanced feature pyramid, and objective re-weighting, respectively for reducing the imbalance at sample, feature, and objective level. Extensive experiments conducted on MS COCO, LVIS and Pascal VOC datasets prove the effectiveness of the overall balanced design.

<img src="https://user-images.githubusercontent.com/40661020/143966392-2633684c-a67a-4269-b71b-afe945c67bcd.png"/>
</div>

The results on COCO 2017val are shown in the below table. (results on test-dev are usually slightly higher than val)

| Architecture |    Backbone     |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                         Config                          |                                                                                                                                                                          Download                                                                                                                                                                           |

| :----------: | :-------------: | :-----: | :-----: | :------: | :------------: | :----: | :-----------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| Faster R-CNN |    R-50-FPN     | pytorch |   1x    |   4.6    |      19.0      |  38.3  |    [config](./libra-faster-rcnn_r50_fpn_1x_coco.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/libra_rcnn/libra_faster_rcnn_r50_fpn_1x_coco/libra_faster_rcnn_r50_fpn_1x_coco_20200130-3afee3a9.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/libra_rcnn/libra_faster_rcnn_r50_fpn_1x_coco/libra_faster_rcnn_r50_fpn_1x_coco_20200130_204655.log.json)               |

|  Fast R-CNN  |    R-50-FPN     | pytorch |   1x    |          |                |        |                                                         |                                                                                                                                                                                                                                                                                                                                                             |
| Faster R-CNN |    R-101-FPN    | pytorch |   1x    |   6.5    |      14.4      |  40.1  |    [config](./libra-faster-rcnn_r101_fpn_1x_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/libra_rcnn/libra_faster_rcnn_r101_fpn_1x_coco/libra_faster_rcnn_r101_fpn_1x_coco_20200203-8dba6a5a.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/libra_rcnn/libra_faster_rcnn_r101_fpn_1x_coco/libra_faster_rcnn_r101_fpn_1x_coco_20200203_001405.log.json)             |

| Faster R-CNN | X-101-64x4d-FPN | pytorch |   1x    |   10.8   |      8.5       |  42.7  | [config](./libra-faster-rcnn_x101-64x4d_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/libra_rcnn/libra_faster_rcnn_x101_64x4d_fpn_1x_coco/libra_faster_rcnn_x101_64x4d_fpn_1x_coco_20200315-3a7d0488.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/libra_rcnn/libra_faster_rcnn_x101_64x4d_fpn_1x_coco/libra_faster_rcnn_x101_64x4d_fpn_1x_coco_20200315_231625.log.json) |
|  RetinaNet   |    R-50-FPN     | pytorch |   1x    |   4.2    |      17.7      |  37.6  |     [config](./libra-retinanet_r50_fpn_1x_coco.py)      |                   [model](https://download.openmmlab.com/mmdetection/v2.0/libra_rcnn/libra_retinanet_r50_fpn_1x_coco/libra_retinanet_r50_fpn_1x_coco_20200205-804d94ce.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/libra_rcnn/libra_retinanet_r50_fpn_1x_coco/libra_retinanet_r50_fpn_1x_coco_20200205_112757.log.json)                   |

We provide config files to reproduce the results in the CVPR 2019 paper [Libra R-CNN](https://arxiv.org/pdf/1904.02701.pdf).

The extended version of [Libra R-CNN](https://arxiv.org/pdf/2108.10175.pdf) is accpeted by IJCV.

```latex
@inproceedings{pang2019libra,
  title={Libra R-CNN: Towards Balanced Learning for Object Detection},
  author={Pang, Jiangmiao and Chen, Kai and Shi, Jianping and Feng, Huajun and Ouyang, Wanli and Dahua Lin},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition},
}

@article{pang2021towards,
  title={Towards Balanced Learning for Instance Recognition},
  author={Pang, Jiangmiao and Chen, Kai and Li, Qi and Xu, Zhihai and Feng, Huajun and Shi, Jianping and Ouyang, Wanli and Lin, Dahua},
  journal={International Journal of Computer Vision},
  volume={129},
  number={5},
  pages={1376--1393},
}
```

# macOS-arm64 下构建方式

- [macOS-arm64 下构建方式](#macos-arm64-下构建方式)
  - [源码安装](#源码安装)
    - [安装构建和编译工具链](#安装构建和编译工具链)
    - [安装依赖包](#安装依赖包)
      - [安装 MMDeploy Converter 依赖](#安装-mmdeploy-converter-依赖)
      - [安装 MMDeploy SDK 依赖](#安装-mmdeploy-sdk-依赖)
      - [安装推理引擎](#安装推理引擎)
    - [编译 MMDeploy](#编译-mmdeploy)
      - [编译 Model Converter](#编译-model-converter)
      - [安装 Model Converter](#安装-model-converter)
      - [编译 SDK 和 Demos](#编译-sdk-和-demos)

## 源码安装

### 安装构建和编译工具链

- cmake

  ```
  brew install cmake
  ```

- clang

  安装 Xcode 或者通过如下命令安装 Command Line Tools

  ```
  xcode-select --install
  ```

### 安装依赖包

#### 安装 MMDeploy Converter 依赖

参考[get_started](../get_started.md)文档，安装conda。

```bash
# install pytoch & mmcv
conda install pytorch==1.9.0 torchvision==0.10.0 -c pytorch
mim install mmengine
mim install "mmcv>=2.0.0rc2"
```

#### 安装 MMDeploy SDK 依赖

如果您只对模型转换感兴趣，那么可以跳过本章节。

<thead>
  <tr>
    <th>名称 </th>
    <th>安装说明 </th>
  </tr>

```

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
    <td>OpenCV<br>(>=3.0) </td>
    <td>
<pre><code>
brew install opencv
</code></pre>
    </td>
</tbody>
</table>

#### 安装推理引擎

MMDeploy 的 Model Converter 和 SDK 共享推理引擎。您可以参考下文，选择自己感兴趣的推理引擎安装。这里重点介绍 Core ML。ONNX Runtime，ncnn 以及 TorchScript 的安装类似 linux 平台，可参考文档 [linux-x86_64](linux-x86_64.md) 进行安装。

Core ML 模型的转化过程中使用 TorchScript 模型作为IR，为了支持含有自定义算子的模型，如 mmdet 中的检测模型，需要安装 libtorch，这里作简单说明。

<table  class="docutils">
<thead>
  <tr>
    <th>名称</th>
    <th>安装包</th>
    <th>安装说明</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>Core ML</td>
    <td>coremltools</td>
    <td>
pip install coremltools==6.0b2
    </td>
  </tr>
  <tr>
  <td>TorchScript</td>
  <td>libtorch</td>
  <td>

<tr>
    <td>
    </td>
  </tr>
  <tr>
  <td>
  1. libtorch暂不提供arm版本的library，故需要自行编译。编译时注意libtorch要和pytorch的版本保持一致，这样编译出的自定义算子才可以加载成功。<br>
  2. 以libtorch 1.9.0为例，可通过如下命令安装:
git clone -b v1.9.0 --recursive https://github.com/pytorch/pytorch.git
cd pytorch
mkdir build && cd build
cmake .. \
    -DCMAKE_BUILD_TYPE=Release \
    -DPYTHON_EXECUTABLE=`which python` \
    -DCMAKE_INSTALL_PREFIX=install \
    -DDISABLE_SVE=ON # low version like 1.9.0 of pytorch need DISABLE_SVE option
make -j4 && make install
export Torch_DIR=$(pwd)/install/share/cmake/Torch
  </td>
  </tr>
</tbody>
</table>

### 编译 MMDeploy

```bash
cd /the/root/path/of/MMDeploy
export MMDEPLOY_DIR=$(pwd)
```

#### 编译 Model Converter

这里介绍使用 Core ML 作为推理后端所需的操作。

- **Core ML**

  Core ML使用 torchscript 作为IR，某些 codebase 如 mmdet 需要编译 torchscript 自定义算子。

```

- **torchscript** 自定义算子

  ```bash
  cd ${MMDEPLOY_DIR}
  mkdir -p build && cd build
  cmake -DMMDEPLOY_TARGET_BACKENDS=coreml -DTorch_DIR=${Torch_DIR} ..
  make -j4 && make install
  ```

参考 [cmake 选项说明](cmake_option.md)

#### 安装 Model Converter

```bash
# requirements/runtime.txt 中依赖项grpcio，通过pip安装的方式无法正常import, 需使用 conda 安装
conda install grpcio
```

```bash
cd ${MMDEPLOY_DIR}
mim install -v -e .
```

**注意**

- 有些依赖项是可选的。运行 `pip install -e .` 将进行最小化依赖安装。 如果需安装其他可选依赖项，请执行`pip install -r requirements/optional.txt`，
  或者 `pip install -e .[optional]`。其中，`[optional]`可以替换为：`all`、`tests`、`build` 或 `optional`。

#### 编译 SDK 和 Demos

下文展示使用 Core ML 作为推理引擎，构建SDK的样例。

- cpu + Core ML

  ```Bash
  cmake .. \
      -DMMDEPLOY_BUILD_SDK=ON \
      -DMMDEPLOY_BUILD_EXAMPLES=ON \

  ```Bash
  cmake .. \
      -DMMDEPLOY_BUILD_SDK_PYTHON_API=ON \
      -DMMDEPLOY_TARGET_DEVICES=cpu \
      -DMMDEPLOY_TARGET_BACKENDS=coreml \
      -DTorch_DIR=${Torch_DIR}

  ```

# Migration of Evaluation and Testing Settings

We update evaluation settings in MMagic 1.x. Important modifications are as following.

- The evaluation field is split to `val_evaluator` and `test_evaluator`. The `interval` is moved to `train_cfg.val_interval`.
- The metrics to evaluation are moved from `test_cfg` to `val_evaluator` and `test_evaluator`.

<thead>
  <tr>
    <th> Original </th>
    <th> New </th>
<tbody>
<tr>
<td valign="top">

```python
train_cfg = None  # Training config
test_cfg = dict(  # Test config
    metrics=['PSNR'],  # Metrics used during testing
    crop_border=scale)  # Crop border during evaluation

evaluation = dict(  # The config to build the evaluation hook
    interval=5000,  # Evaluation interval
    save_image=True,  # Save images during evaluation
    gpu_collect=True)  # Use gpu collect
```

</td>

```python
    dict(type='PSNR', crop_border=scale),  # The name of metrics to evaluate
]

gpu_collect=True)  # Use gpu collect
```

</td>

```python
]

    type='IterBasedTrainLoop', max_iters=300000, val_interval=5000)  # Config of train loop type
val_cfg = dict(type='ValLoop')  # The name of validation loop type
test_cfg = dict(type='TestLoop')  # The name of test loop type
```

</td>

</tr>
</thead>
</table>

We have merged [MMGeneration 1.x](https://github.com/open-mmlab/mmgeneration/tree/1.x) into MMagic. Here is migration of Evaluation and Testing Settings about MMGeneration.

The evaluation field is splited to `val_evaluator` and `test_evaluator`. And it won't support `interval` and `save_best` arguments. The `interval` is moved to `train_cfg.val_interval`, see [the schedule settings](./schedule.md) and the `save_best` is moved to `default_hooks.checkpoint.save_best`.

<thead>

<thead>
  <tr>
    <th> 0.x Version </th>
    <th> 1.x Version </th>
<tbody>
<tr>

```python
    type='GenerativeEvalHook',
    interval=10000,
    metrics=[
        dict(
            type='FID',
            num_images=50000,
            bgr2rgb=True,
            inception_args=dict(type='StyleGAN')),
        dict(type='IS', num_images=50000)
    ],
    best_metric=['fid', 'is'],
    sample_kwargs=dict(sample_model='ema'))
```

</td>

```python
    metrics=[
        dict(
            prefix='FID-Full-50k',
            fake_nums=50000,
            inception_style='StyleGAN',
            sample_model='orig')
        dict(
            type='IS',
            prefix='IS-50k',
            fake_nums=50000)])
# set best config
        interval=10000,

prefix='IS-50k',
        less_keys=['FID-Full-50k/fid'],
        greater_keys=['IS-50k/is'],
        save_optimizer=True,
        save_best=['FID-Full-50k/fid', 'IS-50k/is'],
        rule=['less', 'greater']))
```

</td>

</tr>
</thead>
</table>

To evaluate and test the model correctly, we need to set specific loop in `val_cfg` and `test_cfg`.

<thead>
  <tr>
    <th> Static Model in 0.x Version </th>
    <th> Static Model in 1.x Version </th>
<tbody>
<tr>

```python
total_iters = 1000000

    type='DynamicIterBasedRunner',
    is_dynamic_ddp=False,
    pass_training_status=True)
```

</td>

```python
    by_epoch=False,  # use iteration based training

is_dynamic_ddp=False,
```

</td>

```python
    max_iters=1000000,  # max training iteration
    val_begin=1,
    val_interval=10000)  # evaluation interval
val_cfg = dict(type='MultiValLoop')  # specific loop in validation
test_cfg = dict(type='MultiTestLoop')  # specific loop in testing
```

</td>

</tr>
</thead>
</table>

# Example Project

This is an example README for community `projects/`. You can write your README in your own project. Here are
some recommended parts of a README for others to understand and use your project, you can copy or modify them
according to your project.

## Description \[required\]

You can share any information you would like others to know. For example:

```
Author: @xxx.

This is an implementation of \[XXX\].
```

## Usage \[required\]

### Setup Environment \[required\]

Please refer to [Get Started](https://mmagic.readthedocs.io/en/latest/get_started/I.html) to install
MMagic.

At first, add the current folder to `PYTHONPATH`, so that Python can find your code. Run command in the current directory to add it.

> Please run it every time after you opened a new shell.

```shell
```

### Data Preparation \[optional\]

```shell
```

Prepare the ImageNet-2012 dataset according to the [instruction](https://mmagic.readthedocs.io/en/latest/user_guides/dataset_prepare.html#imagenet).

### Training commands \[optional\]

**To train with single GPU:**

```bash
mim train mmagic configs/examplenet_8xb32_in1k.py
```

**To train with multiple GPUs:**

```bash
mim train mmagic configs/examplenet_8xb32_in1k.py --launcher pytorch --gpus 8
```

**To train with multiple GPUs by slurm:**

```bash
mim train mmagic configs/examplenet_8xb32_in1k.py --launcher slurm \
    --gpus 16 --gpus-per-node 8 --partition $PARTITION
```

### Testing commands \[required\]

**To test with single GPU:**

```bash
mim test mmagic configs/examplenet_8xb32_in1k.py $CHECKPOINT
```

**To test with multiple GPUs:**

```bash
mim test mmagic configs/examplenet_8xb32_in1k.py $CHECKPOINT --launcher pytorch --gpus 8

```

```bash
```

**To test with multiple GPUs by slurm:**

```bash
mim test mmagic configs/examplenet_8xb32_in1k.py $CHECKPOINT --launcher slurm \
```

## Results \[required\]

|       Model        |   Pretrain   | Top-1 (%) | Top-5 (%) |                   Config                   |                Download                |
| :----------------: | :----------: | :-------: | :-------: | :----------------------------------------: | :------------------------------------: |
|  ExampleNet-tiny   | From scratch |   82.33   |   96.15   | [config](configs/examplenet_8xb32_in1k.py) | [model](MODEL-LINK) \| [log](LOG-LINK) |

| ExampleNet-small\* | From scratch |   83.63   |   96.51   | [config](configs/examplenet_8xb32_in1k.py) |          [model](MODEL-LINK)           |
| ExampleNet-base\*  | From scratch |   84.34   |   96.86   | [config](configs/examplenet_8xb32_in1k.py) |          [model](MODEL-LINK)           |

*Models with * are converted from the [official repo](REPO-LINK). The config files of these models are only for inference. We don't ensure these config files' training accuracy and welcome you to contribute your reproduction results.*

You can also paste some visual results here if the model doesn't have quantitative results.

## Citation \[required\]

<!-- Replace to the citation of the paper your project refers to. -->

```bibtex

```bibtex
}
```

## Checklist \[required\]

Here is a checklist of this project's progress. And you can ignore this part if you don't plan to contribute
to MMagic projects.

- [ ] Milestone 1: PR-ready, and acceptable to be one of the `projects/`.

  - [ ] Finish the code

    <!-- The code's design shall follow existing interfaces and convention. For example, each model component should be registered into `mmagic.registry.MODELS` and configurable via a config file. -->

  - [ ] Basic docstrings & proper citation

- [ ] Basic docstrings & proper citation

    <!-- Each major class should contains a docstring, describing its functionality and arguments. If your code is copied or modified from other open-source projects, don't forget to cite the source project in docstring and make sure your behavior is not against its license. Typically, we do not accept any code snippet under GPL license. [A Short Guide to Open Source Licenses](https://medium.com/nationwide-technology/a-short-guide-to-open-source-licenses-cf5b1c329edd) -->

  - [ ] Converted checkpoint and results (Only for reproduction)

    <!-- If you are reproducing the result from a paper, make sure the model in the project can match that results. Also please provide checkpoint links or a checkpoint conversion script for others to get the pre-trained model. -->

  - [ ] Training results

    <!-- If you are reproducing the result from a paper, train your model from scratch and verified that the final result can match the original result. -->

    <!-- Unit tests for the major module are required. [Example](https://github.com/open-mmlab/mmagic/blob/main/tests/test_models/test_backbones/test_vision_transformer.py) -->

  - [ ] Code style

  - [ ] `metafile.yml` and `README.md`

    <!-- It will used for mmagic to acquire your models. [Example](https://github.com/open-mmlab/mmagic/blob/main/configs/mvit/metafile.yml). In particular, you may have to refactor this README into a standard one. [Example](https://github.com/open-mmlab/mmagic/blob/main/configs/swin_transformer/README.md) -->

# KLD

> [Learning High-Precision Bounding Box for Rotated Object Detection via Kullback-Leibler Divergence](https://arxiv.org/pdf/2106.01883.pdf)

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/kld.png" width="800"/>
</div>

Existing rotated object detectors are mostly inherited from the horizontal detection paradigm, as the latter has evolved into a well-developed area. However, these detectors are difficult to perform prominently in high-precision detection due to the limitation of current regression loss design, especially for objects with large aspect ratios. Taking the perspective that horizontal detection is a special case for rotated object detection, in this paper, we are motivated to change the design of rotation regression loss from induction paradigm to deduction methodology, in terms of the relation between rotation and horizontal detection. We show that one essential challenge is how to modulate the coupled parameters in the rotation regression loss, as such the estimated parameters can influence to each other during the dynamic joint optimization, in an adaptive and synergetic way. Specifically, we first convert the rotated bounding box into a 2-D Gaussian distribution, and then calculate the Kullback-Leibler Divergence (KLD) between the Gaussian distributions as the regression loss. By analyzing the gradient of each parameter, we show that KLD (and its derivatives) can dynamically adjust the parameter gradients according to the characteristics of the object. For instance, it will adjust the importance (gradient weight) of the angle parameter according to the aspect ratio. This mechanism can be vital for high-precision detection as a slight angle error would cause a serious accuracy drop for large aspect ratios objects. More importantly, we have proved that KLD is scale invariant. We further show that the KLD loss can be degenerated into the popular $l\_{n}$-norm loss for horizontal detection. Experimental results on seven datasets using different detectors show its consistent superiority

DOTA1.0

|         Backbone         |  mAP  | Angle | lr schd | Mem (GB) | Inf Time (fps) | Aug | Batch Size |                                                              Configs                                                              |                                                                                                                                                                                                    Download                                                                                                                                                                                                    |

| ResNet50 (1024,1024,200) | 64.55 |  oc   |   1x    |   3.38   |      15.7      |  -  |     2      |          [rotated-retinanet-hbox-oc_r50_fpn_1x_dota](../rotated_retinanet/rotated-retinanet-hbox-oc_r50_fpn_1x_dota.py)           |                         [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_hbb_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_r50_fpn_1x_dota_oc-e8a7c7df.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_hbb_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_r50_fpn_1x_dota_oc_20220121_095315.log.json)                         |

| ResNet50 (1024,1024,200) | 69.94 |  oc   |   1x    |   3.39   |      15.6      |  -  |     2      |                [rotated-retinanet-hbox-oc_r50_fpn_kld_1x_dota](./rotated-retinanet-hbox-oc_r50_fpn_kld_1x_dota.py)                |                               [model](https://download.openmmlab.com/mmrotate/v0.1.0/kld/rotated_retinanet_hbb_kld_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_kld_r50_fpn_1x_dota_oc-49c1f937.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kld/rotated_retinanet_hbb_kld_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_kld_r50_fpn_1x_dota_oc_20220125_201832.log.json)                               |

| ResNet50 (1024,1024,200) | 69.86 |  oc   |   1x    |   3.35   |      15.8      |  -  |     2      |         [rotated-retinanet-hbox-oc_r50_fpn_kld-stable_1x_dota](./rotated-retinanet-hbox-oc_r50_fpn_kld-stable_1x_dota.py)         |                 [model](https://download.openmmlab.com/mmrotate/v0.1.0/kld/rotated_retinanet_hbb_kld_stable_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_kld_stable_r50_fpn_1x_dota_oc-92a76443.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kld/rotated_retinanet_hbb_kld_stable_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_kld_stable_r50_fpn_1x_dota_oc_20220403_120631.log.json)                 |

| ResNet50 (1024,1024,200) | 68.42 | le90  |   1x    |   3.38   |      16.9      |  -  |     2      |        [rotated-retinanet-rbox-le90_r50_fpn_1x_dota](../rotated_retinanet/rotated-retinanet-rbox-le90_r50_fpn_1x_dota.py)         |                     [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le90/rotated_retinanet_obb_r50_fpn_1x_dota_le90-c0097bc4.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le90/rotated_retinanet_obb_r50_fpn_1x_dota_le90_20220128_130740.log.json)                     |

| ResNet50 (1024,1024,200) | 70.22 | le90  |   1x    |   3.35   |      16.9      |  -  |     2      |       [rotated-retinanet-rbox-le90_r50_fpn_kld-stable_1x_dota](./rotated-retinanet-rbox-le90_r50_fpn_kld-stable_1x_dota.py)       |             [model](https://download.openmmlab.com/mmrotate/v0.1.0/kld/rotated_retinanet_obb_kld_stable_r50_fpn_1x_dota_le90/rotated_retinanet_obb_kld_stable_r50_fpn_1x_dota_le90-31193e00.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kld/rotated_retinanet_obb_kld_stable_r50_fpn_1x_dota_le90/rotated_retinanet_obb_kld_stable_r50_fpn_1x_dota_le90_20220402_225531.log.json)             |

| ResNet50 (1024,1024,200) | 71.30 | le90  |   1x    |   3.61   |      16.9      |  -  |     2      | [rotated-retinanet-rbox-le90_r50_fpn_kld-stable_adamw-1x_dota](./rotated-retinanet-rbox-le90_r50_fpn_kld-stable_adamw-1x_dota.py) | [model](https://download.openmmlab.com/mmrotate/v0.1.0/kld/rotated_retinanet_obb_kld_stable_r50_adamw_fpn_1x_dota_le90/rotated_retinanet_obb_kld_stable_r50_adamw_fpn_1x_dota_le90-474d9955.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kld/rotated_retinanet_obb_kld_stable_r50_adamw_fpn_1x_dota_le90/rotated_retinanet_obb_kld_stable_r50_adamw_fpn_1x_dota_le90_20220608_003758.log.json) |

|         Backbone         |  mAP  | Angle | lr schd | Mem (GB) | Inf Time (fps) | Aug | Batch Size |                                     Configs                                     |                                                                                                                                                    Download                                                                                                                                                    |
| :----------------------: | :---: | :---: | :-----: | :------: | :------------: | :-: | :--------: | :-----------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

| ResNet50 (1024,1024,200) | 69.80 |  oc   |   1x    |   3.54   |      12.4      |  -  |     2      |        [r3det-oc_r50_fpn_1x_dota](../r3det/r3det-oc_r50_fpn_1x_dota.py)         |                     [model](https://download.openmmlab.com/mmrotate/v0.1.0/r3det/r3det_r50_fpn_1x_dota_oc/r3det_r50_fpn_1x_dota_oc-b1fb045c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/r3det/r3det_r50_fpn_1x_dota_oc/r3det_r50_fpn_1x_dota_oc_20220126_191226.log.json)                     |
| ResNet50 (1024,1024,200) | 71.83 |  oc   |   1x    |   3.54   |      12.4      |  -  |     2      | [r3det-oc_r50_fpn_kld-stable_1x_dota](./r3det-oc_r50_fpn_kld-stable_1x_dota.py) |               [model](https://download.openmmlab.com/mmrotate/v0.1.0/kld/r3det_kld_r50_fpn_1x_dota_oc/r3det_kld_r50_fpn_1x_dota_oc-31866226.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kld/r3det_kld_r50_fpn_1x_dota_oc/r3det_kld_r50_fpn_1x_dota_oc_20220210_114049.log.json)               |

| ResNet50 (1024,1024,200) | 72.12 |  oc   |   1x    |   3.81   |      13.5      |  -  |     2      | [r3det-oc_r50_fpn_kld-stable_1x_dota](./r3det-oc_r50_fpn_kld-stable_1x_dota.py) | [model](https://download.openmmlab.com/mmrotate/v0.1.0/kld/r3det_kld_stable_r50_fpn_1x_dota_oc/r3det_kld_stable_r50_fpn_1x_dota_oc-e011059d.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kld/r3det_kld_stable_r50_fpn_1x_dota_oc/r3det_kld_stable_r50_fpn_1x_dota_oc_20220402_225905.log.json) |

|         Backbone         |  mAP  | Angle | lr schd | Mem (GB) | Inf Time (fps) | Aug | Batch Size |                                   Configs                                   |                                                                                                                                                Download                                                                                                                                                |

| :----------------------: | :---: | :---: | :-----: | :------: | :------------: | :-: | :--------: | :-------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ResNet50 (1024,1024,200) | 70.18 |  oc   |   1x    |   3.23   |      15.6      |  -  |     2      | [r3det-tiny-oc_r50_fpn_1x_dota](../r3det/r3det-tiny-oc_r50_fpn_1x_dota.py)  |       [model](https://download.openmmlab.com/mmrotate/v0.1.0/r3det/r3det_tiny_r50_fpn_1x_dota_oc/r3det_tiny_r50_fpn_1x_dota_oc-c98a616c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/r3det/r3det_tiny_r50_fpn_1x_dota_oc/r3det_tiny_r50_fpn_1x_dota_oc_20220209_171624.log.json)       |

| ResNet50 (1024,1024,200) | 72.76 |  oc   |   1x    |   3.44   |      14.0      |  -  |     2      | [r3det-tiny-oc_r50_fpn_kld_1x_dota](./r3det-tiny-oc_r50_fpn_kld_1x_dota.py) | [model](https://download.openmmlab.com/mmrotate/v0.1.0/kld/r3det_tiny_kld_r50_fpn_1x_dota_oc/r3det_tiny_kld_r50_fpn_1x_dota_oc-589e142a.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kld/r3det_tiny_kld_r50_fpn_1x_dota_oc/r3det_tiny_kld_r50_fpn_1x_dota_oc_20220209_172917.log.json) |

HRSC

HRSC

|      Backbone      |  mAP  | AP50  | AP75  | Angle | lr schd | Mem (GB) | Inf Time (fps) | Aug | Batch Size |                                                           Configs                                                           |                                                                                                                                                                                              Download                                                                                                                                                                                              |

| ResNet50 (800,512) | 52.06 | 84.80 | 58.10 | le90  |   6x    |   1.56   |      38.2      | RR  |     2      |  [rotated-retinanet-rbox-le90_r50_fpn_rr-6x_hrsc](../rotated_retinanet/rotated-retinanet-rbox-le90_r50_fpn_rr-6x_hrsc.py)   |         [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_6x_hrsc_rr_le90/rotated_retinanet_obb_r50_fpn_6x_hrsc_rr_le90-ee4f18af.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_6x_hrsc_rr_le90/rotated_retinanet_obb_r50_fpn_6x_hrsc_rr_le90_20220412_110739.log.json)         |

| ResNet50 (800,512) | 54.15 | 86.20 | 60.60 | le90  |   6x    |   1.56   |      38.2      | RR  |     2      | [rotated-retinanet-rbox-le90_r50_fpn_kld-stable_rr-6x_hrsc](./rotated-retinanet-rbox-le90_r50_fpn_kld-stable_rr-6x_hrsc.py) | [model](https://download.openmmlab.com/mmrotate/v0.1.0/kld/rotated_retinanet_obb_kld_stable_r50_fpn_6x_hrsc_rr_le90/rotated_retinanet_obb_kld_stable_r50_fpn_6x_hrsc_rr_le90-58665364.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kld/rotated_retinanet_obb_kld_stable_r50_fpn_6x_hrsc_rr_le90/rotated_retinanet_obb_kld_stable_r50_fpn_6x_hrsc_rr_le90_20220412_155521.log.json) |

| ResNet50 (800,512) | 45.09 | 79.30 | 46.90 |  oc   |   6x    |   1.56   |      39.2      | RR  |     2      |    [rotated-retinanet-hbox-oc_r50_fpn_rr-6x_hrsc](../rotated_retinanet/rotated-retinanet-hbox-oc_r50_fpn_rr-6x_hrsc.py)     |             [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_hbb_r50_fpn_6x_hrsc_rr_oc/rotated_retinanet_hbb_r50_fpn_6x_hrsc_rr_oc-f37eada6.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_hbb_r50_fpn_6x_hrsc_rr_oc/rotated_retinanet_hbb_r50_fpn_6x_hrsc_rr_oc_20220412_103639.log.json)             |

| ResNet50 (800,512) | 58.17 | 87.00 | 69.30 |  oc   |   6x    |   1.56   |      39.5      | RR  |     2      |   [rotated-retinanet-hbox-oc_r50_fpn_kld-stable_rr-6x_hrsc](./rotated-retinanet-hbox-oc_r50_fpn_kld-stable_rr-6x_hrsc.py)   |     [model](https://download.openmmlab.com/mmrotate/v0.1.0/kld/rotated_retinanet_hbb_kld_stable_r50_fpn_6x_hrsc_rr_oc/rotated_retinanet_hbb_kld_stable_r50_fpn_6x_hrsc_rr_oc-9a4ac8e2.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kld/rotated_retinanet_hbb_kld_stable_r50_fpn_6x_hrsc_rr_oc/rotated_retinanet_hbb_kld_stable_r50_fpn_6x_hrsc_rr_oc_20220413_162440.log.json)     |

```
@inproceedings{yang2021learning,
	title={Learning High-Precision Bounding Box for Rotated Object Detection via Kullback-Leibler Divergence},
	author={Yang, Xue and Yang, Xiaojiang and Yang, Jirui and Ming, Qi and Wang, Wentao and Tian, Qi and Yan, Junchi},
	booktitle={Advances in Neural Information Processing Systems},
	year={2021}
}
```

# 教程3：使用预训练模型推理

MMSegmentation 在 [Model Zoo](../Model_Zoo.md) 中为语义分割提供了预训练的模型，并支持多个标准数据集，包括 Cityscapes、ADE20K 等。
本说明将展示如何使用现有模型对给定图像进行推理。
关于如何在标准数据集上测试现有模型，请参阅本[指南](./4_train_test.md)

MMSegmentation 为用户提供了数个接口，以便轻松使用预训练的模型进行推理。

- [教程3：使用预训练模型推理](#教程3使用预训练模型推理)
  - [推理器](#推理器)
    - [基本使用](#基本使用)
    - [初始化](#初始化)
    - [可视化预测结果](#可视化预测结果)
    - [模型列表](#模型列表)
  - [推理 API](#推理-api)
    - [mmseg.apis.init_model](#mmsegapisinit_model)
    - [mmseg.apis.inference_model](#mmsegapisinference_model)
    - [mmseg.apis.show_result_pyplot](#mmsegapisshow_result_pyplot)

## 推理器

在 MMSegmentation 中，我们提供了最**方便的**方式 `MMSegInferencer` 来使用模型。您只需 3 行代码就可以获得图像的分割掩膜。

### 基本使用

以下示例展示了如何使用 `MMSegInferencer` 对单个图像执行推理。

```
>>> from mmseg.apis import MMSegInferencer
>>> # 将模型加载到内存中
>>> inferencer = MMSegInferencer(model='deeplabv3plus_r18-d8_4xb2-80k_cityscapes-512x1024')
>>> # 推理
>>> inferencer('demo/demo.png', show=True)
```

可视化结果应如下所示：

>>> # 推理
```

    <img src='https://user-images.githubusercontent.com/76149310/221507927-ae01e3a7-016f-4425-b966-7b19cbbe494e.png' />
</div>

此外，您可以使用 `MMSegInferencer` 来处理一个包含多张图片的 `list`：

```
# 输入一个图片 list
>>> images = [image1, image2, ...] # image1 可以是文件路径或 np.ndarray
>>> inferencer(images, show=True, wait_time=0.5) # wait_time 是延迟时间，0 表示无限

# 或输入图像目录
>>> images = $IMAGESDIR
>>> inferencer(images, show=True, wait_time=0.5)

# 保存可视化渲染彩色分割图和预测结果
# out_dir 是保存输出结果的目录，img_out_dir 和 pred_out_dir 为 out_dir 的子目录
# 以保存可视化渲染彩色分割图和预测结果
>>> inferencer(images, out_dir='outputs', img_out_dir='vis', pred_out_dir='pred')
```

推理器有一个可选参数 `return_datasamples`，其默认值为 False，推理器的返回值默认为 `dict` 类型，包括 'visualization' 和 'predictions' 两个 key。

```

如果 `return_datasamples=True` 推理器将返回 [`SegDataSample`](../advanced_guides/structures.md) 或其列表。

```
result = inferencer('demo/demo.png')
# 结果是一个包含 'visualization' 和 'predictions' 两个 key 的 `dict`
# 'visualization' 包含彩色分割图
print(result['visualization'].shape)
# (512, 683, 3)

# 'predictions' 包含带有标签索引的分割掩膜
print(result['predictions'].shape)
# (512, 683)

result = inferencer('demo/demo.png', return_datasamples=True)
print(type(result))
# <class 'mmseg.structures.seg_data_sample.SegDataSample'>

results = inferencer(images)
# 输出为列表
print(type(results['visualization']), results['visualization'][0].shape)
# <class 'list'> (512, 683, 3)
print(type(results['predictions']), results['predictions'][0].shape)
# <class 'list'> (512, 683)

results = inferencer(images, return_datasamples=True)

# <class 'list'>
print(type(results[0]))
```

### 初始化

`MMSegInferencer` 必须使用 `model` 初始化，该 `model` 可以是模型名称或一个 `Config`，甚至可以是配置文件的路径。
模型名称可以在模型的元文件（configs/xxx/metafile.yaml）中找到，比如 maskformer 的一个模型名称是 `maskformer_r50-d32_8xb2-160k_ade20k-512x512`，如果输入模型名称，模型的权重将自动下载。以下是其他输入参数：

- weights（str，可选）- 权重的路径。如果未指定，并且模型是元文件中的模型名称，则权重将从元文件加载。默认为 None。
- classes（list，可选）- 输入类别用于结果渲染，由于分割模型的预测结构是标签索引的分割图，`classes` 是一个相应的标签索引的类别列表。若 classes 没有定义，可视化工具将默认使用 `cityscapes` 的类别。默认为 None。
- palette（list，可选）- 输入调色盘用于结果渲染，它是对应分类的配色列表。若 palette 没有定义，可视化工具将默认使用 `cityscapes` 的调色盘。默认为 None。

- dataset_name（str，可选）- [数据集名称或别名](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/utils/class_names.py#L302-L317)，可视化工具将使用数据集的元信息，如类别和配色，但 `classes` 和 `palette` 具有更高的优先级。默认为 None。
- device（str，可选）- 运行推理的设备。如果无，则会自动使用可用的设备。默认为 None。
- scope（str，可选）- 模型的作用域。默认为 'mmseg'。

### 可视化预测结果

`MMSegInferencer` 有4个用于可视化预测的参数，您可以在初始化推理器时使用它们：

- show（bool）- 是否弹出窗口显示图像。默认为 False。
- wait_time（float）- 显示的间隔。默认值为 0。
- img_out_dir（str）- `out_dir` 的子目录，用于保存渲染有色分割掩膜，因此如果要保存预测掩膜，则必须定义 `out_dir`。默认为 `vis`。
- opacity（int，float）- 分割掩膜的透明度。默认值为 0.8。

这些参数的示例请参考[基本使用](#基本使用)

### 模型列表

在 MMSegmentation 中有一个非常容易列出所有模型名称的方法

```
# models 是一个模型名称列表，它们将自动打印
>>> models = MMSegInferencer.list_models('mmseg')
```

## 推理 API

### mmseg.apis.init_model

从配置文件初始化一个分割器。

参数：

- config（str，`Path` 或 `mmengine.Config`）- 配置文件路径或配置对象。

```

## 推理 API

参数：

- checkpoint（str，可选）- 权重路径。如果为 None，则模型将不会加载任何权重。
- device（str，可选）- CPU/CUDA 设备选项。默认为 'cuda:0'。
- cfg_options（dict，可选）- 用于覆盖所用配置中的某些设置的选项。

返回值：

- nn.Module：构建好的分割器。

示例：

```python
from mmseg.apis import init_model

config_path = 'configs/pspnet/pspnet_r50-d8_4xb2-40k_cityscapes-512x1024.py'
checkpoint_path = 'checkpoints/pspnet_r50-d8_512x1024_40k_cityscapes_20200605_003338-2966598c.pth'

# 初始化不带权重的模型
model = init_model(config_path)

# 初始化模型并加载权重
model = init_model(config_path, checkpoint_path)

# 在 CPU 上的初始化模型并加载权重
model = init_model(config_path, checkpoint_path, 'cpu')
```

### mmseg.apis.inference_model

使用分割器推理图像。

参数：

- model（nn.Module）- 加载的分割器
- imgs（str，np.ndarray 或 list\[str/np.ndarray\]）- 图像文件或加载的图像

返回值：

```

参数：

返回值：

- `SegDataSample` 或 list\[`SegDataSample`\]：如果 imgs 是列表或元组，则返回相同长度的列表类型结果，否则直接返回分割结果。

**注意：** [SegDataSample](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/structures/seg_data_sample.py) 是 MMSegmentation 的数据结构接口，用作不同组件之间的接口。`SegDataSample` 实现抽象数据元素 `mmengine.structures.BaseDataElement`，请参阅 [MMEngine](https://github.com/open-mmlab/mmengine) 中的数据元素[文档](https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/data_element.html)了解更多信息。

`SegDataSample` 中的参数分为几个部分：

- `gt_sem_seg`（`PixelData`）- 语义分割的标注。
- `pred_sem_seg`（`PixelData`）- 语义分割的预测。
- `seg_logits`（`PixelData`）- 模型最后一层的输出结果。

**注意：** [PixelData](https://github.com/open-mmlab/mmengine/blob/main/mmengine/structures/pixel_data.py) 是像素级标注或预测的数据结构，请参阅 [MMEngine](https://github.com/open-mmlab/mmengine) 中的 PixelData [文档](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/data_element.html)了解更多信息。

示例：

```python
from mmseg.apis import init_model, inference_model

img_path = 'demo/demo.png'

result = inference_model(model, img_path)
```

### mmseg.apis.show_result_pyplot

在图像上可视化分割结果。

参数：

- model（nn.Module）- 加载的分割器。
- img（str 或 np.ndarray）- 图像文件名或加载的图像。
- result（`SegDataSample`）- SegDataSample 预测结果。

```

参数：

- opacity（float）- 绘制分割图的不透明度。默认值为 `0.5`，必须在 `(0，1]` 范围内。
- title（str）- pyplot 图的标题。默认值为 ''。
- draw_gt（bool）- 是否绘制 GT SegDataSample。默认为 `True`。
- draw_pred（draws_pred）- 是否绘制预测 SegDataSample。默认为 `True`。
- wait_time（float）- 显示的间隔，0 是表示“无限”的特殊值。默认为 `0`。
- show（bool）- 是否展示绘制的图像。默认为 `True`。
- save_dir（str，可选）- 为所有存储后端保存的文件路径。如果为 `None`，则后端存储将不会保存任何数据。
- out_file（str，可选）- 输出文件的路径。默认为 `None`。

返回值：

- np.ndarray：通道为 RGB 的绘制图像。

示例：

```python
from mmseg.apis import init_model, inference_model, show_result_pyplot

# 从配置文件和权重文件构建模型
model = init_model(config_path, checkpoint_path, device='cuda:0')

# 推理给定图像

# 推理给定图像

# 展示分割结果
vis_image = show_result_pyplot(model, img_path, result)

# 保存可视化结果，输出图像将在 `workdirs/result.png` 路径下找到
vis_iamge = show_result_pyplot(model, img_path, result, out_file='work_dirs/result.png')

# 修改展示图像的时间，注意 0 是表示“无限”的特殊值
vis_image = show_result_pyplot(model, img_path, result, wait_time=5)
```

**注意：** 如果当前设备没有图形用户界面，建议将 `show` 设置为 `False`，并指定 `out_file` 或 `save_dir` 来保存结果。如果您想在窗口上显示结果，则不需要特殊设置。

# Projects

The OpenMMLab ecosystem can only grow through the contributions of the community.
Everyone is welcome to post their implementation of any great ideas in this folder! If you wish to start your own project, please go through the [example project](example_project/) for the best practice. For common questions about projects, please read our [faq](faq.md).

## External Projects

Here we lists some selected external projects released in the community built upon MMOCR:

- [TableMASTER-mmocr](https://github.com/JiaquanYe/TableMASTER-mmocr)
- [WordArt](https://github.com/xdxie/WordArt)

Note: The core maintainers of MMOCR only ensure the results are reproducible and the code quality meets its claim at the time each project was submitted, but they may not be responsible for future maintenance. The original authors take responsibility for maintaining their own projects.

  <img src="resources/mmdet-logo.png" width="600"/>
    <b><font size="5">OpenMMLab 官网</font></b>
    <sup>
      </a>
    </sup>
    <b><font size="5">OpenMMLab 开放平台</font></b>
    <sup>
      </a>
    </sup>
  </div>

[![PyPI](https://img.shields.io/pypi/v/mmdet)](https://pypi.org/project/mmdet)
[![docs](https://img.shields.io/badge/docs-latest-blue)](https://mmdetection.readthedocs.io/en/latest/)
[![badge](https://github.com/open-mmlab/mmdetection/workflows/build/badge.svg)](https://github.com/open-mmlab/mmdetection/actions)
[![codecov](https://codecov.io/gh/open-mmlab/mmdetection/branch/main/graph/badge.svg)](https://codecov.io/gh/open-mmlab/mmdetection)

[![license](https://img.shields.io/github/license/open-mmlab/mmdetection.svg)](https://github.com/open-mmlab/mmdetection/blob/main/LICENSE)
[![open issues](https://isitmaintained.com/badge/open/open-mmlab/mmdetection.svg)](https://github.com/open-mmlab/mmdetection/issues)
[![issue resolution](https://isitmaintained.com/badge/resolution/open-mmlab/mmdetection.svg)](https://github.com/open-mmlab/mmdetection/issues)

[📘使用文档](https://mmdetection.readthedocs.io/zh_CN/latest/) |
[🛠️安装教程](https://mmdetection.readthedocs.io/zh_CN/latest/get_started.html) |
[👀模型库](https://mmdetection.readthedocs.io/zh_CN/latest/model_zoo.html) |
[🆕更新日志](https://mmdetection.readthedocs.io/en/latest/notes/changelog.html) |
[🚀进行中的项目](https://github.com/open-mmlab/mmdetection/projects) |
[🤔报告问题](https://github.com/open-mmlab/mmdetection/issues/new/choose)

</div>

</div>

[English](README.md) | 简体中文

</div>

  <a href="https://discord.com/channels/1037617289144569886/1046608014234370059" style="text-decoration:none;">

<a href="https://space.bilibili.com/1293512903" style="text-decoration:none;">
</div>

<img src="https://github.com/open-mmlab/mmdetection/assets/17425982/6c29886f-ae7a-4a55-8be4-352ee85b7d3e"/>
</div>

## 简介

MMDetection 是一个基于 PyTorch 的目标检测开源工具箱。它是 [OpenMMLab](https://openmmlab.com/) 项目的一部分。

主分支代码目前支持 PyTorch 1.8 及其以上的版本。

<img src="https://user-images.githubusercontent.com/12907710/187674113-2074d658-f2fb-42d1-ac15-9c4a695e64d7.png"/>

<summary>主要特性</summary>

- **模块化设计**

  MMDetection 将检测框架解耦成不同的模块组件，通过组合不同的模块组件，用户可以便捷地构建自定义的检测模型

- **支持多种检测任务**

  MMDetection 支持了各种不同的检测任务，包括**目标检测**，**实例分割**，**全景分割**，以及**半监督目标检测**。

- **速度快**

  基本的框和 mask 操作都实现了 GPU 版本，训练速度比其他代码库更快或者相当，包括 [Detectron2](https://github.com/facebookresearch/detectron2), [maskrcnn-benchmark](https://github.com/facebookresearch/maskrcnn-benchmark) 和 [SimpleDet](https://github.com/TuSimple/simpledet)。

- **性能高**

  MMDetection 这个算法库源自于 COCO 2018 目标检测竞赛的冠军团队 *MMDet* 团队开发的代码，我们在之后持续进行了改进和提升。
  新发布的 [RTMDet](configs/rtmdet) 还在实时实例分割和旋转目标检测任务中取得了最先进的成果，同时也在目标检测模型中取得了最佳的的参数量和精度平衡。

除了 MMDetection 之外，我们还开源了深度学习训练库 [MMEngine](https://github.com/open-mmlab/mmengine) 和计算机视觉基础库 [MMCV](https://github.com/open-mmlab/mmcv)，它们是 MMDetection 的主要依赖。

## 最新进展

### 亮点

## 最新进展

### 亮点

我们很高兴向大家介绍我们在实时目标识别任务方面的最新成果 RTMDet，包含了一系列的全卷积单阶段检测模型。 RTMDet 不仅在从 tiny 到 extra-large 尺寸的目标检测模型上实现了最佳的参数量和精度的平衡，而且在实时实例分割和旋转目标检测任务上取得了最先进的成果。 更多细节请参阅[技术报告](https://arxiv.org/abs/2212.07784)。 预训练模型可以在[这里](configs/rtmdet)找到。

</div>

**v3.1.0** 版本已经在 2023.6.30 发布：

</div>

- 支持 Tracking 类算法，包括多目标跟踪 MOT 算法 SORT、DeepSORT、StrongSORT、OCSORT、ByteTrack、QDTrack 和视频实例分割 VIS 算法 MaskTrackRCNN、Mask2Former-VIS。
- 支持 [ViTDet](projects/ViTDet)
- 支持多模态开放检测算法 [GLIP](configs/glip) 和 [XDecoder](projects/XDecoder) 推理和评估，并同时支持了 COCO 语义分割、COCO Caption、ADE20k 通用分割、RefCOCO 等数据集。后续将支持 GLIP 微调
- 提供了包括 MMDetection 图片任务的 [gradio demo](https://github.com/open-mmlab/mmdetection/blob/dev-3.x/projects/gradio_demo/README.md)，方便用户快速体验

## 安装

请参考[快速入门文档](https://mmdetection.readthedocs.io/zh_CN/latest/get_started.html)进行安装。

## 教程

请阅读[概述](https://mmdetection.readthedocs.io/zh_CN/latest/get_started.html)对 MMDetection 进行初步的了解。

为了帮助用户更进一步了解 MMDetection，我们准备了用户指南和进阶指南，请阅读我们的[文档](https://mmdetection.readthedocs.io/zh_CN/latest/)：

- 用户指南

  <details>

- 用户指南

  <details>

  - [训练 & 测试](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/index.html#train-test)
    - [学习配置文件](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/config.html)
    - [使用已有模型在标准数据集上进行推理](https://mmdetection.readthedocs.io/en/latest/user_guides/inference.html)
    - [数据集准备](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/dataset_prepare.html)
    - [测试现有模型](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/test.html)
    - [在标准数据集上训练预定义的模型](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/train.html)
    - [在自定义数据集上进行训练](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/train.html#train-with-customized-datasets)
    - [在标准数据集上训练自定义模型](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/new_model.html)
    - [模型微调](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/finetune.html)

- [在标准数据集上训练自定义模型](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/new_model.html)
    - [提交测试结果](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/test_results_submission.html)
    - [权重初始化](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/init_cfg.html)
    - [将单阶段检测器作为 RPN](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/single_stage_as_rpn.html)
    - [半监督目标检测](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/semi_det.html)
  - [实用工具](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/index.html#useful-tools)

  </details>

- 进阶指南

  <details>

  - [基础概念](https://mmdetection.readthedocs.io/zh_CN/latest/advanced_guides/index.html#basic-concepts)
  - [组件定制](https://mmdetection.readthedocs.io/zh_CN/latest/advanced_guides/index.html#component-customization)

- [组件定制](https://mmdetection.readthedocs.io/zh_CN/latest/advanced_guides/index.html#component-customization)
  - [How to](https://mmdetection.readthedocs.io/zh_CN/latest/advanced_guides/index.html#how-to)

我们提供了检测的 colab 教程 [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](demo/MMDet_Tutorial.ipynb) 和 实例分割的 colab 教程 [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](demo/MMDet_Tutorial.ipynb)

同时，我们还提供了 [MMDetection 中文解读文案汇总](docs/zh_cn/article.md)

若需要将2.x版本的代码迁移至新版，请参考[迁移文档](https://mmdetection.readthedocs.io/en/latest/migration.html)。

## 基准测试和模型库

测试结果和模型可以在[模型库](docs/zh_cn/model_zoo.md)中找到。

  <b>算法架构</b>
</div>
  <tbody>
      <td>
        <b>Object Detection</b>
      </td>
      <td>
        <b>Instance Segmentation</b>
      </td>
      <td>
        <b>Panoptic Segmentation</b>
      </td>
      <td>
        <b>Other</b>

<b>Object Detection</b>
      </td>
      <td>
      </td>
      <td>
      </td>
      <td>
      </td>
    </tr>
      <td>
        <ul>
            <li><a href="configs/fast_rcnn">Fast R-CNN (ICCV'2015)</a></li>
            <li><a href="configs/faster_rcnn">Faster R-CNN (NeurIPS'2015)</a></li>
            <li><a href="configs/rpn">RPN (NeurIPS'2015)</a></li>
            <li><a href="configs/ssd">SSD (ECCV'2016)</a></li>
            <li><a href="configs/retinanet">RetinaNet (ICCV'2017)</a></li>
            <li><a href="configs/cascade_rcnn">Cascade R-CNN (CVPR'2018)</a></li>
            <li><a href="configs/yolo">YOLOv3 (ArXiv'2018)</a></li>
            <li><a href="configs/cornernet">CornerNet (ECCV'2018)</a></li>
            <li><a href="configs/grid_rcnn">Grid R-CNN (CVPR'2019)</a></li>

<li><a href="configs/cornernet">CornerNet (ECCV'2018)</a></li>
            <li><a href="configs/guided_anchoring">Guided Anchoring (CVPR'2019)</a></li>
            <li><a href="configs/fsaf">FSAF (CVPR'2019)</a></li>
            <li><a href="configs/centernet">CenterNet (CVPR'2019)</a></li>
            <li><a href="configs/libra_rcnn">Libra R-CNN (CVPR'2019)</a></li>
            <li><a href="configs/tridentnet">TridentNet (ICCV'2019)</a></li>
            <li><a href="configs/fcos">FCOS (ICCV'2019)</a></li>
            <li><a href="configs/reppoints">RepPoints (ICCV'2019)</a></li>
            <li><a href="configs/free_anchor">FreeAnchor (NeurIPS'2019)</a></li>
            <li><a href="configs/cascade_rpn">CascadeRPN (NeurIPS'2019)</a></li>
            <li><a href="configs/foveabox">Foveabox (TIP'2020)</a></li>
            <li><a href="configs/double_heads">Double-Head R-CNN (CVPR'2020)</a></li>

<li><a href="configs/foveabox">Foveabox (TIP'2020)</a></li>
            <li><a href="configs/atss">ATSS (CVPR'2020)</a></li>
            <li><a href="configs/nas_fcos">NAS-FCOS (CVPR'2020)</a></li>
            <li><a href="configs/centripetalnet">CentripetalNet (CVPR'2020)</a></li>
            <li><a href="configs/autoassign">AutoAssign (ArXiv'2020)</a></li>
            <li><a href="configs/sabl">Side-Aware Boundary Localization (ECCV'2020)</a></li>
            <li><a href="configs/dynamic_rcnn">Dynamic R-CNN (ECCV'2020)</a></li>
            <li><a href="configs/detr">DETR (ECCV'2020)</a></li>
            <li><a href="configs/paa">PAA (ECCV'2020)</a></li>
            <li><a href="configs/vfnet">VarifocalNet (CVPR'2021)</a></li>
            <li><a href="configs/sparse_rcnn">Sparse R-CNN (CVPR'2021)</a></li>
            <li><a href="configs/yolof">YOLOF (CVPR'2021)</a></li>

<li><a href="configs/sparse_rcnn">Sparse R-CNN (CVPR'2021)</a></li>
            <li><a href="configs/yolox">YOLOX (CVPR'2021)</a></li>
            <li><a href="configs/deformable_detr">Deformable DETR (ICLR'2021)</a></li>
            <li><a href="configs/tood">TOOD (ICCV'2021)</a></li>
            <li><a href="configs/ddod">DDOD (ACM MM'2021)</a></li>
            <li><a href="configs/rtmdet">RTMDet (ArXiv'2022)</a></li>
            <li><a href="configs/conditional_detr">Conditional DETR (ICCV'2021)</a></li>
            <li><a href="configs/dab_detr">DAB-DETR (ICLR'2022)</a></li>
            <li><a href="configs/dino">DINO (ICLR'2023)</a></li>
            <li><a href="configs/glip">GLIP (CVPR'2022)</a></li>
            <li><a href="projects/DiffusionDet">DiffusionDet (ArXiv'2023)</a></li>
            <li><a href="projects/EfficientDet">EfficientDet (CVPR'2020)</a></li>

<li><a href="projects/DiffusionDet">DiffusionDet (ArXiv'2023)</a></li>
            <li><a href="projects/Detic">Detic (ECCV'2022)</a></li>
      </ul>
      </td>
      <td>
        <ul>
          <li><a href="configs/mask_rcnn">Mask R-CNN (ICCV'2017)</a></li>
          <li><a href="configs/cascade_rcnn">Cascade Mask R-CNN (CVPR'2018)</a></li>
          <li><a href="configs/ms_rcnn">Mask Scoring R-CNN (CVPR'2019)</a></li>
          <li><a href="configs/htc">Hybrid Task Cascade (CVPR'2019)</a></li>
          <li><a href="configs/yolact">YOLACT (ICCV'2019)</a></li>
          <li><a href="configs/instaboost">InstaBoost (ICCV'2019)</a></li>
          <li><a href="configs/solo">SOLO (ECCV'2020)</a></li>
          <li><a href="configs/point_rend">PointRend (CVPR'2020)</a></li>
          <li><a href="configs/detectors">DetectoRS (ArXiv'2020)</a></li>

<li><a href="configs/point_rend">PointRend (CVPR'2020)</a></li>
          <li><a href="configs/solov2">SOLOv2 (NeurIPS'2020)</a></li>
          <li><a href="configs/scnet">SCNet (AAAI'2021)</a></li>
          <li><a href="configs/queryinst">QueryInst (ICCV'2021)</a></li>
          <li><a href="configs/mask2former">Mask2Former (ArXiv'2021)</a></li>
          <li><a href="configs/condinst">CondInst (ECCV'2020)</a></li>
          <li><a href="projects/SparseInst">SparseInst (CVPR'2022)</a></li>
          <li><a href="configs/rtmdet">RTMDet (ArXiv'2022)</a></li>
          <li><a href="configs/boxinst">BoxInst (CVPR'2021)</a></li>
        </ul>
      </td>
      <td>
        <ul>
          <li><a href="configs/panoptic_fpn">Panoptic FPN (CVPR'2019)</a></li>
          <li><a href="configs/maskformer">MaskFormer (NeurIPS'2021)</a></li>

</td>
      <td>
        <ul>
        </ul>
      </td>
      <td>
        </ul>
          <li><b>Contrastive Learning</b></li>
        <ul>
        <ul>
          <li><a href="configs/selfsup_pretrain">SwAV (NeurIPS'2020)</a></li>
          <li><a href="configs/selfsup_pretrain">MoCo (CVPR'2020)</a></li>
          <li><a href="configs/selfsup_pretrain">MoCov2 (ArXiv'2020)</a></li>
        </ul>
        </ul>
        </ul>
          <li><b>Distillation</b></li>
        <ul>
        <ul>
          <li><a href="configs/ld">Localization Distillation (CVPR'2022)</a></li>
          <li><a href="configs/lad">Label Assignment Distillation (WACV'2022)</a></li>
        </ul>
        </ul>
          <li><b>Semi-Supervised Object Detection</b></li>

<li><a href="configs/lad">Label Assignment Distillation (WACV'2022)</a></li>
        </ul>
        </ul>
        <ul>
        <ul>
          <li><a href="configs/soft_teacher">Soft Teacher (ICCV'2021)</a></li>
        </ul>
        </ul>
      </ul>
      </td>
    </tr>
</td>
    </tr>
  </tbody>
</table>

  <b>模块组件</b>
</div>
  <tbody>
      <td>
        <b>Backbones</b>
      </td>
      <td>
        <b>Necks</b>
      </td>
      <td>
        <b>Loss</b>
      </td>
      <td>
        <b>Common</b>
      </td>
    </tr>
      <td>
      <ul>
        <li>VGG (ICLR'2015)</li>
        <li>ResNet (CVPR'2016)</li>
        <li>ResNeXt (CVPR'2017)</li>
        <li>MobileNetV2 (CVPR'2018)</li>
        <li><a href="configs/hrnet">HRNet (CVPR'2019)</a></li>

<li>ResNet (CVPR'2016)</li>
        <li><a href="configs/empirical_attention">Generalized Attention (ICCV'2019)</a></li>
        <li><a href="configs/gcnet">GCNet (ICCVW'2019)</a></li>
        <li><a href="configs/res2net">Res2Net (TPAMI'2020)</a></li>
        <li><a href="configs/regnet">RegNet (CVPR'2020)</a></li>
        <li><a href="configs/resnest">ResNeSt (ArXiv'2020)</a></li>
        <li><a href="configs/pvt">PVT (ICCV'2021)</a></li>
        <li><a href="configs/swin">Swin (CVPR'2021)</a></li>
        <li><a href="configs/pvt">PVTv2 (ArXiv'2021)</a></li>
        <li><a href="configs/resnet_strikes_back">ResNet strikes back (ArXiv'2021)</a></li>
        <li><a href="configs/efficientnet">EfficientNet (ArXiv'2021)</a></li>
        <li><a href="configs/convnext">ConvNeXt (CVPR'2022)</a></li>

<li><a href="configs/efficientnet">EfficientNet (ArXiv'2021)</a></li>
        <li><a href="projects/ConvNeXt-V2">ConvNeXtv2 (ArXiv'2023)</a></li>
      </ul>
      </td>
      <td>
      <ul>
        <li><a href="configs/pafpn">PAFPN (CVPR'2018)</a></li>
        <li><a href="configs/nas_fpn">NAS-FPN (CVPR'2019)</a></li>
        <li><a href="configs/carafe">CARAFE (ICCV'2019)</a></li>
        <li><a href="configs/fpg">FPG (ArXiv'2020)</a></li>
        <li><a href="configs/groie">GRoIE (ICPR'2020)</a></li>
        <li><a href="configs/dyhead">DyHead (CVPR'2021)</a></li>
      </ul>
      </td>
      <td>
        <ul>
          <li><a href="configs/ghm">GHM (AAAI'2019)</a></li>
          <li><a href="configs/gfl">Generalized Focal Loss (NeurIPS'2020)</a></li>
          <li><a href="configs/seesaw_loss">Seasaw Loss (CVPR'2021)</a></li>
        </ul>
      </td>
      <td>
        <ul>

<li><a href="configs/seesaw_loss">Seasaw Loss (CVPR'2021)</a></li>
        </ul>
      </td>
      <td>
        <ul>
          <li><a href="configs/faster_rcnn/faster_rcnn_r50_fpn_ohem_1x_coco.py">OHEM (CVPR'2016)</a></li>
          <li><a href="configs/gn">Group Normalization (ECCV'2018)</a></li>
          <li><a href="configs/dcn">DCN (ICCV'2017)</a></li>
          <li><a href="configs/dcnv2">DCNv2 (CVPR'2019)</a></li>
          <li><a href="configs/gn+ws">Weight Standardization (ArXiv'2019)</a></li>
          <li><a href="configs/pisa">Prime Sample Attention (CVPR'2020)</a></li>
          <li><a href="configs/strong_baselines">Strong Baselines (CVPR'2021)</a></li>
          <li><a href="configs/resnet_strikes_back">Resnet strikes back (ArXiv'2021)</a></li>
        </ul>
      </td>
    </tr>
</td>
    </tr>
  </tbody>
</table>

我们在[基于 MMDetection 的项目](./docs/zh_cn/notes/projects.md)中列举了一些其他的支持的算法。

## 常见问题

请参考 [FAQ](docs/zh_cn/notes/faq.md) 了解其他用户的常见问题。

## 贡献指南

</td>
    </tr>
</td>
    </tr>
  </tbody>
</table>

## 常见问题

## 贡献指南

我们感谢所有的贡献者为改进和提升 MMDetection 所作出的努力。我们将正在进行中的项目添加进了[GitHub Projects](https://github.com/open-mmlab/mmdetection/projects)页面，非常欢迎社区用户能参与进这些项目中来。请参考[贡献指南](.github/CONTRIBUTING.md)来了解参与项目贡献的相关指引。

## 致谢

MMDetection 是一款由来自不同高校和企业的研发人员共同参与贡献的开源项目。我们感谢所有为项目提供算法复现和新功能支持的贡献者，以及提供宝贵反馈的用户。 我们希望这个工具箱和基准测试可以为社区提供灵活的代码工具，供用户复现已有算法并开发自己的新模型，从而不断为开源社区提供贡献。

## 引用

如果你在研究中使用了本项目的代码或者性能基准，请参考如下 bibtex 引用 MMDetection。

```
@article{mmdetection,
  title   = {{MMDetection}: Open MMLab Detection Toolbox and Benchmark},
  author  = {Chen, Kai and Wang, Jiaqi and Pang, Jiangmiao and Cao, Yuhang and
             Xiong, Yu and Li, Xiaoxiao and Sun, Shuyang and Feng, Wansen and
             Liu, Ziwei and Xu, Jiarui and Zhang, Zheng and Cheng, Dazhi and

Xiong, Yu and Li, Xiaoxiao and Sun, Shuyang and Feng, Wansen and
             Zhu, Chenchen and Cheng, Tianheng and Zhao, Qijie and Li, Buyu and
             Lu, Xin and Zhu, Rui and Wu, Yue and Dai, Jifeng and Wang, Jingdong
             and Shi, Jianping and Ouyang, Wanli and Loy, Chen Change and Lin, Dahua},
  journal= {arXiv preprint arXiv:1906.07155},
}
```

## 开源许可证

该项目采用 [Apache 2.0 开源许可证](LICENSE)。

## OpenMMLab 的其他项目

- [MMEngine](https://github.com/open-mmlab/mmengine): OpenMMLab 深度学习模型训练基础库
- [MMCV](https://github.com/open-mmlab/mmcv): OpenMMLab 计算机视觉基础库
- [MMPreTrain](https://github.com/open-mmlab/mmpretrain): OpenMMLab 深度学习预训练工具箱
- [MMagic](https://github.com/open-mmlab/mmagic): OpenMMLab 新一代人工智能内容生成（AIGC）工具箱
- [MMDetection](https://github.com/open-mmlab/mmdetection): OpenMMLab 目标检测工具箱

- [MMDetection3D](https://github.com/open-mmlab/mmdetection3d): OpenMMLab 新一代通用 3D 目标检测平台
- [MMRotate](https://github.com/open-mmlab/mmrotate): OpenMMLab 旋转框检测工具箱与测试基准
- [MMYOLO](https://github.com/open-mmlab/mmyolo): OpenMMLab YOLO 系列工具箱与测试基准
- [MMSegmentation](https://github.com/open-mmlab/mmsegmentation): OpenMMLab 语义分割工具箱
- [MMOCR](https://github.com/open-mmlab/mmocr): OpenMMLab 全流程文字检测识别理解工具包
- [MMPose](https://github.com/open-mmlab/mmpose): OpenMMLab 姿态估计工具箱
- [MMHuman3D](https://github.com/open-mmlab/mmhuman3d): OpenMMLab 人体参数化模型工具箱与测试基准
- [MMSelfSup](https://github.com/open-mmlab/mmselfsup): OpenMMLab 自监督学习工具箱与测试基准
- [MMRazor](https://github.com/open-mmlab/mmrazor): OpenMMLab 模型压缩工具箱与测试基准
- [MMFewShot](https://github.com/open-mmlab/mmfewshot): OpenMMLab 少样本学习工具箱与测试基准

- [MMAction2](https://github.com/open-mmlab/mmaction2): OpenMMLab 新一代视频理解工具箱
- [MMTracking](https://github.com/open-mmlab/mmtracking): OpenMMLab 一体化视频目标感知平台
- [MMFlow](https://github.com/open-mmlab/mmflow): OpenMMLab 光流估计工具箱与测试基准
- [MMEditing](https://github.com/open-mmlab/mmediting): OpenMMLab 图像视频编辑工具箱
- [MMGeneration](https://github.com/open-mmlab/mmgeneration): OpenMMLab 图片视频生成模型工具箱
- [MMDeploy](https://github.com/open-mmlab/mmdeploy): OpenMMLab 模型部署框架
- [MIM](https://github.com/open-mmlab/mim): OpenMMlab 项目、算法、模型的统一入口
- [MMEval](https://github.com/open-mmlab/mmeval): 统一开放的跨框架算法评测库
- [Playground](https://github.com/open-mmlab/playground): 收集和展示 OpenMMLab 相关的前沿、有趣的社区项目

## 欢迎加入 OpenMMLab 社区

扫描下方的二维码可关注 OpenMMLab 团队的 [知乎官方账号](https://www.zhihu.com/people/openmmlab)，加入 OpenMMLab 团队的 [官方交流 QQ 群](https://jq.qq.com/?_wv=1027&k=aCvMxdr3)

<img src="resources/zhihu_qrcode.jpg" height="400" />  <img src="resources/qq_group_qrcode.jpg" height="400" />
</div>

我们会在 OpenMMLab 社区为大家

- 📢 分享 AI 框架的前沿核心技术
- 💻 解读 PyTorch 常用模块源码
- 📰 发布 OpenMMLab 的相关新闻
- 🚀 介绍 OpenMMLab 开发的前沿算法
- 🏃 获取更高效的问题答疑和意见反馈
- 🔥 提供与各行各业开发者充分交流的平台

干货满满 📘，等你来撩 💗，OpenMMLab 社区期待您的加入 👬

### broadcasting rule

ncnn BinaryOp accepts blobs with different shape

C = BinaryOp(A, B)

shape notation convention is [w], [w,h], [w,h,c], [w,h,d,c]

* binaryop with scalar and scalar-like

|A|B|C|
|---|---|---|
|[2]|scalar / [1]|[2]|
|[2,3]|scalar / [1] / [1,1]|[2,3]|
|[2,3,4]|scalar / [1] / [1,1] / [1,1,1]|[2,3,4]|
|[2,3,4,5]|scalar / [1] / [1,1] / [1,1,1] / [1,1,1,1]|[2,3,4,5]|

* no broadcast

|A|B|C|
|[2]|[2]|[2]|
|[2,3]|[2,3]|[2,3]|
|[2,3,4]|[2,3,4]|[2,3,4]|
|[2,3,4,5]|[2,3,4,5]|[2,3,4,5]|

* explicit broadcast B

|A|B|C|
|[2,3]|[1,3]|[2,3]|
|[2,3]|[2,1]|[2,3]|
|[2,3,4]|[1,3,4]|[2,3,4]|
|[2,3,4]|[2,1,4]|[2,3,4]|
|[2,3,4]|[2,3,1]|[2,3,4]|
|[2,3,4]|[1,1,4]|[2,3,4]|
|[2,3,4]|[1,3,1]|[2,3,4]|
|[2,3,4]|[2,1,1]|[2,3,4]|
|[2,3,4,5]|[1,3,4,5]|[2,3,4,5]|
|[2,3,4,5]|[2,1,4,5]|[2,3,4,5]|
|[2,3,4,5]|[2,3,1,5]|[2,3,4,5]|
|[2,3,4,5]|[2,3,4,1]|[2,3,4,5]|
|[2,3,4,5]|[1,1,4,5]|[2,3,4,5]|
|[2,3,4,5]|[1,3,1,5]|[2,3,4,5]|
|[2,3,4,5]|[1,3,4,1]|[2,3,4,5]|

|[2,3,4,5]|[2,1,1,5]|[2,3,4,5]|
|[2,3,4,5]|[2,1,4,1]|[2,3,4,5]|
|[2,3,4,5]|[2,3,1,1]|[2,3,4,5]|
|[2,3,4,5]|[1,1,1,5]|[2,3,4,5]|
|[2,3,4,5]|[1,1,4,1]|[2,3,4,5]|
|[2,3,4,5]|[1,3,1,1]|[2,3,4,5]|
|[2,3,4,5]|[2,1,1,1]|[2,3,4,5]|

* implicit broadcast B for inner axis

It broadcasts in the opposite direction of the numpy's implicit broadcasting behavior.

pnnx will insert reshape operator at the appropriate position to convert it to explicit broadcast automatically.

|A|B|C|
|[2,3]|[3]|[2,3]|
|[2,3,4]|[4]|[2,3,4]|
|[2,3,4]|[3,4]|[2,3,4]|
|[2,3,4,5]|[5]|[2,3,4,5]|
|[2,3,4,5]|[4,5]|[2,3,4,5]|
|[2,3,4,5]|[3,4,5]|[2,3,4,5]|

* implicit broadcast B with 1 dimension rank for outer axis

This exists only for compatibility.

When the size is the same, eg. [2,2] and [2], broadcast B for inner axis will be prioritized.

|A|B|C|
|[2,3]|[2]|[2,3]|
|[2,3,4]|[2]|[2,3,4]|
|[2,3,4,5]|[2]|[2,3,4,5]|

# DDRNet

> [Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes](http://arxiv.org/abs/2101.06085)

<a href="https://github.com/ydhongHIT/DDRNet">Official Repo</a>

Semantic segmentation is a key technology for autonomous vehicles to understand the surrounding scenes. The appealing performances of contemporary models usually come at the expense of heavy computations and lengthy inference time, which is intolerable for self-driving. Using light-weight architectures (encoder-decoder or two-pathway) or reasoning on low-resolution images, recent methods realize very fast scene parsing, even running at more than 100 FPS on a single 1080Ti GPU. However, there is still a signiﬁcant gap in performance between these real-time methods and the models based on dilation backbones. To tackle this problem, we proposed a family of efﬁcient backbones specially designed for real-time semantic segmentation. The proposed deep dual-resolution networks (DDRNets) are composed of two deep branches between which multiple bilateral fusions are performed. Additionally, we design a new contextual information extractor named Deep Aggregation Pyramid Pooling Module (DAPPM) to enlarge effective receptive ﬁelds and fuse multi-scale context based on low-resolution feature maps. Our method achieves a new state-of-the-art trade-off between accuracy and speed on both Cityscapes and CamVid dataset. In particular, on a single 2080Ti GPU, DDRNet-23-slim yields 77.4% mIoU at 102 FPS on Cityscapes test set and 74.7% mIoU at 230 FPS on CamVid test set. With widely used test augmentation, our method is superior to most state-of-the-art models and requires much less computation. Codes and trained models are available online.

<img src="https://raw.githubusercontent.com/ydhongHIT/DDRNet/main/figs/DDRNet_seg.png" width="80%"/>
</div>

| Method | Backbone      | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device | mIoU  | mIoU(ms+flip) | config                                                                                                                                    | download                                                                                                                                                                                                                                                                                                                                                                                                             |

| DDRNet | DDRNet23-slim | 1024x1024 | 120000  | 1.70     | 85.85          | A100   | 77.84 | 80.15         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ddrnet/ddrnet_23-slim_in1k-pre_2xb6-120k_cityscapes-1024x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ddrnet/ddrnet_23-slim_in1k-pre_2xb6-120k_cityscapes-1024x1024/ddrnet_23-slim_in1k-pre_2xb6-120k_cityscapes-1024x1024_20230426_145312-6a5e5174.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ddrnet/ddrnet_23-slim_in1k-pre_2xb6-120k_cityscapes-1024x1024/ddrnet_23-slim_in1k-pre_2xb6-120k_cityscapes-1024x1024_20230426_145312.json) |

| DDRNet | DDRNet23      | 1024x1024 | 120000  | 7.26     | 33.41          | A100   | 79.99 | 81.71         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ddrnet/ddrnet_23_in1k-pre_2xb6-120k_cityscapes-1024x1024.py)      | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ddrnet/ddrnet_23_in1k-pre_2xb6-120k_cityscapes-1024x1024/ddrnet_23_in1k-pre_2xb6-120k_cityscapes-1024x1024_20230425_162633-81601db0.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ddrnet/ddrnet_23_in1k-pre_2xb6-120k_cityscapes-1024x1024/ddrnet_23_in1k-pre_2xb6-120k_cityscapes-1024x1024_20230425_162633.json)                     |

## Notes

The pretrained weights in config files are converted from [the official repo](https://github.com/ydhongHIT/DDRNet#pretrained-models).

```bibtex
@article{pan2022deep,
  title={Deep Dual-Resolution Networks for Real-Time and Accurate Semantic Segmentation of Traffic Scenes},

```bibtex
  author={Pan, Huihui and Hong, Yuanduo and Sun, Weichao and Jia, Yisong},
  journal={IEEE Transactions on Intelligent Transportation Systems},
}
```

# 数据集类

## 概览

在 MMOCR 中，所有的数据集都通过不同的基于 [mmengine.BaseDataset](mmengine.dataset.BaseDataset) 的 Dataset 类进行处理。 Dataset 类负责加载数据并进行初始解析，然后将其馈送到 [数据流水线](./transforms.md) 进行数据预处理、增强、格式化等操作。

![Flowchart](https://user-images.githubusercontent.com/22607038/225302639-c65888c5-cda0-4df7-ac79-1348fe16d44a.png)

</div>

在本教程中，我们将介绍 Dataset 类的一些常见接口，以及 MMOCR 中 Dataset 实现的使用以及它们支持的注释类型。

```{tip}
Dataset 类支持一些高级功能，例如懒加载、数据序列化、利用各种数据集包装器执行数据连接、重复和类别平衡。这些内容将不在本教程中介绍，但您可以阅读 {external+mmengine:doc}`MMEngine: BaseDataset <advanced_tutorials/basedataset>` 了解更多详细信息。
```

## 常见接口

现在，让我们看一个具体的示例并学习 Dataset 类的一些典型接口。`OCRDataset` 是 MMOCR 中默认使用的 Dataset 实现，因为它的标注格式足够灵活，支持 *所有* OCR 任务（详见 [OCRDataset](#ocrdataset)）。现在我们将实例化一个 `OCRDataset` 对象，其中将加载 `tests/data/det_toy_dataset` 中的玩具数据集。

```python
from mmocr.datasets import OCRDataset
from mmengine.registry import init_default_scope
init_default_scope('mmocr')

    dict(
        type='LoadImageFromFile'),
    dict(

```python

    dict(
    dict(
        type='LoadOCRAnnotations',
        with_polygon=True,
    ),
    dict(type='RandomCrop', min_side_ratio=0.1),
    dict(type='Resize', scale=(640, 640), keep_ratio=True),
    dict(type='Pad', size=(640, 640)),
    dict(
        type='PackTextDetInputs',
        meta_keys=('img_path', 'ori_shape', 'img_shape'))
]
dataset = OCRDataset(
    data_root='tests/data/det_toy_dataset',
    ann_file='textdet_test.json',
    pipeline=train_pipeline)

```

让我们查看一下这个数据集的大小：

```python
>>> print(len(dataset))

10
```

通常，Dataset 类加载并存储两种类型的信息：（1）**元信息**：储存数据集的属性，例如此数据集中可用的对象类别。 （2）**标注**：图像的路径及其标签。我们可以通过 `dataset.metainfo` 访问元信息：

```python
>>> from pprint import pprint
>>> pprint(dataset.metainfo)

10
```

```python

{'category': [{'id': 0, 'name': 'text'}],
 'dataset_type': 'TextDetDataset',
 'task_name': 'textdet'}
```

对于标注，我们可以通过 `dataset.get_data_info(idx)` 访问它。该方法返回一个字典，其中包含数据集中第 `idx` 个样本的信息。该样本已经经过初步解析，但尚未由 [数据流水线](./transforms.md) 处理。

```python
>>> pprint(dataset.get_data_info(0))

{'height': 720,
 'img_path': 'tests/data/det_toy_dataset/test/img_10.jpg',
 'instances': [{'bbox': [260.0, 138.0, 284.0, 158.0],
                'bbox_label': 0,
                'ignore': True,
                'polygon': [261, 138, 284, 140, 279, 158, 260, 158]},
                ...,
               {'bbox': [1011.0, 157.0, 1079.0, 173.0],
                'polygon': [1011, 157, 1079, 160, 1076, 173, 1011, 170]}],

{'bbox': [1011.0, 157.0, 1079.0, 173.0],
 'sample_idx': 0,
 'seg_map': 'test/gt_img_10.txt',
 'width': 1280}
```

另一方面，我们可以通过 `dataset[idx]` 或 `dataset.__getitem__(idx)` 获取由数据流水线完整处理过后的样本，该样本可以直接馈入模型并执行完整的训练/测试循环。它有两个字段：

- `inputs`：经过数据增强后的图像；
- `data_samples`：包含经过数据增强后的标注和元信息的 [DataSample](./structures.md)，这些元信息可能由一些数据变换产生，并用以记录该样本的某些关键属性。

```python
>>> pprint(dataset[0])

{'data_samples': <TextDetDataSample(

    META INFORMATION
    ori_shape: (720, 1280)
    img_path: 'tests/data/det_toy_dataset/imgs/test/img_10.jpg'
    img_shape: (640, 640)

    DATA FIELDS
    gt_instances: <InstanceData(

            META INFORMATION

            DATA FIELDS
            labels: tensor([0, 0, 0])
            polygons: [array([207.33984 , 104.65409 , 208.34634 ,  84.528305, 231.49594 ,

META INFORMATION

                        86.54088 , 226.46341 , 104.65409 , 207.33984 , 104.65409 ],
                      dtype=float32), array([237.53496 , 103.6478  , 235.52196 ,  84.528305, 365.36096 ,
                        86.54088 , 364.35446 , 107.67296 , 237.53496 , 103.6478  ],
                      dtype=float32), array([105.68293, 166.03773, 105.68293, 151.94969, 177.14471, 150.94339,
                       178.15121, 165.03145, 105.68293, 166.03773], dtype=float32)]
            ignored: tensor([ True, False,  True])
            bboxes: tensor([[207.3398,  84.5283, 231.4959, 104.6541],
                        [235.5220,  84.5283, 365.3610, 107.6730],
                        [105.6829, 150.9434, 178.1512, 166.0377]])
        ) at 0x7f7359f04fa0>
) at 0x7f735a0508e0>,

[235.5220,  84.5283, 365.3610, 107.6730],
 'inputs': tensor([[[129, 111, 131,  ...,   0,   0,   0], ...
                  [ 19,  18,  15,  ...,   0,   0,   0]]], dtype=torch.uint8)}
```

## 数据集类及标注格式

每个数据集实现只能加载特定格式的数据集。这里列出了所有支持的数据集类及其兼容的格式，以及一个示例配置，以演示如何在实践中使用它们。

```{note}
如果您不熟悉配置系统，可以阅读 [数据集配置文件](../user_guides/dataset_prepare.md#数据集配置文件)。
```

### OCRDataset

通常，OCR 数据集中有许多不同类型的标注，在不同的子任务（如文本检测和文本识别）中，格式也经常会有所不同。这些差异可能会导致在使用不同数据集时需要不同的数据加载代码，增加了用户的学习和维护成本。

在 MMOCR 中，我们提出了一种统一的数据集格式，可以适应 OCR 的所有三个子任务：文本检测、文本识别和端到端 OCR。这种设计最大程度地提高了数据集的一致性，允许在不同任务之间重复使用数据标注，也使得数据集管理更加方便。考虑到流行的数据集格式并不一致，MMOCR 提供了 [Dataset Preparer](../user_guides/data_prepare/dataset_preparer.md) 来帮助用户将其数据集转换为 MMOCR 格式。我们也十分鼓励研究人员基于此数据格式开发自己的数据集。

#### 标注格式

#### 标注格式

此标注文件是一个 `.json` 文件，存储一个包含 `metainfo` 和 `data_list` 的 `dict`，前者包括有关数据集的基本信息，后者由每个图片的标注组成。这里呈现了标注文件中的所有字段的列表，但其中某些字段仅会在特定任务中被用到。

```python
{
    "metainfo":
    {
      "dataset_type": "TextDetDataset",  # 可选项: TextDetDataset/TextRecogDataset/TextSpotterDataset
      "task_name": "textdet",  #  可选项: textdet/textspotter/textrecog
      "category": [{"id": 0, "name": "text"}]  # 在 textdet/textspotter 里用到
    },
    "data_list":
    [
      {
        "img_path": "test_img.jpg",
        "height": 604,
        "width": 640,
        "instances":  # 一图内的多个实例
        [
          {
            "bbox": [0, 0, 10, 20],  # textdet/textspotter 内用到, [x1, y1, x2, y2]。
            "bbox_label": 0,  # 对象类别, 在 MMOCR 中恒为 0 (文本)
            "polygon": [0, 0, 0, 10, 10, 20, 20, 0], # textdet/textspotter 内用到。 [x1, y1, x2, y2, ....]
            "text": "mmocr",  # textspotter/textrecog 内用到
            "ignore": False # textspotter/textdet 内用到，决定是否在训练时忽略该实例
          },
          #...
        ],

"text": "mmocr",  # textspotter/textrecog 内用到
          },
          #...
        ],
      }
      #... 多图片
    ]
}
```

#### 示例配置

以下是配置的一部分，我们在 `train_dataloader` 中使用 `OCRDataset` 加载用于文本检测模型的 ICDAR2015 数据集。请注意，`OCRDataset` 可以加载由 Dataset Preparer 准备的任何 OCR 数据集。也就是说，您可以将其用于文本识别和文本检测，但您仍然需要根据不同任务的需求修改 `pipeline` 中的数据变换。

```python
pipeline = [
    dict(
    dict(
    ),
    dict(
]

icdar2015_textdet_train = dict(
    data_root='data/icdar2015',
    ann_file='textdet_train.json',
    filter_cfg=dict(filter_empty_gt=True, min_size=32),
    pipeline=pipeline)

filter_cfg=dict(filter_empty_gt=True, min_size=32),

```

### RecogLMDBDataset

当数据量非常大时，从文件中读取图像或标签可能会很慢。此外，在学术界，大多数场景文本识别数据集的图像和标签都以 lmdb 格式存储。（[示例](https://github.com/clovaai/deep-text-recognition-benchmark)）

为了更接近主流实践并提高数据存储效率，MMOCR支持通过 `RecogLMDBDataset` 从 lmdb 数据集加载图像和标签。

#### 标注格式

MMOCR 会读取 lmdb 数据集中的以下键：

- `num_samples`：描述数据集的数据量的参数。
- 图像和标签的键分别以 `image-000000001` 和 `label-000000001` 的格式命名，索引从1开始。

MMOCR 在 `tests/data/rec_toy_dataset/imgs.lmdb` 中提供了一个 toy lmdb 数据集。您可以使用以下代码片段了解其格式。

```python
>>> import lmdb
>>>
>>> env = lmdb.open('tests/data/rec_toy_dataset/imgs.lmdb')
>>> txn = env.begin()
>>> for k, v in txn.cursor():
>>>     print(k, v)

b'image-000000001' b'\xff...'
b'image-000000002' b'\xff...'
b'image-000000003' b'\xff...'
b'image-000000004' b'\xff...'

b'image-000000005' b'\xff...'
b'image-000000006' b'\xff...'
b'image-000000007' b'\xff...'
b'image-000000008' b'\xff...'
b'image-000000009' b'\xff...'
b'image-000000010' b'\xff...'
b'label-000000001' b'GRAND'
b'label-000000002' b'HOTEL'
b'label-000000003' b'HOTEL'
b'label-000000004' b'PACIFIC'
b'label-000000005' b'03/09/2009'
b'label-000000006' b'ANING'
b'label-000000007' b'Virgin'
b'label-000000008' b'america'
b'label-000000009' b'ATTACK'
b'label-000000010' b'DAVIDSON'
b'num-samples' b'10'

```

#### 示例配置

以下是示例配置的一部分，我们在其中使用 `RecogLMDBDataset` 加载 toy 数据集。由于 `RecogLMDBDataset` 会将图像加载为 numpy 数组，因此如果要在数据管道中成功加载图像，应该记得把`LoadImageFromFile` 替换成 `LoadImageFromNDArray` 。

```python
    dict(
        type='LoadImageFromNDArray'),
    dict(

```python
    dict(
    dict(
        with_text=True,
    ),
    dict(
]

toy_textrecog_train = dict(
    type='RecogLMDBDataset',
    data_root='tests/data/rec_toy_dataset/',
    ann_file='imgs.lmdb',

    dataset=toy_textrecog_train)
```

### RecogTextDataset

在 MMOCR 1.0 之前，MMOCR 0.x 的文本识别任务的输入是文本文件。这些格式已在 MMOCR 1.0 中弃用，这个类随时可能被删除。[更多信息](../migration/dataset.md)

#### 标注格式

文本文件可以是 `txt` 格式或 `jsonl` 格式。简单的 `.txt` 标注通过空格将图像名称和词语标注分隔开，因此这种格式并无法处理文本实例中包含空格的情况。

```text
img1.jpg OpenMMLab
img2.jpg MMOCR
```

`jsonl` 格式使用类似字典的结构来表示标注，其中键 `filename` 和 `text` 存储图像名称和单词标签。

```json
{"filename": "img1.jpg", "text": "OpenMMLab"}

```text
```

```json
{"filename": "img2.jpg", "text": "MMOCR"}
```

#### 示例配置

以下是一个示例配置，我们在训练中使用 `RecogTextDataset` 加载 txt 标签，而在测试中使用 jsonl 标签。

```python
    dict(
    dict(
    ),
    dict(
]

 # loading 0.x txt format annos
 txt_dataset = dict(
     type='RecogTextDataset',
     data_root=data_root,
     ann_file='old_label.txt',
     data_prefix=dict(img_path='imgs'),
     parser_cfg=dict(
         type='LineStrParser',
         keys=['filename', 'text'],
         keys_idx=[0, 1]),
     pipeline=pipeline)

type='LineStrParser',

    dataset=txt_dataset)

 # loading 0.x json line format annos
 jsonl_dataset = dict(
     ann_file='old_label.jsonl',
         type='LineJsonParser',
     pipeline=pipeline))

    dataset=jsonl_dataset)
```

### IcdarDataset

在 MMOCR 1.0 之前，MMOCR 0.x 的文本检测输入采用了类似 COCO 格式的注释。这些格式已在 MMOCR 1.0 中弃用，这个类在将来的任何时候都可能被删除。[更多信息](../migration/dataset.md)

#### 标注格式

```json
{
  "images": [
    {
      "id": 1,
      "width": 800,

#### 标注格式

```json
{
    {
      "id": 1,
      "height": 600,
      "file_name": "test.jpg"
    }
  ],
  "annotations": [
    {
      "id": 1,
      "image_id": 1,
      "category_id": 1,
      "bbox": [0,0,10,10],
      "segmentation": [
          [0,0,10,0,10,10,0,10]
      ],
      "area": 100,
      "iscrowd": 0
    }
  ]
}

```

#### 配置示例

这是配置示例的一部分，其中我们令 `train_dataloader` 使用 `IcdarDataset` 来加载旧标签。

```python
    dict(
    dict(
    ),
    dict(
]

    type='IcdarDatasetDataset',
    data_root='data/det/icdar2015',

type='PackTextDetInputs',
]

    ann_file='instances_training.json',

```

### WildReceiptDataset

该类为 [WildReceipt](https://mmocr.readthedocs.io/en/dev-1.x/user_guides/data_prepare/datasetzoo.html#wildreceipt) 数据集定制。

#### 标注格式

```json
// Close Set
{
  "file_name": "image_files/Image_16/11/d5de7f2a20751e50b84c747c17a24cd98bed3554.jpeg",
  "height": 1200,
  "width": 1600,
  "annotations":
    [
      {
        "box": [550.0, 190.0, 937.0, 190.0, 937.0, 104.0, 550.0, 104.0],
        "text": "SAFEWAY",
        "label": 1
      },
      {

"width": 1600,
    [
      {
        "label": 1
      },
      {
        "box": [1048.0, 211.0, 1074.0, 211.0, 1074.0, 196.0, 1048.0, 196.0],
        "text": "TM",
        "label": 25
      }
    ], //...
}

// Open Set
{
  "file_name": "image_files/Image_12/10/845be0dd6f5b04866a2042abd28d558032ef2576.jpeg",
  "height": 348,
  "width": 348,
    [
      {
        "box": [114.0, 19.0, 230.0, 19.0, 230.0, 1.0, 114.0, 1.0],
        "text": "CHOEUN",
        "label": 2,
        "edge": 1
      },
      {
        "box": [97.0, 35.0, 236.0, 35.0, 236.0, 19.0, 97.0, 19.0],
        "text": "KOREANRESTAURANT",
        "edge": 1
      }
    ]
}
```

#### 配置示例

请参考 [SDMGR 的配置](https://github.com/open-mmlab/mmocr/blob/f30c16ce96bd2393570c04eeb9cf48a7916315cc/configs/kie/sdmgr/sdmgr_novisual_60e_wildreceipt.py)。

# SNPE feature support

Currently mmdeploy integrates the onnx2dlc model conversion and SDK inference, but the following features are not yet supported:

- GPU_FP16 mode
- DSP/AIP quantization
- Operator internal profiling
- UDO operator

# HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_CVPR_2020/html/Cheng_HigherHRNet_Scale-Aware_Representation_Learning_for_Bottom-Up_Human_Pose_Estimation_CVPR_2020_paper.html">HigherHRNet (CVPR'2020)</a></summary>

```bibtex
@inproceedings{cheng2020higherhrnet,
  title={HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation},
  author={Cheng, Bowen and Xiao, Bin and Wang, Jingdong and Shi, Honghui and Huang, Thomas S and Zhang, Lei},
  pages={5386--5395},
}
```

Bottom-up human pose estimation methods have difficulties in predicting the correct pose for small persons due to challenges in scale variation. In this paper, we present HigherHRNet: a novel bottom-up human pose estimation method for learning scale-aware representations using high-resolution feature pyramids. Equipped with multi-resolution supervision for training and multi-resolution aggregation for inference, the proposed approach is able to solve the scale variation challenge in bottom-up multi-person pose estimation and localize keypoints more precisely, especially for small person. The feature pyramid in HigherHRNet consists of feature map outputs from HRNet and upsampled higher-resolution outputs through a transposed convolution. HigherHRNet outperforms the previous best bottom-up method by 2.5% AP for medium person on COCO test-dev, showing its effectiveness in handling scale variation. Furthermore, HigherHRNet achieves new state-of-the-art result on COCO test-dev (70.5% AP) without using refinement or other post-processing techniques, surpassing all existing bottom-up methods. HigherHRNet even surpasses all top-down methods on CrowdPose test (67.6% AP), suggesting its robustness in crowded scene.

<img src="https://user-images.githubusercontent.com/15977946/146515482-975050d6-57da-469a-8dda-201675a404e7.png">
</div>

# 实用工具

本工具允许你在不启动完整训练流程的情况下，直接查看生成的 prompt。如果传入的配置仅为数据集配置（如 `configs/datasets/nq/nq_gen_3dcea1.py`），则展示数据集配置中定义的原始 prompt。若为完整的评测配置（包含模型和数据集），则会展示所选模型运行时实际接收到的 prompt。

运行方式：

```bash
```

- `-n`: 不进入交互模式，默认选择第一个 model （如有）和 dataset。
- `-a`: 查看配置中所有模型和所有数据集组合接收到的 prompt。
- `-p PATTERN`: 不进入交互模式，选择所有与传入正则表达式匹配的数据集。

## Case Analyzer

本工具在已有评测结果的基础上，产出推理错误样本以及带有标注信息的全量样本。

运行方式：

```bash
```

- `-w`：工作路径，默认为 `'./outputs/default'`。

## Lark Bot

用户可以通过配置飞书机器人，实现任务状态的实时监控。飞书机器人的设置文档请[参考这里](https://open.feishu.cn/document/ukTMukTMukTM/ucTM5YjL3ETO24yNxkjN?lang=zh-CN#7a28964d)。

配置方式:

- 打开 `configs/secrets.py` 文件，并在文件中加入以下行：

  ```python
  lark_bot_url = 'YOUR_WEBHOOK_URL'
  ```

  通常， Webhook URL 格式如 https://open.feishu.cn/open-apis/bot/v2/hook/xxxxxxxxxxxxxxxxx 。

- 在完整的评测配置中继承该文件：

  ```python
    _base_ = [
        'secrets.py',
        ...
    ]

```

  ```python
    _base_ = [
        ...
    ]
  ```

  实例可见 `configs/eval.py`。

- 为了避免机器人频繁发消息形成骚扰，默认运行时状态不会自动上报。有需要时，可以通过 `-l` 或 `--lark` 启动状态上报：

  ```bash
  python run.py configs/eval_demo.py -l
  ```

本工具可以快速测试 API 模型的功能是否正常。

运行方式：

```bash
```

本工具可以合并由于 `partitioner` 而产生的分片推理结果。

运行方式：

```bash
```

# Useful Tools (To be updated)

```shell
python tools/analysis_tools/analyze_logs.py plot_curve [--keys ${KEYS}] [--title ${TITLE}] [--legend ${LEGEND}] [--backend ${BACKEND}] [--style ${STYLE}] [--out ${OUT_FILE}]
```

![loss curve image](https://raw.githubusercontent.com/open-mmlab/mmdetection/master/resources/loss_curve.png)

Examples:

  ```shell
  ```

  ```shell
  ```

```shell
  ```

  ```shell
  ```

  ```shell
  ```

  ```text
  ```

`tools/misc/browse_dataset.py` helps the user to browse a detection dataset (both

```shell

```shell
```

In order to serve an `MMRotate` model with [`TorchServe`](https://pytorch.org/serve/), you can follow the steps:

### 1. Convert model from MMRotate to TorchServe

```shell
python tools/deployment/mmrotate2torchserve.py ${CONFIG_FILE} ${CHECKPOINT_FILE} \
```

Example:

```shell
wget -P checkpoint  \
https://download.openmmlab.com/mmrotate/v0.1.0/rotated_faster_rcnn/rotated-faster-rcnn-le90_r50_fpn_1x_dota/rotated_faster_rcnn_r50_fpn_1x_dota_le90-0393aa5c.pth

```shell

python tools/deployment/mmrotate2torchserve.py configs/rotated_faster_rcnn/rotated-faster-rcnn-le90_r50_fpn_1x_dota.py checkpoint/rotated_faster_rcnn_r50_fpn_1x_dota_le90-0393aa5c.pth \
--model-name rotated_faster_rcnn
```

**Note**: ${MODEL_STORE} needs to be an absolute path to a folder.

### 2. Build `mmrotate-serve` docker image

```shell
docker build -t mmrotate-serve:latest docker/serve/
```

### 3. Run `mmrotate-serve`

Check the official docs for [running TorchServe with docker](https://github.com/pytorch/serve/blob/master/docker/README.md#running-torchserve-in-a-production-docker-environment).

In order to run in GPU, you need to install [nvidia-docker](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html). You can omit the `--gpus` argument in order to run in CPU.

Example:

```shell
docker run --rm \
--cpus 8 \
--gpus device=0 \
-p8080:8080 -p8081:8081 -p8082:8082 \
--mount type=bind,source=$MODEL_STORE,target=/home/model-server/model-store \
mmrotate-serve:latest
```

[Read the docs](https://github.com/pytorch/serve/blob/072f5d088cce9bb64b2a18af065886c9b01b317b/docs/rest_api.md/) about the Inference (8080), Management (8081) and Metrics (8082) APis

```shell
curl -O https://raw.githubusercontent.com/open-mmlab/mmrotate/main/demo/demo.jpg
curl http://127.0.0.1:8080/predictions/${MODEL_NAME} -T demo.jpg
```

```shell
```

```json
[
  {
    "class_name": "small-vehicle",
    "bbox": [
      584.9473266601562,
      327.2749938964844,
      38.45665740966797,
      16.898427963256836,
      -0.7229751944541931
    ],
    "score": 0.9766026139259338
  },
  {
    "bbox": [
      152.0239715576172,
      305.92572021484375,
      43.144744873046875,
      18.85024642944336,
      0.014928221702575684
    ],
    "score": 0.972826361656189
  },
  {
    "class_name": "large-vehicle",
    "bbox": [
      160.58056640625,
      437.3690185546875,
      55.6795654296875,
      19.31710433959961,
      0.007036328315734863
    ],
    "score": 0.888836681842804
  },
  {
    "bbox": [
      666.2868041992188,

55.6795654296875,
    ],
  },
  {
    "bbox": [
      1011.3961181640625,
      60.396209716796875,
      21.821645736694336,
      0.8549195528030396
    ],
    "score": 0.8240180015563965
  }
]
```

And you can use `test_torchserver.py` to compare result of torchserver and pytorch, and visualize them.

```shell
[--inference-addr ${INFERENCE_ADDR}] [--device ${DEVICE}] [--score-thr ${SCORE_THR}]
```

Example:

```shell
configs/rotated_faster_rcnn/rotated-faster-rcnn-le90_r50_fpn_1x_dota.py \
rotated_faster_rcnn_r50_fpn_1x_dota_le90-0393aa5c.pth \
rotated_fater_rcnn
```

```

```shell
```

```text
Input shape: (3, 1024, 1024)
Flops: 215.92 GFLOPs
Params: 36.42 M
```

   input shape is (1, 3, 1024, 1024).

2. Some operators are not counted into FLOPs like DCN and custom operators. So models with dcn such as S<sup>2</sup>A-Net and RepPoints based model got wrong flops.
   Refer to [`mmcv.cnn.get_model_complexity_info()`](https://github.com/open-mmlab/mmcv/blob/master/mmcv/cnn/utils/flops_counter.py) for details.

   filename.

```shell
```

E.g.,

```shell

   filename.

```shell
```

E.g.,

```shell
python tools/model_converters/publish_model.py work_dirs/rotated_faster_rcnn/latest.pth rotated_faster_rcnn_r50_fpn_1x_dota_le90_20190801.pth
```

The final output filename will be `rotated_faster_rcnn_r50_fpn_1x_dota_le90_20190801-{hash id}.pth`.

```shell
    ${CHECKPOINT} \
```

${CONFIG} \
```

Examples: Assuming that you have already downloaded the `Rotated Faster R-CNN` model checkpoint to the directory `checkpoints/`.

```shell
       configs/rotated_faster_rcnn/rotated-faster-rcnn-le90_r50_fpn_1x_dota.py \
       checkpoints/rotated_faster_rcnn_r50_fpn_1x_dota_le90-0393aa5c.pth \
```

imports.

```shell
```

imports.

```shell
```

Then, run

```
```

![confusion_matrix_example](https://raw.githubusercontent.com/liuyanyi/doc-image/main/confusion_matrix.png)

# Dummy ResNet Wrapper

This is an example README for community `projects/`. We have provided detailed explanations for each field in the form of html comments, which are visible when you read the source of this README file. If you wish to submit your project to our main repository, then all the fields in this README are mandatory for others to understand what you have achieved in this implementation.

<!-- Share any information you would like others to know. For example:
This is an implementation of \[XXX\]. -->

This project implements a dummy ResNet wrapper, which literally does nothing new but prints "hello world" during initialization.

## Usage

In MMDet3D's root directory, run the following command to train the model:

```bash

```bash
python tools/train.py projects/example_project/configs/fcos3d_dummy-resnet-caffe-dcn_fpn_head-gn_8xb2-1x_nus-mono3d.py
```

In MMDet3D's root directory, run the following command to test the model:

```bash
python tools/test.py projects/example_project/configs/fcos3d_dummy-resnet-caffe-dcn_fpn_head-gn_8xb2-1x_nus-mono3d.py ${CHECKPOINT_PATH}
```

## Results

<!-- List the results as usually done in other model's README. [Example](https://github.com/open-mmlab/mmdetection3d/edit/dev-1.x/configs/fcos3d/README.md)

|                                                     Backbone                                                     | Lr schd | Mem (GB) | Inf time (fps) | mAP  | NDS  |         Download         |
| :--------------------------------------------------------------------------------------------------------------: | :-----: | :------: | :------------: | :--: | :--: | :----------------------: |
| [FCOS3D_dummy](projects/example_project/configs/fcos3d_dummy-resnet-caffe-dcn_fpn_head-gn_8xb2-1x_nus-mono3d.py) |   1x    |   8.69   |                | 29.8 | 37.7 | [model](<>) \| [log](<>) |

<!-- You may remove this section if not applicable. -->

```latex
@inproceedings{wang2021fcos3d,
	title={{FCOS3D: Fully} Convolutional One-Stage Monocular 3D Object Detection},

```latex
	author={Wang, Tai and Zhu, Xinge and Pang, Jiangmiao and Lin, Dahua},
	booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops},
}
# For the original 2D version
@inproceedings{tian2019fcos,
  title     =  {{FCOS: Fully} Convolutional One-Stage Object Detection},
  author    =  {Tian, Zhi and Shen, Chunhua and Chen, Hao and He, Tong},
  booktitle =  {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year      =  {2019}
}
```

booktitle =  {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
}
```

    <!-- The code's design shall follow existing interfaces and convention. For example, each model component should be registered into `mmdet3d.registry.MODELS` and configurable via a config file. -->

  - [ ] A full README

- [ ] A full README

    <!-- Ideally *all* the methods should have [type hints](https://www.pythontutorial.net/python-basics/python-type-hints/) and [docstrings](https://google.github.io/styleguide/pyguide.html#381-docstrings). [Example](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/models/detectors/fcos_mono3d.py) -->

    <!-- Unit tests for each module are required. [Example](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tests/test_models/test_dense_heads/test_fcos_mono3d_head.py) -->

    <!-- It will be parsed by MIM and Inferencer. [Example](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/fcos3d/metafile.yml) -->

  <!-- In particular, you may have to refactor this README into a standard one. [Example](/configs/textdet/dbnet/README.md) -->

## <a href='https://mmaction2.readthedocs.io/en/latest/'>English</a>

## <a href='https://mmaction2.readthedocs.io/zh_CN/latest/'>简体中文</a>

# DSDL: Standard Description Language for DataSet

<!-- [SKIP DEV CHECK] -->

Data is the cornerstone of artificial intelligence. The efficiency of data acquisition, exchange, and application directly impacts the advances in technologies and applications. Over the long history of AI, a vast quantity of data sets have been developed and distributed. However, these datasets are defined in very different forms, which incurs significant overhead when it comes to exchange, integration, and utilization -- it is often the case that one needs to develop a new customized tool or script in order to incorporate a new dataset into a workflow.

To overcome such difficulties, we develop **Data Set Description Language (DSDL)**. More details please visit our [official documents](https://opendatalab.github.io/dsdl-docs/getting_started/overview/), dsdl datasets can be downloaded from our platform [OpenDataLab](https://opendatalab.com/).

## Steps

## Steps

- install dsdl and opendatalab:

  ```
  pip install dsdl
  pip install opendatalab
  ```

- install mmseg and pytorch:
  please refer this [installation documents](https://mmsegmentation.readthedocs.io/en/latest/get_started.html).

- prepare dsdl dataset (take voc2012 as an example)

  - dowaload dsdl dataset (you will need an opendatalab account to do so. [register one now](https://opendatalab.com/))

    ```
    cd data

    odl login
    odl get PASCAL_VOC2012
    ```

    usually, dataset is compressed on opendatalab platform, the downloaded voc 2012 dataset should be like this:

    ```
    data/
    ├── PASCAL_VOC2012
    │   ├── dsdl
    │   │   ├── dsdl_Det_full.zip
    │   │   └── dsdl_SemSeg_full.zip
    │   ├── raw
    │   │   ├── VOC2012test.tar
    │   │   ├── VOCdevkit_18-May-2011.tar
    │   │   └── VOCtrainval_11-May-2012.tar
    │   └── README.md
    └── ...
    ```

  - decompress dataset

    ```
    cd dsdl

│   │   ├── VOCdevkit_18-May-2011.tar
    └── ...
    ```

    ```
    cd dsdl
    unzip dsdl_SemSeg_full.zip
    ```

    as we do not need detection dsdl files, we only decompress the semantic segmentation files here.

    ```
    cd ../raw
    tar -xvf VOCtrainval_11-May-2012.tar
    tar -xvf VOC2012test.tar

    cd ../../
    ```

- change traning config

  open the [voc config file](voc.py) and set some file paths as below:

  ```
  data_root = 'data/PASCAL_VOC2012'
  img_prefix = 'raw/VOCdevkit/VOC2012'
  train_ann = 'dsdl/dsdl_SemSeg_full/set-train/train.yaml'
  val_ann = 'dsdl/dsdl_SemSeg_full/set-val/val.yaml'
  ```

  as dsdl datasets with one task using one dataloader, we can simplly change these file paths to train a model on a different dataset.

- train:

  - using single gpu:

  ```
  python tools/train.py {config_file}
  ```

  - using slrum:

  ```

- train:

  ```
  ```

  ```
  ./tools/slurm_train.sh {partition} {job_name} {config_file} {work_dir} {gpu_nums}
  ```

## Test Results

|  Datasets  |                                                                                        Model                                                                                         | mIoU(%) |          Config           |
| :--------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----: | :-----------------------: |
|  voc2012   |    [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x512_20k_voc12aug/deeplabv3_r50-d8_512x512_20k_voc12aug_20200617_010906-596905ef.pth)    |  76.73  |    [config](./voc.py)     |

| cityscapes | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x1024_40k_cityscapes/deeplabv3_r50-d8_512x1024_40k_cityscapes_20200605_022449-acadc2f8.pth) |  79.01  | [config](./cityscapes.py) |

Here is a practical guide for converting pytorch model to ncnn

resnet18 is used as the example

## pytorch to onnx

The official pytorch tutorial for exporting onnx model

https://pytorch.org/tutorials/advanced/super_resolution_with_caffe2.html

```python
import torchvision
import torch.onnx

# An instance of your model
model = torchvision.models.resnet18()

# An example input you would normally provide to your model's forward() method
x = torch.rand(1, 3, 224, 224)

# Export the model
torch_out = torch.onnx._export(model, x, "resnet18.onnx", export_params=True)
```

## simplify onnx model

The exported resnet18.onnx model may contains many redundant operators such as Shape, Gather and Unsqueeze that is not supported in ncnn

```
Shape not supported yet!
Gather not supported yet!
  # axis=0
Unsqueeze not supported yet!
  # axes 7
  # axes 7
```

Fortunately, daquexian developed a handy tool to eliminate them. cheers!

  # axis=0
  # axes 7
  # axes 7
```

https://github.com/daquexian/onnx-simplifier

```
python3 -m onnxsim resnet18.onnx resnet18-sim.onnx
```

## onnx to ncnn

Finally, you can convert the model to ncnn using tools/onnx2ncnn

```
onnx2ncnn resnet18-sim.onnx resnet18.param resnet18.bin
```

# PointPillars: Fast Encoders for Object Detection from Point Clouds

> [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://arxiv.org/abs/1812.05784)

Object detection in point clouds is an important aspect of many robotics applications such as autonomous driving. In this paper we consider the problem of encoding a point cloud into a format appropriate for a downstream detection pipeline. Recent literature suggests two types of encoders; fixed encoders tend to be fast but sacrifice accuracy, while encoders that are learned from data are more accurate, but slower. In this work we propose PointPillars, a novel encoder which utilizes PointNets to learn a representation of point clouds organized in vertical columns (pillars). While the encoded features can be used with any standard 2D convolutional detection architecture, we further propose a lean downstream network. Extensive experimentation shows that PointPillars outperforms previous encoders with respect to both speed and accuracy by a large margin. Despite only using lidar, our full detection pipeline significantly outperforms the state of the art, even among fusion methods, with respect to both the 3D and bird's eye view KITTI benchmarks. This detection performance is achieved while running at 62 Hz: a 2 - 4 fold runtime improvement. A faster version of our method matches the state of the art at 105 Hz. These benchmarks suggest that PointPillars is an appropriate encoding for object detection in point clouds.

<img src="https://user-images.githubusercontent.com/79644370/143885905-aab6ffcf-7727-495e-90ca-edb8dd5e324b.png" width="800"/>
</div>

We implement PointPillars and provide the results and checkpoints on KITTI, nuScenes, Lyft and Waymo datasets.

### KITTI

|                            Backbone                             |  Class  |   Lr schd   | Mem (GB) | Inf time (fps) |  AP   |                                                                                                                                                                                                         Download                                                                                                                                                                                                         |

|  [SECFPN](./pointpillars_hv_secfpn_8xb6-160e_kitti-3d-car.py)   |   Car   | cyclic 160e |   5.4    |                | 77.6  |       [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-car/hv_pointpillars_secfpn_6x8_160e_kitti-3d-car_20220331_134606-d42d15ed.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-car/hv_pointpillars_secfpn_6x8_160e_kitti-3d-car_20220331_134606.log.json)       |

| [SECFPN](./pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py) | 3 Class | cyclic 160e |   5.5    |                | 64.07 | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class_20220301_150306-37dc2420.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class_20220301_150306.log.json) |

### nuScenes

|                                Backbone                                 | Lr schd | Mem (GB) | Inf time (fps) |  mAP  |  NDS  |                                                                                                                                                                                                     Download                                                                                                                                                                                                     |

|      [SECFPN](./pointpillars_hv_secfpn_sbn-all_8xb4-2x_nus-3d.py)       |   2x    |   16.4   |                | 34.33 | 49.1  |   [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pointpillars/hv_pointpillars_secfpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_secfpn_sbn-all_4x8_2x_nus-3d_20210826_225857-f19d00a3.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pointpillars/hv_pointpillars_secfpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_secfpn_sbn-all_4x8_2x_nus-3d_20210826_225857.log.json)   |

| [SECFPN (FP16)](./pointpillars_hv_secfpn_sbn-all_8xb2-amp-2x_nus-3d.py) |   2x    |   8.37   |                | 35.19 | 50.27 | [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/fp16/hv_pointpillars_secfpn_sbn-all_fp16_2x8_2x_nus-3d/hv_pointpillars_secfpn_sbn-all_fp16_2x8_2x_nus-3d_20201020_222626-c3f0483e.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/fp16/hv_pointpillars_secfpn_sbn-all_fp16_2x8_2x_nus-3d/hv_pointpillars_secfpn_sbn-all_fp16_2x8_2x_nus-3d_20201020_222626.log.json) |

|         [FPN](./pointpillars_hv_fpn_sbn-all_8xb4-2x_nus-3d.py)          |   2x    |   16.3   |                | 39.7  | 53.2  |         [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pointpillars/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d_20210826_104936-fca299c1.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pointpillars/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d_20210826_104936.log.json)         |

|    [FPN (FP16)](./pointpillars_hv_fpn_sbn-all_8xb2-amp-2x_nus-3d.py)    |   2x    |   8.40   |                | 39.26 | 53.26 |       [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/fp16/hv_pointpillars_fpn_sbn-all_fp16_2x8_2x_nus-3d/hv_pointpillars_fpn_sbn-all_fp16_2x8_2x_nus-3d_20201021_120719-269f9dd6.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/fp16/hv_pointpillars_fpn_sbn-all_fp16_2x8_2x_nus-3d/hv_pointpillars_fpn_sbn-all_fp16_2x8_2x_nus-3d_20201021_120719.log.json)       |

### Lyft

### Lyft

|                           Backbone                            | Lr schd | Mem (GB) | Inf time (fps) | Private Score | Public Score |                                                                                                                                                                                                     Download                                                                                                                                                                                                     |

| [SECFPN](./pointpillars_hv_secfpn_sbn-all_8xb2-2x_lyft-3d.py) |   2x    |   12.2   |                |     13.8      |     14.1     | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pointpillars/hv_pointpillars_secfpn_sbn-all_2x8_2x_lyft-3d/hv_pointpillars_secfpn_sbn-all_2x8_2x_lyft-3d_20210829_100455-82b81c39.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pointpillars/hv_pointpillars_secfpn_sbn-all_2x8_2x_lyft-3d/hv_pointpillars_secfpn_sbn-all_2x8_2x_lyft-3d_20210829_100455.log.json) |

|    [FPN](./pointpillars_hv_fpn_sbn-all_8xb2-2x_lyft-3d.py)    |   2x    |   9.2    |                |     14.8      |     15.0     |       [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pointpillars/hv_pointpillars_fpn_sbn-all_2x8_2x_lyft-3d/hv_pointpillars_fpn_sbn-all_2x8_2x_lyft-3d_20210822_095429-0b3d6196.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pointpillars/hv_pointpillars_fpn_sbn-all_2x8_2x_lyft-3d/hv_pointpillars_fpn_sbn-all_2x8_2x_lyft-3d_20210822_095429.log.json)       |

### Waymo

### Waymo

|                                 Backbone                                 | Load Interval |  Class  | Lr schd | Mem (GB) | Inf time (fps) | mAP@L1 | mAPH@L1 | mAP@L2 | **mAPH@L2** |                                                                                                                                                                                                                   Download                                                                                                                                                                                                                   |

|  [SECFPN](./pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymoD5-3d-car.py)   |       5       |   Car   |   2x    |   7.76   |                |  70.2  |  69.6   |  62.6  |    62.1     |       [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_secfpn_sbn_2x16_2x_waymoD5-3d-car/hv_pointpillars_secfpn_sbn_2x16_2x_waymoD5-3d-car_20200901_204315-302fc3e7.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_secfpn_sbn_2x16_2x_waymoD5-3d-car/hv_pointpillars_secfpn_sbn_2x16_2x_waymoD5-3d-car_20200901_204315.log.json)       |

| [SECFPN](./pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymoD5-3d-3class.py) |       5       | 3 Class |   2x    |   8.12   |                |  64.7  |  57.6   |  58.4  |    52.1     | [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_secfpn_sbn_2x16_2x_waymoD5-3d-3class/hv_pointpillars_secfpn_sbn_2x16_2x_waymoD5-3d-3class_20200831_204144-d1a706b1.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_secfpn_sbn_2x16_2x_waymoD5-3d-3class/hv_pointpillars_secfpn_sbn_2x16_2x_waymoD5-3d-3class_20200831_204144.log.json) |

|                               above @ Car                                |               |         |   2x    |   8.12   |                |  68.5  |  67.9   |  60.1  |    59.6     |                                                                                                                                                                                                                                                                                                                                                                                                                                              |

|                            above @ Pedestrian                            |               |         |   2x    |   8.12   |                |  67.8  |  50.6   |  59.6  |    44.3     |                                                                                                                                                                                                                                                                                                                                                                                                                                              |

|                             above @ Cyclist                              |               |         |   2x    |   8.12   |                |  57.7  |  54.4   |  55.5  |    52.4     |                                                                                                                                                                                                                                                                                                                                                                                                                                              |

|   [SECFPN](./pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymo-3d-car.py)    |       1       |   Car   |   2x    |   7.76   |                |  72.1  |  71.5   |  63.6  |    63.1     |                                                                                                                           [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_secfpn_sbn_2x16_2x_waymo-3d-car/hv_pointpillars_secfpn_sbn_2x16_2x_waymo-3d-car.log.json)                                                                                                                            |

|  [SECFPN](./pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymo-3d-3class.py)  |       1       | 3 Class |   2x    |   8.12   |                |  68.8  |  63.3   |  62.6  |    57.6     |                                                                                                                        [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_secfpn_sbn_2x16_2x_waymo-3d-3class/hv_pointpillars_secfpn_sbn_2x16_2x_waymo-3d-3class.log.json)                                                                                                                         |

|                               above @ Car                                |               |         |   2x    |   8.12   |                |  71.6  |  71.0   |  63.1  |    62.5     |                                                                                                                                                                                                                                                                                                                                                                                                                                              |

|                            above @ Pedestrian                            |               |         |   2x    |   8.12   |                |  70.6  |  56.7   |  62.9  |    50.2     |                                                                                                                                                                                                                                                                                                                                                                                                                                              |

|                             above @ Cyclist                              |               |         |   2x    |   8.12   |                |  64.4  |  62.3   |  61.9  |    59.9     |                                                                                                                                                                                                                                                                                                                                                                                                                                              |

#### Note:

- **Metric**: For model trained with 3 classes, the average APH@L2 (mAPH@L2) of all the categories is reported and used to rank the model. For model trained with only 1 class, the APH@L2 is reported and used to rank the model.

- **Data Split**: Here we provide several baselines for waymo dataset, among which D5 means that we divide the dataset into 5 folds and only use one fold for efficient experiments. Using the complete dataset can boost the performance a lot, especially for the detection of cyclist and pedestrian, where more than 5 mAP or mAPH improvement can be expected.
- **Implementation Details**: We basically follow the implementation in the [paper](https://arxiv.org/pdf/1912.04838.pdf) in terms of the network architecture (having a
  stride of 1 for the first convolutional block). Different settings of voxelization, data augmentation and hyper parameters make these baselines outperform those in the paper by about 7 mAP for car and 4 mAP for pedestrian with only a subset of the whole dataset. All of these results are achieved without bells-and-whistles, e.g. ensemble, multi-scale training and test augmentation.

- **License Aggrement**: To comply the [license agreement of Waymo dataset](https://waymo.com/open/terms/), the pre-trained models on Waymo dataset are not released. We still release the training log as a reference to ease the future research.
- `FP16` means Mixed Precision (FP16) is adopted in training. With mixed precision training, we can train PointPillars with nuScenes dataset on 8 Titan XP GPUS with batch size of 2. This will cause OOM error without mixed precision training. The loss scale for PointPillars on nuScenes dataset is specifically tuned to avoid the loss to be Nan. We find 32 is more stable than 512, though loss scale 32 still cause Nan sometimes.

```latex
@inproceedings{lang2019pointpillars,
  title={Pointpillars: Fast encoders for object detection from point clouds},
  author={Lang, Alex H and Vora, Sourabh and Caesar, Holger and Zhou, Lubing and Yang, Jiong and Beijbom, Oscar},

title={Pointpillars: Fast encoders for object detection from point clouds},
  pages={12697--12705},
}
```

# LD

> [Localization Distillation for Dense Object Detection](https://arxiv.org/abs/2102.12252)

Knowledge distillation (KD) has witnessed its powerful capability in learning compact models in object detection. Previous KD methods for object detection mostly focus on imitating deep features within the imitation regions instead of mimicking classification logits due to its inefficiency in distilling localization information. In this paper, by reformulating the knowledge distillation process on localization, we present a novel localization distillation (LD) method which can efficiently transfer the localization knowledge from the teacher to the student. Moreover, we also heuristically introduce the concept of valuable localization region that can aid to selectively distill the semantic and localization knowledge for a certain region. Combining these two new components, for the first time, we show that logit mimicking can outperform feature imitation and localization knowledge distillation is more important and efficient than semantic knowledge for distilling object detectors. Our distillation scheme is simple as well as effective and can be easily applied to different dense object detectors. Experiments show that our LD can boost the AP score of GFocal-ResNet-50 with a single-scale 1× training schedule from 40.1 to 42.1 on the COCO benchmark without any sacrifice on the inference speed.

<img src="https://user-images.githubusercontent.com/40661020/143966265-48a03668-8585-4525-8a86-afa2209d1602.png"/>
</div>

### GFocalV1 with LD

|  Teacher  | Student | Training schedule | Mini-batch size | AP (val) |                      Config                       |                                                                                                                                                        Download                                                                                                                                                        |

| :-------: | :-----: | :---------------: | :-------------: | :------: | :-----------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|    --     |  R-18   |        1x         |        6        |   35.8   |                                                   |                                                                                                                                                                                                                                                                                                                        |

|   R-101   |  R-18   |        1x         |        6        |   36.5   |   [config](./ld_r18-gflv1-r101_fpn_1x_coco.py)    |         [model](https://download.openmmlab.com/mmdetection/v2.0/ld/ld_r18_gflv1_r101_fpn_coco_1x/ld_r18_gflv1_r101_fpn_coco_1x_20220702_062206-330e6332.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/ld/ld_r18_gflv1_r101_fpn_coco_1x/ld_r18_gflv1_r101_fpn_coco_1x_20220702_062206.log.json)         |
|    --     |  R-34   |        1x         |        6        |   38.9   |                                                   |                                                                                                                                                                                                                                                                                                                        |

|   R-101   |  R-34   |        1x         |        6        |   39.9   |   [config](./ld_r34-gflv1-r101_fpn_1x_coco.py)    |         [model](https://download.openmmlab.com/mmdetection/v2.0/ld/ld_r34_gflv1_r101_fpn_coco_1x/ld_r34_gflv1_r101_fpn_coco_1x_20220630_134007-9bc69413.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/ld/ld_r34_gflv1_r101_fpn_coco_1x/ld_r34_gflv1_r101_fpn_coco_1x_20220630_134007.log.json)         |
|    --     |  R-50   |        1x         |        6        |   40.1   |                                                   |                                                                                                                                                                                                                                                                                                                        |

|   R-101   |  R-50   |        1x         |        6        |   41.0   |   [config](./ld_r50-gflv1-r101_fpn_1x_coco.py)    |         [model](https://download.openmmlab.com/mmdetection/v2.0/ld/ld_r50_gflv1_r101_fpn_coco_1x/ld_r50_gflv1_r101_fpn_coco_1x_20220629_145355-8dc5bad8.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/ld/ld_r50_gflv1_r101_fpn_coco_1x/ld_r50_gflv1_r101_fpn_coco_1x_20220629_145355.log.json)         |
|    --     |  R-101  |        2x         |        6        |   44.6   |                                                   |                                                                                                                                                                                                                                                                                                                        |

| R-101-DCN |  R-101  |        2x         |        6        |   45.5   | [config](./ld_r101-gflv1-r101-dcn_fpn_2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/ld/ld_r101_gflv1_r101dcn_fpn_coco_2x/ld_r101_gflv1_r101dcn_fpn_coco_2x_20220629_185920-9e658426.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/ld/ld_r101_gflv1_r101dcn_fpn_coco_2x/ld_r101_gflv1_r101dcn_fpn_coco_2x_20220629_185920.log.json) |

## Note

- Meaning of Config name: ld_r18(student model)\_gflv1(based on gflv1)\_r101(teacher model)\_fpn(neck)\_coco(dataset)\_1x(12 epoch).py

```latex
@Inproceedings{zheng2022LD,
  title={Localization Distillation for Dense Object Detection},
  author= {Zheng, Zhaohui and Ye, Rongguang and Wang, Ping and Ren, Dongwei and Zuo, Wangmeng and Hou, Qibin and Cheng, Mingming},
}
```

# 可视化的迁移

在0.x版中，MMEditing使用`VisualizationHook`来对训练过程中生成的结果进行可视化，在1.x版中，我们将该功能整合到`BasicVisualizationHook` / `VisualizationHook`中，而且遵循MMEngine的设计，我们实现了`ConcatImageVisualizer` / `Visualizer`和一系列`VisBackend`来绘制和保存可视化结果。

<thead>
  <tr>
    <th> 0.x版 </th>
    <th> 1.x版 </th>
  </tr>
</thead>
<tbody>
<tr>

```python
visual_config = dict(
    type='VisualizationHook',
    output_dir='visual',
    interval=1000,
    res_name_list=['gt_img', 'masked_img', 'fake_res', 'fake_img'],
)
```

</td>

```python
    type='ConcatImageVisualizer',
    fn_key='gt_path',
    img_keys=['gt_img', 'input', 'pred_img'],
    bgr2rgb=True)
custom_hooks = [dict(type='BasicVisualizationHook', interval=1)]
```

</td>

</tr>
</tbody>
</table>

要了解更多关于可视化的功能，请参阅[这个教程](../user_guides/visualization.md)。

# 学习配置文件

MMDetection3D 和其他 OpenMMLab 仓库使用 [MMEngine 的配置文件系统](https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/config.html)。它具有模块化和继承性设计，以便于进行各种实验。

## 配置文件的内容

MMDetection3D 采用模块化设计，所有功能的模块可以通过配置文件进行配置。以 PointPillars 为例，我们将根据不同的功能模块介绍配置文件的各个字段。

### 模型配置

在 MMDetection3D 的配置中，我们使用 `model` 字段来配置检测算法的组件。除了 `voxel_encoder`，`backbone` 等神经网络组件外，还需要 `data_preprocessor`，`train_cfg` 和 `test_cfg`。`data_preprocessor` 负责对数据加载器（dataloader）输出的每一批数据进行预处理。模型配置中的 `train_cfg` 和 `test_cfg` 用于设置训练和测试组件的超参数。

```python
    type='VoxelNet',
        type='Det3DDataPreprocessor',
        voxel=True,
        voxel_layer=dict(
            max_num_points=32,
            point_cloud_range=[0, -39.68, -3, 69.12, 39.68, 1],
            voxel_size=[0.16, 0.16, 4],
            max_voxels=(16000, 40000))),
    voxel_encoder=dict(
        type='PillarFeatureNet',
        in_channels=4,
        feat_channels=[64],
        with_distance=False,

max_voxels=(16000, 40000))),
        voxel_size=[0.16, 0.16, 4],
        point_cloud_range=[0, -39.68, -3, 69.12, 39.68, 1]),
    middle_encoder=dict(
        type='PointPillarsScatter', in_channels=64, output_shape=[496, 432]),
        type='SECOND',
        in_channels=64,
        layer_nums=[3, 5, 5],
        layer_strides=[2, 2, 2],
        out_channels=[64, 128, 256]),
        type='SECONDFPN',
        in_channels=[64, 128, 256],
        upsample_strides=[1, 2, 4],
        out_channels=[128, 128, 128]),
    bbox_head=dict(
        type='Anchor3DHead',
        num_classes=3,
        in_channels=384,
        feat_channels=384,
        use_direction_classifier=True,
        assign_per_class=True,
        anchor_generator=dict(
            type='AlignedAnchor3DRangeGenerator',

feat_channels=384,
            ranges=[[0, -39.68, -0.6, 69.12, 39.68, -0.6],
                    [0, -39.68, -0.6, 69.12, 39.68, -0.6],
                    [0, -39.68, -1.78, 69.12, 39.68, -1.78]],
            sizes=[[0.8, 0.6, 1.73], [1.76, 0.6, 1.73], [3.9, 1.6, 1.56]],
            rotations=[0, 1.57],
            reshape_out=False),
        diff_rad_by_sin=True,
        bbox_coder=dict(type='DeltaXYZWLHRBBoxCoder'),
            type='mmdet.FocalLoss',
        loss_bbox=dict(
            type='mmdet.SmoothL1Loss',
            beta=0.1111111111111111,
            loss_weight=2.0),
        loss_dir=dict(
            type='mmdet.CrossEntropyLoss', use_sigmoid=False,
            loss_weight=0.2)),

beta=0.1111111111111111,
    train_cfg=dict(
        assigner=[
            dict(
                type='Max3DIoUAssigner',
                iou_calculator=dict(type='BboxOverlapsNearest3D'),
                pos_iou_thr=0.5,
                neg_iou_thr=0.35,
                min_pos_iou=0.35,
                ignore_iof_thr=-1),
            dict(
            dict(
                pos_iou_thr=0.6,
                neg_iou_thr=0.45,
                min_pos_iou=0.45,

iou_calculator=dict(type='BboxOverlapsNearest3D'),
                ignore_iof_thr=-1)
        ],
        allowed_border=0,
        pos_weight=-1,
        debug=False),
        use_rotate_nms=True,
        nms_across_levels=False,
        nms_thr=0.01,
        score_thr=0.1,
        min_bbox_size=0,
        nms_pre=100,
        max_num=50))
```

在使用[执行器（Runner）](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/runner.html)进行训练、测试和验证时，我们需要配置[数据加载器](https://pytorch.org/docs/stable/data.html?highlight=data%20loader#torch.utils.data.DataLoader)。构建数据加载器需要设置数据集和数据处理流程。由于这部分的配置较为复杂，我们使用中间变量来简化数据加载器配置的编写。

```python
dataset_type = 'KittiDataset'
data_root = 'data/kitti/'
class_names = ['Pedestrian', 'Cyclist', 'Car']
point_cloud_range = [0, -39.68, -3, 69.12, 39.68, 1]
input_modality = dict(use_lidar=True, use_camera=False)

metainfo = dict(classes=class_names)

db_sampler = dict(
    info_path=data_root + 'kitti_dbinfos_train.pkl',
    rate=1.0,
    prepare=dict(
        filter_by_difficulty=[-1],
        filter_by_min_points=dict(Car=5, Pedestrian=5, Cyclist=5)),
    classes=class_names,
    sample_groups=dict(Car=15, Pedestrian=15, Cyclist=15),
    points_loader=dict(
        type='LoadPointsFromFile', coord_type='LIDAR', load_dim=4, use_dim=4))

    dict(type='LoadPointsFromFile', coord_type='LIDAR', load_dim=4, use_dim=4),
    dict(type='ObjectSample', db_sampler=db_sampler, use_ground_plane=True),
    dict(

dict(type='ObjectSample', db_sampler=db_sampler, use_ground_plane=True),
    dict(
        rot_range=[-0.78539816, 0.78539816],
        scale_ratio_range=[0.95, 1.05]),
    dict(
        keys=['points', 'gt_labels_3d', 'gt_bboxes_3d'])
]
    dict(
        type='MultiScaleFlipAug3D',
        pts_scale_ratio=1,
            dict(

dict(
            dict(
                type='PointsRangeFilter', point_cloud_range=point_cloud_range)
        ]),
    dict(type='Pack3DDetInputs', keys=['points'])
]
eval_pipeline = [
]
    batch_size=6,
        type='RepeatDataset',
        times=2,
        dataset=dict(
            type=dataset_type,
            data_root=data_root,
            ann_file='kitti_infos_train.pkl',
            data_prefix=dict(pts='training/velodyne_reduced'),
            pipeline=train_pipeline,

            modality=input_modality,
            test_mode=False,
            metainfo=metainfo,
            box_type_3d='LiDAR')))
    num_workers=1,
        data_prefix=dict(pts='training/velodyne_reduced'),
        ann_file='kitti_infos_val.pkl',
        modality=input_modality,
        metainfo=metainfo,
        box_type_3d='LiDAR'))

```

[评测器](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/evaluation.html)用于计算训练模型在验证和测试数据集上的指标。评测器的配置由一个或一组评价指标配置组成：

```python
    type='KittiMetric',
    ann_file=data_root + 'kitti_infos_val.pkl',
    metric='bbox')
```

由于测试数据集没有标注文件，因此 MMDetection3D 中的 test_dataloader 和 test_evaluator 配置通常等于 val。如果您想要保存在测试数据集上的检测结果，则可以像这样编写配置：

```python
# 在测试集上推理，

```python
# 在测试集上推理，
        data_prefix=dict(pts='testing/velodyne_reduced'),
        ann_file='kitti_infos_test.pkl',
        load_eval_anns=False,
    ann_file=data_root + 'kitti_infos_test.pkl',
    metric='bbox',
    format_only=True,
    submission_prefix='results/kitti-3class/kitti_results')
```

### 训练和测试配置

MMEngine 的执行器使用循环（Loop）来控制训练，验证和测试过程。用户可以使用这些字段设置最大训练轮次和验证间隔：

```python
    type='EpochBasedTrainLoop',
    max_epochs=80,
    val_interval=2)

```python
```

### 优化配置

`optim_wrapper` 是配置优化相关设置的字段。优化器封装不仅提供了优化器的功能，还支持梯度裁剪、混合精度训练等功能。更多内容请看[优化器封装教程](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/optim_wrapper.html)。

```python
optim_wrapper = dict(  # 优化器封装配置
    type='OptimWrapper',  # 优化器封装类型，切换到 AmpOptimWrapper 启动混合精度训练
    optimizer=dict(  # 优化器配置。支持 PyTorch 的各种优化器，请参考 https://pytorch.org/docs/stable/optim.html#algorithms
        type='AdamW', lr=0.001, betas=(0.95, 0.99), weight_decay=0.01),
    clip_grad=dict(max_norm=35, norm_type=2))  # 梯度裁剪选项。设置为 None 禁用梯度裁剪。使用方法请见 https://mmengine.readthedocs.io/zh_CN/latest/tutorials/optim_wrapper.html
```

clip_grad=dict(max_norm=35, norm_type=2))  # 梯度裁剪选项。设置为 None 禁用梯度裁剪。使用方法请见 https://mmengine.readthedocs.io/zh_CN/latest/tutorials/optim_wrapper.html
```

`param_scheduler` 是配置调整优化器超参数（例如学习率和动量）的字段。用户可以组合多个调度器来创建所需要的参数调整策略。更多信息请参考[参数调度器教程](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/param_scheduler.html)和[参数调度器 API 文档](https://mmengine.readthedocs.io/zh_CN/latest/api/optim.html#scheduler)。

```python
    dict(
        type='CosineAnnealingLR',
        T_max=32,
        eta_min=0.01,
        begin=0,
        end=32,
        convert_to_iter_based=True),
    dict(
        T_max=48,
        eta_min=1.0000000000000001e-07,
        begin=32,
        end=80,
    dict(
        type='CosineAnnealingMomentum',
        T_max=32,
        eta_min=0.8947368421052632,
        begin=0,
        end=32,

convert_to_iter_based=True),
    dict(
        T_max=32,
        begin=0,
        end=32,
    dict(
        T_max=48,
        eta_min=1,
        begin=32,
        end=80,
]
```

### 钩子配置

用户可以在训练、验证和测试循环上添加钩子，从而在运行期间插入一些操作。有两种不同的钩子字段，一种是 `default_hooks`，另一种是 `custom_hooks`。

`default_hooks` 是一个钩子配置字典，并且这些钩子是运行时所需要的。它们具有默认优先级，是不需要修改的。如果未设置，执行器将使用默认值。如果要禁用默认钩子，用户可以将其配置设置为 `None`。

```python
    checkpoint=dict(type='CheckpointHook', interval=-1),
    visualization=dict(type='Det3DVisualizationHook'))
```

checkpoint=dict(type='CheckpointHook', interval=-1),
```

`custom_hooks` 是一个由其他钩子配置组成的列表。用户可以开发自己的钩子并将其插入到该字段中。

```python
```

### 运行配置

```python
default_scope = 'mmdet3d'  # 寻找模块的默认注册器域。请参考 https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/registry.html

    mp_cfg=dict(  # 多进程配置
        mp_start_method='fork',  # 使用 fork 来启动多进程。'fork' 通常比 'spawn' 更快，但可能不安全。请参考 https://github.com/pytorch/pytorch/issues/1355
        opencv_num_threads=0),  # 关闭 opencv 的多进程以避免系统超负荷
    dist_cfg=dict(backend='nccl'))  # 分布式配置

vis_backends = [dict(type='LocalVisBackend')]  # 可视化后端。请参考 https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/visualization.html
    type='Det3DLocalVisualizer', vis_backends=vis_backends, name='visualizer')

    by_epoch=True)  # 是否使用 epoch 格式的日志。需要与训练循环的类型保持一致

resume = False  # 是否从 `load_from` 中定义的检查点恢复。如果 `load_from` 为 None，它将恢复 `work_dir` 中的最近检查点。
```

## 配置文件继承

在 `configs/_base_` 文件夹下有 4 个基本组件类型，分别是：数据集（dataset），模型（model），训练策略（schedule）和运行时的默认设置（default runtime）。许多方法，如 SECOND、PointPillars、PartA2、VoteNet 都能够很容易地构建出来。由 `_base_` 下的组件组成的配置，被我们称为 _原始配置（primitive）_。

对于同一个文件夹下的所有配置，推荐**只有一个**对应的 _原始配置_ 文件。所有其他的配置文件都应该继承自这个 _原始配置_ 文件。这样就能保证配置文件的最大继承深度为 3。

为了便于理解，我们建议贡献者继承现有方法。例如，如果在 PointPillars 的基础上做了一些修改，用户可以首先通过指定 `_base_ = '../pointpillars/pointpillars_hv_fpn_sbn-all_8xb4-2x_nus-3d.py'` 来继承基础的 PointPillars 结构，然后修改配置文件中的必要参数以完成继承。

如果您在构建一个与任何现有方法都不共享的全新方法，那么可以在 `configs` 文件夹下创建一个新的例如 `xxx_rcnn` 文件夹。

更多细节请参考 [MMEngine 配置文件教程](https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/config.html)。

```python
_base_ = './pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py'
```

当 `_base_` 是多个文件路径组成的列表式，表示继承多个文件。

```python
_base_ = [
    '../_base_/models/pointpillars_hv_secfpn_kitti.py',
    '../_base_/datasets/kitti-3d-3class.py',
    '../_base_/schedules/cyclic-40e.py', '../_base_/default_runtime.py'
]
```

如果需要检测配置文件，可以通过运行 `python tools/misc/print_config.py /PATH/TO/CONFIG` 来查看完整的配置。

### 忽略基础配置文件里的部分字段

有时，您也许会设置 `_delete_=True` 去忽略基础配置文件里的一些字段。您可以参考 [MMEngine 配置文件教程](https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/config.html) 来获得一些简单的指导。

在 MMDetection3D 里，例如，修改以下 PointPillars 配置中的颈部网络：

```python
    type='MVXFasterRCNN',
    data_preprocessor=dict(voxel_layer=dict(...)),

```python
    pts_voxel_encoder=dict(...),
    pts_middle_encoder=dict(...),
    pts_backbone=dict(...),
    pts_neck=dict(
        type='FPN',
        norm_cfg=dict(type='naiveSyncBN2d', eps=1e-3, momentum=0.01),
        act_cfg=dict(type='ReLU'),
        out_channels=256,
        start_level=0,
        num_outs=3),
    pts_bbox_head=dict(...))
```

`FPN` 和 `SECONDFPN` 使用不同的关键字来构建：

```python
_base_ = '../_base_/models/pointpillars_hv_fpn_nus.py'
```

`_delete_=True` 将使用新的键去替换 `pts_neck` 字段内所有旧的键。

### 在配置文件里使用中间变量

upsample_strides=[1, 2, 4],
```

配置文件里会使用一些中间变量，例如数据集里的 `train_pipeline`/`test_pipeline`。需要注意的是，当修改子配置文件中的中间变量时，用户需要再次将中间变量传递到对应的字段中。例如，我们想使用多尺度策略训练并测试 PointPillars，`train_pipeline`/`test_pipeline` 是我们想要修改的中间变量。

```python
_base_ = './nus-3d.py'
    dict(
    dict(
    dict(

rot_range=[-0.3925, 0.3925],
    dict(
]
    dict(
    dict(
    dict(
        pts_scale_ratio=[0.95, 1.0, 1.05],
            dict(

img_scale=(1333, 800),
            dict(
            dict(
        ]),
]
```

我们首先定义新的 `train_pipeline`/`test_pipeline`，然后传递到数据加载器字段中。

如果用户希望复用 base 文件中的变量，则可以通过使用 `{{_base_.xxx}}` 获取对应变量的拷贝。例如：

```python

a = {{_base_.model}}  # 变量 `a` 等于 `_base_` 中定义的 `model`
```

```python

```

当使用 `tools/train.py` 或者 `tools/test.py` 提交工作时，您可以通过指定 `--cfg-options` 来修改配置文件。

- 更新配置字典的键值

  可以按照原始配置文件中字典的键值顺序指定配置选项。例如，使用 `--cfg-options model.backbone.norm_eval=False` 将模型主干网络中的所有 BN 模块都改为 `train` 模式。

- 更新配置列表中的键值

  在配置文件里，一些配置字典被包含在列表中，例如，训练流程 `train_dataloader.dataset.pipeline` 通常是一个列表，例如 `[dict(type='LoadPointsFromFile'), ...]`。如果您想要将训练流程中的 `'LoadPointsFromFile'` 改成 `'LoadPointsFromDict'`，您需要指定 `--cfg-options data.train.pipeline.0.type=LoadPointsFromDict`。

- 更新列表/元组的值

  如果要更新的值是列表或元组。例如，配置文件通常设置 `model.data_preprocessor.mean=[123.675, 116.28, 103.53]`。如果您想要改变这个均值，您需要指定 `--cfg-options model.data_preprocessor.mean="[127,127,127]"`。注意，引号 `"` 是支持列表/元组数据类型所必需的，并且在指定值的引号内**不允许**有空格。

```

```
```

文件名分为五个部分。所有部分和组件用 `_` 连接，每个部分或组件内的单词应该用 `-` 连接。

- `{algorithm name}`：算法的名称。它可以是检测器的名称，例如 `pointpillars`、`fcos3d` 等。
- `{model component names}`：算法中使用的组件名称，如 voxel_encoder、backbone、neck 等。例如 `second_secfpn_head-dcn-circlenms` 表示使用 SECOND 的 SparseEncoder，SECONDFPN，以及带有 DCN 和 circle NMS 的检测头。
- `{training settings}`：训练设置的信息，例如批量大小，数据增强，损失函数策略，调度器以及训练轮次/迭代。例如 `8xb4-tta-cyclic-20e` 表示使用 8 个 gpu，每个 gpu 有 4 个数据样本，测试增强，余弦退火学习率，训练 20 个 epoch。缩写介绍：
  - `{gpu x batch_per_gpu}`：GPU 数和每个 GPU 的样本数。`bN` 表示每个 GPU 上的批量大小为 N。例如 `4xb4` 是 4 个 GPU，每个 GPU 有 4 个样本数的缩写。
  - `{schedule}`：训练方案，可选项为 `schedule-2x`、`schedule-3x`、`cyclic-20e` 等。`schedule-2x` 和 `schedule-3x` 分别代表 24 epoch 和 36 epoch。`cyclic-20e` 表示 20 epoch。

- `{schedule}`：训练方案，可选项为 `schedule-2x`、`schedule-3x`、`cyclic-20e` 等。`schedule-2x` 和 `schedule-3x` 分别代表 24 epoch 和 36 epoch。`cyclic-20e` 表示 20 epoch。
- `{training dataset information}`：训练数据集名，例如 `kitti-3d-3class`，`nus-3d`，`s3dis-seg`，`scannet-seg`，`waymoD5-3d-car`。这里 `3d` 表示数据集用于 3D 目标检测，`seg` 表示数据集用于点云分割。
- `{testing dataset information}`（可选）：当模型在一个数据集上训练，在另一个数据集上测试时的测试数据集名。如果没有注明，则表示训练和测试的数据集类型相同。

# 常见问题解答

## 概述

我们在这里列出了许多用户常遇到的问题以及相应的解决方案。

- [常见问题解答](#常见问题解答)
  - [概述](#概述)
  - [安装](#安装)
  - [数据](#数据)
  - [训练](#训练)
  - [测试](#测试)

如果您发现任何频繁出现的问题并且有解决方法，欢迎在列表中补充。如果这里的内容没有涵盖您的问题，请使用[提供的模板](https://github.com/open-mmlab/mmaction2/tree/main/.github/ISSUE_TEMPLATE/error-report.md)创建一个问题，并确保在模板中填写所有必要的信息。

## 安装

- **"No module named 'mmcv.ops'"; "No module named 'mmcv.\_ext'"**

  1. 使用 `pip uninstall mmcv` 命令卸载环境中的现有 mmcv。
  2. 参照[安装说明](https://mmcv.readthedocs.io/en/2.x/get_started/installation.html#install-mmcv)安装 mmcv。

- **"OSError: MoviePy Error: creation of None failed because of the following error"**

  使用 `pip install moviepy` 安装。更多信息可以参考[官方安装文档](https://zulko.github.io/moviepy/install.html), 请注意（根据这个 [issue](https://github.com/Zulko/moviepy/issues/693)）：

使用 `pip install moviepy` 安装。更多信息可以参考[官方安装文档](https://zulko.github.io/moviepy/install.html), 请注意（根据这个 [issue](https://github.com/Zulko/moviepy/issues/693)）：

  1. 对于 Windows 用户，[ImageMagick](https://www.imagemagick.org/script/index.php) 不会自动被 MoviePy 检测到，需要修改 `moviepy/config_defaults.py` 文件，提供 ImageMagick 二进制文件 `magick` 的路径，例如 `IMAGEMAGICK_BINARY = "C:\\Program Files\\ImageMagick_VERSION\\magick.exe"`
  2. 对于 Linux 用户，如果 MoviePy 没有检测到 ImageMagick，需要修改 `/etc/ImageMagick-6/policy.xml` 文件，将 `<policy domain="path" rights="none" pattern="@*" />` 注释掉，改为 `<!-- <policy domain="path" rights="none" pattern="@*" /> -->`。

- **"即使我已经安装了 XXCODEBASE，为什么还会收到 'Please install XXCODEBASE to use XXX' 的错误消息?"**

  您收到该错误消息是因为我们的项目无法从 XXCODEBASE 中导入一个函数或类。您可以尝试运行相应的代码行来查看发生了什么。一个可能的原因是，在 OpenMMLAB 的某些代码库中，您需要在安装它们之前先安装 mmcv 和 mmengine。您可以按照[教程](https://mmaction2.readthedocs.io/en/latest/get_started/installation.html#installation)来安装它们。

## 数据

## 数据

- **FileNotFound 错误，例如 `No such file or directory: xxx/xxx/img_00300.jpg`**

  在我们的仓库中，我们将 `start_index=1` 设置为 rawframe 数据集的默认值，将 `start_index=0` 设置为视频数据集的默认值。如果用户遇到数据的第一帧或最后一帧的 FileNotFound 错误，需要检查以 0 或 1 作为偏移量开始的文件，例如 `xxx_00000.jpg` 或 `xxx_00001.jpg`，然后在配置文件中更改数据处理流水线的 `start_index` 值。

- **我们应该如何预处理数据集中的视频？将它们调整为固定大小（所有视频的高宽比相同），例如 `340x256`（1），还是调整它们使得所有视频的短边具有相同的长度（256px 或 320px）（2）？**

  我们尝试过这两种预处理方法，并发现（2）通常是更好的解决方案，因此我们使用（2）作为默认的预处理设置，短边长度为 256px。我们对这些预处理方法进行了基准测试，您可以在[TSN 数据基准测试](https://github.com/open-mmlab/mmaction2/tree/master/configs/recognition/tsn)和[SlowOnly 数据基准测试](https://github.com/open-mmlab/mmaction2/tree/master/configs/recognition/slowonly)中找到结果。

- **数据处理流水线中的项不匹配导致出现类似 `KeyError: 'total_frames'` 的错误**

  我们有用于处理视频和帧的两个处理流水线。

  **对于视频**，我们应该在处理流水线中动态解码视频，所以在这种情况下应该使用 `DecordInit & DecordDecode`、`OpenCVInit & OpenCVDecode` 或 `PyAVInit & PyAVDecode` 这样的配对，例如[这个示例](https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py#L14-L16)。

  **对于帧**，图像已经在离线状态下解码，所以在这种情况下应该使用 `RawFrameDecode` 这样的处理流水线项，例如[这个示例](https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/trn/trn_imagenet-pretrained-r50_8xb16-1x1x8-50e_sthv1-rgb.py#L17)。

  `KeyError: 'total_frames'` 是由于错误地将 `RawFrameDecode` 步骤用于视频，因为当输入是视频时，无法预先获取 `total_frames`。

## 训练

- **如何只使用训练好的识别模型进行主干网络的预训练？**

  为了使用预训练模型进行整个网络的训练，新的配置文件在 `load_from` 中添加了预训练模型的链接。

  要使用主干进行预训练，可以将配置文件中主干部分的 `pretrained` 值更改为权重路径/URL。在训练时，未预料到的键将被忽略。

- **在微调模型时如何固定主干的某些阶段？**

## 训练

  您可以参考 [`def _freeze_stages()`](https://github.com/open-mmlab/mmaction2/blob/main/mmaction/models/backbones/resnet3d.py#L791) 和 [`frozen_stages`](https://github.com/open-mmlab/mmaction2/blob/main/mmaction/models/backbones/resnet3d.py#L369-L370)。
  提醒在配置文件中设置 `find_unused_parameters = True`，以进行分布式训练或测试。

  实际上，除了少数模型，如 C3D 等，用户可以设置 `frozen_stages` 来冻结主干的阶段，因为几乎所有继承自 `ResNet` 和 `ResNet3D` 的主干都支持内部函数 `_freeze_stages()`。

- **如何在配置文件中设置 memcached ？**

  在 MMAction2 中，您可以将 memcached 的参数传递给用于视频数据集的 `class DecordInit` 或用于原始帧数据集的 `RawFrameDecode`。有关更多细节，请参阅 MMEngine 中的 [`class FileClient`](https://github.com/open-mmlab/mmaction2/blob/main/mmaction/data/pipelines/file_client.py)。以下是一个示例，演示如何在原始帧数据集中使用 memcached：

  ```python

```python
  mc_cfg = dict(server_list_cfg='server_list_cfg', client_cfg='client_cfg', sys_path='sys_path')

    ...
    dict(type='RawFrameDecode', io_backend='memcached', **mc_cfg),
    ...
  ]
  ```

- **如何在配置文件中设置 `load_from` 的值以微调模型？**

  在 MMAction2 中，我们将 `load_from=None` 设置为 `configs/_base_/default_runtime.py` 中的默认值，并且由于[继承设计](https://github.com/open-mmlab/mmaction2/tree/main/docs/en/user_guides/config.md)，用户可以直接通过在其配置文件中设置 `load_from` 来更改它。

- **如何在训练时使用 `RawFrameDataset`？**

  在 MMAction2 1.x 版本中，大多数配置文件默认使用 `VideoDataset` 作为数据集类型，这对于文件存储更加友好。如果您想使用 `RawFrameDataset`，需要进行两个修改步骤：

  - `dataset` 相关：
    将 `train_dataloader`/`val_dataloader`/`test_dataloader` 中的 `dataset` 从

    ```
        type=VideoDataset,
        data_prefix=dict(video=xxx),
        ...)
    ```

    修改为

    ```
        type=RawFrameDataset,
        data_prefix=dict(img=xxx),
        filename_tmpl='{:05}.jpg',
        ...)
    ```

...)
    ```

    修改为

    ```
        ...)
    ```

    数据集的其他字段不需要修改。请确保 `filename_tmpl` 与帧数据匹配，并参考[配置文件文档](../user_guides/config.md)了解更多关于配置文件的详细信息。

  - `transform` 相关：在 `train_pipeline`/`val_pipeline`/`test_pipeline` 中删除 `dict(type='DecordInit', **file_client_args)`，将 `dict(type='DecordDecode')` 修改为 `dict(type='RawFrameDecode', **file_client_args)`，并确保在配置文件中定义了 `file_client_args = dict(io_backend='disk')`。

  有关自定义数据集的更多修改，请参考[准备数据集](../user_guides/prepare_dataset.md)和[自定义数据集](../advanced_guides/customize_dataset.md)。

## 测试

- **如何使预测得分在 softmax 内归一化到 \[0, 1\] ?**

  在配置文件中将 `model.cls_head.average_clips` 设置为 `'prob'`。

- **如果模型过大，GPU 内存无法容纳甚至只有一个测试样本怎么办？**

## 测试

  默认情况下，3D 模型使用 10 个 clips x 3 个 crops 进行测试，总共有 30 个视图。对于非常大的模型，即使只有一个测试样本，GPU 内存也无法容纳（因为有 30 个视图）。为了解决这个问题，您可以在配置文件的 `model['test_cfg']` 中设置 `max_testing_views=n`。这样，在前向传播过程中，会使用 n 个视图作为一个批次，以节省 GPU 内存的使用。

```xml
<image>
  <imageName>img/14_03.jpg</imageName>
  <address>341 Southwest 10th Avenue Portland OR</address>
  <lex>
  LIVING,ROOM,THEATERS,KENNY,ZUKE,DELICATESSEN,CLYDE,COMMON,ACE,HOTEL,PORTLAND,ROSE,CITY,BOOKS,STUMPTOWN,COFFEE,ROASTERS,RED,CAP,GARAGE,FISH,GROTTO,SEAFOOD,RESTAURANT,AURA,RESTAURANT,LOUNGE,ROCCO,PIZZA,PASTA,BUFFALO,EXCHANGE,MARK,SPENCER,LIGHT,FEZ,BALLROOM,READING,FRENZY,ROXY,SCANDALS,MARTINOTTI,CAFE,DELI,CROWSENBERG,HALF
  </lex>
  <Resolution x="1280" y="880"/>
  <taggedRectangles>
    <taggedRectangle height="75" width="236" x="375" y="253">
      <tag>LIVING</tag>
    </taggedRectangle>
    <taggedRectangle height="76" width="175" x="639" y="272">
      <tag>ROOM</tag>
    <taggedRectangle height="87" width="281" x="839" y="283">
      <tag>THEATERS</tag>
  </taggedRectangles>
</image>
```

# PPLNN Support

MMDeploy supports ppl.nn v0.8.1 and later. This tutorial is based on Linux systems like Ubuntu-18.04.

1. Please install [pyppl](https://github.com/openppl-public/ppl.nn) following [install-guide](https://github.com/openppl-public/ppl.nn/blob/master/docs/en/building-from-source.md).

2. Install MMDeploy following the [instructions](../01-how-to-build/build_from_source.md).

## Usage

Example:

```bash
python tools/deploy.py \
    configs/mmdet/detection/detection_pplnn_dynamic-800x1344.py \
    /mmdetection_dir/mmdetection/configs/retinanet/retinanet_r50_fpn_1x_coco.py \
    /tmp/snapshots/retinanet_r50_fpn_1x_coco_20200130-c2398f9e.pth \
    --work-dir ../deploy_result \
    --device cuda \
    --log-level INFO
```

# 精度速度测试结果

## Backends

CPU: ncnn, ONNXRuntime, OpenVINO

GPU: ncnn, TensorRT, PPLNN

## 软硬件环境

- Ubuntu 18.04
- ncnn 20211208
- Cuda 11.3
- TensorRT 7.2.3.4
- Docker 20.10.8
- NVIDIA tesla T4 tensor core GPU for TensorRT

### 配置

- 静态图导出
- batch 大小为 1
- 测试时，计算各个数据集中 100 张图片的平均耗时

用户可以直接通过[model profiling](../02-how-to-run/profile_model.md)获得想要的速度测试结果。下面是我们环境中的测试结果：

## 速度测试

<div style="margin-left: 25px;">
<thead>
  <tr>
    <th align="center" colspan="2">mmpretrain</th>
    <th align="center" colspan="5">TensorRT(ms)</th>
    <th align="center" colspan="2">PPLNN(ms)</th>
    <th align="center" colspan="2">ncnn(ms)</th>
    <th align="center" colspan="1">Ascend(ms)</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td align="center" colspan="1" rowspan="2">model</td>
    <td align="center" colspan="1" rowspan="2">spatial</td>
    <td align="center" colspan="3">T4</td>
    <td align="center" colspan="2">JetsonNano2GB</td>

<td align="center" colspan="1" rowspan="2">spatial</td>
    <td align="center" colspan="1">Jetson TX2</td>
    <td align="center" colspan="1">T4</td>
    <td align="center" colspan="1">SnapDragon888</td>
    <td align="center" colspan="1">Adreno660</td>
    <td align="center" colspan="1">Ascend310</td>
  </tr>
  <tr>
    <td align="center" colspan="1">fp32</td>
    <td align="center" colspan="1">fp16</td>
    <td align="center" colspan="1">int8</td>
  </tr>
  <tr>

<td align="center" colspan="1">fp16</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmpretrain/blob/main/configs/resnet/resnet50_8xb32_in1k.py"> ResNet </a></td>
    <td align="center">224x224</td>
    <td align="center">2.97</td>
    <td align="center">1.26</td>
    <td align="center">1.21</td>
    <td align="center">59.32</td>
    <td align="center">30.54</td>
    <td align="center">24.13</td>
    <td align="center">1.30</td>
    <td align="center">33.91</td>
    <td align="center">25.93</td>
    <td align="center">2.49</td>
  </tr>
  <tr>
    <td align="center"> <a href="https://github.com/open-mmlab/mmpretrain/blob/main/configs/resnext/resnext50-32x4d_8xb32_in1k.py"> ResNeXt </a></td>
    <td align="center">4.31</td>
    <td align="center">1.42</td>

<td align="center">224x224</td>
    <td align="center">1.37</td>
    <td align="center">88.10</td>
    <td align="center">49.18</td>
    <td align="center">37.45</td>
    <td align="center">1.36</td>
    <td align="center">133.44</td>
    <td align="center">69.38</td>
    <td align="center">-</td>
  </tr>
  <tr>
    <td align="center"> <a href="https://github.com/open-mmlab/mmpretrain/blob/main/configs/seresnet/seresnet50_8xb32_in1k.py">  SE-ResNet </a></td>
    <td align="center">3.41</td>
    <td align="center">1.66</td>
    <td align="center">1.51</td>
    <td align="center">74.59</td>
    <td align="center">48.78</td>
    <td align="center">29.62</td>
    <td align="center">1.91</td>
    <td align="center">107.84</td>
    <td align="center">80.85</td>
  </tr>
  <tr>

<td align="center">29.62</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmpretrain/blob/main/configs/shufflenet_v2/shufflenet-v2-1x_16xb64_in1k.py">  ShuffleNetV2 </a></td>
    <td align="center">1.19</td>
    <td align="center">1.13</td>
    <td align="center">15.26</td>
    <td align="center">10.23</td>
    <td align="center">7.37</td>
    <td align="center">4.69</td>
    <td align="center">9.55</td>
    <td align="center">10.66</td>
  </tr>
</tbody>
</table>
<thead>
  <tr>
    <th align="center" colspan="2">mmdet part1</th>
    <th align="center" colspan="4">TensorRT(ms)</th>
    <th align="center" colspan="1">PPLNN(ms)</th>
  </tr>
</thead>
<tbody>
  <tr>

<tr>
  </tr>
</thead>
<tbody>
  <tr>
    <td align="center" rowspan="2" colspan="1">model</td>
    <td align="center" rowspan="2" colspan="1">spatial</td>
  </tr>
  <tr>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/main/configs/yolo/yolov3_d53_320_273e_coco.py">YOLOv3</a></td>
    <td align="center">320x320</td>
    <td align="center">14.76</td>
    <td align="center">24.92</td>

<td align="center">320x320</td>
    <td align="center">18.07</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/main/configs/ssd/ssdlite_mobilenetv2_scratch_600e_coco.py">SSD-Lite</a></td>
    <td align="center">8.84</td>
    <td align="center">9.21</td>
    <td align="center">8.04</td>
    <td align="center">1.28</td>
    <td align="center">19.72</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/main/configs/retinanet/retinanet_r50_fpn_1x_coco.py">RetinaNet</a></td>
    <td align="center">800x1344</td>
    <td align="center">97.09</td>
    <td align="center">25.79</td>
    <td align="center">16.88</td>
    <td align="center">780.48</td>
    <td align="center">38.34</td>
  </tr>
  <tr>

<td align="center">97.09</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/main/configs/fcos/fcos_r50_caffe_fpn_gn-head_1x_coco.py">FCOS</a></td>
    <td align="center">84.06</td>
    <td align="center">23.15</td>
    <td align="center">17.68</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/main/configs/fsaf/fsaf_r50_fpn_1x_coco.py">FSAF</a></td>
    <td align="center">82.96</td>
    <td align="center">21.02</td>
    <td align="center">13.50</td>
    <td align="center">30.41</td>
  </tr>
  <tr>

<td align="center">82.96</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/main/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py">Faster R-CNN</a></td>
    <td align="center">88.08</td>
    <td align="center">26.52</td>
    <td align="center">19.14</td>
    <td align="center">733.81</td>
    <td align="center">65.40</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/main/configs/mask_rcnn/mask_rcnn_r50_fpn_1x_coco.py">Mask R-CNN</a></td>
    <td align="center">104.83</td>
    <td align="center">58.27</td>
    <td align="center">86.80</td>
  </tr>
</tbody>
</table>
</div>

<table>
<thead>

<td align="center">-</td>
  </tr>
</tbody>
</table>
</div>

<table>
<thead>
  <tr>
    <th align="center" colspan="2">mmdet part2</th>
    <th align="center" colspan="2">ncnn</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td align="center" rowspan="2">model</td>
    <td align="center" rowspan="2">spatial</td>
  </tr>
  <tr>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/main/configs/yolo/yolov3_mobilenetv2_mstrain-416_300e_coco.py">MobileNetv2-YOLOv3</a></td>
    <td align="center">48.57</td>
    <td align="center">66.55</td>
  </tr>
  <tr>

  </tr>
  <tr>
    <td align="center">44.91</td>
    <td align="center">66.19</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/main/configs/yolox/yolox_tiny_8x8_300e_coco.py">YOLOX</a></td>
    <td align="center">416x416</td>
    <td align="center">111.60</td>
    <td align="center">134.50</td>
  </tr>
</tbody>
</table>
</div>

<thead>
  <tr>
    <th align="center" colspan="2">mmagic</th>
  </tr>
</thead>
<tbody>
  <tr>

<th align="center" colspan="4">TensorRT(ms)</th>
  </tr>
</thead>
<tbody>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmagic/blob/main/configs/esrgan/esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py">ESRGAN</a></td>
    <td align="center">32x32</td>
    <td align="center">12.64</td>
    <td align="center">12.42</td>
    <td align="center">12.45</td>
    <td align="center">7.67</td>
  </tr>
  <tr>

<td align="center">12.64</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmagic/blob/main/configs/srcnn/srcnn_x4k915_1xb16-1000k_div2k.py">SRCNN</a></td>
    <td align="center">0.70</td>
    <td align="center">0.35</td>
    <td align="center">0.26</td>
    <td align="center">58.86</td>
    <td align="center">0.56</td>
  </tr>
</tbody>
</table>
</div>

<thead>
  <tr>
    <th align="center" colspan="2">mmocr</th>
    <th align="center" colspan="3">TensorRT(ms)</th>
  </tr>
</thead>
<tbody>
  <tr>

</tr>
</thead>
<tbody>
  <tr>
  </tr>
  <tr>
  </tr>
    <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py">DBNet</a></td>
    <td align="center">640x640</td>
    <td align="center">10.70</td>
    <td align="center">5.62</td>
    <td align="center">5.00</td>
    <td align="center">34.84</td>

<td align="center">640x640</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmocr/blob/main/configs/textrecog/crnn/crnn_mini-vgg_5e_mj.py">CRNN</a></td>
    <td align="center">1.93 </td>
    <td align="center">1.40</td>
    <td align="center">10.57</td>
    <td align="center">20.00</td>
</tbody>
</table>
</div>

<thead>
  <tr>
    <th align="center" colspan="2">mmseg</th>
  </tr>
</thead>
<tbody>
  <tr>

<th align="center" colspan="1">PPLNN(ms)</th>
  </tr>
</thead>
<tbody>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/fcn/fcn_r50-d8_4xb2-40k_cityscapes-512x1024.py">FCN</a></td>
    <td align="center">512x1024</td>
    <td align="center">128.42</td>
    <td align="center">23.97</td>
    <td align="center">18.13</td>
    <td align="center">1682.54</td>
    <td align="center">27.00</td>
  </tr>
  <tr>

<td align="center">128.42</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/pspnet/pspnet_r50-d8_4xb2-80k_cityscapes-512x1024.py">PSPNet</a></td>
    <td align="center">1x3x512x1024</td>
    <td align="center">119.77</td>
    <td align="center">24.10</td>
    <td align="center">16.33</td>
    <td align="center">1586.19</td>
    <td align="center">27.26</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/deeplabv3/deeplabv3/deeplabv3_r50-d8_4xb2-80k_cityscapes-512x1024.py">DeepLabV3</a></td>
    <td align="center">226.75</td>
    <td align="center">31.80</td>
    <td align="center">19.85</td>
    <td align="center">36.01</td>
  </tr>
  <tr>

<td align="center">226.75</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/deeplabv3plus/deeplabv3plus_r50-d8_4xb2-80k_cityscapes-512x1024.py">DeepLabV3+</a></td>
    <td align="center">151.25</td>
    <td align="center">47.03</td>
    <td align="center">50.38</td>
    <td align="center">2534.96</td>
    <td align="center">34.80</td>
  </tr>
</tbody>
</table>
</div>

## 精度测试

<thead>
  <tr>
    <th align="center">PyTorch</th>
    <th align="center">TorchScript</th>
    <th align="center">ONNX Runtime</th>
    <th align="center" colspan="3">TensorRT</th>
    <th align="center">PPLNN</th>
    <th align="center">Ascend</th>
  </tr>
</thead>

<th align="center">ONNX Runtime</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td align="center">model</td>
    <td align="center">metric</td>
    <td align="center">fp32</td>
    <td align="center">fp16</td>
    <td align="center">int8</td>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmpretrain/blob/main/configs/resnet/resnet18_8xb32_in1k.py">ResNet-18</a></td>
    <td align="center">top-1</td>
    <td align="center">69.90</td>
    <td align="center">69.88</td>
    <td align="center">69.86</td>
    <td align="center">69.91</td>

<td align="center">69.88</td>
  </tr>
  <tr>
    <td align="center">top-5</td>
    <td align="center">89.43</td>
    <td align="center">89.34</td>
    <td align="center">89.33</td>
    <td align="center">89.38</td>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmpretrain/blob/main/configs/resnext/resnext50-32x4d_8xb32_in1k.py">ResNeXt-50</a></td>
    <td align="center">77.90</td>
    <td align="center">77.78</td>
    <td align="center">77.89</td>
  </tr>
  <tr>

<td align="center">77.90</td>
  </tr>
  <tr>
    <td align="center">93.66</td>
    <td align="center">93.64</td>
    <td align="center">93.65</td>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmpretrain/blob/main/configs/seresnet/seresnext50-32x4d_8xb32_in1k.py">SE-ResNet-50</a></td>
    <td align="center">77.74</td>
    <td align="center">77.75</td>
    <td align="center">77.63</td>
    <td align="center">77.73</td>
  </tr>
  <tr>

<td align="center">77.74</td>
  </tr>
  <tr>
    <td align="center">93.84</td>
    <td align="center">93.83</td>
    <td align="center">93.72</td>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmpretrain/blob/main/configs/shufflenet_v1/shufflenet-v1-1x_16xb64_in1k.py">ShuffleNetV1 1.0x</a></td>
    <td align="center">68.13</td>
    <td align="center">67.71</td>
    <td align="center">68.11</td>
  </tr>
  <tr>

<td align="center">68.13</td>
  </tr>
  <tr>
    <td align="center">87.81</td>
    <td align="center">87.58</td>
    <td align="center">87.80</td>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmpretrain/blob/main/configs/shufflenet_v2/shufflenet-v2-1x_16xb64_in1k.py">ShuffleNetV2 1.0x</a></td>
    <td align="center">69.55</td>
    <td align="center">69.54</td>
    <td align="center">69.10</td>
  </tr>
  <tr>

<td align="center">69.55</td>
  </tr>
  <tr>
    <td align="center">88.92</td>
    <td align="center">88.91</td>
    <td align="center">88.58</td>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmpretrain/blob/main/configs/mobilenet_v2/mobilenet-v2_8xb32_in1k.py">MobileNet V2</a></td>
    <td align="center">71.86</td>
    <td align="center">71.87</td>
    <td align="center">70.91</td>
    <td align="center">71.84</td>
  </tr>
  <tr>

<td align="center">71.86</td>
  </tr>
  <tr>
    <td align="center">90.42</td>
    <td align="center">90.40</td>
    <td align="center">89.85</td>
    <td align="center">90.41</td>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmpretrain/blob/main/configs/vision_transformer/vit-base-p16_ft-64xb64_in1k-384.py">Vision Transformer</a></td>
    <td align="center">85.43</td>
    <td align="center">85.42</td>
  </tr>
  <tr>

<td align="center">85.43</td>
  </tr>
  <tr>
    <td align="center">97.77</td>
    <td align="center">97.76</td>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmpretrain/blob/main/configs/swin_transformer/swin-tiny_16xb64_in1k.py">Swin Transformer</a></td>
    <td align="center">81.18</td>
  </tr>
  <tr>

<td align="center">81.18</td>
  </tr>
  <tr>
    <td align="center">95.61</td>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmpretrain/blob/main/configs/efficientformer/efficientformer-l1_8xb128_in1k.py">EfficientFormer</a></td>
    <td align="center">80.46</td>
    <td align="center">80.45</td>
  </tr>
  <tr>
    <td align="center">94.99</td>

  </tr>
  <tr>
    <td align="center">94.98</td>
  </tr>
  </tr>
</tbody>
</table>
</div>
<thead>
  <tr>
    <th align="center" colspan="4">mmdet</th>
    <th align="center">Pytorch</th>
    <th align="center">ONNXRuntime</th>
    <th align="center">OpenVINO</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td align="center">task</td>
    <td align="center">dataset</td>

</tr>
</thead>
<tbody>
  <tr>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/main/configs/yolo/yolov3_d53_320_273e_coco.py">YOLOV3</a></td>
    <td align="center">Object Detection</td>
    <td align="center">COCO2017</td>
    <td align="center">box AP</td>
    <td align="center">33.7</td>
    <td align="center">33.5</td>

  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/main/configs/ssd/ssd300_coco.py">SSD</a></td>
    <td align="center">25.5</td>
  </tr>
  <tr>

<td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/main/configs/retinanet/retinanet_r50_fpn_1x_coco.py">RetinaNet</a></td>
    <td align="center">36.5</td>
    <td align="center">36.4</td>
    <td align="center">36.3</td>
  </tr>
  <tr>
    <td align="center">36.6</td>

<td align="center">box AP</td>
  </tr>
  <tr>
    <td align="center">37.4</td>
    <td align="center">37.2</td>
  </tr>
  <tr>

<td align="center">37.4</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/3.x/configs/centernet/centernet_r18_8xb16-crop512-140e_coco.py">CenterNet</a></td>
    <td align="center">25.9</td>
    <td align="center">26.0</td>
    <td align="center">25.8</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/main/configs/yolox/yolox_s_8x8_300e_coco.py">YOLOX</a></td>

<td align="center">Object Detection</td>
    <td align="center">40.5</td>
    <td align="center">40.3</td>
    <td align="center">29.3</td>
  </tr>
  <tr>
    <td align="center">37.3</td>
    <td align="center">37.1</td>
  </tr>
  <tr>

<td align="center">37.3</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/main/configs/atss/atss_r50_fpn_1x_coco.py">ATSS</a></td>
    <td align="center">39.4</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/main/configs/cascade_rcnn/cascade_rcnn_r50_caffe_fpn_1x_coco.py">Cascade R-CNN</a></td>

    <td align="center">40.4</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/main/configs/gfl/gfl_r50_fpn_1x_coco.py">GFL</a></td>
    <td align="center">40.2</td>
    <td align="center">40.0</td>
  </tr>
  <tr>

<td align="center">40.2</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/main/configs/reppoints/reppoints_moment_r50_fpn_1x_coco.py">RepPoints</a></td>
    <td align="center">37.0</td>
    <td align="center">36.9</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/main/configs/detr/detr_r50_8x2_150e_coco.py">DETR</a></td>

    <td align="center">40.1</td>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmdetection/tree/main/configs/mask_rcnn/mask_rcnn_r50_fpn_1x_coco.py">Mask R-CNN</a></td>
    <td align="center" rowspan="2">Instance Segmentation</td>
    <td align="center" rowspan="2">COCO2017</td>
    <td align="center">38.2</td>
    <td align="center">38.1</td>
    <td align="center">38.0</td>

  </tr>
  <tr>
    <td align="center">mask AP</td>
    <td align="center">34.7</td>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmdetection/blob/main/configs/swin/mask_rcnn_swin-t-p4-w7_fpn_1x_coco.py">Swin-Transformer</a></td>
    <td align="center">42.7</td>

<td align="center" rowspan="2">COCO2017</td>
    <td align="center">42.5</td>
    <td align="center">37.7</td>
  </tr>
  <tr>
    <td align="center">39.3</td>
    <td align="center">35.4</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo/solo_r50_fpn_1x_coco.py">SOLO</a></td>
    <td align="center">Instance Segmentation</td>

<td align="center">Instance Segmentation</td>
    <td align="center">33.1</td>
    <td align="center">32.7</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solov2/solov2_r50_fpn_1x_coco.py">SOLOv2</a></td>
    <td align="center">34.8</td>
    <td align="center">34.5</td>
  </tr>
</tbody>
</table>
</div>

  </tr>
</tbody>
</table>
</div>
<thead>
  <tr>
    <th align="center" colspan="4">mmagic</th>
  </tr>
</thead>
<tbody>
  <tr>
  </tr>
  <tr>

<td align="center">fp32</td>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmagic/blob/main/configs/srcnn/srcnn_x4k915_1xb16-1000k_div2k.py">SRCNN</a></td>
    <td align="center" rowspan="2">Super Resolution</td>
    <td align="center" rowspan="2">Set5</td>
    <td align="center">PSNR</td>
    <td align="center">28.4316</td>
    <td align="center">28.4120</td>
    <td align="center">28.4323</td>
    <td align="center">28.4286</td>
    <td align="center">28.1995</td>
    <td align="center">28.4311</td>
  </tr>
  <tr>
    <td align="center">SSIM</td>
    <td align="center">0.8099</td>
    <td align="center">0.8106</td>
    <td align="center">0.8097</td>
    <td align="center">0.8096</td>
    <td align="center">0.7934</td>

<td align="center">0.8106</td>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmagic/blob/main/configs/esrgan/esrgan_x4c64b23g32_1xb16-400k_div2k.py">ESRGAN</a></td>
    <td align="center">28.2700</td>
    <td align="center">28.2619</td>
    <td align="center">28.2592</td>
    <td align="center"> - </td>
    <td align="center">28.2624</td>
  </tr>
  <tr>
    <td align="center">0.7778</td>
    <td align="center">0.7784</td>
    <td align="center">0.7764</td>
    <td align="center">0.7774</td>

<td align="center">0.7778</td>
    <td align="center">0.7765</td>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmagic/blob/main/configs/esrgan/esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py">ESRGAN-PSNR</a></td>
    <td align="center">30.6428</td>
    <td align="center">30.6306</td>
    <td align="center">30.6444</td>
    <td align="center">30.6430</td>
    <td align="center">27.0426</td>
  </tr>
  <tr>
    <td align="center">0.8559</td>
    <td align="center">0.8565</td>
    <td align="center">0.8558</td>

</tr>
  <tr>
    <td align="center">0.8557</td>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmagic/blob/main/configs/srgan_resnet/srgan_x4c64b16_1xb16-1000k_div2k.py">SRGAN</a></td>
    <td align="center">27.9499</td>
    <td align="center">27.9252</td>
    <td align="center">27.9408</td>
    <td align="center">27.9388</td>
  </tr>
  <tr>
    <td align="center">0.7846</td>
    <td align="center">0.7851</td>
    <td align="center">0.7839</td>

<td align="center">27.9388</td>
  </tr>
  <tr>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmagic/blob/main/configs/srgan_resnet/msrresnet_x4c64b16_1xb16-1000k_div2k.py">SRResNet</a></td>
    <td align="center">30.2252</td>
    <td align="center">30.2069</td>
    <td align="center">30.2300</td>
    <td align="center">30.2294</td>
  </tr>
  <tr>
    <td align="center">0.8491</td>

<td align="center"> - </td>
  </tr>
  <tr>
    <td align="center">0.8497</td>
    <td align="center">0.8488</td>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmagic/blob/main/configs/real_esrgan/realesrnet_c64b23g32_4xb12-lr2e-4-1000k_df2k-ost.py">Real-ESRNet</a></td>
    <td align="center">28.0297</td>
    <td align="center">27.7016</td>
    <td align="center">27.7049</td>
  </tr>
  <tr>

<td align="center">27.7016</td>
  </tr>
  <tr>
    <td align="center">0.8236</td>
    <td align="center">0.8122</td>
    <td align="center">0.8123</td>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmagic/blob/main/configs/edsr/edsr_x4c64b16_1xb16-300k_div2k.py">EDSR</a></td>
    <td align="center">30.2223</td>
    <td align="center">30.2192</td>
    <td align="center">30.2214</td>
    <td align="center">30.2211</td>
    <td align="center">30.1383</td>

<td align="center">30.2192</td>
  </tr>
  <tr>
    <td align="center">0.8500</td>
    <td align="center">0.8507</td>
    <td align="center">0.8469</td>
  </tr>
</tbody>
</table>
</div>
<thead>
  <tr>
    <th align="center" colspan="4">mmocr</th>
  </tr>
</thead>
<tbody>
  <tr>

<th align="center">PPLNN</th>
  </tr>
</thead>
<tbody>
  <tr>
  </tr>
  <tr>
    <td align="center" rowspan="3"><a href="https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py">DBNet*</a></td>
    <td align="center" rowspan="3">TextDetection</td>
    <td align="center" rowspan="3">ICDAR2015</td>
    <td align="center">recall</td>
    <td align="center">0.7310</td>
    <td align="center">0.7308</td>
    <td align="center">0.7304</td>
    <td align="center">0.7198</td>

<td align="center">recall</td>
    <td align="center">0.7179</td>
    <td align="center">0.7111</td>
    <td align="center">0.7309</td>
  </tr>
  <tr>
    <td align="center">precision</td>
    <td align="center">0.8714</td>
    <td align="center">0.8718</td>
    <td align="center">0.8677</td>
    <td align="center">0.8674</td>
    <td align="center">0.8688</td>
  </tr>
  <tr>
    <td align="center">hmean</td>
    <td align="center">0.7950</td>
    <td align="center">0.7949</td>
    <td align="center">0.7868</td>
    <td align="center">0.7856</td>
    <td align="center">0.7821</td>
  </tr>
 <tr>

<td align="center">0.7868</td>
  </tr>
 <tr>
    <td align="center" rowspan="3"><a href="https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/dbnetpp/dbnetpp_resnet50_fpnc_1200e_icdar2015.py">DBNetpp</a></td>
    <td align="center">0.8209</td>
    <td align="center">0.8199</td>
    <td align="center">0.8204</td>
  </tr>
  <tr>
    <td align="center">0.9079</td>
    <td align="center">0.9117</td>

</tr>
  <tr>
    <td align="center">0.9142</td>
  </tr>
  <tr>
    <td align="center">0.8622</td>
    <td align="center">0.8634</td>
    <td align="center">0.8637</td>
    <td align="center">0.8648</td>
  </tr>
  <tr>
    <td align="center" rowspan="3"><a href="https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/psenet/psenet_resnet50_fpnf_600e_icdar2015.py">PSENet</a></td>

<td align="center" rowspan="3">TextDetection</td>
    <td align="center">0.7526</td>
    <td align="center">0.7520</td>
    <td align="center">0.7496</td>
  </tr>
  <tr>
    <td align="center">0.8669</td>
    <td align="center">0.8668</td>
    <td align="center">0.8550</td>
  </tr>
  <tr>
    <td align="center">0.8057</td>
    <td align="center">0.8054</td>
    <td align="center">0.7989</td>

<td align="center">0.8057</td>
  </tr>
  <tr>
    <td align="center" rowspan="3"><a href="https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/panet/panet_resnet18_fpem-ffm_600e_icdar2015.py">PANet</a></td>
    <td align="center">0.7401</td>
    <td align="center">0.7357</td>
    <td align="center">0.7366</td>
  </tr>
  <tr>
    <td align="center">0.8601</td>

<td align="center">0.7401</td>
  </tr>
  <tr>
    <td align="center">0.8570</td>
    <td align="center">0.8586</td>
  </tr>
  <tr>
    <td align="center">0.7955</td>
    <td align="center">0.7917</td>
    <td align="center">0.7930</td>
  </tr>
  <tr>
    <td align="center" rowspan="3"><a href="https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/textsnake/textsnake_resnet50_fpn-unet_1200e_ctw1500.py">TextSnake</a></td>
    <td align="center" rowspan="3">CTW1500</td>

    <td align="center">0.8052</td>
    <td align="center">0.8055</td>
  </tr>
  <tr>
    <td align="center">0.8535</td>
    <td align="center">0.8538</td>
  </tr>
  <tr>
    <td align="center">0.8286</td>
    <td align="center">0.8290</td>

<td align="center">0.8286</td>
  </tr>
  <tr>
    <td align="center" rowspan="3"><a href="https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_icdar2015.py">MaskRCNN</a></td>
    <td align="center">0.7766</td>
    <td align="center">0.7761</td>
    <td align="center">0.7670</td>
  </tr>
  <tr>
    <td align="center">0.8644</td>

  </tr>
  <tr>
    <td align="center">0.8630</td>
    <td align="center">0.8705</td>
  </tr>
  <tr>
    <td align="center">0.8182</td>
    <td align="center">0.8172</td>
    <td align="center">0.8155</td>
  </tr>
  <tr>
    <td align="center">TextRecognition</td>
    <td align="center">IIIT5K</td>
    <td align="center">acc</td>
    <td align="center">0.8067</td>

<td align="center">TextRecognition</td>
    <td align="center">0.8063</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmocr/blob/main/configs/textrecog/sar/sar_resnet31_parallel-decoder_5e_st-sub_mj-sub_sa_real.py">SAR</a></td>
    <td align="center">0.9517</td>
    <td align="center">0.9287</td>
  </tr>
  <tr>

<td align="center">0.9287</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmocr/blob/main/configs/textrecog/satrn/satrn_shallow-small_5e_st_mj.py">SATRN</a></td>
    <td align="center">0.9470</td>
    <td align="center">0.9487</td>
    <td align="center">0.9483</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmocr/blob/main/configs/textrecog/abinet/abinet_20e_st-an_mj.py">ABINet</a></td>

    <td align="center">0.9603</td>
    <td align="center">0.9563</td>
    <td align="center">0.9573</td>
    <td align="center">0.9507</td>
    <td align="center">0.9510</td>
  </tr>
</tbody>
</table>
</div>
<thead>
  <tr>
    <th align="center" colspan="3">mmseg</th>
  </tr>
</thead>
<tbody>
  <tr>

</thead>
<tbody>
  <tr>
  </tr>
  <tr>
    <td align="center">Cityscapes</td>
    <td align="center">mIoU</td>
    <td align="center">72.25</td>
    <td align="center">72.36</td>
    <td align="center">72.35</td>
    <td align="center">74.19</td>
  </tr>
  <tr>

<td align="center">72.36</td>
  </tr>
  <tr>
    <td align="center">78.55</td>
    <td align="center">78.66</td>
    <td align="center">78.26</td>
    <td align="center">78.24</td>
    <td align="center">77.97</td>
    <td align="center">78.09</td>
    <td align="center">78.67</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/deeplabv3/deeplabv3_r50-d8_4xb2-40k_cityscapes-512x1024.py">deeplabv3</a></td>
    <td align="center">79.09</td>

<td align="center">Cityscapes</td>
    <td align="center">79.12</td>
    <td align="center">78.96</td>
    <td align="center">79.06</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/deeplabv3plus/deeplabv3plus_r50-d8_4xb2-40k_cityscapes-512x1024.py">deeplabv3+</a></td>
    <td align="center">79.61</td>
    <td align="center">79.60</td>
    <td align="center">79.43</td>
    <td align="center">79.51</td>
  </tr>
  <tr>

<td align="center">79.60</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/fastscnn/fast_scnn_8xb4-160k_cityscapes-512x1024.py">Fast-SCNN</a></td>
    <td align="center">70.96</td>
    <td align="center">70.93</td>
    <td align="center">70.92</td>
    <td align="center">66.00</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/unet/unet-s5-d16_fcn_4xb4-160k_cityscapes-512x1024.py">UNet</a></td>

    <td align="center">68.95</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/ann/ann_r50-d8_4xb2-40k_cityscapes-512x1024.py">ANN</a></td>
    <td align="center">77.40</td>
    <td align="center">77.32</td>
  </tr>
  <tr>

<td align="center">77.32</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/apcnet/apcnet_r50-d8_4xb2-40k_cityscapes-512x1024.py">APCNet</a></td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/bisenetv1/bisenetv1_r18-d32_4xb4-160k_cityscapes-1024x1024.py">BiSeNetV1</a></td>
    <td align="center">74.44</td>

    <td align="center">74.43</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/bisenetv2/bisenetv2_fcn_4xb4-160k_cityscapes-1024x1024.py">BiSeNetV2</a></td>
    <td align="center">73.21</td>
  </tr>
  <tr>

<td align="center">73.21</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/cgnet/cgnet_fcn_4xb8-60k_cityscapes-512x1024.py">CGNet</a></td>
    <td align="center">68.25</td>
    <td align="center">68.27</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/emanet/emanet_r50-d8_4xb2-80k_cityscapes-512x1024.py">EMANet</a></td>
    <td align="center">77.59</td>

    <td align="center">77.6</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/encnet/encnet_r50-d8_4xb2-40k_cityscapes-512x1024.py">EncNet</a></td>
    <td align="center">75.67</td>
    <td align="center">75.66</td>
  </tr>
  <tr>

<td align="center">75.66</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/erfnet/erfnet_fcn_4xb4-160k_cityscapes-512x1024.py">ERFNet</a></td>
    <td align="center">71.08</td>
    <td align="center">71.07</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/fastfcn/fastfcn_r50-d32_jpu_aspp_4xb2-80k_cityscapes-512x1024.py">FastFCN</a></td>

  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/gcnet/gcnet_r50-d8_4xb2-40k_cityscapes-512x1024.py">GCNet</a></td>
    <td align="center">77.69</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/icnet/icnet_r18-d8_4xb2-80k_cityscapes-832x832.py">ICNet</a></td>

</tr>
  <tr>
    <td align="center">76.29</td>
    <td align="center">76.36</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/isanet/isanet_r50-d8_4xb2-40k_cityscapes-512x1024.py">ISANet</a></td>
    <td align="center">78.49</td>

  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/ocrnet/ocrnet_hr18s_4xb2-40k_cityscapes-512x1024.py">OCRNet</a></td>
    <td align="center">74.30</td>
    <td align="center">73.66</td>
    <td align="center">73.67</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/point_rend/pointrend_r50_4xb2-80k_cityscapes-512x1024.py">PointRend</a></td>

    <td align="center">76.47</td>
    <td align="center">76.41</td>
    <td align="center">76.42</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/sem_fpn/fpn_r50_4xb2-80k_cityscapes-512x1024.py">Semantic FPN</a></td>
    <td align="center">74.52</td>
  </tr>
  <tr>

<td align="center">74.52</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/stdc/stdc1_in1k-pre_4xb12-80k_cityscapes-512x1024.py">STDC</a></td>
    <td align="center">75.10</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/stdc/stdc2_in1k-pre_4xb12-80k_cityscapes-512x1024.py">STDC</a></td>
    <td align="center">77.17</td>

  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/tree/main/configs/upernet/upernet_r50_4xb2-40k_cityscapes-512x1024.py">UPerNet</a></td>
    <td align="center">77.10</td>
    <td align="center">77.19</td>
    <td align="center">77.18</td>
  </tr>
  <tr>

<td align="center">77.19</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segmenter/segmenter_vit-s_fcn_8xb1-160k_ade20k-512x512.py">Segmenter</a></td>
    <td align="center">ADE20K</td>
    <td align="center">44.32</td>
    <td align="center">44.29</td>
    <td align="center">43.34</td>
    <td align="center">43.35</td>
  </tr>
</tbody>
</table>
</div>
<thead>
  <tr>
    <th align="center" colspan="4">mmpose</th>
    <th align="center" colspan="2">TensorRT</th>

<th align="center">Pytorch</th>
  </tr>
</thead>
<tbody>
  <tr>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmpose/blob/main/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_8xb32-210e_coco-256x192.py">HRNet</a></td>
    <td align="center" rowspan="2">Pose Detection</td>
    <td align="center" rowspan="2">COCO</td>
    <td align="center">AP</td>
    <td align="center">0.748</td>

<td align="center" rowspan="2">Pose Detection</td>
  </tr>
  <tr>
    <td align="center">AR</td>
    <td align="center">0.802</td>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmpose/blob/main/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_litehrnet-30_8xb64-210e_coco-256x192.py">LiteHRNet</a></td>
    <td align="center">0.663</td>

  </tr>
  <tr>
    <td align="center">0.728</td>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmpose/blob/main/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_4xmspn50_8xb32-210e_coco-256x192.py">MSPN</a></td>
    <td align="center">0.762</td>

  </tr>
  <tr>
    <td align="center">0.825</td>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmpose/blob/main/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hourglass52_8xb32-210e_coco-256x256.py">Hourglass</a></td>
    <td align="center">0.717</td>

  </tr>
  <tr>
    <td align="center">0.774</td>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmpose/blob/main/configs/body_2d_keypoint/simcc/coco/simcc_mobilenetv2_wo-deconv-8xb64-210e_coco-256x192.py">SimCC</a></td>
    <td align="center">0.607</td>

    <td align="center">0.608</td>
  </tr>
  <tr>
    <td align="center">0.668</td>
    <td align="center">0.672</td>
  </tr>
</tbody>
</table>
</div>

<thead>
  <tr>
    <th align="center" colspan="4">mmrotate</th>
  </tr>
</thead>
<tbody>
  <tr>

<th align="center" colspan="2">TensorRT</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td align="center">metrics</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmrotate/tree/main/configs/rotated_retinanet/rotated-retinanet-hbox-oc_r50_fpn_1x_dota.py">RotatedRetinaNet</a></td>
    <td align="center">Rotated Detection</td>
    <td align="center">DOTA-v1.0</td>
    <td align="center">mAP</td>
    <td align="center">0.698</td>
    <td align="center">0.697</td>

<td align="center">0.698</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmrotate/tree/1.x/configs/oriented_rcnn/oriented-rcnn-le90_r50_fpn_1x_dota.py">Oriented RCNN</a></td>
    <td align="center">0.756</td>
    <td align="center">0.758</td>
    <td align="center">0.730</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmrotate/blob/1.x/configs/gliding_vertex/gliding-vertex-rbox_r50_fpn_1x_dota.py">GlidingVertex</a></td>

<td align="center">Rotated Detection</td>
    <td align="center">0.732</td>
    <td align="center">0.733</td>
    <td align="center">0.731</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmrotate/blob/1.x/configs/roi_trans/roi-trans-le90_r50_fpn_1x_dota.py">RoI Transformer</a></td>
    <td align="center">0.761</td>
  </tr>
</tbody>
</table>
</div>

<thead>
  <tr>
    <th align="center" colspan="4">mmaction2</th>

</tr>
</tbody>
</table>
</div>

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py">TSN</a></td>
    <td align="center" rowspan="2">Recognition</td>

<td align="center" rowspan="2">Recognition</td>
    <td align="center" rowspan="2">Kinetics-400</td>
    <td align="center">69.71</td>
  </tr>
  <tr>
    <td align="center">88.75</td>
  </tr>
<tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/slowfast/slowfast_r50_8xb8-4x16x1-256e_kinetics400-rgb.py">SlowFast</a></td>
    <td align="center">74.45</td>

<td align="center" rowspan="2">Kinetics-400</td>
    <td align="center">75.62</td>
  </tr>
  <tr>
    <td align="center">91.55</td>
    <td align="center">92.10</td>
  </tr>
</tbody>
</table>
</div>
## 备注

- 由于某些数据集在代码库中包含各种分辨率的图像，例如 MMDet，速度基准是通过 MMDeploy 中的静态配置获得的，而性能基准是通过动态配置获得的
- TensorRT 的一些 int8 性能基准测试需要有 tensor core 的 Nvidia 卡，否则性能会大幅下降
- DBNet 在模型 `neck` 使用了`nearest` 插值，TensorRT-7 用了与 Pytorch 完全不同的策略。为了使与 TensorRT-7 兼容，我们重写了`neck`以使用`bilinear`插值，这提高了检测性能。为了获得与 Pytorch 匹配的性能，推荐使用 TensorRT-8+，其插值方法与 Pytorch 相同。
- 对于 mmpose 模型，在模型配置文件中 `flip_test` 需设置为 `False`
- 部分模型在 fp16 模式下可能存在较大的精度损失，请根据具体情况对模型进行调整。

<img src="https://github.com/hustvl/SparseInst/raw/main/assets/banner.gif">
<br>
<br>
Tianheng Cheng, <a href="https://xinggangw.info/">Xinggang Wang</a><sup><span>&#8224;</span></sup>, Shaoyu Chen, Wenqiang Zhang, <a href="https://scholar.google.com/citations?user=pCY-bikAAAAJ&hl=zh-CN">Qian Zhang</a>, <a href="https://scholar.google.com/citations?user=IyyEKyIAAAAJ&hl=zh-CN">Chang Huang</a>, <a href="https://zhaoxiangzhang.net/">Zhaoxiang Zhang</a>, <a href="http://eic.hust.edu.cn/professor/liuwenyu/"> Wenyu Liu</a>
</br>
(<span>&#8224;</span>: corresponding author)
<div>
<a href="https://arxiv.org/abs/2203.12827">[arXiv paper]</a>
<a href="https://openaccess.thecvf.com/content/CVPR2022/papers/Cheng_Sparse_Instance_Activation_for_Real-Time_Instance_Segmentation_CVPR_2022_paper.pdf">[CVPR paper]</a>
<a href="https://drive.google.com/file/d/1xhqQvQ0YVCHd8XQxnCVqef75Hey7kI-d/view?usp=sharing">[slides]</a>
</div>
</div>

</div>
</div>

This is an implementation of [SparseInst](https://github.com/hustvl/SparseInst) based on [MMDetection](https://github.com/open-mmlab/mmdetection/tree/main), [MMCV](https://github.com/open-mmlab/mmcv), and [MMEngine](https://github.com/open-mmlab/mmengine).

**SparseInst** is a conceptually novel, efficient, and fully convolutional framework for real-time instance segmentation.
In contrast to region boxes or anchors (centers), SparseInst adopts a sparse set of **instance activation maps** as object representation, to highlight informative regions for each foreground objects.
Then it obtains the instance-level features by aggregating features according to the highlighted regions for recognition and segmentation.

The bipartite matching compels the instance activation maps to predict objects in a one-to-one style, thus avoiding non-maximum suppression (NMS) in post-processing. Owing to the simple yet effective designs with instance activation maps, SparseInst has extremely fast inference speed and achieves **40 FPS** and **37.9 AP** on COCO (NVIDIA 2080Ti), significantly outperforms the counter parts in terms of speed and accuracy.

<center>
<img src="https://github.com/hustvl/SparseInst/raw/main/assets/sparseinst.png">
</center>

## Usage

```bash

```bash
python tools/train.py projects/SparseInst/configs/sparseinst_r50_iam_8xb8-ms-270k_coco.py
```

```bash
python -m torch.distributed.launch --nnodes=1 --node_rank=0 --nproc_per_node=${NUM_GPUS} --master_port=29506 --master_addr="127.0.0.1" tools/train.py projects/SparseInst/configs/sparseinst_r50_iam_8xb8-ms-270k_coco.py
```

```bash
python tools/test.py projects/SparseInst/configs/sparseinst_r50_iam_8xb8-ms-270k_coco.py ${CHECKPOINT_PATH}
```

## Results

Here we provide the baseline version of SparseInst with ResNet50 backbone.

To find more variants, please visit the [official model zoo](https://github.com/hustvl/SparseInst#models).

```

## Results

| Backbone |  Style  | Lr schd | Mem (GB) | FPS  | mask AP val2017 |                           Config                            |                                                                                                                                                                    Download                                                                                                                                                                    |

| :------: | :-----: | :-----: | :------: | :--: | :-------------: | :---------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|   R-50   | PyTorch |  270k   |   8.7    | 44.3 |      32.9       | [config](./configs/sparseinst_r50_iam_8xb8-ms-270k_coco.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/sparseinst/sparseinst_r50_iam_8xb8-ms-270k_coco/sparseinst_r50_iam_8xb8-ms-270k_coco_20221111_181051-72c711cd.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/sparseinst/sparseinst_r50_iam_8xb8-ms-270k_coco/sparseinst_r50_iam_8xb8-ms-270k_coco_20221111_181051.json) |

If you find SparseInst is useful in your research or applications, please consider giving a star 🌟 to the [official repository](https://github.com/hustvl/SparseInst) and citing SparseInst by the following BibTeX entry.

```BibTeX
@inproceedings{Cheng2022SparseInst,
  title     =   {Sparse Instance Activation for Real-Time Instance Segmentation},
  author    =   {Cheng, Tianheng and Wang, Xinggang and Chen, Shaoyu and Zhang, Wenqiang and Zhang, Qian and Huang, Chang and Zhang, Zhaoxiang and Liu, Wenyu},
  booktitle =   {Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR)},
  year      =   {2022}
}

```

- [x] Test-time correctness

- [ ] Metafile.yml

# STGCN++

[PYSKL: Towards Good Practices for Skeleton Action Recognition](https://arxiv.org/abs/2205.09443)

We present PYSKL: an open-source toolbox for skeleton-based action recognition based on PyTorch. The toolbox supports a wide variety of skeleton action recognition algorithms, including approaches based on GCN and CNN. In contrast to existing open-source skeleton action recognition projects that include only one or two algorithms, PYSKL implements six different algorithms under a unified framework with both the latest and original good practices to ease the comparison of efficacy and efficiency. We also provide an original GCN-based skeleton action recognition model named ST-GCN++, which achieves competitive recognition performance without any complicated attention schemes, serving as a strong baseline. Meanwhile, PYSKL supports the training and testing of nine skeleton-based action recognition benchmarks and achieves state-of-the-art recognition performance on eight of them. To facilitate future research on skeleton action recognition, we also provide a large number of trained models and detailed benchmark results to give some insights. PYSKL is released at this https URL and is actively maintained. We will update this report when we add new features or benchmarks. The current version corresponds to PYSKL v0.2.

### NTU60_XSub_2D

| frame sampling strategy |   modality   | gpus | backbone | top1 acc | testing protocol | FLOPs | params |                  config                   |                  ckpt                   |                  log                   |
| :---------------------: | :----------: | :--: | :------: | :------: | :--------------: | :---: | :----: | :---------------------------------------: | :-------------------------------------: | :------------------------------------: |

|       uniform 100       |    joint     |  8   | STGCN++  |  89.29   |     10 clips     | 1.95G | 1.39M  | [config](/configs/skeleton/stgcnpp/stgcnpp_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcnpp/stgcnpp_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d/stgcnpp_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d_20221228-86e1e77a.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcnpp/stgcnpp_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d/stgcnpp_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.log) |

|       uniform 100       |     bone     |  8   | STGCN++  |  92.30   |     10 clips     | 1.95G | 1.39M  | [config](/configs/skeleton/stgcnpp/stgcnpp_8xb16-bone-u100-80e_ntu60-xsub-keypoint-2d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcnpp/stgcnpp_8xb16-bone-u100-80e_ntu60-xsub-keypoint-2d/stgcnpp_8xb16-bone-u100-80e_ntu60-xsub-keypoint-2d_20221228-cd11a691.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcnpp/stgcnpp_8xb16-bone-u100-80e_ntu60-xsub-keypoint-2d/stgcnpp_8xb16-bone-u100-80e_ntu60-xsub-keypoint-2d.log) |

|       uniform 100       | joint-motion |  8   | STGCN++  |  87.30   |     10 clips     | 1.95G | 1.39M  | [config](/configs/skeleton/stgcnpp/stgcnpp_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-2d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcnpp/stgcnpp_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-2d/stgcnpp_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-2d_20221228-19a34aba.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcnpp/stgcnpp_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-2d/stgcnpp_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-2d.log) |

|       uniform 100       | bone-motion  |  8   | STGCN++  |  88.76   |     10 clips     | 1.95G | 1.39M  | [config](/configs/skeleton/stgcnpp/stgcnpp_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-2d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcnpp/stgcnpp_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-2d/stgcnpp_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-2d_20221228-c02a0749.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcnpp/stgcnpp_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-2d/stgcnpp_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-2d.log) |
|                         |  two-stream  |      |          |  92.61   |                  |       |        |                                           |                                         |                                        |

|                         | four-stream  |      |          |  92.77   |                  |       |        |                                           |                                         |                                        |

### NTU60_XSub_3D

|       uniform 100       |    joint     |  8   | STGCN++  |  89.14   |     10 clips     | 2.96G |  1.4M  | [config](/configs/skeleton/stgcnpp/stgcnpp_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcnpp/stgcnpp_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d/stgcnpp_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d_20221230-4e455ce3.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcnpp/stgcnpp_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d/stgcnpp_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d.log) |

|       uniform 100       |     bone     |  8   | STGCN++  |  90.21   |     10 clips     | 2.96G |  1.4M  | [config](/configs/skeleton/stgcnpp/stgcnpp_8xb16-bone-u100-80e_ntu60-xsub-keypoint-3d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcnpp/stgcnpp_8xb16-bone-u100-80e_ntu60-xsub-keypoint-3d/stgcnpp_8xb16-bone-u100-80e_ntu60-xsub-keypoint-3d_20221230-7f356072.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcnpp/stgcnpp_8xb16-bone-u100-80e_ntu60-xsub-keypoint-3d/stgcnpp_8xb16-bone-u100-80e_ntu60-xsub-keypoint-3d.log) |

|       uniform 100       | joint-motion |  8   | STGCN++  |  86.67   |     10 clips     | 2.96G |  1.4M  | [config](/configs/skeleton/stgcnpp/stgcnpp_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-3d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcnpp/stgcnpp_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-3d/stgcnpp_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-3d_20221230-650de5cc.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcnpp/stgcnpp_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-3d/stgcnpp_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-3d.log) |

|       uniform 100       | bone-motion  |  8   | STGCN++  |  87.45   |     10 clips     | 2.96G |  1.4M  | [config](/configs/skeleton/stgcnpp/stgcnpp_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-3d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcnpp/stgcnpp_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-3d/stgcnpp_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-3d_20221230-b00440d2.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcnpp/stgcnpp_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-3d/stgcnpp_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-3d.log) |
|                         |  two-stream  |      |          |  91.39   |                  |       |        |                                           |                                         |                                        |

|                         | four-stream  |      |          |  91.87   |                  |       |        |                                           |                                         |                                        |

1. The **gpus** indicates the number of gpus we used to get the checkpoint. If you want to use a different number of gpus or videos per gpu, the best way is to set `--auto-scale-lr` when calling `tools/train.py`, this parameter will auto-scale the learning rate according to the actual batch size, and the original batch size.
2. For two-stream fusion, we use **joint : bone = 1 : 1**. For four-stream fusion, we use **joint : joint-motion : bone : bone-motion = 2 : 1 : 2 : 1**. For more details about multi-stream fusion, please refer to this [tutorial](/docs/en/useful_tools.md#multi-stream-fusion).

## Train

```shell
```

## Train

```shell
```

Example: train STGCN++ model on NTU60-2D dataset in a deterministic option with periodic validation.

```shell
python tools/train.py configs/skeleton/stgcnpp/stgcnpp_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py \
```

## Test

```shell
```

Example: test STGCN++ model on NTU60-2D dataset and dump the result to a pickle file.

```shell
python tools/test.py configs/skeleton/stgcnpp/stgcnpp_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py \
```

```shell
```

```BibTeX
@misc{duan2022PYSKL,
  url = {https://arxiv.org/abs/2205.09443},
  author = {Duan, Haodong and Wang, Jiaqi and Chen, Kai and Lin, Dahua},
  title = {PYSKL: Towards Good Practices for Skeleton Action Recognition},
}
```

# Tutorial 6: Visualization

The visualization of images is an important way to measure the quality of image processing, editing and synthesis.
Using `visualizer` in config file can save visual results when training or testing. You can follow [MMEngine Documents](https://github.com/open-mmlab/mmengine/blob/main/docs/en/advanced_tutorials/visualization.md) to learn the usage of visualization. MMagic provides a rich set of visualization functions.
In this tutorial, we introduce the usage of the visualization functions provided by MMagic.

- [Tutorial 6: Visualization](#tutorial-6-visualization)
    - [Visualization configuration of GANs](#visualization-configuration-of-gans)
    - [Visualization configuration of image translation models](#visualization-configuration-of-image-translation-models)
    - [Visualization configuration of diffusion models](#visualization-configuration-of-diffusion-models)

- [Visualization configuration of diffusion models](#visualization-configuration-of-diffusion-models)
    - [Visualization configuration of inpainting models](#visualization-configuration-of-inpainting-models)
    - [Visualization configuration of matting models](#visualization-configuration-of-matting-models)
    - [Visualization configuration of SISR/VSR/VFI models](#visualization-configuration-of-sisrvsrvfi-models)
  - [Visualization Hook](#visualization-hook)
  - [Visualizer](#visualizer)
  - [VisBackend](#visbackend)

In MMagic, the visualization of the training or testing process requires the configuration of three components: VisualizationHook, Visualizer, and VisBackend.

**VisualizationHook** fetches the visualization results of the model output in fixed intervals during training and passes them to Visualizer.

**Visualizer** is responsible for converting the original visualization results into the desired type (png, gif, etc.) and then transferring them to **VisBackend** for storage or display.

### Visualization configuration of GANs

For GAN models, such as StyleGAN and SAGAN, a usual configuration is shown below:

```python
# VisualizationHook
    dict(
        interval=5000,  # visualization interval
        fixed_input=True,  # whether use fixed noise input to generate images
        vis_kwargs_list=dict(type='GAN', name='fake_img')  # pre-defined visualization arguments for GAN models
    )
]
# VisBackend
    dict(type='VisBackend'),  # vis_backend for saving images to file system
    dict(type='WandbVisBackend',  # vis_backend for uploading images to Wandb

)
]
        init_kwargs=dict(
            project='MMagic',   # project name for Wandb
            name='GAN-Visualization-Demo'  # name of the experiment for Wandb
        ))
]
# Visualizer
visualizer = dict(type='Visualizer', vis_backends=vis_backends)
```

If you apply Exponential Moving Average (EMA) to a generator and want to visualize the EMA model, you can modify config of `VisualizationHook` as below:

```python
    dict(
        interval=5000,
        fixed_input=True,
        # vis ema and orig in `fake_img` at the same time
        vis_kwargs_list=dict(
            type='Noise',
            name='fake_img',  # save images with prefix `fake_img`
            sample_model='ema/orig',  # specified kwargs for `NoiseSampler`

vis_kwargs_list=dict(
            target_keys=['ema.fake_img', 'orig.fake_img']  # specific key to visualization
        ))
]
```

### Visualization configuration of image translation models

For Translation models, such as CycleGAN and Pix2Pix, visualization configs can be formed as below:

```python
    dict(
        vis_kwargs_list=[
            dict(
                type='Translation',  # Visualize results on the training set
                name='trans'),  #  save images with prefix `trans`
            dict(
                type='Translationval',  # Visualize results on the validation set
                name='trans_val'),  #  save images with prefix `trans_val`
        ])
]

dict(
        ])
]
            name='Translation-Visualization-Demo'  # name of the experiment for Wandb
        ))
]
```

### Visualization configuration of diffusion models

For Diffusion models, such as Improved-DDPM, we can use the following configuration to visualize the denoising process through a gif:

```python
    dict(

```python
    dict(
        vis_kwargs_list=dict(type='DDPMDenoising'))  # pre-defined visualization argument for DDPM models
]
            name='Diffusion-Visualization-Demo'  # name of the experiment for Wandb
        ))
]
```

### Visualization configuration of inpainting models

For inpainting models, such as AOT-GAN and Global&Local, a usual configuration is shown below:

```python

```python
```

### Visualization configuration of matting models

For matting models, such as DIM and GCA, a usual configuration is shown below:

```python
    fn_key='trimap_path',
    img_keys=['pred_alpha', 'trimap', 'gt_merged', 'gt_alpha'],
```

### Visualization configuration of SISR/VSR/VFI models

For SISR/VSR/VFI models, such as EDSR, EDVR and CAIN, a usual configuration is shown below:

```python

```

```python
    bgr2rgb=False)
```

The specific configuration of the `VisualizationHook`, `Visualizer` and `VisBackend` components are described below

## Visualization Hook

In MMagic, we use `BasicVisualizationHook` and `VisualizationHook` as `VisualizationHook`.
`VisualizationHook` supports three following cases.

(1) Modify `vis_kwargs_list` to visualize the output of the model under specific inputs , which is suitable for visualization of the generated results of GAN and translation results of Image-to-Image-Translation models under specific data input, etc. Below are two typical examples:

```python
# input as dict
vis_kwargs_list = dict(
    type='Noise',  # use 'Noise' sampler to generate model input
    name='fake_img',  # define prefix of saved images
)

# input as list of dict
vis_kwargs_list = [
    dict(type='Arguments',  # use `Arguments` sampler to generate model input
         name='arg_output',  # define prefix of saved images
         vis_mode='gif',  # specific visualization mode as GIF

name='arg_output',  # define prefix of saved images
         forward_kwargs=dict(forward_mode='sampling', sample_kwargs=dict(show_pbar=True))  # specific kwargs for `Arguments` sampler
    ),
    dict(type='Data',  # use `Data` sampler to feed data in dataloader to model as input
         n_samples=36,  # specific how many samples want to generate
         fixed_input=False,  # specific do not use fixed input for each visualization process
    )
]
```

`vis_kwargs_list` takes dict or list of dict as input. Each of dict must contain a `type` field indicating the **type of sampler** used to generate the model input, and each of the dict must also contain the keyword fields necessary for the sampler (e.g. `ArgumentSampler` requires that the argument dictionary contain `forward_kwargs`).

> To be noted that, this content is checked by the corresponding sampler and is not restricted by `BasicVisualizationHook`.

In addition, the other fields are generic fields (e.g. `n_samples`, `n_row`, `name`, `fixed_input`, etc.).
If not passed in, the default values from the BasicVisualizationHook initialization will be used.

For the convenience of users, MMagic has pre-defined visualization parameters for **GAN**, **Translation models**, **SinGAN** and **Diffusion models**, and users can directly use the predefined visualization methods by using the following configuration:

```python
vis_kwargs_list = dict(type='GAN')
vis_kwargs_list = dict(type='SinGAN')
vis_kwargs_list = dict(type='Translation')
vis_kwargs_list = dict(type='TranslationVal')
vis_kwargs_list = dict(type='TranslationTest')
vis_kwargs_list = dict(type='DDPMDenoising')
```

## Visualizer

In MMagic, we implement `ConcatImageVisualizer` and `Visualizer`, which inherit from `mmengine.Visualizer`.

```

The base class of `Visualizer` is `ManagerMixin` and this makes `Visualizer` a globally unique object.
After being instantiated, `Visualizer` can be called at anywhere of the code by `Visualizer.get_current_instance()`, as shown below:

```python
# configs
vis_backends = [dict(type='VisBackend')]
    type='Visualizer', vis_backends=vis_backends, name='visualizer')
```

```python
# `get_instance()` is called for globally unique instantiation
VISUALIZERS.build(cfg.visualizer)

# Once instantiated by the above code, you can call the `get_current_instance` method at any location to get the visualizer
visualizer = Visualizer.get_current_instance()
```

The core interface of `Visualizer` is `add_datasample`.
Through this interface,

```

This interface will call the corresponding drawing function according to the corresponding `vis_mode` to obtain the visualization result in `np.ndarray` type.
Then `show` or `add_image` will be called to directly show the results or pass the visualization result to the predefined vis_backend.

## VisBackend

In general, users do not need to manipulate `VisBackend` objects, only when the current visualization storage can not meet the needs, users will want to manipulate the storage backend directly.
MMagic supports a variety of different visualization backends, including:

- Basic VisBackend of MMEngine: including LocalVisBackend, TensorboardVisBackend and WandbVisBackend. You can follow [MMEngine Documents](https://github.com/open-mmlab/mmengine/blob/main/docs/en/advanced_tutorials/visualization.md) to learn more about them

- VisBackend: Backend for **File System**. Save the visualization results to the corresponding position.
- TensorboardVisBackend: Backend for **Tensorboard**. Send the visualization results to Tensorboard.
- PaviVisBackend: Backend for **Pavi**. Send the visualization results to Tensorboard.
- WandbVisBackend: Backend for **Wandb**. Send the visualization results to Tensorboard.

One `Visualizer` object can have access to any number of VisBackends and users can access to the backend by their class name in their code.

```python
# configs
vis_backends = [dict(type='Visualizer'), dict(type='WandbVisBackend')]
```

```python
# code

# access to the backend by class name
gen_vis_backend = visualizer.get_backend('VisBackend')
gen_wandb_vis_backend = visualizer.get_backend('GenWandbVisBackend')
```

```

When there are multiply VisBackend with the same class name, user must specific name for each VisBackend.

```python
# configs
    dict(type='VisBackend', name='gen_vis_backend_1'),
    dict(type='VisBackend', name='gen_vis_backend_2')
]
```

```python
# code

local_vis_backend_1 = visualizer.get_backend('gen_vis_backend_1')
local_vis_backend_2 = visualizer.get_backend('gen_vis_backend_2')
```

# Visualization

Before reading this tutorial, it is recommended to read MMEngine's {external+mmengine:doc}`MMEngine: Visualization <advanced_tutorials/visualization>` documentation to get a first glimpse of the `Visualizer` definition and usage.

In brief, the [`Visualizer`](mmengine.visualization.Visualizer) is implemented in MMEngine to meet the daily visualization needs, and contains three main functions:

- Implement common drawing APIs, such as [`draw_bboxes`](mmengine.visualization.Visualizer.draw_bboxes) which implements bounding box drawing functions, [`draw_lines`](mmengine.visualization.Visualizer.draw_lines) implements the line drawing function.
- Support writing visualization results, learning rate curves, loss function curves, and verification accuracy curves to various backends, including local disks and common deep learning training logging tools such as [TensorBoard](https://www.tensorflow.org/tensorboard) and [Wandb](https://wandb.ai/site).

- Support calling anywhere in the code to visualize or record intermediate states of the model during training or testing, such as feature maps and validation results.

Based on MMEngine's Visualizer, MMOCR comes with a variety of pre-built visualization tools that can be used by the user by simply modifying the following configuration files.

- The `tools/analysis_tools/browse_dataset.py` script provides a dataset visualization function that draws images and corresponding annotations after Data Transforms, as described in [`browse_dataset.py`](useful_tools.md).

- MMEngine implements `LoggerHook`, which uses `Visualizer` to write the learning rate, loss and evaluation results to the backend set by `Visualizer`. Therefore, by modifying the `Visualizer` backend in the configuration file, for example to ` TensorBoardVISBackend` or `WandbVISBackend`, you can implement logging to common training logging tools such as `TensorBoard` or `WandB`, thus making it easy for users to use these visualization tools to analyze and monitor the training process.
- The `VisualizerHook` is implemented in MMOCR, which uses the `Visualizer` to visualize or store the prediction results of the validation or prediction phase into the backend set by the `Visualizer`, so by modifying the `Visualizer` backend in the configuration file, for example, to ` TensorBoardVISBackend` or `WandbVISBackend`, you can implement storing the predicted images to `TensorBoard` or `Wandb`.

## Configuration

Thanks to the use of the registration mechanism, in MMOCR we can set the behavior of the `Visualizer` by modifying the configuration file. Usually, we define the default configuration for the visualizer in `task/_base_/default_runtime.py`, see [configuration tutorial](config.md) for details.

```Python
    type='TextxxxLocalVisualizer', # use different visualizers for different tasks
```

Based on the above example, we can see that the configuration of `Visualizer` consists of two main parts, namely, the type of `Visualizer` and the visualization backend `vis_backends` it uses.

```

- For different OCR tasks, various visualizers are pre-configured in MMOCR, including [`TextDetLocalVisualizer`](mmocr.visualization.TextDetLocalVisualizer), [`TextRecogLocalVisualizer`](mmocr.visualization.TextRecogLocalVisualizer), [`TextSpottingLocalVisualizer`](mmocr.visualization.TextSpottingLocalVisualizer) and [`KIELocalVisualizer`](mmocr.visualization.KIELocalVisualizer). These visualizers extend the basic Visulizer API according to the characteristics of their tasks and implement the corresponding tag information interface `add_datasamples`. For example, users can directly use `TextDetLocalVisualizer` to visualize labels or predictions for text detection tasks.

- MMOCR sets the visualization backend `vis_backend` to the local visualization backend `LocalVisBackend` by default, saving all visualization results and other training information in a local folder.

## Storage

MMOCR uses the local visualization backend [`LocalVisBackend`](mmengine.visualization.LocalVisBackend) by default, and the model loss, learning rate, model evaluation accuracy and visualization The information stored in `VisualizerHook` and `LoggerHook`, including loss, learning rate, evaluation accuracy will be saved to the `{work_dir}/{config_name}/{time}/{vis_data}` folder by default. In addition, MMOCR also supports other common visualization backends, such as `TensorboardVisBackend` and `WandbVisBackend`, and you only need to change the `vis_backends` type in the configuration file to the corresponding visualization backend. For example, you can store data to `TensorBoard` and `Wandb` by simply inserting the following code block into the configuration file.

```Python

```Python
_base_.visualizer.vis_backends = [
    dict(type='WandbVisBackend'),]
```

## Plot

### Plot the prediction results

MMOCR mainly uses [`VisualizationHook`](mmocr.engine.hooks.VisualizationHook) to plot the prediction results of validation and test, by default `VisualizationHook` is off, and the default configuration is as follows.

```Python
visualization=dict( # user visualization of validation and test results
    enable=False,
    interval=1,
    show=False,
    draw_gt=False,
    draw_pred=False)
```

The following table shows the parameters supported by `VisualizationHook`.

| Parameters |                                                  Description                                                  |
| :--------: | :-----------------------------------------------------------------------------------------------------------: |

|   enable   |        The VisualizationHook is turned on and off by the enable parameter, which is the default state.        |
|  interval  | Controls how much iteration to store or display the results of a val or test if VisualizationHook is enabled. |
|    show    |                           Controls whether to visualize the results of val or test.                           |
|  draw_gt   |               Whether the results of val or test are drawn with or without labeling information               |
| draw_pred  |                              whether to draw predictions for val or test results                              |

If you want to enable `VisualizationHook` related functions and configurations during training or testing, you only need to modify the configuration, take `dbnet_resnet18_fpnc_1200e_icdar2015.py` as an example, draw annotations and predictions at the same time, and display the images, the configuration can be modified as follows

```Python
visualization = _base_.default_hooks.visualization
visualization.update(
    dict(enable=True, show=True, draw_gt=True, draw_pred=True))
```

<img src="https://user-images.githubusercontent.com/24622904/187426573-8448c827-1336-4416-aebc-e7fccce362cd.png" height="200"/>
</div>

If you only want to see the predicted result information you can just let `draw_pred=True`

```Python

</div>

```Python
    dict(enable=True, show=True, draw_gt=False, draw_pred=True))
```

<img src="https://user-images.githubusercontent.com/24622904/187428385-e6a23120-6445-4c55-a265-c550da692087.png" height="300"/>
</div>

The `test.py` procedure is further simplified by providing the  `--show` and `--show-dir` parameters to visualize the annotation and prediction results during the test without modifying the configuration.

```Shell
# Show test results
python tools/test.py configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py dbnet_r18_fpnc_1200e_icdar2015/epoch_400.pth --show

# Specify where to store the prediction results
python tools/test.py configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py dbnet_r18_fpnc_1200e_icdar2015/epoch_400.pth --show-dir imgs/
```

```

</div>

<details>

```bibtex
}
```

<details>

```bibtex

}
```

| [pose_mobilenetv2](/configs/face_2d_keypoint/topdown_heatmap/coco_wholebody_face/td-hm_mobilenetv2_8xb32-60e_coco-wholebody-face-256x256.py) |  256x256   | 0.0611 | [ckpt](https://download.openmmlab.com/mmpose/face/mobilenetv2/mobilenetv2_coco_wholebody_face_256x256-4a3f096e_20210909.pth) | [log](https://download.openmmlab.com/mmpose/face/mobilenetv2/mobilenetv2_coco_wholebody_face_256x256_20210909.log.json) |

# Projects based on MMDetection

There are many projects built upon MMDetection.
We list some of them as examples of how to extend MMDetection for your own projects.
As the page might not be completed, please feel free to create a PR to update this page.

## Projects as an extension

Some projects extend the boundary of MMDetection for deployment or other research fields.
They reveal the potential of what MMDetection can do. We list several of them as below.

- [OTEDetection](https://github.com/opencv/mmdetection): OpenVINO training extensions for object detection.
- [MMDetection3d](https://github.com/open-mmlab/mmdetection3d): OpenMMLab's next-generation platform for general 3D object detection.

## Projects of papers

There are also projects released with papers.
Some of the papers are published in top-tier conferences (CVPR, ICCV, and ECCV), the others are also highly influential.

To make this list also a reference for the community to develop and compare new object detection algorithms, we list them following the time order of top-tier conferences.
Methods already supported and maintained by MMDetection are not listed.

- Involution: Inverting the Inherence of Convolution for Visual Recognition, CVPR21. [\[paper\]](https://arxiv.org/abs/2103.06255)[\[github\]](https://github.com/d-li14/involution)
- Multiple Instance Active Learning for Object Detection, CVPR 2021. [\[paper\]](https://openaccess.thecvf.com/content/CVPR2021/papers/Yuan_Multiple_Instance_Active_Learning_for_Object_Detection_CVPR_2021_paper.pdf)[\[github\]](https://github.com/yuantn/MI-AOD)

- Adaptive Class Suppression Loss for Long-Tail Object Detection, CVPR 2021. [\[paper\]](https://arxiv.org/abs/2104.00885)[\[github\]](https://github.com/CASIA-IVA-Lab/ACSL)
- Generalizable Pedestrian Detection: The Elephant In The Room, CVPR2021. [\[paper\]](https://arxiv.org/abs/2003.08799)[\[github\]](https://github.com/hasanirtiza/Pedestron)
- Group Fisher Pruning for Practical Network Compression, ICML2021. [\[paper\]](https://github.com/jshilong/FisherPruning/blob/main/resources/paper.pdf)[\[github\]](https://github.com/jshilong/FisherPruning)
- Overcoming Classifier Imbalance for Long-tail Object Detection with Balanced Group Softmax, CVPR2020. [\[paper\]](http://openaccess.thecvf.com/content_CVPR_2020/papers/Li_Overcoming_Classifier_Imbalance_for_Long-Tail_Object_Detection_With_Balanced_Group_CVPR_2020_paper.pdf)[\[github\]](https://github.com/FishYuLi/BalancedGroupSoftmax)

- Coherent Reconstruction of Multiple Humans from a Single Image, CVPR2020. [\[paper\]](https://jiangwenpl.github.io/multiperson/)[\[github\]](https://github.com/JiangWenPL/multiperson)
- Look-into-Object: Self-supervised Structure Modeling for Object Recognition, CVPR 2020. [\[paper\]](http://openaccess.thecvf.com/content_CVPR_2020/papers/Zhou_Look-Into-Object_Self-Supervised_Structure_Modeling_for_Object_Recognition_CVPR_2020_paper.pdf)[\[github\]](https://github.com/JDAI-CV/LIO)
- Video Panoptic Segmentation, CVPR2020. [\[paper\]](https://arxiv.org/abs/2006.11339)[\[github\]](https://github.com/mcahny/vps)
- D2Det: Towards High Quality Object Detection and Instance Segmentation, CVPR2020. [\[paper\]](http://openaccess.thecvf.com/content_CVPR_2020/html/Cao_D2Det_Towards_High_Quality_Object_Detection_and_Instance_Segmentation_CVPR_2020_paper.html)[\[github\]](https://github.com/JialeCao001/D2Det)

- CentripetalNet: Pursuing High-quality Keypoint Pairs for Object Detection, CVPR2020. [\[paper\]](https://arxiv.org/abs/2003.09119)[\[github\]](https://github.com/KiveeDong/CentripetalNet)
- Learning a Unified Sample Weighting Network for Object Detection, CVPR 2020. [\[paper\]](http://openaccess.thecvf.com/content_CVPR_2020/html/Cai_Learning_a_Unified_Sample_Weighting_Network_for_Object_Detection_CVPR_2020_paper.html)[\[github\]](https://github.com/caiqi/sample-weighting-network)
- Scale-equalizing Pyramid Convolution for Object Detection, CVPR2020. [\[paper\]](https://arxiv.org/abs/2005.03101) [\[github\]](https://github.com/jshilong/SEPC)
- Revisiting the Sibling Head in Object Detector, CVPR2020. [\[paper\]](https://arxiv.org/abs/2003.07540)[\[github\]](https://github.com/Sense-X/TSD)
- PolarMask: Single Shot Instance Segmentation with Polar Representation, CVPR2020. [\[paper\]](https://arxiv.org/abs/1909.13226)[\[github\]](https://github.com/xieenze/PolarMask)

- Hit-Detector: Hierarchical Trinity Architecture Search for Object Detection, CVPR2020. [\[paper\]](https://arxiv.org/abs/2003.11818)[\[github\]](https://github.com/ggjy/HitDet.pytorch)
- ZeroQ: A Novel Zero Shot Quantization Framework, CVPR2020. [\[paper\]](https://arxiv.org/abs/2001.00281)[\[github\]](https://github.com/amirgholami/ZeroQ)
- CBNet: A Novel Composite Backbone Network Architecture for Object Detection, AAAI2020. [\[paper\]](https://aaai.org/Papers/AAAI/2020GB/AAAI-LiuY.1833.pdf)[\[github\]](https://github.com/VDIGPKU/CBNet)
- RDSNet: A New Deep Architecture for Reciprocal Object Detection and Instance Segmentation, AAAI2020. [\[paper\]](https://arxiv.org/abs/1912.05070)[\[github\]](https://github.com/wangsr126/RDSNet)

- Training-Time-Friendly Network for Real-Time Object Detection, AAAI2020. [\[paper\]](https://arxiv.org/abs/1909.00700)[\[github\]](https://github.com/ZJULearning/ttfnet)
- Cascade RPN: Delving into High-Quality Region Proposal Network with Adaptive Convolution, NeurIPS 2019. [\[paper\]](https://arxiv.org/abs/1909.06720)[\[github\]](https://github.com/thangvubk/Cascade-RPN)
- Reasoning R-CNN: Unifying Adaptive Global Reasoning into Large-scale Object Detection, CVPR2019. [\[paper\]](http://openaccess.thecvf.com/content_CVPR_2019/papers/Xu_Reasoning-RCNN_Unifying_Adaptive_Global_Reasoning_Into_Large-Scale_Object_Detection_CVPR_2019_paper.pdf)[\[github\]](https://github.com/chanyn/Reasoning-RCNN)

- Learning RoI Transformer for Oriented Object Detection in Aerial Images, CVPR2019. [\[paper\]](https://arxiv.org/abs/1812.00155)[\[github\]](https://github.com/dingjiansw101/AerialDetection)
- SOLO: Segmenting Objects by Locations. [\[paper\]](https://arxiv.org/abs/1912.04488)[\[github\]](https://github.com/WXinlong/SOLO)
- SOLOv2: Dynamic, Faster and Stronger. [\[paper\]](https://arxiv.org/abs/2003.10152)[\[github\]](https://github.com/WXinlong/SOLO)
- Dense Peppoints: Representing Visual Objects with Dense Point Sets. [\[paper\]](https://arxiv.org/abs/1912.11473)[\[github\]](https://github.com/justimyhxu/Dense-RepPoints)
- IterDet: Iterative Scheme for Object Detection in Crowded Environments. [\[paper\]](https://arxiv.org/abs/2005.05708)[\[github\]](https://github.com/saic-vul/iterdet)
- Cross-Iteration Batch Normalization. [\[paper\]](https://arxiv.org/abs/2002.05712)[\[github\]](https://github.com/Howal/Cross-iterationBatchNorm)

- A Ranking-based, Balanced Loss Function Unifying Classification and Localisation in Object Detection, NeurIPS2020 [\[paper\]](https://arxiv.org/abs/2009.13592)[\[github\]](https://github.com/kemaloksuz/aLRPLoss)
- RelationNet++: Bridging Visual Representations for Object Detection via Transformer Decoder, NeurIPS2020 [\[paper\]](https://arxiv.org/abs/2010.15831)[\[github\]](https://github.com/microsoft/RelationNet2)
- Generalized Focal Loss V2: Learning Reliable Localization Quality Estimation for Dense Object Detection, CVPR2021[\[paper\]](https://arxiv.org/abs/2011.12885)[\[github\]](https://github.com/implus/GFocalV2)
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, ICCV2021[\[paper\]](https://arxiv.org/abs/2103.14030)[\[github\]](https://github.com/SwinTransformer/)

- Focal Transformer: Focal Self-attention for Local-Global Interactions in Vision Transformers, NeurIPS2021[\[paper\]](https://arxiv.org/abs/2107.00641)[\[github\]](https://github.com/microsoft/Focal-Transformer)
- End-to-End Semi-Supervised Object Detection with Soft Teacher, ICCV2021[\[paper\]](https://arxiv.org/abs/2106.09018)[\[github\]](https://github.com/microsoft/SoftTeacher)
- CBNetV2: A Novel Composite Backbone Network Architecture for Object Detection [\[paper\]](http://arxiv.org/abs/2107.00420)[\[github\]](https://github.com/VDIGPKU/CBNetV2)
- Instances as Queries, ICCV2021 [\[paper\]](https://openaccess.thecvf.com/content/ICCV2021/papers/Fang_Instances_As_Queries_ICCV_2021_paper.pdf)[\[github\]](https://github.com/hustvl/QueryInst)

## <a href='https://mmpose.readthedocs.io/zh_CN/latest/'>简体中文</a>

## <a href='https://mmpose.readthedocs.io/en/latest/'>English</a>

# 常用工具

中文内容建设中，暂时请查阅[英文版文档](../../en/user_guides/useful_tools.md)

如果您愿意参与中文文档的翻译与维护，我们团队将十分感谢您的贡献！欢迎加入我们的社区群与我们取得联系，或直接按照 [如何给 MMPose 贡献代码](../contribution_guide.md) 在 GitHub 上提交 Pull Request。

# Preparing HIDE Dataset

```bibtex
}
```

The test datasets can be download from [here](https://drive.google.com/file/d/1XRomKYJF1H92g1EuD06pCQe4o6HlwB7A/).

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── HIDE
```

# Log and Results Analysis

### Introduction of log analysis tool

`tools/analysis_tools/analyze_logs.py` plots curves of given keys according to the log files.

<div align=center><img src="../_static/image/tools/analysis/analyze_log.jpg" style=" width: 75%; height: 30%; "></div>

```shell
python tools/analysis_tools/analyze_logs.py plot_curve  \
    ${JSON_LOGS}  \
    [--keys ${KEYS}]  \
    [--title ${TITLE}]  \
    [--legend ${LEGEND}]  \
    [--backend ${BACKEND}]  \
    [--style ${STYLE}]  \
    [--out ${OUT_FILE}]  \
    [--window-size ${WINDOW_SIZE}]
```

**Description of all arguments**：

- `json_logs` : The paths of the log files, separate multiple files by spaces.
- `--keys` : The fields of the logs to analyze, separate multiple keys by spaces. Defaults to 'loss'.
- `--title` : The title of the figure. Defaults to use the filename.

- `--legend` : The names of legend, the number of which must be equal to `len(${JSON_LOGS}) * len(${KEYS})`. Defaults to use `"${JSON_LOG}-${KEYS}"`.
- `--backend` : The backend of matplotlib. Defaults to auto selected by matplotlib.
- `--style` : The style of the figure. Default to `whitegrid`.
- `--out` : The path of the output picture. If not set, the figure won't be saved.
- `--window-size`: The shape of the display window. The format should be `'W*H'`. Defaults to `'12*7'`.

```{note}
The `--style` option depends on `seaborn` package, please install it before setting it.
```

### How to plot the loss/accuracy curve

We present some examples here to show how to plot the loss curve of accuracy curve by using the `tools/analysis_tools/analyze_logs.py`

#### Plot the loss curve in training.

```shell

```shell
python tools/analysis_tools/analyze_logs.py plot_curve your_log_json --keys loss --legend loss
```

#### Plot the top-1 accuracy and top-5 accuracy curves, and save the figure to results.jpg.

```shell
python tools/analysis_tools/analyze_logs.py plot_curve your_log_json --keys accuracy/top1 accuracy/top5  --legend top1 top5 --out results.jpg
```

#### Compare the top-1 accuracy of two log files in the same figure.

```shell
python tools/analysis_tools/analyze_logs.py plot_curve log1.json log2.json --keys accuracy/top1 --legend exp1 exp2
```

### How to calculate training time

`tools/analysis_tools/analyze_logs.py` can also calculate the training time according to the log files.

```shell
python tools/analysis_tools/analyze_logs.py cal_train_time \
    ${JSON_LOGS}
    [--include-outliers]
```

```shell
```

**Description of all arguments**:

- `--include-outliers` : If set, include the first time record in each epoch (Sometimes the time of the first iteration is longer).

Example:

```shell
python tools/analysis_tools/analyze_logs.py cal_train_time work_dirs/your_exp/20230206_181002/vis_data/scalars.json
```

The output is expected to be like the below.

```text
-----Analyze train time of work_dirs/your_exp/20230206_181002/vis_data/scalars.json-----
slowest epoch 68, average time is 0.3818
fastest epoch 1, average time is 0.3694
time std over epochs is 0.0020
average iter time: 0.3777 s/iter
```

With the `--out` argument in `tools/test.py`, we can save the inference results of all samples as a file.
And with this result file, we can do further analysis.

```

### How to conduct offline metric evaluation

We provide `tools/analysis_tools/eval_metric.py` to enable the user evaluate the model from the prediction files.

```shell
python tools/analysis_tools/eval_metric.py \
      ${RESULT} \
      [--metric ${METRIC_OPTIONS} ...]
```

- `result`:  The output result file in pickle format from `tools/test.py`.
- `--metric`: The metric and options to evaluate the results. You need to specify at least one metric and you
  can also specify multiple `--metric` to use multiple metrics.

Please refer the [Metric Documentation](mmpretrain.evaluation) to find the available metrics and options.

```{note}
In `tools/test.py`, we support using `--out-item` option to select which kind of results will be saved.

```{note}
Please ensure the `--out-item` is not specified or `--out-item=pred` to use this tool.
```

```shell
# Get the prediction results
python tools/test.py configs/resnet/resnet18_8xb16_cifar10.py \
    https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_b16x8_cifar10_20210528-bd6371c8.pth \
    --out results.pkl

# Eval the top-1 and top-5 accuracy
python tools/analysis_tools/eval_metric.py results.pkl --metric type=Accuracy topk=1,5

# Eval the overall accuracy and the class-wise precision, recall, f1-score
python tools/analysis_tools/eval_metric.py results.pkl --metric type=Accuracy \
    --metric type=SingleLabelMetric items=precision,recall,f1-score average=None
```

### How to plot the confusion matrix for the test result

--metric type=SingleLabelMetric items=precision,recall,f1-score average=None
```

We provide `tools/analysis_tools/confusion_matrix.py` to enable the user plot the confusion matrix from the prediction files.

```shell
python tools/analysis_tools/confusion_matrix.py \
      [--out ${OUT}] \
      [--show] \
      [--show-path ${SHOW_PATH}] \
      [--include-values] \
      [--cmap] \
      [--cfg-options ${CFG_OPTIONS} ...] \
```

- `config`: The config file path.
- `result`:  The output result file in pickle format from `tools/test.py`, or a checkpoint file.
- `--out`: The path to save the confusion matrix in pickle format.
- `--show`: Whether to show the confusion matrix plot.
- `--show-path`: The path to save the confusion matrix plot.
- `--include-values`: Whether to show the values in the confusion matrix plot.

- `--cmap`: The color map to plot the confusion matrix.
- `--cfg-options`: If specified, the key-value pair config will be merged into the config file, for more details please refer to [Learn about Configs](../user_guides/config.md)

```{note}
```

```shell

# Save the confusion matrix in a pickle file

https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_b16x8_cifar10_20210528-bd6371c8.pth \

python tools/analysis_tools/confusion_matrix.py configs/resnet/resnet18_8xb16_cifar10.py results.pkl --out cm.pkl

# Show the confusion matrix plot in a graphical window.
python tools/analysis_tools/confusion_matrix.py configs/resnet/resnet18_8xb16_cifar10.py results.pkl --show
```

### How to visualize the prediction results

We can use `tools/analysis_tools/analyze_results.py` to save the images with the highest scores in successful or failed prediction.

```shell
      [--out-dir ${OUT_DIR}] \
      [--rescale-factor ${RESCALE_FACTOR}] \
```

- `config` : The path of the model config file.

[--topk ${TOPK}] \
```

- `result`:  Output result file in json/pickle format from `tools/test.py`.
- `--out_dir`: Directory to store output files.
- `--topk`: The number of images in successful or failed prediction with the highest `topk` scores to save. If not specified, it will be set to 20.
- `--rescale-factor`: Image rescale factor, which is useful if the output is too large or too small (Too small
  images may cause the prediction tag is too vague).

```{note}

```{note}
```

```shell

# Save the top-10 successful and failed predictions. And enlarge the sample images by 10 times.
       configs/resnet/resnet18_8xb16_cifar10.py \
       results.pkl \
       --out-dir output \
       --topk 10 \
       --rescale-factor 10
```

<details>
<summary align="right"><a href="https://proceedings.neurips.cc/paper/2021/hash/3bbfdde8842a5c44a0323518eec97cbe-Abstract.html">HRFormer (NIPS'2021)</a></summary>

```bibtex
@article{yuan2021hrformer,
  title={HRFormer: High-Resolution Vision Transformer for Dense Predict},
  author={Yuan, Yuhui and Fu, Rao and Huang, Lang and Lin, Weihong and Zhang, Chao and Chen, Xilin and Wang, Jingdong},
  journal={Advances in Neural Information Processing Systems},
}
```

<details>

```bibtex

author={Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll{\'a}r, Piotr and Zitnick, C Lawrence},
}
```

| [pose_hrformer_small](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrformer-small_8xb32-210e_coco-256x192.py) |  256x192   | 0.738 |      0.904      |      0.812      | 0.793 |      0.941      | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrformer/hrformer_small_coco_256x192-5310d898_20220316.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrformer/hrformer_small_coco_256x192_20220316.log.json) |
| [pose_hrformer_small](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrformer-small_8xb32-210e_coco-384x288.py) |  384x288   | 0.757 |      0.905      |      0.824      | 0.807 |      0.941      | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrformer/hrformer_small_coco_384x288-98d237ed_20220316.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrformer/hrformer_small_coco_384x288_20220316.log.json) |

| [pose_hrformer_base](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrformer-base_8xb32-210e_coco-256x192.py) |  256x192   | 0.754 |      0.906      |      0.827      | 0.807 |      0.943      | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrformer/hrformer_base_coco_256x192-6f5f1169_20220316.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrformer/hrformer_base_coco_256x192_20220316.log.json) |
| [pose_hrformer_base](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrformer-base_8xb32-210e_coco-384x288.py) |  384x288   | 0.774 |      0.909      |      0.842      | 0.823 |      0.945      | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrformer/hrformer_base_coco_384x288-ecf0758d_20220316.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrformer/hrformer_base_coco_256x192_20220316.log.json) |

<details>

```bibtex
}
```

<details>

```bibtex

}
```

<details>

```bibtex
}
```

journal={arXiv preprint arXiv:1711.06475},
}
```

MMPose supports training model with combined datasets. [coco-aic-merge](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-aic-256x192-merge.py) and [coco-aic-combine](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-aic-256x192-combine.py) are two examples.

- [coco-aic-merge](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-aic-256x192-merge.py) leverages AIC data with partial keypoints as auxiliary data to train a COCO model
- [coco-aic-combine](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-aic-256x192-combine.py) constructs a combined dataset whose keypoints are the union of COCO and AIC keypoints to train a model that predicts keypoints of both datasets.

Evaluation results on COCO val2017 of models trained with solely COCO dataset and combined dataset as shown below. These models are evaluated with detector having human AP of 56.4 on COCO val2017 dataset.

| Train Set                                    | Arch           | Input Size |  AP   | AP<sup>50</sup> | AP<sup>75</sup> |  AR   | AR<sup>50</sup> |                  ckpt                   |                  log                   |
| :------------------------------------------- | :------------- | :--------: | :---: | :-------------: | :-------------: | :---: | :-------------: | :-------------------------------------: | :------------------------------------: |

| [coco](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192.py) | pose_hrnet_w32 |  256x192   | 0.749 |      0.906      |      0.821      | 0.804 |      0.945      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192-81c58e40_20220909.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192_20220909.log) |

| [coco-aic-merge](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-aic-256x192-merge.py) | pose_hrnet_w32 |  256x192   | 0.757 |      0.907      |      0.829      | 0.809 |      0.944      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-aic-256x192-merge-b05435b9_20221025.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-aic-256x192-merge_20221025.log) |

| [coco-aic-combine](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-aic-256x192-combine.py) | pose_hrnet_w32 |  256x192   | 0.756 |      0.906      |      0.826      | 0.807 |      0.943      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-aic-256x192-combine-4ce66880_20221026.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-aic-256x192-combine_20221026.log) |

In this paper we present a new computer vision task, named video instance segmentation. The goal of this new task is simultaneous detection, segmentation and tracking of instances in videos. In words, it is the first time that the image instance segmentation problem is extended to the video domain. To facilitate research on this new task, we propose a large-scale benchmark called YouTube-VIS, which consists of 2883 high-resolution YouTube videos, a 40-category label set and 131k high-quality instance masks. In addition, we propose a novel algorithm called MaskTrack R-CNN for this task. Our new method introduces a new tracking branch to Mask R-CNN to jointly perform the detection, segmentation and tracking tasks simultaneously. Finally, we evaluate the proposed method and several strong baselines on our new dataset. Experimental results clearly demonstrate the advantages of the proposed algorithm and reveal insight for future improvement. We believe the video instance segmentation task will motivate the community along the line of research for video understanding.

  <img src="https://user-images.githubusercontent.com/34888372/142986554-4f6a2630-92bc-43b4-8509-5173be00402d.png"/>
</div>

```latex
@inproceedings{yang2019video,
  title={Video instance segmentation},
  author={Yang, Linjie and Fan, Yuchen and Xu, Ning},
  pages={5188--5197},
}
```

## Results and models of MaskTrack R-CNN on YouTube-VIS 2019 validation dataset

As mentioned in [Issues #6](https://github.com/youtubevos/MaskTrackRCNN/issues/6#issuecomment-502503505) in MaskTrack R-CNN, the result is kind of unstable for different trials, which ranges from 28 AP to 31 AP when using R-50-FPN as backbone.
The checkpoint provided below is the best one from two experiments.

|     Method      | Base detector | Backbone  |  Style  | Lr schd | Mem (GB) | Inf time (fps) |  AP  |                                 Config                                 |                                                                                                                                                                                    Download                                                                                                                                                                                    |

| MaskTrack R-CNN |  Mask R-CNN   | R-50-FPN  | pytorch |   12e   |   1.61   |       -        | 30.2 | [config](masktrack-rcnn_mask-rcnn_r50_fpn_8xb1-12e_youtubevis2019.py)  |   [model](https://download.openmmlab.com/mmtracking/vis/masktrack_rcnn/masktrack_rcnn_r50_fpn_12e_youtubevis2019/masktrack_rcnn_r50_fpn_12e_youtubevis2019_20211022_194830-6ca6b91e.pth) \| [log](https://download.openmmlab.com/mmtracking/vis/masktrack_rcnn/masktrack_rcnn_r50_fpn_12e_youtubevis2019/masktrack_rcnn_r50_fpn_12e_youtubevis2019_20211022_194830.log.json)   |

| MaskTrack R-CNN |  Mask R-CNN   | R-101-FPN | pytorch |   12e   |   2.27   |       -        | 32.2 | [config](masktrack-rcnn_mask-rcnn_r101_fpn_8xb1-12e_youtubevis2019.py) | [model](https://download.openmmlab.com/mmtracking/vis/masktrack_rcnn/masktrack_rcnn_r101_fpn_12e_youtubevis2019/masktrack_rcnn_r101_fpn_12e_youtubevis2019_20211023_150038-454dc48b.pth) \| [log](https://download.openmmlab.com/mmtracking/vis/masktrack_rcnn/masktrack_rcnn_r101_fpn_12e_youtubevis2019/masktrack_rcnn_r101_fpn_12e_youtubevis2019_20211023_150038.log.json) |

| MaskTrack R-CNN |  Mask R-CNN   | X-101-FPN | pytorch |   12e   |   3.69   |       -        | 34.7 | [config](masktrack-rcnn_mask-rcnn_x101_fpn_8xb1-12e_youtubevis2019.py) | [model](https://download.openmmlab.com/mmtracking/vis/masktrack_rcnn/masktrack_rcnn_x101_fpn_12e_youtubevis2019/masktrack_rcnn_x101_fpn_12e_youtubevis2019_20211023_153205-fff7a102.pth) \| [log](https://download.openmmlab.com/mmtracking/vis/masktrack_rcnn/masktrack_rcnn_x101_fpn_12e_youtubevis2019/masktrack_rcnn_x101_fpn_12e_youtubevis2019_20211023_153205.log.json) |

## Results and models of MaskTrack R-CNN on YouTube-VIS 2021 validation dataset

| MaskTrack R-CNN |  Mask R-CNN   | R-50-FPN  | pytorch |   12e   |   1.61   |       -        | 28.7 | [config](masktrack-rcnn_mask-rcnn_r50_fpn_8xb1-12e_youtubevis2021.py)  |   [model](https://download.openmmlab.com/mmtracking/vis/masktrack_rcnn/masktrack_rcnn_r50_fpn_12e_youtubevis2021/masktrack_rcnn_r50_fpn_12e_youtubevis2021_20211026_044948-10da90d9.pth) \| [log](https://download.openmmlab.com/mmtracking/vis/masktrack_rcnn/masktrack_rcnn_r50_fpn_12e_youtubevis2021/masktrack_rcnn_r50_fpn_12e_youtubevis2021_20211026_044948.log.json)   |

| MaskTrack R-CNN |  Mask R-CNN   | R-101-FPN | pytorch |   12e   |   2.27   |       -        | 31.3 | [config](masktrack-rcnn_mask-rcnn_r101_fpn_8xb1-12e_youtubevis2021.py) | [model](https://download.openmmlab.com/mmtracking/vis/masktrack_rcnn/masktrack_rcnn_r101_fpn_12e_youtubevis2021/masktrack_rcnn_r101_fpn_12e_youtubevis2021_20211026_045509-3c49e4f3.pth) \| [log](https://download.openmmlab.com/mmtracking/vis/masktrack_rcnn/masktrack_rcnn_r101_fpn_12e_youtubevis2021/masktrack_rcnn_r101_fpn_12e_youtubevis2021_20211026_045509.log.json) |

| MaskTrack R-CNN |  Mask R-CNN   | X-101-FPN | pytorch |   12e   |   3.69   |       -        | 33.5 | [config](masktrack-rcnn_mask-rcnn_x101_fpn_8xb1-12e_youtubevis2021.py) | [model](https://download.openmmlab.com/mmtracking/vis/masktrack_rcnn/masktrack_rcnn_x101_fpn_12e_youtubevis2021/masktrack_rcnn_x101_fpn_12e_youtubevis2021_20211026_095943-90831df4.pth) \| [log](https://download.openmmlab.com/mmtracking/vis/masktrack_rcnn/masktrack_rcnn_x101_fpn_12e_youtubevis2021/masktrack_rcnn_x101_fpn_12e_youtubevis2021_20211026_095943.log.json) |

Due to the influence of parameters such as learning rate in default configuration file, we recommend using 8 GPUs for training in order to reproduce accuracy. You can use the following command to start the training.

```shell
# Training MaskTrack R-CNN on YouTube-VIS-2021 dataset with following command.
# The number after config file represents the number of GPUs used. Here we use 8 GPUs.
bash tools/dist_train.sh configs/masktrack_rcnn/masktrack-rcnn_mask-rcnn_r50_fpn_8xb1-12e_youtubevis2021.py 8
```

If you want to know about more detailed usage of `train.py/dist_train.sh/slurm_train.sh`,
please refer to this [document](../../docs/en/user_guides/tracking_train_test.md).

If you want to get the results of the [YouTube-VOS](https://youtube-vos.org/dataset/vis/) val/test set, please use the following command to generate result files that can be used for submission. It will be stored in `./youtube_vis_results.submission_file.zip`, you can modify the saved path in `test_evaluator` of the config.

```shell
# The number after config file represents the number of GPUs used.
bash tools/dist_test_tracking.sh configs/masktrack_rcnn/masktrack-rcnn_mask-rcnn_r50_fpn_8xb1-12e_youtubevis2021.py 8 --checkpoint ${CHECKPOINT_PATH}
```

Use a single GPU to predict a video and save it as a video.

```shell

```shell
python demo/mot_demo.py demo/demo_mot.mp4 configs/masktrack_rcnn/masktrack-rcnn_mask-rcnn_r50_fpn_8xb1-12e_youtubevis2021.py  --checkpoint {CHECKPOINT_PATH} --out vis.mp4
```

If you want to know about more detailed usage of `mot_demo.py`, please refer to this [document](../../docs/en/user_guides/tracking_inference.md).

# Imagenet classification with deep convolutional neural networks

<details>
<summary align="right"><a href="https://proceedings.neurips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf">AlexNet (NeurIPS'2012)</a></summary>

```bibtex
@inproceedings{krizhevsky2012imagenet,
  title={Imagenet classification with deep convolutional neural networks},
  author={Krizhevsky, Alex and Sutskever, Ilya and Hinton, Geoffrey E},
  booktitle={Advances in neural information processing systems},
  pages={1097--1105},
  year={2012}
}
```

We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry

<img src="https://user-images.githubusercontent.com/15977946/146537150-39cd128e-185e-48eb-9d6b-7e9a187d2ae5.png">
</div>

# SDMGR

> [Spatial Dual-Modality Graph Reasoning for Key Information Extraction](https://arxiv.org/abs/2103.14470)

Key information extraction from document images is of paramount importance in office automation. Conventional template matching based approaches fail to generalize well to document images of unseen templates, and are not robust against text recognition errors. In this paper, we propose an end-to-end Spatial Dual-Modality Graph Reasoning method (SDMG-R) to extract key information from unstructured document images. We model document images as dual-modality graphs, nodes of which encode both the visual and textual features of detected text regions, and edges of which represent the spatial relations between neighboring text regions. The key information extraction is solved by iteratively propagating messages along graph edges and reasoning the categories of graph nodes. In order to roundly evaluate our proposed method as well as boost the future research, we release a new dataset named WildReceipt, which is collected and annotated tailored for the evaluation of key information extraction from document images of unseen templates in the wild. It contains 25 key information categories, a total of about 69000 text boxes, and is about 2 times larger than the existing public datasets. Extensive experiments validate that all information including visual features, textual features and spatial relations can benefit key information extraction. It has been shown that SDMG-R can effectively extract key information from document images of unseen templates, and obtain new state-of-the-art results on the recent popular benchmark SROIE and our WildReceipt. Our code and dataset will be publicly released.

<img src="https://user-images.githubusercontent.com/22607038/142580689-18edb4d7-f716-475c-b1c1-e2b934658cee.png"/>
</div>

### WildReceipt

|                                 Method                                 |     Modality     | Macro F1-Score |                                               Download                                               |
| :--------------------------------------------------------------------: | :--------------: | :------------: | :--------------------------------------------------------------------------------------------------: |
|   [sdmgr_unet16](/configs/kie/sdmgr/sdmgr_unet16_60e_wildreceipt.py)   | Visual + Textual |     0.890      | [model](https://download.openmmlab.com/mmocr/kie/sdmgr/sdmgr_unet16_60e_wildreceipt/sdmgr_unet16_60e_wildreceipt_20220825_151648-22419f37.pth) \| [log](https://download.openmmlab.com/mmocr/kie/sdmgr/sdmgr_unet16_60e_wildreceipt/20220825_151648.log) |

| [sdmgr_novisual](/configs/kie/sdmgr/sdmgr_novisual_60e_wildreceipt.py) |     Textual      |     0.873      | [model](https://download.openmmlab.com/mmocr/kie/sdmgr/sdmgr_novisual_60e_wildreceipt/sdmgr_novisual_60e_wildreceipt_20220831_193317-827649d8.pth) \| [log](https://download.openmmlab.com/mmocr/kie/sdmgr/sdmgr_novisual_60e_wildreceipt/20220831_193317.log) |

### WildReceiptOpenset

|                                Method                                 | Modality | Edge F1-Score | Node Macro F1-Score | Node Micro F1-Score |                                 Download                                 |
| :-------------------------------------------------------------------: | :------: | :-----------: | :-----------------: | :-----------------: | :----------------------------------------------------------------------: |

| [sdmgr_novisual_openset](/configs/kie/sdmgr/sdmgr_novisual_60e_wildreceipt-openset.py) | Textual  |     0.792     |        0.931        |        0.940        | [model](https://download.openmmlab.com/mmocr/kie/sdmgr/sdmgr_novisual_60e_wildreceipt-openset/sdmgr_novisual_60e_wildreceipt-openset_20220831_200807-dedf15ec.pth) \| [log](https://download.openmmlab.com/mmocr/kie/sdmgr/sdmgr_novisual_60e_wildreceipt-openset/20220831_200807.log) |

```bibtex
@misc{sun2021spatial,
      title={Spatial Dual-Modality Graph Reasoning for Key Information Extraction},
      author={Hongbin Sun and Zhanghui Kuang and Xiaoyu Yue and Chenhao Lin and Wayne Zhang},
      year={2021},
      eprint={2103.14470},
}
```

# CGNet

> [CGNet: A Light-weight Context Guided Network for Semantic Segmentation](https://arxiv.org/abs/1811.08201)

<a href="https://github.com/wutianyiRosun/CGNet">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/backbones/cgnet.py#L187">Code Snippet</a>

The demand of applying semantic segmentation model on mobile devices has been increasing rapidly. Current state-of-the-art networks have enormous amount of parameters hence unsuitable for mobile devices, while other small memory footprint models follow the spirit of classification network and ignore the inherent characteristic of semantic segmentation. To tackle this problem, we propose a novel Context Guided Network (CGNet), which is a light-weight and efficient network for semantic segmentation. We first propose the Context Guided (CG) block, which learns the joint feature of both local feature and surrounding context, and further improves the joint feature with the global context. Based on the CG block, we develop CGNet which captures contextual information in all stages of the network and is specially tailored for increasing segmentation accuracy. CGNet is also elaborately designed to reduce the number of parameters and save memory footprint. Under an equivalent number of parameters, the proposed CGNet significantly outperforms existing segmentation networks. Extensive experiments on Cityscapes and CamVid datasets verify the effectiveness of the proposed approach. Specifically, without any post-processing and multi-scale testing, the proposed CGNet achieves 64.8% mean IoU on Cityscapes with less than 0.5 M parameters. The source code for the complete system can be found at [this https URL](https://github.com/wutianyiRosun/CGNet).

<img src="https://user-images.githubusercontent.com/24582831/142900351-89559574-79cc-4f57-8f69-5d88765ec38d.png" width="80%"/>
</div>

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                   | download                                                                                                                                                                                                                                                                                                           |

| CGNet  | M3N21    | 680x680   |   60000 | 7.5      | 30.51          | V100   | 65.63 |         68.04 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/cgnet/cgnet_fcn_4xb4-60k_cityscapes-680x680.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/cgnet/cgnet_680x680_60k_cityscapes/cgnet_680x680_60k_cityscapes_20201101_110253-4c0b2f2d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/cgnet/cgnet_680x680_60k_cityscapes/cgnet_680x680_60k_cityscapes-20201101_110253.log.json)     |

| CGNet  | M3N21    | 512x1024  |   60000 | 8.3      | 31.14          | V100   | 68.27 |         70.33 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/cgnet/cgnet_fcn_4xb8-60k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/cgnet/cgnet_512x1024_60k_cityscapes/cgnet_512x1024_60k_cityscapes_20201101_110254-124ea03b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/cgnet/cgnet_512x1024_60k_cityscapes/cgnet_512x1024_60k_cityscapes-20201101_110254.log.json) |

```bibtext
@article{wu2020cgnet,
  title={Cgnet: A light-weight context guided network for semantic segmentation},
  author={Wu, Tianyi and Tang, Sheng and Zhang, Rui and Cao, Juan and Zhang, Yongdong},
  volume={30},
  pages={1169--1179},
}
```

# Tutorial 3: Inference with pre-trained models

MMagic provides APIs for you to easily play with state-of-the-art models on your own images or videos.
Specifically, MMagic supports various fundamental generative models, including:
unconditional Generative Adversarial Networks (GANs), conditional GANs, internal learning, diffusion models, etc.
MMagic also supports various applications, including:
image super-resolution, video super-resolution, video frame interpolation, image inpainting, image matting, image-to-image translation, etc.

In this section, we will specify how to play with our pre-trained models.

- [Tutorial 3: Inference with pre-trained models](#tutorial-3-inference-with-pre-trained-models)
  - [Sample images with unconditional GANs](#sample-images-with-unconditional-gans)
  - [Sample images with conditional GANs](#sample-images-with-conditional-gans)
  - [Sample images with diffusion models](#sample-images-with-diffusion-models)

- [Sample images with conditional GANs](#sample-images-with-conditional-gans)
  - [Run a demo of image inpainting](#run-a-demo-of-image-inpainting)
  - [Run a demo of image matting](#run-a-demo-of-image-matting)
  - [Run a demo of image super-resolution](#run-a-demo-of-image-super-resolution)
  - [Run a demo of facial restoration](#run-a-demo-of-facial-restoration)
  - [Run a demo of video super-resolution](#run-a-demo-of-video-super-resolution)
  - [Run a demo of video frame interpolation](#run-a-demo-of-video-frame-interpolation)
  - [Run a demo of image translation models](#run-a-demo-of-image-translation-models)

## Sample images with unconditional GANs

MMagic provides high-level APIs for sampling images with unconditional GANs. Here is an example of building StyleGAN2-256 and obtaining the synthesized images.

```python
from mmagic.apis import init_model, sample_unconditional_model

```python

# Specify the path to model config and checkpoint file
config_file = 'configs/styleganv2/stylegan2_c2_8xb4_ffhq-1024x1024.py'
# you can download this checkpoint in advance and use a local file path.
checkpoint_file = 'https://download.openmmlab.com/mmediting/stylegan2/stylegan2_c2_ffhq_1024_b4x8_20210407_150045-618c9024.pth'

device = 'cuda:0'
# init a generative model
model = init_model(config_file, checkpoint_file, device=device)
# sample images
fake_imgs = sample_unconditional_model(model, 4)
```

Indeed, we have already provided a more friendly demo script to users. You can use [demo/unconditional_demo.py](../../../demo/unconditional_demo.py) with the following commands:

```shell
python demo/unconditional_demo.py \
    [--save-path ${SAVE_PATH}] \
    [--device ${GPU_ID}]
```

```shell
```

Note that more arguments are also offered to customize your sampling procedure. Please use `python demo/unconditional_demo.py --help` to check more details.

## Sample images with conditional GANs

MMagic provides high-level APIs for sampling images with conditional GANs. Here is an example for building SAGAN-128 and obtaining the synthesized images.

```python
from mmagic.apis import init_model, sample_conditional_model

config_file = 'configs/sagan/sagan_woReLUinplace-Glr1e-4_Dlr4e-4_noaug-ndisc1-8xb32-bigGAN-sch_imagenet1k-128x128.py'
checkpoint_file = 'https://download.openmmlab.com/mmediting/sagan/sagan_128_woReLUinplace_noaug_bigGAN_imagenet1k_b32x8_Glr1e-4_Dlr-4e-4_ndisc1_20210818_210232-3f5686af.pth'

# sample images with random label
fake_imgs = sample_conditional_model(model, 4)

# sample images with the same label
fake_imgs = sample_conditional_model(model, 4, label=0)

# sample images with specific labels
fake_imgs = sample_conditional_model(model, 4, label=[0, 1, 2, 3])
```

Indeed, we have already provided a more friendly demo script to users. You can use [demo/conditional_demo.py](../../../demo/conditional_demo.py) with the following commands:

```shell
python demo/conditional_demo.py \
    [--label] ${LABEL} \
    [--samples-per-classes] ${SAMPLES_PER_CLASSES} \
    [--sample-all-classes] \
```

${CHECKPOINT} \
```

If `--label` is not passed, images with random labels would be generated.
If `--label` is passed, we would generate `${SAMPLES_PER_CLASSES}` images for each input label.
If `sample_all_classes` is set true in command line, `--label` would be ignored and the generator will output images for all categories.

Note that more arguments are also offered to customizing your sampling procedure. Please use `python demo/conditional_demo.py --help` to check more details.

## Sample images with diffusion models

MMagic provides high-level APIs for sampling images with diffusion models. Here is an example for building I-DDPM and obtaining the synthesized images.

```python
from mmagic.apis import init_model, sample_ddpm_model

```python

config_file = 'configs/improved_ddpm/ddpm_cosine-hybird-timestep-4k_16xb8-1500kiters_imagenet1k-64x64.py'
checkpoint_file = 'https://download.openmmlab.com/mmediting/improved_ddpm/ddpm_cosine_hybird_timestep-4k_imagenet1k_64x64_b8x16_1500k_20220103_223919-b8f1a310.pth'
fake_imgs = sample_ddpm_model(model, 4)
```

Indeed, we have already provided a more friendly demo script to users. You can use [demo/ddpm_demo.py](https://github.com/open-mmlab/mmagic/blob/main/demo/ddpm_demo.py) with the following commands:

```shell
python demo/ddpm_demo.py \
```

```shell
```

Note that more arguments are also offered to customizing your sampling procedure. Please use `python demo/ddpm_demo.py --help` to check more details.

## Run a demo of image inpainting

You can use the following commands to test images for inpainting.

```shell
python demo/inpainting_demo.py \
    ${CHECKPOINT_FILE} \
    ${MASKED_IMAGE_FILE} \
    ${MASK_FILE} \
    ${SAVE_FILE} \
    [--imshow] \
```

If `--imshow` is specified, the demo will also show image with opencv. Examples:

```shell
    configs/global_local/gl_256x256_8x12_celeba.py \
    https://download.openmmlab.com/mmediting/inpainting/global_local/gl_256x256_8x12_celeba_20200619-5af0493f.pth \
    tests/data/image/celeba_test.png \
    tests/data/image/bbox_mask.png \

https://download.openmmlab.com/mmediting/inpainting/global_local/gl_256x256_8x12_celeba_20200619-5af0493f.pth \
    tests/data/pred/inpainting_celeba.png
```

The predicted inpainting result will be saved in `tests/data/pred/inpainting_celeba.png`.

## Run a demo of image matting

You can use the following commands to test a pair of images and trimap.

```shell
python demo/matting_demo.py \
    ${IMAGE_FILE} \
    ${TRIMAP_FILE} \
```

```shell
    configs/dim/dim_stage3_v16_pln_1x1_1000k_comp1k.py \
    https://download.openmmlab.com/mmediting/mattors/dim/dim_stage3_v16_pln_1x1_1000k_comp1k_SAD-50.6_20200609_111851-647f24b6.pth \
    tests/data/matting_dataset/merged/GT05.jpg \

https://download.openmmlab.com/mmediting/mattors/dim/dim_stage3_v16_pln_1x1_1000k_comp1k_SAD-50.6_20200609_111851-647f24b6.pth \
    tests/data/matting_dataset/trimap/GT05.png \
    tests/data/pred/GT05.png
```

The predicted alpha matte will be saved in `tests/data/pred/GT05.png`.

## Run a demo of image super-resolution

You can use the following commands to test an image for restoration.

```shell
python demo/restoration_demo.py \
    [--device ${GPU_ID}] \
    [--ref-path ${REF_PATH}]
```

```shell
    configs/esrgan/esrgan_x4c64b23g32_g1_400k_div2k.py \
    https://download.openmmlab.com/mmediting/restorers/esrgan/esrgan_x4c64b23g32_1x16_400k_div2k_20200508-f8ccaf3b.pth \
    tests/data/image/lq/baboon_x4.png \

https://download.openmmlab.com/mmediting/restorers/esrgan/esrgan_x4c64b23g32_1x16_400k_div2k_20200508-f8ccaf3b.pth \
    demo/demo_out_baboon.png
```

You can test Ref-SR by providing `--ref-path`. Examples:

```shell
    configs/ttsr/ttsr-gan_x4_c64b16_g1_500k_CUFED.py \
    https://download.openmmlab.com/mmediting/restorers/ttsr/ttsr-gan_x4_c64b16_g1_500k_CUFED_20210626-2ab28ca0.pth \
    tests/data/frames/sequence/gt/sequence_1/00000000.png \
    demo/demo_out.png \
    --ref-path tests/data/frames/sequence/gt/sequence_1/00000001.png
```

## Run a demo of facial restoration

You can use the following commands to test a face image for restoration.

```shell
python demo/restoration_face_demo.py \
    [--upscale-factor] \
    [--face-size] \
```

${CONFIG_FILE} \
```

```shell
    configs/glean/glean_in128out1024_4xb2-300k_ffhq-celeba-hq.py \
    https://download.openmmlab.com/mmediting/restorers/glean/glean_in128out1024_4x2_300k_ffhq_celebahq_20210812-acbcb04f.pth \
    tests/data/image/face/000001.png \
    tests/data/pred/000001.png \
    --upscale-factor 4
```

## Run a demo of video super-resolution

You can use the following commands to test a video for restoration.

```shell
python demo/restoration_video_demo.py \
    ${INPUT_DIR} \
    ${OUTPUT_DIR} \
    [--window-size=${WINDOW_SIZE}] \
```

It supports both the sliding-window framework and the recurrent framework. Examples:

${INPUT_DIR} \
```

EDVR:

```shell
    configs/edvr/edvrm_wotsa_x4_g8_600k_reds.py \
    https://download.openmmlab.com/mmediting/restorers/edvr/edvrm_wotsa_x4_8x4_600k_reds_20200522-0570e567.pth \
    data/Vid4/BIx4/calendar/ \
    demo/output \
    --window-size=5
```

BasicVSR:

```shell
    configs/basicvsr/basicvsr_reds4.py \
    https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_reds4_20120409-0e599677.pth \
    demo/output
```

The restored video will be saved in `output/`.

## Run a demo of video frame interpolation

You can use the following commands to test a video for frame interpolation.

```shell
python demo/video_interpolation_demo.py \

```shell
    [--fps-multiplier ${FPS_MULTIPLIER}] \
    [--fps ${FPS}]
```

`${INPUT_DIR}` / `${OUTPUT_DIR}` can be a path of video file or the folder of a sequence of ordered images.
If `${OUTPUT_DIR}` is a path of video file, its frame rate can be determined by the frame rate of input video and `fps_multiplier`, or be determined by `fps` directly (the former has higher priority). Examples:

The frame rate of output video is determined by the frame rate of input video and `fps_multiplier`：

```shell
    configs/cain/cain_b5_g1b32_vimeo90k_triplet.py \
    https://download.openmmlab.com/mmediting/video_interpolators/cain/cain_b5_320k_vimeo-triple_20220117-647f3de2.pth \

configs/cain/cain_b5_g1b32_vimeo90k_triplet.py \
    tests/data/test_inference.mp4 \
    tests/data/test_inference_vfi_out.mp4 \
    --fps-multiplier 2.0
```

The frame rate of output video is determined by `fps`:

```shell
    --fps 60.0
```

## Run a demo of image translation models

MMagic provides high-level APIs for translating images by using image translation models. Here is an example of building Pix2Pix and obtaining the translated images.

```python
from mmagic.apis import init_model, sample_img2img_model

```python

config_file = 'configs/pix2pix/pix2pix_vanilla-unet-bn_wo-jitter-flip-4xb1-190kiters_edges2shoes.py'
checkpoint_file = 'https://download.openmmlab.com/mmediting/pix2pix/refactor/pix2pix_vanilla_unet_bn_wo_jitter_flip_1x4_186840_edges2shoes_convert-bgr_20210902_170902-0c828552.pth'
# Specify the path to image you want to translate
image_path = 'tests/data/paired/test/33_AB.jpg'
# translate a single image
translated_image = sample_img2img_model(model, image_path, target_domain='photo')
```

Indeed, we have already provided a more friendly demo script to users. You can use [demo/translation_demo.py](../../../demo/translation_demo.py) with the following commands:

```shell

```

```shell
python demo/translation_demo.py \
    ${IMAGE_PATH}
```

Note that more customized arguments are also offered to customize your sampling procedure. Please use `python demo/translation_demo.py --help` to check more details.

# Semi-automatic Object Detection Annotation with MMDetection and Label-Studio

Please read the [Docs](../../docs/en/user_guides/label_studio.md) for more details.

<details>

```bibtex
}
```

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_ICCV_2019/html/Cao_Cross-Domain_Adaptation_for_Animal_Pose_Estimation_ICCV_2019_paper.html">Animal-Pose (ICCV'2019)</a></summary>

```bibtex
@InProceedings{Cao_2019_ICCV,
    author = {Cao, Jinkun and Tang, Hongyang and Fang, Hao-Shu and Shen, Xiaoyong and Lu, Cewu and Tai, Yu-Wing},
    title = {Cross-Domain Adaptation for Animal Pose Estimation},

author = {Cao, Jinkun and Tang, Hongyang and Fang, Hao-Shu and Shen, Xiaoyong and Lu, Cewu and Tai, Yu-Wing},
    booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
    month = {October},
}
```

Results on AnimalPose validation set (1117 instances)

| [pose_resnet_50](/configs/animal_2d_keypoint/topdown_heatmap/animalpose/td-hm_res50_8xb64-210e_animalpose-256x256.py) |  256x256   | 0.691 |      0.947      |      0.770      | 0.736 |      0.955      | [ckpt](https://download.openmmlab.com/mmpose/animal/resnet/res50_animalpose_256x256-e1f30bff_20210426.pth) | [log](https://download.openmmlab.com/mmpose/animal/resnet/res50_animalpose_256x256_20210426.log.json) |
| [pose_resnet_101](/configs/animal_2d_keypoint/topdown_heatmap/animalpose/td-hm_res101_8xb64-210e_animalpose-256x256.py) |  256x256   | 0.696 |      0.948      |      0.774      | 0.736 |      0.951      | [ckpt](https://download.openmmlab.com/mmpose/animal/resnet/res101_animalpose_256x256-85563f4a_20210426.pth) | [log](https://download.openmmlab.com/mmpose/animal/resnet/res101_animalpose_256x256_20210426.log.json) |

| [pose_resnet_152](/configs/animal_2d_keypoint/topdown_heatmap/animalpose/td-hm_res152_8xb32-210e_animalpose-256x256.py) |  256x256   | 0.704 |      0.938      |      0.786      | 0.748 |      0.946      | [ckpt](https://download.openmmlab.com/mmpose/animal/resnet/res152_animalpose_256x256-a0a7506c_20210426.pth) | [log](https://download.openmmlab.com/mmpose/animal/resnet/res152_animalpose_256x256_20210426.log.json) |

# 自定义损失函数

MMDetection 为用户提供了不同的损失函数。但是默认的配置可能无法适应不同的数据和模型，所以用户可能会希望修改某一个损失函数来适应新的情况。

本教程首先详细的解释计算损失的过程然后给出一些关于如何修改每一个步骤的指导。对损失的修改可以被分为微调和加权。

## 一个损失的计算过程

给定输入（包括预测和目标，以及权重），损失函数会把输入的张量映射到最后的损失标量。映射过程可以分为下面五个步骤：

1. 设置采样方法为对正负样本进行采样。

2. 通过损失核函数获取**元素**或者**样本**损失。

3. 通过权重张量来给损失**逐元素**权重。

4. 把损失张量归纳为一个**标量**。

5. 用一个**张量**给当前损失一个权重。

## 设置采样方法（步骤 1）

对于一些损失函数，需要采样策略来避免正负样本之间的不平衡。

例如，在RPN head中使用`CrossEntropyLoss`时，我们需要在`train_cfg`中设置`RandomSampler`

```python
train_cfg=dict(
    rpn=dict(
        sampler=dict(
            type='RandomSampler',
            num=256,
            pos_fraction=0.5,
            neg_pos_ub=-1,
            add_gt_as_proposals=False))
```

对于其他一些具有正负样本平衡机制的损失，例如 Focal Loss、GHMC 和 QualityFocalLoss，不再需要进行采样。

## 微调损失

微调一个损失主要与步骤 2，4，5 有关，大部分的修改可以在配置文件中指定。这里我们用 [Focal Loss (FL)](https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/losses/focal_loss.py) 作为例子。
下面的代码分别是构建 FL 的方法和它的配置文件，他们是一一对应的。

```python
@LOSSES.register_module()

```python
class FocalLoss(nn.Module):

                 use_sigmoid=True,
                 gamma=2.0,
                 alpha=0.25,
                 reduction='mean',
                 loss_weight=1.0):
```

```python
loss_cls=dict(
    type='FocalLoss',
    use_sigmoid=True,
    gamma=2.0,
    alpha=0.25,
    loss_weight=1.0)
```

### 微调超参数（步骤2）

`gamma` 和 `beta` 是 Focal Loss 中的两个超参数。如果我们想把 `gamma` 的值设为 1.5，把 `alpha` 的值设为 0.5，我们可以在配置文件中按照如下指定：

```python
    gamma=1.5,
    alpha=0.5,
```

### 微调归纳方式（步骤4）

Focal Loss 默认的归纳方式是 `mean`。如果我们想把归纳方式从 `mean` 改成 `sum`，我们可以在配置文件中按照如下指定：

```python
    loss_weight=1.0,
    reduction='sum')
```

### 微调损失权重（步骤5）

这里的损失权重是一个标量，他用来控制多任务学习中不同损失的重要程度，例如，分类损失和回归损失。如果我们想把分类损失的权重设为 0.5，我们可以在配置文件中如下指定：

```

```python
    loss_weight=0.5)
```

## 加权损失（步骤3）

加权损失就是我们逐元素修改损失权重。更具体来说，我们给损失张量乘以一个与他有相同形状的权重张量。所以，损失中不同的元素可以被赋予不同的比例，所以这里叫做逐元素。损失的权重在不同模型中变化很大，而且与上下文相关，但是总的来说主要有两种损失权重：分类损失的 `label_weights` 和边界框的 `bbox_weights`。你可以在相应的头中的 `get_target` 方法中找到他们。这里我们使用 [ATSSHead](https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/dense_heads/atss_head.py#L322) 作为一个例子。它继承了 [AnchorHead](https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/dense_heads/anchor_head.py) ，但是我们重写它的
`get_targets` 方法来产生不同的 `label_weights` 和 `bbox_weights`。

```
class ATSSHead(AnchorHead):

    ...

    def get_targets(self,
                    anchor_list,
                    valid_flag_list,

```

    ...

                    gt_bboxes_list,
                    img_metas,
                    gt_bboxes_ignore_list=None,
                    gt_labels_list=None,
                    label_channels=1,
                    unmap_outputs=True):
```

# 数据流

在本章节中，我们将介绍 [Runner](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/runner.html) 管理的内部模块之间的数据流和数据格式约定。

## 数据流概述

[Runner](https://github.com/open-mmlab/mmengine/blob/main/docs/zh_cn/design/runner.md) 相当于 MMEngine 中的“集成器”。它覆盖了框架的所有方面，并肩负着组织和调度几乎所有模块的责任，这意味着各模块之间的数据流也由 `Runner` 控制。 如 [MMEngine 中的 Runner 文档](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/runner.html)所示，下图展示了基本的数据流。

![Basic dataflow](https://user-images.githubusercontent.com/112053249/199228350-5f80699e-7fd2-4b4c-ac32-0b16b1922c2e.png)

虚线边框、灰色填充形状代表不同的数据格式，而实心框表示模块/方法。由于 MMEngine 极大的灵活性和可扩展性，一些重要的基类可以被继承，并且它们的方法可以被覆写。 上图所示数据流仅适用于当用户没有自定义 `Runner` 中的 `TrainLoop`、`ValLoop` 和 `TestLoop`，并且没有在其自定义模型中覆写 `train_step`、`val_step` 和 `test_step` 方法时。MMSegmentation 中 loop 的默认设置如下：使用`IterBasedTrainLoop` 训练模型，共计 20000 次迭代，并且在每 2000 次迭代后进行一次验证。

```python
train_cfg = dict(type='IterBasedTrainLoop', max_iters=20000, val_interval=2000)
```

```python
```

在上图中，红色线表示 [train_step](./models.md#train_step)，在每次训练迭代中，数据加载器（dataloader）从存储中加载图像并传输到数据预处理器（data preprocessor），数据预处理器会将图像放到特定的设备上，并将数据堆叠到批处理中，之后模型接受批处理数据作为输入，最后将模型的输出发送给优化器（optimizer）。蓝色线表示 [val_step](./models.md#val_step) 和 [test_step](./models.md#test_step)。这两个过程的数据流除了模型输出与 `train_step` 不同外，其余均和 `train_step` 类似。由于在评估时模型参数会被冻结，因此模型的输出将被传递给 [Evaluator](./evaluation.md#ioumetric)。
来计算指标。

## MMSegmentation 中的数据流约定

在上面的图中，我们可以看到基本的数据流。在本节中，我们将分别介绍数据流中涉及的数据的格式约定。

### 数据加载器到数据预处理器

数据加载器（DataLoader）是 MMEngine 的训练和测试流程中的一个重要组件。
从概念上讲，它源于 [PyTorch](https://pytorch.org/) 并保持一致。DataLoader 从文件系统加载数据，原始数据通过数据准备流程后被发送给数据预处理器。

MMSegmentation 在 [PackSegInputs](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/datasets/transforms/formatting.py#L12) 中定义了默认数据格式， 它是 `train_pipeline` 和 `test_pipeline` 的最后一个组件。有关数据转换 `pipeline` 的更多信息，请参阅[数据转换文档](./transforms.md)。

在没有任何修改的情况下，PackSegInputs 的返回值通常是一个包含 `inputs` 和 `data_samples` 的 `dict`。以下伪代码展示了 mmseg 中数据加载器输出的数据类型，它是从数据集中获取的一批数据样本，数据加载器将它们打包成一个字典列表。`inputs` 是输入进模型的张量列表，`data_samples` 包含了输入图像的 meta information 和相应的 ground truth。

```python
dict(
    inputs=List[torch.Tensor],
    data_samples=List[SegDataSample]
)
```

```python
dict(
)
```

**注意：** [SegDataSample](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/structures/seg_data_sample.py) 是 MMSegmentation 的数据结构接口，用于连接不同组件。`SegDataSample` 实现了抽象数据元素 `mmengine.structures.BaseDataElement`，更多信息请在  [MMEngine](https://github.com/open-mmlab/mmengine) 中参阅 [SegDataSample 文档](./structures.md)和[数据元素文档](https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/data_element.html)。

### 数据预处理器到模型

虽然在[上面的图](##数据流概述)中分开绘制了数据预处理器和模型，但数据预处理器是模型的一部分，因此可以在[模型教程](./models.md)中找到数据预处理器章节。

数据预处理器的返回值是一个包含 `inputs` 和 `data_samples` 的字典，其中 `inputs` 是批处理图像的 4D 张量，`data_samples` 中添加了一些用于数据预处理的额外元信息。当传递给网络时，字典将被解包为两个值。 以下伪代码展示了数据预处理器的返回值和模型的输入值。

```python
dict(
    inputs=torch.Tensor,
)
```

```python
class Network(BaseSegmentor):

    def forward(self, inputs: torch.Tensor, data_samples: List[SegDataSample], mode: str):
        pass
```

data_samples=List[SegDataSample]
)
```

```python

        pass
```

**注意：** 模型的前向传播有 3 种模式，由输入参数 mode 控制，更多信息请参阅[模型教程](./models.md)。

### 模型输出

如[模型教程](./models.md#forward) ***（[中文链接待更新](./models.md#forward)）*** 所提到的 3 种前向传播具有 3 种输出。
`train_step` 和 `test_step`（或 `val_step`）分别对应于 `'loss'` 和 `'predict'`。

在 `test_step` 或 `val_step` 中，推理结果会被传递给 `Evaluator` 。您可以参阅[评估文档](./evaluation.md)来获取更多关于 `Evaluator` 的信息。

在推理后，MMSegmentation 中的 [BaseSegmentor](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/models/segmentors/base.py#L15) 会对推理结果进行简单的后处理以打包推理结果。神经网络生成的分割 logits，经过 `argmax` 操作后的分割 mask 和 ground truth（如果存在）将被打包到类似 `SegDataSample` 的实例。 [postprocess_result](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/models/segmentors/base.py#L132) 的返回值是一个 **`SegDataSample`的`List`**。下图显示了这些 `SegDataSample` 实例的关键属性。

![SegDataSample](https://user-images.githubusercontent.com/15952744/209912225-ab46a8d9-904a-43cb-8bf1-8bec4938ed29.png)

与数据预处理器一致，损失函数也是模型的一部分，它是[解码头](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/models/decode_heads/decode_head.py#L142)的属性之一。

在 MMSegmentation 中，`decode_head` 的 [loss_by_feat](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/models/decode_heads/decode_head.py#L291) 方法是用于计算损失的统一接口。

参数：

- seg_logits (Tensor)：解码头前向函数的输出
- batch_data_samples (List\[SegDataSample\])：分割数据样本，通常包括如 `metainfo` 和  `gt_sem_seg` 等信息

返回值：

- dict\[str, Tensor\]：一个损失组件的字典

**注意：**  `train_step` 将损失传递进 OptimWrapper 以更新模型中的权重，更多信息请参阅 [train_step](./models.md#train_step)。

# NPU (华为 昇腾)

## 使用方法

请参考 [MMCV 的安装文档](https://mmcv.readthedocs.io/en/latest/get_started/build.html#build-mmcv-full-on-ascend-npu-machine) 来安装 NPU 版本的 MMCV。

以下展示单机四卡场景的运行指令:

```shell
bash tools/dist_train.sh configs/deeplabv3/deeplabv3_r50-d8_4xb2-40k_cityscapes-512x1024.py 4
```

以下展示单机单卡下的运行指令:

```shell
python tools/train.py configs/deeplabv3/deeplabv3_r50-d8_4xb2-40k_cityscapes-512x1024.py
```

## 模型验证结果

|        Model        | mIoU  | Config                                                                                                                                     | Download                                                                                                                                    |

| :-----------------: | :---: | :----------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------ |
|   [deeplabv3](<>)   | 78.85 | [config](https://github.com/open-mmlab/mmsegmentation/tree/1.x/configs/deeplabv3/deeplabv3_r50-d8_4xb2-40k_cityscapes-512x1024.py)         | [log](https://download.openmmlab.com/mmsegmentation/v0.5/device/npu/deeplabv3_r50-d8_4xb2-40k_cityscapes-512x1024_20230115_205626.json)     |
| [deeplabv3plus](<>) | 79.23 | [config](https://github.com/open-mmlab/mmsegmentation/tree/1.x/configs/deeplabv3plus/deeplabv3plus_r50-d8_4xb2-40k_cityscapes-512x1024.py) | [log](https://download.openmmlab.com/mmsegmentation/v0.5/device/npu/deeplabv3plus_r50-d8_4xb2-40k_cityscapes-512x1024_20230116_043450.json) |

|     [hrnet](<>)     | 78.1  | [config](https://github.com/open-mmlab/mmsegmentation/tree/1.x/configs/hrnet/fcn_hr18_4xb2-40k_cityscapes-512x1024.py)                     | [log](https://download.openmmlab.com/mmsegmentation/v0.5/device/npu/fcn_hr18_4xb2-40k_cityscapes-512x1024_20230116_215821.json)             |
|      [fcn](<>)      | 74.15 | [config](https://github.com/open-mmlab/mmsegmentation/tree/1.x/configs/fcn/fcn_r50-d8_4xb2-40k_cityscapes-512x1024.py)                     | [log](https://download.openmmlab.com/mmsegmentation/v0.5/device/npu/fcn_r50-d8_4xb2-40k_cityscapes-512x1024_20230111_083014.json)           |
|     [icnet](<>)     | 69.25 | [config](https://github.com/open-mmlab/mmsegmentation/tree/1.x/configs/icnet/icnet_r50-d8_4xb2-80k_cityscapes-832x832.py)                  | [log](https://download.openmmlab.com/mmsegmentation/v0.5/device/npu/icnet_r50-d8_4xb2-80k_cityscapes-832x832_20230119_002929.json)          |

|    [pspnet](<>)     | 77.21 | [config](https://github.com/open-mmlab/mmsegmentation/tree/1.x/configs/pspnet/pspnet_r50b-d8_4xb2-80k_cityscapes-512x1024.py)              | [log](https://download.openmmlab.com/mmsegmentation/v0.5/device/npu/pspnet_r50b-d8_4xb2-80k_cityscapes-512x1024_20230114_042721.json)       |
|     [unet](<>)      | 68.86 | [config](https://github.com/open-mmlab/mmsegmentation/tree/1.x/configs/unet/unet-s5-d16_fcn_4xb4-160k_cityscapes-512x1024.py)              | [log](https://download.openmmlab.com/mmsegmentation/v0.5/device/npu/unet-s5-d16_fcn_4xb4-160k_cityscapes-512x1024_20230129_224750.json)     |
|    [upernet](<>)    | 77.81 | [config](https://github.com/open-mmlab/mmsegmentation/tree/1.x/configs/upernet/upernet_r50_4xb2-40k_cityscapes-512x1024.py)                | [log](https://download.openmmlab.com/mmsegmentation/v0.5/device/npu/upernet_r50_4xb2-40k_cityscapes-512x1024_20230129_014634.json)          |

|    [apcnet](<>)     | 78.02 | [config](https://github.com/open-mmlab/mmsegmentation/tree/1.x/configs/apcnet/apcnet_r50-d8_4xb2-40k_cityscapes-512x1024.py)               | [log](https://download.openmmlab.com/mmsegmentation/v0.5/device/npu/apcnet_r50-d8_4xb2-40k_cityscapes-512x1024_20230209_212545.json)        |
|   [bisenetv1](<>)   | 76.04 | [config](https://github.com/open-mmlab/mmsegmentation/tree/1.x/configs/bisenetv1/bisenetv1_r50-d32_4xb4-160k_cityscapes-1024x1024.py)      | [log](https://download.openmmlab.com/mmsegmentation/v0.5/device/npu/bisenetv1_r50-d32_4xb4-160k_cityscapes-1024x1024_20230201_023946.json)  |
|   [bisenetv2](<>)   | 72.44 | [config](https://github.com/open-mmlab/mmsegmentation/tree/1.x/configs/bisenetv2/bisenetv2_fcn_4xb4-amp-160k_cityscapes-1024x1024.py)      | [log](https://download.openmmlab.com/mmsegmentation/v0.5/device/npu/bisenetv2_fcn_4xb4-amp-160k_cityscapes-1024x1024_20230205_215606.json)  |

**注意:**

**注意:**

- 如果没有特别标记，NPU 上的使用混合精度训练的结果与使用 FP32 的 GPU 上的结果相同。

**以上模型结果由华为昇腾团队提供**

# MobileNet V3

> [Searching for MobileNetV3](https://arxiv.org/abs/1905.02244)

**MobileNet V3** is initially described in [the paper](https://arxiv.org/pdf/1905.02244.pdf). MobileNetV3 parameters are obtained by NAS (network architecture search) search, and some practical results of V1 and V2 are inherited, and the attention mechanism of SE channel is attracted, which can be considered as a masterpiece. The author create two new MobileNet models for release: MobileNetV3-Large and MobileNetV3-Small which are targeted for high and low resource use cases. These models are then adapted and applied to the tasks of object detection and semantic segmentation. The author of MobileNet V3 measure its performance on Imagenet classification, COCO object detection, and Cityscapes segmentation.

<img src="https://user-images.githubusercontent.com/26739999/142563801-ef4feacc-ecd7-4d14-a411-8c9d63571749.png" width="60%"/>
</div>

</div>

<details>

<summary>Show the paper's abstract</summary>

<br>

We present the next generation of MobileNets based on a combination of complementary search techniques as well as a novel architecture design. MobileNetV3 is tuned to mobile phone CPUs through a combination of hardware-aware network architecture search (NAS) complemented by the NetAdapt algorithm and then subsequently improved through novel architecture advances. This paper starts the exploration of how automated search algorithms and network design can work together to harness complementary approaches improving the overall state of the art. Through this process we create two new MobileNet models for release: MobileNetV3-Large and MobileNetV3-Small which are targeted for high and low resource use cases. These models are then adapted and applied to the tasks of object detection and semantic segmentation. For the task of semantic segmentation (or any dense pixel prediction), we propose a new efficient segmentation decoder Lite Reduced Atrous Spatial Pyramid Pooling (LR-ASPP). We achieve new state of the art results for mobile classification, detection and segmentation. MobileNetV3-Large is 3.2% more accurate on ImageNet classification while reducing latency by 15% compared to MobileNetV2. MobileNetV3-Small is 4.6% more accurate while reducing latency by 5% compared to MobileNetV2. MobileNetV3-Large detection is 25% faster at roughly the same accuracy as MobileNetV2 on COCO detection. MobileNetV3-Large LR-ASPP is 30% faster than MobileNetV2 R-ASPP at similar accuracy for Cityscapes segmentation.

</br>

```python

predict = inference_model('mobilenet-v3-small-050_3rdparty_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('mobilenet-v3-small-050_3rdparty_in1k', pretrained=True)
```

Train:

```shell
python tools/train.py configs/mobilenet_v3/mobilenet-v3-small_8xb128_in1k.py
```

Test:

```shell

Train:

```shell
```

Test:

```shell
python tools/test.py configs/mobilenet_v3/mobilenet-v3-small-050_8xb128_in1k.py https://download.openmmlab.com/mmclassification/v0/mobilenet_v3/mobilenet-v3-small-050_3rdparty_in1k_20221114-e0b86be1.pth
```

| Model                                    |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                     Config                      |                             Download                             |
| :--------------------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :---------------------------------------------: | :--------------------------------------------------------------: |

| `mobilenet-v3-small-050_3rdparty_in1k`\* | From scratch |    1.59    |   0.02    |   57.91   |   80.19   | [config](mobilenet-v3-small-050_8xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mobilenet_v3/mobilenet-v3-small-050_3rdparty_in1k_20221114-e0b86be1.pth) |
| `mobilenet-v3-small-075_3rdparty_in1k`\* | From scratch |    2.04    |   0.04    |   65.23   |   85.44   | [config](mobilenet-v3-small-075_8xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mobilenet_v3/mobilenet-v3-small-075_3rdparty_in1k_20221114-2011fa76.pth) |
| `mobilenet-v3-small_8xb128_in1k`         | From scratch |    2.54    |   0.06    |   66.68   |   86.74   |   [config](mobilenet-v3-small_8xb128_in1k.py)   | [model](https://download.openmmlab.com/mmclassification/v0/mobilenet_v3/mobilenet-v3-small_8xb128_in1k_20221114-bd1bfcde.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/mobilenet_v3/mobilenet-v3-small_8xb128_in1k_20221114-bd1bfcde.json) |

| `mobilenet-v3-small_3rdparty_in1k`\*     | From scratch |    2.54    |   0.06    |   67.66   |   87.41   |   [config](mobilenet-v3-small_8xb128_in1k.py)   | [model](https://download.openmmlab.com/mmclassification/v0/mobilenet_v3/convert/mobilenet_v3_small-8427ecf0.pth) |
| `mobilenet-v3-large_8xb128_in1k`         | From scratch |    5.48    |   0.23    |   73.49   |   91.31   |   [config](mobilenet-v3-large_8xb128_in1k.py)   | [model](https://download.openmmlab.com/mmclassification/v0/mobilenet_v3/mobilenet-v3-large_8xb128_in1k_20221114-0ed9ed9a.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/mobilenet_v3/mobilenet-v3-large_8xb128_in1k_20221114-0ed9ed9a.json) |
| `mobilenet-v3-large_3rdparty_in1k`\*     | From scratch |    5.48    |   0.23    |   74.04   |   91.34   |   [config](mobilenet-v3-large_8xb128_in1k.py)   | [model](https://download.openmmlab.com/mmclassification/v0/mobilenet_v3/convert/mobilenet_v3_large-3ea3c186.pth) |

*Models with * are converted from the [official repo](https://github.com/pytorch/vision/blob/main/torchvision/models/mobilenetv3.py). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@inproceedings{Howard_2019_ICCV,
    author = {Howard, Andrew and Sandler, Mark and Chu, Grace and Chen, Liang-Chieh and Chen, Bo and Tan, Mingxing and Wang, Weijun and Zhu, Yukun and Pang, Ruoming and Vasudevan, Vijay and Le, Quoc V. and Adam, Hartwig},
    title = {Searching for MobileNetV3},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
}
```

# 准备 RealSRSet 数据集

```bibtex
@inproceedings{zhang2021designing,
    title={Designing a Practical Degradation Model for Deep Blind Image Super-Resolution},
    author={Zhang, Kai and Liang, Jingyun and Van Gool, Luc and Timofte, Radu},
    booktitle={IEEE International Conference on Computer Vision},
    pages={4791--4800},
    year={2021}
}
```

数据集 RealSRSet 可以从 [此处](https://github.com/cszn/BSRGAN/tree/main/testsets/RealSRSet) 下载。

数据集 RealSRSet+5images 可以从 [此处](https://github.com/JingyunLiang/SwinIR/releases/download/v0.0/RealSRSet+5images.zip) 下载。

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── realsrset
|   ├── RealSRSet+5images
```

# 2D Body Keypoint Datasets

MMPose supported datasets:

- Images
  - [COCO](#coco) \[ [Homepage](http://cocodataset.org/) \]
  - [MPII](#mpii) \[ [Homepage](http://human-pose.mpi-inf.mpg.de/) \]
  - [MPII-TRB](#mpii-trb) \[ [Homepage](https://github.com/kennymckormick/Triplet-Representation-of-human-Body) \]
  - [AI Challenger](#aic) \[ [Homepage](https://github.com/AIChallenger/AI_Challenger_2017) \]
  - [CrowdPose](#crowdpose) \[ [Homepage](https://github.com/Jeff-sjtu/CrowdPose) \]
  - [OCHuman](#ochuman) \[ [Homepage](https://github.com/liruilong940607/OCHumanApi) \]
  - [MHP](#mhp) \[ [Homepage](https://lv-mhp.github.io/dataset) \]
  - [Human-Art](#humanart) \[ [Homepage](https://idea-research.github.io/HumanArt/) \]
- Videos

- [MHP](#mhp) \[ [Homepage](https://lv-mhp.github.io/dataset) \]
- Videos
  - [PoseTrack18](#posetrack18) \[ [Homepage](https://posetrack.net/users/download.php) \]
  - [sub-JHMDB](#sub-jhmdb-dataset) \[ [Homepage](http://jhmdb.is.tue.mpg.de/dataset) \]

## COCO

<details>

```bibtex
}
```

}
```

  <img src="https://user-images.githubusercontent.com/100993824/227864552-489d03de-e1b8-4ca2-8ac1-80dd99826cb7.png" height="300px">
</div>

For [COCO](http://cocodataset.org/) data, please download from [COCO download](http://cocodataset.org/#download), 2017 Train/Val is needed for COCO keypoints training and validation.
[HRNet-Human-Pose-Estimation](https://github.com/HRNet/HRNet-Human-Pose-Estimation) provides person detection result of COCO val2017 to reproduce our multi-person pose estimation results.
Please download from [OneDrive](https://1drv.ms/f/s!AhIXJn_J-blWzzDXoz5BeFl8sWM-) or [GoogleDrive](https://drive.google.com/drive/folders/1fRUDNUDxe9fjqcRZ2bnF_TKMlO0nB_dk?usp=sharing).

Optionally, to evaluate on COCO'2017 test-dev, please download the [image-info](https://download.openmmlab.com/mmpose/datasets/person_keypoints_test-dev-2017.json).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── coco
        │-- annotations
        │   │-- person_keypoints_train2017.json
        │   |-- person_keypoints_val2017.json
        │   |-- person_keypoints_test-dev-2017.json
        |-- person_detection_results
        |   |-- COCO_val2017_detections_AP_H_56_person.json
        |   |-- COCO_test-dev2017_detections_AP_H_609_person.json
        │-- train2017
        │   │-- 000000000009.jpg
        │   │-- 000000000025.jpg
        │   │-- 000000000030.jpg
        │   │-- ...

│-- train2017
        │   │-- ...
        `-- val2017
            │-- 000000000139.jpg
            │-- 000000000285.jpg
            │-- 000000000632.jpg
            │-- ...

```

## MPII

<details>

```bibtex
}
```

  <img src="https://user-images.githubusercontent.com/100993824/227864660-e5f51e7d-deca-41d8-9725-8b5432bcc0e6.png" height="300px">
</div>

month = {June}
}
```

</div>

For [MPII](http://human-pose.mpi-inf.mpg.de/) data, please download from [MPII Human Pose Dataset](http://human-pose.mpi-inf.mpg.de/).
We have converted the original annotation files into json format, please download them from [mpii_annotations](https://download.openmmlab.com/mmpose/datasets/mpii_annotations.tar).
Extract them under {MMPose}/data, and make them look like this:

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── mpii
        |── annotations
        |   |── mpii_gt_val.mat
        |   |── mpii_test.json
        |   |── mpii_train.json
        |   |── mpii_trainval.json
        |   `── mpii_val.json
        `── images
            |── 000001163.jpg
            |── 000003072.jpg

```

|   |── mpii_train.json
        `── images

```

During training and inference, the prediction result will be saved as '.mat' format by default. We also provide a tool to convert this '.mat' to more readable '.json' format.

```shell
python tools/dataset/mat2json ${PRED_MAT_FILE} ${GT_JSON_FILE} ${OUTPUT_PRED_JSON_FILE}
```

```shell
python tools/dataset/mat2json work_dirs/res50_mpii_256x256/pred.mat data/mpii/annotations/mpii_val.json pred.json
```

## MPII-TRB

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_ICCV_2019/html/Duan_TRB_A_Novel_Triplet_Representation_for_Understanding_2D_Human_Body_ICCV_2019_paper.html">MPII-TRB (ICCV'2019)</a></summary>

```bibtex
@inproceedings{duan2019trb,
  title={TRB: A Novel Triplet Representation for Understanding 2D Human Body},

```bibtex
  author={Duan, Haodong and Lin, Kwan-Yee and Jin, Sheng and Liu, Wentao and Qian, Chen and Ouyang, Wanli},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision},
  pages={9479--9488},
}
```

  <img src="https://user-images.githubusercontent.com/100993824/227864382-ab722299-6806-4ae4-babb-7bcc5fb09662.png" height="300px">
</div>

For [MPII-TRB](https://github.com/kennymckormick/Triplet-Representation-of-human-Body) data, please download from [MPII Human Pose Dataset](http://human-pose.mpi-inf.mpg.de/).
Please download the annotation files from [mpii_trb_annotations](https://download.openmmlab.com/mmpose/datasets/mpii_trb_annotations.tar).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── mpii

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── mpii
        |   |── mpii_trb_train.json
        |   |── mpii_trb_val.json
        `── images

```

## AIC

<details>

```bibtex
}
```

  <img src="https://user-images.githubusercontent.com/100993824/227864755-dd19644e-fccb-458b-a8c0-de55920261f5.png" height="300px">

year={2017}
}
```

</div>

For [AIC](https://github.com/AIChallenger/AI_Challenger_2017) data, please download from [AI Challenger 2017](https://github.com/AIChallenger/AI_Challenger_2017), 2017 Train/Val is needed for keypoints training and validation.
Please download the annotation files from [aic_annotations](https://download.openmmlab.com/mmpose/datasets/aic_annotations.tar).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── aic
        │   │-- aic_train.json
        │   |-- aic_val.json
        │-- ai_challenger_keypoint_train_20170902
        │   │-- keypoint_train_images_20170902
        │   │   │-- 0000252aea98840a550dac9a78c476ecb9f47ffa.jpg

│   |-- aic_val.json
        │   │   │-- 000050f770985ac9653198495ef9b5c82435d49c.jpg
        │   │   │-- ...
        `-- ai_challenger_keypoint_validation_20170911
            │-- keypoint_validation_images_20170911
                │-- 0002605c53fb92109a3f2de4fc3ce06425c3b61f.jpg
                │-- 0003b55a2c991223e6d8b4b820045bd49507bf6d.jpg
                │-- ...
```

## CrowdPose

<details>

```bibtex

```bibtex
}
```

  <img src="https://user-images.githubusercontent.com/100993824/227864868-54a98493-df3a-44d8-acbc-6ec22043dfb9.png" height="300px">
</div>

For [CrowdPose](https://github.com/Jeff-sjtu/CrowdPose) data, please download from [CrowdPose](https://github.com/Jeff-sjtu/CrowdPose).
Please download the annotation files and human detection results from [crowdpose_annotations](https://download.openmmlab.com/mmpose/datasets/crowdpose_annotations.tar).

For top-down approaches, we follow [CrowdPose](https://arxiv.org/abs/1812.00324) to use the [pre-trained weights](https://pjreddie.com/media/files/yolov3.weights) of [YOLOv3](https://github.com/eriklindernoren/PyTorch-YOLOv3) to generate the detected human bounding boxes.
For model training, we follow [HigherHRNet](https://github.com/HRNet/HigherHRNet-Human-Pose-Estimation) to train models on CrowdPose train/val dataset, and evaluate models on CrowdPose test dataset.

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── crowdpose
        │   │-- mmpose_crowdpose_train.json
        │   │-- mmpose_crowdpose_val.json
        │   │-- mmpose_crowdpose_trainval.json

`── data
        │   │-- mmpose_crowdpose_test.json
        │   │-- det_for_crowd_test_0.1_0.5.json
        │-- images
            │-- 100000.jpg
            │-- 100001.jpg
            │-- 100002.jpg
            │-- ...
```

## OCHuman

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_CVPR_2019/html/Zhang_Pose2Seg_Detection_Free_Human_Instance_Segmentation_CVPR_2019_paper.html">OCHuman (CVPR'2019)</a></summary>

```bibtex
@inproceedings{zhang2019pose2seg,
  title={Pose2seg: Detection free human instance segmentation},
  author={Zhang, Song-Hai and Li, Ruilong and Dong, Xin and Rosin, Paul and Cai, Zixi and Han, Xi and Yang, Dingcheng and Huang, Haozhi and Hu, Shi-Min},

author={Zhang, Song-Hai and Li, Ruilong and Dong, Xin and Rosin, Paul and Cai, Zixi and Han, Xi and Yang, Dingcheng and Huang, Haozhi and Hu, Shi-Min},
  pages={889--898},
}
```

</div>

For [OCHuman](https://github.com/liruilong940607/OCHumanApi) data, please download the images and annotations from [OCHuman](https://github.com/liruilong940607/OCHumanApi),
Move them under $MMPOSE/data, and make them look like this:

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── ochuman
        │   │-- ochuman_coco_format_val_range_0.00_1.00.json
        │   |-- ochuman_coco_format_test_range_0.00_1.00.json
        |-- images
            │-- 000001.jpg

│-- annotations
        |-- images
            │-- 000002.jpg
            │-- 000003.jpg
            │-- ...

```

## MHP

<details>
<summary align="right"><a href="https://dl.acm.org/doi/abs/10.1145/3240508.3240509">MHP (ACM MM'2018)</a></summary>

```bibtex
@inproceedings{zhao2018understanding,
  title={Understanding humans in crowded scenes: Deep nested adversarial learning and a new benchmark for multi-human parsing},
  author={Zhao, Jian and Li, Jianshu and Cheng, Yu and Sim, Terence and Yan, Shuicheng and Feng, Jiashi},
  booktitle={Proceedings of the 26th ACM international conference on Multimedia},
  pages={792--800},
}
```

  <img src="https://user-images.githubusercontent.com/100993824/227865030-2fd33ade-2cc2-4b67-aca0-6dea2124b63c.png" height="300px">

year={2018}
}
```

</div>

For [MHP](https://lv-mhp.github.io/dataset) data, please download from [MHP](https://lv-mhp.github.io/dataset).
Please download the annotation files from [mhp_annotations](https://download.openmmlab.com/mmpose/datasets/mhp_annotations.tar.gz).
Please download and extract them under $MMPOSE/data, and make them look like this:

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── mhp
        │   │-- mhp_train.json
        │   │-- mhp_val.json
        │
        `-- train
        │   │-- images
        │   │   │-- 1004.jpg
        │   │   │-- 10050.jpg
        │   │   │-- ...
        │
        `-- val
        │   │   │-- 10059.jpg
        │   │   │-- 10068.jpg
        │   │   │-- ...
        │
        `-- test

│   │   │-- ...
        │
        `-- val
        │   │   │-- ...
        │
        `-- test
        │   │   │-- 1005.jpg
        │   │   │-- 10052.jpg
        │   │   │-- ...~~~~
```

## Human-Art dataset

<details>
<summary align="right"><a href="https://idea-research.github.io/HumanArt/">Human-Art (CVPR'2023)</a></summary>

```bibtex
@inproceedings{ju2023humanart,
    title={Human-Art: A Versatile Human-Centric Dataset Bridging Natural and Artificial Scenes},
    author={Ju, Xuan and Zeng, Ailing and Jianan, Wang and Qiang, Xu and Lei, Zhang},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
    year={2023}}
```

</div>

year={2023}}
```

</div>

For [Human-Art](https://idea-research.github.io/HumanArt/) data, please download the images and annotation files from [its website](https://idea-research.github.io/HumanArt/). You need to fill in the [data form](https://docs.google.com/forms/d/e/1FAIpQLScroT_jvw6B9U2Qca1_cl5Kmmu1ceKtlh6DJNmWLte8xNEhEw/viewform) to get access to the data.

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
|── data
    │── HumanArt
        │-- images
        │   │-- 2D_virtual_human
        │   │   |-- cartoon
        │   │   |   |-- 000000000000.jpg
        │   │   |   |-- ...
        │   │   |-- digital_art
        │   │   |-- ...
        │   |-- 3D_virtual_human
        │   |-- real_human
        |-- annotations

│   │   |   |-- ...
        │   │   |-- ...
        │   │-- validation_humanart.json
        │   │-- training_humanart_coco.json
        │   │-- HumanArt_validation_detections_AP_H_56_person.json
```

You can choose whether to download other annotation files in Human-Art. If you want to use additional annotation files (e.g. validation set of cartoon), you need to edit the corresponding code in config file.

## PoseTrack18

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_cvpr_2018/html/Andriluka_PoseTrack_A_Benchmark_CVPR_2018_paper.html">PoseTrack18 (CVPR'2018)</a></summary>

```bibtex
@inproceedings{andriluka2018posetrack,
  title={Posetrack: A benchmark for human pose estimation and tracking},

```bibtex
  author={Andriluka, Mykhaylo and Iqbal, Umar and Insafutdinov, Eldar and Pishchulin, Leonid and Milan, Anton and Gall, Juergen and Schiele, Bernt},
  pages={5167--5176},
}
```

  <img src="https://user-images.githubusercontent.com/100993824/227865114-3f98c673-f6d0-4518-ae99-653f475f9fc8.png" height="300px">
</div>

For [PoseTrack18](https://posetrack.net/users/download.php) data, please download from [PoseTrack18](https://posetrack.net/users/download.php).
Please download the annotation files from [posetrack18_annotations](https://download.openmmlab.com/mmpose/datasets/posetrack18_annotations.tar).

We have merged the video-wise separated official annotation files into two json files (posetrack18_train & posetrack18_val.json). We also generate the [mask files](https://download.openmmlab.com/mmpose/datasets/posetrack18_mask.tar) to speed up training.
For top-down approaches, we use [MMDetection](https://github.com/open-mmlab/mmdetection) pre-trained [Cascade R-CNN](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_x101_64x4d_fpn_20e_coco/cascade_rcnn_x101_64x4d_fpn_20e_coco_20200509_224357-051557b1.pth) (X-101-64x4d-FPN) to generate the detected human bounding boxes.

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── posetrack18
        │   │-- posetrack18_train.json

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
        │   │-- posetrack18_val.json
        │   │-- posetrack18_val_human_detections.json
        │   │-- train
        │   │   │-- 000001_bonn_train.json
        │   │   │-- 000002_bonn_train.json
        │   │   │-- ...
        │   │-- val
        │   │   │-- 000342_mpii_test.json
        │   │   │-- 000522_mpii_test.json
        │   │   │-- ...
        │   `-- test
        │       │-- 000001_mpiinew_test.json
        │       │-- 000002_mpiinew_test.json
        │       │-- ...
        │
        `-- images
        │   │-- train
        │   │   │-- 000001_bonn_train
        │   │   │   │-- 000000.jpg
        │   │   │   │-- 000001.jpg
        │   │   │   │-- ...
        │   │   │-- ...
        │   │-- val
        │   │   │-- 000342_mpii_test

│   │   │   │-- ...
        │   │   │-- ...
        │   │-- val
        │   │   │   │-- ...
        │   │   │-- ...
        │   `-- test
        │       │-- 000001_mpiinew_test
        │       │   │-- 000000.jpg
        │       │   │-- 000001.jpg
        │       │   │-- ...
        │       │-- ...
        `-- mask
            │-- train
            │   │-- 000002_bonn_train
            │   │   │-- 000000.jpg
            │   │   │-- 000001.jpg
            │   │   │-- ...
            │   │-- ...
            `-- val
                │-- 000522_mpii_test
                │   │-- 000000.jpg
                │   │-- 000001.jpg
                │   │-- ...
                │-- ...
```

The official evaluation tool for PoseTrack should be installed from GitHub.

```shell
pip install git+https://github.com/svenkreiss/poseval.git
```

## sub-JHMDB dataset

<details>

```shell
```

<details>
<summary align="right"><a href="https://link.springer.com/chapter/10.1007/978-3-030-58580-8_27">RSN (ECCV'2020)</a></summary>

```bibtex
@misc{cai2020learning,
    title={Learning Delicate Local Representations for Multi-Person Pose Estimation},
    author={Yuanhao Cai and Zhicheng Wang and Zhengxiong Luo and Binyi Yin and Angang Du and Haoqian Wang and Xinyu Zhou and Erjin Zhou and Xiangyu Zhang and Jian Sun},
    year={2020},
    eprint={2003.04030},
}
```

  <img src="https://user-images.githubusercontent.com/100993824/227865619-d65f64ae-991d-4693-99c2-caecd1beb1fc.png" height="300px">
</div>

}
```

</div>

For [sub-JHMDB](http://jhmdb.is.tue.mpg.de/dataset) data, please download the [images](<(http://files.is.tue.mpg.de/jhmdb/Rename_Images.tar.gz)>) from [JHMDB](http://jhmdb.is.tue.mpg.de/dataset),
Please download the annotation files from [jhmdb_annotations](https://download.openmmlab.com/mmpose/datasets/jhmdb_annotations.tar).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── jhmdb
        │   │-- Sub1_train.json
        │   |-- Sub1_test.json
        │   │-- Sub2_train.json
        │   |-- Sub2_test.json
        │   │-- Sub3_train.json
        │   |-- Sub3_test.json
        |-- Rename_Images
            │-- brush_hair
            │   │--April_09_brush_hair_u_nm_np1_ba_goo_0

│   │-- Sub3_train.json
            |   │   │--00001.png
            |   │   │--00002.png
            │-- catch
            │-- ...

```

Mat structure is now allocator-aware via an extra allocator parameter with default zero value.

The good-old ncnn::fastMalloc()/ncnn::fastFree() will be used for a null allocator.

You could pass a custom allocator to delegate all memory allocation and deallocation.

```cpp
class Allocator
{
public:
    virtual void* fastMalloc(size_t size) = 0;
    virtual void fastFree(void* ptr) = 0;
};
```

ncnn has already implemented two simple pooled Allocator class, with mutex lock or without it.

```cpp
ncnn::PoolAllocator locked_mempool;
ncnn::UnlockedPoolAllocator unlocked_mempool;
```

the two allocator types in ncnn

* blob allocator

    used to allocate memory for all named blobs, which you could retrieve by Extractor::extract()
* workspace allocator

    used to allocate memory for internal temporary use in layer implementation, such as the temp blob after padding in convolution

by default, all Extractor instance use the two allocator in the default option

You can alter them by ncnn::set_default_option()
or you can set them per Extractor by Extractor::set_blob_allocator()/Extractor::set_workspace_allocator()

blob allocator is guaranteed to be called in-order in layer implementation during each Extractor lifecycle
while workspace allocator may be called synchronously

the practical usage

* one network, one-by-one inference

    shared unlocked blob allocator for all Extractor

    shared locked workspace allocator for all Extractor

* one network, concurrent inference

    shared unlocked blob allocator for all Extractor in each thread

    shared locked workspace allocator for all Extractor among all threads

* concurrent multiple networks, one-by-one inference for each network

    shared unlocked blob allocator for all Extractor of each network

    shared locked workspace allocator for all Extractor among all networks (for saving memory)

shared unlocked blob allocator for all Extractor of each network

* concurrent multiple networks, concurrent inference for each network

    shared unlocked blob allocator for all Extractor of each network in each thread

# Preparing DIOR Dataset

```bibtex
@article{LI2020296,
    title = {Object detection in optical remote sensing images: A survey and a new benchmark},
    journal = {ISPRS Journal of Photogrammetry and Remote Sensing},
    volume = {159},
    pages = {296-307},
    year = {2020},
    issn = {0924-2716},
    doi = {https://doi.org/10.1016/j.isprsjprs.2019.11.023},
    url = {https://www.sciencedirect.com/science/article/pii/S0924271619302825},
    author = {Ke Li and Gang Wan and Gong Cheng and Liqiu Meng and Junwei Han}
```

## Download DIOR dataset

The DIOR dataset can be downloaded from [here](https://gcheng-nwpu.github.io/#Datasets).

The data structure is as follows:

```none
mmrotate
├── mmrotate
├── tools
├── data
│   ├── DIOR
│   │   ├── JPEGImages-trainval
│   │   ├── JPEGImages-test
│   │   ├── Annotations
│   │   │   ├─ Oriented Bounding Boxes
│   │   │   ├─ Horizontal Bounding Boxes
│   │   ├── ImageSets
│   │   │   ├─ Main

│   │   │   ├─ Main
│   │   │   │  ├─ train.txt
│   │   │   │  ├─ val.txt
│   │   │   │  ├─ test.txt
```

## Change base config

Please change `data_root` in `configs/_base_/datasets/dior.py` to `data/dior/`.

# Freihand: A dataset for markerless capture of hand pose and shape from single rgb images

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_ICCV_2019/html/Zimmermann_FreiHAND_A_Dataset_for_Markerless_Capture_of_Hand_Pose_and_ICCV_2019_paper.html">FreiHand (ICCV'2019)</a></summary>

```bibtex
@inproceedings{zimmermann2019freihand,
  title={Freihand: A dataset for markerless capture of hand pose and shape from single rgb images},
  author={Zimmermann, Christian and Ceylan, Duygu and Yang, Jimei and Russell, Bryan and Argus, Max and Brox, Thomas},
  pages={813--822},
}
```

---
name: General questions
about: Ask general questions to get help
title: ''
labels: ''
---

# Chest Radiograph Anatomical Structure Segmentation (CRASS)

This project supports **`Chest Radiograph Anatomical Structure Segmentation (CRASS) `**, which can be downloaded from [here](https://crass.grand-challenge.org/).

A set of consecutively obtained posterior-anterior chest radiograph were selected from a database containing images acquired at two sites in sub Saharan Africa with a high tuberculosis incidence. All subjects were 15 years or older. Images from digital chest radiography units were used (Delft Imaging Systems, The Netherlands) of varying resolutions, with a typical resolution of 1800--2000 pixels, the pixel size was 250 lm isotropic. From the total set of images, 225 were considered to be normal by an expert radiologist, while 333 of the images contained abnormalities. Of the abnormal images, 220 contained abnormalities in the upper area of the lung where the clavicle is located. The data was divided into a training and a test set. The training set consisted of 299 images, the test set of 249 images.

The current data is still incomplete and to be added later.

| Dataset Name                                | Anatomical Region | Task Type    | Modality | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License                                                       |
| ------------------------------------------- | ----------------- | ------------ | -------- | ------------ | --------------------- | ---------------------- | ------------ | ------------------------------------------------------------- |
| [crass](https://crass.grand-challenge.org/) | pulmonary         | segmentation | x_ray    | 2            | 299/-/234             | yes/-/no               | 2021         | [CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/) |

| background |    299     |   98.38    |    -     |    -     |     -     |     -     |
| clavicles  |    299     |    1.62    |    -     |    -     |     -     |     -     |

Note:

![crass](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/x_ray/crass/crass_dataset.png?raw=true)

```
@article{HOGEWEG20121490,
	title={Clavicle segmentation in chest radiographs},
	journal={Medical Image Analysis},
	volume={16},
	number={8},
	pages={1490-1502},
	year={2012}
}
```

pages={1490-1502},
}
```

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `crass/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
```

- download dataset from [here](https://crass.grand-challenge.org/) and decompress data to path `'data/'`.

```none
  ├── mmseg
  │   │   │   │   ├── crass

```

| background |    227     |   98.38    |    57    |  98.39   |     -     |     -     |
| clavicles  |    227     |    1.62    |    57    |   1.61   |     -     |     -     |

```shell
```

```shell
```

```shell
```

<details>

```bibtex
}
```

<details>

```bibtex

author={Graving, Jacob M and Chae, Daniel and Naik, Hemal and Li, Liang and Koger, Benjamin and Costelloe, Blair R and Couzin, Iain D},
}
```

Results on Grévy’s Zebra test set

| Arch                                                       | Input Size | PCK@0.2 |  AUC  | EPE  |                            ckpt                            |                            log                             |
| :--------------------------------------------------------- | :--------: | :-----: | :---: | :--: | :--------------------------------------------------------: | :--------------------------------------------------------: |

| [pose_resnet_50](/configs/animal_2d_keypoint/topdown_heatmap/zebra/td-hm_res50_8xb64-210e_zebra-160x160.py) |  160x160   |  1.000  | 0.914 | 1.87 | [ckpt](https://download.openmmlab.com/mmpose/animal/resnet/res50_zebra_160x160-5a104833_20210407.pth) | [log](https://download.openmmlab.com/mmpose/animal/resnet/res50_zebra_160x160_20210407.log.json) |
| [pose_resnet_101](/configs/animal_2d_keypoint/topdown_heatmap/zebra/td-hm_res101_8xb64-210e_zebra-160x160.py) |  160x160   |  1.000  | 0.915 | 1.83 | [ckpt](https://download.openmmlab.com/mmpose/animal/resnet/res101_zebra_160x160-e8cb2010_20210407.pth) | [log](https://download.openmmlab.com/mmpose/animal/resnet/res101_zebra_160x160_20210407.log.json) |

| [pose_resnet_152](/configs/animal_2d_keypoint/topdown_heatmap/zebra/td-hm_res152_8xb32-210e_zebra-160x160.py) |  160x160   |  1.000  | 0.921 | 1.67 | [ckpt](https://download.openmmlab.com/mmpose/animal/resnet/res152_zebra_160x160-05de71dd_20210407.pth) | [log](https://download.openmmlab.com/mmpose/animal/resnet/res152_zebra_160x160_20210407.log.json) |

# FCOS

> [FCOS: Fully Convolutional One-Stage Object Detection](https://arxiv.org/abs/1904.01355)

We propose a fully convolutional one-stage object detector (FCOS) to solve object detection in a per-pixel prediction fashion, analogue to semantic segmentation. Almost all state-of-the-art object detectors such as RetinaNet, SSD, YOLOv3, and Faster R-CNN rely on pre-defined anchor boxes. In contrast, our proposed detector FCOS is anchor box free, as well as proposal free. By eliminating the predefined set of anchor boxes, FCOS completely avoids the complicated computation related to anchor boxes such as calculating overlapping during training. More importantly, we also avoid all hyper-parameters related to anchor boxes, which are often very sensitive to the final detection performance. With the only post-processing non-maximum suppression (NMS), FCOS with ResNeXt-64x4d-101 achieves 44.7% in AP with single-model and single-scale testing, surpassing previous one-stage detectors with the advantage of being much simpler. For the first time, we demonstrate a much simpler and flexible detection framework achieving improved detection accuracy. We hope that the proposed FCOS framework can serve as a simple and strong alternative for many other instance-level tasks.

<img src="https://user-images.githubusercontent.com/40661020/143882011-45b234bc-d04b-4bbe-a822-94bec057ac86.png"/>
</div>

| Backbone | Style | GN  | MS train | Tricks | DCN | Lr schd | Mem (GB) | Inf time (fps) | box AP |                                         Config                                         |                                                                                                                                                                                          Download                                                                                                                                                                                          |

|   R-50   | caffe |  Y  |    N     |   N    |  N  |   1x    |   3.6    |      22.7      |  36.6  |                   [config](./fcos_r50-caffe_fpn_gn-head_1x_coco.py)                    |                                                        [model](https://download.openmmlab.com/mmdetection/v2.0/fcos/fcos_r50_caffe_fpn_gn-head_1x_coco/fcos_r50_caffe_fpn_gn-head_1x_coco-821213aa.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/fcos/fcos_r50_caffe_fpn_gn-head_1x_coco/20201227_180009.log.json)                                                         |

|   R-50   | caffe |  Y  |    N     |   Y    |  N  |   1x    |   3.7    |       -        |  38.7  |   [config](./fcos_r50-caffe_fpn_gn-head-center-normbbox-centeronreg-giou_1x_coco.py)   |       [model](https://download.openmmlab.com/mmdetection/v2.0/fcos/fcos_center-normbbox-centeronreg-giou_r50_caffe_fpn_gn-head_1x_coco/fcos_center-normbbox-centeronreg-giou_r50_caffe_fpn_gn-head_1x_coco-0a0d75a8.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/fcos/fcos_center-normbbox-centeronreg-giou_r50_caffe_fpn_gn-head_1x_coco/20210105_135818.log.json)       |

|   R-50   | caffe |  Y  |    N     |   Y    |  Y  |   1x    |   3.8    |       -        |  42.3  | [config](./fcos_r50-dcn-caffe_fpn_gn-head-center-normbbox-centeronreg-giou_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/fcos/fcos_center-normbbox-centeronreg-giou_r50_caffe_fpn_gn-head_dcn_1x_coco/fcos_center-normbbox-centeronreg-giou_r50_caffe_fpn_gn-head_dcn_1x_coco-ae4d8b3d.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/fcos/fcos_center-normbbox-centeronreg-giou_r50_caffe_fpn_gn-head_dcn_1x_coco/20210105_224556.log.json) |

|  R-101   | caffe |  Y  |    N     |   N    |  N  |   1x    |   5.5    |      17.3      |  39.1  |                   [config](./fcos_r101-caffe_fpn_gn-head-1x_coco.py)                   |                                                       [model](https://download.openmmlab.com/mmdetection/v2.0/fcos/fcos_r101_caffe_fpn_gn-head_1x_coco/fcos_r101_caffe_fpn_gn-head_1x_coco-0e37b982.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/fcos/fcos_r101_caffe_fpn_gn-head_1x_coco/20210103_155046.log.json)                                                       |

| Backbone |  Style  | GN  | MS train | Lr schd | Mem (GB) | Inf time (fps) | box AP |                            Config                             |                                                                                                                                                            Download                                                                                                                                                            |
| :------: | :-----: | :-: | :------: | :-----: | :------: | :------------: | :----: | :-----------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

|   R-50   |  caffe  |  Y  |    Y     |   2x    |   2.6    |      22.9      |  38.5  | [config](./fcos_r50-caffe_fpn_gn-head_ms-640-800-2x_coco.py)  |  [model](https://download.openmmlab.com/mmdetection/v2.0/fcos/fcos_r50_caffe_fpn_gn-head_mstrain_640-800_2x_coco/fcos_r50_caffe_fpn_gn-head_mstrain_640-800_2x_coco-d92ceeea.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/fcos/fcos_r50_caffe_fpn_gn-head_mstrain_640-800_2x_coco/20201227_161900.log.json)   |
|  R-101   |  caffe  |  Y  |    Y     |   2x    |   5.5    |      17.3      |  40.8  | [config](./fcos_r101-caffe_fpn_gn-head_ms-640-800-2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/fcos/fcos_r101_caffe_fpn_gn-head_mstrain_640-800_2x_coco/fcos_r101_caffe_fpn_gn-head_mstrain_640-800_2x_coco-511424d6.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/fcos/fcos_r101_caffe_fpn_gn-head_mstrain_640-800_2x_coco/20210103_155046.log.json) |

|  X-101   | pytorch |  Y  |    Y     |   2x    |   10.0   |      9.7       |  42.6  | [config](./fcos_x101-64x4d_fpn_gn-head_ms-640-800-2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/fcos/fcos_x101_64x4d_fpn_gn-head_mstrain_640-800_2x_coco/fcos_x101_64x4d_fpn_gn-head_mstrain_640-800_2x_coco-ede514a8.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/fcos/fcos_x101_64x4d_fpn_gn-head_mstrain_640-800_2x_coco/20210114_133041.log.json) |

- The X-101 backbone is X-101-64x4d.
- Tricks means setting `norm_on_bbox`, `centerness_on_reg`, `center_sampling` as `True`.
- DCN means using `DCNv2` in both backbone and head.

```latex
@article{tian2019fcos,
  title={FCOS: Fully Convolutional One-Stage Object Detection},
  author={Tian, Zhi and Shen, Chunhua and Chen, Hao and He, Tong},
  journal={arXiv preprint arXiv:1904.01355},
}
```

# 常见问题

## ModuleNotFoundError

### No module named 'mmengine.config.lazy'

可能是因为已经有旧版本的mmengine缓存在了本机。更新到最新班应该可以解决这个问题。

```shell
pip install --upgrade mmengine
```

### No module named '\_turbomind'

可能是因为：

1. 您没有安装 lmdeploy 的预编译包。`_turbomind`是 turbomind c++ 的 pybind部分，涉及到编译。推荐您直接安装预编译包。

```
```

2. 如果已经安装了，还是出现这个问题，请检查下执行目录。不要在 lmdeploy 的源码根目录下执行 python -m lmdeploy.turbomind.\*下的package，换到其他目录下执行。

## Libs

### libnccl.so.2 not found

确保通过 `pip install lmdeploy` 安装了 lmdeploy (>=v0.0.5)。

如果安装之后，问题还存在，那么就把`libnccl.so.2`的路径加入到环境变量 LD_LIBRARY_PATH 中。

```shell
# 获取nvidia-nccl-cu11 package的安装目录
pip show nvidia-nccl-cu11|grep Location
# 把"libnccl.so.2"的路径加入到 LD_LIBRARY_PATH
export LD_LIBRARY_PATH={Location}/nvidia/nccl/lib:$LD_LIBRARY_PATH
```

### symbol cudaFreeAsync version libcudart.so.11.0 not defined in file libcudart.so.11.0 with link time reference

很可能是机器上的 cuda 版本太低导致的。LMDeploy运行时要求 cuda 不低于 11.2

## Turbomind 推理

## Pytorch 推理

## 服务

## 量化

## 服务

## 量化

### RuntimeError: \[enforce fail at inline_container.cc:337\] . unexpected pos 4566829760 vs 4566829656

请检查你的硬盘空间。

这个错误是因为保存权重时硬盘空间不足导致的，在量化 70B 模型时可能会遇到

# Preparing Diving48

```BibTeX
@inproceedings{li2018resound,
  title={Resound: Towards action recognition without representation bias},
  author={Li, Yingwei and Li, Yi and Vasconcelos, Nuno},
  pages={513--528},
}
```

For basic dataset information, you can refer to the official dataset [website](http://www.svcl.ucsd.edu/projects/resound/dataset.html).

`````{tabs}

````{group-tab} Download by MIM
MIM supports downloading from OpenDataLab and preprocessing Diving48 dataset with one command line.
```Bash
# log in OpenDataLab
odl login
# download and preprocess by MIM
mim download mmaction2 --dataset diving48
```

````

````{group-tab} Download form Official Source

## Step 1. Prepare Annotations

odl login
```

````

You can run the following script to download annotations (considering the correctness of annotation files, we only download V2 version here).
Before we start, please make sure that the directory is located at `$MMACTION2/tools/data/diving48/`.

```shell
bash download_annotations.sh
```

## Step 2. Prepare Videos

You can run the following script to download videos.

```shell
bash download_videos.sh
```

## Step 3. Prepare RGB and Flow

This part is **optional** if you only want to use the video loader.

The frames provided in official compressed file are not complete. You may need to go through the following extraction steps to get the complete frames.

Before extracting, please refer to [install.md](/docs/en/get_started/installation.md) for installing [denseflow](https://github.com/open-mmlab/denseflow).

If you have plenty of SSD space, then we recommend extracting frames there for better I/O performance.

You can run the following script to soft link SSD.

```shell
# execute these two line (Assume the SSD is mounted at "/mnt/SSD/")
mkdir /mnt/SSD/diving48_extracted/
ln -s /mnt/SSD/diving48_extracted/ ../../../data/diving48/rawframes
```

If you only want to play with RGB frames (since extracting optical flow can be time-consuming), consider running the following script to extract **RGB-only** frames using denseflow.

```shell
cd $MMACTION2/tools/data/diving48/
bash extract_rgb_frames.sh
```

```shell
```

If you didn't install denseflow, you can still extract RGB frames using OpenCV by the following script, but it will keep the original size of the images.

```shell
bash extract_rgb_frames_opencv.sh
```

If both are required, run the following script to extract frames.

```shell
bash extract_frames.sh
```

## Step 4. Generate File List

you can run the follow script to generate file list in the format of rawframes and videos.

```shell
bash generate_videos_filelist.sh
bash generate_rawframes_filelist.sh
```

````
`````

### Check Directory Structure

After the whole data process for Diving48 preparation,
you will get the rawframes (RGB + Flow), videos and annotation files for Diving48.

In the context of the whole project (for Diving48 only), the folder structure will look like:

```
mmaction2
├── mmaction
├── tools
├── data

```
mmaction2
├── tools
├── data
│   ├── diving48
│   │   ├── diving48_{train,val}_list_rawframes.txt
│   │   ├── diving48_{train,val}_list_videos.txt
│   │   ├── annotations (optinonal)
│   |   |   ├── Diving48_V2_train.json
│   |   |   ├── Diving48_V2_test.json
│   |   |   ├── Diving48_vocab.json
│   |   ├── videos
│   |   |   ├── _8Vy3dlHg2w_00000.mp4
│   |   |   ├── _8Vy3dlHg2w_00001.mp4
│   |   |   ├── ...
│   |   ├── rawframes (optional)
│   |   |   ├── 2x00lRzlTVQ_00000
│   |   |   |   ├── img_00001.jpg
│   |   |   |   ├── img_00002.jpg
│   |   |   |   ├── ...
│   |   |   |   ├── flow_x_00001.jpg
│   |   |   |   ├── flow_x_00002.jpg
│   |   |   |   ├── flow_y_00001.jpg
│   |   |   |   ├── flow_y_00002.jpg
│   |   |   ├── 2x00lRzlTVQ_00001
│   |   |   ├── ...
```

│   |   |   ├── ...
```

For training and evaluating on Diving48, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).

# Train with Customized Datasets

In this note, you will know how to train and test predefined models with customized datasets. We use the Waymo dataset as an example to describe the whole process.

The basic steps are as below:

1. Prepare the customized dataset
2. Prepare a config
3. Train, test, inference models on the customized dataset.

## Prepare the customized dataset

There are three ways to support a new dataset in MMDetection3D:

1. reorganize the dataset into existing format.
2. reorganize the dataset into a standard format.
3. implement a new dataset.

Usually we recommend to use the first two methods which are usually easier than the third.

In this note, we give an example for converting the data into KITTI format, you can refer to this to reorganize your dataset into kitti format. About the standard format dataset, and you can refer to [customize_dataset.md](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/docs/en/advanced_guides/customize_dataset.md).

**Note**: We take Waymo as the example here considering its format is totally different from other existing formats. For other datasets using similar methods to organize data, like Lyft compared to nuScenes, it would be easier to directly implement the new data converter (for the second approach above) instead of converting it to another format (for the first approach above).

### KITTI dataset format

Firstly, the raw data for 3D object detection from KITTI are typically organized as follows, where `ImageSets` contains split files indicating which files belong to training/validation/testing set, `calib` contains calibration information files, `image_2` and `velodyne` include image data and point cloud data, and `label_2` includes label files for 3D detection.

```
├── tools
├── data
│   ├── kitti
│   │   ├── testing
│   │   │   ├── calib
│   │   │   ├── image_2
│   │   │   ├── velodyne
│   │   ├── training

├── tools
├── data
│   ├── kitti
│   │   │   ├── label_2
```

Specific annotation format is described in the official object development [kit](https://s3.eu-central-1.amazonaws.com/avg-kitti/devkit_object.zip). For example, it consists of the following labels:

```
#Values    Name      Description
----------------------------------------------------------------------------
   1    type         Describes the type of object: 'Car', 'Van', 'Truck',
                     'Pedestrian', 'Person_sitting', 'Cyclist', 'Tram',
                     'Misc' or 'DontCare'
   1    truncated    Float from 0 (non-truncated) to 1 (truncated), where
                     truncated refers to the object leaving image boundaries

'Misc' or 'DontCare'
   1    occluded     Integer (0,1,2,3) indicating occlusion state:
                     0 = fully visible, 1 = partly occluded
                     2 = largely occluded, 3 = unknown
   1    alpha        Observation angle of object, ranging [-pi..pi]
   4    bbox         2D bounding box of object in the image (0-based index):
                     contains left, top, right, bottom pixel coordinates
   3    dimensions   3D object dimensions: height, width, length (in meters)
   3    location     3D object location x,y,z in camera coordinates (in meters)
   1    rotation_y   Rotation ry around Y-axis in camera coordinates [-pi..pi]
   1    score        Only for results: Float, indicating confidence in
                     detection, needed for p/r curves, higher is better.
```

1    score        Only for results: Float, indicating confidence in
```

Assume we use the Waymo dataset.

After downloading the data, we need to implement a function to convert both the input data and annotation format into the KITTI style. Then we can implement `WaymoDataset` inherited from `KittiDataset` to load the data and perform training, and implement `WaymoMetric` inherited from `KittiMetric` for evaluation.

Specifically, we implement a waymo [converter](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/waymo_converter.py) to convert Waymo data into KITTI format and a waymo dataset [class](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/datasets/waymo_dataset.py) to process it, in addition need to add a waymo [metric](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/evaluation/metrics/waymo_metric.py) to evaluate results. Because we preprocess the raw data and reorganize it like KITTI, the dataset class could be implemented more easily by inheriting from KittiDataset. Regarding the dataset evaluation metric, because Waymo has its own evaluation approach, we need further implement a new Waymo metric; more about the metric could refer to [metric_and_evaluator.md](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/metric_and_evaluator.md). Afterward, users can successfully convert the data format and use `WaymoDataset` to train and evaluate the model by `WaymoMetric`.

For more details about the intermediate results of preprocessing of Waymo dataset, please refer to its [waymo_det.md](https://mmdetection3d.readthedocs.io/en/latest/datasets/waymo_det.html).

## Prepare a config

The second step is to prepare configs such that the dataset could be successfully loaded. In addition, adjusting hyperparameters is usually necessary to obtain decent performance in 3D detection.

Suppose we would like to train PointPillars on Waymo to achieve 3D detection for 3 classes, vehicle, cyclist and pedestrian, we need to prepare dataset config like [this](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/datasets/waymoD5-3d-3class.py), model config like [this](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/models/pointpillars_hv_secfpn_waymo.py) and combine them like [this](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/pointpillars/pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymoD5-3d-3class.py), compared to KITTI [dataset config](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/datasets/kitti-3d-3class.py), [model config](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/models/pointpillars_hv_secfpn_kitti.py) and [overall](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py).

## Train a new model

To train a model with the new config, you can simply run

```shell
python tools/train.py configs/pointpillars/pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymoD5-3d-3class.py
```

For more detailed usages, please refer to the [Case 1](https://mmdetection3d.readthedocs.io/en/latest/1_exist_data_model.html).

## Test and inference

To test the trained model, you can simply run

```shell
python tools/test.py configs/pointpillars/pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymoD5-3d-3class.py work_dirs/pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymoD5-3d-3class/latest.pth
```

**Note**: To use Waymo evaluation protocol, you need to follow the [tutorial](https://mmdetection3d.readthedocs.io/en/latest/datasets/waymo_det.html) and prepare files related to metrics computation as official instructions.

For more detailed usages for test and inference, please refer to the [Case 1](https://mmdetection3d.readthedocs.io/en/latest/1_exist_data_model.html).

```json
{
    "imnames": [['8/ballet_106_0.jpg', ...]],
    "wordBB": [[[420.58957   418.85016   448.08478   410.3094    117.745026
                322.30963   322.6857    159.09138   154.27284   260.14597
                431.9315    427.52274   296.86508    99.56819   108.96211  ]
               [512.3321    431.88342   519.4515    499.81183   179.0544
                377.97382   376.4993    203.64464   193.77492   313.61514
                487.58023   484.64633   365.83176   142.49403   144.90457  ]
               [511.92203   428.7077    518.7375    499.0373    172.1684
                378.35858   377.2078    203.3191    193.0739    319.69186
                485.6758    482.571     365.76303   142.31898   144.43858  ]
               [420.1795    415.67444   447.3708    409.53485   110.859024
                322.6944    323.3942    158.76585   153.57182   266.2227
                430.02707   425.44742   296.79636    99.39314   108.49613  ]]

[ 39.82139    66.3395     67.47641    95.43123   233.77136
                 31.528908   70.33074    31.937548   72.534775  269.71988
                143.50633   184.14066    71.96825    22.304657   50.030033 ]], ...],
    "charBB": [[[423.16126397 439.60847343 450.66887979 466.31976402 479.76190495
                504.59927448 418.80489444 450.13965942 464.16775197 480.46891089
                502.46437709 413.02373632 433.01396211 446.7222192  470.28467827
                482.51674486 116.52285438 139.51408587 150.7448586  162.03366629
                322.84717946 333.54881536 343.28386485 363.07416389 323.48968759
                337.98503283 356.66355903 160.48517048 174.1707753  189.64454066
                155.7637383  167.45490471 179.63644201 262.2183876  271.75848874
                284.05396524 298.26103738 432.8464733  449.15387392 468.07231897
                428.11482147 445.61538159 469.24565878 296.86441324 323.6603118

[ 39.82138755  40.18347166  40.44598236  40.79219178  41.08959901
                 41.64111176  66.33948982  67.47640971  68.01403337  68.60595247
                 69.3953105   95.13188979  95.21297344  95.91593691  97.08847413
                 97.75212171 229.94285119 237.26055111 240.66752705 242.74145162
                 31.52890731  31.33842788  31.16401306  30.81155638  69.87135926
                 68.80273568  68.71664209  31.93753588  32.13243303  32.34474067
                 72.53476992  72.88981775  73.28094858 269.71986636 267.92938572
                262.93698624 256.88902439 143.50635029 143.61251781 146.24080653
                184.14064261 185.86853729 188.17713786  71.96823746  53.79651809
                 54.60870874  22.30465649  22.41084398  22.51791234  17.07939535
                 22.63671808  50.03002471  45.81009198  50.49899163]], ...],
    "txt": [['Lines:\nI lost\nKevin ' 'will                ' 'line\nand            '

22.63671808  50.03002471  45.81009198  50.49899163]], ...],
              'and\nthe             ' '(and                ' 'the\nout             '
              'you                 ' "don't\n pkg          "], ...]
}
```

<details>
<summary align="right"><a href="https://arxiv.org/abs/2104.06403">LiteHRNet (CVPR'2021)</a></summary>

```bibtex
@inproceedings{Yulitehrnet21,
  title={Lite-HRNet: A Lightweight High-Resolution Network},
  author={Yu, Changqian and Xiao, Bin and Gao, Changxin and Yuan, Lu and Zhang, Lei and Sang, Nong and Wang, Jingdong},
}
```

<details>

```bibtex
}
```

}
```

| [LiteHRNet-18](/configs/hand_2d_keypoint/topdown_heatmap/coco_wholebody_hand/td-hm_litehrnet-w18_8xb32-210e_coco-wholebody-hand-256x256.py) |  256x256   |  0.795  | 0.830 | 4.77 | [ckpt](https://download.openmmlab.com/mmpose/hand/litehrnet/litehrnet_w18_coco_wholebody_hand_256x256-d6945e6a_20210908.pth) | [log](https://download.openmmlab.com/mmpose/hand/litehrnet/litehrnet_w18_coco_wholebody_hand_256x256_20210908.log.json) |

# ncnn 支持情况

目前对 ncnn 特性使用情况如下：

|      feature       | windows | linux | mac | android |
| :----------------: | :-----: | :---: | :-: | :-----: |
|   fp32 inference   |   ✔️    |  ✔️   | ✔️  |   ✔️    |
| int8 model convert |    -    |  ✔️   | ✔️  |    -    |
|    nchw layout     |   ✔️    |  ✔️   | ✔️  |   ✔️    |
|   Vulkan support   |    -    |  ✔️   | ✔️  |   ✔️    |

以下特性还不能由 mmdeploy 自动开启，需要手动修改 ncnn 编译参数、或在 SDK 中调整运行参数

- bf16 inference
- nc4hw4 layout
- profiling per layer
- 关闭 NCNN_STRING 以减小 so 体积
- 设置线程数和 CPU 亲和力

}
```

<details>

```bibtex
}
```

| [pose_res50](/configs/face_2d_keypoint/topdown_heatmap/coco_wholebody_face/td-hm_res50_8xb32-60e_coco-wholebody-face-256x256.py) |  256x256   | 0.0582 | [ckpt](https://download.openmmlab.com/mmpose/face/resnet/res50_coco_wholebody_face_256x256-5128edf5_20210909.pth) | [log](https://download.openmmlab.com/mmpose/face/resnet/res50_coco_wholebody_face_256x256_20210909.log.json) |

<details>

```bibtex
}
```

<details>

```bibtex

```bibtex
}
```

| [pose_hrnet_w32](/configs/animal_2d_keypoint/topdown_heatmap/animalpose/td-hm_hrnet-w32_8xb64-210e_animalpose-256x256.py) |  256x256   | 0.740 |      0.959      |      0.833      | 0.780 |      0.965      | [ckpt](https://download.openmmlab.com/mmpose/animal/hrnet/hrnet_w32_animalpose_256x256-1aa7f075_20210426.pth) | [log](https://download.openmmlab.com/mmpose/animal/hrnet/hrnet_w32_animalpose_256x256_20210426.log.json) |
| [pose_hrnet_w48](/configs/animal_2d_keypoint/topdown_heatmap/animalpose/td-hm_hrnet-w48_8xb64-210e_animalpose-256x256.py) |  256x256   | 0.738 |      0.958      |      0.831      | 0.778 |      0.962      | [ckpt](https://download.openmmlab.com/mmpose/animal/hrnet/hrnet_w48_animalpose_256x256-34644726_20210426.pth) | [log](https://download.openmmlab.com/mmpose/animal/hrnet/hrnet_w48_animalpose_256x256_20210426.log.json) |

# Robust face landmark estimation under occlusion

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_iccv_2013/html/Burgos-Artizzu_Robust_Face_Landmark_2013_ICCV_paper.html">COFW (ICCV'2013)</a></summary>

```bibtex
@inproceedings{burgos2013robust,
  title={Robust face landmark estimation under occlusion},
  author={Burgos-Artizzu, Xavier P and Perona, Pietro and Doll{\'a}r, Piotr},
  pages={1513--1520},
  year={2013}
}
```

# EfficientFormer

> [EfficientFormer: Vision Transformers at MobileNet Speed](https://arxiv.org/abs/2206.01191)

Vision Transformers (ViT) have shown rapid progress in computer vision tasks, achieving promising results on various benchmarks. However, due to the massive number of parameters and model design, e.g., attention mechanism, ViT-based models are generally times slower than lightweight convolutional networks. Therefore, the deployment of ViT for real-time applications is particularly challenging, especially on resource-constrained hardware such as mobile devices. Recent efforts try to reduce the computation complexity of ViT through network architecture search or hybrid design with MobileNet block, yet the inference speed is still unsatisfactory. This leads to an important question: can transformers run as fast as MobileNet while obtaining high performance? To answer this, we first revisit the network architecture and operators used in ViT-based models and identify inefficient designs. Then we introduce a dimension-consistent pure transformer (without MobileNet blocks) as a design paradigm.  Finally, we perform latency-driven slimming to get a series of final models dubbed EfficientFormer. Extensive experiments show the superiority of EfficientFormer in performance and speed on mobile devices. Our fastest model, EfficientFormer-L1, achieves 79.2% top-1 accuracy on ImageNet-1K with only 1.6 ms inference latency on iPhone 12 (compiled with CoreML), which runs as fast as MobileNetV2×1.4 (1.6 ms, 74.7% top-1), and our largest model, EfficientFormer-L7, obtains 83.3% accuracy with only 7.0 ms latency. Our work proves that properly designed transformers can reach extremely low latency on mobile devices while maintaining high performance.

<img src="https://user-images.githubusercontent.com/18586273/180713426-9d3d77e3-3584-42d8-9098-625b4170d796.png" width="100%"/>
</div>

```python

predict = inference_model('efficientformer-l1_3rdparty_8xb128_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('efficientformer-l1_3rdparty_8xb128_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/efficientformer/efficientformer-l1_8xb128_in1k.py https://download.openmmlab.com/mmclassification/v0/efficientformer/efficientformer-l1_3rdparty_in1k_20220915-cc3e1ac6.pth
```

| Model                                       |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                   Config                    |                             Download                              |
| :------------------------------------------ | :----------: | :--------: | :-------: | :-------: | :-------: | :-----------------------------------------: | :---------------------------------------------------------------: |

| `efficientformer-l1_3rdparty_8xb128_in1k`\* | From scratch |   12.28    |   1.30    |   80.46   |   94.99   | [config](efficientformer-l1_8xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/efficientformer/efficientformer-l1_3rdparty_in1k_20220915-cc3e1ac6.pth) |
| `efficientformer-l3_3rdparty_8xb128_in1k`\* | From scratch |   31.41    |   3.74    |   82.45   |   96.18   | [config](efficientformer-l3_8xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/efficientformer/efficientformer-l3_3rdparty_in1k_20220915-466793d6.pth) |
| `efficientformer-l7_3rdparty_8xb128_in1k`\* | From scratch |   82.23    |   10.16   |   83.40   |   96.60   | [config](efficientformer-l7_8xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/efficientformer/efficientformer-l7_3rdparty_in1k_20220915-185e30af.pth) |

*Models with * are converted from the [official repo](https://github.com/snap-research/EfficientFormer). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@misc{https://doi.org/10.48550/arxiv.2206.01191,
  doi = {10.48550/ARXIV.2206.01191},

  url = {https://arxiv.org/abs/2206.01191},

  author = {Li, Yanyu and Yuan, Geng and Wen, Yang and Hu, Eric and Evangelidis, Georgios and Tulyakov, Sergey and Wang, Yanzhi and Ren, Jian},

  title = {EfficientFormer: Vision Transformers at MobileNet Speed},

  year = {2022},

}
```

# FSAF

> [Feature Selective Anchor-Free Module for Single-Shot Object Detection](https://arxiv.org/abs/1903.00621)

We motivate and present feature selective anchor-free (FSAF) module, a simple and effective building block for single-shot object detectors. It can be plugged into single-shot detectors with feature pyramid structure. The FSAF module addresses two limitations brought up by the conventional anchor-based detection: 1) heuristic-guided feature selection; 2) overlap-based anchor sampling. The general concept of the FSAF module is online feature selection applied to the training of multi-level anchor-free branches. Specifically, an anchor-free branch is attached to each level of the feature pyramid, allowing box encoding and decoding in the anchor-free manner at an arbitrary level. During training, we dynamically assign each instance to the most suitable feature level. At the time of inference, the FSAF module can work jointly with anchor-based branches by outputting predictions in parallel. We instantiate this concept with simple implementations of anchor-free branches and online feature selection strategy. Experimental results on the COCO detection track show that our FSAF module performs better than anchor-based counterparts while being faster. When working jointly with anchor-based branches, the FSAF module robustly improves the baseline RetinaNet by a large margin under various settings, while introducing nearly free inference overhead. And the resulting best model can achieve a state-of-the-art 44.6% mAP, outperforming all existing single-shot detectors on COCO.

<img src="https://user-images.githubusercontent.com/40661020/143887228-c790b542-c0f7-4113-8597-12250c50fe8f.png"/>
</div>

FSAF is an anchor-free method published in CVPR2019 ([https://arxiv.org/pdf/1903.00621.pdf](https://arxiv.org/pdf/1903.00621.pdf)).
Actually it is equivalent to the anchor-based method with only one anchor at each feature map position in each FPN level.
And this is how we implemented it.
Only the anchor-free branch is released for its better compatibility with the current framework and less computational budget.

In the original paper, feature maps within the central 0.2-0.5 area of a gt box are tagged as ignored. However,
it is empirically found that a hard threshold (0.2-0.2) gives a further gain on the performance. (see the table below)

### Results on R50/R101/X101-FPN

| Backbone | ignore range | ms-train | Lr schd | Train Mem (GB) | Train time (s/iter) | Inf time (fps) |   box AP    |                   Config                   |                                                                                                                                                                    Download                                                                                                                                                                     |

|   R-50   |   0.2-0.5    |    N     |   1x    |      3.15      |        0.43         |      12.3      | 36.0 (35.9) |                                            | [model](https://download.openmmlab.com/mmdetection/v2.0/fsaf/fsaf_pscale0.2_nscale0.5_r50_fpn_1x_coco/fsaf_pscale0.2_nscale0.5_r50_fpn_1x_coco_20200715-b555b0e0.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/fsaf/fsaf_pscale0.2_nscale0.5_r50_fpn_1x_coco/fsaf_pscale0.2_nscale0.5_r50_fpn_1x_coco_20200715_094657.log.json) |

|   R-50   |   0.2-0.2    |    N     |   1x    |      3.15      |        0.43         |      13.0      |    37.4     |    [config](./fsaf_r50_fpn_1x_coco.py)     |                                             [model](https://download.openmmlab.com/mmdetection/v2.0/fsaf/fsaf_r50_fpn_1x_coco/fsaf_r50_fpn_1x_coco-94ccc51f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/fsaf/fsaf_r50_fpn_1x_coco/fsaf_r50_fpn_1x_coco_20200428_072327.log.json)                                              |

|  R-101   |   0.2-0.2    |    N     |   1x    |      5.08      |        0.58         |      10.8      | 39.3 (37.9) |    [config](./fsaf_r101_fpn_1x_coco.py)    |                                           [model](https://download.openmmlab.com/mmdetection/v2.0/fsaf/fsaf_r101_fpn_1x_coco/fsaf_r101_fpn_1x_coco-9e71098f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/fsaf/fsaf_r101_fpn_1x_coco/fsaf_r101_fpn_1x_coco_20200428_160348.log.json)                                            |

|  X-101   |   0.2-0.2    |    N     |   1x    |      9.38      |        1.23         |      5.6       | 42.4 (41.0) | [config](./fsaf_x101-64x4d_fpn_1x_coco.py) |                               [model](https://download.openmmlab.com/mmdetection/v2.0/fsaf/fsaf_x101_64x4d_fpn_1x_coco/fsaf_x101_64x4d_fpn_1x_coco-e3f6e6fd.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/fsaf/fsaf_x101_64x4d_fpn_1x_coco/fsaf_x101_64x4d_fpn_1x_coco_20200428_160424.log.json)                                |

- *1x means the model is trained for 12 epochs.*
- *AP values in the brackets represent those reported in the original paper.*
- *All results are obtained with a single model and single-scale test.*
- *X-101 backbone represents ResNext-101-64x4d.*
- *All pretrained backbones use pytorch style.*
- *All models are trained on 8 Titan-XP gpus and tested on a single gpu.*

BibTeX reference is as follows.

```latex
@inproceedings{zhu2019feature,

```latex
  title={Feature Selective Anchor-Free Module for Single-Shot Object Detection},
  author={Zhu, Chenchen and He, Yihui and Savvides, Marios},
  pages={840--849},
}
```

# 第二章：解决模型部署中的难题

在[第一章](https://mmdeploy.readthedocs.io/zh_CN/latest/tutorial/01_introduction_to_model_deployment.html)中，我们部署了一个简单的超分辨率模型，一切都十分顺利。但是，上一个模型还有一些缺陷——图片的放大倍数固定是 4，我们无法让图片放大任意的倍数。现在，我们来尝试部署一个支持动态放大倍数的模型，体验一下在模型部署中可能会碰到的困难。

## 模型部署中常见的难题

在之前的学习中，我们在模型部署上顺风顺水，没有碰到任何问题。这是因为 SRCNN 模型只包含几个简单的算子，而这些卷积、插值算子已经在各个中间表示和推理引擎上得到了完美支持。如果模型的操作稍微复杂一点，我们可能就要为兼容模型而付出大量的功夫了。实际上，模型部署时一般会碰到以下几类困难：

- 模型的动态化。出于性能的考虑，各推理框架都默认模型的输入形状、输出形状、结构是静态的。而为了让模型的泛用性更强，部署时需要在尽可能不影响原有逻辑的前提下，让模型的输入输出或是结构动态化。
- 新算子的实现。深度学习技术日新月异，提出新算子的速度往往快于 ONNX 维护者支持的速度。为了部署最新的模型，部署工程师往往需要自己在 ONNX 和推理引擎中支持新算子。
- 中间表示与推理引擎的兼容问题。由于各推理引擎的实现不同，对 ONNX 难以形成统一的支持。为了确保模型在不同的推理引擎中有同样的运行效果，部署工程师往往得为某个推理引擎定制模型代码，这为模型部署引入了许多工作量。

我们会在后续教程详细讲述解决这些问题的方法。如果对前文中 ONNX、推理引擎、中间表示、算子等名词感觉陌生，不用担心，可以阅读[第一章](https://mmdeploy.readthedocs.io/zh_CN/latest/tutorial/01_introduction_to_model_deployment.html)，了解有关概念。

现在，让我们对原来的 SRCNN 模型做一些小的修改，体验一下模型动态化对模型部署造成的困难，并学习解决该问题的一种方法。

## 问题：实现动态放大的超分辨率模型

在原来的 SRCNN 中，图片的放大比例是写死在模型里的：

```python

```python
class SuperResolutionNet(nn.Module):
    def __init__(self, upscale_factor):
        self.upscale_factor = upscale_factor
        self.img_upsampler = nn.Upsample(
            scale_factor=self.upscale_factor,
            mode='bicubic',
            align_corners=False)

...

def init_torch_model():
    torch_model = SuperResolutionNet(upscale_factor=3)
```

我们使用 upscale_factor 来控制模型的放大比例。初始化模型的时候，我们默认令 upscale_factor 为 3，生成了一个放大 3 倍的 PyTorch 模型。这个 PyTorch 模型最终被转换成了 ONNX 格式的模型。如果我们需要一个放大 4 倍的模型，需要重新生成一遍模型，再做一次到 ONNX 的转换。

现在，假设我们要做一个超分辨率的应用。我们的用户希望图片的放大倍数能够自由设置。而我们交给用户的，只有一个 .onnx 文件和运行超分辨率模型的应用程序。我们在不修改 .onnx 文件的前提下改变放大倍数。

因此，我们必须修改原来的模型，令模型的放大倍数变成推理时的输入。在[第一章](https://mmdeploy.readthedocs.io/zh_CN/latest/tutorial/01_introduction_to_model_deployment.html)中的 Python 脚本的基础上，我们做一些修改，得到这样的脚本：

```python
from torch import nn

```python
from torch.nn.functional import interpolate
import cv2

        self.conv1 = nn.Conv2d(3, 64, kernel_size=9, padding=4)
        self.conv2 = nn.Conv2d(64, 32, kernel_size=1, padding=0)
        self.conv3 = nn.Conv2d(32, 3, kernel_size=5, padding=2)

        self.relu = nn.ReLU()

    def forward(self, x, upscale_factor):
        x = interpolate(x,
                        scale_factor=upscale_factor,
                        mode='bicubic',
                        align_corners=False)
        out = self.relu(self.conv1(x))
        out = self.relu(self.conv2(out))
        out = self.conv3(out)
        return out

    torch_model = SuperResolutionNet()

    # Please read the code about downloading 'srcnn.pth' and 'face.png' in

out = self.conv3(out)
        return out

    # https://mmdeploy.readthedocs.io/zh_CN/latest/tutorial/01_introduction_to_model_deployment.html#pytorch
    state_dict = torch.load('srcnn.pth')['state_dict']

    # Adapt the checkpoint
    for old_key in list(state_dict.keys()):
        new_key = '.'.join(old_key.split('.')[1:])
        state_dict[new_key] = state_dict.pop(old_key)

    torch_model.load_state_dict(state_dict)
    torch_model.eval()
    return torch_model

model = init_torch_model()

input_img = cv2.imread('face.png').astype(np.float32)

# HWC to NCHW
input_img = np.transpose(input_img, [2, 0, 1])
input_img = np.expand_dims(input_img, 0)

# Inference
torch_output = model(torch.from_numpy(input_img), 3).detach().numpy()

# NCHW to HWC
torch_output = np.squeeze(torch_output, 0)
torch_output = np.clip(torch_output, 0, 255)

torch_output = np.transpose(torch_output, [1, 2, 0]).astype(np.uint8)

# Show image
cv2.imwrite("face_torch_2.png", torch_output)
```

SuperResolutionNet 未修改之前，nn.Upsample 在初始化阶段固化了放大倍数，而 PyTorch 的 interpolate 插值算子可以在运行阶段选择放大倍数。因此，我们在新脚本中使用 interpolate 代替 nn.Upsample，从而让模型支持动态放大倍数的超分。 在第 55 行使用模型推理时，我们把放大倍数设置为 3。最后，图片保存在文件 "face_torch_2.png" 中。一切正常的话，"face_torch_2.png" 和 "face_torch.png" 的内容一模一样。

通过简单的修改，PyTorch 模型已经支持了动态分辨率。现在我们来一下尝试导出模型：

```python
x = torch.randn(1, 3, 256, 256)

    torch.onnx.export(model, (x, 3),
                      "srcnn2.onnx",
                      opset_version=11,
                      input_names=['input', 'factor'],
                      output_names=['output'])
```

运行这些脚本时，会报一长串错误。没办法，我们碰到了模型部署中的兼容性问题。

## 解决方法：自定义算子

opset_version=11,
```

直接使用 PyTorch 模型的话，我们修改几行代码就能实现模型输入的动态化。但在模型部署中，我们要花数倍的时间来设法解决这一问题。现在，让我们顺着解决问题的思路，体验一下模型部署的困难，并学习使用自定义算子的方式，解决超分辨率模型的动态化问题。

刚刚的报错是因为 PyTorch 模型在导出到 ONNX 模型时，模型的输入参数的类型必须全部是 torch.Tensor。而实际上我们传入的第二个参数" 3 "是一个整形变量。这不符合 PyTorch 转 ONNX 的规定。我们必须要修改一下原来的模型的输入。为了保证输入的所有参数都是 torch.Tensor 类型的，我们做如下修改：

```python
...

                        scale_factor=upscale_factor.item(),

...

# Note that the second input is torch.tensor(3)
torch_output = model(torch.from_numpy(input_img), torch.tensor(3)).detach().numpy()

...

    torch.onnx.export(model, (x, torch.tensor(3)),

...

```

由于 PyTorch 中 interpolate 的 scale_factor 参数必须是一个数值，我们使用 torch.Tensor.item() 来把只有一个元素的 torch.Tensor 转换成数值。之后，在模型推理时，我们使用 torch.tensor(3) 代替 3，以使得我们的所有输入都满足要求。现在运行脚本的话，无论是直接运行模型，还是导出 ONNX 模型，都不会报错了。

但是，导出 ONNX 时却报了一条 TraceWarning 的警告。这条警告说有一些量可能会追踪失败。这是怎么回事呢？让我们把生成的 srcnn2.onnx 用 Netron 可视化一下：
![image](https://user-images.githubusercontent.com/4560679/157626717-ff60a656-7246-4742-a227-d9f41921ecad.png)

可以发现，虽然我们把模型推理的输入设置为了两个，但 ONNX 模型还是长得和原来一模一样，只有一个叫 " input " 的输入。这是由于我们使用了 torch.Tensor.item() 把数据从 Tensor 里取出来，而导出 ONNX 模型时这个操作是无法被记录的，只好报了一条 TraceWarning。这导致 interpolate 插值函数的放大倍数还是被设置成了" 3 "这个固定值，我们导出的" srcnn2.onnx "和最开始的" srcnn.onnx "完全相同。

直接修改原来的模型似乎行不通，我们得从 PyTorch 转 ONNX 的原理入手，强行令 ONNX 模型明白我们的想法了。

仔细观察 Netron 上可视化出的 ONNX 模型，可以发现在 PyTorch 中无论是使用最早的 nn.Upsample，还是后来的 interpolate，PyTorch 里的插值操作最后都会转换成 ONNX 定义的 Resize 操作。也就是说，所谓 PyTorch 转 ONNX，实际上就是把每个 PyTorch 的操作映射成了 ONNX 定义的算子。

点击该算子，可以看到它的详细参数如下：

![image](https://user-images.githubusercontent.com/4560679/157626910-de33365c-b60a-49f4-ada7-157111afa6e2.png)

其中，展开 scales，可以看到 scales 是一个长度为 4 的一维张量，其内容为 [1, 1, 3, 3], 表示 Resize  操作每一个维度的缩放系数；其类型为 Initializer，表示这个值是根据常量直接初始化出来的。如果我们能够自己生成一个 ONNX 的 Resize 算子，让 scales 成为一个可变量而不是常量，就像它上面的 X 一样，那这个超分辨率模型就能动态缩放了。

现有实现插值的 PyTorch 算子有一套规定好的映射到 ONNX Resize 算子的方法，这些映射出的 Resize 算子的 scales 只能是常量，无法满足我们的需求。我们得自己定义一个实现插值的 PyTorch 算子，然后让它映射到一个我们期望的 ONNX Resize 算子上。

下面的脚本定义了一个 PyTorch 插值算子，并在模型里使用了它。我们先通过运行模型来验证该算子的正确性：

```python
import cv2

class NewInterpolate(torch.autograd.Function):

    @staticmethod

import cv2

    def symbolic(g, input, scales):
        return g.op("Resize",
                    input,
                    g.op("Constant",
                         value_t=torch.tensor([], dtype=torch.float32)),
                    scales,
                    coordinate_transformation_mode_s="pytorch_half_pixel",
                    cubic_coeff_a_f=-0.75,
                    mode_s='cubic',
                    nearest_mode_s="floor")

    def forward(ctx, input, scales):
        scales = scales.tolist()[-2:]
        return interpolate(input,
                           scale_factor=scales,
                           mode='bicubic',
                           align_corners=False)

class StrangeSuperResolutionNet(nn.Module):

mode='bicubic',

        x = NewInterpolate.apply(x, upscale_factor)
        return out

    torch_model = StrangeSuperResolutionNet()

for old_key in list(state_dict.keys()):

factor = torch.tensor([1, 1, 3, 3], dtype=torch.float)

torch_output = model(torch.from_numpy(input_img), factor).detach().numpy()

cv2.imwrite("face_torch_3.png", torch_output)
```

模型运行正常的话，一幅放大3倍的超分辨率图片会保存在"face_torch_3.png"中，其内容和"face_torch.png"完全相同。

在刚刚那个脚本中，我们定义 PyTorch 插值算子的代码如下：

```python

```

```python

                    input,
                    scales,

```

return interpolate(input,
```

在具体介绍这个算子的实现前，让我们先理清一下思路。我们希望新的插值算子有两个输入，一个是被用于操作的图像，一个是图像的放缩比例。前面讲到，为了对接 ONNX 中 Resize 算子的 scales 参数，这个放缩比例是一个 [1, 1, x, x] 的张量，其中 x 为放大倍数。在之前放大3倍的模型中，这个参数被固定成了[1, 1, 3, 3]。因此，在插值算子中，我们希望模型的第二个输入是一个 [1, 1, w, h] 的张量，其中 w 和 h 分别是图片宽和高的放大倍数。

搞清楚了插值算子的输入，再看一看算子的具体实现。算子的推理行为由算子的 forward 方法决定。该方法的第一个参数必须为 ctx，后面的参数为算子的自定义输入，我们设置两个输入，分别为被操作的图像和放缩比例。为保证推理正确，需要把 [1, 1, w, h] 格式的输入对接到原来的 interpolate 函数上。我们的做法是截取输入张量的后两个元素，把这两个元素以 list 的格式传入 interpolate 的 scale_factor 参数。

接下来，我们要决定新算子映射到 ONNX 算子的方法。映射到 ONNX 的方法由一个算子的 symbolic 方法决定。symbolic 方法第一个参数必须是g，之后的参数是算子的自定义输入，和 forward 函数一样。ONNX 算子的具体定义由 g.op 实现。g.op 的每个参数都可以映射到 ONNX 中的算子属性：

![image](https://user-images.githubusercontent.com/4560679/157627349-10ed5483-8bde-47d0-b190-a7a4d0ca2c03.png)

对于其他参数，我们可以照着现在的 [Resize](https://github.com/onnx/onnx/blob/main/docs/Operators.md#Resize) 算子填。而要注意的是，我们现在希望 scales 参数是由输入动态决定的。因此，在填入 ONNX 的 scales 时，我们要把 symbolic 方法的输入参数中的 scales 填入。

接着，让我们把新模型导出成 ONNX 模型：

```python

    torch.onnx.export(model, (x, factor),
                      "srcnn3.onnx",
```

把导出的 " srcnn3.onnx " 进行可视化：

![image](https://user-images.githubusercontent.com/4560679/157627615-b923fcb5-45ab-42c0-87b5-167cd657e04e.png)

可以看到，正如我们所期望的，导出的 ONNX 模型有了两个输入！第二个输入表示图像的放缩比例。

之前在验证 PyTorch 模型和导出 ONNX 模型时，我们宽高的缩放比例设置成了 3x3。现在，在用 ONNX Runtime 推理时，我们尝试使用 4x4 的缩放比例：

```python

input_factor = np.array([1, 1, 4, 4], dtype=np.float32)

```python

ort_session = onnxruntime.InferenceSession("srcnn3.onnx")
ort_inputs = {'input': input_img, 'factor': input_factor}
ort_output = ort_session.run(None, ort_inputs)[0]

ort_output = np.squeeze(ort_output, 0)
ort_output = np.clip(ort_output, 0, 255)
ort_output = np.transpose(ort_output, [1, 2, 0]).astype(np.uint8)
cv2.imwrite("face_ort_3.png", ort_output)
```

运行上面的代码，可以得到一个边长放大4倍的超分辨率图片 "face_ort_3.png"。动态的超分辨率模型生成成功了！只要修改 input_factor，我们就可以自由地控制图片的缩放比例。

我们刚刚的工作，实际上是绕过 PyTorch 本身的限制，凭空“捏”出了一个 ONNX 算子。事实上，我们不仅可以创建现有的 ONNX 算子，还可以定义新的 ONNX 算子以拓展 ONNX 的表达能力。后续教程中我们将介绍自定义新 ONNX 算子的方法。

## 总结

通过学习前两篇教程，我们走完了整个部署流水线，成功部署了支持动态放大倍数的超分辨率模型。在这个过程中，我们既学会了如何简单地调用各框架的API实现模型部署，又学到了如何分析并尝试解决模型部署时碰到的难题。

同样，让我们总结一下本篇教程的知识点：

- 模型部署中常见的几类困难有：模型的动态化；新算子的实现；框架间的兼容。

## 总结

- PyTorch 转 ONNX，实际上就是把每一个操作转化成 ONNX 定义的某一个算子。比如对于 PyTorch 中的 Upsample 和 interpolate，在转 ONNX 后最终都会成为 ONNX 的 Resize 算子。
- 通过修改继承自 torch.autograd.Function 的算子的 symbolic 方法，可以改变该算子映射到 ONNX 算子的行为。

至此，"部署第一个模型“的教程算是告一段落了。是不是觉得学到的知识还不够多？没关系，在接下来的几篇教程中，我们将结合 MMDeploy ，重点介绍 ONNX 中间表示和 ONNX Runtime/TensorRT 推理引擎的知识，让大家学会如何部署更复杂的模型。

```c
// d寄存器全部使用 %P
// d reg matches %P
// a += b * c
float32x2_t _a = vld1_f32(a);
float32x2_t _b = vld1_f32(b);
float32x2_t _c = vld1_f32(c);
asm volatile(
    "vmla.f32  %P0, %P2, %P3"
    : "=w"(_a) // %0
    : "0"(_a),
      "w"(_b), // %2
      "w"(_c)  // %3
    :
);
```
```c
// q寄存器全部使用 %q
// q reg matches %q
// a += b * c
float32x4_t _a = vld1q_f32(a);
float32x4_t _b = vld1q_f32(b);
float32x4_t _c = vld1q_f32(c);
    "vmla.f32  %q0, %q2, %q3"
    : "0"(_a),
    :
);
```
```c
// d寄存器单路使用 %P[0] %P[1]
// 32bit d reg matches %P[0]
// a += b * c[0]
// a += b * c[1]
    "vmla.f32  %P0, %P2, %P3[0]"
    "vmla.f32  %P0, %P2, %P3[1]"
    : "0"(_a),
    :
);
```
```c
// q寄存器单路使用 %e[0] %e[1] %f[0] %f[1]
// 32-bit q reg matches %e[0]

"vmla.f32  %P0, %P2, %P3[1]"
    : "0"(_a),
    :
);
```
```c
// a += b * c[2]
// a += b * c[3]
    "vmla.f32  %q0, %q2, %e3[0]"
    "vmla.f32  %q0, %q2, %e3[1]"
    "vmla.f32  %q0, %q2, %f3[0]"
    "vmla.f32  %q0, %q2, %f3[1]"
    : "0"(_a),
    :
);
```
```c
// q寄存器拆分d寄存器使用 %e %f
// use %e %f to split q reg into two d regs
// a += b * c[0]c[1]
// a += b * c[2]c[3]
float32x2_t _a = vldq_f32(a);
float32x2_t _b = vldq_f32(b);
    "vmla.f32  %P0, %P2, %e3"
    "vmla.f32  %P0, %P2, %f3"
    : "0"(_a),
    :
);
```
```c
// d寄存器声明绑定

    : "0"(_a),
    :
);
```
```c
// specify concrete d reg which want to save
// vmla.f32  d0, d2, d4
register float32x2_t _a asm("d0") = vld1_f32(a);
register float32x2_t _b asm("d2") = vld1_f32(b);
register float32x2_t _c asm("d4") = vld1_f32(c);

    : "0"(_a),
    :
);
```
```c
// q寄存器声明绑定
// bind q reg with data
// vmla.f32  q0, q1, q2
register float32x4_t _a asm("q0") = vld1q_f32(a);
register float32x4_t _b asm("q1") = vld1q_f32(b);
register float32x4_t _c asm("q2") = vld1q_f32(c);

    : "0"(_a),
    :
);
```

如果不是因为编译器的bug，寄存器绑定是用不着的，然而。。。

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=41538

qwq

# Top-down heatmap-based pose estimation

Top-down methods divide the task into two stages: object detection, followed by single-object pose estimation given object bounding boxes. Instead of estimating keypoint coordinates directly, the pose estimator will produce heatmaps which represent the likelihood of being a keypoint, following the paradigm introduced in [Simple Baselines for Human Pose Estimation and Tracking](http://openaccess.thecvf.com/content_ECCV_2018/html/Bin_Xiao_Simple_Baselines_for_ECCV_2018_paper.html).

<img src="https://user-images.githubusercontent.com/15977946/146522977-5f355832-e9c1-442f-a34f-9d24fb0aefa8.png" height=400>
</div>

### COCO-WholeBody-Hand Dataset

|      Model       | Input Size | PCK@0.2 |  AUC  | EPE  |                                       Details and Download                                       |

| :--------------: | :--------: | :-----: | :---: | :--: | :----------------------------------------------------------------------------------------------: |
| HRNetv2-w18+Dark |  256x256   |  0.814  | 0.840 | 4.37 | [hrnetv2_dark_coco_wholebody_hand.md](./coco_wholebody_hand/hrnetv2_dark_coco_wholebody_hand.md) |
|   HRNetv2-w18    |  256x256   |  0.813  | 0.840 | 4.39 |      [hrnetv2_coco_wholebody_hand.md](./coco_wholebody_hand/hrnetv2_coco_wholebody_hand.md)      |
|   HourglassNet   |  256x256   |  0.804  | 0.835 | 4.54 |    [hourglass_coco_wholebody_hand.md](./coco_wholebody_hand/hourglass_coco_wholebody_hand.md)    |
|     SCNet-50     |  256x256   |  0.803  | 0.834 | 4.55 |        [scnet_coco_wholebody_hand.md](./coco_wholebody_hand/scnet_coco_wholebody_hand.md)        |

|    ResNet-50     |  256x256   |  0.800  | 0.833 | 4.64 |       [resnet_coco_wholebody_hand.md](./coco_wholebody_hand/resnet_coco_wholebody_hand.md)       |
|   LiteHRNet-18   |  256x256   |  0.795  | 0.830 | 4.77 |    [litehrnet_coco_wholebody_hand.md](./coco_wholebody_hand/litehrnet_coco_wholebody_hand.md)    |
|   MobileNet-v2   |  256x256   |  0.795  | 0.829 | 4.77 |  [mobilenetv2_coco_wholebody_hand.md](./coco_wholebody_hand/mobilenetv2_coco_wholebody_hand.md)  |

### FreiHand Dataset

Results on FreiHand val & test set

|   Model   | Input Size | PCK@0.2 |  AUC  | EPE  |                   Details and Download                    |
| :-------: | :--------: | :-----: | :---: | :--: | :-------------------------------------------------------: |

| ResNet-50 |  224x224   |  0.999  | 0.868 | 3.27 | [resnet_freihand2d.md](./freihand2d/resnet_freihand2d.md) |

### OneHand10K Dataset

Results on OneHand10K val set

|      Model       | Input Size | PCK@0.2 |  AUC  |  EPE  |                         Details and Download                          |
| :--------------: | :--------: | :-----: | :---: | :---: | :-------------------------------------------------------------------: |
| HRNetv2-w18+Dark |  256x256   |  0.990  | 0.572 | 23.96 | [hrnetv2_dark_onehand10k.md](./onehand10k/hrnetv2_dark_onehand10k.md) |
| HRNetv2-w18+UDP  |  256x256   |  0.990  | 0.571 | 23.88 |  [hrnetv2_udp_onehand10k.md](./onehand10k/hrnetv2_udp_onehand10k.md)  |
|   HRNetv2-w18    |  256x256   |  0.990  | 0.567 | 24.26 |      [hrnetv2_onehand10k.md](./onehand10k/hrnetv2_onehand10k.md)      |

|    ResNet-50     |  256x256   |  0.989  | 0.555 | 25.16 |       [resnet_onehand10k.md](./onehand10k/resnet_onehand10k.md)       |
|   MobileNet-v2   |  256x256   |  0.986  | 0.537 | 28.56 |  [mobilenetv2_onehand10k.md](./onehand10k/mobilenetv2_onehand10k.md)  |

### RHD Dataset

Results on RHD test set

|      Model       | Input Size | PCK@0.2 |  AUC  | EPE  |                  Details and Download                  |
| :--------------: | :--------: | :-----: | :---: | :--: | :----------------------------------------------------: |
| HRNetv2-w18+Dark |  256x256   |  0.992  | 0.903 | 2.18 | [hrnetv2_dark_rhd2d.md](./rhd2d/hrnetv2_dark_rhd2d.md) |
| HRNetv2-w18+UDP  |  256x256   |  0.992  | 0.902 | 2.19 |  [hrnetv2_udp_rhd2d.md](./rhd2d/hrnetv2_udp_rhd2d.md)  |

|   HRNetv2-w18    |  256x256   |  0.992  | 0.902 | 2.21 |      [hrnetv2_rhd2d.md](./rhd2d/hrnetv2_rhd2d.md)      |
|    ResNet-50     |  256x256   |  0.991  | 0.898 | 2.32 |       [resnet_rhd2d.md](./rhd2d/resnet_rhd2d.md)       |
|   MobileNet-v2   |  256x256   |  0.985  | 0.883 | 2.79 |  [mobilenetv2_rhd2d.md](./rhd2d/mobilenetv2_rhd2d.md)  |

# Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation

> [Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation](https://arxiv.org/abs/2011.10033)

State-of-the-art methods for large-scale driving-scene LiDAR segmentation often project the point clouds to 2D space and then process them via 2D convolution. Although this corporation shows the competitiveness in the point cloud, it inevitably alters and abandons the 3D topology and geometric relations. A natural remedy is to utilize the3D voxelization and 3D convolution network. However, we found that in the outdoor point cloud, the improvement obtained in this way is quite limited. An important reason is the property of the outdoor point cloud, namely sparsity and varying density. Motivated by this investigation, we propose a new framework for the outdoor LiDAR segmentation, where cylindrical partition and asymmetrical 3D convolution networks are designed to explore the 3D geometric pat-tern while maintaining these inherent properties. Moreover, a point-wise refinement module is introduced to alleviate the interference of lossy voxel-based label encoding. We evaluate the proposed model on two large-scale datasets, i.e., SemanticKITTI and nuScenes. Our method achieves the 1st place in the leaderboard of SemanticKITTI and outperforms existing methods on nuScenes with a noticeable margin, about 4%. Furthermore, the proposed 3D framework also generalizes well to LiDAR panoptic segmentation and LiDAR 3D detection.

![overview](https://user-images.githubusercontent.com/45515569/228523861-2923082c-37d9-4d4f-aa59-746a8d9284c2.png)

We implement Cylinder3D and provide the result and checkpoints on Semantickitti datasets.

### SemanticKITTI

|                               Method                                | Lr schd | Laser-Polar Mix | Mem (GB) |   mIoU   |                                                                                                                                                                       Download                                                                                                                                                                       |

| :-----------------------------------------------------------------: | :-----: | :-------------: | :------: | :------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| [Cylinder3D](./cylinder3d_8xb2-laser-polar-mix-3x_semantickitti.py) |   3x    |        ✗        |   10.2   | 63.1±0.5 | [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/cylinder3d/cylinder3d_4xb4_3x_semantickitti/cylinder3d_4xb4_3x_semantickitti_20230318_191107-822a8c31.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/cylinder3d/cylinder3d_4xb4_3x_semantickitti/cylinder3d_4xb4_3x_semantickitti_20230318_191107.json) |

| [Cylinder3D](./cylinder3d_8xb2-laser-polar-mix-3x_semantickitti.py) |   3x    |        ✔        |   12.8   |   67.0   |              [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/cylinder3d/cylinder3d_8xb2-amp-laser-polar-mix-3x_semantickitti_20230425_144950-372cdf69.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/cylinder3d/cylinder3d_8xb2-amp-laser-polar-mix-3x_semantickitti_20230425_144950.log)               |

Note: We reproduce the performance comparable with its [official repo](https://github.com/xinge008/Cylinder3D). It's slightly lower than the performance (65.9 mIOU) reported in the paper due to the lack of point-wise refinement and shorter training time.

```latex
@inproceedings{zhu2021cylindrical,
  title={Cylindrical and asymmetrical 3d convolution networks for lidar segmentation},
  author={Zhu, Xinge and Zhou, Hui and Wang, Tai and Hong, Fangzhou and Ma, Yuexin and Li, Wei and Li, Hongsheng and Lin, Dahua},

author={Zhu, Xinge and Zhou, Hui and Wang, Tai and Hong, Fangzhou and Ma, Yuexin and Li, Wei and Li, Hongsheng and Lin, Dahua},
  pages={9939--9948},
}
```

# Dataset Prepare

### Basic Detection Dataset Preparation

MMDetection supports multiple public datasets including COCO, Pascal VOC, CityScapes, and [more](../../../configs/_base_/datasets).

Public datasets like [Pascal VOC](http://host.robots.ox.ac.uk/pascal/VOC/index.html) or mirror and [COCO](https://cocodataset.org/#download) are available from official websites or mirrors. Note: In the detection task, Pascal VOC 2012 is an extension of Pascal VOC 2007 without overlap, and we usually use them together.
It is recommended to download and extract the dataset somewhere outside the project directory and symlink the dataset root to `$MMDETECTION/data` as below.

We provide a script to download datasets such as COCO, you can run `python tools/misc/download_dataset.py --dataset-name coco2017` to download COCO dataset.

For users in China, more datasets can be downloaded from the opensource dataset platform: [OpenDataLab](https://opendatalab.com/?source=OpenMMLab%20GitHub).

For more usage please refer to [dataset-download](./useful_tools.md#dataset-download)

```text
mmdetection
├── mmdet
├── tools
├── data
│   ├── coco
│   │   ├── annotations
│   │   ├── train2017
│   │   ├── val2017
│   │   ├── test2017
│   ├── cityscapes
│   │   ├── leftImg8bit
│   │   │   ├── train
│   │   │   ├── val
│   │   ├── gtFine
│   │   │   ├── val
│   │   ├── VOC2007
```

│   │   │   ├── val
│   │   │   ├── val
```

Some models require additional [COCO-stuff](http://calvin.inf.ed.ac.uk/wp-content/uploads/data/cocostuffdataset/stuffthingmaps_trainval2017.zip) datasets, such as HTC, DetectoRS and SCNet, you can download, unzip, and then move them to the coco folder. The directory should be like this.

```text
├── data
│   ├── coco
│   │   ├── stuffthingmaps
```

Panoptic segmentation models like PanopticFPN require additional [COCO Panoptic](http://images.cocodataset.org/annotations/panoptic_annotations_trainval2017.zip) datasets, you can download, unzip, and then move them to the coco annotation folder. The directory should be like this.

```text
├── data
│   ├── coco

```text
├── data
│   ├── coco
│   │   │   ├── panoptic_train2017.json
│   │   │   ├── panoptic_train2017
│   │   │   ├── panoptic_val2017.json
│   │   │   ├── panoptic_val2017
```

The [cityscapes](https://www.cityscapes-dataset.com/) annotations need to be converted into the coco format using `tools/dataset_converters/cityscapes.py`:

```shell
pip install cityscapesscripts

python tools/dataset_converters/cityscapes.py \
    ./data/cityscapes \
    --nproc 8 \
    --out-dir ./data/cityscapes/annotations
```

### COCO Caption Dataset Preparation

COCO Caption uses the COCO2014 dataset image and uses the annotation of karpathy.

At first, you need to download the COCO2014 dataset.

```shell
python tools/misc/download_dataset.py --dataset-name coco2014 --unzip
```

The dataset will be downloaded to `data/coco` under the current path. Then download the annotation of karpathy.

```shell

```

```shell
cd data/coco/annotations
wget https://storage.googleapis.com/sfr-vision-language-research/datasets/coco_karpathy_train.json
wget https://storage.googleapis.com/sfr-vision-language-research/datasets/coco_karpathy_val.json
wget https://storage.googleapis.com/sfr-vision-language-research/datasets/coco_karpathy_test.json
wget https://storage.googleapis.com/sfr-vision-language-research/datasets/coco_karpathy_val_gt.json
wget https://storage.googleapis.com/sfr-vision-language-research/datasets/coco_karpathy_test_gt.json
```

The final directory structure of the dataset folder that can be directly used for training and testing is as follows:

```text
├── data
│   ├── coco
│   │   │   ├── coco_karpathy_train.json
│   │   │   ├── coco_karpathy_test.json

```text
├── data
│   ├── coco
│   │   │   ├── coco_karpathy_val.json
│   │   │   ├── coco_karpathy_val_gt.json
│   │   │   ├── coco_karpathy_test_gt.json
│   │   ├── train2014
│   │   ├── val2014
│   │   ├── test2014
```

### COCO Semantic Dataset Preparation

There are two types of annotations for COCO semantic segmentation, which differ mainly in the definition of category names, so there are two ways to handle them. The first is to directly use the stuffthingmaps dataset, and the second is to use the panoptic dataset.

**(1) Use stuffthingmaps dataset**

The download link for this dataset is [stuffthingmaps_trainval2017](http://calvin.inf.ed.ac.uk/wp-content/uploads/data/cocostuffdataset/stuffthingmaps_trainval2017.zip). Please download and extract it to the `data/coco` folder.

```text
├── data
│   ├── coco

```text
├── data
│   ├── coco
```

This dataset is different from the standard COCO category annotation in that it includes 172 classes: 80 "thing" classes, 91 "stuff" classes, and 1 "unlabeled" class. The description of each class can be found at https://github.com/nightrome/cocostuff/blob/master/labels.md.

Although only 172 categories are annotated, the maximum label ID in `stuffthingmaps` is 182, and some categories in the middle are not annotated. In addition, the "unlabeled" category of class 0 is removed. Therefore, the relationship between the value at each position in the final `stuffthingmaps` image can be found at https://github.com/kazuto1011/deeplab-pytorch/blob/master/data/datasets/cocostuff/labels.txt.

To train efficiently and conveniently for users, we need to remove 12 unannotated classes before starting training or evaluation. The names of these 12 classes are: `street sign, hat, shoe, eye glasses, plate, mirror, window, desk, door, blender, hair brush`. The category information that can be used for training and evaluation can be found in `mmdet/datasets/coco_semantic.py`.

You can use `tools/dataset_converters/coco_stuff164k.py` to convert the downloaded `stuffthingmaps` to a dataset that can be directly used for training and evaluation. The directory structure of the converted dataset is as follows:

```text
├── data
│   ├── coco
│   │   ├── stuffthingmaps_semseg
```

`stuffthingmaps_semseg` is the newly generated COCO semantic segmentation dataset that can be directly used for training and testing.

**(2) use panoptic dataset**

```

The number of categories in the semantic segmentation dataset generated through panoptic annotation will be less than that generated using the `stuffthingmaps` dataset. First, you need to prepare the panoptic segmentation annotations, and then use the following script to complete the conversion.

```shell
python tools/dataset_converters/prepare_coco_semantic_annos_from_panoptic_annos.py data/coco
```

The directory structure of the converted dataset is as follows:

```text
├── data
│   ├── coco
│   │   │   ├── panoptic_semseg_train2017
│   │   │   ├── panoptic_semseg_val2017
```

```

`panoptic_semseg_train2017` and `panoptic_semseg_val2017` are the newly generated COCO semantic segmentation datasets that can be directly used for training and testing. Note that their category information is the same as that of COCO panoptic segmentation, including both "thing" and "stuff" categories.

### RefCOCO Dataset Preparation

The images and annotations of [RefCOCO](https://github.com/lichengunc/refer) series datasets can be download by running `tools/misc/download_dataset.py`:

```shell
python tools/misc/download_dataset.py --dataset-name refcoco --save-dir data/coco --unzip
```

Then the directory should be like this:

```text
data
├── coco
│   ├── refcoco
│   │   ├── instances.json
│   │   ├── refs(google).p
│   │   └── refs(unc).p
│   ├── refcoco+

```text
data
├── coco
│   ├── refcocog
│   │   └── refs(umd).p
|   |── train2014
```

### ADE20K 2016 Dataset Preparation

The images and annotations of [ADE20K](https://groups.csail.mit.edu/vision/datasets/ADE20K/) dataset can be download by running `tools/misc/download_dataset.py`:

```shell
python tools/misc/download_dataset.py --dataset-name ade20k_2016 --save-dir data --unzip
```

Then move the annotations to the `data/ADEChallengeData2016` directory and run the preprocess script to produce the coco format annotations:

```shell
mv data/annotations_instance data/ADEChallengeData2016/
mv data/categoryMapping.txt data/ADEChallengeData2016/
mv data/imgCatIds.json data/ADEChallengeData2016/

```shell
python tools/dataset_converters/ade20k2coco.py data/ADEChallengeData2016 --task panoptic
python tools/dataset_converters/ade20k2coco.py data/ADEChallengeData2016 --task instance
```

The directory should be like this.

```text
data
├── ADEChallengeData2016
│   ├── ade20k_instance_train.json
│   ├── ade20k_instance_val.json
│   ├── ade20k_panoptic_train
|   |   ├── ADE_train_00000001.png
|   |   ├── ADE_train_00000002.png
|   |   ├── ...
│   ├── ade20k_panoptic_train.json
│   ├── ade20k_panoptic_val
|   |   ├── ADE_val_00000001.png
|   |   ├── ADE_val_00000002.png
|   |   ├── ...
│   ├── ade20k_panoptic_val.json
│   ├── annotations
|   |   ├── training
|   |   |   ├── ADE_train_00000001.png
|   |   |   ├── ADE_train_00000002.png
|   |   |   ├── ...
|   |   ├── validation
|   |   |   ├── ADE_val_00000001.png

|   |   |   ├── ...
|   |   |   ├── ADE_val_00000002.png
|   |   |   ├── ...
│   ├── annotations_instance
|   |   |   ├── ...
|   |   |   ├── ...
│   ├── categoryMapping.txt
│   ├── images
│   |   ├── training
|   |   |   ├── ADE_train_00000001.jpg
|   |   |   ├── ADE_train_00000002.jpg
|   |   |   ├── ...
|   |   |   ├── ADE_val_00000001.jpg
|   |   |   ├── ADE_val_00000002.jpg
|   |   |   ├── ...
│   ├── imgCatIds.json
│   ├── objectInfo150.txt
|   |── sceneCategories.txt
```

|   |   |   ├── ...
```

The above folders include all data of ADE20K's semantic segmentation, instance segmentation, and panoptic segmentation.

### Download from OpenDataLab

By using [OpenDataLab](https://opendatalab.com/), researchers can obtain free formatted datasets in various fields. Through the search function of the platform, researchers may address the dataset they look for quickly and easily. Using the formatted datasets from the platform, researchers can efficiently conduct tasks across datasets.

Currently, MIM supports downloading VOC and COCO datasets from OpenDataLab with one command line. More datasets will be supported in the future. You can also directly download the datasets you need from the OpenDataLab platform and then convert them to the format required by MMDetection.

If you use MIM to download, make sure that the version is greater than v0.3.8. You can use the following command to update:

```Bash
```

```Bash
odl login

# download voc2007 and preprocess by MIM
mim download mmdet --dataset voc2007

# download voc2012 and preprocess by MIM
mim download mmdet --dataset voc2012

mim download mmdet --dataset coco2017
```

# 测试现有模型

我们提供了测试脚本，能够测试一个现有模型在所有数据集（COCO，Pascal VOC，Cityscapes 等）上的性能。我们支持在如下环境下测试：

- 单 GPU 测试
- CPU 测试
- 单节点多 GPU 测试
- 多节点测试

根据以上测试环境，选择合适的脚本来执行测试过程。

```shell
# 单 GPU 测试
    [--out ${RESULT_FILE}] \
    [--show]

# CPU 测试：禁用 GPU 并运行单 GPU 测试脚本
export CUDA_VISIBLE_DEVICES=-1
    [--show]

# 单节点多 GPU 测试
    ${GPU_NUM} \
    [--out ${RESULT_FILE}]
```

`tools/dist_test.sh` 也支持多节点测试，不过需要依赖 PyTorch 的 [启动工具](https://pytorch.org/docs/stable/distributed.html#launch-utility) 。

可选参数：

- `RESULT_FILE`: 结果文件名称，需以 .pkl 形式存储。如果没有声明，则不将结果存储到文件。
- `--show`: 如果开启，检测结果将被绘制在图像上，以一个新窗口的形式展示。它只适用于单 GPU 的测试，是用于调试和可视化的。请确保使用此功能时，你的 GUI 可以在环境中打开。否则，你可能会遇到这么一个错误 `cannot connect to X server`。

- `--show-dir`: 如果指明，检测结果将会被绘制在图像上并保存到指定目录。它只适用于单 GPU 的测试，是用于调试和可视化的。即使你的环境中没有 GUI，这个选项也可使用。

### 样例

假设你已经下载了 checkpoint 文件到 `checkpoints/` 文件下了。

1. 测试 RTMDet 并可视化其结果。按任意键继续下张图片的测试。配置文件和 checkpoint 文件 [在此](https://github.com/open-mmlab/mmdetection/tree/main/configs/rtmdet) 。

   ```shell
   python tools/test.py \
       configs/rtmdet/rtmdet_l_8xb32-300e_coco.py \
       checkpoints/rtmdet_l_8xb32-300e_coco_20220719_112030-5a0be7c4.pth \
       --show
   ```

2. 测试 RTMDet，并为了之后的可视化保存绘制的图像。配置文件和 checkpoint 文件 [在此](https://github.com/open-mmlab/mmdetection/tree/main/configs/rtmdet) 。

   ```shell

```shell
       --show-dir rtmdet_l_8xb32-300e_coco_results
   ```

3. 在 Pascal VOC 数据集上测试 Faster R-CNN，不保存测试结果，测试 `mAP`。配置文件和 checkpoint 文件 [在此](../../../configs/pascal_voc) 。

   ```shell
       configs/pascal_voc/faster-rcnn_r50_fpn_1x_voc0712.py \
       checkpoints/faster_rcnn_r50_fpn_1x_voc0712_20200624-c9895d40.pth
   ```

4. 使用 8 块 GPU 测试 Mask R-CNN，测试 `bbox` 和 `mAP` 。配置文件和 checkpoint 文件 [在此](../../../configs/mask_rcnn) 。

   ```shell
   ./tools/dist_test.sh \
       configs/mask-rcnn_r50_fpn_1x_coco.py \
       checkpoints/mask_rcnn_r50_fpn_1x_coco_20200205-d4b0c5d6.pth \
       8 \
       --out results.pkl
   ```

5. 使用 8 块 GPU 测试 Mask R-CNN，测试**每类**的 `bbox` 和 `mAP`。配置文件和 checkpoint 文件 [在此](../../../configs/mask_rcnn) 。

   ```shell

8 \
   ```

   ```shell
       8
   ```

   该命令生成两个JSON文件 `./work_dirs/coco_instance/test.bbox.json` 和 `./work_dirs/coco_instance/test.segm.json`。

6. 在 COCO test-dev 数据集上，使用 8 块 GPU 测试 Mask R-CNN，并生成 JSON 文件提交到官方评测服务器，配置文件和 checkpoint 文件 [在此](../../../configs/mask_rcnnn) 。你可以在 [config](./././configs/_base_/datasets/coco_instance.py) 的注释中用 test_evaluator 和 test_dataloader 替换原来的 test_evaluator 和 test_dataloader，然后运行：

   ```shell
       configs/cityscapes/mask-rcnn_r50_fpn_1x_cityscapes.py \
       checkpoints/mask_rcnn_r50_fpn_1x_cityscapes_20200227-afe51d5a.pth \
       8
   ```

```shell
       8
   ```

   这行命令生成两个 JSON 文件 `mask_rcnn_test-dev_results.bbox.json` 和 `mask_rcnn_test-dev_results.segm.json`。

7. 在 Cityscapes 数据集上，使用 8 块 GPU 测试 Mask R-CNN，生成 txt 和 png 文件，并上传到官方评测服务器。配置文件和 checkpoint 文件 [在此](../../../configs/cityscapes) 。 你可以在 [config](./././configs/_base_/datasets/cityscapes_instance.py) 的注释中用 test_evaluator 和 test_dataloader 替换原来的 test_evaluator 和 test_dataloader，然后运行：

   ```shell
       8
   ```

   生成的 png 和 txt 文件在 `./work_dirs/cityscapes_metric` 文件夹下。

### 不使用 Ground Truth 标注进行测试

checkpoints/mask_rcnn_r50_fpn_1x_cityscapes_20200227-afe51d5a.pth \
       8
   ```

MMDetection 支持在不使用 ground-truth 标注的情况下对模型进行测试，这需要用到 `CocoDataset`。如果你的数据集格式不是 COCO 格式的，请将其转化成 COCO 格式。如果你的数据集格式是 VOC 或者 Cityscapes，你可以使用 [tools/dataset_converters](https://github.com/open-mmlab/mmdetection/tree/main/tools/dataset_converters) 内的脚本直接将其转化成 COCO 格式。如果是其他格式，可以使用 [images2coco 脚本](https://github.com/open-mmlab/mmdetection/tree/master/tools/dataset_converters/images2coco.py) 进行转换。

```shell
python tools/dataset_converters/images2coco.py \
    ${IMG_PATH} \
    ${CLASSES} \
    ${OUT} \
    [--exclude-extensions]
```

参数：

- `IMG_PATH`: 图片根路径。
- `CLASSES`: 类列表文本文件名。文本中每一行存储一个类别。
- `OUT`: 输出 json 文件名。 默认保存目录和 `IMG_PATH` 在同一级。
- `exclude-extensions`: 待排除的文件后缀名。

在转换完成后，使用如下命令进行测试

```shell
# 单 GPU 测试
    [--show]

```shell
# 单 GPU 测试
    [--show]

    [--show]

    [--show]
```

假设 [model zoo](https://mmdetection.readthedocs.io/en/latest/modelzoo_statistics.html) 中的 checkpoint 文件被下载到了 `checkpoints/` 文件夹下，
我们可以使用以下命令，用 8 块 GPU 在 COCO test-dev 数据集上测试 Mask R-CNN，并且生成 JSON 文件。

```sh
./tools/dist_test.sh \
    configs/mask_rcnn/mask-rcnn_r50_fpn_1x_coco.py \
    checkpoints/mask_rcnn_r50_fpn_1x_coco_20200205-d4b0c5d6.pth \
    8
```

这行命令生成两个 JSON 文件 `./work_dirs/coco_instance/test.bbox.jso` 和 `./work_dirs/coco_instance/test.segm.jsonn`。

### 批量推理

checkpoints/mask_rcnn_r50_fpn_1x_coco_20200205-d4b0c5d6.pth \
    8
```

### 批量推理

MMDetection 在测试模式下，既支持单张图片的推理，也支持对图像进行批量推理。默认情况下，我们使用单张图片的测试，你可以通过修改测试数据配置文件中的 `samples_per_gpu` 来开启批量测试。
开启批量推理的配置文件修改方法为：

```shell
data = dict(train_dataloader=dict(...), val_dataloader=dict(...), test_dataloader=dict(batch_size=2, ...))
```

或者你可以通过将 `--cfg-options` 设置为 `--cfg-options test_dataloader.batch_size=` 来开启它。

## 测试时增强 (TTA)

测试时增强 (TTA) 是一种在测试阶段使用的数据增强策略。它对同一张图片应用不同的增强，例如翻转和缩放，用于模型推理，然后将每个增强后的图像的预测结果合并，以获得更准确的预测结果。为了让用户更容易使用 TTA，MMEngine 提供了 [BaseTTAModel](https://mmengine.readthedocs.io/en/latest/api/generated/mmengine.model.BaseTTAModel.html#mmengine.model.BaseTTAModel) 类，允许用户根据自己的需求通过简单地扩展 BaseTTAModel 类来实现不同的 TTA 策略。

在 MMDetection 中，我们提供了 [DetTTAModel](../../../mmdet/models/test_time_augs/det_tta.py) 类，它继承自 BaseTTAModel。

### 使用案例

### 使用案例

使用 TTA 需要两个步骤。首先，你需要在配置文件中添加 `tta_model` 和 `tta_pipeline`：

```shell
tta_model = dict(
    type='DetTTAModel',
    tta_cfg=dict(nms=dict(
                   type='nms',
                   iou_threshold=0.5),
                   max_per_img=100))

tta_pipeline = [
    dict(type='LoadImageFromFile',
        backend_args=None),
    dict(
        type='TestTimeAug',
        transforms=[[
            dict(type='Resize', scale=(1333, 800), keep_ratio=True)
        ], [ # It uses 2 flipping transformations (flipping and not flipping).
            dict(type='RandomFlip', prob=1.),
            dict(type='RandomFlip', prob=0.)
        ], [
            dict(
               type='PackDetInputs',
               meta_keys=('img_id', 'img_path', 'ori_shape',
                       'img_shape', 'scale_factor', 'flip',
                       'flip_direction'))
       ]])]
```

meta_keys=('img_id', 'img_path', 'ori_shape',
       ]])]
```

第二步，运行测试脚本时，设置 `--tta` 参数，如下所示：

```shell
# 单 GPU 测试
    [--tta]

    [--tta]

# 多 GPU 测试
    [--tta]
```

你也可以自己修改 TTA 配置，例如添加缩放增强：

```shell

img_scales = [(1333, 800), (666, 400), (2000, 1200)]
         backend_args=None),
    dict(

    dict(
            dict(type='Resize', scale=s, keep_ratio=True) for s in img_scales
        ], [
        ], [
            dict(
       ]])]
```

以上数据增强管道将首先对图像执行 3 个多尺度转换，然后执行 2 个翻转转换（翻转和不翻转），最后使用 PackDetInputs 将图像打包到最终结果中。
这里有更多的 TTA 使用案例供您参考：

- [RetinaNet](../../../configs/retinanet/retinanet_tta.py)
- [CenterNet](../../../configs/centernet/centernet_tta.py)
- [YOLOX](../../../configs/rtmdet/rtmdet_tta.py)
- [RTMDet](../../../configs/yolox/yolox_tta.py)

更多高级用法和 TTA 的数据流，请参考 [MMEngine](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/test_time_augmentation.html#data-flow)。我们将在后续支持实例分割 TTA。

# Test a model (To be updated)

- single GPU
- single node multiple GPU
- multiple node

You can use the following commands to infer a dataset.

```shell
# single-gpu

# multi-gpu
./tools/dist_test.sh ${CONFIG_FILE} ${CHECKPOINT_FILE} ${GPU_NUM} [optional arguments]

# multi-node in slurm environment
python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [optional arguments] --launcher slurm
```

Examples:

Inference RotatedRetinaNet on DOTA-1.0 dataset, which can generate compressed files for online [submission](https://captain-whu.github.io/DOTA/evaluation.html). (Please change the [data_root](https://github.com/open-mmlab/mmrotate/tree/main/configs/_base_/datasets/dotav1.py) firstly.)

```shell
python ./tools/test.py  \
  configs/rotated_retinanet/rotated-retinanet-rbox-le90_r50_fpn_1x_dota.py \
  checkpoints/SOME_CHECKPOINT.pth --format-only \
  --eval-options submission_dir=work_dirs/Task1_results
```

or

configs/rotated_retinanet/rotated-retinanet-rbox-le90_r50_fpn_1x_dota.py \
```

or

```shell
./tools/dist_test.sh  \
  checkpoints/SOME_CHECKPOINT.pth 1 --format-only \
```

You can change the test set path in the [data_root](https://github.com/open-mmlab/mmrotate/tree/main/configs/_base_/datasets/dotav1.py) to the val set or trainval set for the offline evaluation.

```shell
python ./tools/test.py \
  checkpoints/SOME_CHECKPOINT.pth --eval mAP
```

or

```shell
  checkpoints/SOME_CHECKPOINT.pth 1 --eval mAP
```

You can also visualize the results.

```shell

```

```shell
  checkpoints/SOME_CHECKPOINT.pth \
  --show-dir work_dirs/vis
```

# Train a model

## Train with a single GPU

```shell
```

If you want to specify the working directory in the command, you can add an argument `--work_dir ${YOUR_WORK_DIR}`.

## Train with multiple GPUs

```shell
```

Optional arguments are:

- `--no-validate` (**not suggested**): By default, the codebase will perform evaluation during the training. To disable this behavior, use `--no-validate`.
- `--work-dir ${WORK_DIR}`: Override the working directory specified in the config file.

- `--resume-from ${CHECKPOINT_FILE}`: Resume from a previous checkpoint file.

Difference between `resume-from` and `load-from`:
`resume-from` loads both the model weights and optimizer status, and the epoch is also inherited from the specified checkpoint. It is usually used for resuming the training process that is interrupted accidentally.
`load-from` only loads the model weights and the training epoch starts from 0. It is usually used for finetuning.

## Train with multiple machines

If you launch with multiple machines simply connected with ethernet, you can simply run following commands:

```shell
NNODES=2 NODE_RANK=0 PORT=$MASTER_PORT MASTER_ADDR=$MASTER_ADDR sh tools/dist_train.sh $CONFIG $GPUS
```

```shell
NNODES=2 NODE_RANK=1 PORT=$MASTER_PORT MASTER_ADDR=$MASTER_ADDR sh tools/dist_train.sh $CONFIG $GPUS
```

```

```shell
```

Usually it is slow if you do not have high speed networking like InfiniBand.

## Manage jobs with Slurm

If you run MMRotate on a cluster managed with [slurm](https://slurm.schedmd.com/), you can use the script `slurm_train.sh`. (This script also supports single machine training.)

```shell
```

If you have just multiple machines connected with ethernet, you can refer to
PyTorch [launch utility](https://pytorch.org/docs/stable/distributed_deprecated.html#launch-utility).

## Launch multiple jobs on a single machine

If you launch multiple jobs on a single machine, e.g., 2 jobs of 4-GPU training on a machine with 8 GPUs,

you need to specify different ports (29500 by default) for each job to avoid communication conflict.

If you use `dist_train.sh` to launch training jobs, you can set the port in commands.

```shell
```

If you use launch training jobs with Slurm, you need to modify the config files (usually the 6th line from the bottom in config files) to set different communication ports.

In `config1.py`,

```python
```

In `config2.py`,

```python
```

Then you can launch two jobs with `config1.py` and `config2.py`.

```shell

```

```python
```

```shell
```

# MMDetection3d Deployment

- [MMDetection3d Deployment](#mmdetection3d-deployment)
  - [Install mmdet3d](#install-mmdet3d)
  - [Convert model](#convert-model)
  - [Model inference](#model-inference)
  - [Supported models](#supported-models)

[MMDetection3d](https://github.com/open-mmlab/mmdetection3d) aka `mmdet3d` is an open source object detection toolbox based on PyTorch, towards the next-generation platform for general 3D detection. It is a part of the [OpenMMLab](https://openmmlab.com/) project.

## Install mmdet3d

These branches are required for mmdet3d deployment

| codebase |  commit   |
| :------: | :-------: |
| mmdet3d  | v1.1.0rc1 |
|   mmcv   | v2.0.0rc1 |
|  mmdet   | v3.0.0rc1 |
|  mmseg   | v1.0.0rc0 |

First checkout and install mmcv/mmdet/mmseg/mmdet3d

```bash
python3 -m pip install openmim --user
python3 -m mim install mmcv==2.0.0rc1 mmdet==3.0.0rc1 mmseg==1.0.0rc0 --user

```bash

git clone https://github.com/open-mmlab/mmdetection3d --branch v1.1.0rc1
cd mmdetection3d
python3 -m pip install .
cd -
```

After installation, `tools/check_env.py` should display mmdet3d version normally

```bash
python3 tools/check_env.py
..
11/11 13:56:19 - mmengine - INFO - **********Codebase information**********
11/11 13:56:19 - mmengine - INFO - mmdet:       3.0.0rc1
11/11 13:56:19 - mmengine - INFO - mmseg:       1.0.0rc0
..
11/11 13:56:19 - mmengine - INFO - mmdet3d:     1.1.0rc1
```

## Convert model

For example, use `tools/deploy.py` to convert centerpoint to onnxruntime format

```bash
export MODEL_CONFIG=/path/to/mmdetection3d/configs/centerpoint/centerpoint_pillar02_second_secfpn_head-circlenms_8xb4-cyclic-20e_nus-3d.py

```bash

export MODEL_PATH=https://download.openmmlab.com/mmdetection3d/v1.0.0_models/centerpoint/centerpoint_02pillar_second_secfpn_circlenms_4x8_cyclic_20e_nus/centerpoint_02pillar_second_secfpn_circlenms_4x8_cyclic_20e_nus_20210816_064624-0f3299c0.pth

export TEST_DATA=/path/to/mmdetection3d/tests/data/nuscenes/sweeps/LIDAR_TOP/n008-2018-09-18-12-07-26-0400__LIDAR_TOP__1537287083900561.pcd.bin

python3 tools/deploy.py configs/mmdet3d/voxel-detection/voxel-detection_onnxruntime_dynamic.py $MODEL_CONFIG $MODEL_PATH $TEST_DATA --work-dir centerpoint
```

This step would generate `end2end.onnx` in `work-dir`

```bash
ls -lah centerpoint
..
-rw-rw-r--  1 rg rg  87M 11月  4 19:48 end2end.onnx
```

## Model inference

```

```bash
..
```

At present, the voxelize preprocessing and postprocessing of mmdet3d are not converted into onnx operations; the C++ SDK has not yet implemented the voxelize calculation.

The caller needs to refer to the corresponding [python implementation](../../../mmdeploy/codebase/mmdet3d/deploy/voxel_detection_model.py) to complete.

## Supported models

|                                                                                model                                                                                 |        task         | dataset  | onnxruntime | openvino | tensorrt\* |
| :------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------: | :------: | :---------: | :------: | :--------: |

| [centerpoint](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/centerpoint/centerpoint_pillar02_second_secfpn_head-circlenms_8xb4-cyclic-20e_nus-3d.py) |   voxel detection   | nuScenes |     ✔️      |    ✔️    |     ✔️     |
|             [pointpillars](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/pointpillars/pointpillars_hv_secfpn_sbn-all_8xb4-2x_nus-3d.py)              |   voxel detection   | nuScenes |     ✔️      |    ✔️    |     ✔️     |
|            [pointpillars](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py)            |   voxel detection   |  KITTI   |     ✔️      |    ✔️    |     ✔️     |
|                   [smoke](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/smoke/smoke_dla34_dlaneck_gn-all_4xb8-6x_kitti-mono3d.py)                    | monocular detection |  KITTI   |     ✔️      |    x     |     ✔️     |

- Make sure trt >= 8.4 for some bug fixed, such as ScatterND, dynamic shape crash and so on.

# 教程4：使用现有模型进行训练和测试

MMSegmentation 支持在多种设备上训练和测试模型。如下文，具体方式分别为单GPU、分布式以及计算集群的训练和测试。通过本教程，您将知晓如何用 MMSegmentation 提供的脚本进行训练和测试。

## 在单GPU上训练和测试

### 在单GPU上训练

`tools/train.py` 文件提供了在单GPU上部署训练任务的方法。

基础用法如下:

```shell
python tools/train.py  ${配置文件} [可选参数]
```

- `--work-dir ${工作路径}`: 重新指定工作路径
- `--amp`: 使用自动混合精度计算
- `--resume`: 从工作路径中保存的最新检查点文件（checkpoint）恢复训练
- `--cfg-options ${需更覆盖的配置}`: 覆盖已载入的配置中的部分设置，并且 以 xxx=yyy 格式的键值对 将被合并到配置文件中。
  比如： '--cfg-option model.encoder.in_channels=6'， 更多细节请看[指导](./1_config.md#Modify-config-through-script-arguments)。

下面是对于多GPU测试的可选参数:

- `--launcher`: 执行器的启动方式。允许选择的参数值有 `none`, `pytorch`, `slurm`, `mpi`。特别的，如果设置为none，测试将非分布式模式下进行。
- `--local_rank`: 分布式中进程的序号。如果没有指定，默认设置为0。

**注意：** 命令行参数 `--resume` 和在配置文件中的参数 `load_from` 的不同之处：

`--resume` 只决定是否继续使用工作路径中最新的检查点，它常常用于恢复被意外打断的训练。

`load_from` 会明确指定被载入的检查点文件，且训练迭代器将从0开始，通常用于微调模型。

如果您希望从指定的检查点上恢复训练您可以使用：

```python
python tools/train.py ${配置文件} --resume --cfg-options load_from=${检查点}
```

```python
```

**在 CPU 上训练**: 如果机器没有 GPU，则在 CPU 上训练的过程是与单GPU训练一致的。如果机器有 GPU 但是不希望使用它们，我们只需要在训练前通过以下方式关闭 GPU 训练功能。

```shell
```

然后运行[上方](###在单GPU上训练)脚本。

### 在单GPU上测试

`tools/test.py` 文件提供了在单 GPU 上启动测试任务的方法。

基础用法如下:

```shell
python tools/test.py ${配置文件} ${模型权重文件} [可选参数]
```

这个工具有几个可选参数，包括：

- `--work-dir`: 如果指定了路径，结果会保存在该路径下。如果没有指定则会保存在 `work_dirs/{配置文件名}` 路径下.
- `--show`: 当 `--show-dir` 没有指定时，可以使用该参数，在程序运行过程中显示预测结果。
- `--show-dir`: 绘制了分割掩膜图片的存储文件夹。如果指定了该参数，则可视化的分割掩膜将被保存到 `work_dir/timestamp/{指定路径}`.
- `--wait-time`: 多次可视化结果的时间间隔。当 `--show` 为激活状态时发挥作用。默认为2。
- `--cfg-options`:  如果被具体指定，以 xxx=yyy 形式的键值对将被合并入配置文件中。

**在CPU上测试**: 如果机器没有GPU，则在CPU上训练的过程是与单GPU训练一致的。如果机器有GPU，但是不希望使用它们，我们只需要在训练前通过以下方式关闭GPUs训练功能。

```shell
```

然后运行[上方](###在单GPU上测试)脚本。

## 多GPU、多机器上训练和测试

### 在多GPU上训练

```shell
```

OpenMMLab2.0 通过 `MMDistributedDataParallel`实现 **分布式** 训练。

`tools/dist_train.sh` 文件提供了在在多GPU上部署训练任务的方法。

基础用法如下:

```shell
sh tools/dist_train.sh ${配置文件} ${GPU数量} [可选参数]
```

可选参数与[上方](###在单GPU上训练)相同并且还增加了可以指定gpu数量的参数。

示例:

```shell
# 模型训练的检查点和日志保存在这个路径下： WORK_DIR=work_dirs/pspnet_r50-d8_4xb4-80k_ade20k-512x512/
# 如果工作路径没有被设定，它将会被自动生成。
sh tools/dist_train.sh configs/pspnet/pspnet_r50-d8_4xb4-80k_ade20k-512x512.py 8 --work-dir work_dirs/pspnet_r50-d8_4xb4-80k_ade20k-512x512
```

**注意**: 在训练过程中，检查点和日志保存在`work_dirs/`下的配置文件的相同文件夹结构下。
不推荐自定义的工作路径，因为评估脚本依赖于源自配置文件名的路径。如果您希望将权重保存在其他地方，请用符号链接，例如：

```shell
ln -s ${您的工作路径} ${MMSEG 路径}/work_dirs
```

### 在多GPU上测试

`tools/dist_test.sh` 文件提供了在多GPU上启动测试任务的方法。

基础用法如下:

```shell
sh tools/dist_test.sh ${配置文件} ${检查点文件} ${GPU数量} [可选参数]
```

```shell
```

基础用法如下:

```shell
```

可选参数与[上方](###在单GPU上测试)相同并且增加了可以指定 gpu 数量的参数。

示例:

```shell
./tools/dist_test.sh configs/pspnet/pspnet_r50-d8_4xb2-40k_cityscapes-512x1024.py \
    checkpoints/pspnet_r50-d8_512x1024_40k_cityscapes_20200605_003338-2966598c.pth 4
```

### 在单台机器上启动多个任务

如果您在单个机器上运行多个任务，比如：在8卡GPU的单个机器上执行2个各需4卡GPU的训练任务，您需要为每个任务具体指定不同端口（默认29500），从而避免通讯冲突。否则，会有报错信息——`RuntimeError: Address already in use`（运行错误：地址被使用）。

如果您使用 `dist_train.sh` 来启动训练任务，您可以通过环境变量 `PORT` 设置端口。

```shell
CUDA_VISIBLE_DEVICES=0,1,2,3 PORT=29500 sh tools/dist_train.sh ${配置文件} 4
CUDA_VISIBLE_DEVICES=4,5,6,7 PORT=29501 sh tools/dist_train.sh ${配置文件} 4
```

### 在多台机器上训练

MMSegmentation 的分布式训练依赖 `torch.distributed`。
因此， 可以通过 PyTorch 的 [运行工具 launch utility](https://pytorch.org/docs/stable/distributed.html#launch-utility) 来进行分布式训练。

```

如果您启动的多台机器简单地通过以太网连接，您可以直接运行下方命令：

在第一个机器上:

```shell
NNODES=2 NODE_RANK=0 PORT=${主节点端口} MASTER_ADDR=${主节点地址} sh tools/dist_train.sh ${配置文件} ${GPUS}
```

在第二个机器上:

```shell
NNODES=2 NODE_RANK=1 PORT=${主节点端口} MASTER_ADDR=${主节点地址} sh tools/dist_train.sh ${配置文件} ${GPUS}
```

通常，如果您没有使用像无限带宽一类的高速网络，这个会过程比较慢。

## 通过 Slurm 管理任务

[Slurm](https://slurm.schedmd.com/) 是一个很好的计算集群作业调度系统。

### 通过 Slurm 在集群上训练

在一个由Slurm管理的集群上，您可以使用`slurm_train.sh`来启动训练任务。它同时支持单节点和多节点的训练。

基础用法如下：

```shell
[GPUS=${GPUS}] sh tools/slurm_train.sh ${分区} ${任务名} ${配置文件} [可选参数]
```

下方是一个通过名为 `dev` 的 Slurm 分区，调用4个 GPU 来训练 PSPNet，并设置工作路径为共享文件系统。

```shell
GPUS=4 sh tools/slurm_train.sh dev pspnet configs/pspnet/pspnet_r50-d8_512x1024_40k_cityscapes.py --work-dir work_dir/pspnet
```

您可以检查 [源码](../../../tools/slurm_train.sh) 来查看全部的参数和环境变量。

```shell
```

### 通过 Slurm 在集群上测试

与训练任务相同， MMSegmentation 提供 `slurm_test.sh` 文件来启动测试任务。

基础用法如下：

```shell
[GPUS=${GPUS}] sh tools/slurm_test.sh ${分区} ${任务名} ${配置文件} ${检查点文件} [可选参数]
```

您可以通过 [源码](../../../tools/slurm_test.sh) 来查看全部的参数和环境变量。

**注意：** 使用 Slurm 时，需要设置端口，可从以下方式中选取一种。

1. 我们更推荐的通过`--cfg-options`设置端口，因为这不会改变原始配置：

   ```shell
   GPUS=4 GPUS_PER_NODE=4 sh tools/slurm_train.sh ${分区} ${任务名} config1.py ${工作路径} --cfg-options env_cfg.dist_cfg.port=29500
   GPUS=4 GPUS_PER_NODE=4 sh tools/slurm_train.sh ${任务名} ${工作路径} config2.py ${工作路径} --cfg-options env_cfg.dist_cfg.port=29501
   ```

2. 通过修改配置文件设置不同的通讯端口：

   在 `config1.py`中:

   ```python
   enf_cfg = dict(dist_cfg=dict(backend='nccl', port=29500))
   ```

   在 `config2.py`中：

   ```python

```

   ```python
   ```

   ```python
   enf_cfg = dict(dist_cfg=dict(backend='nccl', port=29501))
   ```

   然后您可以通过 config1.py 和 config2.py 同时启动两个任务：

   ```shell
   CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 sh tools/slurm_train.sh ${分区} ${任务名} config1.py ${工作路径}
   CUDA_VISIBLE_DEVICES=4,5,6,7 GPUS=4 sh tools/slurm_train.sh ${分区} ${任务名} config2.py ${工作路径}
   ```

3. 在命令行中通过环境变量 `MASTER_PORT` 设置端口 ：

```shell
CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 MASTER_PORT=29500 sh tools/slurm_train.sh ${分区} ${任务名} config1.py ${工作路径}
CUDA_VISIBLE_DEVICES=4,5,6,7 GPUS=4 MASTER_PORT=29501 sh tools/slurm_train.sh ${分区} ${任务名} config2.py ${工作路径}
```

## 测试并保存分割结果

### 基础使用

当需要保存测试输出的分割结果，用 `--out` 指定分割结果输出路径

```shell
python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} --out ${OUTPUT_DIR}
```

以保存模型 `fcn_r50-d8_4xb4-80k_ade20k-512x512` 在 ADE20K 验证数据集上的结果为例：

```shell

```shell
```

```shell
python tools/test.py configs/fcn/fcn_r50-d8_4xb4-80k_ade20k-512x512.py ckpt/fcn_r50-d8_512x512_80k_ade20k_20200614_144016-f8ac5082.pth --out work_dirs/format_results
```

或者通过配置文件定义 `output_dir`。例如在 `configs/fcn/fcn_r50-d8_4xb4-80k_ade20k-512x512.py` 添加 `test_evaluator` 定义：

```python
test_evaluator = dict(type='IoUMetric', iou_metrics=['mIoU'], output_dir='work_dirs/format_results')
```

然后执行相同功能的命令不需要再使用 `--out`：

```shell
python tools/test.py configs/fcn/fcn_r50-d8_4xb4-80k_ade20k-512x512.py ckpt/fcn_r50-d8_512x512_80k_ade20k_20200614_144016-f8ac5082.pth
```

当测试的数据集没有提供标注，评测时没有真值可以参与计算，因此需要设置 `format_only=True`，
同时需要修改 `test_dataloader`，由于没有标注，我们需要在数据增强变换中删掉 `dict(type='LoadAnnotations')`，以下是一个配置示例：

```python
    type='IoUMetric',
    iou_metrics=['mIoU'],

```python
    output_dir='work_dirs/format_results')
        type = 'ADE20KDataset'
        data_root='data/ade/release_test',
        data_prefix=dict(img_path='testing'),
        # 测试数据变换中没有加载标注
        pipeline=[
            dict(type='LoadImageFromFile'),
            dict(type='Resize', scale=(2048, 512), keep_ratio=True),
            dict(type='PackSegInputs')
        ]))
```

然后执行测试命令：

```shell
```

### 测试 Cityscapes 数据集并保存输出分割结果

推荐使用 `CityscapesMetric` 来保存模型在 Cityscapes 数据集上的测试结果，以下是一个配置示例：

```python

```

```python
    type='CityscapesMetric',
    keep_results=True,
        type='CityscapesDataset',
        data_root='data/cityscapes/',
        data_prefix=dict(img_path='leftImg8bit/test'),
            dict(type='Resize', scale=(2048, 1024), keep_ratio=True),
        ]))
```

然后执行相同的命令，例如：

```shell
python tools/test.py configs/fcn/fcn_r18-d8_4xb2-80k_cityscapes-512x1024.py ckpt/fcn_r18-d8_512x1024_80k_cityscapes_20201225_021327-6c50f8b4.pth
```

# CAE

> [Context Autoencoder for Self-Supervised Representation Learning](https://arxiv.org/abs/2202.03026)

We present a novel masked image modeling (MIM) approach, context autoencoder (CAE), for self-supervised learning. We randomly partition the image into two sets: visible patches and masked patches. The CAE architecture consists of: (i) an encoder that takes visible patches as input and outputs their latent representations, (ii) a latent context regressor that predicts the masked patch representations from the visible patch representations that are not updated in this regressor, (iii) a decoder that takes the estimated masked patch representations as input and makes predictions for the masked patches, and (iv) an alignment module that aligns the masked patch representation estimation with the masked patch representations computed from the encoder. In comparison to previous MIM methods that couple the encoding and decoding roles, e.g., using a single module in BEiT, our approach attempts to separate the encoding role (content understanding) from the decoding role (making predictions for masked patches) using different modules, improving the content understanding capability. In addition, our approach makes predictions from the visible patches to the masked patches in the latent representation space that is expected to take on semantics. In addition, we present the explanations about why contrastive pretraining and supervised pretraining perform similarly and why MIM potentially performs better. We demonstrate the effectiveness of our CAE through superior transfer performance in downstream tasks: semantic segmentation, and object detection and instance segmentation.

<img src="https://user-images.githubusercontent.com/30762564/165459947-6c6ef13c-0593-4765-b44e-6da0a079802a.png" width="70%"/>
</div>

```python

predict = inference_model('beit-base-p16_cae-pre_8xb128-coslr-100e_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('cae_beit-base-p16_8xb256-amp-coslr-300e_in1k', pretrained=True)
```

Train:

```shell
python tools/train.py configs/cae/cae_beit-base-p16_8xb256-amp-coslr-300e_in1k.py

Train:

```shell
```

Test:

```shell
python tools/test.py configs/cae/benchmarks/beit-base-p16_8xb128-coslr-100e_in1k.py https://download.openmmlab.com/mmselfsup/1.x/cae/cae_vit-base-p16_16xb128-fp16-coslr-300e_in1k/vit-base-p16_ft-8xb128-coslr-100e-rpe_in1k/vit-base-p16_ft-8xb128-coslr-100e-rpe_in1k_20220825-f3d234cd.pth
```

| Model                                          | Params (M) | Flops (G) |                          Config                           |                                    Download                                    |
| :--------------------------------------------- | :--------: | :-------: | :-------------------------------------------------------: | :----------------------------------------------------------------------------: |

| `cae_beit-base-p16_8xb256-amp-coslr-300e_in1k` |   288.43   |   17.58   | [config](cae_beit-base-p16_8xb256-amp-coslr-300e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/cae/cae_vit-base-p16_8xb256-amp-coslr-300e_in1k/cae_vit-base-p16_8xb256-amp-coslr-300e_in1k_20221230-808170f3.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/cae/cae_vit-base-p16_8xb256-amp-coslr-300e_in1k/cae_vit-base-p16_8xb256-amp-coslr-300e_in1k_20221230-808170f3.json) |

| `beit-base-p16_cae-pre_8xb128-coslr-100e_in1k` | [CAE](https://download.openmmlab.com/mmselfsup/1.x/cae/cae_vit-base-p16_8xb256-amp-coslr-300e_in1k/cae_vit-base-p16_8xb256-amp-coslr-300e_in1k_20221230-808170f3.pth) |   86.68    |   17.58   |   83.20   | [config](benchmarks/beit-base-p16_8xb128-coslr-100e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/cae/cae_vit-base-p16_16xb128-fp16-coslr-300e_in1k/vit-base-p16_ft-8xb128-coslr-100e-rpe_in1k/vit-base-p16_ft-8xb128-coslr-100e-rpe_in1k_20220825-f3d234cd.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/cae/cae_vit-base-p16_16xb128-fp16-coslr-300e_in1k/vit-base-p16_ft-8xb128-coslr-100e-rpe_in1k/vit-base-p16_ft-8xb128-coslr-100e-rpe_in1k_20220825-f3d234cd.json) |

```bibtex
@article{CAE,
  title={Context Autoencoder for Self-Supervised Representation Learning},
  author={Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo,
  Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, Jingdong Wang},

author={Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo,
  journal={ArXiv},
}
```

---
name: "\U0001F4DD others"
about: discussion, suggestion and question
---

## detail | 详细描述 | 詳細な説明

# 第四章：在 PyTorch 中支持更多 ONNX 算子

在[上一篇教程](03_pytorch2onnx.md)中，我们系统地学习了 PyTorch 转 ONNX 的方法，可以发现 PyTorch 对 ONNX 的支持还不错。但在实际的部署过程中，难免碰到模型无法用原生 PyTorch 算子表示的情况。这个时候，我们就得考虑扩充 PyTorch，即在 PyTorch 中支持更多 ONNX 算子。

而要使 PyTorch 算子顺利转换到 ONNX ，我们需要保证以下三个环节都不出错：

- 算子在 PyTorch 中有实现
- 有把该 PyTorch 算子映射成一个或多个 ONNX 算子的方法
- ONNX 有相应的算子

可在实际部署中，这三部分的内容都可能有所缺失。其中最坏的情况是：我们定义了一个全新的算子，它不仅缺少 PyTorch 实现，还缺少 PyTorch 到 ONNX 的映射关系。但所谓车到山前必有路，对于这三个环节，我们也分别都有以下的添加支持的方法：

- PyTorch 算子
  - 组合现有算子
  - 添加 TorchScript 算子
  - 添加普通 C++ 拓展算子
- 映射方法
  - 为 ATen 算子添加符号函数
  - 为 TorchScript 算子添加符号函数
  - 封装成 torch.autograd.Function 并添加符号函数
- ONNX 算子
  - 使用现有 ONNX 算子
  - 定义新 ONNX 算子

那么，面对不同的情况时，就需要我们灵活地选用和组合这些方法。听起来是不是很复杂？别担心，本篇文章中，我们将围绕着三种算子映射方法，学习三个添加算子支持的实例，来理清如何为 PyTorch 算子转 ONNX 算子的三个环节添加支持。

## 支持 ATen 算子

实际的部署过程中，我们都有可能会碰到一个最简单的算子缺失问题： 算子在 ATen 中已经实现了，ONNX 中也有相关算子的定义，但是相关算子映射成 ONNX 的规则没有写。在这种情况下，我们只需要**为 ATen 算子补充描述映射规则的符号函数**就行了。

> [ATen](https://pytorch.org/cppdocs/#aten) 是 PyTorch 内置的 C++ 张量计算库，PyTorch 算子在底层绝大多数计算都是用 ATen 实现的。

上期习题中，我们曾经提到了 ONNX 的 `Asinh` 算子。这个算子在 ATen 中有实现，却缺少了映射到 ONNX 算子的符号函数。在这里，我们来尝试为它补充符号函数，并导出一个包含这个算子的 ONNX 模型。

### 获取 ATen 中算子接口定义

为了编写符号函数，我们需要获得 `asinh` 推理接口的输入参数定义。这时，我们要去 `torch/_C/_VariableFunctions.pyi` 和 `torch/nn/functional.pyi` 这两个文件中搜索我们刚刚得到的这个算子名。这两个文件是编译 PyTorch 时本地自动生成的文件，里面包含了 ATen 算子的 PyTorch 调用接口。通过搜索，我们可以知道 `asinh` 在文件 `torch/_C/_VariableFunctions.pyi` 中，其接口定义为:

```python
def asinh(input: Tensor, *, out: Optional[Tensor]=None) -> Tensor: ...
```

经过这些步骤，我们确认了缺失的算子名为 `asinh`，它是一个有实现的 ATen 算子。我们还记下了 `asinh` 的调用接口。接下来，我们要为它补充符号函数，使它在转换成 ONNX 模型时不再报错。

### 添加符号函数

到目前为止，我们已经多次接触了定义 PyTorch 到 ONNX 映射规则的符号函数了。现在，我们向大家正式介绍一下符号函数。

```

### 添加符号函数

符号函数，可以看成是 PyTorch 算子类的一个静态方法。在把 PyTorch 模型转换成 ONNX 模型时，各个 PyTorch 算子的符号函数会被依次调用，以完成 PyTorch 算子到 ONNX 算子的转换。符号函数的定义一般如下：

```python
def symbolic(g: torch._C.Graph, input_0: torch._C.Value, input_1: torch._C.Value, ...):
```

其中，`torch._C.Graph` 和 `torch._C.Value` 都对应 PyTorch 的 C++ 实现里的一些类。我们在这篇文章不深究它们的细节，只需要知道第一个参数就固定叫 `g`，它表示和计算图相关的内容；后面的每个参数都表示算子的输入，需要和算子的前向推理接口的输入相同。对于 ATen 算子来说，它们的前向推理接口就是上述两个 `.pyi` 文件里的函数接口。

`g` 有一个方法 `op`。在把 PyTorch 算子转换成 ONNX 算子时，需要在符号函数中调用此方法来为最终的计算图添加一个 ONNX 算子。其定义如下：

```python
def op(name: str, input_0: torch._C.Value, input_1: torch._C.Value, ...)
```

其中，第一个参数是算子名称。如果该算子是普通的 ONNX 算子，只需要把它在 ONNX 官方文档里的名称填进去即可（我们稍后再讲其他情况）。

在最简单的情况下，我们只要把 PyTorch 算子的输入用`g.op()`一一对应到 ONNX 算子上即可，并把`g.op()`的返回值作为符号函数的返回值。在情况更复杂时，我们转换一个 PyTorch 算子可能要新建若干个 ONNX 算子。

```

补充完了背景知识，让我们回到 `asinh` 算子上，来为它编写符号函数。我们先去翻阅一下 ONNX 算子文档，学习一下我们在符号函数里的映射关系 `g.op()` 里应该怎么写。[`Asinh` 的文档](https://github.com/onnx/onnx/blob/main/docs/Operators.md#asinh)写道：该算子有一个输入 `input`，一个输出 `output`，二者的类型都为张量。

到这里，我们已经完成了信息收集环节。我们在上一小节得知了 `asinh` 的推理接口定义，在这一小节里收集了 ONNX 算子 `Asinh` 的定义。现在，我们可以用代码来补充这二者的映射关系了。在刚刚导出 `asinh` 算子的代码中，我们添加以下内容：

```python
from torch.onnx.symbolic_registry import register_op

def asinh_symbolic(g, input, *, out=None):
    return g.op("Asinh", input)

register_op('asinh', asinh_symbolic, '', 9)
```

这里的`asinh_symbolic`就是`asinh`的符号函数。从除`g`以外的第二个输入参数开始，其输入参数应该严格对应它在 ATen 中的定义：

```python
```

```

```python
```

在符号函数的函数体中，`g.op("Asinh", input)`则完成了 ONNX 算子的定义。其中，第一个参数`"Asinh"`是算子在 ONNX 中的名称。至于第二个参数 `input`，如我们刚刚在文档里所见，这个算子只有一个输入，因此我们只要把符号函数的输入参数 `input` 对应过去就行。ONNX 的 `Asinh` 的输出和 ATen 的 `asinh` 的输出是一致的，因此我们直接把 `g.op()` 的结果返回即可。

定义完符号函数后，我们要把这个符号函数和原来的 ATen 算子“绑定”起来。这里，我们要用到 `register_op` 这个 PyTorch API 来完成绑定。如示例所示，只需要一行简单的代码即可把符号函数 `asinh_symbolic` 绑定到算子 `asinh` 上：

```python
```

`register_op`的第一个参数是目标 ATen 算子名，第二个是要注册的符号函数，这两个参数很好理解。第三个参数是算子的“域”，对于普通 ONNX 算子，直接填空字符串即可。第四个参数表示向哪个算子集版本注册。我们遵照 ONNX 标准，向第 9 号算子集注册。值得注意的是，这里向第 9 号算子集注册，不代表较新的算子集（第 10 号、第 11 号……）都得到了注册。在示例中，我们先只向第 9 号算子集注册。

整理一下，我们最终的代码如下：

```python

        return torch.asinh(x)

```python

input = torch.rand(1, 3, 10, 10)
torch.onnx.export(model, input, 'asinh.onnx')
```

成功导出的话，`asinh.onnx` 应该长这个样子：

![](https://user-images.githubusercontent.com/47652064/169744691-f14e4fd4-c777-4562-aaa5-a5bf888f21f8.png)

### 测试算子

在完成了一份自定义算子后，我们一定要测试一下算子的正确性。一般我们要用 PyTorch 运行一遍原算子，再用推理引擎（比如 ONNX Runtime）运行一下 ONNX 算子，最后比对两次的运行结果。对于我们刚刚得到的 `asinh.onnx`，可以用如下代码来验证：

```python

torch_output = model(input).detach().numpy()

sess = onnxruntime.InferenceSession('asinh.onnx')
ort_output = sess.run(None, {'0': input.numpy()})[0]

assert np.allclose(torch_output, ort_output)
```

在这份代码里，我们用 PyTorch 做了一遍推理，并把结果转成了 numpy 格式。之后，我们又用 ONNX Runtime 对 onnx 文件做了一次推理。最后，我们使用 `np.allclose` 来保证两个结果张量的误差在一个可以允许的范围内。一切正常的话，运行这段代码后，`assert` 所在行不会报错，程序应该没有任何输出。

## 支持 TorchScript 算子

对于一些比较复杂的运算，仅使用 PyTorch 原生算子是无法实现的。这个时候，就要考虑自定义一个 PyTorch 算子，再把它转换到 ONNX 中了。新增 PyTorch 算子的方法有很多，PyTorch 官方比较推荐的一种做法是[添加 TorchScript 算子](https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html) 。

由于添加算子的方法较繁琐，我们今天跳过新增 TorchScript 算子的内容，以可变形卷积（Deformable Convolution）算子为例，介绍为现有 TorchScript 算子添加 ONNX 支持的方法。

> 可变形卷积（Deformable Convolution）是在 Torchvision 中实现的 TorchScript 算子，虽然尚未得到广泛支持，但是出现在许多模型中。

有了支持 ATen 算子的经验之后，我们可以知道为算子添加符号函数一般要经过以下几步：

1. 获取原算子的前向推理接口。
2. 获取目标 ONNX 算子的定义。
3. 编写符号函数并绑定。

在为可变形卷积添加符号函数时，我们也可以尝试走一遍这个流程。

### 使用 TorchScript 算子

和之前一样，我们首先定义一个包含了算子的模型，为之后转换 ONNX 模型做准备。

```python

        self.conv1 = torch.nn.Conv2d(3, 18, 3)
        self.conv2 = torchvision.ops.DeformConv2d(3, 3, 3)

        return self.conv2(x, self.conv1(x))
```

其中，`torchvision.ops.DeformConv2d` 就是 Torchvision 中的可变形卷积层。相比于普通卷积，可变形卷积的其他参数都大致相同，唯一的区别就是在推理时需要多输入一个表示偏移量的张量。

然后，我们查询算子的前向推理接口。`DeformConv2d` 层最终会调用 `deform_conv2d` 这个算子。我们可以在 `torchvision/csrc/ops/deform_conv2d.cpp` 中查到该算子的调用接口：

```python
m.def(TORCH_SELECTIVE_SCHEMA(

```python
      "torchvision::deform_conv2d(Tensor input,
      Tensor weight,
      Tensor offset,
      ......
      bool use_mask) -> Tensor"));
```

那么接下来，根据之前的经验，我们就是要去 ONNX 官方文档中查找算子的定义了。

### 自定义 ONNX 算子

很遗憾的是，如果我们去 ONNX 的官方算子页面搜索 "deform"，将搜不出任何内容。目前，ONNX 还没有提供可变形卷积的算子，我们要自己定义一个 ONNX 算子了。

我们在前面讲过，`g.op()` 是用来定义 ONNX 算子的函数。对于 ONNX 官方定义的算子，`g.op()` 的第一个参数就是该算子的名称。而对于一个自定义算子，`g.op()` 的第一个参数是一个带命名空间的算子名，比如：

```python
g.op("custom::deform_conv2d, ...)
```

其中，"::"前面的内容就是我们的命名空间。该概念和 C++ 的命名空间类似，是为了防止命名冲突而设定的。如果在 `g.op()` 里不加前面的命名空间，则算子会被默认成 ONNX 的官方算子。

PyTorch 在运行 `g.op()` 时会对官方的算子做检查，如果算子名有误，或者算子的输入类型不正确， `g.op()` 就会报错。为了让我们随心所欲地定义新 ONNX 算子，我们必须设定一个命名空间，给算子取个名，再定义自己的算子。

我们在[第一篇教程](01_introduction_to_model_deployment.md)学过：ONNX 是一套标准，本身并不包括实现。在这里，我们就简略地定义一个 ONNX 可变形卷积算子，而不去写它在某个推理引擎上的实现。在之后的教程中，我们再学习在各个推理引擎中添加新 ONNX 算子支持的方法。此处，我们只关心如何导出一个包含新 ONNX 算子节点的 onnx 文件。因此，我们可以为新算子编写如下简单的符号函数：

```python
@parse_args("v", "v", "v", "v", "v", "i", "i", "i", "i", "i", "i", "i", "i", "none")
def symbolic(g,
        input,
        weight,
        offset,
        mask,
        bias,
        stride_h, stride_w,
        pad_h, pad_w,
        dil_h, dil_w,
        n_weight_grps,
        n_offset_grps,
        use_mask):
    return g.op("custom::deform_conv2d", input, offset)
```

在这个符号函数中，我们以刚刚搜索到的算子输入参数作为符号函数的输入参数，并只用 `input` 和 `offset` 来构造一个简单的 ONNX 算子。

n_offset_grps,
```

这段代码中，最令人疑惑的就是装饰器 `@parse_args` 了。简单来说，TorchScript 算子的符号函数要求标注出每一个输入参数的类型。比如"v"表示 Torch 库里的 `value` 类型，一般用于标注张量，而"i"表示 int 类型，"f"表示 float 类型，"none"表示该参数为空。具体的类型含义可以在 [torch.onnx.symbolic_helper.py](https://github.com/pytorch/pytorch/blob/main/torch/onnx/symbolic_helper.py)中查看。这里输入参数中的 `input, weight, offset, mask, bias` 都是张量，所以用"v"表示。后面的其他参数同理。我们不必纠结于 `@parse_args`的原理，根据实际情况对符号函数的参数标注类型即可。

有了符号函数后，我们通过如下的方式注册符号函数：

```python
register_custom_op_symbolic("torchvision::deform_conv2d", symbolic, 9)
```

和前面的 `register_op` 类似，注册符号函数时，我们要输入算子名、符号函数、算子集版本。与前面不同的是，这里的算子集版本是最早生效版本，在这里设定版本 9，意味着之后的第 10 号、第 11 号……版本集都能使用这个新算子。

最后，我们完整的模型导出代码如下：

```python

```python

from torch.onnx import register_custom_op_symbolic
from torch.onnx.symbolic_helper import parse_args

        input,
        weight,
        offset,
        mask,
        bias,

torch.onnx.export(model, input, 'dcn.onnx')
```

代码成功运行的话，我们应该能得到如下的 ONNX 模型：

```

![](https://user-images.githubusercontent.com/47652064/169744720-51ea91bc-b67b-4911-9e43-0adc1b64d2c1.jpg)

可以看到，我们自定义的 ONNX 算子 `deform_conv2d` 包含了两个输入，一个输出，和我们预想得一样。

## 使用 torch.autograd.Function

最后，我们来学习一种简单的为 PyTorch 添加 C++ 算子实现的方法，来代替较为复杂的新增 TorchScript 算子。同时，我们会用 torch.autograd.Function 封装这个新算子。torch.autograd.Function 能完成算子实现和算子调用的隔离。不管算子是怎么实现的，它封装后的使用体验以及 ONNX 导出方法会和原生的 PyTorch 算子一样。这是我们比较推荐的为算子添加 ONNX 支持的方法。

为了应对更复杂的情况，我们来自定义一个奇怪的 `my_add` 算子。这个算子的输入张量 a, b ，输出 `2a + b` 的值。我们会先把它在 PyTorch 中实现，再把它导出到 ONNX 中。

### 为 PyTorch 添加 C++ 拓展

为 PyTorch 添加简单的 C++ 拓展还是很方便的。对于我们定义的 my_add 算子，可以用以下的 C++ 源文件来实现。我们把该文件命名为 "my_add.cpp"：

```C++
// my_add.cpp

#include <torch/torch.h>

torch::Tensor my_add(torch::Tensor a, torch::Tensor b)
{
    return 2 * a + b;
}

PYBIND11_MODULE(my_lib, m)
{

```C++

{
}

{
    m.def("my_add", my_add);
}
```

由于在 PyTorch 中添加 C++ 拓展和模型部署关系不大，这里我们仅给出这个简单的示例，并不对其原理做过多讲解。

在这段代码中，torch::Tensor 就是 C++ 中 torch 的张量类型，它的加法和乘法等运算符均已重载。因此，我们可以像对普通标量一样对张量做加法和乘法。

轻松地完成了算子的实现后，我们用 `PYBIND11_MODULE` 来为 C++ 函数提供 Python 调用接口。这里的 `my_lib` 是我们未来要在 Python 里导入的模块名。双引号中的 `my_add` 是 Python 调用接口的名称，这里我们对齐 C++ 函数的名称，依然用 "my_add"这个名字。

之后，我们可以编写如下的 Python 代码并命名为 "setup.py"，来编译刚刚的 C++ 文件：

```python
from setuptools import setup
from torch.utils import cpp_extension

setup(name='my_add',
      ext_modules=[cpp_extension.CppExtension('my_lib', ['my_add.cpp'])],
      cmdclass={'build_ext': cpp_extension.BuildExtension})
```

```

这段代码使用了 Python 的 setuptools 编译功能和 PyTorch 的 C++ 拓展工具函数，可以编译包含了 torch 库的 C++ 源文件。这里我们需要填写的只有模块名和模块中的源文件名。我们刚刚把模块命名为 `my_lib`，而源文件只有一个 `my_add.cpp`，因此拓展模块那一行要写成 `ext_modules=[cpp_extension.CppExtension('my_lib', ['my_add.cpp'])],`。

之后，像处理普通的 Python 包一样执行安装命令，我们的 C++ 代码就会自动编译了。

```shell
python setup.py develop
```

### 用 `torch.autograd.Function` 封装

直接用 Python 接口调用 C++ 函数不太“美观”，一种比较优雅的做法是把这个调用接口封装起来。这里我们用 `torch.autograd.Function` 来封装算子的底层调用：

```python
import my_lib
class MyAddFunction(torch.autograd.Function):

    def forward(ctx, a, b):
        return my_lib.my_add(a, b)

    def symbolic(g, a, b):
        two = g.op("Constant", value_t=torch.tensor([2]))
        a = g.op('Mul', a, two)
        return g.op('Add', a, b)
```

@staticmethod
```

我们在前面的教程中已经见过 `torch.autograd.Function`，这里我们正式地对其做一个介绍。`Function` 类本身表示 PyTorch 的一个可导函数，只要为其定义了前向推理和反向传播的实现，我们就可以把它当成一个普通 PyTorch 函数来使用。

PyTorch 会自动调度该函数，合适地执行前向和反向计算。对模型部署来说，`Function` 类有一个很好的性质：如果它定义了 `symbolic` 静态方法，该 `Function` 在执行 `torch.onnx.export()` 时就可以根据 `symbolic` 中定义的规则转换成 ONNX 算子。这个 `symbolic` 就是前面提到的符号函数，只是它的名称必须是 `symbolic` 而已。

在 `forward `函数中，我们用 `my_lib.my_add(a, b)` 就可以调用之前写的C++函数了。这里 `my_lib` 是库名，`my_add` 是函数名，这两个名字是在前面C++的 `PYBIND11_MODULE` 中定义的。

在 `symbolic` 函数中，我们用 `g.op()` 定义了三个算子：常量、乘法、加法。这里乘法和加法的用法和前面提到的 `asinh` 一样，只需要根据 ONNX 算子定义规则把输入参数填入即可。而在定义常量算子时，我们要把 PyTorch 张量的值传入 `value_t` 参数中。

在 ONNX 中，我们需要把新建常量当成一个算子来看待，尽管这个算子并不会以节点的形式出现在 ONNX 模型的可视化结果里。

把算子封装成 Function 后，我们可以把 `my_add` 算子用起来了。

```python
my_add = MyAddFunction.apply

class MyAdd(torch.nn.Module):

```python

    def forward(self, a, b):
        return my_add(a, b)
```

在这份代码里，我们先用 `my_add = MyAddFunction.apply` 获取了一个奇怪的变量。这个变量是用来做什么的呢？其实，`apply`是`torch.autograd.Function` 的一个方法，这个方法完成了 `Function` 在前向推理或者反向传播时的调度。我们在使用 `Function` 的派生类做推理时，不应该显式地调用 `forward`，而应该调用其 `apply` 方法。

这里我们使用 `my_add = MyAddFunction.apply` 把这个调用方法取了一个更简短的别名 `my_add`。以后在使用 `my_add` 算子时，我们应该忽略 `MyAddFunction` 的实现细节，而只通过 `my_add` 这个接口来访问算子。这里 `my_add` 的地位，和 PyTorch 的 `asinh, interpolate, conv2d`等原生函数是类似的。

有了访问新算子的接口后，我们可以进一步把算子封装成一个神经网络中的计算层。我们定义一个叫做的 `MyAdd` 的 `torch.nn.Module`，它封装了`my_add`，就和封装了`conv2d` 的 `torch.nn.Conv2d` 一样。

### 测试算子

费了好大的功夫来“包装”我们的新算子后，我们终于可以来使用它了。和之前的测试流程一样，让我们用下面的代码来导出一个包含新算子的 ONNX 模型，并验证一下它是否正确。

```python
model = MyAdd()

### 测试算子

```python
torch.onnx.export(model, (input, input), 'my_add.onnx')
torch_output = model(input, input).detach().numpy()

sess = onnxruntime.InferenceSession('my_add.onnx')
ort_output = sess.run(None, {'a': input.numpy(), 'b': input.numpy()})[0]

```

在这份代码中，我们直接把 `MyAdd` 作为要导出的模型。我们计算了一个 PyTorch 模型的运行结果，又导出 ONNX 模型，计算了 ONNX 模型在 ONNX Runtime 上的运算结果。如果一切正常的话，这两个结果是一样的，这份代码不会报任何错误，没有任何输出。

![](https://user-images.githubusercontent.com/47652064/169744753-0fb00930-bbca-4636-8681-4ec4e7b31946.jpg)

可视化一下 `my_add.onnx`，可以看出，和我们设计得一样，`my_add` 算子被翻译成了两个 ONNX 算子节点（其中常量算子被放入了 `Mul` 的参数中）。

整理一下，整个流程的 Python 代码如下：

```python

```python

```

## 总结

```

## 总结

在这篇教程中，我们围绕“为 ATen 算子添加符号函数”、“为 TorchScript 算子添加符号函数”、“封装成 `torch.autograd.Function` 并添加符号函数”这三种添加映射关系的方法，讲解了 3 个为 PyTorch 和 ONNX 添加支持的实例。在这个过程中，我们学到了很多零散的知识，来总结一下吧。

- ATen 是 PyTorch 的 C++ 张量运算库。通过查询 torch/\_C/\_VariableFunctions.pyi 和 torch/nn/functional.pyi，我们可以知道 ATen 算子的 Python 接口定义。
- 用 register_op 可以为 ATen 算子补充注册符号函数
- 用 register_custom_op_symbolic 可以为 TorchScript 算子补充注册符号函数
- 如何在 PyTorch 里添加 C++ 拓展
- 如何用 torch.autograd.Function 封装一个自定义 PyTorch 算子
- 如何编写符号函数 symbolic(g, ...)。
- 如何用 g.op() 把一个 PyTorch 算子映射成一个或多个 ONNX 算子，或者是自定义的 ONNX 算子。

这篇教程涉及的代码比较多。如果大家在阅读时碰到了问题，最好去跑一跑代码，改一改代码里的内容，实际感受一下每行代码的意义。

## 上期习题解答

1. PyTorch 目前没有支持 ONNX 的 `Asinh` 算子。我们在 `torch.onnx.symbolic_opset9.py` 中搜索不到 Asinh 的相关内容。

## 上期习题解答

2. 通过在 `torch.onnx.symbolic_opset11.py` 搜索 `BitShift`，我们可以发现 PyTorch 在 `__lshift_` 和 `__rshift_` 里用到了ONNX的 `BitShift` 算子。当输入类型为 `Byte` 时，PyTorch会把算子直接翻译翻译 `BitShift`，以代替乘除 2 的次幂的操作。
3. 对应 `Resize` 算子的第3个参数（`g.op()` 的第4个参数）`scales`。原来的 `scales` 传入 `g.op() `前会经过 `_interpolate_get_scales_if_available()` 函数，一定会被转换成一个常量。为了让 `scales` 由输入决定，我们直接把输入参数中的 `scales` 传入 `g.op()`。

# NRTR

> [NRTR: A No-Recurrence Sequence-to-Sequence Model For Scene Text Recognition](https://arxiv.org/abs/1806.00926)

Scene text recognition has attracted a great many researches due to its importance to various applications. Existing methods mainly adopt recurrence or convolution based networks. Though have obtained good performance, these methods still suffer from two limitations: slow training speed due to the internal recurrence of RNNs, and high complexity due to stacked convolutional layers for long-term feature extraction. This paper, for the first time, proposes a no-recurrence sequence-to-sequence text recognizer, named NRTR, that dispenses with recurrences and convolutions entirely. NRTR follows the encoder-decoder paradigm, where the encoder uses stacked self-attention to extract image features, and the decoder applies stacked self-attention to recognize texts based on encoder output. NRTR relies solely on self-attention mechanism thus could be trained with more parallelization and less complexity. Considering scene image has large variation in text and background, we further design a modality-transform block to effectively transform 2D input images to 1D sequences, combined with the encoder to extract more discriminative features. NRTR achieves state-of-the-art or highly competitive performance on both regular and irregular benchmarks, while requires only a small fraction of training time compared to the best model from the literature (at least 8 times faster).

<img src="https://user-images.githubusercontent.com/22607038/142797203-d9df6c35-868f-4848-8261-c286751fd342.png"/>
</div>

## Dataset

| trainset  | instance_num | repeat_num | source |
| :-------: | :----------: | :--------: | :----: |
| SynthText |   7266686    |     1      | synth  |
|  Syn90k   |   8919273    |     1      | synth  |

| testset | instance_num |   type    |

|                           Methods                           |       Backbone        |        | Regular Text |           |     |           | Irregular Text |        |                           download                            |

| :---------------------------------------------------------: | :-------------------: | :----: | :----------: | :-------: | :-: | :-------: | :------------: | :----: | :-----------------------------------------------------------: |
|                                                             |                       | IIIT5K |     SVT      | IC13-1015 |     | IC15-2077 |      SVTP      |  CT80  |                                                               |
| [NRTR](/configs/textrecog/nrtr/nrtr_modality-transform_6e_st_mj.py) | NRTRModalityTransform | 0.9147 |    0.8841    |  0.9369   |     |  0.7246   |     0.7783     | 0.7500 | [model](https://download.openmmlab.com/mmocr/textrecog/nrtr/nrtr_modality-transform_6e_st_mj/nrtr_modality-transform_6e_st_mj_20220916_103322-bd9425be.pth) \| [log](https://download.openmmlab.com/mmocr/textrecog/nrtr/nrtr_modality-transform_6e_st_mj/20220916_103322.log) |

| [NRTR-TTA](/configs/textrecog/nrtr/nrtr_modality-transform_6e_st_mj.py) | NRTRModalityTransform | 0.9123 |    0.8825    |  0.9310   |     |  0.7492   |     0.7798     | 0.7535 |                                                               |
| [NRTR](/configs/textrecog/nrtr/nrtr_resnet31-1by8-1by4_6e_st_mj.py) |      R31-1/8-1/4      | 0.9483 |    0.8918    |  0.9507   |     |  0.7578   |     0.8016     | 0.8889 | [model](https://download.openmmlab.com/mmocr/textrecog/nrtr/nrtr_resnet31-1by8-1by4_6e_st_mj/nrtr_resnet31-1by8-1by4_6e_st_mj_20220916_103322-a6a2a123.pth) \| [log](https://download.openmmlab.com/mmocr/textrecog/nrtr/nrtr_resnet31-1by8-1by4_6e_st_mj/20220916_103322.log) |
| [NRTR-TTA](/configs/textrecog/nrtr/nrtr_resnet31-1by8-1by4_6e_st_mj.py) |      R31-1/8-1/4      | 0.9443 |    0.8903    |  0.9478   |     |  0.7790   |     0.8078     | 0.8854 |                                                               |

| [NRTR](/configs/textrecog/nrtr/nrtr_resnet31-1by16-1by8_6e_st_mj.py) |     R31-1/16-1/8      | 0.9470 |    0.8918    |  0.9399   |     |  0.7376   |     0.7969     | 0.8854 | [model](https://download.openmmlab.com/mmocr/textrecog/nrtr/nrtr_resnet31-1by16-1by8_6e_st_mj/nrtr_resnet31-1by16-1by8_6e_st_mj_20220920_143358-43767036.pth) \| [log](https://download.openmmlab.com/mmocr/textrecog/nrtr/nrtr_resnet31-1by16-1by8_6e_st_mj/20220920_143358.log) |
| [NRTR-TTA](/configs/textrecog/nrtr/nrtr_resnet31-1by16-1by8_6e_st_mj.py) |     R31-1/16-1/8      | 0.9423 |    0.8903    |  0.9360   |     |  0.7641   |     0.8016     | 0.8854 |                                                               |

```bibtex
@inproceedings{sheng2019nrtr,
  title={NRTR: A no-recurrence sequence-to-sequence model for scene text recognition},
  author={Sheng, Fenfen and Chen, Zhineng and Xu, Bo},
  booktitle={2019 International Conference on Document Analysis and Recognition (ICDAR)},

author={Sheng, Fenfen and Chen, Zhineng and Xu, Bo},
  pages={781--786},
}
```

## 摄像头推理

从版本 v1.1.0 开始，原来的摄像头 API 已被弃用。用户现在可以选择使用推理器（Inferencer）或 Demo 脚本从摄像头读取的视频中进行姿势估计。

### 使用推理器进行摄像头推理

用户可以通过执行以下命令来利用 MMPose Inferencer 对摄像头输入进行人体姿势估计：

```shell
python demo/inferencer_demo.py webcam --pose2d 'human'
```

有关推理器的参数详细信息，请参阅 [推理器文档](/docs/en/user_guides/inference.md)。

### 使用 Demo 脚本进行摄像头推理

除了 `demo/image_demo.py` 之外，所有的 Demo 脚本都支持摄像头输入。

以 `demo/topdown_demo_with_mmdet.py` 为例，用户可以通过在命令中指定 **`--input webcam`** 来使用该脚本对摄像头输入进行推理：

```shell
# inference with webcam
    projects/rtmpose/rtmdet/person/rtmdet_nano_320-8xb32_coco-person.py \
    https://download.openmmlab.com/mmpose/v1/projects/rtmpose/rtmdet_nano_8xb32-100e_coco-obj365-person-05d8511e.pth \
    projects/rtmpose/rtmpose/body_2d_keypoint/rtmpose-m_8xb256-420e_coco-256x192.py \
    https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-aic-coco_pt-aic-coco_420e-256x192-63eb25f7_20230126.pth \
    --input webcam \
    --show
```

# DIM (CVPR'2017)

> **任务**: 图像抠图

<details>
<summary align="right">DIM (CVPR'2017)</summary>

```bibtex
}
```

<br/>

|                                     算法                                      |   SAD    |    MSE    |   GRAD   |   CONN   | GPU 信息 |                                        下载                                        |
| :---------------------------------------------------------------------------: | :------: | :-------: | :------: | :------: | :------: | :--------------------------------------------------------------------------------: |

|                                第一阶段 (原文)                                |   54.6   |   0.017   |   36.7   |   55.3   |    -     |                                         -                                          |
|                                第三阶段 (原文)                                | **50.4** | **0.014** |   31.0   |   50.8   |    -     |                                         -                                          |
|           [第一阶段 (复现)](./dim_stage1-v16_1xb1-1000k_comp1k.py)            |   53.8   |   0.017   |   32.7   |   54.5   |    1     | [模型](https://download.openmmlab.com/mmediting/mattors/dim/dim_stage1_v16_1x1_1000k_comp1k_SAD-53.8_20200605_140257-979a420f.pth) \| [日志](https://download.openmmlab.com/mmediting/mattors/dim/dim_stage1_v16_1x1_1000k_comp1k_20200605_140257.log.json) |

|         [第二阶段 (复现)](./dim_stage2-v16-pln_1xb1-1000k_comp1k.py)          |   52.3   |   0.016   |   29.4   |   52.4   |    1     | [模型](https://download.openmmlab.com/mmediting/mattors/dim/dim_stage2_v16_pln_1x1_1000k_comp1k_SAD-52.3_20200607_171909-d83c4775.pth) \| [日志](https://download.openmmlab.com/mmediting/mattors/dim/dim_stage2_v16_pln_1x1_1000k_comp1k_20200607_171909.log.json) |
|         [第三阶段 (复现)](./dim_stage3-v16-pln_1xb1-1000k_comp1k.py)          |   50.6   |   0.015   | **29.0** | **50.7** |    1     | [模型](https://download.openmmlab.com/mmediting/mattors/dim/dim_stage3_v16_pln_1x1_1000k_comp1k_SAD-50.6_20200609_111851-647f24b6.pth) \| [日志](https://download.openmmlab.com/mmediting/mattors/dim/dim_stage3_v16_pln_1x1_1000k_comp1k_20200609_111851.log.json) |
| [第一阶段 (online merge)](./dim_stage1-v16_1xb1-1000k_comp1k_online-merge.py) |    -     |     -     |    -     |    -     |    -     |                                         -                                          |

**注**

- 第一阶段：训练不带精炼器的编码器-解码器部分。 \\
- 第二阶段：固定编码器-解码器部分，训练精炼器部分。 \\
- 第三阶段：微调整个网络模型。

> 模型在训练过程中的性能不稳定。因此，展示的性能并非来自最后一个模型权重文件，而是训练期间在验证集上取得的最佳性能。

> 不同随机种子的训练性能（最佳性能）的发散程度很大，您可能需要为每个设置运行多个实验以获得上述性能。

## 快速开始

**训练**

<details>

DIM 训练有三个阶段。

**阶段 1**: 训练不带精炼器的编码器-解码器部分。

```shell
# CPU上训练
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/dim/dim_stage1-v16_1xb1-1000k_comp1k.py

# 单个GPU上训练
python tools/train.py configs/dim/dim_stage1-v16_1xb1-1000k_comp1k.py

# 多个GPU上训练
./tools/dist_train.sh configs/dim/dim_stage1-v16_1xb1-1000k_comp1k.py 8
```

**阶段 2**: 固定编码器-解码器部分，训练精炼器部分。

```shell
# CPU上训练
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/dim/dim_stage2-v16-pln_1xb1-1000k_comp1k.py

# 单个GPU上训练
python tools/train.py configs/dim/dim_stage2-v16-pln_1xb1-1000k_comp1k.py

# 多个GPU上训练
./tools/dist_train.sh configs/dim/dim_stage2-v16-pln_1xb1-1000k_comp1k.py 8
```

**阶段 3**: 微调整个网络模型。

```shell
# CPU上训练

# 多个GPU上训练
```

```shell
# CPU上训练
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/dim/dim_stage3-v16-pln_1xb1-1000k_comp1k.py

# 单个GPU上训练
python tools/train.py configs/dim/dim_stage3-v16-pln_1xb1-1000k_comp1k.py

# 多个GPU上训练
./tools/dist_train.sh configs/dim/dim_stage3-v16-pln_1xb1-1000k_comp1k.py 8
```

**测试**

<details>

```shell
# CPU上测试
CUDA_VISIBLE_DEVICES=-1 python tools/test.py onfigs/dim/dim_stage3-v16-pln_1xb1-1000k_comp1k.py https://download.openmmlab.com/mmediting/mattors/dim/dim_stage3_v16_pln_1x1_1000k_comp1k_SAD-50.6_20200609_111851-647f24b6.pth

# 单个GPU上测试

# 单个GPU上测试
python tools/test.py onfigs/dim/dim_stage3-v16-pln_1xb1-1000k_comp1k.py https://download.openmmlab.com/mmediting/mattors/dim/dim_stage3_v16_pln_1x1_1000k_comp1k_SAD-50.6_20200609_111851-647f24b6.pth

# 多个GPU上测试
./tools/dist_test.sh onfigs/dim/dim_stage3-v16-pln_1xb1-1000k_comp1k.py https://download.openmmlab.com/mmediting/mattors/dim/dim_stage3_v16_pln_1x1_1000k_comp1k_SAD-50.6_20200609_111851-647f24b6.pth 8
```

# Solution of FGIA ACCV 2022(1st Place)

This is fine-tuning part of the 1st Place Solution for Webly-supervised Fine-grained Recognition, refer to the ACCV workshop competition in https://www.cvmart.net/race/10412/base.

## Result

<details>

<summary>Show the result</summary>

<br>

**Leaderboard A**

![LB-A](https://user-images.githubusercontent.com/18586273/205498131-5728e470-b4f6-43b7-82a5-5f8e3bd5168e.png)

**Leaderboard B**

![LB-B](https://user-images.githubusercontent.com/18586273/205498171-5a3a3055-370a-4a8b-9779-b686254ebc94.png)

</br>

## Reproduce

For detailed self-supervised pretrain code, please refer to [Self-spervised Pre-training](#self-supervised-pre-training).
For detailed finetuning and inference code, please refer to [this repo](https://github.com/Ezra-Yu/ACCV2022_FGIA_1st).

### Overview of Our Solution

![image](https://user-images.githubusercontent.com/18586273/205498371-31dbc1f4-5814-44bc-904a-f0d32515c7dd.png)

### Our Model

- ViT(MAE-pre-train)   # Pretrained with [MAE](https://github.com/open-mmlab/mmppretrain/tree/main/projects/fgia_accv2022_1st/config/mae_vit-large-p16_8xb512-amp-coslr-1600e_in1k.py)
- Swin-v2(SimMIM-pre-train)   # From [MMPretrain-swin_transformer_v2](https://github.com/open-mmlab/mmppretrain/tree/main/configs/swin_transformer_v2).

\*\*The architectures we use \*\*

- ViT + CE-loss + post-LongTail-Adjusment
- ViT + SubCenterArcFaceWithAdvMargin(CE)
- Swin-B + SubCenterArcFaceWithAdvMargin(SoftMax-EQL)
- Swin-L + SubCenterArcFaceWithAdvMargin(SoftMAx-EQL)

## Self-supervised Pre-training

### Requirements

```shell
PyTorch 1.11.0
torchvision 0.12.0
CUDA 11.3
MMEngine >= 0.1.0
MMCV >= 2.0.0rc0
```

### Preparing the dataset

First you should refactor the folder of your dataset in the following format:

```text

CUDA 11.3
```

```text
mmpretrain
|
|── data
|    |── WebiNat5000
|    |       |── meta
|    |       |    |── train.txt
|    |       |── train
|    |       |── testa
|    |       |── testb
```

The `train`, `testa`, and `testb` folders contain the same content with
those provided by the official website of the competition.

### Start pre-training

First, you should install all these requirements, following this [page](https://mmpretrain.readthedocs.io/en/latest/get_started.html).
Then change your current directory to the root of MMPretrain

```shell
cd $MMPretrain
```

Then you have the following two choices to start pre-training

#### Slurm

If you have a cluster managed by Slurm, you can use the following command:

```shell
## we use 16 NVIDIA 80G A100 GPUs for pre-training

#### Slurm

```shell
GPUS_PER_NODE=8 GPUS=16 SRUN_ARGS=${SRUN_ARGS} bash tools/slurm_train.sh ${PARTITION} ${JOB_NAME} projects/fgia_accv2022_1st/config/mae_vit-large-p16_8xb512-amp-coslr-1600e_in1k.py [optional arguments]
```

#### Pytorch

Or you can use the following two commands to start distributed training on two separate nodes:

```shell
# node 1
NNODES=2 NODE_RANK=0 PORT=${MASTER_PORT} MASTER_ADDR=${MASTER_ADDR} bash tools/dist_train.sh projects/fgia_accv2022_1st/config/mae_vit-large-p16_8xb512-amp-coslr-1600e_in1k.py 8
```

```shell
# node 2
NNODES=2 NODE_RANK=1 PORT=${MASTER_PORT} MASTER_ADDR=${MASTER_ADDR} bash tools/dist_train.sh projects/fgia_accv2022_1st/config/mae_vit-large-p16_8xb512-amp-coslr-1600e_in1k.py 8
```

All these logs and checkpoints will be saved under the folder `work_dirs`in the root.

## Fine-tuning with bag of tricks

```

- [MAE](https://github.com/open-mmlab/mmpretrain/tree/main/configs/mae) |  [Config](https://github.com/Ezra-Yu/ACCV_workshop/tree/master/configs/vit)
- [Swinv2](https://github.com/open-mmlab/mmpretrain/tree/main/configs/swin_transformer_v2) | [Config](https://github.com/Ezra-Yu/ACCV_workshop/tree/master/configs/swin)
- [ArcFace](https://arxiv.org/abs/1801.07698)   |   [Code](https://github.com/Ezra-Yu/ACCV_workshop/blob/master/src/models/arcface_head.py)
- [SubCenterArcFaceWithAdvMargin](https://paperswithcode.com/paper/sub-center-arcface-boosting-face-recognition)   |   [Code](https://github.com/Ezra-Yu/ACCV_workshop/blob/master/src/models/arcface_head.py)
- [Post-LT-adjusment](https://paperswithcode.com/paper/long-tail-learning-via-logit-adjustment)   |   [Code](https://github.com/Ezra-Yu/ACCV_workshop/blob/master/src/models/linear_head_lt.py)

- [SoftMaxEQL](https://paperswithcode.com/paper/the-equalization-losses-gradient-driven)   |   [Code](https://github.com/Ezra-Yu/ACCV_workshop/blob/master/src/models/eql.py)
- FlipTTA [Code](https://github.com/Ezra-Yu/ACCV_workshop/blob/master/src/models/tta_classifier.py)
- clean dataset
- self-emsemble: [Uniform-model-soup](https://arxiv.org/abs/2203.05482) | [code](https://github.com/Ezra-Yu/ACCV_workshop/blob/master/tools/model_soup.py)
- [pseudo](https://lilianweng.github.io/posts/2021-12-05-semi-supervised/)  | [Code](https://github.com/Ezra-Yu/ACCV_workshop/blob/master/tools/creat_pseudo.py)
- bagging-emsemble [Code](https://github.com/Ezra-Yu/ACCV_workshop/blob/master/tools/emsemble.py),

- post-process: [re-distribute-label](https://github.com/Ezra-Yu/ACCV_workshop/blob/master/tools/re-distribute-label.py);

![Overview](https://user-images.githubusercontent.com/18586273/205498258-e5720d83-7006-4aea-86b5-aab1a8998c6c.png)

![image](https://user-images.githubusercontent.com/18586273/205498027-def99b0d-a99a-470b-b292-8d5fc83111fc.png)

#### Used but no improvements

1. Using retrieval paradigm to solve this classification task;
2. Using EfficientNetv2 backbone.

#### Not used but worth to do

1. Try [DiVE](https://arxiv.org/abs/2103.15042) algorithm to improve performance in long tail dataset;
2. Use SimMIM to pre-train Swin-v2 on the competition dataset;
3. refine the re-distribute-label tool.

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_ECCV_2018/html/Ningning_Light-weight_CNN_Architecture_ECCV_2018_paper.html">ShufflenetV2 (ECCV'2018)</a></summary>

```bibtex
@inproceedings{ma2018shufflenet,
  title={Shufflenet v2: Practical guidelines for efficient cnn architecture design},
  author={Ma, Ningning and Zhang, Xiangyu and Zheng, Hai-Tao and Sun, Jian},
  pages={116--131},
}
```

<details>

```bibtex

| [pose_shufflenetv2](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_shufflenetv2_8xb64-210e_mpii-256x256.py) |  256x256   | 0.828 |  0.205   | [ckpt](https://download.openmmlab.com/mmpose/top_down/shufflenetv2/shufflenetv2_mpii_256x256-4fb9df2d_20200925.pth) | [log](https://download.openmmlab.com/mmpose/top_down/shufflenetv2/shufflenetv2_mpii_256x256_20200925.log.json) |

# Train predefined models on standard datasets

MMDetection also provides out-of-the-box tools for training detection models.
This section will show how to train _predefined_ models (under [configs](../../../configs)) on standard datasets i.e. COCO.

## Prepare datasets

Preparing datasets is also necessary for training. See section [Prepare datasets](#prepare-datasets) above for details.

**Note**:
Currently, the config files under `configs/cityscapes` use COCO pre-trained weights to initialize.
If your network connection is slow or unavailable, it's advisable to download existing models before beginning training to avoid errors.

## Learning rate auto scaling

**Important**: The default learning rate in config files is for 8 GPUs and 2 sample per GPU (batch size = 8 * 2 = 16). And it had been set to `auto_scale_lr.base_batch_size` in `config/_base_/schedules/schedule_1x.py`. The learning rate will be automatically scaled based on the value at a batch size of 16. Meanwhile, to avoid affecting other codebases that use mmdet, the default setting for the `auto_scale_lr.enable` flag is `False`.

If you want to enable this feature, you need to add argument `--auto-scale-lr`. And you need to check the config name which you want to use before you process the command, because the config name indicates the default batch size.

By default, it is `8 x 2 = 16 batch size`, like `faster_rcnn_r50_caffe_fpn_90k_coco.py` or `pisa_faster_rcnn_x101_32x4d_fpn_1x_coco.py`. In other cases, you will see the config file name have `_NxM_` in dictating, like `cornernet_hourglass104_mstest_32x3_210e_coco.py` which batch size is `32 x 3 = 96`, or `scnet_x101_64x4d_fpn_8x1_20e_coco.py` which batch size is `8 x 1 = 8`.

**Please remember to check the bottom of the specific config file you want to use, it will have `auto_scale_lr.base_batch_size` if the batch size is not `16`. If you can't find those values, check the config file which in `_base_=[xxx]` and you will find it. Please do not modify its values if you want to automatically scale the LR.**

The basic usage of learning rate auto scaling is as follows.

```shell
python tools/train.py \
    --auto-scale-lr \
    [optional arguments]
```

```shell
```

If you enabled this feature, the learning rate will be automatically scaled according to the number of GPUs on the machine and the batch size of training. See [linear scaling rule](https://arxiv.org/abs/1706.02677) for details. For example, If there are 4 GPUs and 2 pictures on each GPU, `lr = 0.01`, then if there are 16 GPUs and 4 pictures on each GPU, it will automatically scale to `lr = 0.08`.

If you don't want to use it, you need to calculate the learning rate according to the [linear scaling rule](https://arxiv.org/abs/1706.02677) manually then change `optimizer.lr` in specific config file.

## Training on a single GPU

We provide `tools/train.py` to launch training jobs on a single GPU.
The basic usage is as follows.

```shell
```

```shell
```

During training, log files and checkpoints will be saved to the working directory, which is specified by `work_dir` in the config file or via CLI argument `--work-dir`.

By default, the model is evaluated on the validation set every epoch, the evaluation interval can be specified in the config file as shown below.

```python
# evaluate the model every 12 epochs.
train_cfg = dict(val_interval=12)
```

This tool accepts several optional arguments, including:

- `--work-dir ${WORK_DIR}`: Override the working directory.
- `--resume`: resume from the latest checkpoint in the work_dir automatically.
- `--resume ${CHECKPOINT_FILE}`: resume from the specific checkpoint.
- `--cfg-options 'Key=value'`: Overrides other settings in the used config.

**Note:**

There is a difference between `resume` and `load-from`:

**Note:**

`resume` loads both the weights of the model and the state of the optimizer, and it inherits the iteration number from the specified checkpoint, so training does not start again from scratch. `load-from`, on the other hand, only loads the weights of the model, and its training starts from scratch. It is often used for fine-tuning a model. `load-from` needs to be written in the config file, while `resume` is passed as a command line argument.

## Training on CPU

The process of training on the CPU is consistent with single GPU training. We just need to disable GPUs before the training process.

```shell
```

And then run the script [above](#training-on-a-single-GPU).

**Note**:

```shell
```

**Note**:

We do not recommend users to use the CPU for training because it is too slow. We support this feature to allow users to debug on machines without GPU for convenience.

## Training on multiple GPUs

We provide `tools/dist_train.sh` to launch training on multiple GPUs.

```shell
bash ./tools/dist_train.sh \
```

Optional arguments remain the same as stated [above](#training-on-a-single-GPU).

### Launch multiple jobs simultaneously

If you would like to launch multiple jobs on a single machine, e.g., 2 jobs of 4-GPU training on a machine with 8 GPUs,

If you use `dist_train.sh` to launch training jobs, you can set the port in the commands.

```shell

```shell
```

If you launch with multiple machines simply connected with ethernet, you can simply run the following commands:

```shell
```

```shell
```

[Slurm](https://slurm.schedmd.com/) is a good job scheduling system for computing clusters.

On a cluster managed by Slurm, you can use `slurm_train.sh` to spawn training jobs. It supports both single-node and multi-node training.

```shell
```

Below is an example of using 16 GPUs to train Mask R-CNN on a Slurm partition named _dev_, and set the work-dir to some shared file systems.

```shell
GPUS=16 ./tools/slurm_train.sh dev mask_r50_1x configs/mask-rcnn_r50_fpn_1x_coco.py /nfs/xxxx/mask_rcnn_r50_fpn_1x
```

You can check [the source code](../../../tools/slurm_train.sh) to review full arguments and environment variables.

When using Slurm, the port option needs to be set in one of the following ways:

1. Set the port through `--options`. This is more recommended since it does not change the original configs.

   ```shell
   CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config1.py ${WORK_DIR} --cfg-options 'dist_params.port=29500'
   CUDA_VISIBLE_DEVICES=4,5,6,7 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config2.py ${WORK_DIR} --cfg-options 'dist_params.port=29501'
   ```

2. Modify the config files to set different communication ports.

   In `config1.py`, set

   ```python
   dist_params = dict(backend='nccl', port=29500)
   ```

   In `config2.py`, set

   ```python
   dist_params = dict(backend='nccl', port=29501)
   ```

   Then you can launch two jobs with `config1.py` and `config2.py`.

   ```shell

```

   ```python
   ```

   ```shell
   CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config1.py ${WORK_DIR}
   CUDA_VISIBLE_DEVICES=4,5,6,7 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config2.py ${WORK_DIR}
   ```

# Train with customized datasets

In this part, you will know how to train predefined models with customized datasets and then test it. We use the [balloon dataset](https://github.com/matterport/Mask_RCNN/tree/master/samples/balloon) as an example to describe the whole process.

3. Train, test, and infer models on the customized dataset.

There are three ways to support a new dataset in MMDetection:

1. Reorganize the dataset into COCO format.

2. Reorganize the dataset into a middle format.
3. Implement a new dataset.

Usually, we recommend using the first two methods which are usually easier than the third.

In this note, we give an example of converting the data into COCO format.

**Note**: Datasets and metrics have been decoupled except CityScapes since MMDetection 3.0. Therefore, users can use any kind of evaluation metrics for any format of datasets during validation. For example: evaluate on COCO dataset with VOC metric, or evaluate on OpenImages dataset with both VOC and COCO metrics.

### COCO annotation format

The necessary keys of COCO format for instance segmentation are as below, for the complete details, please refer [here](https://cocodataset.org/#format-data).

```json
{
    "images": [image],
    "annotations": [annotation],
    "categories": [category]
}

image = {

```json
{
}

image = {
    "id": int,
    "width": int,
    "height": int,
    "file_name": str,
}

annotation = {
    "id": int,
    "image_id": int,
    "category_id": int,
    "segmentation": RLE or [polygon],
    "area": float,
    "bbox": [x,y,width,height], # (x, y) are the coordinates of the upper left corner of the bbox
    "iscrowd": 0 or 1,
}

categories = [{
    "id": int,
    "name": str,
    "supercategory": str,
}]
```

Assume we use the balloon dataset.
After downloading the data, we need to implement a function to convert the annotation format into the COCO format. Then we can use implemented `CocoDataset` to load the data and perform training and evaluation.

If you take a look at the dataset, you will find the dataset format is as below:

```json
{'base64_img_data': '',
 'file_attributes': {},
 'filename': '34020010494_e5cb88e1c4_k.jpg',
 'fileref': '',

```json
 'regions': {'0': {'region_attributes': {},
   'shape_attributes': {'all_points_x': [1020,
     1000,
     994,
     1003,
     1023,
     1050,
     1089,
     1134,
     1190,
     1265,
     1321,
     1361,
     1403,
     1428,
     1442,
     1445,
     1441,
     1427,
     1400,
     1361,
     1316,
     1269,
     1228,
     1198,
     1207,
     1210,
     1190,
     1177,
     1172,
     1174,
     1170,
     1153,
     1127,
     1104,
     1061,
     1032,
     1020],
    'all_points_y': [963,
     899,
     841,
     787,
     738,
     700,
     663,
     638,
     621,
     619,
     643,
     672,
     720,
     765,
     800,
     860,
     896,
     942,
     990,
     1035,
     1079,
     1112,
     1129,
     1134,
     1144,
     1153,
     1166,
     1166,

619,
     643,
     672,
     720,
     765,
     800,
     860,
     896,
     942,
     990,
     1035,
     1079,
     1112,
     1129,
     1134,
     1144,
     1153,
     1166,
     1166,
     1150,
     1136,
     1129,
     1122,
     1112,
     1084,
     1037,
     989,
     963],
    'name': 'polygon'}}},
 'size': 1115004}
```

The annotation is a JSON file where each key indicates an image's all annotations.
The code to convert the balloon dataset into coco format is as below.

```python

from mmengine.fileio import dump, load
from mmengine.utils import track_iter_progress

def convert_balloon_to_coco(ann_file, out_file, image_prefix):
    data_infos = load(ann_file)

    annotations = []
    images = []
    obj_count = 0
    for idx, v in enumerate(track_iter_progress(data_infos.values())):
        filename = v['filename']
        img_path = osp.join(image_prefix, filename)
        height, width = mmcv.imread(img_path).shape[:2]

filename = v['filename']

        images.append(
            dict(id=idx, file_name=filename, height=height, width=width))

        for _, obj in v['regions'].items():
            assert not obj['region_attributes']
            obj = obj['shape_attributes']
            px = obj['all_points_x']
            py = obj['all_points_y']
            poly = [(x + 0.5, y + 0.5) for x, y in zip(px, py)]
            poly = [p for x in poly for p in x]

            x_min, y_min, x_max, y_max = (min(px), min(py), max(px), max(py))

            data_anno = dict(
                image_id=idx,
                id=obj_count,
                category_id=0,
                bbox=[x_min, y_min, x_max - x_min, y_max - y_min],
                area=(x_max - x_min) * (y_max - y_min),
                segmentation=[poly],
                iscrowd=0)
            annotations.append(data_anno)

area=(x_max - x_min) * (y_max - y_min),
            obj_count += 1

    coco_format_json = dict(
        images=images,
        annotations=annotations,
        categories=[{
            'id': 0,
            'name': 'balloon'
        }])
    dump(coco_format_json, out_file)

    convert_balloon_to_coco(ann_file='data/balloon/train/via_region_data.json',
                            out_file='data/balloon/train/annotation_coco.json',
                            image_prefix='data/balloon/train')
    convert_balloon_to_coco(ann_file='data/balloon/val/via_region_data.json',
                            out_file='data/balloon/val/annotation_coco.json',
                            image_prefix='data/balloon/val')

```

out_file='data/balloon/val/annotation_coco.json',

```

Using the function above, users can successfully convert the annotation file into json format, then we can use `CocoDataset` to train and evaluate the model with `CocoMetric`.

The second step is to prepare a config thus the dataset could be successfully loaded. Assume that we want to use Mask R-CNN with FPN, the config to train the detector on balloon dataset is as below. Assume the config is under directory `configs/balloon/` and named as `mask-rcnn_r50-caffe_fpn_ms-poly-1x_balloon.py`, the config is as below. Please refer [Learn about Configs — MMDetection 3.0.0 documentation](https://mmdetection.readthedocs.io/en/latest/user_guides/config.html) to get detailed information about config files.

```python
# The new config inherits a base config to highlight the necessary modification

```python
_base_ = '../mask_rcnn/mask-rcnn_r50-caffe_fpn_ms-poly-1x_coco.py'

# We also need to change the num_classes in head to match the dataset's annotation
    roi_head=dict(
        bbox_head=dict(num_classes=1), mask_head=dict(num_classes=1)))

# Modify dataset related settings
data_root = 'data/balloon/'
metainfo = {
    'classes': ('balloon', ),
    'palette': [
        (220, 20, 60),
    ]
}
        ann_file='train/annotation_coco.json',
        data_prefix=dict(img='train/')))
        ann_file='val/annotation_coco.json',
        data_prefix=dict(img='val/')))

# Modify metric related settings

metainfo=metainfo,

val_evaluator = dict(ann_file=data_root + 'val/annotation_coco.json')

# We can use the pre-trained Mask RCNN model to obtain higher performance
load_from = 'https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco_bbox_mAP-0.408__segm_mAP-0.37_20200504_163245-42aa3d00.pth'

```

```shell
python tools/train.py configs/balloon/mask-rcnn_r50-caffe_fpn_ms-poly-1x_balloon.py
```

For more detailed usages, please refer to the [training guide](https://mmdetection.readthedocs.io/en/latest/user_guides/train.html#train-predefined-models-on-standard-datasets).

```shell
python tools/test.py configs/balloon/mask-rcnn_r50-caffe_fpn_ms-poly-1x_balloon.py work_dirs/mask-rcnn_r50-caffe_fpn_ms-poly-1x_balloon/epoch_12.pth
```

For more detailed usages, please refer to the [testing guide](https://mmdetection.readthedocs.io/en/latest/user_guides/test.html).

# Preparing NTIRE21 decompression Dataset

```bibtex
@inproceedings{yang2021dataset,
  title={{NTIRE 2021} Challenge on Quality Enhancement of Compressed Video: Dataset and Study},
  author={Ren Yang and Radu Timofte},
  booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops},
}

```

The test datasets can be download from it's [Homepage](https://github.com/RenYang-home/NTIRE21_VEnh).

Please follows the tutorials of the [Homepage](https://github.com/RenYang-home/NTIRE21_VEnh) to generate datasets.

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── NTIRE21_decompression_track1
|   |   ├── GT
|   |   |   ├── 001
|   |   |   |   ├── 001.png
|   |   |   |   ├── ...
|   |   |   ├── ...
|   |   |   ├── 010
|   |   ├── LQ
|   |   |   ├── 001
|   |   |   ├── ...
|   |   |   ├── 010
|   ├── NTIRE21_decompression_track2

|   |   |   ├── 010
|   |   ├── LQ
|   |   |   ├── 001
|   |   |   ├── ...
|   |   |   ├── 010
|   |   ├── GT
|   |   ├── LQ
|   ├── NTIRE21_decompression_track3
|   |   ├── GT
|   |   ├── LQ
```

# t-Distributed Stochastic Neighbor Embedding (t-SNE) Visualization

## Introduction of the t-SNE visualization tool

MMPretrain provides `tools/visualization/vis_tsne.py` tool to visualize the feature embeddings of images by t-SNE. Please install `sklearn` to calculate t-SNE by `pip install scikit-learn`.

**Command**：

```bash
python tools/visualization/vis_tsne.py \
    CONFIG \
    [--checkpoint CHECKPOINT] \
    [--work-dir WORK_DIR] \
    [--test-cfg TEST_CFG] \
    [--vis-stage {backbone,neck,pre_logits}]
    [--class-idx ${CLASS_IDX} [CLASS_IDX ...]]
    [--max-num-class MAX_NUM_CLASS]
    [--max-num-samples MAX_NUM_SAMPLES]
    [--cfg-options CFG_OPTIONS [CFG_OPTIONS ...]]
    [--device DEVICE]
    [--legend]
    [--show]
    [--n-components N_COMPONENTS]
    [--perplexity PERPLEXITY]
    [--early-exaggeration EARLY_EXAGGERATION]
    [--learning-rate LEARNING_RATE]
    [--n-iter N_ITER]
    [--n-iter-without-progress N_ITER_WITHOUT_PROGRESS]
    [--init INIT]
```

[--early-exaggeration EARLY_EXAGGERATION]
```

- `CONFIG`: The path of t-SNE config file.
- `--checkpoint CHECKPOINT`: The path of the checkpoint file.
- `--work-dir WORK_DIR`: The directory to save logs and visualization images.
- `--test-cfg TEST_CFG`: The path of t-SNE config file to load config of test dataloader.
- `--vis-stage {backbone,neck,pre_logits}`: The visualization stage of the model.
- `--class-idx CLASS_IDX [CLASS_IDX ...]`: The categories used to calculate t-SNE.
- `--max-num-class MAX_NUM_CLASS`: The first N categories to apply t-SNE algorithms. Defaults to 20.
- `--max-num-samples MAX_NUM_SAMPLES`: The maximum number of samples per category. Higher number need longer time to calculate. Defaults to 100.

- `--cfg-options CFG_OPTIONS [CFG_OPTIONS ...]`: override some settings in the used config, the key-value pair in xxx=yyy format will be merged into config file. If the value to be overwritten is a list, it should be like key="[a,b]" or key=a,b It also allows nested list/tuple values, e.g. key="[(a,b),(c,d)]" Note that the quotation marks are necessary and that no white space is allowed.
- `--device DEVICE`: Device used for inference.
- `--legend`: Show the legend of all categories.
- `--show`: Display the result in a graphical window.
- `--n-components N_COMPONENTS`: The dimension of results.
- `--perplexity PERPLEXITY`: The perplexity is related to the number of nearest neighbors that is used in other manifold learning algorithms.

- `--early-exaggeration EARLY_EXAGGERATION`: Controls how tight natural clusters in the original space are in the embedded space and how much space will be between them.
- `--learning-rate LEARNING_RATE`: The learning rate for t-SNE is usually in the range[10.0, 1000.0]. If the learning rate is too high, the data may looklike a ball with any point approximately equidistant from its nearestneighbours. If the learning rate is too low, most points may lookcompressed in a dense cloud with few outliers.
- `--n-iter N_ITER`: Maximum number of iterations for the optimization. Should be at least 250.
- `--n-iter-without-progress N_ITER_WITHOUT_PROGRESS`: Maximum number of iterations without progress before we abort the optimization.
- `--init INIT`: The init method.

## How to visualize the t-SNE of a image classifier (such as ResNet)

Here are two examples of running t-SNE visualization on ResNet-18 and ResNet-50 models, trained on CIFAR-10 dataset:

```shell
    configs/resnet/resnet18_8xb16_cifar10.py \
    --checkpoint https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_b16x8_cifar10_20210528-bd6371c8.pth

    configs/resnet/resnet50_8xb16_cifar10.py \
    --checkpoint https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_b16x8_cifar10_20210528-f54bfad9.pth
```

| ResNet-18                                                                                            | ResNet-50                                                                                            |

| ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| <div align=center><img src='https://user-images.githubusercontent.com/42371271/236410521-c4d087da-d16f-48ad-b951-c74d10c68f33.png' height="auto" width="auto" ></div> | <div align=center><img src='https://user-images.githubusercontent.com/42371271/236411844-c97dc514-dad0-401e-ba8f-307d0a385b4e.png' height="auto" width="auto" ></div> |

## How to visualize the t-SNE of a self-supervised model (such as MAE)

Here is an example of running t-SNE visualization on MAE-ViT-base model, trained on ImageNet dataset. The input data is from ImageNet validation set. MAE and some self-supervised pre-training algorithms do not have test_dataloader information. When analyzing such self-supervised algorithms, you need to add test_dataloader information in the config, or you can use '--test-cfg' argument to specify a config file.

```shell
    configs/mae/mae_vit-base-p16_8xb512-amp-coslr-800e_in1k.py \
    --checkpoint https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-800e_in1k/mae_vit-base-p16_8xb512-coslr-800e-fp16_in1k_20220825-5d81fbc4.pth \
    --test-cfg configs/_base_/datasets/imagenet_bs32.py
```

--test-cfg configs/_base_/datasets/imagenet_bs32.py
```

| MAE-ViT-base                                                                                                                                                  |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <div align=center><img src='https://github.com/open-mmlab/mmpretrain/assets/42371271/ee576c0c-abef-43d1-8866-24a5f5fd0cf6' height="auto" width="auto" ></div> |

  <img src="https://github.com/open-mmlab/mmaction2/raw/main/resources/mmaction2_logo.png" width="600"/>
    <sup>
      </a>
    </sup>
    <sup>
      </a>
    </sup>
  </div>

[![Documentation](https://readthedocs.org/projects/mmaction2/badge/?version=latest)](https://mmaction2.readthedocs.io/en/latest/)
[![actions](https://github.com/open-mmlab/mmaction2/workflows/build/badge.svg)](https://github.com/open-mmlab/mmaction2/actions)
[![codecov](https://codecov.io/gh/open-mmlab/mmaction2/branch/main/graph/badge.svg)](https://codecov.io/gh/open-mmlab/mmaction2)

[![PyPI](https://img.shields.io/pypi/v/mmaction2)](https://pypi.org/project/mmaction2/)
[![LICENSE](https://img.shields.io/github/license/open-mmlab/mmaction2.svg)](https://github.com/open-mmlab/mmaction2/blob/main/LICENSE)
[![Average time to resolve an issue](https://isitmaintained.com/badge/resolution/open-mmlab/mmaction2.svg)](https://github.com/open-mmlab/mmaction2/issues)
[![Percentage of issues still open](https://isitmaintained.com/badge/open/open-mmlab/mmaction2.svg)](https://github.com/open-mmlab/mmaction2/issues)

[📘中文文档](https://mmaction2.readthedocs.io/zh_CN/latest/index.html) |
[🛠️安装指南](https://mmaction2.readthedocs.io/zh_CN/get_started/installation.html) |
[👀模型库](https://mmaction2.readthedocs.io/zh_CN/latest/modelzoo_statistics.html) |
[🆕更新日志](https://mmaction2.readthedocs.io/en/latest/notes/changelog.html) |

[🚀进行中项目](https://github.com/open-mmlab/mmaction2/projects) |
[🤔报告问题](https://github.com/open-mmlab/mmaction2/issues/new/choose)

</div>

</div>

[English](/README.md) | 简体中文

## 📄 目录

- [📄 目录](#-目录)
- [🥳 🚀 最新进展](#--最新进展-)
- [📖 简介](#-简介-)
- [🎁 主要功能](#-主要功能-)
- [🛠️ 安装](#️-安装-)
- [👀 模型库](#-模型库-)
- [👨‍🏫 新手入门](#-新手入门-)
- [🎫 许可证](#-许可证-)
- [🖊️ 引用](#️-引用-)
- [🙌 参与贡献](#-参与贡献-)
- [🤝 致谢](#-致谢-)

- [🏗️ OpenMMLab 的其他项目](#️-openmmlab-的其他项目-)
- [❤️ 欢迎加入 OpenMMLab 社区](#️-欢迎加入-openmmlab-社区-)

## 🥳 🚀 最新进展 [🔝](#-table-of-contents)

**默认分支已经从 `master` （当前的`0.x`） 切换到 `main`（之前的 `1.x`），我们建议用户更新至最新版本，其支持更多模型，更强的预训练权重，以及更简洁的代码实现。详情请参阅[迁移指南](https://mmaction2.readthedocs.io/zh_cn/latest/migration.html)**

**Release (2023.07.04)**: v1.1.0 支持以下新功能:

- 支持基于 CLIP 的多模态模型: ActionCLIP(Arxiv'2021) 和 CLIP4clip(ArXiv'2022)
- 支持丰富的 project: 手势识别, 时空行为检测 tutorial, 以及基于 [MMRazor](https://github.com/open-mmlab/mmrazor) 的知识蒸馏
- 支持 HACS-segments 数据集(ICCV'2019), MultiSports 数据集(ICCV'2021), Kinetics-710 数据集(Arxiv'2022)
- 支持 VideoMAE V2(CVPR'2023), VideoMAE(NeurIPS'2022) 支持时空行为检测任务
- 支持 TCANet(CVPR'2021)

- 支持 [纯 Python 风格的配置文件](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#a-pure-python-style-configuration-file-beta) 和使用 MIM 一键下载数据集

## 📖 简介 [🔝](#-table-of-contents)

MMAction2 是一款基于 PyTorch 开发的行为识别开源工具包，是 [open-mmlab](https://github.com/open-mmlab)  项目的一个子项目。

  <img src="https://github.com/open-mmlab/mmaction2/raw/main/resources/mmaction2_overview.gif" width="380px">
  <img src="https://user-images.githubusercontent.com/34324155/123989146-2ecae680-d9fb-11eb-916b-b9db5563a9e5.gif" width="380px">
  <p style="font-size:1.5vw;"> Kinetics-400 数据集行为识别结果（左） 和 NTU-RGB+D-120 数据集基于骨架的行为识别结果（右）</p>
</div>

  <img src="https://user-images.githubusercontent.com/30782254/155710881-bb26863e-fcb4-458e-b0c4-33cd79f96901.gif" width="580px"/><br>

</div>

    <p style="font-size:1.5vw;">Kinetics-400 数据集基于骨骼点的时空行为检测及视频行为识别结果</p>
</div>
  <img src="https://github.com/open-mmlab/mmaction2/raw/main/resources/spatio-temporal-det.gif" width="800px"/><br>
    <p style="font-size:1.5vw;">AVA-2.1 数据集时空行为检测结果</p>
</div>

## 🎁 主要功能 [🔝](#-table-of-contents)

- **模块化设计**： 我们将视频理解框架拆分成了不同模块，用户可以很方便地通过组合不同的模块来构建出自定义的视频理解框架。

- **支持五种主要的视频理解任务**： MMAction2 为视频理解任务实现了多种多样的算法，包括行为识别，时序动作定位，时空动作检测，基于骨骼点的行为识别，以及视频检索。

- **详尽的单元测试和文档**：我们提供了详尽的文档和 API 参考手册，以及单元测试。

## 🛠️ 安装 [🔝](#-table-of-contents)

MMAction2依赖于 [PyTorch](https://pytorch.org/)，[MMCV](https://github.com/open-mmlab/mmcv)，[MMEngine](https://github.com/open-mmlab/mmengine)，[MMDetection](https://github.com/open-mmlab/mmdetection) （可选）和 [MMPose](https://github.com/open-mmlab/mmpose) （可选）

具体步骤请参考 [安装文档](https://mmaction2.readthedocs.io/zh_cn/latest/get_started/installation.html)。

<details close>
<summary>快速安装</summary>

```shell
conda install pytorch torchvision -c pytorch  # 该命令将自动安装最新版的 PyTorch 和 cudatoolkit，请确认此是否匹配你的当前环境。
mim install mmcv
mim install mmdet  # 可选
mim install mmpose  # 可选
git clone https://github.com/open-mmlab/mmaction2.git
cd mmaction2
pip install -v -e .
```

## 👀 模型库 [🔝](#-table-of-contents)

结果及模型位于[模型库](https://mmaction2.readthedocs.io/zh_cn/latest/modelzoo_statistics.html)

<summary>模型支持</summary>

<table style="margin-left:auto;margin-right:auto;font-size:1.3vw;padding:3px 5px;text-align:center;vertical-align:center;">
  <tr>
    <td colspan="5" style="font-weight:bold;">行为识别</td>
  </tr>
  <tr>

<tr>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/c3d/README.md">C3D</a> (CVPR'2014)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/tsn/README.md">TSN</a> (ECCV'2016)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/i3d/README.md">I3D</a> (CVPR'2017)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/c2d/README.md">C2D</a> (CVPR'2018)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/i3d/README.md">I3D Non-Local</a> (CVPR'2018)</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/r2plus1d/README.md">R(2+1)D</a> (CVPR'2018)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/trn/README.md">TRN</a> (ECCV'2018)</td>

<td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/trn/README.md">TRN</a> (ECCV'2018)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/tsm/README.md">TSM</a> (ICCV'2019)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/tsm/README.md">TSM Non-Local</a> (ICCV'2019)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/slowonly/README.md">SlowOnly</a> (ICCV'2019)</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/slowfast/README.md">SlowFast</a> (ICCV'2019)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/csn/README.md">CSN</a> (ICCV'2019)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/tin/README.md">TIN</a> (AAAI'2020)</td>

<td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/tin/README.md">TIN</a> (AAAI'2020)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/tpn/README.md">TPN</a> (CVPR'2020)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/x3d/README.md">X3D</a> (CVPR'2020)</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition_audio/resnet/README.md">MultiModality: Audio</a> (ArXiv'2020)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/tanet/README.md">TANet</a> (ArXiv'2020)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/timesformer/README.md">TimeSformer</a> (ICML'2021)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/projects/actionclip/README.md">ActionCLIP</a> (ArXiv'2021)</td>

<td><a href="https://github.com/open-mmlab/mmaction2/blob/main/projects/actionclip/README.md">ActionCLIP</a> (ArXiv'2021)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/swin/README.md">VideoSwin</a> (CVPR'2022)</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/videomae/README.md">VideoMAE</a> (NeurIPS'2022)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/mvit/README.md">MViT V2</a> (CVPR'2022)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/uniformer/README.md">UniFormer V1</a> (ICLR'2022)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/uniformerv2/README.md">UniFormer V2</a> (Arxiv'2022)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/videomaev2/README.md">VideoMAE V2</a> (CVPR'2023)</td>
  </tr>
  <tr>

<td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/videomaev2/README.md">VideoMAE V2</a> (CVPR'2023)</td>
  </tr>
  <tr>
    <td colspan="5" style="font-weight:bold;">时序动作定位</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/localization/bsn/README.md">BSN</a> (ECCV'2018)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/localization/bmn/README.md">BMN</a> (ICCV'2019)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/localization/tcanet/README.md">TCANet</a> (CVPR'2021)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td colspan="5" style="font-weight:bold;">时空行为检测</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/detection/acrn/README.md">ACRN</a> (ECCV'2018)</td>

<td colspan="5" style="font-weight:bold;">时空行为检测</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/detection/slowonly/README.md">SlowOnly+Fast R-CNN</a> (ICCV'2019)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/detection/slowfast/README.md">SlowFast+Fast R-CNN</a> (ICCV'2019)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/detection/lfb/README.md">LFB</a> (CVPR'2019)</td>
  </tr>
  <tr>
    <td colspan="5" style="font-weight:bold;">基于骨骼点的行为识别</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/skeleton/stgcn/README.md">ST-GCN</a> (AAAI'2018)</td>

</tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/skeleton/2s-agcn/README.md">2s-AGCN</a> (CVPR'2019)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/skeleton/posec3d/README.md">PoseC3D</a> (CVPR'2022)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/skeleton/stgcnpp/README.md">STGCN++</a> (ArXiv'2022)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/projects/ctrgcn/README.md">CTRGCN</a> (CVPR'2021)</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/projects/msg3d/README.md">MSG3D</a> (CVPR'2020)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td colspan="5" style="font-weight:bold;">视频检索</td>
  </tr>
  <tr>

<td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/retrieval/clip4clip/README.md">CLIP4Clip</a> (ArXiv'2022)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>

</table>

<summary>数据集支持</summary>

  <tr>
    <td colspan="4" style="font-weight:bold;">行为识别</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/hmdb51/README.md">HMDB51</a> (<a href="https://serre-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/">官网</a>) (ICCV'2011)</td>

<td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/ucf101/README.md">UCF101</a> (<a href="https://www.crcv.ucf.edu/research/data-sets/ucf101/">官网</a>) (CRCV-IR-12-01)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/activitynet/README.md">ActivityNet</a> (<a href="http://activity-net.org/">官网</a>) (CVPR'2015)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/kinetics/README.md">Kinetics-[400/600/700]</a> (<a href="https://deepmind.com/research/open-source/kinetics/">官网</a>) (CVPR'2017)</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/sthv1/README.md">SthV1</a>  (ICCV'2017)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/sthv2/README.md">SthV2</a> (<a href="https://developer.qualcomm.com/software/ai-datasets/something-something">官网</a>) (ICCV'2017)</td>

<td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/diving48/README.md">Diving48</a> (<a href="http://www.svcl.ucsd.edu/projects/resound/dataset.html">官网</a>) (ECCV'2018)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/jester/README.md">Jester</a> (<a href="https://developer.qualcomm.com/software/ai-datasets/jester">官网</a>) (ICCV'2019)</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/mit/README.md">Moments in Time</a> (<a href="http://moments.csail.mit.edu/">官网</a>) (TPAMI'2019)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/mmit/README.md">Multi-Moments in Time</a> (<a href="http://moments.csail.mit.edu/challenge_iccv_2019.html">官网</a>) (ArXiv'2019)</td>

<td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/hvu/README.md">HVU</a> (<a href="https://github.com/holistic-video-understanding/HVU-Dataset">官网</a>) (ECCV'2020)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/omnisource/README.md">OmniSource</a> (<a href="https://kennymckormick.github.io/omnisource/">官网</a>) (ECCV'2020)</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/gym/README.md">FineGYM</a> (<a href="https://sdolivia.github.io/FineGym/">官网</a>) (CVPR'2020)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/kinetics710/README.md">Kinetics-710</a> (<a href="https://arxiv.org/pdf/2211.09552.pdf">官网</a>) (Arxiv'2022)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td colspan="4" style="font-weight:bold;">时序动作定位</td>
  </tr>
  <tr>

<td></td>
    <td></td>
  </tr>
  <tr>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/thumos14/README.md">THUMOS14</a> (<a href="https://www.crcv.ucf.edu/THUMOS14/download.html">官网</a>) (THUMOS Challenge 2014)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/hacs/README.md">HACS</a> (<a href="https://github.com/hangzhaomit/HACS-dataset">官网</a>) (ICCV'2019)</td>
    <td></td>
  </tr>
  <tr>
    <td colspan="4" style="font-weight:bold;">时空行为检测</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/ucf101_24/README.md">UCF101-24*</a> (<a href="http://www.thumos.info/download.html">官网</a>) (CRCV-IR-12-01)</td>

<tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/jhmdb/README.md">JHMDB*</a> (<a href="http://jhmdb.is.tue.mpg.de/">官网</a>) (ICCV'2015)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/ava/README.md">AVA</a> (<a href="https://research.google.com/ava/index.html">官网</a>) (CVPR'2018)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/ava_kinetics/README.md">AVA-Kinetics</a> (<a href="https://research.google.com/ava/index.html">官网</a>) (Arxiv'2020)</td>
  </tr>
  <tr>
    <td colspan="4" style="font-weight:bold;">基于骨架的行为识别</td>
  </tr>
  <tr>

</tr>
  <tr>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/skeleton/README.md">PoseC3D-FineGYM</a> (<a href="https://kennymckormick.github.io/posec3d/">官网</a>) (ArXiv'2021)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/skeleton/README.md">PoseC3D-NTURGB+D</a> (<a href="https://kennymckormick.github.io/posec3d/">官网</a>) (ArXiv'2021)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/skeleton/README.md">PoseC3D-UCF101</a> (<a href="https://kennymckormick.github.io/posec3d/">官网</a>) (ArXiv'2021)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/skeleton/README.md">PoseC3D-HMDB51</a> (<a href="https://kennymckormick.github.io/posec3d/">官网</a>) (ArXiv'2021)</td>
  </tr>
  <tr>
    <td colspan="4" style="font-weight:bold;">视频检索</td>
  </tr>
  <tr>

</tr>
  <tr>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/video_retrieval/README.md">MSRVTT</a> (<a href="https://www.microsoft.com/en-us/research/publication/msr-vtt-a-large-video-description-dataset-for-bridging-video-and-language/">官网</a>) (CVPR'2016)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

## 👨‍🏫 新手入门 [🔝](#-table-of-contents)

我们提供了一系列简明的教程，帮助新用户轻松上手使用：

- [从 MMAction2 0.X 迁移](https://mmaction2.readthedocs.io/zh_cn/latest/migration.html)
- [学习配置相关知识](https://mmaction2.readthedocs.io/zh_cn/latest/user_guides/config.html)
- [准备数据集](https://mmaction2.readthedocs.io/zh_cn/latest/user_guides/prepare_dataset.html)
- [使用现有模型进行推理](https://mmaction2.readthedocs.io/zh_cn/latest/user_guides/inference.html)
- [训练与测试](https://mmaction2.readthedocs.io/zh_cn/latest/user_guides/train_test.html)

<summary>基于 MMAction2 的社区工作</summary>

- Video Swin Transformer. [\[paper\]](https://arxiv.org/abs/2106.13230)[\[github\]](https://github.com/SwinTransformer/Video-Swin-Transformer)
- Evidential Deep Learning for Open Set Action Recognition, ICCV 2021 **Oral**. [\[paper\]](https://arxiv.org/abs/2107.10161)[\[github\]](https://github.com/Cogito2012/DEAR)
- Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspective, ICCV 2021 **Oral**. [\[paper\]](https://arxiv.org/abs/2103.17263)[\[github\]](https://github.com/xvjiarui/VFS)

## 🎫 许可证 [🔝](#-table-of-contents)

本项目基于 [Apache 2.0 license](LICENSE) 发布。

## 🖊️ 引用 [🔝](#-table-of-contents)

如你发现本项目对你的研究有帮助，请参考如下 bibtex 引用 MMAction2。

```BibTeX
@misc{2020mmaction2,
    title={OpenMMLab's Next Generation Video Understanding Toolbox and Benchmark},
    author={MMAction2 Contributors},

```BibTeX
    howpublished = {\url{https://github.com/open-mmlab/mmaction2}},
    year={2020}
}
```

## 🙌 参与贡献 [🔝](#-table-of-contents)

我们感谢所有的贡献者为改进和提升 MMAction2 所作出的努力。请参考[贡献指南](https://github.com/open-mmlab/mmcv/blob/2.x/CONTRIBUTING.md)来了解参与项目贡献的相关指引。

## 🤝 致谢 [🔝](#-table-of-contents)

MMAction2 是一款由来自不同高校和企业的研发人员共同参与贡献的开源项目。我们感谢所有为项目提供算法复现和新功能支持的贡献者，以及提供宝贵反馈的用户。 我们希望此工具箱可以帮助大家来复现已有的方法和开发新的方法，从而为研究社区贡献力量。

## 🏗️ OpenMMLab 的其他项目 [🔝](#-table-of-contents)

- [MIM](https://github.com/open-mmlab/mim): MIM 是 OpenMMlab 项目、算法、模型的统一入口

## ❤️ 欢迎加入 OpenMMLab 社区 [🔝](#-table-of-contents)

扫描下方的二维码可关注 OpenMMLab 团队的 [知乎官方账号](https://www.zhihu.com/people/openmmlab)，加入 OpenMMLab 团队的 [官方交流 QQ 群](https://jq.qq.com/?_wv=1027&k=aCvMxdr3) 或联络 OpenMMLab 官方微信小助手

<img src="./resources/zhihu_qrcode.jpg" height="400"/> <img src="./resources/xiaozhushou_weixin_qrcode.jpeg" height="400"/>
</div>

# Train with customized models and standard datasets

In this note, you will know how to train, test and inference your own customized models under standard datasets. We use the cityscapes dataset to train a customized Cascade Mask R-CNN R50 model as an example to demonstrate the whole process, which using [`AugFPN`](https://github.com/Gus-Guo/AugFPN) to replace the default `FPN` as neck, and add `Rotate` or `TranslateX` as training-time auto augmentation.

1. Prepare the standard dataset
2. Prepare your own customized model
3. Prepare a config
4. Train, test, and inference models on the standard dataset.

## Prepare the standard dataset

In this note, as we use the standard cityscapes dataset as an example.

It is recommended to symlink the dataset root to `$MMDETECTION/data`.

```none
├── mmdet
├── tools
├── data
│   ├── coco

```none
├── mmdet
├── tools
├── data
│   ├── coco
│   │   │   ├── val
│   │   │   ├── val

```

Or you can set your dataset root through

```bash
export MMDET_DATASETS=$data_root
```

We will replace dataset root with `$MMDET_DATASETS`, so you don't have to modify the corresponding path in config files.

The cityscapes annotations have to be converted into the coco format using `tools/dataset_converters/cityscapes.py`:

```shell
python tools/dataset_converters/cityscapes.py ./data/cityscapes --nproc 8 --out-dir ./data/cityscapes/annotations
```

```shell
```

Currently, the config files in `cityscapes` use COCO pre-trained weights to initialize.
You could download the pre-trained models in advance if the network is unavailable or slow, otherwise, it would cause errors at the beginning of training.

## Prepare your own customized model

The second step is to use your own module or training setting. Assume that we want to implement a new neck called `AugFPN` to replace with the default `FPN` under the existing detector Cascade Mask R-CNN R50. The following implements `AugFPN` under MMDetection.

### 1. Define a new neck (e.g. AugFPN)

Firstly create a new file `mmdet/models/necks/augfpn.py`.

```python
from mmdet.registry import MODELS

class AugFPN(nn.Module):

                in_channels,

```python

                out_channels,
                num_outs,
                start_level=0,
                end_level=-1,
                add_extra_convs=False):
        pass

        # implementation is ignored
        pass
```

### 2. Import the module

You can either add the following line to `mmdet/models/necks/__init__.py`,

```python
from .augfpn import AugFPN
```

or alternatively add

```python
custom_imports = dict(
    imports=['mmdet.models.necks.augfpn'],
    allow_failed_imports=False)
```

to the config file and avoid modifying the original code.

### 3. Modify the config file

```python
neck=dict(
    type='AugFPN',
    in_channels=[256, 512, 1024, 2048],
    out_channels=256,
    num_outs=5)
```

```python
```

For more detailed usages about customizing your own models (e.g. implement a new backbone, head, loss, etc) and runtime training settings (e.g. define a new optimizer, use gradient clip, customize training schedules and hooks, etc), please refer to the guideline [Customize Models](../advanced_guides/customize_models.md) and [Customize Runtime Settings](../advanced_guides/customize_runtime.md) respectively.

The third step is to prepare a config for your own training setting. Assume that we want to add `AugFPN` and `Rotate` or `Translate` augmentation to existing Cascade Mask R-CNN R50 to train the cityscapes dataset, and assume the config is under directory `configs/cityscapes/` and named as `cascade-mask-rcnn_r50_augfpn_autoaug-10e_cityscapes.py`, the config is as below.

```python

```python
# The new config inherits the base configs to highlight the necessary modification
_base_ = [
    '../_base_/models/cascade-mask-rcnn_r50_fpn.py',
    '../_base_/datasets/cityscapes_instance.py', '../_base_/default_runtime.py'
]

    # set None to avoid loading ImageNet pre-trained backbone,
    # instead here we set `load_from` to load from COCO pre-trained detectors.
    backbone=dict(init_cfg=None),
    # replace neck from defaultly `FPN` to our new implemented module `AugFPN`
        type='AugFPN',
        in_channels=[256, 512, 1024, 2048],
        num_outs=5),
    # We also need to change the num_classes in head from 80 to 8, to match the
    # cityscapes dataset's annotation. This modification involves `bbox_head` and `mask_head`.
        bbox_head=[
            dict(
                type='Shared2FCBBoxHead',
                in_channels=256,
                fc_out_channels=1024,

roi_head=dict(
            dict(
                roi_feat_size=7,
                # change the number of classes from defaultly COCO to cityscapes
                num_classes=8,
                bbox_coder=dict(
                    type='DeltaXYWHBBoxCoder',
                    target_means=[0., 0., 0., 0.],
                    target_stds=[0.1, 0.1, 0.2, 0.2]),
                reg_class_agnostic=True,
                loss_cls=dict(
                    type='CrossEntropyLoss',
                    use_sigmoid=False,
                    loss_weight=1.0),
                loss_bbox=dict(type='SmoothL1Loss', beta=1.0,
                               loss_weight=1.0)),
            dict(

dict(
                    target_stds=[0.05, 0.05, 0.1, 0.1]),
            dict(

dict(
                    target_stds=[0.033, 0.033, 0.067, 0.067]),
                loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=1.0))
        ],
        mask_head=dict(
            type='FCNMaskHead',
            num_convs=4,
            in_channels=256,
            conv_out_channels=256,
            # change the number of classes from default COCO to cityscapes

type='FCNMaskHead',
            num_classes=8,
            loss_mask=dict(
                type='CrossEntropyLoss', use_mask=True, loss_weight=1.0))))

# over-write `train_pipeline` for new added `AutoAugment` training setting
    dict(
        type='AutoAugment',
        policies=[
            [dict(
                 type='Rotate',
                 level=5,
                 img_border_value=(124, 116, 104),
                 prob=0.5)
            ],
            [dict(type='Rotate', level=7, img_border_value=(124, 116, 104)),
             dict(
                 type='TranslateX',
                 level=5,
                 prob=0.5,
                 img_border_value=(124, 116, 104))

dict(
                 level=5,
                 prob=0.5,
            ],
        ]),
    dict(
        type='RandomResize',
        scale=[(2048, 800), (2048, 1024)],
    dict(type='PackDetInputs'),
]

# set batch_size per gpu, and set new training pipeline
    num_workers=3,
    # over-write `pipeline` with new training pipeline setting
    dataset=dict(pipeline=train_pipeline))

# Set optimizer

# Set customized learning policy
    dict(
    dict(
        begin=0,
        end=10,
        milestones=[8],

type='LinearLR', start_factor=0.001, by_epoch=False, begin=0, end=500),
    dict(
        begin=0,
        end=10,
]

# train, val, test loop config
train_cfg = dict(max_epochs=10, val_interval=1)

# We can use the COCO pre-trained Cascade Mask R-CNN R50 model for a more stable performance initialization
load_from = 'https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r50_fpn_1x_coco/cascade_mask_rcnn_r50_fpn_1x_coco_20200203-9d4dcb24.pth'
```

```shell
python tools/train.py configs/cityscapes/cascade-mask-rcnn_r50_augfpn_autoaug-10e_cityscapes.py
```

For more detailed usages, please refer to the [training guide](train.md).

```shell

```

```shell
python tools/test.py configs/cityscapes/cascade-mask-rcnn_r50_augfpn_autoaug-10e_cityscapes.py work_dirs/cascade-mask-rcnn_r50_augfpn_autoaug-10e_cityscapes/epoch_10.pth
```

For more detailed usages, please refer to the [testing guide](test.md).

# WIDER FACE

> [WIDER FACE: A Face Detection Benchmark](https://arxiv.org/abs/1511.06523)

Face detection is one of the most studied topics in the computer vision community. Much of the progresses have been made by the availability of face detection benchmark datasets. We show that there is a gap between current face detection performance and the real world requirements. To facilitate future face detection research, we introduce the WIDER FACE dataset, which is 10 times larger than existing datasets. The dataset contains rich annotations, including occlusions, poses, event categories, and face bounding boxes. Faces in the proposed dataset are extremely challenging due to large variations in scale, pose and occlusion, as shown in Fig. 1. Furthermore, we show that WIDER FACE dataset is an effective training source for face detection. We benchmark several representative detection systems, providing an overview of state-of-the-art performance and propose a solution to deal with large scale variation. Finally, we discuss common failure cases that worth to be further investigated.

<img src="https://user-images.githubusercontent.com/40661020/144000364-3320de79-34fc-40a6-938f-bb512f05a4bb.png" height="400"/>
</div>

To use the WIDER Face dataset you need to download it
and extract to the `data/WIDERFace` folder. Annotation in the VOC format
can be found in this [repo](https://github.com/sovrasov/wider-face-pascal-voc-annotations.git).
You should move the annotation files from `WIDER_train_annotations` and `WIDER_val_annotations` folders
to the `Annotation` folders inside the corresponding directories `WIDER_train` and `WIDER_val`.
Also annotation lists `val.txt` and `train.txt` should be copied to `data/WIDERFace` from `WIDER_train_annotations` and `WIDER_val_annotations`.
The directory should be like this:

```
├── mmdet
├── tools
├── data
│   ├── WIDERFace
│   │   ├── WIDER_train
│   |   │   ├──0--Parade
│   |   │   ├── ...
│   |   │   ├── Annotations
│   │   ├── WIDER_val

├── tools
├── data
│   |   │   ├── ...
│   |   │   ├── ...
│   │   ├── val.txt
│   │   ├── train.txt

```

After that you can train the SSD300 on WIDER by launching training with the `ssd300_wider_face.py` config or
create your own config based on the presented one.

```latex
@inproceedings{yang2016wider,
   Author = {Yang, Shuo and Luo, Ping and Loy, Chen Change and Tang, Xiaoou},
   Booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
   Title = {WIDER FACE: A Face Detection Benchmark},
   Year = {2016}
}
```

# GCA (AAAI'2020)

> [Natural Image Matting via Guided Contextual Attention](https://arxiv.org/abs/2001.04069)

> **Task**: Matting

Over the last few years, deep learning based approaches have achieved outstanding improvements in natural image matting. Many of these methods can generate visually plausible alpha estimations, but typically yield blurry structures or textures in the semitransparent area. This is due to the local ambiguity of transparent objects. One possible solution is to leverage the far-surrounding information to estimate the local opacity. Traditional affinity-based methods often suffer from the high computational complexity, which are not suitable for high resolution alpha estimation. Inspired by affinity-based method and the successes of contextual attention in inpainting, we develop a novel end-to-end approach for natural image matting with a guided contextual attention module, which is specifically designed for image matting. Guided contextual attention module directly propagates high-level opacity information globally based on the learned low-level affinity. The proposed method can mimic information flow of affinity-based methods and utilize rich features learned by deep neural networks simultaneously. Experiment results on Composition-1k testing set and this http URL benchmark dataset demonstrate that our method outperforms state-of-the-art approaches in natural image matting.

 <img src="https://user-images.githubusercontent.com/12726765/144176004-c9c26201-f8af-416a-9bea-ccd60bae7913.png" width="400"/>
</div >

|                              Model                               |    Dataset     |    SAD    |    MSE     |   GRAD    |   CONN    | Training Resources |                              Download                               |
| :--------------------------------------------------------------: | :------------: | :-------: | :--------: | :-------: | :-------: | :----------------: | :-----------------------------------------------------------------: |

|      [baseline (our)](./baseline_r34_4xb10-200k_comp1k.py)       | Composition-1K |   34.61   |   0.0083   |   16.21   |   32.12   |         4          | [model](https://download.openmmlab.com/mmediting/mattors/gca/baseline_r34_4x10_200k_comp1k_SAD-34.61_20220620-96f85d56.pth) \| [log](https://download.openmmlab.com/mmediting/mattors/gca/baseline_r34_4x10_200k_comp1k_SAD-34.61_20220620-96f85d56.log) |
|           [GCA (our)](./gca_r34_4xb10-200k_comp1k.py)            | Composition-1K | **33.38** | **0.0081** | **14.96** | **30.59** |         4          | [model](https://download.openmmlab.com/mmediting/mattors/gca/gca_r34_4x10_200k_comp1k_SAD-33.38_20220615-65595f39.pth) \| [log](https://download.openmmlab.com/mmediting/mattors/gca/gca_r34_4x10_200k_comp1k_SAD-33.38_20220615-65595f39.log) |

| [baseline (with DIM pipeline)](./baseline_r34_4xb10-dimaug-200k_comp1k.py) | Composition-1K |   49.95   |   0.0144   |   30.21   |   49.67   |         4          | [model](https://download.openmmlab.com/mmediting/mattors/gca/baseline_dimaug_r34_4x10_200k_comp1k_SAD-49.95_20200626_231612-535c9a11.pth) \| [log](https://download.openmmlab.com/mmediting/mattors/gca/baseline_dimaug_r34_4x10_200k_comp1k_20200626_231612.log.json) |
| [GCA (with DIM pipeline)](./gca_r34_4xb10-dimaug-200k_comp1k.py) | Composition-1K |   49.42   |   0.0129   |   28.07   |   49.47   |         4          | [model](https://download.openmmlab.com/mmediting/mattors/gca/gca_dimaug_r34_4x10_200k_comp1k_SAD-49.42_20200626_231422-8e9cc127.pth) \| [log](https://download.openmmlab.com/mmediting/mattors/gca/gca_dimaug_r34_4x10_200k_comp1k_20200626_231422.log.json) |

<!-- |                   baseline (paper)                    |   40.62   |   0.0106   |   21.53   |   38.43   |         -          |                                              -                                               |
|                      GCA (paper)                      |   35.28   |   0.0091   |   16.92   |   32.53   |         -          |                                              -                                               | -->

**Train**

<details>

```shell
# cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/gca/gca_r34_4xb10-200k_comp1k.py

python tools/train.py configs/gca/gca_r34_4xb10-200k_comp1k.py

./tools/dist_train.sh configs/gca/gca_r34_4xb10-200k_comp1k.py 8
```

```

**Test**

<details>

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/gca/gca_r34_4xb10-200k_comp1k.py https://download.openmmlab.com/mmediting/mattors/gca/gca_r34_4x10_200k_comp1k_SAD-33.38_20220615-65595f39.pth

python tools/test.py configs/gca/gca_r34_4xb10-200k_comp1k.py https://download.openmmlab.com/mmediting/mattors/gca/gca_r34_4x10_200k_comp1k_SAD-33.38_20220615-65595f39.pth

./tools/dist_test.sh configs/gca/gca_r34_4xb10-200k_comp1k.py https://download.openmmlab.com/mmediting/mattors/gca/gca_r34_4x10_200k_comp1k_SAD-33.38_20220615-65595f39.pth 8
```

```bibtex
@inproceedings{li2020natural,
  title={Natural Image Matting via Guided Contextual Attention},
  author={Li, Yaoyi and Lu, Hongtao},
  booktitle={Association for the Advancement of Artificial Intelligence (AAAI)},
}
```

# CenterNet

> [Objects as Points](https://arxiv.org/abs/1904.07850)

Detection identifies objects as axis-aligned boxes in an image. Most successful object detectors enumerate a nearly exhaustive list of potential object locations and classify each. This is wasteful, inefficient, and requires additional post-processing. In this paper, we take a different approach. We model an object as a single point --- the center point of its bounding box. Our detector uses keypoint estimation to find center points and regresses to all other object properties, such as size, 3D location, orientation, and even pose. Our center point based approach, CenterNet, is end-to-end differentiable, simpler, faster, and more accurate than corresponding bounding box based detectors. CenterNet achieves the best speed-accuracy trade-off on the MS COCO dataset, with 28.1% AP at 142 FPS, 37.4% AP at 52 FPS, and 45.1% AP with multi-scale testing at 1.4 FPS. We use the same approach to estimate 3D bounding box in the KITTI benchmark and human pose on the COCO keypoint dataset. Our method performs competitively with sophisticated multi-stage methods and runs in real-time.

<img src="https://user-images.githubusercontent.com/40661020/143873810-85ffa6e7-915b-46a4-9b8f-709e5d7700bb.png"/>
</div>

| Backbone  | DCN | Mem (GB) | Box AP | Flip box AP |                           Config                           |                                                                                                                                                                 Download                                                                                                                                                                 |

| :-------: | :-: | :------: | :----: | :---------: | :--------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ResNet-18 |  N  |   3.45   |  25.9  |    27.3     |    [config](./centernet_r18_8xb16-crop512-140e_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/centernet/centernet_resnet18_140e_coco/centernet_resnet18_140e_coco_20210705_093630-bb5b3bf7.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/centernet/centernet_resnet18_140e_coco/centernet_resnet18_140e_coco_20210705_093630.log.json)             |

| ResNet-18 |  Y  |   3.47   |  29.5  |    30.9     | [config](./centernet_r18-dcnv2_8xb16-crop512-140e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/centernet/centernet_resnet18_dcnv2_140e_coco/centernet_resnet18_dcnv2_140e_coco_20210702_155131-c8cd631f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/centernet/centernet_resnet18_dcnv2_140e_coco/centernet_resnet18_dcnv2_140e_coco_20210702_155131.log.json) |

Note:

- Flip box AP setting is single-scale and `flip=True`.
- Due to complex data enhancement, we find that the performance is unstable and may fluctuate by about 0.4 mAP. mAP 29.4 ~ 29.8 is acceptable in ResNet-18-DCNv2.
- Compared to the source code, we refer to [CenterNet-Better](https://github.com/FateScript/CenterNet-better), and make the following changes
  - fix wrong image mean and variance in image normalization to be compatible with the pre-trained backbone.
  - Use SGD rather than ADAM optimizer and add warmup and grad clip.

- fix wrong image mean and variance in image normalization to be compatible with the pre-trained backbone.
  - Use DistributedDataParallel as other models in MMDetection rather than using DataParallel.

## CenterNet Update

| Backbone  | Style | Lr schd | MS train | Mem (GB) | Box AP |                          Config                          |                                                                                                                                                                               Download                                                                                                                                                                               |

| :-------: | :---: | :-----: | :------: | :------: | :----: | :------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ResNet-50 | caffe |   1x    |   True   |   3.3    |  40.2  | [config](./centernet-update_r50-caffe_fpn_ms-1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/centernet/centernet-update_r50-caffe_fpn_ms-1x_coco/centernet-update_r50-caffe_fpn_ms-1x_coco_20230512_203845-8306baf2.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/centernet/centernet-update_r50-caffe_fpn_ms-1x_coco/centernet-update_r50-caffe_fpn_ms-1x_coco_20230512_203845.log.json) |

CenterNet Update from the paper of [Probabilistic two-stage detection](https://arxiv.org/abs/2103.07461). The author has updated CenterNet to greatly improve performance and convergence speed.
The [Details](https://github.com/xingyizhou/CenterNet2/blob/master/docs/MODEL_ZOO.md) are as follows:

- Using top-left-right-bottom box encoding and GIoU Loss
- Adding regression loss to the center 3x3 region
- Adding more positive pixels for the heatmap loss whose regression loss is small and is within the center3x3 region
- Using RetinaNet-style optimizer (SGD), learning rate rule (0.01 for each batch size 16), and schedule (12 epochs)
- Added FPN neck layers, and assigns objects to FPN levels based on a fixed size range.
- Using standard NMS instead of max pooling

Note: We found that the performance of the r50 model fluctuates greatly and sometimes it does not converge. If the model does not converge, you can try running it again or reduce the learning rate.

```latex

```latex
@article{zhou2019objects,
  title={Objects as Points},
  author={Zhou, Xingyi and Wang, Dequan and Kr{\"a}henb{\"u}hl, Philipp},
  booktitle={arXiv preprint arXiv:1904.07850},
}
```

# KFIoU

> [The KFIoU Loss for Rotated Object Detection](https://arxiv.org/pdf/2101.11952.pdf)

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/kfiou.png" width="800"/>
</div>

Differing from the well-developed horizontal object detection area whereby the computing-friendly IoU based loss is
readily adopted and well fits with the detection metrics. In contrast, rotation detectors often involve a more
complicated loss based on SkewIoU which is unfriendly to gradient-based training. In this paper, we argue that one
effective alternative is to devise an approximate loss who can achieve trend-level alignment with SkewIoU loss instead
of the strict value-level identity. Specifically, we model the objects as Gaussian distribution and adopt Kalman filter to
inherently mimic the mechanism of SkewIoU by its definition, and show its alignment with the SkewIoU at trend-level. This

is in contrast to recent Gaussian modeling based rotation detectors e.g. GWD, KLD that involves a human-specified
distribution distance metric which requires additional hyperparameter tuning. The resulting new loss called KFIoU is
easier to implement and works better compared with exact SkewIoU, thanks to its full differentiability and ability to
handle the non-overlapping cases. We further extend our technique to the 3-D case which also suffers from the same
issues as 2-D detection. Extensive results on various public datasets (2-D/3-D, aerial/text/face images) with different
base detectors show the effectiveness of our approach.

DOTA1.0

DOTA1.0

|         Backbone         |  mAP  | Angle | lr schd | Mem (GB) | Inf Time (fps) | Aug | Batch Size |                                                    Configs                                                     |                                                                                                                                                                                  Download                                                                                                                                                                                  |

| ResNet50 (1024,1024,200) | 64.55 |  oc   |   1x    |   3.38   |      15.7      |  -  |     2      | [rotated-retinanet-hbox-oc_r50_fpn_1x_dota](../rotated_retinanet/rotated-retinanet-hbox-oc_r50_fpn_1x_dota.py) |       [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_hbb_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_r50_fpn_1x_dota_oc-e8a7c7df.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_hbb_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_r50_fpn_1x_dota_oc_20220121_095315.log.json)       |

| ResNet50 (1024,1024,200) | 69.60 | le90  |   1x    |   3.38   |      15.1      |  -  |     2      |  [rotated-retinanet-hbox-le90_r50_fpn_kfiou_1x_dota](./rotated-retinanet-hbox-le90_r50_fpn_kfiou_1x_dota.py)   |   [model](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le90/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le90-03e02f75.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le90/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le90_20220209_173225.log.json)   |

| ResNet50 (1024,1024,200) | 69.76 |  oc   |   1x    |   3.39   |      15.6      |  -  |     2      |    [rotated-retinanet-hbox-oc_r50_fpn_kfiou_1x_dota](./rotated-retinanet-hbox-oc_r50_fpn_kfiou_1x_dota.py)     |       [model](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_oc-c00be030.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_oc_20220126_081643.log.json)       |

| ResNet50 (1024,1024,200) | 69.77 | le135 |   1x    |   3.38   |      15.3      |  -  |     2      | [rotated-retinanet-hbox-le135_r50_fpn_kfiou_1x_dota](./rotated-retinanet-hbox-le135_r50_fpn_kfiou_1x_dota.py)  | [model](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le135/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le135-0eaa4156.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le135/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le135_20220209_173257.log.json) |

|         Backbone         |  mAP  | Angle | lr schd | Mem (GB) | Inf Time (fps) | Aug | Batch Size |                                   Configs                                   |                                                                                                                                                  Download                                                                                                                                                  |
| :----------------------: | :---: | :---: | :-----: | :------: | :------------: | :-: | :--------: | :-------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

| ResNet50 (1024,1024,200) | 69.80 |  oc   |   1x    |   3.54   |      12.4      |  -  |     2      |      [r3det-oc_r50_fpn_1x_dota](../r3det/r3det-oc_r50_fpn_1x_dota.py)       |                   [model](https://download.openmmlab.com/mmrotate/v0.1.0/r3det/r3det_r50_fpn_1x_dota_oc/r3det_r50_fpn_1x_dota_oc-b1fb045c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/r3det/r3det_r50_fpn_1x_dota_oc/r3det_r50_fpn_1x_dota_oc_20220126_191226.log.json)                   |
| ResNet50 (1024,1024,200) | 72.68 |  oc   |   1x    |   3.62   |      12.2      |  -  |     2      | [r3det-oc_r50_fpn_kfiou-ln_1x_dota](./r3det-oc_r50_fpn_kfiou-ln_1x_dota.py) | [model](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/r3det_kfiou_ln_r50_fpn_1x_dota_oc/r3det_kfiou_ln_r50_fpn_1x_dota_oc-8e7f049d.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/r3det_kfiou_ln_r50_fpn_1x_dota_oc/r3det_kfiou_ln_r50_fpn_1x_dota_oc_20220123_074507.log.json) |

```
@misc{yang2022kfiou,

```
      title={The KFIoU Loss for Rotated Object Detection},
      author={Xue Yang and Yue Zhou and Gefan Zhang and Jirui Yang and Wentao Wang and Junchi Yan and Xiaopeng Zhang and Qi Tian},
      eprint={2201.12558},
}
```

# Albu Example

> [Albumentations: fast and flexible image augmentations](https://arxiv.org/abs/1809.06839)

Data augmentation is a commonly used technique for increasing both the size and the diversity of labeled training sets by leveraging input transformations that preserve output labels. In computer vision domain, image augmentations have become a common implicit regularization technique to combat overfitting in deep convolutional neural networks and are ubiquitously used to improve performance. While most deep learning frameworks implement basic image transformations, the list is typically limited to some variations and combinations of flipping, rotating, scaling, and cropping. Moreover, the image processing speed varies in existing tools for image augmentation. We present Albumentations, a fast and flexible library for image augmentations with many various image transform operations available, that is also an easy-to-use wrapper around other augmentation libraries. We provide examples of image augmentations for different computer vision tasks and show that Albumentations is faster than other commonly used image augmentation tools on the most of commonly used image transformations.

<img src="https://user-images.githubusercontent.com/40661020/143870703-74f3ea3f-ae23-4035-9856-746bc3f88464.png" height="400" />
</div>

| Backbone |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                    Config                     |                                                                                                                                                        Download                                                                                                                                                         |

| :------: | :-----: | :-----: | :------: | :------------: | :----: | :-----: | :-------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|   R-50   | pytorch |   1x    |   4.4    |      16.6      |  38.0  |  34.5   | [config](./mask-rcnn_r50_fpn_albu-1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/albu_example/mask_rcnn_r50_fpn_albu_1x_coco/mask_rcnn_r50_fpn_albu_1x_coco_20200208-ab203bcd.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/albu_example/mask_rcnn_r50_fpn_albu_1x_coco/mask_rcnn_r50_fpn_albu_1x_coco_20200208_225520.log.json) |

```latex
@article{2018arXiv180906839B,

```latex
  author = {A. Buslaev, A. Parinov, E. Khvedchenya, V.~I. Iglovikov and A.~A. Kalinin},
  title = "{Albumentations: fast and flexible image augmentations}",
  journal = {ArXiv e-prints},
  eprint = {1809.06839},
  year = 2018
}
```

```text
# img_name, "text"

train/1009_2.png You
train/1017_1.png Rescue
train/1017_2.png mission
```

### opencv to ncnn

* cv::Mat CV_8UC3 -> ncnn::Mat 3 channel + swap RGB/BGR

```cpp
// cv::Mat a(h, w, CV_8UC3);
ncnn::Mat in = ncnn::Mat::from_pixels(a.data, ncnn::Mat::PIXEL_BGR2RGB, a.cols, a.rows);
```

* cv::Mat CV_8UC3 -> ncnn::Mat 3 channel + keep RGB/BGR order

```cpp
ncnn::Mat in = ncnn::Mat::from_pixels(a.data, ncnn::Mat::PIXEL_RGB, a.cols, a.rows);
```

* cv::Mat CV_8UC3 -> ncnn::Mat 1 channel + do RGB2GRAY/BGR2GRAY

```cpp
// cv::Mat rgb(h, w, CV_8UC3);
ncnn::Mat inrgb = ncnn::Mat::from_pixels(rgb.data, ncnn::Mat::PIXEL_RGB2GRAY, rgb.cols, rgb.rows);

// cv::Mat bgr(h, w, CV_8UC3);
ncnn::Mat inbgr = ncnn::Mat::from_pixels(bgr.data, ncnn::Mat::PIXEL_BGR2GRAY, bgr.cols, bgr.rows);
```

* cv::Mat CV_8UC1 -> ncnn::Mat 1 channel

```cpp
// cv::Mat a(h, w, CV_8UC1);
ncnn::Mat in = ncnn::Mat::from_pixels(a.data, ncnn::Mat::PIXEL_GRAY, a.cols, a.rows);
```

* cv::Mat CV_32FC1 -> ncnn::Mat 1 channel

```cpp
```

  * **You could construct ncnn::Mat and fill data into it directly to avoid data copy**

```cpp
// cv::Mat a(h, w, CV_32FC1);
ncnn::Mat in(a.cols, a.rows, 1, (void*)a.data);
in = in.clone();
```

* cv::Mat CV_32FC3 -> ncnn::Mat 3 channel

```cpp
// cv::Mat a(h, w, CV_32FC3);
ncnn::Mat in_pack3(a.cols, a.rows, 1, (void*)a.data, (size_t)4u * 3, 3);
ncnn::Mat in;
ncnn::convert_packing(in_pack3, in, 1);
```

* std::vector < cv::Mat > + CV_32FC1 -> ncnn::Mat multiple channels

```cpp
// std::vector<cv::Mat> a(channels, cv::Mat(h, w, CV_32FC1));
int channels = a.size();
ncnn::Mat in(a[0].cols, a[0].rows, channels);
for (int p=0; p<in.c; p++)
{

```cpp
{
    memcpy(in.channel(p), (const uchar*)a[p].data, in.w * in.h * sizeof(float));
}
```

### ncnn to opencv

* ncnn::Mat 3 channel -> cv::Mat CV_8UC3 + swap RGB/BGR

  * **You may need to call in.substract_mean_normalize() first to scale values from 0..1 to 0..255**

```cpp
// ncnn::Mat in(w, h, 3);
cv::Mat a(in.h, in.w, CV_8UC3);
in.to_pixels(a.data, ncnn::Mat::PIXEL_BGR2RGB);
```

* ncnn::Mat 3 channel -> cv::Mat CV_8UC3 + keep RGB/BGR order

```cpp
in.to_pixels(a.data, ncnn::Mat::PIXEL_RGB);
```

* ncnn::Mat 1 channel -> cv::Mat CV_8UC1

```cpp

```

```cpp
// ncnn::Mat in(w, h, 1);
cv::Mat a(in.h, in.w, CV_8UC1);
in.to_pixels(a.data, ncnn::Mat::PIXEL_GRAY);
```

* ncnn::Mat 1 channel -> cv::Mat CV_32FC1

  * **You could consume or manipulate ncnn::Mat data directly to avoid data copy**

```cpp
// ncnn::Mat in;
cv::Mat a(in.h, in.w, CV_32FC1);
memcpy((uchar*)a.data, in.data, in.w * in.h * sizeof(float));
```

* ncnn::Mat 3 channel -> cv::Mat CV_32FC3

```cpp
ncnn::Mat in_pack3;
ncnn::convert_packing(in, in_pack3, 3);
cv::Mat a(in.h, in.w, CV_32FC3);
memcpy((uchar*)a.data, in_pack3.data, in.w * in.h * 3 * sizeof(float));
```

* ncnn::Mat multiple channels -> std::vector < cv::Mat > + CV_32FC1

```

```cpp
// ncnn::Mat in(w, h, channels);
std::vector<cv::Mat> a(in.c);
{
    a[p] = cv::Mat(in.h, in.w, CV_32FC1);
    memcpy((uchar*)a[p].data, in.channel(p), in.w * in.h * sizeof(float));
}
```

# Preparing Paris Street View Dataset

```bibtex
@inproceedings{pathak2016context,
  title={Context encoders: Feature learning by inpainting},
  author={Pathak, Deepak and Krahenbuhl, Philipp and Donahue, Jeff and Darrell, Trevor and Efros, Alexei A},
  pages={2536--2544},
}
```

Obtain the dataset [here](https://github.com/pathak22/context-encoder/issues/24).

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── paris_street_view
│   │   ├── train
|   |   ├── val

```

# 为 Pix2pix 准备配对数据集

```bibtex
@inproceedings{isola2017image,
  title={Image-to-image translation with conditional adversarial networks},
  author={Isola, Phillip and Zhu, Jun-Yan and Zhou, Tinghui and Efros, Alexei A},
  pages={1125--1134},
}
```

您可以从[此处](http://efrosgans.eecs.berkeley.edu/pix2pix/datasets/)下载配对数据集。然后，您需要解压缩并移动相应的数据集以遵循如下所示的文件夹结构。数据集已经由原作者准备好了。

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── paired
│   │   ├── facades
│   │   ├── maps
|   |   ├── edges2shoes
|   |   |    ├── train
|   |   |    ├── test
```

# TCANet

[Temporal Context Aggregation Network for Temporal Action Proposal Refinement](https://openaccess.thecvf.com/content/CVPR2021/papers/Qing_Temporal_Context_Aggregation_Network_for_Temporal_Action_Proposal_Refinement_CVPR_2021_paper.pdf)

Temporal action proposal generation aims to estimate temporal intervals of actions in untrimmed videos, which is a challenging yet important task in the video understanding field.
The proposals generated by current methods still suffer from inaccurate temporal boundaries and inferior confidence used for retrieval owing to the lack of efficient temporal modeling and effective boundary context utilization.
In this paper, we propose Temporal Context Aggregation Network (TCANet) to generate high-quality action proposals through `local and global` temporal context aggregation and complementary as well as progressive boundary refinement.

Specifically, we first design a Local-Global Temporal Encoder (LGTE), which adopts the channel grouping strategy to efficiently encode both `local and global` temporal inter-dependencies.
Furthermore, both the boundary and internal context of proposals are adopted for frame-level and segment-level boundary regressions, respectively.

Temporal Boundary Regressor (TBR) is designed to combine these two regression granularities in an end-to-end fashion, which achieves the precise boundaries and reliable confidence of proposals through progressive refinement. Extensive experiments are conducted on three challenging datasets: HACS, ActivityNet-v1.3, and THUMOS-14, where TCANet can generate proposals with high precision and recall. By combining with the existing action classifier, TCANet can obtain remarkable temporal action detection performance compared with other methods. Not surprisingly, the proposed TCANet won the 1$^{st}$ place in the CVPR 2020 - HACS challenge leaderboard on temporal action localization task.

<img src="https://user-images.githubusercontent.com/35267818/223302449-8891241c-e84a-4c74-bf31-073d6a75b33a.png" width="800"/>
</div>

### HACS dataset

| feature  | gpus | pretrain |  AUC  | AR@1 | AR@5  | AR@10 | AR@100 | gpu_mem(M) | iter time(s) |                     config                     |                     ckpt                     |                     log                     |
| :------: | :--: | :------: | :---: | :--: | :---: | :---: | :----: | :--------: | :----------: | :--------------------------------------------: | :------------------------------------------: | :-----------------------------------------: |

| SlowOnly |  2   |   None   | 51.39 | 3.61 | 16.92 | 21.94 | 62.80  |     -      |      -       | [config](/configs/localization/tcanet/tcanet_2xb8-700x100-9e_hacs-feature.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/localization/tcanet/tcanet_2xb8-700x100-9e_hacs-feature_20230621-d6bc10b0.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/localization/tcanet/tcanet_2xb8-700x100-9e_hacs-feature.log) |

For more details on data preparation, you can refer to [HACS Data Preparation](/tools/data/hacs/README.md).

## Train

Train TCANet model on HACS dataset with the SlowOnly feature.

```shell
bash tools/dist_train.sh configs/localization/tcanet/tcanet_2xb8-700x100-9e_hacs-feature.py 2
```

## Test

Test TCANet model on HACS dataset with the SlowOnly feature.

```shell

## Test

```shell
python3 tools/test.py  configs/localization/tcanet/tcanet_2xb8-700x100-9e_hacs-feature.py CHECKPOINT.PTH
```

For more details, you can refer to the **Testing** part in the [Training and Test Tutorial](/docs/en/user_guides/train_test.md).

```BibTeX
@inproceedings{qing2021temporal,
  title={Temporal Context Aggregation Network for Temporal Action Proposal Refinement},
  author={Qing, Zhiwu and Su, Haisheng and Gan, Weihao and Wang, Dongliang and Wu, Wei and Wang, Xiang and Qiao, Yu and Yan, Junjie and Gao, Changxin and Sang, Nong},
  pages={485--494},
}
```

### what is packing and why

packing is the form of storing multiple short-sized values as one long-sized value.

element packing is well mapped with the underlying simd register, which usually use one very wide register to store different types of values.

|C|elemsize|elempack|
|double|8|1|
|float|4|1|
|int|4|1|
|short|2|1|
|signed char|1|1|

|arm neon|elemsize|elempack|
|float64x2_t|16|2|
|float32x4_t|16|4|
|int32x4_t|16|4|
|float16x4_t|8|4|
|int8x8_t|8|8|

Though the real count of values doubles when elempack is two, the wide-sized value is still treated as one value in the view of Mat structure. For example, we want to store 40 float values in Mat object, if elempack 1 is used, Mat width is then 40, while 10 if elempack 4 is used.

|dims|w|h|c|cstep|elemsize|elempack|
|---|---|---|---|---|---|---|
|1|40|1|1|40|4|1|
|1|10|1|1|10|16|4|

### packing style convention

In practice, elempack 1, 4, 8 are the most common cases. It is possible to use any other packing style in theory.

The following table show the packing axis used in ncnn for different dimension.

|dims|packing axis|shape before packing|shape after packing|
|---|---|---|---|
|1|w|w|w/elempack|
|2|h|w, h|w, h/elempack|
|3|c|w, h, c|w, h, c/elempack|

If the packing axis dim is not evenly divisible by elempack, zero padding may be used.

```
outw = (w + elempack - 1) / elempack;
```

The following snippet shows the memory layout after elempack=4 on 3-dim Mat

```
// w=2 h=3 c=4 elempack=1
0 1
2 3
4 5

6 7
8 9
10 11

12 13
14 15
16 17

18 19
20 21
22 23

// w=2 h=3 c=1 elempack=4
(0,6,12,18) (1,7,13,19)
(2,8,14,20) (3,9,15,21)
(4,10,16,22) (5,11,17,23)
```

### how to convert elempack

There is a convenient wrapper function provided
```

20 21
22 23

```

```
// convert to elempack 4 if packing axis dim is evenly divisible by elempack
// return the identity Mat otherwise
ncnn::Mat a;
ncnn::Mat a_packed;
ncnn::convert_packing(a, a_packed, 4);
if (a_packed.elempack == 4)
{
    // check if packing is successful
}

// convert to packing 1, aka unpacking, shall be always successful
ncnn::Mat b;
ncnn::Mat b_unpacked;
ncnn::convert_packing(b, b_unpacked, 1);
```

### handle general interleaved data

Here is an example of using convert packing to convert RGB interleaved data to planar

**NOTE:** The following code is just presented to explain what packing is and the conversion process. Do not use it in production due to its poor performance. Do use ncnn::Mat::from_pixels()

```cpp
// rgb_interleaved_u8 is RGB RGB RGB ...
// rgb_interleaved_u8.w = w;

```cpp
// rgb_interleaved_u8.h = h;
// rgb_interleaved_u8.c = 1;
// rgb_interleaved_u8.elemsize = 3;
// rgb_interleaved_u8.elempack = 3;

ncnn::Mat rgb_interleaved_u8(w, h, 1, 3, 3);
ncnn::Mat rgb_planar_u8;

ncnn::convert_packing(rgb_interleaved_u8, rgb_planar_u8, 1);

// rgb_planar_u8 is now RRR ... GGG ... BBB ...
// rgb_planar_u8.w = w;
// rgb_planar_u8.h = h;
// rgb_planar_u8.c = 3;
// rgb_planar_u8.elemsize = 1;
// rgb_planar_u8.elempack = 1;
```

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_cvpr_2016/html/Wei_Convolutional_Pose_Machines_CVPR_2016_paper.html">CPM (CVPR'2016)</a></summary>

```bibtex
@inproceedings{wei2016convolutional,
  title={Convolutional pose machines},
  author={Wei, Shih-En and Ramakrishna, Varun and Kanade, Takeo and Sheikh, Yaser},
  booktitle={Proceedings of the IEEE conference on Computer Vision and Pattern Recognition},
  pages={4724--4732},
}
```

<details>

```bibtex

| [cpm](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_cpm_8xb64-210e_mpii-368x368.py) |  368x368   | 0.876 |  0.285   | [ckpt](https://download.openmmlab.com/mmpose/top_down/cpm/cpm_mpii_368x368-116e62b8_20200822.pth) | [log](https://download.openmmlab.com/mmpose/top_down/cpm/cpm_mpii_368x368_20200822.log.json) |

# Corruption Benchmarking

We provide tools to test object detection and instance segmentation models on the image corruption benchmark defined in [Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming](https://arxiv.org/abs/1907.07484).
This page provides basic tutorials how to use the benchmark.

```latex
@article{michaelis2019winter,
  title={Benchmarking Robustness in Object Detection:
    Autonomous Driving when Winter is Coming},
  author={Michaelis, Claudio and Mitzkus, Benjamin and
    Geirhos, Robert and Rusak, Evgenia and
    Bringmann, Oliver and Ecker, Alexander S. and
    Bethge, Matthias and Brendel, Wieland},
  journal={arXiv:1907.07484},
}
```

![image corruption example](../../../resources/corruptions_sev_3.png)

## About the benchmark

To submit results to the benchmark please visit the [benchmark homepage](https://github.com/bethgelab/robust-detection-benchmark)

The benchmark is modelled after the [imagenet-c benchmark](https://github.com/hendrycks/robustness) which was originally
published in [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://arxiv.org/abs/1903.12261) (ICLR 2019) by Dan Hendrycks and Thomas Dietterich.

The image corruption functions are included in this library but can be installed separately using:

```shell
pip install imagecorruptions
```

Compared to imagenet-c a few changes had to be made to handle images of arbitrary size and greyscale images.
We also modified the 'motion blur' and 'snow' corruptions to remove dependency from a linux specific library,
which would have to be installed separately otherwise. For details please refer to the [imagecorruptions repository](https://github.com/bethgelab/imagecorruptions).

## Inference with pretrained models

We provide a testing script to evaluate a models performance on any combination of the corruptions provided in the benchmark.

### Test a dataset

- [x] single GPU testing
- [ ] multiple GPU testing
- [ ] visualize detection results

You can use the following commands to test a models performance under the 15 corruptions used in the benchmark.

```shell
# single-gpu testing
python tools/analysis_tools/test_robustness.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}]
```

Alternatively different group of corruptions can be selected.

```shell
# noise
python tools/analysis_tools/test_robustness.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] --corruptions noise

# blur

```shell
# noise

# blur
python tools/analysis_tools/test_robustness.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] --corruptions blur

# wetaher
python tools/analysis_tools/test_robustness.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] --corruptions weather

# digital
python tools/analysis_tools/test_robustness.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] --corruptions digital
```

Or a costom set of corruptions e.g.:

```shell
# gaussian noise, zoom blur and snow
python tools/analysis_tools/test_robustness.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [--eval ${EVAL_METRICS}] --corruptions gaussian_noise zoom_blur snow
```

Finally the corruption severities to evaluate can be chosen.
Severity 0 corresponds to clean data and the effect increases from 1 to 5.

```shell
# severity 1

```

```shell
python tools/analysis_tools/test_robustness.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [--eval ${EVAL_METRICS}] --severities 1

# severities 0,2,4
python tools/analysis_tools/test_robustness.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [--eval ${EVAL_METRICS}] --severities 0 2 4
```

## Results for modelzoo models

The results on COCO 2017val are shown in the below table.

|        Model        |      Backbone       |  Style  | Lr schd | box AP clean | box AP corr. | box % | mask AP clean | mask AP corr. | mask % |
| :-----------------: | :-----------------: | :-----: | :-----: | :----------: | :----------: | :---: | :-----------: | :-----------: | :----: |

|    Faster R-CNN     |      R-50-FPN       | pytorch |   1x    |     36.3     |     18.2     | 50.2  |       -       |       -       |   -    |
|    Faster R-CNN     |      R-101-FPN      | pytorch |   1x    |     38.5     |     20.9     | 54.2  |       -       |       -       |   -    |
|    Faster R-CNN     |   X-101-32x4d-FPN   | pytorch |   1x    |     40.1     |     22.3     | 55.5  |       -       |       -       |   -    |
|    Faster R-CNN     |   X-101-64x4d-FPN   | pytorch |   1x    |     41.3     |     23.4     | 56.6  |       -       |       -       |   -    |
|    Faster R-CNN     |    R-50-FPN-DCN     | pytorch |   1x    |     40.0     |     22.4     | 56.1  |       -       |       -       |   -    |

|    Faster R-CNN     | X-101-32x4d-FPN-DCN | pytorch |   1x    |     43.4     |     26.7     | 61.6  |       -       |       -       |   -    |
|     Mask R-CNN      |      R-50-FPN       | pytorch |   1x    |     37.3     |     18.7     | 50.1  |     34.2      |     16.8      |  49.1  |
|     Mask R-CNN      |    R-50-FPN-DCN     | pytorch |   1x    |     41.1     |     23.3     | 56.7  |     37.2      |     20.7      |  55.7  |
|    Cascade R-CNN    |      R-50-FPN       | pytorch |   1x    |     40.4     |     20.1     | 49.7  |       -       |       -       |   -    |
| Cascade Mask R-CNN  |      R-50-FPN       | pytorch |   1x    |     41.2     |     20.7     | 50.2  |     35.7      |     17.6      |  49.3  |

|      RetinaNet      |      R-50-FPN       | pytorch |   1x    |     35.6     |     17.8     | 50.1  |       -       |       -       |   -    |
| Hybrid Task Cascade | X-101-64x4d-FPN-DCN | pytorch |   1x    |     50.6     |     32.7     | 64.7  |     43.8      |     28.1      |  64.0  |

Results may vary slightly due to the stochastic application of the corruptions.

contributors and maintainers pledge to make participation in our project and

include:

  advances

  advances

behavior and are expected to take appropriate, fair, corrective action in
response to any instance of unacceptable behavior.

# profiler

The SDK has ability to record the time consumption of each module in the pipeline. It's closed by default. To use this ability, two steps are required:

- Generate profiler data
- Analyze profiler Data

## Generate profiler data

Using the C interface and classification pipeline as an example, when creating the pipeline, the create api with context information needs to be used, and profiler handle needs to be added to the context. The detailed code is shown below. Running the demo normally will generate profiler data "profiler_data.txt" in the current directory.

```c++

    return 1;
  }

return 1;
  }
    return 1;
  }

    return 1;
  }

if (status != MMDEPLOY_SUCCESS) {
    return 1;
  }

  }

  return 0;
}

```

## Analyze profiler Data

The performance data can be visualized using a script.

```bash
```

mmdeploy_context_destroy(context);

  return 0;
}

```

```bash
```

The parsing results are as follows: "name" represents the name of the node, "n_call" represents the number of calls, "t_mean" represents the average time consumption, "t_50%" and "t_90%" represent the percentiles of the time consumption.

```bash

```

# RTMPose Benchmarks

简体中文 | [English](./README.md)

欢迎社区用户在不同硬件设备上进行推理速度测试，贡献到本项目目录下。

当前已测试：

- CPU
  - Intel i7-11700
- GPU
  - NVIDIA GeForce 1660 Ti
  - NVIDIA GeForce RTX 3090
- Nvidia Jetson
  - AGX Orin
  - Orin NX
- ARM
  - Snapdragon 865

### 人体 2d 关键点 (17 Keypoints)

### Model Info

|                                      Config                                       | Input Size | AP<sup><br>(COCO) | Params(M) | FLOPS(G) |
| :-------------------------------------------------------------------------------: | :--------: | :---------------: | :-------: | :------: |
| [RTMPose-t](../rtmpose/body_2d_keypoint/rtmpose-tiny_8xb256-420e_coco-256x192.py) |  256x192   |       68.5        |   3.34    |   0.36   |
|  [RTMPose-s](../rtmpose/body_2d_keypoint/rtmpose-s_8xb256-420e_coco-256x192.py)   |  256x192   |       72.2        |   5.47    |   0.68   |

|  [RTMPose-m](../rtmpose/body_2d_keypoint/rtmpose-m_8xb256-420e_coco-256x192.py)   |  256x192   |       75.8        |   13.59   |   1.93   |
|  [RTMPose-l](../rtmpose/body_2d_keypoint/rtmpose-l_8xb256-420e_coco-256x192.py)   |  256x192   |       76.5        |   27.66   |   4.16   |
|  [RTMPose-m](../rtmpose/body_2d_keypoint/rtmpose-m_8xb256-420e_coco-384x288.py)   |  384x288   |       77.0        |   13.72   |   4.33   |
|  [RTMPose-l](../rtmpose/body_2d_keypoint/rtmpose-l_8xb256-420e_coco-384x288.py)   |  384x288   |       77.3        |   27.79   |   9.35   |

### Speed Benchmark

图中所示为模型推理时间，单位毫秒。

|   Config    | Input Size | ORT<sup><br>(i7-11700) | TRT-FP16<sup><br>(GTX 1660Ti) | TRT-FP16<sup><br>(RTX 3090) | ncnn-FP16<sup><br>(Snapdragon 865) | TRT-FP16<sup><br>(Jetson AGX Orin) | TRT-FP16<sup><br>(Jetson Orin NX) |

| :---------: | :--------: | :--------------------: | :---------------------------: | :-------------------------: | :--------------------------------: | :--------------------------------: | :-------------------------------: |
| [RTMPose-t](../rtmpose/body_2d_keypoint/rtmpose-tiny_8xb256-420e_coco-256x192.py) |  256x192   |          3.20          |             1.06              |            0.98             |                9.02                |                1.63                |               1.97                |
| [RTMPose-s](../rtmpose/body_2d_keypoint/rtmpose-s_8xb256-420e_coco-256x192.py) |  256x192   |          4.48          |             1.39              |            1.12             |               13.89                |                1.85                |               2.18                |

| [RTMPose-m](../rtmpose/body_2d_keypoint/rtmpose-m_8xb256-420e_coco-256x192.py) |  256x192   |         11.06          |             2.29              |            1.18             |               26.44                |                2.72                |               3.35                |
| [RTMPose-l](../rtmpose/body_2d_keypoint/rtmpose-l_8xb256-420e_coco-256x192.py) |  256x192   |         18.85          |             3.46              |            1.37             |               45.37                |                3.67                |               4.78                |
| [RTMPose-m](../rtmpose/body_2d_keypoint/rtmpose-m_8xb256-420e_coco-384x288.py) |  384x288   |         24.78          |             3.66              |            1.20             |               26.44                |                3.45                |               5.08                |

| [RTMPose-l](../rtmpose/body_2d_keypoint/rtmpose-l_8xb256-420e_coco-384x288.py) |  384x288   |           -            |             6.05              |            1.74             |                 -                  |                4.93                |               7.23                |

### 人体全身 2d 关键点 (133 Keypoints)

| Config                                                                                       | Input Size | Whole AP | Whole AR | FLOPS(G) |
| :------------------------------------------------------------------------------------------- | :--------: | :------: | :------: | :------: |
| [RTMPose-m](../rtmpose/wholebody_2d_keypoint/rtmpose-m_8xb64-270e_coco-wholebody-256x192.py) |  256x192   |   60.4   |   66.7   |   2.22   |
| [RTMPose-l](../rtmpose/wholebody_2d_keypoint/rtmpose-l_8xb64-270e_coco-wholebody-256x192.py) |  256x192   |   63.2   |   69.4   |   4.52   |

| [RTMPose-l](../rtmpose/wholebody_2d_keypoint/rtmpose-l_8xb32-270e_coco-wholebody-384x288.py) |  384x288   |   67.0   |   72.3   |  10.07   |

- 图中所示为模型推理时间，单位毫秒。
- 来自不同社区用户的测试数据用 `|` 分隔开。

| Config                                        | Input Size | ORT<sup><br>(i7-11700) | TRT-FP16<sup><br>(GTX 1660Ti) | TRT-FP16<sup><br>(RTX 3090) | TRT-FP16<sup><br>(Jetson AGX Orin) | TRT-FP16<sup><br>(Jetson Orin NX) |
| :-------------------------------------------- | :--------: | :--------------------: | :---------------------------: | :-------------------------: | :--------------------------------: | :-------------------------------: |

| [RTMPose-m](../rtmpose/wholebody_2d_keypoint/rtmpose-m_8xb64-270e_coco-wholebody-256x192.py) |  256x192   |         13.50          |             4.00              |        1.17 \| 1.84         |                2.79                |               3.51                |
| [RTMPose-l](../rtmpose/wholebody_2d_keypoint/rtmpose-l_8xb64-270e_coco-wholebody-256x192.py) |  256x192   |         23.41          |             5.67              |        1.44 \| 2.61         |                3.80                |               4.95                |
| [RTMPose-l](../rtmpose/wholebody_2d_keypoint/rtmpose-l_8xb32-270e_coco-wholebody-384x288.py) |  384x288   |         44.58          |             7.68              |        1.75 \| 4.24         |                5.08                |               7.20                |

## 如何测试推理速度

我们使用 MMDeploy 提供的 `tools/profiler.py` 脚本进行模型测速。

用户需要准备一个存放测试图片的文件夹`./test_images`，profiler 将随机从该目录下抽取图片用于模型测速。

```shell
python tools/profiler.py \

```shell
    configs/mmpose/pose-detection_simcc_onnxruntime_dynamic.py \
    {RTMPOSE_PROJECT}/rtmpose/body_2d_keypoint/rtmpose-m_8xb256-420e_coco-256x192.py \
    ../test_images \
    --model {WORK_DIR}/end2end.onnx \
    --shape 256x192 \
    --warmup 50 \
    --num-iter 200
```

The result is as follows:

```shell
01/30 15:06:35 - mmengine - INFO - [onnxruntime]-70 times per count: 8.73 ms, 114.50 FPS
01/30 15:06:36 - mmengine - INFO - [onnxruntime]-90 times per count: 9.05 ms, 110.48 FPS
01/30 15:06:37 - mmengine - INFO - [onnxruntime]-110 times per count: 9.87 ms, 101.32 FPS
01/30 15:06:37 - mmengine - INFO - [onnxruntime]-130 times per count: 9.99 ms, 100.10 FPS
01/30 15:06:38 - mmengine - INFO - [onnxruntime]-150 times per count: 10.39 ms, 96.29 FPS

01/30 15:06:39 - mmengine - INFO - [onnxruntime]-170 times per count: 10.77 ms, 92.86 FPS
01/30 15:06:40 - mmengine - INFO - [onnxruntime]-190 times per count: 10.98 ms, 91.05 FPS
01/30 15:06:40 - mmengine - INFO - [onnxruntime]-210 times per count: 11.19 ms, 89.33 FPS
01/30 15:06:41 - mmengine - INFO - [onnxruntime]-230 times per count: 11.16 ms, 89.58 FPS
01/30 15:06:42 - mmengine - INFO - [onnxruntime]-250 times per count: 11.06 ms, 90.41 FPS
----- Settings:
+------------+---------+
| batch size |    1    |
|   shape    | 256x192 |
| iterations |   200   |
|   warmup   |    50   |
----- Results:
+--------+------------+---------+
| Stats  | Latency/ms |   FPS   |
|  Mean  |   11.060   |  90.412 |
| Median |   11.852   |  84.375 |

|  Min   |   7.812    | 128.007 |
|  Max   |   13.690   |  73.044 |
```

If you want to learn more details of profiler, you can refer to the [Profiler Docs](https://mmdeploy.readthedocs.io/en/latest/02-how-to-run/useful_tools.html#profiler).

# SwAV

> [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://arxiv.org/abs/2006.09882)

Unsupervised image representations have significantly reduced the gap with supervised pretraining, notably with the recent achievements of contrastive learning methods. These contrastive methods typically work online and rely on a large number of explicit pairwise feature comparisons, which is computationally challenging. In this paper, we propose an online algorithm, SwAV, that takes advantage of contrastive methods without requiring to compute pairwise comparisons. Specifically, our method simultaneously clusters the data while enforcing consistency between cluster assignments produced for different augmentations (or “views”) of the same image, instead of comparing features directly as in contrastive learning. Simply put, we use a “swapped” prediction mechanism where we predict the code of a view from the representation of another view. Our method can be trained with large and small batches and can scale to unlimited amounts of data. Compared to previous contrastive methods, our method is more memory efficient since it does not require a large memory bank or a special momentum network. In addition, we also propose a new data augmentation strategy, multi-crop, that uses a mix of views with different resolutions in place of two full-resolution views, without increasing the memory or compute requirements.

<img  src="https://user-images.githubusercontent.com/36138628/149724517-9f1e7bdf-04c7-43e3-92f4-2b8fc1399123.png" width="500" />
</div>

```python

predict = inference_model('resnet50_swav-pre_8xb32-linear-coslr-100e_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('swav_resnet50_8xb32-mcrop-coslr-200e_in1k-224px-96px', pretrained=True)
```

Train:

```shell

```

Train:

```shell
python tools/train.py configs/swav/swav_resnet50_8xb32-mcrop-coslr-200e_in1k-224px-96px.py
```

Test:

```shell
python tools/test.py configs/swav/benchmarks/resnet50_8xb512-linear-coslr-90e_in1k.py https://download.openmmlab.com/mmselfsup/1.x/swav/swav_resnet50_8xb32-mcrop-2-6-coslr-200e_in1k-224-96/resnet50_linear-8xb32-coslr-100e_in1k/resnet50_linear-8xb32-coslr-100e_in1k_20220825-80341e08.pth
```

| Model                                                  | Params (M) | Flops (G) |                             Config                             |                             Download                              |

| :----------------------------------------------------- | :--------: | :-------: | :------------------------------------------------------------: | :---------------------------------------------------------------: |
| `swav_resnet50_8xb32-mcrop-coslr-200e_in1k-224px-96px` |   28.35    |   4.11    | [config](swav_resnet50_8xb32-mcrop-coslr-200e_in1k-224px-96px.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/swav/swav_resnet50_8xb32-mcrop-2-6-coslr-200e_in1k-224-96/swav_resnet50_8xb32-mcrop-2-6-coslr-200e_in1k-224-96_20220825-5b3fc7fc.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/swav/swav_resnet50_8xb32-mcrop-2-6-coslr-200e_in1k-224-96/swav_resnet50_8xb32-mcrop-2-6-coslr-200e_in1k-224-96_20220825-5b3fc7fc.json) |

| `resnet50_swav-pre_8xb32-linear-coslr-100e_in1k` | [SWAV](https://download.openmmlab.com/mmselfsup/1.x/swav/swav_resnet50_8xb32-mcrop-2-6-coslr-200e_in1k-224-96/swav_resnet50_8xb32-mcrop-2-6-coslr-200e_in1k-224-96_20220825-5b3fc7fc.pth) |   25.56    |   4.11    |   70.50   | [config](benchmarks/resnet50_8xb512-linear-coslr-90e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/swav/swav_resnet50_8xb32-mcrop-2-6-coslr-200e_in1k-224-96/resnet50_linear-8xb32-coslr-100e_in1k/resnet50_linear-8xb32-coslr-100e_in1k_20220825-80341e08.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/swav/swav_resnet50_8xb32-mcrop-2-6-coslr-200e_in1k-224-96/resnet50_linear-8xb32-coslr-100e_in1k/resnet50_linear-8xb32-coslr-100e_in1k_20220825-80341e08.json) |

```bibtex

```bibtex
  booktitle={NeurIPS},
}
```

# Awesome MMPose

A list of resources related to MMPose. Feel free to contribute!

<img src="https://user-images.githubusercontent.com/13503330/231416285-5467d313-0732-4ada-97e1-12be6ec69a28.png" width="800"/>
</div><br/>

## Contents

- [Tutorials](#tutorials)
- [Papers](#papers)
- [Datasets](#datasets)
- [Projects](#projects)

## Tutorials

- [MMPose Tutorial (Chinese)](https://github.com/TommyZihao/MMPose_Tutorials)

  MMPose 中文视频代码教程，from 同济子豪兄

  <div align=center>
  <img src="https://user-images.githubusercontent.com/13503330/231640277-777f611c-b3d9-4d41-830f-8e48a352fd01.jpg" width="500"/>
  </div><br/>

- [OpenMMLab Course](https://github.com/open-mmlab/OpenMMLabCourse)

  This repository hosts articles, lectures and tutorials on computer vision and OpenMMLab, helping learners to understand algorithms and master our toolboxes in a systematical way.

## Papers

This repository hosts articles, lectures and tutorials on computer vision and OpenMMLab, helping learners to understand algorithms and master our toolboxes in a systematical way.

## Papers

- [\[paper\]](https://arxiv.org/abs/2207.10387) [\[code\]](https://github.com/luminxu/Pose-for-Everything)

  ECCV 2022, Pose for Everything: Towards Category-Agnostic Pose Estimation

- [\[paper\]](https://arxiv.org/abs/2201.04676) [\[code\]](https://github.com/Sense-X/UniFormer)

  ICLR 2022, UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning

- [\[paper\]](https://arxiv.org/abs/2201.07412) [\[code\]](https://github.com/aim-uofa/Poseur)

  ECCV 2022, Poseur:Direct Human Pose Regression with Transformers

- [\[paper\]](https://arxiv.org/abs/2106.03348) [\[code\]](https://github.com/ViTAE-Transformer/ViTAE-Transformer)

  NeurIPS 2022, ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond

NeurIPS 2022, ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond

- [\[paper\]](https://arxiv.org/abs/2204.10762) [\[code\]](https://github.com/ZiyiZhang27/Dite-HRNet)

  IJCAI-ECAI 2021, Dite-HRNet:Dynamic Lightweight High-Resolution Network for Human Pose Estimation

- [\[paper\]](https://arxiv.org/abs/2302.08453) [\[code\]](https://github.com/TencentARC/T2I-Adapter)

  T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models

- [\[paper\]](https://arxiv.org/pdf/2303.11638.pdf) [\[code\]](https://github.com/Gengzigang/PCT)

  CVPR 2023, Human Pose as Compositional Tokens

## Datasets

- [\[github\]](https://github.com/luminxu/Pose-for-Everything) **MP-100**

  Multi-category Pose (MP-100) dataset, which is a 2D pose dataset of 100 object categories containing over 20K instances and is well-designed for developing CAPE algorithms.

Multi-category Pose (MP-100) dataset, which is a 2D pose dataset of 100 object categories containing over 20K instances and is well-designed for developing CAPE algorithms.

  <img src="https://user-images.githubusercontent.com/13503330/231639551-b32ed2ab-aec0-4410-937e-c81a2ac2cb0d.png" width="500"/>

- [\[github\]](https://github.com/facebookresearch/Ego4d/) **Ego4D**

  EGO4D is the world's largest egocentric (first person) video ML dataset and benchmark suite, with 3,600 hrs (and counting) of densely narrated video and a wide range of annotations across five new benchmark tasks. It covers hundreds of scenarios (household, outdoor, workplace, leisure, etc.) of daily life activity captured in-the-wild by 926 unique camera wearers from 74 worldwide locations and 9 different countries.

  <img src="https://user-images.githubusercontent.com/13503330/231640003-d43028cc-6f83-45e7-b76a-8e8f0cddcfcb.png" width="500"/>

## Projects

Waiting for your contribution!

# DRRG

> [Deep relational reasoning graph network for arbitrary shape text detection](https://arxiv.org/abs/2003.07493)

Arbitrary shape text detection is a challenging task due to the high variety and complexity of scenes texts. In this paper, we propose a novel unified relational reasoning graph network for arbitrary shape text detection. In our method, an innovative local graph bridges a text proposal model via Convolutional Neural Network (CNN) and a deep relational reasoning network via Graph Convolutional Network (GCN), making our network end-to-end trainable. To be concrete, every text instance will be divided into a series of small rectangular components, and the geometry attributes (e.g., height, width, and orientation) of the small components will be estimated by our text proposal model. Given the geometry attributes, the local graph construction model can roughly establish linkages between different text components. For further reasoning and deducing the likelihood of linkages between the component and its neighbors, we adopt a graph-based network to perform deep relational reasoning on local graphs. Experiments on public available datasets demonstrate the state-of-the-art performance of our method.

<img src="https://user-images.githubusercontent.com/22607038/142791777-f282300a-fb83-4b5a-a7d4-29f308949f11.png"/>
</div>

### CTW1500

|                 Method                  |                 BackBone                  | Pretrained Model | Training set  |   Test set   | #epochs | Test size | Precision | Recall | Hmean  |                  Download                  |
| :-------------------------------------: | :---------------------------------------: | :--------------: | :-----------: | :----------: | :-----: | :-------: | :-------: | :----: | :----: | :----------------------------------------: |

| [DRRG](/configs/textdet/drrg/drrg_resnet50_fpn-unet_1200e_ctw1500.py) |                 ResNet50                  |        -         | CTW1500 Train | CTW1500 Test |  1200   |    640    |  0.8775   | 0.8179 | 0.8467 | [model](https://download.openmmlab.com/mmocr/textdet/drrg/drrg_resnet50_fpn-unet_1200e_ctw1500/drrg_resnet50_fpn-unet_1200e_ctw1500_20220827_105233-d5c702dd.pth) \\ [log](https://download.openmmlab.com/mmocr/textdet/drrg/drrg_resnet50_fpn-unet_1200e_ctw1500/20220827_105233.log) |
| [DRRG_r50-oclip](/configs/textdet/drrg/drrg_resnet50-oclip_fpn-unet_1200e_ctw1500.py) | [ResNet50-oCLIP](https://download.openmmlab.com/mmocr/backbone/resnet50-oclip-7ba0c533.pth) |        -         | CTW1500 Train | CTW1500 Test |  1200   |           |           |        |        |          [model](<>) \\ [log](<>)          |

```bibtex
@article{zhang2020drrg,
  title={Deep relational reasoning graph network for arbitrary shape text detection},

```bibtex
  author={Zhang, Shi-Xue and Zhu, Xiaobin and Hou, Jie-Bo and Liu, Chang and Yang, Chun and Wang, Hongfa and Yin, Xu-Cheng},
  pages={9699-9708},
}
```

# CondInst

> [CondInst: Conditional Convolutions for Instance
> Segmentation](https://arxiv.org/pdf/2003.05664.pdf)

We propose a simple yet effective instance segmentation framework, termed CondInst (conditional convolutions for instance segmentation). Top-performing instance segmentation methods such as Mask
R-CNN rely on ROI operations (typically ROIPool or ROIAlign) to
obtain the final instance masks. In contrast, we propose to solve instance segmentation from a new perspective. Instead of using instancewise ROIs as inputs to a network of fixed weights, we employ dynamic
instance-aware networks, conditioned on instances. CondInst enjoys two
advantages: 1) Instance segmentation is solved by a fully convolutional
network, eliminating the need for ROI cropping and feature alignment.
2\) Due to the much improved capacity of dynamically-generated conditional convolutions, the mask head can be very compact (e.g., 3 conv.

layers, each having only 8 channels), leading to significantly faster inference. We demonstrate a simpler instance segmentation method that can
achieve improved performance in both accuracy and inference speed. On
the COCO dataset, we outperform a few recent methods including welltuned Mask R-CNN baselines, without longer training schedules needed.

<img src="https://user-images.githubusercontent.com/57584090/203303488-3dbc36da-09a6-4dc8-be9d-d9af27bd1234.png"/>
</div>

</div>

| Backbone |  Style  | MS train | Lr schd | bbox AP | mask AP |                          Config                           |                                                                                                                                                                              Download                                                                                                                                                                              |

| :------: | :-----: | :------: | :-----: | :-----: | :-----: | :-------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|   R-50   | pytorch |    Y     |   1x    |  39.8   |  36.0   | [config](./condinst_r50_fpn_ms-poly-90k_coco_instance.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/condinst/condinst_r50_fpn_ms-poly-90k_coco_instance/condinst_r50_fpn_ms-poly-90k_coco_instance_20221129_125223-4c186406.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/condinst/condinst_r50_fpn_ms-poly-90k_coco_instance/condinst_r50_fpn_ms-poly-90k_coco_instance_20221129_125223.json) |

```latex

```latex
@inproceedings{tian2020conditional,
  title     =  {Conditional Convolutions for Instance Segmentation},
  author    =  {Tian, Zhi and Shen, Chunhua and Chen, Hao},
  booktitle =  {Proc. Eur. Conf. Computer Vision (ECCV)},
  year      =  {2020}
}
```

# 安装 （待更新）

中文版本的安装文档过于陈旧，请参考英文版本。
如果您希望帮助翻译英文版安装文档，请通过issue联系我们
```

## 依赖

- Linux / Windows / Mac
- Python 3.6+
- PyTorch 1.5 或更高
- CUDA 9.0 或更高
- NCCL 2
- GCC 5.4 或更高
- [mmcv](https://github.com/open-mmlab/mmcv)

## 安装

a. 创建并激活 conda 虚拟环境，如 `python 3.8`：

```shell
conda create -n mmagic python=3.8 -y
```

b. 按照 [PyTorch 官方文档](https://pytorch.org/) 安装 PyTorch 和 torchvision，然后安装对应路径下的 `mmcv-full`

如 `cuda 10.1` & `pytorch 1.7`：

```shell
conda install pytorch==1.7.1 torchvision cudatoolkit=10.1 -c pytorch
pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu101/torch1.7/index.html "opencv-python<=4.5.4.60"
```

注 1：过高版本的 `opencv-python` 在使用中存在一些问题，因此在安装时限制其版本。

注 2：确保 CUDA 编译版本和 CUDA 运行版本相匹配。 用户可以参照 [PyTorch 官网](https://pytorch.org/) 对预编译包所支持的 CUDA 版本进行核对。

`例1`：如果 `/usr/local/cuda` 文件夹下已安装了 CUDA 10.1 版本，则需要安装 CUDA 10.1 下预编译的 PyTorch。

```shell
conda install pytorch cudatoolkit=10.1 torchvision -c pytorch
```

```shell
```

`例2`：如果你在 `/usr/local/cuda` 文件夹下已安装 CUDA 9.2 版本，并且想要安装 PyTorch 1.3.1 版本，则需要安装 CUDA 9.2 下预编译的 PyTorch。

```shell
conda install pytorch=1.3.1 cudatoolkit=9.2 torchvision=0.4.2 -c pytorch
```

c. 克隆 MMagic 仓库

```shell
cd mmagic
```

d. 安装相关依赖和 MMagic

```shell
pip install -r requirements.txt
pip install -v -e .  # or "python setup.py develop"
```

如果你是在 macOS 环境下安装，则需将上述命令的最后一行替换为：

```shell
CC=clang CXX=clang++ CFLAGS='-stdlib=libc++' pip install -e .
```

e. 验证安装

安装完成后，可以切换到其他目录（例如 `/home` 目录），并尝试在 python 中导入 mmagic，导入成功则证明安装成功

```shell
$ cd ~
$ python

>>> import mmagic
>>> mmagic.__version__
'0.12.0'
```

注：

1. git commit 的 id 将会被写到版本号中，如 0.6.0+2e7045c。这个版本号也会被保存到训练好的模型中。 推荐用户每次在对本地代码和 github 上的源码进行同步后，执行一次步骤 b。如果 C++/CUDA 代码被修改，就必须进行这一步骤。

'0.12.0'
```

注：

   > 重要：如果你在一个新的 CUDA/PyTorch 版本下重新安装了 mmagic ，请确保删除了`./build`文件夹

   ```shell
   pip uninstall mmagic
   rm -rf ./build
   find . -name "*.so" | xargs rm
   ```

2. 根据上述步骤， MMagic 就会以 `dev` 模式被安装，任何本地的代码修改都会立刻生效，不需要再重新安装一遍（除非用户提交了 commits，并且想更新版本号）。

3. 如果用户想使用 `opencv-python-headless` 而不是 `opencv-python`，可在安装 `MMCV` 前安装 `opencv-python-headless`。

4. 有些模型（例如图像修复任务中的 `EDVR`）依赖于 `mmcv-full` 中的一些 CUDA 算子，具体的依赖可在 `requirements.txt` 中找到。
   如需要，请通过 `pip install -r requirements.txt` 命令来安装 `mmcv-full`，安装过程中会在本地编译 CUDA 算子，这个过程大概需要10分钟。
   另一种方案是安装预编译版本的 `mmcv-full`，请参考 [MMCV 主页](https://github.com/open-mmlab/mmcv#install-with-pip) 获取具体的安装指令。
   此外，如果你要使用的模型不依赖于 CUDA 算子，那么也可以使用 `pip install mmcv`来安装轻量版本的 mmcv，其中 CUDA 算子被移除了。

### CPU 环境下的安装步骤

MMagic 也可以在只有 CPU 的环境下安装（即无法使用 GPU 的环境）。

此外，如果你要使用的模型不依赖于 CUDA 算子，那么也可以使用 `pip install mmcv`来安装轻量版本的 mmcv，其中 CUDA 算子被移除了。

相应的，安装 CPU 版本的 PyTorch 和 MMCV

```shell
```

然而在该环境下，有些功能将被移除，例如：

- Deformable Convolution（可变形卷积）

因此，如果您尝试使用一些包含可变形卷积的模型进行推理，则会出现错误。

### 利用 Docker 镜像安装 MMagic

MMagic 提供了一个 [Dockerfile](https://github.com/open-mmlab/mmagic/blob/master/docker/Dockerfile) 来创建 docker 镜像

```shell
# build an image with PyTorch 1.5, CUDA 10.1
```

运行以下命令：

```shell
```

### 完整的安装脚本

```shell
conda create -n open-mmlab python=3.7 -y

# install latest pytorch prebuilt with the default prebuilt CUDA version (usually the latest)

```

```shell

conda install -c pytorch pytorch torchvision -y
cd mmagic
```

# Configs

We use python files as configs and incorporate modular and inheritance design into our config system, which is convenient to conduct various experiments.

MMPose is equipped with a powerful config system. Cooperating with Registry, a config file can organize all the configurations in the form of python dictionaries and create instances of the corresponding modules.

Here is a simple example of vanilla Pytorch module definition to show how the config system works:

```Python
# Definition of Loss_A in loss_a.py
Class Loss_A(nn.Module):
    def __init__(self, param1, param2):
        self.param1 = param1
        self.param2 = param2
        return x

# Init the module
loss = Loss_A(param1=1.0, param2=True)
```

All you need to do is just to register the module to the pre-defined Registry `MODELS`:

```Python
from mmpose.registry import MODELS

@MODELS.register_module() # register the module to MODELS

```Python

        return x
```

And import the new module in `__init__.py` in the corresponding directory:

```Python
# __init__.py of mmpose/models/losses
from .loss_a.py import Loss_A

__all__ = ['Loss_A']
```

Then you can define the module anywhere you want：

```Python
# config_file.py
loss_cfg = dict(
    type='Loss_A', # specify your registered module via `type`
    param1=1.0,    # pass parameters to __init__() of the module
    param2=True
)

loss = MODELS.build(loss_cfg) # equals to `loss = Loss_A(param1=1.0, param2=True)`
```

```{note}

param1=1.0,    # pass parameters to __init__() of the module
)

```

```{note}
```

Here is a list of pre-defined registries in MMPose:

- `DATASETS`: data-related modules
- `TRANSFORMS`: data transformations
- `MODELS`: all kinds of modules inheriting `nn.Module` (Backbone, Neck, Head, Loss, etc.)
- `VISUALIZERS`: visualization tools
- `VISBACKENDS`: visualizer backend
- `METRICS`: all kinds of evaluation metrics
- `KEYPOINT_CODECS`: keypoint encoder/decoder
- `HOOKS`: all kinds of hooks like `CheckpointHook`

All registries are defined in `$MMPOSE/mmpose/registry.py`.

## Config System

It is best practice to layer your configs in five sections:

- **General**: basic configurations non-related to training or testing, such as Timer, Logger, Visualizer and other Hooks, as well as distributed-related environment settings

- **Data**: dataset, dataloader and data augmentation

- **Training**: resume, weights loading, optimizer, learning rate scheduling, epochs and valid interval etc.

- **Model**: structure, module and loss function etc.

- **Evaluation**: metrics

You can find all the provided configs under `$MMPOSE/configs`. A config can inherit contents from another config.To keep a config file simple and easy to read, we store some necessary but unremarkable configurations to `$MMPOSE/configs/_base_`.You can inspect the complete configurations by：

```Bash
python tools/analysis/print_config.py /PATH/TO/CONFIG
```

### General

```Bash
```

General configuration refers to the necessary configuration non-related to training or testing, mainly including:

- **Default Hooks**: time statistics, training logs, checkpoints etc.

- **Environment**: distributed backend, cudnn, multi-processing etc.

- **Visualizer**: visualization backend and strategy

- **Log**: log level, format, printing and recording interval etc.

Here is the description of General configuration:

```Python
# General
default_scope = 'mmpose'
    timer=dict(type='IterTimerHook'), # time the data processing and model inference
    logger=dict(type='LoggerHook', interval=50), # interval to print logs
    param_scheduler=dict(type='ParamSchedulerHook'), # update lr
        type='CheckpointHook', interval=1, save_best='coco/AP', # interval to save ckpt
        rule='greater'), # rule to judge the metric

checkpoint=dict(
    sampler_seed=dict(type='DistSamplerSeedHook')) # set the distributed seed
    cudnn_benchmark=False, # cudnn benchmark flag
    mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0), # num of opencv threads
    dist_cfg=dict(backend='nccl')) # distributed training backend
vis_backends = [dict(type='LocalVisBackend')] # visualizer backend
visualizer = dict( # Config of visualizer
    type='PoseLocalVisualizer',
    vis_backends=[dict(type='LocalVisBackend')],
log_processor = dict( # Format, interval to log
    type='LogProcessor', window_size=50, by_epoch=True, num_digits=6)
log_level = 'INFO' # The level of logging
```

General configuration is stored alone in the `$MMPOSE/configs/_base_`, and inherited by doing:

```Python

```

```Python
_base_ = ['../../../_base_/default_runtime.py'] # take the config file as the starting point of the relative path
```

```{note}
CheckpointHook:

- save_best: `'coco/AP'` for `CocoMetric`, `'PCK'` for `PCKAccuracy`
- max_keep_ckpts: the maximum checkpoints to keep. Defaults to -1, which means unlimited.

Example:

`default_hooks = dict(checkpoint=dict(save_best='PCK', rule='greater', max_keep_ckpts=1))`
```

### Data

Data configuration refers to the data processing related settings, mainly including:

- **File Client**: data storage backend, default is `disk`, we also support `LMDB`, `S3 Bucket` etc.

- **Dataset**: image and annotation file path

- **Dataloader**: loading configuration, batch size etc.

- **Pipeline**: data augmentation

- **Input Encoder**: encoding the annotation into specific form of target

Here is the description of Data configuration:

```Python
backend_args = dict(backend='local') # data storage backend
dataset_type = 'CocoDataset' # name of dataset
data_mode = 'topdown' # type of the model
data_root = 'data/coco/' # root of the dataset
 # config of codec，to generate targets and decode preds into coordinates
codec = dict(
    type='MSRAHeatmap', input_size=(192, 256), heatmap_size=(48, 64), sigma=2)
train_pipeline = [ # data aug in training
    dict(type='LoadImage', backend_args=backend_args, # image loading
    dict(type='GetBBoxCenterScale'), # calculate center and scale of bbox
    dict(type='RandomBBoxTransform'), # config of scaling, rotation and shifing
    dict(type='RandomFlip', direction='horizontal'), # config of random flipping
    dict(type='RandomHalfBody'), # config of half-body aug

dict(type='RandomFlip', direction='horizontal'), # config of random flipping
    dict(type='TopdownAffine', input_size=codec['input_size']), # update inputs via transform matrix
    dict(
        type='GenerateTarget', # generate targets via transformed inputs
        # typeof targets
        encoder=codec, # get encoder from codec
    dict(type='PackPoseInputs') # pack targets
]
test_pipeline = [ # data aug in testing
    dict(type='LoadImage', backend_args=backend_args), # image loading
]
    batch_size=64, # batch size of each single GPU during training
    num_workers=2, # workers to pre-fetch data for each single GPU

]
    persistent_workers=True, # workers will stay around (with their state) waiting for another call into that dataloader.
    sampler=dict(type='DefaultSampler', shuffle=True), # data sampler, shuffle in traning
        type=dataset_type , # name of dataset
        data_root=data_root, # root of dataset
        data_mode=data_mode, # type of the model
        ann_file='annotations/person_keypoints_train2017.json', # path to annotation file
        data_prefix=dict(img='train2017/'), # path to images
        pipeline=train_pipeline
    ))
    batch_size=32, # batch size of each single GPU during validation

pipeline=train_pipeline
    ))
    sampler=dict(type='DefaultSampler', shuffle=False), # data sampler
        ann_file='annotations/person_keypoints_val2017.json', # path to annotation file
        bbox_file=
        'data/coco/person_detection_results/COCO_val2017_detections_AP_H_56_person.json', # bbox file use for evaluation
        data_prefix=dict(img='val2017/'), # path to images
        pipeline=test_pipeline
    ))
test_dataloader = val_dataloader # use val as test by default
```

```{note}

data_prefix=dict(img='val2017/'), # path to images
    ))
```

```{note}
Common Usages:
- [Resume training](../common_usages/resume_training.md)
- [Automatic mixed precision (AMP) training](../common_usages/amp_training.md)
- [Set the random seed](../common_usages/set_random_seed.md)

```

### Training

Training configuration refers to the training related settings including:

- Resume training

- Model weights loading

- Epochs of training and interval to validate

- Learning rate adjustment strategies like warm-up, scheduling etc.

- Optimizer and initial learning rate

- Advanced tricks like auto learning rate scaling

Here is the description of Training configuration:

```Python
resume = False # resume checkpoints from a given path, the training will be resumed from the epoch when the checkpoint's is saved

```Python
load_from = None # load models as a pre-trained model from a given path
train_cfg = dict(by_epoch=True, max_epochs=210, val_interval=10) # max epochs of training, interval to validate
    dict( # warmup strategy
        type='LinearLR', begin=0, end=500, start_factor=0.001, by_epoch=False),
    dict( # scheduler
        begin=0,
        end=210,
        milestones=[170, 200],
        gamma=0.1,
        by_epoch=True)
]
optim_wrapper = dict(optimizer=dict(type='Adam', lr=0.0005)) # optimizer and initial lr
auto_scale_lr = dict(base_batch_size=512) # auto scale the lr according to batch size
```

### Model

Model configuration refers to model training and inference related settings including:

- Model Structure

- Loss Function

- Output Decoding

```

### Model

- Test-time augmentation

Here is the description of Model configuration, which defines a Top-down Heatmap-based HRNetx32:

```Python
# config of codec, if already defined in data configuration section, no need to define again

    type='TopdownPoseEstimator', # Macro model structure
    data_preprocessor=dict( # data normalization and channel transposition
    backbone=dict( # config of backbone
        in_channels=3,

type='HRNet',
            type='Pretrained', # load pretrained weights to backbone

num_blocks=(4, 4, 4, 4),
            checkpoint='https://download.openmmlab.com/mmpose'
            '/pretrain_models/hrnet_w32-36af842e.pth'),
    ),
    head=dict( # config of head
        type='HeatmapHead',
        in_channels=32,
        out_channels=17,
        deconv_out_channels=None,
        loss=dict(type='KeypointMSELoss', use_target_weight=True), # config of loss function
        decoder=codec), # get decoder from codec
        flip_test=True, # flag of flip test
        flip_mode='heatmap', # heatmap flipping
        shift_heatmap=True,  # shift the flipped heatmap several pixels to get a better performance
    ))
```

### Evaluation

Evaluation configuration refers to metrics commonly used by public datasets for keypoint detection tasks, mainly including:

- AR, AP and mAP

- PCK, PCKh, tPCK

))
```

- AUC

- EPE

- NME

Here is the description of Evaluation configuration, which defines a COCO metric evaluator:

```Python
    type='CocoMetric', # coco AP
    ann_file=data_root + 'annotations/person_keypoints_val2017.json') # path to annotation file
test_evaluator = val_evaluator # use val as test by default
```

## Config File Naming Convention

MMPose follow the style below to name config files：

```Python
```

The filename is divided into four parts:

- **Algorithm Information**: the name of algorithm, such as `topdown-heatmap`, `topdown-rle`

- **Module Information**: list of intermediate modules in the forward order, such as `res101`, `hrnet-w48`

- **Training Information**: settings of training(e.g. `batch_size`, `scheduler`), such as `8xb64-210e`

- **Data Information**: the name of dataset, the reshape of input data, such as `ap10k-256x256`, `zebra-160x160`

Words between different parts are connected by `'_'`, and those from the same part are connected by `'-'`.

To avoid a too long filename, some strong related modules in `{{module info}}` will be omitted, such as `gap` in `RLE` algorithm, `deconv` in `Heatmap-based` algorithm

Contributors are advised to follow the same style.

### Inheritance

This is often used to inherit configurations from other config files. Let's assume two configs like:

`optimizer_cfg.py`:

```Python
```

`resnet50.py`:

```Python
_base_ = ['optimizer_cfg.py']
model = dict(type='ResNet', depth=50)
```

```Python
```

```Python
```

Although we did not define `optimizer` in `resnet50.py`, all configurations in `optimizer.py` will be inherited by setting `_base_ = ['optimizer_cfg.py']`

```Python
cfg = Config.fromfile('resnet50.py')
cfg.optimizer  # ConfigDict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001)
```

### Modification

For configurations already set in previous configs, you can directly modify arguments specific to that module.

`resnet50_lr0.01.py`:

```Python
optimizer = dict(lr=0.01) # modify specific filed
```

Now only `lr` is modified:

```Python
cfg = Config.fromfile('resnet50_lr0.01.py')
cfg.optimizer  # ConfigDict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001)
```

### Delete

```

```Python
```

### Delete

For configurations already set in previous configs, if you wish to modify some specific argument and delete the remainders(in other words, discard the previous and redefine the module), you can set `_delete_=True`.

```Python
_base_ = ['optimizer_cfg.py', 'runtime_cfg.py']
optimizer = dict(_delete_=True, type='SGD', lr=0.01) # discard the previous and redefine the module
```

Now only `type` and `lr` are kept:

```Python
cfg.optimizer  # ConfigDict(type='SGD', lr=0.01)
```

```{note}
If you wish to learn more about advanced usages of the config system, please refer to [MMEngine Config](https://mmengine.readthedocs.io/en/latest/tutorials/config.html).
```

# 自定义模型

我们简单地把模型的各个组件分为五类：

- 主干网络 (backbone)：通常是一个用来提取特征图 (feature map) 的全卷积网络 (FCN network)，例如：ResNet, MobileNet。
- Neck：主干网络和 Head 之间的连接部分，例如：FPN, PAFPN。
- Head：用于具体任务的组件，例如：边界框预测和掩码预测。
- 区域提取器 (roi extractor)：从特征图中提取 RoI 特征，例如：RoI Align。
- 损失 (loss)：在 Head 组件中用于计算损失的部分，例如：FocalLoss, L1Loss, GHMLoss.

## 开发新的组件

### 添加一个新的主干网络

这里，我们以 MobileNet 为例来展示如何开发新组件。

#### 1. 定义一个新的主干网络（以 MobileNet 为例）

新建一个文件 `mmdet/models/backbones/mobilenet.py`

```python

class MobileNet(nn.Module):

    def __init__(self, arg1, arg2):
        pass

    def forward(self, x):  # should return a tuple
        pass
```

#### 2. 导入该模块

你可以添加下述代码到 `mmdet/models/backbones/__init__.py`

```python
from .mobilenet import MobileNet
```

或添加：

```python
    imports=['mmdet.models.backbones.mobilenet'],
```

到配置文件以避免原始代码被修改。

#### 3. 在你的配置文件中使用该主干网络

```python

```

或添加：

```python
```

```python
    ...
        type='MobileNet',
        arg1=xxx,
        arg2=xxx),
    ...
```

### 添加新的 Neck

#### 1. 定义一个 Neck（以 PAFPN 为例）

新建一个文件 `mmdet/models/necks/pafpn.py`

```python

class PAFPN(nn.Module):

                num_outs,
        pass

        pass
```

你可以添加下述代码到 `mmdet/models/necks/__init__.py`

```python
from .pafpn import PAFPN
```

或添加：

```python
    imports=['mmdet.models.necks.pafpn'],

pass
```

```python
```

或添加：

```python
```

#### 3. 修改配置文件

```python
    type='PAFPN',
```

### 添加新的 Head

我们以 [Double Head R-CNN](https://arxiv.org/abs/1904.06493) 为例来展示如何添加一个新的 Head。

首先，添加一个新的 bbox head 到 `mmdet/models/roi_heads/bbox_heads/double_bbox_head.py`。
Double Head R-CNN 在目标检测上实现了一个新的 bbox head。为了实现 bbox head，我们需要使用如下的新模块中三个函数。

```python
from typing import Tuple

from mmcv.cnn import ConvModule
from mmengine.model import BaseModule, ModuleList
from torch import Tensor

from mmdet.models.backbones.resnet import Bottleneck
from mmdet.utils import ConfigType, MultiConfig, OptConfigType, OptMultiConfig

from .bbox_head import BBoxHead

class DoubleConvFCBBoxHead(BBoxHead):
    r"""Bbox head used in Double-Head R-CNN

    .. code-block:: none

                                          /-> cls
                      /-> shared convs ->
                                          \-> reg
        roi features
                                          /-> cls
                      \-> shared fc    ->
                                          \-> reg
    """  # noqa: W605

                 num_convs: int = 0,
                 num_fcs: int = 0,
                 conv_out_channels: int = 1024,
                 fc_out_channels: int = 1024,
                 conv_cfg: OptConfigType = None,
                 norm_cfg: ConfigType = dict(type='BN'),

conv_out_channels: int = 1024,
                 init_cfg: MultiConfig = dict(
                     type='Normal',
                     override=[
                         dict(type='Normal', name='fc_cls', std=0.01),
                         dict(type='Normal', name='fc_reg', std=0.001),
                         dict(
                             type='Xavier',
                             name='fc_branch',
                             distribution='uniform')
                     ]),
                 **kwargs) -> None:
        kwargs.setdefault('with_avg_pool', True)
        super().__init__(init_cfg=init_cfg, **kwargs)

    def forward(self, x_cls: Tensor, x_reg: Tensor) -> Tuple[Tensor]:

```

然后，如有必要，实现一个新的 bbox head。我们打算从 `StandardRoIHead` 来继承新的 `DoubleHeadRoIHead`。我们可以发现 `StandardRoIHead` 已经实现了下述函数。

```python

def forward(self, x_cls: Tensor, x_reg: Tensor) -> Tuple[Tensor]:

```

```python
from typing import List, Optional, Tuple

from mmdet.registry import MODELS, TASK_UTILS
from mmdet.structures import DetDataSample
from mmdet.structures.bbox import bbox2roi
from mmdet.utils import ConfigType, InstanceList
from ..task_modules.samplers import SamplingResult
from ..utils import empty_instances, unpack_gt_instances
from .base_roi_head import BaseRoIHead

class StandardRoIHead(BaseRoIHead):
    """Simplest base roi head including one bbox head and one mask head."""

    def init_assigner_sampler(self) -> None:

    def init_bbox_head(self, bbox_roi_extractor: ConfigType,
                       bbox_head: ConfigType) -> None:

    def init_mask_head(self, mask_roi_extractor: ConfigType,

def init_bbox_head(self, bbox_roi_extractor: ConfigType,

                       mask_head: ConfigType) -> None:

    def forward(self, x: Tuple[Tensor],
                rpn_results_list: InstanceList) -> tuple:

    def loss(self, x: Tuple[Tensor], rpn_results_list: InstanceList,
             batch_data_samples: List[DetDataSample]) -> dict:

    def _bbox_forward(self, x: Tuple[Tensor], rois: Tensor) -> dict:

    def bbox_loss(self, x: Tuple[Tensor],
                  sampling_results: List[SamplingResult]) -> dict:

    def mask_loss(self, x: Tuple[Tensor],
                  sampling_results: List[SamplingResult], bbox_feats: Tensor,
                  batch_gt_instances: InstanceList) -> dict:

    def _mask_forward(self,
                      x: Tuple[Tensor],
                      rois: Tensor = None,
                      pos_inds: Optional[Tensor] = None,

def _mask_forward(self,
                      bbox_feats: Optional[Tensor] = None) -> dict:

    def predict_bbox(self,
                     x: Tuple[Tensor],
                     batch_img_metas: List[dict],
                     rpn_results_list: InstanceList,
                     rcnn_test_cfg: ConfigType,
                     rescale: bool = False) -> InstanceList:

    def predict_mask(self,
                     results_list: InstanceList,

```

Double Head 的修改主要在 bbox_forward 的逻辑中，且它从 `StandardRoIHead` 中继承了其他逻辑。在 `mmdet/models/roi_heads/double_roi_head.py` 中，我们用下述代码实现新的 bbox head：

```python

```python

from .standard_roi_head import StandardRoIHead

class DoubleHeadRoIHead(StandardRoIHead):
    """RoI head for `Double Head RCNN <https://arxiv.org/abs/1904.06493>`_.

    Args:
        reg_roi_scale_factor (float): The scale factor to extend the rois
            used to extract the regression features.
    """

    def __init__(self, reg_roi_scale_factor: float, **kwargs):
        super().__init__(**kwargs)
        self.reg_roi_scale_factor = reg_roi_scale_factor

        """Box head forward function used in both training and testing.

        Args:
            x (tuple[Tensor]): List of multi-level img features.
            rois (Tensor): RoIs with the shape (n, 5) where the first
                column indicates batch id of each RoI.

        Returns:

rois (Tensor): RoIs with the shape (n, 5) where the first

        Returns:
             dict[str, Tensor]: Usually returns a dictionary with keys:

                - `cls_score` (Tensor): Classification scores.
                - `bbox_pred` (Tensor): Box energies / deltas.
                - `bbox_feats` (Tensor): Extract bbox RoI features.
        """
        bbox_cls_feats = self.bbox_roi_extractor(
            x[:self.bbox_roi_extractor.num_inputs], rois)
        bbox_reg_feats = self.bbox_roi_extractor(
            x[:self.bbox_roi_extractor.num_inputs],
            rois,
            roi_scale_factor=self.reg_roi_scale_factor)
        if self.with_shared_head:
            bbox_cls_feats = self.shared_head(bbox_cls_feats)
            bbox_reg_feats = self.shared_head(bbox_reg_feats)
        cls_score, bbox_pred = self.bbox_head(bbox_cls_feats, bbox_reg_feats)

        bbox_results = dict(

bbox_reg_feats = self.shared_head(bbox_reg_feats)

            cls_score=cls_score,
            bbox_pred=bbox_pred,
            bbox_feats=bbox_cls_feats)
        return bbox_results
```

最终，用户需要把该模块添加到 `mmdet/models/bbox_heads/__init__.py` 和 `mmdet/models/roi_heads/__init__.py` 以使相关的注册表可以找到并加载他们。

或者，用户可以添加：

```python
custom_imports=dict(
    imports=['mmdet.models.roi_heads.double_roi_head', 'mmdet.models.roi_heads.bbox_heads.double_bbox_head'])
```

到配置文件并实现相同的目的。

Double Head R-CNN 的配置文件如下：

```python
_base_ = '../faster_rcnn/faster-rcnn_r50_fpn_1x_coco.py'
        type='DoubleHeadRoIHead',
        reg_roi_scale_factor=1.3,
        bbox_head=dict(
            _delete_=True,
            type='DoubleConvFCBBoxHead',
            num_fcs=2,
            conv_out_channels=1024,

_delete_=True,
            fc_out_channels=1024,
            roi_feat_size=7,
            num_classes=80,
            bbox_coder=dict(
                type='DeltaXYWHBBoxCoder',
                target_means=[0., 0., 0., 0.],
                target_stds=[0.1, 0.1, 0.2, 0.2]),
            reg_class_agnostic=False,
            loss_cls=dict(
                type='CrossEntropyLoss', use_sigmoid=False, loss_weight=2.0),
            loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=2.0))))

```

从 MMDetection 2.0 版本起，配置系统支持继承配置以使用户可以专注于修改。
Double Head R-CNN 主要使用了一个新的 `DoubleHeadRoIHead` 和一个新的 `DoubleConvFCBBoxHead`，参数需要根据每个模块的 `__init__` 函数来设置。

### 添加新的损失

假设你想添加一个新的损失 `MyLoss` 用于边界框回归。
为了添加一个新的损失函数，用户需要在 `mmdet/models/losses/my_loss.py` 中实现。
装饰器 `weighted_loss` 可以使损失每个部分加权。

```python

### 添加新的损失

```python

from mmdet.registry import LOSSES

                pred,
                target,

assert reduction_override in (None, 'none', 'mean', 'sum')
        loss_bbox = self.loss_weight * my_loss(
        return loss_bbox
```

然后，用户需要把它加到 `mmdet/models/losses/__init__.py`。

```python
```

或者，你可以添加：

```python
    imports=['mmdet.models.losses.my_loss'])
```

到配置文件来实现相同的目的。

如使用，请修改 `loss_xxx` 字段。
因为 MyLoss 是用于回归的，你需要在 Head 中修改 `loss_xxx` 字段。

```python
loss_bbox=dict(type='MyLoss', loss_weight=1.0))
```

# Rethinking on multi-stage networks for human pose estimation

<details>
<summary align="right"><a href="https://arxiv.org/abs/1901.00148">MSPN (ArXiv'2019)</a></summary>

```bibtex
@article{li2019rethinking,
  title={Rethinking on Multi-Stage Networks for Human Pose Estimation},
  author={Li, Wenbo and Wang, Zhicheng and Yin, Binyi and Peng, Qixiang and Du, Yuming and Xiao, Tianzi and Yu, Gang and Lu, Hongtao and Wei, Yichen and Sun, Jian},
  journal={arXiv preprint arXiv:1901.00148},
}
```

journal={arXiv preprint arXiv:1901.00148},
}
```

Existing pose estimation approaches fall into two categories: single-stage and multi-stage methods. While multi-stage methods are seemingly more suited for the task, their performance in current practice is not as good as single-stage methods. This work studies this issue. We argue that the current multi-stage methods' unsatisfactory performance comes from the insufficiency in various design choices. We propose several improvements, including the single-stage module design, cross stage feature aggregation, and coarse-to-fine supervision. The resulting method establishes the new state-of-the-art on both MS COCO and MPII Human Pose dataset, justifying the effectiveness of a multi-stage architecture. The source code is publicly available for further research.

<img src="https://user-images.githubusercontent.com/15977946/146520722-74c1ef85-9fa3-4b96-8cbf-4a2c86a80adc.png">
</div>

# Benchmark

## Latency benchmark

### Platform

- Cuda 11.3

### Other settings

- Static graph
- Batch size 1
- Synchronize devices after each inference.
- We count the average inference performance of 100 images of the dataset.
- Warm up. For ncnn, we warm up 30 iters for all codebases. As for other backends: for classification, we warm up 1010 iters; for other codebases, we warm up 10 iters.
- Input resolution varies for different datasets of different codebases. All inputs are real images except for `mmagic` because the dataset is not large enough.

Users can directly test the speed through [model profiling](../02-how-to-run/profile_model.md). And here is the benchmark in our environment.

<thead>
  <tr>

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
  </tr>
  <tr>

<td align="center" colspan="1">Ascend310</td>
  </tr>
  <tr>
  </tr>
  <tr>

<td align="center">59.32</td>
  </tr>
  <tr>
  </tr>
  <tr>

</tr>
  <tr>
  </tr>
  <tr>

<td align="center">1.19</td>
  </tr>
</tbody>
</table>
<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
  </tr>
  <tr>

</tr>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>

<td align="center">8.84</td>
  </tr>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>

<td align="center">84.06</td>
  </tr>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>

<td align="center">88.08</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/mask_rcnn/mask_rcnn_r50_fpn_1x_coco.py">Mask R-CNN</a></td>
  </tr>
</tbody>
</table>
</div>

<table>
<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
  </tr>
  <tr>

<td align="center" rowspan="2">spatial</td>
  </tr>
  <tr>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/yolo/yolov3_mobilenetv2_mstrain-416_300e_coco.py">MobileNetv2-YOLOv3</a></td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/ssd/ssdlite_mobilenetv2_scratch_600e_coco.py">SSD-Lite</a></td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/yolox/yolox_tiny_8x8_300e_coco.py">YOLOX</a></td>

<td align="center">66.19</td>
  </tr>
  <tr>
  </tr>
</tbody>
</table>
</div>

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
  </tr>
  <tr>

<td align="center" colspan="1">T4</td>
  </tr>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>

<td align="center">32x32</td>
  </tr>
</tbody>
</table>
</div>

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
  </tr>
  <tr>

<td align="center" colspan="1">Adreno660</td>
  </tr>
  <tr>
  </tr>
    <tr>
  </tr>
  <tr>

</tbody>
</table>
</div>

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>

  </tr>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>

<td align="center">119.77</td>
  </tr>
  <tr>
  </tr>
  <tr>
  </tr>
</tbody>

<td align="center">151.25</td>
  </tr>
</tbody>
</table>
</div>

## Performance benchmark

Users can directly test the performance through [how_to_evaluate_a_model.md](../02-how-to-run/profile_model.md). And here is the benchmark in our environment.

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>

<td align="center">model</td>
  </tr>
  <tr>
  </tr>
  <tr>

</tr>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>

</tr>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>

</tr>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>

</tr>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>

</tr>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>

</tr>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>

</tr>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>

</tr>
  <tr>
  <tr>
  </tr>
  <tr>

<td align="center">top-5</td>
  </tr>
</tbody>
</table>
</div>
<thead>
  <tr>
    <th algin="center">OpenVINO</th>
  </tr>
</thead>
<tbody>
  <tr>

<td align="center">dataset</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/yolo/yolov3_d53_320_273e_coco.py">YOLOV3</a></td>
  </tr>
  <tr>

<td align="center">33.5</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/ssd/ssd300_coco.py">SSD</a></td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/retinanet/retinanet_r50_fpn_1x_coco.py">RetinaNet</a></td>

  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/fcos/fcos_r50_caffe_fpn_gn-head_1x_coco.py">FCOS</a></td>
  </tr>
  <tr>

<td align="center">36.6</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/fsaf/fsaf_r50_fpn_1x_coco.py">FSAF</a></td>
  </tr>
  <tr>

  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/yolox/yolox_s_8x8_300e_coco.py">YOLOX</a></td>
  </tr>
  <tr>

<td align="center">40.3</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py">Faster R-CNN</a></td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/atss/atss_r50_fpn_1x_coco.py">ATSS</a></td>

  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/cascade_rcnn/cascade_rcnn_r50_caffe_fpn_1x_coco.py">Cascade R-CNN</a></td>
  </tr>
  <tr>

<td align="center">40.4</td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/gfl/gfl_r50_fpn_1x_coco.py">GFL</a></td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/reppoints/reppoints_moment_r50_fpn_1x_coco.py">RepPoints</a></td>

  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/detr/detr_r50_8x2_150e_coco.py">DETR</a></td>
  </tr>
  <tr>

<td align="center">40.1</td>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/mask_rcnn/mask_rcnn_r50_fpn_1x_coco.py">Mask R-CNN</a></td>
  </tr>
  <tr>

</tr>
  <tr>
  </tr>
  <tr>
    <td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmdetection/blob/master/configs/swin/mask_rcnn_swin-t-p4-w7_fpn_1x_coco.py">Swin-Transformer</a></td>
  </tr>
  <tr>

<td align="center">42.5</td>
  </tr>
  <tr>
  </tr>
  <tr>

<td align="center">33.1</td>
  </tr>
  <tr>
  </tr>
</tbody>
</table>
</div>
<thead>
  <tr>

<td align="center">34.5</td>
  </tr>
</tbody>
</table>
</div>
<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
  </tr>
  <tr>

</tr>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>

<td align="center">0.8097</td>
  </tr>
  <tr>
  </tr>
  <tr>

<td align="center">0.7784</td>
  </tr>
  <tr>
  </tr>
  <tr>

<td align="center">0.8559</td>
  </tr>
  <tr>
  </tr>
  <tr>

</tr>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>

  </tr>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>

  </tr>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>

<td align="center">30.2214</td>
  </tr>
  <tr>
  </tr>
</tbody>
</table>
</div>
<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>

<th align="center">OpenVINO</th>
  </tr>
</thead>
<tbody>
  <tr>
  </tr>
  <tr>

<td align="center">0.7310</td>
  </tr>
  <tr>
  </tr>
  <tr>
  </tr>
 <tr>

  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmrotate/tree/main/configs/oriented_rcnn/oriented-rcnn-le90_r50_fpn_1x_dota.py">Oriented RCNN</a></td>
  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmrotate/blob/main/configs/gliding_vertex/gliding-vertex-rbox_r50_fpn_1x_dota.py">GlidingVertex</a></td>

  </tr>
  <tr>
    <td align="center"><a href="https://github.com/open-mmlab/mmrotate/blob/main/configs/roi_trans/roi-trans-le90_r50_fpn_1x_dota.py">RoI Transformer</a></td>
  </tr>
</tbody>
</table>
</div>

<thead>
  <tr>

  </tr>
  <tr>
  </tr>
</tbody>
</table>
</div>
## Notes

- As some datasets contain images with various resolutions in codebase like MMDet. The speed benchmark is gained through static configs in MMDeploy, while the performance benchmark is gained through dynamic ones.

- Some int8 performance benchmarks of TensorRT require Nvidia cards with tensor core, or the performance would drop heavily.

- DBNet uses the interpolate mode `nearest` in the neck of the model, which TensorRT-7 applies a quite different strategy from Pytorch. To make the repository compatible with TensorRT-7, we rewrite the neck to use the interpolate mode `bilinear` which improves final detection performance. To get the matched performance with Pytorch, TensorRT-8+ is recommended, which the interpolate methods are all the same as Pytorch.

- Mask AP of Mask R-CNN drops by 1% for the backend. The main reason is that the predicted masks are directly interpolated to original image in PyTorch, while they are at first interpolated to the preprocessed input image of the model and then to original image in other backends.

- MMPose models are tested with `flip_test` explicitly set to `False` in model configs.

- Some models might get low accuracy in fp16 mode. Please adjust the model to avoid value overflow.

# 自定义模型

我们通常把模型的各个组成成分分成 6 种类型：

- 编码器（encoder）：包括 voxel encoder 和 middle encoder 等进入 backbone 前所使用的基于体素的方法，如 `HardVFE` 和 `PointPillarsScatter`。
- 骨干网络（backbone）：通常采用 FCN 网络来提取特征图，如 `ResNet` 和 `SECOND`。
- 颈部网络（neck）：位于 backbones 和 heads 之间的组成模块，如 `FPN` 和 `SECONDFPN`。
- 检测头（head）：用于特定任务的组成模块，如`检测框的预测`和`掩码的预测`。
- RoI 提取器（RoI extractor）：用于从特征图中提取 RoI 特征的组成模块，如 `H3DRoIHead` 和 `PartAggregationROIHead`。
- 损失函数（loss）：heads 中用于计算损失函数的组成模块，如 `FocalLoss`、`L1Loss` 和 `GHMLoss`。

## 开发新的组成模块

### 添加新的编码器

接下来我们以 HardVFE 为例展示如何开发新的组成模块。

#### 1. 定义一个新的体素编码器（如 HardVFE：即 HV-SECOND 中使用的体素特征编码器）

创建一个新文件 `mmdet3d/models/voxel_encoders/voxel_encoder.py`。

```python

from mmdet3d.registry import MODELS

class HardVFE(nn.Module):

        pass

    def forward(self, x):  # 需要返回一个元组
        pass
```

您可以在 `mmdet3d/models/voxel_encoders/__init__.py` 中添加以下代码：

```python
from .voxel_encoder import HardVFE
```

pass

        pass
```

```python
```

或者在配置文件中添加以下代码，从而避免修改源码：

```python
    imports=['mmdet3d.models.voxel_encoders.voxel_encoder'],
```

#### 3. 在配置文件中使用体素编码器

```python
    ...
        type='HardVFE',
        arg1=xxx,
        arg2=yyy),
    ...
)
```

### 添加新的骨干网络

接下来我们以 [SECOND](https://www.mdpi.com/1424-8220/18/10/3337)（Sparsely Embedded Convolutional Detection）为例展示如何开发新的组成模块。

#### 1. 定义一个新的骨干网络（如 SECOND）

创建一个新文件 `mmdet3d/models/backbones/second.py`。

```python
from mmengine.model import BaseModule

class SECOND(BaseModule):

        pass

        pass
```

        pass

        pass
```

您可以在 `mmdet3d/models/backbones/__init__.py` 中添加以下代码：

```python
from .second import SECOND
```

```python
    imports=['mmdet3d.models.backbones.second'],
```

#### 3. 在配置文件中使用骨干网络

```python
    ...
        arg1=xxx,
    ...
)
```

### 添加新的颈部网络

#### 1. 定义一个新的颈部网络（如 SECONDFPN）

创建一个新文件 `mmdet3d/models/necks/second_fpn.py`。

```python

class SECONDFPN(BaseModule):

                 in_channels=[128, 128, 256],
                 out_channels=[256, 256, 256],
                 upsample_strides=[1, 2, 4],

                 norm_cfg=dict(type='BN', eps=1e-3, momentum=0.01),
                 upsample_cfg=dict(type='deconv', bias=False),
                 conv_cfg=dict(type='Conv2d', bias=False),
                 use_conv_for_no_stride=False,
        pass

        # 具体实现忽略
        pass
```

您可以在 `mmdet3d/models/necks/__init__.py` 中添加以下代码：

```python
from .second_fpn import SECONDFPN
```

```python
    imports=['mmdet3d.models.necks.second_fpn'],
```

#### 3. 在配置文件中使用颈部网络

```python
    ...

```python
    ...
    ...
)
```

### 添加新的检测头

接下来我们以 [PartA2 Head](https://arxiv.org/abs/1907.03670) 为例展示如何开发新的检测头。

**注意**：此处展示的 `PartA2 RoI Head` 将用于检测器的第二阶段。对于单阶段的检测头，请参考 `mmdet3d/models/dense_heads/` 中的例子。由于其简单高效，它们更常用于自动驾驶场景下的 3D 检测中。

首先，在 `mmdet3d/models/roi_heads/bbox_heads/parta2_bbox_head.py` 中添加新的 bbox head。`PartA2 RoI Head` 为目标检测实现了一个新的 bbox head。为了实现一个 bbox head，我们通常需要在新模块中实现如下两个函数。有时还需要实现其他相关函数，如 `loss` 和 `get_targets`。

```python

class PartA2BboxHead(BaseModule):
    """PartA2 RoI head."""

                 num_classes,
                 seg_in_channels,
                 part_in_channels,
                 seg_conv_channels=None,
                 part_conv_channels=None,

num_classes,
                 merge_conv_channels=None,
                 down_conv_channels=None,
                 shared_fc_channels=None,
                 cls_channels=None,
                 reg_channels=None,
                 dropout_ratio=0.1,
                 roi_feat_size=14,
                 with_corner_loss=True,
                 bbox_coder=dict(type='DeltaXYZWLHRBBoxCoder'),
                 conv_cfg=dict(type='Conv1d'),
                 norm_cfg=dict(type='BN1d', eps=1e-3, momentum=0.01),
                 loss_bbox=dict(
                     type='SmoothL1Loss', beta=1.0 / 9.0, loss_weight=2.0),
                 loss_cls=dict(
                     type='CrossEntropyLoss',
                     use_sigmoid=True,
                     reduction='none',
                     loss_weight=1.0),

        super(PartA2BboxHead, self).__init__(init_cfg=init_cfg)

    def forward(self, seg_feats, part_feats):
        pass
```

其次，如果有必要的话需要实现一个新的 RoI Head。我们从 `Base3DRoIHead` 中继承得到新的 `PartAggregationROIHead`。我们可以发现 `Base3DRoIHead` 已经实现了如下函数。

```python
from mmdet.models.roi_heads import BaseRoIHead

from mmdet3d.registry import MODELS, TASK_UTILS

class Base3DRoIHead(BaseRoIHead):
    """Base class for 3d RoIHeads."""

                 bbox_head=None,
                 bbox_roi_extractor=None,
                 mask_head=None,
                 mask_roi_extractor=None,
                 train_cfg=None,
                 test_cfg=None,
        super(Base3DRoIHead, self).__init__(

mask_roi_extractor=None,
            bbox_head=bbox_head,
            bbox_roi_extractor=bbox_roi_extractor,
            mask_head=mask_head,
            mask_roi_extractor=mask_roi_extractor,
            train_cfg=train_cfg,
            test_cfg=test_cfg,
            init_cfg=init_cfg)

    def init_bbox_head(self, bbox_roi_extractor: dict,
                       bbox_head: dict) -> None:
        """Initialize box head and box roi extractor.

        Args:
            bbox_roi_extractor (dict or ConfigDict): Config of box
                roi extractor.
            bbox_head (dict or ConfigDict): Config of box in box head.
        """
        self.bbox_roi_extractor = MODELS.build(bbox_roi_extractor)
        self.bbox_head = MODELS.build(bbox_head)

    def init_assigner_sampler(self):

"""

        """Initialize assigner and sampler."""
        self.bbox_assigner = None
        self.bbox_sampler = None
        if self.train_cfg:
            if isinstance(self.train_cfg.assigner, dict):
                self.bbox_assigner = TASK_UTILS.build(self.train_cfg.assigner)
            elif isinstance(self.train_cfg.assigner, list):
                self.bbox_assigner = [
                    TASK_UTILS.build(res) for res in self.train_cfg.assigner
                ]
            self.bbox_sampler = TASK_UTILS.build(self.train_cfg.sampler)

    def init_mask_head(self):
        """Initialize mask head, skip since ``PartAggregationROIHead`` does not
        have one."""
        pass
```

def init_mask_head(self):
        pass
```

接下来主要对 bbox_forward 的逻辑进行修改，同时其继承了来自 `Base3DRoIHead` 的其它逻辑。在 `mmdet3d/models/roi_heads/part_aggregation_roi_head.py` 中，我们实现了新的 RoI Head，如下所示：

```python
from typing import Dict, List, Tuple

from mmdet.models.task_modules import AssignResult, SamplingResult
from mmengine import ConfigDict
from torch.nn import functional as F

from mmdet3d.structures import bbox3d2roi
from mmdet3d.utils import InstanceList
from ...structures.det3d_data_sample import SampleList
from .base_3droi_head import Base3DRoIHead

class PartAggregationROIHead(Base3DRoIHead):
    """Part aggregation roi head for PartA2.

    Args:
        semantic_head (ConfigDict): Config of semantic head.
        num_classes (int): The number of classes.

"""Part aggregation roi head for PartA2.

    Args:
        seg_roi_extractor (ConfigDict): Config of seg_roi_extractor.
        bbox_roi_extractor (ConfigDict): Config of part_roi_extractor.
        bbox_head (ConfigDict): Config of bbox_head.
        train_cfg (ConfigDict): Training config.
        test_cfg (ConfigDict): Testing config.
    """

                 semantic_head: dict,
                 num_classes: int = 3,
                 seg_roi_extractor: dict = None,
                 bbox_head: dict = None,
                 bbox_roi_extractor: dict = None,
                 train_cfg: dict = None,
                 test_cfg: dict = None,
                 init_cfg: dict = None) -> None:
        super(PartAggregationROIHead, self).__init__(

init_cfg: dict = None) -> None:
        assert semantic_head is not None
        self.init_seg_head(seg_roi_extractor, semantic_head)

    def init_seg_head(self, seg_roi_extractor: dict,
                      semantic_head: dict) -> None:
        """Initialize semantic head and seg roi extractor.

        Args:
            seg_roi_extractor (dict): Config of seg
            semantic_head (dict): Config of semantic head.
        """
        self.semantic_head = MODELS.build(semantic_head)
        self.seg_roi_extractor = MODELS.build(seg_roi_extractor)

    @property
    def with_semantic(self):
        """bool: whether the head has semantic branch"""

self.seg_roi_extractor = MODELS.build(seg_roi_extractor)

    @property
        return hasattr(self,
                       'semantic_head') and self.semantic_head is not None

    def predict(self,
                feats_dict: Dict,
                rpn_results_list: InstanceList,
                batch_data_samples: SampleList,
                rescale: bool = False,
                **kwargs) -> InstanceList:
        """Perform forward propagation of the roi head and predict detection
        results on the features of the upstream network.

        Args:
            feats_dict (dict): Contains features from the first stage.
            rpn_results_list (List[:obj:`InstanceData`]): Detection results
                of rpn head.
            batch_data_samples (List[:obj:`Det3DDataSample`]): The Data
                samples. It usually includes information such as

of rpn head.
                `gt_instance_3d`, `gt_panoptic_seg_3d` and `gt_sem_seg_3d`.
            rescale (bool): If True, return boxes in original image space.
                Defaults to False.

        Returns:
            list[:obj:`InstanceData`]: Detection results of each sample
            after the post process.
            Each item usually contains following keys.

            - scores_3d (Tensor): Classification scores, has a shape
              (num_instances, )
            - labels_3d (Tensor): Labels of bboxes, has a shape
              (num_instances, ).
            - bboxes_3d (BaseInstance3DBoxes): Prediction of bboxes,
              contains a tensor with shape (num_instances, C), where
              C >= 7.
        """
        assert self.with_bbox, 'Bbox head must be implemented in PartA2.'

contains a tensor with shape (num_instances, C), where
              C >= 7.
        """
        assert self.with_semantic, 'Semantic head must be implemented' \
                                   ' in PartA2.'

        batch_input_metas = [
            data_samples.metainfo for data_samples in batch_data_samples
        ]
        voxels_dict = feats_dict.pop('voxels_dict')
        # TODO: Split predict semantic and bbox
        results_list = self.predict_bbox(feats_dict, voxels_dict,
                                         batch_input_metas, rpn_results_list,
                                         self.test_cfg)
        return results_list

    def predict_bbox(self, feats_dict: Dict, voxel_dict: Dict,
                     batch_input_metas: List[dict],
                     test_cfg: ConfigDict) -> InstanceList:

batch_input_metas: List[dict],
        """Perform forward propagation of the bbox head and predict detection

        Args:
            voxel_dict (dict): Contains information of voxels.
            batch_input_metas (list[dict], Optional): Batch image meta info.
                Defaults to None.
            test_cfg (Config): Test config.

        Returns:

after the post process.

              C >= 7.
        """
        ...

    def loss(self, feats_dict: Dict, rpn_results_list: InstanceList,
             batch_data_samples: SampleList, **kwargs) -> dict:
        """Perform forward propagation and loss calculation of the detection
        roi on the features of the upstream network.

        Args:

rpn_results_list (List[:obj:`InstanceData`]): Detection results

        Returns:
            dict[str, Tensor]: A dictionary of loss components
        """
        assert len(rpn_results_list) == len(batch_data_samples)
        losses = dict()
        batch_gt_instances_3d = []
        batch_gt_instances_ignore = []
        for data_sample in batch_data_samples:
            batch_gt_instances_3d.append(data_sample.gt_instances_3d)
            if 'ignored_instances' in data_sample:
                batch_gt_instances_ignore.append(data_sample.ignored_instances)
            else:
                batch_gt_instances_ignore.append(None)
        if self.with_semantic:

batch_gt_instances_ignore.append(data_sample.ignored_instances)
            else:
            semantic_results = self._semantic_forward_train(
                feats_dict, voxels_dict, batch_gt_instances_3d)
            losses.update(semantic_results.pop('loss_semantic'))

        sample_results = self._assign_and_sample(rpn_results_list,
                                                 batch_gt_instances_3d)
        if self.with_bbox:
            feats_dict.update(semantic_results)
            bbox_results = self._bbox_forward_train(feats_dict, voxels_dict,
                                                    sample_results)
            losses.update(bbox_results['loss_bbox'])

        return losses
```

此处我们省略了相关函数的更多细节。更多细节请参考[代码](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/models/roi_heads/part_aggregation_roi_head.py)。

return losses
```

最后，用户需要在 `mmdet3d/models/roi_heads/bbox_heads/__init__.py` 和 `mmdet3d/models/roi_heads/__init__.py` 添加模块，从而能被相应的注册器找到并加载。

此外，用户也可以在配置文件中添加以下代码以达到相同的目的。

```python
    imports=['mmdet3d.models.roi_heads.part_aggregation_roi_head', 'mmdet3d.models.roi_heads.bbox_heads.parta2_bbox_head'],
```

`PartAggregationROIHead` 的配置文件如下所示：

```python
    ...
        type='PartAggregationROIHead',
        semantic_head=dict(
            type='PointwiseSemanticHead',
            in_channels=16,
            extra_width=0.2,
            seg_score_thr=0.3,
            num_classes=3,
            loss_seg=dict(
                type='mmdet.FocalLoss',
                use_sigmoid=True,
                reduction='sum',

seg_score_thr=0.3,
                gamma=2.0,
                alpha=0.25,
                loss_weight=1.0),
            loss_part=dict(
                type='mmdet.CrossEntropyLoss',
                loss_weight=1.0)),
        seg_roi_extractor=dict(
            type='Single3DRoIAwareExtractor',
            roi_layer=dict(
                type='RoIAwarePool3d',
                out_size=14,
                max_pts_per_voxel=128,
                mode='max')),
        bbox_roi_extractor=dict(
                mode='avg')),
            type='PartA2BboxHead',

out_size=14,
            seg_in_channels=16,
            part_in_channels=4,
            seg_conv_channels=[64, 64],
            part_conv_channels=[64, 64],
            merge_conv_channels=[128, 128],
            down_conv_channels=[128, 256],
            bbox_coder=dict(type='DeltaXYZWLHRBBoxCoder'),
            shared_fc_channels=[256, 512, 512, 512],
            cls_channels=[256, 256],
            reg_channels=[256, 256],
            dropout_ratio=0.1,
            roi_feat_size=14,
            with_corner_loss=True,
            loss_bbox=dict(
                type='mmdet.SmoothL1Loss',
                beta=1.0 / 9.0,

reduction='sum',
                loss_weight=1.0))),
    ...
)
```

MMDetection 2.0 开始支持配置文件之间的继承，因此用户可以关注配置文件的修改。PartA2 Head 的第二阶段主要使用了新的 `PartAggregationROIHead` 和 `PartA2BboxHead`，需要根据对应模块的 `__init__` 函数来设置参数。

### 添加新的损失函数

假设您想要为检测框的回归添加一个新的损失函数 `MyLoss`。为了添加一个新的损失函数，用户需要在 `mmdet3d/models/losses/my_loss.py` 中实现该函数。装饰器 `weighted_loss` 能够保证对每个元素的损失进行加权平均。

```python
from mmdet.models.losses.utils import weighted_loss

return loss

                pred,
                target,
```

接下来，用户需要在 `mmdet3d/models/losses/__init__.py` 添加该函数。

```python
```

或者在配置文件中添加以下代码以达到相同的目的。

```python
    imports=['mmdet3d.models.losses.my_loss'],
```

```python
```

```python
```

为了使用该函数，用户需要修改 `loss_xxx` 域。由于 `MyLoss` 是用于回归的，您需要修改 head 中的 `loss_bbox` 域。

```python
loss_bbox=dict(type='MyLoss', loss_weight=1.0)
```

# OpenFlamingo

### Prepare the environment

Install [MMPretrain](https://github.com/open-mmlab/mmpretrain) according to this [doc](https://mmpretrain.readthedocs.io/en/latest/get_started.html#installation)

### Start evaluation

#### Slurm

```sh
cd $root
python run.py configs/multimodal/tasks.py --mm-eval --slurm -p $PARTITION
```

#### PyTorch

```sh
cd $root
python run.py configs/multimodal/tasks.py --mm-eval
```

# Preparing DPDD Dataset

```bibtex
}
```

The test datasets can be download from [here](https://drive.google.com/file/d/1dDWUQ_D93XGtcywoUcZE1HOXCV4EuLyw/).

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── DPDD
|   |   ├── inputL
|   |   ├── inputR
|   |   ├── inputC
```

# EDSR (CVPR'2017)

> [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://arxiv.org/abs/1707.02921)

Recent research on super-resolution has progressed with the development of deep convolutional neural networks (DCNN). In particular, residual learning techniques exhibit improved performance. In this paper, we develop an enhanced deep super-resolution network (EDSR) with performance exceeding those of current state-of-the-art SR methods. The significant performance improvement of our model is due to optimization by removing unnecessary modules in conventional residual networks. The performance is further improved by expanding the model size while we stabilize the training procedure. We also propose a new multi-scale deep super-resolution system (MDSR) and training method, which can reconstruct high-resolution images of different upscaling factors in a single model. The proposed methods show superior performance over the state-of-the-art methods on benchmark datasets and prove its excellence by winning the NTIRE2017 Super-Resolution Challenge.

 <img src="https://user-images.githubusercontent.com/7676947/144018090-ed629948-bf68-43ff-b2a9-6213e23f19a5.png" width="400"/>
</div >

Evaluated on RGB channels, `scale` pixels in each border are cropped before evaluation.

|                                Model                                 | Dataset |  PSNR   |  SSIM  | Training Resources |                                            Download                                            |
| :------------------------------------------------------------------: | :-----: | :-----: | :----: | :----------------: | :--------------------------------------------------------------------------------------------: |

| [edsr_x2c64b16_1x16_300k_div2k](./edsr_x2c64b16_1xb16-300k_div2k.py) |  Set5   | 35.7592 | 0.9372 |         1          | [model](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x2c64b16_1x16_300k_div2k_20200604-19fe95ea.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x2c64b16_1x16_300k_div2k_20200604_221933.log.json) |
| [edsr_x2c64b16_1x16_300k_div2k](./edsr_x2c64b16_1xb16-300k_div2k.py) |  Set14  | 31.4290 | 0.8874 |         1          | [model](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x2c64b16_1x16_300k_div2k_20200604-19fe95ea.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x2c64b16_1x16_300k_div2k_20200604_221933.log.json) |

| [edsr_x2c64b16_1x16_300k_div2k](./edsr_x2c64b16_1xb16-300k_div2k.py) |  DIV2K  | 34.5896 | 0.9352 |         1          | [model](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x2c64b16_1x16_300k_div2k_20200604-19fe95ea.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x2c64b16_1x16_300k_div2k_20200604_221933.log.json) |
| [edsr_x3c64b16_1x16_300k_div2k](./edsr_x3c64b16_1xb16-300k_div2k.py) |  Set5   | 32.3301 | 0.8912 |         1          | [model](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x3c64b16_1x16_300k_div2k_20200608-36d896f4.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x3c64b16_1x16_300k_div2k_20200608_114850.log.json) |

| [edsr_x3c64b16_1x16_300k_div2k](./edsr_x3c64b16_1xb16-300k_div2k.py) |  Set14  | 28.4125 | 0.8022 |         1          | [model](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x3c64b16_1x16_300k_div2k_20200608-36d896f4.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x3c64b16_1x16_300k_div2k_20200608_114850.log.json) |
| [edsr_x3c64b16_1x16_300k_div2k](./edsr_x3c64b16_1xb16-300k_div2k.py) |  DIV2K  | 30.9154 | 0.8711 |         1          | [model](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x3c64b16_1x16_300k_div2k_20200608-36d896f4.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x3c64b16_1x16_300k_div2k_20200608_114850.log.json) |

| [edsr_x4c64b16_1x16_300k_div2k](./edsr_x4c64b16_1xb16-300k_div2k.py) |  Set5   | 30.2223 | 0.8500 |         1          | [model](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x4c64b16_1x16_300k_div2k_20200608-3c2af8a3.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x4c64b16_1x16_300k_div2k_20200608_115148.log.json) |
| [edsr_x4c64b16_1x16_300k_div2k](./edsr_x4c64b16_1xb16-300k_div2k.py) |  Set14  | 26.7870 | 0.7366 |         1          | [model](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x4c64b16_1x16_300k_div2k_20200608-3c2af8a3.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x4c64b16_1x16_300k_div2k_20200608_115148.log.json) |

| [edsr_x4c64b16_1x16_300k_div2k](./edsr_x4c64b16_1xb16-300k_div2k.py) |  DIV2K  | 28.9675 | 0.8172 |         1          | [model](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x4c64b16_1x16_300k_div2k_20200608-3c2af8a3.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x4c64b16_1x16_300k_div2k_20200608_115148.log.json) |

**Train**

<details>

```shell
# cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/edsr/edsr_x4c64b16_1xb16-300k_div2k.py

python tools/train.py configs/edsr/edsr_x4c64b16_1xb16-300k_div2k.py

./tools/dist_train.sh configs/edsr/edsr_x4c64b16_1xb16-300k_div2k.py 8
```

**Test**

<details>

```

**Test**

<details>

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/edsr/edsr_x4c64b16_1xb16-300k_div2k.py https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x4c64b16_1x16_300k_div2k_20200608-3c2af8a3.pth

python tools/test.py configs/edsr/edsr_x4c64b16_1xb16-300k_div2k.py https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x4c64b16_1x16_300k_div2k_20200608-3c2af8a3.pth

./tools/dist_test.sh configs/edsr/edsr_x4c64b16_1xb16-300k_div2k.py https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x4c64b16_1x16_300k_div2k_20200608-3c2af8a3.pth 8
```

```

```bibtex
@inproceedings{lim2017enhanced,
  title={Enhanced deep residual networks for single image super-resolution},
  author={Lim, Bee and Son, Sanghyun and Kim, Heewon and Nah, Seungjun and Mu Lee, Kyoung},
  booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition workshops},
  pages={136--144},
}
```

# 可视化

MMSegmentation 1.x 提供了简便的方式监控训练时的状态以及可视化在模型预测时的数据。

## 训练状态监控

MMSegmentation 1.x 使用 TensorBoard 来监控训练时候的状态。

### TensorBoard 的配置

安装 TensorBoard 的过程可以按照 [官方安装指南](https://www.tensorflow.org/install) ，具体的步骤如下：

```shell
pip install tensorboardX
pip install future tensorboard
```

在配置文件 `default_runtime.py` 的 `vis_backend` 中添加 `TensorboardVisBackend`。

```python
vis_backends = [dict(type='LocalVisBackend'),
                dict(type='TensorboardVisBackend')]
    type='SegLocalVisualizer', vis_backends=vis_backends, name='visualizer')
```

### 检查 TensorBoard 中的标量

启动训练实验的命令如下

```shell
python tools/train.py configs/pspnet/pspnet_r50-d8_4xb4-80k_ade20k-512x512.py --work-dir work_dir/test_visual
```

开始训练后找到 `work_dir` 中的 `vis_data` 路径，例如：本次特定测试的 vis_data 路径如下所示：

```shell
work_dirs/test_visual/20220810_115248/vis_data
```

```

```shell
```

vis_data 路径中的标量文件包括了学习率、损失函数和 data_time 等，还记录了指标结果，您可以参考 MMEngine 中的 [记录日志教程](https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/logging.html) 中的日志教程来帮助记录自己定义的数据。 Tensorboard 的可视化结果使用下面的命令执行：

```shell
tensorboard --logdir work_dirs/test_visual/20220810_115248/vis_data
```

## 数据和结果的可视化

### 模型测试或验证期间的可视化数据样本

MMSegmentation 提供了 `SegVisualizationHook` ，它是一个可以用于可视化 ground truth 和在模型测试和验证期间的预测分割结果的[钩子](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/hook.html) 。 它的配置在 `default_hooks` 中，更多详细信息请参见 [执行器教程](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/runner.html)。

例如，在 `_base_/schedules/schedule_20k.py` 中，修改 `SegVisualizationHook` 配置，将 `draw` 设置为 `True` 以启用网络推理结果的存储，`interval` 表示预测结果的采样间隔， 设置为 1 时，将保存网络的每个推理结果。 `interval` 默认设置为 50：

```python

```python
    checkpoint=dict(type='CheckpointHook', by_epoch=False, interval=2000),
    visualization=dict(type='SegVisualizationHook', draw=True, interval=1))

```

启动训练实验后，可视化结果将在 validation loop 存储到本地文件夹中，或者在一个数据集上启动评估模型时，预测结果将存储在本地。本地的可视化的存储结果保存在 `$WORK_DIRS/vis_data` 下的 `vis_image` 中，例如：

```shell
work_dirs/test_visual/20220810_115248/vis_data/vis_image
```

另外，如果在 `vis_backends` 中添加 `TensorboardVisBackend` ，如 [TensorBoard 的配置](###TensorBoard的配置)，我们还可以运行下面的命令在 TensorBoard 中查看它们：

```shell
```

### 可视化单个数据样本

如果你想可视化单个样本数据，我们建议使用 `SegLocalVisualizer` 。

```shell
```

`SegLocalVisualizer`是继承自 MMEngine 中`Visualizer` 类的子类，适用于 MMSegmentation 可视化，有关`Visualizer`的详细信息请参考在 MMEngine 中的[可视化教程](https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/visualization.html) 。

以下是一个关于 `SegLocalVisualizer` 的示例，首先你可以使用下面的命令下载这个案例中的数据：

<img src="https://user-images.githubusercontent.com/24582831/189833109-eddad58f-f777-4fc0-b98a-6bd429143b06.png" width="70%"/>
</div>

```shell
wget https://user-images.githubusercontent.com/24582831/189833109-eddad58f-f777-4fc0-b98a-6bd429143b06.png --output-document aachen_000000_000019_leftImg8bit.png
wget https://user-images.githubusercontent.com/24582831/189833143-15f60f8a-4d1e-4cbb-a6e7-5e2233869fac.png --output-document aachen_000000_000019_gtFine_labelTrainIds.png
```

然后你可以找到他们本地的路径和使用下面的脚本文件对其进行可视化：

```python

```

```python

# `PixelData` 是 MMEngine 中用于定义像素级标注或预测的数据结构。
# 请参考下面的MMEngine数据结构教程文件：
# https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/data_element.html#pixeldata

from mmengine.structures import PixelData

# `SegDataSample` 是在 MMSegmentation 中定义的不同组件之间的数据结构接口，
# 它包括 ground truth、语义分割的预测结果和预测逻辑。
# 详情请参考下面的 `SegDataSample` 教程文件：
# https://github.com/open-mmlab/mmsegmentation/blob/1.x/docs/en/advanced_guides/structures.md

from mmseg.structures import SegDataSample
from mmseg.visualization import SegLocalVisualizer

out_file = 'out_file_cityscapes'
save_dir = './work_dirs'

image = mmcv.imread(
    osp.join(
        osp.dirname(__file__),
        './aachen_000000_000019_leftImg8bit.png'
    ),
    'color')
sem_seg = mmcv.imread(
    osp.join(
        './aachen_000000_000019_gtFine_labelTrainIds.png'  # noqa
    ),
    'unchanged')

),
    'color')
    osp.join(
    ),
sem_seg = torch.from_numpy(sem_seg)
gt_sem_seg_data = dict(data=sem_seg)
gt_sem_seg = PixelData(**gt_sem_seg_data)
data_sample = SegDataSample()
data_sample.gt_sem_seg = gt_sem_seg

seg_local_visualizer = SegLocalVisualizer(
    save_dir=save_dir)

# 数据集的元信息通常包括类名的 `classes` 和
# 用于可视化每个前景颜色的 `palette` 。
# 所有类名和调色板都在此文件中定义：
# https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/utils/class_names.py

seg_local_visualizer.dataset_meta = dict(
    classes=('road', 'sidewalk', 'building', 'wall', 'fence',
             'pole', 'traffic light', 'traffic sign',
             'vegetation', 'terrain', 'sky', 'person', 'rider',
             'car', 'truck', 'bus', 'train', 'motorcycle',
             'bicycle'),
    palette=[[128, 64, 128], [244, 35, 232], [70, 70, 70],

'car', 'truck', 'bus', 'train', 'motorcycle',
             [102, 102, 156], [190, 153, 153], [153, 153, 153],
             [250, 170, 30], [220, 220, 0], [107, 142, 35],
             [152, 251, 152], [70, 130, 180], [220, 20, 60],
             [255, 0, 0], [0, 0, 142], [0, 0, 70],
             [0, 60, 100], [0, 80, 100], [0, 0, 230],
             [119, 11, 32]])

# 当`show=True`时，直接显示结果，
# 当 `show=False`时，结果将保存在本地文件夹中。

seg_local_visualizer.add_datasample(out_file, image,
                                    data_sample, show=False)
```

可视化后的图像结果和它的对应的 ground truth 图像可以在 `./work_dirs/vis_data/vis_image/` 路径找到，文件名字是：`out_file_cityscapes_0.png` ：

<img src="https://user-images.githubusercontent.com/24582831/189835713-c0534054-4bfa-4b75-9254-0afbeb5ff02e.png" width="70%"/>
</div>

</div>

如果你想知道更多的关于可视化的使用指引，你可以参考 MMEngine 中的[可视化教程](<[https://mmengine.readthedocs.io/en/latest/advanced_tutorials/visualization.html](https://github.com/open-mmlab/mmengine/blob/main/docs/zh_cn/advanced_tutorials/visualization.md)>)

# PyTorch 2.0 Compatibility and Benchmark

PyTorch introduced `torch.compile` in its 2.0 release. It compiles your model to speedup trainning & validation. We provide a benchmark result and compatibility of typical models in MMAction2. Except for one model (MViT) that fails to compile, the performance of other models remains consistent before and after compilation.

| Config                                                                    | compiled | Train time / iter (s) | GPU memory (M) | test metric  |
| ------------------------------------------------------------------------- | -------- | --------------------- | -------------- | ------------ |
| tsn_imagenet-pretrained-r50_8xb32-1x1x16-50e_sthv2-rgb                    | False    | 0.50                  | 42537          | 36.55        |
| tsn_imagenet-pretrained-r50_8xb32-1x1x16-50e_sthv2-rgb                    | True     | 0.61                  | 53149          | 36.72        |

| timesformer_divST_8xb8-8x32x1-15e_kinetics400-rgb                         | False    | 0.688                 | 14263          | 77.69        |
| timesformer_divST_8xb8-8x32x1-15e_kinetics400-rgb                         | True     | 0.691                 | 13863          | 77.57        |
| stgcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-2d                          | False    | 0.0305                | 1184           | 91.69        |
| stgcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-2d                          | True     | 0.0298                | 1273           | 91.64        |
| slowonly_r50_8xb16-u48-240e_ntu60-xsub-keypoint                           | False    | 0.498                 | 9581           | 93.6         |

| slowonly_r50_8xb16-u48-240e_ntu60-xsub-keypoint                           | True     | 0.505                 | 11968          | 93.49        |
| slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb            | False    | 0.17                  | 8278           | 20.76        |
| slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb            | True     | 0.1835                | 12004          | 21.67        |
| swin-tiny-p244-w877_in1k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb          | False    | 0.323                 | 21651          | 78.90        |
| swin-tiny-p244-w877_in1k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb          | True     | 0.262                 | 20905          | 78.70        |

| slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb | False    | 0.098                 | 5777           | 75.12        |
| slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb | True     | 0.0942                | 7095           | 75.15        |
| mvit-small-p244_32xb16-16x4x1-200e_kinetics400-rgb                        | Fail     | incompatible          | incompatible   | incompatible |

# ViTDet

This is an implementation of [ViTDet](https://github.com/facebookresearch/detectron2/tree/main/projects/ViTDet) based on [MMDetection](https://github.com/facebookresearch/detectron2/tree/main/projects/ViTDet), [MMCV](https://github.com/open-mmlab/mmcv), and [MMEngine](https://github.com/open-mmlab/mmengine).

## Usage

Follow original [setting](https://github.com/facebookresearch/detectron2/tree/main/projects/ViTDet), this project is trained with total batch size of 64 (16 GPU with 4 images per GPU).

```bash
GPUS=${GPUS} ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} ${CONFIG_FILE} ${WORK_DIR}
```

Below is an example of using 16 GPUs to train VitDet on a Slurm partition named _dev_, and set the work-dir to some shared file systems.

```shell

```

```shell
GPUS=16 ./tools/slurm_train.sh dev vitdet_mask_b projects/ViTDet/configs/vitdet_mask-rcnn_vit-b-mae_lsj-100e.py /nfs/xxxx/vitdet_mask-rcnn_vit-b-mae_lsj-100e
```

```bash
python tools/test.py projects/ViTDet/configs/vitdet_mask-rcnn_vit-b-mae_lsj-100e.py ${CHECKPOINT_PATH}
```

## Results

Based on mmdetection, this project almost aligns the test and train accuracy of the [ViTDet](https://github.com/facebookresearch/detectron2/tree/main/projects/ViTDet).

```

## Results

|                           Method                           | Backbone | Pretrained Model |  Training set  |   Test set   | Epoch | Val Box AP | Val Mask  AP |                                                                                                                                                               Download                                                                                                                                                                |

| :--------------------------------------------------------: | :------: | :--------------: | :------------: | :----------: | :---: | :--------: | :----------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| [ViTDet](./configs/vitdet_mask-rcnn_vit-b-mae_lsj-100e.py) |  ViT-B   |       MAE        | COCO2017 Train | COCO2017 Val |  100  |    51.6    |     45.7     | [model](https://download.openmmlab.com/mmdetection/v3.0/vitdet/vitdet_mask-rcnn_vit-b-mae_lsj-100e/vitdet_mask-rcnn_vit-b-mae_lsj-100e_20230328_153519-e15fe294.pth) / [log](https://download.openmmlab.com/mmdetection/v3.0/vitdet/vitdet_mask-rcnn_vit-b-mae_lsj-100e/vitdet_mask-rcnn_vit-b-mae_lsj-100e_20230328_153519.log.json) |

**Note**:

**Note**:

1. The mask AP is lower than official [repo](https://github.com/facebookresearch/detectron2/tree/main/projects/ViTDet) slightly
2. other model vision will release code and weights in the future

```latex
@article{li2022exploring,
  title={Exploring plain vision transformer backbones for object detection},
  author={Li, Yanghao and Mao, Hanzi and Girshick, Ross and He, Kaiming},
  journal={arXiv preprint arXiv:2203.16527},
}
```

    <!-- It will be parsed by MIM and Inferencer. [Example](https://github.com/open-mmlab/mmdetection/blob/3.x/configs/faster_rcnn/metafile.yml) -->

  <!-- In particular, you may have to refactor this README into a standard one. [Example](https://github.com/open-mmlab/mmdetection/blob/3.x/configs/faster_rcnn/README.md) -->

## 2D Hand Keypoint Demo

本节我们继续通过 demo 脚本演示对单张图片或者视频的 2D 手部关键点的识别。同样的，用户仍要确保开发环境已经安装了 3.0 版本以上的 [MMDetection](https://github.com/open-mmlab/mmdetection) 。

我们在 [mmdet model zoo](/demo/docs/zh_cn/mmdet_modelzoo.md#手部-bounding-box-识别模型) 提供了预训练好的手部 Bounding Box 预测模型，用户可以前往下载。

### 2D 手部图片关键点识别

```shell
    --input ${INPUT_PATH} [--output-root ${OUTPUT_DIR}] \
    [--show] [--device ${GPU_ID or CPU}] [--save-predictions] \
    [--kpt-thr ${KPT_SCORE_THR}] [--bbox-thr ${BBOX_SCORE_THR}]
```

用户可以在 [model zoo](https://mmpose.readthedocs.io/zh_CN/dev-1.x/model_zoo/hand_2d_keypoint.html) 获取预训练好的关键点识别模型。

这里我们用 [onehand10k model](https://download.openmmlab.com/mmpose/hand/hrnetv2/hrnetv2_w18_onehand10k_256x256-30bc9c6b_20210330.pth) 来进行演示：

```shell

```shell
    demo/mmdetection_cfg/cascade_rcnn_x101_64x4d_fpn_1class.py \
    https://download.openmmlab.com/mmpose/mmdet_pretrained/cascade_rcnn_x101_64x4d_fpn_20e_onehand10k-dac19597_20201030.pth \
    configs/hand_2d_keypoint/topdown_heatmap/onehand10k/td-hm_hrnetv2-w18_8xb64-210e_onehand10k-256x256.py \
    https://download.openmmlab.com/mmpose/hand/hrnetv2/hrnetv2_w18_onehand10k_256x256-30bc9c6b_20210330.pth \
    --input tests/data/onehand10k/9.jpg \
    --show --draw-heatmap
```

可视化结果如下：

<img src="https://user-images.githubusercontent.com/26127467/187664103-cfbe0c4e-5876-42f9-9023-5fb58ce00d7b.jpg" height="500px" alt><br>

如果使用了 heatmap-based 模型同时设置了 `--draw-heatmap` ，预测的热图也会跟随关键点一同可视化出来。

如果想本地保存可视化结果可使用如下命令：

```shell

```shell
    --output-root vis_results --show --draw-heatmap
```

如果想本地保存预测结果，需要添加 `--save-predictions` 。

如果想用 CPU 进行 demo 需添加 `--device cpu` ：

```shell

demo/mmdetection_cfg/cascade_rcnn_x101_64x4d_fpn_1class.py \
    --show --draw-heatmap  --device cpu
```

### 2D 手部视频关键点识别推理

视频和图片使用了同样的接口，区别在于视频推理时 `${INPUT_PATH}` 既可以是本地视频文件的路径也可以是视频文件的 **URL** 地址。

```shell

configs/hand_2d_keypoint/topdown_heatmap/onehand10k/td-hm_hrnetv2-w18_8xb64-210e_onehand10k-256x256.py \
    --input demo/resources/<demo_hand.mp4> \
```

<img src="https://user-images.githubusercontent.com/26127467/187665873-3ac836ec-8da5-45e1-8d78-c0abe962bd5e.gif" height="500px" alt><br>

这段视频可以在 [Google Drive](https://raw.githubusercontent.com/open-mmlab/mmpose/master/tests/data/nvgesture/sk_color.avi) 下载到。

### 使用 Inferencer 进行 2D 手部关键点识别推理

Inferencer 提供一个更便捷的推理接口，使得用户可以绕过模型的配置文件和 checkpoint 路径直接使用 model aliases ，支持包括图片路径、视频路径、图片文件夹路径和 webcams 在内的多种输入方式，例如可以这样使用：

```shell
python demo/inferencer_demo.py tests/data/onehand10k \
    --pose2d hand --vis-out-dir vis_results/onehand10k \
    --bbox-thr 0.5 --kpt-thr 0.05
```

该命令会对输入的 `tests/data/onehand10k` 下所有的图片进行推理并且把可视化结果都存入 `vis_results/onehand10k` 文件夹下。

--pose2d hand --vis-out-dir vis_results/onehand10k \
```

<img src="https://user-images.githubusercontent.com/26127467/229824447-b444e92d-9b5b-4a50-9a32-68be3ff8c527.jpg" alt="Image 1" height="200"/> <img src="https://user-images.githubusercontent.com/26127467/229824466-6ae47a40-70a6-451d-94ee-4ffc34204a9c.jpg" alt="Image 2" height="200"/> <img src="https://user-images.githubusercontent.com/26127467/229824477-679201c3-1e0b-45fe-b0c7-bab67b245a10.jpg" alt="Image 3" height="200"/> <img src="https://user-images.githubusercontent.com/26127467/229824488-bd874362-7401-41a5-8209-51bad1563a11.jpg" alt="Image 4" height="200"/>

除此之外， Inferencer 也支持保存预测的姿态结果。具体信息可在 [Inferencer 文档](https://mmpose.readthedocs.io/zh_CN/dev-1.x/user_guides/inference.html) 查看。

### 加速推理

### 加速推理

对于 2D 手部关键点预测模型，用户可以通过修改配置文件中的 `model.test_cfg.flip_test=False` 来加速，如 [onehand10k_hrnetv2](../../configs/hand_2d_keypoint/topdown_heatmap/onehand10k/td-hm_hrnetv2-w18_8xb64-210e_onehand10k-256x256.py#90) 所示。

# Pretraining boosts out-of-domain robustness for pose estimation

<details>
<summary align="right"><a href="https://openaccess.thecvf.com/content/WACV2021/html/Mathis_Pretraining_Boosts_Out-of-Domain_Robustness_for_Pose_Estimation_WACV_2021_paper.html">Horse-10 (WACV'2021)</a></summary>

```bibtex
@inproceedings{mathis2021pretraining,
  title={Pretraining boosts out-of-domain robustness for pose estimation},
  author={Mathis, Alexander and Biasi, Thomas and Schneider, Steffen and Yuksekgonul, Mert and Rogers, Byron and Bethge, Matthias and Mathis, Mackenzie W},
  booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
  pages={1859--1868},
}
```

> **任务**: 视频超分辨率

<details>
<summary align="right"><a href="https://arxiv.org/abs/2104.13371">BasicVSR++ (CVPR'2022)</a></summary>

```bibtex
}
```

SPyNet 的 预训练权重在[这里](https://download.openmmlab.com/mmediting/restorers/basicvsr/spynet_20210409-c6c1bd09.pth)。

|  算法   | REDS4 (BIx4) PSNR/SSIM (RGB) | Vimeo-90K-T (BIx4) PSNR/SSIM (Y) | Vid4 (BIx4) PSNR/SSIM (Y) | UDM10 (BDx4) PSNR/SSIM (Y) | Vimeo-90K-T (BDx4) PSNR/SSIM (Y) | Vid4 (BDx4) PSNR/SSIM (Y) |  GPU 信息   |    Download    |

| :-----: | :--------------------------: | :------------------------------: | :-----------------------: | :------------------------: | :------------------------------: | :-----------------------: | :---------: | :------------: |
| [basicvsr_plusplus_c64n7_8x1_600k_reds4](./basicvsr-pp_c64n7_8xb1-600k_reds4.py) |      **32.3855/0.9069**      |          36.4445/0.9411          |      27.7674/0.8444       |       34.6868/0.9417       |          34.0372/0.9244          |      24.6209/0.7540       | 8 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_600k_reds4_20210217-db622b2f.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_600k_reds4_20210217_113115.log.json) |

| [basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bi](./basicvsr-pp_c64n7_4xb2-300k_vimeo90k-bi.py) |        31.0126/0.8804        |        **37.7864/0.9500**        |    **27.7882/0.8401**     |       33.1211/0.9270       |          33.8972/0.9195          |      23.6086/0.7033       | 4 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bi_20210305-4ef437e2.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bi_20210305_141254.log.json) |

| [basicvsr_plusplus_c64n7_4x2_300k_vimeo90k_bd](./basicvsr-pp_c64n7_4xb2-300k_vimeo90k-bd.py) |        29.2041/0.8528        |          34.7248/0.9351          |      26.4377/0.8074       |     **40.7216/0.9722**     |        **38.2054/0.9550**        |    **29.0400/0.8753**     | 4 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bd_20210305-ab315ab1.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c64n7_8x1_300k_vimeo90k_bd_20210305_140921.log.json) |

<details>
<summary align="left">NTIRE 2021 模型权重文件</summary>

请注意，以下模型是从较小的模型中微调而来的。 这些模型的训练方案将在 MMagic 达到 5k star 时发布。 我们在这里提供预训练的模型。

| 算法                                                                   | 模型                                                                   | 赛道                                                         |

| ---------------------------------------------------------------------- | ---------------------------------------------------------------------- | ------------------------------------------------------------ |
| [basicvsr-pp_c128n25_600k_ntire-vsr](./basicvsr-pp_c128n25_600k_ntire-vsr.py) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c128n25_ntire_vsr_20210311-1ff35292.pth) | NTIRE 2021 Video Super-Resolution                            |
| [basicvsr-pp_c128n25_600k_ntire-decompress-track1](./basicvsr-pp_c128n25_600k_ntire-decompress-track1.py) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c128n25_ntire_decompress_track1_20210223-7b2eba02.pth) | NTIRE 2021 Quality Enhancement of Compressed Video - Track 1 |

| [basicvsr-pp_c128n25_600k_ntire-decompress-track2](./basicvsr-pp_c128n25_600k_ntire-decompress-track2.py) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c128n25_ntire_decompress_track2_20210314-eeae05e6.pth) | NTIRE 2021 Quality Enhancement of Compressed Video - Track 2 |
| [basicvsr-pp_c128n25_600k_ntire-decompress-track3](./basicvsr-pp_c128n25_600k_ntire-decompress-track3.py) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c128n25_ntire_decompress_track3_20210304-6daf4a40.pth) | NTIRE 2021 Quality Enhancement of Compressed Video - Track 3 |

```

## 快速开始

**训练**

<details>

```shell
# CPU上训练

# 单个GPU上训练

# 多个GPU上训练

# 单个GPU上训练

# 多个GPU上训练
```

**测试**

<details>

```shell
# CPU上测试

# 单个GPU上测试

# 多个GPU上测试

# 多个GPU上测试
```

# Migration of Optimizers

We have merged [MMGeneration 1.x](https://github.com/open-mmlab/mmgeneration/tree/1.x) into MMagic. Here is migration of Optimizers about MMGeneration.

In version 0.x, MMGeneration uses PyTorch's native Optimizer, which only provides general parameter optimization.
In version 1.x, we use `OptimizerWrapper` provided by MMEngine.

Compared to PyTorch's `Optimizer`, `OptimizerWrapper` supports the following features:

- `OptimizerWrapper.update_params` implement `zero_grad`, `backward` and `step` in a single function.
- Support gradient accumulation automatically.
- Provide a context manager named `OptimizerWrapper.optim_context` to warp the forward process. `optim_context` can automatically call `torch.no_sync` according to current number of updating iteration. In AMP (auto mixed precision) training, `autocast` is called in `optim_context` as well.

For GAN models, generator and discriminator use different optimizer and training schedule.

To ensure that the GAN model's function signature of `train_step` is consistent with other models, we use `OptimWrapperDict`, inherited from `OptimizerWrapper`, to wrap the optimizer of the generator and discriminator.
To automate this process MMagic implement `MultiOptimWrapperContructor`.
And you should specify this constructor in your config is you want to train GAN model.

The config for the 0.x and 1.x versions are shown below:

<thead>
  <tr>
<tbody>
<tr>

```python
    generator=dict(type='Adam', lr=0.0001, betas=(0.0, 0.999), eps=1e-6),
    discriminator=dict(type='Adam', lr=0.0004, betas=(0.0, 0.999), eps=1e-6))
```

</td>

```python
    constructor='MultiOptimWrapperConstructor',

discriminator=dict(type='Adam', lr=0.0004, betas=(0.0, 0.999), eps=1e-6))
```

</td>

```python
    generator=dict(optimizer=dict(type='Adam', lr=0.0002, betas=(0.0, 0.999), eps=1e-6)),
    discriminator=dict(
        optimizer=dict(type='Adam', lr=0.0004, betas=(0.0, 0.999), eps=1e-6)))
```

</td>

</tr>
</thead>
</table>

> Note that, in the 1.x, MMGeneration uses `OptimWrapper` to realize gradient accumulation. This make the config of `discriminator_steps` (training trick for updating the generator once after multiple updates of the discriminator) and gradient accumulation different between 0.x and 1.x version.

- In 0.x version,  we use `disc_steps`, `gen_steps` and `batch_accumulation_steps` in configs. `disc_steps` and `batch_accumulation_steps` are counted by the number of calls of `train_step` (is also the number of data reads from the dataloader). Therefore the number of consecutive updates of the discriminator is `disc_steps // batch_accumulation_steps`. And for generators, `gen_steps` is the number of times the generator actually updates continuously.
- In 1.x version, we use `discriminator_steps`, `generator_steps` and `accumulative_counts` in configs. `discriminator_steps` and `generator_steps` are the number of consecutive updates to itself before updating other modules.

Take config of BigGAN-128 as example.

<thead>
  <tr>
<tbody>
<tr>

```python
    type='BasiccGAN',
    generator=dict(
        type='BigGANGenerator',
        output_scale=128,
        noise_size=120,

<tbody>
<tr>

```python
        num_classes=1000,
        base_channels=96,
        shared_dim=128,
        with_shared_embedding=True,
        sn_eps=1e-6,
        init_type='ortho',
        act_cfg=dict(type='ReLU', inplace=True),
        split_noise=True,
        auto_sync_bn=False),
        type='BigGANDiscriminator',
        input_scale=128,
        with_spectral_norm=True),
    gan_loss=dict(type='GANLoss', gan_type='hinge'))

# continuous update discriminator for `disc_steps // batch_accumulation_steps = 8 // 8 = 1` times
# continuous update generator for `gen_steps = 1` times

# generators and discriminators perform `batch_accumulation_steps = 8` times gradient accumulations before each update
    disc_steps=8, gen_steps=1, batch_accumulation_steps=8, use_ema=True)
```

</td>

```python
    type='BigGAN',
    num_classes=1000,
    data_preprocessor=dict(type='DataPreprocessor'),

split_noise=True,
    # continuous update discriminator for `discriminator_steps = 1` times
    # continuous update generator for `generator_steps = 1` times
    generator_steps=1,
    discriminator_steps=1)

        # generator perform `accumulative_counts = 8` times gradient accumulations before each update
        accumulative_counts=8,
        optimizer=dict(type='Adam', lr=0.0001, betas=(0.0, 0.999), eps=1e-6)),
        # discriminator perform `accumulative_counts = 8` times gradient accumulations before each update

discriminator=dict(
```

</td>

</tr>
</thead>
</table>

# Adding New Data Transforms

## Customization data transformation

The customized data transformation must inherited from `BaseTransform` and implement `transform` function.
Here we use a simple flipping transformation as example:

```python
import random
from mmcv.transforms import BaseTransform, TRANSFORMS

class MyFlip(BaseTransform):
    def __init__(self, direction: str):
        self.direction = direction

    def transform(self, results: dict) -> dict:
        img = results['img']
        results['img'] = mmcv.imflip(img, direction=self.direction)
```

Moreover, import the new class.

```python
from .my_pipeline import MyFlip
```

Thus, we can instantiate a `MyFlip` object and use it to process the data dict.

```python

transform = MyFlip(direction='horizontal')
data_dict = {'img': np.random.rand(224, 224, 3)}
data_dict = transform(data_dict)

```python

processed_img = data_dict['img']
```

Or, we can use `MyFlip` transformation in data pipeline in our config file.

```python
    ...
    dict(type='MyFlip', direction='horizontal'),
    ...
]
```

Note that if you want to use `MyFlip` in config, you must ensure the file containing `MyFlip` is imported during runtime.

    <sup>
      </a>
    </sup>
    <sup>
      </a>
    </sup>
  </div>

[👀Model Zoo](https://mmaction2.readthedocs.io/en/latest/modelzoo_statistics.html) |
[🆕Update News](https://mmaction2.readthedocs.io/en/latest/notes/changelog.html) |
[🚀Ongoing Projects](https://github.com/open-mmlab/mmaction2/projects) |
[🤔Reporting Issues](https://github.com/open-mmlab/mmaction2/issues/new/choose)

</div>

</div>

- [📄 Table of Contents](#-table-of-contents)
- [🥳 🚀 What's New](#--whats-new-)
- [📖 Introduction](#-introduction-)
- [🎁 Major Features](#-major-features-)
- [🛠️ Installation](#️-installation-)

- [👀 Model Zoo](#-model-zoo-)
- [👨‍🏫 Get Started](#-get-started-)
- [🎫 License](#-license-)
- [🖊️ Citation](#️-citation-)
- [🙌 Contributing](#-contributing-)
- [🤝 Acknowledgement](#-acknowledgement-)
- [🏗️ Projects in OpenMMLab](#️-projects-in-openmmlab-)

## 🥳 🚀 What's New [🔝](#-table-of-contents)

**The default branch has been switched to `main`(previous `1.x`) from `master`(current `0.x`), and we encourage users to migrate to the latest version with more supported models, stronger pre-training checkpoints and simpler coding. Please refer to [Migration Guide](https://mmaction2.readthedocs.io/en/latest/migration.html) for more details.**

**Release (2023.07.04)**: v1.1.0 with the following new features:

- Support [Pure Python style Configuration File](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#a-pure-python-style-configuration-file-beta) and downloading datasets by MIM with one command

## 📖 Introduction [🔝](#-table-of-contents)

MMAction2 is an open-source toolbox for video understanding based on PyTorch.
It is a part of the [OpenMMLab](http://openmmlab.com/) project.

  <p style="font-size:1.5vw;"> Action Recognition on Kinetics-400 (left) and Skeleton-based Action Recognition on NTU-RGB+D-120 (right)</p>
</div>

    <p style="font-size:1.5vw;">Skeleton-based Spatio-Temporal Action Detection and Action Recognition Results on Kinetics-400</p>
</div>

</div>
    <p style="font-size:1.5vw;">Spatio-Temporal Action Detection Results on AVA-2.1</p>
</div>

## 🎁 Major Features [🔝](#-table-of-contents)

- **Modular design**: We decompose a video understanding framework into different components. One can easily construct a customized video understanding framework by combining different modules.

- **Support five major video understanding tasks**: MMAction2 implements various algorithms for multiple video understanding tasks, including action recognition, action localization, spatio-temporal action detection, skeleton-based action detection and video retrieval.

- **Well tested and documented**: We provide detailed documentation and API reference, as well as unit tests.

## 🛠️ Installation [🔝](#-table-of-contents)

MMAction2 depends on [PyTorch](https://pytorch.org/), [MMCV](https://github.com/open-mmlab/mmcv), [MMEngine](https://github.com/open-mmlab/mmengine), [MMDetection](https://github.com/open-mmlab/mmdetection) (optional) and [MMPose](https://github.com/open-mmlab/mmpose) (optional).

Please refer to [install.md](https://mmaction2.readthedocs.io/en/latest/get_started/installation.html) for detailed instructions.

<summary>Quick instructions</summary>

```shell
conda install pytorch torchvision -c pytorch  # This command will automatically install the latest version PyTorch and cudatoolkit, please check whether they match your environment.
mim install mmdet  # optional

mim install mmpose  # optional
```

## 👀 Model Zoo [🔝](#-table-of-contents)

Results and models are available in the [model zoo](https://mmaction2.readthedocs.io/en/latest/model_zoo/modelzoo.html).

<summary>Supported model</summary>

  <tr>
    <td colspan="5" style="font-weight:bold;">Action Recognition</td>
  </tr>
  <tr>

<td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/tsn/README.md">TSN</a> (ECCV'2016)</td>
  </tr>
  <tr>

<td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/tsm/README.md">TSM</a> (ICCV'2019)</td>
  </tr>
  <tr>

<td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/tpn/README.md">TPN</a> (CVPR'2020)</td>
  </tr>
  <tr>
  </tr>
  <tr>

<td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/swin/README.md">VideoSwin</a> (CVPR'2022)</td>
  </tr>
  <tr>
  </tr>
  <tr>
    <td colspan="5" style="font-weight:bold;">Action Localization</td>
  </tr>
  <tr>

</tr>
  <tr>
  </tr>
  <tr>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td colspan="5" style="font-weight:bold;">Spatio-Temporal Action Detection</td>
  </tr>
  <tr>

<td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/detection/slowonly/README.md">SlowOnly+Fast R-CNN</a> (ICCV'2019)</td>
  </tr>
  <tr>
    <td colspan="5" style="font-weight:bold;">Skeleton-based Action Recognition</td>
  </tr>
  <tr>

<td><a href="https://github.com/open-mmlab/mmaction2/blob/main/configs/skeleton/2s-agcn/README.md">2s-AGCN</a> (CVPR'2019)</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td colspan="5" style="font-weight:bold;">Video Retrieval</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>

<tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>

</table>

<summary>Supported dataset</summary>

  <tr>
    <td colspan="4" style="font-weight:bold;">Action Recognition</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/hmdb51/README.md">HMDB51</a> (<a href="https://serre-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/">Homepage</a>) (ICCV'2011)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/ucf101/README.md">UCF101</a> (<a href="https://www.crcv.ucf.edu/research/data-sets/ucf101/">Homepage</a>) (CRCV-IR-12-01)</td>

<td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/activitynet/README.md">ActivityNet</a> (<a href="http://activity-net.org/">Homepage</a>) (CVPR'2015)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/kinetics/README.md">Kinetics-[400/600/700]</a> (<a href="https://deepmind.com/research/open-source/kinetics/">Homepage</a>) (CVPR'2017)</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/sthv2/README.md">SthV2</a> (<a href="https://developer.qualcomm.com/software/ai-datasets/something-something">Homepage</a>) (ICCV'2017)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/diving48/README.md">Diving48</a> (<a href="http://www.svcl.ucsd.edu/projects/resound/dataset.html">Homepage</a>) (ECCV'2018)</td>

<td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/jester/README.md">Jester</a> (<a href="https://developer.qualcomm.com/software/ai-datasets/jester">Homepage</a>) (ICCV'2019)</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/mit/README.md">Moments in Time</a> (<a href="http://moments.csail.mit.edu/">Homepage</a>) (TPAMI'2019)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/mmit/README.md">Multi-Moments in Time</a> (<a href="http://moments.csail.mit.edu/challenge_iccv_2019.html">Homepage</a>) (ArXiv'2019)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/hvu/README.md">HVU</a> (<a href="https://github.com/holistic-video-understanding/HVU-Dataset">Homepage</a>) (ECCV'2020)</td>

<td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/omnisource/README.md">OmniSource</a> (<a href="https://kennymckormick.github.io/omnisource/">Homepage</a>) (ECCV'2020)</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/gym/README.md">FineGYM</a> (<a href="https://sdolivia.github.io/FineGym/">Homepage</a>) (CVPR'2020)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/kinetics710/README.md">Kinetics-710</a> (<a href="https://arxiv.org/pdf/2211.09552.pdf">Homepage</a>) (Arxiv'2022)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td colspan="4" style="font-weight:bold;">Action Localization</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/thumos14/README.md">THUMOS14</a> (<a href="https://www.crcv.ucf.edu/THUMOS14/download.html">Homepage</a>) (THUMOS Challenge 2014)</td>

    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/hacs/README.md">HACS</a> (<a href="https://github.com/hangzhaomit/HACS-dataset">Homepage</a>) (ICCV'2019)</td>
    <td></td>
  </tr>
  <tr>
    <td colspan="4" style="font-weight:bold;">Spatio-Temporal Action Detection</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/ucf101_24/README.md">UCF101-24*</a> (<a href="http://www.thumos.info/download.html">Homepage</a>) (CRCV-IR-12-01)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/jhmdb/README.md">JHMDB*</a> (<a href="http://jhmdb.is.tue.mpg.de/">Homepage</a>) (ICCV'2015)</td>

<td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/jhmdb/README.md">JHMDB*</a> (<a href="http://jhmdb.is.tue.mpg.de/">Homepage</a>) (ICCV'2015)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/ava/README.md">AVA</a> (<a href="https://research.google.com/ava/index.html">Homepage</a>) (CVPR'2018)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/ava_kinetics/README.md">AVA-Kinetics</a> (<a href="https://research.google.com/ava/index.html">Homepage</a>) (Arxiv'2020)</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/multisports/README.md">MultiSports</a> (<a href="https://deeperaction.github.io/datasets/multisports.html">Homepage</a>) (ICCV'2021)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td colspan="4" style="font-weight:bold;">Skeleton-based Action Recognition</td>
  </tr>
  <tr>

<td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/skeleton/README.md">PoseC3D-FineGYM</a> (<a href="https://kennymckormick.github.io/posec3d/">Homepage</a>) (ArXiv'2021)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/skeleton/README.md">PoseC3D-NTURGB+D</a> (<a href="https://kennymckormick.github.io/posec3d/">Homepage</a>) (ArXiv'2021)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/skeleton/README.md">PoseC3D-UCF101</a> (<a href="https://kennymckormick.github.io/posec3d/">Homepage</a>) (ArXiv'2021)</td>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/skeleton/README.md">PoseC3D-HMDB51</a> (<a href="https://kennymckormick.github.io/posec3d/">Homepage</a>) (ArXiv'2021)</td>
  </tr>
  <tr>

</tr>
  <tr>
    <td colspan="4" style="font-weight:bold;">Video Retrieval</td>
  </tr>
  <tr>
    <td><a href="https://github.com/open-mmlab/mmaction2/blob/main/tools/data/video_retrieval/README.md">MSRVTT</a> (<a href="https://www.microsoft.com/en-us/research/publication/msr-vtt-a-large-video-description-dataset-for-bridging-video-and-language/">Homepage</a>) (CVPR'2016)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>

</table>

## 👨‍🏫 Get Started [🔝](#-table-of-contents)

For tutorials, we provide the following user guides for basic usage:

- [Migration from MMAction2 0.X](https://mmaction2.readthedocs.io/en/latest/migration.html)
- [Learn about Configs](https://mmaction2.readthedocs.io/en/latest/user_guides/config.html)
- [Prepare Datasets](https://mmaction2.readthedocs.io/en/latest/user_guides/prepare_dataset.html)
- [Inference with Existing Models](https://mmaction2.readthedocs.io/en/latest/user_guides/inference.html)

- [Training and Testing](https://mmaction2.readthedocs.io/en/latest/user_guides/train_test.html)

<summary>Research works built on MMAction2 by users from community</summary>

## 🎫 License [🔝](#-table-of-contents)

## 🖊️ Citation [🔝](#-table-of-contents)

```BibTeX
}
```

## 🙌 Contributing [🔝](#-table-of-contents)

We appreciate all contributions to improve MMAction2. Please refer to [CONTRIBUTING.md](https://github.com/open-mmlab/mmcv/blob/2.x/CONTRIBUTING.md) in MMCV for more details about the contributing guideline.

## 🤝 Acknowledgement [🔝](#-table-of-contents)

MMAction2 is an open-source project that is contributed by researchers and engineers from various colleges and companies.

We appreciate all the contributors who implement their methods or add new features and users who give valuable feedback.
We wish that the toolbox and benchmark could serve the growing research community by providing a flexible toolkit to reimplement existing methods and develop their new models.

## 🏗️ Projects in OpenMMLab [🔝](#-table-of-contents)

# TinyViT

> [TinyViT: Fast Pretraining Distillation for Small Vision Transformers](https://arxiv.org/abs/2207.10666)

Vision transformer (ViT) recently has drawn great attention in computer vision due to its remarkable model capability. However, most prevailing ViT models suffer from huge number of parameters, restricting their applicability on devices with limited resources. To alleviate this issue, we propose TinyViT, a new family of tiny and efficient small vision transformers pretrained on large-scale datasets with our proposed fast distillation framework. The central idea is to transfer knowledge from large pretrained models to small ones, while enabling small models to get the dividends of massive pretraining data. More specifically, we apply distillation during pretraining for knowledge transfer. The logits of large teacher models are sparsified and stored in disk in advance to save the memory cost and computation overheads. The tiny student transformers are automatically scaled down from a large pretrained model with computation and parameter constraints. Comprehensive experiments demonstrate the efficacy of TinyViT. It achieves a top-1 accuracy of 84.8% on ImageNet-1k with only 21M parameters, being comparable to SwinB pretrained on ImageNet-21k while using 4.2 times fewer parameters. Moreover, increasing image resolutions, TinyViT can reach 86.5% accuracy, being slightly better than Swin-L while using only 11% parameters. Last but not the least, we demonstrate a good transfer ability of TinyViT on various downstream tasks.

<img src="https://github.com/microsoft/Cream/raw/main/TinyViT/.figure/framework.png" width="100%">
</div>

```python

predict = inference_model('tinyvit-5m_3rdparty_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('tinyvit-5m_3rdparty_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/tinyvit/tinyvit-5m_8xb256_in1k.py https://download.openmmlab.com/mmclassification/v0/tinyvit/tinyvit-5m_3rdparty_in1k_20221021-62cb5abf.pth
```

| Model                                          |       Pretrain       | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                     Config                      |                      Download                      |
| :--------------------------------------------- | :------------------: | :--------: | :-------: | :-------: | :-------: | :---------------------------------------------: | :------------------------------------------------: |

| `tinyvit-5m_3rdparty_in1k`\*                   |     From scratch     |    5.39    |   1.29    |   79.02   |   94.74   |       [config](tinyvit-5m_8xb256_in1k.py)       | [model](https://download.openmmlab.com/mmclassification/v0/tinyvit/tinyvit-5m_3rdparty_in1k_20221021-62cb5abf.pth) |
| `tinyvit-5m_in21k-distill-pre_3rdparty_in1k`\* | ImageNet-21k DISTILL |    5.39    |   1.29    |   80.71   |   95.57   |   [config](tinyvit-5m-distill_8xb256_in1k.py)   | [model](https://download.openmmlab.com/mmclassification/v0/tinyvit/tinyvit-5m_in21k-distill-pre_3rdparty_in1k_20221021-d4b010a8.pth) |
| `tinyvit-11m_3rdparty_in1k`\*                  |     From scratch     |   11.00    |   2.05    |   81.44   |   95.79   |      [config](tinyvit-11m_8xb256_in1k.py)       | [model](https://download.openmmlab.com/mmclassification/v0/tinyvit/tinyvit-11m_3rdparty_in1k_20221021-11ccef16.pth) |

| `tinyvit-11m_in21k-distill-pre_3rdparty_in1k`\* | ImageNet-21k DISTILL |   11.00    |   2.05    |   83.19   |   96.53   |  [config](tinyvit-11m-distill_8xb256_in1k.py)   | [model](https://download.openmmlab.com/mmclassification/v0/tinyvit/tinyvit-11m_in21k-distill-pre_3rdparty_in1k_20221021-5d3bc0dc.pth) |
| `tinyvit-21m_3rdparty_in1k`\*                  |     From scratch     |   21.20    |   4.30    |   83.08   |   96.58   |      [config](tinyvit-21m_8xb256_in1k.py)       | [model](https://download.openmmlab.com/mmclassification/v0/tinyvit/tinyvit-21m_3rdparty_in1k_20221021-5346ba34.pth) |
| `tinyvit-21m_in21k-distill-pre_3rdparty_in1k`\* | ImageNet-21k DISTILL |   21.20    |   4.30    |   84.85   |   97.27   |  [config](tinyvit-21m-distill_8xb256_in1k.py)   | [model](https://download.openmmlab.com/mmclassification/v0/tinyvit/tinyvit-21m_in21k-distill-pre_3rdparty_in1k_20221021-3d9b30a2.pth) |

| `tinyvit-21m_in21k-distill-pre_3rdparty_in1k-384px`\* | ImageNet-21k DISTILL |   21.23    |   13.85   |   86.21   |   97.77   | [config](tinyvit-21m-distill_8xb256_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/tinyvit/tinyvit-21m_in21k-distill-pre_3rdparty_in1k-384px_20221021-65be6b3f.pth) |
| `tinyvit-21m_in21k-distill-pre_3rdparty_in1k-512px`\* | ImageNet-21k DISTILL |   21.27    |   27.15   |   86.44   |   97.89   | [config](tinyvit-21m-distill_8xb256_in1k-512px.py) | [model](https://download.openmmlab.com/mmclassification/v0/tinyvit/tinyvit-21m_in21k-distill-pre_3rdparty_in1k-512px_20221021-e42a9bea.pth) |

*Models with * are converted from the [official repo](https://github.com/microsoft/Cream/tree/main/TinyViT). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@InProceedings{tiny_vit,
  title={TinyViT: Fast Pretraining Distillation for Small Vision Transformers},

```bibtex
  author={Wu, Kan and Zhang, Jinnian and Peng, Houwen and Liu, Mengchen and Xiao, Bin and Fu, Jianlong and Yuan, Lu},
  booktitle={European conference on computer vision (ECCV)},
}
```

# iSAID数据集

> **iSAID**: A Large-scale Dataset for Instance Segmentation in Aerial Images

## 数据集介绍

Existing Earth Vision datasets are either suitable for semantic segmentation or object detection. iSAID is the first benchmark dataset for instance segmentation in aerial images. This large-scale and densely annotated dataset contains 655,451 object instances for 15 categories across 2,806 high-resolution images. The distinctive characteristics of iSAID are the following: (a) large number of images with high spatial resolution, (b) fifteen important and commonly occurring categories, (c) large number of instances per category, (d) large count of labelled instances per image, which might help in learning contextual information, (e) huge object scale variation, containing small, medium and large objects, often within the same image, (f) Imbalanced and uneven distribution of objects with varying orientation within images, depicting real-life aerial conditions, (g) several small size objects, with ambiguous appearance, can only be resolved with contextual reasoning, (h) precise instance-level annotations carried out by professional annotators, cross-checked and validated by expert annotators complying with well-defined guidelines.

For more detail, please refer to our [paper](http://openaccess.thecvf.com/content_CVPRW_2019/papers/DOAI/Zamir_iSAID_A_Large-scale_Dataset_for_Instance_Segmentation_in_Aerial_Images_CVPRW_2019_paper.pdf) .

## 数据集准备

iSAID数据集下载链接：[图像数据](https://captain-whu.github.io/DOTA/dataset.html)、[标注数据](https://captain-whu.github.io/iSAID/dataset.html)
请按照[官方仓库](https://github.com/CAPTAIN-WHU/iSAID_Devkit)中所述步骤进行数据预处理（`patch_width`=800,`patch_height`=800,`overlap_area`=200），最终得到的文件夹格式为

```
iSAID_patches
├── test
│   └── images
│       ├── P0006_0_0_800_800.png
│       └── ...
│       └── P0009_0_0_800_800.png
├── train
│   └── instance_only_filtered_train.json
│       ├── P0002_0_0_800_800_instance_color_RGB.png
│       ├── P0002_0_0_800_800_instance_id_RGB.png
│       ├── P0002_0_800_800.png
│       ├── ...
│       ├── P0010_0_0_800_800_instance_color_RGB.png
│       ├── P0010_0_0_800_800_instance_id_RGB.png
│       └── P0010_0_800_800.png
└── val

│       ├── ...
└── val
    └── instance_only_filtered_val.json
    └── images
        ├── P0003_0_0_800_800_instance_color_RGB.png
        ├── P0003_0_0_800_800_instance_id_RGB.png
        ├── P0003_0_0_800_800.png
        ├── ...
        ├── P0004_0_0_800_800_instance_color_RGB.png
        ├── P0004_0_0_800_800_instance_id_RGB.png
        └── P0004_0_0_800_800.png
```

之后，在mmdetection目录下使用以下命令转换json文件格式

```
python projects/iSAID/isaid_json.py /path/to/iSAID
```

## 使用方法

### 训练

```python
python tools/train.py projects/iSAID/configs/mask_rcnn_r50_fpn_1x_isaid.py
```

### 测试

```python
python tools/test.py projects/iSAID/configs/mask_rcnn_r50_fpn_1x_isaid.py ${CHECKPOINT_PATH}
```

```
@inproceedings{waqas2019isaid,
title={iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Images},

```

```
author={Waqas Zamir, Syed and Arora, Aditya and Gupta, Akshita and Khan, Salman and Sun, Guolei and Shahbaz Khan, Fahad and Zhu, Fan and Shao, Ling and Xia, Gui-Song and Bai, Xiang},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops},
pages={28--37},
year={2019}
}
```

```
@InProceedings{Xia_2018_CVPR,
author = {Xia, Gui-Song and Bai, Xiang and Ding, Jian and Zhu, Zhen and Belongie, Serge and Luo, Jiebo and Datcu, Mihai and Pelillo, Marcello and Zhang, Liangpei},
title = {DOTA: A Large-Scale Dataset for Object Detection in Aerial Images},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2018}
}
```

# Pix2Pix (CVPR'2017)

> [Pix2Pix: Image-to-Image Translation with Conditional Adversarial Networks](https://openaccess.thecvf.com/content_cvpr_2017/html/Isola_Image-To-Image_Translation_With_CVPR_2017_paper.html)

We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Moreover, since the release of the pix2pix software associated with this paper, hundreds of twitter users have posted their own artistic experiments using our system. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without handengineering our loss functions either.

<img src="https://user-images.githubusercontent.com/28132635/143053385-1b03356d-43df-423b-88b2-7a82b73d2edd.JPG"/>
</div>

  <b> Results from Pix2Pix trained by mmagic</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/22982797/114269080-4ff0ec00-9a37-11eb-92c4-1525864e0307.PNG" width="800"/>
</div>
We use `FID` and `IS` metrics to evaluate the generation performance of pix2pix.<sup>1</sup>

|                                     Model                                      |   Dataset   |   FID    |  IS   |                                              Download                                              |
| :----------------------------------------------------------------------------: | :---------: | :------: | :---: | :------------------------------------------------------------------------------------------------: |

|           [Ours](./pix2pix_vanilla-unet-bn_1xb1-80kiters_facades.py)           |   facades   | 124.9773 | 1.620 | [model](https://download.openmmlab.com/mmediting/pix2pix/refactor/pix2pix_vanilla_unet_bn_1x1_80k_facades_20210902_170442-c0958d50.pth)   \| [log](https://download.openmmlab.com/mmediting/pix2pix/pix2pix_vanilla_unet_bn_1x1_80k_facades_20210317_172625.log.json)<sup>2</sup> |
|        [Ours](./pix2pix_vanilla-unet-bn_1xb1-220kiters_aerial2maps.py)         | aerial2maps | 122.5856 | 3.137 | [model](https://download.openmmlab.com/mmediting/pix2pix/refactor/pix2pix_vanilla_unet_bn_a2b_1x1_219200_maps_convert-bgr_20210902_170729-59a31517.pth) |
|        [Ours](./pix2pix_vanilla-unet-bn_1xb1-220kiters_maps2aerial.py)         | maps2aerial | 88.4635  | 3.310 | [model](https://download.openmmlab.com/mmediting/pix2pix/refactor/pix2pix_vanilla_unet_bn_b2a_1x1_219200_maps_convert-bgr_20210902_170814-6d2eac4a.pth) |

| [Ours](./pix2pix_vanilla-unet-bn_wo-jitter-flip-1xb4-190kiters_edges2shoes.py) | edges2shoes | 84.3750  | 2.815 | [model](https://download.openmmlab.com/mmediting/pix2pix/refactor/pix2pix_vanilla_unet_bn_wo_jitter_flip_1x4_186840_edges2shoes_convert-bgr_20210902_170902-0c828552.pth) |

| Dataset  |   facades   | aerial2maps  | maps2aerial | edges2shoes |   average    |
| :------: | :---------: | :----------: | :---------: | :---------: | :----------: |
| official | **119.135** |   149.731    |   102.072   | **75.774**  |   111.678    |
|   ours   |  124.9773   | **122.5856** | **88.4635** |   84.3750   | **105.1003** |

`IS` comparison with official:

| Dataset  |  facades  | aerial2maps | maps2aerial | edges2shoes |  average   |
| :------: | :-------: | :---------: | :---------: | :---------: | :--------: |
| official | **1.650** |    2.529    |  **3.552**  |    2.766    |   2.624    |

|   ours   |   1.620   |  **3.137**  |    3.310    |  **2.815**  | **2.7205** |

Note:

1. we strictly follow the [paper](http://openaccess.thecvf.com/content_cvpr_2017/papers/Isola_Image-To-Image_Translation_With_CVPR_2017_paper.pdf) setting in Section 3.3: "*At inference time, we run the generator net in exactly
   the same manner as during the training phase. This differs
   from the usual protocol in that we apply dropout at test time,
   and we apply batch normalization using the statistics of
   the test batch, rather than aggregated statistics of the training batch.*" (i.e., use model.train() mode), thus may lead to slightly different inference results every time.

```latex

```latex
  url={https://openaccess.thecvf.com/content_cvpr_2017/html/Isola_Image-To-Image_Translation_With_CVPR_2017_paper.html},
}
```

# ConvMixer

> [Patches Are All You Need?](https://arxiv.org/abs/2201.09792)

Although convolutional networks have been the dominant architecture for vision tasks for many years, recent experiments have shown that Transformer-based models, most notably the Vision Transformer (ViT), may exceed their performance in some settings. However, due to the quadratic runtime of the self-attention layers in Transformers, ViTs require the use of patch embeddings, which group together small regions of the image into single input features, in order to be applied to larger image sizes. This raises a question: Is the performance of ViTs due to the inherently-more-powerful Transformer architecture, or is it at least partly due to using patches as the input representation? In this paper, we present some evidence for the latter: specifically, we propose the ConvMixer, an extremely simple model that is similar in spirit to the ViT and the even-more-basic MLP-Mixer in that it operates directly on patches as input, separates the mixing of spatial and channel dimensions, and maintains equal size and resolution throughout the network. In contrast, however, the ConvMixer uses only standard convolutions to achieve the mixing steps. Despite its simplicity, we show that the ConvMixer outperforms the ViT, MLP-Mixer, and some of their variants for similar parameter counts and data set sizes, in addition to outperforming classical vision models such as the ResNet.

<img src="https://user-images.githubusercontent.com/42952108/156284977-abf2245e-d9ba-4e0d-8e10-c0664a20f4c8.png" width="100%"/>
</div>

```python

predict = inference_model('convmixer-768-32_3rdparty_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('convmixer-768-32_3rdparty_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/convmixer/convmixer-768-32_10xb64_in1k.py https://download.openmmlab.com/mmclassification/v0/convmixer/convmixer-768-32_3rdparty_10xb64_in1k_20220323-bca1f7b8.pth
```

| Model                               |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                   Config                   |                                  Download                                  |
| :---------------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :----------------------------------------: | :------------------------------------------------------------------------: |

| `convmixer-768-32_3rdparty_in1k`\*  | From scratch |   21.11    |   19.62   |   80.16   |   95.08   | [config](convmixer-768-32_10xb64_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/convmixer/convmixer-768-32_3rdparty_10xb64_in1k_20220323-bca1f7b8.pth) |
| `convmixer-1024-20_3rdparty_in1k`\* | From scratch |   24.38    |   5.55    |   76.94   |   93.36   | [config](convmixer-1024-20_10xb64_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/convmixer/convmixer-1024-20_3rdparty_10xb64_in1k_20220323-48f8aeba.pth) |
| `convmixer-1536-20_3rdparty_in1k`\* | From scratch |   51.63    |   48.71   |   81.37   |   95.61   | [config](convmixer-1536-20_10xb64_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/convmixer/convmixer-1536_20_3rdparty_10xb64_in1k_20220323-ea5786f3.pth) |

*Models with * are converted from the [official repo](https://github.com/locuslab/convmixer). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@misc{trockman2022patches,
      title={Patches Are All You Need?},
      author={Asher Trockman and J. Zico Kolter},
      eprint={2201.09792},
}
```

<details>
<summary align="right"><a href="https://openaccess.thecvf.com/content/ICCV2021/papers/Gu_Removing_the_Bias_of_Integral_Pose_Regression_ICCV_2021_paper.pdf">Debias IPR (ICCV'2021)</a></summary>

```bibtex
@inproceedings{gu2021removing,
    title={Removing the Bias of Integral Pose Regression},
    author={Gu, Kerui and Yang, Linlin and Yao, Angela},
    booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
    pages={11067--11076},
  }
```

<details>

```bibtex

author={He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian},
}
```

<details>

```bibtex
}
```

pages={740--755},
}
```

| [debias-ipr_resnet_50](/configs/body_2d_keypoint/integral_regression/coco/ipr_res50_debias-8xb64-210e_coco-256x256.py) |  256x256   | 0.675 |      0.872      |      0.740      | 0.765 |      0.928      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/integral_regression/coco/ipr_res50_debias-8xb64-210e_coco-256x256-055a7699_20220913.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/integral_regression/coco/ipr_res50_debias-8xb64-210e_coco-256x256-055a7699_20220913.log.json) |

# GLEAN (CVPR'2021)

> [GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution](https://arxiv.org/abs/2012.00739)

We show that pre-trained Generative Adversarial Networks (GANs), e.g., StyleGAN, can be used as a latent bank to improve the restoration quality of large-factor image super-resolution (SR). While most existing SR approaches attempt to generate realistic textures through learning with adversarial loss, our method, Generative LatEnt bANk (GLEAN), goes beyond existing practices by directly leveraging rich and diverse priors encapsulated in a pre-trained GAN. But unlike prevalent GAN inversion methods that require expensive image-specific optimization at runtime, our approach only needs a single forward pass to generate the upscaled image. GLEAN can be easily incorporated in a simple encoder-bank-decoder architecture with multi-resolution skip connections. Switching the bank allows the method to deal with images from diverse categories, e.g., cat, building, human face, and car. Images upscaled by GLEAN show clear improvements in terms of fidelity and texture faithfulness in comparison to existing methods.

 <img src="https://user-images.githubusercontent.com/7676947/144019196-2642f3be-f82e-4fa4-8d96-4161354db9a7.png" width="400"/>
</div >

For the meta info used in training and test, please refer to [here](https://github.com/ckkelvinchan/GLEAN). The results are evaluated on RGB channels.

|                                      Model                                       | Dataset  | PSNR  |    Training Resources    |                                      Download                                       |
| :------------------------------------------------------------------------------: | :------: | :---: | :----------------------: | :---------------------------------------------------------------------------------: |

|                      [glean_cat_8x](./glean_x8_2xb8_cat.py)                      | LSUN-CAT | 23.98 | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/glean/glean_cat_8x_20210614-d3ac8683.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/glean/glean_cat_8x_20210614_145540.log.json) |
|                    [glean_ffhq_16x](./glean_x16_2xb8_ffhq.py)                    |   FFHQ   | 26.91 | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/glean/glean_ffhq_16x_20210527-61a3afad.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/glean/glean_ffhq_16x_20210527_194536.log.json) |

|                     [glean_cat_16x](./glean_x16_2xb8_cat.py)                     | LSUN-CAT | 20.88 | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/glean/glean_cat_16x_20210527-68912543.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/glean/glean_cat_16x_20210527_103708.log.json) |
| [glean_in128out1024_4x2_300k_ffhq_celebahq](./glean_in128out1024_4xb2-300k_ffhq-celeba-hq.py) |   FFHQ   | 27.94 | 4 (Tesla V100-SXM3-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/glean/glean_in128out1024_4x2_300k_ffhq_celebahq_20210812-acbcb04f.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/glean/glean_in128out1024_4x2_300k_ffhq_celebahq_20210812_100549.log.json) |
|                 [glean_fp16_cat_8x](./glean_x8-fp16_2xb8_cat.py)                 | LSUN-CAT |   -   |            -             |                                          -                                          |

|               [glean_fp16_ffhq_16x](./glean_x16-fp16_2xb8_ffhq.py)               |   FFHQ   |   -   |            -             |                                          -                                          |
| [glean_fp16_in128out1024_4x2_300k_ffhq_celebahq](./glean_in128out1024-fp16_4xb2-300k_ffhq-celeba-hq.py) |   FFHQ   |   -   |            -             |                                          -                                          |

**Train**

<details>

```shell
# cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/glean/glean_x8_2xb8_cat.py

python tools/train.py configs/glean/glean_x8_2xb8_cat.py

./tools/dist_train.sh configs/glean/glean_x8_2xb8_cat.py 8
```

```

**Test**

<details>

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/glean/glean_x8_2xb8_cat.py https://download.openmmlab.com/mmediting/restorers/glean/glean_cat_8x_20210614-d3ac8683.pth

python tools/test.py configs/glean/glean_x8_2xb8_cat.py https://download.openmmlab.com/mmediting/restorers/glean/glean_cat_8x_20210614-d3ac8683.pth

./tools/dist_test.sh configs/glean/glean_x8_2xb8_cat.py https://download.openmmlab.com/mmediting/restorers/glean/glean_cat_8x_20210614-d3ac8683.pth 8
```

```

```bibtex
@InProceedings{chan2021glean,
  author = {Chan, Kelvin CK and Wang, Xintao and Xu, Xiangyu and Gu, Jinwei and Loy, Chen Change},
  title = {GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution},
}
```

![NCNN](https://raw.githubusercontent.com/Tencent/ncnn/master/images/256-ncnn.png)

# ncnn

[![License][badge-license]](LICENSE.txt)
[![Download Total Count][badge-download-count]](https://github.com/Tencent/ncnn/releases)
[![codecov][badge-codecov]](https://codecov.io/gh/Tencent/ncnn)
[![Language grade: C/C++][badge-code-quality]](https://lgtm.com/projects/g/Tencent/ncnn/context:cpp)

[badge-license]: https://img.shields.io/badge/license-BSD--3--Clause-blue.svg
[badge-download-count]: https://img.shields.io/github/downloads/Tencent/ncnn/total.svg
[badge-codecov]: https://codecov.io/gh/Tencent/ncnn/branch/master/graph/badge.svg
[badge-code-quality]: https://img.shields.io/lgtm/grade/cpp/g/Tencent/ncnn.svg?logo=lgtm&logoWidth=18

ncnn is a high-performance neural network inference computing framework optimized for mobile platforms.
ncnn is deeply considerate about deployment and uses on mobile phones from the beginning of design.

ncnn does not have third party dependencies. It is cross-platform, and runs faster than all known open source frameworks on mobile phone cpu.
Developers can easily deploy deep learning algorithm models to the mobile platform by using efficient ncnn implementation,
create intelligent APPs, and bring the artificial intelligence to your fingertips.
ncnn is currently being used in many Tencent applications, such as QQ, Qzone, WeChat, Pitu and so on.

ncnn 是一个为手机端极致优化的高性能神经网络前向计算框架。
ncnn 从设计之初深刻考虑手机端的部署和使用。
无第三方依赖，跨平台，手机端 cpu 的速度快于目前所有已知的开源框架。
基于 ncnn，开发者能够将深度学习算法轻松移植到手机端高效执行，
开发出人工智能 APP，将 AI 带到你的指尖。
ncnn 目前已在腾讯多款应用中使用，如：QQ，Qzone，微信，天天 P 图等。

---

## 技术交流 QQ 群：637093648 (超多大佬) 答案：卷卷卷卷卷 （已满）

## Pocky QQ 群（MLIR YES!）: 677104663(超多大佬) 答案：multi-level intermediate representation

## Telegram Group <https://t.me/ncnnyes>

## Discord Channel <https://discord.gg/YRsxgmF>

---

---

## Current building status matrix

| System            | CPU (32bit)                                                         | CPU (64bit)                                                                     | GPU (32bit)                                                     | GPU (64bit)                                                         |
| :---------------- | :------------------------------------------------------------------ | :------------------------------------------------------------------------------ | :-------------------------------------------------------------- | :------------------------------------------------------------------ |

| Linux (GCC)       | [![Build Status][pass-linux-x86-cpu-gcc]][ci-linux-x86-cpu-gcc]     | [![Build Status][pass-linux-x64-cpu-gcc]][ci-linux-x64-cpu-gcc]                 | —                                                               | [![Build Status][pass-linux-x64-gpu-gcc]][ci-linux-x64-gpu-gcc]     |
| Linux (Clang)     | [![Build Status][pass-linux-x86-cpu-clang]][ci-linux-x86-cpu-clang] | [![Build Status][pass-linux-x64-cpu-clang]][ci-linux-x64-cpu-clang]             | —                                                               | [![Build Status][pass-linux-x64-gpu-clang]][ci-linux-x64-gpu-clang] |
| Linux (ARM)       | [![Build Status][pass-linux-arm-cpu-gcc]][ci-linux-arm-cpu-gcc]     | [![Build Status][pass-linux-aarch64-cpu-gcc]][ci-linux-aarch64-cpu-gcc]         | —                                                               | —                                                                   |

| Linux (MIPS)      | [![Build Status][pass-linux-mips-cpu-gcc]][ci-linux-mips-cpu-gcc]   | [![Build Status][pass-linux-mips64-cpu-gcc]][ci-linux-mips64-cpu-gcc]           | —                                                               | —                                                                   |
| Linux (RISC-V)    | —                                                                   | [![Build Status][pass-linux-riscv64-cpu-gcc]][ci-linux-riscv64-cpu-gcc]         | —                                                               | —                                                                   |
| Linux (LoongArch) | —                                                                   | [![Build Status][pass-linux-loongarch64-cpu-gcc]][ci-linux-loongarch64-cpu-gcc] | —                                                               | —                                                                   |

| Windows           | [![Build Status][pass-windows-x86-cpu]][ci-windows-x86-cpu]         | [![Build Status][pass-windows-x64-cpu]][ci-windows-x64-cpu]                     | —                                                               | [![Build Status][pass-windows-x64-gpu]][ci-windows-x64-gpu]         |
| Windows (ARM)     | [![Build Status][pass-windows-arm-cpu]][ci-windows-arm-cpu]         | [![Build Status][pass-windows-arm64-cpu]][ci-windows-arm64-cpu]                 | —                                                               | —                                                                   |
| macOS             | —                                                                   | [![Build Status][pass-macos-x64-cpu]][ci-macos-x64-cpu]                         | —                                                               | [![Build Status][pass-macos-x64-gpu]][ci-macos-x64-gpu]             |

| macOS (ARM)       | —                                                                   | [![Build Status][pass-macos-arm64-cpu]][ci-macos-arm64-cpu]                     | —                                                               | [![Build Status][pass-macos-arm64-gpu]][ci-macos-arm64-gpu]         |
| Android           | [![Build Status][pass-android-armv7-cpu]][ci-android-armv7-cpu]     | [![Build Status][pass-android-armv8-cpu]][ci-android-armv8-cpu]                 | [![Build Status][pass-android-armv7-gpu]][ci-android-armv7-gpu] | [![Build Status][pass-android-armv8-gpu]][ci-android-armv8-gpu]     |
| Android-x86       | [![Build Status][pass-android-x86-cpu]][ci-android-x86-cpu]         | [![Build Status][pass-android-x64-cpu]][ci-android-x64-cpu]                     | [![Build Status][pass-android-x86-gpu]][ci-android-x86-gpu]     | [![Build Status][pass-android-x64-gpu]][ci-android-x64-gpu]         |

| iOS               | [![Build Status][pass-ios-cpu]][ci-ios-cpu]                         | [![Build Status][pass-ios-cpu]][ci-ios-cpu]                                     | —                                                               | [![Build Status][pass-ios-arm64-gpu]][ci-ios-arm64-gpu]             |
| iOS Simulator     | [![Build Status][pass-ios-simulator]][ci-ios-simulator]             | [![Build Status][pass-ios-simulator]][ci-ios-simulator]                         | —                                                               | [![Build Status][pass-ios-simulator-gpu]][ci-ios-simulator-gpu]     |
| WebAssembly       | [![Build Status][pass-web-assembly]][ci-web-assembly]               | —                                                                               | —                                                               | —                                                                   |

| RISC-V GCC/Newlib | [![Build Status][pass-elf-riscv32-cpu-gcc]][ci-elf-riscv32-cpu-gcc] | [![Build Status][pass-elf-riscv64-cpu-gcc]][ci-elf-riscv64-cpu-gcc]             | —                                                               | —                                                                   |

[pass-android-armv7-cpu]: https://img.shields.io/github/actions/workflow/status/Tencent/ncnn/android-armv7-cpu.yml?branch=master
[pass-android-armv7-gpu]: https://img.shields.io/github/actions/workflow/status/Tencent/ncnn/android-armv7-gpu.yml?branch=master
[pass-android-armv8-cpu]: https://img.shields.io/github/actions/workflow/status/Tencent/ncnn/android-armv8-cpu.yml?branch=master
[pass-android-armv8-gpu]: https://img.shields.io/github/actions/workflow/status/Tencent/ncnn/android-armv8-gpu.yml?branch=master
[pass-android-x64-cpu]: https://img.shields.io/github/actions/workflow/status/Tencent/ncnn/android-x64-cpu.yml?branch=master

[ci-macos-x64-cpu]: https://github.com/Tencent/ncnn/actions?query=workflow%3Amacos-x64-cpu
[ci-macos-x64-gpu]: https://github.com/Tencent/ncnn/actions?query=workflow%3Amacos-x64-gpu
[ci-web-assembly]: https://github.com/Tencent/ncnn/actions?query=workflow%3Aweb-assembly
[ci-windows-arm-cpu]: https://github.com/Tencent/ncnn/actions?query=workflow%3Awindows-arm-cpu
[ci-windows-arm64-cpu]: https://github.com/Tencent/ncnn/actions?query=workflow%3Awindows-arm64-cpu
[ci-windows-x64-cpu]: https://github.com/Tencent/ncnn/actions?query=workflow%3Awindows-x64-cpu
[ci-windows-x64-gpu]: https://github.com/Tencent/ncnn/actions?query=workflow%3Awindows-x64-gpu
[ci-windows-x86-cpu]: https://github.com/Tencent/ncnn/actions?query=workflow%3Awindows-x86-cpu

---

## Support most commonly used CNN network

## 支持大部分常用的 CNN 网络

- Classical CNN:
  [VGG](https://github.com/BVLC/caffe/wiki/Model-Zoo#models-used-by-the-vgg-team-in-ilsvrc-2014)

---

  [AlexNet](https://github.com/BVLC/caffe/tree/9b891540183ddc834a02b2bd81b31afae71b2153/models/bvlc_alexnet)
  [GoogleNet](https://github.com/BVLC/caffe/tree/9b891540183ddc834a02b2bd81b31afae71b2153/models/bvlc_googlenet)
  Inception
  ...
- Practical CNN:
  [ResNet](https://github.com/tornadomeet/ResNet)
  [DenseNet](https://github.com/liuzhuang13/DenseNet)
  [SENet](https://github.com/hujie-frank/SENet)
  [FPN](https://github.com/unsky/FPN)
  ...
- Light-weight CNN:
  [SqueezeNet](https://github.com/forresti/SqueezeNet)
  [MobileNetV1](https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md)
  [MobileNetV2/V3](https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet/README.md)
  [ShuffleNetV1](https://github.com/farmingyard/ShuffleNet)

[MobileNetV2/V3](https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet/README.md)
  [ShuffleNetV2](https://github.com/opconty/keras-shufflenetV2)
  [MNasNet](https://github.com/tensorflow/models/tree/master/research/slim/nets/nasnet)
  ...
- Face Detection:
  [MTCNN](https://github.com/ipazc/mtcnn)
  [RetinaFace](https://github.com/biubug6/Pytorch_Retinaface)
  [scrfd](https://github.com/nihui/ncnn-android-scrfd)
  ...
- Detection:
  [VGG-SSD](https://github.com/lzx1413/CAFFE_SSD)
  [MobileNet-SSD](https://github.com/chuanqi305/MobileNet-SSD)
  [SqueezeNet-SSD](https://github.com/chuanqi305/SqueezeNet-SSD)
  [MobileNetV2-SSDLite](https://github.com/chuanqi305/MobileNetv2-SSDLite)
  [MobileNetV3-SSDLite](https://github.com/XiaoyuHuang96/MobilenetV3SSDLite-tfkeras)
  ...
  [Faster-RCNN](https://github.com/rbgirshick/py-faster-rcnn)
  [R-FCN](https://github.com/daijifeng001/R-FCN)
  ...

...
  ...
  [YOLOv2](https://github.com/longcw/yolo2-pytorch)
  [YOLOv3](https://github.com/ultralytics/yolov3)
  [MobileNet-YOLOv3](https://github.com/eric612/MobileNet-YOLO)
  [YOLOv4](https://github.com/Tianxiaomo/pytorch-YOLOv4)
  [YOLOv5](https://github.com/ultralytics/yolov5)
  [YOLOv7](https://github.com/WongKinYiu/yolov7)
  [YOLOX](https://github.com/Megvii-BaseDetection/YOLOX)
  ...
  [NanoDet](https://github.com/RangiLyu/nanodet)
- Segmentation:
  [FCN](https://github.com/unsky/FPN)
  [PSPNet](https://github.com/hszhao/PSPNet)
  [UNet](https://github.com/zhixuhao/unet)
  [YOLACT](https://github.com/dbolya/yolact)
  ...
- Pose Estimation:
  [SimplePose](https://github.com/dog-qiuqiu/Ultralight-SimplePose)
  ...

---

## HowTo

[YOLACT](https://github.com/dbolya/yolact)
  ...
  ...

---

## HowTo

**[how to build ncnn library](https://github.com/Tencent/ncnn/wiki/how-to-build) on Linux / Windows / macOS / Raspberry Pi3, Pi4 / POWER / Android / NVIDIA Jetson / iOS / WebAssembly / AllWinner D1 / Loongson 2K1000**

- [Build for Linux / NVIDIA Jetson / Raspberry Pi3, Pi4 / POWER](https://github.com/Tencent/ncnn/wiki/how-to-build#build-for-linux)
- [Build for Windows x64 using VS2017](https://github.com/Tencent/ncnn/wiki/how-to-build#build-for-windows-x64-using-visual-studio-community-2017)
- [Build for macOS](https://github.com/Tencent/ncnn/wiki/how-to-build#build-for-macos)
- [Build for ARM Cortex-A family with cross-compiling](https://github.com/Tencent/ncnn/wiki/how-to-build#build-for-arm-cortex-a-family-with-cross-compiling)

- [Build for Hisilicon platform with cross-compiling](https://github.com/Tencent/ncnn/wiki/how-to-build#build-for-hisilicon-platform-with-cross-compiling)
- [Build for Android](https://github.com/Tencent/ncnn/wiki/how-to-build#build-for-android)
- [Build for iOS on macOS with xcode](https://github.com/Tencent/ncnn/wiki/how-to-build#build-for-ios-on-macos-with-xcode)
- [Build for WebAssembly](https://github.com/Tencent/ncnn/wiki/how-to-build#build-for-webassembly)
- [Build for AllWinner D1](https://github.com/Tencent/ncnn/wiki/how-to-build#build-for-allwinner-d1)
- [Build for Loongson 2K1000](https://github.com/Tencent/ncnn/wiki/how-to-build#build-for-loongson-2k1000)
- [Build for Termux on Android](https://github.com/Tencent/ncnn/wiki/how-to-build#build-for-termux-on-android)

- [Build for QNX](https://github.com/Tencent/ncnn/wiki/how-to-build#build-for-qnx)

**[download prebuild binary package for android and ios](https://github.com/Tencent/ncnn/releases)**

**[use ncnn with alexnet](https://github.com/Tencent/ncnn/wiki/use-ncnn-with-alexnet) with detailed steps, recommended for beginners :)**

**[ncnn 组件使用指北 alexnet](https://github.com/Tencent/ncnn/wiki/use-ncnn-with-alexnet.zh) 附带详细步骤，新人强烈推荐 :)**

**[use netron for ncnn model visualization](https://netron.app)**

**[out-of-the-box web model conversion](https://convertmodel.com/#outputFormat=ncnn)**

[ncnn low-level operation api](https://github.com/Tencent/ncnn/wiki/low-level-operation-api)

[ncnn param and model file spec](https://github.com/Tencent/ncnn/wiki/param-and-model-file-structure)

[ncnn operation param weight table](https://github.com/Tencent/ncnn/wiki/operation-param-weight-table)

[how to implement custom layer step by step](https://github.com/Tencent/ncnn/wiki/how-to-implement-custom-layer-step-by-step)

---

## FAQ

**[ncnn throw error](https://github.com/Tencent/ncnn/wiki/FAQ-ncnn-throw-error)**

**[ncnn produce wrong result](https://github.com/Tencent/ncnn/wiki/FAQ-ncnn-produce-wrong-result)**

**[ncnn vulkan](https://github.com/Tencent/ncnn/wiki/FAQ-ncnn-vulkan)**

---

## Features

- Supports convolutional neural networks, supports multiple input and multi-branch structure, can calculate part of the branch
- No third-party library dependencies, does not rely on BLAS / NNPACK or any other computing framework
- Pure C++ implementation, cross-platform, supports Android, iOS and so on
- ARM NEON assembly level of careful optimization, calculation speed is extremely high
- Sophisticated memory management and data structure design, very low memory footprint

- Supports multi-core parallel computing acceleration, ARM big.LITTLE CPU scheduling optimization
- Supports GPU acceleration via the next-generation low-overhead Vulkan API
- Extensible model design, supports 8bit quantization and half-precision floating point storage, can import caffe/pytorch/mxnet/onnx/darknet/keras/tensorflow(mlir) models
- Support direct memory zero copy reference load network model
- Can be registered with custom layer implementation and extended
- Well, it is strong, not afraid of being stuffed with 卷 QvQ

## 功能概述

- 支持卷积神经网络，支持多输入和多分支结构，可计算部分分支
- 无任何第三方库依赖，不依赖 BLAS/NNPACK 等计算框架
- 纯 C++ 实现，跨平台，支持 Android / iOS 等
- ARM Neon 汇编级良心优化，计算速度极快
- 精细的内存管理和数据结构设计，内存占用极低
- 支持多核并行计算加速，ARM big.LITTLE CPU 调度优化
- 支持基于全新低消耗的 Vulkan API GPU 加速

- 可扩展的模型设计，支持 8bit [量化](tools/quantize) 和半精度浮点存储，可导入 caffe/pytorch/mxnet/onnx/darknet/keras/tensorflow(mlir) 模型
- 支持直接内存零拷贝引用加载网络模型
- 可注册自定义层实现并扩展
- 恩，很强就是了，不怕被塞卷 QvQ

---

## supported platform matrix

- ✅ = known work and runs fast with good optimization
- ✔️ = known work, but speed may not be fast enough
- ❔ = shall work, not confirmed
- / = not applied

|            | Windows | Linux | Android | macOS | iOS |
| ---------- | ------- | ----- | ------- | ----- | --- |
| intel-cpu  | ✔️      | ✔️    | ❔      | ✔️    | /   |
| intel-gpu  | ✔️      | ✔️    | ❔      | ❔    | /   |
| amd-cpu    | ✔️      | ✔️    | ❔      | ✔️    | /   |
| amd-gpu    | ✔️      | ✔️    | ❔      | ❔    | /   |
| nvidia-gpu | ✔️      | ✔️    | ❔      | ❔    | /   |
| qcom-cpu   | ❔      | ✔️    | ✅      | /     | /   |

| qcom-gpu   | ❔      | ✔️    | ✔️      | /     | /   |
| arm-cpu    | ❔      | ❔    | ✅      | /     | /   |
| arm-gpu    | ❔      | ❔    | ✔️      | /     | /   |
| apple-cpu  | /       | /     | /       | ✔️    | ✅  |
| apple-gpu  | /       | /     | /       | ✔️    | ✔️  |
| ibm-cpu    | /       | ✔️     | /       | /    | /  |

---

## Project examples

- <https://github.com/nihui/ncnn-android-squeezenet>
- <https://github.com/nihui/ncnn-android-styletransfer>
- <https://github.com/nihui/ncnn-android-mobilenetssd>
- <https://github.com/moli232777144/mtcnn_ncnn>
- <https://github.com/nihui/ncnn-android-yolov5>
- <https://github.com/xiang-wuu/ncnn-android-yolov7>
- <https://github.com/nihui/ncnn-android-scrfd> 🤩
- <https://github.com/shaoshengsong/qt_android_ncnn_lib_encrypt_example>

<img src="https://github.com/nihui/ncnn-assets/raw/master/20181217/ncnn-2.jpg" height ="230"/><img src="https://github.com/nihui/ncnn-assets/raw/master/20181217/4.jpg" height ="230"/><img src="https://github.com/nihui/ncnn-assets/raw/master/20181217/ncnn-33.jpg" height ="230"/><img src="https://github.com/nihui/ncnn-assets/raw/master/20181217/ncnn-m.png" height ="230"/><img src="https://github.com/nihui/ncnn-android-yolov5/raw/master/screenshot.jpg" height ="230"/><img src="https://github.com/nihui/ncnn-android-scrfd/raw/master/screenshot.jpg" height ="230"/><br>

- <https://github.com/magicse/ncnn-colorization-siggraph17><br>
<img src="https://user-images.githubusercontent.com/13585785/189326958-f5a8d6f8-caef-49bf-88da-ae494371195d.jpg" width ="700"/>

- <https://github.com/mizu-bai/ncnn-fortran> Call ncnn from Fortran

- <https://github.com/k2-fsa/sherpa> Use ncnn for real-time speech
  recognition (i.e., speech-to-text); also support embedded devices and provide
  mobile Apps (e.g., Android App)

---

## License

[BSD 3 Clause](LICENSE.txt)

## Frequently Asked Questions

### TensorRT

- "WARNING: Half2 support requested on hardware without native FP16 support, performance will be negatively affected."

  Fp16 mode requires a device with full-rate fp16 support.

- "error: parameter check failed at: engine.cpp::setBindingDimensions::1046, condition: profileMinDims.d[i] \<= dimensions.d[i]"

  When building an `ICudaEngine` from an `INetworkDefinition` that has dynamically resizable inputs, users need to specify at least one optimization profile. Which can be set in deploy config:

  ```python
  backend_config = dict(
    common_config=dict(max_workspace_size=1 << 30),
    model_inputs=[
        dict(
            input_shapes=dict(
                input=dict(
                    min_shape=[1, 3, 320, 320],
                    opt_shape=[1, 3, 800, 1344],
                    max_shape=[1, 3, 1344, 1344])))
    ])
  ```

  The input tensor shape should be limited between `min_shape` and `max_shape`.

opt_shape=[1, 3, 800, 1344],
    ])
  ```

- "error: [TensorRT] INTERNAL ERROR: Assertion failed: cublasStatus == CUBLAS_STATUS_SUCCESS"

  TRT 7.2.1 switches to use cuBLASLt (previously it was cuBLAS). cuBLASLt is the defaulted choice for SM version >= 7.0. You may need CUDA-10.2 Patch 1 (Released Aug 26, 2020) to resolve some cuBLASLt issues. Another option is to use the new TacticSource API and disable cuBLASLt tactics if you dont want to upgrade.

### Libtorch

- Error: `libtorch/share/cmake/Caffe2/Caffe2Config.cmake:96 (message):Your installed Caffe2 version uses cuDNN but I cannot find the cuDNN libraries.  Please set the proper cuDNN prefixes and / or install cuDNN.`

  May `export CUDNN_ROOT=/root/path/to/cudnn` to resolve the build error.

### Windows

May `export CUDNN_ROOT=/root/path/to/cudnn` to resolve the build error.

- Error: similar like this `OSError: [WinError 1455] The paging file is too small for this operation to complete. Error loading "C:\Users\cx\miniconda3\lib\site-packages\torch\lib\cudnn_cnn_infer64_8.dll" or one of its dependencies`

  Solution: according to this [post](https://stackoverflow.com/questions/64837376/how-to-efficiently-run-multiple-pytorch-processes-models-at-once-traceback), the issue may be caused by NVidia and will fix in *CUDA release 11.7*. For now one could use the [fixNvPe.py](https://gist.github.com/cobryan05/7d1fe28dd370e110a372c4d268dcb2e5) script to modify the nvidia dlls in the pytorch lib dir.

  `python fixNvPe.py --input=C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\lib\*.dll`

  You can find your pytorch installation path with:

  ```python
  import torch
  print(torch.__file__)
  ```

- enable_language(CUDA) error

  ```

You can find your pytorch installation path with:

  ```python
  ```

  ```
  -- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.19044.
  -- Found CUDA: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.1 (found version "11.1")
  CMake Error at C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCompilerId.cmake:491 (message):
    No CUDA toolset found.
  Call Stack (most recent call first):
    C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCompilerId.cmake:6 (CMAKE_DETERMINE_COMPILER_ID_BUILD)
    C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCompilerId.cmake:59 (__determine_compiler_id_test)
    C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCUDACompiler.cmake:339 (CMAKE_DETERMINE_COMPILER_ID)

C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCUDACompiler.cmake:339 (CMAKE_DETERMINE_COMPILER_ID)
    C:/workspace/mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0/sdk/lib/cmake/MMDeploy/MMDeployConfig.cmake:27 (enable_language)
    CMakeLists.txt:5 (find_package)
  ```

  **Cause：** CUDA Toolkit 11.1 was installed before Visual Studio, so the VS plugin was not installed. Or the version of VS is too new, so that the installation of the VS plugin is skipped during the installation of the CUDA Toolkit

  **Solution：** This problem can be solved by manually copying the four files in `C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\extras\visual_studio_integration\MSBuildExtensions` to `C:\Software\Microsoft Visual Studio\2022\Community\Msbuild\Microsoft\VC\v170\BuildCustomizations` The specific path should be changed according to the actual situation.

### ONNX Runtime

- Under Windows system, when visualizing model inference result failed with the following error:
  ```
  onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Failed to load library, error code: 193
  ```
  **Cause：** In latest Windows systems, there are two `onnxruntime.dll` under the system path, and they will be loaded first, causing conflicts.
  ```
  C:\Windows\SysWOW64\onnxruntime.dll
  C:\Windows\System32\onnxruntime.dll
  ```
  **Solution：** Choose one of the following two options
  1. Copy the dll in the lib directory of the downloaded onnxruntime to the directory where mmdeploy_onnxruntime_ops.dll locates (It is recommended to use Everything to search the ops dll)
  2. Rename the two dlls in the system path so that they cannot be loaded.

### Pip

- pip installed package but could not `import` them.

  Make sure your are using conda pip.

  ```bash
  $ which pip
  # /path/to/.local/bin/pip

### Pip

  ```bash
  $ which pip
  /path/to/miniconda3/lib/python3.9/site-packages/pip
  ```

## 2D Human Pose Demo

We provide demo scripts to perform human pose estimation on images or videos.

### 2D Human Pose Top-Down Image Demo

#### Use full image as input

We provide a demo script to test a single image, using the full image as input bounding box.

```shell
python demo/image_demo.py \
    ${IMG_FILE} ${MMPOSE_CONFIG_FILE} ${MMPOSE_CHECKPOINT_FILE} \
    --out-file ${OUTPUT_FILE} \
    [--device ${GPU_ID or CPU}] \
    [--draw_heatmap]
```

The pre-trained human pose estimation models can be downloaded from [model zoo](https://mmpose.readthedocs.io/en/latest/model_zoo/body_2d_keypoint.html).
Take [coco model](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_coco_256x192-b9e0b3ab_20200708.pth) as an example:

```shell
    tests/data/coco/000000000785.jpg \

```shell
    configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_8xb32-210e_coco-256x192.py \
    https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_coco_256x192-b9e0b3ab_20200708.pth \
    --out-file vis_results.jpg \
    --draw-heatmap
```

To run this demo on CPU:

```shell
    --draw-heatmap \
    --device=cpu
```

<img src="https://user-images.githubusercontent.com/87690686/187824033-2cce0f55-034a-4127-82e2-52744178bc32.jpg" height="500px" alt><br>

#### Use mmdet for human bounding box detection

We provide a demo script to run mmdet for human detection, and mmpose for pose estimation.

Assume that you have already installed [mmdet](https://github.com/open-mmlab/mmdetection) with version >= 3.0.

```shell
    --input ${INPUT_PATH} \
    [--output-root ${OUTPUT_DIR}] [--save-predictions] \
    [--show] [--draw-heatmap] [--device ${GPU_ID or CPU}] \
    [--bbox-thr ${BBOX_SCORE_THR}] [--kpt-thr ${KPT_SCORE_THR}]
```

Example:

```shell

[--bbox-thr ${BBOX_SCORE_THR}] [--kpt-thr ${KPT_SCORE_THR}]
```

Example:

```shell
    configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192.py \
    https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_coco_256x192-c78dce93_20200708.pth \
    --input tests/data/coco/000000197388.jpg --show --draw-heatmap \
    --output-root vis_results/
```

<img src="https://user-images.githubusercontent.com/87690686/187824368-1f1631c3-52bf-4b45-bf9a-a70cd6551e1a.jpg" height="500px" alt><br>

To save the predicted results on disk, please specify `--save-predictions`.

### 2D Human Pose Top-Down Video Demo

The above demo script can also take video as input, and run mmdet for human detection, and mmpose for pose estimation. The difference is, the `${INPUT_PATH}` for videos can be the local path or **URL** link to video file.

Example:

```shell
    https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192-81c58e40_20220909.pth \

https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192-81c58e40_20220909.pth \
    --input tests/data/posetrack18/videos/000001_mpiinew_test/000001_mpiinew_test.mp4 \
    --output-root=vis_results/demo --show --draw-heatmap
```

### 2D Human Pose Bottom-up Image/Video Demo

We also provide a demo script using bottom-up models to estimate the human pose in an image or a video, which does not rely on human detectors.

```shell
python demo/bottomup_demo.py \
    [--show] [--device ${GPU_ID or CPU}] \
    [--kpt-thr ${KPT_SCORE_THR}]
```

Example:

```shell
    configs/body_2d_keypoint/dekr/coco/dekr_hrnet-w32_8xb10-140e_coco-512x512.py \

[--kpt-thr ${KPT_SCORE_THR}]
```

Example:

```shell
    https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/dekr/coco/dekr_hrnet-w32_8xb10-140e_coco-512x512_ac7c17bf-20221228.pth \
    --input tests/data/coco/000000197388.jpg --output-root=vis_results \
    --show --save-predictions
```

<img src="https://user-images.githubusercontent.com/26127467/207224032-a8dab45d-39e4-4b4e-80e0-3c71a64f5f39.jpg" height="300px" alt><br>

### 2D Human Pose Estimation with Inferencer

```shell
python demo/inferencer_demo.py \

```shell
    tests/data/posetrack18/videos/000001_mpiinew_test/000001_mpiinew_test.mp4 \
    --pose2d human --vis-out-dir vis_results/posetrack18
```

This command infers the video and saves the visualization results in the `vis_results/posetrack18` directory.

<img src="https://user-images.githubusercontent.com/26127467/229831445-44c9662b-edc5-4ef0-92a6-13558f0906cc.gif" alt="Image 1" height="300"/>

For top-down models, try to edit the config file. For example,

1. set `model.test_cfg.flip_test=False` in [topdown-res50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_res50_8xb64-210e_coco-256x192.py#L56).

<hr>
<h3>About CUB</h3>

Current release: v1.8.0 (02/16/2018)

We recommend the [CUB Project Website](http://nvlabs.github.com/cub) for further information and examples.

CUB provides state-of-the-art, reusable software components for every layer 
of the CUDA programming model:
- [<b><em>Device-wide primitives</em></b>] (https://nvlabs.github.com/cub/group___device_module.html) 
  - Sort, prefix scan, reduction, histogram, etc.  
  - Compatible with CUDA dynamic parallelism
- [<b><em>Block-wide "collective" primitives</em></b>] (https://nvlabs.github.com/cub/group___block_module.html)
  - I/O, sort, prefix scan, reduction, histogram, etc.  
  - Compatible with arbitrary thread block sizes and types 
- [<b><em>Warp-wide "collective" primitives</em></b>] (https://nvlabs.github.com/cub/group___warp_module.html)
  - Warp-wide prefix scan, reduction, etc.
  - Safe and architecture-specific

- [<b><em>Thread and resource utilities</em></b>](https://nvlabs.github.com/cub/group___thread_module.html)
  - PTX intrinsics, device reflection, texture-caching iterators, caching memory allocators, etc. 

![Orientation of collective primitives within the CUDA software stack](http://nvlabs.github.com/cub/cub_overview.png)

<br><hr>
<h3>A Simple Example</h3>

```C++
#include <cub/cub.cuh>
 
// Block-sorting CUDA kernel
__global__ void BlockSortKernel(int *d_in, int *d_out)
{
     using namespace cub;

     // Specialize BlockRadixSort, BlockLoad, and BlockStore for 128 threads 
     // owning 16 integer items each
     typedef BlockRadixSort<int, 128, 16>                     BlockRadixSort;
     typedef BlockLoad<int, 128, 16, BLOCK_LOAD_TRANSPOSE>   BlockLoad;

// owning 16 integer items each
     typedef BlockStore<int, 128, 16, BLOCK_STORE_TRANSPOSE> BlockStore;
 
     // Allocate shared memory
     __shared__ union {
         typename BlockRadixSort::TempStorage  sort;
         typename BlockLoad::TempStorage       load; 
         typename BlockStore::TempStorage      store; 
     } temp_storage; 

     int block_offset = blockIdx.x * (128 * 16);	  // OffsetT for this block's ment

     // Obtain a segment of 2048 consecutive keys that are blocked across threads
     int thread_keys[16];
     BlockLoad(temp_storage.load).Load(d_in + block_offset, thread_keys);
     __syncthreads();

     // Collectively sort the keys
     BlockRadixSort(temp_storage.sort).Sort(thread_keys);

     // Store the sorted segment

__syncthreads();

     // Store the sorted segment 
     BlockStore(temp_storage.store).Store(d_out + block_offset, thread_keys);
}
```

Each thread block uses cub::BlockRadixSort to collectively sort 
its own input segment.  The class is specialized by the 
data type being sorted, by the number of threads per block, by the number of 
keys per thread, and implicitly by the targeted compilation architecture.  

The cub::BlockLoad and cub::BlockStore classes are similarly specialized.    
Furthermore, to provide coalesced accesses to device memory, these primitives are 
configured to access memory using a striped access pattern (where consecutive threads 
simultaneously access consecutive items) and then <em>transpose</em> the keys into 
a [<em>blocked arrangement</em>](index.html#sec4sec3) of elements across threads.

a [<em>blocked arrangement</em>](index.html#sec4sec3) of elements across threads. 

Once specialized, these classes expose opaque \p TempStorage member types.  
The thread block uses these storage types to statically allocate the union of 
shared memory needed by the thread block.  (Alternatively these storage types 
could be aliased to global memory allocations).

<br><hr>
<h3>Stable Releases</h3>

CUB releases are labeled using version identifiers having three fields: 
*epoch.feature.update*.  The *epoch* field corresponds to support for
a major change in the CUDA programming model.  The *feature* field 
corresponds to a stable set of features, functionality, and interface.  The
*update* field corresponds to a bug-fix or performance update for that
feature set.  At the moment, we do not publicly provide non-stable releases 
such as development snapshots, beta releases or rolling releases.  (Feel free

to contact us if you would like such things.)  See the 
[CUB Project Website](http://nvlabs.github.com/cub) for more information.

<br><hr>
<h3>Contributors</h3>

CUB is developed as an open-source project by [NVIDIA Research](http://research.nvidia.com).  The primary contributor is [Duane Merrill](http://github.com/dumerrill).

<br><hr>
<h3>Open Source License</h3>

CUB is available under the "New BSD" open-source license:

```
Copyright (c) 2010-2011, Duane Merrill.  All rights reserved.
Copyright (c) 2011-2018, NVIDIA CORPORATION.  All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
   *  Redistributions of source code must retain the above copyright
      notice, this list of conditions and the following disclaimer.

*  Redistributions of source code must retain the above copyright
   *  Redistributions in binary form must reproduce the above copyright
      notice, this list of conditions and the following disclaimer in the
      documentation and/or other materials provided with the distribution.
   *  Neither the name of the NVIDIA CORPORATION nor the
      names of its contributors may be used to endorse or promote products
      derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL NVIDIA CORPORATION BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES

(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
```

# Efficient Evaluation

OpenCompass supports custom task partitioners (`Partitioner`), which enable flexible division of evaluation tasks. In conjunction with `Runner`, which controls the platform for task execution, such as a local machine or a cluster, OpenCompass can distribute large evaluation tasks to a vast number of computing nodes. This helps utilize computational resources efficiently and significantly accelerates the evaluation process.

By default, OpenCompass hides these details from users and automatically selects the recommended execution strategies. But users can still customize these strategies of the workflows according to their needs, just by adding the `infer` and/or `eval` fields to the configuration file:

```python
from opencompass.partitioners import SizePartitioner, NaivePartitioner
from opencompass.runners import SlurmRunner
from opencompass.tasks import OpenICLInferTask, OpenICLEvalTask

infer = dict(

    partitioner=dict(type=SizePartitioner, max_task_size=5000),
    runner=dict(
        type=SlurmRunner,
        max_num_workers=64,
        task=dict(type=OpenICLInferTask),
        retry=5),
)

eval = dict(
    partitioner=dict(type=NaivePartitioner),
        type=LocalRunner,
        max_num_workers=32,
        task=dict(type=OpenICLEvalTask)),
)
```

retry=5),
)

)
```

The example above demonstrates the way to configure the execution strategies for the inference and evaluation stages. At the inference stage, the task will be divided into several sub-tasks, each of 5000 samples, and then submitted to the Slurm cluster for execution, where there are at most 64 tasks running in parallel. At the evaluation stage, each single model-dataset pair forms a task, and 32 processes are launched locally to compute the metrics.

The following sections will introduce the involved modules in detail.

## Task Partition (Partitioner)

Due to the long inference time of large language models and the vast amount of evaluation datasets, serial execution of a single evaluation task can be quite time-consuming. OpenCompass allows custom task partitioners (`Partitioner`) to divide large evaluation tasks into numerous independent smaller tasks, thus fully utilizing computational resources via parallel execution. Users can configure the task partitioning strategies for the inference and evaluation stages via `infer.partitioner` and `eval.partitioner`. Below, we will introduce all the partitioning strategies supported by OpenCompass.

### `NaivePartitioner`

This partitioner dispatches each combination of a model and dataset as an independent task. It is the most basic partitioning strategy and does not have any additional parameters.

```python
from opencompass.partitioners import NaivePartitioner

```python

    partitioner=dict(type=NaivePartitioner)
    # ...
)
```

### `SizePartitioner`

For now, this partitioner is not suitable for evaluation tasks (`OpenICLEvalTask`).
```

This partitioner estimates the inference cost (time) of a dataset according to its size, multiplied by an empirical expansion coefficient. It then creates tasks by splitting larger datasets and merging smaller ones to ensure the inference costs of each sub-task are as equal as possible.

The commonly used parameters for this partitioner are as follows:

```python
from opencompass.partitioners import SizePartitioner

    partitioner=dict(
        type=SizePartitioner,
        max_task_size: int = 2000,  # Maximum size of each task
        gen_task_coef: int = 20,  # Expansion coefficient for generative tasks
    ),
    # ...
)
```

type=SizePartitioner,
    ),
    # ...
)
```

`SizePartitioner` estimates the inference cost of a dataset based on the type of the inference task and selects different expansion coefficients accordingly. For generative tasks, such as those using `GenInferencer`, a larger `gen_task_coef` is set; for discriminative tasks, like those using `PPLInferencer`, the number of labels in the prompt is used.

```{note}
Currently, this partitioning strategy is still rather crude and does not accurately reflect the computational difference between generative and discriminative tasks. We look forward to the community proposing better partitioning strategies :)
```

## Execution Backend (Runner)

```

In a multi-card, multi-machine cluster environment, if we want to implement parallel execution of multiple tasks, we usually need to rely on a cluster management system (like Slurm) for task allocation and scheduling. In OpenCompass, task allocation and execution are uniformly handled by the Runner. Currently, it supports both Slurm and PAI-DLC scheduling backends, and also provides a `LocalRunner` to directly launch tasks on the local machine.

### `LocalRunner`

`LocalRunner` is the most basic runner that can run tasks parallelly on the local machine.

```python
from opencompass.runners import LocalRunner
from opencompass.tasks import OpenICLInferTask

    # ...
        max_num_workers=16,  # Maximum number of processes to run in parallel
        task=dict(type=OpenICLEvalTask),  # Task to be run
    )
)
```

```{note}

type=LocalRunner,
    )
)
```

```{note}
The actual number of running tasks are both limited by the actual available GPU resources and the number of workers.
```

### `SlurmRunner`

`SlurmRunner` submits tasks to run on the Slurm cluster. The commonly used configuration fields are as follows:

```python

    # ...
        max_num_workers=16,  # Maximum concurrent evaluation task count
        retry=2,  # Retry count for failed tasks, can prevent accidental errors
    ),
)
```

### `DLCRunner`

max_num_workers=16,  # Maximum concurrent evaluation task count
    ),
)
```

`DLCRunner` submits tasks to run on Alibaba's Deep Learning Center (DLC). This Runner depends on `dlc`. Firstly, you need to prepare `dlc` in the environment:

```bash
cd ~
wget https://dlc-cli.oss-cn-zhangjiakou.aliyuncs.com/light/binary/linux/amd64/dlc
chmod +x ./dlc
sudo ln -rs dlc /usr/local/bin
./dlc config
```

Fill in the necessary information according to the prompts and get the `dlc` configuration file (like `/user/.dlc/config`) to complete the preparation. Then, specify the `DLCRunner` configuration in the configuration file as per the format:

```python
from opencompass.runners import DLCRunner

    # ...
        type=DLCRunner,

    # ...
        aliyun_cfg=dict(
            bashrc_path="/user/.bashrc",  # Path to the bashrc for initializing the running environment
            conda_env_name='opencompass',  # Conda environment for OpenCompass
            dlc_config_path="/user/.dlc/config",  # Configuration file for dlc
            workspace_id='ws-xxx',  # DLC workspace ID
            worker_image='xxx',  # Image url for running tasks
        ),
    ),
)
```

## Task

worker_image='xxx',  # Image url for running tasks
        ),
    ),
)
```

## Task

A Task is a fundamental module in OpenCompass, a standalone script that executes the computationally intensive operations. Each task is designed to load a configuration file to determine parameter settings, and it can be executed in two distinct ways:

2. Instantiate a Task object, then call `task.run()`.
3. Call `get_command` method by passing in the config path and the command template string that contains `{task_cmd}` as a placeholder (e.g. `srun {task_cmd}`). The returned command string will be the full command and can be executed directly.

As of now, OpenCompass supports the following task types:

- `OpenICLInferTask`: Perform LM Inference task based on OpenICL framework.
- `OpenICLEvalTask`: Perform LM Evaluation task based on OpenEval framework.

In the future, more task types will be supported.

# Preparing AVA-Kinetics

```BibTeX
@article{li2020ava,
  title={The ava-kinetics localized human actions video dataset},
  author={Li, Ang and Thotakuri, Meghana and Ross, David A and Carreira, Jo{\~a}o and Vostrikov, Alexander and Zisserman, Andrew},
  journal={arXiv preprint arXiv:2005.00214},
}
```

For basic dataset information, please refer to the official [website](https://research.google.com/ava/index.html).
AVA-Kinetics dataset is a crossover between the AVA Actions and Kinetics datasets. You may want to first prepare the AVA datasets. In this file, we provide commands to prepare the Kinetics part and merge the two parts together.

For model training, we will keep reading from raw frames for the AVA part, but read from videos using `decord` for the Kinetics part to accelerate training.

Before we start, please make sure that the directory is located at `$MMACTION2/tools/data/ava_kinetics/`.

## Step 1. Prepare the Kinetics700 dataset

The Kinetics part of the AVA-Kinetics dataset are sampled from the Kinetics-700 dataset.

It is best if you have prepared the Kinetics-700 dataset (only videos required) following
[Preparing Kinetics](https://github.com/open-mmlab/mmaction2/tree/master/tools/data/kinetics). We will also have alternative method to prepare these videos if you do not have enough storage (coming soon).

We will need the videos of this dataset (`$MMACTION2/data/kinetics700/videos_train`) and the videos file list (`$MMACTION2/data/kinetics700/kinetics700_train_list_videos.txt`), which is generated by [Step 4 in Preparing Kinetics](https://github.com/open-mmlab/mmaction2/tree/master/tools/data/kinetics#step-4-generate-file-list)

The format of the file list should be:

```
Path_to_video_1 label_1\n
Path_to_video_2 label_2\n
...
Path_to_video_n label_n\n
```

```
...
```

The timestamp (start and end of the video) must be contained. For example:

```
class602/o3lCwWyyc_s_000012_000022.mp4 602\n
```

It means that this video clip is the 12th to 22nd seconds of the original video. It is okay if some videos are missing, and we will ignore them in the next steps.

## Step 2. Download Annotations

Download the annotation tar file (recall that the directory should be located at `$MMACTION2/tools/data/ava_kinetics/`).

```shell
wget https://storage.googleapis.com/deepmind-media/Datasets/ava_kinetics_v1_0.tar.gz
tar xf ava_kinetics_v1_0.tar.gz && rm ava_kinetics_v1_0.tar.gz
```

You should have the `ava_kinetics_v1_0` folder at `$MMACTION2/tools/data/ava_kinetics/`.

## Step 3. Cut Videos

```

Use `cut_kinetics.py` to find the desired videos from the Kinetics-700 dataset and trim them to contain only annotated clips. Currently we only use the train set of the Kinetics part to improve training. Validation on the Kinetics part will come soon.

Here is the script:

```shell
python3 cut_kinetics.py --avakinetics_anotation=$AVAKINETICS_ANOTATION \
                        --kinetics_list=$KINETICS_LIST \
                        --avakinetics_root=$AVAKINETICS_ROOT \
                        [--num_workers=$NUM_WORKERS ]
```

Arguments:

- `avakinetics_anotation`: the directory to ava-kinetics anotations. Defaults to `./ava_kinetics_v1_0`.

[--num_workers=$NUM_WORKERS ]
```

- `kinetics_list`: the path to the videos file list as mentioned in Step 1. If you have prepared the Kinetics700 dataset following `mmaction2`, it should be `$MMACTION2/data/kinetics700/kinetics700_train_list_videos.txt`.
- `avakinetics_root`: the directory to save the ava-kinetics dataset. Defaults to `$MMACTION2/data/ava_kinetics`.
- `num_workers`: number of workers used to cut videos. Defaults to -1 and use all available cpus.

There should be about 100k videos. It is OK if some videos are missing and we will ignore them in the next steps.

## Step 4. Extract RGB Frames

This step is similar to Step 4 in [Preparing AVA](https://github.com/open-mmlab/mmaction2/tree/main/tools/data/ava#step-4-extract-rgb-and-flow).

Here we provide a script to extract RGB frames using ffmpeg:

```shell

```shell
python3 extract_rgb_frames.py --avakinetics_root=$AVAKINETICS_ROOT \
                              [--num_workers=$NUM_WORKERS ]
```

- `num_workers`: number of workers used to extract frames. Defaults to -1 and use all available cpus.

If you have installed denseflow, you can also use `build_rawframes.py` to extract RGB frames:

```shell
python3 ../build_rawframes.py ../../../data/ava_kinetics/videos/ ../../../data/ava_kinetics/rawframes/ --task rgb --level 1 --mixed-ext
```

## Step 5. Prepare Annotations

Use `prepare_annotation.py` to prepare the training annotations. It will generate a `kinetics_train.csv` file containning the spatial-temporal annotations for the Kinetics part, localting at `$AVAKINETICS_ROOT`.

```shell

```shell
python3 prepare_annotation.py --avakinetics_anotation=$AVAKINETICS_ANOTATION \
                              --avakinetics_root=$AVAKINETICS_ROOT \
                              [--num_workers=$NUM_WORKERS]
```

- `num_workers`: number of workers used to prepare annotations. Defaults to -1 and use all available cpus.

## Step 6. Fetch Proposal Files

The pre-computed proposals for AVA dataset are provided by FAIR's [Long-Term Feature Banks](https://github.com/facebookresearch/video-long-term-feature-banks). For the Kinetics part, we use `Cascade R-CNN X-101-64x4d-FPN` from [mmdetection](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_x101_64x4d_fpn_1x_coco/cascade_rcnn_x101_64x4d_fpn_1x_coco_20200515_075702-43ce6a30.pth) to fetch the proposals. Here is the script:

```shell
python3 fetch_proposal.py --avakinetics_root=$AVAKINETICS_ROOT \
                          --datalist=$DATALIST \
                          --picklepath=$PICKLEPATH \
                          [--config=$CONFIG ] \
                          [--checkpoint=$CHECKPOINT ]

```

It  will generate a `kinetics_proposal.pkl` file at `$MMACTION2/data/ava_kinetics/`.

[--checkpoint=$CHECKPOINT ]

```

- `datalist`: path to the `kinetics_train.csv` file generated at Step 3.
- `picklepath`: path to save the extracted proposal pickle file.
- `config`: the config file for the human detection model. Defaults to `X-101-64x4d-FPN.py`.
- `checkpoint`: the checkpoint for the human detection model. Defaults to the `mmdetection` pretraining checkpoint.

## Step 7. Merge AVA to AVA-Kinetics

Now we are done with the preparations for the Kinetics part. We need to merge the AVA part into the `ava_kinetics` folder (assuming you have AVA dataset ready at `$MMACTION2/data/ava`). First we make a copy of the AVA anotation to the `ava_kinetics` folder (recall that you are at `$MMACTION2/tools/data/ava_kinetics/`):

```shell

```shell
cp -r ../../../data/ava/annotations/ ../../../data/ava_kinetics/
```

Next we merge the generated anotation files of the Kinetics part to AVA. Please check: you should have two files `kinetics_train.csv` and `kinetics_proposal.pkl` at `$MMACTION2/data/ava_kinetics/` generated from Step 5 and Step 6. Run the following script to merge these two files into `$MMACTION2/data/ava_kinetics/annotations/ava_train_v2.2.csv` and `$MMACTION2/data/ava_kinetics/annotations/ava_dense_proposals_train.FAIR.recall_93.9.pkl` respectively.

```shell
python3 merge_annotations.py --avakinetics_root=$AVAKINETICS_ROOT
```

Finally, we need to merge the rawframes of AVA part. You can either copy/move them or generate soft links. The following script is an example to use soft links:

```shell
python3 softlink_ava.py --avakinetics_root=$AVAKINETICS_ROOT \

```shell
                        --ava_root=$AVA_ROOT
```

- `ava_root`: the directory to save the ava dataset. Defaults to `$MMACTION2/data/ava`.

# Package structures changes

This section is included if you are curious about what has changed between MMSeg 0.x and 1.x.

<table>
<tr>
<td>MMSegmentation 0.x</td>
<td>MMSegmentation 1.x</td>
</tr>
<tr>
<td>mmseg.api</td>
</tr>
<tr>
<td bgcolor=#fcf7f7>- mmseg.core</td>
<td bgcolor=#ecf4eb>+ mmseg.engine</td>
</tr>
<tr>
<td>mmseg.datasets</td>
</tr>
<tr>
<td>mmseg.models</td>
</tr>
<tr>
<td bgcolor=#fcf7f7>- mmseg.ops</td>
<td bgcolor=#ecf4eb>+ mmseg.structure</td>
</tr>
<tr>
<td>mmseg.utils</td>
</tr>
<tr>
<td></td>
<td bgcolor=#ecf4eb>+ mmseg.evaluation</td>
</tr>
<tr>
<td></td>
<td bgcolor=#ecf4eb>+ mmseg.registry</td>
<tr>
</table>

## Removed packages

### `mmseg.core`

In OpenMMLab 2.0, `core` package has been removed. `hooks` and `optimizers` of `core` are moved in `mmseg.engine`, and `evaluation` in `core` is mmseg.evaluation currently.

## `mmseg.ops`

`ops` package included `encoding` and `wrappers`, which are moved in `mmseg.models.utils`.

## Added packages

### `mmseg.engine`

OpenMMLab 2.0 adds a new foundational library for training deep learning, MMEngine. It servers as the training engine of all OpenMMLab codebases.
`engine` package of mmseg is some customized modules for semantic segmentation task, like `SegVisualizationHook` which works for visualizing segmentation mask.

### `mmseg.structure`

In OpenMMLab 2.0, we designed data structure for computer vision task, and in mmseg, we implements `SegDataSample` in `structure` package.

### `mmseg.evaluation`

We move all evaluation metric in `mmseg.evaluation`.

### `mmseg.registry`

We moved registry implementations for all kinds of modules in MMSegmentation in `mmseg.registry`.

## Modified packages

### `mmseg.apis`

OpenMMLab 2.0 tries to support unified interface for multitasking of Computer Vision, and releases much stronger [`Runner`](https://github.com/open-mmlab/mmengine/blob/main/docs/en/design/runner.md), so MMSeg 1.x removed modules in `train.py` and `test.py` renamed `init_segmentor` to `init_model` and `inference_segmentor` to `inference_model`.

Here is the changes of `mmseg.apis`:

|       Function        | Changes                                         |
| :-------------------: | :---------------------------------------------- |
|   `init_segmentor`    | Renamed to `init_model`                         |
| `inference_segmentor` | Rename to `inference_model`                     |

| `show_result_pyplot`  | Implemented based on `SegLocalVisualizer`       |
|     `train_model`     | Removed, use `runner.train` to train.           |
|   `multi_gpu_test`    | Removed, use `runner.test` to test.             |
|   `single_gpu_test`   | Removed, use `runner.test` to test.             |
|   `set_random_seed`   | Removed, use `mmengine.runner.set_random_seed`. |
|  `init_random_seed`   | Removed, use `mmengine.dist.sync_random_seed`.  |

### `mmseg.datasets`

OpenMMLab 2.0 defines the `BaseDataset` to function and interface of dataset, and MMSegmentation 1.x also follow this protocol and defines the `BaseSegDataset` inherited from `BaseDataset`. MMCV 2.x collects general data transforms for multiple tasks e.g. classification, detection, segmentation, so MMSegmentation 1.x uses these data transforms and removes them from mmseg.datasets.

|   Packages/Modules    | Changes                                                                                     |
| :-------------------: | :------------------------------------------------------------------------------------------ |
|   `mmseg.pipelines`   | Moved in `mmcv.transforms`                                                                  |

|    `mmseg.sampler`    | Moved in `mmengine.dataset.sampler`                                                         |
|    `CustomDataset`    | Renamed to `BaseSegDataset` and inherited from `BaseDataset` in MMEngine                    |
| `DefaultFormatBundle` | Replaced with `PackSegInputs`                                                               |
|  `LoadImageFromFile`  | Moved in `mmcv.transforms.LoadImageFromFile`                                                |
|   `LoadAnnotations`   | Moved in `mmcv.transforms.LoadAnnotations`                                                  |
|       `Resize`        | Moved in `mmcv.transforms` and split into `Resize`, `RandomResize` and `RandomChoiceResize` |
|     `RandomFlip`      | Moved in `mmcv.transforms.RandomFlip`                                                       |

|         `Pad`         | Moved in `mmcv.transforms.Pad`                                                              |
|      `Normalize`      | Moved in `mmcv.transforms.Normalize`                                                        |
|       `Compose`       | Moved in `mmcv.transforms.Compose`                                                          |
|    `ImageToTensor`    | Moved in `mmcv.transforms.ImageToTensor`                                                    |

### `mmseg.models`

`models` has not changed a lot, just added the `encoding` and `wrappers` from previous `mmseg.ops`

  <img src="resources/lmdeploy-logo.png" width="450"/>

[![docs](https://img.shields.io/badge/docs-latest-blue)](https://lmdeploy-zh-cn.readthedocs.io/zh_CN/latest/)
[![badge](https://github.com/InternLM/lmdeploy/workflows/lint/badge.svg)](https://github.com/InternLM/lmdeploy/actions)
[![PyPI](https://img.shields.io/pypi/v/lmdeploy)](https://pypi.org/project/lmdeploy)
[![license](https://img.shields.io/github/license/InternLM/lmdeploy.svg)](https://github.com/InternLM/lmdeploy/tree/main/LICENSE)
[![issue resolution](https://img.shields.io/github/issues-closed-raw/InternLM/lmdeploy)](https://github.com/InternLM/lmdeploy/issues)
[![open issues](https://img.shields.io/github/issues-raw/InternLM/lmdeploy)](https://github.com/InternLM/lmdeploy/issues)

</div>

<p align="center">

</div>

    👋 join us on <a href="https://twitter.com/intern_lm" target="_blank">Twitter</a>, <a href="https://discord.gg/xa29JuW87d" target="_blank">Discord</a> and <a href="https://r.vansin.top/?r=internwx" target="_blank">WeChat</a>
</p>

## 更新 🎉

- \[2023/08\] TurboMind 支持 Qwen-7B，动态NTK-RoPE缩放，动态logN缩放
- \[2023/08\] TurboMind 支持 Windows (tp=1)
- \[2023/08\] TurboMind 支持 4-bit 推理，速度是 FP16 的 2.4 倍，是目前最快的开源实现🚀。部署方式请看[这里](./docs/zh_cn/w4a16.md)
- \[2023/08\] LMDeploy 开通了 [HuggingFace Hub](https://huggingface.co/lmdeploy) ，提供开箱即用的 4-bit 模型
- \[2023/08\] LMDeploy 支持使用 [AWQ](https://arxiv.org/abs/2306.00978) 算法进行 4-bit 量化
- \[2023/07\] TurboMind 支持使用 GQA 的 Llama-2 70B 模型
- \[2023/07\] TurboMind 支持 Llama-2 7B/13B 模型

- \[2023/07\] TurboMind 支持 InternLM 的 Tensor Parallel 推理

## 简介

LMDeploy 由 [MMDeploy](https://github.com/open-mmlab/mmdeploy) 和 [MMRazor](https://github.com/open-mmlab/mmrazor) 团队联合开发，是涵盖了 LLM 任务的全套轻量化、部署和服务解决方案。
这个强大的工具箱提供以下核心功能：

- **高效推理引擎 TurboMind**：基于 [FasterTransformer](https://github.com/NVIDIA/FasterTransformer)，我们实现了高效推理引擎 TurboMind，支持 InternLM、LLaMA、vicuna等模型在 NVIDIA GPU 上的推理。

- **交互推理方式**：通过缓存多轮对话过程中 attention 的 k/v，记住对话历史，从而避免重复处理历史会话。

- **多 GPU 部署和量化**：我们提供了全面的模型部署和量化支持，已在不同规模上完成验证。

- **persistent batch 推理**：进一步优化模型执行效率。

  ![PersistentBatchInference](https://github.com/InternLM/lmdeploy/assets/67539920/e3876167-0671-44fc-ac52-5a0f9382493e)

## 支持的模型

`LMDeploy` 支持 `TurboMind` 和 `Pytorch` 两种推理后端

### TurboMind

![PersistentBatchInference](https://github.com/InternLM/lmdeploy/assets/67539920/e3876167-0671-44fc-ac52-5a0f9382493e)

## 支持的模型

> **Note**<br />
> W4A16 推理需要 Ampere 及以上架构的 Nvidia GPU

|   模型   | 模型并行 | FP16 | KV INT8 | W4A16 | W8A8 |
| :------: | :------: | :--: | :-----: | :---: | :--: |
|  Llama   |   Yes    | Yes  |   Yes   |  Yes  |  No  |
|  Llama2  |   Yes    | Yes  |   Yes   |  Yes  |  No  |
| InternLM |   Yes    | Yes  |   Yes   |  Yes  |  No  |

### Pytorch

|  Llama   |   Yes    | Yes  |   No    |  No   |  No  |
|  Llama2  |   Yes    | Yes  |   No    |  No   |  No  |
| InternLM |   Yes    | Yes  |   No    |  No   |  No  |

## 性能

**场景一**: 固定的输入、输出token数（1,2048），测试 output token throughput

**场景二**: 使用真实数据，测试 request throughput

测试配置：LLaMA-7B, NVIDIA A100(80G)

## 性能

TurboMind 的 output token throughput 超过 2000 token/s, 整体比 DeepSpeed 提升约 5% - 15%，比 huggingface transformers 提升 2.3 倍
在 request throughput 指标上，TurboMind 的效率比 vLLM 高 30%

![benchmark](https://github.com/InternLM/lmdeploy/assets/4560679/7775c518-608e-4e5b-be73-7645a444e774)

## 快速上手

### 安装

使用 pip ( python 3.8+) 安装 LMDeploy，或者[源码安装](./docs/zh_cn/build.md)

```shell
```

### 部署 InternLM

#### 获取 InternLM 模型

```shell
# 1. 下载 InternLM 模型

# Make sure you have git-lfs installed (https://git-lfs.com)
git lfs install
git clone https://huggingface.co/internlm/internlm-chat-7b /path/to/internlm-chat-7b

# if you want to clone without large files – just their pointers
# prepend your git clone with the following env var:
GIT_LFS_SKIP_SMUDGE=1

# 2. 转换为 trubomind 要求的格式。默认存放路径为 ./workspace
python3 -m lmdeploy.serve.turbomind.deploy internlm-chat-7b /path/to/internlm-chat-7b

```

#### 使用 turbomind 推理

```shell
```

> turbomind 在使用 FP16 精度推理 InternLM-7B 模型时，显存开销至少需要 15.7G。建议使用 3090, V100，A100等型号的显卡。<br />
> 关闭显卡的 ECC 可以腾出 10% 显存，执行 `sudo nvidia-smi --ecc-config=0` 重启系统生效。

> 使用 Tensor 并发可以利用多张 GPU 进行推理。在 `chat` 时添加参数 `--tp=<num_gpu>` 可以启动运行时 TP。

#### 启动 gradio server

```shell
python3 -m lmdeploy.serve.gradio.app ./workspace
```

![](https://github.com/InternLM/lmdeploy/assets/67539920/08d1e6f2-3767-44d5-8654-c85767cec2ab)

#### 通过 Restful API 部署服务

使用下面的命令启动推理服务：

```shell
python3 -m lmdeploy.serve.openai.api_server ./workspace server_ip server_port --instance_num 32 --tp 1
```

你可以通过命令行方式与推理服务进行对话：

```shell
```

```shell
```

也可以通过 WebUI 方式来对话：

```shell
```

更多详情可以查阅 [restful_api.md](docs/zh_cn/restful_api.md)。

#### 通过容器部署推理服务

```shell
bash workspace/service_docker_up.sh
```

```shell
python3 -m lmdeploy.serve.client {server_ip_addresss}:33337
```

```shell

```shell
```

```shell
```

```shell
python3 -m lmdeploy.serve.gradio.app {server_ip_addresss}:33337
```

其他模型的部署方式，比如 LLaMA，LLaMA-2，vicuna等等，请参考[这里](docs/zh_cn/serving.md)

### 基于 PyTorch 的推理

你必须确保环境中有安装 deepspeed：

```
pip install deepspeed
```

#### 单个 GPU

```shell
python3 -m lmdeploy.pytorch.chat $NAME_OR_PATH_TO_HF_MODEL\
    --max_new_tokens 64 \
    --temperture 0.8 \
    --top_p 0.95 \
    --seed 0
```

#### 使用 DeepSpeed 实现张量并行

```shell
deepspeed --module --num_gpus 2 lmdeploy.pytorch.chat \
    $NAME_OR_PATH_TO_HF_MODEL \
    --seed 0
```

## 量化部署

### Step 1. 获取量化参数

首先，执行量化脚本，获取量化参数

> 执行后，量化需要的各种参数会存放在 $WORK_DIR 中; 接下来的步骤中会用到

```

--top_p 0.95 \
    --seed 0
```

## 量化部署

```

```

### Step 2. 实际量化模型

目前支持对权重的 INT4 量化和 KV Cache 的 INT8 量化，根据需求执行对应脚本即可

#### 权重 INT4 量化

LMDeploy 使用 [AWQ](https://arxiv.org/abs/2306.00978) 算法对模型权重进行量化

> 需要输入第一步的 \`$WORK_DIR\`\` ，量化后的权重也会存在这个文件夹中

```
  --work_dir $WORK_DIR \             # Step 1 保存量化参数的目录
```

[点击这里](./docs/zh_cn/w4a16.md) 查看 weight int4 用法测试结果。

#### KV Cache INT8 量化

--w_group_size 128 \               # 权重量化分组统计尺寸
```

[点击这里](./docs/zh_cn/kv_int8.md) 查看 kv int8 使用方法、实现公式和测试结果。

> **Warning**<br />
> 量化部署不支持运行时 Tensor 并发。如果希望使用 Tensor 并发，需要在 deploy 时配置 tp 参数。

## 贡献指南

我们感谢所有的贡献者为改进和提升 LMDeploy 所作出的努力。请参考[贡献指南](.github/CONTRIBUTING.md)来了解参与项目贡献的相关指引。

## 致谢

- [FasterTransformer](https://github.com/NVIDIA/FasterTransformer)
- [llm-awq](https://github.com/mit-han-lab/llm-awq)

## License

<details>

```bibtex
}
```

<details>

```bibtex

```bibtex
}
```

| [pose_mobilenetv2](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_mobilenetv2_8xb64-210e_coco-256x192.py) |  256x192   | 0.648 |      0.874      |      0.725      | 0.709 |      0.918      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_mobilenetv2_8xb64-210e_coco-256x192-55a04c35_20221016.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_mobilenetv2_8xb64-210e_coco-256x192_20221016.log) |
| [pose_mobilenetv2](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_mobilenetv2_8xb64-210e_coco-384x288.py) |  384x288   | 0.677 |      0.882      |      0.746      | 0.734 |      0.920      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_mobilenetv2_8xb64-210e_coco-384x288-d3ab1457_20221013.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_mobilenetv2_8xb64-210e_coco-384x288_20221013.log) |

# Precompiled package

This document is going to describe the way to build MMDeploy package.

## Prerequisites

- Download and install Miniconda from the [official website](https://docs.conda.io/en/latest/miniconda.html).

- Create conda environments for python 3.6, 3.7, 3.8, 3.9 and 3.10, respectively.

  ```shell
  for PYTHON_VERSION in 3.6 3.7 3.8 3.9 3.10
  do
    conda create --name mmdeploy-$PYTHON_VERSION python=$PYTHON_VERSION -y
  done
  ```

- Prepare MMDeploy dependencies

  Please follow the [build-on-Linux guide](../../docs/en/01-how-to-build/linux-x86_64.md) or [build-on-Windows guide](../../docs/en/01-how-to-build/linux-x86_64.md) to install dependencies of MMDeploy,
  including PyTorch, MMCV, OpenCV, ppl.cv, ONNX Runtime and TensorRT.

  Make sure the environment variables `pplcv_DIR`, `ONNXRUNTIME_DIR`, `TENSORRT_DIR`, `CUDNN_DIR` and `CUDA_TOOLKIT_ROOT_DIR` are exported.

## Run precompiled command

- On Linux platform,

  ```shell
  conda activate mmdeploy-3.6

  ```shell
  pip install pyyaml packaging
  cd the/root/path/of/mmdeploy
  python tools/package_tools/generate_build_config.py --backend 'ort' \
            --system linux --build-mmdeploy --device cpu --build-sdk \
            --build-sdk-monolithic --build-sdk-python --sdk-dynamic-net \
            --output config.yml
  python tools/package_tools/mmdeploy_builder.py --config config.yml --output-dir pack
  ```

- On Windows platform, open `Anaconda Powershell Prompt` from the start menu and execute:

  ```shell
  cd the/root/path/of/MMDeploy
            --system windows --build-mmdeploy --device cpu --build-sdk \

--system windows --build-mmdeploy --device cpu --build-sdk \
  ```

## NCNN Image Watch Plugin for Visual Studio
Image Watch plugin is a good tool for better understanding insight of images. This tiny work offer a ".natvis" file which could add ncnn::Mat class support for Image Watch, and users could debug ncnn::Mat image just like debuging cv::Mat via Image Watch.

To use this plugin, please move this "ImageWatchNCNN.natvis" file to "C:/user/${your user name}/Documents/Visual Studio ${VS_Version}/Visualizers" folder. If not exist this folder, create it(such as: "C:\Users\nihui\Documents\Visual Studio 2017\Visualizers"). 

![](https://github.com/Tencent/ncnn/blob/master/tools/plugin/ncnn_snapshot.png)

See [Image Watch Help](https://imagewatch.azurewebsites.net/ImageWatchHelp/ImageWatchHelp.htm) page for more advanced using tips of Image Watch(For example, get single channel from channels, such as getting confidence heatmap from forward result list {confidence, x1, y1, x2, y2}).

## NNIE Image Watch Plugin for Visual Studio

This image plugin will be a part of NNIE Plugin for NCNN(NPN). NPN will be a WIP, it should be completed by the end of Septembe.

The plugin support SVP_BLOB_TYPE_S32 and SVP_BLOB_TYPE_U8 for now.

![](https://github.com/Tencent/ncnn/blob/master/tools/plugin/nnie_snapshot.png)

# Class Activation Map (CAM) Visualization

## Introduction of the CAM visualization tool

MMPretrain provides `tools/visualization/vis_cam.py` tool to visualize class activation map. Please use `pip install "grad-cam>=1.3.6"` command to install [pytorch-grad-cam](https://github.com/jacobgil/pytorch-grad-cam).

The supported methods are as follows:

| Method       | What it does                                                                                                                 |
| ------------ | ---------------------------------------------------------------------------------------------------------------------------- |
| GradCAM      | Weight the 2D activations by the average gradient                                                                            |
| GradCAM++    | Like GradCAM but uses second order gradients                                                                                 |

| XGradCAM     | Like GradCAM but scale the gradients by the normalized activations                                                           |
| EigenCAM     | Takes the first principle component of the 2D Activations (no class discrimination, but seems to give great results)         |
| EigenGradCAM | Like EigenCAM but with class discrimination: First principle component of Activations\*Grad. Looks like GradCAM, but cleaner |
| LayerCAM     | Spatially weight the activations by positive gradients. Works better especially in lower layers                              |

More CAM methods supported by the new version `pytorch-grad-cam` can also be used but we haven't verified the availability.

```bash
python tools/visualization/vis_cam.py \
    ${IMG} \

```bash
    ${IMG} \
    [--target-layers ${TARGET-LAYERS}] \
    [--preview-model] \
    [--method ${METHOD}] \
    [--target-category ${TARGET-CATEGORY}] \
    [--vit-like] \
    [--num-extra-tokens ${NUM-EXTRA-TOKENS}]
    [--aug_smooth] \
    [--eigen_smooth] \
    [--device ${DEVICE}] \
    [--cfg-options ${CFG-OPTIONS}]
```

- `img`: The target picture path.
- `config`: The path of the model config file.
- `checkpoint`: The path of the checkpoint.
- `--target-layers`: The target layers to get activation maps, one or more network layers can be specified. If not set, use the norm layer of the last block.
- `--preview-model`: Whether to print all network layer names in the model.

- `--method`: Visualization method, supports `GradCAM`, `GradCAM++`, `XGradCAM`, `EigenCAM`, `EigenGradCAM`, `LayerCAM`, which is case insensitive. Defaults to `GradCAM`.
- `--target-category`: Target category, if not set, use the category detected by the given model.
- `--eigen-smooth`: Whether to use the principal component to reduce noise.
- `--aug-smooth`: Whether to use TTA(Test Time Augment) to get CAM.
- `--save-path`: The path to save the CAM visualization image. If not set, the CAM image will not be saved.
- `--vit-like`: Whether the network is ViT-like network.
- `--num-extra-tokens`: The number of extra tokens in ViT-like backbones. If not set, use num_extra_tokens the backbone.
- `--device`: The computing device used. Default to 'cpu'.
- `--cfg-options`: Modifications to the configuration file, refer to [Learn about Configs](../user_guides/config.md).

```{note}

```{note}
The argument `--preview-model` can view all network layers names in the given model. It will be helpful if you know nothing about the model layers when setting `--target-layers`.
```

## How to visualize the CAM of CNN (ResNet-50)

Here are some examples of `target-layers` in ResNet-50, which can be any module or layer:

- `'backbone.layer4'` means the output of the forth ResLayer.
- `'backbone.layer4.2'` means the output of the third BottleNeck block in the forth ResLayer.
- `'backbone.layer4.2.conv1'` means the output of the `conv1` layer in above BottleNeck block.

1. Use different methods to visualize CAM for `ResNet50`, the `target-category` is the predicted result by the given checkpoint, using the default `target-layers`.

   ```shell
   python tools/visualization/vis_cam.py \

```shell
       demo/bird.JPEG \
       configs/resnet/resnet50_8xb32_in1k.py \
       https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_batch256_imagenet_20200708-cfb998bf.pth \
       --method GradCAM
       # GradCAM++, XGradCAM, EigenCAM, EigenGradCAM, LayerCAM
   ```

   | Image                                | GradCAM                                 | GradCAM++                                 | EigenGradCAM                                 | LayerCAM                                 |
   | ------------------------------------ | --------------------------------------- | ----------------------------------------- | -------------------------------------------- | ---------------------------------------- |

| <div align=center><img src='https://user-images.githubusercontent.com/18586273/144429496-628d3fb3-1f6e-41ff-aa5c-1b08c60c32a9.JPEG' height="auto" width="160" ></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/147065002-f1c86516-38b2-47ba-90c1-e00b49556c70.jpg' height="auto" width="150" ></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/147065119-82581fa1-3414-4d6c-a849-804e1503c74b.jpg' height="auto" width="150"></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/147065096-75a6a2c1-6c57-4789-ad64-ebe5e38765f4.jpg' height="auto" width="150"></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/147065129-814d20fb-98be-4106-8c5e-420adcc85295.jpg' height="auto" width="150"></div> |

2. Use different `target-category` to get CAM from the same picture. In `ImageNet` dataset, the category 238 is 'Greater Swiss Mountain dog', the category 281 is 'tabby, tabby cat'.

   ```shell
       demo/cat-dog.png configs/resnet/resnet50_8xb32_in1k.py \
       --target-layers 'backbone.layer4.2' \
       --method GradCAM \
       --target-category 238
       # --target-category 281
   ```

   | Category | Image                                          | GradCAM                                          | XGradCAM                                          | LayerCAM                                          |
   | -------- | ---------------------------------------------- | ------------------------------------------------ | ------------------------------------------------- | ------------------------------------------------- |

| Dog      | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144429526-f27f4cce-89b9-4117-bfe6-55c2ca7eaba6.png' height="auto" width="165" ></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144433562-968a57bc-17d9-413e-810e-f91e334d648a.jpg' height="auto" width="150" ></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144433853-319f3a8f-95f2-446d-b84f-3028daca5378.jpg' height="auto" width="150" ></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144433937-daef5a69-fd70-428f-98a3-5e7747f4bb88.jpg' height="auto" width="150" ></div> |

| Cat      | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144429526-f27f4cce-89b9-4117-bfe6-55c2ca7eaba6.png' height="auto" width="165" ></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144434518-867ae32a-1cb5-4dbd-b1b9-5e375e94ea48.jpg' height="auto" width="150" ></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144434603-0a2fd9ec-c02e-4e6c-a17b-64c234808c56.jpg' height="auto" width="150" ></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144434623-b4432cc2-c663-4b97-aed3-583d9d3743e6.jpg' height="auto" width="150" ></div> |

3. Use `--eigen-smooth` and `--aug-smooth` to improve visual effects.

   ```shell
       demo/dog.jpg  \
       configs/mobilenet_v3/mobilenet-v3-large_8xb128_in1k.py \

```shell
       https://download.openmmlab.com/mmclassification/v0/mobilenet_v3/convert/mobilenet_v3_large-3ea3c186.pth \
       --target-layers 'backbone.layer16' \
       --method LayerCAM \
       --eigen-smooth --aug-smooth
   ```

   | Image                                | LayerCAM                                | eigen-smooth                                | aug-smooth                                | eigen&aug                                 |
   | ------------------------------------ | --------------------------------------- | ------------------------------------------- | ----------------------------------------- | ----------------------------------------- |

| <div align=center><img src='https://user-images.githubusercontent.com/18586273/144557492-98ac5ce0-61f9-4da9-8ea7-396d0b6a20fa.jpg' height="auto" width="160"></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144557541-a4cf7d86-7267-46f9-937c-6f657ea661b4.jpg'  height="auto" width="145" ></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144557547-2731b53e-e997-4dd2-a092-64739cc91959.jpg'  height="auto" width="145" ></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144557545-8189524a-eb92-4cce-bf6a-760cab4a8065.jpg'  height="auto" width="145" ></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144557548-c1e3f3ec-3c96-43d4-874a-3b33cd3351c5.jpg'  height="auto" width="145" ></div> |

## How to visualize the CAM of vision transformer

Here are some examples:

- `'backbone.norm3'` for Swin-Transformer;
- `'backbone.layers.11.ln1'` for ViT;

For ViT-like networks, such as ViT, T2T-ViT and Swin-Transformer, the features are flattened. And for drawing the CAM, we need to specify the `--vit-like` argument to reshape the features into square feature maps.

Besides the flattened features, some ViT-like networks also add extra tokens like the class token in ViT and T2T-ViT, and the distillation token in DeiT. In these networks, the final classification is done on the tokens computed in the last attention block, and therefore, the classification score will not be affected by other features and the gradient of the classification score with respect to them, will be zero. Therefore, you shouldn't use the output of the last attention block as the target layer in these networks.

To exclude these extra tokens, we need know the number of extra tokens. Almost all transformer-based backbones in MMPretrain have the `num_extra_tokens` attribute. If you want to use this tool in a new or third-party network that don't have the `num_extra_tokens` attribute, please specify it the `--num-extra-tokens` argument.

1. Visualize CAM for `Swin Transformer`, using default `target-layers`:

   ```shell
       demo/bird.JPEG  \
       configs/swin_transformer/swin-tiny_16xb64_in1k.py \
       https://download.openmmlab.com/mmclassification/v0/swin-transformer/swin_tiny_224_b16x64_300e_imagenet_20210616_090925-66df6be6.pth \
       --vit-like
   ```

2. Visualize CAM for `Vision Transformer(ViT)`:

   ```shell
       configs/vision_transformer/vit-base-p16_64xb64_in1k-384px.py \

   ```shell
       https://download.openmmlab.com/mmclassification/v0/vit/finetune/vit-base-p16_in21k-pre-3rdparty_ft-64xb64_in1k-384_20210928-98e8652b.pth \
       --vit-like \
       --target-layers 'backbone.layers.11.ln1'
   ```

3. Visualize CAM for `T2T-ViT`:

   ```shell
       configs/t2t_vit/t2t-vit-t-14_8xb64_in1k.py \
       https://download.openmmlab.com/mmclassification/v0/t2t-vit/t2t-vit-t-14_3rdparty_8xb64_in1k_20210928-b7c09b62.pth \
       --target-layers 'backbone.encoder.12.ln1'
   ```

https://download.openmmlab.com/mmclassification/v0/t2t-vit/t2t-vit-t-14_3rdparty_8xb64_in1k_20210928-b7c09b62.pth \
   ```

| Image                                   | ResNet50                                   | ViT                                    | Swin                                    | T2T-ViT                                    |
| --------------------------------------- | ------------------------------------------ | -------------------------------------- | --------------------------------------- | ------------------------------------------ |

| <div align=center><img src='https://user-images.githubusercontent.com/18586273/144429496-628d3fb3-1f6e-41ff-aa5c-1b08c60c32a9.JPEG' height="auto" width="165" ></div> | <div align=center><img src=https://user-images.githubusercontent.com/18586273/144431491-a2e19fe3-5c12-4404-b2af-a9552f5a95d9.jpg  height="auto" width="150" ></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144436218-245a11de-6234-4852-9c08-ff5069f6a739.jpg' height="auto" width="150" ></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144436168-01b0e565-442c-4e1e-910c-17c62cff7cd3.jpg' height="auto" width="150" ></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144436198-51dbfbda-c48d-48cc-ae06-1a923d19b6f6.jpg' height="auto" width="150" ></div> |

# iTPN

> [Integrally Pre-Trained Transformer Pyramid Networks](https://arxiv.org/abs/2211.12735)

In this paper, we present an integral pre-training framework based on masked image modeling (MIM). We advocate for pre-training the backbone and neck jointly so that the transfer gap between MIM and downstream recognition tasks is minimal. We make two technical contributions. First, we unify the reconstruction and recognition necks by inserting a feature pyramid into the pre-training stage. Second, we complement mask image modeling (MIM) with masked feature modeling (MFM) that offers multi-stage supervision to the feature pyramid. The pre-trained models, termed integrally pre-trained transformer pyramid networks (iTPNs), serve as powerful foundation models for visual recognition. In particular, the base/large-level iTPN achieves an 86.2%/87.8% top-1 accuracy on ImageNet-1K, a 53.2%/55.6% box AP on COCO object detection with 1x training schedule using Mask-RCNN, and a 54.7%/57.7% mIoU on ADE20K semantic segmentation using UPerHead -- all these results set new records. Our work inspires the community to work on unifying upstream pre-training and downstream fine-tuning tasks. Code and the pre-trained models will be released at https://github.com/sunsmarterjie/iTPN.

<img src="https://github.com/open-mmlab/mmpretrain/assets/36138628/2e53d5b5-300e-4640-8507-c1173965ca62" width="80%"/>
</div>

<!-- **Use the model**

```python

model = get_model('itpn-clip-b_hivit-base-p16_8xb256-amp-coslr-800e_in1k', pretrained=True)
``` -->

Train:

```shell
python tools/train.py configs/itpn/itpn-pixel_hivit-base-p16_8xb512-amp-coslr-800e_in1k.py
```

Train:

```shell
```

| Model                                                   | Params (M) | Flops (G) |                               Config                               | Download |
| :------------------------------------------------------ | :--------: | :-------: | :----------------------------------------------------------------: | :------: |
| `itpn-clip-b_hivit-base-p16_8xb256-amp-coslr-800e_in1k` |   233.00   |   18.47   | [config](itpn-clip-b_hivit-base-p16_8xb256-amp-coslr-800e_in1k.py) |   N/A    |
| `itpn-pixel_hivit-base-p16_8xb512-amp-coslr-800e_in1k`  |   103.00   |   18.47   | [config](itpn-pixel_hivit-base-p16_8xb512-amp-coslr-800e_in1k.py)  |   N/A    |

| `itpn-pixel_hivit-large-p16_8xb512-amp-coslr-800e_in1k` |   314.00   |   63.98   | [config](itpn-pixel_hivit-large-p16_8xb512-amp-coslr-800e_in1k.py) |   N/A    |

```bibtex
@article{tian2022integrally,
  title={Integrally Pre-Trained Transformer Pyramid Networks},
  author={Tian, Yunjie and Xie, Lingxi and Wang, Zhaozhi and Wei, Longhui and Zhang, Xiaopeng and Jiao, Jianbin and Wang, Yaowei and Tian, Qi and Ye, Qixiang},
  journal={arXiv preprint arXiv:2211.12735},
}
```

# MMPose for AIGC (AI Generated Content)

<img src="https://user-images.githubusercontent.com/13503330/222403836-c65ba905-4bdd-4a44-834c-ff8d5959649d.png" width=1000 />
</div>

English | [简体中文](./README_CN.md)

This project will demonstrate how to use MMPose to generate skeleton images for pose guided AI image generation.

Currently, we support:

- [T2I Adapter](https://huggingface.co/spaces/Adapter/T2I-Adapter)

Please feel free to share interesting pose-guided AIGC projects to us!

## Get Started

### Generate OpenPose-style Skeleton

#### Step 1: Preparation

Run the following commands to prepare the project:

```shell
# install mmpose mmdet
git clone https://github.com/open-mmlab/mmpose.git
cd mmpose
mim install "mmdet>=3.0.0rc6"

# download models
bash download_models.sh
```

#### Step 2: Generate a Skeleton Image

Run the following command to generate a skeleton image:

```shell
# generate a skeleton image

```

```shell
bash mmpose_openpose.sh ../../tests/data/coco/000000000785.jpg
```

The input image and its skeleton are as follows:

<img src="https://user-images.githubusercontent.com/13503330/226527894-f9d98e75-fc6c-49e5-ba39-d6277b03697a.jpg" width=450 /><img src='https://user-images.githubusercontent.com/13503330/226527555-58a86ee6-a886-4986-b1c3-b3841b112995.png' width=450/>
</div>

### Generate MMPose-style Skeleton

**Env Requirements:**

- GCC >= 7.5
- cmake >= 3.14

Run the following commands to install the project:

```shell
bash install_posetracker_linux.sh
```

After installation, files are organized as follows:

```shell
|----mmdeploy-1.0.0-linux-x86_64-cxx11abi
|    |----README.md
|    |----rtmpose-ort
|    |    |----rtmdet-nano
|    |    |----rtmpose-m

```shell
|    |    |----000000147979.jpg
|    |    |----t2i-adapter_skeleton.txt
```

```shell
bash mmpose_style_skeleton.sh \
    mmdeploy-1.0.0-linux-x86_64-cxx11abi/rtmpose-ort/000000147979.jpg
```

For more details, you can refer to [RTMPose](../rtmpose/README.md).

<img src="https://user-images.githubusercontent.com/13503330/222318807-a0a2a87a-cfc4-46cc-b647-e8f8bc44cfe3.jpg" width=300 /><img src='https://user-images.githubusercontent.com/13503330/222318943-6dba5f52-158a-427a-8222-03628addc051.jpg' width=300/>
</div>

### Upload to T2I-Adapter

</div>

The demo page of T2I- Adapter is [Here](https://huggingface.co/spaces/Adapter/T2I-Adapter).

[![Huggingface Gradio](https://img.shields.io/static/v1?label=Demo&message=Huggingface%20Gradio&color=orange)](https://huggingface.co/spaces/ChongMou/T2I-Adapter)

<img src="https://user-images.githubusercontent.com/13503330/226528354-aac50e26-9188-4b81-9692-274415ee5a87.png" width=900 />
</div>

## Gallery

<img src="https://user-images.githubusercontent.com/13503330/226529427-4a34f168-caa2-446b-8aa9-2a60c4082921.png" width=900 />
</div>

# CSN

[Video Classification With Channel-Separated Convolutional Networks](https://openaccess.thecvf.com/content_ICCV_2019/html/Tran_Video_Classification_With_Channel-Separated_Convolutional_Networks_ICCV_2019_paper.html)

Group convolution has been shown to offer great computational savings in various 2D convolutional architectures for image classification. It is natural to ask: 1) if group convolution can help to alleviate the high computational cost of video classification networks; 2) what factors matter the most in 3D group convolutional networks; and 3) what are good computation/accuracy trade-offs with 3D group convolutional networks. This paper studies the effects of different design choices in 3D group convolutional networks for video classification. We empirically demonstrate that the amount of channel interactions plays an important role in the accuracy of 3D group convolutional networks. Our experiments suggest two main findings. First, it is a good practice to factorize 3D convolutions by separating channel interactions and spatiotemporal interactions as this leads to improved accuracy and lower computational cost. Second, 3D channel-separated convolutions provide a form of regularization, yielding lower training accuracy but higher test accuracy compared to 3D convolutions. These two empirical findings lead us to design an architecture -- Channel-Separated Convolutional Network (CSN) -- which is simple, efficient, yet accurate. On Sports1M, Kinetics, and Something-Something, our CSNs are comparable with or better than the state-of-the-art while being 2-3 times more efficient.

<img src="https://user-images.githubusercontent.com/34324155/143017317-1bd7e557-7d99-4964-8b89-ab5280945d54.png" width="800"/>
</div>

| frame sampling strategy | resolution | gpus |        backbone         | pretrain | top1 acc | top5 acc | testing protocol  | FLOPs  | params |             config             |             ckpt              |             log              |
| :---------------------: | :--------: | :--: | :---------------------: | :------: | :------: | :------: | :---------------: | :----: | :----: | :----------------------------: | :---------------------------: | :--------------------------: |

|         32x2x1          |  224x224   |  8   |     ResNet152 (IR)      |  IG65M   |  82.87   |  95.90   | 10 clips x 3 crop | 97.63G | 29.70M | [config](/configs/recognition/csn/ircsn_ig65m-pretrained-r152_8xb12-32x2x1-58e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/csn/ircsn_ig65m-pretrained-r152_8xb12-32x2x1-58e_kinetics400-rgb/ircsn_ig65m-pretrained-r152_8xb12-32x2x1-58e_kinetics400-rgb_20220811-c7a3cc5b.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/csn/ircsn_ig65m-pretrained-r152_8xb12-32x2x1-58e_kinetics400-rgb/ircsn_ig65m-pretrained-r152_8xb12-32x2x1-58e_kinetics400-rgb.log) |

|         32x2x1          |  224x224   |  8   | ResNet152 (IR+BNFrozen) |  IG65M   |  82.84   |  95.92   | 10 clips x 3 crop | 97.63G | 29.70M | [config](/configs/recognition/csn/ircsn_ig65m-pretrained-r152-bnfrozen_8xb12-32x2x1-58e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/csn/ircsn_ig65m-pretrained-r152-bnfrozen_8xb12-32x2x1-58e_kinetics400-rgb/ircsn_ig65m-pretrained-r152-bnfrozen_8xb12-32x2x1-58e_kinetics400-rgb_20220811-7d1dacde.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/csn/ircsn_ig65m-pretrained-r152-bnfrozen_8xb12-32x2x1-58e_kinetics400-rgb/ircsn_ig65m-pretrained-r152-bnfrozen_8xb12-32x2x1-58e_kinetics400-rgb.log) |

|         32x2x1          |  224x224   |  8   | ResNet50 (IR+BNFrozen)  |  IG65M   |  79.44   |  94.26   | 10 clips x 3 crop | 55.90G | 13.13M | [config](/configs/recognition/csn/ircsn_ig65m-pretrained-r50-bnfrozen_8xb12-32x2x1-58e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/csn/ircsn_ig65m-pretrained-r50-bnfrozen_8xb12-32x2x1-58e_kinetics400-rgb/ircsn_ig65m-pretrained-r50-bnfrozen_8xb12-32x2x1-58e_kinetics400-rgb_20220811-44395bae.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/csn/ircsn_ig65m-pretrained-r50-bnfrozen_8xb12-32x2x1-58e_kinetics400-rgb/ircsn_ig65m-pretrained-r50-bnfrozen_8xb12-32x2x1-58e_kinetics400-rgb.log) |

|         32x2x1          |  224x224   |  x   |     ResNet152 (IP)      |   None   |  77.80   |  93.10   | 10 clips x 3 crop | 109.9G | 33.02M | [config](/configs/recognition/csn/ipcsn_r152_32x2x1-180e_kinetics400-rgb.py) | [infer_ckpt](https://download.openmmlab.com/mmaction/recognition/csn/vmz/vmz_ipcsn_from_scratch_r152_32x2x1_180e_kinetics400_rgb_20210617-d565828d.pth) |              x               |
|         32x2x1          |  224x224   |  x   |     ResNet152 (IR)      |   None   |  76.53   |  92.28   | 10 clips x 3 crop | 97.6G  | 29.70M | [config](/configs/recognition/csn/ircsn_r152_32x2x1-180e_kinetics400-rgb.py) | [infer_ckpt](https://download.openmmlab.com/mmaction/recognition/csn/vmz/vmz_ircsn_from_scratch_r152_32x2x1_180e_kinetics400_rgb_20210617-5c933ae1.pth) |              x               |

|         32x2x1          |  224x224   |  x   | ResNet152 (IP+BNFrozen) |  IG65M   |  82.68   |  95.69   | 10 clips x 3 crop | 109.9G | 33.02M | [config](/configs/recognition/csn/ipcsn_ig65m-pretrained-r152-bnfrozen_32x2x1-58e_kinetics400-rgb.py) | [infer_ckpt](https://download.openmmlab.com/mmaction/recognition/csn/vmz/vmz_ipcsn_ig65m_pretrained_r152_32x2x1_58e_kinetics400_rgb_20210617-c3be9793.pth) |              x               |
|         32x2x1          |  224x224   |  x   | ResNet152 (IP+BNFrozen) | Sports1M |  79.07   |  93.82   | 10 clips x 3 crop | 109.9G | 33.02M | [config](/configs/recognition/csn/ipcsn_sports1m-pretrained-r152-bnfrozen_32x2x1-58e_kinetics400-rgb.py) | [infer_ckpt](https://download.openmmlab.com/mmaction/recognition/csn/vmz/vmz_ipcsn_sports1m_pretrained_r152_32x2x1_58e_kinetics400_rgb_20210617-3367437a.pth) |              x               |

|         32x2x1          |  224x224   |  x   | ResNet152 (IR+BNFrozen) | Sports1M |  78.57   |  93.44   | 10 clips x 3 crop | 109.9G | 33.02M | [config](/configs/recognition/csn/ircsn_sports1m-pretrained-r152-bnfrozen_32x2x1-58e_kinetics400-rgb.py) | [infer_ckpt](https://download.openmmlab.com/mmaction/recognition/csn/vmz/vmz_ircsn_sports1m_pretrained_r152_32x2x1_58e_kinetics400_rgb_20210617-b9b10241.pth) |              x               |

1. The **gpus** indicates the number of gpus we used to get the checkpoint. If you want to use a different number of gpus or videos per gpu, the best way is to set `--auto-scale-lr` when calling `tools/train.py`, this parameter will auto-scale the learning rate according to the actual batch size and the original batch size.

3. The **infer_ckpt** means those checkpoints are ported from [VMZ](https://github.com/facebookresearch/VMZ).

## Train

```shell
```

## Train

```shell
```

Example: train CSN model on Kinetics-400 dataset in a deterministic option with periodic validation.

```shell
python tools/train.py configs/recognition/csn/ircsn_ig65m-pretrained-r152_8xb12-32x2x1-58e_kinetics400-rgb.py  \
    --seed=0 --deterministic
```

## Test

```shell
```

Example: test CSN model on Kinetics-400 dataset and dump the result to a pkl file.

```shell
python tools/test.py configs/recognition/csn/ircsn_ig65m-pretrained-r152_8xb12-32x2x1-58e_kinetics400-rgb.py \
```

```shell
```

```BibTeX
@inproceedings{inproceedings,
author = {Wang, Heng and Feiszli, Matt and Torresani, Lorenzo},
year = {2019},
month = {10},
pages = {5551-5560},
title = {Video Classification With Channel-Separated Convolutional Networks},
doi = {10.1109/ICCV.2019.00565}
}
```

```BibTeX
@inproceedings{ghadiyaram2019large,
  title={Large-scale weakly-supervised pre-training for video action recognition},
  author={Ghadiyaram, Deepti and Tran, Du and Mahajan, Dhruv},
  pages={12046--12055},
}
```

We appreciate all contributions to improve MMDeploy. Please refer to [CONTRIBUTING.md](https://github.com/open-mmlab/mmcv/blob/master/CONTRIBUTING.md) in MMCV for more details about the contributing guideline.

# MMPose Deployment

- [MMPose Deployment](#mmpose-deployment)
  - [Installation](#installation)
    - [Install mmpose](#install-mmpose)
    - [Install mmdeploy](#install-mmdeploy)
  - [Model specification](#model-specification)
    - [Backend model inference](#backend-model-inference)
    - [SDK model inference](#sdk-model-inference)

[MMPose](https://github.com/open-mmlab/mmpose/tree/main) aka `mmpose` is an open-source toolbox for pose estimation based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com/) project.

### Install mmpose

Please follow the [best practice](https://mmpose.readthedocs.io/en/latest/installation.html#best-practices) to install mmpose.

### Install mmdeploy

There are several methods to install mmdeploy, among which you can choose an appropriate one according to your target platform and device.

**Method I:** Install precompiled package

You can refer to [get_started](https://mmdeploy.readthedocs.io/en/latest/get_started.html#installation)

**Method II:** Build using scripts

If your target platform is **Ubuntu 18.04 or later version**, we encourage you to run
[scripts](../01-how-to-build/build_from_script.md). For example, the following commands install mmdeploy as well as inference engine - `ONNX Runtime`.

```shell
git clone --recursive -b main https://github.com/open-mmlab/mmdeploy.git
cd mmdeploy
python3 tools/scripts/build_ubuntu_x64_ort.py $(nproc)
export PYTHONPATH=$(pwd)/build/lib:$PYTHONPATH

```shell
export LD_LIBRARY_PATH=$(pwd)/../mmdeploy-dep/onnxruntime-linux-x64-1.8.1/lib/:$LD_LIBRARY_PATH
```

**Method III:** Build from source

If neither **I** nor **II** meets your requirements, [building mmdeploy from source](../01-how-to-build/build_from_source.md) is the last option.

You can use [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/tree/main/tools/deploy.py) to convert mmpose models to the specified backend models. Its detailed usage can be learned from [here](https://github.com/open-mmlab/mmdeploy/tree/main/docs/en/02-how-to-run/convert_model.md#usage).

The command below shows an example about converting `hrnet` model to onnx model that can be inferred by ONNX Runtime.

```shell
# download hrnet model from mmpose model zoo

```shell
mim download mmpose --config td-hm_hrnet-w32_8xb64-210e_coco-256x192 --dest .
# convert mmdet model to onnxruntime model with static shape
    configs/mmpose/pose-detection_onnxruntime_static.py \
    td-hm_hrnet-w32_8xb64-210e_coco-256x192.py \
    hrnet_w32_coco_256x192-c78dce93_20200708.pth \
    demo/resources/human-pose.jpg \
    --work-dir mmdeploy_models/mmpose/ort \
    --show
```

It is crucial to specify the correct deployment config during model conversion. We've already provided builtin deployment config [files](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmpose) of all supported backends for mmpose. The config filename pattern is:

```
pose-detection_{backend}-{precision}_{static | dynamic}_{shape}.py
```

```
```

- **{backend}:** inference backend, such as onnxruntime, tensorrt, pplnn, ncnn, openvino, coreml etc.
- **{precision}:** fp16, int8. When it's empty, it means fp32
- **{static | dynamic}:** static shape or dynamic shape
- **{shape}:** input shape or shape range of a model

Therefore, in the above example, you can also convert `hrnet` to other backend models by changing the deployment config file `pose-detection_onnxruntime_static.py` to [others](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmpose), e.g., converting to tensorrt model by `pose-detection_tensorrt_static-256x192.py`.

```{tip}
When converting mmpose models to tensorrt models, --device should be set to "cuda"
```

## Model specification

Before moving on to model inference chapter, let's know more about the converted model structure which is very important for model inference.

```

The converted model locates in the working directory like `mmdeploy_models/mmpose/ort` in the previous example. It includes:

```
mmdeploy_models/mmpose/ort
├── deploy.json
├── detail.json
├── end2end.onnx
└── pipeline.json
```

in which,

- **end2end.onnx**: backend model which can be inferred by ONNX Runtime
- \***.json**: the necessary information for mmdeploy SDK

The whole package **mmdeploy_models/mmpose/ort** is defined as **mmdeploy SDK model**, i.e., **mmdeploy SDK model** includes both backend model and inference meta information.

### Backend model inference

Take the previous converted `end2end.onnx` model as an example, you can use the following code to inference the model and visualize the results.

```python
from mmdeploy.apis.utils import build_task_processor

```python
from mmdeploy.utils import get_input_shape, load_config

deploy_cfg = 'configs/mmpose/pose-detection_onnxruntime_static.py'
model_cfg = 'td-hm_hrnet-w32_8xb64-210e_coco-256x192.py'
device = 'cpu'
backend_model = ['./mmdeploy_models/mmpose/ort/end2end.onnx']
image = './demo/resources/human-pose.jpg'

# read deploy_cfg and model_cfg
deploy_cfg, model_cfg = load_config(deploy_cfg, model_cfg)

# build task and backend model
task_processor = build_task_processor(model_cfg, deploy_cfg, device)
model = task_processor.build_backend_model(backend_model)

# process input image
input_shape = get_input_shape(deploy_cfg)
model_inputs, _ = task_processor.create_input(image, input_shape)

# do model inference
    result = model.test_step(model_inputs)

# visualize results
task_processor.visualize(
    image=image,
    model=model,
    result=result[0],
    window_name='visualize',

    output_file='output_pose.png')
```

### SDK model inference

TODO

| Model                                                                                                     | Task          | ONNX Runtime | TensorRT | ncnn | PPLNN | OpenVINO |
| :-------------------------------------------------------------------------------------------------------- | :------------ | :----------: | :------: | :--: | :---: | :------: |
| [HRNet](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#hrnet-cvpr-2019)          | PoseDetection |      Y       |    Y     |  Y   |   N   |    Y     |
| [MSPN](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#mspn-arxiv-2019)           | PoseDetection |      Y       |    Y     |  Y   |   N   |    Y     |

| [LiteHRNet](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#litehrnet-cvpr-2021)  | PoseDetection |      Y       |    Y     |  Y   |   N   |    Y     |
| [Hourglass](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/algorithms.html#hourglass-eccv-2016) | PoseDetection |      Y       |    Y     |  Y   |   N   |    Y     |
| [SimCC](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/algorithms.html#simcc-eccv-2022)         | PoseDetection |      Y       |    Y     |  Y   |   N   |    Y     |
| [RTMPose](https://github.com/open-mmlab/mmpose/tree/main/projects/rtmpose)                                | PoseDetection |      Y       |    Y     |  Y   |   N   |    Y     |

| [YoloX-Pose](https://github.com/open-mmlab/mmpose/tree/main/projects/yolox_pose)                          | PoseDetection |      Y       |    Y     |  N   |   N   |    Y     |

# Guided Anchoring

> [Region Proposal by Guided Anchoring](https://arxiv.org/abs/1901.03278)

Region anchors are the cornerstone of modern object detection techniques. State-of-the-art detectors mostly rely on a dense anchoring scheme, where anchors are sampled uniformly over the spatial domain with a predefined set of scales and aspect ratios. In this paper, we revisit this foundational stage. Our study shows that it can be done much more effectively and efficiently. Specifically, we present an alternative scheme, named Guided Anchoring, which leverages semantic features to guide the anchoring. The proposed method jointly predicts the locations where the center of objects of interest are likely to exist as well as the scales and aspect ratios at different locations. On top of predicted anchor shapes, we mitigate the feature inconsistency with a feature adaption module. We also study the use of high-quality proposals to improve detection performance. The anchoring scheme can be seamlessly integrated into proposal methods and detectors. With Guided Anchoring, we achieve 9.1% higher recall on MS COCO with 90% fewer anchors than the RPN baseline. We also adopt Guided Anchoring in Fast R-CNN, Faster R-CNN and RetinaNet, respectively improving the detection mAP by 2.2%, 2.7% and 1.2%.

<img src="https://user-images.githubusercontent.com/40661020/143891529-4c178948-c3fd-4543-ae6e-bb2aa3c8147e.png"/>
</div>

The results on COCO 2017 val is shown in the below table. (results on test-dev are usually slightly higher than val).

| Method |    Backbone     |  Style  | Lr schd | Mem (GB) | Inf time (fps) | AR 1000 |                    Config                    |                                                                                                                                                          Download                                                                                                                                                           |

| :----: | :-------------: | :-----: | :-----: | :------: | :------------: | :-----: | :------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| GA-RPN |    R-50-FPN     |  caffe  |   1x    |   5.3    |      15.8      |  68.4   | [config](./ga-rpn_r50-caffe_fpn_1x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_rpn_r50_caffe_fpn_1x_coco/ga_rpn_r50_caffe_fpn_1x_coco_20200531-899008a6.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_rpn_r50_caffe_fpn_1x_coco/ga_rpn_r50_caffe_fpn_1x_coco_20200531_011819.log.json)   |

| GA-RPN |    R-101-FPN    |  caffe  |   1x    |   7.3    |      13.0      |  69.5   | [config](./ga-rpn_r101-caffe_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_rpn_r101_caffe_fpn_1x_coco/ga_rpn_r101_caffe_fpn_1x_coco_20200531-ca9ba8fb.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_rpn_r101_caffe_fpn_1x_coco/ga_rpn_r101_caffe_fpn_1x_coco_20200531_011812.log.json) |
| GA-RPN | X-101-32x4d-FPN | pytorch |   1x    |   8.5    |      10.0      |  70.6   | [config](./ga-rpn_x101-32x4d_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_rpn_x101_32x4d_fpn_1x_coco/ga_rpn_x101_32x4d_fpn_1x_coco_20200220-c28d1b18.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_rpn_x101_32x4d_fpn_1x_coco/ga_rpn_x101_32x4d_fpn_1x_coco_20200220_221326.log.json) |

| GA-RPN | X-101-64x4d-FPN | pytorch |   1x    |   7.1    |      7.5       |  71.2   | [config](./ga-rpn_x101-64x4d_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_rpn_x101_64x4d_fpn_1x_coco/ga_rpn_x101_64x4d_fpn_1x_coco_20200225-3c6e1aa2.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_rpn_x101_64x4d_fpn_1x_coco/ga_rpn_x101_64x4d_fpn_1x_coco_20200225_152704.log.json) |

|     Method     |    Backbone     |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                        Config                        |                                                                                                                                                                           Download                                                                                                                                                                            |

| GA-Faster RCNN |    R-50-FPN     |  caffe  |   1x    |   5.5    |                |  39.6  | [config](./ga-faster-rcnn_r50-caffe_fpn_1x_coco.py)  |          [model](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_faster_r50_caffe_fpn_1x_coco/ga_faster_r50_caffe_fpn_1x_coco_20200702_000718-a11ccfe6.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_faster_r50_caffe_fpn_1x_coco/ga_faster_r50_caffe_fpn_1x_coco_20200702_000718.log.json)           |

| GA-Faster RCNN |    R-101-FPN    |  caffe  |   1x    |   7.5    |                |  41.5  | [config](./ga-faster-rcnn_r101-caffe_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_faster_r101_caffe_fpn_1x_coco/ga_faster_r101_caffe_fpn_1x_coco_bbox_mAP-0.415_20200505_115528-fb82e499.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_faster_r101_caffe_fpn_1x_coco/ga_faster_r101_caffe_fpn_1x_coco_20200505_115528.log.json) |

| GA-Faster RCNN | X-101-32x4d-FPN | pytorch |   1x    |   8.7    |      9.7       |  43.0  | [config](./ga-faster-rcnn_x101-32x4d_fpn_1x_coco.py) |            [model](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_faster_x101_32x4d_fpn_1x_coco/ga_faster_x101_32x4d_fpn_1x_coco_20200215-1ded9da3.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_faster_x101_32x4d_fpn_1x_coco/ga_faster_x101_32x4d_fpn_1x_coco_20200215_184547.log.json)            |

| GA-Faster RCNN | X-101-64x4d-FPN | pytorch |   1x    |   11.8   |      7.3       |  43.9  | [config](./ga-faster-rcnn_x101-64x4d_fpn_1x_coco.py) |            [model](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_faster_x101_64x4d_fpn_1x_coco/ga_faster_x101_64x4d_fpn_1x_coco_20200215-0fa7bde7.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_faster_x101_64x4d_fpn_1x_coco/ga_faster_x101_64x4d_fpn_1x_coco_20200215_104455.log.json)            |

|  GA-RetinaNet  |    R-50-FPN     |  caffe  |   1x    |   3.5    |      16.8      |  36.9  |  [config](./ga-retinanet_r50-caffe_fpn_1x_coco.py)   |        [model](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_retinanet_r50_caffe_fpn_1x_coco/ga_retinanet_r50_caffe_fpn_1x_coco_20201020-39581c6f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_retinanet_r50_caffe_fpn_1x_coco/ga_retinanet_r50_caffe_fpn_1x_coco_20201020_225450.log.json)        |

|  GA-RetinaNet  |    R-101-FPN    |  caffe  |   1x    |   5.5    |      12.9      |  39.0  |  [config](./ga-retinanet_r101-caffe_fpn_1x_coco.py)  |      [model](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_retinanet_r101_caffe_fpn_1x_coco/ga_retinanet_r101_caffe_fpn_1x_coco_20200531-6266453c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_retinanet_r101_caffe_fpn_1x_coco/ga_retinanet_r101_caffe_fpn_1x_coco_20200531_012847.log.json)      |

|  GA-RetinaNet  | X-101-32x4d-FPN | pytorch |   1x    |   6.9    |      10.6      |  40.5  |  [config](./ga-retinanet_x101-32x4d_fpn_1x_coco.py)  |      [model](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_retinanet_x101_32x4d_fpn_1x_coco/ga_retinanet_x101_32x4d_fpn_1x_coco_20200219-40c56caa.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_retinanet_x101_32x4d_fpn_1x_coco/ga_retinanet_x101_32x4d_fpn_1x_coco_20200219_223025.log.json)      |

|  GA-RetinaNet  | X-101-64x4d-FPN | pytorch |   1x    |   9.9    |      7.7       |  41.3  |  [config](./ga-retinanet_x101-64x4d_fpn_1x_coco.py)  |      [model](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_retinanet_x101_64x4d_fpn_1x_coco/ga_retinanet_x101_64x4d_fpn_1x_coco_20200226-ef9f7f1f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/guided_anchoring/ga_retinanet_x101_64x4d_fpn_1x_coco/ga_retinanet_x101_64x4d_fpn_1x_coco_20200226_221123.log.json)      |

- In the Guided Anchoring paper, `score_thr` is set to 0.001 in Fast/Faster RCNN and 0.05 in RetinaNet for both baselines and Guided Anchoring.

- Performance on COCO test-dev benchmark are shown as follows.

|     Method     | Backbone  | Style | Lr schd | Aug Train | Score thr | AP  | AP_50 | AP_75 | AP_small | AP_medium | AP_large | Download |
| :------------: | :-------: | :---: | :-----: | :-------: | :-------: | :-: | :---: | :---: | :------: | :-------: | :------: | :------: |

| GA-Faster RCNN | R-101-FPN | caffe |   1x    |     F     |   0.05    |     |       |       |          |           |          |          |
| GA-Faster RCNN | R-101-FPN | caffe |   1x    |     F     |   0.001   |     |       |       |          |           |          |          |
|  GA-RetinaNet  | R-101-FPN | caffe |   1x    |     F     |   0.05    |     |       |       |          |           |          |          |
|  GA-RetinaNet  | R-101-FPN | caffe |   2x    |     T     |   0.05    |     |       |       |          |           |          |          |

We provide config files to reproduce the results in the CVPR 2019 paper for [Region Proposal by Guided Anchoring](https://arxiv.org/abs/1901.03278).

```latex
@inproceedings{wang2019region,
    title={Region Proposal by Guided Anchoring},

```latex
    author={Jiaqi Wang and Kai Chen and Shuo Yang and Chen Change Loy and Dahua Lin},
    booktitle={IEEE Conference on Computer Vision and Pattern Recognition},
    year={2019}
}
```

# 配置文件

MMPose 使用 Python 文件作为配置文件，将模块化设计和继承设计结合到配置系统中，便于进行各种实验。

## 简介

MMPose 拥有一套强大的配置系统，在注册器的配合下，用户可以通过一个配置文件来定义整个项目需要用到的所有内容，以 Python 字典形式组织配置信息，传递给注册器完成对应模块的实例化。

下面是一个常见的 Pytorch 模块定义的例子：

```Python
# 在loss_a.py中定义Loss_A类
        return x

# 在需要的地方进行实例化
```

只需要通过一行代码对这个类进行注册：

```Python

@MODELS.register_module() # 注册该类到 MODELS 下
        return x
```

并在对应目录下的 `__init__.py` 中进行 `import`：

```Python

```

我们就可以通过如下方式来从配置文件定义并进行实例化：

```Python
# 在config_file.py中定义

```Python

```

```Python
    type='Loss_A', # 通过type指定类名
    param1=1.0,    # 传递__init__所需的参数
)

loss = MODELS.build(loss_cfg) # 等价于 loss = Loss_A(param1=1.0, param2=True)
```

MMPose 预定义的 Registry 在 `$MMPOSE/mmpose/registry.py` 中，目前支持的有：

- `DATASETS`：数据集

- `TRANSFORMS`：数据变换

- `MODELS`：模型模块（Backbone、Neck、Head、Loss等）

- `VISUALIZERS`：可视化工具

- `VISBACKENDS`：可视化后端

- `METRICS`：评测指标

- `KEYPOINT_CODECS`：编解码器

- `HOOKS`：钩子类

```{note}
需要注意的是，所有新增的模块都需要使用注册器（Registry）进行注册，并在对应目录的 `__init__.py` 中进行 `import`，以便能够使用配置文件构建其实例。
```

## 配置系统

具体而言，一个配置文件主要包含如下五个部分：

- 通用配置：与训练或测试无关的通用配置，如时间统计，模型存储与加载，可视化等相关 Hook，以及一些分布式相关的环境配置

- 数据配置：数据增强策略，Dataset和Dataloader相关配置

- 训练配置：断点恢复、模型权重加载、优化器、学习率调整、训练轮数和测试间隔等

- 模型配置：模型模块、参数、损失函数等

- 评测配置：模型性能评测指标

你可以在 `$MMPOSE/configs` 下找到我们提供的配置文件，配置文件之间通过继承来避免冗余。为了保持配置文件简洁易读，我们将一些必要但不常改动的配置存放到了 `$MMPOSE/configs/_base_` 目录下，如果希望查阅完整的配置信息，你可以运行如下指令：

```Bash
```

### 通用配置

通用配置指与训练或测试无关的必要配置，主要包括：

- **默认Hook**：迭代时间统计，训练日志，参数更新，checkpoint 等

- **环境配置**：分布式后端，cudnn，多进程配置等

- **可视化器**：可视化后端和策略设置

- **日志配置**：日志等级，格式，打印和记录间隔等

下面是通用配置的样例说明：

```Python
# 通用配置
    timer=dict(type='IterTimerHook'), # 迭代时间统计，包括数据耗时和模型耗时
    logger=dict(type='LoggerHook', interval=50), # 日志打印间隔
    param_scheduler=dict(type='ParamSchedulerHook'), # 用于调度学习率更新
        type='CheckpointHook', interval=1, save_best='coco/AP', # ckpt保存间隔，最优ckpt参考指标
        rule='greater'), # 最优ckpt指标评价规则

    sampler_seed=dict(type='DistSamplerSeedHook')) # 分布式随机种子设置
    cudnn_benchmark=False, # cudnn benchmark开关
    mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0), # opencv多线程配置
    dist_cfg=dict(backend='nccl')) # 分布式训练后端设置
vis_backends = [dict(type='LocalVisBackend')] # 可视化器后端设置
visualizer = dict( # 可视化器设置
log_processor = dict( # 训练日志格式、间隔
log_level = 'INFO' # 日志记录等级
```

通用配置一般单独存放到`$MMPOSE/configs/_base_`目录下，通过如下方式进行继承：

```Python
_base_ = ['../../../_base_/default_runtime.py'] # 以运行时的config文件位置为相对路径起点
```

```{note}

- save_best: `'coco/AP'` 用于 `CocoMetric`, `'PCK'` 用于 `PCKAccuracy`
- max_keep_ckpts: 最大保留ckpt数量，默认为-1，代表不限制

```

```{note}

样例:

```

### 数据配置

数据配置指数据处理相关的配置，主要包括：

- **数据后端**：数据供给后端设置，默认为本地硬盘，我们也支持从 LMDB，S3 Bucket 等加载

- **数据集**：图像与标注文件路径

- **加载**：加载策略，批量大小等

- **流水线**：数据增强策略

- **编码器**：根据标注生成特定格式的监督信息

下面是数据配置的样例说明：

```Python
backend_args = dict(backend='local') # 数据加载后端设置，默认从本地硬盘加载
dataset_type = 'CocoDataset' # 数据集类名
data_mode = 'topdown' # 算法结构类型，用于指定标注信息加载策略
data_root = 'data/coco/' # 数据存放路径
 # 定义数据编解码器，用于生成target和对pred进行解码，同时包含了输入图片和输出heatmap尺寸等信息
train_pipeline = [ # 训练时数据增强
    dict(type='LoadImage', backend_args=backend_args, # 加载图片
    dict(type='GetBBoxCenterScale'), # 根据bbox获取center和scale
    dict(type='RandomBBoxTransform'), # 生成随机位移、缩放、旋转变换矩阵

dict(type='LoadImage', backend_args=backend_args, # 加载图片
    dict(type='RandomFlip', direction='horizontal'), # 生成随机翻转变换矩阵
    dict(type='RandomHalfBody'), # 随机半身增强
    dict(type='TopdownAffine', input_size=codec['input_size']), # 根据变换矩阵更新目标数据
    dict(
        type='GenerateTarget', # 根据目标数据生成监督信息
        # 监督信息类型
        encoder=codec, # 传入编解码器，用于数据编码，生成特定格式的监督信息
    dict(type='PackPoseInputs') # 对target进行打包用于训练
]
test_pipeline = [ # 测试时数据增强
    dict(type='LoadImage', backend_args=backend_args), # 加载图片
]
train_dataloader = dict( # 训练数据加载
    batch_size=64, # 批次大小
    num_workers=2, # 数据加载进程数
    persistent_workers=True, # 在不活跃时维持进程不终止，避免反复启动进程的开销

dict(type='PackPoseInputs') # 对target进行打包用于训练
]
    sampler=dict(type='DefaultSampler', shuffle=True), # 采样策略，打乱数据
        type=dataset_type , # 数据集类名
        data_root=data_root, # 数据集路径
        data_mode=data_mode, # 算法类型
        ann_file='annotations/person_keypoints_train2017.json', # 标注文件路径
        data_prefix=dict(img='train2017/'), # 图像路径
        pipeline=train_pipeline # 数据流水线
    ))
    batch_size=32,
    sampler=dict(type='DefaultSampler', shuffle=False), # 采样策略，不进行打乱
        ann_file='annotations/person_keypoints_val2017.json', # 标注文件路径

type=dataset_type , # 数据集类名
        'data/coco/person_detection_results/COCO_val2017_detections_AP_H_56_person.json', # 检测框标注文件，topdown方法专用
        data_prefix=dict(img='val2017/'), # 图像路径
        test_mode=True, # 测试模式开关
        pipeline=test_pipeline # 数据流水线
    ))
test_dataloader = val_dataloader # 默认情况下不区分验证集和测试集，用户根据需要来自行定义
```

```{note}

常用功能可以参考以下教程:
- [恢复训练](../common_usages/resume_training.md)
- [自动混合精度训练](../common_usages/amp_training.md)
- [设置随机种子](../common_usages/set_random_seed.md)

```

### 训练配置

训练配置指训练策略相关的配置，主要包括：

- 从断点恢复训练

- 模型权重加载

- 训练轮数和测试间隔

- 学习率调整策略，如 warmup，scheduler

- 优化器和学习率

- 高级训练策略设置，如自动学习率缩放

下面是训练配置的样例说明：

```Python
resume = False # 断点恢复
load_from = None # 模型权重加载
train_cfg = dict(by_epoch=True, max_epochs=210, val_interval=10) # 训练轮数，测试间隔
    dict( # warmup策略

```Python
        begin=0,
        end=210,
]
optim_wrapper = dict(optimizer=dict(type='Adam', lr=0.0005)) # 优化器和学习率
auto_scale_lr = dict(base_batch_size=512) # 根据batch_size自动缩放学习率
```

### 模型配置

模型配置指模型训练和推理相关的配置，主要包括：

- 模型结构

- 损失函数

- 数据解码策略

- 测试时增强策略

下面是模型配置的样例说明，定义了一个基于 HRNetw32 的 Top-down Heatmap-based 模型：

```Python
# 定义数据编解码器，如果在数据配置部分已经定义过则无需重复定义
# 模型配置
    type='TopdownPoseEstimator', # 模型结构决定了算法流程
    data_preprocessor=dict( # 数据归一化和通道顺序调整，作为模型的一部分

# 模型配置
    backbone=dict( # 骨干网络定义

            type='Pretrained', # 预训练参数，只加载backbone权重用于迁移学习
    ),
    head=dict( # 模型头部
        loss=dict(type='KeypointMSELoss', use_target_weight=True), # 损失函数
        decoder=codec), # 解码器，将heatmap解码成坐标值
        flip_test=True, # 开启测试时水平翻转集成
        flip_mode='heatmap', # 对heatmap进行翻转
        shift_heatmap=True,  # 对翻转后的结果进行平移提高精度
    ))
```

### 评测配置

评测配置指公开数据集中关键点检测任务常用的评测指标，主要包括：

flip_test=True, # 开启测试时水平翻转集成
    ))
```

### 评测配置

- AUC

- EPE

- NME

下面是评测配置的样例说明，定义了一个COCO指标评测器：

```Python
    type='CocoMetric', # coco 评测指标
    ann_file=data_root + 'annotations/person_keypoints_val2017.json') # 加载评测标注数据
test_evaluator = val_evaluator # 默认情况下不区分验证集和测试集，用户根据需要来自行定义
```

## 配置文件命名规则

MMPose 配置文件命名风格如下：

```Python
{{算法信息}}_{{模块信息}}_{{训练信息}}_{{数据信息}}.py
```

文件名总体分为四部分：算法信息，模块信息，训练信息和数据信息。不同部分的单词之间用下划线 `'_'` 连接，同一部分有多个单词用短横线 `'-'` 连接。

- **算法信息**：算法名称，如 `topdown-heatmap`，`topdown-rle` 等

- **模块信息**：按照数据流的顺序列举一些中间的模块，其内容依赖于算法任务，如 `res101`，`hrnet-w48`等

- **训练信息**：训练策略的一些设置，包括 `batch size`，`schedule` 等，如 `8xb64-210e`

- **数据信息**：数据集名称、模态、输入尺寸等，如 `ap10k-256x256`，`zebra-160x160` 等

有时为了避免文件名过长，会省略模型信息中一些强相关的模块，只保留关键信息，如RLE-based算法中的`GAP`，Heatmap-based算法中的 `deconv` 等。

如果你希望向MMPose添加新的方法，你的配置文件同样需要遵守该命名规则。

## 常见用法

### 配置文件的继承

该用法常用于隐藏一些必要但不需要修改的配置，以提高配置文件的可读性。假如有如下两个配置文件：

```Python
```

```Python
```

虽然我们在 `resnet50.py` 中没有定义 optimizer 字段，但由于我们写了 `_base_ = ['optimizer_cfg.py']`，会使这个配置文件获得 `optimizer_cfg.py` 中的所有字段：

```Python
```

### 继承字段的修改

对于继承过来的已经定义好的字典，可以直接指定对应字段进行修改，而不需要重新定义完整的字典：

```Python
optimizer = dict(lr=0.01) # 直接修改对应字段
```

这个配置文件只修改了对应字段`lr`的信息：

```Python

```Python
```

```Python
```

### 删除字典中的字段

如果不仅是需要修改某些字段，还需要删除已定义的一些字段，需要在重新定义这个字典时指定`_delete_=True`，表示将没有在新定义中出现的字段全部删除：

```Python
optimizer = dict(_delete_=True, type='SGD', lr=0.01) # 重新定义字典
```

此时字典中除了 `type` 和 `lr` 以外的内容（`momentum`和`weight_decay`）将被全部删除：

```Python
```

```{note}
如果你希望更深入地了解配置系统的高级用法，可以查看 [MMEngine 教程](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/config.html)。
```

# BEiTv2

> [BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers](https://arxiv.org/abs/2208.06366)

Masked image modeling (MIM) has demonstrated impressive results in self-supervised representation learning by recovering corrupted image patches. However, most existing studies operate on low-level image pixels, which hinders the exploitation of high-level semantics for representation models. In this work, we propose to use a semantic-rich visual tokenizer as the reconstruction target for masked prediction, providing a systematic way to promote MIM from pixel-level to semantic-level. Specifically, we propose vector-quantized knowledge distillation to train the tokenizer, which discretizes a continuous semantic space to compact codes. We then pretrain vision Transformers by predicting the original visual tokens for the masked image patches. Furthermore, we introduce a patch aggregation strategy which associates discrete image patches to enhance global semantic representation. Experiments on image classification and semantic segmentation show that BEiT v2 outperforms all compared MIM methods. On ImageNet-1K (224 size), the base-size BEiT v2 achieves 85.5% top-1 accuracy for fine-tuning and 80.1% top-1 accuracy for linear probing. The large-size BEiT v2 obtains 87.3% top-1 accuracy for ImageNet-1K (224 size) fine-tuning, and 56.7% mIoU on ADE20K for semantic segmentation.

<img src="https://user-images.githubusercontent.com/36138628/203912182-5967a520-d455-49ea-bc67-dcbd500d76bf.png" width="70%"/>
</div>

```python

predict = inference_model('beit-base-p16_beitv2-pre_8xb128-coslr-100e_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('beitv2_beit-base-p16_8xb256-amp-coslr-300e_in1k', pretrained=True)
```

Train:

```shell

```

Train:

```shell
python tools/train.py configs/beitv2/beitv2_beit-base-p16_8xb256-amp-coslr-300e_in1k.py
```

Test:

```shell
python tools/test.py configs/beitv2/benchmarks/beit-base-p16_8xb128-coslr-100e_in1k.py https://download.openmmlab.com/mmselfsup/1.x/beitv2/beitv2_vit-base-p16_8xb256-amp-coslr-300e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k_20221212-d1c0789e.pth
```

| Model                                             | Params (M) | Flops (G) |                            Config                            |                                 Download                                 |

| :------------------------------------------------ | :--------: | :-------: | :----------------------------------------------------------: | :----------------------------------------------------------------------: |
| `beitv2_beit-base-p16_8xb256-amp-coslr-300e_in1k` |   192.81   |   17.58   | [config](beitv2_beit-base-p16_8xb256-amp-coslr-300e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/beitv2/beitv2_vit-base-p16_8xb256-amp-coslr-300e_in1k/beitv2_vit-base-p16_8xb256-amp-coslr-300e_in1k_20221212-a157be30.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/beitv2/beitv2_vit-base-p16_8xb256-amp-coslr-300e_in1k/beitv2_vit-base-p16_8xb256-amp-coslr-300e_in1k_20221212-a157be30.json) |

| Model                                   |                  Pretrain                  | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                  Config                  |                  Download                  |

| :-------------------------------------- | :----------------------------------------: | :--------: | :-------: | :-------: | :-------: | :--------------------------------------: | :----------------------------------------: |
| `beit-base-p16_beitv2-pre_8xb128-coslr-100e_in1k` | [BEITV2](https://download.openmmlab.com/mmselfsup/1.x/beitv2/beitv2_vit-base-p16_8xb256-amp-coslr-300e_in1k/beitv2_vit-base-p16_8xb256-amp-coslr-300e_in1k_20221212-a157be30.pth) |   86.53    |   17.58   |   85.00   |    N/A    | [config](benchmarks/beit-base-p16_8xb128-coslr-100e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/beitv2/beitv2_vit-base-p16_8xb256-amp-coslr-300e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k_20221212-d1c0789e.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/beitv2/beitv2_vit-base-p16_8xb256-amp-coslr-300e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k_20221212-d1c0789e.json) |

| `beit-base-p16_beitv2-in21k-pre_3rdparty_in1k`\* |            BEITV2 ImageNet-21k             |   86.53    |   17.58   |   86.47   |   97.99   | [config](benchmarks/beit-base-p16_8xb64_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/beit/beitv2-base_3rdparty_in1k_20221114-73e11905.pth) |

*Models with * are converted from the [official repo](https://github.com/microsoft/unilm/tree/master/beit2). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@article{beitv2,
    title={{BEiT v2}: Masked Image Modeling with Vector-Quantized Visual Tokenizers},
    author={Zhiliang Peng and Li Dong and Hangbo Bao and Qixiang Ye and Furu Wei},
    year={2022},
    eprint={2208.06366},
}
```

For some reason, if you're not happy with the binary size of the ncnn library, then here is the cheatsheet that helps you to build a minimal ncnn :P

### disable c++ rtti and exceptions

```
cmake -DNCNN_DISABLE_RTTI=ON -DNCNN_DISABLE_EXCEPTION=ON ..
```
* Cannot use RTTI and Exceptions when ncnn functions are called.

### disable vulkan support

```
cmake -DNCNN_VULKAN=OFF ..
```

* Cannot use GPU acceleration.

### disable NCNN_STDIO

```
cmake -DNCNN_STDIO=OFF ..
```

* Cannot load model from files, but can load model from memory or by Android Assets.

    Read more [here](https://github.com/Tencent/ncnn/blob/master/docs/how-to-use-and-FAQ/use-ncnn-with-alexnet.md#load-model).

### disable NCNN_STRING

```
cmake -DNCNN_STRING=OFF ..
```

* Cannot load human-readable param files with visible strings, but can load binary param.bin files.

    Read more [here](https://github.com/Tencent/ncnn/blob/master/docs/how-to-use-and-FAQ/use-ncnn-with-alexnet.md#strip-visible-string)

Read more [here](https://github.com/Tencent/ncnn/blob/master/docs/how-to-use-and-FAQ/use-ncnn-with-alexnet.md#strip-visible-string)

* Cannot identify blobs by string name when calling `Extractor::input / extract`, but can identify them by enum value in `id.h`.

    Read more [here](https://github.com/Tencent/ncnn/blob/master/docs/how-to-use-and-FAQ/use-ncnn-with-alexnet.md#input-and-output).

### disable NCNN_BF16

```
cmake -DNCNN_BF16=OFF ..
```

* Cannot use bf16 storage type in inference.

### disable NCNN_INT8

```
cmake -DNCNN_INT8=OFF ..
```

* Cannot use quantized int8 inference.

### drop pixel drawing functions

```
cmake -DNCNN_PIXEL_DRAWING=OFF ..
```

* Cannot use functions doing drawing basic shape and text like `ncnn::draw_rectangle_xx / ncnn::draw_circle_xx / ncnn::draw_text_xx`, but functions like `Mat::from_pixels / from_pixels_resize` are still available.

### drop pixel rotate and affine functions

```

```
cmake -DNCNN_PIXEL_ROTATE=OFF -DNCNN_PIXEL_AFFINE=OFF ..
```

* Cannot use functions doing rotatation and affine transformation like `ncnn::kanna_rotate_xx / ncnn::warpaffine_bilinear_xx`, but functions like `Mat::from_pixels / from_pixels_resize` are still available. 

### drop pixel functions

```
cmake -DNCNN_PIXEL=OFF ..
```

* Cannot use functions transferring from image to pixels like `Mat::from_pixels / from_pixels_resize / to_pixels / to_pixels_resize`, and need create a Mat and fill in data by hand.

### disable openmp

```
cmake -DNCNN_OPENMP=OFF ..
```

* Cannot use openmp multi-threading acceleration. If you want to run a model in single thread on your target machine, it is recommended to close the option.

### disable avx2 and arm82 optimized kernel

```
cmake -DNCNN_AVX2=OFF -DNCNN_ARM82=OFF ..
```

```
```

* Do not compile optimized kernels using avx2 / arm82 instruction set extensions. If your target machine does not support some of them, it is recommended to close the related options.

### disable runtime cpu instruction dispatch

```
cmake -DNCNN_RUNTIME_CPU=OFF ..
```

* Cannot check supported cpu instruction set extensions and use related optimized kernels in runtime.
* If you know which instruction set extensions are supported on your target machine like avx2 / arm82, you can open related options like `-DNCNN_AVX2=ON / -DNCNN_ARM82=ON` by hand and then sse2 / arm8 version kernels will not be compiled.

### drop layers not used

```
cmake -DWITH_LAYER_absval=OFF -DWITH_LAYER_bnll=OFF ..
```

* If your model does not include some layers, taking absval / bnll as a example above, you can drop them.

```
```

* Some key or dependency layers should not be dropped, like convolution / innerproduct, their dependency like padding / flatten, and activation like relu / clip.

### disable c++ stl

```
cmake -DNCNN_SIMPLESTL=ON ..
```

* STL provided by compiler is no longer depended on, and use `simplestl` provided by ncnn as a replacement. Users also can only use `simplestl` when ncnn functions are called.
* Usually with compiler parameters `-nodefaultlibs -fno-builtin -nostdinc++ -lc`
* Need cmake parameters `cmake -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake -DANDROID_STL=system` to avoid STL conflict when compiling to Android.

### drop optimized kernel not used

* Modify the source code under `ncnn/src/layer/arm/` to delete unnecessary optimized kernels or replace them with empty functions.
* You can also drop layers and related optimized kernels by `-DWITH_LAYER_absval=OFF` as mentioned above.

### drop operators from BinaryOp UnaryOp

* Modify `ncnn/src/layer/binaryop.cpp unaryop.cpp` and `ncnn/src/layer/arm/binaryop.cpp unaryop_arm.cpp` by hand to delete unnecessary operators.

# MMRotate Deployment

- [MMRotate Deployment](#mmrotate-deployment)
    - [Install mmrotate](#install-mmrotate)

[MMRotate](https://github.com/open-mmlab/mmrotate) is an open-source toolbox for rotated object detection based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com/) project.

### Install mmrotate

Please follow the [installation guide](https://mmrotate.readthedocs.io/en/1.x/get_started.html) to install mmrotate.

```shell

```shell
```

**NOTE**:

- Adding `$(pwd)/build/lib` to `PYTHONPATH` is for importing mmdeploy SDK python module - `mmdeploy_runtime`, which will be presented in chapter [SDK model inference](#sdk-model-inference).
- When [inference onnx model by ONNX Runtime](#backend-model-inference), it requests ONNX Runtime library be found. Thus, we add it to `LD_LIBRARY_PATH`.

You can use [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/blob/main/tools/deploy.py) to convert mmrotate models to the specified backend models. Its detailed usage can be learned from [here](https://github.com/open-mmlab/mmdeploy/blob/main/docs/en/02-how-to-run/convert_model.md#usage).

The command below shows an example about converting `rotated-faster-rcnn` model to onnx model that can be inferred by ONNX Runtime.

```shell

# download rotated-faster-rcnn model from mmrotate model zoo
mim download mmrotate --config rotated-faster-rcnn-le90_r50_fpn_1x_dota --dest .
wget https://github.com/open-mmlab/mmrotate/raw/main/demo/dota_demo.jpg

# convert mmrotate model to onnxruntime model with dynamic shape

    configs/mmrotate/rotated-detection_onnxruntime_dynamic.py \
    rotated-faster-rcnn-le90_r50_fpn_1x_dota.py \
    rotated_faster_rcnn_r50_fpn_1x_dota_le90-0393aa5c.pth \
    dota_demo.jpg \
    --work-dir mmdeploy_models/mmrotate/ort \
    --show \
```

It is crucial to specify the correct deployment config during model conversion. We've already provided builtin deployment config [files](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmrotate) of all supported backends for mmrotate. The config filename pattern is:

```
rotated_detection-{backend}-{precision}_{static | dynamic}_{shape}.py
```

```

Therefore, in the above example, you can also convert `rotated-faster-rcnn` to other backend models by changing the deployment config file `rotated-detection_onnxruntime_dynamic` to [others](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmrotate), e.g., converting to tensorrt-fp16 model by `rotated-detection_tensorrt-fp16_dynamic-320x320-1024x1024.py`.

```{tip}
When converting mmrotate models to tensorrt models, --device should be set to "cuda"
```

```

The converted model locates in the working directory like `mmdeploy_models/mmrotate/ort` in the previous example. It includes:

```
mmdeploy_models/mmrotate/ort
```

in which,

The whole package **mmdeploy_models/mmrotate/ort** is defined as **mmdeploy SDK model**, i.e., **mmdeploy SDK model** includes both backend model and inference meta information.

```python

```python

deploy_cfg = 'configs/mmrotate/rotated-detection_onnxruntime_dynamic.py'
model_cfg = './rotated-faster-rcnn-le90_r50_fpn_1x_dota.py'
backend_model = ['./mmdeploy_models/mmrotate/ort/end2end.onnx']
image = './dota_demo.jpg'

    output_file='./output.png')
```

You can also perform SDK model inference like following,

```python
from mmdeploy_runtime import RotatedDetector
import cv2

img = cv2.imread('./dota_demo.jpg')

# create a detector
detector = RotatedDetector(model_path='./mmdeploy_models/mmrotate/ort', device_name='cpu', device_id=0)
# perform inference
det = detector(img)
```

Besides python API, mmdeploy SDK also provides other FFI (Foreign Function Interface), such as C, C++, C#, Java and so on. You can learn their usage from [demos](https://github.com/open-mmlab/mmdeploy/tree/main/demo).

| Model                                                                                             | OnnxRuntime | TensorRT |

| :------------------------------------------------------------------------------------------------ | :---------: | :------: |
| [Rotated RetinaNet](https://github.com/open-mmlab/mmrotate/blob/1.x/configs/rotated_retinanet)    |      Y      |    Y     |
| [Rotated FasterRCNN](https://github.com/open-mmlab/mmrotate/blob/1.x/configs/rotated_faster_rcnn) |      Y      |    Y     |
| [Oriented R-CNN](https://github.com/open-mmlab/mmrotate/blob/1.x/configs/oriented_rcnn)           |      Y      |    Y     |
| [Gliding Vertex](https://github.com/open-mmlab/mmrotate/blob/1.x/configs/gliding_vertex)          |      Y      |    Y     |
| [RTMDET-R](https://github.com/open-mmlab/mmrotate/blob/1.x/configs/rotated_rtmdet)                |      Y      |    Y     |

# ISIC-2017 Task1

This project support **`ISIC-2017 Task1 `**, and the dataset used in this project can be downloaded from [here](https://challenge.isic-archive.com/data/#2017).

The goal of the challenge is to help participants develop image analysis tools to enable the automated diagnosis of melanoma from dermoscopic images.

This challenge provides training data (~2000 images) for participants to engage in all 3 components of lesion image analysis. A separate public validation dataset (~150 images) and blind held-out test dataset (~600 images) will be provided for participants to generate and submit automated results.

| Dataset name                                                     | Anatomical region | Task type    | Modality   | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License                                                                |

| ---------------------------------------------------------------- | ----------------- | ------------ | ---------- | ------------ | --------------------- | ---------------------- | ------------ | ---------------------------------------------------------------------- |
| [ISIC-2017 Task1](https://challenge.isic-archive.com/data/#2017) | full body         | segmentation | dermoscopy | 2            | 2000/150/600          | yes/yes/yes            | 2017         | [CC-0](https://creativecommons.org/share-your-work/public-domain/cc0/) |

| Class Name  | Num. Train | Pct. Train | Num. Val | Pct. Val | Num. Test | Pct. Test |
| :---------: | :--------: | :--------: | :------: | :------: | :-------: | :-------: |
|   normal    |    2000    |   82.86    |   150    |  73.88   |    600    |   70.62   |
| skin lesion |    2000    |   17.14    |   150    |  26.12   |    600    |   29.38   |

Note:

Note:

![bac](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/dermoscopy/isic2017_task1/isic2017_task1.png)

- PyTorch 1.10.0
- pillow(PIL) 9.3.0
- scikit-learn(sklearn) 1.2.0

All the commands below rely on the correct configuration of PYTHONPATH, which should point to the project's directory so that Python can locate the module files. In isic2017_task1/ root directory, run the following line to add the current directory to PYTHONPATH:

```shell
```

```shell
```

- download dataset from [here](https://challenge.isic-archive.com/data/#2017) and decompression data to path 'data/'.
- run script `"python ../../tools/split_seg_dataset.py"` to split dataset and generate `train.txt` and `test.txt`. If the label of official validation set and test set can't be obtained, we generate `train.txt` and `val.txt` from the training set randomly.

```none
  ├── mmseg
  │   │   │   ├── dermoscopy
  │   │   │   │   ├── isic2017_task1
  │   │   │   │   │   │   ├── test.txt

│   │   │   │   │   ├── data
  │   │   │   │   |   │   │   │   └── xxx.png
  │   │   │   │   │   │   │   ├── val
  │   │   │   │   |   │   │   │   └── yyy.png

│   │   │   │   |   │   │   │   ├── ...
```

```shell
```

To train on multiple GPUs, e.g. 8 GPUs, run the following command:

```shell
mim train mmseg ./configs/${CONFIG_PATH}  --launcher pytorch --gpus 8
```

```shell
```

```

## Results

### ISIC-2017 Task1

|     Method      | Backbone | Crop Size |   lr   | mIoU | mDice |                                                                                             config                                                                                              |
| :-------------: | :------: | :-------: | :----: | :--: | :---: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

| fcn_unet_s5-d16 |   unet   |  512x512  |  0.01  |  -   |   -   |  [config](https://github.com/open-mmlab/mmsegmentation/tree/dev-1.x/projects/medical/2d_image/dermoscopy/isic2017_task1/configs/fcn-unet-s5-d16_unet_1xb16-0.01-20k_isic2017-task1-512x512.py)  |
| fcn_unet_s5-d16 |   unet   |  512x512  | 0.001  |  -   |   -   | [config](https://github.com/open-mmlab/mmsegmentation/tree/dev-1.x/projects/medical/2d_image/dermoscopy/isic2017_task1/configs/fcn-unet-s5-d16_unet_1xb16-0.001-20k_isic2017-task1-512x512.py)  |
| fcn_unet_s5-d16 |   unet   |  512x512  | 0.0001 |  -   |   -   | [config](https://github.com/open-mmlab/mmsegmentation/tree/dev-1.x/projects/medical/2d_image/dermoscopy/isic2017_task1/configs/fcn-unet-s5-d16_unet_1xb16-0.0001-20k_isic2017-task1-512x512.py) |

# 准备 Diving48

## 简介

```BibTeX
}
```

用户可参考该数据集的 [官网](http://www.svcl.ucsd.edu/projects/resound/dataset.html)，以获取数据集相关的基本信息。

````{group-tab} 使用 MIM 下载
# MIM 支持下载 Diving48 数据集。用户可以通过一行命令，从 OpenDataLab 进行下载，并进行预处理。
```Bash
# 安装 OpenDataLab CLI 工具
# 登录 OpenDataLab
odl login
# 通过 MIM 进行数据集下载，预处理。注意这将花费较长时间
```

````

````{group-tab} 从官方源下载
## 步骤 1. 下载标注文件

用户可以使用以下命令下载标注文件（考虑到标注的准确性，这里仅下载 V2 版本）。在数据集准备前，请确保命令行当前路径为 `$MMACTION2/tools/data/diving48/`。

```shell
```

## 步骤 2. 准备视频

用户可以使用以下命令下载视频。

```shell
```

## Step 3. 抽取 RGB 帧和光流

如果用户只想使用视频加载训练，则该部分是 **可选项**。

```shell
```

```shell
```

官网提供的帧压缩包并不完整。若想获取完整的数据，可以使用以下步骤解帧。

在抽取视频帧和光流之前，请参考 [安装指南](/docs/zh_cn/get_started/installation.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。

如果拥有大量的 SSD 存储空间，则推荐将抽取的帧存储至 I/O 性能更优秀的 SSD 中。

可以运行以下命令为 SSD 建立软链接。

```shell
# 执行这两行进行抽取（假设 SSD 挂载在 "/mnt/SSD/"）
```

如果用户需要抽取 RGB 帧（因为抽取光流的过程十分耗时），可以考虑运行以下命令使用 denseflow **只抽取 RGB 帧**。

```shell
```

如果用户没有安装 denseflow，则可以运行以下命令使用 OpenCV 抽取 RGB 帧。然而，该方法只能抽取与原始视频分辨率相同的帧。

```shell
```

如果用户想抽取 RGB 帧和光流，则可以运行以下脚本进行抽取。

```shell
```

## 步骤 4. 生成文件列表

用户可以通过运行以下命令生成帧和视频格式的文件列表。

```

```shell
```

```shell
```

````
`````

### 检查文件夹结构

在完成所有 Diving48 数据集准备流程后，
用户可以获得对应的 RGB + 光流文件，视频文件以及标注文件。

在整个 MMAction2 文件夹下，Diving48 的文件结构如下：

```
mmaction2
├── tools
├── data
│   │   ├── annotations（可选）
│   |   |   ├── ...
│   |   ├── rawframes（可选）

│   |   |   ├── ...
│   |   |   ├── ...
```

关于对 Diving48 进行训练和验证，请参考 [训练和测试教程](/docs/en/user_guides/train_test.md)。

# 模型微调

本教程提供了使用预训练模型在其他数据集上进行微调的指导。通过微调，可以获得更好的性能。

- [模型微调](#模型微调)
  - [选择模板配置](#选择模板配置)
  - [修改 Head](#修改-head)
  - [修改数据集](#修改数据集)
  - [修改训练计划](#修改训练计划)
  - [使用预训练模型](#使用预训练模型)
  - [开始训练](#开始训练)

## 概述

在新数据集上进行模型微调有两个步骤。

1. 添加对新数据集的支持。请参考[准备数据集](prepare_dataset.md)和[自定义数据集](../advanced_guides/customize_dataset.md)。
2. 修改配置文件。本教程将讨论这一部分。

## 选择模板配置

这里我们以 `configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py` 为例。我们首先将该配置文件复制到同一文件夹，并将其重命名为 `tsn_ucf101.py`，然后需要注意配置中的四个部分，具体来说，为不存在的键添加新键，并修改现有键的原始键。

## 修改 Head

`cls_head` 中的 `num_classes` 需要更改为新数据集的类别数。预训练模型的权重会被重用，除了最后的预测层。因此，更改类别数是安全的。在我们的例子中，UCF101 有 101 个类别。所以我们将其从 400（Kinetics-400 的类别数）改为 101。

```python
        type='TSNHead',
        num_classes=101  # 将 400 修改为 101
        ))
```

## 修改数据集

```python
        ))
```

## 修改数据集

MMAction2 支持 UCF101、Kinetics-400、Moments in Time、Multi-Moments in Time、THUMOS14、Something-Something V1&V2、ActivityNet 数据集。用户可能需要将上述其中一个数据集适应到他们的特殊数据集上。你可以参考[准备数据集](prepare_dataset.md)和[自定义数据集](../advanced_guides/customize_dataset.md)了解更多细节。在我们的例子中，UCF101 已经由各种数据集类型支持，例如 `VideoDataset`，因此我们将配置修改如下。

```python
dataset_type = 'VideoDataset'
data_root = 'data/ucf101/videos_train/'
data_root_val = 'data/ucf101/videos_val/'
ann_file_train = 'data/ucf101/ucf101_train_list.txt'
ann_file_val = 'data/ucf101/ucf101_val_list.txt'
```

## 修改训练计划

微调通常需要较小的学习率和较少的训练周期。

```python
    max_epochs=50,  # 将 100 修改为 50
    val_interval=1)

    dict(

max_epochs=50,  # 将 100 修改为 50

    dict(
        begin=0,
        end=50,  # 将 100 修改为 50
        milestones=[20, 40],  # 修改 milestones
]

        type='SGD',
        lr=0.005, # 将 0.01 修改为 0.005
        momentum=0.9,
        weight_decay=0.0001),
    clip_grad=dict(max_norm=40, norm_type=2))
```

## 使用预训练模型

为了在整个网络上使用预训练模型，新配置文件在 `load_from` 中添加了预训练模型的链接。我们在 `configs/_base_/default_runtime.py` 中设置 `load_from=None` 作为默认值，并且根据[继承设计](config.md)，用户可以通过在其配置中设置 `load_from` 来直接更改它。

```python
# use the pre-trained model for the whole TSN network

```python
load_from = 'https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb_20220906-cd10898e.pth'  # 模型路径可以在模型库中找到
```

## 开始训练

现在，我们已经完成了微调的配置文件，如下所示：

```python
_base_ = [
    '../../_base_/models/tsn_r50.py', '../../_base_/schedules/sgd_100e.py',
    '../../_base_/default_runtime.py'
]

        ))

file_client_args = dict(io_backend='disk')

    dict(type='DecordInit', **file_client_args),

    dict(type='SampleFrames', clip_len=1, frame_interval=1, num_clips=3),
    dict(type='DecordDecode'),
    dict(type='Resize', scale=(-1, 256)),
    dict(
        type='MultiScaleCrop',
        input_size=224,
        scales=(1, 0.875, 0.75, 0.66),
        random_crop=False,
        max_wh_scale_gap=1),
    dict(type='Resize', scale=(224, 224), keep_ratio=False),
    dict(type='Flip', flip_ratio=0.5),
    dict(type='FormatShape', input_format='NCHW'),
    dict(type='PackActionInputs')
]
    dict(
        type='SampleFrames',
        clip_len=1,
        frame_interval=1,
        num_clips=3,
        test_mode=True),
    dict(type='CenterCrop', crop_size=224),

frame_interval=1,
]
    dict(
        num_clips=25,
    dict(type='TenCrop', crop_size=224),
]

        ann_file=ann_file_train,
        data_prefix=dict(video=data_root),

        ann_file=ann_file_val,
        data_prefix=dict(video=data_root_val),
        pipeline=val_pipeline,
        test_mode=True))

    dict(
        begin=0,
]

val_evaluator = dict(type='AccMetric')

default_hooks = dict(checkpoint=dict(interval=3, max_keep_ckpts=3))

# Default setting for scaling LR automatically
#   - `enable` means enable scaling LR automatically
#       or not by default.
#   - `base_batch_size` = (8 GPUs) x (32 samples per GPU).
auto_scale_lr = dict(enable=False, base_batch_size=256)

load_from = 'https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb_20220906-cd10898e.pth'

```

另一种更简单的方法是继承 kinetics400 配置，并只指定修改的键。请确保自定义配置与 `configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py` 在同一个文件夹中。

```python
_base_ = [
    'tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py'  # 继承模板配置
]

        num_classes=101))  # 将 400 修改为 101

type='TSNHead',

        data_prefix=dict(video=data_root)))
        data_prefix=dict(video=data_root_val)))

    dict(
        begin=0,

    dict(
        begin=0,
]

```

你可以使用以下命令在你的数据集上微调模型。

```shell
python tools/train.py ${CONFIG_FILE} [可选参数]
```

例如：在确定性选项下，在 Kinetics-400 数据集上训练 TSN 模型。

```shell
python tools/train.py configs/recognition/tsn/tsn_ucf101.py  \
  --seed=0 --deterministic
```

```

```shell
```

更多细节，请参考[训练和测试教程](train_test.md)中的**训练**部分。

# Add New Metrics

## Develop with the source code of MMSegmentation

Here we show how to develop a new metric with an example of `CustomMetric` as the following.

1. Create a new file `mmseg/evaluation/metrics/custom_metric.py`.

   ```python
   from typing import List, Sequence

   from mmengine.evaluator import BaseMetric

   from mmseg.registry import METRICS

   @METRICS.register_module()
   class CustomMetric(BaseMetric):

           """
           The metric first processes each batch of data_samples and predictions,
           and appends the processed results to the results list. Then it
           collects all results together from all ranks if distributed training
           is used. Finally, it computes the metrics of the entire dataset.
           """

       def process(self, data_batch: dict, data_samples: Sequence[dict]) -> None:
           pass

       def compute_metrics(self, results: list) -> dict:
           pass

"""

           pass

           pass

       def evaluate(self, size: int) -> dict:
           pass
   ```

   In the above example, `CustomMetric` is a subclass of `BaseMetric`. It has three methods: `process`, `compute_metrics` and `evaluate`.

   - `process()` process one batch of data samples and predictions. The processed results are stored in `self.results` which will be used to compute the metrics after all the data samples are processed. Please refer to [MMEngine documentation](https://github.com/open-mmlab/mmengine/blob/main/docs/en/design/evaluation.md) for more details.

   - `compute_metrics()` is used to compute the metrics from the processed results.

- `compute_metrics()` is used to compute the metrics from the processed results.

   - `evaluate()` is an interface to compute the metrics and return the results. It will be called by `ValLoop` or `TestLoop` in the `Runner`. In most cases, you don't need to override this method, but you can override it if you want to do some extra work.

   **Note:** You might find the details of calling `evaluate()` method in the `Runner` [here](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L366). The `Runner` is the executor of the training and testing process, you can find more details about it at the [engine document](./engine.md).

2. Import the new metric in `mmseg/evaluation/metrics/__init__.py`.

   ```python
   from .custom_metric import CustomMetric
   __all__ = ['CustomMetric', ...]
   ```

3. Add the new metric to the config file.

   ```python
   val_evaluator = dict(type='CustomMetric', arg1=xxx, arg2=xxx)

__all__ = ['CustomMetric', ...]
   ```

   ```python
   test_evaluator = dict(type='CustomMetric', arg1=xxx, arg2=xxx)
   ```

## Develop with the released version of MMSegmentation

The above example shows how to develop a new metric with the source code of MMSegmentation. If you want to develop a new metric with the released version of MMSegmentation, you can follow the following steps.

1. Create a new file `/Path/to/metrics/custom_metric.py`, implement the `process`, `compute_metrics` and `evaluate` methods, `evaluate` method is optional.

2. Import the new metric in your code or config file.

   ```python
   from path.to.metrics import CustomMetric
   ```

   or

   ```python
   custom_imports = dict(imports=['/Path/to/metrics'], allow_failed_imports=False)

```

   or

   ```python

   ```

# 自定义数据集

在本节中，您将了解如何使用自定义数据集训练和测试预定义模型。

基本步骤如下：

1. 准备数据
2. 准备配置文件
3. 在自定义数据集上训练，测试和推理模型

## 数据准备

理想情况下我们可以重新组织自定义的原始数据并将标注格式转换成 KITTI 风格。但是，考虑到对于自定义数据集而言，KITTI 格式的校准文件和 3D 标注难以获得，因此我们在文档中介绍基本的数据格式。

### 基本数据格式

#### 点云格式

目前，我们只支持 `.bin` 格式的点云用于训练和推理。在训练自己的数据集之前，需要将其它格式的点云文件转换成 `.bin` 文件。常见的点云数据格式包括 `.pcd` 和 `.las`，我们列举了一些开源工具作为参考。

1. `.pcd` 转换成 `.bin`：https://github.com/DanielPollithy/pypcd

- 您可以通过以下指令安装 `pypcd`：

  ```bash
  pip install git+https://github.com/DanielPollithy/pypcd.git
  ```

- 您可以使用以下脚本读取 `.pcd` 文件，并将其转换成 `.bin` 格式来保存：

  ```python
  import numpy as np
  from pypcd import pypcd

  pcd_data = pypcd.PointCloud.from_path('point_cloud_data.pcd')
  points = np.zeros([pcd_data.width, 4], dtype=np.float32)
  points[:, 0] = pcd_data.pc_data['x'].copy()
  points[:, 1] = pcd_data.pc_data['y'].copy()
  points[:, 2] = pcd_data.pc_data['z'].copy()
  points[:, 3] = pcd_data.pc_data['intensity'].copy().astype(np.float32)
  with open('point_cloud_data.bin', 'wb') as f:

points[:, 2] = pcd_data.pc_data['z'].copy()
      f.write(points.tobytes())
  ```

2. `.las` 转换成 `.bin`：常见的转换流程为 `.las -> .pcd -> .bin`，`.las -> .pcd` 的转换可以用该[工具](https://github.com/Hitachi-Automotive-And-Industry-Lab/semantic-segmentation-editor)实现。

#### 标签格式

最基本的信息：每个场景的 3D 边界框和类别标签应该包含在 `.txt` 标注文件中。每一行代表特定场景的一个 3D 框，如下所示：

```
# 格式：[x, y, z, dx, dy, dz, yaw, category_name]
1.23 1.42 0.23 3.96 1.65 1.55 1.56 Car
3.51 2.15 0.42 1.05 0.87 1.86 1.23 Pedestrian
...
```

**注意**：对于自定义数据集的评估我们目前只支持 KITTI 评估方法。

3D 框应存储在统一的 3D 坐标系中。

#### 校准格式

对于每个激光雷达收集的点云数据，通常会进行融合并转换到特定的激光雷达坐标系。因此，校准信息文件中通常应该包含每个相机的内参矩阵和激光雷达到每个相机的外参转换矩阵，并保存在 `.txt` 校准文件中，其中 `Px` 表示 `camera_x` 的内参矩阵，`lidar2camx` 表示 `lidar` 到 `camera_x` 的外参转换矩阵。

```
P0
P1
P2
P3
P4
...
lidar2cam0
lidar2cam1
lidar2cam2
lidar2cam3
lidar2cam4
...
```

### 原始数据结构

#### 基于激光雷达的 3D 检测

```
P0
P1
P2
P3
P4
...
...
```

### 原始数据结构

基于激光雷达的 3D 目标检测原始数据通常组织成如下格式，其中 `ImageSets` 包含划分文件，指明哪些文件属于训练/验证集，`points` 包含存储成 `.bin` 格式的点云数据，`labels` 包含 3D 检测的标签文件。

```
├── tools
├── data
│   ├── custom
│   │   │   ├── train.txt
│   │   │   ├── val.txt
│   │   ├── points
│   │   │   ├── 000000.bin
│   │   │   ├── 000001.bin
│   │   │   ├── ...
│   │   ├── labels
│   │   │   ├── 000000.txt
│   │   │   ├── 000001.txt
│   │   │   ├── ...
```

#### 基于视觉的 3D 检测

基于视觉的 3D 目标检测原始数据通常组织成如下格式，其中 `ImageSets` 包含划分文件，指明哪些文件属于训练/验证集，`images` 包含来自不同相机的图像，例如 `camera_x` 获得的图像应放在 `images/images_x` 下，`calibs` 包含校准信息文件，其中存储了每个相机的内参矩阵，`labels` 包含 3D 检测的标签文件。

```
├── tools
├── data
│   │   ├── calibs

```
├── tools
├── data
│   │   │   ├── ...
│   │   ├── images
│   │   │   ├── images_0
│   │   │   │   ├── 000000.png
│   │   │   │   ├── 000001.png
│   │   │   │   ├── ...
│   │   │   ├── images_1
│   │   │   ├── images_2
│   │   │   ├── ...
│   │   │   ├── ...
```

#### 多模态 3D 检测

多模态 3D 目标检测原始数据通常组织成如下格式。不同于基于视觉的 3D 目标检测，`calibs` 里的校准信息文件存储了每个相机的内参矩阵和外参矩阵。

```
├── tools
├── data
│   │   │   ├── ...
│   │   │   ├── ...

│   │   │   ├── ...
│   │   │   ├── ...
│   │   │   ├── ...
│   │   │   ├── ...
```

#### 基于激光雷达的 3D 语义分割

基于激光雷达的 3D 语义分割原始数据通常组织成如下格式，其中 `ImageSets` 包含划分文件，指明哪些文件属于训练/验证集，`points` 包含点云数据，`semantic_mask` 包含逐点级标签。

```
├── tools
├── data
│   │   │   ├── ...
│   │   ├── semantic_mask
│   │   │   ├── ...
```

### 数据转换

按照我们的说明准备好原始数据后，您可以直接使用以下命令生成训练/验证信息文件。

```bash

│   │   │   ├── ...
│   │   │   ├── ...
```

### 数据转换

```bash
python tools/create_data.py custom --root-path ./data/custom --out-dir ./data/custom --extra-tag custom
```

## 自定义数据集示例

在完成数据准备后，我们可以在 `mmdet3d/datasets/my_dataset.py` 中创建一个新的数据集来加载数据。

```python
import mmengine

from mmdet3d.registry import DATASETS
from .det3d_dataset import Det3DDataset

class MyDataset(Det3DDataset):

    # 替换成自定义 pkl 信息文件里的所有类别
    METAINFO = {
        'classes': ('Pedestrian', 'Cyclist', 'Car')
    }

    def parse_ann_info(self, info):
        """Process the `instances` in data info to `ann_info`.

        Args:
            info (dict): Data information of single data sample.

        Returns:
            dict: Annotation information consists of the following keys:

                - gt_bboxes_3d (:obj:`LiDARInstance3DBoxes`):

Returns:

                  3D ground truth bboxes.
                - gt_labels_3d (np.ndarray): Labels of ground truths.
        """
        ann_info = super().parse_ann_info(info)
        if ann_info is None:
            ann_info = dict()
            # 空实例
            ann_info['gt_bboxes_3d'] = np.zeros((0, 7), dtype=np.float32)
            ann_info['gt_labels_3d'] = np.zeros(0, dtype=np.int64)

        # 过滤掉没有在训练中使用的类别
        ann_info = self._remove_dontcare(ann_info)
        gt_bboxes_3d = LiDARInstance3DBoxes(ann_info['gt_bboxes_3d'])
        ann_info['gt_bboxes_3d'] = gt_bboxes_3d
        return ann_info
```

数据预处理后，用户可以通过以下两个步骤来训练自定义数据集：

1. 修改配置文件来使用自定义数据集。
2. 验证自定义数据集标注的正确性。

这里我们以在自定义数据集上训练 PointPillars 为例：

### 准备配置

这里我们演示一个纯点云训练的配置示例：

#### 准备数据集配置

在 `configs/_base_/datasets/custom.py` 中：

```python
# 数据集设置

### 准备配置

```python
# 数据集设置
dataset_type = 'MyDataset'
data_root = 'data/custom/'
class_names = ['Pedestrian', 'Cyclist', 'Car']  # 替换成您的数据集类别
point_cloud_range = [0, -40, -3, 70.4, 40, 1]  # 根据您的数据集进行调整

    dict(
        load_dim=4,  # 替换成您的点云数据维度
        use_dim=4),  # 替换成在训练和推理时实际使用的维度
    dict(
        with_label_3d=True),
    dict(
        type='ObjectNoise',
        num_try=100,
        translation_std=[1.0, 1.0, 0.5],
        global_rot_range=[0.0, 0.0],
        rot_range=[-0.78539816, 0.78539816]),
    dict(

translation_std=[1.0, 1.0, 0.5],
    dict(
    dict(
]
    dict(
        use_dim=4),
]
# 为可视化阶段的数据和 GT 加载构造流水线
    dict(type='Pack3DDetInputs', keys=['points']),
]

]
        times=2,
            ann_file='custom_infos_train.pkl',  # 指定您的训练 pkl 信息
            data_prefix=dict(pts='points'),

num_workers=1,
        data_prefix=dict(pts='points'),
        ann_file='custom_infos_val.pkl',  # 指定您的验证 pkl 信息
    ann_file=data_root + 'custom_infos_val.pkl',  # 指定您的验证 pkl 信息
```

#### 准备模型配置

对于基于体素化的检测器如 SECOND，PointPillars 及 CenterPoint，点云范围（point cloud range）和体素大小（voxel size）应该根据您的数据集做调整。理论上，`voxel_size` 和 `point_cloud_range` 的设置是相关联的。设置较小的 `voxel_size` 将增加体素数以及相应的内存消耗。此外，需要注意以下问题：

如果将 `point_cloud_range` 和 `voxel_size` 分别设置成 `[0, -40, -3, 70.4, 40, 1]` 和 `[0.05, 0.05, 0.1]`，那么中间特征图的形状应该为 `[(1-(-3))/0.1+1, (40-(-40))/0.05, (70.4-0)/0.05]=[41, 1600, 1408]`。更改 `point_cloud_range` 时，请记得依据 `voxel_size` 更改 `middle_encoder` 里中间特征图的形状。

关于 `anchor_range` 的设置，一般需要根据数据集做调整。需要注意的是，`z` 值需要根据点云的位置做相应调整，具体请参考此 [issue](https://github.com/open-mmlab/mmdetection3d/issues/986)。

关于 `anchor_size` 的设置，通常需要计算整个训练集中目标的长、宽、高的平均值作为 `anchor_size`，以获得最好的结果。

在 `configs/_base_/models/pointpillars_hv_secfpn_custom.py` 中：

```python
voxel_size = [0.16, 0.16, 4]  # 根据您的数据集做调整
point_cloud_range = [0, -39.68, -3, 69.12, 39.68, 1]  # 根据您的数据集做调整

            point_cloud_range=point_cloud_range,
            voxel_size=voxel_size,
        voxel_size=voxel_size,
        point_cloud_range=point_cloud_range),
    # `output_shape` 需要根据 `point_cloud_range` 和 `voxel_size` 做相应调整

layer_strides=[2, 2, 2],
        # 根据您的数据集调整 `ranges` 和 `sizes`
            ranges=[
                [0, -39.68, -0.6, 69.12, 39.68, -0.6],
                [0, -39.68, -1.78, 69.12, 39.68, -1.78],
            ],

rotations=[0, 1.57],
            type='mmdet.SmoothL1Loss', beta=1.0 / 9.0, loss_weight=2.0),
    # 模型训练和测试设置
            dict(  # for Pedestrian
            dict(  # for Cyclist

neg_iou_thr=0.35,
            dict(  # for Car
        ],
```

#### 准备整体配置

use_rotate_nms=True,
```

我们将上述的所有配置组合在 `configs/pointpillars/pointpillars_hv_secfpn_8xb6_custom.py` 文件中：

```python
_base_ = [
    '../_base_/models/pointpillars_hv_secfpn_custom.py',
    '../_base_/datasets/custom.py',
]
```

#### 可视化数据集（可选）

为了验证准备的数据和配置是否正确，我们建议在训练和验证前使用 `tools/misc/browse_dataset.py` 脚本可视化数据集和标注。更多细节请参考[可视化文档](https://mmdetection3d.readthedocs.io/zh_CN/dev-1.x/user_guides/visualization.html)。

## 评估

准备好数据和配置之后，您可以遵循我们的文档直接运行训练/测试脚本。

**注意**：我们为自定义数据集提供了 KITTI 风格的评估实现方法。在数据集配置中需要包含如下内容：

```python
```

# MSG3D Project

[Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://arxiv.org/abs/2003.14111)

Spatial-temporal graphs have been widely used by skeleton-based action recognition algorithms to model human action dynamics. To capture robust movement patterns from these graphs, long-range and multi-scale context aggregation and spatial-temporal dependency modeling are critical aspects of a powerful feature extractor. However, existing methods have limitations in achieving (1) unbiased long-range joint relationship modeling under multi-scale operators and (2) unobstructed cross-spacetime information flow for capturing complex spatial-temporal dependencies. In this work, we present (1) a simple method to disentangle multi-scale graph convolutions and (2) a unified spatial-temporal graph convolutional operator named G3D. The proposed multi-scale aggregation scheme disentangles the importance of nodes in different neighborhoods for effective long-range modeling. The proposed G3D module leverages dense cross-spacetime edges as skip connections for direct information propagation across the spatial-temporal graph. By coupling these proposals, we develop a powerful feature extractor named MS-G3D based on which our model outperforms previous state-of-the-art methods on three large-scale datasets: NTU RGB+D 60, NTU RGB+D 120, and Kinetics Skeleton 400.

<img src="https://user-images.githubusercontent.com/58767402/223127347-135bb92b-2dee-46d9-95fc-cebf65c27fc8.png" width="800"/>
</div>

## Usage

### Setup Environment

Please refer to [Installation](https://mmaction2.readthedocs.io/en/latest/get_started/installation.html) to install MMAction2.

Assume that you are located at `$MMACTION2/projects/msg3d`.

Add the current folder to `PYTHONPATH`, so that Python can find your code. Run the following command in the current directory to add it.

```shell
```

### Data Preparation

Prepare the NTU60 dataset according to the [instruction](https://github.com/open-mmlab/mmaction2/blob/main/tools/data/skeleton/README.md).

```

Create a symbolic link from `$MMACTION2/data` to `./data` in the current directory, so that Python can locate your data. Run the following command in the current directory to create the symbolic link.

```shell
ln -s ../../data ./data
```

Prepare the NTU60 dataset according to the [instruction](https://github.com/open-mmlab/mmaction2/blob/1.x/tools/data/skeleton/README.md).

```bash
mim train mmaction configs/msg3d_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py
```

```bash
mim train mmaction configs/msg3d_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py --launcher pytorch --gpus 8
```

```bash

```bash
```

```bash
mim train mmaction configs/msg3d_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py --launcher slurm \
    --gpus 8 --gpus-per-node 8 --partition $PARTITION
```

```bash
mim test mmaction configs/msg3d_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py --checkpoint $CHECKPOINT
```

```bash
mim test mmaction configs/msg3d_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py --checkpoint $CHECKPOINT --launcher pytorch --gpus 8
```

```bash
mim test mmaction configs/msg3d_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py --checkpoint $CHECKPOINT --launcher slurm \
```

## Results

--gpus 8 --gpus-per-node 8 --partition $PARTITION
```

## Results

| frame sampling strategy | modality | gpus | backbone | top1 acc | testing protocol |                     config                     |                     ckpt                     |                     log                     |
| :---------------------: | :------: | :--: | :------: | :------: | :--------------: | :--------------------------------------------: | :------------------------------------------: | :-----------------------------------------: |

|       uniform 100       |  joint   |  8   |  MSG3D   |   92.3   |     10 clips     | [config](./configs/msg3d_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/projects/msg3d/msg3d_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d/msg3d_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d_20230309-73b97296.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/projects/msg3d/msg3d_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d/msg3d_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.log) |

|       uniform 100       |  joint   |  8   |  MSG3D   |   89.6   |     10 clips     | [config](./configs/msg3d_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/projects/msg3d/msg3d_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d/msg3d_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d_20230308-c325d222.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/projects/msg3d/msg3d_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d/msg3d_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d.log) |

```bibtex
@inproceedings{liu2020disentangling,
  title={Disentangling and unifying graph convolutions for skeleton-based action recognition},

```bibtex
  author={Liu, Ziyu and Zhang, Hongwen and Chen, Zhenghao and Wang, Zhiyong and Ouyang, Wanli},
  pages={143--152},
}
```

# Contextual Instance Decoupling for Robust Multi-Person Pose Estimation

<details>
<summary align="right"><a href="https://openaccess.thecvf.com/content/CVPR2022/html/Wang_Contextual_Instance_Decoupling_for_Robust_Multi-Person_Pose_Estimation_CVPR_2022_paper.html">CID (CVPR'2022)</a></summary>

```bibtex
@InProceedings{Wang_2022_CVPR,
    author    = {Wang, Dongkai and Zhang, Shiliang},
    title     = {Contextual Instance Decoupling for Robust Multi-Person Pose Estimation},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2022},
    pages     = {11060-11068}
}
```

Crowded scenes make it challenging to differentiate persons and locate their pose keypoints. This paper proposes the Contextual Instance Decoupling (CID), which presents a new pipeline for multi-person pose estimation. Instead of relying on person bounding boxes to spatially differentiate persons, CID decouples persons in an image into multiple instance-aware feature maps. Each of those feature maps is hence adopted to infer keypoints for a specific person. Compared with bounding box detection, CID is differentiable and robust to detection errors. Decoupling persons into different feature maps allows to isolate distractions from other persons, and explore context cues at scales larger than the bounding box size. Experiments show that CID outperforms previous multi-person pose estimation pipelines on crowded scenes pose estimation benchmarks in both accuracy and efficiency. For instance, it achieves 71.3% AP on CrowdPose, outperforming the recent single-stage DEKR by 5.6%, the bottom-up CenterAttention by 3.7%, and the top-down JCSPPE by 5.3%. This advantage sustains on the commonly used COCO benchmark.

<img src="https://github.com/kennethwdk/CID/raw/main/img/framework.png">
</div>

# Convolutional pose machines

<details>

```bibtex
}
```

booktitle={Proceedings of the IEEE conference on Computer Vision and Pattern Recognition},
}
```

We introduce associative embedding, a novel method for supervising convolutional neural networks for the task of detection and grouping. A number of computer vision problems can be framed in this manner including multi-person pose estimation, instance segmentation, and multi-object tracking. Usually the grouping of detections is achieved with multi-stage pipelines, instead we propose an approach that teaches a network to simultaneously output detections and group assignments. This technique can be easily integrated into any state-of-the-art network architecture that produces pixel-wise predictions. We show how to apply this method to both multi-person pose estimation and instance segmentation and report state-of-the-art performance for multi-person pose on the MPII and MS-COCO datasets.

<img src="https://user-images.githubusercontent.com/15977946/146514331-a599580b-69a5-4ee4-9aaf-4a72f9c25c9a.png">
</div>

# TDAN (CVPR'2020)

> [TDAN: Temporally Deformable Alignment Network for Video Super-Resolution](https://arxiv.org/abs/1812.02898)

Video super-resolution (VSR) aims to restore a photo-realistic high-resolution (HR) video frame from both its corresponding low-resolution (LR) frame (reference frame) and multiple neighboring frames (supporting frames). Due to varying motion of cameras or objects, the reference frame and each support frame are not aligned. Therefore, temporal alignment is a challenging yet important problem for VSR. Previous VSR methods usually utilize optical flow between the reference frame and each supporting frame to wrap the supporting frame for temporal alignment. Therefore, the performance of these image-level wrapping-based models will highly depend on the prediction accuracy of optical flow, and inaccurate optical flow will lead to artifacts in the wrapped supporting frames, which also will be propagated into the reconstructed HR video frame. To overcome the limitation, in this paper, we propose a temporal deformable alignment network (TDAN) to adaptively align the reference frame and each supporting frame at the feature level without computing optical flow. The TDAN uses features from both the reference frame and each supporting frame to dynamically predict offsets of sampling convolution kernels. By using the corresponding kernels, TDAN transforms supporting frames to align with the reference frame. To predict the HR video frame, a reconstruction network taking aligned frames and the reference frame is utilized. Experimental results demonstrate the effectiveness of the proposed TDAN-based VSR model.

 <img src="https://user-images.githubusercontent.com/7676947/144035224-a87cc41e-1352-4ffa-8b07-eda5ace8a0b1.png" width="400"/>
</div >

Evaluated on Y-channel. 8 pixels in each border are cropped before evaluation.

|                                 Model                                  |     Dataset     | PSNR (Y)  | SSIM (Y)  |    Training Resources    |                                 Download                                  |
| :--------------------------------------------------------------------: | :-------------: | :-------: | :-------: | :----------------------: | :-----------------------------------------------------------------------: |
| [tdan_x4_1xb16-lr1e-4-400k_vimeo90k-bi](./tdan_x4_8xb16-lr1e-4-400k_vimeo90k-bi.py) |        -        |     -     |     -     | 8 (Tesla V100-SXM2-32GB) |                                     -                                     |

| [tdan_x4_1xb16-lr1e-4-400k_vimeo90k-bd](./tdan_x4_8xb16-lr1e-4-400k_vimeo90k-bd.py) |        -        |     -     |     -     | 8 (Tesla V100-SXM2-32GB) |                                     -                                     |
| [tdan_x4ft_1xb16-lr5e-5-400k_vimeo90k-bi](./tdan_x4ft_8xb16-lr5e-5-400k_vimeo90k-bi.py) |   Vid4 (BIx4)   | **26.49** | **0.792** | 8 (Tesla V100-SXM2-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bix4_20210528-739979d9.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bix4_20210528_135616.log.json) |
| [tdan_x4ft_1xb16-lr5e-5-400k_vimeo90k-bi](./tdan_x4ft_8xb16-lr5e-5-400k_vimeo90k-bi.py) | SPMCS-30 (BIx4) | **30.42** | **0.856** | 8 (Tesla V100-SXM2-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bix4_20210528-739979d9.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bix4_20210528_135616.log.json) |

| [tdan_x4ft_1xb16-lr5e-5-400k_vimeo90k-bi](./tdan_x4ft_8xb16-lr5e-5-400k_vimeo90k-bi.py) |   Vid4 (BDx4)   |   25.93   |   0.772   | 8 (Tesla V100-SXM2-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bix4_20210528-739979d9.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bix4_20210528_135616.log.json) |
| [tdan_x4ft_1xb16-lr5e-5-400k_vimeo90k-bi](./tdan_x4ft_8xb16-lr5e-5-400k_vimeo90k-bi.py) | SPMCS-30 (BDx4) |   29.69   |   0.842   | 8 (Tesla V100-SXM2-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bix4_20210528-739979d9.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bix4_20210528_135616.log.json) |

| [tdan_x4ft_1xb16-lr5e-5-800k_vimeo90k-bd](./tdan_x4ft_8xb16-lr5e-5-800k_vimeo90k-bd.py) |   Vid4 (BIx4)   |   25.80   |   0.784   | 8 (Tesla V100-SXM2-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bdx4_20210528-c53ab844.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bdx4_20210528_122401.log.json) |
| [tdan_x4ft_1xb16-lr5e-5-800k_vimeo90k-bd](./tdan_x4ft_8xb16-lr5e-5-800k_vimeo90k-bd.py) | SPMCS-30 (BIx4) |   29.56   |   0.851   | 8 (Tesla V100-SXM2-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bdx4_20210528-c53ab844.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bdx4_20210528_122401.log.json) |

| [tdan_x4ft_1xb16-lr5e-5-800k_vimeo90k-bd](./tdan_x4ft_8xb16-lr5e-5-800k_vimeo90k-bd.py) |   Vid4 (BDx4)   | **26.87** | **0.815** | 8 (Tesla V100-SXM2-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bdx4_20210528-c53ab844.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bdx4_20210528_122401.log.json) |
| [tdan_x4ft_1xb16-lr5e-5-800k_vimeo90k-bd](./tdan_x4ft_8xb16-lr5e-5-800k_vimeo90k-bd.py) | SPMCS-30 (BDx4) | **30.77** | **0.868** | 8 (Tesla V100-SXM2-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bdx4_20210528-c53ab844.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bdx4_20210528_122401.log.json) |

**Train**

<details>

TDAN is trained with two stages.

**Train**

<details>

**Stage 1**: Train with a larger learning rate (1e-4)

```shell
# cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/tdan/tdan_x4_1xb16-lr1e-4-400k_vimeo90k-bi.py

python tools/train.py configs/tdan/tdan_x4_1xb16-lr1e-4-400k_vimeo90k-bi.py

./tools/dist_train.sh cconfigs/tdan/tdan_x4_1xb16-lr1e-4-400k_vimeo90k-bi.py 8
```

**Stage 2**: Fine-tune with a smaller learning rate (5e-5)

```shell
# cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/tdan/tdan_x4ft_1xb16-lr5e-5-400k_vimeo90k-bi.py

python tools/train.py configs/tdan/tdan_x4ft_1xb16-lr5e-5-400k_vimeo90k-bi.py

./tools/dist_train.sh configs/tdan/tdan_x4ft_1xb16-lr5e-5-400k_vimeo90k-bi.py 8
```

```

**Test**

<details>

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/tdan/tdan_x4ft_1xb16-lr5e-5-400k_vimeo90k-bi.py https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bix4_20210528-739979d9.pth

python tools/test.py configs/tdan/tdan_x4ft_1xb16-lr5e-5-400k_vimeo90k-bi.py https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bix4_20210528-739979d9.pth

./tools/dist_test.sh configs/tdan/tdan_x4ft_1xb16-lr5e-5-400k_vimeo90k-bi.py https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bix4_20210528-739979d9.pth 8
```

```bibtex
@InProceedings{tian2020tdan,
  title={TDAN: Temporally-Deformable Alignment Network for Video Super-Resolution},
  author={Tian, Yapeng and Zhang, Yulun and Fu, Yun and Xu, Chenliang},
  booktitle = {Proceedings of the IEEE conference on Computer Vision and Pattern Recognition},
}
```

# Dynamic Graph CNN for Learning on Point Clouds

> [Dynamic Graph CNN for Learning on Point Clouds](https://arxiv.org/abs/1801.07829)

Point clouds provide a flexible geometric representation suitable for countless applications in computer graphics; they also comprise the raw output of most 3D data acquisition devices. While hand-designed features on point clouds have long been proposed in graphics and vision, however, the recent overwhelming success of convolutional neural networks (CNNs) for image analysis suggests the value of adapting insight from CNN to the point cloud world. Point clouds inherently lack topological information so designing a model to recover topology can enrich the representation power of point clouds. To this end, we propose a new neural network module dubbed EdgeConv suitable for CNN-based high-level tasks on point clouds including classification and segmentation. EdgeConv acts on graphs dynamically computed in each layer of the network. It is differentiable and can be plugged into existing architectures. Compared to existing modules operating in extrinsic space or treating each point independently, EdgeConv has several appealing properties: It incorporates local neighborhood information; it can be stacked applied to learn global shape properties; and in multi-layer systems affinity in feature space captures semantic characteristics over potentially long distances in the original embedding. We show the performance of our model on standard benchmarks including ModelNet40, ShapeNetPart, and S3DIS.

<img src="https://user-images.githubusercontent.com/30491025/143855852-3d7888ed-2cfc-416c-9ec8-57621edeaa34.png" width="800"/>
</div>

We implement DGCNN and provide the results and checkpoints on S3DIS dataset.

**Notice**: We follow the implementations in the original DGCNN paper and a PyTorch implementation of DGCNN [code](https://github.com/AnTao97/dgcnn.pytorch).

### S3DIS

|                           Method                           | Split  |   Lr schd   | Mem (GB) | Inf time (fps) | mIoU (Val set) |                                                                                                                                                                                                 Download                                                                                                                                                                                                 |

| [DGCNN](./dgcnn_4xb32-cosine-100e_s3dis-seg_test-area1.py) | Area_1 | cosine 100e |   13.1   |                |     68.33      | [model](https://download.openmmlab.com/mmdetection3d/v0.17.0_models/dgcnn/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class/area1/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class_20210731_000734-39658f14.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.17.0_models/dgcnn/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class/area1/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class_20210731_000734.log.json) |

| [DGCNN](./dgcnn_4xb32-cosine-100e_s3dis-seg_test-area2.py) | Area_2 | cosine 100e |   13.1   |                |     40.68      | [model](https://download.openmmlab.com/mmdetection3d/v0.17.0_models/dgcnn/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class/area2/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class_20210731_144648-aea9ecb6.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.17.0_models/dgcnn/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class/area2/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class_20210731_144648.log.json) |

| [DGCNN](./dgcnn_4xb32-cosine-100e_s3dis-seg_test-area3.py) | Area_3 | cosine 100e |   13.1   |                |     69.38      | [model](https://download.openmmlab.com/mmdetection3d/v0.17.0_models/dgcnn/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class/area3/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class_20210801_154629-2ff50ee0.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.17.0_models/dgcnn/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class/area3/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class_20210801_154629.log.json) |

| [DGCNN](./dgcnn_4xb32-cosine-100e_s3dis-seg_test-area4.py) | Area_4 | cosine 100e |   13.1   |                |     50.07      | [model](https://download.openmmlab.com/mmdetection3d/v0.17.0_models/dgcnn/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class/area4/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class_20210802_073551-dffab9cd.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.17.0_models/dgcnn/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class/area4/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class_20210802_073551.log.json) |

| [DGCNN](./dgcnn_4xb32-cosine-100e_s3dis-seg_test-area5.py) | Area_5 | cosine 100e |   13.1   |                |     50.59      | [model](https://download.openmmlab.com/mmdetection3d/v0.17.0_models/dgcnn/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class/area5/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class_20210730_235824-f277e0c5.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.17.0_models/dgcnn/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class/area5/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class_20210730_235824.log.json) |

| [DGCNN](./dgcnn_4xb32-cosine-100e_s3dis-seg_test-area6.py) | Area_6 | cosine 100e |   13.1   |                |     77.94      | [model](https://download.openmmlab.com/mmdetection3d/v0.17.0_models/dgcnn/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class/area6/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class_20210802_154317-e3511b32.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.17.0_models/dgcnn/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class/area6/dgcnn_32x4_cosine_100e_s3dis_seg-3d-13class_20210802_154317.log.json) |

|                           DGCNN                            | 6-fold |             |          |                |     59.43      |                                                                                                                                                                                                                                                                                                                                                                                                          |

- We use XYZ+Color+Normalized_XYZ as input in all the experiments on S3DIS datasets.
- `Area_5` Split means training the model on Area_1, 2, 3, 4, 6 and testing on Area_5.
- `6-fold` Split means the overall result of 6 different splits (Area_1, Area_2, Area_3, Area_4, Area_5 and Area_6 Splits).

- Users need to modify `train_area` and `test_area` in the S3DIS dataset's [config](./configs/_base_/datasets/s3dis_seg-3d-13class.py) to set the training and testing areas, respectively.

## Indeterminism

Since DGCNN testing adopts sliding patch inference which involves random point sampling, and the test script uses fixed random seeds while the random seeds of validation in training are not fixed, the test results may be slightly different from the results reported above.

```latex
@article{dgcnn,
  title={Dynamic Graph CNN for Learning on Point Clouds},
  author={Wang, Yue and Sun, Yongbin and Liu, Ziwei and Sarma, Sanjay E. and Bronstein, Michael M. and Solomon, Justin M.},
  journal={ACM Transactions on Graphics (TOG)},
}
```

# ScanNet Dataset

MMDetection3D supports LiDAR-based detection and segmentation on ScanNet dataset. This page provides specific tutorials about the usage.

## Dataset preparation

For the overall process, please refer to the [README](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/data/scannet/README.md) page for ScanNet.

### Export ScanNet point cloud data

By exporting ScanNet data, we load the raw point cloud data and generate the relevant annotations including semantic labels, instance labels and ground truth bounding boxes.

```shell
python batch_load_scannet_data.py
```

The directory structure before data preparation should be as below

```
├── tools
├── data
│   ├── scannet
│   │   ├── meta_data
│   │   ├── scans
│   │   │   ├── scenexxxx_xx
│   │   ├── batch_load_scannet_data.py
│   │   ├── load_scannet_data.py
│   │   ├── scannet_utils.py
│   │   ├── README.md
```

```

Under folder `scans` there are overall 1201 train and 312 validation folders in which raw point cloud data and relevant annotations are saved. For instance, under folder `scene0001_01` the files are as below:

- `scene0001_01_vh_clean_2.ply`: Mesh file storing coordinates and colors of each vertex. The mesh's vertices are taken as raw point cloud data.
- `scene0001_01.aggregation.json`: Aggregation file including object ID, segments ID and label.
- `scene0001_01_vh_clean_2.0.010000.segs.json`: Segmentation file including segments ID and vertex.
- `scene0001_01.txt`: Meta file including axis-aligned matrix, etc.
- `scene0001_01_vh_clean_2.labels.ply`: Annotation file containing the category of each vertex.

The procedure of exporting ScanNet data by running `python batch_load_scannet_data.py` mainly includes the following 3 steps:

- Export original files to point cloud, instance label, semantic label and bounding box file.
- Downsample raw point cloud and filter invalid classes.
- Save point cloud data and relevant annotation files.

And the core function `export` in `load_scannet_data.py` is as follows:

```python
def export(mesh_file,
           agg_file,
           seg_file,
           meta_file,
           label_map_file,
           output_file=None,
           test_mode=False):

    # label map file: ./data/scannet/meta_data/scannetv2-labels.combined.tsv
    # the various label standards in the label map file, e.g. 'nyu40id'
    label_map = scannet_utils.read_label_mapping(

# label map file: ./data/scannet/meta_data/scannetv2-labels.combined.tsv
        label_map_file, label_from='raw_category', label_to='nyu40id')
    # load raw point cloud data, 6-dims feature: XYZRGB
    mesh_vertices = scannet_utils.read_mesh_vertices_rgb(mesh_file)

    # Load scene axis alignment matrix: a 4x4 transformation matrix
    # transform raw points in sensor coordinate system to a coordinate system
    # which is axis-aligned with the length/width of the room
    lines = open(meta_file).readlines()
    # test set data doesn't have align_matrix
    axis_align_matrix = np.eye(4)
    for line in lines:
        if 'axisAlignment' in line:
            axis_align_matrix = [
                float(x)
                for x in line.rstrip().strip('axisAlignment = ').split(' ')
            ]
            break

axis_align_matrix = [
                float(x)
            ]
            break
    axis_align_matrix = np.array(axis_align_matrix).reshape((4, 4))

    # perform global alignment of mesh vertices
    pts = np.ones((mesh_vertices.shape[0], 4))
    # raw point cloud in homogeneous coordinates, each row: [x, y, z, 1]
    pts[:, 0:3] = mesh_vertices[:, 0:3]
    # transform raw mesh vertices to aligned mesh vertices
    pts = np.dot(pts, axis_align_matrix.transpose())  # Nx4
    aligned_mesh_vertices = np.concatenate([pts[:, 0:3], mesh_vertices[:, 3:]],
                                           axis=1)

    # Load semantic and instance labels
    if not test_mode:
        # each object has one semantic label and consists of several segments
        object_id_to_segs, label_to_segs = read_aggregation(agg_file)
        # many points may belong to the same segment

# each object has one semantic label and consists of several segments
        seg_to_verts, num_verts = read_segmentation(seg_file)
        label_ids = np.zeros(shape=(num_verts), dtype=np.uint32)
        object_id_to_label_id = {}
        for label, segs in label_to_segs.items():
            label_id = label_map[label]
            for seg in segs:
                verts = seg_to_verts[seg]
                # each point has one semantic label
                label_ids[verts] = label_id
        instance_ids = np.zeros(
            shape=(num_verts), dtype=np.uint32)  # 0: unannotated
        for object_id, segs in object_id_to_segs.items():
                # object_id is 1-indexed, i.e. 1,2,3,.,,,.NUM_INSTANCES
                # each point belongs to one object

for seg in segs:
                instance_ids[verts] = object_id
                if object_id not in object_id_to_label_id:
                    object_id_to_label_id[object_id] = label_ids[verts][0]
        # bbox format is [x, y, z, x_size, y_size, z_size, label_id]
        # [x, y, z] is gravity center of bbox, [x_size, y_size, z_size] is axis-aligned
        # [label_id] is semantic label id in 'nyu40id' standard
        # Note: since 3D bbox is axis-aligned, the yaw is 0.
        unaligned_bboxes = extract_bbox(mesh_vertices, object_id_to_segs,
                                        object_id_to_label_id, instance_ids)
        aligned_bboxes = extract_bbox(aligned_mesh_vertices, object_id_to_segs,
                                      object_id_to_label_id, instance_ids)
    ...

aligned_bboxes = extract_bbox(aligned_mesh_vertices, object_id_to_segs,
    ...

    return mesh_vertices, label_ids, instance_ids, unaligned_bboxes, \
        aligned_bboxes, object_id_to_label_id, axis_align_matrix

```

After exporting each scan, the raw point cloud could be downsampled, e.g. to 50000, if the number of points is too large (the raw point cloud won't be downsampled if it's also used in 3D semantic segmentation task). In addition, invalid semantic labels outside of `nyu40id` standard or optional `DONOT CARE` classes should be filtered. Finally, the point cloud data, semantic labels, instance labels and ground truth bounding boxes should be saved in `.npy` files.

### Export ScanNet RGB data (optional)

By exporting ScanNet RGB data, for each scene we load a set of RGB images with corresponding 4x4 pose matrices, and a single 4x4 camera intrinsic matrix. Note, that this step is optional and can be skipped if multi-view detection is not planned to use.

```shell
python extract_posed_images.py
```

Each of 1201 train, 312 validation and 100 test scenes contains a single `.sens` file. For instance, for scene `0001_01` we have `data/scannet/scans/scene0001_01/0001_01.sens`. For this scene all images and poses are extracted to `data/scannet/posed_images/scene0001_01`. Specifically, there will be 300 image files xxxxx.jpg, 300 camera pose files xxxxx.txt and a single `intrinsic.txt` file. Typically, single scene contains several thousand images. By default, we extract only 300 of them with resulting space occupation of \<100 Gb. To extract more images, use `--max-images-per-scene` parameter.

### Create dataset

```shell

```shell
python tools/create_data.py scannet --root-path ./data/scannet \
--out-dir ./data/scannet --extra-tag scannet
```

The above exported point cloud file, semantic label file and instance label file are further saved in `.bin` format. Meanwhile `.pkl` info files are also generated for train or validation. The core function `process_single_scene` of getting data infos is as follows.

```python
def process_single_scene(sample_idx):

    # save point cloud, instance label and semantic label in .bin file respectively, get info['pts_path'], info['pts_instance_mask_path'] and info['pts_semantic_mask_path']
    ...

    # get annotations
    if has_label:
        annotations = {}
        # box is of shape [k, 6 + class]
        aligned_box_label = self.get_aligned_box_label(sample_idx)
        unaligned_box_label = self.get_unaligned_box_label(sample_idx)
        annotations['gt_num'] = aligned_box_label.shape[0]
        if annotations['gt_num'] != 0:

unaligned_box_label = self.get_unaligned_box_label(sample_idx)
            aligned_box = aligned_box_label[:, :-1]  # k, 6
            unaligned_box = unaligned_box_label[:, :-1]
            classes = aligned_box_label[:, -1]  # k
            annotations['name'] = np.array([
                self.label2cat[self.cat_ids2class[classes[i]]]
                for i in range(annotations['gt_num'])
            ])
            # default names are given to aligned bbox for compatibility
            # we also save unaligned bbox info with marked names
            annotations['location'] = aligned_box[:, :3]
            annotations['dimensions'] = aligned_box[:, 3:6]
            annotations['gt_boxes_upright_depth'] = aligned_box
            annotations['unaligned_location'] = unaligned_box[:, :3]
            annotations['unaligned_dimensions'] = unaligned_box[:, 3:6]
            annotations[

annotations['unaligned_location'] = unaligned_box[:, :3]
                'unaligned_gt_boxes_upright_depth'] = unaligned_box
            annotations['index'] = np.arange(
                annotations['gt_num'], dtype=np.int32)
            annotations['class'] = np.array([
                self.cat_ids2class[classes[i]]
            ])
        axis_align_matrix = self.get_axis_align_matrix(sample_idx)
        annotations['axis_align_matrix'] = axis_align_matrix  # 4x4
        info['annos'] = annotations
    return info
```

The directory structure after process should be as below:

```
scannet
├── meta_data
├── batch_load_scannet_data.py
├── load_scannet_data.py
├── scannet_utils.py
├── README.md
├── scans
├── scans_test
├── scannet_instance_data
├── points
│   ├── xxxxx.bin
├── instance_mask
├── semantic_mask

├── scans
├── points
├── seg_info
│   ├── train_label_weight.npy
│   ├── train_resampled_scene_idxs.npy
│   ├── val_label_weight.npy
│   ├── val_resampled_scene_idxs.npy
├── posed_images
│   ├── scenexxxx_xx
│   │   ├── xxxxxx.txt
│   │   ├── xxxxxx.jpg
│   │   ├── intrinsic.txt
├── scannet_infos_train.pkl
├── scannet_infos_val.pkl
├── scannet_infos_test.pkl
```

```

- `points/xxxxx.bin`: The `axis-unaligned` point cloud data after downsample. Since ScanNet 3D detection task takes axis-aligned point clouds as input, while ScanNet 3D semantic segmentation task takes unaligned points, we choose to store unaligned points and their axis-align transform matrix. Note: the points would be axis-aligned in pre-processing pipeline [`GlobalAlignment`](https://github.com/open-mmlab/mmdetection3d/blob/9f0b01caf6aefed861ef4c3eb197c09362d26b32/mmdet3d/datasets/pipelines/transforms_3d.py#L423) of 3D detection task.
- `instance_mask/xxxxx.bin`: The instance label for each point, value range: \[0, NUM_INSTANCES\], 0: unannotated.

- `semantic_mask/xxxxx.bin`: The semantic label for each point, value range: \[1, 40\], i.e. `nyu40id` standard. Note: the `nyu40id` ID will be mapped to train ID in train pipeline `PointSegClassMapping`.
- `seg_info`: The generated infos to support semantic segmentation model training.
  - `train_label_weight.npy`: Weighting factor for each semantic class. Since the number of points in different classes varies greatly, it's a common practice to use label re-weighting to get a better performance.
  - `train_resampled_scene_idxs.npy`: Re-sampling index for each scene. Different rooms will be sampled multiple times according to their number of points to balance training data.
- `posed_images/scenexxxx_xx`: The set of `.jpg` images with `.txt` 4x4 poses and the single `.txt` file with camera intrinsic matrix.

- `scannet_infos_train.pkl`: The train data infos, the detailed info of each scan is as follows:
  - info\['lidar_points'\]: A dict containing all information related to the lidar points.
    - info\['lidar_points'\]\['axis_align_matrix'\]: The transformation matrix to align the axis.
  - info\['pts_semantic_mask_path'\]: The filename of the semantic mask annotation.
  - info\['pts_instance_mask_path'\]: The filename of the instance mask annotation.
  - info\['instances'\]: A list of dict contains all annotations, each dict contains all annotation information of single instance. For the i-th instance:

- info\['instances'\]: A list of dict contains all annotations, each dict contains all annotation information of single instance. For the i-th instance:
    - info\['instances'\]\[i\]\['bbox_3d'\]: List of 6 numbers representing the axis-aligned 3D bounding box of the instance in depth coordinate system, in (x, y, z, l, w, h) order.
    - info\['instances'\]\[i\]\['bbox_label_3d'\]: The label of each 3d bounding boxes.
- `scannet_infos_val.pkl`: The val data infos, which shares the same format as `scannet_infos_train.pkl`.
- `scannet_infos_test.pkl`: The test data infos, which almost shares the same format as `scannet_infos_train.pkl` except for the lack of annotation.

A typical training pipeline of ScanNet for 3D detection is as follows.

```python
    dict(
        coord_type='DEPTH',
        shift_height=True,
        load_dim=6,
        use_dim=[0, 1, 2]),
    dict(

dict(
    dict(
        with_mask_3d=True,
        with_seg_3d=True),
    dict(type='GlobalAlignment', rotation_axis=2),
    dict(type='PointSegClassMapping'),
    dict(type='PointSample', num_points=40000),
    dict(
        type='RandomFlip3D',
        sync_2d=False,
        flip_ratio_bev_horizontal=0.5,
        flip_ratio_bev_vertical=0.5),
    dict(
        rot_range=[-0.087266, 0.087266],
        scale_ratio_range=[1.0, 1.0],
        shift_height=True),
    dict(
        keys=[
            'points', 'gt_bboxes_3d', 'gt_labels_3d', 'pts_semantic_mask',
            'pts_instance_mask'
        ])
]
```

dict(
        keys=[
        ])
]
```

- `GlobalAlignment`: The previous point cloud would be axis-aligned using the axis-aligned matrix.
- `PointSegClassMapping`: Only the valid category IDs will be mapped to class label IDs like \[0, 18) during training.
- Data augmentation:
  - `PointSample`: downsample the input point cloud.
  - `RandomFlip3D`: randomly flip the input point cloud horizontally or vertically.
  - `GlobalRotScaleTrans`: rotate the input point cloud, usually in the range of \[-5, 5\] (degrees) for ScanNet; then scale the input point cloud, usually by 1.0 for ScanNet (which means no scaling); finally translate the input point cloud, usually by 0 for ScanNet  (which means no translation).

A typical training pipeline of ScanNet for 3D semantic segmentation is as below:

```python
    dict(

```python
    dict(
        shift_height=False,
        use_color=True,
        use_dim=[0, 1, 2, 3, 4, 5]),
    dict(
        with_bbox_3d=False,
        with_label_3d=False,
        with_mask_3d=False,
    dict(
        type='PointSegClassMapping'),
    dict(
        type='IndoorPatchPointSample',
        num_points=num_points,
        block_size=1.5,
        ignore_index=len(class_names),
        use_normalized_coord=False,
        enlarge_size=0.2,
        min_unique_num=None),
    dict(type='NormalizePointsColor', color_mean=None),
    dict(type='Pack3DDetInputs', keys=['points', 'pts_semantic_mask'])
]
```

enlarge_size=0.2,
]
```

- `PointSegClassMapping`: Only the valid category ids will be mapped to class label ids like \[0, 20) during training. Other class ids will be converted to `ignore_index` which equals to `20`.
- `IndoorPatchPointSample`: Crop a patch containing a fixed number of points from input point cloud. `block_size` indicates the size of the cropped block, typically `1.5` for ScanNet.
- `NormalizePointsColor`: Normalize the RGB color values of input point cloud by dividing `255`.

## Metrics

## Metrics

- **Object Detection**: Typically mean Average Precision (mAP) is used for evaluation on ScanNet, e.g. `mAP@0.25` and `mAP@0.5`. In detail, a generic function to compute precision and recall for 3D object detection for multiple classes is called. Please refer to [indoor_eval](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/evaluation/functional/indoor_eval.py) for more details.

  **Note**: As introduced in section `Export ScanNet data`, all ground truth 3D bounding box are axis-aligned, i.e. the yaw is zero. So the yaw target of network predicted 3D bounding box is also zero and axis-aligned 3D Non-Maximum Suppression (NMS), which is regardless of rotation, is adopted during post-processing .

- **Semantic Segmentation**: Typically mean Intersection over Union (mIoU) is used for evaluation on ScanNet. In detail, we first compute IoU for multiple classes and then average them to get mIoU, please refer to [seg_eval](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/evaluation/functional/seg_eval.py).

## Testing and Making a Submission

By default, our codebase evaluates semantic segmentation results on the validation set.
If you would like to test the model performance on the online benchmark, add `--format-only` flag in the evaluation script and change `ann_file=data_root + 'scannet_infos_val.pkl'` to `ann_file=data_root + 'scannet_infos_test.pkl'` in the ScanNet dataset's [config](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/_base_/datasets/scannet-seg.py#L126). Remember to specify the `txt_prefix` as the directory to save the testing results.

Taking PointNet++ (SSG) on ScanNet for example, the following command can be used to do inference on test set:

```
./tools/dist_test.sh configs/pointnet2/pointnet2_ssg_16x2_cosine_200e_scannet-seg.py \
    work_dirs/pointnet2_ssg/latest.pth --format-only \
    --eval-options txt_prefix=work_dirs/pointnet2_ssg/test_submission
```

After generating the results, you can basically compress the folder and upload to the [ScanNet evaluation server](http://kaldir.vc.in.tum.de/scannet_benchmark/semantic_label_3d).

# SNGAN (ICLR'2018)

> [Spectral Normalization for Generative Adversarial Networks](https://openreview.net/forum?id=B1QRgziT-)

One of the challenges in the study of generative adversarial networks is the instability of its training. In this paper, we propose a novel weight normalization technique called spectral normalization to stabilize the training of the discriminator. Our new normalization technique is computationally light and easy to incorporate into existing implementations. We tested the efficacy of spectral normalization on CIFAR10, STL-10, and ILSVRC2012 dataset, and we experimentally confirmed that spectrally normalized GANs (SN-GANs) is capable of generating images of better or equal quality relative to the previous training stabilization techniques.

<img src="https://user-images.githubusercontent.com/28132635/143154496-6a03def4-4507-4d80-a948-89a5b747d916.png"/>
</div>

  <b> Results from our SNGAN-PROJ trained in CIFAR10 and ImageNet</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/28132635/125151484-14220b80-e179-11eb-81f7-9391ccaeb841.png" width="400"/> &nbsp;&nbsp;
  <img src="https://user-images.githubusercontent.com/28132635/127621152-7b7a0f2c-c743-485a-bf2e-2beca849a6e6.png" width="400"/>
</div>

|                                Model                                | Dataset  | Inplace ReLU | disc_step | Total Iters\* |  Iter  |   IS    |   FID   |                                Download                                 |

| :-----------------------------------------------------------------: | :------: | :----------: | :-------: | :-----------: | :----: | :-----: | :-----: | :---------------------------------------------------------------------: |
| [SNGAN_Proj-32x32-woInplaceReLU Best IS](./sngan-proj_woReLUinplace_lr2e-4-ndisc5-1xb64_cifar10-32x32.py) | CIFAR10  |     w/o      |     5     |    500000     | 400000 | 9.6919  | 9.8203  | [ckpt](https://download.openmmlab.com/mmediting/sngan_proj/sngan_proj_cifar10_32_lr-2e-4_b64x1_woReLUinplace_is-iter400000_20210709_163823-902ce1ae.pth) \| [Log](https://download.openmmlab.com/mmediting/sngan_proj/sngan_proj_cifar10_32_lr-2e-4_b64x1_woReLUinplace_20210624_065306_fid-ba0862a0_is-902ce1ae.json) |

| [SNGAN_Proj-32x32-woInplaceReLU Best FID](./sngan-proj_woReLUinplace_lr2e-4-ndisc5-1xb64_cifar10-32x32.py) | CIFAR10  |     w/o      |     5     |    500000     | 490000 | 9.5659  | 8.1158  | [ckpt](https://download.openmmlab.com/mmediting/sngan_proj/sngan_proj_cifar10_32_lr-2e-4_b64x1_woReLUinplace_fid-iter490000_20210709_163329-ba0862a0.pth) \| [Log](https://download.openmmlab.com/mmediting/sngan_proj/sngan_proj_cifar10_32_lr-2e-4_b64x1_woReLUinplace_20210624_065306_fid-ba0862a0_is-902ce1ae.json) |

| [SNGAN_Proj-32x32-wInplaceReLU Best IS](./sngan-proj_wReLUinplace_lr2e-4-ndisc5-1xb64_cifar10-32x32.py) | CIFAR10  |      w       |     5     |    500000     | 490000 | 9.5564  | 8.3462  | [ckpt](https://download.openmmlab.com/mmediting/sngan_proj/sngan_proj_cifar10_32_lr-2e-4_b64x1_wReLUinplace_is-iter490000_20210709_202230-cd863c74.pth) \| [Log](https://download.openmmlab.com/mmediting/sngan_proj/sngan_proj_cifar10_32_lr-2e-4_b64x1_wReLUinplace_20210624_063454_is-cd863c74_fid-191b2648.json) |

| [SNGAN_Proj-32x32-wInplaceReLU Best FID](./sngan-proj_wReLUinplace_lr2e-4-ndisc5-1xb64_cifar10-32x32.py) | CIFAR10  |      w       |     5     |    500000     | 490000 | 9.5564  | 8.3462  | [ckpt](https://download.openmmlab.com/mmediting/sngan_proj/sngan_proj_cifar10_32_lr-2e-4-b64x1_wReLUinplace_fid-iter490000_20210709_203038-191b2648.pth) \| [Log](https://download.openmmlab.com/mmediting/sngan_proj/sngan_proj_cifar10_32_lr-2e-4_b64x1_wReLUinplace_20210624_063454_is-cd863c74_fid-191b2648.json) |

| [SNGAN_Proj-128x128-woInplaceReLU Best IS](./sngan-proj_woReLUinplace_Glr2e-4_Dlr5e-5_ndisc5-2xb128_imagenet1k-128x128.py) | ImageNet |     w/o      |     5     |    1000000    | 952000 | 30.0651 | 33.4682 | [ckpt](https://download.openmmlab.com/mmediting/sngan_proj/sngan_proj_imagenet1k_128_Glr2e-4_Dlr5e-5_ndisc5_b128x2_woReLUinplace_is-iter952000_20210730_132027-9c884a21.pth) \| [Log](https://download.openmmlab.com/mmediting/sngan_proj/sngan_proj_imagenet1k_128_Glr2e-4_Dlr5e-5_ndisc5_b128x2_woReLUinplace_20210730_131424_fid-061bf803_is-9c884a21.json) |

| [SNGAN_Proj-128x128-woInplaceReLU Best FID](./sngan-proj_woReLUinplace_Glr2e-4_Dlr5e-5_ndisc5-2xb128_imagenet1k-128x128.py) | ImageNet |     w/o      |     5     |    1000000    | 989000 | 29.5779 | 32.6193 | [ckpt](https://download.openmmlab.com/mmediting/sngan_proj/sngan_proj_imagenet1k_128_Glr2e-4_Dlr5e-5_ndisc5_b128x2_woReLUinplace_fid-iter988000_20210730_131424-061bf803.pth) \| [Log](https://download.openmmlab.com/mmediting/sngan_proj/sngan_proj_imagenet1k_128_Glr2e-4_Dlr5e-5_ndisc5_b128x2_woReLUinplace_20210730_131424_fid-061bf803_is-9c884a21.json) |

| [SNGAN_Proj-128x128-wInplaceReLU Best IS](./sngan-proj_wReLUinplace_Glr2e-4_Dlr5e-5_ndisc5-2xb128_imagenet1k-128x128.py) | ImageNet |      w       |     5     |    1000000    | 944000 | 28.1799 | 34.3383 | [ckpt](https://download.openmmlab.com/mmediting/sngan_proj/sngan_proj_imagenet1k_128_Glr2e-4_Dlr5e-5_ndisc5_b128x2_wReLUinplace_is-iter944000_20210730_132714-ca0ccd07.pth) \| [Log](https://download.openmmlab.com/mmediting/sngan_proj/sngan_proj_imagenet1k_128_Glr2e-4_Dlr5e-5_ndisc5_b128x2_wReLUinplace_20210730_132401_fid-9a682411_is-ca0ccd07.json) |

| [SNGAN_Proj-128x128-wInplaceReLU Best FID](./sngan-proj_wReLUinplace_Glr2e-4_Dlr5e-5_ndisc5-2xb128_imagenet1k-128x128.py) | ImageNet |      w       |     5     |    1000000    | 988000 | 27.7948 | 33.4821 | [ckpt](https://download.openmmlab.com/mmediting/sngan_proj/sngan_proj_imagenet1k_128_Glr2e-4_Dlr5e-5_ndisc5_b128x2_wReLUinplace_fid-iter988000_20210730_132401-9a682411.pth) \| [Log](https://download.openmmlab.com/mmediting/sngan_proj/sngan_proj_imagenet1k_128_Glr2e-4_Dlr5e-5_ndisc5_b128x2_wReLUinplace_20210730_132401_fid-9a682411_is-ca0ccd07.json) |

```
total_iters (biggan/pytorch studio gan) = our_total_iters / disc_step
```

```
```

|           Model           | Dataset  | Inplace ReLU | disc_step | Total Iters | IS (Our Pipeline) | FID (Our Pipeline) | IS (StudioGAN) | FID (StudioGAN) |           Download           |           Original Download link            |
| :-----------------------: | :------: | :----------: | :-------: | :---------: | :---------------: | :----------------: | :------------: | :-------------: | :--------------------------: | :-----------------------------------------: |

| [SAGAN_Proj-32x32 StudioGAN](./sngan-proj-cvt-studioGAN_cifar10-32x32.py) | CIFAR10  |      w       |     5     |   100000    |       9.372       |      10.2011       |     8.677      |     13.248      | [model](https://download.openmmlab.com/mmediting/sngan_proj/sngan_cifar10_convert-studio-rgb_20210709_111346-2979202d.pth) | [model](https://drive.google.com/drive/folders/16s5Cr-V-NlfLyy_uyXEkoNxLBt-8wYSM) |
| [SAGAN_Proj-128x128 StudioGAN](./sngan-proj-cvt-studioGAN_imagenet1k-128x128.py) | ImageNet |      w       |     2     |   1000000   |      30.218       |      29.8199       |     32.247     |     26.792      | [model](https://download.openmmlab.com/mmediting/sngan_proj/sngan_imagenet1k_convert-studio-rgb_20210709_111406-877b1130.pth) | [model](https://drive.google.com/drive/folders/1Ek2wAMlxpajL_M8aub4DKQ9B313K8XhS) |

```

```

```latex
@inproceedings{miyato2018spectral,
  title={Spectral Normalization for Generative Adversarial Networks},
  author={Miyato, Takeru and Kataoka, Toshiki and Koyama, Masanori and Yoshida, Yuichi},

  url={https://openreview.net/forum?id=B1QRgziT-},
}
```

# build sdk

1. open windows powerShell with administrator privileges
   set-ExecutionPolicy RemoteSigned

2. installed opencv (you can skip this step if you have installed it)
   in sdk folder:

   `.\install_opencv.ps1`

3. set environment variable and path

   `. .\set_env.ps1`

   if you use sdk cuda version:

   (**you have to additionally install cuda and cudnn if use sdk cuda version**)

   (**may need to set CUDNN environment variable point to cudnn root folder if use sdk cuda version**)

4. build sdk

   `. .\build_sdk.ps1` \
   (if you installed opencv by install_opencv.ps1)

   or

   `. .\build_sdk.ps1 "path/to/folder/of/OpenCVConfig.cmake"` \
   (if you installed opencv yourself)

   the executable will be generated in:
   `example\cpp\build\Release`

# SAR

> [Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition](https://arxiv.org/abs/1811.00751)

Recognizing irregular text in natural scene images is challenging due to the large variance in text appearance, such as curvature, orientation and distortion. Most existing approaches rely heavily on sophisticated model designs and/or extra fine-grained annotations, which, to some extent, increase the difficulty in algorithm implementation and data collection. In this work, we propose an easy-to-implement strong baseline for irregular scene text recognition, using off-the-shelf neural network components and only word-level annotations. It is composed of a 31-layer ResNet, an LSTM-based encoder-decoder framework and a 2-dimensional attention module. Despite its simplicity, the proposed method is robust and achieves state-of-the-art performance on both regular and irregular scene text recognition benchmarks.

<img src="https://user-images.githubusercontent.com/22607038/142798157-ac68907f-5a8a-473f-a29f-f0532b7fdba0.png"/>
</div>

## Dataset

|  trainset  | instance_num | repeat_num |           source           |
| :--------: | :----------: | :--------: | :------------------------: |
| icdar_2011 |     3567     |     20     |            real            |
| icdar_2013 |     848      |     20     |            real            |
| icdar2015  |     4468     |     20     |            real            |
| coco_text  |    42142     |     20     |            real            |
|   IIIT5K   |     2000     |     20     |            real            |
| SynthText  |   2400000    |     1      |           synth            |
|  SynthAdd  |   1216889    |     1      | synth, 1.6m in [\[1\]](#1) |
|   Syn90k   |   2400000    |     1      |           synth            |

| testset | instance_num |             type              |

| :-----: | :----------: | :---------------------------: |
| IIIT5K  |     3000     |            regular            |
|   SVT   |     647      |            regular            |
|  IC13   |     1015     |            regular            |
|  IC15   |     2077     |           irregular           |
|  SVTP   |     645      | irregular, 639 in [\[1\]](#1) |
|  CT80   |     288      |           irregular           |

|                        Methods                         |  Backbone   |       Decoder        |        | Regular Text |           |     |           | Irregular Text |        |                         download                         |

| :----------------------------------------------------: | :---------: | :------------------: | :----: | :----------: | :-------: | :-: | :-------: | :------------: | :----: | :------------------------------------------------------: |
|                                                        |             |                      | IIIT5K |     SVT      | IC13-1015 |     | IC15-2077 |      SVTP      |  CT80  |                                                          |
| [SAR](/configs/textrecog/sar/sar_r31_parallel_decoder_academic.py) | R31-1/8-1/4 |  ParallelSARDecoder  | 0.9533 |    0.8964    |  0.9369   |     |  0.7602   |     0.8326     | 0.9062 | [model](https://download.openmmlab.com/mmocr/textrecog/sar/sar_resnet31_parallel-decoder_5e_st-sub_mj-sub_sa_real/sar_resnet31_parallel-decoder_5e_st-sub_mj-sub_sa_real_20220915_171910-04eb4e75.pth) \| [log](https://download.openmmlab.com/mmocr/textrecog/sar/sar_resnet31_parallel-decoder_5e_st-sub_mj-sub_sa_real/20220915_171910.log) |

| [SAR-TTA](/configs/textrecog/sar/sar_r31_parallel_decoder_academic.py) | R31-1/8-1/4 |  ParallelSARDecoder  | 0.9510 |    0.8964    |  0.9340   |     |  0.7862   |     0.8372     | 0.9132 |                                                          |
| [SAR](/configs/textrecog/sar/sar_r31_sequential_decoder_academic.py) | R31-1/8-1/4 | SequentialSARDecoder | 0.9553 |    0.9073    |  0.9409   |     |  0.7761   |     0.8093     | 0.8958 | [model](https://download.openmmlab.com/mmocr/textrecog/sar/sar_resnet31_sequential-decoder_5e_st-sub_mj-sub_sa_real/sar_resnet31_sequential-decoder_5e_st-sub_mj-sub_sa_real_20220915_185451-1fd6b1fc.pth) \| [log](https://download.openmmlab.com/mmocr/textrecog/sar/sar_resnet31_sequential-decoder_5e_st-sub_mj-sub_sa_real/20220915_185451.log) |

| [SAR-TTA](/configs/textrecog/sar/sar_r31_sequential_decoder_academic.py) | R31-1/8-1/4 | SequentialSARDecoder | 0.9530 |    0.9073    |  0.9389   |     |  0.8002   |     0.8124     | 0.9028 |                                                          |

```bibtex
@inproceedings{li2019show,
  title={Show, attend and read: A simple and strong baseline for irregular text recognition},
  author={Li, Hui and Wang, Peng and Shen, Chunhua and Zhang, Guyu},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={33},
  number={01},
  pages={8610--8617},
}
```

# STGCN

[Spatial temporal graph convolutional networks for skeleton-based action recognition](https://ojs.aaai.org/index.php/AAAI/article/view/12328)

Dynamics of human body skeletons convey significant information for human action recognition. Conventional approaches for modeling skeletons usually rely on hand-crafted parts or traversal rules, thus resulting in limited expressive power and difficulties of generalization. In this work, we propose a novel model of dynamic skeletons called Spatial-Temporal Graph Convolutional Networks (ST-GCN), which moves beyond the limitations of previous methods by automatically learning both the spatial and temporal patterns from data. This formulation not only leads to greater expressive power but also stronger generalization capability. On two large datasets, Kinetics and NTU-RGBD, it achieves substantial improvements over mainstream methods.

<img src="https://user-images.githubusercontent.com/34324155/142995893-d6618728-072c-46e1-b276-9b88cf21a01c.png" width="800"/>
</div>

|       uniform 100       |    joint     |  8   |  STGCN   |  88.95   |     10 clips     | 3.8G  |  3.1M  | [config](/configs/skeleton/stgcn/stgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d/stgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d_20221129-484a394a.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d/stgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.log) |

|       uniform 100       |     bone     |  8   |  STGCN   |  91.69   |     10 clips     | 3.8G  |  3.1M  | [config](/configs/skeleton/stgcn/stgcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-2d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-2d/stgcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-2d_20221129-c4b44488.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-2d/stgcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-2d.log) |

|       uniform 100       | joint-motion |  8   |  STGCN   |  86.90   |     10 clips     | 3.8G  |  3.1M  | [config](/configs/skeleton/stgcn/stgcn_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-2d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-2d/stgcn_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-2d_20221129-f18eb408.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-2d/stgcn_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-2d.log) |

|       uniform 100       | bone-motion  |  8   |  STGCN   |  87.86   |     10 clips     | 3.8G  |  3.1M  | [config](/configs/skeleton/stgcn/stgcn_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-2d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-2d/stgcn_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-2d_20221129-99c60e2d.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-2d/stgcn_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-2d.log) |
|                         |  two-stream  |      |          |  92.12   |                  |       |        |                                           |                                         |                                        |

|                         | four-stream  |      |          |  92.34   |                  |       |        |                                           |                                         |                                        |

|       uniform 100       |    joint     |  8   |  STGCN   |  88.11   |     10 clips     | 5.7G  |  3.1M  | [config](/configs/skeleton/stgcn/stgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d/stgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d_20221129-850308e1.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d/stgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d.log) |

|       uniform 100       |     bone     |  8   |  STGCN   |  88.76   |     10 clips     | 5.7G  |  3.1M  | [config](/configs/skeleton/stgcn/stgcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-3d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-3d/stgcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-3d_20221129-9c8d2970.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-3d/stgcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-3d.log) |

|       uniform 100       | joint-motion |  8   |  STGCN   |  86.06   |     10 clips     | 5.7G  |  3.1M  | [config](/configs/skeleton/stgcn/stgcn_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-3d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-3d/stgcn_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-3d_20221129-927648ea.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-3d/stgcn_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-3d.log) |

|       uniform 100       | bone-motion  |  8   |  STGCN   |  85.49   |     10 clips     | 5.7G  |  3.1M  | [config](/configs/skeleton/stgcn/stgcn_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-3d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-3d/stgcn_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-3d_20221129-593162ca.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-3d/stgcn_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-3d.log) |
|                         |  two-stream  |      |          |  90.14   |                  |       |        |                                           |                                         |                                        |

|                         | four-stream  |      |          |  90.39   |                  |       |        |                                           |                                         |                                        |

### NTU120_XSub_2D

|       uniform 100       |    joint     |  8   |  STGCN   |  83.19   |     10 clips     | 3.8G  |  3.1M  | [config](/configs/skeleton/stgcn/stgcn_8xb16-joint-u100-80e_ntu120-xsub-keypoint-2d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-joint-u100-80e_ntu120-xsub-keypoint-2d/stgcn_8xb16-joint-u100-80e_ntu120-xsub-keypoint-2d_20221129-612416c6.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-joint-u100-80e_ntu120-xsub-keypoint-2d/stgcn_8xb16-joint-u100-80e_ntu120-xsub-keypoint-2d.log) |

|       uniform 100       |     bone     |  8   |  STGCN   |  83.36   |     10 clips     | 3.8G  |  3.1M  | [config](/configs/skeleton/stgcn/stgcn_8xb16-bone-u100-80e_ntu120-xsub-keypoint-2d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-bone-u100-80e_ntu120-xsub-keypoint-2d/stgcn_8xb16-bone-u100-80e_ntu120-xsub-keypoint-2d_20221129-131e63c3.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-bone-u100-80e_ntu120-xsub-keypoint-2d/stgcn_8xb16-bone-u100-80e_ntu120-xsub-keypoint-2d.log) |

|       uniform 100       | joint-motion |  8   |  STGCN   |  78.87   |     10 clips     | 3.8G  |  3.1M  | [config](/configs/skeleton/stgcn/stgcn_8xb16-joint-motion-u100-80e_ntu120-xsub-keypoint-2d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-joint-motion-u100-80e_ntu120-xsub-keypoint-2d/stgcn_8xb16-joint-motion-u100-80e_ntu120-xsub-keypoint-2d_20221129-7cb38ec2.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-joint-motion-u100-80e_ntu120-xsub-keypoint-2d/stgcn_8xb16-joint-motion-u100-80e_ntu120-xsub-keypoint-2d.log) |

|       uniform 100       | bone-motion  |  8   |  STGCN   |  79.55   |     10 clips     | 3.8G  |  3.1M  | [config](/configs/skeleton/stgcn/stgcn_8xb16-bone-motion-u100-80e_ntu120-xsub-keypoint-2d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-bone-motion-u100-80e_ntu120-xsub-keypoint-2d/stgcn_8xb16-bone-motion-u100-80e_ntu120-xsub-keypoint-2d_20221129-f5b19892.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-bone-motion-u100-80e_ntu120-xsub-keypoint-2d/stgcn_8xb16-bone-motion-u100-80e_ntu120-xsub-keypoint-2d.log) |
|                         |  two-stream  |      |          |  84.84   |                  |       |        |                                           |                                         |                                        |

|                         | four-stream  |      |          |  85.23   |                  |       |        |                                           |                                         |                                        |

### NTU120_XSub_3D

|       uniform 100       |    joint     |  8   |  STGCN   |  82.15   |     10 clips     | 5.7G  |  3.1M  | [config](/configs/skeleton/stgcn/stgcn_8xb16-joint-u100-80e_ntu120-xsub-keypoint-3d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-joint-u100-80e_ntu120-xsub-keypoint-3d/stgcn_8xb16-joint-u100-80e_ntu120-xsub-keypoint-3d_20221129-0484f579.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-joint-u100-80e_ntu120-xsub-keypoint-3d/stgcn_8xb16-joint-u100-80e_ntu120-xsub-keypoint-3d.log) |

|       uniform 100       |     bone     |  8   |  STGCN   |  84.28   |     10 clips     | 5.7G  |  3.1M  | [config](/configs/skeleton/stgcn/stgcn_8xb16-bone-u100-80e_ntu120-xsub-keypoint-3d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-bone-u100-80e_ntu120-xsub-keypoint-3d/stgcn_8xb16-bone-u100-80e_ntu120-xsub-keypoint-3d_20221129-bc007510.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-bone-u100-80e_ntu120-xsub-keypoint-3d/stgcn_8xb16-bone-u100-80e_ntu120-xsub-keypoint-3d.log) |

|       uniform 100       | joint-motion |  8   |  STGCN   |  78.93   |     10 clips     | 5.7G  |  3.1M  | [config](/configs/skeleton/stgcn/stgcn_8xb16-joint-motion-u100-80e_ntu120-xsub-keypoint-3d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-joint-motion-u100-80e_ntu120-xsub-keypoint-3d/stgcn_8xb16-joint-motion-u100-80e_ntu120-xsub-keypoint-3d_20221129-5d54f525.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-joint-motion-u100-80e_ntu120-xsub-keypoint-3d/stgcn_8xb16-joint-motion-u100-80e_ntu120-xsub-keypoint-3d.log) |

|       uniform 100       | bone-motion  |  8   |  STGCN   |  80.02   |     10 clips     | 5.7G  |  3.1M  | [config](/configs/skeleton/stgcn/stgcn_8xb16-bone-motion-u100-80e_ntu120-xsub-keypoint-3d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-bone-motion-u100-80e_ntu120-xsub-keypoint-3d/stgcn_8xb16-bone-motion-u100-80e_ntu120-xsub-keypoint-3d_20221129-3cb0e4e1.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-bone-motion-u100-80e_ntu120-xsub-keypoint-3d/stgcn_8xb16-bone-motion-u100-80e_ntu120-xsub-keypoint-3d.log) |
|                         |  two-stream  |      |          |  85.68   |                  |       |        |                                           |                                         |                                        |

|                         | four-stream  |      |          |  86.19   |                  |       |        |                                           |                                         |                                        |

2. For two-stream fusion, we use **joint : bone = 1 : 1**. For four-stream fusion, we use **joint : joint-motion : bone : bone-motion = 2 : 1 : 2 : 1**. For more details about multi-stream fusion, please refer to this [tutorial](/docs/en/useful_tools.md#multi-stream-fusion.md).

## Train

```shell
```

## Train

```shell
```

Example: train STGCN model on NTU60-2D dataset in a deterministic option with periodic validation.

```shell
python tools/train.py configs/skeleton/stgcn/stgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py \
```

## Test

```shell
```

Example: test STGCN model on NTU60-2D dataset and dump the result to a pickle file.

```shell
python tools/test.py configs/skeleton/stgcn/stgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py \
```

```shell
```

```BibTeX
@inproceedings{yan2018spatial,
  title={Spatial temporal graph convolutional networks for skeleton-based action recognition},
  author={Yan, Sijie and Xiong, Yuanjun and Lin, Dahua},
  booktitle={Thirty-second AAAI conference on artificial intelligence},
}
```

# GHM

> [Gradient Harmonized Single-stage Detector](https://arxiv.org/abs/1811.05181)

Despite the great success of two-stage detectors, single-stage detector is still a more elegant and efficient way, yet suffers from the two well-known disharmonies during training, i.e. the huge difference in quantity between positive and negative examples as well as between easy and hard examples. In this work, we first point out that the essential effect of the two disharmonies can be summarized in term of the gradient. Further, we propose a novel gradient harmonizing mechanism (GHM) to be a hedging for the disharmonies. The philosophy behind GHM can be easily embedded into both classification loss function like cross-entropy (CE) and regression loss function like smooth-L1 (SL1) loss. To this end, two novel loss functions called GHM-C and GHM-R are designed to balancing the gradient flow for anchor classification and bounding box refinement, respectively. Ablation study on MS COCO demonstrates that without laborious hyper-parameter tuning, both GHM-C and GHM-R can bring substantial improvement for single-stage detector. Without any whistles and bells, our model achieves 41.6 mAP on COCO test-dev set which surpasses the state-of-the-art method, Focal Loss (FL) + SL1, by 0.8.

<img src="https://user-images.githubusercontent.com/40661020/143889057-0341f32b-1291-4b9a-8444-52ad266ae709.png"/>
</div>

|    Backbone     |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                       Config                        |                                                                                                                                                           Download                                                                                                                                                            |

| :-------------: | :-----: | :-----: | :------: | :------------: | :----: | :-------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|    R-50-FPN     | pytorch |   1x    |   4.0    |      3.3       |  37.0  |    [config](./retinanet_r50_fpn_ghm-1x_coco.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/ghm/retinanet_ghm_r50_fpn_1x_coco/retinanet_ghm_r50_fpn_1x_coco_20200130-a437fda3.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/ghm/retinanet_ghm_r50_fpn_1x_coco/retinanet_ghm_r50_fpn_1x_coco_20200130_004213.log.json)               |

|    R-101-FPN    | pytorch |   1x    |   6.0    |      4.4       |  39.1  |    [config](./retinanet_r101_fpn_ghm-1x_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/ghm/retinanet_ghm_r101_fpn_1x_coco/retinanet_ghm_r101_fpn_1x_coco_20200130-c148ee8f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/ghm/retinanet_ghm_r101_fpn_1x_coco/retinanet_ghm_r101_fpn_1x_coco_20200130_145259.log.json)             |
| X-101-32x4d-FPN | pytorch |   1x    |   7.2    |      5.1       |  40.7  | [config](./retinanet_x101-32x4d_fpn_ghm-1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/ghm/retinanet_ghm_x101_32x4d_fpn_1x_coco/retinanet_ghm_x101_32x4d_fpn_1x_coco_20200131-e4333bd0.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/ghm/retinanet_ghm_x101_32x4d_fpn_1x_coco/retinanet_ghm_x101_32x4d_fpn_1x_coco_20200131_113653.log.json) |

| X-101-64x4d-FPN | pytorch |   1x    |   10.3   |      5.2       |  41.4  | [config](./retinanet_x101-64x4d_fpn_ghm-1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/ghm/retinanet_ghm_x101_64x4d_fpn_1x_coco/retinanet_ghm_x101_64x4d_fpn_1x_coco_20200131-dd381cef.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/ghm/retinanet_ghm_x101_64x4d_fpn_1x_coco/retinanet_ghm_x101_64x4d_fpn_1x_coco_20200131_113723.log.json) |

```latex
@inproceedings{li2019gradient,
  title={Gradient Harmonized Single-stage Detector},
  author={Li, Buyu and Liu, Yu and Wang, Xiaogang},
  booktitle={AAAI Conference on Artificial Intelligence},
}
```

# Preparing Paired Dataset for Pix2pix

```bibtex
}
```

You can download paired datasets from [here](http://efrosgans.eecs.berkeley.edu/pix2pix/datasets/).
Then, you need to unzip and move corresponding datasets to follow the folder structure shown above. The datasets have been well-prepared by the original authors.

```text
mmagic
├── mmagic
├── tools
├── data
│   │   ├── maps
```

# 添加新数据集

用户可以编写一个继承自 [BasesDataset](https://mmpretrain.readthedocs.io/zh_CN/latest/_modules/mmpretrain/datasets/base_dataset.html#BaseDataset) 的新数据集类，并重载 `load_data_list(self)` 方法，类似 [CIFAR10](https://github.com/open-mmlab/mmpretrain/blob/main/mmpretrain/datasets/cifar.py) 和 [ImageNet](https://github.com/open-mmlab/mmpretrain/blob/main/mmpretrain/datasets/imagenet.py)。

通常，此方法返回一个包含所有样本的列表，其中的每个样本都是一个字典。字典中包含了必要的数据信息，例如 `img` 和 `gt_label`。

假设我们将要实现一个 `Filelist` 数据集，该数据集将使用文件列表进行训练和测试。注释列表的格式如下：

```text
...
```

## 1. 创建数据集类

我们可以在 `mmpretrain/datasets/filelist.py` 中创建一个新的数据集类以加载数据。

```python

        assert isinstance(self.ann_file, str)

assert isinstance(self.ann_file, str)

```

## 2. 添加到库

将新的数据集类加入到 `mmpretrain/datasets/__init__.py` 中：

```python
...

__all__ = [
]
```

### 3. 修改相关配置文件

然后在配置文件中，为了使用 `Filelist`，用户可以按以下方式修改配置

```python
    ...
    )
)
```

```python
    ...
    )
)
```

所有继承 [`BaseDataset`](https://github.com/open-mmlab/mmpretrain/blob/main/mmpretrain/datasets/base_dataset.py) 的数据集类都具有**懒加载**以及**节省内存**的特性，可以参考相关文档 {external+mmengine:doc}`BaseDataset <advanced_tutorials/basedataset>`。

```{note}
如果数据样本时获取的字典中，只包含了 'img_path' 不包含 'img'， 则在 pipeline 中必须包含 'LoadImgFromFile'。
```

# Customize Data Pipelines

1. Write a new transform in a file, e.g., in `my_pipeline.py`. It takes a dict as input and returns a dict.

   ```python

       Args:
       """

   ```

2. Import and use the pipeline in your config file.
   Make sure the import is relative to where your train script is located.

   ```python

custom_imports = dict(imports=['path.to.my_pipeline'], allow_failed_imports=False)

   ]
   ```

3. Visualize the output of your transforms pipeline

   To visualize the output of your transforms pipeline, `tools/misc/browse_dataset.py`
   can help the user to browse a detection dataset (both images and bounding box annotations)
   visually, or save the image to a designated directory. More details can refer to
   [visualization documentation](../user_guides/visualization.md)

# 预训练模型迁移指南

由于在新版本中我们对模型的结构进行了大量的重构和修复，MMOCR 1.x 并不能直接读入旧版的预训练权重。我们在网站上同步更新了所有模型的预训练权重和log，供有需要的用户使用。

此外，我们正在进行针对文本检测任务的权重迁移工具的开发，并计划于近期版本内发布。由于文本识别和关键信息提取模型改动过大，且迁移是有损的，我们暂时不计划作相应支持。如果您有具体的需求，欢迎通过 [Issue](https://github.com/open-mmlab/mmocr/issues) 向我们提问。

# 新闻

- **\[2023.08.10\]** OpenCompass 现已适配 [LMDeploy](https://github.com/InternLM/lmdeploy). 请参考 [评测指南](https://opencompass.readthedocs.io/zh_CN/latest/advanced_guides/evaluation_turbomind.html) 对 **Turbomind** 加速后的模型进行评估.
- **\[2023.08.10\]**  [Qwen-7B](https://github.com/QwenLM/Qwen-7B) 和 [XVERSE-13B](https://github.com/xverse-ai/XVERSE-13B)的评测结果已更新在 OpenCompass [大语言模型评测榜单](https://opencompass.org.cn/leaderboard-llm)!
- **\[2023.08.09\]** 更新更多评测数据集(**CMMLU, TydiQA, SQuAD2.0, DROP**) ，请登录 [大语言模型评测榜单](https://opencompass.org.cn/leaderboard-llm) 查看更多结果! 欢迎添加你的评测数据集到OpenCompass.
- **\[2023.08.07\]** 新增了 [MMBench 评测脚本](tools/eval_mmbench.py) 以支持用户自行获取 [MMBench](https://opencompass.org.cn/MMBench)-dev 的测试结果.
- **\[2023.08.05\]** [GPT-4](https://openai.com/gpt-4) 的评测结果已更新在 OpenCompass [大语言模型评测榜单](https://opencompass.org.cn/leaderboard-llm)!
- **\[2023.07.27\]** 新增了 [CMMLU](https://github.com/haonan-li/CMMLU)! 欢迎更多的数据集加入 OpenCompass.

# 自定义运行时配置

## 自定义优化器设置

优化器相关的配置是由 `optim_wrapper` 管理的，其通常有三个字段：`optimizer`，`paramwise_cfg`，`clip_grad`。更多细节请参考 [OptimWrapper](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/optim_wrapper.html)。如下所示，使用 `AdamW` 作为`优化器`，骨干网络的学习率降低 10 倍，并添加了梯度裁剪。

```python
    # 优化器
        type='AdamW',
        lr=0.0001,
        weight_decay=0.05,
        eps=1e-8,
        betas=(0.9, 0.999)),

    # 参数级学习率及权重衰减系数设置
            'backbone': dict(lr_mult=0.1, decay_mult=1.0),
        },
        norm_decay_mult=0.0),

    # 梯度裁剪
    clip_grad=dict(max_norm=0.01, norm_type=2))
```

### 自定义 PyTorch 支持的优化器

我们已经支持使用所有 PyTorch 实现的优化器，且唯一需要修改的地方就是改变配置文件中的 `optim_wrapper` 字段中的 `optimizer` 字段。例如，如果您想使用 `Adam`（注意这样可能会使性能大幅下降），您可以这样修改：

```python
    optimizer=dict(type='Adam', lr=0.0003, weight_decay=0.0001))
```

```python
```

为了修改模型的学习率，用户只需要修改 `optimizer` 中的 `lr` 字段。用户可以根据 PyTorch 的 [API 文档](https://pytorch.org/docs/stable/optim.html?highlight=optim#module-torch.optim)直接设置参数。

### 自定义并实现优化器

#### 1. 定义新的优化器

一个自定义优化器可以按照如下过程定义：

假设您想要添加一个叫 `MyOptimizer` 的，拥有参数 `a`，`b` 和 `c` 的优化器，您需要创建一个叫做 `mmdet3d/engine/optimizers` 的目录。接下来，应该在目录下某个文件中实现新的优化器，比如 `mmdet3d/engine/optimizers/my_optimizer.py`：

```python

from mmdet3d.registry import OPTIMIZERS

        pass
```

#### 2. 将优化器添加到注册器

为了找到上述定义的优化器模块，该模块首先需要被引入主命名空间。有两种实现方法：

- 修改 `mmdet3d/engine/optimizers/__init__.py` 导入该模块。

  新定义的模块应该在 `mmdet3d/engine/optimizers/__init__.py` 中被导入，从而被找到并且被添加到注册器中：

  ```python
  from .my_optimizer import MyOptimizer
  ```

- 在配置中使用 `custom_imports` 来人工导入新优化器。

  ```python

新定义的模块应该在 `mmdet3d/engine/optimizers/__init__.py` 中被导入，从而被找到并且被添加到注册器中：

  ```python
  ```

  ```python
  custom_imports = dict(imports=['mmdet3d.engine.optimizers.my_optimizer'], allow_failed_imports=False)
  ```

  模块 `mmdet3d.engine.optimizers.my_optimizer` 会在程序开始被导入，且 `MyOptimizer` 类在那时会自动被注册。注意到应该只有包含 `MyOptimizer` 类的包被导入。`mmdet3d.engine.optimizers.my_optimizer.MyOptimizer`**不能**被直接导入。

  事实上，用户可以在这种导入的方法中使用完全不同的文件目录结构，只要保证根目录能在 `PYTHONPATH` 中被定位。

接下来您可以在配置文件的 `optimizer` 字段中使用 `MyOptimizer`。在配置文件中，优化器在 `optimizer` 字段中以如下方式定义：

```python
    optimizer=dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001))
```

为了使用您自己的优化器，该字段可以改为：

```python
```

### 自定义优化器封装构造器

```

```python
```

部分模型可能会拥有一些参数专属的优化器设置，比如 BatchNorm 层的权重衰减 (weight decay)。用户可以通过自定义优化器封装构造器来对那些细粒度的参数进行调优。

```python

from mmdet3d.registry import OPTIM_WRAPPER_CONSTRUCTORS

class MyOptimizerWrapperConstructor(DefaultOptimWrapperConstructor):

                 optim_wrapper_cfg: dict,
                 paramwise_cfg: Optional[dict] = None):
        pass

    def __call__(self, model: nn.Module) -> OptimWrapper:

        return optim_wrapper
```

默认优化器封装构造器在[这里](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/optimizer/default_constructor.py#L18)实现。这部分代码也可以用作新优化器封装构造器的模板。

### 额外的设置

return optim_wrapper
```

### 额外的设置

没有在优化器部分实现的技巧应该通过优化器封装构造器或者钩子来实现（比如逐参数的学习率设置）。我们列举了一些常用的可以稳定训练过程或者加速训练的设置。我们欢迎提供更多类似设置的 PR 和 issue。

- __使用梯度裁剪 (gradient clip) 来稳定训练过程__：一些模型依赖梯度裁剪技术来裁剪训练中的梯度，以稳定训练过程。举例如下：

  ```python
      _delete_=True, clip_grad=dict(max_norm=35, norm_type=2))
  ```

  如果您的配置继承了一个已经设置了 `optim_wrapper` 的基础配置，那么您可能需要 `_delete_=True` 字段来覆盖基础配置中无用的设置。更多细节请参考[配置文档](https://mmdetection3d.readthedocs.io/zh_CN/dev-1.x/user_guides/config.html)。

```

- __使用动量调度器 (momentum scheduler) 来加速模型收敛__：我们支持用动量调度器来根据学习率更改模型的动量，这样可以使模型更快地收敛。动量调度器通常和学习率调度器一起使用，例如，如下配置文件在 [3D 检测](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/schedules/cyclic-20e.py)中被用于加速模型收敛。更多细节请参考 [CosineAnnealingLR](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py#L43) 和 [CosineAnnealingMomentum](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/momentum_scheduler.py#L71) 的实现方法。

  ```python
  param_scheduler = [
      # 学习率调度器
      # 在前 8 个 epoch，学习率从 0 升到 lr * 10
      # 在接下来 12 个 epoch，学习率从 lr * 10 降到 lr * 1e-4
      dict(
          type='CosineAnnealingLR',
          T_max=8,
          eta_min=lr * 10,
          begin=0,
          end=8,
          by_epoch=True,
          convert_to_iter_based=True),

dict(
          T_max=8,
          begin=0,
          end=8,
      dict(
          T_max=12,
          eta_min=lr * 1e-4,
          begin=8,
          end=20,
      # 动量调度器
      # 在前 8 个 epoch，动量从 0 升到 0.85 / 0.95
      # 在接下来 12 个 epoch，动量从 0.85 / 0.95 升到 1
      dict(
          type='CosineAnnealingMomentum',
          T_max=8,
          eta_min=0.85 / 0.95,
          begin=0,
          end=8,
      dict(
          T_max=12,
          eta_min=1,
          begin=8,
          end=20,
          convert_to_iter_based=True)
  ]
  ```

## 自定义训练调度

T_max=12,
          begin=8,
          end=20,
  ]
  ```

## 自定义训练调度

默认情况下我们使用阶梯式学习率衰减的 1 倍训练调度，这会调用 MMEngine 中的 [`MultiStepLR`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py#L144)。我们在[这里](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py)支持了很多其他学习率调度，比如`余弦退火`和`多项式衰减`调度。下面是一些样例：

- 多项式衰减调度：

  ```python
      dict(
          type='PolyLR',
          power=0.9,
          eta_min=1e-4,
          begin=0,
          end=8,
          by_epoch=True)]
  ```

- 余弦退火调度：

  ```python
      dict(
          T_max=8,
          eta_min=lr * 1e-5,
          begin=0,
          end=8,
  ```

## 自定义训练循环控制器

默认情况下，我们在 `train_cfg` 中使用 `EpochBasedTrainLoop`，并在每一个训练 epoch 完成后进行一次验证，如下所示：

```python

begin=0,
          end=8,
  ```

```python
train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=12, val_begin=1, val_interval=1)
```

事实上，[`IterBasedTrainLoop`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L185) 和 [`EpochBasedTrainLoop`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L18) 都支持动态间隔验证，如下所示：

```python
# 在第 365001 次迭代之前，我们每隔 5000 次迭代验证一次。
# 在第 365000 次迭代之后，我们每隔 368750 次迭代验证一次，
# 这意味着我们在训练结束后进行验证。

interval = 5000
max_iters = 368750
dynamic_intervals = [(max_iters // interval * interval + 1, max_iters)]
    type='IterBasedTrainLoop',
    max_iters=max_iters,
    val_interval=interval,
    dynamic_intervals=dynamic_intervals)
```

## 自定义钩子

### 自定义并实现钩子

#### 1. 实现一个新钩子

```

## 自定义钩子

MMEngine 提供了一些实用的[钩子](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/hook.html)，但有些场合用户可能需要实现一个新的钩子。在 v1.1.0rc0 之后，MMDetection3D 在训练时支持基于 MMEngine 自定义钩子。因此用户可以直接在 mmdet3d 或者基于 mmdet3d 的代码库中实现钩子并通过更改训练配置来使用钩子。这里我们给出一个在 mmdet3d 中创建并使用新钩子的例子。

```python
from mmengine.hooks import Hook

from mmdet3d.registry import HOOKS

    def before_run(self, runner) -> None:

    def after_run(self, runner) -> None:

    def before_train(self, runner) -> None:

    def after_train(self, runner) -> None:

    def before_train_epoch(self, runner) -> None:

    def after_train_epoch(self, runner) -> None:

    def before_train_iter(self,
                          runner,
                          batch_idx: int,

def after_train_epoch(self, runner) -> None:

                          runner,
                          data_batch: DATA_BATCH = None) -> None:

    def after_train_iter(self,
                         runner,
                         batch_idx: int,
                         data_batch: DATA_BATCH = None,
                         outputs: Optional[dict] = None) -> None:
```

用户需要根据钩子的功能指定钩子在每个训练阶段时的行为，具体包括如下阶段：`before_run`，`after_run`，`before_train`，`after_train`，`before_train_epoch`，`after_train_epoch`，`before_train_iter`，和 `after_train_iter`。有更多的位点可以插入钩子，详情可参考 [base hook class](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/hook.py#L9)。

#### 2. 注册新钩子

接下来我们需要导入 `MyHook`。假设新钩子位于文件 `mmdet3d/engine/hooks/my_hook.py` 中，有两种实现方法：

- 修改 `mmdet3d/engine/hooks/__init__.py` 导入该模块。

  新定义的模块应该在 `mmdet3d/engine/hooks/__init__.py` 中被导入，从而被找到并且被添加到注册器中：

  ```python
  from .my_hook import MyHook
  ```

  ```python
  ```

- 在配置中使用 `custom_imports` 来人为地导入新钩子。

  ```python
  custom_imports = dict(imports=['mmdet3d.engine.hooks.my_hook'], allow_failed_imports=False)
  ```

#### 3. 更改配置文件

```python
]
```

您可以将字段 `priority` 设置为 `'NORMAL'` 或者 `'HIGHEST'` 来设置钩子的优先级，如下所示：

```python
]
```

默认情况下，注册阶段钩子的优先级为 `'NORMAL'`。

### 使用 MMDetection3D 中实现的钩子

如果 MMDetection3D 中已经实现了该钩子，您可以直接通过更改配置文件来使用该钩子。

#### 例子：`DisableObjectSampleHook`

我们实现了一个名为 [DisableObjectSampleHook](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/engine/hooks/disable_object_sample_hook.py) 的自定义钩子在训练阶段达到指定 epoch 后禁用 `ObjectSample` 增强策略。

如果有需要的话我们可以在配置文件中设置它：

```python

```python
custom_hooks = [dict(type='DisableObjectSampleHook', disable_after_epoch=15)]
```

### 更改默认的运行时钩子

有一些常用的钩子通过 `default_hooks` 注册，它们是：

- `IterTimerHook`：该钩子用来记录加载数据的时间 'data_time' 和模型训练一步的时间 'time'。
- `LoggerHook`：该钩子用来从`执行器（Runner）`的不同组件收集日志并将其写入终端，json 文件，tensorboard 和 wandb 等。
- `ParamSchedulerHook`：该钩子用来更新优化器中的一些超参数，例如学习率和动量。
- `CheckpointHook`：该钩子用来定期地保存检查点。
- `DistSamplerSeedHook`：该钩子用来设置采样和批采样的种子。
- `Det3DVisualizationHook`：该钩子用来可视化验证和测试过程的预测结果。

`IterTimerHook`，`ParamSchedulerHook` 和 `DistSamplerSeedHook` 都很简单，通常不需要修改，因此此处我们将介绍如何使用 `LoggerHook`，`CheckpointHook` 和 `Det3DVisualizationHook`。

#### CheckpointHook

除了定期地保存检查点，[`CheckpointHook`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L18) 提供了其它的可选项例如 `max_keep_ckpts`，`save_optimizer` 等。用户可以设置 `max_keep_ckpts` 只保存少量的检查点或者通过 `save_optimizer` 决定是否保存优化器的状态。参数的更多细节请参考[此处](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L18)。

```python
        max_keep_ckpts=3,
        save_optimizer=True))
```

#### LoggerHook

`LoggerHook` 允许设置日志记录间隔。详细介绍可参考[文档](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/logger_hook.py#L19)。

```python
default_hooks = dict(logger=dict(type='LoggerHook', interval=50))
```

#### Det3DVisualizationHook

`Det3DVisualizationHook` 使用 `DetLocalVisualizer` 来可视化预测结果，`Det3DLocalVisualizer` 支持不同的后端，例如 `TensorboardVisBackend` 和 `WandbVisBackend`（更多细节请参考[文档](https://github.com/open-mmlab/mmengine/blob/main/mmengine/visualization/vis_backend.py)）。用户可以添加多个后端来进行可视化，如下所示。

```python
    visualization=dict(type='Det3DVisualizationHook', draw=True))

```

# SlowFast

[SlowFast Networks for Video Recognition](https://openaccess.thecvf.com/content_ICCV_2019/html/Feichtenhofer_SlowFast_Networks_for_Video_Recognition_ICCV_2019_paper.html)

We present SlowFast networks for video recognition. Our model involves (i) a Slow pathway, operating at low frame rate, to capture spatial semantics, and (ii) a Fast pathway, operating at high frame rate, to capture motion at fine temporal resolution. The Fast pathway can be made very lightweight by reducing its channel capacity, yet can learn useful temporal information for video recognition. Our models achieve strong performance for both action classification and detection in video, and large improvements are pin-pointed as contributions by our SlowFast concept. We report state-of-the-art accuracy on major video recognition benchmarks, Kinetics, Charades and AVA.

<img src="https://user-images.githubusercontent.com/34324155/143044111-94676f64-7ba8-4081-9011-f8054bed7030.png" width="800"/>
</div>

| frame sampling strategy |    scheduler     | resolution | gpus |       backbone       | pretrain | top1 acc | top5 acc | testing protocol  | FLOPs | params |           config           |           ckpt            |           log            |
| :---------------------: | :--------------: | :--------: | :--: | :------------------: | :------: | :------: | :------: | :---------------: | :---: | :----: | :------------------------: | :-----------------------: | :----------------------: |

|         4x16x1          |  Linear+Cosine   |  224x224   |  8   |       ResNet50       |   None   |  75.55   |  92.35   | 10 clips x 3 crop | 36.3G | 34.5M  | [config](/configs/recognition/slowfast/slowfast_r50_8xb8-4x16x1-256e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/slowfast/slowfast_r50_8xb8-4x16x1-256e_kinetics400-rgb/slowfast_r50_8xb8-4x16x1-256e_kinetics400-rgb_20220901-701b0f6f.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/slowfast/slowfast_r50_8xb8-4x16x1-256e_kinetics400-rgb/slowfast_r50_8xb8-4x16x1-256e_kinetics400-rgb.log) |

|          8x8x1          |  Linear+Cosine   |  224x224   |  8   |       ResNet50       |   None   |  76.80   |  92.99   | 10 clips x 3 crop | 66.1G | 34.6M  | [config](/configs/recognition/slowfast/slowfast_r50_8xb8-8x8x1-256e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/slowfast/slowfast_r50_8xb8-8x8x1-256e_kinetics400-rgb/slowfast_r50_8xb8-8x8x1-256e_kinetics400-rgb_20220818-1cb6dfc8.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/slowfast/slowfast_r50_8xb8-8x8x1-256e_kinetics400-rgb/slowfast_r50_8xb8-8x8x1-256e_kinetics400-rgb.log) |

|          8x8x1          | Linear+MultiStep |  224x224   |  8   |       ResNet50       |   None   |  76.65   |  92.86   | 10 clips x 3 crop | 66.1G | 34.6M  | [config](/configs/recognition/slowfast/slowfast_r50_8xb8-8x8x1-steplr-256e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/slowfast/slowfast_r50_8xb8-8x8x1-steplr-256e_kinetics400-rgb/slowfast_r50_8xb8-8x8x1-steplr-256e_kinetics400-rgb_20220818-b62a501f.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/slowfast/slowfast_r50_8xb8-8x8x1-steplr-256e_kinetics400-rgb/slowfast_r50_8xb8-8x8x1-steplr-256e_kinetics400-rgb.log) |

|          8x8x1          |  Linear+Cosine   |  224x224   |  8   |      ResNet101       |   None   |  78.65   |  93.88   | 10 clips x 3 crop | 126G  | 62.9M  | [config](/configs/recognition/slowfast/slowfast_r101_8xb8-8x8x1-256e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/slowfast/slowfast_r101_8xb8-8x8x1-256e_kinetics400-rgb/slowfast_r101_8xb8-8x8x1-256e_kinetics400-rgb_20220818-9c0e09bd.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/slowfast/slowfast_r101_8xb8-8x8x1-256e_kinetics400-rgb/slowfast_r101_8xb8-8x8x1-256e_kinetics400-rgb.log) |

|         4x16x1          |  Linear+Cosine   |  224x224   |  32  | ResNet101 + ResNet50 |   None   |  77.03   |  92.99   | 10 clips x 3 crop | 64.9G | 62.4M  | [config](/configs/recognition/slowfast/slowfast_r101-r50_32xb8-4x16x1-256e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/slowfast/slowfast_r101-r50_32xb8-4x16x1-256e_kinetics400-rgb/slowfast_r101-r50_32xb8-4x16x1-256e_kinetics400-rgb_20220901-a77ac3ee.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/slowfast/slowfast_r101-r50_32xb8-4x16x1-256e_kinetics400-rgb/slowfast_r101-r50_32xb8-4x16x1-256e_kinetics400-rgb.log) |

## Train

```shell
```

Example: train SlowFast model on Kinetics-400 dataset in a deterministic option with periodic validation.

```shell

```shell
```

```shell
python tools/train.py configs/recognition/slowfast/slowfast_r50_8xb8-4x16x1-256e_kinetics400-rgb.py \
```

## Test

```shell
```

Example: test SlowFast model on Kinetics-400 dataset and dump the result to a pkl file.

```shell
python tools/test.py configs/recognition/slowfast/slowfast_r50_8xb8-4x16x1-256e_kinetics400-rgb.py \
```

```

```BibTeX
@inproceedings{feichtenhofer2019slowfast,
  title={Slowfast networks for video recognition},
  author={Feichtenhofer, Christoph and Fan, Haoqi and Malik, Jitendra and He, Kaiming},
  pages={6202--6211},
}
```

# 数据变换与流水线

在 MMOCR 的设计中，数据集的构建与数据准备是相互解耦的。也就是说，[`OCRDataset`](mmocr.datasets.ocr_dataset.OCRDataset) 等数据集构建类负责完成标注文件的读取与解析功能；而数据变换方法（Data Transforms）则进一步实现了数据预处理、数据增强、数据格式化等相关功能。目前，如下表所示，MMOCR 中共实现了 5 类数据变换方法：

|                |                                                                       |                                                                     |
| -------------- | --------------------------------------------------------------------- | ------------------------------------------------------------------- |
| 数据变换类型   | 对应文件                                                              | 功能说明                                                            |
| 数据读取       | loading.py                                                            | 实现了不同格式数据的读取功能。                                      |
| 数据格式化     | formatting.py                                                         | 完成不同任务所需数据的格式化功能。                                  |

| 跨库数据适配器 | adapters.py                                                           | 负责 OpenMMLab 项目内跨库调用的数据格式转换功能。                   |
| 数据增强       | ocr_transforms.py<br>textdet_transforms.py<br>textrecog_transforms.py | 实现了不同任务下的各类数据增强方法。                                |
| 包装类         | wrappers.py                                                           | 实现了对 ImgAug 等常用算法库的包装，使其适配 MMOCR 的内部数据格式。 |

由于每一个数据变换类之间都是相互独立的，因此，在约定好固定的数据存储字段后，我们可以便捷地采用任意的数据变换组合来构建数据流水线（Pipeline）。如下图所示，在 MMOCR 中，一个典型的训练数据流水线主要由**数据读取**、**图像增强**以及**数据格式化**三部分构成，用户只需要在配置文件中定义相关的数据流水线列表，并指定具体所需的数据变换类及其参数即可：

![Flowchart](https://user-images.githubusercontent.com/45810070/192265777-0f1f4b52-1a17-4b1a-8c1b-ba6ef17289d6.jpg)

</div>

```python
train_pipeline_r18 = [
    # 数据读取（图像）
    dict(

</div>

```python
    # 数据读取（图像）
    dict(
        color_type='color_ignore_orientation'),
    # 数据读取（标注）
    dict(
    ),
    # 使用 ImgAug 作数据增强
    dict(
        type='ImgAugWrapper',
        args=[['Fliplr', 0.5],
              dict(cls='Affine', rotate=[-10, 10]), ['Resize', [0.5, 3.0]]]),
    # 使用 MMOCR 内置的图像增强
    # 数据格式化
    dict(
]
```

```{tip}
更多有关数据流水线配置的教程可见[配置文档](../user_guides/config.md#数据流水线配置)。下面，我们将简单介绍 MMOCR 中已支持的数据变换类型。
```

]
```

```{tip}
```

对于每一个数据变换方法，MMOCR 都严格按照文档字符串（docstring）规范在源码中提供了详细的代码注释。例如，每一个数据转换类的头部我们都注释了 “需求字段”（`Required keys`）， “修改字段”（`Modified Keys`）与 “添加字段”（`Added Keys`）。其中，“需求字段”代表该数据转换方法对于输入数据所需包含字段的强制需求，而“修改字段”与“添加字段”则表明该方法可能会在原有数据基础之上修改或添加的字段。例如，`LoadImageFromFile` 实现了图片的读取功能，其需求字段为图像的存储路径 `img_path`，而修改字段则包括了读入的图像信息 `img`，以及图片当前尺寸 `img_shape`，图片原始尺寸 `ori_shape` 等图片属性。

```python
class LoadImageFromFile(MMCV_LoadImageFromFile):
    # 在每一个数据变换方法的头部，我们都提供了详细的代码注释。
    """Load an image from file.

    - img_path

    - img
    - img_shape
    - ori_shape
    """
```

```{note}
在 MMOCR 的数据流水线中，图像及标签等信息被统一保存在字典中。通过统一的字段名，我们可以在不同的数据变换方法间灵活地传递数据。因此，了解 MMOCR 中常用的约定字段名是非常重要的。
```

为方便用户查询，下表列出了 MMOCR 中各数据转换（Data Transform）类常用的字段约定和说明。

- ori_shape
    """
```

```{note}
```

|                  |                                   |                                                                                                                                      |
| ---------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| 字段             | 类型                              | 说明                                                                                                                                 |
| img              | `np.array(dtype=np.uint8)`        | 图像信息，形状为 `(h, w, c)`。                                                                                                       |

| img_shape        | `tuple(int, int)`                 | 当前图像尺寸 `(h, w)`。                                                                                                              |
| ori_shape        | `tuple(int, int)`                 | 图像在初始化时的尺寸 `(h, w)`。                                                                                                      |
| scale            | `tuple(int, int)`                 | 存放用户在 Resize 系列数据变换（Transform）中指定的目标图像尺寸 `(h, w)`。注意：该值未必与变换后的实际图像尺寸相符。                 |
| scale_factor     | `tuple(float, float)`             | 存放用户在 Resize 系列数据变换（Transform）中指定的目标图像缩放因子 `(w_scale, h_scale)`。注意：该值未必与变换后的实际图像尺寸相符。 |

| keep_ratio       | `bool`                            | 是否按等比例对图像进行缩放。                                                                                                         |
| flip             | `bool`                            | 图像是否被翻转。                                                                                                                     |
| flip_direction   | `str`                             | 翻转方向。可选项为 `horizontal`, `vertical`, `diagonal`。                                                                            |
| gt_bboxes        | `np.array(dtype=np.float32)`      | 文本实例边界框的真实标签。                                                                                                           |

| gt_polygons      | `list[np.array(dtype=np.float32)` | 文本实例边界多边形的真实标签。                                                                                                       |
| gt_bboxes_labels | `np.array(dtype=np.int64)`        | 文本实例对应的类别标签。在 MMOCR 中通常为 0，代指 "text" 类别。                                                                      |
| gt_texts         | `list[str]`                       | 与文本实例对应的字符串标注。                                                                                                         |
| gt_ignored       | `np.array(dtype=np.bool_)`        | 是否要在计算目标时忽略该实例（用于检测任务中）。                                                                                     |

## 数据读取 - loading.py

数据读取类主要实现了不同文件格式、后端读取图片及加载标注信息的功能。目前，MMOCR 内部共实现了以下数据读取类的 Data Transforms：

|                    |                                                           |                                                                |                                                                 |
| ------------------ | --------------------------------------------------------- | -------------------------------------------------------------- | --------------------------------------------------------------- |
| 数据转换类名称     | 需求字段                                                  | 修改/添加字段                                                  | 说明                                                            |
| LoadImageFromFile  | `img_path`                                                | `img`<br>`img_shape`<br>`ori_shape`                            | 从图片路径读取图片，支持多种文件存储后端（如 `disk`, `http`, `petrel` 等）及图片解码后端（如 `cv2`, `turbojpeg`, `pillow`, `tifffile`等）。 |

| LoadOCRAnnotations | `bbox`<br>`bbox_label`<br>`polygon`<br>`ignore`<br>`text` | `gt_bboxes`<br>`gt_bboxes_labels`<br>`gt_polygons`<br>`gt_ignored`<br>`gt_texts` | 解析 OCR 任务所需的标注信息。                                   |
| LoadKIEAnnotations | `bboxes` `bbox_labels` `edge_labels`<br>`texts`           | `gt_bboxes`<br>`gt_bboxes_labels`<br>`gt_edge_labels`<br>`gt_texts`<br>`ori_shape` | 解析 KIE 任务所需的标注信息。                                   |

## 数据增强 - xxx_transforms.py

数据增强是文本检测、识别等任务中必不可少的流程之一。目前，MMOCR 中共实现了数十种文本领域内常用的数据增强模块，依据其任务类型，分别为通用 OCR 数据增强模块 [ocr_transforms.py](/mmocr/datasets/transforms/ocr_transforms.py)，文本检测数据增强模块 [textdet_transforms.py](/mmocr/datasets/transforms/textdet_transforms.py)，以及文本识别数据增强模块 [textrecog_transforms.py](/mmocr/datasets/transforms/textrecog_transforms.py)。

具体而言，`ocr_transforms.py` 中实现了随机剪裁、随机旋转等各任务通用的数据增强模块：

|                |                                                                |                                                                |                                                                |
| -------------- | -------------------------------------------------------------- | -------------------------------------------------------------- | -------------------------------------------------------------- |
| 数据转换类名称 | 需求字段                                                       | 修改/添加字段                                                  | 说明                                                           |
| RandomCrop     | `img`<br>`gt_bboxes`<br>`gt_bboxes_labels`<br>`gt_polygons`<br>`gt_ignored`<br>`gt_texts` (optional) | `img`<br>`img_shape`<br>`gt_bboxes`<br>`gt_bboxes_labels`<br>`gt_polygons`<br>`gt_ignored`<br>`gt_texts` (optional) | 随机裁剪，并确保裁剪后的图片至少包含一个文本实例。可选参数为 `min_side_ratio`，用以控制裁剪图片的短边占原始图片的比例，默认值为 `0.4`。 |

| RandomRotate   | `img`<br>`img_shape`<br>`gt_bboxes` (optional)<br>`gt_polygons` (optional) | `img`<br>`img_shape`<br>`gt_bboxes` (optional)<br>`gt_polygons` (optional)<br>`rotated_angle` | 随机旋转，并可选择对旋转后图像的黑边进行填充。                 |
|                |                                                                |                                                                |                                                                |

`textdet_transforms.py` 则实现了文本检测任务中常用的数据增强模块：

|                   |                                       |                                                                     |                                                                      |
| ----------------- | ------------------------------------- | ------------------------------------------------------------------- | -------------------------------------------------------------------- |

| 数据转换类名称    | 需求字段                              | 修改/添加字段                                                       | 说明                                                                 |
| RandomFlip        | `img`<br>`gt_bboxes`<br>`gt_polygons` | `img`<br>`gt_bboxes`<br>`gt_polygons`<br>`flip`<br>`flip_direction` | 随机翻转，支持水平、垂直和对角三种方向的图像翻转。默认使用水平翻转。 |
| FixInvalidPolygon | `gt_polygons`<br>`gt_ignored`         | `gt_polygons`<br>`gt_ignored`                                       | 自动修复或忽略非法多边形标注。                                       |

`textrecog_transforms.py` 中实现了文本识别任务中常用的数据增强模块：

|                 |          |                                                                   |                                                                                                            |

| --------------- | -------- | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| 数据转换类名称  | 需求字段 | 修改/添加字段                                                     | 说明                                                                                                       |
| RescaleToHeight | `img`    | `img`<br>`img_shape`<br>`scale`<br>`scale_factor`<br>`keep_ratio` | 缩放图像至指定高度，并尽可能保持长宽比不变。当 `min_width` 及 `max_width` 被指定时，长宽比则可能会被改变。 |
|                 |          |                                                                   |                                                                                                            |

以上表格仅选择性地对部分数据增强方法作简要介绍，更多数据增强方法介绍请参考[API 文档](../api.rst)或阅读代码内的文档注释。
```

## 数据格式化 - formatting.py

数据格式化负责将图像、真实标签以及其它常用信息等打包成一个字典。不同的任务通常依赖于不同的数据格式化数据变换类。例如：

```

|                     |          |               |                                            |
| ------------------- | -------- | ------------- | ------------------------------------------ |
| 数据转换类名称      | 需求字段 | 修改/添加字段 | 说明                                       |
| PackTextDetInputs   | -        | -             | 用于打包文本检测任务所需要的输入信息。     |
| PackTextRecogInputs | -        | -             | 用于打包文本识别任务所需要的输入信息。     |
| PackKIEInputs       | -        | -             | 用于打包关键信息抽取任务所需要的输入信息。 |

## 跨库数据适配器 - adapters.py

跨库数据适配器打通了 MMOCR 与其他 OpenMMLab 系列算法库如 [MMDetection](https://github.com/open-mmlab/mmdetection) 之间的数据格式，使得跨项目调用其它开源算法库的配置文件及算法成为了可能。目前，MMOCR 实现了 `MMDet2MMOCR` 以及 `MMOCR2MMDet`，使得数据可以在 MMDetection 与 MMOCR 的格式之间自由转换；借助这些适配转换器，用户可以在 MMOCR 算法库内部轻松调用任何 MMDetection 已支持的检测算法，并在 OCR 相关数据集上进行训练。例如，我们以 Mask R-CNN 为例提供了[教程](#todo)，展示了如何在 MMOCR 中使用 MMDetection 的检测算法训练文本检测器。

|                |                                              |                               |                                                |
| -------------- | -------------------------------------------- | ----------------------------- | ---------------------------------------------- |
| 数据转换类名称 | 需求字段                                     | 修改/添加字段                 | 说明                                           |

| MMDet2MMOCR    | `gt_masks` `gt_ignore_flags`                 | `gt_polygons`<br>`gt_ignored` | 将 MMDet 中采用的字段转换为对应的 MMOCR 字段。 |
| MMOCR2MMDet    | `img_shape`<br>`gt_polygons`<br>`gt_ignored` | `gt_masks` `gt_ignore_flags`  | 将 MMOCR 中采用的字段转换为对应的 MMDet 字段。 |

## 包装类 - wrappers.py

为了方便用户在 MMOCR 内部无缝调用常用的 CV 算法库，我们在 wrappers.py 中提供了相应的包装类。其主要打通了 MMOCR 与其它第三方算法库之间的数据格式和转换标准，使得用户可以在 MMOCR 的配置文件内直接配置使用这些第三方库提供的数据变换方法。目前支持的包装类有：

|                    |                                                              |                                                               |                                                               |
| ------------------ | ------------------------------------------------------------ | ------------------------------------------------------------- | ------------------------------------------------------------- |

| 数据转换类名称     | 需求字段                                                     | 修改/添加字段                                                 | 说明                                                          |
| ImgAugWrapper      | `img`<br>`gt_polygons` (optional for text recognition)<br>`gt_bboxes` (optional for text recognition)<br>`gt_bboxes_labels` (optional for text recognition)<br>`gt_ignored` (optional for text recognition)<br>`gt_texts` (optional) | `img`<br>`gt_polygons` (optional for text recognition)<br>`gt_bboxes` (optional for text recognition)<br>`gt_bboxes_labels` (optional for text recognition)<br>`gt_ignored` (optional for text recognition)<br>`img_shape` (optional)<br>`gt_texts` (optional) | [ImgAug](https://github.com/aleju/imgaug) 包装类，用于打通 ImgAug 与 MMOCR 的数据格式及配置，方便用户调用 ImgAug 实现的一系列数据增强方法。 |

| TorchVisionWrapper | `img`                                                        | `img`<br>`img_shape`                                          | [TorchVision](https://github.com/pytorch/vision) 包装类，用于打通 TorchVision 与 MMOCR 的数据格式及配置，方便用户调用 `torchvision.transforms` 中实现的一系列数据变换方法。 |

### `ImgAugWrapper` 示例

例如，在原生的 ImgAug 中，我们可以按照如下代码定义一个 `Sequential` 类型的数据增强流程，对图像分别进行随机翻转、随机旋转和随机缩放：

```python
import imgaug.augmenters as iaa

aug = iaa.Sequential(
  iaa.Fliplr(0.5),                # 以概率 0.5 进行水平翻转
  iaa.Affine(rotate=(-10, 10)),   # 随机旋转 -10 到 10 度
  iaa.Resize((0.5, 3.0))          # 随机缩放到 50% 到 300% 的尺寸
)
```

而在 MMOCR 中，我们可以通过 `ImgAugWrapper` 包装类，将上述数据增强流程直接配置到 `train_pipeline` 中：

```python
dict(
  type='ImgAugWrapper',
  args=[
    ['Fliplr', 0.5],
    dict(cls='Affine', rotate=[-10, 10]),
    ['Resize', [0.5, 3.0]],
  ]
)
```

```python
dict(
  args=[
  ]
)
```

其中，`args` 参数接收一个列表，列表中的每个元素可以是一个列表，也可以是一个字典。如果是列表，则列表的第一个元素为 `imgaug.augmenters` 中的类名，后面的元素为该类的初始化参数；如果是字典，则字典的 `cls` 键对应 `imgaug.augmenters` 中的类名，其他键值对则对应该类的初始化参数。

### `TorchVisionWrapper` 示例

例如，在原生的 TorchVision 中，我们可以按照如下代码定义一个 `Compose` 类型的数据变换流程，对图像进行色彩抖动：

```python
import torchvision.transforms as transforms

aug = transforms.Compose([
  transforms.ColorJitter(
    brightness=32.0 / 255,  # 亮度抖动范围
    saturation=0.5)         # 饱和度抖动范围
])
```

而在 MMOCR 中，我们可以通过 `TorchVisionWrapper` 包装类，将上述数据变换流程直接配置到 `train_pipeline` 中：

```python
dict(
  type='TorchVisionWrapper',
  op='ColorJitter',
  brightness=32.0 / 255,
  saturation=0.5
)
```

其中，`op` 参数为 `torchvision.transforms` 中的类名，后面的参数则对应该类的初始化参数。

# Code Evaluation Service

We support evaluating datasets of multiple programming languages, similar to [humaneval-x](https://huggingface.co/datasets/THUDM/humaneval-x). Before starting, make sure that you have started the code evaluation service. You can refer to the [code-evaluator](https://github.com/Ezra-Yu/code-evaluator) project for the code evaluation service.

## Launching the Code Evaluation Service

Make sure you have installed Docker, then build an image and run a container service.

Build the Docker image:

```shell
git clone https://github.com/Ezra-Yu/code-evaluator.git
cd code-evaluator/docker
sudo docker build -t code-eval:latest .
```

After obtaining the image, create a container with the following commands:

```shell
# Log output format
sudo docker run -it -p 5000:5000 code-eval:latest python server.py

# Run the program in the background
# sudo docker run -itd -p 5000:5000 code-eval:latest python server.py

# Using different ports

# sudo docker run -itd -p 5001:5001 code-eval:latest python server.py --port 5001
```

Ensure that you can access the service and check the following commands (skip this step if you are running the service on a local host):

```shell
ping your_service_ip_address
telnet your_service_ip_address your_service_port
```

```note
If computing nodes cannot connect to the evaluation service, you can directly run `python run.py xxx...`. The resulting code will be saved in the 'outputs' folder. After migration, use [code-evaluator](https://github.com/Ezra-Yu/code-evaluator) directly to get the results (no need to consider the eval_cfg configuration later).
```

## Configuration File

```

We have provided the [configuration file](https://github.com/InternLM/opencompass/blob/main/configs/eval_codegeex2.py) for evaluating huamaneval-x on codegeex2 .

The dataset and related post-processing configuration files can be found at this [link](https://github.com/InternLM/opencompass/tree/main/configs/datasets/humanevalx). Note the `evaluator` field in `humanevalx_eval_cfg_dict`.

```python
from opencompass.openicl.icl_prompt_template import PromptTemplate
from opencompass.openicl.icl_retriever import ZeroRetriever
from opencompass.openicl.icl_inferencer import GenInferencer
from opencompass.datasets import HumanevalXDataset, HumanevalXEvaluator

humanevalx_reader_cfg = dict(
    input_columns=['prompt'], output_column='task_id', train_split='test')

humanevalx_infer_cfg = dict(
    prompt_template=dict(
        type=PromptTemplate,
        template='{prompt}'),
    retriever=dict(type=ZeroRetriever),

    inferencer=dict(type=GenInferencer, max_out_len=1024))

humanevalx_eval_cfg_dict = {
    lang : dict(
            evaluator=dict(
                type=HumanevalXEvaluator,
                language=lang,
                ip_address="localhost",    # replace to your code_eval_server ip_address, port
                port=5000),               # refer to https://github.com/Ezra-Yu/code-evaluator to launch a server
            pred_role='BOT')
    for lang in ['python', 'cpp', 'go', 'java', 'js']   # do not support rust now
}

humanevalx_datasets = [
    dict(
        type=HumanevalXDataset,
        abbr=f'humanevalx-{lang}',
        language=lang,
        path='./data/humanevalx',
        reader_cfg=humanevalx_reader_cfg,
        infer_cfg=humanevalx_infer_cfg,
        eval_cfg=humanevalx_eval_cfg_dict[lang])

language=lang,
    for lang in ['python', 'cpp', 'go', 'java', 'js']
]
```

}
```

<details>

```bibtex
}
```

| [pose_resnet_50](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_res50_8xb64-210e_mpii-256x256.py) |  256x256   | 0.882 |  0.286   | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnet/res50_mpii_256x256-418ffc88_20200812.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnet/res50_mpii_256x256_20200812.log.json) |
| [pose_resnet_101](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_res101_8xb64-210e_mpii-256x256.py) |  256x256   | 0.888 |  0.290   | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnet/res101_mpii_256x256-416f5d71_20200812.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnet/res101_mpii_256x256_20200812.log.json) |

| [pose_resnet_152](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_res152_8xb32-210e_mpii-256x256.py) |  256x256   | 0.889 |  0.303   | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnet/res152_mpii_256x256-3ecba29d_20200812.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnet/res152_mpii_256x256_20200812.log.json) |

# Customize Datasets (To be updated)

## Support new data format

To support a new data format, you can convert them to existing formats (DOTA format). You could choose to convert them offline (before training by a script) or online (implement a new dataset and do the conversion at training).
In MMRotate, we recommend to convert the data into DOTA formats and do the conversion offline, thus you only need to modify the config's data annotation paths and classes after the conversion of your data.

### Reorganize new data formats to existing format

The simplest way is to convert your dataset to existing dataset formats (DOTA).

The annotation txt files in DOTA format:

```text
184 2875 193 2923 146 2932 137 2885 plane 0
66 2095 75 2142 21 2154 11 2107 plane 0
...
```

Each line represents an object and records it as a 10-dimensional array `A`.

- `A[0:8]`: Polygons with format `(x1, y1, x2, y2, x3, y3, x4, y4)`.
- `A[8]`: Category.
- `A[9]`: Difficulty.

...
```

After the data pre-processing, there are two steps for users to train the customized new dataset with existing format (e.g. DOTA format):

1. Modify the config file for using the customized dataset.
2. Check the annotations of the customized dataset.

Here we give an example to show the above two steps, which uses a customized dataset of 5 classes with COCO format to train an existing Cascade Mask R-CNN R50-FPN detector.

#### 1. Modify the config file for using the customized dataset

There are two aspects involved in the modification of config file:

1. The `data` field. Specifically, you need to explicitly add the `classes` fields in `data.train`, `data.val` and `data.test`.

2. The `num_classes` field in the `model` part. Explicitly over-write all the `num_classes` from default value (e.g. 80 in COCO) to your classes number.

In `configs/my_custom_config.py`:

```python

# the new config inherits the base configs to highlight the necessary modification
_base_ = './rotated_retinanet_hbb_r50_fpn_1x_dota_oc'

# 1. dataset settings
dataset_type = 'DOTADataset'
classes = ('a', 'b', 'c', 'd', 'e')
        # explicitly add your class names to the field `classes`
        classes=classes,
        ann_file='path/to/your/train/annotation_data',
        img_prefix='path/to/your/train/image_data'),
    val=dict(

classes=classes,
    val=dict(
        ann_file='path/to/your/val/annotation_data',
        img_prefix='path/to/your/val/image_data'),
        ann_file='path/to/your/test/annotation_data',
        img_prefix='path/to/your/test/image_data'))

# 2. model settings
        type='RotatedRetinaHead',
        # explicitly over-write all the `num_classes` field from default 15 to 5.
        num_classes=15))
```

#### 2. Check the annotations of the customized dataset

Assuming your customized dataset is DOTA format, make sure you have the correct annotations in the customized dataset:

```

- The `classes` fields in your config file should have exactly the same elements and the same order with the `A[8]` in txt annotations. MMRotate automatically maps the uncontinuous `id` in `categories` to the continuous label indices, so the string order of `name` in `categories` field affects the order of label indices. Meanwhile, the string order of `classes` in config affects the label text during visualization of predicted bounding boxes.

## Customize datasets by dataset wrappers

MMRotate also supports many dataset wrappers to mix the dataset or modify the dataset distribution for training.
Currently it supports to three dataset wrappers as below:

- `RepeatDataset`: simply repeat the whole dataset.
- `ClassBalancedDataset`: repeat dataset in a class balanced manner.

- `ConcatDataset`: concat datasets.

### Repeat dataset

We use `RepeatDataset` as wrapper to repeat the dataset. For example, suppose the original dataset is `Dataset_A`, to repeat it, the config looks like the following

```python
dataset_A_train = dict(
        times=N,
        dataset=dict(  # This is the original config of Dataset_A
            type='Dataset_A',
            ...
            pipeline=train_pipeline
        )
    )
```

### Class balanced dataset

We use `ClassBalancedDataset` as wrapper to repeat the dataset based on category
frequency. The dataset to repeat needs to instantiate function `self.get_cat_ids(idx)`
to support `ClassBalancedDataset`.
For example, to repeat `Dataset_A` with `oversample_thr=1e-3`, the config looks like the following

```python

```python
        type='ClassBalancedDataset',
        oversample_thr=1e-3,
            ...
        )
    )
```

### Concatenate dataset

There are three ways to concatenate the dataset.

1. If the datasets you want to concatenate are in the same type with different annotation files, you can concatenate the dataset configs like the following.

   ```python
   dataset_A_train = dict(
       type='Dataset_A',
       ann_file = ['anno_file_1', 'anno_file_2'],
       pipeline=train_pipeline
   )
   ```

```python
   )
   ```

   If the concatenated dataset is used for test or evaluation, this manner supports to evaluate each dataset separately. To test the concatenated datasets as a whole, you can set `separate_eval=False` as below.

   ```python
       separate_eval=False,
   )
   ```

2. In case the dataset you want to concatenate is different, you can concatenate the dataset configs like the following.

   ```python
   dataset_A_train = dict()
   dataset_B_train = dict()

   data = dict(
       imgs_per_gpu=2,
       workers_per_gpu=2,
       train = [
           dataset_A_train,
           dataset_B_train
       ],
       val = dataset_A_val,
       test = dataset_A_test
       )
   ```

workers_per_gpu=2,
       train = [
       ],
       )
   ```

   If the concatenated dataset is used for test or evaluation, this manner also supports to evaluate each dataset separately.

3. We also support to define `ConcatDataset` explicitly as the following.

   ```python
   dataset_A_val = dict()
   dataset_B_val = dict()

       train=dataset_A_train,
       val=dict(
           type='ConcatDataset',
           datasets=[dataset_A_val, dataset_B_val],
           separate_eval=False))
   ```

   This manner allows users to evaluate all the datasets as a single one by setting `separate_eval=False`.

**Note:**

separate_eval=False))
   ```

**Note:**

1. The option `separate_eval=False` assumes the datasets use `self.data_infos` during evaluation. Therefore, COCO datasets do not support this behavior since COCO datasets do not fully rely on `self.data_infos` for evaluation. Combining different types of datasets and evaluating them as a whole is not tested thus is not suggested.
2. Evaluating `ClassBalancedDataset` and `RepeatDataset` is not supported thus evaluating concatenated datasets of these types is also not supported.

A more complex example that repeats `Dataset_A` and `Dataset_B` by N and M times, respectively, and then concatenates the repeated datasets is as the following.

```python
    type='RepeatDataset',
    times=N,
        type='Dataset_A',
        ...
    )
)
dataset_A_val = dict(

```python
    times=N,
        ...
    )
)
    ...
    pipeline=test_pipeline
)
dataset_A_test = dict(
    ...
)
dataset_B_train = dict(
    times=M,
        type='Dataset_B',
        ...
    )
)
    imgs_per_gpu=2,
    train = [
        dataset_A_train,
        dataset_B_train
    ],
    val = dataset_A_val,
    test = dataset_A_test
)

```

# 准备 REDS 数据集

```bibtex
@InProceedings{Nah_2019_CVPR_Workshops_REDS,
  author = {Nah, Seungjun and Baik, Sungyong and Hong, Seokil and Moon, Gyeongsik and Son, Sanghyun and Timofte, Radu and Lee, Kyoung Mu},
  title = {NTIRE 2019 Challenge on Video Deblurring and Super-Resolution: Dataset and Study},
  booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
  year = {2019}
}
```

- 训练集: [REDS 数据集](https://seungjunnah.github.io/Datasets/reds.html).
- 验证集: [REDS 数据集](https://seungjunnah.github.io/Datasets/reds.html) 和 Vid4.

请注意，我们合并了 REDS 的训练集和验证集，以便在 REDS4 划分（在 `EDVR` 中会使用到）和官方验证集划分之间切换。

原始验证集的名称被修改了（clip 000 到 029），以避免与训练集发生冲突（总共 240 个 clip）。具体而言，验证集中的 clips 被改名为 240、241、... 269。

可通过运行以下命令来准备 REDS 数据集:

```shell
python tools/dataset_converters/reds/preprocess_reds_dataset.py ./data/REDS
```

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── REDS
│   │   ├── train_sharp
│   │   │   ├── 000

```

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── REDS
│   │   │   ├── 000
│   │   │   ├── 001
│   │   │   ├── ...
│   │   ├── train_sharp_bicubic
│   │   │   ├── X4
│   │   │   |   ├── 000
│   │   │   |   ├── 001
│   │   │   |   ├── ...
│   │   ├── meta_info_reds4_train.txt
│   │   ├── meta_info_reds4_val.txt
│   │   ├── meta_info_official_train.txt
│   │   ├── meta_info_official_val.txt
│   │   ├── meta_info_REDS_GT.txt
│   ├── REDS4
│   │   ├── GT
│   │   ├── sharp_bicubic
```

## 准备 LMDB 格式的 REDS 数据集

如果您想使用 `LMDB` 以获得更快的 IO 速度，可以通过以下脚本来构建 LMDB 文件：

```shell
python tools/dataset_converters/reds/preprocess_reds_dataset.py --root-path ./data/REDS --make-lmdb
```

## 裁剪为子图

MMagic 支持将 REDS 图像裁剪为子图像以加快 IO。我们提供了这样一个脚本：

```shell
python tools/dataset_converters/reds/crop_sub_images.py --data-root ./data/REDS  -scales 4
```

生成的数据存储在 `REDS` 下，数据结构如下，其中`_sub`表示子图像。

## 裁剪为子图

```shell
```

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── REDS
│   │   │   ├── 000
│   │   │   ├── 001
│   │   │   ├── ...
│   │   ├── train_sharp_sub
│   │   │   ├── 000_s001
│   │   │   ├── 000_s002
│   │   │   ├── ...
│   │   │   ├── 001_s001
│   │   │   ├── ...
│   │   │   ├── X4
│   │   │   │   ├── 000
│   │   │   │   ├── 001
│   │   │   ├── ...
│   │   │   ├── ...
```

请注意，默认情况下，`preprocess_reds_dataset.py` 不会为裁剪后的数据集制作 lmdb 和注释文件。您可能需要为此类操作稍微修改脚本。

# Test on embedded device

Here are the test conclusions of our edge devices. You can directly obtain the results of your own environment with [model profiling](../02-how-to-run/profile_model.md).

## Software and hardware environment

- host OS ubuntu 18.04
- backend SNPE-1.59
- device Mi11 (qcom 888)

## mmpretrain

|                                                          model                                                           |   dataset   | spatial | fp32 top-1 (%) | snpe gpu hybrid fp32 top-1 (%) | latency (ms) |
| :----------------------------------------------------------------------------------------------------------------------: | :---------: | :-----: | :------------: | :----------------------------: | :----------: |
| [ShuffleNetV2](https://github.com/open-mmlab/mmpretrain/blob/main/configs/shufflenet_v2/shufflenet-v2-1x_16xb64_in1k.py) | ImageNet-1k | 224x224 |     69.55      |            69.83\*             |     20±7     |

|    [MobilenetV2](https://github.com/open-mmlab/mmpretrain/blob/main/configs/mobilenet_v2/mobilenet-v2_8xb32_in1k.py)     | ImageNet-1k | 224x224 |     71.86      |            72.14\*             |     15±6     |

tips:

1. The ImageNet-1k dataset is too large to test, only part of the dataset is used (8000/50000)
2. The heating of device will downgrade the frequency, so the time consumption will actually fluctuate. Here are the stable values after running for a period of time. This result is closer to the actual demand.

## mmocr detection

|                                                         model                                                          |  dataset  | spatial  | fp32 hmean | snpe gpu hybrid hmean | latency(ms) |
| :--------------------------------------------------------------------------------------------------------------------: | :-------: | :------: | :--------: | :-------------------: | :---------: |

| [PANet](https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/panet/panet_resnet18_fpem-ffm_600e_icdar2015.py) | ICDAR2015 | 1312x736 |   0.795    |    0.785 @thr=0.9     |  3100±100   |

## mmpose

|                                                                                  model                                                                                  |  dataset   | spatial | snpe hybrid AR@IoU=0.50 | snpe hybrid AP@IoU=0.50 | latency(ms) |
| :---------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :--------: | :-----: | :---------------------: | :---------------------: | :---------: |

| [pose_hrnet_w32](https://github.com/open-mmlab/mmpose/blob/main/configs/animal_2d_keypoint/topdown_heatmap/animalpose/td-hm_hrnet-w32_8xb64-210e_animalpose-256x256.py) | Animalpose | 256x256 |          0.997          |          0.989          |   630±50    |

tips:

- Test `pose_hrnet` using AnimalPose's test dataset instead of val dataset.

## mmseg

|                                                        model                                                         |  dataset   | spatial  | mIoU  | latency(ms) |
| :------------------------------------------------------------------------------------------------------------------: | :--------: | :------: | :---: | :---------: |
| [fcn](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r18-d8_4xb2-80k_cityscapes-512x1024.py) | Cityscapes | 512x1024 | 71.11 |  4915±500   |

tips:

- `fcn` works fine with 512x1024 size. Cityscapes dataset uses 1024x2048 resolution which causes device to reboot.

## Notes

tips:

## Notes

- We needs to manually split the mmdet model into two parts. Because
  - In snpe source code, `onnx_to_ir.py` can only parse onnx input while `ir_to_dlc.py` does not support `topk` operator
  - UDO (User Defined Operator) does not work with `snpe-onnx-to-dlc`
- mmagic model
  - `srcnn` requires cubic resize which snpe does not support
  - `esrgan` converts fine, but loading the model causes the device to reboot
- mmrotate depends on [e2cnn](https://pypi.org/project/e2cnn/) and needs to be installed manually [its Python3.6 compatible branch](https://github.com/QUVA-Lab/e2cnn)

# 如何在自定义数据集上微调模型

在很多场景下，我们需要快速地将模型应用到新的数据集上，但从头训练模型通常很难快速收敛，这种不确定性会浪费额外的时间。
通常，已有的、在大数据集上训练好的模型会比随机初始化提供更为有效的先验信息，粗略来讲，在此基础上的学习我们称之为模型微调。

已经证明，在 ImageNet 数据集上预训练的模型对于其他数据集和其他下游任务有很好的效果。
因此，该教程提供了如何将 [Model Zoo](../modelzoo_statistics.md) 中提供的预训练模型用于其他数据集，已获得更好的效果。

在本教程中，我们提供了一个实践示例和一些关于如何在自己的数据集上微调模型的技巧。

## 第一步：准备你的数据集

按照 [准备数据集](../user_guides/dataset_prepare.md) 准备你的数据集。
假设我们的数据集根文件夹路径为 `data/custom_dataset/`

假设我们想进行有监督图像分类训练，并使用子文件夹格式的 `CustomDataset` 来组织数据集：

```text
├── train
│   │   └── ...
│   └── ...
└── test
    │   └── ...
    └── ...
```

## 第二步：选择一个配置文件作为模板

在这里，我们使用 `configs/resnet/resnet50_8xb32_in1k.py` 作为示例。
首先在同一文件夹下复制一份配置文件，并将其重命名为 `resnet50_8xb32-ft_custom.py`。

```{tip}
按照惯例，配置名称的最后一个字段是数据集，例如，`in1k` 表示 ImageNet-1k，`coco` 表示 coco 数据集
```

```{tip}
```

这个配置的内容是：

```python
_base_ = [
    '../_base_/models/resnet50.py',           # 模型设置
    '../_base_/datasets/imagenet_bs32.py',    # 数据设置
    '../_base_/schedules/imagenet_bs256.py',  # 训练策略设置
    '../_base_/default_runtime.py',           # 运行设置
]
```

## 第三步：修改模型设置

在进行模型微调时，我们通常希望在主干网络（backbone）加载预训练模型，再用我们的数据集训练一个新的分类头（head）。

为了在主干网络加载预训练模型，我们需要修改主干网络的初始化设置，使用
`Pretrained` 类型的初始化函数。另外，在初始化设置中，我们使用 `prefix='backbone'`
来告诉初始化函数需要加载的子模块的前缀，`backbone`即指加载模型中的主干网络。
方便起见，我们这里使用一个在线的权重文件链接，它
会在训练前自动下载对应的文件，你也可以提前下载这个模型，然后使用本地路径。

接下来，新的配置文件需要按照新数据集的类别数目来修改分类头的配置。只需要修改分
类头中的 `num_classes` 设置即可。

另外，当新的小数据集和原本预训练的大数据集中的数据分布较为类似的话，我们在进行微调时会希望
冻结主干网络前面几层的参数，只训练后面层以及分类头的参数，这么做有助于在后续训练中，
保持网络从预训练权重中获得的提取低阶特征的能力。在 MMPretrain 中，
这一功能可以通过简单的一个 `frozen_stages` 参数来实现。比如我们需要冻结前两层网
络的参数，只需要在上面的配置中添加一行：

```{note}
注意，目前并非所有的主干网络都支持 `frozen_stages` 参数。请检查[文档](https://mmpretrain.readthedocs.io/en/latest/api.html#module-mmpretrain.models.backbones)
确认使用的主干网络是否支持这一参数。
```

```python
_base_ = [
]

# >>>>>>>>>>>>>>> 在这里重载模型相关配置 >>>>>>>>>>>>>>>>>>>
        )),
)

prefix='backbone',
        )),
)
# <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
```

```{tip}
这里我们只需要设定我们想要修改的部分配置，其他配置将会自动从我们的基配置文件中获取。
```

## 第四步：修改数据集设置

为了在新数据集上进行微调，我们需要覆盖一些数据集设置，例如数据集类型、数据流水线等。

```python
_base_ = [
]

# 模型设置
        )),
)

# >>>>>>>>>>>>>>> 在这里重载数据配置 >>>>>>>>>>>>>>>>>>>

)),
)

        ann_file='',       # 我们假定使用子文件夹格式，因此需要将标注文件置空
    ))
    ))
# <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
```

## 第五步：修改训练策略设置（可选）

微调所使用的训练超参数一般与默认的超参数不同，它通常需要更小的学习率和更快的学习率衰减。

```python
_base_ = [
]

# 模型设置

'../_base_/schedules/imagenet_bs256.py',  # 训练策略设置
]

# 模型设置
        )),
)

# 数据设置
        ann_file='',
    ))
    ))

# >>>>>>>>>>>>>>> 在这里重载训练策略设置 >>>>>>>>>>>>>>>>>>>
# 优化器超参数
# 学习率策略

# 优化器超参数
# 学习率策略
# <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
```

```{tip}
更多关于配置文件的信息，请参阅[学习配置文件](../user_guides/config.md)
```

## 开始训练

现在，我们完成了用于微调的配置文件，完整的文件如下：

```python
_base_ = [
]

# 模型设置
        )),

type='Pretrained',
        )),
)

# 数据设置
    ))
    ))

# 训练策略设置
```

接下来，我们使用一台 8 张 GPU 的电脑来训练我们的模型，指令如下：

```shell
```

当然，我们也可以使用单张 GPU 来进行训练，使用如下命令：

```shell

```

```shell
```

```shell
```

但是如果我们使用单张 GPU 进行训练的话，需要在数据集设置部分作如下修改：

```python
    ))
    ))
```

这是因为我们的训练策略是针对批次大小（batch size）为 256 设置的。在父配置文件中，
设置了单张 `batch_size=32`，如果使用 8 张 GPU，总的批次大小就是 256。而如果使
用单张 GPU，就必须手动修改 `batch_size=256` 来匹配训练策略。

然而，更大的批次大小需要更大的 GPU 显存，这里有几个简单的技巧来节省显存：

1. 启用自动混合精度训练

   ```shell
   ```

   ```shell
   ```

2. 使用较小的批次大小，例如仍然使用 `batch_size=32`，而不是 256，并启用学习率自动缩放

   ```shell
   ```

   学习率自动缩放功能会根据实际的 batch size 和配置文件中的 `auto_scale_lr.base_batch_size`
   字段对学习率进行线性调整（你可以在基配置文件 `configs/_base_/schedules/imagenet_bs256.py`
   中找到这一字段）

```{note}
以上技巧都有可能对训练效果造成轻微影响。
```

### 在命令行指定预训练模型

如果您不想修改配置文件，您可以使用 `--cfg-options` 将您的预训练模型文件添加到 `init_cfg`.

例如，以下命令也会加载预训练模型：

```shell
bash tools/dist_train.sh configs/tutorial/resnet50_finetune_cifar.py 8 \

# Compatibility

## v1.1.0rc0

### OpenMMLab v2.0 Refactoring

In this version, we make large refactoring based on MMEngine to achieve unified data elements, model interfaces, visualizers, evaluators and other runtime modules across different datasets, tasks and even codebases. A brief summary for this refactoring is as follows:

- Data Element:
  - We add [`Det3DDataSample`](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/structures/det3d_data_sample.py) as the common data element passing through datasets and models. It inherits from [`DetDataSample`](<%5Bhttps://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/structures/det3d_data_sample.py%5D(https://github.com/open-mmlab/mmdetection/blob/dev-3.x/mmdet/structures/det_data_sample.py)>) in mmdetection and implement `InstanceData`, `PixelData`, and
    `LabelData` inheriting from `BaseDataElement` in MMEngine to represent different types of ground truth labels or predictions.
- Datasets:

`LabelData` inheriting from `BaseDataElement` in MMEngine to represent different types of ground truth labels or predictions.
  - We add [`Det3DDataset`](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/datasets/det3d_dataset.py) and [`Seg3DDataset`](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/datasets/seg3d_dataset.py) as the base datasets to inherit from the unified `BaseDataset` in MMEngine. They implement most functions that are commonly used across different datasets and simplify the info loading/processing in the current datasets. Re-defined input arguments and functions can be most re-used in different datasets, which are important for the implementation of customized datasets.

- We define the common keys across different datasets and unify all the info files with a standard protocol. The same info is more clear for users because they share the same key across different dataset infos. Besides, for different settings, such as camera-only and LiDAR-only methods, we no longer need different info formats (like the previous pkl and json files). We can just revise the `parse_data_info` to read the necessary information from a complete info file.
  - We add `train_dataloader`, `val_dataloader` and `test_dataloader` to replace the original `data` in the config. It simplify the levels of data-related fields.
- Data Transforms
  - Based on the basic transforms and wrappers re-implemented and simplified in the latest MMCV, we refactor data transforms to inherit from them.
  - We also adjust the implementation of current data pipelines to make them compatible with our latest data protocol.

- We also adjust the implementation of current data pipelines to make them compatible with our latest data protocol.
  - Normalization, padding of images and voxelization operations are moved to the data-preprocessing.
  - `DefaultFormatBundle3D` and `Collect3D` are replaced with `PackDet3DInputs` to pack the data into the element format as model input.
- Models
  - Unify the model interface as `inputs`, `data_samples`, `return_loss=False`
  - The basic pre-processing before model forward includes: 1) convert input from CPU to GPU tensors; 2) padding images; 3) normalize images; 4) voxelization.
  - Return `loss_dict` during training while return `list[data_sample]` during inference
  - Simply function interfaces in the models
  - Add `preprocess_cfg` in the model configs for pre-processing
- Visualizer

- Simply function interfaces in the models
  - Design a unified visualizer, [`Det3DLocalVisualizer`](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/visualization/local_visualizer.py), based on MMEngine for different 3D tasks and settings
  - Support browsing dataset and visualization hooks based on the [`Det3DLocalVisualizer`](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/visualization/local_visualizer.py)
- Evaluator
  - Decouple evaluators from datasets to make them more flexible: the evaluation codes of each dataset are implemented as a metric class exclusively.
  - Add evaluator information to the current dataset configs
- Registry
  - Refactor all the registries to inherit from root registries in MMEngine
  - When using modules from other codebases, it is necessary to specify the registry scope, such as `mmdet.ResNet`

- Refactor all the registries to inherit from root registries in MMEngine
- Others: Refactor logging, hooks, scheduler, runner and other runtime configs based on MMEngine

## v1.0.0rc1

### Operators Migration

We have adopted CUDA operators compiled from [mmcv](https://github.com/open-mmlab/mmcv/blob/master/mmcv/ops/__init__.py) and removed all the CUDA operators in mmdet3d. We now do not need to compile the CUDA operators in mmdet3d anymore.

### Waymo dataset converter refactoring

In this version we did a major code refactoring that boosted the performance of waymo dataset conversion by multiprocessing.
Meanwhile, we also fixed the imprecise timestamps saving issue in waymo dataset conversion. This change introduces following backward compatibility breaks:

- The point cloud .bin files of waymo dataset need to be regenerated.

  In the .bin files each point occupies 6 `float32` and the meaning of the last `float32` now changed from **imprecise timestamps** to **range frame offset**.
  The **range frame offset** for each point is calculated as`ri * h * w + row * w + col` if the point is from the **TOP** lidar or `-1` otherwise.
  The `h`, `w` denote the height and width of the TOP lidar's range frame.
  The `ri`, `row`, `col` denote the return index, the row and the column of the range frame where each point locates.
  Following tables show the difference across the change:

Before

| Element offset (float32) |  0  |  1  |  2  |     3     |     4      |            5            |
| ------------------------ | :-: | :-: | :-: | :-------: | :--------: | :---------------------: |
| Bytes offset             |  0  |  4  |  8  |    12     |     16     |           20            |

| Meaning                  |  x  |  y  |  z  | intensity | elongation | **imprecise timestamp** |

After

| Element offset (float32) |  0  |  1  |  2  |     3     |     4      |           5            |
| ------------------------ | :-: | :-: | :-: | :-------: | :--------: | :--------------------: |
| Bytes offset             |  0  |  4  |  8  |    12     |     16     |           20           |
| Meaning                  |  x  |  y  |  z  | intensity | elongation | **range frame offset** |

- The objects' point cloud .bin files in the GT-database of waymo dataset need to be regenerated because we also dumped the range frame offset for each point into it.

Before

| Element offset (float32) |  0  |  1  |  2  |     3     |     4      |

Following tables show the difference across the change:

Before

| ------------------------ | :-: | :-: | :-: | :-------: | :--------: |
| Bytes offset             |  0  |  4  |  8  |    12     |     16     |
| Meaning                  |  x  |  y  |  z  | intensity | elongation |

After

- Any configuration that uses waymo dataset with GT Augmentation should change the `db_sampler.points_loader.load_dim` from `5` to `6`.

## v1.0.0rc0

### Coordinate system refactoring

In this version, we did a major code refactoring which improved the consistency among the three coordinate systems (and corresponding box representation), LiDAR, Camera, and Depth. A brief summary for this refactoring is as follows:

- The three coordinate systems are all right-handed now (which means the yaw angle increases in the counterclockwise direction).
- The LiDAR system `(x_size, y_size, z_size)` corresponds to `(l, w, h)` instead of `(w, l, h)`. This is more natural since `l` is parallel with the direction where the yaw angle is zero, and we prefer using the positive direction of the `x` axis as that direction, which is exactly how we define yaw angle in Depth and Camera coordinate systems.
- The APIs for box-related operations are improved and now are more user-friendly.

#### ***NOTICE!!***

Since definitions of box representation have changed, the annotation data of most datasets require updating:

- SUN RGB-D: Yaw angles in the annotation should be reversed.
- KITTI: For LiDAR boxes in GT databases, (x_size, y_size, z_size, yaw) out of (x, y, z, x_size, y_size, z_size) should be converted from the old LiDAR coordinate system to the new one. The training/validation data annotations should be left unchanged since they are under the Camera coordinate system, which is unmodified after the refactoring.
- Waymo: Same as KITTI.
- nuScenes: For LiDAR boxes in training/validation data and GT databases, (x_size, y_size, z_size, yaw) out of (x, y, z, x_size, y_size, z_size) should be converted.
- Lyft: Same as nuScenes.

Please regenerate the data annotation/GT database files or use [`update_data_coords.py`](https://github.com/open-mmlab/mmdetection3d/blob/v1.0.0rc0/tools/update_data_coords.py) to update the data.

To use boxes under Depth and LiDAR coordinate systems, or to convert boxes between different coordinate systems, users should be aware of the difference between the old and new definitions. For example, the rotation, flipping, and bev functions of [`DepthInstance3DBoxes`](https://github.com/open-mmlab/mmdetection3d/blob/v1.0.0rc0/mmdet3d/core/bbox/structures/depth_box3d.py) and [`LiDARInstance3DBoxes`](https://github.com/open-mmlab/mmdetection3d/blob/v1.0.0rc0/mdet3d/core/bbox/structures/lidar_box3d.py) and box conversion [functions](https://github.com/open-mmlab/mmdetection3d/blob/v1.0.0rc0/mmdet3d/core/bbox/structures/box_3d_mode.py) have all been reimplemented in the refactoring.

Consequently, functions like [`output_to_lyft_box`](https://github.com/open-mmlab/mmdetection3d/blob/v1.0.0rc0/mmdet3d/datasets/lyft_dataset.py) undergo small modification to adapt to the new LiDAR/Depth box.

Since the LiDAR system `(x_size, y_size, z_size)` now corresponds to `(l, w, h)` instead of `(w, l, h)`, the anchor sizes for LiDAR boxes are also changed, e.g., from `[1.6, 3.9, 1.56]` to `[3.9, 1.6, 1.56]`.

Functions only involving points are generally unaffected except if they rely on some refactored utility functions such as `rotation_3d_in_axis`.

#### Other BC-breaking or new features:

- `array_converter`: Please refer to [array_converter.py](https://github.com/open-mmlab/mmdetection3d/blob/v1.0.0rc0/mmdet3d/core/utils/array_converter.py). Functions wrapped with `array_converter` can convert array-like input types of `torch.Tensor`, `np.ndarray`, and `list/tuple/float` to `torch.Tensor` to process in an unified PyTorch pipeline. The result may finally be converted back to the input type. Most functions in [utils.py](https://github.com/open-mmlab/mmdetection3d/blob/v1.0.0rc0/mmdet3d/core/bbox/structures/utils.py) are wrapped with `array_converter`.

- [`points_in_boxes`](https://github.com/open-mmlab/mmdetection3d/blob/v1.0.0rc0/mmdet3d/core/bbox/structures/base_box3d.py) and [`points_in_boxes_batch`](https://github.com/open-mmlab/mmdetection3d/blob/v1.0.0rc0/mmdet3d/core/bbox/structures/base_box3d.py) will be deprecated soon. They are renamed to `points_in_boxes_part` and `points_in_boxes_all` respectively, with more detailed docstrings. The major difference of the two functions is that if a point is enclosed by multiple boxes, `points_in_boxes_part` will only return the index of the first enclosing box while `points_in_boxes_all` will return all the indices of enclosing boxes.

- `rotation_3d_in_axis`: Please refer to [utils.py](https://github.com/open-mmlab/mmdetection3d/blob/v1.0.0rc0/mmdet3d/core/bbox/structures/utils.py). Now this function supports multiple input types and more options. The function with the same name in [box_np_ops.py](https://github.com/open-mmlab/mmdetection3d/blob/v1.0.0rc0/mmdet3d/core/bbox/box_np_ops.py) is deleted since we do not need another function to tackle with NumPy data. `rotation_2d`, `points_cam2img`, and `limit_period` in box_np_ops.py are also deleted for the same reason.
- `bev` method of [`CameraInstance3DBoxes`](https://github.com/open-mmlab/mmdetection3d/blob/v1.0.0rc0/mmdet3d/core/bbox/structures/cam_box3d.py): Changed it to be consistent with the definition of bev in Depth and LiDAR coordinate systems.
- Data augmentation utils in [data_augment_utils.py](https://github.com/open-mmlab/mmdetection3d/blob/v1.0.0rc0/mmdet3d/datasets/pipelines/data_augment_utils.py) now follow the rules of a right-handed system.

- We do not need the yaw hacking in KITTI anymore after refining [`get_direction_target`](https://github.com/open-mmlab/mmdetection3d/blob/v1.0.0rc0/mmdet3d/models/dense_heads/train_mixins.py). Interested users may refer to PR [#677](https://github.com/open-mmlab/mmdetection3d/pull/677) .

## 0.16.0

### Returned values of `QueryAndGroup` operation

We modified the returned `grouped_xyz` value of operation `QueryAndGroup` to support PAConv segmentor. Originally, the `grouped_xyz` is centered by subtracting the grouping centers, which represents the relative positions of grouped points. Now, we didn't perform such subtraction and the returned `grouped_xyz` stands for the absolute coordinates of these points.

Note that, the other returned variables of `QueryAndGroup` such as `new_features`, `unique_cnt` and `grouped_idx` are not affected.

### NuScenes coco-style data pre-processing

We remove the rotation and dimension hack in the monocular 3D detection on nuScenes. Specifically, we transform the rotation and dimension of boxes defined by nuScenes devkit to the coordinate system of our `CameraInstance3DBoxes` in the pre-processing and transform them back in the post-processing. In this way, we can remove the corresponding [hack](https://github.com/open-mmlab/mmdetection3d/pull/744/files#diff-5bee5062bd84e6fa25a2fdd71353f6f283dfdc4a66a0316c3b1ca26078c978b6L165) used in the visualization tools. The modification also guarantees the correctness of all the operations based on our `CameraInstance3DBoxes` (such as NMS and flip augmentation) when training monocular 3D detectors.

The modification only influences nuScenes coco-style json files. Please re-run the nuScenes data preparation script if necessary. See more details in the PR [#744](https://github.com/open-mmlab/mmdetection3d/pull/744).

### ScanNet dataset for ImVoxelNet

We adopt a new pre-processing procedure for the ScanNet dataset in order to support ImVoxelNet, which is a multi-view method requiring image data. In previous versions of MMDetection3D, ScanNet dataset was only used for point cloud based 3D detection and segmentation methods. We plan adding ImVoxelNet to our model zoo, thus updating ScanNet correspondingly by adding image-related pre-processing steps. Specifically, we made these changes:

- Add [script](https://github.com/open-mmlab/mmdetection3d/blob/master/data/scannet/extract_posed_images.py) for extracting RGB data.
- Update [script](https://github.com/open-mmlab/mmdetection3d/blob/master/tools/dataset_converters/scannet_data_utils.py) for annotation creating.

- Add instructions in the documents on preparing image data.

Please refer to the ScanNet [README.md](https://github.com/open-mmlab/mmdetection3d/blob/master/data/scannet/README.md/) for more details.

## 0.15.0

### Unified parameter initialization

To unify the parameter initialization in OpenMMLab projects, MMCV supports `BaseModule` that accepts `init_cfg` to allow the modules' parameters initialized in a flexible and unified manner. Now the users need to explicitly call `model.init_weights()` in the training script to initialize the model (as in [here](https://github.com/open-mmlab/mmdetection3d/blob/master/tools/train.py#L183), previously this was handled by the detector. Please refer to PR [#622](https://github.com/open-mmlab/mmdetection3d/pull/622) for details.

### BackgroundPointsFilter

We modified the dataset augmentation function `BackgroundPointsFilter`([here](https://github.com/open-mmlab/mmdetection3d/blob/v0.15.0/mmdet3d/datasets/pipelines/transforms_3d.py#L1132)). In previous version of MMdetection3D, `BackgroundPointsFilter` changes the gt_bboxes_3d's bottom center to the gravity center. In MMDetection3D 0.15.0,

`BackgroundPointsFilter` will not change it. Please refer to PR [#609](https://github.com/open-mmlab/mmdetection3d/pull/609) for details.

### Enhance `IndoorPatchPointSample` transform

We enhance the pipeline function `IndoorPatchPointSample` used in point cloud segmentation task by adding more choices for patch selection. Also, we plan to remove the unused parameter `sample_rate` in the future. Please modify the code as well as the config files accordingly if you use this transform.

## 0.14.0

### Dataset class for 3D segmentation task

We remove a useless parameter `label_weight` from segmentation datasets including `Custom3DSegDataset`, `ScanNetSegDataset` and `S3DISSegDataset` since this weight is utilized in the loss function of model class. Please modify the code as well as the config files accordingly if you use or inherit from these codes.

### ScanNet data pre-processing

We adopt new pre-processing and conversion steps of ScanNet dataset. In previous versions of MMDetection3D, ScanNet dataset was only used for 3D detection task, where we trained on the training set and tested on the validation set. In MMDetection3D 0.14.0, we further support 3D segmentation task on ScanNet, which includes online benchmarking on test set. Since the alignment matrix is not provided for test set data, we abandon the alignment of points in data generation steps to support both tasks. Besides, as 3D segmentation requires per-point prediction, we also remove the down-sampling step in data generation.

- In the new ScanNet processing scripts, we save the unaligned points for all the training, validation and test set. For train and val set with annotations, we also store the `axis_align_matrix` in data infos. For ground-truth bounding boxes, we store boxes in both aligned and unaligned coordinates with key `gt_boxes_upright_depth` and key `unaligned_gt_boxes_upright_depth` respectively in data infos.

- In `ScanNetDataset`, we now load the `axis_align_matrix` as a part of data annotations. If it is not contained in old data infos, we will use identity matrix for compatibility. We also add a transform function `GlobalAlignment` in ScanNet detection data pipeline to align the points.

- Since the aligned boxes share the same key as in old data infos, we do not need to modify the code related to it. But do remember that they are not in the same coordinate system as the saved points.

- There is an `PointSample` pipeline in the data pipelines for ScanNet detection task which down-samples points. So removing down-sampling in data generation will not affect the code.

We have trained a [VoteNet](https://github.com/open-mmlab/mmdetection3d/blob/master/configs/votenet/votenet_8x8_scannet-3d-18class.py) model on the newly processed ScanNet dataset and get similar benchmark results. In order to prepare ScanNet data for both detection and segmentation tasks, please re-run the new pre-processing scripts following the ScanNet [README.md](https://github.com/open-mmlab/mmdetection3d/blob/master/data/scannet/README.md/).

## 0.12.0

### SUNRGBD dataset for ImVoteNet

## 0.12.0

We adopt a new pre-processing procedure for the SUNRGBD dataset in order to support ImVoteNet, which is a multi-modality method requiring both image and point cloud data. In previous versions of MMDetection3D, SUNRGBD dataset was only used for point cloud based 3D detection methods. In MMDetection3D 0.12.0, we add ImVoteNet to our model zoo, thus updating SUNRGBD correspondingly by adding image-related pre-processing steps. Specifically, we made these changes:

- Fix a bug in the image file path in meta data.
- Convert calibration matrices from double to float to avoid type mismatch in further operations.

Please refer to the SUNRGBD [README.md](https://github.com/open-mmlab/mmdetection3d/blob/master/data/sunrgbd/README.md/) for more details.

## 0.6.0

### VoteNet and H3DNet model structure update

## 0.6.0

In MMDetection 0.6.0, we updated the model structures of VoteNet and H3DNet, therefore model checkpoints generated by MMDetection \< 0.6.0 should be first converted to a format compatible with the latest structures via [convert_votenet_checkpoints.py](https://github.com/open-mmlab/mmdetection3d/blob/master/tools/model_converters/convert_votenet_checkpoints.py) and [convert_h3dnet_checkpoints.py](https://github.com/open-mmlab/mmdetection3d/blob/master/tools/model_converters/convert_h3dnet_checkpoints.py) . For more details, please refer to the VoteNet [README.md](https://github.com/open-mmlab/mmdetection3d/tree/master/configs/votenet/README.md/) and H3DNet [README.md](https://github.com/open-mmlab/mmdetection3d/tree/master/configs/h3dnet/README.md/).

  <img width="100%" src="https://github.com/open-mmlab/mmpose/assets/13503330/5b637d76-41dd-4376-9a7f-854cd120799d"/>
</div>

# RTMPose: Real-Time Multi-Person Pose Estimation toolkit based on MMPose

> [RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose](https://arxiv.org/abs/2303.07399)

</div>

近年来，2D 姿态估计的研究在公开数据集上取得了出色的成绩，但是它在工业界的应用仍然受到笨重的模型参数和高推理延迟的影响。为了让前沿姿态估计算法在工业界落地，我们通过实验研究了多人姿态估计算法的五个方面：范式、骨干网络、定位算法、训练策略和部署推理，基于 MMPose 提出了一个高性能的实时多人姿态估计框架 **RTMPose**。我们的 RTMPose-m 模型在 COCO 上取得 **75.8％AP**，在 Intel i7-11700 CPU 上达到 **90+FPS**，在 NVIDIA GTX 1660 Ti GPU 上达到 **430+FPS**。我们同样验证了在算力有限的设备上做实时姿态估计，RTMPose-s 在移动端骁龙865芯片上可以达到 **COCO 72.2%AP**，**70+FPS**。在 MMDeploy 的帮助下，我们的项目支持 CPU、GPU、Jetson、移动端等多种部署环境。

![rtmpose_intro](https://user-images.githubusercontent.com/13503330/219269619-935499e5-bdd9-49ea-8104-3c7796dbd862.png)

- [🙌 社区共建](#-社区共建-)
- [⚡ Pipeline 性能](#-pipeline-性能-)
- [📊 模型库](#-模型库-)
- [👀 可视化](#-可视化-)
- [😎 快速尝试](#-快速尝试-)
- [👨‍🏫 模型训练](#-模型训练-)
- [🏗️ 部署教程](#️-部署教程-)
- [📚 常用功能](#️-常用功能-)
  - [🚀 模型测速](#-模型测速-)
  - [📊 精度验证](#-精度验证-)
- [📜 引用](#-引用-)

## 🥳 最新进展 [🔝](#-table-of-contents)

- 2023 年 6 月：
  - 发布混合数据集训练的 26 点 Body 模型。
- 2023 年 5 月：
  - 添加 [代码示例](./examples/)
  - 发布混合数据集训练的 Hand, Face, Body 模型。
- 2023 年 3 月：发布 RTMPose。RTMPose-m 取得 COCO 验证集 75.8 mAP，推理速度达到 430+ FPS 。

- 添加 [代码示例](./examples/)

<img src="https://user-images.githubusercontent.com/13503330/221138554-110240d8-e887-4b9a-90b1-2fbdc982e9de.gif" width=400 height=300/><img src="https://user-images.githubusercontent.com/13503330/221125176-85015a13-9648-4f0d-a17c-1cbb469efacf.gif" width=250 height=300/><img src="https://user-images.githubusercontent.com/13503330/221125310-7eeb2212-907e-427f-97af-af799d70a4c5.gif" width=250 height=300/>
</div>

<img src="https://github.com/open-mmlab/mmpose/assets/13503330/38aa345e-4ceb-4e73-bc37-5e082735e336" width=450 height=300/><img src="https://user-images.githubusercontent.com/13503330/221125888-15c20faf-0ad5-4afb-828b-a71ccb064582.gif" width=450 height=300/>
</div>

</div>
<img src="https://github.com/open-mmlab/mmpose/assets/13503330/2ecbf9f4-6963-4a14-9801-da10c0a65dac" width=300 height=350/><img src="https://user-images.githubusercontent.com/13503330/221138017-10431ab4-e515-4c32-8fa7-8748e2d17a58.gif" width=600 height=350/>
</div>

### ✨ 主要特性

- 🚀 **高精度，低延迟**

  | Model | AP(COCO) | CPU-FPS | GPU-FPS |
  | :---: | :------: | :-----: | :-----: |
  |   t   |   68.5   |  300+   |  940+   |
  |   s   |   72.2   |  200+   |  710+   |
  |   m   |   75.8   |   90+   |  430+   |
  |   l   |   76.5   |   50+   |  280+   |

- 🛠️ **易部署**

  - 详细的部署代码教程，手把手教你模型部署
  - MMDeploy 助力
  - 支持多种部署后端
    - ONNX
    - TensorRT
    - ncnn
    - OpenVINO 等
  - 支持多种平台
    - Linux
    - Windows
    - NVIDIA Jetson
    - ARM 等

- 🏗️ **为实际业务设计**

  - 提供多种 Pipeline 推理接口和 SDK
    - Python
    - C++
    - C#
    - JAVA 等

## 🙌 社区共建 [🔝](#-table-of-contents)

- 支持多种平台
    - Linux
    - Windows
    - ARM 等

    - Python
    - C++
    - C#
    - JAVA 等

RTMPose 是一个长期优化迭代的项目，致力于业务场景下的高性能实时姿态估计算法的训练、优化和部署，因此我们十分期待来自社区的力量，欢迎分享不同业务场景中 RTMPose 的训练配置与技巧，助力更多的社区用户！

✨ ✨ ✨

- **如果你是 RTMPose 的新用户，我们热切希望你能参与[这份问卷](https://uua478.fanqier.cn/f/xxmynrki)/[Google Questionnaire](https://docs.google.com/forms/d/e/1FAIpQLSfzwWr3eNlDzhU98qzk2Eph44Zio6hi5r0iSwfO9wSARkHdWg/viewform?usp=sf_link)，这对于我们的工作非常重要！**

✨ ✨ ✨

欢迎加入我们的社区交流群获得更多帮助：

- 微信用户群

<div align=left>
<img src="https://user-images.githubusercontent.com/13503330/222647056-875bed70-85ec-455c-9016-c024772915c4.jpg" width=200 />

- Discord Group:
  - 🙌 https://discord.gg/raweFPmdzG 🙌

## ⚡ Pipeline 性能 [🔝](#-table-of-contents)

**说明**

- Pipeline 速度测试时开启了隔帧检测策略，默认检测间隔为 5 帧。
- 环境配置:
  - torch >= 1.7.1
  - onnxruntime 1.12.1
  - TensorRT 8.4.3.1
  - cuDNN 8.3.2
  - CUDA 11.3

**说明**

- 环境配置:
  - CUDA 11.3

| Detection Config                                                    | Pose Config                                                                   | Input Size<sup><br>(Det/Pose) | Model AP<sup><br>(COCO) | Pipeline AP<sup><br>(COCO) | Params (M)<sup><br>(Det/Pose) | Flops (G)<sup><br>(Det/Pose) | ORT-Latency(ms)<sup><br>(i7-11700) | TRT-FP16-Latency(ms)<sup><br>(GTX 1660Ti) |                                                                                                                                  Download                                                                                                                                  |

| [RTMDet-nano](./rtmdet/person/rtmdet_nano_320-8xb32_coco-person.py) | [RTMPose-t](./rtmpose/body_2d_keypoint/rtmpose-t_8xb256-420e_coco-256x192.py) |      320x320<br>256x192       |      40.3<br>67.1       |            64.4            |         0.99<br/>3.34         |        0.31<br/>0.36         |               12.403               |                   2.467                   | [det](https://download.openmmlab.com/mmpose/v1/projects/rtmpose/rtmdet_nano_8xb32-100e_coco-obj365-person-05d8511e.pth)<br/>[pose](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-tiny_simcc-aic-coco_pt-aic-coco_420e-256x192-cfc8f33d_20230126.pth) |

| [RTMDet-nano](./rtmdet/person/rtmdet_nano_320-8xb32_coco-person.py) | [RTMPose-s](./rtmpose/body_2d_keypoint/rtmpose-s_8xb256-420e_coco-256x192.py) |      320x320<br>256x192       |      40.3<br>71.1       |            68.5            |         0.99<br/>5.47         |        0.31<br/>0.68         |               16.658               |                   2.730                   |  [det](https://download.openmmlab.com/mmpose/v1/projects/rtmpose/rtmdet_nano_8xb32-100e_coco-obj365-person-05d8511e.pth)<br/>[pose](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_simcc-aic-coco_pt-aic-coco_420e-256x192-fcb2599b_20230126.pth)   |

| [RTMDet-nano](./rtmdet/person/rtmdet_nano_320-8xb32_coco-person.py) | [RTMPose-m](./rtmpose/body_2d_keypoint/rtmpose-m_8xb256-420e_coco-256x192.py) |      320x320<br>256x192       |      40.3<br>75.3       |            73.2            |        0.99<br/>13.59         |        0.31<br/>1.93         |               26.613               |                   4.312                   |  [det](https://download.openmmlab.com/mmpose/v1/projects/rtmpose/rtmdet_nano_8xb32-100e_coco-obj365-person-05d8511e.pth)<br/>[pose](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-aic-coco_pt-aic-coco_420e-256x192-63eb25f7_20230126.pth)   |

| [RTMDet-nano](./rtmdet/person/rtmdet_nano_320-8xb32_coco-person.py) | [RTMPose-l](./rtmpose/body_2d_keypoint/rtmpose-l_8xb256-420e_coco-256x192.py) |      320x320<br>256x192       |      40.3<br>76.3       |            74.2            |        0.99<br/>27.66         |        0.31<br/>4.16         |               36.311               |                   4.644                   |  [det](https://download.openmmlab.com/mmpose/v1/projects/rtmpose/rtmdet_nano_8xb32-100e_coco-obj365-person-05d8511e.pth)<br/>[pose](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-aic-coco_pt-aic-coco_420e-256x192-f016ffe0_20230126.pth)   |

| [RTMDet-m](./rtmdet/person/rtmdet_m_640-8xb32_coco-person.py)       | [RTMPose-m](./rtmpose/body_2d_keypoint/rtmpose-m_8xb256-420e_coco-256x192.py) |      640x640<br>256x192       |      62.5<br>75.3       |            75.7            |        24.66<br/>13.59        |        38.95<br/>1.93        |                 -                  |                   6.923                   |    [det](https://download.openmmlab.com/mmpose/v1/projects/rtmpose/rtmdet_m_8xb32-100e_coco-obj365-person-235e8209.pth)<br/>[pose](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-aic-coco_pt-aic-coco_420e-256x192-63eb25f7_20230126.pth)    |

| [RTMDet-m](./rtmdet/person/rtmdet_m_640-8xb32_coco-person.py)       | [RTMPose-l](./rtmpose/body_2d_keypoint/rtmpose-l_8xb256-420e_coco-256x192.py) |      640x640<br>256x192       |      62.5<br>76.3       |            76.6            |        24.66<br/>27.66        |        38.95<br/>4.16        |                 -                  |                   7.204                   |    [det](https://download.openmmlab.com/mmpose/v1/projects/rtmpose/rtmdet_m_8xb32-100e_coco-obj365-person-235e8209.pth)<br/>[pose](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-aic-coco_pt-aic-coco_420e-256x192-f016ffe0_20230126.pth)    |

## 📊 模型库 [🔝](#-table-of-contents)

**说明**

- 此处提供的模型采用了多数据集联合训练以提高性能，模型指标不适用于学术比较。
- 表格中为开启了 Flip Test 的测试结果。
- RTMPose 在更多公开数据集上的性能指标可以前往 [Model Zoo](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/algorithms.html) 查看。
- RTMPose 在更多硬件平台上的推理速度可以前往 [Benchmark](./benchmark/README_CN.md) 查看。

- 如果你有希望我们支持的数据集，欢迎[联系我们](https://uua478.fanqier.cn/f/xxmynrki)/[Google Questionnaire](https://docs.google.com/forms/d/e/1FAIpQLSfzwWr3eNlDzhU98qzk2Eph44Zio6hi5r0iSwfO9wSARkHdWg/viewform?usp=sf_link)！

### 人体 2d 关键点

#### 17 Keypoints

- 关键点骨架定义遵循 [COCO](http://cocodataset.org/). 详情见 [meta info](/configs/_base_/datasets/coco.py).
- <img src="https://github.com/open-mmlab/mmpose/assets/13503330/2417e4f7-2203-468f-bad0-e7a6a6bf8251" height="300px">

<summary><b>AIC+COCO</b></summary>

|                                    Config                                     | Input Size | AP<sup><br>(COCO) | PCK@0.1<sup><br>(Body8) | AUC<sup><br>(Body8) | Params<sup><br>(M) | FLOPS<sup><br>(G) | ORT-Latency<sup><br>(ms)<sup><br>(i7-11700) | TRT-FP16-Latency<sup><br>(ms)<sup><br>(GTX 1660Ti) | ncnn-FP16-Latency<sup><br>(ms)<sup><br>(Snapdragon 865) |                                                                    Download                                                                     |

| [RTMPose-t](./rtmpose/body_2d_keypoint/rtmpose-t_8xb256-420e_coco-256x192.py) |  256x192   |       68.5        |          91.28          |        63.38        |        3.34        |       0.36        |                    3.20                     |                        1.06                        |                          9.02                           | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-tiny_simcc-aic-coco_pt-aic-coco_420e-256x192-cfc8f33d_20230126.pth) |

| [RTMPose-s](./rtmpose/body_2d_keypoint/rtmpose-s_8xb256-420e_coco-256x192.py) |  256x192   |       72.2        |          92.95          |        66.19        |        5.47        |       0.68        |                    4.48                     |                        1.39                        |                          13.89                          |  [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_simcc-aic-coco_pt-aic-coco_420e-256x192-fcb2599b_20230126.pth)   |

| [RTMPose-m](./rtmpose/body_2d_keypoint/rtmpose-m_8xb256-420e_coco-256x192.py) |  256x192   |       75.8        |          94.13          |        68.53        |       13.59        |       1.93        |                    11.06                    |                        2.29                        |                          26.44                          |  [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-aic-coco_pt-aic-coco_420e-256x192-63eb25f7_20230126.pth)   |

| [RTMPose-l](./rtmpose/body_2d_keypoint/rtmpose-l_8xb256-420e_coco-256x192.py) |  256x192   |       76.5        |          94.35          |        68.98        |       27.66        |       4.16        |                    18.85                    |                        3.46                        |                          45.37                          |  [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-aic-coco_pt-aic-coco_420e-256x192-f016ffe0_20230126.pth)   |

| [RTMPose-m](./rtmpose/body_2d_keypoint/rtmpose-m_8xb256-420e_coco-384x288.py) |  384x288   |       77.0        |          94.32          |        69.85        |       13.72        |       4.33        |                    24.78                    |                        3.66                        |                            -                            |  [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-aic-coco_pt-aic-coco_420e-384x288-a62a0b32_20230228.pth)   |

| [RTMPose-l](./rtmpose/body_2d_keypoint/rtmpose-l_8xb256-420e_coco-384x288.py) |  384x288   |       77.3        |          94.54          |        70.14        |       27.79        |       9.35        |                      -                      |                        6.05                        |                            -                            |  [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-aic-coco_pt-aic-coco_420e-384x288-97d6cb0f_20230228.pth)   |

<summary><b>Body8</b></summary>

- `*` 代表模型在 7 个开源数据集上训练得到：

- [CrowdPose](https://mmpose.readthedocs.io/en/latest/dataset_zoo/2d_body_keypoint.html#crowdpose)
- `Body8` 代表除了以上提到的 7 个数据集，再加上 [OCHuman](https://mmpose.readthedocs.io/en/latest/dataset_zoo/2d_body_keypoint.html#ochuman) 合并后一起进行评测得到的指标。

|                                     Config                                      | Input Size | AP<sup><br>(COCO) | PCK@0.1<sup><br>(Body8) | AUC<sup><br>(Body8) | Params<sup><br>(M) | FLOPS<sup><br>(G) | ORT-Latency<sup><br>(ms)<sup><br>(i7-11700) | TRT-FP16-Latency<sup><br>(ms)<sup><br>(GTX 1660Ti) | ncnn-FP16-Latency<sup><br>(ms)<sup><br>(Snapdragon 865) |                                                                Download                                                                |

| :-----------------------------------------------------------------------------: | :--------: | :---------------: | :---------------------: | :-----------------: | :----------------: | :---------------: | :-----------------------------------------: | :------------------------------------------------: | :-----------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------: |
| [RTMPose-t\*](./rtmpose/body_2d_keypoint/rtmpose-t_8xb256-420e_coco-256x192.py) |  256x192   |       65.9        |          91.44          |        63.18        |        3.34        |       0.36        |                    3.20                     |                        1.06                        |                          9.02                           | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-t_simcc-body7_pt-body7_420e-256x192-026a1439_20230504.pth) |

| [RTMPose-s\*](./rtmpose/body_2d_keypoint/rtmpose-s_8xb256-420e_coco-256x192.py) |  256x192   |       69.7        |          92.45          |        65.15        |        5.47        |       0.68        |                    4.48                     |                        1.39                        |                          13.89                          | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_simcc-body7_pt-body7_420e-256x192-acd4a1ef_20230504.pth) |
| [RTMPose-m\*](./rtmpose/body_2d_keypoint/rtmpose-m_8xb256-420e_coco-256x192.py) |  256x192   |       74.9        |          94.25          |        68.59        |       13.59        |       1.93        |                    11.06                    |                        2.29                        |                          26.44                          | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-body7_pt-body7_420e-256x192-e48f03d0_20230504.pth) |

| [RTMPose-l\*](./rtmpose/body_2d_keypoint/rtmpose-l_8xb256-420e_coco-256x192.py) |  256x192   |       76.7        |          95.08          |        70.14        |       27.66        |       4.16        |                    18.85                    |                        3.46                        |                          45.37                          | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-body7_pt-body7_420e-256x192-4dba18fc_20230504.pth) |
| [RTMPose-m\*](./rtmpose/body_2d_keypoint/rtmpose-m_8xb256-420e_coco-384x288.py) |  384x288   |       76.6        |          94.64          |        70.38        |       13.72        |       4.33        |                    24.78                    |                        3.66                        |                            -                            | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-body7_pt-body7_420e-384x288-65e718c4_20230504.pth) |

| [RTMPose-l\*](./rtmpose/body_2d_keypoint/rtmpose-l_8xb256-420e_coco-384x288.py) |  384x288   |       78.3        |          95.36          |        71.58        |       27.79        |       9.35        |                      -                      |                        6.05                        |                            -                            | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-body7_pt-body7_420e-384x288-3f5a1437_20230504.pth) |
| [RTMPose-x\*](./rtmpose/body_2d_keypoint/rtmpose-x_8xb256-700e_coco-384x288.py) |  384x288   |       78.8        |            -            |          -          |       49.43        |       17.22       |                      -                      |                         -                          |                            -                            | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-x_simcc-body7_pt-body7_700e-384x288-71d7b7e9_20230629.pth) |

#### 26 Keypoints

- 关键点骨架定义遵循 [Halpe26](https://github.com/Fang-Haoshu/Halpe-FullBody/)，详情见 [meta info](/configs/_base_/datasets/halpe26.py)。
- <img src="https://github.com/open-mmlab/mmpose/assets/13503330/f28ab3ba-833d-4ca7-8739-f97e6cafbab7" height="300px">
- 模型在 `Body8` 上进行训练和评估。

|                                          Config                                           | Input Size | PCK@0.1<sup><br>(Body8) | AUC<sup><br>(Body8) | Params(M) | FLOPS(G) | ORT-Latency<sup><br>(ms)<sup><br>(i7-11700) | TRT-FP16-Latency<sup><br>(ms)<sup><br>(GTX 1660Ti) | ncnn-FP16-Latency<sup><br>(ms)<sup><br>(Snapdragon 865) |                                                                    Download                                                                    |

| :---------------------------------------------------------------------------------------: | :--------: | :---------------------: | :-----------------: | :-------: | :------: | :-----------------------------------------: | :------------------------------------------------: | :-----------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------: |
| [RTMPose-t\*](./rtmpose/body_2d_keypoint/rtmpose-t_8xb1024-700e_body8-halpe26-256x192.py) |  256x192   |          91.89          |        66.35        |   3.51    |   0.37   |                      -                      |                         -                          |                            -                            | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-t_simcc-body7_pt-body7-halpe26_700e-256x192-6020f8a6_20230605.pth) |

| [RTMPose-s\*](./rtmpose/body_2d_keypoint/rtmpose-s_8xb1024-700e_body8-halpe26-256x192.py) |  256x192   |          93.01          |        68.62        |   5.70    |   0.70   |                      -                      |                         -                          |                            -                            | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_simcc-body7_pt-body7-halpe26_700e-256x192-7f134165_20230605.pth) |
| [RTMPose-m\*](./rtmpose/body_2d_keypoint/rtmpose-m_8xb512-700e_body8-halpe26-256x192.py)  |  256x192   |          94.75          |        71.91        |   13.93   |   1.95   |                      -                      |                         -                          |                            -                            | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-body7_pt-body7-halpe26_700e-256x192-4d3e73dd_20230605.pth) |

| [RTMPose-l\*](./rtmpose/body_2d_keypoint/rtmpose-l_8xb512-700e_body8-halpe26-256x192.py)  |  256x192   |          95.37          |        73.19        |   28.11   |   4.19   |                      -                      |                         -                          |                            -                            | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-body7_pt-body7-halpe26_700e-256x192-2abb7558_20230605.pth) |
| [RTMPose-m\*](./rtmpose/body_2d_keypoint/rtmpose-m_8xb512-700e_body8-halpe26-384x288.py)  |  384x288   |          95.15          |        73.56        |   14.06   |   4.37   |                      -                      |                         -                          |                            -                            | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-body7_pt-body7-halpe26_700e-384x288-89e6428b_20230605.pth) |

| [RTMPose-l\*](./rtmpose/body_2d_keypoint/rtmpose-l_8xb512-700e_body8-halpe26-384x288.py)  |  384x288   |          95.56          |        74.38        |   28.24   |   9.40   |                      -                      |                         -                          |                            -                            | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-body7_pt-body7-halpe26_700e-384x288-734182ce_20230605.pth) |
| [RTMPose-x\*](./rtmpose/body_2d_keypoint/rtmpose-x_8xb256-700e_body8-halpe26-384x288.py)  |  384x288   |          95.74          |        74.82        |   50.00   |  17.29   |                      -                      |                         -                          |                            -                            | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-x_simcc-body7_pt-body7-halpe26_700e-384x288-7fb6e239_20230606.pth) |

#### 模型剪枝

**说明**

#### 模型剪枝

**说明**

- 模型剪枝由 [MMRazor](https://github.com/open-mmlab/mmrazor) 提供

|          Config           | Input Size | AP<sup><br>(COCO) | Params<sup><br>(M) | FLOPS<sup><br>(G) | ORT-Latency<sup><br>(ms)<sup><br>(i7-11700) | TRT-FP16-Latency<sup><br>(ms)<sup><br>(GTX 1660Ti) | ncnn-FP16-Latency<sup><br>(ms)<sup><br>(Snapdragon 865) |                                                                      Download                                                                      |
| :-----------------------: | :--------: | :---------------: | :----------------: | :---------------: | :-----------------------------------------: | :------------------------------------------------: | :-----------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------: |

| RTMPose-s-aic-coco-pruned |  256x192   |       69.4        |        3.43        |       0.35        |                      -                      |                         -                          |                            -                            | [Model](https://download.openmmlab.com/mmrazor/v1/pruning/group_fisher/rtmpose-s/group_fisher_finetune_rtmpose-s_8xb256-420e_aic-coco-256x192.pth) |

更多信息，请参考 [GroupFisher Pruning for RTMPose](./rtmpose/pruning/README.md).

- 关键点骨架定义遵循 [COCO-WholeBody](https://github.com/jin-s13/COCO-WholeBody/)，详情见 [meta info](/configs/_base_/datasets/coco_wholebody.py)。
- <img src="https://user-images.githubusercontent.com/100993824/227770977-c8f00355-c43a-467e-8444-d307789cf4b2.png" height="300px">

| Config                          | Input Size | Whole AP | Whole AR | FLOPS<sup><br>(G) | ORT-Latency<sup><br>(ms)<sup><br>(i7-11700) | TRT-FP16-Latency<sup><br>(ms)<sup><br>(GTX 1660Ti) |             Download              |
| :------------------------------ | :--------: | :------: | :------: | :---------------: | :-----------------------------------------: | :------------------------------------------------: | :-------------------------------: |
| [RTMPose-m](./rtmpose/wholebody_2d_keypoint/rtmpose-m_8xb64-270e_coco-wholebody-256x192.py) |  256x192   |   58.2   |   67.4   |       2.22        |                    13.50                    |                        4.00                        | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-coco-wholebody_pt-aic-coco_270e-256x192-cd5e845c_20230123.pth) |

| [RTMPose-l](./rtmpose/wholebody_2d_keypoint/rtmpose-l_8xb64-270e_coco-wholebody-256x192.py) |  256x192   |   61.1   |   70.0   |       4.52        |                    23.41                    |                        5.67                        | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-coco-wholebody_pt-aic-coco_270e-256x192-6f206314_20230124.pth) |
| [RTMPose-l](./rtmpose/wholebody_2d_keypoint/rtmpose-l_8xb32-270e_coco-wholebody-384x288.py) |  384x288   |   64.8   |   73.0   |       10.07       |                    44.58                    |                        7.68                        | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-coco-wholebody_pt-aic-coco_270e-384x288-eaeb96c8_20230125.pth) |

| [RTMPose-x](./rtmpose/wholebody_2d_keypoint/rtmpose-x_8xb32-270e_coco-wholebody-384x288.py) |  384x288   |   65.3   |   73.3   |       18.1        |                      -                      |                         -                          | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-x_simcc-coco-wholebody_pt-body7_270e-384x288-401dfc90_20230629.pth) |

### 动物 2d 关键点 (17 Keypoints)

- 关键点骨架定义遵循 [AP-10K](https://github.com/AlexTheBad/AP-10K/)，详情见 [meta info](/configs/_base_/datasets/ap10k.py)。
- <img src="https://user-images.githubusercontent.com/100993824/227797151-091dc21a-d944-49c9-8b62-cc47fa89e69f.png" height="300px">

|             Config             | Input Size | AP<sup><br>(AP10K) | FLOPS<sup><br>(G) | ORT-Latency<sup><br>(ms)<sup><br>(i7-11700) | TRT-FP16-Latency<sup><br>(ms)<sup><br>(GTX 1660Ti) |             Download             |

| :----------------------------: | :--------: | :----------------: | :---------------: | :-----------------------------------------: | :------------------------------------------------: | :------------------------------: |
| [RTMPose-m](./rtmpose/animal_2d_keypoint/rtmpose-m_8xb64-210e_ap10k-256x256.py) |  256x256   |        72.2        |       2.57        |                   14.157                    |                       2.404                        | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-ap10k_pt-aic-coco_210e-256x256-7a041aa1_20230206.pth) |

### 脸部 2d 关键点 (106 Keypoints)

- 关键点骨架定义遵循 [LaPa](https://github.com/JDAI-CV/lapa-dataset)，详情见 [meta info](/configs/_base_/datasets/lapa.py)。
- <img src="https://github.com/open-mmlab/mmpose/assets/13503330/30fa583e-500c-4356-ac5a-7e9d7d18381a" height="300px">

<summary><b>Face6</b></summary>

- `Face6` and `*` 代表模型在 6 个开源数据集上训练得到：

  - [COCO-Wholebody-Face](https://github.com/jin-s13/COCO-WholeBody/)
  - [WFLW](https://wywu.github.io/projects/LAB/WFLW.html)
  - [300W](https://ibug.doc.ic.ac.uk/resources/300-W/)
  - [COFW](http://www.vision.caltech.edu/xpburgos/ICCV13/)
  - [Halpe](https://github.com/Fang-Haoshu/Halpe-FullBody/)
  - [LaPa](https://github.com/JDAI-CV/lapa-dataset)

|             Config             | Input Size | NME<sup><br>(LaPa) | FLOPS<sup><br>(G) | ORT-Latency<sup><br>(ms)<sup><br>(i7-11700) | TRT-FP16-Latency<sup><br>(ms)<sup><br>(GTX 1660Ti) |             Download             |

| [RTMPose-t\*](./rtmpose/face_2d_keypoint/rtmpose-t_8xb256-120e_lapa-256x256.py) |  256x256   |        1.67        |       0.652       |                      -                      |                         -                          | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-t_simcc-face6_pt-in1k_120e-256x256-df79d9a5_20230529.pth) |
| [RTMPose-s\*](./rtmpose/face_2d_keypoint/rtmpose-s_8xb256-120e_lapa-256x256.py) |  256x256   |        1.59        |       1.119       |                      -                      |                         -                          | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_simcc-face6_pt-in1k_120e-256x256-d779fdef_20230529.pth) |

| [RTMPose-m\*](./rtmpose/face_2d_keypoint/rtmpose-m_8xb256-120e_lapa-256x256.py) |  256x256   |        1.44        |       2.852       |                      -                      |                         -                          | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-face6_pt-in1k_120e-256x256-72a37400_20230529.pth) |

### 手部 2d 关键点 (21 Keypoints)

- 关键点骨架定义遵循 [COCO-WholeBody](https://github.com/jin-s13/COCO-WholeBody/)，详情见 [meta info](/configs/_base_/datasets/coco_wholebody_hand.py)。
- <img src="https://user-images.githubusercontent.com/100993824/227771101-03a27bd8-ccc0-4eb9-a111-660f191a7a16.png" height="300px">

|       Detection Config        | Input Size | Model AP<sup><br>(OneHand10K) | Flops<sup><br>(G) | ORT-Latency<sup><br>(ms)<sup><br>(i7-11700) | TRT-FP16-Latency<sup><br>(ms)<sup><br>(GTX 1660Ti) |        Download        |

| :---------------------------: | :--------: | :---------------------------: | :---------------: | :-----------------------------------------: | :------------------------------------------------: | :--------------------: |
| [RTMDet-nano (试用)](./rtmdet/hand/rtmdet_nano_320-8xb32_hand.py) |  320x320   |             76.0              |       0.31        |                      -                      |                         -                          | [Det Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmdet_nano_8xb32-300e_hand-267f9c8f.pth) |

<summary><b>Hand5</b></summary>

- `Hand5` and `*` 代表模型在 5 个开源数据集上训练得到：
  - [COCO-Wholebody-Hand](https://github.com/jin-s13/COCO-WholeBody/)
  - [OneHand10K](https://www.yangangwang.com/papers/WANG-MCC-2018-10.html)
  - [FreiHand2d](https://lmb.informatik.uni-freiburg.de/projects/freihand/)
  - [RHD2d](https://lmb.informatik.uni-freiburg.de/resources/datasets/RenderedHandposeDataset.en.html)

- [FreiHand2d](https://lmb.informatik.uni-freiburg.de/projects/freihand/)

|                                                    Config                                                    | Input Size | PCK@0.2<sup><br>(COCO-Wholebody-Hand) | PCK@0.2<sup><br>(Hand5) | AUC<sup><br>(Hand5) | FLOPS<sup><br>(G) | ORT-Latency<sup><br>(ms)<sup><br>(i7-11700) | TRT-FP16-Latency<sup><br>(ms)<sup><br>(GTX 1660Ti) |                                                                 Download                                                                 |

| :----------------------------------------------------------------------------------------------------------: | :--------: | :-----------------------------------: | :---------------------: | :-----------------: | :---------------: | :-----------------------------------------: | :------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------: |
| [RTMPose-m\*<sup><br>(试用)](./rtmpose/hand_2d_keypoint/rtmpose-m_8xb32-210e_coco-wholebody-hand-256x256.py) |  256x256   |                 81.5                  |          96.4           |        83.9         |       2.581       |                      -                      |                         -                          | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-hand5_pt-aic-coco_210e-256x256-74fb594_20230320.pth) |

### 预训练模型

### 预训练模型

我们提供了 UDP 预训练的 CSPNeXt 模型参数，训练配置请参考 [pretrain_cspnext_udp folder](./rtmpose/pretrain_cspnext_udp/)。

|    Model     | Input Size | Params<sup><br>(M) | Flops<sup><br>(G) | AP<sup><br>(GT) | AR<sup><br>(GT) |                                                     Download                                                      |
| :----------: | :--------: | :----------------: | :---------------: | :-------------: | :-------------: | :---------------------------------------------------------------------------------------------------------------: |
| CSPNeXt-tiny |  256x192   |        6.03        |       1.43        |      65.5       |      68.9       | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmpose/cspnext-tiny_udp-aic-coco_210e-256x192-cbed682d_20230130.pth) |

|  CSPNeXt-s   |  256x192   |        8.58        |       1.78        |      70.0       |      73.3       | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmpose/cspnext-s_udp-aic-coco_210e-256x192-92f5a029_20230130.pth) |
|  CSPNeXt-m   |  256x192   |       17.53        |       3.05        |      74.8       |      77.7       | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmpose/cspnext-m_udp-aic-coco_210e-256x192-f2f7d6f6_20230130.pth) |
|  CSPNeXt-l   |  256x192   |       32.44        |       5.32        |      77.2       |      79.9       | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmpose/cspnext-l_udp-aic-coco_210e-256x192-273b7631_20230130.pth) |

- [AI Challenger](https://mmpose.readthedocs.io/en/latest/dataset_zoo/2d_body_keypoint.html#aic)

|     Model      | Input Size | Params<sup><br>(M) | Flops<sup><br>(G) | AP<sup><br>(COCO) | PCK@0.2<sup><br>(Body8) | AUC<sup><br>(Body8) |                                      Download                                      |
| :------------: | :--------: | :----------------: | :---------------: | :---------------: | :---------------------: | :-----------------: | :--------------------------------------------------------------------------------: |
| CSPNeXt-tiny\* |  256x192   |        6.03        |       1.43        |       65.9        |          96.34          |        63.80        | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-tiny_udp-body7_210e-256x192-a3775292_20230504.pth) |

|  CSPNeXt-s\*   |  256x192   |        8.58        |       1.78        |       68.7        |          96.59          |        64.92        | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-s_udp-body7_210e-256x192-8c9ccbdb_20230504.pth) |
|  CSPNeXt-m\*   |  256x192   |       17.53        |       3.05        |       73.7        |          97.42          |        68.19        | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-m_udp-body7_210e-256x192-e0c9327b_20230504.pth) |
|  CSPNeXt-l\*   |  256x192   |       32.44        |       5.32        |       75.7        |          97.76          |        69.57        | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-l_udp-body7_210e-256x192-5e9558ef_20230504.pth) |

|  CSPNeXt-m\*   |  384x288   |       17.53        |       6.86        |       75.8        |          97.60          |        70.18        | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-m_udp-body7_210e-384x288-b9bc2b57_20230504.pth) |
|  CSPNeXt-l\*   |  384x288   |       32.44        |       11.96       |       77.2        |          97.89          |        71.23        | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-l_udp-body7_210e-384x288-b15bc30d_20230504.pth) |
|  CSPNeXt-x\*   |  384x288   |       54.92        |       19.96       |       78.1        |          98.00          |        71.79        | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-x_udp-body7_210e-384x288-d28b58e6_20230529.pth) |

#### ImageNet

我们提供了 ImageNet 分类训练的 CSPNeXt 模型参数，更多细节请参考 [RTMDet](https://github.com/open-mmlab/mmdetection/blob/latest/configs/rtmdet/README.md#classification)。

|    Model     | Input Size | Params<sup><br>(M) | Flops<sup><br>(G) | Top-1 (%) | Top-5 (%) |                                                           Download                                                            |
| :----------: | :--------: | :----------------: | :---------------: | :-------: | :-------: | :---------------------------------------------------------------------------------------------------------------------------: |
| CSPNeXt-tiny |  224x224   |        2.73        |       0.34        |   69.44   |   89.45   | [Model](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/cspnext_rsb_pretrain/cspnext-tiny_imagenet_600e-3a2dd350.pth)  |

|  CSPNeXt-s   |  224x224   |        4.89        |       0.66        |   74.41   |   92.23   |   [Model](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/cspnext_rsb_pretrain/cspnext-s_imagenet_600e-ea671761.pth)   |
|  CSPNeXt-m   |  224x224   |       13.05        |       1.93        |   79.27   |   94.79   | [Model](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/cspnext_rsb_pretrain/cspnext-m_8xb256-rsb-a1-600e_in1k-ecb3bbd9.pth) |
|  CSPNeXt-l   |  224x224   |       27.16        |       4.19        |   81.30   |   95.62   | [Model](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/cspnext_rsb_pretrain/cspnext-l_8xb256-rsb-a1-600e_in1k-6a760974.pth) |
|  CSPNeXt-x   |  224x224   |       48.85        |       7.76        |   82.10   |   95.69   | [Model](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/cspnext_rsb_pretrain/cspnext-x_8xb256-rsb-a1-600e_in1k-b3f78edd.pth) |

## 👀 可视化 [🔝](#-table-of-contents)

<img src='https://user-images.githubusercontent.com/13503330/221795678-2c4ae2ec-ac23-4368-8083-0ebeb29f0d3c.gif' width=900/>
<img src="https://user-images.githubusercontent.com/13503330/219270443-421d9b02-fcec-46de-90f2-ce769c67575a.png" width=900 />
</div>

## 😎 快速尝试 [🔝](#-table-of-contents)

我们提供了两种途径来让用户尝试 RTMPose 模型：

- MMPose demo 脚本
- MMDeploy SDK 预编译包 （推荐，速度提升6-10倍）

### MMPose demo 脚本

通过 MMPose 提供的 demo 脚本可以基于 Pytorch 快速进行[模型推理](https://mmpose.readthedocs.io/en/latest/user_guides/inference.html)和效果验证。

**提示：**

- 基于 Pytorch 推理并不能达到 RTMPose 模型的最大推理速度，只用于模型效果验证。
- 输入模型路径可以是本地路径，也可以是下载链接。

```shell
# 前往 mmpose 目录
cd ${PATH_TO_MMPOSE}

# RTMDet 与 RTMPose 联合推理
# 输入模型路径可以是本地路径，也可以是下载链接。

projects/rtmpose/rtmdet/person/rtmdet_nano_320-8xb32_coco-person.py \
    --input {YOUR_TEST_IMG_or_VIDEO} \
    --show

# 摄像头推理
    --show
```

https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-aic-coco_pt-aic-coco_420e-256x192-63eb25f7_20230126.pth \
    --show
```

效果展示：

![topdown_inference_with_rtmdet](https://user-images.githubusercontent.com/13503330/220005020-06bdf37f-6817-4681-a2c8-9dd55e4fbf1e.png)

### MMDeploy SDK 预编译包 （推荐）

MMDeploy 提供了预编译的 SDK，用于对 RTMPose 项目进行 Pipeline 推理，其中推理所用的模型为 SDK 版本。

- 所有的模型必须经过 `tools/deploy.py` 导出后才能使用 PoseTracker 进行推理。
- 导出 SDK 版模型的教程见 [SDK 推理](#%EF%B8%8F-sdk-推理)，推理的详细参数设置见 [Pipeline 推理](#-pipeline-推理)。
- 你可以从 [硬件模型库](https://platform.openmmlab.com/deploee) 直接下载 SDK 版模型（ONNX、 TRT、ncnn 等）。
- 同时我们也支持 [在线模型转换](https://platform.openmmlab.com/deploee/task-convert-list)。

#### Linux\\

说明：

- GCC 版本需大于 7.5
- cmake 版本需大于 3.20

##### Python 推理

1. 安装 mmdeploy_runtime 或者 mmdeploy_runtime_gpu

```shell
# for onnxruntime
pip install mmdeploy-runtime

# for onnxruntime-gpu / tensorrt
pip install mmdeploy-runtime-gpu
```

2. 下载预编译包：

```shell

```shell

```

2. 下载预编译包：

```shell
# onnxruntime
# for ubuntu
wget -c  https://github.com/open-mmlab/mmdeploy/releases/download/v1.0.0/mmdeploy-1.0.0-linux-x86_64-cxx11abi.tar.gz
# 解压并将 third_party 中第三方推理库的动态库添加到 PATH

# for centos7 and lower
wget -c https://github.com/open-mmlab/mmdeploy/releases/download/v1.0.0/mmdeploy-1.0.0-linux-x86_64.tar.gz

# onnxruntime-gpu / tensorrt
wget -c  https://github.com/open-mmlab/mmdeploy/releases/download/v1.0.0/mmdeploy-1.0.0-linux-x86_64-cxx11abi-cuda11.3.tar.gz

wget -c https://github.com/open-mmlab/mmdeploy/releases/download/v1.0.0/mmdeploy-1.0.0-linux-x86_64-cuda11.3.tar.gz
```

```

3. 下载 sdk 模型并解压到 `./example/python` 下。（该模型只用于演示，如需其他模型，请参考 [SDK 推理](#%EF%B8%8F-sdk-推理)）

```shell
# rtmdet-nano + rtmpose-m for cpu sdk
wget https://download.openmmlab.com/mmpose/v1/projects/rtmpose/rtmpose-cpu.zip

unzip rtmpose-cpu.zip
```

4. 使用 `pose_tracker.py` 进行推理：

```shell
# 进入 ./example/python

# 请传入模型目录，而不是模型文件
# 格式：
# python pose_tracker.py cpu {det work-dir} {pose work-dir} {your_video.mp4}

# 示例：
python pose_tracker.py cpu rtmpose-ort/rtmdet-nano/ rtmpose-ort/rtmpose-m/ your_video.mp4

# 摄像头
python pose_tracker.py cpu rtmpose-ort/rtmdet-nano/ rtmpose-ort/rtmpose-m/ 0
```

##### ONNX

```shell
# 下载预编译包
wget https://github.com/open-mmlab/mmdeploy/releases/download/v1.0.0/mmdeploy-1.0.0-linux-x86_64-cxx11abi.tar.gz

# 解压文件
tar -xzvf mmdeploy-1.0.0-linux-x86_64-cxx11abi.tar.gz

# 切换到 sdk 目录

# 下载预编译包

# 解压文件

# 切换到 sdk 目录
cd mmdeploy-1.0.0-linux-x86_64-cxx11abi

# 设置环境变量
source set_env.sh

# 如果系统中没有安装 opencv 3+，请执行以下命令。如果已安装，可略过
bash install_opencv.sh

# 编译可执行程序
bash build_sdk.sh

# 图片推理
./bin/det_pose rtmpose-ort/rtmdet-nano/ rtmpose-ort/rtmpose-m/ your_img.jpg --device cpu

# 视频推理
./bin/pose_tracker rtmpose-ort/rtmdet-nano/ rtmpose-ort/rtmpose-m/ your_video.mp4 --device cpu

# 摄像头推理
./bin/pose_tracker rtmpose-ort/rtmdet-nano/ rtmpose-ort/rtmpose-m/ 0 --device cpu
```

##### TensorRT

```shell
# 下载预编译包
wget https://github.com/open-mmlab/mmdeploy/releases/download/v1.0.0/mmdeploy-1.0.0-linux-x86_64-cxx11abi-cuda11.3.tar.gz

# 解压文件
tar -xzvf mmdeploy-1.0.0-linux-x86_64-cxx11abi-cuda11.3.tar.gz

# 切换到 sdk 目录
cd mmdeploy-1.0.0-linux-x86_64-cxx11abi-cuda11.3

# 解压文件

# 切换到 sdk 目录

# 设置环境变量

# 编译可执行程序

# 图片推理
./bin/det_pose rtmpose-ort/rtmdet-nano/ rtmpose-ort/rtmpose-m/ your_img.jpg --device cuda

# 视频推理
./bin/pose_tracker rtmpose-ort/rtmdet-nano/ rtmpose-ort/rtmpose-m/ your_video.mp4 --device cuda

# 摄像头推理
./bin/pose_tracker rtmpose-ort/rtmdet-nano/ rtmpose-ort/rtmpose-m/ 0 --device cuda
```

详细参数设置见 [Pipeline 推理](#-pipeline-推理)。

#### Windows

```shell
# 下载 [sdk](https://github.com/open-mmlab/mmdeploy/releases/download/v1.0.0/mmdeploy-1.0.0-windows-amd64.zip) 并将 third_party 中第三方推理库的动态库添加到 PATH

# 下载 [sdk](https://github.com/open-mmlab/mmdeploy/releases/download/v1.0.0/mmdeploy-1.0.0-windows-amd64-cuda11.3.zip) 并将 third_party 中第三方推理库的动态库添加到 PATH
```

2. 下载 sdk 模型并解压到 `./example/python` 下。（该模型只用于演示，如需其他模型，请参考 [SDK 推理](#%EF%B8%8F-sdk-推理)）

```shell

```

3. 使用 `pose_tracker.py` 进行推理：

```shell

# 摄像头
```

##### 可执行文件推理

1. 安装 [cmake](https://cmake.org/download/)。
2. 前往 [mmdeploy](https://github.com/open-mmlab/mmdeploy/releases) 下载 win 预编译包。
3. 解压文件，进入 sdk 目录。
4. 使用管理员权限打开 PowerShell，执行以下命令：

```shell
set-ExecutionPolicy RemoteSigned
```

5. 安装 OpenCV：

```shell
```

```shell
# in sdk folder:
.\install_opencv.ps1
```

6. 配置环境变量：

```shell
. .\set_env.ps1
```

7. 编译 sdk：

```shell
# 如果你通过 .\install_opencv.ps1 安装 opencv，直接运行如下指令：
.\build_sdk.ps1
# 如果你自行安装了 opencv，需要指定 OpenCVConfig.cmake 的路径：
.\build_sdk.ps1 "path/to/folder/of/OpenCVConfig.cmake"
```

8. 可执行文件会在如下路径生成：

```shell
example\cpp\build\Release
```

**提示：**

- 基于 Pytorch 推理并不能达到 RTMPose 模型的真实推理速度，只用于模型效果验证。

```shell

```shell

    {PATH_TO_CHECKPOINT}/rtmdet_nano_8xb32-100e_coco-obj365-person-05d8511e.pth \
    {PATH_TO_CHECKPOINT}/rtmpose-m_simcc-aic-coco_pt-aic-coco_420e-256x192-63eb25f7_20230126.pth \
    --show

# 摄像头推理
    --show
```

效果展示：

{PATH_TO_CHECKPOINT}/rtmpose-m_simcc-aic-coco_pt-aic-coco_420e-256x192-63eb25f7_20230126.pth \
    --show
```

效果展示：

## 👨‍🏫 模型训练 [🔝](#-table-of-contents)

请参考 [训练与测试](https://mmpose.readthedocs.io/en/latest/user_guides/train_and_test.html) 进行 RTMPose 的训练。

**提示**：

- 当用户的数据集较小时请根据情况缩小 `batch_size` 和 `base_lr`。
- 模型选择
  - m：推荐首选使用
  - t/s：适用于极端低算力的移动设备，或对推理速度要求严格的场景
  - l：适用于算力较强、对速度不敏感的场景

## 🏗️ 部署教程 [🔝](#-table-of-contents)

本教程将展示如何通过 [MMDeploy](https://github.com/open-mmlab/mmdeploy/tree/main) 部署 RTMPose 项目。

### 🧩 安装

在开始部署之前，首先你需要确保正确安装了 MMPose, MMDetection, MMDeploy，相关安装教程如下：

### 🧩 安装

- [安装 MMPose 与 MMDetection](https://mmpose.readthedocs.io/zh_CN/latest/installation.html)
- [安装 MMDeploy](https://mmdeploy.readthedocs.io/zh_CN/latest/04-supported-codebases/mmpose.html)

根据部署后端的不同，有的后端需要对自定义算子进行编译，请根据需求前往对应的文档确保环境搭建正确：

- [ONNX](https://mmdeploy.readthedocs.io/zh_CN/latest/05-supported-backends/onnxruntime.html)
- [TensorRT](https://mmdeploy.readthedocs.io/zh_CN/latest/05-supported-backends/tensorrt.html)
- [OpenVINO](https://mmdeploy.readthedocs.io/zh_CN/latest/05-supported-backends/openvino.html)
- [更多](https://github.com/open-mmlab/mmdeploy/tree/main/docs/en/05-supported-backends)

### 🛠️ 模型转换

在完成安装之后，你就可以开始模型部署了。通过 MMDeploy 提供的 `tools/deploy.py` 可以方便地将 Pytorch 模型转换到不同的部署后端。

### 🛠️ 模型转换

我们本节演示将 RTMDet 和 RTMPose 模型导出为 ONNX 和 TensorRT 格式，如果你希望了解更多内容请前往 [MMDeploy 文档](https://mmdeploy.readthedocs.io/zh_CN/latest/02-how-to-run/convert_model.html)。

- ONNX 配置

  \- RTMDet：[`detection_onnxruntime_static.py`](https://github.com/open-mmlab/mmdeploy/blob/main/configs/mmdet/detection/detection_onnxruntime_static.py)

  \- RTMPose：[`pose-detection_simcc_onnxruntime_dynamic.py`](https://github.com/open-mmlab/mmdeploy/blob/main/configs/mmpose/pose-detection_simcc_onnxruntime_dynamic.py)

- TensorRT 配置

  \- RTMDet：[`detection_tensorrt_static-320x320.py`](https://github.com/open-mmlab/mmdeploy/blob/main/configs/mmdet/detection/detection_tensorrt_static-320x320.py)

  \- RTMPose：[`pose-detection_simcc_tensorrt_dynamic-256x192.py`](https://github.com/open-mmlab/mmdeploy/blob/main/configs/mmpose/pose-detection_simcc_tensorrt_dynamic-256x192.py)

如果你需要对部署配置进行修改，请参考 [MMDeploy config tutorial](https://mmdeploy.readthedocs.io/zh_CN/latest/02-how-to-run/write_config.html).

本教程中使用的文件结构如下：

```shell
|----mmdeploy
|----mmdetection
|----mmpose
```

#### ONNX

运行如下命令：

```shell
# 前往 mmdeploy 目录
cd ${PATH_TO_MMDEPLOY}

# 转换 RTMDet
    configs/mmdet/detection/detection_onnxruntime_static.py \
    ../mmpose/projects/rtmpose/rtmdet/person/rtmdet_nano_320-8xb32_coco-person.py \

https://download.openmmlab.com/mmpose/v1/projects/rtmpose/rtmdet_nano_8xb32-100e_coco-obj365-person-05d8511e.pth \
    --work-dir rtmpose-ort/rtmdet-nano \
    --show \
    --dump-info   # 导出 sdk info

# 转换 RTMPose
    ../mmpose/projects/rtmpose/rtmpose/body_2d_keypoint/rtmpose-m_8xb256-420e_coco-256x192.py \
    --work-dir rtmpose-ort/rtmpose-m \
    --show \
```

默认导出模型文件为 `{work-dir}/end2end.onnx`

#### TensorRT

运行如下命令：

```shell

# 转换 RTMDet

```

运行如下命令：

```shell

# 转换 RTMDet
    configs/mmdet/detection/detection_tensorrt_static-320x320.py \
    --work-dir rtmpose-trt/rtmdet-nano \
    --device cuda:0 \
    --show \

    configs/mmpose/pose-detection_simcc_tensorrt_dynamic-256x192.py \

    --work-dir rtmpose-trt/rtmpose-m \
    --show \
```

默认导出模型文件为 `{work-dir}/end2end.engine`

🎊 如果模型顺利导出，你将会看到样例图片上的检测结果：

![convert_models](https://user-images.githubusercontent.com/13503330/217726963-7815dd01-561a-4605-b0c6-07b6fe1956c3.png)

#### 高级设置

如果需要使用 TensorRT-FP16，你可以通过修改以下配置开启：

```Python
# in MMDeploy config
backend_config = dict(
    type='tensorrt',
    common_config=dict(
        fp16_mode=True  # 打开 fp16
    ))
```

### 🕹️ SDK 推理

要进行 Pipeline 推理，需要先用 MMDeploy 导出 SDK 版本的 det 和 pose 模型，只需要在参数中加上`--dump-info`。

此处以 onnxruntime 的 cpu 模型为例，运行如下命令：

```shell
# RTMDet
    configs/mmdet/detection/detection_onnxruntime_dynamic.py \

```shell
# RTMDet
    --show \

# RTMPose
    --show \

demo/resources/human-pose.jpg \
    --show \
    --dump-info  # 导出 sdk info
```

默认会导出三个 json 文件：

```shell
|----sdk
     |----end2end.onnx    # ONNX model
     |----end2end.engine  # TensorRT engine file

     |----pipeline.json   #
     |----deploy.json     # json files for the SDK
     |----detail.json     #
```

#### Python API

```Python
# Copyright (c) OpenMMLab. All rights reserved.

import cv2
from mmdeploy_runtime import PoseDetector

    parser = argparse.ArgumentParser(
        description='show how to use sdk python api')
    parser.add_argument('device_name', help='name of device, cuda or cpu')
        'model_path',
        help='path of mmdeploy SDK model dumped by model converter')
    parser.add_argument('image_path', help='path of an image')
        '--bbox',
        default=None,

parser.add_argument('image_path', help='path of an image')
        '--bbox',
        type=int,
        help='bounding box of an object in format (x, y, w, h)')

    img = cv2.imread(args.image_path)

    detector = PoseDetector(
        model_path=args.model_path, device_name=args.device_name, device_id=0)

    if args.bbox is None:
        result = detector(img)
    else:
        # converter (x, y, w, h) -> (left, top, right, bottom)
        print(args.bbox)
        bbox = np.array(args.bbox, dtype=int)
        bbox[2:] += bbox[:2]
        result = detector(img, bbox)
    print(result)

    _, point_num, _ = result.shape
    points = result[:, :, :2].reshape(point_num, 2)
    for [x, y] in points.astype(int):
        cv2.circle(img, (x, y), 1, (0, 255, 0), 2)

    cv2.imwrite('output_pose.png', img)

for [x, y] in points.astype(int):

    main()
```

#### C++ API

```C++
#include "mmdeploy/detector.hpp"

#include "opencv2/imgcodecs/imgcodecs.hpp"
#include "utils/argparse.h"
#include "utils/visualize.h"

DEFINE_ARG_string(model, "Model path");
DEFINE_ARG_string(image, "Input image path");
DEFINE_string(device, "cpu", R"(Device name, e.g. "cpu", "cuda")");
DEFINE_string(output, "detector_output.jpg", "Output image path");

DEFINE_double(det_thr, .5, "Detection score threshold");

  if (!utils::ParseArguments(argc, argv)) {
    return -1;
  }

  cv::Mat img = cv::imread(ARGS_image);
  if (img.empty()) {
    fprintf(stderr, "failed to load image: %s\n", ARGS_image.c_str());
    return -1;
  }

  // construct a detector instance
  mmdeploy::Detector detector(mmdeploy::Model{ARGS_model}, mmdeploy::Device{FLAGS_device});

return -1;
  }

  // apply the detector, the result is an array-like class holding references to
  // `mmdeploy_detection_t`, will be released automatically on destruction
  mmdeploy::Detector::Result dets = detector.Apply(img);

  // visualize
  utils::Visualize v;
  auto sess = v.get_session(img);
  int count = 0;
  for (const mmdeploy_detection_t& det : dets) {
    if (det.score > FLAGS_det_thr) {  // filter bboxes
      sess.add_det(det.bbox, det.label_id, det.score, det.mask, count++);
    }
  }

  if (!FLAGS_output.empty()) {
    cv::imwrite(FLAGS_output, sess.get());
  }

  return 0;
}
```

对于 C++ API 示例，请将 MMDeploy 加入到 CMake 项目中：

```CMake
find_package(MMDeploy REQUIRED)
target_link_libraries(${name} PRIVATE mmdeploy ${OpenCV_LIBS})
```

#### 其他语言

- [C# API 示例](https://github.com/open-mmlab/mmdeploy/tree/main/demo/csharp)

```CMake
```

#### 其他语言

- [JAVA API 示例](https://github.com/open-mmlab/mmdeploy/tree/main/demo/java)

### 🚀 Pipeline 推理

#### 图片推理

如果用户有跟随 MMDeploy 安装教程进行正确编译，在 `mmdeploy/build/bin/` 路径下会看到 `det_pose` 的可执行文件。

```shell
cd ${PATH_TO_MMDEPLOY}/build/bin/

# 单张图片推理
./det_pose rtmpose-ort/rtmdet-nano/ rtmpose-ort/rtmpose-m/ your_img.jpg --device cpu

required arguments:
  det_model           Detection 模型路径 [string]
  pose_model          Pose 模型路径 [string]
  image               输入图片路径 [string]

optional arguments:
  --device            推理设备 "cpu", "cuda" [string = "cpu"]
  --output            导出图片路径 [string = "det_pose_output.jpg"]
  --skeleton          骨架定义文件路径，或使用预定义骨架:
                      "coco" [string = "coco", "coco-wholoebody"]
  --det_label         用于姿势估计的检测标签 [int32 = 0]

--skeleton          骨架定义文件路径，或使用预定义骨架:
                      (0 在 coco 中对应 person)
  --det_thr           检测分数阈值 [double = 0.5]
  --det_min_bbox_size 最小检测框大小 [double = -1]
  --pose_thr          关键点置信度阈值 [double = 0]
```

**API** **示例**

\- [`det_pose.py`](https://github.com/open-mmlab/mmdeploy/blob/main/demo/python/det_pose.py)

\- [`det_pose.cxx`](https://github.com/open-mmlab/mmdeploy/blob/main/demo/csrc/cpp/det_pose.cxx)

#### 视频推理

如果用户有跟随 MMDeploy 安装教程进行正确编译，在 `mmdeploy/build/bin/` 路径下会看到 `pose_tracker` 的可执行文件。

- 将 `input` 输入 `0` 可以使用摄像头推理

```shell

# 视频推理
./pose_tracker rtmpose-ort/rtmdet-nano/ rtmpose-ort/rtmpose-m/ your_video.mp4 --device cpu

  det_model             Detection 模型路径 [string]
  pose_model            Pose 模型路径 [string]
  input                 输入图片路径或摄像头序号 [string]

  --device              推理设备 "cpu", "cuda" [string = "cpu"]
  --output              导出视频路径 [string = ""]
  --output_size         输出视频帧的长边 [int32 = 0]
  --flip                设置为1，用于水平翻转输入 [int32 = 0]
  --show                使用`cv::imshow`时，传递给`cv::waitKey`的延迟;
                        -1: 关闭 [int32 = 1]
  --skeleton            骨架数据的路径或预定义骨架的名称:
                        "coco", "coco-wholebody" [string = "coco"]
  --background          导出视频背景颜色, "default": 原图, "black":
                        纯黑背景 [string = "default"]
  --det_interval        检测间隔 [int32 = 1]
  --det_label           用于姿势估计的检测标签 [int32 = 0]
                        (0 在 coco 中对应 person)
  --det_thr             检测分数阈值 [double = 0.5]
  --det_min_bbox_size   最小检测框大小 [double = -1]
  --det_nms_thr         NMS IOU阈值，用于合并检测到的bboxes和

(0 在 coco 中对应 person)
                        追踪到的目标的 bboxes [double = 0.7]
  --pose_max_num_bboxes 每一帧用于姿势估计的 bboxes 的最大数量
                        [int32 = -1]
  --pose_kpt_thr        可见关键点的阈值 [double = 0.5]
  --pose_min_keypoints  有效姿势的最小关键点数量，-1表示上限(n_kpts/2) [int32 = -1]
  --pose_bbox_scale     将关键点扩展到 bbox 的比例 [double = 1.25]
  --pose_min_bbox_size  最小追踪尺寸，尺寸小于阈值的 bbox 将被剔除 [double = -1]
  --pose_nms_thr        用于抑制重叠姿势的 NMS OKS/IOU阈值。
                        当多个姿态估计重叠到同一目标时非常有用 [double = 0.5]
  --track_iou_thr       追踪 IOU 阈值 [double = 0.4]
  --track_max_missing   最大追踪容错 [int32 = 10]
```

\- [`pose_tracker.py`](https://github.com/open-mmlab/mmdeploy/blob/main/demo/python/pose_tracker.py)

\- [`pose_tracker.cxx`](https://github.com/open-mmlab/mmdeploy/blob/main/demo/csrc/cpp/pose_tracker.cxx)

## 📚 常用功能 [🔝](#-table-of-contents)

### 🚀 模型测速 [🔝](#-table-of-contents)

如果需要测试模型在部署框架下的推理速度，MMDeploy 提供了方便的 `tools/profiler.py` 脚本。

```shell
```

测试结果如下：

```shell

```

如果你希望详细了解 profiler 的更多参数设置与功能，可以前往 [Profiler Docs](https://mmdeploy.readthedocs.io/en/main/02-how-to-run/useful_tools.html#profiler)

### 📊 精度验证 [🔝](#-table-of-contents)

如果需要测试模型在部署框架下的推理精度，MMDeploy 提供了方便的 `tools/test.py` 脚本。

```shell
    --model {PATH_TO_MODEL}/rtmpose_m.pth \
    --device cpu
```

{RTMPOSE_PROJECT}/rtmpose/body_2d_keypoint/rtmpose-m_8xb256-420e_coco-256x192.py \
```

详细内容请参考 [MMDeploys Docs](https://github.com/open-mmlab/mmdeploy/blob/main/docs/zh_cn/02-how-to-run/profile_model.md)

## 📜 引用 [🔝](#-table-of-contents)

如果您觉得 RTMPose 对您的研究工作有所帮助，请考虑引用它：

```bibtex
}

@misc{mmpose2020,

title = {RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose},
}

    title={OpenMMLab Pose Estimation Toolbox and Benchmark},
    author={MMPose Contributors},
    howpublished = {\url{https://github.com/open-mmlab/mmpose}},
}
```

# Preparing SSDD Dataset

```bibtex
@ARTICLE{SSDD2021,
author = {T. Zhang, X. Zhang, J. Li and X. Xu},
title = {SAR ship detection dataset (SSDD): Official release and comprehensive data analysis},
journal = {Remote Senseing},
month = {Sep.},
year = {2021}
volume={13},
number={18},
pages={3690},
}
```

## Download SSDD dataset

The SSDD dataset can be downloaded from [Google drive](https://drive.google.com/file/d/1LmoHBk4xUvm0Zdtm8X7256dHigyFW4Nh/view?usp=sharing).

```none
mmrotate
├── tools
├── data
│   ├── ssdd
│   │   ├── test
│   │   │   ├── all
│   │   │   ├── inshore
│   │   │   ├── offshore
```

Please change `data_root` in `configs/_base_/datasets/ssdd.py` to `data/ssdd/`.

Does the modification introduce changes that break the backward compatibility of the downstream repositories?

If this PR introduces a new feature, it is better to list some use cases here and update the documentation.

- [ ] Pre-commit or other linting tools are used to fix the potential lint issues.

- [ ] Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
- [ ] The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
- [ ] The documentation has been modified accordingly, like docstring or example tutorials.

- [ ] If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects, like MMDet or MMSeg.

# 准备数据集 (待更新)

具体的细节可以参考 [准备数据](https://github.com/open-mmlab/mmrotate/tree/main/tools/data) 下载并组织数据集。

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_cvpr_2014/html/Toshev_DeepPose_Human_Pose_2014_CVPR_paper.html">DeepPose (CVPR'2014)</a></summary>

```bibtex
@inproceedings{toshev2014deeppose,
  title={Deeppose: Human pose estimation via deep neural networks},
  author={Toshev, Alexander and Szegedy, Christian},
  pages={1653--1660},
  year={2014}
}
```

<details>
<summary align="right"><a href="https://arxiv.org/abs/2107.11291">RLE (ICCV'2021)</a></summary>

```bibtex
@inproceedings{li2021human,
  title={Human pose regression with residual log-likelihood estimation},
  author={Li, Jiefeng and Bian, Siyuan and Zeng, Ailing and Wang, Can and Pang, Bo and Liu, Wentao and Lu, Cewu},
  pages={11025--11034},
}
```

booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
}
```

<details>

```bibtex
}
```

<details>

```bibtex

```bibtex
}
```

| [deeppose_resnet_50_rle](/configs/body_2d_keypoint/topdown_regression/mpii/td-reg_res50_rle-8xb64-210e_mpii-256x256.py) |  256x256   | 0.861 |  0.277   | [ckpt](https://download.openmmlab.com/mmpose/top_down/deeppose/deeppose_res50_mpii_256x256_rle-5f92a619_20220504.pth) | [log](https://download.openmmlab.com/mmpose/top_down/deeppose/deeppose_res50_mpii_256x256_rle_20220504.log.json) |

# Inception V3

> [Rethinking the Inception Architecture for Computer Vision](http://arxiv.org/abs/1512.00567)

Convolutional networks are at the core of most state-of-the-art computer vision solutions for a wide variety of tasks. Since 2014 very deep convolutional networks started to become mainstream, yielding substantial gains in various benchmarks. Although increased model size and computational cost tend to translate to immediate quality gains for most tasks (as long as enough labeled data is provided for training), computational efficiency and low parameter count are still enabling factors for various use cases such as mobile vision and big-data scenarios. Here we explore ways to scale up networks in ways that aim at utilizing the added computation as efficiently as possible by suitably factorized convolutions and aggressive regularization. We benchmark our methods on the ILSVRC 2012 classification challenge validation set demonstrate substantial gains over the state of the art: 21.2% top-1 and 5.6% top-5 error for single frame evaluation using a network with a computational cost of 5 billion multiply-adds per inference and with using less than 25 million parameters. With an ensemble of 4 models and multi-crop evaluation, we report 3.5% top-5 error on the validation set (3.6% error on the test set) and 17.3% top-1 error on the validation set.

<img src="https://user-images.githubusercontent.com/26739999/177241797-c103eff4-79bb-414d-aef6-eac323b65a50.png" width="40%"/>
</div>

```python

predict = inference_model('inception-v3_3rdparty_8xb32_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('inception-v3_3rdparty_8xb32_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/inception_v3/inception-v3_8xb32_in1k.py https://download.openmmlab.com/mmclassification/v0/inception-v3/inception-v3_3rdparty_8xb32_in1k_20220615-dcd4d910.pth
```

| Model                                |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                Config                |                                    Download                                     |
| :----------------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :----------------------------------: | :-----------------------------------------------------------------------------: |

| `inception-v3_3rdparty_8xb32_in1k`\* | From scratch |   23.83    |   5.75    |   77.57   |   93.58   | [config](inception-v3_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/inception-v3/inception-v3_3rdparty_8xb32_in1k_20220615-dcd4d910.pth) |

*Models with * are converted from the [official repo](https://github.com/pytorch/vision/blob/main/torchvision/models/inception.py#L28). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@inproceedings{szegedy2016rethinking,
  title={Rethinking the inception architecture for computer vision},
  author={Szegedy, Christian and Vanhoucke, Vincent and Ioffe, Sergey and Shlens, Jon and Wojna, Zbigniew},
  pages={2818--2826},
}
```

# NAFNET (ECCV'2022)

> **任务**: 图像恢复

<details>
<summary align="right">NAFNET (ECCV'2022)</summary>

```bibtex
@article{chen2022simple,
  title={Simple Baselines for Image Restoration},
  author={Chen, Liangyu and Chu, Xiaojie and Zhang, Xiangyu and Sun, Jian},
  journal={arXiv preprint arXiv:2204.04676},
}
```

|                                     方法                                      | 图片尺寸 |       PSNR       |      SSIM      | GPU信息  |                                     下载                                      |
| :---------------------------------------------------------------------------: | :------: | :--------------: | :------------: | :------: | :---------------------------------------------------------------------------: |

| [nafnet_c64eb2248mb12db2222_8xb8-lr1e-3-400k_sidd](./nafnet_c64eb2248mb12db2222_8xb8-lr1e-3-400k_sidd.py) | 256X256  | 40.3045(40.3045) | 0.9253(0.9614) | 1 (A100) | [模型](https://download.openmmlab.com/mmediting/nafnet/NAFNet-SIDD-midc64.pth) \| 日志(即将到来) |
| [nafnet_c64eb11128mb1db1111_8xb8-lr1e-3-400k_gopro](./nafnet_c64eb11128mb1db1111_8xb8-lr1e-3-400k_gopro.py) | 1280x720 | 33.7246(33.7103) | 0.9479(0.9668) | 1 (A100) | [模型](https://download.openmmlab.com/mmediting/nafnet/NAFNet-GoPro-midc64.pth) \| 日志(即将到来) |

Note:

- 评估结果a(b)中，a代表由MMagic测量，b代表由原论文提供。
- PSNR是在RGB通道评估。
- SSIM是平均的分别在RGB通道评估的SSIM, 而原论文使用了3D的SSIM卷积核做统一评估。

## 快速开始

**训练**

<details>

```shell
# CPU上训练
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/nafnet/nafnet_c64eb2248mb12db2222_8xb8-lr1e-3-400k_sidd.py

# 单个GPU上训练
python tools/train.py configs/nafnet/nafnet_c64eb2248mb12db2222_8xb8-lr1e-3-400k_sidd.py

# 多个GPU上训练

# 单个GPU上训练

# 多个GPU上训练
./tools/dist_train.sh configs/nafnet/nafnet_c64eb2248mb12db2222_8xb8-lr1e-3-400k_sidd.py 8
```

更多细节可以参考 [train_test.md](../../docs/zh_cn/user_guides/train_test.md) 中的 **Train a model** 部分。

**测试**

<details>

```shell
# CPU上测试
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/nafnet/nafnet_c64eb2248mb12db2222_8xb8-lr1e-3-400k_sidd.py /path/to/checkpoint

# 单个GPU上测试
python tools/test.py configs/nafnet/nafnet_c64eb2248mb12db2222_8xb8-lr1e-3-400k_sidd.py /path/to/checkpoint

# 多个GPU上测试
./tools/dist_test.sh configs/nafnet/nafnet_c64eb2248mb12db2222_8xb8-lr1e-3-400k_sidd.py /path/to/checkpoint 8
```

预训练模型未来将会上传，敬请等待。
更多细节可以参考 [train_test.md](../../docs/zh_cn/user_guides/train_test.md) 中的 **Test a pre-trained model** 部分。

# Win10 下预编译包的使用

- [Win10 下预编译包的使用](#win10-下预编译包的使用)
  - [准备工作](#准备工作)
    - [ONNX Runtime](#onnx-runtime)
    - [TensorRT](#tensorrt)
  - [模型转换](#模型转换)
    - [ONNX Runtime Example](#onnx-runtime-example)
    - [TensorRT Example](#tensorrt-example)
  - [模型推理](#模型推理)
    - [Backend Inference](#backend-inference)
      - [ONNXRuntime](#onnxruntime)
      - [TensorRT](#tensorrt-1)
    - [Python SDK](#python-sdk)
      - [ONNXRuntime](#onnxruntime-1)
      - [TensorRT](#tensorrt-2)
    - [C SDK](#c-sdk)
      - [ONNXRuntime](#onnxruntime-2)
      - [TensorRT](#tensorrt-3)
  - [可能遇到的问题](#可能遇到的问题)

目前，`MMDeploy`在`Windows`平台下提供`cpu`以及`cuda`两种Device的预编译包，其中`cpu`版支持使用onnxruntime cpu进行推理，`cuda`版支持使用onnxruntime-gpu以及tensorrt进行推理，可以从[Releases](https://github.com/open-mmlab/mmdeploy/releases)获取。。

本篇教程以`mmdeploy-1.2.0-windows-amd64.zip`和`mmdeploy-1.2.0-windows-amd64-cuda11.3.zip`为例，展示预编译包的使用方法。

为了方便使用者快速上手，本教程以分类模型(mmpretrain)为例，展示两种预编译包的使用方法。

预编译包的目录结构如下。

```
.
├── build_sdk.ps1
├── example
├── install_opencv.ps1
├── lib
├── set_env.ps1
└── thirdparty
```

## 准备工作

使用预编译包来进行`模型转换`以及`模型推理`，除了预编译包的中的内容外，还需要安装一些第三方依赖库，下面分别介绍以`ONNX Runtime`、`TensorRT`为推理后端所要进行的准备工作。

两种推理后端环境准备工作中，其中一些操作是共有的，下面先介绍这些共有的操作，再分别介绍各自特有的操作。

首先新建一个工作目录workspace

1. 请按照[get_started](../get_started.md)文档，准备虚拟环境，安装pytorch、torchvision、mmcv。若要使用SDK的C接口，需要安装vs2019+, OpenCV。

   :point_right: 这里建议使用`pip`而不是`conda`安装pytorch、torchvision

2. 克隆mmdeploy仓库

   ```bash
   git clone -b main https://github.com/open-mmlab/mmdeploy.git
   ```

   :point_right: 这里主要为了使用configs文件，所以没有加`--recursive`来下载submodule，也不需要编译`mmdeploy`

3. 安装mmpretrain

   ```bash
   git clone -b main https://github.com/open-mmlab/mmpretrain.git
   cd mmpretrain
   pip install -e .
   ```

4. 准备一个PyTorch的模型文件当作我们的示例

   ```bash
   ```

   这里选择了[resnet18_8xb32_in1k_20210831-fbbb1da6.pth](https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_8xb32_in1k_20210831-fbbb1da6.pth)，对应的训练config为[resnet18_8xb32_in1k.py](https://github.com/open-mmlab/mmpretrain/blob/main/configs/resnet/resnet18_8xb32_in1k.py)

做好以上工作后，当前工作目录的结构应为：

```
.
|-- mmpretrain
|-- mmdeploy
|-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth
```

本节介绍`mmdeploy`使用`ONNX Runtime`推理所特有的环境准备工作

5. 安装`mmdeploy`（模型转换）以及`mmdeploy_runtime`（模型推理Python API）的预编译包

   ```bash
   pip install mmdeploy==1.2.0
   pip install mmdeploy-runtime==1.2.0
   ```

   :point_right: 如果之前安装过，需要先卸载后再安装。

6. 安装onnxruntime package

   ```
   pip install onnxruntime==1.8.1
   ```

7. 下载[`onnxruntime`](https://github.com/microsoft/onnxruntime/releases/tag/v1.8.1)，添加环境变量

   ```
   ```

   将onnxruntime的lib目录添加到PATH里面，如图所示，具体的路径根据个人情况更改。

   ![sys-path](https://user-images.githubusercontent.com/16019484/181463801-1d7814a8-b256-46e9-86f2-c08de0bc150b.png)
   :exclamation: 重启powershell让环境变量生效，可以通过 echo $env:PATH 来检查是否设置成功。

8. 下载 SDK C/cpp Library mmdeploy-1.2.0-windows-amd64.zip

本节介绍`mmdeploy`使用`TensorRT`推理所特有的环境准备工作

   ```bash
   pip install mmdeploy-runtime-gpu==1.2.0
   ```

   :point_right: 如果之前安装过，需要先卸载后再安装

6. 安装CUDA相关内容，并设置环境变量

   - CUDA Toolkit 11.1
   - TensorRT 8.2.3.0 (python包 + 环境变量)
   - cuDNN 8.2.1.0

   其中CUDA的环境变量在安装CUDA Toolkit后会自动添加，TensorRT以及cuDNN解压后需要自行添加运行库的路径到PATH，可参考onnxruntime的设置图例

   :exclamation: 重启powershell让环境变量生效，可以通过 echo $env:PATH 来检查是否设置成功

- cuDNN 8.2.1.0

   :exclamation: 建议只添加一个版本的TensorRT的lib到PATH里面。不建议拷贝TensorRT的dll到C盘的cuda目录，在某些情况下，这样可以暴露dll的版本问题

7. 安装pycuda `pip install pycuda`

8. 下载 SDK C/cpp Library mmdeploy-1.2.0-windows-amd64-cuda11.3.zip

## 模型转换

### ONNX Runtime Example

下面介绍根据之前下载的ckpt来展示如果使用`mmdeploy`预编译包来进行模型转换

经过之前的准备工作，当前的工作目录结构应该为：

```
..
|-- mmdeploy-1.2.0-windows-amd64
`-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth
```

python 转换代码

```python
from mmdeploy.apis import torch2onnx
from mmdeploy.backend.sdk.export_info import export2SDK

img = 'mmpretrain/demo/demo.JPEG'
work_dir = 'work_dir/onnx/resnet'
save_file = 'end2end.onnx'
deploy_cfg = 'mmdeploy/configs/mmpretrain/classification_onnxruntime_dynamic.py'
model_cfg = 'mmpretrain/configs/resnet/resnet18_8xb32_in1k.py'

model_checkpoint = 'resnet18_8xb32_in1k_20210831-fbbb1da6.pth'

# 1. convert model to onnx
torch2onnx(img, work_dir, save_file, deploy_cfg, model_cfg,
  model_checkpoint, device)

# 2. extract pipeline info for sdk use (dump-info)
export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint, device=device)
```

转换后的模型目录结构应该为：

```bash
.\work_dir\
`-- onnx
    `-- resnet
        |-- deploy.json
        |-- detail.json
        |-- end2end.onnx
        `-- pipeline.json
```

### TensorRT Example

下面根据之前下载的ckpt来展示如果使用mmdeploy预编译包来进行模型转换

```
..
|-- mmdeploy-1.2.0-windows-amd64-cuda11.3
```

```python
from mmdeploy.apis.tensorrt import onnx2tensorrt

```

```python
import os

work_dir = 'work_dir/trt/resnet'
deploy_cfg = 'mmdeploy/configs/mmpretrain/classification_tensorrt_static-224x224.py'

# 1. convert model to IR(onnx)

# 2. convert IR to tensorrt
onnx_model = os.path.join(work_dir, save_file)
save_file = 'end2end.engine'
model_id = 0
device = 'cuda'
onnx2tensorrt(work_dir, save_file, model_id, deploy_cfg, onnx_model, device)

# 3. extract pipeline info for sdk use (dump-info)

```

```
`-- trt
    `-- resnet
        |-- end2end.engine
```

## 模型推理

以下内容假定已完成了上述模型转换的两个Example，并得到了上述模型转换后的两个文件夹其中之一或者全部：

```
.\work_dir\onnx\resnet
.\work_dir\trt\resnet
```

当前的工作目录应为：

```
.
`-- work_dir
```

### Backend Inference

:exclamation: 需要强调的一点是，这个接口不是为了做部署的，而是屏蔽了推理后端接口的，用来检验转换的模型是否可以正常推理的。

#### ONNXRuntime

Python 代码

```python
from mmdeploy.apis import inference_model

Python 代码

```python

backend_files = ['work_dir/onnx/resnet/end2end.onnx']
result = inference_model(model_cfg, deploy_cfg, backend_files, img, device)
```

Python 代码

```python

backend_files = ['work_dir/trt/resnet/end2end.engine']
```

### Python SDK

这里介绍如何使用SDK的Python API进行推理

推理代码

```bash

```

推理代码

```bash
python .\mmdeploy\demo\python\image_classification.py cpu .\work_dir\onnx\resnet\ .\mmpretrain\demo\demo.JPEG
```

推理代码

```bash
 python .\mmdeploy\demo\python\image_classification.py cuda .\work_dir\trt\resnet\ .\mmpretrain\demo\demo.JPEG
```

### C SDK

这里介绍如何使用SDK的C API进行推理

1. 添加环境变量

   可根据 SDK 文件夹内的 README.md 使用脚本添加环境变量

2. 编译 examples

   可根据 SDK 文件夹内的 README.md 使用脚本进行编译

3. 推理：

   这里建议使用cmd，这样如果exe运行时如果找不到相关的dll的话会有弹窗

   在mmdeploy-1.2.0-windows-amd64\\example\\cpp\\build\\Release目录下：

   ```
   .\image_classification.exe cpu C:\workspace\work_dir\onnx\resnet\ C:\workspace\mmpretrain\demo\demo.JPEG
   ```

1. 添加环境变量

3. 推理

```

1. 添加环境变量

3. 推理

   在mmdeploy-1.2.0-windows-amd64-cuda11.3\\example\\cpp\\build\\Release目录下：

   ```
   .\image_classification.exe cuda C:\workspace\work_dir\trt\resnet C:\workspace\mmpretrain\demo\demo.JPEG
   ```

## 可能遇到的问题

如遇到问题，可参考[FAQ](../faq.md)

# 结果展示

在评测完成后，评测的结果需要被打印到屏幕或者被保存下来，该过程是由 summarizer 控制的。

```{note}
如果 summarizer 出现在了 config 中，则评测结果输出会按照下述逻辑进行。
如果 summarizer 没有出现在 config 中，则评测结果会按照 `dataset` 中出现的顺序进行输出。
```

## 样例

一个典型的 summerizer 配置文件如下：

```python
summarizer = dict(
    dataset_abbrs = [
        'race',
        'race-high',
        'race-middle',
    ],
    summary_groups=[
        {'name': 'race', 'subsets': ['race-high', 'race-middle']},
    ]
)
```

其输出结果如下：

```text
dataset      version    metric         mode      internlm-7b-hf
-----------  ---------  -------------  ------  ----------------
race         -          naive_average  ppl                76.23
race-high    0c332f     accuracy       ppl                74.53
race-middle  0c332f     accuracy       ppl                77.92
```

```

summarizer 会以 config 中的 `models`, `datasets` 为全集，去尝试读取 `{work_dir}/results/` 路径下的评测分数，并按照 `summarizer.dataset_abbrs` 列表的顺序进行展示。另外，summarizer 会尝试通过 `summarizer.summary_groups` 来进行一些汇总指标的计算。当且仅当 `subsets` 中的值都存在时，对应的 `name` 指标才会生成，这也就是说，若有部分数字缺失，则这个汇总指标也是会缺失的。若分数无法通过上述两种方式被获取到，则 summarizer 会在表格中对应项处使用 `-` 进行表示。

此外，输出结果是有多列的：

- `dataset` 列与 `summarizer.dataset_abbrs` 配置一一对应
- `version` 列是这个数据集的 hash 值，该 hash 值会考虑该数据集模板的评测方式、提示词、输出长度限制等信息。用户可通过该列信息确认两份评测结果是否可比
- `metric` 列是指这个指标的评测方式，具体说明见 [metrics](./metrics.md)
- `mode` 列是指这个推理结果的获取方式，可能的值有 `ppl` / `gen`。对于 `summarizer.summary_groups` 的项，若被 `subsets` 的获取方式都一致，则其值也跟 `subsets` 一致，否则即为 `mixed`
- 其后若干列，一列代表一个模型

## 完整字段说明

summarizer 字段说明如下

- `dataset_abbrs`: (list，可选) 展示列表项。若该项省略，则会输出全部评测结果。
- `summary_groups`: (list，可选) 汇总指标配置。

## 完整字段说明

`summary_groups` 中的字段说明如下：

- `name`: (str) 汇总指标的名称。
- `subsets`: (list) 被汇总指标的名称。注意它不止可以是原始的 `dataset_abbr`，也可以是另一个汇总指标的名称。
- `weights`: (list，可选) 被汇总指标的权重。若该项省略，则默认使用不加权的求平均方法。

注意，我们在 `configs/summarizers/groups` 路径下存放了 MMLU, C-Eval 等数据集的评测结果汇总，建议优先考虑使用。

# Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_cvpr_2018/html/Feng_Wing_Loss_for_CVPR_2018_paper.html">Wingloss (CVPR'2018)</a></summary>

```bibtex
@inproceedings{feng2018wing,
  title={Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks},
  author={Feng, Zhen-Hua and Kittler, Josef and Awais, Muhammad and Huber, Patrik and Wu, Xiao-Jun},
  booktitle={Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on},
  pages ={2235-2245},
}
```

We present a new loss function, namely Wing loss, for robust facial landmark localisation with Convolutional Neural Networks (CNNs). We first compare and analyse different loss functions including L2, L1 and smooth L1. The analysis of these loss functions suggests that, for the training of a CNN-based localisation model, more attention should be paid to small and medium range errors. To this end, we design a piece-wise loss function. The new loss amplifies the impact of errors from the interval (-w, w) by switching from L1 loss to a modified logarithm function. To address the problem of under-representation of samples with large out-of-plane head rotations in the training set, we propose a simple but effective boosting strategy, referred to as pose-based data balancing. In particular, we deal with the data imbalance problem by duplicating the minority training samples and perturbing them by injecting random image rotation, bounding box translation and other data augmentation approaches. Last, the proposed approach is extended to create a two-stage framework for robust facial landmark localisation. The experimental results obtained on AFLW and 300W demonstrate the merits of the Wing loss function, and prove the superiority of the proposed method over the state-of-the-art approaches.

<img src="https://user-images.githubusercontent.com/15977946/146528473-f3228f69-b60e-4807-9c4b-b45997fbc530.png">
</div>

# Breast Cancer Cell Segmentation

This project support **`Breast Cancer Cell Segmentation`**, and the dataset used in this project can be downloaded from [here](https://tianchi.aliyun.com/dataset/dataDetail?dataId=90152).

In this dataset, there are 58 H&E stained histopathology images used in breast cancer cell detection with associated ground truth data available. Routine histology uses the stain combination of hematoxylin and eosin, commonly referred to as H&E. These images are stained since most cells are essentially transparent, with little or no intrinsic pigment. Certain special stains, which bind selectively to particular components, are be used to identify biological structures such as cells. In those images, the challenging problem is cell segmentation for subsequent classification in benign and malignant cells.

| Dataset name                                                                                  | Anatomical region | Task type    | Modality       | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License                                                                                                |
| --------------------------------------------------------------------------------------------- | ----------------- | ------------ | -------------- | ------------ | --------------------- | ---------------------- | ------------ | ------------------------------------------------------------------------------------------------------ |

| [Breast Cancer Cell Segmentation](https://tianchi.aliyun.com/dataset/dataDetail?dataId=90152) | thorax            | segmentation | histopathology | 2            | 58/-/-                | yes/-/-                | 2021         | [CC-BY-SA-NC 4.0](http://creativecommons.org/licenses/by-sa/4.0/?spm=5176.12282016.0.0.3f5b5291ypBxb2) |

|     Class Name     | Num. Train | Pct. Train | Num. Val | Pct. Val | Num. Test | Pct. Test |
| :----------------: | :--------: | :--------: | :------: | :------: | :-------: | :-------: |
|       normal       |     58     |   98.37    |    -     |    -     |     -     |     -     |
| breast cancer cell |     58     |    1.63    |    -     |    -     |     -     |     -     |

Note:

![bac](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/histopathology/breast_cancer_cell_seg/breast_cancer_cell_seg_dataset.png)

```
@inproceedings{gelasca2008evaluation,
  title={Evaluation and benchmark for biological image segmentation},
  author={Gelasca, Elisa Drelie and Byun, Jiyun and Obara, Boguslaw and Manjunath, BS},
  booktitle={2008 15th IEEE international conference on image processing},
  pages={1816--1819},
  year={2008},
}
```

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `breast_cancer_cell_seg/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
```

- download dataset from [here](https://tianchi.aliyun.com/dataset/dataDetail?dataId=90152) and decompression data to path `'data/'`.

```none
  ├── mmseg
  │   │   │   │   ├── breast_cancer_cell_seg

```

|  Class Name  | Num. Train | Pct. Train | Num. Val | Pct. Val | Num. Test | Pct. Test |
| :----------: | :--------: | :--------: | :------: | :------: | :-------: | :-------: |
|  background  |     46     |   98.36    |    12    |  98.41   |     -     |     -     |
| erythrocytes |     46     |    1.64    |    12    |   1.59   |     -     |     -     |

Train models on a single server with one GPU.

```shell
```

Test models on a single server with one GPU.

```shell
```

```shell
```

# 标准与模型库

## 共同设定

- 我们默认使用 4 卡分布式训练
- 所有 PyTorch 风格的 ImageNet 预训练网络由我们自己训练，和 [论文](https://arxiv.org/pdf/1812.01187.pdf) 保持一致。
  我们的 ResNet 网络是基于 ResNetV1c 的变种，在这里输入层的 7x7 卷积被 3个 3x3 取代
- 为了在不同的硬件上保持一致，我们以 `torch.cuda.max_memory_allocated()` 的最大值作为 GPU 占用率，同时设置 `torch.backends.cudnn.benchmark=False`。
  注意，这通常比 `nvidia-smi` 显示的要少
- 我们以网络 forward 和后处理的时间加和作为推理时间，除去数据加载时间。我们使用脚本 `tools/benchmark.py` 来获取推理时间，它在 `torch.backends.cudnn.benchmark=False` 的设定下，计算 200 张图片的平均推理时间
- 在框架中，有两种推理模式
  - `slide` 模式（滑动模式）：测试的配置文件字段 `test_cfg` 会是 `dict(mode='slide', crop_size=(769, 769), stride=(513, 513))`.
    在这个模式下，从原图中裁剪多个小图分别输入网络中进行推理。小图的大小和小图之间的距离由 `crop_size` 和 `stride` 决定，重合区域会进行平均
  - `whole` 模式 （全图模式）：测试的配置文件字段 `test_cfg` 会是 `dict(mode='whole')`. 在这个模式下，全图会被直接输入到网络中进行推理。
    对于 769x769 下训练的模型，我们默认使用 `slide` 进行推理，其余模型用 `whole` 进行推理
- 对于输入大小为 8x+1 （比如769），我们使用 `align_corners=True`。其余情况，对于输入大小为 8x (比如 512，1024)，我们使用 `align_corners=False`

## 基线

### FCN

对于 769x769 下训练的模型，我们默认使用 `slide` 进行推理，其余模型用 `whole` 进行推理

## 基线

### FCN

请参考 [FCN](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/fcn) 获得详细信息。

### PSPNet

请参考 [PSPNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/pspnet) 获得详细信息。

请参考 [DeepLabV3](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/deeplabv3) 获得详细信息。

### PSANet

请参考 [PSANet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/psanet) 获得详细信息。

请参考 [DeepLabV3+](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/deeplabv3plus) 获得详细信息。

请参考 [UPerNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/upernet) 获得详细信息。

请参考 [NonLocal Net](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/nlnet) 获得详细信息。

### EncNet

### EncNet

请参考 [EncNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/encnet) 获得详细信息。

### CCNet

请参考 [CCNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/ccnet) 获得详细信息。

### DANet

请参考 [DANet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/danet) 获得详细信息。

### APCNet

请参考 [APCNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/apcnet) 获得详细信息。

### HRNet

请参考 [HRNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/hrnet) 获得详细信息。

### GCNet

请参考 [GCNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/gcnet) 获得详细信息。

### DMNet

请参考 [DMNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/dmnet) 获得详细信息。

### ANN

请参考 [ANN](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/ann) 获得详细信息。

### OCRNet

### ANN

### OCRNet

请参考 [OCRNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/ocrnet) 获得详细信息。

请参考 [Fast-SCNN](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/fastscnn) 获得详细信息。

请参考 [ResNeSt](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/resnest) 获得详细信息。

请参考 [Semantic FPN](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/semfpn) 获得详细信息。

请参考 [PointRend](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/point_rend) 获得详细信息。

请参考 [MobileNetV2](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/mobilenet_v2) 获得详细信息。

请参考 [MobileNetV3](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/mobilenet_v3) 获得详细信息。

### EMANet

### EMANet

请参考 [EMANet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/emanet) 获得详细信息。

### DNLNet

请参考 [DNLNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/dnlnet) 获得详细信息。

### CGNet

请参考 [CGNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/cgnet) 获得详细信息。

请参考 [Mixed Precision (FP16) Training 在 BiSeNetV2 训练的样例](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/bisenetv2/bisenetv2_fcn_fp16_4x4_1024x1024_160k_cityscapes.py) 获得详细信息。

## 速度标定（待更新）

### 硬件

### 软件环境

- CUDA 10.1

### 训练速度

为了公平比较，我们全部使用 ResNet-101V1c 进行标定。输入大小为 1024x512，批量样本数为 2。

训练速度如下表，指标为每次迭代的时间，以秒为单位，越低越快。

### 软件环境

- CUDA 10.1

### 训练速度

注意：DeepLabV3+ 的输出步长为 8。

# RTMPose

Recent studies on 2D pose estimation have achieved excellent performance on public benchmarks, yet its application in the industrial community still suffers from heavy model parameters and high latency.
In order to bridge this gap, we empirically study five aspects that affect the performance of multi-person pose estimation algorithms: paradigm, backbone network, localization algorithm, training strategy, and deployment inference, and present a high-performance real-time multi-person pose estimation framework, **RTMPose**, based on MMPose.
Our RTMPose-m achieves **75.8% AP** on COCO with **90+ FPS** on an Intel i7-11700 CPU and **430+ FPS** on an NVIDIA GTX 1660 Ti GPU, and RTMPose-l achieves **67.0% AP** on COCO-WholeBody with **130+ FPS**, outperforming existing open-source libraries.
To further evaluate RTMPose's capability in critical real-time applications, we also report the performance after deploying on the mobile device.

### AP-10K Dataset

Results on AP-10K validation set

|   Model   | Input Size |  AP   |             Details and Download             |
| :-------: | :--------: | :---: | :------------------------------------------: |
| RTMPose-m |  256x256   | 0.722 | [rtmpose_cp10k.md](./ap10k/rtmpose_ap10k.md) |

# Mask R-CNN

> [Mask R-CNN](https://arxiv.org/abs/1703.06870)

We present a conceptually simple, flexible, and general framework for object instance segmentation. Our approach efficiently detects objects in an image while simultaneously generating a high-quality segmentation mask for each instance. The method, called Mask R-CNN, extends Faster R-CNN by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition. Mask R-CNN is simple to train and adds only a small overhead to Faster R-CNN, running at 5 fps. Moreover, Mask R-CNN is easy to generalize to other tasks, e.g., allowing us to estimate human poses in the same framework. We show top results in all three tracks of the COCO suite of challenges, including instance segmentation, bounding-box object detection, and person keypoint detection. Without bells and whistles, Mask R-CNN outperforms all existing, single-model entries on every task, including the COCO 2016 challenge winners. We hope our simple and effective approach will serve as a solid baseline and help ease future research in instance-level recognition.

<img src="https://user-images.githubusercontent.com/40661020/143967081-c2552bed-9af2-46c4-ae44-5b3b74e5679f.png"/>
</div>

We support and provide some baseline results on [nuImages dataset](https://www.nuscenes.org/nuimages).
We follow the class mapping in nuScenes dataset, which maps the original categories into 10 foreground categories.
The convert script can be found [here](https://github.com/open-mmlab/mmdetection3d/blob/master/tools/dataset_converters/nuimage_converter.py).
The baseline results include instance segmentation models, e.g., Mask R-CNN, Cascade Mask R-CNN, and HTC.
We will support panoptic segmentation models in the future.

![demo image](../../resources/nuimages_demo.gif)

The dataset converted by the script of v0.6.0 only supports instance segmentation. Since v0.7.0, we also support to produce semantic segmentation mask of each image; thus, we can train HTC or semantic segmentation models using the dataset. To convert the nuImages dataset into COCO format, please use the command below:

```shell
python -u tools/dataset_converters/nuimage_converter.py --data-root ${DATA_ROOT} --version ${VERSIONS} \
                                                    --out-dir ${OUT_DIR} --nproc ${NUM_WORKERS} --extra-tag ${TAG}
```

- `--data-root`: the root of the dataset, defaults to `./data/nuimages`.
- `--version`: the version of the dataset, defaults to `v1.0-mini`. To get the full dataset, please use `--version v1.0-train v1.0-val v1.0-mini`
- `--out-dir`: the output directory of annotations and semantic masks, defaults to `./data/nuimages/annotations/`.

- `--nproc`: number of workers for data preparation, defaults to `4`. Larger number could reduce the preparation time as images are processed in parallel.
- `--extra-tag`: extra tag of the annotations, defaults to `nuimages`. This can be used to separate different annotations processed in different time for study.

### Instance Segmentation

We report Mask R-CNN and Cascade Mask R-CNN results on nuimages.

|       Method       |                                       Backbone                                        | Pretraining | Lr schd | Mem (GB) |                                                                                   Box AP                                                                                   | Mask AP |                                                                                                                                                                                                                        Download                                                                                                                                                                                                                        |

|     Mask R-CNN     |                        [R-50](./mask-rcnn_r50_fpn_1x_nuim.py)                         |     IN      |   1x    |   7.4    |                                                                                    47.8                                                                                    |  38.4   |                                                         [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/mask_rcnn_r50_fpn_1x_nuim/mask_rcnn_r50_fpn_1x_nuim_20201008_195238-e99f5182.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/mask_rcnn_r50_fpn_1x_nuim/mask_rcnn_r50_fpn_1x_nuim_20201008_195238.log.json)                                                         |

|     Mask R-CNN     |                    [R-50](./mask-rcnn_r50_fpn_coco-2x_1x_nuim.py)                     | IN+COCO-2x  |   1x    |   7.4    |                                                                                    49.7                                                                                    |  40.5   |                                         [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/mask_rcnn_r50_fpn_coco-2x_1x_nuim/mask_rcnn_r50_fpn_coco-2x_1x_nuim_20201008_195238-b1742a60.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/mask_rcnn_r50_fpn_coco-2x_1x_nuim/mask_rcnn_r50_fpn_coco-2x_1x_nuim_20201008_195238.log.json)                                         |

|     Mask R-CNN     |                  [R-50-CAFFE](./mask-rcnn_r50_caffe_fpn_1x_nuim.py)                   |     IN      |   1x    |   7.0    |                                                                                    47.7                                                                                    |  38.2   |                                                                                                       [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/mask_rcnn_r50_caffe_fpn_1x_nuim/) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/mask_rcnn_r50_caffe_fpn_1x_nuim/)                                                                                                       |

|     Mask R-CNN     |              [R-50-CAFFE](./mask-rcnn_r50_caffe_fpn_coco-3x_1x_nuim.py)               | IN+COCO-3x  |   1x    |   7.0    |                                                                                    49.9                                                                                    |  40.8   |                             [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/mask_rcnn_r50_caffe_fpn_coco-3x_1x_nuim/mask_rcnn_r50_caffe_fpn_coco-3x_1x_nuim_20201008_195305-661a992e.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/mask_rcnn_r50_caffe_fpn_coco-3x_1x_nuim/mask_rcnn_r50_caffe_fpn_coco-3x_1x_nuim_20201008_195305.log.json)                             |

|     Mask R-CNN     |              [R-50-CAFFE](./mask-rcnn_r50_caffe_fpn_coco-3x_20e_nuim.py)              | IN+COCO-3x  |   20e   |   7.0    |                                                                                    50.6                                                                                    |  41.3   |                           [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/mask_rcnn_r50_caffe_fpn_coco-3x_20e_nuim/mask_rcnn_r50_caffe_fpn_coco-3x_20e_nuim_20201009_125002-5529442c.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/mask_rcnn_r50_caffe_fpn_coco-3x_20e_nuim/mask_rcnn_r50_caffe_fpn_coco-3x_20e_nuim_20201009_125002.log.json)                           |

|     Mask R-CNN     |                       [R-101](./mask-rcnn_r101_fpn_1x_nuim.py)                        |     IN      |   1x    |   10.9   |                                                                                    48.9                                                                                    |  39.1   |                                                       [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/mask_rcnn_r101_fpn_1x_nuim/mask_rcnn_r101_fpn_1x_nuim_20201024_134803-65c7623a.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/mask_rcnn_r101_fpn_1x_nuim/mask_rcnn_r101_fpn_1x_nuim_20201024_134803.log.json)                                                       |

|     Mask R-CNN     |                 [X-101_32x4d](./mask-rcnn_x101_32x4d_fpn_1x_nuim.py)                  |     IN      |   1x    |   13.3   |                                                                                    50.4                                                                                    |  40.5   |                                           [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/mask_rcnn_x101_32x4d_fpn_1x_nuim/mask_rcnn_x101_32x4d_fpn_1x_nuim_20201024_135741-b699ab37.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/mask_rcnn_x101_32x4d_fpn_1x_nuim/mask_rcnn_x101_32x4d_fpn_1x_nuim_20201024_135741.log.json)                                           |

| Cascade Mask R-CNN |                    [R-50](./cascade-mask-rcnn_r50_fpn_1x_nuim.py)                     |     IN      |   1x    |   8.9    |                                                                                    50.8                                                                                    |  40.4   |                                         [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/cascade_mask_rcnn_r50_fpn_1x_nuim/cascade_mask_rcnn_r50_fpn_1x_nuim_20201008_195342-1147c036.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/cascade_mask_rcnn_r50_fpn_1x_nuim/cascade_mask_rcnn_r50_fpn_1x_nuim_20201008_195342.log.json)                                         |

| Cascade Mask R-CNN |                [R-50](./cascade-mask-rcnn_r50_fpn_coco-20e_1x_nuim.py)                | IN+COCO-20e |   1x    |   8.9    |                                                                                    52.8                                                                                    |  42.2   |                       [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/cascade_mask_rcnn_r50_fpn_coco-20e_1x_nuim/cascade_mask_rcnn_r50_fpn_coco-20e_1x_nuim_20201009_124158-ad0540e3.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/cascade_mask_rcnn_r50_fpn_coco-20e_1x_nuim/cascade_mask_rcnn_r50_fpn_coco-20e_1x_nuim_20201009_124158.log.json)                       |

| Cascade Mask R-CNN |               [R-50](./cascade-mask-rcnn_r50_fpn_coco-20e_20e_nuim.py)                | IN+COCO-20e |   20e   |   8.9    |                                                                                    52.8                                                                                    |  42.2   |                     [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/cascade_mask_rcnn_r50_fpn_coco-20e_20e_nuim/cascade_mask_rcnn_r50_fpn_coco-20e_20e_nuim_20201009_124951-40963960.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/cascade_mask_rcnn_r50_fpn_coco-20e_20e_nuim/cascade_mask_rcnn_r50_fpn_coco-20e_20e_nuim_20201009_124951.log.json)                     |

| Cascade Mask R-CNN |                   [R-101](./cascade-mask-rcnn_r101_fpn_1x_nuim.py)                    |     IN      |   1x    |   12.5   |                                                                                    51.5                                                                                    |  40.7   |                                       [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/cascade_mask_rcnn_r101_fpn_1x_nuim/cascade_mask_rcnn_r101_fpn_1x_nuim_20201024_134804-45215b1e.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/cascade_mask_rcnn_r101_fpn_1x_nuim/cascade_mask_rcnn_r101_fpn_1x_nuim_20201024_134804.log.json)                                       |

| Cascade Mask R-CNN |             [X-101_32x4d](./cascade-mask-rcnn_x101_32x4d_fpn_1x_nuim.py)              |     IN      |   1x    |   14.9   |                                                                                    52.8                                                                                    |  41.6   |                           [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/cascade_mask_rcnn_x101_32x4d_fpn_1x_nuim/cascade_mask_rcnn_x101_32x4d_fpn_1x_nuim_20201024_135753-e0e49778.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/cascade_mask_rcnn_x101_32x4d_fpn_1x_nuim/cascade_mask_rcnn_x101_32x4d_fpn_1x_nuim_20201024_135753.log.json)                           |

|  HTC w/o semantic  |                   [R-50](./htc_without_semantic_r50_fpn_1x_nuim.py)                   |     IN      |   1x    |          |                                                                          [model](<>) \| [log](<>)                                                                          |         |                                                                                                                                                                                                                                                                                                                                                                                                                                                        |

|        HTC         |                           [R-50](./htc_r50_fpn_1x_nuim.py)                            |     IN      |   1x    |          | [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/) |         |                                                                                                                                                                                                                                                                                                                                                                                                                                                        |

|        HTC         |                       [R-50](./htc_r50_fpn_coco-20e_1x_nuim.py)                       | IN+COCO-20e |   1x    |   11.6   |                                                                                    53.8                                                                                    |  43.8   |                                                   [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/htc_r50_fpn_coco-20e_1x_nuim/htc_r50_fpn_coco-20e_1x_nuim_20201010_070203-0b53a65e.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/htc_r50_fpn_coco-20e_1x_nuim/htc_r50_fpn_coco-20e_1x_nuim_20201010_070203.log.json)                                                   |

|        HTC         |                      [R-50](./htc_r50_fpn_coco-20e_20e_nuim.py)                       | IN+COCO-20e |   20e   |   11.6   |                                                                                    54.8                                                                                    |  44.4   |                                                 [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/htc_r50_fpn_coco-20e_20e_nuim/htc_r50_fpn_coco-20e_20e_nuim_20201008_211415-d6c60a2c.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/htc_r50_fpn_coco-20e_20e_nuim/htc_r50_fpn_coco-20e_20e_nuim_20201008_211415.log.json)                                                 |

|        HTC         | [X-101_64x4d + DCN_c3-c5](./htc_x101_64x4d_fpn_dconv_c3-c5_coco-20e_16x1_20e_nuim.py) | IN+COCO-20e |   20e   |   13.3   |                                                                                    57.3                                                                                    |  46.4   | [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/htc_x101_64x4d_fpn_dconv_c3-c5_coco-20e_16x1_20e_nuim/htc_x101_64x4d_fpn_dconv_c3-c5_coco-20e_16x1_20e_nuim_20201008_211222-0b16ac4b.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/htc_x101_64x4d_fpn_dconv_c3-c5_coco-20e_16x1_20e_nuim/htc_x101_64x4d_fpn_dconv_c3-c5_coco-20e_16x1_20e_nuim_20201008_211222.log.json) |

**Note**:

**Note**:

1. `IN` means only using ImageNet pre-trained backbone. `IN+COCO-Nx` and `IN+COCO-Ne` means the backbone is first pre-trained on ImageNet, and then the detector is pre-trained on COCO train2017 dataset by `Nx` and `N` epochs schedules, respectively.
2. All the training hyper-parameters follow the standard schedules on COCO dataset except that the images are resized from
   1280 x 720 to 1920 x 1080 (relative ratio 0.8 to 1.2) since the images are in size 1600 x 900.
3. The class order in the detectors released in v0.6.0 is different from the order in the configs because the bug in the conversion script. This bug has been fixed since v0.7.0 and the models trained by the correct class order are also released. If you used nuImages since v0.6.0, please re-convert the data through the conversion script using the above-mentioned command.

# ConvNeXt

> [A ConvNet for the 2020s](https://arxiv.org/abs/2201.03545)

<a href="https://github.com/facebookresearch/ConvNeXt">Official Repo</a>

<a href="https://github.com/open-mmlab/mmclassification/blob/v0.20.1/mmcls/models/backbones/convnext.py#L133">Code Snippet</a>

The "Roaring 20s" of visual recognition began with the introduction of Vision Transformers (ViTs), which quickly superseded ConvNets as the state-of-the-art image classification model. A vanilla ViT, on the other hand, faces difficulties when applied to general computer vision tasks such as object detection and semantic segmentation. It is the hierarchical Transformers (e.g., Swin Transformers) that reintroduced several ConvNet priors, making Transformers practically viable as a generic vision backbone and demonstrating remarkable performance on a wide variety of vision tasks. However, the effectiveness of such hybrid approaches is still largely credited to the intrinsic superiority of Transformers, rather than the inherent inductive biases of convolutions. In this work, we reexamine the design spaces and test the limits of what a pure ConvNet can achieve. We gradually "modernize" a standard ResNet toward the design of a vision Transformer, and discover several key components that contribute to the performance difference along the way. The outcome of this exploration is a family of pure ConvNet models dubbed ConvNeXt. Constructed entirely from standard ConvNet modules, ConvNeXts compete favorably with Transformers in terms of accuracy and scalability, achieving 87.8% ImageNet top-1 accuracy and outperforming Swin Transformers on COCO detection and ADE20K segmentation, while maintaining the simplicity and efficiency of standard ConvNets.

<img src="https://user-images.githubusercontent.com/8370623/148624004-e9581042-ea4d-4e10-b3bd-42c92b02053b.png" width="90%"/>
</div>

### Usage

- ConvNeXt backbone needs to install [MMClassification](https://github.com/open-mmlab/mmclassification) first, which has abundant backbones for downstream tasks.

```shell
pip install mmpretrain>=1.0.0rc7
```

### Pre-trained Models

The pre-trained models on ImageNet-1k or ImageNet-21k are used to fine-tune on the downstream tasks.

|     Model     | Training Data | Params(M) | Flops(G) |                                                                     Download                                                                     |
| :-----------: | :-----------: | :-------: | :------: | :----------------------------------------------------------------------------------------------------------------------------------------------: |

| ConvNeXt-T\*  |  ImageNet-1k  |   28.59   |   4.46   | [model](https://download.openmmlab.com/mmclassification/v0/convnext/downstream/convnext-tiny_3rdparty_32xb128-noema_in1k_20220301-795e9634.pth)  |
| ConvNeXt-S\*  |  ImageNet-1k  |   50.22   |   8.69   | [model](https://download.openmmlab.com/mmclassification/v0/convnext/downstream/convnext-small_3rdparty_32xb128-noema_in1k_20220301-303e75e3.pth) |
| ConvNeXt-B\*  |  ImageNet-1k  |   88.59   |  15.36   | [model](https://download.openmmlab.com/mmclassification/v0/convnext/downstream/convnext-base_3rdparty_32xb128-noema_in1k_20220301-2a0ee547.pth)  |
| ConvNeXt-B\*  | ImageNet-21k  |   88.59   |  15.36   |        [model](https://download.openmmlab.com/mmclassification/v0/convnext/downstream/convnext-base_3rdparty_in21k_20220301-262fd037.pth)        |

| ConvNeXt-L\*  | ImageNet-21k  |  197.77   |  34.37   |       [model](https://download.openmmlab.com/mmclassification/v0/convnext/downstream/convnext-large_3rdparty_in21k_20220301-e6e0ea0a.pth)        |
| ConvNeXt-XL\* | ImageNet-21k  |  350.20   |  60.93   |       [model](https://download.openmmlab.com/mmclassification/v0/convnext/downstream/convnext-xlarge_3rdparty_in21k_20220301-08aa5ddc.pth)       |

*Models with* are converted from the [official repo](https://github.com/facebookresearch/ConvNeXt/tree/main/semantic_segmentation#results-and-fine-tuned-models).\*

### ADE20K

### ADE20K

| Method  | Backbone    | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device | mIoU  | mIoU(ms+flip) | config                                                                                                                                    | download                                                                                                                                                                                                                                                                                                                                                                                             |

| UPerNet | ConvNeXt-T  | 512x512   | 160000  | 4.23     | 19.90          | V100   | 46.11 | 46.62         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/convnext/convnext-tiny_upernet_8xb2-amp-160k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/convnext/upernet_convnext_tiny_fp16_512x512_160k_ade20k/upernet_convnext_tiny_fp16_512x512_160k_ade20k_20220227_124553-cad485de.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/convnext/upernet_convnext_tiny_fp16_512x512_160k_ade20k/upernet_convnext_tiny_fp16_512x512_160k_ade20k_20220227_124553.log.json)         |

| UPerNet | ConvNeXt-S  | 512x512   | 160000  | 5.16     | 15.18          | V100   | 48.56 | 49.02         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/convnext/convnext-small_upernet_8xb2-amp-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/convnext/upernet_convnext_small_fp16_512x512_160k_ade20k/upernet_convnext_small_fp16_512x512_160k_ade20k_20220227_131208-1b1e394f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/convnext/upernet_convnext_small_fp16_512x512_160k_ade20k/upernet_convnext_small_fp16_512x512_160k_ade20k_20220227_131208.log.json)     |

| UPerNet | ConvNeXt-B  | 512x512   | 160000  | 6.33     | 14.41          | V100   | 48.71 | 49.54         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/convnext/convnext-base_upernet_8xb2-amp-160k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/convnext/upernet_convnext_base_fp16_512x512_160k_ade20k/upernet_convnext_base_fp16_512x512_160k_ade20k_20220227_181227-02a24fc6.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/convnext/upernet_convnext_base_fp16_512x512_160k_ade20k/upernet_convnext_base_fp16_512x512_160k_ade20k_20220227_181227.log.json)         |

| UPerNet | ConvNeXt-B  | 640x640   | 160000  | 8.53     | 10.88          | V100   | 52.13 | 52.66         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/convnext/convnext-base_upernet_8xb2-amp-160k_ade20k-640x640.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/convnext/upernet_convnext_base_fp16_640x640_160k_ade20k/upernet_convnext_base_fp16_640x640_160k_ade20k_20220227_182859-9280e39b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/convnext/upernet_convnext_base_fp16_640x640_160k_ade20k/upernet_convnext_base_fp16_640x640_160k_ade20k_20220227_182859.log.json)         |

| UPerNet | ConvNeXt-L  | 640x640   | 160000  | 12.08    | 7.69           | V100   | 53.16 | 53.38         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/convnext/convnext-large_upernet_8xb2-amp-160k_ade20k-640x640.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/convnext/upernet_convnext_large_fp16_640x640_160k_ade20k/upernet_convnext_large_fp16_640x640_160k_ade20k_20220226_040532-e57aa54d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/convnext/upernet_convnext_large_fp16_640x640_160k_ade20k/upernet_convnext_large_fp16_640x640_160k_ade20k_20220226_040532.log.json)     |

| UPerNet | ConvNeXt-XL | 640x640   | 160000  | 26.16\*  | 6.33           | V100   | 53.58 | 54.11         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/convnext/convnext-xlarge_upernet_8xb2-amp-160k_ade20k-640x640.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/convnext/upernet_convnext_xlarge_fp16_640x640_160k_ade20k/upernet_convnext_xlarge_fp16_640x640_160k_ade20k_20220226_080344-95fc38c2.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/convnext/upernet_convnext_xlarge_fp16_640x640_160k_ade20k/upernet_convnext_xlarge_fp16_640x640_160k_ade20k_20220226_080344.log.json) |

Note:

- `Mem (GB)` with * is collected when `cudnn_benchmark=True`, and hardware is V100.

```bibtex
@article{liu2022convnet,
  title={A ConvNet for the 2020s},
  author={Liu, Zhuang and Mao, Hanzi and Wu, Chao-Yuan and Feichtenhofer, Christoph and Darrell, Trevor and Xie, Saining},

```bibtex
  journal={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
}
```

# MaskFeat

> [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://arxiv.org/abs/2112.09133v1)

We present Masked Feature Prediction (MaskFeat) for self-supervised pre-training of video models. Our approach first randomly masks out a portion of the input sequence and then predicts the feature of the masked regions. We study five different types of features and find Histograms of Oriented Gradients (HOG), a hand-crafted feature descriptor, works particularly well in terms of both performance and efficiency. We observe that the local contrast normalization in HOG is essential for good results, which is in line with earlier work using HOG for visual recognition. Our approach can learn abundant visual knowledge and drive large-scale Transformer-based models. Without using extra model weights or supervision, MaskFeat pre-trained on unlabeled videos achieves unprecedented results of 86.7% with MViT-L on Kinetics-400, 88.3% on Kinetics-600, 80.4% on Kinetics-700, 38.8 mAP on AVA, and 75.0% on SSv2. MaskFeat further generalizes to image input, which can be interpreted as a video with a single frame and obtains competitive results on ImageNet.

<img src="https://user-images.githubusercontent.com/48178838/190090285-428f07c0-0887-4ce8-b94f-f719cfd25622.png" width="60%"/>
</div>

```python

predict = inference_model('vit-base-p16_maskfeat-pre_8xb256-coslr-100e_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('maskfeat_vit-base-p16_8xb256-amp-coslr-300e_in1k', pretrained=True)
```

Train:

```shell

```

Train:

```shell
python tools/train.py configs/maskfeat/maskfeat_vit-base-p16_8xb256-amp-coslr-300e_in1k.py
```

Test:

```shell
python tools/test.py configs/maskfeat/benchmarks/vit-base-p16_8xb256-coslr-100e_in1k.py https://download.openmmlab.com/mmselfsup/1.x/maskfeat/maskfeat_vit-base-p16_8xb256-amp-coslr-300e_in1k/vit-base-p16_ft-8xb256-coslr-100e_in1k/vit-base-p16_ft-8xb256-coslr-100e_in1k_20221028-5134431c.pth
```

| Model                                              | Params (M) | Flops (G) |                            Config                             |                                Download                                |

| :------------------------------------------------- | :--------: | :-------: | :-----------------------------------------------------------: | :--------------------------------------------------------------------: |
| `maskfeat_vit-base-p16_8xb256-amp-coslr-300e_in1k` |   85.88    |   17.58   | [config](maskfeat_vit-base-p16_8xb256-amp-coslr-300e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/maskfeat/maskfeat_vit-base-p16_8xb256-amp-coslr-300e_in1k/maskfeat_vit-base-p16_8xb256-amp-coslr-300e_in1k_20221101-6dfc8bf3.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/maskfeat/maskfeat_vit-base-p16_8xb256-amp-coslr-300e_in1k/maskfeat_vit-base-p16_8xb256-amp-coslr-300e_in1k_20221101-6dfc8bf3.json) |

| `vit-base-p16_maskfeat-pre_8xb256-coslr-100e_in1k` | [MASKFEAT](https://download.openmmlab.com/mmselfsup/1.x/maskfeat/maskfeat_vit-base-p16_8xb256-amp-coslr-300e_in1k/maskfeat_vit-base-p16_8xb256-amp-coslr-300e_in1k_20221101-6dfc8bf3.pth) |   86.57    |   17.58   |   83.40   | [config](benchmarks/vit-base-p16_8xb256-coslr-100e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/maskfeat/maskfeat_vit-base-p16_8xb256-amp-coslr-300e_in1k/vit-base-p16_ft-8xb256-coslr-100e_in1k/vit-base-p16_ft-8xb256-coslr-100e_in1k_20221028-5134431c.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/maskfeat/maskfeat_vit-base-p16_8xb256-amp-coslr-300e_in1k/vit-base-p16_ft-8xb256-coslr-100e_in1k/vit-base-p16_ft-8xb256-coslr-100e_in1k_20221028-5134431c.json) |

```bibtex
@InProceedings{wei2022masked,
    author    = {Wei, Chen and Fan, Haoqi and Xie, Saining and Wu, Chao-Yuan and Yuille, Alan and Feichtenhofer, Christoph},
    title     = {Masked Feature Prediction for Self-Supervised Visual Pre-Training},
    booktitle = {CVPR},
}
```

# HRNet

> [Deep High-Resolution Representation Learning for Visual Recognition](https://arxiv.org/abs/1908.07919v2)

High-resolution representations are essential for position-sensitive vision problems, such as human pose estimation, semantic segmentation, and object detection. Existing state-of-the-art frameworks first encode the input image as a low-resolution representation through a subnetwork that is formed by connecting high-to-low resolution convolutions *in series* (e.g., ResNet, VGGNet), and then recover the high-resolution representation from the encoded low-resolution representation. Instead, our proposed network, named as High-Resolution Network (HRNet), maintains high-resolution representations through the whole process. There are two key characteristics: (i) Connect the high-to-low resolution convolution streams *in parallel*; (ii) Repeatedly exchange the information across resolutions. The benefit is that the resulting representation is semantically richer and spatially more precise. We show the superiority of the proposed HRNet in a wide range of applications, including human pose estimation, semantic segmentation, and object detection, suggesting that the HRNet is a stronger backbone for computer vision problems.

<img src="https://user-images.githubusercontent.com/26739999/149920446-cbe05670-989d-4fe6-accc-df20ae2984eb.png" width="100%"/>
</div>

```python

predict = inference_model('hrnet-w18_3rdparty_8xb32_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('hrnet-w18_3rdparty_8xb32_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/hrnet/hrnet-w18_4xb32_in1k.py https://download.openmmlab.com/mmclassification/v0/hrnet/hrnet-w18_3rdparty_8xb32_in1k_20220120-0c10b180.pth
```

| Model                                  |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |              Config               |                                     Download                                     |
| :------------------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :-------------------------------: | :------------------------------------------------------------------------------: |

| `hrnet-w18_3rdparty_8xb32_in1k`\*      | From scratch |   21.30    |   4.33    |   76.75   |   93.44   | [config](hrnet-w18_4xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/hrnet/hrnet-w18_3rdparty_8xb32_in1k_20220120-0c10b180.pth) |
| `hrnet-w30_3rdparty_8xb32_in1k`\*      | From scratch |   37.71    |   8.17    |   78.19   |   94.22   | [config](hrnet-w30_4xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/hrnet/hrnet-w30_3rdparty_8xb32_in1k_20220120-8aa3832f.pth) |
| `hrnet-w32_3rdparty_8xb32_in1k`\*      | From scratch |   41.23    |   8.99    |   78.44   |   94.19   | [config](hrnet-w32_4xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/hrnet/hrnet-w32_3rdparty_8xb32_in1k_20220120-c394f1ab.pth) |

| `hrnet-w40_3rdparty_8xb32_in1k`\*      | From scratch |   57.55    |   12.77   |   78.94   |   94.47   | [config](hrnet-w40_4xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/hrnet/hrnet-w40_3rdparty_8xb32_in1k_20220120-9a2dbfc5.pth) |
| `hrnet-w44_3rdparty_8xb32_in1k`\*      | From scratch |   67.06    |   14.96   |   78.88   |   94.37   | [config](hrnet-w44_4xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/hrnet/hrnet-w44_3rdparty_8xb32_in1k_20220120-35d07f73.pth) |
| `hrnet-w48_3rdparty_8xb32_in1k`\*      | From scratch |   77.47    |   17.36   |   79.32   |   94.52   | [config](hrnet-w48_4xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/hrnet/hrnet-w48_3rdparty_8xb32_in1k_20220120-e555ef50.pth) |

| `hrnet-w64_3rdparty_8xb32_in1k`\*      | From scratch |   128.06   |   29.00   |   79.46   |   94.65   | [config](hrnet-w64_4xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/hrnet/hrnet-w64_3rdparty_8xb32_in1k_20220120-19126642.pth) |
| `hrnet-w18_3rdparty_8xb32-ssld_in1k`\* | From scratch |   21.30    |   4.33    |   81.06   |   95.70   | [config](hrnet-w18_4xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/hrnet/hrnet-w18_3rdparty_8xb32-ssld_in1k_20220120-455f69ea.pth) |
| `hrnet-w48_3rdparty_8xb32-ssld_in1k`\* | From scratch |   77.47    |   17.36   |   83.63   |   96.79   | [config](hrnet-w48_4xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/hrnet/hrnet-w48_3rdparty_8xb32-ssld_in1k_20220120-d0459c38.pth) |

*Models with * are converted from the [official repo](https://github.com/HRNet/HRNet-Image-Classification). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
}
```

# PSPNet

> [Pyramid Scene Parsing Network](https://arxiv.org/abs/1612.01105)

<a href="https://github.com/hszhao/PSPNet">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/decode_heads/psp_head.py#L63">Code Snippet</a>

Scene parsing is challenging for unrestricted open vocabulary and diverse scenes. In this paper, we exploit the capability of global context information by different-region-based context aggregation through our pyramid pooling module together with the proposed pyramid scene parsing network (PSPNet). Our global prior representation is effective to produce good quality results on the scene parsing task, while PSPNet provides a superior framework for pixel-level prediction tasks. The proposed approach achieves state-of-the-art performance on various datasets. It came first in ImageNet scene parsing challenge 2016, PASCAL VOC 2012 benchmark and Cityscapes benchmark. A single PSPNet yields new record of mIoU accuracy 85.4% on PASCAL VOC 2012 and accuracy 80.2% on Cityscapes.

<img src="https://user-images.githubusercontent.com/24582831/142902444-9f93b99e-9261-443b-a0a4-17e78eefb525.png" width="70%"/>
</div>

<img alt="PSPNet-R50-D8" src="https://user-images.githubusercontent.com/47882088/209554973-66804b14-de5a-4f83-b54e-26683a91818a.jpg"/>
PSPNet-R50 D8 model structure
</div>

</div>

| Method        | Backbone      | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                   | download                                                                                                                                                                                                                                                                                                                                                                                                                             |

| PSPNet        | R-50-D8       | 512x1024  |   40000 | 6.1      | 4.07           | V100   | 77.85 |         79.18 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8_4xb2-40k_cityscapes-512x1024.py)            | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x1024_40k_cityscapes/pspnet_r50-d8_512x1024_40k_cityscapes_20200605_003338-2966598c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x1024_40k_cityscapes/pspnet_r50-d8_512x1024_40k_cityscapes_20200605_003338.log.json)                                                                                 |

| PSPNet        | R-101-D8      | 512x1024  |   40000 | 9.6      | 2.68           | V100   | 78.34 |         79.74 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb2-40k_cityscapes-512x1024.py)           | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x1024_40k_cityscapes/pspnet_r101-d8_512x1024_40k_cityscapes_20200604_232751-467e7cf4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x1024_40k_cityscapes/pspnet_r101-d8_512x1024_40k_cityscapes_20200604_232751.log.json)                                                                             |

| PSPNet        | R-50-D8       | 769x769   |   40000 | 6.9      | 1.76           | V100   | 78.26 |         79.88 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8_4xb2-40k_cityscapes-769x769.py)             | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_769x769_40k_cityscapes/pspnet_r50-d8_769x769_40k_cityscapes_20200606_112725-86638686.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_769x769_40k_cityscapes/pspnet_r50-d8_769x769_40k_cityscapes_20200606_112725.log.json)                                                                                     |

| PSPNet        | R-101-D8      | 769x769   |   40000 | 10.9     | 1.15           | V100   | 79.08 |         80.28 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb2-40k_cityscapes-769x769.py)            | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_769x769_40k_cityscapes/pspnet_r101-d8_769x769_40k_cityscapes_20200606_112753-61c6f5be.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_769x769_40k_cityscapes/pspnet_r101-d8_769x769_40k_cityscapes_20200606_112753.log.json)                                                                                 |

| PSPNet        | R-18-D8       | 512x1024  |   80000 | 1.7      | 15.71          | V100   | 74.87 |         76.04 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r18-d8_4xb2-80k_cityscapes-512x1024.py)            | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r18-d8_512x1024_80k_cityscapes/pspnet_r18-d8_512x1024_80k_cityscapes_20201225_021458-09ffa746.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r18-d8_512x1024_80k_cityscapes/pspnet_r18-d8_512x1024_80k_cityscapes-20201225_021458.log.json)                                                                                 |

| PSPNet        | R-50-D8       | 512x1024  |   80000 | -        | -              | V100   | 78.55 |         79.79 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8_4xb2-80k_cityscapes-512x1024.py)            | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x1024_80k_cityscapes/pspnet_r50-d8_512x1024_80k_cityscapes_20200606_112131-2376f12b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x1024_80k_cityscapes/pspnet_r50-d8_512x1024_80k_cityscapes_20200606_112131.log.json)                                                                                 |

| PSPNet        | R-50b-D8 rsb  | 512x1024  |   80000 | 6.2      | 3.82           | V100   | 78.47 |         79.45 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8-rsb_4xb2-adamw-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x1024_80k_cityscapes/pspnet_r50-d8_rsb-pretrain_512x1024_adamw_80k_cityscapes_20220315_123238-588c30be.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x1024_80k_cityscapes/pspnet_r50-d8_rsb-pretrain_512x1024_adamw_80k_cityscapes_20220315_123238.log.json)                                           |

| PSPNet        | R-101-D8      | 512x1024  |   80000 | -        | -              | V100   | 79.76 |         81.01 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb2-80k_cityscapes-512x1024.py)           | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x1024_80k_cityscapes/pspnet_r101-d8_512x1024_80k_cityscapes_20200606_112211-e1e1100f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x1024_80k_cityscapes/pspnet_r101-d8_512x1024_80k_cityscapes_20200606_112211.log.json)                                                                             |

| PSPNet (FP16) | R-101-D8      | 512x1024  |   80000 | 5.34     | 8.77           | V100   | 79.46 |             - | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb2-amp-80k_cityscapes-512x1024.py)       | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_fp16_512x1024_80k_cityscapes/pspnet_r101-d8_fp16_512x1024_80k_cityscapes_20200717_230919-a0875e5c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_fp16_512x1024_80k_cityscapes/pspnet_r101-d8_fp16_512x1024_80k_cityscapes_20200717_230919.log.json)                                                         |

| PSPNet        | R-18-D8       | 769x769   |   80000 | 1.9      | 6.20           | V100   | 75.90 |         77.86 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r18-d8_4xb2-80k_cityscapes-769x769.py)             | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r18-d8_769x769_80k_cityscapes/pspnet_r18-d8_769x769_80k_cityscapes_20201225_021458-3deefc62.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r18-d8_769x769_80k_cityscapes/pspnet_r18-d8_769x769_80k_cityscapes-20201225_021458.log.json)                                                                                     |

| PSPNet        | R-50-D8       | 769x769   |   80000 | -        | -              | V100   | 79.59 |         80.69 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8_4xb2-80k_cityscapes-769x769.py)             | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_769x769_80k_cityscapes/pspnet_r50-d8_769x769_80k_cityscapes_20200606_210121-5ccf03dd.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_769x769_80k_cityscapes/pspnet_r50-d8_769x769_80k_cityscapes_20200606_210121.log.json)                                                                                     |

| PSPNet        | R-101-D8      | 769x769   |   80000 | -        | -              | V100   | 79.77 |         81.06 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb2-80k_cityscapes-769x769.py)            | [model](https://download.oz1z1penmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_769x769_80k_cityscapes/pspnet_r101-d8_769x769_80k_cityscapes_20200606_225055-dba412fa.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_769x769_80k_cityscapes/pspnet_r101-d8_769x769_80k_cityscapes_20200606_225055.log.json)                                                                             |

| PSPNet        | R-18b-D8      | 512x1024  |   80000 | 1.5      | 16.28          | V100   | 74.23 |         75.79 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r18b-d8_4xb2-80k_cityscapes-512x1024.py)           | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r18b-d8_512x1024_80k_cityscapes/pspnet_r18b-d8_512x1024_80k_cityscapes_20201226_063116-26928a60.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r18b-d8_512x1024_80k_cityscapes/pspnet_r18b-d8_512x1024_80k_cityscapes-20201226_063116.log.json)                                                                             |

| PSPNet        | R-50b-D8      | 512x1024  |   80000 | 6.0      | 4.30           | V100   | 78.22 |         79.46 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50b-d8_4xb2-80k_cityscapes-512x1024.py)           | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50b-d8_512x1024_80k_cityscapes/pspnet_r50b-d8_512x1024_80k_cityscapes_20201225_094315-6344287a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50b-d8_512x1024_80k_cityscapes/pspnet_r50b-d8_512x1024_80k_cityscapes-20201225_094315.log.json)                                                                             |

| PSPNet        | R-101b-D8     | 512x1024  |   80000 | 9.5      | 2.76           | V100   | 79.69 |         80.79 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101b-d8_4xb2-80k_cityscapes-512x1024.py)          | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101b-d8_512x1024_80k_cityscapes/pspnet_r101b-d8_512x1024_80k_cityscapes_20201226_170012-3a4d38ab.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101b-d8_512x1024_80k_cityscapes/pspnet_r101b-d8_512x1024_80k_cityscapes-20201226_170012.log.json)                                                                         |

| PSPNet        | R-18b-D8      | 769x769   |   80000 | 1.7      | 6.41           | V100   | 74.92 |         76.90 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r18b-d8_4xb2-80k_cityscapes-769x769.py)            | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r18b-d8_769x769_80k_cityscapes/pspnet_r18b-d8_769x769_80k_cityscapes_20201226_080942-bf98d186.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r18b-d8_769x769_80k_cityscapes/pspnet_r18b-d8_769x769_80k_cityscapes-20201226_080942.log.json)                                                                                 |

| PSPNet        | R-50b-D8      | 769x769   |   80000 | 6.8      | 1.88           | V100   | 78.50 |         79.96 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50b-d8_4xb2-80k_cityscapes-769x769.py)            | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50b-d8_769x769_80k_cityscapes/pspnet_r50b-d8_769x769_80k_cityscapes_20201225_094316-4c643cf6.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50b-d8_769x769_80k_cityscapes/pspnet_r50b-d8_769x769_80k_cityscapes-20201225_094316.log.json)                                                                                 |

| PSPNet        | R-101b-D8     | 769x769   |   80000 | 10.8     | 1.17           | V100   | 78.87 |         80.04 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101b-d8_4xb2-80k_cityscapes-769x769.py)           | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101b-d8_769x769_80k_cityscapes/pspnet_r101b-d8_769x769_80k_cityscapes_20201226_171823-f0e7c293.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101b-d8_769x769_80k_cityscapes/pspnet_r101b-d8_769x769_80k_cityscapes-20201226_171823.log.json)                                                                             |

| PSPNet        | R-50-D32      | 512x1024  |   80000 | 3.0      | 15.21          | V100   | 73.88 |         76.85 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50b-d32_4xb2-80k_cityscapes-512x1024.py)          | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d32_512x1024_80k_cityscapes/pspnet_r50-d32_512x1024_80k_cityscapes_20220316_224840-9092b254.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d32_512x1024_80k_cityscapes/pspnet_r50-d32_512x1024_80k_cityscapes_20220316_224840.log.json)                                                                             |

| PSPNet        | R-50b-D32 rsb | 512x1024  |   80000 | 3.1      | 16.08          | V100   | 74.09 |         77.18 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d32_rsb_4xb2-adamw-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d32_rsb-pretrain_512x1024_adamw_80k_cityscapes/pspnet_r50-d32_rsb-pretrain_512x1024_adamw_80k_cityscapes_20220316_141229-dd9c9610.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d32_rsb-pretrain_512x1024_adamw_80k_cityscapes/pspnet_r50-d32_rsb-pretrain_512x1024_adamw_80k_cityscapes_20220316_141229.log.json) |

| PSPNet        | R-50b-D32     | 512x1024  |   80000 | 2.9      | 15.41          | V100   | 72.61 |         75.51 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50b-d32_4xb2-80k_cityscapes-512x1024.py)          | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50b-d32_512x1024_80k_cityscapes/pspnet_r50b-d32_512x1024_80k_cityscapes_20220311_152152-23bcaf8c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50b-d32_512x1024_80k_cityscapes/pspnet_r50b-d32_512x1024_80k_cityscapes_20220311_152152.log.json)                                                                         |

### ADE20K

### ADE20K

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                     | download                                                                                                                                                                                                                                                                                                                                 |

| PSPNet | R-50-D8  | 512x512   |   80000 | 8.5      | 23.53          | V100   | 41.13 |         41.94 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8_4xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x512_80k_ade20k/pspnet_r50-d8_512x512_80k_ade20k_20200615_014128-15a8b914.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x512_80k_ade20k/pspnet_r50-d8_512x512_80k_ade20k_20200615_014128.log.json)         |

| PSPNet | R-101-D8 | 512x512   |   80000 | 12       | 15.30          | V100   | 43.57 |         44.35 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb4-80k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x512_80k_ade20k/pspnet_r101-d8_512x512_80k_ade20k_20200614_031423-b6e782f0.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x512_80k_ade20k/pspnet_r101-d8_512x512_80k_ade20k_20200614_031423.log.json)     |

| PSPNet | R-50-D8  | 512x512   |  160000 | -        | -              | V100   | 42.48 |         43.44 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8_4xb4-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x512_160k_ade20k/pspnet_r50-d8_512x512_160k_ade20k_20200615_184358-1890b0bd.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x512_160k_ade20k/pspnet_r50-d8_512x512_160k_ade20k_20200615_184358.log.json)     |

| PSPNet | R-101-D8 | 512x512   |  160000 | -        | -              | V100   | 44.39 |         45.35 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x512_160k_ade20k/pspnet_r101-d8_512x512_160k_ade20k_20200615_100650-967c316f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x512_160k_ade20k/pspnet_r101-d8_512x512_160k_ade20k_20200615_100650.log.json) |

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                      | download                                                                                                                                                                                                                                                                                                                                     |

| PSPNet | R-50-D8  | 512x512   |   20000 | 6.1      | 23.59          | V100   | 76.78 |         77.61 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8_4xb4-20k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x512_20k_voc12aug/pspnet_r50-d8_512x512_20k_voc12aug_20200617_101958-ed5dfbd9.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x512_20k_voc12aug/pspnet_r50-d8_512x512_20k_voc12aug_20200617_101958.log.json)     |

| PSPNet | R-101-D8 | 512x512   |   20000 | 9.6      | 15.02          | V100   | 78.47 |         79.25 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb4-20k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x512_20k_voc12aug/pspnet_r101-d8_512x512_20k_voc12aug_20200617_102003-4aef3c9a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x512_20k_voc12aug/pspnet_r101-d8_512x512_20k_voc12aug_20200617_102003.log.json) |

| PSPNet | R-50-D8  | 512x512   |   40000 | -        | -              | V100   | 77.29 |         78.48 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8_4xb4-40k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x512_40k_voc12aug/pspnet_r50-d8_512x512_40k_voc12aug_20200613_161222-ae9c1b8c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x512_40k_voc12aug/pspnet_r50-d8_512x512_40k_voc12aug_20200613_161222.log.json)     |

| PSPNet | R-101-D8 | 512x512   |   40000 | -        | -              | V100   | 78.52 |         79.57 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb4-40k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x512_40k_voc12aug/pspnet_r101-d8_512x512_40k_voc12aug_20200613_161222-bc933b18.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x512_40k_voc12aug/pspnet_r101-d8_512x512_40k_voc12aug_20200613_161222.log.json) |

### Pascal Context

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                            | download                                                                                                                                                                                                                                                                                                                                                             |

| PSPNet | R-101-D8 | 480x480   |   40000 | 8.8      | 9.68           | V100   | 46.60 |         47.78 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb4-40k_pascal-context-480x480.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_480x480_40k_pascal_context/pspnet_r101-d8_480x480_40k_pascal_context_20200911_211210-bf0f5d7c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_480x480_40k_pascal_context/pspnet_r101-d8_480x480_40k_pascal_context-20200911_211210.log.json) |

| PSPNet | R-101-D8 | 480x480   |   80000 | -        | -              | V100   | 46.03 |         47.15 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb4-80k_pascal-context-480x480.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_480x480_80k_pascal_context/pspnet_r101-d8_480x480_80k_pascal_context_20200911_190530-c86d6233.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_480x480_80k_pascal_context/pspnet_r101-d8_480x480_80k_pascal_context-20200911_190530.log.json) |

### Pascal Context 59

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                               | download                                                                                                                                                                                                                                                                                                                                                                         |

| PSPNet | R-101-D8 | 480x480   |   40000 | -        | -              | V100   | 52.02 |         53.54 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb4-40k_pascal-context-59-480x480.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_480x480_40k_pascal_context_59/pspnet_r101-d8_480x480_40k_pascal_context_59_20210416_114524-86d44cd4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_480x480_40k_pascal_context_59/pspnet_r101-d8_480x480_40k_pascal_context_59-20210416_114524.log.json) |

| PSPNet | R-101-D8 | 480x480   |   80000 | -        | -              | V100   | 52.47 |         53.99 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb4-80k_pascal-context-59-480x480.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_480x480_80k_pascal_context_59/pspnet_r101-d8_480x480_80k_pascal_context_59_20210416_114418-fa6caaa2.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_480x480_80k_pascal_context_59/pspnet_r101-d8_480x480_80k_pascal_context_59-20210416_114418.log.json) |

### Dark Zurich and Nighttime Driving

We support evaluation results on these two datasets using models above trained on Cityscapes training set.

| Method | Backbone  | Training Dataset        | Test Dataset              | mIoU  | config                                                                                                                                                  | evaluation checkpoint                                                                                                                                                                                                                                                                                                                                        |

| PSPNet | R-50-D8   | Cityscapes Training set | Dark Zurich               | 10.91 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8_4xb2-40k_cityscapes-512x1024_dark-zurich-1920x1080.py)     | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x1024_40k_cityscapes/pspnet_r50-d8_512x1024_40k_cityscapes_20200605_003338-2966598c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x1024_40k_cityscapes/pspnet_r50-d8_512x1024_40k_cityscapes_20200605_003338.log.json)         |

| PSPNet | R-50-D8   | Cityscapes Training set | Nighttime Driving         | 23.02 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8_4xb2-40k_cityscapes-512x1024_night-driving-1920x1080.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x1024_40k_cityscapes/pspnet_r50-d8_512x1024_40k_cityscapes_20200605_003338-2966598c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x1024_40k_cityscapes/pspnet_r50-d8_512x1024_40k_cityscapes_20200605_003338.log.json)         |

| PSPNet | R-50-D8   | Cityscapes Training set | Cityscapes Validation set | 77.85 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8_4xb2-40k_cityscapes-512x1024.py)                           | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x1024_40k_cityscapes/pspnet_r50-d8_512x1024_40k_cityscapes_20200605_003338-2966598c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x1024_40k_cityscapes/pspnet_r50-d8_512x1024_40k_cityscapes_20200605_003338.log.json)         |

| PSPNet | R-101-D8  | Cityscapes Training set | Dark Zurich               | 10.16 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb2-40k_cityscapes-512x1024_dark-zurich-1920x1080.py)    | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x1024_40k_cityscapes/pspnet_r101-d8_512x1024_40k_cityscapes_20200604_232751-467e7cf4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x1024_40k_cityscapes/pspnet_r101-d8_512x1024_40k_cityscapes_20200604_232751.log.json)     |

| PSPNet | R-101-D8  | Cityscapes Training set | Nighttime Driving         | 20.25 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb2-40k_cityscapes-512x1024_night-driving-1920x1080.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x1024_40k_cityscapes/pspnet_r101-d8_512x1024_40k_cityscapes_20200604_232751-467e7cf4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x1024_40k_cityscapes/pspnet_r101-d8_512x1024_40k_cityscapes_20200604_232751.log.json)     |

| PSPNet | R-101-D8  | Cityscapes Training set | Cityscapes Validation set | 78.34 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb2-40k_cityscapes-512x1024.py)                          | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x1024_40k_cityscapes/pspnet_r101-d8_512x1024_40k_cityscapes_20200604_232751-467e7cf4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x1024_40k_cityscapes/pspnet_r101-d8_512x1024_40k_cityscapes_20200604_232751.log.json)     |

| PSPNet | R-101b-D8 | Cityscapes Training set | Dark Zurich               | 15.54 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101b-d8_4xb2-80k_cityscapes-512x1024_dark-zurich-1920x1080.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101b-d8_512x1024_80k_cityscapes/pspnet_r101b-d8_512x1024_80k_cityscapes_20201226_170012-3a4d38ab.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101b-d8_512x1024_80k_cityscapes/pspnet_r101b-d8_512x1024_80k_cityscapes-20201226_170012.log.json) |

| PSPNet | R-101b-D8 | Cityscapes Training set | Nighttime Driving         | 22.25 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101b-d8_4xb2-80k_cityscapes-512x1024_night-driving-1920x1080.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101b-d8_512x1024_80k_cityscapes/pspnet_r101b-d8_512x1024_80k_cityscapes_20201226_170012-3a4d38ab.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101b-d8_512x1024_80k_cityscapes/pspnet_r101b-d8_512x1024_80k_cityscapes-20201226_170012.log.json) |

| PSPNet | R-101b-D8 | Cityscapes Training set | Cityscapes Validation set | 79.69 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101b-d8_4xb2-80k_cityscapes-512x1024.py)                         | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101b-d8_512x1024_80k_cityscapes/pspnet_r101b-d8_512x1024_80k_cityscapes_20201226_170012-3a4d38ab.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101b-d8_512x1024_80k_cityscapes/pspnet_r101b-d8_512x1024_80k_cityscapes-20201226_170012.log.json) |

### COCO-Stuff 10k

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                           | download                                                                                                                                                                                                                                                                                                                                                                         |

| PSPNet | R-50-D8  | 512x512   |   20000 | 9.6      | 20.5           | V100   | 35.69 |         36.62 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8_4xb4-20k_coco-stuff10k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x512_4x4_20k_coco-stuff10k/pspnet_r50-d8_512x512_4x4_20k_coco-stuff10k_20210820_203258-b88df27f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x512_4x4_20k_coco-stuff10k/pspnet_r50-d8_512x512_4x4_20k_coco-stuff10k_20210820_203258.log.json)     |

| PSPNet | R-101-D8 | 512x512   |   20000 | 13.2     | 11.1           | V100   | 37.26 |         38.52 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb4-20k_coco-stuff10k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x512_4x4_20k_coco-stuff10k/pspnet_r101-d8_512x512_4x4_20k_coco-stuff10k_20210820_232135-76aae482.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x512_4x4_20k_coco-stuff10k/pspnet_r101-d8_512x512_4x4_20k_coco-stuff10k_20210820_232135.log.json) |

| PSPNet | R-50-D8  | 512x512   |   40000 | -        | -              | V100   | 36.33 |         37.24 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8_4xb4-40k_coco-stuff10k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x512_4x4_40k_coco-stuff10k/pspnet_r50-d8_512x512_4x4_40k_coco-stuff10k_20210821_030857-92e2902b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x512_4x4_40k_coco-stuff10k/pspnet_r50-d8_512x512_4x4_40k_coco-stuff10k_20210821_030857.log.json)     |

| PSPNet | R-101-D8 | 512x512   |   40000 | -        | -              | V100   | 37.76 |         38.86 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb4-40k_coco-stuff10k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x512_4x4_40k_coco-stuff10k/pspnet_r101-d8_512x512_4x4_40k_coco-stuff10k_20210821_014022-831aec95.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x512_4x4_40k_coco-stuff10k/pspnet_r101-d8_512x512_4x4_40k_coco-stuff10k_20210821_014022.log.json) |

### COCO-Stuff 164k

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                             | download                                                                                                                                                                                                                                                                                                                                                                                 |

| PSPNet | R-50-D8  | 512x512   |   80000 | 9.6      | 20.5           | V100   | 38.80 |         39.19 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8_4xb4-80k_coco-stuff164k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x512_4x4_80k_coco-stuff164k/pspnet_r50-d8_512x512_4x4_80k_coco-stuff164k_20210707_152034-0e41b2db.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x512_4x4_80k_coco-stuff164k/pspnet_r50-d8_512x512_4x4_80k_coco-stuff164k_20210707_152034.log.json)         |

| PSPNet | R-101-D8 | 512x512   |   80000 | 13.2     | 11.1           | V100   | 40.34 |         40.79 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb4-80k_coco-stuff164k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x512_4x4_80k_coco-stuff164k/pspnet_r101-d8_512x512_4x4_80k_coco-stuff164k_20210707_152034-7eb41789.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x512_4x4_80k_coco-stuff164k/pspnet_r101-d8_512x512_4x4_80k_coco-stuff164k_20210707_152034.log.json)     |

| PSPNet | R-50-D8  | 512x512   |  160000 | -        | -              | V100   | 39.64 |         39.97 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8_4xb4-160k_coco-stuff164k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x512_4x4_160k_coco-stuff164k/pspnet_r50-d8_512x512_4x4_160k_coco-stuff164k_20210707_152004-51276a57.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x512_4x4_160k_coco-stuff164k/pspnet_r50-d8_512x512_4x4_160k_coco-stuff164k_20210707_152004.log.json)     |

| PSPNet | R-101-D8 | 512x512   |  160000 | -        | -              | V100   | 41.28 |         41.66 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb4-160k_coco-stuff164k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x512_4x4_160k_coco-stuff164k/pspnet_r101-d8_512x512_4x4_160k_coco-stuff164k_20210707_152004-4af9621b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x512_4x4_160k_coco-stuff164k/pspnet_r101-d8_512x512_4x4_160k_coco-stuff164k_20210707_152004.log.json) |

| PSPNet | R-50-D8  | 512x512   |  320000 | -        | -              | V100   | 40.53 |         40.75 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8_4xb4-320k_coco-stuff164k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x512_4x4_320k_coco-stuff164k/pspnet_r50-d8_512x512_4x4_320k_coco-stuff164k_20210707_152004-be9610cc.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x512_4x4_320k_coco-stuff164k/pspnet_r50-d8_512x512_4x4_320k_coco-stuff164k_20210707_152004.log.json)     |

| PSPNet | R-101-D8 | 512x512   |  320000 | -        | -              | V100   | 41.95 |         42.42 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb4-320k_coco-stuff164k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x512_4x4_320k_coco-stuff164k/pspnet_r101-d8_512x512_4x4_320k_coco-stuff164k_20210707_152004-72220c60.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x512_4x4_320k_coco-stuff164k/pspnet_r101-d8_512x512_4x4_320k_coco-stuff164k_20210707_152004.log.json) |

### LoveDA

### LoveDA

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                    | download                                                                                                                                                                                                                                                                                                                             |

| PSPNet | R-18-D8  | 512x512   |   80000 | 1.45     | 26.87          | V100   | 48.62 |         47.57 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r18-d8_4xb4-80k_loveda-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r18-d8_512x512_80k_loveda/pspnet_r18-d8_512x512_80k_loveda_20211105_052100-b97697f1.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r18-d8_512x512_80k_loveda/pspnet_r18-d8_512x512_80k_loveda_20211105_052100.log.json)     |

| PSPNet | R-50-D8  | 512x512   |   80000 | 6.14     | 6.60           | V100   | 50.46 |         50.19 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8_4xb4-80k_loveda-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x512_80k_loveda/pspnet_r50-d8_512x512_80k_loveda_20211104_155728-88610f9f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_512x512_80k_loveda/pspnet_r50-d8_512x512_80k_loveda_20211104_155728.log.json)     |

| PSPNet | R-101-D8 | 512x512   |   80000 | 9.61     | 4.58           | V100   | 51.86 |         51.34 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb4-80k_loveda-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x512_80k_loveda/pspnet_r101-d8_512x512_80k_loveda_20211104_153212-1c06c6a8.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_512x512_80k_loveda/pspnet_r101-d8_512x512_80k_loveda_20211104_153212.log.json) |

### Potsdam

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                     | download                                                                                                                                                                                                                                                                                                                                                 |

| PSPNet | R-18-D8  | 512x512   |   80000 | 1.50     | 85.12          | V100   | 77.09 |         78.30 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r18-d8_4xb4-80k_potsdam-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r18-d8_4x4_512x512_80k_potsdam/pspnet_r18-d8_4x4_512x512_80k_potsdam_20211220_125612-7cd046e1.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r18-d8_4x4_512x512_80k_potsdam/pspnet_r18-d8_4x4_512x512_80k_potsdam_20211220_125612.log.json)     |

| PSPNet | R-50-D8  | 512x512   |   80000 | 6.14     | 30.21          | V100   | 78.12 |         78.98 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8_4xb4-80k_potsdam-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_4x4_512x512_80k_potsdam/pspnet_r50-d8_4x4_512x512_80k_potsdam_20211219_043541-2dd5fe67.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_4x4_512x512_80k_potsdam/pspnet_r50-d8_4x4_512x512_80k_potsdam_20211219_043541.log.json)     |

| PSPNet | R-101-D8 | 512x512   |   80000 | 9.61     | 19.40          | V100   | 78.62 |         79.47 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb4-80k_potsdam-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_4x4_512x512_80k_potsdam/pspnet_r101-d8_4x4_512x512_80k_potsdam_20211220_125612-aed036c4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_4x4_512x512_80k_potsdam/pspnet_r101-d8_4x4_512x512_80k_potsdam_20211220_125612.log.json) |

### Vaihingen

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                       | download                                                                                                                                                                                                                                                                                                                                                         |

| PSPNet | R-18-D8  | 512x512   |   80000 | 1.45     | 85.06          | V100   | 71.46 |         73.36 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r18-d8_4xb4-80k_vaihingen-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r18-d8_4x4_512x512_80k_vaihingen/pspnet_r18-d8_4x4_512x512_80k_vaihingen_20211228_160355-52a8a6f6.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r18-d8_4x4_512x512_80k_vaihingen/pspnet_r18-d8_4x4_512x512_80k_vaihingen_20211228_160355.log.json)     |

| PSPNet | R-50-D8  | 512x512   |   80000 | 6.14     | 30.29          | V100   | 72.36 |         73.75 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8_4xb4-80k_vaihingen-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_4x4_512x512_80k_vaihingen/pspnet_r50-d8_4x4_512x512_80k_vaihingen_20211228_160355-382f8f5b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_4x4_512x512_80k_vaihingen/pspnet_r50-d8_4x4_512x512_80k_vaihingen_20211228_160355.log.json)     |

| PSPNet | R-101-D8 | 512x512   |   80000 | 9.61     | 19.97          | V100   | 72.61 |         74.18 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r101-d8_4xb4-80k_vaihingen-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_4x4_512x512_80k_vaihingen/pspnet_r101-d8_4x4_512x512_80k_vaihingen_20211231_230806-8eba0a09.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r101-d8_4x4_512x512_80k_vaihingen/pspnet_r101-d8_4x4_512x512_80k_vaihingen_20211231_230806.log.json) |

### iSAID

### iSAID

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                  | download                                                                                                                                                                                                                                                                                                                                     |

| PSPNet | R-18-D8  | 896x896   |   80000 | 4.52     | 26.91          | V100   | 60.22 |         61.25 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r18-d8_4xb4-80k_isaid-896x896.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r18-d8_4x4_896x896_80k_isaid/pspnet_r18-d8_4x4_896x896_80k_isaid_20220110_180526-e84c0b6a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r18-d8_4x4_896x896_80k_isaid/pspnet_r18-d8_4x4_896x896_80k_isaid_20220110_180526.log.json) |

| PSPNet | R-50-D8  | 896x896   |   80000 | 16.58    | 8.88           | V100   | 65.36 |         66.48 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8_4xb4-80k_isaid-896x896.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_4x4_896x896_80k_isaid/pspnet_r50-d8_4x4_896x896_80k_isaid_20220110_180629-1f21dc32.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pspnet/pspnet_r50-d8_4x4_896x896_80k_isaid/pspnet_r50-d8_4x4_896x896_80k_isaid_20220110_180629.log.json) |

Note:

- `FP16` means Mixed Precision (FP16) is adopted in training.
- `896x896` is the Crop Size of iSAID dataset, which is followed by the implementation of [PointFlow: Flowing Semantics Through Points for Aerial Image Segmentation](https://arxiv.org/pdf/2103.06564.pdf)
- `rsb` is short for 'Resnet strikes back'.

- The `b` in `R-50b` means ResNetV1b, which is a standard ResNet backbone. In MMSegmentation, default backbone is ResNetV1c, which usually performs better in semantic segmentation task.

```bibtex
@inproceedings{zhao2017pspnet,
  title={Pyramid Scene Parsing Network},
  author={Zhao, Hengshuang and Shi, Jianping and Qi, Xiaojuan and Wang, Xiaogang and Jia, Jiaya},
}
```

```bibtex
@article{wightman2021resnet,
  title={Resnet strikes back: An improved training procedure in timm},
  author={Wightman, Ross and Touvron, Hugo and J{\'e}gou, Herv{\'e}},
  journal={arXiv preprint arXiv:2110.00476},
}
```

# Dataset Migration

Based on the new design of [BaseDataset](mmengine.dataset.BaseDataset) in [MMEngine](https://github.com/open-mmlab/mmengine), we have refactored the base OCR dataset class [`OCRDataset`](mmocr.datasets.OCRDataset) in MMOCR 1.0. The following document describes the differences between the old and new dataset formats in MMOCR, and how to migrate from the deprecated version to the latest. For users who do not want to migrate datasets at this time, we also provide a temporary solution in [Section Compatibility](#compatibility).

```{note}
The Key Information Extraction task still uses the original WildReceipt dataset annotation format.
```

## Review of Old Dataset Formats

```{note}
```

MMOCR version 0.x implements a number of dataset classes, such as `IcdarDataset`, `TextDetDataset` for text detection tasks, and `OCRDataset`, `OCRSegDataset` for text recognition tasks. At the same time, the annotations may vary in different formats, such as `.txt`, `.json`, `.jsonl`. Users have to manually configure the `Loader` and the `Parser` while customizing the datasets.

### Text Detection

For the text detection task, `IcdarDataset` uses a COCO-like annotation format.

```json
{
    {
      "id": 1,
    }
  ],
    {
      "id": 1,
      ],
    }
  ]
}
```

"image_id": 1,
      ],
    }
  ]
}
```

The `TextDetDataset` uses the JSON Line storage format, converting COCO-like labels to strings and saves them in `.txt` or `.jsonl` format files.

```text
{"file_name": "test/img_2.jpg", "height": 720, "width": 1280,  "annotations": [{"iscrowd": 0, "category_id": 1, "bbox": [602.0, 173.0,  33.0, 24.0], "segmentation": [[602, 173, 635, 175, 634, 197, 602,  196]]}, {"iscrowd": 0, "category_id": 1, "bbox": [734.0, 310.0, 58.0,  54.0], "segmentation": [[734, 310, 792, 320, 792, 364, 738, 361]]}]}

{"file_name": "test/img_5.jpg", "height": 720, "width": 1280,  "annotations": [{"iscrowd": 1, "category_id": 1, "bbox": [405.0, 409.0,  32.0, 52.0], "segmentation": [[408, 409, 437, 436, 434, 461, 405,  433]]}, {"iscrowd": 1, "category_id": 1, "bbox": [435.0, 434.0, 8.0,  33.0], "segmentation": [[437, 434, 443, 440, 441, 467, 435, 462]]}]}
```

### Text Recognition

For text recognition tasks, there are two annotation formats in MMOCR version 0.x. The simple `.txt` annotations separate image name and word annotation by a blank space, which cannot handle the case when spaces are included in a text instance.

```text
```

The JSON Line format uses a dictionary-like structure to represent the annotations, where the keys `filename` and `text` store the image name and word label, respectively.

```json
```

## New Dataset Format

```json
```

To solve the dataset issues, MMOCR 1.x adopts a unified dataset design introduced in MMEngine. Each annotation file is a `.json` file that stores a `dict`, containing both `metainfo` and `data_list`, where the former includes basic information about the dataset and the latter consists of the label item of each target instance.

```json
{
  "metainfo":
    {
      "classes": ("cat", "dog"),
      // ...
    },
  "data_list":
    [
      {
        "img_path": "xxx/xxx_0.jpg",
        "img_label": 0,
        // ...
      },
      // ...
    ]
}
```

Based on the above structure, we introduced `TextDetDataset`, `TextRecogDataset` for MMOCR-specific tasks.

#### Introduction of the New Format

},
      // ...
    ]
}
```

The `TextDetDataset` holds the information required by the text detection task, such as bounding boxes and labels. We refer users to `tests/data/det_toy_dataset/instances_test.json` which is an example annotation for `TextDetDataset`.

```json
{
    {
      "dataset_type": "TextDetDataset",
      "task_name": "textdet",
      "category": [{"id": 0, "name": "text"}]
    },
    [
      {
        "height": 640,
        "instances":
          [
            {
              "polygon": [0, 0, 0, 10, 10, 20, 20, 0],
              "bbox": [0, 0, 10, 20],
              "bbox_label": 0,
              "ignore": False
            }，
            // ...
          ]
      }
    ]
}
```

"bbox": [0, 0, 10, 20],
            }，
            // ...
          ]
      }
    ]
}
```

The bounding box format is as follows: `[min_x, min_y, max_x, max_y]`

#### Migration Script

We provide a migration script to help users migrate old annotation files to the new format.

```bash
python tools/dataset_converters/textdet/data_migrator.py ${IN_PATH} ${OUT_PATH}
```

| ARGS     | Type                             | Description                                                                                                                                                      |
| -------- | -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |

| in_path  | str                              | （Required）Path to the old annotation file.                                                                                                                     |
| out_path | str                              | （Required）Path to the new annotation file.                                                                                                                     |
| --task   | 'auto', 'textdet', 'textspotter' | Specifies the compatible task for the output dataset annotation. If 'textdet' is specified, the text field in coco format will not be dumped. The default is 'auto', which automatically determines the output format based on the the old annotation files. |

The `TextRecogDataset` holds the information required by the text detection task, such as text and image path. We refer users to `tests/data/rec_toy_dataset/labels.json` which is an example annotation for `TextRecogDataset`.

```json
{
    {
      "dataset_type": "TextRecogDataset",
      "task_name": "textrecog",
    },
    [
      {
            [
              {
                "text": "GRAND"
              }
            ]
      }
    ]
}
```

```bash
python tools/dataset_converters/textrecog/data_migrator.py ${IN_PATH} ${OUT_PATH} --format ${txt, jsonl, lmdb}
```

| ARGS     | Type                   | Description                                       |

```

| -------- | ---------------------- | ------------------------------------------------- |
| in_path  | str                    | （Required）Path to the old annotation file.      |
| out_path | str                    | （Required）Path to the new annotation file.      |
| --format | 'txt', 'jsonl', 'lmdb' | Specify the format of the old dataset annotation. |

## Compatibility

In consideration of the cost to users for data migration, we have temporarily made MMOCR version 1.x compatible with the old MMOCR 0.x format.

```{note}
The code and components used for compatibility with the old data format may be completely removed in a future release. Therefore, we strongly recommend that users migrate their datasets to the new data format.
```

```

Specifically, we provide three dataset classes [IcdarDataset](mmocr.datasets.IcdarDataset), [RecogTextDataset](mmocr.datasets.RecogTextDataset), [RecogLMDBDataset](mmocr.datasets.RecogLMDBDataset) to support the old formats.

1. [IcdarDataset](mmocr.datasets.IcdarDataset) supports COCO-like format annotations for text detection. You just need to add a new dataset config to `configs/textdet/_base_/datasets` and specify its dataset type as `IcdarDataset`.

   ```python
   data_root = 'data/det/icdar2015'
   train_anno_path = 'instances_training.json'

   train_dataset = dict(
       type='IcdarDataset',
       data_root=data_root,
       ann_file=train_anno_path,
       data_prefix=dict(img_path='imgs/'),
       filter_cfg=dict(filter_empty_gt=True, min_size=32),
       pipeline=None)
   ```

   ```

2. [RecogTextDataset](mmocr.datasets.RecogTextDataset) supports `.txt` and `.jsonl` format annotations for text recognition. You just need to add a new dataset config to `configs/textrecog/_base_/datasets` and specify its dataset type as `RecogTextDataset`. For example, the following example shows how to configure and load the 0.x format labels `old_label.txt` and `old_label.jsonl` from the toy dataset.

   ```python
    data_root = 'tests/data/rec_toy_dataset/'

    # loading 0.x txt format annos
    txt_dataset = dict(
        type='RecogTextDataset',
        ann_file='old_label.txt',
        data_prefix=dict(img_path='imgs'),
        parser_cfg=dict(
            type='LineStrParser',
            keys=['filename', 'text'],
            keys_idx=[0, 1]),

data_prefix=dict(img_path='imgs'),
        pipeline=[])

    # loading 0.x json line format annos
    jsonl_dataset = dict(
        ann_file='old_label.jsonl',
            type='LineJsonParser',
        pipeline=[]))
   ```

3. [RecogLMDBDataset](mmocr.datasets.RecogLMDBDataset) supports LMDB format dataset (img+labels) for text recognition. You just need to add a new dataset config to `configs/textrecog/_base_/datasets` and specify its dataset type as `RecogLMDBDataset`. For example, the following example shows how to configure and load the **both labels and images** `imgs.lmdb` from the toy dataset.

- set the dataset type to `RecogLMDBDataset`

```python

```python
# Specify the dataset type as RecogLMDBDataset
 data_root = 'tests/data/rec_toy_dataset/'

 lmdb_dataset = dict(
     type='RecogLMDBDataset',
     ann_file='imgs.lmdb',
     pipeline=None)
```

- replace the [`LoadImageFromFile`](mmocr.datasets.transforms.LoadImageFromFile) with [`LoadImageFromNDArray`](mmocr.datasets.transforms.LoadImageFromNDArray) in the data pipelines in `train_pipeline` and `test_pipeline`., for example：

```python
 train_pipeline = [dict(type='LoadImageFromNDArray')]
```

# PAFPN

> [Path Aggregation Network for Instance Segmentation](https://arxiv.org/abs/1803.01534)

The way that information propagates in neural networks is of great importance. In this paper, we propose Path Aggregation Network (PANet) aiming at boosting information flow in proposal-based instance segmentation framework. Specifically, we enhance the entire feature hierarchy with accurate localization signals in lower layers by bottom-up path augmentation, which shortens the information path between lower layers and topmost feature. We present adaptive feature pooling, which links feature grid and all feature levels to make useful information in each feature level propagate directly to following proposal subnetworks. A complementary branch capturing different views for each proposal is created to further improve mask prediction. These improvements are simple to implement, with subtle extra computational overhead. Our PANet reaches the 1st place in the COCO 2017 Challenge Instance Segmentation task and the 2nd place in Object Detection task without large-batch training. It is also state-of-the-art on MVD and Cityscapes.

<img src="https://user-images.githubusercontent.com/40661020/143968947-5f2d7e8a-a236-4d59-8f2d-7fbb12764845.png"/>
</div>

| Backbone |  style  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                    Config                    |                                                                                                                                                          Download                                                                                                                                                           |

| :------: | :-----: | :-----: | :------: | :------------: | :----: | :-----: | :------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| R-50-FPN | pytorch |   1x    |   4.0    |      17.2      |  37.5  |         | [config](./faster-rcnn_r50_pafpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/pafpn/faster_rcnn_r50_pafpn_1x_coco/faster_rcnn_r50_pafpn_1x_coco_bbox_mAP-0.375_20200503_105836-b7b4b9bd.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pafpn/faster_rcnn_r50_pafpn_1x_coco/faster_rcnn_r50_pafpn_1x_coco_20200503_105836.log.json) |

```latex
@inproceedings{liu2018path,
  author = {Shu Liu and
            Lu Qi and

```latex
            Lu Qi and
            Haifang Qin and
            Jianping Shi and
            Jiaya Jia},
  title = {Path Aggregation Network for Instance Segmentation},
  booktitle = {Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2018}
}
```

# FCENet

> [Fourier Contour Embedding for Arbitrary-Shaped Text Detection](https://arxiv.org/abs/2104.10442)

One of the main challenges for arbitrary-shaped text detection is to design a good text instance representation that allows networks to learn diverse text geometry variances. Most of existing methods model text instances in image spatial domain via masks or contour point sequences in the Cartesian or the polar coordinate system. However, the mask representation might lead to expensive post-processing, while the point sequence one may have limited capability to model texts with highly-curved shapes. To tackle these problems, we model text instances in the Fourier domain and propose one novel Fourier Contour Embedding (FCE) method to represent arbitrary shaped text contours as compact signatures. We further construct FCENet with a backbone, feature pyramid networks (FPN) and a simple post-processing with the Inverse Fourier Transformation (IFT) and Non-Maximum Suppression (NMS). Different from previous methods, FCENet first predicts compact Fourier signatures of text instances, and then reconstructs text contours via IFT and NMS during test. Extensive experiments demonstrate that FCE is accurate and robust to fit contours of scene texts even with highly-curved shapes, and also validate the effectiveness and the good generalization of FCENet for arbitrary-shaped text detection. Furthermore, experimental results show that our FCENet is superior to the state-of-the-art (SOTA) methods on CTW1500 and Total-Text, especially on challenging highly-curved text subset.

<img src="https://user-images.githubusercontent.com/22607038/142791859-1b0ebde4-b151-4c25-ba1b-f354bd8ddc8c.png"/>
</div>

|                 Method                 |                 Backbone                  | Pretrained Model | Training set  |   Test set   | #epochs |  Test size  | Precision | Recall | Hmean  |                 Download                  |
| :------------------------------------: | :---------------------------------------: | :--------------: | :-----------: | :----------: | :-----: | :---------: | :-------: | :----: | :----: | :---------------------------------------: |

| [FCENet_r50dcn](/configs/textdet/fcenet/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500.py) |             ResNet50 + DCNv2              |        -         | CTW1500 Train | CTW1500 Test |  1500   | (736, 1080) |  0.8689   | 0.8296 | 0.8488 | [model](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500_20220825_221510-4d705392.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500/20220825_221510.log) |

| [FCENet_r50-oclip](/configs/textdet/fcenet/fcenet_resnet50-oclip-dcnv2_fpn_1500e_ctw1500.py) | [ResNet50-oCLIP](https://download.openmmlab.com/mmocr/backbone/resnet50-oclip-7ba0c533.pth) |        -         | CTW1500 Train | CTW1500 Test |  1500   | (736, 1080) |  0.8383   | 0.801  | 0.8192 | [model](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50-oclip_fpn_1500e_ctw1500/fcenet_resnet50-oclip_fpn_1500e_ctw1500_20221102_121909-101df7e6.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50-oclip_fpn_1500e_ctw1500/20221102_121909.log) |

### ICDAR2015

|                        Method                         |    Backbone    | Pretrained Model | Training set | Test set  | #epochs |  Test size   | Precision | Recall | Hmean  |                         Download                         |

| :---------------------------------------------------: | :------------: | :--------------: | :----------: | :-------: | :-----: | :----------: | :-------: | :----: | :----: | :------------------------------------------------------: |
| [FCENet_r50](/configs/textdet/fcenet/fcenet_resnet50_fpn_1500e_icdar2015.py) |    ResNet50    |        -         |  IC15 Train  | IC15 Test |  1500   | (2260, 2260) |  0.8243   | 0.8834 | 0.8528 | [model](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50_fpn_1500e_icdar2015/fcenet_resnet50_fpn_1500e_icdar2015_20220826_140941-167d9042.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50_fpn_1500e_icdar2015/20220826_140941.log) |

| [FCENet_r50-oclip](/configs/textdet/fcenet/fcenet_resnet50-oclip_fpn_1500e_icdar2015.py) | ResNet50-oCLIP |        -         |  IC15 Train  | IC15 Test |  1500   | (2260, 2260) |  0.9176   | 0.8098 | 0.8604 | [model](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50-oclip_fpn_1500e_icdar2015/fcenet_resnet50-oclip_fpn_1500e_icdar2015_20221101_150145-5a6fc412.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50-oclip_fpn_1500e_icdar2015/20221101_150145.log) |

### Total Text

|                        Method                         | Backbone | Pretrained Model |  Training set   |    Test set    | #epochs |  Test size  | Precision | Recall | Hmean  |                        Download                         |
| :---------------------------------------------------: | :------: | :--------------: | :-------------: | :------------: | :-----: | :---------: | :-------: | :----: | :----: | :-----------------------------------------------------: |

| [FCENet_r50](/configs/textdet/fcenet/fcenet_resnet50_fpn_1500e_totaltext.py) | ResNet50 |        -         | Totaltext Train | Totaltext Test |  1500   | (1280, 960) |  0.8485   | 0.7810 | 0.8134 | [model](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50_fpn_1500e_totaltext/fcenet_resnet50_fpn_1500e_totaltext-91bd37af.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50_fpn_1500e_totaltext/20221219_201107.log) |

```bibtex
@InProceedings{zhu2021fourier,
      title={Fourier Contour Embedding for Arbitrary-Shaped Text Detection},
      author={Yiqin Zhu and Jianyong Chen and Lingyu Liang and Zhanghui Kuang and Lianwen Jin and Wayne Zhang},
      booktitle = {CVPR}
      }
```

# GET STARTED

MMRotate works on Linux and Windows. It requires Python 3.7+, CUDA 9.2+ and PyTorch 1.6+.

```{note}
```

**Step 0.** Download and install Miniconda from the [official website](https://docs.conda.io/en/latest/miniconda.html).

**Step 1.** Create a conda environment and activate it.

```shell
```

**Step 2.** Install PyTorch following [official instructions](https://pytorch.org/get-started/locally/), e.g.

```shell
conda install pytorch==1.8.0 torchvision==0.9.0 cudatoolkit=10.2 -c pytorch
```

```shell
```

We recommend that users follow our best practices to install MMRotate. However, the whole process is highly customizable. See [Customize Installation](#customize-installation) section for more information.

### Best Practices

**Step 0.** Install [MMEngine](https://github.com/open-mmlab/mmengine) and [MMCV](https://github.com/open-mmlab/mmcv) using [MIM](https://github.com/open-mmlab/mim).

```shell
```

**Step 1.** Install [MMDetection](https://github.com/open-mmlab/mmdetection) as a dependency.

```shell
mim install 'mmdet>=3.0.0rc2'
```

Optionally, you could also build MMDetection from source in case you want to modify the code:

```shell
git clone https://github.com/open-mmlab/mmdetection.git -b dev-3.x
# "-b dev-3.x" means checkout to the `dev-3.x` branch.

```shell
# "-v" means verbose, or more output
# "-e" means installing a project in editable mode,
# thus any local modifications made to the code will take effect without reinstallation.
```

**Step 2.** Install MMRotate.

Case a: If you develop and run mmrotate directly, install it from source:

```shell
git clone https://github.com/open-mmlab/mmrotate.git -b dev-1.x
# "-b dev-1.x" means checkout to the `dev-1.x` branch.
cd mmrotate
```

Case b: If you use mmrotate as a dependency or third-party package, install it with pip:

```shell
pip install mmrotate
```

### Verify the installation

```

```shell
```

To verify whether MMRotate is installed correctly, we provide some sample codes to run an inference demo.

**Step 1.** We need to download config and checkpoint files.

```shell
mim download mmrotate --config oriented-rcnn-le90_r50_fpn_1x_dota --dest .
```

The downloading will take several seconds or more, depending on your network environment. When it is done, you will find two files `oriented-rcnn-le90_r50_fpn_1x_dota.py` and `oriented_rcnn_r50_fpn_1x_dota_le90-6d2b2ce0.pth` in your current folder.

**Step 2.** Verify the inference demo.

Option (a). If you install mmrotate from source, just run the following command.

```shell
python demo/image_demo.py demo/demo.jpg oriented-rcnn-le90_r50_fpn_1x_dota.py oriented_rcnn_r50_fpn_1x_dota_le90-6d2b2ce0.pth --out-file result.jpg
```

```shell
```

You will see a new image `result.jpg` on your current folder, where rotated bounding boxes are plotted on cars, buses, etc.

Option (b). If you install mmrotate with pip, open you python interpreter and copy&paste the following codes.

```python
from mmdet.apis import init_detector, inference_detector
import mmrotate

config_file = 'oriented-rcnn-le90_r50_fpn_1x_dota.py'
checkpoint_file = 'oriented_rcnn_r50_fpn_1x_dota_le90-6d2b2ce0.pth'
model = init_detector(config_file, checkpoint_file, device='cuda:0')
inference_detector(model, 'demo/demo.jpg')
```

You will see a list of arrays printed, indicating the detected rotated bounding boxes.

### Customize Installation

#### CUDA versions

```{note}

```{note}
Installing CUDA runtime libraries is enough if you follow our best practices, because no CUDA code will be compiled locally. However if you hope to compile MMCV from source or develop other CUDA operators, you need to install the complete CUDA toolkit from NVIDIA's [website](https://developer.nvidia.com/cuda-downloads), and its version should match the CUDA version of PyTorch. i.e., the specified version of cudatoolkit in `conda install` command.
```

MMCV contains C++ and CUDA extensions, thus depending on PyTorch in a complex way. MIM solves such dependencies automatically and makes the installation easier. However, it is not a must.

To install MMCV with pip instead of MIM, please follow [MMCV installation guides](https://mmcv.readthedocs.io/en/latest/get_started/installation.html). This requires manually specifying a find-url based on PyTorch version and its CUDA version.

For example, the following command install mmcv built for PyTorch 1.9.x and CUDA 10.2.

```shell
pip install mmcv -f https://download.openmmlab.com/mmcv/dist/cu102/torch1.8/index.html
```

#### Install on Google Colab

[Google Colab](https://research.google.com/) usually has PyTorch installed,
thus we only need to install MMCV and MMDetection with the following commands.

**Step 1.** Install [MMCV](https://github.com/open-mmlab/mmcv) and [MMDetection](https://github.com/open-mmlab/mmdetection) using [MIM](https://github.com/open-mmlab/mim).

```shell
!pip3 install -U openmim
!mim install "mmcv>=2.0.0rc2"
!mim install 'mmdet>=3.0.0rc2'
```

**Step 2.** Install MMRotate from the source.

```shell
!git clone https://github.com/open-mmlab/mmrotate.git -b dev-1.x
%cd mmrotate
!pip install -e .
```

**Step 3.** Verification.

```python
print(mmrotate.__version__)
# Example output: 1.x
```

```{note}

```

```python
```

```{note}
Within Jupyter, the exclamation mark `!` is used to call external executables and `%cd` is a [magic command](https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-cd) to change the current working directory of Python.
```

#### Using MMRotate with Docker

We provide a [Dockerfile](https://github.com/open-mmlab/mmrotate/tree/main/docker/Dockerfile) to build an image. Ensure that your [docker version](https://docs.docker.com/engine/install/) >=19.03.

```shell
# build an image with PyTorch 1.6, CUDA 10.1
docker build -t mmrotate docker/
```

Run it with

```shell
docker run --gpus all --shm-size=8g -it -v {DATA_DIR}:/mmrotate/data mmrotate
```

### Trouble shooting

If you have some issues during the installation, please first view the [FAQ](faq.md) page.

```

You may [open an issue](https://github.com/open-mmlab/mmrotate/issues/new/choose) on GitHub if no solution is found.

# Center-based 3D Object Detection and Tracking

> [Center-based 3D Object Detection and Tracking](https://arxiv.org/abs/2006.11275)

Three-dimensional objects are commonly represented as 3D boxes in a point-cloud. This representation mimics the well-studied image-based 2D bounding-box detection but comes with additional challenges. Objects in a 3D world do not follow any particular orientation, and box-based detectors have difficulties enumerating all orientations or fitting an axis-aligned bounding box to rotated objects. In this paper, we instead propose to represent, detect, and track 3D objects as points. Our framework, CenterPoint, first detects centers of objects using a keypoint detector and regresses to other attributes, including 3D size, 3D orientation, and velocity. In a second stage, it refines these estimates using additional point features on the object. In CenterPoint, 3D object tracking simplifies to greedy closest-point matching. The resulting detection and tracking algorithm is simple, efficient, and effective. CenterPoint achieved state-of-the-art performance on the nuScenes benchmark for both 3D detection and tracking, with 65.5 NDS and 63.8 AMOTA for a single model. On the Waymo Open Dataset, CenterPoint outperforms all previous single model method by a large margin and ranks first among all Lidar-only submissions.

<img src="https://user-images.githubusercontent.com/30491025/143854976-11af75ae-e828-43ad-835d-ac1146f99925.png" width="800"/>
</div>

We implement CenterPoint and provide the result and checkpoints on nuScenes dataset.

`{xxx}` is required field and `[yyy]` is optional.

`{model}`: model type like `centerpoint`.

`{model setting}`: voxel size and voxel type like `01voxel`, `02pillar`.

`{backbone}`: backbone type like `second`.

`{neck}`: neck type like `secfpn`.

`[dcn]`: Whether to use deformable convolution.

`[circle]`: Whether to use circular nms.

`[batch_per_gpu x gpu]`: GPUs and samples per GPU, 4x8 is used by default.

`{schedule}`: training schedule, options are 1x, 2x, 20e, etc. 1x and 2x means 12 epochs and 24 epochs respectively. 20e is adopted in cascade models, which denotes 20 epochs. For 1x/2x, initial learning rate decays by a factor of 10 at the 8/16th and 11/22th epochs. For 20e, initial learning rate decays by a factor of 10 at the 16th and 19th epochs.

`{dataset}`: dataset like nus-3d, kitti-3d, lyft-3d, scannet-3d, sunrgbd-3d. We also indicate the number of classes we are using if there exist multiple settings, e.g., kitti-3d-3class and kitti-3d-car means training on KITTI dataset with 3 classes and single class, respectively.

## Usage

### Test time augmentation

We have supported double-flip and scale augmentation during test time. To use test time augmentation, users need to modify the

## Usage

`test_pipeline` and `test_cfg` in the config.
For example, we change `centerpoint_0075voxel_second_secfpn_circlenms_4x8_cyclic_20e_nus.py` to the following.

```python
_base_ = './centerpoint_0075voxel_second_secfpn_circlenms' \
         '_4x8_cyclic_20e_nus.py'

        pts=dict(
            use_rotate_nms=True,
            max_num=83)))

point_cloud_range = [-54, -54, -5.0, 54, 54, 3.0]
backend_args = None
class_names = [
    'car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier',
    'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone'
]

    dict(
    dict(
        sweeps_num=9,

dict(
    dict(
        use_dim=[0, 1, 2, 3, 4],
        backend_args=backend_args,
        pad_empty_sweeps=True,
        remove_close=True),
    dict(
        flip=True,
        pcd_horizontal_flip=True,
        pcd_vertical_flip=True,
            dict(
            dict(type='RandomFlip3D', sync_2d=False),
            dict(
        ]),
]

dict(
        ]),
]

    val=dict(pipeline=test_pipeline), test=dict(pipeline=test_pipeline))

```

### CenterPoint

|                                           Backbone                                           | Voxel type (voxel size) | Dcn | Circular nms | Mem (GB) | Inf time (fps) |  mAP  |  NDS  |                                                                                                                                                                                                                                               Download                                                                                                                                                                                                                                                |

|    [SECFPN](./centerpoint_voxel01_second_secfpn_head-circlenms_8xb4-cyclic-20e_nus-3d.py)    |       voxel (0.1)       |  ✗  |      ✓       |   5.2    |                | 56.11 | 64.61 |             [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/centerpoint/centerpoint_01voxel_second_secfpn_circlenms_4x8_cyclic_20e_nus/centerpoint_01voxel_second_secfpn_circlenms_4x8_cyclic_20e_nus_20220810_030004-9061688e.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/centerpoint/centerpoint_01voxel_second_secfpn_circlenms_4x8_cyclic_20e_nus/centerpoint_01voxel_second_secfpn_circlenms_4x8_cyclic_20e_nus_20220810_030004.log)             |

|                                     above w/o circle nms                                     |       voxel (0.1)       |  ✗  |      ✗       |          |                |   x   |   x   |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |

|  [SECFPN](./centerpoint_voxel01_second_secfpn_head-dcn-circlenms_8xb4-cyclic-20e_nus-3d.py)  |       voxel (0.1)       |  ✓  |      ✓       |   5.5    |                | 56.10 | 64.69 |     [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/centerpoint/centerpoint_01voxel_second_secfpn_dcn_circlenms_4x8_cyclic_20e_nus/centerpoint_01voxel_second_secfpn_dcn_circlenms_4x8_cyclic_20e_nus_20220810_052355-a6928835.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/centerpoint/centerpoint_01voxel_second_secfpn_dcn_circlenms_4x8_cyclic_20e_nus/centerpoint_01voxel_second_secfpn_dcn_circlenms_4x8_cyclic_20e_nus_20220810_052355.log)     |

|                                     above w/o circle nms                                     |       voxel (0.1)       |  ✓  |      ✗       |          |                |   x   |   x   |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |

|   [SECFPN](./centerpoint_voxel0075_second_secfpn_head-circlenms_8xb4-cyclic-20e_nus-3d.py)   |      voxel (0.075)      |  ✗  |      ✓       |   8.2    |                | 56.54 | 65.17 |         [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/centerpoint/centerpoint_0075voxel_second_secfpn_circlenms_4x8_cyclic_20e_nus/centerpoint_0075voxel_second_secfpn_circlenms_4x8_cyclic_20e_nus_20220810_011659-04cb3a3b.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/centerpoint/centerpoint_0075voxel_second_secfpn_circlenms_4x8_cyclic_20e_nus/centerpoint_0075voxel_second_secfpn_circlenms_4x8_cyclic_20e_nus_20220810_011659.log)         |

|                                     above w/o circle nms                                     |      voxel (0.075)      |  ✗  |      ✗       |          |                | 57.63 | 65.39 |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |

| [SECFPN](./centerpoint_voxel0075_second_secfpn_head-dcn-circlenms_8xb4-cyclic-20e_nus-3d.py) |      voxel (0.075)      |  ✓  |      ✓       |   8.7    |                | 56.92 | 65.27 | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/centerpoint/centerpoint_0075voxel_second_secfpn_dcn_circlenms_4x8_cyclic_20e_nus/centerpoint_0075voxel_second_secfpn_dcn_circlenms_4x8_cyclic_20e_nus_20220810_025930-657f67e0.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/centerpoint/centerpoint_0075voxel_second_secfpn_dcn_circlenms_4x8_cyclic_20e_nus/centerpoint_0075voxel_second_secfpn_dcn_circlenms_4x8_cyclic_20e_nus_20220810_025930.log) |

|                                     above w/o circle nms                                     |      voxel (0.075)      |  ✓  |      ✗       |          |                | 57.43 | 65.63 |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |

|                                     above w/ double flip                                     |      voxel (0.075)      |  ✓  |      ✗       |          |                | 59.73 | 67.39 |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |

|                                      above w/ scale tta                                      |      voxel (0.075)      |  ✓  |      ✗       |          |                | 60.43 | 67.65 |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |

|                              above w/ circle nms w/o scale tta                               |      voxel (0.075)      |  ✓  |      ✗       |          |                | 59.52 | 67.24 |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |

|   [SECFPN](./centerpoint_pillar02_second_secfpn_head-circlenms_8xb4-cyclic-20e_nus-3d.py)    |      pillar (0.2)       |  ✗  |      ✓       |   4.6    |                | 48.70 | 59.62 |           [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/centerpoint/centerpoint_02pillar_second_secfpn_circlenms_4x8_cyclic_20e_nus/centerpoint_02pillar_second_secfpn_circlenms_4x8_cyclic_20e_nus_20220811_031844-191a3822.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/centerpoint/centerpoint_02pillar_second_secfpn_circlenms_4x8_cyclic_20e_nus/centerpoint_02pillar_second_secfpn_circlenms_4x8_cyclic_20e_nus_20220811_031844.log)           |

|                                     above w/o circle nms                                     |      pillar (0.2)       |  ✗  |      ✗       |          |                | 49.12 | 59.66 |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |

|      [SECFPN](./centerpoint_pillar02_second_secfpn_head-dcn_8xb4-cyclic-20e_nus-3d.py)       |      pillar (0.2)       |  ✓  |      ✗       |   4.9    |                | 48.38 | 59.79 |                       [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/centerpoint/centerpoint_02pillar_second_secfpn_dcn_4x8_cyclic_20e_nus/centerpoint_02pillar_second_secfpn_dcn_4x8_cyclic_20e_nus_20220811_045458-808e69ad.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/centerpoint/centerpoint_02pillar_second_secfpn_dcn_4x8_cyclic_20e_nus/centerpoint_02pillar_second_secfpn_dcn_4x8_cyclic_20e_nus_20220811_045458.log)                       |

|                                     above w/ circle nms                                      |      pillar (0.2)       |  ✓  |      ✓       |          |                | 48.79 | 59.65 |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |

**Note:** The model performance after coordinate refactor is slightly different (+/- 0.5 - 1 mAP/NDS) from the performance before coordinate refactor in v0.x branch. We are exploring the reason behind.                                                                                                                                                                                                                                                                                                                                                                                                                                                         |

```latex
@article{yin2021center,
  title={Center-based 3D Object Detection and Tracking},
  author={Yin, Tianwei and Zhou, Xingyi and Kr{\"a}henb{\"u}hl, Philipp},
}
```

# FPG

> [Feature Pyramid Grids](https://arxiv.org/abs/2004.03580)

Feature pyramid networks have been widely adopted in the object detection literature to improve feature representations for better handling of variations in scale. In this paper, we present Feature Pyramid Grids (FPG), a deep multi-pathway feature pyramid, that represents the feature scale-space as a regular grid of parallel bottom-up pathways which are fused by multi-directional lateral connections. FPG can improve single-pathway feature pyramid networks by significantly increasing its performance at similar computation cost, highlighting importance of deep pyramid representations. In addition to its general and uniform structure, over complicated structures that have been found with neural architecture search, it also compares favorably against such approaches without relying on search. We hope that FPG with its uniform and effective nature can serve as a strong component for future work in object recognition.

<img src="https://user-images.githubusercontent.com/40661020/143885611-85902399-2885-4a85-9126-9b9b7464ad08.png"/>
</div>

We benchmark the new training schedule (crop training, large batch, unfrozen BN, 50 epochs) introduced in NAS-FPN.
All backbones are Resnet-50 in pytorch style.

|    Method    |    Neck    | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                           Config                           |                                                                                                                                                                             Download                                                                                                                                                                             |

| Faster R-CNN |    FPG     |   50e   |   20.0   |       -        |  42.3  |    -    |    [config](./faster-rcnn_r50_fpg_crop640-50e_coco.py)     |        [model](https://download.openmmlab.com/mmdetection/v2.0/fpg/faster_rcnn_r50_fpg_crop640_50e_coco/faster_rcnn_r50_fpg_crop640_50e_coco_20220311_011856-74109f42.pth) \|               [log](https://download.openmmlab.com/mmdetection/v2.0/fpg/faster_rcnn_r50_fpg_crop640_50e_coco/faster_rcnn_r50_fpg_crop640_50e_coco_20220311_011856.log.json)        |

| Faster R-CNN | FPG-chn128 |   50e   |   11.9   |       -        |  41.2  |    -    | [config](./faster-rcnn_r50_fpg-chn128_crop640-50e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/fpg/faster_rcnn_r50_fpg-chn128_crop640_50e_coco/faster_rcnn_r50_fpg-chn128_crop640_50e_coco_20220311_011857-9376aa9d.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/fpg/faster_rcnn_r50_fpg-chn128_crop640_50e_coco/faster_rcnn_r50_fpg-chn128_crop640_50e_coco_20220311_011857.log.json) |

| Faster R-CNN |    FPN     |   50e   |   20.0   |       -        |  38.9  |    -    |    [config](./faster-rcnn_r50_fpn_crop640-50e_coco.py)     |        [model](https://download.openmmlab.com/mmdetection/v2.0/fpg/faster_rcnn_r50_fpn_crop640_50e_coco/faster_rcnn_r50_fpn_crop640_50e_coco_20220311_011857-be7c9f42.pth) \|               [log](https://download.openmmlab.com/mmdetection/v2.0/fpg/faster_rcnn_r50_fpn_crop640_50e_coco/faster_rcnn_r50_fpn_crop640_50e_coco_20220311_011857.log.json)        |

|  Mask R-CNN  |    FPG     |   50e   |   23.2   |       -        |  43.0  |  38.1   |     [config](./mask-rcnn_r50_fpg_crop640-50e_coco.py)      |          [model](https://download.openmmlab.com/mmdetection/v2.0/fpg/mask_rcnn_r50_fpg_crop640_50e_coco/mask_rcnn_r50_fpg_crop640_50e_coco_20220311_011857-233b8334.pth) \|                   [log](https://download.openmmlab.com/mmdetection/v2.0/fpg/mask_rcnn_r50_fpg_crop640_50e_coco/mask_rcnn_r50_fpg_crop640_50e_coco_20220311_011857.log.json)          |

|  Mask R-CNN  | FPG-chn128 |   50e   |   15.3   |       -        |  41.7  |  37.1   |  [config](./mask-rcnn_r50_fpg-chn128_crop640-50e_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/fpg/mask_rcnn_r50_fpg-chn128_crop640_50e_coco/mask_rcnn_r50_fpg-chn128_crop640_50e_coco_20220311_011859-043c9b4e.pth) \|     [log](https://download.openmmlab.com/mmdetection/v2.0/fpg/mask_rcnn_r50_fpg-chn128_crop640_50e_coco/mask_rcnn_r50_fpg-chn128_crop640_50e_coco_20220311_011859.log.json)   |

|  Mask R-CNN  |    FPN     |   50e   |   23.2   |       -        |  49.6  |  35.6   |     [config](./mask-rcnn_r50_fpn_crop640-50e_coco.py)      |          [model](https://download.openmmlab.com/mmdetection/v2.0/fpg/mask_rcnn_r50_fpn_crop640_50e_coco/mask_rcnn_r50_fpn_crop640_50e_coco_20220311_011855-a756664a.pth) \|                   [log](https://download.openmmlab.com/mmdetection/v2.0/fpg/mask_rcnn_r50_fpn_crop640_50e_coco/mask_rcnn_r50_fpn_crop640_50e_coco_20220311_011855.log.json)          |

|  RetinaNet   |    FPG     |   50e   |   20.8   |       -        |  40.5  |    -    |     [config](./retinanet_r50_fpg_crop640_50e_coco.py)      |          [model](https://download.openmmlab.com/mmdetection/v2.0/fpg/retinanet_r50_fpg_crop640_50e_coco/retinanet_r50_fpg_crop640_50e_coco_20220311_110809-b0bcf5f4.pth) \|                   [log](https://download.openmmlab.com/mmdetection/v2.0/fpg/retinanet_r50_fpg_crop640_50e_coco/retinanet_r50_fpg_crop640_50e_coco_20220311_110809.log.json)          |

|  RetinaNet   | FPG-chn128 |   50e   |   19.9   |       -        |  39.9  |    -    |  [config](./retinanet_r50_fpg-chn128_crop640_50e_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/fpg/retinanet_r50_fpg-chn128_crop640_50e_coco/retinanet_r50_fpg-chn128_crop640_50e_coco_20220313_104829-ee99a686.pth) \|     [log](https://download.openmmlab.com/mmdetection/v2.0/fpg/retinanet_r50_fpg-chn128_crop640_50e_coco/retinanet_r50_fpg-chn128_crop640_50e_coco_20220313_104829.log.json)   |

**Note**: Chn128 means to decrease the number of channels of features and convs from 256 (default) to 128 in
Neck and BBox Head, which can greatly decrease memory consumption without sacrificing much precision.

```latex
@article{chen2020feature,
  title={Feature pyramid grids},
  author={Chen, Kai and Cao, Yuhang and Loy, Chen Change and Lin, Dahua and Feichtenhofer, Christoph},
  journal={arXiv preprint arXiv:2004.03580},
}
```

}

```

<details>

```bibtex
}
```

<details>
<summary align="right"><a href="https://github.com/Fang-Haoshu/Halpe-FullBody/">AlphaPose (TPAMI'2022)</a></summary>

```bibtex
@article{alphapose,
  author = {Fang, Hao-Shu and Li, Jiefeng and Tang, Hongyang and Xu, Chao and Zhu, Haoyi and Xiu, Yuliang and Li, Yong-Lu and Lu, Cewu},

  title = {AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time},
}
```

|                              Config                              | Input Size | PCK@0.1<sup><br>(Body8) | AUC<sup><br>(Body8) | Params(M) | FLOPS(G) |                              Download                               |
| :--------------------------------------------------------------: | :--------: | :---------------------: | :-----------------: | :-------: | :------: | :-----------------------------------------------------------------: |

| [RTMPose-t\*](/configs/body_2d_keypoint/rtmpose/body8/rtmpose-t_8xb1024-700e_body8-halpe26-256x192.py) |  256x192   |          91.89          |        66.35        |   3.51    |   0.37   | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-t_simcc-body7_pt-body7-halpe26_700e-256x192-6020f8a6_20230605.pth) |
| [RTMPose-s\*](/configs/body_2d_keypoint/rtmpose/body8/rtmpose-s_8xb1024-700e_body8-halpe26-256x192.py) |  256x192   |          93.01          |        68.62        |   5.70    |   0.70   | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_simcc-body7_pt-body7-halpe26_700e-256x192-7f134165_20230605.pth) |

| [RTMPose-m\*](/configs/body_2d_keypoint/rtmpose/body8/rtmpose-m_8xb512-700e_body8-halpe26-256x192.py) |  256x192   |          94.75          |        71.91        |   13.93   |   1.95   | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-body7_pt-body7-halpe26_700e-256x192-4d3e73dd_20230605.pth) |
| [RTMPose-l\*](/configs/body_2d_keypoint/rtmpose/body8/rtmpose-l_8xb512-700e_body8-halpe26-256x192.py) |  256x192   |          95.37          |        73.19        |   28.11   |   4.19   | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-body7_pt-body7-halpe26_700e-256x192-2abb7558_20230605.pth) |

| [RTMPose-m\*](/configs/body_2d_keypoint/rtmpose/body8/rtmpose-m_8xb512-700e_body8-halpe26-384x288.py) |  384x288   |          95.15          |        73.56        |   14.06   |   4.37   | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-body7_pt-body7-halpe26_700e-384x288-89e6428b_20230605.pth) |
| [RTMPose-l\*](/configs/body_2d_keypoint/rtmpose/body8/rtmpose-l_8xb512-700e_body8-halpe26-384x288.py) |  384x288   |          95.56          |        74.38        |   28.24   |   9.40   | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-body7_pt-body7-halpe26_700e-384x288-734182ce_20230605.pth) |

| [RTMPose-x\*](/configs/body_2d_keypoint/rtmpose/body8/rtmpose-x_8xb256-700e_body8-halpe26-384x288.py) |  384x288   |          95.74          |        74.82        |   50.00   |  17.29   | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-x_simcc-body7_pt-body7-halpe26_700e-384x288-7fb6e239_20230606.pth) |

# 模型库

## 通用设置

- 使用分布式训练；
- 为了和其他代码库做公平对比，本文展示的是使用 `torch.cuda.max_memory_allocated()` 在 8 个 GPUs 上得到的最大 GPU 显存占用值，需要注意的是，这些显存占用值通常小于 `nvidia-smi` 显示出来的显存占用值；
- 在模型库中所展示的推理时间是包括网络前向传播和后处理所需的总时间，不包括数据加载所需的时间，模型库中所展示的结果均由 [benchmark.py](https://github.com/open-mmlab/mmdetection/blob/master/tools/analysis_tools/benchmark.py) 脚本文件在 2000 张图像上所计算的平均时间。

## 基准结果

### SECOND

请参考 [SECOND](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/second) 获取更多的细节，我们在 KITTI 和 Waymo 数据集上都给出了相应的基准结果。

### PointPillars

请参考 [PointPillars](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/pointpillars) 获取更多细节，我们在 KITTI 、nuScenes 、Lyft 、Waymo 数据集上给出了相应的基准结果。

### Part-A2

请参考 [Part-A2](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/parta2) 获取更多细节。

### VoteNet

请参考 [VoteNet](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/votenet) 获取更多细节，我们在 ScanNet 和 SUNRGBD 数据集上给出了相应的基准结果。

### Dynamic Voxelization

请参考 [Dynamic Voxelization](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/dynamic_voxelization) 获取更多细节。

### MVXNet

请参考 [MVXNet](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/mvxnet) 获取更多细节。

### RegNetX

请参考 [RegNet](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/regnet) 获取更多细节，我们将 pointpillars 的主干网络替换成 RegNetX，并在 nuScenes 和 Lyft 数据集上给出了相应的基准结果。

### nuImages

我们在 [nuImages 数据集](https://www.nuscenes.org/nuimages) 上也提供基准模型，请参考 [nuImages](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/nuimages) 获取更多细节，我们在该数据集上提供 Mask R-CNN ， Cascade Mask R-CNN 和 HTC 的结果。

### H3DNet

请参考 [H3DNet](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/h3dnet) 获取更多细节。

### 3DSSD

请参考 [3DSSD](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/3dssd) 获取更多细节。

### 3DSSD

请参考 [CenterPoint](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/centerpoint) 获取更多细节。

### SSN

请参考 [SSN](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/ssn) 获取更多细节，我们将 pointpillars 中的检测头替换成 SSN 模型中所使用的 ‘shape-aware grouping heads’，并在 nuScenes 和 Lyft 数据集上给出了相应的基准结果。

### ImVoteNet

请参考 [ImVoteNet](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/imvotenet) 获取更多细节，我们在 SUNRGBD 数据集上给出了相应的结果。

### FCOS3D

请参考 [FCOS3D](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/fcos3d) 获取更多细节，我们在 nuScenes 数据集上给出了相应的结果。

### PointNet++

请参考 [PointNet++](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/pointnet2) 获取更多细节，我们在 ScanNet 和 S3DIS 数据集上给出了相应的结果。

### Group-Free-3D

请参考 [Group-Free-3D](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/groupfree3d) 获取更多细节，我们在 ScanNet 数据集上给出了相应的结果。

### ImVoxelNet

请参考 [ImVoxelNet](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/imvoxelnet) 获取更多细节，我们在 KITTI 数据集上给出了相应的结果。

### PAConv

请参考 [PAConv](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/paconv) 获取更多细节，我们在 S3DIS 数据集上给出了相应的结果。

### DGCNN

请参考 [DGCNN](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/dgcnn) 获取更多细节，我们在 S3DIS 数据集上给出了相应的结果。

### SMOKE

请参考 [SMOKE](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/smoke) 获取更多细节，我们在 KITTI 数据集上给出了相应的结果。

### PGD

请参考 [PGD](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/pgd) 获取更多细节，我们在 KITTI 和 nuScenes 数据集上给出了相应的结果。

### PointRCNN

### PGD

请参考 [PointRCNN](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/point_rcnn) 获取更多细节，我们在 KITTI 数据集上给出了相应的结果。

### MonoFlex

请参考 [MonoFlex](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/monoflex) 获取更多细节，我们在 KITTI 数据集上给出了相应的结果。

### SA-SSD

请参考 [SA-SSD](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/sassd) 获取更多的细节，我们在 KITTI 数据集上给出了相应的基准结果。

### FCAF3D

请参考 [FCAF3D](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/fcaf3d) 获取更多的细节，我们在 ScanNet, S3DIS 和 SUN RGB-D 数据集上给出了相应的基准结果。

### PV-RCNN

请参考 [PV-RCNN](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/pv_rcnn) 获取更多的细节，我们在 KITTI 数据集上给出了相应的基准结果。

### BEVFusion

请参考 [BEVFusion](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/projects/BEVFusion) 获取更多的细节, 我们在 NuScenes 数据集上给出了相应的基准结果。

### CenterFormer

请参考 [CenterFormer](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/projects/CenterFormer) 获取更多的细节, 我们在 Waymo 数据集上给出了相应的基准结果。

### TR3D

请参考 [TR3D](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/projects/TR3D) 获取更多的细节, 我们在 ScanNet, SUN RGB-D 和 S3DIS 数据集上给出了相应的基准结果。

### DETR3D

请参考 [DETR3D](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/projects/DETR3D) 获取更多的细节, 我们在 NuScenes 数据集上给出了相应的基准结果。

### PETR

请参考 [PETR](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/projects/PETR) 获取更多的细节, 我们在 NuScenes 数据集上给出了相应的基准结果。

### TPVFormer

请参考 [TPVFormer](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/projects/TPVFormer) 获取更多的细节, 我们在 NuScenes 数据集上给出了相应的基准结果。

细节请参考 [Mixed Precision (FP16) Training 在 PointPillars 训练的样例](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/pointpillars/pointpillars_hv_fpn_sbn-all_8xb2-amp-2x_nus-3d.py)。

# SimCC: a Simple Coordinate Classification Perspective for Human Pose Estimation

<details>
<summary align="right"><a href="https://arxiv.org/abs/2107.03332">SimCC (ECCV'2022)</a></summary>

```bibtex
@misc{https://doi.org/10.48550/arxiv.2107.03332,
  title={SimCC: a Simple Coordinate Classification Perspective for Human Pose Estimation},
  author={Li, Yanjie and Yang, Sen and Liu, Peidong and Zhang, Shoukui and Wang, Yunxiao and Wang, Zhicheng and Yang, Wankou and Xia, Shu-Tao},
}
```

The 2D heatmap-based approaches have dominated Human Pose Estimation (HPE) for years due to high performance. However, the long-standing quantization error problem in the 2D heatmap-based methods leads to several well-known drawbacks: 1) The performance for the low-resolution inputs is limited; 2) To improve the feature map resolution for higher localization precision, multiple costly upsampling layers are required; 3) Extra post-processing is adopted to reduce the quantization error. To address these issues, we aim to explore a brand new scheme, called \\textit{SimCC}, which reformulates HPE as two classification tasks for horizontal and vertical coordinates. The proposed SimCC uniformly divides each pixel into several bins, thus achieving \\emph{sub-pixel} localization precision and low quantization error. Benefiting from that, SimCC can omit additional refinement post-processing and exclude upsampling layers under certain settings, resulting in a more simple and effective pipeline for HPE. Extensive experiments conducted over COCO, CrowdPose, and MPII datasets show that SimCC outperforms heatmap-based counterparts, especially in low-resolution settings by a large margin.

<img src="https://user-images.githubusercontent.com/13503330/189811385-6395d118-055b-4bad-89e8-f84ffa2c2aa6.png">
</div>

# GGAN (ArXiv'2017)

> [Geometric GAN](https://arxiv.org/abs/1705.02894)

Generative Adversarial Nets (GANs) represent an important milestone for effective generative models, which has inspired numerous variants seemingly different from each other. One of the main contributions of this paper is to reveal a unified geometric structure in GAN and its variants. Specifically, we show that the adversarial generative model training can be decomposed into three geometric steps: separating hyperplane search, discriminator parameter update away from the separating hyperplane, and the generator update along the normal vector direction of the separating hyperplane. This geometric intuition reveals the limitations of the existing approaches and leads us to propose a new formulation called geometric GAN using SVM separating hyperplane that maximizes the margin. Our theoretical analysis shows that the geometric GAN converges to a Nash equilibrium between the discriminator and generator. In addition, extensive numerical results show that the superior performance of geometric GAN.

<img src="https://user-images.githubusercontent.com/28132635/143051600-6a3e5c37-259e-4b77-a847-c6ad1eafa65f.JPG"/>
</div>

  <b> GGAN 64x64, CelebA-Cropped</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/22982797/116691577-9067d800-a9ed-11eb-8ea4-be79884d8502.PNG" width="800"/>
</div>

|                                 Model                                 |    Dataset     |               SWD               | MS-SSIM |   FID   |                                 Download                                 |
| :-------------------------------------------------------------------: | :------------: | :-----------------------------: | :-----: | :-----: | :----------------------------------------------------------------------: |

| [GGAN 64x64](./ggan_dcgan-archi_lr1e-3-1xb128-12Mimgs_celeba-cropped-64x64.py) | CelebA-Cropped |    11.18, 12.21, 39.16/20.85    | 0.3318  | 20.1797 | [model](https://download.openmmlab.com/mmediting/ggan/ggan_celeba-cropped_dcgan-archi_lr-1e-3_64_b128x1_12m.pth)  \| [log](https://download.openmmlab.com/mmediting/ggan/ggan_celeba-cropped_dcgan-archi_lr-1e-3_64_b128x1_12m_20210430_113839.log.json) |
| [GGAN 128x128](./ggan_dcgan-archi_lr1e-4-1xb64-10Mimgs_celeba-cropped-128x128.py) | CelebA-Cropped | 9.81, 11.29, 19.22, 47.79/22.03 | 0.3149  | 18.7647 | [model](https://download.openmmlab.com/mmediting/ggan/ggan_celeba-cropped_dcgan-archi_lr-1e-4_128_b64x1_10m_20210430_143027-516423dc.pth) \| [log](https://download.openmmlab.com/mmediting/ggan/ggan_celeba-cropped_dcgan-archi_lr-1e-4_128_b64x1_10m_20210423_154258.log.json) |

| [GGAN 64x64](./ggan_lsgan-archi_lr1e-4-1xb128-20Mimgs_lsun-bedroom-64x64.py) |  LSUN-Bedroom  |      9.1, 6.2, 12.27/9.19       | 0.0649  | 39.9261 | [model](https://download.openmmlab.com/mmediting/ggan/ggan_lsun-bedroom_lsgan_archi_lr-1e-4_64_b128x1_20m_20210430_143114-5d99b76c.pth) \| [log](https://download.openmmlab.com/mmediting/ggan/ggan_lsun-bedroom_lsgan_archi_lr-1e-4_64_b128x1_20m_20210428_202027.log.json) |

Note: In the original implementation of [GGAN](https://github.com/lim0606/pytorch-geometric-gan), they set `G_iters` to 10. However our framework does not support `G_iters` currently, so we dropped the settings in the original implementation and conducted several experiments with our own settings. We have shown above the experiment results with the lowest `fid` score. \
Original settings and our settings:

|       Model        |    Dataset     | Architecture | optimizer |  lr_G  |  lr_D  | G_iters | D_iters |

| :----------------: | :------------: | :----------: | :-------: | :----: | :----: | :-----: | :-----: |
| GGAN(origin) 64x64 | CelebA-Cropped | dcgan-archi  |  RMSprop  | 0.0002 | 0.0002 |   10    |    1    |
|  GGAN(ours) 64x64  | CelebA-Cropped | dcgan-archi  |   Adam    | 0.001  | 0.001  |    1    |    1    |
| GGAN(origin) 64x64 |  LSUN-Bedroom  | dcgan-archi  |  RMSprop  | 0.0002 | 0.0002 |   10    |    1    |
|  GGAN(ours) 64x64  |  LSUN-Bedroom  | lsgan-archi  |   Adam    | 0.0001 | 0.0001 |    1    |    1    |

```latex
@article{lim2017geometric,
  title={Geometric gan},
  author={Lim, Jae Hyun and Ye, Jong Chul},
  journal={arXiv preprint arXiv:1705.02894},
  url={https://arxiv.org/abs/1705.02894},
}
```

# PVT

> [Pyramid vision transformer: A versatile backbone for dense prediction without convolutions](https://arxiv.org/abs/2102.12122)

Although using convolutional neural networks (CNNs) as backbones achieves great successes in computer vision, this work investigates a simple backbone network useful for many dense prediction tasks without convolutions. Unlike the recently-proposed Transformer model (e.g., ViT) that is specially designed for image classification, we propose Pyramid Vision Transformer~(PVT), which overcomes the difficulties of porting Transformer to various dense prediction tasks. PVT has several merits compared to prior arts. (1) Different from ViT that typically has low-resolution outputs and high computational and memory cost, PVT can be not only trained on dense partitions of the image to achieve high output resolution, which is important for dense predictions but also using a progressive shrinking pyramid to reduce computations of large feature maps. (2) PVT inherits the advantages from both CNN and Transformer, making it a unified backbone in various vision tasks without convolutions by simply replacing CNN backbones. (3) We validate PVT by conducting extensive experiments, showing that it boosts the performance of many downstream tasks, e.g., object detection, semantic, and instance segmentation. For example, with a comparable number of parameters, RetinaNet+PVT achieves 40.4 AP on the COCO dataset, surpassing RetinNet+ResNet50 (36.3 AP) by 4.1 absolute AP. We hope PVT could serve as an alternative and useful backbone for pixel-level predictions and facilitate future researches.

Transformer recently has shown encouraging progresses in computer vision. In this work, we present new baselines by improving the original Pyramid Vision Transformer (abbreviated as PVTv1) by adding three designs, including (1) overlapping patch embedding, (2) convolutional feed-forward networks, and (3) linear complexity attention layers.
With these modifications, our PVTv2 significantly improves PVTv1 on three tasks e.g., classification, detection, and segmentation. Moreover, PVTv2 achieves comparable or better performances than recent works such as Swin Transformer. We hope this work will facilitate state-of-the-art Transformer researches in computer vision.

<img src="https://user-images.githubusercontent.com/40661020/143969989-6f94e695-23b1-4f8f-b406-d589fdc3cfb2.png"/>
</div>

### RetinaNet (PVTv1)

</div>

|  Backbone  | Lr schd | Mem (GB) | box AP |                   Config                   |                                                                                                                                             Download                                                                                                                                             |
| :--------: | :-----: | :------: | :----: | :----------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

|  PVT-Tiny  |   12e   |   8.5    |  36.6  | [config](./retinanet_pvt-t_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/pvt/retinanet_pvt-t_fpn_1x_coco/retinanet_pvt-t_fpn_1x_coco_20210831_103110-17b566bd.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pvt/retinanet_pvt-t_fpn_1x_coco/retinanet_pvt-t_fpn_1x_coco_20210831_103110.log.json) |
| PVT-Small  |   12e   |   14.5   |  40.4  | [config](./retinanet_pvt-s_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/pvt/retinanet_pvt-s_fpn_1x_coco/retinanet_pvt-s_fpn_1x_coco_20210906_142921-b6c94a5b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pvt/retinanet_pvt-s_fpn_1x_coco/retinanet_pvt-s_fpn_1x_coco_20210906_142921.log.json) |

| PVT-Medium |   12e   |   20.9   |  41.7  | [config](./retinanet_pvt-m_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/pvt/retinanet_pvt-m_fpn_1x_coco/retinanet_pvt-m_fpn_1x_coco_20210831_103243-55effa1b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pvt/retinanet_pvt-m_fpn_1x_coco/retinanet_pvt-m_fpn_1x_coco_20210831_103243.log.json) |

### RetinaNet (PVTv2)

| Backbone | Lr schd | Mem (GB) | box AP |                    Config                     |                                                                                                                                                   Download                                                                                                                                                   |

| :------: | :-----: | :------: | :----: | :-------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| PVTv2-B0 |   12e   |   7.4    |  37.1  | [config](./retinanet_pvtv2-b0_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/pvt/retinanet_pvtv2-b0_fpn_1x_coco/retinanet_pvtv2-b0_fpn_1x_coco_20210831_103157-13e9aabe.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pvt/retinanet_pvtv2-b0_fpn_1x_coco/retinanet_pvtv2-b0_fpn_1x_coco_20210831_103157.log.json) |

| PVTv2-B1 |   12e   |   9.5    |  41.2  | [config](./retinanet_pvtv2-b1_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/pvt/retinanet_pvtv2-b1_fpn_1x_coco/retinanet_pvtv2-b1_fpn_1x_coco_20210831_103318-7e169a7d.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pvt/retinanet_pvtv2-b1_fpn_1x_coco/retinanet_pvtv2-b1_fpn_1x_coco_20210831_103318.log.json) |
| PVTv2-B2 |   12e   |   16.2   |  44.6  | [config](./retinanet_pvtv2-b2_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/pvt/retinanet_pvtv2-b2_fpn_1x_coco/retinanet_pvtv2-b2_fpn_1x_coco_20210901_174843-529f0b9a.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pvt/retinanet_pvtv2-b2_fpn_1x_coco/retinanet_pvtv2-b2_fpn_1x_coco_20210901_174843.log.json) |

| PVTv2-B3 |   12e   |   23.0   |  46.0  | [config](./retinanet_pvtv2-b3_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/pvt/retinanet_pvtv2-b3_fpn_1x_coco/retinanet_pvtv2-b3_fpn_1x_coco_20210903_151512-8357deff.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pvt/retinanet_pvtv2-b3_fpn_1x_coco/retinanet_pvtv2-b3_fpn_1x_coco_20210903_151512.log.json) |
| PVTv2-B4 |   12e   |   17.0   |  46.3  | [config](./retinanet_pvtv2-b4_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/pvt/retinanet_pvtv2-b4_fpn_1x_coco/retinanet_pvtv2-b4_fpn_1x_coco_20210901_170151-83795c86.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pvt/retinanet_pvtv2-b4_fpn_1x_coco/retinanet_pvtv2-b4_fpn_1x_coco_20210901_170151.log.json) |

| PVTv2-B5 |   12e   |   18.7   |  46.1  | [config](./retinanet_pvtv2-b5_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/pvt/retinanet_pvtv2-b5_fpn_1x_coco/retinanet_pvtv2-b5_fpn_1x_coco_20210902_201800-3420eb57.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pvt/retinanet_pvtv2-b5_fpn_1x_coco/retinanet_pvtv2-b5_fpn_1x_coco_20210902_201800.log.json) |

```latex
@article{wang2021pyramid,
  title={Pyramid vision transformer: A versatile backbone for dense prediction without convolutions},
  author={Wang, Wenhai and Xie, Enze and Li, Xiang and Fan, Deng-Ping and Song, Kaitao and Liang, Ding and Lu, Tong and Luo, Ping and Shao, Ling},
  journal={arXiv preprint arXiv:2102.12122},
}
```

```latex
@article{wang2021pvtv2,
  title={PVTv2: Improved Baselines with Pyramid Vision Transformer},

journal={arXiv preprint arXiv:2102.12122},
}
```

```latex
  journal={arXiv preprint arXiv:2106.13797},
}
```

# PoolFormer

> [MetaFormer is Actually What You Need for Vision](https://arxiv.org/abs/2111.11418)

Transformers have shown great potential in computer vision tasks. A common belief is their attention-based token mixer module contributes most to their competence. However, recent works show the attention-based module in transformers can be replaced by spatial MLPs and the resulted models still perform quite well. Based on this observation, we hypothesize that the general architecture of the transformers, instead of the specific token mixer module, is more essential to the model's performance. To verify this, we deliberately replace the attention module in transformers with an embarrassingly simple spatial pooling operator to conduct only basic token mixing. Surprisingly, we observe that the derived model, termed as PoolFormer, achieves competitive performance on multiple computer vision tasks. For example, on ImageNet-1K, PoolFormer achieves 82.1% top-1 accuracy, surpassing well-tuned vision transformer/MLP-like baselines DeiT-B/ResMLP-B24 by 0.3%/1.1% accuracy with 35%/52% fewer parameters and 49%/61% fewer MACs. The effectiveness of PoolFormer verifies our hypothesis and urges us to initiate the concept of "MetaFormer", a general architecture abstracted from transformers without specifying the token mixer. Based on the extensive experiments, we argue that MetaFormer is the key player in achieving superior results for recent transformer and MLP-like models on vision tasks. This work calls for more future research dedicated to improving MetaFormer instead of focusing on the token mixer modules. Additionally, our proposed PoolFormer could serve as a starting baseline for future MetaFormer architecture design.

<img src="https://user-images.githubusercontent.com/15921929/144710761-1635f59a-abde-4946-984c-a2c3f22a19d2.png" width="100%"/>
</div>

```python

predict = inference_model('poolformer-s12_3rdparty_32xb128_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('poolformer-s12_3rdparty_32xb128_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/poolformer/poolformer-s12_32xb128_in1k.py https://download.openmmlab.com/mmclassification/v0/poolformer/poolformer-s12_3rdparty_32xb128_in1k_20220414-f8d83051.pth
```

| Model                                    |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                  Config                  |                                Download                                 |
| :--------------------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :--------------------------------------: | :---------------------------------------------------------------------: |

| `poolformer-s12_3rdparty_32xb128_in1k`\* | From scratch |   11.92    |   1.87    |   77.24   |   93.51   | [config](poolformer-s12_32xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/poolformer/poolformer-s12_3rdparty_32xb128_in1k_20220414-f8d83051.pth) |
| `poolformer-s24_3rdparty_32xb128_in1k`\* | From scratch |   21.39    |   3.51    |   80.33   |   95.05   | [config](poolformer-s24_32xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/poolformer/poolformer-s24_3rdparty_32xb128_in1k_20220414-d7055904.pth) |
| `poolformer-s36_3rdparty_32xb128_in1k`\* | From scratch |   30.86    |   5.15    |   81.43   |   95.45   | [config](poolformer-s36_32xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/poolformer/poolformer-s36_3rdparty_32xb128_in1k_20220414-d78ff3e8.pth) |

| `poolformer-m36_3rdparty_32xb128_in1k`\* | From scratch |   56.17    |   8.96    |   82.14   |   95.71   | [config](poolformer-m36_32xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/poolformer/poolformer-m36_3rdparty_32xb128_in1k_20220414-c55e0949.pth) |
| `poolformer-m48_3rdparty_32xb128_in1k`\* | From scratch |   73.47    |   11.80   |   82.51   |   95.95   | [config](poolformer-m48_32xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/poolformer/poolformer-m48_3rdparty_32xb128_in1k_20220414-9378f3eb.pth) |

*Models with * are converted from the [official repo](https://github.com/sail-sg/poolformer). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@inproceedings{yu2022metaformer,
  title={Metaformer is actually what you need for vision},

```bibtex
  author={Yu, Weihao and Luo, Mi and Zhou, Pan and Si, Chenyang and Zhou, Yichen and Wang, Xinchao and Feng, Jiashi and Yan, Shuicheng},
  pages={10819--10829},
}
```

# cmake 编译选项说明

<thead>
  <tr>
    <th>选项</th>
    <th>取值范围</th>
    <th>缺省值</th>
    <th>说明</th>
  </tr>
</thead>
<tbody>

<tr>
    <td>MMDEPLOY_SHARED_LIBS</td>
    <td>{ON, OFF}</td>
    <td>ON</td>
    <td>动态库的编译开关。设置OFF时，编译静态库</td>
  </tr>

<tr>
    <td>MMDEPLOY_BUILD_SDK</td>
    <td>OFF</td>
    <td>MMDeploy SDK 编译开关</td>
  </tr>

<tr>
  <td>MMDEPLOY_BUILD_SDK_MONOLITHIC</td>
  <td>{ON, OFF}</td>
  <td>OFF</td>
  <td>编译生成单个 lib 文件</td>
  </tr>

<tr>
  <td>MMDEPLOY_BUILD_TEST</td>
  <td>MMDeploy SDK 的单元测试程序编译开关</td>
  </tr>

<tr>
    <td>MMDEPLOY_BUILD_SDK_PYTHON_API</td>
    <td>SDK python package的编译开关</td>
  </tr>

<tr>
    <td>MMDEPLOY_BUILD_SDK_CSHARP_API</td>
    <td>SDK C# package的编译开关</td>
  </tr>

<tr>
    <td>MMDEPLOY_BUILD_SDK_JAVA_API</td>

<td>{ON, OFF}</td>
  </tr>

<tr>
    <td>SDK Java package的编译开关</td>
  </tr>

<tr>
    <td>MMDEPLOY_BUILD_EXAMPLES</td>
    <td>是否编译 demo</td>
  </tr>

<tr>
    <td>MMDEPLOY_SPDLOG_EXTERNAL</td>
    <td>是否使用系统自带的 spdlog 安装包</td>
  </tr>

<tr>
    <td>MMDEPLOY_ZIP_MODEL</td>
    <td>是否使用 zip 格式的 sdk 目录</td>
  </tr>

<tr>
    <td>MMDEPLOY_COVERAGE</td>
    <td>额外增加编译选项，以生成代码覆盖率报表</td>
  </tr>

<tr>
    <td>MMDEPLOY_TARGET_DEVICES</td>
    <td>{"cpu", "cuda"}</td>
    <td>cpu</td>
    <td>设置目标设备。当有多个设备时，设备名称之间使用分号隔开。 比如，-DMMDEPLOY_TARGET_DEVICES="cpu;cuda"</td>
  </tr>

<tr>
    <td>MMDEPLOY_TARGET_BACKENDS</td>

<td>{"cpu", "cuda"}</td>
  </tr>

<tr>
    <td>{"trt", "ort", "pplnn", "ncnn", "openvino", "torchscript", "snpe", "coreml", "tvm"}</td>
    <td>N/A</td>
    <td> <b>默认情况下，SDK不设置任何后端</b>, 因为它与应用场景高度相关。 当选择多个后端时， 中间使用分号隔开。比如，<pre><code>-DMMDEPLOY_TARGET_BACKENDS="trt;ort;pplnn;ncnn;openvino"</code></pre>
    构建时，几乎每个后端，都需设置一些路径变量，用来查找依赖包。<br>
    1. <b>trt</b>: 表示 TensorRT。需要设置 <code>TENSORRT_DIR</code> 和 <code>CUDNN_DIR</code>。
-DTENSORRT_DIR=$env:TENSORRT_DIR
-DCUDNN_DIR=$env:CUDNN_DIR
    2. <b>ort</b>: 表示 ONNXRuntime。需要设置 <code>ONNXRUNTIME_DIR</code>。
<pre><code>-DONNXRUNTIME_DIR=$env:ONNXRUNTIME_DIR</code></pre>
    3. <b>pplnn</b>: 表示 PPL.NN。需要设置 <code>pplnn_DIR</code>。<br>
    4. <b>ncnn</b>：表示 ncnn。需要设置 <code>ncnn_DIR</code>。 <br>
    5. <b>openvino</b>: 表示 OpenVINO。需要设置 <code>InferenceEngine_DIR</code>。<br>

4. <b>ncnn</b>：表示 ncnn。需要设置 <code>ncnn_DIR</code>。 <br>
    6. <b>torchscript</b>: 表示 TorchScript。需要设置<code>Torch_DIR</code>。<br>
    7. <b>snpe</b>: 表示 qcom snpe。需要环境变量设置 SNPE_ROOT。<br>
    8. <b>coreml</b>: 表示 Core ML。目前在进行模型转换时需要设置 <code>Torch_DIR</code>。 <br>
    9. <b>tvm</b>: 表示 TVM。需要设置 <code>TVM_DIR</code>。<br>
   </td>
  </tr>

<tr>
    <td>MMDEPLOY_CODEBASES</td>
    <td>{"mmpretrain", "mmdet", "mmseg", "mmagic", "mmocr", "all"}</td>
    <td>all</td>
    <td>用来设置SDK后处理组件，加载 OpenMMLab 算法仓库的后处理功能。如果选择多个 codebase，中间使用分号隔开。比如，<code>-DMMDEPLOY_CODEBASES="mmcls;mmdet"</code>。也可以通过 <code>-DMMDEPLOY_CODEBASES=all</code> 方式，加载所有 codebase。</td>
  </tr>

</tbody>
</table>

# Model Analysis

## 统计模型参数量与计算量

MMPose 提供了 `tools/analysis_tools/get_flops.py` 来统计模型的参数量与计算量。

```shell
python tools/analysis_tools/get_flops.py ${CONFIG_FILE} [--shape ${INPUT_SHAPE}] [--cfg-options ${CFG_OPTIONS}]
```

参数说明：

`CONFIG_FILE` : 模型配置文件的路径。

`--shape`: 模型的输入张量形状。

`--input-constructor`: 如果指定为 `batch`，将会生成一个 `batch tensor` 来计算 FLOPs。

`--batch-size`：如果 `--input-constructor` 指定为 `batch`，将会生成一个随机 `tensor`，形状为 `(batch_size, 3, **input_shape)` 来计算 FLOPs。

`--cfg-options`: 如果指定，可选的 `cfg` 的键值对将会被合并到配置文件中。

示例：

```shell
python tools/analysis_tools/get_flops.py configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192.py
```

结果如下：

```text
Input shape: (1, 3, 256, 192)
Flops: 7.7 GFLOPs
Params: 28.54 M
```

```{note}
目前该工具仍处于实验阶段，我们不能保证统计结果绝对正确，一些算子（比如 GN 或自定义算子）没有被统计到 FLOPs 中。
```

## 分析训练日志

MMPose 提供了 `tools/analysis_tools/analyze_logs.py` 来对训练日志进行简单的分析，包括：

- 将日志绘制成损失和精度曲线图
- 统计训练速度

```

```{note}
```

## 分析训练日志

- 统计训练速度

### 绘制损失和精度曲线图

该功能依赖于 `seaborn`，请先运行 `pip install seaborn` 安装依赖包。

![log_curve](https://user-images.githubusercontent.com/87690686/188538215-5d985aaa-59f8-44cf-b6f9-10890d599e9c.png)

```shell
python tools/analysis_tools/analyze_logs.py plot_curve ${JSON_LOGS} [--keys ${KEYS}] [--title ${TITLE}] [--legend ${LEGEND}] [--backend ${BACKEND}] [--style ${STYLE}] [--out ${OUT_FILE}]
```

示例：

- 绘制损失曲线

  ```shell
  python tools/analysis_tools/analyze_logs.py plot_curve log.json --keys loss_kpt --legend loss_kpt
  ```

- 绘制精度曲线并导出为 PDF 文件

  ```shell
  python tools/analysis_tools/analyze_logs.py plot_curve log.json --keys acc_pose --out results.pdf
  ```

- 将多个日志文件绘制在同一张图上

  ```shell

```

  ```shell
  ```

  ```shell
  python tools/analysis_tools/analyze_logs.py plot_curve log1.json log2.json --keys loss_kpt --legend run1 run2 --title loss_kpt --out loss_kpt.png
  ```

### 统计训练速度

```shell
python tools/analysis_tools/analyze_logs.py cal_train_time ${JSON_LOGS} [--include-outliers]
```

示例：

```shell
python tools/analysis_tools/analyze_logs.py cal_train_time log.json
```

结果如下：

```text
-----Analyze train time of hrnet_w32_256x192.json-----
slowest epoch 56, average time is 0.6924
fastest epoch 1, average time is 0.6502
time std over epochs is 0.0085
average iter time: 0.6688 s/iter
```

# 数据集准备

## 前言

经过数十年的发展，OCR 领域涌现出了一系列的相关数据集，这些数据集往往采用风格各异的格式来提供文本的标注文件，使得用户在使用这些数据集时不得不进行格式转换。因此，为了方便用户进行数据集准备，我们提供了[一键式的数据准备脚本](./data_prepare/dataset_preparer.md)，使得用户仅需使用一行命令即可完成数据集准备的全部步骤。

在这一节，我们将介绍一个典型的数据集准备流程：

1. [下载数据集并将其格式转换为 MMOCR 支持的格式](#数据集下载及格式转换)
2. [修改配置文件](#修改配置文件)

然而，如果你已经有了 MMOCR 支持的格式的数据集，那么第一步就不是必须的。你可以阅读[数据集类及标注格式](../basic_concepts/datasets.md#数据集类及标注格式)来了解更多细节。

## 数据集下载及格式转换

以 ICDAR 2015 数据集的文本检测任务准备步骤为例，你可以执行以下命令来完成数据集准备：

```shell
python tools/dataset_converters/prepare_dataset.py icdar2015 --task textdet
```

命令执行完成后，数据集将被下载并转换至 MMOCR 格式，文件目录结构如下：

```text
data/icdar2015
├── textdet_imgs
│   ├── test
│   └── train
├── textdet_test.json
└── textdet_train.json
```

数据准备完毕以后，你也可以通过使用我们提供的数据集浏览工具 [browse_dataset.py](./useful_tools.md#数据集可视化工具) 来可视化数据集的标签是否被正确生成，例如：

```bash
python tools/analysis_tools/browse_dataset.py configs/textdet/_base_/datasets/icdar2015.py
```

## 修改配置文件

### 单数据集训练

```bash
```

## 修改配置文件

### 单数据集训练

在使用新的数据集时，我们需要对其图像、标注文件的路径等基础信息进行配置。`configs/xxx/_base_/datasets/` 路径下已预先配置了 MMOCR 中常用的数据集（当你使用 `prepare_dataset.py` 来准备数据集时，这个配置文件通常会在数据集准备就绪后自动生成），这里我们以 ICDAR 2015 数据集为例（见 `configs/textdet/_base_/datasets/icdar2015.py`）：

```Python
icdar2015_textdet_data_root = 'data/icdar2015' # 数据集根目录

# 训练集配置
    data_root=icdar2015_textdet_data_root,               # 数据根目录
    ann_file='textdet_train.json',                       # 标注文件名称
    filter_cfg=dict(filter_empty_gt=True, min_size=32),  # 数据过滤
# 测试集配置
icdar2015_textdet_test = dict(
    data_root=icdar2015_textdet_data_root,
```

在配置好数据集后，我们还需要在相应的算法模型配置文件中导入想要使用的数据集。例如，在 ICDAR 2015 数据集上训练 "DBNet_R18" 模型：

```Python
_base_ = [

ann_file='textdet_test.json',
```

```Python
_base_ = [
    '_base_dbnet_r18_fpnc.py',
    '../_base_/datasets/icdar2015.py',  # 导入数据集配置文件
]

icdar2015_textdet_train = _base_.icdar2015_textdet_train            # 指定训练集
icdar2015_textdet_train.pipeline = _base_.train_pipeline   # 指定训练集使用的数据流水线
icdar2015_textdet_test = _base_.icdar2015_textdet_test              # 指定测试集
icdar2015_textdet_test.pipeline = _base_.test_pipeline     # 指定测试集使用的数据流水线

    dataset=icdar2015_textdet_train)    # 在 train_dataloader 中指定使用的训练数据集

dataset=icdar2015_textdet_train)    # 在 train_dataloader 中指定使用的训练数据集

    dataset=icdar2015_textdet_test)    # 在 val_dataloader 中指定使用的验证数据集

```

### 多数据集训练

此外，基于 [`ConcatDataset`](mmocr.datasets.ConcatDataset)，用户还可以使用多个数据集组合来训练或测试模型。用户只需在配置文件中将 dataloader 中的 dataset 类型设置为 `ConcatDataset`，并指定对应的数据集列表即可。

```Python
train_list = [ic11, ic13, ic15]
        type='ConcatDataset', datasets=train_list, pipeline=train_pipeline))
```

例如，以下配置使用了 MJSynth 数据集进行训练，并使用 6 个学术数据集（CUTE80, IIIT5K, SVT, SVTP, ICDAR2013, ICDAR2015）进行测试。

```Python
_base_ = [ # 导入所有需要使用的数据集配置
    '../_base_/datasets/mjsynth.py',
    '../_base_/datasets/cute80.py',
    '../_base_/datasets/iiit5k.py',
    '../_base_/datasets/svt.py',
    '../_base_/datasets/svtp.py',

'../_base_/datasets/mjsynth.py',
    '../_base_/datasets/icdar2013.py',
    '../_base_/schedules/schedule_adadelta_5e.py',
    '_base_crnn_mini-vgg.py',
]

# 训练集列表
train_list = [_base_.mjsynth_textrecog_train]
# 测试集列表
test_list = [
    _base_.cute80_textrecog_test, _base_.iiit5k_textrecog_test, _base_.svt_textrecog_test,
    _base_.svtp_textrecog_test, _base_.icdar2013_textrecog_test, _base_.icdar2015_textrecog_test
]

# 使用 ConcatDataset 来级联列表中的多个数据集
train_dataset = dict(
       type='ConcatDataset', datasets=train_list, pipeline=_base_.train_pipeline)
test_dataset = dict(
       type='ConcatDataset', datasets=test_list, pipeline=_base_.test_pipeline)

    batch_size=192 * 4,
    num_workers=32,

    dataset=train_dataset)

    dataset=test_dataset)

val_dataloader = test_dataloader
```

# Overview

## What is MMAction2

MMAction2 is an open source toolkit based on PyTorch, supporting numerous video understanding models, including **action recognition, skeleton-based action recognition, spatio-temporal action detection and temporal action localization**. Moreover, it supports widely-used academic datasets and offers many useful tools, assisting users in exploring various aspects of models and datasets, as well as implementing high-quality algorithms. Generally, the toolkit boasts the following features:

**One-stop, Multi-model**: MMAction2 supports various video understanding tasks and implements state-of-the-art models for action recognition, localization, detection.

**Modular Design**: The modular design of MMAction2 enables users to define and reuse modules in the model as required.

**Various Useful Tools**: MMAction2 provides an array of analysis tools, such as visualizers, validation scripts, evaluators, etc., to aid users in troubleshooting, fine-tuning, or comparing models.

**Powered by OpenMMLab**: Similar to other algorithm libraries in the OpenMMLab family, MMAction2 adheres to OpenMMLab's rigorous development guidelines and interface conventions, considerably reducing the learning cost for users familiar with other OpenMMLab projects. Furthermore, due to the unified interfaces among OpenMMLab projects, it is easy to call models implemented in other OpenMMLab projects (such as MMClassification) in MMAction2, which greatly facilitates cross-domain research and real-world applications.

<table><tr>
  <td><img src="https://github.com/open-mmlab/mmaction2/raw/main/resources/mmaction2_overview.gif" width="380px">

    <p style="text-align: center;">Action Recognition</p></td>
  <td><img src="https://user-images.githubusercontent.com/34324155/123989146-2ecae680-d9fb-11eb-916b-b9db5563a9e5.gif" width="380px"><br>
    <p style="text-align: center;">Skeleton-based Action Recognition</p></td>
</table></tr>
  <td><img src="https://user-images.githubusercontent.com/30782254/155710881-bb26863e-fcb4-458e-b0c4-33cd79f96901.gif" width="380px">
    <p style="text-align: center;">Spatio-Temporal Action Detection</p></td>
  <td><img src="https://github.com/open-mmlab/mmaction2/raw/main/resources/spatio-temporal-det.gif" width="380px"><br>

## How to use the documentation

We have prepared a wealth of documents to meet your various needs:

<summary><b>For the basic usage of MMAction2</b></summary>

- [Installation](installation.md)
- [Quick Run](quick_run.md)
- [Inference with existing models](../user_guides/inference.md)

<summary><b>For training on supported dataset</b></summary>

- [Learn about Configs](../user_guides/config.md)
- [Prepare Dataset](../user_guides/prepare_dataset.md)
- [Training and Test](../user_guides/train_test.md)

<summary><b>For looking for some common issues</b></summary>

- [FAQ](faq.md)
- [Useful tools](../useful_tools.md)

<summary><b>For a general understanding about MMAction2</b></summary>

- [A 20-Minute Guide to MMAction2 FrameWork](guide_to_framework.md)
- [Dataflow in MMAction2](../advanced_guides/dataflow.md)

<summary><b>For advanced usage about custom training</b></summary>

- [Customize Model](../advanced_guides/customize_models.md)
- [Customize Dataset](../advanced_guides/customize_dataset.md)
- [Customize Data Pipeline](../advanced_guides/customize_pipeline.md)
- [Customize Optimizer](../advanced_guides/customize_optimizer.md)
- [Customize Logging](../advanced_guides/customize_logging.md)

<summary><b>For supported model zoo and dataset zoo</b></summary>

- [Model Zoo](../modelzoo_statistics.md)
- [Dataset Zoo](../datasetzoo_statistics.md)

<summary><b>For migration from MMAction2 0.x</b></summary>

- [Migration](../migration.md)

<summary><b>For researchers and developers who are willing to contribute to MMAction2</b></summary>

- [How to contribute to MMAction2](contribution_guide.md)

# Usage

**step 0.** install the local nuget package

You should build csharp api first, it will generate a nuget package, or you can download our prebuit package. You may refer to [this](https://stackoverflow.com/a/55167481) on how to install local nuget package.

**step 1.** Add runtime dll to the system path

If you built csharp api from source and didn't build static lib, you should add the built dll to your system path. The same is to opencv, etc.

And don't forget to install backend dependencies. Take tensorrt backend as example, you have to install cudatoolkit, cudnn and tensorrt. The version of backend dependencies that our prebuit nuget package used will be offered in release note.

| backend     | dependencies                  |
| ----------- | ----------------------------- |
| tensorrt    | cudatoolkit, cudnn, tensorrt  |
| onnxruntime | onnxruntime / onnxruntime-gpu |

**step 2.** Open Demo.sln and build solution.

**step 3.** Prepare the model.

You can either convert your model according to this [tutorial](../../docs/en/tutorials/how_to_convert_model.md) or download the test models from [OneDrive](https://1drv.ms/u/s!Aqis6w3rjKXSh2dXZ5OqbZIZSu9P?e=nefSdY) or [BaiduYun](https://pan.baidu.com/s/1VJkLo2oqHos6ZWDT7xamFg?pwd=STAR). The web drive contains onnx and tensorrt models and the test models are converted under environment of cuda11.1 + cudnn8.2.1 + tensorrt 8.2.3.0 + GTX2070s.

*Note*:

- a) If you want to use the tensorrt model from the link, make sure your environment and your gpu architecture is same with above.
- b) When you use the downloaded onnx model, you have to edit `deploy.json`, edit `end2end.engine` to `end2end.onnx` and `tensorrt` to `onnxruntime`.

**step 4.** Set one project as startup project and run it.

## <a href='https://lmdeploy.readthedocs.io/en/latest/'>English</a>

## <a href='https://lmdeploy.readthedocs.io/zh_CN/latest/'>简体中文</a>

# ResNeSt

> [ResNeSt: Split-Attention Networks](https://arxiv.org/abs/2004.08955)

<a href="https://github.com/zhanghang1989/ResNeSt">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/backbones/resnest.py#L271">Code Snippet</a>

It is well known that featuremap attention and multi-path representation are important for visual recognition. In this paper, we present a modularized architecture, which applies the channel-wise attention on different network branches to leverage their success in capturing cross-feature interactions and learning diverse representations. Our design results in a simple and unified computation block, which can be parameterized using only a few variables. Our model, named ResNeSt, outperforms EfficientNet in accuracy and latency trade-off on image classification. In addition, ResNeSt has achieved superior transfer learning results on several public benchmarks serving as the backbone, and has been adopted by the winning entries of COCO-LVIS challenge. The source code for complete system and pretrained models are publicly available.

<img src="https://user-images.githubusercontent.com/24582831/142902526-3cf33345-7e40-47a6-985e-4381857e21df.png" width="60%"/>
</div>

| Method     | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                         | download                                                                                                                                                                                                                                                                                                                                                                               |

| FCN        | S-101-D8 | 512x1024  |   80000 |     11.4 | 2.39           | V100   | 77.56 | 78.98         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/resnest/resnest_s101-d8_fcn_4xb2-80k_cityscapes-512x1024.py)           | [model](https://download.openmmlab.com/mmsegmentation/v0.5/resnest/fcn_s101-d8_512x1024_80k_cityscapes/fcn_s101-d8_512x1024_80k_cityscapes_20200807_140631-f8d155b3.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/resnest/fcn_s101-d8_512x1024_80k_cityscapes/fcn_s101-d8_512x1024_80k_cityscapes-20200807_140631.log.json)                                         |

| PSPNet     | S-101-D8 | 512x1024  |   80000 |     11.8 | 2.52           | V100   | 78.57 | 79.19         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/resnest/resnest_s101-d8_pspnet_4xb2-80k_cityscapes512x1024.py)         | [model](https://download.openmmlab.com/mmsegmentation/v0.5/resnest/pspnet_s101-d8_512x1024_80k_cityscapes/pspnet_s101-d8_512x1024_80k_cityscapes_20200807_140631-c75f3b99.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/resnest/pspnet_s101-d8_512x1024_80k_cityscapes/pspnet_s101-d8_512x1024_80k_cityscapes-20200807_140631.log.json)                             |

| DeepLabV3  | S-101-D8 | 512x1024  |   80000 |     11.9 | 1.88           | V100   | 79.67 | 80.51         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/resnest/resnest_s101-d8_deeplabv3_4xb2-80k_cityscapes-512x1024.py)     | [model](https://download.openmmlab.com/mmsegmentation/v0.5/resnest/deeplabv3_s101-d8_512x1024_80k_cityscapes/deeplabv3_s101-d8_512x1024_80k_cityscapes_20200807_144429-b73c4270.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/resnest/deeplabv3_s101-d8_512x1024_80k_cityscapes/deeplabv3_s101-d8_512x1024_80k_cityscapes-20200807_144429.log.json)                 |

| DeepLabV3+ | S-101-D8 | 512x1024  |   80000 |     13.2 | 2.36           | V100   | 79.62 | 80.27         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/resnest/resnest_s101-d8_deeplabv3plus_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/resnest/deeplabv3plus_s101-d8_512x1024_80k_cityscapes/deeplabv3plus_s101-d8_512x1024_80k_cityscapes_20200807_144429-1239eb43.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/resnest/deeplabv3plus_s101-d8_512x1024_80k_cityscapes/deeplabv3plus_s101-d8_512x1024_80k_cityscapes-20200807_144429.log.json) |

### ADE20K

### ADE20K

| Method     | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                     | download                                                                                                                                                                                                                                                                                                                                                               |

| FCN        | S-101-D8 | 512x512   |  160000 |     14.2 | 12.86          | V100   | 45.62 | 46.16         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/resnest/resnest_s101-d8_fcn_4xb4-160k_ade20k-512x512.py)           | [model](https://download.openmmlab.com/mmsegmentation/v0.5/resnest/fcn_s101-d8_512x512_160k_ade20k/fcn_s101-d8_512x512_160k_ade20k_20200807_145416-d3160329.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/resnest/fcn_s101-d8_512x512_160k_ade20k/fcn_s101-d8_512x512_160k_ade20k-20200807_145416.log.json)                                         |

| PSPNet     | S-101-D8 | 512x512   |  160000 |     14.2 | 13.02          | V100   | 45.44 | 46.28         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/resnest/resnest_s101-d8_pspnet_4xb4-160k_ade20k-512x512.py)        | [model](https://download.openmmlab.com/mmsegmentation/v0.5/resnest/pspnet_s101-d8_512x512_160k_ade20k/pspnet_s101-d8_512x512_160k_ade20k_20200807_145416-a6daa92a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/resnest/pspnet_s101-d8_512x512_160k_ade20k/pspnet_s101-d8_512x512_160k_ade20k-20200807_145416.log.json)                             |

| DeepLabV3  | S-101-D8 | 512x512   |  160000 |     14.6 | 9.28           | V100   | 45.71 | 46.59         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/resnest/resnest_s101-d8_deeplabv3_4xb4-160k_ade20k-512x512.py)     | [model](https://download.openmmlab.com/mmsegmentation/v0.5/resnest/deeplabv3_s101-d8_512x512_160k_ade20k/deeplabv3_s101-d8_512x512_160k_ade20k_20200807_144503-17ecabe5.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/resnest/deeplabv3_s101-d8_512x512_160k_ade20k/deeplabv3_s101-d8_512x512_160k_ade20k-20200807_144503.log.json)                 |

| DeepLabV3+ | S-101-D8 | 512x512   |  160000 |     16.2 | 11.96          | V100   | 46.47 | 47.27         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/resnest/resnest_s101-d8_deeplabv3plus_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/resnest/deeplabv3plus_s101-d8_512x512_160k_ade20k/deeplabv3plus_s101-d8_512x512_160k_ade20k_20200807_144503-27b26226.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/resnest/deeplabv3plus_s101-d8_512x512_160k_ade20k/deeplabv3plus_s101-d8_512x512_160k_ade20k-20200807_144503.log.json) |

```bibtex
@article{zhang2020resnest,
title={ResNeSt: Split-Attention Networks},
author={Zhang, Hang and Wu, Chongruo and Zhang, Zhongyue and Zhu, Yi and Zhang, Zhi and Lin, Haibin and Sun, Yue and He, Tong and Muller, Jonas and Manmatha, R. and Li, Mu and Smola, Alexander},
journal={arXiv preprint arXiv:2004.08955},
}
```

# 3DSSD: Point-based 3D Single Stage Object Detector

> [3DSSD: Point-based 3D Single Stage Object Detector](https://arxiv.org/abs/2002.10187)

Currently, there have been many kinds of voxel-based 3D single stage detectors, while point-based single stage methods are still underexplored. In this paper, we first present a lightweight and effective point-based 3D single stage object detector, named 3DSSD, achieving a good balance between accuracy and efficiency. In this paradigm, all upsampling layers and refinement stage, which are indispensable in all existing point-based methods, are abandoned to reduce the large computation cost. We novelly propose a fusion sampling strategy in downsampling process to make detection on less representative points feasible. A delicate box prediction network including a candidate generation layer, an anchor-free regression head with a 3D center-ness assignment strategy is designed to meet with our demand of accuracy and speed. Our paradigm is an elegant single stage anchor-free framework, showing great superiority to other existing methods. We evaluate 3DSSD on widely used KITTI dataset and more challenging nuScenes dataset. Our method outperforms all state-of-the-art voxel-based single stage methods by a large margin, and has comparable performance to two stage point-based methods as well, with inference speed more than 25 FPS, 2x faster than former state-of-the-art point-based methods.

<img src="https://user-images.githubusercontent.com/30491025/143854187-54ed1257-a046-4764-81cd-d2c8404137d3.png" width="800"/>
</div>

We implement 3DSSD and provide the results and checkpoints on KITTI datasets.

Some settings in our implementation are different from the [official implementation](https://github.com/Jia-Research-Lab/3DSSD), which bring marginal differences to the performance on KITTI datasets in our experiments. To simplify and unify the models of our implementation, we skip them in our models. These differences are listed as below:

1. We keep the scenes without any object while the official code skips these scenes in training. In the official implementation, only 3229 and 3394 samples are used as training and validation sets, respectively. In our implementation, we keep using 3712 and 3769 samples as training and validation sets, respectively, as those used for all the other models in our implementation on KITTI datasets.

2. We do not modify the decay of `batch normalization` during training.
3. While using [`DataBaseSampler`](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/datasets/pipelines/dbsampler.py#L80) for data augmentation, the official code uses road planes as reference to place the sampled objects while we do not.
4. We perform detection using LIDAR coordinates while the official code uses camera coordinates.

### KITTI

|                    Backbone                    | Class | Lr schd | Mem (GB) | Inf time (fps) |           mAP            |                                                                                                                                                Download                                                                                                                                                |

| :--------------------------------------------: | :---: | :-----: | :------: | :------------: | :----------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| [PointNet2SAMSG](./3dssd_4xb4_kitti-3d-car.py) |  Car  |   72e   |   4.7    |                | 78.58(81.27)<sup>1</sup> | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/3dssd/3dssd_4x4_kitti-3d-car/3dssd_4x4_kitti-3d-car_20210818_203828-b89c8fc4.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/3dssd/3dssd_4x4_kitti-3d-car/3dssd_4x4_kitti-3d-car_20210818_203828.log.json) |

\[1\]: We report two different 3D object detection performance here. 78.58mAP is evaluated by our evaluation code and 81.27mAP is evaluated by the official development kit （so as that used in the paper and official code of 3DSSD ）. We found that the commonly used Python implementation of [`rotate_iou`](https://github.com/traveller59/second.pytorch/blob/e42e4a0e17262ab7d180ee96a0a36427f2c20a44/second/core/non_max_suppression/nms_gpu.py#L605) which is used in our KITTI dataset evaluation, is different from the official implementation in [KITTI benchmark](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d).

```latex
@inproceedings{yang20203dssd,
    author = {Zetong Yang and Yanan Sun and Shu Liu and Jiaya Jia},
    title = {3DSSD: Point-based 3D Single Stage Object Detector},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    year = {2020}
}
```

# 评估指标

在评测阶段，我们一般以数据集本身的特性来选取对应的评估策略，最主要的依据为**标准答案的类型**，一般以下几种类型：

- **选项**：常见于分类任务，判断题以及选择题，目前这类问题的数据集占比最大，有 MMLU, CEval数据集等等，评估标准一般使用准确率--`ACCEvaluator`。
- **短语**：常见于问答以及阅读理解任务，这类数据集主要包括CLUE_CMRC, CLUE_DRCD, DROP数据集等等，评估标准一般使用匹配率--`EMEvaluator`。
- **句子**：常见于翻译以及生成伪代码、命令行任务中，主要包括Flores, Summscreen, Govrepcrs, Iwdlt2017数据集等等，评估标准一般使用BLEU(Bilingual Evaluation Understudy)--`BleuEvaluator`。
- **段落**：常见于文本摘要生成的任务，常用的数据集主要包括Lcsts, TruthfulQA, Xsum数据集等等，评估标准一般使用ROUGE（Recall-Oriented Understudy for Gisting Evaluation）--`RougeEvaluator`。
- **代码**：常见于代码生成的任务，常用的数据集主要包括Humaneval，MBPP数据集等等，评估标准一般使用执行通过率以及 `pass@k`，目前 Opencompass 支持的有`MBPPEvaluator`、`HumanEvaluator`。

还有一类**打分类型**评测任务没有标准答案，比如评判一个模型的输出是否存在有毒，可以直接使用相关 API 服务进行打分，目前支持的有 `ToxicEvaluator`，目前有 realtoxicityprompts 数据集使用此评测方式。

## 已支持评估指标

## 已支持评估指标

目前 OpenCompass 中，常用的 Evaluator 主要放在 [`opencompass/openicl/icl_evaluator`](https://github.com/InternLM/opencompass/tree/main/opencompass/openicl/icl_evaluator)文件夹下， 还有部分数据集特有指标的放在 [`opencompass/datasets`](https://github.com/InternLM/opencompass/tree/main/opencompass/datasets) 的部分文件中。以下是汇总：

| 评估指标           | 评估策略             | 常用后处理方式              | 数据集                                                               |
| ------------------ | -------------------- | --------------------------- | -------------------------------------------------------------------- |
| `ACCEvaluator`     | 正确率               | `first_capital_postprocess` | agieval, ARC, bbh, mmlu, ceval, commonsenseqa, crowspairs, hellaswag |
| `EMEvaluator`      | 匹配率               | None, dataset_specification | drop, CLUE_CMRC, CLUE_DRCD                                           |

| `BleuEvaluator`    | BLEU                 | None, `flores`              | flores, iwslt2017, summscreen, govrepcrs                             |
| `RougeEvaluator`   | ROUGE                | None, dataset_specification | lcsts, truthfulqa, Xsum, XLSum                                       |
| `HumanEvaluator`   | pass@k               | `humaneval_postprocess`     | humaneval_postprocess                                                |
| `MBPPEvaluator`    | 执行通过率           | None                        | mbpp                                                                 |
| `ToxicEvaluator`   | PerspectiveAPI       | None                        | realtoxicityprompts                                                  |

| `AGIEvalEvaluator` | 正确率               | None                        | agieval                                                              |
| `AUCROCEvaluator`  | AUC-ROC              | None                        | jigsawmultilingual, civilcomments                                    |
| `MATHEvaluator`    | 正确率               | `math_postprocess`          | math                                                                 |
| `MccEvaluator`     | Matthews Correlation | None                        | --                                                                   |
| `SquadEvaluator`   | F1-scores            | None                        | --                                                                   |

## 如何配置

评估标准配置一般放在数据集配置文件中，最终的 xxdataset_eval_cfg 会传给 `dataset.infer_cfg` 作为实例化的一个参数。

## 如何配置

下面是 `govrepcrs_eval_cfg` 的定义， 具体可查看 [configs/datasets/govrepcrs](https://github.com/InternLM/opencompass/tree/main/configs/datasets/govrepcrs)。

```python
from opencompass.openicl.icl_evaluator import BleuEvaluator
from opencompass.datasets import GovRepcrsDataset
from opencompass.utils.text_postprocessors import general_cn_postprocess

govrepcrs_reader_cfg = dict(.......)
govrepcrs_infer_cfg = dict(.......)

# 评估指标的配置
govrepcrs_eval_cfg = dict(
    evaluator=dict(type=BleuEvaluator),            # 使用常用翻译的评估器BleuEvaluator
    pred_role='BOT',                               # 接受'BOT' 角色的输出
    pred_postprocessor=dict(type=general_cn_postprocess),      # 预测结果的后处理
    dataset_postprocessor=dict(type=general_cn_postprocess))   # 数据集标准答案的后处理

govrepcrs_datasets = [
    dict(
        type=GovRepcrsDataset,                 # 数据集类名
        path='./data/govrep/',                 # 数据集路径

    dict(
        abbr='GovRepcrs',                      # 数据集别名
        reader_cfg=govrepcrs_reader_cfg,       # 数据集读取配置文件，配置其读取的split，列等
        infer_cfg=govrepcrs_infer_cfg,         # 数据集推理的配置文件，主要 prompt 相关
        eval_cfg=govrepcrs_eval_cfg)           # 数据集结果的评估配置文件，评估标准以及前后处理。
]
```

# 从 MMSegmentation 0.x 迁移

## 引言

本指南介绍了 MMSegmentation 0.x 和 MMSegmentation1.x 在表现和 API 方面的基本区别，以及这些与迁移过程的关系。

## 新的依赖

MMSegmentation 1.x 依赖于一些新的软件包，您可以准备一个新的干净环境，然后根据[安装教程](../get_started.md)重新安装。

或手动安装以下软件包。

1. [MMEngine](https://github.com/open-mmlab/mmengine)：MMEngine 是 OpenMMLab 2.0 架构的核心，我们将许多与计算机视觉无关的内容从 MMCV 拆分到 MMEngine 中。

2. [MMCV](https://github.com/open-mmlab/mmcv)：OpenMMLab 的计算机视觉包。这不是一个新的依赖，但您需要将其升级到 **2.0.0** 或以上的版本。

3. [MMClassification](https://github.com/open-mmlab/mmclassification)（可选）：OpenMMLab 的图像分类工具箱和基准。这不是一个新的依赖，但您需要将其升级到 **1.0.0rc6** 版本。

4. [MMDetection](https://github.com/open-mmlab/mmdetection)(可选): OpenMMLab 的目标检测工具箱和基准。这不是一个新的依赖，但您需要将其升级到 **3.0.0** 或以上的版本。

## 启动训练

OpenMMLab 2.0 的主要改进是发布了 MMEngine，它为启动训练任务的统一接口提供了通用且强大的执行器。

与 MMSeg 0.x 相比，MMSeg 1.x 在 `tools/train.py` 中提供的命令行参数更少

<tr>
<td>功能</td>
<td>原版</td>
<td>新版</td>
</tr>
<tr>
<td>加载预训练模型</td>
<td>--load_from=$CHECKPOINT</td>
<td>--cfg-options load_from=$CHECKPOINT</td>

<tr>
</tr>
<tr>
</tr>
<tr>
<td>从特定检查点恢复训练</td>
<td>--resume-from=$CHECKPOINT</td>
<td>--resume=$CHECKPOINT</td>
</tr>
<tr>
<td>从最新的检查点恢复训练</td>
<td>--auto-resume</td>
<td>--resume='auto'</td>
</tr>
<tr>
<td>训练期间是否不评估检查点</td>
<td>--no-validate</td>
<td>--cfg-options val_cfg=None val_dataloader=None val_evaluator=None</td>
</tr>
<tr>
<td>指定训练设备</td>
<td>--gpu-id=$DEVICE_ID</td>
<td>-</td>
</tr>
<tr>
<td>是否为不同进程设置不同的种子</td>
<td>--diff-seed</td>
<td>--cfg-options randomness.diff_rank_seed=True</td>
</tr>
<td>是否为 CUDNN 后端设置确定性选项</td>
<td>--deterministic</td>
<td>--cfg-options randomness.deterministic=True</td>
</table>

## 测试启动

与训练启动类似，MMSegmentation 1.x 的测试启动脚本在 tools/test.py 中仅提供关键命令行参数，以下是测试启动脚本的区别，更多关于测试启动的细节请参考[这里](../user_guides/4_train_test.md)。

<tr>
<td>0.x</td>
<td>1.x</td>
</tr>
<tr>

<tr>
</tr>
<tr>
<td>指定评测指标</td>
<td>--eval mIoU</td>
<td>--cfg-options test_evaluator.type=IoUMetric</td>
</tr>
<tr>
<td>测试时数据增强</td>
<td>--aug-test</td>
<td>--tta</td>
</tr>
<tr>
<td>测试时是否只保存预测结果不计算评测指标</td>
<td>--format-only</td>
<td>--cfg-options test_evaluator.format_only=True</td>
</tr>
</table>

## 配置文件

### 模型设置

`model.backend`、`model.neck`、`model.decode_head` 和 `model.loss` 字段没有更改。

添加 `model.data_preprocessor` 字段以配置 `DataPreProcessor`，包括：

- `mean`（Sequence，可选）：R、G、B 通道的像素平均值。默认为 None。

- `std`（Sequence，可选）：R、G、B 通道的像素标准差。默认为 None。

- `size`（Sequence，可选）：固定的填充大小。

- `size_divisor`（int，可选）：填充图像可以被当前值整除。

- `seg_pad_val`（float，可选）：分割图的填充值。默认值：255。

- `padding_mode`（str）：填充类型。默认值：'constant'。

  - constant：常量值填充，值由 pad_val 指定。

- `bgr_to_rgb`（bool）：是否将图像从 BGR 转换为 RGB。默认为 False。

- `rgb_to_bgr`（bool）：是否将图像从 RGB 转换为 BGR。默认为 False。

**注：**
有关详细信息，请参阅[模型文档](../advanced_guides/models.md)。

### 数据集设置

**data** 的更改：

**注：**

### 数据集设置

原版 `data` 字段被拆分为 `train_dataloader`、`val_dataloader` 和 `test_dataloader`，允许我们以细粒度配置它们。例如，您可以在训练和测试期间指定不同的采样器和批次大小。
`samples_per_gpu` 重命名为 `batch_size`。
`workers_per_gpu` 重命名为 `num_workers`。

<tr>
<td>

```python
    samples_per_gpu=4,
    workers_per_gpu=4,
    train=dict(...),
    val=dict(...),
    test=dict(...),
)
```

</td>
<tr>
<td>

```python
    batch_size=4,
    dataset=dict(...),
    sampler=dict(type='DefaultSampler', shuffle=True)  # 必须
)

    sampler=dict(type='DefaultSampler', shuffle=False)  # 必须
)

```

</td>
</tr>
</table>

**数据增强变换流程**变更

batch_size=4,
)

```

</td>
</tr>
</table>

- 原始格式转换 **`ToTensor`**、**`ImageToTensor`**、**`Collect`** 组合为 [`PackSegInputs`](mmseg.datasets.transforms.PackSegInputs)
- 我们不建议在数据集流程中执行 **`Normalize`** 和 **Pad**。请将其从流程中删除，并将其设置在 `data_preprocessor` 字段中。
- MMSeg 1.x 中原始的 **`Resize`** 已更改为 **`RandomResize `**，输入参数 `img_scale` 重命名为 `scale`，`keep_ratio` 的默认值修改为 False。
- 原始的 `test_pipeline` 将单尺度和多尺度测试结合在一起，在 MMSeg 1.x 中，我们将其分为 `test_pipeline` 和 `tta_pipeline`。

**注：**
我们将一些数据转换工作转移到数据预处理器中，如归一化，请参阅[文档](package.md)了解更多详细信息。

训练流程

<tr>
<td>

```python
    dict(type='LoadAnnotations', reduce_zero_label=True),
    dict(type='Resize', img_scale=(2560, 640), ratio_range=(0.5, 2.0)),

dict(type='LoadAnnotations', reduce_zero_label=True),
    dict(type='Pad', size=crop_size, pad_val=0, seg_pad_val=255),
    dict(type='Collect', keys=['img', 'gt_semantic_seg']),
]
```

</td>
<tr>
<td>

```python
    dict(
        scale=(2560, 640),
        ratio_range=(0.5, 2.0),
]
```

</td>
</tr>
</table>

测试流程

dict(type='RandomFlip', prob=0.5),
]
```

</td>
</tr>
</table>

测试流程

<tr>
<td>

```python
    dict(
        img_scale=(2560, 640),
        # img_ratios=[0.5, 0.75, 1.0, 1.25, 1.5, 1.75],
        ])
]
```

</td>
<tr>
<td>

```python
    dict(type='Resize', scale=(2560, 640), keep_ratio=True),
]
img_ratios = [0.5, 0.75, 1.0, 1.25, 1.5, 1.75]

]
    dict(type='LoadImageFromFile', backend_args=None),
    dict(
            [
                dict(type='Resize', scale_factor=r, keep_ratio=True)
                for r in img_ratios
            ],
            [
                dict(type='RandomFlip', prob=0., direction='horizontal'),
                dict(type='RandomFlip', prob=1., direction='horizontal')
            ], [dict(type='LoadAnnotations')], [dict(type='PackSegInputs')]
        ])
]
```

</td>
</tr>
</table>

**`evaluation`** 中的更改：

- **`evaluation`** 字段被拆分为 `val_evaluator` 和 `test_evaluator `。而且不再支持 `interval` 和 `save_best` 参数。
  `interval` 已移动到 `train_cfg.val_interval`，`save_best` 已移动到 `default_hooks.checkpoint.save_best`。`pre_eval` 已删除。
- `IoU` 已更改为 `IoUMetric`。

<tr>
<td>

```python

`interval` 已移动到 `train_cfg.val_interval`，`save_best` 已移动到 `default_hooks.checkpoint.save_best`。`pre_eval` 已删除。

<tr>
<td>

```python
evaluation = dict(interval=2000, metric='mIoU', pre_eval=True)
```

</td>
<tr>
<td>

```python
```

</td>
</tr>
</table>

### Optimizer 和 Schedule 设置

**`optimizer`** 和 **`optimizer_config`** 中的更改：

- 现在我们使用 `optim_wrapper` 字段来指定优化过程的所有配置。以及 `optimizer` 是 `optim_wrapper` 的一个子字段。
- `paramwise_cfg` 也是 `optim_wrapper` 的一个子字段，以替代 `optimizer`。
- `optimizer_config` 现在被删除，它的所有配置都被移动到 `optim_wrapper` 中。
- `grad_clip` 重命名为 `clip_grad`。

<tr>
<td>

```python
optimizer = dict(type='AdamW', lr=0.0001, weight_decay=0.0005)
optimizer_config = dict(grad_clip=dict(max_norm=1, norm_type=2))
```

</td>
<tr>
<td>

```python

```

</td>
<tr>
<td>

```python
    optimizer=dict(type='AdamW', lr=0.0001, weight_decay=0.0005),
    clip_grad=dict(max_norm=1, norm_type=2))
```

</td>
</tr>
</table>

**`lr_config`** 中的更改：

- 我们将 `lr_config` 字段删除，并使用新的 `param_scheduler` 替代。
- 我们删除了与 `warmup` 相关的参数，因为我们使用 scheduler 组合来实现该功能。

新的 scheduler 组合机制非常灵活，您可以使用它来设计多种学习率/动量曲线。有关详细信息，请参见[教程](TODO)。

<tr>
<td>

```python
    policy='poly',
    warmup='linear',
    warmup_iters=1500,
    warmup_ratio=1e-6,
    power=1.0,
    min_lr=0.0,
    by_epoch=False)
```

</td>
<tr>
<td>

```python
    dict(
        type='LinearLR', start_factor=1e-6, by_epoch=False, begin=0, end=1500),
    dict(
        power=1.0,
        begin=1500,

<td>

```python
    dict(
    dict(
        eta_min=0.0,
    )
]
```

</td>
</tr>
</table>

**`runner`** 中的更改：

原版 `runner` 字段中的大多数配置被移动到 `train_cfg`、`val_cfg` 和 `test_cfg` 中，以在训练、验证和测试中配置 loop。

<tr>
<td>

```python
runner = dict(type='IterBasedRunner', max_iters=20000)
```

</td>
<tr>
<td>

```python
# `val_interval` 是旧版本的 `evaluation.interval`。
val_cfg = dict(type='ValLoop') # 使用默认的验证循环。
test_cfg = dict(type='TestLoop') # 使用默认的测试循环。
```

</td>
</tr>
</table>

```

</td>
</tr>
</table>

事实上，在 OpenMMLab 2.0 中，我们引入了 `Loop` 来控制训练、验证和测试中的行为。`Runner` 的功能也发生了变化。您可以在 [MMMEngine](https://github.com/open-mmlab/mmengine/) 的[执行器教程](https://github.com/open-mmlab/mmengine/blob/main/docs/zh_cn/design/runner.md) 中找到更多的详细信息。

### 运行时设置

**`checkpoint_config`** 和 **`log_config`** 中的更改：

`checkpoint_config` 被移动到 `default_hooks.checkpoint` 中，`log_config` 被移动到 `default_hooks.logger` 中。
并且我们将许多钩子设置从脚本代码移动到运行时配置的 `default_hooks` 字段中。

```python
    # 记录每次迭代的时间。

    # 每50次迭代打印一次日志。

    # 启用参数调度程序。

    # 每2000次迭代保存一次检查点。

# 启用参数调度程序。

    # 在分布式环境中设置采样器种子。

    # 验证结果可视化。
```

此外，我们将原版 logger 拆分为 logger 和 visualizer。logger 用于记录信息，visualizer 用于在不同的后端显示 logger，如 terminal 和 TensorBoard。

<tr>
<td>

```python
    interval=100,
    hooks=[
    ])
```

</td>
<tr>
<td>

```python
    ...
    logger=dict(type='LoggerHook', interval=100),
)
```

</td>
</tr>
</table>

dict(type='TensorboardVisBackend')]
```

</td>
</tr>
</table>

**`load_from`** 和 **`resume_from`** 中的更改：

- 删除 `resume_from`。我们使用 `resume` 和 `load_from` 来替换它。
  - 如果 `resume=True` 且 `load_from` 为 **not None**，则从 `load_from` 中的检查点恢复训练。
  - 如果 `resume=True` 且 `load_from` 为 **None**，则尝试从工作目录中的最新检查点恢复。
  - 如果 `resume=False` 且 `load_from` 为 **not None**，则只加载检查点，而不继续训练。
  - 如果 `resume=False` 且 `load_from` 为 **None**，则不加载或恢复。

**`dist_params`** 中的更改：`dist_params` 字段现在是 `env_cfg` 的子字段。并且 `env_cfg` 中还有一些新的配置。

```python
    # 是否启用 cudnn_benchmark

    # 设置多进程参数

    # 设置分布式参数
)
```

**`workflow`** 的改动:`workflow` 相关功能被删除。

# 设置多进程参数

    # 设置分布式参数
)
```

新字段 **`visualizer`**：visualizer 是 OpenMMLab 2.0 体系结构中的新设计。我们在 runner 中使用 visualizer 实例来处理结果和日志可视化，并保存到不同的后端。更多详细信息，请参阅[可视化教程](../user_guides/visualization.md)。

新字段 **`default_scope`**：搜索所有注册模块的起点。MMSegmentation 中的 `default_scope` 为 `mmseg`。请参见[注册器教程](https://github.com/open-mmlab/mmengine/blob/main/docs/zh_cn/advanced_tutorials/registry.md)了解更多详情。

# Build for Linux-x86_64

- [Build for Linux-x86_64](#build-for-linux-x86_64)
  - [Install Toolchains](#install-toolchains)
  - [Install Dependencies](#install-dependencies)
    - [Install Dependencies for Model Converter](#install-dependencies-for-model-converter)
    - [Install Dependencies for SDK](#install-dependencies-for-sdk)
    - [Install Inference Engines for MMDeploy](#install-inference-engines-for-mmdeploy)
  - [Build MMDeploy](#build-mmdeploy)
    - [Build Model Converter](#build-model-converter)
    - [Install Model Converter](#install-model-converter)
    - [Build SDK and Demo](#build-sdk-and-demo)

## Install Toolchains

- cmake

  **Make sure cmake version >= 3.14.0**. The below script shows how to install cmake 3.20.0. You can find more versions [here](https://cmake.org/install).

  ```bash
  wget https://github.com/Kitware/CMake/releases/download/v3.20.0/cmake-3.20.0-linux-x86_64.tar.gz

```bash
  tar -xzvf cmake-3.20.0-linux-x86_64.tar.gz
  sudo ln -sf $(pwd)/cmake-3.20.0-linux-x86_64/bin/* /usr/bin/
  ```

- GCC 7+

  MMDeploy requires compilers that support C++17.

  ```bash
  # Add repository if ubuntu < 18.04
  sudo add-apt-repository ppa:ubuntu-toolchain-r/test
  sudo apt-get update
  sudo apt-get install gcc-7
  sudo apt-get install g++-7
  ```

## Install Dependencies

### Install Dependencies for Model Converter

<thead>
  <tr>
    <th>NAME </th>
    <th>INSTALLATION </th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>conda </td>
    <td>Please install conda according to the official <a href="https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html">guide</a>. <br>
    Create a conda virtual environment and activate it. <br>
conda create -n mmdeploy python=3.7 -y
conda activate mmdeploy
    </td>

Create a conda virtual environment and activate it. <br>
    </td>
  </tr>
  <tr>
    <td>PyTorch <br>(>=1.8.0) </td>
    <td>
    Install PyTorch>=1.8.0 by following the <a href="https://pytorch.org/">official instructions</a>. Be sure the CUDA version PyTorch requires matches that in your host.
conda install pytorch==1.8.0 torchvision==0.9.0 cudatoolkit=11.1 -c pytorch -c conda-forge
    </td>
  </tr>
  <tr>
    <td>mmcv </td>
    <td>Install mmcv as follows. Refer to the <a href="https://github.com/open-mmlab/mmcv/tree/2.x#installation">guide</a> for details.
export cu_version=cu111 # cuda 11.1
export torch_version=torch1.8
    </td>
  </tr>
</tbody>
</table>

### Install Dependencies for SDK

You can skip this chapter if you are only interested in the model converter.

    </td>
  </tr>
</tbody>
</table>

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
    <td>
    On Ubuntu >=18.04,
sudo apt-get install libopencv-dev
    On Ubuntu 16.04, OpenCV has to be built from the source code. Please refer to the <a href="https://docs.opencv.org/3.4/d7/d9f/tutorial_linux_install.html">guide</a>.
    </td>

</tr>
  <tr>
    <td>pplcv </td>
    <td>A high-performance image processing library of openPPL.<br>
  <b>It is optional which only be needed if <code>cuda</code> platform is required.</b><br>
git clone https://github.com/openppl-public/ppl.cv.git
cd ppl.cv
export PPLCV_DIR=$(pwd)
git checkout tags/v0.7.0 -b v0.7.0
./build.sh cuda
   </td>
  </tr>
</tbody>

cd ppl.cv
   </td>
  </tr>
</tbody>
</table>

### Install Inference Engines for MMDeploy

Both MMDeploy's model converter and SDK share the same inference engines.

You can select you interested inference engines and do the installation by following the given commands.

<thead>
  <tr>
    <th>NAME</th>
    <th>PACKAGE</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>ONNXRuntime</td>
    <td>onnxruntime<br>(>=1.8.1) </td>
    <td>
    1. Install python package
       <pre><code>pip install onnxruntime==1.8.1</code></pre>
    2. Download the linux prebuilt binary package from <a href="https://github.com/microsoft/onnxruntime/releases/tag/v1.8.1">here</a>.  Extract it and export environment variables as below:

wget https://github.com/microsoft/onnxruntime/releases/download/v1.8.1/onnxruntime-linux-x64-1.8.1.tgz
tar -zxvf onnxruntime-linux-x64-1.8.1.tgz
cd onnxruntime-linux-x64-1.8.1
export ONNXRUNTIME_DIR=$(pwd)
export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH
    </td>
  </tr>
  <tr>
    <td rowspan="2">TensorRT<br> </td>
    <td>TensorRT <br> </td>
    <td>
   1. Login <a href="https://www.nvidia.com/">NVIDIA</a> and download the TensorRT tar file that matches the CPU architecture and CUDA version you are using from <a href="https://developer.nvidia.com/nvidia-tensorrt-download">here</a>. Follow the <a href="https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html#installing-tar">guide</a> to install TensorRT. <br>

2. Here is an example of installing TensorRT 8.2 GA Update 2 for Linux x86_64 and CUDA 11.x that you can refer to. First of all, click <a href="https://developer.nvidia.com/compute/machine-learning/tensorrt/secure/8.2.3.0/tars/tensorrt-8.2.3.0.linux.x86_64-gnu.cuda-11.4.cudnn8.2.tar.gz">here</a> to download CUDA 11.x TensorRT 8.2.3.0 and then install it and other dependency like below:
cd /the/path/of/tensorrt/tar/gz/file
tar -zxvf TensorRT-8.2.3.0.Linux.x86_64-gnu.cuda-11.4.cudnn8.2.tar.gz
pip install TensorRT-8.2.3.0/python/tensorrt-8.2.3.0-cp37-none-linux_x86_64.whl
export TENSORRT_DIR=$(pwd)/TensorRT-8.2.3.0
export LD_LIBRARY_PATH=$TENSORRT_DIR/lib:$LD_LIBRARY_PATH
pip install pycuda
   </td>
  </tr>
  <tr>
    <td>cuDNN </td>
    <td>
    1. Download cuDNN that matches the CPU architecture, CUDA version and TensorRT version you are using from <a href="https://developer.nvidia.com/rdp/cudnn-archive"> cuDNN Archive</a>. <br>

<td>
In the above TensorRT's installation example, it requires cudnn8.2. Thus, you can download <a href="https://developer.nvidia.com/compute/machine-learning/cudnn/secure/8.2.1.32/11.3_06072021/cudnn-11.3-linux-x64-v8.2.1.32.tgz">CUDA 11.x cuDNN 8.2</a><br>
    2. Extract the compressed file and set the environment variables
cd /the/path/of/cudnn/tgz/file
tar -zxvf cudnn-11.3-linux-x64-v8.2.1.32.tgz
export CUDNN_DIR=$(pwd)/cuda
export LD_LIBRARY_PATH=$CUDNN_DIR/lib64:$LD_LIBRARY_PATH
   </td>
  </tr>
  <tr>
    <td>PPL.NN</td>
    <td>ppl.nn </td>
    <td>
    1. Please follow the <a href="https://github.com/openppl-public/ppl.nn/blob/master/docs/en/building-from-source.md">guide</a> to build <code>ppl.nn</code> and install <code>pyppl</code>.<br>

1. Please follow the <a href="https://github.com/openppl-public/ppl.nn/blob/master/docs/en/building-from-source.md">guide</a> to build <code>ppl.nn</code> and install <code>pyppl</code>.<br>
    2. Export pplnn's root path to environment variable
cd ppl.nn
export PPLNN_DIR=$(pwd)
    </td>
  </tr>
  <tr>
    <td>OpenVINO</td>
    <td>openvino </td>
    <td>1. Install <a href="https://docs.openvino.ai/2021.4/get_started.html">OpenVINO</a> package
pip install openvino-dev
2. <b>Optional</b>. If you want to use OpenVINO in MMDeploy SDK, please install and configure it by following the <a href="https://docs.openvino.ai/2021.4/openvino_docs_install_guides_installing_openvino_linux.html#install-openvino">guide</a>.
    </td>
  </tr>
  <tr>
    <td>ncnn </td>
    <td>1. Download and build ncnn according to its <a href="https://github.com/Tencent/ncnn/wiki/how-to-build">wiki</a>.

</td>
  </tr>
  <tr>
Make sure to enable <code>-DNCNN_PYTHON=ON</code> in your build command. <br>
2. Export ncnn's root path to environment variable
cd ncnn
export NCNN_DIR=$(pwd)
export LD_LIBRARY_PATH=${NCNN_DIR}/build/install/lib/:$LD_LIBRARY_PATH
3. Install pyncnn
cd ${NCNN_DIR}/python
pip install -e .
    </td>
  </tr>
  <tr>
    <td>TorchScript</td>
    <td>libtorch</td>
    <td>
    1. Download libtorch from <a href="https://pytorch.org/get-started/locally/">here</a>. Please note that only <b>Pre-cxx11 ABI</b> and <b>version 1.8.1+</b> on Linux platform are supported by now. For previous versions of libtorch, you can find them in the <a href="https://github.com/pytorch/pytorch/issues/40961#issuecomment-1017317786">issue comment</a>. <br>

2. Take Libtorch1.8.1+cu111 as an example. You can install it like this:
   <pre><code>
wget https://download.pytorch.org/libtorch/cu111/libtorch-shared-with-deps-1.8.1%2Bcu111.zip
unzip libtorch-shared-with-deps-1.8.1+cu111.zip
cd libtorch
export Torch_DIR=$(pwd)
export LD_LIBRARY_PATH=$Torch_DIR/lib:$LD_LIBRARY_PATH
   </code></pre>
    </td>
  </tr>
  <tr>
    <td>Ascend</td>
    <td>CANN</td>
    <td>
    1. Install CANN follow <a href="https://www.hiascend.com/document/detail/en/CANNCommunityEdition/51RC1alphaX/softwareinstall/instg/atlasdeploy_03_0002.html">official guide</a>.<br>
    2. Setup environment
export ASCEND_TOOLKIT_HOME="/usr/local/Ascend/ascend-toolkit/latest"
    </td>
  </tr>
  <tr>
  <td>TVM</td>
  <td>
  1. Install TVM follow <a href="https://tvm.apache.org/docs/install/from_source.html">official guide</a>.<br>
  2. Setup environment
  <pre><code>
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${TVM_HOME}/build

2. Setup environment
export PYTHONPATH=${TVM_HOME}/python:${PYTHONPATH}
  </code></pre>
  </td>
  </tr>
</tbody>
</table>

Note: <br>
If you want to make the above environment variables permanent, you could add them to <code>~/.bashrc</code>. Take the ONNXRuntime for example,

```bash
echo '# set env for onnxruntime' >> ~/.bashrc
echo "export ONNXRUNTIME_DIR=${ONNXRUNTIME_DIR}" >> ~/.bashrc
echo "export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH" >> ~/.bashrc
```

## Build MMDeploy

```bash
```

#### Build Model Converter

If one of inference engines among ONNXRuntime, TensorRT, ncnn and libtorch is selected, you have to build the corresponding custom ops.

- **ONNXRuntime** Custom Ops

  ```bash

  ```bash
  cmake -DCMAKE_CXX_COMPILER=g++-7 -DMMDEPLOY_TARGET_BACKENDS=ort -DONNXRUNTIME_DIR=${ONNXRUNTIME_DIR} ..
  make -j$(nproc) && make install
  ```

- **TensorRT** Custom Ops

  ```bash
  cmake -DCMAKE_CXX_COMPILER=g++-7 -DMMDEPLOY_TARGET_BACKENDS=trt -DTENSORRT_DIR=${TENSORRT_DIR} -DCUDNN_DIR=${CUDNN_DIR} ..
  ```

- **ncnn** Custom Ops

  ```bash
  cmake -DCMAKE_CXX_COMPILER=g++-7 -DMMDEPLOY_TARGET_BACKENDS=ncnn -Dncnn_DIR=${NCNN_DIR}/build/install/lib/cmake/ncnn ..
  ```

- **TorchScript** Custom Ops

  ```bash
  cmake -DCMAKE_CXX_COMPILER=g++-7 -DMMDEPLOY_TARGET_BACKENDS=torchscript -DTorch_DIR=${Torch_DIR} ..
  ```

  ```

Please check [cmake build option](cmake_option.md).

#### Install Model Converter

```bash
```

**Note**

- Some dependencies are optional. Simply running `pip install -e .` will only install the minimum runtime requirements.
  To use optional dependencies, install them manually with `pip install -r requirements/optional.txt` or specify desired extras when calling `pip` (e.g. `pip install -e .[optional]`).
  Valid keys for the extras field are: `all`, `tests`, `build`, `optional`.
- It is recommended to [install patch for cuda10](https://developer.nvidia.com/cuda-10.2-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=runfilelocal), otherwise GEMM related errors may occur when model runs

### Build SDK and Demo

MMDeploy provides two recipes as shown below for building SDK with ONNXRuntime and TensorRT as inference engines respectively.
You can also activate other engines after the model.

- cpu + ONNXRuntime

  ```Bash
  cmake .. \
      -DCMAKE_CXX_COMPILER=g++-7 \
      -DMMDEPLOY_TARGET_BACKENDS=ort \
      -DONNXRUNTIME_DIR=${ONNXRUNTIME_DIR}

  ```

- cuda + TensorRT

  ```Bash
  cmake .. \
      -DMMDEPLOY_TARGET_DEVICES="cuda;cpu" \
      -DMMDEPLOY_TARGET_BACKENDS=trt \
      -Dpplcv_DIR=${PPLCV_DIR}/cuda-build/install/lib/cmake/ppl \

-DMMDEPLOY_BUILD_EXAMPLES=ON \
      -DTENSORRT_DIR=${TENSORRT_DIR} \
      -DCUDNN_DIR=${CUDNN_DIR}

  ```

- pplnn

  ```Bash
  cmake .. \
      -DMMDEPLOY_TARGET_BACKENDS=pplnn \
      -Dpplcv_DIR=${PPLCV_DIR}/cuda-build/cuda-build/install/lib/cmake/ppl \
      -Dpplnn_DIR=${PPLNN_DIR}/pplnn-build/install/lib/cmake/ppl

  ```

- cuda + TensorRT + onnxruntime + openvino + ncnn

-Dpplnn_DIR=${PPLNN_DIR}/pplnn-build/install/lib/cmake/ppl

  ```

  If the [ncnn auto-install script](../../../tools/scripts/build_ubuntu_x64_ncnn.py) is used, protobuf will be installed in mmdeploy-dep/pbinstall in the same directory as mmdeploy.

  ```Bash
  export PROTO_DIR=/path/to/mmdeploy-dep/pbinstall
  cmake .. \
      -DMMDEPLOY_TARGET_BACKENDS="trt;ort;ncnn" \
      -DCUDNN_DIR=${CUDNN_DIR} \
      -DONNXRUNTIME_DIR=${ONNXRUNTIME_DIR} \
      -DInferenceEngine_DIR=${InferenceEngine_DIR} \
      -Dncnn_DIR=${NCNN_DIR}/build/install/lib/cmake/ncnn \

-DCUDNN_DIR=${CUDNN_DIR} \
      -DProtobuf_LIBRARIES=${PROTO_DIR}/lib/libprotobuf.so \
      -DProtobuf_PROTOC_EXECUTABLE=${PROTO_DIR}/bin/protoc \
      -DProtobuf_INCLUDE_DIR=${PROTO_DIR}/pbinstall/include
```

# 准备数据集

MMAction2 支持许多现有的数据集。在本章中，我们将引导您准备 MMAction2 的数据集。

- [准备数据集](#准备数据集)
  - [关于视频数据格式的说明](#关于视频数据格式的说明)
  - [使用内置数据集](#使用内置数据集)
  - [使用自定义数据集](#使用自定义数据集)
    - [动作识别](#动作识别)
    - [基于骨骼的动作识别](#基于骨骼的动作识别)
    - [时空动作检测](#时空动作检测)
    - [时序动作定位](#时序动作定位)
  - [使用混合数据集进行训练](#使用混合数据集进行训练)
    - [重复数据集](#重复数据集)
  - [浏览数据集](#浏览数据集)

## 关于视频数据格式的说明

MMAction2 支持两种类型的数据格式：原始帧和视频。前者在之前的项目（如 [TSN](https://github.com/yjxiong/temporal-segment-networks)）中被广泛使用。当 SSD 可用时，这种方法运行速度很快，但无法满足日益增长的数据集需求（例如，最新的 [Kinetics](https://www.deepmind.com/open-source/kinetics) 数据集有 65 万个视频，总帧数将占用几 TB 的空间）。后者可以节省空间，但必须在执行时进行计算密集型的视频解码。为了加快视频解码速度，我们支持几种高效的视频加载库，如 [decord](https://github.com/zhreshold/decord)、[PyAV](https://github.com/PyAV-Org/PyAV) 等。

## 使用内置数据集

MMAction2 已经支持许多数据集，我们在路径 `$MMACTION2/tools/data/` 下提供了用于数据准备的 shell 脚本，请参考[支持的数据集](../datasetzoo_satatistics.md)以获取准备特定数据集的详细信息。

最简单的方法是将您的数据集转换为现有的数据集格式：

- `RawFrameDataset` 和 `VideoDataset` 用于[动作识别](#动作识别)

- `PoseDataset` 用于[基于骨骼的动作识别](#基于骨骼的动作识别)
- `AVADataset` 用于[时空动作检测](#时空动作检测)
- `ActivityNetDataset` 用于[时序动作定位](#时序动作定位)

在数据预处理之后，用户需要进一步修改配置文件以使用数据集。以下是在原始帧格式中使用自定义数据集的示例。

在 `configs/task/method/my_custom_config.py` 中：

```python
...
# 数据集设置
dataset_type = 'RawframeDataset'
data_root = 'path/to/your/root'
data_root_val = 'path/to/your/root_val'
ann_file_train = 'data/custom/custom_train_list.txt'
ann_file_val = 'data/custom/custom_val_list.txt'
ann_file_test = 'data/custom/custom_val_list.txt'
...
    videos_per_gpu=32,
        ...),
    val=dict(
        ...),
        ann_file=ann_file_test,
        ...))
...
```

### 动作识别

动作识别有两种类型的注释文件。

- `RawFrameDataset` 的原始帧注释

ann_file=ann_file_val,
        ...),
        ...))
...
```

### 动作识别

  原始帧数据集的注释是一个包含多行的文本文件，每一行表示一个视频的 `frame_directory`（相对路径）、视频的 `total_frames` 和视频的 `label`，用空格分隔。

  以下是一个示例。

  ```
  some/directory-1 163 1
  some/directory-2 122 1
  some/directory-3 258 2
  some/directory-4 234 2
  some/directory-5 295 3
  some/directory-6 121 3
  ```

- `VideoDataset` 的视频注释

  视频数据集的注释是一个包含多行的文本文件，每一行表示一个样本视频，包括 `filepath`（相对路径）和 `label`，用空格分隔。

  以下是一个示例。

  ```
  some/path/000.mp4 1
  some/path/001.mp4 1
  some/path/002.mp4 2
  some/path/003.mp4 2
  some/path/004.mp4 3
  some/path/005.mp4 3
  ```

### 基于骨骼点的动作识别

该任务基于骨骼序列（关键点的时间序列）识别动作类别。我们提供了一些方法来构建自定义的骨骼数据集。

- 从 RGB 视频数据构建

  您需要从视频中提取关键点数据，并将其转换为支持的格式。我们提供了一个[教程](https://github.com/open-mmlab/mmaction2/tree/main/configs/skeleton/posec3d/custom_dataset_training.md)，详细介绍了如何执行。

- 从现有关键点数据构建

  假设您已经有了 coco 格式的关键点数据，您可以将它们收集到一个 pickle 文件中。

  每个 pickle 文件对应一个动作识别数据集。pickle 文件的内容是一个字典，包含两个字段：`split` 和 `annotations`

  1. Split：`split` 字段的值是一个字典：键是拆分名称，值是属于特定剪辑的视频标识符列表。
  2. Annotations：`annotations` 字段的值是一个骨骼注释列表，每个骨骼注释是一个字典，包含以下字段：
     - `frame_dir`（str）：对应视频的标识符。
     - `total_frames`（int）：此视频中的帧数。
     - `img_shape`（tuple\[int\]）：视频帧的形状，一个包含两个元素的元组，格式为 `(height, width)`。仅对 2D 骨骼需要。
     - `original_shape`（tuple\[int\]）：与 `img_shape` 相同。
     - `label`（int）：动作标签。
     - `keypoint`（np.ndarray，形状为 `[M x T x V x C]`）：关键点注释。
       - M：人数；
       - T：帧数（与 `total_frames` 相同）；
       - V：关键点数量（NTURGB+D 3D 骨骼为 25，Coco 为 17，OpenPose 为 18 等）；
       - C：关键点坐标的维数（2D 关键点为 C=2，3D 关键点为 C=3）。
     - `keypoint_score`（np.ndarray，形状为 `[M x T x V]`）：关键点的置信度分数。仅对 2D 骨骼需要。

  以下是一个示例：

  ```
  {

- C：关键点坐标的维数（2D 关键点为 C=2，3D 关键点为 C=3）。

  以下是一个示例：

  ```
  {
      "split":
          {
              'xsub_train':
                  ['S001C001P001R001A001', ...],
              'xsub_val':
                  ['S001C001P003R001A001', ...],
              ...
          }

      "annotations:
          [
              {
                  {
                      'frame_dir': 'S001C001P001R001A001',
                      'label': 0,
                      'img_shape': (1080, 1920),
                      'original_shape': (1080, 1920),
                      'total_frames': 103,
                      'keypoint': array([[[[1032. ,  334.8], ...]]])
                      'keypoint_score': array([[[0.934 , 0.9766, ...]]])
                  },
                  {
                      'frame_dir': 'S001C001P003R001A001',
                      ...
                  },
                  ...

              }

},
                  {
                      ...
                  },
                  ...

              }
          ]
  }
  ```

  支持其他关键点格式需要进行进一步修改，请参考[自定义数据集](../advanced_guides/customize_dataset.md)。

### 时空动作检测

MMAction2 支持基于 `AVADataset` 的时空动作检测任务。注释包含真实边界框和提议边界框。

- 真实边界框
  真实边界框是一个包含多行的 csv 文件，每一行是一个帧的检测样本，格式如下：

  video_identifier, time_stamp, lt_x, lt_y, rb_x, rb_y, label, entity_id
  每个字段的含义如下：
  `video_identifier`：对应视频的标识符
  `time_stamp`：当前帧的时间戳
  `lt_x`：左上角点的规范化 x 坐标
  `lt_y`：左上角点的规范化 y 坐标
  `rb_y`：右下角点的规范化 x 坐标
  `rb_y`：右下角点的规范化 y 坐标
  `label`：动作标签
  `entity_id`：一个唯一的整数，允许将此框与该视频相邻帧中描绘同一个人的其他框连接起来

  以下是一个示例：

  ```
  _-Z6wFjXtGQ,0902,0.063,0.049,0.524,0.996,12,0
  _-Z6wFjXtGQ,0902,0.063,0.049,0.524,0.996,74,0
  ...
  ```

- 提议边界框
  提议边界框是由一个人体检测器生成的 pickle 文件，通常需要在目标数据集上进行微调。pickle 文件包含一个带有以下数据结构的字典：

  `{'video_identifier,time_stamp': bbox_info}`

  video_identifier（str）：对应视频的标识符

...
  ```

- 提议边界框

  time_stamp（int）：当前帧的时间戳
  bbox_info（np.ndarray，形状为`[n, 5]`）：检测到的边界框，\<x1> \<y1> \<x2> \<y2> \<score>。x1、x2、y1、y2 是相对于帧大小归一化的值，范围为 0.0-1.0。

### 时序动作定位

我们支持基于 `ActivityNetDataset` 的时序动作定位。ActivityNet 数据集的注释是一个 json 文件。每个键是一个视频名，相应的值是视频的元数据和注释。

以下是一个示例：

```
{
  "video1": {
      "duration_second": 211.53,
      "duration_frame": 6337,
      "annotations": [
          {
              "segment": [
                  30.025882995319815,
                  205.2318595943838
              ],
              "label": "Rock climbing"
          }
      ],
      "feature_frame": 6336,
      "fps": 30.0,
      "rfps": 29.9579255898
  },
  "video2": {...
  }
  ...
}
```

MMAction2 还支持混合数据集进行训练。目前，它支持重复数据集。

### 重复数据集

我们使用 `RepeatDataset` 作为包装器来重复数据集。例如，假设原始数据集为 `Dataset_A`，要重复它，配置如下所示

```python

},
  }
  ...
}
```

### 重复数据集

```python
        times=N,
        dataset=dict(  # 这是 Dataset_A 的原始配置
            ...
        )
    )
```

## 浏览数据集

即将推出...

# What's New in MMOCR 1.x

Here are some highlights of MMOCR 1.x compared to 0.x.

1. **New engines**. MMOCR 1.x is based on [MMEngine](https://github.com/open-mmlab/mmengine), which provides a general and powerful runner that allows more flexible customizations and significantly simplifies the entrypoints of high-level interfaces.

2. **Unified interfaces**. As a part of the OpenMMLab 2.0 projects, MMOCR 1.x unifies and refactors the interfaces and internal logics of train, testing, datasets, models, evaluation, and visualization. All the OpenMMLab 2.0 projects share the same design in those interfaces and logics to allow the emergence of multi-task/modality algorithms.

3. **Cross project calling**. Benefiting from the unified design, you can use the models implemented in other OpenMMLab projects, such as MMDet. We provide an example of how to use MMDetection's Mask R-CNN through `MMDetWrapper`. Check our documents for more details. More wrappers will be released in the future.

5. **More documentation and tutorials**. We add a bunch of documentation and tutorials to help users get started more smoothly.

6. **One-stop Dataset Preparaion**. Multiple datasets are instantly ready with only one line of command, via our [Dataset Preparer](https://mmocr.readthedocs.io/en/dev-1.x/user_guides/data_prepare/dataset_preparer.html).

7. **Embracing more `projects/`**: We now introduce `projects/` folder, where some experimental features, frameworks and models can be placed, only needed to satisfy the minimum requirement on the code quality. Everyone is welcome to post their implementation of any great ideas in this folder! Learn more from our [example project](https://github.com/open-mmlab/mmocr/blob/dev-1.x/projects/example_project/).

8. **More models**. MMOCR 1.0 supports more tasks and more state-of-the-art models!

ncnn openmp 最佳实践

### ncnn占用过多cpu资源

   使用ncnn推理运算，cpu占用非常高甚至所有核心占用都接近100%。

   如果还有其它线程或进程需要较多的cpu资源，运行速度下降严重。

### cpu占用高的根本原因

1. ncnn使用openmp API控制多线程加速推理计算。默认情况下，线程数等于cpu内核数。如果推理需要高频率运行，必然占用大部分
   cpu资源。

2. openmp内部维护一个线程池，线程池最大可用线程数等于cpu内核数。(核心过多时最大限制是15？）获取和归还线程时需要同步。

   为了提高效率，几乎所有omp实现都使用了自旋锁同步(simpleomp除外)。自旋锁默认的spin time是200ms。因此一个线程被调度后，
   需要忙等待最多200ms。

### 为什么使用vulkan加速后cpu占用依然很高。

1. 加载参数文件时也使用了openmp，这部分是在cpu上运行的。

2. 显存上传前和下载后的 fp32 fp16转换是在cpu上执行的，这部分逻辑也使用了openmp。

### 解决方法

```
1. 绑核
```
   如果使用有大小核cpu的设备，建议通过ncnn::set_cpu_powersave(int)绑定大核或小核，注意windows系统不支持绑核。顺便说一下，ncnn支持不同的模型运行在不同的核心。假设硬件平台有2个大核，4个小核，你想把netA运行在大核，netB运行在小核。
   可以通过std::thread or pthread创建两个线程，运行如下代码：
   
   ```
   void thread_1()
   {
      ncnn::set_cpu_powersave(2); // bind to big cores
      netA.opt.num_threads = 2;
   }

   void thread_2()
   {
      ncnn::set_cpu_powersave(1); // bind to little cores
      netB.opt.num_threads = 4;
   }
   ```

```
2. 使用更少的线程数。
```

netA.opt.num_threads = 2;
   }

   {
   }
   ```

```
```
   通过ncnn::set_omp_num_threads(int)或者net.opt.num_threads字段设置线程数为cpu内核数的一半或更小。如果使用clang的libomp，
   建议线程数不超过8，如果使用其它omp库，建议线程数不超过4。
```
3. 减小openmp blocktime。
```
   可以修改ncnn::set_kmp_blocktime(int)或者修改net.opt.openmp_blocktime，这个参数是ncnn API设置的spin time，默认是20ms。
   可以根据情况设置更小的值，或者直接改为0。

   局限：目前只有clang的libomp库有实现，vcomp和libgomp都没有相应接口，如果不是使用clang编译的，这个值默认还是200ms。
   如果使用vcomp或libgomp, 可以使用环境变量OMP_WAIT_POLICY=PASSIVE禁用spin time，如果使用simpleomp,不需要设置这个参数。
```
4. 限制openmp线程池可用线程数量。
```
   即使减小了openmp线程数量，cpu占用率仍然可能会很高。这在cpu核心特别多的服务器上比较常见。这是因为线程池中的等待线程使用
   自旋锁忙等待，可以通过限制线程池可用线程数量减轻这种影响。

   一般可以通过设置OMP_THREAD_LIMIT环境变量。simpleomp目前不支持这一特性，不需要设置。注意这个环境变量仅在程序启动前设置才有效。
```
5. 完全禁用openmp
```
   如果只有一个cpu核心，或者使用vulkan加速，建议关闭openmp, cmake编译时指定-DNCNN_OPENMP=OFF即可。

## <a href='https://mmdeploy.readthedocs.io/en/latest/'>English</a>

## <a href='https://mmdeploy.readthedocs.io/zh_CN/latest/'>简体中文</a>

## <a href='https://mmdetection3d.readthedocs.io/en/latest/'>English</a>

## <a href='https://mmdetection3d.readthedocs.io/zh_CN/latest/'>简体中文</a>

### use ncnn with own project

After building ncnn, there is one or more library files generated. Consider integrating ncnn into your own project, you may use ncnn's installating provided cmake config file, or by manually specify library path(s).

**with cmake**

Ensure your project is built by cmake. Then in your project's CMakeLists.txt, add these lines:

```cmake
set(ncnn_DIR "<ncnn_install_dir>/lib/cmake/ncnn" CACHE PATH "Directory that contains ncnnConfig.cmake")
find_package(ncnn REQUIRED)
target_link_libraries(my_target ncnn)
```
After this, both the header file search path ("including directories") and library paths are configured automatically, including vulkan related dependencies.

Note: you have to change `<ncnn_install_dir>` to your machine's directory, it is the directory that contains `ncnnConfig.cmake`.

For the prebuilt ncnn release packages, ncnnConfig is located in:
- for `ncnn-YYYYMMDD-windows-vs2019`, it is `lib/cmake/ncnn`

- for `ncnn-YYYYMMDD-android-vulkan`, it is `${ANDROID_ABI}/lib/cmake/ncnn` (`${ANDROID_ABI}` is defined in NDK's cmake toolchain file)
- other prebuilt release packages are with similar condition

**manually specify**

You may also manually specify ncnn library path and including directory. Note that if you use ncnn with vulkan, it is also required to specify vulkan related dependencies.

For example, on Visual Studio debug mode with vulkan required, the lib paths are:
```
E:\github\ncnn\build\vs2019-x64\install\lib\ncnnd.lib
E:\lib\VulkanSDK\1.2.148.0\Lib\vulkan-1.lib
E:\github\ncnn\build\vs2019-x64\install\lib\SPIRVd.lib
E:\github\ncnn\build\vs2019-x64\install\lib\glslangd.lib
E:\github\ncnn\build\vs2019-x64\install\lib\MachineIndependentd.lib
E:\github\ncnn\build\vs2019-x64\install\lib\OGLCompilerd.lib

E:\github\ncnn\build\vs2019-x64\install\lib\OSDependentd.lib
E:\github\ncnn\build\vs2019-x64\install\lib\GenericCodeGend.lib
```
And for its release mode, lib paths are:
```
E:\github\ncnn\build\vs2019-x64\install\lib\ncnn.lib
E:\github\ncnn\build\vs2019-x64\install\lib\SPIRV.lib
E:\github\ncnn\build\vs2019-x64\install\lib\glslang.lib
E:\github\ncnn\build\vs2019-x64\install\lib\MachineIndependent.lib
E:\github\ncnn\build\vs2019-x64\install\lib\OGLCompiler.lib
E:\github\ncnn\build\vs2019-x64\install\lib\OSDependent.lib
E:\github\ncnn\build\vs2019-x64\install\lib\GenericCodeGen.lib
```

# Projects based on MMPose

There are many projects built upon MMPose. We list some of them as examples of how to extend MMPose for your own projects. As the page might not be completed, please feel free to create a PR to update this page.

Some projects extend the boundary of MMPose for deployment or other research fields. They reveal the potential of what MMPose can do. We list several of them as below.

- [Anime Face Detector](https://github.com/hysts/anime-face-detector): An anime face landmark detection toolbox.
- [PosePipeline](https://github.com/peabody124/PosePipeline): Open-Source Human Pose Estimation Pipeline for Clinical Research

There are also projects released with papers. Some of the papers are published in top-tier conferences (CVPR, ICCV, and ECCV), the others are also highly influential. We list some of these works as a reference for the community to develop and compare new pose estimation algorithms. Methods already supported and maintained by MMPose are not listed.

- Pose for Everything: Towards Category-Agnostic Pose Estimation, ECCV 2022. [\[paper\]](https://arxiv.org/abs/2207.10387)[\[github\]](https://github.com/luminxu/Pose-for-Everything)
- UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning, ICLR 2022. [\[paper\]](https://arxiv.org/abs/2201.04676)[\[github\]](https://github.com/Sense-X/UniFormer)

- Poseur:Direct Human Pose Regression with Transformers, ECCV 2022. [\[paper\]](https://arxiv.org/abs/2201.07412)[\[github\]](https://github.com/aim-uofa/Poseur)
- ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond, NeurIPS 2022. [\[paper\]](https://arxiv.org/abs/2106.03348)[\[github\]](https://github.com/ViTAE-Transformer/ViTAE-Transformer)
- Dite-HRNet:Dynamic Lightweight High-Resolution Network for Human Pose Estimation, IJCAI-ECAI 2021. [\[paper\]](https://arxiv.org/abs/2204.10762)[\[github\]](https://github.com/ZiyiZhang27/Dite-HRNet)

# Welcome to Projects of MMagic

Welcome to the MMagic community!
The MMagic ecosystem consists of tutorials, libraries, and projects from a broad set of researchers in academia and industry, ML and application engineers.
The goal of this ecosystem is to support, accelerate, and aid in your AIGC exploration with MMagic for image, video, 3D content generation, editing and processing.

Here are a few projects that are built upon MMagic. They are examples of how to use MMagic as a library, to make your projects more maintainable.
Please find more projects in [MMagic Ecosystem](https://openmmlab.com/ecosystem).

## Show your projects on OpenMMLab Ecosystem

You can submit your project so that it can be shown on the homepage of [OpenMMLab](https://openmmlab.com/ecosystem).

## Add example projects to MMagic

Here is an [example project](./example_project) about how to add your projects to MMagic.
You can copy and create your own project from the [example project](./example_project).

We also provide some documentation listed below for your reference:

- [Contribution Guide](https://mmagic.readthedocs.io/en/latest/community/contributing.html)

  The guides for new contributors about how to add your projects to MMagic.

- [New Model Guide](https://mmagic.readthedocs.io/en/latest/howto/models.html)

  The documentation of adding new models.

- [Discussions](https://github.com/open-mmlab/mmagic/discussions)

  Welcome to start a discussion!

## Projects of libraries and toolboxes

- [PowerVQE](https://github.com/ryanxingql/powervqe): Open framework for quality enhancement of compressed videos based on PyTorch and MMagic.

- [VR-Baseline](https://github.com/linjing7/VR-Baseline): Video Restoration Toolbox.

- [Derain-Toolbox](https://github.com/biubiubiiu/derain-toolbox): Single Image Deraining Toolbox and Benchmark

## Projects of research papers

- [Towards Interpretable Video Super-Resolution via Alternating Optimization, ECCV 2022](https://arxiv.org/abs/2207.10765)[\[github\]](https://github.com/caojiezhang/DAVSR)

- [SepLUT:Separable Image-adaptive Lookup Tables for Real-time Image Enhancement, ECCV 2022](https://arxiv.org/abs/2207.08351)[\[github\]](https://github.com/ImCharlesY/SepLUT)

- [TTVSR: Learning Trajectory-Aware Transformer for Video Super-Resolution, CVPR 2022](https://arxiv.org/abs/2204.04216)[\[github\]](https://github.com/researchmm/TTVSR)

- [Arbitrary-Scale Image Synthesis, CVPR 2022](https://arxiv.org/pdf/2204.02273.pdf)[\[github\]](https://github.com/vglsd/ScaleParty)

- [Investigating Tradeoffs in Real-World Video Super-Resolution(RealBasicVSR), CVPR 2022](https://arxiv.org/abs/2111.12704)[\[github\]](https://github.com/ckkelvinchan/RealBasicVSR)

- [BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment, CVPR 2022](https://arxiv.org/abs/2104.13371)[\[github\]](https://github.com/ckkelvinchan/BasicVSR_PlusPlus)

- [Multi-Scale Memory-Based Video Deblurring, CVPR 2022](https://arxiv.org/abs/2204.02977)[\[github\]](https://github.com/jibo27/MemDeblur)

- [AdaInt:Learning Adaptive Intervals for 3D Lookup Tables on Real-time Image Enhancement, CVPR 2022](https://arxiv.org/abs/2204.13983)[\[github\]](https://github.com/ImCharlesY/AdaInt)

- [A New Dataset and Transformer for Stereoscopic Video Super-Resolution, CVPRW 2022](https://openaccess.thecvf.com/content/CVPR2022W/NTIRE/papers/Imani_A_New_Dataset_and_Transformer_for_Stereoscopic_Video_Super-Resolution_CVPRW_2022_paper.pdf)[\[github\]](https://github.com/H-deep/Trans-SVSR)

- [Liquid warping GAN with attention: A unified framework for human image synthesis, TPAMI 2021](https://arxiv.org/pdf/2011.09055.pdf)[\[github\]](https://github.com/iPERDance/iPERCore)

- [BasicVSR:The Search for Essential Components in Video Super-Resolution and Beyond, CVPR 2021](https://arxiv.org/abs/2012.02181)[\[github\]](https://github.com/ckkelvinchan/BasicVSR-IconVSR)

- [GLEAN:Generative Latent Bank for Large-Factor Image Super-Resolution, CVPR 2021](https://arxiv.org/abs/2012.00739)[\[github\]](https://github.com/ckkelvinchan/GLEAN)

- [DAN:Unfolding the Alternating Optimization for Blind Super Resolution, NeurIPS 2020](https://arxiv.org/abs/2010.02631v4)[\[github\]](https://github.com/AlexZou14/DAN-Basd-on-Openmmlab)

# RTMDet: An Empirical Study of Designing Real-Time Object Detectors

> [RTMDet: An Empirical Study of Designing Real-Time Object Detectors](https://arxiv.org/abs/2212.07784)

In this paper, we aim to design an efficient real-time object detector that exceeds the YOLO series and is easily extensible for many object recognition tasks such as instance segmentation and rotated object detection. To obtain a more efficient model architecture, we explore an architecture that has compatible capacities in the backbone and neck, constructed by a basic building block that consists of large-kernel depth-wise convolutions. We further introduce soft labels when calculating matching costs in the dynamic label assignment to improve accuracy. Together with better training techniques, the resulting object detector, named RTMDet, achieves 52.8% AP on COCO with 300+ FPS on an NVIDIA 3090 GPU, outperforming the current mainstream industrial detectors. RTMDet achieves the best parameter-accuracy trade-off with tiny/small/medium/large/extra-large model sizes for various application scenarios, and obtains new state-of-the-art performance on real-time instance segmentation and rotated object detection. We hope the experimental results can provide new insights into designing versatile real-time object detectors for many object recognition tasks.

<img src="https://user-images.githubusercontent.com/12907710/208070055-7233a3d8-955f-486a-82da-b714b3c3bbd6.png"/>
</div>

### Object Detection

|    Model    | size | box AP | Params(M) | FLOPS(G) | TRT-FP16-Latency(ms)<br>RTX3090 | TRT-FP16-Latency(ms)<br>T4 |                   Config                   |                                                                                                                                                Download                                                                                                                                                |

| :---------: | :--: | :----: | :-------: | :------: | :-----------------------------: | :------------------------: | :----------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| RTMDet-tiny | 640  |  41.1  |    4.8    |   8.1    |              0.98               |            2.34            | [config](./rtmdet_tiny_8xb32-300e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet_tiny_8xb32-300e_coco/rtmdet_tiny_8xb32-300e_coco_20220902_112414-78e30dcc.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet_tiny_8xb32-300e_coco/rtmdet_tiny_8xb32-300e_coco_20220902_112414.log.json) |

|  RTMDet-s   | 640  |  44.6  |   8.89    |   14.8   |              1.22               |            2.96            |  [config](./rtmdet_s_8xb32-300e_coco.py)   |       [model](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet_s_8xb32-300e_coco/rtmdet_s_8xb32-300e_coco_20220905_161602-387a891e.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet_s_8xb32-300e_coco/rtmdet_s_8xb32-300e_coco_20220905_161602.log.json)       |
|  RTMDet-m   | 640  |  49.4  |   24.71   |  39.27   |              1.62               |            6.41            |  [config](./rtmdet_m_8xb32-300e_coco.py)   |       [model](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet_m_8xb32-300e_coco/rtmdet_m_8xb32-300e_coco_20220719_112220-229f527c.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet_m_8xb32-300e_coco/rtmdet_m_8xb32-300e_coco_20220719_112220.log.json)       |

|  RTMDet-l   | 640  |  51.5  |   52.3    |  80.23   |              2.44               |           10.32            |  [config](./rtmdet_l_8xb32-300e_coco.py)   |       [model](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet_l_8xb32-300e_coco/rtmdet_l_8xb32-300e_coco_20220719_112030-5a0be7c4.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet_l_8xb32-300e_coco/rtmdet_l_8xb32-300e_coco_20220719_112030.log.json)       |
|  RTMDet-x   | 640  |  52.8  |   94.86   |  141.67  |              3.10               |           18.80            |  [config](./rtmdet_x_8xb32-300e_coco.py)   |       [model](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet_x_8xb32-300e_coco/rtmdet_x_8xb32-300e_coco_20220715_230555-cc79b9ae.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet_x_8xb32-300e_coco/rtmdet_x_8xb32-300e_coco_20220715_230555.log.json)       |

**Note**:

**Note**:

1. We implement a fast training version of RTMDet in [MMYOLO](https://github.com/open-mmlab/mmyolo). Its training speed is **2.6 times faster** and memory requirement is lower! Try it [here](https://github.com/open-mmlab/mmyolo/tree/main/configs/rtmdet)!
2. The inference speed of RTMDet is measured with TensorRT 8.4.3, cuDNN 8.2.0, FP16, batch size=1, and without NMS.
3. For a fair comparison, the config of bbox postprocessing is changed to be consistent with YOLOv5/6/7 after [PR#9494](https://github.com/open-mmlab/mmdetection/pull/9494), bringing about 0.1~0.3% AP improvement.

RTMDet-Ins is the state-of-the-art real-time instance segmentation on coco dataset:

|      Model      | size | box AP | mask AP | Params(M) | FLOPS(G) | TRT-FP16-Latency(ms) |                     Config                     |                                                                                                                                                        Download                                                                                                                                                        |
| :-------------: | :--: | :----: | :-----: | :-------: | :------: | :------------------: | :--------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

| RTMDet-Ins-tiny | 640  |  40.5  |  35.4   |    5.6    |   11.8   |         1.70         | [config](./rtmdet-ins_tiny_8xb32-300e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet-ins_tiny_8xb32-300e_coco/rtmdet-ins_tiny_8xb32-300e_coco_20221130_151727-ec670f7e.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet-ins_tiny_8xb32-300e_coco/rtmdet-ins_tiny_8xb32-300e_coco_20221130_151727.log.json) |
|  RTMDet-Ins-s   | 640  |  44.0  |  38.7   |   10.18   |   21.5   |         1.93         |  [config](./rtmdet-ins_s_8xb32-300e_coco.py)   |       [model](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet-ins_s_8xb32-300e_coco/rtmdet-ins_s_8xb32-300e_coco_20221121_212604-fdc5d7ec.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet-ins_s_8xb32-300e_coco/rtmdet-ins_s_8xb32-300e_coco_20221121_212604.log.json)       |

|  RTMDet-Ins-m   | 640  |  48.8  |  42.1   |   27.58   |  54.13   |         2.69         |  [config](./rtmdet-ins_m_8xb32-300e_coco.py)   |       [model](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet-ins_m_8xb32-300e_coco/rtmdet-ins_m_8xb32-300e_coco_20221123_001039-6eba602e.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet-ins_m_8xb32-300e_coco/rtmdet-ins_m_8xb32-300e_coco_20221123_001039.log.json)       |
|  RTMDet-Ins-l   | 640  |  51.2  |  43.7   |   57.37   |  106.56  |         3.68         |  [config](./rtmdet-ins_l_8xb32-300e_coco.py)   |       [model](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet-ins_l_8xb32-300e_coco/rtmdet-ins_l_8xb32-300e_coco_20221124_103237-78d1d652.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet-ins_l_8xb32-300e_coco/rtmdet-ins_l_8xb32-300e_coco_20221124_103237.log.json)       |

|  RTMDet-Ins-x   | 640  |  52.4  |  44.6   |   102.7   |  182.7   |         5.31         |  [config](./rtmdet-ins_x_8xb16-300e_coco.py)   |       [model](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet-ins_x_8xb16-300e_coco/rtmdet-ins_x_8xb16-300e_coco_20221124_111313-33d4595b.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet-ins_x_8xb16-300e_coco/rtmdet-ins_x_8xb16-300e_coco_20221124_111313.log.json)       |

**Note**:

1. The inference speed of RTMDet-Ins is measured on an NVIDIA 3090 GPU with TensorRT 8.4.3, cuDNN 8.2.0, FP16, batch size=1. Top 100 masks are kept and the post process latency is included.

### Rotated Object Detection

RTMDet-R achieves state-of-the-art on various remote sensing datasets.

[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/rtmdet-an-empirical-study-of-designing-real/one-stage-anchor-free-oriented-object-3)](https://paperswithcode.com/sota/one-stage-anchor-free-oriented-object-3?p=rtmdet-an-empirical-study-of-designing-real)

Models and configs of RTMDet-R are available in [MMRotate](https://github.com/open-mmlab/mmrotate/tree/1.x/configs/rotated_rtmdet).

|  Backbone   | pretrain |  Aug  | mmAP  | mAP50 | mAP75 | Params(M) | FLOPS(G) | TRT-FP16-Latency(ms) |                                                            Config                                                             |                                                                                                                                                                       Download                                                                                                                                                                       |

| RTMDet-tiny |    IN    |  RR   | 47.37 | 75.36 | 50.64 |   4.88    |  20.45   |         4.40         |        [config](https://github.com/open-mmlab/mmrotate/edit/1.x/configs/rotated_rtmdet/rotated_rtmdet_tiny-3x-dota.py)        |                             [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_tiny-3x-dota/rotated_rtmdet_tiny-3x-dota-9d821076.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_tiny-3x-dota/rotated_rtmdet_tiny-3x-dota_20221201_120814.json)                             |

| RTMDet-tiny |    IN    | MS+RR | 53.59 | 79.82 | 58.87 |   4.88    |  20.45   |         4.40         |      [config](https://github.com/open-mmlab/mmrotate/edit/1.x/configs/rotated_rtmdet/rotated_rtmdet_tiny-3x-dota_ms.py)       |                       [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_tiny-3x-dota_ms/rotated_rtmdet_tiny-3x-dota_ms-f12286ff.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_tiny-3x-dota_ms/rotated_rtmdet_tiny-3x-dota_ms_20221113_201235.log)                        |

|  RTMDet-s   |    IN    |  RR   | 48.16 | 76.93 | 50.59 |   8.86    |  37.62   |         4.86         |         [config](https://github.com/open-mmlab/mmrotate/edit/1.x/configs/rotated_rtmdet/rotated_rtmdet_s-3x-dota.py)          |                                   [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_s-3x-dota/rotated_rtmdet_s-3x-dota-11f6ccf5.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_s-3x-dota/rotated_rtmdet_s-3x-dota_20221124_081442.json)                                   |

|  RTMDet-s   |    IN    | MS+RR | 54.43 | 79.98 | 60.07 |   8.86    |  37.62   |         4.86         |        [config](https://github.com/open-mmlab/mmrotate/edit/1.x/configs/rotated_rtmdet/rotated_rtmdet_s-3x-dota_ms.py)        |                             [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_s-3x-dota_ms/rotated_rtmdet_s-3x-dota_ms-20ead048.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_s-3x-dota_ms/rotated_rtmdet_s-3x-dota_ms_20221113_201055.json)                             |

|  RTMDet-m   |    IN    |  RR   | 50.56 | 78.24 | 54.47 |   24.67   |  99.76   |         7.82         |         [config](https://github.com/open-mmlab/mmrotate/edit/1.x/configs/rotated_rtmdet/rotated_rtmdet_m-3x-dota.py)          |                                   [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_m-3x-dota/rotated_rtmdet_m-3x-dota-beeadda6.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_m-3x-dota/rotated_rtmdet_m-3x-dota_20221122_011234.json)                                   |

|  RTMDet-m   |    IN    | MS+RR | 55.00 | 80.26 | 61.26 |   24.67   |  99.76   |         7.82         |        [config](https://github.com/open-mmlab/mmrotate/edit/1.x/configs/rotated_rtmdet/rotated_rtmdet_m-3x-dota_ms.py)        |                             [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_m-3x-dota_ms/rotated_rtmdet_m-3x-dota_ms-c71eb375.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_m-3x-dota_ms/rotated_rtmdet_m-3x-dota_ms_20221122_011234.json)                             |

|  RTMDet-l   |    IN    |  RR   | 51.01 | 78.85 | 55.21 |   52.27   |  204.21  |        10.82         |         [config](https://github.com/open-mmlab/mmrotate/edit/1.x/configs/rotated_rtmdet/rotated_rtmdet_l-3x-dota.py)          |                                   [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_l-3x-dota/rotated_rtmdet_l-3x-dota-23992372.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_l-3x-dota/rotated_rtmdet_l-3x-dota_20221122_011241.json)                                   |

|  RTMDet-l   |    IN    | MS+RR | 55.52 | 80.54 | 61.47 |   52.27   |  204.21  |        10.82         |        [config](https://github.com/open-mmlab/mmrotate/edit/1.x/configs/rotated_rtmdet/rotated_rtmdet_l-3x-dota_ms.py)        |                             [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_l-3x-dota_ms/rotated_rtmdet_l-3x-dota_ms-2738da34.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_l-3x-dota_ms/rotated_rtmdet_l-3x-dota_ms_20221122_011241.json)                             |

|  RTMDet-l   |   COCO   | MS+RR | 56.74 | 81.33 | 63.45 |   52.27   |  204.21  |        10.82         | [config](https://github.com/open-mmlab/mmrotate/edit/1.x/configs/rotated_rtmdet/rotated_rtmdet_l-coco_pretrain-3x-dota_ms.py) | [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_l-coco_pretrain-3x-dota_ms/rotated_rtmdet_l-coco_pretrain-3x-dota_ms-06d248a2.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_l-coco_pretrain-3x-dota_ms/rotated_rtmdet_l-coco_pretrain-3x-dota_ms_20221113_202010.json) |

### Classification

We also provide the imagenet classification configs of the RTMDet backbone. Find more details in the [classification folder](./classification).

|    Model     | resolution | Params(M) | Flops(G) | Top-1 (%) | Top-5 (%) |                                                              Download                                                               |

| :----------: | :--------: | :-------: | :------: | :-------: | :-------: | :---------------------------------------------------------------------------------------------------------------------------------: |
| CSPNeXt-tiny |  224x224   |   2.73    |   0.34   |   69.44   |   89.45   |    [model](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/cspnext_rsb_pretrain/cspnext-tiny_imagenet_600e-3a2dd350.pth)     |
|  CSPNeXt-s   |  224x224   |   4.89    |   0.66   |   74.41   |   92.23   |      [model](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/cspnext_rsb_pretrain/cspnext-s_imagenet_600e-ea671761.pth)      |
|  CSPNeXt-m   |  224x224   |   13.05   |   1.93   |   79.27   |   94.79   | [model](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/cspnext_rsb_pretrain/cspnext-m_8xb256-rsb-a1-600e_in1k-ecb3bbd9.pth) |

|  CSPNeXt-l   |  224x224   |   27.16   |   4.19   |   81.30   |   95.62   | [model](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/cspnext_rsb_pretrain/cspnext-l_8xb256-rsb-a1-600e_in1k-6a760974.pth) |
|  CSPNeXt-x   |  224x224   |   48.85   |   7.76   |   82.10   |   95.69   | [model](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/cspnext_rsb_pretrain/cspnext-x_8xb256-rsb-a1-600e_in1k-b3f78edd.pth) |

```latex
}
```

</div>

## Deployment Tutorial

}
```

</div>

Here is a basic example of deploy RTMDet with [MMDeploy-1.x](https://github.com/open-mmlab/mmdeploy/tree/1.x).

### Step1. Install MMDeploy

Before starting the deployment, please make sure you install MMDetection and MMDeploy-1.x correctly.

- Install MMDetection, please refer to the [MMDetection installation guide](https://mmdetection.readthedocs.io/en/latest/get_started.html).
- Install MMDeploy-1.x, please refer to the [MMDeploy-1.x installation guide](https://mmdeploy.readthedocs.io/en/1.x/get_started.html#installation).

If you want to deploy RTMDet with ONNXRuntime, TensorRT, or other inference engine,
please make sure you have installed the corresponding dependencies and MMDeploy precompiled packages.

### Step2. Convert Model

After the installation, you can enjoy the model deployment journey starting from converting PyTorch model to backend model by running MMDeploy's `tools/deploy.py`.

The detailed model conversion tutorial please refer to the [MMDeploy document](https://mmdeploy.readthedocs.io/en/1.x/02-how-to-run/convert_model.html).
Here we only give the example of converting RTMDet.

MMDeploy supports converting dynamic and static models. Dynamic models support different input shape, but the inference speed is slower than static models.
To achieve the best performance, we suggest converting RTMDet with static setting.

- If you only want to use ONNX, please use [`configs/mmdet/detection/detection_onnxruntime_static.py`](https://github.com/open-mmlab/mmdeploy/blob/1.x/configs/mmdet/detection/detection_onnxruntime_static.py) as the deployment config.

- If you want to use TensorRT, please use [`configs/mmdet/detection/detection_tensorrt_static-640x640.py`](https://github.com/open-mmlab/mmdeploy/blob/1.x/configs/mmdet/detection/detection_tensorrt_static-640x640.py).

If you want to customize the settings in the deployment config for your requirements, please refer to [MMDeploy config tutorial](https://mmdeploy.readthedocs.io/en/1.x/02-how-to-run/write_config.html).

After preparing the deployment config, you can run the `tools/deploy.py` script to convert your model.
Here we take converting RTMDet-s to TensorRT as an example:

```shell
# go to the mmdeploy folder

# download RTMDet-s checkpoint
wget -P checkpoint https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet_s_8xb32-300e_coco/rtmdet_s_8xb32-300e_coco_20220905_161602-387a891e.pth

# run the command to start model conversion
  configs/mmdet/detection/detection_tensorrt_static-640x640.py \

  ${PATH_TO_MMDET}/configs/rtmdet/rtmdet_s_8xb32-300e_coco.py \
  checkpoint/rtmdet_s_8xb32-300e_coco_20220905_161602-387a891e.pth \
  demo/resources/det.jpg \
  --work-dir ./work_dirs/rtmdet \
  --device cuda:0 \
  --show
```

If the script runs successfully, you will see the following files:

```
|----work_dirs
     |----rtmdet
          |----end2end.onnx  # ONNX model
          |----end2end.engine  # TensorRT engine file
```

After this, you can check the inference results with MMDeploy Model Converter API:

```python

result = inference_model(
  model_cfg='${PATH_TO_MMDET}/configs/rtmdet/rtmdet_s_8xb32-300e_coco.py',
  deploy_cfg='${PATH_TO_MMDEPLOY}/configs/mmdet/detection/detection_tensorrt_static-640x640.py',
  backend_files=['work_dirs/rtmdet/end2end.engine'],
  img='demo/resources/det.jpg',

deploy_cfg='${PATH_TO_MMDEPLOY}/configs/mmdet/detection/detection_tensorrt_static-640x640.py',
  device='cuda:0')
```

#### Advanced Setting

To convert the model with TRT-FP16, you can enable the fp16 mode in your deploy config:

```python
        fp16_mode=True  # enable fp16
    ))
```

To reduce the end to end inference speed with the inference engine, we suggest you to adjust the post-processing setting of the model.
We set a very low score threshold during training and testing to achieve better COCO mAP.
However, in actual usage scenarios, a relatively high score threshold (e.g. 0.3) is usually used.

You can adjust the score threshold and the number of detection boxes in your model config according to the actual usage to reduce the time-consuming of post-processing.

```python
# in MMDetection config

```python
        nms_pre=1000,  # keep top-k score bboxes before nms
        score_thr=0.3,  # score threshold to filter bboxes
        nms=dict(type='nms', iou_threshold=0.65),
        max_per_img=100)  # only keep top-100 as the final results.
)
```

### Step3. Inference with SDK

We provide both Python and C++ inference API with MMDeploy SDK.

To use SDK, you need to dump the required info during converting the model. Just add `--dump-info` to the model conversion command:

```shell
  --work-dir ./work_dirs/rtmdet-sdk \
  --show \
  --dump-info  # dump sdk info
```

checkpoint/rtmdet_s_8xb32-300e_coco_20220905_161602-387a891e.pth \
  --show \
```

After running the command, it will dump 3 json files additionally for the SDK:

```
     |----rtmdet-sdk
          # json files for the SDK
          |----pipeline.json
          |----deploy.json
          |----detail.json
```

Here is a basic example of SDK Python API:

```python
from mmdeploy_python import Detector
import cv2

img = cv2.imread('demo/resources/det.jpg')

detector = Detector(model_path='work_dirs/rtmdet-sdk', device_name='cuda', device_id=0)
# run the inference
bboxes, labels, _ = detector(img)
# Filter the result according to threshold
indices = [i for i in range(len(bboxes))]

for index, bbox, label_id in zip(indices, bboxes, labels):
  [left, top, right, bottom], score = bbox[0:4].astype(int),  bbox[4]
  if score < 0.3:
      continue
  # draw bbox
  cv2.rectangle(img, (left, top), (right, bottom), (0, 255, 0))

cv2.imwrite('output_detection.png', img)
```

Here is a basic example of SDK C++ API:

```C++
#include <cstdlib>
#include <opencv2/opencv.hpp>

int main() {
  const char* device_name = "cuda";
  int device_id = 0;
  std::string model_path = "work_dirs/rtmdet-sdk";
  std::string image_path = "demo/resources/det.jpg";

  // 1. load model
  mmdeploy::Model model(model_path);
  // 2. create predictor
  mmdeploy::Detector detector(model, mmdeploy::Device{device_name, device_id});
  // 3. read image
  // 4. inference

// 2. create predictor
  auto dets = detector.Apply(img);
  // 5. deal with the result. Here we choose to visualize it
  for (int i = 0; i < dets.size(); ++i) {
    const auto& box = dets[i].bbox;
    fprintf(stdout, "box %d, left=%.2f, top=%.2f, right=%.2f, bottom=%.2f, label=%d, score=%.4f\n",
            i, box.left, box.top, box.right, box.bottom, dets[i].label_id, dets[i].score);
    if (bboxes[i].score < 0.3) {
      continue;
    }
    cv::rectangle(img, cv::Point{(int)box.left, (int)box.top},
                  cv::Point{(int)box.right, (int)box.bottom}, cv::Scalar{0, 255, 0});
  }
  cv::imwrite("output_detection.png", img);
  return 0;
}
```

To build C++ example, please add MMDeploy package in your CMake project as following:

```cmake

}
  return 0;
}
```

```cmake
```

#### Other languages

- [C# API Examples](https://github.com/open-mmlab/mmdeploy/tree/1.x/demo/csharp)
- [JAVA API Examples](https://github.com/open-mmlab/mmdeploy/tree/1.x/demo/java)

### Deploy RTMDet Instance Segmentation Model

We support RTMDet-Ins ONNXRuntime and TensorRT deployment after [MMDeploy v1.0.0rc2](https://github.com/open-mmlab/mmdeploy/tree/v1.0.0rc2). And its deployment process is almost consistent with the detection model.

#### Step1. Install MMDeploy >= v1.0.0rc2

Please refer to the [MMDeploy-1.x installation guide](https://mmdeploy.readthedocs.io/en/1.x/get_started.html#installation) to install the latest version.
Please remember to replace the pre-built package with the latest version.

The v1.0.0rc2 package can be downloaded from [v1.0.0rc2 release page](https://github.com/open-mmlab/mmdeploy/releases/tag/v1.0.0rc2).

Step2. Convert Model

This step has no difference with the previous tutorial. The only thing you need to change is switching to the RTMDet-Ins deploy config:

- If you want to use ONNXRuntime, please use [`configs/mmdet/instance-seg/instance-seg_rtmdet-ins_onnxruntime_static-640x640.py`](https://github.com/open-mmlab/mmdeploy/blob/dev-1.x/configs/mmdet/instance-seg/instance-seg_rtmdet-ins_onnxruntime_static-640x640.py) as the deployment config.
- If you want to use TensorRT, please use [`configs/mmdet/instance-seg/instance-seg_rtmdet-ins_tensorrt_static-640x640.py`](https://github.com/open-mmlab/mmdeploy/blob/dev-1.x/configs/mmdet/instance-seg/instance-seg_rtmdet-ins_tensorrt_static-640x640.py).

Here we take converting RTMDet-Ins-s to TensorRT as an example:

```shell

```shell

wget -P checkpoint https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet-ins_s_8xb32-300e_coco/rtmdet-ins_s_8xb32-300e_coco_20221121_212604-fdc5d7ec.pth

  configs/mmdet/instance-seg/instance-seg_rtmdet-ins_tensorrt_static-640x640.py \
  ${PATH_TO_MMDET}/configs/rtmdet/rtmdet-ins_s_8xb32-300e_coco.py \
  checkpoint/rtmdet-ins_s_8xb32-300e_coco_20221121_212604-fdc5d7ec.pth \
  --work-dir ./work_dirs/rtmdet-ins \
  --show
```

```
     |----rtmdet-ins
```

```
```

```python

  model_cfg='${PATH_TO_MMDET}/configs/rtmdet/rtmdet-ins_s_8xb32-300e_coco.py',
  deploy_cfg='${PATH_TO_MMDEPLOY}/configs/mmdet/instance-seg/instance-seg_rtmdet-ins_tensorrt_static-640x640.py',
  backend_files=['work_dirs/rtmdet-ins/end2end.engine'],
```

### Model Config

backend_files=['work_dirs/rtmdet-ins/end2end.engine'],
```

In MMDetection's config, we use `model` to set up detection algorithm components. In addition to neural network components such as `backbone`, `neck`, etc, it also requires `data_preprocessor`, `train_cfg`, and `test_cfg`. `data_preprocessor` is responsible for processing a batch of data output by dataloader. `train_cfg`, and `test_cfg` in the model config are for training and testing hyperparameters of the components.Taking RTMDet as an example, we will introduce each field in the config according to different function modules:

```python
    type='RTMDet',  # The name of detector

```python
        type='DetDataPreprocessor',  # The type of the data preprocessor. Refer to https://mmdetection.readthedocs.io/en/latest/api.html#mmdet.models.data_preprocessors.DetDataPreprocessor
        mean=[103.53, 116.28, 123.675],  # Mean values used to pre-training the pre-trained backbone models, ordered in R, G, B
        std=[57.375, 57.12, 58.395],  # Standard variance used to pre-training the pre-trained backbone models, ordered in R, G, B
        bgr_to_rgb=False,  # whether to convert image from BGR to RGB
        batch_augments=None),  # Batch-level augmentations
        type='CSPNeXt',  # The type of backbone network. Refer to https://mmdetection.readthedocs.io/en/latest/api.html#mmdet.models.backbones.CSPNeXt

backbone=dict(  # The config of backbone
        arch='P5',  # Architecture of CSPNeXt, from {P5, P6}. Defaults to P5
        expand_ratio=0.5,  # Ratio to adjust the number of channels of the hidden layer. Defaults to 0.5
        deepen_factor=1,  # Depth multiplier, multiply number of blocks in CSP layer by this amount. Defaults to 1.0
        widen_factor=1,  # Width multiplier, multiply number of channels in each layer by this amount. Defaults to 1.0
        channel_attention=True,  # Whether to add channel attention in each stage. Defaults to True
        norm_cfg=dict(type='SyncBN'),  # Dictionary to construct and config norm layer. Defaults to dict(type=’BN’, requires_grad=True)
        act_cfg=dict(type='SiLU', inplace=True)),  # Config dict for activation layer. Defaults to dict(type=’SiLU’)

act_cfg=dict(type='SiLU', inplace=True)),  # Config dict for activation layer. Defaults to dict(type=’SiLU’)
        type='CSPNeXtPAFPN',  # The type of neck is CSPNeXtPAFPN. Refer to https://mmdetection.readthedocs.io/en/latest/api.html#mmdet.models.necks.CSPNeXtPAFPN
        in_channels=[256, 512, 1024],  # Number of input channels per scale
        out_channels=256,  # Number of output channels (used at each scale)
        num_csp_blocks=3,  # Number of bottlenecks in CSPLayer. Defaults to 3
        expand_ratio=0.5,  # Ratio to adjust the number of channels of the hidden layer. Default: 0.5
        norm_cfg=dict(type='SyncBN'),  # Config dict for normalization layer. Default: dict(type=’BN’)
        act_cfg=dict(type='SiLU', inplace=True)),  # Config dict for activation layer. Default: dict(type=’Swish’)

act_cfg=dict(type='SiLU', inplace=True)),  # Config dict for activation layer. Default: dict(type=’Swish’)
        type='RTMDetSepBNHead',  # The type of bbox_head is RTMDetSepBNHead. RTMDetHead with separated BN layers and shared conv layers. Refer to https://mmdetection.readthedocs.io/en/latest/api.html#mmdet.models.dense_heads.RTMDetSepBNHead
        num_classes=80,  # Number of categories excluding the background category
        in_channels=256,  # Number of channels in the input feature map
        stacked_convs=2,  # Whether to share conv layers between stages. Defaults to True
        feat_channels=256,  # Feature channels of convolutional layers in the head
            type='MlvlPointGenerator',  # The methods use MlvlPointGenerator. Refer to https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/task_modules/prior_generators/point_generator.py#L92

offset=0,  # The offset of points, the value is normalized with corresponding stride. Defaults to 0.5
            strides=[8, 16, 32]),  # Strides of anchors in multiple feature levels in order (w, h)
        bbox_coder=dict(type='DistancePointBBoxCoder'),  # Distance Point BBox coder.This coder encodes gt bboxes (x1, y1, x2, y2) into (top, bottom, left,right) and decode it back to the original. Refer to https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/task_modules/coders/distance_point_bbox_coder.py#L9
            type='QualityFocalLoss',  # Type of loss for classification branch. Refer to https://mmdetection.readthedocs.io/en/latest/api.html#mmdet.models.losses.QualityFocalLoss
            use_sigmoid=True,  # Whether sigmoid operation is conducted in QFL. Defaults to True
            beta=2.0,  # The beta parameter for calculating the modulating factor. Defaults to 2.0

use_sigmoid=True,  # Whether sigmoid operation is conducted in QFL. Defaults to True
            loss_weight=1.0),  #  Loss weight of current loss
        loss_bbox=dict(  # Config of loss function for the regression branch
            type='GIoULoss',  # Type of loss. Refer to https://mmdetection.readthedocs.io/en/latest/api.html#mmdet.models.losses.GIoULoss
            loss_weight=2.0),  # Loss weight of the regression branch
        with_objectness=False,  # Whether to add an objectness branch. Defaults to True
        exp_on_reg=True,  # Whether to use .exp() in regression
        share_conv=True,  # Whether to share conv layers between stages. Defaults to True
        pred_kernel_size=1,  # Kernel size of prediction layer. Defaults to 1
        norm_cfg=dict(type='SyncBN'),  # Config dict for normalization layer. Defaults to dict(type='BN', momentum=0.03, eps=0.001)

norm_cfg=dict(type='SyncBN'),  # Config dict for normalization layer. Defaults to dict(type='BN', momentum=0.03, eps=0.001)
        act_cfg=dict(type='SiLU', inplace=True)),  # Config dict for activation layer. Defaults to dict(type='SiLU')
    train_cfg=dict(  # Config of training hyperparameters for ATSS
        assigner=dict(  # Config of assigner
            type='DynamicSoftLabelAssigner',   # Type of assigner. DynamicSoftLabelAssigner computes matching between predictions and ground truth with dynamic soft label assignment. Refer to https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/task_modules/assigners/dynamic_soft_label_assigner.py#L40
            topk=13),  # Select top-k predictions to calculate dynamic k best matches for each gt. Defaults to 13
        allowed_border=-1,  # The border allowed after padding for valid anchors
        pos_weight=-1,  # The weight of positive samples during training
        debug=False),  # Whether to set the debug mode

pos_weight=-1,  # The weight of positive samples during training
    test_cfg=dict(  # Config for testing hyperparameters for ATSS
        nms_pre=30000,  # The number of boxes before NMS
        min_bbox_size=0,  # The allowed minimal box size
        score_thr=0.001,  # Threshold to filter out boxes
        nms=dict(  # Config of NMS in the second stage
            type='nms',  # Type of NMS
            iou_threshold=0.65),  # NMS threshold
        max_per_img=300),  # Max number of detections of each image
)
```

# Learn to train and test

## Train

This section will show how to train existing models on supported datasets.
The following training environments are supported:

- CPU
- single node multiple GPUs
- multiple nodes

You can also manage jobs with Slurm.

Important:

- You can change the evaluation interval during training by modifying the `train_cfg` as
  `train_cfg = dict(val_interval=10)`. That means evaluating the model every 10 epochs.
- The default learning rate in all config files is for 8 GPUs.
  According to the [Linear Scaling Rule](https://arxiv.org/abs/1706.02677),
  you need to set the learning rate proportional to the batch size if you use different GPUs or images per GPU,
  e.g., `lr=0.01` for 8 GPUs * 1 img/gpu and lr=0.04 for 16 GPUs * 2 imgs/gpu.
- During training, log files and checkpoints will be saved to the working directory,
  which is specified by CLI argument `--work-dir`. It uses `./work_dirs/CONFIG_NAME` as default.

- If you want the mixed precision training, simply specify CLI argument `--amp`.

#### 1. Train on CPU

The model is default put on cuda device.
Only if there are no cuda devices, the model will be put on cpu.
So if you want to train the model on CPU, you need to `export CUDA_VISIBLE_DEVICES=-1` to disable GPU visibility first.
More details in [MMEngine](https://github.com/open-mmlab/mmengine/blob/ca282aee9e402104b644494ca491f73d93a9544f/mmengine/runner/runner.py#L849-L850).

```shell script
CUDA_VISIBLE_DEVICES=-1 python tools/train.py ${CONFIG_FILE} [optional arguments]
```

An example of training the MOT model QDTrack on CPU:

CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/qdtrack/qdtrack_faster-rcnn_r50_fpn_8xb2-4e_mot17halftrain_test-mot17halfval.py
```

```

#### 2. Train on single GPU

If you want to train the model on single GPU, you can directly use the `tools/train.py` as follows.

```

You can use `export CUDA_VISIBLE_DEVICES=$GPU_ID` to select the GPU.

An example of training the MOT model QDTrack on single GPU:

CUDA_VISIBLE_DEVICES=2 python tools/train.py configs/qdtrack/qdtrack_faster-rcnn_r50_fpn_8xb2-4e_mot17halftrain_test-mot17halfval.py
```

#### 3. Train on single node multiple GPUs

bash ./tools/dist_train.sh ${CONFIG_FILE} ${GPU_NUM} [optional arguments]
```

If you would like to launch multiple jobs on a single machine,

```

e.g., 2 jobs of 4-GPU training on a machine with 8 GPUs,

For example, you can set the port in commands as follows.

```

An example of training the MOT model QDTrack on single node multiple GPUs:

bash ./tools/dist_train.sh configs/qdtrack/qdtrack_faster-rcnn_r50_fpn_8xb2-4e_mot17halftrain_test-mot17halfval.py 8
```

#### 4. Train on multiple nodes

```

```

```

#### 5. Train with Slurm

On a cluster managed by Slurm, you can use `slurm_train.sh` to spawn training jobs.
It supports both single-node and multi-node training.

bash ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} ${CONFIG_FILE} ${WORK_DIR} ${GPUS}
```

An example of training the MOT model QDTrack with Slurm:

```

PORT=29501 \
GPUS_PER_NODE=8 \
SRUN_ARGS="--quotatype=reserved" \
bash ./tools/slurm_train.sh \
mypartition \
mottrack
configs/qdtrack/qdtrack_faster-rcnn_r50_fpn_8xb2-4e_mot17halftrain_test-mot17halfval.py
./work_dirs/QDTrack \
8
```

## Test

This section will show how to test existing models on supported datasets.
The following testing environments are supported:

- CPU

- In MOT, some algorithms like `DeepSORT`, `SORT`, `StrongSORT` need load the weight of the `reid` and the weight of the `detector` separately.
  Other algorithms such as `ByteTrack`, `OCSORT` and `QDTrack` don't need. So we provide `--checkpoint`, `--detector` and `--reid` to load weights.

Other algorithms such as `ByteTrack`, `OCSORT` and `QDTrack` don't need. So we provide `--checkpoint`, `--detector` and `--reid` to load weights.
- We provide two ways to evaluate and test models, video_basede test and image_based test. some algorithms like `StrongSORT`, `Mask2former` only support
  video_based test. if your GPU memory can't fit the entire video, you can switch test way by set sampler type.
  For example:
  video_based test: `sampler=dict(type='DefaultSampler', shuffle=False, round_up=False)`
  image_based test: `sampler=dict(type='TrackImgSampler')`
- You can set the results saving path by modifying the key `outfile_prefix` in evaluator.
  For example, `val_evaluator = dict(outfile_prefix='results/sort_mot17')`.
  Otherwise, a temporal file will be created and will be removed after evaluation.
- If you just want the formatted results without evaluation, you can set `format_only=True`.

Otherwise, a temporal file will be created and will be removed after evaluation.
  For example, `test_evaluator = dict(type='MOTChallengeMetric', metric=['HOTA', 'CLEAR', 'Identity'], outfile_prefix='sort_mot17_results', format_only=True)`

#### 1. Test on CPU

So if you want to test the model on CPU, you need to `export CUDA_VISIBLE_DEVICES=-1` to disable GPU visibility first.

CUDA_VISIBLE_DEVICES=-1 python tools/test_tracking.py ${CONFIG_FILE} [optional arguments]
```

An example of testing the MOT model SORT on CPU:

```

CUDA_VISIBLE_DEVICES=-1 python tools/test_tracking.py configs/sort/sort_faster-rcnn_r50_fpn_8xb2-4e_mot17halftrain_test-mot17halfval.py --detector ${CHECKPOINT_FILE}
```

#### 2. Test on single GPU

If you want to test the model on single GPU, you can directly use the `tools/test_tracking.py` as follows.

python tools/test_tracking.py ${CONFIG_FILE} [optional arguments]
```

An example of testing the MOT model QDTrack on single GPU:

CUDA_VISIBLE_DEVICES=2 python tools/test_tracking.py configs/qdtrack/qdtrack_faster-rcnn_r50_fpn_8xb2-4e_mot17halftrain_test-mot17halfval.py --detector ${CHECKPOINT_FILE}
```

#### 3. Test on single node multiple GPUs

```

We provide `tools/dist_test_tracking.sh` to launch testing on multiple GPUs.

bash ./tools/dist_test_tracking.sh ${CONFIG_FILE} ${GPU_NUM} [optional arguments]
```

An example of testing the MOT model DeepSort on single node multiple GPUs:

bash ./tools/dist_test_tracking.sh configs/qdtrack/qdtrack_faster-rcnn_r50_fpn_8xb2-4e_mot17halftrain_test-mot17halfval.py 8 --detector ${CHECKPOINT_FILE} --reid ${CHECKPOINT_FILE}
```

#### 4. Test on multiple nodes

You can test on multiple nodes, which is similar with "Train on multiple nodes".

#### 5. Test with Slurm

On a cluster managed by Slurm, you can use `slurm_test_tracking.sh` to spawn testing jobs.
It supports both single-node and multi-node testing.

[GPUS=${GPUS}] bash tools/slurm_test_tracking.sh ${PARTITION} ${JOB_NAME} ${CONFIG_FILE} [optional arguments]
```

```

An example of testing the VIS model Mask2former with Slurm:

GPUS=8
bash tools/slurm_test_tracking.sh \
vis \
configs/mask2former_vis/mask2former_r50_8xb2-8e_youtubevis2021.py \
--checkpoint ${CHECKPOINT_FILE}
```

# 优化器的迁移

我们已经将[MMGeneration 1.x](https://github.com/open-mmlab/mmgeneration/tree/1.x)合并至MMagic。以下是针对MMGeneration中优化器的迁移事项。

在0.x版中，MMGeneration使用PyTorch自带的优化器，其只提供了通用参数优化，而在1.x版中，我们则使用了MMEngine提供的`OptimizerWrapper`。

对比PyTorch自带的`Optimizer`，`OptimizerWrapper`可以支持如下功能：

- `OptimizerWrapper.update_params`在一个单一的函数中就实现了`zero_grad`，`backward`和`step`
- 支持梯度自动累积
- 提供一个名为`OptimizerWrapper.optim_context`的上下文管理器来封装前向进程，`optim_context`会根据当前更新迭代数目来自动调用`torch.no_sync`，在AMP(Auto Mixed Precision)训练中，`autocast`也会在`optim_context`中被调用。

对GAN模型，生成器和鉴别器采用不同的优化器和训练策略。要使GAN模型的`train_step`函数签名和其它模型的保持一致，我们使用从`OptimizerWrapper`继承下来的`OptimWrapperDict`来封装生成器和鉴别器的优化器，为了便于该流程的自动化MMagic实现了`MultiOptimWrapperContructor`构造器。如你想训练GAN模型，那么应该在你的配置中指定该构造器。

如下是0.x版和1.x版的配置对比

<thead>
  <tr>
  </tr>
</thead>
<tbody>
<tr>

```python

<tr>
  </tr>
</thead>
<tbody>
<tr>

```python
```

</td>

```python
```

</td>

</tr>
</tbody>

</table>

> 注意，在1.x版中，MMGeneration使用`OptimWrapper`来实现梯度累加，这就会导致在0.x版和1.x版之间，`discriminator_steps`配置（用于在多次更新鉴别器之后更新一次生成器的训练技巧）与梯度累加均出现不一致问题。

- 在0.x版中，我们在配置里使用`disc_steps`，`gen_steps`和`batch_accumulation_steps` 。`disc_steps`和`batch_accumulation_steps`会根据`train_step`的调用次数来进行统计（亦即dataloader中数据的读取次数）。因此鉴别器的一段连续性更新次数为`disc_steps // batch_accumulation_steps`。且对于生成器，`gen_steps`是生成器实际的一段连续性更新次数

- 但在1.x版中，我们在配置里则使用了`discriminator_steps`，`generator_steps` 和`accumulative_counts`。`discriminator_steps`和`generator_steps`指的是自身在更新其它模型之前的一段连续性的更新次数
  以BigGAN-128配置为例。

<thead>
  <tr>
  </tr>
</thead>
<tbody>
<tr>

```python

num_classes=1000,

# 连续性更新鉴别器`disc_steps // batch_accumulation_steps = 8 // 8 = 1`次
# 连续性更新生成器`gen_steps = 1`次
# 生成器与鉴别器在每次更新之前执行`batch_accumulation_steps = 8`次梯度累加
```

</td>

```python

with_shared_embedding=True,
    # 连续性更新鉴别器`discriminator_steps = 1`次
    # 连续性更新生成器`generator_steps = 1`次

        # 生成器在每次更新之前执行`accumulative_counts = 8`次梯度累加
        # 鉴别器在每次更新之前执行`accumulative_counts = 8`次梯度累加

optimizer=dict(type='Adam', lr=0.0001, betas=(0.0, 0.999), eps=1e-6)),
```

</td>

</tr>
</tbody>

</table>

|device|gpu|api version|driver version|squeezenet|mobilenetssd|yolov3|
|intel-i7-7700|Intel(R) HD Graphics 630 (Kaby Lake GT2)|1.1.90|18.3.4|y|y|y|
|GTX-1060|GeForce GTX 1060 3GB|1.1.95|418.172.0|y|y|y|
|AMD-Radeon R9 M290X|AMD RADV PITCAIRN (LLVM 7.0.1)|1.1.70|18.3.4|y|y|y|
|iphone-5s|Apple A7 GPU|1.0.82|0.2.1825|y|y|y|
|huawei-nexus6p|Adreno (TM) 430|1.0.49|35.601.2388|y|y|y
|vivo-y1731ca|Adreno (TM) 505|1.0.61|37.845.1429|y|n|n|
|vivo-y85a|Adreno (TM) 506|1.0.61|2.944.3349|y|n|n|
|vivo-x9s|Adreno (TM) 510|1.0.61|42.917.1172|y|y|y|
|meizu-15|Adreno (TM) 512|1.0.38|29.189.223|n|n|n|
|chuizi-jianguo-pro2|Adreno (TM) 512|1.0.38|21.219.2615|n|n|n|
|xiaomi-note3|Adreno (TM) 512|1.0.38|39.369.2305|n|n|n|
|oppo-r11|Adreno (TM) 512|1.0.38|42.977.756|n|n|n|
|xiaomi-6x|Adreno (TM) 512|1.0.61|14.322.3739|y|y|y|
|oppo-r11s+|Adreno (TM) 512|1.0.61|35.1004.3936|y|y|y|
|vivo-x20a|Adreno (TM) 512|1.0.61|43.10.3141|y|y|y|

|vivo-v1816a|Adreno (TM) 512|1.0.61|43.10.3141|y|y|y|
|vivo-z1|Adreno (TM) 512|1.0.61|43.10.3141|y|y|y|
|xiaomi-redmi-note5|Adreno (TM) 512|1.0.61|63.219.2354|y|y|y|
|google-pixel|Adreno (TM) 530|1.1.87|512.354.0|y|y|y|
|nubia-z17|Adreno (TM) 540|1.0.38|1.28.32|n|n|n|
|samsung-galaxys8+|Adreno (TM) 540|1.0.61|29.896.3583|y|y|y|
|oneplus-5t|Adreno (TM) 540|1.0.61|18.1023.2233|y|y|y|
|google-pixel2|Adreno (TM) 540|1.1.66|512.313.0|y|y|y|
|essential-ph-1|Adreno (TM) 540|1.1.66|512.319.0|y|y|y|
|vivo-x23|Adreno (TM) 615|1.0.66|33.870.3328|y|y|y|
|vivo-v1813ba|Adreno (TM) 615|1.0.66|33.870.3328|y|y|y|
|xiaomi-8se|Adreno (TM) 616|1.0.66|30.913.18|y|y|y|
|vivo-nex-a|Adreno (TM) 616|1.0.66|33.870.3328|y|y|y|
|xiaomi-mix2s|Adreno (TM) 630|1.0.61|4.91.2976|y|y|y|
|heisha-SKR-A0|Adreno (TM) 630|1.0.61|36.173.3586|y|y|y|

|heisha-SKR-A0|Adreno (TM) 630|1.0.66|47.448.1532|y|y|y|
|oneplus-6|Adreno (TM) 630|1.1.66|512.324.0|y|y|y|
|vivo-iQOO|Adreno (TM) 640|1.1.87|512.361.0|y|y|y|
|meitu-m8s|Mali-T880|1.0.14|500.910.1017|n|n|n|
|huawei-p10|Mali-G71|1.0.53|151.949.2145|n|n|n|
|huawei-mate9|Mali-G71|1.0.53|151.949.2145|n|n|n|
|oppo-a73|Mali-G71|1.0.47|575.795.1934|n|n|n|
|vivo-y97|Mali-G72|1.0.58|240.537.3580|n|n|n|
|huawei-mate10|Mali-G72|1.0.66|14.0.0|y|y|y|
|huawei-v10|Mali-G72|1.0.66|14.0.0|y|y|y|
|huawei-vce-al00|Mali-G72|1.0.66|14.0.0|y|y|y|
|huawei-mate20|Mali-G76|1.0.66|14.0.0|y|y|y|
|huawei-pct-al10|Mali-G76|1.0.66|14.0.0|y|y|y|

# Preparing Jester

```BibTeX
@InProceedings{Materzynska_2019_ICCV,
  author = {Materzynska, Joanna and Berger, Guillaume and Bax, Ingo and Memisevic, Roland},
  title = {The Jester Dataset: A Large-Scale Video Dataset of Human Gestures},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops},
}
```

For basic dataset information, you can refer to the dataset [website](https://developer.qualcomm.com/software/ai-datasets/jester).
Before we start, please make sure that the directory is located at `$MMACTION2/tools/data/jester/`.

First of all, you have to sign in and download annotations to `$MMACTION2/data/jester/annotations` on the official [website](https://developer.qualcomm.com/software/ai-datasets/jester).

## Step 2. Prepare RGB Frames

Since the [jester website](https://developer.qualcomm.com/software/ai-datasets/jester) doesn't provide the original video data and only extracted RGB frames are available, you have to directly download RGB frames from [jester website](https://developer.qualcomm.com/software/ai-datasets/jester).

You can download all RGB frame parts on [jester website](https://developer.qualcomm.com/software/ai-datasets/jester) to `$MMACTION2/data/jester/` and use the following command to extract.

```shell
cd $MMACTION2/data/jester/
cat 20bn-jester-v1-?? | tar zx
cd $MMACTION2/tools/data/jester/
```

For users who only want to use RGB frames, you can skip to step 5 to generate file lists in the format of rawframes. Since the prefix of official JPGs is "%05d.jpg" (e.g., "00001.jpg"),
we add `"filename_tmpl='{:05}.jpg'"` to the dict of `data.train`, `data.val` and `data.test` in the config files related with jester like this:

```
    videos_per_gpu=16,

```
        data_prefix=data_root,
    val=dict(
        data_prefix=data_root_val,
        pipeline=val_pipeline),
```

## Step 3. Extract Flow

This part is **optional** if you only want to use RGB frames.

```shell
mkdir /mnt/SSD/jester_extracted/
ln -s /mnt/SSD/jester_extracted/ ../../../data/jester/rawframes
```

Then, you can run the following script to extract optical flow based on RGB frames.

```shell
bash extract_flow.sh
```

## Step 4. Encode Videos

You can run the following script to encode videos.

```shell
bash encode_videos.sh
```

## Step 5. Generate File List

You can run the follow script to generate file list in the format of rawframes and videos.

```shell

```shell
```

```shell
bash generate_{rawframes, videos}_filelist.sh
```

## Step 5. Check Directory Structure

After the whole data process for Jester preparation,
you will get the rawframes (RGB + Flow), and annotation files for Jester.

In the context of the whole project (for Jester only), the folder structure will look like:

```
mmaction2
├── tools
├── data
│   ├── jester
│   │   ├── jester_{train,val}_list_rawframes.txt
│   │   ├── jester_{train,val}_list_videos.txt
│   |   |   ├── 1.mp4
│   |   |   ├── 2.mp4
│   |   |   ├──...
│   |   ├── rawframes
│   |   |   ├── 1
│   |   |   |   ├── 00001.jpg
│   |   |   |   ├── 00002.jpg

│   |   |   ├──...
│   |   |   ├── 1
│   |   |   ├── 2
│   |   |   ├── ...

```

For training and evaluating on Jester, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).

}
```

<details>
<summary align="right"><a href="https://ieeexplore.ieee.org/abstract/document/6130513/">AFLW (ICCVW'2011)</a></summary>

```bibtex
@inproceedings{koestinger2011annotated,
  title={Annotated facial landmarks in the wild: A large-scale, real-world database for facial landmark localization},
  author={Koestinger, Martin and Wohlhart, Paul and Roth, Peter M and Bischof, Horst},
  booktitle={2011 IEEE international conference on computer vision workshops (ICCV workshops)},
  pages={2144--2151},
}
```

Results on AFLW dataset

booktitle={2011 IEEE international conference on computer vision workshops (ICCV workshops)},
}
```

The model is trained on AFLW train and evaluated on AFLW full and frontal.

| Arch                                              | Input Size | NME<sub>*full*</sub> | NME<sub>*frontal*</sub> |                       ckpt                        |                        log                        |
| :------------------------------------------------ | :--------: | :------------------: | :---------------------: | :-----------------------------------------------: | :-----------------------------------------------: |

| [pose_hrnetv2_w18_dark](/configs/face_2d_keypoint/topdown_heatmap/aflw/td-hm_hrnetv2-w18_dark-8xb64-60e_aflw-256x256.py) |  256x256   |         1.35         |          1.19           | [ckpt](https://download.openmmlab.com/mmpose/face/darkpose/hrnetv2_w18_aflw_256x256_dark-219606c0_20210125.pth) | [log](https://download.openmmlab.com/mmpose/face/darkpose/hrnetv2_w18_aflw_256x256_dark_20210125.log.json) |

# K-Net

> [K-Net: Towards Unified Image Segmentation](https://arxiv.org/abs/2106.14855)

<a href="https://github.com/ZwwWayne/K-Net/">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.23.0/mmseg/models/decode_heads/knet_head.py#L392">Code Snippet</a>

Semantic, instance, and panoptic segmentations have been addressed using different and specialized frameworks despite their underlying connections. This paper presents a unified, simple, and effective framework for these essentially similar tasks. The framework, named K-Net, segments both instances and semantic categories consistently by a group of learnable kernels, where each kernel is responsible for generating a mask for either a potential instance or a stuff class. To remedy the difficulties of distinguishing various instances, we propose a kernel update strategy that enables each kernel dynamic and conditional on its meaningful group in the input image. K-Net can be trained in an end-to-end manner with bipartite matching, and its training and inference are naturally NMS-free and box-free. Without bells and whistles, K-Net surpasses all previous published state-of-the-art single-model results of panoptic segmentation on MS COCO test-dev split and semantic segmentation on ADE20K val split with 55.2% PQ and 54.3% mIoU, respectively. Its instance segmentation performance is also on par with Cascade Mask R-CNN on MS COCO with 60%-90% faster inference speeds. Code and models will be released at [this https URL](https://github.com/ZwwWayne/K-Net/).

<img src="https://user-images.githubusercontent.com/24582831/157008300-9f40905c-b8e8-4a2a-9593-c1177fa35b2c.png" width="90%"/>
</div>

### ADE20K

| Method           | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device | mIoU  | mIoU(ms+flip) | config                                                                                                                                  | download                                                                                                                                                                                                                                                                                                                                                                                                         |

| KNet + FCN       | R-50-D8  | 512x512   | 80000   | 7.01     | 19.24          | V100   | 43.60 | 45.12         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/knet/knet-s3_r50-d8_fcn_8xb2-adamw-80k_ade20k-512x512.py)       | [model](https://download.openmmlab.com/mmsegmentation/v0.5/knet/knet_s3_fcn_r50-d8_8x2_512x512_adamw_80k_ade20k/knet_s3_fcn_r50-d8_8x2_512x512_adamw_80k_ade20k_20220228_043751-abcab920.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/knet/knet_s3_fcn_r50-d8_8x2_512x512_adamw_80k_ade20k/knet_s3_fcn_r50-d8_8x2_512x512_adamw_80k_ade20k_20220228_043751.log.json)                         |

| KNet + PSPNet    | R-50-D8  | 512x512   | 80000   | 6.98     | 20.04          | V100   | 44.18 | 45.58         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/knet/knet-s3_r50-d8_pspnet_8xb2-adamw-80k_ade20k-512x512.py)    | [model](https://download.openmmlab.com/mmsegmentation/v0.5/knet/knet_s3_pspnet_r50-d8_8x2_512x512_adamw_80k_ade20k/knet_s3_pspnet_r50-d8_8x2_512x512_adamw_80k_ade20k_20220228_054634-d2c72240.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/knet/knet_s3_pspnet_r50-d8_8x2_512x512_adamw_80k_ade20k/knet_s3_pspnet_r50-d8_8x2_512x512_adamw_80k_ade20k_20220228_054634.log.json)             |

| KNet + DeepLabV3 | R-50-D8  | 512x512   | 80000   | 7.42     | 12.10          | V100   | 45.06 | 46.11         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/knet/knet-s3_r50-d8_deeplabv3_8xb2-adamw-80k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/knet/knet_s3_deeplabv3_r50-d8_8x2_512x512_adamw_80k_ade20k/knet_s3_deeplabv3_r50-d8_8x2_512x512_adamw_80k_ade20k_20220228_041642-00c8fbeb.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/knet/knet_s3_deeplabv3_r50-d8_8x2_512x512_adamw_80k_ade20k/knet_s3_deeplabv3_r50-d8_8x2_512x512_adamw_80k_ade20k_20220228_041642.log.json) |

| KNet + UperNet   | R-50-D8  | 512x512   | 80000   | 7.34     | 17.11          | V100   | 43.45 | 44.07         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/knet/knet-s3_r50-d8_upernet_8xb2-adamw-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/knet/knet_s3_upernet_r50-d8_8x2_512x512_adamw_80k_ade20k/knet_s3_upernet_r50-d8_8x2_512x512_adamw_80k_ade20k_20220304_125657-215753b0.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/knet/knet_s3_upernet_r50-d8_8x2_512x512_adamw_80k_ade20k/knet_s3_upernet_r50-d8_8x2_512x512_adamw_80k_ade20k_20220304_125657.log.json)         |

| KNet + UperNet   | Swin-T   | 512x512   | 80000   | 7.57     | 15.56          | V100   | 45.84 | 46.27         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/knet/knet-s3_swin-t_upernet_8xb2-adamw-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/knet/knet_s3_upernet_swin-t_8x2_512x512_adamw_80k_ade20k/knet_s3_upernet_swin-t_8x2_512x512_adamw_80k_ade20k_20220303_133059-7545e1dc.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/knet/knet_s3_upernet_swin-t_8x2_512x512_adamw_80k_ade20k/knet_s3_upernet_swin-t_8x2_512x512_adamw_80k_ade20k_20220303_133059.log.json)         |

| KNet + UperNet   | Swin-L   | 512x512   | 80000   | 13.5     | 8.29           | V100   | 52.05 | 53.24         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/knet/knet-s3_swin-l_upernet_8xb2-adamw-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/knet/knet_s3_upernet_swin-l_8x2_512x512_adamw_80k_ade20k/knet_s3_upernet_swin-l_8x2_512x512_adamw_80k_ade20k_20220303_154559-d8da9a90.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/knet/knet_s3_upernet_swin-l_8x2_512x512_adamw_80k_ade20k/knet_s3_upernet_swin-l_8x2_512x512_adamw_80k_ade20k_20220303_154559.log.json)         |

| KNet + UperNet   | Swin-L   | 640x640   | 80000   | 13.54    | 8.29           | V100   | 52.21 | 53.34         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/knet/knet-s3_swin-l_upernet_8xb2-adamw-80k_ade20k-640x640.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/knet/knet_s3_upernet_swin-l_8x2_640x640_adamw_80k_ade20k/knet_s3_upernet_swin-l_8x2_640x640_adamw_80k_ade20k_20220301_220747-8787fc71.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/knet/knet_s3_upernet_swin-l_8x2_640x640_adamw_80k_ade20k/knet_s3_upernet_swin-l_8x2_640x640_adamw_80k_ade20k_20220301_220747.log.json)         |

Note:

- All experiments of K-Net are implemented with 8 V100 (32G) GPUs with 2 samplers per GPU.

# Citation

```bibtex
@inproceedings{zhang2021knet,
    title={{K-Net: Towards} Unified Image Segmentation},
    author={Wenwei Zhang and Jiangmiao Pang and Kai Chen and Chen Change Loy},
    year={2021},
    booktitle={NeurIPS},
}
```

```bibtex
}
```

<details>

```bibtex
}
```

booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
}
```

| [pose_hrnet_w32_dark](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_hrnet-w32_dark-8xb64-210e_mpii-256x256.py) |  256x256   | 0.904 |  0.354   | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_mpii_256x256_dark-f1601c5b_20200927.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_mpii_256x256_dark_20200927.log.json) |

| [pose_hrnet_w48_dark](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_hrnet-w48_dark-8xb64-210e_mpii-256x256.py) |  256x256   | 0.905 |  0.360   | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_mpii_256x256_dark-0decd39f_20200927.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_mpii_256x256_dark_20200927.log.json) |

# Preparing SRSDD Dataset

```bibtex
@ARTICLE{SRSDD2021,
author = {S. Lei, D. Lu and X. Qiu},
title = {SRSDD-v1.0: A high-resolution SAR rotation ship
detection dataset},
month = {Dec.},
year = {2021},
number={24},
pages={5104},
}
```

## Download SRSDD dataset

The SRSDD dataset can be downloaded from [Google drive](https://drive.google.com/file/d/1QtCjih1ChOmG-TOPUTlsL3WbMh0L-1zp/view?usp=sharing).

```none
mmrotate
├── tools
├── data
│   ├── srsdd
│   │   ├── test
```

Please change `data_root` in `configs/_base_/datasets/srsdd.py` to `data/srsdd/`.

# HRFormer: High-Resolution Vision Transformer for Dense Predict

<details>

```bibtex
}
```

We present a High-Resolution Transformer (HRFormer) that learns high-resolution representations for dense
prediction tasks, in contrast to the original Vision Transformer that produces low-resolution representations
and has high memory and computational cost. We take advantage of the multi-resolution parallel design

introduced in high-resolution convolutional networks (HRNet), along with local-window self-attention
that performs self-attention over small non-overlapping image windows, for improving the memory and
computation efficiency. In addition, we introduce a convolution into the FFN to exchange information
across the disconnected image windows. We demonstrate the effectiveness of the HighResolution Transformer
on both human pose estimation and semantic segmentation tasks, e.g., HRFormer outperforms Swin
transformer by 1.3 AP on COCO pose estimation with 50% fewer parameters and 30% fewer FLOPs.
Code is available at: https://github.com/HRNet/HRFormer

<img src="https://user-images.githubusercontent.com/28900607/155838218-a6aa12b5-5855-45ed-922e-1cfe74f63027.png">
</div>

# Contribution Guide

OpenMMLab welcomes everyone who is interested in contributing to our projects and accepts contribution in the form of PR.

## What is PR

`PR` is the abbreviation of `Pull Request`. Here's the definition of `PR` in the [official document](https://docs.github.com/en/github/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests) of Github.

```
Pull requests let you tell others about changes you have pushed to a branch in a repository on GitHub. Once a pull request is opened, you can discuss and review the potential changes with collaborators and add follow-up commits before your changes are merged into the base branch.
```

## Basic Workflow

1. Get the most recent codebase

```

2. Checkout a new branch from `dev-1.x` or `dev` branch, depending on the version of the codebase you want to contribute to. The main differences between `dev-1.x` and `dev` is that `dev-1.x` depends on MMEngine additionally and it's the main branch we maintains. We strongly recommend you pull request based on more advanced `dev-1.x` branch.

3. Commit your changes ([Don't forget to use pre-commit hooks!](#3-commit-your-changes))

4. Push your changes and create a PR

5. Discuss and review your code

6. Merge your branch to `dev-1.x` / `dev` branch

## Procedures in detail

### 1. Get the most recent codebase

- When you work on your first PR

  Fork the OpenMMLab repository: click the **fork** button at the top right corner of Github page
  ![avatar](https://user-images.githubusercontent.com/34888372/224920532-dc11f696-1175-436a-8c0f-1966f5ca33d1.png)

  Clone forked repository to local

  ```bash

![avatar](https://user-images.githubusercontent.com/34888372/224920532-dc11f696-1175-436a-8c0f-1966f5ca33d1.png)

  ```bash
  git clone git@github.com:XXX/mmdetection3d.git
  ```

  Add source repository to upstream

  ```bash
  git remote add upstream git@github.com:open-mmlab/mmdetection3d
  ```

- After your first PR

  Checkout the latest branch of the local repository and pull the latest branch of the source repository. Here we assume that you are working on the `dev-1.x` branch.

  ```bash
  git checkout dev-1.x
  git pull upstream dev-1.x
  ```

### 2. Checkout a new branch from the `dev-1.x` / `dev` branch

```bash
git checkout -b branchname
```

```{tip}
To make commit history clear, we strongly recommend you checkout the `dev-1.x` branch before creating a new branch.
```

### 3. Commit your changes

- If you are a first-time contributor, please install and initialize pre-commit hooks from the repository root directory first.

  ```bash

```

  ```bash
  pip install -U pre-commit
  pre-commit install
  ```

- Commit your changes as usual. Pre-commit hooks will be triggered to stylize your code before each commit.

  ```bash
  # coding
  git add [files]
  git commit -m 'messages'
  ```

  ```{note}
  Sometimes your code may be changed by pre-commit hooks. In this case, please remember to re-stage the modified files and commit again.
  ```

### 4. Push your changes to the forked repository and create a PR

- Push the branch to your forked remote repository

  ```bash
  git push origin branchname
  ```

- Create a PR
  ![avatar](https://user-images.githubusercontent.com/34888372/224922548-69455db9-68d1-4d92-a007-afcd2814b1c1.png)

```bash
  ```

- Revise PR message template to describe your motivation and modifications made in this PR. You can also link the related issue to the PR manually in the PR message (For more information, checkout the [official guidance](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue)).

- Specifically, if you are contributing to `dev-1.x`, you will have to change the base branch of the PR to `dev-1.x` in the PR page, since the default base branch is `master`.

  ![avatar](https://user-images.githubusercontent.com/34888372/224923009-1d611a30-0bfc-4fe5-93a2-96cc88a18886.png)

- You can also ask a specific person to review the changes you've proposed.

### 5. Discuss and review your code

- Modify your codes according to reviewers' suggestions and then push your changes.

### 6.  Merge your branch to the `dev-1.x` / `dev` branch and delete the branch

- After the PR is merged by the maintainer, you can delete the branch you created in your forked repository.

  ```bash
  git branch -d branchname # delete local branch
  git push origin --delete branchname # delete remote branch
  ```

## PR Specs

- Good: Add a parameter to box head to support custom conv-layer number

# Customize Runtime Settings

## Customize optimization settings

Optimization related configuration is now all managed by `optim_wrapper` which usually has three fields: `optimizer`, `paramwise_cfg`, `clip_grad`. Please refer to [OptimWrapper](https://mmengine.readthedocs.io/en/latest/tutorials/optim_wrapper.html) for more details. See the example below, where `AdamW` is used as an `optimizer`, the learning rate of the backbone is reduced by a factor of 10, and gradient clipping is added.

```python
    # optimizer
        eps=1e-8,

    # Parameter-level learning rate and weight decay settings
        },

    # gradient clipping
```

custom_keys={
        },

```

### Customize optimizer supported by PyTorch

We already support to use all the optimizers implemented by PyTorch, and the only modification is to change the `optimizer` field in `optim_wrapper` field of config files. For example, if you want to use `Adam` (note that the performance could drop a lot), the modification could be as the following:

```python
```

To modify the learning rate of the model, the users only need to modify the `lr` in `optimizer`. The users can directly set arguments following the [API doc](https://pytorch.org/docs/stable/optim.html?highlight=optim#module-torch.optim) of PyTorch.

### Customize self-implemented optimizer

#### 1. Define a new optimizer

A customized optimizer could be defined as following:

Assume you want to add a optimizer named `MyOptimizer`, which has arguments `a`, `b`, and `c`.
You need to create a new directory named `mmdet3d/engine/optimizers`, and then implement the new optimizer in a file, e.g., in `mmdet3d/engine/optimizers/my_optimizer.py`:

```python

        pass
```

#### 2. Add the optimizer to registry

To find the above module defined above, this module should be imported into the main namespace at first. There are two options to achieve it.

- Modify `mmdet3d/engine/optimizers/__init__.py` to import it.

  The newly defined module should be imported in `mmdet3d/engine/optimizers/__init__.py` so that the registry will find the new module and add it:

The newly defined module should be imported in `mmdet3d/engine/optimizers/__init__.py` so that the registry will find the new module and add it:

  ```python
  ```

- Use `custom_imports` in the config to manually import it.

  ```python
  ```

  The module `mmdet3d.engine.optimizers.my_optimizer` will be imported at the beginning of the program and the class `MyOptimizer` is then automatically registered.
  Note that only the package containing the class `MyOptimizer` should be imported.
  `mmdet3d.engine.optimizers.my_optimizer.MyOptimizer` **cannot** be imported directly.

  Actually users can use a totally different file directory structure with this importing method, as long as the module root is located in `PYTHONPATH`.

Then you can use `MyOptimizer` in `optimizer` field in `optim_wrapper` field of config files. In the configs, the optimizers are defined by the field `optimizer` like the following:

```python
```

To use your own optimizer, the field can be changed to:

```python
```

### Customize optimizer wrapper constructor

Some models may have some parameter-specific settings for optimization, e.g. weight decay for BatchNorm layers.
The users can do those fine-grained parameter tuning through customizing optimizer wrapper constructor.

```python

```python

        pass

```

The default optimizer wrapper constructor is implemented [here](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/optimizer/default_constructor.py#L18), which could also serve as a template for the new optimizer wrapper constructor.

### Additional settings

Tricks not implemented by the optimizer should be implemented through optimizer wrapper constructor (e.g., set parameter-wise learning rates) or hooks. We list some common settings that could stabilize the training or accelerate the training. Feel free to create PR, issue for more settings.

- __Use gradient clip to stabilize training__:
  Some models need gradient clip to clip the gradients to stabilize the training process. An example is as below:

  ```python
  ```

  If your config inherits the base config which already sets the `optim_wrapper`, you might need `_delete_=True` to override the unnecessary settings. See the [config documentation](https://mmdetection3d.readthedocs.io/en/dev-1.x/user_guides/config.html) for more details.

- __Use momentum schedule to accelerate model convergence__:

  We support momentum scheduler to modify model's momentum according to learning rate, which could make the model converge in a faster way.
  Momentum scheduler is usually used with LR scheduler, for example, the following config is used in [3D detection](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/schedules/cyclic-20e.py) to accelerate convergence.
  For more details, please refer to the implementation of [CosineAnnealingLR](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py#L43) and [CosineAnnealingMomentum](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/momentum_scheduler.py#L71).

  ```python
      # learning rate scheduler
      # During the first 8 epochs, learning rate increases from 0 to lr * 10
      # during the next 12 epochs, learning rate decreases from lr * 10 to lr * 1e-4
      dict(

# During the first 8 epochs, learning rate increases from 0 to lr * 10
      dict(
          T_max=8,
          begin=0,
          end=8,
      dict(
          T_max=12,
          begin=8,
          end=20,
      # momentum scheduler
      # During the first 8 epochs, momentum increases from 0 to 0.85 / 0.95
      # during the next 12 epochs, momentum increases from 0.85 / 0.95 to 1
      dict(
          T_max=8,
          begin=0,
          end=8,
      dict(

eta_min=0.85 / 0.95,
          begin=0,
          end=8,
      dict(
          T_max=12,
          begin=8,
          end=20,
  ]
  ```

## Customize training schedules

By default we use step learning rate with 1x schedule, this calls [`MultiStepLR`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py#L144) in MMEngine.
We support many other learning rate schedule [here](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py), such as `CosineAnnealingLR` and `PolyLR` schedules. Here are some examples:

- Poly schedule:

  ```python
      dict(
          begin=0,
          end=8,
  ```

```python
      dict(
          begin=0,
          end=8,
  ```

- CosineAnnealing schedule:

  ```python
      dict(
          T_max=8,
          begin=0,
          end=8,
  ```

## Customize train loop

By default, `EpochBasedTrainLoop` is used in `train_cfg` and validation is done after every train epoch, as follows:

```python
```

Actually, both [`IterBasedTrainLoop`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L185) and [`EpochBasedTrainLoop`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L18) support dynamic interval, see the following example:

```python

```python
# Before 365001th iteration, we do evaluation every 5000 iterations.
# After 365000th iteration, we do evaluation every 368750 iterations,
# which means that we do evaluation at the end of training.

```

## Customize hooks

### Customize self-implemented hooks

#### 1. Implement a new hook

MMEngine provides many useful [hooks](https://mmengine.readthedocs.io/en/latest/tutorials/hook.html), but there are some occasions when the users might need to implement a new hook. MMDetection3D supports customized hooks in training based on MMEngine after v1.1.0rc0. Thus the users could implement a hook directly in mmdet3d or their mmdet3d-based codebases and use the hook by only modifying the config in training.

Here we give an example of creating a new hook in mmdet3d and using it in training.

```python

                          runner,

                         runner,
```

runner,
```

Depending on the functionality of the hook, users need to specify what the hook will do at each stage of the training in `before_run`, `after_run`, `before_train`, `after_train`, `before_train_epoch`, `after_train_epoch`, `before_train_iter`, and `after_train_iter`. There are more points where hooks can be inserted, refer to [base hook class](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/hook.py#L9) for more details.

#### 2. Register the new hook

Then we need to make `MyHook` imported. Assuming the file is in `mmdet3d/engine/hooks/my_hook.py`, there are two ways to do that:

- Modify `mmdet3d/engine/hooks/__init__.py` to import it.

  The newly defined module should be imported in `mmdet3d/engine/hooks/__init__.py` so that the registry will find the new module and add it:

  ```python
  ```

  ```python
  ```

#### 3. Modify the config

```python
]
```

You can also set the priority of the hook by adding key `priority` to `'NORMAL'` or `'HIGHEST'` as below:

```python
]
```

By default the hook's priority is set as `NORMAL` during registration.

### Use hooks implemented in MMDetection3D

dict(type='MyHook', a=a_value, b=b_value, priority='NORMAL')
]
```

If the hook is already implemented in MMDetection3D, you can directly modify the config to use the hook as below.

#### Example: `DisableObjectSampleHook`

We implement a customized hook named [DisableObjectSampleHook](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/engine/hooks/disable_object_sample_hook.py) to disable `ObjectSample` augmentation during training after specified epoch.

We can set it in the config file if needed:

```python
```

### Modify default runtime hooks

There are some common hooks that are registered through `default_hooks`, they are

- `IterTimerHook`: A hook that logs 'data_time' for loading data and 'time' for a model training step.

- `LoggerHook`: A hook that collects logs from different components of `Runner` and writes them to terminal, json file, tensorboard and wandb etc.
- `ParamSchedulerHook`: A hook that updates some hyper-parameters in optimizer, e.g., learning rate and momentum.
- `CheckpointHook`: A hook that saves checkpoints periodically.
- `DistSamplerSeedHook`: A hook that sets the seed for sampler and batch_sampler.
- `Det3DVisualizationHook`: A hook used to visualize validation and testing process prediction results.

`IterTimerHook`, `ParamSchedulerHook` and `DistSamplerSeedHook` are simple and no need to be modified usually, so here we reveal what we can do with `LoggerHook`, `CheckpointHook` and `Det3DVisualizationHook`.

Except saving checkpoints periodically, [`CheckpointHook`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L18) provides other options such as `max_keep_ckpts`, `save_optimizer` and etc. The users could set `max_keep_ckpts` to only save small number of checkpoints or decide whether to store state dict of optimizer by `save_optimizer`. More details of the arguments are [here](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L18).

```python
```

The `LoggerHook` enables setting intervals. Detailed instructions can be found in the [docstring](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/logger_hook.py#L19).

```python
```

```python
```

`Det3DVisualizationHook` use `DetLocalVisualizer` to visualize prediction results, and `Det3DLocalVisualizer` current supports different backends, e.g., `TensorboardVisBackend` and `WandbVisBackend` (see [docstring](https://github.com/open-mmlab/mmengine/blob/main/mmengine/visualization/vis_backend.py) for more details). The users could add multi backends to do visualization as follows.

```python

```

# UPerNet

> [Unified Perceptual Parsing for Scene Understanding](https://arxiv.org/pdf/1807.10221.pdf)

<a href="https://github.com/CSAILVision/unifiedparsing">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/decode_heads/uper_head.py#L13">Code Snippet</a>

Humans recognize the visual world at multiple levels: we effortlessly categorize scenes and detect objects inside, while also identifying the textures and surfaces of the objects along with their different compositional parts. In this paper, we study a new task called Unified Perceptual Parsing, which requires the machine vision systems to recognize as many visual concepts as possible from a given image. A multi-task framework called UPerNet and a training strategy are developed to learn from heterogeneous image annotations. We benchmark our framework on Unified Perceptual Parsing and show that it is able to effectively segment a wide range of concepts from images. The trained networks are further applied to discover visual knowledge in natural scenes. Models are available at [this https URL](https://github.com/CSAILVision/unifiedparsing).

<img src="https://user-images.githubusercontent.com/24582831/142903077-44e8e0da-7276-4bda-bd2b-0df1680ca845.png" width="70%"/>
</div>

| Method  | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                        | download                                                                                                                                                                                                                                                                                                                                           |

| UPerNet | R-50     | 512x1024  |   40000 | 6.4      | 4.25           | V100   | 77.10 |         78.37 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/upernet/upernet_r50_4xb2-40k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r50_512x1024_40k_cityscapes/upernet_r50_512x1024_40k_cityscapes_20200605_094827-aa54cb54.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r50_512x1024_40k_cityscapes/upernet_r50_512x1024_40k_cityscapes_20200605_094827.log.json)     |

| UPerNet | R-101    | 512x1024  |   40000 | 7.4      | 3.79           | V100   | 78.69 |         80.11 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/upernet/upernet_r101_4xb2-40k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r101_512x1024_40k_cityscapes/upernet_r101_512x1024_40k_cityscapes_20200605_094933-ebce3b10.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r101_512x1024_40k_cityscapes/upernet_r101_512x1024_40k_cityscapes_20200605_094933.log.json) |

| UPerNet | R-50     | 769x769   |   40000 | 7.2      | 1.76           | V100   | 77.98 |         79.70 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/upernet/upernet_r50_4xb2-40k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r50_769x769_40k_cityscapes/upernet_r50_769x769_40k_cityscapes_20200530_033048-92d21539.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r50_769x769_40k_cityscapes/upernet_r50_769x769_40k_cityscapes_20200530_033048.log.json)         |

| UPerNet | R-101    | 769x769   |   40000 | 8.4      | 1.56           | V100   | 79.03 |         80.77 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/upernet/upernet_r101_4xb2-40k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r101_769x769_40k_cityscapes/upernet_r101_769x769_40k_cityscapes_20200530_040819-83c95d01.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r101_769x769_40k_cityscapes/upernet_r101_769x769_40k_cityscapes_20200530_040819.log.json)     |

| UPerNet | R-50     | 512x1024  |   80000 | -        | -              | V100   | 78.19 |         79.19 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/upernet/upernet_r50_4xb2-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r50_512x1024_80k_cityscapes/upernet_r50_512x1024_80k_cityscapes_20200607_052207-848beca8.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r50_512x1024_80k_cityscapes/upernet_r50_512x1024_80k_cityscapes_20200607_052207.log.json)     |

| UPerNet | R-101    | 512x1024  |   80000 | -        | -              | V100   | 79.40 |         80.46 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/upernet/upernet_r101_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r101_512x1024_80k_cityscapes/upernet_r101_512x1024_80k_cityscapes_20200607_002403-f05f2345.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r101_512x1024_80k_cityscapes/upernet_r101_512x1024_80k_cityscapes_20200607_002403.log.json) |

| UPerNet | R-50     | 769x769   |   80000 | -        | -              | V100   | 79.39 |         80.92 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/upernet/upernet_r50_4xb2-80k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r50_769x769_80k_cityscapes/upernet_r50_769x769_80k_cityscapes_20200607_005107-82ae7d15.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r50_769x769_80k_cityscapes/upernet_r50_769x769_80k_cityscapes_20200607_005107.log.json)         |

| UPerNet | R-101    | 769x769   |   80000 | -        | -              | V100   | 80.10 |         81.49 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/upernet/upernet_r101_4xb2-80k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r101_769x769_80k_cityscapes/upernet_r101_769x769_80k_cityscapes_20200607_001014-082fc334.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r101_769x769_80k_cityscapes/upernet_r101_769x769_80k_cityscapes_20200607_001014.log.json)     |

### ADE20K

### ADE20K

| Method  | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                    | download                                                                                                                                                                                                                                                                                                                           |

| UPerNet | R-50     | 512x512   |   80000 | 8.1      | 23.40          | V100   | 40.70 |         41.81 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/upernet/upernet_r50_4xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r50_512x512_80k_ade20k/upernet_r50_512x512_80k_ade20k_20200614_144127-ecc8377b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r50_512x512_80k_ade20k/upernet_r50_512x512_80k_ade20k_20200614_144127.log.json)         |

| UPerNet | R-101    | 512x512   |   80000 | 9.1      | 20.34          | V100   | 42.91 |         43.96 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/upernet/upernet_r101_4xb4-80k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r101_512x512_80k_ade20k/upernet_r101_512x512_80k_ade20k_20200614_185117-32e4db94.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r101_512x512_80k_ade20k/upernet_r101_512x512_80k_ade20k_20200614_185117.log.json)     |

| UPerNet | R-50     | 512x512   |  160000 | -        | -              | V100   | 42.05 |         42.78 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/upernet/upernet_r50_4xb4-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r50_512x512_160k_ade20k/upernet_r50_512x512_160k_ade20k_20200615_184328-8534de8d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r50_512x512_160k_ade20k/upernet_r50_512x512_160k_ade20k_20200615_184328.log.json)     |

| UPerNet | R-101    | 512x512   |  160000 | -        | -              | V100   | 43.82 |         44.85 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/upernet/upernet_r101_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r101_512x512_160k_ade20k/upernet_r101_512x512_160k_ade20k_20200615_161951-91b32684.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r101_512x512_160k_ade20k/upernet_r101_512x512_160k_ade20k_20200615_161951.log.json) |

| Method  | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                     | download                                                                                                                                                                                                                                                                                                                               |

| UPerNet | R-50     | 512x512   |   20000 | 6.4      | 23.17          | V100   | 74.82 |         76.35 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/upernet/upernet_r50_4xb4-20k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r50_512x512_20k_voc12aug/upernet_r50_512x512_20k_voc12aug_20200617_165330-5b5890a7.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r50_512x512_20k_voc12aug/upernet_r50_512x512_20k_voc12aug_20200617_165330.log.json)     |

| UPerNet | R-101    | 512x512   |   20000 | 7.5      | 19.98          | V100   | 77.10 |         78.29 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/upernet/upernet_r101_4xb4-20k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r101_512x512_20k_voc12aug/upernet_r101_512x512_20k_voc12aug_20200617_165629-f14e7f27.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r101_512x512_20k_voc12aug/upernet_r101_512x512_20k_voc12aug_20200617_165629.log.json) |

| UPerNet | R-50     | 512x512   |   40000 | -        | -              | V100   | 75.92 |         77.44 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/upernet/upernet_r50_4xb4-40k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r50_512x512_40k_voc12aug/upernet_r50_512x512_40k_voc12aug_20200613_162257-ca9bcc6b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r50_512x512_40k_voc12aug/upernet_r50_512x512_40k_voc12aug_20200613_162257.log.json)     |

| UPerNet | R-101    | 512x512   |   40000 | -        | -              | V100   | 77.43 |         78.56 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/upernet/upernet_r101_4xb4-40k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r101_512x512_40k_voc12aug/upernet_r101_512x512_40k_voc12aug_20200613_163549-e26476ac.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/upernet/upernet_r101_512x512_40k_voc12aug/upernet_r101_512x512_40k_voc12aug_20200613_163549.log.json) |

```bibtex
@inproceedings{xiao2018unified,
  title={Unified perceptual parsing for scene understanding},
  author={Xiao, Tete and Liu, Yingcheng and Zhou, Bolei and Jiang, Yuning and Sun, Jian},
  pages={418--434},
}
```

<details>

```bibtex
}
```

<details>

```bibtex

title={Learning to Estimate 3D Hand Pose from Single RGB Images},
}
```

| [pose_mobilenet_v2](/configs/hand_2d_keypoint/topdown_heatmap/rhd2d/td-hm_mobilenetv2_8xb64-210e_rhd2d-256x256.py) |  256x256   |  0.985  | 0.883 | 2.79 | [ckpt](https://download.openmmlab.com/mmpose/hand/mobilenetv2/mobilenetv2_rhd2d_256x256-85fa02db_20210330.pth) | [log](https://download.openmmlab.com/mmpose/hand/mobilenetv2/mobilenetv2_rhd2d_256x256_20210330.log.json) |

# Deep high-resolution representation learning for visual recognition

<details>

```bibtex
}
```

High-resolution representations are essential for position-sensitive vision problems, such as human pose estimation, semantic segmentation, and object detection. Existing state-of-the-art frameworks first encode the input image as a low-resolution representation through a subnetwork that is formed by connecting high-to-low resolution convolutions in series (e.g., ResNet, VGGNet), and then recover the high-resolution representation from the encoded low-resolution representation. Instead, our proposed network, named as High-Resolution Network (HRNet), maintains high-resolution representations through the whole process. There are two key characteristics: (i) Connect the high-to-low resolution convolution streams in parallel and (ii) repeatedly exchange the information across resolutions. The benefit is that the resulting representation is semantically richer and spatially more precise. We show the superiority of the proposed HRNet in a wide range of applications, including human pose estimation, semantic segmentation, and object detection, suggesting that the HRNet is a stronger backbone for computer vision problems.

<img src="https://user-images.githubusercontent.com/15977946/146519239-b3416e9a-0f3a-493a-9cc5-2d257b7fae01.png">
</div>

# Knowledge Distillation Based on MMRazor

Knowledge Distillation is a classic model compression method. The core idea is to "imitate" a teacher model (or multi-model ensemble) with better performance and more complex structure by guiding a lightweight student model, improving the performance of the student model without changing its structure. [MMRazor](https://github.com/open-mmlab/mmrazor) is a model compression toolkit for model slimming and AutoML, which supports several KD algorithms. In this project, we take TSM-MobileNetV2 as an example to show how to use MMRazor to perform knowledge distillation on action recognition models. You could refer to more [MMRazor](https://github.com/open-mmlab/mmrazor) for more model compression algorithms.

This is an implementation of MMRazor Knowledge Distillation Application, we provide action recognition configs and models for MMRazor.

## Usage

## Usage

- [MMRazor v1.0.0](https://github.com/open-mmlab/mmrazor/tree/v1.0.0) or higher

There are two install modes:

Option (a). Install as a Python package

```shell
mim install "mmrazor>=1.0.0"
```

Option (b). Install from source

```shell
git clone https://github.com/open-mmlab/mmrazor.git
cd mmrazor
```

Please refer to [Get Started](https://mmaction2.readthedocs.io/en/latest/get_started/installation.html) to install MMAction2.

```shell
```

```shell
```

Prepare the Kinetics400 dataset according to the [instruction](https://github.com/open-mmlab/mmaction2/blob/main/tools/data/kinetics/README.md).

```shell
```

```bash
mim train mmrazor configs/kd_logits_tsm-res50_tsm-mobilenetv2_8xb16_k400.py
```

```bash
mim train mmrazor configs/kd_logits_tsm-res50_tsm-mobilenetv2_8xb16_k400.py --launcher pytorch --gpus 8
```

```bash
mim train mmrazor configs/kd_logits_tsm-res50_tsm-mobilenetv2_8xb16_k400.py --launcher slurm \

```

```bash
```

Please convert the knowledge distillation checkpoint to student-only checkpoint with following commands, you will get a checkpoint with a '\_student.pth' suffix under the same directory as the original checkpoint. Then take the student-only checkpoint for testing.

```bash
mim run mmrazor convert_kd_ckpt_to_student $CHECKPOINT
```

```bash
mim test mmaction tsm_imagenet-pretrained-mobilenetv2_8xb16-1x1x8-100e_kinetics400-rgb.py --checkpoint $CHECKPOINT
```

```bash
mim test mmaction tsm_imagenet-pretrained-mobilenetv2_8xb16-1x1x8-100e_kinetics400-rgb.py --checkpoint $CHECKPOINT --launcher pytorch --gpus 8
```

```bash

```

```bash
mim test mmaction tsm_imagenet-pretrained-mobilenetv2_8xb16-1x1x8-100e_kinetics400-rgb.py --checkpoint $CHECKPOINT --launcher slurm \
```

| Location |   Dataset    |    Teacher     |      Student      |     Acc     | Acc(T) | Acc(S) |        Config         | Download                                                                      |
| :------: | :----------: | :------------: | :---------------: | :---------: | :----: | :----: | :-------------------: | :---------------------------------------------------------------------------- |

|  logits  | Kinetics-400 | [TSM-ResNet50] | [TSM-MobileNetV2] | 69.60(+0.9) | 73.22  | 68.71  | [config][distill_tsm] | [teacher][tsm_r50_pth] \| [model][distill_pth_tsm] \| [log][distill_log_tsm]  |
|  logits  | Kinetics-400 |   [TSN-Swin]   |  [TSN-ResNet50]   | 75.54(+1.4) | 79.22  | 74.12  | [config][distill_tsn] | [teacher][tsn_swin_pth] \| [model][distill_pth_tsn] \| [log][distill_log_tsn] |

```latex
@article{huang2022knowledge,
  title={Knowledge Distillation from A Stronger Teacher},
  author={Huang, Tao and You, Shan and Wang, Fei and Qian, Chen and Xu, Chang},
  journal={arXiv preprint arXiv:2205.10536},
}
```

author={Huang, Tao and You, Shan and Wang, Fei and Qian, Chen and Xu, Chang},
}
```

[distill_log_tsm]: https://download.openmmlab.com/mmaction/v1.0/projects/knowledge_distillation/kd_logits_tsm-res50_tsm-mobilenetv2_8xb16_k400/kd_logits_tsm-res50_tsm-mobilenetv2_8xb16_k400.log
[distill_log_tsn]: https://download.openmmlab.com/mmaction/v1.0/projects/knowledge_distillation/kd_logits_tsn-swin_tsn-r50_1x1x8_k400/kd_logits_tsn-swin_tsn-r50_1x1x8_k400.log
[distill_pth_tsm]: https://download.openmmlab.com/mmaction/v1.0/projects/knowledge_distillation/kd_logits_tsm-res50_tsm-mobilenetv2_8xb16_k400/kd_logits_tsm-res50_tsm-mobilenetv2_8xb16_k400_20230517-c3e8aa0d.pth
[distill_pth_tsn]: https://download.openmmlab.com/mmaction/v1.0/projects/knowledge_distillation/kd_logits_tsn-swin_tsn-r50_1x1x8_k400/kd_logits_tsn-swin_tsn-r50_1x1x8_k400_student_20230530-f938d404.pth

# Conditional DETR

> [Conditional DETR for Fast Training Convergence](https://arxiv.org/abs/2108.06152)

The DETR approach applies the transformer encoder and decoder architecture to object detection and achieves promising performance. In this paper, we handle the critical issue, slow training convergence, and present a conditional cross-attention mechanism for fast DETR training. Our approach is motivated by that the cross-attention in DETR relies highly on the content embeddings and that the spatial embeddings make minor contributions, increasing the need for high-quality content embeddings and thus increasing the training difficulty.

<img src="https://github.com/Atten4Vis/ConditionalDETR/blob/main/.github/attention-maps.png?raw=true"/>
</div>

</div>

Our conditional DETR learns a conditional spatial query from the decoder embedding for decoder multi-head cross-attention. The benefit is that through the conditional spatial query, each cross-attention head is able to attend to a band containing a distinct region, e.g., one object extremity or a region inside the object box (Figure 1). This narrows down the spatial range for localizing the distinct regions for object classification and box regression, thus relaxing the dependence on the content embeddings and easing the training. Empirical results show that conditional DETR converges 6.7x faster for the backbones R50 and R101 and 10x faster for stronger backbones DC5-R50 and DC5-R101.

<img src="https://github.com/Atten4Vis/ConditionalDETR/raw/main/.github/conditional-detr.png" width="48%"/>

<img src="https://github.com/Atten4Vis/ConditionalDETR/raw/main/.github/convergence-curve.png" width="48%"/>
</div>

We provide the config files and models for Conditional DETR: [Conditional DETR for Fast Training Convergence](https://arxiv.org/abs/2108.06152).

| Backbone |      Model       | Lr schd | Mem (GB) | Inf time (fps) | box AP |                      Config                       |                                                                                                                                                                        Download                                                                                                                                                                        |

| :------: | :--------------: | :-----: | :------: | :------------: | :----: | :-----------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|   R-50   | Conditional DETR |   50e   |          |                |  41.1  | [config](./conditional-detr_r50_8xb2-50e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/conditional_detr/conditional-detr_r50_8xb2-50e_coco/conditional-detr_r50_8xb2-50e_coco_20221121_180202-c83a1dc0.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/conditional_detr/conditional-detr_r50_8xb2-50e_coco/conditional-detr_r50_8xb2-50e_coco_20221121_180202.log.json) |

```latex

```latex
@inproceedings{meng2021-CondDETR,
  title       = {Conditional DETR for Fast Training Convergence},
  author      = {Meng, Depu and Chen, Xiaokang and Fan, Zejia and Zeng, Gang and Li, Houqiang and Yuan, Yuhui and Sun, Lei and Wang, Jingdong},
  booktitle   = {Proceedings of the IEEE International Conference on Computer Vision (ICCV)},
  year        = {2021}
}
```

}
```

<details>
<summary align="right"><a href="https://arxiv.org/abs/1710.03740">FP16 (ArXiv'2017)</a></summary>

```bibtex
@article{micikevicius2017mixed,
  title={Mixed precision training},
  author={Micikevicius, Paulius and Narang, Sharan and Alben, Jonah and Diamos, Gregory and Elsen, Erich and Garcia, David and Ginsburg, Boris and Houston, Michael and Kuchaiev, Oleksii and Venkatesh, Ganesh and others},
  journal={arXiv preprint arXiv:1710.03740},
}
```

<details>

```bibtex

| [pose_resnet_50_fp16](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_res50_fp16-8xb64-210e_coco-256x192.py) |  256x192   | 0.716 |      0.898      |      0.798      | 0.772 |      0.937      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res50_fp16-8xb64-210e_coco-256x192-463da051_20220927.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res50_fp16-8xb64-210e_coco-256x192_20220927.log) |

# Build for Android

- [Build for Android](#build-for-android)
  - [Build From Source](#build-from-source)
    - [Install Toolchains](#install-toolchains)
    - [Install Dependencies](#install-dependencies)
      - [Install Dependencies for SDK](#install-dependencies-for-sdk)
    - [Build MMDeploy](#build-mmdeploy)
      - [Build SDK and Demos](#build-sdk-and-demos)

MMDeploy provides cross compile for android platform.

Model converter is executed on linux platform, and SDK is executed on android platform.

Here are two steps for android build.

1. Build model converter on linux, please refer to [How to build linux](linux-x86_64.md)

2. Build SDK using android toolchain on linux.

This doc is only for how to build SDK using android toolchain on linux.

## Build From Source

### Install Toolchains

- cmake

- cmake

  **Make sure cmake version >= 3.14.0**. If not, you can follow instructions below to install cmake 3.20.0. For more versions of cmake, please refer to [cmake website](https://cmake.org/install).

  ```bash
  ```

- ANDROID NDK 19+

  **Make sure android ndk version >= 19.0**. If not, you can follow instructions below to install android ndk r23c. For more versions of android ndk, please refer to [android ndk website](https://developer.android.com/ndk/downloads).

  ```bash
  wget https://dl.google.com/android/repository/android-ndk-r23c-linux.zip
  unzip android-ndk-r23c-linux.zip
  cd android-ndk-r23c

```bash
  export NDK_PATH=${PWD}
  ```

### Install Dependencies

#### Install Dependencies for SDK

You can skip this chapter if only interested in model converter.

<table>
<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
    <td>
export OPENCV_VERSION=4.6.0
wget https://github.com/opencv/opencv/releases/download/${OPENCV_VERSION}/opencv-${OPENCV_VERSION}-android-sdk.zip
unzip opencv-${OPENCV_VERSION}-android-sdk.zip
export OPENCV_ANDROID_SDK_DIR=${PWD}/OpenCV-android-sdk
    </td>

</tr>
  <tr>
    <td>A high-performance neural network inference computing framework supporting for android.</br>

    </td>

</tr>
  <tr>
  <b> Now, MMDeploy supports 20220721 and has to use <code>git clone</code> to download it. For supported android ABI, see <a href='https://github.com/Tencent/ncnn/releases'> here </a>. </b><br>
git clone -b 20220721 https://github.com/Tencent/ncnn.git
cd ncnn
git submodule update --init
export NCNN_DIR=${PWD}

export ANDROID_ABI=arm64-v8a

mkdir -p build\_${ANDROID_ABI}
cd build\_${ANDROID_ABI}

cmake -DCMAKE_TOOLCHAIN_FILE=${NDK_PATH}/build/cmake/android.toolchain.cmake -DANDROID_ABI="${ANDROID_ABI}" -DANDROID_PLATFORM=android-30 -DNCNN_VULKAN=ON -DNCNN_DISABLE_EXCEPTION=OFF -DNCNN_DISABLE_RTTI=OFF -DANDROID_USE_LEGACY_TOOLCHAIN_FILE=False ..
make -j$(nproc) install

</td>
  </tr>
  <tr>
  <td>OpenJDK </td>
  <td>It is necessary for building Java API.</br>

</td>
  </tr>
  <tr>
  See <a href='https://github.com/open-mmlab/mmdeploy/tree/main/csrc/mmdeploy/apis/java/README.md'> Java API build </a> for building tutorials.
  </td>
  </tr>
</tbody>
</table>

### Build MMDeploy

#### Build SDK and Demos

MMDeploy provides a recipe as shown below for building SDK with ncnn as inference engine for android.

- cpu + ncnn
  ```Bash
  export ANDROID_ABI=arm64-v8a
  mkdir -p build_${ANDROID_ABI} && cd build_${ANDROID_ABI}
  cmake .. \
      -DMMDEPLOY_BUILD_SDK_JAVA_API=ON \
      -DOpenCV_DIR=${OPENCV_ANDROID_SDK_DIR}/sdk/native/jni/abi-${ANDROID_ABI} \
      -Dncnn_DIR=${NCNN_DIR}/build_${ANDROID_ABI}/install/lib/cmake/ncnn \
      -DMMDEPLOY_TARGET_BACKENDS=ncnn \
      -DMMDEPLOY_SHARED_LIBS=OFF \

-Dncnn_DIR=${NCNN_DIR}/build_${ANDROID_ABI}/install/lib/cmake/ncnn \
      -DCMAKE_TOOLCHAIN_FILE=${NDK_PATH}/build/cmake/android.toolchain.cmake \
      -DANDROID_USE_LEGACY_TOOLCHAIN_FILE=False \
      -DANDROID_ABI=${ANDROID_ABI} \
      -DANDROID_PLATFORM=android-30 \
      -DANDROID_CPP_FEATURES="rtti exceptions"

  ```

Please check [cmake build option spec](cmake_option.md)

# Mask2Former

> [Masked-attention Mask Transformer for Universal Image Segmentation](http://arxiv.org/abs/2112.01527)

Image segmentation is about grouping pixels with different semantics, e.g., category or instance membership, where each choice of semantics defines a task. While only the semantics of each task differ, current research focuses on designing specialized architectures for each task. We present Masked-attention Mask Transformer (Mask2Former), a new architecture capable of addressing any image segmentation task (panoptic, instance or semantic). Its key components include masked attention, which extracts localized features by constraining cross-attention within predicted mask regions. In addition to reducing the research effort by at least three times, it outperforms the best specialized architectures by a significant margin on four popular datasets. Most notably, Mask2Former sets a new state-of-the-art for panoptic segmentation (57.8 PQ on COCO), instance segmentation (50.1 AP on COCO) and semantic segmentation (57.7 mIoU on ADE20K).

<img src="https://camo.githubusercontent.com/455d3116845b1d580b1f8a8542334b9752fdf39364deee2951cdd231524c7725/68747470733a2f2f626f77656e63303232312e6769746875622e696f2f696d616765732f6d61736b666f726d657276325f7465617365722e706e67" height="300"/>
</div>

Mask2Former requires COCO and [COCO-panoptic](http://images.cocodataset.org/annotations/panoptic_annotations_trainval2017.zip) dataset for training and evaluation. You need to download and extract it in the COCO dataset path.

```none
├── mmdet
├── tools
├── data
│   ├── coco
|   |   |   ├── instances_train2017.json
|   |   |   ├── instances_val2017.json
```

### Panoptic segmentation

```

| Backbone | style   | Pretrain     | Lr schd | Mem (GB) | Inf time (fps) | PQ   | box mAP | mask mAP | Config                                                                          | Download                                                                                                                                                                                                                                                                                                                                                                                                                                                           |

| R-50     | pytorch | ImageNet-1K  | 50e     | 13.9     | -              | 52.0 | 44.5    | 41.8     | [config](./mask2former_r50_8xb2-lsj-50e_coco-panoptic.py)                       | [model](https://download.openmmlab.com/mmdetection/v3.0/mask2former/mask2former_r50_8xb2-lsj-50e_coco-panoptic/mask2former_r50_8xb2-lsj-50e_coco-panoptic_20230118_125535-54df384a.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/mask2former/mask2former_r50_8xb2-lsj-50e_coco-panoptic/mask2former_r50_8xb2-lsj-50e_coco-panoptic_20230118_125535.log.json)                                                                                       |

| R-101    | pytorch | ImageNet-1K  | 50e     | 16.1     | -              | 52.4 | 45.3    | 42.4     | [config](./mask2former_r101_8xb2-lsj-50e_coco-panoptic.py)                      | [model](https://download.openmmlab.com/mmdetection/v3.0/mask2former/mask2former_r101_8xb2-lsj-50e_coco-panoptic/mask2former_r101_8xb2-lsj-50e_coco-panoptic_20220329_225104-c74d4d71.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask2former/mask2former_r101_lsj_8x2_50e_coco-panoptic/mask2former_r101_lsj_8x2_50e_coco-panoptic_20220329_225104.log.json)                                                                                     |

| Swin-T   | -       | ImageNet-1K  | 50e     | 15.9     | -              | 53.4 | 46.3    | 43.4     | [config](./mask2former_swin-t-p4-w7-224_8xb2-lsj-50e_coco-panoptic.py)          | [model](https://download.openmmlab.com/mmdetection/v3.0/mask2former/mask2former_swin-t-p4-w7-224_8xb2-lsj-50e_coco-panoptic/mask2former_swin-t-p4-w7-224_8xb2-lsj-50e_coco-panoptic_20220326_224553-3ec9e0ae.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask2former/mask2former_swin-t-p4-w7-224_lsj_8x2_50e_coco-panoptic/mask2former_swin-t-p4-w7-224_lsj_8x2_50e_coco-panoptic_20220326_224553.log.json)                                     |

| Swin-S   | -       | ImageNet-1K  | 50e     | 19.1     | -              | 54.5 | 47.8    | 44.5     | [config](./mask2former_swin-s-p4-w7-224_8xb2-lsj-50e_coco-panoptic.py)          | [model](https://download.openmmlab.com/mmdetection/v3.0/mask2former/mask2former_swin-s-p4-w7-224_8xb2-lsj-50e_coco-panoptic/mask2former_swin-s-p4-w7-224_8xb2-lsj-50e_coco-panoptic_20220329_225200-4a16ded7.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask2former/mask2former_swin-s-p4-w7-224_lsj_8x2_50e_coco-panoptic/mask2former_swin-s-p4-w7-224_lsj_8x2_50e_coco-panoptic_20220329_225200.log.json)                                     |

| Swin-B   | -       | ImageNet-1K  | 50e     | 26.0     | -              | 55.1 | 48.2    | 44.9     | [config](./mask2former_swin-b-p4-w12-384_8xb2-lsj-50e_coco-panoptic.py)         | [model](https://download.openmmlab.com/mmdetection/v3.0/mask2former/mask2former_swin-b-p4-w12-384_8xb2-lsj-50e_coco-panoptic/mask2former_swin-b-p4-w12-384_8xb2-lsj-50e_coco-panoptic_20220331_002244-8a651d82.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask2former/mask2former_swin-b-p4-w12-384_lsj_8x2_50e_coco-panoptic/mask2former_swin-b-p4-w12-384_lsj_8x2_50e_coco-panoptic_20220331_002244.log.json)                                 |

| Swin-B   | -       | ImageNet-21K | 50e     | 25.8     | -              | 56.3 | 50.0    | 46.3     | [config](./mask2former_swin-b-p4-w12-384-in21k_8xb2-lsj-50e_coco-panoptic.py)   | [model](https://download.openmmlab.com/mmdetection/v3.0/mask2former/mask2former_swin-b-p4-w12-384-in21k_8xb2-lsj-50e_coco-panoptic/mask2former_swin-b-p4-w12-384-in21k_8xb2-lsj-50e_coco-panoptic_20220329_230021-05ec7315.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask2former/mask2former_swin-b-p4-w12-384-in21k_lsj_8x2_50e_coco-panoptic/mask2former_swin-b-p4-w12-384-in21k_lsj_8x2_50e_coco-panoptic_20220329_230021.log.json)         |

| Swin-L   | -       | ImageNet-21K | 100e    | 21.1     | -              | 57.6 | 52.2    | 48.5     | [config](./mask2former_swin-l-p4-w12-384-in21k_16xb1-lsj-100e_coco-panoptic.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/mask2former/mask2former_swin-l-p4-w12-384-in21k_16xb1-lsj-100e_coco-panoptic/mask2former_swin-l-p4-w12-384-in21k_16xb1-lsj-100e_coco-panoptic_20220407_104949-82f8d28d.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask2former/mask2former_swin-l-p4-w12-384-in21k_lsj_16x1_100e_coco-panoptic/mask2former_swin-l-p4-w12-384-in21k_lsj_16x1_100e_coco-panoptic_20220407_104949.log.json) |

### Instance segmentation

| Backbone | style   | Pretrain    | Lr schd | Mem (GB) | Inf time (fps) | box mAP | mask mAP | Config                                                        | Download                                                                                                                                                                                                                                                                                                                                                                                   |

| R-50     | pytorch | ImageNet-1K | 50e     | 13.7     | -              | 45.7    | 42.9     | [config](./mask2former_r50_8xb2-lsj-50e_coco.py)              | [model](https://download.openmmlab.com/mmdetection/v3.0/mask2former/mask2former_r50_8xb2-lsj-50e_coco/mask2former_r50_8xb2-lsj-50e_coco_20220506_191028-41b088b6.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask2former/mask2former_r50_lsj_8x2_50e_coco/mask2former_r50_lsj_8x2_50e_coco_20220506_191028.log.json)                                                     |

| R-101    | pytorch | ImageNet-1K | 50e     | 15.5     | -              | 46.7    | 44.0     | [config](./mask2former_r101_8xb2-lsj-50e_coco.py)             | [model](https://download.openmmlab.com/mmdetection/v3.0/mask2former/mask2former_r101_8xb2-lsj-50e_coco/mask2former_r101_8xb2-lsj-50e_coco_20220426_100250-ecf181e2.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask2former/mask2former_r101_lsj_8x2_50e_coco/mask2former_r101_lsj_8x2_50e_coco_20220426_100250.log.json)                                                 |

| Swin-T   | -       | ImageNet-1K | 50e     | 15.3     | -              | 47.7    | 44.7     | [config](./mask2former_swin-t-p4-w7-224_8xb2-lsj-50e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/mask2former/mask2former_swin-t-p4-w7-224_8xb2-lsj-50e_coco/mask2former_swin-t-p4-w7-224_8xb2-lsj-50e_coco_20220508_091649-01b0f990.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask2former/mask2former_swin-t-p4-w7-224_lsj_8x2_50e_coco/mask2former_swin-t-p4-w7-224_lsj_8x2_50e_coco_20220508_091649.log.json) |

| Swin-S   | -       | ImageNet-1K | 50e     | 18.8     | -              | 49.3    | 46.1     | [config](./mask2former_swin-s-p4-w7-224_8xb2-lsj-50e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/mask2former/mask2former_swin-s-p4-w7-224_8xb2-lsj-50e_coco/mask2former_swin-s-p4-w7-224_8xb2-lsj-50e_coco_20220504_001756-c9d0c4f2.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask2former/mask2former_swin-s-p4-w7-224_lsj_8x2_50e_coco/mask2former_swin-s-p4-w7-224_lsj_8x2_50e_coco_20220504_001756.log.json) |

### Note

1. The performance is unstable. The `Mask2Former-R50-coco-panoptic` may fluctuate about 0.2 PQ. The models other than `Mask2Former-R50-coco-panoptic` were trained with mmdet 2.x and have been converted for mmdet 3.x.

2. We have trained the instance segmentation models many times (see more details in [PR 7571](https://github.com/open-mmlab/mmdetection/pull/7571)). The results of the trained models are relatively stable (+- 0.2), and have a certain gap (about 0.2 AP) in comparison with the results in the [paper](http://arxiv.org/abs/2112.01527). However, the performance of the model trained with the official code is unstable and may also be slightly lower than the reported results as mentioned in the [issue](https://github.com/facebookresearch/Mask2Former/issues/46).

```latex
@article{cheng2021mask2former,
  title={Masked-attention Mask Transformer for Universal Image Segmentation},
  author={Bowen Cheng and Ishan Misra and Alexander G. Schwing and Alexander Kirillov and Rohit Girdhar},
}
```

### Step 1: Implement a new hook

MMEngine has implemented commonly used [hooks](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/hook.md) for training and test,
When users have requirements for customization, they can follow examples below.
For example, if some hyper-parameter of the model needs to be changed when model training, we can implement a new hook for it:

```python
from typing import Optional, Sequence

from mmengine.model import is_model_wrapper

from mmseg.registry import HOOKS

class NewHook(Hook):
    """Docstring for NewHook.
    """

    def __init__(self, a: int, b: int) -> None:
        self.a = a
        self.b = b

                          runner,

def __init__(self, a: int, b: int) -> None:
        self.a = a
        self.b = b

                          runner,
                          data_batch: Optional[Sequence[dict]] = None) -> None:
        cur_iter = runner.iter
        # acquire this model when it is in a wrapper
        if is_model_wrapper(runner.model):
          model = runner.model.module
        model.hyper_parameter = self.a * cur_iter + self.b
```

### Step 2: Import a new hook

The module which is defined above needs to be imported into main namespace first to ensure being registered.
We assume `NewHook` is implemented in `mmseg/engine/hooks/new_hook.py`, there are two ways to import it:

- Import it by modifying `mmseg/engine/hooks/__init__.py`.
  Modules should be imported in `mmseg/engine/hooks/__init__.py` thus these new modules can be found and added by registry.

```python
from .new_hook import NewHook

__all__ = [..., NewHook]

Modules should be imported in `mmseg/engine/hooks/__init__.py` thus these new modules can be found and added by registry.

```python

```

- Import it manually by `custom_imports` in config file.

```python
custom_imports = dict(imports=['mmseg.engine.hooks.new_hook'], allow_failed_imports=False)
```

### Step 3: Modify config file

Users can set and use customized hooks in training and test followed methods below.
The execution priority of hooks at the same place of `Runner` can be referred [here](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/hook.md#built-in-hooks),
Default priority of customized hook is `NORMAL`.

```python
    dict(type='NewHook', a=a_value, b=b_value, priority='ABOVE_NORMAL')
]
```

## Customize optimizer

### Step 1: Implement a new optimizer

```python
]
```

We recommend the customized optimizer implemented in `mmseg/engine/optimizers/my_optimizer.py`. Here is an example of a new optimizer `MyOptimizer` which has parameters `a`, `b` and `c`:

```python
from mmseg.registry import OPTIMIZERS

```

### Step 2: Import a new optimizer

We assume `MyOptimizer` is implemented in `mmseg/engine/optimizers/my_optimizer.py`, there are two ways to import it:

- Import it by modifying `mmseg/engine/optimizers/__init__.py`.
  Modules should be imported in `mmseg/engine/optimizers/__init__.py` thus these new modules can be found and added by registry.

```python
```

```python
custom_imports = dict(imports=['mmseg.engine.optimizers.my_optimizer'], allow_failed_imports=False)
```

Then it needs to modify `optimizer` in `optim_wrapper` of config file, if users want to use customized `MyOptimizer`, it can be modified as:

```python
optim_wrapper = dict(type='OptimWrapper',
                     optimizer=dict(type='MyOptimizer',
                                    a=a_value, b=b_value, c=c_value),
                     clip_grad=None)
```

## Customize optimizer constructor

### Step 1: Implement a new optimizer constructor

a=a_value, b=b_value, c=c_value),
```

Optimizer constructor is used to create optimizer and optimizer wrapper for model training, which has powerful functions like specifying learning rate and weight decay for different model layers.
Here is an example for a customized optimizer constructor.

```python
from mmseg.registry import OPTIM_WRAPPER_CONSTRUCTORS

class LearningRateDecayOptimizerConstructor(DefaultOptimWrapperConstructor):

```

Default optimizer constructor is implemented [here](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/optimizer/default_constructor.py#L19).

return my_optimizer
```

It can also be used as base class of new optimizer constructor.

### Step 2: Import a new optimizer constructor

We assume `MyOptimizerConstructor` is implemented in `mmseg/engine/optimizers/my_optimizer_constructor.py`, there are two ways to import it:

```python
from .my_optimizer_constructor import MyOptimizerConstructor
```

```python

```python
```

```python
custom_imports = dict(imports=['mmseg.engine.optimizers.my_optimizer_constructor'], allow_failed_imports=False)
```

Then it needs to modify `constructor` in `optim_wrapper` of config file, if users want to use customized `MyOptimizerConstructor`, it can be modified as:

```python
                     constructor='MyOptimizerConstructor',
```

# 准备骨架数据集

```BibTeX
@misc{duan2021revisiting,
      title={Revisiting Skeleton-based Action Recognition},
      author={Haodong Duan and Yue Zhao and Kai Chen and Dian Shao and Dahua Lin and Bo Dai},
      eprint={2104.13586},
}
```

## 简介

MMAction2 发布 [Revisiting Skeleton-based Action Recognition](https://arxiv.org/abs/2104.13586) 论文中所使用的骨架标注。
默认使用 [Faster-RCNN](https://github.com/open-mmlab/mmdetection/blob/master/configs/faster_rcnn/faster_rcnn_r50_caffe_fpn_mstrain_1x_coco-person.py) 作为人体检测器，
使用 [HRNet-w32](https://github.com/open-mmlab/mmpose/blob/master/configs/top_down/hrnet/coco/hrnet_w32_coco_256x192.py) 作为单人姿态估计模型。
对于 FineGYM 数据集，MMAction2 使用的是运动员的真实框标注，而非检测器所出的框。目前，MMAction2 已发布 FineGYM 和 NTURGB-D Xsub 部分的骨架标注，其他数据集的标注也将很快发布。

## 准备标注文件

目前，MMAction2 支持 HMDB51, UCF101, FineGYM 和 NTURGB+D 数据集。对于 FineGYM 数据集，用户可以使用以下脚本下载标注文件。

```shell
bash download_annotations.sh ${DATASET}
```

## 准备标注文件

```shell
```

由于 NTURGB+D 数据集的 [使用条例](http://rose1.ntu.edu.sg/Datasets/actionRecognition.asp)，MMAction2 并未直接发布实验中所使用的标注文件。
因此，这里提供生成 NTURGB+D 数据集中视频的姿态标注文件，这将生成一个 dict 数据并将其保存为一个 pickle 文件。
用户可以生成一个 list 用以包含对应视频的 dict 数据，并将其保存为一个 pickle 文件。
之后，用户可以获得 `ntu60_xsub_train.pkl`, `ntu60_xsub_val.pkl`, `ntu120_xsub_train.pkl`, `ntu120_xsub_val.pkl` 文件用于训练。

对于无法进行姿态提取的用户，这里提供了上述流程的输出结果，分别对应 NTURGB-D 数据集的 4 个部分：

- NTURGB+D \[2D Skeleton\]: https://download.openmmlab.com/mmaction/v1.0/skeleton/data/ntu60_2d.pkl
- NTURGB+D \[3D Skeleton\]: https://download.openmmlab.com/mmaction/v1.0/skeleton/data/ntu60_3d.pkl
- NTURGB+D 120 \[2D Skeleton\]: https://download.openmmlab.com/mmaction/v1.0/skeleton/data/ntu120_2d.pkl
- NTURGB+D 120 \[3D Skeleton\]: https://download.openmmlab.com/mmaction/v1.0/skeleton/data/ntu120_3d.pkl

- GYM \[2D Skeleton\]: https://download.openmmlab.com/mmaction/v1.0/skeleton/data/gym_2d.pkl
  - GYM 2D 姿态标注文件是基于运动员的真实标注框生成的，用户可以从这个[链接](https://download.openmmlab.com/mmaction/pyskl/data/gym/gym_gt_bboxes.pkl)下载真实标注框。如果你在项目中使用了该数据，请引用 [PoseConv3D](https://arxiv.org/abs/2104.13586)
- UCF101 \[2D Skeleton\]: https://download.openmmlab.com/mmaction/v1.0/skeleton/data/ucf101_2d.pkl
- HMDB51 \[2D Skeleton\]: https://download.openmmlab.com/mmaction/v1.0/skeleton/data/hmdb51_2d.pkl
- Diving48 \[2D Skeleton\]: https://download.openmmlab.com/mmaction/v1.0/skeleton/data/diving48_2d.pkl
- Kinetics400 \[2D Skeleton\]: https://download.openmmlab.com/mmaction/v1.0/skeleton/data/k400_2d.pkl (只包含数据列表，没有姿态标注文件)

由于 Kinetics400 数据集姿态标注文件过大，我们不提供阿里云的下载链接，请使用此[链接](https://mycuhk-my.sharepoint.com/:u:/g/personal/1155136485_link_cuhk_edu_hk/EeyDCVskqLtClMVVwqD53acBF2FEwkctp3vtRbkLfnKSTw?e=B3SZlM)下载 `kpfiles`，解压到 `$MMACTION2/data/k400` 目录下，用于 Kinetics400 的训练和测试。

若想生成单个视频的 2D 姿态标注文件，用户在安装 mmdetection 和 mmpose 之后，可使用以下脚本进行 NTURGB+D 视频的姿态提取：

```python
python ntu_pose_extraction.py S001C001P001R001A001_rgb.avi S001C001P001R001A001.pkl
```

请注意，由于 mmpose 算法库升级，此脚本的推理结果与提供的姿态点数据集可能略有差异。

在用户获得数据集某部分所有视频的姿态标注文件（如 `ntu60_xsub_val`）后，可以将其集合成一个 list 数据并保存为 `ntu60_xsub_val.pkl`。用户可用这些大型 pickle 文件进行训练和测试。

## PoseC3D 的标注文件格式

这里简单介绍 PoseC3D 的标注文件格式。以 `gym_train.pkl` 为例：`gym_train.pkl` 存储一个长度为 20484 的 list，list 的每一项为单个视频的骨架标注 dict。每个 dict 的内容如下：

- keypoint：关键点坐标，大小为 N（#人数）x T（时序长度）x K（#关键点, 这里为17）x 2 （x，y 坐标）的 numpy array 数据类型

- keypoint_score：关键点的置信分数，大小为 N（#人数）x T（时序长度）x K（#关键点, 这里为17）的 numpy array 数据类型
- frame_dir: 对应视频名
- label: 动作类别
- img_shape: 每一帧图像的大小
- original_shape: 同 `img_shape`
- total_frames: 视频时序长度

如用户想使用自己的数据集训练 PoseC3D，可以参考 [Custom Dataset Training](https://github.com/open-mmlab/mmaction2/blob/master/configs/skeleton/posec3d/custom_dataset_training.md)。

## 可视化

为了可视化骨架数据，用户需要准备 RGB 的视频。详情可参考 \[visualize_heatmap_volume\]。这里提供一些 NTU-60 和 FineGYM 上的例子

<table>
<thead>
  <tr>
    <td>
  <b> 姿态估计结果 </b>
  <br/>
  <img src="https://user-images.githubusercontent.com/34324155/116529341-6fc95080-a90f-11eb-8f0d-57fdb35d1ba4.gif" width="455"/>
  <br/>
  <br/>
  <img src="https://user-images.githubusercontent.com/34324155/116531676-04cd4900-a912-11eb-8db4-a93343bedd01.gif" width="455"/>
</div></td>
    <td>
  <b> 关键点热力图三维可视化 </b>
  <br/>

    <td>
  <br/>
  <img src="https://user-images.githubusercontent.com/34324155/116529336-6dff8d00-a90f-11eb-807e-4d9168997655.gif" width="256"/>
  <br/>
  <br/>
  <img src="https://user-images.githubusercontent.com/34324155/116531658-00a12b80-a912-11eb-957b-561c280a86da.gif" width="256"/>
    <td>
  <b> 肢体热力图三维可视化 </b>
  <br/>
  <img src="https://user-images.githubusercontent.com/34324155/116529322-6a6c0600-a90f-11eb-81df-6fbb36230bd0.gif" width="256"/>
  <br/>
  <br/>
  <img src="https://user-images.githubusercontent.com/34324155/116531649-fed76800-a911-11eb-8ca9-0b4e58f43ad9.gif" width="256"/>
  </tr>
</thead>
</table>

## 如何将 NTU RGB+D 原始数据转化为 MMAction2 格式 （转换好的标注文件目前仅适用于 GCN 模型）

这里介绍如何将 NTU RGB+D 原始数据转化为 MMAction2 格式。首先，需要从 https://github.com/shahroudy/NTURGB-D 下载原始 NTU-RGBD 60 和 NTU-RGBD 120 数据集的原始骨架数据。

对于 NTU-RGBD 60 数据集，可使用以下脚本

```python

```python
python gen_ntu_rgbd_raw.py --data-path your_raw_nturgbd60_skeleton_path --ignored-sample-path NTU_RGBD_samples_with_missing_skeletons.txt --out-folder your_nturgbd60_output_path --task ntu60
```

对于 NTU-RGBD 120 数据集，可使用以下脚本

```python
python gen_ntu_rgbd_raw.py --data-path your_raw_nturgbd120_skeleton_path --ignored-sample-path NTU_RGBD120_samples_with_missing_skeletons.txt --out-folder your_nturgbd120_output_path --task ntu120
```

## 转换其他第三方项目的骨骼标注

MMAction2 提供脚本以将其他第三方项目的骨骼标注转至 MMAction2 格式，如：

- BABEL: `babel2mma2.py`

**待办项**：

- [x] FineGYM
- [x] NTU60_XSub
- [x] NTU120_XSub
- [x] NTU60_XView
- [x] NTU120_XSet
- [x] UCF101
- [x] HMDB51
- [x] Kinetics

# Evaluation pipeline on MMBench

## Intro to each data sample in MMBench

MMBecnh is split into **dev** and **test** split, and each data sample in each split contains the following field:

```
img: the raw data of an image
question: the question
options: the concated options
category: the leaf category
l2-category: the l2-level category
options_dict: the dict contains all options
index: the unique identifier of current question
context (optional): the context to a question, which is optional.
answer: the target answer to current question. (only exists in the dev split, and is keep confidential for the test split on our evaluation server)
```

## Load MMBench

We provide a code snippet as an example of loading MMBench

```python
import base64
import io

import pandas as pd
from PIL import Image
from torch.utils.data import Dataset

def decode_base64_to_image(base64_string):
    image_data = base64.b64decode(base64_string)
    image = Image.open(io.BytesIO(image_data))

    return image

class MMBenchDataset(Dataset):
                 data_file,
                 sys_prompt='There are several options:'):
        self.df = pd.read_csv(data_file, sep='\t')
        self.sys_prompt = sys_prompt

    def __len__(self):
        return len(self.df)

    def __getitem__(self, idx):
        index = self.df.iloc[idx]['index']
        image = self.df.iloc[idx]['image']
        image = decode_base64_to_image(image)
        question = self.df.iloc[idx]['question']
        answer = self.df.iloc[idx]['answer'] if 'answer' in self.df.iloc[0].keys() else None
        catetory = self.df.iloc[idx]['category']
        l2_catetory = self.df.iloc[idx]['l2-category']

        option_candidate = ['A', 'B', 'C', 'D', 'E']
        options = {

catetory = self.df.iloc[idx]['category']

        options = {
            cand: self.load_from_df(idx, cand)
            for cand in option_candidate
            if self.load_from_df(idx, cand) is not None
        }
        options_prompt = f'{self.sys_prompt}\n'
        for key, item in options.items():
            options_prompt += f'{key}. {item}\n'

        hint = self.load_from_df(idx, 'hint')
        data = {
            'img': image,
            'question': question,
            'answer': answer,
            'options': options_prompt,
            'category': catetory,
            'l2-category': l2_catetory,
            'options_dict': options,
            'index': index,
            'context': hint,
        }
   def load_from_df(self, idx, key):
        if key in self.df.iloc[idx] and not pd.isna(self.df.iloc[idx][key]):

'index': index,
        }
            return self.df.iloc[idx][key]
        else:
            return None
```

## How to construct the inference prompt

```python
if data_sample['context'] is not None:
    prompt = data_sample['context'] + ' ' + data_sample['question'] + ' ' + data_sample['options']
else:
    prompt = data_sample['question'] + ' ' + data_sample['options']
```

For example:
Question: Which category does this image belong to?
A. Oil Painting
B. Sketch
C. Digital art
D. Photo

<img src="https://github-production-user-asset-6210df.s3.amazonaws.com/34324155/255581681-1364ef43-bd27-4eb5-b9e5-241327b1f920.png" width="50%"/>
</div>

```python
prompt = """
###Human: Question: Which category does this image belong to?

</div>

```python
There are several options: A. Oil Painting, B. Sketch, C. Digital art, D. Photo
###Assistant:
"""
```

You can make custom modifications to the prompt

## How to save results:

You should dump your model's predictions into an excel(.xlsx) file, and this file should contain the following fields:

```
A: The first choice
B: The second choice
C: The third choice
D: The fourth choice
prediction: The prediction of your model to current question
l2_category: the l2-level category
index: the l2-level category
```

If there are any questions with fewer than four options, simply leave those fields blank.

## TRT 自定义算子

- [TRT 自定义算子](#trt-自定义算子)

- [Type Constraints](#type-constraints-10)

<dl>
</dl>

<dl>

<dl>
</dl>

### Git clone ncnn repo with submodule

```
git clone https://github.com/Tencent/ncnn.git
cd ncnn
```

- [Git clone ncnn repo with submodule](#git-clone-ncnn-repo-with-submodule)
- [Build for Linux](#build-for-linux)
  - [Nvidia Jetson](#nvidia-jetson)
  - [Raspberry Pi](#raspberry-pi)
  - [POWER](#power)
  - [Intel oneAPI](#intel-oneapi)
  - [Verification](#verification)
- [Build for Windows x64 using Visual Studio Community 2017](#build-for-windows-x64-using-visual-studio-community-2017)
- [Build for macOS](#build-for-macos)
- [Build for ARM Cortex-A family with cross-compiling](#build-for-arm-cortex-a-family-with-cross-compiling)
- [Build for Hisilicon platform with cross-compiling](#build-for-hisilicon-platform-with-cross-compiling)
- [Build for iOS on macOS with xcode](#build-for-ios-on-macos-with-xcode)
- [Build for WebAssembly](#build-for-webassembly)
- [Build for AllWinner D1](#build-for-allwinner-d1)

- [Build for Loongson 2K1000](#build-for-loongson-2k1000)
- [Build for Termux on Android](#build-for-termux-on-android)
- [Build for QNX](#build-for-qnx)

***

### Build for Linux

Install required build dependencies:

* git
* g++
* cmake
* protocol buffer (protobuf) headers files and protobuf compiler
* vulkan header files and loader library
* glslang
* (optional) opencv  # For building examples

Generally if you have Intel, AMD or Nvidia GPU from last 10 years, Vulkan can be easily used.

* glslang

On some systems there are no Vulkan drivers easily available at the moment (October 2020), so you might need to disable use of Vulkan on them. This applies to Raspberry Pi 3 (but there is experimental open source Vulkan driver in the works, which is not ready yet). Nvidia Tegra series devices (like Nvidia Jetson) should support Vulkan. Ensure you have most recent software installed for best experience.

On Debian, Ubuntu or Raspberry Pi OS, you can install all required dependencies using: 
```shell
sudo apt install build-essential git cmake libprotobuf-dev protobuf-compiler libvulkan-dev vulkan-utils libopencv-dev
```
On Redhat or Centos, you can install all required dependencies using: 
```shell

```
```shell
sudo yum install build-essential git cmake libprotobuf-dev protobuf-compiler libvulkan-dev vulkan-utils libopencv-dev
```
To use Vulkan backend install Vulkan header files, a vulkan driver loader, GLSL to SPIR-V compiler and vulkaninfo tool. Preferably from your distribution repositories. Alternatively download and install full Vulkan SDK (about 200MB in size; it contains all header files, documentation and prebuilt loader, as well some extra tools and source code of everything) from https://vulkan.lunarg.com/sdk/home

```shell
wget https://sdk.lunarg.com/sdk/download/1.2.189.0/linux/vulkansdk-linux-x86_64-1.2.189.0.tar.gz?Human=true -O vulkansdk-linux-x86_64-1.2.189.0.tar.gz
tar -xf vulkansdk-linux-x86_64-1.2.189.0.tar.gz
export VULKAN_SDK=$(pwd)/1.2.189.0/x86_64
```

```

To use Vulkan after building ncnn later, you will also need to have Vulkan driver for your GPU. For AMD and Intel GPUs these can be found in Mesa graphics driver, which usually is installed by default on all distros (i.e. `sudo apt install mesa-vulkan-drivers` on Debian/Ubuntu). For Nvidia GPUs the proprietary Nvidia driver must be downloaded and installed (some distros will allow easier installation in some way). After installing Vulkan driver, confirm Vulkan libraries and driver are working, by using `vulkaninfo` or `vulkaninfo | grep deviceType`, it should list GPU device type. If there are more than one GPU installed (including the case of integrated GPU and discrete GPU, commonly found in laptops), you might need to note the order of devices to use later on.

#### Nvidia Jetson

The Vulkan driver is a default component of the Linux For Tegra BSP release, check [the device list](https://developer.nvidia.com/embedded/vulkan).

```shell
cd ncnn
mkdir -p build
cd build
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE=../toolchains/jetson.toolchain.cmake -DNCNN_VULKAN=ON -DNCNN_BUILD_EXAMPLES=ON ..
make -j$(nproc)
```

#### Raspberry Pi

Vulkan drivers do exists, but are not mature. You are free to experiment at your own discretion, and report results and performance.

```shell
cd ncnn
cd build
cmake -DCMAKE_BUILD_TYPE=Release -DNCNN_VULKAN=ON -DNCNN_BUILD_EXAMPLES=ON ..
```

You can add `-GNinja` to `cmake` above to use Ninja build system (invoke build using `ninja` or `cmake --build .`).

```

For Rasberry Pi 3 on 32bit OS, add `-DCMAKE_TOOLCHAIN_FILE=../toolchains/pi3.toolchain.cmake` to cmake. You can also consider disabling Vulkan support as the Vulkan drivers for Rasberry Pi are still not mature, but it doesn't hurt to build the support in, but not use it.

#### POWER

For POWER9 with Clang 13 or higher:

```shell
cd ncnn
cd build
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE=../toolchains/power9le-linux-gnu-vsx.clang.toolchain.cmake -DNCNN_VULKAN=ON -DNCNN_BUILD_EXAMPLES=ON ..
```

cd build
```

Earlier versions of Clang may fail to build ncnn due to [Bug 49864](https://github.com/llvm/llvm-project/issues/49864). To use GCC instead, use the `power9le-linux-gnu-vsx.toolchain.cmake` toolchain file instead. Note that according to benchmarks, Clang appears to produce noticeably faster CPU inference than GCC for POWER9 targets.

For POWER8 instead of POWER9, use the `power8le-linux-gnu-vsx.clang.toolchain.cmake` or `power8le-linux-gnu-vsx.toolchain.cmake` toolchain file instead. POWER8 will be slower than POWER9.

Note that the POWER toolchain files only support little-endian mode.

#### Intel oneAPI

Besides the prerequests in this section, Intel oneAPI BaseKit and HPCKit should be installed. They are available from https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit.html and https://www.intel.com/content/www/us/en/developer/tools/oneapi/hpc-toolkit.html freely.

Intel oneAPI offers two kinds of compilers, the classic `icc/icpc` and the LLVM based `icx/icpx`. To build with these compilers, add `CC=icc CXX=icpc` or `CC=icx CXX=icpx` before the `cmake` command. When compiling with `icc/icpc`, cmake will warn that `xop`, `avx512`, and `bf16` extensions are not supported by the compiler, while `icx/icpx` works well.

Both of these compilers have been tested and passed the ncnn benchmark successfully. The results have been included in ncnn benchmark readme. Generally, `icx/icpx` are likely to show better performance than `icc/icpc` and the quantized models can benefit from the extensions `icx/icpx` supports.

#### Verification

Verify build by running some examples:

```shell
cd ../examples
../build/examples/squeezenet ../images/256-ncnn.png
[0 AMD RADV FIJI (LLVM 10.0.1)]  queueC=1[4]  queueG=0[1]  queueT=0[1]
[0 AMD RADV FIJI (LLVM 10.0.1)]  bugsbn1=0  buglbia=0  bugcopc=0  bugihfa=0
[0 AMD RADV FIJI (LLVM 10.0.1)]  fp16p=1  fp16s=1  fp16a=0  int8s=1  int8a=1
532 = 0.163452
920 = 0.093140
716 = 0.061584
```

You can also run benchmarks (the 4th argument is a GPU device index to use, refer to `vulkaninfo`, if you have more than one GPU):

```shell
cd ../benchmark
../build/benchmark/benchncnn 10 $(nproc) 0 0

```shell
num_threads = 4
powersave = 0
gpu_device = 0
cooling_down = 1
          squeezenet  min =    4.68  max =    4.99  avg =    4.85
     squeezenet_int8  min =   38.52  max =   66.90  avg =   48.52
...
```

To run benchmarks on a CPU, set the 5th argument to `-1`.

***

### Build for Windows x64 using Visual Studio Community 2017

Download and Install Visual Studio Community 2017 from https://visualstudio.microsoft.com/vs/community/

Start the command prompt: `Start → Programs → Visual Studio 2017 → Visual Studio Tools → x64 Native Tools Command Prompt for VS 2017`

> You can also search `x64 Native Tools Command Prompt for VS 2017` directly.

Download protobuf-3.11.2 from https://github.com/google/protobuf/archive/v3.11.2.zip

Build protobuf library:

```shell
cd <protobuf-root-dir>
mkdir protobuf_build
cd protobuf_build
cmake -A x64 -DCMAKE_INSTALL_PREFIX=%cd%/install -Dprotobuf_BUILD_TESTS=OFF -Dprotobuf_MSVC_STATIC_RUNTIME=OFF ../cmake
cmake --build . --config Release -j 2
cmake --build . --config Release --target install
```
(optional) Download and install Vulkan SDK from https://vulkan.lunarg.com/sdk/home

Build ncnn library (replace <protobuf-root-dir> with a proper path):

```shell
cd <ncnn-root-dir>
mkdir -p protobuf_build
cmake -A x64 -DCMAKE_INSTALL_PREFIX=%cd%/install -Dprotobuf_DIR=<protobuf-root-dir>/protobuf_build/install/cmake -DNCNN_VULKAN=ON ..
```

```

Note: To speed up compilation process on multi core machines, configuring `cmake` to use `jom` or `ninja` using `-G` flag is recommended.

***
### Build for macOS

We've published ncnn to [brew](https://formulae.brew.sh/formula/ncnn#default) now, you can just use following method to install ncnn if you have the Xcode Command Line Tools installed.

```shell
brew update
brew install ncnn
```

Or if you want to compile and build ncnn locally, first install Xcode or Xcode Command Line Tools according to your needs.

Then install `protobuf` and `libomp` via homebrew

```shell
brew install protobuf libomp
```

Download and install Vulkan SDK from <https://vulkan.lunarg.com/sdk/home>

```shell
wget https://sdk.lunarg.com/sdk/download/1.2.189.0/mac/vulkansdk-macos-1.2.189.0.dmg?Human=true -O vulkansdk-macos-1.2.189.0.dmg
hdiutil attach vulkansdk-macos-1.2.189.0.dmg

```shell
sudo /Volumes/vulkansdk-macos-1.2.189.0/InstallVulkan.app/Contents/MacOS/InstallVulkan --root `pwd`/vulkansdk-macos-1.2.189.0 --accept-licenses --default-answer --confirm-command install
hdiutil detach /Volumes/vulkansdk-macos-1.2.189.0

# setup env
export VULKAN_SDK=`pwd`/vulkansdk-macos-1.2.189.0/macOS
```

```shell
cd build

cmake -DCMAKE_OSX_ARCHITECTURES="x86_64;arm64" \
    -DVulkan_INCLUDE_DIR=`pwd`/../vulkansdk-macos-1.2.189.0/MoltenVK/include \
    -DVulkan_LIBRARY=`pwd`/../vulkansdk-macos-1.2.189.0/MoltenVK/dylib/macOS/libMoltenVK.dylib \
    -DNCNN_VULKAN=ON -DNCNN_BUILD_EXAMPLES=ON ..

cmake --build . -j 4
cmake --build . --target install
```

-DNCNN_VULKAN=ON -DNCNN_BUILD_EXAMPLES=ON ..

```

*Note: If you encounter `libomp` related errors during installation, you can also check our GitHub Actions at [here](https://github.com/Tencent/ncnn/blob/d91cccf/.github/workflows/macos-x64-gpu.yml#L50-L68) to install and use `openmp`.*
***

### Build for ARM Cortex-A family with cross-compiling
Download ARM toolchain from https://developer.arm.com/open-source/gnu-toolchain/gnu-a/downloads

```shell
export PATH="<your-toolchain-compiler-path>:${PATH}"
```

Alternatively install a cross-compiler provided by the distribution (i.e. on Debian / Ubuntu, you can do `sudo apt install g++-arm-linux-gnueabi g++-arm-linux-gnueabihf g++-aarch64-linux-gnu`).

Depending on your needs build one or more of the below targets.

AArch32 target with soft float (arm-linux-gnueabi)
```shell
mkdir -p build-arm-linux-gnueabi
cd build-arm-linux-gnueabi

```shell
cmake -DCMAKE_TOOLCHAIN_FILE=../toolchains/arm-linux-gnueabi.toolchain.cmake ..
```

AArch32 target with hard float (arm-linux-gnueabihf)
```shell
mkdir -p build-arm-linux-gnueabihf
cd build-arm-linux-gnueabihf
cmake -DCMAKE_TOOLCHAIN_FILE=../toolchains/arm-linux-gnueabihf.toolchain.cmake ..
```

AArch64 GNU/Linux target (aarch64-linux-gnu)
```shell
mkdir -p build-aarch64-linux-gnu
cd build-aarch64-linux-gnu
cmake -DCMAKE_TOOLCHAIN_FILE=../toolchains/aarch64-linux-gnu.toolchain.cmake ..
```

***

### Build for Hisilicon platform with cross-compiling
Download and install Hisilicon SDK. The toolchain should be in `/opt/hisi-linux/x86-arm`

```shell
cd build

# Choose one cmake toolchain file depends on your target platform

```shell
cd build

cmake -DCMAKE_TOOLCHAIN_FILE=../toolchains/hisiv300.toolchain.cmake ..
cmake -DCMAKE_TOOLCHAIN_FILE=../toolchains/hisiv500.toolchain.cmake ..
cmake -DCMAKE_TOOLCHAIN_FILE=../toolchains/himix100.toolchain.cmake ..
cmake -DCMAKE_TOOLCHAIN_FILE=../toolchains/himix200.toolchain.cmake ..

make install
```

***

### Build for Android
You can use the pre-build ncnn-android-lib.zip from https://github.com/Tencent/ncnn/releases

Download Android NDK from http://developer.android.com/ndk/downloads/index.html and install it, for example:

```shell
unzip android-ndk-r21d-linux-x86_64.zip
export ANDROID_NDK=<your-ndk-root-path>
```

(optional) remove the hardcoded debug flag in Android NDK [android-ndk issue](https://github.com/android-ndk/ndk/issues/243)
```
# open $ANDROID_NDK/build/cmake/android.toolchain.cmake for ndk < r23

```
# or $ANDROID_NDK/build/cmake/android-legacy.toolchain.cmake for ndk >= r23
# delete "-g" line
list(APPEND ANDROID_COMPILER_FLAGS
  -g
  -DANDROID
```

Build armv7 library

```shell
mkdir -p build-android-armv7
cd build-android-armv7

cmake -DCMAKE_TOOLCHAIN_FILE="$ANDROID_NDK/build/cmake/android.toolchain.cmake" \
    -DANDROID_ABI="armeabi-v7a" -DANDROID_ARM_NEON=ON \
    -DANDROID_PLATFORM=android-14 ..

# If you want to enable Vulkan, platform api version >= android-24 is needed
    -DANDROID_PLATFORM=android-24 -DNCNN_VULKAN=ON ..

# If you use cmake >= 3.21 and ndk-r23

-DANDROID_ABI="armeabi-v7a" -DANDROID_ARM_NEON=ON \

# you need to add -DANDROID_USE_LEGACY_TOOLCHAIN_FILE=False option for working optimization flags

```

Pick `build-android-armv7/install` folder for further JNI usage.

Build aarch64 library:

```shell
mkdir -p build-android-aarch64
cd build-android-aarch64

cmake -DCMAKE_TOOLCHAIN_FILE="$ANDROID_NDK/build/cmake/android.toolchain.cmake"\
    -DANDROID_ABI="arm64-v8a" \
    -DANDROID_PLATFORM=android-21 ..

-DANDROID_PLATFORM=android-24 -DNCNN_VULKAN=ON ..

```

Pick `build-android-aarch64/install` folder for further JNI usage.

***

### Build for iOS on macOS with xcode
You can use the pre-build ncnn.framework glslang.framework and openmp.framework from https://github.com/Tencent/ncnn/releases

Install xcode

You can replace ```-DENABLE_BITCODE=0``` to ```-DENABLE_BITCODE=1``` in the following cmake arguments if you want to build bitcode enabled libraries.

Download and install openmp for multithreading inference feature on iPhoneOS
```shell
wget https://github.com/llvm/llvm-project/releases/download/llvmorg-11.0.0/openmp-11.0.0.src.tar.xz
tar -xf openmp-11.0.0.src.tar.xz
cd openmp-11.0.0.src

# apply some compilation fix
sed -i'' -e '/.size __kmp_unnamed_critical_addr/d' runtime/src/z_Linux_asm.S

sed -i'' -e 's/__kmp_unnamed_critical_addr/___kmp_unnamed_critical_addr/g' runtime/src/z_Linux_asm.S

mkdir -p build-ios
cd build-ios

cmake -DCMAKE_TOOLCHAIN_FILE=<ncnn-root-dir>/toolchains/ios.toolchain.cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=install \
    -DIOS_PLATFORM=OS -DENABLE_BITCODE=0 -DENABLE_ARC=0 -DENABLE_VISIBILITY=0 -DIOS_ARCH="armv7;arm64;arm64e" \
    -DPERL_EXECUTABLE=/usr/local/bin/perl \
    -DLIBOMP_ENABLE_SHARED=OFF -DLIBOMP_OMPT_SUPPORT=OFF -DLIBOMP_USE_HWLOC=OFF ..

# copy openmp library and header files to xcode toolchain sysroot
# <xcode-dir> is usually /Applications/Xcode.app or /Applications/Xcode-beta.app depends on your Xcode version

sudo cp install/include/* <xcode-dir>/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include
sudo cp install/lib/libomp.a <xcode-dir>/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/lib
```

Download and install openmp for multithreading inference feature on iPhoneSimulator
```shell

mkdir -p build-ios-sim
cd build-ios-sim

    -DIOS_PLATFORM=SIMULATOR -DENABLE_BITCODE=0 -DENABLE_ARC=0 -DENABLE_VISIBILITY=0 -DIOS_ARCH="i386;x86_64" \

sudo cp install/include/* <xcode-dir>/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include

sudo cp install/lib/libomp.a <xcode-dir>/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/lib
```

Package openmp framework:
```shell
cd <openmp-root-dir>

mkdir -p openmp.framework/Versions/A/Headers
mkdir -p openmp.framework/Versions/A/Resources
ln -s A openmp.framework/Versions/Current
ln -s Versions/Current/Headers openmp.framework/Headers
ln -s Versions/Current/Resources openmp.framework/Resources
ln -s Versions/Current/openmp openmp.framework/openmp
lipo -create build-ios/install/lib/libomp.a build-ios-sim/install/lib/libomp.a -o openmp.framework/Versions/A/openmp
cp -r build-ios/install/include/* openmp.framework/Versions/A/Headers/

sed -e 's/__NAME__/openmp/g' -e 's/__IDENTIFIER__/org.llvm.openmp/g' -e 's/__VERSION__/11.0/g' <ncnn-root-dir>/Info.plist > openmp.framework/Versions/A/Resources/Info.plist
```

Download and install Vulkan SDK from https://vulkan.lunarg.com/sdk/home
```shell

# setup env
```

Build library for iPhoneOS:

```shell

# setup env
```

```shell

cmake -DCMAKE_TOOLCHAIN_FILE=../toolchains/ios.toolchain.cmake -DIOS_PLATFORM=OS -DIOS_ARCH="armv7;arm64;arm64e" \
    -DENABLE_BITCODE=0 -DENABLE_ARC=0 -DENABLE_VISIBILITY=0 \
    -DOpenMP_C_FLAGS="-Xclang -fopenmp" -DOpenMP_CXX_FLAGS="-Xclang -fopenmp" \
    -DOpenMP_C_LIB_NAMES="libomp" -DOpenMP_CXX_LIB_NAMES="libomp" \
    -DOpenMP_libomp_LIBRARY="/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/lib/libomp.a" \
    -DNCNN_BUILD_BENCHMARK=OFF ..

# vulkan is only available on arm64 devices
cmake -DCMAKE_TOOLCHAIN_FILE=../toolchains/ios.toolchain.cmake -DIOS_PLATFORM=OS64 -DIOS_ARCH="arm64;arm64e" \

    -DVulkan_INCLUDE_DIR=$VULKAN_SDK/../MoltenVK/include \
    -DVulkan_LIBRARY=$VULKAN_SDK/../MoltenVK/dylib/iOS/libMoltenVK.dylib \
    -DNCNN_VULKAN=ON -DNCNN_BUILD_BENCHMARK=OFF ..

```

Build library for iPhoneSimulator:

```shell

cmake -DCMAKE_TOOLCHAIN_FILE=../toolchains/ios.toolchain.cmake -DIOS_PLATFORM=SIMULATOR -DIOS_ARCH="i386;x86_64" \

    -DOpenMP_libomp_LIBRARY="/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/lib/libomp.a" \

```

Package glslang framework:
```shell

mkdir -p glslang.framework/Versions/A/Headers
mkdir -p glslang.framework/Versions/A/Resources
ln -s A glslang.framework/Versions/Current
ln -s Versions/Current/Headers glslang.framework/Headers
ln -s Versions/Current/Resources glslang.framework/Resources
ln -s Versions/Current/glslang glslang.framework/glslang

libtool -static build-ios/install/lib/libglslang.a build-ios/install/lib/libMachineIndependent.a build-ios/install/lib/libGenericCodeGen.a build-ios/install/lib/libSPIRV.a build-ios/install/lib/libOGLCompiler.a build-ios/install/lib/libOSDependent.a -o build-ios/install/lib/libglslang_combined.a
libtool -static build-ios-sim/install/lib/libglslang.a build-ios-sim/install/lib/libMachineIndependent.a build-ios-sim/install/lib/libGenericCodeGen.a build-ios-sim/install/lib/libSPIRV.a build-ios-sim/install/lib/libOGLCompiler.a build-ios-sim/install/lib/libOSDependent.a -o build-ios-sim/install/lib/libglslang_combined.a
lipo -create build-ios/install/lib/libglslang_combined.a build-ios-sim/install/lib/libglslang_combined.a -o glslang.framework/Versions/A/glslang

cp -r build/install/include/glslang glslang.framework/Versions/A/Headers/
sed -e 's/__NAME__/glslang/g' -e 's/__IDENTIFIER__/org.khronos.glslang/g' -e 's/__VERSION__/1.0/g' Info.plist > glslang.framework/Versions/A/Resources/Info.plist
```

Package ncnn framework:
```shell

mkdir -p ncnn.framework/Versions/A/Headers
mkdir -p ncnn.framework/Versions/A/Resources
ln -s A ncnn.framework/Versions/Current
ln -s Versions/Current/Headers ncnn.framework/Headers
ln -s Versions/Current/Resources ncnn.framework/Resources
ln -s Versions/Current/ncnn ncnn.framework/ncnn
lipo -create build-ios/install/lib/libncnn.a build-ios-sim/install/lib/libncnn.a -o ncnn.framework/Versions/A/ncnn
cp -r build-ios/install/include/* ncnn.framework/Versions/A/Headers/

sed -e 's/__NAME__/ncnn/g' -e 's/__IDENTIFIER__/com.tencent.ncnn/g' -e 's/__VERSION__/1.0/g' Info.plist > ncnn.framework/Versions/A/Resources/Info.plist
```

Pick `ncnn.framework` `glslang.framework` and `openmp.framework` folder for app development.

***

### Build for WebAssembly

Install Emscripten

```shell
git clone https://github.com/emscripten-core/emsdk.git
cd emsdk
./emsdk install 3.1.28
./emsdk activate 3.1.28

source emsdk_env.sh
```

Build without any extension for general compatibility:
```shell
cd build
cmake -DCMAKE_TOOLCHAIN_FILE=$EMSDK/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake \
    -DNCNN_THREADS=OFF -DNCNN_OPENMP=OFF -DNCNN_SIMPLEOMP=OFF -DNCNN_SIMPLEOCV=ON -DNCNN_RUNTIME_CPU=OFF -DNCNN_SSE2=OFF -DNCNN_AVX2=OFF -DNCNN_AVX=OFF \

-DNCNN_THREADS=OFF -DNCNN_OPENMP=OFF -DNCNN_SIMPLEOMP=OFF -DNCNN_SIMPLEOCV=ON -DNCNN_RUNTIME_CPU=OFF -DNCNN_SSE2=OFF -DNCNN_AVX2=OFF -DNCNN_AVX=OFF \
    -DNCNN_BUILD_TOOLS=OFF -DNCNN_BUILD_EXAMPLES=OFF -DNCNN_BUILD_BENCHMARK=OFF ..
```

Build with WASM SIMD extension:
```shell
mkdir -p build-simd
cd build-simd
    -DNCNN_THREADS=OFF -DNCNN_OPENMP=OFF -DNCNN_SIMPLEOMP=OFF -DNCNN_SIMPLEOCV=ON -DNCNN_RUNTIME_CPU=OFF -DNCNN_SSE2=ON -DNCNN_AVX2=OFF -DNCNN_AVX=OFF \
```

Build with WASM Thread extension:
```shell
mkdir -p build-threads
cd build-threads

```

```shell
    -DNCNN_THREADS=ON -DNCNN_OPENMP=ON -DNCNN_SIMPLEOMP=ON -DNCNN_SIMPLEOCV=ON -DNCNN_RUNTIME_CPU=OFF -DNCNN_SSE2=OFF -DNCNN_AVX2=OFF -DNCNN_AVX=OFF \
```

Build with WASM SIMD and Thread extension:
```shell
mkdir -p build-simd-threads
cd build-simd-threads
    -DNCNN_THREADS=ON -DNCNN_OPENMP=ON -DNCNN_SIMPLEOMP=ON -DNCNN_SIMPLEOCV=ON -DNCNN_RUNTIME_CPU=OFF -DNCNN_SSE2=ON -DNCNN_AVX2=OFF -DNCNN_AVX=OFF \
```

-DNCNN_BUILD_TOOLS=OFF -DNCNN_BUILD_EXAMPLES=OFF -DNCNN_BUILD_BENCHMARK=OFF ..
```

Pick `build-XYZ/install` folder for further usage.

***

### Build for AllWinner D1

Download c906 toolchain package from https://occ.t-head.cn/community/download?id=4046947553902661632

```shell
tar -xf Xuantie-900-gcc-linux-5.10.4-glibc-x86_64-V2.2.6-20220516.tar.gz
export RISCV_ROOT_PATH=/home/nihui/osd/Xuantie-900-gcc-linux-5.10.4-glibc-x86_64-V2.2.6
```

You need to fix riscv_vector.h header for workaround vfrec7/vfrsqrt7 bug.

Open ```$RISCV_ROOT_PATH/lib/gcc/riscv64-unknown-linux-gnu/10.2.0/include/riscv_vector.h```, goto the file end, you will find three ```#endif```, and apply changes as the following
```c
#endif

#define vfrec7_v_f32m1(x, vl) vfrdiv_vf_f32m1(x, 1.f, vl)
#define vfrec7_v_f32m2(x, vl) vfrdiv_vf_f32m2(x, 1.f, vl)
#define vfrec7_v_f32m4(x, vl) vfrdiv_vf_f32m4(x, 1.f, vl)

```c
#endif

#define vfrec7_v_f32m8(x, vl) vfrdiv_vf_f32m8(x, 1.f, vl)
#define vfrec7_v_f16m1(x, vl) vfrdiv_vf_f16m1(x, 1.f, vl)
#define vfrec7_v_f16m2(x, vl) vfrdiv_vf_f16m2(x, 1.f, vl)
#define vfrec7_v_f16m4(x, vl) vfrdiv_vf_f16m4(x, 1.f, vl)
#define vfrec7_v_f16m8(x, vl) vfrdiv_vf_f16m8(x, 1.f, vl)

#define vfrsqrt7_v_f32m1(x, vl) vfrdiv_vf_f32m1(vfsqrt_v_f32m1(x, vl), 1.f, vl)
#define vfrsqrt7_v_f32m2(x, vl) vfrdiv_vf_f32m2(vfsqrt_v_f32m2(x, vl), 1.f, vl)
#define vfrsqrt7_v_f32m4(x, vl) vfrdiv_vf_f32m4(vfsqrt_v_f32m4(x, vl), 1.f, vl)
#define vfrsqrt7_v_f32m8(x, vl) vfrdiv_vf_f32m8(vfsqrt_v_f32m8(x, vl), 1.f, vl)
#define vfrsqrt7_v_f16m1(x, vl) vfrdiv_vf_f16m1(vfsqrt_v_f16m1(x, vl), 1.f, vl)
#define vfrsqrt7_v_f16m2(x, vl) vfrdiv_vf_f16m2(vfsqrt_v_f16m2(x, vl), 1.f, vl)

#define vfrsqrt7_v_f16m4(x, vl) vfrdiv_vf_f16m4(vfsqrt_v_f16m4(x, vl), 1.f, vl)
#define vfrsqrt7_v_f16m8(x, vl) vfrdiv_vf_f16m8(vfsqrt_v_f16m8(x, vl), 1.f, vl)

#endif
#endif
```

Build ncnn with riscv-v vector and simpleocv enabled:
```shell
mkdir -p build-c906
cd build-c906
cmake -DCMAKE_TOOLCHAIN_FILE=../toolchains/c906-v226.toolchain.cmake \
    -DCMAKE_BUILD_TYPE=release -DNCNN_OPENMP=OFF -DNCNN_THREADS=OFF -DNCNN_RUNTIME_CPU=OFF -DNCNN_RVV=ON \
    -DNCNN_SIMPLEOCV=ON -DNCNN_BUILD_EXAMPLES=ON ..
```

Pick `build-c906/install` folder for further usage.

You can upload binary inside `build-c906/examples` folder and run on D1 board for testing.

***

### Build for Loongson 2K1000

```

***

For gcc version < 8.5, you need to fix msa.h header for workaround msa fmadd/fmsub/maddv/msubv bug.

Open ```/usr/lib/gcc/mips64el-linux-gnuabi64/8/include/msa.h```, find ```__msa_fmadd``` and ```__msa_fmsub``` and apply changes as the following
```c
// #define __msa_fmadd_w __builtin_msa_fmadd_w
// #define __msa_fmadd_d __builtin_msa_fmadd_d
// #define __msa_fmsub_w __builtin_msa_fmsub_w
// #define __msa_fmsub_d __builtin_msa_fmsub_d
#define __msa_fmadd_w(a, b, c) __builtin_msa_fmadd_w(c, b, a)
#define __msa_fmadd_d(a, b, c) __builtin_msa_fmadd_d(c, b, a)
#define __msa_fmsub_w(a, b, c) __builtin_msa_fmsub_w(c, b, a)
#define __msa_fmsub_d(a, b, c) __builtin_msa_fmsub_d(c, b, a)
```

find ```__msa_maddv``` and ```__msa_msubv``` and apply changes as the following
```c

```

```c
// #define __msa_maddv_b __builtin_msa_maddv_b
// #define __msa_maddv_h __builtin_msa_maddv_h
// #define __msa_maddv_w __builtin_msa_maddv_w
// #define __msa_maddv_d __builtin_msa_maddv_d
// #define __msa_msubv_b __builtin_msa_msubv_b
// #define __msa_msubv_h __builtin_msa_msubv_h
// #define __msa_msubv_w __builtin_msa_msubv_w
// #define __msa_msubv_d __builtin_msa_msubv_d
#define __msa_maddv_b(a, b, c) __builtin_msa_maddv_b(c, b, a)
#define __msa_maddv_h(a, b, c) __builtin_msa_maddv_h(c, b, a)
#define __msa_maddv_w(a, b, c) __builtin_msa_maddv_w(c, b, a)
#define __msa_maddv_d(a, b, c) __builtin_msa_maddv_d(c, b, a)
#define __msa_msubv_b(a, b, c) __builtin_msa_msubv_b(c, b, a)
#define __msa_msubv_h(a, b, c) __builtin_msa_msubv_h(c, b, a)
#define __msa_msubv_w(a, b, c) __builtin_msa_msubv_w(c, b, a)

#define __msa_msubv_d(a, b, c) __builtin_msa_msubv_d(c, b, a)
```

Build ncnn with mips msa and simpleocv enabled:
```shell
cd build
cmake -DNCNN_DISABLE_RTTI=ON -DNCNN_DISABLE_EXCEPTION=ON -DNCNN_RUNTIME_CPU=OFF -DNCNN_MSA=ON -DNCNN_MMI=ON -DNCNN_SIMPLEOCV=ON ..
cmake --build . -j 2
```

Pick `build/install` folder for further usage.

You can run binary inside `build/examples` folder for testing.

***

### Build for Termux on Android

Install app Termux on your phone,and install Ubuntu in Termux.

 If you want use ssh, just install openssh in Termux

```shell
pkg install proot-distro
proot-distro install ubuntu
```

or you can see what system can be installed using `proot-distro list`

If you want use ssh, just install openssh in Termux

```shell
```

or you can see what system can be installed using `proot-distro list` 

while you install ubuntu successfully, using `proot-distro login ubuntu` to login Ubuntu.

Then make ncnn,no need to install any other dependencies.

```shell
cd ncnn
cd build
cmake -DCMAKE_BUILD_TYPE=Release -DNCNN_BUILD_EXAMPLES=ON -DNCNN_PLATFORM_API=OFF -DNCNN_SIMPLEOCV=ON ..
```

Then you can run a test

> on my Pixel 3 XL using Qualcomm 845,cant load `256-ncnn.png`

```shell
../build/examples/squeezenet ../images/128-ncnn.png
```

### Build for QNX

Set QNX environment variables

```shell
export QNX_HOST=/opt/qnx710/host/linux/x86_64
export QNX_TARGET=/opt/qnx710/target/qnx7
```

Create ld link to solve 'cannot find ld' issue

```shell
cd ${QNX_HOST}/usr/bin/

```shell
```

```shell
ln -s aarch64-unknown-nto-qnx7.1.0-ld ld
```

Build ncnn with cmake

```shell
cd ncnn
mkdir -p build-qnx
cd build-qnx
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE=../toolchains/qnx710.toolchain.cmake ..
```

Pick `build-qnx/install` folder for further usage.

# The Devil is in the Details: Delving into Unbiased Data Processing for Human Pose Estimation

<details>

```bibtex
}
```

Recently, the leading performance of human pose estimation is dominated by top-down methods. Being a fundamental component in training and inference, data processing has not been systematically considered in pose estimation community, to the best of our knowledge. In this paper, we focus on this problem and find that the devil of top-down pose estimator is in the biased data processing. Specifically, by investigating the standard data processing in state-of-the-art approaches mainly including data transformation and encoding-decoding, we find that the results obtained by common flipping strategy are unaligned with the original ones in inference. Moreover, there is statistical error in standard encoding-decoding during both training and inference. Two problems couple together and significantly degrade the pose estimation performance. Based on quantitative analyses, we then formulate a principled way to tackle this dilemma. Data is processed in continuous space based on unit length (the intervals between pixels) instead of in discrete space with pixel, and a combined classification and regression approach is adopted to perform encoding-decoding. The Unbiased Data Processing (UDP) for human pose estimation can be achieved by combining the two together. UDP not only boosts the performance of existing methods by a large margin but also plays a important role in result reproducing and future exploration. As a model-agnostic approach, UDP promotes SimpleBaseline-ResNet50-256x192 by 1.5 AP (70.2 to 71.7) and HRNet-W32-256x192 by 1.7 AP (73.5 to 75.2) on COCO test-dev set. The HRNet-W48-384x288 equipped with UDP achieves 76.5 AP and sets a new state-of-the-art for human pose estimation. The source code is publicly available for further research.

<img src="https://user-images.githubusercontent.com/15977946/146524686-ddfe1356-77bd-46a0-a6cd-7ff418f65675.png">
</div>

# CSPNet

> [CSPNet: A New Backbone that can Enhance Learning Capability of CNN](https://arxiv.org/abs/1911.11929)

Neural networks have enabled state-of-the-art approaches to achieve incredible results on computer vision tasks such as object detection. However, such success greatly relies on costly computation resources, which hinders people with cheap devices from appreciating the advanced technology. In this paper, we propose Cross Stage Partial Network (CSPNet) to mitigate the problem that previous works require heavy inference computations from the network architecture perspective. We attribute the problem to the duplicate gradient information within network optimization. The proposed networks respect the variability of the gradients by integrating feature maps from the beginning and the end of a network stage, which, in our experiments, reduces computations by 20% with equivalent or even superior accuracy on the ImageNet dataset, and significantly outperforms state-of-the-art approaches in terms of AP50 on the MS COCO object detection dataset. The CSPNet is easy to implement and general enough to cope with architectures based on ResNet, ResNeXt, and DenseNet. Source code is at this https URL.

<img src="https://user-images.githubusercontent.com/18586273/159420842-6147c687-a488-460c-8bb2-4ea5276c26c7.png" width="60%"/>
</div>

```python

predict = inference_model('cspdarknet50_3rdparty_8xb32_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('cspdarknet50_3rdparty_8xb32_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/cspnet/cspdarknet50_8xb32_in1k.py https://download.openmmlab.com/mmclassification/v0/cspnet/cspdarknet50_3rdparty_8xb32_in1k_20220329-bd275287.pth
```

| `cspdarknet50_3rdparty_8xb32_in1k`\* | From scratch |   27.64    |   5.04    |   80.05   |   95.07   | [config](cspdarknet50_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/cspnet/cspdarknet50_3rdparty_8xb32_in1k_20220329-bd275287.pth) |
| `cspresnet50_3rdparty_8xb32_in1k`\*  | From scratch |   21.62    |   3.48    |   79.55   |   94.68   | [config](cspresnet50_8xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/cspnet/cspresnet50_3rdparty_8xb32_in1k_20220329-dd6dddfb.pth) |
| `cspresnext50_3rdparty_8xb32_in1k`\* | From scratch |   20.57    |   3.11    |   79.96   |   94.96   | [config](cspresnext50_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/cspnet/cspresnext50_3rdparty_8xb32_in1k_20220329-2cc84d21.pth) |

*Models with * are converted from the [official repo](https://github.com/rwightman/pytorch-image-models). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@inproceedings{wang2020cspnet,
  title={CSPNet: A new backbone that can enhance learning capability of CNN},
  author={Wang, Chien-Yao and Liao, Hong-Yuan Mark and Wu, Yueh-Hua and Chen, Ping-Yang and Hsieh, Jun-Wei and Yeh, I-Hau},
  booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops},
  pages={390--391},
}
```

# Deep high-resolution representation learning for human pose estimation

<details>

```bibtex
}
```

}
```

In this paper, we are interested in the human pose estimation problem with a focus on learning reliable highresolution representations. Most existing methods recover high-resolution representations from low-resolution representations produced by a high-to-low resolution network. Instead, our proposed network maintains high-resolution representations through the whole process. We start from a high-resolution subnetwork as the first stage, gradually add high-to-low resolution subnetworks one by one to form more stages, and connect the mutliresolution subnetworks in parallel. We conduct repeated multi-scale fusions such that each of the high-to-low resolution representations receives information from other parallel representations over and over, leading to rich highresolution representations. As a result, the predicted keypoint heatmap is potentially more accurate and spatially more precise. We empirically demonstrate the effectiveness

of our network through the superior pose estimation results over two benchmark datasets: the COCO keypoint detection
dataset and the MPII Human Pose dataset. In addition, we show the superiority of our network in pose tracking on the PoseTrack dataset.

<img src="https://user-images.githubusercontent.com/15977946/146518010-9c1d6078-dd9c-4610-94a9-cbcb60aa87c0.png">
</div>

# 可视化

中文内容建设中，暂时请查阅[英文版文档](../../en/user_guides/visualization.md)

# C3D

[Learning Spatiotemporal Features with 3D Convolutional Networks](https://openaccess.thecvf.com/content_iccv_2015/html/Tran_Learning_Spatiotemporal_Features_ICCV_2015_paper.html)

We propose a simple, yet effective approach for spatiotemporal feature learning using deep 3-dimensional convolutional networks (3D ConvNets) trained on a large scale supervised video dataset. Our findings are three-fold: 1) 3D ConvNets are more suitable for spatiotemporal feature learning compared to 2D ConvNets; 2) A homogeneous architecture with small 3x3x3 convolution kernels in all layers is among the best performing architectures for 3D ConvNets; and 3) Our learned features, namely C3D (Convolutional 3D), with a simple linear classifier outperform state-of-the-art methods on 4 different benchmarks and are comparable with current best methods on the other 2 benchmarks. In addition, the features are compact: achieving 52.8% accuracy on UCF101 dataset with only 10 dimensions and also very efficient to compute due to the fast inference of ConvNets. Finally, they are conceptually very simple and easy to train and use.

<img src="https://user-images.githubusercontent.com/34324155/143043383-8c26f5d6-d45e-47ae-be18-c23456eb84b9.png" width="800"/>
</div>

### UCF-101

| frame sampling strategy | resolution | gpus | backbone | pretrain | top1 acc | top5 acc | testing protocol  | FLOPs | params |                config                |                ckpt                |                log                |
| :---------------------: | :--------: | :--: | :------: | :------: | :------: | :------: | :---------------: | :---: | :----: | :----------------------------------: | :--------------------------------: | :-------------------------------: |

|         16x1x1          |  112x112   |  8   |   c3d    | sports1m |  83.08   |  95.93   | 10 clips x 1 crop | 38.5G | 78.4M  | [config](/configs/recognition/c3d/c3d_sports1m-pretrained_8xb30-16x1x1-45e_ucf101-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/c3d/c3d_sports1m-pretrained_8xb30-16x1x1-45e_ucf101-rgb/c3d_sports1m-pretrained_8xb30-16x1x1-45e_ucf101-rgb_20220811-31723200.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/c3d/c3d_sports1m-pretrained_8xb30-16x1x1-45e_ucf101-rgb/c3d_sports1m-pretrained_8xb30-16x1x1-45e_ucf101-rgb.log) |

1. The author of C3D normalized UCF-101 with volume mean and used SVM to classify videos, while we normalized the dataset with RGB mean value and used a linear classifier.

For more details on data preparation, you can refer to [UCF101](/tools/data/ucf101/README.md).

## Train

```shell
```

Example: train C3D model on UCF-101 dataset in a deterministic option with periodic validation.

```shell
python tools/train.py configs/recognition/c3d/c3d_sports1m-pretrained_8xb30-16x1x1-45e_ucf101-rgb.py \
```

```shell
```

## Test

```shell
```

Example: test C3D model on UCF-101 dataset and dump the result to a pkl file.

```shell
python tools/test.py configs/recognition/c3d_sports1m-pretrained_8xb30-16x1x1-45e_ucf101-rgb.py \
```

```BibTeX
@ARTICLE{2014arXiv1412.0767T,
author = {Tran, Du and Bourdev, Lubomir and Fergus, Rob and Torresani, Lorenzo and Paluri, Manohar},

```BibTeX
title = {Learning Spatiotemporal Features with 3D Convolutional Networks},
keywords = {Computer Science - Computer Vision and Pattern Recognition},
year = 2014,
month = dec,
eid = {arXiv:1412.0767}
}
```

```text
# timage/img_name text 1 text

timage/001.jpg RONALDO 1 RONALDO
timage/002.jpg 7 1 7
timage/003.jpg SEACREST 1 SEACREST
timage/004.jpg BEACH 1 BEACH
```

# KITTI 数据集

本页提供了有关在 MMDetection3D 中使用 KITTI 数据集的具体教程。

## 数据准备

您可以在[这里](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d)下载 KITTI 3D 检测数据并解压缩所有 zip 文件。此外，您可以在[这里](https://download.openmmlab.com/mmdetection3d/data/train_planes.zip)下载道路平面信息，其在训练过程中作为一个可选项，用来提高模型的性能。道路平面信息由 [AVOD](https://github.com/kujason/avod) 生成，更多细节请参考[此处](https://github.com/kujason/avod/issues/19)。

像准备数据集的一般方法一样，建议将数据集根目录链接到 `$MMDETECTION3D/data`。

在我们处理之前，文件夹结构应按如下方式组织：

```
├── tools
├── data
│   ├── kitti
│   │   │   ├── planes (optional)
```

### 创建 KITTI 数据集

```

为了创建 KITTI 点云数据，首先需要加载原始的点云数据并生成相关的包含目标标签和标注框的数据标注文件，同时还需要为 KITTI 数据集生成每个单独的训练目标的点云数据，并将其存储在 `data/kitti/kitti_gt_database` 的 `.bin` 格式的文件中，此外，需要为训练数据或者验证数据生成 `.pkl` 格式的包含数据信息的文件。随后，通过运行下面的命令来创建最终的 KITTI 数据：

```bash
mkdir ./data/kitti/ && mkdir ./data/kitti/ImageSets

# 下载数据划分
wget -c  https://raw.githubusercontent.com/traveller59/second.pytorch/master/second/data/ImageSets/test.txt --no-check-certificate --content-disposition -O ./data/kitti/ImageSets/test.txt
wget -c  https://raw.githubusercontent.com/traveller59/second.pytorch/master/second/data/ImageSets/train.txt --no-check-certificate --content-disposition -O ./data/kitti/ImageSets/train.txt

wget -c  https://raw.githubusercontent.com/traveller59/second.pytorch/master/second/data/ImageSets/val.txt --no-check-certificate --content-disposition -O ./data/kitti/ImageSets/val.txt
wget -c  https://raw.githubusercontent.com/traveller59/second.pytorch/master/second/data/ImageSets/trainval.txt --no-check-certificate --content-disposition -O ./data/kitti/ImageSets/trainval.txt

python tools/create_data.py kitti --root-path ./data/kitti --out-dir ./data/kitti --extra-tag kitti --with-plane
```

需要注意的是，如果您的本地磁盘没有充足的存储空间来存储转换后的数据，您可以通过改变 `--out-dir` 来指定其他任意的存储路径。如果您没有准备 `planes` 数据，您需要移除 `--with-plane` 标志。

处理后的文件夹结构应该如下：

```
kitti
├── ImageSets
│   ├── test.txt
│   ├── train.txt
│   ├── trainval.txt
│   ├── val.txt
├── testing
│   ├── calib
│   ├── image_2
│   ├── velodyne
│   ├── velodyne_reduced

```
kitti
│   ├── calib
├── training
│   ├── calib
│   ├── label_2
│   ├── planes (optional)
├── kitti_gt_database
├── kitti_infos_train.pkl
├── kitti_infos_val.pkl
├── kitti_dbinfos_train.pkl
├── kitti_infos_test.pkl
├── kitti_infos_trainval.pkl
```

- `kitti_gt_database/xxxxx.bin`：训练数据集中包含在 3D 标注框中的点云数据。
- `kitti_infos_train.pkl`：训练数据集，该字典包含了两个键值：`metainfo` 和 `data_list`。`metainfo` 包含数据集的基本信息，例如 `categories`, `dataset` 和 `info_version`。`data_list` 是由字典组成的列表，每个字典（以下简称 `info`）包含了单个样本的所有详细信息。
  - info\['sample_idx'\]：该样本在整个数据集的索引。
  - info\['images'\]：多个相机捕获的图像信息。是一个字典，包含 5 个键值：`CAM0`, `CAM1`, `CAM2`, `CAM3`, `R0_rect`。
    - info\['images'\]\['R0_rect'\]：校准旋转矩阵，是一个 4x4 数组。
    - info\['images'\]\['CAM2'\]：包含 `CAM2` 相机传感器的信息。

- info\['images'\]：多个相机捕获的图像信息。是一个字典，包含 5 个键值：`CAM0`, `CAM1`, `CAM2`, `CAM3`, `R0_rect`。
      - info\['images'\]\['CAM2'\]\['img_path'\]：图像的文件名。
      - info\['images'\]\['CAM2'\]\['height'\]：图像的高。
      - info\['images'\]\['CAM2'\]\['width'\]：图像的宽。
      - info\['images'\]\['CAM2'\]\['cam2img'\]：相机到图像的变换矩阵，是一个 4x4 数组。
      - info\['images'\]\['CAM2'\]\['lidar2cam'\]：激光雷达到相机的变换矩阵，是一个 4x4 数组。
      - info\['images'\]\['CAM2'\]\['lidar2img'\]：激光雷达到图像的变换矩阵，是一个 4x4 数组。
    - info\['lidar_points'\]：是一个字典，包含了激光雷达点相关的信息。
      - info\['lidar_points'\]\['lidar_path'\]：激光雷达点云数据的文件名。
      - info\['lidar_points'\]\['num_pts_feats'\]：点的特征维度。
      - info\['lidar_points'\]\['Tr_velo_to_cam'\]：Velodyne 坐标到相机坐标的变换矩阵，是一个 4x4 数组。
      - info\['lidar_points'\]\['Tr_imu_to_velo'\]：IMU 坐标到 Velodyne 坐标的变换矩阵，是一个 4x4 数组。
    - info\['instances'\]：是一个字典组成的列表。每个字典包含单个实例的所有标注信息。对于其中的第 i 个实例，我们有：

- info\['lidar_points'\]\['Tr_imu_to_velo'\]：IMU 坐标到 Velodyne 坐标的变换矩阵，是一个 4x4 数组。
      - info\['instances'\]\[i\]\['bbox'\]：长度为 4 的列表，以 (x1, y1, x2, y2) 的顺序表示实例的 2D 边界框。
      - info\['instances'\]\[i\]\['bbox_3d'\]：长度为 7 的列表，以 (x, y, z, l, h, w, yaw) 的顺序表示实例的 3D 边界框。
      - info\['instances'\]\[i\]\['bbox_label'\]：是一个整数，表示实例的 2D 标签，-1 代表忽略。
      - info\['instances'\]\[i\]\['bbox_label_3d'\]：是一个整数，表示实例的 3D 标签，-1 代表忽略。
      - info\['instances'\]\[i\]\['depth'\]：3D 边界框投影到相关图像平面的中心点的深度。
      - info\['instances'\]\[i\]\['num_lidar_pts'\]：3D 边界框内的激光雷达点数。
      - info\['instances'\]\[i\]\['center_2d'\]：3D 边界框投影的 2D 中心。
      - info\['instances'\]\[i\]\['difficulty'\]：KITTI 官方定义的困难度，包括简单、适中、困难。
      - info\['instances'\]\[i\]\['truncated'\]：从 0（非截断）到 1（截断）的浮点数，其中截断指的是离开检测图像边界的检测目标。
      - info\['instances'\]\[i\]\['occluded'\]：整数 (0,1,2,3) 表示目标的遮挡状态：0 = 完全可见，1 = 部分遮挡，2 = 大面积遮挡，3 = 未知。

- info\['instances'\]\[i\]\['truncated'\]：从 0（非截断）到 1（截断）的浮点数，其中截断指的是离开检测图像边界的检测目标。
      - info\['instances'\]\[i\]\['group_ids'\]：用于多部分的物体。
    - info\['plane'\]（可选）：地平面信息。

更多细节请参考 [kitti_converter.py](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/kitti_converter.py) 和 [update_infos_to_v2.py](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/update_infos_to_v2.py)。

## 训练流程

下面展示了一个使用 KITTI 数据集进行 3D 目标检测的典型流程：

```python
    dict(
        load_dim=4, # x, y, z, intensity
    dict(
        type='LoadAnnotations3D', with_bbox_3d=True, with_label_3d=True),
    dict(type='ObjectSample', db_sampler=db_sampler),
    dict(

dict(type='ObjectSample', db_sampler=db_sampler),
    dict(
    dict(
    dict(
]
```

- 数据增强：
  - `ObjectNoise`：对场景中的每个真实标注框目标添加噪音。
  - `RandomFlip3D`：对输入点云数据进行随机地水平翻转或者垂直翻转。
  - `GlobalRotScaleTrans`：对输入点云数据进行旋转。

## 评估

使用 8 个 GPU 以及 KITTI 指标评估的 PointPillars 的示例如下：

```shell

]
```

- 数据增强：

## 评估

```shell
bash tools/dist_test.sh configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py work_dirs/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class/latest.pth 8
```

## 度量指标

KITTI 官方使用全类平均精度（mAP）和平均方向相似度（AOS）来评估 3D 目标检测的性能，更多细节请参考[官方网站](http://www.cvlibs.net/datasets/kitti/eval_3dobject.php)和[论文](http://www.cvlibs.net/publications/Geiger2012CVPR.pdf)。

MMDetection3D 采用相同的方法在 KITTI 数据集上进行评估，下面展示了一个评估结果的例子：

```
Car AP@0.70, 0.70, 0.70:
bbox AP:97.9252, 89.6183, 88.1564
bev  AP:90.4196, 87.9491, 85.1700
3d   AP:88.3891, 77.1624, 74.4654
aos  AP:97.70, 89.11, 87.38
Car AP@0.70, 0.50, 0.50:
bev  AP:98.3509, 90.2042, 89.6102
3d   AP:98.2800, 90.1480, 89.4736
```

## 测试和提交

使用 8 个 GPU 在 KITTI 上测试 PointPillars 并生成对排行榜的提交的示例如下：

```

## 测试和提交

- 首先，你需要在你的配置文件中修改 `test_dataloader` 和 `test_evaluator` 字典，如下所示：

  ```python
  data_root = 'data/kitti/'
  test_dataloader = dict(
          ann_file='kitti_infos_test.pkl',
          load_eval_anns=False,
          data_prefix=dict(pts='testing/velodyne_reduced')))
  test_evaluator = dict(
      ann_file=data_root + 'kitti_infos_test.pkl',
      format_only=True,
      pklfile_prefix='results/kitti-3class/kitti_results',
      submission_prefix='results/kitti-3class/kitti_results')
  ```

- 接下来，你可以运行如下测试脚本。

  ```shell
  ./tools/dist_test.sh configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py work_dirs/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class/latest.pth 8
  ```

```shell
  ```

在生成 `results/kitti-3class/kitti_results/xxxxx.txt` 后，您可以提交这些文件到 KITTI 官方网站进行基准测试，更多细节请参考 [KITTI 官方网站](http://www.cvlibs.net/datasets/kitti/index.php)。

### Prepare S3DIS Data

We follow the procedure in [pointnet](https://github.com/charlesq34/pointnet).

1. Download S3DIS data by filling this [Google form](https://docs.google.com/forms/d/e/1FAIpQLScDimvNMCGhy_rmBA2gHfDu3naktRm6A8BPwAWWDv-Uhm6Shw/viewform?c=0&w=1). Download the `Stanford3dDataset_v1.2_Aligned_Version.zip` file and unzip it. Link or move the folder to this level of directory.

2. In this directory, extract point clouds and annotations by running `python collect_indoor3d_data.py`.

3. Enter the project root directory, generate training data by running

```bash
python tools/create_data.py s3dis --root-path ./data/s3dis --out-dir ./data/s3dis --extra-tag s3dis
```

The overall process could be achieved through the following script

```bash
python collect_indoor3d_data.py
cd ../..
```

The directory structure after pre-processing should be as below

```
s3dis

```

```
s3dis
├── indoor3d_util.py
├── collect_indoor3d_data.py
├── Stanford3dDataset_v1.2_Aligned_Version
├── s3dis_data
├── points
│   ├── Area_1_label_weight.npy
│   ├── Area_1_resampled_scene_idxs.npy
│   ├── Area_2_label_weight.npy
│   ├── Area_2_resampled_scene_idxs.npy
│   ├── Area_3_label_weight.npy
│   ├── Area_3_resampled_scene_idxs.npy
│   ├── Area_4_label_weight.npy
│   ├── Area_4_resampled_scene_idxs.npy
│   ├── Area_5_label_weight.npy
│   ├── Area_5_resampled_scene_idxs.npy
│   ├── Area_6_label_weight.npy
│   ├── Area_6_resampled_scene_idxs.npy
├── s3dis_infos_Area_1.pkl
├── s3dis_infos_Area_2.pkl
├── s3dis_infos_Area_3.pkl
├── s3dis_infos_Area_4.pkl
├── s3dis_infos_Area_5.pkl

├── s3dis_infos_Area_6.pkl

```

# Customize Losses

MMDetection provides users with different loss functions. But the default configuration may be not applicable for different datasets or models, so users may want to modify a specific loss to adapt the new situation.

This tutorial first elaborate the computation pipeline of losses, then give some instructions about how to modify each step. The modification can be categorized as tweaking and weighting.

## Computation pipeline of a loss

Given the input prediction and target, as well as the weights, a loss function maps the input tensor to the final loss scalar. The mapping can be divided into five steps:

1. Set the sampling method to sample positive and negative samples.

2. Get **element-wise** or **sample-wise** loss by the loss kernel function.

3. Weighting the loss with a weight tensor **element-wisely**.

4. Reduce the loss tensor to a **scalar**.

5. Weighting the loss with a **scalar**.

## Set sampling method (step 1)

For some loss functions, sampling strategies are needed to avoid imbalance between positive and negative samples.

For example, when using `CrossEntropyLoss` in RPN head, we need to set `RandomSampler` in `train_cfg`

```python
    rpn=dict(
            num=256,
```

For some other losses which have positive and negative sample balance mechanism such as Focal Loss, GHMC, and QualityFocalLoss, the sampler is no more necessary.

## Tweaking loss

Tweaking a loss is more related with step 2, 4, 5, and most modifications can be specified in the config.
Here we take [Focal Loss (FL)](../../../mmdet/models/losses/focal_loss.py) as an example.

The following code sniper are the construction method and config of FL respectively, they are actually one to one correspondence.

```python

```

```python
```

### Tweaking hyper-parameters (step 2)

`gamma` and `beta` are two hyper-parameters in the Focal Loss. Say if we want to change the value of `gamma` to be 1.5 and `alpha` to be 0.5, then we can specify them in the config as follows:

```python

```python
```

### Tweaking the way of reduction (step 3)

The default way of reduction is `mean` for FL. Say if we want to change the reduction from `mean` to `sum`, we can specify it in the config as follows:

```python
```

### Tweaking loss weight (step 5)

The loss weight here is a scalar which controls the weight of different losses in multi-task learning, e.g. classification loss and regression loss. Say if we want to change to loss weight of classification loss to be 0.5, we can specify it in the config as follows:

```python
```

## Weighting loss (step 3)

```python
```

Weighting loss means we re-weight the loss element-wisely. To be more specific, we multiply the loss tensor with a weight tensor which has the same shape. As a result, different entries of the loss can be scaled differently, and so called element-wisely.
The loss weight varies across different models and highly context related, but overall there are two kinds of loss weights, `label_weights` for classification loss and `bbox_weights` for bbox regression loss. You can find them in the `get_target` method of the corresponding head. Here we take [ATSSHead](../../../mmdet/models/dense_heads/atss_head.py#L322) as an example, which inherit [AnchorHead](../../../mmdet/models/dense_heads/anchor_head.py) but overwrite its `get_targets` method which yields different `label_weights` and `bbox_weights`.

```

    ...

```

    ...

```

# DenseNet

> [Densely Connected Convolutional Networks](https://arxiv.org/abs/1608.06993)

Recent work has shown that convolutional networks can be substantially deeper, more accurate, and efficient to train if they contain shorter connections between layers close to the input and those close to the output. In this paper, we embrace this observation and introduce the Dense Convolutional Network (DenseNet), which connects each layer to every layer in a feed-forward fashion. Whereas traditional convolutional networks with L layers have L connections - one between each layer and its subsequent layer - our network has L(L+1)/2 direct connections. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers. DenseNets have several compelling advantages: they alleviate the vanishing-gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters. We evaluate our proposed architecture on four highly competitive object recognition benchmark tasks (CIFAR-10, CIFAR-100, SVHN, and ImageNet). DenseNets obtain significant improvements over the state-of-the-art on most of them, whilst requiring less computation to achieve high performance.

<img src="https://user-images.githubusercontent.com/42952108/162675098-9a670883-b13a-4a5a-a9c9-06c39c616a0a.png" width="100%"/>
</div>

```python

predict = inference_model('densenet121_3rdparty_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('densenet121_3rdparty_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/densenet/densenet121_4xb256_in1k.py https://download.openmmlab.com/mmclassification/v0/densenet/densenet121_4xb256_in1k_20220426-07450f99.pth
```

| Model                         |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                Config                |                                        Download                                        |
| :---------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :----------------------------------: | :------------------------------------------------------------------------------------: |

| `densenet121_3rdparty_in1k`\* | From scratch |    7.98    |   2.88    |   74.96   |   92.21   | [config](densenet121_4xb256_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/densenet/densenet121_4xb256_in1k_20220426-07450f99.pth) |
| `densenet169_3rdparty_in1k`\* | From scratch |   14.15    |   3.42    |   76.08   |   93.11   | [config](densenet169_4xb256_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/densenet/densenet169_4xb256_in1k_20220426-a2889902.pth) |
| `densenet201_3rdparty_in1k`\* | From scratch |   20.01    |   4.37    |   77.32   |   93.64   | [config](densenet201_4xb256_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/densenet/densenet201_4xb256_in1k_20220426-05cae4ef.pth) |

| `densenet161_3rdparty_in1k`\* | From scratch |   28.68    |   7.82    |   77.61   |   93.83   | [config](densenet161_4xb256_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/densenet/densenet161_4xb256_in1k_20220426-ee6a80a9.pth) |

*Models with * are converted from the [official repo](https://github.com/pytorch/vision/blob/main/torchvision/models/densenet.py). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@misc{https://doi.org/10.48550/arxiv.1608.06993,
      doi = {10.48550/ARXIV.1608.06993},
      url = {https://arxiv.org/abs/1608.06993},
      author = {Huang, Gao and Liu, Zhuang and van der Maaten, Laurens and Weinberger, Kilian Q.},
      keywords = {Computer Vision and Pattern Recognition (cs.CV), Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},
      title = {Densely Connected Convolutional Networks},

title = {Densely Connected Convolutional Networks},
      publisher = {arXiv},
      year = {2016},
      copyright = {arXiv.org perpetual, non-exclusive license}
}
```

# 分析工具

除了训练/测试脚本外，我们在 `tools/` 目录下还提供了许多有用的工具。

## 分析工具链接

- [](#分析工具)
  - [分析工具](#分析工具)
    - [准备模型进行发布](#准备模型进行发布)
  - [杂项](#杂项)
    - [评估指标](#评估指标)
    - [打印完整配置](#打印完整配置)
    - [检查视频](#检查视频)
    - [多流融合](#多流融合)

## 模型转换

### 准备模型进行发布

`tools/deployment/publish_model.py` 帮助用户准备他们的模型进行发布。

在将模型上传到 AWS 之前，您可能想要：

（1）将模型权重转换为 CPU 张量。
（2）删除优化器状态信息。
（3）计算权重文件的哈希值，并将哈希值添加到文件名中。

```shell
python tools/deployment/publish_model.py ${INPUT_FILENAME} ${OUTPUT_FILENAME}
```

例如，

```shell
python tools/deployment/publish_model.py work_dirs/tsn_r50_8xb32-1x1x3-100e_kinetics400-rgb/latest.pth tsn_r50_1x1x3_100e_kinetics400_rgb.pth
```

最终输出的文件名将是 `tsn_r50_8xb32-1x1x3-100e_kinetics400-rgb-{hash id}.pth`。

## 杂项

### 评估指标

`tools/analysis_tools/eval_metric.py` 根据配置文件评估保存在文件中的结果的某些指标。

保存的结果文件是通过在 `tools/test.py` 中设置参数 `--out ${RESULT_FILE}` 来创建的，以指示结果文件，其中存储了整个模型的最终输出。

```shell

## 杂项

### 评估指标

```shell
python tools/analysis/eval_metric.py ${CONFIG_FILE} ${RESULT_FILE} [--eval ${EVAL_METRICS}] [--cfg-options ${CFG_OPTIONS}] [--eval-options ${EVAL_OPTIONS}]
```

### 打印完整配置

`tools/analysis_tools/print_config.py` 逐字打印整个配置，展开所有导入项。

```shell
python tools/analysis_tools/print_config.py ${CONFIG} [-h] [--options ${OPTIONS [OPTIONS...]}]
```

### 检查视频

`tools/analysis_tools/check_videos.py` 使用指定的视频编码器迭代由输入配置文件指定的所有样本，查找无效的视频（损坏或缺失），并将相应的文件路径保存到输出文件中。请注意，删除无效视频后，用户需要重新生成视频文件列表。

```shell
python tools/analysis_tools/check_videos.py ${CONFIG} [-h] [--options OPTIONS [OPTIONS ...]] [--cfg-options CFG_OPTIONS [CFG_OPTIONS ...]] [--output-file OUTPUT_FILE] [--split SPLIT] [--decoder DECODER] [--num-processes NUM_PROCESSES] [--remove-corrupted-videos]
```

### 多流融合

```

### 多流融合

`tools/analysis_tools/report_accuracy.py` 使用推理保存的结果（在测试时设置 `--dump res.pkl`）来融合多流预测分数，即后融合（late fusion）。

```shell
python tools/analysis_tools/report_accuracy.py [--preds ${RESULT_PKL_1 [RESULT_PKL_2 ...]}] [--coefficients ${COEFFICIENT_1 [COEFFICIENT_2, ...]}] [--apply-softmax]
```

以 joint-bone 融合为例，这是基于骨骼动作识别任务的一种常见实践。

```shell
python tools/analysis_tools/report_accuracy.py --preds demo/fuse/joint.pkl demo/fuse/bone.pkl --coefficients 1.0 1.0
```

```
Mean Class Accuracy: 0.9180
Top 1 Accuracy: 0.9333
Top 5 Accuracy: 0.9833
```

# Stacked hourglass networks for human pose estimation

<details>
<summary align="right"><a href="https://link.springer.com/chapter/10.1007/978-3-319-46484-8_29">Hourglass (ECCV'2016)</a></summary>

```bibtex
@inproceedings{newell2016stacked,
  title={Stacked hourglass networks for human pose estimation},
  author={Newell, Alejandro and Yang, Kaiyu and Deng, Jia},
  pages={483--499},
  year={2016},
}
```

}
```

This work introduces a novel convolutional network architecture for the task of human pose estimation. Features are processed across all scales and consolidated to best capture the various spatial relationships associated with the body. We show how repeated bottom-up, top-down processing used in conjunction with intermediate supervision is critical to improving the performance of the network. We refer to the architecture as a "stacked hourglass" network based on the successive steps of pooling and upsampling that are done to produce a final set of predictions. State-of-the-art results are achieved on the FLIC and MPII benchmarks outcompeting all recent methods.

<img src="https://user-images.githubusercontent.com/15977946/146516901-cee086aa-6b7c-4f79-82bd-1168f03ea63b.png">
</div>

# BSN

[Bsn: Boundary sensitive network for temporal action proposal generation](https://openaccess.thecvf.com/content_ECCV_2018/html/Tianwei_Lin_BSN_Boundary_Sensitive_ECCV_2018_paper.html)

Temporal action proposal generation is an important yet challenging problem, since temporal proposals with rich action content are indispensable for analysing real-world videos with long duration and high proportion irrelevant content. This problem requires methods not only generating proposals with precise temporal boundaries, but also retrieving proposals to cover truth action instances with high recall and high overlap using relatively fewer proposals. To address these difficulties, we introduce an effective proposal generation method, named Boundary-Sensitive Network (BSN), which adopts "local to global" fashion. Locally, BSN first locates temporal boundaries with high probabilities, then directly combines these boundaries as proposals. Globally, with Boundary-Sensitive Proposal feature, BSN retrieves proposals by evaluating the confidence of whether a proposal contains an action within its region. We conduct experiments on two challenging datasets: ActivityNet-1.3 and THUMOS14, where BSN outperforms other state-of-the-art temporal action proposal generation methods with high recall and high temporal precision. Finally, further experiments demonstrate that by combining existing action classifiers, our method significantly improves the state-of-the-art temporal action detection performance.

<img src="https://user-images.githubusercontent.com/34324155/143016692-69efafbd-cec6-47f1-af45-371d0ff78a97.png" width="800"/>
</div>

### ActivityNet feature

|    feature    | gpus | pretrain |  AUC  | AR@1  | AR@5  | AR@10 | AR@100 |   gpu_mem(M)    | iter time(s) |                   config                   |                   ckpt                   |                   log                    |
| :-----------: | :--: | :------: | :---: | :---: | :---: | :---: | :----: | :-------------: | :----------: | :----------------------------------------: | :--------------------------------------: | :--------------------------------------: |

| cuhk_mean_100 |  1   |   None   | 66.26 | 32.71 | 48.43 | 55.28 | 74.27  | 43(TEM)+25(PEM) |      -       | [config_TEM](/configs/localization/bsn/bsn_tem_1xb16-400x100-20e_activitynet-feature.py) [config_PGM](/configs/localization/bsn/bsn_pgm_400x100_activitynet-feature.py) [config_PEM](/configs/localization/bsn/bsn_pem_1xb16-400x100-20e_activitynet-feature.py) | [ckpt_TEM](https://download.openmmlab.com/mmaction/v1.0/localization/bsn/bsn_tem_1xb16-400x100-20e_activitynet-feature_20220908-9da79951.pth) [ckpt_PEM](https://download.openmmlab.com/mmaction/v1.0/localization/bsn/bsn_pem_1xb16-400x100-20e_activitynet-feature_20220908-ec2eb21d.pth) | [log_tem](https://download.openmmlab.com/mmaction/v1.0/localization/bsn/bsn_tem_1xb16-400x100-20e_activitynet-feature.log) [log_pem](https://download.openmmlab.com/mmaction/v1.0/localization/bsn/bsn_pem_1xb16-400x100-20e_activitynet-feature.log) |

1. The **gpus** indicates the number of gpu we used to get the checkpoint.

   According to the [Linear Scaling Rule](https://arxiv.org/abs/1706.02677), you may set the learning rate proportional to the batch size if you use different GPUs or videos per GPU,
   e.g., lr=0.01 for 4 GPUs x 2 video/gpu and lr=0.08 for 16 GPUs x 4 video/gpu.
2. For feature column, cuhk_mean_100 denotes the widely used cuhk activitynet feature extracted by [anet2016-cuhk](https://github.com/yjxiong/anet2016-cuhk).

For more details on data preparation, you can refer to [ActivityNet Data Preparation](/tools/data/activitynet/README.md).

## Training and Test

The traing of the BSN model is three-stages. Firstly train the Temporal evaluation module (TEM):

```shell
python3 tools/train.py configs/localization/bsn/bsn_tem_1xb16-400x100-20e_activitynet-feature.py
```

```shell
```

After training use the TEM module to generate the probabilities sequence (actionness, starting, and ending) for the training and validation dataset:

```shell
python tools/test.py configs/localization/bsn/bsn_tem_1xb16-400x100-20e_activitynet-feature.py \
    work_dirs/bsn_400x100_20e_1xb16_activitynet_feature/tem_epoch_20.pth
```

The second step is to run the Proposal generation module (PGM) to generate Boundary-Sensitive Proposal (BSP) feature for the training and validation dataset:

```shell
python tools/misc/bsn_proposal_generation.py configs/localization/bsn/bsn_pgm_400x100_activitynet-feature.py --mode train
python tools/misc/bsn_proposal_generation.py configs/localization/bsn/bsn_pgm_400x100_activitynet-feature.py --mode test
```

The last step is to train (and validate) the Proposal evaluation module (PEM):

```shell

```

```shell
python python tools/train.py configs/localization/bsn/bsn_pem_1xb16-400x100-20e_activitynet-feature.py
```

(Optional) You can use the following command to generate a formatted proposal file, which will be fed into the action classifier (Currently supports only SSN and P-GCN, not including TSN, I3D etc.) to get the classification result of proposals.

```shell
python tools/data/activitynet/convert_proposal_format.py
```

```BibTeX
@inproceedings{lin2018bsn,
  title={Bsn: Boundary sensitive network for temporal action proposal generation},
  author={Lin, Tianwei and Zhao, Xu and Su, Haisheng and Wang, Chongjing and Yang, Ming},
  pages={3--19},
}
```

  <img src="resources/mmpose-logo.png" width="450"/>
    <sup>
      </a>
    </sup>
    <sup>
      </a>
    </sup>
  </div>

[![Documentation](https://readthedocs.org/projects/mmpose/badge/?version=latest)](https://mmpose.readthedocs.io/en/latest/?badge=latest)
[![actions](https://github.com/open-mmlab/mmpose/workflows/build/badge.svg)](https://github.com/open-mmlab/mmpose/actions)
[![codecov](https://codecov.io/gh/open-mmlab/mmpose/branch/latest/graph/badge.svg)](https://codecov.io/gh/open-mmlab/mmpose)
[![PyPI](https://img.shields.io/pypi/v/mmpose)](https://pypi.org/project/mmpose/)

[![LICENSE](https://img.shields.io/github/license/open-mmlab/mmpose.svg)](https://github.com/open-mmlab/mmpose/blob/main/LICENSE)
[![Average time to resolve an issue](https://isitmaintained.com/badge/resolution/open-mmlab/mmpose.svg)](https://github.com/open-mmlab/mmpose/issues)
[![Percentage of issues still open](https://isitmaintained.com/badge/open/open-mmlab/mmpose.svg)](https://github.com/open-mmlab/mmpose/issues)

[📘文档](https://mmpose.readthedocs.io/zh_CN/latest/) |
[🛠️安装](https://mmpose.readthedocs.io/zh_CN/latest/installation.html) |
[👀模型库](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo.html) |
[📜论文库](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/algorithms.html) |
[🆕更新日志](https://mmpose.readthedocs.io/zh_CN/latest/notes/changelog.html) |
[🤔报告问题](https://github.com/open-mmlab/mmpose/issues/new/choose) |
[🔥RTMPose](/projects/rtmpose/)

</div>

</div>

  <a href="https://discord.com/channels/1037617289144569886/1072798105428299817" style="text-decoration:none;">

<a href="https://twitter.com/OpenMMLab" style="text-decoration:none;">

<img src="https://user-images.githubusercontent.com/25839884/219026751-d7d14cce-a7c9-4e82-9942-8375fca65b99.png" width="3%" alt="" /></a>
</div>

[English](./README.md) | 简体中文

MMPose 是一款基于 PyTorch 的姿态分析的开源工具箱，是 [OpenMMLab](https://github.com/open-mmlab) 项目的成员之一。

主分支代码目前支持 **PyTorch 1.8 以上**的版本。

https://user-images.githubusercontent.com/15977946/124654387-0fd3c500-ded1-11eb-84f6-24eeddbf4d91.mp4

<summary><b>主要特性</b></summary>

- **支持多种人体姿态分析相关任务**

  MMPose 支持当前学界广泛关注的主流姿态分析任务：主要包括 2D多人姿态估计、2D手部姿态估计、2D人脸关键点检测、133关键点的全身人体姿态估计、3D人体形状恢复、服饰关键点检测、动物关键点检测等。
  具体请参考 [功能演示](demo/docs/zh_cn/)。

- **更高的精度和更快的速度**

  MMPose 复现了多种学界最先进的人体姿态分析模型，包括“自顶向下”和“自底向上”两大类算法。MMPose 相比于其他主流的代码库，具有更高的模型精度和训练速度。
  具体请参考 [基准测试](docs/en/notes/benchmark.md)（英文）。

- **支持多样的数据集**

  MMPose 支持了很多主流数据集的准备和构建，如 COCO、 MPII 等。 具体请参考 [数据集](docs/zh_cn/dataset_zoo)。

  MMPose 将统一的人体姿态分析框架解耦成不同的模块组件，通过组合不同的模块组件，用户可以便捷地构建自定义的人体姿态分析模型。

- **详尽的单元测试和文档**

  MMPose 提供了详尽的说明文档，API 接口说明，全面的单元测试，以供社区参考。

## 最新进展

- 我们支持了三个新的数据集:
  - (CVPR 2023) [Human-Art](https://github.com/IDEA-Research/HumanArt)
  - (CVPR 2022) [Animal Kingdom](https://github.com/sutdcv/Animal-Kingdom)
  - (AAAI 2020) [LaPa](https://github.com/JDAI-CV/lapa-dataset/)

![yolox-pose_intro](https://user-images.githubusercontent.com/26127467/226655503-3cee746e-6e42-40be-82ae-6e7cae2a4c7e.jpg)

- (AAAI 2020) [LaPa](https://github.com/JDAI-CV/lapa-dataset/)

- 欢迎使用 [*MMPose 项目*](/projects/README.md)。在这里，您可以发现 MMPose 中的最新功能和算法，并且可以通过最快的方式与社区分享自己的创意和代码实现。向 MMPose 中添加新功能从此变得简单丝滑：

  - 提供了一种简单迅捷的方式，将新的算法、功能和应用添加到 MMPose 中
  - 更灵活的代码结构和风格，更少的限制，更简短的代码审核流程
  - 通过独立项目的形式，利用 MMPose 的强大功能，同时不被代码框架所束缚
  - 最新添加的项目包括：
    - [RTMPose](/projects/rtmpose/)
    - [YOLOX-Pose](/projects/yolox_pose/)
    - [MMPose4AIGC](/projects/mmpose4aigc/)
    - [Simple Keypoints](/projects/skps/)
  - 从简单的 [示例项目](/projects/example_project/) 开启您的 MMPose 代码贡献者之旅吧，让我们共同打造更好用的 MMPose！

<br/>

- 2023-07-04：MMPose [v1.1.0](https://github.com/open-mmlab/mmpose/releases/tag/v1.1.0) 正式发布了，主要更新包括:

  - 支持新数据集：Human-Art、Animal Kingdom、LaPa。
  - 支持新的配置文件风格，支持 IDE 跳转和搜索。
  - 提供更强性能的 RTMPose 模型。
  - 迁移 3D 姿态估计算法。
  - 加速推理脚本，全部 demo 脚本支持摄像头推理。

- 支持新数据集：Human-Art、Animal Kingdom、LaPa。

  请查看完整的 [版本说明](https://github.com/open-mmlab/mmpose/releases/tag/v1.1.0) 以了解更多 MMPose v1.1.0 带来的更新!

## 0.x / 1.x 迁移

MMPose v1.0.0 是一个重大更新，包括了大量的 API 和配置文件的变化。目前 v1.0.0 中已经完成了一部分算法的迁移工作，剩余的算法将在后续的版本中陆续完成，我们将在下面的列表中展示迁移进度。

<summary><b>迁移进度</b></summary>

| 算法名称                          |  迁移进度   |
| :-------------------------------- | :---------: |
| MTUT (CVPR 2019)                  |             |
| MSPN (ArXiv 2019)                 |    done     |
| InterNet (ECCV 2020)              |             |
| DEKR (CVPR 2021)                  |    done     |
| HigherHRNet (CVPR 2020)           |             |
| DeepPose (CVPR 2014)              |    done     |
| RLE (ICCV 2021)                   |    done     |
| SoftWingloss (TIP 2021)           |    done     |
| VideoPose3D (CVPR 2019)           |    done     |

| Hourglass (ECCV 2016)             |    done     |
| LiteHRNet (CVPR 2021)             |    done     |
| AdaptiveWingloss (ICCV 2019)      |    done     |
| SimpleBaseline2D (ECCV 2018)      |    done     |
| PoseWarper (NeurIPS 2019)         |             |
| SimpleBaseline3D (ICCV 2017)      |    done     |
| HMR (CVPR 2018)                   |             |
| UDP (CVPR 2020)                   |    done     |
| VIPNAS (CVPR 2021)                |    done     |
| Wingloss (CVPR 2018)              |    done     |
| DarkPose (CVPR 2020)              |    done     |
| Associative Embedding (NIPS 2017) | in progress |
| VoxelPose (ECCV 2020)             |             |
| RSN (ECCV 2020)                   |    done     |
| CID (CVPR 2022)                   |    done     |
| CPM (CVPR 2016)                   |    done     |

| HRNet (CVPR 2019)                 |    done     |
| HRNetv2 (TPAMI 2019)              |    done     |
| SCNet (CVPR 2020)                 |    done     |

如果您使用的算法还没有完成迁移，您也可以继续使用访问 [0.x 分支](https://github.com/open-mmlab/mmpose/tree/0.x) 和 [旧版文档](https://mmpose.readthedocs.io/zh_CN/0.x/)

## 安装

关于安装的详细说明请参考[安装文档](https://mmpose.readthedocs.io/zh_CN/latest/installation.html)。

## 教程

我们提供了一系列简明的教程，帮助 MMPose 的新用户轻松上手使用：

1. MMPose 的基本使用方法：

   - [20 分钟上手教程](https://mmpose.readthedocs.io/zh_CN/latest/guide_to_framework.html)
   - [Demos](https://mmpose.readthedocs.io/zh_CN/latest/demos.html)
   - [模型推理](https://mmpose.readthedocs.io/zh_CN/latest/user_guides/inference.html)
   - [配置文件](https://mmpose.readthedocs.io/zh_CN/latest/user_guides/configs.html)

- [模型推理](https://mmpose.readthedocs.io/zh_CN/latest/user_guides/inference.html)
   - [准备数据集](https://mmpose.readthedocs.io/zh_CN/latest/user_guides/prepare_datasets.html)
   - [训练与测试](https://mmpose.readthedocs.io/zh_CN/latest/user_guides/train_and_test.html)

2. 对于希望基于 MMPose 进行开发的研究者和开发者：

   - [编解码器](https://mmpose.readthedocs.io/zh_CN/latest/advanced_guides/codecs.html)
   - [数据流](https://mmpose.readthedocs.io/zh_CN/latest/advanced_guides/dataflow.html)
   - [实现新模型](https://mmpose.readthedocs.io/zh_CN/latest/advanced_guides/implement_new_models.html)
   - [自定义数据集](https://mmpose.readthedocs.io/zh_CN/latest/advanced_guides/customize_datasets.html)
   - [自定义数据变换](https://mmpose.readthedocs.io/zh_CN/latest/advanced_guides/customize_transforms.html)
   - [自定义优化器](https://mmpose.readthedocs.io/zh_CN/latest/advanced_guides/customize_optimizer.html)

- [自定义数据变换](https://mmpose.readthedocs.io/zh_CN/latest/advanced_guides/customize_transforms.html)
   - [自定义日志](https://mmpose.readthedocs.io/zh_CN/latest/advanced_guides/customize_logging.html)
   - [模型部署](https://mmpose.readthedocs.io/zh_CN/latest/advanced_guides/how_to_deploy.html)
   - [模型分析工具](https://mmpose.readthedocs.io/zh_CN/latest/advanced_guides/model_analysis.html)
   - [迁移指南](https://mmpose.readthedocs.io/zh_CN/latest/migration.html)

3. 对于希望加入开源社区，向 MMPose 贡献代码的研究者和开发者：

   - [参与贡献代码](https://mmpose.readthedocs.io/zh_CN/latest/contribution_guide.html)

4. 对于使用过程中的常见问题：

   - [FAQ](https://mmpose.readthedocs.io/zh_CN/latest/faq.html)

## 模型库

各个模型的结果和设置都可以在对应的 config（配置）目录下的 **README.md** 中查看。
整体的概况也可也在 [模型库](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo.html) 页面中查看。

<summary><b>支持的算法</b></summary>

## 模型库

- [x] [DeepPose](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/algorithms.html#deeppose-cvpr-2014) (CVPR'2014)
- [x] [CPM](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#cpm-cvpr-2016) (CVPR'2016)
- [x] [Hourglass](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#hourglass-eccv-2016) (ECCV'2016)
- [x] [SimpleBaseline3D](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/algorithms.html#simplebaseline3d-iccv-2017) (ICCV'2017)
- [ ] [Associative Embedding](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/algorithms.html#associative-embedding-nips-2017) (NeurIPS'2017)
- [x] [SimpleBaseline2D](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/algorithms.html#simplebaseline2d-eccv-2018) (ECCV'2018)

- [x] [DSNT](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/algorithms.html#dsnt-2018) (ArXiv'2021)
- [x] [HRNet](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#hrnet-cvpr-2019) (CVPR'2019)
- [x] [IPR](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/algorithms.html#ipr-eccv-2018) (ECCV'2018)
- [x] [VideoPose3D](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/algorithms.html#videopose3d-cvpr-2019) (CVPR'2019)
- [x] [HRNetv2](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#hrnetv2-tpami-2019) (TPAMI'2019)
- [x] [MSPN](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#mspn-arxiv-2019) (ArXiv'2019)
- [x] [SCNet](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#scnet-cvpr-2020) (CVPR'2020)

- [ ] [HigherHRNet](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#higherhrnet-cvpr-2020) (CVPR'2020)
- [x] [RSN](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#rsn-eccv-2020) (ECCV'2020)
- [ ] [InterNet](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/algorithms.html#internet-eccv-2020) (ECCV'2020)
- [ ] [VoxelPose](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/algorithms.html#voxelpose-eccv-2020) (ECCV'2020)
- [x] [LiteHRNet](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#litehrnet-cvpr-2021) (CVPR'2021)
- [x] [ViPNAS](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#vipnas-cvpr-2021) (CVPR'2021)
- [x] [Debias-IPR](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/algorithms.html#debias-ipr-iccv-2021) (ICCV'2021)

- [x] [SimCC](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/algorithms.html#simcc-eccv-2022) (ECCV'2022)

<summary><b>支持的技术</b></summary>

- [x] [FPN](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/techniques.html#fpn-cvpr-2017) (CVPR'2017)
- [x] [FP16](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/techniques.html#fp16-arxiv-2017) (ArXiv'2017)
- [x] [Wingloss](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/techniques.html#wingloss-cvpr-2018) (CVPR'2018)
- [x] [AdaptiveWingloss](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/techniques.html#adaptivewingloss-iccv-2019) (ICCV'2019)
- [x] [DarkPose](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/techniques.html#darkpose-cvpr-2020) (CVPR'2020)

- [x] [UDP](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/techniques.html#udp-cvpr-2020) (CVPR'2020)
- [x] [Albumentations](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/techniques.html#albumentations-information-2020) (Information'2020)
- [x] [SoftWingloss](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/techniques.html#softwingloss-tip-2021) (TIP'2021)
- [x] [RLE](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/techniques.html#rle-iccv-2021) (ICCV'2021)

<summary><b>支持的数据集</b></summary>

- [x] [AFLW](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#aflw-iccvw-2011) \[[主页](https://www.tugraz.at/institute/icg/research/team-bischof/lrs/downloads/aflw/)\] (ICCVW'2011)

- [x] [sub-JHMDB](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#jhmdb-iccv-2013) \[[主页](http://jhmdb.is.tue.mpg.de/dataset)\] (ICCV'2013)
- [x] [COFW](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#cofw-iccv-2013) \[[主页](http://www.vision.caltech.edu/xpburgos/ICCV13/)\] (ICCV'2013)
- [x] [MPII](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#mpii-cvpr-2014) \[[主页](http://human-pose.mpi-inf.mpg.de/)\] (CVPR'2014)
- [x] [Human3.6M](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#human3-6m-tpami-2014) \[[主页](http://vision.imar.ro/human3.6m/description.php)\] (TPAMI'2014)
- [x] [COCO](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#coco-eccv-2014) \[[主页](http://cocodataset.org/)\] (ECCV'2014)
- [x] [CMU Panoptic](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#cmu-panoptic-iccv-2015) (ICCV'2015)

- [x] [DeepFashion](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#deepfashion-cvpr-2016) \[[主页](http://mmlab.ie.cuhk.edu.hk/projects/DeepFashion/LandmarkDetection.html)\] (CVPR'2016)
- [x] [300W](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#300w-imavis-2016) \[[主页](https://ibug.doc.ic.ac.uk/resources/300-W/)\] (IMAVIS'2016)
- [x] [RHD](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#rhd-iccv-2017) \[[主页](https://lmb.informatik.uni-freiburg.de/resources/datasets/RenderedHandposeDataset.en.html)\] (ICCV'2017)
- [x] [CMU Panoptic](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#cmu-panoptic-iccv-2015) \[[主页](http://domedb.perception.cs.cmu.edu/)\] (ICCV'2015)

- [x] [AI Challenger](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#ai-challenger-arxiv-2017) \[[主页](https://github.com/AIChallenger/AI_Challenger_2017)\] (ArXiv'2017)
- [x] [MHP](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#mhp-acm-mm-2018) \[[主页](https://lv-mhp.github.io/dataset)\] (ACM MM'2018)
- [x] [WFLW](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#wflw-cvpr-2018) \[[主页](https://wywu.github.io/projects/LAB/WFLW.html)\] (CVPR'2018)
- [x] [PoseTrack18](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#posetrack18-cvpr-2018) \[[主页](https://posetrack.net/users/download.php)\] (CVPR'2018)

- [x] [OCHuman](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#ochuman-cvpr-2019) \[[主页](https://github.com/liruilong940607/OCHumanApi)\] (CVPR'2019)
- [x] [CrowdPose](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#crowdpose-cvpr-2019) \[[主页](https://github.com/Jeff-sjtu/CrowdPose)\] (CVPR'2019)
- [x] [MPII-TRB](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#mpii-trb-iccv-2019) \[[主页](https://github.com/kennymckormick/Triplet-Representation-of-human-Body)\] (ICCV'2019)
- [x] [FreiHand](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#freihand-iccv-2019) \[[主页](https://lmb.informatik.uni-freiburg.de/projects/freihand/)\] (ICCV'2019)

- [x] [Animal-Pose](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#animal-pose-iccv-2019) \[[主页](https://sites.google.com/view/animal-pose/)\] (ICCV'2019)
- [x] [OneHand10K](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#onehand10k-tcsvt-2019) \[[主页](https://www.yangangwang.com/papers/WANG-MCC-2018-10.html)\] (TCSVT'2019)
- [x] [Vinegar Fly](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#vinegar-fly-nature-methods-2019) \[[主页](https://github.com/jgraving/DeepPoseKit-Data)\] (Nature Methods'2019)
- [x] [Desert Locust](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#desert-locust-elife-2019) \[[主页](https://github.com/jgraving/DeepPoseKit-Data)\] (Elife'2019)

- [x] [Grévy’s Zebra](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#grevys-zebra-elife-2019) \[[主页](https://github.com/jgraving/DeepPoseKit-Data)\] (Elife'2019)
- [x] [ATRW](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#atrw-acm-mm-2020) \[[主页](https://cvwc2019.github.io/challenge.html)\] (ACM MM'2020)
- [x] [Halpe](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#halpe-cvpr-2020) \[[主页](https://github.com/Fang-Haoshu/Halpe-FullBody/)\] (CVPR'2020)
- [x] [COCO-WholeBody](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#coco-wholebody-eccv-2020) \[[主页](https://github.com/jin-s13/COCO-WholeBody/)\] (ECCV'2020)

- [x] [MacaquePose](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#macaquepose-biorxiv-2020) \[[主页](http://www.pri.kyoto-u.ac.jp/datasets/macaquepose/index.html)\] (bioRxiv'2020)
- [x] [InterHand2.6M](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#interhand2-6m-eccv-2020) \[[主页](https://mks0601.github.io/InterHand2.6M/)\] (ECCV'2020)
- [x] [AP-10K](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#ap-10k-neurips-2021) \[[主页](https://github.com/AlexTheBad/AP-10K)\] (NeurIPS'2021)
- [x] [Horse-10](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#horse-10-wacv-2021) \[[主页](http://www.mackenziemathislab.org/horse10)\] (WACV'2021)

- [x] [Human-Art](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#human-art-cvpr-2023) \[[主页](https://idea-research.github.io/HumanArt/)\] (CVPR'2023)
- [x] [LaPa](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/datasets.html#lapa-aaai-2020) \[[主页](https://github.com/JDAI-CV/lapa-dataset)\] (AAAI'2020)

<summary><b>支持的骨干网络</b></summary>

- [x] [AlexNet](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#alexnet-neurips-2012) (NeurIPS'2012)
- [x] [VGG](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#vgg-iclr-2015) (ICLR'2015)
- [x] [ResNet](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#resnet-cvpr-2016) (CVPR'2016)

- [x] [ResNext](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#resnext-cvpr-2017) (CVPR'2017)
- [x] [SEResNet](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#seresnet-cvpr-2018) (CVPR'2018)
- [x] [ShufflenetV1](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#shufflenetv1-cvpr-2018) (CVPR'2018)
- [x] [ShufflenetV2](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#shufflenetv2-eccv-2018) (ECCV'2018)
- [x] [MobilenetV2](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#mobilenetv2-cvpr-2018) (CVPR'2018)
- [x] [ResNetV1D](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#resnetv1d-cvpr-2019) (CVPR'2019)

- [x] [ResNeSt](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#resnest-arxiv-2020) (ArXiv'2020)
- [x] [Swin](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#swin-cvpr-2021) (CVPR'2021)
- [x] [HRFormer](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#hrformer-nips-2021) (NIPS'2021)
- [x] [PVT](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#pvt-iccv-2021) (ICCV'2021)
- [x] [PVTV2](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo_papers/backbones.html#pvtv2-cvmj-2022) (CVMJ'2022)

### 模型需求

我们将跟进学界的最新进展，并支持更多算法和框架。如果您对 MMPose 有任何功能需求，请随时在 [MMPose Roadmap](https://github.com/open-mmlab/mmpose/issues/2258) 中留言。

## 参与贡献

我们非常欢迎用户对于 MMPose 做出的任何贡献，可以参考 [贡献指南](https://mmpose.readthedocs.io/zh_CN/latest/contribution_guide.html) 文件了解更多细节。

## 致谢

## 参与贡献

## 致谢

MMPose 是一款由不同学校和公司共同贡献的开源项目。我们感谢所有为项目提供算法复现和新功能支持的贡献者，以及提供宝贵反馈的用户。
我们希望该工具箱和基准测试可以为社区提供灵活的代码工具，供用户复现现有算法并开发自己的新模型，从而不断为开源社区提供贡献。

## 引用

如果您觉得 MMPose 对您的研究有所帮助，请考虑引用它：

```bibtex
}
```

## 许可证

该项目采用 [Apache 2.0 license](LICENSE) 开源协议。

## OpenMMLab的其他项目

扫描下方的二维码可关注 OpenMMLab 团队的 [知乎官方账号](https://www.zhihu.com/people/openmmlab)，联络 OpenMMLab [官方微信小助手](https://user-images.githubusercontent.com/25839884/205872898-e2e6009d-c6bb-4d27-8d07-117e697a3da8.jpg)或加入 OpenMMLab 团队的 [官方交流 QQ 群](https://jq.qq.com/?_wv=1027&k=K0QI8ByU)

<img src="https://user-images.githubusercontent.com/25839884/205870927-39f4946d-8751-4219-a4c0-740117558fd7.jpg" height="400"><img src="https://user-images.githubusercontent.com/25839884/205872898-e2e6009d-c6bb-4d27-8d07-117e697a3da8.jpg" height="400"><img src="https://user-images.githubusercontent.com/25839884/203904835-62392033-02d4-4c73-a68c-c9e4c1e2b07f.jpg" height="400">
</div>

# TSN

[Temporal segment networks: Towards good practices for deep action recognition](https://link.springer.com/chapter/10.1007/978-3-319-46484-8_2)

Deep convolutional networks have achieved great success for visual recognition in still images. However, for action recognition in videos, the advantage over traditional methods is not so evident. This paper aims to discover the principles to design effective ConvNet architectures for action recognition in videos and learn these models given limited training samples. Our first contribution is temporal segment network (TSN), a novel framework for video-based action recognition. which is based on the idea of long-range temporal structure modeling. It combines a sparse temporal sampling strategy and video-level supervision to enable efficient and effective learning using the whole action video. The other contribution is our study on a series of good practices in learning ConvNets on video data with the help of temporal segment network. Our approach obtains the state-the-of-art performance on the datasets of HMDB51 ( 69.4%) and UCF101 (94.2%). We also visualize the learned ConvNet models, which qualitatively demonstrates the effectiveness of temporal segment network and the proposed good practices.

<img src="https://user-images.githubusercontent.com/34324155/143019237-8823045b-dfa3-45cc-a992-ee83ab9d8459.png" width="800"/>
</div>

| frame sampling strategy | scheduler | resolution | gpus | backbone  | pretrain | top1 acc | top5 acc |  testing protocol  | FLOPs  | params |              config              |                           ckpt |                           log |
| :---------------------: | :-------: | :--------: | :--: | :-------: | :------: | :------: | :------: | :----------------: | :----: | :----: | :------------------------------: | -----------------------------: | ----------------------------: |

|          1x1x3          | MultiStep |  224x224   |  8   | ResNet50  | ImageNet |  72.83   |  90.65   | 25 clips x 10 crop | 102.7G | 24.33M | [config](/configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb_20220906-cd10898e.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.log) |

|          1x1x5          | MultiStep |  224x224   |  8   | ResNet50  | ImageNet |  73.80   |  91.21   | 25 clips x 10 crop | 102.7G | 24.33M | [config](/configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x5-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x5-100e_kinetics400-rgb/tsn_imagenet-pretrained-r50_8xb32-1x1x5-100e_kinetics400-rgb_20220906-65d68713.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x5-100e_kinetics400-rgb/tsn_imagenet-pretrained-r50_8xb32-1x1x5-100e_kinetics400-rgb.log) |

|          1x1x8          | MultiStep |  224x224   |  8   | ResNet50  | ImageNet |  74.12   |  91.34   | 25 clips x 10 crop | 102.7G | 24.33M | [config](/configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb/tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb_20220906-2692d16c.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb/tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb.log) |

|       dense-1x1x5       | MultiStep |  224x224   |  8   | ResNet50  | ImageNet |  71.37   |  89.67   | 25 clips x 10 crop | 102.7G | 24.33M | [config](/configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-dense-1x1x5-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-dense-1x1x5-100e_kinetics400-rgb/tsn_imagenet-pretrained-r50_8xb32-dense-1x1x5-100e_kinetics400-rgb_20220906-dcbc6e01.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-dense-1x1x5-100e_kinetics400-rgb/tsn_imagenet-pretrained-r50_8xb32-dense-1x1x5-100e_kinetics400-rgb.log) |

|          1x1x8          | MultiStep |  224x224   |  8   | ResNet101 | ImageNet |  75.89   |  92.07   | 25 clips x 10 crop | 195.8G | 43.32M | [config](/configs/recognition/tsn/tsn_imagenet-pretrained-r101_8xb32-1x1x8-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-r101_8xb32-1x1x8-100e_kinetics400-rgb/tsn_imagenet-pretrained-r101_8xb32-1x1x8-100e_kinetics400-rgb_20220906-23cff032.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-r101_8xb32-1x1x8-100e_kinetics400-rgb/tsn_imagenet-pretrained-r101_8xb32-1x1x8-100e_kinetics400-rgb.log) |

### Something-Something V2

| frame sampling strategy | scheduler | resolution | gpus | backbone | pretrain | top1 acc | top5 acc |  testing protocol  | FLOPs  | params |              config              |                           ckpt |                            log |

| :---------------------: | :-------: | :--------: | :--: | :------: | :------: | :------: | :------: | :----------------: | :----: | :----: | :------------------------------: | -----------------------------: | -----------------------------: |
|          1x1x8          | MultiStep |  224x224   |  8   | ResNet50 | ImageNet |  35.51   |  67.09   | 25 clips x 10 crop | 102.7G | 24.33M | [config](/configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x8-50e_sthv2-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x8-50e_sthv2-rgb/tsn_imagenet-pretrained-r50_8xb32-1x1x8-50e_sthv2-rgb_20230313-06ad7d03.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x8-50e_sthv2-rgb/tsn_imagenet-pretrained-r50_8xb32-1x1x8-50e_sthv2-rgb.log) |

|         1x1x16          | MultiStep |  224x224   |  8   | ResNet50 | ImageNet |  36.91   |  68.77   | 25 clips x 10 crop | 102.7G | 24.33M | [config](/configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x16-50e_sthv2-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x16-50e_sthv2-rgb/tsn_imagenet-pretrained-r50_8xb32-1x1x16-50e_sthv2-rgb_20230221-85bcc1c3.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x16-50e_sthv2-rgb/tsn_imagenet-pretrained-r50_8xb32-1x1x16-50e_sthv2-rgb.log) |

### Using backbones from 3rd-party in TSN

It's possible and convenient to use a 3rd-party backbone for TSN under the framework of MMAction2, here we provide some examples for:

- [x] Backbones from [MMClassification](https://github.com/open-mmlab/mmclassification/)
- [x] Backbones from [TorchVision](https://github.com/pytorch/vision/)

- [x] Backbones from [TIMM (pytorch-image-models)](https://github.com/rwightman/pytorch-image-models)

| frame sampling strategy | scheduler | resolution | gpus |     backbone     | pretrain | top1 acc | top5 acc |  testing protocol  | FLOPs  | params |            config             |                         ckpt |                         log |
| :---------------------: | :-------: | :--------: | :--: | :--------------: | :------: | :------: | :------: | :----------------: | :----: | :----: | :---------------------------: | ---------------------------: | --------------------------: |

|          1x1x3          | MultiStep |  224x224   |  8   |    ResNext101    | ImageNet |  72.95   |  90.36   | 25 clips x 10 crop | 200.3G | 42.95M | [config](/configs/recognition/tsn/custom_backbones/tsn_imagenet-pretrained-rn101-32x4d_8xb32-1x1x3-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-rn101-32x4d_8xb32-1x1x3-100e_kinetics400-rgb/tsn_imagenet-pretrained-rn101-32x4d_8xb32-1x1x3-100e_kinetics400-rgb_20221209-de2d5615.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-rn101-32x4d_8xb32-1x1x3-100e_kinetics400-rgb/tsn_imagenet-pretrained-rn101-32x4d_8xb32-1x1x3-100e_kinetics400-rgb.log) |

|          1x1x3          | MultiStep |  224x224   |  8   |   DenseNet161    | ImageNet |  72.07   |  90.15   | 25 clips x 10 crop | 194.6G | 27.36M | [config](/configs/recognition/tsn/custom_backbones/tsn_imagenet-pretrained-dense161_8xb32-1x1x3-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-dense161_8xb32-1x1x3-100e_kinetics400-rgb/tsn_imagenet-pretrained-dense161_8xb32-1x1x3-100e_kinetics400-rgb_20220906-5f4c0daf.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-dense161_8xb32-1x1x3-100e_kinetics400-rgb/tsn_imagenet-pretrained-dense161_8xb32-1x1x3-100e_kinetics400-rgb.log) |

|          1x1x3          | MultiStep |  224x224   |  8   | Swin Transformer | ImageNet |  77.03   |  92.61   | 25 clips x 10 crop | 386.7G | 87.15M | [config](/configs/recognition/tsn/custom_backbones/tsn_imagenet-pretrained-swin-transformer_8xb32-1x1x3-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-swin-transformer_8xb32-1x1x3-100e_kinetics400-rgb/tsn_imagenet-pretrained-swin-transformer_8xb32-1x1x3-100e_kinetics400-rgb_20220906-65ed814e.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-swin-transformer_8xb32-1x1x3-100e_kinetics400-rgb/tsn_imagenet-pretrained-swin-transformer_8xb32-1x1x3-100e_kinetics400-rgb.log) |

|          1x1x8          | MultiStep |  224x224   |  8   | Swin Transformer | ImageNet |  79.22   |  94.20   | 25 clips x 10 crop | 386.7G | 87.15M | [config](/configs/recognition/tsn/custom_backbones/tsn_imagenet-pretrained-swin-transformer_32xb8-1x1x8-50e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-swin-transformer_32xb8-1x1x8-50e_kinetics400-rgb/tsn_imagenet-pretrained-swin-transformer_32xb8-1x1x8-50e_kinetics400-rgb_20230530-428f0064.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-swin-transformer_32xb8-1x1x8-50e_kinetics400-rgb/tsn_imagenet-pretrained-swin-transformer_32xb8-1x1x8-50e_kinetics400-rgb.log) |

1. Note that some backbones in TIMM are not supported due to multiple reasons. Please refer to [PR #880](https://github.com/open-mmlab/mmaction2/pull/880) for details.

For more details on data preparation, you can refer to

- [Kinetics](/tools/data/kinetics/README.md)
- [Something-something V2](/tools/data/sthv2/README.md)

## Train

```shell
```

Example: train TSN model on Kinetics-400 dataset in a deterministic option.

```shell

```shell
```

```shell
python tools/train.py configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py  \
```

## Test

```shell
```

Example: test TSN model on Kinetics-400 dataset and dump the result to a pkl file.

```shell
python tools/test.py configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py \
```

```BibTeX

```

```BibTeX
@inproceedings{wang2016temporal,
  title={Temporal segment networks: Towards good practices for deep action recognition},
  author={Wang, Limin and Xiong, Yuanjun and Wang, Zhe and Qiao, Yu and Lin, Dahua and Tang, Xiaoou and Van Gool, Luc},
  pages={20--36},
}
```

# 训练技巧

MMSegmentation 支持如下训练技巧：

## 主干网络和解码头组件使用不同的学习率 (Learning Rate, LR)

在语义分割里，一些方法会让解码头组件的学习率大于主干网络的学习率，这样可以获得更好的表现或更快的收敛。

在 MMSegmentation 里面，您也可以在配置文件里添加如下行来让解码头组件的学习率是主干组件的10倍。

```python
optim_wrapper=dict(
    paramwise_cfg = dict(
            'head': dict(lr_mult=10.)}))
```

通过这种修改，任何被分组到 `'head'` 的参数的学习率都将乘以10。您也可以参照 [MMEngine 文档](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/optim_wrapper.html#id6)  获取更详细的信息。

## 在线难样本挖掘 (Online Hard Example Mining, OHEM)

MMSegmentation 中实现了像素采样器，训练时可以对特定像素进行采样，例如 OHEM(Online Hard Example Mining)，可以解决样本不平衡问题，
如下例子是使用 PSPNet 训练并采用 OHEM 策略的配置：

```python
_base_ = './pspnet_r50-d8_512x1024_40k_cityscapes.py'
model=dict(
        sampler=dict(type='OHEMPixelSampler', thresh=0.7, min_kept=100000)) )
```

通过这种方式，只有置信分数在0.7以下的像素值点会被拿来训练。在训练时我们至少要保留100000个像素值点。如果 `thresh` 并未被指定，前 `min_kept`
个损失的像素值点才会被选择。

## 类别平衡损失 (Class Balanced Loss)

对于不平衡类别分布的数据集，您也许可以改变每个类别的损失权重。这里以 cityscapes 数据集为例：

```

```python
            type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0,
            # DeepLab 对 cityscapes 使用这种权重
            class_weight=[0.8373, 0.9180, 0.8660, 1.0345, 1.0166, 0.9969, 0.9754,
                        1.0489, 0.8786, 1.0023, 0.9539, 0.9843, 1.1116, 0.9037,
                        1.0865, 1.0955, 1.0865, 1.1529, 1.0507])))
```

`class_weight` 将被作为 `weight` 参数，传递给 `CrossEntropyLoss`。详细信息请参照 [PyTorch 文档](https://pytorch.org/docs/stable/nn.html?highlight=crossentropy#torch.nn.CrossEntropyLoss) 。

## 同时使用多种损失函数 (Multiple Losses)

对于训练时损失函数的计算，我们目前支持多个损失函数同时使用。 以 `unet` 使用 `DRIVE` 数据集训练为例，
使用 `CrossEntropyLoss` 和 `DiceLoss` 的 `1:3` 的加权和作为损失函数。配置文件写为:

```python

```python
_base_ = './fcn_unet_s5-d16_64x64_40k_drive.py'
    decode_head=dict(loss_decode=[
        dict(type='CrossEntropyLoss', loss_name='loss_ce', loss_weight=1.0),
        dict(type='DiceLoss', loss_name='loss_dice', loss_weight=3.0)
    ]),
    auxiliary_head=dict(loss_decode=[
    ]),
)
```

通过这种方式，确定训练过程中损失函数的权重 `loss_weight` 和在训练日志里的名字 `loss_name`。

注意： `loss_name` 的名字必须带有 `loss_` 前缀，这样它才能被包括在计算图里。

# 教程5：模型部署

# MMSegmentation 模型部署

- [教程5：模型部署](#教程5模型部署)
- [MMSegmentation 模型部署](#mmsegmentation-模型部署)
    - [安装 mmseg](#安装-mmseg)
    - [安装 mmdeploy](#安装-mmdeploy)
  - [模型规范](#模型规范)
    - [后端模型推理](#后端模型推理)
    - [SDK 模型推理](#sdk-模型推理)
  - [模型支持列表](#模型支持列表)
  - [注意事项](#注意事项)

[MMSegmentation](https://github.com/open-mmlab/mmsegmentation/tree/main) 又称`mmseg`，是一个基于 PyTorch 的开源对象分割工具箱。它是 [OpenMMLab](https://openmmlab.com/) 项目的一部分。

## 安装

### 安装 mmseg

请参考[官网安装指南](https://mmsegmentation.readthedocs.io/en/latest/get_started.html)。

### 安装 mmdeploy

mmdeploy 有以下几种安装方式:

**方式一：** 安装预编译包

请参考[安装概述](https://mmdeploy.readthedocs.io/zh_CN/latest/get_started.html#mmdeploy)

**方式二：** 一键式脚本安装

如果部署平台是 **Ubuntu 18.04 及以上版本**， 请参考[脚本安装说明](../01-how-to-build/build_from_script.md)，完成安装过程。
比如，以下命令可以安装 mmdeploy 以及配套的推理引擎——`ONNX Runtime`.

```shell

```shell
```

**说明**:

- 把 `$(pwd)/build/lib` 添加到 `PYTHONPATH`，目的是为了加载 mmdeploy SDK python 包 `mmdeploy_runtime`，在章节 [SDK模型推理](#sdk模型推理)中讲述其用法。
- 在[使用 ONNX Runtime推理后端模型](#后端模型推理)时，需要加载自定义算子库，需要把 ONNX Runtime 库的路径加入环境变量 `LD_LIBRARY_PATH`中。
  **方式三：** 源码安装

在方式一、二都满足不了的情况下，请参考[源码安装说明](../01-how-to-build/build_from_source.md) 安装 mmdeploy 以及所需推理引擎。

## 模型转换

你可以使用 [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/tree/main/tools/deploy.py) 把 mmseg 模型一键式转换为推理后端模型。

## 模型转换

该工具的详细使用说明请参考[这里](https://github.com/open-mmlab/mmdeploy/tree/main/docs/en/02-how-to-run/convert_model.md#usage).

以下，我们将演示如何把 `unet` 转换为 onnx 模型。

```shell

# download unet model from mmseg model zoo
mim download mmsegmentation --config unet-s5-d16_fcn_4xb4-160k_cityscapes-512x1024 --dest .

# convert mmseg model to onnxruntime model with dynamic shape
    configs/mmseg/segmentation_onnxruntime_dynamic.py \
    unet-s5-d16_fcn_4xb4-160k_cityscapes-512x1024.py \
    fcn_unet_s5-d16_4x4_512x1024_160k_cityscapes_20211210_145204-6860854e.pth \
    demo/resources/cityscapes.png \
    --work-dir mmdeploy_models/mmseg/ort \
    --show \
```

转换的关键之一是使用正确的配置文件。项目中已内置了各后端部署[配置文件](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmseg)。
文件的命名模式是：

```

--device cpu \
    --show \
```

文件的命名模式是：

```
segmentation_{backend}-{precision}_{static | dynamic}_{shape}.py
```

其中：

- **{backend}:** 推理后端名称。比如，onnxruntime、tensorrt、pplnn、ncnn、openvino、coreml 等等
- **{precision}:** 推理精度。比如，fp16、int8。不填表示 fp32
- **{static | dynamic}:** 动态、静态 shape
- **{shape}:** 模型输入的 shape 或者 shape 范围

在上例中，你也可以把 `unet` 转为其他后端模型。比如使用`segmentation_tensorrt-fp16_dynamic-512x1024-2048x2048.py`，把模型转为 tensorrt-fp16 模型。

```{tip}
当转 tensorrt 模型时, --device 需要被设置为 "cuda"
```

## 模型规范

在使用转换后的模型进行推理之前，有必要了解转换结果的结构。 它存放在 `--work-dir` 指定的路路径下。

上例中的`mmdeploy_models/mmseg/ort`，结构如下：

```
mmdeploy_models/mmseg/ort
```

重要的是：

- **end2end.onnx**: 推理引擎文件。可用 ONNX Runtime 推理
- \***.json**:  mmdeploy SDK 推理所需的 meta 信息

```
```

重要的是：

整个文件夹被定义为**mmdeploy SDK model**。换言之，**mmdeploy SDK model**既包括推理引擎，也包括推理 meta 信息。

## 模型推理

### 后端模型推理

以上述模型转换后的 `end2end.onnx` 为例，你可以使用如下代码进行推理：

```python

deploy_cfg = 'configs/mmseg/segmentation_onnxruntime_dynamic.py'
model_cfg = './unet-s5-d16_fcn_4xb4-160k_cityscapes-512x1024.py'
backend_model = ['./mmdeploy_models/mmseg/ort/end2end.onnx']
image = './demo/resources/cityscapes.png'

    output_file='./output_segmentation.png')
```

### SDK 模型推理

你也可以参考如下代码，对 SDK model 进行推理：

```python
from mmdeploy_runtime import Segmentor
import cv2

img = cv2.imread('./demo/resources/cityscapes.png')

# create a classifier
segmentor = Segmentor(model_path='./mmdeploy_models/mmseg/ort', device_name='cpu', device_id=0)
seg = segmentor(img)

# visualize inference result
## random a palette with size 256x3

palette = np.random.randint(0, 256, size=(256, 3))
color_seg = np.zeros((seg.shape[0], seg.shape[1], 3), dtype=np.uint8)
for label, color in enumerate(palette):
    color_seg[seg == label, :] = color
# convert to BGR
color_seg = color_seg[..., ::-1]
img = img * 0.5 + color_seg * 0.5
img = img.astype(np.uint8)
cv2.imwrite('output_segmentation.png', img)
```

除了python API，mmdeploy SDK 还提供了诸如 C、C++、C#、Java等多语言接口。
你可以参考[样例](https://github.com/open-mmlab/mmdeploy/tree/main/demo)学习其他语言接口的使用方法。

## 模型支持列表

| Model                                                                                                     | TorchScript | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVino |
| :-------------------------------------------------------------------------------------------------------- | :---------: | :---------: | :------: | :--: | :---: | :------: |

| [FCN](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/fcn)                                 |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |
| [PSPNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/pspnet)[\*](#static_shape)        |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |
| [DeepLabV3](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/deeplabv3)                     |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |
| [DeepLabV3+](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/deeplabv3plus)                |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |

| [Fast-SCNN](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/fastscnn)[\*](#static_shape)   |      Y      |      Y      |    Y     |  N   |   Y   |    Y     |
| [UNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/unet)                               |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |
| [ANN](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/ann)[\*](#static_shape)              |      Y      |      Y      |    Y     |  N   |   N   |    N     |
| [APCNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/apcnet)                           |      Y      |      Y      |    Y     |  Y   |   N   |    N     |

| [BiSeNetV1](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/bisenetv1)                     |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |
| [BiSeNetV2](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/bisenetv2)                     |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |
| [CGNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/cgnet)                             |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |
| [DMNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/dmnet)                             |      ?      |      Y      |    N     |  N   |   N   |    N     |

| [DNLNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/dnlnet)                           |      ?      |      Y      |    Y     |  Y   |   N   |    Y     |
| [EMANet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/emanet)                           |      Y      |      Y      |    Y     |  N   |   N   |    Y     |
| [EncNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/encnet)                           |      Y      |      Y      |    Y     |  N   |   N   |    Y     |
| [ERFNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/erfnet)                           |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |

| [FastFCN](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/fastfcn)                         |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |
| [GCNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/gcnet)                             |      Y      |      Y      |    Y     |  N   |   N   |    N     |
| [ICNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/icnet)[\*](#static_shape)          |      Y      |      Y      |    Y     |  N   |   N   |    Y     |
| [ISANet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/isanet)[\*](#static_shape)        |      N      |      Y      |    Y     |  N   |   N   |    Y     |

| [NonLocal Net](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/nonlocal_net)               |      ?      |      Y      |    Y     |  Y   |   N   |    Y     |
| [OCRNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/ocrnet)                           |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |
| [PointRend](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/point_rend)[\*](#static_shape) |      Y      |      Y      |    Y     |  N   |   N   |    N     |
| [Semantic FPN](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/sem_fpn)                    |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |

| [STDC](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/stdc)                               |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |
| [UPerNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/upernet)[\*](#static_shape)      |      N      |      Y      |    Y     |  N   |   N   |    N     |
| [DANet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/danet)                             |      ?      |      Y      |    Y     |  N   |   N   |    Y     |
| [Segmenter](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/segmenter)[\*](#static_shape)  |      N      |      Y      |    Y     |  Y   |   N   |    Y     |

| [SegFormer](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/segformer)[\*](#static_shape)  |      ?      |      Y      |    Y     |  N   |   N   |    Y     |
| [SETR](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/setr)                               |      ?      |      Y      |    N     |  N   |   N   |    Y     |
| [CCNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/ccnet)                             |      ?      |      N      |    N     |  N   |   N   |    N     |
| [PSANet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/psanet)                           |      ?      |      N      |    N     |  N   |   N   |    N     |

| [DPT](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/dpt)                                 |      ?      |      N      |    N     |  N   |   N   |    N     |

## 注意事项

- 所有 mmseg 模型仅支持 "whole" 推理模式。

- <i id=“static_shape”>PSPNet，Fast-SCNN</i> 仅支持静态输入，因为多数推理框架的 [nn.AdaptiveAvgPool2d](https://github.com/open-mmlab/mmsegmentation/blob/0c87f7a0c9099844eff8e90fa3db5b0d0ca02fee/mmseg/models/decode_heads/psp_head.py#L38) 不支持动态输入。

- 对于仅支持静态形状的模型，应使用静态形状的部署配置文件，例如 `configs/mmseg/segmentation_tensorrt_static-1024x2048.py`

- 对于喜欢部署模型生成概率特征图的用户，将 `codebase_config = dict(with_argmax=False)` 放在部署配置中就足够了。

# Tutorial 5: Using metrics in MMagic

MMagic supports **17 metrics** to assess the quality of models.

Please refer to [Train and Test in MMagic](../user_guides/train_test.md) for usages.

Here, we will specify the details of different metrics one by one.

The structure of this guide are as follows:

- [Tutorial 5: Using metrics in MMagic](#tutorial-5-using-metrics-in-mmagic)
  - [MAE](#mae)
  - [MSE](#mse)
  - [PSNR](#psnr)
  - [SNR](#snr)
  - [SSIM](#ssim)
  - [NIQE](#niqe)
  - [SAD](#sad)
  - [MattingMSE](#mattingmse)
  - [GradientError](#gradienterror)
  - [ConnectivityError](#connectivityerror)
  - [FID and TransFID](#fid-and-transfid)
  - [IS and TransIS](#is-and-transis)
  - [Precision and Recall](#precision-and-recall)
  - [PPL](#ppl)
  - [SWD](#swd)
  - [MS-SSIM](#ms-ssim)
  - [Equivarience](#equivarience)

## MAE

MAE is Mean Absolute Error metric for image.
To evaluate with MAE, please add the following configuration in the config file:

```python

- [Equivarience](#equivarience)

## MAE

```python
    dict(type='MAE'),
]
```

## MSE

MSE is Mean Squared Error metric for image.
To evaluate with MSE, please add the following configuration in the config file:

```python
    dict(type='MSE'),
]
```

## PSNR

PSNR is Peak Signal-to-Noise Ratio. Our implement refers to https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio.
To evaluate with PSNR, please add the following configuration in the config file:

```python
    dict(type='PSNR'),
]
```

## SNR

SNR is Signal-to-Noise Ratio. Our implementation refers to https://en.wikipedia.org/wiki/Signal-to-noise_ratio.
To evaluate with SNR, please add the following configuration in the config file:

```python
    dict(type='SNR'),
]
```

## SSIM

```python
]
```

## SSIM

SSIM is structural similarity for image, proposed in [Image quality assessment: from error visibility to structural similarity](https://live.ece.utexas.edu/publications/2004/zwang_ssim_ieeeip2004.pdf). The results of our implementation are the same as that of the official released MATLAB code in https://ece.uwaterloo.ca/~z70wang/research/ssim/.
To evaluate with SSIM, please add the following configuration in the config file:

```python
    dict(type='SSIM'),
]
```

## NIQE

NIQE is Natural Image Quality Evaluator metric, proposed in [Making a "Completely Blind" Image Quality Analyzer](http://www.live.ece.utexas.edu/publications/2013/mittal2013.pdf). Our implementation could produce almost the same results as the official MATLAB codes: http://live.ece.utexas.edu/research/quality/niqe_release.zip.

To evaluate with NIQE, please add the following configuration in the config file:

```python
    dict(type='NIQE'),
]
```

## SAD

SAD is Sum of Absolute Differences metric for image matting. This metric compute per-pixel absolute difference and sum across all pixels.
To evaluate with SAD, please add the following configuration in the config file:

```python
    dict(type='SAD'),
]
```

## MattingMSE

MattingMSE is Mean Squared Error metric for image matting.
To evaluate with MattingMSE, please add the following configuration in the config file:

```python
    dict(type='MattingMSE'),
]
```

## GradientError

GradientError is Gradient error for evaluating alpha matte prediction.
To evaluate with GradientError, please add the following configuration in the config file:

```python
    dict(type='GradientError'),
]
```

## ConnectivityError

```python
]
```

ConnectivityError is Connectivity error for evaluating alpha matte prediction.
To evaluate with ConnectivityError, please add the following configuration in the config file:

```python
    dict(type='ConnectivityError'),
]
```

## FID and TransFID

Fréchet Inception Distance is a measure of similarity between two datasets of images. It was shown to correlate well with the human judgment of visual quality and is most often used to evaluate the quality of samples of Generative Adversarial Networks. FID is calculated by computing the Fréchet distance between two Gaussians fitted to feature representations of the Inception network.

In `MMagic`, we provide two versions for FID calculation. One is the commonly used PyTorch version and the other one is used in StyleGAN paper. Meanwhile, we have compared the difference between these two implementations in the StyleGAN2-FFHQ1024 model (the details can be found [here](https://github.com/open-mmlab/mmagic/blob/main/configs/styleganv2/README.md)). Fortunately, there is a marginal difference in the final results. Thus, we recommend users adopt the more convenient PyTorch version.

**About PyTorch version and Tero's version:** The commonly used PyTorch version adopts the modified InceptionV3 network to extract features for real and fake images. However, Tero's FID requires a [script module](https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/pretrained/metrics/inception-2015-12-05.pt) for Tensorflow InceptionV3. Note that applying this script module needs `PyTorch >= 1.6.0`.

**About extracting real inception data:** For the users' convenience, the real features will be automatically extracted at test time and saved locally, and the stored features will be automatically read at the next test. Specifically, we will calculate a hash value based on the parameters used to calculate the real features, and use the hash value to mark the feature file, and when testing, if the `inception_pkl` is not set, we will look for the feature in `MMAGIC_CACHE_DIR` (~/.cache/openmmlab/mmagic/). If cached inception pkl is not found, then extracting will be performed.

To use the FID metric, you should add the metric in a config file like this:

```python
metrics = [
    dict(
        type='FrechetInceptionDistance',
        prefix='FID-Full-50k',
        fake_nums=50000,
        inception_style='StyleGAN',
        sample_model='ema')
]
```

```python
metrics = [
    dict(
]
```

If you work on an new machine, then you can copy the `pkl` files in `MMAGIC_CACHE_DIR` and copy them to new machine and set `inception_pkl` field.

```python
metrics = [
    dict(
        inception_pkl=
        'work_dirs/inception_pkl/inception_state-capture_mean_cov-full-33ad4546f8c9152e4b3bdb1b0c08dbaf.pkl',  # copied from old machine
]
```

`TransFID` has same usage as `FID`, but it's designed for translation models like `Pix2Pix` and `CycleGAN`, which is adapted for our evaluator. You can refer
to [evaluation](../user_guides/train_test.md) for details.

## IS and TransIS

Inception score is an objective metric for evaluating the quality of generated images, proposed in [Improved Techniques for Training GANs](https://arxiv.org/pdf/1606.03498.pdf). It uses an InceptionV3 model to predict the class of the generated images, and suppose that 1) If an image is of high quality, it will be categorized into a specific class. 2) If images are of high diversity, the range of images' classes will be wide. So the KL-divergence of the conditional probability and marginal probability can indicate the quality and diversity of generated images. You can see the complete implementation in `metrics.py`, which refers to https://github.com/sbarratt/inception-score-pytorch/blob/master/inception_score.py.
If you want to evaluate models with `IS` metrics, you can add the `metrics` into your config file like this:

```python
# at the end of the configs/biggan/biggan_2xb25-500kiters_cifar10-32x32.py

```python
metrics = [
    xxx,
    dict(
        type='IS',
        prefix='IS-50k',
]
```

To be noted that, the selection of Inception V3 and image resize method can significantly influence the final IS score. Therefore, we strongly recommend users may download the [Tero's script model of Inception V3](https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/pretrained/metrics/inception-2015-12-05.pt) (load this script model need torch >= 1.6) and use `Bicubic` interpolation with `Pillow` backend.

Corresponding to config, you can set `resize_method` and `use_pillow_resize` for image resizing. You can also set `inception_style` as `StyleGAN` for recommended tero's inception model, or `PyTorch` for torchvision's implementation. For environment without internet, you can download the inception's weights, and set `inception_path` to your inception model.

We also perform a survey on the influence of data loading pipeline and the version of pretrained Inception V3 on the IS result. All IS are evaluated on the same group of images which are randomly selected from the ImageNet dataset.

<details> <summary> Show the Comparison Results </summary>

|                            Code Base                            | Inception V3 Version | Data Loader Backend | Resize Interpolation Method |          IS           |
| :-------------------------------------------------------------: | :------------------: | :-----------------: | :-------------------------: | :-------------------: |

|   [OpenAI (baseline)](https://github.com/openai/improved-gan)   |      Tensorflow      |       Pillow        |       Pillow Bicubic        | **312.255 +/- 4.970** |
| [StyleGAN-Ada](https://github.com/NVlabs/stylegan2-ada-pytorch) | Tero's Script Model  |       Pillow        |       Pillow Bicubic        |   311.895 +/ 4.844    |
|                          mmagic (Ours)                          |  Pytorch Pretrained  |         cv2         |        cv2 Bilinear         |   322.932 +/- 2.317   |
|                          mmagic (Ours)                          |  Pytorch Pretrained  |         cv2         |         cv2 Bicubic         |   324.604 +/- 5.157   |

|                          mmagic (Ours)                          |  Pytorch Pretrained  |         cv2         |       Pillow Bicubic        |   318.161 +/- 5.330   |
|                          mmagic (Ours)                          |  Pytorch Pretrained  |       Pillow        |       Pillow Bilinear       |   313.126 +/- 5.449   |
|                          mmagic (Ours)                          |  Pytorch Pretrained  |       Pillow        |        cv2 Bilinear         |    318.021+/-3.864    |
|                          mmagic (Ours)                          |  Pytorch Pretrained  |       Pillow        |       Pillow Bicubic        |   317.997 +/- 5.350   |

|                          mmagic (Ours)                          | Tero's Script Model  |         cv2         |        cv2 Bilinear         |   318.879 +/- 2.433   |
|                          mmagic (Ours)                          | Tero's Script Model  |         cv2         |         cv2 Bicubic         |   316.125 +/- 5.718   |
|                          mmagic (Ours)                          | Tero's Script Model  |         cv2         |       Pillow Bicubic        | **312.045 +/- 5.440** |
|                          mmagic (Ours)                          | Tero's Script Model  |       Pillow        |       Pillow Bilinear       |   308.645 +/- 5.374   |

|                          mmagic (Ours)                          | Tero's Script Model  |       Pillow        |       Pillow Bicubic        |   311.733 +/- 5.375   |

`TransIS` has same usage as `IS`, but it's designed for translation models like `Pix2Pix` and `CycleGAN`, which is adapted for our evaluator. You can refer

## Precision and Recall

Our `Precision and Recall` implementation follows the version used in StyleGAN2. In this metric, a VGG network will be adopted to extract the features for images. Unfortunately, we have not found a PyTorch VGG implementation leading to similar results with Tero's version used in StyleGAN2. (About the differences, please see this [file](https://github.com/open-mmlab/mmagic/blob/main/configs/styleganv2/README.md).) Thus, in our implementation, we adopt [Teor's VGG](https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/pretrained/metrics/vgg16.pt) network by default. Importantly, applying this script module needs `PyTorch >= 1.6.0`. If with a lower PyTorch version, we will use the PyTorch official VGG network for feature extraction.

To evaluate with `P&R`, please add the following configuration in the config file:

```python
metrics = [

```python
metrics = [
    dict(type='PrecisionAndRecall', fake_nums=50000, prefix='PR-50K')
]
```

## PPL

Perceptual path length measures the difference between consecutive images (their VGG16 embeddings) when interpolating between two random inputs. Drastic changes mean that multiple features have changed together and that they might be entangled. Thus, a smaller PPL score appears to indicate higher overall image quality by experiments. \
As a basis for our metric, we use a perceptually-based pairwise image distance that is calculated as a weighted difference between two VGG16 embeddings, where the weights are fit so that the metric agrees with human perceptual similarity judgments.

If we subdivide a latent space interpolation path into linear segments, we can define the total perceptual length of this segmented path as the sum of perceptual differences over each segment, and a natural definition for the perceptual path length would be the limit of this sum under infinitely fine subdivision, but in practice we approximate it using a small subdivision `` $`\epsilon=10^{-4}`$ ``.
The average perceptual path length in latent `space` Z, over all possible endpoints, is therefore

`` $$`L_Z = E[\frac{1}{\epsilon^2}d(G(slerp(z_1,z_2;t))), G(slerp(z_1,z_2;t+\epsilon)))]`$$ ``

Computing the average perceptual path length in latent `space` W is carried out in a similar fashion:

Where `` $`z_1, z_2 \sim P(z)`$ ``, and `` $` t \sim U(0,1)`$ `` if we set `sampling` to full, `` $` t \in \{0,1\}`$ `` if we set `sampling` to end. `` $` G`$ `` is the generator(i.e. `` $` g \circ f`$ `` for style-based networks), and `` $` d(.,.)`$ `` evaluates the perceptual distance between the resulting images.We compute the expectation by taking 100,000 samples (set `num_images` to 50,000 in our code).

You can find the complete implementation in `metrics.py`, which refers to https://github.com/rosinality/stylegan2-pytorch/blob/master/ppl.py.
If you want to evaluate models with `PPL` metrics, you can add the `metrics` into your config file like this:

```python
# at the end of the configs/styleganv2/stylegan2_c2_ffhq_1024_b4x8.py
metrics = [
    xxx,

```python
metrics = [
    xxx,
    dict(type='PerceptualPathLength', fake_nums=50000, prefix='ppl-w')
]
```

## SWD

Sliced Wasserstein distance is a discrepancy measure for probability distributions, and smaller distance indicates generated images look like the real ones. We obtain the Laplacian pyramids of every image and extract patches from the Laplacian pyramids as descriptors, then SWD can be calculated by taking the sliced Wasserstein distance of the real and fake descriptors.
You can see the complete implementation in `metrics.py`, which refers to https://github.com/tkarras/progressive_growing_of_gans/blob/master/metrics/sliced_wasserstein.py.
If you want to evaluate models with `SWD` metrics, you can add the `metrics` into your config file like this:

```python
# at the end of the configs/dcgan/dcgan_1xb128-5epoches_lsun-bedroom-64x64.py
metrics = [
    dict(
        type='SWD',
        prefix='swd',

```python
metrics = [
    dict(
        fake_nums=16384,
        sample_model='orig',
        image_shape=(3, 64, 64))
]
```

## MS-SSIM

Multi-scale structural similarity is used to measure the similarity of two images. We use MS-SSIM here to measure the diversity of generated images, and a low MS-SSIM score indicates the high diversity of generated images. You can see the complete implementation in `metrics.py`, which refers to https://github.com/tkarras/progressive_growing_of_gans/blob/master/metrics/ms_ssim.py.
If you want to evaluate models with `MS-SSIM` metrics, you can add the `metrics` into your config file like this:

```python
metrics = [
    dict(
        type='MS_SSIM', prefix='ms-ssim', fake_nums=10000,
        sample_model='orig')
]
```

## Equivarience

metrics = [
    dict(
]
```

Equivarience of generative models refer to the exchangeability of model forward and geometric transformations. Currently this metric is only calculated for StyleGANv3,
you can see the complete implementation in `metrics.py`, which refers to https://github.com/NVlabs/stylegan3/blob/main/metrics/equivariance.py.
If you want to evaluate models with `Equivarience` metrics, you can add the `metrics` into your config file like this:

```python
# at the end of the configs/styleganv3/stylegan3-t_gamma2.0_8xb4-fp16-noaug_ffhq-256x256.py
metrics = [
    dict(
        type='Equivariance',
        sample_mode='ema',
        prefix='EQ',
        eq_cfg=dict(
            compute_eqt_int=True, compute_eqt_frac=True, compute_eqr=True))
]
```

# 准备 UCF101-24

## 简介

```BibTeX
@article{Soomro2012UCF101AD,
  title={UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild},
  author={K. Soomro and A. Zamir and M. Shah},
  year={2012},
  volume={abs/1212.0402}
}
```

用户可参考该数据集的 [官网](http://www.thumos.info/download.html)，以获取数据集相关的基本信息。
在数据集准备前，请确保命令行当前路径为 `$MMACTION2/tools/data/ucf101_24/`。

## 下载和解压

用户可以从 [这里](https://drive.google.com/drive/folders/1BvGywlAGrACEqRyfYbz3wzlVV3cDFkct) 下载 RGB 帧，光流和标注文件。
该数据由 [MOC](https://github.com/MCG-NJU/MOC-Detector/blob/master/readme/Dataset.md) 代码库提供，
参考自 [act-detector](https://github.com/vkalogeiton/caffe/tree/act-detector) 和 [corrected-UCF101-Annots](https://github.com/gurkirt/corrected-UCF101-Annots)。

**注意**：UCF101-24 的标注文件来自于 [这里](https://github.com/gurkirt/corrected-UCF101-Annots)，该标注文件相对于其他标注文件更加准确。

用户在下载 `UCF101_v2.tar.gz` 文件后，需将其放置在 `$MMACTION2/tools/data/ucf101_24/` 目录下，并使用以下指令进行解压：

```shell
tar -zxvf UCF101_v2.tar.gz
```

## 检查文件夹结构

```shell
```

## 检查文件夹结构

经过解压后，用户将得到 `rgb-images` 文件夹，`brox-images` 文件夹和 `UCF101v2-GT.pkl` 文件。

在整个 MMAction2 文件夹下，UCF101_24 的文件结构如下：

```
mmaction2
├── tools
├── data
│   ├── ucf101_24
│   |   ├── brox-images
│   |   |   ├── Basketball
│   |   |   |   ├── v_Basketball_g01_c01
│   |   |   |   |   ├── 00001.jpg
│   |   |   |   |   ├── 00002.jpg
│   |   |   |   |   ├── ...
│   |   |   |   |   ├── 00140.jpg
│   |   |   |   |   ├── 00141.jpg
│   |   |   ├── ...
│   |   |   ├── WalkingWithDog
│   |   |   |   ├── v_WalkingWithDog_g01_c01
│   |   |   |   ├── v_WalkingWithDog_g25_c04
│   |   ├── rgb-images

│   |   |   ├── ...
│   |   ├── UCF101v2-GT.pkl

```

**注意**：`UCF101v2-GT.pkl` 作为一个缓存文件，它包含 6 个项目：

1. `labels` (list)：24 个行为类别名称组成的列表
2. `gttubes` (dict)：每个视频对应的基准 tubes 组成的字典
   **gttube** 是由标签索引和 tube 列表组成的字典
   **tube** 是一个 `nframes` 行和 5 列的 numpy array，每一列的形式如 `<frame index> <x1> <y1> <x2> <y2>`
3. `nframes` (dict)：用以表示每个视频对应的帧数，如 `'HorseRiding/v_HorseRiding_g05_c02': 151`
4. `train_videos` (list)：包含 `nsplits=1` 的元素，每一项都包含了训练视频的列表
5. `test_videos` (list)：包含 `nsplits=1` 的元素，每一项都包含了测试视频的列表
6. `resolution` (dict)：每个视频对应的分辨率（形如 (h,w)），如 `'FloorGymnastics/v_FloorGymnastics_g09_c03': (240, 320)`

# FastFCN

> [FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation](https://arxiv.org/abs/1903.11816)

<a href="https://github.com/wuhuikai/FastFCN">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.18.0/mmseg/models/necks/jpu.py#L12">Code Snippet</a>

Modern approaches for semantic segmentation usually employ dilated convolutions in the backbone to extract high-resolution feature maps, which brings heavy computation complexity and memory footprint. To replace the time and memory consuming dilated convolutions, we propose a novel joint upsampling module named Joint Pyramid Upsampling (JPU) by formulating the task of extracting high-resolution feature maps into a joint upsampling problem. With the proposed JPU, our method reduces the computation complexity by more than three times without performance loss. Experiments show that JPU is superior to other upsampling modules, which can be plugged into many existing approaches to reduce computation complexity and improve performance. By replacing dilated convolutions with the proposed JPU module, our method achieves the state-of-the-art performance in Pascal Context dataset (mIoU of 53.13%) and ADE20K dataset (final score of 0.5584) while running 3 times faster.

<img src="https://user-images.githubusercontent.com/24582831/142901374-6e0252ab-6e0f-4acd-86ad-1e9f49be3185.png" width="70%"/>
</div>

| Method              | Backbone       | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                    | download                                                                                                                                                                                                                                                                                                                                                                                                           |

| FastFCN + DeepLabV3 | R-50-D32       | 512x1024  |   80000 | 5.67     | 2.64           | V100   | 79.12 | 80.58         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fastfcn/fastfcn_r50-d32_jpu_aspp_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_aspp_512x1024_80k_cityscapes/fastfcn_r50-d32_jpu_aspp_512x1024_80k_cityscapes_20210928_053722-5d1a2648.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_aspp_512x1024_80k_cityscapes/fastfcn_r50-d32_jpu_aspp_512x1024_80k_cityscapes_20210928_053722.log.json)                 |

| FastFCN + DeepLabV3 | R-50-D32 (4x4) | 512x1024  |   80000 | 9.79     | -              | V100   | 79.52 | 80.91         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fastfcn/fastfcn_r50-d32_jpu_aspp_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_aspp_4x4_512x1024_80k_cityscapes/fastfcn_r50-d32_jpu_aspp_4x4_512x1024_80k_cityscapes_20210924_214357-72220849.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_aspp_4x4_512x1024_80k_cityscapes/fastfcn_r50-d32_jpu_aspp_4x4_512x1024_80k_cityscapes_20210924_214357.log.json) |

| FastFCN + PSPNet    | R-50-D32       | 512x1024  |   80000 | 5.67     | 4.40           | V100   | 79.26 | 80.86         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fastfcn/fastfcn_r50-d32_jpu_psp_4xb2-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_psp_512x1024_80k_cityscapes/fastfcn_r50-d32_jpu_psp_512x1024_80k_cityscapes_20210928_053722-57749bed.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_psp_512x1024_80k_cityscapes/fastfcn_r50-d32_jpu_psp_512x1024_80k_cityscapes_20210928_053722.log.json)                     |

| FastFCN + PSPNet    | R-50-D32 (4x4) | 512x1024  |   80000 | 9.94     | -              | V100   | 78.76 | 80.03         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fastfcn/fastfcn_r50-d32_jpu_psp_4xb2-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_psp_4x4_512x1024_80k_cityscapes/fastfcn_r50-d32_jpu_psp_4x4_512x1024_80k_cityscapes_20210925_061841-77e87b0a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_psp_4x4_512x1024_80k_cityscapes/fastfcn_r50-d32_jpu_psp_4x4_512x1024_80k_cityscapes_20210925_061841.log.json)     |

| FastFCN + EncNet    | R-50-D32       | 512x1024  |   80000 | 8.15     | 4.77           | V100   | 77.97 | 79.92         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fastfcn/fastfcn_r50-d32_jpu_enc_4xb2-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_enc_512x1024_80k_cityscapes/fastfcn_r50-d32_jpu_enc_512x1024_80k_cityscapes_20210928_030036-78da5046.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_enc_512x1024_80k_cityscapes/fastfcn_r50-d32_jpu_enc_512x1024_80k_cityscapes_20210928_030036.log.json)                     |

| FastFCN + EncNet    | R-50-D32 (4x4) | 512x1024  |   80000 | 15.45    | -              | V100   |  78.6 | 80.25         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fastfcn/fastfcn_r50-d32_jpu_enc_4xb2-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_enc_4x4_512x1024_80k_cityscapes/fastfcn_r50-d32_jpu_enc_4x4_512x1024_80k_cityscapes_20210926_093217-e1eb6dbb.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_enc_4x4_512x1024_80k_cityscapes/fastfcn_r50-d32_jpu_enc_4x4_512x1024_80k_cityscapes_20210926_093217.log.json)     |

### ADE20K

### ADE20K

| Method              | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                | download                                                                                                                                                                                                                                                                                                                                                                           |

| FastFCN + DeepLabV3 | R-50-D32 | 512x1024  |   80000 | 8.46     | 12.06          | V100   | 41.88 | 42.91         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fastfcn/fastfcn_r50-d32_jpu_aspp_4xb4-80k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_aspp_512x512_80k_ade20k/fastfcn_r50-d32_jpu_aspp_512x512_80k_ade20k_20211013_190619-3aa40f2d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_aspp_512x512_80k_ade20k/fastfcn_r50-d32_jpu_aspp_512x512_80k_ade20k_20211013_190619.log.json)     |

| FastFCN + DeepLabV3 | R-50-D32 | 512x1024  |  160000 | -        | -              | V100   | 43.58 | 44.92         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fastfcn/fastfcn_r50-d32_jpu_aspp_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_aspp_512x512_160k_ade20k/fastfcn_r50-d32_jpu_aspp_512x512_160k_ade20k_20211008_152246-27036aee.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_aspp_512x512_160k_ade20k/fastfcn_r50-d32_jpu_aspp_512x512_160k_ade20k_20211008_152246.log.json) |

| FastFCN + PSPNet    | R-50-D32 | 512x1024  |   80000 | 8.02     | 19.21          | V100   | 41.40 | 42.12         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fastfcn/fastfcn_r50-d32_jpu_psp_4xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_psp_512x512_80k_ade20k/fastfcn_r50-d32_jpu_psp_512x512_80k_ade20k_20210930_225137-993d07c8.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_psp_512x512_80k_ade20k/fastfcn_r50-d32_jpu_psp_512x512_80k_ade20k_20210930_225137.log.json)         |

| FastFCN + PSPNet    | R-50-D32 | 512x1024  |  160000 | -        | -              | V100   | 42.63 | 43.71         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fastfcn/fastfcn_r50-d32_jpu_psp_4xb4-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_psp_512x512_160k_ade20k/fastfcn_r50-d32_jpu_psp_512x512_160k_ade20k_20211008_105455-e8f5a2fd.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_psp_512x512_160k_ade20k/fastfcn_r50-d32_jpu_psp_512x512_160k_ade20k_20211008_105455.log.json)     |

| FastFCN + EncNet    | R-50-D32 | 512x1024  |   80000 | 9.67     | 17.23          | V100   | 40.88 | 42.36         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fastfcn/fastfcn_r50-d32_jpu_enc_4xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_enc_512x512_80k_ade20k/fastfcn_r50-d32_jpu_enc_512x512_80k_ade20k_20210930_225214-65aef6dd.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_enc_512x512_80k_ade20k/fastfcn_r50-d32_jpu_enc_512x512_80k_ade20k_20210930_225214.log.json)         |

| FastFCN + EncNet    | R-50-D32 | 512x1024  |  160000 | -        | -              | V100   | 42.50 | 44.21         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fastfcn/fastfcn_r50-d32_jpu_enc_4xb4-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_enc_512x512_160k_ade20k/fastfcn_r50-d32_jpu_enc_512x512_160k_ade20k_20211008_105456-d875ce3c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fastfcn/fastfcn_r50-d32_jpu_enc_512x512_160k_ade20k/fastfcn_r50-d32_jpu_enc_512x512_160k_ade20k_20211008_105456.log.json)     |

Note:

- `4x4` means 4 GPUs with 4 samples per GPU in training, default setting is 4 GPUs with 2 samples per GPU in training.

Note:

- Results of [DeepLabV3 (mIoU: 79.32)](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3), [PSPNet (mIoU: 78.55)](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet) and [ENCNet (mIoU: 77.94)](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/encnet) can be found in each original repository.

```bibtex
@article{wu2019fastfcn,
title={Fastfcn: Rethinking dilated convolution in the backbone for semantic segmentation},
author={Wu, Huikai and Zhang, Junge and Huang, Kaiqi and Liang, Kongming and Yu, Yizhou},
journal={arXiv preprint arXiv:1903.11816},
}
```

# TTSR (CVPR'2020)

<details>
<summary align="right">TTSR (CVPR'2020)</summary>

```bibtex
@inproceedings{yang2020learning,
  title={Learning texture transformer network for image super-resolution},
  author={Yang, Fuzhi and Yang, Huan and Fu, Jianlong and Lu, Hongtao and Guo, Baining},
  pages={5791--5800},
}
```

<br/>

|                                    算法                                    | scale |      CUFED       |   GPU 信息   |                                             下载                                              |
| :------------------------------------------------------------------------: | :---: | :--------------: | :----------: | :-------------------------------------------------------------------------------------------: |

| [ttsr-rec_x4_c64b16_g1_200k_CUFED](./ttsr-rec_x4c64b16_1xb9-200k_CUFED.py) |  x4   | 25.2433 / 0.7491 | 1 (TITAN Xp) | [模型](https://download.openmmlab.com/mmediting/restorers/ttsr/ttsr-rec_x4_c64b16_g1_200k_CUFED_20210525-b0dba584.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/ttsr/ttsr-rec_x4_c64b16_g1_200k_CUFED_20210525-b0dba584.log.json) |
| [ttsr-gan_x4_c64b16_g1_500k_CUFED](./ttsr-gan_x4c64b16_1xb9-500k_CUFED.py) |  x4   | 24.6075 / 0.7234 | 1 (TITAN Xp) | [模型](https://download.openmmlab.com/mmediting/restorers/ttsr/ttsr-gan_x4_c64b16_g1_500k_CUFED_20210626-2ab28ca0.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/ttsr/ttsr-gan_x4_c64b16_g1_500k_CUFED_20210626-2ab28ca0.log.json) |

## 快速开始

**训练**

<details>

```shell
# CPU上训练
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/ttsr/ttsr-gan_x4c64b16_1xb9-500k_CUFED.py

# 单个GPU上训练
python tools/train.py configs/ttsr/ttsr-gan_x4c64b16_1xb9-500k_CUFED.py

```shell
# CPU上训练

# 单个GPU上训练

# 多个GPU上训练
./tools/dist_train.sh configs/ttsr/ttsr-gan_x4c64b16_1xb9-500k_CUFED.py 8
```

**测试**

<details>

```shell
# CPU上测试
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/ttsr/ttsr-gan_x4c64b16_1xb9-500k_CUFED.py https://download.openmmlab.com/mmediting/restorers/ttsr/ttsr-gan_x4_c64b16_g1_500k_CUFED_20210626-2ab28ca0.pth

# 单个GPU上测试
python tools/test.py configs/ttsr/ttsr-gan_x4c64b16_1xb9-500k_CUFED.py https://download.openmmlab.com/mmediting/restorers/ttsr/ttsr-gan_x4_c64b16_g1_500k_CUFED_20210626-2ab28ca0.pth

# 多个GPU上测试

# 多个GPU上测试
./tools/dist_test.sh configs/ttsr/ttsr-gan_x4c64b16_1xb9-500k_CUFED.py https://download.openmmlab.com/mmediting/restorers/ttsr/ttsr-gan_x4_c64b16_g1_500k_CUFED_20210626-2ab28ca0.pth 8
```

# 数据集迁移

在 OpenMMLab 2.0 系列算法库基于 [MMEngine](https://github.com/open-mmlab/mmengine) 设计了统一的数据集基类 [BaseDataset](mmengine.dataset.BaseDataset)，并制定了数据集标注文件规范。基于此，我们在 MMOCR 1.0 版本中重构了 OCR 任务数据集基类 [`OCRDataset`](mmocr.datasets.OCRDataset)。以下文档将介绍 MMOCR 中新旧数据集格式的区别，以及如何将旧数据集迁移至新版本中。对于暂不方便进行数据迁移的用户，我们也在[第三节](#兼容性)提供了临时的代码兼容方案。

```{note}
关键信息抽取任务仍采用原有的 WildReceipt 数据集标注格式。
```

## 旧版数据格式回顾

针对不同任务，MMOCR 0.x 版本实现了多种不同的数据集类型，如文本检测任务的 `IcdarDataset`，`TextDetDataset`；文本识别任务的 `OCRDataset`，`OCRSegDataset` 等。而不同的数据集类型同时还可能存在多种不同的标注及文件存储后端，如 `.txt`、`.json`、`.jsonl` 等，使得用户在自定义数据集时需要配置各类数据加载器 (`Loader`) 以及数据解析器 (`Parser`)。这不仅增加了用户的使用难度，也带来了许多问题和隐患。例如，以 `.txt` 格式存储的简单 `OCDDataset` 在遇到包含空格的文本标注时将会报错。

### 文本检测

文本检测任务中，`IcdarDataset` 采用了与通用目标检测 COCO 数据集一致的标注格式。

```json
{
    {
      "id": 1,
    }
  ],
    {
      "id": 1,

"width": 800,
    }
  ],
    {
      "id": 1,
      ],
    }
  ]
}
```

而 `TextDetDataset` 则采用了 JSON Line 的存储格式，将类似 COCO 格式的标签转换成文本存放在 `.txt` 或 `.jsonl` 格式文件中。

```text

```

### 文本识别

对于文本识别任务，MMOCR 0.x 版本中存在两种数据标注格式。其中 `.txt` 格式的标注文件每一行共有两个字段，分别存放了图片名以及标注的文本内容，并以空格分隔。

```text
```

而 JSON Line 格式则使用 `json.dumps` 将 JSON 格式的标注转换为文本内容后存放在 .jsonl 文件中，其内容形似一个字典，将文件名和文本标注信息分别存放在 `filename` 和 `text` 字段中。

```json
```

## 新版数据格式

为解决 0.x 版本中数据集格式过于混杂的情况，MMOCR 1.x 采用了基于 MMEngine 设计的统一数据标准。每一个数据标注文件存放在 `.json` 文件中，并使用类似字典的格式分别存放了数据集的元信息（`metainfo`）与具体的标注内容（`data_list`）。

```json
{
    {
      // ...
    },
    [
      {

```json
{
    {
      // ...
    },
    [
      {
        // ...
      },
      // ...
    ]
}
```

基于此，我们针对 MMOCR 特有的任务设计了 `TextDetDataset`、`TextRecogDataset`。

### 文本检测

#### 新版格式介绍

`TextDetDataset` 中存放了文本检测任务所需的边界盒标注、文件名等信息。由于文本检测任务中只有 1 个类别，因此我们将其类别 id 默认设置为 0，而背景类则为 1。`tests/data/det_toy_dataset/instances_test.json` 中存放了一个文本检测任务的数据标注示例，用户可以参考该文件来将自己的数据集转换为我们支持的格式。

```json
{
    {
    },
    [
      {
          [
            {
            },
            // ...

"polygon": [0, 0, 0, 10, 10, 20, 20, 0],
            },
            // ...
          ]
      }
    ]
}
```

其中，`bbox` 字段的格式为 `[min_x, min_y, max_x, max_y]`。

#### 迁移脚本

为帮助用户将旧版本标注文件迁移至新格式，我们提供了迁移脚本。使用方法如下：

```bash
```

| 参数     | 类型                             | 说明                                                                                                                                               |
| -------- | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| in_path  | str                              | （必须）旧版标注的路径                                                                                                                             |

| out_path | str                              | （必须）新版标注的路径                                                                                                                             |
| --task   | 'auto', 'textdet', 'textspotter' | 指定输出数据集标注的所兼容的任务。若指定为 textdet ，则不会转存 coco 格式中的 text 字段。默认为 auto，即根据旧版标注的格式自动决定输出的标注格式。 |

### 文本识别

`TextRecogDataset` 中存放了文本识别任务所需的文本内容，通常而言，文本识别数据集中的每一张图片都仅包含一个文本实例。我们在 `tests/data/rec_toy_dataset/labels.json` 提供了一个简单的识别数据格式示例，用户可以参考该文件以进一步了解其中的细节。

```json
{
    {
    },
    [
      {
            [
              {
              }
            ]
      }
    ]
}
```

"data_list":
    [
      {
            [
              {
              }
            ]
      }
    ]
}
```

#### 迁移脚本

```bash
```

| 参数     | 类型                   | 说明                       |
| -------- | ---------------------- | -------------------------- |
| in_path  | str                    | （必须）旧版标注的路径     |
| out_path | str                    | （必须）新版标注的路径     |
| --format | 'txt', 'jsonl', 'lmdb' | 指定旧版数据集标注的格式。 |

## 兼容性

考虑到用户对数据迁移所需的成本，我们在 MMOCR 1.x 版本中暂时对 MMOCR 0.x  旧版本格式进行了兼容。

```{note}
用于兼容旧数据格式的代码和组件可能在未来的版本中被完全移除。因此，我们强烈建议用户将数据集迁移至新的数据格式标准。
```

## 兼容性

```{note}
```

具体而言，我们提供了三个临时的数据集类 [IcdarDataset](mmocr.datasets.IcdarDataset), [RecogTextDataset](mmocr.datasets.RecogTextDataset), [RecogLMDBDataset](mmocr.datasets.RecogLMDBDataset) 来兼容旧格式的标注文件。分别对应了 MMOCR 0.x 版本中的文本检测数据集 `IcdarDataset`，`.txt`、`.jsonl` 和 `LMDB` 格式的文本识别数据标注。其使用方式与 0.x 版本一致。

1. [IcdarDataset](mmocr.datasets.IcdarDataset) 支持 0.x 版本文本检测任务的 COCO 标注格式。只需要在 `configs/textdet/_base_/datasets` 中添加新的数据集配置文件，并指定其数据集类型为 `IcdarDataset` 即可。

   ```python
     data_root = 'data/det/icdar2015'

     train_dataset = dict(
         type='IcdarDataset',
         data_root=data_root,
         ann_file='instances_training.json',
         data_prefix=dict(img_path='imgs/'),
         filter_cfg=dict(filter_empty_gt=True, min_size=32),
         pipeline=None)
   ```

ann_file='instances_training.json',
   ```

2. [RecogTextDataset](mmocr.datasets.RecogTextDataset) 支持 0.x 版本文本识别任务的 `txt` 和 `jsonl` 标注格式。只需要在 `configs/textrecog/_base_/datasets` 中添加新的数据集配置文件，并指定其数据集类型为 `RecogTextDataset` 即可。例如，以下示例展示了如何配置并读取 toy dataset 中的旧格式标签 `old_label.txt` 以及 `old_label.jsonl`。

   ```python

    # 读取旧版 txt 格式识别数据标签

    # 读取旧版 json line 格式识别数据标签

pipeline=[])

   ```

3. [RecogLMDBDataset](mmocr.datasets.RecogLMDBDataset) 支持 0.x 版本文本识别任务**图像+文字**的 `LMDB` 标注格式。只需要在 `configs/textrecog/_base_/datasets` 中添加新的数据集配置文件，并指定其数据集类型为 `RecogLMDBDataset` 即可。例如，以下示例展示了如何配置并读取 toy dataset 中的 `imgs.lmdb`，该 `lmdb` 文件**包含标签和图像**。

   ```python
   # 将数据集类型设定为 RecogLMDBDataset

    lmdb_dataset = dict(
        type='RecogLMDBDataset',
        ann_file='imgs.lmdb',
        pipeline=None)
   ```

data_root = 'tests/data/rec_toy_dataset/'

   ```

   还需把 `train_pipeline` 及 `test_pipeline` 中的数据读取方法如 [`LoadImageFromFile`](mmocr.datasets.transforms.LoadImageFromFile) 替换为 [`LoadImageFromNDArray`](mmocr.datasets.transforms.LoadImageFromNDArray)：

   ```python
    train_pipeline = [dict(type='LoadImageFromNDArray')]
   ```

# VideoMAE

[VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://arxiv.org/abs/2203.12602)

Pre-training video transformers on extra large-scale datasets is generally required to achieve premier performance on relatively small datasets. In this paper, we show that video masked autoencoders (VideoMAE) are data-efficient learners for self-supervised video pre-training (SSVP). We are inspired by the recent ImageMAE and propose customized video tube masking with an extremely high ratio. This simple design makes video reconstruction a more challenging self-supervision task, thus encouraging extracting more effective video representations during this pre-training process. We obtain three important findings on SSVP: (1) An extremely high proportion of masking ratio (i.e., 90% to 95%) still yields favorable performance of VideoMAE. The temporally redundant video content enables a higher masking ratio than that of images. (2) VideoMAE achieves impressive results on very small datasets (i.e., around 3k-4k videos) without using any extra data. (3) VideoMAE shows that data quality is more important than data quantity for SSVP. Domain shift between pre-training and target datasets is an important issue. Notably, our VideoMAE with the vanilla ViT can achieve 87.4% on Kinetics-400, 75.4% on Something-Something V2, 91.3% on UCF101, and 62.6% on HMDB51, without using any extra data.

<img src="https://user-images.githubusercontent.com/35267818/191656296-14f28f4a-203f-4eeb-a4c3-c2efdb6d1ab4.png" width="800"/>
</div>

|         16x4x1          | short-side 320 |  ViT-B   |   81.3   |   95.0   | 81.5 \[[VideoMAE](https://github.com/MCG-NJU/VideoMAE/blob/main/MODEL_ZOO.md)\] | 95.1 \[[VideoMAE](https://github.com/MCG-NJU/VideoMAE/blob/main/MODEL_ZOO.md)\] | 5 clips x 3 crops | 180G  |  87M   | [config](/configs/recognition/videomae/vit-base-p16_videomae-k400-pre_16x4x1_kinetics-400.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/videomae/vit-base-p16_videomae-k400-pre_16x4x1_kinetics-400_20221013-860a3cd3.pth) \[1\] |

|         16x4x1          | short-side 320 |  ViT-L   |   85.3   |   96.7   | 85.2 \[[VideoMAE](https://github.com/MCG-NJU/VideoMAE/blob/main/MODEL_ZOO.md)\] | 96.8 \[[VideoMAE](https://github.com/MCG-NJU/VideoMAE/blob/main/MODEL_ZOO.md)\] | 5 clips x 3 crops | 597G  |  305M  | [config](/configs/recognition/videomae/vit-large-p16_videomae-k400-pre_16x4x1_kinetics-400.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/videomae/vit-large-p16_videomae-k400-pre_16x4x1_kinetics-400_20221013-229dbb03.pth) \[1\] |

\[1\] The models are ported from the repo [VideoMAE](https://github.com/MCG-NJU/VideoMAE) and tested on our data. Currently, we only support the testing of VideoMAE models, training will be available soon.

```shell
```

```shell
python tools/test.py configs/recognition/videomae/vit-base-p16_videomae-k400-pre_16x4x1_kinetics-400.py \
```

```BibTeX
@inproceedings{tong2022videomae,
  title={Video{MAE}: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training},
  author={Zhan Tong and Yibing Song and Jue Wang and Limin Wang},
  booktitle={Advances in Neural Information Processing Systems},
}
```

}
```

<details>

```bibtex
}
```

| [pose_resnet_50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_res50_8xb64-210e_coco-256x192.py) |  256x192   | 0.718 |      0.898      |      0.796      | 0.774 |      0.934      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res50_8xb64-210e_coco-256x192-04af38ce_20220923.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res50_8xb64-210e_coco-256x192_20220923.log) |
| [pose_resnet_50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_res50_8xb64-210e_coco-384x288.py) |  384x288   | 0.731 |      0.900      |      0.799      | 0.782 |      0.937      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res50_8xb64-210e_coco-384x288-7b8db90e_20220923.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res50_8xb64-210e_coco-384x288_20220923.log) |

| [pose_resnet_101](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_res101_8xb64-210e_coco-256x192.py) |  256x192   | 0.728 |      0.904      |      0.809      | 0.783 |      0.942      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res101_8xb64-210e_coco-256x192-065d3625_20220926.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res101_8xb64-210e_coco-256x192_20220926.log) |
| [pose_resnet_101](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_res101_8xb32-210e_coco-384x288.py) |  384x288   | 0.749 |      0.906      |      0.817      | 0.799 |      0.941      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res101_8xb64-210e_coco-256x192-065d3625_20220926.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res101_8xb64-210e_coco-256x192_20220926.log) |

| [pose_resnet_152](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_res152_8xb32-210e_coco-256x192.py) |  256x192   | 0.736 |      0.904      |      0.818      | 0.791 |      0.942      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res152_8xb32-210e_coco-256x192-0345f330_20220928.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res152_8xb32-210e_coco-256x192_20220928.log) |
| [pose_resnet_152](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_res152_8xb32-210e_coco-384x288.py) |  384x288   | 0.750 |      0.908      |      0.821      | 0.800 |      0.942      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res152_8xb32-210e_coco-384x288-7fbb906f_20220927.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res152_8xb32-210e_coco-384x288_20220927.log) |

# PANet

> [Efficient and Accurate Arbitrary-Shaped Text Detection with Pixel Aggregation Network](https://arxiv.org/abs/1908.05900)

Scene text detection, an important step of scene text reading systems, has witnessed rapid development with convolutional neural networks. Nonetheless, two main challenges still exist and hamper its deployment to real-world applications. The first problem is the trade-off between speed and accuracy. The second one is to model the arbitrary-shaped text instance. Recently, some methods have been proposed to tackle arbitrary-shaped text detection, but they rarely take the speed of the entire pipeline into consideration, which may fall short in practical this http URL this paper, we propose an efficient and accurate arbitrary-shaped text detector, termed Pixel Aggregation Network (PAN), which is equipped with a low computational-cost segmentation head and a learnable post-processing. More specifically, the segmentation head is made up of Feature Pyramid Enhancement Module (FPEM) and Feature Fusion Module (FFM). FPEM is a cascadable U-shaped module, which can introduce multi-level information to guide the better segmentation. FFM can gather the features given by the FPEMs of different depths into a final feature for segmentation. The learnable post-processing is implemented by Pixel Aggregation (PA), which can precisely aggregate text pixels by predicted similarity vectors. Experiments on several standard benchmarks validate the superiority of the proposed PAN. It is worth noting that our method can achieve a competitive F-measure of 79.9% at 84.2 FPS on CTW1500.

<img src="https://user-images.githubusercontent.com/22607038/142795741-0e1ea962-1596-47c2-8671-27bbe87d0df8.png"/>
</div>

|                            Method                            | Pretrained Model | Training set  |   Test set   | #epochs | Test size | Precision | Recall | Hmean  |                            Download                            |
| :----------------------------------------------------------: | :--------------: | :-----------: | :----------: | :-----: | :-------: | :-------: | :----: | :----: | :------------------------------------------------------------: |

| [PANet](/configs/textdet/panet/panet_resnet18_fpem-ffm_600e_ctw1500.py) |     ImageNet     | CTW1500 Train | CTW1500 Test |   600   |    640    |  0.8208   | 0.7376 | 0.7770 | [model](https://download.openmmlab.com/mmocr/textdet/panet/panet_resnet18_fpem-ffm_600e_ctw1500/panet_resnet18_fpem-ffm_600e_ctw1500_20220826_144818-980f32d0.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/panet/panet_resnet18_fpem-ffm_600e_ctw1500/20220826_144818.log) |

|                           Method                           | Pretrained Model |  Training set   |    Test set    | #epochs | Test size | Precision | Recall | Hmean  |                           Download                           |
| :--------------------------------------------------------: | :--------------: | :-------------: | :------------: | :-----: | :-------: | :-------: | :----: | :----: | :----------------------------------------------------------: |

| [PANet](/configs/textdet/panet/panet_resnet18_fpem-ffm_600e_icdar2015.py) |     ImageNet     | ICDAR2015 Train | ICDAR2015 Test |   600   |    736    |  0.8455   | 0.7323 | 0.7848 | [model](https://download.openmmlab.com/mmocr/textdet/panet/panet_resnet18_fpem-ffm_600e_icdar2015/panet_resnet18_fpem-ffm_600e_icdar2015_20220826_144817-be2acdb4.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/panet/panet_resnet18_fpem-ffm_600e_icdar2015/20220826_144817.log) |

```bibtex
@inproceedings{WangXSZWLYS19,
  author={Wenhai Wang and Enze Xie and Xiaoge Song and Yuhang Zang and Wenjia Wang and Tong Lu and Gang Yu and Chunhua Shen},
  title={Efficient and Accurate Arbitrary-Shaped Text Detection With Pixel Aggregation Network},
  booktitle={ICCV},
  pages={8439--8448},
  }
```

<details>

```bibtex
}
```

<details>

```bibtex

}
```

| [pose_hrnet_w32](/configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_hrnet-w32_8xb64-210e_coco-wholebody-256x192.py) |  256x192   |  0.678  |  0.755  |  0.543  |  0.661  |  0.630  |  0.708  |  0.467  |  0.566  |  0.536   |  0.636   | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_coco_wholebody_256x192-853765cd_20200918.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_coco_wholebody_256x192_20200918.log.json) |
| [pose_hrnet_w32](/configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_hrnet-w32_8xb64-210e_coco-wholebody-384x288.py) |  384x288   |  0.700  |  0.772  |  0.585  |  0.691  |  0.726  |  0.783  |  0.515  |  0.603  |  0.586   |  0.673   | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_coco_wholebody_384x288-78cacac3_20200922.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_coco_wholebody_384x288_20200922.log.json) |

| [pose_hrnet_w48](/configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_hrnet-w48_8xb32-210e_coco-wholebody-256x192.py) |  256x192   |  0.701  |  0.776  |  0.675  |  0.787  |  0.656  |  0.743  |  0.535  |  0.639  |  0.579   |  0.681   | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_coco_wholebody_256x192-643e18cb_20200922.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_coco_wholebody_256x192_20200922.log.json) |
| [pose_hrnet_w48](/configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_hrnet-w48_8xb32-210e_coco-wholebody-384x288.py) |  384x288   |  0.722  |  0.791  |  0.696  |  0.801  |  0.776  |  0.834  |  0.587  |  0.678  |  0.632   |  0.717   | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_coco_wholebody_384x288-6e061c6a_20200922.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_coco_wholebody_384x288_20200922.log.json) |

# RTMPose

|   Model   | Input Size | PCK@0.2 |  AUC  | EPE  |                                  Details and Download                                  |
| :-------: | :--------: | :-----: | :---: | :--: | :------------------------------------------------------------------------------------: |
| RTMPose-m |  256x256   |  0.815  | 0.837 | 4.51 | [rtmpose_coco_wholebody_hand.md](./coco_wholebody_hand/rtmpose_coco_wholebody_hand.md) |

ncnn openmp best practice

### CPU loadaverage is too high with ncnn.

   When inference the neural network with ncnn, the cpu occupancy is very high even all CPU cores occupancy close to 100%.

   If there are other threads or processes that require more cpu resources, the running speed of the program will drop severely.

### The root cause of high CPU usage

1. ncnn uses openmp API to speed up the inference compute. the thread count equals to the cpu core   count. If the computing work need to run frequently, it must consume many cpu resources.

2. There is a thread pool managed by openmp, the pool size is equal to the cpu core size. (the max  vulue is 15 if there are much more cpu cores?)
   Openmp need to sync the thread when acquiring and returning threads to the pool. In order to improve efficiency, almost all omp implementations use spinlock synchronization (except for simpleomp).

Openmp need to sync the thread when acquiring and returning threads to the pool. In order to improve efficiency, almost all omp implementations use spinlock synchronization (except for simpleomp). 
   The default spin time of the spinlock is 200ms. So after a thread is scheduled, the thread need to busy-wait up to 200ms.

### Why the CPU usage is still high even using vulkan GPU acceleration.

1. Openmp is also used when loading the param bin file, and this part runs on cpu.

2. The fp32 to fp16 conversion before and after the GPU memory upload is executed on the cpu, and this part of the logic also uses openmp.

### Solution
```
1. Bind to the specific cpu core.
```
   If you use a device with large and small core CPUs, it is recommended to bind large or small cores through ncnn::set_cpu_powersave(int). Note that Windows does not support binding cores. By the way,  it's possible to have multiple threadpool using openmp. A new threadpool will be created for a new thread scope.

Suppose your platform is 2 big cores + 4 little cores, and you want to execute model A on 2 big cores and model B on 4 little cores concurrently.

create two threads via std::thread or pthread
   ```
   {
   }

   {
   }
   ```
   
```
2. Use fewer threads.
```
   Set the number of threads to half of the cpu cores count or less through ncnn::set_omp_num_threads(int)  or change net.opt.num_threads field. If you are coding with clang libomp, it's recommended that the number of threads does not exceed 8. If you use other omp libraries, it is recommended that the number of threads does not exceed 4.
```
3. Reduce openmp spinlock blocktime.
```
   You can modify openmp blocktime by call ncnn::set_kmp_blocktime(int) method or modify net.opt.openmp_blocktime field.

```
```
   This argument is the spin time set by the ncnn API, and the default is 20ms.You can set a smaller value according to
   the situation, or directly change it to 0.

   Limitations: At present, only the libomp library of clang is implemented. Neither vcomp nor libgomp have corresponding interfaces.
   If it is not compiled with clang, this value is still 200ms by default.
   If you use vcomp or libgomp, you can use the environment variable OMP_WAIT_POLICY=PASSIVE to disable spin time. If you use simpleomp,
   It's no need to set this parameter.
```
4. Limit the number of threads available in the openmp thread pool.
```
   Even if the number of openmp threads is reduced, the CPU occupancy rate may still be high. This is more common on servers with
   particularly many CPU cores.

```
   particularly many CPU cores. 
   This is because the waiting threads in the thread pool use a spinlock to busy-wait, which can be reducedby limiting the number of
   threads available in the thread pool.

   Generally, you can set the OMP_THREAD_LIMIT environment variable. simpleomp currently does not support this feature so it's no need to be set.
   Note that this environment variable is only valid if it is set before the program starts.
```
5. Disable openmp completely
```
   If there is only one cpu core, or use the vulkan gpu acceleration, it is recommended to disable openmp, just specify -DNCNN_OPENMP=OFF
   when compiling with cmake.

# DIC (CVPR'2020)

<details>

<summary align="right">DIC (CVPR'2020)</summary>

```bibtex
@inproceedings{ma2020deep,
  title={Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation},
  author={Ma, Cheng and Jiang, Zhenyu and Rao, Yongming and Lu, Jiwen and Zhou, Jie},
  pages={5569--5578},
}
```

<br/>

在 `dic_gan_x8c48b6_g4_150k_CelebAHQ` 的日志中，DICGAN 在 CelebA-HQ 测试集的前9张图片上进行了验证，因此下表中的 `PSNR/SSIM` 与日志数据不同。

`GPU 信息`: 训练过程中的 GPU 信息.

|                                     算法                                     | scale |    CelebA-HQ     |      GPU 信息       |                                         下载                                         |

| :--------------------------------------------------------------------------: | :---: | :--------------: | :-----------------: | :----------------------------------------------------------------------------------: |
|     [dic_x8c48b6_g4_150k_CelebAHQ](./dic_x8c48b6_4xb2-150k_celeba-hq.py)     |  x8   | 25.2319 / 0.7422 | 4 (Tesla PG503-216) | [模型](https://download.openmmlab.com/mmediting/restorers/dic/dic_x8c48b6_g4_150k_CelebAHQ_20210611-5d3439ca.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/dic/dic_x8c48b6_g4_150k_CelebAHQ_20210611-5d3439ca.log.json) |
| [dic_gan_x8c48b6_g4_500k_CelebAHQ](./dic_gan-x8c48b6_4xb2-500k_celeba-hq.py) |  x8   | 23.6241 / 0.6721 | 4 (Tesla PG503-216) | [模型](https://download.openmmlab.com/mmediting/restorers/dic/dic_gan_x8c48b6_g4_500k_CelebAHQ_20210625-3b89a358.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/dic/dic_gan_x8c48b6_g4_500k_CelebAHQ_20210625-3b89a358.log.json) |

## 快速开始

**训练**

<details>

## 快速开始

**训练**

<details>

```shell
# CPU上训练
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/dic/dic_gan-x8c48b6_4xb2-500k_celeba-hq.py

# 单个GPU上训练
python tools/train.py configs/dic/dic_gan-x8c48b6_4xb2-500k_celeba-hq.py

# 多个GPU上训练
./tools/dist_train.sh configs/dic/dic_gan-x8c48b6_4xb2-500k_celeba-hq.py 8
```

**测试**

<details>

```shell
# CPU上测试
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/dic/dic_gan-x8c48b6_4xb2-500k_celeba-hq.py https://download.openmmlab.com/mmediting/restorers/dic/dic_gan_x8c48b6_g4_500k_CelebAHQ_20210625-3b89a358.pth

# 单个GPU上测试
python tools/test.py configs/dic/dic_gan-x8c48b6_4xb2-500k_celeba-hq.py https://download.openmmlab.com/mmediting/restorers/dic/dic_gan_x8c48b6_g4_500k_CelebAHQ_20210625-3b89a358.pth

# 多个GPU上测试

# 多个GPU上测试
./tools/dist_test.sh configs/dic/dic_gan-x8c48b6_4xb2-500k_celeba-hq.py https://download.openmmlab.com/mmediting/restorers/dic/dic_gan_x8c48b6_g4_500k_CelebAHQ_20210625-3b89a358.pth 8
```

  <img src="docs/zh_cn/_static/image/logo.svg" width="500px"/>
  <br />
  <br />

[![docs](https://readthedocs.org/projects/opencompass/badge)](https://opencompass.readthedocs.io/zh_CN)
[![license](https://img.shields.io/github/license/InternLM/opencompass.svg)](https://github.com/InternLM/opencompass/blob/main/LICENSE)

<!-- [![PyPI](https://badge.fury.io/py/opencompass.svg)](https://pypi.org/project/opencompass/) -->

[🌐Website](https://opencompass.org.cn/) |
[📘Documentation](https://opencompass.readthedocs.io/zh_CN/latest/index.html) |
[🛠️Installation](https://opencompass.readthedocs.io/zh_CN/latest/get_started.html#id1) |
[🤔Reporting Issues](https://github.com/InternLM/opencompass/issues/new/choose)

</div>

</div>

    👋 加入我们的<a href="https://twitter.com/intern_lm" target="_blank">推特</a>、<a href="https://discord.gg/xa29JuW87d" target="_blank">Discord</a> 和 <a href="https://r.vansin.top/?r=internwx" target="_blank">微信社区</a>
</p>

## 🧭	欢迎

来到**OpenCompass**！

就像指南针在我们的旅程中为我们导航一样，我们希望OpenCompass能够帮助你穿越评估大型语言模型的重重迷雾。OpenCompass提供丰富的算法和功能支持，期待OpenCompass能够帮助社区更便捷地对NLP模型的性能进行公平全面的评估。

> **🔥 注意**<br />
> 我们正式启动 OpenCompass 共建计划，诚邀社区用户为 OpenCompass 提供更具代表性和可信度的客观评测数据集!
> 点击 [Issue](https://github.com/InternLM/opencompass/issues/248) 获取更多数据集.
> 让我们携手共进，打造功能强大易用的大模型评测平台！

## 🚀 最新进展 <a><img width="35" height="20" src="https://user-images.githubusercontent.com/12782558/212848161-5e783dd6-11e8-4fe0-bbba-39ffb77730be.png"></a>

- **\[2023.08.25\]**  欢迎 [**TigerBot**](https://github.com/TigerResearch/TigerBot) 团队采用OpenCompass对模型进行系统评估。我们非常感谢社区在提升LLM评估的透明度和可复现性上所做的努力。
  🔥🔥🔥.

  🔥🔥🔥.
- **\[2023.08.21\]** [**Lagent**](https://github.com/InternLM/lagent) 正式发布，它是一个轻量级、开源的基于大语言模型的智能体（agent）框架。我们正与Lagent团队紧密合作，推进支持基于Lagent的大模型工具能力评测 ! 🔥🔥🔥.
- **\[2023.08.18\]** OpenCompass现已支持**多模态评测**，支持10+多模态评测数据集，包括 **MMBench, SEED-Bench, COCO-Caption, Flickr-30K, OCR-VQA, ScienceQA** 等. 多模态评测榜单即将上线，敬请期待!
- **\[2023.08.18\]** [数据集页面](https://opencompass.org.cn/dataset-detail/MMLU) 现已在OpenCompass官网上线，欢迎更多社区评测数据集加入OpenCompass !
- **\[2023.08.11\]** 官网榜单上新增了[模型对比](https://opencompass.org.cn/model-compare/GPT-4,ChatGPT,LLaMA-2-70B,LLaMA-65B)功能，希望该功能可以协助提供更多发现！
- **\[2023.08.11\]** 新增了 [LEval](https://github.com/OpenLMLab/LEval) 评测支持.

> [更多](docs/zh_cn/notes/news.md)

## ✨ 介绍

OpenCompass 是面向大模型评测的一站式平台。其主要特点如下：

- **开源可复现**：提供公平、公开、可复现的大模型评测方案

- **全面的能力维度**：五大维度设计，提供 50+ 个数据集约 30 万题的的模型评测方案，全面评估模型能力

## ✨ 介绍

- **丰富的模型支持**：已支持 20+ HuggingFace 及 API 模型

- **分布式高效评测**：一行命令实现任务分割和分布式评测，数小时即可完成千亿模型全量评测

- **多样化评测范式**：支持零样本、小样本及思维链评测，结合标准型或对话型提示词模板，轻松激发各种模型最大性能

- **灵活化拓展**：想增加新模型或数据集？想要自定义更高级的任务分割策略，甚至接入新的集群管理系统？OpenCompass 的一切均可轻松扩展！

## 📊 性能榜单

我们将陆续提供开源模型和API模型的具体性能榜单，请见 [OpenCompass Leaderbaord](https://opencompass.org.cn/rank) 。如需加入评测，请提供模型仓库地址或标准的 API 接口至邮箱  `opencompass@pjlab.org.cn`.

<p align="right"><a href="#top">🔝返回顶部</a></p>

## 📖 数据集支持

  <tbody>
      <td>
        <b>语言</b>
      </td>
      <td>
        <b>知识</b>
      </td>
      <td>
        <b>推理</b>
      </td>
      <td>
        <b>学科</b>
      </td>
      <td>
        <b>理解</b>
      </td>
    </tr>
      <td>
<summary><b>字词释义</b></summary>

- WiC

</td>
      <td>
        <b>学科</b>
      </td>
      <td>
        <b>理解</b>
      </td>
    </tr>
      <td>

- WiC
- SummEdits

<summary><b>成语习语</b></summary>

- CHID

<summary><b>语义相似度</b></summary>

- AFQMC
- BUSTM

<summary><b>指代消解</b></summary>

- CLUEWSC
- WSC
- WinoGrande

<summary><b>翻译</b></summary>

- Flores

      </td>
      <td>
<summary><b>知识问答</b></summary>

- BoolQ
- CommonSenseQA
- NaturalQuestion
- TrivialQA

<summary><b>多语种问答</b></summary>

- TyDi-QA

      </td>
      <td>
<summary><b>文本蕴含</b></summary>

- CMNLI
- OCNLI
- OCNLI_FC
- AX-b
- AX-g
- CB
- RTE

<summary><b>常识推理</b></summary>

- StoryCloze
- StoryCloze-CN（即将上线）
- COPA
- ReCoRD
- HellaSwag
- PIQA
- SIQA

- OCNLI
- OCNLI_FC
- AX-b
- AX-g
- CB
- RTE

- COPA
- ReCoRD
- PIQA
- SIQA

<summary><b>数学推理</b></summary>

- MATH
- GSM8K

<summary><b>定理应用</b></summary>

- TheoremQA

<summary><b>代码</b></summary>

- HumanEval
- MBPP

<summary><b>综合推理</b></summary>

- BBH

      </td>
      <td>
<summary><b>初中/高中/大学/职业考试</b></summary>

- GAOKAO-2023
- CEval
- AGIEval
- MMLU
- GAOKAO-Bench
- CMMLU
- ARC

      </td>
      <td>
<summary><b>阅读理解</b></summary>

- C3
- CMRC
- DRCD
- MultiRC
- RACE

<summary><b>内容总结</b></summary>

- CSL
- LCSTS
- XSum

<summary><b>内容分析</b></summary>

- EPRSTMT
- LAMBADA
- TNEWS

      </td>
    </tr>
</td>
    </tr>
  </tbody>
</table>

- CSL
- LCSTS
- XSum

- EPRSTMT
- LAMBADA
- TNEWS

      </td>
    </tr>
</td>
    </tr>
  </tbody>
</table>

## 📖 模型支持

  <tbody>
      <td>
        <b>开源模型</b>
      </td>
      <td>
        <b>API 模型</b>
      </td>
      <!-- <td>
        <b>自定义模型</b>
      </td> -->
    </tr>
      <td>

- LLaMA
- Vicuna
- Alpaca
- Baichuan
- WizardLM
- ChatGLM-6B
- ChatGLM2-6B
- MPT
- Falcon
- TigerBot
- MOSS
- ……

</td>
<td>

- OpenAI
- Claude (即将推出)
- PaLM (即将推出)
- ……

</td>
<!-- <td>

- GLM
- ……

</td> -->
</tr>
  </tbody>
</table>

## 🛠️ 安装

下面展示了快速安装以及准备数据集的步骤。

```Python
conda create --name opencompass python=3.10 pytorch torchvision pytorch-cuda -c nvidia -c pytorch -y
conda activate opencompass
git clone https://github.com/InternLM/opencompass opencompass
cd opencompass

# 下载数据集到 data/ 处
wget https://github.com/InternLM/opencompass/releases/download/0.1.1/OpenCompassData.zip
unzip OpenCompassData.zip
```

有部分第三方功能,如 Humaneval 以及 Llama,可能需要额外步骤才能正常运行，详细步骤请参考[安装指南](https://opencompass.readthedocs.io/zh_CN/latest/get_started.html)。

## 🏗️ ️评测

确保按照上述步骤正确安装 OpenCompass 并准备好数据集后，请阅读[快速上手](https://opencompass.readthedocs.io/zh_CN/latest/get_started.html#id3)了解如何运行一个评测任务。

更多教程请查看我们的[文档](https://opencompass.readthedocs.io/zh_CN/latest/index.html)。

## 🔜 路线图

- [ ] 主观评测
  - [ ] 发布主观评测榜单
  - [ ] 发布主观评测数据集
- [ ] 长文本
  - [ ] 支持广泛的长文本评测集
  - [ ] 发布长文本评测榜单
- [ ] 代码能力
  - [ ] 发布代码能力评测榜单
  - [ ] 提供非Python语言的评测服务
- [ ] 智能体
  - [ ] 支持丰富的智能体方案
  - [ ] 提供智能体评测榜单
- [ ] 鲁棒性
  - [ ] 支持各类攻击方法

## 👷‍♂️ 贡献

- [ ] 长文本
- [ ] 代码能力
- [ ] 智能体
- [ ] 鲁棒性

## 👷‍♂️ 贡献

我们感谢所有的贡献者为改进和提升 OpenCompass 所作出的努力。请参考[贡献指南](https://opencompass.readthedocs.io/zh_CN/latest/notes/contribution_guide.html)来了解参与项目贡献的相关指引。

## 🤝 致谢

该项目部分的代码引用并修改自 [OpenICL](https://github.com/Shark-NLP/OpenICL)。

该项目部分的数据集和提示词实现修改自 [chain-of-thought-hub](https://github.com/FranxYao/chain-of-thought-hub), [instruct-eval](https://github.com/declare-lab/instruct-eval)

## 🖊️ 引用

```bibtex
@misc{2023opencompass,
    title={OpenCompass: A Universal Evaluation Platform for Foundation Models},
    author={OpenCompass Contributors},
    howpublished = {\url{https://github.com/InternLM/OpenCompass}},
}
```

# Pascal VOC

> [The Pascal Visual Object Classes (VOC) Challenge](https://link.springer.com/article/10.1007/s11263-009-0275-4)

The Pascal Visual Object Classes (VOC) challenge is a benchmark in visual object category recognition and detection, providing the vision and machine learning communities with a standard dataset of images and annotation, and standard evaluation procedures. Organised annually from 2005 to present, the challenge and its associated dataset has become accepted as the benchmark for object detection.

This paper describes the dataset and evaluation procedure. We review the state-of-the-art in evaluated methods for both classification and detection, analyse whether the methods are statistically different, what they are learning from the images (e.g. the object or its context), and what the methods find easy or confuse. The paper concludes with lessons learnt in the three year history of the challenge, and proposes directions for future improvement and extension.

<img src="https://user-images.githubusercontent.com/40661020/143969235-6bb4d665-0470-4bae-825c-492eb4582127.png" height="600"/>
</div>

</div>

|  Architecture   | Backbone |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                         Config                         |                                                                                                                                                                                                                                                       Download                                                                                                                                                                                                                                                       |

| Faster R-CNN C4 |   R-50   |  caffe  |   18k   |          |       -        |  80.9  | [config](./faster-rcnn_r50-caffe-c4_ms-18k_voc0712.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/pascal_voc/faster_rcnn_r50_caffe_c4_mstrain_18k_voc0712//home/dong/code_sensetime/2022Q1/mmdetection/work_dirs/prepare_voc/gather/pascal_voc/faster_rcnn_r50_caffe_c4_mstrain_18k_voc0712/faster_rcnn_r50_caffe_c4_mstrain_18k_voc0712_20220314_234327-847a14d2.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pascal_voc/faster_rcnn_r50_caffe_c4_mstrain_18k_voc0712/faster_rcnn_r50_caffe_c4_mstrain_18k_voc0712_20220314_234327.log.json) |

|  Faster R-CNN   |   R-50   | pytorch |   1x    |   2.6    |       -        |  80.4  |     [config](./faster-rcnn_r50_fpn_1x_voc0712.py)      |                                                                                              [model](https://download.openmmlab.com/mmdetection/v2.0/pascal_voc/faster_rcnn_r50_fpn_1x_voc0712/faster_rcnn_r50_fpn_1x_voc0712_20220320_192712-54bef0f3.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pascal_voc/faster_rcnn_r50_fpn_1x_voc0712/faster_rcnn_r50_fpn_1x_voc0712_20220320_192712.log.json)                                                                                              |

|    Retinanet    |   R-50   | pytorch |   1x    |   2.1    |       -        |  77.3  |      [config](./retinanet_r50_fpn_1x_voc0712.py)       |                                                                                                     [model](https://download.openmmlab.com/mmdetection/v2.0/pascal_voc/retinanet_r50_fpn_1x_voc0712/retinanet_r50_fpn_1x_voc0712_20200617-47cbdd0e.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pascal_voc/retinanet_r50_fpn_1x_voc0712/retinanet_r50_fpn_1x_voc0712_20200616_014642.log.json)                                                                                                      |

|     SSD300      |  VGG16   |    -    |  120e   |    -     |       -        |  76.5  |             [config](./ssd300_voc0712.py)              |                                                                                                                              [model](https://download.openmmlab.com/mmdetection/v2.0/pascal_voc/ssd300_voc0712/ssd300_voc0712_20220320_194658-17edda1b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pascal_voc/ssd300_voc0712/ssd300_voc0712_20220320_194658.log.json)                                                                                                                              |

|     SSD512      |  VGG16   |    -    |  120e   |    -     |       -        |  79.5  |             [config](./ssd512_voc0712.py)              |                                                                                                                              [model](https://download.openmmlab.com/mmdetection/v2.0/pascal_voc/ssd512_voc0712/ssd512_voc0712_20220320_194717-03cefefe.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pascal_voc/ssd512_voc0712/ssd512_voc0712_20220320_194717.log.json)                                                                                                                              |

```latex
@Article{Everingham10,
   author = "Everingham, M. and Van~Gool, L. and Williams, C. K. I. and Winn, J. and Zisserman, A.",
   title = "The Pascal Visual Object Classes (VOC) Challenge",
   journal = "International Journal of Computer Vision",
   volume = "88",
   year = "2010",
   number = "2",
   month = jun,

title = "The Pascal Visual Object Classes (VOC) Challenge",
   pages = "303--338",
}
```

## <a href='https://mmagic.readthedocs.io/en/latest/'>English</a>

## <a href='https://mmagic.readthedocs.io/zh_CN/latest/'>简体中文</a>

# DMNet

> [Dynamic Multi-scale Filters for Semantic Segmentation](https://openaccess.thecvf.com/content_ICCV_2019/papers/He_Dynamic_Multi-Scale_Filters_for_Semantic_Segmentation_ICCV_2019_paper.pdf)

<a href="https://github.com/Junjun2016/DMNet">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/decode_heads/dm_head.py#L93">Code Snippet</a>

Multi-scale representation provides an effective way toaddress scale variation of objects and stuff in semantic seg-mentation. Previous works construct multi-scale represen-tation by utilizing different filter sizes, expanding filter sizeswith dilated filters or pooling grids, and the parameters ofthese filters are fixed after training. These methods oftensuffer from heavy computational cost or have more param-eters, and are not adaptive to the input image during in-ference. To address these problems, this paper proposes aDynamic Multi-scale Network (DMNet) to adaptively cap-ture multi-scale contents for predicting pixel-level semanticlabels. DMNet is composed of multiple Dynamic Convolu-tional Modules (DCMs) arranged in parallel, each of whichexploits context-aware filters to estimate semantic represen-tation for a specific scale. The outputs of multiple DCMsare further integrated for final segmentation. We conductextensive experiments to evaluate our DMNet on three chal-lenging semantic segmentation and scene parsing datasets,PASCAL VOC 2012, Pascal-Context, and ADE20K. DMNetachieves a new record 84.4% mIoU on PASCAL VOC 2012test set without MS COCO pre-trained and post-processing,and also obtains state-of-the-art performance on Pascal-Context and ADE20K.

<img src="https://user-images.githubusercontent.com/24582831/142900781-6215763f-8b71-4e0b-a6b1-c41372db2aa0.png" width="70%"/>
</div>

| DMNet  | R-50-D8  | 512x1024  |   40000 | 7.0      | 3.66           | V100   | 77.78 |         79.14 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dmnet/dmnet_r50-d8_4xb2-40k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r50-d8_512x1024_40k_cityscapes/dmnet_r50-d8_512x1024_40k_cityscapes_20201215_042326-615373cf.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r50-d8_512x1024_40k_cityscapes/dmnet_r50-d8_512x1024_40k_cityscapes-20201215_042326.log.json)     |

| DMNet  | R-101-D8 | 512x1024  |   40000 | 10.6     | 2.54           | V100   | 78.37 |         79.72 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dmnet/dmnet_r101-d8_4xb2-40k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r101-d8_512x1024_40k_cityscapes/dmnet_r101-d8_512x1024_40k_cityscapes_20201215_043100-8291e976.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r101-d8_512x1024_40k_cityscapes/dmnet_r101-d8_512x1024_40k_cityscapes-20201215_043100.log.json) |

| DMNet  | R-50-D8  | 769x769   |   40000 | 7.9      | 1.57           | V100   | 78.49 |         80.27 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dmnet/dmnet_r50-d8_4xb2-40k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r50-d8_769x769_40k_cityscapes/dmnet_r50-d8_769x769_40k_cityscapes_20201215_093706-e7f0e23e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r50-d8_769x769_40k_cityscapes/dmnet_r50-d8_769x769_40k_cityscapes-20201215_093706.log.json)         |

| DMNet  | R-101-D8 | 769x769   |   40000 | 12.0     | 1.01           | V100   | 77.62 |         78.94 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dmnet/dmnet_r101-d8_4xb2-40k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r101-d8_769x769_40k_cityscapes/dmnet_r101-d8_769x769_40k_cityscapes_20201215_081348-a74261f6.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r101-d8_769x769_40k_cityscapes/dmnet_r101-d8_769x769_40k_cityscapes-20201215_081348.log.json)     |

| DMNet  | R-50-D8  | 512x1024  |   80000 | -        | -              | V100   | 79.07 |         80.22 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dmnet/dmnet_r50-d8_4xb2-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r50-d8_512x1024_80k_cityscapes/dmnet_r50-d8_512x1024_80k_cityscapes_20201215_053728-3c8893b9.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r50-d8_512x1024_80k_cityscapes/dmnet_r50-d8_512x1024_80k_cityscapes-20201215_053728.log.json)     |

| DMNet  | R-101-D8 | 512x1024  |   80000 | -        | -              | V100   | 79.64 |         80.67 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dmnet/dmnet_r101-d8_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r101-d8_512x1024_80k_cityscapes/dmnet_r101-d8_512x1024_80k_cityscapes_20201215_031718-fa081cb8.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r101-d8_512x1024_80k_cityscapes/dmnet_r101-d8_512x1024_80k_cityscapes-20201215_031718.log.json) |

| DMNet  | R-50-D8  | 769x769   |   80000 | -        | -              | V100   | 79.22 |         80.55 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dmnet/dmnet_r50-d8_4xb2-80k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r50-d8_769x769_80k_cityscapes/dmnet_r50-d8_769x769_80k_cityscapes_20201215_034006-6060840e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r50-d8_769x769_80k_cityscapes/dmnet_r50-d8_769x769_80k_cityscapes-20201215_034006.log.json)         |

| DMNet  | R-101-D8 | 769x769   |   80000 | -        | -              | V100   | 79.19 |         80.65 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dmnet/dmnet_r101-d8_4xb2-80k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r101-d8_769x769_80k_cityscapes/dmnet_r101-d8_769x769_80k_cityscapes_20201215_082810-7f0de59a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r101-d8_769x769_80k_cityscapes/dmnet_r101-d8_769x769_80k_cityscapes-20201215_082810.log.json)     |

### ADE20K

| DMNet  | R-50-D8  | 512x512   |   80000 | 9.4      | 20.95          | V100   | 42.37 |         43.62 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dmnet/dmnet_r50-d8_4xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r50-d8_512x512_80k_ade20k/dmnet_r50-d8_512x512_80k_ade20k_20201215_144744-f89092a6.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r50-d8_512x512_80k_ade20k/dmnet_r50-d8_512x512_80k_ade20k-20201215_144744.log.json)         |

| DMNet  | R-101-D8 | 512x512   |   80000 | 13.0     | 13.88          | V100   | 45.34 |         46.13 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dmnet/dmnet_r101-d8_4xb4-80k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r101-d8_512x512_80k_ade20k/dmnet_r101-d8_512x512_80k_ade20k_20201215_104812-bfa45311.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r101-d8_512x512_80k_ade20k/dmnet_r101-d8_512x512_80k_ade20k-20201215_104812.log.json)     |

| DMNet  | R-50-D8  | 512x512   |  160000 | -        | -              | V100   | 43.15 |         44.17 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dmnet/dmnet_r50-d8_4xb4-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r50-d8_512x512_160k_ade20k/dmnet_r50-d8_512x512_160k_ade20k_20201215_115313-025ab3f9.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r50-d8_512x512_160k_ade20k/dmnet_r50-d8_512x512_160k_ade20k-20201215_115313.log.json)     |

| DMNet  | R-101-D8 | 512x512   |  160000 | -        | -              | V100   | 45.42 |         46.76 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dmnet/dmnet_r101-d8_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r101-d8_512x512_160k_ade20k/dmnet_r101-d8_512x512_160k_ade20k_20201215_111145-a0bc02ef.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dmnet/dmnet_r101-d8_512x512_160k_ade20k/dmnet_r101-d8_512x512_160k_ade20k-20201215_111145.log.json) |

```bibtex
@InProceedings{He_2019_ICCV,
author = {He, Junjun and Deng, Zhongying and Qiao, Yu},
title = {Dynamic Multi-Scale Filters for Semantic Segmentation},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2019}
}
```

# WGAN-GP (NeurIPS'2017)

> [Improved Training of Wasserstein GANs](https://arxiv.org/abs/1704.00028)

Generative Adversarial Networks (GANs) are powerful generative models, but suffer from training instability. The recently proposed Wasserstein GAN (WGAN) makes progress toward stable training of GANs, but sometimes can still generate only low-quality samples or fail to converge. We find that these problems are often due to the use of weight clipping in WGAN to enforce a Lipschitz constraint on the critic, which can lead to undesired behavior. We propose an alternative to clipping weights: penalize the norm of gradient of the critic with respect to its input. Our proposed method performs better than standard WGAN and enables stable training of a wide variety of GAN architectures with almost no hyperparameter tuning, including 101-layer ResNets and language models over discrete data. We also achieve high quality generations on CIFAR-10 and LSUN bedrooms.

<img src="https://user-images.githubusercontent.com/28132635/143154792-de359728-101b-4ad1-90c0-ef3c1572d184.png"/>
</div>

  <b> WGAN-GP 128, CelebA-Cropped</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/12726765/113997469-c00e3f00-988a-11eb-81dc-19b05698b74b.png" width="800"/>
</div>

|                              Model                               |    Dataset     |      Details       |              SWD              | MS-SSIM |                               Download                               |
| :--------------------------------------------------------------: | :------------: | :----------------: | :---------------------------: | :-----: | :------------------------------------------------------------------: |

| [WGAN-GP 128](./wgangp_GN_1xb64-160kiters_celeba-cropped-128x128.py) | CelebA-Cropped |         GN         | 5.87, 9.76, 9.43, 18.84/10.97 | 0.2601  | [model](https://download.openmmlab.com/mmediting/wgangp/wgangp_GN_celeba-cropped_128_b64x1_160k_20210408_170611-f8a99336.pth) |
| [WGAN-GP 128](./wgangp_GN-GP-50_1xb64-160kiters_lsun-bedroom-128x128.py) |  LSUN-Bedroom  | GN, GP-lambda = 50 | 11.7, 7.87, 9.82, 25.36/13.69 |  0.059  | [model](https://download.openmmlab.com/mmediting/wgangp/wgangp_GN_GP-50_lsun-bedroom_128_b64x1_130k_20210408_170509-56f2a37c.pth) |

```latex
@article{gulrajani2017improved,
  title={Improved Training of Wasserstein GANs},
  author={Gulrajani, Ishaan and Ahmed, Faruk and Arjovsky, Martin and Dumoulin, Vincent and Courville, Aaron},
  journal={arXiv preprint arXiv:1704.00028},
  url={https://arxiv.org/abs/1704.00028},
}
```

# 教程 5：使用评价指标

MMagic支持**17个指标**以评估模型质量。

有关用法，请参阅[MMagic中的训练与测试](../user_guides/train_test.md)。

在这里，我们将逐个介绍不同指标的详细信息。

本文的结构如下:

01. [MAE](#mae)
02. [MSE](#mse)
03. [PSNR](#psnr)
04. [SNR](#snr)
05. [SSIM](#ssim)
06. [NIQE](#niqe)
07. [SAD](#sad)
08. [MattingMSE](#mattingmse)
09. [GradientError](#gradienterror)
10. [ConnectivityError](#connectivityerror)
11. [FID and TransFID](#fid-and-transfid)
12. [IS and TransIS](#is-and-transis)
13. [Precision and Recall](#precision-and-recall)
14. [PPL](#ppl)
15. [SWD](#swd)
16. [MS-SSIM](#ms-ssim)
17. [Equivarience](#equivarience)

## MAE

MAE是图像的平均绝对误差。
要使用MAE进行评估，请在配置文件中添加以下配置:

```python
]
```

## MSE

MSE是图像的均方误差。
要使用MSE进行评估，请在配置文件中添加以下配置:

```python
]
```

## PSNR

PSNR是峰值信噪比。我们的实现方法来自https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio。
要使用PSNR进行评估，请在配置文件中添加以下配置:

```python
]
```

## SNR

]
```

## PSNR

```python
]
```

## SNR

SNR是信噪比。我们的实现方法来自 https://en.wikipedia.org/wiki/Signal-to-noise_ratio。
要使用SNR进行评估，请在配置文件中添加以下配置:

```python
]
```

## SSIM

SSIM是图像的结构相似度，在[图像质量评估:从错误可见性到结构相似度](https://live.ece.utexas.edu/publications/2004/zwang_ssim_ieeeip2004.pdf)中提出。我们实现的结果与https://ece.uwaterloo.ca/~z70wang/research/ssim/官方发布的MATLAB代码相同。
要使用SSIM进行评估，请在配置文件中添加以下配置:

```python
]
```

## NIQE

NIQE是自然图像质量评估指标，在[制作'完全盲'图像质量分析仪](http://www.live.ece.utexas.edu/publications/2013/mittal2013.pdf)中提出。我们的实现可以产生几乎与官方MATLAB代码相同的结果:http://live.ece.utexas.edu/research/quality/niqe_release.zip。
要使用NIQE进行评估，请在配置文件中添加以下配置:

```python
]
```

## SAD

SAD是图像抠图的绝对误差和。该指标计算每个像素的绝对差和所有像素的总和。
要使用SAD进行评估，请在配置文件中添加以下配置:

```python

```python
]
```

## SAD

```python
]
```

MattingMSE是图像抠图的均方误差。
要使用MattingMSE进行评估，请在配置文件中添加以下配置:

```python
]
```

GradientError是用于评估alpha matte预测的梯度误差。
要使用GradientError进行评估，请在配置文件中添加以下配置:

```python
]
```

ConnectivityError是用于评估alpha matte预测的连通性误差。
要使用ConnectivityError进行评估，请在配置文件中添加以下配置:

```python
]
```

## FID 和 TransFID

Fréchet初始距离是两个图像数据集之间相似度的度量。它被证明与人类对视觉质量的判断有很好的相关性，最常用于评估生成对抗网络样本的质量。FID是通过计算两个高斯函数之间的Fréchet距离来计算的，这些高斯函数适合于Inception网络的特征表示。

dict(type='ConnectivityError'),
]
```

在`MMagic`中，我们提供了两个版本的FID计算。一个是常用的PyTorch版本，另一个用于StyleGAN。同时，我们在StyleGAN2-FFHQ1024模型中比较了这两种实现之间的差异(详细信息可以在这里找到\[https://github.com/open-mmlab/mmagic/blob/main/configs/styleganv2/README.md\])。幸运的是，最终结果只是略有不同。因此，我们建议用户采用更方便的PyTorch版本。

**关于PyTorch版本和Tero版本:** 常用的PyTorch版本采用修改后的InceptionV3网络提取真假图像特征。然而，Tero的FID需要Tensorflow InceptionV3的[脚本模块](https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/pretrained/metrics/inception-2015-12-05.pt)。注意，应用此脚本模块需要' PyTorch >= 1.6.0 '。

**关于提取真实的初始数据:** 为了方便用户，在测试时自动提取真实的特征并保存在本地，存储的特征在下次测试时自动读取。具体来说，我们将根据用于计算实际特性的参数计算一个哈希值，并使用哈希值来标记特性文件，在测试时，如果' inception_pkl '没有设置，我们将在' MMAGIC_CACHE_DIR ' (~/.cache/openmmlab/mmagic/)中寻找该特性。如果未找到缓存的初始pkl，则将执行提取。

要使用FID指标，请在配置文件中添加以下配置:

```python
metrics = [
    dict(

```python
metrics = [
    dict(
]
```

如果您在一台新机器上工作，那么您可以复制'MMAGIC_CACHE_DIR'中的'pkl'文件，将它们复制到新机器并设置'inception_pkl'字段。

```python
metrics = [
    dict(
]
```

'TransFID'与'FID'的用法相同，但TransFID是为'Pix2Pix'和'CycleGAN'等翻译模型设计的，适用于我们的评估器。更多信息您可以参考[evaluation](../user_guides/train_test.md)。

## IS 和 TransIS

sample_model='ema')
]
```

Inception评分是评估生成图像质量的客观指标，在[改进的训练GANs技术](https://arxiv.org/pdf/1606.03498.pdf)中提出。它使用一个InceptionV3模型来预测生成的图像的类别，并假设: 1)如果图像质量高，它将被归类到特定的类别。2)如果图像具有较高的多样性，则图像的类别范围将很广。因此，条件概率和边际概率的kl -散度可以指示生成图像的质量和多样性。您可以在'metrics.py'中看到完整的实现，它指向https://github.com/sbarratt/inception-score-pytorch/blob/master/inception_score.py。
如果您想使用'IS'指标评估模型，请在配置文件中添加以下配置:

```python
metrics = [
    xxx,
    dict(
]
```

metrics = [
    xxx,
    dict(
]
```

需要注意的是，Inception V3的选择和图像大小的调整方法会显著影响最终的IS评分。因此，我们强烈建议用户可以下载[Tero's script model of Inception V3](https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/pretrained/metrics/inception-2015-12-05.pt)(加载此脚本模型需要torch >= 1.6)，并使用'Bicubic'插值与'Pillow'后端。

对应于config，您可以设置'resize_method'和'use_pillow_resize'用于图像大小的调整。您也可以将'inception_style'设置为'StyleGAN'用于推荐的tero的初始模型，或'PyTorch'用于torchvision的实现。对于没有互联网的环境，您可以下载初始的权重，并将'inception_path'设置为您的初始模型。

我们还调查了数据加载管线和预训练的Inception V3版本对IS结果的影响。所有IS都在同一组图像上进行评估，这些图像是从ImageNet数据集中随机选择的。

<details> <summary> 显示对比结果 </summary>

'TransIS'与'IS'的用法相同，但TransIS是为'Pix2Pix'和'CycleGAN'这样的翻译模型设计的，这是为我们的评估器改编的。更多信息可参考[evaluation](../user_guides/train_test.md)。

我们的'Precision and Recall'实现遵循StyleGAN2中使用的版本。在该度量中，采用VGG网络对图像进行特征提取。不幸的是，我们还没有发现PyTorch VGG实现与StyleGAN2中使用的Tero版本产生类似的结果。(关于差异，请参阅这个[文件](https://github.com/open-mmlab/mmagicing/blob/main/configs/styleganv2/README.md)。)因此，在我们的实现中，我们默认采用[Teor's VGG](https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/pretrained/metrics/vgg16.pt)网络。需要注意的是，应用这个脚本模块需要'PyTorch >= 1.6.0'。如果使用较低的PyTorch版本，我们将使用PyTorch官方VGG网络进行特征提取。
要使用' P&R '进行评估，请在配置文件中添加以下配置:

```python
metrics = [
]
```

## PPL

当在两个随机输入之间进行插值时，感知路径长度测量连续图像（其VGG16嵌入）之间的差异。剧烈的变化意味着多个特征一起发生了变化，它们可能会叠加在一起。通过实验表明，较小的PPL分数表明整体图像质量较高。
作为该指标的基础，我们使用基于感知的成对图像距离，该距离被计算为两个VGG16嵌入之间的加权差，其中权重被拟合，从而评价指标与人类的感知相似性判断一致。

]
```

## PPL

如果我们将潜在空间插值路径细分为线性段，我们可以将该分段路径的总感知长度定义为每个段上感知差异的总和，并且感知路径长度的自然定义将是无限细分下的总和的极限，但在实践中，我们使用一个小的细分`` $`\epsilon=10^｛-4｝`$ ``来近似它。
因此，潜在`space`Z中所有可能端点的平均感知路径长度为

以类似的方式计算潜在 `space` W中的平均感知路径长度：:

当 `` $`z_1, z_2 \sim P(z)`$ ``， 如果我们设置 `sampling` 为 ` full`，  则 `` $` t \sim U(0,1)`$ ``， 如果设置 `sampling` 为 `end`，则`` $` t \in \{0,1\}`$ ``。 `` $` G`$ `` 是生成器(i.e. `` $` g \circ f`$ `` 用于style-based网络)， `` $` d(.,.)`$ `` 用于计算结果图像之间的感知距离。我们通过取100,000个样本来计算期望(在代码中将' num_images '设置为50,000)。

您可以在'metrics.py'中找到完整的实现，参考https://github.com/rosinality/stylegan2-pytorch/blob/master/ppl.py。
如果您想使用'PPL'指标评估模型，请在配置文件中添加以下配置:

```python
metrics = [
    xxx,
]
```

## SWD

切片Wasserstein距离是概率分布的差异度量，距离越小表示生成的图像越真实。我们获得每个图像的拉普拉斯金字塔，并从拉普拉斯金字塔中提取小块作为描述符，然后可以通过获取真实和伪描述符切片的Wasserstein距离来计算SWD。
您可以在'metrics.py'中看到完整的实现，参考https://github.com/tkarras/progressive_growing_of_gans/blob/master/metrics/sliced_wasserstein.py。
如果您想使用'SWD'指标评估模型，请在配置文件中添加以下配置:

```python
metrics = [
    dict(
]
```

## MS-SSIM

metrics = [
    dict(
]
```

## MS-SSIM

采用多尺度结构相似度来衡量两幅图像的相似度。我们在这里使用MS-SSIM来衡量生成图像的多样性，MS-SSIM得分低表示生成图像的多样性高。您可以在'metrics.py'中看到完整的实现，参考https://github.com/tkarras/progressive_growing_of_gans/blob/master/metrics/ms_ssim.py。
如果您想使用'MS-SSIM'指标评估模型，请在配置文件中添加以下配置:

```python
metrics = [
    dict(
]
```

生成模型的等价性是指模型正变换和几何变换的互换性。目前这个指标只针对StyleGANv3计算，您可以在'metrics.py'中看到完整的实现，参考https://github.com/NVlabs/stylegan3/blob/main/metrics/equivariance.py。
如果您想使用'Equivarience'指标评估模型，请在配置文件中添加以下配置:

```python
metrics = [
    dict(

```python
metrics = [
    dict(
]
```

<details>

```bibtex
}
```

<details>

```bibtex
}
```

| [pose_hourglass_52](/configs/hand_2d_keypoint/topdown_heatmap/coco_wholebody_hand/td-hm_hourglass52_8xb32-210e_coco-wholebody-hand-256x256.py) |  256x256   |  0.804  | 0.835 | 4.54 | [ckpt](https://download.openmmlab.com/mmpose/hand/hourglass/hourglass52_coco_wholebody_hand_256x256-7b05c6db_20210909.pth) | [log](https://download.openmmlab.com/mmpose/hand/hourglass/hourglass52_coco_wholebody_hand_256x256_20210909.log.json) |

# Model Deployment

MMDet3D 1.1 fully relies on [MMDeploy](https://mmdeploy.readthedocs.io/) to deploy models.
Please stay tuned and this document will be update soon.

# Aggregated residual transformations for deep neural networks

<details>

```bibtex
}
```

We present a simple, highly modularized network architecture for image classification. Our network is constructed by repeating a building block that aggregates a set of transformations with the same topology. Our simple design results in a homogeneous, multi-branch architecture that has only a few hyper-parameters to set. This strategy exposes a new dimension, which we call "cardinality" (the size of the set of transformations), as an essential factor in addition to the dimensions of depth and width. On the ImageNet-1K dataset, we empirically show that even under the restricted condition of maintaining complexity, increasing cardinality is able to improve classification accuracy. Moreover, increasing cardinality is more effective than going deeper or wider when we increase the capacity. Our models, named ResNeXt, are the foundations of our entry to the ILSVRC 2016 classification task in which we secured 2nd place. We further investigate ResNeXt on an ImageNet-5K set and the COCO detection set, also showing better results than its ResNet counterpart. The code and models are publicly available online.

<img src="https://user-images.githubusercontent.com/15977946/146540039-cd3950dd-a6d2-4836-9af2-f7f990e98f77.png">
</div>

# Dataflow

In this chapter, we will introduce the dataflow and data format convention between the internal modules managed by the [Runner](https://mmengine.readthedocs.io/en/latest/tutorials/runner.html).

## Overview of dataflow

The [Runner](https://github.com/open-mmlab/mmengine/blob/main/docs/en/design/runner.md) is an "integrator" in MMEngine. It covers all aspects of the framework and shoulders the responsibility of organizing and scheduling nearly all modules, that means the dataflow between all modules also controlled by the `Runner`. As illustrated in the [Runner document of MMEngine](https://mmengine.readthedocs.io/en/latest/tutorials/runner.html), the following diagram shows the basic dataflow.

The dashed border, gray filled shapes represent different data formats, while solid boxes represent modules/methods. Due to the great flexibility and extensibility of MMEngine, some critical base classes can be inherited and their methods can be overridden. The diagram above only holds when users are not customizing `TrainLoop`, `ValLoop`, and `TestLoop` in `Runner`, and are not overriding `train_step`, `val_step` and `test_step` method in their custom model. The default setting of loops in MMSegmentation is as follows, it uses `IterBasedTrainLoop` to train models with 20000 iterations in total and do evaluation each 2000 iterations.

```python
```

```python
```

In the above diagram, the red line indicates the [train_step](./models.md#train_step). At each training iteration, dataloader loads images from storage and transfer to data preprocessor, data preprocessor would put images to the specific device and stack data to batch, then model accepts the batch data as inputs, finally the outputs of the model would be sent to optimizer. The blue line indicates [val_step](./models.md#val_step) and [test_step](./models.md#test_step). The dataflow of these two process is similar to the `train_step` except the outputs of model, since model parameters are freezed when doing evaluation, the model output would be transferred to [Evaluator](./evaluation.md#ioumetric) to compute metrics.

## Dataflow convention in MMSegmentation

From the diagram above, we could see the basic dataflow. In this section, we would introduce format convention of data involved in this dataflow, respectively.

### DataLoader to Data Preprocessor

DataLoader is an essential component in training and testing pipelines of MMEngine. Conceptually, it is derived from and consistent with [PyTorch](https://pytorch.org/). DataLoader loads data from filesystem and the original data passes through data preparation pipeline, then it would be sent to Data Preprocessor.

MMSegmentation defines the default data format at [PackSegInputs](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/datasets/transforms/formatting.py#L12), it's the last component of `train_pipeline` and `test_pipeline`. Please refer to [data transform documentation](./transforms.md) for more information about data transform `pipeline`.

Without any modifications, the return value of PackSegInputs is usually a `dict` and has only two keys, `inputs` and `data_samples`. The following pseudo-code shows the data types of the data loader output in mmseg, which is a batch of fetched data samples from the dataset, and data loader packs them into a dictionary of the list. `inputs` is the list of input tensors to the model and `data_samples` contains a list of input images' meta information and corresponding ground truth.

```python
dict(
)
```

```python
dict(
)
```

**Note:** [SegDataSample](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/structures/seg_data_sample.py) is a data structure interface of MMSegmentation, it is used as an interface between different components. `SegDataSample` implements the abstract data element `mmengine.structures.BaseDataElement`, please refer to [the SegDataSample documentation](./structures.md) and [data element documentation](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/data_element.html) in [MMEngine](https://github.com/open-mmlab/mmengine) for more information.

### Data Preprocessor to Model

Though drawn separately in the diagram [above](#overview-of-dataflow), data_preprocessor is a part of the model and thus can be found in [Model tutorial](./models.md) at data preprocessor chapter.

The return value of data preprocessor is a dictionary, containing `inputs` and `data_samples`, `inputs` is batched images, a 4D tensor, and some additional meta info used in data preprocesses would be added to the `data_samples`. When transferred to the network, the dictionary would be unpacked to two values. The following pseudo-codes show the return value of the data preprocessor and the input values of model.

```python
dict(
)
```

```python

        pass
```

**Note:** Model forward has 3 kinds of mode, which is controlled by input argumentmode, please refer [model tutorial](./models.md) for more details.

pass
```

### Model output

As [model tutorial](./models.md#forward) mentioned 3 kinds of mode forward with 3 kinds of output. `train_step`and `test_step`(or `val_step`) correspond to `'loss'` and `'predict'` respectively.

In `test_step` or `val_step`, the inference results would be transferred to `Evaluator`. You might read the [evaluation document](./evaluation.md) for more information about `Evaluator`.

After inference, the [BaseSegmentor](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/models/segmentors/base.py#L15) in MMSegmentation would do a simple post process to pack inference results, the segmentation logits produced by the neural network, segmentation mask after the `argmax` operation and ground truth(if exists) would be packed into a similar `SegDataSample` instance. The return value of [postprocess_result](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/models/segmentors/base.py#L132) is a **`List` of `SegDataSample`**. Following diagram shows the key properties of these `SegDataSample` instances.

The same as Data Preprocessor, loss function is also a part of the model, it's a property of [decode head](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/models/decode_heads/decode_head.py#L142).

In MMSegmentation, the method [loss_by_feat](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/models/decode_heads/decode_head.py#L291) of `decode_head` is an unified interface used to compute loss.

Parameters:

- seg_logits (Tensor): The output from decode head forward function.
- batch_data_samples (List\[:obj:`SegDataSample`\]): The seg data samples. It usually includes information such as `metainfo` and `gt_sem_seg`.

Returns:

- dict\[str, Tensor\]: a dictionary of loss components

**Note:** The `train_step` transfers the loss into OptimWrapper to update the weights in model, please refer [train_step](./models.md#train_step) for more details.

This is an example README for community `projects/`. We have provided detailed explanations for each field in the form of html comments, which are visible when you read the source of this README file. If you wish to submit your project to our main repository, then all the fields in this README are mandatory for others to understand what you have achieved in this implementation. For more details, read our [contribution guide](https://mmdetection.readthedocs.io/en/main/notes/contribution_guide.html) or approach us in [Discussions](https://github.com/open-mmlab/mmdetection/discussions).

## Usage

## Usage

```bash
python tools/train.py projects/example_project/configs/faster-rcnn_dummy-resnet_fpn_1x_coco.py
```

```bash
python -m torch.distributed.launch --nnodes=1 --node_rank=0 --nproc_per_node=${NUM_GPUS} --master_port=29506 --master_addr="127.0.0.1" tools/train.py projects/example_project/configs/faster-rcnn_dummy-resnet_fpn_1x_coco.py
```

```bash

```

```bash
python tools/test.py projects/example_project/configs/faster-rcnn_dummy-resnet_fpn_1x_coco.py ${CHECKPOINT_PATH}
```

## Results

<!-- List the results as usually done in other model's README. [Example](https://github.com/open-mmlab/mmdetection/blob/main/configs/faster_rcnn/README.md#results-and-models)

|                                Method                                 |  Backbone   | Pretrained Model |  Training set  |   Test set   | #epoch | box AP |         Download         |
| :-------------------------------------------------------------------: | :---------: | :--------------: | :------------: | :----------: | :----: | :----: | :----------------------: |

| [Faster R-CNN dummy](configs/faster-rcnn_dummy-resnet_fpn_1x_coco.py) | DummyResNet |        -         | COCO2017 Train | COCO2017 Val |   12   | 0.8853 | [model](<>) \| [log](<>) |

```latex
@article{Ren_2017,
   title={Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks},
   journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
   publisher={Institute of Electrical and Electronics Engineers (IEEE)},
   author={Ren, Shaoqing and He, Kaiming and Girshick, Ross and Sun, Jian},
   year={2017},
   month={Jun},
}
```

author={Ren, Shaoqing and He, Kaiming and Girshick, Ross and Sun, Jian},
}
```

# DeiT

> [Training data-efficient image transformers & distillation through attention](https://arxiv.org/abs/2012.12877)

Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. However, these visual transformers are pre-trained with hundreds of millions of images using an expensive infrastructure, thereby limiting their adoption.   In this work, we produce a competitive convolution-free transformer by training on Imagenet only. We train them on a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop evaluation) on ImageNet with no external data.   More importantly, we introduce a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention. We show the interest of this token-based distillation, especially when using a convnet as a teacher. This leads us to report results competitive with convnets for both Imagenet (where we obtain up to 85.2% accuracy) and when transferring to other tasks. We share our code and models.

<img src="https://user-images.githubusercontent.com/26739999/143225703-c287c29e-82c9-4c85-a366-dfae30d198cd.png" width="40%"/>
</div>

```python

predict = inference_model('deit-tiny_4xb256_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('deit-tiny_4xb256_in1k', pretrained=True)
```

Train:

```shell
python tools/train.py configs/deit/deit-tiny_4xb256_in1k.py
```

Test:

```shell

Train:

```shell
```

Test:

```shell
python tools/test.py configs/deit/deit-tiny_4xb256_in1k.py https://download.openmmlab.com/mmclassification/v0/deit/deit-tiny_pt-4xb256_in1k_20220218-13b382a0.pth
```

| Model                                             |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                       Config                       |                       Download                       |
| :------------------------------------------------ | :----------: | :--------: | :-------: | :-------: | :-------: | :------------------------------------------------: | :--------------------------------------------------: |

| `deit-tiny_4xb256_in1k`                           | From scratch |    5.72    |   1.26    |   74.50   |   92.24   |         [config](deit-tiny_4xb256_in1k.py)         | [model](https://download.openmmlab.com/mmclassification/v0/deit/deit-tiny_pt-4xb256_in1k_20220218-13b382a0.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/deit/deit-tiny_pt-4xb256_in1k_20220218-13b382a0.json) |
| `deit-tiny-distilled_3rdparty_in1k`\*             | From scratch |    5.91    |   1.27    |   74.51   |   91.90   |    [config](deit-tiny-distilled_4xb256_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/deit/deit-tiny-distilled_3rdparty_pt-4xb256_in1k_20211216-c429839a.pth) |

| `deit-small_4xb256_in1k`                          | From scratch |   22.05    |   4.61    |   80.69   |   95.06   |        [config](deit-small_4xb256_in1k.py)         | [model](https://download.openmmlab.com/mmclassification/v0/deit/deit-small_pt-4xb256_in1k_20220218-9425b9bb.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/deit/deit-small_pt-4xb256_in1k_20220218-9425b9bb.json) |
| `deit-small-distilled_3rdparty_in1k`\*            | From scratch |   22.44    |   4.63    |   81.17   |   95.40   |   [config](deit-small-distilled_4xb256_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/deit/deit-small-distilled_3rdparty_pt-4xb256_in1k_20211216-4de1d725.pth) |

| `deit-base_16xb64_in1k`                           | From scratch |   86.57    |   17.58   |   81.76   |   95.81   |         [config](deit-base_16xb64_in1k.py)         | [model](https://download.openmmlab.com/mmclassification/v0/deit/deit-base_pt-16xb64_in1k_20220216-db63c16c.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/deit/deit-base_pt-16xb64_in1k_20220216-db63c16c.json) |
| `deit-base_3rdparty_in1k`\*                       | From scratch |   86.57    |   17.58   |   81.79   |   95.59   |         [config](deit-base_16xb64_in1k.py)         | [model](https://download.openmmlab.com/mmclassification/v0/deit/deit-base_3rdparty_pt-16xb64_in1k_20211124-6f40c188.pth) |
| `deit-base-distilled_3rdparty_in1k`\*             | From scratch |   87.34    |   17.67   |   83.33   |   96.49   |    [config](deit-base-distilled_16xb64_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/deit/deit-base-distilled_3rdparty_pt-16xb64_in1k_20211216-42891296.pth) |

| `deit-base_224px-pre_3rdparty_in1k-384px`\*       |    224px     |   86.86    |   55.54   |   83.04   |   96.31   |      [config](deit-base_16xb32_in1k-384px.py)      | [model](https://download.openmmlab.com/mmclassification/v0/deit/deit-base_3rdparty_ft-16xb32_in1k-384px_20211124-822d02f2.pth) |
| `deit-base-distilled_224px-pre_3rdparty_in1k-384px`\* |    224px     |   87.63    |   55.65   |   85.55   |   97.35   | [config](deit-base-distilled_16xb32_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/deit/deit-base-distilled_3rdparty_ft-16xb32_in1k-384px_20211216-e48d6000.pth) |

*Models with * are converted from the [official repo](https://github.com/facebookresearch/deit/blob/f5123946205daf72a88783dae94cabff98c49c55/models.py#L168). The config files of these models are only for inference. We haven't reproduce the training results.*

MMPretrain doesn't support training the distilled version DeiT.

And we provide distilled version checkpoints for inference only.
```

```bibtex
@InProceedings{pmlr-v139-touvron21a,
  title =     {Training data-efficient image transformers &amp; distillation through attention},
  author =    {Touvron, Hugo and Cord, Matthieu and Douze, Matthijs and Massa, Francisco and Sablayrolles, Alexandre and Jegou, Herve},
  booktitle = {International Conference on Machine Learning},
  pages =     {10347--10357},
  year =      {2021},
  volume =    {139},
  month =     {July}
}
```

# 混合数据集训练

MMPose 提供了一个灵活、便捷的工具 `CombinedDataset` 来进行混合数据集训练。它作为一个封装器，可以包含多个子数据集，并将来自不同子数据集的数据转换成一个统一的格式，以用于模型训练。使用 `CombinedDataset` 的数据处理流程如下图所示。

![combined_dataset_pipeline](https://user-images.githubusercontent.com/26127467/223333154-fb88e511-810a-423c-b755-c791d296bc43.jpg)

本篇教程的后续部分将通过一个结合 COCO 和 AI Challenger (AIC) 数据集的例子详细介绍如何配置 `CombinedDataset`。

## COCO & AIC 数据集混合案例

COCO 和 AIC 都是 2D 人体姿态数据集。但是，这两个数据集在关键点的数量和排列顺序上有所不同。下面是分别来自这两个数据集的图片及关键点：

<img src="https://user-images.githubusercontent.com/26127467/223335806-748498af-8da4-4666-a6d3-337e4a8996f0.png" height="300px" alt><br>

有些关键点（例如“左手”）在两个数据集中都有定义，但它们具有不同的序号。具体来说，“左手”关键点在 COCO 数据集中的序号为 9，在AIC数据集中的序号为 5。此外，每个数据集都包含独特的关键点，另一个数据集中不存在。例如，面部关键点（序号为0〜4）仅在 COCO 数据集中定义，而“头顶”（序号为 12）和“颈部”（序号为 13）关键点仅在 AIC 数据集中存在。以下的维恩图显示了两个数据集中关键点之间的关系。

<img src="https://user-images.githubusercontent.com/26127467/223338755-d838dd39-901b-4e7d-af8b-b94b5f5f9ef3.png" height="200px" alt><br>

接下来，我们会介绍两种混合数据集的方式：

- [将 AIC 合入 COCO 数据集](#将-aic-合入-coco-数据集)
- [合并 AIC 和 COCO 数据集](#合并-aic-和-coco-数据集)

### 将 AIC 合入 COCO 数据集

如果用户想提高其模型在 COCO 或类似数据集上的性能，可以将 AIC 数据集作为辅助数据。此时应该仅选择 AIC 数据集中与 COCO 数据集共享的关键点，忽略其余关键点。此外，还需要将这些被选择的关键点在 AIC 数据集中的序号进行转换，以匹配在 COCO 数据集中对应关键点的序号。

<img src="https://user-images.githubusercontent.com/26127467/223348541-d1f9e3b7-7e60-41b5-bf68-22e61b34bb2b.png" height="200px" alt><br>

在这种情况下，来自 COCO 的数据不需要进行转换。此时 COCO 数据集可通过如下方式配置：

```python
dataset_coco = dict(
    type='CocoDataset',
    data_root='data/coco/',
    ann_file='annotations/person_keypoints_train2017.json',
    data_prefix=dict(img='train2017/'),
    pipeline=[], # `pipeline` 应为空列表，因为 COCO 数据不需要转换
)
```

对于 AIC 数据集，需要转换关键点的顺序。MMPose 提供了一个 `KeypointConverter` 转换器来实现这一点。以下是配置 AIC 子数据集的示例：

```python
dataset_aic = dict(
    type='AicDataset',

pipeline=[], # `pipeline` 应为空列表，因为 COCO 数据不需要转换
)
```

```python
    data_root='data/aic/',
    ann_file='annotations/aic_train.json',
    data_prefix=dict(img='ai_challenger_keypoint_train_20170902/'
                     'keypoint_train_images_20170902/'),
    pipeline=[
        dict(
            type='KeypointConverter',
            num_keypoints=17,  # 与 COCO 数据集关键点数一致
            mapping=[  # 需要列出所有带转换关键点的序号
                (0, 6),  # 0 (AIC 中的序号) -> 6 (COCO 中的序号)
                (1, 8),
                (2, 10),
                (3, 5),
                (4, 7),
                (5, 9),
                (6, 12),
                (7, 14),
                (8, 16),
                (9, 11),
                (10, 13),
                (11, 15),
            ])
    ],
)
```

(6, 12),
                (7, 14),
                (8, 16),
                (9, 11),
                (10, 13),
                (11, 15),
            ])
    ],
)
```

`KeypointConverter` 会将原序号在 0 到 11 之间的关键点的序号转换为在 5 到 16 之间的对应序号。同时，在 AIC 中序号为为 12 和 13 的关键点将被删除。另外，目标序号在 0 到 4 之间的关键点在 `mapping` 参数中没有定义，这些点将被设为不可见，并且不会在训练中使用。

子数据集都完成配置后, 混合数据集 `CombinedDataset` 可以通过如下方式配置:

```python
dataset = dict(
    type='CombinedDataset',
    # 混合数据集关键点顺序和 COCO 数据集相同，
    # 所以使用 COCO 数据集的描述信息
    metainfo=dict(from_file='configs/_base_/datasets/coco.py'),
    datasets=[dataset_coco, dataset_aic],
    # `train_pipeline` 包含了常用的数据预处理，
    # 比如图片读取、数据增广等
)
```

MMPose 提供了一份完整的 [配置文件](https://github.com/open-mmlab/mmpose/blob/dev-1.x/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-aic-256x192-merge.py) 来将 AIC 合入 COCO 数据集并用于训练网络。用户可以查阅这个文件以获取更多细节，或者参考这个文件来构建新的混合数据集。

### 合并 AIC 和 COCO 数据集

将 AIC 合入 COCO 数据集的过程中丢弃了部分 AIC 数据集中的标注信息。如果用户想要使用两个数据集中的所有信息，可以将两个数据集合并，即在两个数据集中取关键点的并集。

<img src="https://user-images.githubusercontent.com/26127467/223356617-075e0ab1-0ed3-426d-bc88-4f16be93f0ba.png" height="200px" alt><br>

在这种情况下，COCO 和 AIC 数据集都需要使用 `KeypointConverter` 来调整它们关键点的顺序：

```python
        dict(
            num_keypoints=19,  # 并集中有 19 个关键点
            mapping=[
                (0, 0),
                (1, 1),
                # 省略
                (16, 16),
            ])
    ])

ann_file='annotations/aic_train.json',
        dict(
            mapping=[
                (0, 6),
                # 省略
                (12, 17),
                (13, 18),
            ])
    ],
)
```

合并后的数据集有 19 个关键点，这与 COCO 或 AIC 数据集都不同，因此需要一个新的数据集描述信息文件。[coco_aic.py](https://github.com/open-mmlab/mmpose/blob/dev-1.x/configs/_base_/datasets/coco_aic.py) 是一个描述信息文件的示例，它基于 [coco.py](https://github.com/open-mmlab/mmpose/blob/dev-1.x/configs/_base_/datasets/coco.py) 并进行了以下几点修改:

- 添加了 AIC 数据集的文章信息；
- 在 `keypoint_info` 中添加了“头顶”和“颈部”这两个只在 AIC 中定义的关键点；
- 在 `skeleton_info` 中添加了“头顶”和“颈部”间的连线；
- 拓展 `joint_weights` 和 `sigmas` 以添加新增关键点的信息。

完成以上步骤后，合并数据集 `CombinedDataset` 可以通过以下方式配置：

```python
    # 使用新的描述信息文件

```python
    metainfo=dict(from_file='configs/_base_/datasets/coco_aic.py'),
)
```

此外，在使用混合数据集时，由于关键点数量的变化，模型的输出通道数也要做相应调整。如果用户用混合数据集训练了模型，但是要在 COCO 数据集上评估模型，就需要从模型输出的关键点中取出一个子集来匹配 COCO 中的关键点格式。可以通过 `test_cfg` 中的 `output_keypoint_indices` 参数自定义此子集。这个 [配置文件](https://github.com/open-mmlab/mmpose/blob/dev-1.x/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-aic-256x192-combine.py) 展示了如何用 AIC 和 COCO 合并后的数据集训练模型并在 COCO 数据集上进行测试。用户可以查阅这个文件以获取更多细节，或者参考这个文件来构建新的混合数据集。

# Pytorch

## Chat in command line

LMDeploy support chatting with PyTorch models with submodule `lmdeploy.pytorch.chat`.

This submodule allow user to chat with language model through command line, and optionally accelerate model using backends like deepspeed.

**Example 1**: Chat with default setting

```shell
python -m lmdeploy.pytorch.chat $PATH_TO_HF_MODEL
```

**Example 2**: Disable sampling and chat history

```shell
python -m lmdeploy.pytorch.chat \
    $PATH_TO_LLAMA_MODEL_IN_HF_FORMAT \
    --temperature 0 --max-histroy 0
```

**Example 3**: Accelerate with deepspeed inference

```shell
    --accel deepspeed
```

Note: to use deepspeed, you need to install deepspeed, and if hope to accelerate InternLM, you need a customized version <https://github.com/wangruohui/DeepSpeed/tree/support_internlm_0.10.0>

**Example 4**: Tensor parallel the model on 2 GPUs

```shell

```shell
    --accel deepspeed \
```

This module also allow the following control commands to change generation behaviors during chat.

- `exit`: terminate and exit chat
- `config set key=value`: change generation config `key` to `value`, e.g. config temperature=0 disable sampling for following chats
- `clear`: clear chat history

### Simple diagram of components

```mermaid
graph LR;
    subgraph model specific adapter
        p((user_input))-->tokenize-->id((input_ids))-->decorate
        tmpl_ids((template_ids))-->decorate;
    end
    subgraph generate
        model[CausalLM_model.generate]-->gen_result(("gen_result"))
        gen_result-->hid
        gen_result-->attn((attention))
    end
    subgraph streamer
        model-->s[streamer]--value-->decode_single--token-->output
    end
    subgraph session_manager

gen_result-->hid
    end
    end
        prepend_history-->fullid((complete_ids));
        trim-->prepend_history
    end
    decorate-->prepend_history
    hid((history_ids))-->trim;
    attn-->trim;
    fullid-->model
    tokenizer((tokenizer))-->decode_single
    tokenizer-->tokenize
    p-->genconfig(GenConfig)-->model
```

# How to add test units for backend ops

This tutorial introduces how to add unit test for backend ops. When you add a custom op under `backend_ops`, you need to add the corresponding test unit. Test units of ops are included in `tests/test_ops/test_ops.py`.

## Prerequisite

- `Compile new ops`: After adding a new custom op, needs to recompile the relevant backend, referring to [build.md](../01-how-to-build/build_from_source.md).

## 1. Add the test program test_XXXX()

You can put unit test for ops in `tests/test_ops/`. Usually, the following program template can be used for your custom op.

### example of ops unit test

```python
@pytest.mark.parametrize('backend', [TEST_TENSORRT, TEST_ONNXRT])        # 1.1 backend test class
@pytest.mark.parametrize('pool_h,pool_w,spatial_scale,sampling_ratio',   # 1.2 set parameters of op
                         [(2, 2, 1.0, 2), (4, 4, 2.0, 4)])               # [(# Examples of op test parameters),...]
def test_roi_align(backend,

[(2, 2, 1.0, 2), (4, 4, 2.0, 4)])               # [(# Examples of op test parameters),...]
                   pool_h,                                               # set parameters of op
                   pool_w,
                   spatial_scale,
                   sampling_ratio,
                   input_list=None,
                   save_dir=None):
    backend.check_env()

    if input_list is None:
        input = torch.rand(1, 1, 16, 16, dtype=torch.float32)            # 1.3 op input data initialization
        single_roi = torch.tensor([[0, 0, 0, 4, 4]], dtype=torch.float32)
    else:
        input = torch.tensor(input_list[0], dtype=torch.float32)
        single_roi = torch.tensor(input_list[1], dtype=torch.float32)

    from mmcv.ops import roi_align

    def wrapped_function(torch_input, torch_rois):                       # 1.4 initialize op model to be tested

from mmcv.ops import roi_align

        return roi_align(torch_input, torch_rois, (pool_w, pool_h),
                         spatial_scale, sampling_ratio, 'avg', True)

    wrapped_model = WrapFunction(wrapped_function).eval()

    with RewriterContext(cfg={}, backend=backend.backend_name, opset=11): # 1.5 call the backend test class interface
        backend.run_and_validate(
            wrapped_model, [input, single_roi],
            'roi_align',
            input_names=['input', 'rois'],
            output_names=['roi_feat'],
            save_dir=save_dir)
```

### 1.1 backend test class

We provide some functions and classes for difference backends, such as `TestOnnxRTExporter`, `TestTensorRTExporter`, `TestNCNNExporter`.

### 1.2 set parameters of op

```

Set some parameters of op, such as ’pool_h‘, ’pool_w‘, ’spatial_scale‘, ’sampling_ratio‘ in roi_align. You can set multiple parameters to test op.

### 1.3 op input data initialization

Initialization required input data.

### 1.4 initialize op model to be tested

The model containing custom op usually has two forms.

- `torch model`: Torch model with custom operators. Python code related to op is required, refer to `roi_align` unit test.
- `onnx model`: Onnx model with custom operators. Need to call onnx api to build, refer to `multi_level_roi_align` unit test.

### 1.5 call the backend test class interface

Call the backend test class `run_and_validate` to run and verify the result output by the op on the backend.

```python
    def run_and_validate(self,
                         model,

```python
                         model,
                         input_list,
                         model_name='tmp',
                         tolerate_small_mismatch=False,
                         do_constant_folding=True,
                         dynamic_axes=None,
                         output_names=None,
                         input_names=None,
                         expected_result=None,
                         save_dir=None):
```

#### Parameter Description

- `model`: Input model to be tested and it can be torch model or any other backend model.
- `input_list`: List of test data, which is mapped to the order of input_names.
- `model_name`: The name of the model.
- `tolerate_small_mismatch`: Whether to allow small errors in the verification of results.

- `do_constant_folding`: Whether to use constant light folding to optimize the model.
- `dynamic_axes`: If you need to use dynamic dimensions, enter the dimension information.
- `output_names`: The node name of the output node.
- `input_names`: The node name of the input node.
- `expected_result`: Expected ground truth values for verification.
- `save_dir`: The folder used to save the output files.

## 2. Test Methods

Use pytest to call the test function to test ops.

```bash
pytest tests/test_ops/test_ops.py::test_XXXX
```

# Adaptive Wing Loss for Robust Face Alignment via Heatmap Regression

<details>
<summary align="right"><a href="https://arxiv.org/pdf/1904.07399.pdf">AdaptiveWingloss (ICCV'2019)</a></summary>

```bibtex
@inproceedings{wang2019adaptive,
  title={Adaptive wing loss for robust face alignment via heatmap regression},
  author={Wang, Xinyao and Bo, Liefeng and Fuxin, Li},
  booktitle={Proceedings of the IEEE/CVF international conference on computer vision},
  pages={6971--6981},
}
```

Heatmap regression with a deep network has become one of the mainstream approaches to localize facial landmarks. However, the loss function for heatmap regression is rarely studied. In this paper, we analyze the ideal loss function properties for heatmap regression in face alignment problems. Then we propose a novel loss function, named Adaptive Wing loss, that is able to adapt its shape to different types of ground truth heatmap pixels. This adaptability penalizes loss more on foreground pixels while less on background pixels. To address the imbalance between foreground and background pixels, we also propose Weighted Loss Map, which assigns high weights on foreground and difficult background pixels to help training process focus more on pixels that are crucial to landmark localization. To further improve face alignment accuracy, we introduce boundary prediction and CoordConv with boundary coordinates. Extensive experiments on different benchmarks, including COFW, 300W and WFLW, show our approach outperforms the state-of-the-art by a significant margin on

various evaluation metrics. Besides, the Adaptive Wing loss also helps other heatmap regression tasks.

<img src="https://user-images.githubusercontent.com/15977946/148007960-a06a34d8-8090-49e1-80db-6bbe4a7e7e8d.png">
</div>

# ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation

<details>
<summary align="right"><a href="https://arxiv.org/abs/2204.12484">ViTPose (NeurIPS'2022)</a></summary>

```bibtex
  xu2022vitpose,
  title={ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation},
  author={Yufei Xu and Jing Zhang and Qiming Zhang and Dacheng Tao},
}
```

Although no specific domain knowledge is considered in the design, plain vision transformers have shown excellent performance in visual recognition tasks. However, little effort has been made to reveal the potential of such simple structures for pose estimation tasks. In this paper, we show the surprisingly good capabilities of plain vision transformers for pose estimation from various aspects, namely simplicity in model structure, scalability in model size, flexibility in training paradigm, and transferability of knowledge between models, through a simple baseline model called ViTPose. Specifically, ViTPose employs plain and non-hierarchical vision transformers as backbones to extract features for a given person instance and a lightweight decoder for pose estimation. It can be scaled up from 100M to 1B parameters by taking the advantages of the scalable model capacity and high parallelism of transformers, setting a new Pareto front between throughput and performance. Besides, ViTPose is very flexible regarding the attention type, input resolution, pre-training and finetuning strategy, as well as dealing with multiple pose tasks. We also empirically demonstrate that the knowledge of large ViTPose models can be easily transferred to small ones via a simple knowledge token. Experimental results show that our basic ViTPose model outperforms representative methods on the challenging MS COCO Keypoint Detection benchmark, while the largest model sets a new state-of-the-art.

<img src="https://user-images.githubusercontent.com/26127467/224964357-d3d000fc-768b-4087-96d6-9291c86a3e8a.png">
</div>

# ERFNet

> [ERFNet: Efficient Residual Factorized ConvNet for Real-time Semantic Segmentation](http://www.robesafe.uah.es/personal/eduardo.romera/pdfs/Romera17tits.pdf)

<a href="https://github.com/Eromera/erfnet_pytorch">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.20.0/mmseg/models/backbones/erfnet.py#L321">Code Snippet</a>

Semantic segmentation is a challenging task that addresses most of the perception needs of intelligent vehicles (IVs) in an unified way. Deep neural networks excel at this task, as they can be trained end-to-end to accurately classify multiple object categories in an image at pixel level. However, a good tradeoff between high quality and computational resources is yet not present in the state-of-the-art semantic segmentation approaches, limiting their application in real vehicles. In this paper, we propose a deep architecture that is able to run in real time while providing accurate semantic segmentation. The core of our architecture is a novel layer that uses residual connections and factorized convolutions in order to remain efficient while retaining remarkable accuracy. Our approach is able to run at over 83 FPS in a single Titan X, and 7 FPS in a Jetson TX1 (embedded device). A comprehensive set of experiments on the publicly available Cityscapes data set demonstrates that our system achieves an accuracy that is similar to the state of the art, while being orders of magnitude faster to compute than other architectures that achieve top precision. The resulting tradeoff makes our model an ideal approach for scene understanding in IV applications. The code is publicly available at: https://github.com/Eromera/erfnet.

<img src="https://user-images.githubusercontent.com/24582831/143479729-ea7951f6-1a3c-47d6-aaee-62c5759c0638.png" width="60%"/>
</div>

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device | mIoU | mIoU(ms+flip) | config                                                                                                                      | download                                                                                                                                                                                                                                                                                                                                                     |

| ERFNet | ERFNet   | 512x1024  |  160000 | 6.04     | 15.26          | V100   | 72.5 | 74.75         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/erfnet/erfnet_fcn_4xb4-160k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/erfnet/erfnet_fcn_4x4_512x1024_160k_cityscapes/erfnet_fcn_4x4_512x1024_160k_cityscapes_20220704_162145-dc90157a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/erfnet/erfnet_fcn_4x4_512x1024_160k_cityscapes/erfnet_fcn_4x4_512x1024_160k_cityscapes_20220704_162145.log.json) |

Note:

- The model is trained from scratch.

- Last deconvolution layer in the [original paper](https://github.com/Eromera/erfnet_pytorch/blob/master/train/erfnet.py#L123) is replaced by a naive `FCNHead` decoder head and a bilinear upsampling layer, found more effective and efficient.

- This model performance is sensitive to the seed values used, please refer to the log file for the specific settings of the seed. If you choose a different seed, the results might differ from the table results.

```bibtex
@article{romera2017erfnet,
  title={Erfnet: Efficient residual factorized convnet for real-time semantic segmentation},
  author={Romera, Eduardo and Alvarez, Jos{\'e} M and Bergasa, Luis M and Arroyo, Roberto},
  volume={19},
  number={1},
  pages={263--272},
}
```

# 数据预处理

## 在数据预处理前

我们推荐用户将数据集的路径软链接到 `$MMDETECTION3D/data`。如果你的文件夹结构和以下所展示的结构不一致，你可能需要改变配置文件中相应的数据路径。

```
├── tools
├── data
│   │   ├── maps
│   ├── kitti
│   ├── waymo
│   ├── lyft
│   │   ├── v1.01-train
│   │   │   ├── v1.01-train (训练数据)
│   │   │   ├── lidar (训练激光雷达)
│   │   │   ├── images (训练图片)
│   │   │   ├── maps (训练地图)
│   │   ├── v1.01-test
│   │   │   ├── v1.01-test (测试数据)
│   │   │   ├── lidar (测试激光雷达)
│   │   │   ├── images (测试图片)

│   │   │   ├── maps (测试地图)
│   │   ├── test.txt
│   │   ├── sample_submission.csv
│   ├── s3dis
│   │   ├── Stanford3dDataset_v1.2_Aligned_Version
│   │   ├── collect_indoor3d_data.py
│   │   ├── indoor3d_util.py
│   │   ├── scans_test
│   ├── sunrgbd
│   │   ├── OFFICIAL_SUNRGBD
│   │   ├── matlab
│   │   ├── sunrgbd_data.py
│   │   ├── sunrgbd_utils.py

```

## 数据下载和预处理

### KITTI

在[这里](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d)下载 KITTI 的 3D 检测数据。通过运行以下指令对 KITTI 数据进行预处理：

```bash

```

### KITTI

```bash

# 下载数据划分文件

```

然后通过运行以下指令生成信息文件：

```bash
python tools/create_data.py kitti --root-path ./data/kitti --out-dir ./data/kitti --extra-tag kitti
```

在使用 slurm 的环境下，用户需要使用下面的指令：

```bash
sh tools/create_data.sh <partition> kitti
```

**小贴士**：

- **现成的标注文件**：我们已经提供了离线处理好的 [KITTI 标注文件](#数据集标注文件列表)。您直接下载他们并放到 `data/kitti/` 目录下。然而，如果你想在点云检测方法中使用 `ObjectSample` 这一数据增强，你可以再额外使用以下命令来生成物体标注框数据库：

```bash
python tools/create_data.py kitti --root-path ./data/kitti --out-dir ./data/kitti --extra-tag kitti --only-gt-database
```

### Waymo

```bash
```

### Waymo

在[这里](https://waymo.com/open/download/)下载 Waymo 公开数据集 1.2 版本，在[这里](https://drive.google.com/drive/folders/18BVuF_RYJF0NjZpt8SnfzANiakoRMf0o?usp=sharing)下载其数据划分文件。然后，将 `.tfrecord` 文件置于 `data/waymo/waymo_format/` 目录下的相应位置，并将数据划分的 `.txt` 文件置于 `data/waymo/kitti_format/ImageSets` 目录下。在[这里](https://console.cloud.google.com/storage/browser/waymo_open_dataset_v_1_2_0/validation/ground_truth_objects)下载验证集的真实标签（`.bin` 文件）并将其置于 `data/waymo/waymo_format/`。提示：你可以使用 `gsutil` 来用命令下载大规模的数据集。更多细节请参考此[工具](https://github.com/RalphMao/Waymo-Dataset-Tool)。完成以上各步后，可以通过运行以下指令对 Waymo 数据进行预处理：

```bash
```

注意:

- 如果你的硬盘空间大小不足以存储转换后的数据，你可以将 `--out-dir` 参数设定为别的路径。你只需要记得在那个路径下创建文件夹并下载数据，然后在数据预处理完成后将其链接回 `data/waymo/kitti_format` 即可。

**小贴士**：

```

注意:

**小贴士**：

- **现成的标注文件**: 我们已经提供了离线处理好的 [Waymo 标注文件](#数据集标注文件列表)。您直接下载他们并放到 `data/waymo/kitti_format/` 目录下。然而，您还是需要自己使用上面的脚本将 Waymo 的原始数据还需要转成 kitti 格式。

- **Waymo-mini**： 如果你只是为了验证某些方法或者 debug, 你可以使用我们提供的 [Waymo-mini](https://download.openmmlab.com/mmdetection3d/data/waymo/waymo_mini_kitti_format.tar.gz)。它只包含原始数据集中训练集中的 2 个 segments 和 验证集中的 1 个 segment。您只需要下载并且解压到 `data/waymo/`，即可使用它：

  ```bash
  tar -xzvf waymo_mini_kitti_format.tar.gz -C ./data/waymo
  ```

- **更快的评估**： 如果你想在 Waymo 上进行更快的评估，你可以下载已经预处理好的[元信息文件](https://download.openmmlab.com/mmdetection3d/data/waymo/idx2metainfo.pkl)并将其放置在 `data/waymo/waymo_format/` 目录下。接着，你可以按照以下来修改数据集的配置：

  ```python
  val_evaluator = dict(
      type='WaymoMetric',
      ann_file='./data/waymo/kitti_format/waymo_infos_val.pkl',
      waymo_bin_file='./data/waymo/waymo_format/gt.bin',

```python
      data_root='./data/waymo/waymo_format',
      backend_args=backend_args,
      convert_kitti_format=True,
      idx2metainfo='data/waymo/waymo_format/idx2metainfo.pkl'
      )
  ```

  目前这种方式仅限于纯点云检测任务。

### NuScenes

在[这里](https://www.nuscenes.org/download)下载 nuScenes 数据集 1.0 版本的完整数据文件。通过运行以下指令对 nuScenes 数据进行预处理：

```bash
```

**小贴士**：

- **现成的标注文件**：我们已经提供了离线处理好的 [NuScenes 标注文件](#数据集标注文件列表)。您直接下载他们并放到 `data/nuscenes/` 目录下。然而，如果你想在点云检测方法中使用 `ObjectSample` 这一数据增强，你可以再额外使用以下命令来生成物体标注框数据库：

```bash
python tools/create_data.py nuscenes --root-path ./data/nuscenes --out-dir ./data/nuscenes --extra-tag nuscenes --only-gt-database
```

### Lyft

```bash
```

### Lyft

在[这里](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/data)下载 Lyft 3D 检测数据。通过运行以下指令对 Lyft 数据进行预处理：

```bash
python tools/create_data.py lyft --root-path ./data/lyft --out-dir ./data/lyft --extra-tag lyft --version v1.01
python tools/data_converter/lyft_data_fixer.py --version v1.01 --root-folder ./data/lyft
```

注意，为了文件结构的清晰性，我们遵从了 Lyft 数据原先的文件夹名称。请按照上面展示出的文件结构对原始文件夹进行重命名。同样值得注意的是，第二行命令的目的是为了修复一个损坏的激光雷达数据文件。更多细节请参考[该讨论](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/discussion/110000)。

在[这里](http://semantic-kitti.org/dataset.html#download)下载 SemanticKITTI 数据集并解压所有文件。通过运行以下指令对 SemanticKITTI 数据进行预处理：

```bash
python ./tools/create_data.py semantickitti --root-path ./data/semantickitti --out-dir ./data/semantickitti --extra-tag semantickitti
```

**小贴士**：

```bash
```

**小贴士**：

- **现成的标注文件**. 我们已经提供了离线处理好的 [SemanticKITTI 标注文件](#数据集标注文件列表)。您直接下载他们并放到 `data/semantickitti` 目录下。

### S3DIS、ScanNet 和 SUN RGB-D

请参考 S3DIS [README](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/data/s3dis/README.md) 文件以对其进行数据预处理。

请参考 ScanNet [README](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/data/scannet/README.md) 文件以对其进行数据预处理。

请参考 SUN RGB-D [README](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/data/sunrgbd/README.md) 文件以对其进行数据预处理。

**小贴士**：对于 S3DIS, ScanNet 和 SUN RGB-D 数据集，我们已经提供了离线处理好的 [标注文件](#数据集标注文件列表)。您可以直接下载他们并放到 `data/${DATASET}/` 目录下。然而，您还是需要自己利用我们的脚本来生成点云文件以及语义掩膜文件(如果该数据集有的话)。

### 自定义数据集

关于如何使用自定义数据集，请参考[自定义数据集](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/docs/zh_cn/advanced_guides/customize_dataset.md)。

### 更新数据信息

### 自定义数据集

### 更新数据信息

如果你之前已经使用 v1.0.0rc1-v1.0.0rc4 版的 mmdetection3d 创建数据信息，现在你想使用最新的 v1.1.0 版 mmdetection3d，你需要更新数据信息文件。

```bash
python tools/dataset_converters/update_infos_to_v2.py --dataset ${DATA_SET} --pkl-path ${PKL_PATH} --out-dir ${OUT_DIR}
```

- `--dataset`：数据集名。
- `--pkl-path`：指定数据信息 pkl 文件路径。
- `--out-dir`：输出数据信息 pkl 文件目录。

例如：

```bash
python tools/dataset_converters/update_infos_to_v2.py --dataset kitti --pkl-path ./data/kitti/kitti_infos_trainval.pkl --out-dir ./data/kitti
```

### 数据集标注文件列表

我们提供了离线生成好的数据集标注文件以供参考。为了方便，您也可以直接使用他们。

```

|                                                         数据集                                                         |                                                                                                              训练集标注文件                                                                                                               |                                                                                                           验证集标注文件                                                                                                           |                                                        测试集标注文件                                                        |

|                                                         KITTI                                                          |                                                                  [kitti_infos_train.pkl](https://download.openmmlab.com/mmdetection3d/data/kitti/kitti_infos_train.pkl)                                                                   |                                                                 [kitti_infos_val.pkl](https://download.openmmlab.com/mmdetection3d/data/kitti/kitti_infos_val.pkl)                                                                 |               [kitti_infos_test](https://download.openmmlab.com/mmdetection3d/data/kitti/kitti_infos_test.pkl)               |

|                                                        NuScenes                                                        | [nuscenes_infos_train.pkl](https://download.openmmlab.com/mmdetection3d/data/nuscenes/nuscenes_infos_train.pkl) [nuscenes_mini_infos_train.pkl](https://download.openmmlab.com/mmdetection3d/data/nuscenes/nuscenes_mini_infos_train.pkl) | [nuscenes_infos_val.pkl](https://download.openmmlab.com/mmdetection3d/data/nuscenes/nuscenes_infos_val.pkl)  [nuscenes_mini_infos_val.pkl](https://download.openmmlab.com/mmdetection3d/data/nuscenes/nuscenes_mini_infos_val.pkl) |                                                                                                                              |

|                                                         Waymo                                                          |         [waymo_infos_train.pkl](https://download.openmmlab.com/mmdetection3d/data/waymo/waymo_infos_train.pkl)  [waymo_mini_infos_train.pkl](https://download.openmmlab.com/mmdetection3d/data/waymo/waymo_mini_infos_train.pkl)          |          [waymo_infos_val.pkl](https://download.openmmlab.com/mmdetection3d/data/waymo/waymo_infos_val.pkl)  [waymo_mini_infos_val.pkl](https://download.openmmlab.com/mmdetection3d/data/waymo/waymo_mini_infos_val.pkl)          |             [waymo_infos_test.pkl](https://download.openmmlab.com/mmdetection3d/data/waymo/waymo_infos_test.pkl)             |

| [Waymo-mini kitti-format data](https://download.openmmlab.com/mmdetection3d/data/waymo/waymo_mini_kitti_format.tar.gz) |                                                                                                                                                                                                                                           |                                                                                                                                                                                                                                    |                                                                                                                              |

|                                                       SUN RGB-D                                                        |                                                               [sunrgbd_infos_train.pkl](https://download.openmmlab.com/mmdetection3d/data/sunrgbd/sunrgbd_infos_train.pkl)                                                                |                                                              [sunrgbd_infos_val.pkl](https://download.openmmlab.com/mmdetection3d/data/sunrgbd/sunrgbd_infos_val.pkl)                                                              |                                                                                                                              |

|                                                        ScanNet                                                         |                                                               [scannet_infos_train.pkl](https://download.openmmlab.com/mmdetection3d/data/scannet/scannet_infos_train.pkl)                                                                |                                                              [scannet_infos_val.pkl](https://download.openmmlab.com/mmdetection3d/data/scannet/scannet_infos_val.pkl)                                                              |          [scannet_infos_test.pkl](https://download.openmmlab.com/mmdetection3d/data/scannet/scannet_infos_test.pkl)          |

|                                                     SemanticKitti                                                      |                                                      [semantickitti_infos_train.pkl](https://download.openmmlab.com/mmdetection3d/data/semantickitti/semantickitti_infos_train.pkl)                                                       |                                                     [semantickitti_infos_val.pkl](https://download.openmmlab.com/mmdetection3d/data/semantickitti/semantickitti_infos_val.pkl)                                                     | [semantickitti_infos_test.pkl](https://download.openmmlab.com/mmdetection3d/data/semantickitti/semantickitti_infos_test.pkl) |

- [Single Image](#single-image)
- [Browse Dataset](#browse-dataset)
- [Visualizer Hook](#visualizer-hook)

## Single Image

`demo/image_demo.py` helps the user to visualize the prediction result of a single image, including the skeleton and heatmaps.

```shell
python demo/image_demo.py ${IMG} ${CONFIG} ${CHECKPOINT} [-h] [--out-file OUT_FILE] [--device DEVICE] [--draw-heatmap]
```

| ARGS                  | Description                      |
| --------------------- | -------------------------------- |
| `IMG`                 | The path to the test image.      |
| `CONFIG`              | The path to the config file.     |
| `CHECKPOINT`          | The path to the checkpoint file. |
| `--out-file OUT_FILE` | Path to output file.             |
| `--device DEVICE`     | Device used for inference.       |
| `--draw-heatmap`      | Visualize the predicted heatmap. |

Here is an example of Heatmap visualization:

![000000196141](https://user-images.githubusercontent.com/13503330/222373580-88d93603-e00e-45e9-abdd-f504a62b4ca5.jpg)

## Browse Dataset

```shell
```

```shell
```

```

```shell
```

## Visualizer Hook

During validation and testing, users can specify certain arguments to visualize the output of trained models.

To visualize in external window during testing:

```shell
python tools/test.py ${CONFIG} ${CHECKPOINT} --show
```

During validation:

```shell
python tools/train.py ${CONFIG} --work-dir ${WORK_DIR} --show --interval ${INTERVAL}
```

It is suggested to use large `INTERVAL` (e.g., 50) if users want to visualize during validation, since the wait time for each visualized instance will make the validation process very slow.

To save visualization results in `SHOW_DIR` during testing:

```shell
python tools/test.py ${CONFIG} ${CHECKPOINT} --show-dir=${SHOW_DIR}
```

```shell
python tools/train.py ${CONFIG} --work-dir ${WORK_DIR} --show-dir=${SHOW_DIR}
```

More details about visualization arguments can be found in [train_and_test](./train_and_test.md).

```shell
```

If you use a heatmap-based method and want to visualize predicted heatmaps, you can manually specify `output_heatmaps=True` for `model.test_cfg` in config file. Another way is to add `--cfg-options='model.test_cfg.output_heatmaps=True'` at the end of your command.

Visualization example (top: decoded keypoints; bottom: predicted heatmap):
![vis_pred](https://user-images.githubusercontent.com/26127467/187578902-30ef7bb0-9a93-4e03-bae0-02aeccf7f689.jpg)

For top-down models, each sample only contains one instance. So there will be multiple visualization results for each image.

# 教程 3：预训练权重推理（待更新）

我们针对特定任务提供了一些脚本，可以对单张图像进行推理。

#### 图像补全

您可以使用以下命令，输入一张测试图像以及缺损部位的遮罩图像，实现对测试图像的补全。

```shell
```

如果指定了 --imshow ，演示程序将使用 opencv 显示图像。例子：

```shell
    configs/global_local/gl_8xb12_celeba-256x256.py \
    tests/data/inpainting/celeba_test.png \
    tests/data/inpainting/bbox_mask.png \
    tests/data/inpainting/inpainting_celeba.png
```

补全结果将保存在 `tests/data/inpainting/inpainting_celeba.png` 中。

#### 抠图

您可以使用以下命令，输入一张测试图像以及对应的三元图（trimap），实现对测试图像的抠图。

```shell

```shell
```

```shell
    configs/dim/dim_stage3-v16-pln_1000k-1xb1_comp1k.py \
    tests/data/matting_dataset/pred/GT05.png
```

预测的 alpha 遮罩将保存在 `tests/data/matting_dataset/pred/GT05.png` 中。

#### 图像超分辨率

您可以使用以下命令来测试要恢复的图像。

```shell
```

如果指定了 `--imshow` ，演示程序将使用 opencv 显示图像。例子：

```shell

${CHECKPOINT_FILE} \
```

```shell
    configs/esrgan/esrgan_x4c64b23g32_400k-1xb16_div2k.py \
```

您可以通过提供 `--ref-path` 参数来测试基于参考的超分辨率算法。例子：

```shell
    configs/ttsr/ttsr-gan_x4c64b16_500k-1xb9_CUFED.py \
```

#### 人脸图像超分辨率

您可以使用以下命令来测试要恢复的人脸图像。

```shell

demo/demo_out.png \
```

```shell
```

```shell
    configs/glean/glean_in128out1024_300k-4xb2_ffhq-celeba-hq.py \
    tests/data/image/face/pred.png \
```

#### 视频超分辨率

您可以使用以下命令来测试视频以进行恢复。

```shell
```

```shell
```

它同时支持滑动窗口框架和循环框架。 例子：

EDVR:

```shell
    configs/edvr/edvrm_wotsa_reds_600k-8xb8.py \
```

BasicVSR:

```shell
    configs/basicvsr/basicvsr_2xb4_reds4.py \
```

复原的视频将保存在 ` demo/output/` 中。

#### 视频插帧

您可以使用以下命令来测试视频插帧。

```shell

```

#### 视频插帧

```shell
```

`${INPUT_DIR}` 和 `${OUTPUT_DIR}` 可以是视频文件路径或存放一系列有序图像的文件夹。
若 `${OUTPUT_DIR}` 是视频文件地址，其帧率可由输入视频帧率和 `fps_multiplier` 共同决定，也可由 `fps` 直接给定(其中前者优先级更高）。例子：

由输入视频帧率和 `fps_multiplier` 共同决定输出视频的帧率：

```shell
    configs/cain/cain_g1b32_1xb5_vimeo90k-triplet.py \
    tests/data/frames/test_inference.mp4 \
    tests/data/frames/test_inference_vfi_out.mp4 \
```

由 `fps` 直接给定输出视频的帧率：

```shell

--fps-multiplier 2.0
```

```shell
    --fps 60.0
```

#### 图像生成

```shell
python demo/generation_demo.py \
    [--unpaired-path ${UNPAIRED_IMAGE_FILE}] \
```

如果指定了 `--unpaired-path` （用于 CycleGAN），模型将执行未配对的图像到图像的转换。 如果指定了 `--imshow` ，演示也将使用opencv显示图像。 例子：

针对配对数据：

```shell
    configs/example_config.py \
    work_dirs/example_exp/example_model_20200202.pth \
    demo/demo.jpg \
    demo/demo_out.jpg
```

针对未配对数据（用 opencv 显示图像）：

```shell
python demo/generation_demo.py 、

configs/example_config.py \
```

```shell
    demo/demo_out.jpg \
    --unpaired-path demo/demo_unpaired.jpg \
    --imshow
```

# Real-ESRGAN (ICCVW'2021)

<details>
<summary align="right"><a href="https://arxiv.org/abs/2107.10833">Real-ESRGAN (ICCVW'2021)</a></summary>

```bibtex
@inproceedings{wang2021real,
  title={Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic data},
  author={Wang, Xintao and Xie, Liangbin and Dong, Chao and Shan, Ying},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)},
  pages={1905--1914},
}
```

<br/>

在 RGB 通道上进行评估，指标为 `PSNR/SSIM`。

|                                       算法                                        |      Set5      |         GPU 信息         |                                       下载                                        |

| :-------------------------------------------------------------------------------: | :------------: | :----------------------: | :-------------------------------------------------------------------------------: |
| [realesrnet_c64b23g32_12x4_lr2e-4_1000k_df2k_ost](./realesrnet_c64b23g32_4xb12-lr2e-4-1000k_df2k-ost.py) | 28.0297/0.8236 | 4 (Tesla V100-SXM2-32GB) | [模型](https://download.openmmlab.com/mmediting/restorers/real_esrgan/realesrnet_c64b23g32_12x4_lr2e-4_1000k_df2k_ost_20210816-4ae3b5a4.pth)/日志 |
| [realesrgan_c64b23g32_12x4_lr1e-4_400k_df2k_ost](./realesrgan_c64b23g32_4xb12-lr1e-4-400k_df2k-ost.py) | 26.2204/0.7655 | 4 (Tesla V100-SXM2-32GB) | [模型](https://download.openmmlab.com/mmediting/restorers/real_esrgan/realesrgan_c64b23g32_12x4_lr1e-4_400k_df2k_ost_20211010-34798885.pth) /[日志](https://download.openmmlab.com/mmediting/restorers/real_esrgan/realesrgan_c64b23g32_12x4_lr1e-4_400k_df2k_ost_20210922_142838.log.json) |

## 快速开始

**训练**

<details>

## 快速开始

**训练**

<details>

```shell
# CPU上训练
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/real_esrgan/realesrgan_c64b23g32_4xb12-lr1e-4-400k_df2k-ost.py

# 单个GPU上训练
python tools/train.py configs/real_esrgan/realesrgan_c64b23g32_4xb12-lr1e-4-400k_df2k-ost.py

# 多个GPU上训练
./tools/dist_train.sh configs/real_esrgan/realesrgan_c64b23g32_4xb12-lr1e-4-400k_df2k-ost.py 8
```

**测试**

<details>

```shell
# CPU上测试
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/real_esrgan/realesrgan_c64b23g32_4xb12-lr1e-4-400k_df2k-ost.py https://download.openmmlab.com/mmediting/restorers/real_esrgan/realesrgan_c64b23g32_12x4_lr1e-4_400k_df2k_ost_20211010-34798885.pth

# 单个GPU上测试

# 单个GPU上测试
python tools/test.py configs/real_esrgan/realesrgan_c64b23g32_4xb12-lr1e-4-400k_df2k-ost.py https://download.openmmlab.com/mmediting/restorers/real_esrgan/realesrgan_c64b23g32_12x4_lr1e-4_400k_df2k_ost_20211010-34798885.pth

# 多个GPU上测试
./tools/dist_test.sh configs/real_esrgan/realesrgan_c64b23g32_4xb12-lr1e-4-400k_df2k-ost.py https://download.openmmlab.com/mmediting/restorers/real_esrgan/realesrgan_c64b23g32_12x4_lr1e-4_400k_df2k_ost_20211010-34798885.pth 8
```

# Preparing HRSC Dataset

```bibtex
@conference{hrsc,
    author = {Zikun Liu. and Liu Yuan. and Lubin Weng. and Yiping Yang.},
    title = {A High Resolution Optical Satellite Image Dataset for Ship Recognition and Some New Baselines},
    booktitle = {Proceedings of the 6th International Conference on Pattern Recognition Applications and Methods - ICPRAM,},
    year = {2017},
    pages = {324-331},
    publisher = {SciTePress},
    organization = {INSTICC},
    doi = {10.5220/0006120603240331},
    isbn = {978-989-758-222-6},
    issn = {2184-4313},
}
```

## Download HRSC dataset

The HRSC dataset can be downloaded from [here](https://aistudio.baidu.com/aistudio/datasetdetail/54106).

```none
mmrotate
├── tools
├── data
│   ├── hrsc
│   │   ├── FullDataSet
│   │   │   ├─ AllImages
│   │   │   ├─ Annotations
│   │   │   ├─ LandMask
│   │   │   ├─ Segmentations
```

├── tools
├── data
│   ├── hrsc
```

Please change `data_root` in `configs/_base_/datasets/hrsc.py` to `data/hrsc/`.

# 如何设计自己的损失函数

`losses` 在 `MMagic` 中注册为 `LOSSES`。
在 MMagic 中设计自己的损失函数，步骤和在 MMagic 中自定义任何其他模型类似。
本节主要具体介绍了如何在 MMagic 中实现自定义的损失函数。
本教程建议您在实现自定义的损失函数时，应该遵循本教程相同的设计，这样在我们的框架中使用您新定义的损失函数，就不需要额外的工作。

本指南包括：

- [设计你自己的损失函数](#如何设计自己的损失函数)
  - [支持损失函数介绍](#支持的损失函数介绍)
  - [设计一个新的损失函数](#设计一个新的损失函数)
    - [MSELoss 的一个例子](#MSELoss-的一个例子)
    - [DiscShiftLoss 的一个例子](#DiscShiftLoss-的一个例子)
    - [GANWithCustomizedLoss 的一个例子](#GANWithCustomizedLoss-的一个例子)
  - [可用损失函数](#可用损失函数)
    - [常规损失函数](#常规损失函数)
    - [损失函数组件](#损失函数组件)

## 支持的损失函数介绍

为了方便使用，您可以直接使用我们为具体算法设置的默认损失计算过程，如lsgan、biggan、styleganv2等。
以`stylegan2`为例，我们使用R1梯度惩罚和生成器路径长度正则化作为可配置损失，用户可以调整相关参数，如 `r1_loss_weight` 和 `g_reg_weight`。

```python
# stylegan2_base.py
loss_config =dict(
     r1_loss_weight=10。 / 2. * d_reg_interval,
     r1_interval=d_reg_interval,
     norm_mode='HWC',
     g_reg_interval=g_reg_interval,
     g_reg_weight=2。 * g_reg_interval,
     pl_batch_shrink=2)

     type='StyleGAN2',
     xxx,

r1_interval=d_reg_interval,

     xxx,
     loss_config=loss_config)
```

## 设计一个新的损失函数

### MSELoss 的一个例子

一般来说，要实现一个损失模块，我们会编写一个函数实现，然后用类实现包装它。 以MSELoss为例：

```python
@masked_loss
def mse_loss(pred，target)：
     return F.mse_loss（pred，target，reduction='none'）

Class MSELoss(nn.Module)：

     def __init__(self, loss_weight=1.0, reduction='mean', sample_wise=False):
         # 代码可以在``mmagic/models/losses/pixelwise_loss.py``中找到

     def forward(self, pred, target, weight=None, **kwargs):
```

根据这个损失函数的定义，我们现在可以简单地通过在配置文件中定义它来使用：

```python
pixel_loss=dict(type='MSELoss', loss_weight=1.0, reduction='mean')
```

请注意，上面的`pixel_loss`必须在模型中定义。 详情请参考[自定义模型](./models.md)。 与自定义模型类似，为了使用您自己实现的损失函数，您需要在编写后在`mmagic/models/losses/__init__.py`中导入该损失函数。

```

### DiscShiftLoss 的一个例子

一般来说，要实现一个损失模块，我们会编写一个函数实现，然后用类实现包装它。
但是，在 MMagic 中，我们提供了另一个统一的接口 data_info 供用户定义输入参数和数据项之间的映射。

```python
def disc_shift_loss(pred)：
     return pred**2

@MODULES.register_module()
Class DiscShiftLoss(nn.Module)：

     def __init__(self, loss_weight=1.0, data_info=None):
         super(DiscShiftLoss，self).__init__()
         # 代码可以在``mmagic/models/losses/disc_auxiliary_loss.py``中找到

     def forward(self, *args, **kwargs):
```

这种损失模块设计的目标是允许在生成模型(`MODELS`)中自动使用它，而无需其他复杂代码来定义数据和关键字参数之间的映射。 因此，与 OpenMMLab 中的其他框架不同，我们的损失模块包含一个特殊的关键字 data_info，它是一个定义输入参数与生成模型数据之间映射的字典。 以`DiscShiftLoss`为例，用户在编写配置文件时，可能会用到这个loss，如下：

```python
dict(type='DiscShiftLoss',
    loss_weight=0.001 * 0.5,
    data_info=dict(pred='disc_pred_real'))
```

```python
```

`data_info` 中的信息告诉模块使用 `disc_pred_real` 数据作为 `pred` 参数的输入张量。 一旦 `data_info` 不为 `None`，我们的损失模块将自动构建计算图。

```python
class DiscShiftLoss(nn.Module):

    def __init__(self, loss_weight=1.0, data_info=None):
        super(DiscShiftLoss, self).__init__()
        self.data_info = data_info

    def forward(self, *args, **kwargs):
        # use data_info to build computational path
        if self.data_info is not None:
            # parse the args and kwargs
            if len(args) == 1:
                assert isinstance(args[0], dict), (
                    'You should offer a dictionary containing network outputs '
                    'for building up computational graph of this loss module.')
                outputs_dict = args[0]
            elif 'outputs_dict' in kwargs:

'for building up computational graph of this loss module.')
                assert len(args) == 0, (
                    'If the outputs dict is given in keyworded arguments, no'
                    ' further non-keyworded arguments should be offered.')
                outputs_dict = kwargs.pop('outputs_dict')
            else:
                raise NotImplementedError(
                    'Cannot parsing your arguments passed to this loss module.'
                    ' Please check the usage of this module')
            # link the outputs with loss input args according to self.data_info
            loss_input_dict = {
                k: outputs_dict[v]
                for k, v in self.data_info.items()
            }
            kwargs.update(loss_input_dict)
            kwargs.update(dict(weight=self.loss_weight))
            return disc_shift_loss(**kwargs)
        else:

}
        else:
            # if you have not define how to build computational graph, this
            # module will just directly return the loss as usual.
            return disc_shift_loss(*args, weight=self.loss_weight, **kwargs)

    def loss_name():
        return 'loss_disc_shift'

```

如这部分代码所示，一旦用户设置了“data_info”，损失模块将收到一个包含所有必要数据和模块的字典，该字典由训练过程中的“MODELS”提供。 如果此字典作为非关键字参数给出，则应将其作为第一个参数提供。 如果您使用关键字参数，请将其命名为 `outputs_dict`。

### GANWithCustomizedLoss 的一个例子

为了构建计算图，生成模型必须提供包含各种数据的字典。 仔细观察任何生成模型，你会发现我们将各种特征和模块收集到字典中。 我们在这里提供了一个自定义的`GANWithCustomizedLoss`来展示这个过程。

```python
class GANWithCustomizedLoss(BaseModel):

    def __init__(self, gan_loss, disc_auxiliary_loss, gen_auxiliary_loss,
                 *args, **kwargs):
        # ...
        if gan_loss is not None:

        # ...
            self.gan_loss = MODULES.build(gan_loss)
        else:
            self.gan_loss = None

        if disc_auxiliary_loss:
            self.disc_auxiliary_losses = MODULES.build(disc_auxiliary_loss)
            if not isinstance(self.disc_auxiliary_losses, nn.ModuleList):
                self.disc_auxiliary_losses = nn.ModuleList(
                    [self.disc_auxiliary_losses])
        else:
            self.disc_auxiliary_loss = None

        if gen_auxiliary_loss:
            self.gen_auxiliary_losses = MODULES.build(gen_auxiliary_loss)
            if not isinstance(self.gen_auxiliary_losses, nn.ModuleList):
                self.gen_auxiliary_losses = nn.ModuleList(
                    [self.gen_auxiliary_losses])
        else:
            self.gen_auxiliary_losses = None

self.gen_auxiliary_losses = nn.ModuleList(
        else:

    def train_step(self, data: dict,
                   optim_wrapper: OptimWrapperDict) -> Dict[str, Tensor]:
        # ...

        # get data dict to compute losses for disc
        data_dict_ = dict(
            iteration=curr_iter,
            gen=self.generator,
            disc=self.discriminator,
            disc_pred_fake=disc_pred_fake,
            disc_pred_real=disc_pred_real,
            fake_imgs=fake_imgs,
            real_imgs=real_imgs)

        loss_disc, log_vars_disc = self._get_disc_loss(data_dict_)

        # ...

    def _get_disc_loss(self, outputs_dict):
        # Construct losses dict. If you hope some items to be included in the
        # computational graph, you have to add 'loss' in its name. Otherwise,
        # items without 'loss' in their name will just be used to print

# computational graph, you have to add 'loss' in its name. Otherwise,
        # information.
        losses_dict = {}
        # gan loss
        losses_dict['loss_disc_fake'] = self.gan_loss(
            outputs_dict['disc_pred_fake'], target_is_real=False, is_disc=True)
        losses_dict['loss_disc_real'] = self.gan_loss(
            outputs_dict['disc_pred_real'], target_is_real=True, is_disc=True)

        # disc auxiliary loss
        if self.with_disc_auxiliary_loss:
            for loss_module in self.disc_auxiliary_losses:
                loss_ = loss_module(outputs_dict)
                if loss_ is None:
                    continue

                # the `loss_name()` function return name as 'loss_xxx'
                if loss_module.loss_name() in losses_dict:
                    losses_dict[loss_module.loss_name(
                    )] = losses_dict[loss_module.loss_name()] + loss_

if loss_module.loss_name() in losses_dict:
                else:
                    losses_dict[loss_module.loss_name()] = loss_
        loss, log_var = self.parse_losses(losses_dict)

        return loss, log_var

```

在这里，`_get_disc_loss` 将帮助自动组合各种损失函数。

因此，只要用户设计相同规则的损失模块，就可以在生成模型的训练中插入任何一种损失，无需对模型代码进行其他修改。 您只需要在配置文件中定义 `data_info` 即可。

## 可用损失函数

我们在配置中列出了可用的损失示例，如下所示。

### 常规损失函数

<thead>
  <tr>
    <th>Method</th>
    <th>class</th>
    <th>Example</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>vanilla gan loss</td>
    <td>mmagic.models.GANLoss</td>
<td>

```python
# dic gan
loss_gan=dict(
    type='GANLoss',
    gan_type='vanilla',
    loss_weight=0.001,
)

```

</td>

</tr>
  <tr>
    <td>lsgan loss</td>
<td>
</td>

</tr>
  <tr>
    <td>wgan loss</td>

gan_type='vanilla',
)

```

</td>

</tr>
  <tr>
<td>
</td>

</tr>
  <tr>
    <td>

```python
# deepfillv1
    gan_type='wgan',
    loss_weight=0.0001,
)
```

</td>

</tr>
  <tr>
    <td>hinge loss</td>
    <td>

```python
# deepfillv2
    gan_type='hinge',
    loss_weight=0.1,
)
```

</td>

</tr>
  <tr>
    <td>smgan loss</td>
<td>

```python
# aot-gan
    gan_type='smgan',
    loss_weight=0.01,
)
```

</td>

</tr>
  <tr>
    <td>gradient penalty</td>
    <td>mmagic.models.GradientPenaltyLoss</td>
    <td>

```python
loss_gp=dict(type='GradientPenaltyLoss', loss_weight=10.)
```

</td>

</tr>
  <tr>
    <td>discriminator shift loss</td>

<td>mmagic.models.GradientPenaltyLoss</td>
    <td>

```python
```

</td>

</tr>
  <tr>
    <td>mmagic.models.DiscShiftLoss</td>
    <td>

```python
loss_disc_shift=dict(type='DiscShiftLoss', loss_weight=0.001)

```

</td>

</tr>
  <tr>
    <td>clip loss</td>
    <td>mmagic.models.CLIPLoss</td>
    <td></td>

</tr>
  <tr>
    <td>L1 composition loss</td>
    <td>mmagic.models.L1CompositionLoss</td>
    <td></td>

</tr>
  <tr>
    <td>MSE composition loss</td>
    <td>mmagic.models.MSECompositionLoss</td>
    <td></td>

</tr>
  <tr>
    <td>charbonnier composition loss</td>
    <td>mmagic.models.CharbonnierCompLoss</td>
    <td>

```python
# dim
loss_comp=dict(type='CharbonnierCompLoss', loss_weight=0.5)
```

</td>

</tr>
  <tr>
    <td>face id Loss</td>
    <td>mmagic.models.FaceIdLoss</td>
    <td></td>

</tr>
  <tr>
    <td>light cnn feature loss</td>

```

</td>

</tr>
  <tr>
    <td></td>

</tr>
  <tr>
    <td>mmagic.models.LightCNNFeatureLoss</td>
    <td>

```python
# dic gan
feature_loss=dict(
    type='LightCNNFeatureLoss',
    pretrained=pretrained_light_cnn,
    criterion='l1')
```

</td>

</tr>
  <tr>
    <td>gradient loss</td>
    <td>mmagic.models.GradientLoss</td>
    <td></td>

</tr>
  <tr>
    <td>l1 Loss</td>
    <td>mmagic.models.L1Loss</td>
    <td>

```python
# dic gan
pixel_loss=dict(type='L1Loss', loss_weight=1.0, reduction='mean')
```

</td>

</tr>
  <tr>
    <td>mse loss</td>
    <td>mmagic.models.MSELoss</td>
    <td>

```python
# dic gan
align_loss=dict(type='MSELoss', loss_weight=0.1, reduction='mean')
```

</td>

</tr>
  <tr>
    <td>charbonnier loss</td>
    <td>mmagic.models.CharbonnierLoss</td>
    <td>

```python
# dim
loss_alpha=dict(type='CharbonnierLoss', loss_weight=0.5)
```

</td>

```

</td>

</tr>
  <tr>
    <td>

```python
# dim
```

</td>

</tr>
  <tr>
    <td>masked total variation loss</td>
    <td>mmagic.models.MaskedTVLoss</td>
    <td>

```python
# partial conv
loss_tv=dict(
    type='MaskedTVLoss',
    loss_weight=0.1
)

```

</td>

</tr>
  <tr>
    <td>perceptual loss</td>
    <td>mmagic.models.PerceptualLoss</td>
    <td>

```python
# real_basicvsr
perceptual_loss=dict(
    type='PerceptualLoss',
    layer_weights={
        '2': 0.1,
        '7': 0.1,
        '16': 1.0,
        '25': 1.0,
        '34': 1.0,
    },
    vgg_type='vgg19',
    perceptual_weight=1.0,
    style_weight=0,
    norm_img=False)

```

</td>

</tr>
  <tr>
    <td>transferal perceptual loss</td>
    <td>mmagic.models.TransferalPerceptualLoss</td>
    <td>

```python
# ttsr
transferal_perceptual_loss=dict(
    type='TransferalPerceptualLoss',

<tr>
    <td>

```python
# ttsr
    loss_weight=1e-2,
    use_attention=False,
    criterion='mse')

```

</td>
  </tr>
</tbody>
</table>

### 损失函数组件

对于“GANWithCustomizedLoss”，我们提供了几个组件来构建自定义损失。

| Method                               | class                                       |
| ------------------------------------ | ------------------------------------------- |
| clip loss component                  | mmagic.models.CLIPLossComps                 |
| discriminator shift loss component   | mmagic.models. DiscShiftLossComps           |
| gradient penalty loss component      | mmagic.models. GradientPenaltyLossComps     |
| r1 gradient penalty component        | mmagic.models. R1GradientPenaltyComps       |
| face Id loss component               | mmagic.models. FaceIdLossComps              |

| gan loss component                   | mmagic.models. GANLossComps                 |
| generator path regularizer component | mmagic.models.GeneratorPathRegularizerComps |

<details>

```bibtex
}
```

<details>

```bibtex

| [deeppose_resnet_50](/configs/body_2d_keypoint/topdown_regression/coco/td-reg_res50_8xb64-210e_coco-256x192.py) |  256x192   | 0.541 |      0.824      |      0.601      | 0.649 |      0.893      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_regression/coco/td-reg_res50_8xb64-210e_coco-256x192-72ef04f3_20220913.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_regression/coco/td-reg_res50_8xb64-210e_coco-256x192-72ef04f3_20220913.log.json) |
| [deeppose_resnet_101](/configs/body_2d_keypoint/topdown_regression/coco/td-reg_res101_8xb64-210e_coco-256x192.py) |  256x192   | 0.562 |      0.831      |      0.629      | 0.670 |      0.900      | [ckpt](https://download.openmmlab.com/mmpose/top_down/deeppose/deeppose_res101_coco_256x192-2f247111_20210205.pth) | [log](https://download.openmmlab.com/mmpose/top_down/deeppose/deeppose_res101_coco_256x192_20210205.log.json) |

| [deeppose_resnet_152](/configs/body_2d_keypoint/topdown_regression/coco/td-reg_res152_8xb64-210e_coco-256x192.py) |  256x192   | 0.584 |      0.842      |      0.659      | 0.688 |      0.907      | [ckpt](https://download.openmmlab.com/mmpose/top_down/deeppose/deeppose_res152_coco_256x192-7df89a88_20210205.pth) | [log](https://download.openmmlab.com/mmpose/top_down/deeppose/deeppose_res152_coco_256x192_20210205.log.json) |

<details>
<summary align="right"><a href="https://arxiv.org/abs/2004.08955">ResNeSt (ArXiv'2020)</a></summary>

```bibtex
  title={ResNeSt: Split-Attention Networks},
  author={Zhang, Hang and Wu, Chongruo and Zhang, Zhongyue and Zhu, Yi and Zhang, Zhi and Lin, Haibin and Sun, Yue and He, Tong and Muller, Jonas and Manmatha, R. and Li, Mu and Smola, Alexander},
  journal={arXiv preprint arXiv:2004.08955},
}
```

<details>

```bibtex

| [pose_resnest_50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnest50_8xb64-210e_coco-256x192.py) |  256x192   | 0.720 |      0.899      |      0.800      | 0.775 |      0.939      | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnest/resnest50_coco_256x192-6e65eece_20210320.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnest/resnest50_coco_256x192_20210320.log.json) |
| [pose_resnest_50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnest50_8xb64-210e_coco-384x288.py) |  384x288   | 0.737 |      0.900      |      0.811      | 0.789 |      0.937      | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnest/resnest50_coco_384x288-dcd20436_20210320.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnest/resnest50_coco_384x288_20210320.log.json) |

| [pose_resnest_101](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnest101_8xb64-210e_coco-256x192.py) |  256x192   | 0.725 |      0.900      |      0.807      | 0.781 |      0.939      | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnest/resnest101_coco_256x192-2ffcdc9d_20210320.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnest/resnest101_coco_256x192_20210320.log.json) |
| [pose_resnest_101](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnest101_8xb32-210e_coco-384x288.py) |  384x288   | 0.745 |      0.905      |      0.818      | 0.798 |      0.942      | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnest/resnest101_coco_384x288-80660658_20210320.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnest/resnest101_coco_384x288_20210320.log.json) |

| [pose_resnest_200](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnest200_8xb64-210e_coco-256x192.py) |  256x192   | 0.731 |      0.905      |      0.812      | 0.787 |      0.943      | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnest/resnest200_coco_256x192-db007a48_20210517.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnest/resnest200_coco_256x192_20210517.log.json) |
| [pose_resnest_200](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnest200_8xb16-210e_coco-384x288.py) |  384x288   | 0.753 |      0.907      |      0.827      | 0.805 |      0.943      | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnest/resnest200_coco_384x288-b5bb76cb_20210517.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnest/resnest200_coco_384x288_20210517.log.json) |

| [pose_resnest_269](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnest269_8xb32-210e_coco-256x192.py) |  256x192   | 0.737 |      0.907      |      0.819      | 0.792 |      0.943      | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnest/resnest269_coco_256x192-2a7882ac_20210517.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnest/resnest269_coco_256x192_20210517.log.json) |
| [pose_resnest_269](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnest269_8xb16-210e_coco-384x288.py) |  384x288   | 0.754 |      0.908      |      0.828      | 0.805 |      0.943      | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnest/resnest269_coco_384x288-b142b9fb_20210517.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnest/resnest269_coco_384x288_20210517.log.json) |

# 开始你的第一步

## 依赖

在本节中，我们将展示如何使用 PyTorch 准备环境。

MMDetection3D 支持在 Linux，Windows（实验性支持），MacOS 上运行，它需要 Python 3.7 以上，CUDA 9.2 以上和 PyTorch 1.6 以上。

```{note}
如果您对 PyTorch 有经验并且已经安装了它，您可以直接跳转到[下一小节](#安装流程)。否则，您可以按照下述步骤进行准备。
```

**步骤 0.** 从[官方网站](https://docs.conda.io/en/latest/miniconda.html)下载并安装 Miniconda。

**步骤 1.** 创建并激活一个 conda 环境。

```shell
```

**步骤 2.** 基于 [PyTorch 官方说明](https://pytorch.org/get-started/locally/)安装 PyTorch，例如：

在 GPU 平台上：

```shell
```

在 CPU 平台上：

```shell
```

## 安装流程

我们推荐用户参照我们的最佳实践安装 MMDetection3D。不过，整个过程也是可定制化的，更多信息请参考[自定义安装](#自定义安装)章节。

### 最佳实践

**步骤 0.** 使用 [MIM](https://github.com/open-mmlab/mim) 安装 [MMEngine](https://github.com/open-mmlab/mmengine)，[MMCV](https://github.com/open-mmlab/mmcv) 和 [MMDetection](https://github.com/open-mmlab/mmdetection)。

```shell

```shell
mim install 'mmcv>=2.0.0rc4'
mim install 'mmdet>=3.0.0'
```

**注意**：在 MMCV-v2.x 中，`mmcv-full` 改名为 `mmcv`，如果您想安装不包含 CUDA 算子的 `mmcv`，您可以使用 `mim install "mmcv-lite>=2.0.0rc4"` 安装精简版。

**步骤 1.** 安装 MMDetection3D。

方案 a：如果您开发并直接运行 mmdet3d，从源码安装它：

```shell
git clone https://github.com/open-mmlab/mmdetection3d.git -b dev-1.x
# "-b dev-1.x" 表示切换到 `dev-1.x` 分支。
# "-v" 指详细说明，或更多的输出
# "-e" 表示在可编辑模式下安装项目，因此对代码所做的任何本地修改都会生效，从而无需重新安装。
```

方案 b：如果您将 mmdet3d 作为依赖或第三方 Python 包使用，使用 MIM 安装：

```shell
mim install "mmdet3d>=1.1.0rc0"
```

注意：

1. 如果您希望使用 `opencv-python-headless` 而不是 `opencv-python`，您可以在安装 MMCV 之前安装它。

2. 一些安装依赖是可选的。简单地运行 `pip install -v -e .` 将会安装最低运行要求的版本。如果想要使用一些可选依赖项，例如 `albumentations` 和 `imagecorruptions`，可以使用 `pip install -r requirements/optional.txt` 进行手动安装，或者在使用 `pip` 时指定所需的附加功能（例如 `pip install -v -e .[optional]`），支持附加功能的有效键值包括 `all`、`tests`、`build` 以及 `optional`。

我们已经支持 `spconv 2.0`。如果用户已经安装 `spconv 2.0`，代码会默认使用 `spconv 2.0`，它会比原生 `mmcv spconv` 使用更少的 GPU 内存。用户可以使用下列的命令来安装 `spconv 2.0`：

   ```shell
   pip install cumm-cuxxx
   pip install spconv-cuxxx
   ```

   `xxx` 表示环境中的 CUDA 版本。

   例如，使用 CUDA 10.2，对应命令是 `pip install cumm-cu102 && pip install spconv-cu102`。

   支持的 CUDA 版本包括 10.2，11.1，11.3 和 11.4。用户也可以通过源码编译来安装。更多细节请参考[spconv v2.x](https://github.com/traveller59/spconv)。

   我们也支持 `Minkowski Engine` 作为稀疏卷积的后端。如果需要，请参考[安装指南](https://github.com/NVIDIA/MinkowskiEngine#installation) 或者使用 `pip` 来安装：

   ```shell
   conda install openblas-devel -c anaconda
   export CPLUS_INCLUDE_PATH=CPLUS_INCLUDE_PATH:${YOUR_CONDA_ENVS_DIR}/include
   # replace ${YOUR_CONDA_ENVS_DIR} to your anaconda environment path e.g. `/home/username/anaconda3/envs/openmmlab`.
   pip install -U git+https://github.com/NVIDIA/MinkowskiEngine -v --no-deps --install-option="--blas_include_dirs=/opt/conda/include" --install-option="--blas=openblas"
   ```

pip install -U git+https://github.com/NVIDIA/MinkowskiEngine -v --no-deps --install-option="--blas_include_dirs=/opt/conda/include" --install-option="--blas=openblas"
   ```

   我们还支持 `Torchsparse` 作为稀疏卷积的后端。如果需要，请参考[安装指南](https://github.com/mit-han-lab/torchsparse#installation) 或者使用 `pip` 来安装：

   ```shell
   sudo apt install libsparsehash-dev
   pip install --upgrade git+https://github.com/mit-han-lab/torchsparse.git@v1.4.0
   ```

   或者通过以下安装绕过sudo权限

   ```shell
   conda install -c bioconda sparsehash
    # replace ${YOUR_CONDA_ENVS_DIR} to your anaconda environment path e.g. `/home/username/anaconda3/envs/openmmlab`.
   ```

3. 我们的代码目前不能在只有 CPU 的环境（CUDA 不可用）下编译。

### 验证安装

为了验证 MMDetection3D 是否安装正确，我们提供了一些示例代码来执行模型推理。

**步骤 1.** 我们需要下载配置文件和模型权重文件。

```shell

```

### 验证安装

```shell
mim download mmdet3d --config pointpillars_hv_secfpn_8xb6-160e_kitti-3d-car --dest .
```

下载将需要几秒钟或更长时间，这取决于您的网络环境。完成后，您会在当前文件夹中发现两个文件 `pointpillars_hv_secfpn_8xb6-160e_kitti-3d-car.py` 和 `hv_pointpillars_secfpn_6x8_160e_kitti-3d-car_20220331_134606-d42d15ed.pth`。

**步骤 2.** 推理验证。

方案 a：如果您从源码安装 MMDetection3D，那么直接运行以下命令进行验证：

```shell
python demo/pcd_demo.py demo/data/kitti/000008.bin pointpillars_hv_secfpn_8xb6-160e_kitti-3d-car.py hv_pointpillars_secfpn_6x8_160e_kitti-3d-car_20220331_134606-d42d15ed.pth --show
```

您会看到一个带有点云的可视化界面，其中包含有在汽车上绘制的检测框。

**注意**：

如果您想输入一个 `.ply` 文件，您可以使用如下函数将它转换成 `.bin` 格式。然后您可以使用转化的 `.bin` 文件来运行样例。请注意在使用此脚本之前，您需要安装 `pandas` 和 `plyfile`。这个函数也可以用于训练 `ply 数据`时作为数据预处理来使用。

```python
from plyfile import PlyData

def convert_ply(input_path, output_path):

```python

    plydata = PlyData.read(input_path)  # 读取文件
    data = plydata.elements[0].data  # 读取数据
    data_pd = pd.DataFrame(data)  # 转换成 DataFrame
    data_np = np.zeros(data_pd.shape, dtype=np.float)  # 初始化数组来存储数据
    property_names = data[0].dtype.names  # 读取属性名称
    for i, name in enumerate(
            property_names):  # 通过属性读取数据
        data_np[:, i] = data_pd[name]
    data_np.astype(np.float32).tofile(output_path)
```

例如：

```python
convert_ply('./test.ply', './test.bin')
```

如果您有其他格式的点云数据（`.off`，`.obj` 等），您可以使用 `trimesh` 将它们转化成 `.ply`。

```python
import trimesh

def to_ply(input_path, output_path, original_type):
    mesh = trimesh.load(input_path, file_type=original_type)  # 读取文件
    mesh.export(output_path, file_type='ply')  # 转换成 ply
```

例如：

```python
to_ply('./test.obj', './test.ply', 'obj')
```

mesh = trimesh.load(input_path, file_type=original_type)  # 读取文件
```

例如：

```python
```

方案 b：如果您使用 MIM 安装 MMDetection3D，那么可以打开您的 Python 解析器，复制并粘贴以下代码：

```python
from mmdet3d.apis import init_model, inference_detector

config_file = 'pointpillars_hv_secfpn_8xb6-160e_kitti-3d-car.py'
checkpoint_file = 'hv_pointpillars_secfpn_6x8_160e_kitti-3d-car_20220331_134606-d42d15ed.pth'
model = init_model(config_file, checkpoint_file)
inference_detector(model, 'demo/data/kitti/000008.bin')
```

您将会看到一个包含 `Det3DDataSample` 的列表，预测结果在 `pred_instances_3d` 里面，包含有检测框，类别和得分。

### 自定义安装

#### CUDA 版本

在安装 PyTorch 时，您需要指定 CUDA 的版本。如果您不清楚应该选择哪一个，请遵循我们的建议：

- 对于 Ampere 架构的 NVIDIA GPU，例如 GeForce 30 系列以及 NVIDIA A100，CUDA 11 是必需的。
- 对于更早的 NVIDIA GPU，CUDA 11 是向后兼容的，但 CUDA 10.2 提供更好的兼容性，并且更轻量。

请确保 GPU 驱动版本满足最低的版本需求。更多信息请参考此[表格](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cuda-major-component-versions__table-cuda-toolkit-driver-versions)。

```{note}
如果您遵循我们的最佳实践，您只需要安装 CUDA 运行库，这是因为不需要在本地编译 CUDA 代码。但如果您希望从源码编译 MMCV，或者开发其他 CUDA 算子，那么您需要从 NVIDIA 的[官网](https://developer.nvidia.com/cuda-downloads)安装完整的 CUDA 工具链，并且该版本应该与 PyTorch 的 CUDA 版本相匹配，比如在 `conda install` 指令里指定 cudatoolkit 版本。
```

#### 不通过 MIM 安装 MMEngine

如果想要使用 pip 而不是 MIM 安装 MMEngine，请参考 [MMEngine 安装指南](https://mmengine.readthedocs.io/zh_CN/latest/get_started/installation.html)。

例如，您可以通过以下指令安装 MMEngine：

```shell
```

#### 不通过 MIM 安装 MMCV

MMCV 包含 C++ 和 CUDA 拓展，因此其对 PyTorch 的依赖更复杂。MIM 会自动解决此类依赖关系并使安装更容易。但这不是必需的。

```shell
```

如果想要使用 pip 而不是 MIM 安装 MMCV，请参考 [MMCV 安装指南](https://mmcv.readthedocs.io/zh_CN/2.x/get_started/installation.html)。这需要用指定 url 的形式手动指定对应的 PyTorch 和 CUDA 版本。

例如，下述指令将会安装基于 PyTorch 1.12.x 和 CUDA 11.6 编译的 MMCV：

```shell
pip install "mmcv>=2.0.0rc4" -f https://download.openmmlab.com/mmcv/dist/cu116/torch1.12.0/index.html
```

#### 在 Google Colab 中安装

[Google Colab](https://colab.research.google.com/) 通常已经安装了 PyTorch，因此我们只需要用如下命令安装 MMEngine，MMCV，MMDetection 和 MMDetection3D 即可。

**步骤 1.** 使用 [MIM](https://github.com/open-mmlab/mim) 安装 [MMEngine](https://github.com/open-mmlab/mmengine)，[MMCV](https://github.com/open-mmlab/mmcv) 和 [MMDetection](https://github.com/open-mmlab/mmdetection)。

```shell
!pip3 install openmim
!mim install mmengine
!mim install "mmcv>=2.0.0rc4,<2.1.0"
!mim install "mmdet>=3.0.0,<3.1.0"
```

```shell
```

**步骤 2.** 从源码安装 MMDetection3D。

```shell
!git clone https://github.com/open-mmlab/mmdetection3d.git -b dev-1.x
%cd mmdetection3d
```

**步骤 3.** 验证安装是否成功。

```python
import mmdet3d
print(mmdet3d.__version__)
# 预期输出：1.1.0rc0 或其它版本号。
```

```{note}
在 Jupyter Notebook 中，感叹号 `!` 用于执行外部命令，而 `%cd` 是一个[魔术命令](https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-cd)，用于切换 Python 的工作路径。
```

#### 通过 Docker 使用 MMDetection3D

我们提供了 [Dockerfile](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/docker/Dockerfile) 来构建一个镜像。请确保您的 [docker 版本](https://docs.docker.com/engine/install/) >= 19.03。

```shell
# 基于 PyTorch 1.9，CUDA 11.1 构建镜像
# 如果您想要其他版本，只需要修改 Dockerfile
docker build -t mmdetection3d docker/
```

用以下命令运行 Docker 镜像：

```shell
docker run --gpus all --shm-size=8g -it -v {DATA_DIR}:/mmdetection3d/data mmdetection3d
```

### 故障排除

```

```shell
```

### 故障排除

如果您在安装过程中遇到一些问题，请先参考 [FAQ](notes/faq.md) 页面。如果没有找到对应的解决方案，您也可以在 GitHub [提一个问题](https://github.com/open-mmlab/mmdetection3d/issues/new/choose)。

### 使用多个 MMDetection3D 版本进行开发

训练和测试的脚本已经在 `PYTHONPATH` 中进行了修改，以确保脚本使用当前目录中的 MMDetection3D。

要使环境中安装默认版本的 MMDetection3D 而不是当前正在使用的，可以删除出现在相关脚本中的代码：

```shell
```

# YOLOX-Pose

This project implements a YOLOX-based human pose estimator, utilizing the approach outlined in **YOLO-Pose: Enhancing YOLO for Multi Person Pose Estimation Using Object Keypoint Similarity Loss** (CVPRW 2022). This pose estimator is lightweight and quick, making it well-suited for crowded scenes.

<img src="https://user-images.githubusercontent.com/26127467/226655503-3cee746e-6e42-40be-82ae-6e7cae2a4c7e.jpg" alt><br>

## Usage

- Python 3.7 or higher
- [MMEngine](https://github.com/open-mmlab/mmengine) v0.6.0 or higher
- [MMCV](https://github.com/open-mmlab/mmcv) v2.0.0rc4 or higher
- [MMDetection](https://github.com/open-mmlab/mmdetection) v3.0.0rc6 or higher
- [MMYOLO](https://github.com/open-mmlab/mmyolo) v0.5.0 or higher
- [MMPose](https://github.com/open-mmlab/mmpose) v1.0.0rc1 or higher

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. **In `yolox-pose/` root directory**, run the following line to add the current directory to `PYTHONPATH`:

```shell
```

### Inference

Users can apply YOLOX-Pose models to estimate human poses using the inferencer found in the MMPose core package. Use the command below:

```shell
python demo/inferencer_demo.py $INPUTS \
    --pose2d $CONFIG --pose2d-weights $CHECKPOINT --scope mmyolo \
    [--show] [--vis-out-dir $VIS_OUT_DIR] [--pred-out-dir $PRED_OUT_DIR]
```

For more information on using the inferencer, please see [this document](https://mmpose.readthedocs.io/en/latest/user_guides/inference.html#out-of-the-box-inferencer).

Here's an example code:

```

```shell
python demo/inferencer_demo.py ../../tests/data/coco/000000000785.jpg \
    --pose2d configs/yolox-pose_s_8xb32-300e_coco.py \
    --pose2d-weights https://download.openmmlab.com/mmpose/v1/projects/yolox-pose/yolox-pose_s_8xb32-300e_coco-9f5e3924_20230321.pth \
    --scope mmyolo --vis-out-dir vis_results
```

This will create an output image `vis_results/000000000785.jpg`, which appears like:

<img src="https://user-images.githubusercontent.com/26127467/226552585-19b91294-9751-4599-98e7-5dae071a1761.jpg" height="360px" alt><br>

### Training & Testing

#### Data Preparation

Prepare the COCO dataset according to the [instruction](https://mmpose.readthedocs.io/en/latest/dataset_zoo/2d_body_keypoint.html#coco).

#### Commands

```shell

```shell
bash tools/dist_train.sh $CONFIG 8 --amp
```

**To train with slurm:**

```shell
bash tools/slurm_train.sh $PARTITION $JOBNAME $CONFIG $WORKDIR --amp
```

```shell
python tools/test.py $CONFIG $CHECKPOINT
```

```shell
bash tools/dist_test.sh $CONFIG $CHECKPOINT 8
```

```shell
bash tools/slurm_test.sh $PARTITION $JOBNAME $CONFIG $CHECKPOINT
```

### Results

Results on COCO val2017

|                              Model                              | Input Size |  AP   | AP<sup>50</sup> | AP<sup>75</sup> |  AR   | AR<sup>50</sup> |                                 Download                                 |

| :-------------------------------------------------------------: | :--------: | :---: | :-------------: | :-------------: | :---: | :-------------: | :----------------------------------------------------------------------: |
| [YOLOX-tiny-Pose](./configs/yolox-pose_tiny_4xb64-300e_coco.py) |    416     | 0.518 |      0.799      |      0.545      | 0.566 |      0.841      | [model](https://download.openmmlab.com/mmpose/v1/projects/yolox-pose/yolox-pose_tiny_4xb64-300e_coco-c47dd83b_20230321.pth) \| [log](https://download.openmmlab.com/mmpose/v1/projects/yolox-pose/yolox-pose_tiny_4xb64-300e_coco_20230321.json) |

|    [YOLOX-s-Pose](./configs/yolox-pose_s_8xb32-300e_coco.py)    |    640     | 0.632 |      0.875      |      0.692      | 0.676 |      0.907      | [model](https://download.openmmlab.com/mmpose/v1/projects/yolox-pose/yolox-pose_s_8xb32-300e_coco-9f5e3924_20230321.pth) \| [log](https://download.openmmlab.com/mmpose/v1/projects/yolox-pose/yolox-pose_s_8xb32-300e_coco_20230321.json) |
|    [YOLOX-m-Pose](./configs/yolox-pose_m_4xb64-300e_coco.py)    |    640     | 0.685 |      0.897      |      0.753      | 0.727 |      0.925      | [model](https://download.openmmlab.com/mmpose/v1/projects/yolox-pose/yolox-pose_m_4xb64-300e_coco-cbd11d30_20230321.pth) \| [log](https://download.openmmlab.com/mmpose/v1/projects/yolox-pose/yolox-pose_m_4xb64-300e_coco_20230321.json) |

|    [YOLOX-l-Pose](./configs/yolox-pose_l_4xb64-300e_coco.py)    |    640     | 0.706 |      0.907      |      0.775      | 0.747 |      0.934      | [model](https://download.openmmlab.com/mmpose/v1/projects/yolox-pose/yolox-pose_l_4xb64-300e_coco-122e4cf8_20230321.pth) \| [log](https://download.openmmlab.com/mmpose/v1/projects/yolox-pose/yolox-pose_l_4xb64-300e_coco_20230321.json) |

We have only trained models with an input size of 640, as we couldn't replicate the performance enhancement mentioned in the paper when increasing the input size from 640 to 960. We warmly welcome any contributions if you can successfully reproduce the results from the paper!

**NEW!**

[MMYOLO](https://github.com/open-mmlab/mmyolo/blob/dev/configs/yolox/README.md#yolox-pose) also supports YOLOX-Pose and achieves better performance. Their models are fully compatible with this project. Here are their results on COCO val2017:

|  Backbone  | Size | Batch Size | AMP | RTMDet-Hyp | Mem (GB) |  AP  |                                   Config                                   |                                   Download                                    |
| :--------: | :--: | :--------: | :-: | :--------: | :------: | :--: | :------------------------------------------------------------------------: | :---------------------------------------------------------------------------: |

| YOLOX-tiny | 416  |   8xb32    | Yes |    Yes     |   5.3    | 52.8 | [config](https://github.com/open-mmlab/mmyolo/blob/dev/configs/yolox/pose/yolox-pose_tiny_8xb32-300e-rtmdet-hyp_coco.py) | [model](https://download.openmmlab.com/mmyolo/v0/yolox/pose/yolox-pose_tiny_8xb32-300e-rtmdet-hyp_coco/yolox-pose_tiny_8xb32-300e-rtmdet-hyp_coco_20230427_080351-2117af67.pth) \| [log](https://download.openmmlab.com/mmyolo/v0/yolox/pose/yolox-pose_tiny_8xb32-300e-rtmdet-hyp_coco/yolox-pose_tiny_8xb32-300e-rtmdet-hyp_coco_20230427_080351.log.json) |

|  YOLOX-s   | 640  |   8xb32    | Yes |    Yes     |   10.7   | 63.7 | [config](https://github.com/open-mmlab/mmyolo/blob/dev/configs/yolox/pose/yolox-pose_s_8xb32-300e-rtmdet-hyp_coco.py) | [model](https://download.openmmlab.com/mmyolo/v0/yolox/pose/yolox-pose_s_8xb32-300e-rtmdet-hyp_coco/yolox-pose_s_8xb32-300e-rtmdet-hyp_coco_20230427_005150-e87d843a.pth) \| [log](https://download.openmmlab.com/mmyolo/v0/yolox/pose/yolox-pose_s_8xb32-300e-rtmdet-hyp_coco/yolox-pose_s_8xb32-300e-rtmdet-hyp_coco_20230427_005150.log.json) |

|  YOLOX-m   | 640  |   8xb32    | Yes |    Yes     |   19.2   | 69.3 | [config](https://github.com/open-mmlab/mmyolo/blob/dev/configs/yolox/pose/yolox-pose_m_8xb32-300e-rtmdet-hyp_coco.py) | [model](https://download.openmmlab.com/mmyolo/v0/yolox/pose/yolox-pose_m_8xb32-300e-rtmdet-hyp_coco/yolox-pose_m_8xb32-300e-rtmdet-hyp_coco_20230427_094024-bbeacc1c.pth) \| [log](https://download.openmmlab.com/mmyolo/v0/yolox/pose/yolox-pose_m_8xb32-300e-rtmdet-hyp_coco/yolox-pose_m_8xb32-300e-rtmdet-hyp_coco_20230427_094024.log.json) |

|  YOLOX-l   | 640  |   8xb32    | Yes |    Yes     |   30.3   | 71.1 | [config](https://github.com/open-mmlab/mmyolo/blob/dev/configs/yolox/pose/yolox-pose_l_8xb32-300e-rtmdet-hyp_coco.py) | [model](https://download.openmmlab.com/mmyolo/v0/yolox/pose/yolox-pose_l_8xb32-300e-rtmdet-hyp_coco/yolox-pose_l_8xb32-300e-rtmdet-hyp_coco_20230427_041140-82d65ac8.pth) \| [log](https://download.openmmlab.com/mmyolo/v0/yolox/pose/yolox-pose_l_8xb32-300e-rtmdet-hyp_coco/yolox-pose_l_8xb32-300e-rtmdet-hyp_coco_20230427_041140.log.json) |

If this project benefits your work, please kindly consider citing the original papers:

```bibtex
@inproceedings{maji2022yolo,
  title={YOLO-Pose: Enhancing YOLO for Multi Person Pose Estimation Using Object Keypoint Similarity Loss},
  author={Maji, Debapriya and Nagori, Soyeb and Mathew, Manu and Poddar, Deepak},
  pages={2637--2646},
}
```

booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
}
```

```bibtex
@article{yolox2021,
  title={{YOLOX}: Exceeding YOLO Series in 2021},
  author={Ge, Zheng and Liu, Songtao and Wang, Feng and Li, Zeming and Sun, Jian},
  journal={arXiv preprint arXiv:2107.08430},
}
```

Additionally, please cite our work as well:

```bibtex
}
```

<details>

```bibtex
}
```

<details>

```bibtex

author={Sandler, Mark and Howard, Andrew and Zhu, Menglong and Zhmoginov, Andrey and Chen, Liang-Chieh},
}
```

<details>

```bibtex
}
```

| [simcc_mobilenetv2_wo_deconv](/configs/body_2d_keypoint/simcc/coco/simcc_mobilenetv2_wo-deconv-8xb64-210e_coco-256x192.py) |  256x192   | 0.620 |      0.855      |      0.697      | 0.678 |      0.902      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/simcc/coco/simcc_mobilenetv2_wo-deconv-8xb64-210e_coco-256x192-4b0703bb_20221010.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/simcc/coco/simcc_mobilenetv2_wo-deconv-8xb64-210e_coco-256x192-4b0703bb_20221010.log.json) |

### image roi crop + convert to ncnn::Mat

```
+--------------+
|   y          |           /-------/
| x +-------+  |          +-------+|
|   |     roih |im_h  =>  |      roih
|   +-roiw--+  |          +-roiw--+/
|              |
+-----im_w-----+
```
```cpp
ncnn::Mat in = ncnn::Mat::from_pixels_roi(im.data, ncnn::Mat::PIXEL_RGB, im_w, im_h, x, y, roiw, roih);
```
For Android Application, it is :
```cpp
ncnn::Mat in = ncnn::Mat::from_android_bitmap_roi(env, image, ncnn::Mat::PIXEL_RGBA2RGB, x, y, roiw, roih);
```

### image roi crop + resize + convert to ncnn::Mat

```
|   y          |           /----/
| x +-------+  |          +----+|
|   |     roih |im_h  =>  |  target_h
|   +-roiw--+  |          |    ||
|              |          +----+/
+-----im_w-----+         target_w
```
```cpp
ncnn::Mat in = ncnn::Mat::from_pixels_roi_resize(im.data, ncnn::Mat::PIXEL_RGB, im_w, im_h, x, y, roiw, roih, target_w, target_h);
```
```cpp

```
```cpp
```
```cpp
ncnn::Mat in = ncnn::Mat::from_android_bitmap_roi_resize(env, image, ncnn::Mat::PIXEL_RGBA2RGB, x, y, roiw, roih, target_w, target_h);
```

### ncnn::Mat export image + offset paste

```
                +--------------+
 /-------/      |   y          |
+-------+|      | x +-------+  |
|       h|  =>  |   |       h  |im_h
+---w---+/      |   +---w---+  |
                |              |
                +-----im_w-----+
```
```cpp
const unsigned char* data = im.data + (y * im_w + x) * 3;
out.to_pixels(data, ncnn::Mat::PIXEL_RGB, im_w * 3);
```

### ncnn::Mat export image + resize + roi paste

```
            +--------------+
 /----/     |   y          |
+----+|     | x +-------+  |
|    h| =>  |   |      roih|im_h
|    ||     |   +-roiw--+  |
+-w--+/     |              |
            +-----im_w-----+
```

/----/     |   y          |
+-w--+/     |              |
```
```cpp
out.to_pixels_resize(data, ncnn::Mat::PIXEL_RGB, roiw, roih, im_w * 3);
```

### image roi crop + resize
```
|   y          |
| x +-------+  |          +----+
|   |      roih|im_h  =>  |  target_h
|   +-roiw--+  |          |    |
|              |          +----+
```
```cpp
ncnn::resize_bilinear_c3(data, roiw, roih, im_w * 3, outdata, target_w, target_h, target_w * 3);
```

### image resize + offset paste
```
            |   y          |
+----+      | x +-------+  |
|    h  =>  |   |     roih |im_h
|    |      |   +-roiw--+  |
+-w--+      |              |
```
```cpp

|   y          |
+-w--+      |              |
```
```cpp
unsigned char* outdata = im.data + (y * im_w + x) * 3;
ncnn::resize_bilinear_c3(data, w, h, w * 3, outdata, roiw, roih, im_w * 3);
```

### image roi crop + resize + roi paste
```
+--------------+         +-----------------+
|   y          |         |  roiy           |
| x +-------+  |         |roix----------+  |
|   |       h  |im_h  => |   |     target_h|outim_h
|   +---w---+  |         |   |          |  |
|              |         |   +-target_w-+  |
+-----im_w-----+         +-----outim_w-----+
```
```cpp
unsigned char* outdata = outim.data + (roiy * outim_w + roix) * 3;
ncnn::resize_bilinear_c3(data, w, h, im_w * 3, outdata, target_w, target_h, outim_w * 3);
```

### image roi crop + rotate
```
|   y          |

```

```
|   y          |
| x +-------+  |          +---+
|   |  < <  h  |im_h  =>  | ^ |w
|   +---w---+  |          | ^ |
|              |          +---+
+-----im_w-----+            h
```
```cpp
ncnn::kanna_rotate_c3(data, w, h, im_w * 3, outdata, h, w, h * 3, 6);
```

### image rotate + offset paste
```
             +--------------+
             |   y          |
 +---+       | x +-------+  |
 | ^ |h  =>  |   |  < <  w  |im_h
 | ^ |       |   +---h---+  |
 +---+       |              |
   w         +-----im_w-----+
```
```cpp
ncnn::kanna_rotate_c3(data, w, h, w * 3, outdata, h, w, im_w * 3, 7);
```

### image roi crop + rotate + roi paste
```
|   y          |         |        roiy     |

```

```
|   y          |         |        roiy     |
| x +-------+  |         |   roix  +---+   |
|   |  < <  h  |im_h  => |         | ^ w   |outim_h
|   +---w---+  |         |         | ^ |   |
|              |         |         +-h-+   |
```
```cpp
ncnn::kanna_rotate_c3(data, w, h, im_w * 3, outdata, h, w, outim_w * 3, 6);
```

## 数据集准备

### 基础检测数据集准备

MMDetection 支持多个公共数据集，包括 [COCO](https://cocodataset.org/)， [Pascal VOC](http://host.robots.ox.ac.uk/pascal/VOC)， [Cityscapes](https://www.cityscapes-dataset.com/) 和 [其他更多数据集](https://github.com/open-mmlab/mmdetection/tree/main/configs/_base_/datasets)。

一些公共数据集，比如 Pascal VOC 及其镜像数据集，或者 COCO 等数据集都可以从官方网站或者镜像网站获取。注意：在检测任务中，Pascal VOC 2012 是 Pascal VOC 2007 的无交集扩展，我们通常将两者一起使用。 我们建议将数据集下载，然后解压到项目外部的某个文件夹内，然后通过符号链接的方式，将数据集根目录链接到 `$MMDETECTION/data` 文件夹下， 如果你的文件夹结构和下方不同的话，你需要在配置文件中改变对应的路径。

我们提供了下载 COCO 等数据集的脚本，你可以运行 `python tools/misc/download_dataset.py --dataset-name coco2017` 下载 COCO 数据集。 对于中国境内的用户，我们也推荐通过开源数据平台 [OpenDataLab](https://opendatalab.com/?source=OpenMMLab%20GitHub) 来下载数据，以获得更好的下载体验。

更多用法请参考[数据集下载](./useful_tools.md#dataset-download)

```text
├── mmdet
├── tools
├── data
│   ├── coco

├── mmdet
├── tools
├── data
│   ├── coco
│   │   │   ├── val
│   │   │   ├── val
```

有些模型需要额外的 [COCO-stuff](http://calvin.inf.ed.ac.uk/wp-content/uploads/data/cocostuffdataset/stuffthingmaps_trainval2017.zip) 数据集，比如 HTC，DetectoRS 和 SCNet，你可以下载并解压它们到 `coco` 文件夹下。文件夹会是如下结构：

```text
├── data
│   ├── coco
```

PanopticFPN 等全景分割模型需要额外的 [COCO Panoptic](http://images.cocodataset.org/annotations/panoptic_annotations_trainval2017.zip) 数据集，你可以下载并解压它们到 `coco/annotations` 文件夹下。文件夹会是如下结构：

```text
├── data
│   ├── coco

```text
├── data
│   ├── coco
```

Cityscape 数据集的标注格式需要转换，以与 COCO 数据集标注格式保持一致，使用 `tools/dataset_converters/cityscapes.py` 来完成转换：

```shell

    --nproc 8 \
```

### COCO Caption 数据集准备

COCO Caption 采用的是 COCO2014 数据集作为图片，并且使用了 karpathy 的标注，

首先你需要下载 COCO2014 数据集

```shell
```

数据集会下载到当前路径的 `data/coco` 下。然后下载 karpathy 的标注

```shell

```

```shell
```

最终直接可用于训练和测试的数据集文件夹结构如下：

```text
├── data
│   ├── coco
```

### COCO semantic 数据集准备

```

COCO 语义分割有两种类型标注，主要差别在于类别名定义不一样，因此处理方式也有两种，第一种是直接使用 stuffthingmaps 数据集，第二种是使用 panoptic 数据集。

**(1) 使用 stuffthingmaps 数据集**

该数据集的下载地址为 [stuffthingmaps_trainval2017](http://calvin.inf.ed.ac.uk/wp-content/uploads/data/cocostuffdataset/stuffthingmaps_trainval2017.zip)，请下载后解压到 `data/coco` 文件夹下。

```text
├── data
│   ├── coco
```

该数据集不同于标准的 COCO 类别标注，其包括 172 个类： 80 thing 类、91 stuff 类和 1 个 'unlabeled'，其每个类别的说明见 https://github.com/nightrome/cocostuff/blob/master/labels.md

虽然只标注了 172 个类别，但是 `stuffthingmaps` 中最大标签 id 是 182，中间有些类别是没有标注的，并且第 0 类的 `unlabeled` 类别被移除。因此最终的 `stuffthingmaps` 图片中每个位置的值对应的类别关系见 https://github.com/kazuto1011/deeplab-pytorch/blob/master/data/datasets/cocostuff/labels.txt

考虑到训练高效和方便用户，在开启训练或者评估前，我们需要将没有标注的 12 个类移除，这 12 个类的名字为： `street sign、hat、shoe、eye glasses、plate、mirror、window、desk、door、blender、hair brush`，最终可用于训练和评估的类别信息见 `mmdet/datasets/coco_semantic.py`

你可以使用 `tools/dataset_converters/coco_stuff164k.py` 来完成将下载的 `stuffthingmaps` 转换为直接可以训练和评估的数据集，转换后的数据集文件夹结构如下：

```text
├── data
│   ├── coco
```

`stuffthingmaps_semseg` 即为新生成的可以直接训练和测试的 COCO 语义分割数据集。

**(2) 使用 panoptic 数据集**

通过 panoptic 标注生成的语义分割数据集类别数相比使用 `stuffthingmaps` 数据集生成的会少一些。首先你需要准备全景分割标注，然后使用如下脚本完成转换

```shell
```

转换后的数据集文件夹结构如下：

```text
├── data
│   ├── coco

```

```text
├── data
│   ├── coco
```

`panoptic_semseg_train2017` 和 `panoptic_semseg_val2017` 即为新生成的可以直接训练和测试的 COCO 语义分割数据集。注意其类别信息就是 COCO 全景分割的类别信息，包括 thing 和 stuff。

### RefCOCO 数据集准备

[RefCOCO](https://github.com/lichengunc/refer)系列数据集的图像和注释可以通过运行 `tools/misc/download_dataset.py` 下载:

```shell
```

然后，目录应该是这样的：

```text
data
├── coco

```

### ADE20K 数据集准备

[ADE20K](http://groups.csail.mit.edu/vision/datasets/ADE20K/)数据集的图像和注释可以通过运行 `tools/misc/download_dataset.py` 下载:

```shell
```

然后将注释移至`data/ADEChallengeData2016`目录，并运行预处理脚本以产生coco格式注释：

```shell
```

```text
data

```

```text
data
|   |   ├── ...
|   |   ├── ...
|   |   |   ├── ...
|   |   |   ├── ...
|   |   |   ├── ...

|   |   |   ├── ...
|   |   |   ├── ...
|   |   |   ├── ...
|   |   |   ├── ...
```

上述文件夹包括ADE20K的语义分割、实例分割和泛在分割的所有数据。

### 从 OpenDataLab 中下载

[OpenDataLab](https://opendatalab.com/) 为人工智能研究者提供免费开源的数据集，通过 OpenDataLab，研究者可以获得格式统一的各领域经典数据集。通过平台的搜索功能，研究者可以迅速便捷地找到自己所需数据集；通过平台的统一格式，研究者可以便捷地对跨数据集任务进行开发。

目前，MIM 支持使用一条命令行从 OpenDataLab 中下载 VOC 和 COCO 数据集，后续将支持更多数据集。你也可以直接访问 OpenDataLab 平台下载你所需的数据集，然后将其转化为 MMDetection 所要求的格式。

如果使用 MIM 下载，请确保版本大于 v0.3.8，你可以使用如下命令更新:

```Bash

```Bash
```

```Bash
odl login

```

# Preparing Deraining Dataset

```bibtex
}
```

The test datasets (Rain100H, Rain100L, Test100, Test1200, Test2800) can be download from [here](https://drive.google.com/file/d/1P_-RAvltEoEhfT-9GrWRdpEi6NSswTs8/).

```text
mmagic
├── mmagic
├── tools
├── data
```

# 配置数据集

本节教程主要关注如何选择和配置所需要的数据集。请确保你已按照[数据集准备](../get_started.md#数据集准备)中的步骤下载好数据集。

## 数据集配置文件目录结构

首先简单介绍一下 OpenCompass `configs/datasets` 目录下的结构，如下所示：

```text
configs/datasets/
├── agieval
├── apps
├── ARC_c
├── ...
├── CLUE_afqmc  # 数据集
│   ├── CLUE_afqmc_gen_901306.py  # 不同版本数据集配置文件
│   ├── CLUE_afqmc_gen.py
│   ├── CLUE_afqmc_ppl_378c5b.py
│   ├── CLUE_afqmc_ppl_6507d7.py
│   ├── CLUE_afqmc_ppl_7b0c1e.py
│   └── CLUE_afqmc_ppl.py
├── ...
├── XLSum
├── Xsum
└── z_bench
```

在 `configs/datasets` 目录结构下，我们直接展平所有数据集，在各个数据集对应的文件夹下存在多个数据集配置。

数据集配置文件名由以下命名方式构成 `{数据集名称}_{评测方式}_{prompt版本号}.py`，以 `CLUE_afqmc/CLUE_afqmc_gen_db509b.py` 为例，该配置文件则为中文通用能力下的 `CLUE_afqmc` 数据集，对应的评测方式为 `gen`，即生成式评测，对应的prompt版本号为 `db509b`；同样的， `CLUE_afqmc_ppl_00b348.py` 指评测方式为`ppl`即判别式评测，prompt版本号为 `00b348` 。

除此之外，不带版本号的文件，例如： `CLUE_afqmc_gen.py` 则指向该评测方式最新的prompt配置文件，通常来说会是精度最高的prompt。

## 数据集选择

在各个数据集配置文件中，数据集将会被定义在 `{}_datasets` 变量当中，例如下面 `CLUE_afqmc/CLUE_afqmc_gen_db509b.py` 中的 `afqmc_datasets`。

```python

## 数据集选择

```python
afqmc_datasets = [
    dict(
        abbr="afqmc-dev",
        type=AFQMCDataset_V2,
        path="./data/CLUE/AFQMC/dev.json",
        reader_cfg=afqmc_reader_cfg,
        infer_cfg=afqmc_infer_cfg,
        eval_cfg=afqmc_eval_cfg,
    ),
]
```

以及 `CLUE_cmnli/CLUE_cmnli_ppl_b78ad4.py` 中的 `cmnli_datasets`。

```python
cmnli_datasets = [
    dict(
        type=HFDataset,
        abbr='cmnli',
        path='json',
        split='train',
        data_files='./data/CLUE/cmnli/cmnli_public/dev.json',
        reader_cfg=cmnli_reader_cfg,
        infer_cfg=cmnli_infer_cfg,
        eval_cfg=cmnli_eval_cfg)
]
```

以上述两个数据集为例， 如果用户想同时评测这两个数据集，可以在 `configs` 目录下新建一个配置文件，我们使用  `mmengine` 配置中直接import的机制来构建数据集部分的参数，如下所示：

```python
from mmengine.config import read_base

with read_base():
    from .datasets.CLUE_afqmc.CLUE_afqmc_gen_db509b import afqmc_datasets

```python

    from .datasets.CLUE_cmnli.CLUE_cmnli_ppl_b78ad4 import cmnli_datasets

datasets = []
datasets += afqmc_datasets
datasets += cmnli_datasets
```

用户可以根据需要，选择不同能力不同数据集以及不同评测方式的配置文件来构建评测脚本中数据集的部分。

有关如何启动评测任务，以及如何评测自建数据集可以参考相关文档。

# Preparing OmniSource

```BibTeX
@article{duan2020omni,
  title={Omni-sourced Webly-supervised Learning for Video Recognition},
  author={Duan, Haodong and Zhao, Yue and Xiong, Yuanjun and Liu, Wentao and Lin, Dahua},
  journal={arXiv preprint arXiv:2003.13042},
}
```

We release a subset of the OmniSource web dataset used in the paper [Omni-sourced Webly-supervised Learning for Video Recognition](https://arxiv.org/abs/2003.13042). Since all web dataset in OmniSource are built based on the Kinetics-400 taxonomy, we select those web data related to the 200 classes in Mini-Kinetics subset (which is proposed in [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://arxiv.org/pdf/1712.04851.pdf)).

We provide data from all sources that are related to the 200 classes in Mini-Kinetics (including Kinetics trimmed clips, Kinetics untrimmed videos, images from Google and Instagram, video clips from Instagram).  To obtain this dataset, please first fill in the [request form](https://docs.google.com/forms/d/e/1FAIpQLSd8_GlmHzG8FcDbW-OEu__G7qLgOSYZpH-i5vYVJcu7wcb_TQ/viewform?usp=sf_link). We will share the download link to you after your request is received. Since we release all data crawled from the web without any filtering, the dataset is large and it may take some time to download them. We describe the size of the datasets in the following table:

|  Dataset Name   | #samples |  Size   |  Teacher Model   | #samples after filtering | #samples similar to k200_val |
| :-------------: | :------: | :-----: | :--------------: | :----------------------: | :--------------------------: |

|   k200_train    |  76030   |  45.6G  |       N/A        |           N/A            |             N/A              |
|    k200_val     |   4838   |  2.9G   |       N/A        |           N/A            |             N/A              |
| googleimage_200 | 3050880  | 265.5G  |   TSN-R50-8seg   |         1188695          |             967              |
|  insimage_200   | 3654650  | 224.4G  |   TSN-R50-8seg   |          879726          |             116              |
|  insvideo_200   |  732855  | 1487.6G | SlowOnly-8x8-R50 |          330680          |             956              |
| k200_raw_train  |  76027   | 963.5G  | SlowOnly-8x8-R50 |           N/A            |             N/A              |

The file structure of our uploaded OmniSource dataset looks like:

```
OmniSource/
├── annotations
│   ├── googleimage_200

```
│   │   ├── googleimage_200.txt                       File list of all valid images crawled from Google.
│   │   ├── tsn_8seg_googleimage_200_duplicate.txt    Positive file list of images crawled from Google, which is similar to a validation example.
│   │   ├── tsn_8seg_googleimage_200.txt              Positive file list of images crawled from Google, filtered by the teacher model.
│   │   └── tsn_8seg_googleimage_200_wodup.txt        Positive file list of images crawled from Google, filtered by the teacher model, after de-duplication.
│   ├── insimage_200
│   │   ├── insimage_200.txt
│   │   ├── tsn_8seg_insimage_200_duplicate.txt
│   │   ├── tsn_8seg_insimage_200.txt
│   │   └── tsn_8seg_insimage_200_wodup.txt
│   ├── insvideo_200
│   │   ├── insvideo_200.txt
│   │   ├── slowonly_8x8_insvideo_200_duplicate.txt
│   │   ├── slowonly_8x8_insvideo_200.txt

│   │   └── slowonly_8x8_insvideo_200_wodup.txt
│   ├── k200_actions.txt                              The list of action names of the 200 classes in MiniKinetics.
│   ├── K400_to_MiniKinetics_classidx_mapping.json    The index mapping from Kinetics-400 to MiniKinetics.
│   ├── kinetics_200
│   │   ├── k200_train.txt
│   │   └── k200_val.txt
│   ├── kinetics_raw_200
│   │   └── slowonly_8x8_kinetics_raw_200.json        Kinetics Raw Clips filtered by the teacher model.
│   └── webimage_200
│       └── tsn_8seg_webimage_200_wodup.txt           The union of `tsn_8seg_googleimage_200_wodup.txt` and `tsn_8seg_insimage_200_wodup.txt`
├── googleimage_200                                   (10 volumes)
│   ├── vol_0.tar
│   ├── ...
│   └── vol_9.tar

│   ├── ...
├── insimage_200                                      (10 volumes)
│   ├── ...
├── insvideo_200                                      (20 volumes)
│   ├── vol_00.tar
│   ├── ...
│   └── vol_19.tar
├── kinetics_200_train
│   └── kinetics_200_train.tar
├── kinetics_200_val
│   └── kinetics_200_val.tar
└── kinetics_raw_200_train                            (16 volumes)
    ├── vol_0.tar
    ├── ...
    └── vol_15.tar
```

For data preparation, you need to first download those data. For `kinetics_200` and 3 web datasets: `googleimage_200`, `insimage_200` and `insvideo_200`, you just need to extract each volume and merge their contents.

For Kinetics raw videos, since loading long videos is very heavy, you need to first trim it into clips. Here we provide a script named `trim_raw_video.py`. It trims a long video into 10-second clips and remove the original raw video. You can use it to trim the Kinetics raw video.

The data should be placed in `data/OmniSource/`. When data preparation finished, the folder structure of `data/OmniSource` looks like (We omit the files not needed in training & testing for simplicity):

```
data/OmniSource/
│   │   └── tsn_8seg_googleimage_200_wodup.txt    Positive file list of images crawled from Google, filtered by the teacher model, after de-duplication.

│   │   └── slowonly_8x8_kinetics_raw_200.json    Kinetics Raw Clips filtered by the teacher model.
│       └── tsn_8seg_webimage_200_wodup.txt       The union of `tsn_8seg_googleimage_200_wodup.txt` and `tsn_8seg_insimage_200_wodup.txt`
├── googleimage_200
│   ├── 000
|   │   ├── 00
|   │   │   ├── 000001.jpg
|   │   │   ├── ...
|   │   │   └── 000901.jpg
|   │   ├── ...
|   │   ├── 19
│   ├── ...
│   └── 199
├── insimage_200
│   ├── 000
|   │   ├── abseil
|   │   │   ├── 1J9tKWCNgV_0.jpg
|   │   │   ├── ...
|   │   │   └── 1J9tKWCNgV_0.jpg
|   │   ├── abseiling
│   ├── ...
│   └── 199
├── insvideo_200
│   ├── 000
|   │   │   ├── B00arxogubl.mp4
|   │   │   ├── ...
|   │   │   └── BzYsP0HIvbt.mp4
│   ├── ...
│   └── 199

│   ├── 000
|   │   │   ├── ...
│   ├── ...
│   └── 199
│   ├── 0074cdXclLU.mp4
|   ├── ...
|   ├── zzzlyL61Fyo.mp4
│   ├── 01fAWEHzudA.mp4
|   ├── ...
|   ├── zymA_6jZIz4.mp4
└── kinetics_raw_200_train
│   ├── pref_
│   |   ├── ___dTOdxzXY
|   │   │   ├── part_0.mp4
|   │   │   ├── ...
|   │   │   ├── part_6.mp4
│   |   ├── ...
│   |   └── _zygwGDE2EM
│   ├── ...
│   └── prefZ
```

# Task Execution and Monitoring

## Launching an Evaluation Task

The program entry for the evaluation task is `run.py`, its usage is as follows:

```shell
python run.py $Config {--slurm | --dlc | None} [-p PARTITION] [-q QUOTATYPE] [--debug] [-m MODE] [-r [REUSE]] [-w WORKDIR] [-l] [--dry-run]
```

Here are some examples for launching the task in different environments:

- Running locally: `run.py $Config`, where `$Config` does not contain fields 'eval' and 'infer'.
- Running with Slurm: `run.py $Config --slurm -p $PARTITION_name`.
- Running on ALiYun DLC: `run.py $Config --dlc --aliyun-cfg $AliYun_Cfg`, tutorial will come later.
- Customized run: `run.py $Config`, where `$Config` contains fields 'eval' and 'infer', and you are able to customize the way how each task will be split and launched. See [Evaluation document](./evaluation.md).

The parameter explanation is as follows:

- `-p`: Specify the slurm partition;

- `-q`: Specify the slurm quotatype (default is None), with optional values being reserved, auto, spot. This parameter may only be used in some slurm variants;
- `--debug`: When enabled, inference and evaluation tasks will run in single-process mode, and output will be echoed in real-time for debugging;
- `-m`: Running mode, default is `all`. It can be specified as `infer` to only run inference and obtain output results; if there are already model outputs in `{WORKDIR}`, it can be specified as `eval` to only run evaluation and obtain evaluation results; if the evaluation results are ready, it can be specified as `viz` to only run visualization, which summarizes the results in tables; if specified as `all`, a full run will be performed, which includes inference, evaluation, and visualization.

- `-r`: Reuse existing inference results, and skip the finished tasks. If followed by a timestamp, the result under that timestamp in the workspace path will be reused; otherwise, the latest result in the specified workspace path will be reused.
- `-w`: Specify the working path, default is `./outputs/default`.
- `-l`: Enable status reporting via Lark bot.
- `--dry-run`: When enabled, inference and evaluation tasks will be dispatched but won't actually run for debugging.

Using run mode `-m all` as an example, the overall execution flow is as follows:

1. Read the configuration file, parse out the model, dataset, evaluator, and other configuration information
2. The evaluation task mainly includes three stages: inference `infer`, evaluation `eval`, and visualization `viz`. After task division by Partitioner, they are handed over to Runner for parallel execution. Individual inference and evaluation tasks are abstracted into `OpenICLInferTask` and `OpenICLEvalTask` respectively.

3. After each stage ends, the visualization stage will read the evaluation results in `results/` to generate a table.

## Task Monitoring: Lark Bot

Users can enable real-time monitoring of task status by setting up a Lark bot. Please refer to [this document](https://open.feishu.cn/document/ukTMukTMukTM/ucTM5YjL3ETO24yNxkjN?lang=zh-CN#7a28964d) for setting up the Lark bot.

1. Open the `configs/lark.py` file, and add the following line:

   ```python
   lark_bot_url = 'YOUR_WEBHOOK_URL'
   ```

   Typically, the Webhook URL is formatted like this: https://open.feishu.cn/open-apis/bot/v2/hook/xxxxxxxxxxxxxxxxx .

2. Inherit this file in the complete evaluation configuration:

   ```python
     from mmengine.config import read_base

     with read_base():
         from .lark import lark_bot_url

   ```

   ```python

   ```

3. To avoid frequent messages from the bot becoming a nuisance, status updates are not automatically reported by default. You can start status reporting using `-l` or `--lark` when needed:

   ```bash
   python run.py configs/eval_demo.py -p {PARTITION} -l
   ```

## Run Results

All run results will be placed in `outputs/default/` directory by default, the directory structure is shown below:

```
outputs/default/
├── 20200220_120000
├── ...
├── 20230220_183030
│   ├── configs
│   ├── logs
│   │   ├── eval
│   │   └── infer
│   ├── predictions
│   │   └── MODEL1
│   └── results
│       └── MODEL1
```

Each timestamp contains the following content:

- configs folder, which stores the configuration files corresponding to each run with this timestamp as the output directory;

```

- logs folder, which stores the output log files of the inference and evaluation phases, each folder will store logs in subfolders by model;
- predictions folder, which stores the inferred json results, with a model subfolder;
- results folder, which stores the evaluated json results, with a model subfolder.

Also, all `-r` without specifying a corresponding timestamp will select the newest folder by sorting as the output directory.

## Introduction of Summerizer (to be updated)

# ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection

> [ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection](https://arxiv.org/abs/2106.01178)

In this paper, we introduce the task of multi-view RGB-based 3D object detection as an end-to-end optimization problem. To address this problem, we propose ImVoxelNet, a novel fully convolutional method of 3D object detection based on posed monocular or multi-view RGB images. The number of monocular images in each multiview input can variate during training and inference; actually, this number might be unique for each multi-view input. ImVoxelNet successfully handles both indoor and outdoor scenes, which makes it general-purpose. Specifically, it achieves state-of-the-art results in car detection on KITTI (monocular) and nuScenes (multi-view) benchmarks among all methods that accept RGB images. Moreover, it surpasses existing RGB-based 3D object detection methods on the SUN RGB-D dataset. On ScanNet, ImVoxelNet sets a new benchmark for multi-view 3D object detection.

<img src="https://user-images.githubusercontent.com/36950400/143871445-38a55168-b8cd-4520-8ed6-f5c8c8ea304a.png" width="800"/>
</div>

We implement a monocular 3D detector ImVoxelNet and provide its results and checkpoints on KITTI dataset.
Results for SUN RGB-D, ScanNet and nuScenes are currently available in ImVoxelNet authors
[repo](https://github.com/saic-vul/imvoxelnet) (based on mmdetection3d).

### KITTI

|                    Backbone                    | Class | Lr schd | Mem (GB) | Inf time (fps) |  mAP  |                                                                                                                                                               Download                                                                                                                                                               |

| :--------------------------------------------: | :---: | :-----: | :------: | :------------: | :---: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| [ResNet-50](./imvoxelnet_8xb4_kitti-3d-car.py) |  Car  |   3x    |          |                | 17.26 | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/imvoxelnet/imvoxelnet_4x8_kitti-3d-car/imvoxelnet_4x8_kitti-3d-car_20210830_003014-3d0ffdf4.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/imvoxelnet/imvoxelnet_4x8_kitti-3d-car/imvoxelnet_4x8_kitti-3d-car_20210830_003014.log.json) |

### SUN RGB-D

|                      Backbone                       | Lr schd | Mem (GB) | Inf time (fps) | mAP@0.25 | mAP@0.5 |                                                                                                                                                                           Download                                                                                                                                                                           |
| :-------------------------------------------------: | :-----: | :------: | :------------: | :------: | :-----: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

| [ResNet-50](./imvoxelnet_4x2_sunrgbd-3d-10class.py) |   2x    |   7.2    |      22.5      |  40.96   |  13.50  | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/imvoxelnet/imvoxelnet_4x2_sunrgbd-3d-10class/imvoxelnet_4x2_sunrgbd-3d-10class_20220809_184416-29ca7d2e.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/imvoxelnet/imvoxelnet_4x2_sunrgbd-3d-10class/imvoxelnet_4x2_sunrgbd-3d-10class_20220809_184416.log.json) |

```latex
@article{rukhovich2021imvoxelnet,
  title={ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection},
  author={Danila Rukhovich, Anna Vorontsova, Anton Konushin},
  journal={arXiv preprint arXiv:2106.01178},
}
```

# Preparing GYM

```BibTeX
@inproceedings{shao2020finegym,
  title={Finegym: A hierarchical video dataset for fine-grained action understanding},
  author={Shao, Dian and Zhao, Yue and Dai, Bo and Lin, Dahua},
  pages={2616--2625},
}
```

For basic dataset information, please refer to the official [project](https://sdolivia.github.io/FineGym/) and the [paper](https://arxiv.org/abs/2004.06704).
We currently provide the data pre-processing pipeline for GYM99.
Before we start, please make sure that the directory is located at `$MMACTION2/tools/data/gym/`.

First of all, you can run the following script to prepare annotations.

```shell
```

Then, you can run the following script to prepare videos.

```shell
```

The codes are adapted from the [official crawler](https://github.com/activitynet/ActivityNet/tree/master/Crawler/Kinetics). Note that this might take a long time.

```shell
```

## Step 3. Trim Videos into Events

First, you need to trim long videos into events based on the annotation of GYM with the following scripts.

```shell
python trim_event.py
```

## Step 4. Trim Events into Subactions

Then, you need to trim events into subactions based on the annotation of GYM with the following scripts. We use the two stage trimming for better efficiency (trimming multiple short clips from a long video can be extremely inefficient, since you need to go over the video many times).

```shell
python trim_subaction.py
```

## Step 5. Extract RGB and Flow

```shell
```

This part is **optional** if you only want to use the video loader for RGB model training.

Run the following script to extract both rgb and flow using "tvl1" algorithm.

```shell
```

## Step 6. Generate file list for GYM99 based on extracted subactions

You can use the following script to generate train / val lists for GYM99.

```shell
python generate_file_list.py
```

## Step 7. Folder Structure

After the whole data pipeline for GYM preparation. You can get the subaction clips, event clips, raw videos and GYM99 train/val lists.

In the context of the whole project (for GYM only), the full folder structure will look like:

```
mmaction2
├── tools
├── data
│   ├── gym
|   |   ├── annotations

```
mmaction2
├── tools
├── data
│   ├── gym
|   |   |   ├── gym99_train_org.txt
|   |   |   ├── gym99_val_org.txt
|   |   |   ├── gym99_train.txt
|   |   |   ├── gym99_val.txt
|   |   |   ├── annotation.json
|   |   |   └── event_annotation.json
│   │   ├── videos
|   |   |   ├── 0LtLS9wROrk.mp4
|   |   |   ├── ...
|   |   |   └── zfqS-wCJSsw.mp4
│   │   ├── events
|   |   |   ├── 0LtLS9wROrk_E_002407_002435.mp4
|   |   |   ├── ...
|   |   |   └── zfqS-wCJSsw_E_006732_006824.mp4
│   │   ├── subactions
|   |   |   ├── 0LtLS9wROrk_E_002407_002435_A_0003_0005.mp4
|   |   |   ├── ...
|   |   |   └── zfqS-wCJSsw_E_006244_006252_A_0000_0007.mp4
|   |   └── subaction_frames
```

For training and evaluating on GYM, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).

# 自定义运行参数

运行参数配置包括许多有用的功能，如权重文件保存、日志配置等等，在本教程中，我们将介绍如何配置这些功能。

## 保存权重文件

权重文件保存功能是一个在训练阶段默认注册的钩子， 你可以通过配置文件中的 `default_hooks.checkpoint` 字段配置它。

```{note}
钩子机制在 OpenMMLab 开源算法库中应用非常广泛。通过钩子，你可以在不修改运行器的主要执行逻辑的情况下插入许多功能。

可以通过{external+mmengine:doc}`相关文章 <tutorials/hook>`进一步理解钩子。
```

**默认配置:**

```python
    ...
    checkpoint = dict(type='CheckpointHook', interval=1)
    ...
)
```

下面是一些[权重文件钩子(CheckpointHook)](mmengine.hooks.CheckpointHook)的常用可配置参数。

- **`interval`** (int): 文件保存周期。如果使用-1，它将永远不会保存权重。
- **`by_epoch`** (bool): 选择 **`interval`** 是基于epoch还是基于iteration， 默认为 `True`.
- **`out_dir`** (str): 保存权重文件的根目录。如果不指定，检查点将被保存在工作目录中。如果指定，检查点将被保存在 **`out_dir`** 的子文件夹中。
- **`max_keep_ckpts`** (int): 要保留的权重文件数量。在某些情况下，为了节省磁盘空间，我们希望只保留最近的几个权重文件。默认为 -1，也就是无限制。
- **`save_best`** (str, List[str]): 如果指定，它将保存具有最佳评估结果的权重。
  通常情况下，你可以直接使用`save_best="auto"`来自动选择评估指标。

而如果你想要更高级的配置，请参考[权重文件钩子(CheckpointHook)](tutorials/hook.md#checkpointhook)。

## 权重加载 / 断点训练

在配置文件中，你可以加载指定模型权重或者断点继续训练，如下所示:

```python
# 从指定权重文件加载
load_from = "Your checkpoint path"

# 是否从加载的断点继续训练
```

`load_from` 字段可以是本地路径，也可以是HTTP路径。你可以从检查点恢复训练，方法是指定 `resume=True`。

```{tip}
你也可以通过指定 `load_from=None` 和 `resume=True` 启用从最新的断点自动恢复。
Runner执行器将自动从工作目录中找到最新的权重文件。
```

如果你用我们的 `tools/train.py` 脚本来训练模型，你只需使用 `--resume` 参数来恢复训练，就不用手动修改配置文件了。如下所示:

```bash
# 自动从最新的断点恢复
python tools/train.py configs/resnet/resnet50_8xb32_in1k.py --resume

# 从指定的断点恢复
python tools/train.py configs/resnet/resnet50_8xb32_in1k.py --resume checkpoints/resnet.pth
```

## 随机性(Randomness)配置

为了让实验尽可能是可复现的， 我们在 `randomness` 字段中提供了一些控制随机性的选项。

默认情况下，我们不会在配置文件中指定随机数种子，在每次实验中，程序会生成一个不同的随机数种子。

**默认配置:**

```python
randomness = dict(seed=None, deterministic=False)
```

**默认配置:**

```python
```

为了使实验更具可复现性，你可以指定一个种子并设置 `deterministic=True`。
`deterministic` 选项的使用效果可以在[这里](https://pytorch.org/docs/stable/notes/randomness.html#cuda-convolution-benchmarking)找到。

## 日志配置

日志的配置与多个字段有关。

在`log_level`字段中，你可以指定全局日志级别。参见 {external+python:ref}`Logging Levels<levels>` 以获得日志级别列表。

```python
```

在 `default_hooks.logger` 字段中，你可以指定训练和测试期间的日志间隔。
而所有可用的参数可以在[日志钩子文档](tutorials/hook.md#loggerhook)中找到。

```python
    ...
    # 每100次迭代就打印一次日志
    ...
)
```

在 `log_processor` 字段中，你可以指定日志信息的平滑方法。
通常，我们使用一个长度为10的窗口来平滑日志中的值，并输出所有信息的平均值。
如果你想特别指定某些信息的平滑方法，请参阅{external+mmengine:doc}`日志处理器文档 <advanced_tutorials/logging>`。

```python
# 默认设置，它将通过一个10长度的窗口平滑训练日志中的值
log_processor = dict(window_size=10)
```

```python
```

在 `visualizer` 字段中，你可以指定多个后端来保存日志信息，如TensorBoard和WandB。
更多的细节可以在[可视化工具](#visualizer)找到。

## 自定义钩子

上述许多功能是由钩子实现的，你也可以通过修改 `custom_hooks` 字段来插入其他的自定义钩子。
下面是 MMEngine 和 MMPretrain 中的一些钩子，你可以直接使用，例如：

- [EMAHook](mmpretrain.engine.hooks.EMAHook)
- [SyncBuffersHook](mmengine.hooks.SyncBuffersHook)
- [EmptyCacheHook](mmengine.hooks.EmptyCacheHook)
- [ClassNumCheckHook](mmpretrain.engine.hooks.ClassNumCheckHook)
- ......

例如，EMA（Exponential Moving Average）在模型训练中被广泛使用，你可以以下方式启用它：

```python
    dict(type='EMAHook', momentum=4e-5, priority='ABOVE_NORMAL'),
]
```

## 验证可视化

验证可视化钩子是一个验证过程中默认注册的钩子。
你可以在 `default_hooks.visualization` 字段中来配置它。

默认情况下，我们禁用这个钩子，你可以通过指定 `enable=True` 来启用它。而更多的参数可以在
[可视化钩子文档](mmpretrain.engine.hooks.VisualizationHook)中找到。

```python
    ...

```python
    ...
    visualization=dict(type='VisualizationHook', enable=False),
    ...
)
```

这个钩子将在验证数据集中选择一部分图像，在每次验证过程中记录并可视化它们的预测结果。
你可以用它来观察训练期间模型在实际图像上的性能变化。

此外，如果你的验证数据集中的图像很小（\<100， 如Cifra数据集），
你可以指定 `rescale_factor` 来缩放它们，如 `rescale_factor=2.`, 将可视化的图像放大两倍。

`Visualizer` 用于记录训练和测试过程中的各种信息，包括日志、图像和标量。
默认情况下，记录的信息将被保存在工作目录下的 `vis_data` 文件夹中。

**默认配置:**

```python
    type='UniversalVisualizer',
    vis_backends=[
        dict(type='LocalVisBackend'),
    ]
)
```

通常，最有用的功能是将日志和标量如 `loss` 保存到不同的后端。
例如，要把它们保存到 TensorBoard，只需像下面这样设置：

```python
        dict(type='TensorboardVisBackend'),
    ]
)
```

或者像下面这样把它们保存到 WandB：

```python

type='UniversalVisualizer',
    ]
)
```

```python
        dict(type='WandbVisBackend'),
    ]
)
```

## 环境配置

在 `env_cfg` 字段中，你可以配置一些底层的参数，如 cuDNN、多进程和分布式通信。

**在修改这些参数之前，请确保你理解这些参数的含义。**

```python
    # 是否启用cudnn基准测试

    # 设置多进程参数

    # 设置分布式参数
)
```

<details>

```bibtex
}
```

<details>

```bibtex

| [pose_hrnet_w32](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_hrnet-w32_8xb64-210e_mpii-256x256.py) |  256x256   | 0.900 |  0.334   | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_mpii_256x256-6c4f923f_20200812.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_mpii_256x256_20200812.log.json) |
| [pose_hrnet_w48](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_hrnet-w48_8xb64-210e_mpii-256x256.py) |  256x256   | 0.901 |  0.337   | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_mpii_256x256-92cab7bd_20200812.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_mpii_256x256_20200812.log.json) |

# Training and Test

- [Training and Test](#training-and-test)
  - [Training](#training)
    - [Training with your PC](#training-with-your-pc)
    - [Training with multiple GPUs](#training-with-multiple-gpus)
    - [Training with multiple machines](#training-with-multiple-machines)
      - [Multiple machines in the same network](#multiple-machines-in-the-same-network)
      - [Multiple machines managed with slurm](#multiple-machines-managed-with-slurm)
  - [Test](#test)
    - [Test with your PC](#test-with-your-pc)
    - [Test with multiple GPUs](#test-with-multiple-gpus)
    - [Test with multiple machines](#test-with-multiple-machines)
      - [Multiple machines in the same network](#multiple-machines-in-the-same-network-1)
      - [Multiple machines managed with slurm](#multiple-machines-managed-with-slurm-1)

### Training with your PC

```shell
```

By default, MMAction2 prefers GPU to CPU. If you want to train a model on CPU, please empty `CUDA_VISIBLE_DEVICES` or set it to -1 to make GPU invisible to the program.

```bash
```
````

| `--work-dir WORK_DIR`                 | The target folder to save logs and checkpoints. Defaults to a folder with the same name of the config file under `./work_dirs`.                                     |
| `--resume [RESUME]`                   | Resume training. If a path is specified, resume from it, while if not specified, try to auto resume from the latest checkpoint.                                     |

| `--auto-scale-lr`                     | Auto scale the learning rate according to the actual batch size and the original batch size.                                                                        |
| `--seed`                              | Random seed.                                                                                                                                                        |
| `--diff-rank-seed`                    | Whether or not set different seeds for different ranks.                                                                                                             |

| `--deterministic`                     | Whether to set deterministic options for CUDNN backend.                                                                                                             |
| `--cfg-options CFG_OPTIONS`           | Override some settings in the used config, the key-value pair in xxx=yyy format will be merged into the config file. If the value to be overwritten is a list, it should be of the form of either `key="[a,b]"` or `key=a,b`. The argument also allows nested list/tuple values, e.g. `key="[(a,b),(c,d)]"`. Note that the quotation marks are necessary and that no white space is allowed. |
| `--launcher {none,pytorch,slurm,mpi}` | Options for job launcher. Defaults to `none`.                                                                                                                       |

### Training with multiple GPUs

```shell

```shell
bash tools/dist_train.sh ${CONFIG} ${GPUS} [PY_ARGS]
```

| ARGS       | Description                                                                        |
| ---------- | ---------------------------------------------------------------------------------- |
| `CONFIG`   | The path to the config file.                                                       |
| `GPUS`     | The number of GPUs to be used.                                                     |
| `[PYARGS]` | The other optional arguments of `tools/train.py`, see [here](#train-with-your-pc). |

communication port of the launcher to 29666 by the following command:

```shell
PORT=29666 bash tools/dist_train.sh ${CONFIG} ${GPUS} [PY_ARGS]
```

```shell
```

```shell
CUDA_VISIBLE_DEVICES=0,1,2,3 PORT=29500 bash tools/dist_train.sh ${CONFIG} 4 [PY_ARGS]
CUDA_VISIBLE_DEVICES=4,5,6,7 PORT=29501 bash tools/dist_train.sh ${CONFIG} 4 [PY_ARGS]
```

### Training with multiple machines

```shell
```

```shell
```

```

```shell
```

The following extra environment variables need to be specified to train or test models with multiple machines:

| ENV_VARS      | Description                                                                                           |
| ------------- | ----------------------------------------------------------------------------------------------------- |
| `NNODES`      | The total number of machines. Defaults to 1.                                                          |
| `NODE_RANK`   | The index of the local machine. Defaults to 0.                                                        |
| `PORT`        | The communication port, it should be the same in all machines. Defaults to 29500.                     |
| `MASTER_ADDR` | The IP address of the master machine, it should be the same in all machines. Defaults to `127.0.0.1`. |

If you run MMAction2 on a cluster managed with [slurm](https://slurm.schedmd.com/), you can use the script `slurm_train.sh`.

```shell
[ENV_VARS] bash tools/slurm_train.sh ${PARTITION} ${JOB_NAME} ${CONFIG} [PY_ARGS]
```

| ARGS        | Description                                                                        |
| ----------- | ---------------------------------------------------------------------------------- |
| `PARTITION` | The partition to use in your cluster.                                              |
| `JOB_NAME`  | The name of your job, you can name it as you like.                                 |

| `CONFIG`    | The path to the config file.                                                       |
| `[PYARGS]`  | The other optional arguments of `tools/train.py`, see [here](#train-with-your-pc). |

Here are the environment variables can be used to configure the slurm job.

| `GPUS`          | The number of GPUs to be used. Defaults to 8.                                                              |

## Test

```shell
```

By default, MMAction2 prefers GPU to CPU. If you want to test a model on CPU, please empty `CUDA_VISIBLE_DEVICES` or set it to -1 to make GPU invisible to the program.

```bash
```
````

```bash
```
````

| `CHECKPOINT_FILE`                     | The path to the checkpoint file (It can be a http link)                                                                                                             |

| `--work-dir WORK_DIR`                 | The directory to save the file containing evaluation metrics. Defaults to a folder with the same name of the config file under `./work_dirs`.                       |

| `--interval INTERVAL`                 | The interval of samples to visualize. Defaults to 1.                                                                                                                |
| `--wait-time WAIT_TIME`               | The display time of every window (in seconds). Defaults to 2.                                                                                                       |

```shell
bash tools/dist_test.sh ${CONFIG} ${CHECKPOINT} ${GPUS} [PY_ARGS]
```

| ARGS         | Description                                                                      |
| ------------ | -------------------------------------------------------------------------------- |
| `CONFIG`     | The path to the config file.                                                     |
| `CHECKPOINT` | The path to the checkpoint file (It can be a http link)                          |
| `GPUS`       | The number of GPUs to be used.                                                   |

| `[PYARGS]`   | The other optional arguments of `tools/test.py`, see [here](#test-with-your-pc). |

```shell
PORT=29666 bash tools/dist_test.sh ${CONFIG} ${CHECKPOINT} ${GPUS} [PY_ARGS]
```

```shell
CUDA_VISIBLE_DEVICES=0,1,2,3 PORT=29500 bash tools/dist_test.sh ${CONFIG} ${CHECKPOINT} 4 [PY_ARGS]
CUDA_VISIBLE_DEVICES=4,5,6,7 PORT=29501 bash tools/dist_test.sh ${CONFIG} ${CHECKPOINT} 4 [PY_ARGS]
```

```

```shell
NNODES=2 NODE_RANK=0 PORT=$MASTER_PORT MASTER_ADDR=$MASTER_ADDR bash tools/dist_test.sh $CONFIG $CHECKPOINT $GPUS
```

```shell
NNODES=2 NODE_RANK=1 PORT=$MASTER_PORT MASTER_ADDR=$MASTER_ADDR bash tools/dist_test.sh $CONFIG $CHECKPOINT $GPUS
```

If you run MMAction2 on a cluster managed with [slurm](https://slurm.schedmd.com/), you can use the script `slurm_test.sh`.

```shell
[ENV_VARS] bash tools/slurm_test.sh ${PARTITION} ${JOB_NAME} ${CONFIG} ${CHECKPOINT} [PY_ARGS]
```

```shell
```

| `PARTITION`  | The partition to use in your cluster.                                            |
| `JOB_NAME`   | The name of your job, you can name it as you like.                               |

## Build from source

- make sure local gcc version no less than 9, which can be conformed by `gcc --version`.
- install packages for compiling and running:
  ```shell
  pip install -r requirements.txt
  ```
- install [nccl](https://docs.nvidia.com/deeplearning/nccl/install-guide/index.html), set environment variables:
  ```shell
  export NCCL_ROOT_DIR=/path/to/nccl/build
  export NCCL_LIBRARIES=/path/to/nccl/build/lib
  ```
- install rapidjson
- install openmpi, installing from source is recommended.
  ```shell
  wget https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.5.tar.gz
  tar -xzf openmpi-*.tar.gz && cd openmpi-*
  ./configure --with-cuda
  make -j$(nproc)
  make install
  ```
- build and install lmdeploy:
  ```shell
  mkdir build && cd build
  sh ../generate.sh
  ```

# PConv (ECCV'2018)

> [Image Inpainting for Irregular Holes Using Partial Convolutions](https://arxiv.org/abs/1804.07723)

> **Task**: Inpainting

Existing deep learning based image inpainting methods use a standard convolutional network over the corrupted image, using convolutional filter responses conditioned on both valid pixels as well as the substitute values in the masked holes (typically the mean value). This often leads to artifacts such as color discrepancy and blurriness. Post-processing is usually used to reduce such artifacts, but are expensive and may fail. We propose the use of partial convolutions, where the convolution is masked and renormalized to be conditioned on only valid pixels. We further include a mechanism to automatically generate an updated mask for the next layer as part of the forward pass. Our model outperforms other methods for irregular masks. We show qualitative and quantitative comparisons with other methods to validate our approach.

 <img src="https://user-images.githubusercontent.com/12726765/144175613-1bc9ad1b-072d-4c1f-a97d-1af5be2590bd.png" width="400"/>
</div >

|                         Model                          | Mask Type | Resolution | Train Iters |    Dataset    | l1 error |  PSNR  | SSIM  | Training Resources |                              Download                              |
| :----------------------------------------------------: | :-------: | :--------: | :---------: | :-----------: | :------: | :----: | :---: | :----------------: | :----------------------------------------------------------------: |
| [PConv_Stage1](./pconv_stage1_8xb12_places-256x256.py) | free-form |  256x256   |    500k     | Places365-val |    -     |   -    |   -   |         8          |                                 -                                  |

| [PConv_Stage2](./pconv_stage2_4xb2_places-256x256.py)  | free-form |  256x256   |    500k     | Places365-val |  8.776   | 22.762 | 0.801 |         4          | [model](https://download.openmmlab.com/mmediting/inpainting/pconv/pconv_256x256_stage2_4x2_places_20200619-1ffed0e8.pth) \| [log](https://download.openmmlab.com/mmediting/inpainting/pconv/pconv_256x256_stage2_4x2_places_20200619-1ffed0e8.log.json) |
| [PConv_Stage1](./pconv_stage1_8xb1_celeba-256x256.py)  | free-form |  256x256   |    500k     |  CelebA-val   |    -     |   -    |   -   |         8          |                                 -                                  |

| [PConv_Stage2](./pconv_stage2_4xb2_celeba-256x256.py)  | free-form |  256x256   |    500k     |  CelebA-val   |  5.990   | 25.404 | 0.853 |         4          | [model](https://download.openmmlab.com/mmediting/inpainting/pconv/pconv_256x256_stage2_4x2_celeba_20200619-860f8b95.pth) \| [log](https://download.openmmlab.com/mmediting/inpainting/pconv/pconv_256x256_stage2_4x2_celeba_20200619-860f8b95.log.json) |

**Train**

<details>

```shell
# cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/partial_conv/pconv_stage2_4xb2_places-256x256.py

python tools/train.py configs/partial_conv/pconv_stage2_4xb2_places-256x256.py

./tools/dist_train.sh configs/partial_conv/pconv_stage2_4xb2_places-256x256.py 8
```

```

**Test**

<details>

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/partial_conv/pconv_stage2_4xb2_places-256x256.py https://download.openmmlab.com/mmediting/inpainting/pconv/pconv_256x256_stage2_4x2_places_20200619-1ffed0e8.pth

python tools/test.py configs/partial_conv/pconv_stage2_4xb2_places-256x256.py https://download.openmmlab.com/mmediting/inpainting/pconv/pconv_256x256_stage2_4x2_places_20200619-1ffed0e8.pth

./tools/dist_test.sh configs/partial_conv/pconv_stage2_4xb2_places-256x256.py https://download.openmmlab.com/mmediting/inpainting/pconv/pconv_256x256_stage2_4x2_places_20200619-1ffed0e8.pth 8
```

```bibtex
@inproceedings{liu2018image,
  title={Image inpainting for irregular holes using partial convolutions},
  author={Liu, Guilin and Reda, Fitsum A and Shih, Kevin J and Wang, Ting-Chun and Tao, Andrew and Catanzaro, Bryan},
  pages={85--100},
}
```

# Dataset Preparation

It is recommended to symlink the dataset root to `$MMDETECTION3D/data`.
If your folder structure is different from the following, you may need to change the corresponding paths in config files.

```
├── tools
├── data
│   │   ├── maps
│   ├── kitti
│   ├── waymo
│   ├── lyft
│   │   │   ├── v1.01-train (train_data)
│   │   │   ├── lidar (train_lidar)

│   ├── lyft
│   │   │   ├── images (train_images)
│   │   │   ├── maps (train_maps)
│   │   │   ├── v1.01-test (test_data)
│   │   │   ├── lidar (test_lidar)
│   │   │   ├── images (test_images)
│   │   │   ├── maps (test_maps)
│   ├── s3dis

│   ├── semantickitti
│   │   ├── sequences
│   │   │   ├── 00
│   │   │   │   ├── labels
│   │   │   │   ├── velodyne
│   │   │   ├── 01
│   │   │   ├── ..
│   │   │   ├── 22

```

## Download and Data Preparation

### KITTI

1. Download KITTI 3D detection data [HERE](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d). Alternatively, you
   can download the dataset from [OpenDataLab](https://opendatalab.com/) using MIM. The command scripts are the following:

```bash
# log in OpenDataLab. Note that you should register an account on [OpenDataLab](https://opendatalab.com/) before.
pip install odl
odl login
mim download mmdet3d --dataset kitti
```

odl login
```

2. Prepare KITTI data splits by running:

```bash

# Download data split
```

```

3. Generate info files by running:

```bash
```

In an environment using slurm, users may run the following command instead:

```bash
```

**Tips**:

- **Ready-made Annotations**. We have also provided kitti data annotation files generated offline [here](#summary-of-annotation-files). You could download them and place them under `data/kitti/`. However, if you want to use `ObjectSample` Augmentation in LiDAR-based detection methods, you should additionally generate groundtruth database files and annotations.

  ```bash
  python tools/create_data.py kitti --root-path ./data/kitti --out-dir ./data/kitti --extra-tag kitti --only-gt-database
  ```

```bash
  ```

### Waymo

Download Waymo open dataset V1.4.1 [HERE](https://waymo.com/open/download/) and its data split [HERE](https://drive.google.com/drive/folders/18BVuF_RYJF0NjZpt8SnfzANiakoRMf0o?usp=sharing). Then put `.tfrecord` files into corresponding folders in `data/waymo/waymo_format/` and put the data split `.txt` files into `data/waymo/kitti_format/ImageSets`. Download ground truth `.bin` file for validation set [HERE](https://console.cloud.google.com/storage/browser/waymo_open_dataset_v_1_4_1/validation/ground_truth_objects) and put it into `data/waymo/waymo_format/`. A tip is that you can use `gsutil` to download the large-scale dataset with commands. You can take this [tool](https://github.com/RalphMao/Waymo-Dataset-Tool) as an example for more details. Subsequently, prepare waymo data by running:

```bash

```bash
```

Note that:

- In case the preprocessing of Waymo dataset is slow or blocked, consider reducing the value of `--workers`. If this doesn't resolve the issue, you could set `--workers` as 0 to avoid using multiprocess.

- If your local disk does not have enough space for saving converted data, you can change the `--out-dir` to anywhere else. Just remember to create folders and prepare data there in advance and link them back to `data/waymo/kitti_format` after the data conversion.

**Tips**:

- **Ready-made Annotations**. We have provided the annotation files generated offline [here](#summary-of-annotation-files). However, the original Waymo data still needs to be converted to `kitti-format` data by yourself.

- **Waymo-mini**. If you just want to use a part of Waymo Dataset to verify some methods or debug quickly, you could use our provided [Waymo-mini](https://download.openmmlab.com/mmdetection3d/data/waymo/waymo_mini_kitti_format.tar.gz) which only contains two segments in train split and one segment in val split from the original dataset. All the images, point clouds and annotations in this compressed file have been processed offline so that you can directly download and unzip it to `data/waymo/`:

  ```bash
  ```

- **Faster evaluation**. If you want faster evaluation on Waymo, you can download the preprocessed [metainfo](https://download.openmmlab.com/mmdetection3d/data/waymo/idx2metainfo.pkl) containing `contextname` and `timestamp` to the directory `data/waymo/waymo_format/` and then modify the dataset config as the following:

  ```python

```python
      )
  ```

  Now, this trick is only used for LiDAR-based detection methods.

1. Download nuScenes V1.0 full dataset data [HERE](https://www.nuscenes.org/download). Alternatively, you
   can download the dataset from [OpenDataLab](https://opendatalab.com/) using MIM. The downloading and unzipping command scripts are the following:

```bash
odl login
mim download mmdet3d --dataset nuscenes
```

odl login
```

2. Prepare nuscenes data by running:

```bash
```

**Tips**:

- **Ready-made Annotations**. We have also provided NuScenes data annotation files generated offline [here](#summary-of-annotation-files). You could download them and place them under `data/nuscenes/`. However, if you want to use `ObjectSample` Augmentation in LiDAR-based detection methods, you should additionally generate groundtruth database files and annotations.

```bash
```

### Lyft

Download Lyft 3D detection data [HERE](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/data). Prepare Lyft data by running:

```bash

```

### Lyft

```bash
python tools/dataset_converters/lyft_data_fixer.py --version v1.01 --root-folder ./data/lyft
```

Note that we follow the original folder names for clear organization. Please rename the raw folders as shown above. Also note that the second command serves the purpose of fixing a corrupted lidar data file. Please refer to the [discussion](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/discussion/110000) for more details.

1. Download SemanticKITTI dataset [HERE](http://semantic-kitti.org/dataset.html#download) and unzip all zip files. Alternatively, you

```bash
odl login
mim download mmdet3d --dataset semantickitti
```

2. Generate info files by running:

```bash
```

**Tips**:

- **Ready-made Annotations**. We have also provided SemanticKITTI data annotation files generated offline [here](#summary-of-annotation-files). You could download them and place them under `data/semantickitti/`.

### S3DIS, ScanNet and SUN RGB-D

To prepare S3DIS data, please see its [README](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/data/s3dis/README.md).

To prepare ScanNet data, please see its [README](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/data/scannet/README.md).

To prepare SUN RGB-D data, please see its [README](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/data/sunrgbd/README.md).

**Tips**: For S3DIS, ScanNet and SUN RGB-D datasets, we have also provided data annotation files generated offline [here](#summary-of-annotation-files). You could download them and place them under `data/${DATASET}/`. However, you also need to generate point cloud files and semantic/instance masks files (if it has) by yourself.

### Customized Datasets

For using custom datasets, please refer to [Customize Datasets](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/docs/en/advanced_guides/customize_dataset.md).

### Update data infos

If you have used v1.0.0rc1-v1.0.0rc4 mmdetection3d to create data infos before, and now you want to use the newest v1.1.0 mmdetection3d, you need to update the data infos file.

```bash
```

- `--dataset` : Name of dataset.
- `--pkl-path` : Specify the data infos pkl file path.
- `--out-dir` : Output direction of the data infos pkl file.

Example:

```bash
```

### Summary of annotation files

We provide ready-made annotation files we generated offline for reference. You can directly use these files for convenice.

```

|                                                        Dataset                                                         |                                                                                                           Train annotation file                                                                                                           |                                                                                                        Val annotation file                                                                                                         |                                                    Test information file                                                     |

# Vessel Assessment and Measurement Platform for Images of the REtina

This project support **`Vessel Assessment and Measurement Platform for Images of the REtina`**, and the dataset used in this project can be downloaded from [here](https://vampire.computing.dundee.ac.uk/vesselseg.html).

In order to promote evaluation of vessel segmentation on ultra-wide field-of-view (UWFV) fluorescein angriogram (FA) frames, we make public 8 frames from two different sequences, the manually annotated images and the result of our automatic vessel segmentation algorithm.

| Dataset name                                                     | Anatomical region | Task type    | Modality               | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License                                                         |

| ---------------------------------------------------------------- | ----------------- | ------------ | ---------------------- | ------------ | --------------------- | ---------------------- | ------------ | --------------------------------------------------------------- |
| [Vampire](https://vampire.computing.dundee.ac.uk/vesselseg.html) | vessel            | segmentation | fluorescein angriogram | 2            | 8/-/-                 | yes/-/-                | 2017         | [CC-BY-NC 4.0](https://creativecommons.org/licenses/by-sa/4.0/) |

| background |     8      |   96.75    |    -     |    -     |     -     |     -     |
|   vessel   |     8      |    3.25    |    -     |    -     |     -     |     -     |

Note:

Note:

![bac](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/fluorescein_angriogram/vampire/vampire_dataset.png)

```bibtex

@inproceedings{perez2011improving,
  title={Improving vessel segmentation in ultra-wide field-of-view retinal fluorescein angiograms},
  author={Perez-Rovira, Adria and Zutis, K and Hubschman, Jean Pierre and Trucco, Emanuele},
  booktitle={2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society},
  pages={2614--2617},
}

@article{perez2011rerbee,
  title={RERBEE: robust efficient registration via bifurcations and elongated elements applied to retinal fluorescein angiogram sequences},

organization={IEEE}
}

  author={Perez-Rovira, Adria and Cabido, Raul and Trucco, Emanuele and McKenna, Stephen J and Hubschman, Jean Pierre},
  journal={IEEE Transactions on Medical Imaging},
  volume={31},
  pages={140--150},
}

```

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `vampire/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
```

- download dataset from [here](https://vampire.computing.dundee.ac.uk/vesselseg.html) and decompression data to path `'data/'`.

- run script `python ../../tools/split_seg_dataset.py` to split dataset. For the Bacteria_detection dataset, as there is no test or validation dataset, we sample 20% samples from the whole dataset as the validation dataset and 80% samples for training data and make two filename lists `train.txt` and `val.txt`. As we set the random seed as the hard code, we eliminated the randomness, the dataset split actually can be reproducible.

```none
  ├── mmseg
  │   │   │   ├── fluorescein_angriogram
  │   │   │   │   ├── vampire

```

| background |     6      |   97.48    |    2     |  94.54   |     -     |     -     |
|   vessel   |     6      |    2.52    |    2     |   5.46   |     -     |     -     |

```shell
```

```shell
```

<details>

```bibtex
}
```

<details>

```bibtex

| [mspn_50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_mspn50_8xb32-210e_coco-256x192.py) |  256x192   | 0.723 |      0.895      |      0.794      | 0.788 |      0.934      | [ckpt](https://download.openmmlab.com/mmpose/top_down/mspn/mspn50_coco_256x192-8fbfb5d0_20201123.pth) | [log](https://download.openmmlab.com/mmpose/top_down/mspn/mspn50_coco_256x192_20201123.log.json) |
| [2xmspn_50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_2xmspn50_8xb32-210e_coco-256x192.py) |  256x192   | 0.754 |      0.903      |      0.826      | 0.816 |      0.942      | [ckpt](https://download.openmmlab.com/mmpose/top_down/mspn/2xmspn50_coco_256x192-c8765a5c_20201123.pth) | [log](https://download.openmmlab.com/mmpose/top_down/mspn/2xmspn50_coco_256x192_20201123.log.json) |

| [3xmspn_50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_3xmspn50_8xb32-210e_coco-256x192.py) |  256x192   | 0.758 |      0.904      |      0.830      | 0.821 |      0.943      | [ckpt](https://download.openmmlab.com/mmpose/top_down/mspn/3xmspn50_coco_256x192-e348f18e_20201123.pth) | [log](https://download.openmmlab.com/mmpose/top_down/mspn/3xmspn50_coco_256x192_20201123.log.json) |
| [4xmspn_50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_4xmspn50_8xb32-210e_coco-256x192.py) |  256x192   | 0.765 |      0.906      |      0.835      | 0.826 |      0.943      | [ckpt](https://download.openmmlab.com/mmpose/top_down/mspn/4xmspn50_coco_256x192-7b837afb_20201123.pth) | [log](https://download.openmmlab.com/mmpose/top_down/mspn/4xmspn50_coco_256x192_20201123.log.json) |

# Albumentations: fast and flexible image augmentations

<details>
<summary align="right"><a href="https://www.mdpi.com/649002">Albumentations (Information'2020)</a></summary>

```bibtex
@article{buslaev2020albumentations,
  title={Albumentations: fast and flexible image augmentations},
  author={Buslaev, Alexander and Iglovikov, Vladimir I and Khvedchenya, Eugene and Parinov, Alex and Druzhinin, Mikhail and Kalinin, Alexandr A},
  journal={Information},
  volume={11},
  pages={125},
  publisher={Multidisciplinary Digital Publishing Institute}
}
```

# ArcFace

> [ArcFace: Additive Angular Margin Loss for Deep Face Recognition](https://arxiv.org/abs/1801.07698)

Recently, a popular line of research in face recognition is adopting margins in the well-established softmax loss function to maximize class separability. In this paper, we first introduce an Additive Angular Margin Loss (ArcFace), which not only has a clear geometric interpretation but also significantly enhances the discriminative power. Since ArcFace is susceptible to the massive label noise, we further propose sub-center ArcFace, in which each class contains K sub-centers and training samples only need to be close to any of the K positive sub-centers. Sub-center ArcFace encourages one dominant sub-class that contains the majority of clean faces and non-dominant sub-classes that include hard or noisy faces. Based on this self-propelled isolation, we boost the performance through automatically purifying raw web faces under massive real-world noise. Besides discriminative feature embedding, we also explore the inverse problem, mapping feature vectors to face images. Without training any additional generator or discriminator, the pre-trained ArcFace model can generate identity-preserved face images for both subjects inside and outside the training data only by using the network gradient and Batch Normalization (BN) priors. Extensive experiments demonstrate that ArcFace can enhance the discriminative feature embedding as well as strengthen the generative face synthesis.

<img src="https://user-images.githubusercontent.com/24734142/212606212-8ffc3cd2-dbc1-4abf-8924-22167f3f6e34.png" width="80%"/>
</div>

**Retrieve image**

```python
from mmpretrain import ImageRetrievalInferencer

inferencer = ImageRetrievalInferencer('resnet50-arcface_inshop', prototype='demo/')
predict = inferencer('demo/dog.jpg', topk=2)[0]
print(predict[0])
print(predict[1])
```

```python

model = get_model('resnet50-arcface_inshop', pretrained=True)
```

Train:

```shell
python tools/train.py configs/arcface/resnet50-arcface_8xb32_inshop.py
```

Train:

```shell
```

Test:

```shell
python tools/test.py configs/arcface/resnet50-arcface_8xb32_inshop.py https://download.openmmlab.com/mmclassification/v0/arcface/resnet50-arcface_inshop_20230202-b766fe7f.pth
```

### Image Retrieval on InShop

|           Model           |                      Pretrain                      | Params(M) | Flops(G) | Recall@1 | mAP@10 |                    Config                    |                      Download                      |
| :-----------------------: | :------------------------------------------------: | :-------: | :------: | :------: | :----: | :------------------------------------------: | :------------------------------------------------: |

| `resnet50-arcface_inshop` | [ImageNet-21k-mill](https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_3rdparty-mill_in21k_20220331-faac000b.pth) |   31.69   |  16.48   |  90.18   | 69.30  | [config](./resnet50-arcface_8xb32_inshop.py) | [model](https://download.openmmlab.com/mmclassification/v0/arcface/resnet50-arcface_inshop_20230202-b766fe7f.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/arcface/resnet50-arcface_inshop_20230202-b766fe7f.log) |

```bibtex
@inproceedings{deng2018arcface,
title={ArcFace: Additive Angular Margin Loss for Deep Face Recognition},
author={Deng, Jiankang and Guo, Jia and Niannan, Xue and Zafeiriou, Stefanos},
booktitle={CVPR},
}
```

# Build lmdeploy manylinux wheel

## Prepare docker image

To build all docker images you can use the convenient script:

```bash
./build_all_docker.sh
# Build with pushing
WITH_PUSH=true ./build_all_docker.sh
```

To build a docker image with specific cuda version or manylinux-docker version, you may use:

```bash
MANY_LINUX_VERSION=2014 GPU_ARCH_VERSION=11.8 ./build_docker.sh
```

## Build lmdeploy wheel

```bash
./build_all_wheel.sh
```

# Preparing Something-Something V2

```BibTeX
@misc{goyal2017something,
      title={The "something something" video database for learning and evaluating visual common sense},
      author={Raghav Goyal and Samira Ebrahimi Kahou and Vincent Michalski and Joanna Materzyńska and Susanne Westphal and Heuna Kim and Valentin Haenel and Ingo Fruend and Peter Yianilos and Moritz Mueller-Freitag and Florian Hoppe and Christian Thurau and Ingo Bax and Roland Memisevic},
      year={2017},
      eprint={1706.04261},
}
```

For basic dataset information, you can refer to the dataset [website](https://developer.qualcomm.com/software/ai-datasets/something-something).

MIM supports downloading from OpenDataLab and preprocessing Something-Something V2 dataset with one command line.
```Bash

```Bash
odl login
mim download mmaction2 --dataset sthv2
```

````

First of all, you have to sign in and download annotations to `$MMACTION2/data/sthv2/annotations` on the official [website](https://20bn.com/datasets/something-something/v2).
Before we start, please make sure that the directory is located at `$MMACTION2/tools/data/sthv2/`.

Then, you can download all data parts to `$MMACTION2/data/sthv2/` and use the following command to uncompress.

```shell
cd $MMACTION2/data/sthv2/
cat 20bn-something-something-v2-?? | tar zx
cd $MMACTION2/tools/data/sthv2/
```

## Step 3. Extract RGB and Flow

```

```shell
mkdir /mnt/SSD/sthv2_extracted/
ln -s /mnt/SSD/sthv2_extracted/ ../../../data/sthv2/rawframes
```

```shell
```

```shell
```

```shell
```

```shell
```

```shell
```

````
`````

After the whole data process for Something-Something V2 preparation,
you will get the rawframes (RGB + Flow), videos and annotation files for Something-Something V2.

In the context of the whole project (for Something-Something V2 only), the folder structure will look like:

```
mmaction2

```
mmaction2
├── tools
├── data
│   ├── sthv2
│   │   ├── sthv2_{train,val}_list_rawframes.txt(Optional)
│   │   ├── sthv2_{train,val}_list_videos.txt
│   │   ├── annotations(Optional)
│   |   |   ├──...
│   |   ├── rawframes(Optional)
│   |   |   ├── 1
│   |   |   ├── 2
│   |   |   ├── ...

```

For training and evaluating on Something-Something V2, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).

<details>

```bibtex
}
```

<details>

```bibtex
}
```

}
```

| [pose_hourglass_52](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_hourglass52_8xb64-210e_mpii-256x256.py) |  256x256   | 0.889 |  0.317   | [ckpt](https://download.openmmlab.com/mmpose/top_down/hourglass/hourglass52_mpii_256x256-ae358435_20200812.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hourglass/hourglass52_mpii_256x256_20200812.log.json) |

| [pose_hourglass_52](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_hourglass52_8xb32-210e_mpii-384x384.py) |  384x384   | 0.894 |  0.367   | [ckpt](https://download.openmmlab.com/mmpose/top_down/hourglass/hourglass52_mpii_384x384-04090bc3_20200812.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hourglass/hourglass52_mpii_384x384_20200812.log.json) |

<details>
<summary align="right"><a href="https://arxiv.org/abs/1711.08229">IPR (ECCV'2018)</a></summary>

```bibtex
@inproceedings{sun2018integral,
  title={Integral human pose regression},
  author={Sun, Xiao and Xiao, Bin and Wei, Fangyin and Liang, Shuang and Wei, Yichen},
  pages={529--545},
}
```

<details>

```bibtex
}
```

<details>

| [ipr_resnet_50](/configs/body_2d_keypoint/integral_regression/coco/ipr_res50_8xb64-210e_coco-256x256.py) |  256x256   | 0.633 |      0.860      |      0.703      | 0.730 |      0.919      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/integral_regression/coco/ipr_res50_8xb64-210e_coco-256x256-a3898a33_20220913.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/integral_regression/coco/ipr_res50_8xb64-210e_coco-256x256-a3898a33_20220913.log.json) |

# MaskFormer

> [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://arxiv.org/abs/2107.06278)

Modern approaches typically formulate semantic segmentation as a per-pixel classification task, while instance-level segmentation is handled with an alternative mask classification. Our key insight: mask classification is sufficiently general to solve both semantic- and instance-level segmentation tasks in a unified manner using the exact same model, loss, and training procedure. Following this observation, we propose MaskFormer, a simple mask classification model which predicts a set of binary masks, each associated with a single global class label prediction. Overall, the proposed mask classification-based method simplifies the landscape of effective approaches to semantic and panoptic  segmentation tasks and shows excellent empirical results. In particular, we observe that MaskFormer outperforms per-pixel classification baselines when the number of classes is large. Our mask classification-based method outperforms both current state-of-the-art semantic (55.6 mIoU on ADE20K) and panoptic segmentation (52.7 PQ on COCO) models.

<img src="https://camo.githubusercontent.com/29fb22298d506ce176caad3006a7b05ef2603ca12cece6c788b7e73c046e8bc9/68747470733a2f2f626f77656e63303232312e6769746875622e696f2f696d616765732f6d61736b666f726d65722e706e67" height="300"/>
</div>

MaskFormer requires COCO and [COCO-panoptic](http://images.cocodataset.org/annotations/panoptic_annotations_trainval2017.zip) dataset for training and evaluation. You need to download and extract it in the COCO dataset path.

```none
├── mmdet
├── tools
├── data
│   ├── coco
```

```

| Backbone |  style  | Lr schd | Mem (GB) | Inf time (fps) |   PQ   |   SQ   |   RQ   | PQ_th  | SQ_th  | RQ_th  | PQ_st  | SQ_st  | RQ_st  |                           Config                           |                                                                                                                                                                                        Download                                                                                                                                                                                        |

|   R-50   | pytorch |   75e   |   16.2   |       -        | 46.757 | 80.297 | 57.176 | 50.829 | 81.125 | 61.798 | 40.610 | 79.048 | 50.199 |      [config](./maskformer_r50_ms-16xb1-75e_coco.py)       |                           [model](https://download.openmmlab.com/mmdetection/v3.0/maskformer/maskformer_r50_ms-16xb1-75e_coco/maskformer_r50_ms-16xb1-75e_coco_20230116_095226-baacd858.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/maskformer/maskformer_r50_ms-16xb1-75e_coco/maskformer_r50_ms-16xb1-75e_coco_20230116_095226.log.json)                           |

|  Swin-L  | pytorch |  300e   |   27.2   |       -        | 53.249 | 81.704 | 64.231 | 58.798 | 82.923 | 70.282 | 44.874 | 79.863 | 55.097 | [config](./maskformer_swin-l-p4-w12_64xb1-ms-300e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/maskformer/maskformer_swin-l-p4-w12_64xb1-ms-300e_coco/maskformer_swin-l-p4-w12_64xb1-ms-300e_coco_20220326_221612-c63ab967.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/maskformer/maskformer_swin-l-p4-w12_mstrain_64x1_300e_coco/maskformer_swin-l-p4-w12_mstrain_64x1_300e_coco_20220326_221612.log.json) |

### Note

1. The `R-50` version was mentioned in Table XI, in paper [Masked-attention Mask Transformer for Universal Image Segmentation](https://arxiv.org/abs/2112.01527).
2. The models were trained with mmdet 2.x and have been converted for mmdet 3.x.

```latex
@inproceedings{cheng2021maskformer,
  title={Per-Pixel Classification is Not All You Need for Semantic Segmentation},

```latex
  author={Bowen Cheng and Alexander G. Schwing and Alexander Kirillov},
  journal={NeurIPS},
}
```

# 准备 UCF-101

## 简介

```BibTeX
}
```

用户可参考该数据集的 [官网](https://www.crcv.ucf.edu/research/data-sets/ucf101/)，以获取数据集相关的基本信息。
在数据集准备前，请确保命令行当前路径为 `$MMACTION2/tools/data/ucf101/`。

首先，用户可运行以下脚本下载标注文件。

```shell
```

## 步骤 2. 准备视频文件

之后，用户可运行以下脚本准备视频文件。

```shell
```

用户可使用以下脚本，对原视频进行裁剪，得到密集编码且更小尺寸的视频。

```
python ../resize_videos.py ../../../data/ucf101/videos/ ../../../data/ucf101/videos_256p_dense_cache --dense --level 2 --ext avi
```

## 步骤 3. 抽取视频帧和光流

如果拥有大量的 SSD 存储空间，则推荐将抽取的帧存储至 I/O 性能更优秀的 SSD 中。所抽取的视频帧和光流约占据 100 GB 的存储空间。

```shell
mkdir /mnt/SSD/ucf101_extracted/
ln -s /mnt/SSD/ucf101_extracted/ ../../../data/ucf101/rawframes
```

```shell
```

```shell
```

如果用户想抽取 RGB 帧和光流，则可以运行以下脚本使用 "tvl1" 算法进行抽取。

```shell
```

```shell
```

## 步骤 5. 检查文件夹结构

在完成所有 UCF-101 数据集准备流程后，

在整个 MMAction2 文件夹下，UCF-101 的文件结构如下：

```
mmaction2
├── tools
├── data
│   ├── ucf101
│   │   ├── ucf101_{train,val}_split_{1,2,3}_rawframes.txt
│   │   ├── ucf101_{train,val}_split_{1,2,3}_videos.txt

```
mmaction2
├── tools
├── data
│   │   │   ├── ApplyEyeMakeup
│   │   │   │   ├── v_ApplyEyeMakeup_g01_c01.avi

│   │   │   ├── YoYo
│   │   │   │   ├── v_YoYo_g25_c05.avi
│   │   ├── rawframes
│   │   │   │   ├── v_ApplyEyeMakeup_g01_c01
│   │   │   │   │   ├── img_00001.jpg
│   │   │   │   │   ├── img_00002.jpg
│   │   │   │   │   ├── ...
│   │   │   │   │   ├── flow_x_00001.jpg
│   │   │   │   │   ├── flow_x_00002.jpg
│   │   │   │   │   ├── flow_y_00001.jpg
│   │   │   │   │   ├── flow_y_00002.jpg
│   │   │   ├── ...
│   │   │   │   ├── v_YoYo_g01_c01
│   │   │   │   ├── v_YoYo_g25_c05

```

关于对 UCF-101 进行训练和验证，请参考 [训练和测试教程](/docs/en/user_guides/train_test.md)。

</div>

## Usage

We implement it based on [mmdetection](https://github.com/open-mmlab/mmdetection/), please refer to [mmdetection/projects/XDecoder](https://github.com/open-mmlab/mmdetection/tree/main/projects/XDecoder) for more details.

# DR HAGIS: Diabetic Retinopathy, Hypertension, Age-related macular degeneration and Glacuoma ImageS

This project supports **`DR HAGIS: Diabetic Retinopathy, Hypertension, Age-related macular degeneration and Glacuoma ImageS`**, which can be downloaded from [here](https://paperswithcode.com/dataset/dr-hagis).

The DR HAGIS database has been created to aid the development of vessel extraction algorithms suitable for retinal screening programmes. Researchers are encouraged to test their segmentation algorithms using this database. All thirty-nine fundus images were obtained from a diabetic retinopathy screening programme in the UK. Hence, all images were taken from diabetic patients.

Besides the fundus images, the manual segmentation of the retinal surface vessels is provided by an expert grader. These manually segmented images can be used as the ground truth to compare and assess the automatic vessel extraction algorithms. Masks of the FOV are provided as well to quantify the accuracy of vessel extraction within the FOV only. The images were acquired in different screening centers, therefore reflecting the range of image resolutions, digital cameras and fundus cameras used in the clinic. The fundus images were captured using a Topcon TRC-NW6s, Topcon TRC-NW8 or a Canon CR DGi fundus camera with a horizontal 45 degree field-of-view (FOV). The images are 4752x3168 pixels, 3456x2304 pixels, 3126x2136 pixels, 2896x1944 pixels or 2816x1880 pixels in size.

| Dataset name                                            | Anatomical region | Task type    | Modality           | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License |
| ------------------------------------------------------- | ----------------- | ------------ | ------------------ | ------------ | --------------------- | ---------------------- | ------------ | ------- |
| [DR HAGIS](https://paperswithcode.com/dataset/dr-hagis) | head and neck     | segmentation | fundus photography | 2            | 40/-/-                | yes/-/-                | 2017         | -       |

| background |     40     |   96.38    |    -     |    -     |     -     |     -     |
|   vessel   |     40     |    3.62    |    -     |    -     |     -     |     -     |

Note:

![bac](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/fundus_photography/dr_hagis/dr_hagis_dataset.png)

## Usage

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `dr_hagis/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
```

- download dataset from [here](https://paperswithcode.com/dataset/dr-hagis) and decompress data to path `'data/'`.

```none
  ├── mmseg
  │   │   │   │   ├── dr_hagis

│   │   │   │   │   ├── tools
```

| background |     32     |   96.21    |    8     |  97.12   |     -     |     -     |

|   vessel   |     32     |    3.79    |    8     |   2.88   |     -     |     -     |

```shell
```

```shell
```

If this work is helpful for your research, please consider citing the below paper.

```
@article{holm2017dr,
  title={DR HAGIS—a fundus image database for the automatic extraction of retinal surface vessels from diabetic patients},
  author={Holm, Sven and Russell, Greg and Nourrit, Vincent and McLoughlin, Niall},
  journal={Journal of Medical Imaging},
  volume={4},
  pages={014503--014503},
  publisher={Society of Photo-Optical Instrumentation Engineers}
}
```

# DeiT III: Revenge of the ViT

> [DeiT III: Revenge of the ViT](https://arxiv.org/abs/2204.07118)

A Vision Transformer (ViT) is a simple neural architecture amenable to serve several computer vision tasks. It has limited built-in architectural priors, in contrast to more recent architectures that incorporate priors either about the input data or of specific tasks. Recent works show that ViTs benefit from self-supervised pre-training, in particular BerT-like pre-training like BeiT. In this paper, we revisit the supervised training of ViTs. Our procedure builds upon and simplifies a recipe introduced for training ResNet-50. It includes a new simple data-augmentation procedure with only 3 augmentations, closer to the practice in self-supervised learning. Our evaluations on Image classification (ImageNet-1k with and without pre-training on ImageNet-21k), transfer learning and semantic segmentation show that our procedure outperforms by a large margin previous fully supervised training recipes for ViT. It also reveals that the performance of our ViT trained with supervision is comparable to that of more recent architectures. Our results could serve as better baselines for recent self-supervised approaches demonstrated on ViT.

<img src="https://user-images.githubusercontent.com/24734142/192964480-46726469-21d9-4e45-a06a-87c6a57c3367.png" width="90%"/>
</div>

```python

predict = inference_model('deit3-small-p16_3rdparty_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('deit3-small-p16_3rdparty_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/deit3/deit3-small-p16_64xb64_in1k.py https://download.openmmlab.com/mmclassification/v0/deit3/deit3-small-p16_3rdparty_in1k_20221008-0f7c70cf.pth
```

| Model                                             |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                     Config                     |                         Download                         |
| :------------------------------------------------ | :----------: | :--------: | :-------: | :-------: | :-------: | :--------------------------------------------: | :------------------------------------------------------: |

| `deit3-small-p16_3rdparty_in1k`\*                 | From scratch |   22.06    |   4.61    |   81.35   |   95.31   |    [config](deit3-small-p16_64xb64_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/deit3/deit3-small-p16_3rdparty_in1k_20221008-0f7c70cf.pth) |
| `deit3-small-p16_3rdparty_in1k-384px`\*           | From scratch |   22.21    |   15.52   |   83.43   |   96.68   | [config](deit3-small-p16_64xb64_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/deit3/deit3-small-p16_3rdparty_in1k-384px_20221008-a2c1a0c7.pth) |
| `deit3-small-p16_in21k-pre_3rdparty_in1k`\*       | ImageNet-21k |   22.06    |   4.61    |   83.06   |   96.77   |    [config](deit3-small-p16_64xb64_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/deit3/deit3-small-p16_in21k-pre_3rdparty_in1k_20221009-dcd90827.pth) |

| `deit3-small-p16_in21k-pre_3rdparty_in1k-384px`\* | ImageNet-21k |   22.21    |   15.52   |   84.84   |   97.48   | [config](deit3-small-p16_64xb64_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/deit3/deit3-small-p16_in21k-pre_3rdparty_in1k-384px_20221009-de116dd7.pth) |
| `deit3-medium-p16_3rdparty_in1k`\*                | From scratch |   38.85    |   8.00    |   82.99   |   96.22   |   [config](deit3-medium-p16_64xb64_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/deit3/deit3-medium-p16_3rdparty_in1k_20221008-3b21284d.pth) |
| `deit3-medium-p16_in21k-pre_3rdparty_in1k`\*      | ImageNet-21k |   38.85    |   8.00    |   84.56   |   97.19   |   [config](deit3-medium-p16_64xb64_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/deit3/deit3-medium-p16_in21k-pre_3rdparty_in1k_20221009-472f11e2.pth) |

| `deit3-base-p16_3rdparty_in1k`\*                  | From scratch |   86.59    |   17.58   |   83.80   |   96.55   |    [config](deit3-base-p16_64xb64_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/deit3/deit3-base-p16_3rdparty_in1k_20221008-60b8c8bf.pth) |
| `deit3-base-p16_3rdparty_in1k-384px`\*            | From scratch |   86.88    |   55.54   |   85.08   |   97.25   | [config](deit3-base-p16_64xb32_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/deit3/deit3-base-p16_3rdparty_in1k-384px_20221009-e19e36d4.pth) |
| `deit3-base-p16_in21k-pre_3rdparty_in1k`\*        | ImageNet-21k |   86.59    |   17.58   |   85.70   |   97.75   |    [config](deit3-base-p16_64xb64_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/deit3/deit3-base-p16_in21k-pre_3rdparty_in1k_20221009-87983ca1.pth) |

| `deit3-base-p16_in21k-pre_3rdparty_in1k-384px`\*  | ImageNet-21k |   86.88    |   55.54   |   86.73   |   98.11   | [config](deit3-base-p16_64xb32_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/deit3/deit3-base-p16_in21k-pre_3rdparty_in1k-384px_20221009-5e4e37b9.pth) |
| `deit3-large-p16_3rdparty_in1k`\*                 | From scratch |   304.37   |   61.60   |   84.87   |   97.01   |    [config](deit3-large-p16_64xb64_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/deit3/deit3-large-p16_3rdparty_in1k_20221009-03b427ea.pth) |
| `deit3-large-p16_3rdparty_in1k-384px`\*           | From scratch |   304.76   |  191.21   |   85.82   |   97.60   | [config](deit3-large-p16_64xb16_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/deit3/deit3-large-p16_3rdparty_in1k-384px_20221009-4317ce62.pth) |

| `deit3-large-p16_in21k-pre_3rdparty_in1k`\*       | ImageNet-21k |   304.37   |   61.60   |   86.97   |   98.24   |    [config](deit3-large-p16_64xb64_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/deit3/deit3-large-p16_in21k-pre_3rdparty_in1k_20221009-d8d27084.pth) |
| `deit3-large-p16_in21k-pre_3rdparty_in1k-384px`\* | ImageNet-21k |   304.76   |  191.21   |   87.73   |   98.51   | [config](deit3-large-p16_64xb16_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/deit3/deit3-large-p16_in21k-pre_3rdparty_in1k-384px_20221009-75fea03f.pth) |
| `deit3-huge-p14_3rdparty_in1k`\*                  | From scratch |   632.13   |  167.40   |   85.21   |   97.36   |    [config](deit3-huge-p14_64xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/deit3/deit3-huge-p14_3rdparty_in1k_20221009-e107bcb7.pth) |

| `deit3-huge-p14_in21k-pre_3rdparty_in1k`\*        | ImageNet-21k |   632.13   |  167.40   |   87.19   |   98.26   |    [config](deit3-huge-p14_64xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/deit3/deit3-huge-p14_in21k-pre_3rdparty_in1k_20221009-19b8a535.pth) |

*Models with * are converted from the [official repo](https://github.com/facebookresearch/deit/blob/main/models_v2.py#L171). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@article{Touvron2022DeiTIR,
  title={DeiT III: Revenge of the ViT},
  author={Hugo Touvron and Matthieu Cord and Herve Jegou},
  journal={arXiv preprint arXiv:2204.07118},
}
```

# DreamBooth (2022)

> [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://arxiv.org/abs/2208.12242)

> **Task**: Text2Image

Large text-to-image models achieved a remarkable leap in the evolution of AI, enabling high-quality and diverse synthesis of images from a given text prompt. However, these models lack the ability to mimic the appearance of subjects in a given reference set and synthesize novel renditions of them in different contexts. In this work, we present a new approach for "personalization" of text-to-image diffusion models. Given as input just a few images of a subject, we fine-tune a pretrained text-to-image model such that it learns to bind a unique identifier with that specific subject. Once the subject is embedded in the output domain of the model, the unique identifier can be used to synthesize novel photorealistic images of the subject contextualized in different scenes. By leveraging the semantic prior embedded in the model with a new autogenous class-specific prior preservation loss, our technique enables synthesizing the subject in diverse scenes, poses, views and lighting conditions that do not appear in the reference images. We apply our technique to several previously-unassailable tasks, including subject recontextualization, text-guided view synthesis, and artistic rendering, all while preserving the subject's key features. We also provide a new dataset and evaluation protocol for this new task of subject-driven generation.

<img src="https://user-images.githubusercontent.com/28132635/232406756-04dc1fbe-abde-4bb5-950c-ad3e125d5252.png">
</div>

## Configs

|                                     Model                                      | Dataset | Download |
| :----------------------------------------------------------------------------: | :-----: | :------: |
|                         [DreamBooth](./dreambooth.py)                          |    -    |    -     |
|  [DreamBooth (Finetune Text Encoder)](./dreambooth-finetune_text_encoder.py)   |    -    |    -     |
|      [DreamBooth with Prior-Preservation Loss](./dreambooth-prior_pre.py)      |    -    |    -     |
|                    [DreamBooth LoRA](./dreambooth-lora.py)                     |    -    |    -     |
| [DreamBooth LoRA with Prior-Preservation Loss](./dreambooth-lora-prior_pre.py) |    -    |    -     |

1. Download [data](https://drive.google.com/drive/folders/1BO_dyz-p65qhBRRMRA4TbZ8qW4rB99JZ) and save to `data/dreambooth/`

The file structure will be like this:

```text
data
└── dreambooth
    └──imgs
       ├── alvan-nee-Id1DBHv4fbg-unsplash.jpeg
       ├── alvan-nee-bQaAJCbNq3g-unsplash.jpeg
       ├── alvan-nee-brFsZ7qszSY-unsplash.jpeg
       └── alvan-nee-eoqnr8ikwFE-unsplash.jpeg
```

2. Start training with the following command:

```bash
bash tools/dist_train.sh configs/dreambooth/dreambooth.py 1
# or
bash tools/dist_train.sh configs/dreambooth/dreambooth-lora.py 1
```

<thead>
  <tr>
    <td>
  <img src="https://user-images.githubusercontent.com/28132635/232682088-26424e69-f697-49bc-a706-d03245ff25b1.png" width="400"/>
  <br/>
  <b>'dreambooth'</b>
    <td>

<img src="https://user-images.githubusercontent.com/28132635/232682088-26424e69-f697-49bc-a706-d03245ff25b1.png" width="400"/>
  <br/>
    <td>
  <img src="https://user-images.githubusercontent.com/28132635/232682057-fbc99047-e2d0-433e-bbc5-4f2d4ec18191.png" width="400"/>
  <br/>
  <b>'dreambooth-lora'</b>
    <td>
</thead>
</table>

## Use ToMe to accelerate your training and inference

We support **[tomesd](https://github.com/dbolya/tomesd)** now! It is developed for stable-diffusion-based models referring to [ToMe](https://github.com/facebookresearch/ToMe), an efficient ViT speed-up tool based on token merging. To work on with **tomesd** in `mmagic`, you just need to add `tomesd_cfg` to `model` in [DreamBooth](./dreambooth.py). The only requirement is `torch >= 1.12.1` in order to properly support `torch.Tensor.scatter_reduce()` functionality. Please do check it before running the demo.

```python

```python
    type='DreamBooth',
    ...
    tomesd_cfg=dict(ratio=0.5),
    ...
    val_prompts=val_prompts)
```

For more details, you can refer to [Stable Diffusion Acceleration](../stable_diffusion/README.md#use-tome-to-accelerate-your-stable-diffusion-model).

## Comments

Our codebase for the stable diffusion models builds heavily on [diffusers codebase](https://github.com/huggingface/diffusers) and the model weights are from [stable-diffusion-1.5](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_controlnet.py).

Thanks for the efforts of the community!

```bibtex
@article{ruiz2022dreambooth,
  title={Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation},
  author={Ruiz, Nataniel and Li, Yuanzhen and Jampani, Varun and Pritch, Yael and Rubinstein, Michael and Aberman, Kfir},
  journal={arXiv preprint arXiv:2208.12242},
}
```

  <img src="resources/mmdeploy-logo.png" width="450"/>
    <sup>
        <a href="https://openmmlab.com">
      </a>
    </sup>
    <sup>
      </a>
    </sup>
  </div>

[![docs](https://img.shields.io/badge/docs-latest-blue)](https://mmdeploy.readthedocs.io/en/latest/)
[![badge](https://github.com/open-mmlab/mmdeploy/workflows/build/badge.svg)](https://github.com/open-mmlab/mmdeploy/actions)
[![codecov](https://codecov.io/gh/open-mmlab/mmdeploy/branch/main/graph/badge.svg)](https://codecov.io/gh/open-mmlab/mmdeploy)

[![license](https://img.shields.io/github/license/open-mmlab/mmdeploy.svg)](https://github.com/open-mmlab/mmdeploy/tree/main/LICENSE)
[![issue resolution](https://img.shields.io/github/issues-closed-raw/open-mmlab/mmdeploy)](https://github.com/open-mmlab/mmdeploy/issues)
[![open issues](https://img.shields.io/github/issues-raw/open-mmlab/mmdeploy)](https://github.com/open-mmlab/mmdeploy/issues)

</div>

<a href="https://www.youtube.com/openmmlab" style="text-decoration:none;">
</div>

## Highlights

The MMDeploy 1.x has been released, which is adapted to upstream codebases from OpenMMLab 2.0. Please **align the version** when using it.
The default branch has been switched to `main` from `master`. MMDeploy 0.x (`master`) will be deprecated and new features will only be added to MMDeploy 1.x (`main`) in future.

| mmdeploy | mmengine |   mmcv   |  mmdet   | others |
| :------: | :------: | :------: | :------: | :----: |
|  0.x.y   |    -     | \<=1.x.y | \<=2.x.y | 0.x.y  |
|  1.x.y   |  0.x.y   |  2.x.y   |  3.x.y   | 1.x.y  |

[deploee](https://platform.openmmlab.com/deploee/) offers over 2,300 AI models in ONNX, NCNN, TRT and OpenVINO formats. Featuring a built-in list of real hardware devices, deploee enables users to convert Torch models into any target inference format for profiling purposes.

MMDeploy is an open-source deep learning model deployment toolset. It is a part of the [OpenMMLab](https://openmmlab.com/) project.

  <img src="resources/introduction.png">
</div>

## Main features

### Fully support OpenMMLab models

The currently supported codebases and models are as follows, and more will be included in the future

- [mmpretrain](docs/en/04-supported-codebases/mmpretrain.md)
- [mmdet](docs/en/04-supported-codebases/mmdet.md)
- [mmseg](docs/en/04-supported-codebases/mmseg.md)

- [mmagic](docs/en/04-supported-codebases/mmagic.md)
- [mmocr](docs/en/04-supported-codebases/mmocr.md)
- [mmpose](docs/en/04-supported-codebases/mmpose.md)
- [mmdet3d](docs/en/04-supported-codebases/mmdet3d.md)
- [mmrotate](docs/en/04-supported-codebases/mmrotate.md)
- [mmaction2](docs/en/04-supported-codebases/mmaction2.md)

### Multiple inference backends are available

The supported Device-Platform-InferenceBackend matrix is presented as following, and more will be compatible.

The benchmark can be found from [here](docs/en/03-benchmark/benchmark.md)

<div style="width: fit-content; margin: auto;">
<table>
  <tr>
    <th>Device / <br> Platform</th>
    <th>Linux</th>
    <th>Windows</th>
    <th>macOS</th>
    <th>Android</th>
  </tr>
  <tr>
    <th>x86_64 <br> CPU</th>
    <td>

<table>
  <tr>
  </tr>
  <tr>
    <td>
        <sub><a href="https://github.com/open-mmlab/mmdeploy/actions/workflows/backend-ort.yml"><img src="https://img.shields.io/github/actions/workflow/status/open-mmlab/mmdeploy/backend-ort.yml"></a></sub> <sub>onnxruntime</sub> <br>
        <sub><a href="https://github.com/open-mmlab/mmdeploy/actions/workflows/backend-pplnn.yml"><img src="https://img.shields.io/github/actions/workflow/status/open-mmlab/mmdeploy/backend-pplnn.yml"></a></sub> <sub>pplnn</sub> <br>
        <sub><a href="https://github.com/open-mmlab/mmdeploy/actions/workflows/backend-ncnn.yml"><img src="https://img.shields.io/github/actions/workflow/status/open-mmlab/mmdeploy/backend-ncnn.yml"></a></sub> <sub>ncnn</sub> <br>

<sub><a href="https://github.com/open-mmlab/mmdeploy/actions/workflows/backend-torchscript.yml"><img src="https://img.shields.io/github/actions/workflow/status/open-mmlab/mmdeploy/backend-torchscript.yml"></a></sub> <sub>LibTorch</sub> <br>
        <sub><img src="https://img.shields.io/badge/build-no%20status-lightgrey"></sub> <sub>OpenVINO</sub> <br>
        <sub><img src="https://img.shields.io/badge/build-no%20status-lightgrey"></sub> <sub>TVM</sub> <br>
    </td>
    <td>
        <sub><img src="https://img.shields.io/badge/build-no%20status-lightgrey"></sub> <sub>onnxruntime</sub> <br>
        <sub><img src="https://img.shields.io/badge/build-no%20status-lightgrey"></sub> <sub>ncnn</sub> <br>
    </td>
    <td align="center">
        -
    </td>
        -
    </td>
  </tr>

<tr>
    <th>ARM <br> CPU</th>
    <td>

</td>
        -
    </td>
        -
    </td>
  </tr>

<tr>
    <td>
        <sub><a href="https://github.com/open-mmlab/mmdeploy/actions/workflows/build.yml"><img src="https://byob.yarr.is/open-mmlab/mmdeploy/cross_build_aarch64"></a></sub> <sub>ncnn</sub> <br>
    </td>
        -
    </td>
        -
    </td>
    </td>
  </tr>

<tr>
    <th>RISC-V</th>
    <td>
        <sub><a href="https://github.com/open-mmlab/mmdeploy/actions/workflows/linux-riscv64-gcc.yml"><img src="https://img.shields.io/github/actions/workflow/status/open-mmlab/mmdeploy/linux-riscv64-gcc.yml"></a></sub> <sub>ncnn</sub> <br>
    </td>

</td>
        -
    </td>
        -
    </td>
        -
    </td>
  </tr>

<tr>
    <th>NVIDIA <br> GPU</th>
    <td>
        <sub><a href="https://github.com/open-mmlab/mmdeploy/actions/workflows/build.yml"><img src="https://byob.yarr.is/open-mmlab/mmdeploy/build_cuda113_linux"></a></sub> <sub>onnxruntime</sub> <br>
        <sub><a href="https://github.com/open-mmlab/mmdeploy/actions/workflows/build.yml"><img src="https://byob.yarr.is/open-mmlab/mmdeploy/build_cuda113_linux"></a></sub> <sub>TensorRT</sub> <br>
        <sub><img src="https://img.shields.io/badge/build-no%20status-lightgrey"></sub> <sub>LibTorch</sub> <br>
    </td>
    <td>

</td>
    <td>
        <sub><a href="https://github.com/open-mmlab/mmdeploy/actions/workflows/build.yml"><img src="https://byob.yarr.is/open-mmlab/mmdeploy/build_cuda113_windows"></a></sub> <sub>onnxruntime</sub> <br>
        <sub><a href="https://github.com/open-mmlab/mmdeploy/actions/workflows/build.yml"><img src="https://byob.yarr.is/open-mmlab/mmdeploy/build_cuda113_windows"></a></sub> <sub>TensorRT</sub> <br>
    </td>
        -
    </td>
        -
    </td>
  </tr>

<tr>
    <th>NVIDIA <br> Jetson</th>
    <td>
        <sub><img src="https://img.shields.io/badge/build-no%20status-lightgrey"></sub> <sub>TensorRT</sub> <br>
    </td>
        -
    </td>
        -
    </td>
        -
    </td>
  </tr>

<tr>
    <th>Huawei <br> ascend310</th>
    <td>

</td>
        -
    </td>
        -
    </td>
        -
    </td>
  </tr>

<tr>
    <td>
        <sub><a href="https://github.com/open-mmlab/mmdeploy/actions/workflows/backend-ascend.yml"><img src="https://img.shields.io/github/actions/workflow/status/open-mmlab/mmdeploy/backend-ascend.yml"></a></sub> <sub>CANN</sub> <br>
    </td>
        -
    </td>
        -
    </td>
        -
    </td>
  </tr>

<tr>
    <th>Rockchip</th>
    <td>
        <sub><a href="https://github.com/open-mmlab/mmdeploy/actions/workflows/backend-rknn.yml"><img src="https://img.shields.io/github/actions/workflow/status/open-mmlab/mmdeploy/backend-rknn.yml"></a></sub> <sub>RKNN</sub> <br>
    </td>
        -
    </td>
        -
    </td>
        -
    </td>

</td>
        -
    </td>
        -
    </td>
        -
    </td>
  </tr>

<tr>
    <th>Apple M1</th>
        -
    </td>
        -
    </td>
    <td>
        <sub><a href="https://github.com/open-mmlab/mmdeploy/actions/workflows/backend-coreml.yml"><img src="https://img.shields.io/github/actions/workflow/status/open-mmlab/mmdeploy/backend-coreml.yml"></a></sub> <sub>CoreML</sub> <br>
    </td>
        -
    </td>
  </tr>

<tr>
    <th>Adreno <br> GPU</th>
        -
    </td>
        -
    </td>
        -
    </td>
    <td>
        <sub><a href="https://github.com/open-mmlab/mmdeploy/actions/workflows/backend-snpe.yml"><img src="https://img.shields.io/github/actions/workflow/status/open-mmlab/mmdeploy/backend-snpe.yml"></a></sub> <sub>SNPE</sub> <br>

<sub><a href="https://github.com/open-mmlab/mmdeploy/actions/workflows/backend-ncnn.yml"><img src="https://img.shields.io/github/actions/workflow/status/open-mmlab/mmdeploy/backend-ncnn.yml"></a></sub> <sub>ncnn</sub> <br>
    </td>
  </tr>

<tr>
    <th>Hexagon <br> DSP</th>
        -
    </td>
        -
    </td>
        -
    </td>
    <td>
    </td>
  </tr>
</table>
</div>

### Efficient and scalable C/C++ SDK Framework

All kinds of modules in the SDK can be extended, such as `Transform` for image processing, `Net` for Neural Network inference, `Module` for postprocessing and so on

## [Documentation](https://mmdeploy.readthedocs.io/en/latest/)

Please read [getting_started](docs/en/get_started.md) for the basic usage of MMDeploy. We also provide tutoials about:

- [Build](docs/en/01-how-to-build/build_from_source.md)
  - [Build from Docker](docs/en/01-how-to-build/build_from_docker.md)
  - [Build from Script](docs/en/01-how-to-build/build_from_script.md)
  - [Build for Linux](docs/en/01-how-to-build/linux-x86_64.md)
  - [Build for macOS](docs/en/01-how-to-build/macos-arm64.md)
  - [Build for Win10](docs/en/01-how-to-build/windows.md)
  - [Build for Android](docs/en/01-how-to-build/android.md)
  - [Build for Jetson](docs/en/01-how-to-build/jetsons.md)
  - [Build for SNPE](docs/en/01-how-to-build/snpe.md)
  - [Cross Build for aarch64](docs/en/01-how-to-build/cross_build_ncnn_aarch64.md)
- User Guide
  - [How to convert model](docs/en/02-how-to-run/convert_model.md)
  - [How to write config](docs/en/02-how-to-run/write_config.md)

  - [How to profile model](docs/en/02-how-to-run/profile_model.md)
  - [How to quantize model](docs/en/02-how-to-run/quantize_model.md)
  - [Useful tools](docs/en/02-how-to-run/useful_tools.md)
- Developer Guide
  - [Architecture](docs/en/07-developer-guide/architecture.md)
  - [How to support new models](docs/en/07-developer-guide/support_new_model.md)
  - [How to support new backends](docs/en/07-developer-guide/support_new_backend.md)
  - [How to partition model](docs/en/07-developer-guide/partition_model.md)
  - [How to test rewritten model](docs/en/07-developer-guide/test_rewritten_models.md)
  - [How to test backend ops](docs/en/07-developer-guide/add_backend_ops_unittest.md)
  - [How to do regression test](docs/en/07-developer-guide/regression_test.md)
- Custom Backend Ops
  - [ncnn](docs/en/06-custom-ops/ncnn.md)

- [How to do regression test](docs/en/07-developer-guide/regression_test.md)
  - [ONNXRuntime](docs/en/06-custom-ops/onnxruntime.md)
  - [tensorrt](docs/en/06-custom-ops/tensorrt.md)
- [FAQ](docs/en/faq.md)
- [Contributing](.github/CONTRIBUTING.md)

## Benchmark and Model zoo

You can find the supported models from [here](docs/en/03-benchmark/supported_models.md) and their performance in the [benchmark](docs/en/03-benchmark/benchmark.md).

We appreciate all contributions to MMDeploy. Please refer to [CONTRIBUTING.md](.github/CONTRIBUTING.md) for the contributing guideline.

We would like to sincerely thank the following teams for their contributions to [MMDeploy](https://github.com/open-mmlab/mmdeploy):

- [OpenPPL](https://github.com/openppl-public)
- [OpenVINO](https://github.com/openvinotoolkit/openvino)
- [ncnn](https://github.com/Tencent/ncnn)

If you find this project useful in your research, please consider citing:

```BibTeX
@misc{=mmdeploy,
    title={OpenMMLab's Model Deployment Toolbox.},
    author={MMDeploy Contributors},
    howpublished = {\url{https://github.com/open-mmlab/mmdeploy}},
}
```

## License

- [MMPretrain](https://github.com/open-mmlab/mmpretrain): OpenMMLab pre-training toolbox and benchmark.

- [MMYOLO](https://github.com/open-mmlab/mmyolo): OpenMMLab YOLO series toolbox and benchmark

### input data and extract output
```cpp
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include "net.h"

int main()
{
    cv::Mat img = cv::imread("image.ppm", CV_LOAD_IMAGE_GRAYSCALE);
    int w = img.cols;
    int h = img.rows;

    // subtract 128, norm to -1 ~ 1
    ncnn::Mat in = ncnn::Mat::from_pixels_resize(img.data, ncnn::Mat::PIXEL_GRAY, w, h, 60, 60);
    float mean[1] = { 128.f };
    float norm[1] = { 1/128.f };
    in.substract_mean_normalize(mean, norm);

    ncnn::Net net;
    net.load_param("model.param");
    net.load_model("model.bin");

    ncnn::Extractor ex = net.create_extractor();
    ex.set_light_mode(true);
    ex.set_num_threads(4);

    ex.input("data", in);

    ncnn::Mat feat;
    ex.extract("output", feat);

    return 0;
}

```

### print Mat content
```cpp
void pretty_print(const ncnn::Mat& m)
{
    for (int q=0; q<m.c; q++)
    {
        const float* ptr = m.channel(q);
        for (int z=0; z<m.d; z++)
        {

}

```

```cpp
{
    {
        {
            for (int y=0; y<m.h; y++)
            {
                for (int x=0; x<m.w; x++)
                {
                    printf("%f ", ptr[x]);
                }
                ptr += m.w;
                printf("\n");
            }
            printf("\n");
        }
        printf("------------------------\n");
    }
}
```

### visualize Mat content
```cpp
void visualize(const char* title, const ncnn::Mat& m)
{
    std::vector<cv::Mat> normed_feats(m.c);

    for (int i=0; i<m.c; i++)
    {
        cv::Mat tmp(m.h, m.w, CV_32FC1, (void*)(const float*)m.channel(i));

        cv::normalize(tmp, normed_feats[i], 0, 255, cv::NORM_MINMAX, CV_8U);

        cv::cvtColor(normed_feats[i], normed_feats[i], cv::COLOR_GRAY2BGR);

        // check NaN
        for (int y=0; y<m.h; y++)

cv::cvtColor(normed_feats[i], normed_feats[i], cv::COLOR_GRAY2BGR);

        {
            const float* tp = tmp.ptr<float>(y);
            uchar* sp = normed_feats[i].ptr<uchar>(y);
            for (int x=0; x<m.w; x++)
            {
                float v = tp[x];
                if (v != v)
                {
                    sp[0] = 0;
                    sp[1] = 0;
                    sp[2] = 255;
                }

                sp += 3;
            }
        }
    }

    int tw = m.w < 10 ? 32 : m.w < 20 ? 16 : m.w < 40 ? 8 : m.w < 80 ? 4 : m.w < 160 ? 2 : 1;
    int th = (m.c - 1) / tw + 1;

    cv::Mat show_map(m.h * th, m.w * tw, CV_8UC3);
    show_map = cv::Scalar(127);

    // tile
    {
        int ty = i / tw;
        int tx = i % tw;

        normed_feats[i].copyTo(show_map(cv::Rect(tx * m.w, ty * m.h, m.w, m.h)));
    }

// tile
    {

    }

    cv::resize(show_map, show_map, cv::Size(0,0), 2, 2, cv::INTER_NEAREST);
    cv::imshow(title, show_map);
}
```

### FAQ
Q ncnn的起源

A 深度学习算法要在手机上落地，caffe依赖太多，手机上也没有cuda，需要个又快又小的前向网络实现

Q ncnn名字的来历

A cnn就是卷积神经网络的缩写，开头的n算是一语n关。比如new/next(全新的实现)，naive(ncnn是naive实现)，neon(ncnn最初为手机优化)，up主名字(←_←)

Q 支持哪些平台

A 跨平台，支持 android / ios / linux / windows / macos，也支持裸机跑

Q 计算精度如何

A armv7 neon float 不遵照 ieee754 标准，有些采用快速实现(如exp sin等)，速度快但确保精度足够高

Q logo

A up主是mc玩家，所以灵魂手绘像素猫，还可以找到ncnn...

# LeNet

> [Backpropagation Applied to Handwritten Zip Code Recognition](https://ieeexplore.ieee.org/document/6795724)

The ability of learning networks to generalize can be greatly enhanced by providing constraints from the task domain. This paper demonstrates how such constraints can be integrated into a backpropagation network through the architecture of the network. This approach has been successfully applied to the recognition of handwritten zip code digits provided by the U.S. Postal Service. A single network learns the entire recognition operation, going from the normalized image of the character to the final classification.

<img src="https://user-images.githubusercontent.com/26739999/142561080-cd1c4bdc-8739-46ca-bc32-76d462a32901.png" width="50%"/>
</div>

```
@ARTICLE{6795724,
  author={Y. {LeCun} and B. {Boser} and J. S. {Denker} and D. {Henderson} and R. E. {Howard} and W. {Hubbard} and L. D. {Jackel}},

</div>

```
  journal={Neural Computation},
  title={Backpropagation Applied to Handwritten Zip Code Recognition},
  year={1989},
  volume={1},
  pages={541-551},
  doi={10.1162/neco.1989.1.4.541}}
}
```

# Controlnet Animation (2023)

> [Controlnet](https://github.com/lllyasviel/ControlNet) Application

> **Task**: controlnet_animation

It is difficult to avoid video frame flickering when using stable diffusion to generate video frame by frame.
Here we reproduce a method that effectively avoids video flickering, that is, using controlnet and multi-frame rendering.
[ControlNet](https://github.com/lllyasviel/ControlNet) is a neural network structure to control diffusion models by adding extra conditions.
[Multi-frame rendering](https://xanthius.itch.io/multi-frame-rendering-for-stablediffusion) is a community method to reduce flickering.
We use controlnet with hed condition and stable diffusion img2img for multi-frame rendering.

## Demos

prompt key words: a handsome man, silver hair, smiling, play basketball

## Demos

  <video src="https://user-images.githubusercontent.com/12782558/227149757-fd054d32-554f-45d5-9f09-319184866d85.mp4" width=1024/>
</div>

prompt key words: a handsome man

  <video src="https://user-images.githubusercontent.com/12782558/227152129-d70d5f76-a6fc-4d23-97d1-a94abd08f95a.mp4" width=1024/>
</div>

&#8195;

**Change prompt to get different result**

prompt key words: a girl, black hair, white pants, smiling, play basketball

  <video src="https://user-images.githubusercontent.com/12782558/227216038-38599164-2384-4a79-b65e-f98785d466bf.mp4" width=512/>
</div>

## Pretrained models

We use pretrained model from hugging face.

</div>

|                    Model                    | Dataset |                                     Download                                      |
| :-----------------------------------------: | :-----: | :-------------------------------------------------------------------------------: |
| [anythingv3 config](./anythingv3_config.py) |    -    | [stable diffusion model](https://huggingface.co/Linaqruf/anything-v3.0/tree/main) |

There are two ways to try controlnet animation.

### 1. Use MMagic inference API.

Running the following codes, you can get an generated animation video.

```python

# Create a MMEdit instance and infer
editor = MMagicInferencer(model_name='controlnet_animation')

prompt = 'a girl, black hair, T-shirt, smoking, best quality, extremely detailed'
negative_prompt = 'longbody, lowres, bad anatomy, bad hands, missing fingers, ' + \

                  'extra digit, fewer digits, cropped, worst quality, low quality'

# you can download the example video with this link
# https://user-images.githubusercontent.com/12782558/227418400-80ad9123-7f8e-4c1a-8e19-0892ebad2a4f.mp4
video = '/path/to/your/input/video.mp4'
save_path = '/path/to/your/output/video.mp4'

# Do the inference to get result
editor.infer(video=video, prompt=prompt, negative_prompt=negative_prompt, save_path=save_path)
```

### 2. Use controlnet animation gradio demo.

```python
python demo/gradio_controlnet_animation.py
```

## Play animation with SAM

We also provide a demo to play controlnet animation with sam, for details, please see [OpenMMLab PlayGround](https://github.com/open-mmlab/playground/blob/main/mmediting_sam/README.md).

```bibtex
@misc{zhang2023adding,

```bibtex
  title={Adding Conditional Control to Text-to-Image Diffusion Models},
  author={Lvmin Zhang and Maneesh Agrawala},
  year={2023},
  eprint={2302.05543},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}
```

**Text Detection, Text Recognition and Text Spotting**

```text
# x1,y1,x2,y2,x3,y3,x4,y4,trans

72,25,326,25,326,64,72,64,TAN WOON YANN
50,82,440,82,440,121,50,121,BOOK TA .K(TAMAN DAYA) SDN BND
205,121,285,121,285,139,205,139,789417-W
```

<details>
<summary align="right"><a href="https://arxiv.org/abs/2105.10154">ViPNAS (CVPR'2021)</a></summary>

```bibtex
@article{xu2021vipnas,
  title={ViPNAS: Efficient Video Pose Estimation via Neural Architecture Search},
  author={Xu, Lumin and Guan, Yingda and Jin, Sheng and Liu, Wentao and Qian, Chen and Luo, Ping and Ouyang, Wanli and Wang, Xiaogang},
}
```

<details>

```bibtex

| [S-ViPNAS-MobileNetV3](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_vipnas-mbv3_8xb64-210e_coco-256x192.py) |  256x192   | 0.700 |      0.887      |      0.783      | 0.758 |      0.929      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_vipnas-mbv3_8xb64-210e_coco-256x192-e0987441_20221010.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_vipnas-mbv3_8xb64-210e_coco-256x192_20221010.log) |
| [S-ViPNAS-Res50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_vipnas-res50_8xb64-210e_coco-256x192.py) |  256x192   | 0.711 |      0.894      |      0.787      | 0.769 |      0.934      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_vipnas-res50_8xb64-210e_coco-256x192-35d4bff9_20220917.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_vipnas-res50_8xb64-210e_coco-256x192_20220917.log) |

<details>

```bibtex
}
```

<details>

```bibtex
}
```

}
```

| [pose_hourglass_52](/configs/face_2d_keypoint/topdown_heatmap/coco_wholebody_face/td-hm_hourglass52_8xb32-60e_coco-wholebody-face-256x256.py) |  256x256   | 0.0587 | [ckpt](https://download.openmmlab.com/mmpose/face/hourglass/hourglass52_coco_wholebody_face_256x256-6994cf2e_20210909.pth) | [log](https://download.openmmlab.com/mmpose/face/hourglass/hourglass52_coco_wholebody_face_256x256_20210909.log.json) |

# 常见问题

我们在这里列出了一些常见问题及其相应的解决方案。如果您发现任何常见问题并有方法
帮助解决，欢迎随时丰富列表。如果这里的内容没有涵盖您的问题，请按照
[提问模板](https://github.com/open-mmlab/mmpretrain/issues/new/choose)
在 GitHub 上提出问题，并补充模板中需要的信息。

## 安装

- MMEngine, MMCV 与 MMPretrain 的兼容问题

  这里我们列举了各版本 MMPretrain 对 MMEngine 和 MMCV 版本的依赖，请选择合适的 MMEngine 和 MMCV 版本来避免安装和使用中的问题。

  | MMPretrain 版本 |   MMEngine 版本   |    MMCV 版本     |
  | :-------------: | :---------------: | :--------------: |
  |  1.0.1 (main)   | mmengine >= 0.8.3 |  mmcv >= 2.0.0   |
  |      1.0.0      | mmengine >= 0.8.0 |  mmcv >= 2.0.0   |
  |    1.0.0rc8     | mmengine >= 0.7.1 | mmcv >= 2.0.0rc4 |
  |    1.0.0rc7     | mmengine >= 0.5.0 | mmcv >= 2.0.0rc4 |

  ```{note}
  由于 `dev` 分支处于频繁开发中，MMEngine 和 MMCV 版本依赖可能不准确。如果您在使用
  `dev` 分支时遇到问题，请尝试更新 MMEngine 和 MMCV 到最新版。
  ```

- 使用 Albumentations

  如果你希望使用 `albumentations` 相关的功能，我们建议使用 `pip install -r requirements/optional.txt` 或者
  `pip install -U albumentations>=0.3.2 --no-binary qudida,albumentations` 命令进行安装。

```

  如果你直接使用 `pip install albumentations>=0.3.2` 来安装，它会同时安装 `opencv-python-headless`
  （即使你已经安装了 `opencv-python`）。具体细节可参阅
  [官方文档](https://albumentations.ai/docs/getting_started/installation/#note-on-opencv-dependencies)。

## 通用问题

### 如果我对源码进行了改动，需要重新安装以使改动生效吗？

如果你遵照[最佳实践](../get_started.md#最佳实践)的指引，从源码安装 mmpretrain，那么任何本地修改都不需要重新安装即可生效。

### 如何在多个 MMPretrain 版本下进行开发？

通常来说，我们推荐通过不同虚拟环境来管理多个开发目录下的 MMPretrain。
但如果你希望在不同目录（如 mmpretrain-0.21, mmpretrain-0.23 等）使用同一个环境进行开发，
我们提供的训练和测试 shell 脚本会自动使用当前目录的 mmpretrain，其他 Python 脚本
则可以在命令前添加 `` PYTHONPATH=`pwd`  `` 来使用当前目录的代码。

反过来，如果你希望 shell 脚本使用环境中安装的 MMPretrain，而不是当前目录的，
则可以去掉 shell 脚本中如下一行代码：

```shell
```

### `load_from` 和 `init_cfg` 之间的关系是什么？

- `load_from`: 如果`resume=False`，只导入模型权重，主要用于加载训练过的模型；

```shell
```

  如果 `resume=True` ，加载所有的模型权重、优化器状态和其他训练信息，主要用于恢复中断的训练。

- `init_cfg`: 你也可以指定`init=dict(type="Pretrained", checkpoint=xxx)`来加载权重，
  表示在模型权重初始化时加载权重，通常在训练的开始阶段执行。
  主要用于微调预训练模型，你可以在骨干网络的配置中配置它，还可以使用 `prefix` 字段来只加载对应的权重，例如：

  ```python
        depth=50,
        init_cfg=dict(type='Pretrained', checkpoints=xxx, prefix='backbone'),
    )
    ...
  )
  ```

参见 [微调模型](./finetune_custom_dataset.md) 以了解更多关于模型微调的细节。

### `default_hooks` 和 `custom_hooks` 之间有什么区别？

几乎没有区别。通常，`default_hooks` 字段用于指定几乎所有实验都会使用的钩子，
而 `custom_hooks` 字段指部分实验特有的钩子。

另一个区别是 `default_hooks` 是一个字典，而 `custom_hooks` 是一个列表，请不要混淆。

### 在训练期间，我没有收到训练日志，这是什么原因？

如果你的训练数据集很小，而批处理量却很大，我们默认的日志间隔可能太大，无法记录你的训练日志。

你可以缩减日志间隔，再试一次，比如:

```python
    ...

```python
    ...
    logger=dict(type='LoggerHook', interval=10),
    ...
)
```

### 如何基于其它数据集训练，例如我自己的数据集或者是 COCO 数据集？

我们提供了 [具体示例](./pretrain_custom_dataset.md) 来展示如何在其它数据集上进行训练。

# How to design your own models

MMagic is built upon MMEngine and MMCV, which enables users to design new models quickly, train and evaluate them easily.
In this section, you will learn how to design your own models.

- [How to design your own models](#how-to-design-your-own-models)
  - [Overview of models in MMagic](#overview-of-models-in-mmagic)
  - [An example of SRCNN](#an-example-of-srcnn)
    - [Step 1: Define the network of SRCNN](#step-1-define-the-network-of-srcnn)
    - [Step 2: Define the model of SRCNN](#step-2-define-the-model-of-srcnn)
    - [Step 3: Start training SRCNN](#step-3-start-training-srcnn)
  - [An example of DCGAN](#an-example-of-dcgan)
    - [Step 1: Define the network of DCGAN](#step-1-define-the-network-of-dcgan)
    - [Step 2: Design the model of DCGAN](#step-2-design-the-model-of-dcgan)
    - [Step 3: Start training DCGAN](#step-3-start-training-dcgan)
  - [References](#references)

- [Step 2: Design the model of DCGAN](#step-2-design-the-model-of-dcgan)

## Overview of models in MMagic

In MMagic, one algorithm can be splited two compents: **Model** and **Module**.

- **Model** are topmost wrappers and always inherint from `BaseModel` provided in MMEngine. **Model** is responsible to network forward, loss calculation and backward, parameters updating, etc. In MMagic, **Model** should be registered as `MODELS`.
- **Module** includes the neural network **architectures** to train or inference, pre-defined **loss classes**, and **data preprocessors** to preprocess the input data batch. **Module** always present as elements of **Model**. In MMagic, **Module** should be registered as **MODULES**.

Take DCGAN model as an example, [generator](https://github.com/open-mmlab/mmagic/blob/main/mmagic/models/editors/dcgan/dcgan_generator.py) and [discriminator](https://github.com/open-mmlab/mmagic/blob/main/mmagic/models/editors/dcgan/dcgan_discriminator.py) are the **Module**, which generate images and discriminate real or fake images. [`DCGAN`](https://github.com/open-mmlab/mmagic/blob/main/mmagic/models/editors/dcgan/dcgan.py) is the **Model**, which take data from dataloader and train generator and discriminator alternatively.

You can find the implementation of **Model** and **Module** by the following link.

- **Model**:
  - [Editors](https://github.com/open-mmlab/mmagic/tree/main/mmagic/models/editors)
- **Module**:
  - [Layers](https://github.com/open-mmlab/mmagic/tree/main/mmagic/models/layers)
  - [Losses](https://github.com/open-mmlab/mmagic/tree/main/mmagic/models/losses)

  - [Data Preprocessor](https://github.com/open-mmlab/mmagic/tree/main/mmagic/models/data_preprocessors)

## An example of SRCNN

Here, we take the implementation of the classical image super-resolution model, SRCNN \[1\], as an example.

### Step 1: Define the network of SRCNN

SRCNN is the first deep learning method for single image super-resolution \[1\].
To implement the network architecture of SRCNN,
we need to create a new file `mmagic/models/editors/srgan/sr_resnet.py` and implement `class MSRResNet`.

In this step, we implement `class MSRResNet` by inheriting from `mmengine.models.BaseModule` and define the network architecture in `__init__` function.
In particular, we need to use `@MODELS.register_module()` to add the implementation of `class MSRResNet` into the registration of MMagic.

```python

```python
from mmagic.registry import MODELS

from mmagic.models.utils import (PixelShufflePack, ResidualBlockNoBN,
                                 default_init_weights, make_layer)

class MSRResNet(BaseModule):
    """Modified SRResNet.

    A compacted version modified from SRResNet in "Photo-Realistic Single
    Image Super-Resolution Using a Generative Adversarial Network".

    It uses residual blocks without BN, similar to EDSR.
    Currently, it supports x2, x3 and x4 upsampling scale factor.

    Args:
        in_channels (int): Channel number of inputs.
        out_channels (int): Channel number of outputs.
        mid_channels (int): Channel number of intermediate features.
            Default: 64.

out_channels (int): Channel number of outputs.
        num_blocks (int): Block number in the trunk network. Default: 16.
        upscale_factor (int): Upsampling factor. Support x2, x3 and x4.
            Default: 4.
    """
    _supported_upscale_factors = [2, 3, 4]

                 in_channels,
                 out_channels,
                 mid_channels=64,
                 num_blocks=16,
                 upscale_factor=4):

        self.out_channels = out_channels
        self.mid_channels = mid_channels
        self.num_blocks = num_blocks

        self.conv_first = nn.Conv2d(
            in_channels, mid_channels, 3, 1, 1, bias=True)
        self.trunk_net = make_layer(
            ResidualBlockNoBN, num_blocks, mid_channels=mid_channels)

in_channels, mid_channels, 3, 1, 1, bias=True)

        # upsampling
        if self.upscale_factor in [2, 3]:
            self.upsample1 = PixelShufflePack(
                mid_channels,
                self.upscale_factor,
                upsample_kernel=3)
        elif self.upscale_factor == 4:
                mid_channels, mid_channels, 2, upsample_kernel=3)
            self.upsample2 = PixelShufflePack(
        else:
            raise ValueError(
                f'Unsupported scale factor {self.upscale_factor}. '
                f'Currently supported ones are '
                f'{self._supported_upscale_factors}.')

        self.conv_hr = nn.Conv2d(
            mid_channels, mid_channels, 3, 1, 1, bias=True)

f'Currently supported ones are '

        self.conv_last = nn.Conv2d(
            mid_channels, out_channels, 3, 1, 1, bias=True)

            mode='bilinear',

        # activation function
        self.lrelu = nn.LeakyReLU(negative_slope=0.1, inplace=True)

        self.init_weights()

        """Init weights for models.

        Args:
            pretrained (str, optional): Path for pretrained weights. If given
                None, pretrained weights will not be loaded. Defaults to None.
            strict (boo, optional): Whether strictly load the pretrained model.
                Defaults to True.
        """

strict (boo, optional): Whether strictly load the pretrained model.
        """

        for m in [self.conv_first, self.conv_hr, self.conv_last]:
            default_init_weights(m, 0.1)

```

Then, we implement the `forward` function of  `class MSRResNet`, which takes as input tensor and then returns the results from `MSRResNet`.

```python
        """Forward function.

        Args:
            x (Tensor): Input tensor with shape (n, c, h, w).

        Returns:
            Tensor: Forward results.
        """

        feat = self.lrelu(self.conv_first(x))
        out = self.trunk_net(feat)

            out = self.upsample1(out)
            out = self.upsample2(out)

        out = self.conv_last(self.lrelu(self.conv_hr(out)))
        upsampled_img = self.img_upsampler(x)

out = self.upsample1(out)

        out += upsampled_img
        return out
```

After the implementation of `class MSRResNet`, we need to update the model list in `mmagic/models/editors/__init__.py`, so that we can import and use `class MSRResNet` by `mmagic.models.editors`.

```python
from .srgan.sr_resnet import MSRResNet
```

### Step 2: Define the model of SRCNN

After the implementation of the network architecture,
we need to define our model `class BaseEditModel` and implement the forward loop of `class BaseEditModel`.

To implement `class BaseEditModel`,
we create a new file `mmagic/models/base_models/base_edit_model.py`.
Specifically, `class BaseEditModel` inherits from `mmengine.model.BaseModel`.
In the `__init__` function, we define the loss functions, training and testing configurations, networks of `class BaseEditModel`.

```python
from typing import List, Optional

from mmengine.model import BaseModel

from mmagic.structures import DataSample

class BaseEditModel(BaseModel):
    """Base model for image and video editing.

    It must contain a generator that takes frames as inputs and outputs an
    interpolated frame. It also has a pixel-wise loss for training.

    Args:
        generator (dict): Config for the generator structure.
        pixel_loss (dict): Config for pixel-wise loss.
        train_cfg (dict): Config for training. Default: None.
        test_cfg (dict): Config for testing. Default: None.
        init_cfg (dict, optional): The weight initialized config for
            :class:`BaseModule`.
        data_preprocessor (dict, optional): The pre-process config of

init_cfg (dict, optional): The weight initialized config for
            :class:`BaseDataPreprocessor`.

    Attributes:
        init_cfg (dict, optional): Initialization config dict.
        data_preprocessor (:obj:`BaseDataPreprocessor`): Used for
            pre-processing data sampled by dataloader to the format accepted by
            :meth:`forward`. Default: None.
    """

                 generator,
                 pixel_loss,
                 init_cfg=None,
                 data_preprocessor=None):
            init_cfg=init_cfg, data_preprocessor=data_preprocessor)

        self.train_cfg = train_cfg
        self.test_cfg = test_cfg

        # generator
        self.generator = MODELS.build(generator)

        # loss

self.train_cfg = train_cfg

        # loss
        self.pixel_loss = MODELS.build(pixel_loss)
```

Since `mmengine.model.BaseModel` provides the basic functions of the algorithmic model,
such as weights initialize, batch inputs preprocess, parse losses, and update model parameters.
Therefore, the subclasses inherit from BaseModel, i.e., `class BaseEditModel` in this example,
only need to implement the forward method,
which implements the logic to calculate loss and predictions.

Specifically, the implemented `forward` function of `class BaseEditModel` takes as input `batch_inputs` and `data_samples` and return results according to mode arguments.

```python
                batch_inputs: torch.Tensor,
                data_samples: Optional[List[DataSample]] = None,
                mode: str = 'tensor',
                **kwargs):

        """Returns losses or predictions of training, validation, testing, and
        simple inference process.

        ``forward`` method of BaseModel is an abstract method, its subclasses
        must implement this method.

        Accepts ``batch_inputs`` and ``data_samples`` processed by
        :attr:`data_preprocessor`, and returns results according to mode
        arguments.

        During non-distributed training, validation, and testing process,
        ``forward`` will be called by ``BaseModel.train_step``,
        ``BaseModel.val_step`` and ``BaseModel.val_step`` directly.

        During distributed data parallel training process,
        ``MMSeparateDistributedDataParallel.train_step`` will first call
        ``DistributedDataParallel.forward`` to enable automatic

During distributed data parallel training process,
        gradient synchronization, and then call ``forward`` to get training
        loss.

        Args:
            batch_inputs (torch.Tensor): batch input tensor collated by
                :attr:`data_preprocessor`.
            data_samples (List[BaseDataElement], optional):
                data samples collated by :attr:`data_preprocessor`.
            mode (str): mode should be one of ``loss``, ``predict`` and
                ``tensor``

                - ``loss``: Called by ``train_step`` and return loss ``dict``
                  used for logging
                - ``predict``: Called by ``val_step`` and ``test_step``
                  and return list of ``BaseDataElement`` results used for
                  computing metric.

- ``predict``: Called by ``val_step`` and ``test_step``
                - ``tensor``: Called by custom use to get ``Tensor`` type
                  results.

        Returns:
            ForwardResults:

                - If ``mode == loss``, return a ``dict`` of loss tensor used
                  for backward and logging.
                - If ``mode == predict``, return a ``list`` of
                  :obj:`BaseDataElement` for computing metric
                  and getting inference result.
                - If ``mode == tensor``, return a tensor or ``tuple`` of tensor
                  or ``dict or tensor for custom use.
        """

            return self.forward_tensor(batch_inputs, data_samples, **kwargs)

            return self.forward_inference(batch_inputs, data_samples, **kwargs)

return self.forward_tensor(batch_inputs, data_samples, **kwargs)

            return self.forward_train(batch_inputs, data_samples, **kwargs)
```

Specifically, in `forward_tensor`, `class BaseEditModel` returns the forward tensors of the network directly.

```python
    def forward_tensor(self, batch_inputs, data_samples=None, **kwargs):
        """Forward tensor.
            Returns result of simple forward.

        Args:

        Returns:
            Tensor: result of simple forward.
        """

        feats = self.generator(batch_inputs, **kwargs)

        return feats
```

Returns:
        """

```

In `forward_inference` function, `class BaseEditModel` first converts the forward tensors to images and then returns the images as output.

```python
    def forward_inference(self, batch_inputs, data_samples=None, **kwargs):
        """Forward inference.
            Returns predictions of validation, testing, and simple inference.

        Args:

        Returns:
            List[DataSample]: predictions.
        """

        feats = self.forward_tensor(batch_inputs, data_samples, **kwargs)
        feats = self.data_preprocessor.destructor(feats)
        predictions = []

"""

        for idx in range(feats.shape[0]):
            predictions.append(
                DataSample(
                    pred_img=feats[idx].to('cpu'),
                    metainfo=data_samples[idx].metainfo))

```

In `forward_train`, `class BaseEditModel` calculate the loss function and returns a dictionary contains the losses as output.

```python
    def forward_train(self, batch_inputs, data_samples=None, **kwargs):
        """Forward training.
            Returns dict of losses of training.

        Args:

        Returns:

:attr:`data_preprocessor`.

        Returns:
            dict: Dict of losses.
        """

        gt_imgs = [data_sample.gt_img.data for data_sample in data_samples]
        batch_gt_data = torch.stack(gt_imgs)

        loss = self.pixel_loss(feats, batch_gt_data)

        return dict(loss=loss)

```

After the implementation of `class BaseEditModel`,
we need to update the model list in `mmagic/models/__init__.py`,
so that we can import and use `class BaseEditModel` by `mmagic.models`.

```python
from .base_models.base_edit_model import BaseEditModel
```

### Step 3: Start training SRCNN

After implementing the network architecture and the forward loop of SRCNN,
now we can create a new file `configs/srcnn/srcnn_x4k915_g1_1000k_div2k.py`

```

to set the configurations needed by training SRCNN.

In the configuration file, we need to specify the parameters of our model, `class BaseEditModel`, including the generator network architecture, loss function, additional training and testing configuration, and data preprocessor of input tensors. Please refer to the [Introduction to the loss in MMagic](./losses.md) for more details of losses in MMagic.

```python
    type='BaseEditModel',
        type='SRCNNNet',
        channels=(3, 64, 32, 3),
        kernel_sizes=(9, 1, 5),
        upscale_factor=scale),
    pixel_loss=dict(type='L1Loss', loss_weight=1.0, reduction='mean'),
        type='DataPreprocessor',
        mean=[0., 0., 0.],
        std=[255., 255., 255.],

pixel_loss=dict(type='L1Loss', loss_weight=1.0, reduction='mean'),
    ))
```

We also need to specify the training dataloader and testing dataloader according to create your own dataloader.
Finally we can start training our own model by：

```python
python train.py configs/srcnn/srcnn_x4k915_g1_1000k_div2k.py
```

## An example of DCGAN

Here, we take the implementation of the classical gan model, DCGAN \[2\], as an example.

### Step 1: Define the network of DCGAN

DCGAN is a classical image generative adversarial network \[2\]. To implement the network architecture of DCGAN, we need to create tow new files `mmagic/models/editors/dcgan/dcgan_generator.py` and `mmagic/models/editors/dcgan/dcgan_discriminator.py`, and implement generator (`class DCGANGenerator`) and discriminator (`class DCGANDiscriminator`).

In this step, we implement `class DCGANGenerator`, `class DCGANDiscriminator` and define the network architecture in `__init__` function.
In particular, we need to use `@MODULES.register_module()` to add the generator and discriminator into the registration of MMagic.

Take the following code as example:

```python
from mmcv.runner import load_checkpoint
from mmcv.utils.parrots_wrapper import _BatchNorm
from mmengine.logging import MMLogger
from mmengine.model.utils import normal_init

from mmagic.models.builder import MODULES
from ..common import get_module_device

class DCGANGenerator(nn.Module):
                 output_scale,
                 out_channels=3,
                 base_channels=1024,
                 input_scale=4,
                 noise_size=100,
                 default_norm_cfg=dict(type='BN'),
                 default_act_cfg=dict(type='ReLU'),

input_scale=4,
                 out_act_cfg=dict(type='Tanh'),
                 pretrained=None):
        self.output_scale = output_scale
        self.base_channels = base_channels
        self.input_scale = input_scale
        self.noise_size = noise_size

        # the number of times for upsampling
        self.num_upsamples = int(np.log2(output_scale // input_scale))

        # output 4x4 feature map
        self.noise2feat = ConvModule(
            noise_size,
            base_channels,
            kernel_size=4,
            stride=1,
            padding=0,
            conv_cfg=dict(type='ConvTranspose2d'),
            norm_cfg=default_norm_cfg,
            act_cfg=default_act_cfg)

        # build up upsampling backbone (excluding the output layer)
        upsampling = []

norm_cfg=default_norm_cfg,

        curr_channel = base_channels
        for _ in range(self.num_upsamples - 1):
            upsampling.append(
                ConvModule(
                    curr_channel,
                    curr_channel // 2,
                    kernel_size=4,
                    stride=2,
                    padding=1,
                    conv_cfg=dict(type='ConvTranspose2d'),
                    norm_cfg=default_norm_cfg,
                    act_cfg=default_act_cfg))

            curr_channel //= 2

        self.upsampling = nn.Sequential(*upsampling)

        # output layer
        self.output_layer = ConvModule(
            curr_channel,
            out_channels,
            stride=2,
            padding=1,
            norm_cfg=None,

out_channels,
            stride=2,
            act_cfg=out_act_cfg)
```

Then, we implement the `forward` function of  `DCGANGenerator`, which takes as `noise` tensor or `num_batches` and then returns the results from `DCGANGenerator`.

```python
    def forward(self, noise, num_batches=0, return_noise=False):
        noise_batch = noise_batch.to(get_module_device(self))
        x = self.noise2feat(noise_batch)
        x = self.upsampling(x)
        x = self.output_layer(x)
        return x
```

If you want to implement specific weights initialization method for you network, you need add `init_weights` function by yourself.

```python
    def init_weights(self, pretrained=None):
        if isinstance(pretrained, str):
            logger = MMLogger.get_current_instance()
            load_checkpoint(self, pretrained, strict=False, logger=logger)

if isinstance(pretrained, str):
        elif pretrained is None:
            for m in self.modules():
                if isinstance(m, (nn.Conv2d, nn.ConvTranspose2d)):
                    normal_init(m, 0, 0.02)
                elif isinstance(m, _BatchNorm):
                    nn.init.normal_(m.weight.data)
                    nn.init.constant_(m.bias.data, 0)
        else:
            raise TypeError('pretrained must be a str or None but'
                            f' got {type(pretrained)} instead.')
```

After the implementation of class `DCGANGenerator`, we need to update the model list in `mmagic/models/editors/__init__.py`, so that we can import and use class `DCGANGenerator` by `mmagic.models.editors`.

Implementation of Class `DCGANDiscriminator` follows the similar logic, and you can find the implementation [here](https://github.com/open-mmlab/mmagic/blob/main/mmagic/models/editors/dcgan/dcgan_discriminator.py).

### Step 2: Design the model of DCGAN

After the implementation of the network **Module**, we need to define our **Model** class `DCGAN`.

Your **Model** should inherit from [`BaseModel`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/model/base_model/base_model.py#L16) provided by MMEngine and implement three functions, `train_step`, `val_step` and `test_step`.

- `train_step`: This function is responsible to update the parameters of the network and called by MMEngine's Loop ([`IterBasedTrainLoop`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L183) or [`EpochBasedTrainLoop`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L18)). `train_step` take data batch and [`OptimWrapper`](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/optim_wrapper.md) as input and return a dict of log.
- `val_step`: This function is responsible for getting output for validation during the training process. and is called by [`MultiValLoop`](https://github.com/open-mmlab/mmagic/blob/main/mmagic/engine/runner/multi_loops.py#L19).
- `test_step`: This function is responsible for getting output in test process and is called by [`MultiTestLoop`](https://github.com/open-mmlab/mmagic/blob/main/mmagic/engine/runner/multi_loops.py#L274).

> Note that, in `train_step`, `val_step` and `test_step`, `DataPreprocessor` is called to preprocess the input data batch before feed them to the neural network. To know more about `DataPreprocessor` please refer to this [file](https://github.com/open-mmlab/mmagic/blob/main/mmagic/models/data_preprocessors/gen_preprocessor.py) and this [tutorial](https://github.com/open-mmlab/mmengine/blob/main/docs/zh_cn/tutorials/model.md#%E6%95%B0%E6%8D%AE%E5%A4%84%E7%90%86%E5%99%A8datapreprocessor).

For simplify using, we provide [`BaseGAN`](https://github.com/open-mmlab/mmagic/blob/main/mmagic/models/base_models/base_gan.py) class in MMagic, which implements generic `train_step`, `val_step` and `test_step` function for GAN models. With `BaseGAN` as base class, each specific GAN algorithm only need to implement `train_generator` and `train_discriminator`.

In `train_step`, we support data preprocessing, gradient accumulation (realized by [`OptimWrapper`](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/optim_wrapper.md)) and expontial moving averate (EMA) realized by [(`ExponentialMovingAverage`)](https://github.com/open-mmlab/mmagic/blob/main/mmagic/models/base_models/average_model.py#L19). With `BaseGAN.train_step`, each specific GAN algorithm only need to implement `train_generator` and `train_discriminator`.

```python
        message_hub = MessageHub.get_current_instance()
        curr_iter = message_hub.get_info('iter')
        data = self.data_preprocessor(data, True)
        disc_optimizer_wrapper: OptimWrapper = optim_wrapper['discriminator']
        disc_accu_iters = disc_optimizer_wrapper._accumulative_counts

        # train discriminator, use context manager provided by MMEngine

disc_accu_iters = disc_optimizer_wrapper._accumulative_counts

        with disc_optimizer_wrapper.optim_context(self.discriminator):
            # train_discriminator should be implemented!
            log_vars = self.train_discriminator(
                **data, optimizer_wrapper=disc_optimizer_wrapper)

        # add 1 to `curr_iter` because iter is updated in train loop.
        # Whether to update the generator. We update generator with
        # discriminator is fully updated for `self.n_discriminator_steps`
        # iterations. And one full updating for discriminator contains
        # `disc_accu_counts` times of grad accumulations.
        if (curr_iter + 1) % (self.discriminator_steps * disc_accu_iters) == 0:
            set_requires_grad(self.discriminator, False)
            gen_optimizer_wrapper = optim_wrapper['generator']
            gen_accu_iters = gen_optimizer_wrapper._accumulative_counts

set_requires_grad(self.discriminator, False)

            log_vars_gen_list = []
            # init optimizer wrapper status for generator manually
            gen_optimizer_wrapper.initialize_count_status(
                self.generator, 0, self.generator_steps * gen_accu_iters)
            # update generator, use context manager provided by MMEngine
            for _ in range(self.generator_steps * gen_accu_iters):
                with gen_optimizer_wrapper.optim_context(self.generator):
                    # train_generator should be implemented!
                    log_vars_gen = self.train_generator(
                        **data, optimizer_wrapper=gen_optimizer_wrapper)

                log_vars_gen_list.append(log_vars_gen)
            log_vars_gen = gather_log_vars(log_vars_gen_list)

**data, optimizer_wrapper=gen_optimizer_wrapper)

            log_vars_gen.pop('loss', None)  # remove 'loss' from gen logs

            set_requires_grad(self.discriminator, True)

            # only do ema after generator update
            if self.with_ema_gen and (curr_iter + 1) >= (
                    self.ema_start * self.discriminator_steps *
                    disc_accu_iters):
                self.generator_ema.update_parameters(
                    self.generator.module
                    if is_model_wrapper(self.generator) else self.generator)

            log_vars.update(log_vars_gen)

        # return the log dict
        return log_vars
```

In `val_step` and `test_step`, we call data preprocessing and `BaseGAN.forward` progressively.

```python
    def val_step(self, data: dict) -> SampleList:

return log_vars
```

```python
        data = self.data_preprocessor(data)
        # call `forward`
        outputs = self(**data)
        return outputs

    def test_step(self, data: dict) -> SampleList:
        # call `orward`
```

Then, we implement `train_generator` and `train_discriminator` in `DCGAN` class.

```python
from typing import Dict, Tuple

from mmengine.optim import OptimWrapper

from .base_gan import BaseGAN

class DCGAN(BaseGAN):
    def disc_loss(self, disc_pred_fake: Tensor,
                  disc_pred_real: Tensor) -> Tuple:
        losses_dict = dict()

        losses_dict['loss_disc_fake'] = F.binary_cross_entropy_with_logits(
            disc_pred_fake, 0. * torch.ones_like(disc_pred_fake))
        losses_dict['loss_disc_real'] = F.binary_cross_entropy_with_logits(
            disc_pred_real, 1. * torch.ones_like(disc_pred_real))

    def gen_loss(self, disc_pred_fake: Tensor) -> Tuple:
        losses_dict['loss_gen'] = F.binary_cross_entropy_with_logits(
            disc_pred_fake, 1. * torch.ones_like(disc_pred_fake))

    def train_discriminator(
            self, inputs, data_sample,
            optimizer_wrapper: OptimWrapper) -> Dict[str, Tensor]:

return loss, log_var

        real_imgs = inputs['img']

        num_batches = real_imgs.shape[0]

        noise_batch = self.noise_fn(num_batches=num_batches)
        with torch.no_grad():
            fake_imgs = self.generator(noise=noise_batch, return_noise=False)

        disc_pred_fake = self.discriminator(fake_imgs)
        disc_pred_real = self.discriminator(real_imgs)

        parsed_losses, log_vars = self.disc_loss(disc_pred_fake,
                                                 disc_pred_real)
        optimizer_wrapper.update_params(parsed_losses)

    def train_generator(self, inputs, data_sample,
                        optimizer_wrapper: OptimWrapper) -> Dict[str, Tensor]:
        num_batches = inputs['img'].shape[0]

        noise = self.noise_fn(num_batches=num_batches)

optimizer_wrapper: OptimWrapper) -> Dict[str, Tensor]:

        fake_imgs = self.generator(noise=noise, return_noise=False)

        parsed_loss, log_vars = self.gen_loss(disc_pred_fake)

        optimizer_wrapper.update_params(parsed_loss)
```

After the implementation of `class DCGAN`, we need to update the model list in `mmagic/models/__init__.py`, so that we can import and use `class DCGAN` by `mmagic.models`.

### Step 3: Start training DCGAN

After implementing the network **Module** and the **Model** of DCGAN,
now we can create a new file `configs/dcgan/dcgan_1xb128-5epoches_lsun-bedroom-64x64.py`
to set the configurations needed by training DCGAN.

In the configuration file, we need to specify the parameters of our model, `class DCGAN`, including the generator network architecture and data preprocessor of input tensors.

```python
    type='DCGAN',
    noise_size=100,
    data_preprocessor=dict(type='GANDataPreprocessor'),
    generator=dict(type='DCGANGenerator', output_scale=64, base_channels=1024),
        type='DCGANDiscriminator',
        input_scale=64,
        output_scale=4,
        out_channels=1))
```

We also need to specify the training dataloader and testing dataloader according to [create your own dataloader](dataset.md).

```python
python train.py configs/dcgan/dcgan_1xb128-5epoches_lsun-bedroom-64x64.py
```

## References

```python
```

1. Dong, Chao and Loy, Chen Change and He, Kaiming and Tang, Xiaoou. Image Super-Resolution Using Deep Convolutional Networks\[J\]. IEEE transactions on pattern analysis and machine intelligence, 2015.

2. Radford, Alec, Luke Metz, and Soumith Chintala. "Unsupervised representation learning with deep convolutional generative adversarial networks." arXiv preprint arXiv:1511.06434 (2015).

# 常见问题解答

我们列出了一些用户和开发者在开发过程中会遇到的常见问题以及对应的解决方案，如果您发现了任何频繁出现的问题，请随时扩充本列表，非常欢迎您提出的任何解决方案。如果您在环境配置、模型训练等工作中遇到任何的问题，请使用[问题模板](https://github.com/open-mmlab/mmdetection3d/blob/master/.github/ISSUE_TEMPLATE/error-report.md)来创建相应的 issue，并将所需的所有信息填入到问题模板中，我们会尽快解决您的问题。

## MMEngine/MMCV/MMDet/MMDet3D 安装

- 跟 MMEngine, MMCV, MMDetection 和 MMDetection3D 相关的编译问题; "ConvWS is already registered in conv layer"; "AssertionError: MMCV==xxx is used but incompatible. Please install mmcv>=xxx, \<=xxx."

- MMDetection3D 需要的 MMEngine, MMCV 和 MMDetection 的版本列在了下面。请安装正确版本的 MMEngine、MMCV 和 MMDetection 以避免相关的安装问题。

  | MMDetection3D 版本 |      MMEngine 版本       |        MMCV 版本        |     MMDetection 版本     |
  | ------------------ | :----------------------: | :---------------------: | :----------------------: |
  | dev-1.x            | mmengine>=0.7.1, \<1.0.0 | mmcv>=2.0.0rc4, \<2.1.0 |  mmdet>=3.0.0, \<3.1.0   |

| dev-1.x            | mmengine>=0.7.1, \<1.0.0 | mmcv>=2.0.0rc4, \<2.1.0 |  mmdet>=3.0.0, \<3.1.0   |
  | main               | mmengine>=0.7.1, \<1.0.0 | mmcv>=2.0.0rc4, \<2.1.0 |  mmdet>=3.0.0, \<3.1.0   |
  | v1.1.0rc3          | mmengine>=0.1.0, \<1.0.0 | mmcv>=2.0.0rc3, \<2.1.0 | mmdet>=3.0.0rc0, \<3.1.0 |
  | v1.1.0rc2          | mmengine>=0.1.0, \<1.0.0 | mmcv>=2.0.0rc3, \<2.1.0 | mmdet>=3.0.0rc0, \<3.1.0 |
  | v1.1.0rc1          | mmengine>=0.1.0, \<1.0.0 | mmcv>=2.0.0rc0, \<2.1.0 | mmdet>=3.0.0rc0, \<3.1.0 |
  | v1.1.0rc0          | mmengine>=0.1.0, \<1.0.0 | mmcv>=2.0.0rc0, \<2.1.0 | mmdet>=3.0.0rc0, \<3.1.0 |

  **注意**：如果你想安装 mmdet3d-v1.0.0rcx，可以在[此处](https://mmdetection3d.readthedocs.io/en/latest/faq.html#mmcv-mmdet-mmdet3d-installation)找到 MMDetection，MMSegmentation 和 MMCV 的兼容版本。请选择正确版本的 MMCV、MMDetection 和 MMSegmentation 以避免安装问题。

- 如果您在 `import open3d` 时遇到下面的问题：

  `OSError: /lib/x86_64-linux-gnu/libm.so.6: version 'GLIBC_2.27' not found`

  请将 open3d 的版本降级至 0.9.0.0，因为最新版 open3d 需要 'GLIBC_2.27' 文件的支持， Ubuntu 16.04 系统中缺失该文件，且该文件仅存在于 Ubuntu 18.04 及之后的系统中。

- 如果您在 `import pycocotools` 时遇到版本错误的问题，这是由于 nuscenes-devkit 需要安装 pycocotools，然而 mmdet 依赖于 mmpycocotools，当前的解决方案如下所示，我们将会在之后全面支持 pycocotools ：

  ```shell
  pip uninstall pycocotools mmpycocotools
  pip install mmpycocotools
  ```

  **注意**： 我们已经在 0.13.0 及之后的版本中全面支持 pycocotools。

- 如果您在导入 pycocotools 相关包时遇到下面的问题：

  `ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject`

  请将 pycocotools 的版本降级至 2.0.1，这是由于最新版本的 pycocotools 与 numpy \< 1.20.0 不兼容。或者通过下面的方式从源码进行编译来安装最新版本的 pycocotools ：

  `pip install -e "git+https://github.com/cocodataset/cocoapi#egg=pycocotools&subdirectory=PythonAPI"`

  或者

  `pip install -e "git+https://github.com/ppwwyyxx/cocoapi#egg=pycocotools&subdirectory=PythonAPI"`

或者

- 如果您使用 cuda-9.0 的环境并遇到关于 numba 的错误， 您应该检查下 numba 的版本。在 cuda-9.0 环境中，高版本的 numba 是不支持的，我们建议安装 numba==0.53.0.

## 如何标注点云？

MMDetection3D 不支持点云标注。我们提供一些开源的标注工具供参考：

- [SUSTechPOINTS](https://github.com/naurril/SUSTechPOINTS)
- [LATTE](https://github.com/bernwang/latte)

此外，我们改进了 [LATTE](https://github.com/bernwang/latte) 以便更方便的标注。更多的细节请参考[这里](https://arxiv.org/abs/2011.10174)。

# MobileNetV2

> [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://arxiv.org/abs/1801.04381)

<a href="https://github.com/tensorflow/models/tree/master/research/deeplab">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/backbones/mobilenet_v2.py#L14">Code Snippet</a>

In this paper we describe a new mobile architecture, MobileNetV2, that improves the state of the art performance of mobile models on multiple tasks and benchmarks as well as across a spectrum of different model sizes. We also describe efficient ways of applying these mobile models to object detection in a novel framework we call SSDLite. Additionally, we demonstrate how to build mobile semantic segmentation models through a reduced form of DeepLabv3 which we call Mobile DeepLabv3.

The MobileNetV2 architecture is based on an inverted residual structure where the input and output of the residual block are thin bottleneck layers opposite to traditional residual models which use expanded representations in the input an MobileNetV2 uses lightweight depthwise convolutions to filter features in the intermediate expansion layer. Additionally, we find that it is important to remove non-linearities in the narrow layers in order to maintain representational power. We demonstrate that this improves performance and provide an intuition that led to this design. Finally, our approach allows decoupling of the input/output domains from the expressiveness of the transformation, which provides a convenient framework for further analysis. We measure our performance on Imagenet classification, COCO object detection, VOC image segmentation. We evaluate the trade-offs between accuracy, and number of operations measured by multiply-adds (MAdd), as well as the number of parameters.

<img src="https://user-images.githubusercontent.com/24582831/142901935-fa22700e-4b77-477f-90b9-334a4197506f.png" width="50%"/>
</div>

| Method     | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                              | download                                                                                                                                                                                                                                                                                                                                                                                                 |

| FCN        | M-V2-D8  | 512x1024  |   80000 |      3.4 | 14.2           | A100   | 71.19 | 73.34         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mobilenet_v2/mobilenet-v2-d8_fcn_4xb2-80k_cityscapes-512x1024.py)           | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v2/mobilenet-v2-d8_fcn_4xb2-80k_cityscapes-512x1024/mobilenet-v2-d8_fcn_4xb2-80k_cityscapes-512x1024-20230224_185436-13fef4ea.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v2/mobilenet-v2-d8_fcn_4xb2-80k_cityscapes-512x1024/mobilenet-v2-d8_fcn_4xb2-80k_cityscapes-512x1024_20230224_185436.json) |

| PSPNet     | M-V2-D8  | 512x1024  |   80000 |      3.6 | 11.2           | V100   | 70.23 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mobilenet_v2/mobilenet-v2-d8_pspnet_4xb2-80k_cityscapes-512x1024.py)        | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v2/pspnet_m-v2-d8_512x1024_80k_cityscapes/pspnet_m-v2-d8_512x1024_80k_cityscapes_20200825_124817-19e81d51.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v2/pspnet_m-v2-d8_512x1024_80k_cityscapes/pspnet_m-v2-d8_512x1024_80k_cityscapes-20200825_124817.log.json)                                     |

| DeepLabV3  | M-V2-D8  | 512x1024  |   80000 |      3.9 | 8.4            | V100   | 73.84 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mobilenet_v2/mobilenet-v2-d8_deeplabv3_4xb2-80k_cityscapes-512x1024.py)     | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v2/deeplabv3_m-v2-d8_512x1024_80k_cityscapes/deeplabv3_m-v2-d8_512x1024_80k_cityscapes_20200825_124836-bef03590.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v2/deeplabv3_m-v2-d8_512x1024_80k_cityscapes/deeplabv3_m-v2-d8_512x1024_80k_cityscapes-20200825_124836.log.json)                         |

| DeepLabV3+ | M-V2-D8  | 512x1024  |   80000 |      5.1 | 8.4            | V100   | 75.20 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mobilenet_v2/mobilenet-v2-d8_deeplabv3plus_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v2/deeplabv3plus_m-v2-d8_512x1024_80k_cityscapes/deeplabv3plus_m-v2-d8_512x1024_80k_cityscapes_20200825_124836-d256dd4b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v2/deeplabv3plus_m-v2-d8_512x1024_80k_cityscapes/deeplabv3plus_m-v2-d8_512x1024_80k_cityscapes-20200825_124836.log.json)         |

### ADE20K

### ADE20K

| Method     | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                          | download                                                                                                                                                                                                                                                                                                                                                                         |

| FCN        | M-V2-D8  | 512x512   |  160000 |      6.5 | 64.4           | V100   | 19.71 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mobilenet_v2/mobilenet-v2-d8_fcn_4xb4-160k_ade20k-512x512.py)           | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v2/fcn_m-v2-d8_512x512_160k_ade20k/fcn_m-v2-d8_512x512_160k_ade20k_20200825_214953-c40e1095.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v2/fcn_m-v2-d8_512x512_160k_ade20k/fcn_m-v2-d8_512x512_160k_ade20k-20200825_214953.log.json)                                         |

| PSPNet     | M-V2-D8  | 512x512   |  160000 |      6.5 | 57.7           | V100   | 29.68 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mobilenet_v2/mobilenet-v2-d8_pspnet_4xb4-160k_ade20k-512x512.py)        | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v2/pspnet_m-v2-d8_512x512_160k_ade20k/pspnet_m-v2-d8_512x512_160k_ade20k_20200825_214953-f5942f7a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v2/pspnet_m-v2-d8_512x512_160k_ade20k/pspnet_m-v2-d8_512x512_160k_ade20k-20200825_214953.log.json)                             |

| DeepLabV3  | M-V2-D8  | 512x512   |  160000 |      6.8 | 39.9           | V100   | 34.08 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mobilenet_v2/mobilenet-v2-d8_deeplabv3_4xb4-160k_ade20k-512x512.py)     | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v2/deeplabv3_m-v2-d8_512x512_160k_ade20k/deeplabv3_m-v2-d8_512x512_160k_ade20k_20200825_223255-63986343.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v2/deeplabv3_m-v2-d8_512x512_160k_ade20k/deeplabv3_m-v2-d8_512x512_160k_ade20k-20200825_223255.log.json)                 |

| DeepLabV3+ | M-V2-D8  | 512x512   |  160000 |      8.2 | 43.1           | V100   | 34.02 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mobilenet_v2/mobilenet-v2-d8_deeplabv3plus_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v2/deeplabv3plus_m-v2-d8_512x512_160k_ade20k/deeplabv3plus_m-v2-d8_512x512_160k_ade20k_20200825_223255-465a01d4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v2/deeplabv3plus_m-v2-d8_512x512_160k_ade20k/deeplabv3plus_m-v2-d8_512x512_160k_ade20k-20200825_223255.log.json) |

```bibtex
}
```

# PointRend

> [PointRend: Image Segmentation as Rendering](https://arxiv.org/abs/1912.08193)

<a href="https://github.com/facebookresearch/detectron2/tree/master/projects/PointRend">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/decode_heads/point_head.py#L36">Code Snippet</a>

We present a new method for efficient high-quality image segmentation of objects and scenes. By analogizing classical computer graphics methods for efficient rendering with over- and undersampling challenges faced in pixel labeling tasks, we develop a unique perspective of image segmentation as a rendering problem. From this vantage, we present the PointRend (Point-based Rendering) neural network module: a module that performs point-based segmentation predictions at adaptively selected locations based on an iterative subdivision algorithm. PointRend can be flexibly applied to both instance and semantic segmentation tasks by building on top of existing state-of-the-art models. While many concrete implementations of the general idea are possible, we show that a simple design already achieves excellent results. Qualitatively, PointRend outputs crisp object boundaries in regions that are over-smoothed by previous methods. Quantitatively, PointRend yields significant gains on COCO and Cityscapes, for both instance and semantic segmentation. PointRend's efficiency enables output resolutions that are otherwise impractical in terms of memory or computation compared to existing approaches. Code has been made available at [this https URL](https://github.com/facebookresearch/detectron2/tree/main/projects/PointRend).

<img src="https://user-images.githubusercontent.com/24582831/142902293-5db49cdd-4b1b-4940-9067-2acd6196c700.png" width="60%"/>
</div>

| Method    | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                             | download                                                                                                                                                                                                                                                                                                                                                         |

| PointRend | R-50     | 512x1024  |   80000 |      3.1 | 8.48           | V100   | 76.47 | 78.13         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/point_rend/pointrend_r50_4xb2-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/point_rend/pointrend_r50_512x1024_80k_cityscapes/pointrend_r50_512x1024_80k_cityscapes_20200711_015821-bb1ff523.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/point_rend/pointrend_r50_512x1024_80k_cityscapes/pointrend_r50_512x1024_80k_cityscapes-20200715_214714.log.json)     |

| PointRend | R-101    | 512x1024  |   80000 |      4.2 | 7.00           | V100   | 78.30 | 79.97         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/point_rend/pointrend_r101_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/point_rend/pointrend_r101_512x1024_80k_cityscapes/pointrend_r101_512x1024_80k_cityscapes_20200711_170850-d0ca84be.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/point_rend/pointrend_r101_512x1024_80k_cityscapes/pointrend_r101_512x1024_80k_cityscapes-20200715_214824.log.json) |

### ADE20K

### ADE20K

| Method    | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                         | download                                                                                                                                                                                                                                                                                                                                         |

| PointRend | R-50     | 512x512   |  160000 |      5.1 | 17.31          | V100   | 37.64 | 39.17         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/point_rend/pointrend_r50_4xb4-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/point_rend/pointrend_r50_512x512_160k_ade20k/pointrend_r50_512x512_160k_ade20k_20200807_232644-ac3febf2.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/point_rend/pointrend_r50_512x512_160k_ade20k/pointrend_r50_512x512_160k_ade20k-20200807_232644.log.json)     |

| PointRend | R-101    | 512x512   |  160000 |      6.1 | 15.50          | V100   | 40.02 | 41.60         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/point_rend/pointrend_r101_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/point_rend/pointrend_r101_512x512_160k_ade20k/pointrend_r101_512x512_160k_ade20k_20200808_030852-8834902a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/point_rend/pointrend_r101_512x512_160k_ade20k/pointrend_r101_512x512_160k_ade20k-20200808_030852.log.json) |

```bibtex
@inproceedings{kirillov2020pointrend,
  title={Pointrend: Image segmentation as rendering},
  author={Kirillov, Alexander and Wu, Yuxin and He, Kaiming and Girshick, Ross},
  pages={9799--9808},
}
```

MMDetection works on Linux, Windows, and macOS. It requires Python 3.7+, CUDA 9.2+, and PyTorch 1.8+.

```{note}
```

```shell
```

```shell
```

```shell
```

```shell
```

```shell
```

We recommend that users follow our best practices to install MMDetection. However, the whole process is highly customizable. See [Customize Installation](#customize-installation) section for more information.

```shell
mim install "mmcv>=2.0.0"
```

**Note:** In MMCV-v2.x, `mmcv-full` is rename to `mmcv`, if you want to install `mmcv` without CUDA ops, you can use `mim install "mmcv-lite>=2.0.0rc1"` to install the lite version.

**Step 1.** Install MMDetection.

Case a: If you develop and run mmdet directly, install it from source:

```shell

```shell
git clone https://github.com/open-mmlab/mmdetection.git
```

Case b: If you use mmdet as a dependency or third-party package, install it with MIM:

```shell
mim install mmdet
```

To verify whether MMDetection is installed correctly, we provide some sample codes to run an inference demo.

```shell
mim download mmdet --config rtmdet_tiny_8xb32-300e_coco --dest .
```

```shell
```

The downloading will take several seconds or more, depending on your network environment. When it is done, you will find two files `rtmdet_tiny_8xb32-300e_coco.py` and `rtmdet_tiny_8xb32-300e_coco_20220902_112414-78e30dcc.pth` in your current folder.

Case a: If you install MMDetection from source, just run the following command.

```shell
python demo/image_demo.py demo/demo.jpg rtmdet_tiny_8xb32-300e_coco.py --weights rtmdet_tiny_8xb32-300e_coco_20220902_112414-78e30dcc.pth --device cpu
```

You will see a new image `demo.jpg` on your `./outputs/vis` folder, where bounding boxes are plotted on cars, benches, etc.

Case b: If you install MMDetection with MIM, open your python interpreter and copy&paste the following codes.

```python

```python

config_file = 'rtmdet_tiny_8xb32-300e_coco.py'
checkpoint_file = 'rtmdet_tiny_8xb32-300e_coco_20220902_112414-78e30dcc.pth'
model = init_detector(config_file, checkpoint_file, device='cpu')  # or device='cuda:0'
```

You will see a list of `DetDataSample`, and the predictions are in the `pred_instance`, indicating the detected bounding boxes, labels, and scores.

## Tracking Installation

We recommend that users follow our best practices to install MMDetection for for tracking task.

```shell
```

```shell
```

```shell
pip install -v -e . -r requirements/tracking.txt
```

```shell
mim install mmdet[tracking]
```

**Step 2.** Install TrackEval.

```shell
pip install git+https://github.com/JonathonLuiten/TrackEval.git
```

```shell

```shell
mim download mmdet --config bytetrack_yolox_x_8xb4-amp-80e_crowdhuman-mot17halftrain_test-mot17halfval --dest .
```

The downloading will take several seconds or more, depending on your network environment. When it is done, you will find two files `bytetrack_yolox_x_8xb4-amp-80e_crowdhuman-mot17halftrain_test-mot17halfval.py` and `bytetrack_yolox_x_crowdhuman_mot17-private-half_20211218_205500-1985c9f0.pth` in your current folder.

```shell
python demo/mot_demo.py demo/demo_mot.mp4 bytetrack_yolox_x_8xb4-amp-80e_crowdhuman-mot17halftrain_test-mot17halfval.py --checkpoint bytetrack_yolox_x_crowdhuman_mot17-private-half_20211218_205500-1985c9f0.pth --out mot.mp4
```

```

You will see a new video `mot.mp4` on your folder, where bounding boxes are plotted on person.

Case b: If you install MMDetection with MIM, open your python interpreter and demo/mot_demo.py, then run it like Case a.

```{note}

```{note}
Installing CUDA runtime libraries is enough if you follow our best practices, because no CUDA code will be compiled locally. However, if you hope to compile MMCV from source or develop other CUDA operators, you need to install the complete CUDA toolkit from NVIDIA's [website](https://developer.nvidia.com/cuda-downloads), and its version should match the CUDA version of PyTorch. i.e., the specified version of cudatoolkit in the `conda install` command.
```

#### Install MMEngine without MIM

To install MMEngine with pip instead of MIM, please follow [MMEngine installation guides](https://mmengine.readthedocs.io/en/latest/get_started/installation.html).

For example, you can install MMEngine by the following command.

```shell
```

To install MMCV with pip instead of MIM, please follow [MMCV installation guides](https://mmcv.readthedocs.io/en/2.x/get_started/installation.html). This requires manually specifying a find-url based on the PyTorch version and its CUDA version.

For example, the following command installs MMCV built for PyTorch 1.12.x and CUDA 11.6.

```shell
pip install "mmcv>=2.0.0" -f https://download.openmmlab.com/mmcv/dist/cu116/torch1.12.0/index.html
```

#### Install on CPU-only platforms

MMDetection can be built for CPU-only environments. In CPU mode you can train (requires MMCV version >= 2.0.0rc1), test, or infer a model.

However, some functionalities are gone in this mode:

- Deformable Convolution
- Modulated Deformable Convolution
- ROI pooling
- Deformable ROI pooling
- CARAFE
- SyncBatchNorm

- CARAFE
- CrissCrossAttention
- MaskedConv2d
- Temporal Interlace Shift
- nms_cuda
- sigmoid_focal_loss_cuda
- bbox_overlaps

If you try to train/test/infer a model containing the above ops, an error will be raised.
The following table lists affected algorithms.

|                        Operator                         |                                          Model                                           |
| :-----------------------------------------------------: | :--------------------------------------------------------------------------------------: |
| Deformable Convolution/Modulated Deformable Convolution | DCN、Guided Anchoring、RepPoints、CentripetalNet、VFNet、CascadeRPN、NAS-FCOS、DetectoRS |

|                      MaskedConv2d                       |                                     Guided Anchoring                                     |
|                         CARAFE                          |                                          CARAFE                                          |
|                      SyncBatchNorm                      |                                         ResNeSt                                          |

[Google Colab](https://colab.research.google.com/) usually has PyTorch installed,
thus we only need to install MMEngine, MMCV, and MMDetection with the following commands.

**Step 1.** Install [MMEngine](https://github.com/open-mmlab/mmengine) and [MMCV](https://github.com/open-mmlab/mmcv) using [MIM](https://github.com/open-mmlab/mim).

```shell

```shell
!mim install "mmcv>=2.0.0,<2.1.0"
```

**Step 2.** Install MMDetection from the source.

```shell
!git clone https://github.com/open-mmlab/mmdetection.git
%cd mmdetection
```

```python
import mmdet
print(mmdet.__version__)
# Example output: 3.0.0, or an another version.
```

```{note}
```

#### Use MMDetection with Docker

We provide a [Dockerfile](../../docker/Dockerfile) to build an image. Ensure that your [docker version](https://docs.docker.com/engine/install/) >=19.03.

```shell

```shell
# build an image with PyTorch 1.9, CUDA 11.1
docker build -t mmdetection docker/
```

Run it with

```shell
docker run --gpus all --shm-size=8g -it -v {DATA_DIR}:/mmdetection/data mmdetection
```

### Troubleshooting

If you have some issues during the installation, please first view the [FAQ](notes/faq.md) page.
You may [open an issue](https://github.com/open-mmlab/mmdetection/issues/new/choose) on GitHub if no solution is found.

### Use Multiple Versions of MMDetection in Development

Training and testing scripts have already been modified in `PYTHONPATH` in order to make sure the scripts are using their own versions of MMDetection.

To install the default version of MMDetection in your environment, you can exclude the follow code in the relative scripts:

```shell
```

# 教程1：了解配置文件

我们将模块化和继承性设计融入到我们的配置文件系统中，方便进行各种实验。如果您想查看配置文件，你可以运行 `python tools/misc/print_config.py /PATH/TO/CONFIG` 来查看完整的配置文件。你也可以通过传递参数 `--cfg-options xxx.yyy=zzz` 来查看更新的配置信息。

## 配置文件的结构

在  `config/_base_ ` 文件夹下面有4种基本组件类型： 数据集(dataset)，模型(model)，训练策略(schedule)和运行时的默认设置(default runtime)。许多模型都可以很容易地通过组合这些组件进行实现，比如 DeepLabV3，PSPNet。使用 `_base_` 下的组件构建的配置信息叫做原始配置 (primitive)。

对于同一个文件夹下的所有配置文件，建议**只有一个**对应的**原始配置文件**。所有其他的配置文件都应该继承自这个原始配置文件，从而保证每个配置文件的最大继承深度为 3。

为了便于理解，我们建议社区贡献者从现有的方法继承。例如，如果您在 DeepLabV3 基础上进行了一些修改，用户可以先通过指定 `_base_ = ../deeplabv3/deeplabv3_r50-d8_4xb2-40k_cityscapes-512x1024.py` 继承基本的 DeepLabV3 结构，然后在配置文件中修改必要的字段。

如果你正在构建一个全新的方法，它不与现有的任何方法共享基本组件，您可以在`config`下创建一个新的文件夹`xxxnet` ，详细文档请参考[mmengine](https://mmengine.readthedocs.io/en/latest/tutorials/config.html)。

## 配置文件命名风格

我们遵循以下格式来命名配置文件，建议社区贡献者遵循相同的风格。

```text

```text
```

配置文件的文件名分为五个部分，组成文件名每一个部分和组件之间都用`_`连接，每个部分或组件中的每个单词都要用`-`连接。

- `{algorithm name}`: 算法的名称，如 `deeplabv3`,  `pspnet` 等。
- `{model component names}`: 算法中使用的组件名称，如主干(backbone)、解码头(head)等。例如，`r50-d8 `表示使用ResNet50主干网络，并使用主干网络的8倍下采样输出作为下一级的输入。
- `{training settings}`: 训练时的参数设置，如 `batch size`、数据增强(augmentation)、损失函数(loss)、学习率调度器(learning rate scheduler)和训练轮数(epochs/iterations)。例如: `4xb4-ce-linearlr-40K` 意味着使用4个gpu，每个gpu4个图像，使用交叉熵损失函数(CrossEntropy)，线性学习率调度程序，训练40K iterations。
  一些缩写:
  - `{gpu x batch_per_gpu}`: GPU数量和每个GPU的样本数。`bN ` 表示每个GPU的batch size为N，如 `8xb2` 为8个gpu x 每个gpu2张图像的缩写。如果未提及，则默认使用 `4xb4 `。
  - `{schedule}`: 训练计划，选项有`20k`，`40k`等。`20k ` 和 `40k` 分别表示20000次迭代(iterations)和40000次迭代(iterations)。

- `{schedule}`: 训练计划，选项有`20k`，`40k`等。`20k ` 和 `40k` 分别表示20000次迭代(iterations)和40000次迭代(iterations)。
- `{training dataset information}`: 训练数据集名称，如 `cityscapes `， `ade20k ` 等，以及输入分辨率。例如: `cityscapes-768x768  `表示使用 `cityscapes` 数据集进行训练，输入分辨率为`768x768 `。
- `{testing dataset information}` (可选): 测试数据集名称。当您的模型在一个数据集上训练但在另一个数据集上测试时，请将测试数据集名称添加到此处。如果没有这一部分，则意味着模型是在同一个数据集上进行训练和测试的。

## PSPNet 的一个例子

为了帮助用户熟悉对这个现代语义分割系统的完整配置文件和模块，我们对使用ResNet50V1c作为主干网络的PSPNet的配置文件作如下的简要注释和说明。要了解更详细的用法和每个模块对应的替换方法，请参阅API文档。

```python
_base_ = [
] # 我们可以在基本配置文件的基础上 构建新的配置文件
```

`_base_/models/pspnet_r50-d8.py`是使用ResNet50V1c作为主干网络的PSPNet的基本模型配置文件。

```python
# 模型设置
norm_cfg = dict(type='SyncBN', requires_grad=True)  # 分割框架通常使用 SyncBN

```

```python
# 模型设置
data_preprocessor = dict(  # 数据预处理的配置项，通常包括图像的归一化和增强
    type='SegDataPreProcessor',  # 数据预处理的类型
    mean=[123.675, 116.28, 103.53],  # 用于归一化输入图像的平均值
    std=[58.395, 57.12, 57.375],  # 用于归一化输入图像的标准差
    bgr_to_rgb=True,  # 是否将图像从 BGR 转为 RGB
    pad_val=0,  # 图像的填充值
    seg_pad_val=255)  # 'gt_seg_map'的填充值
    type='EncoderDecoder',  # 分割器(segmentor)的名字
    pretrained='open-mmlab://resnet50_v1c',  # 加载使用 ImageNet 预训练的主干网络
        type='ResNetV1c',  # 主干网络的类别，更多细节请参考 mmseg/models/backbones/resnet.py
        depth=50,  # 主干网络的深度，通常为 50 和 101
        num_stages=4,  # 主干网络状态(stages)的数目
        out_indices=(0, 1, 2, 3),  # 每个状态(stage)产生的特征图输出的索引
        dilations=(1, 1, 2, 4),  # 每一层(layer)的空心率(dilation rate)

num_stages=4,  # 主干网络状态(stages)的数目
        strides=(1, 2, 1, 1),  # 每一层(layer)的步长(stride)
        norm_cfg=norm_cfg,  # 归一化层(norm layer)的配置项
        norm_eval=False,  # 是否冻结 BN 里的统计项
        contract_dilation=True),  # 当空洞率 > 1, 是否压缩第一个空洞层
        type='PSPHead',  # 解码头(decode head)的类别。可用选项请参 mmseg/models/decode_heads
        in_channels=2048,  # 解码头的输入通道数
        in_index=3,  # 被选择特征图(feature map)的索引
        channels=512,  # 解码头中间态(intermediate)的通道数
        pool_scales=(1, 2, 3, 6),  # PSPHead 平均池化(avg pooling)的规模(scales)。 细节请参考文章内容
        dropout_ratio=0.1,  # 进入最后分类层(classification layer)之前的 dropout 比例
        num_classes=19,  # 分割前景的种类数目。 通常情况下，cityscapes 为19，VOC为21，ADE20k 为150
        norm_cfg=norm_cfg,  # 归一化层的配置项

dropout_ratio=0.1,  # 进入最后分类层(classification layer)之前的 dropout 比例
        align_corners=False,  # 解码过程中调整大小(resize)的 align_corners 参数
        loss_decode=dict(  # 解码头(decode_head)里的损失函数的配置项
            type='CrossEntropyLoss',  # 分割时使用的损失函数的类别
            use_sigmoid=False,  # 分割时是否使用 sigmoid 激活
            loss_weight=1.0)),  # 解码头的损失权重
        type='FCNHead',  # 辅助头(auxiliary head)的种类。可用选项请参考 mmseg/models/decode_heads
        in_channels=1024,  # 辅助头的输入通道数
        in_index=2,  # 被选择的特征图(feature map)的索引
        channels=256,  # 辅助头中间态(intermediate)的通道数
        num_convs=1,  # FCNHead 里卷积(convs)的数目，辅助头中通常为1
        concat_input=False,  # 在分类层(classification layer)之前是否连接(concat)输入和卷积的输出

        loss_decode=dict(  # 辅助头(auxiliary head)里的损失函数的配置项
            loss_weight=0.4)),  # 辅助头损失的权重，默认设置为0.4
    # 模型训练和测试设置项
    train_cfg=dict(),  # train_cfg 当前仅是一个占位符
    test_cfg=dict(mode='whole'))  # 测试模式，可选参数为 'whole' 和 'slide'. 'whole': 在整张图像上全卷积(fully-convolutional)测试。 'slide': 在输入图像上做滑窗预测
```

`_base_/datasets/cityscapes.py`是数据集的基本配置文件。

```python
# 数据集设置
dataset_type = 'CityscapesDataset'  # 数据集类型，这将被用来定义数据集
data_root = 'data/cityscapes/'  # 数据的根路径
crop_size = (512, 1024)  # 训练时的裁剪大小
train_pipeline = [  # 训练流程
    dict(type='LoadImageFromFile'),  # 第1个流程，从文件路径里加载图像

    dict(type='LoadAnnotations'),  # 第2个流程，对于当前图像，加载它的标注图像
    dict(type='RandomResize',  # 调整输入图像大小(resize)和其标注图像的数据增广流程
        scale=(2048, 1024),  # 图像裁剪的大小
        ratio_range=(0.5, 2.0),  # 数据增广的比例范围
        keep_ratio=True),  # 调整图像大小时是否保持纵横比
    dict(type='RandomCrop',  # 随机裁剪当前图像和其标注图像的数据增广流程
        crop_size=crop_size,  # 随机裁剪的大小
        cat_max_ratio=0.75),  # 单个类别可以填充的最大区域的比
    dict(type='RandomFlip',  # 翻转图像和其标注图像的数据增广流程
        prob=0.5),  # 翻转图像的概率
    dict(type='PhotoMetricDistortion'),  # 光学上使用一些方法扭曲当前图像和其标注图像的数据增广流程
    dict(type='PackSegInputs')  # 打包用于语义分割的输入数据
]
    dict(type='Resize',  # 使用调整图像大小(resize)增强
        scale=(2048, 1024),  # 图像缩放的大小
        keep_ratio=True),  # 在调整图像大小时是否保留长宽比
    # 在' Resize '之后添加标注图像

dict(type='Resize',  # 使用调整图像大小(resize)增强
    # 不需要做调整图像大小(resize)的数据变换
    dict(type='LoadAnnotations'),  # 加载数据集提供的语义分割标注
]
train_dataloader = dict(  # 训练数据加载器(dataloader)的配置
    batch_size=2,  # 每一个GPU的batch size大小
    num_workers=2,  # 为每一个GPU预读取数据的进程个数
    persistent_workers=True,  # 在一个epoch结束后关闭worker进程，可以加快训练速度
    sampler=dict(type='InfiniteSampler', shuffle=True),  # 训练时进行随机洗牌(shuffle)
    dataset=dict(  # 训练数据集配置
        type=dataset_type,  # 数据集类型，详见mmseg/datassets/
        data_root=data_root,  # 数据集的根目录
            img_path='leftImg8bit/train', seg_map_path='gtFine/train'),  # 训练数据的前缀
        pipeline=train_pipeline)) # 数据处理流程，它通过之前创建的train_pipeline传递。
    batch_size=1,  # 每一个GPU的batch size大小
    num_workers=4,  # 为每一个GPU预读取数据的进程个数

pipeline=train_pipeline)) # 数据处理流程，它通过之前创建的train_pipeline传递。
    sampler=dict(type='DefaultSampler', shuffle=False),  # 训练时不进行随机洗牌(shuffle)
    dataset=dict(  # 测试数据集配置
            img_path='leftImg8bit/val', seg_map_path='gtFine/val'),  # 测试数据的前缀
        pipeline=test_pipeline))  # 数据处理流程，它通过之前创建的test_pipeline传递。
# 精度评估方法，我们在这里使用 IoUMetric 进行评估
```

```python
optimizer = dict(type='SGD', # 优化器种类，更多细节可参考 https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/optimizer/default_constructor.py

```python
                lr=0.01,  # 优化器的学习率，参数的使用细节请参照对应的 PyTorch 文档
                momentum=0.9,  # 动量大小 (Momentum)
                weight_decay=0.0005)  # SGD 的权重衰减 (weight decay)
optim_wrapper = dict(type='OptimWrapper',  # 优化器包装器(Optimizer wrapper)为更新参数提供了一个公共接口
                    optimizer=optimizer,  # 用于更新模型参数的优化器(Optimizer)
                    clip_grad=None)  # 如果 'clip_grad' 不是None，它将是 ' torch.nn.utils.clip_grad' 的参数。
# 学习策略
    dict(
        type='PolyLR',  # 调度流程的策略，同样支持 Step, CosineAnnealing, Cyclic 等. 请从 https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py 参考 LrUpdater 的细节
        eta_min=1e-4,  # 训练结束时的最小学习率
        power=0.9,  # 多项式衰减 (polynomial decay) 的幂
        begin=0,  # 开始更新参数的时间步(step)

eta_min=1e-4,  # 训练结束时的最小学习率
        end=40000,  # 停止更新参数的时间步(step)
        by_epoch=False)  # 是否按照 epoch 计算训练时间
]
# 40k iteration 的训练计划
# 默认钩子(hook)配置
    timer=dict(type='IterTimerHook'),  # 记录迭代过程中花费的时间
    logger=dict(type='LoggerHook', interval=50, log_metric_by_epoch=False),  # 从'Runner'的不同组件收集和写入日志
    param_scheduler=dict(type='ParamSchedulerHook'),  # 更新优化器中的一些超参数，例如学习率
    checkpoint=dict(type='CheckpointHook', by_epoch=False, interval=4000),  # 定期保存检查点(checkpoint)
    sampler_seed=dict(type='DistSamplerSeedHook'))  # 用于分布式训练的数据加载采样器
```

```python
# 将注册表的默认范围设置为mmseg

```

```python
)
load_from = None  # 从文件中加载检查点(checkpoint)
resume = False  # 是否从已有的模型恢复
```

这些都是用于训练和测试PSPNet的配置文件，要加载和解析它们，我们可以使用[MMEngine](https://github.com/open-mmlab/mmengine)实现的[Config](https://mmengine.readthedocs.io/en/latest/tutorials/config.html)。

```python

```

```shell

'sampler': {'type': 'InfiniteSampler', 'shuffle': True},
```

`cfg `是`mmengine.config.Config `的一个实例。它的接口与dict对象相同，也允许将配置值作为属性访问。更多信息请参见[MMEngine](https://github.com/open-mmlab/mmengine)中的[config tutorial](https://mmengine.readthedocs.io/en/latest/tutorials/config.html)。

## FAQ

### 忽略基础配置文件里的一些字段

## FAQ

有时，您可以设置`_delete_=True `来忽略基本配置文件中的某些字段。您可以参考[MMEngine](https://github.com/open-mmlab/mmengine)中的[config tutorial](https://mmengine.readthedocs.io/en/latest/tutorials/config.html)来获得一些简单的指导。

例如，在MMSegmentation中，如果您想在下面的配置文件`pspnet.py `中修改PSPNet的主干网络:

```python
        depth=50,

channels=512,
```

用以下代码加载并解析配置文件`pspnet.py`:

```python

```

```shell

'in_channels': 2048,
```

`ResNet`和`HRNet`使用不同的关键字构建，编写一个新的配置文件`hrnet.py`，如下所示:

```python

num_channels=(64, )),
```

用以下代码加载并解析配置文件`hrnet.py`:

```python
```

```shell

'pretrained': 'open-mmlab://msra/hrnetv2_w32',

'in_index': 3,
```

`_delete_=True` 将用新的键去替换 `backbone` 字段内所有旧的键。

配置文件中会使用一些中间变量，例如数据集(datasets)字段里的 `train_pipeline`/`test_pipeline`。 需要注意的是，在子配置文件里修改中间变量时，您需要再次传递这些变量给对应的字段。例如，我们想改变在训练或测试PSPNet时采用的多尺度策略 (multi scale strategy)，`train_pipeline`/`test_pipeline` 是我们需要修改的中间变量。

```python

]
]

```

我们首先需要定义新的 `train_pipeline`/`test_pipeline` 然后传递到 `dataset` 里。

类似的，如果我们想从 `SyncBN` 切换到 `BN` 或者 `MMSyncBN`，我们需要替换配置文件里的每一个 `norm_cfg`。

```python
```

## 通过脚本参数修改配置文件

在[training script](https://github.com/open-mmlab/mmsegmentation/blob/1.x/tools/train.py)和[testing script](https://github.com/open-mmlab/mmsegmentation/blob/1.x/tools/test.py)中，我们支持脚本参数 `--cfg-options`，它可以帮助用户覆盖所使用的配置中的一些设置，`xxx=yyy` 格式的键值对将合并到配置文件中。

例如，这是一个简化的脚本 `demo_script.py `:

```python

return args

    main()
```

一个配置文件示例 `demo_config.py` 如下所示:

```python
    depth=50,
```

运行 `demo_script.py`:

```shell
```

```shell
```

通过脚本参数修改配置:

```shell
```

```shell

```

```shell
```

```shell
```

- 更新列表/元组的值。

  如果要更新的值是一个 list 或 tuple。例如，需要在配置文件 `demo_config.py ` 的 `backbone ` 中设置 `stride =(1,2,1,1) `。
  如果您想更改这个键，你可以用两种方式进行指定:

  1. `--cfg-options backbone.strides="(1, 1, 1, 1)"`. 注意引号 " 是支持 list/tuple 数据类型所必需的。

     ```shell
     ```

     ```shell
     ```

```

  2. `--cfg-options backbone.strides=1,1,1,1`. 注意，在指定的值中**不允许**有空格。

     另外，如果原来的类型是tuple，通过这种方式修改后会自动转换为list。

     ```shell
     ```

     ```shell
     ```

```{note}
  这种修改方法仅支持修改string、int、float、boolean、None、list和tuple类型的配置项。
  具体来说，对于list和tuple类型的配置项，它们内部的元素也必须是上述七种类型之一。
```

# 教程 8：模型部署指南

[MMDeploy](https://github.com/open-mmlab/mmdeploy) 是 OpenMMLab 的部署仓库，负责包括 MMClassification、MMDetection、MMagic 等在内的各算法库的部署工作。
你可以从[这里](https://mmdeploy.readthedocs.io/zh_CN/latest/04-supported-codebases/mmagic.html)获取 MMDeploy 对 MMClassification 部署支持的最新文档。

本文的结构如下：

- [安装](#安装)
- [模型转换](#模型转换)
- [模型规范](#模型规范)
- [模型推理](#模型推理)
  - [后端模型推理](#后端模型推理)
  - [SDK 模型推理](#sdk-模型推理)
- [模型支持列表](#模型支持列表)

## 安装

请参考[此处](../get_started/install.md)安装 mmagic。然后，按照[说明](https://mmdeploy.readthedocs.io/zh_CN/latest/get_started.html#mmdeploy)安装 mmdeploy。

```{note}
如果安装的是 mmdeploy 预编译包，那么也请通过 'git clone https://github.com/open-mmlab/mmdeploy.git --depth=1' 下载 mmdeploy 源码。因为它包含了部署时要用到的配置文件
```

## 模型转换

```{note}
```

## 模型转换

假设在安装步骤中，mmagic 和 mmdeploy 代码库在同级目录下，并且当前的工作目录为 mmagic 的根目录，那么以 [ESRGAN](../../../configs/esrgan/esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py) 模型为例，你可以从[此处](https://download.openmmlab.com/mmediting/restorers/esrgan/esrgan_psnr_x4c64b23g32_1x16_1000k_div2k_20200420-bf5c993c.pth)下载对应的 checkpoint，并使用以下代码将之转换为 onnx 模型：

```python

img = 'tests/data/image/face/000001.png'
work_dir = 'mmdeploy_models/mmagic/onnx'
deploy_cfg = '../mmdeploy/configs/mmagic/super-resolution/super-resolution_onnxruntime_dynamic.py'
model_cfg = 'configs/esrgan/esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py'
model_checkpoint = 'esrgan_psnr_x4c64b23g32_1x16_1000k_div2k_20200420-bf5c993c.pth'

# 2. extract pipeline info for inference by MMDeploy SDK
```

转换的关键之一是使用正确的配置文件。项目中已内置了各后端部署[配置文件](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmagic)。
文件的命名模式是：

```
{task}/{task}_{backend}-{precision}_{static | dynamic}_{shape}.py
```

其中：

- **{task}:** mmagic 中的任务

在上例中，你也可以把 `ESRGAN` 转为其他后端模型。比如使用`super-resolution_tensorrt-fp16_dynamic-32x32-512x512.py`，把模型转为 tensorrt-fp16 模型。

```{tip}

```{tip}
```

## 模型规范

上例中的`mmdeploy_models/mmagic/onnx`，结构如下：

```
mmdeploy_models/mmagic/onnx
```

重要的是：

- ***xxx*.json**:  mmdeploy SDK 推理所需的 meta 信息

## 模型推理

### 后端模型推理

```python

backend_model = ['mmdeploy_models/mmagic/onnx/end2end.onnx']
image = 'tests/data/image/lq/baboon_x4.png'

    output_file='output_restorer.bmp')
```

```python

image=image,
```

```python
from mmdeploy_python import Restorer
import cv2

img = cv2.imread('tests/data/image/lq/baboon_x4.png')

# create a predictor
restorer = Restorer(model_path='mmdeploy_models/mmagic/onnx', device_name='cpu', device_id=0)
result = restorer(img)

cv2.imwrite('output_restorer.bmp', result)
```

## 模型支持列表

请参考[这里](https://mmdeploy.readthedocs.io/zh_CN/latest/04-supported-codebases/mmagic.html#id7)

# 概述

本章将向你介绍 MMPose 的整体框架，并提供详细的教程链接。

## 什么是 MMPose

![overview](https://user-images.githubusercontent.com/13503330/191004511-508d3ec6-9ead-4c52-a522-4d9aa1f26027.png)

MMPose 是一款基于 Pytorch 的姿态估计开源工具箱，是 OpenMMLab 项目的成员之一，包含了丰富的 2D 多人姿态估计、2D 手部姿态估计、2D 人脸关键点检测、133关键点全身人体姿态估计、动物关键点检测、服饰关键点检测等算法以及相关的组件和模块，下面是它的整体框架：

MMPose 由 **8** 个主要部分组成，apis、structures、datasets、codecs、models、engine、evaluation 和 visualization。

- **apis** 提供用于模型推理的高级 API

- **structures** 提供 bbox、keypoint 和 PoseDataSample 等数据结构

- **datasets** 支持用于姿态估计的各种数据集

  - **transforms** 包含各种数据增强变换

- **codecs** 提供姿态编解码器：编码器用于将姿态信息（通常为关键点坐标）编码为模型学习目标（如热力图），解码器则用于将模型输出解码为姿态估计结果

- **models** 以模块化结构提供了姿态估计模型的各类组件

  - **pose_estimators** 定义了所有姿态估计模型类
  - **data_preprocessors** 用于预处理模型的输入数据
  - **backbones** 包含各种骨干网络
  - **necks** 包含各种模型颈部组件
  - **heads** 包含各种模型头部
  - **losses** 包含各种损失函数

- **engine** 包含与姿态估计任务相关的运行时组件

  - **hooks** 提供运行时的各种钩子

- **evaluation** 提供各种评估模型性能的指标

- **visualization** 用于可视化关键点骨架和热力图等信息

## 如何使用本指南

- **heads** 包含各种模型头部

## 如何使用本指南

针对不同类型的用户，我们准备了详细的指南：

1. 安装说明：

   - [安装](./installation.md)

2. MMPose 的基本使用方法：

   - [20 分钟上手教程](./guide_to_framework.md)
   - [Demos](./demos.md)
   - [模型推理](./user_guides/inference.md)
   - [配置文件](./user_guides/configs.md)
   - [准备数据集](./user_guides/prepare_datasets.md)
   - [训练与测试](./user_guides/train_and_test.md)

3. 对于希望基于 MMPose 进行开发的研究者和开发者：

   - [编解码器](./advanced_guides/codecs.md)
   - [数据流](./advanced_guides/dataflow.md)
   - [实现新模型](./advanced_guides/implement_new_models.md)
   - [自定义数据集](./advanced_guides/customize_datasets.md)
   - [自定义数据变换](./advanced_guides/customize_transforms.md)
   - [自定义优化器](./advanced_guides/customize_optimizer.md)
   - [自定义日志](./advanced_guides/customize_logging.md)
   - [模型部署](./advanced_guides/how_to_deploy.md)

- [自定义优化器](./advanced_guides/customize_optimizer.md)
   - [模型分析工具](./advanced_guides/model_analysis.md)
   - [迁移指南](./migration.md)

4. 对于希望加入开源社区，向 MMPose 贡献代码的研究者和开发者：

   - [参与贡献代码](./contribution_guide.md)

5. 对于使用过程中的常见问题：

   - [FAQ](./faq.md)

# 混淆矩阵

MMPretrain 提供 `tools/analysis_tools/confusion_matrix.py` 工具来分析预测结果的混淆矩阵。关于混淆矩阵的介绍，可参考[链接](https://zh.wikipedia.org/zh-cn/%E6%B7%B7%E6%B7%86%E7%9F%A9%E9%98%B5)。

## 命令行使用

**命令行**：

```shell
    [--show] \
    [--show-path] \
    [--include-values] \
    [--cmap ${CMAP}] \
```

- `checkpoint`：权重路径。
- `--show`：是否展示混淆矩阵的 matplotlib 可视化结果，默认不展示。
- `--show-path`：如果 `show` 为 True，可视化结果的保存路径。
- `--include-values`：是否在可视化结果上添加数值。
- `--cmap`：可视化结果使用的颜色映射图，即 `cmap`，默认为 `viridis`。
- `--cfg-options`：对配置文件的修改，参考[学习配置文件](../user_guides/config.md)。

**使用示例**：

```shell
    https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_b16x8_cifar10_20210528-f54bfad9.pth \
    --show
```

**输出图片**：

configs/resnet/resnet50_8xb16_cifar10.py \
    --show
```

**输出图片**：

<div align=center><img src="https://user-images.githubusercontent.com/26739999/210298124-49ae00f7-c8fd-488a-a4da-58c285e9c1f1.png" style=" width: auto; height: 40%; "></div>

## 基础用法

```python
>>> import torch
>>> from mmpretrain.evaluation import ConfusionMatrix
>>> y_pred = [0, 1, 1, 3]
>>> y_true = [0, 2, 1, 3]
>>> ConfusionMatrix.calculate(y_pred, y_true, num_classes=4)
tensor([[1, 0, 0, 0],
        [0, 1, 0, 0],
        [0, 0, 0, 1]])
>>> # plot the confusion matrix
>>> import matplotlib.pyplot as plt
>>> y_score = torch.rand((1000, 10))
>>> y_true = torch.randint(10, (1000, ))
>>> matrix = ConfusionMatrix.calculate(y_score, y_true)
>>> ConfusionMatrix().plot(matrix)
>>> plt.show()
```

## 结合评估器使用

```python

```

## 结合评估器使用

```python
>>> from mmpretrain.structures import DataSample
>>> from mmengine.evaluator import Evaluator
>>> data_samples = [
...     DataSample().set_gt_label(i%5).set_pred_score(torch.rand(5))
...     for i in range(1000)
... ]
>>> evaluator = Evaluator(metrics=ConfusionMatrix())
>>> evaluator.process(data_samples)
>>> evaluator.evaluate(1000)
{'confusion_matrix/result': tensor([[37, 37, 48, 43, 35],
         [35, 51, 32, 46, 36],
         [45, 28, 39, 42, 46],
         [42, 40, 40, 35, 43],
         [40, 39, 41, 37, 43]])}
```

# Supported RKNN feature

Currently, MMDeploy only tests rk3588 and rv1126 with linux platform.

The following features cannot be automatically enabled by mmdeploy and you need to manually modify the configuration in MMDeploy like [here](https://github.com/open-mmlab/mmdeploy/tree/main/configs/_base_/backends/rknn.py).

- target_platform other than default
- quantization settings
- optimization level other than 1

# How to join the technical Community Groups with QQ  ？

- Open QQ -> click the group chat search-> search group number 637093648, enter the answer to the question: conv conv conv conv conv → join the group chat → ready to accept the Turing test(a joke)
- Open QQ -> search Pocky group: 677104663 (lots experts), the answer to the question

# How to watch the author's on live in Bilibili？

- nihui：[水竹院落](https://live.bilibili.com/1264617)

# Compilation

- ## How to download the full source code？

   git clone --recursive https://github.com/Tencent/ncnn/
   
   or
   
   download [ncnn-xxxxx-full-source.zip](https://github.com/Tencent/ncnn/releases)

- ## How to cross-compile？How to set the cmake toolchain？
  
   See https://github.com/Tencent/ncnn/wiki/how-to-build

- ## The submodules were not downloaded! Please update submodules with "git submodule update --init" and try again

   As above, download the full source code. Or follow the prompts to execute: git submodule update --init

As above, download the full source code. Or follow the prompts to execute: git submodule update --init

- ## Could NOT find Protobuf (missing: Protobuf_INCLUDE_DIR)
  
   sudo apt-get install libprotobuf-dev protobuf-compiler

- ## Could NOT find CUDA (missing: CUDA_TOOLKIT_ROOT_DIR CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY)

   https://github.com/Tencent/ncnn/issues/1873

- ## Could not find a package configuration file provided by "OpenCV" with any of the following names: OpenCVConfig.cmake opencv-config.cmake

   sudo apt-get install libopencv-dev

   or customized compile and install ，with set(OpenCV_DIR {the dir OpenCVConfig.cmake exist})

- ## Could not find a package configuration file provided by "ncnn" with any of the following names: ncnnConfig.cmake ncnn-config.cmake

   set(ncnn_DIR { the dir ncnnConfig.cmake exist})

- ## Vulkan not found, 

   - cmake requires version >= 3.10, otherwise there is no FindVulkan.cmake

   - android-api >= 24

set(ncnn_DIR { the dir ncnnConfig.cmake exist})

   - macos has to run the install script first

- ## How to install vulkan sdk

    - See https://www.vulkan.org/tools#download-these-essential-development-tools
    - But There was a frequent problem that the project need glslang lib in ncnn not official vulkan

- ## xxx.lib not found（be specified by system/compiler）

   undefined reference to __kmpc_for_static_init_4 __kmpc_for_static_fini __kmpc_fork_call ...

   Need to link openmp

   undefined reference to vkEnumerateInstanceExtensionProperties vkGetInstanceProcAddr vkQueueSubmit ...

   need vulkan-1.lib

   undefined reference to glslang::InitializeProcess() glslang::TShader::TShader(EShLanguage) ...

   need glslang.lib OGLCompiler.lib SPIRV.lib OSDependent.lib

   undefined reference to AAssetManager_fromJava AAssetManager_open AAsset_seek ...

need glslang.lib OGLCompiler.lib SPIRV.lib OSDependent.lib

   Add android to find_library and target_like_libraries 

   find_package(ncnn)

- ## undefined reference to typeinfo for ncnn::Layer

   opencv rtti -> opencv-mobile

- ## undefined reference to __cpu_model

   upgrade compiler / libgcc_s libgcc

- ## unrecognized command line option "-mavx2"

   upgrade gcc

- ## Why is the compiled ncnn-android library so large？

   See https://github.com/Tencent/ncnn/wiki/build-for-android.zh and see How to trim smaller ncnn

- ## ncnnoptimize and custom layer

   ncnnoptimize first before adding a custom layer to avoid ncnnoptimize not being able to handle custom layer saves.

- ## rtti/exceptions Conflict

   The reason for the conflict is that the libraries used in the project are configured differently, so analyze whether you need to turn them on or off according to your actual situation. ncnn is ON by default, add the following two parameters when recompiling ncnn.
   - ON: -DNCNN_DISABLE_RTTI=OFF -DNCNN_DISABLE_EXCEPTION=OFF
   - OFF: -DNCNN_DISABLE_RTTI=ON -DNCNN_DISABLE_EXCEPTION=ON

- ## error: undefined symbol: ncnn::Extractor::extract(char const*, ncnn::Mat&)

   Possible scenarios.
   - Try upgrading the NDK version of Android Studio

# How do I add the ncnn library to my project and how does the cmake method work?

Compile ncnn,and make install. linux/windows should set/export ncnn_DIR points to the directory containing ncnnConfig.cmake under the install directory

- ## android

- ## ios

- ## ios

- ## linux

- ## windows

- ## macos

- ## arm linux

# Convert model issues

- ## caffe

   `./caffe2ncnn caffe.prototxt caffe.caffemodel ncnn.param ncnn.bin`

- ## mxnet

   ` ./mxnet2ncnn mxnet-symbol.json mxnet.params ncnn.param ncnn.bin`

- ## darknet

   [https://github.com/xiangweizeng/darknet2ncnn](https://github.com/xiangweizeng/darknet2ncnn)

- ## pytorch - onnx

   [use ncnn with pytorch or onnx](https://github.com/Tencent/ncnn/wiki/use-ncnn-with-pytorch-or-onnx)

- ## tensorflow 1.x/2.x - keras

   [https://github.com/MarsTechHAN/keras2ncnn](https://github.com/MarsTechHAN/keras2ncnn) **[@MarsTechHAN](https://github.com/MarsTechHAN)**

- ## tensorflow 2.x - mlir

[https://github.com/MarsTechHAN/keras2ncnn](https://github.com/MarsTechHAN/keras2ncnn) **[@MarsTechHAN](https://github.com/MarsTechHAN)**

   [Converting tensorflow2 models to ncnn via MLIR](https://zhuanlan.zhihu.com/p/152535430) **@[nihui](https://www.zhihu.com/people/nihui-2)**

- ## Shape not supported yet! Gather not supported yet! Cast not supported yet!

   onnx-simplifier shape

- ## convertmodel

   [https://convertmodel.com/](https://convertmodel.com/) **[@大老师](https://github.com/daquexian)**

- ## netron

   [https://github.com/lutzroeder/netron](https://github.com/lutzroeder/netron)

- ## How to generate a model with fixed shape？

   Input      0=w 1=h 2=c

- ## why gpu can speedup

- ## How to convert ncnnoptimize to fp16 model

   `ncnnoptimize model.param model.bin yolov5s-opt.param yolov5s-opt.bin 65536`

- ## How to use ncnnoptimize  checking the FLOPS / memory usage of your model

- ## How to modify the model to support dynamics shape？

   Interp Reshape

- ## How to convert a model into code embedded in a program？

   use ncnn2mem

- ## How to encrypt the model？

   See https://zhuanlan.zhihu.com/p/268327784

- ## The ncnn model transferred under Linux, Windows/MacOS/Android/... Can I use it directly?

   Yes, for all platforms

- ## How to remove post-processing and export onnx？

   Ref：

   Referring to an article by UP <https://zhuanlan.zhihu.com/p/128974102>, step 3 is to remove the post-processing and then export the onnx, where removing the post-processing can be the result of removing the subsequent steps when testing within the project.

- ## pytorch layers can't export to onnx？

 Mode 1:

   ONNX_ATEN_FALLBACK
Fully customizable op, first change to one that can export (e.g. concat slice), go to ncnn and then modify param

 Way 2.

 Mode 1:

 Way 2.

 You can try this with PNNX, see the following article for a general description:

   1. [Windows/Linux/macOS steps for compiling PNNX](https://zhuanlan.zhihu.com/p/431833958)

   2. [Learn in 5 minutes! Converting TorchScript models to ncnn models with PNNX](https://zhuanlan.zhihu.com/p/427512763)

# Using

- ## vkEnumeratePhysicalDevices failed -3

- ## vkCreateInstance failed -9

   Please upgrade your GPU driver if you meet this crash or error.
   Here are the download sites for some brands of GPU drivers. We have provided some driver download pages here.
   [Intel](https://downloadcenter.intel.com/product/80939/Graphics-Drivers), [AMD](https://www.amd.com/en/support), [Nvidia](https://) www.nvidia.com/Download/index.aspx)

- ## ModuleNotFoundError: No module named 'ncnn.ncnn'

   python setup.py develop

- ## fopen nanodet-m.param failed

   path should be working dir

   File not found or not readable. Make sure that XYZ.param/XYZ.bin is accessible.

- ## find_blob_index_by_name data / output / ... failed

   layer name vs blob name
   
   param.bin use xxx.id.h enum

- ## parse magic failed

- ## param is too old, please regenerate

   The model maybe has problems

   Your model file is being the old format converted by an old caffe2ncnn tool.

   Checkout the latest ncnn code, build it and regenerate param and model binary files, and that should work.

   Make sure that your param file starts with the magic number 7767517.

   you may find more info on use-ncnn-with-alexnet
   
   When adding the softmax layer yourself, you need to add 1=1

- ## set_vulkan_compute failed, network use_vulkan_compute disabled

   Set net.opt.use_vulkan_compute = true before load_param / load_model;

- ## How to exexcite multiple blob inputs, multiple blob outputs？
   Multiple execute `ex.input()` and `ex.extract()` like following
    ```
    ex.input("data1", in_1);
    ex.input("data2", in_2);
    ex.extract("output1", out_1);
    ex.extract("output2", out_2);
    ```
- ## Multiple executions of Extractor extract double the calculation？

   No

- ## How to see the elapsed time for every layer？

   cmake -DNCNN_BENCHMARK=ON ..

- ## How to convert a cv::Mat CV_8UC3 BGR image

   from_pixels to_pixels

- ## How to convert float data to ncnn::Mat

   First of all, you need to manage the memory you request yourself, at this point ncnn::Mat will not automatically free up the float data you pass over to it
   ``` c++
   std::vector<float> testData(60, 1.0); // use std::vector<float> to manage memory requests and releases yourself

``` c++
   ncnn::Mat in1 = ncnn::Mat(60, (void*)testData.data()).reshape(4, 5, 3); // just pass the pointer to the float data as a void*, and even specify the dimension (up says it's best to use reshape to solve the channel gap)
   float* a = new float[60]; // New a piece of memory yourself, you need to release it later
   ncnn::Mat in2 = ncnn::Mat(60, (void*)a).reshape(4, 5, 3).clone(); // use the same method as above, clone() to transfer data owner
   ```

# RTMPose inference with ONNXRuntime

This example shows how to run RTMPose inference with ONNXRuntime in Python.

### 1. Install onnxruntime inference engine.

Choose one of the following ways to install onnxruntime.

- CPU version

```bash
export ONNXRUNTIME_DIR=$(pwd)/onnxruntime-linux-x64-1.8.1
```

- GPU version

```bash
pip install onnxruntime-gpu==1.8.1
wget https://github.com/microsoft/onnxruntime/releases/download/v1.8.1/onnxruntime-linux-x64-gpu-1.8.1.tgz
tar -zxvf onnxruntime-linux-x64-gpu-1.8.1.tgz
export ONNXRUNTIME_DIR=$(pwd)/onnxruntime-linux-x64-gpu-1.8.1
```

### 2. Convert model to onnx files

- Install `mim` tool.

```bash
```

- Download `mmpose` model.

```bash

```

```bash
```

```bash
# choose one rtmpose model
mim download mmpose --config rtmpose-m_8xb64-270e_coco-wholebody-256x192 --dest .
```

- Clone `mmdeploy` repo.

```bash
git clone https://github.com/open-mmlab/mmdeploy.git
```

- Convert model to onnx files.

```bash
python mmdeploy/tools/deploy.py \
    mmdeploy/configs/mmpose/pose-detection_simcc_onnxruntime_dynamic.py \
    mmpose/rtmpose-m_8xb64-270e_coco-wholebody-256x192.py \
    mmpose/rtmpose-m_simcc-coco-wholebody_pt-aic-coco_270e-256x192-cd5e845c_20230123.pth \
    mmdeploy/demo/resources/human-pose.jpg \
    --work-dir mmdeploy_model/mmpose/ort \
```

## Run demo

### Install dependencies

```bash
```

### Usage:

```bash
python main.py \
    {ONNX_FILE} \
    {IMAGE_FILE} \

--device cuda \
```

## Run demo

```bash
```

### Usage:

```bash
    --device {DEVICE} \
    --save-path {SAVE_PATH}
```

### Description of all arguments

- `ONNX_FILE`: The path of onnx file
- `IMAGE_FILE`: The path of image file
- `DEVICE`: The device to run the model, default is `cpu`
- `SAVE_PATH`: The path to save the output image, default is `output.jpg`

# Customize Dataset

In this tutorial, we will introduce some methods about how to customize your own dataset by online conversion.

- [Customize Dataset](#customize-dataset)
  - [General understanding of the Dataset in MMAction2](#general-understanding-of-the-dataset-in-mmaction2)
  - [Customize new datasets](#customize-new-datasets)
  - [Customize keypoint format for PoseDataset](#customize-keypoint-format-for-posedataset)

## General understanding of the Dataset in MMAction2

- [Customize new datasets](#customize-new-datasets)

MMAction2 provides task-specific `Dataset` class, e.g. `VideoDataset`/`RawframeDataset` for action recognition, `AVADataset` for spatio-temporal action detection, `PoseDataset` for skeleton-based action recognition. These task-specific datasets only require the implementation of `load_data_list(self)` for generating a data list from the annotation file. The remaining functions are automatically handled by the superclass (i.e., `BaseActionDataset` and `BaseDataset`). The following table shows the inheritance relationship and the main method of the modules.

| Class Name                     | Class Method                                                                                                                                                               |

| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `MMAction2::VideoDataset`      | `load_data_list(self)` <br> Build data list from the annotation file.                                                                                                      |
| `MMAction2::BaseActionDataset` | `get_data_info(self, idx)` <br> Given the `idx`, return the corresponding data sample from the data list.                                                                  |
| `MMEngine::BaseDataset`        | `__getitem__(self, idx)` <br> Given the `idx`, call `get_data_info` to get the data sample, then call the `pipeline` to perform transforms and augmentation in `train_pipeline` or `val_pipeline` . |

## Customize new datasets

Although offline conversion is the preferred method for utilizing your own data in most cases, MMAction2 offers a convenient process for creating a customized `Dataset` class. As mentioned previously, task-specific datasets only require the implementation of `load_data_list(self)` for generating a data list from the annotation file. It is noteworthy that the elements in the `data_list` are `dict` with fields that are essential for the subsequent processes in the `pipeline`.

Taking `VideoDataset` as an example, `train_pipeline`/`val_pipeline` require `'filename'` in `DecordInit` and `'label'` in `PackActionInputs`. Consequently, the data samples in the `data_list` must contain 2 fields: `'filename'` and `'label'`.
Please refer to [customize pipeline](customize_pipeline.md) for more details about the `pipeline`.

```
```

```
```

However, `AVADataset` is more complex, data samples in the `data_list` consist of several fields about the video data. Moreover, it overwrites `get_data_info(self, idx)` to convert keys that are indispensable in the spatio-temporal action detection pipeline.

```python

class AVADataset(BaseActionDataset):
  ...

  def load_data_list(self) -> List[dict]:
      ...
        video_info = dict(
            frame_dir=frame_dir,
            video_id=video_id,
            timestamp=int(timestamp),
            img_key=img_key,
            shot_info=shot_info,
            fps=self._FPS,
            ann=ann)
            data_list.append(video_info)
        data_list.append(video_info)
      return data_list

  def get_data_info(self, idx: int) -> dict:
      ...
      ann = data_info.pop('ann')

data_list.append(video_info)

      ...
      data_info['gt_bboxes'] = ann['gt_bboxes']
      data_info['gt_labels'] = ann['gt_labels']
      data_info['entity_ids'] = ann['entity_ids']
      return data_info
```

## Customize keypoint format for PoseDataset

MMAction2 currently supports three keypoint formats: `coco`, `nturgb+d` and `openpose`. If you use one of these formats, you may simply specify the corresponding format in the following modules:

For Graph Convolutional Networks, such as AAGCN, STGCN, ...

- `pipeline`: argument `dataset` in `JointToBone`.
- `backbone`: argument `graph_cfg` in Graph Convolutional Networks.

For PoseC3D:

- `pipeline`: In `Flip`, specify `left_kp` and `right_kp` based on the symmetrical relationship between keypoints.

- `pipeline`: In `GeneratePoseTarget`, specify `skeletons`, `left_limb`, `right_limb` if `with_limb` is `True`, and `left_kp`, `right_kp` if `with_kp` is `True`.

If using a custom keypoint format, it is necessary to include a new graph layout in both the `backbone` and `pipeline`. This layout will define the keypoints and their connection relationship.

Taking the `coco` dataset as an example, we define a layout named `coco` in `Graph`. The `inward` connections of this layout comprise all node connections, with each **centripetal** connection consisting of a tuple of nodes. Additional settings for `coco` include specifying the number of nodes as `17` the `node 0` as the central node.

```python

self.num_node = 17
self.inward = [(15, 13), (13, 11), (16, 14), (14, 12), (11, 5),

```python

                (12, 6), (9, 7), (7, 5), (10, 8), (8, 6), (5, 0),
                (6, 0), (1, 0), (3, 1), (2, 0), (4, 2)]
self.center = 0
```

Similarly, we define the `pairs` in `JointToBone`, adding a bone of `(0, 0)` to align the number of bones to the nodes. The `pairs` of coco dataset are shown below, and the order of `pairs` in `JointToBone` is irrelevant.

```python

self.pairs = ((0, 0), (1, 0), (2, 0), (3, 1), (4, 2), (5, 0),
              (6, 0), (7, 5), (8, 6), (9, 7), (10, 8), (11, 0),
              (12, 0), (13, 11), (14, 12), (15, 13), (16, 14))
```

To use your custom keypoint format, simply define the aforementioned settings as your graph structure and specify them in your config file as shown below, In this example, we will use `STGCN`, with `n` denoting the number of classes and `custom_dataset` defined in `Graph` and `JointToBone`.

```python

```python
  type='RecognizerGCN',
  backbone=dict(
      type='STGCN', graph_cfg=dict(layout='custom_dataset', mode='stgcn_spatial')),
  cls_head=dict(type='GCNHead', num_classes=n, in_channels=256))

  ...
  dict(type='GenSkeFeat', dataset='custom_dataset'),
  ...]

  ...
  ...]

  ...
  ...]

```

Top-down methods divide the task into two stages: object detection, followed by single-object pose estimation given object bounding boxes Instead of estimating keypoint coordinates directly, the pose estimator will produce heatmaps which represent the
likelihood of being a keypoint, following the paradigm introduced in [Simple Baselines for Human Pose Estimation and Tracking](http://openaccess.thecvf.com/content_ECCV_2018/html/Bin_Xiao_Simple_Baselines_for_ECCV_2018_paper.html).

</div>

### Animal-Pose Dataset

|   Model    | Input Size |  AP   |  AR   |                   Details and Download                    |
| :--------: | :--------: | :---: | :---: | :-------------------------------------------------------: |

| HRNet-w32  |  256x256   | 0.740 | 0.780 |  [hrnet_animalpose.md](./animalpose/hrnet_animalpose.md)  |
| HRNet-w48  |  256x256   | 0.738 | 0.778 |  [hrnet_animalpose.md](./animalpose/hrnet_animalpose.md)  |
| ResNet-152 |  256x256   | 0.704 | 0.748 | [resnet_animalpose.md](./animalpose/resnet_animalpose.md) |
| ResNet-101 |  256x256   | 0.696 | 0.736 | [resnet_animalpose.md](./animalpose/resnet_animalpose.md) |
| ResNet-50  |  256x256   | 0.691 | 0.736 | [resnet_animalpose.md](./animalpose/resnet_animalpose.md) |

|   Model    | Input Size |  AP   |                 Details and Download                 |
| :--------: | :--------: | :---: | :--------------------------------------------------: |
| HRNet-w48  |  256x256   | 0.728 |       [hrnet_ap10k.md](./ap10k/hrnet_ap10k.md)       |

| HRNet-w32  |  256x256   | 0.722 |       [hrnet_ap10k.md](./ap10k/hrnet_ap10k.md)       |
| ResNet-101 |  256x256   | 0.681 |      [resnet_ap10k.md](./ap10k/resnet_ap10k.md)      |
| ResNet-50  |  256x256   | 0.680 |      [resnet_ap10k.md](./ap10k/resnet_ap10k.md)      |
| CSPNeXt-m  |  256x256   | 0.703 | [cspnext_udp_ap10k.md](./ap10k/cspnext_udp_ap10k.md) |

### Desert Locust Dataset

Results on Desert Locust test set

|   Model    | Input Size |  AUC  | EPE  |             Details and Download              |
| :--------: | :--------: | :---: | :--: | :-------------------------------------------: |
| ResNet-152 |  160x160   | 0.925 | 1.49 | [resnet_locust.md](./locust/resnet_locust.md) |
| ResNet-101 |  160x160   | 0.907 | 2.03 | [resnet_locust.md](./locust/resnet_locust.md) |

| ResNet-50  |  160x160   | 0.900 | 2.27 | [resnet_locust.md](./locust/resnet_locust.md) |

### Grévy’s Zebra Dataset

|   Model    | Input Size |  AUC  | EPE  |            Details and Download            |
| :--------: | :--------: | :---: | :--: | :----------------------------------------: |
| ResNet-152 |  160x160   | 0.921 | 1.67 | [resnet_zebra.md](./zebra/resnet_zebra.md) |
| ResNet-101 |  160x160   | 0.915 | 1.83 | [resnet_zebra.md](./zebra/resnet_zebra.md) |
| ResNet-50  |  160x160   | 0.914 | 1.87 | [resnet_zebra.md](./zebra/resnet_zebra.md) |

### Animal-Kingdom Dataset

Results on AnimalKingdom test set

|   Model   | Input Size |     class     | PCK(0.05) |                 Details and Download                  |

| :-------: | :--------: | :-----------: | :-------: | :---------------------------------------------------: |
| HRNet-w32 |  256x256   |      P1       |  0.6323   | [hrnet_animalkingdom.md](./ak/hrnet_animalkingdom.md) |
| HRNet-w32 |  256x256   |      P2       |  0.3741   | [hrnet_animalkingdom.md](./ak/hrnet_animalkingdom.md) |
| HRNet-w32 |  256x256   |  P3_mammals   |   0.571   | [hrnet_animalkingdom.md](./ak/hrnet_animalkingdom.md) |
| HRNet-w32 |  256x256   | P3_amphibians |  0.5358   | [hrnet_animalkingdom.md](./ak/hrnet_animalkingdom.md) |
| HRNet-w32 |  256x256   |  P3_reptiles  |   0.51    | [hrnet_animalkingdom.md](./ak/hrnet_animalkingdom.md) |
| HRNet-w32 |  256x256   |   P3_birds    |  0.7671   | [hrnet_animalkingdom.md](./ak/hrnet_animalkingdom.md) |

| HRNet-w32 |  256x256   |   P3_fishes   |  0.6406   | [hrnet_animalkingdom.md](./ak/hrnet_animalkingdom.md) |

# Whole-Body Human Pose Estimation in the Wild

<details>

```bibtex
}
```

# mlir2ncnn

## Compile

**Clone LLVM**
```bash
https://github.com/llvm/llvm-project.git
git checkout -b mlir <a_working_commit_id>
```
Current working commit id is 74e6030bcbcc8e628f9a99a424342a0c656456f9:
```
$ git log

commit 74e6030bcbcc8e628f9a99a424342a0c656456f9 (HEAD -> main, origin/main, origin/HEAD)
Author: Craig Topper <craig.topper@sifive.com>
Date:   Thu Mar 4 22:30:38 2021 -0800

    [TargetLowering] Use HandleSDNodes to prevent nodes from being deleted by recursive calls in getNegatedExpression.
```

It is determined by query lastest git commit date of `tools/mlir` directory.

**Compile mlir**
```bash
cd llvm-project
mkdir build
cd build
cmake -G Ninja -DCMAKE_INSTALL_PREFIX=install -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DLLVM_ENABLE_PROJECTS="mlir" -DLLVM_TARGETS_TO_BUILD="" -DLLVM_INCLUDE_EXAMPLES=OFF -DLLVM_INCLUDE_TESTS=OFF ../llvm/
ninja -j8
ninja install
```

**Compile mlir2ncnn**
```bash
cd tools/mlir
cd build

ninja -j8
```

```bash
cd build
cmake .. -D LLVM_DIR=<path/to/your/llvm_install/lib/cmake/llvm>
make
```

## Usage

**Export `.mlir`**

See https://zhuanlan.zhihu.com/p/152535430

**Usage mlir2ncnn**

```
./mlir2ncnn pix2pix.mlir pix2pix.param pix2pix.bin
```

# Understanding humans in crowded scenes: Deep nested adversarial learning and a new benchmark for multi-human parsing

<details>

```bibtex
}
```

# Annotated facial landmarks in the wild: A large-scale, real-world database for facial landmark localization

<details>

```bibtex
}
```

# Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution

> [Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution ](https://arxiv.org/abs/2007.16100)

Self-driving cars need to understand 3D scenes efficiently and accurately in order to drive safely. Given the limited hardware resources, existing 3D perception models are not able to recognize small instances (e.g., pedestrians, cyclists) very well due to the low-resolution voxelization and aggressive downsampling. To this end, we propose Sparse Point-Voxel Convolution (SPVConv), a lightweight 3D module that equips the vanilla Sparse Convolution with the high-resolution point-based branch. With negligible overhead, this point-based branch is able to preserve the fine details even from large outdoor scenes. To explore the spectrum of efficient 3D models, we first define a flexible architecture design space based on SPVConv, and we then present 3D Neural Architecture Search (3D-NAS) to search the optimal network architecture over this diverse design space efficiently and effectively. Experimental results validate that the resulting SPVNAS model is fast and accurate: it outperforms the state-of-the-art MinkowskiNet by 3.3%, ranking 1st on the competitive SemanticKITTI leaderboard. It also achieves 8x computation reduction and 3x measured speedup over MinkowskiNet with higher accuracy. Finally, we transfer our method to 3D object detection, and it achieves consistent improvements over the one-stage detection baseline on KITTI.

<img src="https://user-images.githubusercontent.com/72679458/226509154-80c27d8e-c138-426a-b92e-72846997b5b3.png" width="800"/>
</div>

We implement SPVCNN with [TorchSparse](https://github.com/mit-han-lab/torchsparse) backend and provide the result and checkpoints on SemanticKITTI datasets.

|                                 Method                                  | Lr schd | Laser-Polar Mix | Mem (GB) | mIoU |                                                                                                                                                                    Download                                                                                                                                                                     |

| :---------------------------------------------------------------------: | :-----: | :-------------: | :------: | :--: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|        [SPVCNN-W16](./spvcnn_w16_8xb2-amp-15e_semantickitti.py)         |   15e   |        ✗        |   3.9    | 61.8 | [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/spvcnn/spvcnn_w16_8xb2-15e_semantickitti/spvcnn_w16_8xb2-15e_semantickitti_20230321_011645-a2734d85.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/spvcnn/spvcnn_w16_8xb2-15e_semantickitti/spvcnn_w16_8xb2-15e_semantickitti_20230321_011645.log) |

|        [SPVCNN-W20](./spvcnn_w20_8xb2-amp-15e_semantickitti.py)         |   15e   |        ✗        |   4.2    | 62.6 | [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/spvcnn/spvcnn_w20_8xb2-15e_semantickitti/spvcnn_w20_8xb2-15e_semantickitti_20230321_011649-519e7eff.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/spvcnn/spvcnn_w20_8xb2-15e_semantickitti/spvcnn_w20_8xb2-15e_semantickitti_20230321_011649.log) |
|        [SPVCNN-W32](./spvcnn_w32_8xb2-amp-15e_semantickitti.py)         |   15e   |        ✗        |   5.4    | 64.3 | [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/spvcnn/spvcnn_w32_8xb2-15e_semantickitti/spvcnn_w32_8xb2-15e_semantickitti_20230308_113324-f7c0c5b4.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/spvcnn/spvcnn_w32_8xb2-15e_semantickitti/spvcnn_w32_8xb2-15e_semantickitti_20230308_113324.log) |

| [SPVCNN-W32](./spvcnn_w32_8xb2-amp-laser-polar-mix-3x_semantickitti.py) |   3x    |        ✔        |   7.2    | 68.7 |                [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/spvcnn/spvcnn_w32_8xb2-amp-laser-polar-mix-3x_semantickitti_20230425_125908-d68a68b7.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/spvcnn/spvcnn_w32_8xb2-amp-laser-polar-mix-3x_semantickitti_20230425_125908.log)                |

**Note:** We follow the implementation in SPVNAS original [repo](https://github.com/mit-han-lab/spvnas) and W16\\W20\\W32 indicates different number of channels.

**Note:** Due to TorchSparse backend, the model performance is unstable with TorchSparse backend and may fluctuate by about 1.5 mIoU for different random seeds.

```latex
@inproceedings{tang2020searching,
  title={Searching efficient 3d architectures with sparse point-voxel convolution},

```latex
  author={Tang, Haotian and Liu, Zhijian and Zhao, Shengyu and Lin, Yujun and Lin, Ji and Wang, Hanrui and Han, Song},
  booktitle={Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XXVIII},
  pages={685--702},
}
```

# Customize Datasets

In this note, you will know how to train and test predefined models with customized datasets.

1. Prepare data
3. Train, test and inference models on the customized dataset

The ideal situation is that we can reorganize the customized raw data and convert the annotation format into KITTI style. However, considering some calibration files and 3D annotations in KITTI format are difficult to obtain for customized datasets, we introduce the basic data format in the doc.

### Basic Data Format

#### Point cloud Format

Currently, we only support `.bin` format point cloud for training and inference. Before training on your own datasets, you need to convert your point cloud files with other formats to `.bin` files. The common point cloud data formats include `.pcd` and `.las`, we list some open-source tools for reference.

1. Convert `.pcd` to `.bin`: https://github.com/DanielPollithy/pypcd

- You can install `pypcd` with the following command:

  ```bash
  ```

- You can use the following script to read the `.pcd` file and convert it to `.bin` format for saving:

  ```python

  ```

points[:, 3] = pcd_data.pc_data['intensity'].copy().astype(np.float32)
  ```

2. Convert `.las` to `.bin`: The common conversion path is `.las -> .pcd -> .bin`, and the conversion path `.las -> .pcd` can be achieved through [this tool](https://github.com/Hitachi-Automotive-And-Industry-Lab/semantic-segmentation-editor).

#### Label Format

The most basic information: 3D bounding box and category label of each scene need to be contained in the `.txt` annotation file. Each line represents a 3D box in a certain scene as follow:

```
# format: [x, y, z, dx, dy, dz, yaw, category_name]
...
```

**Note**: Currently we only support KITTI Metric evaluation for customized datasets evaluation.

The 3D Box should be stored in unified 3D coordinates.

#### Calibration Format

...
```

For the point cloud data collected by each LiDAR, they are usually fused and converted to a certain LiDAR coordinate. So typically the calibration information file should contain the intrinsic matrix of each camera and the transformation extrinsic matrix from the LiDAR to each camera in `.txt` calibration file, while `Px` represents the intrinsic matrix of `camera_x` and `lidar2camx` represents the transformation extrinsic matrix from the `lidar` to `camera_x`.

```
P0
P1
P2
P3
P4
...
...
```

### Raw Data Structure

#### LiDAR-Based 3D Detection

```
P0
P1
P2
P3
P4
...
...
```

The raw data for LiDAR-based 3D object detection are typically organized as follows, where `ImageSets` contains split files indicating which files belong to training/validation set, `points` includes point cloud data which are supposed to be stored in `.bin` format and `labels` includes label files for 3D detection.

```
├── tools
├── data
│   │   │   ├── ...
│   │   │   ├── ...
```

#### Vision-Based 3D Detection

│   │   │   ├── ...
│   │   │   ├── ...
```

The raw data for vision-based 3D object detection are typically organized as follows, where `ImageSets` contains split files indicating which files belong to training/validation set, `images` contains the images from different cameras, for example, images from `camera_x` need to be placed in `images/images_x`, `calibs` contains calibration information files which store the camera intrinsic matrix of each camera, and `labels` includes label files for 3D detection.

```
├── tools
├── data
│   │   │   ├── ...

│   │   │   ├── ...
│   │   │   ├── ...
│   │   │   ├── ...
```

#### Multi-Modality 3D Detection

The raw data for multi-modality 3D object detection are typically organized as follows. Different from vision-based 3D object detection, calibration information files in `calibs` store the camera intrinsic matrix of each camera and extrinsic matrix.

```
├── tools
├── data
│   │   │   ├── ...

│   │   │   ├── ...
│   │   │   ├── ...
│   │   │   ├── ...
│   │   │   ├── ...
```

#### LiDAR-Based 3D Semantic Segmentation

The raw data for LiDAR-based 3D semantic segmentation are typically organized as follows, where `ImageSets` contains split files indicating which files belong to training/validation set, `points` includes point cloud data, and `semantic_mask` includes point-level label.

```
├── tools
├── data

```
├── tools
├── data
│   │   │   ├── ...
│   │   │   ├── ...
```

### Data Converter

Once you prepared the raw data following our instruction, you can directly use the following command to generate training/validation information files.

```bash
```

## An example of customized dataset

Once we finish data preparation, we can create a new dataset in `mmdet3d/datasets/my_dataset.py` to load the data.

```python

```python

    # replace with all the classes in customized pkl info file
    }

        Args:

        Returns:

        """
            # empty instance

if ann_info is None:

        # filter the gt classes not used in training
```

After the data pre-processing, there are two steps for users to train the customized new dataset:

Here we take training PointPillars on customized dataset as an example:

### Prepare a config

Here we demonstrate a config sample for pure point cloud training.

#### Prepare dataset config

In `configs/_base_/datasets/custom.py`:

```python

```python
class_names = ['Pedestrian', 'Cyclist', 'Car']  # replace with your dataset class
point_cloud_range = [0, -40, -3, 70.4, 40, 1]  # adjust according to your dataset

    dict(
        load_dim=4,  # replace with your point cloud data dimension
        use_dim=4),  # replace with the actual dimension used in training and inference
    dict(
    dict(

with_bbox_3d=True,
    dict(
    dict(
    dict(
]
    dict(
]

coord_type='LIDAR',
]
# construct a pipeline for data and gt loading in show function
]
        times=2,
            ann_file='custom_infos_train.pkl',  # specify your training pkl info

pipeline=train_pipeline,
        ann_file='custom_infos_val.pkl',  # specify your validation pkl info
    ann_file=data_root + 'custom_infos_val.pkl',  # specify your validation pkl info
```

#### Prepare model config

```

For voxel-based detectors such as SECOND, PointPillars and CenterPoint, the point cloud range and voxel size should be adjusted according to your dataset.
Theoretically, `voxel_size` is linked to the setting of `point_cloud_range`. Setting a smaller `voxel_size` will increase the voxel num and the corresponding memory consumption. In addition, the following issues need to be noted:

If the `point_cloud_range` and `voxel_size` are set to be `[0, -40, -3, 70.4, 40, 1]` and `[0.05, 0.05, 0.1]` respectively, then the shape of intermediate feature map should be `[(1-(-3))/0.1+1, (40-(-40))/0.05, (70.4-0)/0.05]=[41, 1600, 1408]`. When changing `point_cloud_range`, remember to change the shape of intermediate feature map in `middle_encoder` according to the `voxel_size`.

Regarding the setting of `anchor_range`, it is generally adjusted according to dataset. Note that `z` value needs to be adjusted accordingly to the position of the point cloud, please refer to this [issue](https://github.com/open-mmlab/mmdetection3d/issues/986).

Regarding the setting of `anchor_size`, it is usually necessary to count the average length, width and height of objects in the entire training dataset as `anchor_size` to obtain the best results.

In `configs/_base_/models/pointpillars_hv_secfpn_custom.py`:

```python
voxel_size = [0.16, 0.16, 4]  # adjust according to your dataset
point_cloud_range = [0, -39.68, -3, 69.12, 39.68, 1]  # adjust according to your dataset

voxel_layer=dict(
    # the `output_shape` should be adjusted according to `point_cloud_range`
    # and `voxel_size`

in_channels=[64, 128, 256],
        # adjust the `ranges` and `sizes` according to your dataset
            ranges=[
            ],

diff_rad_by_sin=True,

dict(  # for Cyclist
        ],
```

#### Prepare overall config

We combine all the configs above in `configs/pointpillars/pointpillars_hv_secfpn_8xb6_custom.py`:

```python
_base_ = [

nms_pre=100,
```

```python
_base_ = [
]
```

#### Visualize your dataset (optional)

To validate whether your prepared data and config are correct, it's highly recommended to use `tools/misc/browse_dataset.py` script
to visualize your dataset and annotations before training and validation. Please refer to [visualization doc](https://mmdetection3d.readthedocs.io/en/dev-1.x/user_guides/visualization.html) for more details.

Once the data and config have been prepared, you can directly run the training/testing script following our doc.

**Note**: We only provide an implementation for KITTI style evaluation for the customized dataset. It should be included in the dataset config:

```python
```

# Code Migration

MMOCR has been designed in a way that there are a lot of shortcomings in the initial version in order to balance the tasks of text detection, recognition and key information extraction. In this 1.0 release, MMOCR synchronizes its new model architecture to align as much as possible with the overall OpenMMLab design and to achieve structural uniformity within the algorithm library. Although this upgrade is not fully backward compatible, we summarize the changes that may be of interest to developers for those who need them.

## Fundamental Changes

Functional boundaries of modules has not been clearly defined in MMOCR 0.x. In MMOCR 1.0, we address this issue by refactoring the design of model modules. Here are some major changes in 1.0:

- MMOCR 1.0 no longer supports named entity recognition tasks since it's not in the scope of OCR.

- The module that computes the loss in a model is named as *Module Loss*, which is also responsible for the conversion of gold annotations into loss targets. Another module, *Postprocessor*, is responsible for decoding the model raw output into `DataSample` for the corresponding task at prediction time.

- The inputs of all models are now organized as a dictionary that consists of two keys: `inputs`, containing the original features of the images, and `List[DataSample]`, containing the meta-information of the images. At training time, the output format of a model is standardized to a dictionary containing the loss tensors. Similarly, a model generates a sequence of `DataSample`s containing the prediction outputs in testing.

- In MMOCR 0.x, the majority of classes named `XXLoss` have the implementations closely bound to the corresponding model, while their names made users hard to tell them apart from other generic losses like `DiceLoss`. In 1.0, they are renamed to the form `XXModuleLoss`. (e.g. `DBLoss` was renamed to `DBModuleLoss`). The key to their configurations in config files is also changed from `loss` to `module_loss`.

- The names of generic loss classes that are not related to the model implementation are kept as `XXLoss`. (e.g. [`MaskedBCELoss`](mmocr.models.common.losses.MaskedBCELoss)) They are all placed under `mmocr/models/common/losses`.

- Changes under `mmocr/models/common/losses`: `DiceLoss` is renamed to [`MaskedDiceLoss`](mmocr.models.common.losses.MaskedDiceLoss). `FocalLoss` has been removed.

- MMOCR 1.0 adds a *Dictionary* module which originates from *label converter*. It is used in text recognition and key information extraction tasks.

## Text Detection Models

### Key Changes (TL;DR)

- The model weights from MMOCR 0.x still works in the 1.0, but the fields starting with `bbox_head` in the state dict `state_dict` need to be renamed to `det_head`.

- `XXTargets` transforms, which were responsible for genearting detection targets, have been merged into `XXModuleLoss`.

### SingleStageTextDetector

- The original inheritance chain was `mmdet.BaseDetector->SingleStageDetector->SingleStageTextDetector`. Now `SingleStageTextDetector` is directly inherited from `BaseDetector` without extra dependency on MMDetection, and `SingleStageDetector` is deleted.

- `bbox_head` is renamed to `det_head`.

- `train_cfg`, `test_cfg` and `pretrained` fields are removed.

- `forward_train()` and `simple_test()` are refactored to `loss()` and `predict()`. The part of `simple_test()` that was responsible for splitting the raw output of the model and feeding it into `head.get_bounary()` is integrated into `BaseTextDetPostProcessor`.

- `TextDetectorMixin` has been removed since its implementation overlaps with `TextDetLocalVisualizer`.

### Head

- `HeadMixin`, the base class that `XXXHead` had to inherit from in version 0.x, has been replaced by `BaseTextDetHead`. `get_boundary()` and `resize_boundary()` are now rewritten as `__call__()` and `rescale()`  in `BaseTextDetPostProcessor`.

### ModuleLoss

- Data transforms `XXXTargets` in text detection tasks are all moved to `XXXModuleLoss._get_target_single()`. Target-related configurations are no longer specified in the data pipeline but in `XXXLoss` instead.

### Postprocessor

- The logic in the original `XXXPostprocessor.__call__()` are transferred to the refactored `XXXPostprocessor.get_text_instances()`.

- `BasePostprocessor` is refactored to `BaseTextDetPostProcessor`. This base class splits and processes the model output predictions one by one and supports automatic scaling of the output polygon or bounding box based on `scale_factor`.

## Text Recognition

- Due to the change of the character order and some bugs in the model architecture being fixed, the recognition model weights in 0.x can no longer be directly used in 1.0. We will provide a migration script and tutorial for those who need it.

- The support of SegOCR has been removed. TPS-CRNN will still be supported in a later version.

- Test time augmentation will be supported in the upcoming release.

- *Label converter* module has been removed and its functions have been split into *Dictionary*, *ModuleLoss* and *Postprocessor*.

- The definition of `max_seq_len` has been unified and now it represents the original output length of the model.

### Label Converter

- The original label converters had spelling errors (written as label convertors). We fixed them by removing label converters from this project.

- The part responsible for converting characters/strings to and from numeric indexes was extracted to *Dictionary*.

- In older versions, different label converters would have different special character sets and character order. In version 0.x, the character order was as follows.

| Converter                       | Character order                           |
| ------------------------------- | ----------------------------------------- |

| `AttnConvertor`, `ABIConvertor` | `<UKN>`, `<BOS/EOS>`, `<PAD>`, characters |
| `CTCConvertor`                  | `<BLK>`, `<UKN>`, characters              |

In 1.0, instead of designing different dictionaries and character orders for different tasks, we have a unified *Dictionary* implementation with the character order always as characters, \<BOS/EOS>, \<PAD>, \<UKN>. \<BLK> in `CTCConvertor` has been equivalently replaced by \<PAD>.

- *Label convertor* originally supported three ways to initialize dictionaries: `dict_type`, `dict_file` and `dict_list`, which are now reduced to `dict_file` only in `Dictionary`. Also, we have put those pre-defined character sets originally supported in `dict_type` into `dicts/` directory now. The corresponding mapping is as follows:

| MMOCR 0.x: `dict_type` | MMOCR 1.0: Dict path                   |
  | ---------------------- | -------------------------------------- |
  | DICT90                 | dicts/english_digits_symbols.txt       |
  | DICT91                 | dicts/english_digits_symbols_space.txt |
  | DICT36                 | dicts/lower_english_digits.txt         |
  | DICT37                 | dicts/lower_english_digits_space.txt   |

- The implementation of `str2tensor()` in *label converter* has been moved to `ModuleLoss.get_targets()`. The following table shows the correspondence between the old and new method implementations. Note that the old and new implementations are not identical.

  | MMOCR 0.x                                                 | MMOCR 1.0                               | Note                                                                                                     |

| --------------------------------------------------------- | --------------------------------------- | -------------------------------------------------------------------------------------------------------- |
  | `ABIConvertor.str2tensor()`, `AttnConvertor.str2tensor()` | `BaseTextRecogModuleLoss.get_targets()` | The different implementations between `ABIConvertor.str2tensor()` and `AttnConvertor.str2tensor()` have been unified in the new version. |
  | `CTCConvertor.str2tensor()`                               | `CTCModuleLoss.get_targets()`           |                                                                                                          |

- The implementation of `tensor2idx()` in *label converter* has been moved to `Postprocessor.get_single_prediction()`. The following table shows the correspondence between the old and new method implementations. Note that the old and new implementations are not identical.

| MMOCR 0.x                                                 | MMOCR 1.0                                        |
  | --------------------------------------------------------- | ------------------------------------------------ |
  | `ABIConvertor.tensor2idx()`, `AttnConvertor.tensor2idx()` | `AttentionPostprocessor.get_single_prediction()` |
  | `CTCConvertor.tensor2idx()`                               | `CTCPostProcessor.get_single_prediction()`       |

## Key Information Extraction

- Due to changes in the inputs to the model, the model weights obtained in 0.x can no longer be directly used in 1.0.

### KIEDataset & OpensetKIEDataset

- The part that reads data is kept in `WildReceiptDataset`.

- The part that additionally processes the nodes and edges is moved to `LoadKIEAnnotation`.

- The part that uses dictionaries to transform text is moved to `SDMGRHead.convert_text()`, with the help of *Dictionary*.

- The part of `compute_relation()` that computes the relationships between text boxes is moved to `SDMGRHead.compute_relations()`. It's now done inside the model.

- The part that evaluates the model performance is done in [`F1Metric`](mmocr.evaluation.metric.F1Metric).

- The part of `OpensetKIEDataset` that processes model's edge outputs is moved to `SDMGRPostProcessor`.

### SDMGR

- `show_result()` is integrated into `KIEVisualizer`.

- The part of `forward_test()` that post-processes the output is organized in `SDMGRPostProcessor`.

## Utils Migration

Utility functions are now grouped together under `mmocr/utils/`. Here are the scopes of the files in this directory:

- bbox_utils.py: bounding box related functions.
- check_argument.py: used to check argument type.
- collect_env.py: used to collect running environment.

- data_converter_utils.py: used for data format conversion.
- fileio.py: file input and output related functions.
- img_utils.py: image processing related functions.
- mask_utils.py: mask related functions.
- ocr.py: used for MMOCR inference.
- parsers.py: used for parsing datasets.
- polygon_utils.py: polygon related functions.
- setup_env.py: used for initialize MMOCR.
- string_utils.py: string related functions.
- typing.py: defines the abbreviation of types used in MMOCR.

In OpenMMLab, all the inference operations are unified into a new interface - `Inferencer`. `Inferencer` is designed to expose a neat and simple API to users, and shares very similar interface across different OpenMMLab libraries.

In MMOCR, Inferencers are constructed in different levels of task abstraction.

- Standard Inferencer: Following OpenMMLab's convention, each fundamental task in MMOCR has a standard Inferencer, namely `TextDetInferencer` (text detection), `TextRecInferencer` (text recognition), `TextSpottingInferencer` (end-to-end OCR), and `KIEInferencer` (key information extraction). They are designed to perform inference on a single task, and can be chained together to perform inference on a series of tasks. They also share very similar interface, have standard input/output protocol, and overall follow the OpenMMLab design.

- **MMOCRInferencer**: We also provide `MMOCRInferencer`, a convenient inference interface only designed for MMOCR. It encapsulates and chains all the Inferencers in MMOCR, so users can use this Inferencer to perform a series of tasks on an image and directly get the final result in an end-to-end manner. *However, it has a relatively different interface from other standard Inferencers, and some of standard Inferencer functionalities might be sacrificed for the sake of simplicity.*

For new users, we recommend using **MMOCRInferencer** to test out different combinations of models.

If you are a developer and wish to integrate the models into your own project, we recommend using **standard Inferencers**, as they are more flexible and standardized, equipped with full functionalities.

## Basic Usage

````{group-tab} MMOCRInferencer

As of now, `MMOCRInferencer` can perform inference on the following tasks:

- Text detection
- Text recognition

- OCR (text detection + text recognition)
- Key information extraction (text detection + text recognition + key information extraction)
- *OCR (text spotting)* (coming soon)

For convenience, `MMOCRInferencer` provides both Python and command line interfaces. For example, if you want to perform OCR inference on `demo/demo_text_ocr.jpg` with `DBNet` as the text detection model and `CRNN` as the text recognition model, you can simply run the following command:

::::{tabs}

:::{code-tab} python
>>> from mmocr.apis import MMOCRInferencer
>>> # Load models into memory
>>> ocr = MMOCRInferencer(det='DBNet', rec='SAR')
>>> # Perform inference
>>> ocr('demo/demo_text_ocr.jpg', show=True)
:::

:::{code-tab} bash
python tools/infer.py demo/demo_text_ocr.jpg --det DBNet --rec SAR --show
:::
::::

:::

:::
::::

The resulting OCR output will be displayed in a new window:

    <img src="https://user-images.githubusercontent.com/22607038/220563262-e9c1ab52-9b96-4d9c-bcb6-f55ff0b9e1be.png" height="250"/>
</div>

```{note}
If you are running MMOCR on a server without GUI or via SSH tunnel with X11 forwarding disabled, the `show` option will not work. However, you can still save visualizations to files by setting `out_dir` and `save_vis=True` arguments. Read [Dumping Results](#dumping-results) for details.
```

Depending on the initialization arguments, `MMOCRInferencer` can run in different modes. For example, it can run in KIE mode if it is initialized with `det`, `rec` and `kie` specified.

>>> kie = MMOCRInferencer(det='DBNet', rec='SAR', kie='SDMGR')

>>> kie('demo/demo_kie.jpeg', show=True)
:::

python tools/infer.py demo/demo_kie.jpeg --det DBNet --rec SAR --kie SDMGR --show
:::

::::

The output image should look like this:

    <img src="https://user-images.githubusercontent.com/22607038/220569700-fd4894bc-f65a-405e-95e7-ebd2d614aedd.png" height="250"/>
</div>
<br />

You may have found that the Python interface and the command line interface of `MMOCRInferencer` are very similar. The following sections will use the Python interface as an example to introduce the usage of `MMOCRInferencer`. For more information about the command line interface, please refer to [Command Line Interface](#command-line-interface).

````

````{group-tab} Standard Inferencer

````

In general, all the standard Inferencers across OpenMMLab share a very similar interface. The following example shows how to use `TextDetInferencer` to perform inference on a single image.

```python
>>> from mmocr.apis import TextDetInferencer
>>> inferencer = TextDetInferencer(model='DBNet')
>>> # Inference
>>> inferencer('demo/demo_text_ocr.jpg', show=True)
```

The visualization result should look like:

    <img src="https://user-images.githubusercontent.com/22607038/221418215-2431d0e9-e16e-4deb-9c52-f8b86801706a.png" height="250"/>
</div>

````

`````

## Initialization

Each Inferencer must be initialized with a model. You can also choose the inference device during initialization.

### Model Initialization

For each task, `MMOCRInferencer` takes two arguments in the form of `xxx` and `xxx_weights` (e.g. `det` and `det_weights`) for initialization, and there are many ways to initialize a model for inference. We will take `det` and `det_weights` as an example to illustrate some typical ways to initialize a model.

- To infer with MMOCR's pre-trained model, passing its name to the argument `det` can work. The weights will be automatically downloaded and loaded from OpenMMLab's model zoo. Check [Weights](../modelzoo.md#weights) for available model names.

  ```python
  >>> MMOCRInferencer(det='DBNet')
  ```

- To load custom config and weight, you can pass the path to the config file to `det` and the path to the weight to `det_weights`.

  ```python

```python
  ```

  ```python
  >>> MMOCRInferencer(det='path/to/dbnet_config.py', det_weights='path/to/dbnet.pth')
  ```

You may click on the "Standard Inferencer" tab to find more initialization methods.

````

Every standard `Inferencer` accepts two parameters, `model` and `weights`. (In `MMOCRInferencer`, they are referred to as `xxx` and `xxx_weights`)

- `model` takes either the name of a model, or the path to a config file as input. The name of a model is obtained from the model's metafile ([Example](https://github.com/open-mmlab/mmocr/blob/1.x/configs/textdet/dbnet/metafile.yml)) indexed from [model-index.yml](https://github.com/open-mmlab/mmocr/blob/1.x/model-index.yml). You can find the list of available weights [here](../modelzoo.md#weights).

- `weights` accepts the path to a weight file.

<br />

There are various ways to initialize a model.

- To infer with MMOCR's pre-trained model,  you can pass its name to `model`. The weights will be automatically downloaded and loaded from OpenMMLab's model zoo.

  ```python
  >>> from mmocr.apis import TextDetInferencer
  >>> inferencer = TextDetInferencer(model='DBNet')
  ```

  ```{note}
  The model type must match the Inferencer type.
  ```

  You can load another weight by passing its path/url to `weights`.

  ```python
  >>> inferencer = TextDetInferencer(model='DBNet', weights='path/to/dbnet.pth')
  ```

- To load custom config and weight, you can pass the path to the config file to `model` and the path to the weight to `weights`.

  ```python
  >>> inferencer = TextDetInferencer(model='path/to/dbnet_config.py', weights='path/to/dbnet.pth')
  ```

```python
  ```

- By default, [MMEngine](https://github.com/open-mmlab/mmengine/) dumps config to the weight. If you have a weight trained on MMEngine, you can also pass the path to the weight file to `weights` without specifying `model`:

  ```python
  >>> # It will raise an error if the config file cannot be found in the weight
  >>> inferencer = TextDetInferencer(weights='path/to/dbnet.pth')
  ```

- Passing config file to `model` without specifying `weight` will result in a randomly initialized model.

````
`````

### Device

Each Inferencer instance is bound to a device.
By default, the best device is automatically decided by [MMEngine](https://github.com/open-mmlab/mmengine/). You can also alter the device by specifying the `device` argument. For example, you can use the following code to create an Inferencer on GPU 1.

```python

```python
>>> inferencer = MMOCRInferencer(det='DBNet', device='cuda:1')
```

````

```python
>>> inferencer = TextDetInferencer(model='DBNet', device='cuda:1')
```

````

`````

To create an Inferencer on CPU:

```python
>>> inferencer = MMOCRInferencer(det='DBNet', device='cpu')
```

````

```python
>>> inferencer = TextDetInferencer(model='DBNet', device='cpu')
```

````

`````

Refer to [torch.device](torch.device) for all the supported forms.

## Inference

Once the Inferencer is initialized, you can directly pass in the raw data to be inferred and get the inference results from return values.

### Input

````{tab} MMOCRInferencer / TextDetInferencer / TextRecInferencer / TextSpottingInferencer

Input can be either of these types:

- str: Path/URL to the image.

  ```python

### Input

  ```python
  >>> inferencer('demo/demo_text_ocr.jpg')
  ```

- array: Image in numpy array. It should be in BGR order.

  ```python
  >>> import mmcv
  >>> array = mmcv.imread('demo/demo_text_ocr.jpg')
  >>> inferencer(array)
  ```

- list: A list of basic types above. Each element in the list will be processed separately.

  ```python
  >>> inferencer(['img_1.jpg', 'img_2.jpg])
  >>> # You can even mix the types
  >>> inferencer(['img_1.jpg', array])
  ```

- str: Path to the directory. All images in the directory will be processed.

  ```python
  >>> inferencer('tests/data/det_toy_dataset/imgs/test/')
  ```

````

````{tab} KIEInferencer

Input can be a dict or list[dict], where each dictionary contains
following keys:

```python
  ```

````

- `img` (str or ndarray): Path to the image or the image itself. If KIE Inferencer is used in no-visual mode, this key is not required.
If it's an numpy array, it should be in BGR order.
- `img_shape` (tuple(int, int)): Image shape in (H, W). Only required when KIE Inferencer is used in no-visual mode and no `img` is provided.
- `instances` (list[dict]): A list of instances.

Each `instance` looks like the following:

```python
{
    # A nested list of 4 numbers representing the bounding box of
    # the instance, in (x1, y1, x2, y2) order.
    "bbox": np.array([[x1, y1, x2, y2], [x1, y1, x2, y2], ...],
                    dtype=np.int32),

    # List of texts.
    "texts": ['text1', 'text2', ...],
}
```

````
`````

### Output

"bbox": np.array([[x1, y1, x2, y2], [x1, y1, x2, y2], ...],

}
```

````
`````

### Output

By default, each `Inferencer` returns the prediction results in a dictionary format.

- `visualization` contains the visualized predictions. But it's an empty list by default unless `return_vis=True`.

- `predictions` contains the predictions results in a json-serializable format. As presented below, the contents are slightly different depending on the task type.

  `````{tabs}

  :::{group-tab} MMOCRInferencer

  ```python
  {
      'predictions' : [
        # Each instance corresponds to an input image
        {
          'det_polygons': [...],  # 2d list of length (N,), format: [x1, y1, x2, y2, ...]
          'det_scores': [...],  # float list of length (N,)
          'det_bboxes': [...],   # 2d list of shape (N, 4), format: [min_x, min_y, max_x, max_y]

'det_scores': [...],  # float list of length (N,)
          'rec_texts': [...],  # str list of length (N,)
          'rec_scores': [...],  # float list of length (N,)
          'kie_labels': [...],  # node labels, length (N, )
          'kie_scores': [...],  # node scores, length (N, )
          'kie_edge_scores': [...],  # edge scores, shape (N, N)
          'kie_edge_labels': [...]  # edge labels, shape (N, N)
        },
        ...
      ],
      'visualization' : [
        array(..., dtype=uint8),
      ]
  }
  ```

  :::

  :::{group-tab} Standard Inferencer

  ````{tabs}
  ```{code-tab} python TextDetInferencer

  {
        {
          'polygons': [...],  # 2d list of len (N,) in the format of [x1, y1, x2, y2, ...]

{
        {
          'bboxes': [...],  # 2d list of shape (N, 4), in the format of [min_x, min_y, max_x, max_y]
          'scores': [...]  # list of float, len (N, )
        },
      ]
      ]
  }
  ```

  ```{code-tab} python TextRecInferencer
  {
        {
          'text': '...',  # a string
          'scores': 0.1,  # a float
        },
        ...
      ]
      ]
  }
  ```

  ```{code-tab} python TextSpottingInferencer
  {
        {

{
        {
          'texts': ['...',]  # list of texts, len (N, )
        },
      ]
      ]
  }
  ```

  ```{code-tab} python KIEInferencer
  {
        {
          'labels': [...],  # node label, len (N,)
          'scores': [...],  # node scores, len (N, )
          'edge_scores': [...],  # edge scores, shape (N, N)
          'edge_labels': [...],  # edge labels, shape (N, N)
        },
      ]
      ]
  }
  ```
  ````

  :::

  `````

'edge_labels': [...],  # edge labels, shape (N, N)
        },
      ]
      ]
  }
  ```
  ````

  :::

  `````

If you wish to get the raw outputs from the model, you can set `return_datasamples` to `True` to get the original [DataSample](structures.md), which will be stored in `predictions`.

### Dumping Results

Apart from obtaining predictions from the return value, you can also export the predictions/visualizations to files by setting `out_dir` and `save_pred`/`save_vis` arguments.

```python
>>> inferencer('img_1.jpg', out_dir='outputs/', save_pred=True, save_vis=True)
```

Results in the directory structure like:

```text
outputs
├── preds
│   └── img_1.json
└── vis
    └── img_1.jpg
```

The filename of each file is the same as the corresponding input image filename. If the input image is an array, the filename will be a number starting from 0.

### Batch Inference

```

You can customize the batch size by setting `batch_size`. The default batch size is 1.

## API

Here are extensive lists of parameters that you can use.

````{tabs}

```{group-tab} MMOCRInferencer

**MMOCRInferencer.\_\_init\_\_():**

| Arguments     | Type                                                 | Default | Description                                                                                                                                                                      |
| ------------- | ---------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

| `det`         | str or [Weights](../modelzoo.html#weights), optional | None    | Pretrained text detection algorithm. It's the path to the config file or the model name defined in metafile.                                                                     |
| `det_weights` | str, optional                                        | None    | Path to the custom checkpoint file of the selected det model. If it is not specified and "det" is a model name of metafile, the weights will be loaded from metafile.            |
| `rec`         | str or [Weights](../modelzoo.html#weights), optional | None    | Pretrained text recognition algorithm. It’s the path to the config file or the model name defined in metafile.                                                                   |

| `rec_weights` | str, optional                                        | None    | Path to the custom checkpoint file of the selected rec model. If it is not specified and “rec” is a model name of metafile, the weights will be loaded from metafile.            |
| `kie` \[1\]   | str or [Weights](../modelzoo.html#weights), optional | None    | Pretrained key information extraction algorithm. It’s the path to the config file or the model name defined in metafile.                                                         |
| `kie_weights` | str, optional                                        | None    | Path to the custom checkpoint file of the selected kie model. If it is not specified and “kie” is a model name of metafile, the weights will be loaded from metafile.            |

| `device`      | str, optional                                        | None    | Device used for inference, accepting all allowed strings by `torch.device`. E.g., 'cuda:0' or 'cpu'. If None, the available device will be automatically used. Defaults to None. |

\[1\]: `kie` is only effective when both text detection and recognition models are specified.

**MMOCRInferencer.\_\_call\_\_()**

| Arguments            | Type                    | Default      | Description                                                                                      |
| -------------------- | ----------------------- | ------------ | ------------------------------------------------------------------------------------------------ |
| `inputs`             | str/list/tuple/np.array | **required** | It can be a path to an image/a folder, an np array or a list/tuple (with img paths or np arrays) |

| `return_datasamples` | bool                    | False        | Whether to return results as DataSamples. If False, the results will be packed into a dict.      |
| `batch_size`         | int                     | 1            | Inference batch size.                                                                            |
| `det_batch_size`     | int, optional           | None         | Inference batch size for text detection model. Overwrite batch_size if it is not None.           |
| `rec_batch_size`     | int, optional           | None         | Inference batch size for text recognition model. Overwrite batch_size if it is not None.         |
| `kie_batch_size`     | int, optional           | None         | Inference batch size for KIE model. Overwrite batch_size if it is not None.                      |

| `return_vis`         | bool                    | False        | Whether to return the visualization result.                                                      |
| `print_result`       | bool                    | False        | Whether to print the inference result to the console.                                            |
| `show`               | bool                    | False        | Whether to display the visualization results in a popup window.                                  |
| `wait_time`          | float                   | 0            | The interval of show(s).                                                                         |
| `out_dir`            | str                     | `results/`   | Output directory of results.                                                                     |

| `save_vis`           | bool                    | False        | Whether to save the visualization results to `out_dir`.                                          |
| `save_pred`          | bool                    | False        | Whether to save the inference results to `out_dir`.                                              |

```

```{group-tab} Standard Inferencer

**Inferencer.\_\_init\_\_():**

| Arguments | Type                                                 | Default | Description                                                                                                                                                                      |

| --------- | ---------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `model`   | str or [Weights](../modelzoo.html#weights), optional | None    | Path to the config file or the model name defined in metafile.                                                                                                                   |
| `weights` | str, optional                                        | None    | Path to the custom checkpoint file of the selected det model. If it is not specified and "det" is a model name of metafile, the weights will be loaded from metafile.            |

| `device`  | str, optional                                        | None    | Device used for inference, accepting all allowed strings by `torch.device`. E.g., 'cuda:0' or 'cpu'. If None, the available device will be automatically used. Defaults to None. |

**Inferencer.\_\_call\_\_()**

| Arguments            | Type                    | Default      | Description                                                                                                      |
| -------------------- | ----------------------- | ------------ | ---------------------------------------------------------------------------------------------------------------- |
| `inputs`             | str/list/tuple/np.array | **required** | It can be a path to an image/a folder, an np array or a list/tuple (with img paths or np arrays)                 |

| `return_datasamples` | bool                    | False        | Whether to return results as DataSamples. If False, the results will be packed into a dict.                      |
| `batch_size`         | int                     | 1            | Inference batch size.                                                                                            |
| `progress_bar`       | bool                    | True         | Whether to show a progress bar.                                                                                  |
| `return_vis`         | bool                    | False        | Whether to return the visualization result.                                                                      |

| `print_result`       | bool                    | False        | Whether to print the inference result to the console.                                                            |
| `show`               | bool                    | False        | Whether to display the visualization results in a popup window.                                                  |
| `wait_time`          | float                   | 0            | The interval of show(s).                                                                                         |
| `draw_pred`          | bool                    | True         | Whether to draw predicted bounding boxes. *Only applicable on `TextDetInferencer` and `TextSpottingInferencer`.* |

| `out_dir`            | str                     | `results/`   | Output directory of results.                                                                                     |
| `save_vis`           | bool                    | False        | Whether to save the visualization results to `out_dir`.                                                          |
| `save_pred`          | bool                    | False        | Whether to save the inference results to `out_dir`.                                                              |

```
````

## Command Line Interface

```{note}
This section is only applicable to `MMOCRInferencer`.
```

You can use `tools/infer.py` to perform inference through `MMOCRInferencer`.
Its general usage is as follows:

```bash

```{note}
```

```bash
python tools/infer.py INPUT_PATH [--det DET] [--det-weights ...] ...
```

where `INPUT_PATH` is a required field, which should be a path to an image or a folder. Command-line parameters follow the mapping relationship with the Python interface parameters as follows:

- To convert the Python interface parameters to the command line ones, you need to add two `--` in front of the Python interface parameters, and replace the underscore `_` with the hyphen `-`. For example, `out_dir` becomes `--out-dir`.
- For boolean type parameters, putting the parameter in the command is equivalent to specifying it as True. For example, `--show` will specify the `show` parameter as True.

In addition, the command line will not display the inference result by default. You can use the `--print-result` parameter to view the inference result.

```bash
python tools/infer.py demo/demo_text_ocr.jpg --det DBNet --rec SAR --show --print-result
```

Running this command will give the following result:

```bash
{'predictions': [{'rec_texts': ['CBank', 'Docbcba', 'GROUP', 'MAUN', 'CROBINSONS', 'AOCOC', '916M3', 'BOO9', 'Oven', 'BRANDS', 'ARETAIL', '14', '70<UKN>S', 'ROUND', 'SALE', 'YEAR', 'ALLY', 'SALE', 'SALE'],
'rec_scores': [0.9753464579582214, ...], 'det_polygons': [[551.9930285844646, 411.9138765335083, 553.6153911653112,
383.53195309638977, 620.2410061195247, 387.33785033226013, 618.6186435386782, 415.71977376937866], ...], 'det_scores': [0.8230461478233337, ...]}]}
```

# Global&Local (ToG'2017)

> [Globally and Locally Consistent Image Completion](http://iizuka.cs.tsukuba.ac.jp/projects/completion/data/completion_sig2017.pdf)

We present a novel approach for image completion that results in images that are both locally and globally consistent. With a fully-convolutional neural network, we can complete images of arbitrary resolutions by flling in missing regions of any shape. To train this image completion network to be consistent, we use global and local context discriminators that are trained to distinguish real images from completed ones. The global discriminator looks at the entire image to assess if it is coherent as a whole, while the local discriminator looks only at a small area centered at the completed region to ensure the local consistency of the generated patches. The image completion network is then trained to fool the both context discriminator networks, which requires it to generate images that are indistinguishable from real ones with regard to overall consistency as well as in details. We show that our approach can be used to complete a wide variety of scenes. Furthermore, in contrast with the patch-based approaches such as PatchMatch, our approach can generate fragments that do not appear elsewhere in the image, which allows us to naturally complete the image.

 <img src="https://user-images.githubusercontent.com/12726765/144175196-51dfda11-f7e1-4c7e-abed-42799f757bef.png" width="400"/>
</div >

*Note that we do not apply the post-processing module in Global&Local for a fair comparison with current deep inpainting methods.*

|                    Model                     |       Dataset       |  Mask Type  | Resolution | Train Iters |   Test Set    | l1 error |  PSNR  | SSIM  | Training Resources |                        Download                         |
| :------------------------------------------: | :-----------------: | :---------: | :--------: | :---------: | :-----------: | :------: | :----: | :---: | :----------------: | :-----------------------------------------------------: |

| [Global&Local](./gl_8xb12_places-256x256.py) | Places365-Challenge | square bbox |  256x256   |    500k     | Places365-val |  11.164  | 23.152 | 0.862 |         8          | [model](https://download.openmmlab.com/mmediting/inpainting/global_local/gl_256x256_8x12_places_20200619-52a040a8.pth) \| [log](https://download.openmmlab.com/mmediting/inpainting/global_local/gl_256x256_8x12_places_20200619-52a040a8.log.json) |
| [Global&Local](./gl_8xb12_celeba-256x256.py) |      CelebA-HQ      | square bbox |  256x256   |    500k     |  CelebA-val   |  6.678   | 26.780 | 0.904 |         8          | [model](https://download.openmmlab.com/mmediting/inpainting/global_local/gl_256x256_8x12_celeba_20200619-5af0493f.pth) \| [log](https://download.openmmlab.com/mmediting/inpainting/global_local/gl_256x256_8x12_celeba_20200619-5af0493f.log.json) |

**Train**

<details>

**Train**

<details>

```shell
# cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/global_local/gl_8xb12_places-256x256.py

python tools/train.py configs/global_local/gl_8xb12_places-256x256.py

./tools/dist_train.sh configs/global_local/gl_8xb12_places-256x256.py 8
```

**Test**

<details>

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/global_local/gl_8xb12_places-256x256.py https://download.openmmlab.com/mmediting/inpainting/global_local/gl_256x256_8x12_places_20200619-52a040a8.pth

python tools/test.py configs/global_local/gl_8xb12_places-256x256.py https://download.openmmlab.com/mmediting/inpainting/global_local/gl_256x256_8x12_places_20200619-52a040a8.pth

./tools/dist_test.sh configs/global_local/gl_8xb12_places-256x256.py https://download.openmmlab.com/mmediting/inpainting/global_local/gl_256x256_8x12_places_20200619-52a040a8.pth 8
```

```bibtex
@article{iizuka2017globally,
  title={Globally and locally consistent image completion},
  author={Iizuka, Satoshi and Simo-Serra, Edgar and Ishikawa, Hiroshi},
  journal={ACM Transactions on Graphics (ToG)},
  volume={36},
  pages={1--14},
}
```

# 如何进行回归测试

<!-- -->

这篇教程介绍了如何进行回归测试。部署配置文件由`每个codebase的回归配置文件`，`推理框架配置信息`组成。

- [如何进行回归测试](#如何进行回归测试)
  - [1. 环境搭建](#1-环境搭建)
    - [MMDeploy的安装及配置](#mmdeploy的安装及配置)
    - [Python环境依赖](#python环境依赖)
  - [2. 用法](#2-用法)
    - [参数解析](#参数解析)
    - [注意事项](#注意事项)
  - [例子](#例子)
  - [3. 回归测试配置文件](#3-回归测试配置文件)
    - [示例及参数解析](#示例及参数解析)
  - [4. 生成的报告](#4-生成的报告)
    - [模板](#模板)
  - [5. 支持的后端](#5-支持的后端)
  - [6. 支持的Codebase及其Metric](#6-支持的codebase及其metric)
  - [7. 注意事项](#7-注意事项)
  - [8. 常见问题](#8-常见问题)

## 1. 环境搭建

### MMDeploy的安装及配置

本章节的内容，需要提前根据[build 文档](../01-how-to-build/build_from_source.md)将 MMDeploy 安装配置好之后，才能进行。

### Python环境依赖

需要安装 test 的环境

```shell
pip install -r requirements/tests.txt
```

如果在使用过程是 numpy 报错，则更新一下 numpy

```shell
pip install -U numpy
```

## 2. 用法

```shell
python ./tools/regression_test.py \
    --codebase "${CODEBASE_NAME}" \
    --backends "${BACKEND}" \
    [--models "${MODELS}"] \
    --work-dir "${WORK_DIR}" \

```

## 2. 用法

```shell
    --device "${DEVICE}" \
    [--performance 或 -p] \
    [--checkpoint-dir "$CHECKPOINT_DIR"]
```

### 参数解析

- `--codebase` : 需要测试的 codebase，eg.`mmdet`, 测试多个 `mmpretrain mmdet ...`
- `--backends` : 筛选测试的后端, 默认测全部`backend`, 也可传入若干个后端，例如 `onnxruntime tesnsorrt`。如果需要一同进行 SDK 的测试，需要在 `tests/regression/${codebase}.yml` 里面的 `sdk_config` 进行配置。
- `--models` : 指定测试的模型, 默认测试 `yml` 中所有模型, 也可传入若干个模型名称，模型名称可参考相关yml配置文件。例如 `ResNet SE-ResNet "Mask R-CNN"`。注意的是，可传入只有字母和数字组成模型名称，例如 `resnet seresnet maskrcnn`。
- `--work-dir` : 模型转换、报告生成的路径，默认是`../mmdeploy_regression_working_dir`，注意路径中不要含空格等特殊字符。
- `--checkpoint-dir`: PyTorch 模型文件下载保存路径，默认是`../mmdeploy_checkpoints`，注意路径中不要含空格等特殊字符。
- `--device` : 使用的设备，默认 `cuda`。

- `--log-level` : 设置日记的等级，选项包括`'CRITICAL'， 'FATAL'， 'ERROR'， 'WARN'， 'WARNING'， 'INFO'， 'DEBUG'， 'NOTSET'`。默认是`INFO`。
- `-p` 或 `--performance` : 是否测试精度，加上则测试转换+精度，不加上则只测试转换

### 注意事项

对于 Windows 用户：

1. 要在 shell 命令中使用 `&&` 连接符，需要下载并使用 `PowerShell 7 Preview 5+`。
2. 如果您使用 conda env，可能需要在 regression_test.py 中将 `python3` 更改为 `python`，因为 `%USERPROFILE%\AppData\Local\Microsoft\WindowsApps` 目录中有 `python3.exe`。

## 例子

1. 测试 mmdet 和 mmpose 的所有 backend 的 **转换+精度**

```shell
    --codebase mmdet mmpose \
    --work-dir "../mmdeploy_regression_working_dir" \
    --device "cuda" \
    --performance
```

2. 测试 mmdet 和 mmpose 的某几个 backend 的 **转换+精度**

```shell
    --backends onnxruntime tensorrt \

```shell
    -p
```

3. 测试 mmdet 和 mmpose 的某几个 backend，**只测试转换**

```shell
```

4. 测试 mmdet 和 mmpretrain 的某几个 models，**只测试转换**

```shell
    --models ResNet SE-ResNet "Mask R-CNN" \
```

## 3. 回归测试配置文件

### 示例及参数解析

```yaml
globals:
  codebase_dir: ../mmocr # 回归测试的 codebase 路径
  checkpoint_force_download: False # 回归测试是否重新下载模型即使其已经存在
  images: # 测试使用图片
    img_densetext_det: &img_densetext_det ../mmocr/demo/demo_densetext_det.jpg

codebase_dir: ../mmocr # 回归测试的 codebase 路径
    img_demo_text_det: &img_demo_text_det ../mmocr/demo/demo_text_det.jpg
    img_demo_text_ocr: &img_demo_text_ocr ../mmocr/demo/demo_text_ocr.jpg
    img_demo_text_recog: &img_demo_text_recog ../mmocr/demo/demo_text_recog.jpg
  metric_info: &metric_info # 指标参数
    hmean-iou: # 命名根据 metafile.Results.Metrics
      eval_name: hmean-iou # 命名根据 test.py --metrics args 入参名称
      metric_key: 0_hmean-iou:hmean # 命名根据 eval 写入 log 的 key name
      tolerance: 0.1 # 容忍的阈值区间
      task_name: Text Detection # 命名根据模型 metafile.Results.Task
      dataset: ICDAR2015 #命名根据模型 metafile.Results.Dataset
    word_acc: # 同上
      eval_name: acc
      metric_key: 0_word_acc_ignore_case
      tolerance: 0.2
      task_name: Text Recognition
      dataset: IIIT5K

word_acc: # 同上
  convert_image_det: &convert_image_det # det转换会使用到的图片
    input_img: *img_densetext_det
    test_img: *img_demo_text_det
  convert_image_rec: &convert_image_rec
    input_img: *img_demo_text_recog
    test_img: *img_demo_text_recog
  backend_test: &default_backend_test True # 是否对 backend 进行精度测试
  sdk: # SDK 配置文件
    sdk_detection_dynamic: &sdk_detection_dynamic configs/mmocr/text-detection/text-detection_sdk_dynamic.py
    sdk_recognition_dynamic: &sdk_recognition_dynamic configs/mmocr/text-recognition/text-recognition_sdk_dynamic.py

onnxruntime:
  pipeline_ort_recognition_static_fp32: &pipeline_ort_recognition_static_fp32
    convert_image: *convert_image_rec # 转换过程中使用的图片
    backend_test: *default_backend_test # 是否进行后端测试，存在则判断，不存在则视为 False

pipeline_ort_recognition_static_fp32: &pipeline_ort_recognition_static_fp32
    sdk_config: *sdk_recognition_dynamic # 是否进行SDK测试，存在则使用特定的 SDK config 进行测试，不存在则视为不进行 SDK 测试
    deploy_config: configs/mmocr/text-recognition/text-recognition_onnxruntime_static.py # 使用的 deploy cfg 路径，基于 mmdeploy 的路径

  pipeline_ort_recognition_dynamic_fp32: &pipeline_ort_recognition_dynamic_fp32
    convert_image: *convert_image_rec
    backend_test: *default_backend_test
    sdk_config: *sdk_recognition_dynamic
    deploy_config: configs/mmocr/text-recognition/text-recognition_onnxruntime_dynamic.py

  pipeline_ort_detection_dynamic_fp32: &pipeline_ort_detection_dynamic_fp32
    convert_image: *convert_image_det
    deploy_config: configs/mmocr/text-detection/text-detection_onnxruntime_dynamic.py

tensorrt:
  pipeline_trt_recognition_dynamic_fp16: &pipeline_trt_recognition_dynamic_fp16

deploy_config: configs/mmocr/text-detection/text-detection_onnxruntime_dynamic.py

tensorrt:
    deploy_config: configs/mmocr/text-recognition/text-recognition_tensorrt-fp16_dynamic-1x32x32-1x32x640.py

  pipeline_trt_detection_dynamic_fp16: &pipeline_trt_detection_dynamic_fp16
    sdk_config: *sdk_detection_dynamic
    deploy_config: configs/mmocr/text-detection/text-detection_tensorrt-fp16_dynamic-320x320-2240x2240.py

openvino:
  # 此处省略，内容同上
ncnn:
pplnn:
torchscript:

models:
  - name: crnn # 模型名称
    metafile: configs/textrecog/crnn/metafile.yml # 模型对应的 metafile 的路径，相对于 codebase 的路径

ncnn:
pplnn:

models:
    codebase_model_config_dir: configs/textrecog/crnn # `model_configs` 的父文件夹路径，相对于 codebase 的路径
    model_configs: # 需要测试的 config 名称
      - crnn_academic_dataset.py
    pipelines: # 使用的 pipeline
      - *pipeline_ort_recognition_dynamic_fp32

  - name: dbnet
    metafile: configs/textdet/dbnet/metafile.yml
    codebase_model_config_dir: configs/textdet/dbnet
    model_configs:
      - dbnet_r18_fpnc_1200e_icdar2015.py
    pipelines:
      - *pipeline_ort_detection_dynamic_fp32
      - *pipeline_trt_detection_dynamic_fp16

      # 特殊的 pipeline 可以这样加入
      - convert_image: xxx
        backend_test: xxx
        sdk_config: xxx
        deploy_config: configs/mmocr/text-detection/xxx
```

## 4. 生成的报告

### 模板

# 特殊的 pipeline 可以这样加入
```

## 4. 生成的报告

### 模板

|      | Model    | Model Config      | Task             | Checkpoint     | Dataset    | Backend  | Deploy Config   | Static or Dynamic | Precision Type | Conversion Result | metric_1    | metric_2    | metric_n    | Test Pass    |
| ---- | -------- | ----------------- | ---------------- | -------------- | ---------- | -------- | --------------- | ----------------- | -------------- | ----------------- | ----------- | ----------- | ----------- | ------------ |
| 序号 | 模型名称 | model config 路径 | 执行的 task name | `.pth`模型路径 | 数据集名称 | 后端名称 | deploy cfg 路径 | 动态 or 静态      | 测试精度       | 模型转换结果      | 指标 1 数值 | 指标 2 数值 | 指标 n 数值 | 后端测试结果 |

### 示例

这是 MMOCR 生成的报告

### 示例

|     | Model | Model Config                                                     | Task             | Checkpoint                                                                                                   | Dataset   | Backend         | Deploy Config                                                                          | Static or Dynamic | Precision Type | Conversion Result | hmean-iou | word_acc | Test Pass |

| --- | ----- | ---------------------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------ | --------- | --------------- | -------------------------------------------------------------------------------------- | ----------------- | -------------- | ----------------- | --------- | -------- | --------- |
| 0   | crnn  | ../mmocr/configs/textrecog/crnn/crnn_academic_dataset.py         | Text Recognition | ../mmdeploy_checkpoints/mmocr/crnn/crnn_academic-a723a1c5.pth                                                | IIIT5K    | Pytorch         | -                                                                                      | -                 | -              | -                 | -         | 80.5     | -         |

| 1   | crnn  | ../mmocr/configs/textrecog/crnn/crnn_academic_dataset.py         | Text Recognition | ${WORK_DIR}/mmocr/crnn/onnxruntime/static/crnn_academic-a723a1c5/end2end.onnx                                | x         | onnxruntime     | configs/mmocr/text-recognition/text-recognition_onnxruntime_dynamic.py                 | static            | fp32           | True              | -         | 80.67    | True      |
| 2   | crnn  | ../mmocr/configs/textrecog/crnn/crnn_academic_dataset.py         | Text Recognition | ${WORK_DIR}/mmocr/crnn/onnxruntime/static/crnn_academic-a723a1c5                                             | x         | SDK-onnxruntime | configs/mmocr/text-recognition/text-recognition_sdk_dynamic.py                         | static            | fp32           | True              | -         | x        | False     |

| 3   | dbnet | ../mmocr/configs/textdet/dbnet/dbnet_r18_fpnc_1200e_icdar2015.py | Text Detection   | ../mmdeploy_checkpoints/mmocr/dbnet/dbnet_r18_fpnc_sbn_1200e_icdar2015_20210329-ba3ab597.pth                 | ICDAR2015 | Pytorch         | -                                                                                      | -                 | -              | -                 | 0.795     | -        | -         |
| 4   | dbnet | ../mmocr/configs/textdet/dbnet/dbnet_r18_fpnc_1200e_icdar2015.py | Text Detection   | ../mmdeploy_checkpoints/mmocr/dbnet/dbnet_r18_fpnc_sbn_1200e_icdar2015_20210329-ba3ab597.pth                 | ICDAR     | onnxruntime     | configs/mmocr/text-detection/text-detection_onnxruntime_dynamic.py                     | dynamic           | fp32           | True              | -         | -        | True      |

| 5   | dbnet | ../mmocr/configs/textdet/dbnet/dbnet_r18_fpnc_1200e_icdar2015.py | Text Detection   | ${WORK_DIR}/mmocr/dbnet/tensorrt/dynamic/dbnet_r18_fpnc_sbn_1200e_icdar2015_20210329-ba3ab597/end2end.engine | ICDAR     | tensorrt        | configs/mmocr/text-detection/text-detection_tensorrt-fp16_dynamic-320x320-2240x2240.py | dynamic           | fp16           | True              | 0.793302  | -        | True      |
| 6   | dbnet | ../mmocr/configs/textdet/dbnet/dbnet_r18_fpnc_1200e_icdar2015.py | Text Detection   | ${WORK_DIR}/mmocr/dbnet/tensorrt/dynamic/dbnet_r18_fpnc_sbn_1200e_icdar2015_20210329-ba3ab597                | ICDAR     | SDK-tensorrt    | configs/mmocr/text-detection/text-detection_sdk_dynamic.py                             | dynamic           | fp16           | True              | 0.795073  | -        | True      |

## 5. 支持的后端

- [x] ONNX Runtime
- [x] TensorRT
- [x] PPLNN
- [x] ncnn
- [x] OpenVINO
- [x] TorchScript
- [x] SNPE
- [x] MMDeploy SDK

## 5. 支持的后端

- [x] PPLNN
- [x] ncnn
- [x] SNPE

## 6. 支持的Codebase及其Metric

| Codebase   | Metric   | Support            |
| ---------- | -------- | ------------------ |
| mmdet      | bbox     | :heavy_check_mark: |
|            | segm     | :heavy_check_mark: |
|            | PQ       | :x:                |
| mmpretrain | accuracy | :heavy_check_mark: |
| mmseg      | mIoU     | :heavy_check_mark: |
| mmpose     | AR       | :heavy_check_mark: |
|            | AP       | :heavy_check_mark: |
| mmocr      | hmean    | :heavy_check_mark: |
|            | acc      | :heavy_check_mark: |
| mmagic     | PSNR     | :heavy_check_mark: |
|            | SSIM     | :heavy_check_mark: |

## 7. 注意事项

暂无

## 8. 常见问题

暂无

# Useful Hooks

MMDetection and MMEngine provide users with various useful hooks including log hooks, `NumClassCheckHook`, etc. This tutorial introduces the functionalities and usages of hooks implemented in MMDetection. For using hooks in MMEngine, please read the [API documentation in MMEngine](https://github.com/open-mmlab/mmengine/tree/main/docs/en/tutorials/hook.md).

## CheckInvalidLossHook

## NumClassCheckHook

## MemoryProfilerHook

[Memory profiler hook](https://github.com/open-mmlab/mmdetection/blob/main/mmdet/engine/hooks/memory_profiler_hook.py) records memory information including virtual memory, swap memory, and the memory of the current process. This hook helps grasp the memory usage of the system and discover potential memory leak bugs. To use this hook, users should install `memory_profiler` and `psutil` by `pip install memory_profiler psutil` first.

### Usage

To use this hook, users should add the following code to the config file.

```python

### Usage

```python
    dict(type='MemoryProfilerHook', interval=50)
]
```

### Result

During training, you can see the messages in the log recorded by `MemoryProfilerHook` as below.

```text
The system has 250 GB (246360 MB + 9407 MB) of memory and 8 GB (5740 MB + 2452 MB) of swap memory in total. Currently 9407 MB (4.4%) of memory and 5740 MB (29.9%) of swap memory were consumed. And the current training process consumed 5434 MB of memory.
```

```text
2022-04-21 08:49:56,881 - mmengine - INFO - Memory information available_memory: 246360 MB, used_memory: 9407 MB, memory_utilization: 4.4 %, available_swap_memory: 5740 MB, used_swap_memory: 2452 MB, swap_memory_utilization: 29.9 %, current_process_memory: 5434 MB
```

## SetEpochInfoHook

## SyncNormHook

## SyncRandomSizeHook

## YOLOXLrUpdaterHook

## YOLOXModeSwitchHook

## How to implement a custom hook

```

In general, there are 20 points where hooks can be inserted from the beginning to the end of model training. The users can implement custom hooks and insert them at different points in the process of training to do what they want.

- global points: `before_run`, `after_run`
- points in training: `before_train`, `before_train_epoch`, `before_train_iter`, `after_train_iter`, `after_train_epoch`, `after_train`
- points in validation: `before_val`, `before_val_epoch`, `before_val_iter`, `after_val_iter`, `after_val_epoch`, `after_val`
- points at testing: `before_test`, `before_test_epoch`, `before_test_iter`, `after_test_iter`, `after_test_epoch`,  `after_test`
- other points: `before_save_checkpoint`, `after_save_checkpoint`

For example, users can implement a hook to check loss and terminate training when loss goes NaN. To achieve that, there are three steps to go:

1. Implement a new hook that inherits the `Hook` class in MMEngine, and implement `after_train_iter` method which checks whether loss goes NaN after every `n` training iterations.
2. The implemented hook should be registered in `HOOKS` by `@HOOKS.register_module()` as shown in the code below.
3. Add `custom_hooks = [dict(type='MemoryProfilerHook', interval=50)]` in the config file.

```python
from typing import Optional

from mmdet.registry import HOOKS

class CheckInvalidLossHook(Hook):
    """Check invalid loss hook.

    This hook will regularly check whether the loss is valid
    during training.

    Args:
        interval (int): Checking interval (every k iterations).
            Default: 50.
    """

    def __init__(self, interval: int = 50) -> None:
        self.interval = interval

                         runner: Runner,
                         data_batch: Optional[dict] = None,
        """Regularly check whether the loss is valid every n iterations.

        Args:
            runner (:obj:`Runner`): The runner of the training process.
            batch_idx (int): The index of the current batch in the train loop.

Args:
            data_batch (dict, Optional): Data from dataloader.
            outputs (dict, Optional): Outputs from model. Defaults to None.
        """
        if self.every_n_train_iters(runner, self.interval):
            assert torch.isfinite(outputs['loss']), \
                runner.logger.info('loss become infinite or NaN!')
```

Please read [customize_runtime](../advanced_guides/customize_runtime.md) for more about implementing a custom hook.

# PCam (PatchCamelyon)

This project supports **`Patch Camelyon (PCam) `**, which can be downloaded from [here](https://opendatalab.com/PCam).

PatchCamelyon is an image classification dataset. It consists of 327680 color images (96 x 96px) extracted from histopathologic scans of lymph node sections. Each image is annotated with a binary label indicating presence of metastatic tissue. PCam provides a new benchmark for machine learning models: bigger than CIFAR10, smaller than ImageNet, trainable on a single GPU.

| Dataset Name                         | Anatomical Region | Task Type    | Modality       | Num. Classes | Train/Val/Test images | Train/Val/Test Labeled | Release Date | License                                                       |

| ------------------------------------ | ----------------- | ------------ | -------------- | ------------ | --------------------- | ---------------------- | ------------ | ------------------------------------------------------------- |
| [Pcam](https://opendatalab.com/PCam) | throax            | segmentation | histopathology | 2            | 327680/-/-            | yes/-/-                | 2018         | [CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/) |

|    Class Name     | Num. Train | Pct. Train | Num. Val | Pct. Val | Num. Test | Pct. Test |
| :---------------: | :--------: | :--------: | :------: | :------: | :-------: | :-------: |
|    background     |   214849   |   63.77    |    -     |    -     |     -     |     -     |
| metastatic tissue |   131832   |   36.22    |    -     |    -     |     -     |     -     |

Note:

Note:

![pcam](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/histopathology/pcam/pcam_dataset.png?raw=true)

```
@inproceedings{veeling2018rotation,
	title={Rotation equivariant CNNs for digital pathology},
	author={Veeling, Bastiaan S and Linmans, Jasper and Winkens, Jim and Cohen, Taco and Welling, Max},
	booktitle={International Conference on Medical image computing and computer-assisted intervention},
	pages={210--218},
	year={2018},
}
```

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `pcam/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
```

- download dataset from [here](https://opendatalab.com/PCam) and decompress data to path `'data/'`.

```shell
mkdir data & cd data
pip install opendatalab
odl get PCam
mv ./PCam/raw/pcamv1 ./
rm -rf PCam
cd ..
```

```none
  ├── mmseg
  │   │   │   │   ├── pcam

```

|    background     |   171948   |   63.82    |  42901   |   63.6   |     -     |     -     |

| metastatic tissue |   105371   |   36.18    |  26461   |   36.4   |     -     |     -     |

```shell
```

```shell
```

# R3Det

> [R3Det: Refined Single-Stage Detector with Feature Refinement for Rotating Object](https://arxiv.org/pdf/1908.05612.pdf)

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/r3det.png" width="800"/>
</div>

Rotation detection is a challenging task due to the difficulties of locating the multi-angle objects and separating them effectively from the background. Though considerable progress has been made, for practical settings, there still exist challenges for rotating objects with large aspect ratio, dense distribution and category extremely imbalance. In this paper, we propose an end-to-end refined single-stage rotation detector for fast and accurate object detection by using a progressive regression approach from coarse to fine granularity. Considering the shortcoming of feature misalignment in existing refined single stage detector, we design a feature refinement module to improve detection performance by getting more accurate features. The key idea of feature refinement module is to re-encode the position information of the current refined bounding box to the corresponding feature points through pixel-wise feature interpolation to realize feature reconstruction and alignment. For more accurate rotation estimation, an approximate SkewIoU loss is proposed to solve the problem that the calculation of SkewIoU is not derivable. Experiments on three popular remote sensing public datasets DOTA, HRSC2016, UCAS-AOD as well as one scene text dataset ICDAR2015 show the effectiveness of our approach.

DOTA1.0

|         Backbone         |  mAP  | Angle | lr schd | Mem (GB) | Inf Time (fps) | Aug | Batch Size |                                                    Configs                                                     |                                                                                                                                                                            Download                                                                                                                                                                            |

| ResNet50 (1024,1024,200) | 64.55 |  oc   |   1x    |   3.38   |      15.7      |  -  |     2      | [rotated-retinanet-hbox-oc_r50_fpn_1x_dota](../rotated_retinanet/rotated-retinanet-hbox-oc_r50_fpn_1x_dota.py) | [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_hbb_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_r50_fpn_1x_dota_oc-e8a7c7df.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_hbb_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_r50_fpn_1x_dota_oc_20220121_095315.log.json) |

| ResNet50 (1024,1024,200) | 69.80 |  oc   |   1x    |   3.54   |      12.4      |  -  |     2      |                        [r3det-oc_r50_fpn_1x_dota](../r3det/r3det-oc_r50_fpn_1x_dota.py)                        |                                             [model](https://download.openmmlab.com/mmrotate/v0.1.0/r3det/r3det_r50_fpn_1x_dota_oc/r3det_r50_fpn_1x_dota_oc-b1fb045c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/r3det/r3det_r50_fpn_1x_dota_oc/r3det_r50_fpn_1x_dota_oc_20220126_191226.log.json)                                             |

| ResNet50 (1024,1024,200) | 70.18 |  oc   |   1x    |   3.23   |      15.6      |  -  |     2      |                   [r3det-tiny-oc_r50_fpn_1x_dota](../r3det/r3det-tiny-oc_r50_fpn_1x_dota.py)                   |                                   [model](https://download.openmmlab.com/mmrotate/v0.1.0/r3det/r3det_tiny_r50_fpn_1x_dota_oc/r3det_tiny_r50_fpn_1x_dota_oc-c98a616c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/r3det/r3det_tiny_r50_fpn_1x_dota_oc/r3det_tiny_r50_fpn_1x_dota_oc_20220209_171624.log.json)                                   |

```
@inproceedings{yang2021r3det,
    title={R3Det: Refined Single-Stage Detector with Feature Refinement for Rotating Object},
    author={Yang, Xue and Yan, Junchi and Feng, Ziming and He, Tao},
    booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
    volume={35},
    number={4},
    pages={3163--3171},
}

```

# Changelog of v3.x

## v3.1.0 (30/6/2023)

- Supports tracking algorithms including multi-object tracking (MOT) algorithms SORT, DeepSORT, StrongSORT, OCSORT, ByteTrack, QDTrack, and video instance segmentation (VIS) algorithm MaskTrackRCNN, Mask2Former-VIS.
- Support [ViTDet](../../../projects/ViTDet)
- Supports inference and evaluation of multimodal algorithms [GLIP](../../../configs/glip) and [XDecoder](../../../projects/XDecoder), and also supports datasets such as COCO semantic segmentation, COCO Caption, ADE20k general segmentation, and RefCOCO. GLIP fine-tuning will be supported in the future.
- Provides a [gradio demo](https://github.com/open-mmlab/mmdetection/blob/dev-3.x/projects/gradio_demo/README.md) for image type tasks of MMDetection, making it easy for users to experience.

- Support DSDL Dataset (#9801)
- Support iSAID dataset (#10028)
- Support VISION dataset (#10530)
- Release SoftTeacher checkpoints (#10119)

- Release `centernet-update_r50-caffe_fpn_ms-1x_coco` checkpoints  (#10327)
- Support SIoULoss (#10290)
- Support Eqlv2 loss (#10120)
- Support CopyPaste when mask is not available (#10509)
- Support MIM to download ODL dataset (#10460)
- Support new config (#10566)

- Fix benchmark scripts error in windows (#10128)
- Fix error of `YOLOXModeSwitchHook` does not switch the mode when resumed from the checkpoint after switched (#10116)
- Fix pred and weight dims unmatch in SmoothL1Loss (#10423)

- Update MMDet_Tutorial.ipynb (#10081)
- Support to hide inference progress (#10519)
- Replace mmcls with mmpretrain  (#10545)

A total of 29 developers contributed to this release.

Thanks @lovelykite, @minato-ellie, @freepoet, @wufan-tb, @yalibian, @keyakiluo, @gihanjayatilaka, @i-aki-y, @xin-li-67, @RangeKing, @JingweiZhang12, @MambaWong, @lucianovk, @tall-josh, @xiuqhou, @jamiechoi1995, @YQisme, @yechenzhi, @bjzhb666, @xiexinch, @jamiechoi1995, @yarkable, @Renzhihan, @nijkah, @amaizr, @Lum1104, @zwhus, @Czm369, @hhaAndroid

## v3.0.0 (6/4/2023)

- Support Semi-automatic annotation Base [Label-Studio](../../../projects/LabelStudio) (#10039)
- Support [EfficientDet](../../../projects/EfficientDet) in projects (#9810)

- File I/O migration and reconstruction (#9709)
- Release DINO Swin-L 36e model (#9927)

- Fix benchmark script (#9865)
- Fix the crop method of PolygonMasks (#9858)

- Fix Albu augmentation with the mask shape (#9918)
- Fix `RTMDetIns` prior generator device error (#9964)
- Fix `img_shape` in data pipeline (#9966)
- Fix cityscapes import error (#9984)
- Fix `solov2_r50_fpn_ms-3x_coco.py` config error (#10030)
- Fix Conditional DETR AP and Log (#9889)
- Fix accepting an unexpected argument local-rank in PyTorch 2.0 (#10050)
- Fix `common/ms_3x_coco-instance.py` config error (#10056)
- Fix compute flops error (#10051)
- Delete `data_root` in `CocoOccludedSeparatedMetric` to fix bug (#9969)
- Unifying metafile.yml (#9849)

- Added BoxInst r101 config (#9967)
- Added config migration guide (#9960)
- Added more social networking links (#10021)
- Added RTMDet config introduce (#10042)
- Added visualization docs (#9938, #10058)

- Refined data_prepare docs (#9935)
- Added support for setting the cache_size_limit parameter of dynamo in PyTorch 2.0 (#10054)
- Updated coco_metric.py (#10033)
- Update type hint (#10040)

A total of 19 developers contributed to this release.

Thanks @IRONICBo, @vansin, @RangeKing, @Ghlerrix, @okotaku, @JosonChan1998, @zgzhengSE, @bobo0810, @yechenzh, @Zheng-LinXiao, @LYMDLUT, @yarkable, @xiejiajiannb, @chhluo, @BIGWangYuDong, @RangiLy, @zwhus, @hhaAndroid, @ZwwWayne

## v3.0.0rc6 (24/2/2023)

- Support [Boxinst](../../../configs/boxinst), [Objects365 Dataset](../../../configs/objects365), and [Separated and Occluded COCO metric](../user_guides/useful_tools.md#COCO-Separated-&-Occluded-Mask-Metric)

- Support [ConvNeXt-V2](../../../projects/ConvNeXt-V2), [DiffusionDet](../../../projects/DiffusionDet), and inference of [EfficientDet](../../../projects/EfficientDet) and [Detic](../../../projects/Detic) in `Projects`
- Refactor [DETR](../../../configs/detr) series and support [Conditional-DETR](../../../configs/conditional_detr), [DAB-DETR](../../../configs/dab_detr), and [DINO](../../../configs/detr)
- Support `DetInferencer` for inference, Test Time Augmentation, and automatically importing modules from registry
- Support RTMDet-Ins ONNXRuntime and TensorRT [deployment](../../../configs/rtmdet/README.md#deployment-tutorial)
- Support [calculating FLOPs of detectors](../user_guides/useful_tools.md#Model-Complexity)

- Support [Boxinst](https://arxiv.org/abs/2012.02310) (#9525)
- Support [Objects365 Dataset](https://openaccess.thecvf.com/content_ICCV_2019/papers/Shao_Objects365_A_Large-Scale_High-Quality_Dataset_for_Object_Detection_ICCV_2019_paper.pdf) (#9600)

- Support [ConvNeXt-V2](http://arxiv.org/abs/2301.00808) in `Projects` (#9619)
- Support [DiffusionDet](https://arxiv.org/abs/2211.09788) in `Projects` (#9639, #9768)
- Support [Detic](http://arxiv.org/abs/2201.02605) inference in `Projects` (#9645)
- Support [EfficientDet](https://arxiv.org/abs/1911.09070) inference in `Projects` (#9645)
- Support [Separated and Occluded COCO metric](https://arxiv.org/abs/2210.10046) (#9710)
- Support auto import modules from registry (#9143)
- Refactor DETR series and support Conditional-DETR, DAB-DETR and DINO (#9646)
- Support `DetInferencer` for inference (#9561)
- Support Test Time Augmentation (#9452)
- Support calculating FLOPs of detectors (#9777)

- Fix deprecating old type alias due to new version of numpy (#9625, #9537)

- Fix VOC metrics (#9784)
- Fix the wrong link of RTMDet-x log (#9549)
- Fix RTMDet link in README (#9575)
- Fix MMDet get flops error (#9589)
- Fix `use_depthwise` in RTMDet (#9624)
- Fix `albumentations` augmentation post process with masks (#9551)
- Fix DETR series Unit Test (#9647)
- Fix `LoadPanopticAnnotations` bug (#9703)
- Fix `isort` CI (#9680)
- Fix amp pooling overflow (#9670)
- Fix docstring about noise in DINO (#9747)
- Fix potential bug in `MultiImageMixDataset` (#9764)

- Replace NumPy transpose with PyTorch permute to speed-up (#9762)
- Deprecate `sklearn` (#9725)
- Add RTMDet-Ins deployment guide (#9823)
- Update RTMDet config and README (#9603)
- Replace the models used in the tutorial document with RTMDet (#9843)

- Adjust the minimum supported python version to 3.7 (#9602)
- Support modifying palette through configuration (#9445)
- Update README document in `Project` (#9599)
- Replace `github` with `gitee` in `.pre-commit-config-zh-cn.yaml` file (#9586)
- Use official `isort` in `.pre-commit-config.yaml` file (#9701)
- Change MMCV minimum version to `2.0.0rc4` for `dev-3.x` (#9695)
- Add Chinese version of single_stage_as_rpn.md and test_results_submission.md (#9434)
- Add OpenDataLab download link (#9605, #9738)
- Add type hints of several layers (#9346)
- Add typehint for `DarknetBottleneck` (#9591)
- Add dockerfile (#9659)
- Add twitter, discord, medium, and youtube link (#9775)
- Prepare for merging refactor-detr (#9656)
- Add metafile to ConditionalDETR, DABDETR and DINO (#9715)

- Support to modify `non_blocking` parameters (#9723)
- Comment repeater visualizer register (#9740)
- Update user guide: `finetune.md` and `inference.md` (#9578)

- @NoFish-528 made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/9346>
- @137208 made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/9434>
- @lyviva made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/9625>
- @zwhus made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/9589>
- @zylo117 made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/9670>
- @chg0901 made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/9740>

- @DanShouzhu made their first contribution in https://github.com/open-mmlab/mmdetection/pull/9578

A total of 27 developers contributed to this release.

Thanks @JosonChan1998, @RangeKing, @NoFish-528, @likyoo, @Xiangxu-0103, @137208, @PeterH0323, @tianleiSHI, @wufan-tb, @lyviva, @zwhus, @jshilong, @Li-Qingyun, @sanbuphy, @zylo117, @triple-Mu, @KeiChiTse, @LYMDLUT, @nijkah, @chg0901, @DanShouzhu, @zytx121, @vansin, @BIGWangYuDong, @hhaAndroid, @RangiLyu, @ZwwWayne

## v3.0.0rc5 (26/12/2022)

- Support [RTMDet](https://arxiv.org/abs/2212.07784) instance segmentation models. The technical report of RTMDet is on [arxiv](https://arxiv.org/abs/2212.07784)

- Support SSHContextModule in paper [SSH: Single Stage Headless Face Detector](https://arxiv.org/abs/1708.03979).

- Support [RTMDet](https://arxiv.org/abs/2212.07784) instance segmentation models and improve RTMDet test config (#9494)
- Support SSHContextModule in paper [SSH: Single Stage Headless Face Detector](https://arxiv.org/abs/1708.03979) (#8953)
- Release [CondInst](https://arxiv.org/abs/2003.05664) pre-trained model (#9406)

- Fix CondInst predict error when `batch_size` is greater than 1 in inference (#9400)
- Fix the bug of visualization when the dtype of the pipeline output image is not uint8 in browse dataset (#9401)
- Fix `analyze_logs.py` to plot mAP and calculate train time correctly (#9409)
- Fix backward inplace error with `PAFPN` (#9450)

- Fix config import links in model converters (#9441)
- Fix `DeformableDETRHead` object has no attribute `loss_single` (#9477)
- Fix the logic of pseudo bboxes predicted by teacher model in SemiBaseDetector (#9414)
- Fix demo API in instance segmentation tutorial (#9226)
- Fix `analyze_results` (#9380)
- Fix the error that Readthedocs API cannot be displayed (#9510)
- Fix the error when there are no prediction results and support visualize the groundtruth of TTA (#9840)

- Remove legacy `builder.py` (#9479)
- Make sure the pipeline argument shape is in `(width, height)` order (#9324)
- Add `.pre-commit-config-zh-cn.yaml` file (#9388)
- Refactor dataset metainfo to lowercase (#9469)
- Add PyTorch 1.13 checking in CI (#9478)
- Adjust `FocalLoss` and `QualityFocalLoss` to allow different kinds of targets (#9481)
- Refactor `setup.cfg` (#9370)

- Clip saturation value to valid range `[0, 1]` (#9391)
- Only keep meta and state_dict when publishing model (#9356)
- Add segm evaluator in ms-poly_3x_coco_instance config (#9524)
- Update deployment guide (#9527)
- Update zh_cn `faq.md` (#9396)
- Update `get_started` (#9480)
- Update the zh_cn user_guides of `useful_tools.md` and `useful_hooks.md` (#9453)
- Add type hints for `bfp` and `channel_mapper` (#9410)
- Add type hints of several losses (#9397)
- Add type hints and update docstring for task modules (#9468)

- @lihua199710 made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/9388>
- @twmht made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/9450>

- @tianleiSHI made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/9453>
- @kitecats made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/9481>
- @QJC123654 made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/9468>

A total of 20 developers contributed to this release.

Thanks @liuyanyi, @RangeKing, @lihua199710, @MambaWong, @sanbuphy, @Xiangxu-0103, @twmht, @JunyaoHu, @Chan-Sun, @tianleiSHI, @zytx121, @kitecats, @QJC123654, @JosonChan1998, @lvhan028, @Czm369, @BIGWangYuDong, @RangiLyu, @hhaAndroid, @ZwwWayne

## v3.0.0rc4 (23/11/2022)

- Support [CondInst](https://arxiv.org/abs/2003.05664)

- Add `projects/` folder, which will be a place for some experimental models/features.
- Support [SparseInst](https://arxiv.org/abs/2203.12827) in [`projects`](./projects/SparseInst/README.md)

- Support [CondInst](https://arxiv.org/abs/2003.05664) (#9223)
- Add `projects/` folder, which will be a place for some experimental models/features (#9341)
- Support [SparseInst](https://arxiv.org/abs/2203.12827) in [`projects`](./projects/SparseInst/README.md) (#9377)

- Fix `pixel_decoder_type` discrimination in MaskFormer Head. (#9176)
- Fix wrong padding value in cached MixUp (#9259)
- Rename `utils/typing.py` to `utils/typing_utils.py` to fix `collect_env` error (#9265)
- Fix resume arg conflict (#9287)
- Fix the configs of Faster R-CNN with caffe backbone (#9319)
- Fix torchserve and update related documentation (#9343)

- Fix bbox refine bug with sigmooid activation (#9538)

- Update the docs of GIoU Loss in README (#8810)
- Handle dataset wrapper in `inference_detector` (#9144)
- Update the type of `counts` in COCO’s compressed RLE (#9274)
- Support saving config file in `print_config` (#9276)
- Update docs about video inference (#9305)
- Update guide about model deployment (#9344)
- Fix doc typos of useful tools (#9177)
- Allow to resume from specific checkpoint in CLI (#9284)
- Update FAQ about windows installation issues of pycocotools (#9292)

- @Daa98 made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/9274>
- @lvhan028 made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/9344>

A total of 12 developers contributed to this release.

Thanks @sanbuphy, @Czm369, @Daa98, @jbwang1997, @BIGWangYuDong, @JosonChan1998, @lvhan028, @RunningLeon, @RangiLyu, @Daa98, @ZwwWayne, @hhaAndroid

## v3.0.0rc3 (4/11/2022)

Upgrade the minimum version requirement of MMEngine to 0.3.0 to use `ignore_key` of `ConcatDataset` for training VOC datasets (#9058)

- Support [CrowdDet](https://arxiv.org/abs/2003.09163) and [EIoU Loss](https://ieeexplore.ieee.org/document/9429909)
- Support training detection models in Detectron2
- Refactor Fast R-CNN

- Support [CrowdDet](https://arxiv.org/abs/2003.09163) (#8744)
- Support training detection models in Detectron2 with examples of Mask R-CNN, Faster R-CNN, and RetinaNet (#8672)
- Support [EIoU Loss](https://ieeexplore.ieee.org/document/9429909) (#9086)

- Fix `XMLDataset` image size error (#9216)
- Fix bugs of empty_instances when predicting without nms in roi_head (#9015)
- Fix the config file of DETR (#9158)
- Fix SOLOv2 cannot dealing with empty gt image (#9192)
- Fix inference demo (#9153)
- Add `ignore_key` in VOC `ConcatDataset` (#9058)
- Fix dumping results issue in test scripts. (#9241)
- Fix configs of training coco subsets on MMDet 3.x (#9225)
- Fix corner2hbox of HorizontalBoxes for supporting empty bboxes (#9140)

- Refactor Fast R-CNN (#9132)
- Clean requirements of mmcv-full due to SyncBN (#9207)
- Support training detection models in detectron2 (#8672)
- Add `box_type` support for `DynamicSoftLabelAssigner` (#9179)
- Make scipy as a default dependency in runtime (#9187)
- Update eval_metric (#9062)
- Add `seg_map_suffix` in `BaseDetDataset` (#9088)

- @Wwupup made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/9086>
- @sanbuphy made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/9153>
- @cxiang26 made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/9158>
- @JosonChan1998 made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/9225>

Thanks @wanghonglie, @Wwupup, @sanbuphy, @BIGWangYuDong, @liuyanyi, @cxiang26, @jbwang1997, @ZwwWayne, @yuyoujiang, @RangiLyu, @hhaAndroid, @JosonChan1998, @Czm369

## v3.0.0rc2 (21/10/2022)

- Support [imagenet pre-training](configs/rtmdet/cspnext_imagenet_pretrain) for RTMDet's backbone

- Support [imagenet pre-training](configs/rtmdet/cspnext_imagenet_pretrain) for RTMDet's backbone (#8887)
- Add `CrowdHumanDataset` and Metric (#8430)
- Add `FixShapeResize` to support resize of fixed shape (#8665)

- Fix `ConcatDataset` Import Error (#8909)
- Fix `CircleCI` and `readthedoc` build failed (#8980, #8963)
- Fix bitmap mask translate when `out_shape` is different (#8993)
- Fix inconsistency in `Conv2d` weight channels (#8948)
- Fix bugs when plotting loss curve by analyze_logs.py (#8944)
- Fix type change of labels in `albumentations` (#9074)
- Fix some docs and types error (#8818)
- Update memory occupation of `RTMDet` in metafile (#9098)
- Fix wrong arguments of `OpenImageMetrics` in the config (#9061)

- Refactor standard roi head with `box type` (#8658)

- Support mask concatenation in `BitmapMasks` and `PolygonMasks` (#9006)
- Update PyTorch and dependencies' version in dockerfile (#8845)
- Update `robustness_eval.py` and `print_config` (#8452)
- Make compatible with `ConfigDict` and `dict` in `dense_heads` (#8942)
- Support logging coco metric copypaste (#9012)
- Remove `Normalize` transform (#8913)
- Support jittering the color of different instances of the same class (#8988)
- Add assertion for missing key in `PackDetInputs` (#8982)

- @Chan-Sun made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/8909>
- @MambaWong made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/8913>

- @yuyoujiang made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/8437>
- @sltlls made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/8944>
- @Nioolek made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/8845>
- @wufan-tb made their first contribution in <https://github.com/open-mmlab/mmdetection/pull/9061>

Thanks @RangiLyu, @jbwang1997, @wanghonglie, @Chan-Sun, @RangeKing, @chhluo, @MambaWong, @yuyoujiang, @hhaAndroid, @sltlls, @Nioolek, @ZwwWayne, @wufan-tb

## v3.0.0rc1 (26/9/2022)

- Release a high-precision, low-latency single-stage object detector [RTMDet](configs/rtmdet).

- Fix UT to be compatible with PyTorch 1.6 (#8707)
- Fix `NumClassCheckHook` bug when model is wrapped (#8794)
- Update the right URL of R-50-FPN with BoundedIoULoss (#8805)
- Fix potential bug of indices in RandAugment (#8826)
- Fix some types and links (#8839, #8820, #8793, #8868)
- Fix incorrect background fill values in `FSAF` and `RepPoints` Head (#8813)

- Refactored anchor head and base head with `box type` (#8625)
- Refactored `SemiBaseDetector` and `SoftTeacher` (#8786)
- Add list to dict keys to avoid modify loss dict (#8828)
- Update `analyze_results.py` , `analyze_logs.py` and `loading.py` (#8430, #8402, #8784)
- Support dump results in `test.py` (#8814)
- Check empty predictions in `DetLocalVisualizer._draw_instances` (#8830)
- Fix `floordiv` warning in `SOLO` (#8738)

A total of 16 developers contributed to this release.

Thanks @ZwwWayne, @jbwang1997, @Czm369, @ice-tong, @Zheng-LinXiao, @chhluo, @RangiLyu, @liuyanyi, @wanghonglie, @levan92, @JiayuXu0, @nye0, @hhaAndroid, @xin-li-67, @shuxp, @zytx121

## v3.0.0rc0 (31/8/2022)

We are excited to announce the release of MMDetection 3.0.0rc0. MMDet 3.0.0rc0 is the first version of MMDetection 3.x, a part of the OpenMMLab 2.0 projects. Built upon the new [training engine](https://github.com/open-mmlab/mmengine), MMDet 3.x unifies the interfaces of the dataset, models, evaluation, and visualization with faster training and testing speed. It also provides a general semi-supervised object detection framework and strong baselines.

1. **New engine**. MMDet 3.x is based on [MMEngine](https://github.com/open-mmlab/mmengine), which provides a universal and powerful runner that allows more flexible customizations and significantly simplifies the entry points of high-level interfaces.

2. **Unified interfaces**. As a part of the OpenMMLab 2.0 projects, MMDet 3.x unifies and refactors the interfaces and internal logic of training, testing, datasets, models, evaluation, and visualization. All the OpenMMLab 2.0 projects share the same design in those interfaces and logic to allow the emergence of multi-task/modality algorithms.

3. **Faster speed**. We optimize the training and inference speed for common models and configurations, achieving a faster or similar speed than [Detection2](https://github.com/facebookresearch/detectron2/). Model details of benchmark will be updated in [this note](./benchmark.md#comparison-with-detectron2).

4. **General semi-supervised object detection**. Benefitting from the unified interfaces, we support a general semi-supervised learning framework that works with all the object detectors supported in MMDet 3.x. Please refer to [semi-supervised object detection](../user_guides/semi_det.md) for details.

5. **Strong baselines**. We release strong baselines of many popular models to enable fair comparisons among state-of-the-art models.

6. **New features and algorithms**:

   - Enable all the single-stage detectors to serve as region proposal networks
   - [SoftTeacher](https://arxiv.org/abs/2106.09018)
   - [the updated CenterNet](https://arxiv.org/abs/2103.07461)

7. **More documentation and tutorials**. We add a bunch of documentation and tutorials to help users get started more smoothly. Read it [here](https://mmdetection.readthedocs.io/en/3.x/).

MMDet 3.x has undergone significant changes for better design, higher efficiency, more flexibility, and more unified interfaces.
Besides the changes in API, we briefly list the major breaking changes in this section.
Users can also refer to the [API doc](https://mmdetection.readthedocs.io/en/3.x/) for more details.

- MMDet 3.x runs on PyTorch>=1.6. We have deprecated the support of PyTorch 1.5 to embrace mixed precision training and other new features since PyTorch 1.6. Some models can still run on PyTorch 1.5, but the full functionality of MMDet 3.x is not guaranteed.
- MMDet 3.x relies on MMEngine to run. MMEngine is a new foundational library for training deep learning models of OpenMMLab and is the core dependency of OpenMMLab 2.0 projects. The dependencies of file IO and training are migrated from MMCV 1.x to MMEngine.

- MMDet 3.x relies on MMCV>=2.0.0rc0. Although MMCV no longer maintains the training functionalities since 2.0.0rc0, MMDet 3.x relies on the data transforms, CUDA operators, and image processing interfaces in MMCV. Note that the package `mmcv` is the version that provides pre-built CUDA operators and `mmcv-lite` does not since MMCV 2.0.0rc0, while `mmcv-full` has been deprecated since 2.0.0rc0.

- MMDet 3.x uses Runner in [MMEngine](https://github.com/open-mmlab/mmengine) rather than that in MMCV. The new Runner implements and unifies the building logic of the dataset, model, evaluation, and visualizer. Therefore, MMDet 3.x no longer maintains the building logic of those modules in `mmdet.train.apis` and `tools/train.py`. Those codes have been migrated into [MMEngine](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/runner.py). Please refer to the [migration guide of Runner in MMEngine](https://mmengine.readthedocs.io/en/latest/migration/runner.html) for more details.
- The Runner in MMEngine also supports testing and validation. The testing scripts are also simplified, which has similar logic to that in training scripts to build the runner.

- Learning rate and momentum schedules have been migrated from Hook to [Parameter Scheduler in MMEngine](https://mmengine.readthedocs.io/en/latest/tutorials/param_scheduler.html). Please refer to the [migration guide of Parameter Scheduler in MMEngine](https://mmengine.readthedocs.io/en/latest/migration/param_scheduler.html) for more details.

- The [Runner in MMEngine](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/runner.py) uses a different config structure to ease the understanding of the components in the runner. Users can read the [config example of MMDet 3.x](../user_guides/config.md) or refer to the [migration guide in MMEngine](https://mmengine.readthedocs.io/en/latest/migration/runner.html) for migration details.
- The file names of configs and models are also refactored to follow the new rules unified across OpenMMLab 2.0 projects. The names of checkpoints are not updated for now as there is no BC-breaking of model weights between MMDet 3.x and 2.x. We will progressively replace all the model weights with those trained in MMDet 3.x. Please refer to the [user guides of config](../user_guides/config.md) for more details.

The Dataset classes implemented in MMDet 3.x all inherit from the `BaseDetDataset`, which inherits from the [BaseDataset in MMEngine](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/basedataset.html). In addition to the changes in interfaces, there are several changes in Dataset in MMDet 3.x.

- All the datasets support serializing the internal data list to reduce the memory when multiple workers are built for data loading.
- The internal data structure in the dataset is changed to be self-contained (without losing information like class names in MMDet 2.x) while keeping simplicity.
- The evaluation functionality of each dataset has been removed from the dataset so that some specific evaluation metrics like COCO AP can be used to evaluate the prediction on other datasets.

The data transforms in MMDet 3.x all inherits from `BaseTransform` in MMCV>=2.0.0rc0, which defines a new convention in OpenMMLab 2.0 projects.

Besides the interface changes, there are several changes listed below:

- The functionality of some data transforms (e.g., `Resize`) are decomposed into several transforms to simplify and clarify the usages.
- The format of data dict processed by each data transform is changed according to the new data structure of dataset.
- Some inefficient data transforms (e.g., normalization and padding) are moved into data preprocessor of model to improve data loading and training speed.
- The same data transforms in different OpenMMLab 2.0 libraries have the same augmentation implementation and the logic given the same arguments, i.e., `Resize` in MMDet 3.x and MMSeg 1.x will resize the image in the exact same manner given the same arguments.

#### Model

#### Model

The models in MMDet 3.x all inherit from `BaseModel` in MMEngine, which defines a new convention of models in OpenMMLab 2.0 projects.
Users can refer to [the tutorial of the model in MMengine](https://mmengine.readthedocs.io/en/latest/tutorials/model.html) for more details.
Accordingly, there are several changes as the following:

- The model interfaces, including the input and output formats, are significantly simplified and unified following the new convention in MMDet 3.x.

  Specifically, all the input data in training and testing are packed into `inputs` and `data_samples`, where `inputs` contains model inputs like a list of image tensors, and `data_samples` contains other information of the current data sample such as ground truths, region proposals, and model predictions. In this way, different tasks in MMDet 3.x can share the same input arguments, which makes the models more general and suitable for multi-task learning and some flexible training paradigms like semi-supervised learning.

- The model has a data preprocessor module, which is used to pre-process the input data of the model. In MMDet 3.x, the data preprocessor usually does the necessary steps to form the input images into a batch, such as padding. It can also serve as a place for some special data augmentations or more efficient data transformations like normalization.
- The internal logic of the model has been changed. In MMdet 2.x, model uses `forward_train`, `forward_test`, `simple_test`, and `aug_test` to deal with different model forward logics. In MMDet 3.x and OpenMMLab 2.0, the forward function has three modes: 'loss', 'predict', and 'tensor' for training, inference, and tracing or other purposes, respectively.
  The forward function calls `self.loss`, `self.predict`, and `self._forward` given the modes 'loss', 'predict', and 'tensor', respectively.

The forward function calls `self.loss`, `self.predict`, and `self._forward` given the modes 'loss', 'predict', and 'tensor', respectively.

The evaluation in MMDet 2.x strictly binds with the dataset. In contrast, MMDet 3.x decomposes the evaluation from dataset so that all the detection datasets can evaluate with COCO AP and other metrics implemented in MMDet 3.x.
MMDet 3.x mainly implements corresponding metrics for each dataset, which are manipulated by [Evaluator](https://mmengine.readthedocs.io/en/latest/design/evaluator.html) to complete the evaluation.
Users can build an evaluator in MMDet 3.x to conduct offline evaluation, i.e., evaluate predictions that may not produce in MMDet 3.x with the dataset as long as the dataset and the prediction follow the dataset conventions. More details can be found in the [tutorial in mmengine](https://mmengine.readthedocs.io/en/latest/tutorials/evaluation.html).

The functions of visualization in MMDet 2.x are removed. Instead, in OpenMMLab 2.0 projects, we use [Visualizer](https://mmengine.readthedocs.io/en/latest/design/visualization.html) to visualize data. MMDet 3.x implements `DetLocalVisualizer` to allow visualization of ground truths, model predictions, feature maps, etc., at any place. It also supports sending the visualization data to any external visualization backends such as Tensorboard.

- Optimized training and testing speed of FCOS, RetinaNet, Faster R-CNN, Mask R-CNN, and Cascade R-CNN. The training speed of those models with some common training strategies is also optimized, including those with synchronized batch normalization and mixed precision training.

- Support mixed precision training of all the models. However, some models may get undesirable performance due to some numerical issues. We will update the documentation and list the results (accuracy of failure) of mixed precision training.
- Release strong baselines of some popular object detectors. Their accuracy and pre-trained checkpoints will be released.

- DeepFashion dataset: the config and results have been updated.

1. Support a general semi-supervised learning framework that works with all the object detectors supported in MMDet 3.x. Please refer to [semi-supervised object detection](../user_guides/semi_det.md) for details.
2. Enable all the single-stage detectors to serve as region proposal networks. We give [an example of using FCOS as RPN](../user_guides/single_stage_as_rpn.md).
3. Support a semi-supervised object detection algorithm: [SoftTeacher](https://arxiv.org/abs/2106.09018).

4. Support [the updated CenterNet](https://arxiv.org/abs/2103.07461).
5. Support data structures `HorizontalBoxes` and `BaseBoxes` to encapsulate different kinds of bounding boxes. We are migrating to use data structures of boxes to replace the use of pure tensor boxes. This will unify the usages of different kinds of bounding boxes in MMDet 3.x and MMRotate 1.x to simplify the implementation and reduce redundant codes.

### Planned changes

We list several planned changes of MMDet 3.0.0rc0 so that the community could more comprehensively know the progress of MMDet 3.x. Feel free to create a PR, issue, or discussion if you are interested, have any suggestions and feedback, or want to participate.

1. Test-time augmentation: which is supported in MMDet 2.x, is not implemented in this version due to the limited time slot. We will support it in the following releases with a new and simplified design.
2. Inference interfaces: unified inference interfaces will be supported in the future to ease the use of released models.
3. Interfaces of useful tools that can be used in Jupyter Notebook or Colab: more useful tools that are implemented in the `tools` directory will have their python interfaces so that they can be used in Jupyter Notebook, Colab, and downstream libraries.
4. Documentation: we will add more design docs, tutorials, and migration guidance so that the community can deep dive into our new design, participate the future development, and smoothly migrate downstream libraries to MMDet 3.x.

5. Wandb visualization: MMDet 2.x supports data visualization since v2.25.0, which has not been migrated to MMDet 3.x for now. Since WandB provides strong visualization and experiment management capabilities, a `DetWandBVisualizer` and maybe a hook are planned to fully migrate those functionalities from MMDet 2.x.
6. Full support of WiderFace dataset (#8508) and Fast R-CNN: we are verifying their functionalities and will fix related issues soon.
7. Migrate DETR-series algorithms (#8655, #8533) and YOLOv3 on IPU (#8552) from MMDet 2.x.

Thanks @shuxp, @wanghonglie, @Czm369, @BIGWangYuDong, @zytx121, @jbwang1997, @chhluo, @jshilong, @RangiLyu, @hhaAndroid, @ZwwWayne

# SRGAN (CVPR'2016)

> [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://arxiv.org/abs/1609.04802)

Despite the breakthroughs in accuracy and speed of single image super-resolution using faster and deeper convolutional neural networks, one central problem remains largely unsolved: how do we recover the finer texture details when we super-resolve at large upscaling factors? The behavior of optimization-based super-resolution methods is principally driven by the choice of the objective function. Recent work has largely focused on minimizing the mean squared reconstruction error. The resulting estimates have high peak signal-to-noise ratios, but they are often lacking high-frequency details and are perceptually unsatisfying in the sense that they fail to match the fidelity expected at the higher resolution. In this paper, we present SRGAN, a generative adversarial network (GAN) for image super-resolution (SR). To our knowledge, it is the first framework capable of inferring photo-realistic natural images for 4x upscaling factors. To achieve this, we propose a perceptual loss function which consists of an adversarial loss and a content loss. The adversarial loss pushes our solution to the natural image manifold using a discriminator network that is trained to differentiate between the super-resolved images and original photo-realistic images. In addition, we use a content loss motivated by perceptual similarity instead of similarity in pixel space. Our deep residual network is able to recover photo-realistic textures from heavily downsampled images on public benchmarks. An extensive mean-opinion-score (MOS) test shows hugely significant gains in perceptual quality using SRGAN. The MOS scores obtained with SRGAN are closer to those of the original high-resolution images than to those obtained with any state-of-the-art method.

 <img src="https://user-images.githubusercontent.com/7676947/144035016-8ed4a80b-2d8b-4947-848b-3f8e917a9273.png" width="400"/>
</div >

|     Model    |Dataset|  PSNR | SSIM | Training Resources |                             Download     |
| :-------: | :--------: | :--------: | :-------: | :--------: | :--------: | :----------------: |
| [msrresnet_x4c64b16_1x16_300k_div2k](./msrresnet_x4c64b16_1xb16-1000k_div2k.py) | Set5|30.2252  | 0.8491   | 1          | [model](https://download.openmmlab.com/mmediting/restorers/srresnet_srgan/msrresnet_x4c64b16_1x16_300k_div2k_20200521-61556be5.pth) | [log](https://download.openmmlab.com/mmediting/restorers/srresnet_srgan/msrresnet_x4c64b16_1x16_300k_div2k_20200521_110246.log.json) |

| [msrresnet_x4c64b16_1x16_300k_div2k](./msrresnet_x4c64b16_1xb16-1000k_div2k.py) | Set14|26.7762   |  0.7369   | 1          | [model](https://download.openmmlab.com/mmediting/restorers/srresnet_srgan/msrresnet_x4c64b16_1x16_300k_div2k_20200521-61556be5.pth) | [log](https://download.openmmlab.com/mmediting/restorers/srresnet_srgan/msrresnet_x4c64b16_1x16_300k_div2k_20200521_110246.log.json) |
| [msrresnet_x4c64b16_1x16_300k_div2k](./msrresnet_x4c64b16_1xb16-1000k_div2k.py) | DIV2K|28.9748   | 0.8178   |  1          | [model](https://download.openmmlab.com/mmediting/restorers/srresnet_srgan/msrresnet_x4c64b16_1x16_300k_div2k_20200521-61556be5.pth) | [log](https://download.openmmlab.com/mmediting/restorers/srresnet_srgan/msrresnet_x4c64b16_1x16_300k_div2k_20200521_110246.log.json) |

| [srgan_x4c64b16_1x16_1000k_div2k](./srgan_x4c64b16_1xb16-1000k_div2k.py) |Set5|27.9499  | 0.7846   |1          | [model](https://download.openmmlab.com/mmediting/restorers/srresnet_srgan/srgan_x4c64b16_1x16_1000k_div2k_20200606-a1f0810e.pth) | [log](https://download.openmmlab.com/mmediting/restorers/srresnet_srgan/srgan_x4c64b16_1x16_1000k_div2k_20200506_191442.log.json) |
| [srgan_x4c64b16_1x16_1000k_div2k](./srgan_x4c64b16_1xb16-1000k_div2k.py) |Set14|24.7383   |0.6491   |  1          | [model](https://download.openmmlab.com/mmediting/restorers/srresnet_srgan/srgan_x4c64b16_1x16_1000k_div2k_20200606-a1f0810e.pth) | [log](https://download.openmmlab.com/mmediting/restorers/srresnet_srgan/srgan_x4c64b16_1x16_1000k_div2k_20200506_191442.log.json) |

| [srgan_x4c64b16_1x16_1000k_div2k](./srgan_x4c64b16_1xb16-1000k_div2k.py) |DIV2K|26.5697   |0.7365   |  1          | [model](https://download.openmmlab.com/mmediting/restorers/srresnet_srgan/srgan_x4c64b16_1x16_1000k_div2k_20200606-a1f0810e.pth) | [log](https://download.openmmlab.com/mmediting/restorers/srresnet_srgan/srgan_x4c64b16_1x16_1000k_div2k_20200506_191442.log.json) |

**Train**

<details>

```shell
# cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/srgan_resnet/srgan_x4c64b16_1xb16-1000k_div2k.py

python tools/train.py configs/srgan_resnet/srgan_x4c64b16_1xb16-1000k_div2k.py

./tools/dist_train.sh configs/srgan_resnet/srgan_x4c64b16_1xb16-1000k_div2k.py 8
```

```

**Test**

<details>

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/srgan_resnet/srgan_x4c64b16_1xb16-1000k_div2k.py https://download.openmmlab.com/mmediting/restorers/srresnet_srgan/srgan_x4c64b16_1x16_1000k_div2k_20200606-a1f0810e.pth

python tools/test.py configs/srgan_resnet/srgan_x4c64b16_1xb16-1000k_div2k.py https://download.openmmlab.com/mmediting/restorers/srresnet_srgan/srgan_x4c64b16_1x16_1000k_div2k_20200606-a1f0810e.pth

./tools/dist_test.sh configs/srgan_resnet/srgan_x4c64b16_1xb16-1000k_div2k.py https://download.openmmlab.com/mmediting/restorers/srresnet_srgan/srgan_x4c64b16_1x16_1000k_div2k_20200606-a1f0810e.pth 8
```

```bibtex
@inproceedings{ledig2016photo,
  title={Photo-realistic single image super-resolution using a generative adversarial network},
  author={Ledig, Christian and Theis, Lucas and Husz{\'a}r, Ferenc and Caballero, Jose and Cunningham, Andrew and Acosta, Alejandro and Aitken, Andrew and Tejani, Alykhan and Totz, Johannes and Wang, Zehan},
}
```

  <img src="resources/mmdet3d-logo.png" width="600"/>
    <sup>
      </a>
    </sup>
    <sup>
      </a>
    </sup>
  </div>

[![PyPI](https://img.shields.io/pypi/v/mmdet3d)](https://pypi.org/project/mmdet3d)
[![docs](https://img.shields.io/badge/docs-latest-blue)](https://mmdetection3d.readthedocs.io/zh_CN/latest/)
[![badge](https://github.com/open-mmlab/mmdetection3d/workflows/build/badge.svg)](https://github.com/open-mmlab/mmdetection3d/actions)
[![codecov](https://codecov.io/gh/open-mmlab/mmdetection3d/branch/main/graph/badge.svg)](https://codecov.io/gh/open-mmlab/mmdetection3d)

[![license](https://img.shields.io/github/license/open-mmlab/mmdetection3d.svg)](https://github.com/open-mmlab/mmdetection3d/blob/main/LICENSE)
[![open issues](https://isitmaintained.com/badge/open/open-mmlab/mmdetection3d.svg)](https://github.com/open-mmlab/mmdetection3d/issues)
[![issue resolution](https://isitmaintained.com/badge/resolution/open-mmlab/mmdetection3d.svg)](https://github.com/open-mmlab/mmdetection3d/issues)

[📘使用文档](https://mmdetection3d.readthedocs.io/zh_CN/latest/) |
[🛠️安装教程](https://mmdetection3d.readthedocs.io/zh_CN/latest/get_started.html) |
[👀模型库](https://mmdetection3d.readthedocs.io/zh_CN/latest/model_zoo.html) |
[🆕更新日志](https://mmdetection3d.readthedocs.io/en/latest/notes/changelog.html) |
[🚀进行中的项目](https://github.com/open-mmlab/mmdetection3d/projects) |
[🤔报告问题](https://github.com/open-mmlab/mmdetection3d/issues/new/choose)

</div>

</div>

</div>

## 简介

MMDetection3D 是一个基于 PyTorch 的目标检测开源工具箱，下一代面向 3D 检测的平台。它是 [OpenMMlab](https://openmmlab.com/) 项目的一部分。

主分支代码目前支持 PyTorch 1.8 以上的版本。

![demo image](resources/mmdet3d_outdoor_demo.gif)

- **支持多模态/单模态的检测器**

  支持多模态/单模态检测器，包括 MVXNet，VoteNet，PointPillars 等。

- **支持户内/户外的数据集**

  支持室内/室外的 3D 检测数据集，包括 ScanNet，SUNRGB-D，Waymo，nuScenes，Lyft，KITTI。对于 nuScenes 数据集，我们也支持 [nuImages 数据集](https://github.com/open-mmlab/mmdetection3d/tree/main/configs/nuimages)。

- **与 2D 检测器的自然整合**

  [MMDetection](https://github.com/open-mmlab/mmdetection/blob/3.x/docs/zh_cn/model_zoo.md) 支持的 **300+ 个模型，40+ 的论文算法**，和相关模块都可以在此代码库中训练或使用。

- **性能高**

  训练速度比其他代码库更快。下表可见主要的对比结果。更多的细节可见[基准测评文档](./docs/zh_cn/notes/benchmarks.md)。我们对比了每秒训练的样本数（值越高越好）。其他代码库不支持的模型被标记为 `✗`。

  |       Methods       | MMDetection3D | [OpenPCDet](https://github.com/open-mmlab/OpenPCDet) | [votenet](https://github.com/facebookresearch/votenet) | [Det3D](https://github.com/poodarchu/Det3D) |

|       Methods       | MMDetection3D | [OpenPCDet](https://github.com/open-mmlab/OpenPCDet) | [votenet](https://github.com/facebookresearch/votenet) | [Det3D](https://github.com/poodarchu/Det3D) |
  | :-----------------: | :-----------: | :--------------------------------------------------: | :----------------------------------------------------: | :-----------------------------------------: |
  |       VoteNet       |      358      |                          ✗                           |                           77                           |                      ✗                      |
  |  PointPillars-car   |      141      |                          ✗                           |                           ✗                            |                     140                     |
  | PointPillars-3class |      107      |                          44                          |                           ✗                            |                      ✗                      |

| PointPillars-3class |      107      |                          44                          |                           ✗                            |                      ✗                      |
  |       SECOND        |      40       |                          30                          |                           ✗                            |                      ✗                      |
  |       Part-A2       |      17       |                          14                          |                           ✗                            |                      ✗                      |

和 [MMDetection](https://github.com/open-mmlab/mmdetection)，[MMCV](https://github.com/open-mmlab/mmcv) 一样，MMDetection3D 也可以作为一个库去支持各式各样的项目。

## 最新进展

### 亮点

**我们将 `1.1` 分支重命名为 `main` 并将默认分支从 `master` 切换到 `main`。我们鼓励用户迁移到最新版本，请参考 [迁移指南](docs/en/migration.md)以了解更多细节。**

## 最新进展

### 亮点

我们在 SemanticKITTI 上构建了一个全面的点云语义分割基准，包括 Cylinder3D 、MinkUNet 和 SPVCNN 方法。其中，改进后的 MinkUNetv2 在验证集上可以达到 70.3 mIoU。我们还在 `projects` 中支持了 BEVFusion 的训练和全新的 3D 占有网格预测网络 TPVFormer。更多关于 3D 感知的新功能正在进行中。请继续关注！

**v1.2.0** 版本已经在 2023.7.4 发布：

- 在 `mmdet3d/config`中支持 [新Config样式](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#a-pure-python-style-configuration-file-beta)
- 在 `projects` 中支持 [DSVT](<(https://arxiv.org/abs/2301.06051)>) 的推理
- 支持通过 `mim` 从 [OpenDataLab](https://opendatalab.com/) 下载数据集

**v1.1.1** 版本已经在 2023.5.30 发布：

- 在 `projects` 中支持 [TPVFormer](https://arxiv.org/pdf/2302.07817.pdf)
- 在 `projects` 中支持 BEVFusion 的训练
- 支持基于激光雷达的 3D 语义分割基准

## 安装

请参考[快速入门文档](https://mmdetection3d.readthedocs.io/zh_CN/latest/get_started.html)进行安装。

## 教程

<details>
<summary>用户指南</summary>

## 安装

## 教程

<details>

- [训练 & 测试](https://mmdetection3d.readthedocs.io/zh_CN/latest/user_guides/index.html#train-test)
  - [学习配置文件](https://mmdetection3d.readthedocs.io/zh_CN/latest/user_guides/config.html)
  - [坐标系](https://mmdetection3d.readthedocs.io/zh_CN/latest/user_guides/coord_sys_tutorial.html)
  - [数据预处理](https://mmdetection3d.readthedocs.io/zh_CN/latest/user_guides/dataset_prepare.html)
  - [自定义数据预处理流程](https://mmdetection3d.readthedocs.io/zh_CN/latest/user_guides/data_pipeline.html)
  - [在标注数据集上测试和训练](https://mmdetection3d.readthedocs.io/zh_CN/latest/user_guides/train_test.html)
  - [推理](https://mmdetection3d.readthedocs.io/zh_CN/latest/user_guides/inference.html)
  - [在自定义数据集上进行训练](https://mmdetection3d.readthedocs.io/zh_CN/latest/user_guides/new_data_model.html)

- [推理](https://mmdetection3d.readthedocs.io/zh_CN/latest/user_guides/inference.html)
- [实用工具](https://mmdetection3d.readthedocs.io/zh_CN/latest/user_guides/index.html#useful-tools)

<details>
<summary>进阶教程</summary>

- [数据集](https://mmdetection3d.readthedocs.io/zh_CN/latest/advanced_guides/index.html#datasets)
  - [KITTI 数据集](https://mmdetection3d.readthedocs.io/zh_CN/latest/advanced_guides/datasets/kitti.html)
  - [NuScenes 数据集](https://mmdetection3d.readthedocs.io/zh_CN/latest/advanced_guides/datasets/nuscenes.html)
  - [Lyft 数据集](https://mmdetection3d.readthedocs.io/zh_CN/latest/advanced_guides/datasets/lyft.html)
  - [Waymo 数据集](https://mmdetection3d.readthedocs.io/zh_CN/latest/advanced_guides/datasets/waymo.html)
  - [SUN RGB-D 数据集](https://mmdetection3d.readthedocs.io/zh_CN/latest/advanced_guides/datasets/sunrgbd.html)

- [SUN RGB-D 数据集](https://mmdetection3d.readthedocs.io/zh_CN/latest/advanced_guides/datasets/sunrgbd.html)
  - [ScanNet 数据集](https://mmdetection3d.readthedocs.io/zh_CN/latest/advanced_guides/datasets/scannet.html)
  - [S3DIS 数据集](https://mmdetection3d.readthedocs.io/zh_CN/latest/advanced_guides/datasets/s3dis.html)
  - [SemanticKITTI 数据集](https://mmdetection3d.readthedocs.io/zh_CN/latest/advanced_guides/datasets/semantickitti.html)
- [支持的任务](https://mmdetection3d.readthedocs.io/zh_CN/latest/advanced_guides/index.html#supported-tasks)
  - [基于激光雷达的 3D 检测](https://mmdetection3d.readthedocs.io/zh_CN/latest/advanced_guides/supported_tasks/lidar_det3d.html)
  - [基于视觉的 3D 检测](https://mmdetection3d.readthedocs.io/zh_CN/latest/advanced_guides/supported_tasks/vision_det3d.html)
  - [基于激光雷达的 3D 语义分割](https://mmdetection3d.readthedocs.io/zh_CN/latest/advanced_guides/supported_tasks/lidar_sem_seg3d.html)

- [基于激光雷达的 3D 语义分割](https://mmdetection3d.readthedocs.io/zh_CN/latest/advanced_guides/supported_tasks/lidar_sem_seg3d.html)
- [自定义项目](https://mmdetection3d.readthedocs.io/zh_CN/latest/advanced_guides/index.html#customization)
  - [自定义数据集](https://mmdetection3d.readthedocs.io/zh_CN/latest/advanced_guides/customize_dataset.html)
  - [自定义模型](https://mmdetection3d.readthedocs.io/zh_CN/latest/advanced_guides/customize_models.html)
  - [自定义运行时配置](https://mmdetection3d.readthedocs.io/zh_CN/latest/advanced_guides/customize_runtime.html)

</div>
  <tbody>
      <td>
        <b>主干网络</b>
      </td>
      <td>
        <b>检测头</b>
      </td>
      <td>
        <b>特性</b>
      </td>
    </tr>
      <td>
      <ul>
        <li><a href="configs/pointnet2">PointNet (CVPR'2017)</a></li>

<b>检测头</b>
      </td>
      <td>
        <b>特性</b>
      </td>
    </tr>
      <td>
      <ul>
        <li><a href="configs/pointnet2">PointNet++ (NeurIPS'2017)</a></li>
        <li><a href="configs/dgcnn">DGCNN (TOG'2019)</a></li>
        <li>DLA (CVPR'2018)</li>
        <li>MinkResNet (CVPR'2019)</li>
        <li><a href="configs/minkunet">MinkUNet (CVPR'2019)</a></li>
        <li><a href="configs/cylinder3d">Cylinder3D (CVPR'2021)</a></li>
      </ul>
      </td>
      <td>
      <ul>
        <li><a href="configs/free_anchor">FreeAnchor (NeurIPS'2019)</a></li>
      </ul>
      </td>
      <td>
      <ul>
        <li><a href="configs/dynamic_voxelization">Dynamic Voxelization (CoRL'2019)</a></li>
      </ul>
      </td>
    </tr>
</td>
    </tr>
  </tbody>
</table>

  <b>算法模型</b>
</div>

</ul>
      </td>
    </tr>
</td>
    </tr>
  </tbody>
</table>

</div>
  <tbody>
    <tr align="center" valign="middle">
      <td>
        <b>激光雷达 3D 目标检测</b>
      </td>
      <td>
        <b>相机 3D 目标检测</b>
      </td>
      <td>
        <b>多模态 3D 目标检测</b>
      </td>
      <td>
        <b>3D 语义分割</b>
      </td>
    </tr>
      <td>
        <li><b>室外</b></li>
        <ul>
            <li><a href="configs/second">SECOND (Sensor'2018)</a></li>
            <li><a href="configs/pointpillars">PointPillars (CVPR'2019)</a></li>
            <li><a href="configs/ssn">SSN (ECCV'2020)</a></li>
            <li><a href="configs/3dssd">3DSSD (CVPR'2020)</a></li>
            <li><a href="configs/sassd">SA-SSD (CVPR'2020)</a></li>
            <li><a href="configs/point_rcnn">PointRCNN (CVPR'2019)</a></li>
            <li><a href="configs/parta2">Part-A2 (TPAMI'2020)</a></li>

<li><a href="configs/point_rcnn">PointRCNN (CVPR'2019)</a></li>
            <li><a href="configs/centerpoint">CenterPoint (CVPR'2021)</a></li>
            <li><a href="configs/pv_rcnn">PV-RCNN (CVPR'2020)</a></li>
            <li><a href="projects/CenterFormer">CenterFormer (ECCV'2022)</a></li>
        </ul>
        <li><b>室内</b></li>
        <ul>
            <li><a href="configs/votenet">VoteNet (ICCV'2019)</a></li>
            <li><a href="configs/h3dnet">H3DNet (ECCV'2020)</a></li>
            <li><a href="configs/groupfree3d">Group-Free-3D (ICCV'2021)</a></li>
            <li><a href="configs/fcaf3d">FCAF3D (ECCV'2022)</a></li>
            <li><a href="projects/TR3D">TR3D (ArXiv'2023)</a></li>
      </ul>
      </td>
      <td>
        <ul>
          <li><a href="configs/imvoxelnet">ImVoxelNet (WACV'2022)</a></li>

</ul>
      </td>
      <td>
        <ul>
          <li><a href="configs/smoke">SMOKE (CVPRW'2020)</a></li>
          <li><a href="configs/fcos3d">FCOS3D (ICCVW'2021)</a></li>
          <li><a href="configs/pgd">PGD (CoRL'2021)</a></li>
          <li><a href="configs/monoflex">MonoFlex (CVPR'2021)</a></li>
          <li><a href="projects/DETR3D">DETR3D (CoRL'2021)</a></li>
          <li><a href="projects/PETR">PETR (ECCV'2022)</a></li>
        </ul>
        <li><b>Indoor</b></li>
        <ul>
        </ul>
      </td>
      <td>
        <ul>
          <li><a href="configs/mvxnet">MVXNet (ICRA'2019)</a></li>
          <li><a href="projects/BEVFusion">BEVFusion (ICRA'2023)</a></li>
        </ul>
        <ul>

<li><a href="configs/mvxnet">MVXNet (ICRA'2019)</a></li>
        </ul>
        <ul>
          <li><a href="configs/imvotenet">ImVoteNet (CVPR'2020)</a></li>
        </ul>
      </td>
      <td>
        <ul>
          <li><a href="configs/minkunet">MinkUNet (CVPR'2019)</a></li>
          <li><a href="configs/spvcnn">SPVCNN (ECCV'2020)</a></li>
          <li><a href="configs/cylinder3d">Cylinder3D (CVPR'2021)</a></li>
          <li><a href="projects/TPVFormer">TPVFormer (CVPR'2023)</a></li>
        </ul>
        <ul>
          <li><a href="configs/pointnet2">PointNet++ (NeurIPS'2017)</a></li>
          <li><a href="configs/paconv">PAConv (CVPR'2021)</a></li>
          <li><a href="configs/dgcnn">DGCNN (TOG'2019)</a></li>
        </ul>
      </ul>
      </td>
    </tr>
</td>
    </tr>
  </tbody>
</table>

<li><a href="configs/dgcnn">DGCNN (TOG'2019)</a></li>
        </ul>
      </ul>
      </td>
    </tr>
</td>
    </tr>
  </tbody>
</table>

|               | ResNet | VoVNet | Swin-T | PointNet++ | SECOND | DGCNN | RegNetX | DLA | MinkResNet | Cylinder3D | MinkUNet |
| :-----------: | :----: | :----: | :----: | :--------: | :----: | :---: | :-----: | :-: | :--------: | :--------: | :------: |
|    SECOND     |   ✗    |   ✗    |   ✗    |     ✗      |   ✓    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |
| PointPillars  |   ✗    |   ✗    |   ✗    |     ✗      |   ✓    |   ✗   |    ✓    |  ✗  |     ✗      |     ✗      |    ✗     |
|  FreeAnchor   |   ✗    |   ✗    |   ✗    |     ✗      |   ✗    |   ✗   |    ✓    |  ✗  |     ✗      |     ✗      |    ✗     |
|    VoteNet    |   ✗    |   ✗    |   ✗    |     ✓      |   ✗    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |

|    H3DNet     |   ✗    |   ✗    |   ✗    |     ✓      |   ✗    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |
|     3DSSD     |   ✗    |   ✗    |   ✗    |     ✓      |   ✗    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |
|    Part-A2    |   ✗    |   ✗    |   ✗    |     ✗      |   ✓    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |
|    MVXNet     |   ✓    |   ✗    |   ✗    |     ✗      |   ✓    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |
|  CenterPoint  |   ✗    |   ✗    |   ✗    |     ✗      |   ✓    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |
|      SSN      |   ✗    |   ✗    |   ✗    |     ✗      |   ✗    |   ✗   |    ✓    |  ✗  |     ✗      |     ✗      |    ✗     |

|   ImVoteNet   |   ✓    |   ✗    |   ✗    |     ✓      |   ✗    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |
|    FCOS3D     |   ✓    |   ✗    |   ✗    |     ✗      |   ✗    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |
|  PointNet++   |   ✗    |   ✗    |   ✗    |     ✓      |   ✗    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |
| Group-Free-3D |   ✗    |   ✗    |   ✗    |     ✓      |   ✗    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |
|  ImVoxelNet   |   ✓    |   ✗    |   ✗    |     ✗      |   ✗    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |
|    PAConv     |   ✗    |   ✗    |   ✗    |     ✓      |   ✗    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |

|     DGCNN     |   ✗    |   ✗    |   ✗    |     ✗      |   ✗    |   ✓   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |
|     SMOKE     |   ✗    |   ✗    |   ✗    |     ✗      |   ✗    |   ✗   |    ✗    |  ✓  |     ✗      |     ✗      |    ✗     |
|      PGD      |   ✓    |   ✗    |   ✗    |     ✗      |   ✗    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |
|   MonoFlex    |   ✗    |   ✗    |   ✗    |     ✗      |   ✗    |   ✗   |    ✗    |  ✓  |     ✗      |     ✗      |    ✗     |
|    SA-SSD     |   ✗    |   ✗    |   ✗    |     ✗      |   ✓    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |
|    FCAF3D     |   ✗    |   ✗    |   ✗    |     ✗      |   ✗    |   ✗   |    ✗    |  ✗  |     ✓      |     ✗      |    ✗     |

|    PV-RCNN    |   ✗    |   ✗    |   ✗    |     ✗      |   ✓    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |
|  Cylinder3D   |   ✗    |   ✗    |   ✗    |     ✗      |   ✗    |   ✗   |    ✗    |  ✗  |     ✗      |     ✓      |    ✗     |
|   MinkUNet    |   ✗    |   ✗    |   ✗    |     ✗      |   ✗    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✓     |
|    SPVCNN     |   ✗    |   ✗    |   ✗    |     ✗      |   ✗    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✓     |
|   BEVFusion   |   ✗    |   ✗    |   ✓    |     ✗      |   ✓    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |
| CenterFormer  |   ✗    |   ✗    |   ✗    |     ✗      |   ✓    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |

|     TR3D      |   ✗    |   ✗    |   ✗    |     ✗      |   ✗    |   ✗   |    ✗    |  ✗  |     ✓      |     ✗      |    ✗     |
|    DETR3D     |   ✓    |   ✓    |   ✗    |     ✗      |   ✗    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |
|     PETR      |   ✗    |   ✓    |   ✗    |     ✗      |   ✗    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |
|   TPVFormer   |   ✓    |   ✗    |   ✗    |     ✗      |   ✗    |   ✗   |    ✗    |  ✗  |     ✗      |     ✗      |    ✗     |

**注意：**[MMDetection](https://github.com/open-mmlab/mmdetection/blob/3.x/docs/zh_cn/model_zoo.md) 支持的基于 2D 检测的 **300+ 个模型，40+ 的论文算法**在 MMDetection3D 中都可以被训练或使用。

## 常见问题

## 贡献指南

我们感谢所有的贡献者为改进和提升 MMDetection3D 所作出的努力。请参考[贡献指南](docs/en/notes/contribution_guides.md)来了解参与项目贡献的相关指引。

## 致谢

## 常见问题

## 贡献指南

## 致谢

MMDetection3D 是一款由来自不同高校和企业的研发人员共同参与贡献的开源项目。我们感谢所有为项目提供算法复现和新功能支持的贡献者，以及提供宝贵反馈的用户。我们希望这个工具箱和基准测试可以为社区提供灵活的代码工具，供用户复现已有算法并开发自己的新的 3D 检测模型。

## 引用

如果你觉得本项目对你的研究工作有所帮助，请参考如下 bibtex 引用 MMdetection3D：

```latex
@misc{mmdet3d2020,
    title={{MMDetection3D: OpenMMLab} next-generation platform for general {3D} object detection},
    author={MMDetection3D Contributors},
    howpublished = {\url{https://github.com/open-mmlab/mmdetection3d}},
}
```

## 开源许可证

扫描下方的二维码可关注 OpenMMLab 团队的[知乎官方账号](https://www.zhihu.com/people/openmmlab)，加入 OpenMMLab 团队的[官方交流 QQ 群](https://jq.qq.com/?_wv=1027&k=K0QI8ByU)，或通过添加微信“Open小喵Lab”加入官方交流微信群。

<img src="https://user-images.githubusercontent.com/58739961/187154320-f3312cdf-31f2-4316-9dbb-8d7b0e1b7e08.jpg" height="400" />  <img src="https://user-images.githubusercontent.com/25839884/203904835-62392033-02d4-4c73-a68c-c9e4c1e2b07f.jpg" height="400" />  <img src="https://user-images.githubusercontent.com/58739961/187151778-d17c1368-125f-4fde-adbe-38cc6eb3be98.jpg" height="400" />
</div>

# 教程 2：准备数据集

在本节中，我们将详细介绍如何准备数据并在本仓库的不同任务中采用适当的数据集。

我们支持不同任务的多个数据集。
在MMagic中，有两种方法可以将数据集用于训练和测试模型：

1. 直接使用下载的数据集
2. 在使用下载的数据集之前对其进行预处理

本文的结构如下：

- \[教程 2：准备数据集\](#教程 2：准备数据集)
  - [下载数据集](#下载数据集)
  - [准备数据集](#准备数据集)
  - [MMagic中的数据集概述](#MMagic中的数据集概述)

## 下载数据集

首先，建议您从官方的页面下载数据集。
大多数数据集在下载后都是可用的，因此您只需确保文件夹结构正确，无需进一步准备。
例如，您可以通过从[主页](http://toflow.csail.mit.edu/)下载，来简单地准备Vimeo90K-triplet数据集.

## 准备数据集

一些数据集需要在训练或测试之前进行预处理。我们在
[tools/dataset_converters](https://github.com/open-mmlab/mmagic/tree/main/tools/dataset_converters)中支持许多用来准备数据集的脚本。
您可以遵循每个数据集的教程来运行脚本。例如，我们建议将DIV2K图像裁剪为子图像。我们提供了一个脚本来准备裁剪的DIV2K数据集。可以运行以下命令：

```shell
```

## MMagic中的数据集概述

我们支持详细的教程，并根据不同的任务进行拆分。

请查看我们的数据集概览，了解不同任务的数据准备。

如果您对MMagic中数据集的更多细节感兴趣，请查看[进阶教程](../howto/dataset.md)。

本节我们继续使用 demo 脚本演示 2D 人体关键点的识别。同样的，用户仍要确保开发环境已经安装了 3.0 版本以上的 [mmdet](https://github.com/open-mmlab/mmdetection) 。

### 2D 人体姿态 Top-Down 图片检测

#### 使用整张图片作为输入进行检测

此时输入的整张图片会被当作 bounding box 使用。

```shell
```

用户可以在 [model zoo](https://mmpose.readthedocs.io/zh_CN/latest/model_zoo/body_2d_keypoint.html) 获取预训练好的关键点识别模型。

这里我们用 [coco model](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_coco_256x192-b9e0b3ab_20200708.pth) 来进行演示：

```shell

```

使用 CPU 推理：

```shell
```

可视化结果如下：

#### 使用 MMDet 做人体 bounding box 检测

使用 MMDet 进行识别的命令如下所示：

```shell

```shell
```

结合我们的具体例子：

```shell
```

可视化结果如下：

--input tests/data/coco/000000197388.jpg --show --draw-heatmap \
```

可视化结果如下：

想要本地保存识别结果，用户需要加上 `--save-predictions` 。

### 2D 人体姿态 Top-Down 视频检测

我们的脚本同样支持视频作为输入，由 MMDet 完成人体检测后 MMPose 完成 Top-Down 的姿态预估，视频推理时 `${INPUT_PATH}` 既可以是本地视频文件的路径也可以是视频文件的 **URL** 地址。

例如：

```shell

```

### 2D 人体姿态 Bottom-Up 图片和视频识别检测

除了 Top-Down ，我们也支持 Bottom-Up 不依赖人体识别器的人体姿态预估识别，使用方式如下：

```shell
```

结合具体示例如下：

```shell

--input tests/data/coco/000000197388.jpg --output-root=vis_results \
```

其可视化结果如图所示：

### 使用 Inferencer 进行 2D 人体姿态识别检测

```shell
```

该命令会对输入的 `tests/data/posetrack18` 下的视频进行推理并且把可视化结果存入 `vis_results/posetrack18` 文件夹下。

Inferencer 支持保存姿态的检测结果，具体的使用可参考 [inferencer document](https://mmpose.readthedocs.io/zh_CN/dev-1.x/user_guides/inference.html) 。

### 加速推理

### 加速推理

对于 top-down 结构的模型，用户可以通过修改配置文件来加速，更多具体例子可以参考：

1. 设置 `model.test_cfg.flip_test=False`，如 [topdown-res50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_res50_8xb64-210e_coco-256x192.py#L56) 所示。
2. 使用更快的人体 bounding box 检测器，可参考 [MMDetection](https://mmdetection.readthedocs.io/zh_CN/3.x/model_zoo.html) 。

RTMPose-Deploy 是一个进行 RTMPose 本地化部署的 C++ 代码示例。

目前，RTMPose-Deploy 已完成在 Windows 系统上使用 OnnxRuntime CPU 和TensorRT 对 RTMDet 和 RTMPose 完成了部署。

| 部署框架        | 仓库                                                                 |
| --------------- | -------------------------------------------------------------------- |
| ONNXRuntime-CPU | [RTMPose-Deploy](https://github.com/HW140701/RTMPose-Deploy)         |
| TensorRT        | [rtmpose_tensorrt](https://github.com/Dominic23331/rtmpose_tensorrt) |

# MacaquePose: A novel ‘in the wild’macaque monkey pose dataset for markerless motion capture

<details>
<summary align="right"><a href="https://www.ncbi.nlm.nih.gov/pmc/articles/pmc7874091/">MacaquePose (bioRxiv'2020)</a></summary>

```bibtex
@article{labuguen2020macaquepose,
  title={MacaquePose: A novel ‘in the wild’macaque monkey pose dataset for markerless motion capture},
  author={Labuguen, Rollyn and Matsumoto, Jumpei and Negrete, Salvador and Nishimaru, Hiroshi and Nishijo, Hisao and Takada, Masahiko and Go, Yasuhiro and Inoue, Ken-ichi and Shibata, Tomohiro},
  journal={bioRxiv},
  publisher={Cold Spring Harbor Laboratory}
}
```

# Overview

This chapter introduces you to the framework of MMSegmentation, and the basic conception of semantic segmentation. It also provides links to detailed tutorials about MMSegmentation.

## What is semantic segmentation?

Semantic segmentation is the task of clustering parts of an image together that belong to the same object class.
It is a form of pixel-level prediction because each pixel in an image is classified according to a category.
Some example benchmarks for this task are [Cityscapes](https://www.cityscapes-dataset.com/benchmarks/), [PASCAL VOC](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/) and [ADE20K](https://groups.csail.mit.edu/vision/datasets/ADE20K/).
Models are usually evaluated with the Mean Intersection-Over-Union (Mean IoU) and Pixel Accuracy metrics.

## What is MMSegmentation?

MMSegmentation is a toolbox that provides a framework for unified implementation and evaluation of semant
ic segmentation methods,

and contains high-quality implementations of popular semantic segmentation methods and datasets.

MMSeg consists of 7 main parts including apis, structures, datasets, models, engine, evaluation and visualization.

- **apis** provides high-level APIs for model inference.

- **structures** provides segmentation data structure `SegDataSample`.

- **datasets** supports various datasets for semantic segmentation.

  - **transforms** contains a lot of useful data augmentation transforms.

- **models** is the most vital part for segmentors and contains different components of a segmentor.

  - **segmentors** defines all of the segmentation model classes.
  - **data_preprocessors** works for preprocessing the input data of the model.
  - **backbones** contains various backbone networks that transform an image to feature maps.

- **data_preprocessors** works for preprocessing the input data of the model.
  - **necks** contains various neck components that connect the backbone and heads.
  - **decode_heads** contains various head components that take feature map as input and predict segmentation results.
  - **losses** contains various loss functions.

- **engine** is a part for runtime components that extends function of [MMEngine](https://github.com/open-mmlab/mmengine).

  - **optimizers** provides optimizers and optimizer wrappers.
  - **hooks** provides various hooks of the runner.

- **evaluation** provides different metrics for evaluating model performance.

- **visualization** is for visualizing segmentation results.

## How to use this documentation

Here is a detailed step-by-step guide to learn more about MMSegmentation:

1. For installation instructions, please see [get_started](getting_started.md).

2. For beginners, MMSegmentation is the best place to start the journey of semantic segmentation
   as there are many SOTA and classic segmentation [models](model_zoo.md),
   and it is easier to carry out a segmentation task by plugging together building blocks and convenient high-level apis.
   Refer to the tutorials below for the basic usage of MMSegmentation:

   - [Config](user_guides/1_config.md)
   - [Dataset Preparation](user_guides/2_dataset_prepare.md)
   - [Inference](user_guides/3_inference.md)
   - [Train and Test](user_guides/4_train_test.md)

3. If you would like to learn about the fundamental classes and features that make MMSegmentation work,
   please refer to the tutorials below to dive deeper:

   - [Data flow](advanced_guides/data_flow.md)

please refer to the tutorials below to dive deeper:

   - [Structures](advanced_guides/structures.md)
   - [Models](advanced_guides/models.md)
   - [Datasets](advanced_guides/datasets.md)
   - [Evaluation](advanced_guides/evaluation.md)

4. MMSegmentation also provide tutorials for customization and advanced research,
   please refer to the below guides to build your own segmentation project:

   - [Add new models](advanced_guides/add_models.md)
   - [Add new datasets](advanced_guides/add_datasets.md)
   - [Add new transforms](advanced_guides/add_transforms.md)
   - [Customize runtime](advanced_guides/customize_runtime.md)

5. If you are more familiar with MMSegmentation v0.x, there is documentation about migration from MMSegmentation v0.x to v1.x

   - [migration](migration/index.rst)

- [Paper with code](https://paperswithcode.com/task/semantic-segmentation/codeless#task-home)

## 2D Human Whole-Body Pose Demo

### 2D 人体全身姿态 Top-Down 图片识别

```shell
```

用户可以在 [model zoo](https://mmpose.readthedocs.io/zh_CN/dev-1.x/model_zoo/2d_wholebody_keypoint.html) 获取预训练好的关键点识别模型。

这里我们用 [coco-wholebody_vipnas_res50_dark](https://download.openmmlab.com/mmpose/top_down/vipnas/vipnas_res50_wholebody_256x192_dark-67c0ce35_20211112.pth) 来进行演示：

```shell
    configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_vipnas-res50_dark-8xb64-210e_coco-wholebody-256x192.py \
    https://download.openmmlab.com/mmpose/top_down/vipnas/vipnas_res50_wholebody_256x192_dark-67c0ce35_20211112.pth \
    --out-file vis_results.jpg
```

使用 CPU 推理：

```shell

--out-file vis_results.jpg
```

使用 CPU 推理：

```shell
```

#### 使用 MMDet 进行人体 bounding box 检测

使用 MMDet 进行识别的命令格式如下：

```shell
```

具体可例如：

```shell

```

具体可例如：

```shell
    configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_hrnet-w48_dark-8xb32-210e_coco-wholebody-384x288.py \
    https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_coco_wholebody_384x288_dark-f5726563_20200918.pth \
    --input tests/data/coco/000000196141.jpg \
    --output-root vis_results/ --show
```

### 2D 人体全身姿态 Top-Down 视频识别检测

我们的脚本同样支持视频作为输入，由 MMDet 完成人体检测后 MMPose 完成 Top-Down 的姿态预估。

例如：

```shell

例如：

```shell
    --input https://user-images.githubusercontent.com/87690686/137440639-fb08603d-9a35-474e-b65f-46b5c06b68d6.mp4 \
```

可视化结果如下：

<img src="https://user-images.githubusercontent.com/87690686/190854069-634e1142-d13c-4863-9930-1120057ca77e.gif" height="350px" alt><br>

### 使用 Inferencer 进行 2D 人体全身姿态识别

```

可视化结果如下：

```shell
python demo/inferencer_demo.py tests/data/crowdpose \
    --pose2d wholebody --vis-out-dir vis_results/crowdpose
```

该命令会对输入的 `tests/data/crowdpose` 下所有图片进行推理并且把可视化结果存入 `vis_results/crowdpose` 文件夹下。

<img src="https://user-images.githubusercontent.com/26127467/229832887-31edb6d5-bcf0-44a4-a66f-9d523061a6e9.jpg" alt="Image 1" height="200"/> <img src="https://user-images.githubusercontent.com/26127467/229832908-bc82dbc9-5e43-4800-acc7-a7da85a653c7.jpg" alt="Image 2" height="200"/>

Inferencer 支持保存姿态的检测结果，具体的使用可参考 [Inferencer 文档](https://mmpose.readthedocs.io/zh_CN/dev-1.x/user_guides/#inferencer-a-unified-inference-interface) 。

### 加速推理

### 加速推理

1. 设置 `model.test_cfg.flip_test=False`，用户可参考 [pose_hrnet_w48_dark+](/configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_hrnet-w48_dark-8xb32-210e_coco-wholebody-384x288.py#L90) 。
2. 使用更快的人体 bounding box 检测器，如 [MMDetection](https://mmdetection.readthedocs.io/zh_CN/3.x/model_zoo.html) 。

**We provide lots of useful tools under the `tools/` directory.**

## MOT Test-time Parameter Search

`tools/analysis_tools/mot/mot_param_search.py` can search the parameters of the `tracker` in MOT models.
It is used as the same manner with `tools/test.py` but **different** in the configs.

Here is an example that shows how to modify the configs:

1. Define the desirable evaluation metrics to record.

   For example, you can define the `evaluator` as

   ```python
   test_evaluator=dict(type='MOTChallengeMetrics', metric=['HOTA', 'CLEAR', 'Identity'])
   ```

   Of course, you can also customize the content of `metric` in `test_evaluator`. You are free to choose one or more of `['HOTA', 'CLEAR', 'Identity']`.

2. Define the parameters and the values to search.

   Assume you have a tracker like

   ```python
   model=dict(
       tracker=dict(
           type='BaseTracker',
           obj_score_thr=0.5,
           match_iou_thr=0.5
       )
   )
   ```

Assume you have a tracker like

   ```python
       )
   )
   ```

   If you want to search the parameters of the tracker, just change the value to a list as follow

   ```python
           obj_score_thr=[0.4, 0.5, 0.6],
           match_iou_thr=[0.4, 0.5, 0.6, 0.7]
       )
   )
   ```

   Then the script will test the totally 12 cases and log the results.

## MOT Error Visualize

`tools/analysis_tools/mot/mot_error_visualize.py` can visualize errors for multiple object tracking.
This script needs the result of inference. By Default, the **red** bounding box denotes false positive, the **yellow** bounding box denotes the false negative and the **blue** bounding box denotes ID switch.

```
python tools/analysis_tools/mot/mot_error_visualize.py \
    ${CONFIG_FILE}\
    --input ${INPUT} \

```
    --result-dir ${RESULT_DIR} \
    [--output-dir ${OUTPUT}] \
    [--fps ${FPS}] \
    [--show] \
    [--backend ${BACKEND}]
```

The `RESULT_DIR` contains the inference results of all videos and the inference result is a `txt` file.

Optional arguments:

- `OUTPUT`: Output of the visualized demo. If not specified, the `--show` is obligate to show the video on the fly.
- `FPS`: FPS of the output video.
- `--show`: Whether show the video on the fly.
- `BACKEND`: The backend to visualize the boxes. Options are `cv2` and `plt`.

`tools/analysis_tools/mot/browse_dataset.py` can visualize the training dataset to check whether the dataset configuration is correct.

**Examples:**

```shell
python tools/analysis_tools/browse_dataset.py ${CONFIG_FILE} [--show-interval ${SHOW_INTERVAL}]
```

- `SHOW_INTERVAL`: The interval of show (s).

```shell
```

- `--show`: Whether show the images on the fly.

# Rotated RepPoints

> [RepPoints: Point Set Representation for Object Detection.](https://arxiv.org/abs/1904.11490)

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/reppoints.png" width="800"/>
</div>

Modern object detectors rely heavily on rectangular bounding boxes, such as anchors, proposals and the final predictions, to represent objects at various recognition stages. The bounding box is convenient to use but provides only a coarse localization of objects and leads to a correspondingly coarse extraction of object features. In this paper, we present \\textbf{RepPoints} (representative points), a new finer representation of objects as a set of sample points useful for both localization and recognition. Given ground truth localization and recognition targets for training, RepPoints learn to automatically arrange themselves in a manner that bounds the spatial extent of an object and indicates semantically significant local areas. They furthermore do not require the use of anchors to sample a space of bounding boxes. We show that an anchor-free object detector based on RepPoints can be as effective as the state-of-the-art anchor-based detection methods, with 46.5 AP and 67.4 AP50 on the COCO test-dev detection benchmark, using ResNet-101 model.

DOTA1.0

|         Backbone         |  mAP  | Angle | lr schd | Mem (GB) | Inf Time (fps) | Aug | Batch Size |                                                 Configs                                                  |                                                                                                                                                                    Download                                                                                                                                                                    |

| ResNet50 (1024,1024,200) | 59.44 |  oc   |   1x    |   3.45   |      15.6      |  -  |     2      | [rotated-reppoints-qbox_r50_fpn_1x_dota](../rotated_reppoints/rotated-reppoints-qbox_r50_fpn_1x_dota.py) | [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_reppoints/rotated_reppoints_r50_fpn_1x_dota_oc/rotated_reppoints_r50_fpn_1x_dota_oc-d38ce217.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_reppoints/rotated_reppoints_r50_fpn_1x_dota_oc/rotated_reppoints_r50_fpn_1x_dota_oc_20220205_145010.log.json) |

```
@inproceedings{yang2019reppoints,
  title={RepPoints: Point Set Representation for Object Detection},
  author={Yang, Ze and Liu, Shaohui and Hu, Han and Wang, Liwei and Lin, Stephen},
  booktitle={The IEEE International Conference on Computer Vision (ICCV)},
  month={Oct},
}
```

# Bag of tricks for image classification with convolutional neural networks

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_CVPR_2019/html/He_Bag_of_Tricks_for_Image_Classification_with_Convolutional_Neural_Networks_CVPR_2019_paper.html">ResNetV1D (CVPR'2019)</a></summary>

```bibtex
@inproceedings{he2019bag,
  title={Bag of tricks for image classification with convolutional neural networks},
  author={He, Tong and Zhang, Zhi and Zhang, Hang and Zhang, Zhongyue and Xie, Junyuan and Li, Mu},
  pages={558--567},
}
```

pages={558--567},
}
```

Much of the recent progress made in image classification research can be credited to training procedure refinements, such as changes in data augmentations and optimization methods. In the literature, however, most refinements are either briefly mentioned as implementation details or only visible in source code. In this paper, we will examine a collection of such refinements and empirically evaluate their impact on the final model accuracy through ablation study. We will show that, by combining these refinements together, we are able to improve various CNN models significantly. For example, we raise ResNet-50’s top-1 validation accuracy from 75.3% to 79.29% on ImageNet. We will also demonstrate that improvement on image classification accuracy leads to better transfer learning performance in other application domains such as object detection and semantic
segmentation.

<img src="https://user-images.githubusercontent.com/15977946/146539869-6deeea09-8f28-4b19-8737-2d9b0d080189.png">
</div>

**Text Detection/Spotting**

```json
{
  "cats": {},
  "anns": {
      "45346": {
          "mask":[468.9,286.7,468.9,295.2,493.0,295.8,493.0,287.2],
          "class":"machine printed",
          "bbox":[468.9,286.7,24.1,9.1],
          "image_id":522579,
          "id":167312,
          "language":"english",
          "area":55.5,
          "utf8_string":"the",
          "legibility":"legible"
      },
      // ...
  },
  "imgs": {
      "522579": {
          "file_name":"COCO_train2014_000000522579.jpg",
          "height":476,
          "width":640,
          "id":522579,
          "set":"train",
      },
      // ...
  },
  "imgToAnns": {
      "522579": [167294, 167295, 167296, 167297, 167298, 167299, 167300, 167301, 167302, 167303, 167304, 167305, 167306, 167307, 167308, 167309, 167310, 167311, 167312, 167313, 167314, 167315, 167316, 167317],
      // ...
  },
  "info": {}
}
```

# CrowdDet

> [Detection in Crowded Scenes: One Proposal, Multiple Predictions](https://arxiv.org/abs/2003.09163)

We propose a simple yet effective proposal-based object detector, aiming at detecting highly-overlapped instances in crowded scenes. The key of our approach is to let each proposal predict a set of correlated instances rather than a single one in previous proposal-based frameworks. Equipped with new techniques such as EMD Loss and Set NMS, our detector can effectively handle the difficulty of detecting highly overlapped objects. On a FPN-Res50 baseline, our detector can obtain 4.9% AP gains on challenging CrowdHuman dataset and 1.0% MR^−2 improvements on CityPersons dataset, without bells and whistles. Moreover, on less crowed datasets like COCO, our approach can still achieve moderate improvement, suggesting the proposed method is robust to crowdedness. Code and pre-trained models will be released at https://github.com/megvii-model/CrowdDetection.

<img src="https://github.com/Purkialo/images/blob/master/CrowdDet_arch.jpg"/>
</div>

| Backbone |  RM   |  Style  | Mem (GB) | Inf time (fps) | box AP |                             Config                              |                                                                                                                                                                                            Download                                                                                                                                                                                            |

| R-50-FPN | False | pytorch |   4.4    |       -        |  90.0  |    [config](./crowddet-rcnn_r50_fpn_8xb2-30e_crowdhuman.py)     |               [model](https://download.openmmlab.com/mmdetection/v3.0/crowddet/crowddet-rcnn_r50_fpn_8xb2-30e_crowdhuman/crowddet-rcnn_r50_fpn_8xb2-30e_crowdhuman_20221023_174954-dc319c2d.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/crowddet/crowddet-rcnn_r50_fpn_8xb2-30e_crowdhuman/crowddet-rcnn_r50_fpn_8xb2-30e_crowdhuman_20221023_174954.log.json)               |

| R-50-FPN | True  | pytorch |   4.8    |       -        | 90.32  | [config](./crowddet-rcnn_refine_r50_fpn_8xb2-30e_crowdhuman.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/crowddet/crowddet-rcnn_refine_r50_fpn_8xb2-30e_crowdhuman/crowddet-rcnn_refine_r50_fpn_8xb2-30e_crowdhuman_20221024_215917-45602806.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/crowddet/crowddet-rcnn_refine_r50_fpn_8xb2-30e_crowdhuman/crowddet-rcnn_refine_r50_fpn_8xb2-30e_crowdhuman_20221024_215917.log.json) |

Note:

- RM indicates whether to use the refine module.
- The dataset for training and testing this model is `CrowdHuman`, and the metric of `box AP` is calculated by `mmdet/evaluation/metrics/crowdhuman_metric.py`.

```latex
@inproceedings{Chu_2020_CVPR,
  title={Detection in Crowded Scenes: One Proposal, Multiple Predictions},
  author={Chu, Xuangeng and Zheng, Anlin and Zhang, Xiangyu and Sun, Jian},

```latex
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
}
```

# Data Transform Migration

In MMOCR version 0.x, we implemented a series of **Data Transform** methods in `mmocr/datasets/pipelines/xxx_transforms.py`. However, these modules are scattered all over the place and lack a standardized design. Therefore, we refactored all the data transform modules in MMOCR version 1.x. According to the task type, they are now defined in `ocr_transforms.py`, `textdet_transforms.py`, and `textrecog_transforms.py`, respectively, under `mmocr/datasets/transforms`. Specifically, `ocr_transforms.py` implements the data augmentation methods for OCR-related tasks in general, while `textdet_transforms.py` and `textrecog_transforms.py` implement data augmentation transforms related to text detection and text recognition tasks, respectively.

Since some of the modules were renamed, merged or separated during the refactoring process, the new interface and default parameters may be inconsistent with the old version. Therefore, this migration guide will introduce how to configure the new data transforms to achieve the identical behavior as the old version.

## Configuration Migration Guide

### Data Formatting Related Data Transforms

1. `Collect` + `CustomFormatBundle` -> [`PackTextDetInputs`](mmocr.datasets.transforms.formatting.PackTextDetInputs)/[`PackTextRecogInputs`](mmocr.datasets.transforms.formatting.PackTextRecogInputs)

`PackxxxInputs` implements both `Collect` and `CustomFormatBundle` functions, and no longer has `key` parameters, the generation of training targets is moved to be done in `loss` modules.

<thead>
  <tr>
    <th>MMOCR 0.x Configuration</th>
    <th>MMOCR 1.x Configuration</th>
  </tr>
  <tbody><tr>
  <td valign="top">

```python
dict(
    type='CustomFormatBundle',

<thead>
  <tr>
  </tr>

```python
dict(
    keys=['gt_shrink', 'gt_shrink_mask', 'gt_thr', 'gt_thr_mask'],
    meta_keys=['img_path', 'ori_shape', 'img_shape'],
    visualize=dict(flag=False, boundary_key='gt_shrink')),
dict(
    type='Collect',
    keys=['img', 'gt_shrink', 'gt_shrink_mask', 'gt_thr', 'gt_thr_mask'])
```

</td><td>

```python
dict(
  type='PackTextDetInputs',
  meta_keys=('img_path', 'ori_shape', 'img_shape'))
```

</td></tr>
</thead>
</table>

### Data Augmentation Related Data Transforms

1. `ResizeOCR` -> [`Resize`](mmocr.datasets.transforms.Resize), [`RescaleToHeight`](mmocr.datasets.transforms.RescaleToHeight), [`PadToWidth`](mmocr.datasets.transforms.PadToWidth)

   The original `ResizeOCR` is now split into three data augmentation modules.

The original `ResizeOCR` is now split into three data augmentation modules.

   When `keep_aspect_ratio=False`, it is equivalent to `Resize` in version 1.x. Its configuration can be modified as follows.

<thead>
  <tr>
  </tr>

```python
dict(
  type='ResizeOCR',
  height=32,
  min_width=100,
  max_width=100,
  keep_aspect_ratio=False)
```

</td><td>

```python
dict(
  type='Resize',
  scale=(100, 32),
  keep_ratio=False)
```

</thead>
</table>

When `keep_aspect_ratio=True` and `max_width=None`. The image will be rescaled to a fixed size alongside the height while keeping the aspect ratio the same as the origin.

<thead>
  <tr>
  </tr>

```python
dict(
  min_width=32,

<thead>
  <tr>
  </tr>

```python
dict(
  max_width=None,
  width_downsample_ratio = 1.0 / 16
  keep_aspect_ratio=True)
```

</td><td>

```python
dict(
  type='RescaleToHeight',
  width_divisor=16),
```

</thead>
</table>

When `keep_aspect_ratio=True` and `max_width` is a fixed value. The image will be rescaled to a fixed size alongside the height while keeping the aspect ratio the same as the origin. Then, the width will be padded or cropped to `max_width`. That is to say, the shape of the output image is always `(height, max_width)`.

<thead>
  <tr>
  </tr>

```python
dict(

<th>MMOCR 0.x Configuration</th>
  </tr>

```python
dict(
  width_downsample_ratio = 1.0 / 16,
```

</td><td>

```python
dict(
dict(
  type='PadToWidth',
  width=100)
```

</thead>
</table>

2. `RandomRotateTextDet` &  `RandomRotatePolyInstances` -> [`RandomRotate`](mmocr.datasets.transforms.RandomRotate)

   We implemented all random rotation-related data augmentation in `RandomRotate` in version 1.x. Its default behavior is identical to the `RandomRotateTextDet` in version 0.x.

```{note}
  The default value of "max_angle" might be different from the old version, so the users are suggested to manually set the number.
```

<thead>
  <tr>

```

<thead>
  <tr>
  </tr>

```python
dict(type='RandomRotateTextDet')
```

</td><td>

```python
dict(type='RandomRotate', max_angle=10)
```

</thead>
</table>

For `RandomRotatePolyInstances`，it is supposed to set `use_canvas=True`。

<thead>
  <tr>
  </tr>

```python
dict(
  type='RandomRotatePolyInstances',
  rotate_ratio=0.5, # Specify the execution probability
  max_angle=60,
  pad_with_fixed_color=False)
```

</td><td>

```python
# Wrap the data transforms with RandomApply and specify the execution probability
dict(
  type='RandomApply',
  transforms=[
    dict(type='RandomRotate',
    max_angle=60,
    pad_with_fixed_color=False,
    use_canvas=True)],
  prob=0.5) # Specify the execution probability
```

dict(
```

</thead>
</table>

```{note}
In version 0.x, some data augmentation methods specified execution probability by defining an internal variable "xxx_ratio", such as "rotate_ratio", "crop_ratio", etc. In version 1.x, these parameters have been removed. Now we can use "RandomApply" to wrap different data transforms and specify their execution probabilities.
```

3. `RandomCropFlip` -> [`TextDetRandomCropFlip`](mmocr.datasets.transforms.TextDetRandomCropFlip)

   Currently, only the method name has been changed, and other parameters remain the same.

4. `RandomCropPolyInstances` -> [`RandomCrop`](mmocr.datasets.transforms.RandomCrop)

   In MMOCR version 1.x, `crop_ratio` and `instance_key` are removed. The `gt_polygons` is now used as the target for cropping.

In MMOCR version 1.x, `crop_ratio` and `instance_key` are removed. The `gt_polygons` is now used as the target for cropping.

<thead>
  <tr>
  </tr>

```python
dict(
  type='RandomCropPolyInstances',
  instance_key='gt_masks',
  crop_ratio=0.8, # Specify the execution probability
  min_side_ratio=0.3)
```

</td><td>

```python
dict(
  transforms=[dict(type='RandomCrop', min_side_ratio=0.3)],
  prob=0.8) # Specify the execution probability
```

</thead>
</table>

5. `RandomCropInstances` -> [`TextDetRandomCrop`](mmocr.datasets.transforms.TextDetRandomCrop)

<thead>
  <tr>

<thead>
  <tr>
  </tr>

```python
dict(
  type='RandomCropInstances',
  target_size=(800，800),
  instance_key='gt_kernels')
```

</td><td>

```python
dict(
  type='TextDetRandomCrop',
  target_size=(800，800))
```

</thead>
</table>

6. `EastRandomCrop` -> [`RandomCrop`](mmocr.datasets.transforms.RandomCrop) + [`Resize`](mmocr.datasets.transforms.Resize) + [`mmengine.Pad`](mmcv.transforms.Pad)

   `EastRandomCrop` was implemented by applying cropping, scaling and padding to the input image. Now, the same effect can be achieved by combining three data transforms.

<thead>
  <tr>
  </tr>

<thead>
  <tr>
  </tr>

```python
dict(
  type='EastRandomCrop',
  max_tries=10,
  min_crop_side_ratio=0.1,
  target_size=(640, 640))
```

</td><td>

```python
dict(type='RandomCrop', min_side_ratio=0.1),
dict(type='Resize', scale=(640,640), keep_ratio=True),
dict(type='Pad', size=(640,640))
```

</thead>
</table>

7. `RandomScaling` -> [`mmengine.RandomResize`](mmcv.transforms.RandomResize)

   The `RandomScaling` is now replaced with [`mmengine.RandomResize`](mmcv.transforms.RandomResize).

<thead>
  <tr>
  </tr>

```python
 dict(
  type='RandomScaling',
  size=800,
  scale=(0.75, 2.5))
```

</td><td>

```python
dict(
  type='RandomResize',
  scale=(800, 800),
  ratio_range=(0.75, 2.5),
  keep_ratio=True)
```

```python
 dict(
  size=800,
```

</td><td>

```python
dict(
```

</thead>
</table>

```{note}

dict(
```

</thead>
</table>

```{note}
By default, the data pipeline will search for the corresponding data transforms from the register of the current *scope*, and if that data transform does not exist, it will continue to search in the upstream library, such as MMCV and MMEngine. For example, the `RandomResize` transform is not implemented in MMOCR, but it can be directly called in the configuration, as the program will automatically search for it from MMCV. In addition, you can also specify *scope* by adding a prefix. For example, `mmengine.RandomResize` will force it to use `RandomResize` implemented in MMEngine, which is useful when a method of the same name exists in both upstream and downstream libraries. It is noteworthy that all of the data transforms implemented in MMCV are registered to MMEngine, that is why we use `mmengine.RandomResize` but not `mmcv.RandomResize`.
```

```

8. `SquareResizePad` -> [`Resize`](mmocr.datasets.transforms.Resize) + [`SourceImagePad`](mmocr.datasets.transforms.SourceImagePad)

   `SquareResizePad` implements two branches and uses one of them randomly based on the `pad_ratio`. Specifically, one branch first resizes the image and then pads it to a certain size; while the other branch only resizes the image. To enhance the reusability of the different modules, we split this data transform into a combination of `Resize` + `SourceImagePad` in version 1.x, and control the branches via `RandomChoice`.

<thead>
  <tr>
  </tr>

```python
dict(
  type='SquareResizePad',
  target_size=800,
  pad_ratio=0.6)
```

</td><td>

```python
dict(
  type='RandomChoice',
    [
      dict(
        type='Resize',
        scale=800,
      dict(
        type='SourceImagePad',

```

</td><td>

```python
dict(
    [
      dict(
      dict(
        target_scale=800)
    ],
    [
      dict(
        keep_ratio=False)
    ]
  ],
  prob=[0.4, 0.6]), # Probability of selection of two combinations
```

</thead>
</table>

```{note}
In version 1.x, the random choice wrapper "RandomChoice" replaces "OneOfWrapper", allowing random selection of data transform combinations.
```

9. `RandomWrapper` -> [`mmengine.RandomApply`](mmcv.transforms.RandomApply)

   In version 1.x, the `RandomWrapper` wrapper has been replaced with `RandomApply` in MMEngine, which is used to specify the probability of performing a data transform. And the probability `p` is now named `prob`.

<thead>
  <tr>
  </tr>

<thead>
  <tr>
  </tr>

```python
 dict(
  type='RandomWrapper',
  p=0.25,
      dict(type='PyramidRescale'),
  ])
```

</td><td>

```python
dict(
  prob=0.25,
    dict(type='PyramidRescale'),
  ])
```

</thead>
</table>

10. `OneOfWrapper` -> [`mmengine.RandomChoice`](mmcv.transforms.RandomChoice)

    The random choice wrapper is now renamed to `RandomChoice` and is used in exactly the same way as before.

11. `ScaleAspectJitter` -> [`ShortScaleAspectJitter`](mmocr.datasets.transforms.ShortScaleAspectJitter), [`BoundedScaleAspectJitter`](mmocr.datasets.transforms.BoundedScaleAspectJitter)

    The `ScaleAspectJitter` implemented several different image size jittering strategies, which has now been split into several independent data transforms.

The `ScaleAspectJitter` implemented several different image size jittering strategies, which has now been split into several independent data transforms.

    When `resize_type='indep_sample_in_range'`, it is equivalent to `RandomResize`.

<thead>
  <tr>
  </tr>

```python
dict(
  type='ScaleAspectJitter',
  img_scale=None,
  keep_ratio=False,
  resize_type='indep_sample_in_range',
  scale_range=(640, 2560))
```

</td><td>

```python
 dict(
  scale=(640, 640),
  ratio_range=(1.0, 4.125),
  resize_type='Resize',
  keep_ratio=True))
```

</thead>
</table>

When `resize_type='long_short_bound'`, we implemented `BoundedScaleAspectJitter`, which randomly rescales the image so that the long and short sides of the image are around the bound; then jitters the aspect ratio.

<thead>
  <tr>

<thead>
  <tr>
  </tr>

```python
dict(
  img_scale=[(3000, 736)],  # Unused
  ratio_range=(0.7, 1.3),
  aspect_ratio_range=(0.9, 1.1),
  multiscale_mode='value',
  long_size_bound=800,
  short_size_bound=480,
  resize_type='long_short_bound',
```

</td><td>

```python
dict(
  type='BoundedScaleAspectJitter',
  aspect_ratio_range=(0.9, 1.1))
```

</thead>
</table>

When `resize_type='round_min_img_scale'`, we implemented `ShortScaleAspectJitter`, which rescales the image for its shorter side to reach the `short_size` and then jitters its aspect ratio, finally rescales the shape guaranteed to be divided by scale_divisor.

<thead>
  <tr>

<thead>
  <tr>
  </tr>

```python
dict(
  img_scale=[(3000, 640)],
```

</td><td>

```python
dict(
  type='ShortScaleAspectJitter',
  short_size=640,
  scale_divisor=32),
```

</thead>
</table>

# FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection

> [FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection](https://arxiv.org/abs/2104.10956)

Monocular 3D object detection is an important task for autonomous driving considering its advantage of low cost. It is much more challenging than conventional 2D cases due to its inherent ill-posed property, which is mainly reflected in the lack of depth information. Recent progress on 2D detection offers opportunities to better solving this problem. However, it is non-trivial to make a general adapted 2D detector work in this 3D task. In this paper, we study this problem with a practice built on a fully convolutional single-stage detector and propose a general framework FCOS3D. Specifically, we first transform the commonly defined 7-DoF 3D targets to the image domain and decouple them as 2D and 3D attributes. Then the objects are distributed to different feature levels with consideration of their 2D scales and assigned only according to the projected 3D-center for the training procedure. Furthermore, the center-ness is redefined with a 2D Gaussian distribution based on the 3D-center to fit the 3D target formulation. All of these make this framework simple yet effective, getting rid of any 2D detection or 2D-3D correspondence priors. Our solution achieves 1st place out of all the vision-only methods in the nuScenes 3D detection challenge of NeurIPS 2020.

<img src="https://user-images.githubusercontent.com/30491025/143856739-93b7c4ff-e116-4824-8cc3-8cf1a433a84c.png" width="800"/>
</div>

FCOS3D is a general anchor-free, one-stage monocular 3D object detector adapted from the original 2D version FCOS.
It serves as a baseline built on top of mmdetection and mmdetection3d for 3D detection based on monocular vision.

Currently we first support the benchmark on the large-scale nuScenes dataset, which achieved 1st place out of all the vision-only methods in the [nuScenes 3D detecton challenge](https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Camera) of NeurIPS 2020.

![demo image](../../resources/browse_dataset_mono.png)

## Usage

After supporting FCOS3D and monocular 3D object detection in v0.13.0, the coco-style 2D json info files will include related annotations by default

## Usage

(see [here](https://github.com/open-mmlab/mmdetection3d/blob/master/tools/dataset_converters/nuscenes_converter.py#L333) if you would like to change the parameter).
So you can just follow the data preparation steps given in the documentation, then all the needed infos are ready together.

### Training and Inference

The way to training and inference a monocular 3D object detector is the same as others in mmdetection and mmdetection3d. You can basically follow the [documentation](https://mmdetection3d.readthedocs.io/en/latest/1_exist_data_model.html#train-predefined-models-on-standard-datasets) and change the `config`, `work_dirs`, etc. accordingly.

We implement test time augmentation for the dense outputs of detection heads, which is more effective than merging predicted boxes at last.
You can turn on it by setting `flip=True` in the `test_pipeline`.

### Training with finetune

Due to the scale and measurements of depth is different from those of other regression targets, we first train the model with depth weight equal to 0.2 for a more stable training procedure. For a stronger detector with better performance, please finetune the model with depth weight changed to 1.0 as shown in the [config](./fcos3d_r101-caffe-dcn_fpn_head-gn_8xb2-1x_nus-mono3d_finetune.py). Note that the path of `load_from` needs to be changed to yours accordingly.

### Visualizing prediction results

We also provide visualization functions to show the monocular 3D detection results. Simply follow the [documentation](https://mmdetection3d.readthedocs.io/en/latest/1_exist_data_model.html#test-existing-models-on-standard-datasets) and use the `single-gpu testing` command. You only need to add the `--show` flag and specify `--show-dir` to store the visualization results.

|                                        Backbone                                         | Lr schd | Mem (GB) | Inf time (fps) | mAP  | NDS  |                                                                                                                                                                                                                             Download                                                                                                                                                                                                                             |

|      [ResNet101 w/ DCN](./fcos3d_r101-caffe-dcn_fpn_head-gn_8xb2-1x_nus-mono3d.py)      |   1x    |   8.69   |                | 29.8 | 37.7 |                   [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_20210715_235813-4bed5239.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_20210715_235813.log.json)                   |

| [above w/ finetune](./fcos3d_r101-caffe-dcn_fpn_head-gn_8xb2-1x_nus-mono3d_finetune.py) |   1x    |   8.69   |                | 32.1 | 39.5 | [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_finetune/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_finetune_20210717_095645-8d806dc2.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_finetune/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_finetune_20210717_095645.log.json) |

|                                      above w/ tta                                       |   1x    |   8.69   |                | 33.1 | 40.3 |                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |

```latex
}

booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops},
}
}
```

## batchnorm
```
Input       A            0 1 A 0 0 0
MemoryData  sub/y        0 1 sub/y 16 0 0
BinaryOp    sub          2 1 A sub/y sub 1
MemoryData  div/y        0 1 div/y 16 0 0
BinaryOp    div          2 1 sub div/y div 3
MemoryData  mul/y        0 1 mul/y 16 0 0
BinaryOp    mul          2 1 div mul/y mul 2
MemoryData  BiasAdd/bias 0 1 BiasAdd/bias 16 0 0
BinaryOp    BiasAdd      2 1 mul BiasAdd/bias BiasAdd 0
```
## convolution
```
Convolution Conv2D       1 1 A Conv2D 10 3 1 1 0 0 270
MemoryData  biases/read  0 1 biases/read 10 0 0
BinaryOp    BiasAdd      2 1 Conv2D biases/read BiasAdd 0
```
## innerproduct
```
Input        A           0 1 A 0 0 0
MemoryData   biases/read 0 1 biases/read 10 0 0
InnerProduct MatMul      1 1 A MatMul 10 0 2560
BinaryOp     conv6       2 1 MatMul biases/read conv6 0
```
## leakyrelu
```
Split       splitncnn_0  1 2 A A_splitncnn_0 A_splitncnn_1

```
```
MemoryData  mul_1/x      0 1 mul_1/x 0 0 0
BinaryOp    mul_1        2 1 mul_1/x A_splitncnn_1 mul_1 2
BinaryOp    leaky        2 1 mul_1 A_splitncnn_0 leaky 4
```
## prelu
```
MemoryData  prelu/alpha  0 1 prelu/alpha 10 0 0
ReLU        prelu/Relu   1 1 A_splitncnn_1 prelu/Relu 0.000000
UnaryOp     prelu/Neg    1 1 A_splitncnn_0 prelu/Neg 1
ReLU        prelu/Relu_1 1 1 prelu/Neg prelu/Relu_1 0.000000
UnaryOp     prelu/Neg_1  1 1 prelu/Relu_1 prelu/Neg_1 1
BinaryOp    prelu/Mul    2 1 prelu/alpha prelu/Neg_1 prelu/Mul 2
BinaryOp    prelu/add    2 1 prelu/Relu prelu/Mul prelu/add 0
```
## softmax
```
Split       splitncnn_4  1 2 A A_splitncnn_0 A_splitncnn_1

```
## softmax
```
Reduction   Max          1 1 A_splitncnn_1 Max 4 -2 1.000000
BinaryOp    sub          2 1 A_splitncnn_0 Max sub 1
UnaryOp     Exp          1 1 sub Exp 7
Split       splitncnn_5  1 2 Exp Exp_splitncnn_0 Exp_splitncnn_1
Reduction   Sum          1 1 Exp_splitncnn_1 Sum 0 -2 1.000000
BinaryOp    prob         2 1 Exp_splitncnn_0 Sum prob 3
```

<details>

```bibtex
}
```

<details>
<summary align="right"><a href="https://elifesciences.org/articles/47994">Desert Locust (Elife'2019)</a></summary>

```bibtex

}
```

| [pose_resnet_50](/configs/animal_2d_keypoint/topdown_heatmap/locust/td-hm_res50_8xb64-210e_locust-160x160.py) |  160x160   |  1.000  | 0.900 | 2.27 | [ckpt](https://download.openmmlab.com/mmpose/animal/resnet/res50_locust_160x160-9efca22b_20210407.pth) | [log](https://download.openmmlab.com/mmpose/animal/resnet/res50_locust_160x160_20210407.log.json) |
| [pose_resnet_101](/configs/animal_2d_keypoint/topdown_heatmap/locust/td-hm_res101_8xb64-210e_locust-160x160.py) |  160x160   |  1.000  | 0.907 | 2.03 | [ckpt](https://download.openmmlab.com/mmpose/animal/resnet/res101_locust_160x160-d77986b3_20210407.pth) | [log](https://download.openmmlab.com/mmpose/animal/resnet/res101_locust_160x160_20210407.log.json) |

| [pose_resnet_152](/configs/animal_2d_keypoint/topdown_heatmap/locust/td-hm_res152_8xb32-210e_locust-160x160.py) |  160x160   |  1.000  | 0.925 | 1.49 | [ckpt](https://download.openmmlab.com/mmpose/animal/resnet/res152_locust_160x160-4ea9b372_20210407.pth) | [log](https://download.openmmlab.com/mmpose/animal/resnet/res152_locust_160x160_20210407.log.json) |

# 常用工具

## 可视化工具

### 数据集可视化工具

MMOCR 提供了数据集可视化工具 `tools/visualizations/browse_datasets.py` 以辅助用户排查可能遇到的数据集相关的问题。用户只需要指定所使用的训练配置文件（通常存放在如 `configs/textdet/dbnet/xxx.py` 文件中）或数据集配置（通常存放在 `configs/textdet/_base_/datasets/xxx.py` 文件中）路径。该工具将依据输入的配置文件类型自动将经过数据流水线（data pipeline）处理过的图像及其对应的标签，或原始图片及其对应的标签绘制出来。

#### 支持参数

```bash
python tools/visualizations/browse_dataset.py \
    [-t, --task ${DATASET_TASK}] \
```

| 参数名              | 类型                                  | 描述                                                                                                                                             |

| ------------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| config              | str                                   | (必须) 配置文件路径。                                                                                                                            |
| -o, --output-dir    | str                                   | 如果图形化界面不可用，请指定一个输出路径来保存可视化结果。                                                                                       |
| -p, --phase         | str                                   | 用于指定需要可视化的数据集切片，如 "train", "test", "val"。当数据集存在多个变种时，也可以通过该参数来指定待可视化的切片。                        |
| -m, --mode          | `original`, `transformed`, `pipeline` | 用于指定数据可视化的模式。`original`：原始模式，仅可视化数据集的原始标注；`transformed`：变换模式，展示经过所有数据变换步骤的最终图像；`pipeline`：流水线模式，展示数据变换过程中每一个中间步骤的变换图像。默认使用 `transformed` 变换模式。 |

| -t, --task          | `auto`, `textdet`, `textrecog`        | 用于指定可视化数据集的任务类型。`auto`：自动模式，将依据给定的配置文件自动选择合适的任务类型，如果无法自动获取任务类型，则需要用户手动指定为 `textdet` 文本检测任务 或 `textrecog` 文本识别任务。默认采用 `auto` 自动模式。 |
| -n, --show-number   | int                                   | 指定需要可视化的样本数量。若该参数缺省则默认将可视化全部图片。                                                                                   |
| -i, --show-interval | float                                 | 可视化图像间隔时间，默认为 2 秒。                                                                                                                |
| --cfg-options       | float                                 | 用于覆盖配置文件中的参数，详见[示例](./config.md#command-line-modification)。                                                                    |

#### 用法示例

以下示例演示了如何使用该工具可视化 "DBNet_R50_icdar2015" 模型使用的训练数据。

```Bash
# 使用默认参数可视化 "dbnet_r50dcn_v2_fpnc_1200e_icadr2015" 模型的训练数据

#### 用法示例

```Bash
python tools/visualizations/browse_dataset.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py
```

默认情况下，可视化模式为 "transformed"，您将看到经由数据流水线变换过后的图像和标注：

<center class="half">
    <img src="https://user-images.githubusercontent.com/24622904/187611542-01e9aa94-fc12-4756-964b-a0e472522a3a.jpg" width="250"/><img src="https://user-images.githubusercontent.com/24622904/187611555-3f5ea616-863d-4538-884f-bccbebc2f7e7.jpg" width="250"/><img src="https://user-images.githubusercontent.com/24622904/187611581-88be3970-fbfe-4f62-8cdf-7a8a7786af29.jpg" width="250"/>
</center>

如果您只想可视化原始数据集，只需将模式设置为 "original"：

```Bash
python tools/visualizations/browse_dataset.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py -m original
```

</center>

```Bash
```

<div align=center><img src="https://user-images.githubusercontent.com/22607038/206646570-382d0f26-908a-4ab4-b1a7-5cc31fa70c5f.jpg" style=" width: auto; height: 40%; "></div>

或者，您也可以使用 "pipeline" 模式来可视化整个数据流水线的中间结果：

```Bash
python tools/visualizations/browse_dataset.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py -m pipeline
```

<div align=center><img src="https://user-images.githubusercontent.com/22607038/206637571-287640c0-1f55-453f-a2fc-9f9734b9593f.jpg" style=" width: auto; height: 40%; "></div>

另外，用户还可以通过指定数据集配置文件的路径来可视化数据集的原始图像及其对应的标注，例如：

```Bash
python tools/visualizations/browse_dataset.py configs/textrecog/_base_/datasets/icdar2015.py
```

```Bash
```

部分数据集可能有多个变体。例如，`icdar2015` 文本识别数据集的[配置文件](/configs/textrecog/_base_/datasets/icdar2015.py)中包含两个测试集变体，分别为 `icdar2015_textrecog_test` 和 `icdar2015_1811_textrecog_test`，如下所示：

```python
    ann_file='textrecog_test.json',
    # ...
    )

icdar2015_1811_textrecog_test = dict(
    ann_file='textrecog_test_1811.json',
    # ...
)
```

在这种情况下，用户可以通过指定 `-p` 参数来可视化不同的变体，例如，使用以下命令可视化 `icdar2015_1811_textrecog_test` 变体：

```Bash
python tools/visualizations/browse_dataset.py configs/textrecog/_base_/datasets/icdar2015.py -p icdar2015_1811_textrecog_test
```

基于该工具，用户可以轻松地查看数据集的原始图像及其对应的标注，以便于检查数据集的标注是否正确。

### 优化器参数策略可视化工具

MMOCR提供了优化器参数可视化工具 `tools/visualizations/vis_scheduler.py` 以辅助用户排查优化器的超参数调度器（无需训练），支持学习率（learning rate）和动量（momentum）。

#### 工具简介

```bash
python tools/visualizations/vis_scheduler.py \

#### 工具简介

```bash
    [-p, --parameter ${PARAMETER_NAME}] \
    [-d, --dataset-size ${DATASET_SIZE}] \
    [-n, --ngpus ${NUM_GPUs}] \
    [-s, --save-path ${SAVE_PATH}] \
    [--title ${TITLE}] \
    [--style ${STYLE}] \
    [--window-size ${WINDOW_SIZE}] \
    [--cfg-options]
```

- **`-p, parameter`**: 可视化参数名，只能为 `["lr", "momentum"]` 之一， 默认为 `"lr"`.
- **`-d, --dataset-size`**: 数据集的大小。如果指定，`build_dataset` 将被跳过并使用这个大小作为数据集大小，默认使用 `build_dataset` 所得数据集的大小。
- **`-n, --ngpus`**: 使用 GPU 的数量, 默认为1。
- **`-s, --save-path`**: 保存的可视化图片的路径，默认不保存。
- `--title`: 可视化图片的标题，默认为配置文件名。
- `--style`: 可视化图片的风格，默认为 `whitegrid`。
- `--window-size`: 可视化窗口大小，如果没有指定，默认为 `12*7`。如果需要指定，按照格式 \`W\*H'。
- `--cfg-options`: 对配置文件的修改，参考[学习配置文件](../user_guides/config.md)。

```{note}

```{note}
部分数据集在解析标注阶段比较耗时，可直接将 `-d, dataset-size` 指定数据集的大小，以节约时间。
```

#### 如何在开始训练前可视化学习率曲线

你可以使用如下命令来绘制配置文件 `configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py` 将会使用的变化率曲线：

```bash
python tools/visualizations/vis_scheduler.py configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py -d 100
```

<div align=center><img src="https://user-images.githubusercontent.com/43344034/232755081-cad8fe62-349d-400a-bc38-7f5d17824011.png" style=" width: auto; height: 40%; "></div>

## 分析工具

### 离线评测工具

对于已保存的预测结果，我们提供了离线评测脚本 `tools/analysis_tools/offline_eval.py`。例如，以下代码演示了如何使用该工具对 "PSENet" 模型的输出结果进行离线评估：

```Bash
# 初次运行测试脚本时，用户可以通过指定 --save-preds 参数来保存模型的输出结果
python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} --save-preds
# 示例：对 PSENet 进行测试

```Bash
python tools/test.py configs/textdet/psenet/psenet_r50_fpnf_600e_icdar2015.py epoch_600.pth --save-preds

# 之后即可使用已保存的输出文件进行离线评估
python tools/analysis_tool/offline_eval.py ${CONFIG_FILE} ${PRED_FILE}
# 示例：对已保存的 PSENet 结果进行离线评估
python tools/analysis_tools/offline_eval.py configs/textdet/psenet/psenet_r50_fpnf_600e_icdar2015.py work_dirs/psenet_r50_fpnf_600e_icdar2015/epoch_600.pth_predictions.pkl
```

`--save-preds` 默认将输出结果保存至 `work_dir/CONFIG_NAME/MODEL_NAME_predictions.pkl`

此外，基于此工具，用户也可以将其他算法库获取的预测结果转换成 MMOCR 支持的格式，从而使用 MMOCR 内置的评估指标来对其他算法库的模型进行评测。

| 参数          | 类型  | 说明                                                             |
| ------------- | ----- | ---------------------------------------------------------------- |
| config        | str   | （必须）配置文件路径。                                           |

| pkl_results   | str   | （必须）预先保存的预测结果文件。                                 |
| --cfg-options | float | 用于覆写配置文件中的指定参数。[示例](./config.md#命令行修改配置) |

### 计算 FLOPs 和参数量

我们提供一个计算 FLOPs 和参数量的方法，首先我们使用以下命令安装依赖。

```shell
pip install fvcore
```

计算 FLOPs 和参数量的脚本使用方法如下：

```shell
python tools/analysis_tools/get_flops.py ${config} --shape ${IMAGE_SHAPE}
```

| 参数    | 类型   | 说明                                                               |
| ------- | ------ | ------------------------------------------------------------------ |
| config  | str    | （必须) 配置文件路径。                                             |
| --shape | int\*2 | 计算 FLOPs 使用的图片尺寸，如 `--shape 320 320`。 默认为 `640 640` |

获取 `dbnet_resnet18_fpnc_100k_synthtext.py` FLOPs 和参数量的示例命令如下。

```shell

```shell
python tools/analysis_tools/get_flops.py configs/textdet/dbnet/dbnet_resnet18_fpnc_100k_synthtext.py --shape 1024 1024
```

输出如下：

```shell
input shape is  (1, 3, 1024, 1024)
| module                    | #parameters or shape | #flops  |
| :------------------------ | :------------------- | :------ |
| model                     | 12.341M              | 63.955G |
| backbone                  | 11.177M              | 38.159G |
| backbone.conv1            | 9.408K               | 2.466G  |
| backbone.conv1.weight     | (64, 3, 7, 7)        |         |
| backbone.bn1              | 0.128K               | 83.886M |
| backbone.bn1.weight       | (64,)                |         |
| backbone.bn1.bias         | (64,)                |         |
| backbone.layer1           | 0.148M               | 9.748G  |

| backbone.layer1.0         | 73.984K              | 4.874G  |
| backbone.layer1.1         | 73.984K              | 4.874G  |
| backbone.layer2           | 0.526M               | 8.642G  |
| backbone.layer2.0         | 0.23M                | 3.79G   |
| backbone.layer2.1         | 0.295M               | 4.853G  |
| backbone.layer3           | 2.1M                 | 8.616G  |
| backbone.layer3.0         | 0.919M               | 3.774G  |
| backbone.layer3.1         | 1.181M               | 4.842G  |
| backbone.layer4           | 8.394M               | 8.603G  |
| backbone.layer4.0         | 3.673M               | 3.766G  |
| backbone.layer4.1         | 4.721M               | 4.837G  |
| neck                      | 0.836M               | 14.887G |

| neck.lateral_convs        | 0.246M               | 2.013G  |
| neck.lateral_convs.0.conv | 16.384K              | 1.074G  |
| neck.lateral_convs.1.conv | 32.768K              | 0.537G  |
| neck.lateral_convs.2.conv | 65.536K              | 0.268G  |
| neck.lateral_convs.3.conv | 0.131M               | 0.134G  |
| neck.smooth_convs         | 0.59M                | 12.835G |
| neck.smooth_convs.0.conv  | 0.147M               | 9.664G  |
| neck.smooth_convs.1.conv  | 0.147M               | 2.416G  |
| neck.smooth_convs.2.conv  | 0.147M               | 0.604G  |
| neck.smooth_convs.3.conv  | 0.147M               | 0.151G  |
| det_head                  | 0.329M               | 10.909G |
| det_head.binarize         | 0.164M               | 10.909G |

| det_head.binarize.0       | 0.147M               | 9.664G  |
| det_head.binarize.1       | 0.128K               | 20.972M |
| det_head.binarize.3       | 16.448K              | 1.074G  |
| det_head.binarize.4       | 0.128K               | 83.886M |
| det_head.binarize.6       | 0.257K               | 67.109M |
| det_head.threshold        | 0.164M               |         |
| det_head.threshold.0      | 0.147M               |         |
| det_head.threshold.1      | 0.128K               |         |
| det_head.threshold.3      | 16.448K              |         |
| det_head.threshold.4      | 0.128K               |         |
| det_head.threshold.6      | 0.257K               |         |

!!!Please be cautious if you use the results in papers. You may need to check if all ops are supported and verify that the flops computation is correct.
```

# Vision Transformer

> [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://arxiv.org/abs/2010.11929)

**Vision Transformer**, known as **ViT**, succeeded in using a full transformer to outperform previous works that based on convolutional networks in vision field. ViT splits image into patches to feed the multi-head attentions, concatenates a learnable class token for final prediction and adds a learnable position embeddings for relative positional message between patches. Based on these three techniques with attentions, ViT provides a brand-new pattern to build a basic structure in vision field.

The strategy works even better when coupled with large datasets pre-trainings. Because of its simplicity and effectiveness, some after works in classification field are originated from ViT. And even in recent multi-modality field, ViT-based method still plays a role in it.

<img src="https://user-images.githubusercontent.com/26739999/142579081-b5718032-6581-472b-8037-ea66aaa9e278.png" width="70%"/>
</div>

<details>

<br>

</div>

<details>

<br>

While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.
</br>

</br>

```python

predict = inference_model('vit-base-p32_in21k-pre_3rdparty_in1k-384px', 'demo/bird.JPEG')
```

```python

model = get_model('vit-base-p32_in21k-pre_3rdparty_in1k-384px', pretrained=True)
```

Train:

```shell
python tools/train.py configs/vision_transformer/vit-base-p16_32xb128-mae_in1k.py
```

Test:

```shell

Train:

```shell
```

Test:

```shell
python tools/test.py configs/vision_transformer/vit-base-p32_64xb64_in1k-384px.py https://download.openmmlab.com/mmclassification/v0/vit/finetune/vit-base-p32_in21k-pre-3rdparty_ft-64xb64_in1k-384_20210928-9cea8599.pth
```

| Model                                           |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                    Config                    |                           Download                           |
| :---------------------------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :------------------------------------------: | :----------------------------------------------------------: |

| `vit-base-p32_in21k-pre_3rdparty_in1k-384px`\*  | ImageNet-21k |   88.30    |   13.06   |   84.01   |   97.08   | [config](vit-base-p32_64xb64_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/vit/finetune/vit-base-p32_in21k-pre-3rdparty_ft-64xb64_in1k-384_20210928-9cea8599.pth) |
| `vit-base-p16_32xb128-mae_in1k`                 | From scratch |   86.57    |   17.58   |   82.37   |   96.15   |  [config](vit-base-p16_32xb128-mae_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/vit/vit-base-p16_pt-32xb128-mae_in1k_20220623-4c544545.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/vit/vit-base-p16_pt-32xb128-mae_in1k_20220623-4c544545.log) |

| `vit-base-p16_in21k-pre_3rdparty_in1k-384px`\*  | ImageNet-21k |   86.86    |   55.54   |   85.43   |   97.77   | [config](vit-base-p16_64xb64_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/vit/finetune/vit-base-p16_in21k-pre-3rdparty_ft-64xb64_in1k-384_20210928-98e8652b.pth) |
| `vit-large-p16_in21k-pre_3rdparty_in1k-384px`\* | ImageNet-21k |   304.72   |  191.21   |   85.63   |   97.63   | [config](vit-large-p16_64xb64_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/vit/finetune/vit-large-p16_in21k-pre-3rdparty_ft-64xb64_in1k-384_20210928-b20ba619.pth) |

*Models with * are converted from the [official repo](https://github.com/google-research/vision_transformer/blob/88a52f8892c80c10de99194990a517b4d80485fd/vit_jax/models.py#L208). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
  dosovitskiy2021an,

```bibtex
  title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
  author={Alexey Dosovitskiy and Lucas Beyer and Alexander Kolesnikov and Dirk Weissenborn and Xiaohua Zhai and Thomas Unterthiner and Mostafa Dehghani and Matthias Minderer and Georg Heigold and Sylvain Gelly and Jakob Uszkoreit and Neil Houlsby},
  url={https://openreview.net/forum?id=YicbFdNTTy}
}
```

# 使用GroupFisher剪枝RTMPose

# 概述

我们尝试使用 GroupFisher 算法对 RTMPose 模型进行剪枝。具体来说，我们将一个 RTMPose 模型剪枝到与较小的 RTMPose 模型相同的大小，例如将 RTMPose-S 剪枝到 RTMPose-T 的大小。
实验表明，剪枝后的模型比具有相似大小和推理速度的 RTMPose 模型具有更好的性能（AP）。

我们使用能自动确定剪枝结构的 GroupFisher 剪枝算法，将 RTMPose-S 剪枝到 RTMPose-T 的大小。
此外，我们提供了两个版本的剪枝模型，其中一个只使用 coco 数据集，另一个同时使用 coco 和 ai-challenge 数据集。

# 实验结果

| Arch                                                                  | Input Size |  AP   | AP<sup>50</sup> | AP<sup>75</sup> |  AR   | AR<sup>50</sup> | Flops | Params |                   ckpt                    |      log       |
| :-------------------------------------------------------------------- | :--------: | :---: | :-------------: | :-------------: | :---: | :-------------: | :---: | :----: | :---------------------------------------: | :------------: |

| [rtmpose-s-pruned](./group_fisher_finetune_rtmpose-s_8xb256-420e_coco-256x192.py) |  256x192   | 0.691 |      0.885      |      0.765      | 0.745 |      0.925      | 0.34  |  3.42  | [pruned][rp_sc_p] \| [finetuned][rp_sc_f] | [log][rp_sc_l] |
| [rtmpose-s-aic-coco-pruned](./group_fisher_finetune_rtmpose-s_8xb256-420e_aic-coco-256x192.py) |  256x192   | 0.694 |      0.884      |      0.771      | 0.747 |      0.922      | 0.35  |  3.43  | [pruned][rp_sa_p] \| [finetuned][rp_sa_f] | [log][rp_sa_l] |

我们需要三个步骤来将 GroupFisher 应用于你的模型，包括剪枝（Prune），微调（Finetune），部署（Deploy）。
注意：请使用torch>=1.12，因为我们需要fxtracer来自动解析模型。

### Prune

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 PORT=29500 ./tools/dist_train.sh \
  {config_folder}/group_fisher_{normalization_type}_prune_{model_name}.py 8 \
  --work-dir $WORK_DIR
```

在剪枝配置文件中，你需要填写以下参数。

```python
"""
_base_ (str): The path to your pretrained model checkpoint.
pretrained_path (str): The path to your pretrained model checkpoint.

--work-dir $WORK_DIR
```

```python
"""

interval (int): Interval between pruning two channels. You should ensure you
    can reach your target pruning ratio when the training ends.
normalization_type (str): GroupFisher uses two methods to normlized the channel
    importance, including ['flops','act']. The former uses flops, while the
    latter uses the memory occupation of activation feature maps.
lr_ratio (float): Ratio to decrease lr rate. As pruning progress is unstable,
    you need to decrease the original lr rate until the pruning training work
    steadly without getting nan.

target_flop_ratio (float): The target flop ratio to prune your model.
input_shape (Tuple): input shape to measure the flops.
"""
```

在剪枝结束后，你将获得一个剪枝模型的 checkpoint，该 checkpoint 的名称为 flops\_{target_flop_ratio}.pth，位于你的 workdir 中。

### Finetune

```bash

"""
```

```bash
   {config_folder}/group_fisher_{normalization_type}_finetune_{model_name}.py 8 \
```

微调时也有一些参数需要你填写。

```python
"""
_base_(str): The path to your pruning config file.
pruned_path (str): The path to the checkpoint of the pruned model.
finetune_lr (float): The lr rate to finetune. Usually, we directly use the lr
    rate of the pretrain.
"""
```

在微调结束后，除了最佳模型的 checkpoint 外，还有一个 fix_subnet.json，它记录了剪枝模型的结构。它将在部署时使用。

### Test

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 PORT=29500 ./tools/dist_test.sh \
   {config_folder}/group_fisher_{normalization_type}_finetune_{model_name}.py {checkpoint_path} 8
```

### Deploy

```bash
```

### Deploy

对于剪枝模型，你只需要使用剪枝部署 config 来代替预训练 config 来部署模型的剪枝版本。如果你不熟悉 MMDeploy，请参看[MMDeploy document](https://mmdeploy.readthedocs.io/en/latest/02-how-to-run/convert_model.html)。

```bash
python {mmdeploy}/tools/deploy.py \
    {mmdeploy}/{mmdeploy_config}.py \
    {config_folder}/group_fisher_{normalization_type}_deploy_{model_name}.py \
    {path_to_finetuned_checkpoint}.pth \
    {mmdeploy}/tests/data/tiger.jpeg
```

部署配置文件有如下参数：

```python
"""
_base_ (str): The path to your pretrain config file.
fix_subnet (Union[dict,str]): The dict store the pruning structure or the
    json file including it.
divisor (int): The divisor the make the channel number divisible.
"""
```

divisor 设置十分重要，我们建议你在尝试 \[1,2,4,8,16,32\]，以找到最佳设置。

json file including it.
"""
```

[GroupFisher in MMRazor](https://github.com/open-mmlab/mmrazor/tree/main/configs/pruning/base/group_fisher)

[rp_sa_f]: https://download.openmmlab.com/mmrazor/v1/pruning/group_fisher/rtmpose-s/group_fisher_finetune_rtmpose-s_8xb256-420e_aic-coco-256x192.pth
[rp_sa_l]: https://download.openmmlab.com/mmrazor/v1/pruning/group_fisher/rtmpose-s/group_fisher_finetune_rtmpose-s_8xb256-420e_aic-coco-256x192.json
[rp_sa_p]: https://download.openmmlab.com/mmrazor/v1/pruning/group_fisher/rtmpose-s/group_fisher_prune_rtmpose-s_8xb256-420e_aic-coco-256x192.pth
[rp_sc_f]: https://download.openmmlab.com/mmrazor/v1/pruning/group_fisher/rtmpose-s/group_fisher_finetune_rtmpose-s_8xb256-420e_coco-256x192.pth

# rtmpose_tensorrt

This repository is use the TensorRT to deploy RTMDet and RTMPose. Your computer should have these components:

- NVIDIA GPU
- CUDA
- cudnn
- TensorRT 8.x
- OPENCV
- VS2019

The effect of the code is as follows:

![mabaoguo](https://github.com/Dominic23331/rtmpose_tensorrt/assets/53283758/568563be-a31d-4d03-9629-842dad3745e2)

### I. Convert Model

#### 1. RTMDet

When you start to convert a RTMDet model, you can use **convert_rtmdet.py** to convert pth file to onnx.

```shell
python convert_rtmdet.py --config <model cfg> --checkpoint <checkpoint> --output <output path>
```

Note that RTMDet should be the mmdetection version, and the conversion of mmyolo is not supported.

#### 2. RTMPose

You can use mmdeploy to convert RTMPose. The mmdeploy config file should use **configs/mmpose/pose-detection_simcc_onnxruntime_dynamic.py**.  The convert command as follow:

```shell

```shell
python tools/deploy.py <deploy cfg> <model cfg> <checkpoint> <image path>
```

#### 3. Convert to TensorRT engine file

You can use trtexec to convert an ONNX file to engine file. The command as follow:

```
trtexec --onnx=<ONNX file> --saveEngine=<output file>
```

**Note that the engine files included in the project are only for storing examples. As the engine files generated by TensorRT are related to hardware, it is necessary to regenerate the engine files on the computer where the code needs to be run.**

### II. Run

At first, you should fill in the model locations for RTMDet and RTMPose as follows:

```c++
// set engine file path
string detEngineFile = "./model/rtmdet.engine";
string poseEngineFile = "./model/rtmpose_m.engine";
```

```c++
```

Then, you can set the cap to video file or camera.

```
// open cap
cv::VideoCapture cap(0);
```

If you want to change iou threshold or confidence threshold, you can change them when you initialize RTMDet model.

```
RTMDet det_model(detEngineFile, logger, 0.5, 0.65);
```

Finally, you can run the **main.cpp** file to get result.

# Structure Aware Single-stage 3D Object Detection from Point Cloud

> [Structure Aware Single-stage 3D Object Detection from Point Cloud](<%5Bhttps://arxiv.org/abs/2104.02323%5D(https://openaccess.thecvf.com/content_CVPR_2020/papers/He_Structure_Aware_Single-Stage_3D_Object_Detection_From_Point_Cloud_CVPR_2020_paper.pdf)>)

3D object detection from point cloud data plays an essential role in autonomous driving. Current single-stage detectors are efficient by progressively downscaling the 3D point clouds in a fully convolutional manner. However, the downscaled features inevitably lose spatial information and cannot make full use of the structure information of 3D point cloud, degrading their localization precision. In this work, we propose to improve the localization precision of single-stage detectors by explicitly leveraging the structure information of 3D point cloud. Specifically, we design an auxiliary network which converts the convolutional features in the backbone network back to point-level representations. The auxiliary network is jointly optimized, by two point-level supervisions, to guide the convolutional features in the backbone network to be aware of the object structure. The auxiliary network can be detached after training and therefore introduces no extra computation in the inference stage. Besides, considering that single-stage detectors suffer from the discordance between the predicted bounding boxes and corresponding classification confidences, we develop an efficient part-sensitive warping operation to align the confidences to the predicted bounding boxes. Our proposed detector ranks at the top of KITTI 3D/BEV detection leaderboards and runs at 25 FPS for inference.

<img src="https://user-images.githubusercontent.com/30491025/172526367-c8b9bdf7-f901-4f2f-8855-bfd55c39f8d1.png" width="800"/>
</div>

We implement SA-SSD and provide the results and checkpoints on KITTI dataset.

```latex
@InProceedings{he2020sassd,
    title={Structure Aware Single-stage 3D Object Detection from Point Cloud},
    author={He, Chenhang and Zeng, Hui and Huang, Jianqiang and Hua, Xian-Sheng and Zhang, Lei},
    booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
}
```

# DeepFillv2 (CVPR'2019)

> [Free-Form Image Inpainting with Gated Convolution](https://arxiv.org/abs/1806.03589)

We present a generative image inpainting system to complete images with free-form mask and guidance. The system is based on gated convolutions learned from millions of images without additional labelling efforts. The proposed gated convolution solves the issue of vanilla convolution that treats all input pixels as valid ones, generalizes partial convolution by providing a learnable dynamic feature selection mechanism for each channel at each spatial location across all layers. Moreover, as free-form masks may appear anywhere in images with any shape, global and local GANs designed for a single rectangular mask are not applicable. Thus, we also present a patch-based GAN loss, named SN-PatchGAN, by applying spectral-normalized discriminator on dense image patches. SN-PatchGAN is simple in formulation, fast and stable in training. Results on automatic image inpainting and user-guided extension demonstrate that our system generates higher-quality and more flexible results than previous methods. Our system helps user quickly remove distracting objects, modify image layouts, clear watermarks and edit faces.

 <img src="https://user-images.githubusercontent.com/12726765/144175160-75473789-924f-490b-ab25-4c4f252fa55f.png" width="400"/>
</div >

**Places365-Challenge**

|                       Model                       | Mask Type | Resolution | Train Iters |    Dataset    | l1 error |  PSNR  | SSIM  | Training Resources |                                Download                                 |
| :-----------------------------------------------: | :-------: | :--------: | :---------: | :-----------: | :------: | :----: | :---: | :----------------: | :---------------------------------------------------------------------: |

| [DeepFillv2](./deepfillv2_8xb2_places-256x256.py) | free-form |  256x256   |    100k     | Places365-val |  8.635   | 22.398 | 0.815 |         8          | [model](https://download.openmmlab.com/mmediting/inpainting/deepfillv2/deepfillv2_256x256_8x2_places_20200619-10d15793.pth) \| [log](https://download.openmmlab.com/mmediting/inpainting/deepfillv2/deepfillv2_256x256_8x2_places_20200619-10d15793.log.json) |

**CelebA-HQ**

|                       Model                       | Mask Type | Resolution | Train Iters |  Dataset   | l1 error |  PSNR  | SSIM  | Training Resources |                                  Download                                  |
| :-----------------------------------------------: | :-------: | :--------: | :---------: | :--------: | :------: | :----: | :---: | :----------------: | :------------------------------------------------------------------------: |

| [DeepFillv2](./deepfillv2_8xb2_celeba-256x256.py) | free-form |  256x256   |     20k     | CelebA-val |  5.411   | 25.721 | 0.871 |         8          | [model](https://download.openmmlab.com/mmediting/inpainting/deepfillv2/deepfillv2_256x256_8x2_celeba_20200619-c96e5f12.pth) \| [log](https://download.openmmlab.com/mmediting/inpainting/deepfillv2/deepfillv2_256x256_8x2_celeba_20200619-c96e5f12.log.json) |

**Train**

<details>

```shell
# cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/deepfillv2/deepfillv2_8xb2_places-256x256.py

python tools/train.py configs/deepfillv2/deepfillv2_8xb2_places-256x256.py

./tools/dist_train.sh configs/deepfillv2/deepfillv2_8xb2_places-256x256.py 8
```

```

**Test**

<details>

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/deepfillv2/deepfillv2_8xb2_places-256x256.py https://download.openmmlab.com/mmediting/inpainting/deepfillv2/deepfillv2_256x256_8x2_places_20200619-10d15793.pth

python tools/test.py configs/deepfillv2/deepfillv2_8xb2_places-256x256.py https://download.openmmlab.com/mmediting/inpainting/deepfillv2/deepfillv2_256x256_8x2_places_20200619-10d15793.pth

./tools/dist_test.sh configs/deepfillv2/deepfillv2_8xb2_places-256x256.py https://download.openmmlab.com/mmediting/inpainting/deepfillv2/deepfillv2_256x256_8x2_places_20200619-10d15793.pth 8
```

```bibtex
@inproceedings{yu2019free,
  title={Free-form image inpainting with gated convolution},
  author={Yu, Jiahui and Lin, Zhe and Yang, Jimei and Shen, Xiaohui and Lu, Xin and Huang, Thomas S},
  pages={4471--4480},
}
```

<details>

```bibtex
}
```

<details>

```bibtex

```bibtex
}
```

Results on COCO val2017 without multi-scale test

| [CID](/configs/body_2d_keypoint/cid/coco/cid_hrnet-w32_8xb20-140e_coco-512x512.py) |  512x512   | 0.704 |      0.894      |      0.775      | 0.753 |      0.928      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/cid/coco/cid_hrnet-w32_8xb20-140e_coco-512x512_42b7e6e6-20230207.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/cid/coco/cid_hrnet-w32_8xb20-140e_coco-512x512_20230207.json) |
| [CID](/configs/body_2d_keypoint/cid/coco/cid_hrnet-w48_8xb20-140e_coco-512x512.py) |  512x512   | 0.715 |      0.900      |      0.782      | 0.765 |      0.935      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/cid/coco/cid_hrnet-w48_8xb20-140e_coco-512x512_a36c3ecf-20230207.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/cid/coco/cid_hrnet-w48_8xb20-140e_coco-512x512_20230207.json) |

# SSN: Shape Signature Networks for Multi-class Object Detection from Point Clouds

> [SSN: Shape Signature Networks for Multi-class Object Detection from Point Clouds](https://arxiv.org/abs/2004.02774)

Multi-class 3D object detection aims to localize and classify objects of multiple categories from point clouds. Due to the nature of point clouds, i.e. unstructured, sparse and noisy, some features benefit-ting multi-class discrimination are underexploited, such as shape information. In this paper, we propose a novel 3D shape signature to explore the shape information from point clouds. By incorporating operations of symmetry, convex hull and chebyshev fitting, the proposed shape sig-nature is not only compact and effective but also robust to the noise, which serves as a soft constraint to improve the feature capability of multi-class discrimination. Based on the proposed shape signature, we develop the shape signature networks (SSN) for 3D object detection, which consist of pyramid feature encoding part, shape-aware grouping heads and explicit shape encoding objective. Experiments show that the proposed method performs remarkably better than existing methods on two large-scale datasets. Furthermore, our shape signature can act as a plug-and-play component and ablation study shows its effectiveness and good scalability.

<img src="https://user-images.githubusercontent.com/79644370/144024507-9c1f23c1-5e5a-49c8-b346-ff37e30adc3a.png" width="800"/>
</div>

We implement PointPillars with Shape-aware grouping heads used in the SSN and provide the results and checkpoints on the nuScenes and Lyft dataset.

|                                            Backbone                                             | Lr schd | Mem (GB) | Inf time (fps) |  mAP  |  NDS  |                                                                                                                                                                                                                       Download                                                                                                                                                                                                                       |

|           [SECFPN](../pointpillars/pointpillars_hv_secfpn_sbn-all_8xb4-2x_nus-3d.py)            |   2x    |   16.4   |                | 35.17 | 49.76 |                     [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_secfpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_secfpn_sbn-all_4x8_2x_nus-3d_20200620_230725-0817d270.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_secfpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_secfpn_sbn-all_4x8_2x_nus-3d_20200620_230725.log.json)                     |

|                        [SSN](./ssn_hv_secfpn_sbn-all_16xb2-2x_nus-3d.py)                        |   2x    |   3.6    |                | 40.91 | 54.44 |                                              [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/ssn/hv_ssn_secfpn_sbn-all_2x16_2x_nus-3d/hv_ssn_secfpn_sbn-all_2x16_2x_nus-3d_20210830_101351-51915986.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/ssn/hv_ssn_secfpn_sbn-all_2x16_2x_nus-3d/hv_ssn_secfpn_sbn-all_2x16_2x_nus-3d_20210830_101351.log.json)                                              |

| [RegNetX-400MF-SECFPN](../regnet/pointpillars_hv_regnet-400mf_secfpn_sbn-all_8xb4-2x_nus-3d.py) |   2x    |   16.4   |                | 41.15 | 55.20 | [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/regnet/hv_pointpillars_regnet-400mf_secfpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_regnet-400mf_secfpn_sbn-all_4x8_2x_nus-3d_20200620_230334-53044f32.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/regnet/hv_pointpillars_regnet-400mf_secfpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_regnet-400mf_secfpn_sbn-all_4x8_2x_nus-3d_20200620_230334.log.json) |

|          [RegNetX-400MF-SSN](./ssn_hv_regnet-400mf_secfpn_sbn-all_16xb2-2x_nus-3d.py)           |   2x    |   5.1    |                | 46.65 | 58.24 |                    [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/ssn/hv_ssn_regnet-400mf_secfpn_sbn-all_2x16_2x_nus-3d/hv_ssn_regnet-400mf_secfpn_sbn-all_2x16_2x_nus-3d_20210829_210615-361e5e04.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/ssn/hv_ssn_regnet-400mf_secfpn_sbn-all_2x16_2x_nus-3d/hv_ssn_regnet-400mf_secfpn_sbn-all_2x16_2x_nus-3d_20210829_210615.log.json)                    |

### Lyft

### Lyft

|                                   Backbone                                    | Lr schd | Mem (GB) | Inf time (fps) | Private Score | Public Score |                                                                                                                                                                                                      Download                                                                                                                                                                                                      |

|  [SECFPN](../pointpillars/pointpillars_hv_secfpn_sbn-all_8xb2-2x_lyft-3d.py)  |   2x    |   12.2   |                |     13.9      |     14.1     |  [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_secfpn_sbn-all_2x8_2x_lyft-3d/hv_pointpillars_secfpn_sbn-all_2x8_2x_lyft-3d_20210517_204807-2518e3de.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_secfpn_sbn-all_2x8_2x_lyft-3d/hv_pointpillars_secfpn_sbn-all_2x8_2x_lyft-3d_20210517_204807.log.json)  |

|              [SSN](./ssn_hv_secfpn_sbn-all_16xb2-2x_lyft-3d.py)               |   2x    |   8.5    |                |     17.5      |     17.5     |                           [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/ssn/hv_ssn_secfpn_sbn-all_2x16_2x_lyft-3d/hv_ssn_secfpn_sbn-all_2x16_2x_lyft-3d_20210822_134731-46841b41.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/ssn/hv_ssn_secfpn_sbn-all_2x16_2x_lyft-3d/hv_ssn_secfpn_sbn-all_2x16_2x_lyft-3d_20210822_134731.log.json)                           |

| [RegNetX-400MF-SSN](./ssn_hv_regnet-400mf_secfpn_sbn-all_16xb1-2x_lyft-3d.py) |   2x    |   7.4    |                |     17.9      |      18      | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/ssn/hv_ssn_regnet-400mf_secfpn_sbn-all_1x16_2x_lyft-3d/hv_ssn_regnet-400mf_secfpn_sbn-all_1x16_2x_lyft-3d_20210829_122825-d93475a1.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/ssn/hv_ssn_regnet-400mf_secfpn_sbn-all_1x16_2x_lyft-3d/hv_ssn_regnet-400mf_secfpn_sbn-all_1x16_2x_lyft-3d_20210829_122825.log.json) |

Note:

Note:

The main difference of the shape-aware grouping heads with the original SECOND FPN heads is that the former groups objects with similar sizes and shapes together, and design shape-specific heads for each group. Heavier heads (with more convolutions and large strides) are designed for large objects while smaller heads for small objects. Note that there may appear different feature map sizes in the outputs, so an anchor generator tailored to these feature maps is also needed in the implementation.

Users could try other settings in terms of the head design. Here we basically refer to the implementation [HERE](https://github.com/xinge008/SSN).

```latex
@inproceedings{zhu2020ssn,
  title={SSN: Shape Signature Networks for Multi-class Object Detection from Point Clouds},
  author={Zhu, Xinge and Ma, Yuexin and Wang, Tai and Xu, Yan and Shi, Jianping and Lin, Dahua},
  booktitle={Proceedings of the European Conference on Computer Vision},
}
```

# StyleGANv2 (CVPR'2020)

> [Analyzing and Improving the Image Quality of Stylegan](https://openaccess.thecvf.com/content_CVPR_2020/html/Karras_Analyzing_and_Improving_the_Image_Quality_of_StyleGAN_CVPR_2020_paper.html)

The style-based GAN architecture (StyleGAN) yields state-of-the-art results in data-driven unconditional generative image modeling. We expose and analyze several of its characteristic artifacts, and propose changes in both model architecture and training methods to address them. In particular, we redesign the generator normalization, revisit progressive growing, and regularize the generator to encourage good conditioning in the mapping from latent codes to images. In addition to improving image quality, this path length regularizer yields the additional benefit that the generator becomes significantly easier to invert. This makes it possible to reliably attribute a generated image to a particular network. We furthermore visualize how well the generator utilizes its output resolution, and identify a capacity problem, motivating us to train larger models for additional quality improvements. Overall, our improved model redefines the state of the art in unconditional image modeling, both in terms of existing distribution quality metrics as well as perceived image quality.

<img src="https://user-images.githubusercontent.com/28132635/143055738-58b15493-ab87-436d-94fd-c4c19e4a0225.JPG"/>
</div>

  <b> Results (compressed) from StyleGAN2 config-f trained by mmagic</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/12726765/113825919-25433100-97b4-11eb-84f7-5c66b3cfbc68.png" width="800"/>
</div>

|                                  Model                                   |   Dataset   |     Comment     | FID50k | Precision50k | Recall50k |                                  Download                                   |
| :----------------------------------------------------------------------: | :---------: | :-------------: | :----: | :----------: | :-------: | :-------------------------------------------------------------------------: |

| [stylegan2_c2_8xb4_ffhq-1024x1024](./stylegan2_c2_8xb4_ffhq-1024x1024.py) |    FFHQ     | official weight | 2.8134 |    62.856    |  49.400   | [model](https://download.openmmlab.com/mmediting/stylegan2/official_weights/stylegan2-ffhq-config-f-official_20210327_171224-bce9310c.pth) |
| [stylegan2_c2_8xb4_lsun-car-384x512](./stylegan2_c2_8xb4_lsun-car-384x512.py) |  LSUN_CAR   | official weight | 5.4316 |    65.986    |  48.190   | [model](https://download.openmmlab.com/mmediting/stylegan2/official_weights/stylegan2-car-config-f-official_20210327_172340-8cfe053c.pth) |
| [stylegan2_c2_8xb4-800kiters_lsun-horse-256x256](./stylegan2_c2_8xb4-800kiters_lsun-horse-256x256.py) | LSUN_HORSE  | official weight |   -    |      -       |     -     | [model](https://download.openmmlab.com/mmediting/stylegan2/official_weights/stylegan2-horse-config-f-official_20210327_173203-ef3e69ca.pth) |

| [stylegan2_c2_8xb4-800kiters_lsun-church-256x256](./stylegan2_c2_8xb4-800kiters_lsun-church-256x256.py) | LSUN_CHURCH | official weight |   -    |      -       |     -     | [model](https://download.openmmlab.com/mmediting/stylegan2/official_weights/stylegan2-church-config-f-official_20210327_172657-1d42b7d1.pth) |
| [stylegan2_c2_8xb4-800kiters_lsun-cat-256x256](./stylegan2_c2_8xb4-800kiters_lsun-cat-256x256.py) |  LSUN_CAT   | official weight |   -    |      -       |     -     | [model](https://download.openmmlab.com/mmediting/stylegan2/official_weights/stylegan2-cat-config-f-official_20210327_172444-15bc485b.pth) |
| [stylegan2_c2_8xb4-800kiters_ffhq-256x256](./stylegan2_c2_8xb4-800kiters_ffhq-256x256.py) |    FFHQ     |  our training   | 3.992  |    69.012    |  40.417   | [model](https://download.openmmlab.com/mmediting/stylegan2/stylegan2_c2_ffhq_256_b4x8_20210407_160709-7890ae1f.pth) |

| [stylegan2_c2_8xb4_ffhq-1024x1024](./stylegan2_c2_8xb4_ffhq-1024x1024.py) |    FFHQ     |  our training   | 2.8185 |    68.236    |  49.583   | [model](https://download.openmmlab.com/mmediting/stylegan2/stylegan2_c2_ffhq_1024_b4x8_20210407_150045-618c9024.pth) |
| [stylegan2_c2_8xb4_lsun-car-384x512](./stylegan2_c2_8xb4_lsun-car-384x512.py) |  LSUN_CAR   |  our training   | 2.4116 |    66.760    |  50.576   | [model](https://download.openmmlab.com/mmediting/stylegan2/stylegan2_c2_lsun-car_384x512_b4x8_1800k_20210424_160929-fc9072ca.pth) |

## FP16 Support and Experiments

Currently, we have supported FP16 training for StyleGAN2, and here are the results for the mixed-precision training. (Experiments for FFHQ1024 will come soon.)

  <b> Evaluation FID for FP32 and FP16 training </b>
  <br/>
  <img src="https://user-images.githubusercontent.com/12726765/117523645-18e90880-afec-11eb-9327-da14362b8bfd.png" width="600"/>
</div>

<b> Evaluation FID for FP32 and FP16 training </b>
  <br/>
</div>

As shown in the figure, we provide **3** ways to do mixed-precision training for `StyleGAN2`:

- [stylegan2_c2_fp16_PL-no-scaler](./stylegan2_c2-PL_8xb4-fp16-partial-GD-no-scaler-800kiters_ffhq-256x256.py): In this setting, we try our best to follow the official FP16 implementation in [StyleGAN2-ADA](https://github.com/NVlabs/stylegan2-ada). Similar to the official version, we only adopt FP16 training for the higher-resolution feature maps (the last 4 stages in G and the first 4 stages). Note that we do not adopt the `clamp` way to avoid gradient overflow used in the official implementation. We use the `autocast` function from `torch.cuda.amp` package.

- [stylegan2_c2_fp16-globalG-partialD_PL-R1-no-scaler](./stylegan2_c2-PL-R1_8xb4-fp16-globalG-partialD-no-scaler-800kiters_ffhq-256x256.py): In this config, we try to adopt mixed-precision training for the whole generator, but in partial discriminator (the first 4 higher-resolution stages). Note that we do not apply the loss scaler in the path length loss and gradient penalty loss. Because we always meet divergence after adopting the loss scaler to scale the gradient in these two losses.
- [stylegan2_c2_apex_fp16_PL-R1-no-scaler](./stylegan2_c2-PL-R1_8xb4-apex-fp16-no-scaler-800kiters_ffhq-256x256.py): In this setting, we adopt the [APEX](https://github.com/NVIDIA/apex) toolkit to implement mixed-precision training with multiple loss/gradient scalers. In APEX, you can assign different loss scalers for the generator and the discriminator respectively. Note that we still ignore the gradient scaler in the path length loss and gradient penalty loss.

|                                  Model                                   |                 Comment                 | Dataset | FID50k |                                   Download                                   |
| :----------------------------------------------------------------------: | :-------------------------------------: | :-----: | :----: | :--------------------------------------------------------------------------: |
| [stylegan2_c2_8xb4-800kiters_ffhq-256x256](./stylegan2_c2_8xb4-800kiters_ffhq-256x256.py) |                baseline                 | FFHQ256 | 3.992  | [ckpt](https://download.openmmlab.com/mmediting/stylegan2/stylegan2_c2_ffhq_256_b4x8_20210407_160709-7890ae1f.pth) |

| [stylegan2_c2-PL_8xb4-fp16-partial-GD-no-scaler-800kiters_ffhq-256x256](./stylegan2_c2-PL_8xb4-fp16-partial-GD-no-scaler-800kiters_ffhq-256x256.py) |         partial layers in fp16          | FFHQ256 | 4.331  | [ckpt](https://download.openmmlab.com/mmediting/stylegan2/stylegan2_c2_fp16_partial-GD_PL-no-scaler_ffhq_256_b4x8_800k_20210508_114854-dacbe4c9.pth) |
| [stylegan2_c2-PL-R1_8xb4-fp16-globalG-partialD-no-scaler-800kiters_ffhq-256x256](./stylegan2_c2-PL-R1_8xb4-fp16-globalG-partialD-no-scaler-800kiters_ffhq-256x256.py) |           the whole G in fp16           | FFHQ256 | 4.362  | [ckpt](https://download.openmmlab.com/mmediting/stylegan2/stylegan2_c2_fp16-globalG-partialD_PL-R1-no-scaler_ffhq_256_b4x8_800k_20210508_114930-ef8270d4.pth) |

| [stylegan2_c2-PL-R1_8xb4-apex-fp16-no-scaler-800kiters_ffhq-256x256](./stylegan2_c2-PL-R1_8xb4-apex-fp16-no-scaler-800kiters_ffhq-256x256.py) | the whole G&D in fp16 + two loss scaler | FFHQ256 | 4.614  | [ckpt](https://download.openmmlab.com/mmediting/stylegan2/stylegan2_c2_apex_fp16_PL-R1-no-scaler_ffhq_256_b4x8_800k_20210508_114701-c2bb8afd.pth) |

As shown in this table, `P&R50k_full` is the metric used in StyleGANv1 and StyleGANv2. `full` indicates that we use the whole dataset for extracting the real distribution, e.g., 70000 images in FFHQ dataset. However, adopting the VGG16 provided from [Tero](https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/pretrained/metrics/vgg16.pt) requires that your PyTorch version must fulfill `>=1.6.0`. Be careful about using the PyTorch's VGG16 to extract features, which will cause higher precision and recall.

```latex
@inproceedings{karras2020analyzing,
  title={Analyzing and improving the image quality of stylegan},

```latex
  author={Karras, Tero and Laine, Samuli and Aittala, Miika and Hellsten, Janne and Lehtinen, Jaakko and Aila, Timo},
  pages={8110--8119},
  url={https://openaccess.thecvf.com/content_CVPR_2020/html/Karras_Analyzing_and_Improving_the_Image_Quality_of_StyleGAN_CVPR_2020_paper.html},
}
```

</div>

English | [简体中文](README_CN.md)

</div>

Our RTMPose-m achieves **75.8% AP** on COCO with **90+ FPS** on an Intel i7-11700 CPU and **430+ FPS** on an NVIDIA GTX 1660 Ti GPU.
To further evaluate RTMPose's capability in critical real-time applications, we also report the performance after deploying on the mobile device. Our RTMPose-s achieves **72.2% AP** on COCO with **70+ FPS** on a Snapdragon 865 chip, outperforming existing open-source libraries.
With the help of MMDeploy, our project supports various platforms like CPU, GPU, NVIDIA Jetson, and mobile devices and multiple inference backends such as ONNXRuntime, TensorRT, ncnn, etc.

- [🙌 Community](#-community-)
- [⚡ Pipeline Performance](#-pipeline-performance-)
- [📊 Model Zoo](#-model-zoo-)
- [👀 Visualization](#-visualization-)
- [😎 Get Started](#-get-started-)
- [👨‍🏫 How to Train](#-how-to-train-)
- [🏗️ How to Deploy](#️-how-to-deploy-)
- [📚 Common Usage](#️-common-usage-)
  - [🚀 Inference Speed Test](#-inference-speed-test-)
  - [📊 Model Test](#-model-test-)
- [📜 Citation](#-citation-)

- Jun. 2023:

- [🚀 Inference Speed Test](#-inference-speed-test-)

  - Release 26-keypoint Body models trained on combined datasets.
- May. 2023:
  - Add [code examples](./examples/) of RTMPose.
  - Release Hand, Face, Body models trained on combined datasets.
- Mar. 2023: RTMPose is released. RTMPose-m runs at 430+ FPS and achieves 75.8 mAP on COCO val set.

</div>

</div>

</div>
</div>

### ✨ Major Features

- 🚀 **High efficiency and high accuracy**

- 🛠️ **Easy to deploy**

  - Step-by-step deployment tutorials.

|   s   |   72.2   |  200+   |  710+   |

  - Support various backends including
    - ONNX
    - TensorRT
    - ncnn
    - OpenVINO
    - etc.
  - Support various platforms including
    - Linux
    - Windows
    - ARM
    - etc.

- 🏗️ **Design for practical applications**

  - Pipeline inference API and SDK for
    - Python
    - C++
    - C#
    - JAVA
    - etc.

## 🙌 Community [🔝](#-table-of-contents)

RTMPose is a long-term project dedicated to the training, optimization and deployment of high-performance real-time pose estimation algorithms in practical scenarios, so we are looking forward to the power from the community. Welcome to share the training configurations and tricks based on RTMPose in different business applications to help more community users!

✨ ✨ ✨

✨ ✨ ✨

- **If you are a new user of RTMPose, we eagerly hope you can fill out this [Google Questionnaire](https://docs.google.com/forms/d/e/1FAIpQLSfzwWr3eNlDzhU98qzk2Eph44Zio6hi5r0iSwfO9wSARkHdWg/viewform?usp=sf_link)/[Chinese version](https://uua478.fanqier.cn/f/xxmynrki), it's very important for our work!**

✨ ✨ ✨

Feel free to join our community group for more help:

- WeChat Group:

</div>

## ⚡ Pipeline Performance [🔝](#-table-of-contents)

**Notes**

- Pipeline latency is tested under skip-frame settings, the detection interval is 5 frames by defaults.
- Flip test is NOT used.
- Env Setup:
  - ncnn 20221128
  - CUDA 11.3

  - CUDA 11.3

## 📊 Model Zoo [🔝](#-table-of-contents)

**Notes**

- Since all models are trained on multi-domain combined datasets for practical applications, results are **not** suitable for academic comparison.
- More results of RTMPose on public benchmarks can refer to [Model Zoo](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/algorithms.html)

- Flip test is used.
- Inference speed measured on more hardware platforms can refer to [Benchmark](./benchmark/README.md)
- If you have datasets you would like us to support, feel free to [contact us](https://docs.google.com/forms/d/e/1FAIpQLSfzwWr3eNlDzhU98qzk2Eph44Zio6hi5r0iSwfO9wSARkHdWg/viewform?usp=sf_link)/[联系我们](https://uua478.fanqier.cn/f/xxmynrki).

### Body 2d

- Keypoints are defined as [COCO](http://cocodataset.org/). For details please refer to the [meta info](/configs/_base_/datasets/coco.py).

- [MS COCO](https://mmpose.readthedocs.io/en/latest/dataset_zoo/2d_body_keypoint.html#coco)

- Keypoints are defined as [Halpe26](https://github.com/Fang-Haoshu/Halpe-FullBody/). For details please refer to the [meta info](/configs/_base_/datasets/halpe26.py).
- Models are trained and evaluated on `Body8`.

#### Model Pruning

**Notes**

**Notes**

- Model pruning is supported by [MMRazor](https://github.com/open-mmlab/mmrazor)

For more details, please refer to [GroupFisher Pruning for RTMPose](./rtmpose/pruning/README.md).

### WholeBody 2d (133 Keypoints)

- Keypoints are defined as [COCO-WholeBody](https://github.com/jin-s13/COCO-WholeBody/). For details please refer to the [meta info](/configs/_base_/datasets/coco_wholebody.py).

### Animal 2d (17 Keypoints)

- Keypoints are defined as [AP-10K](https://github.com/AlexTheBad/AP-10K/). For details please refer to the [meta info](/configs/_base_/datasets/ap10k.py).

### Face 2d (106 Keypoints)

- Keypoints are defined as [LaPa](https://github.com/JDAI-CV/lapa-dataset). For details please refer to the [meta info](/configs/_base_/datasets/lapa.py).

- `Face6` and `*` denote model trained on 6 public datasets:

### Hand 2d (21 Keypoints)

- Keypoints are defined as [COCO-WholeBody](https://github.com/jin-s13/COCO-WholeBody/). For details please refer to the [meta info](/configs/_base_/datasets/coco_wholebody_hand.py).

| [RTMDet-nano<sup><br>(alpha version)](./rtmdet/hand/rtmdet_nano_320-8xb32_hand.py) |  320x320   |             76.0              |       0.31        |                      -                      |                         -                          | [Det Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmdet_nano_8xb32-300e_hand-267f9c8f.pth) |

- `Hand5` and `*` denote model trained on 5 public datasets:

- [OneHand10K](https://www.yangangwang.com/papers/WANG-MCC-2018-10.html)

|                                                        Config                                                         | Input Size | PCK@0.2<sup><br>(COCO-Wholebody-Hand) | PCK@0.2<sup><br>(Hand5) | AUC<sup><br>(Hand5) | FLOPS<sup><br>(G) | ORT-Latency<sup><br>(ms)<sup><br>(i7-11700) | TRT-FP16-Latency<sup><br>(ms)<sup><br>(GTX 1660Ti) |                                                                 Download                                                                 |

| :-------------------------------------------------------------------------------------------------------------------: | :--------: | :-----------------------------------: | :---------------------: | :-----------------: | :---------------: | :-----------------------------------------: | :------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------: |
| [RTMPose-m\*<sup><br>(alpha version)](./rtmpose/hand_2d_keypoint/rtmpose-m_8xb32-210e_coco-wholebody-hand-256x256.py) |  256x256   |                 81.5                  |          96.4           |        83.9         |       2.581       |                      -                      |                         -                          | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-hand5_pt-aic-coco_210e-256x256-74fb594_20230320.pth) |

### Pretrained Models

We provide the UDP pretraining configs of the CSPNeXt backbone. Find more details in the [pretrain_cspnext_udp folder](./rtmpose/pretrain_cspnext_udp/).

We also provide the ImageNet classification pre-trained weights of the CSPNeXt backbone. Find more details in [RTMDet](https://github.com/open-mmlab/mmdetection/blob/latest/configs/rtmdet/README.md#classification).

## 👀 Visualization [🔝](#-table-of-contents)

</div>

## 😎 Get Started [🔝](#-table-of-contents)

We provide two appoaches to try RTMPose:

- MMPose demo scripts
- Pre-compiled MMDeploy SDK (Recommend, 6-10 times faster)

### MMPose demo scripts

</div>

MMPose provides demo scripts to conduct [inference with existing models](https://mmpose.readthedocs.io/en/latest/user_guides/inference.html).

**Note:**

- Inferencing with Pytorch can not reach the maximum speed of RTMPose, just for verification.
- Model file can be either a local path or a download link

```shell
# go to the mmpose folder

# inference with rtmdet

projects/rtmpose/rtmpose/body_2d_keypoint/rtmpose-m_8xb256-420e_coco-256x192.py \
    --show

    --show
```

Result is as follows:

### Pre-compiled MMDeploy SDK (Recommended)

MMDeploy provides a precompiled SDK for Pipeline reasoning on RTMPose projects, where the model used for reasoning is the SDK version.

- All models must by exported by `tools/deploy.py` before PoseTracker can be used for inference.
- For the tutorial of exporting the SDK version model, see [SDK Reasoning](#%EF%B8%8F-step3-inference-with-sdk), and for detailed parameter settings of inference, see [Pipeline Reasoning](#-step4-pipeline-inference).
- Exported SDK models (ONNX, TRT, ncnn, etc.) can be downloaded from [OpenMMLab Deploee](https://platform.openmmlab.com/deploee).
- You can also convert `.pth` models into SDK [online](https://platform.openmmlab.com/deploee/task-convert-list).

#### Linux

Env Requirements:

- GCC >= 7.5
- cmake >= 3.20

##### Python Inference

#### Linux

- GCC >= 7.5

1. Install mmdeploy_runtime or mmdeploy_runtime_gpu

```shell

```

2. Download Pre-compiled files.

```shell
# unzip then add third party runtime libraries to the PATH

```

3. Download the sdk models and unzip to `./example/python`. (If you need other models, please export sdk models refer to [SDK Reasoning](#%EF%B8%8F-step3-inference-with-sdk))

```shell

```

4. Inference with `pose_tracker.py`:

```shell
# go to ./example/python

# Please pass the folder of the model, not the model file
# Format:

# Example:

```shell

# Format:

# Example:

# webcam
```

##### ONNX

```shell
# Download pre-compiled files

# Unzip files

# Go to the sdk folder

# Init environment

# If opencv 3+ is not installed on your system, execute the following command.
# If it is installed, skip this command

# Compile executable programs

# Inference for an image

# Inference for a video

# Inference using webcam
```

```shell

```

For details, see [Pipeline Inference](#-step4-pipeline-inference).

```

```shell
# download [sdk](https://github.com/open-mmlab/mmdeploy/releases/download/v1.0.0/mmdeploy-1.0.0-windows-amd64.zip) add third party runtime libraries to the PATH

# download [sdk](https://github.com/open-mmlab/mmdeploy/releases/download/v1.0.0/mmdeploy-1.0.0-windows-amd64-cuda11.3.zip) add third party runtime libraries to the PATH
```

2. Download the sdk models and unzip to `./example/python`. (If you need other models, please export sdk models refer to [SDK Reasoning](#%EF%B8%8F-step3-inference-with-sdk))

```shell

```shell

```

3. Inference with `pose_tracker.py`:

```shell

```

##### Executable Inference

1. Install [CMake](https://cmake.org/download/).
2. Download the [pre-compiled SDK](https://github.com/open-mmlab/mmdeploy/releases).
3. Unzip the SDK and go to the `sdk` folder.
4. open windows powerShell with administrator privileges

```shell
```

5. Install OpenCV:

```shell
```

6. Set environment variables:

```shell

```shell
```

```shell
```

```shell
```

7. Compile the SDK:

```shell
# (if you installed opencv by .\install_opencv.ps1)
# (if you installed opencv yourself)
```

8. the executable will be generated in:

```shell
```

**Note:**

```shell

```shell

    --show

    --show
```

    --show
```

## 👨‍🏫 How to Train [🔝](#-table-of-contents)

Please refer to [Train and Test](https://mmpose.readthedocs.io/en/latest/user_guides/train_and_test.html).

**Tips**:

- Please accordinally reduce `batch_size` and `base_lr` when your dataset is small.
- Guidelines to choose a model
  - m: Recommended and Preferred Use
  - t/s: For mobile devices with extremely low computing power, or scenarios with stringent inference speed requirements
  - l: Suitable for scenarios with strong computing power and not sensitive to speed

## 🏗️ How to Deploy [🔝](#-table-of-contents)

Here is a basic example of deploy RTMPose with [MMDeploy](https://github.com/open-mmlab/mmdeploy/tree/main).

### 🧩 Step1. Install MMDeploy

Before starting the deployment, please make sure you install MMPose and MMDeploy correctly.

- Install MMPose, please refer to the [MMPose installation guide](https://mmpose.readthedocs.io/en/latest/installation.html).
- Install MMDeploy, please refer to the [MMDeploy installation guide](https://mmdeploy.readthedocs.io/en/latest/get_started.html#installation).

Depending on the deployment backend, some backends require compilation of custom operators, so please refer to the corresponding document to ensure the environment is built correctly according to your needs:

- [ONNX RUNTIME SUPPORT](https://mmdeploy.readthedocs.io/en/latest/05-supported-backends/onnxruntime.html)
- [TENSORRT SUPPORT](https://mmdeploy.readthedocs.io/en/latest/05-supported-backends/tensorrt.html)
- [OPENVINO SUPPORT](https://mmdeploy.readthedocs.io/en/latest/05-supported-backends/openvino.html)
- [More](https://github.com/open-mmlab/mmdeploy/tree/main/docs/en/05-supported-backends)

### 🛠️ Step2. Convert Model

The detailed model conversion tutorial please refer to the [MMDeploy document](https://mmdeploy.readthedocs.io/en/latest/02-how-to-run/convert_model.html). Here we only give the example of converting RTMPose.

Here we take converting RTMDet-nano and RTMPose-m to ONNX/TensorRT as an example.

- If you only want to use ONNX, please use:
  - [`detection_onnxruntime_static.py`](https://github.com/open-mmlab/mmdeploy/blob/main/configs/mmdet/detection/detection_onnxruntime_static.py) for RTMDet.
  - [`pose-detection_simcc_onnxruntime_dynamic.py`](https://github.com/open-mmlab/mmdeploy/blob/main/configs/mmpose/pose-detection_simcc_onnxruntime_dynamic.py) for RTMPose.
- If you want to use TensorRT, please use：

  - [`detection_tensorrt_static-320x320.py`](https://github.com/open-mmlab/mmdeploy/blob/main/configs/mmdet/detection/detection_tensorrt_static-320x320.py) for RTMDet.
  - [`pose-detection_simcc_tensorrt_dynamic-256x192.py`](https://github.com/open-mmlab/mmdeploy/blob/main/configs/mmpose/pose-detection_simcc_tensorrt_dynamic-256x192.py) for RTMPose.

If you want to customize the settings in the deployment config for your requirements, please refer to [MMDeploy config tutorial](https://mmdeploy.readthedocs.io/en/latest/02-how-to-run/write_config.html).

In this tutorial, we organize files as follows:

```shell
```

#### ONNX

```shell

# run the command to convert RTMDet
# Model file can be either a local path or a download link

    --show \
    --dump-info  # dump sdk info

# run the command to convert RTMPose

    --show \
```

The converted model file is `{work-dir}/end2end.onnx` by defaults.

```shell

    --show \

    --show \

    --show \
```

The converted model file is `{work-dir}/end2end.engine` by defaults.

🎊 If the script runs successfully, you will see the following files:

```Python
    ))
```

### 🕹️ Step3. Inference with SDK

To use SDK, you need to dump the required info during converting the model. Just add --dump-info to the model conversion command.

```shell
# RTMDet

    --show \

# RTMPose
    --show \
```

    --show \
```

```shell
|----{work-dir}

```

```Python

import cv2

parser.add_argument('device_name', help='name of device, cuda or cpu')
        '--bbox',
        type=int,

    else:

print(args.bbox)

    main()
```

```C++

    return -1;
  }

    return -1;
  }

    return -1;
  }

    }
  }

if (det.score > FLAGS_det_thr) {  // filter bboxes
    }
  }

  }

  return 0;
}
```

```CMake
```

- [C# API Examples](https://github.com/open-mmlab/mmdeploy/tree/main/demo/csharp)
- [JAVA API Examples](https://github.com/open-mmlab/mmdeploy/tree/main/demo/java)

## 🚀 Step4. Pipeline Inference

### Inference for images

If the user has MMDeploy compiled correctly, you will see the `det_pose` executable under the `mmdeploy/build/bin/`.

```shell

# inference for an image

  det_model           Object detection model path [string]
  pose_model          Pose estimation model path [string]
  image               Input image path [string]

  --device            Device name, e.g. "cpu", "cuda" [string = "cpu"]
  --output            Output image path [string = "det_pose_output.jpg"]
  --skeleton          Path to skeleton data or name of predefined skeletons:
  --det_label         Detection label use for pose estimation [int32 = 0]
                      (0 refers to 'person' in coco)
  --det_thr           Detection score threshold [double = 0.5]
  --det_min_bbox_size Detection minimum bbox size [double = -1]
  --pose_thr          Pose key-point threshold [double = 0]
```

--det_thr           Detection score threshold [double = 0.5]
```

#### API Example

- [`det_pose.py`](https://github.com/open-mmlab/mmdeploy/blob/main/demo/python/det_pose.py)
- [`det_pose.cxx`](https://github.com/open-mmlab/mmdeploy/blob/main/demo/csrc/cpp/det_pose.cxx)

### Inference for a video

If the user has MMDeploy compiled correctly, you will see the `pose_tracker` executable under the `mmdeploy/build/bin/`.

- pass `0` to `input` can inference from a webcam

```shell

# inference for a video

  det_model             Object detection model path [string]
  pose_model            Pose estimation model path [string]
  input                 Input video path or camera index [string]

det_model             Object detection model path [string]

  --device              Device name, e.g. "cpu", "cuda" [string = "cpu"]
  --output              Output video path or format string [string = ""]
  --output_size         Long-edge of output frames [int32 = 0]
  --flip                Set to 1 for flipping the input horizontally [int32 = 0]
  --show                Delay passed to `cv::waitKey` when using `cv::imshow`;
                        -1: disable [int32 = 1]
  --skeleton            Path to skeleton data or name of predefined skeletons:
  --background          Output background, "default": original image, "black":
                        black background [string = "default"]
  --det_interval        Detection interval [int32 = 1]

--background          Output background, "default": original image, "black":
  --det_label           Detection label use for pose estimation [int32 = 0]
                        (0 refers to 'person' in coco)
  --det_thr             Detection score threshold [double = 0.5]
  --det_min_bbox_size   Detection minimum bbox size [double = -1]
  --det_nms_thr         NMS IOU threshold for merging detected bboxes and
                        bboxes from tracked targets [double = 0.7]
  --pose_max_num_bboxes Max number of bboxes used for pose estimation per frame
  --pose_kpt_thr        Threshold for visible key-points [double = 0.5]
  --pose_min_keypoints  Min number of key-points for valid poses, -1 indicates
                        ceil(n_kpts/2) [int32 = -1]
  --pose_bbox_scale     Scale for expanding key-points to bbox [double = 1.25]

ceil(n_kpts/2) [int32 = -1]
  --pose_min_bbox_size  Min pose bbox size, tracks with bbox size smaller than
                        the threshold will be dropped [double = -1]
  --pose_nms_thr        NMS OKS/IOU threshold for suppressing overlapped poses,
                        useful when multiple pose estimations collapse to the
                        same target [double = 0.5]
  --track_iou_thr       IOU threshold for associating missing tracks
                        [double = 0.4]
  --track_max_missing   Max number of missing frames before a missing tracks is
                        removed [int32 = 10]
```

- [`pose_tracker.py`](https://github.com/open-mmlab/mmdeploy/blob/main/demo/python/pose_tracker.py)
- [`pose_tracker.cxx`](https://github.com/open-mmlab/mmdeploy/blob/main/demo/csrc/cpp/pose_tracker.cxx)

## 📚 Common Usage [🔝](#-table-of-contents)

### 🚀 Inference Speed Test [🔝](#-table-of-contents)

If you need to test the inference speed of the model under the deployment framework, MMDeploy provides a convenient `tools/profiler.py` script.

The user needs to prepare a folder for the test images `./test_images`, the profiler will randomly read images from this directory for the model speed test.

```shell
```

```shell

```

```shell

```

### 📊 Model Test [🔝](#-table-of-contents)

If you need to test the inference accuracy of the model on the deployment backend, MMDeploy provides a convenient `tools/test.py` script.

```shell

```shell
```

You can also refer to [MMDeploy Docs](https://github.com/open-mmlab/mmdeploy/blob/main/docs/en/02-how-to-run/profile_model.md) for more details.

## 📜 Citation [🔝](#-table-of-contents)

If you find RTMPose useful in your research, please consider cite:

```bibtex

url = {https://arxiv.org/abs/2303.07399},
}

}
```

# 准备 MultiSports

## 介绍

```BibTeX
@inproceedings{li2021multisports,
  title={Multisports: A multi-person video dataset of spatio-temporally localized sports actions},
  author={Li, Yixuan and Chen, Lei and He, Runyu and Wang, Zhenzhi and Wu, Gangshan and Wang, Limin},
  pages={13536--13545},
}
```

关于基本数据集信息，请参考官方 [项目](https://deeperaction.github.io/datasets/multisports.html) 和 [论文](https://arxiv.org/abs/2105.07404)。
在我们开始之前，请确保目录位于 `$MMACTION2/tools/data/multisports/`。

## 第一步：准备标注

首先，你必须从官方 [网站](https://github.com/MCG-NJU/MultiSports) 下载标注和视频到 `$MMACTION2/data/multisports`，请同时下载人物检测框并将其放到 `$MMACTION2/data/multisports`。

## 第二步：准备视频

在这一步之前，请确保文件夹结构如下：

```
mmaction2
├── tools
├── data
│   ├── multisports
│   |   ├── MultiSports_box.zip
│   |   ├── trainval
│   |   |   ├── aerobic_gymnastics.zip
│   |   |   ├── basketball.zip

```
mmaction2
├── tools
├── data
│   |   |   ├── multisports_GT.pkl
│   |   |   ├──...
│   |   ├── test
│   |   |   ├──...
```

然后，你可以使用以下命令进行解压。

```shell
cd $MMACTION2/data/multisports/
unzip MultiSports_box.zip
cd $MMACTION2/data/multisports/trainval
find . -name '*.zip' -exec unzip {} \;
cd $MMACTION2/data/multisports/test
cd $MMACTION2/tools/data/multisports/
```

## 第三步：转换标注文件

你可以运行以下脚本来转换我们需要的标注文件和候选框。

```shell
python parse_anno.py
```

## 第五步：检查目录结构

完成整个数据处理后，你将得到 MultiSports 数据集的视频和标注文件。

在整个项目的目录中（仅针对 MultiSports），文件夹结构如下：

```
mmaction2
├── tools
├── data
│   |   ├── annotations

```

```
mmaction2
├── tools
├── data
|   │   |   ├── multisports_dense_proposals_test.recall_96.13.pkl
|   │   |   ├── multisports_dense_proposals_train.recall_96.13.pkl
|   │   |   ├── multisports_dense_proposals_val.recall_96.13.pkl
|   │   |   ├── multisports_GT.pkl
|   │   |   ├── multisports_train.csv
|   │   |   ├── multisports_val.csv
│   |   |   ├── aerobic_gymnastics
|   │   |   |   ├── v__wAgwttPYaQ_c001.mp4
|   │   |   |   ├── v__wAgwttPYaQ_c002.mp4
|   │   |   |   ├── ...
│   |   |   ├── basketball
|   │   |   |   ├── v_-6Os86HzwCs_c001.mp4
|   │   |   |   ├── v_-6Os86HzwCs_c002.mp4
│   |   |   ├──...
│   |   ├── test
|   │   |   |   ├── v_2KroSzspz-c_c001.mp4

│   |   |   ├──...
│   |   ├── test
|   │   |   |   ├── v_2KroSzspz-c_c002.mp4
|   │   |   |   ├── v_1tefH1iPbGM_c001.mp4
|   │   |   |   ├── v_1tefH1iPbGM_c002.mp4
│   |   |   ├──...
```

我们不需要项目下的 zip 文件，你可以按照自己的意愿处理它们。
关于在 MultiSports 上进行训练和评估，请参考 [训练和测试教程](/docs/en/user_guides/train_test.md)。

# End-to-end Recovery of Human Shape and Pose

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_cvpr_2018/html/Kanazawa_End-to-End_Recovery_of_CVPR_2018_paper.html">HMR (CVPR'2018)</a></summary>

```bibtex
@inProceedings{kanazawaHMR18,
  title={End-to-end Recovery of Human Shape and Pose},
  author = {Angjoo Kanazawa
  and Michael J. Black
  and David W. Jacobs
  and Jitendra Malik},
  booktitle={Computer Vision and Pattern Recognition (CVPR)},
}
```

We describe Human Mesh Recovery (HMR), an end-to-end framework for reconstructing a full 3D mesh of a human body from a single RGB image. In contrast to most current methods that compute 2D or 3D joint locations, we produce a richer and more useful mesh representation that is parameterized by shape and 3D joint angles. The main objective is to minimize the reprojection loss of keypoints, which allows our model to be trained using in-the-wild images that only have ground truth 2D annotations. However, the reprojection loss alone is highly underconstrained. In this work we address this problem by introducing an adversary trained to tell whether human body shape and pose are real or not using a large database of 3D human meshes. We show that HMR can be trained with and without using any paired 2D-to-3D supervision. We do not rely on intermediate 2D keypoint detections and infer 3D pose and shape parameters directly from image pixels. Our model runs in real-time given a bounding box containing the person. We demonstrate our approach on various images in-the-wild and out-perform previous optimization-based methods that output 3D meshes and show competitive results on tasks such as 3D joint location estimation and part segmentation.

<img src="https://user-images.githubusercontent.com/15977946/146515822-a271ec4d-4045-4456-a7cb-740018317053.png">
</div>

# InternImage Classification

This is the implementation of [InternImage](https://arxiv.org/abs/2211.05778) for image classification.

## Usage

Please refer to [Get Started](https://mmpretrain.readthedocs.io/en/latest/get_started.html) documentation of MMPretrain to finish installation.

Please install DCNv3. Run the command below following the [ InternImage official installation instructions](https://github.com/OpenGVLab/InternImage/blob/master/classification/README.md).

```shell
cd ops_dcnv3
sh ./make.sh
```

### Training and Test Commands

At first, you need to add the current folder to `PYTHONPATH`, so that Python can find your model files. In `projects/internimage_classification/` root directory, please run command below to add it.

```shell
```

#### Training

##### On Local Single GPU

```bash
# train with mim
mim train mmpretrain ${CONFIG} --work-dir ${WORK_DIR}

# a specific command example

```shell
```

```bash

mim train mmpretrain configs/internimage-tiny_8xb128_in1k-224.py \
	--work-dir work_dirs/internimage-tiny_8xb128_in1k-224/
```

##### On Multiple GPUs

```bash
mim train mmpretrain ${CONFIG} \
    --launcher pytorch --gpus 8
```

##### On Multiple GPUs with Slurm

```bash
    --launcher slurm --gpus 16 --gpus-per-node 8 \
    --partition ${PARTITION}
```

#### Test

Please download the pretrain weight provided by [OpenGVLab](https://github.com/OpenGVLab/) from [here](https://huggingface.co/OpenGVLab/InternImage/tree/main)

```bash
# test with mim
mim test mmpretrain ${CONFIG} -C ${CHECKPOINT}

```bash

mim test mmpretrain configs/internimage-tiny_8xb128_in1k-224.py -C /PATH/TO/internimage_t_1k_224.pth
```

```bash
# a specific command examples, 8 GPUs here
mim test mmpretrain configs/internimage_t_1k_224.py \
	-C /PATH/TO/internimage_t_1k_224.pth \
```

```bash
mim test mmpretrain ${CONFIG} \
    -C ${CHECKPOINT}
    --launcher slurm --gpus 8 --gpus-per-node 8 \
    --partition ${PARTITION} \
    $PY_ARGS
```

Note: `PY_ARGS` is other optional args.

## Results on ImageNet1K

The accuracy of different models on ImageNet1K,

--partition ${PARTITION} \
    $PY_ARGS
```

|      name      | resolution |  acc@1  |  acc@5  |                          config                           |                                              weight                                               |
| :------------: | :--------: | :-----: | :-----: | :-------------------------------------------------------: | :-----------------------------------------------------------------------------------------------: |
| InternImage-T  |    224     | 83.4700 | 96.5340 |  [config](./configs/internimage-tiny_8xb128_in1k-224.py)  |    [model](https://huggingface.co/OpenGVLab/InternImage/resolve/main/internimage_t_1k_224.pth)    |

| InternImage-S  |    224     | 84.1640 | 96.9320 | [config](./configs/internimage-small_8xb128_in1k-224.py)  |    [model](https://huggingface.co/OpenGVLab/InternImage/resolve/main/internimage_s_1k_224.pth)    |
| InternImage-B  |    224     | 84.8660 | 97.1820 |  [config](./configs/internimage-base_8xb128_in1k-224.py)  |    [model](https://huggingface.co/OpenGVLab/InternImage/resolve/main/internimage_b_1k_224.pth)    |
| InternImage-L  |    384     | 87.7060 | 98.3820 | [config](./configs/internimage-large_8xb128_in1k-384.py)  | [model](https://huggingface.co/OpenGVLab/InternImage/resolve/main/internimage_l_22kto1k_384.pth)  |
| InternImage-XL |    384     | 88.0460 | 98.5620 | [config](./configs/internimage-xlagre_8xb128_in1k-384.py) | [model](https://huggingface.co/OpenGVLab/InternImage/resolve/main/internimage_xl_22kto1k_384.pth) |

| InternImage-H  |    640     | 89.5500 | 98.8500 |  [config](./configs/internimage-huge_8xb128_in1k-640.py)  | [model](https://huggingface.co/OpenGVLab/InternImage/resolve/main/internimage_h_22kto1k_640.pth)  |
| InternImage-G  |    512     | 90.0580 | 98.9700 | [config](./configs/internimage-giant_8xb128_in1k-512.py)  | [model](https://huggingface.co/OpenGVLab/InternImage/resolve/main/internimage_g_22kto1k_512.pth)  |

```bibtex
@article{wang2022internimage,
  title={InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions},
  author={Wang, Wenhai and Dai, Jifeng and Chen, Zhe and Huang, Zhenhang and Li, Zhiqi and Zhu, Xizhou and Hu, Xiaowei and Lu, Tong and Lu, Lewei and Li, Hongsheng and others},
  journal={arXiv preprint arXiv:2211.05778},
}
```

# Learn about Configs (To be updated)

The mmrotate is built upon the [mmdet](https://github.com/open-mmlab/mmdetection),
thus it is highly recommended to learn the basics of [mmdet](https://mmdetection.readthedocs.io/en/latest/).

## Modify a config through script arguments

When submitting jobs using "tools/train.py" or "tools/test.py", you may specify `--cfg-options` to in-place modify the config.

  For example, `--cfg-options model.backbone.norm_eval=False` changes all BN modules in model backbones to `train` mode.

For example, `--cfg-options model.backbone.norm_eval=False` changes all BN modules in model backbones to `train` mode.

  Some config dicts are composed as a list in your config. For example, the training pipeline `data.train.pipeline` is normally a list
  e.g. `[dict(type='LoadImageFromFile'), ...]`. If you want to change `'LoadImageFromFile'` to `'LoadImageFromWebcam'` in the pipeline,
  you may specify `--cfg-options data.train.pipeline.0.type=LoadImageFromWebcam`.

  If the value to be updated is a list or a tuple. For example, the config file normally sets `workflow=[('train', 1)]`. If you want to
  change this key, you may specify `--cfg-options workflow="[(train,1),(val,1)]"`. Note that the quotation mark " is necessary to
  support list/tuple data types, and that **NO** white space is allowed inside the quotation marks in the specified value.

## Config file naming convention

support list/tuple data types, and that **NO** white space is allowed inside the quotation marks in the specified value.

```text
{model}_[model setting]_{backbone}_{neck}_[norm setting]_[misc]_[gpu x batch_per_gpu]_{dataset}_{data setting}_{angle version}
```

- `{model}`: model type like `rotated_faster_rcnn`, `rotated_retinanet`, etc.
- `[model setting]`: specific setting for some model, like `hbb` for `rotated_retinanet`, etc.
- `{backbone}`: backbone type like `r50` (ResNet-50), `swin_tiny` (SWIN-tiny).
- `{neck}`: neck type like `fpn`,  `refpn`.
- `[norm_setting]`: `bn` (Batch Normalization) is used unless specified, other norm layer types could be `gn` (Group Normalization), `syncbn` (Synchronized Batch Normalization).

  `gn-head`/`gn-neck` indicates GN is applied in head/neck only, while `gn-all` means GN is applied in the entire model, e.g. backbone, neck, head.
- `[misc]`: miscellaneous setting/plugins of the model, e.g. `dconv`, `gcb`, `attention`, `albu`, `mstrain`.
- `[gpu x batch_per_gpu]`: GPUs and samples per GPU, `1xb2` is used by default.
- `{dataset}`: dataset like `dota`.
- `{angle version}`: like `oc`, `le135`, or `le90`.

## An example of RotatedRetinaNet

To help the users have a basic idea of a complete config and the modules in a modern detection system,
we make brief comments on the config of RotatedRetinaNet using ResNet50 and FPN as the following. For more
detailed usage and the corresponding alternative for each module, please refer to the API documentation.

```python

```python
angle_version = 'oc'  # The angle version
    type='RotatedRetinaNet',  # The name of detector
        type='ResNet',  # The type of the backbone
        depth=50,  # The depth of backbone
        out_indices=(0, 1, 2, 3),  # The index of output feature maps produced in each stages
        frozen_stages=1,  # The weights in the first 1 stage are fronzen
        zero_init_residual=False,  # Whether to use zero init for last norm layer in resblocks to let them behave as identity.

        init_cfg=dict(type='Pretrained', checkpoint='torchvision://resnet50')),  # The ImageNet pretrained backbone to be loaded
        type='FPN',  # The neck of detector is FPN. We also support 'ReFPN'
        start_level=1,  # Index of the start input backbone level used to build the feature pyramid
        add_extra_convs='on_input',  # It specifies the source feature map of the extra convs

add_extra_convs='on_input',  # It specifies the source feature map of the extra convs
        type='RotatedRetinaHead',# The type of bbox head is 'RRetinaHead'
        num_classes=15,  # Number of classes for classification
        in_channels=256,  # Input channels for bbox head
        stacked_convs=4,  # Number of stacking convs of the head
        feat_channels=256,  # Number of hidden channels
        assign_by_circumhbbox='oc',  # The angle version of obb2hbb
            type='RotatedAnchorGenerator',  # The type of anchor generator
            octave_base_scale=4,  # The base scale of octave.
            scales_per_octave=3,  #  Number of scales for each octave.
            ratios=[1.0, 0.5, 2.0],  # The ratio between height and width.

scales_per_octave=3,  #  Number of scales for each octave.
            strides=[8, 16, 32, 64, 128]),  # The strides of the anchor generator. This is consistent with the FPN feature strides.
            type='DeltaXYWHAOBBoxCoder',  # Type of box coder.
            angle_range='oc',  # The angle version of box coder.
            norm_factor=None,  # The norm factor of box coder.
            edge_swap=False,  # The edge swap flag of box coder.
            proj_xy=False,  # The project flag of box coder.
            target_means=(0.0, 0.0, 0.0, 0.0, 0.0),  # The target means used to encode and decode boxes
            target_stds=(1.0, 1.0, 1.0, 1.0, 1.0)),  # The standard variance used to encode and decode boxes

target_stds=(1.0, 1.0, 1.0, 1.0, 1.0)),  # The standard variance used to encode and decode boxes
            type='FocalLoss',  # Type of loss for classification branch
            use_sigmoid=True,  #  Whether the prediction is used for sigmoid or softmax
            gamma=2.0,  # The gamma for calculating the modulating factor
            alpha=0.25,  # A balanced form for Focal Loss
            loss_weight=1.0),  # Loss weight of the classification branch
            type='L1Loss',  # Type of loss
            loss_weight=1.0)),  # Loss weight of the regression branch
    train_cfg=dict(  # Config of training hyperparameters
            type='MaxIoUAssigner',  # Type of assigner
            pos_iou_thr=0.5,  # IoU >= threshold 0.5 will be taken as positive samples

assigner=dict(  # Config of assigner
            neg_iou_thr=0.4,  # IoU < threshold 0.4 will be taken as negative samples
            min_pos_iou=0,  # The minimal IoU threshold to take boxes as positive samples
            ignore_iof_thr=-1,  # IoF threshold for ignoring bboxes
            iou_calculator=dict(type='RBboxOverlaps2D')),  # Type of Calculator for IoU
        allowed_border=-1,  # The border allowed after padding for valid anchors.
        pos_weight=-1,  # The weight of positive samples during training.
    test_cfg=dict(  # Config of testing hyperparameters
        nms_pre=2000,  # The number of boxes before NMS
        score_thr=0.05,  # Threshold to filter out boxes

nms_pre=2000,  # The number of boxes before NMS
        nms=dict(iou_thr=0.1), # NMS threshold
        max_per_img=2000))  # The number of boxes to be kept after NMS.
dataset_type = 'DOTADataset'  # Dataset type, this will be used to define the dataset
data_root = '../datasets/split_1024_dota1_0/'  # Root path of data
img_norm_cfg = dict(  # Image normalization config to normalize the input images
    mean=[123.675, 116.28, 103.53],  # Mean values used to pre-training the pre-trained backbone models
    std=[58.395, 57.12, 57.375],  # Standard variance used to pre-training the pre-trained backbone models
    to_rgb=True)  # The channel orders of image used to pre-training the pre-trained backbone models
train_pipeline = [  # Training pipeline

    dict(type='LoadAnnotations',  # Second pipeline to load annotations for current image
         with_bbox=True),  # Whether to use bounding box, True for detection
    dict(type='RResize',  # Augmentation pipeline that resize the images and their annotations
         img_scale=(1024, 1024)),  # The largest scale of image
    dict(type='RRandomFlip',  # Augmentation pipeline that flip the images and their annotations
         flip_ratio=0.5,  # The ratio or probability to flip
         version='oc'),  # The angle version
    dict(
        type='Normalize',  # Augmentation pipeline that normalize the input images
        mean=[123.675, 116.28, 103.53],  # These keys are the same of img_norm_cfg since the
        std=[58.395, 57.12, 57.375],  # keys of img_norm_cfg are used here as arguments
        to_rgb=True),
    dict(type='Pad',  # Padding config

std=[58.395, 57.12, 57.375],  # keys of img_norm_cfg are used here as arguments
         size_divisor=32),  # The number the padded images should be divisible
    dict(type='DefaultFormatBundle'),  # Default format bundle to gather data in the pipeline
    dict(type='Collect',  # Pipeline that decides which keys in the data should be passed to the detector
         keys=['img', 'gt_bboxes', 'gt_labels'])
]
    dict(
        type='MultiScaleFlipAug',  # An encapsulation that encapsulates the testing augmentations
        img_scale=(1024, 1024),  # Decides the largest scale for testing, used for the Resize pipeline
        flip=False,  # Whether to flip images during testing
            dict(type='RResize'),  # Use resize augmentation
            dict(

flip=False,  # Whether to flip images during testing
            dict(
                type='Normalize',  # Normalization config, the values are from img_norm_cfg
                mean=[123.675, 116.28, 103.53],
                std=[58.395, 57.12, 57.375],
                to_rgb=True),
            dict(type='Pad',  # Padding config to pad images divisible by 32.
                 size_divisor=32),
            dict(type='DefaultFormatBundle'),  # Default format bundle to gather data in the pipeline
            dict(type='Collect',  # Collect pipeline that collect necessary keys for testing.
                 keys=['img'])
        ])
]
    samples_per_gpu=2,  # Batch size of a single GPU
    workers_per_gpu=2,  # Worker to pre-fetch data for each single GPU
    train=dict(  # Train dataset config
        type='DOTADataset',  # Type of dataset
        ann_file=

workers_per_gpu=2,  # Worker to pre-fetch data for each single GPU
        ann_file=
        '../datasets/split_1024_dota1_0/trainval/annfiles/',  # Path of annotation file
        img_prefix=
        '../datasets/split_1024_dota1_0/trainval/images/',  # Prefix of image path
        pipeline=[  # pipeline, this is passed by the train_pipeline created before.
            dict(type='LoadAnnotations', with_bbox=True),
            dict(type='RResize', img_scale=(1024, 1024)),
            dict(type='RRandomFlip', flip_ratio=0.5, version='oc'),
            dict(
                type='Normalize',
            dict(type='DefaultFormatBundle'),

std=[58.395, 57.12, 57.375],
            dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'])
        ],
        version='oc'),
    val=dict(  # Validation dataset config
        type='DOTADataset',
        ann_file=
        '../datasets/split_1024_dota1_0/trainval/annfiles/',
        '../datasets/split_1024_dota1_0/trainval/images/',
            dict(
                type='MultiScaleFlipAug',
                img_scale=(1024, 1024),
                flip=False,
                transforms=[
                    dict(type='RResize'),
                    dict(
                        type='Normalize',
                        mean=[123.675, 116.28, 103.53],
                        std=[58.395, 57.12, 57.375],
                        to_rgb=True),

type='Normalize',
                    dict(type='Pad', size_divisor=32),
                    dict(type='DefaultFormatBundle'),
                    dict(type='Collect', keys=['img'])
                ])
        ],
    test=dict(  # Test dataset config, modify the ann_file for test-dev/test submission
        ann_file=
        '../datasets/split_1024_dota1_0/test/images/',
        pipeline=[  # Pipeline is passed by test_pipeline created before
            dict(

dict(
                    dict(
                ])
        ],
        version='oc'))
    interval=12,  # Evaluation interval
    metric='mAP')  # Metrics used during evaluation
optimizer = dict(  # Config used to build optimizer
    type='SGD',  # Type of optimizers
    lr=0.0025,  # Learning rate of optimizers
    momentum=0.9,  # Momentum
    weight_decay=0.0001)  # Weight decay of SGD

type='SGD',  # Type of optimizers
optimizer_config = dict(  # Config used to build the optimizer hook
    grad_clip=dict(
        max_norm=35,
        norm_type=2))
lr_config = dict(  # Learning rate scheduler config used to register LrUpdater hook
    policy='step',  # The policy of scheduler
    warmup='linear',  # The warmup policy, also support `exp` and `constant`.
    warmup_iters=500,  # The number of iterations for warmup
    warmup_ratio=0.3333333333333333,  # The ratio of the starting learning rate used for warmup
    step=[8, 11])  # Steps to decay the learning rate
    type='EpochBasedRunner',  # Type of runner to use (i.e. IterBasedRunner or EpochBasedRunner)
    max_epochs=12) # Runner that runs the workflow in total max_epochs. For IterBasedRunner use `max_iters`
checkpoint_config = dict(  # Config to set the checkpoint hook

max_epochs=12) # Runner that runs the workflow in total max_epochs. For IterBasedRunner use `max_iters`
    interval=12)  # The save interval is 12
log_config = dict(  # config to register logger hook
    interval=50,  # Interval to print the log
    hooks=[
        # dict(type='TensorboardLoggerHook')  # The Tensorboard logger is also supported
        dict(type='TextLoggerHook')
    ])  # The logger used to record the training process.
dist_params = dict(backend='nccl')  # Parameters to setup distributed training, the port can also be set.
load_from = None  # load models as a pre-trained model from a given path. This will not resume training.
resume_from = None  # Resume checkpoints from a given path, the training will be resumed from the epoch when the checkpoint's is saved.

workflow = [('train', 1)]  # Workflow for runner. [('train', 1)] means there is only one workflow and the workflow named 'train' is executed once. The workflow trains the model by 12 epochs according to the total_epochs.
work_dir = './work_dirs/rotated_retinanet_hbb_r50_fpn_1x_dota_oc'  # Directory to save the model checkpoints and logs for the current experiments.
```

## FAQ

It's worth noting that when modifying intermediate variables in the children configs, the user needs to pass the intermediate variables into corresponding fields again.
For example, we would like to use an offline multi-scale strategy to train an RoI-Trans. `train_pipeline` are intermediate variables we would like to modify.

```python
_base_ = ['./roi-trans-le90_r50_fpn_1x_dota.py']

data_root = '../datasets/split_ms_dota1_0/'
angle_version = 'le90'
    dict(type='RResize', img_scale=(1024, 1024)),
    dict(
        type='RRandomFlip',
        flip_ratio=[0.25, 0.25, 0.25],
        direction=['horizontal', 'vertical', 'diagonal'],
        version=angle_version),
    dict(
        type='PolyRandomRotate',
        rotate_ratio=0.5,
        angles_range=180,
        auto_bound=False,

auto_bound=False,
    dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'])
]
        ann_file=data_root + 'trainval/annfiles/',
        img_prefix=data_root + 'trainval/images/'),
    val=dict(
        ann_file=data_root + 'test/images/',
        img_prefix=data_root + 'test/images/'))
```

We first define the new `train_pipeline`/`test_pipeline` and pass them into `data`.

```python
_base_ = './roi-trans-le90_r50_fpn_1x_dota.py'

```python
    ...)
```

# 安装

我们推荐用户按照我们的最佳实践来安装 MMPose。但除此之外，如果您想根据
您的习惯完成安装流程，也可以参见 [自定义安装](#自定义安装) 一节来获取更多信息。

  - [依赖环境](#依赖环境)
  - [最佳实践](#最佳实践)
    - [从源码安装 MMPose](#从源码安装-mmpose)
    - [作为 Python 包安装](#作为-python-包安装)
  - [验证安装](#验证安装)
  - [自定义安装](#自定义安装)
    - [CUDA 版本](#cuda-版本)
    - [不使用 MIM 安装 MMEngine](#不使用-mim-安装-mmengine)
    - [在 CPU 环境中安装](#在-cpu-环境中安装)
    - [在 Google Colab 中安装](#在-google-colab-中安装)
    - [通过 Docker 使用 MMPose](#通过-docker-使用-mmpose)
  - [故障解决](#故障解决)

## 依赖环境

在本节中，我们将演示如何准备 PyTorch 相关的依赖环境。

MMPose 适用于 Linux、Windows 和 macOS。它需要 Python 3.7+、CUDA 9.2+ 和 PyTorch 1.8+。

如果您对配置 PyTorch 环境已经很熟悉，并且已经完成了配置，可以直接进入下一节：[安装](#安装-mmpose)。否则，请依照以下步骤完成配置。

**第 1 步** 从[官网](https://docs.conda.io/en/latest/miniconda.html) 下载并安装 Miniconda。

**第 2 步** 创建一个 conda 虚拟环境并激活它。

```shell
```

**第 3 步** 按照[官方指南](https://pytorch.org/get-started/locally/) 安装 PyTorch。例如：

在 GPU 平台：

```shell

```shell
```

在 GPU 平台：

```shell
```

以上命令会自动安装最新版的 PyTorch 与对应的 cudatoolkit，请检查它们是否与您的环境匹配。
```

在 CPU 平台：

```shell
```

**第 4 步** 使用 [MIM](https://github.com/open-mmlab/mim) 安装 [MMEngine](https://github.com/open-mmlab/mmengine) 和 [MMCV](https://github.com/open-mmlab/mmcv/tree/2.x)

```shell
mim install "mmcv>=2.0.1"
```

请注意，MMPose 中的一些推理示例脚本需要使用 [MMDetection](https://github.com/open-mmlab/mmdetection) (mmdet) 检测人体。如果您想运行这些示例脚本，可以通过运行以下命令安装 mmdet:

```shell
mim install "mmdet>=3.1.0"
```

## 最佳实践

根据具体需求，我们支持两种安装模式: 从源码安装（推荐）和作为 Python 包安装

### 从源码安装（推荐）

如果基于 MMPose 框架开发自己的任务，需要添加新的功能，比如新的模型或是数据集，或者使用我们提供的各种工具。从源码按如下方式安装 mmpose：

```shell

```shell
```

## 最佳实践

```shell
cd mmpose
# "-v" 表示输出更多安装相关的信息
# "-e" 表示以可编辑形式安装，这样可以在不重新安装的情况下，让本地修改直接生效
```

### 作为 Python 包安装

如果只是希望调用 MMPose 的接口，或者在自己的项目中导入 MMPose 中的模块。直接使用 mim 安装即可。

```shell
mim install "mmpose>=1.1.0"
```

## 验证安装

为了验证 MMPose 是否安装正确，您可以通过以下步骤运行模型推理。

**第 1 步** 我们需要下载配置文件和模型权重文件

```shell
mim download mmpose --config td-hm_hrnet-w48_8xb32-210e_coco-256x192  --dest .
```

下载过程往往需要几秒或更多的时间，这取决于您的网络环境。完成之后，您会在当前目录下找到这两个文件：`td-hm_hrnet-w48_8xb32-210e_coco-256x192.py` 和 `hrnet_w48_coco_256x192-b9e0b3ab_20200708.pth`, 分别是配置文件和对应的模型权重文件。

**第 2 步** 验证推理示例

如果您是**从源码安装**的 mmpose，可以直接运行以下命令进行验证：

```shell

```shell
    td-hm_hrnet-w48_8xb32-210e_coco-256x192.py \
    hrnet_w48_coco_256x192-b9e0b3ab_20200708.pth \
```

如果一切顺利，您将会得到这样的可视化结果：

![image](https://user-images.githubusercontent.com/87690686/187824033-2cce0f55-034a-4127-82e2-52744178bc32.jpg)

代码会将预测的关键点和热图绘制在图像中的人体上，并保存到当前文件夹下的 `vis_results.jpg`。

如果您是**作为 Python 包安装**，可以打开您的 Python 解释器，复制并粘贴如下代码：

```python
from mmpose.apis import inference_topdown, init_model
from mmpose.utils import register_all_modules

register_all_modules()

config_file = 'td-hm_hrnet-w48_8xb32-210e_coco-256x192.py'
checkpoint_file = 'hrnet_w48_coco_256x192-b9e0b3ab_20200708.pth'
model = init_model(config_file, checkpoint_file, device='cpu')  # or device='cuda:0'

# 请准备好一张带有人体的图片
results = inference_topdown(model, 'demo.jpg')
```

```

示例图片 `demo.jpg` 可以从 [Github](https://raw.githubusercontent.com/open-mmlab/mmpose/main/tests/data/coco/000000000785.jpg) 下载。
推理结果是一个 `PoseDataSample` 列表，预测结果将会保存在 `pred_instances` 中，包括检测到的关键点位置和置信度。

## 自定义安装

### CUDA 版本

安装 PyTorch 时，需要指定 CUDA 版本。如果您不清楚选择哪个，请遵循我们的建议：

- 对于 Ampere 架构的 NVIDIA GPU，例如 GeForce 30 系列 以及 NVIDIA A100，CUDA 11 是必需的。
- 对于更早的 NVIDIA GPU，CUDA 11 是向后兼容 (backward compatible) 的，但 CUDA 10.2 能够提供更好的兼容性，也更加轻量。

请确保您的 GPU 驱动版本满足最低的版本需求，参阅[这张表](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cuda-major-component-versions__table-cuda-toolkit-driver-versions)。

```{note}
如果按照我们的最佳实践进行安装，CUDA 运行时库就足够了，因为我们提供相关 CUDA 代码的预编译，您不需要进行本地编译。
但如果您希望从源码进行 MMCV 的编译，或是进行其他 CUDA 算子的开发，那么就必须安装完整的 CUDA 工具链，参见
[NVIDIA 官网](https://developer.nvidia.com/cuda-downloads)，另外还需要确保该 CUDA 工具链的版本与 PyTorch 安装时

的配置相匹配（如用 `conda install` 安装 PyTorch 时指定的 cudatoolkit 版本）。
```

### 不使用 MIM 安装 MMEngine

若不使用 mim 安装 MMEngine，请遵循 [ MMEngine 安装指南](https://mmengine.readthedocs.io/zh_CN/latest/get_started/installation.html).

例如，您可以通过以下命令安装 MMEngine:

```shell
```

### 不使用 MIM 安装 MMCV

MMCV 包含 C++ 和 CUDA 扩展，因此其对 PyTorch 的依赖比较复杂。MIM 会自动解析这些
依赖，选择合适的 MMCV 预编译包，使安装更简单，但它并不是必需的。

若不使用 mim 来安装 MMCV，请遵照 [MMCV 安装指南](https://mmcv.readthedocs.io/zh_CN/2.x/get_started/installation.html)。
它需要您用指定 url 的形式手动指定对应的 PyTorch 和 CUDA 版本。

举个例子，如下命令将会安装基于 PyTorch 1.10.x 和 CUDA 11.3 编译的 mmcv。

```shell
pip install 'mmcv>=2.0.1' -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10/index.html
```

### 在 CPU 环境中安装

MMPose 可以仅在 CPU 环境中安装，在 CPU 模式下，您可以完成训练、测试和模型推理等所有操作。

```shell
```

在 CPU 模式下，MMCV 的部分功能将不可用，通常是一些 GPU 编译的算子，如 `Deformable Convolution`。MMPose 中大部分的模型都不会依赖这些算子，但是如果您尝试使用包含这些算子的模型来运行训练、测试或推理，将会报错。

### 在 Google Colab 中安装

[Google Colab](https://colab.research.google.com/) 通常已经包含了 PyTorch 环境，因此我们只需要安装 MMEngine, MMCV 和 MMPose 即可，命令如下：

**第 1 步** 使用 [MIM](https://github.com/open-mmlab/mim) 安装 [MMEngine](https://github.com/open-mmlab/mmengine) 和 [MMCV](https://github.com/open-mmlab/mmcv/tree/2.x)

```shell
!mim install "mmcv>=2.0.1"
```

**第 2 步** 从源码安装 mmpose

```shell
!git clone https://github.com/open-mmlab/mmpose.git
%cd mmpose
```

**第 3 步** 验证

```python
import mmpose
print(mmpose.__version__)
# 预期输出： 1.1.0
```

```{note}

```shell
%cd mmpose
```

**第 3 步** 验证

```python
```

```{note}
在 Jupyter 中，感叹号 `!` 用于执行外部命令，而 `%cd` 是一个[魔术命令](https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-cd)，用于切换 Python 的工作路径。
```

### 通过 Docker 使用 MMPose

MMPose 提供 [Dockerfile](https://github.com/open-mmlab/mmpose/blob/master/docker/Dockerfile)
用于构建镜像。请确保您的 [Docker 版本](https://docs.docker.com/engine/install/) >=19.03。

```shell
# 构建默认的 PyTorch 1.8.0，CUDA 10.1 版本镜像
# 如果您希望使用其他版本，请修改 Dockerfile
docker build -t mmpose docker/
```

**注意**：请确保您已经安装了 [nvidia-container-toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker)。

```shell
docker run --gpus all --shm-size=8g -it -v {DATA_DIR}:/mmpose/data mmpose
```

`{DATA_DIR}` 是您本地存放用于 MMPose 训练、测试、推理等流程的数据目录。

## 故障解决

如果您在安装过程中遇到了什么问题，请先查阅[常见问题](faq.md)。如果没有找到解决方法，可以在 GitHub

```

## 故障解决

上[提出 issue](https://github.com/open-mmlab/mmpose/issues/new/choose)。

<details>

```bibtex
}
```

<details>

```bibtex

}
```

<details>

```bibtex
}
```

}
```

| [pose_hrnet_w32](/configs/body_2d_keypoint/topdown_heatmap/mpii/cspnext-m_udp_8xb64-210e_mpii-256x256.py) |  256x256   | 0.902 |  0.303   | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-m_udp-mpii_pt-in1k_210e-256x256-68d0402f_20230208.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-m_udp-mpii_pt-in1k_210e-256x256-68d0402f_20230208.json) |

# 准备 HIDE 数据集

```bibtex
}
```

测试数据集可以从 [此处](https://drive.google.com/file/d/1XRomKYJF1H92g1EuD06pCQe4o6HlwB7A/) 下载。

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── HIDE
```

# SOLO

> [SOLO: Segmenting Objects by Locations](https://arxiv.org/abs/1912.04488)

We present a new, embarrassingly simple approach to instance segmentation in images. Compared to many other dense prediction tasks, e.g., semantic segmentation, it is the arbitrary number of instances that have made instance segmentation much more challenging. In order to predict a mask for each instance, mainstream approaches either follow the 'detect-thensegment' strategy as used by Mask R-CNN, or predict category masks first then use clustering techniques to group pixels into individual instances. We view the task of instance segmentation from a completely new perspective by introducing the notion of "instance categories", which assigns categories to each pixel within an instance according to the instance's location and size, thus nicely converting instance mask segmentation into a classification-solvable problem. Now instance segmentation is decomposed into two classification tasks. We demonstrate a much simpler and flexible instance segmentation framework with strong performance, achieving on par accuracy with Mask R-CNN and outperforming recent singleshot instance segmenters in accuracy. We hope that this very simple and strong framework can serve as a baseline for many instance-level recognition tasks besides instance segmentation.

<img src="https://user-images.githubusercontent.com/40661020/143998371-10e6f14b-4506-481d-91a7-5f8f58213307.png"/>
</div>

### SOLO

| Backbone |  Style  | MS train | Lr schd | Mem (GB) | Inf time (fps) | mask AP |                                                                                                                                Download                                                                                                                                |
| :------: | :-----: | :------: | :-----: | :------: | :------------: | :-----: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

|   R-50   | pytorch |    N     |   1x    |   8.0    |      14.0      |  33.1   | [model](https://download.openmmlab.com/mmdetection/v2.0/solo/solo_r50_fpn_1x_coco/solo_r50_fpn_1x_coco_20210821_035055-2290a6b8.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/solo/solo_r50_fpn_1x_coco/solo_r50_fpn_1x_coco_20210821_035055.log.json) |
|   R-50   | pytorch |    Y     |   3x    |   7.4    |      14.0      |  35.9   | [model](https://download.openmmlab.com/mmdetection/v2.0/solo/solo_r50_fpn_3x_coco/solo_r50_fpn_3x_coco_20210901_012353-11d224d7.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/solo/solo_r50_fpn_3x_coco/solo_r50_fpn_3x_coco_20210901_012353.log.json) |

### Decoupled SOLO

| Backbone |  Style  | MS train | Lr schd | Mem (GB) | Inf time (fps) | mask AP |                                                                                                                                                    Download                                                                                                                                                    |
| :------: | :-----: | :------: | :-----: | :------: | :------------: | :-----: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

|   R-50   | pytorch |    N     |   1x    |   7.8    |      12.5      |  33.9   | [model](https://download.openmmlab.com/mmdetection/v2.0/solo/decoupled_solo_r50_fpn_1x_coco/decoupled_solo_r50_fpn_1x_coco_20210820_233348-6337c589.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/solo/decoupled_solo_r50_fpn_1x_coco/decoupled_solo_r50_fpn_1x_coco_20210820_233348.log.json) |
|   R-50   | pytorch |    Y     |   3x    |   7.9    |      12.5      |  36.7   | [model](https://download.openmmlab.com/mmdetection/v2.0/solo/decoupled_solo_r50_fpn_3x_coco/decoupled_solo_r50_fpn_3x_coco_20210821_042504-7b3301ec.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/solo/decoupled_solo_r50_fpn_3x_coco/decoupled_solo_r50_fpn_3x_coco_20210821_042504.log.json) |

- Decoupled SOLO has a decoupled head which is different from SOLO head.
  Decoupled SOLO serves as an efficient and equivalent variant in accuracy
  of SOLO. Please refer to the corresponding config files for details.

Decoupled SOLO serves as an efficient and equivalent variant in accuracy

### Decoupled Light SOLO

| Backbone |  Style  | MS train | Lr schd | Mem (GB) | Inf time (fps) | mask AP |                                                                                                                                                                Download                                                                                                                                                                |
| :------: | :-----: | :------: | :-----: | :------: | :------------: | :-----: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

|   R-50   | pytorch |    Y     |   3x    |   2.2    |      31.2      |  32.9   | [model](https://download.openmmlab.com/mmdetection/v2.0/solo/decoupled_solo_light_r50_fpn_3x_coco/decoupled_solo_light_r50_fpn_3x_coco_20210906_142703-e70e226f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/solo/decoupled_solo_light_r50_fpn_3x_coco/decoupled_solo_light_r50_fpn_3x_coco_20210906_142703.log.json) |

- Decoupled Light SOLO using decoupled structure similar to Decoupled
  SOLO head, with light-weight head and smaller input size, Please refer
  to the corresponding config files for details.

```latex
@inproceedings{wang2020solo,
  title     =  {{SOLO}: Segmenting Objects by Locations},
  author    =  {Wang, Xinlong and Kong, Tao and Shen, Chunhua and Jiang, Yuning and Li, Lei},
}
```

<details>

```bibtex
}
```

<details>

```bibtex

}
```

| [pose_hrnetv2_w18](/configs/hand_2d_keypoint/topdown_heatmap/coco_wholebody_hand/td-hm_hrnetv2-w18_8xb32-210e_coco-wholebody-hand-256x256.py) |  256x256   |  0.813  | 0.840 | 4.39 | [ckpt](https://download.openmmlab.com/mmpose/hand/hrnetv2/hrnetv2_w18_coco_wholebody_hand_256x256-1c028db7_20210908.pth) | [log](https://download.openmmlab.com/mmpose/hand/hrnetv2/hrnetv2_w18_coco_wholebody_hand_256x256_20210908.log.json) |

<details>

```bibtex
}
```

<details>

```bibtex
}
```

}
```

| [pose_resnet_50](/configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_res50_8xb64-210e_coco-wholebody-256x192.py) |  256x192   |  0.652  |  0.738  |  0.615  |  0.749  |  0.606  |  0.715  |  0.460  |  0.584  |  0.521   |  0.633   | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnet/res50_coco_wholebody_256x192-9e37ed88_20201004.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnet/res50_coco_wholebody_256x192_20201004.log.json) |
| [pose_resnet_50](/configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_res50_8xb64-210e_coco-wholebody-384x288.py) |  384x288   |  0.666  |  0.747  |  0.634  |  0.763  |  0.731  |  0.811  |  0.536  |  0.646  |  0.574   |  0.670   | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnet/res50_coco_wholebody_384x288-ce11e294_20201004.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnet/res50_coco_wholebody_384x288_20201004.log.json) |

| [pose_resnet_101](/configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_res101_8xb32-210e_coco-wholebody-256x192.py) |  256x192   |  0.669  |  0.753  |  0.637  |  0.766  |  0.611  |  0.722  |  0.463  |  0.589  |  0.531   |  0.645   | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnet/res101_coco_wholebody_256x192-7325f982_20201004.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnet/res101_coco_wholebody_256x192_20201004.log.json) |
| [pose_resnet_101](/configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_res101_8xb32-210e_coco-wholebody-384x288.py) |  384x288   |  0.692  |  0.770  |  0.680  |  0.799  |  0.746  |  0.820  |  0.548  |  0.657  |  0.597   |  0.693   | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnet/res101_coco_wholebody_384x288-6c137b9a_20201004.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnet/res101_coco_wholebody_384x288_20201004.log.json) |

| [pose_resnet_152](/configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_res152_8xb32-210e_coco-wholebody-256x192.py) |  256x192   |  0.682  |  0.764  |  0.661  |  0.787  |  0.623  |  0.728  |  0.481  |  0.607  |  0.548   |  0.661   | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnet/res152_coco_wholebody_256x192-5de8ae23_20201004.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnet/res152_coco_wholebody_256x192_20201004.log.json) |
| [pose_resnet_152](/configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_res152_8xb32-210e_coco-wholebody-384x288.py) |  384x288   |  0.704  |  0.780  |  0.693  |  0.813  |  0.751  |  0.824  |  0.559  |  0.666  |  0.610   |  0.705   | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnet/res152_coco_wholebody_384x288-eab8caa8_20201004.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnet/res152_coco_wholebody_384x288_20201004.log.json) |

# Quasi-Dense Similarity Learning for Multiple Object Tracking

Similarity learning has been recognized as a crucial step for object tracking. However, existing multiple object tracking methods only use sparse ground truth matching as the training objective, while ignoring the majority of the informative regions on the images. In this paper, we present Quasi-Dense Similarity Learning, which densely samples hundreds of region proposals on a pair of images for contrastive learning. We can directly combine this similarity learning with existing detection methods to build Quasi-Dense Tracking (QDTrack) without turning to displacementregression or motion priors. We also find that the resulting distinctive feature space admits a simple nearest neighbor search at the inference time. Despite its simplicity, QD-Track outperforms all existing methods on MOT, BDD100K, Waymo, and TAO tracking benchmarks. It achieves 68.7 MOTA at 20.3 FPS on MOT17 without using external training data. Compared to methods with similar detectors, it boosts almost 10 points of MOTA and significantly decreases the number of ID switches on BDD100K and Waymo datasets.

  <img src="https://user-images.githubusercontent.com/48645550/158332287-79fb379b-d817-4aa8-8530-5f9d172b3ca7.png"/>
  <img src="https://user-images.githubusercontent.com/48645550/158332524-8ccaab0e-d379-4c6b-83e5-d75398af02bf.png"/>
</div>

| Method  |   Detector   | Train Set  | Test Set | Public | Inf time (fps) | HOTA | MOTA | IDF1 |  FP  |  FN   | IDSw. |                                      Config                                       |                                                                                                                                        Download                                                                                                                                        |

| :-----: | :----------: | :--------: | :------: | :----: | :------------: | :--: | :--: | :--: | :--: | :---: | :---: | :-------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| QDTrack | Faster R-CNN | half-train | half-val |   N    |       -        | 57.1 | 68.1 | 68.6 | 7707 | 42732 | 1083  | [config](qdtrack_faster-rcnn_r50_fpn_8xb2-4e_mot17halftrain_test-mot17halfval.py) | [model](https://download.openmmlab.com/mmtracking/mot/qdtrack/mot_dataset/qdtrack_faster-rcnn_r50_fpn_4e_mot17_20220315_145635-76f295ef.pth) \| [log](https://download.openmmlab.com/mmtracking/mot/qdtrack/mot_dataset/qdtrack_faster-rcnn_r50_fpn_4e_mot17_20220315_145635.log.json) |

```shell
# Training QDTrack on mot17-half-train dataset with following command.
bash tools/dist_train.sh configs/qdtrack/qdtrack_faster-rcnn_r50_fpn_8xb2-4e_mot17halftrain_test-mot17halfval.py 8
```

```

**4.1 Example on MOTxx-halfval dataset**

```shell
# Example 1: Test on motXX-half-val set
bash tools/dist_test_tracking.sh configs/qdtrack/qdtrack_faster-rcnn_r50_fpn_8xb2-4e_mot17halftrain_test-mot17halfval.py 8 --checkpoint ${CHECKPOINT_PATH}
```

**4.2 use video_baesd to evaluating and testing**
we also provide two_ways(img_based or video_based) to evaluating and testing.
if you want to use video_based to evaluating and testing, you can modify config as follows

```
    sampler=dict(type='DefaultSampler', shuffle=False, round_up=False))
```

If you want to know about more detailed usage of `test_tracking.py/dist_test_tracking.sh/slurm_test_tracking.sh`,

sampler=dict(type='DefaultSampler', shuffle=False, round_up=False))
```

```shell
python demo/mot_demo.py demo/demo_mot.mp4 configs/qdtrack/qdtrack_faster-rcnn_r50_fpn_8xb2-4e_mot17halftrain_test-mot17halfval.py --checkpoint ${CHECKPOINT_PATH} --out mot.mp4
```

```latex
@inproceedings{pang2021quasi,
  title={Quasi-dense similarity learning for multiple object tracking},
  author={Pang, Jiangmiao and Qiu, Linlu and Li, Xia and Chen, Haofeng and Li, Qi and Darrell, Trevor and Yu, Fisher},

title={Quasi-dense similarity learning for multiple object tracking},
  pages={164--173},
}
```

<details>

```bibtex
}
```

<details>

```bibtex
}
```

| [rtmpose_m](/configs/hand_2d_keypoint/rtmpose/coco_wholebody_hand/rtmpose-m_8xb32-210e_coco-wholebody-hand-256x256.py) |  256x256   |  0.815  | 0.837 | 4.51 | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-coco-wholebody-hand_pt-aic-coco_210e-256x256-99477206_20230228.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-coco-wholebody-hand_pt-aic-coco_210e-256x256-99477206_20230228.json) |

## 安装

```shell

```shell
```

**说明**:

**方式三：** 源码安装

## 模型转换

```shell

```shell

    --show \
```

文件的命名模式是：

```
```

其中：

文件的命名模式是：

```
```

其中：

```{tip}
```

## 模型规范

```
```

重要的是：

## 模型推理

### 后端模型推理

```python

## 模型推理

### 后端模型推理

```python

```

```python
import cv2

```

## 模型支持列表

| [SegFormer](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/segformer)[\*](#static_shape)  |      Y      |      Y      |    Y     |  N   |   N   |    Y     |

## 注意事项

## 注意事项

# Data Transforms and Pipeline

In the design of MMOCR, dataset construction and preparation are decoupled. That is, dataset construction classes such as [`OCRDataset`](mmocr.datasets.ocr_dataset.OCRDataset) are responsible for loading and parsing annotation files; while data transforms further apply data preprocessing, augmentation, formatting, and other related functions. Currently, there are five types of data transforms implemented in MMOCR, as shown in the following table.

|                                  |                                                                       |                                                                                                     |
| -------------------------------- | --------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |

| Transforms Type                  | File                                                                  | Description                                                                                         |
| Data Loading                     | loading.py                                                            | Implemented the data loading functions.                                                             |
| Data Formatting                  | formatting.py                                                         | Formatting the data required by different tasks.                                                    |
| Cross Project Data Adapter       | adapters.py                                                           | Converting the data format between other OpenMMLab projects and MMOCR.                              |

| Data Augmentation Functions      | ocr_transforms.py<br>textdet_transforms.py<br>textrecog_transforms.py | Various built-in data augmentation methods designed for different tasks.                            |
| Wrappers of Third Party Packages | wrappers.py                                                           | Wrapping the transforms implemented in popular third party packages such as [ImgAug](https://github.com/aleju/imgaug), and adapting them to MMOCR format. |

Since each data transform class is independent of each other, we can easily combine any data transforms to build a data pipeline after we have defined the data fields. As shown in the following figure, in MMOCR, a typical training data pipeline consists of three stages: **data loading**, **data augmentation**, and **data formatting**. Users only need to define the data pipeline list in the configuration file and specify the specific data transform class and its parameters:

![Flowchart](https://user-images.githubusercontent.com/45810070/192265787-86c02e51-5f4a-4686-b2a1-255a7db98ada.jpg)

</div>

```python
    # Loading images
    dict(
    # Loading annotations
    dict(
    ),
    # Data augmentation
    dict(
    # Data formatting
    dict(
]
```

```{tip}

dict(type='Pad', size=(640, 640)),
    dict(
]
```

```{tip}
More tutorials about data pipeline configuration can be found in the [Config Doc](../user_guides/config.md#data-pipeline-configuration). Next, we will briefly introduce the data transforms supported in MMOCR according to their categories.
```

```

For each data transform, MMOCR provides a detailed docstring. For example, in the header of each data transform class, we annotate `Required Keys`, `Modified Keys` and `Added Keys`. The `Required Keys` represent the mandatory fields that should be included in the input required by the data transform, while the `Modified Keys` and `Added Keys` indicate that the transform may modify or add the fields into the original data. For example, `LoadImageFromFile` implements the image loading function, whose `Required Keys` is the image path `img_path`, and the `Modified Keys` includes the loaded image `img`, the current size of the image `img_shape`, the original size of the image `ori_shape`, and other image attributes.

```python
    # We provide detailed docstring for each data transform.

    - img_path

    - img

# We provide detailed docstring for each data transform.

    - img_path

    - img
    """
```

```{note}
In the data pipeline of MMOCR, the image and label information are saved in a dictionary. By using the unified fields, the data can be freely transferred between different data transforms. Therefore, it is very important to understand the conventional fields used in MMOCR.
```

For your convenience, the following table lists the conventional keys used in MMOCR data transforms.

|                  |                                   |                                                                                                                                                         |

| ---------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Key              | Type                              | Description                                                                                                                                             |
| img              | `np.array(dtype=np.uint8)`        | Image array, shape of `(h, w, c)`.                                                                                                                      |
| img_shape        | `tuple(int, int)`                 | Current image size `(h, w)`.                                                                                                                            |

| ori_shape        | `tuple(int, int)`                 | Original image size `(h, w)`.                                                                                                                           |
| scale            | `tuple(int, int)`                 | Stores the target image size `(h, w)` specified by the user in the `Resize` data transform series. Note: This value may not correspond to the actual image size after the transformation. |
| scale_factor     | `tuple(float, float)`             | Stores the target image scale factor `(w_scale, h_scale)` specified by the user in the `Resize` data transform series. Note: This value may not correspond to the actual image size after the transformation. |
| keep_ratio       | `bool`                            | Boolean flag determines whether to keep the aspect ratio while scaling images.                                                                          |

| flip             | `bool`                            | Boolean flags to indicate whether the image has been flipped.                                                                                           |
| flip_direction   | `str`                             | Flipping direction, options are `horizontal`, `vertical`, `diagonal`.                                                                                   |
| gt_bboxes        | `np.array(dtype=np.float32)`      | Ground-truth bounding boxes.                                                                                                                            |
| gt_polygons      | `list[np.array(dtype=np.float32)` | Ground-truth polygons.                                                                                                                                  |

| gt_bboxes_labels | `np.array(dtype=np.int64)`        | Category label of bounding boxes. By default, MMOCR uses `0` to represent "text" instances.                                                             |
| gt_texts         | `list[str]`                       | Ground-truth text content of the instance.                                                                                                              |
| gt_ignored       | `np.array(dtype=np.bool_)`        | Boolean flag indicating whether ignoring the instance (used in text detection).                                                                         |

## Data Loading

Data loading transforms mainly implement the functions of loading data from different formats and backends. Currently, the following data loading transforms are implemented in MMOCR:

|                    |                                                           |                                                                |                                                                 |
| Transforms Name    | Required Keys                                             | Modified/Added Keys                                            | Description                                                     |

| LoadImageFromFile  | `img_path`                                                | `img`<br>`img_shape`<br>`ori_shape`                            | Load image from the specified path，supporting different file storage backends (e.g. `disk`, `http`, `petrel`) and decoding backends (e.g. `cv2`, `turbojpeg`, `pillow`, `tifffile`). |
| LoadOCRAnnotations | `bbox`<br>`bbox_label`<br>`polygon`<br>`ignore`<br>`text` | `gt_bboxes`<br>`gt_bboxes_labels`<br>`gt_polygons`<br>`gt_ignored`<br>`gt_texts` | Parse the annotation required by OCR task.                      |
| LoadKIEAnnotations | `bboxes` `bbox_labels` `edge_labels`<br>`texts`           | `gt_bboxes`<br>`gt_bboxes_labels`<br>`gt_edge_labels`<br>`gt_texts`<br>`ori_shape` | Parse the annotation required by KIE task.                      |

## Data Augmentation

Data augmentation is an indispensable process in text detection and recognition tasks. Currently, MMOCR has implemented dozens of data augmentation modules commonly used in OCR fields, which are classified into [ocr_transforms.py](/mmocr/datasets/transforms/ocr_transforms.py), [textdet_transforms.py](/mmocr/datasets/transforms/textdet_transforms.py), and [textrecog_transforms.py](/mmocr/datasets/transforms/textrecog_transforms.py).

Specifically, `ocr_transforms.py` implements generic OCR data augmentation modules such as `RandomCrop` and `RandomRotate`:

|                 |                                                               |                                                                |                                                                |

| --------------- | ------------------------------------------------------------- | -------------------------------------------------------------- | -------------------------------------------------------------- |
| Transforms Name | Required Keys                                                 | Modified/Added Keys                                            | Description                                                    |
| RandomCrop      | `img`<br>`gt_bboxes`<br>`gt_bboxes_labels`<br>`gt_polygons`<br>`gt_ignored`<br>`gt_texts` (optional) | `img`<br>`img_shape`<br>`gt_bboxes`<br>`gt_bboxes_labels`<br>`gt_polygons`<br>`gt_ignored`<br>`gt_texts` (optional) | Randomly crop the image and make sure the cropped image contains at least one text instance. The optional parameter is `min_side_ratio`, which controls the ratio of the short side of the cropped image to the original image, the default value is `0.4`. |

| RandomRotate    | `img`<br>`img_shape`<br>`gt_bboxes` (optional)<br>`gt_polygons` (optional) | `img`<br>`img_shape`<br>`gt_bboxes` (optional)<br>`gt_polygons` (optional)<br>`rotated_angle` | Randomly rotate the image and optionally fill the blank areas of the rotated image. |
|                 |                                                               |                                                                |                                                                |

`textdet_transforms.py` implements text detection related data augmentation modules:

|                   |                                       |                                                                     |                                                                                 |

| ----------------- | ------------------------------------- | ------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| Transforms Name   | Required Keys                         | Modified/Added Keys                                                 | Description                                                                     |
| RandomFlip        | `img`<br>`gt_bboxes`<br>`gt_polygons` | `img`<br>`gt_bboxes`<br>`gt_polygons`<br>`flip`<br>`flip_direction` | Random flip, support `horizontal`, `vertical` and `diagonal` modes. Defaults to `horizontal`. |
| FixInvalidPolygon | `gt_polygons`<br>`gt_ignored`         | `gt_polygons`<br>`gt_ignored`                                       | Automatically fixing the invalid polygons included in the annotations.          |

`textrecog_transforms.py` implements text recognition related data augmentation modules:

|                 |               |                                                                   |                                                                                                             |
| --------------- | ------------- | ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| Transforms Name | Required Keys | Modified/Added Keys                                               | Description                                                                                                 |
| RescaleToHeight | `img`         | `img`<br>`img_shape`<br>`scale`<br>`scale_factor`<br>`keep_ratio` | Scales the image to the specified height while keeping the aspect ratio. When `min_width` and `max_width` are specified, the aspect ratio may be changed. |

|                 |               |                                                                   |                                                                                                             |

The above table only briefly introduces some selected data augmentation methods, for more information please refer to the [API documentation](../api.rst) or the code docstrings.
```

## Data Formatting

Data formatting transforms are responsible for packaging images, ground truth labels, and other information into a dictionary. Different tasks usually rely on different formatting transforms. For example:

|                     |               |                     |                                               |
| ------------------- | ------------- | ------------------- | --------------------------------------------- |
| Transforms Name     | Required Keys | Modified/Added Keys | Description                                   |

| PackTextDetInputs   | -             | -                   | Pack the inputs required by text detection.   |
| PackTextRecogInputs | -             | -                   | Pack the inputs required by text recognition. |
| PackKIEInputs       | -             | -                   | Pack the inputs required by KIE.              |

## Cross Project Data Adapters

The cross-project data adapters bridge the data formats between MMOCR and other OpenMMLab libraries such as [MMDetection](https://github.com/open-mmlab/mmdetection), making it possible to call models implemented in other OpenMMLab projects. Currently, MMOCR has implemented [`MMDet2MMOCR`](mmocr.datasets.transforms.MMDet2MMOCR) and [`MMOCR2MMDet`](mmocr.datasets.transforms.MMOCR2MMDet), allowing data to be converted between MMDetection and MMOCR formats; with these adapters, users can easily train any detectors supported by MMDetection in MMOCR. For example, we provide a [tutorial](#todo) to show how to train Mask R-CNN as a text detector in MMOCR.

|                 |                                              |                               |                                            |

|                 |                                              |                               |                                            |
| --------------- | -------------------------------------------- | ----------------------------- | ------------------------------------------ |
| Transforms Name | Required Keys                                | Modified/Added Keys           | Description                                |
| MMDet2MMOCR     | `gt_masks` `gt_ignore_flags`                 | `gt_polygons`<br>`gt_ignored` | Convert the fields used in MMDet to MMOCR. |
| MMOCR2MMDet     | `img_shape`<br>`gt_polygons`<br>`gt_ignored` | `gt_masks` `gt_ignore_flags`  | Convert the fields used in MMOCR to MMDet. |

## Wrappers

To facilitate the use of popular third-party CV libraries in MMOCR, we provide wrappers in `wrappers.py` to unify the data format between MMOCR and other third-party libraries. Users can directly configure the data transforms provided by these libraries in the configuration file of MMOCR. The supported wrappers are as follows:

|                    |                                                              |                                                               |                                                               |

| Transforms Name    | Required Keys                                                | Modified/Added Keys                                           | Description                                                   |
| ImgAugWrapper      | `img`<br>`gt_polygons` (optional for text recognition)<br>`gt_bboxes` (optional for text recognition)<br>`gt_bboxes_labels` (optional for text recognition)<br>`gt_ignored` (optional for text recognition)<br>`gt_texts` (optional) | `img`<br>`gt_polygons` (optional for text recognition)<br>`gt_bboxes` (optional for text recognition)<br>`gt_bboxes_labels` (optional for text recognition)<br>`gt_ignored` (optional for text recognition)<br>`img_shape` (optional)<br>`gt_texts` (optional) | [ImgAug](https://github.com/aleju/imgaug) wrapper, which bridges the data format and configuration between ImgAug and MMOCR, allowing users to config the data augmentation methods supported by ImgAug in MMOCR. |

| TorchVisionWrapper | `img`                                                        | `img`<br>`img_shape`                                          | [TorchVision](https://github.com/pytorch/vision) wrapper, which bridges the data format and configuration between TorchVision and MMOCR, allowing users to config the data transforms supported by `torchvision.transforms` in MMOCR. |

### `ImgAugWrapper` Example

For example, in the original ImgAug, we can define a `Sequential` type data augmentation pipeline as follows to perform random flipping, random rotation and random scaling on the image:

```python

  iaa.Fliplr(0.5),                # horizontally flip 50% of all images
  iaa.Affine(rotate=(-10, 10)),   # rotate by -10 to +10 degrees
  iaa.Resize((0.5, 3.0))          # scale images to 50-300% of their size
)
```

In MMOCR, we can directly configure the above data augmentation pipeline in `train_pipeline` as follows:

```python

iaa.Resize((0.5, 3.0))          # scale images to 50-300% of their size
)
```

```python
dict(
  args=[
  ]
)
```

Specifically, the `args` parameter accepts a list, and each element in the list can be a list or a dictionary. If it is a list, the first element of the list is the class name in `imgaug.augmenters`, and the following elements are the initialization parameters of the class; if it is a dictionary, the `cls` key corresponds to the class name in `imgaug.augmenters`, and the other key-value pairs correspond to the initialization parameters of the class.

### `TorchVisionWrapper` Example

For example, in the original TorchVision, we can define a `Compose` type data transformation pipeline as follows to perform color jittering on the image:

```python

```python

    brightness=32.0 / 255,  # brightness jittering range
    saturation=0.5)         # saturation jittering range
])
```

In MMOCR, we can directly configure the above data transformation pipeline in `train_pipeline` as follows:

```python
dict(
)
```

Specifically, the `op` parameter is the class name in `torchvision.transforms`, and the following parameters correspond to the initialization parameters of the class.

# CoNIC: Colon Nuclei Identification and Counting Challenge

This project supports **`CoNIC: Colon Nuclei Identification and Counting Challenge`**, which can be downloaded from [here](https://drive.google.com/drive/folders/1il9jG7uA4-ebQ_lNmXbbF2eOK9uNwheb).

Nuclear segmentation, classification and quantification within Haematoxylin & Eosin stained histology images enables the extraction of interpretable cell-based features that can be used in downstream explainable models in computational pathology (CPath). To help drive forward research and innovation for automatic nuclei recognition in CPath, we organise the Colon Nuclei Identification and Counting (CoNIC) Challenge. The challenge requires researchers to develop algorithms that perform segmentation, classification and counting of 6 different types of nuclei within the current largest known publicly available nuclei-level dataset in CPath, containing around half a million labelled nuclei.

### Task Information

The CONIC challenge has 2 tasks:

- Task 1: Nuclear segmentation and classification.

The first task requires participants to segment nuclei within the tissue, while also classifying each nucleus into one of the following categories: epithelial, lymphocyte, plasma, eosinophil, neutrophil or connective tissue.

- Task 2: Prediction of cellular composition.

For the second task, we ask participants to predict how many nuclei of each class are present in each input image.

The output of Task 1 can be directly used to perform Task 2, but these can be treated as independent tasks. Therefore, if it is preferred, prediction of cellular composition can be treated as a stand alone regression task.

***NOTE：We only consider `Task 1` in the following sections.***

| Dataset name                                             | Anatomical region | Task type    | Modality       | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License                                                                                                      |
| -------------------------------------------------------- | ----------------- | ------------ | -------------- | ------------ | --------------------- | ---------------------- | ------------ | ------------------------------------------------------------------------------------------------------------ |

| [CoNIC202](https://conic-challenge.grand-challenge.org/) | abdomen           | segmentation | histopathology | 7            | 4981/-/-              | yes/-/-                | 2022         | [Attribution-NonCommercial-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-nc-sa/4.0/) |

| background |    4981    |   83.97    |    -     |    -     |     -     |     -     |
| neutrophil |    1218    |    0.13    |    -     |    -     |     -     |     -     |
| epithelial |    4256    |   10.31    |    -     |    -     |     -     |     -     |
| lymphocyte |    4473    |    1.85    |    -     |    -     |     -     |     -     |
|   plasma   |    3316    |    0.55    |    -     |    -     |     -     |     -     |
| eosinophil |    1456    |    0.1     |    -     |    -     |     -     |     -     |

| connective |    4613    |    3.08    |    -     |    -     |     -     |     -     |

Note:

![bac](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/histopathology/conic2022_seg/conic2022_seg_dataset.png)

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `conic2022_seg/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
```

- download dataset from [here](https://drive.google.com/drive/folders/1il9jG7uA4-ebQ_lNmXbbF2eOK9uNwheb/) and move data to path `'data/CoNIC_Challenge'`. The directory should be like:
  ```shell
  data/CoNIC_Challenge
        ├── README.txt
        ├── by-nc-sa.md
        ├── counts.csv
        ├── images.npy
        ├── labels.npy
        └── patch_info.csv
  ```

```shell
  ```

```none
  ├── mmseg
  │   │   │   │   ├── conic2022_seg

```

| background |    3984    |   84.06    |   997    |  83.65   |     -     |     -     |

| neutrophil |    956     |    0.12    |   262    |   0.13   |     -     |     -     |
| epithelial |    3400    |   10.26    |   856    |  10.52   |     -     |     -     |
| lymphocyte |    3567    |    1.83    |   906    |   1.96   |     -     |     -     |
|   plasma   |    2645    |    0.55    |   671    |   0.56   |     -     |     -     |
| eosinophil |    1154    |    0.1     |   302    |   0.1    |     -     |     -     |
| connective |    3680    |    3.08    |   933    |   3.08   |     -     |     -     |

```shell
```

```shell
```

```

```shell
```

## Organizers

- Simon Graham (TIA, PathLAKE)
- Mostafa Jahanifar (TIA, PathLAKE)
- Dang Vu (TIA)
- Giorgos Hadjigeorghiou (TIA, PathLAKE)
- Thomas Leech (TIA, PathLAKE)
- David Snead (UHCW, PathLAKE)
- Shan Raza (TIA, PathLAKE)
- Fayyaz Minhas (TIA, PathLAKE)
- Nasir Rajpoot (TIA, PathLAKE)

TIA: Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, United Kingdom

UHCW: Department of Pathology, University Hospitals Coventry and Warwickshire, United Kingdom

PathLAKE: Pathology Image Data Lake for Analytics Knowledge & Education, , University Hospitals Coventry and Warwickshire, United Kingdom

```
@inproceedings{graham2021lizard,
  title={Lizard: A large-scale dataset for colonic nuclear instance segmentation and classification},
  author={Graham, Simon and Jahanifar, Mostafa and Azam, Ayesha and Nimir, Mohammed and Tsang, Yee-Wah and Dodd, Katherine and Hero, Emily and Sahota, Harvir and Tank, Atisha and Benes, Ksenija and others},
  pages={684--693},

}
@article{graham2021conic,
  title={Conic: Colon nuclei identification and counting challenge 2022},
  author={Graham, Simon and Jahanifar, Mostafa and Vu, Quoc Dang and Hadjigeorghiou, Giorgos and Leech, Thomas and Snead, David and Raza, Shan E Ahmed and Minhas, Fayyaz and Rajpoot, Nasir},
  journal={arXiv preprint arXiv:2111.14485},
}
```

# RIFormer

> [RIFormer: Keep Your Vision Backbone Effective But Removing Token Mixer](https://arxiv.org/abs/2304.05659)

RIFormer is a way to keep a vision backbone effective while removing token mixers in its basic building blocks. Equipped with our proposed optimization strategy, we are able to build an extremely simple vision backbone with encouraging performance, while enjoying the high efficiency during inference. RIFormer shares nearly the same macro and micro design as MetaFormer, but safely removing all token mixers. The quantitative results show that our networks outperform many prevailing backbones with faster inference speed on ImageNet-1K.

<img src="https://user-images.githubusercontent.com/48375204/223930120-dc075c8e-0513-42eb-9830-469a45c1d941.png" width="65%"/>
</div>

<details>

<br>

This paper studies how to keep a vision backbone effective while removing token mixers in its basic building blocks. Token mixers, as self-attention for vision transformers (ViTs), are intended to perform information communication between different spatial tokens but suffer from considerable computational cost and latency. However, directly removing them will lead to an incomplete model structure prior, and thus brings a significant accuracy drop. To this end, we first develop an RepIdentityFormer base on the re-parameterizing idea, to study the token mixer free model architecture. And we then explore the improved learning paradigm to break the limitation of simple token mixer free backbone, and summarize the empirical practice into 5 guidelines. Equipped with the proposed optimization strategy, we are able to build an extremely simple vision backbone with encouraging performance, while enjoying the high efficiency during inference. Extensive experiments and ablative analysis also demonstrate that the inductive bias of network architecture, can be incorporated into simple network structure with appropriate optimization strategy. We hope this work can serve as a starting point for the exploration of optimization-driven efficient network design.

</br>

## How to use

The checkpoints provided are all `training-time` models. Use the reparameterize tool or `switch_to_deploy` interface to switch them to more efficient `inference-time` architecture, which not only has fewer parameters but also less calculations.

Use `classifier.backbone.switch_to_deploy()` interface to switch the RIFormer models into inference mode.

```python
>>> from mmpretrain import get_model, inference_model
>>>
>>> model = get_model("riformer-s12_in1k", pretrained=True)
>>> results = inference_model(model, 'demo/demo.JPEG')
>>> print( (results['pred_class'], results['pred_score']) )
('sea snake', 0.7827484011650085)
>>>
>>> # switch to deploy mode
>>> model.backbone.switch_to_deploy()
('sea snake', 0.7827480435371399)
```

```python
>>>

```

```python
>>>
>>> model.eval()
>>> inputs = torch.rand(1, 3, 224, 224).to(model.data_preprocessor.device)
>>> # To get classification scores.
>>> out = model(inputs)
>>> print(out.shape)
torch.Size([1, 1000])
>>> # To extract features.
>>> outs = model.extract_feat(inputs)
>>> print(outs[0].shape)
torch.Size([1, 512])
>>>
>>> out_deploy = model(inputs)
>>> assert torch.allclose(out, out_deploy, rtol=1e-4, atol=1e-5) # pass without error
```

Place the ImageNet dataset to the `data/imagenet/` directory, or prepare datasets according to the [docs](https://mmpretrain.readthedocs.io/en/latest/user_guides/dataset_prepare.html#prepare-dataset).

*224×224*

*224×224*

Download Checkpoint:

```shell
wget https://download.openmmlab.com/mmclassification/v1/riformer/riformer-s12_32xb128_in1k_20230406-6741ce71.pth
```

Test use unfused model:

```shell
python tools/test.py configs/riformer/riformer-s12_8xb128_in1k.py riformer-s12_32xb128_in1k_20230406-6741ce71.pth
```

Reparameterize checkpoint:

```shell
python tools/model_converters/reparameterize_model.py configs/riformer/riformer-s12_8xb128_in1k.py riformer-s12_32xb128_in1k_20230406-6741ce71.pth riformer-s12_deploy.pth
```

Test use fused model:

```shell
python tools/test.py configs/riformer/deploy/riformer-s12-deploy_8xb128_in1k.py riformer-s12_deploy.pth
```

For more configurable parameters, please refer to the [API](https://mmpretrain.readthedocs.io/en/latest/api/generated/mmpretrain.models.backbones.RIFormer.html#mmpretrain.models.backbones.RIFormer).

<details>

<summary><b>How to use the reparameterization tool</b>(click to show)</summary>

<br>

<details>

<br>

Use provided tool to reparameterize the given model and save the checkpoint:

```bash
python tools/convert_models/reparameterize_model.py ${CFG_PATH} ${SRC_CKPT_PATH} ${TARGET_CKPT_PATH}
```

`${CFG_PATH}` is the config file path, `${SRC_CKPT_PATH}` is the source chenpoint file path, `${TARGET_CKPT_PATH}` is the target deploy weight file path.

```shell
# download the weight

# reparameterize unfused weight to fused weight
```

To use reparameterized weights, you can use the deploy model config file such as the [s12_deploy example](./deploy/riformer-s12-deploy_8xb128_in1k.py):

```text

```

```text
# in riformer-s12-deploy_8xb128_in1k.py
_base_ = '../deploy/riformer-s12-deploy_8xb128_in1k.py'  # basic s12 config

model = dict(backbone=dict(deploy=True))  # switch model into deploy mode
```

```shell
```

</br>

### ImageNet-1k

|         Model         | resolution | Params(M) | Flops(G) | Top-1 (%) | Top-5 (%) |                    Config                     |                                         Download                                          |
| :-------------------: | :--------: | :-------: | :------: | :-------: | :-------: | :-------------------------------------------: | :---------------------------------------------------------------------------------------: |

|   riformer-s12_in1k   |  224x224   |   11.92   |   1.82   |   76.90   |   93.06   |    [config](./riformer-s12_8xb128_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v1/riformer/riformer-s12_32xb128_in1k_20230406-6741ce71.pth) |
|   riformer-s24_in1k   |  224x224   |   21.39   |   3.41   |   80.28   |   94.80   |    [config](./riformer-s24_8xb128_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v1/riformer/riformer-s24_32xb128_in1k_20230406-fdab072a.pth) |
|   riformer-s36_in1k   |  224x224   |   30.86   |   5.00   |   81.29   |   95.41   |    [config](./riformer-s36_8xb128_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v1/riformer/riformer-s36_32xb128_in1k_20230406-fdfcd3b0.pth) |

|   riformer-m36_in1k   |  224x224   |   56.17   |   8.80   |   82.57   |   95.99   |    [config](./riformer-m36_8xb128_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v1/riformer/riformer-m36_32xb128_in1k_20230406-2fcb9d9b.pth) |
|   riformer-m48_in1k   |  224x224   |   73.47   |  11.59   |   82.75   |   96.11   |    [config](./riformer-m48_8xb64_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v1/riformer/riformer-m48_32xb128_in1k_20230406-2b9d1abf.pth) |
| riformer-s12_384_in1k |  384x384   |   11.92   |   5.36   |   78.29   |   93.93   | [config](./riformer-s12_8xb128_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v1/riformer/riformer-s12_32xb128_in1k-384px_20230406-145eda4c.pth) |

| riformer-s24_384_in1k |  384x384   |   21.39   |  10.03   |   81.36   |   95.40   | [config](./riformer-s24_8xb128_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v1/riformer/riformer-s24_32xb128_in1k-384px_20230406-bafae7ab.pth) |
| riformer-s36_384_in1k |  384x384   |   30.86   |  14.70   |   82.22   |   95.95   | [config](./riformer-s36_8xb64_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v1/riformer/riformer-s36_32xb128_in1k-384px_20230406-017ed3c4.pth) |
| riformer-m36_384_in1k |  384x384   |   56.17   |  25.87   |   83.39   |   96.40   | [config](./riformer-m36_8xb64_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v1/riformer/riformer-m36_32xb128_in1k-384px_20230406-66a6f764.pth) |

| riformer-m48_384_in1k |  384x384   |   73.47   |  34.06   |   83.70   |   96.60   | [config](./riformer-m48_8xb64_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v1/riformer/riformer-m48_32xb128_in1k-384px_20230406-2e874826.pth) |

The config files of these models are only for inference.

```bibtex
@inproceedings{wang2023riformer,
  title={RIFormer: Keep Your Vision Backbone Effective But Removing Token Mixer},
  author={Wang, Jiahao and Zhang, Songyang and Liu, Yong and Wu, Taiqiang and Yang, Yujiu and Liu, Xihui and Chen, Kai and Luo, Ping and Lin, Dahua},
  year={2023}
}
```

# Quick run

```python
```

Or you can just run the following command.

```bash
python demo/mmagic_inference_demo.py \
    --model-name stable_diffusion \
    --text "A panda is having dinner at KFC" \
    --result-out-dir ./output/sd_res.png
```

You will see a new image `sd_res.png` in folder `output/`, which contained generated samples.

What's more, if you want to make these photos much more clear,
you only need several lines of codes for image super-resolution by MMagic!

```python

```python
config = 'configs/esrgan/esrgan_x4c64b23g32_1xb16-400k_div2k.py'
checkpoint = 'https://download.openmmlab.com/mmediting/restorers/esrgan/esrgan_x4c64b23g32_1x16_400k_div2k_20200508-f8ccaf3b.pth'
img_path = 'tests/data/image/lq/baboon_x4.png'
editor = MMagicInferencer('esrgan', model_config=config, model_ckpt=checkpoint)
output = editor.infer(img=img_path,result_out_dir='output.png')
```

Now, you can check your fancy photos in `output.png`.

This tutorial will show how to use the following APIs：

- [**`list_models`**](mmpretrain.apis.list_models): List available model names in MMPreTrain.
- [**`get_model`**](mmpretrain.apis.get_model): Get a model from model name or model config.
- [**`inference_model`**](mmpretrain.apis.inference_model): Inference a model with the correspondding
  inferencer. It's a shortcut for a quick start, and for advanced usage, please use the below inferencer
  directly.
- Inferencers:
  1. [**`ImageClassificationInferencer`**](mmpretrain.apis.ImageClassificationInferencer):
     Perform image classification on the given image.
  2. [**`ImageRetrievalInferencer`**](mmpretrain.apis.ImageRetrievalInferencer):
     Perform image-to-image retrieval from the given image on a given image set.
  3. [**`ImageCaptionInferencer`**](mmpretrain.apis.ImageCaptionInferencer):
     Generate a caption on the given image.

3. [**`ImageCaptionInferencer`**](mmpretrain.apis.ImageCaptionInferencer):
  4. [**`VisualQuestionAnsweringInferencer`**](mmpretrain.apis.VisualQuestionAnsweringInferencer):
     Answer a question according to the given image.
  5. [**`VisualGroundingInferencer`**](mmpretrain.apis.VisualGroundingInferencer):
     Locate an object from the description on the given image.
  6. [**`TextToImageRetrievalInferencer`**](mmpretrain.apis.TextToImageRetrievalInferencer):
     Perform text-to-image retrieval from the given description on a given image set.
  7. [**`ImageToTextRetrievalInferencer`**](mmpretrain.apis.ImageToTextRetrievalInferencer):
     Perform image-to-text retrieval from the given image on a series of text.
  8. [**`NLVRInferencer`**](mmpretrain.apis.NLVRInferencer):
     Perform Natural Language for Visual Reasoning on a given image-pair and text.
  9. [**`FeatureExtractor`**](mmpretrain.apis.FeatureExtractor):

Perform Natural Language for Visual Reasoning on a given image-pair and text.
     Extract features from the image files by a vision backbone.

## List available models

list all the models in MMPreTrain.

```python
>>> from mmpretrain import list_models
>>> list_models()
['barlowtwins_resnet50_8xb256-coslr-300e_in1k',
 'beit-base-p16_beit-in21k-pre_3rdparty_in1k',
 ...]
```

`list_models` supports Unix filename pattern matching, you can use \*\* * \*\* to match any character.

```python
>>> list_models("*convnext-b*21k")
['convnext-base_3rdparty_in21k',
 'convnext-base_in21k-pre-3rdparty_in1k-384px',
 'convnext-base_in21k-pre_3rdparty_in1k']
```

You can use the `list_models` method of inferencers to get the available models of the correspondding tasks.

```python
>>> from mmpretrain import ImageCaptionInferencer
>>> ImageCaptionInferencer.list_models()

```python
['blip-base_3rdparty_caption',
 'blip2-opt2.7b_3rdparty-zeroshot_caption',
 'flamingo_3rdparty-zeroshot_caption',
 'ofa-base_3rdparty-finetuned_caption']
```

## Get a model

you can use `get_model` get the model.

```python
>>> from mmpretrain import get_model

# Get model without loading pre-trained weight.
>>> model = get_model("convnext-base_in21k-pre_3rdparty_in1k")

# Get model and load the default checkpoint.
>>> model = get_model("convnext-base_in21k-pre_3rdparty_in1k", pretrained=True)

# Get model and load the specified checkpoint.
>>> model = get_model("convnext-base_in21k-pre_3rdparty_in1k", pretrained="your_local_checkpoint_path")

# Get model with extra initialization arguments, for example, modify the num_classes in head.
>>> model = get_model("convnext-base_in21k-pre_3rdparty_in1k", head=dict(num_classes=10))

# Another example, remove the neck and head, and output from stage 1, 2, 3 in backbone
>>> model_headless = get_model("resnet18_8xb32_in1k", head=None, neck=None, backbone=dict(out_indices=(1, 2, 3)))
```

The obtained model is a usual PyTorch module.

```python
>>> model = get_model('convnext-base_in21k-pre_3rdparty_in1k', pretrained=True)
>>> x = torch.rand((1, 3, 224, 224))
>>> y = model(x)
>>> print(type(y), y.shape)
<class 'torch.Tensor'> torch.Size([1, 1000])
```

## Inference on given images

Here is an example to inference an [image](https://github.com/open-mmlab/mmpretrain/raw/main/demo/demo.JPEG) by the ResNet-50 pre-trained classification model.

```python
>>> from mmpretrain import inference_model

```python
>>> image = 'https://github.com/open-mmlab/mmpretrain/raw/main/demo/demo.JPEG'
>>> # If you have no graphical interface, please set `show=False`
>>> result = inference_model('resnet50_8xb32_in1k', image, show=True)
>>> print(result['pred_class'])
sea snake
```

The `inference_model` API is only for demo and cannot keep the model instance or inference on multiple
samples. You can use the inferencers for multiple calling.

```python
>>> from mmpretrain import ImageClassificationInferencer
>>> inferencer = ImageClassificationInferencer('resnet50_8xb32_in1k')
>>> # Note that the inferencer output is a list of result even if the input is a single sample.
>>> result = inferencer('https://github.com/open-mmlab/mmpretrain/raw/main/demo/demo.JPEG')[0]
sea snake
>>>
>>> # You can also use is for multiple images.

sea snake
>>>
>>> image_list = ['demo/demo.JPEG', 'demo/bird.JPEG'] * 16
>>> results = inferencer(image_list, batch_size=8)
>>> print(len(results))
32
>>> print(results[1]['pred_class'])
house finch, linnet, Carpodacus mexicanus
```

Usually, the result for every sample is a dictionary. For example, the image classification result is a dictionary containing `pred_label`, `pred_score`, `pred_scores` and `pred_class` as follows:

```python
{
    "pred_label": 65,
    "pred_score": 0.6649366617202759,
    "pred_class":"sea snake",
    "pred_scores": array([..., 0.6649366617202759, ...], dtype=float32)
}
```

You can configure the inferencer by arguments, for example, use your own config file and checkpoint to
inference images by CUDA.

```python

```python
>>> config = 'configs/resnet/resnet50_8xb32_in1k.py'
>>> checkpoint = 'https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_8xb32_in1k_20210831-ea4938fc.pth'
>>> inferencer = ImageClassificationInferencer(model=config, pretrained=checkpoint, device='cuda')
>>> result = inferencer(image)[0]
sea snake
```

## Inference by a Gradio demo

We also provide a gradio demo for all supported tasks and you can find it in [projects/gradio_demo/launch.py](https://github.com/open-mmlab/mmpretrain/blob/main/projects/gradio_demo/launch.py).

Please install `gradio` by `pip install -U gradio` at first.

Here is the interface preview:

<img src="https://user-images.githubusercontent.com/26739999/236147750-90ccb517-92c0-44e9-905e-1473677023b1.jpg" width="100%"/>

## Extract Features From Image

Compared with `model.extract_feat`, `FeatureExtractor` is used to extract features from the image files directly, instead of a batch of tensors.
In a word, the input of `model.extract_feat` is `torch.Tensor`, the input of `FeatureExtractor` is images.

```python
>>> from mmpretrain import FeatureExtractor, get_model
>>> model = get_model('resnet50_8xb32_in1k', backbone=dict(out_indices=(0, 1, 2, 3)))
>>> extractor = FeatureExtractor(model)
>>> features = extractor('https://github.com/open-mmlab/mmpretrain/raw/main/demo/demo.JPEG')[0]
>>> features[0].shape, features[1].shape, features[2].shape, features[3].shape
(torch.Size([256]), torch.Size([512]), torch.Size([1024]), torch.Size([2048]))
```

# 评测 LMDeploy 模型

我们支持评测使用 [LMDeploy](https://github.com/InternLM/lmdeploy) 加速过的大语言模型。LMDeploy 由 MMDeploy 和 MMRazor 团队联合开发，是涵盖了 LLM 任务的全套轻量化、部署和服务解决方案。 **TurboMind** 是 LMDeploy 推出的高效推理引擎。OpenCompass 对 TurboMind 进行了适配，本教程将介绍如何使用 OpenCompass 来对 TurboMind 加速后的模型进行评测。

## 环境配置

### 安装 OpenCompass

请根据 OpenCompass [安装指南](https://opencompass.readthedocs.io/en/latest/get_started.html) 来安装算法库和准备数据集。

### 安装 LMDeploy

使用 pip 安装 LMDeploy (python 3.8+)：

```shell
```

## 评测

我们使用 InternLM 作为例子来介绍如何评测。

### 第一步: 获取 InternLM 模型

```shell
# 1. Download InternLM model(or use the cached model's checkpoint)

# 2. Convert InternLM model to turbomind's format, which will be in "./workspace" by default

```

### 第二步: 验证转换后的模型

```shell
python -m lmdeploy.turbomind.chat ./workspace
```

### 第三步: 评测转换后的模型

在 OpenCompass 项目目录执行：

```shell
python run.py configs/eval_internlm_chat_7b_turbomind.py -w outputs/turbomind
```

当模型完成推理和指标计算后，我们便可获得模型的评测结果。

# Instance-aware Image Colorization (CVPR'2020)

> [Instance-Aware Image Colorization](https://openaccess.thecvf.com/content_CVPR_2020/html/Su_Instance-Aware_Image_Colorization_CVPR_2020_paper.html)

> **任务**: 图像上色

## 摘要

Image colorization is inherently an ill-posed problem with multi-modal uncertainty. Previous methods leverage the deep neural network to map input grayscale images to plausible color outputs directly. Although these learning-based methods have shown impressive performance, they usually fail on the input images that contain multiple objects. The leading cause is that existing models perform learning and colorization on the entire image. In the absence of a clear figure-ground separation, these models cannot effectively locate and learn meaningful object-level semantics. In this paper, we propose a method for achieving instance-aware colorization. Our network architecture leverages an off-the-shelf object detector to obtain cropped object images and uses an instance colorization network to extract object-level features. We use a similar network to extract the full-image features and apply a fusion module to full object-level and image-level features to predict the final colors. Both colorization networks and fusion modules are learned from a large-scale dataset. Experimental results show that our work outperforms existing methods on different quality metrics and achieves state-of-the-art performance on image colorization.

 <img src="https://github.com/ericsujw/InstColorization/blob/master/imgs/teaser.png?raw=true" width="400"/>
</div >

## 结果和模型

|                                              Model                                               |                                                 Download                                                 |
| :----------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------: |
| [instance_aware_colorization_officiial](./inst-colorizatioon_full_official_cocostuff-256x256.py) | [model](https://download.openmmlab.com/mmediting/inst_colorization/inst-colorizatioon_full_official_cocostuff-256x256-5b9d4eee.pth) |

## 快速开始

<details>
<summary>图像上色模型</summary>

您可以使用以下命令来对一张图像进行上色。

```shell

## 快速开始

<details>

```shell
python demo/colorization_demo.py configs/inst_colorization/inst-colorizatioon_full_official_cocostuff-256x256.py https://download.openmmlab.com/mmediting/inst_colorization/inst-colorizatioon_full_official_cocostuff-256x256-5b9d4eee.pth input.jpg output.jpg
```

更多细节可以参考 [Tutorial 3: inference with pre-trained models](https://mmagic.readthedocs.io/en/latest/user_guides/3_inference.html)。

<details>
<summary align="right">Instance-aware Image Colorization (CVPR'2020)</summary>

```bibtex
@inproceedings{Su-CVPR-2020,
  author = {Su, Jheng-Wei and Chu, Hung-Kuo and Huang, Jia-Bin},
  title = {Instance-aware Image Colorization},
}
```

# Android 下构建方式

- [Android 下构建方式](#android-下构建方式)

MMDeploy 为 android 平台提供交叉编译的构建方式.

MMDeploy converter 部分在 linux 平台上执行,SDK 部分在 android 平台上执行.

MMDeploy 的交叉编译分为两步:

1. 在 linux 平台上构建 MMDeploy converter. 请根据 [How to build linux](linux-x86_64.md) 进行构建.

2. 使用 android 工具链构建 MMDeploy SDK.

本文档仅提供在 linux 平台上使用 android 工具链进行交叉编译构建 MMDeploy SDK 的方法.

## 源码安装

- cmake

  **保证 cmake的版本 >= 3.14.0**. 如果不是,可以参考以下命令安装 3.20.0 版本. 如要获取其他版本,请参考 [这里](https://cmake.org/install)

  ```bash
  ```

tar -xzvf cmake-3.20.0-linux-x86_64.tar.gz
  ```

  **保证 android ndk 的版本 >= 19.0**. 如果不是,可以参考以下命令安装 r23c 版本. 如要获取其他版本,请参考 [这里](https://developer.android.com/ndk/downloads)

  ```bash
  ```

### 安装依赖包

如果您只对模型转换感兴趣,那么可以跳过本章节.

<table>
<thead>
  <tr>
    <th>安装方式 </th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>
    </td>

</tr>
  <tr>
    <td>ncnn 是支持 android 平台的高效神经网络推理计算框架</br>

    </td>

</tr>
  <tr>
  <b> 目前, MMDeploy 支持 ncnn 的 20220721 版本, 且必须使用<code>git clone</code> 下载源码的方式安装。请到 <a href='https://github.com/Tencent/ncnn/releases'> 这里 </a> 查询 ncnn 支持的 android ABI。</b><br>

cd ncnn

mkdir -p build_${ANDROID_ABI}
cd build_${ANDROID_ABI}

</td>
  </tr>
    <tr>
  <td>编译Java API之前需要先准备OpenJDK开发环境</br>

</td>
  </tr>
    <tr>
  请参考 <a href='https://github.com/open-mmlab/mmdeploy/tree/main/csrc/mmdeploy/apis/java/README.md'> Java API 编译 </a> 进行构建.
  </td>
  </tr>
</tbody>
</table>

下文展示构建 SDK 的样例，用 ncnn 作为推理引擎。

- cpu + ncnn
  ```Bash
  cmake .. \

-DMMDEPLOY_SHARED_LIBS=OFF \

  ```

# Dynamic Voxelization

> [End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds](https://arxiv.org/abs/1910.06528)

Recent work on 3D object detection advocates point cloud voxelization in birds-eye view, where objects preserve their physical dimensions and are naturally separable. When represented in this view, however, point clouds are sparse and have highly variable point density, which may cause detectors difficulties in detecting distant or small objects (pedestrians, traffic signs, etc.). On the other hand, perspective view provides dense observations, which could allow more favorable feature encoding for such cases. In this paper, we aim to synergize the birds-eye view and the perspective view and propose a novel end-to-end multi-view fusion (MVF) algorithm, which can effectively learn to utilize the complementary information from both. Specifically, we introduce dynamic voxelization, which has four merits compared to existing voxelization methods, i) removing the need of pre-allocating a tensor with fixed size; ii) overcoming the information loss due to stochastic point/voxel dropout; iii) yielding deterministic voxel embeddings and more stable detection outcomes; iv) establishing the bi-directional relationship between points and voxels, which potentially lays a natural foundation for cross-view feature fusion. By employing dynamic voxelization, the proposed feature fusion architecture enables each point to learn to fuse context information from different views. MVF operates on points and can be naturally extended to other approaches using LiDAR point clouds. We evaluate our MVF model extensively on the newly released Waymo Open Dataset and on the KITTI dataset and demonstrate that it significantly improves detection accuracy over the comparable single-view PointPillars baseline.

<img src="https://user-images.githubusercontent.com/30491025/143856017-98b77ecb-7c13-4164-9c1d-e3011a7645e6.png" width="600"/>
</div>

We implement Dynamic Voxelization proposed in  and provide its results and models on KITTI dataset.

### KITTI

|                               Model                                |  Class  |  Lr schd   | Mem (GB) | Inf time (fps) |  mAP  |                                                                                                                                                                                                                 Download                                                                                                                                                                                                                 |

|       [SECOND](./second_dv_secfpn_8xb6-80e_kitti-3d-car.py)        |   Car   | cyclic 80e |   5.5    |                | 78.83 |                     [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/dynamic_voxelization/dv_second_secfpn_6x8_80e_kitti-3d-car/dv_second_secfpn_6x8_80e_kitti-3d-car_20200620_235228-ac2c1c0c.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/dynamic_voxelization/dv_second_secfpn_6x8_80e_kitti-3d-car/dv_second_secfpn_6x8_80e_kitti-3d-car_20200620_235228.log.json)                     |

|  [SECOND](./second_dv_secfpn_8xb2-cosine-80e_kitti-3d-3class.py)   | 3 Class | cosine 80e |   5.5    |                | 65.27 | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/dynamic_voxelization/dv_second_secfpn_2x8_cosine_80e_kitti-3d-3class/dv_second_secfpn_2x8_cosine_80e_kitti-3d-3class_20210831_054106-e742d163.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/dynamic_voxelization/dv_second_secfpn_2x8_cosine_80e_kitti-3d-3class/dv_second_secfpn_2x8_cosine_80e_kitti-3d-3class_20210831_054106.log.json) |

| [PointPillars](./pointpillars_dv_secfpn_8xb6-160e_kitti-3d-car.py) |   Car   | cyclic 80e |   4.7    |                | 77.76 |       [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/dynamic_voxelization/dv_pointpillars_secfpn_6x8_160e_kitti-3d-car/dv_pointpillars_secfpn_6x8_160e_kitti-3d-car_20200620_230844-ee7b75c9.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/dynamic_voxelization/dv_pointpillars_secfpn_6x8_160e_kitti-3d-car/dv_pointpillars_secfpn_6x8_160e_kitti-3d-car_20200620_230844.log.json)       |

```latex
@article{zhou2019endtoend,
    title={End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds},
    author={Yin Zhou and Pei Sun and Yu Zhang and Dragomir Anguelov and Jiyang Gao and Tom Ouyang and James Guo and Jiquan Ngiam and Vijay Vasudevan},
    year={2019},
    eprint={1910.06528},
}
```

# 模型评测

模型评测过程会分别在 [ValLoop](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L300) 和 [TestLoop](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L373) 中被执行，用户可以在训练期间或使用配置文件中简单设置的测试脚本进行模型性能评估。`ValLoop` 和 `TestLoop` 属于 [Runner](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/runner.py#L59)，它们会在第一次被调用时构建。由于 `dataloader` 与 `evaluator` 是必需的参数，所以要成功构建 `ValLoop`，在构建 `Runner` 时必须设置 `val_dataloader` 和 `val_evaluator`，`TestLoop` 亦然。有关 Runner 设计的更多信息，请参阅 [MMEngine](https://github.com/open-mmlab/mmengine) 的[文档](https://github.com/open-mmlab/mmengine/blob/main/docs/zh_cn/design/runner.md)。

<center>
  <img src='../../../resources/test_step.png' />
  <center>测试/验证 数据流</center>
</center>

在 MMSegmentation 中，默认情况下，我们将 dataloader 和 metrics 的设置写在数据集配置文件中，并将 evaluation loop 的配置写在 `schedule_x` 配置文件中。

例如，在 ADE20K 配置文件 `configs/_base_/dataset/ADE20K.py` 中，在第37到48行，我们配置了 `val_dataloader`，在第51行，我们选择 `IoUMetric` 作为 evaluator，并设置 `mIoU` 作为指标：

```python
            img_path='images/validation',
            seg_map_path='annotations/validation'),

```

为了能够在训练期间进行评估模型，我们将评估配置添加到了 `configs/schedules/schedule_40k.py` 文件的第15至16行：

```python
```

使用以上两种设置，MMSegmentation 在 40K 迭代训练期间，每 4000 次迭代进行一次模型 **mIoU** 指标的评估。

如果我们希望在训练后测试模型，则需要将 `test_dataloader`、`test_evaluator` 和 `test_cfg` 配置添加到配置文件中。

```python

```python

test_evaluator = dict(type='IoUMetric', iou_metrics=['mIoU'])
```

在 MMSegmentation 中，默认情况下，`test_dataloader` 和 `test_evaluator` 的设置与 `ValLoop` 的 dataloader 和 evaluator 相同，我们可以修改这些设置以满足我们的需要。

## IoUMetric

```

MMSegmentation 基于 [MMEngine](https://github.com/open-mmlab/mmengine) 提供的 [BaseMetric](https://github.com/open-mmlab/mmengine/blob/main/mmengine/evaluator/metric.py) 实现 [IoUMetric](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/evaluation/metrics/iou_metric.py) 和 [CityscapesMetric](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/evaluation/metrics/citys_metric.py)，以评估模型的性能。有关统一评估接口的更多详细信息，请参阅[文档](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/evaluation.html)。

这里我们简要介绍 `IoUMetric` 的参数和两种主要方法。

除了 `collect_device` 和 `prefix` 之外，`IoUMetric` 的构建还包含一些其他参数。

构造函数的参数：

- ignore_index（int）- 将在评估中忽略的类别索引。默认值：255。
- iou_metrics（list\[str\] | str）- 需要计算的指标，可选项包括 'mIoU'、'mDice' 和 'mFscore'。
- nan_to_num（int，可选）- 如果指定，NaN 值将被用户定义的数字替换。默认值：None。
- beta（int）- 决定综合评分中 recall 的权重。默认值：1。

- collect_device（str）- 用于在分布式训练期间从不同进程收集结果的设备名称。必须是 'cpu' 或 'gpu'。默认为 'cpu'。
- prefix（str，可选）- 将添加到指标名称中的前缀，以消除不同 evaluator 的同名指标的歧义。如果参数中未提供前缀，则将使用 self.default_prefix 进行替代。默认为 None。

`IoUMetric` 实现 IoU 指标的计算，`IoUMetric` 的两个核心方法是 `process` 和 `compute_metrics`。

- `process` 方法处理一批 data 和 data_samples。
- `compute_metrics` 方法根据处理的结果计算指标。

### IoUMetric.process

参数：

- data_batch（Any）- 来自 dataloader 的一批数据。
- data_samples（Sequence\[dict\]）- 模型的一批输出。

返回值：

此方法没有返回值，因为处理的结果将存储在 `self.results` 中，以在处理完所有批次后进行指标的计算。

### IoUMetric.compute_metrics

参数：

- results（list）- 每个批次的处理结果。

返回值：

- Dict\[str，float\] - 计算的指标。指标的名称为 key，值是相应的结果。key 主要包括 **aAcc**、**mIoU**、**mAcc**、**mDice**、**mFscore**、**mPrecision**、**mPrecall**。

## CityscapesMetric

参数：

返回值：

`CityscapesMetric` 使用由 Cityscapes 官方提供的 [CityscapesScripts](https://github.com/mcordts/cityscapesScripts) 进行模型性能的评估。

### 使用方法

在使用之前，请先安装 `cityscapesscripts` 包：

```shell
```

由于 `IoUMetric` 在 MMSegmentation 中作为默认的 evaluator 使用，如果您想使用 `CityscapesMetric`，则需要自定义配置文件。在自定义配置文件中，应按如下方式替换默认 evaluator。

```python
val_evaluator = dict(type='CityscapesMetric', output_dir='tmp')
```

### 接口

构造函数的参数：

- output_dir (str) - 预测结果输出的路径
- ignore_index (int) - 将在评估中忽略的类别索引。默认值：255。
- format_only (bool) - 只为提交进行结果格式化而不进行评估。当您希望将结果格式化为特定格式并将其提交给测试服务器时有用。默认为 False。
- keep_results (bool) - 是否保留结果。当 `format_only` 为 True 时，`keep_results` 必须为 True。默认为 False。
- collect_device (str) - 用于在分布式训练期间从不同进程收集结果的设备名称。必须是 'cpu' 或 'gpu'。默认为 'cpu'。

- prefix (str，可选) - 将添加到指标名称中的前缀，以消除不同 evaluator 的同名指标的歧义。如果参数中未提供前缀，则将使用 self.default_prefix 进行替代。默认为 None。

#### CityscapesMetric.process

该方法将在图像上绘制 mask，并将绘制的图像保存到 `work_dir` 中。

参数：

- data_batch（dict）- 来自 dataloader 的一批数据。

返回值：

#### CityscapesMetric.compute_metrics

此方法将调用 `cityscapessscripts.evaluation.evalPixelLevelSemanticLabeling` 工具来计算指标。

参数：

- results（list）- 数据集的测试结果。

返回值：

- dict\[str:float\] - Cityscapes 评测结果。

# BMN

[Bmn: Boundary-matching network for temporal action proposal generation](https://openaccess.thecvf.com/content_ICCV_2019/html/Lin_BMN_Boundary-Matching_Network_for_Temporal_Action_Proposal_Generation_ICCV_2019_paper.html)

Temporal action proposal generation is an challenging and promising task which aims to locate temporal regions in real-world videos where action or event may occur. Current bottom-up proposal generation methods can generate proposals with precise boundary, but cannot efficiently generate adequately reliable confidence scores for retrieving proposals. To address these difficulties, we introduce the Boundary-Matching (BM) mechanism to evaluate confidence scores of densely distributed proposals, which denote a proposal as a matching pair of starting and ending boundaries and combine all densely distributed BM pairs into the BM confidence map. Based on BM mechanism, we propose an effective, efficient and end-to-end proposal generation method, named Boundary-Matching Network (BMN), which generates proposals with precise temporal boundaries as well as reliable confidence scores simultaneously. The two-branches of BMN are jointly trained in an unified framework. We conduct experiments on two challenging datasets: THUMOS-14 and ActivityNet-1.3, where BMN shows significant performance improvement with remarkable efficiency and generalizability. Further, combining with existing action classifier, BMN can achieve state-of-the-art temporal action detection performance.

<img src="https://user-images.githubusercontent.com/34324155/143016479-2ca7e8b6-a17b-4a4c-b4c9-ae731935cd91.png" width="800"/>
</div>

|    feature    | gpus | pretrain |  AUC  | AR@1  | AR@5  | AR@10 | AR@100 | gpu_mem(M) | iter time(s) |                    config                    |                    ckpt                    |                    log                    |
| :-----------: | :--: | :------: | :---: | :---: | :---: | :---: | :----: | :--------: | :----------: | :------------------------------------------: | :----------------------------------------: | :---------------------------------------: |

| cuhk_mean_100 |  2   |   None   | 67.25 | 32.89 | 49.43 | 56.64 | 75.29  |    5412    |      -       | [config](/configs/localization/bmn/bmn_2xb8-400x100-9e_activitynet-feature.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/localization/bmn/bmn_2xb8-400x100-9e_activitynet-feature_20220908-79f92857.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/localization/bmn/bmn_2xb8-400x100-9e_activitynet-feature.log) |

3. We evaluate the action detection performance of BMN, using  [anet_cuhk_2017](https://download.openmmlab.com/mmaction/localization/cuhk_anet17_pred.json) submission for ActivityNet2017 Untrimmed Video Classification Track to assign label for each action proposal.

\*We train BMN with the [official repo](https://github.com/JJBOY/BMN-Boundary-Matching-Network), evaluate its proposal generation and action detection performance with [anet_cuhk_2017](https://download.openmmlab.com/mmaction/localization/cuhk_anet17_pred.json) for label assigning.

## Train

Train BMN model on ActivityNet features dataset.

```shell
bash tools/dist_train.sh configs/localization/bmn/bmn_2xb8-400x100-9e_activitynet-feature.py 2
```

## Train

```shell
```

## Test

Test BMN on ActivityNet feature dataset.

```shell
python3 tools/test.py  configs/localization/bmn/bmn_2xb8-400x100-9e_activitynet-feature.py CHECKPOINT.PTH
```

```BibTeX
@inproceedings{lin2019bmn,
  title={Bmn: Boundary-matching network for temporal action proposal generation},
  author={Lin, Tianwei and Liu, Xiao and Li, Xin and Ding, Errui and Wen, Shilei},
  pages={3889--3898},
}
```

```BibTeX
@article{zhao2017cuhk,

booktitle={Proceedings of the IEEE International Conference on Computer Vision},
}
```

```BibTeX
  title={Cuhk \& ethz \& siat submission to activitynet challenge 2017},
  author={Zhao, Y and Zhang, B and Wu, Z and Yang, S and Zhou, L and Yan, S and Wang, L and Xiong, Y and Lin, D and Qiao, Y and others},
  journal={arXiv preprint arXiv:1710.08011},
}
```

Thank you for your interest in this project! Please refer to the following
sections on how to contribute code and bug reports.

### Reporting bugs

Before submitting a question or bug report, please take a moment of your time
and ensure that your issue isn't already discussed in the project documentation
provided at [pybind11.readthedocs.org][] or in the [issue tracker][]. You can
also check [gitter][] to see if it came up before.

Assuming that you have identified a previously unknown problem or an important
question, it's essential that you submit a self-contained and minimal piece of
code that reproduces the problem. In other words: no external dependencies,
isolate the function(s) that cause breakage, submit matched and complete C++
and Python snippets that can be easily compiled and run in isolation; or
ideally make a small PR with a failing test case that can be used as a starting
point.

## Pull requests

point.

Contributions are submitted, reviewed, and accepted using GitHub pull requests.
Please refer to [this article][using pull requests] for details and adhere to
the following rules to make the process as smooth as possible:

* Make a new branch for every feature you're working on.
* Make small and clean pull requests that are easy to review but make sure they
  do add value by themselves.
* Add tests for any new functionality and run the test suite (`cmake --build
  build --target pytest`) to ensure that no existing features break.
* Please run [`pre-commit`][pre-commit] to check your code matches the
  project style. (Note that `gawk` is required.) Use `pre-commit run
  --all-files` before committing (or use installed-mode, check pre-commit docs)
  to verify your code passes before pushing to save time.

--all-files` before committing (or use installed-mode, check pre-commit docs)
* This project has a strong focus on providing general solutions using a
  minimal amount of code, thus small pull requests are greatly preferred.

### Licensing of contributions

pybind11 is provided under a BSD-style license that can be found in the
``LICENSE`` file. By using, distributing, or contributing to this project, you
agree to the terms and conditions of this license.

You are under no obligation whatsoever to provide any bug fixes, patches, or
upgrades to the features, functionality or performance of the source code
("Enhancements") to anyone; however, if you choose to make your Enhancements
available either publicly, or directly to the author of this software, without
imposing a separate written license agreement for such Enhancements, then you
hereby grant the following license: a non-exclusive, royalty-free perpetual

license to install, use, modify, prepare derivative works, incorporate into
other computer software, distribute, and sublicense such enhancements or
derivative works thereof, in binary and source code form.

## Development of pybind11

### Quick setup

To setup a quick development environment, use [`nox`](https://nox.thea.codes).
This will allow you to do some common tasks with minimal setup effort, but will
take more time to run and be less flexible than a full development environment.
If you use [`pipx run nox`](https://pipx.pypa.io), you don't even need to
install `nox`. Examples:

```bash
# List all available sessions
nox -l

# Run linters
nox -s lint

# Run tests on Python 3.9
nox -s tests-3.9

# Build and preview docs
nox -s docs -- serve

# Build SDists and wheels
nox -s build
```

### Full setup

nox -l

nox -s lint

```

To setup an ideal development environment, run the following commands on a
system with CMake 3.14+:

```bash
python3 -m venv venv
source venv/bin/activate
pip install -r tests/requirements.txt
cmake -S . -B build -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON
cmake --build build -j4
```

Tips:

* You can use `virtualenv` (from PyPI) instead of `venv` (which is Python 3
  only).
* You can select any name for your environment folder; if it contains "env" it
  will be ignored by git.
* If you don’t have CMake 3.14+, just add “cmake” to the pip install command.
* You can use `-DPYBIND11_FINDPYTHON=ON` to use FindPython on CMake 3.12+
* In classic mode, you may need to set `-DPYTHON_EXECUTABLE=/path/to/python`.
  FindPython uses `-DPython_ROOT_DIR=/path/to` or
  `-DPython_EXECUTABLE=/path/to/python`.

### Configuration options

In CMake, configuration options are given with “-D”. Options are stored in the
build directory, in the `CMakeCache.txt` file, so they are remembered for each
build directory. Two selections are special - the generator, given with `-G`,
and the compiler, which is selected based on environment variables `CXX` and
similar, or `-DCMAKE_CXX_COMPILER=`. Unlike the others, these cannot be changed
after the initial run.

The valid options are:

* `-DCMAKE_BUILD_TYPE`: Release, Debug, MinSizeRel, RelWithDebInfo
* `-DPYBIND11_FINDPYTHON=ON`: Use CMake 3.12+’s FindPython instead of the
  classic, deprecated, custom FindPythonLibs
* `-DPYBIND11_NOPYTHON=ON`: Disable all Python searching (disables tests)
* `-DBUILD_TESTING=ON`: Enable the tests

classic, deprecated, custom FindPythonLibs
* `-DDOWNLOAD_CATCH=ON`: Download catch to build the C++ tests
* `-DOWNLOAD_EIGEN=ON`: Download Eigen for the NumPy tests
* `-DPYBIND11_INSTALL=ON/OFF`: Enable the install target (on by default for the
  master project)
* `-DUSE_PYTHON_INSTALL_DIR=ON`: Try to install into the python dir

<details><summary>A few standard CMake tricks: (click to expand)</summary><p>

* Use `cmake --build build -v` to see the commands used to build the files.
* Use `cmake build -LH` to list the CMake options with help.
* Use `ccmake` if available to see a curses (terminal) gui, or `cmake-gui` for
  a completely graphical interface (not present in the PyPI package).
* Use `cmake --build build -j12` to build with 12 cores (for example).
* Use `-G` and the name of a generator to use something different. `cmake

  --help` lists the generators available.
      - On Unix, setting `CMAKE_GENERATER=Ninja` in your environment will give
        you automatic mulithreading on all your CMake projects!
* Open the `CMakeLists.txt` with QtCreator to generate for that IDE.
* You can use `-DCMAKE_EXPORT_COMPILE_COMMANDS=ON` to generate the `.json` file
  that some tools expect.

</p></details>

To run the tests, you can "build" the check target:

```bash
cmake --build build --target check
```

`--target` can be spelled `-t` in CMake 3.15+. You can also run individual
tests with these targets:

* `pytest`: Python tests only, using the
[pytest](https://docs.pytest.org/en/stable/) framework
* `cpptest`: C++ tests only
* `test_cmake_build`: Install / subdirectory tests

If you want to build just a subset of tests, use

`-DPYBIND11_TEST_OVERRIDE="test_callbacks.cpp;test_pickling.cpp"`. If this is
empty, all tests will be built.

You may also pass flags to the `pytest` target by editing `tests/pytest.ini` or
by using the `PYTEST_ADDOPTS` environment variable
(see [`pytest` docs](https://docs.pytest.org/en/2.7.3/customize.html#adding-default-options)). As an example:

```bash
env PYTEST_ADDOPTS="--capture=no --exitfirst" \
    cmake --build build --target pytest
# Or using abbreviated flags
env PYTEST_ADDOPTS="-s -x" cmake --build build --target pytest
```

### Formatting

All formatting is handled by pre-commit.

Install with brew (macOS) or pip (any OS):

```bash
# Any OS
python3 -m pip install pre-commit

# OR macOS with homebrew:
brew install pre-commit
```

```bash
# Any OS

```

Then, you can run it on the items you've added to your staging area, or all
files:

```bash
pre-commit run
# OR
```

And, if you want to always use it, you can install it as a git hook (hence the
name, pre-commit):

```bash
```

### Clang-Format

As of v2.6.2, pybind11 ships with a [`clang-format`][clang-format]
configuration file at the top level of the repo (the filename is
`.clang-format`). Currently, formatting is NOT applied automatically, but
manually using `clang-format` for newly developed files is highly encouraged.
To check if a file needs formatting:

```bash
clang-format -style=file --dry-run some.cpp
```

The output will show things to be fixed, if any. To actually format the file:

```bash
clang-format -style=file -i some.cpp
```

```bash
```

```bash
```

Note that the `-style-file` option searches the parent directories for the
`.clang-format` file, i.e. the commands above can be run in any subdirectory
of the pybind11 repo.

### Clang-Tidy

[`clang-tidy`][clang-tidy] performs deeper static code analyses and is
more complex to run, compared to `clang-format`, but support for `clang-tidy`
is built into the pybind11 CMake configuration. To run `clang-tidy`, the
following recipe should work. Run the `docker` command from the top-level
directory inside your pybind11 git clone. Files will be modified in place,
so you can use git to monitor the changes.

```bash
docker run --rm -v $PWD:/mounted_pybind11 -it silkeh/clang:12
apt-get update && apt-get install -y python3-dev python3-pytest

```bash
cmake -S /mounted_pybind11/ -B build -DCMAKE_CXX_CLANG_TIDY="$(which clang-tidy);-fix" -DDOWNLOAD_EIGEN=ON -DDOWNLOAD_CATCH=ON -DCMAKE_CXX_STANDARD=17
cmake --build build -j 2 -- --keep-going
```

### Include what you use

To run include what you use, install (`brew install include-what-you-use` on
macOS), then run:

```bash
cmake -S . -B build-iwyu -DCMAKE_CXX_INCLUDE_WHAT_YOU_USE=$(which include-what-you-use)
cmake --build build
```

The report is sent to stderr; you can pipe it into a file if you wish.

### Build recipes

This builds with the Intel compiler (assuming it is in your path, along with a
recent CMake and Python 3):

```bash
. venv/bin/activate
pip install pytest

```bash
cmake -S . -B build-intel -DCMAKE_CXX_COMPILER=$(which icpc) -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON -DPYBIND11_WERROR=ON
```

This will test the PGI compilers:

```bash
docker run --rm -it -v $PWD:/pybind11 nvcr.io/hpc/pgi-compilers:ce
apt-get update && apt-get install -y python3-dev python3-pip python3-pytest
wget -qO- "https://cmake.org/files/v3.18/cmake-3.18.2-Linux-x86_64.tar.gz" | tar --strip-components=1 -xz -C /usr/local
cmake -S pybind11/ -B build
```

### Explanation of the SDist/wheel building design

> These details below are _only_ for packaging the Python sources from git. The
> SDists and wheels created do not have any extra requirements at all and are
> completely normal.

The main objective of the packaging system is to create SDists (Python's source

distribution packages) and wheels (Python's binary distribution packages) that
include everything that is needed to work with pybind11, and which can be
installed without any additional dependencies. This is more complex than it
appears: in order to support CMake as a first class language even when using
the PyPI package, they must include the _generated_ CMake files (so as not to
require CMake when installing the `pybind11` package itself). They should also
provide the option to install to the "standard" location
(`<ENVROOT>/include/pybind11` and `<ENVROOT>/share/cmake/pybind11`) so they are
easy to find with CMake, but this can cause problems if you are not an
environment or using ``pyproject.toml`` requirements. This was solved by having

two packages; the "nice" pybind11 package that stores the includes and CMake
files inside the package, that you get access to via functions in the package,
and a `pybind11-global` package that can be included via `pybind11[global]` if
you want the more invasive but discoverable file locations.

If you want to install or package the GitHub source, it is best to have Pip 10
or newer on Windows, macOS, or Linux (manylinux1 compatible, includes most
distributions).  You can then build the SDists, or run any procedure that makes
SDists internally, like making wheels or installing.

```bash
# Editable development install example
python3 -m pip install -e .
```

Since Pip itself does not have an `sdist` command (it does have `wheel` and
`install`), you may want to use the upcoming `build` package:

```bash
python3 -m pip install build

```

```bash

# Normal package
python3 -m build -s .

# Global extra
PYBIND11_GLOBAL_SDIST=1 python3 -m build -s .
```

If you want to use the classic "direct" usage of `python setup.py`, you will
need CMake 3.15+ and either `make` or `ninja` preinstalled (possibly via `pip
install cmake ninja`), since directly running Python on `setup.py` cannot pick
up and install `pyproject.toml` requirements. As long as you have those two
things, though, everything works the way you would expect:

```bash
python3 setup.py sdist

PYBIND11_GLOBAL_SDIST=1 python3 setup.py sdist
```

A detailed explanation of the build procedure design for developers wanting to
work on or maintain the packaging system is as follows:

#### 1. Building from the source directory

```

When you invoke any `setup.py` command from the source directory, including
`pip wheel .` and `pip install .`, you will activate a full source build. This
is made of the following steps:

1. If the tool is PEP 518 compliant, like Pip 10+, it will create a temporary
   virtual environment and install the build requirements (mostly CMake) into
   it. (if you are not on Windows, macOS, or a manylinux compliant system, you
   can disable this with `--no-build-isolation` as long as you have CMake 3.15+
   installed)
2. The environment variable `PYBIND11_GLOBAL_SDIST` is checked - if it is set
   and truthy, this will be make the accessory `pybind11-global` package,
   instead of the normal `pybind11` package. This package is used for
   installing the files directly to your environment root directory, using

instead of the normal `pybind11` package. This package is used for
   `pybind11[global]`.
2. `setup.py` reads the version from `pybind11/_version.py` and verifies it
   matches `includes/pybind11/detail/common.h`.
3. CMake is run with `-DCMAKE_INSTALL_PREIFX=pybind11`. Since the CMake install
   procedure uses only relative paths and is identical on all platforms, these
   files are valid as long as they stay in the correct relative position to the
   includes. `pybind11/share/cmake/pybind11` has the CMake files, and
   `pybind11/include` has the includes. The build directory is discarded.
4. Simpler files are placed in the SDist: `tools/setup_*.py.in`,
   `tools/pyproject.toml` (`main` or `global`)
5. The package is created by running the setup function in the
   `tools/setup_*.py`.  `setup_main.py` fills in Python packages, and
   `setup_global.py` fills in only the data/header slots.

6. A context manager cleans up the temporary CMake install directory (even if
   an error is thrown).

### 2. Building from SDist

Since the SDist has the rendered template files in `tools` along with the
includes and CMake files in the correct locations, the builds are completely
trivial and simple. No extra requirements are required. You can even use Pip 9
if you really want to.

[pre-commit]: https://pre-commit.com
[clang-format]: https://clang.llvm.org/docs/ClangFormat.html
[clang-tidy]: https://clang.llvm.org/extra/clang-tidy/
[pybind11.readthedocs.org]: http://pybind11.readthedocs.org/en/latest
[issue tracker]: https://github.com/pybind/pybind11/issues
[gitter]: https://gitter.im/pybind/Lobby
[using pull requests]: https://help.github.com/articles/using-pull-requests

# EVA

> [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://arxiv.org/abs/2211.07636)

We launch EVA, a vision-centric foundation model to explore the limits of visual representation at scale using only publicly accessible data. EVA is a vanilla ViT pre-trained to reconstruct the masked out image-text aligned vision features conditioned on visible image patches. Via this pretext task, we can efficiently scale up EVA to one billion parameters, and sets new records on a broad range of representative vision downstream tasks, such as image recognition, video action recognition, object detection, instance segmentation and semantic segmentation without heavy supervised training. Moreover, we observe quantitative changes in scaling EVA result in qualitative changes in transfer learning performance that are not present in other models. For instance, EVA takes a great leap in the challenging large vocabulary instance segmentation task: our model achieves almost the same state-of-the-art performance on LVISv1.0 dataset with over a thousand categories and COCO dataset with only eighty categories. Beyond a pure vision encoder, EVA can also serve as a vision-centric, multi-modal pivot to connect images and text. We find initializing the vision tower of a giant CLIP from EVA can greatly stabilize the training and outperform the training from scratch counterpart with much fewer samples and less compute, providing a new direction for scaling up and accelerating the costly training of multi-modal foundation models.

<img src="https://user-images.githubusercontent.com/24734142/205410193-f1164e56-c117-4165-86f5-4cbfd797bc87.png" width="70%"/>
</div>

```python

predict = inference_model('vit-base-p16_eva-mae-style-pre_8xb128-coslr-100e_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('eva-mae-style_vit-base-p16_16xb256-coslr-400e_in1k', pretrained=True)
```

Train:

```shell

```

Train:

```shell
python tools/train.py configs/eva/eva-mae-style_vit-base-p16_16xb256-coslr-400e_in1k.py
```

Test:

```shell
python tools/test.py configs/eva/benchmarks/vit-base-p16_8xb128-coslr-100e_in1k.py https://download.openmmlab.com/mmselfsup/1.x/eva/eva-mae-style_vit-base-p16_16xb256-coslr-400e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k_20221226-f61cf992.pth
```

| Model                                                | Params (M) | Flops (G) |                             Config                              |                              Download                              |

| :--------------------------------------------------- | :--------: | :-------: | :-------------------------------------------------------------: | :----------------------------------------------------------------: |
| `eva-mae-style_vit-base-p16_16xb256-coslr-400e_in1k` |   111.78   |   17.58   | [config](eva-mae-style_vit-base-p16_16xb256-coslr-400e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/eva/eva-mae-style_vit-base-p16_16xb256-coslr-400e_in1k/eva-mae-style_vit-base-p16_16xb256-coslr-400e_in1k_20221226-26d90f07.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/eva/eva-mae-style_vit-base-p16_16xb256-coslr-400e_in1k/eva-mae-style_vit-base-p16_16xb256-coslr-400e_in1k_20221226-26d90f07.json) |
| `beit-l-p14_3rdparty-eva_in21k`\*                    |   303.18   |   81.08   |                 [config](eva-l-p14_headless.py)                 | [model](https://download.openmmlab.com/mmclassification/v0/eva/eva-l-p14_3rdparty-mim_in21k_20221213-3a5da50b.pth) |

| `beit-l-p14_eva-pre_3rdparty_in21k`\*                |   303.18   |   81.08   |                 [config](eva-l-p14_headless.py)                 | [model](https://download.openmmlab.com/mmclassification/v0/eva/eva-l-p14_mim-pre_3rdparty_in21k_20221213-8f194fa2.pth) |
| `beit-g-p16_3rdparty-eva_30m`\*                      |  1011.32   |  203.52   |                 [config](eva-g-p16_headless.py)                 | [model](https://download.openmmlab.com/mmclassification/v0/eva/eva-g-p16_3rdparty_30m_20221213-7bed23ee.pth) |
| `beit-g-p14_3rdparty-eva_30m`\*                      |  1011.60   |  267.17   |                 [config](eva-g-p14_headless.py)                 | [model](https://download.openmmlab.com/mmclassification/v0/eva/eva-g-p14_3rdparty_30m_20221213-3b7aca97.pth) |

| `beit-g-p14_eva-30m-pre_3rdparty_in21k`\*            |  1011.60   |  267.17   |                 [config](eva-g-p14_headless.py)                 | [model](https://download.openmmlab.com/mmclassification/v0/eva/eva-g-p14_30m-pre_3rdparty_in21k_20221213-d72285b7.pth) |

*Models with * are converted from the [official repo](https://github.com/baaivision/EVA). The config files of these models are only for inference. We haven't reproduce the training results.*

| `vit-base-p16_eva-mae-style-pre_8xb128-coslr-100e_in1k` | [EVA MAE STYLE](https://download.openmmlab.com/mmselfsup/1.x/eva/eva-mae-style_vit-base-p16_16xb256-coslr-400e_in1k/eva-mae-style_vit-base-p16_16xb256-coslr-400e_in1k_20221226-26d90f07.pth) |   86.57    |   17.58   |   83.70   |    N/A    | [config](benchmarks/vit-base-p16_8xb128-coslr-100e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/eva/eva-mae-style_vit-base-p16_16xb256-coslr-400e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k_20221226-f61cf992.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/eva/eva-mae-style_vit-base-p16_16xb256-coslr-400e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k_20221226-f61cf992.json) |

| `vit-base-p16_eva-mae-style-pre_8xb2048-linear-coslr-100e_in1k` | [EVA MAE STYLE](https://download.openmmlab.com/mmselfsup/1.x/eva/eva-mae-style_vit-base-p16_16xb256-coslr-400e_in1k/eva-mae-style_vit-base-p16_16xb256-coslr-400e_in1k_20221226-26d90f07.pth) |   86.57    |   17.58   |   69.00   |    N/A    | [config](benchmarks/vit-base-p16_8xb2048-linear-coslr-100e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/eva/eva-mae-style_vit-base-p16_16xb256-coslr-400e_in1k/vit-base-p16_linear-8xb2048-coslr-100e_in1k/vit-base-p16_linear-8xb2048-coslr-100e_in1k_20221226-ef51bf09.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/eva/eva-mae-style_vit-base-p16_16xb256-coslr-400e_in1k/vit-base-p16_linear-8xb2048-coslr-100e_in1k/vit-base-p16_linear-8xb2048-coslr-100e_in1k_20221226-ef51bf09.json) |

| `beit-l-p14_eva-pre_3rdparty_in1k-196px`\* | [EVA](https://download.openmmlab.com/mmclassification/v0/eva/eva-l-p14_3rdparty-mim_in21k_20221213-3a5da50b.pth) |   304.14   |   61.57   |   87.94   |   98.5    | [config](eva-l-p14_8xb16_in1k-196px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/eva/eva-l-p14_mim-pre_3rdparty_in1k-196px_20221214-2adf4d28.pth) |
| `beit-l-p14_eva-in21k-pre_3rdparty_in1k-196px`\* |              EVA ImageNet-21k              |   304.14   |   61.57   |   88.58   |   98.65   | [config](eva-l-p14_8xb16_in1k-196px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/eva/eva-l-p14_mim-in21k-pre_3rdparty_in1k-196px_20221213-b730c7e7.pth) |

| `beit-l-p14_eva-pre_3rdparty_in1k-336px`\* | [EVA](https://download.openmmlab.com/mmclassification/v0/eva/eva-l-p14_3rdparty-mim_in21k_20221213-3a5da50b.pth) |   304.53   |  191.10   |   88.66   |   98.75   | [config](eva-l-p14_8xb16_in1k-336px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/eva/eva-l-p14_mim-pre_3rdparty_in1k-336px_20221214-07785cfd.pth) |
| `beit-l-p14_eva-in21k-pre_3rdparty_in1k-336px`\* |              EVA ImageNet-21k              |   304.53   |  191.10   |   89.17   |   98.86   | [config](eva-l-p14_8xb16_in1k-336px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/eva/eva-l-p14_mim-in21k-pre_3rdparty_in1k-336px_20221213-f25b7634.pth) |

| `beit-g-p14_eva-30m-in21k-pre_3rdparty_in1k-336px`\* | [EVA 30M ImageNet-21k](https://download.openmmlab.com/mmclassification/v0/eva/eva-g-p14_30m-pre_3rdparty_in21k_20221213-d72285b7.pth) |  1013.01   |  620.64   |   89.61   |   98.93   | [config](eva-g-p14_8xb16_in1k-336px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/eva/eva-g-p14_30m-in21k-pre_3rdparty_in1k-336px_20221213-210f9071.pth) |
| `beit-g-p14_eva-30m-in21k-pre_3rdparty_in1k-560px`\* | [EVA 30M ImageNet-21k](https://download.openmmlab.com/mmclassification/v0/eva/eva-g-p14_30m-pre_3rdparty_in21k_20221213-d72285b7.pth) |  1014.45   |  1906.76  |   89.71   |   98.96   | [config](eva-g-p14_8xb16_in1k-560px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/eva/eva-g-p14_30m-in21k-pre_3rdparty_in1k-560px_20221213-fa1c3652.pth) |

```bibtex
@article{EVA,
  title={EVA: Exploring the Limits of Masked Visual Representation Learning at Scale},
  author={Fang, Yuxin and Wang, Wen and Xie, Binhui and Sun, Quan and Wu, Ledell and Wang, Xinggang and Huang, Tiejun and Wang, Xinlong and Cao, Yue},
  journal={arXiv preprint arXiv:2211.07636},
}
```

# Overview

Welcome to MMagic! In this section, you will know about

- [Overview](#overview)
  - [What is MMagic?](#what-is-mmagic)
  - [Why should I use MMagic?](#why-should-i-use-mmagic)
  - [Get started](#get-started)
  - [User guides](#user-guides)
    - [Advanced guides](#advanced-guides)
    - [How to](#how-to)

## What is MMagic?

MMagic (**M**ultimodal **A**dvanced, **G**enerative, and **I**ntelligent **C**reation) is an open-source AIGC toolbox for professional AI researchers and machine learning engineers to explore image and video processing, editing and generation.

MMagic allows researchers and engineers to use pre-trained state-of-the-art models, train and develop new customized models easily.

MMagic supports various foundamental generative models, including:

- Unconditional Generative Adversarial Networks (GANs)
- Conditional Generative Adversarial Networks (GANs)
- Internal Learning
- Diffusion Models
- And many other generative models are coming soon!

MMagic supports various applications, including:

- Text-to-Image
- Image-to-image translation
- 3D-aware generation
- Image super-resolution
- Video super-resolution
- Video frame interpolation
- Image inpainting
- Image matting
- Image restoration
- Image colorization
- Image generation
- And many other applications are coming soon!

    <video width="100%" controls>
        <source src="https://user-images.githubusercontent.com/49083766/233564593-7d3d48ed-e843-4432-b610-35e3d257765c.mp4" type="video/mp4">
        <object data="https://user-images.githubusercontent.com/49083766/233564593-7d3d48ed-e843-4432-b610-35e3d257765c.mp4" width="100%">
        </object>
    </video>
</div>
</br>

## Why should I use MMagic?

</object>
    </video>
</div>
</br>

For installation instructions, please see [Installation](install.md).

## User guides

For beginners, we suggest learning the basic usage of MMagic from [user_guides](../user_guides/config.md).

### Advanced guides

For users who are familiar with MMagic, you may want to learn the design of MMagic, as well as how to extend the repo, how to use multiple repos and other advanced usages, please refer to [advanced_guides](../advanced_guides/evaluator.md).

### How to

For users who want to use MMagic to do something, please refer to [How to](../howto/models.md).

# 模型支持列表

自测完成的 model-backend 组合：

| Model config                                                                                             | Codebase         | TorchScript | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO | Ascend | RKNN |
| :------------------------------------------------------------------------------------------------------- | :--------------- | :---------: | :---------: | :------: | :--: | :---: | :------: | :----: | :--: |
| [RetinaNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/retinanet)                        | MMDetection      |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   Y    |  Y   |
| [Faster R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/faster_rcnn)                   | MMDetection      |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   Y    |  N   |

| [YOLOv3](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/yolo)                                | MMDetection      |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |   Y    |  Y   |
| [YOLOX](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/yolox)                                | MMDetection      |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |   N    |  Y   |
| [FCOS](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/fcos)                                  | MMDetection      |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |   N    |  N   |
| [FSAF](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/fsaf)                                  | MMDetection      |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   N    |  Y   |

| [Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/mask_rcnn)                       | MMDetection      |      Y      |      Y      |    Y     |  N   |   N   |    Y     |   N    |  N   |
| [SSD](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/ssd)[\*](#note)                         | MMDetection      |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |   N    |  Y   |
| [FoveaBox](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/foveabox)                          | MMDetection      |      Y      |      Y      |    N     |  N   |   N   |    Y     |   N    |  N   |
| [ATSS](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/atss)                                  | MMDetection      |      N      |      Y      |    Y     |  N   |   N   |    Y     |   N    |  N   |

| [GFL](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/gfl)                                    | MMDetection      |      N      |      Y      |    Y     |  N   |   ?   |    Y     |   N    |  N   |
| [Cascade R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/cascade_rcnn)                 | MMDetection      |      N      |      Y      |    Y     |  N   |   Y   |    Y     |   N    |  N   |
| [Cascade Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/cascade_rcnn)            | MMDetection      |      N      |      Y      |    Y     |  N   |   N   |    Y     |   N    |  N   |
| [Swin Transformer](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/swin)[\*](#note)           | MMDetection      |      N      |      Y      |    Y     |  N   |   N   |    Y     |   N    |  N   |

| [VFNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/vfnet)                                | MMDetection      |      N      |      N      |    N     |  N   |   N   |    Y     |   N    |  N   |
| [RepPoints](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/reppoints)                        | MMDetection      |      N      |      N      |    Y     |  N   |   ?   |    Y     |   N    |  N   |
| [DETR](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/detr)                                  | MMDetection      |      N      |      Y      |    Y     |  N   |   ?   |    N     |   N    |  N   |
| [CenterNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/centernet)                        | MMDetection      |      N      |      Y      |    Y     |  N   |   ?   |    Y     |   N    |  N   |

| [SOLO](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo)                                  | MMDetection      |      N      |      Y      |    N     |  N   |   N   |    Y     |   N    |  N   |
| [SOLOv2](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solov2)                              | MMDetection      |      N      |      Y      |    N     |  N   |   N   |    Y     |   N    |  N   |
| [ResNet](https://github.com/open-mmlab/mmpretrain/tree/main/configs/resnet)                              | MMPretrain       |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   Y    |  Y   |
| [ResNeXt](https://github.com/open-mmlab/mmpretrain/tree/main/configs/resnext)                            | MMPretrain       |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   Y    |  Y   |

| [SE-ResNet](https://github.com/open-mmlab/mmpretrain/tree/main/configs/seresnet)                         | MMPretrain       |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   Y    |  Y   |
| [MobileNetV2](https://github.com/open-mmlab/mmpretrain/tree/main/configs/mobilenet_v2)                   | MMPretrain       |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   Y    |  Y   |
| [ShuffleNetV1](https://github.com/open-mmlab/mmpretrain/tree/main/configs/shufflenet_v1)                 | MMPretrain       |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   Y    |  Y   |
| [ShuffleNetV2](https://github.com/open-mmlab/mmpretrain/tree/main/configs/shufflenet_v2)                 | MMPretrain       |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   Y    |  Y   |

| [VisionTransformer](https://github.com/open-mmlab/mmpretrain/tree/main/configs/vision_transformer)       | MMPretrain       |      Y      |      Y      |    Y     |  Y   |   ?   |    Y     |   Y    |  N   |
| [SwinTransformer](https://github.com/open-mmlab/mmpretrain/tree/main/configs/swin_transformer)           | MMPretrain       |      Y      |      Y      |    Y     |  N   |   ?   |    N     |   ?    |  N   |
| [MobileOne](https://github.com/open-mmlab/mmpretrain/tree/main/configs/mobileone)                        | MMPretrain       |      N      |      Y      |    Y     |  N   |   N   |    N     |   N    |  N   |
| [FCN](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/fcn)                                | MMSegmentation   |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   Y    |  Y   |

| [PSPNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/pspnet)[\*static](#note)         | MMSegmentation   |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   Y    |  Y   |
| [DeepLabV3](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/deeplabv3)                    | MMSegmentation   |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   Y    |  N   |
| [DeepLabV3+](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/deeplabv3plus)               | MMSegmentation   |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   Y    |  N   |
| [Fast-SCNN](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/fastscnn)[\*static](#note)    | MMSegmentation   |      Y      |      Y      |    Y     |  N   |   Y   |    Y     |   N    |  Y   |

| [UNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/unet)                              | MMSegmentation   |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   Y    |  Y   |
| [ANN](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/ann)[\*](#note)                     | MMSegmentation   |      Y      |      Y      |    Y     |  N   |   N   |    N     |   N    |  N   |
| [APCNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/apcnet)                          | MMSegmentation   |      Y      |      Y      |    Y     |  Y   |   N   |    N     |   N    |  Y   |
| [BiSeNetV1](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/bisenetv1)                    | MMSegmentation   |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |   N    |  Y   |

| [BiSeNetV2](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/bisenetv2)                    | MMSegmentation   |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |   N    |  N   |
| [CGNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/cgnet)                            | MMSegmentation   |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |   N    |  Y   |
| [DMNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/dmnet)                            | MMSegmentation   |      ?      |      Y      |    N     |  N   |   N   |    N     |   N    |  N   |
| [DNLNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/dnlnet)                          | MMSegmentation   |      ?      |      Y      |    Y     |  Y   |   N   |    Y     |   N    |  N   |

| [EMANet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/emanet)                          | MMSegmentation   |      Y      |      Y      |    Y     |  N   |   N   |    Y     |   N    |  N   |
| [EncNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/encnet)                          | MMSegmentation   |      Y      |      Y      |    Y     |  N   |   N   |    Y     |   N    |  N   |
| [ERFNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/erfnet)                          | MMSegmentation   |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |   N    |  Y   |
| [FastFCN](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/fastfcn)                        | MMSegmentation   |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |   N    |  N   |

| [GCNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/gcnet)                            | MMSegmentation   |      Y      |      Y      |    Y     |  N   |   N   |    N     |   N    |  N   |
| [ICNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/icnet)[\*](#note)                 | MMSegmentation   |      Y      |      Y      |    Y     |  N   |   N   |    Y     |   N    |  N   |
| [ISANet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/isanet)[\*static](#note)         | MMSegmentation   |      N      |      Y      |    Y     |  N   |   N   |    Y     |   N    |  Y   |
| [NonLocal Net](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/nonlocal_net)              | MMSegmentation   |      ?      |      Y      |    Y     |  Y   |   N   |    Y     |   N    |  N   |

| [OCRNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/ocrnet)                          | MMSegmentation   |      ?      |      Y      |    Y     |  Y   |   N   |    Y     |   N    |  Y   |
| [PointRend](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/point_rend)                   | MMSegmentation   |      Y      |      Y      |    Y     |  N   |   N   |    Y     |   N    |  N   |
| [Semantic FPN](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/sem_fpn)                   | MMSegmentation   |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |   N    |  Y   |
| [STDC](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/stdc)                              | MMSegmentation   |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |   N    |  Y   |

| [UPerNet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/upernet)[\*](#note)             | MMSegmentation   |      ?      |      Y      |    Y     |  N   |   N   |    N     |   N    |  Y   |
| [DANet](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/danet)                            | MMSegmentation   |      ?      |      Y      |    Y     |  N   |   N   |    N     |   N    |  N   |
| [Segmenter](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/segmenter) [\*static](#note)  | MMSegmentation   |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |   N    |  N   |
| [SRCNN](https://github.com/open-mmlab/mmagic/tree/main/configs/srcnn)                                    | MMagic           |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   N    |  N   |

| [ESRGAN](https://github.com/open-mmlab/mmagic/tree/main/configs/esrgan)                                  | MMagic           |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   N    |  N   |
| [SRGAN](https://github.com/open-mmlab/mmagic/tree/main/configs/srgan_resnet)                             | MMagic           |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   N    |  N   |
| [SRResNet](https://github.com/open-mmlab/mmagic/tree/main/configs/srgan_resnet)                          | MMagic           |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   N    |  N   |
| [Real-ESRGAN](https://github.com/open-mmlab/mmagic/tree/main/configs/real_esrgan)                        | MMagic           |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   N    |  N   |

| [EDSR](https://github.com/open-mmlab/mmagic/tree/main/configs/edsr)                                      | MMagic           |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |   N    |  N   |
| [RDN](https://github.com/open-mmlab/mmagic/tree/main/configs/rdn)                                        | MMagic           |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   N    |  N   |
| [DBNet](https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/dbnet)                             | MMOCR            |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |   Y    |  N   |
| [DBNetpp](https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/dbnetpp)                         | MMOCR            |      Y      |      Y      |    Y     |  ?   |   ?   |    Y     |   ?    |  N   |

| [PANet](https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/panet)                             | MMOCR            |      Y      |      Y      |    Y     |  Y   |   ?   |    Y     |   Y    |  N   |
| [PSENet](https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/psenet)                           | MMOCR            |      Y      |      Y      |    Y     |  Y   |   ?   |    Y     |   Y    |  N   |
| [TextSnake](https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/textsnake)                     | MMOCR            |      Y      |      Y      |    Y     |  Y   |   ?   |    ?     |   ?    |  N   |
| [MaskRCNN](https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/maskrcnn)                       | MMOCR            |      Y      |      Y      |    Y     |  ?   |   ?   |    ?     |   ?    |  N   |

| [CRNN](https://github.com/open-mmlab/mmocr/blob/main/configs/textrecog/crnn)                             | MMOCR            |      Y      |      Y      |    Y     |  Y   |   Y   |    N     |   N    |  N   |
| [SAR](https://github.com/open-mmlab/mmocr/blob/main/configs/textrecog/sar)                               | MMOCR            |      N      |      Y      |    N     |  N   |   N   |    N     |   N    |  N   |
| [SATRN](https://github.com/open-mmlab/mmocr/blob/main/configs/textrecog/satrn)                           | MMOCR            |      Y      |      Y      |    Y     |  N   |   N   |    N     |   N    |  N   |
| [ABINet](https://github.com/open-mmlab/mmocr/blob/main/configs/textrecog/abinet)                         | MMOCR            |      Y      |      Y      |    Y     |  N   |   N   |    N     |   N    |  N   |

| [HRNet](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#hrnet-cvpr-2019)         | MMPose           |      N      |      Y      |    Y     |  Y   |   N   |    Y     |   N    |  N   |
| [MSPN](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#mspn-arxiv-2019)          | MMPose           |      N      |      Y      |    Y     |  Y   |   N   |    Y     |   N    |  N   |
| [LiteHRNet](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#litehrnet-cvpr-2021) | MMPose           |      N      |      Y      |    Y     |  N   |   N   |    Y     |   N    |  N   |
| [Hourglass](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#hourglass-eccv-2016) | MMPose           |      N      |      Y      |    Y     |  Y   |   N   |    Y     |   N    |  N   |

| [SimCC](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/algorithms.html#simcc-eccv-2022)        | MMPose           |      N      |      Y      |    Y     |  Y   |   N   |    N     |   N    |  N   |
| [PointPillars](https://github.com/open-mmlab/mmdetection3d/tree/main/configs/pointpillars)               | MMDetection3d    |      ?      |      Y      |    Y     |  N   |   N   |    Y     |   N    |  N   |
| [CenterPoint (pillar)](https://github.com/open-mmlab/mmdetection3d/tree/main/configs/centerpoint)        | MMDetection3d    |      ?      |      Y      |    Y     |  N   |   N   |    Y     |   N    |  N   |
| [RotatedRetinaNet](https://github.com/open-mmlab/mmrotate/blob/main/configs/rotated_retinanet/README.md) | RotatedDetection |      N      |      Y      |    Y     |  N   |   N   |    N     |   N    |  N   |

| [Oriented RCNN](https://github.com/open-mmlab/mmrotate/blob/main/configs/oriented_rcnn/README.md)        | RotatedDetection |      N      |      Y      |    Y     |  N   |   N   |    N     |   N    |  N   |
| [Gliding Vertex](https://github.com/open-mmlab/mmrotate/blob/main/configs/gliding_vertex/README.md)      | RotatedDetection |      N      |      N      |    Y     |  N   |   N   |    N     |   N    |  N   |

## Note

- Tag:
  - static: This model only support static export. Please use `static` deploy config, just like $MMDEPLOY_DIR/configs/mmseg/segmentation_tensorrt_static-1024x2048.py.
- SSD: When you convert SSD model, you need to use min shape deploy config just like 300x300-512x512 rather than 320x320-1344x1344, for example $MMDEPLOY_DIR/configs/mmdet/detection/detection_tensorrt_dynamic-300x300-512x512.py.
- YOLOX: YOLOX with ncnn only supports static shape.
- Swin Transformer: For TensorRT, only version 8.4+ is supported.

- SAR: Chinese text recognition model is not supported as the protobuf size of ONNX is limited.

# Human pose regression with residual log-likelihood estimation

<details>

```bibtex
}
```

Heatmap-based methods dominate in the field of human pose estimation by modelling the output distribution through likelihood heatmaps. In contrast, regressionbased methods are more efficient but suffer from inferior performance. In this work, we explore maximum likelihood estimation (MLE) to develop an efficient and effective regression-based methods. From the perspective of MLE, adopting different regression losses is making different assumptions about the output density function. A density function closer to the true distribution leads to a better regression performance. In light of this, we propose a novel regression paradigm with Residual Log-likelihood Estimation (RLE) to capture the underlying output distribution. Concretely, RLE learns the change of the distribution instead of the unreferenced underlying distribution to facilitate the training process. With the proposed reparameterization design, our method is compatible with offthe-shelf flow models. The proposed method is effective, efficient and flexible. We show its potential in various human pose estimation tasks with comprehensive experiments. Compared to the conventional regression paradigm, regression with RLE bring 12.4 mAP improvement on MSCOCO without any test-time overhead. Moreover, for the first time, especially on multi-person pose estimation, our regression method is superior to the heatmap-based methods.

<img src="https://user-images.githubusercontent.com/15977946/166661858-e1149715-02cf-4393-b948-81c8e39f247d.png">
</div>

Annotation data is a time-consuming and laborious task. This article introduces how to perform semi-automatic annotation using the RTMDet algorithm in MMDetection in conjunction with Label-Studio software. Specifically, using RTMDet to predict image annotations and then refining the annotations with Label-Studio. Community users can refer to this process and methodology and apply it to other fields.

- RTMDet: RTMDet is a high-precision single-stage object detection algorithm developed by OpenMMLab, open-sourced in the MMDetection object detection toolbox. Its open-source license is Apache 2.0, and it can be used freely without restrictions by industrial users.

- [Label Studio](https://github.com/heartexlabs/label-studio) is an excellent annotation software covering the functionality of dataset annotation in areas such as image classification, object detection, and segmentation.

In this article, we will use [cat](https://download.openmmlab.com/mmyolo/data/cat_dataset.zip) images for semi-automatic annotation.

## Environment Configuration

To begin with, you need to create a virtual environment and then install PyTorch and MMCV. In this article, we will specify the versions of PyTorch and MMCV. Next, you can install MMDetection, Label-Studio, and label-studio-ml-backend using the following steps:

Create a virtual environment:

```shell
conda create -n rtmdet python=3.9 -y
conda activate rtmdet
```

Install PyTorch:

```shell
# Linux and Windows CPU only
pip install torch==1.10.1+cpu torchvision==0.11.2+cpu torchaudio==0.10.1 -f https://download.pytorch.org/whl/cpu/torch_stable.html
# Linux and Windows CUDA 11.3
pip install torch==1.10.1+cu113 torchvision==0.11.2+cu113 torchaudio==0.10.1 -f https://download.pytorch.org/whl/cu113/torch_stable.html
# OSX
pip install torch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1
```

Install MMCV:

```shell

# OSX
```

```shell
# Installing mmcv will automatically install mmengine
```

Install MMDetection:

```shell
git clone https://github.com/open-mmlab/mmdetection
```

Install Label-Studio and label-studio-ml-backend:

```shell
# Installing Label-Studio may take some time, if the version is not found, please use the official source
pip install label-studio==1.7.2
pip install label-studio-ml==1.0.9
```

Download the rtmdet weights:

```shell
cd path/to/mmetection
mkdir work_dirs
cd work_dirs
wget https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet_m_8xb32-300e_coco/rtmdet_m_8xb32-300e_coco_20220719_112220-229f527c.pth
```

## Start the Service

Start the RTMDet backend inference service:

```shell

label-studio-ml start projects/LabelStudio/backend_template --with \

```

```shell

config_file=configs/rtmdet/rtmdet_m_8xb32-300e_coco.py \
checkpoint_file=./work_dirs/rtmdet_m_8xb32-300e_coco_20220719_112220-229f527c.pth \
device=cpu \
--port 8003
# Set device=cpu to use CPU inference, and replace cpu with cuda:0 to use GPU inference.
```

![](https://cdn.vansin.top/picgo20230330131601.png)

The RTMDet backend inference service has now been started. To configure it in the Label-Studio web system, use http://localhost:8003 as the backend inference service.

Now, start the Label-Studio web service:

```shell
label-studio start
```

![](https://cdn.vansin.top/picgo20230330132913.png)

Open your web browser and go to http://localhost:8080/ to see the Label-Studio interface.

![](https://cdn.vansin.top/picgo20230330133118.png)

Register a user and then create an RTMDet-Semiautomatic-Label project.

![](https://cdn.vansin.top/picgo20230330133333.png)

Download the example cat images by running the following command and import them using the Data Import button:

```shell

wget https://download.openmmlab.com/mmyolo/data/cat_dataset.zip && unzip cat_dataset.zip
```

![](https://cdn.vansin.top/picgo20230330133628.png)

![](https://cdn.vansin.top/picgo20230330133715.png)

Then, select the Object Detection With Bounding Boxes template.

![](https://cdn.vansin.top/picgo20230330133807.png)

```shell
airplane
apple
backpack
banana
baseball_bat
baseball_glove
bear
bed
bench
bicycle
bird
boat
book
bottle
bowl
broccoli
bus
cake
car
carrot
cat
cell_phone
chair
clock
couch
cow
cup
dining_table
dog
donut
elephant
fire_hydrant
fork
frisbee
giraffe
hair_drier
handbag
horse
hot_dog
keyboard
kite
knife
laptop
microwave
motorcycle

car
carrot
cat
chair
clock
couch
cow
cup
dog
donut
elephant
fork
frisbee
giraffe
handbag
horse
hot_dog
keyboard
kite
knife
laptop
microwave
mouse
orange
oven
parking_meter
person
pizza
potted_plant
refrigerator
remote
sandwich
scissors
sheep
sink
skateboard
skis
snowboard
spoon
sports_ball
stop_sign
suitcase
surfboard
teddy_bear
tennis_racket
tie
toaster
toilet
toothbrush
traffic_light
train
truck
tv
umbrella
vase
wine_glass
zebra
```

Then, copy and add the above categories to Label-Studio and click Save.

![](https://cdn.vansin.top/picgo20230330134027.png)

In the Settings, click Add Model to add the RTMDet backend inference service.

![](https://cdn.vansin.top/picgo20230330134320.png)

Click Validate and Save, and then click Start Labeling.

![](https://cdn.vansin.top/picgo20230330134424.png)

If you see Connected as shown below, the backend inference service has been successfully added.

![](https://cdn.vansin.top/picgo20230330134554.png)

## Start Semi-Automatic Labeling

Click on Label to start labeling.

![](https://cdn.vansin.top/picgo20230330134804.png)

We can see that the RTMDet backend inference service has successfully returned the predicted results and displayed them on the image. However, we noticed that the predicted bounding boxes for the cats are a bit too large and not very accurate.

![](https://cdn.vansin.top/picgo20230403104419.png)

We manually adjust the position of the cat bounding box, and then click Submit to complete the annotation of this image.

![](https://cdn.vansin.top/picgo/20230403105923.png)

After submitting all images, click export to export the labeled dataset in COCO format.

![](https://cdn.vansin.top/picgo20230330135921.png)

Use VS Code to open the unzipped folder to see the labeled dataset, which includes the images and the annotation files in JSON format.

![](https://cdn.vansin.top/picgo20230330140321.png)

At this point, the semi-automatic labeling is complete. We can use this dataset to train a more accurate model in MMDetection and then continue semi-automatic labeling on newly collected images with this model. This way, we can iteratively expand the high-quality dataset and improve the accuracy of the model.

## Use MMYOLO as the Backend Inference Service

If you want to use Label-Studio in MMYOLO, you can refer to replacing the config_file and checkpoint_file with the configuration file and weight file of MMYOLO when starting the backend inference service.

```shell

```shell

config_file= path/to/mmyolo_config.py \
checkpoint_file= path/to/mmyolo_weights.pth \
# device=cpu is for using CPU inference. If using GPU inference, replace cpu with cuda:0.
```

Rotation object detection and instance segmentation are still under development, please stay tuned.

|operation|param id|param phase|default value|weight order|
|:---:|:---:|:---:|:---:|:---:|
|AbsVal|||
|ArgMax|0|out_max_val|0|
||1|topk|1|
|BatchNorm|0|channels|0|slope mean variance bias|
||1|eps|0.f|
|Bias|0|bias_data_size|0|
|BinaryOp|0|op_type|0|
||1|with_scalar|0|
||2|b|0.f|
|BNLL|||
|Cast|0|type_from|0|
||1|type_to|0|
|Clip|0|min|-FLT_MAX|
||1|max|FLT_MAX|
|Concat|0|axis|0|
|Convolution|0|num_output|0|weight bias|
||1|kernel_w|0|
||2|dilation_w|1|
||3|stride_w|1|
||4|pad_left|0|
||5|bias_term|0|
||6|weight_data_size|0|
||8|int8_scale_term|0|
||9|activation_type|0|
||10|activation_params|[ ]|
||11|kernel_h|kernel_w|
||12|dilation_h|dilation_w|
||13|stride_h|stride_w|
||15|pad_right|pad_left|
||14|pad_top|pad_left|
||16|pad_bottom|pad_top|
||17|impl_type|0|
||18|pad_value|0.f|
|ConvolutionDepthWise|0|num_output|0|weight bias|
||7|group|1|

|Crop|0|woffset|0|
||1|hoffset|0|
||2|coffset|0|
||3|outw|0|
||4|outh|0|
||5|outc|0|
||6|woffset2|0|
||7|hoffset2|0|
||8|coffset2|0|
||9|starts|[ ]|
||10|ends|[ ]|
||11|axes|[ ]|
|Deconvolution|0|num_output|0|weight bias|
||18|output_pad_right|0|

||19|output_pad_bottom|output_pad_right|
||20|output_w|0|
||21|output_h|output_w|
|DeconvolutionDepthWise|0|num_output|0|weight bias|
|Dequantize|0|scale|1.f|bias|
||1|bias_term|0|
||2|bias_data_size|0|
|DetectionOutput|0|num_class|0|
||1|nms_threshold|0.05f|
||2|nms_top_k|300|
||3|keep_top_k|100|
||4|confidence_threshold|0.5f|
||5|variances[0]|0.1f|
||6|variances[1]|0.1f|
||7|variances[2]|0.2f|

||8|variances[3]|0.2f|
|Dropout|0|scale|1.f|
|Eltwise|0|op_type|0|
||1|coeffs|[ ]|
|ELU|0|alpha|0.1f|
|Embed|0|num_output|0|weight bias|
||1|input_dim|0|
||2|bias_term|0|
||3|weight_data_size|0|
|Exp|0|base|-1.f|
||1|scale|1.f|
||2|shift|0.f|
|ExpandDims|0|expand_w|0|
||1|expand_h|0|
||2|expand_c|0|
||3|axes|[ ]|
|Flatten|||
|HardSigmoid|0|alpha|0.2f||
||1|beta|0.5f|
|HardSwish|0|alpha|0.2f||
|InnerProduct|0|num_output|0|weight bias|
||2|weight_data_size|0|
|Input|0|w|0|
||1|h|0|
||2|c|0|
|InstanceNorm|0|channels|0|gamma bias|
||1|eps|0.001f|
|Interp|0|resize_type|0|
||1|height_scale|1.f|
||2|width_scale|1.f|
||3|output_height|0|
||4|output_width|0|
|Log|0|base|-1.f|

|LRN|0|region_type|0|
||1|local_size|5|
||2|alpha|1.f|
||3|beta|0.75f|
||4|bias|1.f|
|LSTM|0|num_output|0|
||1|weight_data_size|1|
||2|direction|0|
|MemoryData|0|w|0|
||1|h|0|
||2|c|0|
|Mish|||
|MVN|0|normalize_variance|0|
||1|across_channels|0|
||2|eps|0.0001f|
|Noop|||
|Normalize|0|across_spatial|0|scale|
||4|across_channel|0|
||1|channel_shared|0|
||9|eps_mode|0|
||3|scale_data_size|0|
|Packing|0|out_packing|1|
||1|use_padding|0|
||2|cast_type_from|0|
||3|cast_type_to|0|
||4|storage_type_from|0|
||5|storage_type_to|0|
|Padding|0|top|0|per_channel_pad_data|
||1|bottom|0|
||2|left|0|
||3|right|0|
||4|type|0|
||5|value|0.f|
||6|per_channel_pad_data_size|0|
||7|front|0|
||8|behind|0|
|Permute|0|order_type|0|
|PixelShuffle|0|upscale_factor|1|

|Pooling|0|pooling_type(0: max 1: avg)|0|
||2|stride_w|1|
||12|stride_h|stride_w|
||3|pad_left|0|
||14|pad_right|pad_left|
||13|pad_top|pad_left|
||15|pad_bottom|pad_top|
||4|global_pooling|0|
||5|pad_mode|0|
|Power|0|power|1.f|
|PReLU|0|num_slope|0|slope|
|PriorBox|0|min_sizes|[ ]|
||1|max_sizes|[ ]|
||2|aspect_ratios|[ ]|
||3|varainces[0]|0.f|
||4|varainces[1]|0.f|
||5|varainces[2]|0.f|
||6|varainces[3]|0.f|
||7|flip|1|
||8|clip|0|
||9|image_width|0|
||10|image_height|0|
||11|step_width|-233.f|
||12|step_height|-233.f|
||13|offset|0.f|
||14|step_mmdetection|0|
||15|center_mmdetection|0|
|Proposal|0|feat_stride|16|
||1|base_size|16|
||2|pre_nms_topN|6000|
||3|after_nms_topN|300|
||4|num_thresh|0.7f|
||5|min_size|16|

|PSROIPooling|0|pooled_width|7|
||1|pooled_height|7|
||2|spatial_scale|0.0625f|
||3|output_dim|0|
|Quantize|0|scale|1.f|
|Reduction|0|operation|0|
||1|dim|0|
||2|coeff|1.f|
||4|keepdims|0|
|ReLU|0|slope|0.f|
|Reorg|0|stride|0|
|Requantize|0|scale_in|1.f|bias|
||1|scale_out|1.f|
||3|bias_data_size|0|
||4|fusion_relu|0|
|Reshape|0|w|-233|
||1|h|-233|
||2|c|-233|
||3|permute|0|
|ROIAlign|0|pooled_width|0|
||1|pooled_height|0|
||2|spatial_scale|1.f|
||3|sampling_ratio|0|
||4|aligned|0|
||5|version|0|
|ROIPooling|0|pooled_width|0|
|Scale|0|scale_data_size|0|scale bias|
|SELU|0|alpha|1.67326324f||
||1|lambda|1.050700987f|
|ShuffleChannel|0|group|1|
|Sigmoid|||
|Slice|0|slices|[ ]|
||1|axis|0|

|Softmax|0|axis|0|
|Split|||
|SPP|0|pooling_type|0|
||1|pyramid_height|1|
|Squeeze|0|squeeze_w|0|
||1|squeeze_h|0|
||2|squeeze_c|0|
|StatisticsPooling|0|include_stddev|0|
|Swish|||
|TanH|||
|Threshold|0|threshold|0.f|
|Tile|0|dim|0|
||1|tiles|1|
|UnaryOp|0|op_type|0|
|YoloDetectionOutput|0|num_class|20|
||1|num_box|5|
||2|confidence_threshold|0.01f|
||3|num_threshold|0.45f|
||4|biases|[]|
|Yolov3DetectionOutput|0|num_class|20|
||5|mask|[]|
||6|anchors_scale|[]|
|RNN|0|num_output|0|
||1|weight_data_size|0|
|MultiHeadAttention|0|embed_dim|0|
||1|num_head|1|

# ResNeSt

<img src="https://user-images.githubusercontent.com/26739999/142573827-a8189607-614b-4385-b579-b0db148b3db7.png" width="60%"/>
</div>

```
@misc{zhang2020resnest,
      title={ResNeSt: Split-Attention Networks},
      author={Hang Zhang and Chongruo Wu and Zhongyue Zhang and Yi Zhu and Haibin Lin and Zhi Zhang and Yue Sun and Tong He and Jonas Mueller and R. Manmatha and Mu Li and Alexander Smola},
      eprint={2004.08955},
}
```

# 使用现有模型进行推理

本文将展示如何使用以下API：

- [**`list_models`**](mmpretrain.apis.list_models): 列举 MMPretrain 中所有可用模型名称
- [**`get_model`**](mmpretrain.apis.get_model): 通过模型名称或模型配置文件获取模型
- [**`inference_model`**](mmpretrain.apis.inference_model): 使用与模型相对应任务的推理器进行推理。主要用作快速
  展示。如需配置进阶用法，还需要直接使用下列推理器。
- 推理器:
     对给定图像执行图像分类。
     从给定的一系列图像中，检索与给定图像最相似的图像。
     生成给定图像的一段描述。
     根据给定的图像回答问题。
     根据一段描述，从给定图像中找到一个与描述对应的对象。
     从给定的一系列图像中，检索与给定文本最相似的图像。

根据一段描述，从给定图像中找到一个与描述对应的对象。
     从给定的一系列文本中，检索与给定图像最相似的文本。
     对给定的一对图像和一段文本进行自然语言视觉推理（NLVR 任务）。
     通过视觉主干网络从图像文件提取特征。

## 列举可用模型

列出 MMPreTrain 中的所有已支持的模型。

```python
 ...]
```

`list_models` 支持 Unix 文件名风格的模式匹配，你可以使用 \*\* * \*\* 匹配任意字符。

```python
```

你还可以使用推理器的 `list_models` 方法获取对应任务可用的所有模型。

```

```python
```

## 获取模型

选定需要的模型后，你可以使用 `get_model` 获取特定模型。

```python

# 不加载预训练权重的模型

# 加载默认的权重文件

# 加载制定的权重文件

# 指定额外的模型初始化参数，例如修改 head 中的 num_classes。

# 另外一个例子：移除模型的 neck，head 模块，直接从 backbone 中的 stage 1, 2, 3 输出
```

获得的模型是一个通常的 PyTorch Module

```python
```

## 在给定图像上进行推理

这里是一个例子，我们将使用 ResNet-50 预训练模型对给定的 [图像](https://github.com/open-mmlab/mmpretrain/raw/main/demo/demo.JPEG) 进行分类。

```python
>>> # 如果你没有图形界面，请设置 `show=False`

sea snake
```

上述 `inference_model` 接口可以快速进行模型推理，但它每次调用都需要重新初始化模型，也无法进行多个样本的推理。
因此我们需要使用推理器来进行多次调用。

```python
>>> # 注意推理器的输出始终为一个结果列表，即使输入只有一个样本
sea snake
>>>
>>> # 你可以对多张图像进行批量推理
32
```

32
```

通常，每个样本的结果都是一个字典。比如图像分类的结果是一个包含了 `pred_label`、`pred_score`、`pred_scores`、`pred_class` 等字段的字典：

```python
{
}
```

你可以为推理器配置额外的参数，比如使用你自己的配置文件和权重文件，在 CUDA 上进行推理：

```python
sea snake
```

## 使用 Gradio 推理示例

sea snake
```

我们还提供了一个基于 gradio 的推理示例，提供了 MMPretrain 所支持的所有任务的推理展示功能，你可以在 [projects/gradio_demo/launch.py](https://github.com/open-mmlab/mmpretrain/blob/main/projects/gradio_demo/launch.py) 找到这一例程。

请首先使用 `pip install -U gradio` 安装 `gradio` 库。

这里是界面效果预览：

## 从图像中提取特征

与 `model.extract_feat` 相比，`FeatureExtractor` 用于直接从图像文件中提取特征，而不是从一批张量中提取特征。简单说，`model.extract_feat` 的输入是 `torch.Tensor`，`FeatureExtractor` 的输入是图像。

```

```
```

<details>

```bibtex
}
```

<details>

```bibtex
}
```

<details>

| [simcc_resnet_50](/configs/body_2d_keypoint/simcc/coco/simcc_res50_8xb64-210e_coco-256x192.py) |  256x192   | 0.721 |      0.897      |      0.798      | 0.781 |      0.937      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/simcc/coco/simcc_res50_8xb64-210e_coco-256x192-8e0f5b59_20220919.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/simcc/coco/simcc_res50_8xb64-210e_coco-256x192-8e0f5b59_20220919.log.json) |
| [simcc_resnet_50](/configs/body_2d_keypoint/simcc/coco/simcc_res50_8xb32-140e_coco-384x288.py) |  384x288   | 0.735 |      0.899      |      0.800      | 0.790 |      0.939      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/simcc/coco/simcc_res50_8xb32-140e_coco-384x288-45c3ba34_20220913.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/simcc/coco/simcc_res50_8xb32-140e_coco-384x288-45c3ba34_20220913.log.json) |

<details>

```bibtex
}
```

<details>
<summary align="right"><a href="https://arxiv.org/abs/2108.12617">AP-10K (NeurIPS'2021)</a></summary>

```bibtex
@misc{yu2021ap10k,
      title={AP-10K: A Benchmark for Animal Pose Estimation in the Wild},
      author={Hang Yu and Yufei Xu and Jing Zhang and Wei Zhao and Ziyu Guan and Dacheng Tao},
      eprint={2108.12617},
}
```

year={2021},
}
```

| Arch                                       | Input Size |  AP   | AP<sup>50</sup> | AP<sup>75</sup> | AP<sup>M</sup> | AP<sup>L</sup> |                    ckpt                     |                    log                     |
| :----------------------------------------- | :--------: | :---: | :-------------: | :-------------: | :------------: | :------------: | :-----------------------------------------: | :----------------------------------------: |

| [pose_resnet_50](/configs/animal_2d_keypoint/topdown_heatmap/ap10k/td-hm_res50_8xb64-210e_ap10k-256x256.py) |  256x256   | 0.680 |      0.926      |      0.738      |     0.552      |     0.687      | [ckpt](https://download.openmmlab.com/mmpose/animal/resnet/res50_ap10k_256x256-35760eb8_20211029.pth) | [log](https://download.openmmlab.com/mmpose/animal/resnet/res50_ap10k_256x256-35760eb8_20211029.log.json) |
| [pose_resnet_101](/configs/animal_2d_keypoint/topdown_heatmap/ap10k/td-hm_res101_8xb64-210e_ap10k-256x256.py) |  256x256   | 0.681 |      0.921      |      0.751      |     0.545      |     0.690      | [ckpt](https://download.openmmlab.com/mmpose/animal/resnet/res101_ap10k_256x256-9edfafb9_20211029.pth) | [log](https://download.openmmlab.com/mmpose/animal/resnet/res101_ap10k_256x256-9edfafb9_20211029.log.json) |

# Prepare Dataset

MMAction2 supports many existing datasets. In this chapter, we will lead you to prepare datasets for MMAction2.

- [Prepare Dataset](#prepare-dataset)
  - [Notes on Video Data Format](#notes-on-video-data-format)
  - [Use built-in datasets](#use-built-in-datasets)
  - [Use a custom dataset](#use-a-custom-dataset)
    - [Action Recognition](#action-recognition)
    - [Skeleton-based Action Recognition](#skeleton-based-action-recognition)
    - [Spatio-temporal Action Detection](#spatio-temporal-action-detection)
    - [Temporal Action Localization](#temporal-action-localization)
  - [Use mixed datasets for training](#use-mixed-datasets-for-training)
    - [Repeat dataset](#repeat-dataset)
  - [Browse dataset](#browse-dataset)

## Notes on Video Data Format

MMAction2 supports two types of data formats: raw frames and video. The former is widely used in previous projects such as [TSN](https://github.com/yjxiong/temporal-segment-networks).

This is fast when SSD is available but fails to scale to the fast-growing datasets.
(For example, the newest edition of [Kinetics](https://www.deepmind.com/open-source/kinetics) has 650K  videos and the total frames will take up several TBs.)
The latter saves much space but has to do the computation intensive video decoding at execution time.
To make video decoding faster, we support several efficient video loading libraries, such as [decord](https://github.com/zhreshold/decord), [PyAV](https://github.com/PyAV-Org/PyAV), etc.

MMAction2 already supports many datasets, we provide shell scripts for data preparation under the path `$MMACTION2/tools/data/`, please refer to [supported datasets](../datasetzoo_satatistics.md) for details to prepare specific datasets.

The simplest way is to convert your dataset to existing dataset formats:

- `RawFrameDataset` and `VideoDataset` for [Action Recognition](#action-recognition)
- `PoseDataset` for [Skeleton-based Action Recognition](#skeleton-based-action-recognition)
- `AVADataset` for [Spatio-temporal Action Detection](#spatio-temporal-action-detection)
- `ActivityNetDataset` for [Temporal Action Localization](#temporal-action-localization)

After the data pre-processing, the users need to further modify the config files to use the dataset.
Here is an example of using a custom dataset in rawframe format.

In `configs/task/method/my_custom_config.py`:

```python
...
...

...
        ...),
    val=dict(
        ...),
        ...))
...
```

### Action Recognition

There are two kinds of annotation files for action recognition.

- rawframe annotaiton for `RawFrameDataset`

  The annotation of a rawframe dataset is a text file with multiple lines,
  and each line indicates `frame_directory` (relative path) of a video,
  `total_frames` of a video and the `label` of a video, which are split by a whitespace.

  Here is an example.

  ```

Here is an example.

  ```
  ```

- video annotation for `VideoDataset`

  The annotation of a video dataset is a text file with multiple lines,
  and each line indicates a sample video with the `filepath` (relative path) and `label`,
  which are split by a whitespace.

  ```
  ```

### Skeleton-based Action Recognition

The task recognizes the action class based on the skeleton sequence (time sequence of keypoints). We provide some methods to build your custom skeleton dataset.

- Build from RGB video data

  You need to extract keypoints data from video and convert it to a supported format, we provide a [tutorial](https://github.com/open-mmlab/mmaction2/tree/main/configs/skeleton/posec3d/custom_dataset_training.md) with detailed instructions.

- Build from existing keypoint data

  Assuming that you already have keypoint data in coco formats, you can gather them into a pickle file.

  Each pickle file corresponds to an action recognition dataset. The content of a pickle file is a dictionary with two fields: `split` and `annotations`

  1. Split: The value of the `split` field is a dictionary: the keys are the split names, while the values are lists of video identifiers that belong to the specific clip.

1. Split: The value of the `split` field is a dictionary: the keys are the split names, while the values are lists of video identifiers that belong to the specific clip.
  2. Annotations: The value of the `annotations` field is a list of skeleton annotations, each skeleton annotation is a dictionary, containing the following fields:
     - `frame_dir` (str): The identifier of the corresponding video.
     - `total_frames` (int): The number of frames in this video.
     - `img_shape` (tuple\[int\]): The shape of a video frame, a tuple with two elements, in the format of `(height, width)`. Only required for 2D skeletons.
     - `original_shape` (tuple\[int\]): Same as `img_shape`.
     - `label` (int): The action label.
     - `keypoint` (np.ndarray, with shape `[M x T x V x C]`): The keypoint annotation.
       - M: number of persons;
       - T: number of frames (same as `total_frames`);

- `keypoint` (np.ndarray, with shape `[M x T x V x C]`): The keypoint annotation.
       - V: number of keypoints (25 for NTURGB+D 3D skeleton, 17 for CoCo, 18 for OpenPose, etc. );
       - C: number of dimensions for keypoint coordinates (C=2 for 2D keypoint, C=3 for 3D keypoint).
     - `keypoint_score` (np.ndarray, with shape `[M x T x V]`): The confidence score of keypoints. Only required for 2D skeletons.

  Here is an example:

  ```
  {
      "split":
          {
              ...
          }

          [
              {
                  {

[
              {
                  {
                  },
                  {
                      ...
                  },
                  ...

              }
          ]
  }
  ```

  Support other keypoint formats needs further modification, please refer to [customize dataset](../advanced_guides/customize_dataset.md).

### Spatio-temporal Action Detection

MMAction2 supports the task based on `AVADataset`. The annotation contains groundtruth bbox and proposal bbox.

- groundtruth bbox

  groundtruth bbox is a csv file with multiple lines, and each line is a detection sample of one frame, with following formats:

  each field means:
  `video_identifier` : The identifier of the corresponding video
  `time_stamp`: The time stamp of current frame
  `lt_x`: The normalized x-coordinate of the left top point of bounding box
  `lt_y`: The normalized y-coordinate of the left top point of bounding box
  `rb_y`: The normalized x-coordinate of the right bottom point of bounding box
  `rb_y`: The normalized y-coordinate of the right bottom point of bounding box
  `label`: The action label
  `entity_id`: a unique integer allowing this box to be linked to other boxes depicting the same person in adjacent frames of this video

`label`: The action label

  ```
  ...
  ```

- proposal bbox
  proposal bbox is a pickle file generated by a person detector, and usually needs to be fine-tuned on the target dataset. The pickle file contains a dict with below data structure:

  video_identifier (str): The identifier of the corresponding video
  time_stamp (int): The time stamp of current frame
  bbox_info (np.ndarray, with shape `[n, 5]`): Detected bbox, \<x1> \<y1> \<x2> \<y2> \<score>. x1, x2, y1, y2 are normalized with respect to frame size, which are between 0.0-1.0.

### Temporal Action Localization

We support Temporal Action Localization based on `ActivityNetDataset`. The annotation of ActivityNet dataset is a json file. Each key is a video name and the corresponding value is the meta data and annotation for the video.

```
{
          {
              ],
          }
      ],
  },
  }
  ...
}
```

MMAction2 also supports to mix dataset for training. Currently it supports to repeat dataset.

We use `RepeatDataset` as wrapper to repeat the dataset. For example, suppose the original dataset as `Dataset_A`,

to repeat it, the config looks like the following

```python
        times=N,
            ...
        )
    )
```

coming soon...

# LAD

> [Improving Object Detection by Label Assignment Distillation](https://arxiv.org/abs/2108.10520)

Label assignment in object detection aims to assign targets, foreground or background, to sampled regions in an image. Unlike labeling for image classification, this problem is not well defined due to the object's bounding box. In this paper, we investigate the problem from a perspective of distillation, hence we call Label Assignment Distillation (LAD). Our initial motivation is very simple, we use a teacher network to generate labels for the student. This can be achieved in two ways: either using the teacher's prediction as the direct targets (soft label), or through the hard labels dynamically assigned by the teacher (LAD). Our experiments reveal that: (i) LAD is more effective than soft-label, but they are complementary. (ii) Using LAD, a smaller teacher can also improve a larger student significantly, while soft-label can't. We then introduce Co-learning LAD, in which two networks simultaneously learn from scratch and the role of teacher and student are dynamically interchanged. Using PAA-ResNet50 as a teacher, our LAD techniques can improve detectors PAA-ResNet101 and PAA-ResNeXt101 to 46AP and 47.5AP on the COCO test-dev set. With a stronger teacher PAA-SwinB, we improve the students PAA-ResNet50 to 43.7AP by only 1x schedule training and standard setting, and PAA-ResNet101 to 47.9AP, significantly surpassing the current methods.

<img src="https://user-images.githubusercontent.com/40661020/143894499-c2a3a243-988f-4604-915b-17918732bf03.png"/>
</div>

We provide config files to reproduce the object detection results in the
WACV 2022 paper for Improving Object Detection by Label Assignment
Distillation.

### PAA with LAD

| Teacher | Student | Training schedule | AP (val) |                      Config                      |                                                                                                                                               Download                                                                                                                                               |

| :-----: | :-----: | :---------------: | :------: | :----------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|   --    |  R-50   |        1x         |   40.4   |     [config](../paa/paa_r50_fpn_1x_coco.py)      |                     [model](https://download.openmmlab.com/mmdetection/v2.0/paa/paa_r50_fpn_1x_coco/paa_r50_fpn_1x_coco_20200821-936edec3.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/paa/paa_r50_fpn_1x_coco/paa_r50_fpn_1x_coco_20200821-936edec3.log.json)                      |

|   --    |  R-101  |        1x         |   42.6   |     [config](../paa/paa_r101_fpn_1x_coco.py)     |                   [model](https://download.openmmlab.com/mmdetection/v2.0/paa/paa_r101_fpn_1x_coco/paa_r101_fpn_1x_coco_20200821-0a1825a4.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/paa/paa_r101_fpn_1x_coco/paa_r101_fpn_1x_coco_20200821-0a1825a4.log.json)                    |
|  R-101  |  R-50   |        1x         |   41.4   | [config](./lad_r50-paa-r101_fpn_2xb8_coco_1x.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/lad/lad_r50_paa_r101_fpn_coco_1x/lad_r50_paa_r101_fpn_coco_1x_20220708_124246-74c76ff0.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/lad/lad_r50_paa_r101_fpn_coco_1x/lad_r50_paa_r101_fpn_coco_1x_20220708_124246.log.json) |

|  R-50   |  R-101  |        1x         |   43.2   | [config](./lad_r101-paa-r50_fpn_2xb8_coco_1x.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/lad/lad_r101_paa_r50_fpn_coco_1x/lad_r101_paa_r50_fpn_coco_1x_20220708_124357-9407ac54.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/lad/lad_r101_paa_r50_fpn_coco_1x/lad_r101_paa_r50_fpn_coco_1x_20220708_124357.log.json) |

## Note

- Meaning of Config name: lad_r50(student model)\_paa(based on paa)\_r101(teacher model)\_fpn(neck)\_coco(dataset)\_1x(12 epoch).py
- Results may fluctuate by about 0.2 mAP.
- 2 GPUs are used, 8 samples per GPU.

```latex
@inproceedings{nguyen2021improving,
  title={Improving Object Detection by Label Assignment Distillation},
  author={Chuong H. Nguyen and Thuy C. Nguyen and Tuan N. Tang and Nam L. H. Phan},
  booktitle = {WACV},
}
```

# Preparing VideoLQ Dataset

```bibtex
@inproceedings{chan2022investigating,
  title = {Investigating Tradeoffs in Real-World Video Super-Resolution},
  booktitle = {IEEE Conference on Computer Vision and Pattern Recognition},
}
```

You can download the dataset using [Dropbox](https://www.dropbox.com/sh/hc06f1livdhutbo/AAAMPy92EOqVjRN8waT0ie8ja?dl=0) / [Google Drive](https://drive.google.com/drive/folders/1-1iJRNdqdFZWOnoUU4xG1Z1QhwsGwMDy?usp=sharing) / [OneDrive](https://entuedu-my.sharepoint.com/:f:/g/personal/chan0899_e_ntu_edu_sg/ErSugvUBxoBMlvSAHhqT5BEB9-4ZaqxzJIcc9uvVa8JGHg?e=WpHJTc).

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── VideoLQ
│   │   ├── 000
│   │   │   ├── 00000000.png
│   │   │   ├── 00000001.png
│   │   │   ├── ...
│   │   ├── 001
│   │   ├── 002
│   │   ├── ...
```

# RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose

<details>

```bibtex
}

```

Recent studies on 2D pose estimation have achieved excellent performance on public benchmarks, yet its application in the industrial community still suffers from heavy model parameters and high latency. In order to bridge this gap, we empirically explore key factors in pose estimation including paradigm, model architecture, training strategy, and deployment, and present a high-performance real-time multi-person pose estimation framework, RTMPose, based on MMPose. Our RTMPose-m achieves 75.8% AP on COCO with 90+ FPS on an Intel i7-11700 CPU and 430+ FPS on an NVIDIA GTX 1660 Ti GPU, and RTMPose-l achieves 67.0% AP on COCO-WholeBody with 130+ FPS. To further evaluate RTMPose’s capability in critical real-time applications, we also report the performance after deploying on the mobile device. Our RTMPoses achieves 72.2% AP on COCO with 70+ FPS on a Snapdragon 865 chip, outperforming existing open-source libraries. Code and models are released at https://github.com/open-mmlab/mmpose/tree/main/projects/rtmpose.

<img src="https://user-images.githubusercontent.com/15977946/225237680-6e3640c1-7b03-4cd9-9bb2-ca28adb8710d.png">
</div>

# 准备 GYM

## 简介

```BibTeX
}
```

请参照 [项目主页](https://sdolivia.github.io/FineGym/) 及 [原论文](https://sdolivia.github.io/FineGym/) 以获取数据集基本信息。
MMAction2 当前支持 GYM99 的数据集预处理。
在开始之前，用户需确保当前目录为 `$MMACTION2/tools/data/gym/`。

## 1. 准备标注文件

首先，用户可以使用如下脚本下载标注文件并进行预处理：

```shell
```

## 2. 准备视频

用户可以使用以下脚本准备视频，视频准备代码修改自 [ActivityNet 爬虫](https://github.com/activitynet/ActivityNet/tree/master/Crawler/Kinetics)。
注意这一步骤将花费较长时间。

```shell
```

## 3. 裁剪长视频至动作级别

用户首先需要使用以下脚本将 GYM 中的长视频依据标注文件裁剪至动作级别。

```shell
```

## 4. 裁剪动作视频至分动作级别

```shell
```

```shell
```

随后，用户需要使用以下脚本将 GYM 中的动作视频依据标注文件裁剪至分动作级别。将视频的裁剪分成两个级别可以带来更高的效率（在长视频中裁剪多个极短片段异常耗时）。

```shell
```

## 5. 提取 RGB 帧和光流

如果用户仅使用 video loader，则可以跳过本步。

在提取之前，请参考 [安装指南](/docs/zh_cn/get_started/installation.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。

用户可使用如下脚本同时抽取 RGB 帧和光流（提取光流时使用 tvl1 算法）：

```shell
```

## 6. 基于提取出的分动作生成文件列表

用户可使用以下脚本为 GYM99 生成训练及测试的文件列表：

```shell
```

## 7. 目录结构

在完整完成 GYM 的数据处理后，将得到帧文件夹（RGB 帧和光流帧），动作视频片段，分动作视频片段以及训练测试所用标注文件。

在整个项目目录下（仅针对 GYM），完整目录结构如下所示：

```
mmaction2
├── tools
├── data
│   ├── gym

│   ├── gym
|   |   |   ├── ...
|   |   |   ├── ...
|   |   |   ├── ...
```

关于 GYM 数据集上的训练与测试，请参照 [训练教程](/docs/zh_cn/user_guides/train_test.md)。

# 自定义训练配置

## 自定义优化相关的配置

优化相关的配置现在已全部集成到 `optim_wrapper` 中，通常包含三个域：`optimizer`, `paramwise_cfg`，`clip_grad`，具体细节见 [OptimWrapper](https://mmengine.readthedocs.io/en/latest/tutorials/optim_wrapper.md)。下面这个例子中，使用了 `AdamW` 作为优化器，主干部分的学习率缩小到原来的十分之一，以及添加了梯度裁剪。

```python
    # 优化器
        eps=1e-8,

    # 参数层面的学习率和正则化设置
        },

    # 梯度裁剪
```

### 自定义 Pytorch 中优化器设置

我们已经支持了 Pytorch 中实现的所有优化器，要使用这些优化器唯一要做就是修改配置文件中的 `optimi_wrapper` 中的 `optimzer` 域。比如，如果想要使用 `ADAM` 作为优化器（可能会导致性能下降），所需要做的修改如下。

```python
```

```python
```

要修改模型的学习率，用户只需要修改 `optimizer` 中的 `lr` 域。用户可以直接参考 PyToch 的 [API doc](https://pytorch.org/docs/stable/optim.html?highlight=optim#module-torch.optim) 来进行参数的设置。

### 自定义优化器

#### 1. 定义一个新优化器

自定义优化器可以定义的方式如下：

假设你想要添加一个名为 `MyOptimizer` 的优化器，它包含三个参数 `a`，`b`，`c`。你需要新建一个名为
`mmdet/engine/optimizers` 的文件夹。然后在文件（比如，`mmdet/engine/optimizers/my_optimizer.py`）实现一个新的优化器。

```python
from mmdet.registry import OPTIMIZERS

```

#### 2. 导入自定义的优化器

为了能找到上面的所定义的模块，这个模块必须要先导入到主命名空间中。有两种方式可以实现这一点。

- 修改 `mmdet/engine/optimizers/__init__.py` 来导入模块。

  新定义的模块必须导入到 `mmdet/engine/optimizers/__init__.py`，这样注册器才能找到该模块并添加它。

```python
```

- 在配置文件使用 `custom_imports` 来手动导入模块。

```python

新定义的模块必须导入到 `mmdet/engine/optimizers/__init__.py`，这样注册器才能找到该模块并添加它。

```python
```

```python
custom_imports = dict(imports=['mmdet.engine.optimizers.my_optimizer'], allow_failed_imports=False)
```

`mmdet.engine.optimizers.my_optimizer` 模块将在程序开始时导入，之后 `MyOptimizer` 类会被自动注册。注意：应该导入 `MyOptimizer` 所在的文件，即 `mmdet.engine.optimizers.my_optimizer`，而不是 `mmdet.engine.optimizers.my_optimizer.MyOptimizer`。

实际上，用户也可以在别的目录结构下来进行导入模块，只要改模块可以在 `PYTHONPATH` 中找到。

接下来，你可以在配置文件中的 `optim_wrapper` 域中的中 `optimizer` 域中设置你实现的优化器 `MyOptimizer`。在配置文件中，优化器在 `optimizer` 域中的配置方式如下：

```python
```

为了使用你的优化器，可以进行如下修改

```python
```

### 自定义优化器包装构造类

```

```python
```

一些模型可能存在一些特定参数的优化设置，比如，BN 层的权重衰减。用户可以通过自定义优化器包装构造类来实现这些精细化的参数调整。

```python
from mmengine.optim import DefaultOptiWrapperConstructor

from mmdet.registry import OPTIM_WRAPPER_CONSTRUCTORS

```

优化器包装构造类的具体实现见[这里](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/optimizer/default_constructor.py#L18)，用户以它为模板，来实现新的优化器包装构造类。

### 额外的设置

一些没有被优化器实现的技巧（比如，参数层面的学习率设置）应该通过优化器包装构造类来实现或者钩子。我们列出了一些常用的设置用于稳定训练或者加速训练。请随意创建 PR，发布更多设置。

### 额外的设置

- __使用梯度裁剪来稳定训练__:
  一些模型需要进行梯度裁剪来稳定训练过程，例子如下：

  ```python
  ```

  如果你的配置已经集成了基础配置（包含了 `optim_wrapper` 的配置），那么你需要添加 `_delete_=True` 来覆盖掉不需要的设置。具体见[配置相关的文档](https://mmdetection.readthedocs.io/en/latest/tutorials/config.html)。

  我们支持动量调度器根据学习率修改模型的动量，这可以使模型以更快的方式收敛。动量调度器通常与学习率调度器一起使用，例如 [3D 检测](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/schedules/cyclic-20e.py) 中使用以下配置以加速收敛。
  更多细节请参考 [CosineAnnealingLR](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py#L43) 和 [CosineAnnealingMomentum](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/momentum_scheduler.py#L71) 的具体实现。

  ```python
      # 学习率调度器
      # 在前 8 个 epoch, 学习率从 0 增大到 lr * 10

```python
      # 学习率调度器
      # 在接下来 12 个 epoch, 学习率从 lr * 10 减小到 lr * 1e-4
      dict(
          T_max=8,
          begin=0,
          end=8,
      dict(
          T_max=12,
          begin=8,
          end=20,
      # 动量调度器
      # 在前 8 个 epoch, 动量从 0 增大到 0.85 / 0.95
      # 在接下来 12 个 epoch, 学习率从 0.85 / 0.95 增大到 1
      dict(
          T_max=8,
          begin=0,
          end=8,
      dict(
          T_max=12,
          begin=8,
          end=20,

by_epoch=True,
      dict(
          T_max=12,
          begin=8,
          end=20,
  ]
  ```

## 自定义训练策略

默认情况下，我们使用 1x 的学习率调整策略，这会条用 MMEngine 中的 [MultiStepLR](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py#L139)。
我们支持许多其他学习率调整策略，具体见[这里](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py)，例如 `CosineAnnealingLR` 和 `PolyLR` 策略。下面有些例子

- 多项式学习率调整策略:

  ```python
      dict(
          begin=0,
          end=8,
  ```

- 余弦退火学习率调整策略

  ```python
      dict(
          T_max=8,
          begin=0,
          end=8,

```python
      dict(
          T_max=8,
          begin=0,
          end=8,

  ```

## 自定义训练循环

默认情况下，在 `train_cfg` 中使用 `EpochBasedTrainLoop`，并且在每个 epoch 训练之后进行验证，如下所示。

```python
```

实际上，[`IterBasedTrainLoop`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L183%5D) 和\[`EpochBasedTrainLoop`\](https:// github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L18) 支持动态区间的方式进行验证，见下例。

```python
# 在第 365001 次迭代之前，我们每 5000 次迭代进行一次评估。
# 在第 365000 次迭代后，我们每 368750 次迭代进行一次评估，
# 这意味着我们在训练结束时进行评估。

```

## 自定义钩子

```

## 自定义钩子

### 自定义自行实现的钩子

#### 1. 实现一个新的钩子

MMEngine 提供了许多有用的[钩子](https://mmdetection.readthedocs.io/en/latest/tutorials/hooks.html)，但在某些情况下用户可能需要实现新的钩子。MMDetection 在 v3.0 中支持自定义钩子。因此，用户可以直接在 mmdet 或其基于 mmdet 的代码库中实现钩子，并通过仅在训练中修改配置来使用钩子。
这里我们给出一个在 mmdet 中创建一个新的钩子并在训练中使用它的例子。

```python

                          runner,

                          runner,

                         runner,
```

根据钩子的功能，用户需要在 `before_run`、`after_run`、`before_train`、`after_train`、`before_train_epoch`、`after_train_epoch`、`before_train_iter` 和 `after_train_iter`。还有更多可以插入钩子的点，更多细节请参考 [base hook class](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/hook.py#L9)。

然后我们需要导入 `MyHook`。假设该文件位于 `mmdet/engine/hooks/my_hook.py` 中，有两种方法可以做到这一点：

- 修改 `mmdet/engine/hooks/__init__.py` 以导入它。

  新定义的模块应该在 `mmdet/engine/hooks/__init__.py` 中导入，以便注册表找到新模块并添加它：

```python
```

- 在配置中使用 `custom_imports` 手动导入它

```python
```

```python
custom_imports = dict(imports=['mmdet.engine.hooks.my_hook'], allow_failed_imports=False)
```

```python
]
```

你还可以通过修改键 `priority` 的值为 `NORMAL` 或 `HIGHEST` 来设置挂钩的优先级，如下所示

```python
]
```

默认情况下，钩子的优先级在注册期间设置为 `NORMAL`。

### 使用 MMDetection 中实现的钩子

如果 MMDetection 中已经实现了该钩子，你可以直接修改配置以使用该钩子，如下所示

#### 例子: `NumClassCheckHook`

我们实现了一个名为 [NumClassCheckHook](https://github.com/open-mmlab/mmdetection/blob/main/mmdet/engine/hooks/num_class_check_hook.py) 的自定义钩子来检查 `num_classes` 是否在 head 中和 `dataset` 中的 `classes` 的长度相匹配。

我们在 [default_runtime.py](https://github.com/open-mmlab/mmdetection/blob/main/configs/_base_/default_runtime.py) 中设置它。

```python
```

### 修改默认运行时钩子

有一些常见的钩子是通过 `default_hooks` 注册的，它们是

- `IterTimerHook`：记录 “data_time” 用于加载数据和 “time” 用于模型训练步骤的钩子。
- `LoggerHook`：从`Runner`的不同组件收集日志并将它们写入终端、JSON文件、tensorboard和 wandb 等的钩子。
- `ParamSchedulerHook`：更新优化器中一些超参数的钩子，例如学习率和动量。
- `CheckpointHook`：定期保存检查点的钩子。
- `DistSamplerSeedHook`：为采样器和批处理采样器设置种子的钩子。
- `DetVisualizationHook`：用于可视化验证和测试过程预测结果的钩子。

`IterTimerHook`、`ParamSchedulerHook` 和 `DistSamplerSeedHook` 很简单，通常不需要修改，所以这里我们将展示如何使用 `LoggerHook`、`CheckpointHook` 和 `DetVisualizationHook`。

除了定期保存检查点，[`CheckpointHook`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L19) 提供了其他选项，例如`max_keep_ckpts`、`save_optimizer ` 等。用户可以设置 `max_keep_ckpts` 只保存少量检查点或通过 `save_optimizer` 决定是否存储优化器的状态字典。参数的更多细节在[这里](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L19)可以找到。

```python
```

`LoggerHook` 可以设置间隔。详细用法可以在 [docstring](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/logger_hook.py#L18) 中找到。

```python
```

#### DetVisualizationHook

```python
```

`DetVisualizationHook` 使用 `DetLocalVisualizer` 来可视化预测结果，`DetLocalVisualizer` 支持不同的后端，例如 `TensorboardVisBackend` 和 `WandbVisBackend` （见 [docstring](https://github.com/open-mmlab/mmengine/blob/main/mmengine/visualization/vis_backend.py) 了解更多细节）。用户可以添加多个后端来进行可视化，如下所示。

```python
    visualization=dict(type='DetVisualizationHook', draw=True))

```

# Hyper-parameter Scheduler Visualization

This tool aims to help the user to check the hyper-parameter scheduler of the optimizer (without training), which support the "learning rate" or "momentum"

## Introduce the scheduler visualization tool

```bash
python tools/visualization/vis_scheduler.py \
```

- `config`: The path of a model config file.
- **`-p, --parameter`**: The param to visualize its change curve, choose from "lr" and "momentum". Default to use "lr".
- **`-d, --dataset-size`**: The size of the datasets. If set，`build_dataset` will be skipped and `${DATASET_SIZE}` will be used as the size. Default to use the function `build_dataset`.

- **`-n, --ngpus`**: The number of GPUs used in training, default to be 1.
- **`-s, --save-path`**: The learning rate curve plot save path, default not to save.
- `--title`: Title of figure. If not set, default to be config file name.
- `--style`: Style of plt. If not set, default to be `whitegrid`.
- `--window-size`: The shape of the display window. If not specified, it will be set to `12*7`. If used, it must be in the format `'W*H'`.

```{note}
Loading annotations maybe consume much time, you can directly specify the size of the dataset with `-d, dataset-size` to save time.
```

## How to plot the learning rate curve without training

```

You can use the following command to plot the step learning rate schedule used in the config `configs/swin_transformer/swin-base_16xb64_in1k.py`:

```bash
python tools/visualization/vis_scheduler.py configs/swin_transformer/swin-base_16xb64_in1k.py --dataset-size 1281167 --ngpus 16
```

<div align=center><img src="https://user-images.githubusercontent.com/26739999/226544329-cf3a3d45-6ab3-48aa-8972-2c2a58c35e62.png" style=" width: auto; height: 40%; "></div>

# KV Cache Quantization and Test Results

For the LLaMa-7B fp16 model with a maximum length of 2048, the server requires approximately 1030MB of GPU memory to store kv_cache for each concurrent session created. This means that even an A100 80G can only serve a limited number of users.

To reduce runtime GPU memory usage, we have implemented PTQ quantization for kv cache, using the following formula:

```bash
zp = (min+max) / 2
scale = (max-min) / 255
quant: q = round( (f-zp) / scale)
dequant: f = q * scale + zp
```

## How to Enable KV Cache INT8

### **Step One**

Convert the Hugging Face model format to the TurboMind inference format to create a workspace directory.

```bash
```

If you already have a workspace directory, skip this step.

### **Step Two**

Get the quantization parameters.

```bash
python3 -m lmdeploy.lite.apis.kv_qparams \

```

```bash
  --work_dir /path/to/internlm-chat-7b  \             # Directory of the Hugging Face model
  --turbomind_dir workspace/trition_models/weights/ \ # Directory to save the quantization parameters
  --kv_sym False \                                    # Symmetric or asymmetric quantization, default is False
  --num_tp 1  \                                       # Number of GPUs used for Tensor parallelization, keep it consistent with deploy.py
```

`kv_qparams` will generate fp32 scaling factors in the `weights` directory. The file format is a binary produced by `numpy.tofile`.

You can also first set `turbomind_dir` to a private directory, then copy the scaling factors into `workspace/trition_models/weights/`.

### **Step Three**

Modify `workspace/trition_models/weights/config.ini`:

- Set use_context_fmha to 0, which means turning off flashattention
- Set quant_policy to 4. This means enabling kv_cache int8

This is because there are two versions of flashattention, v1 and v2, and kv_cache int8 has also previously realized the symmetric version.

Considering there are four combinations of kernels needed to be implemented, premature optimization when the algorithm is uncertain can be disastrous for software.

### **Step Four**

Test the chat performance.

```bash
```

## GPU Memory Test

The test object is the [internlm-chat-7b](https://huggingface.co/internlm/internlm-chat-7b) model.
Testing method:

1. Use `deploy.py` to convert the model, modify the maximum concurrency in the `workspace` configuration; adjust the number of requests in `llama_config.ini`.

2. Compile and run `bin/llama_triton_example` to obtain the GPU memory situation of the fp16 version under different batch_size.
3. Enable quantization, re-run `bin/llama_triton_example` to obtain the GPU memory situation of the int8 version under different batch_size.

Below shows the comparison of GPU memory between the two versions:

| batch_size | fp16 memory(MiB) | int8 memory(MiB) | diff(MiB) |
| :--------: | :--------------: | :--------------: | :-------: |
|     8      |      22337       |      18241       |   -4096   |
|     16     |      30593       |      22369       |   -8224   |
|     32     |      47073       |      30625       |  -16448   |
|     48     |      63553       |      38881       |  -24672   |

Compared to directly quantizing Weight (such as [GPTQ-for-LLaMa](https://github.com/qwopqwop200/GPTQ-for-LLaMa/)), we have done a comparative estimation of memory growth in the 7B model for both methods, with some data from [llama.cpp](https://github.com/ggerganov/llama.cpp).

![](../../resources/batch_memory.png)

As can be seen, the fp16 version requires 1030MB of GPU memory for each concurrency, so quantizing kv_cache can significantly reduce the rate of increase of runtime memory.

## Accuracy Test

The test object is the [internlm-chat-7b](https://huggingface.co/internlm/internlm-chat-7b) command model.

Below is the result of PTQ quantization of `kCacheKVInt8` method with only 128 randomly selected data from the c4 dataset. The accuracy was tested using [opencompass](https://github.com/InternLM/opencompass) before and after quantization.

|     task      |     dataset     |    metric     | int8  | fp16  | diff  |
| :-----------: | :-------------: | :-----------: | :---: | :---: | :---: |
|   Language    |   winogrande    |   accuracy    | 60.77 | 61.48 | -0.71 |
|   Knowledge   |       nq        |     score     | 2.69  | 2.60  | +0.09 |
|   Reasoning   |      gsm8k      |   accuracy    | 33.28 | 34.72 | -1.44 |
|   Reasoning   |       bbh       | naive_average | 20.12 | 20.51 | -0.39 |
| Understanding | openbookqa_fact |   accuracy    | 82.40 | 82.20 | +0.20 |
| Understanding |   eprstmt-dev   |   accuracy    | 90.62 | 88.75 | +1.87 |

|    Safety     |   crows_pairs   |   accuracy    | 32.56 | 31.43 | +1.13 |

Note that both `kCacheKVInt8` and `WeightInt4` methods can be enabled at the same time.

# Wide-ResNet

> [Wide Residual Networks](https://arxiv.org/abs/1605.07146)

Deep residual networks were shown to be able to scale up to thousands of layers and still have improving performance. However, each fraction of a percent of improved accuracy costs nearly doubling the number of layers, and so training very deep residual networks has a problem of diminishing feature reuse, which makes these networks very slow to train. To tackle these problems, in this paper we conduct a detailed experimental study on the architecture of ResNet blocks, based on which we propose a novel architecture where we decrease depth and increase width of residual networks. We call the resulting network structures wide residual networks (WRNs) and show that these are far superior over their commonly used thin and very deep counterparts. For example, we demonstrate that even a simple 16-layer-deep wide residual network outperforms in accuracy and efficiency all previous deep residual networks, including thousand-layer-deep networks, achieving new state-of-the-art results on CIFAR, SVHN, COCO, and significant improvements on ImageNet.

<img src="https://user-images.githubusercontent.com/26739999/156701329-2c7ec7bc-23da-401b-86bf-dea8567ccee8.png" width="90%"/>
</div>

```python

predict = inference_model('wide-resnet50_3rdparty_8xb32_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('wide-resnet50_3rdparty_8xb32_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/wrn/wide-resnet50_8xb32_in1k.py https://download.openmmlab.com/mmclassification/v0/wrn/wide-resnet50_3rdparty_8xb32_in1k_20220304-66678344.pth
```

| Model                                      |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                   Config                   |                              Download                               |
| :----------------------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :----------------------------------------: | :-----------------------------------------------------------------: |

| `wide-resnet50_3rdparty_8xb32_in1k`\*      | From scratch |   68.88    |   11.44   |   78.48   |   94.08   |   [config](wide-resnet50_8xb32_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/wrn/wide-resnet50_3rdparty_8xb32_in1k_20220304-66678344.pth) |
| `wide-resnet101_3rdparty_8xb32_in1k`\*     | From scratch |   126.89   |   22.81   |   78.84   |   94.28   |   [config](wide-resnet101_8xb32_in1k.py)   | [model](https://download.openmmlab.com/mmclassification/v0/wrn/wide-resnet101_3rdparty_8xb32_in1k_20220304-8d5f9d61.pth) |
| `wide-resnet50_3rdparty-timm_8xb32_in1k`\* | From scratch |   68.88    |   11.44   |   81.45   |   95.53   | [config](wide-resnet50_timm_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/wrn/wide-resnet50_3rdparty-timm_8xb32_in1k_20220304-83ae4399.pth) |

*Models with * are converted from the [timm](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/resnet.py). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@INPROCEEDINGS{Zagoruyko2016WRN,
    author = {Sergey Zagoruyko and Nikos Komodakis},
    title = {Wide Residual Networks},
    booktitle = {BMVC},
    year = {2016}}
```

# 数据集格式转换脚本

MMPose 提供了一些工具来帮助用户处理数据集。

## Animal Pose 数据集

<details>

```bibtex
}
```

对于 [Animal-Pose](https://sites.google.com/view/animal-pose/)，可以从[官方网站](https://sites.google.com/view/animal-pose/)下载图像和标注。脚本 `tools/dataset_converters/parse_animalpose_dataset.py` 将原始标注转换为 MMPose 兼容的格式。预处理的[标注文件](https://download.openmmlab.com/mmpose/datasets/animalpose_annotations.tar)可用。如果您想自己生成标注，请按照以下步骤操作：

1. 下载图片与标注信息并解压到 `$MMPOSE/data`，按照以下格式组织：

   ```text

   ```text
   mmpose
   ├── mmpose
   ├── docs
   ├── tests
   ├── tools
   ├── configs
   `── data
       │── animalpose
           │
           │-- VOC2012
           │   │-- Annotations
           │   │-- ImageSets
           │   │-- JPEGImages
           │   │-- SegmentationClass
           │   │-- SegmentationObject
           │
           │-- animalpose_image_part2
           │   │-- cat
           │   │-- cow
           │   │-- dog
           │   │-- horse
           │   │-- sheep
           │
           │-- PASCAL2011_animal_annotation
           │   │-- cat
           │   │   |-- 2007_000528_1.xml
           │   │   |-- 2007_000549_1.xml
           │   │   │-- ...
           │   │-- cow
           │   │-- dog
           │   │-- horse
           │   │-- sheep
           │
           │-- annimalpose_anno2
           │   │-- cat
           │   │   |-- ca1.xml
           │   │   |-- ca2.xml
           │   │   │-- ...

│   │-- sheep
           │
           │   │-- cat
           │   │   │-- ...
           │   │-- cow
           │   │-- dog
           │   │-- horse
           │   │-- sheep
   ```

2. 运行脚本

   ```bash
   python tools/dataset_converters/parse_animalpose_dataset.py
   ```

   生成的标注文件将保存在 `$MMPOSE/data/animalpose/annotations` 中。

开源作者没有提供官方的 train/val/test 划分，我们选择来自 PascalVOC 的图片作为 train & val，train+val 一共 3600 张图片，5117 个标注。其中 2798 张图片，4000 个标注用于训练，810 张图片，1117 个标注用于验证。测试集包含 1000 张图片，1000 个标注用于评估。

## COFW 数据集

<details>

```bibtex

```bibtex
}
```

对于 COFW 数据集，请从 [COFW Dataset (Color Images)](https://data.caltech.edu/records/20099) 进行下载。

将 `COFW_train_color.mat` 和 `COFW_test_color.mat` 移动到 `$MMPOSE/data/cofw/`，确保它们按照以下格式组织：

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── cofw
        |── COFW_train_color.mat
        |── COFW_test_color.mat
```

运行 `pip install h5py` 安装依赖，然后在 `$MMPOSE` 下运行脚本：

```bash
python tools/dataset_converters/parse_cofw_dataset.py
```

最终结果为：

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── cofw
        |   |── cofw_train.json

├── mmpose
├── docs
├── tests
├── tools
`── data
    │── cofw
        |   |── cofw_test.json
        |── images
            |── 000001.jpg
            |── 000002.jpg
```

## DeepposeKit 数据集

<details>

```bibtex
}
```

journal={Elife},
}
```

对于 [Vinegar Fly](https://github.com/jgraving/DeepPoseKit-Data)，[Desert Locust](https://github.com/jgraving/DeepPoseKit-Data), 和 [Grévy’s Zebra](https://github.com/jgraving/DeepPoseKit-Data) 数据集，请从 [DeepPoseKit-Data](https://github.com/jgraving/DeepPoseKit-Data) 下载数据。

`tools/dataset_converters/parse_deepposekit_dataset.py` 脚本可以将原始标注转换为 MMPose 支持的格式。我们已经转换好的标注文件可以在这里下载：

- [vinegar_fly_annotations](https://download.openmmlab.com/mmpose/datasets/vinegar_fly_annotations.tar)
- [locust_annotations](https://download.openmmlab.com/mmpose/datasets/locust_annotations.tar)
- [zebra_annotations](https://download.openmmlab.com/mmpose/datasets/zebra_annotations.tar)

如果你希望自己转换数据，请按照以下步骤操作：

1. 下载原始图片和标注，并解压到 `$MMPOSE/data`，将它们按照以下格式组织：

   ```text
   mmpose
   ├── mmpose
   ├── docs
   ├── tests
   ├── tools
   `── data
       |

   ```text
   mmpose
   ├── mmpose
   ├── docs
   ├── tests
   ├── tools
   `── data
       |
       |── DeepPoseKit-Data
       |   `── datasets
       |       |── fly
       |       |   |── annotation_data_release.h5
       |       |   |── skeleton.csv
       |       |   |── ...
       |       |
       |       |── locust
       |       |   |── ...
       |       |
       |       `── zebra
       |           |── annotation_data_release.h5
       |           |── skeleton.csv
       |           |── ...
       |
       │── fly
           `-- images
               │-- 0.jpg
               │-- 1.jpg
               │-- ...
   ```

|           |── skeleton.csv
       |           |── ...
       |
       │── fly
           `-- images
               │-- 0.jpg
               │-- 1.jpg
               │-- ...
   ```

   图片也可以在 [vinegar_fly_images](https://download.openmmlab.com/mmpose/datasets/vinegar_fly_images.tar)，[locust_images](https://download.openmmlab.com/mmpose/datasets/locust_images.tar) 和[zebra_images](https://download.openmmlab.com/mmpose/datasets/zebra_images.tar) 下载。

2. 运行脚本：

   ```bash
   python tools/dataset_converters/parse_deepposekit_dataset.py
   ```

   生成的标注文件将保存在 $MMPOSE/data/fly/annotations`，`$MMPOSE/data/locust/annotations`和`$MMPOSE/data/zebra/annotations\` 中。

由于官方数据集中没有提供测试集，我们随机选择了 90% 的图片用于训练，剩下的 10% 用于测试。

## Macaque 数据集

<details>

```bibtex

<details>

```bibtex
}
```

对于 [MacaquePose](http://www2.ehub.kyoto-u.ac.jp/datasets/macaquepose/index.html) 数据集，请从 [这里](http://www2.ehub.kyoto-u.ac.jp/datasets/macaquepose/index.html) 下载数据。

`tools/dataset_converters/parse_macaquepose_dataset.py` 脚本可以将原始标注转换为 MMPose 支持的格式。我们已经转换好的标注文件可以在 [这里](https://download.openmmlab.com/mmpose/datasets/macaque_annotations.tar) 下载。

   ```text
   mmpose
   ├── mmpose
   ├── docs
   ├── tests
   ├── tools
   `── data
       │── macaque
           │-- annotations.csv
           │-- images
           │   │-- 01418849d54b3005.jpg
           │   │-- 0142d1d1a6904a70.jpg
           │   │-- 01ef2c4c260321b7.jpg
           │   │-- 020a1c75c8c85238.jpg
           │   │-- 020b1506eef2557d.jpg
           │   │-- ...
   ```

2. 运行脚本：

   ```bash
   python tools/dataset_converters/parse_macaquepose_dataset.py
   ```

   生成的标注文件将保存在 `$MMPOSE/data/macaque/annotations` 中。

## Human3.6M 数据集

<details>

```bibtex

```bibtex
}
```

对于 [Human3.6M](http://vision.imar.ro/human3.6m/description.php) 数据集，请从官网下载数据，放置到 `$MMPOSE/data/h36m` 下。

然后执行 [预处理脚本](/tools/dataset_converters/preprocess_h36m.py)。

```bash
python tools/dataset_converters/preprocess_h36m.py --metadata {path to metadata.xml} --original data/h36m
```

这将在全帧率（50 FPS）和降频帧率（10 FPS）下提取相机参数和姿势注释。处理后的数据应具有以下结构：

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    ├── h36m
        ├── annotation_body3d
        |   ├── cameras.pkl
        |   ├── fps50

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    ├── h36m
        |   ├── fps50
        |   |   ├── h36m_test.npz
        |   |   ├── h36m_train.npz
        |   |   ├── joint2d_rel_stats.pkl
        |   |   ├── joint2d_stats.pkl
        |   |   ├── joint3d_rel_stats.pkl
        |   |   `── joint3d_stats.pkl
        |   `── fps10
        |       ├── h36m_test.npz
        |       ├── h36m_train.npz
        |       ├── joint2d_rel_stats.pkl
        |       ├── joint2d_stats.pkl
        |       ├── joint3d_rel_stats.pkl
        |       `── joint3d_stats.pkl
        `── images
            ├── S1
            |   ├── S1_Directions_1.54138969
            |   |   ├── S1_Directions_1.54138969_00001.jpg
            |   |   ├── S1_Directions_1.54138969_00002.jpg
            |   |   ├── ...
            |   ├── ...
            ├── S5
            ├── S6
            ├── S7
            ├── S8
            ├── S9

|   |   ├── ...
            |   ├── ...
            ├── S5
            ├── S6
            ├── S7
            ├── S8
            ├── S9
            `── S11
```

然后，标注信息需要转换为 MMPose 支持的 COCO 格式。这可以通过运行以下命令完成：

```bash
python tools/dataset_converters/h36m_to_coco.py
```

## MPII 数据集

<details>

```bibtex
}
```

对于 [MPII](http://human-pose.mpi-inf.mpg.de/) 数据集，请从官网下载数据，放置到 `$MMPOSE/data/mpii` 下。

我们提供了一个脚本来将 `.mat` 格式的标注文件转换为 `.json` 格式。这可以通过运行以下命令完成：

```shell

}
```

```shell
python tools/dataset_converters/mat2json ${PRED_MAT_FILE} ${GT_JSON_FILE} ${OUTPUT_PRED_JSON_FILE}
```

例如：

```shell
```

## Label Studio 数据集

<details>
<summary align="right"><a href="https://github.com/heartexlabs/label-studio/">Label Studio</a></summary>

```bibtex
@misc{Label Studio,
  title={{Label Studio}: Data labeling software},
  url={https://github.com/heartexlabs/label-studio},
  note={Open source software available from https://github.com/heartexlabs/label-studio},
  author={
    Maxim Tkachenko and
    Mikhail Malyuk and
    Andrey Holmanyuk and
    Nikolai Liubimov},
  year={2020-2022},
}
```

author={
}
```

对于 [Label Studio](https://github.com/heartexlabs/label-studio/) 用户，请依照 [Label Studio 转换工具文档](./label_studio.md) 中的方法进行标注，并将结果导出为 Label Studio 标准的 `.json` 文件，将 `Labeling Interface` 中的 `Code` 保存为 `.xml` 文件。

我们提供了一个脚本来将 Label Studio 标准的 `.json` 格式标注文件转换为 COCO 标准的 `.json` 格式。这可以通过运行以下命令完成：

```shell
python tools/dataset_converters/labelstudio2coco.py ${LS_JSON_FILE} ${LS_XML_FILE} ${OUTPUT_COCO_JSON_FILE}
```

例如：

```shell
```

# 类别激活图（CAM）可视化

## 类别激活图可视化工具介绍

MMPretrain 提供 `tools/visualization/vis_cam.py` 工具来可视化类别激活图。请使用 `pip install "grad-cam>=1.3.6"` 安装依赖的 [pytorch-grad-cam](https://github.com/jacobgil/pytorch-grad-cam)。

目前支持的方法有：

|    Method    |                                           What it does                                            |
| :----------: | :-----------------------------------------------------------------------------------------------: |
|   GradCAM    |                                  使用平均梯度对 2D 激活进行加权                                   |
|  GradCAM++   |                                  类似 GradCAM，但使用了二阶梯度                                   |
|   XGradCAM   |                         类似 GradCAM，但通过归一化的激活对梯度进行了加权                          |
|   EigenCAM   |                     使用 2D 激活的第一主成分（无法区分类别，但效果似乎不错）                      |
| EigenGradCAM | 类似 EigenCAM，但支持类别区分，使用了激活 * 梯度的第一主成分，看起来和 GradCAM 差不多，但是更干净 |

|   LayerCAM   |                        使用正梯度对激活进行空间加权，对于浅层有更好的效果                         |

也可以使用新版本 `pytorch-grad-cam` 支持的更多 CAM 方法，但我们尚未验证可用性。

**命令行**：

```bash
    ${IMG} \
```

- `img`：目标图片路径。
- `--target-layers`：所查看的网络层名称，可输入一个或者多个网络层，如果不设置，将使用最后一个`block`中的`norm`层。
- `--preview-model`：是否查看模型所有网络层。

```

- `--method`：类别激活图图可视化的方法，目前支持 `GradCAM`, `GradCAM++`, `XGradCAM`, `EigenCAM`, `EigenGradCAM`, `LayerCAM`，不区分大小写。如果不设置，默认为 `GradCAM`。
- `--target-category`：查看的目标类别，如果不设置，使用模型检测出来的类别做为目标类别。
- `--save-path`：保存的可视化图片的路径，默认不保存。
- `--eigen-smooth`：是否使用主成分降低噪音，默认不开启。
- `--vit-like`: 是否为 `ViT` 类似的 Transformer-based 网络
- `--num-extra-tokens`: `ViT` 类网络的额外的 tokens 通道数，默认使用主干网络的 `num_extra_tokens`。
- `--aug-smooth`：是否使用测试时增强
- `--device`：使用的计算设备，如果不设置，默认为'cpu'。

```{note}
在指定 `--target-layers` 时，如果不知道模型有哪些网络层，可使用命令行添加 `--preview-model` 查看所有网络层名称；
```

## 如何可视化 CNN 网络的类别激活图（如 ResNet-50）

`--target-layers` 在 `Resnet-50` 中的一些示例如下：

- `'backbone.layer4'`，表示第四个 `ResLayer` 层的输出。
- `'backbone.layer4.2'` 表示第四个 `ResLayer` 层中第三个 `BottleNeck` 块的输出。

```

- `'backbone.layer4.2.conv1'` 表示上述 `BottleNeck` 块中 `conv1` 层的输出。

1. 使用不同方法可视化 `ResNet50`，默认 `target-category` 为模型检测的结果，使用默认推导的 `target-layers`。

   ```shell
   ```

2. 同一张图不同类别的激活图效果图，在 `ImageNet` 数据集中，类别 238 为 'Greater Swiss Mountain dog'，类别 281 为 'tabby, tabby cat'。

   ```shell

   ```shell
   ```

3. 使用 `--eigen-smooth` 以及 `--aug-smooth` 获取更好的可视化效果。

   ```shell

demo/dog.jpg  \
   ```

## 如何可视化 Transformer 类型网络的类别激活图

`--target-layers` 在 Transformer-based 网络中的一些示例如下：

- Swin-Transformer 中：`'backbone.norm3'`

- ViT 中：`'backbone.layers.11.ln1'`

对于 Transformer-based 的网络，比如 ViT、T2T-ViT 和 Swin-Transformer，特征是被展平的。为了绘制 CAM 图，我们需要指定 `--vit-like` 选项，从而让被展平的特征恢复方形的特征图。

除了特征被展平之外，一些类 ViT 的网络还会添加额外的 tokens。比如 ViT 和 T2T-ViT 中添加了分类 token，DeiT 中还添加了蒸馏 token。在这些网络中，分类计算在最后一个注意力模块之后就已经完成了，分类得分也只和这些额外的 tokens 有关，与特征图无关，也就是说，分类得分对这些特征图的导数为 0。因此，我们不能使用最后一个注意力模块的输出作为 CAM 绘制的目标层。

另外，为了去除这些额外的 toekns 以获得特征图，我们需要知道这些额外 tokens 的数量。MMPretrain 中几乎所有 Transformer-based 的网络都拥有 `num_extra_tokens` 属性。而如果你希望将此工具应用于新的，或者第三方的网络，而且该网络没有指定 `num_extra_tokens` 属性，那么可以使用 `--num-extra-tokens` 参数手动指定其数量。

1. 对 `Swin Transformer` 使用默认 `target-layers` 进行 CAM 可视化：

   ```shell

   ```shell
   ```

2. 对 `Vision Transformer(ViT)` 进行 CAM 可视化：

   ```shell
   ```

3. 对 `T2T-ViT` 进行 CAM 可视化：

   ```shell

```

   ```shell
   ```

# 如何在自定义数据集上进行模型预训练

在本教程中，我们提供了一个实践示例和一些有关如何在您自己的数据集上进行训练的技巧。

在 MMPretrain 中，我们支持用户直接调用 MMPretrain 的 `CustomDataset` （类似于 `torchvision` 的 `ImageFolder`）, 该数据集能自动的读取给的路径下的图片。你只需要准备你的数据集路径，并修改配置文件，即可轻松使用 MMPretrain 进行预训练。

假设我们想使用 MAE 算法进行图像自监督训练，并使用子文件夹格式的 `CustomDataset` 来组织数据集：

```text
├── sample1.png
├── sample2.png
├── sample3.png
├── sample4.png
└── ...
```

在本教程中，我们使用 `configs/mae/mae_vit-base-p16_8xb512-amp-coslr-300e_in1k.py` 作为一个示例进行介绍。
首先在同一文件夹下复制一份配置文件，并将其重命名为 `mae_vit-base-p16_8xb512-amp-coslr-300e_custom.py`。

```{tip}
```

这个配置文件的内容如下：

```python
_base_ = [
    '../_base_/models/mae_vit-base-p16.py',
    '../_base_/datasets/imagenet_bs512_mae.py',
]

# optimizer wrapper

'../_base_/models/mae_vit-base-p16.py',
]

    loss_scale='dynamic',
        lr=1.5e-4 * 4096 / 256,
        betas=(0.9, 0.95),
        weight_decay=0.05),
            'ln': dict(decay_mult=0.0),
            'bias': dict(decay_mult=0.0),
            'mask_token': dict(decay_mult=0.),
            'cls_token': dict(decay_mult=0.)
        }))

# learning rate scheduler
    dict(
        type='LinearLR',
        start_factor=0.0001,
        begin=0,
        end=40,
    dict(
        T_max=260,
        begin=40,
        end=300,
        convert_to_iter_based=True)
]

    dict(
        begin=40,
        end=300,
]

# runtime settings
train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=300)
    # only keeps the latest 3 checkpoints
    checkpoint=dict(type='CheckpointHook', interval=1, max_keep_ckpts=3))

randomness = dict(seed=0, diff_rank_seed=True)

# auto resume

# NOTE: `auto_scale_lr` is for automatically scaling LR
# based on the actual training batch size.
auto_scale_lr = dict(base_batch_size=4096)
```

## 第三步：修改数据集设置

- 重载数据集设置中的 `type` 为 `'CustomDataset'`
- 重载数据集设置中的 `data_root` 为 `data/custom_dataset`
- 重载数据集设置中的 `ann_file` 为空字符串，这是因为我们使用子文件格式的 `CustomDataset`，需要将配置文件置空
- 重载数据集设置中的 `data_prefix` 为空字符串，这是因为我们希望使用数据集根目录下的所有数据进行训练，并不需要将其拆分为不同子集。

修改后的文件应如下：

```python
_base_ = [

```python
_base_ = [
]

# >>>>>>>>>>>>>>> 在此重载数据设置 >>>>>>>>>>>>>>>>>>>
        data_root='data/custom_dataset/',
        data_prefix='',    # 使用 `data_root` 路径下所有数据
        with_label=False,
    )
)

betas=(0.9, 0.95),
        }))

    dict(
        begin=0,
        end=40,
    dict(
        begin=40,
        end=300,
]

# only keeps the latest 3 checkpoints

```

使用上述配置文件，你就能够轻松的在自定义数据集上使用 `MAE` 算法来进行预训练了。

## 另一个例子：在 COCO 数据集上训练 MAE

```{note}
你可能需要参考[文档](https://github.com/open-mmlab/mmdetection/blob/3.x/docs/en/get_started.md)安装 MMDetection 来使用 `mmdet.CocoDataset`。
```

与在自定义数据集上进行预训练类似，我们在本教程中也提供了一个使用 COCO 数据集进行预训练的示例。修改后的文件如下：

```python
# >>>>>>>>>>>>>>>>>>>>> Start of Changed >>>>>>>>>>>>>>>>>>>>>>>>>
_base_ = [
    '../_base_/datasets/imagenet_mae.py',
]

        type='mmdet.CocoDataset',
        data_root='data/coco/',

'../_base_/default_runtime.py',
]

        ann_file='annotations/instances_train2017.json',  # 仅用于加载图片，不会使用标签
    )
)

        }))
    dict(

'cls_token': dict(decay_mult=0.)
        }))
    dict(
        begin=0,
        end=40,
    dict(
        begin=40,
        end=300,
]
```

# Preparing Unpaired Dataset for CycleGAN

```bibtex
}
```

You can download unpaired datasets from [here](https://people.eecs.berkeley.edu/~taesung_park/CycleGAN/datasets/).

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── unpaired
|   |   ├── horse2zebra
|   |   ├── summer2winter_yosemite
|   |   |    ├── trainA
|   |   |    ├── trainB
|   |   |    ├── testA
|   |   |    ├── testB
```

# 实用的钩子

MMDetection 和 MMEngine 为用户提供了多种多样实用的钩子（Hook），包括 `MemoryProfilerHook`、`NumClassCheckHook` 等等。
这篇教程介绍了 MMDetection 中实现的钩子功能及使用方式。若使用 MMEngine 定义的钩子请参考 [MMEngine 的钩子API文档](https://github.com/open-mmlab/mmengine/tree/main/docs/en/tutorials/hook.md).

[内存分析钩子](https://github.com/open-mmlab/mmdetection/blob/main/mmdet/engine/hooks/memory_profiler_hook.py)
记录了包括虚拟内存、交换内存、当前进程在内的所有内存信息，它能够帮助捕捉系统的使用状况与发现隐藏的内存泄露问题。为了使用这个钩子，你需要先通过 `pip install memory_profiler psutil` 命令安装 `memory_profiler` 和 `psutil`。

### 使用

为了使用这个钩子，使用者需要添加如下代码至 config 文件

```python
]
```

### 结果

在训练中，你会看到 `MemoryProfilerHook` 记录的如下信息：

```text

### 使用

```python
]
```

### 结果

```text
```

```text
```

## 如何实现自定义钩子

通常，从模型训练的开始到结束，共有20个点位可以执行钩子。我们可以实现自定义钩子在不同点位执行，以便在训练中实现自定义操作。

比如，我们要实现一个检查 loss 的钩子，当损失为 NaN 时自动结束训练。我们可以把这个过程分为三步：

1. 在 MMEngine 实现一个继承于 `Hook` 类的新钩子，并实现 `after_train_iter` 方法用于检查每 `n` 次训练迭代后损失是否变为 NaN 。
2. 使用 `@HOOKS.register_module()` 注册实现好了的自定义钩子，如下列代码所示。
3. 在配置文件中添加 `custom_hooks = [dict(type='MemoryProfilerHook', interval=50)]`

```python

```python

    Args:
    """

        Args:

outputs: Optional[dict] = None) -> None:

        Args:
        """
```

请参考 [自定义训练配置](../advanced_guides/customize_runtime.md) 了解更多与自定义钩子相关的内容。

> [Image Super-Resolution Using Deep Convolutional Networks](https://arxiv.org/abs/1501.00092)

We propose a deep learning method for single image super-resolution (SR). Our method directly learns an end-to-end mapping between the low/high-resolution images. The mapping is represented as a deep convolutional neural network (CNN) that takes the low-resolution image as the input and outputs the high-resolution one. We further show that traditional sparse-coding-based SR methods can also be viewed as a deep convolutional network. But unlike traditional methods that handle each component separately, our method jointly optimizes all layers. Our deep CNN has a lightweight structure, yet demonstrates state-of-the-art restoration quality, and achieves fast speed for practical on-line usage. We explore different network structures and parameter settings to achieve trade-offs between performance and speed. Moreover, we extend our network to cope with three color channels simultaneously, and show better overall reconstruction quality.

 <img src="https://user-images.githubusercontent.com/7676947/144034831-79f48aae-196e-42e7-92b9-069149733e3e.png" width="400"/>
</div >

| [srcnn_x4k915_1x16_1000k_div2k](./srcnn_x4k915_1xb16-1000k_div2k.py) |  Set5   | 28.4316 | 0.8099 |         1          | [model](https://download.openmmlab.com/mmediting/restorers/srcnn/srcnn_x4k915_1x16_1000k_div2k_20200608-4186f232.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/srcnn/srcnn_x4k915_1x16_1000k_div2k_20200608_120159.log.json) |
| [srcnn_x4k915_1x16_1000k_div2k](./srcnn_x4k915_1xb16-1000k_div2k.py) |  Set14  | 25.6486 | 0.7014 |         1          | [model](https://download.openmmlab.com/mmediting/restorers/srcnn/srcnn_x4k915_1x16_1000k_div2k_20200608-4186f232.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/srcnn/srcnn_x4k915_1x16_1000k_div2k_20200608_120159.log.json) |

| [srcnn_x4k915_1x16_1000k_div2k](./srcnn_x4k915_1xb16-1000k_div2k.py) |  DIV2K  | 27.7460 | 0.7854 |         1          | [model](https://download.openmmlab.com/mmediting/restorers/srcnn/srcnn_x4k915_1x16_1000k_div2k_20200608-4186f232.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/srcnn/srcnn_x4k915_1x16_1000k_div2k_20200608_120159.log.json) |

**Train**

<details>

```shell
# cpu train

```

**Test**

<details>

```

**Test**

<details>

```shell
# cpu test

```

```

```bibtex
}
```

</div>

### AVA2.2

Currently, we use the training set of AVA-Kinetics and evaluate on the AVA2.2 validation dataset.

| frame sampling strategy | resolution | gpus | backbone  |   pretrain   | mAP  |                     config                      |                      ckpt                      |                      log                      |
| :---------------------: | :--------: | :--: | :-------: | :----------: | :--: | :---------------------------------------------: | :--------------------------------------------: | :-------------------------------------------: |

|         16x4x1          |    raw     |  8   | ViT Base  | Kinetics-400 | 33.6 | [config](/configs/detection/videomae/vit-base-p16_videomae-k400-pre_8xb8-16x4x1-20e-adamw_ava-kinetics-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/videomae/vit-base-p16_videomae-k400-pre_8xb8-16x4x1-20e-adamw_ava-kinetics-rgb/vit-base-p16_videomae-k400-pre_8xb8-16x4x1-20e-adamw_ava-kinetics-rgb_20230314-3dafab75.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/videomae/vit-base-p16_videomae-k400-pre_8xb8-16x4x1-20e-adamw_ava-kinetics-rgb/vit-base-p16_videomae-k400-pre_8xb8-16x4x1-20e-adamw_ava-kinetics-rgb.log) |

|         16x4x1          |    raw     |  8   | ViT Large | Kinetics-400 | 38.7 | [config](/configs/detection/videomae/vit-large-p16_videomae-k400-pre_8xb8-16x4x1-20e-adamw_ava-kinetics-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/videomae/vit-large-p16_videomae-k400-pre_8xb8-16x4x1-20e-adamw_ava-kinetics-rgb/vit-large-p16_videomae-k400-pre_8xb8-16x4x1-20e-adamw_ava-kinetics-rgb_20230314-bf93c9ea.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/videomae/vit-large-p16_videomae-k400-pre_8xb8-16x4x1-20e-adamw_ava-kinetics-rgb/vit-large-p16_videomae-k400-pre_8xb8-16x4x1-20e-adamw_ava-kinetics-rgb.log) |

## Train

```shell
```

Example: train the ViT base model on AVA-Kinetics in a deterministic option.

```shell

```shell
```

```shell
python tools/train.py configs/detection/ava_kinetics/vit-base-p16_videomae-k400-pre_8xb8-16x4x1-20e-adamw_ava-kinetics-rgb.py \
    --cfg-options randomness.seed=0 randomness.deterministic=True
```

## Test

```shell
```

Example: test the ViT base model on AVA-Kinetics and dump the result to a pkl file.

```shell
python tools/test.py configs/detection/ava_kinetics/vit-base-p16_videomae-k400-pre_8xb8-16x4x1-20e-adamw_ava-kinetics-rgb.py \
```

```shell
```

```BibTeX
}
```

the typical usage
```
ncnnoptimize mobilenet.param mobilenet.bin mobilenet-opt.param mobilenet-opt.bin 65536 
```

operator fusion
* batchnorm - scale
* convolution - batchnorm
* convolutiondepthwise - batchnorm
* deconvolution - batchnorm
* deconvolutiondepthwise - batchnorm
* innerproduct - batchnorm
* convolution - relu
* convolutiondepthwise - relu
* deconvolution - relu
* deconvolutiondepthwise - relu
* innerproduct - relu

eliminate noop operator
* innerproduct - dropout
* flatten after global pooling

prefer better operator
* replace convolution with innerproduct after global pooling

# GRoIE

> [A novel Region of Interest Extraction Layer for Instance Segmentation](https://arxiv.org/abs/2004.13665)

Given the wide diffusion of deep neural network architectures for computer vision tasks, several new applications are nowadays more and more feasible. Among them, a particular attention has been recently given to instance segmentation, by exploiting the results achievable by two-stage networks (such as Mask R-CNN or Faster R-CNN), derived from R-CNN. In these complex architectures, a crucial role is played by the Region of Interest (RoI) extraction layer, devoted to extracting a coherent subset of features from a single Feature Pyramid Network (FPN) layer attached on top of a backbone.

This paper is motivated by the need to overcome the limitations of existing RoI extractors which select only one (the best) layer from FPN. Our intuition is that all the layers of FPN retain useful information. Therefore, the proposed layer (called Generic RoI Extractor - GRoIE) introduces non-local building blocks and attention mechanisms to boost the performance.
A comprehensive ablation study at component level is conducted to find the best set of algorithms and parameters for the GRoIE layer. Moreover, GRoIE can be integrated seamlessly with every two-stage architecture for both object detection and instance segmentation tasks. Therefore, the improvements brought about by the use of GRoIE in different state-of-the-art architectures are also evaluated. The proposed layer leads up to gain a 1.1% AP improvement on bounding box detection and 1.7% AP improvement on instance segmentation.

<img src="https://user-images.githubusercontent.com/40661020/143891453-afdcdaf4-a868-4a28-ad20-dc710a517a76.png"/>
</div>

By Leonardo Rossi, Akbar Karimi and Andrea Prati from
[IMPLab](http://implab.ce.unipr.it/).

We provide configs to reproduce the results in the paper for
"*A novel Region of Interest Extraction Layer for Instance Segmentation*"
on COCO object detection.

This paper is motivated by the need to overcome to the limitations of existing
RoI extractors which select only one (the best) layer from FPN.

Our intuition is that all the layers of FPN retain useful information.

Therefore, the proposed layer (called Generic RoI Extractor - **GRoIE**)
introduces non-local building blocks and attention mechanisms to boost the
performance.

The results on COCO 2017 minival (5k images) are shown in the below table.

### Application of GRoIE to different architectures

| Backbone  |     Method      | Lr schd | box AP | mask AP |                                Config                                |                                                                                                                                                                                                                   Download                                                                                                                                                                                                                   |

| R-50-FPN  | Faster Original |   1x    |  37.4  |         |       [config](../faster_rcnn/faster-rcnn_r50_fpn_1x_coco.py)        |                                                                  [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130_204655.log.json)                                                                   |

| R-50-FPN  |     + GRoIE     |   1x    |  38.3  |         |           [config](./faste-rcnn_r50_fpn_groie_1x_coco.py)            |                                                         [model](https://download.openmmlab.com/mmdetection/v2.0/groie/faster_rcnn_r50_fpn_groie_1x_coco/faster_rcnn_r50_fpn_groie_1x_coco_20200604_211715-66ee9516.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/groie/faster_rcnn_r50_fpn_groie_1x_coco/faster_rcnn_r50_fpn_groie_1x_coco_20200604_211715.log.json)                                                         |

| R-50-FPN  |   Grid R-CNN    |   1x    |  39.1  |         |        [config](./grid-rcnn_r50_fpn_gn-head-groie_1x_coco.py)        |                                             [model](https://download.openmmlab.com/mmdetection/v2.0/groie/grid_rcnn_r50_fpn_gn-head_groie_1x_coco/grid_rcnn_r50_fpn_gn-head_groie_1x_coco_20200605_202059-4b75d86f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/groie/grid_rcnn_r50_fpn_gn-head_groie_1x_coco/grid_rcnn_r50_fpn_gn-head_groie_1x_coco_20200605_202059.log.json)                                             |

| R-50-FPN  |     + GRoIE     |   1x    |        |         |        [config](./grid-rcnn_r50_fpn_gn-head-groie_1x_coco.py)        |                                                                                                                                                                                                                                                                                                                                                                                                                                              |

| R-50-FPN  |   Mask R-CNN    |   1x    |  38.2  |  34.7   |         [config](../mask_rcnn/mask-rcnn_r50_fpn_1x_coco.py)          |                                                                        [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_fpn_1x_coco/mask_rcnn_r50_fpn_1x_coco_20200205-d4b0c5d6.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_fpn_1x_coco/mask_rcnn_r50_fpn_1x_coco_20200205_050542.log.json)                                                                         |

| R-50-FPN  |     + GRoIE     |   1x    |  39.0  |  36.0   |            [config](./mask-rcnn_r50_fpn_groie_1x_coco.py)            |                                                             [model](https://download.openmmlab.com/mmdetection/v2.0/groie/mask_rcnn_r50_fpn_groie_1x_coco/mask_rcnn_r50_fpn_groie_1x_coco_20200604_211715-50d90c74.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/groie/mask_rcnn_r50_fpn_groie_1x_coco/mask_rcnn_r50_fpn_groie_1x_coco_20200604_211715.log.json)                                                             |

| R-50-FPN  |     GC-Net      |   1x    |  40.7  |  36.5   | [config](../gcnet/mask-rcnn_r50-syncbn-gcb-r4-c3-c5_fpn_1x_coco.py)  |                  [model](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r50_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco/mask_rcnn_r50_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco_20200202-50b90e5c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r50_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco/mask_rcnn_r50_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco_20200202_085547.log.json)                   |

| R-50-FPN  |     + GRoIE     |   1x    |  41.0  |  37.8   |  [config](./mask-rcnn_r50_fpn_syncbn-r4-gcb-c3-c5-groie_1x_coco.py)  | [model](https://download.openmmlab.com/mmdetection/v2.0/groie/mask_rcnn_r50_fpn_syncbn-backbone_r4_gcb_c3-c5_groie_1x_coco/mask_rcnn_r50_fpn_syncbn-backbone_r4_gcb_c3-c5_groie_1x_coco_20200604_211715-42eb79e1.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/groie/mask_rcnn_r50_fpn_syncbn-backbone_r4_gcb_c3-c5_groie_1x_coco/mask_rcnn_r50_fpn_syncbn-backbone_r4_gcb_c3-c5_groie_1x_coco_20200604_211715-42eb79e1.pth) |

| R-101-FPN |     GC-Net      |   1x    |  42.2  |  37.8   | [config](../gcnet/mask-rcnn_r101-syncbn-gcb-r4-c3-c5_fpn_1x_coco.py) |                [model](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r101_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco/mask_rcnn_r101_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco_20200206-8407a3f0.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r101_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco/mask_rcnn_r101_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco_20200206_142508.log.json)                 |

| R-101-FPN |     + GRoIE     |   1x    |  42.6  |  38.7   | [config](./mask-rcnn_r101_fpn_syncbn-r4-gcb_c3-c5-groie_1x_coco.py)  | [model](https://download.openmmlab.com/mmdetection/v2.0/groie/mask_rcnn_r101_fpn_syncbn-backbone_r4_gcb_c3-c5_groie_1x_coco/mask_rcnn_r101_fpn_syncbn-backbone_r4_gcb_c3-c5_groie_1x_coco_20200607_224507-8daae01c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/groie/mask_rcnn_r101_fpn_syncbn-backbone_r4_gcb_c3-c5_groie_1x_coco/mask_rcnn_r101_fpn_syncbn-backbone_r4_gcb_c3-c5_groie_1x_coco_20200607_224507.log.json) |

If you use this work or benchmark in your research, please cite this project.

```latex
@inproceedings{rossi2021novel,
  title={A novel region of interest extraction layer for instance segmentation},
  author={Rossi, Leonardo and Karimi, Akbar and Prati, Andrea},
  booktitle={2020 25th International Conference on Pattern Recognition (ICPR)},
  pages={2203--2209},
}
```

## Contact

booktitle={2020 25th International Conference on Pattern Recognition (ICPR)},
}
```

## Contact

The implementation of GRoIE is currently maintained by
[Leonardo Rossi](https://github.com/hachreak/).

# VGG

> [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://arxiv.org/abs/1409.1556)

In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.

<img src="https://user-images.githubusercontent.com/26739999/142578905-9be586ec-f6fd-4bfb-bbba-432f599d3b9b.png" width="60%"/>
</div>

```python

predict = inference_model('vgg11_8xb32_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('vgg11_8xb32_in1k', pretrained=True)
```

Train:

```shell
python tools/train.py configs/vgg/vgg11_8xb32_in1k.py
```

Test:

```shell

Train:

```shell
```

Test:

```shell
python tools/test.py configs/vgg/vgg11_8xb32_in1k.py https://download.openmmlab.com/mmclassification/v0/vgg/vgg11_batch256_imagenet_20210208-4271cd6c.pth
```

| Model                |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |             Config              |                                               Download                                               |
| :------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :-----------------------------: | :--------------------------------------------------------------------------------------------------: |

| `vgg11_8xb32_in1k`   | From scratch |   132.86   |   7.63    |   68.75   |   88.87   |  [config](vgg11_8xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/vgg/vgg11_batch256_imagenet_20210208-4271cd6c.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/vgg/vgg11_batch256_imagenet_20210208-4271cd6c.json) |
| `vgg13_8xb32_in1k`   | From scratch |   133.05   |   11.34   |   70.02   |   89.46   |  [config](vgg13_8xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/vgg/vgg13_batch256_imagenet_20210208-4d1d6080.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/vgg/vgg13_batch256_imagenet_20210208-4d1d6080.json) |

| `vgg16_8xb32_in1k`   | From scratch |   138.36   |   15.50   |   71.62   |   90.49   |  [config](vgg16_8xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/vgg/vgg16_batch256_imagenet_20210208-db26f1a5.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/vgg/vgg16_batch256_imagenet_20210208-db26f1a5.json) |
| `vgg19_8xb32_in1k`   | From scratch |   143.67   |   19.67   |   72.41   |   90.80   |  [config](vgg19_8xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/vgg/vgg19_batch256_imagenet_20210208-e6920e4a.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/vgg/vgg19_batch256_imagenet_20210208-e6920e4a.json) |

| `vgg11bn_8xb32_in1k` | From scratch |   132.87   |   7.64    |   70.67   |   90.16   | [config](vgg11bn_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/vgg/vgg11_bn_batch256_imagenet_20210207-f244902c.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/vgg/vgg11_bn_batch256_imagenet_20210207-f244902c.json) |
| `vgg13bn_8xb32_in1k` | From scratch |   133.05   |   11.36   |   72.12   |   90.66   | [config](vgg13bn_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/vgg/vgg13_bn_batch256_imagenet_20210207-1a8b7864.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/vgg/vgg13_bn_batch256_imagenet_20210207-1a8b7864.json) |

| `vgg16bn_8xb32_in1k` | From scratch |   138.37   |   15.53   |   73.74   |   91.66   | [config](vgg16bn_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/vgg/vgg16_bn_batch256_imagenet_20210208-7e55cd29.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/vgg/vgg16_bn_batch256_imagenet_20210208-7e55cd29.json) |
| `vgg19bn_8xb32_in1k` | From scratch |   143.68   |   19.70   |   74.68   |   92.27   | [config](vgg19bn_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/vgg/vgg19_bn_batch256_imagenet_20210208-da620c4f.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/vgg/vgg19_bn_batch256_imagenet_20210208-da620c4f.json) |

```bibtex
@article{simonyan2014very,
  title={Very deep convolutional networks for large-scale image recognition},
  author={Simonyan, Karen and Zisserman, Andrew},
  journal={arXiv preprint arXiv:1409.1556},
}
```

## Usage

Please refer to the [building documentation of MMCV](https://mmcv.readthedocs.io/en/latest/get_started/build.html#build-mmcv-full-on-ascend-npu-machine) to install MMCV on NPU devices

Here we use 4 NPUs on your computer to train the model with the following command:

```shell
```

```shell
```

- If not specially marked, the results on NPU with amp are the basically same as those on the GPU with FP32.

# EVA-02

> [EVA-02: A Visual Representation for Neon Genesis](https://arxiv.org/abs/2303.11331)

We launch EVA-02, a next-generation Transformer-based visual representation pre-trained to reconstruct strong and robust language-aligned vision features via masked image modeling. With an updated plain Transformer architecture as well as extensive pre-training from an open & accessible giant CLIP vision encoder, EVA-02 demonstrates superior performance compared to prior state-of-the-art approaches across various representative vision tasks, while utilizing significantly fewer parameters and compute budgets. Notably, using exclusively publicly accessible training data, EVA-02 with only 304M parameters achieves a phenomenal 90.0 fine-tuning top-1 accuracy on ImageNet-1K val set.  Additionally, our EVA-02-CLIP can reach up to 80.4 zero-shot top-1 on ImageNet-1K, outperforming the previous largest & best open-sourced CLIP with only ~1/6 parameters and ~1/6 image-text training data. We offer four EVA-02 variants in various model sizes, ranging from 6M to 304M parameters, all with impressive performance. To facilitate open accessand open research, we release the complete suite of EVA-02 to the community.

<img src="https://user-images.githubusercontent.com/40905160/229037980-b83dceb5-41d6-406c-a20b-63b83c80136d.png" width="70%" alt="TrV builds upon the original plain ViT architecture and includes several enhancements: SwinGLU FFN, sub-LN, 2D RoPE, and JAX weight initialization. To keep the parameter & FLOPs consistent with the baseline, the FFN hidden dim of SwiGLU is 2/3× of the typical MLP counterpart."/>
</div>

```python

predict = inference_model('vit-tiny-p14_eva02-in21k-pre_3rdparty_in1k-336px', 'demo/bird.JPEG')
```

```python

model = get_model('vit-tiny-p14_eva02-in21k-pre_3rdparty_in1k-336px', pretrained=True)
inputs = torch.rand(1, 3, 336, 336)

```

Train:

```shell
python tools/train.py configs/eva02/eva02-tiny-p14_in1k.py
```

Test:

```shell
python tools/test.py configs/eva02/eva02-tiny-p14_in1k.py /path/to/eva02-tiny-p14_in1k.pth
```

| Model                             | Params (M) | Flops (G) |                Config                 |                                                   Download                                                    |

| :-------------------------------- | :--------: | :-------: | :-----------------------------------: | :-----------------------------------------------------------------------------------------------------------: |
| `vit-tiny-p14_eva02-pre_in21k`\*  |    5.50    |   1.70    | [config](eva02-tiny-p14_headless.py)  | [model](https://download.openmmlab.com/mmpretrain/v1.0/eva02/eva02-tiny-p14_pre_in21k_20230505-d703e7b1.pth)  |
| `vit-small-p14_eva02-pre_in21k`\* |   21.62    |   6.14    | [config](eva02-small-p14_headless.py) | [model](https://download.openmmlab.com/mmpretrain/v1.0/eva02/eva02-small-p14_pre_in21k_20230505-3175f463.pth) |
| `vit-base-p14_eva02-pre_in21k`\*  |   85.77    |   23.22   | [config](eva02-base-p14_headless.py)  | [model](https://download.openmmlab.com/mmpretrain/v1.0/eva02/eva02-base-p14_pre_in21k_20230505-2f2d4d3c.pth)  |

| `vit-large-p14_eva02-pre_in21k`\* |   303.29   |   81.15   | [config](eva02-large-p14_headless.py) | [model](https://download.openmmlab.com/mmpretrain/v1.0/eva02/eva02-large-p14_pre_in21k_20230505-9072de5d.pth) |
| `vit-large-p14_eva02-pre_m38m`\*  |   303.29   |   81.15   | [config](eva02-large-p14_headless.py) | [model](https://download.openmmlab.com/mmpretrain/v1.0/eva02/eva02-large-p14_pre_m38m_20230505-b8a1a261.pth)  |

- The input size / patch size of MIM pre-trained EVA-02 is `224x224` / `14x14`.

*Models with * are converted from the [official repo](https://github.com/baaivision/EVA).*

#### (*w/o* IN-21K intermediate fine-tuning)

| Model                                                 |      Pretrain      | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |               Config                |                         Download                          |

| :---------------------------------------------------- | :----------------: | :--------: | :-------: | :-------: | :-------: | :---------------------------------: | :-------------------------------------------------------: |
| `vit-tiny-p14_eva02-in21k-pre_3rdparty_in1k-336px`\*  | EVA02 ImageNet-21k |    5.76    |   4.68    |   80.69   |   95.54   | [config](./eva02-tiny-p14_in1k.py)  | [model](https://download.openmmlab.com/mmpretrain/v1.0/eva02/eva02-tiny-p14_in21k-pre_3rdparty_in1k-336px_20230505-a4e8708a.pth) |
| `vit-small-p14_eva02-in21k-pre_3rdparty_in1k-336px`\* | EVA02 ImageNet-21k |   22.13    |   15.48   |   85.78   |   97.60   | [config](./eva02-small-p14_in1k.py) | [model](https://download.openmmlab.com/mmpretrain/v1.0/eva02/eva02-small-p14_in21k-pre_3rdparty_in1k-336px_20230505-9c5b0e85.pth) |

| `vit-base-p14_eva02-in21k-pre_3rdparty_in1k-448px`\*  | EVA02 ImageNet-21k |   87.13    |  107.11   |   88.29   |   98.53   | [config](./eva02-base-p14_in1k.py)  | [model](https://download.openmmlab.com/mmpretrain/v1.0/eva02/eva02-base-p14_in21k-pre_3rdparty_in1k-448px_20230505-8ad211c5.pth) |

*Models with * are converted from the  [official repo](https://github.com/baaivision/EVA/tree/master/EVA-02). The config files of these models are only for inference. We haven't reproduce the training results.*

#### (*w* IN-21K intermediate fine-tuning)

| `vit-base-p14_eva02-in21k-pre_in21k-medft_3rdparty_in1k-448px`\* | EVA02 ImageNet-21k |   87.13    |  107.11   |   88.47   |   98.62   | [config](./eva02-base-p14_in1k.py)  | [model](https://download.openmmlab.com/mmpretrain/v1.0/eva02/eva02-base-p14_in21k-pre_in21k-medft_3rdparty_in1k-448px_20230505-5cd4d87f.pth) |
| `vit-large-p14_eva02-in21k-pre_in21k-medft_3rdparty_in1k-448px`\* | EVA02 ImageNet-21k |   305.08   |  362.33   |   89.65   |   98.95   | [config](./eva02-large-p14_in1k.py) | [model](https://download.openmmlab.com/mmpretrain/v1.0/eva02/eva02-large-p14_in21k-pre_in21k-medft_3rdparty_in1k-448px_20230505-926d1599.pth) |

| `vit-large-p14_eva02_m38m-pre_in21k-medft_3rdparty_in1k-448px`\* |  EVA02 Merged-38M  |   305.10   |  362.33   |   89.83   |   99.00   | [config](./eva02-large-p14_in1k.py) | [model](https://download.openmmlab.com/mmpretrain/v1.0/eva02/eva02-large-p14_m38m-pre_in21k-medft_3rdparty_in1k-448px_20230505-150dc5ed.pth) |

```bibtex
@article{EVA-02,
  title={EVA-02: A Visual Representation for Neon Genesis},
  author={Yuxin Fang and Quan Sun and Xinggang Wang and Tiejun Huang and Xinlong Wang and Yue Cao},
  journal={arXiv preprint arXiv:2303.11331},
}
```

# Benchmark and Model Zoo (To be updated)

- [Rotated RetinaNet-OBB/HBB](https://github.com/open-mmlab/mmrotate/tree/main/configs/rotated_retinanet/README.md) (ICCV'2017)
- [Rotated FasterRCNN-OBB](https://github.com/open-mmlab/mmrotate/tree/main/configs/rotated_faster_rcnn/README.md) (TPAMI'2017)
- [Rotated RepPoints-OBB](https://github.com/open-mmlab/mmrotate/tree/main/configs/rotated_reppoints/README.md) (ICCV'2019)
- [Rotated FCOS](https://github.com/open-mmlab/mmrotate/tree/main/configs/rotated_fcos/README.md) (ICCV'2019)
- [RoI Transformer](https://github.com/open-mmlab/mmrotate/tree/main/configs/roi_trans/README.md) (CVPR'2019)
- [Gliding Vertex](https://github.com/open-mmlab/mmrotate/tree/main/configs/gliding_vertex/README.md) (TPAMI'2020)
- [Rotated ATSS-OBB](https://github.com/open-mmlab/mmrotate/tree/main/configs/rotated_atss/README.md) (CVPR'2020)
- [CSL](https://github.com/open-mmlab/mmrotate/tree/main/configs/csl/README.md) (ECCV'2020)

- [R<sup>3</sup>Det](https://github.com/open-mmlab/mmrotate/tree/main/configs/r3det/README.md) (AAAI'2021)
- [S<sup>2</sup>A-Net](https://github.com/open-mmlab/mmrotate/tree/main/configs/s2anet/README.md) (TGRS'2021)
- [ReDet](https://github.com/open-mmlab/mmrotate/tree/main/configs/redet/README.md) (CVPR'2021)
- [Beyond Bounding-Box](https://github.com/open-mmlab/mmrotate/tree/main/configs/cfa/README.md) (CVPR'2021)
- [Oriented R-CNN](https://github.com/open-mmlab/mmrotate/tree/main/configs/oriented_rcnn/README.md) (ICCV'2021)
- [GWD](https://github.com/open-mmlab/mmrotate/tree/main/configs/gwd/README.md) (ICML'2021)
- [KLD](https://github.com/open-mmlab/mmrotate/tree/main/configs/kld/README.md) (NeurIPS'2021)
- [SASM](configs/sasm_reppoints/README.md) (AAAI'2022)
- [Oriented RepPoints](https://github.com/open-mmlab/mmrotate/tree/main/configs/oriented_reppoints/README.md) (CVPR'2022)

- [KFIoU](https://github.com/open-mmlab/mmrotate/tree/main/configs/kfiou/README.md) (arXiv)
- [H2RBox](https://github.com/open-mmlab/mmrotate/tree/main/configs/h2rbox/README.md) (arXiv)

## Results on DOTA v1.0

|          Backbone          |  mAP  | Angle | lr schd | Mem (GB) | Inf Time (fps) |  Aug  | Batch Size |                                                                      Configs                                                                      |                                                                                                                                                                                                      Download                                                                                                                                                                                                      |

|  ResNet50 (1024,1024,200)  | 59.44 |  oc   |   1x    |   3.45   |      15.6      |   -   |     2      |                [rotated-reppoints-qbox_r50_fpn_1x_dota](../../configs/rotated_reppoints/rotated-reppoints-qbox_r50_fpn_1x_dota.py)                |                                   [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_reppoints/rotated_reppoints_r50_fpn_1x_dota_oc/rotated_reppoints_r50_fpn_1x_dota_oc-d38ce217.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_reppoints/rotated_reppoints_r50_fpn_1x_dota_oc/rotated_reppoints_r50_fpn_1x_dota_oc_20220205_145010.log.json)                                   |

|  ResNet50 (1024,1024,200)  | 64.55 |  oc   |   1x    |   3.38   |      15.7      |   -   |     2      |             [rotated-retinanet-hbox-oc_r50_fpn_1x_dota](../../configs/rotated_retinanet/rotated-retinanet-hbox-oc_r50_fpn_1x_dota.py)             |                           [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_hbb_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_r50_fpn_1x_dota_oc-e8a7c7df.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_hbb_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_r50_fpn_1x_dota_oc_20220121_095315.log.json)                           |

|  ResNet50 (1024,1024,200)  | 65.59 |  oc   |   1x    |   3.12   |      18.5      |   -   |     2      |                     [rotated_atss_hbb_r50_fpn_1x_dota_oc](../../configs/rotated_atss/rotated_atss_hbb_r50_fpn_1x_dota_oc.py)                      |                                          [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_atss/rotated_atss_hbb_r50_fpn_1x_dota_oc/rotated_atss_hbb_r50_fpn_1x_dota_oc-eaa94033.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_atss/rotated_atss_hbb_r50_fpn_1x_dota_oc/rotated_atss_hbb_r50_fpn_1x_dota_oc_20220402_002230.log.json)                                          |

|  ResNet50 (1024,1024,200)  | 66.45 |  oc   |   1x    |   3.53   |      15.3      |   -   |     2      |                         [sasm-reppoints-qbox_r50_fpn_1x_dota](../../configs/sasm/sasm-reppoints-qbox_r50_fpn_1x_dota.py)                          |                                                      [model](https://download.openmmlab.com/mmrotate/v0.1.0/sasm/sasm_reppoints_r50_fpn_1x_dota_oc/sasm_reppoints_r50_fpn_1x_dota_oc-6d9edded.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/sasm/sasm_reppoints_r50_fpn_1x_dota_oc/sasm_reppoints_r50_fpn_1x_dota_oc_20220205_144938.log.json)                                                      |

|  ResNet50 (1024,1024,200)  | 68.42 | le90  |   1x    |   3.38   |      16.9      |   -   |     2      |           [rotated-retinanet-rbox-le90_r50_fpn_1x_dota](../../configs/rotated_retinanet/rotated-retinanet-rbox-le90_r50_fpn_1x_dota.py)           |                       [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le90/rotated_retinanet_obb_r50_fpn_1x_dota_le90-c0097bc4.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le90/rotated_retinanet_obb_r50_fpn_1x_dota_le90_20220128_130740.log.json)                       |

|  ResNet50 (1024,1024,200)  | 68.79 | le90  |   1x    |   2.36   |      22.4      |   -   |     2      |                [rotated-retinanet-rbox-le90_r50_fpn_amp-1x_dota](../../configs/rotated-retinanet-rbox-le90_r50_fpn_amp-1x_dota.py)                |             [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_fp16_1x_dota_le90/rotated_retinanet_obb_r50_fpn_fp16_1x_dota_le90-01de71b5.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_fp16_1x_dota_le90/rotated_retinanet_obb_r50_fpn_fp16_1x_dota_le90_20220303_183714.log.json)             |

|  ResNet50 (1024,1024,200)  | 69.49 | le135 |   1x    |   4.05   |      8.6       |   -   |     2      |                        [g_reppoints_r50_fpn_1x_dota_le135](../../configs/g_reppoints/g_reppoints_r50_fpn_1x_dota_le135.py)                        |                                               [model](https://download.openmmlab.com/mmrotate/v0.1.0/g_reppoints/g_reppoints_r50_fpn_1x_dota_le135/g_reppoints_r50_fpn_1x_dota_le135-b840eed7.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/g_reppoints/g_reppoints_r50_fpn_1x_dota_le135/g_reppoints_r50_fpn_1x_dota_le135_20220202_233631.log.json)                                               |

|  ResNet50 (1024,1024,200)  | 69.51 | le90  |   1x    |   4.40   |      24.0      |   -   |     2      | [rotated-retinanet-rbox-le90_r50_fpn_csl-gaussian_amp-1x_dota](../../configs/csl/rotated-retinanet-rbox-le90_r50_fpn_csl-gaussian_amp-1x_dota.py) | [model](https://download.openmmlab.com/mmrotate/v0.1.0/csl/rotated_retinanet_obb_csl_gaussian_r50_fpn_fp16_1x_dota_le90/rotated_retinanet_obb_csl_gaussian_r50_fpn_fp16_1x_dota_le90-b4271aed.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/csl/rotated_retinanet_obb_csl_gaussian_r50_fpn_fp16_1x_dota_le90/rotated_retinanet_obb_csl_gaussian_r50_fpn_fp16_1x_dota_le90_20220321_010033.log.json) |

|  ResNet50 (1024,1024,200)  | 69.55 |  oc   |   1x    |   3.39   |      15.5      |   -   |     2      |                [rotated-retinanet-hbox-oc_r50_fpn_gwd_1x_dota](../../configs/gwd/rotated-retinanet-hbox-oc_r50_fpn_gwd_1x_dota.py)                |                                 [model](https://download.openmmlab.com/mmrotate/v0.1.0/gwd/rotated_retinanet_hbb_gwd_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_gwd_r50_fpn_1x_dota_oc-41fd7805.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/gwd/rotated_retinanet_hbb_gwd_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_gwd_r50_fpn_1x_dota_oc_20220120_152421.log.json)                                 |

|  ResNet50 (1024,1024,200)  | 69.60 | le90  |   1x    |   3.38   |      15.1      |   -   |     2      |           [rotated-retinanet-hbox-le90_r50_fpn_kfiou_1x_dota](../../configs/kfiou/rotated-retinanet-hbox-le90_r50_fpn_kfiou_1x_dota.py)           |                       [model](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le90/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le90-03e02f75.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le90/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le90_20220209_173225.log.json)                       |

|  ResNet50 (1024,1024,200)  | 69.63 | le135 |   1x    |   3.45   |      16.1      |   -   |     2      |                                     [cfa-qbox_r50_fpn_1x_dota](../../configs/cfa/cfa-qbox_r50_fpn_1x_dota.py)                                     |                                                                       [model](https://download.openmmlab.com/mmrotate/v0.1.0/cfa/cfa_r50_fpn_1x_dota_le135/cfa_r50_fpn_1x_dota_le135-aed1cbc6.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/cfa/cfa_r50_fpn_1x_dota_le135/cfa_r50_fpn_1x_dota_le135_20220205_144859.log.json)                                                                       |

|  ResNet50 (1024,1024,200)  | 69.76 |  oc   |   1x    |   3.39   |      15.6      |   -   |     2      |             [rotated-retinanet-hbox-oc_r50_fpn_kfiou_1x_dota](../../configs/kfiou/rotated-retinanet-hbox-oc_r50_fpn_kfiou_1x_dota.py)             |                           [model](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_oc-c00be030.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_oc_20220126_081643.log.json)                           |

|  ResNet50 (1024,1024,200)  | 69.77 | le135 |   1x    |   3.38   |      15.3      |   -   |     2      |          [rotated-retinanet-hbox-le135_r50_fpn_kfiou_1x_dota](../../configs/kfiou/rotated-retinanet-hbox-le135_r50_fpn_kfiou_1x_dota.py)          |                     [model](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le135/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le135-0eaa4156.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le135/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le135_20220209_173257.log.json)                     |

|  ResNet50 (1024,1024,200)  | 69.79 | le135 |   1x    |   3.38   |      17.2      |   -   |     2      |          [rotated-retinanet-rbox-le135_r50_fpn_1x_dota](../../configs/rotated_retinanet/rotated-retinanet-rbox-le135_r50_fpn_1x_dota.py)          |                     [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le135/rotated_retinanet_obb_r50_fpn_1x_dota_le135-e4131166.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le135/rotated_retinanet_obb_r50_fpn_1x_dota_le135_20220128_130755.log.json)                     |

|  ResNet50 (1024,1024,200)  | 69.80 |  oc   |   1x    |   3.54   |      12.4      |   -   |     2      |                                    [r3det-oc_r50_fpn_1x_dota](../../configs/r3det/r3det-oc_r50_fpn_1x_dota.py)                                    |                                                                       [model](https://download.openmmlab.com/mmrotate/v0.1.0/r3det/r3det_r50_fpn_1x_dota_oc/r3det_r50_fpn_1x_dota_oc-b1fb045c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/r3det/r3det_r50_fpn_1x_dota_oc/r3det_r50_fpn_1x_dota_oc_20220126_191226.log.json)                                                                       |

|  ResNet50 (1024,1024,200)  | 69.94 |  oc   |   1x    |   3.39   |      15.6      |   -   |     2      |                [rotated-retinanet-hbox-oc_r50_fpn_kld_1x_dota](../../configs/kld/rotated-retinanet-hbox-oc_r50_fpn_kld_1x_dota.py)                |                                 [model](https://download.openmmlab.com/mmrotate/v0.1.0/kld/rotated_retinanet_hbb_kld_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_kld_r50_fpn_1x_dota_oc-49c1f937.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kld/rotated_retinanet_hbb_kld_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_kld_r50_fpn_1x_dota_oc_20220125_201832.log.json)                                 |

|  ResNet50 (1024,1024,200)  | 70.18 |  oc   |   1x    |   3.23   |      15.6      |   -   |     2      |                               [r3det-tiny-oc_r50_fpn_1x_dota](../../configs/r3det/r3det-tiny-oc_r50_fpn_1x_dota.py)                               |                                                             [model](https://download.openmmlab.com/mmrotate/v0.1.0/r3det/r3det_tiny_r50_fpn_1x_dota_oc/r3det_tiny_r50_fpn_1x_dota_oc-c98a616c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/r3det/r3det_tiny_r50_fpn_1x_dota_oc/r3det_tiny_r50_fpn_1x_dota_oc_20220209_171624.log.json)                                                             |

|  ResNet50 (1024,1024,200)  | 70.64 | le90  |   1x    |   3.12   |      18.2      |   -   |     2      |                   [rotated_atss_obb_r50_fpn_1x_dota_le90](../../configs/rotated_atss/rotated_atss_obb_r50_fpn_1x_dota_le90.py)                    |                                      [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_atss/rotated_atss_obb_r50_fpn_1x_dota_le90/rotated_atss_obb_r50_fpn_1x_dota_le90-e029ca06.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_atss/rotated_atss_obb_r50_fpn_1x_dota_le90/rotated_atss_obb_r50_fpn_1x_dota_le90_20220402_002048.log.json)                                      |

|  ResNet50 (1024,1024,200)  | 70.70 | le90  |   1x    |   4.18   |                |   -   |     2      |                  [rotated-fcos-hbox-le90_r50_fpn_1x_dota](../../configs/rotated_fcos/rotated-fcos-hbox-le90_r50_fpn_1x_dota.py)                   |                          [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_sep_angle_r50_fpn_1x_dota_le90/rotated_fcos_sep_angle_r50_fpn_1x_dota_le90-0be71a0c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_sep_angle_r50_fpn_1x_dota_le90/rotated_fcos_sep_angle_r50_fpn_1x_dota_le90_20220409_023250.log.json)                          |

|  ResNet50 (1024,1024,200)  | 71.28 | le90  |   1x    |   4.18   |                |   -   |     2      |                       [rotated-fcos-le90_r50_fpn_1x_dota](../../configs/rotated_fcos/rotated-fcos-le90_r50_fpn_1x_dota.py)                        |                                              [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_r50_fpn_1x_dota_le90/rotated_fcos_r50_fpn_1x_dota_le90-d87568ed.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_r50_fpn_1x_dota_le90/rotated_fcos_r50_fpn_1x_dota_le90_20220413_163526.log.json)                                              |

|  ResNet50 (1024,1024,200)  | 71.76 | le90  |   1x    |   4.23   |                |   -   |     2      |     [rotated-fcos-hbox-le90_r50_fpn_csl-gaussian_1x_dota](../../configs/rotated_fcos/rotated-fcos-hbox-le90_r50_fpn_csl-gaussian_1x_dota.py)      |                    [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_csl_gaussian_r50_fpn_1x_dota_le90/rotated_fcos_csl_gaussian_r50_fpn_1x_dota_le90-4e044ad2.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_csl_gaussian_r50_fpn_1x_dota_le90/rotated_fcos_csl_gaussian_r50_fpn_1x_dota_le90_20220409_080616.log.json)                    |

|  ResNet50 (1024,1024,200)  | 71.83 |  oc   |   1x    |   3.54   |      12.4      |   -   |     2      |                          [r3det-oc_r50_fpn_kld-stable_1x_dota](../../configs/kld/r3det-oc_r50_fpn_kld-stable_1x_dota.py)                          |                                                                 [model](https://download.openmmlab.com/mmrotate/v0.1.0/kld/r3det_kld_r50_fpn_1x_dota_oc/r3det_kld_r50_fpn_1x_dota_oc-31866226.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kld/r3det_kld_r50_fpn_1x_dota_oc/r3det_kld_r50_fpn_1x_dota_oc_20220210_114049.log.json)                                                                 |

|  ResNet50 (1024,1024,200)  | 71.89 | le90  |   1x    |   4.18   |                |   -   |     2      |                   [rotated-fcos-le90_r50_fpn_kld_1x_dota](../../configs/rotated_fcos/rotated-fcos-le90_r50_fpn_kld_1x_dota.py)                    |                                      [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_kld_r50_fpn_1x_dota_le90/rotated_fcos_kld_r50_fpn_1x_dota_le90-ecafdb2b.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_kld_r50_fpn_1x_dota_le90/rotated_fcos_kld_r50_fpn_1x_dota_le90_20220409_202939.log.json)                                      |

|  ResNet50 (1024,1024,200)  | 71.94 | le135 |   1x    |   3.45   |      16.1      |   -   |     2      |              [oriented-reppoints-qbox_r50_fpn_1x_dota](../../configs/oriented_reppoints/oriented-reppoints-qbox_r50_fpn_1x_dota.py)               |                           [model](https://download.openmmlab.com/mmrotate/v0.1.0/orientedreppoints/oriented_reppoints_r50_fpn_1x_dota_le135/oriented_reppoints_r50_fpn_1x_dota_le135-ef072de9.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/orientedreppoints/oriented_reppoints_r50_fpn_1x_dota_le135/oriented_reppoints_r50_fpn_1x_dota_le135_20220505_092853.log.json)                           |

|  ResNet50 (1024,1024,200)  | 72.29 | le135 |   1x    |   3.19   |      18.8      |   -   |     2      |                  [rotated_atss_obb_r50_fpn_1x_dota_le135](../../configs/rotated_atss/rotated_atss_obb_r50_fpn_1x_dota_le135.py)                   |                                    [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_atss/rotated_atss_obb_r50_fpn_1x_dota_le135/rotated_atss_obb_r50_fpn_1x_dota_le135-eab7bc12.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_atss/rotated_atss_obb_r50_fpn_1x_dota_le135/rotated_atss_obb_r50_fpn_1x_dota_le135_20220402_002138.log.json)                                    |

|  ResNet50 (1024,1024,200)  | 72.68 |  oc   |   1x    |   3.62   |      12.2      |   -   |     2      |                           [r3det-oc_r50_fpn_kfiou-ln_1x_dota](../../configs/kfiou/r3det-oc_r50_fpn_kfiou-ln_1x_dota.py)                           |                                                     [model](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/r3det_kfiou_ln_r50_fpn_1x_dota_oc/r3det_kfiou_ln_r50_fpn_1x_dota_oc-8e7f049d.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/r3det_kfiou_ln_r50_fpn_1x_dota_oc/r3det_kfiou_ln_r50_fpn_1x_dota_oc_20220123_074507.log.json)                                                     |

|  ResNet50 (1024,1024,200)  | 72.76 |  oc   |   1x    |   3.44   |      14.0      |   -   |     2      |                            [r3det-tiny-oc_r50_fpn_kld_1x_dota](../../configs/kld/r3det-tiny-oc_r50_fpn_kld_1x_dota.py)                            |                                                       [model](https://download.openmmlab.com/mmrotate/v0.1.0/kld/r3det_tiny_kld_r50_fpn_1x_dota_oc/r3det_tiny_kld_r50_fpn_1x_dota_oc-589e142a.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kld/r3det_tiny_kld_r50_fpn_1x_dota_oc/r3det_tiny_kld_r50_fpn_1x_dota_oc_20220209_172917.log.json)                                                       |

|  ResNet50 (1024,1024,200)  | 73.23 | le90  |   1x    |   8.45   |      16.4      |   -   |     2      |                    [gliding-vertex-rbox_r50_fpn_1x_dota](../../configs/gliding_vertex/gliding-vertex-rbox_r50_fpn_1x_dota.py)                     |                                        [model](https://download.openmmlab.com/mmrotate/v0.1.0/gliding_vertex/gliding_vertex_r50_fpn_1x_dota_le90/gliding_vertex_r50_fpn_1x_dota_le90-12e7423c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/gliding_vertex/gliding_vertex_r50_fpn_1x_dota_le90/gliding_vertex_r50_fpn_1x_dota_le90_20220129_085529.log.json)                                        |

|  ResNet50 (1024,1024,200)  | 73.40 | le90  |   1x    |   8.46   |      16.5      |   -   |     2      |             [rotated-faster-rcnn-le90_r50_fpn_1x_dota](../../configs/rotated_faster_rcnn/rotated-faster-rcnn-le90_r50_fpn_1x_dota.py)             |                         [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_faster_rcnn/rotated_faster_rcnn_r50_fpn_1x_dota_le90/rotated_faster_rcnn_r50_fpn_1x_dota_le90-0393aa5c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_faster_rcnn/rotated_faster_rcnn_r50_fpn_1x_dota_le90/rotated_faster_rcnn_r50_fpn_1x_dota_le90_20220131_082156.log.json)                         |

|  ResNet50 (1024,1024,200)  | 73.45 |  oc   |   40e   |   3.45   |      16.1      |   -   |     2      |                                    [cfa-qbox_r50_fpn_40e_dota](../../configs/cfa/cfa-qbox_r50_fpn_40e_dota.py)                                    |                                                                           [model](https://download.openmmlab.com/mmrotate/v0.1.0/cfa/cfa_r50_fpn_40e_dota_oc/cfa_r50_fpn_40e_dota_oc-2f387232.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/cfa/cfa_r50_fpn_40e_dota_oc/cfa_r50_fpn_40e_dota_oc_20220209_171237.log.json)                                                                           |

|  ResNet50 (1024,1024,200)  | 73.91 | le135 |   1x    |   3.14   |      15.5      |   -   |     2      |                               [s2anet-le135_r50_fpn_1x_dota](../../configs/s2anet/s2anet-le135_r50_fpn_1x_dota.py)                                |                                                              [model](https://download.openmmlab.com/mmrotate/v0.1.0/s2anet/s2anet_r50_fpn_1x_dota_le135/s2anet_r50_fpn_1x_dota_le135-5dfcf396.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/s2anet/s2anet_r50_fpn_1x_dota_le135/s2anet_r50_fpn_1x_dota_le135_20220124_163529.log.json)                                                              |

|  ResNet50 (1024,1024,200)  | 74.19 | le135 |   1x    |   2.17   |      17.4      |   -   |     2      |                               [s2anet-le135_r50_fpn_amp-1x_dota](../../configs/s2anet-le135_r50_fpn_amp-1x_dota.py)                               |                                                    [model](https://download.openmmlab.com/mmrotate/v0.1.0/s2anet/s2anet_r50_fpn_fp16_1x_dota_le135/s2anet_r50_fpn_fp16_1x_dota_le135-5cac515c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/s2anet/s2anet_r50_fpn_fp16_1x_dota_le135/s2anet_r50_fpn_fp16_1x_dota_le135_20220303_194910.log.json)                                                    |

|  ResNet50 (1024,1024,200)  | 75.63 | le90  |   1x    |   7.37   |      21.2      |   -   |     2      |                         [oriented-rcnn-le90_r50_fpn_amp-1x_dota](../../configs/oriented-rcnn-le90_r50_fpn_amp-1x_dota.py)                         |                                 [model](https://download.openmmlab.com/mmrotate/v0.1.0/oriented_rcnn/oriented_rcnn_r50_fpn_fp16_1x_dota_le90/oriented_rcnn_r50_fpn_fp16_1x_dota_le90-57c88621.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/oriented_rcnn/oriented_rcnn_r50_fpn_fp16_1x_dota_le90/oriented_rcnn_r50_fpn_fp16_1x_dota_le90_20220303_195049.log.json)                                 |

|  ResNet50 (1024,1024,200)  | 75.69 | le90  |   1x    |   8.46   |      16.2      |   -   |     2      |                      [oriented-rcnn-le90_r50_fpn_1x_dota](../../configs/oriented_rcnn/oriented-rcnn-le90_r50_fpn_1x_dota.py)                      |                                           [model](https://download.openmmlab.com/mmrotate/v0.1.0/oriented_rcnn/oriented_rcnn_r50_fpn_1x_dota_le90/oriented_rcnn_r50_fpn_1x_dota_le90-6d2b2ce0.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/oriented_rcnn/oriented_rcnn_r50_fpn_1x_dota_le90/oriented_rcnn_r50_fpn_1x_dota_le90_20220127_100150.log.json)                                           |

|  ResNet50 (1024,1024,200)  | 75.75 | le90  |   1x    |   7.56   |      19.3      |   -   |     2      |                             [roi-trans-le90_r50_fpn_amp-1x_dota](../../configs/roi-trans-le90_r50_fpn_amp-1x_dota.py)                             |                                             [model](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_r50_fpn_fp16_1x_dota_le90/roi_trans_r50_fpn_fp16_1x_dota_le90-62eb88b1.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_r50_fpn_fp16_1x_dota_le90/roi_trans_r50_fpn_fp16_1x_dota_le90_20220303_193513.log.json)                                             |

| ReResNet50 (1024,1024,200) | 75.99 | le90  |   1x    |   7.71   |      13.3      |   -   |     2      |                              [redet-le90_re50_refpn_amp-1x_dota](../../configs/redet-le90_re50_refpn_amp-1x_dota.py)                              |                                                   [model](https://download.openmmlab.com/mmrotate/v0.1.0/redet/redet_re50_refpn_fp16_1x_dota_le90/redet_re50_refpn_fp16_1x_dota_le90-1e34da2d.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/redet/redet_re50_refpn_fp16_1x_dota_le90/redet_re50_refpn_fp16_1x_dota_le90_20220303_194725.log.json)                                                   |

|  ResNet50 (1024,1024,200)  | 76.08 | le90  |   1x    |   8.67   |      14.4      |   -   |     2      |                            [roi-trans-le90_r50_fpn_1x_dota](../../configs/roi_trans/roi-trans-le90_r50_fpn_1x_dota.py)                            |                                                       [model](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_r50_fpn_1x_dota_le90/roi_trans_r50_fpn_1x_dota_le90-d1f0b77a.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_r50_fpn_1x_dota_le90/roi_trans_r50_fpn_1x_dota_le90_20220130_132727.log.json)                                                       |

|  ResNet50 (1024,1024,500)  | 76.50 | le90  |   1x    |          |      17.5      | MS+RR |     2      |     [rotated-retinanet-rbox-le90_r50_fpn_rr-1x_dota-ms](../../configs/rotated_retinanet/rotated-retinanet-rbox-le90_r50_fpn_rr-1x_dota-ms.py)     |           [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_ms_rr_le90/rotated_retinanet_obb_r50_fpn_1x_dota_ms_rr_le90-1da1ec9c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_ms_rr_le90/rotated_retinanet_obb_r50_fpn_1x_dota_ms_rr_le90_20220210_114843.log.json)           |

| ReResNet50 (1024,1024,200) | 76.68 | le90  |   1x    |   9.32   |      10.9      |   -   |     2      |                               [redet-le90_re50_refpn_1x_dota](../../configs/redet/redet-le90_re50_refpn_1x_dota.py)                               |                                                                 [model](https://download.openmmlab.com/mmrotate/v0.1.0/redet/redet_re50_fpn_1x_dota_le90/redet_re50_fpn_1x_dota_le90-724ab2da.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/redet/redet_re50_fpn_1x_dota_le90/redet_re50_fpn_1x_dota_le90_20220130_132751.log.json)                                                                 |

| Swin-tiny (1024,1024,200)  | 77.51 | le90  |   1x    |          |      10.9      |   -   |     2      |                      [roi-trans-le90_swin-tiny_fpn_1x_dota](../../configs/roi_trans/roi-trans-le90_swin-tiny_fpn_1x_dota.py)                      |                                           [model](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_swin_tiny_fpn_1x_dota_le90/roi_trans_swin_tiny_fpn_1x_dota_le90-ddeee9ae.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_swin_tiny_fpn_1x_dota_le90/roi_trans_swin_tiny_fpn_1x_dota_le90_20220131_083622.log.json)                                           |

|  ResNet50 (1024,1024,500)  | 79.66 | le90  |   1x    |          |      14.4      | MS+RR |     2      |                      [roi_trans_r50_fpn_1x_dota_ms_rr_le90](../../configs/roi_trans/roi_trans_r50_fpn_1x_dota_ms_rr_le90.py)                      |                                           [model](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_r50_fpn_1x_dota_ms_rr_le90/roi_trans_r50_fpn_1x_dota_ms_rr_le90-fa99496f.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_r50_fpn_1x_dota_ms_rr_le90/roi_trans_r50_fpn_1x_dota_ms_rr_le90_20220205_171729.log.json)                                           |

| ReResNet50 (1024,1024,500) | 79.87 | le90  |   1x    |          |      10.9      | MS+RR |     2      |                         [redet-le90_re50_refpn_rr-1x_dota-ms](../../configs/redet/redet-le90_re50_refpn_rr-1x_dota-ms.py)                         |                                                     [model](https://download.openmmlab.com/mmrotate/v0.1.0/redet/redet_re50_fpn_1x_dota_ms_rr_le90/redet_re50_fpn_1x_dota_ms_rr_le90-fc9217b5.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/redet/redet_re50_fpn_1x_dota_ms_rr_le90/redet_re50_fpn_1x_dota_ms_rr_le90_20220206_105343.log.json)                                                     |

|  ResNet50 (1024,1024,200)  | 68.75 | le90  |   1x    |   6.25   |                |   -   |     2      |                          [h2rbox-le90_r50_fpn_adamw-1x_dota](../../configs/h2rbox/h2rbox-le90_r50_fpn_adamw-1x_dota.py)                           |                                                        [model](https://download.openmmlab.com/mmrotate/v1.0/h2rbox/h2rbox-le90_r50_fpn_adamw-1x_dota/h2rbox-le90_r50_fpn_adamw-1x_dota-d02c933a.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/h2rbox/h2rbox-le90_r50_fpn_adamw-1x_dota/h2rbox-le90_r50_fpn_adamw-1x_dota-20221124_153420.json)                                                        |

- `MS` means multiple scale image split.
- `RR` means random rotation.

The above models are trained with 1 * 1080Ti/2080Ti and inferred with 1 * 2080Ti.

# Results Summary

After the evaluation is complete, the results need to be printed on the screen or saved. This process is controlled by the summarizer.

```{note}
If the summarizer appears in the overall config, all the evaluation results will be output according to the following logic.
If the summarizer does not appear in the overall config, the evaluation results will be output in the order they appear in the `dataset` config.
```

## Example

A typical summarizer configuration file is as follows:

```python
        'race',
    ],
    ]
)
```

The output is:

```text

```text
```

The summarizer tries to read the evaluation scores from the `{work_dir}/results/` directory using the `models` and `datasets` in the config as the full set. It then displays them in the order of the `summarizer.dataset_abbrs` list. Moreover, the summarizer tries to compute some aggregated metrics using `summarizer.summary_groups`. The `name` metric is only generated if and only if all values in `subsets` exist. This means if some scores are missing, the aggregated metric will also be missing. If scores can't be fetched by the above methods, the summarizer will use `-` in the respective cell of the table.

In addition, the output consists of multiple columns:

- The `dataset` column corresponds to the `summarizer.dataset_abbrs` configuration.
- The `version` column is the hash value of the dataset, which considers the dataset's evaluation method, prompt words, output length limit, etc. Users can verify whether two evaluation results are comparable using this column.
- The `metric` column indicates the evaluation method of this metric. For specific details, [metrics](./metrics.md).
- The `mode` column indicates how the inference result is obtained. Possible values are `ppl` / `gen`. For items in `summarizer.summary_groups`, if the methods of obtaining `subsets` are consistent, its value will be the same as subsets, otherwise it will be `mixed`.
- The subsequent columns represent different models.

## Field Description

The fields of summarizer are explained as follows:

- `dataset_abbrs`: (list, optional) Display list items. If omitted, all evaluation results will be output.

- `summary_groups`: (list, optional) Configuration for aggregated metrics.

The fields in `summary_groups` are:

- `name`: (str) Name of the aggregated metric.
- `subsets`: (list) Names of the metrics that are aggregated. Note that it can not only be the original `dataset_abbr` but also the name of another aggregated metric.
- `weights`: (list, optional) Weights of the metrics being aggregated. If omitted, the default is to use unweighted averaging.

Please note that we have stored the summary groups of datasets like MMLU, C-Eval, etc., under the `configs/summarizers/groups` path. It's recommended to consider using them first.

# CSPNeXt ImageNet Pre-training

In this folder, we provide the imagenet pre-training config of RTMDet's backbone CSPNeXt.

To train with these configs, please install [MMPreTrain](https://github.com/open-mmlab/mmpretrain) first.

Install by MIM:

```shell
mim install mmpretrain
```

or install by pip:

```shell
pip install mmpretrain
```

## Prepare Dataset

To pre-train on ImageNet, you need to prepare the dataset first. Please refer to the [guide](https://mmpretrain.readthedocs.io/en/latest/user_guides/dataset_prepare.html#imagenet).

## How to Train

You can use the classification config in the same way as the detection config.

For single-GPU training, run:

```shell
```

For multi-GPU training, run:

```shell
```

```

```shell
```

More details can be found in [user guides](https://mmdetection.readthedocs.io/en/latest/user_guides/train.html).

# Build for macOS-arm64

- [Build for macOS-arm64](#build-for-macos-arm64)

- cmake

  ```
  ```

- clang

  install Xcode or Command Line Tools

  ```
  ```

Please refer to [get_started](../get_started.md) to install conda.

```bash
# install pytorch & mmcv

```

```bash
```

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
    <td>
    </td>
</tbody>
</table>

This document focus on Core ML. The installation of ONNX Runtime, ncnn and TorchScript is similar to the linux platform, please refer to the document [linux-x86_64](linux-x86_64.md) for installation.

The TorchScript model is used as the IR in the conversion process of the Core ML model. In order to support the custom operator in some models like detection models in mmdet, libtorch needs to be installed.

<thead>
  <tr>
    <th>INSTALLATION</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>
    </td>
  </tr>
  <tr>
  <td>

<tr>
    <td>
    </td>
  </tr>
  <tr>
  <td>
  1. Libtorch doesn't provide prebuilt arm library for macOS, so you need to compile it yourself. Please note that the version of libtorch must be consistent with the version of pytorch. <br>
  2. Take LibTorch 1.9.0 as an example. You can install it like this:
cd pytorch
cmake .. \
  </td>
  </tr>
</tbody>
</table>

```bash
```

  </td>
  </tr>
</tbody>
</table>

```bash
```

### Build Model Converter

  Core ML uses torchscript as IR, to convert models in some codebases like mmdet, you need to compile torchscript custom operators

- **torchscript** custom operators

  ```bash
  ```

### Install Model Converter

```bash
# You should use `conda install` to install the grpcio in requirements/runtime.txt
```

```bash
```

**Note**

```

```bash
```

**Note**

The following shows an example of building an SDK using Core ML as the inference engine.

  ```Bash
  cmake .. \

  ```

# Meta Template

## 背景

在 LLM 的 Supervised Fine-Tuning (SFT) 过程中，我们常常会根据实际的要求往对话内注入一些预定义的字符串，以求模型能按照一定的要求输出内容。例如，在一些 `chat` 模型的微调中，我们可能会在每段对话的开头加入系统层级的指令，并约定一套的格式表示用户与模型之间的对话。在一段对话中，模型期望文本的格式可能如下：

```Bash
Meta instruction: You are now a helpful and harmless AI assistant.
HUMAN: Hi!<eoh>\n
Bot: Hello! How may I assist you?<eob>\n
```

在评测时，我们也需要按照约定的格式输入问题，模型才能发挥出其最大的性能。

此外， API 模型也存在着类似的情况。一般 API 的对话模型都允许用户在调用时传入历史对话，还有些模型也允许传入 SYSTEM 层级的指令。为了更好地评测 API 模型的能力，我们希望在评测 API 模型时可以尽量让数据更贴合 API 模型本身的多轮对话模板，而并非把所有内容塞进一段指令当中。

因此，我们需要针对不同模型指定不同的解析模板。在 OpenCompass 中，我们将这套解析模板其称为 **Meta Template**。Meta Template 与模型的配置相绑定，在运行时与数据集的对话式模板相结合，最终产生最适合当前模型的 prompt。

```Python
# 指定时只需要把 meta_template 字段传入模型
models = [
    dict(
        type='AnyModel',
        meta_template = ...,  # meta tmplate
    )
]
```

接下来，我们会介绍 Meta Template 在两种模型上的配置方法。建议读者在阅读本章前，先了解[对话式模板](./prompt_template.md#对话式-prompt)的基本语法。

```{note}

dict(
    )
]
```

```{note}
在某些情况下（例如对基座的测试），我们并不需要在正常对话中注入任何的指令，此时我们可以将 meta template 置空。在这种情况下，模型接收到的 prompt 仅由数据集配置定义，是一个普通的字符串。若数据集配置使用的是对话式模板，不同角色的发言将会由 \n 拼接而成。
```

## 应用在语言模型上

下图展示了在 2-shot learning 的情况下，数据从数据集中经过 prompt template 和 meta template，最终构建出 prompt 的几种情况。读者可以该图为参考，方便理解后续的章节。

![](https://user-images.githubusercontent.com/22607038/251195073-85808807-6359-44df-8a19-9f5d00c591ec.png)

我们将会结合几个例子讲解 meta template 的定义方式。

假设根据数据集的对话式模板，产生了下面的 PromptList：

```python
PromptList([
    dict(role='HUMAN', prompt='1+1=?'),
    dict(role='BOT', prompt='2'),
    dict(role='HUMAN', prompt='2+2=?'),
    dict(role='BOT', prompt='4'),
])
```

我们希望把这段对话传到一个已经经过 SFT 的模型。模型约定的对话中不同的角色的发言以`<角色名>:`开头，并固定以一个特殊 token 和 \\n 结尾。以下是模型期望接收到的完整字符串：

```Plain
<HUMAN>: 1+1=?<eoh>
<BOT>: 2<eob>
<HUMAN>: 2+2=?<eoh>
<BOT>: 4<eob>

])
```

```Plain
```

在 meta template 中，我们只需要把每轮对话的格式抽象为如下配置即可：

```Python
# model meta template
meta_template = dict(
    round=[
          dict(role='HUMAN', begin='<HUMAN>: ', end='<eoh>\n'),
          dict(role='BOT', begin='<BOT>: ', end='<eob>\n'),
    ],
 )
```

有的数据集中可能会引入 SYSTEM 级别的角色：

```python
    dict(role='SYSTEM', fallback_role='HUMAN', prompt='Solve the following math questions'),
])
```

假设模型同样接受 SYSTEM 这个角色，且期望输入为：

```Bash
<SYSTEM>: Solve the following math questions<eosys>\n
<HUMAN>: 1+1=?<eoh>\n
<BOT>: 2<eob>\n
<HUMAN>: 2+2=?<eoh>\n
<BOT>: 4<eob>\n
end of conversation
```

])
```

```Bash
```

我们就可以把 SYSTEM 角色的定义放进 `reserved_roles` 中。`reserved_roles` 中的角色不会在常规对话中出现，但允许数据集配置的对话式模板在 `begin` 或者 `end` 中调用。

```Python
    round=[
    ],
    reserved_roles=[dict(role='SYSTEM', begin='<SYSTEM>: ', end='<eosys>\n'),],
 ),
```

若模型并不接受 SYSTEM 角色，则**不需要**配置此项，也能正常运行。这种情况下，模型会接收到的字符串变成了：

```Python
<HUMAN>: Solve the following math questions<eoh>\n
```

这是因为在 OpenCompass 预定义的数据集中，每个 `SYSTEM` 发言都会有一个 `fallback_role='HUMAN'`，即若 meta template 中的 `SYSTEM` 角色不存在，发言者会被切换至 `HUMAN` 角色。

```

有的模型还可能需要考虑在对话开始或结束时嵌入其它字符串，如系统指令：

```Bash
```

此时，我们可以通过指定 `begin` 和 `end` 参数指定这些字符串。

```Python
    round=[
    ],
    begin="Meta instruction: You are now a helpful and harmless AI assistant.",
    end="end of conversion",
 ),
```

reserved_roles=[dict(role='SYSTEM', begin='<SYSTEM>: ', end='<eosys>\n'),],
 ),
```

在**生成式**的任务评测中，我们也不会将答案直接输入模型，而是通过截断 prompt，在保留上文的同时，把模型输出的答案留空。

```Bash
<BOT>:
```

我们只需要把 BOT 的配置中把 `generate` 字段置为 True ，OpenCompass 即会将 BOT 的最后一句话留给模型生成：

```Python
    round=[
          dict(role='BOT', begin='<BOT>: ', end='<eob>\n', generate=True),
    ],
 ),
```

 ),
```

需要注意的是，`generate` 仅影响生成式推理。在进行判别式推理时，模型接受到的 prompt 仍然是完整的。

### 全量字段介绍

```Bash
models = [
    dict(meta_template = dict(
            begin="Meta instruction: You are now a helpful and harmless AI assistant.",
            round=[
                    dict(role='HUMAN', begin='HUMAN: ', end='<eoh>\n'),  # begin and end can be a list of strings or integers.
                    dict(role='THOUGHTS', begin='THOUGHTS: ', end='<eot>\n', prompt='None'), # Here we can set the default prompt, which may be overridden by the speicfic dataset
                    dict(role='BOT', begin='BOT: ', generate=True, end='<eob>\n'),
            ],
            end="end of conversion",
            reserved_roles=[dict(role='SYSTEM', begin='SYSTEM: ', end='\n'),],
            eos_token_id=10000,
         ),
     )
]

],
         ),
     )
]
```

meta_template 是一个字典，该字典可以包含以下数个字段：

- `begin`，`end` ：(str，可选) prompt 的开头和结尾，通常是一些系统级别的指令。

- `round`：(list) 每一轮对话的模板格式。每轮对话的 prompt 内容由数据集配置的对话式模板控制。

- `reserved_roles`:（list，可选）指定 `round` 中并未出现，但有可能在数据集配置中用到的的预留角色，例如 `SYSTEM` 角色。

- `eos_token_id`:（int, 可选）：指定了该模型的 eos token 的 id。如果不设置，则默认为 tokenizer 中的 eos token id。它的主要作用是在生成式任务中，截取模型的输出结果，因此一般应该被设置为 generate=True 的项所对应的 end 的第一个 token id。

meta_template 的 `round` 指定了一轮对话中每个角色说话的格式，接受一个字典组成的列表，每个字典的关键字如下：

- `role`（str）: 参与对话的角色名，该字符串并不影响实际的 prompt。

- `begin`, `end` (str): 指定该角色在说话时的固定开头或结尾。

- `prompt` (str)：角色的 prompt。在 meta template 中允许留空，但此时必须在数据集配置的 prompt 中指定。

- `generate` (bool): 指定为 True 时，该角色即为模型扮演的角色。在生成任务中，模型接收到的 prompt 会截止到该角色的 `begin` 处，剩下的内容由模型补全。

## 应用在 API 模型上

API 模型的 meta template 与普通模型的 meta template 类似，但配置更为简单。用户可以根据情况，直接使用下面的两种配置之一，即可以多轮对话的方式评测 API 模型：

```Bash
# 若 API 模型不支持 system 指令
meta_template=dict(
    round=[
        dict(role='HUMAN', api_role='HUMAN'),
        dict(role='BOT', api_role='BOT', generate=True)
    ],
)

# 若 API 模型支持 system 指令
    round=[
    ],
    reserved_roles=[
        dict(role='SYSTEM', api_role='SYSTEM'),
    ],
)
```

### 原理

尽管不同 API 模型接受的数据结构不一，但总体上不乏共通之处。接受对话历史的接口里通常允许用户传入以下三个角色的 prompt：

- 用户

- 机器人

- 系统 （可选）

],
    ],
)
```

### 原理

- 用户

- 机器人

- 系统 （可选）

据此 OpenCompass 为 API 模型预设了三个 `api_role`：`HUMAN`, `BOT`, `SYSTEM`，同时约定 API 模型接受的输入除了普通字符串外，还有一种以 `PromptList` 结构表示对话的中间格式。API 模型会将对话重新以多轮对话格式打包，发送至后端。但要激活此功能，需要用户使用上面的 meta template 中把数据集 prompt 模板中的角色 `role` 映射到对应的 `api_role` 中。下图展示了 API 模型接受的输入与 Prompt Template 、Meta Template 之间的关系。

![](https://user-images.githubusercontent.com/22607038/251195872-63aa7d30-045a-4837-84b5-11b09f07fb18.png)

## 调试

如果需要调试 prompt，建议在准备好配置文件后，使用 `tools/prompt_viewer.py` 脚本预览模型实际接收到的 prompt。阅读[这里](../tools.md#prompt-viewer)了解更多。

# 模型部署（待更新）

MMDet3D 1.1 完全基于 [MMDeploy](https://mmdeploy.readthedocs.io/) 來部署模型。
我们将在下一个版本完善这个文档。

# onnxruntime Support

## Introduction of ONNX Runtime

**ONNX Runtime** is a cross-platform inference and training accelerator compatible with many popular ML/DNN frameworks. Check its [github](https://github.com/microsoft/onnxruntime) for more information.

*Please note that only **onnxruntime>=1.8.1** of on Linux platform is supported by now.*

### Install ONNX Runtime python package

- CPU Version

```bash
pip install onnxruntime==1.8.1 # if you want to use cpu version
```

- GPU Version

```bash
pip install onnxruntime-gpu==1.8.1 # if you want to use gpu version
```

### Install float16 conversion tool (optional)

If you want to use float16 precision, install the tool by running the following script:

```bash
pip install onnx onnxconverter-common
```

## Build custom ops

### Download ONNXRuntime Library

```bash
```

Download `onnxruntime-linux-*.tgz` library from ONNX Runtime [releases](https://github.com/microsoft/onnxruntime/releases/tag/v1.8.1), extract it, expose `ONNXRUNTIME_DIR` and finally add the lib path to `LD_LIBRARY_PATH` as below:

```bash

```

In X64 GPU:

```bash

cd onnxruntime-linux-x64-gpu-1.8.1

```

In Arm GPU:

```bash
# Arm not have 1.8.1 version package
wget https://github.com/microsoft/onnxruntime/releases/download/v1.10.0/onnxruntime-linux-aarch64-1.10.0.tgz

tar -zxvf onnxruntime-linux-aarch64-1.10.0.tgz
cd onnxruntime-linux-aarch64-1.10.0
```

You can also go to [ONNX Runtime Release](https://github.com/microsoft/onnxruntime/releases) to find corresponding release version package.

### Build on Linux

```bash
cd ${MMDEPLOY_DIR} # To MMDeploy root directory
mkdir -p build && cd build
cmake -DMMDEPLOY_TARGET_DEVICES='cpu' -DMMDEPLOY_TARGET_BACKENDS=ort -DONNXRUNTIME_DIR=${ONNXRUNTIME_DIR} ..
make -j$(nproc) && make install
```

```bash

```

```bash
cmake -DMMDEPLOY_TARGET_DEVICES='cuda' -DMMDEPLOY_TARGET_BACKENDS=ort -DONNXRUNTIME_DIR=${ONNXRUNTIME_DIR} ..
```

## How to convert a model

- You could follow the instructions of tutorial [How to convert model](../02-how-to-run/convert_model.md)

## How to add a new custom op

## Reminder

- The custom operator is not included in [supported operator list](https://github.com/microsoft/onnxruntime/blob/master/docs/OperatorKernels.md) in ONNX Runtime.
- The custom operator should be able to be exported to ONNX.

#### Main procedures

Take custom operator `roi_align` for example.

1. Create a `roi_align` directory in ONNX Runtime source directory `${MMDEPLOY_DIR}/csrc/backend_ops/onnxruntime/`
2. Add header and source file into `roi_align` directory `${MMDEPLOY_DIR}/csrc/backend_ops/onnxruntime/roi_align/`

3. Add unit test into `tests/test_ops/test_ops.py`
   Check [here](../../../tests/test_ops/test_ops.py) for examples.

**Finally, welcome to send us PR of adding custom operators for ONNX Runtime in MMDeploy.** :nerd_face:

- [How to export Pytorch model with custom op to ONNX and run it in ONNX Runtime](https://github.com/onnx/tutorials/blob/master/PyTorchCustomOperator/README.md)
- [How to add a custom operator/kernel in ONNX Runtime](https://onnxruntime.ai/docs/reference/operators/add-custom-op.html)

# RDN (CVPR'2018)

> [Residual Dense Network for Image Super-Resolution](https://arxiv.org/abs/1802.08797)

A very deep convolutional neural network (CNN) has recently achieved great success for image super-resolution (SR) and offered hierarchical features as well. However, most deep CNN based SR models do not make full use of the hierarchical features from the original low-resolution (LR) images, thereby achieving relatively-low performance. In this paper, we propose a novel residual dense network (RDN) to address this problem in image SR. We fully exploit the hierarchical features from all the convolutional layers. Specifically, we propose residual dense block (RDB) to extract abundant local features via dense connected convolutional layers. RDB further allows direct connections from the state of preceding RDB to all the layers of current RDB, leading to a contiguous memory (CM) mechanism. Local feature fusion in RDB is then used to adaptively learn more effective features from preceding and current local features and stabilizes the training of wider network. After fully obtaining dense local features, we use global feature fusion to jointly and adaptively learn global hierarchical features in a holistic way. Extensive experiments on benchmark datasets with different degradation models show that our RDN achieves favorable performance against state-of-the-art methods.

 <img src="https://user-images.githubusercontent.com/7676947/144034203-c3a4ac55-d815-4180-a345-f80ab5ca68b6.png" width="400"/>
</div >

The metrics are `PSNR and SSIM` .

|                               Model                                | Dataset |  PSNR   |  SSIM  | Training Resources |                                             Download                                             |
| :----------------------------------------------------------------: | :-----: | :-----: | :----: | :----------------: | :----------------------------------------------------------------------------------------------: |

| [rdn_x4c64b16_g1_1000k_div2k](./rdn_x4c64b16_1xb16-1000k_div2k.py) |  Set5   | 30.4922 | 0.8548 |    1 (TITAN Xp)    | [model](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x4c64b16_g1_1000k_div2k_20210419-3577d44f.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x4c64b16_g1_1000k_div2k_20210419-3577d44f.log.json) |
| [rdn_x4c64b16_g1_1000k_div2k](./rdn_x4c64b16_1xb16-1000k_div2k.py) |  Set14  | 26.9570 | 0.7423 |    1 (TITAN Xp)    | [model](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x4c64b16_g1_1000k_div2k_20210419-3577d44f.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x4c64b16_g1_1000k_div2k_20210419-3577d44f.log.json) |

| [rdn_x4c64b16_g1_1000k_div2k](./rdn_x4c64b16_1xb16-1000k_div2k.py) |  DIV2K  | 29.1925 | 0.8233 |    1 (TITAN Xp)    | [model](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x4c64b16_g1_1000k_div2k_20210419-3577d44f.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x4c64b16_g1_1000k_div2k_20210419-3577d44f.log.json) |
| [rdn_x3c64b16_g1_1000k_div2k](./rdn_x3c64b16_1xb16-1000k_div2k.py) |  Set5   | 32.6051 | 0.8943 |    1 (TITAN Xp)    | [model](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x3c64b16_g1_1000k_div2k_20210419-b93cb6aa.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x3c64b16_g1_1000k_div2k_20210419-b93cb6aa.log.json) |

| [rdn_x3c64b16_g1_1000k_div2k](./rdn_x3c64b16_1xb16-1000k_div2k.py) |  Set14  | 28.6338 | 0.8077 |    1 (TITAN Xp)    | [model](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x3c64b16_g1_1000k_div2k_20210419-b93cb6aa.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x3c64b16_g1_1000k_div2k_20210419-b93cb6aa.log.json) |
| [rdn_x3c64b16_g1_1000k_div2k](./rdn_x3c64b16_1xb16-1000k_div2k.py) |  DIV2K  | 31.2153 | 0.8763 |    1 (TITAN Xp)    | [model](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x3c64b16_g1_1000k_div2k_20210419-b93cb6aa.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x3c64b16_g1_1000k_div2k_20210419-b93cb6aa.log.json) |

| [rdn_x2c64b16_g1_1000k_div2k](./rdn_x2c64b16_1xb16-1000k_div2k.py) |  Set5   | 35.9883 | 0.9385 |    1 (TITAN Xp)    | [model](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x2c64b16_g1_1000k_div2k_20210419-dc146009.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x2c64b16_g1_1000k_div2k_20210419-dc146009.log.json) |
| [rdn_x2c64b16_g1_1000k_div2k](./rdn_x2c64b16_1xb16-1000k_div2k.py) |  Set14  | 31.8366 | 0.8920 |    1 (TITAN Xp)    | [model](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x2c64b16_g1_1000k_div2k_20210419-dc146009.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x2c64b16_g1_1000k_div2k_20210419-dc146009.log.json) |

| [rdn_x2c64b16_g1_1000k_div2k](./rdn_x2c64b16_1xb16-1000k_div2k.py) |  DIV2K  | 34.9392 | 0.9380 |    1 (TITAN Xp)    | [model](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x2c64b16_g1_1000k_div2k_20210419-dc146009.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x2c64b16_g1_1000k_div2k_20210419-dc146009.log.json) |

**Train**

<details>

```shell
# cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/rdn/rdn_x4c64b16_1xb16-1000k_div2k.py

python tools/train.py configs/rdn/rdn_x4c64b16_1xb16-1000k_div2k.py

./tools/dist_train.sh configs/rdn/rdn_x4c64b16_1xb16-1000k_div2k.py 8
```

**Test**

<details>

```

**Test**

<details>

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/rdn/rdn_x4c64b16_1xb16-1000k_div2k.py https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x4c64b16_g1_1000k_div2k_20210419-3577d44f.pth

python tools/test.py configs/rdn/rdn_x4c64b16_1xb16-1000k_div2k.py https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x4c64b16_g1_1000k_div2k_20210419-3577d44f.pth

./tools/dist_test.sh configs/rdn/rdn_x4c64b16_1xb16-1000k_div2k.py https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x4c64b16_g1_1000k_div2k_20210419-3577d44f.pth 8
```

```

```bibtex
@inproceedings{zhang2018residual,
  title={Residual dense network for image super-resolution},
  author={Zhang, Yulun and Tian, Yapeng and Kong, Yu and Zhong, Bineng and Fu, Yun},
  pages={2472--2481},
}
```

<a href="https://github.com/sail-sg/poolformer/tree/main/segmentation">Official Repo</a>

<a href="https://github.com/open-mmlab/mmclassification/blob/v0.23.0/mmcls/models/backbones/poolformer.py#L198">Code Snippet</a>

Transformers have shown great potential in computer vision tasks. A common belief is their attention-based token mixer module contributes most to their competence. However, recent works show the attention-based module in transformers can be replaced by spatial MLPs and the resulted models still perform quite well. Based on this observation, we hypothesize that the general architecture of the transformers, instead of the specific token mixer module, is more essential to the model's performance. To verify this, we deliberately replace the attention module in transformers with an embarrassingly simple spatial pooling operator to conduct only the most basic token mixing. Surprisingly, we observe that the derived model, termed as PoolFormer, achieves competitive performance on multiple computer vision tasks. For example, on ImageNet-1K, PoolFormer achieves 82.1% top-1 accuracy, surpassing well-tuned vision transformer/MLP-like baselines DeiT-B/ResMLP-B24 by 0.3%/1.1% accuracy with 35%/52% fewer parameters and 48%/60% fewer MACs. The effectiveness of PoolFormer verifies our hypothesis and urges us to initiate the concept of "MetaFormer", a general architecture abstracted from transformers without specifying the token mixer. Based on the extensive experiments, we argue that MetaFormer is the key player in achieving superior results for recent transformer and MLP-like models on vision tasks. This work calls for more future research dedicated to improving MetaFormer instead of focusing on the token mixer modules. Additionally, our proposed PoolFormer could serve as a starting baseline for future MetaFormer architecture design. Code is available at [this https URL](https://github.com/sail-sg/poolformer)

<img src="https://user-images.githubusercontent.com/15921929/144710761-1635f59a-abde-4946-984c-a2c3f22a19d2.png" width="70%"/>
</div>

```bibtex
}
```

### Usage

- PoolFormer backbone needs to install [MMClassification](https://github.com/open-mmlab/mmclassification) first, which has abundant backbones for downstream tasks.

```shell
pip install "mmpretrain>=1.0.0rc7"
```

- The pretrained models could also be downloaded from [PoolFormer config of MMClassification](https://github.com/open-mmlab/mmclassification/tree/master/configs/poolformer).

### ADE20K

### ADE20K

| Method | Backbone       | Crop Size | pretrain    | Batch Size | Lr schd | Mem (GB) | Inf time (fps) | Device | mIoU  | mIoU(ms+flip) | mIoU\* | mIoU\*(ms+flip) | config                                                                                                                            | download                                                                                                                                                                                                                                                                                                                                                                     |

| FPN    | PoolFormer-S12 | 512x512   | ImageNet-1K | 32         | 40000   | 4.17     | 23.48          | V100   | 36.68 |             - | 37.07  |               - | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/poolformer/fpn_poolformer_s12_8xb4-40k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/poolformer/fpn_poolformer_s12_8x4_512x512_40k_ade20k/fpn_poolformer_s12_8x4_512x512_40k_ade20k_20220501_115154-b5aa2f49.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/poolformer/fpn_poolformer_s12_8x4_512x512_40k_ade20k/fpn_poolformer_s12_8x4_512x512_40k_ade20k_20220501_115154.log.json) |

| FPN    | PoolFormer-S24 | 512x512   | ImageNet-1K | 32         | 40000   | 5.47     | 15.74          | V100   | 40.12 |             - | 40.36  |               - | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/poolformer/fpn_poolformer_s24_8xb4-40k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/poolformer/fpn_poolformer_s24_8x4_512x512_40k_ade20k/fpn_poolformer_s24_8x4_512x512_40k_ade20k_20220503_222049-394a7cf7.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/poolformer/fpn_poolformer_s24_8x4_512x512_40k_ade20k/fpn_poolformer_s24_8x4_512x512_40k_ade20k_20220503_222049.log.json) |

| FPN    | PoolFormer-S36 | 512x512   | ImageNet-1K | 32         | 40000   | 6.77     | 11.34          | V100   | 41.61 |             - | 41.81  |               - | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/poolformer/fpn_poolformer_s36_8xb4-40k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/poolformer/fpn_poolformer_s36_8x4_512x512_40k_ade20k/fpn_poolformer_s36_8x4_512x512_40k_ade20k_20220501_151122-b47e607d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/poolformer/fpn_poolformer_s36_8x4_512x512_40k_ade20k/fpn_poolformer_s36_8x4_512x512_40k_ade20k_20220501_151122.log.json) |

| FPN    | PoolFormer-M36 | 512x512   | ImageNet-1K | 32         | 40000   | 8.59     | 8.97           | V100   | 41.95 |             - | 42.35  |               - | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/poolformer/fpn_poolformer_m36_8xb4-40k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/poolformer/fpn_poolformer_m36_8x4_512x512_40k_ade20k/fpn_poolformer_m36_8x4_512x512_40k_ade20k_20220501_164230-3dc83921.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/poolformer/fpn_poolformer_m36_8x4_512x512_40k_ade20k/fpn_poolformer_m36_8x4_512x512_40k_ade20k_20220501_164230.log.json) |

| FPN    | PoolFormer-M48 | 512x512   | ImageNet-1K | 32         | 40000   | 10.48    | 6.69           | V100   | 42.43 |             - | 42.76  |               - | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/poolformer/fpn_poolformer_m48_8xb4-40k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/poolformer/fpn_poolformer_m48_8x4_512x512_40k_ade20k/fpn_poolformer_m48_8x4_512x512_40k_ade20k_20220504_003923-64168d3b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/poolformer/fpn_poolformer_m48_8x4_512x512_40k_ade20k/fpn_poolformer_m48_8x4_512x512_40k_ade20k_20220504_003923.log.json) |

Note:

- We replace `AlignedResize` in original PoolFormer implementation to `Resize + ResizeToMultiple`.

- `mIoU` with * is collected when `Resize + ResizeToMultiple` is adopted in `test_pipeline`, so do `mIoU` in logs.

- The Test Time Augmentation i.e., "ms+flip" in MMSegmentation v1.x is developing, stay tuned!

- Linux | Windows | macOS
- torchvision 0.7.0
- CUDA 10.1
- NCCL 2
- GCC 5.4.0 or higher

## Environment Setup

```{note}
If you are experienced with PyTorch and have already installed it, just skip this part and jump to the [next section](#installation-steps). Otherwise, you can follow these steps for the preparation.
```

```shell
```

```{code-tab} shell GPU Platform
```

```{code-tab} shell CPU Platform
```

````

## Installation Steps

```

```

````

We recommend that users follow our best practices to install MMOCR. However, the whole process is highly customizable. See [Customize Installation](#customize-installation) section for more information.

**Step 0.** Install  [MMEngine](https://github.com/open-mmlab/mmengine), [MMCV](https://github.com/open-mmlab/mmcv) and [MMDetection](https://github.com/open-mmlab/mmdetection) using [MIM](https://github.com/open-mmlab/mim).

```shell
```

**Step 1.** Install MMOCR.

If you wish to run and develop MMOCR directly, install it from **source** (recommended).

If you use MMOCR as a dependency or third-party package, install it via **MIM**.

````{group-tab} Install from Source

```shell

```shell

git clone https://github.com/open-mmlab/mmocr.git
cd mmocr
# "-v" increases pip's verbosity.
# "-e" means installing the project in editable mode,
# That is, any local modifications on the code will take effect immediately.

```

````

````{group-tab} Install via MIM

```shell

mim install mmocr

```

````

`````

**Step 2. (Optional)** If you wish to use any transform involving `albumentations` (For example, `Albu` in ABINet's pipeline), or any dependency for building documentation or running unit tests, please install the dependency using the following command:

```shell
# install albu
pip install -r requirements/albu.txt
# install the dependencies for building documentation and running unit tests
```

````

```

````

```shell
pip install albumentations>=1.1.0 --no-binary qudida,albumentations
```

````

`````

```{note}

We recommend checking the environment after installing `albumentations` to
ensure that `opencv-python` and `opencv-python-headless` are not installed together, otherwise it might cause unexpected issues. If that's unfortunately the case, please uninstall `opencv-python-headless` to make sure MMOCR's visualization utilities can work.

Refer
to [albumentations's official documentation](https://albumentations.ai/docs/getting_started/installation/#note-on-opencv-dependencies) for more details.

```

You may verify the installation via this inference demo.

````{tab} Python

Run the following code in a Python interpreter:

```python

```

```python
>>> ocr = MMOCRInferencer(det='DBNet', rec='CRNN')
>>> ocr('demo/demo_text_ocr.jpg', show=True, print_result=True)
```
````

````{tab} Shell

If you installed MMOCR from source, you can run the following in MMOCR's root directory:

```shell
python tools/infer.py demo/demo_text_ocr.jpg --det DBNet --rec CRNN --show --print-result
```
````

`````

You should be able to see a pop-up image and the inference result printed out in the console upon successful verification.

    <img src="https://user-images.githubusercontent.com/24622904/187825445-d30cbfa6-5549-4358-97fe-245f08f4ed94.jpg" height="250"/>
</div>
<br />

```bash
# Inference result

</div>
<br />

```bash
{'predictions': [{'rec_texts': ['cbanks', 'docecea', 'grouf', 'pwate', 'chobnsonsg', 'soxee', 'oeioh', 'c', 'sones', 'lbrandec', 'sretalg', '11', 'to8', 'round', 'sale', 'year',
'ally', 'sie', 'sall'], 'rec_scores': [...], 'det_polygons': [...], 'det_scores':
[...]}]}
```

```{note}
If you are running MMOCR on a server without GUI or via SSH tunnel with X11 forwarding disabled, you may not see the pop-up window.
```

```{note}
```

### Install MMCV without MIM

```

```shell
pip install `mmcv>=2.0.0rc1`  -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10/index.html
```

MMOCR can be built for CPU-only environment. In CPU mode you can train (requires MMCV version >= 1.4.4), test or inference a model.

If you try to train/test/inference a model containing above ops, an error will be raised.

|                        Operator                         |                          Model                          |
| :-----------------------------------------------------: | :-----------------------------------------------------: |
| Deformable Convolution/Modulated Deformable Convolution | DBNet (r50dcnv2), DBNet++ (r50dcnv2), FCENet (r50dcnv2) |
|                      SyncBatchNorm                      |                      PANet, PSENet                      |

### Using MMOCR with Docker

We provide a [Dockerfile](https://github.com/open-mmlab/mmocr/blob/master/docker/Dockerfile) to build an image.

```shell
docker build -t mmocr docker/
```

Run it with

```shell
```

Run it with

```shell
docker run --gpus all --shm-size=8g -it -v {DATA_DIR}:/mmocr/data mmocr
```

## Dependency on MMEngine, MMCV & MMDetection

MMOCR has different version requirements on MMEngine, MMCV and MMDetection at each release to guarantee the implementation correctness. Please refer to the table below and ensure the package versions fit the requirement.

| MMOCR          | MMEngine                    | MMCV                       | MMDetection                 |
| -------------- | --------------------------- | -------------------------- | --------------------------- |
| dev-1.x        | 0.7.1 \<= mmengine \< 1.1.0 | 2.0.0rc4 \<= mmcv \< 2.1.0 | 3.0.0rc5 \<= mmdet \< 3.2.0 |
| 1.0.1          | 0.7.1 \<= mmengine \< 1.1.0 | 2.0.0rc4 \<= mmcv \< 2.1.0 | 3.0.0rc5 \<= mmdet \< 3.2.0 |
| 1.0.0          | 0.7.1 \<= mmengine \< 1.0.0 | 2.0.0rc4 \<= mmcv \< 2.1.0 | 3.0.0rc5 \<= mmdet \< 3.1.0 |

| 1.0.0rc6       | 0.6.0 \<= mmengine \< 1.0.0 | 2.0.0rc4 \<= mmcv \< 2.1.0 | 3.0.0rc5 \<= mmdet \< 3.1.0 |
| 1.0.0rc\[4-5\] | 0.1.0 \<= mmengine \< 1.0.0 | 2.0.0rc1 \<= mmcv \< 2.1.0 | 3.0.0rc0 \<= mmdet \< 3.1.0 |
| 1.0.0rc\[0-3\] | 0.0.0 \<= mmengine \< 0.2.0 | 2.0.0rc1 \<= mmcv \< 2.1.0 | 3.0.0rc0 \<= mmdet \< 3.1.0 |

# Label Studio 标注工具转COCO脚本

[Label Studio](https://labelstud.io/) 是一款广受欢迎的深度学习标注工具，可以对多种任务进行标注，然而对于关键点标注，Label Studio 无法直接导出成 MMPose 所需要的 COCO 格式。本文将介绍如何使用Label Studio 标注关键点数据，并利用 [labelstudio2coco.py](../../../tools/dataset_converters/labelstudio2coco.py) 工具将其转换为训练所需的格式。

## Label Studio 标注要求

根据 COCO 格式的要求，每个标注的实例中都需要包含关键点、分割和 bbox 的信息，然而 Label Studio 在标注时会将这些信息分散在不同的实例中，因此需要按一定规则进行标注，才能正常使用后续的脚本。

1. 标签接口设置

对于一个新建的 Label Studio 项目，首先要设置它的标签接口。这里需要有三种类型的标注：`KeyPointLabels`、`PolygonLabels`、`RectangleLabels`，分别对应 COCO 格式中的`keypoints`、`segmentation`、`bbox`。以下是一个标签接口的示例，可以在项目的`Settings`中找到`Labeling Interface`，点击`Code`，粘贴使用该示例。

```xml
<View>

<Label value="person" background="#0DA39E"/>
</View>
```

2. 标注顺序

由于需要将多个标注实例中的不同类型标注组合到一个实例中，因此采取了按特定顺序标注的方式，以此来判断各标注是否位于同一个实例。标注时须按照 **KeyPointLabels -> PolygonLabels/RectangleLabels** 的顺序标注，其中 KeyPointLabels 的顺序和数量要与 MMPose 配置文件中的`dataset_info`的关键点顺序和数量一致， PolygonLabels 和 RectangleLabels 的标注顺序可以互换，且可以只标注其中一个，只要保证一个实例的标注中，以关键点开始，以非关键点结束即可。下图为标注的示例：

*注：bbox 和 area 会根据靠后的 PolygonLabels/RectangleLabels 来计算，如若先标 PolygonLabels，那么bbox会是靠后的 RectangleLabels 的范围，面积为矩形的面积，反之则是多边形外接矩形和多边形的面积*

3. 导出标注

上述标注完成后，需要将标注进行导出。选择项目界面的`Export`按钮，选择`JSON`格式，再点击`Export`即可下载包含标签的 JSON 格式文件。

*注：上述文件中仅仅包含标签，不包含原始图片，因此需要额外提供标注对应的图片。由于 Label Studio 会对过长的文件名进行截断，因此不建议直接使用上传的文件，而是使用`Export`功能中的导出 COCO 格式工具，使用压缩包内的图片文件夹。*

## 转换工具脚本的使用

转换工具脚本位于`tools/dataset_converters/labelstudio2coco.py`，使用方式如下：

```bash
```

其中`config.xml`的内容为标签接口设置中提到的`Labeling Interface`中的`Code`，`project-1-at-2023-05-13-09-22-91b53efa.json`即为导出标注时导出的 Label Studio 格式的 JSON 文件，`output/result.json`为转换后得到的 COCO 格式的 JSON 文件路径，若路径不存在，该脚本会自动创建路径。

随后，将图片的文件夹放置在输出目录下，即可完成 COCO 数据集的转换。目录结构示例如下：

```bash
.
├── images

```

若想在 MMPose 中使用该数据集，可以进行类似如下的修改：

```python

```

```python
)
```

# Tutorial 3: Inference with existing models

MMSegmentation provides pre-trained models for semantic segmentation in [Model Zoo](../model_zoo.md), and supports multiple standard datasets, including Cityscapes, ADE20K, etc.
This note will show how to use existing models to inference on given images.
As for how to test existing models on standard datasets, please see this [guide](./4_train_test.md)

MMSegmentation provides several interfaces for users to easily use pre-trained models for inference.

- [Tutorial 3: Inference with existing models](#tutorial-3-inference-with-existing-models)
  - [Inferencer](#inferencer)
    - [Basic Usage](#basic-usage)
    - [Initialization](#initialization)
    - [Visualize prediction](#visualize-prediction)
    - [List model](#list-model)
  - [Inference API](#inference-api)

- [mmseg.apis.init_model](#mmsegapisinit_model)

## Inferencer

We provide the most **convenient** way to use the model in MMSegmentation `MMSegInferencer`. You can get segmentation mask for an image with only 3 lines of code.

### Basic Usage

The following example shows how to use `MMSegInferencer` to perform inference on a single image.

```
```

</div>

Moreover, you can use `MMSegInferencer` to process a list of images:

```

</div>

```
# Input a list of images
>>> images = [image1, image2, ...] # image1 can be a file path or a np.ndarray
>>> inferencer(images, show=True, wait_time=0.5) # wait_time is delay time, and 0 means forever

# Or input image directory

# Save visualized rendering color maps and predicted results
# out_dir is the directory to save the output results, img_out_dir and pred_out_dir are subdirectories of out_dir
# to save visualized rendering color maps and predicted results
```

There is a optional parameter of inferencer, `return_datasamples`, whose default value is False, and return value of inferencer is a `dict` type by default, including 2 keys 'visualization' and 'predictions'.

If `return_datasamples=True` inferencer will return [`SegDataSample`](../advanced_guides/structures.md), or list of it.

```
# result is a `dict` including 2 keys 'visualization' and 'predictions'
# 'visualization' includes color segmentation map

# 'predictions' includes segmentation mask with label indice

# The output is list

```

### Initialization

`MMSegInferencer` must be initialized from a `model`, which can be a model name or a `Config` even a path of config file.
The model names can be found in models' metafile (configs/xxx/metafile.yaml), like one model name of maskformer is `maskformer_r50-d32_8xb2-160k_ade20k-512x512`, and if input model name and the weights of the model will be download automatically. Below are other input parameters:

- weights (str, optional) -  Path to the checkpoint. If it is not specified and model is a model name of metafile, the weights will be loaded from metafile. Defaults to None.

- classes (list, optional) - Input classes for result rendering, as the prediction of segmentation model is a segment map with label indices, `classes` is a list which includes items responding to the label indices. If classes is not defined, visualizer will take `cityscapes` classes by default. Defaults to None.
- palette (list, optional) - Input palette for result rendering, which is a list of colors responding to the classes. If the palette is not defined, the visualizer will take the palette of `cityscapes` by default. Defaults to None.

- dataset_name (str, optional) - [Dataset name or alias](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/utils/class_names.py#L302-L317), visualizer will use the meta information of the dataset i.e. classes and palette, but the `classes` and `palette` have higher priority. Defaults to None.
- device (str, optional) - Device to run inference. If None, the available device will be automatically used. Defaults to None.
- scope (str, optional) - The scope of the model. Defaults to 'mmseg'.

### Visualize prediction

`MMSegInferencer` supports 4 parameters for visualize prediction, you can use them when call initialized inferencer:

- show (bool) - Whether to display the image in a popup window. Defaults to False.
- wait_time (float) - The interval of show (s). Defaults to 0.
- img_out_dir (str) - Subdirectory of `out_dir`, used to save rendering color segmentation mask, so `out_dir` must be defined if you would like to save predicted mask. Defaults to 'vis'.

- opacity (int, float) - The transparency of segmentation mask. Defaults to 0.8.

The examples of these parameters is in [Basic Usage](#basic-usage)

### List model

There is a very easy to list all model names in MMSegmentation

```
# models is a list of model names, and them will print automatically
```

## Inference API

Initialize a segmentor from config file.

- config (str, `Path`, or `mmengine.Config`) - Config file path or the config object.
- checkpoint (str, optional) - Checkpoint path. If left as None, the model will not load any weights.
- device (str, optional) - CPU/CUDA device option. Default 'cuda:0'.

- cfg_options (dict, optional) - Options to override some settings in the used config.

Returns:

- nn.Module: The constructed segmentor.

Example:

```python

# initialize model without checkpoint

# init model and load checkpoint

# init model and load checkpoint on CPU
```

Inference image(s) with the segmentor.

- model (nn.Module) - The loaded segmentor

```

- imgs (str, np.ndarray, or list\[str/np.ndarray\]) - Either image files or loaded images

Returns:

- `SegDataSample` or list\[`SegDataSample`\]: If imgs is a list or tuple, the same length list type results will be returned, otherwise return the segmentation results directly.

**Note:** [SegDataSample](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/structures/seg_data_sample.py) is a data structure interface of MMSegmentation, it is used as interfaces between different components. `SegDataSample` implement the abstract data element `mmengine.structures.BaseDataElement`, please refer to data element [documentation](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/data_element.html) in [MMEngine](https://github.com/open-mmlab/mmengine) for more information.

The attributes in `SegDataSample` are divided into several parts:

- `gt_sem_seg` (`PixelData`) - Ground truth of semantic segmentation.
- `pred_sem_seg` (`PixelData`) - Prediction of semantic segmentation.
- `seg_logits` (`PixelData`) - Predicted logits of semantic segmentation.

**Note** [PixelData](https://github.com/open-mmlab/mmengine/blob/main/mmengine/structures/pixel_data.py) is the data structure for pixel-level annotations or predictions, please refer to PixelData [documentation](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/data_element.html) in [MMEngine](https://github.com/open-mmlab/mmengine) for more information.

Example:

```python

```

Visualize the segmentation results on the image.

- model (nn.Module) - The loaded segmentor.
- img (str or np.ndarray) - Image filename or loaded image.
- result (`SegDataSample`) - The prediction SegDataSample result.
- opacity (float) - Opacity of painted segmentation map. Default `0.5`, must be in `(0, 1]` range.
- title (str) - The title of pyplot figure. Default is ''.
- draw_gt (bool) - Whether to draw GT SegDataSample. Default to `True`.
- draw_pred (draws_pred) - Whether to draw Prediction SegDataSample. Default to `True`.
- wait_time (float) - The interval of show (s), 0 is the special value that means "forever". Default to `0`.
- show (bool) - Whether to display the drawn image. Default to `True`.

- save_dir (str, optional) - Save file dir for all storage backends. If it is `None`, the backend storage will not save any data.
- out_file (str, optional) - Path to output file. Default to `None`.

Returns:

- np.ndarray: the drawn image which channel is RGB.

Example:

```python

# inference on given image

# display the segmentation result

# save the visualization result, the output image would be found at the path `work_dirs/result.png`

# Modify the time of displaying images, note that 0 is the special value that means "forever"
```

**Note:** If your current device doesn't have graphical user interface, it is recommended that setting `show` to `False` and specify the `out_file` or `save_dir` to save the results. If you would like to display the result on a window, no special settings are required.

# Build from Source

## Download

```shell
git clone -b main git@github.com:open-mmlab/mmdeploy.git --recursive
```

Note:

- If fetching submodule fails, you could get submodule manually by following instructions:

  ```shell
  cd mmdeploy
  git clone git@github.com:NVIDIA/cub.git third_party/cub
  cd third_party/cub
  git checkout c3cceac115

  # go back to third_party directory and git clone pybind11
  cd ..
  git clone git@github.com:pybind/pybind11.git pybind11
  cd pybind11
  git checkout 70a58c5

  cd ..
  git clone git@github.com:gabime/spdlog.git spdlog
  cd spdlog
  git checkout 9e8e52c048
  ```

- If it fails when `git clone` via `SSH`, you can try the `HTTPS` protocol like this:

  ```shell
  git clone -b main https://github.com/open-mmlab/mmdeploy.git --recursive
  ```

## Build

Please visit the following links to find out how to build MMDeploy according to the target platform.

- [Linux-x86_64](linux-x86_64.md)
- [Windows](windows.md)
- [MacOS](macos-arm64.md)

## Build

- [Android-aarch64](android.md)
- [NVIDIA Jetson](jetsons.md)
- [SNPE](snpe.md)
- [RISC-V](riscv.md)
- [Rockchip](rockchip.md)

# 如何支持新的后端

MMDeploy 支持了许多后端推理引擎，但我们依然非常欢迎新后端的贡献。在本教程中，我们将介绍在 MMDeploy 中支持新后端的一般过程。

## 必要条件

在对 MMDeploy 添加新的后端引擎之前，需要先检查所要支持的新后端是否符合一些要求:

- 后端必须能够支持 ONNX 作为 IR。
- 如果后端需要“.onnx”文件以外的模型文件或权重文件，则需要添加将“.onnx”文件转换为模型文件或权重文件的转换工具，该工具可以是 Python API、脚本或可执行程序。
- 强烈建议新后端可提供 Python 接口来加载后端文件和推理以进行验证。

## 支持后端转换

MMDeploy 中的后端必须支持 ONNX，因此后端能直接加载“.onnx”文件，或者使用转换工具将“.onnx”转换成自己的格式。在本节中，我们将介绍支持后端转换的步骤。

1. 在 `mmdeploy/utils/constants.py` 文件中添加新推理后端变量，以表示支持的后端名称。

   **示例**：

   ```Python
   # mmdeploy/utils/constants.py

   class Backend(AdvancedEnum):
       # 以现有的TensorRT为例
       TENSORRT = 'tensorrt'
   ```

2. 在 `mmdeploy/backend/` 目录下添加相应的库(一个包括 `__init__.py` 的文件夹),例如， `mmdeploy/backend/tensorrt` 。在 `__init__.py` 中，必须有一个名为 `is_available` 的函数检查用户是否安装了后端库。如果检查通过，则将加载库的剩余文件。

   **例子**:

   ```Python
   # mmdeploy/backend/tensorrt/__init__.py

   def is_available():
       return importlib.util.find_spec('tensorrt') is not None

   if is_available():

**例子**:

   ```Python

       from .utils import from_onnx, load, save
       from .wrapper import TRTWrapper

       __all__ = [
           'from_onnx', 'save', 'load', 'TRTWrapper'
       ]
   ```

3. 在 `configs/_base_/backends` 目录中创建一个配置文件(例如， `configs/_base_/backends/tensorrt.py` )。如果新后端引擎只是将“.onnx”文件作为输入，那么新的配置可以很简单,对应配置只需包含一个表示后端名称的字段(但也应该与 `mmdeploy/utils/constants.py` 中的名称相同)。

   **例子**

   ```python
   backend_config = dict(type='tensorrt')
   ```

   但如果后端需要其他文件，则从“.onnx”文件转换为后端文件所需的参数也应包含在配置文件中。

   **例子**

   ```Python

   backend_config = dict(
       type='tensorrt',
       common_config=dict(
           fp16_mode=False, max_workspace_size=0))
   ```

   在拥有一个基本的后端配置文件后，您已经可以通过继承轻松构建一个完整的部署配置。有关详细信息，请参阅我们的[配置教程](../02-how-to-run/write_config.md)。下面是一个例子：

   ```Python
   _base_ = ['../_base_/backends/tensorrt.py']

```

   ```Python

   codebase_config = dict(type='mmpretrain', task='Classification')
   onnx_config = dict(input_shape=None)
   ```

4. 如果新后端需要模型文件或权重文件而不是“.onnx”文件，则需要在相应的文件夹中创建一个 `onnx2backend.py` 文件(例如,创建 `mmdeploy/backend/tensorrt/onnx2tensorrt.py` )。然后在文件中添加一个转换函数`onnx2backend`。该函数应将给定的“.onnx”文件转换为给定工作目录中所需的后端文件。对函数的其他参数和实现细节没有要求，您可以使用任何工具进行转换。下面有些例子：

   **使用python脚本**

   ```Python
   def onnx2openvino(input_info: Dict[str, Union[List[int], torch.Size]],
                     output_names: List[str], onnx_path: str, work_dir: str):

       input_names = ','.join(input_info.keys())
       input_shapes = ','.join(str(list(elem)) for elem in input_info.values())
       output = ','.join(output_names)

       mo_args = f'--input_model="{onnx_path}" '\
                 f'--output_dir="{work_dir}" ' \

output = ','.join(output_names)

                 f'--output="{output}" ' \
                 f'--input="{input_names}" ' \
                 f'--input_shape="{input_shapes}" ' \
                 f'--disable_fusing '
       command = f'mo.py {mo_args}'
       mo_output = run(command, stdout=PIPE, stderr=PIPE, shell=True, check=True)
   ```

   **使用可执行文件**

   ```Python
   def onnx2ncnn(onnx_path: str, work_dir: str):
       onnx2ncnn_path = get_onnx2ncnn_path()
       save_param, save_bin = get_output_model_file(onnx_path, work_dir)
       call([onnx2ncnn_path, onnx_path, save_param, save_bin])\
   ```

5. 在 `mmdeploy/apis` 中创建新后端库并声明对应 APIs

   **例子**

   ```Python
   # mmdeploy/apis/ncnn/__init__.py

   from mmdeploy.backend.ncnn import is_available

   __all__ = ['is_available']

       from mmdeploy.backend.ncnn.onnx2ncnn import (onnx2ncnn,

from mmdeploy.backend.ncnn import is_available

                                                    get_output_model_file)
       __all__ += ['onnx2ncnn', 'get_output_model_file']
   ```

   从 BaseBackendManager 派生类，实现 `to_backend` 类方法。

   **例子**

   ```Python
   @classmethod
    def to_backend(cls,
                   ir_files: Sequence[str],
                   deploy_cfg: Any,
                   work_dir: str,
                   log_level: int = logging.INFO,
                   device: str = 'cpu',
                   **kwargs) -> Sequence[str]:
        return ir_files
   ```

6. 将 OpenMMLab 的模型转换后(如有必要)并在后端引擎上进行推理。如果在测试时发现一些不兼容的算子，可以尝试按照[重写器教程](support_new_model.md)为后端重写原始模型或添加自定义算子。

7. 为新后端引擎代码添加相关注释和单元测试:).

## 支持后端推理

尽管后端引擎通常用C/C++实现，但如果后端提供Python推理接口，则测试和调试非常方便。我们鼓励贡献者在MMDeploy的Python接口中支持新后端推理。在本节中，我们将介绍支持后端推理的步骤。

## 支持后端推理

1. 添加一个名为 `wrapper.py` 的文件到 `mmdeploy/backend/{backend}` 中相应后端文件夹。例如， `mmdeploy/backend/tensorrt/wrapper` 。此模块应实现并注册一个封装类，该类继承 `mmdeploy/backend/base/base_wrapper.py` 中的基类 `BaseWrapper` 。

   **例子**

   ```Python
   from mmdeploy.utils import Backend
   from ..base import BACKEND_WRAPPER, BaseWrapper

   @BACKEND_WRAPPER.register_module(Backend.TENSORRT.value)
   class TRTWrapper(BaseWrapper):
   ```

2. 封装类可以在函数 `__init__` 中初始化引擎以及在 `forward` 函数中进行推理。请注意，该 `__init__` 函数必须接受一个参数 `output_names` 并将其传递给基类以确定输出张量的顺序。其中 `forward` 输入和输出变量应表示tensors的名称和值的字典。

3. 为了方便性能测试，该类应该定义一个 `execute` 函数，只调用后端引擎的推理接口。该 `forward` 函数应在预处理数据后调用 `execute` 函数。

   **例子**

   ```Python
   from mmdeploy.utils.timer import TimeCounter

**例子**

   ```Python

   @BACKEND_WRAPPER.register_module(Backend.ONNXRUNTIME.value)
   class ORTWrapper(BaseWrapper):

       def __init__(self,
                    onnx_file: str,
                    device: str,
                    output_names: Optional[Sequence[str]] = None):
           # Initialization
           #
           # ...
           super().__init__(output_names)

       def forward(self, inputs: Dict[str,
                                      torch.Tensor]) -> Dict[str, torch.Tensor]:
           # Fetch data
           # ...

           self.__ort_execute(self.io_binding)

   		# Postprocess data
           # ...

       @TimeCounter.count_time('onnxruntime')
       def __ort_execute(self, io_binding: ort.IOBinding):
   		# Only do the inference
           self.sess.run_with_iobinding(io_binding)
   ```

@TimeCounter.count_time('onnxruntime')
   ```

4. 从 `BaseBackendManager` 派生接口类，实现 `build_wrapper` 静态方法

   **例子**

   ```Python
        @BACKEND_MANAGERS.register('onnxruntime')
        class ONNXRuntimeManager(BaseBackendManager):
            @classmethod
            def build_wrapper(cls,
                              backend_files: Sequence[str],
                              device: str = 'cpu',
                              input_names: Optional[Sequence[str]] = None,
                              output_names: Optional[Sequence[str]] = None,
                              deploy_cfg: Optional[Any] = None,
                from .wrapper import ORTWrapper
                return ORTWrapper(
                    onnx_file=backend_files[0],
                    device=device,

from .wrapper import ORTWrapper
                    output_names=output_names)
   ```

5. 为新后端引擎代码添加相关注释和单元测试 :).

## 将MMDeploy作为第三方库时添加新后端

前面的部分展示了如何在 MMDeploy 中添加新的后端，这需要更改其源代码。但是，如果我们将 MMDeploy 视为第三方，则上述方法不再有效。为此，添加一个新的后端需要我们预先安装另一个名为 `aenum` 的包。我们可以直接通过`pip install aenum`进行安装。

成功安装 `aenum` 后，我们可以通过以下方式使用它来添加新的后端：

```python
from mmdeploy.utils.constants import Backend
from aenum import extend_enum

try:
    Backend.get('backend_name')
except Exception:
    extend_enum(Backend, 'BACKEND', 'backend_name')
```

我们可以在使用 MMDeploy 的重写逻辑之前运行上面的代码，这就完成了新后端的添加。

# How to Pretrain with Custom Dataset

In this tutorial, we provide a practice example and some tips on how to train on your own dataset.

In MMPretrain, We support the `CustomDataset` (similar to the `ImageFolder` in `torchvision`),  which is able to read the images within the specified folder directly. You only need to prepare the path information of the custom dataset and edit the config.

Here, we assume you want to do unsupervised training, and use the sub-folder format `CustomDataset` to
organize your dataset as:

```text
└── ...
```

Here, we would like to use `configs/mae/mae_vit-base-p16_8xb512-amp-coslr-300e_in1k.py` as the example. We

└── ...
```

first copy this config file to the same folder and rename it as
`mae_vit-base-p16_8xb512-amp-coslr-300e_custom.py`.

```{tip}
```

```python
_base_ = [
]

```

## Step-3: Edit the dataset related config

- Override the `type` of dataset settings as `'CustomDataset'`
- Override the `data_root` of dataset settings as `data/custom_dataset`.
- Override the `ann_file` of dataset settings as an empty string since we assume you are using the sub-folder
  format `CustomDataset`.
- Override the `data_prefix` of dataset settings as an empty string since we are using the whole dataset under
  the `data_root`, and you don't need to split samples into different subset and set the `data_prefix`.

The modified config will be like:

```python
_base_ = [

```python
_base_ = [
]

# >>>>>>>>>>>>>>> Override dataset settings here >>>>>>>>>>>>>>>>>>>
        data_prefix='',    # The `data_root` is the data_prefix directly.
    )
)

```

By using the edited config file, you are able to train a self-supervised model with MAE algorithm on the custom dataset.

## Another example: Train MAE on COCO Dataset

```{note}
You need to install MMDetection to use the `mmdet.CocoDataset` follow this [documentation](https://github.com/open-mmlab/mmdetection/blob/3.x/docs/en/get_started.md)
```

Follow the aforementioned idea, we also present an example of how to train MAE on COCO dataset.  The edited file will be like this:

```python
_base_ = [
]

```python
_base_ = [
]

        ann_file='annotations/instances_train2017.json',  # Only for loading images, and the labels won't be used.
    )
)

        }))

    dict(
        begin=0,
        end=40,
    dict(
        begin=40,
        end=300,
]

```

# 更多工具介绍

除 `deploy.py` 以外， tools 目录下有很多实用工具

## torch2onnx

把 OpenMMLab 模型转 onnx 格式。

### 用法

```bash
    ${DEPLOY_CFG} \
    ${MODEL_CFG} \
```

### 参数说明

- `deploy_cfg` : The path of the deploy config file in MMDeploy codebase.
- `model_cfg` : The path of model config file in OpenMMLab codebase.
- `checkpoint` : The path of the model checkpoint file.
- `img` : The path of the image file used to convert the model.
- `--work-dir` : Directory to save output ONNX models Default is `./work-dir`.
- `--device` : The device used for conversion. If not specified, it will be set to `cpu`.
- `--log-level` : To set log level which in `'CRITICAL', 'FATAL', 'ERROR', 'WARN', 'WARNING', 'INFO', 'DEBUG', 'NOTSET'`. If not specified, it will be set to `INFO`.

## extract

有 `Mark` 节点的 onnx 模型会被分成多个子图，这个工具用来提取 onnx 模型中的子图。

### 用法

```bash
python tools/extract.py \

## extract

### 用法

```bash
    ${INPUT_MODEL} \
    ${OUTPUT_MODEL} \
    --start ${PARITION_START} \
    --end ${PARITION_END} \
```

### 参数说明

- `input_model` : The path of input ONNX model. The output ONNX model will be extracted from this model.
- `output_model` : The path of output ONNX model.
- `--start` : The start point of extracted model with format `<function_name>:<input/output>`. The `function_name` comes from the decorator `@mark`.
- `--end` : The end point of extracted model with format `<function_name>:<input/output>`. The `function_name` comes from the decorator `@mark`.

### 注意事项

要支持模型分块，必须在 onnx 模型中添加  mark 节点，用`@mark` 修饰。
下面这个例子里 mark 了 `multiclass_nms`，在 NMS 前设置 `end=multiclass_nms:input` 提取子图。

### 注意事项

```python
@mark('multiclass_nms', inputs=['boxes', 'scores'], outputs=['dets', 'labels'])
def multiclass_nms(*args, **kwargs):
    """Wrapper function for `_multiclass_nms`."""
```

## onnx2pplnn

这个工具可以把 onnx 模型转成 pplnn 格式。

### 用法

```bash
python tools/onnx2pplnn.py \
    ${ONNX_PATH} \
    ${OUTPUT_PATH} \
    --opt-shapes [224,224] \
```

### 参数说明

- `onnx_path`: The path of the `ONNX` model to convert.
- `output_path`: The converted `PPLNN` algorithm path in json format.
- `device`: The device of the model during conversion.
- `opt-shapes`: Optimal shapes for PPLNN optimization. The shape of each tensor should be wrap with "[]" or "()" and the shapes of tensors should be separated by ",".

- `--log-level`: To set log level which in `'CRITICAL', 'FATAL', 'ERROR', 'WARN', 'WARNING', 'INFO', 'DEBUG', 'NOTSET'`. If not specified, it will be set to `INFO`.

## onnx2tensorrt

这个工具把 onnx 转成 trt .engine 格式。

### 用法

```bash
python tools/onnx2tensorrt.py \
    ${OUTPUT} \
    --device-id 0 \
    --calib-file  /path/to/file
```

### 参数说明

- `onnx_path` : The ONNX model path to convert.
- `output` : The path of output TensorRT engine.
- `--device-id` : The device index, default to `0`.
- `--calib-file` : The calibration data used to calibrate engine to int8.

## onnx2ncnn

onnx 转 ncnn

### 用法

```bash
python tools/onnx2ncnn.py \
    ${NCNN_PARAM} \
    ${NCNN_BIN} \
```

### 参数说明

- `onnx_path` : The path of the `ONNX` model to convert from.
- `output_param` : The converted `ncnn` param path.
- `output_bin` : The converted `ncnn` bin path.

## profiler

这个工具用来测试 torch 和 trt 等后端的速度，注意测试不包含前后处理。

### 用法

```bash
    ${IMAGE_DIR} \

### 用法

```bash
    --model ${MODEL} \
    --shape ${SHAPE} \
    --num-iter ${NUM_ITER} \
    --warmup ${WARMUP} \
    --cfg-options ${CFG_OPTIONS} \
    --batch-size ${BATCH_SIZE} \
    --img-ext ${IMG_EXT}
```

### 参数说明

- `image_dir` : The directory to image files that used to test the model.
- `--model` : The path of the model to be tested.
- `--shape` : Input shape of the model by `HxW`, e.g., `800x1344`. If not specified, it would use `input_shape` from deploy config.
- `--num-iter` : Number of iteration to run inference. Default is `100`.
- `--warmup` : Number of iteration to warm-up the machine. Default is `10`.

- `--device` : The device type. If not specified, it will be set to `cuda:0`.
- `--cfg-options` : Optional key-value pairs to be overrode for model config.
- `--batch-size`: the batch size for test inference. Default is `1`. Note that not all models support `batch_size>1`.
- `--img-ext`: the file extensions for input images from `image_dir`. Defaults to `['.jpg', '.jpeg', '.png', '.ppm', '.bmp', '.pgm', '.tif']`.

### 使用举例

```shell
    configs/mmpretrain/classification_tensorrt_dynamic-224x224-224x224.py \
    ../mmpretrain/configs/resnet/resnet18_8xb32_in1k.py \
    ../mmpretrain/demo/ \
    --model work-dirs/mmpretrain/resnet/trt/end2end.engine \
    --shape 224x224 \
    --num-iter 100 \
    --warmup 10 \
    --batch-size 1
```

输出：

```text

```

输出：

```text
|   shape    | 224x224 |
| iterations |   100   |
|   warmup   |    10   |
|  Mean  |   1.535    | 651.656 |
| Median |   1.665    | 600.569 |
|  Min   |   1.308    | 764.341 |
|  Max   |   1.689    | 591.983 |
```

## generate_md_table

生成mmdeploy支持的后端表。

### 用法

```shell
python tools/generate_md_table.py \
    ${YML_FILE} \
    --backends ${BACKENDS}
```

### 参数说明

- `yml_file:`  输入 yml 配置路径
- `output:` 输出markdown文件路径
- `--backends:` 要输出的后端，默认为 onnxruntime tensorrt torchscript pplnn openvino ncnn

### 使用举例

从 mmocr.yml 生成mmdeploy支持的后端表

```shell

```

### 参数说明

### 使用举例

```shell
python tools/generate_md_table.py tests/regression/mmocr.yml tests/regression/mmocr.md --backends onnxruntime tensorrt torchscript pplnn openvino ncnn
```

输出：

| model                                                                                | task            | onnxruntime | tensorrt | torchscript | pplnn | openvino | ncnn |
| :----------------------------------------------------------------------------------- | :-------------- | :---------: | :------: | :---------: | :---: | :------: | :--: |
| [DBNet](https://github.com/open-mmlab/mmocr/tree/main/configs/textdet/dbnet)         | TextDetection   |      Y      |    Y     |      Y      |   Y   |    Y     |  Y   |

| [DBNetpp](https://github.com/open-mmlab/mmocr/tree/main/configs/textdet/dbnetpp)     | TextDetection   |      Y      |    Y     |      N      |   N   |    Y     |  Y   |
| [PANet](https://github.com/open-mmlab/mmocr/tree/main/configs/textdet/panet)         | TextDetection   |      Y      |    Y     |      Y      |   Y   |    Y     |  Y   |
| [PSENet](https://github.com/open-mmlab/mmocr/tree/main/configs/textdet/psenet)       | TextDetection   |      Y      |    Y     |      Y      |   Y   |    Y     |  Y   |
| [TextSnake](https://github.com/open-mmlab/mmocr/tree/main/configs/textdet/textsnake) | TextDetection   |      Y      |    Y     |      Y      |   N   |    N     |  N   |

| [MaskRCNN](https://github.com/open-mmlab/mmocr/tree/main/configs/textdet/maskrcnn)   | TextDetection   |      Y      |    Y     |      Y      |   N   |    N     |  N   |
| [CRNN](https://github.com/open-mmlab/mmocr/tree/main/configs/textrecog/crnn)         | TextRecognition |      Y      |    Y     |      Y      |   Y   |    N     |  Y   |
| [SAR](https://github.com/open-mmlab/mmocr/tree/main/configs/textrecog/sar)           | TextRecognition |      Y      |    N     |      Y      |   N   |    N     |  N   |
| [SATRN](https://github.com/open-mmlab/mmocr/tree/main/configs/textrecog/satrn)       | TextRecognition |      Y      |    Y     |      Y      |   N   |    N     |  N   |

| [ABINet](https://github.com/open-mmlab/mmocr/tree/main/configs/textrecog/abinet)     | TextRecognition |      Y      |    Y     |      Y      |   N   |    N     |  N   |

Does the modification introduce changes that break the back-compatibility of the downstream repos?

3. If the modification has potential influence on downstream projects, this PR should be tested with downstream projects.

# 2D Fashion Landmark Dataset

- [DeepFashion](#deepfashion) \[ [Homepage](http://mmlab.ie.cuhk.edu.hk/projects/DeepFashion/LandmarkDetection.html) \]
- [DeepFashion2](#deepfashion2) \[ [Homepage](https://github.com/switchablenorms/DeepFashion2) \]

## DeepFashion (Fashion Landmark Detection, FLD)

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_cvpr_2016/html/Liu_DeepFashion_Powering_Robust_CVPR_2016_paper.html">DeepFashion (CVPR'2016)</a></summary>

```bibtex
@inproceedings{liuLQWTcvpr16DeepFashion,
 author = {Liu, Ziwei and Luo, Ping and Qiu, Shi and Wang, Xiaogang and Tang, Xiaoou},
 title = {DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations},

author = {Liu, Ziwei and Luo, Ping and Qiu, Shi and Wang, Xiaogang and Tang, Xiaoou},
 booktitle = {Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
 month = {June},
 year = {2016}
}
```

<details>
<summary align="right"><a href="https://link.springer.com/chapter/10.1007/978-3-319-46475-6_15">DeepFashion (ECCV'2016)</a></summary>

```bibtex
@inproceedings{liuYLWTeccv16FashionLandmark,
 author = {Liu, Ziwei and Yan, Sijie and Luo, Ping and Wang, Xiaogang and Tang, Xiaoou},
 title = {Fashion Landmark Detection in the Wild},
 booktitle = {European Conference on Computer Vision (ECCV)},
 month = {October},
 }
```

  <img src="https://user-images.githubusercontent.com/100993824/227774588-443fc5cc-7842-472a-abd5-827f0e3fd27f.png" height="150px">
</div>

year = {2016}
 }
```

</div>

For [DeepFashion](http://mmlab.ie.cuhk.edu.hk/projects/DeepFashion/LandmarkDetection.html) dataset, images can be downloaded from [download](http://mmlab.ie.cuhk.edu.hk/projects/DeepFashion/LandmarkDetection.html).
Please download the annotation files from [fld_annotations](https://download.openmmlab.com/mmpose/datasets/fld_annotations.tar).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── fld
        │   │-- fld_upper_train.json
        │   |-- fld_upper_val.json
        │   |-- fld_upper_test.json
        │   │-- fld_lower_train.json
        │   |-- fld_lower_val.json
        │   |-- fld_lower_test.json
        │   │-- fld_full_train.json

│   |-- fld_upper_test.json
        │   |-- fld_full_val.json
        │   |-- fld_full_test.json
        │-- img
        │   │-- img_00000001.jpg
        │   │-- img_00000002.jpg
        │   │-- img_00000003.jpg
        │   │-- img_00000004.jpg
        │   │-- img_00000005.jpg
        │   │-- ...
```

## DeepFashion2

<details>
<summary align="right"><a href="https://arxiv.org/pdf/1901.07973.pdf">DeepFashion2 (CVPR'2019)</a></summary>

```bibtex
@article{DeepFashion2,
  author = {Yuying Ge and Ruimao Zhang and Lingyun Wu and Xiaogang Wang and Xiaoou Tang and Ping Luo},
  title={A Versatile Benchmark for Detection, Pose Estimation, Segmentation and Re-Identification of Clothing Images},
}
```

title={A Versatile Benchmark for Detection, Pose Estimation, Segmentation and Re-Identification of Clothing Images},
}
```

For [DeepFashion2](https://github.com/switchablenorms/DeepFashion2) dataset, images can be downloaded from [download](https://drive.google.com/drive/folders/125F48fsMBz2EF0Cpqk6aaHet5VH399Ok?usp=sharing).
Please download the [annotation files](https://drive.google.com/file/d/1RM9l9EaB9ULRXhoCS72PkCXtJ4Cn4i6O/view?usp=share_link). These annotation files are converted by [deepfashion2_to_coco.py](https://github.com/switchablenorms/DeepFashion2/blob/master/evaluation/deepfashion2_to_coco.py).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── deepfashion2
        │── train
            │-- deepfashion2_short_sleeved_outwear_train.json
            │-- deepfashion2_short_sleeved_dress_train.json

├── tools
`── data
        │── train
            │-- deepfashion2_skirt_train.json
            │-- deepfashion2_sling_dress_train.json
            │-- ...
            │-- image
            │   │-- 000001.jpg
            │   │-- 000002.jpg
            │   │-- 000003.jpg
            │   │-- 000004.jpg
            │   │-- 000005.jpg
            │   │-- ...
        │── validation
            │-- deepfashion2_short_sleeved_dress_validation.json
            │-- deepfashion2_long_sleeved_shirt_validation.json
            │-- deepfashion2_trousers_validation.json
            │-- deepfashion2_skirt_validation.json
            │-- ...
            │-- image
            │   │-- ...
```

# 贡献代码

欢迎加入 MMagic 社区，我们致力于打造新一代人工智能内容生成（AIGC）工具箱，我们欢迎任何类型的贡献，包括但不限于

**修复错误**

修复代码实现错误的步骤如下：

1. 如果提交的代码改动较大，建议先提交 issue，并正确描述 issue 的现象、原因和复现方式，讨论后确认修复方案。
2. 修复错误并补充相应的单元测试，提交拉取请求。

**新增功能或组件**

1. 如果新功能或模块涉及较大的代码改动，建议先提交 issue，确认功能的必要性。
2. 实现新增功能并添单元测试，提交拉取请求。

**文档补充**

修复文档可以直接提交拉取请求

添加文档或将文档翻译成其他语言步骤如下

1. 提交 issue，确认添加文档的必要性。
2. 添加文档，提交拉取请求。

### 拉取请求工作流

如果你对拉取请求不了解，没关系，接下来的内容将会从零开始，一步一步地指引你如何创建一个拉取请求。如果你想深入了解拉取请求的开发模式，可以参考 github [官方文档](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests)

#### 1. 复刻仓库

当你第一次提交拉取请求时，先复刻 OpenMMLab 原代码库，点击 GitHub 页面右上角的 **Fork** 按钮，复刻后的代码库将会出现在你的 GitHub 个人主页下。

将代码克隆到本地

```shell
git clone git@github.com:{username}/mmagic.git
```

添加原代码库为上游代码库

```bash
git remote add upstream git@github.com:open-mmlab/mmagic
```

检查 remote 是否添加成功，在终端输入 `git remote -v`

```bash

```shell
```

```bash
```

```bash
origin	git@github.com:{username}/mmagic.git (fetch)
origin	git@github.com:{username}/mmagic.git (push)
upstream	git@github.com:open-mmlab/mmagic (fetch)
upstream	git@github.com:open-mmlab/mmagic (push)
```

```{note}
这里对 origin 和 upstream 进行一个简单的介绍，当我们使用 git clone 来克隆代码时，会默认创建一个 origin 的 remote，它指向我们克隆的代码库地址，而 upstream 则是我们自己添加的，用来指向原始代码库地址。当然如果你不喜欢他叫 upstream，也可以自己修改，比如叫 open-mmlab。我们通常向 origin 提交代码（即 fork 下来的远程仓库），然后向 upstream 提交一个 pull request。如果提交的代码和最新的代码发生冲突，再从 upstream 拉取最新的代码，和本地分支解决冲突，再提交到 origin。
```

#### 2. 配置 pre-commit

在本地开发环境中，我们使用 [pre-commit](https://pre-commit.com/#intro) 来检查代码风格，以确保代码风格的统一。在提交代码，需要先安装 pre-commit（需要在 mmagic 目录下执行）:

```shell
```

检查 pre-commit 是否配置成功，并安装 `.pre-commit-config.yaml` 中的钩子：

```shell

```shell
```

```shell
```

```{note}
如果你是中国用户，由于网络原因，可能会出现安装失败的情况，这时可以使用国内源

pre-commit install -c .pre-commit-config-zh-cn.yaml

pre-commit run --all-files -c .pre-commit-config-zh-cn.yaml
```

如果安装过程被中断，可以重复执行 `pre-commit run ...` 继续安装。

如果提交的代码不符合代码风格规范，pre-commit 会发出警告，并自动修复部分错误。

如果我们想临时绕开 pre-commit 的检查提交一次代码，可以在 `git commit` 时加上 `--no-verify`（需要保证最后推送至远程仓库的代码能够通过 pre-commit 检查）。

```shell
```

#### 3. 创建开发分支

```shell
```

安装完 pre-commit 之后，我们需要基于 main 创建开发分支，建议的分支命名规则为 `username/pr_name`。

```shell
```

在后续的开发中，如果本地仓库的 main 分支落后于 upstream 的 main 分支，我们需要先拉取 upstream 的代码进行同步，再执行上面的命令

```shell
git pull upstream main
```

#### 4. 提交代码并在本地通过单元测试

- mmagic 引入了 mypy 来做静态类型检查，以增加代码的鲁棒性。因此我们在提交代码时，需要补充 Type Hints。具体规则可以参考[教程](https://zhuanlan.zhihu.com/p/519335398)。

- 提交的代码同样需要通过单元测试

  ```shell
  # 通过全量单元测试

  # 我们需要保证提交的代码能够通过修改模块的单元测试，以 runner 为例
  ```

  如果你由于缺少依赖无法运行修改模块的单元测试，可以参考[指引-单元测试](#单元测试)

- 如果修改/添加了文档，参考[指引](#文档渲染)确认文档渲染正常。

#### 5. 推送代码到远程

代码通过单元测试和 pre-commit 检查后，将代码推送到远程仓库，如果是第一次推送，可以在 `git push` 后加上 `-u` 参数以关联远程分支

```shell
```

这样下次就可以直接使用 `git push` 命令推送代码了，而无需指定分支和远程仓库。

#### 6. 提交拉取请求（PR）

```shell
```

(1) 在 GitHub 的 Pull request 界面创建拉取请求

(2) 根据指引修改 PR 描述，以便于其他开发者更好地理解你的修改

描述规范详见[拉取请求规范](#拉取请求规范)

&#160;

**注意事项**

(a) PR 描述应该包含修改理由、修改内容以及修改后带来的影响，并关联相关 Issue（具体方式见[文档](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue)）

(b) 如果是第一次为 OpenMMLab 做贡献，需要签署 CLA

(c) 检查提交的 PR 是否通过 CI（集成测试）

mmagic 会在不同的平台（Linux、Window、Mac），基于不同版本的 Python、PyTorch、CUDA 对提交的代码进行单元测试，以保证代码的正确性，如果有任何一个没有通过，我们可点击上图中的 `Details` 来查看具体的测试信息，以便于我们修改代码。

(3) 如果 PR 通过了 CI，那么就可以等待其他开发者的 review，并根据 reviewer 的意见，修改代码，并重复 [4](#4-提交代码并本地通过单元测试)-[5](#5-推送代码到远程) 步骤，直到 reviewer 同意合入 PR。

所有 reviewer 同意合入 PR 后，我们会尽快将 PR 合并到主分支。

#### 7. 解决冲突

随着时间的推移，我们的代码库会不断更新，这时候，如果你的 PR 与主分支存在冲突，你需要解决冲突，解决冲突的方式有两种：

```shell
git rebase upstream/main
```

或者

```shell
git merge upstream/main
```

```shell
```

或者

```shell
```

如果你非常善于处理冲突，那么可以使用 rebase 的方式来解决冲突，因为这能够保证你的 commit log 的整洁。如果你不太熟悉 `rebase` 的使用，那么可以使用 `merge` 的方式来解决冲突。

### 指引

#### 单元测试

在提交修复代码错误或新增特性的拉取请求时，我们应该尽可能的让单元测试覆盖所有提交的代码，计算单元测试覆盖率的方法如下

```shell
python -m coverage run -m pytest /path/to/test_file
python -m coverage html
# check file in htmlcov/index.html
```

#### 文档渲染

在提交修复代码错误或新增特性的拉取请求时，可能会需要修改/新增模块的 docstring。我们需要确认渲染后的文档样式是正确的。
本地生成渲染后的文档的方法如下

```shell
make html
```

### 代码风格

[PEP8](https://www.python.org/dev/peps/pep-0008/) 作为 OpenMMLab 算法库首选的代码规范，我们使用以下工具检查和格式化代码

- [flake8](https://github.com/PyCQA/flake8): Python 官方发布的代码规范检查工具，是多个检查工具的封装
- [isort](https://github.com/timothycrosley/isort): 自动调整模块导入顺序的工具

- [yapf](https://github.com/google/yapf): Google 发布的代码规范检查工具
- [codespell](https://github.com/codespell-project/codespell): 检查单词拼写是否有误
- [mdformat](https://github.com/executablebooks/mdformat): 检查 markdown 文件的工具
- [docformatter](https://github.com/myint/docformatter): 格式化 docstring 的工具

yapf 和 isort 的配置可以在 [setup.cfg](../../../setup.cfg) 找到

通过配置 [pre-commit hook](https://pre-commit.com/) ，我们可以在提交代码时自动检查和格式化 `flake8`、`yapf`、`isort`、`trailing whitespaces`、`markdown files`，修复 `end-of-files`、`double-quoted-strings`、`python-encoding-pragma`、`mixed-line-ending`，调整 `requirments.txt` 的包顺序。
pre-commit 钩子的配置可以在 [.pre-commit-config](../../../.pre-commit-config.yaml) 找到。

pre-commit 具体的安装使用方式见[拉取请求](#2-配置-pre-commit)。

更具体的规范请参考 [OpenMMLab 代码规范](contributing.md#代码风格)。

C++ 和 CUDA 的代码规范遵从 [Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html)

### 拉取请求规范

1. 使用 [pre-commit hook](https://pre-commit.com)，尽量减少代码风格相关问题

2. 一个`拉取请求`对应一个短期分支

3. 粒度要细，一个`拉取请求`只做一件事情，避免超大的`拉取请求`

   - Bad：实现 Faster R-CNN
   - Acceptable：给 Faster R-CNN 添加一个 box head
   - Good：给 box head 增加一个参数来支持自定义的 conv 层数

4. 每次 Commit 时需要提供清晰且有意义 commit 信息

5. 提供清晰且有意义的`拉取请求`描述

   - 标题写明白任务名称，一般格式:\[Prefix\] Short description of the pull request (Suffix)
   - prefix: 新增功能 \[Feature\], 修 bug \[Fix\], 文档相关 \[Docs\], 开发中 \[WIP\] (暂时不会被review)
   - 描述里介绍`拉取请求`的主要修改内容，结果，以及对其他部分的影响, 参考`拉取请求`模板
   - 关联相关的`议题` (issue) 和其他`拉取请求`

6. 如果引入了其他三方库，或借鉴了三方库的代码，请确认他们的许可证和 mmagic 兼容，并在借鉴的代码上补充 `This code is inspired from http://`

## 代码规范

### 代码规范标准

#### PEP 8 —— Python 官方代码规范

- 关联相关的`议题` (issue) 和其他`拉取请求`

## 代码规范

### 代码规范标准

[Python 官方的代码风格指南](https://www.python.org/dev/peps/pep-0008/)，包含了以下几个方面的内容：

- 代码布局，介绍了 Python 中空行、断行以及导入相关的代码风格规范。比如一个常见的问题：当我的代码较长，无法在一行写下时，何处可以断行？

- 表达式，介绍了 Python 中表达式空格相关的一些风格规范。

- 尾随逗号相关的规范。当列表较长，无法一行写下而写成如下逐行列表时，推荐在末项后加逗号，从而便于追加选项、版本控制等。

  ```python
  # Correct:
  FILES = ['setup.cfg', 'tox.ini']
  # Correct:
  FILES = [
      'setup.cfg',
      'tox.ini',
  ]
  # Wrong:
  FILES = ['setup.cfg', 'tox.ini',]
  # Wrong:
  FILES = [
      'tox.ini'
  ]
  ```

- 命名相关规范、注释相关规范、类型注解相关规范，我们将在后续章节中做详细介绍。

  "A style guide is about consistency. Consistency with this style guide is important. Consistency within a project is more important. Consistency within one module or function is the most important." PEP 8 -- Style Guide for Python Code

:::{note}

:::{note}
PEP 8 的代码规范并不是绝对的，项目内的一致性要优先于 PEP 8 的规范。OpenMMLab 各个项目都在 setup.cfg 设定了一些代码规范的设置，请遵照这些设置。一个例子是在 PEP 8 中有如下一个例子：

```python
# Correct:
hypot2 = x*x + y*y
# Wrong:
hypot2 = x * x + y * y
```

这一规范是为了指示不同优先级，但 OpenMMLab 的设置中通常没有启用 yapf 的 `ARITHMETIC_PRECEDENCE_INDICATION` 选项，因而格式规范工具不会按照推荐样式格式化，以设置为准。
:::

#### Google 开源项目风格指南

[Google 使用的编程风格指南](https://google.github.io/styleguide/pyguide.html)，包括了 Python 相关的章节。相较于 PEP 8，该指南提供了更为详尽的代码指南。该指南包括了语言规范和风格规范两个部分。

其中，语言规范对 Python 中很多语言特性进行了优缺点的分析，并给出了使用指导意见，如异常、Lambda 表达式、列表推导式、metaclass 等。

风格规范的内容与 PEP 8 较为接近，大部分约定建立在 PEP 8 的基础上，也有一些更为详细的约定，如函数长度、TODO 注释、文件与 socket 对象的访问等。

推荐将该指南作为参考进行开发，但不必严格遵照，一来该指南存在一些 Python 2 兼容需求，例如指南中要求所有无基类的类应当显式地继承 Object, 而在仅使用 Python 3 的环境中，这一要求是不必要的，依本项目中的惯例即可。二来 OpenMMLab 的项目作为框架级的开源软件，不必对一些高级技巧过于避讳，尤其是 MMCV。但尝试使用这些技巧前应当认真考虑是否真的有必要，并寻求其他开发人员的广泛评估。

另外需要注意的一处规范是关于包的导入，在该指南中，要求导入本地包时必须使用路径全称，且导入的每一个模块都应当单独成行，通常这是不必要的，而且也不符合目前项目的开发惯例，此处进行如下约定：

```python
# Correct

```python
# Correct
from mmagic.cnn.bricks import (Conv2d, build_norm_layer, DropPath, MaxPool2d,
                             Linear)
from ..utils import ext_loader

# Wrong
from mmagic.cnn.bricks import Conv2d, build_norm_layer, DropPath, MaxPool2d, \
                            Linear  # 使用括号进行连接，而不是反斜杠
from ...utils import is_str  # 最多向上回溯一层，过多的回溯容易导致结构混乱
```

OpenMMLab 项目使用 pre-commit 工具自动格式化代码，详情见[贡献代码](contributing.md#代码风格)。

### 命名规范

#### 命名规范的重要性

优秀的命名是良好代码可读的基础。基础的命名规范对各类变量的命名做了要求，使读者可以方便地根据代码名了解变量是一个类 / 局部变量 / 全局变量等。而优秀的命名则需要代码作者对于变量的功能有清晰的认识，以及良好的表达能力，从而使读者根据名称就能了解其含义，甚至帮助了解该段代码的功能。

#### 基础命名规范

| 类型            | 公有             | 私有               |
| --------------- | ---------------- | ------------------ |
| 模块            | lower_with_under | \_lower_with_under |
| 包              | lower_with_under |                    |

| 类              | CapWords         | \_CapWords         |
| 异常            | CapWordsError    |                    |
| 函数（方法）    | lower_with_under | \_lower_with_under |
| 函数 / 方法参数 | lower_with_under |                    |
| 全局 / 类内常量 | CAPS_WITH_UNDER  | \_CAPS_WITH_UNDER  |
| 全局 / 类内变量 | lower_with_under | \_lower_with_under |
| 变量            | lower_with_under | \_lower_with_under |
| 局部变量        | lower_with_under |                    |

注意：

- 尽量避免变量名与保留字冲突，特殊情况下如不可避免，可使用一个后置下划线，如 class\_
- 尽量不要使用过于简单的命名，除了约定俗成的循环变量 i，文件变量 f，错误变量 e 等。
- 不会被用到的变量可以命名为 \_，逻辑检查器会将其忽略。

#### 命名技巧

良好的变量命名需要保证三点：

1. 含义准确，没有歧义
2. 长短适中
3. 前后统一

```python
# Wrong
class Masks(metaclass=ABCMeta):  # 命名无法表现基类；Instance or Semantic？
    pass

# Correct
class BaseInstanceMasks(metaclass=ABCMeta):
    pass

# Wrong，不同地方含义相同的变量尽量用统一的命名

3. 前后统一

```python
# Wrong
    pass

# Correct
    pass

def __init__(self, inplanes, planes):
    pass

def __init__(self, in_channels, out_channels):
    pass
```

常见的函数命名方法：

- 动宾命名法：crop_img, init_weights
- 动宾倒置命名法：imread, bbox_flip

注意函数命名与参数的顺序，保证主语在前，符合语言习惯：

- check_keys_exist(key, container)
- check_keys_contain(container, key)

注意避免非常规或统一约定的缩写，如 nb -> num_blocks，in_nc -> in_channels

### docstring 规范

#### 为什么要写 docstring

docstring 是对一个类、一个函数功能与 API 接口的详细描述，有两个功能，一是帮助其他开发者了解代码功能，方便 debug 和复用代码；二是在 Readthedocs 文档中自动生成相关的 API reference 文档，帮助不了解源代码的社区用户使用相关功能。

#### 如何写 docstring

与注释不同，一份规范的 docstring 有着严格的格式要求，以便于 Python 解释器以及 sphinx 进行文档解析，详细的 docstring 约定参见 [PEP 257](https://www.python.org/dev/peps/pep-0257/)。此处以例子的形式介绍各种文档的标准格式，参考格式为 [Google 风格](https://zh-google-styleguide.readthedocs.io/en/latest/google-python-styleguide/python_style_rules/#comments)。

1. 模块文档

   代码风格规范推荐为每一个模块（即 Python 文件）编写一个 docstring，但目前 OpenMMLab 项目大部分没有此类 docstring，因此不做硬性要求。

   ```python
   """A one line summary of the module or program, terminated by a period.

   Leave one blank line. The rest of this docstring should contain an
   overall description of the module or program. Optionally, it may also
   contain a brief description of exported classes and functions and/or usage
   examples.

   Typical usage example:

   foo = ClassFoo()
   bar = foo.FunctionBar()
   """
   ```

contain a brief description of exported classes and functions and/or usage
   examples.

   """
   ```

2. 类文档

   类文档是我们最常需要编写的，此处，按照 OpenMMLab 的惯例，我们使用了与 Google 风格不同的写法。如下例所示，文档中没有使用 Attributes 描述类属性，而是使用 Args 描述 __init__ 函数的参数。

   在 Args 中，遵照 `parameter (type): Description.` 的格式，描述每一个参数类型和功能。其中，多种类型可使用 `(float or str)` 的写法，可以为 None 的参数可以写为 `(int, optional)`。

   ```python
   class BaseRunner(metaclass=ABCMeta):
       """The base class of Runner, a training helper for PyTorch.

       All subclasses should implement the following APIs:

       - ``run()``
       - ``train()``
       - ``val()``
       - ``save_checkpoint()``

       Args:
           model (:obj:`torch.nn.Module`): The model to be run.
           batch_processor (callable, optional): A callable method that process
               a data batch. The interface of this method should be

batch_processor (callable, optional): A callable method that process
               ``batch_processor(model, data, train_mode) -> dict``.
               Defaults to None.
           optimizer (dict or :obj:`torch.optim.Optimizer`, optional): It can be
               either an optimizer (in most cases) or a dict of optimizers
               (in models that requires more than one optimizer, e.g., GAN).
           work_dir (str, optional): The working directory to save checkpoints
               and logs. Defaults to None.
           logger (:obj:`logging.Logger`): Logger used during training.
                Defaults to None. (The default value is just for backward
                compatibility)
           meta (dict, optional): A dict records some import information such as
               environment info and seed, which will be logged in logger hook.

compatibility)
           max_epochs (int, optional): Total training epochs. Defaults to None.
           max_iters (int, optional): Total training iterations. Defaults to None.
       """

                    model,
                    batch_processor=None,
                    optimizer=None,
                    work_dir=None,
                    logger=None,
                    meta=None,
                    max_iters=None,
                    max_epochs=None):
           ...
   ```

   另外，在一些算法实现的主体类中，建议加入原论文的链接；如果参考了其他开源代码的实现，则应加入 modified from，而如果是直接复制了其他代码库的实现，则应加入 copied from ，并注意源码的 License。如有必要，也可以通过 .. math:: 来加入数学公式

   ```python
   # 参考实现
   # This func is modified from `detectron2

```python
   # 参考实现
   # <https://github.com/facebookresearch/detectron2/blob/ffff8acc35ea88ad1cb1806ab0f00b4c1c5dbfd9/detectron2/structures/masks.py#L387>`_.

   # 复制代码
   # This code was copied from the `ubelt
   # library<https://github.com/Erotemic/ubelt>`_.

   # 引用论文 & 添加公式
   class LabelSmoothLoss(nn.Module):
       r"""Initializer for the label smoothed cross entropy loss.

       Refers to `Rethinking the Inception Architecture for Computer Vision
       <https://arxiv.org/abs/1512.00567>`_.

       This decreases gap between output scores and encourages generalization.
       Labels provided to forward can be one-hot like vectors (NxC) or class
       indices (Nx1).
       And this accepts linear combination of one-hot like labels from mixup or
       cutmix except multi-label task.

       Args:
           label_smooth_val (float): The degree of label smoothing.

And this accepts linear combination of one-hot like labels from mixup or

       Args:
           num_classes (int, optional): Number of classes. Defaults to None.
           mode (str): Refers to notes, Options are "original", "classy_vision",
               "multi_label". Defaults to "classy_vision".
           reduction (str): The method used to reduce the loss.
               Options are "none", "mean" and "sum". Defaults to 'mean'.
           loss_weight (float):  Weight of the loss. Defaults to 1.0.

       Note:
           if the ``mode`` is "original", this will use the same label smooth
           method as the original paper as:

           .. math::
               (1-\epsilon)\delta_{k, y} + \frac{\epsilon}{K}

           where :math:`\epsilon` is the ``label_smooth_val``, :math:`K` is
           the ``num_classes`` and :math:`\delta_{k,y}` is Dirac delta,

where :math:`\epsilon` is the ``label_smooth_val``, :math:`K` is
           which equals 1 for k=y and 0 otherwise.

           if the ``mode`` is "classy_vision", this will use the same label
           smooth method as the `facebookresearch/ClassyVision
           <https://github.com/facebookresearch/ClassyVision/blob/main/classy_vision/losses/label_smoothing_loss.py>`_ repo as:

           .. math::
               \frac{\delta_{k, y} + \epsilon/K}{1+\epsilon}

           if the ``mode`` is "multi_label", this will accept labels from
           multi-label task and smoothing them as:

           .. math::
               (1-2\epsilon)\delta_{k, y} + \epsilon
   ```

```{note}
注意 \`\`here\`\`、\`here\`、"here" 三种引号功能是不同。

在 reStructured 语法中，\`\`here\`\` 表示一段代码；\`here\` 表示斜体；"here" 无特殊含义，一般可用来表示字符串。其中 \`here\` 的用法与 Markdown 中不同，需要多加留意。
另外还有 :obj:\`type\` 这种更规范的表示类的写法，但鉴于长度，不做特别要求，一般仅用于表示非常用类型。
```

```

3. 方法（函数）文档

   函数文档与类文档的结构基本一致，但需要加入返回值文档。对于较为复杂的函数和类，可以使用 Examples 字段加入示例；如果需要对参数加入一些较长的备注，可以加入 Note 字段进行说明。

   对于使用较为复杂的类或函数，比起看大段大段的说明文字和参数文档，添加合适的示例更能帮助用户迅速了解其用法。需要注意的是，这些示例最好是能够直接在 Python 交互式环境中运行的，并给出一些相对应的结果。如果存在多个示例，可以使用注释简单说明每段示例，也能起到分隔作用。

   ```python
   def import_modules_from_strings(imports, allow_failed_imports=False):
       """Import modules from the given list of strings.

       Args:
           imports (list | str | None): The given module names to be imported.
           allow_failed_imports (bool): If True, the failed imports will return
               None. Otherwise, an ImportError is raise. Defaults to False.

       Returns:
           List[module] | module | None: The imported modules.
           All these three lines in docstring will be compiled into the same

Returns:
           line in readthedocs.

       Examples:
           >>> osp, sys = import_modules_from_strings(
           ...     ['os.path', 'sys'])
           >>> import os.path as osp_
           >>> import sys as sys_
           >>> assert osp == osp_
           >>> assert sys == sys_
       """
       ...
   ```

   如果函数接口在某个版本发生了变化，需要在 docstring 中加入相关的说明，必要时添加 Note 或者 Warning 进行说明，例如：

   ```python
   class CheckpointHook(Hook):
       """Save checkpoints periodically.

       Args:
           out_dir (str, optional): The root directory to save checkpoints. If
               not specified, ``runner.work_dir`` will be used by default. If
               specified, the ``out_dir`` will be the concatenation of
               ``out_dir`` and the last level directory of ``runner.work_dir``.

specified, the ``out_dir`` will be the concatenation of
               Defaults to None. `Changed in version 1.3.15.`
           file_client_args (dict, optional): Arguments to instantiate a
               FileClient. See :class:`mmagic.fileio.FileClient` for details.
               Defaults to None. `New in version 1.3.15.`

       Warning:
           Before v1.3.15, the ``out_dir`` argument indicates the path where the
           checkpoint is stored. However, in v1.3.15 and later, ``out_dir``
           indicates the root directory and the final path to save checkpoint is
           the concatenation of out_dir and the last level directory of
           ``runner.work_dir``. Suppose the value of ``out_dir`` is
           "/path/of/A" and the value of ``runner.work_dir`` is "/path/of/B",
           then the final path will be "/path/of/A/B".
   ```

   如果参数或返回值里带有需要展开描述字段的 dict，则应该采用如下格式：

"/path/of/A" and the value of ``runner.work_dir`` is "/path/of/B",
   ```

   ```python
   def func(x):
       r"""
       Args:
           x (None): A dict with 2 keys, ``padded_targets``, and ``targets``.

               - ``targets`` (list[Tensor]): A list of tensors.
                 Each tensor has the shape of :math:`(T_i)`. Each
                 element is the index of a character.
               - ``padded_targets`` (Tensor): A tensor of shape :math:`(N)`.
                 Each item is the length of a word.

       Returns:
           dict: A dict with 2 keys, ``padded_targets``, and ``targets``.

           - ``targets`` (list[Tensor]): A list of tensors.
             Each tensor has the shape of :math:`(T_i)`. Each
             element is the index of a character.
           - ``padded_targets`` (Tensor): A tensor of shape :math:`(N)`.

Each tensor has the shape of :math:`(T_i)`. Each
             Each item is the length of a word.
       """
       return x
   ```

```{important}
为了生成 readthedocs 文档，文档的编写需要按照 ReStructrued 文档格式，否则会产生文档渲染错误，在提交 PR 前，最好生成并预览一下文档效果。
语法规范参考：

- [reStructuredText Primer - Sphinx documentation](https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#)
- [Example Google Style Python Docstrings ‒ napoleon 0.7 documentation](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html#example-google)
```

### 注释规范

#### 为什么要写注释

对于一个开源项目，团队合作以及社区之间的合作是必不可少的，因而尤其要重视合理的注释。不写注释的代码，很有可能过几个月自己也难以理解，造成额外的阅读和修改成本。

#### 如何写注释

最需要写注释的是代码中那些技巧性的部分。如果你在下次代码审查的时候必须解释一下，那么你应该现在就给它写注释。对于复杂的操作，应该在其操作开始前写上若干行注释。对于不是一目了然的代码，应在其行尾添加注释。
—— Google 开源项目风格指南

```python
# We use a weighted dictionary search to find out where i is in

```python
# the array. We extrapolate position based on the largest num
# in the array and the array size and then do binary search to
# get the exact number.
if i & (i-1) == 0:  # True if i is 0 or a power of 2.
```

为了提高可读性, 注释应该至少离开代码2个空格.
另一方面, 绝不要描述代码. 假设阅读代码的人比你更懂Python, 他只是不知道你的代码要做什么.

```python
# Wrong:
# Now go through the b array and make sure whenever i occurs
# the next element is i+1

# Wrong:
if i & (i-1) == 0:  # True if i bitwise and i-1 is 0.
```

在注释中，可以使用 Markdown 语法，因为开发人员通常熟悉 Markdown 语法，这样可以便于交流理解，如可使用单反引号表示代码和变量（注意不要和 docstring 中的 ReStructured 语法混淆）

```python
# `_reversed_padding_repeated_twice` is the padding to be passed to
# `F.pad` if needed (e.g., for non-zero padding types that are
# implemented as two ops: padding + conv). `F.pad` accepts paddings in

# reverse order than the dimension.
self._reversed_padding_repeated_twice = _reverse_repeat_tuple(self.padding, 2)
```

#### 注释示例

1. 出自 `mmcv/utils/registry.py`，对于较为复杂的逻辑结构，通过注释，明确了优先级关系。

   ```python
   # self.build_func will be set with the following priority:
   # 1. build_func
   # 2. parent.build_func
   # 3. build_from_cfg
   if build_func is None:
       if parent is not None:
           self.build_func = parent.build_func
       else:
           self.build_func = build_from_cfg
   else:
       self.build_func = build_func
   ```

2. 出自 `mmcv/runner/checkpoint.py`，对于 bug 修复中的一些特殊处理，可以附带相关的 issue 链接，帮助其他人了解 bug 背景。

   ```python
   def _save_ckpt(checkpoint, file):
       # The 1.6 release of PyTorch switched torch.save to use a new
       # zipfile-based file format. It will cause RuntimeError when a

```python
       # checkpoint was saved in high version (PyTorch version>=1.6.0) but
       # loaded in low version (PyTorch version<1.6.0). More details at
       # https://github.com/open-mmlab/mmpose/issues/904
       if digit_version(TORCH_VERSION) >= digit_version('1.6.0'):
           torch.save(checkpoint, file, _use_new_zipfile_serialization=False)
       else:
           torch.save(checkpoint, file)
   ```

### 类型注解

#### 为什么要写类型注解

类型注解是对函数中变量的类型做限定或提示，为代码的安全性提供保障、增强代码的可读性、避免出现类型相关的错误。
Python 没有对类型做强制限制，类型注解只起到一个提示作用，通常你的 IDE 会解析这些类型注解，然后在你调用相关代码时对类型做提示。另外也有类型注解检查工具，这些工具会根据类型注解，对代码中可能出现的问题进行检查，减少 bug 的出现。
需要注意的是，通常我们不需要注释模块中的所有函数：

1. 公共的 API 需要注释
2. 在代码的安全性，清晰性和灵活性上进行权衡是否注释
3. 对于容易出现类型相关的错误的代码进行注释
4. 难以理解的代码请进行注释
5. 若代码中的类型已经稳定，可以进行注释. 对于一份成熟的代码，多数情况下，即使注释了所有的函数，也不会丧失太多的灵活性.

#### 如何写类型注解

1. 函数 / 方法类型注解，通常不对 self 和 cls 注释。

   ```python
   from typing import Optional, List, Tuple

   # 全部位于一行
   def my_method(self, first_var: int) -> int:
       pass

   # 另起一行
   def my_method(
           self, first_var: int,
           second_var: float) -> Tuple[MyLongType1, MyLongType1, MyLongType1]:
       pass

   # 单独成行（具体的应用场合与行宽有关，建议结合 yapf 自动化格式使用）
       self, first_var: int, second_var: float
   ) -> Tuple[MyLongType1, MyLongType1, MyLongType1]:
       pass

   # 引用尚未被定义的类型
   class MyClass:
                    stack: List["MyClass"]) -> None:
           pass
   ```

   注：类型注解中的类型可以是 Python 内置类型，也可以是自定义类，还可以使用 Python 提供的 wrapper 类对类型注解进行装饰，一些常见的注解如下：

   ```python
   # 数值类型
   from numbers import Number

   # 可选类型，指参数可以为 None

pass
   ```

   ```python
   # 数值类型

   from typing import Optional
   def foo(var: Optional[int] = None):
       pass

   # 联合类型，指同时接受多种类型
   from typing import Union
   def foo(var: Union[float, str]):
       pass

   from typing import Sequence  # 序列类型
   from typing import Iterable  # 可迭代类型
   from typing import Any  # 任意类型
   from typing import Callable  # 可调用类型

   from typing import List, Dict  # 列表和字典的泛型类型
   from typing import Tuple  # 元组的特殊格式
   # 虽然在 Python 3.9 中，list, tuple 和 dict 本身已支持泛型，但为了支持之前的版本
   # 我们在进行类型注解时还是需要使用 List, Tuple, Dict 类型
   # 另外，在对参数类型进行注解时，尽量使用 Sequence & Iterable & Mapping
   # List, Tuple, Dict 主要用于返回值类型注解
   # 参见 https://docs.python.org/3/library/typing.html#typing.List
   ```

2. 变量类型注解，一般用于难以直接推断其类型时

   ```python
   # Recommend: 带类型注解的赋值
   a: Foo = SomeUndecoratedFunction()

# 参见 https://docs.python.org/3/library/typing.html#typing.List
   ```

   ```python
   a: List[int]: [1, 2, 3]         # List 只支持单一类型泛型，可使用 Union
   b: Tuple[int, int] = (1, 2)     # 长度固定为 2
   c: Tuple[int, ...] = (1, 2, 3)  # 变长
   d: Dict[str, int] = {'a': 1, 'b': 2}

   # Not Recommend：行尾类型注释
   # 虽然这种方式被写在了 Google 开源指南中，但这是一种为了支持 Python 2.7 版本
   # 而补充的注释方式，鉴于我们只支持 Python 3, 为了风格统一，不推荐使用这种方式。
   a = SomeUndecoratedFunction()  # type: Foo
   a = [1, 2, 3]  # type: List[int]
   b = (1, 2, 3)  # type: Tuple[int, ...]
   c = (1, "2", 3.5)  # type: Tuple[int, Text, float]
   ```

3. 泛型

   上文中我们知道，typing 中提供了 list 和 dict 的泛型类型，那么我们自己是否可以定义类似的泛型呢？

   ```python
   from typing import TypeVar, Generic

   KT = TypeVar('KT')
   VT = TypeVar('VT')

   class Mapping(Generic[KT, VT]):
       def __init__(self, data: Dict[KT, VT]):
           self._data = data

from typing import TypeVar, Generic

       def __getitem__(self, key: KT) -> VT:
           return self._data[key]
   ```

   使用上述方法，我们定义了一个拥有泛型能力的映射类，实际用法如下：

   ```python
   mapping = Mapping[str, float]({'a': 0.5})
   value: float = example['a']
   ```

   另外，我们也可以利用 TypeVar 在函数签名中指定联动的多个类型：

   ```python
   from typing import TypeVar, List

   T = TypeVar('T')  # Can be anything
   A = TypeVar('A', str, bytes)  # Must be str or bytes

   def repeat(x: T, n: int) -> List[T]:
       """Return a list containing n references to x."""
       return [x]*n

   def longest(x: A, y: A) -> A:
       """Return the longest of two strings."""
       return x if len(x) >= len(y) else y
   ```

更多关于类型注解的写法请参考 [typing](https://docs.python.org/3/library/typing.html)。

#### 类型注解检查工具

"""Return the longest of two strings."""
   ```

[mypy](https://mypy.readthedocs.io/en/stable/) 是一个 Python 静态类型检查工具。根据你的类型注解，mypy 会检查传参、赋值等操作是否符合类型注解，从而避免可能出现的 bug。

例如如下的一个  Python 脚本文件 test.py:

```python
def foo(var: int) -> float:
    return float(var)

a: str = foo('2.0')
b: int = foo('3.0')  # type: ignore
```

运行 mypy test.py 可以得到如下检查结果，分别指出了第 4 行在函数调用和返回值赋值两处类型错误。而第 5 行同样存在两个类型错误，由于使用了 type: ignore 而被忽略了，只有部分特殊情况可能需要此类忽略。

```
test.py:4: error: Incompatible types in assignment (expression has type "float", variable has type "int")
test.py:4: error: Argument 1 to "foo" has incompatible type "str"; expected "int"
Found 2 errors in 1 file (checked 1 source file)
```

MMDetection3D and other OpenMMLab repositories use [MMEngine's config system](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html). It has a modular and inheritance design, which is convenient to conduct various experiments.

MMDetection3D uses a modular design, all modules with different functions can be configured through the config. Taking PointPillars as an example, we will introduce each field in the config according to different function modules.

In MMDetection3D's config, we use `model` to setup detection algorithm components. In addition to neural network components such as `voxel_encoder`, `backbone` etc, it also requires `data_preprocessor`, `train_cfg`, and `test_cfg`. `data_preprocessor` is responsible for processing a batch of data output by dataloader. `train_cfg` and `test_cfg` in the model config are training and testing hyperparameters of the components.

```python

```python

out_channels=[64, 128, 256]),

            dict(
            dict(

dict(
            dict(
        ],
```

nms_across_levels=False,
```

[Dataloaders](https://pytorch.org/docs/stable/data.html?highlight=data%20loader#torch.utils.data.DataLoader) are required for the training, validation, and testing of the [runner](https://mmengine.readthedocs.io/en/latest/tutorials/runner.html). Dataset and data pipeline need to be set to build the dataloader. Due to the complexity of this part, we use intermediate variables to simplify the writing of dataloader configs.

```python

    rate=1.0,

    rate=1.0,

    dict(

type='GlobalRotScaleTrans',
    dict(
]
    dict(
            dict(
            dict(
        ]),

translation_std=[0, 0, 0]),
            dict(
        ]),
]
]
        times=2,

```

```python
```

Since the test dataset has no annotation files, the test_dataloader and test_evaluator config in MMDetection3D are generally equal to the val's. If you want to save the detection results on the test dataset, you can write the config like this:

```python

```python
```

submission_prefix='results/kitti-3class/kitti_results')
```

Users can set the maximum training epochs and validation intervals with these fields:

```python
```

```python

```python
    clip_grad=dict(max_norm=35, norm_type=2))  # Gradient clip option. Set None to disable gradient clip. Find usage in https://mmengine.readthedocs.io/en/latest/tutorials/optim_wrapper.html
```

clip_grad=dict(max_norm=35, norm_type=2))  # Gradient clip option. Set None to disable gradient clip. Find usage in https://mmengine.readthedocs.io/en/latest/tutorials/optim_wrapper.html
```

`param_scheduler` is a field that configures methods of adjusting optimization hyperparameters such as learning rate and momentum. Users can combine multiple schedulers to create a desired parameter adjustment strategy. Find more in [parameter scheduler tutorial](https://mmengine.readthedocs.io/en/latest/tutorials/param_scheduler.html) and [parameter scheduler API documents](https://mmengine.readthedocs.io/en/latest/api/optim.html#scheduler).

```python
    dict(
        T_max=32,
        begin=0,
        end=32,
    dict(
        T_max=48,
        begin=32,
        end=80,

    dict(
        T_max=48,
        begin=32,
        end=80,
    dict(
        T_max=32,
        begin=0,
        end=32,
    dict(
        T_max=48,
        begin=32,
        end=80,
]
```

`default_hooks` is a dict of hook configs, and they are the hooks must be required at the runtime. They have default priority which should not be modified. If not set, runner will use the default values. To disable a default hook, users can set its config to `None`.

```python
```

```python
```

```python
default_scope = 'mmdet3d'  # The default registry scope to find modules. Refer to https://mmengine.readthedocs.io/en/latest/advanced_tutorials/registry.html

```python

    dist_cfg=dict(backend='nccl'))  # Distribution configs

```

There are 4 basic component types under `configs/_base_`, dataset, model, schedule, default_runtime.
Many methods could be easily constructed with one of these models like SECOND, PointPillars, PartA2, VoteNet.
The configs that are composed of components from `_base_` are called _primitive_.

For example, if some modification is made based on PointPillars, users may first inherit the basic PointPillars structure by specifying `_base_ = '../pointpillars/pointpillars_hv_fpn_sbn-all_8xb4-2x_nus-3d.py'`, then modify the necessary fields in the config files.

If you are building an entirely new method that does not share the structure with any of the existing methods, you may create a folder `xxx_rcnn` under `configs`.

Please refer to [MMEngine config tutorial](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html) for detailed documentation.

```python
```

```python
_base_ = [
]
```

```python
_base_ = [
]
```

You may refer to [MMEngine config tutorial](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html) for a simple illustration.

In MMDetection3D, for example, to change the neck of PointPillars with the following config:

```python

data_preprocessor=dict(voxel_layer=dict(...)),
```

`FPN` and `SECONDFPN` use different keywords to construct:

```python
```

The `_delete_=True` would replace all old keys in `pts_neck` field with new keys.

out_channels=[128, 128, 128]),
```

It's worth noting that when modifying intermediate variables in the children configs, user needs to pass the intermediate variables into corresponding fields again.
For example, we would like to use a multi-scale strategy to train and test a PointPillars, `train_pipeline`/`test_pipeline` are intermediate variables we would like to modify.

```python
    dict(
    dict(

load_dim=5,
    dict(
    dict(
    dict(
]
    dict(
    dict(

]
    dict(
    dict(
    dict(
            dict(
            dict(
        ]),
]
```

]
```

```python

a = {{_base_.model}}  # variable `a` is equal to the `model` defined in `_base_`
```

- Update config keys of dict chains

- Update keys inside a list of configs

  e.g. `[dict(type='LoadPointsFromFile'), ...]`. If you want to change `'LoadPointsFromFile'` to `'LoadPointsFromDict'` in the pipeline,
  you may specify `--cfg-options data.train.pipeline.0.type=LoadPointsFromDict`.

- Update values of list/tuple

```
```

```

- `{algorithm name}`: The name of the algorithm. It can be a detector name such as `pointpillars`, `fcos3d`, etc.
- `{model component names}`: Names of the components used in the algorithm such as voxel_encoder, backbone, neck, etc. For example, `second_secfpn_head-dcn-circlenms` means using SECOND's SparseEncoder, SECONDFPN and a detection head with DCN and circle NMS.
- `{training settings}`: Information of training settings such as batch size, augmentations, loss trick, scheduler, and epochs/iterations. For example: `8xb4-tta-cyclic-20e` means using 8-gpus x 4-samples-per-gpu, test time augmentation, cyclic annealing learning rate, and train 20 epochs.
  - `{gpu x batch_per_gpu}`: GPUs and samples per GPU. `bN` indicates N batch size per GPU. E.g. `4xb4` is the short term of 4-GPUs x 4-samples-per-GPU.

  - `{schedule}`: training schedule, options are `schedule-2x`, `schedule-3x`, `cyclic-20e`, etc.
    `schedule-2x` and `schedule-3x` mean 24 epochs and 36 epochs respectively.
    `cyclic-20e` means 20 epochs respectively.
- `{training dataset information}`: Training dataset names like `kitti-3d-3class`, `nus-3d`, `s3dis-seg`, `scannet-seg`, `waymoD5-3d-car`. Here `3d` means dataset used for 3D object detection, and `seg` means dataset used for point cloud segmentation.

# mmdeploy Architecture

This article mainly introduces the functions of each directory of mmdeploy and how it works from model conversion to real inference.

## Take a general look at the directory structure

The entire mmdeploy can be seen as two independent parts: model conversion and SDK.

We introduce the entire repo directory structure and functions, without having to study the source code, just have an impression.

Peripheral directory features:

```bash
$ tree -L 1
.
├── CMakeLists.txt    # Compile custom operator and cmake configuration of SDK
├── configs                   # Algorithm library configuration for model conversion
├── csrc                          # SDK and custom operator
├── demo                      # FFI interface examples in various languages, such as csharp, java, python, etc.
├── docker                   # docker build
├── mmdeploy           # python package for model conversion
├── requirements      # python requirements

├── service                    # Some small boards not support python, we use C/S mode for model conversion, here is server code
├── tests                         # unittest
├── third_party           # 3rd party dependencies required by SDK and FFI
└── tools                        # Tools are also the entrance to all functions, such as onnx2xx.py, profiler.py, test.py, etc.
```

It should be clear

- Model conversion mainly depends on `tools`, `mmdeploy` and small part of `csrc` directory;
- SDK is consist of three directories: `csrc`, `third_party` and `demo`.

## Model Conversion

Here we take ViT of mmpretrain as model example, and take ncnn as inference backend example. Other models and inferences are similar.

Let's take a look at the mmdeploy/mmdeploy directory structure and get an impression:

```bash
.

```bash
.
├── apis                             # The api used by tools is implemented here, such as onnx2ncnn.py
│   ├── calibration.py          # trt dedicated collection of quantitative data
│   ├── core                              # Software infrastructure
│   ├── extract_model.py  # Use it to export part of onnx
│   ├── inference.py             # Abstract function, which will actually call torch/ncnn specific inference
│   ├── ncnn                            # ncnn Wrapper
│   └── visualize.py              # Still an abstract function, which will actually call torch/ncnn specific inference and visualize
..
├── backend                  # Backend wrapper
│   ├── base                            # Because there are multiple backends, there must be an OO design for the base class
│   ├── ncnn                           # This calls the ncnn python interface for model conversion

│   │   ├── init_plugins.py           # Find the path of ncnn custom operators and ncnn tools
│   │   ├── onnx2ncnn.py            # Wrap `mmdeploy_onnx2ncnn` into a python interface
│   │   ├── quant.py                       # Wrap `ncnn2int8` as a python interface
│   │   └── wrapper.py                  # Wrap pyncnn forward API
..
├── codebase                #  Algorithm rewriter
│   ├── base                          # There are multiple algorithms here that we need a bit of OO design
│   ├── mmpretrain                      #  mmpretrain related model rewrite
│   │   ├── deploy                       # mmpretrain implementation of base abstract task/model/codebase
│   │   └── models                      # Real model rewrite
│   │       ├── backbones                 # Rewrites of backbone network parts, such as multiheadattention

│   │       ├── heads                           # Such as MultiLabelClsHead
│   │       ├── necks                            # Such as GlobalAveragePooling
│..
├── core                         # Software infrastructure of rewrite mechanism
├── mmcv                     #  Rewrite mmcv
├── pytorch                 #  Rewrite pytorch operator for ncnn, such as Gemm
..
```

Each line above needs to be read, don't skip it.

When typing `tools/deploy.py` to convert ViT, these are 3 things:

1. Rewrite of mmpretrain ViT forward
2. ncnn does not support `gather` opr, customize and load it with libncnn.so
3. Run exported ncnn model with real inference, render output, and make sure the result is correct

### 1. Rewrite `forward`

Because when exporting ViT to onnx, it generates some operators that ncnn doesn't support perfectly, mmdeploy's solution is to hijack the forward code and change it. The output onnx is suitable for ncnn.

For example, rewrite the process of `conv -> shape -> concat_const -> reshape` to `conv -> reshape` to trim off the redundant `shape` and `concat` operator.

All mmpretrain algorithm rewriters are in the `mmdeploy/codebase/mmpretrain/models` directory.

### 2. Custom Operator

Operators customized for ncnn are in the `csrc/mmdeploy/backend_ops/ncnn/` directory, and are loaded together with `libncnn.so` after compilation. The essence is in hotfix ncnn, which currently implements these operators:

- topk
- tensorslice
- shape
- gather
- expand
- constantofshape

### 3. Model Conversion and testing

- topk
- shape
- gather
- expand

We first use the modified `mmdeploy_onnx2ncnn`to convert model, then inference with`pyncnn` and custom ops.

When encountering a framework such as snpe that does not support python well, we use C/S mode: wrap a server with protocols such as gRPC, and forward the real inference output.

For Rendering, mmdeploy directly uses the rendering API of upstream algorithm codebase.

## SDK

After the model conversion completed, the SDK compiled with C++ can be used to execute on different platforms.

Let's take a look at the csrc/mmdeploy directory structure:

```bash
.
├── apis           # csharp, java, go, Rust and other FFI interfaces
├── backend_ops    # Custom operators for each inference framework
├── CMakeLists.txt
├── codebase       # The type of results preferred by each algorithm framework, such as multi-use bbox for detection task

├── core           # Abstraction of graph, operator, device and so on
├── device         # Implementation of CPU/GPU device abstraction
├── execution      # Implementation of the execution abstraction
├── graph          # Implementation of graph abstraction
├── model          # Implement both zip-compressed and uncompressed work directory
├── net            # Implementation of net, such as wrap ncnn forward C API
├── preprocess     # Implement preprocess
└── utils          # OCV tools
```

The essence of the SDK is to design a set of abstraction of the computational graph, and combine the **multiple models'**

- preprocess
- inference
- postprocess

Provide FFI in multiple languages at the same time.

# From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network

> [From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network](https://arxiv.org/abs/1907.03670)

3D object detection from LiDAR point cloud is a challenging problem in 3D scene understanding and has many practical applications. In this paper, we extend our preliminary work PointRCNN to a novel and strong point-cloud-based 3D object detection framework, the part-aware and aggregation neural network (Part-A2 net). The whole framework consists of the part-aware stage and the part-aggregation stage. Firstly, the part-aware stage for the first time fully utilizes free-of-charge part supervisions derived from 3D ground-truth boxes to simultaneously predict high quality 3D proposals and accurate intra-object part locations. The predicted intra-object part locations within the same proposal are grouped by our new-designed RoI-aware point cloud pooling module, which results in an effective representation to encode the geometry-specific features of each 3D proposal. Then the part-aggregation stage learns to re-score the box and refine the box location by exploring the spatial relationship of the pooled intra-object part locations. Extensive experiments are conducted to demonstrate the performance improvements from each component of our proposed framework. Our Part-A2 net outperforms all existing 3D detection methods and achieves new state-of-the-art on KITTI 3D object detection dataset by utilizing only the LiDAR point cloud data.

<img src="https://user-images.githubusercontent.com/79644370/143882774-6fc5f736-10d1-499a-8929-ca0768419049.png" width="800"/>
</div>

We implement Part-A^2 and provide its results and checkpoints on KITTI dataset.

### KITTI

|                            Backbone                             |  Class  |  Lr schd   | Mem (GB) | Inf time (fps) |  mAP  |                                                                                                                                                                                                   Download                                                                                                                                                                                                   |

| [SECFPN](./parta2_hv_secfpn_8xb2-cyclic-80e_kitti-3d-3class.py) | 3 Class | cyclic 80e |   4.1    |                | 68.33 | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/parta2/hv_PartA2_secfpn_2x8_cyclic_80e_kitti-3d-3class/hv_PartA2_secfpn_2x8_cyclic_80e_kitti-3d-3class_20210831_022017-454a5344.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/parta2/hv_PartA2_secfpn_2x8_cyclic_80e_kitti-3d-3class/hv_PartA2_secfpn_2x8_cyclic_80e_kitti-3d-3class_20210831_022017.log.json) |

|  [SECFPN](./parta2_hv_secfpn_8xb2-cyclic-80e_kitti-3d-car.py)   |   Car   | cyclic 80e |   4.0    |                | 79.08 |       [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/parta2/hv_PartA2_secfpn_2x8_cyclic_80e_kitti-3d-car/hv_PartA2_secfpn_2x8_cyclic_80e_kitti-3d-car_20210831_022017-cb7ff621.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/parta2/hv_PartA2_secfpn_2x8_cyclic_80e_kitti-3d-car/hv_PartA2_secfpn_2x8_cyclic_80e_kitti-3d-car_20210831_022017.log.json)       |

```latex
@article{shi2020points,
  title={From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network},
  author={Shi, Shaoshuai and Wang, Zhe and Shi, Jianping and Wang, Xiaogang and Li, Hongsheng},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
}
```

# Docker Image

We provide a [Dockerfile](Dockerfile) to build an image.

```shell
```

Run it with

```shell
```

**Note**:
Versions defined in this [Dockerfile](Dockerfile) is not up-to-date.
If you use this Dockerfile in your project, you probably want to make some updates.
Feel free to submit an issue or PR for the update.

# 自定义数据集

在本教程中，我们将介绍如何通过在线转换来自定义你的数据集。

- [自定义数据集](#自定义数据集)
  - [MMAction2 数据集概述](#mmaction2-数据集概述)
  - [定制新的数据集](#定制新的数据集)
  - [为 PoseDataset 自定义关键点格式](#为-posedataset-自定义关键点格式)

## MMAction2 数据集概述

MMAction2 提供了任务特定的 `Dataset` 类，例如用于动作识别的 `VideoDataset`/`RawframeDataset`，用于时空动作检测的 `AVADataset`，用于基于骨骼的动作识别的`PoseDataset`。这些任务特定的数据集只需要实现 `load_data_list(self)` 来从注释文件生成数据列表。剩下的函数由超类（即 `BaseActionDataset` 和 `BaseDataset`）自动处理。下表显示了模块的继承关系和主要方法。

| 类名                           | 类方法                                                                                                                                                        |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |

| `MMAction2::VideoDataset`      | `load_data_list(self)` <br> 从注释文件中构建数据列表。                                                                                                        |
| `MMAction2::BaseActionDataset` | `get_data_info(self, idx)` <br> 给定 `idx`，从数据列表中返回相应的数据样本。                                                                                  |
| `MMEngine::BaseDataset`        | `__getitem__(self, idx)` <br> 给定 `idx`，调用 `get_data_info` 获取数据样本，然后调用 `pipeline` 在 `train_pipeline` 或 `val_pipeline` 中执行数据变换和增强。 |

## 定制新的数据集类

大多数情况下，把你的数据集离线转换成指定格式是首选方法，但 MMAction2 提供了一个方便的过程来创建一个定制的 `Dataset` 类。如前所述，任务特定的数据集只需要实现 `load_data_list(self)` 来从注释文件生成数据列表。请注意，`data_list` 中的元素是包含后续流程中必要字段的 `dict`。

以 `VideoDataset` 为例，`train_pipeline`/`val_pipeline` 在 `DecordInit` 中需要 `'filename'`，在 `PackActionInputs` 中需要 `'label'`。因此，`data_list` 中的数据样本必须包含2个字段：`'filename'`和`'label'`。
请参考[定制数据流水线](customize_pipeline.md)以获取有关 `pipeline` 的更多详细信息。

```
```

`AVADataset` 会更加复杂，`data_list` 中的数据样本包含有关视频数据的几个字段。此外，它重写了 `get_data_info(self, idx)` 以转换在时空动作检测数据流水线中需要用的字段。

```python

  ...

   def load_data_list(self) -> List[dict]:
      ...
            ann=ann)

timestamp=int(timestamp),
            ann=ann)

      ...
```

## 为 PoseDataset 自定义关键点格式

MMAction2 目前支持三种关键点格式：`coco`，`nturgb+d` 和 `openpose`。如果你使用其中一种格式，你可以简单地在以下模块中指定相应的格式：

对于图卷积网络，如 AAGCN，STGCN，...

- `pipeline`：在 `JointToBone` 中的参数 `dataset`。
- `backbone`：在图卷积网络中的参数 `graph_cfg`。

对于 PoseC3D：

- `pipeline`：在 `Flip` 中，根据关键点的对称关系指定 `left_kp` 和 `right_kp`。
- `pipeline`：在 `GeneratePoseTarget` 中，如果 `with_limb` 为 `True`，指定`skeletons`，`left_limb`，`right_limb`，如果 `with_kp` 为 `True`，指定`left_kp` 和 `right_kp`。

如果使用自定义关键点格式，需要在 `backbone` 和 `pipeline` 中都包含一个新的图布局。这个布局将定义关键点及其连接关系。

以 `coco` 数据集为例，我们在 `Graph` 中定义了一个名为 `coco` 的布局。这个布局的 `inward` 连接包括所有节点连接，每个**向心**连接由一个节点元组组成。`coco`的额外设置包括将节点数指定为 `17`，将 `node 0` 设为中心节点。

```python

```

同样，我们在 `JointToBone` 中定义了 `pairs`，添加了一个 bone `(0, 0)` 以使 bone 的数量对齐到 joint。coco数据集的 `pairs` 如下所示，`JointToBone` 中的 `pairs` 的顺序无关紧要。

```python

self.pairs = ((0, 0), (1, 0), (2, 0), (3, 1), (4, 2),
                (5, 0), (6, 0), (7, 5), (8, 6), (9, 7),
                (10, 8), (11, 0), (12, 0), (13, 11), (14, 12),
                (15, 13), (16, 14))
```

(5, 0), (6, 0), (7, 5), (8, 6), (9, 7),
```

要使用你的自定义关键点格式，只需定义上述设置为你的图结构，并在你的配置文件中指定它们，如下所示。在这个例子中，我们将使用 `STGCN`，其中 `n` 表示类别的数量，`custom_dataset` 在 `Graph` 和 `JointToBone` 中定义。

```python

  ...
  ...]

  ...
  ...]

  ...
  ...]

```

只需简单地指定自定义布局，你就可以使用你自己的关键点格式进行训练和测试了。通过这种方式，MMAction2 为用户提供了很大的灵活性，允许用户自定义他们的数据集和关键点格式，以满足他们特定的需求。

...]

  ...
  ...]

```

以上就是关于如何自定义你的数据集的一些方法。希望这个教程能帮助你理解MMAction2的数据集结构，并教给你如何根据自己的需求创建新的数据集。虽然这可能需要一些编程知识，但是 MMAction2 试图使这个过程尽可能简单。通过了解这些基本概念，你将能够更好地控制你的数据，从而改进你的模型性能。

# SAM

> [Segment Anything](https://arxiv.org/abs/2304.02643)

We introduce the Segment Anything (SA) project: a new task, model, and dataset for image segmentation. Using our efficient model in a data collection loop, we built the largest segmentation dataset to date (by far), with over 1 billionmasks on 11M licensed and privacy respecting images. The model is designed and trained to be promptable, so it can transfer zero-shot to new image distributions and tasks. We evaluate its capabilities on numerous tasks and find that its zero-shot performance is impressive – often competitive with or even superior to prior fully supervised results. We are releasing the Segment Anything Model (SAM) and corresponding dataset (SA-1B) of 1B masks and 11M images at https://segment-anything.com to foster research into foundation models for computer vision.

<img src="https://user-images.githubusercontent.com/36138628/231106092-261ff035-dd3b-4a8b-b2e7-e91f195090a1.png" width="100%"/>
</div>

```python

model = get_model('vit-base-p16_sam-pre_3rdparty_sa1b-1024px', pretrained=True)
inputs = torch.rand(1, 3, 1024, 1024)
```

| Model                                          | Params (M) | Flops (G) |                 Config                  |                                             Download                                             |

| :--------------------------------------------- | :--------: | :-------: | :-------------------------------------: | :----------------------------------------------------------------------------------------------: |
| `vit-base-p16_sam-pre_3rdparty_sa1b-1024px`\*  |   89.67    |  486.00   | [config](vit-base-p16_sam_headless.py)  | [model](https://download.openmmlab.com/mmclassification/v1/vit_sam/vit-base-p16_sam-pre_3rdparty_sa1b-1024px_20230411-2320f9cc.pth) |
| `vit-large-p16_sam-pre_3rdparty_sa1b-1024px`\* |   308.00   |  1494.00  | [config](vit-large-p16_sam_headless.py) | [model](https://download.openmmlab.com/mmclassification/v1/vit_sam/vit-large-p16_sam-pre_3rdparty_sa1b-1024px_20230411-595feafd.pth) |
| `vit-huge-p16_sam-pre_3rdparty_sa1b-1024px`\*  |   637.00   |  2982.00  | [config](vit-huge-p16_sam_headless.py)  | [model](https://download.openmmlab.com/mmclassification/v1/vit_sam/vit-huge-p16_sam-pre_3rdparty_sa1b-1024px_20230411-3f13c653.pth) |

*Models with * are converted from the [official repo](https://github.com/facebookresearch/segment-anything/). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@article{kirillov2023segany,
  title={Segment Anything},
  author={Kirillov, Alexander and Mintun, Eric and Ravi, Nikhila and Mao, Hanzi and Rolland, Chloe and Gustafson, Laura and Xiao, Tete and Whitehead, Spencer and Berg, Alexander C. and Lo, Wan-Yen and Doll{\'a}r, Piotr and Girshick, Ross},
  journal={arXiv:2304.02643},
}
```

> **任务**: 图像修复

<details>
<summary align="right">Global&Local (ToG'2017)</summary>

```bibtex
}
```

<br/>

*请注意，为了与当前的深度图像修复方法进行公平比较，我们没有在 Global&Local 中使用后处理模块。*

|                     算法                     |  掩膜类型   | 分辨率  | 训练集容量 |    测试集     | l1 损失 |  PSNR  | SSIM  | GPU 信息 |                                           下载                                            |
| :------------------------------------------: | :---------: | :-----: | :--------: | :-----------: | :-----: | :----: | :---: | :------: | :---------------------------------------------------------------------------------------: |

| [Global&Local](./gl_8xb12_places-256x256.py) | square bbox | 256x256 |    500k    | Places365-val | 11.164  | 23.152 | 0.862 |    8     | [模型](https://download.openmmlab.com/mmediting/inpainting/global_local/gl_256x256_8x12_places_20200619-52a040a8.pth) \| [日志](https://download.openmmlab.com/mmediting/inpainting/global_local/gl_256x256_8x12_places_20200619-52a040a8.log.json) |

|                     算法                     |  掩膜类型   | 分辨率  | 训练集容量 |   测试集   | l1 损失 |  PSNR  | SSIM  | GPU 信息 |                                             下载                                             |
| :------------------------------------------: | :---------: | :-----: | :--------: | :--------: | :-----: | :----: | :---: | :------: | :------------------------------------------------------------------------------------------: |

| [Global&Local](./gl_8xb12_celeba-256x256.py) | square bbox | 256x256 |    500k    | CelebA-val |  6.678  | 26.780 | 0.904 |    8     | [模型](https://download.openmmlab.com/mmediting/inpainting/global_local/gl_256x256_8x12_celeba_20200619-5af0493f.pth) \| [日志](https://download.openmmlab.com/mmediting/inpainting/global_local/gl_256x256_8x12_celeba_20200619-5af0493f.log.json) |

## 快速开始

**训练**

<details>

```shell
# CPU上训练

# 单个GPU上训练

# 多个GPU上训练
```

**测试**

<details>

```shell
# CPU上测试

```

**测试**

<details>

```shell
# CPU上测试

# 单个GPU上测试

# 多个GPU上测试
```

## Binaries

We provide nuget package on our [release page](https://github.com/open-mmlab/mmdeploy/releases). Currently the prebuilt package only support tensorrt and onnxruntiem backend.

To use the nuget package, you also need to download the backend dependencies. For example, if you want to use the tensorrt backend, you should install cudatoolkit, cudnn and tensorrt, remember to add the dll directories to your system path. The version of backend dependencies that our prebuit nuget package used will be offered in release note.

## From Source

- Environment required by building sdk
- .NET Framework 4.8 / .NET core 3.1
- Visual Studio 2019+

### Installation

**Step 0.** Build sdk.

Before building the c# api, you need to build sdk first. Please follow this [tutorial](../../../docs/en/build/windows.md)/[教程](../../../docs/zh_cn/build/windows.md) to build sdk. Remember to set the MMDEPLOY_BUILD_SDK_CSHARP_API option to ON. We recommend setting `MMDEPLOY_SHARED_LIBS` to OFF and use the static third party libraries(pplcv, opencv, etc.). If so, you only need add the backend dependencies to your system path, or you need to add all dependencies.

If you follow the tutorial, the mmdeploy.dll will be built in `build\bin\release`. Make sure the expected dll is in that path or the next step will throw a file-not-exist error.

**Step 1.** Build MMDeploy nuget package.

There are two methods to build the nuget package.

(*option 1*) Use the command.

If your environment is well prepared, you can just go to the `csrc\apis\csharp` folder, open a terminal and type the following command, the nupkg will be built in `csrc\apis\csharp\MMDeploy\bin\Release\MMDeployCSharp.1.2.0.nupkg`.

```shell
dotnet build --configuration Release -p:Version=1.2.0
```

(*option 2*) Open MMDeploy.sln && Build.

You can set the package-version through `Properties -> Package Version`. The default version is 1.2.0 if you don't set it.

If you encounter missing dependencies, follow the instructions for MSVC.

# 准备 Vid4 数据集

```bibtex
@article{xue2019video,
  title={On Bayesian adaptive video super resolution},
  author={Liu, Ce and Sun, Deqing},
  pages={346--360},
  year={2013},
}
```

可以从 [此处](https://drive.google.com/file/d/1ZuvNNLgR85TV_whJoHM7uVb-XW1y70DW/view?usp=sharing) 下载 Vid4 数据集，其中包含了由两种下采样方法得到的图片：

1. BIx4 包含了由双线性插值下采样得到的图片
2. BDx4 包含了由 `σ=1.6` 的高斯核模糊，然后每4个像素进行一次采样得到的图片

请注意，应为 Vid4 数据集准备一个如下所列的标注文件（例如 meta_info_Vid4_GT.txt）。

```text
calendar 41 (576,720,3)
city 34 (576,704,3)
foliage 49 (480,720,3)
walk 47 (480,720,3)
```

对于 ToFlow，应准备直接上采样的数据集，为此，我们提供了一个脚本：

```shell
python tools/dataset_converters/vid4/preprocess_vid4_dataset.py --data-root ./data/Vid4/BIx4
```

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── Vid4
│   │   ├── GT
│   │   │   ├── calendar
│   │   │   ├── city
│   │   │   ├── foliage

```

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── Vid4
│   │   ├── GT
│   │   │   ├── walk
│   │   ├── BDx4
│   │   ├── BIx4
│   │   ├── BIx4up_direct
│   │   ├── meta_info_Vid4_GT.txt
```

}
```

<details>

```bibtex
}
```

| [pose_cspnext_m](/configs/animal_2d_keypoint/topdown_heatmap/ap10k/cspnext-m_udp_8xb64-210e_ap10k-256x256.py) |  256x256   | 0.703 |      0.944      |      0.776      |     0.513      |     0.710      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-m_udp-ap10k_pt-in1k_210e-256x256-1f2d947a_20230123.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-m_udp-ap10k_pt-in1k_210e-256x256-1f2d947a_20230123.json) |

# EDVR (CVPRW'2019)

<details>
<summary align="right">EDVR (CVPRW'2019)</summary>

```bibtex
@InProceedings{wang2019edvr,
  author    = {Wang, Xintao and Chan, Kelvin C.K. and Yu, Ke and Dong, Chao and Loy, Chen Change},
  title     = {EDVR: Video restoration with enhanced deformable convolutional networks},
  booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
  month     = {June},
  year      = {2019},
}
```

<br/>

在 RGB 通道上进行评估。

|                                       算法                                       |      REDS4       |         GPU 信息         |                                       下载                                       |
| :------------------------------------------------------------------------------: | :--------------: | :----------------------: | :------------------------------------------------------------------------------: |

|         [edvrm_wotsa_x4_8x4_600k_reds](./edvrm_wotsa_8xb4-600k_reds.py)          | 30.3430 / 0.8664 |            8             | [模型](https://download.openmmlab.com/mmediting/restorers/edvr/edvrm_wotsa_x4_8x4_600k_reds_20200522-0570e567.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/edvr/edvrm_wotsa_x4_8x4_600k_reds_20200522_141644.log.json) |
|               [edvrm_x4_8x4_600k_reds](./edvrm_8xb4-600k_reds.py)                | 30.4194 / 0.8684 |            8             | [模型](https://download.openmmlab.com/mmediting/restorers/edvr/edvrm_x4_8x4_600k_reds_20210625-e29b71b5.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/edvr/edvrm_x4_8x4_600k_reds_20200622_102544.log.json) |

| [edvrl_wotsa_c128b40_8x8_lr2e-4_600k_reds4](./edvrl_wotsa-c128b40_8xb8-lr2e-4-600k_reds4.py) | 31.0010 / 0.8784 | 8 (Tesla V100-PCIE-32GB) | [模型](https://download.openmmlab.com/mmediting/restorers/edvr/edvrl_wotsa_c128b40_8x8_lr2e-4_600k_reds4_20211228-d895a769.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/edvr/edvrl_wotsa_c128b40_8x8_lr2e-4_600k_reds4_20211228_144658.log.json) |
| [edvrl_c128b40_8x8_lr2e-4_600k_reds4](./edvrl_c128b40_8xb8-lr2e-4-600k_reds4.py) | 31.0467 / 0.8793 | 8 (Tesla V100-PCIE-32GB) | [模型](https://download.openmmlab.com/mmediting/restorers/edvr/edvrl_c128b40_8x8_lr2e-4_600k_reds4_20220104-4509865f.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/edvr/edvrl_c128b40_8x8_lr2e-4_600k_reds4_20220104_171823.log.json) |

## 快速开始

**训练**

<details>

```shell
# CPU上训练
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/edvr/edvrm_8xb4-600k_reds.py

# 单个GPU上训练

## 快速开始

**训练**

<details>

```shell
# CPU上训练

# 单个GPU上训练
python tools/train.py configs/edvr/edvrm_8xb4-600k_reds.py

# 多个GPU上训练
./tools/dist_train.sh configs/edvr/edvrm_8xb4-600k_reds.py 8
```

**测试**

<details>

```shell
# CPU上测试
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/edvr/edvrm_8xb4-600k_reds.py https://download.openmmlab.com/mmediting/restorers/edvr/edvrm_x4_8x4_600k_reds_20210625-e29b71b5.pth

# 单个GPU上测试
python tools/test.py configs/edvr/edvrm_8xb4-600k_reds.py https://download.openmmlab.com/mmediting/restorers/edvr/edvrm_x4_8x4_600k_reds_20210625-e29b71b5.pth

# 多个GPU上测试

# 单个GPU上测试

# 多个GPU上测试
./tools/dist_test.sh configs/edvr/edvrm_8xb4-600k_reds.py https://download.openmmlab.com/mmediting/restorers/edvr/edvrm_x4_8x4_600k_reds_20210625-e29b71b5.pth 8
```

# Preparing Places365 Dataset

```bibtex
 @article{zhou2017places,
   title={Places: A 10 million Image Database for Scene Recognition},
   author={Zhou, Bolei and Lapedriza, Agata and Khosla, Aditya and Oliva, Aude and Torralba, Antonio},
   publisher={IEEE}
 }

```

Prepare the data from [Places365](http://places2.csail.mit.edu/download.html).

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── Places
│   │   ├── data_large
│   │   ├── val_large
|   |   ├── meta
|   |   |    ├── places365_train_challenge.txt
|   |   |    ├── places365_val.txt

```

# Tutorial 4: Train and test with existing models

MMSegmentation supports training and testing models on a variety of devices, which are described below for single-GPU, distributed, and cluster training and testing, respectively. Through this tutorial, you will learn how to train and test using the scripts provided by MMSegmentation.

## Training and testing on a single GPU

### Training on a single GPU

```shell
python tools/train.py  ${CONFIG_FILE} [optional arguments]
```

- `--amp`: Use auto mixed precision training.
- `--resume`: Resume from the latest checkpoint in the work_dir automatically.
- `--cfg-options ${OVERRIDE_CONFIGS}`: Override some settings in the used config, and the key-value pair in xxx=yyy format will be merged into the config file.

  For example, '--cfg-option model.encoder.in_channels=6'. Please see this [guide](./1_config.md#Modify-config-through-script-arguments) for more details.

Below are the optional arguments for the multi-gpu test:

- `--launcher`: Items for distributed job initialization launcher. Allowed choices are `none`, `pytorch`, `slurm`, `mpi`. Especially, if set to none, it will test in a non-distributed mode.
- `--local_rank`: ID for local rank. If not specified, it will be set to 0.

**Note:** Difference between the argument `--resume` and the field `load_from` in the config file:

`--resume` only determines whether to resume from the latest checkpoint in the work_dir. It is usually used for resuming the training process that is interrupted accidentally.

`load_from` will specify the checkpoint to be loaded and the training iteration starts from 0. It is usually used for fine-tuning.

If you would like to resume training from a specific checkpoint, you can use:

```python
python tools/train.py ${CONFIG_FILE} --resume --cfg-options load_from=${CHECKPOINT}
```

**Training on CPU**: The process of training on the CPU is consistent with single GPU training if a machine does not have GPU. If it has GPUs but not wanting to use them, we just need to disable GPUs before the training process.

```shell
```

And then run the script [above](#training-on-a-single-gpu).

### Testing on a single GPU

We provide `tools/test.py` to launch training jobs on a single GPU.

```shell

```shell
```

- `--work-dir`: If specified, results will be saved in this directory. If not specified, the results will be automatically saved to `work_dirs/{CONFIG_NAME}`.
- `--show`: Show prediction results at runtime, available when `--show-dir` is not specified.
- `--show-dir`: Directory where painted images will be saved. If specified, the visualized segmentation mask will be saved to the `work_dir/timestamp/show_dir`.
- `--wait-time`: The interval of show (s), which takes effect when `--show` is activated. Default to 2.
- `--cfg-options`:  If specified, the key-value pair in xxx=yyy format will be merged into the config file.

- `--tta`: Test time augmentation option.

**Testing on CPU**: The process of testing on the CPU is consistent with single GPU testing if a machine does not have GPU. If it has GPUs but not wanting to use them, we just need to disable GPUs before the training process.

```shell
```

then run the script [above](#testing-on-a-single-gpu).

## Training and testing on multiple GPUs and multiple machines

### Training on multiple GPUs

OpenMMLab2.0 implements **distributed** training with `MMDistributedDataParallel`.

The basic usage is as follows:

```shell
sh tools/dist_train.sh ${CONFIG_FILE} ${GPU_NUM} [optional arguments]
```

Optional arguments remain the same as stated [above](#training-on-a-single-gpu) and have additional arguments to specify the number of GPUs.

An example:

```

```shell
# checkpoints and logs saved in WORK_DIR=work_dirs/pspnet_r50-d8_4xb4-80k_ade20k-512x512/
# If work_dir is not set, it will be generated automatically.
```

**Note**: During training, checkpoints and logs are saved in the same folder structure as the config file under `work_dirs/`. A custom work directory is not recommended since evaluation scripts infer work directories from the config file name. If you want to save your weights somewhere else, please use a symlink, for example:

```shell
ln -s ${YOUR_WORK_DIRS} ${MMSEG}/work_dirs
```

### Testing on multiple GPUs

We provide `tools/dist_test.sh` to launch testing on multiple GPUs.

```shell

```shell
```

```shell
sh tools/dist_test.sh ${CONFIG_FILE} ${CHECKPOINT_FILE} ${GPU_NUM} [optional arguments]
```

Optional arguments remain the same as stated [above](#testing-on-a-single-gpu) and have additional arguments to specify the number of GPUs.

```shell
```

### Launch multiple jobs on a single machine

If you launch multiple jobs on a single machine, e.g., 2 jobs of 4-GPU training on a machine with 8 GPUs, you need to specify different ports (29500 by default) for each job to avoid communication conflict. Otherwise, there will be an error message saying `RuntimeError: Address already in use`.

If you use `dist_train.sh` to launch training jobs, you can set the port in commands with the environment variable `PORT`.

```shell
CUDA_VISIBLE_DEVICES=0,1,2,3 PORT=29500 sh tools/dist_train.sh ${CONFIG_FILE} 4
CUDA_VISIBLE_DEVICES=4,5,6,7 PORT=29501 sh tools/dist_train.sh ${CONFIG_FILE} 4
```

MMSegmentation relies on `torch.distributed` package for distributed training.
Thus, as a basic usage, one can launch distributed training via PyTorch's [launch utility](https://pytorch.org/docs/stable/distributed.html#launch-utility).

```shell
NNODES=2 NODE_RANK=0 PORT=${MASTER_PORT} MASTER_ADDR=${MASTER_ADDR} sh tools/dist_train.sh ${CONFIG_FILE} ${GPUS}
```

```shell
NNODES=2 NODE_RANK=1 PORT=${MASTER_PORT} MASTER_ADDR=${MASTER_ADDR} sh tools/dist_train.sh ${CONFIG_FILE} ${GPUS}
```

```

```shell
```

### Training on a cluster with Slurm

```shell
[GPUS=${GPUS}] sh tools/slurm_train.sh ${PARTITION} ${JOB_NAME} ${CONFIG_FILE} [optional arguments]
```

Below is an example of using 4 GPUs to train PSPNet on a Slurm partition named _dev_, and set the work-dir to some shared file systems.

```shell
```

```shell
```

### Testing on a cluster with Slurm

Similar to the training task, MMSegmentation provides `slurm_test.sh` to launch testing jobs.

```shell
[GPUS=${GPUS}] sh tools/slurm_test.sh ${PARTITION} ${JOB_NAME} ${CONFIG_FILE} ${CHECKPOINT_FILE} [optional arguments]
```

You can check [the source code](../../../tools/slurm_test.sh) to review full arguments and environment variables.

**Note:** When using Slurm, the port option needs to be set in one of the following ways:

1. Set the port through `--cfg-options`. This is more recommended since it does not change the original configs.

   ```shell

   ```shell
   GPUS=4 GPUS_PER_NODE=4 sh tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config1.py ${WORK_DIR} --cfg-options env_cfg.dist_cfg.port=29500
   GPUS=4 GPUS_PER_NODE=4 sh tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config2.py ${WORK_DIR} --cfg-options env_cfg.dist_cfg.port=29501
   ```

   In `config1.py`:

   ```python
   ```

   In `config2.py`:

   ```python
   ```

   Then you can launch two jobs with config1.py and config2.py.

   ```shell
   CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 sh tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config1.py ${WORK_DIR}
   CUDA_VISIBLE_DEVICES=4,5,6,7 GPUS=4 sh tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config2.py ${WORK_DIR}
   ```

CUDA_VISIBLE_DEVICES=4,5,6,7 GPUS=4 sh tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config2.py ${WORK_DIR}
   ```

3. Set the port in the command using the environment variable 'MASTER_PORT':

```shell
CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 MASTER_PORT=29500 sh tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config1.py ${WORK_DIR}
CUDA_VISIBLE_DEVICES=4,5,6,7 GPUS=4 MASTER_PORT=29501 sh tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config2.py ${WORK_DIR}
```

## Testing and saving segment files

When you want to save the results, you can use `--out` to specify the output directory.

```shell
```

Here is an example to save the predicted results from model `fcn_r50-d8_4xb4-80k_ade20k-512x512` on ADE20k validatation dataset.

```shell

```shell
```

You also can modify the config file to define `output_dir`. We also take
`fcn_r50-d8_4xb4-80k_ade20k-512x512` as example just add
`test_evaluator` in `configs/fcn/fcn_r50-d8_4xb4-80k_ade20k-512x512.py`

```python
```

then run command without `--out`:

```shell
```

If you would like to only save the predicted results without evaluation as annotation is not released by the official dataset, you can set `format_only=True` and modify `test_dataloader`.

```

As there is no annotation in dataset, we remove `dict(type='LoadAnnotations')` from `test_dataloader` Here is the example configuration:

```python
        # we don't load annotation in test transform pipeline.

        ]))
```

then run test command:

```shell
```

### Testing Cityscape dataset and save predicted segment files

We recommend `CityscapesMetric` which is the wrapper of Cityscapes'sdk, when you want to
save the predicted results of Cityscape test dataset to submit them in [Cityscape test server](https://www.cityscapes-dataset.com/submit/). Here is the example configuration:

```python

output_dir='work_dirs/format_results')
        ]))
```

then run test command, for example:

```shell
```

# Top-down SimCC-based pose estimation

Top-down methods divide the task into two stages: object detection, followed by single-object pose estimation given object bounding boxes. At the 2nd stage, SimCC  based methods reformulate human pose estimation as two classification tasks for horizontal and vertical coordinates, and uniformly divide each pixel into several bins, thus obtain the keypoint coordinates given the features extracted from the bounding box area, following the paradigm introduced in [SimCC: a Simple Coordinate Classification Perspective for Human Pose Estimation](https://arxiv.org/abs/2107.03332).

</div>

### COCO Dataset

</div>

|             Model             | Input Size |  AP   |  AR   |               Details and Download                |
| :---------------------------: | :--------: | :---: | :---: | :-----------------------------------------------: |
|        ResNet-50+SimCC        |  384x288   | 0.735 | 0.790 |      [resnet_coco.md](./coco/resnet_coco.md)      |
|        ResNet-50+SimCC        |  256x192   | 0.721 | 0.781 |      [resnet_coco.md](./coco/resnet_coco.md)      |
|  S-ViPNAS-MobileNet-V3+SimCC  |  256x192   | 0.695 | 0.755 |      [vipnas_coco.md](./coco/vipnas_coco.md)      |
| MobileNet-V2+SimCC(wo/deconv) |  256x192   | 0.620 | 0.678 | [mobilenetv2_coco.md](./coco/mobilenetv2_coco.md) |

# RepVGG

> [RepVGG: Making VGG-style ConvNets Great Again](https://arxiv.org/abs/2101.03697)

RepVGG is a VGG-style convolutional architecture. It has the following advantages:

1. The model has a VGG-like plain (a.k.a. feed-forward) topology 1 without any branches. I.e., every layer takes the output of its only preceding layer as input and feeds the output into its only following layer.
2. The model’s body uses only 3 × 3 conv and ReLU.
3. The concrete architecture (including the specific depth and layer widths) is instantiated with no automatic search, manual refinement, compound scaling, nor other heavy designs.

<img src="https://user-images.githubusercontent.com/26739999/142573223-f7f14d32-ea08-43a1-81ad-5a6a83ee0122.png" width="60%"/>
</div>

<details>

<br>

</div>

<details>

<br>
We present a simple but powerful architecture of convolutional neural network, which has a VGG-like inference-time body composed of nothing but a stack of 3x3 convolution and ReLU, while the training-time model has a multi-branch topology. Such decoupling of the training-time and inference-time architecture is realized by a structural re-parameterization technique so that the model is named RepVGG. On ImageNet, RepVGG reaches over 80% top-1 accuracy, which is the first time for a plain model, to the best of our knowledge. On NVIDIA 1080Ti GPU, RepVGG models run 83% faster than ResNet-50 or 101% faster than ResNet-101 with higher accuracy and show favorable accuracy-speed trade-off compared to the state-of-the-art models like EfficientNet and RegNet.
</br>

```python
from mmpretrain import inference_model, get_model

</br>

```python

model = get_model('repvgg-A0_8xb32_in1k', pretrained=True)
model.backbone.switch_to_deploy()
predict = inference_model(model, 'demo/bird.JPEG')
```

```python

```

Train:

```shell
python tools/train.py configs/repvgg/repvgg-A0_8xb32_in1k.py
```

Test:

```shell

Train:

```shell
```

Test:

```shell
python tools/test.py configs/repvgg/repvgg-A0_8xb32_in1k.py https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-A0_8xb32_in1k_20221213-60ae8e23.pth
```

Test with reparameterized model:

```shell
python tools/test.py configs/repvgg/repvgg-A0_8xb32_in1k.py repvgg_A0_deploy.pth --cfg-options model.backbone.deploy=True
```

**Reparameterization**

The checkpoints provided are all `training-time` models. Use the reparameterize tool to switch them to more efficient `inference-time` architecture, which not only has fewer parameters but also less calculations.

```bash
```

`${CFG_PATH}` is the config file, `${SRC_CKPT_PATH}` is the source chenpoint file, `${TARGET_CKPT_PATH}` is the target deploy weight file path.

```

To use reparameterized weights, the config file must switch to the deploy config files.

```bash
python tools/test.py ${deploy_cfg} ${deploy_checkpoint} --metrics accuracy
```

You can also use `backbone.switch_to_deploy()` to switch to the deploy mode in Python code. For example:

```python
from mmpretrain.models import RepVGG

backbone = RepVGG(arch='A0')
backbone.switch_to_deploy()
```

| Model                         |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |               Config                |                                        Download                                         |

| :---------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :---------------------------------: | :-------------------------------------------------------------------------------------: |
| `repvgg-A0_8xb32_in1k`        | From scratch |    8.31    |   1.36    |   72.37   |   90.56   |  [config](repvgg-A0_8xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-A0_8xb32_in1k_20221213-60ae8e23.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-A0_8xb32_in1k_20221213-60ae8e23.log) |
| `repvgg-A1_8xb32_in1k`        | From scratch |   12.79    |   2.36    |   74.23   |   91.80   |  [config](repvgg-A1_8xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-A1_8xb32_in1k_20221213-f81bf3df.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-A1_8xb32_in1k_20221213-f81bf3df.log) |

| `repvgg-A2_8xb32_in1k`        | From scratch |   25.50    |   5.12    |   76.49   |   93.09   |  [config](repvgg-A2_8xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-A2_8xb32_in1k_20221213-a8767caf.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-A2_8xb32_in1k_20221213-a8767caf.log) |
| `repvgg-B0_8xb32_in1k`        | From scratch |    3.42    |   15.82   |   75.27   |   92.21   |  [config](repvgg-B0_8xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-B0_8xb32_in1k_20221213-5091ecc7.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-B0_8xb32_in1k_20221213-5091ecc7.log) |

| `repvgg-B1_8xb32_in1k`        | From scratch |   51.83    |   11.81   |   78.19   |   94.04   |  [config](repvgg-B1_8xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-B1_8xb32_in1k_20221213-d17c45e7.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-B1_8xb32_in1k_20221213-d17c45e7.log) |
| `repvgg-B1g2_8xb32_in1k`      | From scratch |   41.36    |   8.81    |   77.87   |   93.99   | [config](repvgg-B1g2_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-B1g2_8xb32_in1k_20221213-ae6428fd.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-B1g2_8xb32_in1k_20221213-ae6428fd.log) |

| `repvgg-B1g4_8xb32_in1k`      | From scratch |   36.13    |   7.30    |   77.81   |   93.77   | [config](repvgg-B1g4_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-B1g4_8xb32_in1k_20221213-a7a4aaea.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-B1g4_8xb32_in1k_20221213-a7a4aaea.log) |
| `repvgg-B2_8xb32_in1k`        | From scratch |   80.32    |   18.37   |   78.58   |   94.23   |  [config](repvgg-B2_8xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-B2_8xb32_in1k_20221213-d8b420ef.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-B2_8xb32_in1k_20221213-d8b420ef.log) |

| `repvgg-B2g4_8xb32_in1k`      | From scratch |   55.78    |   11.33   |   79.44   |   94.72   | [config](repvgg-B2g4_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-B2g4_8xb32_in1k_20221213-0c1990eb.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-B2g4_8xb32_in1k_20221213-0c1990eb.log) |
| `repvgg-B3_8xb32_in1k`        | From scratch |   110.96   |   26.21   |   80.58   |   95.33   |  [config](repvgg-B3_8xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-B3_8xb32_in1k_20221213-927a329a.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-B3_8xb32_in1k_20221213-927a329a.log) |

| `repvgg-B3g4_8xb32_in1k`      | From scratch |   75.63    |   16.06   |   80.26   |   95.15   | [config](repvgg-B3g4_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-B3g4_8xb32_in1k_20221213-e01cb280.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-B3g4_8xb32_in1k_20221213-e01cb280.log) |
| `repvgg-D2se_3rdparty_in1k`\* | From scratch |   120.39   |   32.84   |   81.81   |   95.94   | [config](repvgg-D2se_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/repvgg/repvgg-D2se_3rdparty_4xb64-autoaug-lbs-mixup-coslr-200e_in1k_20210909-cf3139b7.pth) |

*Models with * are converted from the [official repo](https://github.com/DingXiaoH/RepVGG/blob/9f272318abfc47a2b702cd0e916fca8d25d683e7/repvgg.py#L250). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@inproceedings{ding2021repvgg,

```bibtex
  title={Repvgg: Making vgg-style convnets great again},
  author={Ding, Xiaohan and Zhang, Xiangyu and Ma, Ningning and Han, Jungong and Ding, Guiguang and Sun, Jian},
  pages={13733--13742},
}
```

<details>

```bibtex
}
```

<details>

```bibtex

title = {2D Human Pose Estimation: New Benchmark and State of the Art Analysis},
}
```

| [pose_seresnet_50](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_seresnet50_8xb64-210e_mpii-256x256.py) |  256x256   | 0.884 |  0.292   | [ckpt](https://download.openmmlab.com/mmpose/top_down/seresnet/seresnet50_mpii_256x256-1bb21f79_20200927.pth) | [log](https://download.openmmlab.com/mmpose/top_down/seresnet/seresnet50_mpii_256x256_20200927.log.json) |
| [pose_seresnet_101](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_seresnet101_8xb64-210e_mpii-256x256.py) |  256x256   | 0.884 |  0.295   | [ckpt](https://download.openmmlab.com/mmpose/top_down/seresnet/seresnet101_mpii_256x256-0ba14ff5_20200927.pth) | [log](https://download.openmmlab.com/mmpose/top_down/seresnet/seresnet101_mpii_256x256_20200927.log.json) |

| [pose_seresnet_152\*](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_seresnet152_8xb32-210e_mpii-256x256.py) |  256x256   | 0.884 |  0.287   | [ckpt](https://download.openmmlab.com/mmpose/top_down/seresnet/seresnet152_mpii_256x256-6ea1e774_20200927.pth) | [log](https://download.openmmlab.com/mmpose/top_down/seresnet/seresnet152_mpii_256x256_20200927.log.json) |

Note that * means without imagenet pre-training.

# Migration of Schedule Settings

We update schedule settings in MMagic 1.x. Important modifications are as following.

- Now we use `optim_wrapper` field to specify all configuration about the optimization process. And the `optimizer` is a sub field of `optim_wrapper` now.
- The `lr_config` field is removed and we use new `param_scheduler` to replace it.
- The `total_iters` field is moved to `train_cfg` as `max_iters`, `val_cfg` and `test_cfg`, which configure the loop in training, validation and test.

<thead>
  <tr>
<tbody>
<tr>

```python
optimizers = dict(generator=dict(type='Adam', lr=1e-4, betas=(0.9, 0.999)))  # Config used to build optimizer, support all the optimizers in PyTorch whose arguments are also the same as those in PyTorch
total_iters = 300000 # Total training iters
lr_config = dict( # Learning rate scheduler config used to register LrUpdater hook

    policy='Step', by_epoch=False, step=[200000], gamma=0.5)  # The policy of scheduler
```

</td>

```python
    dict(
        type='OptimWrapper',
        optimizer=dict(type='Adam', lr=1e-4),
    )
)  # Config used to build optimizer, support all the optimizers in PyTorch whose arguments are also the same as those in PyTorch.
param_scheduler = dict(  # Config of learning policy
    type='MultiStepLR', by_epoch=False, milestones=[200000], gamma=0.5)  # The policy of scheduler
```

</td>

</tr>
</thead>
</table>

```

</td>

</tr>
</thead>
</table>

> More details of schedule settings are shown in [MMEngine Documents](https://github.com/open-mmlab/mmengine/blob/main/docs/en/migration/param_scheduler.md).

# Get started: Install and Run MMSeg

MMSegmentation works on Linux, Windows and macOS. It requires Python 3.7+, CUDA 10.2+ and PyTorch 1.8+.

**Note:**
If you are experienced with PyTorch and have already installed it, just skip this part and jump to the [next section](##installation). Otherwise, you can follow these steps for the preparation.

```shell
```

```shell
```

```shell
```

```shell
```

```shell
```

We recommend that users follow our best practices to install MMSegmentation. However, the whole process is highly customizable. See [Customize Installation](#customize-installation) section for more information.

```shell
```

**Step 1.** Install MMSegmentation.

Case a: If you develop and run mmseg directly, install it from source:

```shell
git clone -b main https://github.com/open-mmlab/mmsegmentation.git
cd mmsegmentation
# '-v' means verbose, or more output
# '-e' means installing a project in editable mode,

```

Case b: If you use mmsegmentation as a dependency or third-party package, install it with pip:

```shell
pip install "mmsegmentation>=1.0.0"
```

To verify whether MMSegmentation is installed correctly, we provide some sample codes to run an inference demo.

```shell
mim download mmsegmentation --config pspnet_r50-d8_4xb2-40k_cityscapes-512x1024 --dest .
```

```shell
```

The downloading will take several seconds or more, depending on your network environment. When it is done, you will find two files `pspnet_r50-d8_4xb2-40k_cityscapes-512x1024.py` and `pspnet_r50-d8_512x1024_40k_cityscapes_20200605_003338-2966598c.pth` in your current folder.

Option (a). If you install mmsegmentation from source, just run the following command.

```shell
python demo/image_demo.py demo/demo.png configs/pspnet/pspnet_r50-d8_4xb2-40k_cityscapes-512x1024.py pspnet_r50-d8_512x1024_40k_cityscapes_20200605_003338-2966598c.pth --device cuda:0 --out-file result.jpg
```

You will see a new image `result.jpg` on your current folder, where segmentation masks are covered on all objects.

```

Option (b). If you install mmsegmentation with pip, open you python interpreter and copy&paste the following codes.

```python
from mmseg.apis import inference_model, init_model, show_result_pyplot

config_file = 'pspnet_r50-d8_4xb2-40k_cityscapes-512x1024.py'
checkpoint_file = 'pspnet_r50-d8_512x1024_40k_cityscapes_20200605_003338-2966598c.pth'

model = init_model(config_file, checkpoint_file, device='cuda:0')

# test a single image and show the results
img = 'demo/demo.png'  # or img = mmcv.imread(img), which will only load it once
result = inference_model(model, img)
# visualize the results in a new window
show_result_pyplot(model, img, result, show=True)
# or save the visualization results to image files
# you can change the opacity of the painted segmentation map in (0, 1].

show_result_pyplot(model, img, result, show=True, out_file='result.jpg', opacity=0.5)
# test a video and show the results
video = mmcv.VideoReader('video.mp4')
for frame in video:
   result = inference_segmentor(model, frame)
   show_result_pyplot(model, result, wait_time=1)
```

You can modify the code above to test a single image or a video, both of these options can verify that the installation was successful.

**Note:**

For example, the following command install mmcv==2.0.0 built for PyTorch 1.10.x and CUDA 11.3.

```shell
pip install mmcv==2.0.0 -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10/index.html
```

MMSegmentation can be built for CPU only environment. In CPU mode you can train (requires MMCV version >= 2.0.0), test or inference a model.

thus we only need to install MMCV and MMSegmentation with the following commands.

**Step 1.** Install [MMCV](https://github.com/open-mmlab/mmcv) using [MIM](https://github.com/open-mmlab/mim).

```shell
!mim install "mmcv>=2.0.0"
```

**Step 2.** Install MMSegmentation from the source.

```shell
!git clone https://github.com/open-mmlab/mmsegmentation.git
%cd mmsegmentation
!git checkout main
```

```python
import mmseg
print(mmseg.__version__)
```

**Note:**

### Using MMSegmentation with Docker

We provide a [Dockerfile](https://github.com/open-mmlab/mmsegmentation/blob/main/docker/Dockerfile) to build an image. Ensure that your [docker version](https://docs.docker.com/engine/install/) >=19.03.

```shell
# build an image with PyTorch 1.11, CUDA 11.3
docker build -t mmsegmentation docker/
```

Run it with

```shell
docker run --gpus all --shm-size=8g -it -v {DATA_DIR}:/mmsegmentation/data mmsegmentation
```

### Optional Dependencies

#### Install GDAL

[GDAL](https://gdal.org/) is a translator library for raster and vector geospatial data formats. Install GDAL to read complex formats and extremely large remote sensing images.

```shell
conda install GDAL
```

```shell
```

You may [open an issue](https://github.com/open-mmlab/mmsegmentation/issues/new/choose) on GitHub if no solution is found.

# Visual Attention Network (VAN) for Segmentation

This repo is a PyTorch implementation of applying **VAN** (**Visual Attention Network**) to semantic segmentation.

The code is an integration from [VAN-Segmentation](https://github.com/Visual-Attention-Network/VAN-Segmentation/blob/main/README.md?plain=1)

More details can be found in [**Visual Attention Network**](https://arxiv.org/abs/2202.09741).

```bib
@article{guo2022visual,
  title={Visual Attention Network},
  author={Guo, Meng-Hao and Lu, Cheng-Ze and Liu, Zheng-Ning and Cheng, Ming-Ming and Hu, Shi-Min},
  journal={arXiv preprint arXiv:2202.09741},
}
```

## Results

**Notes**: Pre-trained models can be found in [TsingHua Cloud](https://cloud.tsinghua.edu.cn/d/0100f0cea37d41ba8d08/).

Results can be found in [VAN-Segmentation](https://github.com/Visual-Attention-Network/VAN-Segmentation/blob/main/README.md?plain=1)

We provide evaluation results of the converted weights.

| Method  |   Backbone   | mIoU  |                                                                    Download                                                                    |
| :-----: | :----------: | :---: | :--------------------------------------------------------------------------------------------------------------------------------------------: |
| UPerNet |    VAN-B2    | 49.35 | [model](https://download.openmmlab.com/mmsegmentation/v0.5/van_3rdparty/van-b2-in1kpre_upernet_3rdparty_512x512-ade20k_20230522-19c58aee.pth)  |
| UPerNet |    VAN-B3    | 49.71 | [model](https://download.openmmlab.com/mmsegmentation/v0.5/van_3rdparty/van-b3-in1kpre_upernet_3rdparty_512x512-ade20k_20230522-653bd6b7.pth)  |

| UPerNet |    VAN-B4    | 51.56 | [model](https://download.openmmlab.com/mmsegmentation/v0.5/van_3rdparty/van-b4-in1kpre_upernet_3rdparty_512x512-ade20k_20230522-653bd6b7.pth)  |
| UPerNet | VAN-B4-in22k | 52.61 | [model](https://download.openmmlab.com/mmsegmentation/v0.5/van_3rdparty/van-b4-in22kpre_upernet_3rdparty_512x512-ade20k_20230522-4a4d744a.pth) |
| UPerNet | VAN-B5-in22k | 53.11 | [model](https://download.openmmlab.com/mmsegmentation/v0.5/van_3rdparty/van-b5-in22kpre_upernet_3rdparty_512x512-ade20k_20230522-5bb6f2b4.pth) |
| UPerNet | VAN-B6-in22k | 54.25 | [model](https://download.openmmlab.com/mmsegmentation/v0.5/van_3rdparty/van-b6-in22kpre_upernet_3rdparty_512x512-ade20k_20230522-e226b363.pth) |

|   FPN   |    VAN-B0    | 38.65 |   [model](https://download.openmmlab.com/mmsegmentation/v0.5/van_3rdparty/van-b0-in1kpre_fpn_3rdparty_512x512-ade20k_20230522-75a76298.pth)    |
|   FPN   |    VAN-B1    | 43.22 |   [model](https://download.openmmlab.com/mmsegmentation/v0.5/van_3rdparty/van-b1-in1kpre_fpn_3rdparty_512x512-ade20k_20230522-104499ff.pth)    |
|   FPN   |    VAN-B2    | 46.84 |   [model](https://download.openmmlab.com/mmsegmentation/v0.5/van_3rdparty/van-b2-in1kpre_fpn_3rdparty_512x512-ade20k_20230522-7074e6f8.pth)    |
|   FPN   |    VAN-B3    | 48.32 |   [model](https://download.openmmlab.com/mmsegmentation/v0.5/van_3rdparty/van-b3-in1kpre_fpn_3rdparty_512x512-ade20k_20230522-2c3b7f5e.pth)    |

## Preparation

Install MMSegmentation and download ADE20K according to the guidelines in MMSegmentation.

## Requirement

```shell
```

```shell
```

```shell
```

If you use 4 GPUs for training by default. Run:

```bash

```shell
```

```bash
bash tools/dist_train.sh projects/van/configs/van/van-b2_pre1k_upernet_4xb2-160k_ade20k-512x512.py 4
```

To evaluate the model, an example is:

```bash
bash tools/dist_train.sh projects/van/configs/van/van-b2_pre1k_upernet_4xb2-160k_ade20k-512x512.py work_dirs/van-b2_pre1k_upernet_4xb2-160k_ade20k-512x512/iter_160000.pth 4 --eval mIoU
```

## FLOPs

To calculate FLOPs for a model, run:

```bash
bash tools/analysis_tools/get_flops.py projects/van/configs/van/van-b2_pre1k_upernet_4xb2-160k_ade20k-512x512.py --shape 512 512
```

## Acknowledgment

```bash
```

Our implementation is mainly based on [mmsegmentation](https://github.com/open-mmlab/mmsegmentation/tree/v0.12.0), [Swin-Transformer](https://github.com/SwinTransformer/Swin-Transformer-Semantic-Segmentation), [PoolFormer](https://github.com/sail-sg/poolformer), [Enjoy-Hamburger](https://github.com/Gsunshine/Enjoy-Hamburger) and [VAN-Segmentation](https://github.com/Visual-Attention-Network/VAN-Segmentation/blob/main/README.md?plain=1). Thanks for their authors.

## LICENSE

This repo is under the Apache-2.0 license. For commercial use, please contact the authors.

# Mapillary Vistas Dataset

Support **`Mapillary Vistas Dataset`**

This project implements **`Mapillary Vistas Dataset`**

Preparing `Mapillary Vistas Dataset` dataset following [Mapillary Vistas Dataset Preparing Guide](https://github.com/open-mmlab/mmsegmentation/tree/main/projects/mapillary_dataset/docs/en/user_guides/2_dataset_prepare.md)

```none
  ├── mmseg
  ├── tools
  ├── configs
  ├── data
  │   ├── mapillary
  │   │   ├── training
  │   │   │   ├── images
  │   │   │   ├── v1.2
  |   │   │   │   ├── instances
  |   │   │   │   ├── labels
  |   │   │   │   ├── labels_mask
  |   │   │   │   └── panoptic
  │   │   │   ├── v2.0
  |   │   │   │   ├── panoptic
  |   │   │   │   └── polygons
  │   │   ├── validation

|   │   │   │   ├── labels_mask
```

```bash
# Dataset train commands
# at `mmsegmentation` folder
bash tools/dist_train.sh projects/mapillary_dataset/configs/deeplabv3plus_r101-d8_4xb2-240k_mapillay_v1-512x1024.py 4
```

- [x] Milestone 3: Good to be a part of our core package!

  - [x] Type hints and docstrings

  - [x] Unit tests

  - [x] Code polishing

  - [x] Metafile.yml

- [x] Move your modules into the core package following the codebase's file hierarchy structure.

- [x] Refactor your modules into the core package following the codebase's file hierarchy structure.

# Evaluation

```{note}
Before reading this document, we recommend that you first read {external+mmengine:doc}`MMEngine: Model Accuracy Evaluation Basics <tutorials/evaluation>`.
```

## Metrics

MMOCR implements widely-used evaluation metrics for text detection, text recognition and key information extraction tasks based on the {external+mmengine:doc}`MMEngine: BaseMetric <design/evaluation>` base class. Users can specify the metric used in the validation and test phases by modifying the `val_evaluator` and `test_evaluator` fields in the configuration file. For example, the following config shows how to use `HmeanIOUMetric` to evaluate the model performance in text detection task.

```python

# In addition, MMOCR also supports the combined evaluation of multiple metrics for the same task, such as using WordMetric and CharMetric at the same time

    dict(type='WordMetric', mode=['exact', 'ignore_case', 'ignore_case_symbol']),
    dict(type='CharMetric')
]
```

```{tip}
More evaluation related configurations can be found in the [evaluation configuration tutorial](../user_guides/config.md#evaluation-configuration).
```

As shown in the following table, MMOCR currently supports 5 evaluation metrics for text detection, text recognition, and key information extraction tasks, including `HmeanIOUMetric`, `WordMetric`, `CharMetric`, `OneMinusNEDMetric`, and `F1Metric`.

|                                         |         |                                                   |                                                                       |

|                                         |         |                                                   |                                                                       |
| --------------------------------------- | ------- | ------------------------------------------------- | --------------------------------------------------------------------- |
| Metric                                  | Task    | Input Field                                       | Output Field                                                          |
| [HmeanIOUMetric](#hmeanioumetric)       | TextDet | `pred_polygons`<br>`pred_scores`<br>`gt_polygons` | `recall`<br>`precision`<br>`hmean`                                    |
| [WordMetric](#wordmetric)               | TextRec | `pred_text`<br>`gt_text`                          | `word_acc`<br>`word_acc_ignore_case`<br>`word_acc_ignore_case_symbol` |

| [CharMetric](#charmetric)               | TextRec | `pred_text`<br>`gt_text`                          | `char_recall`<br>`char_precision`                                     |
| [OneMinusNEDMetric](#oneminusnedmetric) | TextRec | `pred_text`<br>`gt_text`                          | `1-N.E.D`                                                             |
| [F1Metric](#f1metric)                   | KIE     | `pred_labels`<br>`gt_labels`                      | `macro_f1`<br>`micro_f1`                                              |

In general, the evaluation metric used in each task is conventionally determined. Users usually do not need to understand or manually modify the internal implementation of the evaluation metric. However, to facilitate more customized requirements, this document will further introduce the specific implementation details and configurable parameters of the built-in metrics in MMOCR.

### HmeanIOUMetric

[HmeanIOUMetric](mmocr.evaluation.metrics.hmean_iou_metric.HmeanIOUMetric) is one of the most widely used evaluation metrics in text detection tasks, because it calculates the harmonic mean (H-mean) between the detection precision (P) and recall rate (R). The `HmeanIOUMetric` can be calculated by the following equation:

```{math}
H = \frac{2}{\frac{1}{P} + \frac{1}{R}} = \frac{2PR}{P+R}
```

```{math}
```

In addition, since it is equivalent to the F-score (also known as F-measure or F-metric) when {math}`\beta = 1`, `HmeanIOUMetric` is sometimes written as `F1Metric` or `f1-score`:

```{math}
F_1=(1+\beta^2)\cdot\frac{PR}{\beta^2\cdot P+R} = \frac{2PR}{P+R}
```

In MMOCR, the calculation of `HmeanIOUMetric` can be summarized as the following steps:

1. Filter out invalid predictions

   - Filter out predictions with a score is lower than `pred_score_thrs`
   - Filter out predictions overlapping with `ignored` ground truth boxes with an overlap ratio higher than `ignore_precision_thr`

   It is worth noting that `pred_score_thrs` will **automatically search** for the **best threshold** within a certain range by default, and users can also customize the search range by manually modifying the configuration file:

   ```python

```python
   # By default, HmeanIOUMetric searches the best threshold within the range [0.3, 0.9] with a step size of 0.1
   val_evaluator = dict(type='HmeanIOUMetric', pred_score_thrs=dict(start=0.3, stop=0.9, step=0.1))
   ```

2. Calculate the IoU matrix

   - At the data processing stage, `HmeanIOUMetric` will calculate and maintain an {math}`M \times N` IoU matrix `iou_metric` for the convenience of the subsequent bounding box pairing step. Here, M and N represent the number of label bounding boxes and filtered prediction bounding boxes, respectively. Therefore, each element of this matrix stores the IoU between the m-th label bounding box and the n-th prediction bounding box.

3. Compute the number of GT samples that can be accurately matched based on the corresponding pairing strategy

   Although `HmeanIOUMetric` can be calculated by a fixed formula, there may still be some subtle differences in the specific implementations. These differences mainly reflect the use of different strategies to match gt and predicted bounding boxes, which leads to the difference in final scores. Currently, MMOCR supports two matching strategies, namely `vanilla` and `max_matching`, for the `HmeanIOUMetric`. As shown below, users can specify the matching strategies in the config.

   - `vanilla` matching strategy

     By default, `HmeanIOUMetric` adopts the `vanilla` matching strategy, which is consistent with the `hmean-iou` implementation in MMOCR 0.x and the **official** text detection competition evaluation standard of ICDAR series. The matching strategy adopts the first-come-first-served matching method to pair the labels and predictions.

     ```python

```python
     # By default, HmeanIOUMetric adopts 'vanilla' matching strategy
     val_evaluator = dict(type='HmeanIOUMetric')
     ```

   - `max_matching` matching strategy

     To address the shortcomings of the existing matching mechanism, MMOCR has implemented a more efficient matching strategy to maximize the number of matches.

     ```python
     # Specify to use 'max_matching' matching strategy
     val_evaluator = dict(type='HmeanIOUMetric', strategy='max_matching')
     ```

   ```{note}
   We recommend that research-oriented developers use the default `vanilla` matching strategy to ensure consistency with other papers. For industry-oriented developers, you can use the `max_matching` matching strategy to achieve optimized performance.
   ```

4. Compute the final evaluation score according to the aforementioned matching strategy

### WordMetric

```

[WordMetric](mmocr.evaluation.metrics.recog_metric.WordMetric) implements **word-level** text recognition evaluation metrics and includes three text matching modes, namely `exact`, `ignore_case`, and `ignore_case_symbol`. Users can freely combine the output of one or more text matching modes in the configuration file by modifying the `mode` field.

```python
# Use WordMetric for text recognition task
    dict(type='WordMetric', mode=['exact', 'ignore_case', 'ignore_case_symbol'])
]
```

- `exact`：Full matching mode, i.e., only when the predicted text and the ground truth text are exactly the same, the predicted text is considered to be correct.
- `ignore_case`：The mode ignores the case of the predicted text and the ground truth text.

- `ignore_case_symbol`：The mode ignores the case and symbols of the predicted text and the ground truth text. This is also the text recognition accuracy reported by most academic papers. The performance reported by MMOCR uses the `ignore_case_symbol` mode by default.

Assume that the real label is `MMOCR!` and the model output is `mmocr`. The `WordMetric` scores under the three matching modes are: `{'exact': 0, 'ignore_case': 0, 'ignore_case_symbol': 1}`.

### CharMetric

[CharMetric](mmocr.evaluation.metrics.recog_metric.CharMetric) implements **character-level** text recognition evaluation metrics that are **case-insensitive**.

```python
# Use CharMetric for text recognition task
val_evaluator = [dict(type='CharMetric')]
```

```python
```

Specifically, `CharMetric` will output two evaluation metrics, namely `char_precision` and `char_recall`. Let the number of correctly predicted characters (True Positive) be {math}`\sigma_{tp}`, then the precision *P* and recall *R* can be calculated by the following equation:

```{math}
P=\frac{\sigma_{tp}}{\sigma_{pred}}, R = \frac{\sigma_{tp}}{\sigma_{gt}}
```

where {math}`\sigma_{gt}` and {math}`\sigma_{pred}` represent the total number of characters in the label text and the predicted text, respectively.

For example, assume that the label text is "MM**O**CR" and the predicted text is "mm**0**cR**1**". The score of the `CharMetric` is:

```{math}
P=\frac{4}{6}, R=\frac{4}{5}
```

### OneMinusNEDMetric

```{math}
```

[OneMinusNEDMetric(1-N.E.D)](mmocr.evaluation.metrics.recog_metric.OneMinusNEDMetric) is commonly used for text recognition evaluation of Chinese or English **text line-level** annotations. Unlike the full matching metric that requires the prediction and the gt text to be exactly the same, `1-N.E.D` uses the normalized [edit distance](https://en.wikipedia.org/wiki/Edit_distance) (also known as Levenshtein Distance) to measure the difference between the predicted and the gt text, so that the performance difference of the model can be better distinguished when evaluating long texts. Assume that the real and predicted texts are {math}`s_i` and {math}`\hat{s_i}`, respectively, and their lengths are {math}`l_{i}` and {math}`\hat{l_i}`, respectively. The `OneMinusNEDMetric` score can be calculated by the following formula:

```{math}
score = 1 - \frac{1}{N}\sum_{i=1}^{N}\frac{D(s_i, \hat{s_{i}})}{max(l_{i},\hat{l_{i}})}
```

```{math}
```

where *N* is the total number of samples, and {math}`D(s_1, s_2)` is the edit distance between two strings.

For example, assume that the real label is "OpenMMLabMMOCR", the prediction of model A is "0penMMLabMMOCR", and the prediction of model B is "uvwxyz". The results of the full matching and `OneMinusNEDMetric` evaluation metrics are as follows:

|         |            |            |
| ------- | ---------- | ---------- |
|         | Full-match | 1 - N.E.D. |
| Model A | 0          | 0.92857    |
| Model B | 0          | 0          |

As shown in the table above, although the model A only predicted one letter incorrectly, both models got 0 in when using full-match strategy. However, the `OneMinusNEDMetric` evaluation metric can better distinguish the performance of the two models on **long texts**.

### F1Metric

[F1Metric](mmocr.evaluation.metrics.f_metric.F1Metric) implements the F1-Metric evaluation metric for KIE tasks and provides two modes, namely `micro` and `macro`.

```python
    dict(type='F1Metric', mode=['micro', 'macro'],
]
```

- `micro` mode: Calculate the global F1-Metric score based on the total number of True Positive, False Negative, and False Positive.

- `macro` mode：Calculate the F1-Metric score for each class and then take the average.

### Customized Metric

MMOCR supports the implementation of customized evaluation metrics for users who pursue higher customization. In general, users only need to create a customized evaluation metric class `CustomizedMetric` and inherit {external+mmengine:doc}`MMEngine: BaseMetric <design/evaluation>`. Then, the data format processing method `process` and the metric calculation method `compute_metrics` need to be overwritten respectively. Finally, add it to the `METRICS` registry to implement any customized evaluation metric.

```python
from mmocr.registry import METRICS

class CustomizedMetric(BaseMetric):

    def process(self, data_batch: Sequence[Dict], predictions: Sequence[Dict]):
    """ process receives two parameters, data_batch stores the gt label information, and predictions stores the predicted results.
    """

""" process receives two parameters, data_batch stores the gt label information, and predictions stores the predicted results.
    """
        pass

    """ compute_metric receives the results of the process method as input and returns the evaluation results.
    """
        pass
```

```{note}
More details can be found in {external+mmengine:doc}`MMEngine Documentation: BaseMetric <design/evaluation>`.
```

# ViPNAS: Efficient Video Pose Estimation via Neural Architecture Search

<details>

```bibtex
}
```

Human pose estimation has achieved significant progress in recent years. However, most of the recent methods focus on improving accuracy using complicated models and ignoring real-time efficiency. To achieve a better trade-off between accuracy and efficiency, we propose a novel neural architecture search (NAS) method, termed ViPNAS, to search networks in both spatial and temporal levels for fast online video pose estimation. In the spatial level, we carefully design the search space with five different dimensions including network depth, width, kernel size, group number, and attentions. In the temporal level, we search from a series of temporal feature fusions to optimize the total accuracy and speed across multiple video frames. To the best of our knowledge, we are the first to search for the temporal feature fusion and automatic computation allocation in videos. Extensive experiments demonstrate the effectiveness of our approach on the challenging COCO2017 and PoseTrack2018 datasets. Our discovered model family, S-ViPNAS and T-ViPNAS, achieve significantly higher inference speed (CPU real-time) without sacrificing the accuracy compared to the previous state-of-the-art methods.

<img src="https://user-images.githubusercontent.com/15977946/146527709-b8e55e40-426c-4958-8a35-b8d259e69efa.png">
</div>

---
name: "\U0001F41B bug issue"
about: submit a bug report +_+
---

## context | 编译/运行环境 | バックグラウンド

1.
2.
3.

## more | 其他 | その他

# AOT-GAN (TVCG'2021)

> [AOT-GAN: Aggregated Contextual Transformations for High-Resolution Image Inpainting](https://arxiv.org/abs/2104.01431.pdf)

## 摘要

 <img src="https://user-images.githubusercontent.com/12756472/169230414-3ca7fb6b-cf2a-401f-8696-71df75a08c32.png"/>
</div >

## 结果与模型

|                        算法                        |      掩膜类型      | 分辨率  | 训练集容量 |    测试集     | l1 损失 | PSNR  | SSIM  |        GPU 信息         |                              下载                              |
| :------------------------------------------------: | :----------------: | :-----: | :--------: | :-----------: | :-----: | :---: | :---: | :---------------------: | :------------------------------------------------------------: |

| [AOT-GAN](./aot-gan_smpgan_4xb4_places-512x512.py) | free-form (50-60%) | 512x512 |    500k    | Places365-val |  7.07   | 19.01 | 0.682 | 4 (GeForce GTX 1080 Ti) | [模型](https://download.openmmlab.com/mmediting/inpainting/aot_gan/AOT-GAN_512x512_4x12_places_20220509-6641441b.pth) \| [日志](https://download.openmmlab.com/mmediting/inpainting/aot_gan/AOT-GAN_512x512_4x12_places_20220509-6641441b.json) |

| 评估指标        | 掩膜缺损 | 论文结果 | 复现结果 |
| :-------------- | :------- | :------- | :------- |
| L1 (10^-2)      | 1 – 10%  | 0.55     | 0.54     |
| (lower better)  | 10 – 20% | 1.19     | 1.47     |
|                 | 20 – 30% | 2.11     | 2.79     |
|                 | 30 – 40% | 3.20     | 4.38     |
|                 | 40 – 50% | 4.51     | 6.28     |
|                 | 50 – 60% | 7.07     | 10.16    |
| PSNR            | 1 – 10%  | 34.79    | inf      |
| (higher better) | 10 – 20% | 29.49    | 31.22    |
|                 | 20 – 30% | 26.03    | 27.65    |

|                 | 30 – 40% | 23.58    | 25.06    |
|                 | 40 – 50% | 21.65    | 23.01    |
|                 | 50 – 60% | 19.01    | 20.05    |
| SSIM            | 1 – 10%  | 0.976    | 0.982    |
| (higher better) | 10 – 20% | 0.940    | 0.951    |
|                 | 20 – 30% | 0.890    | 0.911    |
|                 | 30 – 40% | 0.835    | 0.866    |
|                 | 40 – 50% | 0.773    | 0.815    |
|                 | 50 – 60% | 0.682    | 0.739    |

## 快速开始

**训练**

<details>

```shell
# CPU上训练
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/aot_gan/aot-gan_smpgan_4xb4_places-512x512.py

# 单个GPU上训练
python tools/train.py configs/aot_gan/aot-gan_smpgan_4xb4_places-512x512.py

# 多个GPU上训练

# 单个GPU上训练

# 多个GPU上训练
./tools/dist_train.sh configs/aot_gan/aot-gan_smpgan_4xb4_places-512x512.py 8
```

**测试**

<details>

```shell
# CPU上测试
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/aot_gan/aot-gan_smpgan_4xb4_places-512x512.py https://download.openmmlab.com/mmediting/inpainting/aot_gan/AOT-GAN_512x512_4x12_places_20220509-6641441b.pth

# 单个GPU上测试
python tools/test.py configs/aot_gan/aot-gan_smpgan_4xb4_places-512x512.py https://download.openmmlab.com/mmediting/inpainting/aot_gan/AOT-GAN_512x512_4x12_places_20220509-6641441b.pth

# 多个GPU上测试

# 多个GPU上测试
./tools/dist_test.sh configs/aot_gan/aot-gan_smpgan_4xb4_places-512x512.py https://download.openmmlab.com/mmediting/inpainting/aot_gan/AOT-GAN_512x512_4x12_places_20220509-6641441b.pth 8
```

## 引用

```bibtex
@inproceedings{yan2021agg,
  author = {Zeng, Yanhong and Fu, Jianlong and Chao, Hongyang and Guo, Baining},
  title = {Aggregated Contextual Transformations for High-Resolution Image Inpainting},
  booktitle = {Arxiv},
  pages={-},
}
```

本系列文档我们会来介绍如何使用提供了的脚本进行完成基本的推理 demo ，本节先介绍如何对 top-down 结构和动物的 2D 姿态进行单张图片和视频推理，请确保你已经安装了 3.0 以上版本的 [MMDetection](https://github.com/open-mmlab/mmdetection) 。

### 2D 动物图片姿态识别推理

```shell
```

用户可以在 [model zoo](https://mmpose.readthedocs.io/zh_CN/dev-1.x/model_zoo/animal_2d_keypoint.html) 获取预训练好的关键点识别模型。

这里我们用 [animalpose model](https://download.openmmlab.com/mmpose/animal/hrnet/hrnet_w32_animalpose_256x256-1aa7f075_20210426.pth) 来进行演示：

```shell

```shell
```

可视化结果如下：

`--det-cat-id=15` 参数用来指定模型只检测 `cat` 类型，这是基于 COCO 数据集的数据。

**COCO 数据集动物信息**

COCO 数据集共包含 80 个类别，其中有 10 种常见动物，类别如下：

(14: 'bird', 15: 'cat', 16: 'dog', 17: 'horse', 18: 'sheep', 19: 'cow', 20: 'elephant', 21: 'bear', 22: 'zebra', 23: 'giraffe')

对于其他类型的动物，我们也提供了一些训练好的动物检测模型，用户可以前往 [detection model zoo](/demo/docs/zh_cn/mmdet_modelzoo.md) 下载。

```shell

https://download.openmmlab.com/mmpose/animal/hrnet/hrnet_w32_animalpose_256x256-1aa7f075_20210426.pth \
```

如果想本地保存预测结果，需要使用 `--save-predictions` 。

```shell
```

仅使用 CPU：

```shell

--output-root vis_results --save-predictions --draw-heatmap --det-cat-id=15
```

仅使用 CPU：

```shell
```

### 2D 动物视频姿态识别推理

例如：

```shell

例如：

```shell
```

这段视频可以在 [Google Drive](https://drive.google.com/file/d/18d8K3wuUpKiDFHvOx0mh1TEwYwpOc5UO/view?usp=sharing) 下载。

### 使用 Inferencer 进行 2D 动物姿态识别推理

```shell
```

该命令会对输入的 `tests/data/ap10k` 下所有的图片进行推理并且把可视化结果都存入 `vis_results/ap10k` 文件夹下。

Inferencer 同样支持保存预测结果，更多的信息可以参考 [Inferencer 文档](https://mmpose.readthedocs.io/en/dev-1.x/user_guides/inference.html#inferencer-a-unified-inference-interface) 。

### 加速推理

用户可以通过修改配置文件来加速，更多具体例子可以参考：

### 加速推理

1. 设置 `model.test_cfg.flip_test=False`，如 [animalpose_hrnet-w32](../../configs/animal_2d_keypoint/topdown_heatmap/animalpose/td-hm_hrnet-w32_8xb64-210e_animalpose-256x256.py#85) 所示。
2. 使用更快的 bounding box 检测器，可参考 [MMDetection](https://mmdetection.readthedocs.io/zh_CN/3.x/model_zoo.html) 。

# Conformer

> [Conformer: Local Features Coupling Global Representations for Visual Recognition](https://arxiv.org/abs/2105.03889)

Within Convolutional Neural Network (CNN), the convolution operations are good at extracting local features but experience difficulty to capture global representations. Within visual transformer, the cascaded self-attention modules can capture long-distance feature dependencies but unfortunately deteriorate local feature details. In this paper, we propose a hybrid network structure, termed Conformer, to take advantage of convolutional operations and self-attention mechanisms for enhanced representation learning. Conformer roots in the Feature Coupling Unit (FCU), which fuses local features and global representations under different resolutions in an interactive fashion. Conformer adopts a concurrent structure so that local features and global representations are retained to the maximum extent. Experiments show that Conformer, under the comparable parameter complexity, outperforms the visual transformer (DeiT-B) by 2.3% on ImageNet. On MSCOCO, it outperforms ResNet-101 by 3.7% and 3.6% mAPs for object detection and instance segmentation, respectively, demonstrating the great potential to be a general backbone network.

<img src="https://user-images.githubusercontent.com/26739999/144957687-926390ed-6119-4e4c-beaa-9bc0017fe953.png" width="90%"/>
</div>

```python

predict = inference_model('conformer-tiny-p16_3rdparty_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('conformer-tiny-p16_3rdparty_in1k', pretrained=True)
```

Train:

```shell
python tools/train.py configs/conformer/conformer-small-p32_8xb128_in1k.py
```

Test:

```shell

Train:

```shell
```

Test:

```shell
python tools/test.py configs/conformer/conformer-tiny-p16_8xb128_in1k.py https://download.openmmlab.com/mmclassification/v0/conformer/conformer-tiny-p16_3rdparty_8xb128_in1k_20211206-f6860372.pth
```

| Model                                 |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                    Config                    |                                Download                                |
| :------------------------------------ | :----------: | :--------: | :-------: | :-------: | :-------: | :------------------------------------------: | :--------------------------------------------------------------------: |

| `conformer-tiny-p16_3rdparty_in1k`\*  | From scratch |   23.52    |   4.90    |   81.31   |   95.60   | [config](conformer-tiny-p16_8xb128_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/conformer/conformer-tiny-p16_3rdparty_8xb128_in1k_20211206-f6860372.pth) |
| `conformer-small-p16_3rdparty_in1k`\* | From scratch |   37.67    |   10.31   |   83.32   |   96.46   | [config](conformer-small-p16_8xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/conformer/conformer-small-p16_3rdparty_8xb128_in1k_20211206-3065dcf5.pth) |
| `conformer-small-p32_8xb128_in1k`     | From scratch |   38.85    |   7.09    |   81.96   |   96.02   | [config](conformer-small-p32_8xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/conformer/conformer-small-p32_8xb128_in1k_20211206-947a0816.pth) |

| `conformer-base-p16_3rdparty_in1k`\*  | From scratch |   83.29    |   22.89   |   83.82   |   96.59   | [config](conformer-base-p16_8xb128_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/conformer/conformer-base-p16_3rdparty_8xb128_in1k_20211206-bfdf8637.pth) |

*Models with * are converted from the [official repo](https://github.com/pengzhiliang/Conformer/blob/main/models.py#L89). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@article{peng2021conformer,
      title={Conformer: Local Features Coupling Global Representations for Visual Recognition},
      author={Zhiliang Peng and Wei Huang and Shanzhi Gu and Lingxi Xie and Yaowei Wang and Jianbin Jiao and Qixiang Ye},
      journal={arXiv preprint arXiv:2105.03889},
}
```

# How to convert model

This tutorial briefly introduces how to export an OpenMMlab model to a specific backend using MMDeploy tools.
Notes:

- Supported backends are [ONNXRuntime](../05-supported-backends/onnxruntime.md), [TensorRT](../05-supported-backends/tensorrt.md), [ncnn](../05-supported-backends/ncnn.md), [PPLNN](../05-supported-backends/pplnn.md), [OpenVINO](../05-supported-backends/openvino.md).
- Supported codebases are [MMPretrain](../04-supported-codebases/mmpretrain.md), [MMDetection](../04-supported-codebases/mmdet.md), [MMSegmentation](../04-supported-codebases/mmseg.md), [MMOCR](../04-supported-codebases/mmocr.md), [MMagic](../04-supported-codebases/mmagic.md).

## How to convert models from Pytorch to other backends

1. Install and build your target backend. You could refer to [ONNXRuntime-install](../05-supported-backends/onnxruntime.md), [TensorRT-install](../05-supported-backends/tensorrt.md), [ncnn-install](../05-supported-backends/ncnn.md), [PPLNN-install](../05-supported-backends/pplnn.md), [OpenVINO-install](../05-supported-backends/openvino.md) for more information.
2. Install and build your target codebase. You could refer to [MMPretrain-install](https://mmpretrain.readthedocs.io/en/latest/get_started.html#installation), [MMDetection-install](https://mmdetection.readthedocs.io/en/latest/get_started.html#installation), [MMSegmentation-install](https://mmsegmentation.readthedocs.io/en/latest/get_started.html#installation), [MMOCR-install](https://mmocr.readthedocs.io/en/latest/get_started/install.html#installation-steps), [MMagic-install](https://mmagic.readthedocs.io/en/latest/get_started/install.html#installation).

### Usage

```bash
    --show \
```

- `deploy_cfg` : The deployment configuration of mmdeploy for the model, including the type of inference framework, whether quantize, whether the input shape is dynamic, etc. There may be a reference relationship between configuration files, `mmdeploy/mmpretrain/classification_ncnn_static.py` is an example.
- `model_cfg` : Model configuration for algorithm library, e.g. `mmpretrain/configs/vision_transformer/vit-base-p32_ft-64xb64_in1k-384.py`, regardless of the path to mmdeploy.
- `checkpoint` : torch model path. It can start with http/https, see the implementation of `mmcv.FileClient` for details.

- `img` : The path to the image or point cloud file used for testing during the model conversion.
- `--test-img` : The path of the image file that is used to test the model. If not specified, it will be set to `None`.
- `--work-dir` : The path of the work directory that is used to save logs and models.
- `--calib-dataset-cfg` : Only valid in int8 mode. The config used for calibration. If not specified, it will be set to `None` and use the "val" dataset in the model config for calibration.
- `--device` : The device used for model conversion. If not specified, it will be set to `cpu`. For trt, use `cuda:0` format.
- `--show` : Whether to show detection outputs.

- `--dump-info` : Whether to output information for SDK.

### How to find the corresponding deployment config of a PyTorch model

1. Find the model's codebase folder in `configs/`.  For converting a yolov3 model, you need to check `configs/mmdet` folder.
2. Find the model's task folder in `configs/codebase_folder/`. For a yolov3 model, you need to check `configs/mmdet/detection` folder.
3. Find the deployment config file in `configs/codebase_folder/task_folder/`. For deploying a yolov3 model to the onnx backend, you could use `configs/mmdet/detection/detection_onnxruntime_dynamic.py`.

### Example

```bash

$PATH_TO_MMDET/checkpoints/yolo/yolov3_d53_mstrain-608_273e_coco_20210518_115020-a2c3acb8.pth \
    --show \
```

## How to evaluate the exported models

You can try to evaluate model, referring to [how_to_evaluate_a_model](profile_model.md).

## List of supported models exportable to other backends

Refer to [Support model list](../03-benchmark/supported_models.md)

# 如何写模型转换配置

<!-- This tutorial describes how to write a config for model conversion and deployment. A deployment config includes `onnx config`, `codebase config`, `backend config`. -->

这篇教程介绍了如何编写模型转换和部署的配置文件。部署配置文件由`ONNX配置`， `代码库配置`， `推理框架配置`组成。

- [如何写模型转换配置](#如何写模型转换配置)
  - [1. 如何编写ONNX配置](#1-如何编写onnx配置)
    - [ONNX配置参数说明](#onnx配置参数说明)
      - [示例](#示例)
    - [动态尺寸输入和输出配置](#动态尺寸输入和输出配置)
      - [示例](#示例-1)
  - [2. 如何编写代码库配置](#2-如何编写代码库配置)
    - [代码库配置参数说明](#代码库配置参数说明)
      - [示例](#示例-2)
  - [3. 如何编写推理框架配置](#3-如何编写推理框架配置)
    - [示例](#示例-3)
  - [4. 部署配置完整示例](#4-部署配置完整示例)
  - [5. 部署配置文件命名规则](#5-部署配置文件命名规则)
    - [示例](#示例-4)
  - [6. 如何编写模型配置文件](#6-如何编写模型配置文件)

## 1. 如何编写ONNX配置

ONNX 配置描述了如何将PyTorch模型转换为ONNX模型。

### ONNX配置参数说明

- `type`: 配置类型。 默认为 `onnx`。

- `export_params`: 如果指定，将导出模型所有参数。如果您只想导出未训练模型将此项设置为 False。

- `keep_initializers_as_inputs`:
  如果为 True，则所有初始化器（通常对应为参数）也将作为输入导出，添加到计算图中。 如果为 False，则初始化器不会作为输入导出，不添加到计算图中，仅将非参数输入添加到计算图中。

- `opset_version`: ONNX的算子集版本，默认为11。

- `save_file`: 输出ONNX模型文件。

- `input_names`: 模型计算图中输入节点的名称。

- `output_names`: 模型计算图中输出节点的名称。

- `input_shape`: 模型输入张量的高度和宽度。

#### 示例

```python
onnx_config = dict(
    type='onnx',
    export_params=True,
    keep_initializers_as_inputs=False,
    opset_version=11,
    save_file='end2end.onnx',
    input_names=['input'],
    output_names=['output'],
    input_shape=None)
```

### 动态尺寸输入和输出配置

如果模型要求动态尺寸的输入和输出，您需要在ONNX配置中加入dynamic_axes配置。

- `dynamic_axes`: 描述输入和输出的维度信息。

#### 示例

```python
    dynamic_axes={
        'input': {
            0: 'batch',
            2: 'height',
            3: 'width'
        },
        'dets': {
            1: 'num_dets',
        },
        'labels': {

3: 'width'
        },
        'dets': {
        },
        },
    }
```

## 2. 如何编写代码库配置

代码库主要指OpenMMLab 系列模型代码库，代码库配置由OpenMMLab 系列模型代码库的简称和OpenMMLab 系列模型任务类型组成。

### 代码库配置参数说明

- `type`: OpenMMLab 系列模型代码库的简称， 包括 `mmpretrain`， `mmdet`， `mmseg`， `mmocr`， `mmagic`。
- `task`: OpenMMLab 系列模型任务类型， 具体请参考 [OpenMMLab 系列模型任务列表](#list-of-tasks-in-all-codebases)。

#### 示例

```python
codebase_config = dict(type='mmpretrain', task='Classification')
```

## 3. 如何编写推理框架配置

推理框架配置主要用于指定模型运行在哪个推理框架，并提供模型在推理框架运行时所需的信息，具体参考 [ONNX Runtime](../05-supported-backends/onnxruntime.md)， [TensorRT](../05-supported-backends/tensorrt.md)， [ncnn](../05-supported-backends/ncnn.md)， [PPLNN](../05-supported-backends/pplnn.md)。

- `type`: 模型推理框架, 包括 `onnxruntime`， `ncnn`， `pplnn`， `tensorrt`， `openvino`。

#### 示例

```python

#### 示例

```python
        fp16_mode=False, max_workspace_size=1 << 30),
        dict(
                    min_shape=[1, 3, 512, 1024],
                    opt_shape=[1, 3, 1024, 2048],
                    max_shape=[1, 3, 2048, 2048])))
    ])
```

## 4. 部署配置完整示例

这里我们提供了一个以TensorRT为推理框架的基于mmpretrain图像分类任务的完整部署配置示例。

```python

        fp16_mode=False,
        max_workspace_size=1 << 30),
        dict(
                    min_shape=[1, 3, 224, 224],
                    opt_shape=[4, 3, 224, 224],
                    max_shape=[64, 3, 224, 224])))])

input=dict(

        'input': {
            3: 'width'
        },
        'output': {
            0: 'batch'
        }
    },
    input_shape=[224, 224])
```

## 5. 部署配置文件命名规则

```bash
(task name)_(backend name)_(dynamic or static).py
```

- `task name`: 模型任务类型。
- `backend name`: 推理框架名称。注意：如果您使用了量化，您需要指出量化类型。例如  `tensorrt-int8`。
- `dynamic or static`: 动态或者静态尺寸导出。 注意：如果推理框架需要明确的形状信息，您需要添加输入大小的描述，格式为`高度 x 宽度`。 例如 `dynamic-512x1024-2048x2048`, 这意味着最小输入形状是`512x1024`，最大输入形状是`2048x2048`。

#### 示例

```bash

#### 示例

```bash
detection_tensorrt-int8_dynamic-320x320-1344x1344.py
```

## 6. 如何编写模型配置文件

请根据模型具体任务的代码库，编写模型配置文件。 模型配置文件用于初始化模型，详情请参考[MMPretrain](https://github.com/open-mmlab/mmpretrain/blob/main/docs/zh_CN/user_guides/config.md)，[MMDetection](https://github.com/open-mmlab/mmdetection/blob/3.x/docs/zh_cn/user_guides/config.md)， [MMSegmentation](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/zh_cn/user_guides/1_config.md)， [MMOCR](https://github.com/open-mmlab/mmocr/blob/main/docs/en/user_guides/config.md)，[MMagic](https://github.com/open-mmlab/mmagic/blob/main/docs/en/user_guides/config.md)。

---
name: Reimplementation Questions
about: Ask about questions during model reimplementation
title: ''
labels: reimplementation
---

**Notice**

There are several common situations in the reimplementation issues as below

1. Reimplement a model in the model zoo using the provided configs
2. Reimplement a model in the model zoo on other dataset (e.g., custom datasets)
3. Reimplement a custom model but all the components are implemented in MMDetection
4. Reimplement a custom model with new modules implemented by yourself

There are several things to do for different cases as below.

- For case 1 & 3, please follow the steps in the following sections thus we could help to quick identify the issue.
- For case 2 & 4, please understand that we are not able to do much help here because we usually do not know the full code and the users should be responsible to the code they write.

- One suggestion for case 2 & 4 is that the users should first check whether the bug lies in the self-implemented code or the original code. For example, users can first make sure that the same model runs well on supported datasets. If you still need help, please describe what you have done and what you obtain in the issue, and follow the steps in the following sections and try as clear as possible so that we can better help you.

**Checklist**

1. I have searched related issues but cannot get the expected help.
2. The issue has not been fixed in the latest version.

**Describe the issue**

A clear and concise description of what the problem you meet and what have you done.

**Reproduction**

1. What command or script did you run?

```none
A placeholder for the command.
```

2. What config dir you run?

```none
```

```none
A placeholder for the config.
```

3. Did you make any modifications on the code or config? Did you understand what you have modified?
4. What dataset did you use?

**Environment**

1. Please run `python mmdet/utils/collect_env.py` to collect necessary environment information and paste it here.
2. You may add addition that may be helpful for locating the problem, such as
   1. How you installed PyTorch \[e.g., pip, conda, source\]
   2. Other environment variables that may be related (such as `$PATH`, `$LD_LIBRARY_PATH`, `$PYTHONPATH`, etc.)

**Results**

If applicable, paste the related results here, e.g., what you expect and what you get.

```none
A placeholder for results comparison
```

**Issue fix**

```none
```

If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!

The squeezenet android example project has been moved to https://github.com/nihui/ncnn-android-squeezenet

# Projects based on MMPretrain

There are many projects built upon MMPretrain(MMClassification previsously).
We list some of them as examples of how to extend MMPretrain(MMClassification previsously) for your own projects.

- [OpenMixup](https://github.com/Westlake-AI/openmixup): an open-source toolbox for supervised, self-, and semi-supervised visual representation learning with mixup based on PyTorch, especially for mixup-related methods.
- [AI Power](https://github.com/ykk648/AI_power): AI toolbox and pretrain models.
- [OpenBioSeq](https://github.com/Westlake-AI/OpenBioSeq): an open-source supervised and self-supervised bio-sequence representation learning toolbox based on PyTorch.

To make this list also a reference for the community to develop and compare new image classification algorithms, we list them following the time order of top-tier conferences.
Methods already supported and maintained by MMPretrain(MMClassification previsously) are not listed.

- Involution: Inverting the Inherence of Convolution for Visual Recognition, CVPR21. [[paper]](https://arxiv.org/abs/2103.06255)[[github]](https://github.com/d-li14/involution)
- Convolution of Convolution: Let Kernels Spatially Collaborate, CVPR22. [[paper]](https://openaccess.thecvf.com/content/CVPR2022/papers/Zhao_Convolution_of_Convolution_Let_Kernels_Spatially_Collaborate_CVPR_2022_paper.pdf)[[github]](https://github.com/Genera1Z/ConvolutionOfConvolution)

# UniFormer

[UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning](https://arxiv.org/abs/2201.04676)

It is a challenging task to learn rich and multi-scale spatiotemporal semantics from high-dimensional videos, due to large local redundancy and complex global dependency between video frames. The recent advances in this research have been mainly driven by 3D convolutional neural networks and vision transformers. Although 3D convolution can efficiently aggregate local context to suppress local redundancy from a small 3D neighborhood, it lacks the capability to capture global dependency because of the limited receptive field. Alternatively, vision transformers can effectively capture long-range dependency by self-attention mechanism, while having the limitation on reducing local redundancy with blind similarity comparison among all the tokens in each layer. Based on these observations, we propose a novel Unified transFormer (UniFormer) which seamlessly integrates merits of 3D convolution and spatiotemporal self-attention in a concise transformer format, and achieves a preferable balance between computation and accuracy. Different from traditional transformers, our relation aggregator can tackle both spatiotemporal redundancy and dependency, by learning local and global token affinity respectively in shallow and deep layers. We conduct extensive experiments on the popular video benchmarks, e.g., Kinetics-400, Kinetics-600, and Something-Something V1&V2. With only ImageNet-1K pretraining, our UniFormer achieves 82.9%/84.8% top-1 accuracy on Kinetics-400/Kinetics-600, while requiring 10x fewer GFLOPs than other state-of-the-art methods. For Something-Something V1 and V2, our UniFormer achieves new state-of-the-art performances of 60.9% and 71.2% top-1 accuracy respectively.

<img src="https://raw.githubusercontent.com/Sense-X/UniFormer/main/figures/framework.png"/>
</div>

| frame sampling strategy |   resolution   |  backbone   | top1 acc | top5 acc | [reference](https://github.com/Sense-X/UniFormer/blob/main/video_classification/README.md) top1 acc | [reference](https://github.com/Sense-X/UniFormer/blob/main/video_classification/README.md) top5 acc | mm-Kinetics top1 acc | mm-Kinetics top5 acc | testing protocol | FLOPs | params |                                              config                                               |                                                                           ckpt                                                                           |

|         16x4x1          | short-side 320 | UniFormer-S |   80.9   |   94.6   |                                                80.8                                                 |                                                94.7                                                 |         80.9         |         94.6         | 4 clips x 1 crop | 41.8G | 21.4M  | [config](/configs/recognition/uniformer/uniformer-small_imagenet1k-pre_16x4x1_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv1/uniformer-small_imagenet1k-pre_16x4x1_kinetics400-rgb_20221219-c630a037.pth) |

|         16x4x1          | short-side 320 | UniFormer-B |   82.0   |   95.0   |                                                82.0                                                 |                                                95.1                                                 |         82.0         |         95.0         | 4 clips x 1 crop | 96.7G | 49.8M  | [config](/configs/recognition/uniformer/uniformer-base_imagenet1k-pre_16x4x1_kinetics400-rgb.py)  | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv1/uniformer-base_imagenet1k-pre_16x4x1_kinetics400-rgb_20221219-157c2e66.pth)  |

|         32x4x1          | short-side 320 | UniFormer-B |   83.1   |   95.3   |                                                82.9                                                 |                                                95.4                                                 |         83.0         |         95.3         | 4 clips x 1 crop |  59G  | 49.8M  | [config](/configs/recognition/uniformer/uniformer-base_imagenet1k-pre_32x4x1_kinetics400-rgb.py)  | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv1/uniformer-base_imagenet1k-pre_32x4x1_kinetics400-rgb_20221219-b776322c.pth)  |

The models are ported from the repo [UniFormer](https://github.com/Sense-X/UniFormer/blob/main/video_classification/README.md) and tested on our data. Currently, we only support the testing of UniFormer models, training will be available soon.

2. The values in `top1/5 acc` is tested on the same data list as the original repo, and the label map is provided by [UniFormer](https://drive.google.com/drive/folders/17VB-XdF3Kfr9ORmnGyXCxTMs86n0L4QL). The total videos are available at [Kinetics400](https://pan.baidu.com/s/1t5K0FRz3PGAT-37-3FwAfg) (BaiduYun password: g5kp), which consists of 19787 videos.

3. The values in columns named after "mm-Kinetics" are the testing results on the Kinetics dataset held by MMAction2, which is also used by other models in MMAction2. Due to the differences between various versions of Kinetics dataset, there is a little gap between `top1/5 acc` and `mm-Kinetics top1/5 acc`. For a fair comparison with other models, we report both results here. Note that we simply report the inference results, since the training set is different between UniFormer and other models, the results are lower than that tested on the author's version.
4. Since the original models for Kinetics-400/600/700 adopt different [label file](https://drive.google.com/drive/folders/17VB-XdF3Kfr9ORmnGyXCxTMs86n0L4QL), we simply map the weight according to the label name. New label map for Kinetics-400/600/700 can be found [here](https://github.com/open-mmlab/mmaction2/tree/main/tools/data/kinetics).

5. Due to some difference between [SlowFast](https://github.com/facebookresearch/SlowFast) and MMAction, there are some gaps between their performances.

## Test

```shell
```

Example: test UniFormer-S model on Kinetics-400 dataset and dump the result to a pkl file.

```shell
python tools/test.py configs/recognition/uniformer/uniformer-small_imagenet1k-pre_16x4x1_kinetics400-rgb.py \
```

```BibTeX
  li2022uniformer,
  title={UniFormer: Unified Transformer for Efficient Spatial-Temporal Representation Learning},

```BibTeX
  author={Kunchang Li and Yali Wang and Gao Peng and Guanglu Song and Yu Liu and Hongsheng Li and Yu Qiao},
  url={https://openreview.net/forum?id=nBU_u6DLvoK}
}
```

The deployment of OpenMMLab codebases, including MMDetection, MMPretrain and so on are supported by [MMDeploy](https://github.com/open-mmlab/mmdeploy).
The latest deployment guide for MMDetection can be found from [here](https://mmdeploy.readthedocs.io/en/dev-1.x/04-supported-codebases/mmdet.html).

This tutorial is organized as follows:

- [Convert model](#convert-model)
- [Model specification](#model-specification)
- [Model inference](#model-inference)
  - [Backend model inference](#backend-model-inference)
  - [SDK model inference](#sdk-model-inference)
- [Supported models](#supported-models)

Please follow the [guide](https://mmdetection.readthedocs.io/en/latest/get_started.html) to install mmdet. And then install mmdeploy from source by following [this](https://mmdeploy.readthedocs.io/en/1.x/get_started.html#installation) guide.

```{note}

```{note}
If you install mmdeploy prebuilt package, please also clone its repository by 'git clone https://github.com/open-mmlab/mmdeploy.git --depth=1' to get the deployment config files.
```

Suppose mmdetection and mmdeploy repositories are in the same directory, and the working directory is the root path of mmdetection.

Take [Faster R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/faster_rcnn/faster-rcnn_r50_fpn_1x_coco.py) model as an example. You can download its checkpoint from [here](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth), and then convert it to onnx model as follows:

```python

img = 'demo/demo.jpg'
work_dir = 'mmdeploy_models/mmdet/onnx'
deploy_cfg = '../mmdeploy/configs/mmdet/detection/detection_onnxruntime_dynamic.py'

model_cfg = 'configs/faster_rcnn/faster-rcnn_r50_fpn_1x_coco.py'
model_checkpoint = 'faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth'

           model_checkpoint, device)

export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint,
           device=device)
```

It is crucial to specify the correct deployment config during model conversion. MMDeploy has already provided builtin deployment config [files](https://github.com/open-mmlab/mmdeploy/tree/1.x/configs/mmdet) of all supported backends for mmdetection, under which the config file path follows the pattern:

```
```

```
```

- **{task}:** task in mmdetection.

  There are two of them. One is `detection` and the other is `instance-seg`, indicating instance segmentation.

  mmdet models like `RetinaNet`, `Faster R-CNN` and `DETR` and so on belongs to `detection` task. While `Mask R-CNN` is one of `instance-seg` models.

  **DO REMEMBER TO USE** `detection/detection_*.py` deployment config file when trying to convert detection models and use `instance-seg/instance-seg_*.py` to deploy instance segmentation models.

Therefore, in the above example, you can also convert `Faster R-CNN` to tensorrt-fp16 model by `detection_tensorrt-fp16_dynamic-320x320-1344x1344.py`.

```{tip}
When converting mmdet models to tensorrt models, --device should be set to "cuda"
```

The converted model locates in the working directory like `mmdeploy_models/mmdet/onnx` in the previous example. It includes:

```
mmdeploy_models/mmdet/onnx
```

in which,

- ***xxx*.json**: the necessary information for mmdeploy SDK

The whole package **mmdeploy_models/mmdet/onnx** is defined as **mmdeploy SDK model**, i.e., **mmdeploy SDK model** includes both backend model and inference meta information.

```python

backend_model = ['mmdeploy_models/mmdet/onnx/end2end.onnx']
image = 'demo/demo.jpg'

    output_file='output_detection.png')
```

```python
import cv2

img = cv2.imread('demo/demo.jpg')

detector = Detector(model_path='mmdeploy_models/mmdet/onnx',
                    device_name='cpu', device_id=0)

import cv2

bboxes, labels, masks = detector(img)

    [left, top, right, bottom], score = bbox[0:4].astype(int), bbox[4]
    if score < 0.3:
        continue

    cv2.rectangle(img, (left, top), (right, bottom), (0, 255, 0))

```

Besides python API, mmdeploy SDK also provides other FFI (Foreign Function Interface), such as C, C++, C#, Java and so on. You can learn their usage from [demos](https://github.com/open-mmlab/mmdeploy/tree/1.x/demo).

Please refer to [here](https://mmdeploy.readthedocs.io/en/1.x/04-supported-codebases/mmdet.html#supported-models) for the supported model list.

# Deep Hough Voting for 3D Object Detection in Point Clouds

> [Deep Hough Voting for 3D Object Detection in Point Clouds](https://arxiv.org/abs/1904.09664)

Current 3D object detection methods are heavily influenced by 2D detectors. In order to leverage architectures in 2D detectors, they often convert 3D point clouds to regular grids (i.e., to voxel grids or to bird's eye view images), or rely on detection in 2D images to propose 3D boxes. Few works have attempted to directly detect objects in point clouds. In this work, we return to first principles to construct a 3D detection pipeline for point cloud data and as generic as possible. However, due to the sparse nature of the data -- samples from 2D manifolds in 3D space -- we face a major challenge when directly predicting bounding box parameters from scene points: a 3D object centroid can be far from any surface point thus hard to regress accurately in one step. To address the challenge, we propose VoteNet, an end-to-end 3D object detection network based on a synergy of deep point set networks and Hough voting. Our model achieves state-of-the-art 3D detection on two large datasets of real 3D scans, ScanNet and SUN RGB-D with a simple design, compact model size and high efficiency. Remarkably, VoteNet outperforms previous methods by using purely geometric information without relying on color images.

<img src="https://user-images.githubusercontent.com/79644370/143888295-af7435b4-9f75-4669-b5f8-a19ae24a051c.png" width="800"/>
</div>

We implement VoteNet and provide the result and checkpoints on ScanNet and SUNRGBD datasets.

### ScanNet

|                  Backbone                  | Lr schd | Mem (GB) | Inf time (fps) | AP@0.25 | AP@0.5 |                                                                                                                                                                  Download                                                                                                                                                                  |

| :----------------------------------------: | :-----: | :------: | :------------: | :-----: | :----: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| [PointNet++](./votenet_8xb8_scannet-3d.py) |   3x    |   4.1    |                |  62.34  | 40.82  | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/votenet/votenet_8x8_scannet-3d-18class/votenet_8x8_scannet-3d-18class_20210823_234503-cf8134fa.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/votenet/votenet_8x8_scannet-3d-18class/votenet_8x8_scannet-3d-18class_20210823_234503.log.json) |

### SUNRGBD

|                  Backbone                   | Lr schd | Mem (GB) | Inf time (fps) | AP@0.25 | AP@0.5 |                                                                                                                                                                    Download                                                                                                                                                                    |
| :-----------------------------------------: | :-----: | :------: | :------------: | :-----: | :----: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

| [PointNet++](./votenet_8xb16_sunrgbd-3d.py) |   3x    |   8.1    |                |  59.78  | 35.77  | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/votenet/votenet_16x8_sunrgbd-3d-10class/votenet_16x8_sunrgbd-3d-10class_20210820_162823-bf11f014.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/votenet/votenet_16x8_sunrgbd-3d-10class/votenet_16x8_sunrgbd-3d-10class_20210820_162823.log.json) |

**Notice**: If your current mmdetection3d version >= 0.6.0, and you are using the checkpoints downloaded from the above links or using checkpoints trained with mmdetection3d version \< 0.6.0, the checkpoints have to be first converted via [tools/model_converters/convert_votenet_checkpoints.py](../../tools/model_converters/convert_votenet_checkpoints.py):

```
python ./tools/model_converters/convert_votenet_checkpoints.py ${ORIGINAL_CHECKPOINT_PATH} --out=${NEW_CHECKPOINT_PATH}
```

```
```

Then you can use the converted checkpoints following [get_started.md](../../docs/en/get_started.md).

Since test data preparation randomly downsamples the points, and the test script uses fixed random seeds while the random seeds of validation in training are not fixed, the test results may be slightly different from the results reported above.

## IoU loss

Adding IoU loss (simply = 1-IoU) boosts VoteNet's performance. To use IoU loss, add this loss term to the config file:

```python
iou_loss=dict(type='AxisAlignedIoULoss', reduction='sum', loss_weight=10.0 / 3.0)
```

|                        Backbone                         | Lr schd | Mem (GB) | Inf time (fps) | AP@0.25 | AP@0.5 | Download |
| :-----------------------------------------------------: | :-----: | :------: | :------------: | :-----: | :----: | :------: |

| [PointNet++](./votenet_head-iouloss_8xb8_scannet-3d.py) |   3x    |   4.1    |                |  63.81  | 44.21  |    /     |

For now, we only support calculating IoU loss for axis-aligned bounding boxes since the CUDA op of general 3D IoU calculation does not implement the backward method. Therefore, IoU loss can only be used for ScanNet dataset for now.

```latex
@inproceedings{qi2019deep,
    author = {Qi, Charles R and Litany, Or and He, Kaiming and Guibas, Leonidas J},
    title = {Deep Hough Voting for 3D Object Detection in Point Clouds},
    booktitle = {Proceedings of the IEEE International Conference on Computer Vision},
}
```

    <sup>
      </a>
    </sup>
    <sup>
      </a>
    </sup>
  </div>

[![docs](https://img.shields.io/badge/docs-latest-blue)](https://mmdetection3d.readthedocs.io/en/latest/)

[🆕Update News](https://mmdetection3d.readthedocs.io/en/latest/notes/changelog.html) |
[🚀Ongoing Projects](https://github.com/open-mmlab/mmdetection3d/projects) |
[🤔Reporting Issues](https://github.com/open-mmlab/mmdetection3d/issues/new/choose)

</div>

</div>

</div>

MMDetection3D is an open source object detection toolbox based on PyTorch, towards the next-generation platform for general 3D detection. It is a part of the [OpenMMLab](https://openmmlab.com/) project.

The main branch works with **PyTorch 1.8+**.

<summary>Major features</summary>

- **Support multi-modality/single-modality detectors out of box**

  It directly supports multi-modality/single-modality detectors including MVXNet, VoteNet, PointPillars, etc.

- **Support indoor/outdoor 3D detection out of box**

  It directly supports popular indoor and outdoor 3D detection datasets, including ScanNet, SUNRGB-D, Waymo, nuScenes, Lyft, and KITTI. For nuScenes dataset, we also support [nuImages dataset](https://github.com/open-mmlab/mmdetection3d/tree/main/configs/nuimages).

- **Natural integration with 2D detection**

  All the about **300+ models, methods of 40+ papers**, and modules supported in [MMDetection](https://github.com/open-mmlab/mmdetection/blob/3.x/docs/en/model_zoo.md) can be trained or used in this codebase.

- **High efficiency**

  It trains faster than other codebases. The main results are as below. Details can be found in [benchmark.md](./docs/en/notes/benchmarks.md). We compare the number of samples trained per second (the higher, the better). The models that are not supported by other codebases are marked by `✗`.

|       VoteNet       |      358      |                          ✗                           |                           77                           |                      ✗                      |

|       Part-A2       |      17       |                          14                          |                           ✗                            |                      ✗                      |

Like [MMDetection](https://github.com/open-mmlab/mmdetection) and [MMCV](https://github.com/open-mmlab/mmcv), MMDetection3D can also be used as a library to support different projects on top of it.

**We have renamed the branch `1.1` to `main` and switched the default branch from `master` to `main`. We encourage users to migrate to the latest version, though it comes with some cost. Please refer to [Migration Guide](docs/en/migration.md) for more details.**

We have constructed a comprehensive LiDAR semantic segmentation benchmark on SemanticKITTI, including Cylinder3D, MinkUNet and SPVCNN methods. Noteworthy, the improved MinkUNetv2 can achieve 70.3 mIoU on the validation set of SemanticKITTI. We have also supported the training of BEVFusion and an occupancy prediction method, TPVFomrer, in our `projects`. More new features about 3D perception are on the way. Please stay tuned!

**v1.2.0** was released in 4/7/2023

- Support [New Config Type](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#a-pure-python-style-configuration-file-beta) in `mmdet3d/config`
- Support the inference of [DSVT](<(https://arxiv.org/abs/2301.06051)>) in `projects`
- Support downloading datasets from [OpenDataLab](https://opendatalab.com/) using `mim`

**v1.1.1** was released in 30/5/2023:

- Support [TPVFormer](https://arxiv.org/pdf/2302.07817.pdf) in `projects`
- Support the training of BEVFusion in `projects`

- Support lidar-based 3D semantic segmentation benchmark

Please refer to [Installation](https://mmdetection3d.readthedocs.io/en/latest/get_started.html) for installation instructions.

For detailed user guides and advanced guides, please refer to our [documentation](https://mmdetection3d.readthedocs.io/en/latest/):

<details>
<summary>User Guides</summary>

- [Train & Test](https://mmdetection3d.readthedocs.io/en/latest/user_guides/index.html#train-test)
  - [Learn about Configs](https://mmdetection3d.readthedocs.io/en/latest/user_guides/config.html)
  - [Coordinate System](https://mmdetection3d.readthedocs.io/en/latest/user_guides/coord_sys_tutorial.html)
  - [Dataset Preparation](https://mmdetection3d.readthedocs.io/en/latest/user_guides/dataset_prepare.html)

- [Dataset Preparation](https://mmdetection3d.readthedocs.io/en/latest/user_guides/dataset_prepare.html)
  - [Customize Data Pipelines](https://mmdetection3d.readthedocs.io/en/latest/user_guides/data_pipeline.html)
  - [Test and Train on Standard Datasets](https://mmdetection3d.readthedocs.io/en/latest/user_guides/train_test.html)
  - [Inference](https://mmdetection3d.readthedocs.io/en/latest/user_guides/inference.html)
  - [Train with Customized Datasets](https://mmdetection3d.readthedocs.io/en/latest/user_guides/new_data_model.html)
- [Useful Tools](https://mmdetection3d.readthedocs.io/en/latest/user_guides/index.html#useful-tools)

<details>
<summary>Advanced Guides</summary>

- [Datasets](https://mmdetection3d.readthedocs.io/en/latest/advanced_guides/index.html#datasets)
  - [KITTI Dataset](https://mmdetection3d.readthedocs.io/en/latest/advanced_guides/datasets/kitti.html)

  - [NuScenes Dataset](https://mmdetection3d.readthedocs.io/en/latest/advanced_guides/datasets/nuscenes.html)
  - [Lyft Dataset](https://mmdetection3d.readthedocs.io/en/latest/advanced_guides/datasets/lyft.html)
  - [Waymo Dataset](https://mmdetection3d.readthedocs.io/en/latest/advanced_guides/datasets/waymo.html)
  - [SUN RGB-D Dataset](https://mmdetection3d.readthedocs.io/en/latest/advanced_guides/datasets/sunrgbd.html)
  - [ScanNet Dataset](https://mmdetection3d.readthedocs.io/en/latest/advanced_guides/datasets/scannet.html)
  - [S3DIS Dataset](https://mmdetection3d.readthedocs.io/en/latest/advanced_guides/datasets/s3dis.html)
  - [SemanticKITTI Dataset](https://mmdetection3d.readthedocs.io/en/latest/advanced_guides/datasets/semantickitti.html)

- [SemanticKITTI Dataset](https://mmdetection3d.readthedocs.io/en/latest/advanced_guides/datasets/semantickitti.html)
- [Supported Tasks](https://mmdetection3d.readthedocs.io/en/latest/advanced_guides/index.html#supported-tasks)
  - [LiDAR-Based 3D Detection](https://mmdetection3d.readthedocs.io/en/latest/advanced_guides/supported_tasks/lidar_det3d.html)
  - [Vision-Based 3D Detection](https://mmdetection3d.readthedocs.io/en/latest/advanced_guides/supported_tasks/vision_det3d.html)
  - [LiDAR-Based 3D Semantic Segmentation](https://mmdetection3d.readthedocs.io/en/latest/advanced_guides/supported_tasks/lidar_sem_seg3d.html)
- [Customization](https://mmdetection3d.readthedocs.io/en/latest/advanced_guides/index.html#customization)
  - [Customize Datasets](https://mmdetection3d.readthedocs.io/en/latest/advanced_guides/customize_dataset.html)
  - [Customize Models](https://mmdetection3d.readthedocs.io/en/latest/advanced_guides/customize_models.html)

- [Customize Models](https://mmdetection3d.readthedocs.io/en/latest/advanced_guides/customize_models.html)
  - [Customize Runtime Settings](https://mmdetection3d.readthedocs.io/en/latest/advanced_guides/customize_runtime.html)

## Overview of Benchmark and Model Zoo

Results and models are available in the [model zoo](docs/en/model_zoo.md).

  <b>Components</b>
</div>
  <tbody>
      <td>
      </td>
      <td>
        <b>Heads</b>
      </td>
      <td>
        <b>Features</b>
      </td>
    </tr>
      <td>
      <ul>

<li><a href="configs/regnet">RegNet (CVPR'2020)</a></li>
      </ul>
      </td>
      <td>
      <ul>
      </ul>
      </td>
      <td>
      <ul>
      </ul>
      </td>
    </tr>
</td>
    </tr>
  </tbody>
</table>

  <b>Architectures</b>
</div>
  <tbody>
      <td>
        <b>LiDAR-based 3D Object Detection</b>
      </td>
      <td>
        <b>Camera-based 3D Object Detection</b>
      </td>
      <td>
        <b>Multi-modal 3D Object Detection</b>

<td>
      </td>
      <td>
      </td>
      <td>
      </td>
      <td>
        <b>3D Semantic Segmentation</b>
      </td>
    </tr>
      <td>
        <li><b>Outdoor</b></li>
        <ul>

<li><a href="configs/parta2">Part-A2 (TPAMI'2020)</a></li>
        </ul>
        <ul>
      </ul>
      </td>
      <td>
        <ul>

<td>
        <ul>
        </ul>
        <ul>
        </ul>
      </td>
      <td>
        <ul>
        </ul>
        <ul>

        </ul>
        <ul>
        </ul>
      </td>
      <td>
        <ul>
        </ul>
        <ul>
        </ul>
      </ul>
      </td>
    </tr>
</td>
    </tr>
  </tbody>
</table>

**Note:** All the about **500+ models, methods of 90+ papers** in 2D detection supported by [MMDetection](https://github.com/open-mmlab/mmdetection/blob/3.x/docs/en/model_zoo.md) can be trained or used in this codebase.

## FAQ

## FAQ

We appreciate all contributions to improve MMDetection3D. Please refer to [CONTRIBUTING.md](docs/en/notes/contribution_guides.md) for the contributing guideline.

MMDetection3D is an open source project that is contributed by researchers and engineers from various colleges and companies. We appreciate all the contributors as well as users who give valuable feedbacks. We wish that the toolbox and benchmark could serve the growing research community by providing a flexible toolkit to reimplement existing methods and develop their own new 3D detectors.

```latex

```latex
}
```

## License

# CAIN (AAAI'2020)

> [Channel Attention Is All You Need for Video Frame Interpolation](https://aaai.org/ojs/index.php/AAAI/article/view/6693/6547)

> **Task**: Video Interpolation

Prevailing video frame interpolation techniques rely heavily on optical flow estimation and require additional model complexity and computational cost; it is also susceptible to error propagation in challenging scenarios with large motion and heavy occlusion. To alleviate the limitation, we propose a simple but effective deep neural network for video frame interpolation, which is end-to-end trainable and is free from a motion estimation network component. Our algorithm employs a special feature reshaping operation, referred to as PixelShuffle, with a channel attention, which replaces the optical flow computation module. The main idea behind the design is to distribute the information in a feature map into multiple channels and extract motion information by attending the channels for pixel-level frame synthesis. The model given by this principle turns out to be effective in the presence of challenging motion and occlusion. We construct a comprehensive evaluation benchmark and demonstrate that the proposed approach achieves outstanding performance compared to the existing models with a component for optical flow computation.

 <img src="https://user-images.githubusercontent.com/56712176/149734064-1da0cebf-6953-4106-a29a-43acd7386a80.png" width="400"/>
</div >

The learning rate adjustment strategy is `Step LR scheduler with min_lr clipping`.

|                                  Model                                  |  Dataset   |  PSNR   |  SSIM  |    Training Resources    |                                      Download                                      |
| :---------------------------------------------------------------------: | :--------: | :-----: | :----: | :----------------------: | :--------------------------------------------------------------------------------: |

| [cain_b5_g1b32_vimeo90k_triplet](./cain_g1b32_1xb5_vimeo90k-triplet.py) | vimeo90k-T | 34.6010 | 0.9578 | 1 (Tesla V100-SXM2-32GB) | [model](https://download.openmmlab.com/mmediting/video_interpolators/cain/cain_b5_g1b32_vimeo90k_triplet_20220530-3520b00c.pth)/[log](https://download.openmmlab.com/mmediting/video_interpolators/cain/cain_b5_g1b32_vimeo90k_triplet_20220530-3520b00c.log.json) |

**Train**

<details>

```shell
# cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/cain/cain_g1b32_1xb5_vimeo90k-triplet.py

python tools/train.py configs/cain/cain_g1b32_1xb5_vimeo90k-triplet.py

./tools/dist_train.sh configs/cain/cain_g1b32_1xb5_vimeo90k-triplet.py 8
```

```

**Test**

<details>

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/cain/cain_g1b32_1xb5_vimeo90k-triplet.py https://download.openmmlab.com/mmediting/video_interpolators/cain/cain_b5_g1b32_vimeo90k_triplet_20220530-3520b00c.pth

python tools/test.py configs/cain/cain_g1b32_1xb5_vimeo90k-triplet.py https://download.openmmlab.com/mmediting/video_interpolators/cain/cain_b5_g1b32_vimeo90k_triplet_20220530-3520b00c.pth

./tools/dist_test.sh configs/cain/cain_g1b32_1xb5_vimeo90k-triplet.py https://download.openmmlab.com/mmediting/video_interpolators/cain/cain_b5_g1b32_vimeo90k_triplet_20220530-3520b00c.pth 8
```

```bibtex
@inproceedings{choi2020channel,
  title={Channel attention is all you need for video frame interpolation},
  author={Choi, Myungsub and Kim, Heewon and Han, Bohyung and Xu, Ning and Lee, Kyoung Mu},
  number={07},
  pages={10663--10671},
}
```

# 新增自定义数据集

## 新增自定义数据集

在这里，我们展示如何构建一个新的数据集。

1. 创建一个新文件 `mmseg/datasets/example.py`

   ```python
   from mmseg.registry import DATASETS
   from .basesegdataset import BaseSegDataset

   @DATASETS.register_module()
   class ExampleDataset(BaseSegDataset):

       METAINFO = dict(
           classes=('xxx', 'xxx', ...),
           palette=[[x, x, x], [x, x, x], ...])

       def __init__(self, aeg1, arg2):
           pass
   ```

2. 在 `mmseg/datasets/__init__.py` 中导入模块

   ```python
   from .example import ExampleDataset
   ```

3. 通过创建一个新的数据集配置文件 `configs/_base_/datasets/example_dataset.py` 来使用它

   ```python
   dataset_type = 'ExampleDataset'
   data_root = 'data/example/'
   ...
   ```

4. 在 `mmseg/utils/class_names.py` 中补充数据集元信息

   ```python
   def example_classes():
       return [
           'xxx', 'xxx',
           ...
       ]

   def example_palette():
       return [
           [x, x, x], [x, x, x],
           ...
       ]
   dataset_aliases ={

return [
           ...
       ]

       return [
           ...
       ]
       'example': ['example', ...],
       ...
   }
   ```

**注意：** 如果新数据集不满足 mmseg 的要求，则需要在 `tools/dataset_converters/` 中准备一个数据集预处理脚本

## 通过重新组织数据来定制数据集

最简单的方法是将您的数据集进行转化，并组织成文件夹的形式。

如下的文件结构就是一个例子。

```none
├── data
│   ├── my_dataset
│   │   │   │   ├── xxx{img_suffix}
│   │   │   │   ├── yyy{img_suffix}
│   │   │   │   ├── zzz{img_suffix}
│   │   │   ├── val
│   │   │   │   ├── xxx{seg_map_suffix}
│   │   │   │   ├── yyy{seg_map_suffix}
│   │   │   │   ├── zzz{seg_map_suffix}
│   │   │   ├── val

```

一个训练对将由 img_dir/ann_dir 里同样首缀的文件组成。

有些数据集不会发布测试集或测试集的标注，如果没有测试集的标注，我们就无法在本地进行评估模型，因此我们在配置文件中将验证集设置为默认测试集。

关于如何构建自己的数据集或实现新的数据集类，请参阅[数据集指南](./datasets.md)以获取更多详细信息。

│   │   │   ├── val

```

**注意：** 标注是跟图像同样的形状 (H, W)，其中的像素值的范围是 `[0, num_classes - 1]`。
您也可以使用 [pillow](https://pillow.readthedocs.io/en/stable/handbook/concepts.html#palette) 的 `'P'` 模式去创建包含颜色的标注。

## 通过混合数据去定制数据集

MMSegmentation 同样支持混合数据集去训练。
当前它支持拼接 (concat), 重复 (repeat) 和多图混合 (multi-image mix) 数据集。

### 重复数据集

我们使用 `RepeatDataset` 作为包装 (wrapper) 去重复数据集。
例如，假设原始数据集是 `Dataset_A`，为了重复它，配置文件如下：

```python
    times=N,
    dataset=dict(  # 这是 Dataset_A 数据集的原始配置
        ...
    )
)
```

### 拼接数据集

如果要拼接不同的数据集，可以按如下方式连接数据集配置。

```python
dataset_A_train = dict()
dataset_B_train = dict()
concatenate_dataset = dict(
    type='ConcatDataset',
    datasets=[dataset_A_train, dataset_B_train])
```

```python
```

下面是一个更复杂的示例，它分别重复 `Dataset_A` 和 `Dataset_B` N 次和 M 次，然后连接重复的数据集。

```python
    times=N,
        ...
    )
)
    ...
)
    ...
)
    times=M,
        ...
    )
)
        datasets=[dataset_A_train, dataset_B_train]))

val_dataloader = dict(dataset=dataset_A_val)
test_dataloader = dict(dataset=dataset_A_test)

```

```

您可以参考 mmengine 的基础数据集[教程](https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/basedataset.html)以了解更多详细信息

### 多图混合集

我们使用 `MultiImageMixDataset` 作为包装（wrapper）去混合多个数据集的图片。
`MultiImageMixDataset`可以被类似 mosaic 和 mixup 的多图混合数据増广使用。

`MultiImageMixDataset` 与 `Mosaic` 数据増广一起使用的例子：

```python
    dict(type='RandomMosaic', prob=1),
    dict(type='Resize', img_scale=(1024, 512), keep_ratio=True),
]

    type='MultiImageMixDataset',
        reduce_zero_label=False,
        img_dir=data_root + "images/train",
        ann_dir=data_root + "annotations/train",

reduce_zero_label=False,
            dict(type='LoadAnnotations'),
        ]
    ),
    pipeline=train_pipeline
)

```

# 3D Hand Keypoint Datasets

- [InterHand2.6M](#interhand26m) \[ [Homepage](https://mks0601.github.io/InterHand2.6M/) \]

## InterHand2.6M

<details>
<summary align="right"><a href="https://link.springer.com/content/pdf/10.1007/978-3-030-58565-5_33.pdf">InterHand2.6M (ECCV'2020)</a></summary>

```bibtex
@InProceedings{Moon_2020_ECCV_InterHand2.6M,
author = {Moon, Gyeongsik and Yu, Shoou-I and Wen, He and Shiratori, Takaaki and Lee, Kyoung Mu},
title = {InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image},
booktitle = {European Conference on Computer Vision (ECCV)},
}
```

}
```

  <img src="https://user-images.githubusercontent.com/100993824/227867693-e36c2cbd-4089-4734-a310-37cb5db6cafa.png" height="200px">
</div>

For [InterHand2.6M](https://mks0601.github.io/InterHand2.6M/), please download from [InterHand2.6M](https://mks0601.github.io/InterHand2.6M/).
Please download the annotation files from [annotations](https://drive.google.com/drive/folders/1pWXhdfaka-J0fSAze0MsajN0VpZ8e8tO).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── interhand2.6m
        |   |── all
        |   |── human_annot
        |   |── machine_annot
        |   |── skeleton.txt
        |   |── subject.txt
        |
        `── images
        |   |── train
        |   |   |-- Capture0 ~ Capture26
        |   |── val
        |   |   |-- Capture0

|   |── skeleton.txt
        |
        `── images
        |   |── train
        |   |── val
        |   |── test
        |   |   |-- Capture0 ~ Capture7
```

# S3DIS 数据集

## 数据集的准备

对于数据集准备的整体流程，请参考 S3DIS 的[指南](https://github.com/open-mmlab/mmdetection3d/blob/master/data/s3dis/README.md/)。

### 提取 S3DIS 数据

通过从原始数据中提取 S3DIS 数据，我们将点云数据读取并保存下相关的标注信息，例如语义分割标签和实例分割标签。

数据提取前的目录结构应该如下所示：

```
├── tools
├── data
│   ├── s3dis
│   │   │   ├── Area_1
│   │   │   │   ├── conferenceRoom_1
│   │   │   │   ├── office_1
│   │   │   ├── Area_2
│   │   │   ├── Area_3
│   │   │   ├── Area_4
│   │   │   ├── Area_5
│   │   │   ├── Area_6
```

在 `Stanford3dDataset_v1.2_Aligned_Version` 目录下，所有房间依据所属区域被分为 6 组。
我们通常使用 5 个区域进行训练，然后在余下 1 个区域上进行测试 (被余下的 1 个区域通常为区域 5)。
在每个区域的目录下包含有多个房间的文件夹，每个文件夹是一个房间的原始点云数据和相关的标注信息。
例如，在 `Area_1/office_1` 目录下的文件如下所示：

- `office_1.txt`：一个 txt 文件存储着原始点云数据每个点的坐标和颜色信息。

- `Annotations/`：这个文件夹里包含有此房间中实例物体的信息 (以 txt 文件的形式存储)。每个 txt 文件表示一个实例，例如：

  - `chair_1.txt`：存储有该房间中一把椅子的点云数据。

  如果我们将 `Annotations/` 下的所有 txt 文件合并起来，得到的点云就和 `office_1.txt` 中的点云是一致的。

你可以通过 `python collect_indoor3d_data.py` 指令进行 S3DIS 数据的提取。
主要步骤包括：

- 从原始 txt 文件中读取点云数据、语义分割标签和实例分割标签。
- 将点云数据和相关标注文件存储下来。

这其中的核心函数 `indoor3d_util.py` 中的 `export` 函数实现如下：

```python
def export(anno_path, out_filename):
    """将原始数据集的文件转化为点云、语义分割标签和实例分割掩码文件。
    我们将同一房间中所有实例的点进行聚合。

    参数列表:
        anno_path (str): 标注信息的路径，例如 Area_1/office_2/Annotations/
        out_filename (str): 保存点云和标签的路径
        file_format (str): txt 或 numpy，指定保存的文件格式

    注意:
        点云在处理过程中被整体移动了，保存下的点最小位于原点 (即没有负数坐标值)
    """
    points_list = []
    ins_idx = 1  # 实例标签从 1 开始，因此最终实例标签为 0 的点就是无标注的点

file_format (str): txt 或 numpy，指定保存的文件格式

    注意:
    """

    # `anno_path` 的一个例子：Area_1/office_1/Annotations
    # 其中以 txt 文件存储有该房间中所有实例物体的点云
    for f in glob.glob(osp.join(anno_path, '*.txt')):
        # get class name of this instance
        one_class = osp.basename(f).split('_')[0]
        if one_class not in class_names:  # 某些房间有 'staris' 类物体
            one_class = 'clutter'
        points = np.loadtxt(f)
        labels = np.ones((points.shape[0], 1)) * class2label[one_class]
        ins_labels = np.ones((points.shape[0], 1)) * ins_idx
        ins_idx += 1
        points_list.append(np.concatenate([points, labels, ins_labels], 1))

    data_label = np.concatenate(points_list, 0)  # [N, 8], (pts, rgb, sem, ins)
    # 将点云对齐到原点
    xyz_min = np.amin(data_label, axis=0)[0:3]
    data_label[:, 0:3] -= xyz_min

data_label = np.concatenate(points_list, 0)  # [N, 8], (pts, rgb, sem, ins)
    # 将点云对齐到原点

    np.save(f'{out_filename}_point.npy', data_label[:, :6].astype(np.float32))
    np.save(f'{out_filename}_sem_label.npy', data_label[:, 6].astype(np.int64))
    np.save(f'{out_filename}_ins_label.npy', data_label[:, 7].astype(np.int64))

```

上述代码中，我们读取 `Annotations/` 下的所有点云实例，将其合并得到整体房屋的点云，同时生成语义/实例分割的标签。
在提取完每个房间的数据后，点云、语义分割和实例分割的标签文件应以 `.npy` 的格式被保存下来。

### 创建数据集

```shell
python tools/create_data.py s3dis --root-path ./data/s3dis \
--out-dir ./data/s3dis --extra-tag s3dis
```

上述指令首先读取以 `.npy` 格式存储的点云、语义分割和实例分割标签文件，然后进一步将它们以 `.bin` 格式保存。
同时，每个区域 `.pkl` 格式的信息文件也会被保存下来。

数据预处理后的目录结构如下所示：

```
s3dis
├── points

├── points
```

- `points/xxxxx.bin`：提取的点云数据。
- `instance_mask/xxxxx.bin`：每个点云的实例标签，取值范围为 \[0, ${实例个数}\]，其中 0 代表未标注的点。

```

- `semantic_mask/xxxxx.bin`：每个点云的语义标签，取值范围为 \[0, 12\]。
- `s3dis_infos_Area_1.pkl`：区域 1 的数据信息，每个房间的详细信息如下：
  - info\['point_cloud'\]: {'num_features': 6, 'lidar_idx': sample_idx}.
  - info\['pts_path'\]: `points/xxxxx.bin` 点云的路径。
  - info\['pts_instance_mask_path'\]: `instance_mask/xxxxx.bin` 实例标签的路径。
  - info\['pts_semantic_mask_path'\]: `semantic_mask/xxxxx.bin` 语义标签的路径。
- `seg_info`：为支持语义分割任务所生成的信息文件。
  - `Area_1_label_weight.npy`：每一语义类别的权重系数。因为 S3DIS 中属于不同类的点的数量相差很大，一个常见的操作是在计算损失时对不同类别进行加权 (label re-weighting) 以得到更好的分割性能。
  - `Area_1_resampled_scene_idxs.npy`：每一个场景 (房间) 的重采样标签。在训练过程中，我们依据每个场景的点的数量，会对其进行不同次数的重采样，以保证训练数据均衡。

## 训练流程

S3DIS 上 3D 语义分割的一种典型数据载入流程如下所示：

```python
class_names = ('ceiling', 'floor', 'wall', 'beam', 'column', 'window', 'door',

## 训练流程

```python
               'table', 'chair', 'sofa', 'bookcase', 'board', 'clutter')
num_points = 4096
    dict(
    dict(
    dict(
    dict(
        block_size=1.0,
        ignore_index=None,
        use_normalized_coord=True,
        enlarge_size=None,
        min_unique_num=num_points // 4,
        eps=0.0),
    dict(

enlarge_size=None,
        eps=0.0),
    dict(
        rot_range=[-3.141592653589793, 3.141592653589793],  # [-pi, pi]
        scale_ratio_range=[0.8, 1.2],
    dict(
        type='RandomJitterPoints',
        jitter_std=[0.01, 0.01, 0.01],
        clip_range=[-0.05, 0.05]),
    dict(type='RandomDropPointsColor', drop_ratio=0.2),
]
```

- `PointSegClassMapping`：在训练过程中，只有被使用的类别的序号会被映射到类似 \[0, 13) 范围内的类别标签。其余的类别序号会被转换为 `ignore_index` 所制定的忽略标签，在本例中是 `13`。
- `IndoorPatchPointSample`：从输入点云中裁剪一个含有固定数量点的小块 (patch)。`block_size` 指定了裁剪块的边长，在 S3DIS 上这个数值一般设置为 `1.0`。
- `NormalizePointsColor`：将输入点的颜色信息归一化，通过将 RGB 值除以 `255` 来实现。
- 数据增广：
  - `GlobalRotScaleTrans`：对输入点云进行随机旋转和放缩变换。
  - `RandomJitterPoints`：通过对每一个点施加不同的噪声向量以实现对点云的随机扰动。

- 数据增广：
  - `RandomDropPointsColor`：以 `drop_ratio` 的概率随机将点云的颜色值全部置零。

## 度量指标

通常我们使用平均交并比 (mean Intersection over Union, mIoU) 作为 S3DIS 语义分割任务的度量指标。
具体而言，我们先计算所有类别的 IoU，然后取平均值作为 mIoU。
更多实现细节请参考 [seg_eval.py](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/evaluation/functional/seg_eval.py)。

正如在 `提取 S3DIS 数据` 一节中所提及的，S3DIS 通常在 5 个区域上进行训练，然后在余下的 1 个区域上进行测试。但是在其他论文中，也有不同的划分方式。
为了便于灵活划分训练和测试的子集，我们首先定义子数据集 (sub-dataset) 来表示每一个区域，然后根据区域划分对其进行合并，以得到完整的训练集。
以下是在区域 1、2、3、4、6 上训练并在区域 5 上测试的一个配置文件例子：

```python
dataset_type = 'S3DISSegDataset'
data_root = './data/s3dis/'
train_area = [1, 2, 3, 4, 6]
test_area = 5
    batch_size=8,

'table', 'chair', 'sofa', 'bookcase', 'board', 'clutter')
        ann_files=[f's3dis_infos_Area_{i}.pkl' for i in train_area],
        data_prefix=data_prefix,
        scene_idxs=[
            f'seg_info/Area_{i}_resampled_scene_idxs.npy' for i in train_area
        ],
        test_mode=False))

        ann_files=f's3dis_infos_Area_{test_area}.pkl',
        scene_idxs=f'seg_info/Area_{test_area}_resampled_scene_idxs.npy',
```

可以看到，我们通过将多个相应路径构成的列表 (list) 输入 `ann_files` 和 `scene_idxs` 以实现训练测试集的划分。
如果修改训练测试区域的划分，只需要简单修改 `train_area` 和 `test_area` 即可。

</div>

### 300W Dataset

Results on 300W dataset

|    Model    | Input Size | NME<sub>*common*</sub> | NME<sub>*challenge*</sub> | NME<sub>*full*</sub> | NME<sub>*test*</sub> |           Details and Download            |

| :---------: | :--------: | :--------------------: | :-----------------------: | :------------------: | :------------------: | :---------------------------------------: |
| HRNetv2-w18 |  256x256   |          2.92          |           5.64            |         3.45         |         4.10         | [hrnetv2_300w.md](./300w/hrnetv2_300w.md) |

### AFLW Dataset

|      Model       | Input Size | NME<sub>*full*</sub> | NME<sub>*frontal*</sub> |                Details and Download                 |
| :--------------: | :--------: | :------------------: | :---------------------: | :-------------------------------------------------: |
| HRNetv2-w18+Dark |  256x256   |         1.35         |          1.19           | [hrnetv2_dark_aflw.md](./aflw/hrnetv2_dark_aflw.md) |

|   HRNetv2-w18    |  256x256   |         1.41         |          1.27           |      [hrnetv2_aflw.md](./aflw/hrnetv2_aflw.md)      |

### COCO-WholeBody-Face Dataset

|      Model       | Input Size |  NME   |                                       Details and Download                                       |
| :--------------: | :--------: | :----: | :----------------------------------------------------------------------------------------------: |
| HRNetv2-w18+Dark |  256x256   | 0.0513 | [hrnetv2_dark_coco_wholebody_face.md](./coco_wholebody_face/hrnetv2_dark_coco_wholebody_face.md) |
|     SCNet-50     |  256x256   | 0.0567 |        [scnet_coco_wholebody_face.md](./coco_wholebody_face/scnet_coco_wholebody_face.md)        |

|   HRNetv2-w18    |  256x256   | 0.0569 |      [hrnetv2_coco_wholebody_face.md](./coco_wholebody_face/hrnetv2_coco_wholebody_face.md)      |
|    ResNet-50     |  256x256   | 0.0582 |       [resnet_coco_wholebody_face.md](./coco_wholebody_face/resnet_coco_wholebody_face.md)       |
|   HourglassNet   |  256x256   | 0.0587 |    [hourglass_coco_wholebody_face.md](./coco_wholebody_face/hourglass_coco_wholebody_face.md)    |
|   MobileNet-v2   |  256x256   | 0.0611 |  [mobilenetv2_coco_wholebody_face.md](./coco_wholebody_face/mobilenetv2_coco_wholebody_face.md)  |

### COFW Dataset

Results on COFW dataset

|    Model    | Input Size | NME  |           Details and Download            |
| :---------: | :--------: | :--: | :---------------------------------------: |
| HRNetv2-w18 |  256x256   | 3.48 | [hrnetv2_cofw.md](./cofw/hrnetv2_cofw.md) |

### WFLW  Dataset

Results on WFLW  dataset

|  Model  | Input Size | NME<sub>*test*</sub> | NME<sub>*pose*</sub> | NME<sub>*illumination*</sub> | NME<sub>*occlusion*</sub> | NME<sub>*blur*</sub> | NME<sub>*makeup*</sub> | NME<sub>*expression*</sub> |  Details and Download  |
| :-----: | :--------: | :------------------: | :------------------: | :--------------------------: | :-----------------------: | :------------------: | :--------------------: | :------------------------: | :--------------------: |
| HRNetv2-w18+Dark |  256x256   |         3.98         |         6.98         |             3.96             |           4.78            |         4.56         |          3.89          |            4.29            | [hrnetv2_dark_wflw.md](./wflw/hrnetv2_dark_wflw.md) |

| HRNetv2-w18+AWing |  256x256   |         4.02         |         6.94         |             3.97             |           4.78            |         4.59         |          3.87          |            4.28            | [hrnetv2_awing_wflw.md](./wflw/hrnetv2_awing_wflw.md) |
| HRNetv2-w18 |  256x256   |         4.06         |         6.97         |             3.99             |           4.83            |         4.58         |          3.94          |            4.33            | [hrnetv2_wflw.md](./wflw/hrnetv2_wflw.md) |

<details>
<summary align="right">EDSR (CVPR'2017)</summary>

```bibtex
}
```

<br/>

|                                 算法                                 |       Set5       |      Set14       |      DIV2K       | GPU 信息 |                                     下载                                     |
| :------------------------------------------------------------------: | :--------------: | :--------------: | :--------------: | :------: | :--------------------------------------------------------------------------: |

| [edsr_x2c64b16_1x16_300k_div2k](./edsr_x2c64b16_1xb16-300k_div2k.py) | 35.7592 / 0.9372 | 31.4290 / 0.8874 | 34.5896 / 0.9352 |    1     | [模型](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x2c64b16_1x16_300k_div2k_20200604-19fe95ea.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x2c64b16_1x16_300k_div2k_20200604_221933.log.json) |
| [edsr_x3c64b16_1x16_300k_div2k](./edsr_x3c64b16_1xb16-300k_div2k.py) | 32.3301 / 0.8912 | 28.4125 / 0.8022 | 30.9154 / 0.8711 |    1     | [模型](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x3c64b16_1x16_300k_div2k_20200608-36d896f4.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x3c64b16_1x16_300k_div2k_20200608_114850.log.json) |

| [edsr_x4c64b16_1x16_300k_div2k](./edsr_x4c64b16_1xb16-300k_div2k.py) | 30.2223 / 0.8500 | 26.7870 / 0.7366 | 28.9675 / 0.8172 |    1     | [模型](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x4c64b16_1x16_300k_div2k_20200608-3c2af8a3.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/edsr/edsr_x4c64b16_1x16_300k_div2k_20200608_115148.log.json) |

## 快速开始

**训练**

<details>

```shell
# CPU上训练

# 单个GPU上训练

# 多个GPU上训练
```

**测试**

<details>

```shell
# CPU上测试

```

**测试**

<details>

```shell
# CPU上测试

# 单个GPU上测试

# 多个GPU上测试
```

<details>
<summary align="right">RDN (CVPR'2018)</summary>

```bibtex
}
```

<br/>

|                                算法                                |       Set5       |      Set14       |      DIV2K       |   GPU 信息   |                                    下载                                    |
| :----------------------------------------------------------------: | :--------------: | :--------------: | :--------------: | :----------: | :------------------------------------------------------------------------: |

| [rdn_x2c64b16_g1_1000k_div2k](./rdn_x2c64b16_1xb16-1000k_div2k.py) | 35.9883 / 0.9385 | 31.8366 / 0.8920 | 34.9392 / 0.9380 | 1 (TITAN Xp) | [模型](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x2c64b16_g1_1000k_div2k_20210419-dc146009.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x2c64b16_g1_1000k_div2k_20210419-dc146009.log.json) |
| [rdn_x3c64b16_g1_1000k_div2k](./rdn_x3c64b16_1xb16-1000k_div2k.py) | 32.6051 / 0.8943 | 28.6338 / 0.8077 | 31.2153 / 0.8763 | 1 (TITAN Xp) | [模型](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x3c64b16_g1_1000k_div2k_20210419-b93cb6aa.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x3c64b16_g1_1000k_div2k_20210419-b93cb6aa.log.json) |

| [rdn_x4c64b16_g1_1000k_div2k](./rdn_x4c64b16_1xb16-1000k_div2k.py) | 30.4922 / 0.8548 | 26.9570 / 0.7423 | 29.1925 / 0.8233 | 1 (TITAN Xp) | [模型](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x4c64b16_g1_1000k_div2k_20210419-3577d44f.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/rdn/rdn_x4c64b16_g1_1000k_div2k_20210419-3577d44f.log.json) |

## 快速开始

**训练**

<details>

```shell
# CPU上训练

# 单个GPU上训练

# 多个GPU上训练
```

**测试**

<details>

```shell
# CPU上测试

```

**测试**

<details>

```shell
# CPU上测试

# 单个GPU上测试

# 多个GPU上测试
```

<details>

```bibtex
}
```

<details>

```bibtex

}
```

<details>

```bibtex
}
```

| [S-ViPNAS-MobileNetV3_dark](/configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_vipnas-mbv3_dark-8xb64-210e_coco-wholebody-256x192.py) |  256x192   |  0.632  |  0.710  |  0.530  |  0.660  |  0.672  |  0.771  |  0.404  |  0.519  |  0.508   |  0.607   | [ckpt](https://download.openmmlab.com/mmpose/top_down/vipnas/vipnas_mbv3_coco_wholebody_256x192_dark-e2158108_20211205.pth) | [log](https://download.openmmlab.com/mmpose/top_down/vipnas/vipnas_mbv3_coco_wholebody_256x192_dark_20211205.log.json) |

| [S-ViPNAS-Res50_dark](/configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_vipnas-res50_dark-8xb64-210e_coco-wholebody-256x192.py) |  256x192   |  0.650  |  0.732  |  0.550  |  0.686  |  0.684  |  0.783  |  0.437  |  0.554  |  0.528   |  0.632   | [ckpt](https://download.openmmlab.com/mmpose/top_down/vipnas/vipnas_res50_wholebody_256x192_dark-67c0ce35_20211112.pth) | [log](https://download.openmmlab.com/mmpose/top_down/vipnas/vipnas_res50_wholebody_256x192_dark_20211112.log.json) |

# MMagic Demo

There are some mmagic demos in this folder. We provide python command line usage here to run these demos and more guidance could also be found in the [documentation](https://mmagic.readthedocs.io/en/latest/user_guides/3_inference.html)

Table of contents:

[1. Download sample images or videos](#1-download-sample-images-or-videos)

[2. MMagic inference demo](#2-mmagic-inference-demo)

&#8195;    [2.1. Check supported tasks and models](#21-check-supported-tasks-and-models)

&#8195;    [2.2. Perform inference with command line](#22-perform-inference-with-command-line)

&#8195;      [2.2.1. Conditional GANs example](#221-conditional-gans)

&#8195;      [2.2.2. Inpainting example](#222-inpainting)

&#8195;      [2.2.3. Matting example](#223-matting)

&#8195;      [2.2.4. Image Super-Resolution example](#224-image-super-resolution)

&#8195;      [2.2.5. Image Translation example](#225-image-translation)

&#8195;      [2.2.6. Unconditional GANs example](#226-unconditional-gans)

&#8195;      [2.2.7. Video Interpolation example](#227-video-interpolation)

&#8195;      [2.2.8. Video Super-Resolution example](#228-video-super-resolution)

&#8195;      [2.2.9. Text-to-Image example](#229-text-to-image)

&#8195;      [2.2.10. 3D-aware Generation example](#2210-3d-aware-generation)

&#8195;      [2.2.11. Image Restoration example](#2211-image-restoration)

[3. Other demos](#3-other-demos)

## 1. Download sample images or videos

We prepared some images and videos for you to run demo with. After MMagic is well installed, you could use demos in this folder to infer these data.
Download with python script [download_inference_resources.py](./download_inference_resources.py).

```shell
# cd mmagic demo path
cd mmagic/demo

# see all resources
python download_inference_resources.py --print-all
# see all task types

```shell

python download_inference_resources.py --print-task-type
# see resources of one specific task
python download_inference_resources.py --print-task 'Inpainting'
# download all resources to default dir '../resources'
python download_inference_resources.py
# download resources of one task
python download_inference_resources.py --task 'Inpainting'
# download to the directory you want
python download_inference_resources.py --root-dir '../resources'
```

## 2. MMagic inference demo

### 2.1 Check supported tasks and models

print all supported models for inference.

```shell
python mmagic_inference_demo.py --print-supported-models
```

print all supported tasks for inference.

```shell
python mmagic_inference_demo.py --print-supported-tasks
```

print all supported models for one task, take 'Image2Image' for example.

```shell

```

```shell
```

```shell
python mmagic_inference_demo.py --print-task-supported-models 'Image2Image'
```

### 2.2 Perform inference with command line

You can use the following commands to perform inference with a MMagic model.

Usage of python API can also be found in this [tutotial](./mmagic_inference_tutorial.ipynb).

```shell
    [--img] \
    [--video] \
    [--label] \
    [--trimap] \
    [--mask] \
    [--result-out-dir] \
    [--model-name] \
    [--model-setting] \
    [--model-config] \
    [--model-ckpt] \
    [--device ] \
    [--extra-parameters]
```

Examples for each kind of task:

#### 2.2.1 Conditional GANs

```shell
python mmagic_inference_demo.py \
        --model-name biggan \
        --label 1 \

[--extra-parameters]
```

```shell
        --label 1 \
        --result-out-dir ../resources/output/conditional/demo_conditional_biggan_res.jpg
```

#### 2.2.2 Inpainting

```shell
        --model-name global_local  \
        --img ../resources/input/inpainting/celeba_test.png \
        --mask ../resources/input/inpainting/bbox_mask.png \
        --result-out-dir ../../resources/output/inpainting/demo_inpainting_global_local_res.jpg
```

#### 2.2.3 Matting

```shell
        --model-name gca  \
        --img ../resources/input/matting/GT05.jpg \
        --trimap ../resources/input/matting/GT05_trimap.jpg \
        --result-out-dir ../resources/output/matting/demo_matting_gca_res.png
```

#### 2.2.4 Image Super-resolution

```shell

--result-out-dir ../resources/output/matting/demo_matting_gca_res.png
```

```shell
        --model-name esrgan \
        --img ../resources/input/restoration/0901x2.png \
        --result-out-dir ../resources/output/restoration/demo_restoration_esrgan_res.png
```

```shell
        --model-name ttsr \
        --ref ../resources/input/restoration/0901x2.png \
        --result-out-dir ../resources/output/restoration/demo_restoration_ttsr_res.png
```

#### 2.2.5 Image translation

```shell
        --model-name pix2pix \
        --img ../resources/input/translation/gt_mask_0.png \
        --result-out-dir ../resources/output/translation/demo_translation_pix2pix_res.png
```

#### 2.2.6 Unconditional GANs

```shell
        --model-name styleganv1 \

```

```shell
        --result-out-dir ../resources/output/unconditional/demo_unconditional_styleganv1_res.jpg
```

#### 2.2.7 Video interpolation

```shell
        --model-name flavr \
        --video ../resources/input/video_interpolation/b-3LLDhc4EU_000000_000010.mp4 \
        --result-out-dir ../resources/output/video_interpolation/demo_video_interpolation_flavr_res.mp4
```

#### 2.2.8 Video Super-Resolution

BasicVSR / BasicVSR++ / IconVSR / RealBasicVSR

```shell
        --model-name basicvsr \
        --video ../resources/input/video_restoration/QUuC4vJs_000084_000094_400x320.mp4 \
        --result-out-dir ../resources/output/video_restoration/demo_video_restoration_basicvsr_res.mp4
```

EDVR

```shell
        --model-name edvr \
        --extra-parameters window_size=5 \

```

EDVR

```shell
        --result-out-dir ../resources/output/video_restoration/demo_video_restoration_edvr_res.mp4
```

TDAN

```shell
        --model-name tdan \
        --model-setting 2
```

#### 2.2.9 Text-to-Image

```shell
        --model-name stable_diffusion \
        --text "A panda is having dinner at KFC" \
        --result-out-dir ../resources/output/text2image/demo_text2image_stable_diffusion_res.png
```

#### 2.2.10 3D-aware Generation

```shell

--text "A panda is having dinner at KFC" \
```

```shell
    --model-name eg3d \
    --result-out-dir ../resources/output/eg3d-output
```

#### 2.2.11 Image Restoration

```shell
        --model-name nafnet \
        --result-out-dir ../resources/output/restoration/demo_restoration_nafnet_res.png
```

# MixMIM

> [MixMIM: Mixed and Masked Image Modeling for Efficient Visual Representation Learning](https://arxiv.org/abs/2205.13137)

In this study, we propose Mixed and Masked Image Modeling (MixMIM), a
simple but efficient MIM method that is applicable to various hierarchical Vision
Transformers. Existing MIM methods replace a random subset of input tokens with
a special [MASK] symbol and aim at reconstructing original image tokens from
the corrupted image. However, we find that using the [MASK] symbol greatly
slows down the training and causes training-finetuning inconsistency, due to the
large masking ratio (e.g., 40% in BEiT). In contrast, we replace the masked tokens
of one image with visible tokens of another image, i.e., creating a mixed image.
We then conduct dual reconstruction to reconstruct the original two images from
the mixed input, which significantly improves efficiency. While MixMIM can

be applied to various architectures, this paper explores a simpler but stronger
hierarchical Transformer, and scales with MixMIM-B, -L, and -H. Empirical
results demonstrate that MixMIM can learn high-quality visual representations
efficiently. Notably, MixMIM-B with 88M parameters achieves 85.1% top-1
accuracy on ImageNet-1K by pretraining for 600 epochs, setting a new record for
neural networks with comparable model sizes (e.g., ViT-B) among MIM methods.
Besides, its transferring performances on the other 6 datasets show MixMIM has
better FLOPs / performance tradeoff than previous MIM methods

<img src="https://user-images.githubusercontent.com/56866854/202853730-d26fb3d7-e5e8-487a-aad5-e3d4600cef87.png"/>
</div>

```python

</div>

```python

predict = inference_model('mixmim-base_mixmim-pre_8xb128-coslr-100e_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('mixmim_mixmim-base_16xb128-coslr-300e_in1k', pretrained=True)
```

Train:

```shell
python tools/train.py configs/mixmim/mixmim_mixmim-base_16xb128-coslr-300e_in1k.py
```

Test:

```shell

Train:

```shell
```

Test:

```shell
python tools/test.py configs/mixmim/benchmarks/mixmim-base_8xb128-coslr-100e_in1k.py https://download.openmmlab.com/mmselfsup/1.x/mixmim/mixmim-base-p16_16xb128-coslr-300e_in1k/mixmim-base-p16_ft-8xb128-coslr-100e_in1k/mixmim-base-p16_ft-8xb128-coslr-100e_in1k_20221208-41ecada9.pth
```

| Model                                        | Params (M) | Flops (G) |                         Config                          |                                      Download                                      |
| :------------------------------------------- | :--------: | :-------: | :-----------------------------------------------------: | :--------------------------------------------------------------------------------: |

| `mixmim_mixmim-base_16xb128-coslr-300e_in1k` |   114.67   |   16.35   | [config](mixmim_mixmim-base_16xb128-coslr-300e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/mixmim/mixmim-base-p16_16xb128-coslr-300e_in1k/mixmim-base-p16_16xb128-coslr-300e_in1k_20221208-44fe8d2c.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mixmim/mixmim-base-p16_16xb128-coslr-300e_in1k/mixmim-base-p16_16xb128-coslr-300e_in1k_20221208-44fe8d2c.json) |

| `mixmim-base_mixmim-pre_8xb128-coslr-100e_in1k` | [MIXMIM](https://download.openmmlab.com/mmselfsup/1.x/mixmim/mixmim-base-p16_16xb128-coslr-300e_in1k/mixmim-base-p16_16xb128-coslr-300e_in1k_20221208-44fe8d2c.pth) |   88.34    |   16.35   |   84.63   | [config](benchmarks/mixmim-base_8xb128-coslr-100e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/mixmim/mixmim-base-p16_16xb128-coslr-300e_in1k/mixmim-base-p16_ft-8xb128-coslr-100e_in1k/mixmim-base-p16_ft-8xb128-coslr-100e_in1k_20221208-41ecada9.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mixmim/mixmim-base-p16_16xb128-coslr-300e_in1k/mixmim-base-p16_ft-8xb128-coslr-100e_in1k/mixmim-base-p16_ft-8xb128-coslr-100e_in1k_20221208-41ecada9.json) |

```bibtex
@article{MixMIM2022,
  author  = {Jihao Liu, Xin Huang, Yu Liu, Hongsheng Li},
  journal = {arXiv:2205.13137},
  title   = {MixMIM: Mixed and Masked Image Modeling for Efficient Visual Representation Learning},
  year    = {2022},
}
```

# Torchserve Deployment

In order to serve an `MMPretrain` model with [`TorchServe`](https://pytorch.org/serve/), you can follow the steps:

## 1. Convert model from MMPretrain to TorchServe

```shell
python tools/torchserve/mmpretrain2torchserve.py ${CONFIG_FILE} ${CHECKPOINT_FILE} \
```

```{note}
${MODEL_STORE} needs to be an absolute path to a folder.
```

Example:

```shell
python tools/torchserve/mmpretrain2torchserve.py \
  configs/resnet/resnet18_8xb32_in1k.py \
  checkpoints/resnet18_8xb32_in1k_20210831-fbbb1da6.pth \
  --output-folder ./checkpoints \
  --model-name resnet18_in1k
```

## 2. Build `mmpretrain-serve` docker image

```shell
docker build -t mmpretrain-serve:latest docker/serve/
```

## 3. Run `mmpretrain-serve`

In order to run in GPU, you need to install [nvidia-docker](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html). You can omit the `--gpus` argument in order to run in GPU.

Example:

```shell
--name mar \
--cpus 8 \
--mount type=bind,source=`realpath ./checkpoints`,target=/home/model-server/model-store \
mmpretrain-serve:latest
```

```{note}
`realpath ./checkpoints` points to the absolute path of "./checkpoints", and you can replace it with the absolute path where you store torchserve models.
```

[Read the docs](https://github.com/pytorch/serve/blob/master/docs/rest_api.md) about the Inference (8080), Management (8081) and Metrics (8082) APis

## 4. Test deployment

```shell

```

```shell
curl http://127.0.0.1:8080/predictions/${MODEL_NAME} -T demo/demo.JPEG
```

```json
{
  "pred_label": 58,
  "pred_score": 0.38102269172668457,
  "pred_class": "water snake"
}
```

And you can use `test_torchserver.py` to compare result of TorchServe and PyTorch, and visualize them.

```shell
python tools/torchserve/test_torchserver.py ${IMAGE_FILE} ${CONFIG_FILE} ${CHECKPOINT_FILE} ${MODEL_NAME}
[--inference-addr ${INFERENCE_ADDR}] [--device ${DEVICE}]
```

Example:

```shell
python tools/torchserve/test_torchserver.py \
  demo/demo.JPEG \
  resnet18_in1k
```

# Customize Logging

MMAction2 produces a lot of logs during the running process, such as loss, iteration time, learning rate, etc. In this section, we will introduce you how to output custom log. More details about the logging system, please refer to [MMEngine Tutorial](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/logging.html).

- [Customize Logging](#customize-logging)
  - [Flexible Logging System](#flexible-logging-system)
  - [Customize log](#customize-log)
  - [Export the debug log](#export-the-debug-log)

## Flexible Logging System

The MMAction2 logging system is configured by the `LogProcessor` in [default_runtime](https://github.com/open-mmlab/mmaction2/tree/main/configs/_base_/default_runtime.py) by default, which is equivalent to:

```python
log_processor = dict(type='LogProcessor', window_size=20, by_epoch=True)
```

```python
```

By default, the `LogProcessor` captures all fields that begin with `loss` returned by `model.forward`. For instance, in the following model, `loss1` and `loss2` will be logged automatically without any additional configuration.

```python

class ToyModel(BaseModel):
    def __init__(self) -> None:
        self.linear = nn.Linear(1, 1)

    def forward(self, img, label, mode):
        feat = self.linear(img)
        loss1 = (feat - label).pow(2)
        loss2 = (feat - label).abs()
        return dict(loss1=loss1, loss2=loss2)
```

The output log follows the following format:

```
08/21 02:58:41 - mmengine - INFO - Epoch(train) [1][10/25]  lr: 1.0000e-02  eta: 0:00:00  time: 0.0019  data_time: 0.0004  loss1: 0.8381  loss2: 0.9007  loss: 1.7388

```
08/21 02:58:41 - mmengine - INFO - Epoch(train) [1][20/25]  lr: 1.0000e-02  eta: 0:00:00  time: 0.0029  data_time: 0.0010  loss1: 0.1978  loss2: 0.4312  loss: 0.6290
```

`LogProcessor` will output the log in the following format:

- The prefix of the log:
  - epoch mode(`by_epoch=True`): `Epoch(train) [{current_epoch}/{current_iteration}]/{dataloader_length}`
  - iteration mode(`by_epoch=False`): `Iter(train) [{current_iteration}/{max_iteration}]`
- Learning rate (`lr`): The learning rate of the last iteration.
- Time:
  - `time`: The averaged time for inference of the last `window_size` iterations.
  - `data_time`: The averaged time for loading data of the last `window_size` iterations.
  - `eta`: The estimated time of arrival to finish the training.

- `data_time`: The averaged time for loading data of the last `window_size` iterations.
- Loss: The averaged loss output by model of the last `window_size` iterations.

log_processor outputs the epoch based log by default(`by_epoch=True`). To get an expected log matched with the `train_cfg`, we should set the same value for `by_epoch` in `train_cfg` and `log_processor`.
```

Based on the rules above, the code snippet will count the average value of the loss1 and the loss2 every 20 iterations. More types of statistical methods, please refer to [mmengine.runner.LogProcessor](mmengine.runner.LogProcessor).

## Customize log

The logging system could not only log the `loss`, `lr`, .etc but also collect and output the custom log. For example, if we want to statistic the intermediate loss:

The `ToyModel` calculate `loss_tmp` in forward, but don't save it into the return dict.

```python

```python
from mmengine.logging import MessageHub

        loss_tmp = (feat - label).abs()
        loss = loss_tmp.pow(2)

        # update the intermediate `loss_tmp` in the message hub
        message_hub.update_scalar('train/loss_tmp', loss_tmp.sum())
```

Add the `loss_tmp` into the config:

```python
    type='LogProcessor',
    window_size=20,
    by_epoch=True,
    custom_cfg=[
        # statistic the loss_tmp with the averaged value
            dict(
                data_src='loss_tmp',
                window_size=20,
                method_name='mean')
        ])
```

# statistic the loss_tmp with the averaged value
            dict(
        ])
```

The `loss_tmp` will be added to the output log:

```
08/21 03:40:31 - mmengine - INFO - Epoch(train) [1][10/25]  lr: 1.0000e-02  eta: 0:00:00  time: 0.0026  data_time: 0.0008  loss_tmp: 0.0097  loss: 0.0000
08/21 03:40:31 - mmengine - INFO - Epoch(train) [1][20/25]  lr: 1.0000e-02  eta: 0:00:00  time: 0.0028  data_time: 0.0013  loss_tmp: 0.0065  loss: 0.0000
```

## Export the debug log

To export the debug log to the `work_dir`, you can set log_level in config file as follows:

```
log_level='DEBUG'
```

```
08/21 18:16:22 - mmengine - DEBUG - Get class `LocalVisBackend` from "vis_backend" registry in "mmengine"
08/21 18:16:22 - mmengine - DEBUG - An `LocalVisBackend` instance is built from registry, its implementation can be found in mmengine.visualization.vis_backend

08/21 18:16:22 - mmengine - DEBUG - Get class `RuntimeInfoHook` from "hook" registry in "mmengine"
08/21 18:16:22 - mmengine - DEBUG - An `RuntimeInfoHook` instance is built from registry, its implementation can be found in mmengine.hooks.runtime_info_hook
08/21 18:16:22 - mmengine - DEBUG - Get class `IterTimerHook` from "hook" registry in "mmengine"
...
```

Besides, logs of different ranks will be saved in `debug` mode if you are training your model with the shared storage. The hierarchy of the log is as follows:

```text
./tmp
├── tmp.log
├── tmp_rank1.log
├── tmp_rank2.log
├── tmp_rank3.log
├── tmp_rank4.log
├── tmp_rank5.log
├── tmp_rank6.log
└── tmp_rank7.log
...
└── tmp_rank63.log
```

The log of Multiple machines with independent storage:

```text
# device: 0:
work_dir/
└── exp_name_logs
    ├── exp_name.log

...
```

```text
work_dir/
    ├── exp_name_rank1.log
    ├── exp_name_rank2.log
    ├── exp_name_rank3.log
    ...
    └── exp_name_rank7.log

# device: 7:
work_dir/
    ├── exp_name_rank56.log
    ├── exp_name_rank57.log
    ├── exp_name_rank58.log
    ...
    └── exp_name_rank63.log
```

# OFA

> [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://arxiv.org/abs/2202.03052)

In this work, we pursue a unified paradigm for multimodal pretraining to break the scaffolds of complex task/modality-specific customization. We propose OFA, a Task-Agnostic and Modality-Agnostic framework that supports Task Comprehensiveness. OFA unifies a diverse set of cross-modal and unimodal tasks, including image generation, visual grounding, image captioning, image classification, language modeling, etc., in a simple sequence-to-sequence learning framework. OFA follows the instruction-based learning in both pretraining and finetuning stages, requiring no extra task-specific layers for downstream tasks. In comparison with the recent state-of-the-art vision & language models that rely on extremely large cross-modal datasets, OFA is pretrained on only 20M publicly available image-text pairs. Despite its simplicity and relatively small-scale training data, OFA achieves new SOTAs in a series of cross-modal tasks while attaining highly competitive performances on uni-modal tasks. Our further analysis indicates that OFA can also effectively transfer to unseen tasks and unseen domains.

<img src="https://user-images.githubusercontent.com/26739999/236164275-2429bf20-6e2a-4325-acc2-6117f9b53a53.png" width="80%"/>
</div>

```python

result = inference_model('ofa-base_3rdparty-finetuned_caption', 'demo/cat-dog.png')
print(result)
# {'pred_caption': 'a dog and a kitten sitting next to each other'}
```

Test:

```shell
python tools/test.py configs/ofa/ofa-base_finetuned_refcoco.py https://download.openmmlab.com/mmclassification/v1/ofa/ofa-base_3rdparty_refcoco_20230418-2797d3ab.pth
```

### Image Caption on COCO

```

| Model                                   | Params (M) | BLEU-4 | CIDER  |                 Config                  |                                               Download                                               |
| :-------------------------------------- | :--------: | :----: | :----: | :-------------------------------------: | :--------------------------------------------------------------------------------------------------: |
| `ofa-base_3rdparty-finetuned_caption`\* |   182.24   | 42.64  | 144.50 | [config](ofa-base_finetuned_caption.py) | [model](https://download.openmmlab.com/mmclassification/v1/ofa/ofa-base_3rdparty_coco-caption_20230418-de18914e.pth) |

*Models with * are converted from the [official repo](https://github.com/OFA-Sys/OFA). The config files of these models are only for inference. We haven't reproduce the training results.*

### Visual Grounding on RefCOCO

| Model                                   | Params (M) | Accuracy (testA) | Accuracy (testB) |                 Config                  |                                     Download                                     |
| :-------------------------------------- | :--------: | :--------------: | :--------------: | :-------------------------------------: | :------------------------------------------------------------------------------: |
| `ofa-base_3rdparty-finetuned_refcoco`\* |   182.24   |      90.49       |      83.63       | [config](ofa-base_finetuned_refcoco.py) | [model](https://download.openmmlab.com/mmclassification/v1/ofa/ofa-base_3rdparty_refcoco_20230418-2797d3ab.pth) |

### Visual Question Answering on VQAv2

| Model                               | Params (M) | Accuracy |               Config                |                                                     Download                                                     |
| :---------------------------------- | :--------: | :------: | :---------------------------------: | :--------------------------------------------------------------------------------------------------------------: |
| `ofa-base_3rdparty-finetuned_vqa`\* |   182.24   |  78.00   | [config](ofa-base_finetuned_vqa.py) | [model](https://download.openmmlab.com/mmclassification/v1/ofa/ofa-base_3rdparty_coco-vqa_20230418-f38539a5.pth) |
| `ofa-base_3rdparty-zeroshot_vqa`\*  |   182.24   |  58.32   | [config](ofa-base_zeroshot_vqa.py)  | [model](https://download.openmmlab.com/mmclassification/v1/ofa/ofa-base_3rdparty_pretrain_20230418-dccfc07f.pth) |

```bibtex
@article{wang2022ofa,
  author    = {Peng Wang and
               An Yang and
               Rui Men and
               Junyang Lin and
               Shuai Bai and
               Zhikang Li and
               Jianxin Ma and
               Chang Zhou and
               Jingren Zhou and
               Hongxia Yang},
  title     = {OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence
               Learning Framework},
  volume    = {abs/2202.03052},
  year      = {2022}
}
```

# 2D Animal Keypoint Dataset

- [Animal-Pose](#animal-pose) \[ [Homepage](https://sites.google.com/view/animal-pose/) \]
- [AP-10K](#ap-10k) \[ [Homepage](https://github.com/AlexTheBad/AP-10K/) \]
- [Horse-10](#horse-10) \[ [Homepage](http://www.mackenziemathislab.org/horse10) \]
- [MacaquePose](#macaquepose) \[ [Homepage](http://www.pri.kyoto-u.ac.jp/datasets/macaquepose/index.html) \]
- [Vinegar Fly](#vinegar-fly) \[ [Homepage](https://github.com/jgraving/DeepPoseKit-Data) \]
- [Desert Locust](#desert-locust) \[ [Homepage](https://github.com/jgraving/DeepPoseKit-Data) \]
- [Grévy’s Zebra](#grvys-zebra) \[ [Homepage](https://github.com/jgraving/DeepPoseKit-Data) \]
- [ATRW](#atrw) \[ [Homepage](https://cvwc2019.github.io/challenge.html) \]

- [Animal Kingdom](#Animal-Kindom) \[ [Homepage](https://openaccess.thecvf.com/content/CVPR2022/html/Ng_Animal_Kingdom_A_Large_and_Diverse_Dataset_for_Animal_Behavior_CVPR_2022_paper.html) \]

## Animal-Pose

<details>

```bibtex
}
```

booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
}
```

  <img src="https://user-images.githubusercontent.com/100993824/227796953-95ae1e30-5323-43f8-9a19-c4c2326e9835.png" height="200px">
</div>

For [Animal-Pose](https://sites.google.com/view/animal-pose/) dataset, we prepare the dataset as follows:

1. Download the images of [PASCAL VOC2012](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/#data), especially the five categories (dog, cat, sheep, cow, horse), which we use as trainval dataset.
2. Download the [test-set](https://drive.google.com/drive/folders/1DwhQobZlGntOXxdm7vQsE4bqbFmN3b9y?usp=sharing) images with raw annotations (1000 images, 5 categories).

3. We have pre-processed the annotations to make it compatible with MMPose. Please download the annotation files from [annotations](https://download.openmmlab.com/mmpose/datasets/animalpose_annotations.tar). If you would like to generate the annotations by yourself, please check our dataset parsing [codes](/tools/dataset_converters/parse_animalpose_dataset.py).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── animalpose
        │
        │-- VOC2012
        │   │-- Annotations
        │   │-- ImageSets
        │   │-- JPEGImages
        │   │-- SegmentationClass
        │   │-- SegmentationObject
        │
        │-- animalpose_image_part2
        │   │-- cat
        │   │-- cow
        │   │-- dog

│   │-- JPEGImages
        │
        │   │-- cat
        │   │-- cow
        │   │-- dog
        │   │-- horse
        │   │-- sheep
        │
        │   │-- animalpose_train.json
        │   |-- animalpose_val.json
        │   |-- animalpose_trainval.json
        │   │-- animalpose_test.json
        │
        │-- PASCAL2011_animal_annotation
        │   │-- cat
        │   │   |-- 2007_000528_1.xml
        │   │   |-- 2007_000549_1.xml
        │   │   │-- ...
        │   │-- cow
        │   │-- dog
        │   │-- horse
        │   │-- sheep
        │
        │-- annimalpose_anno2
        │   │-- cat
        │   │   |-- ca1.xml
        │   │   |-- ca2.xml
        │   │   │-- ...
        │   │-- cow
        │   │-- dog
        │   │-- horse
        │   │-- sheep

```

The official dataset does not provide the official train/val/test set split.

│   │   │-- ...
        │   │-- cow
        │   │-- dog
        │   │-- horse
        │   │-- sheep

```

We choose the images from PascalVOC for train & val. In total, we have 3608 images and 5117 annotations for train+val, where
2798 images with 4000 annotations are used for training, and 810 images with 1117 annotations are used for validation.
Those images from other sources (1000 images with 1000 annotations) are used for testing.

## AP-10K

<details>

```bibtex
}
```

}
```

  <img src="https://user-images.githubusercontent.com/100993824/227797151-091dc21a-d944-49c9-8b62-cc47fa89e69f.png" height="200px">
</div>

For [AP-10K](https://github.com/AlexTheBad/AP-10K/) dataset, images and annotations can be downloaded from [download](https://drive.google.com/file/d/1-FNNGcdtAQRehYYkGY1y4wzFNg4iWNad/view?usp=sharing).
Note, this data and annotation data is for non-commercial use only.

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── ap10k
        │   │-- ap10k-train-split1.json
        │   |-- ap10k-train-split2.json
        │   |-- ap10k-train-split3.json
        │   │-- ap10k-val-split1.json
        │   |-- ap10k-val-split2.json
        │   |-- ap10k-val-split3.json

│   |-- ap10k-train-split2.json
        │   |-- ap10k-test-split1.json
        │   |-- ap10k-test-split2.json
        │   |-- ap10k-test-split3.json
        │-- data
        │   │-- 000000000001.jpg
        │   │-- 000000000002.jpg
        │   │-- ...

```

The annotation files in 'annotation' folder contains 50 labeled animal species. There are total 10,015 labeled images with 13,028 instances in the AP-10K dataset. We randonly split them into train, val, and test set following the ratio of 7:1:2.

## Horse-10

<details>

```bibtex

```bibtex
}
```

  <img src="https://user-images.githubusercontent.com/100993824/227797934-32bc1b2c-7957-4a29-94df-8e431842ab3b.png" height="200px">
</div>

For [Horse-10](http://www.mackenziemathislab.org/horse10) dataset, images can be downloaded from [download](http://www.mackenziemathislab.org/horse10).

</div>

Please download the annotation files from [horse10_annotations](https://download.openmmlab.com/mmpose/datasets/horse10_annotations.tar). Note, this data and annotation data is for non-commercial use only, per the authors (see http://horse10.deeplabcut.org for more information).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── horse10
        │   │-- horse10-train-split1.json
        │   |-- horse10-train-split2.json
        │   |-- horse10-train-split3.json
        │   │-- horse10-test-split1.json
        │   |-- horse10-test-split2.json
        │   |-- horse10-test-split3.json
        │-- labeled-data
        │   │-- BrownHorseinShadow
        │   │-- BrownHorseintoshadow
        │   │-- ...

```

│   |-- horse10-test-split3.json
        │   │-- ...

```

## MacaquePose

<details>

```bibtex
}
```

  <img src="https://user-images.githubusercontent.com/100993824/227799576-f10f8469-9432-4139-beb4-195037dee72c.png" height="200px">
</div>

}
```

</div>

For [MacaquePose](http://www.pri.kyoto-u.ac.jp/datasets/macaquepose/index.html) dataset, images can be downloaded from [download](http://www.pri.kyoto-u.ac.jp/datasets/macaquepose/index.html).
Please download the annotation files from [macaque_annotations](https://download.openmmlab.com/mmpose/datasets/macaque_annotations.tar).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── macaque
        │   │-- macaque_train.json
        │   |-- macaque_test.json
        │-- images
        │   │-- 01418849d54b3005.jpg
        │   │-- 0142d1d1a6904a70.jpg
        │   │-- 01ef2c4c260321b7.jpg
        │   │-- 020a1c75c8c85238.jpg
        │   │-- 020b1506eef2557d.jpg
        │   │-- ...

```

│   │-- 0142d1d1a6904a70.jpg
        │   │-- ...

```

Since the official dataset does not provide the test set, we randomly select 12500 images for training, and the rest for evaluation (see [code](/tools/dataset/parse_macaquepose_dataset.py)).

## Vinegar Fly

<details>
<summary align="right"><a href="https://www.nature.com/articles/s41592-018-0234-5">Vinegar Fly (Nature Methods'2019)</a></summary>

```bibtex
@article{pereira2019fast,
  title={Fast animal pose estimation using deep neural networks},
  author={Pereira, Talmo D and Aldarondo, Diego E and Willmore, Lindsay and Kislin, Mikhail and Wang, Samuel S-H and Murthy, Mala and Shaevitz, Joshua W},
  journal={Nature methods},
  volume={16},
  pages={117--125},
  publisher={Nature Publishing Group}
}
```

journal={Nature methods},
}
```

  <img src="https://user-images.githubusercontent.com/100993824/227802774-bb4e4ef2-2ade-42ad-80f1-97f2a7faa9e2.png" height="200px">
</div>

For [Vinegar Fly](https://github.com/jgraving/DeepPoseKit-Data) dataset, images can be downloaded from [vinegar_fly_images](https://download.openmmlab.com/mmpose/datasets/vinegar_fly_images.tar).
Please download the annotation files from [vinegar_fly_annotations](https://download.openmmlab.com/mmpose/datasets/vinegar_fly_annotations.tar).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── fly
        │   │-- fly_train.json
        │   |-- fly_test.json
        │-- images
        │   │-- 0.jpg
        │   │-- 1.jpg
        │   │-- 2.jpg
        │   │-- 3.jpg

        │-- images
        │   │-- 0.jpg
        │   │-- 1.jpg
        │   │-- 2.jpg
        │   │-- 3.jpg
        │   │-- ...

```

Since the official dataset does not provide the test set, we randomly select 90% images for training, and the rest (10%) for evaluation (see [code](/tools/dataset_converters/parse_deepposekit_dataset.py)).

## Desert Locust

<details>

```bibtex
}
```

}
```

  <img src="https://user-images.githubusercontent.com/100993824/227802779-09d0ec8c-8971-4c67-a315-e2d1355f7f72.png" height="200px">
</div>

For [Desert Locust](https://github.com/jgraving/DeepPoseKit-Data) dataset, images can be downloaded from [locust_images](https://download.openmmlab.com/mmpose/datasets/locust_images.tar).
Please download the annotation files from [locust_annotations](https://download.openmmlab.com/mmpose/datasets/locust_annotations.tar).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── locust
        │   │-- locust_train.json
        │   |-- locust_test.json
        │-- images
        │   │-- 0.jpg
        │   │-- 1.jpg
        │   │-- 2.jpg
        │   │-- 3.jpg
        │   │-- ...

```

│   │-- locust_train.json
        │-- images
        │   │-- 0.jpg
        │   │-- 1.jpg
        │   │-- 2.jpg
        │   │-- 3.jpg
        │   │-- ...

```

## Grévy’s Zebra

<details>

```bibtex
}
```

}
```

  <img src="https://user-images.githubusercontent.com/100993824/227802783-ace952bb-1ff9-4720-80a8-c63cc9e714b6.png" height="200px">
</div>

For [Grévy’s Zebra](https://github.com/jgraving/DeepPoseKit-Data) dataset, images can be downloaded from [zebra_images](https://download.openmmlab.com/mmpose/datasets/zebra_images.tar).
Please download the annotation files from [zebra_annotations](https://download.openmmlab.com/mmpose/datasets/zebra_annotations.tar).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── zebra
        │   │-- zebra_train.json
        │   |-- zebra_test.json
        │-- images
        │   │-- 0.jpg
        │   │-- 1.jpg
        │   │-- 2.jpg
        │   │-- 3.jpg
        │   │-- ...

```

│   │-- zebra_train.json
        │-- images
        │   │-- 0.jpg
        │   │-- 1.jpg
        │   │-- 2.jpg
        │   │-- 3.jpg
        │   │-- ...

```

## ATRW

<details>
<summary align="right"><a href="https://arxiv.org/abs/1906.05586">ATRW (ACM MM'2020)</a></summary>

```bibtex
@inproceedings{li2020atrw,
  title={ATRW: A Benchmark for Amur Tiger Re-identification in the Wild},
  author={Li, Shuyuan and Li, Jianguo and Tang, Hanlin and Qian, Rui and Lin, Weiyao},
  booktitle={Proceedings of the 28th ACM International Conference on Multimedia},
  pages={2590--2598},
}
```

booktitle={Proceedings of the 28th ACM International Conference on Multimedia},
}
```

  <img src="https://user-images.githubusercontent.com/100993824/227797386-fce99241-8a0e-4a40-a179-dad013e6c5a4.png" height="200px">
</div>

ATRW captures images of the Amur tiger (also known as Siberian tiger, Northeast-China tiger) in the wild.
For [ATRW](https://cvwc2019.github.io/challenge.html) dataset, please download images from
[Pose_train](https://lilablobssc.blob.core.windows.net/cvwc2019/train/atrw_pose_train.tar.gz),
[Pose_val](https://lilablobssc.blob.core.windows.net/cvwc2019/train/atrw_pose_val.tar.gz), and
[Pose_test](https://lilablobssc.blob.core.windows.net/cvwc2019/test/atrw_pose_test.tar.gz).
Note that in the ATRW official annotation files, the key "file_name" is written as "filename". To make it compatible with
other coco-type json files, we have modified this key.

Please download the modified annotation files from [atrw_annotations](https://download.openmmlab.com/mmpose/datasets/atrw_annotations.tar).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── atrw
        │   │-- keypoint_train.json
        │   │-- keypoint_val.json
        │   │-- keypoint_trainval.json
        │-- images
        │   │-- train
        │   │   │-- 000002.jpg
        │   │   │-- 000003.jpg
        │   │   │-- ...
        │   │-- val
        │   │   │-- 000001.jpg
        │   │   │-- 000013.jpg
        │   │   │-- ...
        │   │-- test
        │   │   │-- 000000.jpg
        │   │   │-- 000004.jpg
        │   │   │-- ...

```

## Animal Kingdom

<details>

│   │   │-- 000013.jpg
        │   │   │-- ...
        │   │-- test
        │   │   │-- ...

```

<details>
<summary align="right"><a href="https://arxiv.org/abs/2204.08129">Animal Kingdom (CVPR'2022)</a></summary>
  <img src="https://github.com/open-mmlab/mmpose/assets/53283758/8591989e-91fa-4f6e-99c8-48b614de862e" height="200px">
</div>

```bibtex
@inproceedings{Ng_2022_CVPR,
    author    = {Ng, Xun Long and Ong, Kian Eng and Zheng, Qichen and Ni, Yun and Yeo, Si Yong and Liu, Jun},
    title     = {Animal Kingdom: A Large and Diverse Dataset for Animal Behavior Understanding},
    pages     = {19023-19034}
 }
```

 }
```

For [Animal Kingdom](https://github.com/sutdcv/Animal-Kingdom) dataset, images can be downloaded from [here](https://forms.office.com/pages/responsepage.aspx?id=drd2NJDpck-5UGJImDFiPVRYpnTEMixKqPJ1FxwK6VZUQkNTSkRISTNORUI2TDBWMUpZTlQ5WUlaSyQlQCN0PWcu).
Please Extract dataset under {MMPose}/data, and make them look like this:

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── ak
        |--annotations
        │  │-- ak_P1
        │  │   │-- train.json
        │  │   │-- test.json
        │  │-- ak_P2
        │  │-- ak_P3_amphibian
        │  │-- ak_P3_bird
        │  │-- ak_P3_fish

│  │-- ak_P3_amphibian
        │  │-- ak_P3_mammal
        │  │-- ak_P3_reptile
        │      │-- train.json
        │      │-- test.json
        │-- images
        │   │-- AAACXZTV
        │   │   │--AAACXZTV_f000059.jpg
        │   │   │--...
        │   │-- AAAUILHH
        │   │   │--AAAUILHH_f000098.jpg
        │   │   │--...
        │   │-- ...
```

<details>

```bibtex
}
```

<details>

```bibtex

author={Hang Yu and Yufei Xu and Jing Zhang and Wei Zhao and Ziyu Guan and Dacheng Tao},
}
```

| [pose_hrnet_w32](/configs/animal_2d_keypoint/topdown_heatmap/ap10k/td-hm_hrnet-w32_8xb64-210e_ap10k-256x256.py) |  256x256   | 0.722 |      0.935      |      0.789      |     0.557      |     0.729      | [ckpt](https://download.openmmlab.com/mmpose/animal/hrnet/hrnet_w32_ap10k_256x256-18aac840_20211029.pth) | [log](https://download.openmmlab.com/mmpose/animal/hrnet/hrnet_w32_ap10k_256x256-18aac840_20211029.log.json) |
| [pose_hrnet_w48](/configs/animal_2d_keypoint/topdown_heatmap/ap10k/td-hm_hrnet-w48_8xb64-210e_ap10k-256x256.py) |  256x256   | 0.728 |      0.936      |      0.802      |     0.577      |     0.735      | [ckpt](https://download.openmmlab.com/mmpose/animal/hrnet/hrnet_w48_ap10k_256x256-d95ab412_20211029.pth) | [log](https://download.openmmlab.com/mmpose/animal/hrnet/hrnet_w48_ap10k_256x256-d95ab412_20211029.log.json) |

We provide several profiling tools to benchmark our models.

## profile with dataset

Download the dataset below or create your own dataset.

```bash
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
```

Profiling your model with `profile_throughput.py`

```bash
python profile_throughput.py \
 ShareGPT_V3_unfiltered_cleaned_split.json \
 /path/to/your/model \
 --concurrency 64
```

## profile without dataset

`profile_generation.py` perform benchmark with dummy data.

```bash
python profile_generation.py \
 --concurrency 8 --input_seqlen 0 --output_seqlen 2048
```

## profile serving

Tools above profile models with Python API. `profile_serving.py` is used to do benchmark on serving.

```bash

python profile_serving.py \

```bash

    ${TritonServerAddress} \
    /path/to/tokenizer \
    ShareGPT_V3_unfiltered_cleaned_split.json \
    --concurrency 64
```

# RepMLP

> [RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition](https://arxiv.org/abs/2105.01883)

We propose RepMLP, a multi-layer-perceptron-style neural network building block for image recognition, which is composed of a series of fully-connected (FC) layers. Compared to convolutional layers, FC layers are more efficient, better at modeling the long-range dependencies and positional patterns, but worse at capturing the local structures, hence usually less favored for image recognition. We propose a structural re-parameterization technique that adds local prior into an FC to make it powerful for image recognition. Specifically, we construct convolutional layers inside a RepMLP during training and merge them into the FC for inference. On CIFAR, a simple pure-MLP model shows performance very close to CNN. By inserting RepMLP in traditional CNN, we improve ResNets by 1.8% accuracy on ImageNet, 2.9% for face recognition, and 2.3% mIoU on Cityscapes with lower FLOPs. Our intriguing findings highlight that combining the global representational capacity and positional perception of FC with the local prior of convolution can improve the performance of neural network with faster speed on both the tasks with translation invariance (e.g., semantic segmentation) and those with aligned images and positional patterns (e.g., face recognition).

<img src="https://user-images.githubusercontent.com/18586273/155455288-a17a5c48-11af-4b74-995a-cf7183f0e2d2.png" width="80%"/>
</div>

```python

model = get_model('repmlp-base_3rdparty_8xb64_in1k', pretrained=True)
```

```python

```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/repmlp/repmlp-base_8xb64_in1k.py https://download.openmmlab.com/mmclassification/v0/repmlp/repmlp-base_3rdparty_8xb64_in1k_20220330-1cb1f11b.pth
```

```bash
```

```bash

```bash
```

```python
from mmpretrain.models import RepMLPNet

backbone = RepMLPNet(arch='B', img_size=224, reparam_conv_kernels=(1, 3))
```

| Model                                     |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                  Config                   |                               Download                                |
| :---------------------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :---------------------------------------: | :-------------------------------------------------------------------: |

| `repmlp-base_3rdparty_8xb64_in1k`\*       | From scratch |   68.24    |   6.71    |   80.41   |   95.14   |    [config](repmlp-base_8xb64_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/repmlp/repmlp-base_3rdparty_8xb64_in1k_20220330-1cb1f11b.pth) |
| `repmlp-base_3rdparty_8xb64_in1k-256px`\* | From scratch |   96.45    |   9.69    |   81.11   |   95.50   | [config](repmlp-base_8xb64_in1k-256px.py) | [model](https://download.openmmlab.com/mmclassification/v0/repmlp/repmlp-base_3rdparty_8xb64_in1k-256px_20220330-7c5a91ce.pth) |

*Models with * are converted from the [official repo](https://github.com/DingXiaoH/RepMLP/blob/072d8516beba83d75dfe6ebb12f625abad4b53d5/repmlpnet.py#L278). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@article{ding2021repmlp,
  title={Repmlp: Re-parameterizing convolutions into fully-connected layers for image recognition},

```bibtex
  author={Ding, Xiaohan and Xia, Chunlong and Zhang, Xiangyu and Chu, Xiaojie and Han, Jungong and Ding, Guiguang},
  journal={arXiv preprint arXiv:2105.01883},
}
```

# A New Dataset and Boundary-Attention Semantic Segmentation for Face Parsing

<details>
<summary align="right"><a href="https://aaai.org/ojs/index.php/AAAI/article/view/6832/6686">LaPa (AAAI'2020)</a></summary>

```bibtex
@inproceedings{liu2020new,
  title={A New Dataset and Boundary-Attention Semantic Segmentation for Face Parsing.},
  author={Liu, Yinglu and Shi, Hailin and Shen, Hao and Si, Yue and Wang, Xiaobo and Mei, Tao},
  booktitle={AAAI},
  pages={11637--11644},
}
```

# FreeAnchor

> [FreeAnchor: Learning to Match Anchors for Visual Object Detection](https://arxiv.org/abs/1909.02466)

Modern CNN-based object detectors assign anchors for ground-truth objects under the restriction of object-anchor Intersection-over-Unit (IoU). In this study, we propose a learning-to-match approach to break IoU restriction, allowing objects to match anchors in a flexible manner. Our approach, referred to as FreeAnchor, updates hand-crafted anchor assignment to "free" anchor matching by formulating detector training as a maximum likelihood estimation (MLE) procedure. FreeAnchor targets at learning features which best explain a class of objects in terms of both classification and localization. FreeAnchor is implemented by optimizing detection customized likelihood and can be fused with CNN-based detectors in a plug-and-play manner. Experiments on COCO demonstrate that FreeAnchor consistently outperforms their counterparts with significant margins.

<img src="https://user-images.githubusercontent.com/40661020/143886006-8374bf9b-28af-442e-9abf-eb16562bb7d3.png"/>
</div>

|  Backbone   |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                      Config                      |                                                                                                                                                                                   Download                                                                                                                                                                                    |

| :---------: | :-----: | :-----: | :------: | :------------: | :----: | :----------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|    R-50     | pytorch |   1x    |   4.9    |      18.4      |  38.7  |    [config](./freeanchor_r50_fpn_1x_coco.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/free_anchor/retinanet_free_anchor_r50_fpn_1x_coco/retinanet_free_anchor_r50_fpn_1x_coco_20200130-0f67375f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/free_anchor/retinanet_free_anchor_r50_fpn_1x_coco/retinanet_free_anchor_r50_fpn_1x_coco_20200130_095625.log.json)               |

|    R-101    | pytorch |   1x    |   6.8    |      14.9      |  40.3  |    [config](./freeanchor_r101_fpn_1x_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/free_anchor/retinanet_free_anchor_r101_fpn_1x_coco/retinanet_free_anchor_r101_fpn_1x_coco_20200130-358324e6.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/free_anchor/retinanet_free_anchor_r101_fpn_1x_coco/retinanet_free_anchor_r101_fpn_1x_coco_20200130_100723.log.json)             |
| X-101-32x4d | pytorch |   1x    |   8.1    |      11.1      |  41.9  | [config](./freeanchor_x101-32x4d_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/free_anchor/retinanet_free_anchor_x101_32x4d_fpn_1x_coco/retinanet_free_anchor_x101_32x4d_fpn_1x_coco_20200130-d4846968.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/free_anchor/retinanet_free_anchor_x101_32x4d_fpn_1x_coco/retinanet_free_anchor_x101_32x4d_fpn_1x_coco_20200130_095627.log.json) |

- We use 8 GPUs with 2 images/GPU.
- For more settings and models, please refer to the [official repo](https://github.com/zhangxiaosong18/FreeAnchor).

```latex
@inproceedings{zhang2019freeanchor,
  title   =  {{FreeAnchor}: Learning to Match Anchors for Visual Object Detection},
  author  =  {Zhang, Xiaosong and Wan, Fang and Liu, Chang and Ji, Rongrong and Ye, Qixiang},
  booktitle =  {Neural Information Processing Systems},
  year    =  {2019}
}
```

# MMPretrain Deployment

- [MMPretrain Deployment](#mmpretrain-deployment)
    - [Install mmpretrain](#install-mmpretrain)
  - [Model Specification](#model-specification)

[MMPretrain](https://github.com/open-mmlab/mmpretrain) aka `mmpretrain` is an open-source image classification toolbox based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com) project.

### Install mmpretrain

Please follow this [quick guide](https://github.com/open-mmlab/mmpretrain/tree/main#installation) to install mmpretrain.

```shell

```shell
```

You can use [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/tree/main/tools/deploy.py) to convert mmpretrain models to the specified backend models. Its detailed usage can be learned from [here](https://github.com/open-mmlab/mmdeploy/tree/main/docs/en/02-how-to-run/convert_model.md#usage).

The command below shows an example about converting `resnet18` model to onnx model that can be inferred by ONNX Runtime.

```shell

# download resnet18 model from mmpretrain model zoo

```shell

mim download mmpretrain --config resnet18_8xb32_in1k --dest .

# convert mmpretrain model to onnxruntime model with dynamic shape
    configs/mmpretrain/classification_onnxruntime_dynamic.py \
    resnet18_8xb32_in1k.py \
    resnet18_8xb32_in1k_20210831-fbbb1da6.pth \
    --work-dir mmdeploy_models/mmpretrain/ort \
    --show \
```

It is crucial to specify the correct deployment config during model conversion. We've already provided builtin deployment config [files](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmpretrain) of all supported backends for mmpretrain. The config filename pattern is:

```
classification_{backend}-{precision}_{static | dynamic}_{shape}.py
```

```
```

- **{backend}:** inference backend, such as onnxruntime, tensorrt, pplnn, ncnn, openvino, coreml and etc.

Therefore, in the above example, you can also convert `resnet18` to other backend models by changing the deployment config file `classification_onnxruntime_dynamic.py` to [others](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmpretrain), e.g., converting to tensorrt-fp16 model by `classification_tensorrt-fp16_dynamic-224x224-224x224.py`.

```{tip}
When converting mmpretrain models to tensorrt models, --device should be set to "cuda"
```

## Model Specification

```

The converted model locates in the working directory like `mmdeploy_models/mmpretrain/ort` in the previous example. It includes:

```
mmdeploy_models/mmpretrain/ort
```

in which,

The whole package **mmdeploy_models/mmpretrain/ort** is defined as **mmdeploy SDK model**, i.e., **mmdeploy SDK model** includes both backend model and inference meta information.

Take the previous converted `end2end.onnx` model as an example, you can use the following code to inference the model.

```python

```python

deploy_cfg = 'configs/mmpretrain/classification_onnxruntime_dynamic.py'
model_cfg = './resnet18_8xb32_in1k.py'
backend_model = ['./mmdeploy_models/mmpretrain/ort/end2end.onnx']
image = 'tests/data/tiger.jpeg'

    output_file='output_classification.png')
```

```python
from mmdeploy_runtime import Classifier
import cv2

img = cv2.imread('tests/data/tiger.jpeg')

classifier = Classifier(model_path='./mmdeploy_models/mmpretrain/ort', device_name='cpu', device_id=0)
result = classifier(img)
# show inference result
for label_id, score in result:
    print(label_id, score)
```

| Model                                                                                              | TorchScript | ONNX Runtime | TensorRT | ncnn | PPLNN | OpenVINO |
| :------------------------------------------------------------------------------------------------- | :---------: | :----------: | :------: | :--: | :---: | :------: |
| [ResNet](https://github.com/open-mmlab/mmpretrain/tree/main/configs/resnet)                        |      Y      |      Y       |    Y     |  Y   |   Y   |    Y     |
| [ResNeXt](https://github.com/open-mmlab/mmpretrain/tree/main/configs/resnext)                      |      Y      |      Y       |    Y     |  Y   |   Y   |    Y     |
| [SE-ResNet](https://github.com/open-mmlab/mmpretrain/tree/main/configs/seresnet)                   |      Y      |      Y       |    Y     |  Y   |   Y   |    Y     |

| [MobileNetV2](https://github.com/open-mmlab/mmpretrain/tree/main/configs/mobilenet_v2)             |      Y      |      Y       |    Y     |  Y   |   Y   |    Y     |
| [ShuffleNetV1](https://github.com/open-mmlab/mmpretrain/tree/main/configs/shufflenet_v1)           |      Y      |      Y       |    Y     |  Y   |   Y   |    Y     |
| [ShuffleNetV2](https://github.com/open-mmlab/mmpretrain/tree/main/configs/shufflenet_v2)           |      Y      |      Y       |    Y     |  Y   |   Y   |    Y     |
| [VisionTransformer](https://github.com/open-mmlab/mmpretrain/tree/main/configs/vision_transformer) |      Y      |      Y       |    Y     |  Y   |   ?   |    Y     |

| [SwinTransformer](https://github.com/open-mmlab/mmpretrain/tree/main/configs/swin_transformer)     |      Y      |      Y       |    Y     |  N   |   ?   |    Y     |
| [MobileOne](https://github.com/open-mmlab/mmpretrain/tree/main/configs/mobileone)                  |      Y      |      Y       |    Y     |  Y   |   ?   |    Y     |
| [EfficientNet](https://github.com/open-mmlab/mmpretrain/tree/main/configs/efficientnet)            |      Y      |      Y       |    Y     |  N   |   ?   |    Y     |
| [Conformer](https://github.com/open-mmlab/mmpretrain/tree/main/configs/conformer)                  |      Y      |      Y       |    Y     |  N   |   ?   |    Y     |

| [EfficientFormer](https://github.com/open-mmlab/mmpretrain/tree/main/configs/efficientformer)      |      Y      |      Y       |    Y     |  N   |   ?   |    Y     |

- [MacaquePose](#macaquepose) \[ [Homepage](http://pri.ehub.kyoto-u.ac.jp/datasets/macaquepose/index.html) \]

        │
        │   │-- cat
        │   │-- cow
        │   │-- dog
        │   │-- horse
        │   │-- sheep
        │
        │
        │   │-- cat
        │   │   │-- ...
        │   │-- cow
        │   │-- dog
        │   │-- horse
        │   │-- sheep
        │
        │   │-- cat
        │   │   │-- ...
        │   │-- cow
        │   │-- dog
        │   │-- horse
        │   │-- sheep
```

│   │   │-- ...
        │   │-- cow
        │   │-- dog
        │   │-- horse
        │   │-- sheep
```

## AP-10K

<details>

```bibtex
}
```

        │-- data
        │   │-- ...
```

<details>

```bibtex

</div>

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
        │   │-- ...
```

        │   │-- ...
```

<details>

```bibtex
}
```

</div>

}
```

</div>

For [MacaquePose](http://pri.ehub.kyoto-u.ac.jp/datasets/macaquepose/index.html) dataset, images can be downloaded from [download](http://pri.ehub.kyoto-u.ac.jp/datasets/macaquepose/download.php).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
        │-- images
        │   │-- ...
```

        │   │-- ...
```

<details>

```bibtex
}
```

        │-- images
        │   │-- 0.jpg
        │   │-- 1.jpg
        │   │-- 2.jpg
        │   │-- 3.jpg
        │   │-- ...
```

<details>

```bibtex
}
```

}
```

</div>

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── locust
        │-- images
        │   │-- 0.jpg
        │   │-- 1.jpg
        │   │-- 2.jpg
        │   │-- 3.jpg
        │   │-- ...
```

        │-- images
        │   │-- 0.jpg
        │   │-- 1.jpg
        │   │-- 2.jpg
        │   │-- 3.jpg
        │   │-- ...
```

<details>

```bibtex
}
```

}
```

</div>

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── zebra
        │-- images
        │   │-- 0.jpg
        │   │-- 1.jpg
        │   │-- 2.jpg
        │   │-- 3.jpg
        │   │-- ...
```

        │-- images
        │   │-- 0.jpg
        │   │-- 1.jpg
        │   │-- 2.jpg
        │   │-- 3.jpg
        │   │-- ...
```

## ATRW

<details>

```bibtex
}
```

}
```

</div>

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── atrw
        │-- images
        │   │-- train
        │   │   │-- ...
        │   │-- val
        │   │   │-- ...
        │   │-- test
        │   │   │-- ...
```

<details>

        │   │   │-- ...
        │   │-- test
        │   │   │-- ...
```

<details>
</div>

```bibtex
 }
```

# VoxelPose: Towards Multi-Camera 3D Human Pose Estimation in Wild Environment

<details>
<summary align="right"><a href="https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123460188.pdf">VoxelPose (ECCV'2020)</a></summary>

```bibtex
@inproceedings{tumultipose,
  title={VoxelPose: Towards Multi-Camera 3D Human Pose Estimation in Wild Environment},
  author={Tu, Hanyue and Wang, Chunyu and Zeng, Wenjun},
  booktitle={ECCV},
}
```

author={Tu, Hanyue and Wang, Chunyu and Zeng, Wenjun},
}
```

We present VoxelPose to estimate 3D poses of multiple people from multiple camera views. In contrast to the previous efforts which require to establish cross-view correspondence based on noisy and incomplete 2D pose estimates, VoxelPose directly operates in the 3D space therefore avoids making incorrect decisions in each camera view. To achieve this goal, features in all camera views are aggregated in the 3D voxel space and fed into Cuboid Proposal Network (CPN) to localize all people. Then we propose Pose Regression Network (PRN) to estimate a detailed 3D pose for each proposal. The approach is robust to occlusion which occurs frequently in practice. Without bells and whistles, it outperforms the previous methods on several public datasets.

<img src="https://user-images.githubusercontent.com/15977946/148036780-b73a878c-7244-480b-b56e-ac739396856b.png">
</div>

# 准备 Classic5 数据集

```bibtex
@article{zhang2017beyond,
  title={Beyond a {Gaussian} denoiser: Residual learning of deep {CNN} for image denoising},
  author={Zhang, Kai and Zuo, Wangmeng and Chen, Yunjin and Meng, Deyu and Zhang, Lei},
  volume={26},
  pages={3142-3155},
}
```

测试数据集可以从 [此处](https://github.com/cszn/DnCNN/tree/master/testsets) 下载。

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── Classic5
```

> [Masked-attention Mask Transformer for Universal Image Segmentation](https://arxiv.org/abs/2112.01527)

<a href="https://github.com/facebookresearch/Mask2Former">Official Repo</a>

<a href="https://github.com/open-mmlab/mmdetection/blob/3.x/mmdet/models/dense_heads/mask2former_head.py">Code Snippet</a>

### Usage

### Usage

- Mask2Former model needs to install [MMDetection](https://github.com/open-mmlab/mmdetection) first.

```shell
pip install "mmdet>=3.0.0rc4"
```

| Method      | Backbone       | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device | mIoU  | mIoU(ms+flip) |                                                                                                                                                    config | download                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |

| Mask2Former | R-50-D32       | 512x1024  | 90000   |     5.67 | 9.17           | A100   | 80.44 |             - |                      [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mask2former/mask2former_r50_8xb2-90k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_r50_8xb2-90k_cityscapes-512x1024/mask2former_r50_8xb2-90k_cityscapes-512x1024_20221202_140802-ffd9d750.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_r50_8xb2-90k_cityscapes-512x1024/mask2former_r50_8xb2-90k_cityscapes-512x1024_20221202_140802.json)                                                                                      |

| Mask2Former | R-101-D32      | 512x1024  | 90000   |     6.81 | 7.11           | A100   | 80.80 |             - |                     [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mask2former/mask2former_r101_8xb2-90k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_r101_8xb2-90k_cityscapes-512x1024/mask2former_r101_8xb2-90k_cityscapes-512x1024_20221130_031628-43e68666.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_r101_8xb2-90k_cityscapes-512x1024/mask2former_r101_8xb2-90k_cityscapes-512x1024_20221130_031628.json))                                                                                 |

| Mask2Former | Swin-T         | 512x1024  | 90000   |     6.36 | 7.18           | A100   | 81.71 |             - |                   [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mask2former/mask2former_swin-t_8xb2-90k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_swin-t_8xb2-90k_cityscapes-512x1024/mask2former_swin-t_8xb2-90k_cityscapes-512x1024_20221127_144501-36c59341.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_swin-t_8xb2-90k_cityscapes-512x1024/mask2former_swin-t_8xb2-90k_cityscapes-512x1024_20221127_144501.json))                                                                         |

| Mask2Former | Swin-S         | 512x1024  | 90000   |     8.09 | 5.57           | A100   | 82.57 |             - |                   [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mask2former/mask2former_swin-s_8xb2-90k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_swin-s_8xb2-90k_cityscapes-512x1024/mask2former_swin-s_8xb2-90k_cityscapes-512x1024_20221127_143802-9ab177f6.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_swin-s_8xb2-90k_cityscapes-512x1024/mask2former_swin-s_8xb2-90k_cityscapes-512x1024_20221127_143802.json))                                                                         |

| Mask2Former | Swin-B (in22k) | 512x1024  | 90000   |    10.89 | 4.32           | A100   | 83.52 |             - | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mask2former/mask2former_swin-b-in22k-384x384-pre_8xb2-90k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_swin-b-in22k-384x384-pre_8xb2-90k_cityscapes-512x1024/mask2former_swin-b-in22k-384x384-pre_8xb2-90k_cityscapes-512x1024_20221203_045030-9a86a225.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_swin-b-in22k-384x384-pre_8xb2-90k_cityscapes-512x1024/mask2former_swin-b-in22k-384x384-pre_8xb2-90k_cityscapes-512x1024_20221203_045030.json)) |

| Mask2Former | Swin-L (in22k) | 512x1024  | 90000   |    15.83 | 2.86           | A100   | 83.65 |             - | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mask2former/mask2former_swin-l-in22k-384x384-pre_8xb2-90k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_swin-l-in22k-384x384-pre_8xb2-90k_cityscapes-512x1024/mask2former_swin-l-in22k-384x384-pre_8xb2-90k_cityscapes-512x1024_20221202_141901-28ad20f1.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_swin-l-in22k-384x384-pre_8xb2-90k_cityscapes-512x1024/mask2former_swin-l-in22k-384x384-pre_8xb2-90k_cityscapes-512x1024_20221202_141901.json)) |

### ADE20K

### ADE20K

| Method      | Backbone       | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device | mIoU  | mIoU(ms+flip) |                                                                                                                                                config | download                                                                                                                                                                                                                                                                                                                                                                                                                                                    |

| Mask2Former | R-50-D32       | 512x512   | 160000  |     3.31 | 26.59          | A100   | 47.87 |             - |                      [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mask2former/mask2former_r50_8xb2-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_r50_8xb2-160k_ade20k-512x512/mask2former_r50_8xb2-160k_ade20k-512x512_20221204_000055-2d1f55f1.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_r50_8xb2-160k_ade20k-512x512/mask2former_r50_8xb2-160k_ade20k-512x512_20221204_000055.json))                                                                                     |

| Mask2Former | R-101-D32      | 512x512   | 160000  |     4.09 | 22.97          | A100   | 48.60 |             - |                     [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mask2former/mask2former_r101_8xb2-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_r101_8xb2-160k_ade20k-512x512/mask2former_r101_8xb2-160k_ade20k-512x512_20221203_233905-b7135890.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_r101_8xb2-160k_ade20k-512x512/mask2former_r101_8xb2-160k_ade20k-512x512_20221203_233905.json))                                                                                 |

| Mask2Former | Swin-T         | 512x512   | 160000  |     3826 | 23.82          | A100   | 48.66 |             - |                   [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mask2former/mask2former_swin-t_8xb2-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_swin-t_8xb2-160k_ade20k-512x512/mask2former_swin-t_8xb2-160k_ade20k-512x512_20221203_234230-7d64e5dd.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_swin-t_8xb2-160k_ade20k-512x512/mask2former_swin-t_8xb2-160k_ade20k-512x512_20221203_234230.json))                                                                         |

| Mask2Former | Swin-S         | 512x512   | 160000  |     3.74 | 19.69          | A100   | 51.24 |             - |                   [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mask2former/mask2former_swin-s_8xb2-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_swin-s_8xb2-160k_ade20k-512x512/mask2former_swin-s_8xb2-160k_ade20k-512x512_20221204_143905-e715144e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_swin-s_8xb2-160k_ade20k-512x512/mask2former_swin-s_8xb2-160k_ade20k-512x512_20221204_143905.json))                                                                         |

| Mask2Former | Swin-B         | 640x640   | 160000  |     5.66 | 12.48          | A100   | 52.44 |             - |  [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mask2former/mask2former_swin-b-in1k-384x384-pre_8xb2-160k_ade20k-640x640.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_swin-b-in1k-384x384-pre_8xb2-160k_ade20k-640x640/mask2former_swin-b-in1k-384x384-pre_8xb2-160k_ade20k-640x640_20221129_125118-a4a086d2.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_swin-b-in1k-384x384-pre_8xb2-160k_ade20k-640x640/mask2former_swin-b-in1k-384x384-pre_8xb2-160k_ade20k-640x640_20221129_125118.json))     |

| Mask2Former | Swin-B (in22k) | 640x640   | 160000  |     5.66 | 12.43          | A100   | 53.90 |             - | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mask2former/mask2former_swin-b-in22k-384x384-pre_8xb2-160k_ade20k-640x640.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_swin-b-in22k-384x384-pre_8xb2-160k_ade20k-640x640/mask2former_swin-b-in22k-384x384-pre_8xb2-160k_ade20k-640x640_20221203_235230-7ec0f569.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_swin-b-in22k-384x384-pre_8xb2-160k_ade20k-640x640/mask2former_swin-b-in22k-384x384-pre_8xb2-160k_ade20k-640x640_20221203_235230.json)) |

| Mask2Former | Swin-L (in22k) | 640x640   | 160000  |     8.86 | 8.81           | A100   | 56.01 |             - | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mask2former/mask2former_swin-l-in22k-384x384-pre_8xb2-160k_ade20k-640x640.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_swin-l-in22k-384x384-pre_8xb2-160k_ade20k-640x640/mask2former_swin-l-in22k-384x384-pre_8xb2-160k_ade20k-640x640_20221203_235933-7120c214.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mask2former/mask2former_swin-l-in22k-384x384-pre_8xb2-160k_ade20k-640x640/mask2former_swin-l-in22k-384x384-pre_8xb2-160k_ade20k-640x640_20221203_235933.json)) |

Note:

- All experiments of Mask2Former are implemented with 8 A100 GPUs with 2 samplers per GPU.

Note:

- As mentioned at [the official repo](https://github.com/facebookresearch/Mask2Former/issues/5), the results of Mask2Former are relatively not stable, the result of Mask2Former(swin-s) on ADE20K dataset in the table is the medium result obtained by training 5 times following the suggestion of the author.
- The ResNet backbones utilized in MaskFormer models are standard `ResNet` rather than `ResNetV1c`.
- Test time augmentation is not supported in MMSegmentation 1.x version yet, we would add "ms+flip" results as soon as possible.

```bibtex
@inproceedings{cheng2021mask2former,
}

author={Bowen Cheng and Ishan Misra and Alexander G. Schwing and Alexander Kirillov and Rohit Girdhar},
}
}
```

# 边、端设备测试结果

这里给出我们边、端设备的测试结论，用户可以直接通过 [model profiling](../02-how-to-run/profile_model.md) 获得自己环境的结果。

## 软硬件环境

## mmpretrain 模型

tips:

1. ImageNet-1k 数据集较大，仅使用一部分测试（8000/50000）
2. 边、端设备发热会降频，因此耗时实际上会波动。这里给出运行一段时间后、稳定的数值。这个结果更贴近实际需求

## mmocr 检测

## mmpose 模型

tips:

- 测试 pose_hrnet 用的是 AnimalPose 的 test dataset，而非 val dataset

## mmseg

tips:

## mmseg

tips:

- fcn 用 512x1024 尺寸运行正常。Cityscapes 数据集 1024x2048 分辨率会导致设备重启

## 其他模型

- mmdet 需要手动把模型拆成两部分。因为
  - snpe 源码中 `onnx_to_ir.py` 仅能解析输入，`ir_to_dlc.py` 还不支持 topk
  - UDO （用户自定义算子）无法和 `snpe-onnx-to-dlc` 配合使用
- mmagic 模型
  - srcnn 需要 cubic resize，snpe 不支持
  - esrgan 可正常转换，但加载模型会导致设备重启
- mmrotate 依赖 [e2cnn](https://pypi.org/project/e2cnn/) ，需要手动安装 [其 Python3.6
  兼容分支](https://github.com/QUVA-Lab/e2cnn)

# 数据变换（待更新）

按照惯例，我们使用 `Dataset` 和 `DataLoader` 进行多进程的数据加载。`Dataset` 返回字典类型的数据，数据内容为模型 `forward` 方法的各个参数。由于在目标检测中，输入的图像数据具有不同的大小，我们在 `MMCV` 里引入一个新的 `DataContainer` 类去收集和分发不同大小的输入数据。更多细节请参考[这里](https://github.com/open-mmlab/mmcv/blob/master/mmcv/parallel/data_container.py)。

数据的准备流程和数据集是解耦的。通常一个数据集定义了如何处理标注数据（annotations）信息，而一个数据流程定义了准备一个数据字典的所有步骤。一个流程包括一系列的操作，每个操作都把一个字典作为输入，然后再输出一个新的字典给下一个变换操作。

我们在下图展示了一个经典的数据处理流程。蓝色块是数据处理操作，随着数据流程的处理，每个操作都可以在结果字典中加入新的键（标记为绿色）或更新现有的键（标记为橙色）。

![pipeline figure](../../../resources/data_pipeline.png)

这些操作可以分为数据加载（data loading）、预处理（pre-processing）、格式变化（formatting）和测试时数据增强（test-time augmentation）。

下面的例子是 `Faster R-CNN` 的一个流程：

```python

]
    dict(
        ])
]
```

# 测试

## 单机单卡测试

你可以使用 `tools/test.py` 在电脑上用 CPU 或是 GPU 进行模型的测试。

以下是测试脚本的完整用法：

```shell
```

默认情况下，MMPretrain 会自动调用你的 GPU 进行测试。如果你有 GPU 但仍想使用 CPU 进行测试，请设置环境变量 `CUDA_VISIBLE_DEVICES` 为空或者 -1 来对禁用 GPU。

```bash
```
````

| 参数                                  | 描述                                                                                                                                                                |

| `CHECKPOINT_FILE`                     | 权重文件路径（支持 http 链接，你可以在[这里](https://mmpretrain.readthedocs.io/en/latest/modelzoo_statistics.html)寻找需要的权重文件）。                            |
| `--work-dir WORK_DIR`                 | 用来保存测试指标结果的文件夹。                                                                                                                                      |
| `--out OUT`                           | 用来保存测试输出的文件。                                                                                                                                            |
| `--out-item OUT_ITEM`                 | 指定测试输出文件的内容，可以为 "pred" 或 "metrics"，其中 "pred" 表示保存所有模型输出，这些数据可以用于离线测评；"metrics" 表示输出测试指标。默认为 "pred"。         |

| `--show-dir SHOW_DIR`                 | 用于保存可视化预测结果图像的文件夹。                                                                                                                                |
| `--show`                              | 在窗口中显示预测结果图像。                                                                                                                                          |
| `--interval INTERVAL`                 | 每隔多少样本进行一次预测结果可视化。                                                                                                                                |

| `--wait-time WAIT_TIME`               | 每个窗口的显示时间（单位为秒）。                                                                                                                                    |
| `--tta`                               | 是否开启 Test-Time-Aug (TTA). 如果配置文件有 `tta_pipeline` 和 `tta_model`，将使用这些配置指定 TTA transforms，并且决定如何融合 TTA 的结果。 否则，通过平均分类分数使用 flip TTA。 |

## 单机多卡测试

## 单机多卡测试

```shell
```

| 参数              | 描述                                                                                                                                     |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `CONFIG_FILE`     | 配置文件的路径。                                                                                                                         |
| `CHECKPOINT_FILE` | 权重文件路径（支持 http 链接，你可以在[这里](https://mmpretrain.readthedocs.io/en/latest/modelzoo_statistics.html)寻找需要的权重文件）。 |
| `GPU_NUM`         | 使用的 GPU 数量。                                                                                                                        |

| `[PY_ARGS]`       | `tools/test.py` 支持的其他可选参数，参见[上文](#单机单卡测试)。                                                                          |

```shell
```

如果你希望使用不同的 GPU 进行多项测试任务，可以在启动时指定不同的通讯端口和不同的可用设备。

```shell
```

## 多机测试

如果你希望使用同一局域网下连接的多台电脑进行一个测试任务，可以使用如下命令：

在第一台机器上：

```shell
```

在第二台机器上：

```shell

在第一台机器上：

```shell
```

在第二台机器上：

```shell
```

| 环境变量      | 描述                                           |

如果你在 [slurm](https://slurm.schedmd.com/) 集群上，可以使用 `tools/slurm_test.sh` 脚本启动任务。

```shell
```

```shell
```

| 参数              | 描述                                                                                                                                     |
| `PARTITION`       | 使用的集群分区。                                                                                                                         |
| `JOB_NAME`        | 任务的名称，你可以随意起一个名字。                                                                                                       |
| `CONFIG_FILE`     | 配置文件路径。                                                                                                                           |

| 环境变量        | 描述                                                                                       |

# Rotated Faster R-CNN

> [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://papers.nips.cc/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf)

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/faster_rcnn.png" width="800"/>
</div>

State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet \[7\] and Fast R-CNN \[5\] have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully-convolutional network that simultaneously predicts object bounds and objectness scores at each position. RPNs are trained end-to-end to generate highquality region proposals, which are used by Fast R-CNN for detection. With a simple alternating optimization, RPN and Fast R-CNN can be trained to share convolutional features. For the very deep VGG-16 model \[19\], our detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007 (73.2% mAP) and 2012 (70.4% mAP) using 300 proposals per image.

```
}
```

扫描下方的二维码可关注 OpenMMLab 团队的 [知乎官方账号](https://www.zhihu.com/people/openmmlab)，加入 OpenMMLab 团队的 [官方交流 QQ 群](https://jq.qq.com/?_wv=1027&k=aCvMxdr3)，或通过添加微信“Open小喵Lab”加入官方交流微信群， 或者加入我们的 [Slack 社区](https://join.slack.com/t/mmocrworkspace/shared_invite/zt-1ifqhfla8-yKnLO_aKhVA2h71OrK8GZw)

<img src="https://raw.githubusercontent.com/open-mmlab/mmcv/master/docs/en/_static/zhihu_qrcode.jpg" height="400" />  <img src="https://raw.githubusercontent.com/open-mmlab/mmcv/master/docs/en/_static/qq_group_qrcode.jpg" height="400" />  <img src="https://raw.githubusercontent.com/open-mmlab/mmcv/master/docs/en/_static/wechat_qrcode.jpg" height="400" />
</div>

# DeepLabV3

> [Rethinking atrous convolution for semantic image segmentation](https://arxiv.org/abs/1706.05587)

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/decode_heads/aspp_head.py#L54">Code Snippet</a>

In this work, we revisit atrous convolution, a powerful tool to explicitly adjust filter's field-of-view as well as control the resolution of feature responses computed by Deep Convolutional Neural Networks, in the application of semantic image segmentation. To handle the problem of segmenting objects at multiple scales, we design modules which employ atrous convolution in cascade or in parallel to capture multi-scale context by adopting multiple atrous rates. Furthermore, we propose to augment our previously proposed Atrous Spatial Pyramid Pooling module, which probes convolutional features at multiple scales, with image-level features encoding global context and further boost performance. We also elaborate on implementation details and share our experience on training our system. The proposed \`DeepLabv3' system significantly improves over our previous DeepLab versions without DenseCRF post-processing and attains comparable performance with other state-of-art models on the PASCAL VOC 2012 semantic image segmentation benchmark.

<img alt="DEEPLABv3_ResNet-D8" src="https://user-images.githubusercontent.com/61172629/209305311-87ff9e36-b7cd-46d7-8b4c-9e26e10c27d0.jpg"/>
DEEPLABv3_ResNet-D8 model structure
</div>

| Method           | Backbone        | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                      | download                                                                                                                                                                                                                                                                                                                                                                                               |

| DeepLabV3        | R-50-D8         | 512x1024  |   40000 | 6.1      | 2.57           | V100   | 79.09 |         80.45 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r50-d8_4xb2-40k_cityscapes-512x1024.py)         | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x1024_40k_cityscapes/deeplabv3_r50-d8_512x1024_40k_cityscapes_20200605_022449-acadc2f8.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x1024_40k_cityscapes/deeplabv3_r50-d8_512x1024_40k_cityscapes_20200605_022449.log.json)                                 |

| DeepLabV3        | R-101-D8        | 512x1024  |   40000 | 9.6      | 1.92           | V100   | 77.12 |         79.61 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101-d8_4xb2-40k_cityscapes-512x1024.py)        | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x1024_40k_cityscapes/deeplabv3_r101-d8_512x1024_40k_cityscapes_20200605_012241-7fd3f799.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x1024_40k_cityscapes/deeplabv3_r101-d8_512x1024_40k_cityscapes_20200605_012241.log.json)                             |

| DeepLabV3        | R-50-D8         | 769x769   |   40000 | 6.9      | 1.11           | V100   | 78.58 |         79.89 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r50-d8_4xb2-40k_cityscapes-769x769.py)          | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_769x769_40k_cityscapes/deeplabv3_r50-d8_769x769_40k_cityscapes_20200606_113723-7eda553c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_769x769_40k_cityscapes/deeplabv3_r50-d8_769x769_40k_cityscapes_20200606_113723.log.json)                                     |

| DeepLabV3        | R-101-D8        | 769x769   |   40000 | 10.9     | 0.83           | V100   | 79.27 |         80.11 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101-d8_4xb2-40k_cityscapes-769x769.py)         | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_769x769_40k_cityscapes/deeplabv3_r101-d8_769x769_40k_cityscapes_20200606_113809-c64f889f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_769x769_40k_cityscapes/deeplabv3_r101-d8_769x769_40k_cityscapes_20200606_113809.log.json)                                 |

| DeepLabV3        | R-18-D8         | 512x1024  |   80000 | 1.7      | 13.78          | V100   | 76.70 |         78.27 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r18-d8_4xb2-80k_cityscapes-512x1024.py)         | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r18-d8_512x1024_80k_cityscapes/deeplabv3_r18-d8_512x1024_80k_cityscapes_20201225_021506-23dffbe2.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r18-d8_512x1024_80k_cityscapes/deeplabv3_r18-d8_512x1024_80k_cityscapes-20201225_021506.log.json)                                 |

| DeepLabV3        | R-50-D8         | 512x1024  |   80000 | -        | -              | V100   | 79.32 |         80.57 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r50-d8_4xb2-80k_cityscapes-512x1024.py)         | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x1024_80k_cityscapes/deeplabv3_r50-d8_512x1024_80k_cityscapes_20200606_113404-b92cfdd4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x1024_80k_cityscapes/deeplabv3_r50-d8_512x1024_80k_cityscapes_20200606_113404.log.json)                                 |

| DeepLabV3        | R-101-D8        | 512x1024  |   80000 | -        | -              | V100   | 80.20 |         81.21 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101-d8_4xb2-80k_cityscapes-512x1024.py)        | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x1024_80k_cityscapes/deeplabv3_r101-d8_512x1024_80k_cityscapes_20200606_113503-9e428899.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x1024_80k_cityscapes/deeplabv3_r101-d8_512x1024_80k_cityscapes_20200606_113503.log.json)                             |

| DeepLabV3 (FP16) | R-101-D8        | 512x1024  |   80000 | 5.75     | 3.86           | V100   | 80.48 |             - | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101-d8_4xb2-amp-80k_cityscapes-512x1024.py)    | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_fp16_512x1024_80k_cityscapes/deeplabv3_r101-d8_fp16_512x1024_80k_cityscapes_20200717_230920-774d9cec.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_fp16_512x1024_80k_cityscapes/deeplabv3_r101-d8_fp16_512x1024_80k_cityscapes_20200717_230920.log.json)         |

| DeepLabV3        | R-18-D8         | 769x769   |   80000 | 1.9      | 5.55           | V100   | 76.60 |         78.26 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r18-d8_4xb2-80k_cityscapes-769x769.py)          | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r18-d8_769x769_80k_cityscapes/deeplabv3_r18-d8_769x769_80k_cityscapes_20201225_021506-6452126a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r18-d8_769x769_80k_cityscapes/deeplabv3_r18-d8_769x769_80k_cityscapes-20201225_021506.log.json)                                     |

| DeepLabV3        | R-50-D8         | 769x769   |   80000 | -        | -              | V100   | 79.89 |         81.06 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r50-d8_4xb2-80k_cityscapes-769x769.py)          | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_769x769_80k_cityscapes/deeplabv3_r50-d8_769x769_80k_cityscapes_20200606_221338-788d6228.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_769x769_80k_cityscapes/deeplabv3_r50-d8_769x769_80k_cityscapes_20200606_221338.log.json)                                     |

| DeepLabV3        | R-101-D8        | 769x769   |   80000 | -        | -              | V100   | 79.67 |         80.81 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101-d8_4xb2-80k_cityscapes-769x769.py)         | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_769x769_80k_cityscapes/deeplabv3_r101-d8_769x769_80k_cityscapes_20200607_013353-60e95418.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_769x769_80k_cityscapes/deeplabv3_r101-d8_769x769_80k_cityscapes_20200607_013353.log.json)                                 |

| DeepLabV3        | R-101-D16-MG124 | 512x1024  |   40000 | 4.7      | 6.96           | V100   | 76.71 |         78.63 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101-d16-mg124_4xb2-40k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d16-mg124_512x1024_40k_cityscapes/deeplabv3_r101-d16-mg124_512x1024_40k_cityscapes_20200908_005644-67b0c992.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d16-mg124_512x1024_40k_cityscapes/deeplabv3_r101-d16-mg124_512x1024_40k_cityscapes-20200908_005644.log.json) |

| DeepLabV3        | R-101-D16-MG124 | 512x1024  |   80000 | -        | -              | V100   | 78.36 |         79.84 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101-d16-mg124_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d16-mg124_512x1024_80k_cityscapes/deeplabv3_r101-d16-mg124_512x1024_80k_cityscapes_20200908_005644-57bb8425.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d16-mg124_512x1024_80k_cityscapes/deeplabv3_r101-d16-mg124_512x1024_80k_cityscapes-20200908_005644.log.json) |

| DeepLabV3        | R-18b-D8        | 512x1024  |   80000 | 1.6      | 13.93          | V100   | 76.26 |         77.88 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r18b-d8_4xb2-80k_cityscapes-512x1024.py)        | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r18b-d8_512x1024_80k_cityscapes/deeplabv3_r18b-d8_512x1024_80k_cityscapes_20201225_094144-46040cef.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r18b-d8_512x1024_80k_cityscapes/deeplabv3_r18b-d8_512x1024_80k_cityscapes-20201225_094144.log.json)                             |

| DeepLabV3        | R-50b-D8        | 512x1024  |   80000 | 6.0      | 2.74           | V100   | 79.63 |         80.98 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r50b-d8_4xb2-80k_cityscapes-512x1024.py)        | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50b-d8_512x1024_80k_cityscapes/deeplabv3_r50b-d8_512x1024_80k_cityscapes_20201225_155148-ec368954.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50b-d8_512x1024_80k_cityscapes/deeplabv3_r50b-d8_512x1024_80k_cityscapes-20201225_155148.log.json)                             |

| DeepLabV3        | R-101b-D8       | 512x1024  |   80000 | 9.5      | 1.81           | V100   | 80.01 |         81.21 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101b-d8_4xb2-80k_cityscapes-512x1024.py)       | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101b-d8_512x1024_80k_cityscapes/deeplabv3_r101b-d8_512x1024_80k_cityscapes_20201226_171821-8fd49503.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101b-d8_512x1024_80k_cityscapes/deeplabv3_r101b-d8_512x1024_80k_cityscapes-20201226_171821.log.json)                         |

| DeepLabV3        | R-18b-D8        | 769x769   |   80000 | 1.8      | 5.79           | V100   | 75.63 |         77.51 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r18b-d8_4xb2-80k_cityscapes-769x769.py)         | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r18b-d8_769x769_80k_cityscapes/deeplabv3_r18b-d8_769x769_80k_cityscapes_20201225_094144-fdc985d9.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r18b-d8_769x769_80k_cityscapes/deeplabv3_r18b-d8_769x769_80k_cityscapes-20201225_094144.log.json)                                 |

| DeepLabV3        | R-50b-D8        | 769x769   |   80000 | 6.8      | 1.16           | V100   | 78.80 |         80.27 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r50b-d8_4xb2-80k_cityscapes-769x769.py)         | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50b-d8_769x769_80k_cityscapes/deeplabv3_r50b-d8_769x769_80k_cityscapes_20201225_155404-87fb0cf4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50b-d8_769x769_80k_cityscapes/deeplabv3_r50b-d8_769x769_80k_cityscapes-20201225_155404.log.json)                                 |

| DeepLabV3        | R-101b-D8       | 769x769   |   80000 | 10.7     | 0.82           | V100   | 79.41 |         80.73 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101b-d8_4xb2-80k_cityscapes-769x769.py)        | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101b-d8_769x769_80k_cityscapes/deeplabv3_r101b-d8_769x769_80k_cityscapes_20201226_190843-9142ee57.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101b-d8_769x769_80k_cityscapes/deeplabv3_r101b-d8_769x769_80k_cityscapes-20201226_190843.log.json)                             |

### ADE20K

### ADE20K

| Method    | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                           | download                                                                                                                                                                                                                                                                                                                                                   |

| DeepLabV3 | R-50-D8  | 512x512   |   80000 | 8.9      | 14.76          | V100   | 42.42 |         43.28 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r50-d8_4xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x512_80k_ade20k/deeplabv3_r50-d8_512x512_80k_ade20k_20200614_185028-0bb3f844.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x512_80k_ade20k/deeplabv3_r50-d8_512x512_80k_ade20k_20200614_185028.log.json)         |

| DeepLabV3 | R-101-D8 | 512x512   |   80000 | 12.4     | 10.14          | V100   | 44.08 |         45.19 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101-d8_4xb4-80k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x512_80k_ade20k/deeplabv3_r101-d8_512x512_80k_ade20k_20200615_021256-d89c7fa4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x512_80k_ade20k/deeplabv3_r101-d8_512x512_80k_ade20k_20200615_021256.log.json)     |

| DeepLabV3 | R-50-D8  | 512x512   |  160000 | -        | -              | V100   | 42.66 |         44.09 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r50-d8_4xb4-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x512_160k_ade20k/deeplabv3_r50-d8_512x512_160k_ade20k_20200615_123227-5d0ee427.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x512_160k_ade20k/deeplabv3_r50-d8_512x512_160k_ade20k_20200615_123227.log.json)     |

| DeepLabV3 | R-101-D8 | 512x512   |  160000 | -        | -              | V100   | 45.00 |         46.66 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101-d8_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x512_160k_ade20k/deeplabv3_r101-d8_512x512_160k_ade20k_20200615_105816-b1f72b3b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x512_160k_ade20k/deeplabv3_r101-d8_512x512_160k_ade20k_20200615_105816.log.json) |

| Method    | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                            | download                                                                                                                                                                                                                                                                                                                                                       |

| DeepLabV3 | R-50-D8  | 512x512   |   20000 | 6.1      | 13.88          | V100   | 76.17 |         77.42 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r50-d8_4xb4-20k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x512_20k_voc12aug/deeplabv3_r50-d8_512x512_20k_voc12aug_20200617_010906-596905ef.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x512_20k_voc12aug/deeplabv3_r50-d8_512x512_20k_voc12aug_20200617_010906.log.json)     |

| DeepLabV3 | R-101-D8 | 512x512   |   20000 | 9.6      | 9.81           | V100   | 78.70 |         79.95 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101-d8_4xb4-20k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x512_20k_voc12aug/deeplabv3_r101-d8_512x512_20k_voc12aug_20200617_010932-8d13832f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x512_20k_voc12aug/deeplabv3_r101-d8_512x512_20k_voc12aug_20200617_010932.log.json) |

| DeepLabV3 | R-50-D8  | 512x512   |   40000 | -        | -              | V100   | 77.68 |         78.78 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r50-d8_4xb4-40k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x512_40k_voc12aug/deeplabv3_r50-d8_512x512_40k_voc12aug_20200613_161546-2ae96e7e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x512_40k_voc12aug/deeplabv3_r50-d8_512x512_40k_voc12aug_20200613_161546.log.json)     |

| DeepLabV3 | R-101-D8 | 512x512   |   40000 | -        | -              | V100   | 77.92 |         79.18 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101-d8_4xb4-40k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x512_40k_voc12aug/deeplabv3_r101-d8_512x512_40k_voc12aug_20200613_161432-0017d784.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x512_40k_voc12aug/deeplabv3_r101-d8_512x512_40k_voc12aug_20200613_161432.log.json) |

| Method    | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                  | download                                                                                                                                                                                                                                                                                                                                                                               |

| DeepLabV3 | R-101-D8 | 480x480   |   40000 | 9.2      | 7.09           | V100   | 46.55 |         47.81 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101-d8_4xb4-40k_pascal-context-480x480.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_480x480_40k_pascal_context/deeplabv3_r101-d8_480x480_40k_pascal_context_20200911_204118-1aa27336.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_480x480_40k_pascal_context/deeplabv3_r101-d8_480x480_40k_pascal_context-20200911_204118.log.json) |

| DeepLabV3 | R-101-D8 | 480x480   |   80000 | -        | -              | V100   | 46.42 |         47.53 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101-d8_4xb4-80k_pascal-context-480x480.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_480x480_80k_pascal_context/deeplabv3_r101-d8_480x480_80k_pascal_context_20200911_170155-2a21fff3.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_480x480_80k_pascal_context/deeplabv3_r101-d8_480x480_80k_pascal_context-20200911_170155.log.json) |

| Method    | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                     | download                                                                                                                                                                                                                                                                                                                                                                                           |

| DeepLabV3 | R-101-D8 | 480x480   |   40000 | -        | -              | V100   | 52.61 |         54.28 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101-d8_4xb4-40k_pascal-context-59-480x480.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_480x480_40k_pascal_context_59/deeplabv3_r101-d8_480x480_40k_pascal_context_59_20210416_110332-cb08ea46.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_480x480_40k_pascal_context_59/deeplabv3_r101-d8_480x480_40k_pascal_context_59-20210416_110332.log.json) |

| DeepLabV3 | R-101-D8 | 480x480   |   80000 | -        | -              | V100   | 52.46 |         54.09 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101-d8_4xb4-80k_pascal-context-59-480x480.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_480x480_80k_pascal_context_59/deeplabv3_r101-d8_480x480_80k_pascal_context_59_20210416_113002-26303993.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_480x480_80k_pascal_context_59/deeplabv3_r101-d8_480x480_80k_pascal_context_59-20210416_113002.log.json) |

| Method    | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                 | download                                                                                                                                                                                                                                                                                                                                                                                           |

| DeepLabV3 | R-50-D8  | 512x512   |   20000 | 9.6      | 10.8           | V100   | 34.66 |         36.08 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r50-d8_4xb4-20k_coco-stuff10k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x512_4x4_20k_coco-stuff10k/deeplabv3_r50-d8_512x512_4x4_20k_coco-stuff10k_20210821_043025-b35f789d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x512_4x4_20k_coco-stuff10k/deeplabv3_r50-d8_512x512_4x4_20k_coco-stuff10k_20210821_043025.log.json)     |

| DeepLabV3 | R-101-D8 | 512x512   |   20000 | 13.2     | 8.7            | V100   | 37.30 |         38.42 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101-d8_4xb4-20k_coco-stuff10k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x512_4x4_20k_coco-stuff10k/deeplabv3_r101-d8_512x512_4x4_20k_coco-stuff10k_20210821_043025-c49752cb.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x512_4x4_20k_coco-stuff10k/deeplabv3_r101-d8_512x512_4x4_20k_coco-stuff10k_20210821_043025.log.json) |

| DeepLabV3 | R-50-D8  | 512x512   |   40000 | -        | -              | V100   | 35.73 |         37.09 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r50-d8_4xb4-40k_coco-stuff10k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x512_4x4_40k_coco-stuff10k/deeplabv3_r50-d8_512x512_4x4_40k_coco-stuff10k_20210821_043305-dc76f3ff.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x512_4x4_40k_coco-stuff10k/deeplabv3_r50-d8_512x512_4x4_40k_coco-stuff10k_20210821_043305.log.json)     |

| DeepLabV3 | R-101-D8 | 512x512   |   40000 | -        | -              | V100   | 37.81 |         38.80 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101-d8_4xb4-40k_coco-stuff10k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x512_4x4_40k_coco-stuff10k/deeplabv3_r101-d8_512x512_4x4_40k_coco-stuff10k_20210821_043305-636cb433.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x512_4x4_40k_coco-stuff10k/deeplabv3_r101-d8_512x512_4x4_40k_coco-stuff10k_20210821_043305.log.json) |

| Method    | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                   | download                                                                                                                                                                                                                                                                                                                                                                                                   |

| DeepLabV3 | R-50-D8  | 512x512   |   80000 | 9.6      | 10.8           | V100   | 39.38 |         40.03 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r50-d8_4xb4-80k_coco-stuff164k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x512_4x4_80k_coco-stuff164k/deeplabv3_r50-d8_512x512_4x4_80k_coco-stuff164k_20210709_163016-88675c24.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x512_4x4_80k_coco-stuff164k/deeplabv3_r50-d8_512x512_4x4_80k_coco-stuff164k_20210709_163016.log.json)         |

| DeepLabV3 | R-101-D8 | 512x512   |   80000 | 13.2     | 8.7            | V100   | 40.87 |         41.50 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101-d8_4xb4-80k_coco-stuff164k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x512_4x4_80k_coco-stuff164k/deeplabv3_r101-d8_512x512_4x4_80k_coco-stuff164k_20210709_201252-13600dc2.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x512_4x4_80k_coco-stuff164k/deeplabv3_r101-d8_512x512_4x4_80k_coco-stuff164k_20210709_201252.log.json)     |

| DeepLabV3 | R-50-D8  | 512x512   |  160000 | -        | -              | V100   | 41.09 |         41.69 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r50-d8_4xb4-160k_coco-stuff164k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x512_4x4_160k_coco-stuff164k/deeplabv3_r50-d8_512x512_4x4_160k_coco-stuff164k_20210709_163016-49f2812b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x512_4x4_160k_coco-stuff164k/deeplabv3_r50-d8_512x512_4x4_160k_coco-stuff164k_20210709_163016.log.json)     |

| DeepLabV3 | R-101-D8 | 512x512   |  160000 | -        | -              | V100   | 41.82 |         42.49 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101-d8_4xb4-160k_coco-stuff164k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x512_4x4_160k_coco-stuff164k/deeplabv3_r101-d8_512x512_4x4_160k_coco-stuff164k_20210709_155402-f035acfd.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x512_4x4_160k_coco-stuff164k/deeplabv3_r101-d8_512x512_4x4_160k_coco-stuff164k_20210709_155402.log.json) |

| DeepLabV3 | R-50-D8  | 512x512   |  320000 | -        | -              | V100   | 41.37 |         42.22 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r50-d8_4xb4-320k_coco-stuff164k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x512_4x4_320k_coco-stuff164k/deeplabv3_r50-d8_512x512_4x4_320k_coco-stuff164k_20210709_155403-51b21115.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r50-d8_512x512_4x4_320k_coco-stuff164k/deeplabv3_r50-d8_512x512_4x4_320k_coco-stuff164k_20210709_155403.log.json)     |

| DeepLabV3 | R-101-D8 | 512x512   |  320000 | -        | -              | V100   | 42.61 |         43.42 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3/deeplabv3_r101-d8_4xb4-320k_coco-stuff164k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x512_4x4_320k_coco-stuff164k/deeplabv3_r101-d8_512x512_4x4_320k_coco-stuff164k_20210709_155402-3cbca14d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3/deeplabv3_r101-d8_512x512_4x4_320k_coco-stuff164k/deeplabv3_r101-d8_512x512_4x4_320k_coco-stuff164k_20210709_155402.log.json) |

Note:

- `D-8` here corresponding to the output stride 8 setting for DeepLab series.

@article{chen2017rethinking,
  title={Rethinking atrous convolution for semantic image segmentation},

  author={Chen, Liang-Chieh and Papandreou, George and Schroff, Florian and Adam, Hartwig},
  journal={arXiv preprint arXiv:1706.05587},
}
```

# PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

> [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://arxiv.org/abs/1706.02413)

Few prior works study deep learning on point sets. PointNet by Qi et al. is a pioneer in this direction. However, by design PointNet does not capture local structures induced by the metric space points live in, limiting its ability to recognize fine-grained patterns and generalizability to complex scenes. In this work, we introduce a hierarchical neural network that applies PointNet recursively on a nested partitioning of the input point set. By exploiting metric space distances, our network is able to learn local features with increasing contextual scales. With further observation that point sets are usually sampled with varying densities, which results in greatly decreased performance for networks trained on uniform densities, we propose novel set learning layers to adaptively combine features from multiple scales. Experiments show that our network called PointNet++ is able to learn deep point set features efficiently and robustly. In particular, results significantly better than state-of-the-art have been obtained on challenging benchmarks of 3D point clouds.

<img src="https://user-images.githubusercontent.com/79644370/143885530-ae53ed38-8132-4bb7-85a7-d2577de7de3f.png" width="800"/>
</div>

We implement PointNet++ and provide the result and checkpoints on ScanNet and S3DIS datasets.

**Notice**: The original PointNet++ paper used step learning rate schedule. We discovered that cosine schedule achieves much better results and adopt it in our implementations. We also use a larger `weight_decay` factor because we find it consistently improves the performance.

|                                    Method                                     |   Input   |   Lr schd   | Mem (GB) | Inf time (fps) | mIoU (Val set) | mIoU (Test set) | Download                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |

| [PointNet++ (SSG)](./pointnet2_ssg_2xb16-cosine-200e_scannet-seg-xyz-only.py) |    XYZ    | cosine 200e |   1.9    |                |     53.91      |                 | [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointnet2/pointnet2_ssg_xyz-only_16x2_cosine_200e_scannet_seg-3d-20class/pointnet2_ssg_xyz-only_16x2_cosine_200e_scannet_seg-3d-20class_20210514_143628-4e341a48.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointnet2/pointnet2_ssg_xyz-only_16x2_cosine_200e_scannet_seg-3d-20class/pointnet2_ssg_xyz-only_16x2_cosine_200e_scannet_seg-3d-20class_20210514_143628.log.json) |

|     [PointNet++ (SSG)](./pointnet2_ssg_2xb16-cosine-200e_scannet-seg.py)      | XYZ+Color | cosine 200e |   1.9    |                |     54.44      |                 | [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointnet2/pointnet2_ssg_16x2_cosine_200e_scannet_seg-3d-20class/pointnet2_ssg_16x2_cosine_200e_scannet_seg-3d-20class_20210514_143644-ee73704a.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointnet2/pointnet2_ssg_16x2_cosine_200e_scannet_seg-3d-20class/pointnet2_ssg_16x2_cosine_200e_scannet_seg-3d-20class_20210514_143644.log.json)                                     |

| [PointNet++ (MSG)](./pointnet2_msg_2xb16-cosine-250e_scannet-seg-xyz-only.py) |    XYZ    | cosine 250e |   2.4    |                |     54.26      |                 | [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointnet2/pointnet2_msg_xyz-only_16x2_cosine_250e_scannet_seg-3d-20class/pointnet2_msg_xyz-only_16x2_cosine_250e_scannet_seg-3d-20class_20210514_143838-b4a3cf89.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointnet2/pointnet2_msg_xyz-only_16x2_cosine_250e_scannet_seg-3d-20class/pointnet2_msg_xyz-only_16x2_cosine_250e_scannet_seg-3d-20class_20210514_143838.log.json) |

|     [PointNet++ (MSG)](./pointnet2_msg_2xb16-cosine-250e_scannet-seg.py)      | XYZ+Color | cosine 250e |   2.4    |                |     55.05      |                 | [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointnet2/pointnet2_msg_16x2_cosine_250e_scannet_seg-3d-20class/pointnet2_msg_16x2_cosine_250e_scannet_seg-3d-20class_20210514_144009-24477ab1.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointnet2/pointnet2_msg_16x2_cosine_250e_scannet_seg-3d-20class/pointnet2_msg_16x2_cosine_250e_scannet_seg-3d-20class_20210514_144009.log.json)                                     |

- The original PointNet++ paper conducted experiments on the ScanNet V1 dataset, while later point cloud segmentor papers often used ScanNet V2. Following common practice, we report results on the ScanNet V2 dataset.

- Since ScanNet dataset doesn't provide ground-truth labels for the test set, users can only evaluate test set performance by submitting to its online benchmark [website](http://kaldir.vc.in.tum.de/scannet_benchmark/). However, users are only allowed to submit once every two weeks. Therefore, we currently report val set mIoU. Test set performance may be added in the future.

- To generate submission file for ScanNet online benchmark, you need to modify the ScanNet dataset's [config](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/_base_/datasets/scannet-seg.py#L126). Change `ann_file=data_root + 'scannet_infos_val.pkl'` to `ann_file=data_root + 'scannet_infos_test.pkl'`, and then simply run:

  ```shell
  python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} --format-only --options 'txt_prefix=exps/pointnet2_scannet_results'
  ```

```shell
  ```

  This will save the prediction results as `txt` files in `exps/pointnet2_scannet_results/`. Then, go to this folder and zip all files into `pn2_scannet.zip`. Now you can submit it to the online benchmark and wait for the test set result. More instructions can be found at their official [website](http://kaldir.vc.in.tum.de/scannet_benchmark/documentation#submission-policy).

### S3DIS

### S3DIS

|                              Method                               | Split  |  Lr schd   | Mem (GB) | Inf time (fps) | mIoU (Val set) |                                                                                                                                                                                                            Download                                                                                                                                                                                                            |

| [PointNet++ (SSG)](./pointnet2_ssg_2xb16-cosine-50e_s3dis-seg.py) | Area_5 | cosine 50e |   3.6    |                |     56.93      | [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointnet2/pointnet2_ssg_16x2_cosine_50e_s3dis_seg-3d-13class/pointnet2_ssg_16x2_cosine_50e_s3dis_seg-3d-13class_20210514_144205-995d0119.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointnet2/pointnet2_ssg_16x2_cosine_50e_s3dis_seg-3d-13class/pointnet2_ssg_16x2_cosine_50e_s3dis_seg-3d-13class_20210514_144205.log.json) |

| [PointNet++ (MSG)](./pointnet2_msg_2xb16-cosine-80e_s3dis-seg.py) | Area_5 | cosine 80e |   3.6    |                |     58.04      | [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointnet2/pointnet2_msg_16x2_cosine_80e_s3dis_seg-3d-13class/pointnet2_msg_16x2_cosine_80e_s3dis_seg-3d-13class_20210514_144307-b2059817.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointnet2/pointnet2_msg_16x2_cosine_80e_s3dis_seg-3d-13class/pointnet2_msg_16x2_cosine_80e_s3dis_seg-3d-13class_20210514_144307.log.json) |

Since PointNet++ testing adopts sliding patch inference which involves random point sampling, and the test script uses fixed random seeds while the random seeds of validation in training are not fixed, the test results may be slightly different from the results reported above.

```latex
@inproceedings{qi2017pointnet++,
  title={PointNet++ deep hierarchical feature learning on point sets in a metric space},
  author={Qi, Charles R and Yi, Li and Su, Hao and Guibas, Leonidas J},
  booktitle={Proceedings of the 31st International Conference on Neural Information Processing Systems},
  pages={5105--5114},
}
```

# W4A16 LLM Model Deployment

LMDeploy supports LLM model inference of 4-bit weight, with the minimum requirement for NVIDIA graphics cards being sm80, such as A10, A100, Geforce 30/40 series.

Before proceeding with the inference, please ensure that lmdeploy is installed.

```shell
```

## 4-bit LLM model Inference

You can download the pre-quantized 4-bit weight models from LMDeploy's [model zoo](https://huggingface.co/lmdeploy) and conduct inference using the following command.

Alternatively, you can quantize 16-bit weights to 4-bit weights following the ["4-bit Weight Quantization"](#4-bit-weight-quantization) section, and then perform inference as per the below instructions.

Take the 4-bit Llama-2-chat-7B model from the model zoo as an example:

```shell
```

```shell
```

As demonstrated in the command below, first convert the model's layout using `turbomind.deploy`, and then you can interact with the AI assistant in the terminal

```shell

## Convert the model's layout and store it in the default path, ./workspace.

## inference
```

## Serve with gradio

If you wish to interact with the model via web ui, please initiate the gradio server as indicated below:

```shell
```

Subsequently, you can open the website `http://{ip_addr}:{port}` in your browser and interact with the model

```

## Inference Performance

We benchmarked the Llama-2-7B-chat and Llama-2-13B-chat models with 4-bit quantization on NVIDIA GeForce RTX 4090 using [profile_generation.py](https://github.com/InternLM/lmdeploy/blob/main/benchmark/profile_generation.py). And we measure the token generation throughput (tokens/s) by setting a single prompt token and generating 512 tokens. All the results are measured for single batch inference.

Memory (GB) comparison results between 4-bit and 16-bit model with context size 2048 and 4096 respectively,

```shell
```

## 4-bit Weight Quantization

It includes two steps:

- generate quantization parameter
- quantize model according to the parameter

### Step 1: Generate Quantization Parameter

```shell
  --calib_dataset 'c4' \             # Calibration dataset, supports c4, ptb, wikitext2, pileval

```shell
  --calib_samples 128 \              # Number of samples in the calibration set, if memory is insufficient, you can appropriately reduce this
  --calib_seqlen 2048 \              # Length of a single piece of text, if memory is insufficient, you can appropriately reduce this
  --work_dir $WORK_DIR \             # Folder storing Pytorch format quantization statistics parameters and post-quantization weight
```

### Step2: Quantize Weights

LMDeploy employs AWQ algorithm for model weight quantization.

```shell
  --w_bits 4 \                       # Bit number for weight quantization
  --w_group_size 128 \               # Group size for weight quantization statistics
  --work_dir $WORK_DIR \             # Directory saving quantization parameters from Step 1

--w_group_size 128 \               # Group size for weight quantization statistics
```

After the quantization is complete, the quantized model is saved to `$WORK_DIR`. Then you can proceed with model inference according to the instructions in the ["4-Bit Weight Model Inference"](#4-bit-llm-model-inference) section.

# NPU (华为昇腾)

## 使用方法

首先，请参考[MMCV](https://mmcv.readthedocs.io/zh_CN/latest/get_started/build.html#npu-mmcv-full) 安装带有 NPU 支持的 MMCV与 [mmengine](https://mmengine.readthedocs.io/en/latest/get_started/installation.html#build-from-source) 。

使用如下命令，可以利用 8 个 NPU 训练模型（以 edsr 为例）：

```shell
bash tools/dist_train.sh configs/edsr/edsr_x2c64b16_1xb16-300k_div2k.py 8
```

或者，使用如下命令，在一个 NPU 上训练模型（以 edsr 为例）：

```shell
python tools/train.py configs/edsr/edsr_x2c64b16_1xb16-300k_div2k.py
```

## 经过验证的模型

|                                           Model                                            | Dataset | PSNR  | SSIM | Download                                                                                       |
| :----------------------------------------------------------------------------------------: | ------- | :---: | :--- | :--------------------------------------------------------------------------------------------- |

| [edsr_x2c64b16_1x16_300k_div2k](https://github.com/open-mmlab/mmagic/blob/main/configs/edsr/edsr_x2c64b16_1xb16-300k_div2k.py) | DIV2K   | 35.83 | 0.94 | [log](https://download.openmmlab.com/mmediting/device/npu/edsr/edsr_x2c64b16_1xb16-300k_div2k.log) |

**注意:**

- 如果没有特别标记，NPU 上的结果与使用 FP32 的 GPU 上的结果相同。

**以上所有模型权重及训练日志均由华为昇腾团队提供**

## 0.6.0 (05/05/2022)

1. A new recognition algorithm [MASTER](https://arxiv.org/abs/1910.02562) has been added into MMOCR, which was the championship solution for the "ICDAR 2021 Competition on Scientific Table Image Recognition to Latex"! The model pre-trained on SynthText and MJSynth is available for testing! Credit to @JiaquanYe
2. [DBNet++](https://arxiv.org/abs/2202.10304) has been released now! A new Adaptive Scale Fusion module has been equipped for feature enhancement. Benefiting from this, the new model achieved 2% better h-mean score than its predecessor on the ICDAR2015 dataset.
3. Three more dataset converters are added: LSVT, RCTW and HierText. Check the dataset zoo ([Det](https://mmocr.readthedocs.io/en/latest/datasets/det.html#) & [Recog](https://mmocr.readthedocs.io/en/latest/datasets/recog.html) ) to explore further information.

4. To enhance the data storage efficiency, MMOCR now supports loading both images and labels from .lmdb format annotations for the text recognition task. To enable such a feature, the new lmdb_converter.py is ready for use to pack your cropped images and labels into an lmdb file. For a detailed tutorial, please refer to the following sections and the [doc](https://mmocr.readthedocs.io/en/latest/tools.html#convert-text-recognition-dataset-to-lmdb-format).
5. Testing models on multiple datasets is a widely used evaluation strategy. MMOCR now supports automatically reporting mean scores when there is more than one dataset to evaluate, which enables a more convenient comparison between checkpoints. [Doc](https://mmocr.readthedocs.io/en/latest/tutorials/dataset_types.html#getting-mean-evaluation-scores)

6. Evaluation is more flexible and customizable now. For text detection tasks, you can set the score threshold range where the best results might come out. ([Doc](https://mmocr.readthedocs.io/en/latest/tutorials/dataset_types.html#evaluation)) If too many results are flooding your text recognition train log, you can trim it by specifying a subset of metrics in evaluation config. Check out the [Evaluation](https://mmocr.readthedocs.io/en/latest/tutorials/dataset_types.html#ocrdataset) section for details.
7. MMOCR provides a script to convert the .json labels obtained by the popular annotation toolkit **Labelme** to MMOCR-supported data format. @Y-M-Y contributed a log analysis tool that helps users gain a better understanding of the entire training process. Read [tutorial docs](https://mmocr.readthedocs.io/en/latest/tools.html) to get started.

### Lmdb Dataset

Reading images or labels from files can be slow when data are excessive, e.g. on a scale of millions. Besides, in academia, most of the scene text recognition datasets are stored in lmdb format, including images and labels. To get closer to the mainstream practice and enhance the data storage efficiency, MMOCR now officially supports loading images and labels from lmdb datasets via a new pipeline [LoadImageFromLMDB](https://github.com/open-mmlab/mmocr/blob/878383b9de8d0e598f31fbb844ffcb0c305deb8b/mmocr/datasets/pipelines/loading.py#L140).
This section is intended to serve as a quick walkthrough for you to master this update and apply it to facilitate your research.

#### Specifications

To better align with the academic community, MMOCR now requires the following specifications for lmdb datasets:

- The parameter describing the data volume of the dataset is `num-samples` instead of `total_number` (deprecated).

- Images and labels are stored with keys in the form of `image-000000001` and `label-000000001`, respectively.

#### Usage

1. Use existing academic lmdb datasets if they meet the specifications; or the tool provided by MMOCR to pack images & annotations into a lmdb dataset.

- Previously, MMOCR had a function `txt2lmdb` (deprecated) that only supported converting labels to lmdb format. However, it is quite different from academic lmdb datasets, which usually contain both images and labels. Now MMOCR provides a new utility [lmdb_converter](https://github.com/open-mmlab/mmocr/blob/main/tools/data/utils/lmdb_converter.py) to convert recognition datasets with both images and labels to lmdb format.

- Say that your recognition data in MMOCR's format are organized as follows. (See an example in [ocr_toy_dataset](https://github.com/open-mmlab/mmocr/tree/main/tests/data/ocr_toy_dataset)).

  ```text
  # Directory structure

  ├──img_path
  |      |—— img1.jpg
  |      |—— img2.jpg
  |      |—— ...
  |——label.txt (or label.jsonl)

  # Annotation format

  label.txt:  img1.jpg HELLO
              img2.jpg WORLD
              ...

  label.jsonl:    {'filename':'img1.jpg', 'text':'HELLO'}
                  {'filename':'img2.jpg', 'text':'WORLD'}
                  ...
  ```

- Then pack these files up:

  ```bash
  python tools/data/utils/lmdb_converter.py  {PATH_TO_LABEL} {OUTPUT_PATH} --i {PATH_TO_IMAGES}
  ```

- Check out [tools.md](https://github.com/open-mmlab/mmocr/blob/main/docs/en/tools.md) for more details.

2. The second step is to modify the configuration files. For example, to train CRNN on MJ and ST datasets:

- Set parser as `LineJsonParser` and `file_format` as 'lmdb' in [dataset config](https://github.com/open-mmlab/mmocr/blob/main/configs/_base_/recog_datasets/ST_MJ_train.py#L9)

  ```python
  # configs/_base_/recog_datasets/ST_MJ_train.py
  train1 = dict(
      type='OCRDataset',
      img_prefix=train_img_prefix1,
      ann_file=train_ann_file1,
      loader=dict(
          type='AnnFileLoader',
          repeat=1,
          file_format='lmdb',
          parser=dict(
              type='LineJsonParser',
              keys=['filename', 'text'],
          )),
      pipeline=None,
      test_mode=False)
  ```

- Use `LoadImageFromLMDB` in [pipeline](https://github.com/open-mmlab/mmocr/blob/main/configs/_base_/recog_pipelines/crnn_pipeline.py#L4):

  ```python
  # configs/_base_/recog_pipelines/crnn_pipeline.py

```python
      dict(type='LoadImageFromLMDB', color_type='grayscale'),
      ...
  ```

3. You are good to go! Start training and MMOCR will load data from your lmdb dataset.

### New Features & Enhancements

- Add analyze_logs in tools and its description in docs by @Y-M-Y in https://github.com/open-mmlab/mmocr/pull/899
- Add LSVT Data Converter by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/896
- Add RCTW dataset converter by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/914
- Support computing mean scores in UniformConcatDataset by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/981
- Support loading images and labels from lmdb file by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/982
- Add recog2lmdb and new toy dataset files by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/979

- Add labelme converter for textdet and textrecog by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/972
- Update CircleCI configs by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/918
- Update Git Action by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/930
- More customizable fields in dataloaders by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/933
- Skip CIs when docs are modified by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/941
- Rename Github tests, fix ignored paths by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/946
- Support latest MMCV by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/959
- Support dynamic threshold range in eval_hmean by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/962

- Update the version requirement of mmdet in docker by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/966
- Replace `opencv-python-headless` with `open-python` by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/970
- Update Dataset Configs by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/980
- Add SynthText dataset config by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/983
- Automatically report mean scores when applicable by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/995
- Add DBNet++ by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/973
- Add MASTER by @JiaquanYe in https://github.com/open-mmlab/mmocr/pull/807

- Allow choosing metrics to report in text recognition tasks by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/989
- Add HierText converter by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/948
- Fix lint_only in CircleCI by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/998

- Fix CircleCi Main Branch Accidentally Run PR Stage Test by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/927
- Fix a deprecate warning about mmdet.datasets.pipelines.formating by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/944
- Fix a Bug in ResNet plugin by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/967
- revert a wrong setting in db_r18 cfg by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/978

- Fix TotalText Anno version issue by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/945
- Update installation step of `albumentations` by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/984
- Fix ImgAug transform by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/949
- Fix GPG key error in CI and docker by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/988
- update label.lmdb by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/991
- correct meta key by @garvan2021 in https://github.com/open-mmlab/mmocr/pull/926
- Use new image by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/976
- Fix Data Converter Issues by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/955

### Docs

### Docs

- Update CONTRIBUTING.md by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/905
- Fix the misleading description in test.py by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/908
- Update recog.md for lmdb Generation by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/934
- Add MMCV by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/954
- Add wechat QR code to CN readme by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/960
- Update CONTRIBUTING.md by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/947
- Use QR codes from MMCV by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/971
- Renew dataset_types.md by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/997

- @Y-M-Y made their first contribution in https://github.com/open-mmlab/mmocr/pull/899

**Full Changelog**: https://github.com/open-mmlab/mmocr/compare/v0.5.0...v0.6.0

## 0.5.0 (31/03/2022)

1. MMOCR now supports SPACE recognition! (What a prominent feature!) Users only need to convert the recognition annotations that contain spaces from a plain `.txt` file to JSON line format `.jsonl`, and then revise a few configurations to enable the `LineJsonParser`. For more information, please read our step-by-step [tutorial](https://mmocr.readthedocs.io/en/latest/tutorials/blank_recog.html).

2. [Tesseract](https://github.com/tesseract-ocr/tesseract) is now available in MMOCR! While MMOCR is more flexible to support various downstream tasks, users might sometimes not be satisfied with DL models and would like to turn to effective legacy solutions. Therefore, we offer this option in `mmocr.utils.ocr` by wrapping Tesseract as a detector and/or recognizer. Users can easily create an MMOCR object by `MMOCR(det=’Tesseract’, recog=’Tesseract’)`. Credit to @garvan2021
3. We release data converters for **16** widely used OCR datasets, including multiple scenarios such as document, handwritten, and scene text. Now it is more convenient to generate annotation files for these datasets. Check the dataset zoo ( [Det](https://mmocr.readthedocs.io/en/latest/datasets/det.html#) & [Recog](https://mmocr.readthedocs.io/en/latest/datasets/recog.html) ) to explore further information.

4. Special thanks to @EighteenSprings @BeyondYourself @yangrisheng, who had actively participated in documentation translation!

### Migration Guide - ResNet

Some refactoring processes are still going on. For text recognition models, we unified the [`ResNet-like` architectures](https://github.com/open-mmlab/mmocr/blob/72f945457324e700f0d14796dd10a51535c01a57/mmocr/models/textrecog/backbones/resnet.py) which are used as backbones. By introducing stage-wise and block-wise plugins, the refactored ResNet is highly flexible to support existing models, like ResNet31 and ResNet45, and other future designs of ResNet variants.

#### Plugin

- `Plugin` is a module category inherited from MMCV's implementation of `PLUGIN_LAYERS`, which can be inserted between each stage of ResNet or into a basicblock. You can find a simple implementation of plugin at [mmocr/models/textrecog/plugins/common.py](https://github.com/open-mmlab/mmocr/blob/72f945457324e700f0d14796dd10a51535c01a57/mmocr/models/textrecog/plugins/common.py), or click the button below.

  <details close>
    <summary>Plugin Example</summary>

  ```python
  @PLUGIN_LAYERS.register_module()
  class Maxpool2d(nn.Module):
      """A wrapper around nn.Maxpool2d().

      Args:
          kernel_size (int or tuple(int)): Kernel size for max pooling layer
          stride (int or tuple(int)): Stride for max pooling layer
          padding (int or tuple(int)): Padding for pooling layer
      """

      def __init__(self, kernel_size, stride, padding=0, **kwargs):
          super(Maxpool2d, self).__init__()

padding (int or tuple(int)): Padding for pooling layer
      """

          self.model = nn.MaxPool2d(kernel_size, stride, padding)

      def forward(self, x):
          """
          Args:
              x (Tensor): Input feature map

          Returns:
              Tensor: The tensor after Maxpooling layer.
          """
          return self.model(x)
  ```

#### Stage-wise Plugins

- ResNet is composed of stages, and each stage is composed of blocks. E.g., ResNet18 is composed of 4 stages, and each stage is composed of basicblocks. For each stage, we provide two ports to insert stage-wise plugins by giving `plugins` parameters in ResNet.

  ```text
  [port1: before stage] ---> [stage] ---> [port2: after stage]
  ```

```text
  ```

- E.g. Using a ResNet with four stages as example. Suppose we want to insert an additional convolution layer before each stage, and an additional convolution layer at stage 1, 2, 4. Then you can define the special ResNet18 like this

  ```python
  resnet18_speical = ResNet(
          # for simplicity, some required
          # parameters are omitted
          plugins=[
              dict(
                  cfg=dict(
                  type='ConvModule',
                  kernel_size=3,
                  stride=1,
                  padding=1,
                  norm_cfg=dict(type='BN'),
                  act_cfg=dict(type='ReLU')),
                  stages=(True, True, True, True),
                  position='before_stage')
              dict(
                  cfg=dict(
                  stride=1,

dict(
                  cfg=dict(
                  stride=1,
                  stages=(True, True, False, True),
                  position='after_stage')
          ])
  ```

- You can also insert more than one plugin in each port and those plugins will be executed in order. Let's take ResNet in [MASTER](https://arxiv.org/abs/1910.02562) as an example:

    <summary>Multiple Plugins Example</summary>

  - ResNet in Master is based on ResNet31. And after each stage, a module named `GCAModule` will be used. The `GCAModule` is inserted before the stage-wise convolution layer in ResNet31. In conlusion, there will be two plugins at `after_stage` port in the same time.

    ```python
    resnet_master = ResNet(
                    # for simplicity, some required

```python
                    # parameters are omitted
                    plugins=[
                        dict(
                            cfg=dict(type='Maxpool2d', kernel_size=2, stride=(2, 2)),
                            stages=(True, True, False, False),
                            position='before_stage'),
                        dict(
                            cfg=dict(type='Maxpool2d', kernel_size=(2, 1), stride=(2, 1)),
                            stages=(False, False, True, False),
                        dict(
                            cfg=dict(type='GCAModule', kernel_size=3, stride=1, padding=1),
                            stages=[True, True, True, True],
                            position='after_stage'),
                        dict(
                            cfg=dict(
                                type='ConvModule',

position='after_stage'),
                        dict(
                            cfg=dict(
                                kernel_size=3,
                                stride=1,
                                padding=1,
                                norm_cfg=dict(type='BN'),
                                act_cfg=dict(type='ReLU')),
                            stages=(True, True, True, True),
                            position='after_stage')
                    ])

    ```

    </details>

  - In each plugin, we will pass two parameters (`in_channels`, `out_channels`) to support operations that need the information of current channels.

#### Block-wise Plugin (Experimental)

- In each plugin, we will pass two parameters (`in_channels`, `out_channels`) to support operations that need the information of current channels.

- We also refactored the `BasicBlock` used in ResNet. Now it can be customized with block-wise plugins. Check [here](https://github.com/open-mmlab/mmocr/blob/72f945457324e700f0d14796dd10a51535c01a57/mmocr/models/textrecog/layers/conv_layer.py) for more details.

- BasicBlock is composed of two convolution layer in the main branch and a shortcut branch. We provide four ports to insert plugins.

  ```text
      [port1: before_conv1] ---> [conv1] --->
      [port2: after_conv1] ---> [conv2] --->
      [port3: after_conv2] ---> +(shortcut) ---> [port4: after_shortcut]
  ```

- In each plugin, we will pass a parameter `in_channels` to support operations that need the information of current channels.

- E.g. Build a ResNet with customized BasicBlock with an additional convolution layer before conv1:

    <summary>Block-wise Plugin Example</summary>

  ```python
  resnet_31 = ResNet(
          in_channels=3,
          stem_channels=[64, 128],
          block_cfgs=dict(type='BasicBlock'),
          arch_layers=[1, 2, 5, 3],
          arch_channels=[256, 256, 512, 512],
          strides=[1, 1, 1, 1],
          plugins=[
              dict(
                  cfg=dict(type='Maxpool2d',
                  kernel_size=2,
                  stride=(2, 2)),
                  stages=(True, True, False, False),
                  position='before_stage'),
              dict(
                  kernel_size=(2, 1),
                  stride=(2, 1)),
                  stages=(False, False, True, False),
              dict(
                  cfg=dict(

stages=(False, False, True, False),
              dict(
                  cfg=dict(
                  stride=1,
          ])
  ```

#### Full Examples

<summary>ResNet without plugins</summary>

- ResNet45 is used in ASTER and ABINet without any plugins.

  ```python
  resnet45_aster = ResNet(
      in_channels=3,
      stem_channels=[64, 128],
      block_cfgs=dict(type='BasicBlock', use_conv1x1='True'),
      arch_layers=[3, 4, 6, 6, 3],
      arch_channels=[32, 64, 128, 256, 512],
      strides=[(2, 2), (2, 2), (2, 1), (2, 1), (2, 1)])

  resnet45_abi = ResNet(
      stem_channels=32,

arch_channels=[32, 64, 128, 256, 512],

      strides=[2, 1, 2, 1, 1])
  ```

<summary>ResNet with plugins</summary>

- ResNet31 is a typical architecture to use stage-wise plugins. Before the first three stages, Maxpooling layer is used. After each stage, a convolution layer with BN and ReLU is used.

  ```python
      block_cfgs=dict(type='BasicBlock'),
      arch_layers=[1, 2, 5, 3],
      arch_channels=[256, 256, 512, 512],
      strides=[1, 1, 1, 1],
      plugins=[
          dict(
              cfg=dict(type='Maxpool2d',
              kernel_size=2,
              stride=(2, 2)),

strides=[1, 1, 1, 1],
      plugins=[
          dict(
              stages=(True, True, False, False),
              position='before_stage'),
          dict(
              kernel_size=(2, 1),
              stride=(2, 1)),
              stages=(False, False, True, False),
          dict(
              cfg=dict(
              type='ConvModule',
              kernel_size=3,
              stride=1,
              padding=1,
              norm_cfg=dict(type='BN'),
              act_cfg=dict(type='ReLU')),
              stages=(True, True, True, True),
              position='after_stage')
      ])
  ```

### Migration Guide - Dataset Annotation Loader

stages=(True, True, True, True),
      ])
  ```

The annotation loaders, `LmdbLoader` and `HardDiskLoader`, are unified into `AnnFileLoader` for a more consistent design and wider support on different file formats and storage backends. `AnnFileLoader` can load the annotations from `disk`(default), `http` and `petrel` backend, and parse the annotation in `txt` or `lmdb` format. `LmdbLoader` and `HardDiskLoader` are deprecated, and users are recommended to modify their configs to use the new `AnnFileLoader`. Users can migrate their legacy loader `HardDiskLoader` referring to the following example:

```python
# Legacy config
train = dict(
    ...
    loader=dict(
        type='HardDiskLoader',
        ...))

# Suggested config
    ...
        type='AnnFileLoader',
        file_storage_backend='disk',

type='HardDiskLoader',
        ...))

    ...
        file_format='txt',
        ...))
```

Similarly, using `AnnFileLoader` with `file_format='lmdb'` instead of `LmdbLoader` is strongly recommended.

- Update mmcv install by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/775
- Upgrade isort by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/771
- Automatically infer device for inference if not speicifed by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/781
- Add open-mmlab precommit hooks by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/787
- Add windows CI by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/790
- Add CurvedSyntext150k Converter by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/719

- Add FUNSD Converter by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/808
- Support loading annotation file with petrel/http backend by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/793
- Support different seeds on different ranks by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/820
- Support json in recognition converter by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/844
- Add args and docs for multi-machine training/testing by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/849
- Add warning info for LineStrParser by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/850
- Deploy openmmlab-bot by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/876
- Add Tesserocr Inference  by @garvan2021 in https://github.com/open-mmlab/mmocr/pull/814

- Add LV Dataset Converter by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/871
- Add SROIE Converter by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/810
- Add NAF Converter by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/815
- Add DeText Converter by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/818
- Add IMGUR Converter by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/825
- Add ILST Converter by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/833
- Add KAIST Converter by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/835
- Add IC11 (Born-digital Images) Data Converter by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/857
- Add IC13 (Focused Scene Text) Data Converter by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/861

- Add BID Converter by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/862
- Add Vintext Converter by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/864
- Add MTWI Data Converter by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/867
- Add COCO Text v2 Data Converter by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/872
- Add ReCTS Data Converter by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/892
- Refactor ResNets by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/809

- Bump mmdet version to 2.20.0 in Dockerfile by @GPhilo in https://github.com/open-mmlab/mmocr/pull/763
- Update mmdet version limit by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/773
- Minimum version requirement of albumentations by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/769

- Disable worker in the dataloader of gpu unit test by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/780
- Standardize the type of torch.device in ocr.py by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/800
- Use RECOGNIZER instead of DETECTORS by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/685
- Add num_classes to configs of ABINet by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/805
- Support loading space character from dict file by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/854
- Description in tools/data/utils/txt2lmdb.py by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/870
- ignore_index in SARLoss by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/869
- Fix a bug that may cause inplace operation error by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/884

- Use hyphen instead of underscores in script args by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/890

### Docs

- Add deprecation message for deploy tools by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/801
- Reorganizing OpenMMLab projects in readme by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/806
- Add demo/README_zh.md by @EighteenSprings in https://github.com/open-mmlab/mmocr/pull/802
- Add detailed version requirement table by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/778
- Correct misleading section title in training.md by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/819
- Update README_zh-CN  document URL by @BeyondYourself in https://github.com/open-mmlab/mmocr/pull/823
- translate testing.md. by @yangrisheng in https://github.com/open-mmlab/mmocr/pull/822

- Fix confused description for load-from and resume-from by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/842
- Add documents getting_started in docs/zh by @BeyondYourself in https://github.com/open-mmlab/mmocr/pull/841
- Add the model serving translation document by @BeyondYourself in https://github.com/open-mmlab/mmocr/pull/845
- Update docs about installation on Windows by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/852
- Update tutorial notebook by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/853
- Update Instructions for New Data Converters by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/900
- Brief installation instruction in README by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/897

- update doc for ILST, VinText, BID by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/902
- Fix typos in readme by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/903
- Recog dataset doc by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/893
- Reorganize the directory structure section in det.md by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/894

- @GPhilo made their first contribution in https://github.com/open-mmlab/mmocr/pull/763
- @xinke-wang made their first contribution in https://github.com/open-mmlab/mmocr/pull/801
- @EighteenSprings made their first contribution in https://github.com/open-mmlab/mmocr/pull/802
- @BeyondYourself made their first contribution in https://github.com/open-mmlab/mmocr/pull/823
- @yangrisheng made their first contribution in https://github.com/open-mmlab/mmocr/pull/822

- @Mountchicken made their first contribution in https://github.com/open-mmlab/mmocr/pull/844
- @garvan2021 made their first contribution in https://github.com/open-mmlab/mmocr/pull/814

**Full Changelog**: https://github.com/open-mmlab/mmocr/compare/v0.4.1...v0.5.0

## v0.4.1 (27/01/2022)

1. Visualizing edge weights in OpenSet KIE is now supported! https://github.com/open-mmlab/mmocr/pull/677
2. Some configurations have been optimized to significantly speed up the training and testing processes! Don't worry - you can still tune these parameters in case these modifications do not work. https://github.com/open-mmlab/mmocr/pull/757
3. Now you can use CPU to train/debug your model! https://github.com/open-mmlab/mmocr/pull/752

4. We have fixed a severe bug that causes users unable to call `mmocr.apis.test` with our pre-built wheels. https://github.com/open-mmlab/mmocr/pull/667

- Show edge score for openset kie by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/677
- Download flake8 from github as pre-commit hooks by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/695
- Deprecate the support for 'python setup.py test' by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/722
- Disable multi-processing feature of cv2 to speed up data loading by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/721
- Extend ctw1500 converter to support text fields by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/729
- Extend totaltext converter to support text fields by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/728

- Speed up training by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/739
- Add setup multi-processing both in train and test.py by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/757
- Support CPU training/testing by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/752
- Support specify gpu for testing and training with gpu-id instead of gpu-ids and gpus  by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/756
- Remove unnecessary custom_import from test.py  by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/758

- Fix satrn onnxruntime test by @AllentDan in https://github.com/open-mmlab/mmocr/pull/679
- Support both ConcatDataset and UniformConcatDataset by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/675

- Fix bugs of show_results in single_gpu_test by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/667
- Fix a bug for sar decoder when bi-rnn is used by @MhLiao in https://github.com/open-mmlab/mmocr/pull/690
- Fix opencv version to avoid some bugs by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/694
- Fix py39 ci error by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/707
- Update visualize.py by @TommyZihao in https://github.com/open-mmlab/mmocr/pull/715
- Fix link of config by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/726
- Use yaml.safe_load instead of load by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/753
- Add necessary keys to test_pipelines to enable test-time visualization by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/754

### Docs

### Docs

- Fix recog.md by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/674
- Add config tutorial by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/683
- Add MMSelfSup/MMRazor/MMDeploy in readme by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/692
- Add recog & det model summary by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/693
- Update docs link by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/710
- add pull request template.md by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/711
- Add website links to readme by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/731
- update readme according to standard by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/742

- @MhLiao made their first contribution in https://github.com/open-mmlab/mmocr/pull/690

- @TommyZihao made their first contribution in https://github.com/open-mmlab/mmocr/pull/715

**Full Changelog**: https://github.com/open-mmlab/mmocr/compare/v0.4.0...v0.4.1

## v0.4.0 (15/12/2021)

1. We release a new text recognition model - [ABINet](https://arxiv.org/pdf/2103.06495.pdf) (CVPR 2021, Oral). With it dedicated model design and useful data augmentation transforms, ABINet can achieve the best performance on irregular text recognition tasks. [Check it out!](https://mmocr.readthedocs.io/en/latest/textrecog_models.html#read-like-humans-autonomous-bidirectional-and-iterative-language-modeling-for-scene-text-recognition)
2. We are also working hard to fulfill the requests from our community.

   [OpenSet KIE](https://mmocr.readthedocs.io/en/latest/kie_models.html#wildreceiptopenset) is one of the achievement, which extends the application of SDMGR from text node classification to node-pair relation extraction. We also provide
   a demo script to convert WildReceipt to open set domain, though it cannot
   take the full advantage of OpenSet format. For more information, please read our
   [tutorial](https://mmocr.readthedocs.io/en/latest/tutorials/kie_closeset_openset.html).
3. APIs of models can be exposed through TorchServe. [Docs](https://mmocr.readthedocs.io/en/latest/model_serving.html)

### Breaking Changes & Migration Guide

#### Postprocessor

Some refactoring processes are still going on. For all text detection models, we unified their `decode` implementations into a new module category, `POSTPROCESSOR`, which is responsible for decoding different raw outputs into boundary instances. In all text detection configs, the `text_repr_type` argument in `bbox_head` is deprecated and will be removed in the future release.

**Migration Guide**: Find a similar line from detection model's config:

```
text_repr_type=xxx,
```

And replace it with

```
postprocessor=dict(type='{MODEL_NAME}Postprocessor', text_repr_type=xxx)),
```

Take a snippet of PANet's config as an example. Before the change, its config for `bbox_head` looks like:

```
        type='PANHead',
        text_repr_type='poly',
        in_channels=[128, 128, 128, 128],

```
        out_channels=6,
        module_loss=dict(type='PANModuleLoss')),
```

Afterwards:

```
    type='PANHead',
    in_channels=[128, 128, 128, 128],
    out_channels=6,
    module_loss=dict(type='PANModuleLoss'),
    postprocessor=dict(type='PANPostprocessor', text_repr_type='poly')),
```

There are other postprocessors and each takes different arguments. Interested users can find their interfaces or implementations in `mmocr/models/textdet/postprocess` or through our [api docs](https://mmocr.readthedocs.io/en/latest/api.html#textdet-postprocess).

#### New Config Structure

We reorganized the `configs/` directory by extracting reusable sections into `configs/_base_`. Now the directory tree of `configs/_base_` is organized as follows:

```
_base_
├── det_datasets
├── det_models
├── det_pipelines
├── recog_datasets
├── recog_models

```
_base_
├── recog_pipelines
└── schedules
```

Most of model configs are making full use of base configs now, which makes the overall structural clearer and facilitates fair
comparison across models. Despite the seemingly significant hierarchical difference, **these changes would not break the backward compatibility** as the names of model configs remain the same.

- Support openset kie by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/498
- Add converter for the Open Images v5 text annotations by Krylov et al. by @baudm in https://github.com/open-mmlab/mmocr/pull/497
- Support Chinese for kie show result by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/464
- Add TorchServe support for text detection and recognition by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/522

- Save filename in text detection test results by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/570
- Add codespell pre-commit hook and fix typos by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/520
- Avoid duplicate placeholder docs in CN by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/582
- Save results to json file for kie. by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/589
- Add SAR_CN to ocr.py by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/579
- mim extension for windows by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/641
- Support muitiple pipelines for different datasets by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/657
- ABINet Framework by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/651

### Refactoring

- Refactor textrecog config structure by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/617
- Refactor text detection config by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/626
- refactor transformer modules by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/618
- refactor textdet postprocess by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/640

### Docs

- C++ example section by @apiaccess21 in https://github.com/open-mmlab/mmocr/pull/593
- install.md Chinese section by @A465539338 in https://github.com/open-mmlab/mmocr/pull/364
- Add Chinese Translation of deployment.md. by @fatfishZhao in https://github.com/open-mmlab/mmocr/pull/506
- Fix a model link and add the metafile for SATRN by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/473
- Improve docs style by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/474

- Enhancement & sync Chinese docs by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/492
- TorchServe docs by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/539
- Update docs menu by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/564
- Docs for KIE CloseSet & OpenSet by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/573
- Fix broken links by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/576
- Docstring for text recognition models by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/562
- Add MMFlow & MIM by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/597
- Add MMFewShot by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/621
- Update model readme by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/604

- Add input size check to model_inference by @mpena-vina in https://github.com/open-mmlab/mmocr/pull/633
- Docstring for textdet models by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/561
- Add MMHuman3D in readme by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/644
- Use shared menu from theme instead by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/655
- Refactor docs structure by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/662
- Docs fix by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/664

### Enhancements

- Use bounding box around polygon instead of within polygon by @alexander-soare in https://github.com/open-mmlab/mmocr/pull/469
- Add CITATION.cff by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/476

- Add py3.9 CI by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/475
- update model-index.yml by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/484
- Use container in CI by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/502
- CircleCI Setup by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/611
- Remove unnecessary custom_import from train.py by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/603
- Change the upper version of mmcv to 1.5.0 by @zhouzaida in https://github.com/open-mmlab/mmocr/pull/628
- Update CircleCI by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/631
- Pass custom_hooks to MMCV by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/609
- Skip CI when some specific files were changed by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/642

- Add markdown linter in pre-commit hook by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/643
- Use shape from loaded image by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/652
- Cancel previous runs that are not completed by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/666

- Modify algorithm "sar" weights path in metafile by @ShoupingShan in https://github.com/open-mmlab/mmocr/pull/581
- Fix Cuda CI by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/472
- Fix image export in test.py for KIE models by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/486
- Allow invalid polygons in intersection and union by default  by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/471
- Update checkpoints' links for SATRN by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/518

- Fix converting to onnx bug because of changing key from img_shape to resize_shape by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/523
- Fix PyTorch 1.6 incompatible checkpoints by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/540
- Fix paper field in metafiles by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/550
- Unify recognition task names in metafiles by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/548
- Fix py3.9 CI by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/563
- Always map location to cpu when loading checkpoint by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/567
- Fix wrong model builder in recog_test_imgs by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/574
- Improve dbnet r50 by fixing img std by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/578

- Fix resource warning: unclosed file by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/577
- Fix bug that same start_point for different texts in draw_texts_by_pil by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/587
- Keep original texts for kie by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/588
- Fix random seed by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/600
- Fix DBNet_r50 config by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/625
- Change SBC case to DBC case by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/632
- Fix kie demo by @innerlee in https://github.com/open-mmlab/mmocr/pull/610
- fix type check by @cuhk-hbsun in https://github.com/open-mmlab/mmocr/pull/650
- Remove depreciated image validator in totaltext converter by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/661

- Fix change locals() dict by @Fei-Wang in https://github.com/open-mmlab/mmocr/pull/663
- fix #614: textsnake targets by @HolyCrap96 in https://github.com/open-mmlab/mmocr/pull/660

- @alexander-soare made their first contribution in https://github.com/open-mmlab/mmocr/pull/469
- @A465539338 made their first contribution in https://github.com/open-mmlab/mmocr/pull/364
- @fatfishZhao made their first contribution in https://github.com/open-mmlab/mmocr/pull/506
- @baudm made their first contribution in https://github.com/open-mmlab/mmocr/pull/497
- @ShoupingShan made their first contribution in https://github.com/open-mmlab/mmocr/pull/581
- @apiaccess21 made their first contribution in https://github.com/open-mmlab/mmocr/pull/593
- @zhouzaida made their first contribution in https://github.com/open-mmlab/mmocr/pull/628

- @mpena-vina made their first contribution in https://github.com/open-mmlab/mmocr/pull/633
- @Fei-Wang made their first contribution in https://github.com/open-mmlab/mmocr/pull/663

**Full Changelog**: https://github.com/open-mmlab/mmocr/compare/v0.3.0...0.4.0

## v0.3.0 (25/8/2021)

1. We add a new text recognition model -- SATRN! Its pretrained checkpoint achieves the best performance over other provided text recognition models. A lighter version of SATRN is also released which can obtain ~98% of the performance of the original model with only 45 MB in size. ([@2793145003](https://github.com/2793145003)) [#405](https://github.com/open-mmlab/mmocr/pull/405)

2. Improve the demo script, `ocr.py`, which supports applying end-to-end text detection, text recognition and key information extraction models on images with easy-to-use commands. Users can find its full documentation in the demo section. ([@samayala22](https://github.com/samayala22), [@manjrekarom](https://github.com/manjrekarom)) [#371](https://github.com/open-mmlab/mmocr/pull/371), [#386](https://github.com/open-mmlab/mmocr/pull/386), [#400](https://github.com/open-mmlab/mmocr/pull/400), [#374](https://github.com/open-mmlab/mmocr/pull/374), [#428](https://github.com/open-mmlab/mmocr/pull/428)
3. Our documentation is reorganized into a clearer structure. More useful contents are on the way! [#409](https://github.com/open-mmlab/mmocr/pull/409), [#454](https://github.com/open-mmlab/mmocr/pull/454)

4. The requirement of `Polygon3` is removed since this project is no longer maintained or distributed. We unified all its references to equivalent substitutions in `shapely` instead. [#448](https://github.com/open-mmlab/mmocr/pull/448)

1. Upgrade version requirement of MMDetection to 2.14.0 to avoid bugs [#382](https://github.com/open-mmlab/mmocr/pull/382)
2. MMOCR now has its own model and layer registries inherited from MMDetection's or MMCV's counterparts. ([#436](https://github.com/open-mmlab/mmocr/pull/436)) The modified hierarchical structure of the model registries are now organized as follows.

```text
mmcv.MODELS -> mmdet.BACKBONES -> BACKBONES
mmcv.MODELS -> mmdet.NECKS -> NECKS
mmcv.MODELS -> mmdet.ROI_EXTRACTORS -> ROI_EXTRACTORS
mmcv.MODELS -> mmdet.HEADS -> HEADS
mmcv.MODELS -> mmdet.LOSSES -> LOSSES
mmcv.MODELS -> mmdet.DETECTORS -> DETECTORS
mmcv.ACTIVATION_LAYERS -> ACTIVATION_LAYERS
mmcv.UPSAMPLE_LAYERS -> UPSAMPLE_LAYERS
```

```

To migrate your old implementation to our new backend, you need to change the import path of any registries and their corresponding builder functions (including `build_detectors`) from `mmdet.models.builder` to `mmocr.models.builder`. If you have referred to any model or layer of MMDetection or MMCV in your model config, you need to add `mmdet.` or `mmcv.` prefix to its name to inform the model builder of the right namespace to work on.

Interested users may check out [MMCV's tutorial on Registry](https://mmcv.readthedocs.io/en/latest/understand_mmcv/registry.html) for in-depth explanations on its mechanism.

- Automatically replace SyncBN with BN for inference [#420](https://github.com/open-mmlab/mmocr/pull/420), [#453](https://github.com/open-mmlab/mmocr/pull/453)

- Support batch inference for CRNN and SegOCR [#407](https://github.com/open-mmlab/mmocr/pull/407)
- Support exporting documentation in pdf or epub format [#406](https://github.com/open-mmlab/mmocr/pull/406)
- Support `persistent_workers` option in data loader [#459](https://github.com/open-mmlab/mmocr/pull/459)

- Remove depreciated key in kie_test_imgs.py [#381](https://github.com/open-mmlab/mmocr/pull/381)
- Fix dimension mismatch in batch testing/inference of DBNet [#383](https://github.com/open-mmlab/mmocr/pull/383)
- Fix the problem of dice loss which stays at 1 with an empty target given [#408](https://github.com/open-mmlab/mmocr/pull/408)
- Fix a wrong link in ocr.py ([@naarkhoo](https://github.com/naarkhoo)) [#417](https://github.com/open-mmlab/mmocr/pull/417)

- Fix undesired assignment to "pretrained" in test.py [#418](https://github.com/open-mmlab/mmocr/pull/418)
- Fix a problem in polygon generation of DBNet [#421](https://github.com/open-mmlab/mmocr/pull/421), [#443](https://github.com/open-mmlab/mmocr/pull/443)
- Skip invalid annotations in totaltext_converter [#438](https://github.com/open-mmlab/mmocr/pull/438)
- Add zero division handler in poly utils, remove Polygon3 [#448](https://github.com/open-mmlab/mmocr/pull/448)

- Replace lanms-proper with lanms-neo to support installation on Windows (with special thanks to [@gen-ko](https://github.com/gen-ko) who has re-distributed this package!)
- Support MIM [#394](https://github.com/open-mmlab/mmocr/pull/394)
- Add tests for PyTorch 1.9 in CI [#401](https://github.com/open-mmlab/mmocr/pull/401)

- Enables fullscreen layout in readthedocs [#413](https://github.com/open-mmlab/mmocr/pull/413)
- General documentation enhancement [#395](https://github.com/open-mmlab/mmocr/pull/395)
- Update version checker [#427](https://github.com/open-mmlab/mmocr/pull/427)
- Add copyright info [#439](https://github.com/open-mmlab/mmocr/pull/439)
- Update citation information [#440](https://github.com/open-mmlab/mmocr/pull/440)

We thank [@2793145003](https://github.com/2793145003), [@samayala22](https://github.com/samayala22), [@manjrekarom](https://github.com/manjrekarom), [@naarkhoo](https://github.com/naarkhoo), [@gen-ko](https://github.com/gen-ko), [@duanjiaqi](https://github.com/duanjiaqi), [@gaotongxiao](https://github.com/gaotongxiao), [@cuhk-hbsun](https://github.com/cuhk-hbsun), [@innerlee](https://github.com/innerlee), [@wdsd641417025](https://github.com/wdsd641417025) for their contribution to this release!

## v0.2.1 (20/7/2021)

1. Upgrade to use MMCV-full **>= 1.3.8** and MMDetection **>= 2.13.0** for latest features

2. Add ONNX and TensorRT export tool, supporting the deployment of DBNet, PSENet, PANet and CRNN (experimental) [#278](https://github.com/open-mmlab/mmocr/pull/278), [#291](https://github.com/open-mmlab/mmocr/pull/291), [#300](https://github.com/open-mmlab/mmocr/pull/300), [#328](https://github.com/open-mmlab/mmocr/pull/328)
3. Unified parameter initialization method which uses init_cfg in config files [#365](https://github.com/open-mmlab/mmocr/pull/365)

- Support TextOCR dataset [#293](https://github.com/open-mmlab/mmocr/pull/293)
- Support Total-Text dataset [#266](https://github.com/open-mmlab/mmocr/pull/266), [#273](https://github.com/open-mmlab/mmocr/pull/273), [#357](https://github.com/open-mmlab/mmocr/pull/357)

- Support grouping text detection box into lines [#290](https://github.com/open-mmlab/mmocr/pull/290), [#304](https://github.com/open-mmlab/mmocr/pull/304)
- Add benchmark_processing script that benchmarks data loading process [#261](https://github.com/open-mmlab/mmocr/pull/261)
- Add SynthText preprocessor for text recognition models [#351](https://github.com/open-mmlab/mmocr/pull/351), [#361](https://github.com/open-mmlab/mmocr/pull/361)
- Support batch inference during testing [#310](https://github.com/open-mmlab/mmocr/pull/310)
- Add user-friendly OCR inference script [#366](https://github.com/open-mmlab/mmocr/pull/366)

- Fix improper class ignorance in SDMGR Loss [#221](https://github.com/open-mmlab/mmocr/pull/221)

- Fix potential numerical zero division error in DRRG [#224](https://github.com/open-mmlab/mmocr/pull/224)
- Fix installing requirements with pip and mim [#242](https://github.com/open-mmlab/mmocr/pull/242)
- Fix dynamic input error of DBNet [#269](https://github.com/open-mmlab/mmocr/pull/269)
- Fix space parsing error in LineStrParser [#285](https://github.com/open-mmlab/mmocr/pull/285)
- Fix textsnake decode error [#264](https://github.com/open-mmlab/mmocr/pull/264)
- Correct isort setup [#288](https://github.com/open-mmlab/mmocr/pull/288)
- Fix a bug in SDMGR config [#316](https://github.com/open-mmlab/mmocr/pull/316)
- Fix kie_test_img for KIE nonvisual [#319](https://github.com/open-mmlab/mmocr/pull/319)
- Fix metafiles [#342](https://github.com/open-mmlab/mmocr/pull/342)
- Fix different device problem in FCENet [#334](https://github.com/open-mmlab/mmocr/pull/334)

- Ignore improper tailing empty characters in annotation files [#358](https://github.com/open-mmlab/mmocr/pull/358)
- Docs fixes [#247](https://github.com/open-mmlab/mmocr/pull/247), [#255](https://github.com/open-mmlab/mmocr/pull/255), [#265](https://github.com/open-mmlab/mmocr/pull/265), [#267](https://github.com/open-mmlab/mmocr/pull/267), [#268](https://github.com/open-mmlab/mmocr/pull/268), [#270](https://github.com/open-mmlab/mmocr/pull/270), [#276](https://github.com/open-mmlab/mmocr/pull/276), [#287](https://github.com/open-mmlab/mmocr/pull/287), [#330](https://github.com/open-mmlab/mmocr/pull/330), [#355](https://github.com/open-mmlab/mmocr/pull/355), [#367](https://github.com/open-mmlab/mmocr/pull/367)

- Fix NRTR config [#356](https://github.com/open-mmlab/mmocr/pull/356), [#370](https://github.com/open-mmlab/mmocr/pull/370)

- Add backend for resizeocr [#244](https://github.com/open-mmlab/mmocr/pull/244)
- Skip image processing pipelines in SDMGR novisual [#260](https://github.com/open-mmlab/mmocr/pull/260)
- Speedup DBNet [#263](https://github.com/open-mmlab/mmocr/pull/263)
- Update mmcv installation method in workflow [#323](https://github.com/open-mmlab/mmocr/pull/323)
- Add part of Chinese documentations [#353](https://github.com/open-mmlab/mmocr/pull/353), [#362](https://github.com/open-mmlab/mmocr/pull/362)
- Add support for ConcatDataset with two workflows [#348](https://github.com/open-mmlab/mmocr/pull/348)
- Add list_from_file and list_to_file utils [#226](https://github.com/open-mmlab/mmocr/pull/226)
- Speed up sort_vertex [#239](https://github.com/open-mmlab/mmocr/pull/239)

- Support distributed evaluation of KIE [#234](https://github.com/open-mmlab/mmocr/pull/234)
- Add pretrained FCENet on IC15 [#258](https://github.com/open-mmlab/mmocr/pull/258)
- Support CPU for OCR demo [#227](https://github.com/open-mmlab/mmocr/pull/227)
- Avoid extra image pre-processing steps [#375](https://github.com/open-mmlab/mmocr/pull/375)

## v0.2.0 (18/5/2021)

1. Add the NER approach Bert-softmax (NAACL'2019)
2. Add the text detection method DRRG (CVPR'2020)
3. Add the text detection method FCENet (CVPR'2021)
4. Increase the ease of use via adding text detection and recognition end-to-end demo, and colab online demo.
5. Simplify the installation.

- Add Bert-softmax for Ner task [#148](https://github.com/open-mmlab/mmocr/pull/148)

- Add DRRG [#189](https://github.com/open-mmlab/mmocr/pull/189)
- Add FCENet [#133](https://github.com/open-mmlab/mmocr/pull/133)
- Add end-to-end demo [#105](https://github.com/open-mmlab/mmocr/pull/105)
- Support batch inference [#86](https://github.com/open-mmlab/mmocr/pull/86) [#87](https://github.com/open-mmlab/mmocr/pull/87) [#178](https://github.com/open-mmlab/mmocr/pull/178)
- Add TPS preprocessor for text recognition [#117](https://github.com/open-mmlab/mmocr/pull/117) [#135](https://github.com/open-mmlab/mmocr/pull/135)
- Add demo documentation [#151](https://github.com/open-mmlab/mmocr/pull/151) [#166](https://github.com/open-mmlab/mmocr/pull/166) [#168](https://github.com/open-mmlab/mmocr/pull/168) [#170](https://github.com/open-mmlab/mmocr/pull/170) [#171](https://github.com/open-mmlab/mmocr/pull/171)

- Add checkpoint for Chinese recognition [#156](https://github.com/open-mmlab/mmocr/pull/156)
- Add metafile [#175](https://github.com/open-mmlab/mmocr/pull/175) [#176](https://github.com/open-mmlab/mmocr/pull/176) [#177](https://github.com/open-mmlab/mmocr/pull/177) [#182](https://github.com/open-mmlab/mmocr/pull/182) [#183](https://github.com/open-mmlab/mmocr/pull/183)
- Add support for numpy array inference [#74](https://github.com/open-mmlab/mmocr/pull/74)

- Fix the duplicated point bug due to transform for textsnake [#130](https://github.com/open-mmlab/mmocr/pull/130)
- Fix CTC loss NaN [#159](https://github.com/open-mmlab/mmocr/pull/159)
- Fix error raised if result is empty in demo [#144](https://github.com/open-mmlab/mmocr/pull/141)
- Fix results missing if one image has a large number of boxes [#98](https://github.com/open-mmlab/mmocr/pull/98)
- Fix package missing in dockerfile [#109](https://github.com/open-mmlab/mmocr/pull/109)

- Simplify installation procedure via removing compiling [#188](https://github.com/open-mmlab/mmocr/pull/188)
- Speed up panet post processing so that it can detect dense texts [#188](https://github.com/open-mmlab/mmocr/pull/188)
- Add zh-CN README [#70](https://github.com/open-mmlab/mmocr/pull/70) [#95](https://github.com/open-mmlab/mmocr/pull/95)
- Support windows [#89](https://github.com/open-mmlab/mmocr/pull/89)
- Add Colab [#147](https://github.com/open-mmlab/mmocr/pull/147) [#199](https://github.com/open-mmlab/mmocr/pull/199)
- Add 1-step installation using conda environment [#193](https://github.com/open-mmlab/mmocr/pull/193) [#194](https://github.com/open-mmlab/mmocr/pull/194) [#195](https://github.com/open-mmlab/mmocr/pull/195)

## v0.1.0 (7/4/2021)

- MMOCR is released.

### Main Features

- Support text detection, text recognition and the corresponding downstream tasks such as key information extraction.
- For text detection, support both single-step (`PSENet`, `PANet`, `DBNet`, `TextSnake`) and two-step (`MaskRCNN`) methods.
- For text recognition, support CTC-loss based method `CRNN`; Encoder-decoder (with attention) based methods `SAR`, `Robustscanner`; Segmentation based method `SegOCR`; Transformer based method `NRTR`.
- For key information extraction, support GCN based method `SDMG-R`.
- Provide checkpoints and log files for all of the methods above.

### the comprehensive model loading api table

|load from|alexnet.param|alexnet.param.bin|alexnet.bin|
|file path|load_param(const char*)|load_param_bin(const char*)|load_model(const char*)|
|file descriptor|load_param(FILE*)|load_param_bin(FILE*)|load_model(FILE*)|
|file memory|load_param_mem(const char*)|load_param(const unsigned char*)|load_model(const unsigned char*)|
|android asset|load_param(AAsset*)|load_param_bin(AAsset*)|load_model(AAsset*)|
|android asset path|load_param(AAssetManager*, const char*)|load_param_bin(AAssetManager*, const char*)|load_model(AAssetManager*, const char*)|
|custom IO reader|load_param(const DataReader&)|load_param_bin(const DataReader&)|load_model(const DataReader&)|

### points to note

1. Either of the following combination shall be enough for loading model
    * alexnet.param + alexnet.bin
    * alexnet.param.bin + alexnet.bin

2. Never modify Net opt member after loading

3. Most loading functions return 0 if success, except loading alexnet.param.bin and alexnet.bin from file memory, which returns the bytes consumed after loading
    * int Net::load_param(const unsigned char*)
    * int Net::load_model(const unsigned char*)

4. It is recommended to load model from Android asset directly to avoid copying them to sdcard on Android platform

5. The custom IO reader interface can be used to implement on-the-fly model decryption and loading

# Otter

> [Otter: A Multi-Modal Model with In-Context Instruction Tuning](https://arxiv.org/abs/2305.03726)

Large language models (LLMs) have demonstrated significant universal capabilities as few/zero-shot learners in various tasks due to their pre-training on vast amounts of text data, as exemplified by GPT-3, which boosted to InstrctGPT and ChatGPT, effectively following natural language instructions to accomplish real-world tasks. In this paper, we propose to introduce instruction tuning into multi-modal models, motivated by the Flamingo model's upstream interleaved format pretraining dataset. We adopt a similar approach to construct our MultI-Modal In-Context Instruction Tuning (MIMIC-IT) dataset. We then introduce Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following ability and in-context learning. We also optimize OpenFlamingo's implementation for researchers, democratizing the required training resources from 1$\times$ A100 GPU to 4$\times$ RTX-3090 GPUs, and integrate both OpenFlamingo and Otter into Huggingface Transformers for more researchers to incorporate the models into their customized training and inference pipelines.

<img src="https://camo.githubusercontent.com/70613ab882a7827808148a2c577029d544371e707b0832a0b01151c54ce553c3/68747470733a2f2f692e706f7374696d672e63632f5477315a304243572f6f7474657276302d322d64656d6f2e706e67" width="80%"/>
</div>

```python

model = get_model('otter-9b_3rdparty_caption', pretrained=True, device='cuda', generation_cfg=dict(max_new_tokens=50))
out = inference_model(model, 'demo/cat-dog.png')
print(out)
# {'pred_caption': 'The image features two adorable small puppies sitting next to each other on the grass. One puppy is brown and white, while the other is tan and white. They appear to be relaxing outdoors, enjoying each other'}
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/otter/otter-9b_caption.py https://download.openmmlab.com/mmclassification/v1/otter/otter-9b-adapter_20230613-51c5be8d.pth
```

| Model                         | Params (M) |  BLEU-4  |  CIDER   |            Config             |                                                 Download                                                 |
| :---------------------------- | :--------: | :------: | :------: | :---------------------------: | :------------------------------------------------------------------------------------------------------: |

| `otter-9b_3rdparty_caption`\* |  8220.45   | Upcoming | Upcoming | [config](otter-9b_caption.py) | [model](https://download.openmmlab.com/mmclassification/v1/otter/otter-9b-adapter_20230613-51c5be8d.pth) |

*Models with * are converted from the [official repo](https://github.com/Luodian/Otter/tree/main). The config files of these models are only for inference. We haven't reproduce the training results.*

| Model                     | Params (M) | Accuracy |          Config           |                                                 Download                                                 |
| :------------------------ | :--------: | :------: | :-----------------------: | :------------------------------------------------------------------------------------------------------: |

| `otter-9b_3rdparty_vqa`\* |  8220.45   | Upcoming | [config](otter-9b_vqa.py) | [model](https://download.openmmlab.com/mmclassification/v1/otter/otter-9b-adapter_20230613-51c5be8d.pth) |

```bibtex
@article{li2023otter,
  title={Otter: A Multi-Modal Model with In-Context Instruction Tuning},
  author={Li, Bo and Zhang, Yuanhan and Chen, Liangyu and Wang, Jinghao and Yang, Jingkang and Liu, Ziwei},
  journal={arXiv preprint arXiv:2305.03726},
}

@article{li2023mimicit,
    title={MIMIC-IT: Multi-Modal In-Context Instruction Tuning},

journal={arXiv preprint arXiv:2305.03726},
}

    author={Bo Li and Yuanhan Zhang and Liangyu Chen and Jinghao Wang and Fanyi Pu and Jingkang Yang and Chunyuan Li and Ziwei Liu},
    year={2023},
    eprint={2306.05425},
}
```

# Preparing AVA

```BibTeX
@inproceedings{gu2018ava,
  title={Ava: A video dataset of spatio-temporally localized atomic visual actions},
  author={Gu, Chunhui and Sun, Chen and Ross, David A and Vondrick, Carl and Pantofaru, Caroline and Li, Yeqing and Vijayanarasimhan, Sudheendra and Toderici, George and Ricco, Susanna and Sukthankar, Rahul and others},
  pages={6047--6056},
}
```

Before we start, please make sure that the directory is located at `$MMACTION2/tools/data/ava/`.

```shell
```

```shell
```

This command will download `ava_v2.1.zip` for AVA `v2.1` annotation. If you need the AVA `v2.2` annotation, you can try the following script.

```shell
VERSION=2.2 bash download_annotations.sh
```

Then, use the following script to prepare videos. The codes are adapted from the [official crawler](https://github.com/cvdfoundation/ava-dataset).
Note that this might take a long time.

```shell
```

Or you can use the following command to downloading AVA videos in parallel using a python script.

```shell
bash download_videos_parallel.sh
```

Note that if you happen to have sudoer or have [GNU parallel](https://www.gnu.org/software/parallel/) on your machine,
you can speed up the procedure by downloading in parallel.

```shell
# sudo apt-get install parallel
bash download_videos_gnu_parallel.sh

```shell
```

Cut each video from its 15th to 30th minute and make them at 30 fps.

```shell
bash cut_videos.sh
```

## Step 4. Extract RGB and Flow

If you have plenty of SSD space, then we recommend extracting frames there for better I/O performance. And you can run the following script to soft link the extracted frames.

```shell
mkdir /mnt/SSD/ava_extracted/
ln -s /mnt/SSD/ava_extracted/ ../data/ava/rawframes/
```

```shell

```

```shell
```

If you didn't install denseflow, you can still extract RGB frames using ffmpeg by the following script.

```shell
bash extract_rgb_frames_ffmpeg.sh
```

```shell
```

## Step 5. Fetch Proposal Files

The scripts are adapted from FAIR's [Long-Term Feature Banks](https://github.com/facebookresearch/video-long-term-feature-banks).

Run the following scripts to fetch the pre-computed proposal list.

```shell
bash fetch_ava_proposals.sh
```

## Step 6. Folder Structure

After the whole data pipeline for AVA preparation.
you can get the rawframes (RGB + Flow), videos and annotation files for AVA.

```shell
```

In the context of the whole project (for AVA only), the *minimal* folder structure will look like:
(*minimal* means that some data are not necessary: for example, you may want to evaluate AVA using the original video format.)

```
mmaction2
├── tools
├── data
│   ├── ava
│   │   |   ├── ava_dense_proposals_train.FAIR.recall_93.9.pkl
│   │   |   ├── ava_dense_proposals_val.FAIR.recall_93.9.pkl
│   │   |   ├── ava_dense_proposals_test.FAIR.recall_93.9.pkl
│   │   |   ├── ava_train_v2.1.csv
│   │   |   ├── ava_val_v2.1.csv
│   │   |   ├── ava_train_excluded_timestamps_v2.1.csv
│   │   |   ├── ava_val_excluded_timestamps_v2.1.csv
│   │   |   ├── ava_action_list_v2.1_for_activitynet_2018.pbtxt
│   │   │   ├── 053oq2xB3oU.mkv

│   │   │   ├── 0f39OWEqJ24.mp4
│   │   │   ├── ...
│   │   ├── videos_15min
│   │   │   ├── ...
│   │   │   ├── 053oq2xB3oU
|   │   │   │   ├── img_00001.jpg
|   │   │   │   ├── img_00002.jpg
|   │   │   │   ├── ...
```

For training and evaluating on AVA, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).

1. O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014

  <img src="docs/zh_cn/_static/image/mmagic-logo.png" width="500px"/>
  </div>
    <sup>
      </a>
    </sup>
    <sup>
      </a>
    </sup>
  </div>

[![docs](https://img.shields.io/badge/docs-latest-blue)](https://mmagic.readthedocs.io/zh_CN/latest/)

[![badge](https://github.com/open-mmlab/mmagic/workflows/build/badge.svg)](https://github.com/open-mmlab/mmagic/actions)
[![codecov](https://codecov.io/gh/open-mmlab/mmagic/branch/master/graph/badge.svg)](https://codecov.io/gh/open-mmlab/mmagic)
[![license](https://img.shields.io/github/license/open-mmlab/mmagic.svg)](https://github.com/open-mmlab/mmagic/blob/main/LICENSE)
[![open issues](https://isitmaintained.com/badge/open/open-mmlab/mmagic.svg)](https://github.com/open-mmlab/mmagic/issues)
[![issue resolution](https://isitmaintained.com/badge/resolution/open-mmlab/mmagic.svg)](https://github.com/open-mmlab/mmagic/issues)

[📘使用文档](https://mmagic.readthedocs.io/zh_CN/latest/) |
[🛠️安装教程](https://mmagic.readthedocs.io/zh_CN/latest/get_started/install.html) |

[📊模型库](https://mmagic.readthedocs.io/zh_CN/latest/model_zoo/overview.html) |
[🆕更新记录](https://mmagic.readthedocs.io/zh_CN/latest/changelog.html) |
[🚀进行中的项目](https://github.com/open-mmlab/mmagic/projects) |
[🤔提出问题](https://github.com/open-mmlab/mmagic/issues)

</div>

</div>

### 最新的 [**MMagic v1.0.1**](https://github.com/open-mmlab/mmagic/releases/tag/v1.0.1) 版本已经在 \[26/05/2023\] 发布:

- 支持 StableDiffusion tomesd 加速.
- 支持所有 inpainting/matting/image restoration 模型的 inferencer.
- 支持 animated drawings.
- 支持 Style-Based Global Appearance Flow for Virtual Try-On.
- 修复 pip install 时 inferencer 无法使用的问题.

我们正式发布 MMagic v1.0.0 版本，源自 [MMEditing](https://github.com/open-mmlab/mmediting) 和 [MMGeneration](https://github.com/open-mmlab/mmgeneration)。

经过 OpenMMLab 2.0 框架的迭代更新以及与 MMGeneration 的合并，MMEditing 已经成为了一个支持基于 GAN 和 CNN 的底层视觉算法的强大工具。而今天，MMEditing 将更加拥抱生成式 AI（Generative AI），正式更名为 **MMagic**（**M**ultimodal **A**dvanced, **G**enerative, and **I**ntelligent **C**reation），致力于打造更先进、更全面的 AIGC 开源算法库。MMagic 将为广大研究者与 AIGC 爱好者们提供更加快捷灵活的实验支持，助力你的 AIGC 探索之旅。

以下是此次版本发布的重点新功能:

**1. 新算法**

我们支持了4个新任务以及11个新算法。

  - GLIDE
  - EG3D
  - NAFNet
  - SwinIR

针对 Diffusion Model，我们提供了以下“魔法”

- 支持基于 Stable Diffusion 与 Disco Diffusion 的图像生成.
- 支持 Dreambooth 以及 DreamBooth LoRA 等 Finetune 方法.
- 支持 ControlNet 进行可控性的文本到图像生成.

- 支持 xFormers 加速和优化策略，提高训练与推理效率.
- 支持基于 MultiFrame Render 的视频生成.
- 支持通过 Wrapper 调用 Diffusers 的基础模型以及采样策略.

**3. 框架升级**

通过 OpenMMLab 2.0 框架的 MMEngine 和 MMCV， MMagic 在以下几方面完成升级：

- 重构 DataSample，支持 batch 维度的组合与拆分.
- 重构 DataPreprocessor，并统一各种任务在训练与推理时的数据格式.
- 重构 MultiValLoop 与 MultiTestLoop，同时支持生成类型指标（e.g. FID）与重建类型指标（e.g. SSIM） 的评测，同时支持一次性评测多个数据集
- 支持本地可视化以及使用 tensorboard 或 wandb的可视化.
- 支持 33+ 算法 Pytorch 2.0 加速.

**MMagic** 已经支持了[MMEditing](https://github.com/open-mmlab/mmediting)和[MMGeneration](https://github.com/open-mmlab/mmgeneration)中的全量任务、模型、优化函数和评价指标 ，并基于[MMEngine](https://github.com/open-mmlab/mmengine)统一了各组件接口 😍。

如果想了解更多版本更新细节和历史信息，请阅读[更新日志](docs/zh_cn/changelog.md)。如果想从[旧版本](https://github.com/open-mmlab/mmagic/tree/master) MMEditing 0.x 迁移到新版本 MMagic 1.x，请阅读[迁移文档](docs/zh_cn/migration/overview.md)。

## 📄 目录

- [📖 介绍](#-介绍)
- [🙌 参与贡献](#-参与贡献)
- [🛠️ 安装](#%EF%B8%8F-安装)
- [📊 模型库](#-模型库)
- [🤝 致谢](#-致谢)
- [🖊️ 引用](#%EF%B8%8F-引用)
- [🎫 许可证](#-许可证)
- [🏗️ ️OpenMMLab 的其他项目](#%EF%B8%8F-️openmmlab-的其他项目)

## 📖 介绍

MMagic 是基于 PyTorch 的图像&视频编辑和生成开源工具箱。是 [OpenMMLab](https://openmmlab.com/) 项目的成员之一。

目前 MMagic 支持多种图像和视频的生成/编辑任务。

主分支代码的最佳实践基于 **Python 3.8+** 和 **PyTorch 1.9+** 。

### ✨ 主要特性

- **SOTA 算法**

  MMagic 提供了处理、编辑、生成图像和视频的 SOTA 算法。

- **强有力且流行的应用**

  MMagic 支持了流行的图像修复、图文生成、3D生成、图像修补、抠图、超分辨率和生成等任务的应用。特别是 MMagic 支持了 Stable Diffusion 的微调和许多激动人心的 diffusion 应用，例如 ControlNet 动画生成。MMagic 也支持了 GANs 的插值，投影，编辑和其他流行的应用。请立即开始你的 AIGC 探索之旅！

- **高效的框架**

MMagic 支持了流行的图像修复、图文生成、3D生成、图像修补、抠图、超分辨率和生成等任务的应用。特别是 MMagic 支持了 Stable Diffusion 的微调和许多激动人心的 diffusion 应用，例如 ControlNet 动画生成。MMagic 也支持了 GANs 的插值，投影，编辑和其他流行的应用。请立即开始你的 AIGC 探索之旅！

  通过 OpenMMLab 2.0 框架的 MMEngine 和 MMCV， MMagic 将编辑框架分解为不同的组件，并且可以通过组合不同的模块轻松地构建自定义的编辑器模型。我们可以像搭建“乐高”一样定义训练流程，提供丰富的组件和策略。在 MMagic 中，你可以使用不同的 APIs 完全控制训练流程。得益于 [MMSeparateDistributedDataParallel](https://github.com/open-mmlab/mmengine/blob/main/mmengine/model/wrappers/seperate_distributed.py), 动态模型结构的分布式训练可以轻松实现。

## 🙌 参与贡献

越来越多社区贡献者的加入使我们的算法库日益发展。最近由社区贡献的项目包括：

- [GLIDE](projects/glide/configs/README.md) 来自 @Taited.
- [Restormer](configs/restormer/README.md) 来自 @AlexZou14.
- [SwinIR](configs/swinir/README.md) 来自 @Zdafeng.

为使向 MMagic 中添加项目更加容易，我们开启了 [Projects](projects/README.md) 。

感谢您为改善 MMagic 所做的所有贡献。请参阅 MMCV 中的 [CONTRIBUTING.md](https://github.com/open-mmlab/mmcv/blob/main/CONTRIBUTING_zh-CN.md) 和 MMEngine 中的 [CONTRIBUTING.md](https://github.com/open-mmlab/mmengine/blob/main/CONTRIBUTING_zh-CN.md) 以获取贡献指南。

## 🛠️ 安装

MMagic 依赖 [PyTorch](https://pytorch.org/)，[MMEngine](https://github.com/open-mmlab/mmengine) 和 [MMCV](https://github.com/open-mmlab/mmcv)，以下是安装的简要步骤。

**步骤 1.**
依照[官方教程](https://pytorch.org/get-started/locally/)安装 PyTorch 。

**步骤 2.**
使用 [MIM](https://github.com/open-mmlab/mim) 安装 MMCV，MMEngine 和 MMagic 。

```
```

**步骤 3.**
验证 MMagic 安装成功。

```shell
cd ~
```

**开始使用**

```

**步骤 3.**

```shell
cd ~
```

**开始使用**

成功安装 MMagic 后，你可以很容易地上手使用 MMagic！仅需几行代码，你就可以使用 MMagic 完成文本生成图像！

```python
```

请参考[快速运行](docs/zh_cn/get_started/quick_run.md)和[推理演示](docs/zh_cn/user_guides/inference.md)获取 MMagic 的基本用法。

**从源码安装 MMagic**

使用以下命令从源码安装 MMagic，你可以选择不使用已发布的稳定版本，而在最新开发的版本上进行实验。

```
cd mmagic
```

更详细的安装指南请参考 [安装指南](docs/zh_cn/get_started/install.md) 。

## 📊 模型库

  <b>支持的算法</b>
</div>
  <tbody>

## 📊 模型库

</div>
  <tbody>
      <td>
      </td>
      <td>
      </td>
      <td>
      </td>
      <td>
      </td>
    </tr>
      <td>
        <ul>
      </ul>
      </td>
      <td>
        <ul>

</td>
      <td>
        <ul>
        </ul>
      </td>
      <td>
        <ul>

</td>
      <td>
        <ul>
        </ul>
      </td>
      <td>
        <ul>

<li><a href="configs/ttsr/README.md">TTSR (CVPR'2020)</a></li>
        </ul>
      </td>
    </tr>
</td>
    </tr>
  </tbody>
<tbody>
      <td>
      </td>
      <td>
      </td>
      <td>
      </td>
      <td>
      </td>
    </tr>
      <td>
        <ul>

<li><a href="configs/tof/README.md">TOF (IJCV'2019)</a></li>
      </ul>
      </td>
      <td>
        <ul>
        </ul>
      </td>
      <td>
        <ul>
        </ul>
      </td>
      <td>
        <ul>

</ul>
      </td>
      <td>
        <ul>
        </ul>
      </td>
      <td>
        <ul>
        </ul>
      </td>
    </tr>
</td>
    </tr>
  </tbody>
<tbody>
      <td>
      </td>
      <td>
      </td>
      <td>
      </td>
      <td>
      </td>
    </tr>
      <td>
        <ul>

<li><a href="configs/deepfillv1/README.md">DeepFillv1 (CVPR'2018)</a></li>
        </ul>
      </td>
      <td>
        <ul>
        </ul>
      </td>
      <td>
        <ul>

<li><a href="configs/guided_diffusion/README.md">Guided Diffusion (NeurIPS'2021)</a></li>
        </ul>
      </td>
      <td>
        <ul>
        </ul>
      </td>
    </tr>
</td>
    </tr>
  </tbody>
</table>

请参考[模型库](https://mmagic.readthedocs.io/zh_CN/latest/model_zoo/overview.html)了解详情。

## 🤝 致谢

</td>
    </tr>
</td>
    </tr>
  </tbody>
</table>

## 🤝 致谢

MMagic 是一款由不同学校和公司共同贡献的开源项目。我们感谢所有为项目提供算法复现和新功能支持的贡献者，以及提供宝贵反馈的用户。我们希望该工具箱和基准测试可以为社区提供灵活的代码工具，供用户复现现有算法并开发自己的新模型，从而不断为开源社区提供贡献。

</a>

## 🖊️ 引用

如果 MMagic 对您的研究有所帮助，请按照如下 bibtex 引用它。

```bibtex
}
```

```bibtex

year = {2023}
}
```

```bibtex
}
```

## 🎫 许可证

本项目开源自 [Apache 2.0 license](LICENSE)。

## 🏗️ ️OpenMMLab 的其他项目

- [MMEngine](https://github.com/open-mmlab/mmengine): OpenMMLab MMEngine.
- [MMPreTrain](https://github.com/open-mmlab/mmpretrain): OpenMMLab 预训练工具箱

- [MMOCR](https://github.com/open-mmlab/mmocr): OpenMMLab 全流程文字检测识别理解工具箱

扫描下方的二维码可关注 OpenMMLab 团队的 [知乎官方账号](https://www.zhihu.com/people/openmmlab)，加入 OpenMMLab 团队的 [官方交流 QQ 群](https://jq.qq.com/?_wv=1027&k=K0QI8ByU)，或通过群主小喵加入微信官方交流群。

<img src="docs/zh_cn/_static/image/zhihu_qrcode.jpg" height="500" />  <img src="https://user-images.githubusercontent.com/25839884/203927852-e15def4d-a0eb-4dfc-9bfb-7cf09ea945d0.png" height="500" /> <img src="https://raw.githubusercontent.com/open-mmlab/mmcv/master/docs/en/_static/wechat_qrcode.jpg" height="500" />
</div>

</div>

# Train

In this tutorial, we will introduce how to use the scripts provided in MMPretrain to start a training task. If
you need, we also have some practice examples about [how to pretrain with custom dataset](../notes/pretrain_custom_dataset.md)
and [how to finetune with custom dataset](../notes/finetune_custom_dataset.md).

## Train with your PC

```shell
```

By default, MMPretrain prefers GPU to CPU. If you want to train a model on CPU, please empty `CUDA_VISIBLE_DEVICES` or set it to -1 to make GPU invisible to the program.

```bash
```
````

```bash
```
````

| `--no-pin-memory`                     | Whether to disable the `pin_memory` option in dataloaders.                                                                                                          |
| `--no-persistent-workers`             | Whether to disable the `persistent_workers` option in dataloaders.                                                                                                  |

```shell
```

| `[PY_ARGS]`   | The other optional arguments of `tools/train.py`, see [here](#train-with-your-pc). |

```shell
```

different ports and visible devices.

```shell
```

### Multiple machines in the same network

```shell

```shell
```

```shell
```

Comparing with multi-GPUs in a single machine, you need to specify some extra environment variables:

### Multiple machines managed with slurm

If you run MMPretrain on a cluster managed with [slurm](https://slurm.schedmd.com/), you can use the script `tools/slurm_train.sh`.

```shell
```

```

| `GPUS_PER_NODE` | The number of GPUs to be allocated per node..                                                              |

# Data Preparation for Rotation Detection

It is recommended to symlink the dataset root to `$MMROTATE/data`.

Datasets supported in MMRotate:

- [DOTA Dataset](dota/README.md) \[ [Homepage](https://captain-whu.github.io/DOTA/) \]
- [DIOR Dataset](dior/README.md) \[ [Homepage](https://gcheng-nwpu.github.io/#Datasets) \]
- [SSDD Dataset](ssdd/README.md)
- [HRSC Dataset](hrsc/README.md)
- [HRSID Dataset](hrsid/README.md)
- [SRSDD Dataset](srsdd/README.md)
- [RSDD Dataset](rsdd/README.md)

</div>

|    Backbone     |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                     Config                      |                                                                                                                                                                            Download                                                                                                                                                                             |

| :-------------: | :-----: | :-----: | :------: | :------------: | :----: | :-----: | :---------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|    R-50-FPN     |  caffe  |   1x    |   4.3    |                |  38.0  |  34.4   | [config](./mask-rcnn_r50-caffe_fpn_1x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_caffe_fpn_1x_coco/mask_rcnn_r50_caffe_fpn_1x_coco_bbox_mAP-0.38__segm_mAP-0.344_20200504_231812-0ebd1859.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_caffe_fpn_1x_coco/mask_rcnn_r50_caffe_fpn_1x_coco_20200504_231812.log.json)    |

|    R-50-FPN     | pytorch |   1x    |   4.4    |      16.1      |  38.2  |  34.7   |    [config](./mask-rcnn_r50_fpn_1x_coco.py)     |                                  [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_fpn_1x_coco/mask_rcnn_r50_fpn_1x_coco_20200205-d4b0c5d6.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_fpn_1x_coco/mask_rcnn_r50_fpn_1x_coco_20200205_050542.log.json)                                  |
| R-50-FPN (FP16) | pytorch |   1x    |   3.6    |      24.1      |  38.1  |  34.7   |  [config](./mask-rcnn_r50_fpn_amp-1x_coco.py)   |                             [model](https://download.openmmlab.com/mmdetection/v2.0/fp16/mask_rcnn_r50_fpn_fp16_1x_coco/mask_rcnn_r50_fpn_fp16_1x_coco_20200205-59faf7e4.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/fp16/mask_rcnn_r50_fpn_fp16_1x_coco/mask_rcnn_r50_fpn_fp16_1x_coco_20200205_130539.log.json)                             |

|    R-50-FPN     | pytorch |   2x    |    -     |       -        |  39.2  |  35.4   |    [config](./mask-rcnn_r50_fpn_2x_coco.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_fpn_2x_coco/mask_rcnn_r50_fpn_2x_coco_bbox_mAP-0.392__segm_mAP-0.354_20200505_003907-3e542a40.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_fpn_2x_coco/mask_rcnn_r50_fpn_2x_coco_20200505_003907.log.json)               |
|    R-101-FPN    |  caffe  |   1x    |          |                |  40.4  |  36.4   | [config](./mask-rcnn_r101-caffe_fpn_1x_coco.py) |                [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r101_caffe_fpn_1x_coco/mask_rcnn_r101_caffe_fpn_1x_coco_20200601_095758-805e06c1.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r101_caffe_fpn_1x_coco/mask_rcnn_r101_caffe_fpn_1x_coco_20200601_095758.log.json)                 |

|    R-101-FPN    | pytorch |   1x    |   6.4    |      13.5      |  40.0  |  36.1   |    [config](./mask-rcnn_r101_fpn_1x_coco.py)    |                                [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r101_fpn_1x_coco/mask_rcnn_r101_fpn_1x_coco_20200204-1efe0ed5.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r101_fpn_1x_coco/mask_rcnn_r101_fpn_1x_coco_20200204_144809.log.json)                                |
|    R-101-FPN    | pytorch |   2x    |    -     |       -        |  40.8  |  36.6   |    [config](./mask-rcnn_r101_fpn_2x_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r101_fpn_2x_coco/mask_rcnn_r101_fpn_2x_coco_bbox_mAP-0.408__segm_mAP-0.366_20200505_071027-14b391c7.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r101_fpn_2x_coco/mask_rcnn_r101_fpn_2x_coco_20200505_071027.log.json)             |

| X-101-32x4d-FPN | pytorch |   1x    |   7.6    |      11.3      |  41.9  |  37.5   | [config](./mask-rcnn_x101-32x4d_fpn_1x_coco.py) |                    [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_x101_32x4d_fpn_1x_coco/mask_rcnn_x101_32x4d_fpn_1x_coco_20200205-478d0b67.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_x101_32x4d_fpn_1x_coco/mask_rcnn_x101_32x4d_fpn_1x_coco_20200205_034906.log.json)                    |
| X-101-32x4d-FPN | pytorch |   2x    |    -     |       -        |  42.2  |  37.8   | [config](./mask-rcnn_x101-32x4d_fpn_2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_x101_32x4d_fpn_2x_coco/mask_rcnn_x101_32x4d_fpn_2x_coco_bbox_mAP-0.422__segm_mAP-0.378_20200506_004702-faef898c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_x101_32x4d_fpn_2x_coco/mask_rcnn_x101_32x4d_fpn_2x_coco_20200506_004702.log.json) |

| X-101-64x4d-FPN | pytorch |   1x    |   10.7   |      8.0       |  42.8  |  38.4   | [config](./mask-rcnn_x101-64x4d_fpn_1x_coco.py) |                    [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_x101_64x4d_fpn_1x_coco/mask_rcnn_x101_64x4d_fpn_1x_coco_20200201-9352eb0d.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_x101_64x4d_fpn_1x_coco/mask_rcnn_x101_64x4d_fpn_1x_coco_20200201_124310.log.json)                    |
| X-101-64x4d-FPN | pytorch |   2x    |    -     |       -        |  42.7  |  38.1   | [config](./mask-rcnn_x101-64x4d_fpn_2x_coco.py) |                [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_x101_64x4d_fpn_2x_coco/mask_rcnn_x101_64x4d_fpn_2x_coco_20200509_224208-39d6f70c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_x101_64x4d_fpn_2x_coco/mask_rcnn_x101_64x4d_fpn_2x_coco_20200509_224208.log.json)                 |

| X-101-32x8d-FPN | pytorch |   1x    |   10.6   |       -        |  42.8  |  38.3   | [config](./mask-rcnn_x101-32x8d_fpn_1x_coco.py) |                [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_x101_32x8d_fpn_1x_coco/mask_rcnn_x101_32x8d_fpn_1x_coco_20220630_173841-0aaf329e.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_x101_32x8d_fpn_1x_coco/mask_rcnn_x101_32x8d_fpn_1x_coco_20220630_173841.log.json)                 |

## Pre-trained Models

We also train some models with longer schedules and multi-scale training. The users could finetune them for downstream tasks.

|                             Backbone                             |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                         Config                          |                                                                                                                                                                                                    Download                                                                                                                                                                                                     |

|     [R-50-FPN](./mask-rcnn_r50-caffe_fpn_ms-poly-2x_coco.py)     |  caffe  |   2x    |   4.3    |                |  40.3  |  36.5   | [config](./mask-rcnn_r50-caffe_fpn_ms-poly-2x_coco.py)  | [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_caffe_fpn_mstrain-poly_2x_coco/mask_rcnn_r50_caffe_fpn_mstrain-poly_2x_coco_bbox_mAP-0.403__segm_mAP-0.365_20200504_231822-a75c98ce.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_caffe_fpn_mstrain-poly_2x_coco/mask_rcnn_r50_caffe_fpn_mstrain-poly_2x_coco_20200504_231822.log.json) |

|     [R-50-FPN](./mask-rcnn_r50-caffe_fpn_ms-poly-3x_coco.py)     |  caffe  |   3x    |   4.3    |                |  40.8  |  37.0   | [config](./mask-rcnn_r50-caffe_fpn_ms-poly-3x_coco.py)  | [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco_bbox_mAP-0.408__segm_mAP-0.37_20200504_163245-42aa3d00.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco_20200504_163245.log.json)  |

|        [R-50-FPN](./mask-rcnn_r50_fpn_ms-poly-3x_coco.py)        | pytorch |   3x    |   4.1    |                |  40.9  |  37.1   |    [config](./mask-rcnn_r50_fpn_ms-poly-3x_coco.py)     |                            [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_fpn_mstrain-poly_3x_coco/mask_rcnn_r50_fpn_mstrain-poly_3x_coco_20210524_201154-21b550bb.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_fpn_mstrain-poly_3x_coco/mask_rcnn_r50_fpn_mstrain-poly_3x_coco_20210524_201154.log.json)                             |

|    [R-101-FPN](./mask-rcnn_r101-caffe_fpn_ms-poly-3x_coco.py)    |  caffe  |   3x    |   5.9    |                |  42.9  |  38.5   | [config](./mask-rcnn_r101-caffe_fpn_ms-poly-3x_coco.py) |                   [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r101_caffe_fpn_mstrain-poly_3x_coco/mask_rcnn_r101_caffe_fpn_mstrain-poly_3x_coco_20210526_132339-3c33ce02.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn_r101_caffe_fpn_mstrain-poly_3x_coco/mask_rcnn_r101_caffe_fpn_mstrain-poly_3x_coco_20210526_132339.log.json)                    |

|       [R-101-FPN](./mask-rcnn_r101_fpn_ms-poly-3x_coco.py)       | pytorch |   3x    |   6.1    |                |  42.7  |  38.5   |    [config](./mask-rcnn_r101_fpn_ms-poly-3x_coco.py)    |                          [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r101_fpn_mstrain-poly_3x_coco/mask_rcnn_r101_fpn_mstrain-poly_3x_coco_20210524_200244-5675c317.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r101_fpn_mstrain-poly_3x_coco/mask_rcnn_r101_fpn_mstrain-poly_3x_coco_20210524_200244.log.json)                           |

| [x101-32x4d-FPN](./mask-rcnn_x101-32x4d_fpn_ms-poly-3x_coco.py)  | pytorch |   3x    |   7.3    |                |  43.6  |  39.0   | [config](./mask-rcnn_x101-32x4d_fpn_ms-poly-3x_coco.py) |              [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_x101_32x4d_fpn_mstrain-poly_3x_coco/mask_rcnn_x101_32x4d_fpn_mstrain-poly_3x_coco_20210524_201410-abcd7859.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_x101_32x4d_fpn_mstrain-poly_3x_coco/mask_rcnn_x101_32x4d_fpn_mstrain-poly_3x_coco_20210524_201410.log.json)               |

| [X-101-32x8d-FPN](./mask-rcnn_x101-32x8d_fpn_ms-poly-3x_coco.py) | pytorch |   1x    |   10.4   |                |  43.4  |  39.0   | [config](./mask-rcnn_x101-32x8d_fpn_ms-poly-1x_coco.py) |              [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_x101_32x8d_fpn_mstrain-poly_1x_coco/mask_rcnn_x101_32x8d_fpn_mstrain-poly_1x_coco_20220630_170346-b4637974.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_x101_32x8d_fpn_mstrain-poly_1x_coco/mask_rcnn_x101_32x8d_fpn_mstrain-poly_1x_coco_20220630_170346.log.json)               |

| [X-101-32x8d-FPN](./mask-rcnn_x101-32x8d_fpn_ms-poly-3x_coco.py) | pytorch |   3x    |   10.3   |                |  44.3  |  39.5   | [config](./mask-rcnn_x101-32x8d_fpn_ms-poly-3x_coco.py) |              [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_x101_32x8d_fpn_mstrain-poly_3x_coco/mask_rcnn_x101_32x8d_fpn_mstrain-poly_3x_coco_20210607_161042-8bd2c639.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_x101_32x8d_fpn_mstrain-poly_3x_coco/mask_rcnn_x101_32x8d_fpn_mstrain-poly_3x_coco_20210607_161042.log.json)               |

| [X-101-64x4d-FPN](./mask-rcnn_x101-64x4d_fpn_ms-poly_3x_coco.py) | pytorch |   3x    |   10.4   |                |  44.5  |  39.7   | [config](./mask-rcnn_x101-64x4d_fpn_ms-poly_3x_coco.py) |              [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_x101_64x4d_fpn_mstrain-poly_3x_coco/mask_rcnn_x101_64x4d_fpn_mstrain-poly_3x_coco_20210526_120447-c376f129.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_x101_64x4d_fpn_mstrain-poly_3x_coco/mask_rcnn_x101_64x4d_fpn_mstrain-poly_3x_coco_20210526_120447.log.json)               |

```latex
@article{He_2017,
   title={Mask R-CNN},
   journal={2017 IEEE International Conference on Computer Vision (ICCV)},
   publisher={IEEE},
   author={He, Kaiming and Gkioxari, Georgia and Dollar, Piotr and Girshick, Ross},
   month={Oct}
}
```

# 支持新数据集

尽管 OpenCompass 已经包含了大多数常用数据集，用户在支持新数据集的时候需要完成以下几个步骤：

1. 在 `opencompass/datasets` 文件夹新增数据集脚本 `mydataset.py`, 该脚本需要包含：

   - 数据集及其加载方式，需要定义一个 `MyDataset` 类，实现数据集加载方法 `load`，该方法为静态方法，需要返回 `datasets.Dataset` 类型的数据。这里我们使用 huggingface dataset 作为数据集的统一接口，避免引入额外的逻辑。具体示例如下：

   ```python
   import datasets
   from .base import BaseDataset

   class MyDataset(BaseDataset):

       @staticmethod
       def load(**kwargs) -> datasets.Dataset:
           pass
   ```

   - （可选）如果 OpenCompass 已有的评测器不能满足需要，需要用户定义 `MyDatasetlEvaluator` 类，实现评分方法 `score`，需要根据输入的 `predictions` 和 `references` 列表，得到需要的字典。由于一个数据集可能存在多种 metric，需要返回一个 metrics 以及对应 scores 的相关字典。具体示例如下：

   ```python
   from opencompass.openicl.icl_evaluator import BaseEvaluator

   class MyDatasetlEvaluator(BaseEvaluator):

       def score(self, predictions: List, references: List) -> dict:
           pass

   ```

   - （可选）如果 OpenCompass 已有的后处理方法不能满足需要，需要用户定义 `mydataset_postprocess` 方法，根据输入的字符串得到相应后处理的结果。具体示例如下：

   ```python

pass

   ```

   ```python
   def mydataset_postprocess(text: str) -> str:
       pass
   ```

2. 在定义好数据集加载、评测以及数据后处理等方法之后，需要在配置文件中新增以下配置：

   ```python
   from opencompass.datasets import MyDataset, MyDatasetlEvaluator, mydataset_postprocess

   mydataset_eval_cfg = dict(
       evaluator=dict(type=MyDatasetlEvaluator),
       pred_postprocessor=dict(type=mydataset_postprocess))

   mydataset_datasets = [
       dict(
           type=MyDataset,
           ...,
           reader_cfg=...,
           infer_cfg=...,
           eval_cfg=mydataset_eval_cfg)
   ]
   ```

   配置好数据集之后，其他需要的配置文件直接参考[快速上手](../get_started.md)教程即可。

# Waymo 数据集

本文档页包含了关于 MMDetection3D 中 Waymo 数据集用法的教程。

## 数据集准备

在准备 Waymo 数据集之前，如果您之前只安装了 `requirements/build.txt` 和 `requirements/runtime.txt` 中的依赖，请通过运行如下指令额外安装 Waymo 数据集所依赖的官方包：

```
# tf 2.1.0.
# tf 2.0.0
# tf 1.15.0
```

或者

```
```

和准备数据集的通用方法一致，我们推荐将数据集根目录软链接至 `$MMDETECTION3D/data`。
由于原始 Waymo 数据的格式基于 `tfrecord`，我们需要将原始数据进行预处理，以便于训练和测试时使用。我们的方法是将它们转换为 KITTI 格式。

处理之前，文件目录结构组织如下：

```
├── tools
├── data
│   ├── waymo

```

│   ├── waymo

```

您可以在[这里](https://waymo.com/open/download/)下载 1.2 版本的 Waymo 公开数据集，并在[这里](https://drive.google.com/drive/folders/18BVuF_RYJF0NjZpt8SnfzANiakoRMf0o?usp=sharing)下载其训练/验证/测试集拆分文件。接下来，请将 `tfrecord` 文件放入 `data/waymo/waymo_format/` 下的对应文件夹，并将 txt 格式的数据集拆分文件放入 `data/waymo/kitti_format/ImageSets`。在[这里](https://console.cloud.google.com/storage/browser/waymo_open_dataset_v_1_2_0/validation/ground_truth_objects)下载验证集使用的 bin 格式真实标注 (Ground Truth) 文件并放入 `data/waymo/waymo_format/`。小窍门：您可以使用 `gsutil` 来在命令行下载大规模数据集。您可以将该[工具](https://github.com/RalphMao/Waymo-Dataset-Tool) 作为一个例子来查看更多细节。之后，通过运行如下指令准备 Waymo 数据：

```bash
```

```bash
```

请注意，如果您的本地磁盘没有足够空间保存转换后的数据，您可以将 `--out-dir` 改为其他目录；只要在创建文件夹、准备数据并转换格式后，将数据文件链接到 `data/waymo/kitti_format` 即可。

在数据转换后，文件目录结构应组织如下：

```
├── tools
├── data
│   ├── waymo

```

因为 Waymo 数据的来源包含数个相机，这里我们将每个相机对应的图像和标签文件分别存储，并将相机位姿 (pose) 文件存储下来以供后续处理连续多帧的点云。我们使用 `{a}{bbb}{ccc}` 的名称编码方式为每帧数据命名，其中 `a` 是不同数据拆分的前缀（`0` 指代训练集，`1` 指代验证集，`2` 指代测试集），`bbb` 是分割部分 (segment) 的索引，而 `ccc` 是帧索引。您可以轻而易举地按照如上命名规则定位到所需的帧。我们将训练和验证所需数据按 KITTI 的方式集合在一起，然后将训练集/验证集/测试集的索引存储在 `ImageSet` 下的文件中。

## 训练

考虑到原始数据集中的数据有很多相似的帧，我们基本上可以主要使用一个子集来训练我们的模型。在我们初步的基线中，我们在每五帧图片中加载一帧。得益于我们的超参数设置和数据增强方案，我们得到了比 Waymo [原论文](https://arxiv.org/pdf/1912.04838.pdf)中更好的性能。请移步 `configs/pointpillars/` 下的 README.md 以查看更多配置和性能相关的细节。我们会尽快发布一个更完整的 Waymo 基准榜单 (benchmark)。

## 评估

## 评估

为了在 Waymo 数据集上进行检测性能评估，请按照[此处指示](https://github.com/waymo-research/waymo-open-dataset/blob/master/docs/quick_start.md)构建用于计算评估指标的二进制文件 `compute_detection_metrics_main`，并将它置于 `mmdet3d/core/evaluation/waymo_utils/` 下。您基本上可以按照下方命令安装 `bazel`，然后构建二进制文件：

```shell

```

接下来，您就可以在 Waymo 上评估您的模型了。如下示例是使用 8 个图形处理器 (GPU) 在 Waymo 上用 Waymo 评价指标评估 PointPillars 模型的情景：

```shell
```

如果需要生成 bin 文件，需要在配置文件的 `test_evaluator` 中指定 `pklfile_prefix`，因此你可以在命令后添加 `--cfg-options "test_evaluator.pklfile_prefix=xxxx"`。

**注意**:

1. 有时用 `bazel` 构建 `compute_detection_metrics_main` 的过程中会出现如下错误：`'round' 不是 'std' 的成员` (`'round' is not a member of 'std'`)。我们只需要移除该文件中，`round` 前的 `std::`。

**注意**:

2. 考虑到 Waymo 上评估一次耗时不短，我们建议只在模型训练结束时进行评估。

3. 为了在 CUDA 9 环境使用 TensorFlow，我们建议通过编译 TensorFlow 源码的方式使用。除了官方教程之外，您还可以参考该[链接](https://github.com/SmileTM/Tensorflow2.X-GPU-CUDA9.0)以寻找可能合适的预编译包以及编译源码的实用攻略。

## 测试并提交到官方服务器

如下是一个使用 8 个图形处理器在 Waymo 上测试 PointPillars，生成 bin 文件并提交结果到官方榜单的例子：

如果你想生成 bin 文件并提交到服务器中，在运行测试指令前你需要在配置文件的 `test_evaluator` 中指定 `submission_prefix`。

在生成 bin 文件后，您可以简单地构建二进制文件 `create_submission`，并按照[指示](https://github.com/waymo-research/waymo-open-dataset/blob/master/docs/quick_start.md/)创建一个提交文件。下面是一些示例：

```shell

```

如果想用官方评估服务器评估您在验证集上的结果，您可以使用同样的方法生成提交文件，只需确保您在运行如上指令前更改 `submission.txtpb` 中的字段值即可。

# How to do regression test

This tutorial describes how to do regression test. The deployment configuration file contains  codebase config and inference config.

### 1. Python Environment

```shell
```

If pip throw an exception, try to upgrade numpy.

```shell
```

## 2. Usage

```shell
```

- `--codebase` : The codebase to test, eg.`mmdet`. If you want to test multiple codebase, use `mmpretrain mmdet ...`

[--checkpoint-dir "$CHECKPOINT_DIR"]
```

- `--backends` : The backend to test. By default, all `backend`s would be tested. You can use `onnxruntime tesensorrt`to choose several backends. If you also need to test the SDK, you need to configure the `sdk_config` in `tests/regression/${codebase}.yml`.
- `--models` : Specify the model to be tested. All models in `yml` are tested by default. You can also give some model names. For the model name, please refer to the relevant yml configuration file. For example `ResNet SE-ResNet "Mask R-CNN"`. Model name can only contain numbers and letters.
- `--work-dir` : The directory of model convert and report, use `../mmdeploy_regression_working_dir` by default.
- `--checkpoint-dir`: The path of downloaded torch model, use `../mmdeploy_checkpoints` by default.
- `--device` : device type, use `cuda` by default

- `--log-level` : These options are available:`'CRITICAL', 'FATAL', 'ERROR', 'WARN', 'WARNING', 'INFO', 'DEBUG',  'NOTSET'`. The default value is `INFO`.
- `-p` or `--performance` : Test precision or not. If not enabled, only model convert would be tested.

### Notes

For Windows user:

1. To use the `&&` connector in shell commands, you need to download `PowerShell 7 Preview 5+`.
2. If you are using conda env, you may need to change `python3` to `python` in regression_test.py because there is `python3.exe` in `%USERPROFILE%\AppData\Local\Microsoft\WindowsApps` directory.

## Example

1. Test all backends of mmdet and mmpose for **model convert and precision**

```shell

```shell
```

2. Test **model convert and precision** of some backends of mmdet and mmpose

```shell
    -p
```

3. Test some backends of mmdet and mmpose, **only test model convert**

```shell
```

4. Test some models of mmdet and mmpretrain, **only test model convert**

```shell

```

```shell
```

## 3. Regression Test Tonfiguration

### Example and parameter description

```yaml
globals:
  codebase_dir: ../mmocr # codebase path to test
  checkpoint_force_download: False # whether to redownload the model even if it already exists
  images:
  metric_info: &metric_info
    hmean-iou: # metafile.Results.Metrics
      eval_name: hmean-iou #  test.py --metrics args

metric_info: &metric_info
      metric_key: 0_hmean-iou:hmean # the key name of eval log
      tolerance: 0.1 # tolerated threshold interval
      task_name: Text Detection # the name of metafile.Results.Task
      dataset: ICDAR2015 # the name of metafile.Results.Dataset
    word_acc: # same as hmean-iou, also a kind of metric
  convert_image_det: &convert_image_det # the image that will be used by detection model convert
  backend_test: &default_backend_test True # whether test model precision for backend
  sdk: # SDK config

input_img: *img_demo_text_recog

    convert_image: *convert_image_rec # the image used by model conversion
    backend_test: *default_backend_test # whether inference on the backend
    sdk_config: *sdk_recognition_dynamic # test SDK or not. If it exists, use a specific SDK config for testing
    deploy_config: configs/mmocr/text-recognition/text-recognition_onnxruntime_static.py # the deploy cfg path to use, based on mmdeploy path

pipeline_ort_recognition_dynamic_fp32: &pipeline_ort_recognition_dynamic_fp32

tensorrt:

pipeline_trt_detection_dynamic_fp16: &pipeline_trt_detection_dynamic_fp16

openvino:
  # same as onnxruntime backend configuration
ncnn:
pplnn:

models:
  - name: crnn # model name
    metafile: configs/textrecog/crnn/metafile.yml # the path of model metafile, based on codebase path
    codebase_model_config_dir: configs/textrecog/crnn # the basepath of `model_configs`, based on codebase path
    model_configs: # the config name to teset
    pipelines: # pipeline name

- crnn_academic_dataset.py

      # special pipeline can be added like this
```

## 4. Generated Report

This is an example of mmocr regression test report.

backend_test: xxx
```

## 5. Supported Backends

- [x] PPLNN
- [x] ncnn
- [x] SNPE

- [x] PPLNN
- [x] ncnn
- [x] SNPE

## 6. Supported Codebase and Metrics

|            | PQ       | :x:                |

# Contributing to OpenCompass

- [Contributing to OpenCompass](#contributing-to-opencompass)
  - [What is PR](#what-is-pr)
  - [Basic Workflow](#basic-workflow)
  - [Procedures in detail](#procedures-in-detail)
    - [1. Get the most recent codebase](#1-get-the-most-recent-codebase)
    - [2. Checkout a new branch from `main` branch](#2-checkout-a-new-branch-from-main-branch)
    - [3. Commit your changes](#3-commit-your-changes)
    - [4. Push your changes to the forked repository and create a PR](#4-push-your-changes-to-the-forked-repository-and-create-a-pr)
    - [5. Discuss and review your code](#5-discuss-and-review-your-code)
    - [6.  Merge your branch to `main` branch and delete the branch](#6--merge-your-branch-to-main-branch-and-delete-the-branch)
  - [Code style](#code-style)
    - [Python](#python)

Thanks for your interest in contributing to OpenCompass! All kinds of contributions are welcome, including but not limited to the following.

- Fix typo or bugs

- [Python](#python)

- Add documentation or translate the documentation into other languages
- Add new features and components

```
```

2. Checkout a new branch from `main` branch.

```

6. Merge your branch to `main` branch

  Fork the OpenCompass repository: click the **fork** button at the top right corner of Github page
  ![avatar](https://github.com/InternLM/opencompass/assets/22607038/851ed33d-02db-49c9-bf94-7c62eee89eb2)

  ```bash
  git clone git@github.com:XXX/opencompass.git
  ```

  ```bash
  git remote add upstream git@github.com:InternLM/opencompass.git
  ```

  Checkout the latest branch of the local repository and pull the latest branch of the source repository.

  ```bash
  git checkout main

```

  ```bash
  git pull upstream main
  ```

### 2. Checkout a new branch from `main` branch

```bash
git checkout main -b branchname
```

  ```bash
  ```

  ```bash
  # coding
  ```

  ```{note}
  ```

  ```bash

```

  ```bash
  ```

  ![avatar](https://github.com/InternLM/opencompass/assets/22607038/08feb221-b145-4ea8-8e20-05f143081604)

### 6. Merge your branch to `main` branch and delete the branch

  ```bash
  ```

## Code style

### Python

Style configurations of yapf and isort can be found in [setup.cfg](https://github.com/open-mmlab/OpenCompass/blob/main/setup.cfg).

# 20 分钟了解 MMPose 架构设计

MMPose 1.0 与之前的版本有较大改动，对部分模块进行了重新设计和组织，降低代码冗余度，提升运行效率，降低学习难度。

MMPose 1.0 采用了全新的模块结构设计以精简代码，提升运行效率，降低学习难度。对于有一定深度学习基础的用户，本章节提供了对 MMPose 架构设计的总体介绍。不论你是**旧版 MMPose 的用户**，还是**希望直接从 MMPose 1.0 上手的新用户**，都可以通过本教程了解如何构建一个基于 MMPose 1.0 的项目。

```{note}
本教程包含了使用 MMPose 1.0 时开发者会关心的内容：

- 整体代码架构与设计逻辑

- 如何用config文件管理模块

- 如何使用自定义数据集

- 如何添加新的模块（骨干网络、模型头部、损失函数等）
```

以下是这篇教程的目录：

- [20 分钟了解 MMPose 架构设计](#20-分钟了解-mmpose-架构设计)
  - [总览](#总览)
  - [Step1：配置文件](#step1配置文件)
  - [Step2：数据](#step2数据)
    - [数据集元信息](#数据集元信息)
    - [数据集](#数据集)
    - [数据流水线](#数据流水线)
      - [i. 数据增强](#i-数据增强)
      - [ii. 数据变换](#ii-数据变换)
      - [iii. 数据编码](#iii-数据编码)
      - [iv. 数据打包](#iv-数据打包)
  - [Step3: 模型](#step3-模型)
    - [前处理器（DataPreprocessor）](#前处理器datapreprocessor)
    - [主干网络（Backbone）](#主干网络backbone)
    - [颈部模块（Neck）](#颈部模块neck)
    - [预测头（Head）](#预测头head)

## 总览

![overall-cn](https://user-images.githubusercontent.com/13503330/187830967-f2d7bf40-6261-42f3-91a5-ae045fa0dc0c.png)

- [颈部模块（Neck）](#颈部模块neck)

## 总览

一般来说，开发者在项目开发过程中经常接触内容的主要有**五个**方面：

- **通用**：环境、钩子（Hook）、模型权重存取（Checkpoint）、日志（Logger）等

- **数据**：数据集、数据读取（Dataloader）、数据增强等

- **训练**：优化器、学习率调整等

- **模型**：主干网络、颈部模块（Neck）、预测头模块（Head）、损失函数等

- **评测**：评测指标（Metric）、评测器（Evaluator）等

其中**通用**、**训练**和**评测**相关的模块往往由训练框架提供，开发者只需要调用和调整参数，不需要自行实现，开发者主要实现的是**数据**和**模型**部分。

## Step1：配置文件

在MMPose中，我们通常 python 格式的配置文件，用于整个项目的定义、参数管理，因此我们强烈建议第一次接触 MMPose 的开发者，查阅 [配置文件](./user_guides/configs.md) 学习配置文件的定义。

## Step2：数据

MMPose 数据的组织主要包含三个方面：

- 数据集元信息

- 数据集

- 数据流水线

### 数据集元信息

元信息指具体标注之外的数据集信息。姿态估计数据集的元信息通常包括：关键点和骨骼连接的定义、对称性、关键点性质（如关键点权重、标注标准差、所属上下半身）等。这些信息在数据在数据处理、模型训练和测试中有重要作用。在 MMPose 中，数据集的元信息使用 python 格式的配置文件保存，位于 `$MMPOSE/configs/_base_/datasets` 目录下。

### 数据集元信息

在 MMPose 中使用自定义数据集时，你需要增加对应的元信息配置文件。以 MPII 数据集（`$MMPOSE/configs/_base_/datasets/mpii.py`）为例：

```Python
    ),
        0:
        dict(
            id=0,
        ## 内容省略
    },
        0:

name='right_ankle',
            id=0,
        ## 内容省略
    },
        0:
        ## 内容省略
    },
    ],
    # 使用 COCO 数据集中提供的 sigmas 值
    sigmas=[
    ])
```

在模型配置文件中，你需要为自定义数据集指定对应的元信息配置文件。假如该元信息配置文件路径为 `$MMPOSE/configs/_base_/datasets/custom.py`，指定方式如下：

```python

        # 指定对应的元信息配置文件
        ...),
    )
        ...),
    )
```

### 数据集

在 MMPose 中使用自定义数据集时，我们推荐将数据转化为已支持的格式（如 COCO 或 MPII），并直接使用我们提供的对应数据集实现。如果这种方式不可行，则用户需要实现自己的数据集类。

...),
    )
```

### 数据集

MMPose 中的大部分 2D 关键点数据集**以 COCO 形式组织**，为此我们提供了基类 [BaseCocoStyleDataset](/mmpose/datasets/datasets/base/base_coco_style_dataset.py)。我们推荐用户继承该基类，并按需重写它的方法（通常是 `__init__()` 和 `_load_annotations()` 方法），以扩展到新的 2D 关键点数据集。

```{note}
关于COCO数据格式的详细说明请参考 [COCO](./dataset_zoo/2d_body_keypoint.md) 。
```

```{note}
在 MMPose 中 bbox 的数据格式采用 `xyxy`，而不是 `xywh`，这与 [MMDetection](https://github.com/open-mmlab/mmdetection) 等其他 OpenMMLab 成员保持一致。为了实现不同 bbox 格式之间的转换，我们提供了丰富的函数：`bbox_xyxy2xywh`、`bbox_xywh2xyxy`、`bbox_xyxy2cs`等。这些函数定义在`$MMPOSE/mmpose/structures/bbox/transforms.py`。
```

下面我们以MPII数据集的实现（`$MMPOSE/mmpose/datasets/datasets/body/mpii_dataset.py`）为例：

```Python

```Python

                 ## 内容省略
                 ## 内容省略):

headbox_file_type = headbox_file[-3:]

            ## 内容省略
            )

                                         [2, 0, 1])

headboxes_src = np.transpose(headbox_dict['headboxes_src'],
                                         [2, 0, 1])

        ann_id = 0

# unify shape with coco datasets

            }

'keypoints_visible': keypoints_visible,
            }

```

在对MPII数据集进行支持时，由于MPII需要读入 `head_size` 信息来计算 `PCKh`，因此我们在`__init__()`中增加了 `headbox_file`，并重载了 `_load_annotations()` 来完成数据组织。

如果自定义数据集无法被 `BaseCocoStyleDataset` 支持，你需要直接继承 [MMEngine](https://github.com/open-mmlab/mmengine) 中提供的 `BaseDataset` 基类。具体方法请参考相关[文档](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/basedataset.html)。

### 数据流水线

一个典型的数据流水线配置如下：

```Python

### 数据流水线

```Python
]
]
```

在关键点检测任务中，数据一般会在三个尺度空间中变换：

- **原始图片空间**：图片存储时的原始空间，不同图片的尺寸不一定相同

- **输入图片空间**：模型输入的图片尺度空间，所有**图片**和**标注**被缩放到输入尺度，如 `256x256`，`256x192` 等

- **输出尺度空间**：模型输出和训练监督信息所在的尺度空间，如`64x64(热力图)`，`1x1(回归坐标值)`等

数据在三个空间中变换的流程如图所示：

![migration-cn](https://user-images.githubusercontent.com/13503330/187831574-13804daf-f498-47c2-ba43-64b8e6ffe3dd.png)

在MMPose中，数据变换所需要的模块在`$MMPOSE/mmpose/datasets/transforms`目录下，它们的工作流程如图所示：

![transforms-cn](https://user-images.githubusercontent.com/13503330/187831611-8db89e20-95c7-42bc-8b0d-700fadf60328.png)

#### i. 数据增强

数据增强中常用的变换存放在 `$MMPOSE/mmpose/datasets/transforms/common_transforms.py` 中，如 `RandomFlip`、`RandomHalfBody` 等。

对于 top-down 方法，`Shift`、`Rotate`、`Resize` 操作由 `RandomBBoxTransform`来实现；对于 bottom-up 方法，这些则是由 `BottomupRandomAffine` 实现。

```{note}
值得注意的是，大部分数据变换都依赖于 `bbox_center` 和 `bbox_scale`，它们可以通过 `GetBBoxCenterScale` 来得到。
```

#### ii. 数据变换

我们使用仿射变换，将图像和坐标标注从原始图片空间变换到输入图片空间。这一操作在 top-down 方法中由 `TopdownAffine` 完成，在 bottom-up 方法中则由 `BottomupRandomAffine` 完成。

#### iii. 数据编码

```

在模型训练时，数据从原始空间变换到输入图片空间后，需要使用 `GenerateTarget` 来生成训练所需的监督目标（比如用坐标值生成高斯热图），我们将这一过程称为编码（Encode），反之，通过高斯热图得到对应坐标值的过程称为解码（Decode）。

在 MMPose 中，我们将编码和解码过程集合成一个编解码器（Codec），在其中实现 `encode()` 和 `decode()`。

目前 MMPose 支持生成以下类型的监督目标：

- `heatmap`: 高斯热图

- `keypoint_label`: 关键点标签（如归一化的坐标值）

- `keypoint_xy_label`: 单个坐标轴关键点标签

- `heatmap+keypoint_label`: 同时生成高斯热图和关键点标签

- `multiscale_heatmap`: 多尺度高斯热图

生成的监督目标会按以下关键字进行封装：

- `heatmaps`：高斯热图

- `keypoint_labels`：关键点标签（如归一化的坐标值）

- `keypoint_x_labels`：x 轴关键点标签

- `keypoint_y_labels`：y 轴关键点标签

- `keypoint_weights`：关键点权重

```Python

    Added Keys (depends on the args):
        - heatmaps
        - keypoint_labels
        - keypoint_x_labels
        - keypoint_y_labels
        - keypoint_weights

Added Keys (depends on the args):
        - heatmaps
    """
```

值得注意的是，我们对 top-down 和 bottom-up 的数据格式进行了统一，这意味着标注信息中会新增一个维度来代表同一张图里的不同目标（如人），格式为：

```Python
```

- top-down：`[B, 1, K, D]`

当前已经支持的编解码器定义在 `$MMPOSE/mmpose/codecs` 目录下，如果你需要自定新的编解码器，可以前往[编解码器](./user_guides/codecs.md)了解更多详情。

#### iv. 数据打包

数据经过前处理变换后，最终需要通过 `PackPoseInputs` 打包成数据样本。该操作定义在 `$MMPOSE/mmpose/datasets/transforms/formatting.py` 中。

打包过程会将数据流水线中用字典 `results` 存储的数据转换成用 MMPose 所需的标准数据结构， 如 `InstanceData`，`PixelData`，`PoseDataSample` 等。

具体而言，我们将数据样本内容分为 `gt`（标注真值） 和 `pred`（模型预测）两部分，它们都包含以下数据项：

- **instances**(numpy.array)：实例级别的原始标注或预测结果，属于原始尺度空间

- **instance_labels**(torch.tensor)：实例级别的训练标签（如归一化的坐标值、关键点可见性），属于输出尺度空间

- **fields**(torch.tensor)：像素级别的训练标签（如高斯热图）或预测结果，属于输出尺度空间

下面是 `PoseDataSample` 底层实现的例子：

```Python
    # meta
        img_shape=(600, 900),  # [h, w, c]
        crop_size=(256, 192),  # [h, w]
    )

pred_instances.keypoint_scores = np.random.rand(1, 17)

```

## Step3: 模型

在 MMPose 1.0中，模型由以下几部分构成：

- **预处理器（DataPreprocessor）**：完成图像归一化和通道转换等前处理

- **主干网络 （Backbone）**：用于特征提取

- **颈部模块（Neck）**：GAP，FPN 等可选项

- **预测头（Head）**：用于实现核心算法功能和损失函数定义

我们在 `$MMPOSE/models/pose_estimators/base.py` 下为姿态估计模型定义了一个基类 `BasePoseEstimator`，所有的模型（如 `TopdownPoseEstimator`）都需要继承这个基类，并重载对应的方法。

在模型的 `forward()` 方法中提供了三种不同的模式：

- `mode == 'loss'`：返回损失函数计算的结果，用于模型训练

- `mode == 'predict'`：返回输入尺度下的预测结果，用于模型推理

- `mode == 'tensor'`：返回输出尺度下的模型输出，即只进行模型前向传播，用于模型导出

开发者需要在 `PoseEstimator` 中按照模型结构调用对应的 `Registry` ，对模块进行实例化。以 top-down 模型为例：

```Python

```

### 前处理器（DataPreprocessor）

```

从 MMPose 1.0 开始，我们在模型中添加了新的前处理器模块，用以完成图像归一化、通道顺序变换等操作。这样做的好处是可以利用 GPU 等设备的计算能力加快计算，并使模型在导出和部署时更具完整性。

在配置文件中，一个常见的 `data_preprocessor` 如下：

```Python
```

它会将输入图片的通道顺序从 `bgr` 转换为 `rgb`，并根据 `mean` 和 `std` 进行数据归一化。

### 主干网络（Backbone）

MMPose 实现的主干网络存放在 `$MMPOSE/mmpose/models/backbones` 目录下。

在实际开发中，开发者经常会使用预训练的网络权重进行迁移学习，这能有效提升模型在小数据集上的性能。 在 MMPose 中，只需要在配置文件 `backbone` 的 `init_cfg` 中设置：

```Python
```

如果你想只加载一个训练好的 checkpoint 的 backbone 部分，你需要指明一下前缀 `prefix`:

```Python

checkpoint='PATH/TO/YOUR_MODEL_WEIGHTS.pth'),
```

```Python
```

其中 `checkpoint` 既可以是本地路径，也可以是下载链接。因此，如果你想使用 Torchvision 提供的预训练模型（比如ResNet50），可以使用：

```Python
```

除了这些常用的主干网络以外，你还可以从 MMClassification 等其他 OpenMMLab 项目中方便地迁移主干网络，它们都遵循同一套配置文件格式，并提供了预训练权重可供使用。

需要强调的是，如果你加入了新的主干网络，需要在模型定义时进行注册：

```Python
```

同时在 `$MMPOSE/mmpose/models/backbones/__init__.py` 下进行 `import`，并加入到 `__all__` 中，才能被配置文件正确地调用。

### 颈部模块（Neck）

颈部模块通常是介于主干网络和预测头之间的模块，在部分模型算法中会用到，常见的颈部模块有：

  `FeatureMapProcessor` 是一个通用的 PyTorch 模块，旨在通过选择、拼接和缩放等非参数变换将主干网络输出的特征图转换成适合预测头的格式。以下是一些操作的配置方式及效果示意图:

  - 选择操作

  - 选择操作

    ```python
    ```

  - 拼接操作

    ```python
    ```

    拼接之前，其它特征图会被缩放到和序号为 0 的特征图相同的尺寸。

  - 缩放操作

    ```python
    ```

### 预测头（Head）

通常来说，预测头是模型算法实现的核心，用于控制模型的输出，并进行损失函数计算。

<img src="https://user-images.githubusercontent.com/26127467/227109402-94106e4b-b941-4ce9-8201-c64920d82ed1.png" height="120px" alt><br>

MMPose 中 Head 相关的模块定义在 `$MMPOSE/mmpose/models/heads` 目录下，开发者在自定义预测头时需要继承我们提供的基类 `BaseHead`，并重载以下三个方法对应模型推理的三种模式：

- loss()

具体而言，`predict()` 返回的应是输入图片尺度下的结果，因此需要调用 `self.decode()` 对网络输出进行解码，这一过程实现在 `BaseHead` 中已经实现，它会调用编解码器提供的 `decode()` 方法来完成解码。

另一方面，我们会在 `predict()` 中进行测试时增强。在进行预测时，一个常见的测试时增强技巧是进行翻转集成。即，将一张图片先进行一次推理，再将图片水平翻转进行一次推理，推理的结果再次水平翻转回去，对两次推理的结果进行平均。这个技巧能有效提升模型的预测稳定性。

下面是在 `RegressionHead` 中定义 `predict()` 的例子：

```Python

test_cfg: ConfigType = {}) -> Predictions:

    else:

```

`loss()`除了进行损失函数的计算，还会进行 accuracy 等训练时指标的计算，并通过一个字典 `losses` 来传递:

```Python

batch_coords.unsqueeze_(dim=1)  # (B, N, K, D)
```

```Python
    thr=0.05,

```

每个 batch 的数据都打包成了 `batch_data_samples`。以 Regression-based 方法为例，训练所需的归一化的坐标值和关键点权重可以用如下方式获取：

```Python
])
```

以下为 `RegressionHead` 中完整的 `loss()` 实现：

```Python

])
```

```Python

    ])

    else:

# calculate accuracy
        thr=0.05,

```

# HTC

> [Hybrid Task Cascade for Instance Segmentation](https://arxiv.org/abs/1901.07518)

Cascade is a classic yet powerful architecture that has boosted performance on various tasks. However, how to introduce cascade to instance segmentation remains an open question. A simple combination of Cascade R-CNN and Mask R-CNN only brings limited gain. In exploring a more effective approach, we find that the key to a successful instance segmentation cascade is to fully leverage the reciprocal relationship between detection and segmentation. In this work, we propose a new framework, Hybrid Task Cascade (HTC), which differs in two important aspects: (1) instead of performing cascaded refinement on these two tasks separately, it interweaves them for a joint multi-stage processing; (2) it adopts a fully convolutional branch to provide spatial context, which can help distinguishing hard foreground from cluttered background. Overall, this framework can learn more discriminative features progressively while integrating complementary features together in each stage. Without bells and whistles, a single HTC obtains 38.4 and 1.5 improvement over a strong Cascade Mask R-CNN baseline on MSCOCO dataset. Moreover, our overall system achieves 48.6 mask AP on the test-challenge split, ranking 1st in the COCO 2018 Challenge Object Detection Task.

<img src="https://user-images.githubusercontent.com/40661020/143893906-e05acaa6-d46f-4c11-84e8-bb9940a95b44.png"/>
</div>

HTC requires COCO and [COCO-stuff](http://calvin.inf.ed.ac.uk/wp-content/uploads/data/cocostuffdataset/stuffthingmaps_trainval2017.zip) dataset for training. You need to download and extract it in the COCO dataset path.

```none
├── mmdet
├── tools
├── data
│   ├── coco
|   |   ├── stuffthingmaps
```

```

|    Backbone     |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                      Config                      |                                                                                                                                                   Download                                                                                                                                                    |

| :-------------: | :-----: | :-----: | :------: | :------------: | :----: | :-----: | :----------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|    R-50-FPN     | pytorch |   1x    |   8.2    |      5.8       |  42.3  |  37.4   |        [config](./htc_r50_fpn_1x_coco.py)        |                           [model](https://download.openmmlab.com/mmdetection/v2.0/htc/htc_r50_fpn_1x_coco/htc_r50_fpn_1x_coco_20200317-7332cf16.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/htc/htc_r50_fpn_1x_coco/htc_r50_fpn_1x_coco_20200317_070435.log.json)                           |

|    R-50-FPN     | pytorch |   20e   |   8.2    |       -        |  43.3  |  38.3   |       [config](./htc_r50_fpn_20e_coco.py)        |                         [model](https://download.openmmlab.com/mmdetection/v2.0/htc/htc_r50_fpn_20e_coco/htc_r50_fpn_20e_coco_20200319-fe28c577.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/htc/htc_r50_fpn_20e_coco/htc_r50_fpn_20e_coco_20200319_070313.log.json)                         |
|    R-101-FPN    | pytorch |   20e   |   10.2   |      5.5       |  44.8  |  39.6   |       [config](./htc_r101_fpn_20e_coco.py)       |                       [model](https://download.openmmlab.com/mmdetection/v2.0/htc/htc_r101_fpn_20e_coco/htc_r101_fpn_20e_coco_20200317-9b41b48f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/htc/htc_r101_fpn_20e_coco/htc_r101_fpn_20e_coco_20200317_153107.log.json)                       |

| X-101-32x4d-FPN | pytorch |   20e   |   11.4   |      5.0       |  46.1  |  40.5   | [config](./htc_x101-32x4d_fpn_16xb1-20e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/htc/htc_x101_32x4d_fpn_16x1_20e_coco/htc_x101_32x4d_fpn_16x1_20e_coco_20200318-de97ae01.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/htc/htc_x101_32x4d_fpn_16x1_20e_coco/htc_x101_32x4d_fpn_16x1_20e_coco_20200318_034519.log.json) |
| X-101-64x4d-FPN | pytorch |   20e   |   14.5   |      4.4       |  47.0  |  41.4   | [config](./htc_x101-64x4d_fpn_16xb1-20e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/htc/htc_x101_64x4d_fpn_16x1_20e_coco/htc_x101_64x4d_fpn_16x1_20e_coco_20200318-b181fd7a.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/htc/htc_x101_64x4d_fpn_16x1_20e_coco/htc_x101_64x4d_fpn_16x1_20e_coco_20200318_081711.log.json) |

- In the HTC paper and COCO 2018 Challenge, `score_thr` is set to 0.001 for both baselines and HTC.

- We use 8 GPUs with 2 images/GPU for R-50 and R-101 models, and 16 GPUs with 1 image/GPU for X-101 models.
  If you would like to train X-101 HTC with 8 GPUs, you need to change the lr from 0.02 to 0.01.

We also provide a powerful HTC with DCN and multi-scale training model. No testing augmentation is used.

|    Backbone     |  Style  |  DCN  | training scales | Lr schd | box AP | mask AP |                                  Config                                  |                                                                                                                                                                                                             Download                                                                                                                                                                                                              |

| X-101-64x4d-FPN | pytorch | c3-c5 |    400~1400     |   20e   |  50.4  |  43.8   | [config](./htc_x101-64x4d-dconv-c3-c5_fpn_ms-400-1400-16xb1-20e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/htc/htc_x101_64x4d_fpn_dconv_c3-c5_mstrain_400_1400_16x1_20e_coco/htc_x101_64x4d_fpn_dconv_c3-c5_mstrain_400_1400_16x1_20e_coco_20200312-946fd751.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/htc/htc_x101_64x4d_fpn_dconv_c3-c5_mstrain_400_1400_16x1_20e_coco/htc_x101_64x4d_fpn_dconv_c3-c5_mstrain_400_1400_16x1_20e_coco_20200312_203410.log.json) |

We provide config files to reproduce the results in the CVPR 2019 paper for [Hybrid Task Cascade](https://arxiv.org/abs/1901.07518).

```latex
@inproceedings{chen2019hybrid,
  title={Hybrid task cascade for instance segmentation},

```latex
  author={Chen, Kai and Pang, Jiangmiao and Wang, Jiaqi and Xiong, Yu and Li, Xiaoxiao and Sun, Shuyang and Feng, Wansen and Liu, Ziwei and Shi, Jianping and Ouyang, Wanli and Chen Change Loy and Dahua Lin},
}
```

    <sup>
      </a>
    </sup>
    <sup>
      </a>
    </sup>
  </div>

[![docs](https://img.shields.io/badge/docs-latest-blue)](https://mmrotate.readthedocs.io/en/latest/)

[📘使用文档](https://mmrotate.readthedocs.io/zh_CN/1.x/) |
[🛠️安装教程](https://mmrotate.readthedocs.io/zh_CN/1.x/get_started.html) |
[👀模型库](https://mmrotate.readthedocs.io/zh_CN/1.x/model_zoo.html) |
[🆕更新日志](https://mmrotate.readthedocs.io/en/1.x/notes/changelog.html) |
[🚀进行中的项目](https://github.com/open-mmlab/mmrotate/projects) |
[🤔报告问题](https://github.com/open-mmlab/mmrotate/issues/new/choose)

</div>

</div>

## 介绍

</div>

</div>

## 介绍

MMRotate 是一款基于 PyTorch 的旋转框检测的开源工具箱，是 [OpenMMLab](http://openmmlab.org/) 项目的成员之一。

主分支代码目前支持 **PyTorch 1.6 以上**的版本。

- **支持多种角度表示法**

  MMRotate 提供了三种主流的角度表示法以满足不同论文的配置。

  MMRotate 将旋转框检测任务解耦成不同的模块组件，通过组合不同的模块组件，用户可以便捷地构建自定义的旋转框检测算法模型。

- **强大的基准模型与SOTA**

  MMRotate 提供了旋转框检测任务中最先进的算法和强大的基准模型.

## 最新进展

### 亮点

我们很高兴向大家介绍我们在实时目标识别任务方面的最新成果 RTMDet，包含了一系列的全卷积单阶段检测模型。 RTMDet 不仅在从 tiny 到 extra-large 尺寸的目标检测模型上实现了最佳的参数量和精度的平衡，而且在实时实例分割和旋转目标检测任务上取得了最先进的成果。 更多细节请参阅[技术报告](https://arxiv.org/abs/2212.07784)。 预训练模型可以在[这里](configs/rotated_rtmdet)找到。

</div>

最新的 **v1.0.0rc1** 版本已经在 2022.12.30 发布:

- 支持了 [RTMDet](configs/rotated_rtmdet) 的旋转目标检测模型。RTMDet 的技术报告发布在了 [arxiv](https://arxiv.org/abs/2212.07784) 上。
- 支持了 [H2RBox](configs/h2rbox) 模型。H2RBox 的技术报告发布在了 [arxiv](https://arxiv.org/abs/2210.06742) 上。

## 安装

请参考[快速入门文档](https://mmrotate.readthedocs.io/zh_CN/1.x/get_started.html)进行安装。

## 教程

## 安装

## 教程

请阅读[概述](https://mmrotate.readthedocs.io/zh_CN/1.x/get_started.html)对 MMDetection 进行初步的了解。

为了帮助用户更进一步了解 MMDetection，我们准备了用户指南和进阶指南，请阅读我们的[文档](https://mmrotate.readthedocs.io/zh_CN/1.x/)：

- 用户指南
  - [训练 & 测试](https://mmrotate.readthedocs.io/zh_CN/1.x/user_guides/index.html#train-test)
    - [学习配置文件](https://mmrotate.readthedocs.io/zh_CN/1.x/user_guides/config.html)
    - [使用已有模型在标准数据集上进行推理](https://mmrotate.readthedocs.io/en/1.x/user_guides/inference.html)
    - [数据集准备](https://mmrotate.readthedocs.io/zh_CN/1.x/user_guides/dataset_prepare.html)
    - [测试现有模型](https://mmrotate.readthedocs.io/zh_CN/1.x/user_guides/train_test.html#test)
    - [在标准数据集上训练预定义的模型](https://mmrotate.readthedocs.io/zh_CN/1.x/user_guides/train_test.html#train)

- [测试现有模型](https://mmrotate.readthedocs.io/zh_CN/1.x/user_guides/train_test.html#test)
    - [提交测试结果](https://mmrotate.readthedocs.io/zh_CN/1.x/user_guides/test_results_submission.html)
  - [实用工具](https://mmrotate.readthedocs.io/zh_CN/1.x/user_guides/index.html#useful-tools)
- 进阶指南
  - [基础概念](https://mmrotate.readthedocs.io/zh_CN/1.x/advanced_guides/index.html#basic-concepts)
  - [组件定制](https://mmrotate.readthedocs.io/zh_CN/1.x/advanced_guides/index.html#component-customization)
  - [How to](https://mmrotate.readthedocs.io/zh_CN/1.x/advanced_guides/index.html#how-to)

我们提供了旋转检测的 colab 教程 [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](demo/MMRotate_Tutorial.ipynb)。

若需要将0.x版本的代码迁移至新版，请参考[迁移文档](https://mmrotate.readthedocs.io/zh_CN/1.x/migration.html)。

## 模型库

各个模型的结果和设置都可以在对应的 config（配置）目录下的 *README.md* 中查看。

## 模型库

整体的概况也可也在 [模型库](docs/zh_cn/model_zoo.md) 页面中查看。

## 数据准备

请参考 [data_preparation.md](tools/data/README.md) 进行数据集准备。

## 常见问题

请参考 [FAQ](docs/en/notes/faq.md) 了解其他用户的常见问题。

## 参与贡献

我们非常欢迎用户对于 MMRotate 做出的任何贡献，可以参考 [CONTRIBUTION.md](.github/CONTRIBUTING.md) 文件了解更多细节。

## 致谢

## 常见问题

## 参与贡献

## 致谢

MMRotate 是一款由不同学校和公司共同贡献的开源项目。我们感谢所有为项目提供算法复现和新功能支持的贡献者，以及提供宝贵反馈的用户。我们感谢 [SJTU 学生创新中心](https://www.si.sjtu.edu.cn/)在项目之初提供的大量计算资源。我们希望该工具箱和基准测试可以为社区提供灵活的代码工具，供用户复现现有算法并开发自己的新模型，从而不断为开源社区提供贡献。

## 引用

如果你在研究中使用了本项目的代码或者性能基准，请参考如下 bibtex 引用 MMRotate。

```bibtex
}
```

## 许可证

pages = {7331–7334},
}
```

## 许可证

- [MMClassification](https://github.com/open-mmlab/mmclassification): OpenMMLab 图像分类工具箱

扫描下方的二维码可关注 OpenMMLab 团队的 [知乎官方账号](https://www.zhihu.com/people/openmmlab)，加入 OpenMMLab 团队的 [官方交流 QQ 群](https://jq.qq.com/?_wv=1027&k=aCvMxdr3)或联络 OpenMMLab 官方微信小助手

<img src="https://raw.githubusercontent.com/open-mmlab/mmcv/master/docs/en/_static/zhihu_qrcode.jpg" height="400"><img src="https://raw.githubusercontent.com/open-mmlab/mmcv/master/docs/en/_static/wechat_qrcode.jpg" height="400"><img src="https://raw.githubusercontent.com/open-mmlab/mmcv/master/docs/en/_static/qq_group_qrcode.jpg" height="400">
</div>

# YOLOF

> [You Only Look One-level Feature](https://arxiv.org/abs/2103.09460)

This paper revisits feature pyramids networks (FPN) for one-stage detectors and points out that the success of FPN is due to its divide-and-conquer solution to the optimization problem in object detection rather than multi-scale feature fusion. From the perspective of optimization, we introduce an alternative way to address the problem instead of adopting the complex feature pyramids - {\\em utilizing only one-level feature for detection}. Based on the simple and efficient solution, we present You Only Look One-level Feature (YOLOF). In our method, two key components, Dilated Encoder and Uniform Matching, are proposed and bring considerable improvements. Extensive experiments on the COCO benchmark prove the effectiveness of the proposed model. Our YOLOF achieves comparable results with its feature pyramids counterpart RetinaNet while being 2.5× faster. Without transformer layers, YOLOF can match the performance of DETR in a single-level feature manner with 7× less training epochs. With an image size of 608×608, YOLOF achieves 44.3 mAP running at 60 fps on 2080Ti, which is 13% faster than YOLOv4.

<img src="https://user-images.githubusercontent.com/40661020/144001639-257374ef-7d4f-412b-a783-88abdd22f277.png"/>
</div>

| Backbone | Style | Epoch | Lr schd | Mem (GB) | box AP |                  Config                  |                                                                                                                                         Download                                                                                                                                         |
| :------: | :---: | :---: | :-----: | :------: | :----: | :--------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

| R-50-C5  | caffe |   Y   |   1x    |   8.3    |  37.5  | [config](./yolof_r50-c5_8xb8-1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/yolof/yolof_r50_c5_8x8_1x_coco/yolof_r50_c5_8x8_1x_coco_20210425_024427-8e864411.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/yolof/yolof_r50_c5_8x8_1x_coco/yolof_r50_c5_8x8_1x_coco_20210425_024427.log.json) |

**Note**:

1. We find that the performance is unstable and may fluctuate by about 0.3 mAP. mAP 37.4 ~ 37.7 is acceptable in YOLOF_R_50_C5_1x. Such fluctuation can also be found in the [original implementation](https://github.com/chensnathan/YOLOF).
2. In addition to instability issues, sometimes there are large loss fluctuations and NAN, so there may still be problems with this project, which will be improved subsequently.

```latex
@inproceedings{chen2021you,
  title={You Only Look One-level Feature},

```latex
  author={Chen, Qiang and Wang, Yingming and Yang, Tong and Zhang, Xiangyu and Cheng, Jian and Sun, Jian},
}
```

# 安装

1. 准备 OpenCompass 运行环境：

   ```bash
   conda create --name opencompass python=3.10 pytorch torchvision pytorch-cuda -c nvidia -c pytorch -y
   conda activate opencompass
   ```

   如果你希望自定义 PyTorch 版本或相关的 CUDA 版本，请参考 [官方文档](https://pytorch.org/get-started/locally/) 准备 PyTorch 环境。需要注意的是，OpenCompass 要求 `pytorch>=1.13`。

2. 安装 OpenCompass：

   ```bash
   git clone https://github.com/InternLM/opencompass.git
   cd opencompass
   ```

3. 安装 humaneval（可选）：

   如果你需要**在 humaneval 数据集上评估模型代码能力**，请执行此步骤，否则忽略这一步。

   <details>
   <summary><b>点击查看详细</b></summary>

   ```bash
   git clone https://github.com/openai/human-eval.git
   cd human-eval
   pip install -r requirements.txt
   cd ..
   ```

   请仔细阅读 `human_eval/execution.py` **第48-57行**的注释，了解执行模型生成的代码可能存在的风险，如果接受这些风险，请取消**第58行**的注释，启用代码执行评测。

   </details>

4. 安装 Llama（可选）：

   如果你需要**使用官方实现评测 Llama / Llama-2 / Llama-2-chat 模型**，请执行此步骤，否则忽略这一步。

   <details>

   <details>

   ```bash
   git clone https://github.com/facebookresearch/llama.git
   cd llama
   cd ..
   ```

   你可以在 `configs/models` 下找到所有 Llama / Llama-2 / Llama-2-chat 模型的配置文件示例。([示例](https://github.com/InternLM/opencompass/blob/eb4822a94d624a4e16db03adeb7a59bbd10c2012/configs/models/llama2_7b_chat.py))

# 数据集准备

OpenCompass 支持的数据集主要包括两个部分：

1. Huggingface 数据集： [Huggingface Dataset](https://huggingface.co/datasets) 提供了大量的数据集，这部分数据集运行时会**自动下载**。

2. 自建以及第三方数据集：OpenCompass 还提供了一些第三方数据集及自建**中文**数据集。运行以下命令**手动下载解压**。

在 OpenCompass 项目根目录下运行下面命令，将数据集准备至 `${OpenCompass}/data` 目录下：

```bash
```

```bash
```

OpenCompass 已经支持了大多数常用于性能比较的数据集，具体支持的数据集列表请直接在 `configs/datasets` 下进行查找。

# 快速上手

OpenCompass 的评测以配置文件为中心，必须包含 `datasets` 和 `models` 字段，配置需要评测的模型以及数据集，使用入口 'run.py' 启动。

我们会以测试 [OPT-125M](https://huggingface.co/facebook/opt-125m) 以及 [OPT-350M](https://huggingface.co/facebook/opt-350m) 预训练基座模型在 [SIQA](https://huggingface.co/datasets/social_i_qa) 和 [Winograd](https://huggingface.co/datasets/winogrande) 上的性能为例，带领你熟悉 OpenCompass 的一些基本功能。
本次的测试的配置文件为[configs/eval_demo.py](https://github.com/InternLM/opencompass/blob/main/configs/eval_demo.py)。

运行前确保已经安装了 OpenCompass，本实验可以在单张 _GTX-1660-6G_ 显卡上成功运行。
更大参数的模型，如 Llama-7B, 可参考 [configs](https://github.com/InternLM/opencompass/tree/main/configs) 中其他例子。

由于 OpenCompass 默认使用并行的方式进行评测，为了便于及时发现问题，我们可以在首次启动时使用 debug 模式运行，该模式会将任务串行执行，并会实时输出任务的执行进度。

```bash

```bash
python run.py configs/eval_demo.py -w outputs/demo --debug
```

如果一切正常，屏幕上会出现 "Starting inference process"：

```bash
Loading cached processed dataset at .cache/huggingface/datasets/social_i_qa/default/0.1.0/674d85e42ac7430d3dcd4de7007feaffcb1527c535121e09bab2803fbcc925f8/cache-742512eab30e8c9c.arrow
[2023-07-12 18:23:55,076] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
```

此时可以使用 `ctrl+c` 中断 debug 模式的执行，并运行以下命令进行并行评测：

```bash
python run.py configs/eval_demo.py -w outputs/demo
```

运行 demo 期间，我们来介绍一下本案例中的配置内容以及启动选项。

## 步骤详解

### 数据集列表 `datasets`

以下为 `configs/eval_demo.py` 中与数据集相关的配置片段：

```python
from mmengine.config import read_base  # 使用 mmengine.read_base() 读取基础配置

    # 直接从预设数据集配置中读取需要的数据集配置

```python

    from .datasets.winograd.winograd_ppl import winograd_datasets  # 读取 Winograd 的配置，基于 PPL (perplexity) 进行评测
    from .datasets.siqa.siqa_gen import siqa_datasets  # 读取 SIQA 的配置，基于生成式进行评测

datasets = [*siqa_datasets, *winograd_datasets]       # 最后 config 需要包含所需的评测数据集列表 datasets
```

[configs/datasets](https://github.com/InternLM/OpenCompass/blob/main/configs/datasets) 包含各种数据集预先定义好的配置文件；
部分数据集文件夹下有 'ppl' 和 'gen' 两类配置文件，表示使用的评估方式，其中 `ppl` 表示使用判别式评测， `gen` 表示使用生成式评测。

[configs/datasets/collections](https://github.com/InternLM/OpenCompass/blob/main/configs/datasets/collections) 存放了各类数据集集合，方便做综合评测。

更多介绍可查看 [数据集配置](./user_guides/dataset_prepare.md)。

### 模型列表 `models`

OpenCompass 支持直接在配置中指定待测试的模型列表，对于 HuggingFace 模型来说，用户通常无需添加代码。下面为相关的配置片段：

```python
# 提供直接使用 HuggingFaceCausalLM 模型的接口

```python
from opencompass.models import HuggingFaceCausalLM

# OPT-350M
opt350m = dict(
       type=HuggingFaceCausalLM,
       # 以下参数为 HuggingFaceCausalLM 相关的初始化参数
       path='facebook/opt-350m',
       tokenizer_path='facebook/opt-350m',
       tokenizer_kwargs=dict(
           padding_side='left',
           truncation_side='left',
           proxies=None,
           trust_remote_code=True),
       model_kwargs=dict(device_map='auto'),
       # 下列参数为所有模型均需设定的初始化参数，非 HuggingFaceCausalLM 独有
       abbr='opt350m',                    # 模型简称，用于结果展示
       max_seq_len=2048,              # 模型能接受的最大序列长度
       max_out_len=100,                   # 最长生成 token 数
       batch_size=64,                     # 批次大小
       run_cfg=dict(num_gpus=1),          # 运行配置，用于指定资源需求
    )

# OPT-125M
opt125m = dict(

max_out_len=100,                   # 最长生成 token 数
    )

# OPT-125M
       # 以下参数为 HuggingFaceCausalLM 的初始化参数
       path='facebook/opt-125m',
       tokenizer_path='facebook/opt-125m',
       abbr='opt125m',                # 模型简称，用于结果展示
       max_out_len=100,               # 最长生成 token 数
       batch_size=128,                # 批次大小
       run_cfg=dict(num_gpus=1),      # 运行配置，用于指定资源需求
    )

models = [opt350m, opt125m]
```

HuggingFace 中的 'facebook/opt-350m' 以及 'facebook/opt-125m' 权重会在运行时自动下载。

run_cfg=dict(num_gpus=1),      # 运行配置，用于指定资源需求
    )

```

关于模型配置的更多介绍可阅读 [准备模型](./user_guides/models.md)。

### 启动评测

配置文件准备完毕后，我们可以使用 debug 模式启动任务，以检查模型加载、数据集读取是否出现异常，如未正确读取缓存等。

```shell
```

但 `--debug` 模式下只能逐一序列执行任务，因此检查无误后，可关闭 `--debug` 模式，使程序充分利用多卡资源

```shell
```

以下是一些与评测相关的参数，可以帮助你根据自己的环境情况配置更高效的推理任务。

- `-w outputs/demo`: 评测日志及结果保存目录。若不指定，则默认为 `outputs/default`
- `-r`: 重启上一次（中断的）评测
- `--mode all`: 指定进行某一阶段的任务
  - all: 进行全阶段评测，包括推理和评估
  - infer: 仅进行各个数据集上的推理
  - eval: 仅基于推理结果进行评估
  - viz: 仅展示评估结果
- `--max-partition-size 2000`: 数据集拆分尺寸，部分数据集可能比较大，利用此参数将其拆分成多个子任务，能有效利用资源。但如果拆分过细，则可能因为模型本身加载时间过长，反而速度更慢
- `--max-num-workers 32`: 最大并行启动任务数，在 Slurm 等分布式环境中，该参数用于指定最大提交任务数；在本地环境中，该参数用于指定最大并行执行的任务数，注意实际并行执行任务数受制于 GPU 等资源数，并不一定为该数字。

如果你不是在本机进行评测，而是使用 slurm 集群，可以指定如下参数：

- `--slurm`: 使用 slurm 在集群提交任务
- `--partition(-p) my_part`: slurm 集群分区
- `--retry 2`: 任务出错重试次数

```{tip}
这个脚本同样支持将任务提交到阿里云深度学习中心（DLC）上运行，以及更多定制化的评测策略。请参考 [评测任务发起](./user_guides/experimentation.md#评测任务发起) 了解更多细节。
```

## 评测结果

评测完成后，会打印评测结果表格如下：

```text
dataset    version    metric    mode      opt350m    opt125m
---------  ---------  --------  ------  ---------  ---------
siqa       e78df3     accuracy  gen         21.55      12.44
winograd   b6c7ed     accuracy  ppl         51.23      49.82
```

所有过程的日志，预测，以及最终结果会放在 `outputs/demo/` 目录下。目录结构如下所示：

```text
├── 20230220_183030   # 一次实验
│   ├── configs       # 每次实验都会在此处存下用于追溯的 config
│   ├── logs          # 运行日志
│   │   ├── eval
│   │   └── infer
│   ├── predictions   # 储存了每个任务的推理结果
│   ├── results       # 储存了每个任务的评测结果

│   │   ├── eval
│   └── summary       # 汇总每次实验的所有评测结果
├── ...
```

打印评测结果的过程可被进一步定制化，用于输出一些数据集的平均分 (例如 MMLU, C-Eval 等)。

关于评测结果输出的更多介绍可阅读 [结果展示](./user_guides/summarizer.md)。

## 更多教程

想要更多了解 OpenCompass, 可以点击下列链接学习。

- [数据集配置](./user_guides/dataset_prepare.md)
- [准备模型](./user_guides/models.md)
- [任务运行和监控](./user_guides/experimentation.md)
- [如何调Prompt](./prompt/overview.md)
- [结果展示](./user_guides/summarizer.md)
- [学习配置文件](./user_guides/config.md)

# Bactteria detection with darkfield microscopy

This project supports **`Bactteria detection with darkfield microscopy`**, which can be downloaded from [here](https://tianchi.aliyun.com/dataset/94411).

Spirochaeta is a genus of bacteria classified within the phylum Spirochaetes. Included in this dataset are 366 darkfield microscopy images and manually annotated masks which can be used for classification and segmentation purposes. Detecting bacteria in blood could have a huge significance for research in both the medical and computer science field.

It was gathered and annotated by students (hand-on experience)
It has more than just one targeted class (blood cell and bacteria were annotated)
It is highly imbalanced, so naive loss functions would work less properly

| Dataset name                                                    | Anatomical region | Task type    | Modality   | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License                                                         |
| --------------------------------------------------------------- | ----------------- | ------------ | ---------- | ------------ | --------------------- | ---------------------- | ------------ | --------------------------------------------------------------- |
| [Bactteria detection](https://tianchi.aliyun.com/dataset/94411) | bacteria          | segmentation | microscopy | 3            | 366/-/-               | yes/-/-                | 2017         | [CC-BY-NC 4.0](https://creativecommons.org/licenses/by-sa/4.0/) |

|  background  |    366     |    85.9    |    -     |    -     |     -     |     -     |
| erythrocytes |    345     |   13.03    |    -     |    -     |     -     |     -     |
| spirochaete  |    288     |    1.07    |    -     |    -     |     -     |     -     |

Note:

![bac](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/microscopy_images/bactteria_detection/bactteria_detection_dataset.png)

## Usage

- pillow (PIL) v9.3.0
- scikit-learn (sklearn) v1.2.0

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `bactteria_detection/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
```

- Download dataset from [here](https://tianchi.aliyun.com/dataset/94411) and save it to the `data/` directory .
- Decompress data to path `data/`. This will create a new folder named `data/Bacteria_detection_with_darkfield_microscopy_datasets/`, which contains the original image data.

- run script `python tools/prepare_dataset.py` to format data and change folder structure as below.

```none
  ├── mmseg
  │   │   │   ├── microscopy_images
  │   │   │   │   ├── bactteria_detection

│   │   │   ├── microscopy_images
  │   │   │   │   │   │   ├── Bacteria_detection_with_darkfield_microscopy_datasets
```

```

|  background  |    292     |   85.66    |    74    |   86.7   |     -     |     -     |
| erythrocytes |    274     |   13.25    |    71    |  12.29   |     -     |     -     |
| spirochaete  |    231     |    1.09    |    57    |   1.01   |     -     |     -     |

```shell
```

```shell
```

```

```shell
```

## Results

### Bactteria detection with darkfield microscopy

***Note: The following experimental results are based on the data randomly partitioned according to the above method described in the dataset preparing section.***

|     Method      | Backbone | Crop Size |   lr   | mIoU  | mDice |                                          config                                          |         download         |

| :-------------: | :------: | :-------: | :----: | :---: | :---: | :--------------------------------------------------------------------------------------: | :----------------------: |
| fcn_unet_s5-d16 |   unet   |  512x512  |  0.01  | 76.48 | 84.68 |  [config](./configs/fcn-unet-s5-d16_unet_1xb16-0.01-20k_bactteria-detection-512x512.py)  | [model](<>) \| [log](<>) |
| fcn_unet_s5-d16 |   unet   |  512x512  | 0.001  | 61.06 | 63.69 | [config](./configs/fcn-unet-s5-d16_unet_1xb16-0.001-20k_bactteria-detection-512x512.py)  | [model](<>) \| [log](<>) |
| fcn_unet_s5-d16 |   unet   |  512x512  | 0.0001 | 58.87 | 62.42 | [config](./configs/fcn-unet-s5-d16_unet_1xb16-0.0001-20k_bactteria-detection-512x512.py) | [model](<>) \| [log](<>) |

# 半监督目标检测

半监督目标检测同时利用标签数据和无标签数据进行训练，一方面可以减少模型对检测框数量的依赖，另一方面也可以利用大量的未标记数据进一步提高模型。

按照以下流程进行半监督目标检测：

- [半监督目标检测](#半监督目标检测)
  - [准备和拆分数据集](#准备和拆分数据集)
  - [配置多分支数据流程](#配置多分支数据流程)
  - [配置半监督数据加载](#配置半监督数据加载)
  - [配置半监督模型](#配置半监督模型)
  - [配置MeanTeacherHook](#配置meanteacherhook)
  - [配置TeacherStudentValLoop](#配置teacherstudentvalloop)

## 准备和拆分数据集

我们提供了数据集下载脚本，默认下载 coco2017 数据集，并且自动解压。

```shell
python tools/misc/download_dataset.py
```

解压后的数据集目录如下：

```plain
├── data
│   ├── coco
│   │   │   ├── image_info_unlabeled2017.json
│   │   │   ├── instances_train2017.json
│   │   │   ├── instances_val2017.json
│   │   ├── unlabeled2017
```

半监督目标检测在 coco 数据集上有两种比较通用的实验设置：

（1）将 `train2017` 按照固定百分比（1%，2%，5% 和 10%）划分出一部分数据作为标签数据集，剩余的训练集数据作为无标签数据集，同时考虑划分不同的训练集数据作为标签数据集对半监督训练的结果影响较大，所以采用五折交叉验证来评估算法性能。我们提供了数据集划分脚本：

```shell
python tools/misc/split_coco.py
```

```shell
```

该脚本默认会按照 1%，2%，5% 和 10% 的标签数据占比划分 `train2017`，每一种划分会随机重复 5 次，用于交叉验证。生成的半监督标注文件名称格式如下：

- 标签数据集标注名称格式：`instances_train2017.{fold}@{percent}.json`

- 无标签数据集名称标注：`instances_train2017.{fold}@{percent}-unlabeled.json`

其中，`fold` 用于交叉验证，`percent` 表示标签数据的占比。 划分后的数据集目录结构如下：

```plain
├── data
│   ├── coco
│   │   ├── semi_anns
│   │   │   ├── instances_train2017.1@1.json
│   │   │   ├── instances_train2017.1@1-unlabeled.json
│   │   │   ├── instances_train2017.1@2.json
│   │   │   ├── instances_train2017.1@2-unlabeled.json
│   │   │   ├── instances_train2017.1@5.json
│   │   │   ├── instances_train2017.1@5-unlabeled.json

│   │   │   ├── instances_train2017.1@10.json
│   │   │   ├── instances_train2017.1@10-unlabeled.json
│   │   │   ├── instances_train2017.2@1.json
│   │   │   ├── instances_train2017.2@1-unlabeled.json
```

（2）将 `train2017` 作为标签数据集，`unlabeled2017` 作为无标签数据集。由于 `image_info_unlabeled2017.json` 没有 `categories` 信息，无法初始化 `CocoDataset` ，所以需要将 `instances_train2017.json` 的 `categories` 写入 `image_info_unlabeled2017.json` ，另存为 `instances_unlabeled2017.json`，相关脚本如下：

```python
from mmengine.fileio import load, dump

anns_train = load('instances_train2017.json')
anns_unlabeled = load('image_info_unlabeled2017.json')
anns_unlabeled['categories'] = anns_train['categories']

dump(anns_unlabeled, 'instances_unlabeled2017.json')
```

处理后的数据集目录如下：

```plain
├── data
│   ├── coco
│   │   │   ├── instances_unlabeled2017.json
```

## 配置多分支数据流程

半监督学习有两个主要的方法，分别是
[一致性正则化](https://research.nvidia.com/sites/default/files/publications/laine2017iclr_paper.pdf)

```

和[伪标签](https://www.researchgate.net/profile/Dong-Hyun-Lee/publication/280581078_Pseudo-Label_The_Simple_and_Efficient_Semi-Supervised_Learning_Method_for_Deep_Neural_Networks/links/55bc4ada08ae092e9660b776/Pseudo-Label-The-Simple-and-Efficient-Semi-Supervised-Learning-Method-for-Deep-Neural-Networks.pdf) 。
一致性正则化往往需要一些精心的设计，而伪标签的形式比较简单，更容易拓展到下游任务。我们主要采用了基于伪标签的教师学生联合训练的半监督目标检测框架，对于标签数据和无标签数据需要配置不同的数据流程：
（1）标签数据的数据流程：

```python
# pipeline used to augment labeled data,
# which will be sent to student model for supervised training.
sup_pipeline = [
    dict(type='LoadImageFromFile',backend_args = backend_args),
    dict(type='RandomResize', scale=scale, keep_ratio=True),

dict(type='LoadAnnotations', with_bbox=True),
    dict(type='RandAugment', aug_space=color_space, aug_num=1),
    dict(type='FilterAnnotations', min_gt_bbox_wh=(1e-2, 1e-2)),
    dict(type='MultiBranch', sup=dict(type='PackDetInputs'))
]
```

（2）无标签的数据流程：

```python
# pipeline used to augment unlabeled data weakly,
# which will be sent to teacher model for predicting pseudo instances.
weak_pipeline = [
    dict(
                   'scale_factor', 'flip', 'flip_direction',
                   'homography_matrix')),
]

# pipeline used to augment unlabeled data strongly,
# which will be sent to student model for unsupervised training.
strong_pipeline = [

'homography_matrix')),
]

    dict(
        type='RandomOrder',
            dict(type='RandAugment', aug_space=color_space, aug_num=1),
            dict(type='RandAugment', aug_space=geometric, aug_num=1),
        ]),
    dict(type='RandomErasing', n_patches=(1, 5), ratio=(0, 0.2)),
    dict(
]

# pipeline used to augment unlabeled data into different views
unsup_pipeline = [
    dict(type='LoadImageFromFile', backend_args = backend_args),

]

    dict(type='LoadEmptyAnnotations'),
    dict(
        type='MultiBranch',
        unsup_teacher=weak_pipeline,
        unsup_student=strong_pipeline,
    )
]
```

## 配置半监督数据加载

（1）构建半监督数据集。使用 `ConcatDataset` 拼接标签数据集和无标签数据集。

```python
labeled_dataset = dict(
    ann_file='annotations/instances_train2017.json',
    pipeline=sup_pipeline)

unlabeled_dataset = dict(
    ann_file='annotations/instances_unlabeled2017.json',
    data_prefix=dict(img='unlabeled2017/'),
    filter_cfg=dict(filter_empty_gt=False),
    pipeline=unsup_pipeline)

    batch_size=batch_size,

data_prefix=dict(img='unlabeled2017/'),

    num_workers=num_workers,
        type='GroupMultiSourceSampler',
        batch_size=batch_size,
        source_ratio=[1, 4]),
        type='ConcatDataset', datasets=[labeled_dataset, unlabeled_dataset]))
```

（2）使用多源数据集采样器。 使用 `GroupMultiSourceSampler` 从 `labeled_dataset` 和 `labeled_dataset` 采样数据组成 batch , `source_ratio` 控制 batch 中标签数据和无标签数据的占比。`GroupMultiSourceSampler` 还保证了同一个 batch 中的图片具有相近的长宽比例，如果不需要保证batch内图片的长宽比例，可以使用 `MultiSourceSampler`。`GroupMultiSourceSampler` 采样示意图如下：

<img src="https://user-images.githubusercontent.com/40661020/186149261-8cf28e92-de5c-4c8c-96e1-13558b2e27f7.jpg"/>
</div>

</div>

`sup=1000` 表示标签数据集的规模为 1000 ，`sup_h=200` 表示标签数据集中长宽比大于等于1的图片规模为 200，`sup_w=800` 表示标签数据集中长宽比小于1的图片规模为 800 ，`unsup=9000` 表示无标签数据集的规模为 9000 ，`unsup_h=1800` 表示无标签数据集中长宽比大于等于1的图片规模为 1800，`unsup_w=7200` 表示标签数据集中长宽比小于1的图片规模为 7200 ，`GroupMultiSourceSampler` 每次按照标签数据集和无标签数据集的图片的总体长宽比分布随机选择一组，然后按照 `source_ratio` 从两个数据集中采样组成 batch ，因此标签数据集和无标签数据集重复采样次数不同。

## 配置半监督模型

我们选择 `Faster R-CNN` 作为 `detector` 进行半监督训练，以半监督目标检测算法 `SoftTeacher` 为例，模型的配置可以继承 `_base_/models/faster-rcnn_r50_fpn.py`，将检测器的骨干网络替换成 `caffe` 风格。
注意，与监督训练的配置文件不同的是，`Faster R-CNN` 作为 `detector`，是作为 `model`的一个属性，而不是 `model` 。此外，还需要将`data_preprocessor`设置为`MultiBranchDataPreprocessor`，用于处理不同数据流程图片的填充和归一化。
最后，可以通过 `semi_train_cfg` 和 `semi_test_cfg` 配置半监督训练和测试需要的参数。

```python
_base_ = [
    '../_base_/models/faster-rcnn_r50_fpn.py', '../_base_/default_runtime.py',

```python
_base_ = [
    '../_base_/datasets/semi_coco_detection.py'
]

detector = _base_.model
detector.data_preprocessor = dict(
    type='DetDataPreprocessor',
    mean=[103.530, 116.280, 123.675],
    std=[1.0, 1.0, 1.0],
    bgr_to_rgb=False,
    pad_size_divisor=32)
detector.backbone = dict(
    type='ResNet',
    depth=50,
    frozen_stages=1,
    norm_cfg=dict(type='BN', requires_grad=False),
    norm_eval=True,
    style='caffe',
    init_cfg=dict(
        type='Pretrained',
        checkpoint='open-mmlab://detectron2/resnet50_caffe'))

    type='SoftTeacher',
    detector=detector,
        type='MultiBranchDataPreprocessor',
        data_preprocessor=detector.data_preprocessor),
    semi_train_cfg=dict(
        freeze_teacher=True,

        sup_weight=1.0,
        unsup_weight=4.0,
        pseudo_label_initial_score_thr=0.5,
        rpn_pseudo_thr=0.9,
        cls_pseudo_thr=0.9,
        reg_pseudo_thr=0.02,
        jitter_times=10,
        jitter_scale=0.06,
        min_pseudo_bbox_wh=(1e-2, 1e-2)),
    semi_test_cfg=dict(predict_on='teacher'))
```

此外，我们也支持其他检测模型进行半监督训练，比如，`RetinaNet` 和 `Cascade R-CNN`。由于 `SoftTeacher` 仅支持 `Faster R-CNN`，所以需要将其替换为 `SemiBaseDetector`，示例如下：

```python
_base_ = [
    '../_base_/models/retinanet_r50_fpn.py', '../_base_/default_runtime.py',
]

    type='SemiBaseDetector',

]

        unsup_weight=1.0,
```

沿用 `SoftTeacher` 的半监督训练配置，将 `batch_size` 改为 2 ，`source_ratio` 改为 `[1, 1]`，`RetinaNet`，`Faster R-CNN`， `Cascade R-CNN` 以及 `SoftTeacher` 在 10% coco 训练集上的监督训练和半监督训练的实验结果如下：

|      Model       |   Detector    | BackBone | Style | sup-0.1-coco mAP | semi-0.1-coco mAP |
| :--------------: | :-----------: | :------: | :---: | :--------------: | :---------------: |
| SemiBaseDetector |   RetinaNet   | R-50-FPN | caffe |       23.5       |       27.7        |

| SemiBaseDetector | Faster R-CNN  | R-50-FPN | caffe |       26.7       |       28.4        |
| SemiBaseDetector | Cascade R-CNN | R-50-FPN | caffe |       28.0       |       29.7        |
|   SoftTeacher    | Faster R-CNN  | R-50-FPN | caffe |       26.7       |       31.1        |

## 配置MeanTeacherHook

通常，教师模型采用对学生模型指数滑动平均（EMA）的方式进行更新，进而教师模型随着学生模型的优化而优化，可以通过配置 `custom_hooks` 实现：

```python
custom_hooks = [dict(type='MeanTeacherHook')]
```

## 配置TeacherStudentValLoop

由于教师学生联合训练框架存在两个模型，我们可以用 `TeacherStudentValLoop` 替换 `ValLoop`，在训练的过程中同时检验两个模型的精度。

```python
val_cfg = dict(type='TeacherStudentValLoop')
```

# Textsnake

> [TextSnake: A Flexible Representation for Detecting Text of Arbitrary Shapes](https://arxiv.org/abs/1807.01544)

Driven by deep neural networks and large scale datasets, scene text detection methods have progressed substantially over the past years, continuously refreshing the performance records on various standard benchmarks. However, limited by the representations (axis-aligned rectangles, rotated rectangles or quadrangles) adopted to describe text, existing methods may fall short when dealing with much more free-form text instances, such as curved text, which are actually very common in real-world scenarios. To tackle this problem, we propose a more flexible representation for scene text, termed as TextSnake, which is able to effectively represent text instances in horizontal, oriented and curved forms. In TextSnake, a text instance is described as a sequence of ordered, overlapping disks centered at symmetric axes, each of which is associated with potentially variable radius and orientation. Such geometry attributes are estimated via a Fully Convolutional Network (FCN) model. In experiments, the text detector based on TextSnake achieves state-of-the-art or comparable performance on Total-Text and SCUT-CTW1500, the two newly published benchmarks with special emphasis on curved text in natural images, as well as the widely-used datasets ICDAR 2015 and MSRA-TD500. Specifically, TextSnake outperforms the baseline on Total-Text by more than 40% in F-measure.

<img src="https://user-images.githubusercontent.com/22607038/142795949-2525ead4-865b-4762-baaa-e977cfd6ac66.png"/>
</div>

| [TextSnake](/configs/textdet/textsnake/textsnake_resnet50_fpn-unet_1200e_ctw1500.py) |                 ResNet50                  |        -         | CTW1500 Train | CTW1500 Test |  1200   |    736    |  0.8535   | 0.8052 | 0.8286 | [model](https://download.openmmlab.com/mmocr/textdet/textsnake/textsnake_resnet50_fpn-unet_1200e_ctw1500/textsnake_resnet50_fpn-unet_1200e_ctw1500_20220825_221459-c0b6adc4.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/textsnake/textsnake_resnet50_fpn-unet_1200e_ctw1500/20220825_221459.log) |

| [TextSnake_r50-oclip](/configs/textdet/textsnake/textsnake_resnet50-oclip_fpn-unet_1200e_ctw1500.py) | [ResNet50-oCLIP](https://download.openmmlab.com/mmocr/backbone/resnet50-oclip-7ba0c533.pth) |        -         | CTW1500 Train | CTW1500 Test |  1200   |    736    |  0.8869   | 0.8215 | 0.8529 | [model](https://download.openmmlab.com/mmocr/textdet/textsnake/textsnake_resnet50-oclip_fpn-unet_1200e_ctw1500/textsnake_resnet50-oclip_fpn-unet_1200e_ctw1500_20221101_134814-a216e5b2.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/textsnake/textsnake_resnet50-oclip_fpn-unet_1200e_ctw1500/20221101_134814.log) |

```bibtex
@article{long2018textsnake,
  title={TextSnake: A Flexible Representation for Detecting Text of Arbitrary Shapes},
  author={Long, Shangbang and Ruan, Jiaqiang and Zhang, Wenjie and He, Xin and Wu, Wenhao and Yao, Cong},
  pages={20-36},
}
```

# ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spotting

<div>
<a href="https://arxiv.org/abs/2105.03620">[arXiv paper]</a>
<a href="https://ieeexplore.ieee.org/document/9525302">[TPAMI paper]</a>
</div>

This is an implementation of [ABCNetV2](https://github.com/aim-uofa/AdelaiDet) based on [MMOCR](https://github.com/open-mmlab/mmocr/tree/dev-1.x), [MMCV](https://github.com/open-mmlab/mmcv), and [MMEngine](https://github.com/open-mmlab/mmengine).

**ABCNetV2**  contributions are four-fold: 1) For the first time, we adaptively fit arbitrarily-shaped text by a parameterized Bezier curve, which, compared with segmentation-based methods, can not only provide structured output but also controllable representation. 2) We design a novel BezierAlign layer for extracting accurate convolution features of a text instance of arbitrary shapes, significantly improving the precision of recognition over previous methods. 3) Different from previous methods, which often suffer from complex post-processing and sensitive hyper-parameters, our ABCNet v2 maintains a simple pipeline with the only post-processing non-maximum suppression (NMS). 4) As the performance of text recognition closely depends on feature alignment, ABCNet v2 further adopts a simple yet effective coordinate convolution to encode the position of the convolutional filters, which leads to a considerable improvement with negligible computation overhead. Comprehensive experiments conducted on various bilingual (English and Chinese) benchmark datasets demonstrate that ABCNet v2 can achieve state-of-the-art performance while maintaining very high efficiency.

<center>
<img src="https://user-images.githubusercontent.com/24622904/213096846-2557e0ac-ca18-4c4f-88c1-569107f48f9b.png">
</center>

## Usage

- [MIM](https://github.com/open-mmlab/mim)
- [MMOCR](https://github.com/open-mmlab/mmocr)

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `ABCNet/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
# Linux
# Windows PowerShell
$env:PYTHONPATH=Get-Location
```

if the data is not in `ABCNet/`, you can link the data into `ABCNet/`:

```shell
# Linux
ln -s ${DataPath} $PYTHONPATH

```

```shell
# Linux
New-Item -ItemType SymbolicLink -Path $env:PYTHONPATH -Name data  -Target ${DataPath}
```

In the current directory, run the following command to test the model:

```bash
mim test mmocr config/abcnet_v2/abcnet-v2_resnet50_bifpn_500e_icdar2015.py --work-dir work_dirs/ --checkpoint ${CHECKPOINT_PATH}
```

## Results

Here we provide the baseline version of ABCNet with ResNet50 backbone.

To find more variants, please visit the [official model zoo](https://github.com/aim-uofa/AdelaiDet/blob/master/configs/BAText/README.md).

|         Name          | Pretrained Model | E2E-None-Hmean | det-Hmean |                                                                   Download                                                                   |

| :-------------------: | :--------------: | :------------: | :-------: | :------------------------------------------------------------------------------------------------------------------------------------------: |
| v2-icdar2015-finetune |    SynthText     |     0.6628     |  0.8886   | [model](https://download.openmmlab.com/mmocr/textspotting/abcnet-v2/abcnet-v2_resnet50_bifpn/abcnet-v2_resnet50_bifpn_500e_icdar2015-5e4cc7ed.pth) |

If you find ABCNetV2 useful in your research or applications, please cite ABCNetV2 with the following BibTeX entry.

```BibTeX
@ARTICLE{9525302,
  author={Liu, Yuliang and Shen, Chunhua and Jin, Lianwen and He, Tong and Chen, Peng and Liu, Chongyu and Chen, Hao},
  title={ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spotting},
  volume={},
  number={},
  pages={1-1},
  doi={10.1109/TPAMI.2021.3107437}}
```

```

    <!-- The code's design shall follow existing interfaces and convention. For example, each model component should be registered into `mmocr.registry.MODELS` and configurable via a config file. -->

    <!-- Ideally *all* the methods should have [type hints](https://www.pythontutorial.net/python-basics/python-type-hints/) and [docstrings](https://google.github.io/styleguide/pyguide.html#381-docstrings). [Example](https://github.com/open-mmlab/mmocr/blob/76637a290507f151215d299707c57cea5120976e/mmocr/utils/polygon_utils.py#L80-L96) -->

    <!-- Unit tests for each module are required. [Example](https://github.com/open-mmlab/mmocr/blob/76637a290507f151215d299707c57cea5120976e/tests/test_utils/test_polygon_utils.py#L97-L106) -->

    <!-- It will be parsed by MIM and Inferencer. [Example](https://github.com/open-mmlab/mmocr/blob/1.x/configs/textdet/dbnet/metafile.yml) -->

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily receiving feedbacks. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

2. The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
3. If the modification has a dependency on downstream projects of a newer version, this PR should be tested with all supported versions of downstream projects.

# 混合精度训练的迁移

在0.x版中，MMEditing并不支持对整体前向过程的混合精度训练。相反，用户必须使用`auto_fp16`装饰器来适配特定子模块，然后再将子模块的参数转化成fp16。这样就可以拥有对模型参数的更细粒度的控制，但是该方法使用起来很繁琐，而且用户需要自己处理一些操作，比如训练过程中损失函数的缩放

MMagic 1.x版使用了MMEngine提供的`AmpOptimWrapper`，在`AmpOptimWrapper.update_params`中，梯度缩放和`GradScaler`更新将被自动执行，且在`optim_context`上下文管理其中，`auto_cast`被应用到整个前向过程中。

具体来说，0.x版和1.x版之间的差异如下所示：

<thead>
  <tr>
  </tr>
</thead>
<tbody>
<tr>

```python
# 配置
runner = dict(fp16_loss_scaler=dict(init_scale=512))
```

```python
# 代码
from mmedit.models.builder import build_model
from mmedit.core.runners.fp16_utils import auto_fp16

class DemoModule(nn.Module):
    def __init__(self, cfg):
        self.net = build_model(cfg)

    @auto_fp16
        return self.net(x)

class DemoModel(nn.Module):

        super().__init__(self)
        self.demo_network = DemoModule(cfg)

def forward(self, x):

    def train_step(self,
                   data_batch,
                   optimizer,
                   ddp_reducer=None,
                   loss_scaler=None,
                   use_apex_amp=False,
                   running_status=None):
        # 从data_batch中获取数据
        inputs = data_batch['img']
        output = self.demo_network(inputs)

        loss, log_vars = self.get_loss(data_dict_)

        if ddp_reducer is not None:
            ddp_reducer.prepare_for_backward(_find_tensors(loss_disc))

        if loss_scaler:
            # 添加fp16支持
            loss_scaler.scale(loss_disc).backward()
        elif use_apex_amp:
            from apex import amp
            with amp.scale_loss(loss_disc, optimizer,
                    loss_id=0) as scaled_loss_disc:

elif use_apex_amp:
                scaled_loss_disc.backward()
        else:
            loss_disc.backward()

            loss_scaler.unscale_(optimizer)
            loss_scaler.step(optimizer)
        else:
            optimizer.step()
```

</td>

```python
# 配置
    constructor='OptimWrapperConstructor',
        optimizer=dict(type='Adam', lr=0.0001, betas=(0.0, 0.999), eps=1e-06),
        type='AmpOptimWrapper',  # 使用amp封装器
        loss_scale='dynamic'),
        optimizer=dict(type='Adam', lr=0.0004, betas=(0.0, 0.999), eps=1e-06),
        loss_scale='dynamic'))
```

```python
# 代码

optimizer=dict(type='Adam', lr=0.0004, betas=(0.0, 0.999), eps=1e-06),
```

```python
# 代码
from mmagic.registry import MODULES

        self.net = MODULES.build(cfg)

class DemoModel(BaseModel):

    def train_step(self, data, optim_wrapper):

        with optim_wrapper.optim_context(self.discriminator):
            output = self.demo_network(inputs)
        loss_dict = self.get_loss(output)
        # 使用`BaseModel`提供的parse_loss
        loss, log_vars = self.parse_loss(loss_dict)
        optimizer_wrapper.update_params(loss)

loss_dict = self.get_loss(output)

```

</td>

</tr>
</tbody>
</table>

若要避免用户操作配置文件，MMagic在`train.py`里提供了`--amp` 选项，其可以让用户在不修改配置文件的情况下启动混合精度训练，用户可以使用以下命令启动混合精度训练：

```bash
bash tools/dist_train.sh CONFIG GPUS --amp

# 对slurm用户
bash tools/slurm_train.sh PARTITION JOB_NAME CONFIG WORK_DIR --amp
```

# FLAVR (arXiv'2020)

> [FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation](https://arxiv.org/pdf/2012.08512.pdf)

Most modern frame interpolation approaches rely on explicit bidirectional optical flows between adjacent frames, thus are sensitive to the accuracy of underlying flow estimation in handling occlusions while additionally introducing computational bottlenecks unsuitable for efficient deployment. In this work, we propose a flow-free approach that is completely end-to-end trainable for multi-frame video interpolation. Our method, FLAVR, is designed to reason about non-linear motion trajectories and complex occlusions implicitly from unlabeled videos and greatly simplifies the process of training, testing and deploying frame interpolation models. Furthermore, FLAVR delivers up to 6× speed up compared to the current state-of-the-art methods for multi-frame interpolation while consistently demonstrating superior qualitative and quantitative results compared with prior methods on popular benchmarks including Vimeo-90K, Adobe-240FPS, and GoPro. Finally, we show that frame interpolation is a competitive self-supervised pre-training task for videos via demonstrating various novel applications of FLAVR including action recognition, optical flow estimation, motion magnification, and video object tracking. Code and trained models are provided in the supplementary material.

 <img src="https://user-images.githubusercontent.com/56712176/169070212-52acdcea-d732-4441-9983-276e2e40b195.png" width="400"/>
</div >

|                                   Model                                    |  Dataset   | scale |  PSNR   |  SSIM   | Training Resources  |                                    Download                                    |
| :------------------------------------------------------------------------: | :--------: | :---: | :-----: | :-----: | :-----------------: | :----------------------------------------------------------------------------: |

| [flavr_in4out1_g8b4_vimeo90k_septuplet](./flavr_in4out1_8xb4_vimeo90k-septuplet.py) | vimeo90k-T |  x2   | 36.3340 | 0.96015 | 8 (Tesla PG503-216) | [model](https://download.openmmlab.com/mmediting/video_interpolators/flavr/flavr_in4out1_g8b4_vimeo90k_septuplet_20220509-c2468995.pth) \| [log](https://download.openmmlab.com/mmediting/video_interpolators/flavr/flavr_in4out1_g8b4_vimeo90k_septuplet_20220509-c2468995.log.json) |

Note: FLAVR for x8 VFI task will supported in the future.

**Train**

<details>

```shell
# cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/flavr/flavr_in4out1_8xb4_vimeo90k-septuplet.py

python tools/train.py configs/flavr/flavr_in4out1_8xb4_vimeo90k-septuplet.py

./tools/dist_train.sh configs/flavr/flavr_in4out1_8xb4_vimeo90k-septuplet.py 8
```

```

**Test**

<details>

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/flavr/flavr_in4out1_8xb4_vimeo90k-septuplet.py https://download.openmmlab.com/mmediting/video_interpolators/flavr/flavr_in4out1_g8b4_vimeo90k_septuplet_20220509-c2468995.pth

python tools/test.py configs/flavr/flavr_in4out1_8xb4_vimeo90k-septuplet.py https://download.openmmlab.com/mmediting/video_interpolators/flavr/flavr_in4out1_g8b4_vimeo90k_septuplet_20220509-c2468995.pth

./tools/dist_test.sh configs/flavr/flavr_in4out1_8xb4_vimeo90k-septuplet.py https://download.openmmlab.com/mmediting/video_interpolators/flavr/flavr_in4out1_g8b4_vimeo90k_septuplet_20220509-c2468995.pth 8
```

```bibtex
@article{kalluri2020flavr,
  title={Flavr: Flow-agnostic video representations for fast frame interpolation},
  author={Kalluri, Tarun and Pathak, Deepak and Chandraker, Manmohan and Tran, Du},
  journal={arXiv preprint arXiv:2012.08512},
}
```

# Projects

There are also selected external projects released in the community that use MMSegmentation:

- [SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation](https://github.com/visual-attention-network/segnext)
- [Vision Transformer Adapter for Dense Predictions](https://github.com/czczup/ViT-Adapter)
- [UniFormer: Unifying Convolution and Self-attention for Visual Recognition](https://github.com/Sense-X/UniFormer)
- [Multi-Scale High-Resolution Vision Transformer for Semantic Segmentation](https://github.com/facebookresearch/HRViT)

- [ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias](https://github.com/ViTAE-Transformer/ViTAE-Transformer)
- [DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation](https://github.com/lhoyer/DAFormer)
- [MPViT : Multi-Path Vision Transformer for Dense Prediction](https://github.com/youngwanLEE/MPViT)
- [TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation](https://github.com/hustvl/TopFormer)

Note: These projects are supported and maintained by their own contributors. The core maintainers of MMSegmentation only ensure the results are reproducible and the code quality meets its claim at the time each project was submitted, but they may not be responsible for future maintenance.

# 新增评测指标

## 使用 MMSegmentation 的源代码进行开发

在这里，我们用 `CustomMetric` 作为例子来展示如何开发一个新的评测指标。

1. 创建一个新文件 `mmseg/evaluation/metrics/custom_metric.py`。

   ```python

           """
           """

           pass

           pass

           pass
   ```

pass

           pass

           pass
   ```

   在上面的示例中，`CustomMetric` 是 `BaseMetric` 的子类。它有三个方法：`process`，`compute_metrics` 和 `evaluate`。

   - `process()` 处理一批数据样本和预测。处理后的结果需要显示地传给 `self.results` ，将在处理所有数据样本后用于计算指标。更多细节请参考 [MMEngine 文档](https://github.com/open-mmlab/mmengine/blob/main/docs/zh_cn/design/evaluation.md)

   - `compute_metrics()` 用于从处理后的结果中计算指标。

   - `evaluate()` 是一个接口，用于计算指标并返回结果。它将由 `ValLoop` 或 `TestLoop` 在 `Runner` 中调用。在大多数情况下，您不需要重写此方法，但如果您想做一些额外的工作，可以重写它。

   **注意：** 您可以在[这里](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L366) 找到 `Runner` 调用 `evaluate()` 方法的过程。`Runner` 是训练和测试过程的执行器，您可以在[训练引擎文档](./engine.md)中找到有关它的详细信息。

2. 在 `mmseg/evaluation/metrics/__init__.py` 中导入新的指标。

   ```python
   ```

3. 在配置文件中设置新的评测指标

   ```python

   ```python
   ```

   ```python
   ```

## 使用发布版本的 MMSegmentation 进行开发

上面的示例展示了如何使用 MMSegmentation 的源代码开发新指标。如果您想使用 MMSegmentation 的发布版本开发新指标，可以按照以下步骤操作。

1. 创建一个新文件 `/Path/to/metrics/custom_metric.py`，实现 `process`，`compute_metrics` 和 `evaluate` 方法，`evaluate` 方法是可选的。

2. 在代码或配置文件中导入新的指标。

   ```python
   ```

   或者

   ```python

   ```

<details>

```bibtex
}
```

<details>

```bibtex

| [pose_shufflenetv1](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_shufflenetv1_8xb64-210e_coco-256x192.py) |  256x192   | 0.587 |      0.849      |      0.654      | 0.654 |      0.896      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_shufflenetv1_8xb64-210e_coco-256x192-7a7ea4f4_20221013.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_shufflenetv1_8xb64-210e_coco-256x192_20221013.log) |
| [pose_shufflenetv1](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_shufflenetv1_8xb64-210e_coco-384x288.py) |  384x288   | 0.626 |      0.862      |      0.696      | 0.687 |      0.903      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_shufflenetv1_8xb64-210e_coco-384x288-8342f8ba_20221013.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_shufflenetv1_8xb64-210e_coco-384x288_20221013.log) |

# Overview

MMOCR is an open source toolkit based on [PyTorch](https://pytorch.org/) and [MMDetection](https://github.com/open-mmlab/mmdetection), supporting numerous OCR-related models, including text detection, text recognition, and key information extraction. In addition, it supports widely-used academic datasets and provides many useful tools, assisting users in exploring various aspects of models and datasets and implementing high-quality algorithms. Generally, it has the following features.

- **One-stop, Multi-model**: MMOCR supports various OCR-related tasks and implements the latest models for text detection, recognition, and key information extraction.
- **Modular Design**: MMOCR's modular design allows users to define and reuse modules in the model on demand.
- **Various Useful Tools**: MMOCR provides a number of analysis tools, including visualizers, validation scripts, evaluators, etc., to help users troubleshoot, finetune or compare models.

- **Powered by [OpenMMLab](https://openmmlab.com/)**: Like other algorithm libraries in OpenMMLab family, MMOCR follows OpenMMLab's rigorous development guidelines and interface conventions, significantly reducing the learning cost of users familiar with other projects in OpenMMLab family. In addition, benefiting from the unified interfaces among OpenMMLab, you can easily call the models implemented in other OpenMMLab projects (e.g. MMDetection) in MMOCR, facilitating cross-domain research and real-world applications.

Together with the release of OpenMMLab 2.0, MMOCR now also comes to its 1.0.0 version, which has made significant BC-breaking changes, resulting in less code redundancy, higher code efficiency and an overall more systematic and consistent design.

Considering that there are some backward incompatible changes in this version compared to 0.x, we have prepared a detailed [migration guide](../migration/overview.md). It lists all the changes made in the new version and the steps required to migrate. We hope this guide can help users familiar with the old framework to complete the upgrade as quickly as possible. Though this may take some time, we believe that the new features brought by MMOCR and the OpenMMLab ecosystem will make it all worthwhile. 😊

Next, please read the section according to your actual needs.

- We recommend that beginners go through [Quick Run](quick_run.md) to get familiar with MMOCR and master the usage of MMOCR by reading the examples in **User Guides**.
- Intermediate and advanced developers are suggested to learn the background, conventions, and recommended implementations of each component from **Basic Concepts**.
- Read our [FAQ](faq.md) to find answers to frequently asked questions.

- If you can't find the answers you need in the documentation, feel free to raise an [issue](https://github.com/open-mmlab/mmocr/issues).
- Everyone is welcome to be a contributor! Read the [contribution guide](../notes/contribution_guide.md) to learn how to contribute to MMOCR!

# IndexNet (ICCV'2019)

> [Indices Matter: Learning to Index for Deep Image Matting](https://arxiv.org/abs/1908.00672)

We show that existing upsampling operators can be unified with the notion of the index function. This notion is inspired by an observation in the decoding process of deep image matting where indices-guided unpooling can recover boundary details much better than other upsampling operators such as bilinear interpolation. By looking at the indices as a function of the feature map, we introduce the concept of learning to index, and present a novel index-guided encoder-decoder framework where indices are self-learned adaptively from data and are used to guide the pooling and upsampling operators, without the need of supervision. At the core of this framework is a flexible network module, termed IndexNet, which dynamically predicts indices given an input. Due to its flexibility, IndexNet can be used as a plug-in applying to any off-the-shelf convolutional networks that have coupled downsampling and upsampling stages.

 <img src="https://user-images.githubusercontent.com/12726765/144176083-52604501-1f46-411d-b81a-cad0eb4b529f.png" width="400"/>
</div >

|                                Model                                 |    Dataset     |   SAD    |    MSE    |   GRAD   | CONN | Training Resources |                                Download                                 |
| :------------------------------------------------------------------: | :------------: | :------: | :-------: | :------: | :--: | :----------------: | :---------------------------------------------------------------------: |

|        [M2O DINs (our)](./indexnet_mobv2_1xb16-78k_comp1k.py)        | Composition-1k | **45.6** | **0.012** | **25.5** | 44.8 |         1          | [model](https://download.openmmlab.com/mmediting/mattors/indexnet/indexnet_mobv2_1x16_78k_comp1k_SAD-45.6_20200618_173817-26dd258d.pth) \| [log](https://download.openmmlab.com/mmediting/mattors/indexnet/indexnet_mobv2_1x16_78k_comp1k_20200618_173817.log.json) |
| [M2O DINs (with DIM pipeline)](./indexnet_mobv2-dimaug_1xb16-78k_comp1k.py) | Composition-1k |   50.1   |   0.016   |   30.8   | 49.5 |         1          | [model](https://download.openmmlab.com/mmediting/mattors/indexnet/indexnet_dimaug_mobv2_1x16_78k_comp1k_SAD-50.1_20200626_231857-af359436.pth) \| [log](https://download.openmmlab.com/mmediting/mattors/indexnet/indexnet_dimaug_mobv2_1x16_78k_comp1k_20200626_231857.log.json) |

<!-- |                    M2O DINs (paper)                    |   45.8   |   0.013   |   25.9   | **43.7** |         -          |                     -->

> The performance of training (best performance) with different random seeds diverges in a large range. You may need to run several experiments for each setting to obtain the above performance.

**Train**

<details>

```shell
# cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/indexnet/indexnet_mobv2_1xb16-78k_comp1k.py

python tools/train.py configs/indexnet/indexnet_mobv2_1xb16-78k_comp1k.py

./tools/dist_train.sh configs/indexnet/indexnet_mobv2_1xb16-78k_comp1k.py 8
```

**Test**

<details>

```

**Test**

<details>

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/indexnet/indexnet_mobv2_1xb16-78k_comp1k.py https://download.openmmlab.com/mmediting/mattors/indexnet/indexnet_mobv2_1x16_78k_comp1k_SAD-45.6_20200618_173817-26dd258d.pth

python tools/test.py configs/indexnet/indexnet_mobv2_1xb16-78k_comp1k.py https://download.openmmlab.com/mmediting/mattors/indexnet/indexnet_mobv2_1x16_78k_comp1k_SAD-45.6_20200618_173817-26dd258d.pth

./tools/dist_test.sh configs/indexnet/indexnet_mobv2_1xb16-78k_comp1k.py https://download.openmmlab.com/mmediting/mattors/indexnet/indexnet_mobv2_1x16_78k_comp1k_SAD-45.6_20200618_173817-26dd258d.pth 8
```

```

```bibtex
@inproceedings{hao2019indexnet,
  title={Indices Matter: Learning to Index for Deep Image Matting},
  author={Lu, Hao and Dai, Yutong and Shen, Chunhua and Xu, Songcen},
  booktitle={Proc. IEEE/CVF International Conference on Computer Vision (ICCV)},
}
```

# PETR

This is an README for `PETR`.

Author: @SekiroRong.
This is an implementation of *PETR*.

## Usage

```bash
python tools/train.py projects/PETR/config/petr/petr_vovnet_gridmask_p4_800x320.py
```

```bash
python tools/test.py projects/PETR/config/petr/petr_vovnet_gridmask_p4_800x320.py ${CHECKPOINT_PATH}
```

## Results

```

## Results

 You should claim whether this is based on the pre-trained weights, which are converted from the official release; or it's a reproduced result obtained from retraining the model in this project. -->

This Result is trained by petr_vovnet_gridmask_p4_800x320.py and use [weights](https://drive.google.com/file/d/1ABI5BoQCkCkP4B0pO5KBJ3Ni0tei0gZi/view?usp=sharing) as pretrain weight.

|                                   Backbone                                    | Lr schd | Mem (GB) | Inf time (fps) | mAP  | NDS  |                                                                                                      Download                                                                                                       |

| :---------------------------------------------------------------------------: | :-----: | :------: | :------------: | :--: | :--: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| [petr_vovnet_gridmask_p4_800x320](configs/petr_vovnet_gridmask_p4_800x320.py) |   1x    |   7.62   |      18.7      | 38.3 | 43.5 | [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/petr/petr_vovnet_gridmask_p4_800x320-e2191752.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/petr/20221222_232156.log) |

# 第六章： TensorRT 模型构建与推理

模型部署入门教程继续更新啦！相信经过前几期的学习，大家已经对 ONNX 这一中间表示有了一个比较全面的认识，但是在具体的生产环境中，ONNX 模型常常需要被转换成能被具体推理后端使用的模型格式。本篇教程我们就和大家一起来认识大名鼎鼎的推理后端 TensorRT。

## TensorRT 简介

TensorRT 是由 NVIDIA 发布的深度学习框架，用于在其硬件上运行深度学习推理。TensorRT 提供量化感知训练和离线量化功能，用户可以选择 INT8 和 FP16 两种优化模式，将深度学习模型应用到不同任务的生产部署，如视频流、语音识别、推荐、欺诈检测、文本生成和自然语言处理。TensorRT 经过高度优化，可在 NVIDIA GPU 上运行， 并且可能是目前在 NVIDIA GPU 运行模型最快的推理引擎。关于 TensorRT 更具体的信息可以访问 [TensorRT官网](https://developer.nvidia.com/tensorrt) 了解。

## 安装 TensorRT

默认在一台有 NVIDIA 显卡的机器上，提前安装好 [CUDA](https://developer.nvidia.com/cuda-toolkit-archive) 和 [CUDNN](https://developer.nvidia.com/rdp/cudnn-archive)，登录 NVIDIA 官方网站下载和主机 CUDA 版本适配的 TensorRT 压缩包即可。

以 CUDA 版本是 10.2 为例，选择适配 CUDA 10.2 的 [zip 包](https://developer.nvidia.com/compute/machine-learning/tensorrt/secure/8.2.5.1/zip/tensorrt-8.2.5.1.windows10.x86_64.cuda-10.2.cudnn8.2.zip)，下载完成后，有 conda 虚拟环境的用户可以优先切换到虚拟环境中，然后在 powershell 中执行类似如下的命令安装并测试：

```shell
cd \the\path\of\tensorrt\zip\file

```shell
Expand-Archive TensorRT-8.2.5.1.Windows10.x86_64.cuda-10.2.cudnn8.2.zip .
$env:TENSORRT_DIR = "$pwd\TensorRT-8.2.5.1"
$env:path = "$env:TENSORRT_DIR\lib;" + $env:path
pip install $env:TENSORRT_DIR\python\tensorrt-8.2.5.1-cp36-none-win_amd64.whl
python -c "import tensorrt;print(tensorrt.__version__)"
```

上述命令会在安装后检查 TensorRT 版本，如果打印结果是 8.2.5.1，说明安装 Python 包成功了。

### Linux

和在 Windows 环境下安装类似，默认在一台有 NVIDIA 显卡的机器上，提前安装好 [CUDA](https://developer.nvidia.com/cuda-toolkit-archive) 和 [CUDNN](https://developer.nvidia.com/rdp/cudnn-archive)，登录 NVIDIA 官方网站下载和主机 CUDA 版本适配的 TensorRT 压缩包即可。

以 CUDA 版本是 10.2 为例，选择适配 CUDA 10.2 的 [tar 包](https://developer.nvidia.com/compute/machine-learning/tensorrt/secure/8.2.5.1/tars/tensorrt-8.2.5.1.linux.x86_64-gnu.cuda-10.2.cudnn8.2.tar.gz)，然后执行类似如下的命令安装并测试：

```shell
tar -zxvf TensorRT-8.2.5.1.linux.x86_64-gnu.cuda-10.2.cudnn8.2.tar.gz
export TENSORRT_DIR=$(pwd)/TensorRT-8.2.5.1

```shell
pip install TensorRT-8.2.5.1/python/tensorrt-8.2.5.1-cp37-none-linux_x86_64.whl
```

如果发现打印结果是 8.2.5.1，说明安装 Python 包成功了。

### Jetson

对于 Jetson 平台，我们有非常详细的安装环境配置教程，可参考 [MMDeploy 安装文档](https://github.com/open-mmlab/mmdeploy/tree/main/docs/zh_cn/01-how-to-build/jetsons.md)。需要注意的是，在 Jetson 上配置的 CUDA 版本 TensorRT 版本与 JetPack 强相关的，我们选择适配硬件的版本即可。配置好环境后，通过 `python -c "import tensorrt;print(tensorrt.__version__)"` 查看TensorRT版本是否正确。

## 模型构建

我们使用 TensorRT 生成模型主要有两种方式：

1. 直接通过 TensorRT 的 API 逐层搭建网络；
2. 将中间表示的模型转换成 TensorRT 的模型，比如将 ONNX 模型转换成 TensorRT 模型。

接下来，我们将用 Python 和 C++ 语言分别使用这两种方式构建 TensorRT 模型，并将生成的模型进行推理。

### 直接构建

## 模型构建

### 直接构建

利用 TensorRT 的 API 逐层搭建网络，这一过程类似使用一般的训练框架，如使用 Pytorch 或者TensorFlow 搭建网络。需要注意的是对于权重部分，如卷积或者归一化层，需要将权重内容赋值到 TensorRT 的网络中。本文就不详细展示，只搭建一个对输入做池化的简单网络。

#### 使用 Python API 构建

首先是使用 Python API 直接搭建 TensorRT 网络，这种方法主要是利用 `tensorrt.Builder` 的 `create_builder_config` 和 `create_network` 功能，分别构建 config 和 network，前者用于设置网络的最大工作空间等参数，后者就是网络主体，需要对其逐层添加内容。

此外，需要定义好输入和输出名称，将构建好的网络序列化，保存成本地文件。值得注意的是：如果想要网络接受不同分辨率的输入输出，需要使用 `tensorrt.Builder` 的 `create_optimization_profile` 函数，并设置最小、最大的尺寸。

实现代码如下：

```python
import tensorrt as trt

verbose = True
IN_NAME = 'input'
OUT_NAME = 'output'
IN_H = 224
IN_W = 224
BATCH_SIZE = 1

EXPLICIT_BATCH = 1 << (int)(
    trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)

TRT_LOGGER = trt.Logger(trt.Logger.VERBOSE) if verbose else trt.Logger()

IN_H = 224
IN_W = 224

with trt.Builder(TRT_LOGGER) as builder, builder.create_builder_config(
) as config, builder.create_network(EXPLICIT_BATCH) as network:
    # define network
    input_tensor = network.add_input(
        name=IN_NAME, dtype=trt.float32, shape=(BATCH_SIZE, 3, IN_H, IN_W))
    pool = network.add_pooling(
        input=input_tensor, type=trt.PoolingType.MAX, window_size=(2, 2))
    pool.stride = (2, 2)
    pool.get_output(0).name = OUT_NAME
    network.mark_output(pool.get_output(0))

    # serialize the model to engine file
    profile = builder.create_optimization_profile()
    profile.set_shape_input('input', *[[BATCH_SIZE, 3, IN_H, IN_W]]*3)
    builder.max_batch_size = 1
    config.max_workspace_size = 1 << 30
    engine = builder.build_engine(network, config)

profile.set_shape_input('input', *[[BATCH_SIZE, 3, IN_H, IN_W]]*3)
    with open('model_python_trt.engine', mode='wb') as f:
        f.write(bytearray(engine.serialize()))
        print("generating file done!")
```

#### 使用 C++ API 构建

对于想要直接用 C++ 语言构建网络的小伙伴来说，整个流程和上述 Python 的执行过程非常类似，需要注意的点主要有：

1. `nvinfer1:: createInferBuilder` 对应 Python 中的 `tensorrt.Builder`，需要传入 `ILogger` 类的实例，但是 `ILogger` 是一个抽象类，需要用户继承该类并实现内部的虚函数。不过此处我们直接使用了 TensorRT 包解压后的 samples 文件夹 ../samples/common/logger.h 文件里的实现 `Logger` 子类。
2. 设置 TensorRT 模型的输入尺寸，需要多次调用 `IOptimizationProfile` 的 `setDimensions` 方法，比 Python `略繁琐一些。IOptimizationProfile` 需要用 `createOptimizationProfile` 函数，对应 Python 的 `create_builder_config` 函数。

实现代码如下：

```cpp
#include <iostream>

#include <NvInfer.h>
#include <../samples/common/logger.h>

using namespace nvinfer1;
using namespace sample;

实现代码如下：

```cpp

const char* IN_NAME = "input";
const char* OUT_NAME = "output";
static const int IN_H = 224;
static const int IN_W = 224;
static const int BATCH_SIZE = 1;
static const int EXPLICIT_BATCH = 1 << (int)(NetworkDefinitionCreationFlag::kEXPLICIT_BATCH);

int main(int argc, char** argv)
{
        // Create builder
        Logger m_logger;
        IBuilder* builder = createInferBuilder(m_logger);
        IBuilderConfig* config = builder->createBuilderConfig();

        // Create model to populate the network
        INetworkDefinition* network = builder->createNetworkV2(EXPLICIT_BATCH);
        ITensor* input_tensor = network->addInput(IN_NAME, DataType::kFLOAT, Dims4{ BATCH_SIZE, 3, IN_H, IN_W });
        IPoolingLayer* pool = network->addPoolingNd(*input_tensor, PoolingType::kMAX, DimsHW{ 2, 2 });

IPoolingLayer* pool = network->addPoolingNd(*input_tensor, PoolingType::kMAX, DimsHW{ 2, 2 });
        pool->setStrideNd(DimsHW{ 2, 2 });
        pool->getOutput(0)->setName(OUT_NAME);
        network->markOutput(*pool->getOutput(0));

        // Build engine
        IOptimizationProfile* profile = builder->createOptimizationProfile();
        profile->setDimensions(IN_NAME, OptProfileSelector::kMIN, Dims4(BATCH_SIZE, 3, IN_H, IN_W));
        profile->setDimensions(IN_NAME, OptProfileSelector::kOPT, Dims4(BATCH_SIZE, 3, IN_H, IN_W));
        profile->setDimensions(IN_NAME, OptProfileSelector::kMAX, Dims4(BATCH_SIZE, 3, IN_H, IN_W));
        config->setMaxWorkspaceSize(1 << 20);
        ICudaEngine* engine = builder->buildEngineWithConfig(*network, *config);

        // Serialize the model to engine file
        IHostMemory* modelStream{ nullptr };
        assert(engine != nullptr);
        modelStream = engine->serialize();

// Serialize the model to engine file

        std::ofstream p("model.engine", std::ios::binary);
        if (!p) {
                std::cerr << "could not open output file to save model" << std::endl;
                return -1;
        }
        p.write(reinterpret_cast<const char*>(modelStream->data()), modelStream->size());
        std::cout << "generating file done!" << std::endl;

        // Release resources
        modelStream->destroy();
        network->destroy();
        engine->destroy();
        builder->destroy();
        config->destroy();
        return 0;
}
```

### IR 转换模型

除了直接通过 TensorRT 的 API 逐层搭建网络并序列化模型，TensorRT 还支持将中间表示的模型（如 ONNX）转换成 TensorRT 模型。

#### 使用 Python API 转换

我们首先使用 Pytorch 实现一个和上文一致的模型，即只对输入做一次池化并输出；然后将 Pytorch 模型转换成 ONNX 模型；最后将 ONNX 模型转换成 TensorRT 模型。

这里主要使用了 TensorRT 的 `OnnxParser` 功能，它可以将 ONNX 模型解析到 TensorRT 的网络中。最后我们同样可以得到一个 TensorRT 模型，其功能与上述方式实现的模型功能一致。

实现代码如下：

```python

onnx_model = 'model.onnx'

class NaiveModel(torch.nn.Module):
        self.pool = torch.nn.MaxPool2d(2, 2)

        return self.pool(x)

device = torch.device('cuda:0')

# generate ONNX model
torch.onnx.export(NaiveModel(), torch.randn(1, 3, 224, 224), onnx_model, input_names=['input'], output_names=['output'], opset_version=11)
onnx_model = onnx.load(onnx_model)

# create builder and network
logger = trt.Logger(trt.Logger.ERROR)
builder = trt.Builder(logger)

network = builder.create_network(EXPLICIT_BATCH)

# parse onnx
parser = trt.OnnxParser(network, logger)

if not parser.parse(onnx_model.SerializeToString()):
    error_msgs = ''
    for error in range(parser.num_errors):
        error_msgs += f'{parser.get_error(error)}\n'
    raise RuntimeError(f'Failed to parse onnx, {error_msgs}')

config = builder.create_builder_config()
config.max_workspace_size = 1<<20
profile = builder.create_optimization_profile()

profile.set_shape('input', [1,3 ,224 ,224], [1,3,224, 224], [1,3 ,224 ,224])
config.add_optimization_profile(profile)
# create engine
with torch.cuda.device(device):

with open('model.engine', mode='wb') as f:
    f.write(bytearray(engine.serialize()))

    print("generating file done!")
```

IR 转换时，如果有多 Batch、多输入、动态 shape 的需求，都可以通过多次调用 `set_shape` 函数进行设置。`set_shape` 函数接受的传参分别是：输入节点名称，可接受的最小输入尺寸，最优的输入尺寸，可接受的最大输入尺寸。一般要求这三个尺寸的大小关系为单调递增。

#### 使用 C++ API 转换

介绍了如何用 Python 语言将 ONNX 模型转换成 TensorRT 模型后，再介绍下如何用 C++ 将 ONNX 模型转换成 TensorRT 模型。这里通过 `NvOnnxParser`，我们可以将上一小节转换时得到的 ONNX 文件直接解析到网络中。

实现代码如下：

```cpp

#include <NvOnnxParser.h>

using namespace nvonnxparser;

{

{
        const auto explicitBatch = 1U << static_cast<uint32_t>(NetworkDefinitionCreationFlag::kEXPLICIT_BATCH);

        INetworkDefinition* network = builder->createNetworkV2(explicitBatch);

        // Parse ONNX file
        IParser* parser = nvonnxparser::createParser(*network, m_logger);
        bool parser_status = parser->parseFromFile("model.onnx", static_cast<int>(ILogger::Severity::kWARNING));

        // Get the name of network input
        Dims dim = network->getInput(0)->getDimensions();
        if (dim.d[0] == -1)  // -1 means it is a dynamic model
        {
                const char* name = network->getInput(0)->getName();

Dims dim = network->getInput(0)->getDimensions();
        {
                IOptimizationProfile* profile = builder->createOptimizationProfile();
                profile->setDimensions(name, OptProfileSelector::kMIN, Dims4(1, dim.d[1], dim.d[2], dim.d[3]));
                profile->setDimensions(name, OptProfileSelector::kOPT, Dims4(1, dim.d[1], dim.d[2], dim.d[3]));
                profile->setDimensions(name, OptProfileSelector::kMAX, Dims4(1, dim.d[1], dim.d[2], dim.d[3]));
                config->addOptimizationProfile(profile);
        }

        if (!p) {
                return -1;
        }
        std::cout << "generate file success!" << std::endl;

        return 0;
}
```

### 模型推理

前面，我们使用了两种构建 TensorRT 模型的方式，分别用 Python 和 C++ 两种语言共生成了四个 TensorRT 模型，这四个模型的功能理论上是完全一致的。
接下来，我们将分别使用 Python 和 C++ 两种语言对生成的 TensorRT 模型进行推理。

#### 使用 Python API 推理

return 0;
}
```

### 模型推理

首先是使用 Python API 推理 TensorRT 模型，这里部分代码参考了 [MMDeploy](https://github.com/open-mmlab/mmdeploy)。运行下面代码，可以发现输入一个 `1x3x224x224` 的张量，输出一个 `1x3x112x112` 的张量，完全符合我们对输入池化后结果的预期。

```python
from typing import Union, Optional, Sequence,Dict,Any

class TRTWrapper(torch.nn.Module):
    def __init__(self,engine: Union[str, trt.ICudaEngine],
                 output_names: Optional[Sequence[str]] = None) -> None:
        self.engine = engine
        if isinstance(self.engine, str):
            with trt.Logger() as logger, trt.Runtime(logger) as runtime:
                with open(self.engine, mode='rb') as f:
                    engine_bytes = f.read()
                self.engine = runtime.deserialize_cuda_engine(engine_bytes)

with open(self.engine, mode='rb') as f:
        self.context = self.engine.create_execution_context()
        names = [_ for _ in self.engine]
        input_names = list(filter(self.engine.binding_is_input, names))
        self._input_names = input_names
        self._output_names = output_names

        if self._output_names is None:
            output_names = list(set(names) - set(input_names))
            self._output_names = output_names

    def forward(self, inputs: Dict[str, torch.Tensor]):
        assert self._input_names is not None
        assert self._output_names is not None
        bindings = [None] * (len(self._input_names) + len(self._output_names))
        profile_id = 0
        for input_name, input_tensor in inputs.items():
            # check if input shape is valid

profile_id = 0
            profile = self.engine.get_profile_shape(profile_id, input_name)
            assert input_tensor.dim() == len(
                profile[0]), 'Input dim is different from engine profile.'
            for s_min, s_input, s_max in zip(profile[0], input_tensor.shape,
                                             profile[2]):
                assert s_min <= s_input <= s_max, \
                    'Input shape should be between ' \
                    + f'{profile[0]} and {profile[2]}' \
                    + f' but get {tuple(input_tensor.shape)}.'
            idx = self.engine.get_binding_index(input_name)

            # All input tensors must be gpu variables
            assert 'cuda' in input_tensor.device.type
            input_tensor = input_tensor.contiguous()
            if input_tensor.dtype == torch.long:

assert 'cuda' in input_tensor.device.type
                input_tensor = input_tensor.int()
            self.context.set_binding_shape(idx, tuple(input_tensor.shape))
            bindings[idx] = input_tensor.contiguous().data_ptr()

        # create output tensors
        outputs = {}
        for output_name in self._output_names:
            idx = self.engine.get_binding_index(output_name)
            dtype = torch.float32
            shape = tuple(self.context.get_binding_shape(idx))

            device = torch.device('cuda')
            output = torch.empty(size=shape, dtype=dtype, device=device)
            outputs[output_name] = output
            bindings[idx] = output.data_ptr()
        self.context.execute_async_v2(bindings,
                                      torch.cuda.current_stream().cuda_stream)

bindings[idx] = output.data_ptr()

model = TRTWrapper('model.engine', ['output'])
output = model(dict(input = torch.randn(1, 3, 224, 224).cuda()))
```

#### 使用 C++ API 推理

最后，在很多实际生产环境中，我们都会使用 C++ 语言完成具体的任务，以达到更加高效的代码运行效果，另外 TensoRT 的用户一般也都更看重其在 C++ 下的使用，所以我们也用 C++ 语言实现一遍模型推理，这也可以和用 Python API 推理模型做一个对比。

实现代码如下：

```cpp

#define CHECK(status) \
    do\
    {\
        auto ret = (status);\
        if (ret != 0)\
        {\
            std::cerr << "Cuda failure: " << ret << std::endl;\
            abort();\
        }\
    } while (0)

}\
    } while (0)

void doInference(IExecutionContext& context, float* input, float* output, int batchSize)
{
        const ICudaEngine& engine = context.getEngine();

        // Pointers to input and output device buffers to pass to engine.
        // Engine requires exactly IEngine::getNbBindings() number of buffers.
        assert(engine.getNbBindings() == 2);
        void* buffers[2];

        // In order to bind the buffers, we need to know the names of the input and output tensors.
        // Note that indices are guaranteed to be less than IEngine::getNbBindings()
        const int inputIndex = engine.getBindingIndex(IN_NAME);

// Note that indices are guaranteed to be less than IEngine::getNbBindings()
        const int outputIndex = engine.getBindingIndex(OUT_NAME);

        // Create GPU buffers on device
        CHECK(cudaMalloc(&buffers[inputIndex], batchSize * 3 * IN_H * IN_W * sizeof(float)));
        CHECK(cudaMalloc(&buffers[outputIndex], batchSize * 3 * IN_H * IN_W /4 * sizeof(float)));

        // Create stream
        cudaStream_t stream;
        CHECK(cudaStreamCreate(&stream));

        // DMA input batch data to device, infer on the batch asynchronously, and DMA output back to host
        CHECK(cudaMemcpyAsync(buffers[inputIndex], input, batchSize * 3 * IN_H * IN_W * sizeof(float), cudaMemcpyHostToDevice, stream));
        context.enqueue(batchSize, buffers, stream, nullptr);
        CHECK(cudaMemcpyAsync(output, buffers[outputIndex], batchSize * 3 * IN_H * IN_W / 4 * sizeof(float), cudaMemcpyDeviceToHost, stream));

CHECK(cudaMemcpyAsync(output, buffers[outputIndex], batchSize * 3 * IN_H * IN_W / 4 * sizeof(float), cudaMemcpyDeviceToHost, stream));
        cudaStreamSynchronize(stream);

        // Release stream and buffers
        cudaStreamDestroy(stream);
        CHECK(cudaFree(buffers[inputIndex]));
        CHECK(cudaFree(buffers[outputIndex]));
}

{
        // create a model using the API directly and serialize it to a stream
        char *trtModelStream{ nullptr };
        size_t size{ 0 };

        std::ifstream file("model.engine", std::ios::binary);
        if (file.good()) {
                file.seekg(0, file.end);
                size = file.tellg();
                file.seekg(0, file.beg);
                trtModelStream = new char[size];
                assert(trtModelStream);
                file.read(trtModelStream, size);
                file.close();
        }

assert(trtModelStream);
        }

        IRuntime* runtime = createInferRuntime(m_logger);
        assert(runtime != nullptr);
        ICudaEngine* engine = runtime->deserializeCudaEngine(trtModelStream, size, nullptr);
        IExecutionContext* context = engine->createExecutionContext();
        assert(context != nullptr);

        // generate input data
        float data[BATCH_SIZE * 3 * IN_H * IN_W];
        for (int i = 0; i < BATCH_SIZE * 3 * IN_H * IN_W; i++)
                data[i] = 1;

        // Run inference
        float prob[BATCH_SIZE * 3 * IN_H * IN_W /4];
        doInference(*context, data, prob, BATCH_SIZE);

        // Destroy the engine
        context->destroy();
        runtime->destroy();
        return 0;
}
```

## 总结

doInference(*context, data, prob, BATCH_SIZE);

        return 0;
}
```

## 总结

通过本文的学习，我们掌握了两种构建 TensorRT 模型的方式：直接通过 TensorRT 的 API 逐层搭建网络；将中间表示的模型转换成 TensorRT 的模型。不仅如此，我们还分别用 C++ 和 Python 两种语言完成了 TensorRT 模型的构建及推理，相信大家都有所收获！在下一篇文章中，我们将和大家一起学习何添加 TensorRT 自定义算子，敬请期待哦~

## FAQ

1. Could not find: cudnn64_8.dll. Is it on your PATH?
   首先检查下自己的环境变量中是否包含 cudnn64_8.dll 所在的路径，若发现 cudnn 的路径在 C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v10.2\\bin 中，但是里面只有 cudnn64_7.dll。解决方法是去 NVIDIA 官网下载 cuDNN zip 包，解压后，复制其中的 cudnn64_8.dll 到 CUDA Toolkit 的 bin 目录下。这时也可以复制一份 cudnn64_7.dll，然后将复制的那份改名成 cudnn64_8.dll，同样可以解决这个问题。

# MoCoV3

> [An Empirical Study of Training Self-Supervised Vision Transformers](https://arxiv.org/abs/2104.02057)

This paper does not describe a novel method. Instead, it studies a straightforward, incremental, yet must-know baseline given the recent progress in computer vision: self-supervised learning for Vision Transformers (ViT). While the training recipes for standard convolutional networks have been highly mature and robust, the recipes for ViT are yet to be built, especially in the self-supervised scenarios where training becomes more challenging. In this work, we go back to basics and investigate the effects of several fundamental components for training self-supervised ViT. We observe that instability is a major issue that degrades accuracy, and it can be hidden by apparently good results. We reveal that these results are indeed partial failure, and they can be improved when training is made more stable. We benchmark ViT results in MoCo v3 and several other self-supervised frameworks, with ablations in various aspects. We discuss the currently positive evidence as well as challenges and open questions. We hope that this work will provide useful data points and experience for future research.

<img  src="https://user-images.githubusercontent.com/36138628/151305362-e6e8ea35-b3b8-45f6-8819-634e67083218.png" width="500" />
</div>

```python

predict = inference_model('resnet50_mocov3-100e-pre_8xb128-linear-coslr-90e_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('mocov3_resnet50_8xb512-amp-coslr-100e_in1k', pretrained=True)
```

Train:

```shell

```

Train:

```shell
python tools/train.py configs/mocov3/mocov3_resnet50_8xb512-amp-coslr-100e_in1k.py
```

Test:

```shell
python tools/test.py configs/mocov3/benchmarks/resnet50_8xb128-linear-coslr-90e_in1k.py https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_resnet50_8xb512-amp-coslr-100e_in1k/resnet50_linear-8xb128-coslr-90e_in1k/resnet50_linear-8xb128-coslr-90e_in1k_20220927-8f7d937e.pth
```

| `mocov3_resnet50_8xb512-amp-coslr-100e_in1k`       |   68.01    |   4.11    |    [config](mocov3_resnet50_8xb512-amp-coslr-100e_in1k.py)    | [model](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_resnet50_8xb512-amp-coslr-100e_in1k/mocov3_resnet50_8xb512-amp-coslr-100e_in1k_20220927-f1144efa.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_resnet50_8xb512-amp-coslr-100e_in1k/mocov3_resnet50_8xb512-amp-coslr-100e_in1k_20220927-f1144efa.json) |

| `mocov3_resnet50_8xb512-amp-coslr-300e_in1k`       |   68.01    |   4.11    |    [config](mocov3_resnet50_8xb512-amp-coslr-300e_in1k.py)    | [model](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_resnet50_8xb512-amp-coslr-300e_in1k/mocov3_resnet50_8xb512-amp-coslr-300e_in1k_20220927-1e4f3304.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_resnet50_8xb512-amp-coslr-300e_in1k/mocov3_resnet50_8xb512-amp-coslr-300e_in1k_20220927-1e4f3304.json) |
| `mocov3_resnet50_8xb512-amp-coslr-800e_in1k`       |   68.01    |   4.11    |    [config](mocov3_resnet50_8xb512-amp-coslr-800e_in1k.py)    | [model](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_resnet50_8xb512-amp-coslr-800e_in1k/mocov3_resnet50_8xb512-amp-coslr-800e_in1k_20220927-e043f51a.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_resnet50_8xb512-amp-coslr-800e_in1k/mocov3_resnet50_8xb512-amp-coslr-800e_in1k_20220927-e043f51a.json) |

| `mocov3_vit-small-p16_16xb256-amp-coslr-300e_in1k` |   84.27    |   4.61    | [config](mocov3_vit-small-p16_16xb256-amp-coslr-300e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_vit-small-p16_16xb256-amp-coslr-300e_in1k/mocov3_vit-small-p16_16xb256-amp-coslr-300e_in1k-224_20220826-08bc52f7.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_vit-small-p16_16xb256-amp-coslr-300e_in1k/mocov3_vit-small-p16_16xb256-amp-coslr-300e_in1k-224_20220826-08bc52f7.json) |

| `mocov3_vit-base-p16_16xb256-amp-coslr-300e_in1k`  |   215.68   |   17.58   | [config](mocov3_vit-base-p16_16xb256-amp-coslr-300e_in1k.py)  | [model](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_vit-base-p16_16xb256-amp-coslr-300e_in1k/mocov3_vit-base-p16_16xb256-amp-coslr-300e_in1k-224_20220826-25213343.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_vit-base-p16_16xb256-amp-coslr-300e_in1k/mocov3_vit-base-p16_16xb256-amp-coslr-300e_in1k-224_20220826-25213343.json) |

| `mocov3_vit-large-p16_64xb64-amp-coslr-300e_in1k`  |   652.78   |   61.60   | [config](mocov3_vit-large-p16_64xb64-amp-coslr-300e_in1k.py)  | [model](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_vit-large-p16_64xb64-amp-coslr-300e_in1k/mocov3_vit-large-p16_64xb64-amp-coslr-300e_in1k-224_20220829-9b88a442.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_vit-large-p16_64xb64-amp-coslr-300e_in1k/mocov3_vit-large-p16_64xb64-amp-coslr-300e_in1k-224_20220829-9b88a442.json) |

| `resnet50_mocov3-100e-pre_8xb128-linear-coslr-90e_in1k` | [MOCOV3 100-Epochs](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_resnet50_8xb512-amp-coslr-100e_in1k/mocov3_resnet50_8xb512-amp-coslr-100e_in1k_20220927-f1144efa.pth) |   25.56    |   4.11    |   69.60   | [config](benchmarks/resnet50_8xb128-linear-coslr-90e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_resnet50_8xb512-amp-coslr-100e_in1k/resnet50_linear-8xb128-coslr-90e_in1k/resnet50_linear-8xb128-coslr-90e_in1k_20220927-8f7d937e.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_resnet50_8xb512-amp-coslr-100e_in1k/resnet50_linear-8xb128-coslr-90e_in1k/resnet50_linear-8xb128-coslr-90e_in1k_20220927-8f7d937e.json) |

| `resnet50_mocov3-300e-pre_8xb128-linear-coslr-90e_in1k` | [MOCOV3 300-Epochs](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_resnet50_8xb512-amp-coslr-300e_in1k/mocov3_resnet50_8xb512-amp-coslr-300e_in1k_20220927-1e4f3304.pth) |   25.56    |   4.11    |   72.80   | [config](benchmarks/resnet50_8xb128-linear-coslr-90e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_resnet50_8xb512-amp-coslr-300e_in1k/resnet50_linear-8xb128-coslr-90e_in1k/resnet50_linear-8xb128-coslr-90e_in1k_20220927-d21ddac2.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_resnet50_8xb512-amp-coslr-300e_in1k/resnet50_linear-8xb128-coslr-90e_in1k/resnet50_linear-8xb128-coslr-90e_in1k_20220927-d21ddac2.json) |

| `resnet50_mocov3-800e-pre_8xb128-linear-coslr-90e_in1k` | [MOCOV3 800-Epochs](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_resnet50_8xb512-amp-coslr-800e_in1k/mocov3_resnet50_8xb512-amp-coslr-800e_in1k_20220927-e043f51a.pth) |   25.56    |   4.11    |   74.40   | [config](benchmarks/resnet50_8xb128-linear-coslr-90e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_resnet50_8xb512-amp-coslr-800e_in1k/resnet50_linear-8xb128-coslr-90e_in1k/resnet50_linear-8xb128-coslr-90e_in1k_20220927-0e97a483.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_resnet50_8xb512-amp-coslr-800e_in1k/resnet50_linear-8xb128-coslr-90e_in1k/resnet50_linear-8xb128-coslr-90e_in1k_20220927-0e97a483.json) |

| `vit-small-p16_mocov3-pre_8xb128-linear-coslr-90e_in1k` | [MOCOV3](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_vit-small-p16_16xb256-amp-coslr-300e_in1k/mocov3_vit-small-p16_16xb256-amp-coslr-300e_in1k-224_20220826-08bc52f7.pth) |   22.05    |   4.61    |   73.60   | [config](benchmarks/vit-small-p16_8xb128-linear-coslr-90e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_vit-small-p16_16xb256-amp-coslr-300e_in1k/vit-small-p16_linear-8xb128-coslr-90e_in1k/vit-small-p16_linear-8xb128-coslr-90e_in1k_20220826-376674ef.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_vit-small-p16_16xb256-amp-coslr-300e_in1k/vit-small-p16_linear-8xb128-coslr-90e_in1k/vit-small-p16_linear-8xb128-coslr-90e_in1k_20220826-376674ef.json) |

| `vit-base-p16_mocov3-pre_8xb64-coslr-150e_in1k` | [MOCOV3](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_vit-base-p16_16xb256-amp-coslr-300e_in1k/mocov3_vit-base-p16_16xb256-amp-coslr-300e_in1k-224_20220826-25213343.pth) |   86.57    |   17.58   |   83.00   | [config](benchmarks/vit-base-p16_8xb64-coslr-150e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_vit-base-p16_16xb256-amp-coslr-300e_in1k/vit-base-p16_ft-8xb64-coslr-150e_in1k/vit-base-p16_ft-8xb64-coslr-150e_in1k_20220826-f1e6c442.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_vit-base-p16_16xb256-amp-coslr-300e_in1k/vit-base-p16_ft-8xb64-coslr-150e_in1k/vit-base-p16_ft-8xb64-coslr-150e_in1k_20220826-f1e6c442.json) |

| `vit-base-p16_mocov3-pre_8xb128-linear-coslr-90e_in1k` | [MOCOV3](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_vit-base-p16_16xb256-amp-coslr-300e_in1k/mocov3_vit-base-p16_16xb256-amp-coslr-300e_in1k-224_20220826-25213343.pth) |   86.57    |   17.58   |   76.90   | [config](benchmarks/vit-base-p16_8xb128-linear-coslr-90e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_vit-base-p16_16xb256-amp-coslr-300e_in1k/vit-base-p16_linear-8xb128-coslr-90e_in1k/vit-base-p16_linear-8xb128-coslr-90e_in1k_20220826-83be7758.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_vit-base-p16_16xb256-amp-coslr-300e_in1k/vit-base-p16_linear-8xb128-coslr-90e_in1k/vit-base-p16_linear-8xb128-coslr-90e_in1k_20220826-83be7758.json) |

| `vit-large-p16_mocov3-pre_8xb64-coslr-100e_in1k` | [MOCOV3](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_vit-large-p16_64xb64-amp-coslr-300e_in1k/mocov3_vit-large-p16_64xb64-amp-coslr-300e_in1k-224_20220829-9b88a442.pth) |   304.33   |   61.60   |   83.70   | [config](benchmarks/vit-large-p16_8xb64-coslr-100e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_vit-large-p16_64xb64-amp-coslr-300e_in1k/vit-large-p16_ft-8xb64-coslr-100e_in1k/vit-large-p16_ft-8xb64-coslr-100e_in1k_20220829-878a2f7f.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mocov3/mocov3_vit-large-p16_64xb64-amp-coslr-300e_in1k/vit-large-p16_ft-8xb64-coslr-100e_in1k/vit-large-p16_ft-8xb64-coslr-100e_in1k_20220829-878a2f7f.json) |

```bibtex
@InProceedings{Chen_2021_ICCV,
    title     = {An Empirical Study of Training Self-Supervised Vision Transformers},
    author    = {Chen, Xinlei and Xie, Saining and He, Kaiming},

```bibtex
    year      = {2021}
}
```

# 分支

本文档旨在全面解释 MMOCR 中每个分支的目的和功能。

## 分支概述

### 1. `main`

`main` 分支是 MMOCR 项目的默认分支。它包含了 MMOCR 的最新稳定版本，目前包含了 MMOCR 1.x（例如 v1.0.0）的代码。`main` 分支确保用户能够使用最新和最可靠的软件版本。

### 2. `dev-1.x`

`dev-1.x` 分支用于开发 MMOCR 的下一个版本。此分支将在发版前进行依赖性测试，通过的提交将会合成到新版本中，并被发布到 `main` 分支。通过设置单独的开发分支，项目可以在不影响 `main` 分支稳定性的情况下继续发展。**所有 PR 应合并到 `dev-1.x` 分支。**

### 3. `0.x`

`0.x` 分支用作 MMOCR 0.x（例如 v0.6.3）的存档。此分支将不再积极接受更新或改进，但它仍可作为历史参考，或供尚未升级到 MMOCR 1.x 的用户使用。

### 4. `1.x`

它是 `main` 分支的别名，旨在实现从兼容性时期平稳过渡。它将在 2023 年的年中删除。

```{note}
分支映射在 2023.04.06 发生了变化。有关旧分支映射和迁移指南，请参阅[分支迁移指南](../migration/branches.md)。
```

# Data flow in MMagic \[Coming Soon!\]

# Downstream tasks

## Detection

For detection tasks, please use MMDetection. First, make sure you have installed [MIM](https://github.com/open-mmlab/mim), which is also a project of OpenMMLab.

```shell
mim install 'mmdet>=3.0.0rc0'
```

Besides, please refer to MMDet for [installation](https://mmdetection.readthedocs.io/en/dev-3.x/get_started.html) and [data preparation](https://mmdetection.readthedocs.io/en/dev-3.x/user_guides/dataset_prepare.html)

### Train

After installation, you can run MMDetection with simple command.

```shell
# distributed version
bash tools/benchmarks/mmdetection/mim_dist_train_c4.sh ${CONFIG} ${PRETRAIN} ${GPUS}
bash tools/benchmarks/mmdetection/mim_dist_train_fpn.sh ${CONFIG} ${PRETRAIN} ${GPUS}

# slurm version
bash tools/benchmarks/mmdetection/mim_slurm_train_c4.sh ${PARTITION} ${CONFIG} ${PRETRAIN}
bash tools/benchmarks/mmdetection/mim_slurm_train_fpn.sh ${PARTITION} ${CONFIG} ${PRETRAIN}
```

```

- `${CONFIG}`: Use config file path in MMDetection directly. And for some algorithms, we also have some
  modified config files which can be found in the `benchmarks` folder under the correspondding algorithm
  folder. You can also writing your config file from scratch.
- `${PRETRAIN}`: the pre-trained model file.
- `${GPUS}`: The number of GPUs that you want to use to train. We adopt 8 GPUs for detection tasks by default.

Example:

```shell
bash ./tools/benchmarks/mmdetection/mim_dist_train_c4.sh \
  configs/byol/benchmarks/mask-rcnn_r50-c4_ms-1x_coco.py \
  https://download.openmmlab.com/mmselfsup/1.x/byol/byol_resnet50_16xb256-coslr-200e_in1k/byol_resnet50_16xb256-coslr-200e_in1k_20220825-de817331.pth 8
```

### Test

After training, you can also run the command below to test your model.

```shell

```

### Test

```shell
bash tools/benchmarks/mmdetection/mim_dist_test.sh ${CONFIG} ${CHECKPOINT} ${GPUS}

bash tools/benchmarks/mmdetection/mim_slurm_test.sh ${PARTITION} ${CONFIG} ${CHECKPOINT}
```

- `${CONFIG}`: Use config file name in MMDetection directly. And for some algorithms, we also have some
- `${CHECKPOINT}`: The fine-tuned detection model that you want to test.
- `${GPUS}`: The number of GPUs that you want to use to test. We adopt 8 GPUs for detection tasks by default.

Example:

```shell
bash ./tools/benchmarks/mmdetection/mim_dist_test.sh \
configs/byol/benchmarks/mask-rcnn_r50_fpn_ms-1x_coco.py \

Example:

```shell
https://download.openmmlab.com/mmselfsup/1.x/byol/byol_resnet50_16xb256-coslr-200e_in1k/byol_resnet50_16xb256-coslr-200e_in1k_20220825-de817331.pth 8
```

## Segmentation

For semantic segmentation task, we use MMSegmentation. First, make sure you have installed [MIM](https://github.com/open-mmlab/mim), which is also a project of OpenMMLab.

```shell
mim install 'mmsegmentation>=1.0.0rc0'
```

Besides, please refer to MMSegmentation for [installation](https://mmsegmentation.readthedocs.io/en/dev-1.x/get_started.html) and [data preparation](https://mmsegmentation.readthedocs.io/en/dev-1.x/user_guides/2_dataset_prepare.html).

### Train

After installation, you can run MMSegmentation with simple command.

```shell
bash tools/benchmarks/mmsegmentation/mim_dist_train.sh ${CONFIG} ${PRETRAIN} ${GPUS}

```shell

bash tools/benchmarks/mmsegmentation/mim_slurm_train.sh ${PARTITION} ${CONFIG} ${PRETRAIN}
```

- `${CONFIG}`: Use config file path in MMSegmentation directly. And for some algorithms, we also have some
- `${GPUS}`: The number of GPUs that you want to use to train. We adopt 4 GPUs for segmentation tasks by default.

Example:

```shell
bash ./tools/benchmarks/mmsegmentation/mim_dist_train.sh \
configs/benchmarks/mmsegmentation/voc12aug/fcn_r50-d8_4xb4-20k_voc12aug-512x512.py \
https://download.openmmlab.com/mmselfsup/1.x/byol/byol_resnet50_16xb256-coslr-200e_in1k/byol_resnet50_16xb256-coslr-200e_in1k_20220825-de817331.pth 4
```

### Test

```

### Test

```shell
bash tools/benchmarks/mmsegmentation/mim_dist_test.sh ${CONFIG} ${CHECKPOINT} ${GPUS}

bash tools/benchmarks/mmsegmentation/mim_slurm_test.sh ${PARTITION} ${CONFIG} ${CHECKPOINT}
```

- `${CONFIG}`: Use config file name in MMSegmentation directly. And for some algorithms, we also have some
- `${CHECKPOINT}`: The fine-tuned segmentation model that you want to test.
- `${GPUS}`: The number of GPUs that you want to use to test. We adopt 4 GPUs for segmentation tasks by default.

Example:

```shell

Example:

```shell
bash ./tools/benchmarks/mmsegmentation/mim_dist_test.sh  fcn_r50-d8_4xb4-20k_voc12aug-512x512.py \
```

# RGBPoseConv3D

RGBPoseConv3D is a framework that jointly use 2D human skeletons and RGB appearance for human action recognition. It is a 3D CNN with two streams, with the architecture borrowed from SlowFast. In RGBPoseConv3D:

- The RGB stream corresponds to the `slow` stream in SlowFast; The Skeleton stream corresponds to the `fast` stream in SlowFast.
- The input resolution of RGB frames is `4x` larger than the pseudo heatmaps.
- Bilateral connections are used for early feature fusion between the two modalities.

<img src="https://user-images.githubusercontent.com/34324155/209961351-6def0074-9b05-43fc-8210-a1cdaaed6536.png" width=70%/>
</div>

```BibTeX
@inproceedings{duan2022revisiting,
  title={Revisiting skeleton-based action recognition},
  author={Duan, Haodong and Zhao, Yue and Chen, Kai and Lin, Dahua and Dai, Bo},
  pages={2969--2978},
}
```

title={Revisiting skeleton-based action recognition},
}
```

## How to train RGBPoseConv3D (on NTURGB+D, for example)?

#### Step 0. Data Preparation

Besides the skeleton annotations, you also need RGB videos to train RGBPoseConv3D. You need to download them from the official website of [NTURGB+D](https://rose1.ntu.edu.sg/dataset/actionRecognition/) and put these videos in `$MMACTION2/data/nturgbd_raw`. After that, you need to use the provided script to compress the raw videos (from `1920x1080` to `960x540`) and switch the suffix to `.mp4`:

```bash
# That step is mandatory, unless you know how to modify the code & config to make it work for raw videos!
python tools/data/skeleton/compress_nturgbd.py
```

After that, you will find processed videos in `$MMACTION2/data/nturgbd_videos`, named like `S001C001P001R001A001.mp4`.

#### Step 1. Pretraining

```

You first need to train the RGB-only and Pose-only model on the target dataset, the pretrained checkpoints will be used to initialize the RGBPoseConv3D model.

You can either train these two models from scratch with provided configs files:

```bash
# We train each model for 180 epochs. By default, we use 8 GPUs.
# Train the RGB-only model
bash tools/dist_train.sh configs/skeleton/posec3d/rgbpose_conv3d/rgb_only.py 8
# Train the Pose-only model
bash tools/dist_train.sh configs/skeleton/posec3d/rgbpose_conv3d/pose_only.py 8
```

or directly download and use the provided pretrain models:

```

|    Dataset    |                                Config                                |                                 Checkpoint                                 | Top-1 (1 clip testing) | Top-1 (10 clip testing) |
| :-----------: | :------------------------------------------------------------------: | :------------------------------------------------------------------------: | :--------------------: | :---------------------: |
| NTURGB+D XSub |  [rgb_config](/configs/skeleton/posec3d/rgbpose_conv3d/rgb_only.py)  | [rgb_ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/posec3d/rgbpose_conv3d/rgb_only_20230228-576b9f86.pth) |          94.9          |          95.4           |

| NTURGB+D XSub | [pose_config](/configs/skeleton/posec3d/rgbpose_conv3d/pose_only.py) | [pose_ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/posec3d/rgbpose_conv3d/pose_only_20230228-fa40054e.pth) |          93.1          |          93.5           |

#### Step 2. Generate the initializing weight for RGBPoseConv3D

You can use the provided [IPython notebook](/configs/skeleton/posec3d/rgbpose_conv3d/merge_pretrain.ipynb) to merge two pretrained models into a single `rgbpose_conv3d_init.pth`.

You can do it your own or directly download and use the provided [rgbpose_conv3d_init.pth](https://download.openmmlab.com/mmaction/v1.0/skeleton/posec3d/rgbpose_conv3d/rgbpose_conv3d_init_20230228-09b7684b.pth).

#### Step 3. Finetune RGBPoseConv3D

You can use our provided config files to finetune RGBPoseConv3D, jointly with two modalities (RGB & Pose):

```bash
# We finetune RGBPoseConv3D for 20 epochs on NTURGB+D XSub (8 GPUs)

```bash
bash tools/dist_train.sh configs/skeleton/posec3d/rgbpose_conv3d/rgbpose_conv3d.py 8
# After finetuning, you can test the model with the following command (8 GPUs)
bash tools/dist_test.sh configs/skeleton/posec3d/rgbpose_conv3d/rgbpose_conv3d.py $CKPT 8 --dump result.pkl
```

**Notes**

1. We use linear scaling learning rate (`Initial LR` ∝ `Batch Size`). If you change the training batch size, remember to change the initial LR proportionally.

2. Though optimized, multi-clip testing may consumes large amounts of time. For faster inference, you may change the test_pipeline to disable the multi-clip testing, this may lead to a small drop in recognition performance. Below is the guide:

   ```python
   test_pipeline = [

```python
       dict(type='MMUniformSampleFrames', clip_len=dict(RGB=8, Pose=32), num_clips=10, test_mode=True), # change `num_clips=10` to `num_clips=1`
       dict(type='MMDecode'),
       dict(type='MMCompact', hw_ratio=1., allow_imgpad=True),
       dict(type='Resize', scale=(256, 256), keep_ratio=False),
       dict(type='GeneratePoseTarget', sigma=0.7, use_score=True, with_kp=True, with_limb=False, scaling=0.25),
       dict(type='FormatShape', input_format='NCTHW'),
       dict(type='PackActionInputs', collect_keys=('imgs', 'heatmap_imgs'))
   ]
   ```

## Results

On action recognition with multiple modalities (RGB & Pose), RGBPoseConv3D can achieve better recognition performance than the late fusion baseline.

|    Dataset    |        Fusion         |        Config         |         Checkpoint         | RGB Stream Top-1<br>(1-clip / 10-clip) | Pose Stream Top-1<br>(1-clip / 10-clip) | 2 Stream Top-1 (1:1)<br>(1-clip / 10-clip) |

| :-----------: | :-------------------: | :-------------------: | :------------------------: | :------------------------------------: | :-------------------------------------: | :----------------------------------------: |
| NTURGB+D XSub |      Late Fusion      | [rgb_config](/configs/skeleton/posec3d/rgbpose_conv3d/rgb_only.py)<br>[pose_config](/configs/skeleton/posec3d/rgbpose_conv3d/pose_only.py) | [rgb_ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/posec3d/rgbpose_conv3d/rgb_only_20230228-576b9f86.pth)<br>[pose_ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/posec3d/rgbpose_conv3d/pose_only_20230228-fa40054e.pth) |              94.9 / 95.4               |               93.1 / 93.5               |                96.0 / 96.2                 |

| NTURGB+D XSub | Early Fusion + Late Fusion | [config](/configs/skeleton/posec3d/rgbpose_conv3d/rgbpose_conv3d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/posec3d/rgbpose_conv3d/rgbpose_conv3d_20230301-ac7b0e77.pth) |              96.2 / 96.4               |               96.0 / 96.2               |                96.6 / 96.8                 |

**Notes**

For both `Late Fusion` and `Early Fusion + Late Fusion`, we combine the action scores based on two modalities with 1:1 ratio to get the final prediction.

# MVX-Net: Multimodal VoxelNet for 3D Object Detection

> [MVX-Net: Multimodal VoxelNet for 3D Object Detection](https://arxiv.org/abs/1904.01649)

Many recent works on 3D object detection have focused on designing neural network architectures that can consume point cloud data. While these approaches demonstrate encouraging performance, they are typically based on a single modality and are unable to leverage information from other modalities, such as a camera. Although a few approaches fuse data from different modalities, these methods either use a complicated pipeline to process the modalities sequentially, or perform late-fusion and are unable to learn interaction between different modalities at early stages. In this work, we present PointFusion and VoxelFusion: two simple yet effective early-fusion approaches to combine the RGB and point cloud modalities, by leveraging the recently introduced VoxelNet architecture. Evaluation on the KITTI dataset demonstrates significant improvements in performance over approaches which only use point cloud data. Furthermore, the proposed method provides results competitive with the state-of-the-art multimodal algorithms, achieving top-2 ranking in five of the six bird's eye view and 3D detection categories on the KITTI benchmark, by using a simple single stage network.

<img src="https://user-images.githubusercontent.com/79644370/143880819-560675ca-e7e3-4d77-8808-ea661ff8e6e6.png" width="800"/>
</div>

We implement MVX-Net and provide its results and models on KITTI dataset.

### KITTI

|                              Backbone                               |  Class  |  Lr schd   | Mem (GB) | Inf time (fps) | mAP  |                                                                                                                                                                                                  Download                                                                                                                                                                                                   |

| [SECFPN](./mvxnet_fpn_dv_second_secfpn_8xb2-80e_kitti-3d-3class.py) | 3 Class | cosine 80e |   6.7    |                | 63.5 | [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/mvxnet/mvxnet_fpn_dv_second_secfpn_8xb2-80e_kitti-3d-3class/mvxnet_fpn_dv_second_secfpn_8xb2-80e_kitti-3d-3class-8963258a.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/mvxnet/mvxnet_fpn_dv_second_secfpn_8xb2-80e_kitti-3d-3class/mvxnet_fpn_dv_second_secfpn_8xb2-80e_kitti-3d-3class-20230424_132228.log) |

```latex
@inproceedings{sindagi2019mvx,
  title={MVX-Net: Multimodal voxelnet for 3D object detection},
  author={Sindagi, Vishwanath A and Zhou, Yin and Tuzel, Oncel},
  booktitle={2019 International Conference on Robotics and Automation (ICRA)},
  pages={7276--7282},
}
```

# HiViT

> [HiViT: A Simple and More Efficient Design of Hierarchical Vision Transformer](https://arxiv.org/abs/2205.14949)

Recently, masked image modeling (MIM) has offered a new methodology of self-supervised pre-training of vision transformers. A key idea of efficient implementation is to discard the masked image patches (or tokens) throughout the target network (encoder), which requires the encoder to be a plain vision transformer (e.g., ViT), albeit hierarchical vision transformers (e.g., Swin Transformer) have potentially better properties in formulating vision inputs. In this paper, we offer a new design of hierarchical vision transformers named HiViT (short for Hierarchical ViT) that enjoys both high efficiency and good performance in MIM. The key is to remove the unnecessary "local inter-unit operations", deriving structurally simple hierarchical vision transformers in which mask-units can be serialized like plain vision transformers. For this purpose, we start with Swin Transformer and (i) set the masking unit size to be the token size in the main stage of Swin Transformer, (ii) switch off inter-unit self-attentions before the main stage, and (iii) eliminate all operations after the main stage. Empirical studies demonstrate the advantageous performance of HiViT in terms of fully-supervised, self-supervised, and transfer learning. In particular, in running MAE on ImageNet-1K, HiViT-B reports a +0.6% accuracy gain over ViT-B and a 1.9$\times$ speed-up over Swin-B, and the performance gain generalizes to downstream tasks of detection and segmentation. Code will be made publicly available.

<img src="https://github.com/open-mmlab/mmpretrain/assets/36138628/4a99cf9d-15df-4866-8750-bd2c3db5d894" width="80%"/>
</div>

<!-- **Predict image**

```python

predict = inference_model('hivit-tiny-p16_16xb64_in1k', 'demo/bird.JPEG')
```

<!-- **Use the model** -->

<!-- ```python

model = get_model('hivit-tiny-p16_16xb64_in1k', pretrained=True)
``` -->

Train:

```shell
python tools/train.py configs/hivit/hivit-tiny-p16_16xb64_in1k.py
```

<!-- Test:

```shell

Train:

```shell
```

<!-- Test:

```shell
python tools/test.py configs/hivit/hivit-tiny-p16_16xb64_in1k.py None
``` -->

| Model                         |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) |                  Config                  | Download |
| :---------------------------- | :----------: | :--------: | :-------: | :-------: | :--------------------------------------: | :------: |
| `hivit-tiny-p16_16xb64_in1k`  | From scratch |   19.18    |   4.60    |   82.10   | [config](hivit-tiny-p16_16xb64_in1k.py)  |   N/A    |
| `hivit-small-p16_16xb64_in1k` | From scratch |   37.53    |   9.07    |    N/A    | [config](hivit-small-p16_16xb64_in1k.py) |   N/A    |
| `hivit-base-p16_16xb64_in1k`  | From scratch |   79.05    |   18.47   |    N/A    | [config](hivit-base-p16_16xb64_in1k.py)  |   N/A    |

```bibtex

```bibtex
@inproceedings{zhanghivit,
  title={HiViT: A Simpler and More Efficient Design of Hierarchical Vision Transformer},
  author={Zhang, Xiaosong and Tian, Yunjie and Xie, Lingxi and Huang, Wei and Dai, Qi and Ye, Qixiang and Tian, Qi},
}
```

## Get Model Params & FLOPs

MMPose provides `tools/analysis_tools/get_flops.py` to get model parameters and FLOPs.

```shell
```

`CONFIG_FILE` : The path of a model config file.

`--shape`: The input shape to the model.

`--input-constructor`: If specified as batch, it will generate a batch tensor to calculate FLOPs.

`--batch-size`：If `--input-constructor` is specified as batch, it will generate a random tensor with shape `(batch_size, 3, **input_shape)` to calculate FLOPs.

`--cfg-options`: If specified, the key-value pair optional `cfg` will be merged into config file.

Example:

```shell
```

We will get the following results:

```text

```

```text
```

```{note}
This tool is still experimental and we do not guarantee that the number is absolutely correct. Some operators are not counted into FLOPs like GN and custom operators.
```

MMPose provides `tools/analysis_tools/analyze_logs.py` to analyze the training log. The log file can be either a json file or a text file. The json file is recommended, because it is more convenient to parse and visualize.

Currently, the following functions are supported:

- Plot loss/accuracy curves
- Calculate training time

### Plot Loss/Accuracy Curves

The function depends on `seaborn`, please install it first by running `pip install seaborn`.

```shell

```shell
```

Examples:

- Plot loss curve

  ```shell
  ```

- Plot accuracy curve and export to PDF file

  ```shell
  ```

- Plot multiple log files on the same figure

  ```shell
  ```

### Calculate Training Time

```shell
```

Examples:

```shell

```

```shell
```

Examples:

```shell
```

```text
```

<details>

```bibtex
}
```

<details>

```bibtex

| [pose_seresnet_50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_seresnet50_8xb64-210e_coco-256x192.py) |  256x192   | 0.729 |      0.903      |      0.807      | 0.784 |      0.941      | [ckpt](https://download.openmmlab.com/mmpose/top_down/seresnet/seresnet50_coco_256x192-25058b66_20200727.pth) | [log](https://download.openmmlab.com/mmpose/top_down/seresnet/seresnet50_coco_256x192_20200727.log.json) |
| [pose_seresnet_50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_seresnet50_8xb64-210e_coco-384x288.py) |  384x288   | 0.748 |      0.904      |      0.819      | 0.799 |      0.941      | [ckpt](https://download.openmmlab.com/mmpose/top_down/seresnet/seresnet50_coco_384x288-bc0b7680_20200727.pth) | [log](https://download.openmmlab.com/mmpose/top_down/seresnet/seresnet50_coco_384x288_20200727.log.json) |

| [pose_seresnet_101](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_seresnet101_8xb64-210e_coco-256x192.py) |  256x192   | 0.734 |      0.905      |      0.814      | 0.790 |      0.941      | [ckpt](https://download.openmmlab.com/mmpose/top_down/seresnet/seresnet101_coco_256x192-83f29c4d_20200727.pth) | [log](https://download.openmmlab.com/mmpose/top_down/seresnet/seresnet101_coco_256x192_20200727.log.json) |
| [pose_seresnet_101](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_seresnet101_8xb32-210e_coco-384x288.py) |  384x288   | 0.754 |      0.907      |      0.823      | 0.805 |      0.943      | [ckpt](https://download.openmmlab.com/mmpose/top_down/seresnet/seresnet101_coco_384x288-48de1709_20200727.pth) | [log](https://download.openmmlab.com/mmpose/top_down/seresnet/seresnet101_coco_384x288_20200727.log.json) |

| [pose_seresnet_152\*](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_seresnet152_8xb32-210e_coco-256x192.py) |  256x192   | 0.730 |      0.899      |      0.810      | 0.787 |      0.939      | [ckpt](https://download.openmmlab.com/mmpose/top_down/seresnet/seresnet152_coco_256x192-1c628d79_20200727.pth) | [log](https://download.openmmlab.com/mmpose/top_down/seresnet/seresnet152_coco_256x192_20200727.log.json) |
| [pose_seresnet_152\*](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_seresnet152_8xb48-210e_coco-384x288.py) |  384x288   | 0.753 |      0.906      |      0.824      | 0.806 |      0.945      | [ckpt](https://download.openmmlab.com/mmpose/top_down/seresnet/seresnet152_coco_384x288-58b23ee8_20200727.pth) | [log](https://download.openmmlab.com/mmpose/top_down/seresnet/seresnet152_coco_384x288_20200727.log.json) |

# Hand keypoint detection in single images using multiview bootstrapping

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_cvpr_2017/html/Simon_Hand_Keypoint_Detection_CVPR_2017_paper.html">CMU Panoptic HandDB (CVPR'2017)</a></summary>

```bibtex
@inproceedings{simon2017hand,
  title={Hand keypoint detection in single images using multiview bootstrapping},
  author={Simon, Tomas and Joo, Hanbyul and Matthews, Iain and Sheikh, Yaser},
  pages={1145--1153},
}
```

# EncNet

> [Context Encoding for Semantic Segmentation](https://arxiv.org/abs/1803.08904)

<a href="https://github.com/zhanghang1989/PyTorch-Encoding">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/decode_heads/enc_head.py#L63">Code Snippet</a>

Recent work has made significant progress in improving spatial resolution for pixelwise labeling with Fully Convolutional Network (FCN) framework by employing Dilated/Atrous convolution, utilizing multi-scale features and refining boundaries. In this paper, we explore the impact of global contextual information in semantic segmentation by introducing the Context Encoding Module, which captures the semantic context of scenes and selectively highlights class-dependent featuremaps. The proposed Context Encoding Module significantly improves semantic segmentation results with only marginal extra computation cost over FCN. Our approach has achieved new state-of-the-art results 51.7% mIoU on PASCAL-Context, 85.9% mIoU on PASCAL VOC 2012. Our single model achieves a final score of 0.5567 on ADE20K test set, which surpass the winning entry of COCO-Place Challenge in 2017. In addition, we also explore how the Context Encoding Module can improve the feature representation of relatively shallow networks for the image classification on CIFAR-10 dataset. Our 14 layer network has achieved an error rate of 3.45%, which is comparable with state-of-the-art approaches with over 10 times more layers. The source code for the complete system are publicly available.

<img src="https://user-images.githubusercontent.com/24582831/142901276-b364fbbf-3bdb-4000-9d31-b9a135e30935.png" width="70%"/>
</div>

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                         | download                                                                                                                                                                                                                                                                                                                                                 |

| EncNet | R-50-D8  | 512x1024  |   40000 | 8.6      | 4.58           | V100   | 75.67 |         77.08 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/encnet/encnet_r50-d8_4xb2-40k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r50-d8_512x1024_40k_cityscapes/encnet_r50-d8_512x1024_40k_cityscapes_20200621_220958-68638a47.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r50-d8_512x1024_40k_cityscapes/encnet_r50-d8_512x1024_40k_cityscapes-20200621_220958.log.json)     |

| EncNet | R-101-D8 | 512x1024  |   40000 | 12.1     | 2.66           | V100   | 75.81 |         77.21 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/encnet/encnet_r101-d8_4xb2-40k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r101-d8_512x1024_40k_cityscapes/encnet_r101-d8_512x1024_40k_cityscapes_20200621_220933-35e0a3e8.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r101-d8_512x1024_40k_cityscapes/encnet_r101-d8_512x1024_40k_cityscapes-20200621_220933.log.json) |

| EncNet | R-50-D8  | 769x769   |   40000 | 9.8      | 1.82           | V100   | 76.24 |         77.85 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/encnet/encnet_r50-d8_4xb2-40k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r50-d8_769x769_40k_cityscapes/encnet_r50-d8_769x769_40k_cityscapes_20200621_220958-3bcd2884.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r50-d8_769x769_40k_cityscapes/encnet_r50-d8_769x769_40k_cityscapes-20200621_220958.log.json)         |

| EncNet | R-101-D8 | 769x769   |   40000 | 13.7     | 1.26           | V100   | 74.25 |         76.25 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/encnet/encnet_r101-d8_4xb2-40k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r101-d8_769x769_40k_cityscapes/encnet_r101-d8_769x769_40k_cityscapes_20200621_220933-2fafed55.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r101-d8_769x769_40k_cityscapes/encnet_r101-d8_769x769_40k_cityscapes-20200621_220933.log.json)     |

| EncNet | R-50-D8  | 512x1024  |   80000 | -        | -              | V100   | 77.94 |         79.13 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/encnet/encnet_r50-d8_4xb2-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r50-d8_512x1024_80k_cityscapes/encnet_r50-d8_512x1024_80k_cityscapes_20200622_003554-fc5c5624.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r50-d8_512x1024_80k_cityscapes/encnet_r50-d8_512x1024_80k_cityscapes-20200622_003554.log.json)     |

| EncNet | R-101-D8 | 512x1024  |   80000 | -        | -              | V100   | 78.55 |         79.47 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/encnet/encnet_r101-d8_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r101-d8_512x1024_80k_cityscapes/encnet_r101-d8_512x1024_80k_cityscapes_20200622_003555-1de64bec.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r101-d8_512x1024_80k_cityscapes/encnet_r101-d8_512x1024_80k_cityscapes-20200622_003555.log.json) |

| EncNet | R-50-D8  | 769x769   |   80000 | -        | -              | V100   | 77.44 |         78.72 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/encnet/encnet_r50-d8_4xb2-80k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r50-d8_769x769_80k_cityscapes/encnet_r50-d8_769x769_80k_cityscapes_20200622_003554-55096dcb.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r50-d8_769x769_80k_cityscapes/encnet_r50-d8_769x769_80k_cityscapes-20200622_003554.log.json)         |

| EncNet | R-101-D8 | 769x769   |   80000 | -        | -              | V100   | 76.10 |         76.97 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/encnet/encnet_r101-d8_4xb2-80k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r101-d8_769x769_80k_cityscapes/encnet_r101-d8_769x769_80k_cityscapes_20200622_003555-470ef79d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r101-d8_769x769_80k_cityscapes/encnet_r101-d8_769x769_80k_cityscapes-20200622_003555.log.json)     |

### ADE20K

| EncNet | R-50-D8  | 512x512   |   80000 | 10.1     | 22.81          | V100   | 39.53 |         41.17 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/encnet/encnet_r50-d8_4xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r50-d8_512x512_80k_ade20k/encnet_r50-d8_512x512_80k_ade20k_20200622_042412-44b46b04.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r50-d8_512x512_80k_ade20k/encnet_r50-d8_512x512_80k_ade20k-20200622_042412.log.json)         |

| EncNet | R-101-D8 | 512x512   |   80000 | 13.6     | 14.87          | V100   | 42.11 |         43.61 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/encnet/encnet_r101-d8_4xb4-80k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r101-d8_512x512_80k_ade20k/encnet_r101-d8_512x512_80k_ade20k_20200622_101128-dd35e237.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r101-d8_512x512_80k_ade20k/encnet_r101-d8_512x512_80k_ade20k-20200622_101128.log.json)     |

| EncNet | R-50-D8  | 512x512   |  160000 | -        | -              | V100   | 40.10 |         41.71 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/encnet/encnet_r50-d8_4xb4-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r50-d8_512x512_160k_ade20k/encnet_r50-d8_512x512_160k_ade20k_20200622_101059-b2db95e0.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r50-d8_512x512_160k_ade20k/encnet_r50-d8_512x512_160k_ade20k-20200622_101059.log.json)     |

| EncNet | R-101-D8 | 512x512   |  160000 | -        | -              | V100   | 42.61 |         44.01 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/encnet/encnet_r101-d8_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r101-d8_512x512_160k_ade20k/encnet_r101-d8_512x512_160k_ade20k_20200622_073348-7989641f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/encnet/encnet_r101-d8_512x512_160k_ade20k/encnet_r101-d8_512x512_160k_ade20k-20200622_073348.log.json) |

```bibtex
@InProceedings{Zhang_2018_CVPR,
author = {Zhang, Hang and Dana, Kristin and Shi, Jianping and Zhang, Zhongyue and Wang, Xiaogang and Tyagi, Ambrish and Agrawal, Amit},
title = {Context Encoding for Semantic Segmentation},
}
```

# 可视化

在阅读本教程之前，建议先阅读 MMEngine 的 [Visualization](https://github.com/open-mmlab/mmengine/blob/main/docs/en/advanced_tutorials/visualization.md) 文档，以对 `Visualizer` 的定义和用法有一个初步的了解。

简而言之，`Visualizer` 在 MMEngine 中实现以满足日常可视化需求，并包含以下三个主要功能：

- 实现通用的绘图 API，例如 [`draw_bboxes`](mmengine.visualization.Visualizer.draw_bboxes) 实现了绘制边界框的功能，[`draw_lines`](mmengine.visualization.Visualizer.draw_lines) 实现了绘制线条的功能。
- 支持将可视化结果、学习率曲线、损失函数曲线以及验证精度曲线写入到各种后端中，包括本地磁盘以及常见的深度学习训练日志工具，例如 [TensorBoard](https://www.tensorflow.org/tensorboard) 和 [Wandb](https://wandb.ai/site)。
- 支持在代码的任何位置调用以可视化或记录模型在训练或测试期间的中间状态，例如特征图和验证结果。

基于 MMEngine 的 `Visualizer`，MMDet 提供了各种预构建的可视化工具，用户可以通过简单地修改以下配置文件来使用它们。

- `tools/analysis_tools/browse_dataset.py` 脚本提供了一个数据集可视化功能，可以在数据经过数据转换后绘制图像和相应的注释，具体描述请参见[`browse_dataset.py`](useful_tools.md#Visualization)。

- MMEngine实现了`LoggerHook`，使用`Visualizer`将学习率、损失和评估结果写入由`Visualizer`设置的后端。因此，通过修改配置文件中的`Visualizer`后端，例如修改为`TensorBoardVISBackend`或`WandbVISBackend`，可以实现日志记录到常用的训练日志工具，如`TensorBoard`或`WandB`，从而方便用户使用这些可视化工具来分析和监控训练过程。

- 在MMDet中实现了`VisualizerHook`，它使用`Visualizer`将验证或预测阶段的预测结果可视化或存储到由`Visualizer`设置的后端。因此，通过修改配置文件中的`Visualizer`后端，例如修改为`TensorBoardVISBackend`或`WandbVISBackend`，可以将预测图像存储到`TensorBoard`或`Wandb`中。

## 配置

由于使用了注册机制，在MMDet中我们可以通过修改配置文件来设置`Visualizer`的行为。通常，我们会在`configs/_base_/default_runtime.py`中为可视化器定义默认配置，详细信息请参见[配置教程](config.md)。

```Python
```

基于上面的例子，我们可以看到`Visualizer`的配置由两个主要部分组成，即`Visualizer`类型和其使用的可视化后端`vis_backends`。

- 用户可直接使用`DetLocalVisualizer`来可视化支持任务的标签或预测结果。

vis_backends=vis_backends,
```

- MMDet默认将可视化后端`vis_backend`设置为本地可视化后端`LocalVisBackend`，将所有可视化结果和其他训练信息保存在本地文件夹中。

## 存储

MMDet默认使用本地可视化后端[`LocalVisBackend`](mmengine.visualization.LocalVisBackend)，`VisualizerHook`和`LoggerHook`中存储的模型损失、学习率、模型评估精度和可视化信息，包括损失、学习率、评估精度将默认保存到`{work_dir}/{config_name}/{time}/{vis_data}`文件夹中。此外，MMDet还支持其他常见的可视化后端，例如`TensorboardVisBackend`和`WandbVisBackend`，您只需要在配置文件中更改`vis_backends`类型为相应的可视化后端即可。例如，只需在配置文件中插入以下代码块即可将数据存储到`TensorBoard`和`Wandb`中。

```Python
# https://mmengine.readthedocs.io/en/latest/api/visualization.html
    dict(type='LocalVisBackend'), #
```

## 绘图

### 绘制预测结果

```

## 绘图

### 绘制预测结果

MMDet主要使用[`DetVisualizationHook`](mmdet.engine.hooks.DetVisualizationHook)来绘制验证和测试的预测结果，默认情况下`DetVisualizationHook`是关闭的，其默认配置如下。

```Python
visualization=dict( #用户可视化验证和测试结果
    type='DetVisualizationHook',
    draw=False,
    show=False)
```

以下表格展示了`DetVisualizationHook`支持的参数。

|   参数   |                                       描述                                       |
| :------: | :------------------------------------------------------------------------------: |
|   draw   |          DetVisualizationHook通过enable参数打开和关闭，默认状态为关闭。          |
| interval | 控制在DetVisualizationHook启用时存储或显示验证或测试结果的间隔，单位为迭代次数。 |
|   show   |                         控制是否可视化验证或测试的结果。                         |

如果您想在训练或测试期间启用 `DetVisualizationHook` 相关功能和配置，您只需要修改配置文件，以 `configs/rtmdet/rtmdet_tiny_8xb32-300e_coco.py` 为例，同时绘制注释和预测，并显示图像，配置文件可以修改如下：

```Python
visualization.update(dict(draw=True, show=True))
```

<img src="https://user-images.githubusercontent.com/17425982/224883427-1294a7ba-14ab-4d93-9152-55a7b270b1f1.png" height="300"/>
</div>

`test.py`程序提供了`--show`和`--show-dir`参数，可以在测试过程中可视化注释和预测结果，而不需要修改配置文件，从而进一步简化了测试过程。

```Shell
# 展示测试结果
python tools/test.py configs/rtmdet/rtmdet_tiny_8xb32-300e_coco.py https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet_tiny_8xb32-300e_coco/rtmdet_tiny_8xb32-300e_coco_20220902_112414-78e30dcc.pth --show

# 指定存储预测结果的位置

python tools/test.py configs/rtmdet/rtmdet_tiny_8xb32-300e_coco.py https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet_tiny_8xb32-300e_coco/rtmdet_tiny_8xb32-300e_coco_20220902_112414-78e30dcc.pth --show-dir imgs/
```

</div>

# 准备 SIDD 数据集

```bibtex
}
```

训练数据集可以从 [此处](https://drive.google.com/file/d/1UHjWZzLPGweA9ZczmV8lFSRcIxqiOVJw/) 下载。验证数据集可以从 [此处](https://drive.google.com/file/d/1Fw6Ey1R-nCHN9WEpxv0MnMqxij-ECQYJ/) 下载。测试数据集可以从 [此处](https://drive.google.com/file/d/11vfqV-lqousZTuAit1Qkqghiv_taY0KZ/) 下载。

测试数据集需要从 mat 文件中导出，为此，我们提供了一个脚本：

```shell
python tools/dataset_converters/sidd/preprocess_sidd_test_dataset.py --data-root ./data/SIDD/test --out-dir ./data/SIDD/test
```

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── SIDD
|   |   ├── train
|   |   |   ├── gt
|   |   |   ├── noisy
|   |   ├── val
|   |   |   ├── input_crops
|   |   |   ├── target_crops
|   |   ├── test

├── tools
├── data
|   ├── SIDD
|   |   |   ├── gt
|   |   ├── val
|   |   ├── test
|   |   |   ├── gt
```

# Chain of Thought

## Background

During the process of reasoning, CoT (Chain of Thought) method is an efficient way to help LLMs deal complex questions, for example: math problem and relation inference. In OpenCompass, we support multiple types of CoT method.

![image](https://github.com/InternLM/opencompass/assets/28834990/45d60e0e-02a1-49aa-b792-40a1f95f9b9e)

## 1. Zero Shot CoT

You can change the `PromptTemplate` of the dataset config, by simply add *Let's think step by step* to realize a Zero-Shot CoT prompt for your evaluation:

```python
qa_infer_cfg = dict(
        template="Answer the question:\nQ: {question}?\nLet's think step by step:\n"
    ),
    retriever=dict(type=ZeroRetriever)
)
```

## 2. Few Shot CoT

Few-shot CoT can make LLMs easy to follow your instructions and get better answers. For few-shot CoT, add your CoT template to `PromptTemplate` like following config to create a one-shot prompt:

```python

```python
        template=
'''Question: Mark's basketball team scores 25 2 pointers, 8 3 pointers and 10 free throws.  Their opponents score double the 2 pointers but half the 3 pointers and free throws.  What's the total number of points scored by both teams added together?
Let's think step by step
Answer:
Mark's team scores 25 2 pointers, meaning they scored 25*2= 50 points in 2 pointers.
His team also scores 6 3 pointers, meaning they scored 8*3= 24 points in 3 pointers
They scored 10 free throws, and free throws count as one point so they scored 10*1=10 points in free throws.
All together his team scored 50+24+10= 84 points
Mark's opponents scored double his team's number of 2 pointers, meaning they scored 50*2=100 points in 2 pointers.
His opponents scored half his team's number of 3 pointers, meaning they scored 24/2= 12 points in 3 pointers.

They also scored half Mark's team's points in free throws, meaning they scored 10/2=5 points in free throws.
All together Mark's opponents scored 100+12+5=117 points
The total score for the game is both team's scores added together, so it is 84+117=201 points
The answer is 201

Question: {question}\nLet's think step by step:\n{answer}
'''),
)
```

## 3. Self-Consistency

'''),
)
```

The SC (Self-Consistency) method is proposed in [this paper](https://arxiv.org/abs/2203.11171), which will sample multiple reasoning paths for the question, and make majority voting to the generated answers for LLMs. This method displays remarkable proficiency among reasoning tasks with high accuracy but may consume more time and resources when inferencing, because of the majority voting strategy. In OpenCompass, You can easily implement the SC method by replacing `GenInferencer` with `SCInferencer` in the dataset configuration and setting the corresponding parameters like:

```python
# This SC gsm8k config can be found at: opencompass.configs.datasets.gsm8k.gsm8k_gen_a3e34a.py
gsm8k_infer_cfg = dict(
    inferencer=dict(
        type=SCInferencer, # Replace GenInferencer with SCInferencer.

        generation_kwargs=dict(do_sample=True, temperature=0.7, top_k=40),  # Set sample parameters to make sure model generate various output, only works for models load from HuggingFace now.
        infer_type='SC',
        sc_size = SAMPLE_SIZE
    )
)
gsm8k_eval_cfg = dict(sc_size=SAMPLE_SIZE)
```

```{note}
OpenCompass defaults to use argmax for sampling the next token. Therefore, if the sampling parameters are not specified, the model's inference results will be completely consistent each time, and multiple rounds of evaluation will be ineffective.
```

Where `SAMPLE_SIZE` is the number of reasoning paths in Self-Consistency, higher value usually outcome higher performance. The following figure from the original SC paper demonstrates the relation between reasoning paths and performance in several reasoning tasks:

![image](https://github.com/InternLM/opencompass/assets/28834990/05c7d850-7076-43ca-b165-e6251f9b3001)

From the figure, it can be seen that in different reasoning tasks, performance tends to improve as the number of reasoning paths increases. However, for some tasks, increasing the number of reasoning paths may reach a limit, and further increasing the number of paths may not bring significant performance improvement. Therefore, it is necessary to conduct experiments and adjustments on specific tasks to find the optimal number of reasoning paths that best suit the task.

## 4. Tree-of-Thoughts

In contrast to the conventional CoT approach that considers only a single reasoning path, Tree-of-Thoughts (ToT) allows the language model to explore multiple diverse reasoning paths simultaneously. The model evaluates the reasoning process through self-assessment and makes global choices by conducting lookahead or backtracking when necessary. Specifically, this process is divided into the following four stages:

**1. Thought Decomposition**

Based on the nature of the problem, break down the problem into multiple intermediate steps. Each step can be a phrase, equation, or writing plan, depending on the nature of the problem.

**2. Thought Generation**

Assuming that solving the problem requires k steps, there are two methods to generate reasoning content:

- Independent sampling: For each state, the model independently extracts k reasoning contents from the CoT prompts, without relying on other reasoning contents.

- Sequential generation: Sequentially use "prompts" to guide the generation of reasoning content, where each reasoning content may depend on the previous one.

**3. Heuristic Evaluation**

Use heuristic methods to evaluate the contribution of each generated reasoning content to problem-solving. This self-evaluation is based on the model's self-feedback and involves designing prompts to have the model score multiple generated results.

**4. Search Algorithm Selection**

Based on the methods of generating and evaluating reasoning content, select an appropriate search algorithm. For example, you can use breadth-first search (BFS) or depth-first search (DFS) algorithms to systematically explore the thought tree, conducting lookahead and backtracking.

In OpenCompass, ToT parameters need to be set according to the requirements. Below is an example configuration for the 24-Point game from the [official paper](https://arxiv.org/pdf/2305.10601.pdf). Currently, ToT inference is supported only with Huggingface models:

```python
# This ToT Game24 config can be found at: opencompass/configs/datasets/game24/game24_gen_8dfde3.py.
from opencompass.datasets import (Game24Dataset, game24_postprocess,
                                  Game24Evaluator, Game24PromptWrapper)

generation_kwargs = dict(temperature=0.7)

game24_infer_cfg = dict(
        prompt_template=dict(
        template='{input}'), # Directly pass the input content, as the Prompt needs to be specified in steps
    inferencer=dict(type=ToTInferencer, # Replace GenInferencer with ToTInferencer
                    generation_kwargs=generation_kwargs,

retriever=dict(type=ZeroRetriever),
                    method_generate='propose',  # Method for generating reasoning content, can be independent sampling (sample) or sequential generation (propose)
                    method_evaluate='value', # Method for evaluating reasoning content, can be voting (vote) or scoring (value)
                    method_select='greedy', # Method for selecting reasoning content, can be greedy (greedy) or random (sample)
                    n_evaluate_sample=3,
                    n_select_sample=5,
                    task_wrapper=dict(type=Game24PromptWrapper) # This Wrapper class includes the prompts for each step and methods for generating and evaluating reasoning content, needs customization according to the task
                    ))
```

))
```

If you want to use the ToT method on a custom dataset, you'll need to make additional configurations in the `opencompass.datasets.YourDataConfig.py` file to set up the `YourDataPromptWrapper` class. This is required for handling the thought generation and heuristic evaluation step within the ToT framework. For reasoning tasks similar to the game 24-Point, you can refer to the implementation in `opencompass/datasets/game24.py` for guidance.

}
```

<details>

```bibtex
}
```

<details>

booktitle={Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on},
}
```

<details>

```bibtex
}
```

The model is trained on WFLW train set.

# SoftTeacher

> [End-to-End Semi-Supervised Object Detection with Soft Teacher](https://arxiv.org/abs/2106.09018)

This paper presents an end-to-end semi-supervised object detection approach, in contrast to previous more complex multi-stage methods. The end-to-end training gradually improves pseudo label qualities during the curriculum, and the more and more accurate pseudo labels in turn benefit object detection training. We also propose two simple yet effective techniques within this framework: a soft teacher mechanism where the classification loss of each unlabeled bounding box is weighed by the classification score produced by the teacher network; a box jittering approach to select reliable pseudo boxes for the learning of box regression. On the COCO benchmark, the proposed approach outperforms previous methods by a large margin under various labeling ratios, i.e. 1%, 5% and 10%. Moreover, our approach proves to perform also well when the amount of labeled data is relatively large. For example, it can improve a 40.9 mAP baseline detector trained using the full COCO training set by +3.6 mAP, reaching 44.5 mAP, by leveraging the 123K unlabeled images of COCO. On the state-of-the-art Swin Transformer based object detector (58.9 mAP on test-dev), it can still significantly improve the detection accuracy by +1.5 mAP, reaching 60.4 mAP, and improve the instance segmentation accuracy by +1.2 mAP, reaching 52.4 mAP. Further incorporating with the Object365 pre-trained model, the detection accuracy reaches 61.3 mAP and the instance segmentation accuracy reaches 53.0 mAP, pushing the new state-of-the-art.

<img src="https://user-images.githubusercontent.com/40661020/186086683-f8a69813-d09c-4c3f-a86a-e233a708cd38.png"/>
</div>

|    Model    |   Detector   | Labeled Dataset | Iteration | box AP |                                  Config                                   |                                                                                                                                                                                                            Download                                                                                                                                                                                                            |

| SoftTeacher | Faster R-CNN |     COCO-1%     |   180k    |  19.9  | [config](./soft-teacher_faster-rcnn_r50-caffe_fpn_180k_semi-0.01-coco.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/soft_teacher/soft-teacher_faster-rcnn_r50-caffe_fpn_180k_semi-0.01-coco/soft-teacher_faster-rcnn_r50-caffe_fpn_180k_semi-0_20230330_233412-3c8f6d4a.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/soft_teacher/soft-teacher_faster-rcnn_r50-caffe_fpn_180k_semi-0.01-coco/soft-teacher_faster-rcnn_r50-caffe_fpn_180k_semi-0_20230330_233412.log.json) |

| SoftTeacher | Faster R-CNN |     COCO-2%     |   180k    |  24.9  | [config](./soft-teacher_faster-rcnn_r50-caffe_fpn_180k_semi-0.02-coco.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/soft_teacher/soft-teacher_faster-rcnn_r50-caffe_fpn_180k_semi-0.02-coco/soft-teacher_faster-rcnn_r50-caffe_fpn_180k_semi-0_20230331_020244-c0d2c3aa.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/soft_teacher/soft-teacher_faster-rcnn_r50-caffe_fpn_180k_semi-0.02-coco/soft-teacher_faster-rcnn_r50-caffe_fpn_180k_semi-0_20230331_020244.log.json) |

| SoftTeacher | Faster R-CNN |     COCO-5%     |   180k    |  30.4  | [config](./soft-teacher_faster-rcnn_r50-caffe_fpn_180k_semi-0.05-coco.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/soft_teacher/soft-teacher_faster-rcnn_r50-caffe_fpn_180k_semi-0.05-coco/soft-teacher_faster-rcnn_r50-caffe_fpn_180k_semi-0_20230331_070656-308798ad.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/soft_teacher/soft-teacher_faster-rcnn_r50-caffe_fpn_180k_semi-0.05-coco/soft-teacher_faster-rcnn_r50-caffe_fpn_180k_semi-0_20230331_070656.log.json) |

| SoftTeacher | Faster R-CNN |    COCO-10%     |   180k    |  33.8  | [config](./soft-teacher_faster-rcnn_r50-caffe_fpn_180k_semi-0.1-coco.py)  |  [model](https://download.openmmlab.com/mmdetection/v3.0/soft_teacher/soft-teacher_faster-rcnn_r50-caffe_fpn_180k_semi-0.1-coco/soft-teacher_faster-rcnn_r50-caffe_fpn_180k_semi-0_20230330_232113-b46f78d0.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/soft_teacher/soft-teacher_faster-rcnn_r50-caffe_fpn_180k_semi-0.1-coco/soft-teacher_faster-rcnn_r50-caffe_fpn_180k_semi-0_20230330_232113.log.json)  |

```latex
@article{xu2021end,
  title={End-to-End Semi-Supervised Object Detection with Soft Teacher},
  author={Xu, Mengde and Zhang, Zheng and Hu, Han and Wang, Jianfeng and Wang, Lijuan and Wei, Fangyun and Bai, Xiang and Liu, Zicheng},
  journal={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
}
```

# SUN RGB-D Dataset

For the overall process, please refer to the [README](https://github.com/open-mmlab/mmdetection3d/blob/master/data/sunrgbd/README.md) page for SUN RGB-D.

### Download SUN RGB-D data and toolbox

Download SUNRGBD data [HERE](http://rgbd.cs.princeton.edu/data/). Then, move `SUNRGBD.zip`, `SUNRGBDMeta2DBB_v2.mat`, `SUNRGBDMeta3DBB_v2.mat` and `SUNRGBDtoolbox.zip` to the `OFFICIAL_SUNRGBD` folder, unzip the zip files.

The directory structure before data preparation should be as below:

```
sunrgbd
├── matlab
│   ├── extract_rgbd_data_v1.m
│   ├── extract_rgbd_data_v2.m
│   ├── extract_split.m
├── OFFICIAL_SUNRGBD
│   ├── SUNRGBD
│   ├── SUNRGBDMeta2DBB_v2.mat
│   ├── SUNRGBDMeta3DBB_v2.mat
│   ├── SUNRGBDtoolbox
```

### Extract data and annotations for 3D detection from raw data

Extract SUN RGB-D annotation data from raw annotation data by running (this requires MATLAB installed on your machine):

```bash

```

```bash
matlab -nosplash -nodesktop -r 'extract_split;quit;'
matlab -nosplash -nodesktop -r 'extract_rgbd_data_v2;quit;'
matlab -nosplash -nodesktop -r 'extract_rgbd_data_v1;quit;'
```

The main steps include:

- Extract train and val split.
- Extract data for 3D detection from raw data.
- Extract and format detection annotation from raw data.

The main component of `extract_rgbd_data_v2.m` which extracts point cloud data from depth map is as follows:

```matlab
data = SUNRGBDMeta(imageId);
data.depthpath(1:16) = '';
data.depthpath = strcat('../OFFICIAL_SUNRGBD', data.depthpath);
data.rgbpath(1:16) = '';
data.rgbpath = strcat('../OFFICIAL_SUNRGBD', data.rgbpath);

% extract point cloud from depth map
[rgb,points3d,depthInpaint,imsize]=read3dPoints(data);
rgb(isnan(points3d(:,1)),:) = [];

points3d(isnan(points3d(:,1)),:) = [];
points3d_rgb = [points3d, rgb];

% MAT files are 3x smaller than TXT files. In Python we can use
% scipy.io.loadmat('xxx.mat')['points3d_rgb'] to load the data.
mat_filename = strcat(num2str(imageId,'%06d'), '.mat');
txt_filename = strcat(num2str(imageId,'%06d'), '.txt');
% save point cloud data
parsave(strcat(depth_folder, mat_filename), points3d_rgb);
```

The main component of `extract_rgbd_data_v1.m` which extracts annotation is as follows:

```matlab
% Write 2D and 3D box label
data2d = data;
fid = fopen(strcat(det_label_folder, txt_filename), 'w');
for j = 1:length(data.groundtruth3DBB)
    centroid = data.groundtruth3DBB(j).centroid;  % 3D bbox center
    classname = data.groundtruth3DBB(j).classname;  % class name

    orientation = data.groundtruth3DBB(j).orientation;  % 3D bbox orientation
    coeffs = abs(data.groundtruth3DBB(j).coeffs);  % 3D bbox size
    box2d = data2d.groundtruth2DBB(j).gtBb2D;  % 2D bbox
    fprintf(fid, '%s %d %d %d %d %f %f %f %f %f %f %f %f\n', classname, box2d(1), box2d(2), box2d(3), box2d(4), centroid(1), centroid(2), centroid(3), coeffs(1), coeffs(2), coeffs(3), orientation(1), orientation(2));
end
fclose(fid);
```

The above two scripts call functions such as `read3dPoints` from the [toolbox](https://rgbd.cs.princeton.edu/data/SUNRGBDtoolbox.zip) provided by SUN RGB-D.

The directory structure after extraction should be as follows.

```
sunrgbd
├── matlab

```
sunrgbd
├── matlab
├── sunrgbd_trainval
│   ├── calib
│   ├── depth
│   ├── image
│   ├── label
│   ├── label_v1
│   ├── seg_label
│   ├── train_data_idx.txt
│   ├── val_data_idx.txt
```

Under each following folder there are overall 5285 train files and 5050 val files:

- `calib`: Camera calibration information in `.txt`
- `depth`: Point cloud saved in `.mat` (xyz+rgb)
- `image`: Image data in `.jpg`
- `label`: Detection annotation data in `.txt` (version 2)
- `label_v1`: Detection annotation data in `.txt` (version 1)
- `seg_label`: Segmentation annotation data in `.txt`

Currently, we use v1 data for training and testing, so the version 2 labels are unused.

Please run the command below to create the dataset.

```shell

```shell
python tools/create_data.py sunrgbd --root-path ./data/sunrgbd \
--out-dir ./data/sunrgbd --extra-tag sunrgbd
```

or (if in a slurm environment)

```
bash tools/create_data.sh <job_name> sunrgbd
```

The above point cloud data are further saved in `.bin` format. Meanwhile `.pkl` info files are also generated for saving annotation and metadata.

The directory structure after processing should be as follows.

```
sunrgbd
├── matlab
│   ├── ...
│   ├── ...
│   ├── ...
├── points
├── sunrgbd_infos_train.pkl
├── sunrgbd_infos_val.pkl
```

- `points/xxxxxx.bin`: The point cloud data after downsample.
- `sunrgbd_infos_train.pkl`: The train data infos, the detailed info of each scene is as follows:

```

  - info\['lidar_points'\]: A dict containing all information related to the the lidar points.
  - info\['images'\]: A dict containing all information relate to the image data.
    - info\['images'\]\['CAM0'\]\['img_path'\]: The filename of the image.
    - info\['images'\]\['CAM0'\]\['depth2img'\]: Transformation matrix from depth to image with shape (4, 4).
    - info\['images'\]\['CAM0'\]\['height'\]: The height of image.
    - info\['images'\]\['CAM0'\]\['width'\]: The width of image.

- info\['images'\]\['CAM0'\]\['height'\]: The height of image.
  - info\['instances'\]: A list of dict contains all the annotations of this frame. Each dict corresponds to annotations of single instance. For the i-th instance:
    - info\['instances'\]\[i\]\['bbox_3d'\]: List of 7 numbers representing the 3D bounding box in depth coordinate system.
    - info\['instances'\]\[i\]\['bbox'\]: List of 4 numbers representing the 2D bounding box of the instance, in (x1, y1, x2, y2) order.
    - info\['instances'\]\[i\]\['bbox_label_3d'\]: An int indicates the 3D label of instance and the -1 indicates ignore class.
    - info\['instances'\]\[i\]\['bbox_label'\]: An int indicates the 2D label of instance and the -1 indicates ignore class.
- `sunrgbd_infos_val.pkl`: The val data infos, which shares the same format as `sunrgbd_infos_train.pkl`.

## Train pipeline

A typical train pipeline of SUN RGB-D for point cloud only 3D detection is as follows.

```python
    dict(
    dict(type='LoadAnnotations3D'),
    dict(
    ),
    dict(
        rot_range=[-0.523599, 0.523599],
        scale_ratio_range=[0.85, 1.15],
    dict(type='PointSample', num_points=20000),
    dict(
]
```

Data augmentation for point clouds:

- `RandomFlip3D`: randomly flip the input point cloud horizontally or vertically.

keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
]
```

- `GlobalRotScaleTrans`: rotate the input point cloud, usually in the range of \[-30, 30\] (degrees) for SUN RGB-D; then scale the input point cloud, usually in the range of \[0.85, 1.15\] for SUN RGB-D; finally translate the input point cloud, usually by 0 for SUN RGB-D (which means no translation).
- `PointSample`: downsample the input point cloud.

A typical train pipeline of SUN RGB-D for multi-modality (point cloud and image) 3D detection is as follows.

```python
    dict(

shift_height=True,
    dict(type='Resize', scale=(1333, 600), keep_ratio=True),
    dict(type='RandomFlip', flip_ratio=0.0),
    dict(
    ),
    dict(
    dict(
        keys=['points', 'gt_bboxes_3d', 'gt_labels_3d','img', 'gt_bboxes', 'gt_bboxes_labels'])
]
```

Data augmentation for images:

- `Resize`: resize the input image, `keep_ratio=True` means the ratio of the image is kept unchanged.
- `RandomFlip`: randomly flip the input image.

]
```

The image augmentation functions are implemented in [MMDetection](https://github.com/open-mmlab/mmdetection/tree/dev-3.x/mmdet/datasets/transforms).

## Metrics

Same as ScanNet, typically mean Average Precision (mAP) is used for evaluation on SUN RGB-D, e.g. `mAP@0.25` and `mAP@0.5`. In detail, a generic function to compute precision and recall for 3D object detection for multiple classes is called. Please refer to [indoor_eval](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/evaluation/functional/indoor_eval.py) for more details.

Since SUN RGB-D consists of image data, detection on image data is also feasible. For instance, in ImVoteNet, we first train an image detector, and we also use mAP for evaluation, e.g. `mAP@0.5`. We use the `eval_map` function from [MMDetection](https://github.com/open-mmlab/mmdetection) to calculate mAP.

# FoveaBox

> [FoveaBox: Beyond Anchor-based Object Detector](https://arxiv.org/abs/1904.03797)

We present FoveaBox, an accurate, flexible, and completely anchor-free framework for object detection. While almost all state-of-the-art object detectors utilize predefined anchors to enumerate possible locations, scales and aspect ratios for the search of the objects, their performance and generalization ability are also limited to the design of anchors. Instead, FoveaBox directly learns the object existing possibility and the bounding box coordinates without anchor reference. This is achieved by: (a) predicting category-sensitive semantic maps for the object existing possibility, and (b) producing category-agnostic bounding box for each position that potentially contains an object. The scales of target boxes are naturally associated with feature pyramid representations. In FoveaBox, an instance is assigned to adjacent feature levels to make the model more accurate.We demonstrate its effectiveness on standard benchmarks and report extensive experimental analysis. Without bells and whistles, FoveaBox achieves state-of-the-art single model performance on the standard COCO and Pascal VOC object detection benchmark. More importantly, FoveaBox avoids all computation and hyper-parameters related to anchor boxes, which are often sensitive to the final detection performance. We believe the simple and effective approach will serve as a solid baseline and help ease future research for object detection.

<img src="https://user-images.githubusercontent.com/40661020/143885497-332d38a7-b492-4f51-b9d2-ef9d4ad4412a.png"/>
</div>

FoveaBox is an accurate, flexible and completely anchor-free object detection system for object detection framework, as presented in our paper [https://arxiv.org/abs/1904.03797](https://arxiv.org/abs/1904.03797):
Different from previous anchor-based methods, FoveaBox directly learns the object existing possibility and the bounding box coordinates without anchor reference. This is achieved by: (a) predicting category-sensitive semantic maps for the object existing possibility, and (b) producing category-agnostic bounding box for each position that potentially contains an object.

### Results on R50/101-FPN

| Backbone |  Style  | align | ms-train | Lr schd | Mem (GB) | Inf time (fps) | box AP |                               Config                                |                                                                                                                                                                                                        Download                                                                                                                                                                                                         |

|   R-50   | pytorch |   N   |    N     |   1x    |   5.6    |      24.1      |  36.5  |              [config](./fovea_r50_fpn_4xb4-1x_coco.py)              |                                                               [model](https://download.openmmlab.com/mmdetection/v2.0/foveabox/fovea_r50_fpn_4x4_1x_coco/fovea_r50_fpn_4x4_1x_coco_20200219-ee4d5303.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/foveabox/fovea_r50_fpn_4x4_1x_coco/fovea_r50_fpn_4x4_1x_coco_20200219_223025.log.json)                                                               |

|   R-50   | pytorch |   N   |    N     |   2x    |   5.6    |       -        |  37.2  |              [config](./fovea_r50_fpn_4xb4-2x_coco.py)              |                                                               [model](https://download.openmmlab.com/mmdetection/v2.0/foveabox/fovea_r50_fpn_4x4_2x_coco/fovea_r50_fpn_4x4_2x_coco_20200203-2df792b1.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/foveabox/fovea_r50_fpn_4x4_2x_coco/fovea_r50_fpn_4x4_2x_coco_20200203_112043.log.json)                                                               |

|   R-50   | pytorch |   Y   |    N     |   2x    |   8.1    |      19.4      |  37.9  |       [config](./fovea_r50_fpn_gn-head-align_4xb4-2x_coco.py)       |                                   [model](https://download.openmmlab.com/mmdetection/v2.0/foveabox/fovea_align_r50_fpn_gn-head_4x4_2x_coco/fovea_align_r50_fpn_gn-head_4x4_2x_coco_20200203-8987880d.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/foveabox/fovea_align_r50_fpn_gn-head_4x4_2x_coco/fovea_align_r50_fpn_gn-head_4x4_2x_coco_20200203_134252.log.json)                                   |

|   R-50   | pytorch |   Y   |    Y     |   2x    |   8.1    |      18.3      |  40.4  | [config](./fovea_r50_fpn_gn-head-align_ms-640-800-4xb4-2x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/foveabox/fovea_align_r50_fpn_gn-head_mstrain_640-800_4x4_2x_coco/fovea_align_r50_fpn_gn-head_mstrain_640-800_4x4_2x_coco_20200205-85ce26cb.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/foveabox/fovea_align_r50_fpn_gn-head_mstrain_640-800_4x4_2x_coco/fovea_align_r50_fpn_gn-head_mstrain_640-800_4x4_2x_coco_20200205_112557.log.json)   |

|  R-101   | pytorch |   N   |    N     |   1x    |   9.2    |      17.4      |  38.6  |             [config](./fovea_r101_fpn_4xb4-1x_coco.py)              |                                                             [model](https://download.openmmlab.com/mmdetection/v2.0/foveabox/fovea_r101_fpn_4x4_1x_coco/fovea_r101_fpn_4x4_1x_coco_20200219-05e38f1c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/foveabox/fovea_r101_fpn_4x4_1x_coco/fovea_r101_fpn_4x4_1x_coco_20200219_011740.log.json)                                                             |

|  R-101   | pytorch |   N   |    N     |   2x    |   11.7   |       -        |  40.0  |             [config](./fovea_r101_fpn_4xb4-2x_coco.py)              |                                                             [model](https://download.openmmlab.com/mmdetection/v2.0/foveabox/fovea_r101_fpn_4x4_2x_coco/fovea_r101_fpn_4x4_2x_coco_20200208-02320ea4.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/foveabox/fovea_r101_fpn_4x4_2x_coco/fovea_r101_fpn_4x4_2x_coco_20200208_202059.log.json)                                                             |

|  R-101   | pytorch |   Y   |    N     |   2x    |   11.7   |      14.7      |  40.0  |      [config](./fovea_r101_fpn_gn-head-align_4xb4-2x_coco.py)       |                                 [model](https://download.openmmlab.com/mmdetection/v2.0/foveabox/fovea_align_r101_fpn_gn-head_4x4_2x_coco/fovea_align_r101_fpn_gn-head_4x4_2x_coco_20200208-c39a027a.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/foveabox/fovea_align_r101_fpn_gn-head_4x4_2x_coco/fovea_align_r101_fpn_gn-head_4x4_2x_coco_20200208_203337.log.json)                                 |

|  R-101   | pytorch |   Y   |    Y     |   2x    |   11.7   |      14.7      |  42.0  | [config](./fovea_r101_fpn_gn-head-align_ms-640-800-4xb4-2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/foveabox/fovea_align_r101_fpn_gn-head_mstrain_640-800_4x4_2x_coco/fovea_align_r101_fpn_gn-head_mstrain_640-800_4x4_2x_coco_20200208-649c5eb6.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/foveabox/fovea_align_r101_fpn_gn-head_mstrain_640-800_4x4_2x_coco/fovea_align_r101_fpn_gn-head_mstrain_640-800_4x4_2x_coco_20200208_202124.log.json) |

\[1\] *1x and 2x mean the model is trained for 12 and 24 epochs, respectively.* \
\[2\] *Align means utilizing deformable convolution to align the cls branch.* \
\[3\] *All results are obtained with a single model and without any test time data augmentation.*\
\[4\] *We use 4 GPUs for training.*

Any pull requests or issues are welcome.

Please consider citing our paper in your publications if the project helps your research. BibTeX reference is as follows.

```latex
@article{kong2019foveabox,
  title={FoveaBox: Beyond Anchor-based Object Detector},
  author={Kong, Tao and Sun, Fuchun and Liu, Huaping and Jiang, Yuning and Shi, Jianbo},
  journal={arXiv preprint arXiv:1904.03797},
}
```

# Reversible Vision Transformers

> [Reversible Vision Transformers](https://openaccess.thecvf.com/content/CVPR2022/papers/Mangalam_Reversible_Vision_Transformers_CVPR_2022_paper.pdf)

**RevViT** is initially described in [Reversible Vision Tranformers](https://openaccess.thecvf.com/content/CVPR2022/papers/Mangalam_Reversible_Vision_Transformers_CVPR_2022_paper.pdf), which introduce the reversible idea into vision transformer, to reduce the GPU memory footprint required for training.

<img src="https://github.com/facebookresearch/SlowFast/raw/main/projects/rev/teaser.png" width="70%"/>
</div>

<details>

<br>

<br>
We present Reversible Vision Transformers, a memory efficient architecture design for visual recognition. By decoupling the GPU memory footprint from the depth of the model, Reversible Vision Transformers enable memory efficient scaling of transformer architectures. We adapt two popular models, namely Vision Transformer and Multiscale Vision Transformers, to reversible variants and benchmark extensively across both model sizes and tasks of image classification, object detection and video classification. Reversible Vision Transformers achieve a reduced memory footprint of up to 15.5× at identical model complexity, parameters and accuracy, demonstrating the promise of reversible vision transformers as an efficient backbone for resource limited training regimes. Finally, we find that the additional computational burden of recomputing activations is more than overcome for deeper models, where throughput can increase up to 3.9× over their non-reversible counterparts.
</br>

</br>

```python

predict = inference_model('revvit-small_3rdparty_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('revvit-small_3rdparty_in1k', pretrained=True)
```

Test:

```shell
python tools/test.py configs/revvit/revvit-small_8xb256_in1k.py https://download.openmmlab.com/mmclassification/v0/revvit/revvit-base_3rdparty_in1k_20221213-87a7b0a5.pth
```

```

| Model                          |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                Config                 |                                       Download                                       |
| :----------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :-----------------------------------: | :----------------------------------------------------------------------------------: |
| `revvit-small_3rdparty_in1k`\* | From scratch |   22.44    |   4.58    |   79.87   |   94.90   | [config](revvit-small_8xb256_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/revvit/revvit-base_3rdparty_in1k_20221213-87a7b0a5.pth) |

| `revvit-base_3rdparty_in1k`\*  | From scratch |   87.34    |   17.49   |   81.81   |   95.56   | [config](revvit-base_8xb256_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/revvit/revvit-small_3rdparty_in1k_20221213-a3a34f5c.pth) |

*Models with * are converted from the [official repo](https://github.com/facebookresearch/SlowFast). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@inproceedings{mangalam2022reversible,
  title={Reversible Vision Transformers},
  author={Mangalam, Karttikeya and Fan, Haoqi and Li, Yanghao and Wu, Chao-Yuan and Xiong, Bo and Feichtenhofer, Christoph and Malik, Jitendra},
  pages={10830--10840},
}
```

We recommend that users follow our best practices to install MMPose. However, the whole process is highly customizable. See [Customize Installation](#customize-installation) section for more information.

  - [Prerequisites](#prerequisites)
  - [Best Practices](#best-practices)
    - [Build MMPose from source](#build-mmpose-from-source)
    - [Install as a Python package](#install-as-a-python-package)
  - [Customize Installation](#customize-installation)
    - [CUDA versions](#cuda-versions)
    - [Install MMEngine without MIM](#install-mmengine-without-mim)
    - [Install MMCV without MIM](#install-mmcv-without-mim)
    - [Install on CPU-only platforms](#install-on-cpu-only-platforms)
    - [Install on Google Colab](#install-on-google-colab)
    - [Using MMPose with Docker](#using-mmpose-with-docker)
  - [Verify the installation](#verify-the-installation)
  - [Trouble shooting](#trouble-shooting)

- [Using MMPose with Docker](#using-mmpose-with-docker)

MMPose works on Linux, Windows and macOS. It requires Python 3.7+, CUDA 9.2+ and PyTorch 1.8+.

If you are experienced with PyTorch and have already installed it, you can skip this part and jump to the [MMPose Installation](#install-mmpose). Otherwise, you can follow these steps for the preparation.

```shell
```

```shell

```

```shell
```

```

```shell
```

**Step 3.** Install [MMEngine](https://github.com/open-mmlab/mmengine) and [MMCV](https://github.com/open-mmlab/mmcv/tree/2.x) using [MIM](https://github.com/open-mmlab/mim).

```shell
```

Note that some of the demo scripts in MMPose require [MMDetection](https://github.com/open-mmlab/mmdetection) (mmdet)  for human detection. If you want to run these demo scripts with mmdet, you can easily install mmdet as a dependency by running:

```shell

```shell
```

### Build MMPose from source

To develop and run mmpose directly, install it from source:

```shell
cd mmpose
```

To use mmpose as a dependency or third-party package, install it with pip:

```shell
```

To verify that MMPose is installed correctly, you can run an inference demo with the following steps.

```shell
```

```shell
```

The downloading will take several seconds or more, depending on your network environment. When it is done, you will find two files `td-hm_hrnet-w48_8xb32-210e_coco-256x192.py` and `hrnet_w48_coco_256x192-b9e0b3ab_20200708.pth` in your current folder.

**Step 2.** Run the inference demo.

Option (A). If you install mmpose from source, just run the following command under the folder `$MMPOSE`:

```shell
```

tests/data/coco/000000000785.jpg \
```

If everything goes fine, you will be able to get the following visualization result from `vis_results.jpg` in your current folder, which displays the predicted keypoints and heatmaps overlaid on the person in the image.

Option (B). If you install mmpose with pip, open you python interpreter and copy & paste the following codes.

```python

# please prepare an image with person
```

The `demo.jpg` can be downloaded from [Github](https://raw.githubusercontent.com/open-mmlab/mmpose/main/tests/data/coco/000000000785.jpg).

The inference results will be a list of `PoseDataSample`, and the predictions are in the `pred_instances`, indicating the detected keypoint locations and scores.

### Install MMEngine without MIM

To install MMEngine with pip instead of MIM, please follow [MMEngine installation guides](https://mmengine.readthedocs.io/zh_CN/latest/get_started/installation.html).

```shell
```

To install MMCV with pip instead of MIM, please follow [MMCV installation guides](https://mmcv.readthedocs.io/en/2.x/get_started/installation.html). This requires manually specifying a find-url based on PyTorch version and its CUDA version.

For example, the following command install mmcv built for PyTorch 1.10.x and CUDA 11.3.

```shell
```

```shell
```

MMPose can be built for CPU only environment. In CPU mode you can train, test or inference a model.

However, some functionalities are missing in this mode, usually GPU-compiled ops like `Deformable Convolution`. Most models in MMPose don't depend on these ops, but if you try to train/test/infer a model containing these ops, an error will be raised.

thus we only need to install MMEngine, MMCV and MMPose with the following commands.

**Step 1.** Install [MMEngine](https://github.com/open-mmlab/mmengine) and [MMCV](https://github.com/open-mmlab/mmcv/tree/2.x) using [MIM](https://github.com/open-mmlab/mim).

```shell
```

**Step 2.** Install MMPose from the source.

```shell
```

```shell
%cd mmpose
```

```python
# Example output: 1.1.0
```

```{note}
Note that within Jupyter, the exclamation mark `!` is used to call external executables and `%cd` is a [magic command](https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-cd) to change the current working directory of Python.
```

### Using MMPose with Docker

We provide a [Dockerfile](https://github.com/open-mmlab/mmpose/blob/master/docker/Dockerfile) to build an image. Ensure that your [docker version](https://docs.docker.com/engine/install/) >=19.03.

```shell
# build an image with PyTorch 1.8.0, CUDA 10.1, CUDNN 7.
```

```shell
```

**Important:** Make sure you've installed the [nvidia-container-toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker).

Run it with

```shell
```

`{DATA_DIR}` is your local folder containing all the datasets for mmpose.

```{note}
If you encounter the error message like `permission denied`, please add `sudo` at the start of the command and try it again.
```

If you have some issues during the installation, please first view the [FAQ](./faq.md) page.
You may [open an issue](https://github.com/open-mmlab/mmpose/issues/new/choose) on GitHub if no solution is found.

# BarlowTwins

> [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://arxiv.org/abs/2103.03230)

Self-supervised learning (SSL) is rapidly closing the gap with supervised methods on large computer vision benchmarks. A successful approach to SSL is to learn embeddings which are invariant to distortions of the input sample. However, a recurring issue with this approach is the existence of trivial constant solutions. Most current methods avoid such solutions by careful implementation details. We propose an objective function that naturally avoids collapse by measuring the cross-correlation matrix between the outputs of two identical networks fed with distorted versions of a sample, and making it as close to the identity matrix as possible. This causes the embedding vectors of distorted versions of a sample to be similar, while minimizing the redundancy between the components of these vectors. The method is called Barlow Twins, owing to neuroscientist H. Barlow's redundancy-reduction principle applied to a pair of identical networks. Barlow Twins does not require large batches nor asymmetry between the network twins such as a predictor network, gradient stopping, or a moving average on the weight updates. Intriguingly it benefits from very high-dimensional output vectors. Barlow Twins outperforms previous methods on ImageNet for semi-supervised classification in the low-data regime, and is on par with current state of the art for ImageNet classification with a linear classifier head, and for transfer tasks of classification and object detection.

<img src="https://user-images.githubusercontent.com/36138628/163914714-082de804-0b5f-4024-94f9-880e6ef334fa.png" width="800" />
</div>

```python

predict = inference_model('resnet50_barlowtwins-pre_8xb32-linear-coslr-100e_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('barlowtwins_resnet50_8xb256-coslr-300e_in1k', pretrained=True)
```

Train:

```shell

```

Train:

```shell
python tools/train.py configs/barlowtwins/barlowtwins_resnet50_8xb256-coslr-300e_in1k.py
```

Test:

```shell
python tools/test.py configs/barlowtwins/benchmarks/resnet50_8xb32-linear-coslr-100e_in1k.py https://download.openmmlab.com/mmselfsup/1.x/barlowtwins/barlowtwins_resnet50_8xb256-coslr-300e_in1k/resnet50_linear-8xb32-coslr-100e_in1k/resnet50_linear-8xb32-coslr-100e_in1k_20220825-52fde35f.pth
```

| Model                                         | Params (M) | Flops (G) |                          Config                          |                                     Download                                     |

| :-------------------------------------------- | :--------: | :-------: | :------------------------------------------------------: | :------------------------------------------------------------------------------: |
| `barlowtwins_resnet50_8xb256-coslr-300e_in1k` |   174.54   |   4.11    | [config](barlowtwins_resnet50_8xb256-coslr-300e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/barlowtwins/barlowtwins_resnet50_8xb256-coslr-300e_in1k/barlowtwins_resnet50_8xb256-coslr-300e_in1k_20220825-57307488.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/barlowtwins/barlowtwins_resnet50_8xb256-coslr-300e_in1k/barlowtwins_resnet50_8xb256-coslr-300e_in1k_20220825-57307488.json) |

| `resnet50_barlowtwins-pre_8xb32-linear-coslr-100e_in1k` | [BARLOWTWINS](https://download.openmmlab.com/mmselfsup/1.x/barlowtwins/barlowtwins_resnet50_8xb256-coslr-300e_in1k/barlowtwins_resnet50_8xb256-coslr-300e_in1k_20220825-57307488.pth) |   25.56    |   4.11    |   71.80   | [config](benchmarks/resnet50_8xb32-linear-coslr-100e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/barlowtwins/barlowtwins_resnet50_8xb256-coslr-300e_in1k/resnet50_linear-8xb32-coslr-100e_in1k/resnet50_linear-8xb32-coslr-100e_in1k_20220825-52fde35f.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/barlowtwins/barlowtwins_resnet50_8xb256-coslr-300e_in1k/resnet50_linear-8xb32-coslr-100e_in1k/resnet50_linear-8xb32-coslr-100e_in1k_20220825-52fde35f.json) |

```bibtex
@inproceedings{zbontar2021barlow,
  title={Barlow twins: Self-supervised learning via redundancy reduction},
  author={Zbontar, Jure and Jing, Li and Misra, Ishan and LeCun, Yann and Deny, St{\'e}phane},
}
```

# Custom Dataset Training with PoseC3D

We provide a step-by-step tutorial on how to train your custom dataset with PoseC3D.

1. First, you should know that action recognition with PoseC3D requires skeleton information only and for that you need to prepare your custom annotation files (for training and validation). To start with, you need to install MMDetection and MMPose. Then you need to take advantage of [ntu_pose_extraction.py](https://github.com/open-mmlab/mmaction2/blob/90fc8440961987b7fe3ee99109e2c633c4e30158/tools/data/skeleton/ntu_pose_extraction.py) as shown in [Prepare Annotations](https://github.com/open-mmlab/mmaction2/blob/master/tools/data/skeleton/README.md#prepare-annotations) to extract 2D keypoints for each video in your custom dataset. The command looks like (assuming the name of your video is `some_video_from_my_dataset.mp4`):

   ```shell
   # You can use the above command to generate pickle files for all of your training and validation videos.

```shell
   python ntu_pose_extraction.py some_video_from_my_dataset.mp4 some_video_from_my_dataset.pkl
   ```

   @kennymckormick's [note](https://github.com/open-mmlab/mmaction2/issues/1216#issuecomment-950130079):

   > One only thing you may need to change is that: since ntu_pose_extraction.py is developed specifically for pose extraction of NTU videos, you can skip the [ntu_det_postproc](https://github.com/open-mmlab/mmaction2/blob/90fc8440961987b7fe3ee99109e2c633c4e30158/tools/data/skeleton/ntu_pose_extraction.py#L307) step when using this script for extracting pose from your custom video datasets.

2. Then, you will collect all the pickle files into one list for training (and, of course, for validation) and save them as a single file (like `custom_dataset_train.pkl` or `custom_dataset_val.pkl`). At that time, you finalize preparing annotation files for your custom dataset.

3. Next, you may use the following script (with some alterations according to your needs) for training as shown in [PoseC3D/Train](https://github.com/open-mmlab/mmaction2/blob/master/configs/skeleton/posec3d/README.md#train): `python tools/train.py configs/skeleton/posec3d/slowonly_r50_u48_240e_ntu120_xsub_keypoint.py --work-dir work_dirs/slowonly_r50_u48_240e_ntu120_xsub_keypoint --validate --test-best --gpus 2 --seed 0 --deterministic`:

   - Before running the above script, you need to modify the variables to initialize with your newly made annotation files:

     ```python
     model = dict(
         ...
         cls_head=dict(
             ...
             num_classes=4,    # Your class number

```python
         ...
             ...
             ...
         ),
         ...
     )

     ann_file_train = 'data/posec3d/custom_dataset_train.pkl'  # Your annotation for training
     ann_file_val = 'data/posec3d/custom_dataset_val.pkl'      # Your annotation for validation

     load_from = 'pretrained_weight.pth'       # Your can use released weights for initialization, set to None if training from scratch

     # You can also alter the hyper parameters or training schedule
     ```

With that, your machine should start its work to let you grab a cup of coffee and watch how the training goes.

# MMAction2 Deployment

- [MMAction2 Deployment](#mmaction2-deployment)
    - [Install mmaction2](#install-mmaction2)
    - [Convert video recognition model](#convert-video-recognition-model)
  - [Model Inference](#model-inference)
      - [Video recognition SDK model inference](#video-recognition-sdk-model-inference)

[MMAction2](https://github.com/open-mmlab/mmaction2) is an open-source toolbox for video understanding based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com) project.

### Install mmaction2

Please follow the [installation guide](https://github.com/open-mmlab/mmaction2/tree/main#installation) to install mmaction2.

**Method I：** Install precompiled package

**Method II：** Build using scripts

```shell

```shell
```

You can use [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/tree/main/tools/deploy.py) to convert mmaction2 models to the specified backend models. Its detailed usage can be learned from [here](https://github.com/open-mmlab/mmdeploy/tree/main/docs/en/02-how-to-run/convert_model.md#usage).

When using `tools/deploy.py`, it is crucial to specify the correct deployment config. We've already provided builtin deployment config [files](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmaction) of all supported backends for mmaction2, under which the config file path follows the pattern:

```
```

其中：

- **{task}:** task in mmaction2.
- **{2d/3d}:** model type

In the next part，we will take `tsn` model from `video recognition` task as an example, showing how to convert them to onnx model that can be inferred by ONNX Runtime.

### Convert video recognition model

```shell

# download tsn model from mmaction2 model zoo

```shell

mim download mmaction2 --config tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb --dest .

# convert mmaction2 model to onnxruntime model with dynamic shape
    configs/mmaction/video-recognition/video-recognition_2d_onnxruntime_static.py \
    tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb \
    tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb_20220906-cd10898e.pth \
    tests/data/arm_wrestling.mp4 \
    --work-dir mmdeploy_models/mmaction/tsn/ort \
    --show \
```

The converted model locates in the working directory like `mmdeploy_models/mmaction/tsn/ort` in the previous example. It includes:

```

```
mmdeploy_models/mmaction/tsn/ort
```

in which,

The whole package **mmdeploy_models/mmaction/tsn/ort** is defined as **mmdeploy SDK model**, i.e., **mmdeploy SDK model** includes both backend model and inference meta information.

## Model Inference

Take the previous converted `end2end.onnx` mode of `tsn` as an example, you can use the following code to inference the model and visualize the results.

```python

```python

deploy_cfg = 'configs/mmaction/video-recognition/video-recognition_2d_onnxruntime_static.py'
model_cfg = 'tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb'
backend_model = ['./mmdeploy_models/mmaction2/tsn/ort/end2end.onnx']
image = 'tests/data/arm_wrestling.mp4'

# show top5-results
pred_scores = result[0].pred_scores.item.tolist()

top_index = np.argsort(pred_scores)[::-1]
for i in range(5):
    index = top_index[i]
    print(index, pred_scores[index])
```

Given the above SDK model of `tsn` you can also perform SDK model inference like following,

#### Video recognition SDK model inference

```python
from mmdeploy_runtime import VideoRecognizer
import cv2

# refer to demo/python/video_recognition.py
# def SampleFrames(cap, clip_len, frame_interval, num_clips):
#  ...

cap = cv2.VideoCapture('tests/data/arm_wrestling.mp4')

clips, info = SampleFrames(cap, 1, 1, 25)

# create a recognizer
recognizer = VideoRecognizer(model_path='./mmdeploy_models/mmaction/tsn/ort', device_name='cpu', device_id=0)
result = recognizer(clips, info)
```

```

> MMAction2 only API of c, c++ and python for now.

| Model                                                                                      | TorchScript | ONNX Runtime | TensorRT | ncnn | PPLNN | OpenVINO |
| :----------------------------------------------------------------------------------------- | :---------: | :----------: | :------: | :--: | :---: | :------: |
| [TSN](https://github.com/open-mmlab/mmaction2/tree/main/configs/recognition/tsn)           |      Y      |      Y       |    Y     |  N   |   N   |    N     |

| [SlowFast](https://github.com/open-mmlab/mmaction2/tree/main/configs/recognition/slowfast) |      Y      |      Y       |    Y     |  N   |   N   |    N     |
| [TSM](https://github.com/open-mmlab/mmaction2/tree/main/configs/recognition/tsm)           |      Y      |      Y       |    Y     |  N   |   N   |    N     |
| [X3D](https://github.com/open-mmlab/mmaction2/tree/main/configs/recognition/x3d)           |      Y      |      Y       |    Y     |  N   |   N   |    N     |

# 权重初始化

在训练过程中，适当的初始化策略有利于加快训练速度或获得更⾼的性能。 [MMCV](https://github.com/open-mmlab/mmcv/blob/master/mmcv/cnn/utils/weight_init.py) 提供了一些常⽤的初始化模块的⽅法，如 `nn.Conv2d`。 MMdetection 中的模型初始化主要使⽤ `init_cfg`。⽤⼾可以通过以下两个步骤来初始化模型：

1. 在 `model_cfg` 中为模型或其组件定义 `init_cfg`，但⼦组件的 `init_cfg` 优先级更⾼，会覆盖⽗模块的 `init_cfg` 。
2. 像往常一样构建模型，然后显式调⽤ `model.init_weights()` ⽅法，此时模型参数将会被按照配置文件写法进行初始化。

MMdetection 初始化工作流的高层 API 调用流程是：

### 描述

它的数据类型是 dict 或者 list\[dict\]，包含了下列键值:

- `type` (str)，包含 `INTIALIZERS` 中的初始化器名称，后面跟着初始化器的参数。
- `layer`（str 或 list\[str\]），包含 Pytorch 或 MMCV 中基本层的名称，以及将被初始化的可学习参数，例如 `'Conv2d'`，`'DeformConv2d'`。
- `override` (dict 或 list\[dict\])，包含不继承⾃ `BaseModule` 且其初始化配置与 `layer` 键中的其他层不同的⼦模块。 `type` 中定义的初始化器将适⽤于 `layer` 中定义的所有层，因此如果⼦模块不是 `BaseModule` 的派⽣类但可以与 `layer` 中的层相同的⽅式初始化，则不需要使⽤ `override`。`override` 包含了：
  - `type` 后跟初始化器的参数；
  - `name` 用以指⽰将被初始化的⼦模块。

### 初始化参数

- `type` 后跟初始化器的参数；

### 初始化参数

从 `mmcv.runner.BaseModule` 或 `mmdet.models` 继承一个新模型。这里我们用 FooModel 来举个例子。

```python

                 arg1,
                 arg2,
		...
```

- 直接在代码中使⽤ `init_cfg` 初始化模型

  ```python

                  arg1,
                  arg2,
    		super(FooModel, self).__init__(init_cfg)
    	    ...
  ```

- 在 `mmcv.Sequential` 或 `mmcv.ModuleList` 代码中直接使⽤ `init_cfg` 初始化模型

  ```python

                	arg1,
                	arg2,
                	init_cfg=None):

```python

                	arg1,
                	arg2,
        	...
        	self.conv1 = ModuleList(init_cfg=XXX)
  ```

- 使⽤配置⽂件中的 `init_cfg` 初始化模型

  ```python
  	...
    	model = dict(
        	type='FooModel',
        	arg1=XXX,
        	arg2=XXX,
        	init_cfg=XXX),
            ...
  ```

### init_cfg 的使用

1. 用 `layer` 键初始化模型

   如果我们只定义了 `layer`, 它只会在 `layer` 键中初始化网络层。

   注意： `layer` 键对应的值是 Pytorch 的带有 weights 和 bias 属性的类名（因此不⽀持 `MultiheadAttention` 层）。

- 定义⽤于初始化具有相同配置的模块的 `layer` 键。

  ```python
  # ⽤相同的配置初始化整个模块
  ```

- 定义⽤于初始化具有不同配置的层的 `layer` 键。

  ```python
  init_cfg = [dict(type='Constant', layer='Conv1d', val=1),
              dict(type='Constant', layer='Conv2d', val=2),

# ⽤相同的配置初始化整个模块
  ```

  ```python
              dict(type='Constant', layer='Linear', val=3)]
  # nn.Conv1d 将被初始化为 dict(type='Constant', val=1)
  # nn.Conv2d 将被初始化为 dict(type='Constant', val=2)
  # nn.Linear 将被初始化为 dict(type='Constant', val=3)
  ```

2. 使⽤ `override` 键初始化模型

- 当使⽤属性名初始化某些特定部分时，我们可以使⽤ `override` 键， `override` 中的值将忽略 init_cfg 中的值。

  ```python
  # layers：

  # self.feat and self.cls 将被初始化为 dict(type='Constant', val=1, bias=2)
  # 叫 'reg' 的模块将被初始化为 dict(type='Constant', val=3, bias=4)
  ```

# self.feat and self.cls 将被初始化为 dict(type='Constant', val=1, bias=2)
  ```

- 如果 init_cfg 中的 `layer` 为 None，则只会初始化 override 中有 name 的⼦模块，⽽ override 中的 type 和其他参数可以省略。

  ```python
  # layers：

  # self.feat and self.cls 将被 Pytorch 初始化
  # 叫 'reg' 的模块将被 dict(type='Constant', val=1, bias=2) 初始化
  ```

- 如果我们不定义 `layer` 或 `override` 键，它不会初始化任何东西。

- 无效的使用

  ```python
  # override 没有 name 键的话是无效的

  # override 有 name 键和其他参数但是没有 type 键也是无效的
  ```

3. 使⽤预训练模型初始化模型

   ```python

init_cfg = dict(type='Constant', layer=['Conv1d','Conv2d'], val=1, bias=2,
  ```

   ```python
                checkpoint='torchvision://resnet50')
   ```

更多细节可以参考 [MMEngine](https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/initialize.html) 的文档

## You can find keras2ncnn tools here

[https://github.com/MarsTechHAN/keras2ncnn](https://github.com/MarsTechHAN/keras2ncnn)  
[https://github.com/azeme1/keras2ncnn](https://github.com/azeme1/keras2ncnn)

----
### From tensorflow 2.x, you can also export mlir and use mlir2ncnn which is maintained by the official.

The source code is located here: [https://github.com/Tencent/ncnn/tree/master/tools/mlir](https://github.com/Tencent/ncnn/tree/master/tools/mlir)  
For Chinese, you can refer the guide here [https://zhuanlan.zhihu.com/p/152535430](https://zhuanlan.zhihu.com/p/152535430)

# Add a dataset

Although OpenCompass has already included most commonly used datasets, users need to follow the steps below to support a new dataset if wanted:

1. Add a dataset script `mydataset.py` to the `opencompass/datasets` folder. This script should include:

   - The dataset and its loading method. Define a `MyDataset` class that implements the data loading method `load` as a static method. This method should return data of type `datasets.Dataset`. We use the Hugging Face dataset as the unified interface for datasets to avoid introducing additional logic. Here's an example:

   ```python

           pass
   ```

```python

           pass
   ```

   - (Optional) If the existing evaluators in OpenCompass do not meet your needs, you need to define a `MyDatasetEvaluator` class that implements the scoring method `score`. This method should take `predictions` and `references` as input and return the desired dictionary. Since a dataset may have multiple metrics, the method should return a dictionary containing the metrics and their corresponding scores. Here's an example:

   ```python

   class MyDatasetEvaluator(BaseEvaluator):

           pass
   ```

class MyDatasetEvaluator(BaseEvaluator):

           pass
   ```

   - (Optional) If the existing postprocessors in OpenCompass do not meet your needs, you need to define the `mydataset_postprocess` method. This method takes an input string and returns the corresponding postprocessed result string. Here's an example:

   ```python
       pass
   ```

2. After defining the dataset loading, data postprocessing, and evaluator methods, you need to add the following configurations to the configuration file:

   ```python
   from opencompass.datasets import MyDataset, MyDatasetEvaluator, mydataset_postprocess

       evaluator=dict(type=MyDatasetEvaluator),

       dict(
           ...,

pred_postprocessor=dict(type=mydataset_postprocess))

       dict(
           ...,
   ]
   ```

   Once the dataset is configured, you can refer to the instructions on [Get started](../get_started.md) for other requirements.

# Linux-x86_64 下构建方式

- [Linux-x86_64 下构建方式](#linux-x86_64-下构建方式)

## 源码安装

- cmake

  **保证 cmake的版本 >= 3.14.0**。如果不是，可以参考以下命令安装 3.20.0 版本。如需获取其他版本，请参考[这里](https://cmake.org/install)。

  ```bash
  ```

- GCC 7+

  MMDeploy SDK 使用了 C++17 特性，因此需要安装gcc 7+以上的版本。

  ```bash
  # 如果 Ubuntu 版本 < 18.04，需要加入仓库

sudo ln -sf $(pwd)/cmake-3.20.0-linux-x86_64/bin/* /usr/bin/
  ```

- GCC 7+

  ```bash
  ```

### 安装依赖包

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
    <td>请参考<a href="https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html">官方说明</a>安装 conda。<br> 通过 conda 创建并激活 Python 环境。<br>
    </td>
  </tr>
  <tr>
    <td>安装 PyTorch，要求版本是 torch>=1.8.0。可查看<a href="https://pytorch.org/">官网</a>获取更多详细的安装教程。请确保 PyTorch 要求的 CUDA 版本和您主机的 CUDA 版本是一致<br>

</tr>
  <tr>
    </td>
  </tr>
  <tr>
    <td>参考如下命令安装 mmcv。更多安装方式，可查看 <a href="https://github.com/open-mmlab/mmcv">mmcv 官网</a><br>
    </td>
  </tr>
</tbody>
</table>

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
    <td>
    在 Ubuntu 18.04 及以上版本

<th>名称 </th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>
    在 Ubuntu 16.04 中，需要源码安装 OpenCV。您可以参考<a href="https://docs.opencv.org/3.4/d7/d9f/tutorial_linux_install.html">此处</a>.
    </td>

</tr>
  <tr>
    <td>pplcv 是 openPPL 开发的高性能图像处理库。 <b>此依赖项为可选项，只有在 cuda 平台下，才需安装。</b><br>
cd ppl.cv
   </td>
  </tr>
</tbody>
</table>

MMDeploy 的 Model Converter 和 SDK 共享推理引擎。您可以参考下文，选择自己感兴趣的推理引擎安装。

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
    <td>
    1. 安装 onnxruntime 的 python 包

<tr>
  </tr>
</thead>
<tbody>
  <tr>
    <td>
    2. 从<a href="https://github.com/microsoft/onnxruntime/releases/tag/v1.8.1">这里</a>下载 onnxruntime 的预编译包。参考如下命令，解压压缩包并设置环境变量
    </td>
  </tr>
  <tr>
    <td>
   1. 登录 <a href="https://www.nvidia.com/">NVIDIA 官网</a>，从<a href="https://developer.nvidia.com/nvidia-tensorrt-download">这里</a>选取并下载 TensorRT tar 包。要保证它和您机器的 CPU 架构以及 CUDA 版本是匹配的。<br>

<td>
   您可以参考这份<a href="https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html#installing-tar">指南</a>安装 TensorRT。<br>
   1. 这里也有一份 TensorRT 8.2 GA Update 2 在 Linux x86_64 和 CUDA 11.x 下的安装示例，供您参考。首先，点击<a href="https://developer.nvidia.com/compute/machine-learning/tensorrt/secure/8.2.3.0/tars/tensorrt-8.2.3.0.linux.x86_64-gnu.cuda-11.4.cudnn8.2.tar.gz">此处</a>下载 CUDA 11.x TensorRT 8.2.3.0。然后，根据如下命令，安装并配置 TensorRT 以及相关依赖。
   </td>
  </tr>
  <tr>

   </td>
  </tr>
  <tr>
    <td>
    1. 从 <a href="https://developer.nvidia.com/rdp/cudnn-archive">cuDNN Archive</a> 选择和您环境中 CPU 架构、CUDA 版本以及 TensorRT 版本配套的 cuDNN。以前文 TensorRT 安装说明为例，它需要 cudnn8.2。因此，可以下载 <a href="https://developer.nvidia.com/compute/machine-learning/cudnn/secure/8.2.1.32/11.3_06072021/cudnn-11.3-linux-x64-v8.2.1.32.tgz">CUDA 11.x cuDNN 8.2</a><br>
    2. 解压压缩包，并设置环境变量
   </td>
  </tr>
  <tr>
    <td>
    1. 请参考 ppl.nn 的 <a href="https://github.com/openppl-public/ppl.nn/blob/master/docs/en/building-from-source.md">安装文档</a> 编译 ppl.nn，并安装 pyppl<br>
    2. 将 pplnn 的根目录写入环境变量
cd ppl.nn

1. 请参考 ppl.nn 的 <a href="https://github.com/openppl-public/ppl.nn/blob/master/docs/en/building-from-source.md">安装文档</a> 编译 ppl.nn，并安装 pyppl<br>
cd ppl.nn
    </td>
  </tr>
  <tr>
    <td>1. 安装 <a href="https://docs.openvino.ai/2021.4/get_started.html">OpenVINO</a>
2. <b>可选</b>. 如果您想在 MMDeploy SDK 中使用 OpenVINO，请根据<a href="https://docs.openvino.ai/2021.4/openvino_docs_install_guides_installing_openvino_linux.html#install-openvino">指南</a>安装并配置它
    </td>
  </tr>
  <tr>
    <td>1. 请参考 ncnn的 <a href="https://github.com/Tencent/ncnn/wiki/how-to-build">wiki</a> 编译 ncnn。
编译时，请打开<code>-DNCNN_PYTHON=ON</code><br>
2. 将 ncnn 的根目录写入环境变量
cd ncnn
3. 安装 pyncnn
    </td>
  </tr>

cd ncnn
    </td>
  </tr>
  <tr>
  <td>
  1. Download libtorch from <a href="https://pytorch.org/get-started/locally/">here</a>. Please note that only <b>Pre-cxx11 ABI</b> and <b>version 1.8.1+</b> on Linux platform are supported by now. For previous versions of libtorch, you can find them in the <a href="https://github.com/pytorch/pytorch/issues/40961#issuecomment-1017317786">issue comment</a>. <br>
  2. Take Libtorch1.8.1+cu111 as an example. You can install it like this:
  </td>
  </tr>
  <tr>

  </td>
  </tr>
  <tr>
    <td>
    1. 按照 <a href="https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/60RC1alpha02/softwareinstall/instg/atlasdeploy_03_0002.html">官方指引</a> 安装 CANN 工具集.<br>
    2. 配置环境
    </td>
  </tr>
  <tr>
    <td>TVM</td>
    <td>
    1. 按照 <a href="https://tvm.apache.org/docs/install/from_source.html">官方指引</a>安装 TVM.<br>
    2. 配置环境
    </td>
  </tr>
</tbody>
</table>

注意: <br>
如果您想使上述环境变量永久有效，可以把它们加入<code>~/.bashrc</code>。以 ONNXRuntime 的环境变量为例，

```bash

    </td>
  </tr>
</tbody>
</table>

注意: <br>

```bash
```

```bash
```

如果您选择了ONNXRuntime，TensorRT，ncnn 和 torchscript 任一种推理后端，您需要编译对应的自定义算子库。

- **ONNXRuntime** 自定义算子

  ```bash
  ```

- **TensorRT** 自定义算子

  ```bash

```bash
  ```

- **ncnn** 自定义算子

  ```bash
  ```

  ```bash
  cmake -DMMDEPLOY_TARGET_BACKENDS=torchscript -DTorch_DIR=${Torch_DIR} ..
  ```

```bash
```

**注意**

- cuda10 建议安装[补丁包](https://developer.nvidia.com/cuda-10.2-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=runfilelocal)，否则模型运行可能出现 GEMM 相关错误

下文展示一些构建 SDK 的样例。您可以参考它们，激活其他的推理引擎。

  ```Bash
  cmake .. \

  ```

  ```Bash
  cmake .. \

  ```

  ```Bash
  cmake .. \

  ```

- pplnn

  ```Bash
  cmake .. \

-DMMDEPLOY_TARGET_BACKENDS=pplnn \

  ```

  如果使用了 [ncnn 自动安装脚本](../../../tools/scripts/build_ubuntu_x64_ncnn.py), protobuf 会安装在 mmdeploy 同级目录的 mmdeploy-dep/pbinstall 中。

  ```Bash
  cmake .. \

-DTENSORRT_DIR=${TENSORRT_DIR} \
```

# SOLOv2

> [SOLOv2: Dynamic and Fast Instance Segmentation](https://arxiv.org/abs/2003.10152)

In this work, we aim at building a simple, direct, and fast instance segmentation
framework with strong performance. We follow the principle of the SOLO method of
Wang et al. "SOLO: segmenting objects by locations". Importantly, we take one
step further by dynamically learning the mask head of the object segmenter such
that the mask head is conditioned on the location. Specifically, the mask branch
is decoupled into a mask kernel branch and mask feature branch, which are
responsible for learning the convolution kernel and the convolved features
respectively. Moreover, we propose Matrix NMS (non maximum suppression) to
significantly reduce the inference time overhead due to NMS of masks. Our
Matrix NMS performs NMS with parallel matrix operations in one shot, and
yields better results. We demonstrate a simple direct instance segmentation

system, outperforming a few state-of-the-art methods in both speed and accuracy.
A light-weight version of SOLOv2 executes at 31.3 FPS and yields 37.1% AP.
Moreover, our state-of-the-art results in object detection (from our mask byproduct)
and panoptic segmentation show the potential to serve as a new strong baseline
for many instance-level recognition tasks besides instance segmentation.

<img src="https://user-images.githubusercontent.com/48282753/167235090-f20dab74-43a5-44ed-9f11-4e5f08866f45.png"/>
</div>

### SOLOv2

</div>

### SOLOv2

|  Backbone  |  Style  | MS train | Lr schd | Mem (GB) | mask AP |                    Config                     |                                                                                                                                                Download                                                                                                                                                |

| :--------: | :-----: | :------: | :-----: | :------: | :-----: | :-------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|    R-50    | pytorch |    N     |   1x    |   5.1    |  34.8   |     [config](./solov2_r50_fpn_1x_coco.py)     |      [model](https://download.openmmlab.com/mmdetection/v2.0/solov2/solov2_r50_fpn_1x_coco/solov2_r50_fpn_1x_coco_20220512_125858-a357fa23.pth)           \| [log](https://download.openmmlab.com/mmdetection/v2.0/solov2/solov2_r50_fpn_1x_coco/solov2_r50_fpn_1x_coco_20220512_125858.log.json)      |

|    R-50    | pytorch |    Y     |   3x    |   5.1    |  37.5   |   [config](./solov2_r50_fpn_ms-3x_coco.py)    |      [model](https://download.openmmlab.com/mmdetection/v2.0/solov2/solov2_r50_fpn_3x_coco/solov2_r50_fpn_3x_coco_20220512_125856-fed092d4.pth)           \| [log](https://download.openmmlab.com/mmdetection/v2.0/solov2/solov2_r50_fpn_3x_coco/solov2_r50_fpn_3x_coco_20220512_125856.log.json)      |
|   R-101    | pytorch |    Y     |   3x    |   6.9    |  39.1   |   [config](./solov2_r101_fpn_ms-3x_coco.py)   |     [model](https://download.openmmlab.com/mmdetection/v2.0/solov2/solov2_r101_fpn_3x_coco/solov2_r101_fpn_3x_coco_20220511_095119-c559a076.pth)         \| [log](https://download.openmmlab.com/mmdetection/v2.0/solov2/solov2_r101_fpn_3x_coco/solov2_r101_fpn_3x_coco_20220511_095119.log.json)     |

| R-101(DCN) | pytorch |    Y     |   3x    |   7.1    |  41.2   | [config](./solov2_r101-dcn_fpn_ms-3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/solov2/solov2_r101_dcn_fpn_3x_coco/solov2_r101_dcn_fpn_3x_coco_20220513_214734-16c966cb.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/solov2/solov2_r101_dcn_fpn_3x_coco/solov2_r101_dcn_fpn_3x_coco_20220513_214734.log.json) |
| X-101(DCN) | pytorch |    Y     |   3x    |   11.3   |  42.4   | [config](./solov2_x101-dcn_fpn_ms-3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/solov2/solov2_x101_dcn_fpn_3x_coco/solov2_x101_dcn_fpn_3x_coco_20220513_214337-aef41095.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/solov2/solov2_x101_dcn_fpn_3x_coco/solov2_x101_dcn_fpn_3x_coco_20220513_214337.log.json) |

### Light SOLOv2

| Backbone |  Style  | MS train | Lr schd | Mem (GB) | mask AP |                     Config                     |                                                                                                                                                  Download                                                                                                                                                  |
| :------: | :-----: | :------: | :-----: | :------: | :-----: | :--------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

|   R-18   | pytorch |    Y     |   3x    |   9.1    |  29.7   | [config](./solov2-light_r18_fpn_ms-3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/solov2/solov2_light_r18_fpn_3x_coco/solov2_light_r18_fpn_3x_coco_20220511_083717-75fa355b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/solov2/solov2_light_r18_fpn_3x_coco/solov2_light_r18_fpn_3x_coco_20220511_083717.log.json) |
|   R-34   | pytorch |    Y     |   3x    |   9.3    |  31.9   | [config](./solov2-light_r34_fpn_ms-3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/solov2/solov2_light_r34_fpn_3x_coco/solov2_light_r34_fpn_3x_coco_20220511_091839-e51659d3.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/solov2/solov2_light_r34_fpn_3x_coco/solov2_light_r34_fpn_3x_coco_20220511_091839.log.json) |

|   R-50   | pytorch |    Y     |   3x    |   9.9    |  33.7   | [config](./solov2-light_r50_fpn_ms-3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/solov2/solov2_light_r50_fpn_3x_coco/solov2_light_r50_fpn_3x_coco_20220512_165256-c93a6074.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/solov2/solov2_light_r50_fpn_3x_coco/solov2_light_r50_fpn_3x_coco_20220512_165256.log.json) |

```latex
@article{wang2020solov2,
  title={SOLOv2: Dynamic and Fast Instance Segmentation},
  author={Wang, Xinlong and Zhang, Rufeng and  Kong, Tao and Li, Lei and Shen, Chunhua},
  journal={Proc. Advances in Neural Information Processing Systems (NeurIPS)},
}
```

# Cross-Domain Adaptation for Animal Pose Estimation

<details>

```bibtex
}
```

# Frequently Asked Questions

We list some common troubles faced by many users and their corresponding solutions here. Feel free to enrich the list if you find any frequent issues and have ways to help others to solve them. If the contents here do not cover your issue, please create an issue using the [provided templates](https://github.com/open-mmlab/mmdetection/blob/main/.github/ISSUE_TEMPLATE/error-report.md/) and make sure you fill in all required information in the template.

## PyTorch 2.0 Support

The vast majority of algorithms in MMDetection now support PyTorch 2.0 and its `torch.compile` function. Users only need to install MMDetection 3.0.0rc7 or later versions to enjoy this feature. If any unsupported algorithms are found during use, please feel free to give us feedback. We also welcome contributions from the community to benchmark the speed improvement brought by using the `torch.compile` function.

To enable the `torch.compile` function, simply add `--cfg-options compile=True` after `train.py` or `test.py`. For example, to enable `torch.compile` for RTMDet, you can use the following command:

```shell
# Single GPU
python tools/train.py configs/rtmdet/rtmdet_s_8xb32-300e_coco.py  --cfg-options compile=True

# Single node multiple GPUs
./tools/dist_train.sh configs/rtmdet/rtmdet_s_8xb32-300e_coco.py 8 --cfg-options compile=True

# Single node multiple GPUs + AMP
./tools/dist_train.sh configs/rtmdet/rtmdet_s_8xb32-300e_coco.py 8 --cfg-options compile=True --amp
```

It is important to note that PyTorch 2.0's support for dynamic shapes is not yet fully developed. In most object detection algorithms, not only are the input shapes dynamic, but the loss calculation and post-processing parts are also dynamic. This can lead to slower training speeds when using the `torch.compile` function. Therefore, if you wish to enable the `torch.compile` function, you should follow these principles:

1. Input images to the network are fixed shape, not multi-scale
2. set `torch._dynamo.config.cache_size_limit` parameter. TorchDynamo will convert and cache the Python bytecode, and the compiled functions will be stored in the cache. When the next check finds that the function needs to be recompiled, the function will be recompiled and cached. However, if the number of recompilations exceeds the maximum value set (64), the function will no longer be cached or recompiled. As mentioned above, the loss calculation and post-processing parts of the object detection algorithm are also dynamically calculated, and these functions need to be recompiled every time. Therefore, setting the `torch._dynamo.config.cache_size_limit` parameter to a smaller value can effectively reduce the compilation time

In MMDetection, you can set the `torch._dynamo.config.cache_size_limit` parameter through the environment variable `DYNAMO_CACHE_SIZE_LIMIT`. For example, the command is as follows:

```shell

```shell
export DYNAMO_CACHE_SIZE_LIMIT = 4

```

About the common questions about PyTorch 2.0's dynamo, you can refer to [here](https://pytorch.org/docs/stable/dynamo/faq.html)

Compatibility issue between MMCV and MMDetection; "ConvWS is already registered in conv layer"; "AssertionError: MMCV==xxx is used but incompatible. Please install mmcv>=xxx, \<=xxx."

Compatible MMDetection, MMEngine, and MMCV versions are shown as below. Please choose the correct version of MMCV to avoid installation issues.

| MMDetection version |      MMCV version       |     MMEngine version     |
| :-----------------: | :---------------------: | :----------------------: |
|        main         |  mmcv>=2.0.0, \<2.1.0   | mmengine>=0.7.1, \<1.0.0 |
|        3.1.0        |  mmcv>=2.0.0, \<2.1.0   | mmengine>=0.7.1, \<1.0.0 |
|        3.0.0        |  mmcv>=2.0.0, \<2.1.0   | mmengine>=0.7.1, \<1.0.0 |
|      3.0.0rc6       | mmcv>=2.0.0rc4, \<2.1.0 | mmengine>=0.6.0, \<1.0.0 |
|      3.0.0rc5       | mmcv>=2.0.0rc1, \<2.1.0 | mmengine>=0.3.0, \<1.0.0 |
|      3.0.0rc4       | mmcv>=2.0.0rc1, \<2.1.0 | mmengine>=0.3.0, \<1.0.0 |
|      3.0.0rc3       | mmcv>=2.0.0rc1, \<2.1.0 | mmengine>=0.3.0, \<1.0.0 |
|      3.0.0rc2       | mmcv>=2.0.0rc1, \<2.1.0 | mmengine>=0.1.0, \<1.0.0 |
|      3.0.0rc1       | mmcv>=2.0.0rc1, \<2.1.0 | mmengine>=0.1.0, \<1.0.0 |

|      3.0.0rc0       | mmcv>=2.0.0rc1, \<2.1.0 | mmengine>=0.1.0, \<1.0.0 |

**Note:**

1. If you want to install mmdet-v2.x, the compatible MMDetection and MMCV versions table can be found at [here](https://mmdetection.readthedocs.io/en/stable/faq.html#installation). Please choose the correct version of MMCV to avoid installation issues.
2. In MMCV-v2.x, `mmcv-full` is rename to `mmcv`, if you want to install `mmcv` without CUDA ops, you can install `mmcv-lite`.

- "No module named 'mmcv.ops'"; "No module named 'mmcv.\_ext'".

  1. Uninstall existing `mmcv-lite` in the environment using `pip uninstall mmcv-lite`.
  2. Install `mmcv` following the [installation instruction](https://mmcv.readthedocs.io/en/2.x/get_started/installation.html).

- "Microsoft Visual C++ 14.0 or graeter is required" during installation on Windows.

  This error happens when building the 'pycocotools.\_mask' extension of pycocotools and the environment lacks corresponding C++ compilation dependencies. You need to download it at Microsoft officials [visual-cpp-build-tools](https://visualstudio.microsoft.com/zh-hans/visual-cpp-build-tools/),  select the "Use C ++ Desktop Development" option to install the minimum dependencies, and then reinstall pycocotools.

- Using Albumentations

  If you would like to use `albumentations`, we suggest using `pip install -r requirements/albu.txt` or
  `pip install -U albumentations --no-binary qudida,albumentations`.
  If you simply use `pip install albumentations>=0.3.2`, it will install `opencv-python-headless` simultaneously (even though you have already installed `opencv-python`).

If you simply use `pip install albumentations>=0.3.2`, it will install `opencv-python-headless` simultaneously (even though you have already installed `opencv-python`).
  Please refer to the [official documentation](https://albumentations.ai/docs/getting_started/installation/#note-on-opencv-dependencies) for details.

- ModuleNotFoundError is raised when using some algorithms

  Some extra dependencies are required for Instaboost, Panoptic Segmentation, LVIS dataset, etc. Please note the error message and install corresponding packages, e.g.,

  ```shell
  # for instaboost
  pip install instaboostfast
  # for panoptic segmentation
  pip install git+https://github.com/cocodataset/panopticapi.git
  # for LVIS dataset
  pip install git+https://github.com/lvis-dataset/lvis-api.git
  ```

## Coding

- Do I need to reinstall mmdet after some code modifications

# for LVIS dataset
  ```

## Coding

  If you follow the best practice and install mmdet with `pip install -e .`, any local modifications made to the code will take effect without reinstallation.

- How to develop with multiple MMDetection versions

  You can have multiple folders like mmdet-3.0, mmdet-3.1.
  When you run the train or test script, it will adopt the mmdet package in the current folder.

  To use the default MMDetection installed in the environment rather than the one you are working with, you can remove the following line in those scripts:

  ```shell
  PYTHONPATH="$(dirname $0)/..":$PYTHONPATH
  ```

## PyTorch/CUDA Environment

- "RTX 30 series card fails when building MMCV or MMDet"

  1. Temporary work-around: do `MMCV_WITH_OPS=1 MMCV_CUDA_ARGS='-gencode=arch=compute_80,code=sm_80' pip install -e .`.

     The common issue is `nvcc fatal : Unsupported gpu architecture 'compute_86'`. This means that the compiler should optimize for sm_86, i.e., nvidia 30 series card, but such optimizations have not been supported by CUDA toolkit 11.0.
     This work-around modifies the compile flag by adding `MMCV_CUDA_ARGS='-gencode=arch=compute_80,code=sm_80'`, which tells `nvcc` to optimize for **sm_80**, i.e., Nvidia A100. Although A100 is different from the 30 series card, they use similar ampere architecture. This may hurt the performance but it works.
  2. PyTorch developers have updated that the default compiler flags should be fixed by [pytorch/pytorch#47585](https://github.com/pytorch/pytorch/pull/47585). So using PyTorch-nightly may also be able to solve the problem, though we have not tested it yet.

- "invalid device function" or "no kernel image is available for execution".

  1. Check if your cuda runtime version (under `/usr/local/`), `nvcc --version` and `conda list cudatoolkit` version match.
  2. Run `python mmdet/utils/collect_env.py` to check whether PyTorch, torchvision, and MMCV are built for the correct GPU architecture.
     You may need to set `TORCH_CUDA_ARCH_LIST` to reinstall MMCV.
     The GPU arch table could be found [here](https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#gpu-feature-list),
     i.e. run `TORCH_CUDA_ARCH_LIST=7.0 pip install mmcv` to build MMCV for Volta GPUs.
     The compatibility issue could happen when using old GPUS, e.g., Tesla K80 (3.7) on colab.
  3. Check whether the running environment is the same as that when mmcv/mmdet has compiled.
     For example, you may compile mmcv using CUDA 10.0 but run it on CUDA 9.0 environments.

- "undefined symbol" or "cannot open xxx.so".

For example, you may compile mmcv using CUDA 10.0 but run it on CUDA 9.0 environments.

  1. If those symbols are CUDA/C++ symbols (e.g., libcudart.so or GLIBCXX), check whether the CUDA/GCC runtimes are the same as those used for compiling mmcv,
     i.e. run `python mmdet/utils/collect_env.py` to see if `"MMCV Compiler"`/`"MMCV CUDA Compiler"` is the same as `"GCC"`/`"CUDA_HOME"`.
  2. If those symbols are PyTorch symbols (e.g., symbols containing caffe, aten, and TH), check whether the PyTorch version is the same as that used for compiling mmcv.
  3. Run `python mmdet/utils/collect_env.py` to check whether PyTorch, torchvision, and MMCV are built by and running on the same environment.

- setuptools.sandbox.UnpickleableException: DistutilsSetupError("each element of 'ext_modules' option must be an Extension instance or 2-tuple")

  1. If you are using miniconda rather than anaconda, check whether Cython is installed as indicated in [#3379](https://github.com/open-mmlab/mmdetection/issues/3379).
     You need to manually install Cython first and then run command `pip install -r requirements.txt`.
  2. You may also need to check the compatibility between the `setuptools`, `Cython`, and `PyTorch` in your environment.

- "Segmentation fault".

  1. Check you GCC version and use GCC 5.4. This usually caused by the incompatibility between PyTorch and the environment (e.g., GCC \< 4.9 for PyTorch). We also recommend the users to avoid using GCC 5.5 because many feedbacks report that GCC 5.5 will cause "segmentation fault" and simply changing it to GCC 5.4 could solve the problem.

2. Check whether PyTorch is correctly installed and could use CUDA op, e.g. type the following command in your terminal.

     ```shell
     python -c 'import torch; print(torch.cuda.is_available())'
     ```

     And see whether they could correctly output results.

  3. If Pytorch is correctly installed, check whether MMCV is correctly installed.

     ```shell
     python -c 'import mmcv; import mmcv.ops'
     ```

     If MMCV is correctly installed, then there will be no issue of the above two commands.

  4. If MMCV and Pytorch is correctly installed, you man use `ipdb`, `pdb` to set breakpoints or directly add 'print' in mmdetection code and see which part leads the segmentation fault.

- "Loss goes Nan"

  1. Check if the dataset annotations are valid: zero-size bounding boxes will cause the regression loss to be Nan due to the commonly used transformation for box regression. Some small size (width or height are smaller than 1) boxes will also cause this problem after data augmentation (e.g., instaboost). So check the data and try to filter out those zero-size boxes and skip some risky augmentations on the small-size boxes when you face the problem.
  2. Reduce the learning rate: the learning rate might be too large due to some reasons, e.g., change of batch size. You can rescale them to the value that could stably train the model.
  3. Extend the warmup iterations: some models are sensitive to the learning rate at the start of the training. You can extend the warmup iterations, e.g., change the `warmup_iters` from 500 to 1000 or 2000.

4. Add gradient clipping: some models requires gradient clipping to stabilize the training process. The default of `grad_clip` is `None`, you can add gradient clippint to avoid gradients that are too large, i.e., set `optim_wrapper=dict(clip_grad=dict(max_norm=35, norm_type=2))` in your config file.

- "GPU out of memory"

  1. There are some scenarios when there are large amount of ground truth boxes, which may cause OOM during target assignment. You can set `gpu_assign_thr=N` in the config of assigner thus the assigner will calculate box overlaps through CPU when there are more than N GT boxes.

  2. Set `with_cp=True` in the backbone. This uses the sublinear strategy in PyTorch to reduce GPU memory cost in the backbone.

  3. Try mixed precision training using following the examples in `config/fp16`. The `loss_scale` might need further tuning for different models.

3. Try mixed precision training using following the examples in `config/fp16`. The `loss_scale` might need further tuning for different models.

  4. Try to use `AvoidCUDAOOM` to avoid GPU out of memory. It will first retry after calling `torch.cuda.empty_cache()`. If it still fails, it will then retry by converting the type of inputs to FP16 format. If it still fails, it will try to copy inputs from GPUs to CPUs to continue computing. Try AvoidOOM in you code to make the code continue to run when GPU memory runs out:

     ```python
     from mmdet.utils import AvoidCUDAOOM

     output = AvoidCUDAOOM.retry_if_cuda_oom(some_function)(input1, input2)
     ```

     You can also try `AvoidCUDAOOM` as a decorator to make the code continue to run when GPU memory runs out:

     ```python

     @AvoidCUDAOOM.retry_if_cuda_oom
     def function(*args, **kwargs):
         ...
         return xxx
     ```

```python

         ...
         return xxx
     ```

- "RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one"

  1. This error indicates that your module has parameters that were not used in producing loss. This phenomenon may be caused by running different branches in your code in DDP mode.
  2. You can set `find_unused_parameters = True` in the config to solve the above problems, but this will slow down the training speed.

2. You can set `find_unused_parameters = True` in the config to solve the above problems, but this will slow down the training speed.
  3. You can set `detect_anomalous_params = True` in the config or `model_wrapper_cfg = dict(type='MMDistributedDataParallel', detect_anomalous_params=True)` (More details please refer to [MMEngine](https://github.com/open-mmlab/mmengine/blob/main/mmengine/model/wrappers/distributed.py#L91)) to get the name of those unused parameters. Note `detect_anomalous_params = True` will slow down the training speed, so it is recommended for debugging only.

- Save the best model

  It can be turned on by configuring `default_hooks = dict(checkpoint=dict(type='CheckpointHook', interval=1, save_best='auto'),`. In the case of the `auto` parameter, the first key in the returned evaluation result will be used as the basis for selecting the best model. You can also directly set the key in the evaluation result to manually set it, for example, `save_best='coco/bbox_mAP'`.

- COCO Dataset, AP or AR = -1
  1. According to the definition of COCO dataset, the small and medium areas in an image are less than 1024 (32\*32), 9216 (96\*96), respectively.
  2. If the corresponding area has no object, the result of AP and AR will set to -1.

## Model

- `style` in ResNet

2. If the corresponding area has no object, the result of AP and AR will set to -1.

## Model

  The `style` parameter in ResNet allows either `pytorch` or `caffe` style. It indicates the difference in the Bottleneck module. Bottleneck is a stacking structure of `1x1-3x3-1x1` convolutional layers. In the case of `caffe` mode, the convolution layer with `stride=2` is the first `1x1` convolution, while in `pyorch` mode, it is the second `3x3` convolution has `stride=2`. A sample code is as below:

  ```python
  if self.style == 'pytorch':
        self.conv1_stride = 1
        self.conv2_stride = stride
  else:
        self.conv1_stride = stride
        self.conv2_stride = 1
  ```

- ResNeXt parameter description

self.conv1_stride = 1
  else:
  ```

  ResNeXt comes from the paper [`Aggregated Residual Transformations for Deep Neural Networks`](https://arxiv.org/abs/1611.05431). It introduces  group and uses “cardinality” to control the number of groups to achieve a balance between accuracy and complexity. It controls the basic width and grouping parameters of the internal Bottleneck module through two hyperparameters `baseWidth` and `cardinality`. An example configuration name in MMDetection is `mask_rcnn_x101_64x4d_fpn_mstrain-poly_3x_coco.py`, where `mask_rcnn` represents the algorithm using Mask R-CNN, `x101` represents the backbone network using ResNeXt-101, and `64x4d` represents that the bottleneck block has 64 group and each group has basic width of 4.

- `norm_eval` in backbone

  Since the detection model is usually large and the input image resolution is high, this will result in a small batch of the detection model, which will make the variance of the statistics calculated by BatchNorm during the training process very large and not as stable as the statistics obtained during the pre-training of the backbone network . Therefore, the `norm_eval=True` mode is generally used in training, and the BatchNorm statistics in the pre-trained backbone network are directly used. The few algorithms that use large batches are the `norm_eval=False` mode, such as NASFPN. For the backbone network without ImageNet pre-training and the batch is relatively small, you can consider using `SyncBN`.

# Migration of Runtime Settings

We update runtime settings in MMagic 1.x. Important modifications are as following.

- The `checkpoint_config` is moved to `default_hooks.checkpoint` and the `log_config` is moved to `default_hooks.logger`. And we move many hooks settings from the script code to the `default_hooks` field in the runtime configuration.
- The `resume_from` is removed. And we use `resume` to replace it.
  - If resume=True and load_from is not None, resume training from the checkpoint in load_from.
  - If resume=True and load_from is None, try to resume from the latest checkpoint in the work directory.
  - If resume=False and load_from is not None, only load the checkpoint, not resume training.
  - If resume=False and load_from is None, do not load nor resume.
- The `dist_params` field is a sub field of `env_cfg` now. And there are some new configurations in the `env_cfg`.
- The `workflow` related functionalities are removed.

- New field `visualizer`: The visualizer is a new design. We use a visualizer instance in the runner to handle results & log visualization and save to different backends,  like Local, TensorBoard and Wandb.
- New field `default_scope`: The start point to search module for all registries.

<thead>
  <tr>
<tbody>
<tr>

```python
checkpoint_config = dict(  # Config to set the checkpoint hook, Refer to https://github.com/open-mmlab/mmcv/blob/master/mmcv/runner/hooks/checkpoint.py for implementation.
    interval=5000,  # The save interval is 5000 iterations
    save_optimizer=True,  # Also save optimizers
    by_epoch=False)  # Count by iterations
log_config = dict(  # Config to register logger hook

save_optimizer=True,  # Also save optimizers
    interval=100,  # Interval to print the log
    hooks=[
        dict(type='TextLoggerHook', by_epoch=False),  # The logger used to record the training process
        dict(type='TensorboardLoggerHook'),  # The Tensorboard logger is also supported
    ])
visual_config = None  # Visual config, we do not use it.
dist_params = dict(backend='nccl')  # Parameters to setup distributed training, the port can also be set
log_level = 'INFO'  # The level of logging
load_from = None # load models as a pre-trained model from a given path. This will not resume training
resume_from = None # Resume checkpoints from a given path, the training will be resumed from the iteration when the checkpoint's is saved

workflow = [('train', 1)]  # Workflow for runner. [('train', 1)] means there is only one workflow and the workflow named 'train' is executed once. Keep this unchanged when training current matting models
```

</td>

```python
default_hooks = dict(  # Used to build default hooks
    checkpoint=dict(  # Config to set the checkpoint hook
        interval=5000,  # The save interval is 5000 iterations
        by_epoch=False,  # Count by iterations
        out_dir=save_dir,
    ),
    logger=dict(type='LoggerHook', interval=100),  # Config to register logger hook
)
default_scope = 'mmedit' # Used to set registries location

param_scheduler=dict(type='ParamSchedulerHook'),
)
env_cfg = dict(  # Parameters to setup distributed training, the port can also be set
    mp_cfg=dict(mp_start_method='fork', opencv_num_threads=4),
)
log_processor = dict(type='LogProcessor', window_size=100, by_epoch=False)  # Used to build log processor
resume = False  # Resume checkpoints from a given path, the training will be resumed from the epoch when the checkpoint's is saved.
```

</td>

</tr>
</thead>
</table>

<details>
<summary align="right">LIIF (CVPR'2021)</summary>

```bibtex
}
```

<br/>

|                             算法                              | scale | Set5<br>PSNR / SSIM | Set14<br>PSNR / SSIM | DIV2K <br>PSNR / SSIM |   GPU 信息   |                              下载                              |
| :-----------------------------------------------------------: | :---: | :-----------------: | :------------------: | :-------------------: | :----------: | :------------------------------------------------------------: |

| [liif_edsr_norm_c64b16_g1_1000k_div2k](./liif-edsr-norm_c64b16_1xb16-1000k_div2k.py) |  x2   |  35.7131 / 0.9366   |   31.5579 / 0.8889   |   34.6647 / 0.9355    | 1 (TITAN Xp) | [模型](https://download.openmmlab.com/mmediting/restorers/liif/liif_edsr_norm_c64b16_g1_1000k_div2k_20210715-ab7ce3fc.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/liif/liif_edsr_norm_c64b16_g1_1000k_div2k_20210715-ab7ce3fc.log.json) |
|                               △                               |  x3   |  32.3805 / 0.8915   |   28.4605 / 0.8039   |   30.9808 / 0.8724    |      △       |                               △                                |
|                               △                               |  x4   |  30.2748 / 0.8509   |   26.8415 / 0.7381   |   29.0245 / 0.8187    |      △       |                               △                                |

|                               △                               |  x6   |  27.1187 / 0.7774   |   24.7461 / 0.6444   |   26.7770 / 0.7425    |      △       |                               △                                |
|                               △                               |  x18  |  20.8516 / 0.5406   |   20.0096 / 0.4525   |   22.1987 / 0.5955    |      △       |                               △                                |
|                               △                               |  x30  |  18.8467 / 0.5010   |   18.1321 / 0.3963   |   20.5050 / 0.5577    |      △       |                               △                                |

| [liif_rdn_norm_c64b16_g1_1000k_div2k](./liif-rdn-norm_c64b16_1xb16-1000k_div2k.py) |  x2   |  35.7874 / 0.9366   |   31.6866 / 0.8896   |   34.7548 / 0.9356    | 1 (TITAN Xp) | [模型](https://download.openmmlab.com/mmediting/restorers/liif/liif_rdn_norm_c64b16_g1_1000k_div2k_20210717-22d6fdc8.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/liif/liif_rdn_norm_c64b16_g1_1000k_div2k_20210717-22d6fdc8.log.json) |
|                               △                               |  x3   |  32.4992 / 0.8923   |   28.4905 / 0.8037   |   31.0744 / 0.8731    |      △       |                               △                                |
|                               △                               |  x4   |  30.3835 / 0.8513   |   26.8734 / 0.7373   |   29.1101 / 0.8197    |      △       |                               △                                |

|                               △                               |  x6   |  27.1914 / 0.7751   |   24.7824 / 0.6434   |   26.8693 / 0.7437    |      △       |                               △                                |
|                               △                               |  x18  |  20.8913 / 0.5329   |   20.1077 / 0.4537   |   22.2972 / 0.5950    |      △       |                               △                                |
|                               △                               |  x30  |  18.9354 / 0.4864   |   18.1448 / 0.3942   |   20.5663 / 0.5560    |      △       |                               △                                |

注：

- △ 指同上。
- 这两个配置仅在 _testing pipeline_ 上有所不同。 所以他们使用相同的检查点。
- 数据根据 [EDSR](../edsr/README.md) 进行正则化。
- 在 RGB 通道上进行评估，在评估之前裁剪每个边界中的 `scale` 像素。

## 快速开始

**训练**

<details>

```shell
# CPU上训练

## 快速开始

**训练**

<details>

```shell
# CPU上训练

# 单个GPU上训练

# 多个GPU上训练
```

**测试**

<details>

```shell
# CPU上测试

# 单个GPU上测试

# 单个GPU上测试

# 多个GPU上测试
```

# 可视化

阅读本文前建议先阅读 {external+mmengine:doc}`MMEngine: 可视化 <advanced_tutorials/visualization>` 以初步了解 Visualizer 的定义及相关用法。

简单来说，MMEngine 中实现了用于满足日常可视化需求的可视化器件 [`Visualizer`](mmengine.visualization.Visualizer)，其主要包含三个功能：

- 实现了常用的绘图 API，例如 [`draw_bboxes`](mmengine.visualization.Visualizer.draw_bboxes) 实现了边界盒的绘制功能，[`draw_lines`](mmengine.visualization.Visualizer.draw_lines) 实现了线条的绘制功能。
- 支持将可视化结果、学习率曲线、损失函数曲线以及验证精度曲线等写入多种后端中，包括本地磁盘以及常用的深度学习训练日志记录工具，如 [TensorBoard](https://www.tensorflow.org/tensorboard) 和 [WandB](https://wandb.ai/site)。
- 支持在代码中的任意位置进行调用，例如在训练或测试过程中可视化或记录模型的中间状态，如特征图及验证结果等。

基于 MMEngine 的 Visualizer，MMOCR 内预置了多种可视化工具，用户仅需简单修改配置文件即可使用：

- `tools/analysis_tools/browse_dataset.py` 脚本提供了数据集可视化功能，其可以绘制经过数据变换（Data Transforms）之后的图像及对应的标注内容，详见 [`browse_dataset.py`](useful_tools.md)。

- MMEngine 中实现了 `LoggerHook`，该 Hook 利用 `Visualizer` 将学习率、损失以及评估结果等数据写入  `Visualizer` 设置的后端中，因此通过修改配置文件中的 `Visualizer` 后端，比如修改为`TensorBoardVISBackend` 或 `WandbVISBackend`，可以实现将日志到 `TensorBoard` 或 `WandB` 等常见的训练日志记录工具中，从而方便用户使用这些可视化工具来分析和监控训练流程。
- MMOCR 中实现了`VisualizerHook`，该 Hook 利用 `Visualizer` 将验证阶段或预测阶段的预测结果进行可视化或储存至 `Visualizer` 设置的后端中，因此通过修改配置文件中的 `Visualizer` 后端，比如修改为`TensorBoardVISBackend` 或 `WandbVISBackend`，可以实现将预测的图像存储到 `TensorBoard` 或 `Wandb`中。

## 配置

得益于注册机制的使用，在 MMOCR 中，我们可以通过修改配置文件来设置可视化器件 `Visualizer` 的行为。通常，我们在 `task/_base_/default_runtime.py` 中定义可视化相关的默认配置， 详见[配置教程](config.md)。

```Python
    type='TextxxxLocalVisualizer',  # 不同任务使用不同的可视化器
```

```Python
```

依据以上示例，我们可以看出 `Visualizer` 的配置主要由两个部分组成，即，`Visualizer`的类型以及其采用的可视化后端 `vis_backends`。

- 针对不同的 OCR 任务，MMOCR 中预置了多种可视化器件，包括 [`TextDetLocalVisualizer`](mmocr.visualization.TextDetLocalVisualizer)，[`TextRecogLocalVisualizer`](mmocr.visualization.TextRecogLocalVisualizer)，[`TextSpottingLocalVisualizer`](mmocr.visualization.TextSpottingLocalVisualizer) 以及[`KIELocalVisualizer`](mmocr.visualization.KIELocalVisualizer)。这些可视化器件依照自身任务的特点对基础的 Visulizer API 进行了拓展，并实现了相应的标签信息接口 `add_datasamples`。例如，用户可以直接使用 `TextDetLocalVisualizer` 来可视化文本检测任务的标签或预测结果。
- MMOCR 默认将可视化后端 `vis_backend` 设置为本地可视化后端 `LocalVisBackend`，将所有可视化结果及其他训练信息保存在本地文件夹中。

## 存储

## 存储

MMOCR 默认使用本地可视化后端 [`LocalVisBackend`](mmengine.visualization.LocalVisBackend)，`VisualizerHook` 和`LoggerHook` 中存储的模型损失、学习率、模型评估精度以及可视化结果等信息将被默认保存至`{work_dir}/{config_name}/{time}/{vis_data}` 文件夹。此外，MMOCR 也支持其它常用的可视化后端，如 `TensorboardVisBackend` 以及 `WandbVisBackend`用户只需要将配置文件中的 `vis_backends` 类型修改为对应的可视化后端即可。例如，用户只需要在配置文件中插入以下代码块，即可将数据存储至 `TensorBoard` 以及 `WandB`中。

```Python
```

## 绘制

### 绘制预测结果信息

MMOCR 主要利用 [`VisualizationHook`](mmocr.engine.hooks.VisualizationHook)validation 和 test 的预测结果, 默认情况下 `VisualizationHook`为关闭状态，默认配置如下：

```Python
visualization=dict( # 用户可视化 validation 和 test 的结果
```

下表为 `VisualizationHook` 支持的参数：

type='VisualizationHook',
```

|   参数    |                                        说明                                         |
| :-------: | :---------------------------------------------------------------------------------: |
|  enable   |          VisualizationHook 的开启和关闭由参数enable控制默认是关闭的状态，           |
| interval  | 在VisualizationHook开启的情况下,用以控制多少iteration 存储或展示 val 或 test 的结果 |
|   show    |                          控制是否可视化 val 或 test 的结果                          |
|  draw_gt  |                         val 或 test 的结果是否绘制标注信息                          |
| draw_pred |                         val 或 test 的结果是否绘制预测结果                          |

如果在训练或者测试过程中想开启 `VisualizationHook` 相关功能和配置，仅需修改配置即可，以 `dbnet_resnet18_fpnc_1200e_icdar2015.py`为例， 同时绘制标注和预测，并且将图像展示，配置可进行如下修改

```Python

```Python
```

</div>

如果只想查看预测结果信息可以只让`draw_pred=True`

```Python
```

</div>

在 `test.py` 过程中进一步简化，提供了 `--show` 和 `--show-dir`两个参数，无需修改配置即可视化测试过程中绘制标注和预测结果。

```Shell
# 展示test 结果

# 指定预测结果的存储位置

```Shell
# 展示test 结果

```

</div>

# Preparing Kinetics-710

```BibTeX
@misc{li2022uniformerv2,
      title={UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer},
      author={Kunchang Li and Yali Wang and Yinan He and Yizhuo Li and Yi Wang and Limin Wang and Yu Qiao},
      eprint={2211.09552},
}
```

For basic dataset information, please refer to the [paper](https://arxiv.org/pdf/2211.09552.pdf). The scripts can be used for preparing kinetics-710. MMAction2 supports Kinetics-710
dataset as a concat dataset, which means only provides a list of annotation files, and makes use of the original data of Kinetics-400/600/700 dataset. You could refer to the [config](/configs/recognition/uniformerv2/uniformerv2-base-p16-res224_clip_u8_kinetics710-rgb.py)
for details, which also provides a template config about how to use concat dataset in MMAction2.

Before we start, please make sure that the directory is located at `$MMACTION2`.

## Step 1. Download Kinetics 400/600/700

Kinetics-710 is a video benchmark based on Kinetics-400/600/700, which merges the training set of these Kinetics datasets, and deletes the repeated videos according to Youtube IDs. MMAction2 provides an annotation file based on the Kinetics-400/600/700 on [OpenDataLab](https://opendatalab.com/). So we suggest you download Kinetics-400/600/700 first from OpenDataLab by [MIM](https://github.com/open-mmlab/mim).

```shell
odl login
# download Kinetics-400/600/700, note that this might take a long time.
mim download mmaction2 --dataset kinetics400
mim download mmaction2 --dataset kinetics600
mim download mmaction2 --dataset kinetics700

```

## Step 2. Download Kinetics-710 Annotations

```

We provide the annotation list of Kinetics-710 corresponding to OpenDataLab version Kinetics, you could download it from aliyun and unzip it to the `$MMACTION2/data/`

```shell
wget -P data https://download.openmmlab.com/mmaction/dataset/kinetics710/annotations.zip
cd data && unzip annotations.zip && cd ..

```

## Step 3. Folder Structure

After the whole data pipeline for Kinetics preparation.
you can get the videos and annotation files for Kinetics-710.

In the context of the whole project (for Kinetics only), the *minimal* folder structure will look like:
(*minimal* means that some data are not necessary: for example, you may want to evaluate kinetics using the original video format.)

```
mmaction2
├── tools
├── data
│   ├── kinetics400
│   │   ├── videos_train

```
mmaction2
├── tools
├── data
│   │   ├── videos_val
│   │   │   ├── jf7RDuUTrsQ.mp4
│   │   │   ├── ...
│   ├── kinetics600
│   │   │   ├── vol_00
│   │   │   │   ├── -A5JFdMXB_k_000018_000028.mp4
│   │   │   ├── ...
│   │   │   ├── vol63
│   ├── kinetics700
│   │   │   │   ├── -Paa0R0tQ1w_000009_000019.mp4
│   │   │   ├── ...
│   ├── kinetics710
│   │   ├── k400_train_list_videos.txt
│   │   ├── k400_val_list_videos.txt
│   │   ├── k600_train_list_videos.txt
│   │   ├── k600_val_list_videos.txt
│   │   ├── k700_train_list_videos.txt
│   │   ├── k700_val_list_videos.txt
```

For training and evaluating on Kinetics, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).

# Migrating from MMDetection 2.x to 3.x

MMDetection 3.x is a significant update that includes many changes to API and configuration files. This document aims to help users migrate from MMDetection 2.x to 3.x.
We divided the migration guide into the following sections:

- [Configuration file migration](./config_migration.md)
- [API and Registry migration](./api_and_registry_migration.md)
- [Dataset migration](./dataset_migration.md)
- [Model migration](./model_migration.md)
- [Frequently Asked Questions](./migration_faq.md)

If you encounter any problems during the migration process, feel free to raise an issue. We also welcome contributions to this document.

<details>

```bibtex
}
```

<details>

```bibtex

| [pose_shufflenetv2](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_shufflenetv2_8xb64-210e_coco-256x192.py) |  256x192   | 0.602 |      0.857      |      0.672      | 0.668 |      0.902      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_shufflenetv2_8xb64-210e_coco-256x192-51fb931e_20221014.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_shufflenetv2_8xb64-210e_coco-256x192_20221014.log) |
| [pose_shufflenetv2](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_shufflenetv2_8xb64-210e_coco-384x288.py) |  384x288   | 0.638 |      0.866      |      0.707      | 0.699 |      0.910      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_shufflenetv2_8xb64-210e_coco-384x288-d30ab55c_20221014.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_shufflenetv2_8xb64-210e_coco-384x288_20221014.log) |

# ABCNet: Real-time Scene Text Spotting with Adaptive Bezier-Curve Network

<div>
<a href="https://arxiv.org/abs/2002.10200">[arXiv paper]</a>
<a href="https://openaccess.thecvf.com/content_CVPR_2020/papers/Liu_ABCNet_Real-Time_Scene_Text_Spotting_With_Adaptive_Bezier-Curve_Network_CVPR_2020_paper.pdf">[CVPR paper]</a>
</div>

This is an implementation of [ABCNet](https://github.com/aim-uofa/AdelaiDet) based on [MMOCR](https://github.com/open-mmlab/mmocr/tree/dev-1.x), [MMCV](https://github.com/open-mmlab/mmcv), and [MMEngine](https://github.com/open-mmlab/mmengine).

**ABCNet** is a conceptually novel, efficient, and fully convolutional framework for text spotting, which address the problem by proposing the Adaptive Bezier-Curve Network (ABCNet). Our contributions are three-fold: 1) For the first time, we adaptively fit arbitrarily-shaped text by a parameterized Bezier curve. 2) We design a novel BezierAlign layer for extracting accurate convolution features of a text instance with arbitrary shapes, significantly improving the precision compared with previous methods. 3) Compared with standard bounding box detection, our Bezier curve detection introduces negligible computation overhead, resulting in superiority of our method in both efficiency and accuracy. Experiments on arbitrarily-shaped benchmark datasets, namely Total-Text and CTW1500, demonstrate that ABCNet achieves state-of-the-art accuracy, meanwhile significantly improving the speed. In particular, on Total-Text, our realtime version is over 10 times faster than recent state-of-the-art methods with a competitive recognition accuracy.

<center>
<img src="https://user-images.githubusercontent.com/24622904/205641295-d57c225f-4b2e-4954-b604-ba8c8afc23cb.png">
</center>

## Usage

```shell
# Linux
```

```shell
# Linux

```

```shell
# Linux
```

In the current directory, run the following command to train the model:

```bash
mim train mmocr config/abcnet/abcnet_resnet50_fpn_500e_icdar2015.py --work-dir work_dirs/
```

```bash
mim train mmocr config/abcnet/abcnet_resnet50_fpn_500e_icdar2015.py --work-dir work_dirs/ --launcher pytorch --gpus 8
```

```bash
mim test mmocr config/abcnet/abcnet_resnet50_fpn_500e_icdar2015.py --work-dir work_dirs/ --checkpoint ${CHECKPOINT_PATH}
```

## Results

```

## Results

|         Name          |                                  Pretrained Model                                  | E2E-None-Hmean | det-Hmean |                                  Download                                  |
| :-------------------: | :--------------------------------------------------------------------------------: | :------------: | :-------: | :------------------------------------------------------------------------: |

| v1-icdar2015-finetune | [SynthText](https://download.openmmlab.com/mmocr/textspotting/abcnet/abcnet_resnet50_fpn_500e_icdar2015/abcnet_resnet50_fpn_pretrain-d060636c.pth) |     0.6127     |  0.8753   | [model](https://download.openmmlab.com/mmocr/textspotting/abcnet/abcnet_resnet50_fpn_500e_icdar2015/abcnet_resnet50_fpn_500e_icdar2015-326ac6f4.pth) \| [log](https://download.openmmlab.com/mmocr/textspotting/abcnet/abcnet_resnet50_fpn_500e_icdar2015/20221210_170401.log) |

If you find ABCNet useful in your research or applications, please cite ABCNet with the following BibTeX entry.

```BibTeX
@inproceedings{liu2020abcnet,
  title     =  {{ABCNet}: Real-time Scene Text Spotting with Adaptive Bezier-Curve Network},
  author    =  {Liu, Yuliang and Chen, Hao and Shen, Chunhua and He, Tong and Jin, Lianwen and Wang, Liangwei},
  booktitle =  {Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR)},
}
```

booktitle =  {Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR)},
}
```

# RepPoints

> [RepPoints: Point Set Representation for Object Detection](https://arxiv.org/abs/1904.11490)

Modern object detectors rely heavily on rectangular bounding boxes, such as anchors, proposals and the final predictions, to represent objects at various recognition stages. The bounding box is convenient to use but provides only a coarse localization of objects and leads to a correspondingly coarse extraction of object features. In this paper, we present RepPoints(representative points), a new finer representation of objects as a set of sample points useful for both localization and recognition. Given ground truth localization and recognition targets for training, RepPoints learn to automatically arrange themselves in a manner that bounds the spatial extent of an object and indicates semantically significant local areas. They furthermore do not require the use of anchors to sample a space of bounding boxes. We show that an anchor-free object detector based on RepPoints can be as effective as the state-of-the-art anchor-based detection methods, with 46.5 AP and 67.4 AP50 on the COCO test-dev detection benchmark, using ResNet-101 model.

<img src="https://user-images.githubusercontent.com/40661020/143972514-93247220-4dad-4eb3-a51b-a1115dc7d449.png"/>
</div>

## Introdution

By [Ze Yang](https://yangze.tech/), [Shaohui Liu](http://b1ueber2y.me/), and [Han Hu](https://ancientmooner.github.io/).

We provide code support and configuration files to reproduce the results in the paper for
["RepPoints: Point Set Representation for Object Detection"](https://arxiv.org/abs/1904.11490) on COCO object detection.

**RepPoints**, initially described in [arXiv](https://arxiv.org/abs/1904.11490), is a new representation method for visual objects, on which visual understanding tasks are typically centered. Visual object representation, aiming at both geometric description and appearance feature extraction, is conventionally achieved by `bounding box + RoIPool (RoIAlign)`. The bounding box representation is convenient to use; however, it provides only a rectangular localization of objects that lacks geometric precision and may consequently degrade feature quality. Our new representation, RepPoints, models objects by a `point set` instead of a `bounding box`, which learns to adaptively position themselves over an object in a manner that circumscribes the object’s `spatial extent` and enables `semantically aligned feature extraction`. This richer and more flexible representation maintains the convenience of bounding boxes while facilitating various visual understanding applications. This repo demonstrated the effectiveness of RepPoints for COCO object detection.

Another feature of this repo is the demonstration of an `anchor-free detector`, which can be as effective as state-of-the-art anchor-based detection methods. The anchor-free detector can utilize either `bounding box` or `RepPoints` as the basic object representation.

The results on COCO 2017val are shown in the table below.

|  Method   |   Backbone    | GN  | Anchor | convert func | Lr schd | Mem (GB) | Inf time (fps) | box AP |                                 Config                                  |                                                                                                                                                                                                                 Download                                                                                                                                                                                                                  |

|   BBox    |   R-50-FPN    |  Y  | single |      -       |   1x    |   3.9    |      15.9      |  36.4  |      [config](./reppoints-bbox_r50_fpn-gn_head-gn-grid_1x_coco.py)      |                                     [model](https://download.openmmlab.com/mmdetection/v2.0/reppoints/bbox_r50_grid_fpn_gn-neck%2Bhead_1x_coco/bbox_r50_grid_fpn_gn-neck%2Bhead_1x_coco_20200329_145916-0eedf8d1.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/reppoints/bbox_r50_grid_fpn_gn-neck%2Bhead_1x_coco/bbox_r50_grid_fpn_gn-neck%2Bhead_1x_coco_20200329_145916.log.json)                                      |

|   BBox    |   R-50-FPN    |  Y  |  none  |      -       |   1x    |   3.9    |      15.4      |  37.4  |  [config](./reppoints-bbox_r50-center_fpn-gn_head-gn-grid_1x_coco.py)   |                                     [model](https://download.openmmlab.com/mmdetection/v2.0/reppoints/bbox_r50_grid_fpn_gn-neck%2Bhead_1x_coco/bbox_r50_grid_fpn_gn-neck%2Bhead_1x_coco_20200329_145916-0eedf8d1.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/reppoints/bbox_r50_grid_fpn_gn-neck%2Bhead_1x_coco/bbox_r50_grid_fpn_gn-neck%2Bhead_1x_coco_20200329_145916.log.json)                                      |

| RepPoints |   R-50-FPN    |  N  |  none  |    moment    |   1x    |   3.3    |      18.5      |  37.0  |             [config](./reppoints-moment_r50_fpn_1x_coco.py)             |                                                         [model](https://download.openmmlab.com/mmdetection/v2.0/reppoints/reppoints_moment_r50_fpn_1x_coco/reppoints_moment_r50_fpn_1x_coco_20200330-b73db8d1.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/reppoints/reppoints_moment_r50_fpn_1x_coco/reppoints_moment_r50_fpn_1x_coco_20200330_233609.log.json)                                                         |

| RepPoints |   R-50-FPN    |  Y  |  none  |    moment    |   1x    |   3.9    |      17.5      |  38.1  |       [config](./reppoints-moment_r50_fpn-gn_head-gn_1x_coco.py)        |                       [model](https://download.openmmlab.com/mmdetection/v2.0/reppoints/reppoints_moment_r50_fpn_gn-neck%2Bhead_1x_coco/reppoints_moment_r50_fpn_gn-neck%2Bhead_1x_coco_20200329_145952-3e51b550.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/reppoints/reppoints_moment_r50_fpn_gn-neck%2Bhead_1x_coco/reppoints_moment_r50_fpn_gn-neck%2Bhead_1x_coco_20200329_145952.log.json)                        |

| RepPoints |   R-50-FPN    |  Y  |  none  |    moment    |   2x    |   3.9    |       -        |  38.6  |       [config](./reppoints-moment_r50_fpn-gn_head-gn_2x_coco.py)        |                           [model](https://download.openmmlab.com/mmdetection/v2.0/reppoints/reppoints_moment_r50_fpn_gn-neck%2Bhead_2x_coco/reppoints_moment_r50_fpn_gn-neck%2Bhead_2x_coco_20200329-91babaa2.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/reppoints/reppoints_moment_r50_fpn_gn-neck%2Bhead_2x_coco/reppoints_moment_r50_fpn_gn-neck%2Bhead_2x_coco_20200329_150020.log.json)                           |

| RepPoints |   R-101-FPN   |  Y  |  none  |    moment    |   2x    |   5.8    |      13.7      |  40.5  |       [config](./reppoints-moment_r101_fpn-gn_head-gn_2x_coco.py)       |                         [model](https://download.openmmlab.com/mmdetection/v2.0/reppoints/reppoints_moment_r101_fpn_gn-neck%2Bhead_2x_coco/reppoints_moment_r101_fpn_gn-neck%2Bhead_2x_coco_20200329-4fbc7310.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/reppoints/reppoints_moment_r101_fpn_gn-neck%2Bhead_2x_coco/reppoints_moment_r101_fpn_gn-neck%2Bhead_2x_coco_20200329_132205.log.json)                         |

| RepPoints | R-101-FPN-DCN |  Y  |  none  |    moment    |   2x    |   5.9    |      12.1      |  42.9  | [config](./reppoints-moment_r101-dconv-c3-c5_fpn-gn_head-gn_2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/reppoints/reppoints_moment_r101_fpn_dconv_c3-c5_gn-neck%2Bhead_2x_coco/reppoints_moment_r101_fpn_dconv_c3-c5_gn-neck%2Bhead_2x_coco_20200329-3309fbf2.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/reppoints/reppoints_moment_r101_fpn_dconv_c3-c5_gn-neck%2Bhead_2x_coco/reppoints_moment_r101_fpn_dconv_c3-c5_gn-neck%2Bhead_2x_coco_20200329_132134.log.json) |

| RepPoints | X-101-FPN-DCN |  Y  |  none  |    moment    |   2x    |   7.1    |      9.3       |  44.2  | [config](./reppoints-moment_x101-dconv-c3-c5_fpn-gn_head-gn_2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/reppoints/reppoints_moment_x101_fpn_dconv_c3-c5_gn-neck%2Bhead_2x_coco/reppoints_moment_x101_fpn_dconv_c3-c5_gn-neck%2Bhead_2x_coco_20200329-f87da1ea.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/reppoints/reppoints_moment_x101_fpn_dconv_c3-c5_gn-neck%2Bhead_2x_coco/reppoints_moment_x101_fpn_dconv_c3-c5_gn-neck%2Bhead_2x_coco_20200329_132201.log.json) |

- `R-xx`, `X-xx` denote the ResNet and ResNeXt architectures, respectively.
- `DCN` denotes replacing 3x3 conv with the 3x3 deformable convolution in `c3-c5` stages of backbone.

- `none` in the `anchor` column means 2-d `center point` (x,y) is used to represent the initial object hypothesis. `single` denotes one 4-d anchor box (x,y,w,h) with IoU based label assign criterion is adopted.
- `moment`, `partial MinMax`, `MinMax` in the `convert func` column are three functions to convert a point set to a pseudo box.
- Note the results here are slightly different from those reported in the paper, due to framework change. While the original paper uses an [MXNet](https://mxnet.apache.org/) implementation, we re-implement the method in [PyTorch](https://pytorch.org/) based on mmdetection.

```latex

```latex
}
```

# 为推理 ops 添加测试单元

本教程介绍如何为后端 ops 添加单元测试。在 backend_ops 目录下添加自定义 op 时，需要添加相应的测试单元。op 的单元测试在 `test/test_ops/test_ops.py` 中。

添加新的自定义 op 后，需要重新编译，引用 [build.md](../01-how-to-build/build_from_source.md) 。

## ops 单元测试样例

```python
                         [(2, 2, 1.0, 2), (4, 4, 2.0, 4)])               # [（# Examples of op test parameters）,...]
                   pool_w,

save_dir=None):

    else:

backend.run_and_validate(
```

mmdeploy 支持的模型有两种格式：

- torch 模型：参考 roi_align 单元测试，必须要求 op 相关 Python 代码
- onnx 模型：参考 multi_level_roi_align 单元测试，需要调用 onnx api 进行构建

调用 `run_and_validate` 即可运行

```python
                         model,
```

|          参数           |                 说明                  |

expected_result=None,
```

|          参数           |                 说明                  |
| :---------------------: | :-----------------------------------: |
|          model          |           要测试的输入模型            |
|       input_list        | 测试数据列表，映射到input_names的顺序 |
| tolerate_small_mismatch |     是否允许验证结果出现精度误差      |
|   do_constant_folding   |           是否使用常量折叠            |
|      output_names       |             输出节点名字              |
|       input_names       |             输入节点名字              |
|     expected_result     |          期望的 ground truth          |
|        save_dir         |             结果保存目录              |

## 测试模型

用 `pytest` 调用 ops 测试

```bash
```

## 中文解读文案汇总（待更新）

### 1 官方解读文案（v2.x）

#### 1.1 框架解读

- **[轻松掌握 MMDetection 整体构建流程(一)](https://zhuanlan.zhihu.com/p/337375549)**
- **[轻松掌握 MMDetection 整体构建流程(二)](https://zhuanlan.zhihu.com/p/341954021)**
- **[轻松掌握 MMDetection 中 Head 流程](https://zhuanlan.zhihu.com/p/343433169)**

#### 1.2 算法解读

- **[轻松掌握 MMDetection 中常用算法(一)：RetinaNet 及配置详解](https://zhuanlan.zhihu.com/p/346198300)**
- **[轻松掌握 MMDetection 中常用算法(二)：Faster R-CNN|Mask R-CNN](https://zhuanlan.zhihu.com/p/349807581)**
- [轻松掌握 MMDetection 中常用算法(三)：FCOS](https://zhuanlan.zhihu.com/p/358056615)
- [轻松掌握 MMDetection 中常用算法(四)：ATSS](https://zhuanlan.zhihu.com/p/358125611)
- [轻松掌握 MMDetection 中常用算法(五)：Cascade R-CNN](https://zhuanlan.zhihu.com/p/360952172)
- [轻松掌握 MMDetection 中常用算法(六)：YOLOF](https://zhuanlan.zhihu.com/p/370758213)
- [轻松掌握 MMDetection 中常用算法(七)：CenterNet](https://zhuanlan.zhihu.com/p/374891478)
- [轻松掌握 MMDetection 中常用算法(八)：YOLACT](https://zhuanlan.zhihu.com/p/376347955)

- [轻松掌握 MMDetection 中常用算法(九)：AutoAssign](https://zhuanlan.zhihu.com/p/378581552)
- [YOLOX 在 MMDetection 中复现全流程解析](https://zhuanlan.zhihu.com/p/398545304)
- [喂喂喂！你可以减重了！小模型 - MMDetection 新增SSDLite 、 MobileNetV2YOLOV3 两大经典算法](https://zhuanlan.zhihu.com/p/402781143)

#### 1.3 工具解读

- [OpenMMLab 中混合精度训练 AMP 的正确打开方式](https://zhuanlan.zhihu.com/p/375224982)
- [小白都能看懂！手把手教你使用混淆矩阵分析目标检测](https://zhuanlan.zhihu.com/p/443499860)
- [MMDetection 图像缩放 Resize 详细说明 OpenMMLab](https://zhuanlan.zhihu.com/p/381117525)
- [拿什么拯救我的 4G 显卡](https://zhuanlan.zhihu.com/p/430123077)
- [MMDet居然能用MMCls的Backbone？论配置文件的打开方式](https://zhuanlan.zhihu.com/p/436865195)

#### 1.4 知乎问答

- [COCO数据集上1x模式下为什么不采用多尺度训练?](https://www.zhihu.com/question/462170786/answer/1915119662)

- [MMDetection中SOTA论文源码中将训练过程中BN层的eval打开?](https://www.zhihu.com/question/471189603/answer/2195540892)
- [基于PyTorch的MMDetection中训练的随机性来自何处？](https://www.zhihu.com/question/453511684/answer/1839683634)
- [单阶段、双阶段、anchor-based、anchor-free 这四者之间有什么联系吗？](https://www.zhihu.com/question/428972054/answer/1619925296)
- [目标检测的深度学习方法，有推荐的书籍或资料吗？](https://www.zhihu.com/question/391577080/answer/1612593817)
- [大佬们，刚入学研究生，想入门目标检测，有什么学习路线可以入门的？](https://www.zhihu.com/question/343768934/answer/1612580715)
- [目标检测领域还有什么可以做的？](https://www.zhihu.com/question/280703314/answer/1627885518)
- [如何看待Transformer在CV上的应用前景，未来有可能替代CNN吗？](https://www.zhihu.com/question/437495132/answer/1686380553)
- [MMDetection如何学习源码？](https://www.zhihu.com/question/451585041/answer/1832498963)

- [如何具体上手实现目标检测呢？](https://www.zhihu.com/question/341401981/answer/1848561187)

#### 1.5 其他

- **[不得不知的 MMDetection 学习路线(个人经验版)](https://zhuanlan.zhihu.com/p/369826931)**
- [OpenMMLab 社区专访之 YOLOX 复现篇](https://zhuanlan.zhihu.com/p/405913343)

### 2 社区解读文案（v2.x）

- [手把手带你实现经典检测网络 Mask R-CNN 的推理](https://zhuanlan.zhihu.com/p/414082071)

# Changelog of v1.1

### v1.2.0 (4/7/2023)

#### Highlights

- Support [New Config Type](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#a-pure-python-style-configuration-file-beta) in `mmdet3d/config`  (#2608)
- Support the inference of [DSVT](<(https://arxiv.org/abs/2301.06051)>) in `projects`  (#2606)
- Support downloading datasets from [OpenDataLab](https://opendatalab.com/) using `mim`  (#2593)

#### New Features

#### Improvements

- Enhanced visualization in interactive form (#2611)
- Update README.md and Model Zoo (#2599, #2600)
- Speed up S3DIS data preparation (#2585)

#### Bug Fixes

- Remove PointRCNN in benchmark training (#2610)
- Fix wrong indoor detection visualization (#2625)
- Fix MinkUNet download link (#2590)
- Fix the formula in the `readthedocs` (#2580)

#### Contributors

A total of 5 developers contributed to this release.

@sunjiahao1999, @Xiangxu-0103, @JingweiZhang12, @col14m, @zhulf0804

### v1.1.1 (30/5/2023)

- Support [TPVFormer](https://arxiv.org/pdf/2302.07817.pdf) in `projects` (#2399, #2517, #2535)
- Support the training of BEVFusion in `projects` (#2546)
- Support lidar-based 3D semantic segmentation benchmark (#2530, #2559)

- Support the training of \[BEVFusion\] in `projects` (#2558)

- Support lidar-based 3D Semantic Segmentation Benchmark (#2530, #2559)
- Support test-time augmentation for Segmentor (#2382)
- Support `Minkowski ConvModule` and `Residual` Block (#2528)
- Support the visualization of multi-view images in multi-modal methods (#2453)

- Upload checkpoints and training log of PETR (#2555)
- Replace `np.float` by default `float` in segmentation evaluation (#2527)
- Add docs of converting SemanticKITTI datasets (#2515)
- Support different colors for different classes in visualization (#2500)
- Support tensor-like operations for `BaseInstance3DBoxes` and `BasePoint`
- Add information of LiDAR Segmentation in NuScenes annotation files
- Provide annotation files of datasets generated offline (#2457)
- Refactor document structure (#2429)

- Complete typehints and docstring (#2396, #2457, #2468, #2464, #2485)

- Fix the bug of abnormal loss when training SECOND in Automatic mixed precision(AMP) mode (#2452)
- Add a warning in function `post_process_coords` in mmdet3d/dataset/convert_utils.py (#2557)
- Fix invalid configs (#2477, #2536)
- Fix bugs of unit test (#2466)
- Update `local-rank` argument in test.py for pytorch 2.0 (#2469)
- Fix docker file (#2451)
- Fix demo and visualization (#2453)
- Fix SUN RGB-D data converter (#2440)
- Fix readthedocs building (#2459, #2419, #2505, #2396)
- Fix CI #(2445)
- Fix the version error of `torch` in github merge stage test (#2424)
- Loose the version restriction of `numba` (#2416)

A total of 10 developers contributed to this release.

@sunjiahao1999, @Xiangxu-0103, @JingweiZhang12, @chriscarving, @jaan1729, @pd-michaelstanley, @filaPro, @kabouzeid, @A-new-b, @lbin

### v1.1.0 (6/4/2023)

- Support [Cylinder3D](https://arxiv.org/pdf/2011.10033.pdf) (#2291, #2344, #2350)
- Support [MinkUnet](https://arxiv.org/abs/1904.08755) (#2294, #2358)
- Support [SPVCNN](https://arxiv.org/abs/2007.16100) (#2320，#2372)
- Support [TR3D](https://arxiv.org/abs/2302.02858) detector in `projects` (#2274)
- Support the inference of [BEVFusion](https://arxiv.org/abs/2205.13542) in `projects` (#2175)
- Support [DETR3D](https://arxiv.org/abs/2110.06922) in `projects` (#2173)

- Support PolarMix and LaserMix augmentation (#2265, #2302)
- Support loading annotation of panoptic segmentation (#2223)
- Support panoptic segmentation metric (#2230)
- Add inferencer for LiDAR-based, monocular and multi-modality 3D detection (#2208, #2190, #2342)
- Add inferencer for LiDAR-based segmentation (#2304)

- Support `lazy_init` for CBGSDataset (#2271)

- Support generating annotation files for test set on Waymo  (#2180)
- Enhance the support for SemanticKitti (#2253, #2323)
- File I/O migration and reconstruction (#2319)
- Support `format_only` option for Lyft, NuScenes and Waymo datasets (#2333, #2151)
- Replace `np.transpose` with `torch.permute` to speed up (#2277)
- Allow setting local-rank for pytorch 2.0 (#2387)

- Fix the problem of reversal of length and width when drawing heatmap in CenterFormer (#2362)
- Deprecate old type alias due to the new version of numpy (#2339)
- Lose `trimesh` version requirements to fix numpy random state (#2340)
- Fix the device mismatch error in CenterPoint (#2308)
- Fix bug of visualization when there are no bboxes (#2231)
- Fix bug of counting ignore index in IOU in segmentation evaluation (#2229)

A total of 14 developers contributed to this release.

@ZLTJohn, @SekiroRong, @shufanwu, @vansin, @triple-Mu, @404Vector, @filaPro, @sunjiahao1999, @Ginray, @Xiangxu-0103, @JingweiZhang12, @DezeZhao, @ZCMax, @roger-lcc

### v1.1.0rc3 (7/1/2023)

- Support [CenterFormer](https://arxiv.org/abs/2209.05588) in `projects` (#2175)
- Support [PETR](https://arxiv.org/abs/2203.05625) in `projects` (#2173)

- Refactor ImVoxelNet on SUN RGB-D into mmdet3d v1.1 (#2141)

- Remove legacy builder.py (#2061)
- Update `customize_dataset` documentation (#2153)
- Update tutorial of LiDAR-based detection (#2120)

- Fix the configs of FCOS3D and PGD (#2191)
- Fix numpy's `ValueError` in update_infos_to_v2.py (#2162)
- Fix parameter missing in Det3DVisualizationHook (#2118)
- Fix memory overflow in the rotated box IoU calculation (#2134)
- Fix lidar2cam error in update_infos_to_v2.py for nus and lyft dataset (#2110)
- Fix error of data type in Waymo metrics (#2109)
- Update `bbox_3d` information in `cam_instances` for mono3d detection task (#2046)
- Fix label saving of Waymo dataset (#2096)

@SekiroRong, @ZLTJohn, @vansin, @shanmo, @VVsssssk, @ZCMax, @Xiangxu-0103, @JingweiZhang12, @Tai-Wang, @lianqing11

### v1.1.0rc2 (2/12/2022)

- Support [PV-RCNN](https://arxiv.org/abs/1912.13192)
- Speed up evaluation on Waymo dataset

- Support [PV-RCNN](https://arxiv.org/abs/1912.13192) (#1597, #2045)
- Speed up evaluation on Waymo dataset (#2008)
- Refactor FCAF3D into the framework of mmdet3d v1.1 (#1945)
- Refactor S3DIS dataset into the framework of mmdet3d v1.1 (#1984)
- Add `Projects/` folder and the first example project (#2042)

- Rename `CLASSES` and `PALETTE` to `classes` and `palette` respectively (#1932)
- Update `metainfo` in pkl files and add `categories` into metainfo (#1934)
- Show instance statistics before and after through the pipeline (#1863)
- Add configs of DGCNN for different testing areas (#1967)
- Remove testing utils from `tests/utils/` to `mmdet3d/testing/` (#2012)
- Add typehint for code in `models/layers/` (#2014)
- Refine documentation (#1891, #1994)
- Refine voxelization for better speed (#2062)

- Fix loop visualization error about point cloud (#1914)
- Fix image conversion of Waymo to avoid information loss (#1979)
- Fix evaluation on KITTI testset (#2005)
- Fix sampling bug in `IoUNegPiecewiseSampler` (#2017)
- Fix point cloud range in CenterPoint (#1998)
- Fix some loading bugs and support FOV-image-based mode on Waymo dataset (#1942)
- Fix dataset conversion utils (#1923, #2040, #1971)
- Update metafiles in all the configs (#2006)

@vavanade, @oyel, @thinkthinking, @PeterH0323， @274869388, @cxiang26, @lianqing11, @VVsssssk, @ZCMax, @Xiangxu-0103, @JingweiZhang12, @Tai-Wang

### v1.1.0rc1 (11/10/2022)

- Support a camera-only 3D detection baseline on Waymo, [MV-FCOS3D++](https://arxiv.org/abs/2207.12716)

- Support a camera-only 3D detection baseline on Waymo, [MV-FCOS3D++](https://arxiv.org/abs/2207.12716), with new evaluation metrics and transformations (#1716)
- Refactor PointRCNN in the framework of mmdet3d v1.1 (#1819)

- Add `auto_scale_lr` in config to support training with auto-scale learning rates (#1807)
- Fix CI (#1813, #1865, #1877)
- Update `browse_dataset.py` script (#1817)
- Update SUN RGB-D and Lyft datasets documentation (#1833)
- Rename `convert_to_datasample` to `add_pred_to_datasample` in detectors (#1843)
- Update customized dataset documentation (#1845)
- Update `Det3DLocalVisualization` and visualization documentation (#1857)
- Add the code of generating `cam_sync_labels` for Waymo dataset (#1870)
- Update dataset transforms typehints (#1875)

- Fix missing registration of models in [setup_env.py](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/utils/setup_env.py) (#1808)
- Fix the data base sampler bugs when using the ground plane data (#1812)
- Add output directory existing check during visualization (#1828)
- Fix bugs of nuScenes dataset for monocular 3D detection (#1837)
- Fix visualization hook to support the visualization of different data modalities (#1839)
- Fix monocular 3D detection demo (#1864)
- Fix the lack of `num_pts_feats` key in nuscenes dataset and complete docstring (#1882)

@ZwwWayne, @Tai-Wang, @lianqing11, @VVsssssk, @ZCMax, @Xiangxu-0103, @JingweiZhang12, @tpoisonooo, @ice-tong, @jshilong

### v1.1.0rc0 (1/9/2022)

We are excited to announce the release of MMDetection3D 1.1.0rc0.

MMDet3D 1.1.0rc0 is the first version of MMDetection3D 1.1, a part of the OpenMMLab 2.0 projects.
Built upon the new [training engine](https://github.com/open-mmlab/mmengine) and [MMDet 3.x](https://github.com/open-mmlab/mmdetection/tree/3.x),
MMDet3D 1.1 unifies the interfaces of dataset, models, evaluation, and visualization with faster training and testing speed.
It also provides a standard data protocol for different datasets, modalities, and tasks for 3D perception.
We will support more strong baselines in the future release, with our latest exploration on camera-only 3D detection from videos.

1. **New engines**. MMDet3D 1.1 is based on [MMEngine](https://github.com/open-mmlab/mmengine) and [MMDet 3.x](https://github.com/open-mmlab/mmdetection/tree/3.x), which provides a universal and powerful runner that allows more flexible customizations and significantly simplifies the entry points of high-level interfaces.

2. **Unified interfaces**. As a part of the OpenMMLab 2.0 projects, MMDet3D 1.1 unifies and refactors the interfaces and internal logics of train, testing, datasets, models, evaluation, and visualization. All the OpenMMLab 2.0 projects share the same design in those interfaces and logics to allow the emergence of multi-task/modality algorithms.

3. **Standard data protocol for all the datasets, modalities, and tasks for 3D perception**. Based on the unified base datasets inherited from MMEngine, we also design a standard data protocol that defines and unifies the common keys across different datasets, tasks, and modalities. It significantly simplifies the usage of multiple datasets and data modalities for multi-task frameworks and eases dataset customization. Please refer to the [documentation of customized datasets](../advanced_guides/customize_dataset.md) for details.

4. **Strong baselines**. We will release strong baselines of many popular models to enable fair comparisons among state-of-the-art models.

5. **More documentation and tutorials**. We add a bunch of documentation and tutorials to help users get started more smoothly. Read it [here](https://mmdetection3d.readthedocs.io/en/1.1/).

MMDet3D 1.1 has undergone significant changes to have better design, higher efficiency, more flexibility, and more unified interfaces.
Besides the changes of API, we briefly list the major breaking changes in this section.
Users can also refer to the [compatibility documentation](./compatibility.md) and [API doc](https://mmdetection3d.readthedocs.io/en/1.1/) for more details.

- MMDet3D 1.1 runs on PyTorch>=1.6. We have deprecated the support of PyTorch 1.5 to embrace the mixed precision training and other new features since PyTorch 1.6. Some models can still run on PyTorch 1.5, but the full functionality of MMDet3D 1.1 is not guaranteed.

- MMDet3D 1.1 relies on MMEngine to run. MMEngine is a new foundational library for training deep learning models of OpenMMLab and are widely depended by OpenMMLab 2.0 projects. The dependencies of file IO and training are migrated from MMCV 1.x to MMEngine.
- MMDet3D 1.1 relies on MMCV>=2.0.0rc0. Although MMCV no longer maintains the training functionalities since 2.0.0rc0, MMDet3D 1.1 relies on the data transforms, CUDA operators, and image processing interfaces in MMCV. Note that the package `mmcv` is the version that provides pre-built CUDA operators and `mmcv-lite` does not since MMCV 2.0.0rc0, while `mmcv-full` has been deprecated since 2.0.0rc0.
- MMDet3D 1.1 is based on MMDet 3.x, which is also a part of OpenMMLab 2.0 projects.

- MMDet3D 1.1 uses Runner in [MMEngine](https://github.com/open-mmlab/mmengine) rather than that in MMCV. The new Runner implements and unifies the building logic of dataset, model, evaluation, and visualizer. Therefore, MMDet3D 1.1 no longer relies on the building logics of those modules in `mmdet3d.train.apis` and `tools/train.py`. Those code have been migrated into [MMEngine](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/runner.py). Please refer to the [migration guide of Runner in MMEngine](https://mmengine.readthedocs.io/en/latest/migration/runner.html) for more details.

- Learning rate and momentum scheduling has been migrated from Hook to [Parameter Scheduler in MMEngine](https://mmengine.readthedocs.io/en/latest/tutorials/param_scheduler.html). Please refer to the [migration guide of Parameter Scheduler in MMEngine](https://mmengine.readthedocs.io/en/latest/migration/param_scheduler.html) for more details.

- The [Runner in MMEngine](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/runner.py) uses a different config structure to ease the understanding of the components in runner. Users can read the [config example of MMDet3D 1.1](../user_guides/config.md) or refer to the [migration guide in MMEngine](https://mmengine.readthedocs.io/en/latest/migration/runner.html) for migration details.
- The file names of configs and models are also refactored to follow the new rules unified across OpenMMLab 2.0 projects. The names of checkpoints are not updated for now as there is no BC-breaking of model weights between MMDet3D 1.1 and 1.0.x. We will progressively replace all the model weights by those trained in MMDet3D 1.1. Please refer to the [user guides of config](../user_guides/config.md) for more details.

The Dataset classes implemented in MMDet3D 1.1 all inherits from the `Det3DDataset` and `Seg3DDataset`, which inherits from the [BaseDataset in MMEngine](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/basedataset.html). In addition to the changes of interfaces, there are several changes of Dataset in MMDet3D 1.1.

- All the datasets support to serialize the internal data list to reduce the memory when multiple workers are built for data loading.
- The internal data structure in the dataset is changed to be self-contained (without losing information like class names in MMDet3D 1.0.x) while keeping simplicity.
- Common keys across different datasets and data modalities are defined and all the info files are unified into a standard protocol.
- The evaluation functionality of each dataset has been removed from dataset so that some specific evaluation metrics like KITTI AP can be used to evaluate the prediction on other datasets.

The data transforms in MMDet3D 1.1 all inherits from `BaseTransform` in MMCV>=2.0.0rc0, which defines a new convention in OpenMMLab 2.0 projects.
Besides the interface changes, there are several changes listed as below:

#### Model

#### Model

The models in MMDet3D 1.1 all inherits from `BaseModel` in MMEngine, which defines a new convention of models in OpenMMLeb 2.0 projects.
Users can refer to [the tutorial of model in MMengine](https://mmengine.readthedocs.io/en/latest/tutorials/model.html) for more details.

- The model interfaces, including the input and output formats, are significantly simplified and unified following the new convention in MMDet3D 1.1.

  Specifically, all the input data in training and testing are packed into `inputs` and `data_samples`, where `inputs` contains model inputs like a dict contain a list of image tensors and the point cloud data, and `data_samples` contains other information of the current data sample such as ground truths, region proposals, and model predictions. In this way, different tasks in MMDet3D 1.1 can share the same input arguments, which makes the models more general and suitable for multi-task learning and some flexible training paradigms like semi-supervised learning.

- The model has a data preprocessor module, which are used to pre-process the input data of model. In MMDet3D 1.1, the data preprocessor usually does necessary steps to form the input images into a batch, such as padding. It can also serve as a place for some special data augmentations or more efficient data transformations like normalization.
- The internal logic of model have been changed. In MMDet3D 1.1, model uses `forward_train`, `forward_test`, `simple_test`, and `aug_test` to deal with different model forward logics. In MMDet3D 1.1 and OpenMMLab 2.0, the forward function has three modes: 'loss', 'predict', and 'tensor' for training, inference, and tracing or other purposes, respectively.

The evaluation in MMDet3D 1.0.x strictly binds with the dataset. In contrast, MMDet3D 1.1 decomposes the evaluation from dataset, so that all the detection dataset can evaluate with KITTI AP and other metrics implemented in MMDet3D 1.1.
MMDet3D 1.1 mainly implements corresponding metrics for each dataset, which are manipulated by [Evaluator](https://mmengine.readthedocs.io/en/latest/design/evaluator.html) to complete the evaluation.
Users can build evaluator in MMDet3D 1.1 to conduct offline evaluation, i.e., evaluate predictions that may not produced in MMDet3D 1.1 with the dataset as long as the dataset and the prediction follows the dataset conventions. More details can be find in the [tutorial in mmengine](https://mmengine.readthedocs.io/en/latest/tutorials/evaluation.html).

The functions of visualization in MMDet3D 1.1 are removed. Instead, in OpenMMLab 2.0 projects, we use [Visualizer](https://mmengine.readthedocs.io/en/latest/design/visualization.html) to visualize data. MMDet3D 1.1 implements `Det3DLocalVisualizer` to allow visualization of 2D and 3D data, ground truths, model predictions, and feature maps, etc., at any place. It also supports to send the visualization data to any external visualization backends such as Tensorboard.

We list several planned changes of MMDet3D 1.1.0rc0 so that the community could more comprehensively know the progress of MMDet3D 1.1. Feel free to create a PR, issue, or discussion if you are interested, have any suggestions and feedbacks, or want to participate.

1. Test-time augmentation: which is supported in MMDet3D 1.0.x, is not implemented in this version due to limited time slot. We will support it in the following releases with a new and simplified design.

3. Interfaces of useful tools that can be used in notebook: more useful tools that implemented in the `tools` directory will have their python interfaces so that they can be used through notebook and in downstream libraries.
4. Documentation: we will add more design docs, tutorials, and migration guidance so that the community can deep dive into our new design, participate the future development, and smoothly migrate downstream libraries to MMDet3D 1.1.
5. Wandb visualization: MMDet 2.x supports data visualization since v2.25.0, which has not been migrated to MMDet 3.x for now. Since Wandb provides strong visualization and experiment management capabilities, a `DetWandbVisualizer` and maybe a hook are planned to fully migrated those functionalities in MMDet 2.x and a `Det3DWandbVisualizer` will be supported in MMDet3D 1.1 accordingly.

6. Will support recent new features added in MMDet3D 1.0.x and our recent exploration on camera-only 3D detection from videos: we will refactor these models and support them with benchmarks and models soon.

# 准备 AVA

## 简介

```BibTeX
}
```

请参照 [官方网站](https://research.google.com/ava/index.html) 以获取数据集基本信息。
在开始之前，用户需确保当前目录为 `$MMACTION2/tools/data/ava/`。

```shell
```

这一命令将下载 `ava_v2.1.zip` 以得到 AVA v2.1 标注文件。如用户需要 AVA v2.2 标注文件，可使用以下脚本：

```shell
```

## 2. 下载视频

用户可以使用以下脚本准备视频，视频准备代码修改自 [官方爬虫](https://github.com/cvdfoundation/ava-dataset)。

```shell
```

```shell
```

## 2. 下载视频

```shell
```

亦可使用以下脚本，使用 python 并行下载 AVA 数据集视频：

```shell
```

## 3. 截取视频

截取每个视频中的 15 到 30 分钟，设定帧率为 30。

```shell
```

## 4. 提取 RGB 帧和光流

如果用户有足够的 SSD 空间，那么建议将视频抽取为 RGB 帧以提升 I/O 性能。用户可以使用以下脚本为抽取得到的帧文件夹建立软连接：

```shell
# 执行以下脚本 (假设 SSD 被挂载在 "/mnt/SSD/")
```

如果用户只使用 RGB 帧（由于光流提取非常耗时），可执行以下脚本使用 denseflow 提取 RGB 帧：

```shell
```

如果用户未安装 denseflow，可执行以下脚本使用 ffmpeg 提取 RGB 帧：

```shell
```

如果同时需要 RGB 帧和光流，可使用如下脚本抽帧：

```shell
```

## 5. 下载 AVA 上人体检测结果

```

```shell
```

```shell
```

以下脚本修改自 [Long-Term Feature Banks](https://github.com/facebookresearch/video-long-term-feature-banks)。

可使用以下脚本下载 AVA 上预先计算的人体检测结果：

```shell
```

## 6. 目录结构

在完整完成 AVA 的数据处理后，将得到帧文件夹（RGB 帧和光流帧），视频以及标注文件。

在整个项目目录下（仅针对 AVA），*最简* 目录结构如下所示：

```
mmaction2
├── tools
├── data
│   ├── ava

│   │   │   ├── ...
│   │   │   ├── ...
```

关于 AVA 数据集上的训练与测试，请参照 [训练教程](/docs/zh_cn/user_guides/train_test.md)。

# EdgeNeXt

> [EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications](https://arxiv.org/abs/2206.10589)

In the pursuit of achieving ever-increasing accuracy, large and complex neural networks are usually developed. Such models demand high computational resources and therefore cannot be deployed on edge devices. It is of great interest to build resource-efficient general purpose networks due to their usefulness in several application areas. In this work, we strive to effectively combine the strengths of both CNN and Transformer models and propose a new efficient hybrid architecture EdgeNeXt. Specifically in EdgeNeXt, we introduce split depth-wise transpose attention (SDTA) encoder that splits input tensors into multiple channel groups and utilizes depth-wise convolution along with self-attention across channel dimensions to implicitly increase the receptive field and encode multi-scale features. Our extensive experiments on classification, detection and segmentation tasks, reveal the merits of the proposed approach, outperforming state-of-the-art methods with comparatively lower compute requirements. Our EdgeNeXt model with 1.3M parameters achieves 71.2% top-1 accuracy on ImageNet-1K, outperforming MobileViT with an absolute gain of 2.2% with 28% reduction in FLOPs. Further, our EdgeNeXt model with 5.6M parameters achieves 79.4% top-1 accuracy on ImageNet-1K.

<img src="https://github.com/mmaaz60/EdgeNeXt/raw/main/images/EdgeNext.png" width="100%"/>
</div>

```python

predict = inference_model('edgenext-xxsmall_3rdparty_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('edgenext-xxsmall_3rdparty_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/edgenext/edgenext-xxsmall_8xb256_in1k.py https://download.openmmlab.com/mmclassification/v0/edgenext/edgenext-xxsmall_3rdparty_in1k_20220801-7ca8a81d.pth
```

| Model                                |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                   Config                    |                                 Download                                 |
| :----------------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :-----------------------------------------: | :----------------------------------------------------------------------: |

| `edgenext-xxsmall_3rdparty_in1k`\*   | From scratch |    1.33    |   0.26    |   71.20   |   89.91   |  [config](edgenext-xxsmall_8xb256_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/edgenext/edgenext-xxsmall_3rdparty_in1k_20220801-7ca8a81d.pth) |
| `edgenext-xsmall_3rdparty_in1k`\*    | From scratch |    2.34    |   0.53    |   74.86   |   92.31   |  [config](edgenext-xsmall_8xb256_in1k.py)   | [model](https://download.openmmlab.com/mmclassification/v0/edgenext/edgenext-xsmall_3rdparty_in1k_20220801-974f9fe7.pth) |
| `edgenext-small_3rdparty_in1k`\*     | From scratch |    5.59    |   1.25    |   79.41   |   94.53   |   [config](edgenext-small_8xb256_in1k.py)   | [model](https://download.openmmlab.com/mmclassification/v0/edgenext/edgenext-small_3rdparty_in1k_20220801-d00db5f8.pth) |

| `edgenext-small-usi_3rdparty_in1k`\* | From scratch |    5.59    |   1.25    |   81.06   |   95.34   | [config](edgenext-small_8xb256-usi_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/edgenext/edgenext-small_3rdparty-usi_in1k_20220801-ae6d8dd3.pth) |
| `edgenext-base_3rdparty_in1k`\*      | From scratch |   18.51    |   3.81    |   82.48   |   96.20   |   [config](edgenext-base_8xb256_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/edgenext/edgenext-base_3rdparty_in1k_20220801-9ade408b.pth) |
| `edgenext-base_3rdparty-usi_in1k`\*  | From scratch |   18.51    |   3.81    |   83.67   |   96.70   | [config](edgenext-base_8xb256-usi_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/edgenext/edgenext-base_3rdparty-usi_in1k_20220801-909e8939.pth) |

*Models with * are converted from the [official repo](https://github.com/mmaaz60/EdgeNeXt). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@article{Maaz2022EdgeNeXt,
    title={EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications},
    author={Muhammad Maaz and Abdelrahman Shaker and Hisham Cholakkal and Salman Khan and Syed Waqas Zamir and Rao Muhammad Anwer and Fahad Shahbaz Khan},
    journal={2206.10589},
}
```

# Finetuning Models

Detectors pre-trained on the COCO dataset can serve as a good pre-trained model for other datasets, e.g., CityScapes and KITTI Dataset.
This tutorial provides instructions for users to use the models provided in the [Model Zoo](../model_zoo.md) for other datasets to obtain better performance.

There are two steps to finetune a model on a new dataset.

- Add support for the new dataset following [Customize Datasets](../advanced_guides/customize_dataset.md).
- Modify the configs as will be discussed in this tutorial.

Take the finetuning process on Cityscapes Dataset as an example, the users need to modify five parts in the config.

## Inherit base configs

To release the burden and reduce bugs in writing the whole configs, MMDetection V3.0 support inheriting configs from multiple existing configs. To finetune a Mask RCNN model, the new config needs to inherit

`_base_/models/mask-rcnn_r50_fpn.py` to build the basic structure of the model. To use the Cityscapes Dataset, the new config can also simply inherit `_base_/datasets/cityscapes_instance.py`. For runtime settings such as logger settings, the new config needs to inherit `_base_/default_runtime.py`. For training schedules, the new config can to inherit `_base_/schedules/schedule_1x.py`. These configs are in the `configs` directory and the users can also choose to write the whole contents rather than use inheritance.

```python
_base_ = [
    '../_base_/datasets/cityscapes_instance.py', '../_base_/default_runtime.py',
    '../_base_/schedules/schedule_1x.py'
]
```

## Modify head

Then the new config needs to modify the head according to the class numbers of the new datasets. By only changing `num_classes` in the roi_head, the weights of the pre-trained models are mostly reused except for the final prediction head.

```python

```python
            type='Shared2FCBBoxHead',
                type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
            loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=1.0)),
```

## Modify dataset

conv_out_channels=256,
```

The users may also need to prepare the dataset and write the configs about dataset, refer to [Customize Datasets](../advanced_guides/customize_dataset.md) for more detail. MMDetection V3.0 already supports VOC, WIDERFACE, COCO, LIVS, OpenImages, DeepFashion, Objects365, and Cityscapes Dataset.

## Modify training schedule

The finetuning hyperparameters vary from the default schedule. It usually requires a smaller learning rate and fewer training epochs

```python
# lr is set for a batch size of 8
optim_wrapper = dict(optimizer=dict(lr=0.01))

    dict(
    dict(
        begin=0,
        end=8,
        milestones=[7],

    dict(
        begin=0,
        end=8,
]

# max_epochs
train_cfg = dict(max_epochs=8)

# log config
default_hooks = dict(logger=dict(interval=100)),
```

## Use pre-trained model

To use the pre-trained model, the new config adds the link of pre-trained models in the `load_from`. The users might need to download the model weights before training to avoid the download time during training.

```python
load_from = 'https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco_bbox_mAP-0.408__segm_mAP-0.37_20200504_163245-42aa3d00.pth'  # noqa
```

<details>
<summary align="right"><a href="https://arxiv.org/abs/1409.1556">VGG (ICLR'2015)</a></summary>

```bibtex
}
```

<details>

```bibtex
}
```

| [vgg](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_vgg16-bn_8xb64-210e_coco-256x192.py) |  256x192   | 0.699 |      0.890      |      0.769      | 0.754 |      0.927      | [ckpt](https://download.openmmlab.com/mmpose/top_down/vgg/vgg16_bn_coco_256x192-7e7c58d6_20210517.pth) | [log](https://download.openmmlab.com/mmpose/top_down/vgg/vgg16_bn_coco_256x192_20210517.log.json) |

# Changelog of v1.0.x

### v1.0.0 (6/4/2023)

- Add BN in FPN to avoid loss Nan in MVXNet (#2282)
- Update `s3dis_data_utils.py` (#2232)

- Fix precision error when using mixed precision on CenterPoint (#2341)
- Replace `np.transpose` with `torch.permute` to speed up (@2273)
- Update links of SECOND checkpoints (#2185)

@JingweiZhang12, @ZCMax, @Xiangxu-0103, @vansinhu, @cs1488, @sunjiahao1999, @Ginray

### v1.0.0rc7 (7/1/2023)

- Support training and testing on MLU (#2167)

A total of 1 developers contributed to this release.
@mengpenghui

### v1.0.0rc6 (2/12/2022)

- Add `Projects/` folder and the first example project (#2082)

- Update Waymo converter to save storage space (#1759)
- Update model link and performance of CenterPoint (#1916)

- Fix GPU memory occupancy problem in PointRCNN (#1928)
- Fix sampling bug in `IoUNegPiecewiseSampler` (#2018)

@oyel, @zzj403, @VVsssssk, @Tai-Wang, @tpoisonooo, @JingweiZhang12, @ZCMax

### v1.0.0rc5 (11/10/2022)

- Support ImVoxelNet on SUN RGB-D (#1738)

- Fix the cross-codebase reference problem in metafile README (#1644)
- Update the Chinese documentation about getting started (#1715)
- Fix docs link and add docs link checker (#1811)

- Fix a visualization bug that is potentially triggered by empty prediction labels (#1725)
- Fix point cloud segmentation visualization bug due to wrong parameter passing (#1858)
- Fix Nan loss bug during PointRCNN training (#1874)

@ZwwWayne, @Tai-Wang, @filaPro, @VVsssssk, @ZCMax, @Xiangxu-0103, @holtvogt, @tpoisonooo, @lianqing01

### v1.0.0rc4 (8/8/2022)

- Support [FCAF3D](https://arxiv.org/pdf/2112.00322.pdf)

- Support [FCAF3D](https://arxiv.org/pdf/2112.00322.pdf) (#1547)
- Add the transformation to support multi-camera 3D object detection (#1580)
- Support lift-splat-shoot view transformer (#1598)

- Remove the limitation of the maximum number of points during SUN RGB-D preprocessing (#1555)
- Support circle CI (#1647)
- Add mim to extras_require in setup.py (#1560, #1574)
- Update dockerfile package version (#1697)

- Flip yaw angle for DepthInstance3DBoxes.overlaps (#1548, #1556)
- Fix DGCNN configs (#1587)

- Fix bbox head not registered bug (#1625)
- Fix missing objects in S3DIS preprocessing (#1665)
- Fix spconv2.0 model loading bug (#1699)

@Tai-Wang, @ZwwWayne, @filaPro, @lianqing11, @ZCMax, @HuangJunJie2017, @Xiangxu-0103, @ChonghaoSima, @VVsssssk

### v1.0.0rc3 (8/6/2022)

- Support [SA-SSD](https://openaccess.thecvf.com/content_CVPR_2020/papers/He_Structure_Aware_Single-Stage_3D_Object_Detection_From_Point_Cloud_CVPR_2020_paper.pdf)

- Support [SA-SSD](https://openaccess.thecvf.com/content_CVPR_2020/papers/He_Structure_Aware_Single-Stage_3D_Object_Detection_From_Point_Cloud_CVPR_2020_paper.pdf) (#1337)

- Add Chinese documentation for vision-only 3D detection (#1438)

- Update CenterPoint pretrained models that are compatible with refactored coordinate systems (#1450)
- Configure myst-parser to parse anchor tag in the documentation (#1488)
- Replace markdownlint with mdformat for avoiding installing ruby (#1489)
- Add missing `gt_names` when getting annotation info in Custom3DDataset (#1519)
- Support S3DIS full ceph training (#1542)
- Rewrite the installation and FAQ documentation (#1545)

- Fix the incorrect registry name when building RoI extractors (#1460)
- Fix the potential problems caused by the registry scope update when composing pipelines (#1466) and using CocoDataset (#1536)
- Fix the missing selection with `order` in the [box3d_nms](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/core/post_processing/box3d_nms.py) introduced by [#1403](https://github.com/open-mmlab/mmdetection3d/pull/1403) (#1479)

- Update the [PointPillars config](https://github.com/open-mmlab/mmdetection3d/blob/master/configs/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-car.py) to make it consistent with the log (#1486)
- Fix heading anchor in documentation (#1490)
- Fix the compatibility of mmcv in the dockerfile (#1508)
- Make overwrite_spconv packaged when building whl (#1516)
- Fix the requirement of mmcv and mmdet (#1537)
- Update configs of PartA2 and support its compatibility with spconv 2.0 (#1538)

@Xiangxu-0103, @ZCMax, @jshilong, @filaPro, @atinfinity, @Tai-Wang, @wenbo-yu, @yi-chen-isuzu, @ZwwWayne, @wchen61, @VVsssssk, @AlexPasqua, @lianqing11

### v1.0.0rc2 (1/5/2022)

- Support spconv 2.0
- Support MinkowskiEngine with MinkResNet
- Support training models on custom datasets with only point clouds
- Update Registry to distinguish the scope of built functions

- Replace mmcv.iou3d with a set of bird-eye-view (BEV) operators to unify the operations of rotated boxes

- Add loader arguments in the configuration files (#1388)
- Support [spconv 2.0](https://github.com/traveller59/spconv) when the package is installed. Users can still use spconv 1.x in MMCV with CUDA 9.0 (only cost more memory) without losing the compatibility of model weights between two versions (#1421)
- Support MinkowskiEngine with MinkResNet (#1422)

- Add the documentation for model deployment (#1373, #1436)
- Add Chinese documentation of
  - Speed benchmark (#1379)
  - LiDAR-based 3D detection (#1368)
  - LiDAR 3D segmentation (#1420)
  - Coordinate system refactoring (#1384)

- Support training models on custom datasets with only point clouds (#1393)
- Replace mmcv.iou3d with a set of bird-eye-view (BEV) operators to unify the operations of rotated boxes (#1403, #1418)
- Update Registry to distinguish the scope of building functions (#1412, #1443)
- Replace recommonmark with myst_parser for documentation rendering (#1414)

- Fix the show pipeline in the [browse_dataset.py](https://github.com/open-mmlab/mmdetection3d/blob/master/tools/misc/browse_dataset.py) (#1376)
- Fix missing __init__ files after coordinate system refactoring (#1383)
- Fix the incorrect yaw in the visualization caused by coordinate system refactoring (#1407)
- Fix `NaiveSyncBatchNorm1d` and `NaiveSyncBatchNorm2d` to support non-distributed cases and more general inputs (#1435)

@ZCMax, @ZwwWayne, @Tai-Wang, @VVsssssk, @HanaRo, @JoeyforJoy, @ansonlcy, @filaPro, @jshilong, @Xiangxu-0103, @deleomike

### v1.0.0rc1 (1/4/2022)

#### Compatibility

- We migrate all the mmdet3d ops to mmcv and do not need to compile them when installing mmdet3d.
- To fix the imprecise timestamp and optimize its saving method, we reformat the point cloud data during Waymo data conversion. The data conversion time is also optimized significantly by supporting parallel processing. Please re-generate KITTI format Waymo data if necessary. See more details in the [compatibility documentation](https://github.com/open-mmlab/mmdetection3d/blob/master/docs/en/compatibility.md).

- We update some of the model checkpoints after the refactor of coordinate systems. Please stay tuned for the release of the remaining model checkpoints.

|               | Fully Updated | Partially Updated | In Progress | No Influcence |
| ------------- | :-----------: | :---------------: | :---------: | :-----------: |
| SECOND        |               |         ✓         |             |               |
| PointPillars  |               |         ✓         |             |               |
| FreeAnchor    |       ✓       |                   |             |               |
| VoteNet       |       ✓       |                   |             |               |
| H3DNet        |       ✓       |                   |             |               |
| 3DSSD         |               |         ✓         |             |               |
| Part-A2       |       ✓       |                   |             |               |
| MVXNet        |       ✓       |                   |             |               |

| CenterPoint   |               |                   |      ✓      |               |
| SSN           |       ✓       |                   |             |               |
| ImVoteNet     |       ✓       |                   |             |               |
| FCOS3D        |               |                   |             |       ✓       |
| PointNet++    |               |                   |             |       ✓       |
| Group-Free-3D |               |                   |             |       ✓       |
| ImVoxelNet    |       ✓       |                   |             |               |
| PAConv        |               |                   |             |       ✓       |
| DGCNN         |               |                   |             |       ✓       |

| SMOKE         |               |                   |             |       ✓       |
| PGD           |               |                   |             |       ✓       |
| MonoFlex      |               |                   |             |       ✓       |

- Migrate all the mmdet3d ops to mmcv
- Support parallel waymo data converter
- Add ScanNet instance segmentation dataset with metrics
- Better compatibility for windows with CI support, op migration and bug fixes
- Support loading annotations from Ceph

- Add ScanNet instance segmentation dataset with metrics (#1230)
- Support different random seeds for different ranks (#1321)
- Support loading annotations from Ceph (#1325)
- Support resuming from the latest checkpoint automatically (#1329)
- Add windows CI (#1345)

- Update the table format and OpenMMLab project orders in [README.md](https://github.com/open-mmlab/mmdetection3d/blob/master/README.md) (#1272, #1283)
- Migrate all the mmdet3d ops to mmcv (#1240, #1286, #1290, #1333)
- Add `with_plane` flag in the KITTI data conversion (#1278)
- Update instructions and links in the documentation (#1300, 1309, #1319)
- Support parallel Waymo dataset converter and ground truth database generator (#1327)
- Add quick installation commands to [getting_started.md](https://github.com/open-mmlab/mmdetection3d/blob/master/docs/en/getting_started.md) (#1366)

- Update nuimages configs to use new nms config style (#1258)
- Fix the usage of np.long for windows compatibility (#1270)
- Fix the incorrect indexing in `BasePoints` (#1274)

- Fix the incorrect indexing in the [pillar_scatter.forward_single](https://github.com/open-mmlab/mmdetection3d/blob/dev/mmdet3d/models/middle_encoders/pillar_scatter.py#L38) (#1280)
- Fix unit tests that use GPUs (#1301)
- Fix incorrect feature dimensions in `DynamicPillarFeatureNet` caused by previous upgrading of `PillarFeatureNet` (#1302)
- Remove the `CameraPoints` constraint in `PointSample` (#1314)
- Fix imprecise timestamps saving of Waymo dataset (#1327)

@ZCMax, @ZwwWayne, @wHao-Wu, @Tai-Wang, @wangruohui, @zjwzcx, @Xiangxu-0103, @EdAyers, @hongye-dev, @zhanggefan

### v1.0.0rc0 (18/2/2022)

- We refactor our three coordinate systems to make their rotation directions and origins more consistent, and further remove unnecessary hacks in different datasets and models. Therefore, please re-generate data infos or convert the old version to the new one with our provided scripts. We will also provide updated checkpoints in the next version. Please refer to the [compatibility documentation](https://github.com/open-mmlab/mmdetection3d/blob/v1.0.0.dev0/docs/en/compatibility.md) for more details.
- Unify the camera keys for consistent transformation between coordinate systems on different datasets. The modification changes the key names to `lidar2img`, `depth2img`, `cam2img`, etc., for easier understanding. Customized codes using legacy keys may be influenced.

- The next release will begin to move files of CUDA ops to [MMCV](https://github.com/open-mmlab/mmcv). It will influence the way to import related functions. We will not break the compatibility but will raise a warning first and please prepare to migrate it.

- Support new monocular 3D detectors: [PGD](https://github.com/open-mmlab/mmdetection3d/tree/v1.0.0.dev0/configs/pgd), [SMOKE](https://github.com/open-mmlab/mmdetection3d/tree/v1.0.0.dev0/configs/smoke), [MonoFlex](https://github.com/open-mmlab/mmdetection3d/tree/v1.0.0.dev0/configs/monoflex)
- Support a new LiDAR-based detector: [PointRCNN](https://github.com/open-mmlab/mmdetection3d/tree/v1.0.0.dev0/configs/point_rcnn)
- Support a new backbone: [DGCNN](https://github.com/open-mmlab/mmdetection3d/tree/v1.0.0.dev0/configs/dgcnn)
- Support 3D object detection on the S3DIS dataset
- Support compilation on Windows
- Full benchmark for PAConv on S3DIS

- Further enhancement for documentation, especially on the Chinese documentation

- Support 3D object detection on the S3DIS dataset (#835)
- Support PointRCNN (#842, #843, #856, #974, #1022, #1109, #1125)
- Support DGCNN (#896)
- Support PGD (#938, #940, #948, #950, #964, #1014, #1065, #1070, #1157)
- Support SMOKE (#939, #955, #959, #975, #988, #999, #1029)
- Support MonoFlex (#1026, #1044, #1114, #1115, #1183)
- Support CPU Training (#1196)

- Support point sampling based on distance metric (#667, #840)
- Refactor coordinate systems (#677, #774, #803, #899, #906, #912, #968, #1001)
- Unify camera keys in PointFusion and transformations between different systems (#791, #805)
- Refine documentation (#792, #827, #829, #836, #849, #854, #859, #1111, #1113, #1116, #1121, #1132, #1135, #1185, #1193, #1226)

- Add a script to support benchmark regression (#808)
- Benchmark PAConvCUDA on S3DIS (#847)
- Support to download pdf and epub documentation (#850)
- Change the `repeat` setting in Group-Free-3D configs to reduce training epochs (#855)
- Support KITTI AP40 evaluation metric (#927)
- Add the mmdet3d2torchserve tool for SECOND (#977)
- Add code-spell pre-commit hook and fix typos (#995)
- Support the latest numba version (#1043)
- Set a default seed to use when the random seed is not specified (#1072)
- Distribute mix-precision models to each algorithm folder (#1074)
- Add abstract and a representative figure for each algorithm (#1086)
- Upgrade pre-commit hook (#1088, #1217)
- Support augmented data and ground truth visualization (#1092)
- Add local yaw property for `CameraInstance3DBoxes` (#1130)
- Lock the required numba version to 0.53.0 (#1159)

- Support the usage of plane information for KITTI dataset (#1162)
- Deprecate the support for "python setup.py test" (#1164)
- Reduce the number of multi-process threads to accelerate training (#1168)
- Support 3D flip augmentation for semantic segmentation (#1181)
- Update README format for each model (#1195)

- Fix compiling errors on Windows (#766)
- Fix the deprecated nms setting in the ImVoteNet config (#828)
- Use the latest `wrap_fp16_model` import from mmcv (#861)
- Remove 2D annotations generation on Lyft (#867)
- Update index files for the Chinese documentation to be consistent with the English version (#873)
- Fix the nested list transpose in the CenterPoint head (#879)
- Fix deprecated pretrained model loading for RegNet (#889)

- Fix the incorrect dimension indices of rotations and testing config in the CenterPoint test time augmentation (#892)
- Fix and improve visualization tools (#956, #1066, #1073)
- Fix PointPillars FLOPs calculation error (#1075)
- Fix missing dimension information in the SUN RGB-D data generation (#1120)
- Fix incorrect anchor range settings in the PointPillars [config](https://github.com/open-mmlab/mmdetection3d/blob/master/configs/_base_/models/hv_pointpillars_secfpn_kitti.py) for KITTI (#1163)
- Fix incorrect model information in the RegNet metafile (#1184)
- Fix bugs in non-distributed multi-gpu training and testing (#1197)
- Fix a potential assertion error when generating corners from an empty box (#1212)
- Upgrade bazel version according to the requirement of Waymo Devkit (#1223)

@THU17cyz, @wHao-Wu, @wangruohui, @Wuziyi616, @filaPro, @ZwwWayne, @Tai-Wang, @DCNSW, @xieenze, @robin-karlsson0, @ZCMax, @Otteri

### v0.18.1 (1/2/2022)

- Support Flip3D augmentation in semantic segmentation task (#1182)
- Update regnet metafile (#1184)
- Add point cloud annotation tools introduction in FAQ (#1185)
- Add missing explanations of `cam_intrinsic` in the nuScenes dataset doc (#1193)

- Fix the rotation matrix while rotation axis=0 (#1182)
- Fix the bug in non-distributed multi-gpu training/testing (#1197)
- Fix a potential bug when generating corners for empty bounding boxes (#1212)

A total of 4 developers contributed to this release.

@ZwwWayne, @ZCMax, @Tai-Wang, @wHao-Wu

### v0.18.0 (1/1/2022)

- Update the required minimum version of mmdet and mmseg

- Use the official markdownlint hook and add codespell hook for pre-committing (#1088)
- Improve CI operation (#1095, #1102, #1103)
- Use shared menu content from OpenMMLab's theme and remove duplicated contents from config (#1111)
- Refactor the structure of documentation (#1113, #1121)
- Update the required minimum version of mmdet and mmseg (#1147)

- Fix symlink failure on Windows (#1096)
- Fix the upper bound of mmcv version in the mminstall requirements (#1104)
- Fix API documentation compilation and mmcv build errors (#1116)
- Fix figure links and pdf documentation compilation (#1132, #1135)

### v0.17.3 (1/12/2021)

- Change the default show value to `False` in show_result function to avoid unnecessary errors (#1034)
- Improve the visualization of detection results with colorized points in [single_gpu_test](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/apis/test.py#L11) (#1050)
- Clean unnecessary custom_imports in entrypoints (#1068)

- Update mmcv version in the Dockerfile (#1036)
- Fix the memory-leak problem when loading checkpoints in [init_model](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/apis/inference.py#L36) (#1045)
- Fix incorrect velocity indexing when formatting boxes on nuScenes (#1049)
- Explicitly set cuda device ID in [init_model](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/apis/inference.py#L36) to avoid memory allocation on unexpected devices (#1056)

- Fix PointPillars FLOPs calculation error (#1076)

@wHao-Wu, @Tai-Wang, @ZCMax, @MilkClouds, @aldakata

### v0.17.2 (1/11/2021)

- Update Group-Free-3D and FCOS3D bibtex (#985)
- Update the solutions for incompatibility of pycocotools in the FAQ (#993)
- Add Chinese documentation for the KITTI (#1003) and Lyft (#1010) dataset tutorial
- Add the H3DNet checkpoint converter for incompatible keys (#1007)

- Update mmdetection and mmsegmentation version in the Dockerfile (#992)
- Fix links in the Chinese documentation (#1015)

@Tai-Wang, @wHao-Wu, @ZwwWayne, @ZCMax

### v0.17.1 (1/10/2021)

- Support a faster but non-deterministic version of hard voxelization
- Completion of dataset tutorials and the Chinese documentation
- Improved the aesthetics of the documentation format

- Add Chinese documentation for training on customized datasets and designing customized models (#729, #820)
- Support a faster but non-deterministic version of hard voxelization (#904)
- Update paper titles and code details for metafiles (#917)
- Add a tutorial for KITTI dataset (#953)
- Use Pytorch sphinx theme to improve the format of documentation (#958)
- Use the docker to accelerate CI (#971)

- Fix the sphinx version used in the documentation (#902)
- Fix a dynamic scatter bug that discards the first voxel by mistake when all input points are valid (#915)

- Fix the inconsistent variable names used in the [unit test](https://github.com/open-mmlab/mmdetection3d/blob/master/tests/test_models/test_voxel_encoder/test_voxel_generator.py) for voxel generator (#919)
- Upgrade to use `build_prior_generator` to replace the legacy `build_anchor_generator` (#941)
- Fix a minor bug caused by a too small difference set in the FreeAnchor Head (#944)

@DCNSW, @zhanggefan, @mickeyouyou, @ZCMax, @wHao-Wu, @tojimahammatov, @xiliu8006, @Tai-Wang

### v0.17.0 (1/9/2021)

- Unify the camera keys for consistent transformation between coordinate systems on different datasets. The modification change the key names to `lidar2img`, `depth2img`, `cam2img`, etc. for easier understanding. Customized codes using legacy keys may be influenced.

- Update PointFusion to support unified camera keys (#791)
- Add Chinese documentation for customized dataset (#792), data pipeline (#827), customized runtime (#829), 3D Detection on ScanNet (#836), nuScenes (#854) and Waymo (#859)
- Unify camera keys used in transformation between different systems (#805)
- Add a tutorial for 3D detection on the Lyft dataset (#849)

@THU17cyz, @wHao-Wu, @wangruohui, @Wuziyi616, @filaPro, @ZwwWayne, @Tai-Wang, @DCNSW, @xieenze, @robin-karlsson0, @ZCMax

### v0.16.0 (1/8/2021)

- Remove the rotation and dimension hack in the monocular 3D detection on nuScenes by applying corresponding transformation in the pre-processing and post-processing. The modification only influences nuScenes coco-style json files. Please re-run the data preparation scripts if necessary. See more details in the PR #744.
- Add a new pre-processing module for the ScanNet dataset in order to support multi-view detectors. Please run the updated scripts to extract the RGB data and its annotations. See more details in the PR #696.

- Support to use [MIM](https://github.com/open-mmlab/mim) with pip installation
- Support PAConv [models and benchmarks](https://github.com/open-mmlab/mmdetection3d/tree/master/configs/paconv) on S3DIS
- Enhance the documentation especially on dataset tutorials

- Support RGB images on ScanNet for multi-view detectors (#696)
- Support FLOPs and number of parameters calculation (#736)
- Support to use [MIM](https://github.com/open-mmlab/mim) with pip installation (#782)
- Support PAConv models and benchmarks on the S3DIS dataset (#783, #809)

- Refactor Group-Free-3D to make it inherit BaseModule from MMCV (#704)
- Modify the initialization methods of FCOS3D to be consistent with the refactored approach (#705)
- Benchmark the Group-Free-3D [models](https://github.com/open-mmlab/mmdetection3d/tree/master/configs/groupfree3d) on ScanNet (#710)
- Add Chinese documentation for Getting Started (#725), FAQ (#730), Model Zoo (#735), Demo (#745), Quick Run (#746), Data Preparation (#787) and Configs (#788)
- Add documentation for semantic segmentation on ScanNet and S3DIS (#743, #747, #806, #807)

- Add a parameter `max_keep_ckpts` to limit the maximum number of saved Group-Free-3D checkpoints (#765)
- Add documentation for 3D detection on SUN RGB-D and nuScenes (#770, #793)
- Remove mmpycocotools in the Dockerfile (#785)

- Fix versions of OpenMMLab dependencies (#708)
- Convert `rt_mat` to `torch.Tensor` in coordinate transformation for compatibility (#709)
- Fix the `bev_range` initialization in `ObjectRangeFilter` according to the `gt_bboxes_3d` type (#717)
- Fix Chinese documentation and incorrect doc format due to the incompatible Sphinx version (#718)
- Fix a potential bug when setting `interval == 1` in [analyze_logs.py](https://github.com/open-mmlab/mmdetection3d/blob/master/tools/analysis_tools/analyze_logs.py) (#720)
- Update the structure of Chinese documentation (#722)
- Fix FCOS3D FPN BC-Breaking caused by the code refactoring in MMDetection (#739)

- Fix wrong `in_channels` when `with_distance=True` in the [Dynamic VFE Layers](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/models/voxel_encoders/voxel_encoder.py#L87) (#749)
- Fix the dimension and yaw hack of FCOS3D on nuScenes (#744, #794, #795, #818)
- Fix the missing default `bbox_mode` in the `show_multi_modality_result` (#825)

@yinchimaoliang, @gopi231091, @filaPro, @ZwwWayne, @ZCMax, @hjin2902, @wHao-Wu, @Wuziyi616, @xiliu8006, @THU17cyz, @DCNSW, @Tai-Wang

### v0.15.0 (1/7/2021)

- Support [PAConv](https://arxiv.org/abs/2103.14635)
- Support monocular/multi-view 3D detector [ImVoxelNet](https://arxiv.org/abs/2106.01178) on KITTI
- Support Transformer-based 3D detection method [Group-Free-3D](https://arxiv.org/abs/2104.00678) on ScanNet
- Add documentation for tasks including LiDAR-based 3D detection, vision-only 3D detection and point-based 3D semantic segmentation

- Add dataset documents like ScanNet

- Support Group-Free-3D on ScanNet (#539)
- Support PAConv modules (#598, #599)
- Support ImVoxelNet on KITTI (#627, #654)

- Add unit tests for pipeline functions `LoadImageFromFileMono3D`, `ObjectNameFilter` and `ObjectRangeFilter` (#615)
- Enhance [IndoorPatchPointSample](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/datasets/pipelines/transforms_3d.py) (#617)
- Refactor model initialization methods based MMCV (#622)
- Add Chinese docs (#629)
- Add documentation for LiDAR-based 3D detection (#642)
- Unify intrinsic and extrinsic matrices for all datasets (#653)
- Add documentation for point-based 3D semantic segmentation (#663)
- Add documentation of ScanNet for 3D detection (#664)
- Refine docs for tutorials (#666)

- Add documentation for vision-only 3D detection (#669)
- Refine docs for Quick Run and Useful Tools (#686)

- Fix the bug of [BackgroundPointsFilter](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/datasets/pipelines/transforms_3d.py) using the bottom center of ground truth (#609)
- Fix [LoadMultiViewImageFromFiles](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/datasets/pipelines/loading.py) to unravel stacked multi-view images to list to be consistent with DefaultFormatBundle (#611)
- Fix the potential bug in [analyze_logs](https://github.com/open-mmlab/mmdetection3d/blob/master/tools/analysis_tools/analyze_logs.py) when the training resumes from a checkpoint or is stopped before evaluation (#634)
- Fix test commands in docs and make some refinements (#635)

- Fix wrong config paths in unit tests (#641)

### v0.14.0 (1/6/2021)

- Support the point cloud segmentation method [PointNet++](https://arxiv.org/abs/1706.02413)

- Support PointNet++ (#479, #528, #532, #541)
- Support RandomJitterPoints transform for point cloud segmentation (#584)
- Support RandomDropPointsColor transform for point cloud segmentation (#585)

- Move the point alignment of ScanNet from data pre-processing to pipeline (#439, #470)
- Add compatibility document to provide detailed descriptions of BC-breaking changes (#504)
- Add MMSegmentation installation requirement (#535)
- Support points rotation even without bounding box in GlobalRotScaleTrans for point cloud segmentaiton (#540)
- Support visualization of detection results and dataset browse for nuScenes Mono-3D dataset (#542, #582)
- Support faster implementation of KNN (#586)

- Support RegNetX models on Lyft dataset (#589)
- Remove a useless parameter `label_weight` from segmentation datasets including `Custom3DSegDataset`, `ScanNetSegDataset` and `S3DISSegDataset` (#607)

- Fix a corrupted lidar data file in Lyft dataset in [data_preparation](https://github.com/open-mmlab/mmdetection3d/tree/master/docs/data_preparation.md) (#546)
- Fix evaluation bugs in nuScenes and Lyft dataset (#549)
- Fix converting points between coordinates with specific transformation matrix in the [coord_3d_mode.py](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/core/bbox/structures/coord_3d_mode.py) (#556)
- Support PointPillars models on Lyft dataset (#578)
- Fix the bug of demo with pre-trained VoteNet model on ScanNet (#600)

### v0.13.0 (1/5/2021)

- Support a monocular 3D detection method [FCOS3D](https://arxiv.org/abs/2104.10956)
- Support ScanNet and S3DIS semantic segmentation dataset
- Enhancement of visualization tools for dataset browsing and demos, including support of visualization for multi-modality data and point cloud segmentation.

- Support ScanNet semantic segmentation dataset (#390)
- Support monocular 3D detection on nuScenes (#392)
- Support multi-modality visualization (#405)
- Support nuimages visualization (#408)
- Support monocular 3D detection on KITTI (#415)
- Support online visualization of semantic segmentation results (#416)
- Support ScanNet test results submission to online benchmark (#418)
- Support S3DIS data pre-processing and dataset class (#433)
- Support FCOS3D (#436, #442, #482, #484)

- Support dataset browse for multiple types of datasets (#467)
- Adding paper-with-code (PWC) metafile for each model in the model zoo (#485)

- Support dataset browsing for SUNRGBD, ScanNet or KITTI points and detection results (#367)
- Add the pipeline to load data using file client (#430)
- Support to customize the type of runner (#437)
- Make pipeline functions process points and masks simultaneously when sampling points (#444)
- Add waymo unit tests (#455)
- Split the visualization of projecting points onto image from that for only points (#480)
- Efficient implementation of PointSegClassMapping (#489)
- Use the new model registry from mmcv (#495)

- Fix Pytorch 1.8 Compilation issue in the [scatter_points_cuda.cu](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/ops/voxel/src/scatter_points_cuda.cu) (#404)
- Fix [dynamic_scatter](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/ops/voxel/src/scatter_points_cuda.cu) errors triggered by empty point input (#417)
- Fix the bug of missing points caused by using break incorrectly in the voxelization (#423)
- Fix the missing `coord_type` in the waymo dataset [config](https://github.com/open-mmlab/mmdetection3d/blob/master/configs/_base_/datasets/waymoD5-3d-3class.py) (#441)

- Fix errors in four unittest functions of [configs](https://github.com/open-mmlab/mmdetection3d/blob/master/configs/ssn/hv_ssn_secfpn_sbn-all_2x16_2x_lyft-3d.py), [test_detectors.py](https://github.com/open-mmlab/mmdetection3d/blob/master/tests/test_models/test_detectors.py), [test_heads.py](https://github.com/open-mmlab/mmdetection3d/blob/master/tests/test_models/test_heads/test_heads.py) (#453)
- Fix 3DSSD training errors and simplify configs (#462)
- Clamp 3D votes projections to image boundaries in ImVoteNet (#463)
- Update out-of-date names of pipelines in the [config](https://github.com/open-mmlab/mmdetection3d/blob/master/configs/benchmark/hv_pointpillars_secfpn_3x8_100e_det3d_kitti-3d-car.py) of pointpillars benchmark (#474)

- Fix the lack of a placeholder when unpacking RPN targets in the [h3d_bbox_head.py](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/models/roi_heads/bbox_heads/h3d_bbox_head.py) (#508)
- Fix the incorrect value of `K` when creating pickle files for SUN RGB-D (#511)

### v0.12.0 (1/4/2021)

- Support a new multi-modality method [ImVoteNet](https://arxiv.org/abs/2001.10692).
- Support PyTorch 1.7 and 1.8
- Refactor the structure of tools and [train.py](https://github.com/open-mmlab/mmdetection3d/blob/master/tools/train.py)/[test.py](https://github.com/open-mmlab/mmdetection3d/blob/master/tools/test.py)

- Support LiDAR-based semantic segmentation metrics (#332)
- Support [ImVoteNet](https://arxiv.org/abs/2001.10692) (#352, #384)
- Support the KNN GPU operation (#360, #371)

- Add FAQ for common problems in the documentation (#333)
- Refactor the structure of tools (#339)

- Refactor [train.py](https://github.com/open-mmlab/mmdetection3d/blob/master/tools/train.py) and [test.py](https://github.com/open-mmlab/mmdetection3d/blob/master/tools/test.py) (#343)
- Support demo on nuScenes (#353)
- Add 3DSSD checkpoints (#359)
- Update the Bibtex of CenterPoint (#368)
- Add citation format and reference to other OpenMMLab projects in the README (#374)
- Upgrade the mmcv version requirements (#376)
- Add numba and numpy version requirements in FAQ (#379)
- Avoid unnecessary for-loop execution of vfe layer creation (#389)
- Update SUNRGBD dataset documentation to stress the requirements for training ImVoteNet (#391)
- Modify vote head to support 3DSSD (#396)

- Fix missing keys `coord_type` in database sampler config (#345)
- Rename H3DNet configs (#349)

- Fix CI by using ubuntu 18.04 in github workflow (#350)
- Add assertions to avoid 4-dim points being input to [points_in_boxes](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/ops/roiaware_pool3d/points_in_boxes.py) (#357)
- Fix the SECOND results on Waymo in the corresponding [README](https://github.com/open-mmlab/mmdetection3d/tree/master/configs/second) (#363)
- Fix the incorrect adopted pipeline when adding val to workflow (#370)
- Fix a potential bug when indices used in the backwarding in ThreeNN (#377)
- Fix a compilation error triggered by [scatter_points_cuda.cu](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/ops/voxel/src/scatter_points_cuda.cu) in PyTorch 1.7 (#393)

### v0.11.0 (1/3/2021)

- Support more friendly visualization interfaces based on open3d

- Support a faster and more memory-efficient implementation of DynamicScatter
- Refactor unit tests and details of configs

- Support new visualization methods based on open3d (#284, #323)

- Refactor unit tests (#303)
- Move the key `train_cfg` and `test_cfg` into the model configs (#307)
- Update [README](https://github.com/open-mmlab/mmdetection3d/blob/master/README.md/) with [Chinese version](https://github.com/open-mmlab/mmdetection3d/blob/master/README_zh-CN.md/) and [instructions for getting started](https://github.com/open-mmlab/mmdetection3d/blob/master/docs/getting_started.md/). (#310, #316)
- Support a faster and more memory-efficient implementation of DynamicScatter (#318, #326)

- Fix an unsupported bias setting in the unit test for centerpoint head (#304)
- Fix errors due to typos in the centerpoint head (#308)

- Fix a minor bug in [points_in_boxes.py](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/ops/roiaware_pool3d/points_in_boxes.py) when tensors are not in the same device. (#317)
- Fix warning of deprecated usages of nonzero during training with PyTorch 1.6 (#330)

### v0.10.0 (1/2/2021)

- Preliminary release of API for SemanticKITTI dataset.
- Documentation and demo enhancement for better user experience.
- Fix a number of underlying minor bugs and add some corresponding important unit tests.

- Support SemanticKITTI dataset preliminarily (#287)

- Add tag to README in configurations for specifying different uses (#262)
- Update instructions for evaluation metrics in the documentation (#265)

- Add nuImages entry in [README.md](https://github.com/open-mmlab/mmdetection3d/blob/master/README.md/) and gif demo (#266, #268)
- Add unit test for voxelization (#275)

- Fixed the issue of unpacking size in [furthest_point_sample.py](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/ops/furthest_point_sample/furthest_point_sample.py) (#248)
- Fix bugs for 3DSSD triggered by empty ground truths (#258)
- Remove models without checkpoints in model zoo statistics of documentation (#259)
- Fix some unclear installation instructions in [getting_started.md](https://github.com/open-mmlab/mmdetection3d/blob/master/docs/getting_started.md/) (#269)
- Fix relative paths/links in the documentation (#271)

- Fix a minor bug in [scatter_points_cuda.cu](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/ops/voxel/src/scatter_points_cuda.cu) when num_features != 4 (#275)
- Fix the bug about missing text files when testing on KITTI (#278)
- Fix issues caused by inplace modification of tensors in `BaseInstance3DBoxes` (#283)
- Fix log analysis for evaluation and adjust the documentation accordingly (#285)

### v0.9.0 (31/12/2020)

- Documentation refactoring with better structure, especially about how to implement new models and customized datasets.
- More compatible with refactored point structure by bug fixes in ground truth sampling.

- Documentation refactoring (#242)

- Fix point structure related bugs in ground truth sampling (#211)
- Fix loading points in ground truth sampling augmentation on nuScenes (#221)

- Fix channel setting in the SeparateHead of CenterPoint (#228)
- Fix evaluation for indoors 3D detection in case of less classes in prediction (#231)
- Remove unreachable lines in nuScenes data converter (#235)
- Minor adjustments of numpy implementation for perspective projection and prediction filtering criterion in KITTI evaluation (#241)

### v0.8.0 (30/11/2020)

- Refactor points structure with more constructive and clearer implementation.
- Support axis-aligned IoU loss for VoteNet with better performance.
- Update and enhance [SECOND](https://github.com/open-mmlab/mmdetection3d/tree/master/configs/second) benchmark on Waymo.

- Support axis-aligned IoU loss for VoteNet. (#194)

- Support points structure for consistent processing of all the point related representation. (#196, #204)

- Enhance [SECOND](https://github.com/open-mmlab/mmdetection3d/tree/master/configs/second) benchmark on Waymo with stronger baselines. (#205)
- Add model zoo statistics and polish the documentation. (#201)

### v0.7.0 (1/11/2020)

- Support a new method [SSN](https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123700579.pdf) with benchmarks on nuScenes and Lyft datasets.
- Update benchmarks for SECOND on Waymo, CenterPoint with TTA on nuScenes and models with mixed precision training on KITTI and nuScenes.

- Support semantic segmentation on nuImages and provide [HTC](https://arxiv.org/abs/1901.07518) models with configurations and performance for reference.

- Modified primitive head which can support the setting on SUN-RGBD dataset (#136)
- Support semantic segmentation and [HTC](https://github.com/open-mmlab/mmdetection3d/tree/master/configs/nuimages) with models for reference on nuImages dataset (#155)
- Support [SSN](https://github.com/open-mmlab/mmdetection3d/tree/master/configs/ssn) on nuScenes and Lyft datasets (#147, #174, #166, #182)
- Support double flip for test time augmentation of CenterPoint with updated benchmark (#143)

- Update [SECOND](https://github.com/open-mmlab/mmdetection3d/tree/master/configs/second) benchmark with configurations for reference on Waymo (#166)

- Delete checkpoints on Waymo to comply its specific license agreement (#180)
- Update models and instructions with [mixed precision training](https://github.com/open-mmlab/mmdetection3d/tree/master/configs/fp16) on KITTI and nuScenes (#178)

- Fix incorrect code weights in anchor3d_head when introducing mixed precision training (#173)
- Fix the incorrect label mapping on nuImages dataset (#155)

### v0.6.1 (11/10/2020)

- Support mixed precision training of voxel-based methods
- Support docker with PyTorch 1.6.0
- Update baseline configs and results ([CenterPoint](https://github.com/open-mmlab/mmdetection3d/tree/master/configs/centerpoint) on nuScenes and [PointPillars](https://github.com/open-mmlab/mmdetection3d/tree/master/configs/pointpillars) on Waymo with full dataset)

- Switch model zoo to download.openmmlab.com

- Support dataset pipeline `VoxelBasedPointSampler` to sample multi-sweep points based on voxelization. (#125)
- Support mixed precision training of voxel-based methods (#132)
- Support docker with PyTorch 1.6.0 (#160)

- Reduce requirements for the case exclusive of Waymo (#121)
- Switch model zoo to download.openmmlab.com (#126)
- Update docs related to Waymo (#128)
- Add version assertion in the [init file](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/__init__.py) (#129)
- Add evaluation interval setting for CenterPoint (#131)
- Add unit test for CenterPoint (#133)
- Update [PointPillars](https://github.com/open-mmlab/mmdetection3d/tree/master/configs/pointpillars) baselines on Waymo with full dataset (#142)
- Update [CenterPoint](https://github.com/open-mmlab/mmdetection3d/tree/master/configs/centerpoint) results with models and logs (#154)

- Fix a bug of visualization in multi-batch case (#120)
- Fix bugs in dcn unit test (#130)
- Fix dcn bias bug in centerpoint (#137)
- Fix dataset mapping in the evaluation of nuScenes mini dataset (#140)
- Fix origin initialization in `CameraInstance3DBoxes` (#148, #150)
- Correct documentation link in the getting_started.md (#159)
- Fix model save path bug in gather_models.py (#153)
- Fix image padding shape bug in `PointFusion` (#162)

### v0.6.0 (20/9/2020)

- Support new methods [H3DNet](https://arxiv.org/abs/2006.05682), [3DSSD](https://arxiv.org/abs/2002.10187), [CenterPoint](https://arxiv.org/abs/2006.11275).
- Support new dataset [Waymo](https://waymo.com/open/) (with PointPillars baselines) and [nuImages](https://www.nuscenes.org/nuimages) (with Mask R-CNN and Cascade Mask R-CNN baselines).

- Support Batch Inference
- Support Pytorch 1.6
- Start to publish `mmdet3d` package to PyPI since v0.5.0. You can use mmdet3d through `pip install mmdet3d`.

#### Backwards Incompatible Changes

- Support Batch Inference (#95, #103, #116): MMDetection3D v0.6.0 migrates to support batch inference based on MMDetection >= v2.4.0. This change influences all the test APIs in MMDetection3D and downstream codebases.
- Start to use collect environment function from MMCV (#113): MMDetection3D v0.6.0 migrates to use `collect_env` function in MMCV.
  `get_compiler_version` and `get_compiling_cuda_version` compiled in `mmdet3d.ops.utils` are removed. Please import these two functions from `mmcv.ops`.

`get_compiler_version` and `get_compiling_cuda_version` compiled in `mmdet3d.ops.utils` are removed. Please import these two functions from `mmcv.ops`.

- Support [nuImages](https://www.nuscenes.org/nuimages) dataset by converting them into coco format and release Mask R-CNN and Cascade Mask R-CNN baseline models (#91, #94)
- Support to publish to PyPI in github-action (#17, #19, #25, #39, #40)
- Support CBGSDataset and make it generally applicable to all the supported datasets (#75, #94)
- Support [H3DNet](https://arxiv.org/abs/2006.05682) and release models on ScanNet dataset (#53, #58, #105)
- Support Fusion Point Sampling used in [3DSSD](https://arxiv.org/abs/2002.10187) (#66)
- Add `BackgroundPointsFilter` to filter background points in data pipeline (#84)
- Support pointnet2 with multi-scale grouping in backbone and refactor pointnets (#82)
- Support dilated ball query used in [3DSSD](https://arxiv.org/abs/2002.10187) (#96)

- Support [3DSSD](https://arxiv.org/abs/2002.10187) and release models on KITTI dataset (#83, #100, #104)
- Support [CenterPoint](https://arxiv.org/abs/2006.11275) and release models on nuScenes dataset (#49, #92)
- Support [Waymo](https://waymo.com/open/) dataset and release PointPillars baseline models (#118)
- Allow `LoadPointsFromMultiSweeps` to pad empty sweeps and select multiple sweeps randomly (#67)

- Fix all warnings and bugs in PyTorch 1.6.0 (#70, #72)
- Update issue templates (#43)
- Update unit tests (#20, #24, #30)
- Update documentation for using `ply` format point cloud data (#41)
- Use points loader to load point cloud data in ground truth (GT) samplers (#87)
- Unify version file of OpenMMLab projects by using `version.py` (#112)

- Remove unnecessary data preprocessing commands of SUN RGB-D dataset (#110)

- Rename CosineAnealing to CosineAnnealing (#57)
- Fix device inconsistent bug in 3D IoU computation (#69)
- Fix a minor bug in json2csv of lyft dataset (#78)
- Add missed test data for pointnet modules (#85)
- Fix `use_valid_flag` bug in `CustomDataset` (#106)

### v0.5.0 (9/7/2020)

MMDetection3D is released.

# 准备 OmniSource

## 简介

```BibTeX
}
```

MMAction2 中发布了 OmniSource 网络数据集的一个子集 (来自论文 [Omni-sourced Webly-supervised Learning for Video Recognition](https://arxiv.org/abs/2003.13042))。
OmniSource 数据集中所有类别均来自 Kinetics-400。MMAction2 所提供的子集包含属于 Mini-Kinetics 数据集 200 类动作的网络数据 (Mini-inetics 数据集由论文 [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://arxiv.org/pdf/1712.04851.pdf) 提出)。

MMAction2 提供所有数据源中属于 Mini-Kinetics 200 类动作的数据，这些数据源包含：Kinetics 数据集，Kinetics 原始数据集（未经裁剪的长视频），来自 Google 和 Instagram 的网络图片，来自 Instagram 的网络视频。为获取这一数据集，用户需先填写 [数据申请表](https://docs.google.com/forms/d/e/1FAIpQLSd8_GlmHzG8FcDbW-OEu__G7qLgOSYZpH-i5vYVJcu7wcb_TQ/viewform?usp=sf_link)。在接收到申请后，下载链接将被发送至用户邮箱。由于发布的数据集均为爬取所得的原始数据，数据集较大，下载需要一定时间。下表中提供了 OmniSource 数据集各个分量的统计信息。

|   数据集名称    | 样本个数 | 所占空间 | 过滤使用的 Teacher 模型 | 过滤后的样本个数 | 与 k200_val 中样本相似（疑似重复）的样本个数 |
| :-------------: | :------: | :------: | :---------------------: | :--------------: | :------------------------------------------: |
|   k200_train    |  76030   |  45.6G   |           N/A           |       N/A        |                     N/A                      |
|    k200_val     |   4838   |   2.9G   |           N/A           |       N/A        |                     N/A                      |
| googleimage_200 | 3050880  |  265.5G  |      TSN-R50-8seg       |     1188695      |                     967                      |

|  insimage_200   | 3654650  |  224.4G  |      TSN-R50-8seg       |      879726      |                     116                      |
|  insvideo_200   |  732855  | 1487.6G  |    SlowOnly-8x8-R50     |      330680      |                     956                      |
| k200_raw_train  |  76027   |  963.5G  |    SlowOnly-8x8-R50     |       N/A        |                     N/A                      |

MMAction2 所发布的 OmniSource 数据集目录结构如下所示：

```
│   │   ├── googleimage_200.txt                       从 Google 爬取到的所有图片列表
│   │   ├── tsn_8seg_googleimage_200_duplicate.txt    从 Google 爬取到的，疑似与 k200-val 中样本重复的正样本列表
│   │   ├── tsn_8seg_googleimage_200.txt              从 Google 爬取到的，经过 teacher 模型过滤的正样本列表
│   │   └── tsn_8seg_googleimage_200_wodup.txt        从 Google 爬取到的，经过 teacher 模型过滤及去重的正样本列表

│   ├── k200_actions.txt                              MiniKinetics 中 200 类动作的名称
│   ├── K400_to_MiniKinetics_classidx_mapping.json    Kinetics 中的类索引至 MiniKinetics 中的类索引的映射
│   └── kinetics_raw_200
│       └── slowonly_8x8_kinetics_raw_200.json        经 teacher 模型过滤后的 Kinetics 原始视频片段
├── googleimage_200                                   共 10 卷

│   ├── ...
├── insimage_200                                      共 10 卷
│   ├── ...
├── insvideo_200                                      共 20 卷
│   ├── ...
└── kinetics_raw_200_train                            共 16 卷
    ├── ...
```

## 数据准备

用户需要首先完成数据下载，对于 `kinetics_200` 和三个网络数据集 `googleimage_200`, `insimage_200`, `insvideo_200`，用户仅需解压各压缩卷并将其合并至一处。

对于 Kinetics 原始视频，由于直接读取长视频非常耗时，用户需要先将其分割为小段。MMAction2 提供了名为 `trim_raw_video.py` 的脚本，用于将长视频分割至 10 秒的小段（分割完成后删除长视频）。用户可利用这一脚本分割长视频。

所有数据应位于 `data/OmniSource/` 目录下。完成数据准备后，`data/OmniSource/` 目录的结构应如下所示（为简洁，省去了训练及测试时未使用的文件）：

```
│   ├── 000
|   │   ├── 00

│   ├── 000
|   │   ├── 00
|   │   │   ├── ...
|   │   ├── ...
|   │   ├── 19
│   ├── ...
│   └── 199
│   ├── 000
|   │   │   ├── ...
│   ├── ...
│   └── 199
│   ├── 000
|   │   │   ├── ...
│   ├── ...
│   └── 199
|   ├── ...
|   ├── ...
│   ├── pref_
|   │   │   ├── ...
│   |   ├── ...

|   ├── ...
│   ├── pref_
|   │   │   ├── ...
│   |   ├── ...
│   ├── ...
│   └── prefZ
```

# Kvasir-SEG Segmented Polyp Dataset from Aliyun (Kvasir SEG Aliyun)

This project supports **`Kvasir-SEG Segmented Polyp Dataset from Aliyun (Kvasir SEG Aliyun) `**, which can be downloaded from [here](https://tianchi.aliyun.com/dataset/84385).

Colorectal cancer is the second most common cancer type among women and third most common among men. Polyps are precursors to colorectal cancer and therefore important to detect and remove at an early stage. Polyps are found in nearly half of the individuals at age 50 that undergo a colonoscopy screening, and their frequency increase with age.Polyps are abnormal tissue growth from the mucous membrane, which is lining the inside of the GI tract, and can sometimes be cancerous. Colonoscopy is the gold standard for detection and assessment of these polyps with subsequent biopsy and removal of the polyps. Early disease detection has a huge impact on survival from colorectal cancer. Increasing the detection of polyps has been shown to decrease risk of colorectal cancer. Thus, automatic detection of more polyps at an early stage can play a crucial role in prevention and survival from colorectal cancer.

The Kvasir-SEG dataset is based on the previous Kvasir dataset, which is the first multi-class dataset for gastrointestinal (GI) tract disease detection and classification. It contains annotated polyp images and their corresponding masks. The pixels depicting polyp tissue, the ROI, are represented by the foreground (white mask), while the background (in black) does not contain positive pixels. These images were collected and verified by experienced gastroenterologists from Vestre Viken Health Trust in Norway. The classes include anatomical landmarks, pathological findings and endoscopic procedures.

| Dataset Name                                           | Anatomical Region | Task Type    | Modality  | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License                                                   |

| ------------------------------------------------------ | ----------------- | ------------ | --------- | ------------ | --------------------- | ---------------------- | ------------ | --------------------------------------------------------- |
| [kvasir-seg](https://tianchi.aliyun.com/dataset/84385) | abdomen           | segmentation | endoscopy | 2            | 1000/-/-              | yes/-/-                | 2020         | [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/) |

| background |    1000    |   84.72    |    -     |    -     |     -     |     -     |
|   polyp    |    1000    |   15.28    |    -     |    -     |     -     |     -     |

Note:

Note:

![kvasir_seg_aliyun](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/endoscopy_images/kvasir_seg_aliyun/kvasir_seg_aliyun_dataset.png?raw=true)

```
@inproceedings{jha2020kvasir,
	title={Kvasir-seg: A segmented polyp dataset},
	author={Jha, Debesh and Smedsrud, Pia H and Riegler, Michael A and Halvorsen, P{\aa}l and Lange, Thomas de and Johansen, Dag and Johansen, H{\aa}vard D},
	booktitle={International Conference on Multimedia Modeling},
	pages={451--462},
	year={2020},
	organization={Springer}
 }
```

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `kvasir_seg_aliyun/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
```

- download dataset from [here](https://tianchi.aliyun.com/dataset/84385) and decompression data to path 'data/.'.

```none
  ├── mmseg
  │   │   │   ├── endoscopy
  │   │   │   │   ├── kvasir_seg_aliyun

```

| background |    800     |   84.66    |   200    |  84.94   |     -     |     -     |
|   polyp    |    800     |   15.34    |   200    |  15.06   |     -     |     -     |

```shell
```

```shell
```

```shell
```

# 学习基础知识 (待更新)

本章将向您介绍旋转目标检测的基本概念，以及旋转目标检测的框架 MMRotate，并提供了详细教程的链接。

## 什么是旋转目标检测

### 问题定义

受益于通用检测的蓬勃发展，目前绝大多数的旋转检测模型都是基于经典的通用检测器。随着检测任务的发展，
水平框在一些细分领域上已经无法满足研究人员的需求。通过重新定义目标表示形式以及增加回归自由度数量
的操作来实现旋转矩形框、四边形甚至任意形状检测，我们称之为旋转目标检测。如何更加高效地进行高精度
的旋转目标检测已成为当下的研究热点。下面列举一些旋转目标检测已经被应用或者有巨大潜力的领域：
人脸识别、场景文字、遥感影像、自动驾驶、医学图像、机器人抓取等。

### 什么是旋转框

旋转目标检测与通用目标检测最大的不同就是用旋转框标注来代替水平框标注，它们的定义如下：

- 水平框: 宽沿 `x` 轴方向，高沿 `y` 轴方向的矩形。通常可以用2个对角顶点的坐标表示
  `(x_i, y_i)`  (i = 1, 2)，也可以用中心点坐标以及宽和高表示
  `(x_center, y_center, width, height)`。
- 旋转框: 由水平框绕中心点旋转一个角度 `angle` 得到，通过添加一个弧度参数得到其旋转框定义法
  `(x_center, y_center, width, height, theta)`。其中，`theta = angle * pi / 180`，
  单位为 `rad`。当旋转的角度为90°的倍数时，旋转框退化为水平框。标注软件导出的旋转框标注通常为多边形
  `(xr_i, yr_i)` (i = 1, 2, 3, 4)，在训练时需要转换为旋转框定义法。

```{note}
在 MMRotate 中，角度参数的单位均为弧度。
```

### 旋转方向

旋转框可以由水平框绕其中心点顺时针旋转或逆时针旋转得到。旋转方向和坐标系的选择密切相关。
图像空间采用右手坐标系 `(y，x)`，其中 y 是`上->下`，x 是`左->右`。
此时存在2种相反的旋转方向：

- 顺时针 (`CW`)

`CW` 的示意图

```
0-------------------> x (0 rad)
|  A-------------B

```

### 旋转方向

`CW` 的示意图

```
|  |             |
|  |     box     h
|  |   angle=0   |
|  D------w------C
v
y (pi/2 rad)

```

`CW` 的旋转矩阵

```{math}
\begin{pmatrix}
\cos\alpha & -\sin\alpha \\
\sin\alpha & \cos\alpha
\end{pmatrix}
```

`CW` 的旋转变换

```{math}
P_A=
\begin{pmatrix} x_A \\ y_A\end{pmatrix}
=
\begin{pmatrix} x_{center} \\ y_{center}\end{pmatrix} +
\begin{pmatrix}\cos\alpha & -\sin\alpha \\
\sin\alpha & \cos\alpha\end{pmatrix}
\begin{pmatrix} -0.5w \\ -0.5h\end{pmatrix} \\
=
\begin{pmatrix} x_{center}-0.5w\cos\alpha+0.5h\sin\alpha
\\
y_{center}-0.5w\sin\alpha-0.5h\cos\alpha\end{pmatrix}
```

- 逆时针 (`CCW`)

`CCW` 的示意图

```
|  |             |
|  |     box     h
v
y (-pi/2 rad)

```

`CCW` 的旋转矩阵

```{math}

```
|  |             |
|  |     box     h
v

```

```{math}
\cos\alpha & \sin\alpha \\
-\sin\alpha & \cos\alpha
```

`CCW` 的旋转变换

```{math}
P_A=
=
\begin{pmatrix}\cos\alpha & \sin\alpha \\
-\sin\alpha & \cos\alpha\end{pmatrix}
=
\begin{pmatrix} x_{center}-0.5w\cos\alpha-0.5h\sin\alpha
\\
y_{center}+0.5w\sin\alpha-0.5h\cos\alpha\end{pmatrix}
```

在MMCV中可以设置旋转方向的算子有：

- box_iou_rotated (默认为 `CW`)
- nms_rotated (默认为 `CW`)
- RoIAlignRotated (默认为 `CCW`)
- RiRoIAlignRotated (默认为 `CCW`)。

```{note}
在MMRotate 中，旋转框的旋转方向均为 `CW`。
```

### 旋转框定义法

由于 `theta` 定义范围的不同，在旋转目标检测中逐渐派生出如下3种旋转框定义法：

```{note}
```

### 旋转框定义法

- {math}`D_{oc^{\prime}}` : OpenCV 定义法，`angle∈(0, 90°]`，`theta∈(0, pi / 2]`， `width` 与 `x` 正半轴之间的夹角为正的锐角。该定义法源于 OpenCV 中的 `cv2.minAreaRect` 函数，
  其返回值为 `(0, 90°]`。
- {math}`D_{le135}` : 长边 135° 定义法，`angle∈[-45°, 135°)`，`theta∈[-pi / 4, 3 * pi / 4)` 并且 `width > height`。
- {math}`D_{le90}` : 长边 90° 定义法，`angle∈[-90°, 90°)`，`theta∈[-pi / 2, pi / 2)` 并且 `width > height`。

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/angle_def.png" width=100%/>
</div>

三种定义法之间的转换关系在 MMRotate 内部并不涉及，因此不多做介绍。如果想了解更多的细节，可以参考这篇[博客](https://zhuanlan.zhihu.com/p/459018810)。

```{note}
MMRotate 同时支持上述三种旋转框定义法，可以通过配置文件灵活切换。
```

需要注意的是，在 4.5.1 之前的版本中，`cv2.minAreaRect` 的返回值为 `[-90°, 0°)`
（[参考资料](https://github.com/opencv/opencv/issues/19749)）。为了便于区分，
将老版本的 OpenCV 定义法记作 {math}`D_{oc}`。

```

- {math}`D_{oc^{\prime}}`: OpenCV 定义法，`opencv>=4.5.1`，`angle∈(0, 90°]`，`theta∈(0, pi / 2]`。
- {math}`D_{oc}`: 老版的 OpenCV 定义法，`opencv<4.5.1`，`angle∈[-90°, 0°)`，`theta∈[-pi / 2, 0)`。

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/opencv.png" width=50%/>
</div>

两种 OpenCV 定义法的转换关系如下：

```{math}
D_{oc^{\prime}}\left( w_{oc^{\prime}},h_{oc^{\prime}},\theta _{oc^{\prime}} \right) =\begin{cases}
	D_{oc}\left( h_{oc},w_{oc},\theta _{oc}+\pi /2 \right) , otherwise\\
	D_{oc}\left( w_{oc},h_{oc},\theta _{oc}+\pi \right) ,\theta _{oc}=-\pi /2\\
\end{cases}
\\
D_{oc}\left( w_{oc},h_{oc},\theta _{oc} \right) =\begin{cases}
	D_{oc^{\prime}}\left( h_{oc^{\prime}},w_{oc^{\prime}},\theta _{oc^{\prime}}-\pi /2 \right) , otherwise\\

\\
	D_{oc^{\prime}}\left( w_{oc^{\prime}},h_{oc^{\prime}},\theta _{oc^{\prime}}-\pi \right) , \theta _{oc^{\prime}}=\pi /2\\
```

```{note}
不管您使用的 OpenCV 版本是多少, MMRotate 都会将 OpenCV 定义法的 theta 转换为 (0, pi / 2]。
```

### 评估

评估 mAP 的代码中涉及 IoU 的计算，可以直接计算旋转框 IoU，也可以将旋转框转换为多边形，然后
计算多边形 IoU (DOTA 在线评估使用的是计算多边形 IoU)。

## 什么是 MMRotate

MMRotate 是一个为旋转目标检测方法提供统一训练和评估框架的工具箱，以下是其整体框架：:

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/mmrotate-arch.png" width=50%/>
</div>

MMRotate 包括四个部分, `datasets`, `models`, `core` and `apis`.

- `datasets` 用于数据加载和数据增强。 在这部分,我们支持了各种旋转目标检测数据集和数据增强预处理。

- `models` 包括模型和损失函数。

- `core` 为模型训练和评估提供工具。

- `apis` 为模型训练、测试和推理提供高级 API。

MMRotate 的模块设计如下图所示：

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/framework.png" width=100%/>
</div>

其中由于旋转框定义法不同而需要注意的地方有如下几个：

- 读取标注
- 数据增强
- 指派样本
- 评估指标

## 如何使用教程

下面是 MMRotate 详细的分步指南:

1. 关于安装说明, 请参阅 [安装](install.md).

2. [开始](get_started.md) 介绍了 MMRotate 的基本用法.

3. 如果想要更加深入了解 MMRotate，请参阅以下教程:

- [配置](tutorials/customize_config.md)
- [自定义数据集](tutorials/customize_dataset.md)
- [自定义模型](tutorials/customize_models.md)
- [自定义运行时](tutorials/customize_runtime.md)

# 模型微调

在 COCO 数据集上预训练的检测器可以作为其他数据集（例如 CityScapes 和 KITTI 数据集）优质的预训练模型。
本教程将指导用户如何把 [ModelZoo](../model_zoo.md) 中提供的模型用于其他数据集中并使得当前所训练的模型获得更好性能。

以下是在新数据集中微调模型需要的两个步骤。

- 按 [教程2：自定义数据集](../advanced_guides/customize_dataset.md) 中的方法对新数据集添加支持中的方法对新数据集添加支持
- 按照本教程中所讨论方法，修改配置信息

接下来将会以 Cityscapes Dataset 上的微调过程作为例子，具体讲述用户需要在配置中修改的五个部分。

## 继承基础配置

为了减轻编写整个配置的负担并减少漏洞的数量， MMDetection V3.0 支持从多个现有配置中继承配置信息。微调 MaskRCNN 模型的时候，新的配置信息需要使用从 `_base_/models/mask_rcnn_r50_fpn.py` 中继承的配置信息来构建模型的基本结构。当使用 Cityscapes 数据集时，新的配置信息可以简便地从`_base_/datasets/cityscapes_instance.py` 中继承。对于训练过程的运行设置部分，例如 `logger settings`，配置文件可以从 `_base_/default_runtime.py` 中继承。对于训练计划的配置则可以从`_base_/schedules/schedule_1x.py` 中继承。这些配置文件存放于 `configs` 目录下，用户可以选择全部内容的重新编写而不是使用继承方法。

```python
_base_ = [
    '../_base_/models/mask_rcnn_r50_fpn.py',
]
```

## Head 的修改

_base_ = [
]
```

## Head 的修改

接下来新的配置还需要根据新数据集的类别数量对 Head 进行修改。只需要对 roi_head 中的 `num_classes`进行修改。修改后除了最后的预测模型的 Head 之外，预训练模型的权重的大部分都会被重新使用。

```python

loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=1.0)),
```

## 数据集的修改

用户可能还需要准备数据集并编写有关数据集的配置，可在 [Customize Datasets](../advanced_guides/customize_dataset.md) 中获取更多信息。目前 MMDetection V3.0 的配置文件已经支持 VOC、WIDERFACE、COCO、LIVS、OpenImages、DeepFashion、Objects365 和 Cityscapes Dataset 的数据集信息。

## 训练策略的修改

微调超参数与默认的训练策略不同。它通常需要更小的学习率和更少的训练回合。

```python
# 优化器
# batch size 为 8 时的 lr 配置

# 学习率
    dict(
    dict(
        begin=0,
        end=8,
]

# 设置 max epoch

dict(
        begin=0,
        end=8,
]

# 设置 log config

```

## 使用预训练模型

如果要使用预训练模型，可以在 `load_from` 中查阅新的配置信息，用户需要在训练开始之前下载好需要的模型权重，从而避免在训练过程中浪费了宝贵时间。

```python
```

# FAQ

We list some common issues faced by many users and their corresponding solutions here.
Feel free to enrich the list if you find any frequent issues and have ways to help others to solve them.
If the contents here do not cover your issue, please create an issue using the [provided templates](/.github/ISSUE_TEMPLATE/error-report.md) and make sure you fill in all required information in the template.

Compatibility issue between MMCV and MMPose; "AssertionError: MMCV==xxx is used but incompatible. Please install mmcv>=xxx, \<=xxx."

Here are the version correspondences between `mmdet`, `mmcv` and `mmpose`:

- mmdet 2.x \<=> mmpose 0.x \<=> mmcv 1.x
- mmdet 3.x \<=> mmpose 1.x \<=> mmcv 2.x

Detailed compatible MMPose and MMCV versions are shown as below. Please choose the correct version of MMCV to avoid installation issues.

### MMPose 1.x

| MMPose version |      MMCV/MMEngine version      |
| :------------: | :-----------------------------: |

|     1.1.0      |  mmcv>=2.0.1, mmengine>=0.8.0   |
|     1.0.0      |  mmcv>=2.0.0, mmengine>=0.7.0   |
|    1.0.0rc1    | mmcv>=2.0.0rc4, mmengine>=0.6.0 |
|    1.0.0rc0    | mmcv>=2.0.0rc0, mmengine>=0.0.1 |
|    1.0.0b0     | mmcv>=2.0.0rc0, mmengine>=0.0.1 |

### MMPose 0.x

| MMPose version |       MMCV version        |
| :------------: | :-----------------------: |
|      0.x       | mmcv-full>=1.3.8, \<1.8.0 |
|     0.29.0     | mmcv-full>=1.3.8, \<1.7.0 |
|     0.28.1     | mmcv-full>=1.3.8, \<1.7.0 |
|     0.28.0     | mmcv-full>=1.3.8, \<1.6.0 |
|     0.27.0     | mmcv-full>=1.3.8, \<1.6.0 |
|     0.26.0     | mmcv-full>=1.3.8, \<1.6.0 |
|     0.25.1     | mmcv-full>=1.3.8, \<1.6.0 |
|     0.25.0     | mmcv-full>=1.3.8, \<1.5.0 |
|     0.24.0     | mmcv-full>=1.3.8, \<1.5.0 |
|     0.23.0     | mmcv-full>=1.3.8, \<1.5.0 |

|     0.22.0     | mmcv-full>=1.3.8, \<1.5.0 |
|     0.21.0     | mmcv-full>=1.3.8, \<1.5.0 |
|     0.20.0     | mmcv-full>=1.3.8, \<1.4.0 |
|     0.19.0     | mmcv-full>=1.3.8, \<1.4.0 |
|     0.18.0     | mmcv-full>=1.3.8, \<1.4.0 |
|     0.17.0     | mmcv-full>=1.3.8, \<1.4.0 |
|     0.16.0     | mmcv-full>=1.3.8, \<1.4.0 |
|     0.14.0     | mmcv-full>=1.1.3, \<1.4.0 |
|     0.13.0     | mmcv-full>=1.1.3, \<1.4.0 |
|     0.12.0     |  mmcv-full>=1.1.3, \<1.3  |
|     0.11.0     |  mmcv-full>=1.1.3, \<1.3  |
|     0.10.0     |  mmcv-full>=1.1.3, \<1.3  |
|     0.9.0      |  mmcv-full>=1.1.3, \<1.3  |
|     0.8.0      |  mmcv-full>=1.1.1, \<1.2  |
|     0.7.0      |  mmcv-full>=1.1.1, \<1.2  |

- **Unable to install xtcocotools**

  1. Try to install it using pypi manually `pip install xtcocotools`.

  2. If step1 does not work. Try to install it from [source](https://github.com/jin-s13/xtcocoapi).

  ```
  git clone https://github.com/jin-s13/xtcocoapi
  cd xtcocoapi
  python setup.py install
  ```

- **No matching distribution found for xtcocotools>=1.6**

  1. Install cython by `pip install cython`.
  2. Install xtcocotools from [source](https://github.com/jin-s13/xtcocoapi).

  ```
  ```

  1. Uninstall existing mmcv in the environment using `pip uninstall mmcv`.
  2. Install mmcv following [mmcv installation instruction](https://mmcv.readthedocs.io/en/2.x/get_started/installation.html).

## Data

- **What if my custom dataset does not have bounding box label?**

## Data

  We can estimate the bounding box of a person as the minimal box that tightly bounds all the keypoints.

- **What is `COCO_val2017_detections_AP_H_56_person.json`? Can I train pose models without it?**

  "COCO_val2017_detections_AP_H_56_person.json" contains the "detected" human bounding boxes for COCO validation set, which are generated by FasterRCNN.
  One can choose to use gt bounding boxes to evaluate models, by setting `bbox_file=None` in `val_dataloader.dataset` in config. Or one can use detected boxes to evaluate
  the generalizability of models, by setting `bbox_file='COCO_val2017_detections_AP_H_56_person.json'`.

- **RuntimeError: Address already in use**

  Set the environment variables `MASTER_PORT=XXX`. For example:

  ```shell

  ```shell
  MASTER_PORT=29517 GPUS=16 GPUS_PER_NODE=8 CPUS_PER_TASK=2 ./tools/slurm_train.sh train res50 configs/body_2d_keypoint/topdown_regression/coco/td-reg_res50_8xb64-210e_coco-256x192.py work_dirs/res50_coco_256x192
  ```

- **"Unexpected keys in source state dict" when loading pre-trained weights**

  It's normal that some layers in the pretrained model are not used in the pose model. ImageNet-pretrained classification network and the pose network may have different architectures (e.g. no classification head). So some unexpected keys in source state dict is actually expected.

- **How to use trained models for backbone pre-training ?**

  Refer to [Migration - Step3: Model - Backbone](../migration.md).

  When training, the unexpected keys will be ignored.

- **How to visualize the training accuracy/loss curves in real-time ?**

Refer to [Migration - Step3: Model - Backbone](../migration.md).

  Modify `vis_backends` in config file like:

  ```python
  vis_backends = [
    dict(type='TensorboardVisBackend')
  ]
  ```

  You can refer to [user_guides/visualization.md](../user_guides/visualization.md).

- **Log info is NOT printed**

  Use smaller log interval. For example, change `interval=50` to `interval=1` in the config:

  ```python
  # hooks
  default_hooks = dict(logger=dict(interval=1))
  ```

- **How to evaluate on MPII test dataset?**
  Since we do not have the ground-truth for test dataset, we cannot evaluate it 'locally'.

```

  If you would like to evaluate the performance on test set, you have to upload the pred.mat (which is generated during testing) to the official server via email, according to [the MPII guideline](http://human-pose.mpi-inf.mpg.de/#evaluation).

- **For top-down 2d pose estimation, why predicted joint coordinates can be out of the bounding box (bbox)?**
  We do not directly use the bbox to crop the image. bbox will be first transformed to center & scale, and the scale will be multiplied by a factor (1.25) to include some context. If the ratio of width/height is different from that of model input (possibly 192/256), we will adjust the bbox.

- **How to run mmpose on CPU?**

  Run demos with `--device=cpu`.

- **How to speed up inference?**

  A few approaches may help to improve the inference speed:

  1. Set `flip_test=False` in `init_cfg` in the config file.
  2. For top-down models, use faster human bounding box detector, see [MMDetection](https://mmdetection.readthedocs.io/en/3.x/model_zoo.html).

# FAQ

## Outline

- [FAQ](#faq)
  - [Outline](#outline)
  - [Data](#data)
  - [Testing](#testing)

If the contents here do not cover your issue, please create an issue using the [provided templates](https://github.com/open-mmlab/mmaction2/tree/main/.github/ISSUE_TEMPLATE/error-report.md) and make sure to fill in all required information in the template.

  1. Uninstall existing mmcv in the environment using `pip uninstall mmcv`
  2. Install mmcv following the [installation instruction](https://mmcv.readthedocs.io/en/2.x/get_started/installation.html#install-mmcv)

  Refer to [install.md](https://github.com/open-mmlab/mmaction2/blob/master/docs/install.md#requirements)

  1. For Windows users, [ImageMagick](https://www.imagemagick.org/script/index.php) will not be automatically detected by MoviePy, there is a need to modify `moviepy/config_defaults.py` file by providing the path to the ImageMagick binary called `magick`, like `IMAGEMAGICK_BINARY = "C:\\Program Files\\ImageMagick_VERSION\\magick.exe"`
  2. For Linux users, there is a need to modify the `/etc/ImageMagick-6/policy.xml` file by commenting out `<policy domain="path" rights="none" pattern="@*" />` to `<!-- <policy domain="path" rights="none" pattern="@*" /> -->`, if ImageMagick is not detected by moviepy.

- **"Why I got the error message 'Please install XXCODEBASE to use XXX' even if I have already installed XXCODEBASE?"**

  You got that error message because our project failed to import a function or a class from XXCODEBASE. You can try to run the corresponding line to see what happens. One possible reason is, for some codebases in OpenMMLAB, you need to install mmcv and mmengine before you install them. You could follow this [tutorial](https://mmaction2.readthedocs.io/en/latest/get_started/installation.html#installation) to install them.

## Data

- **FileNotFound like `No such file or directory: xxx/xxx/img_00300.jpg`**

  In our repo, we set `start_index=1` as default value for rawframe dataset, and `start_index=0` as default value for video dataset.
  If users encounter FileNotFound error for the first or last frame of the data, there is a need to check the files begin with offset 0 or 1,

If users encounter FileNotFound error for the first or last frame of the data, there is a need to check the files begin with offset 0 or 1,
  that is `xxx_00000.jpg` or `xxx_00001.jpg`, and then change the `start_index` value of data pipeline in configs.

- **How should we preprocess the videos in the dataset? Resizing them to a fix size(all videos with the same height-width ratio) like `340x256` (1) or resizing them so that the short edges of all videos are of the same length (256px or 320px) (2)**

  We have tried both preprocessing approaches and found (2) is a better solution in general, so we use (2) with short edge length 256px as the default preprocessing setting. We benchmarked these preprocessing approaches and you may find the results in [TSN Data Benchmark](https://github.com/open-mmlab/mmaction2/tree/master/configs/recognition/tsn) and [SlowOnly Data Benchmark](https://github.com/open-mmlab/mmaction2/tree/master/configs/recognition/slowonly).

- **Mismatched data pipeline items lead to errors like `KeyError: 'total_frames'`**

  We have both pipeline for processing videos and frames.

  **For videos**, We should decode them on the fly in the pipeline, so pairs like `DecordInit & DecordDecode`, `OpenCVInit & OpenCVDecode`, `PyAVInit & PyAVDecode` should be used for this case like [this example](https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py#L14-L16).

  **For Frames**, the image has been decoded offline, so pipeline item `RawFrameDecode` should be used for this case like [this example](https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/trn/trn_imagenet-pretrained-r50_8xb16-1x1x8-50e_sthv1-rgb.py#L17).

  `KeyError: 'total_frames'` is caused by incorrectly using `RawFrameDecode` step for videos, since when the input is a video, it can not get the `total_frames` beforehand.

`KeyError: 'total_frames'` is caused by incorrectly using `RawFrameDecode` step for videos, since when the input is a video, it can not get the `total_frames` beforehand.

- **How to just use trained recognizer models for backbone pre-training?**

  In order to use the pre-trained model for the whole network, the new config adds the link of pre-trained models in the `load_from`.

  And to use backbone for pre-training, you can change `pretrained` value in the backbone dict of config files to the checkpoint path / url.

- **How to fix stages of backbone when finetuning a model?**

  You can refer to [`def _freeze_stages()`](https://github.com/open-mmlab/mmaction2/blob/main/mmaction/models/backbones/resnet3d.py#L791) and [`frozen_stages`](https://github.com/open-mmlab/mmaction2/blob/main/mmaction/models/backbones/resnet3d.py#L369-L370).

Reminding to set `find_unused_parameters = True` in config files for distributed training or testing.

  Actually, users can set `frozen_stages` to freeze stages in backbones except C3D model, since almost all backbones inheriting from `ResNet` and `ResNet3D` support the inner function `_freeze_stages()`.

- **How to set memcached setting in config files?**

  In MMAction2, you can pass memcached kwargs to `class DecordInit` for video dataset or `RawFrameDecode` for rawframes dataset.
  For more details, you can refer to \[`class FileClient`\] in MMEngine for more details.
  Here is an example to use memcached for rawframes dataset:

  ```python

    ...
    ...
  ]
  ```

- **How to set `load_from` value in config files to finetune models?**

    ...
    ...
  ]
  ```

  In MMAction2, We set `load_from=None` as default in `configs/_base_/default_runtime.py` and owing to [inheritance design](https://github.com/open-mmlab/mmaction2/tree/main/docs/en/user_guides/config.md),
  users can directly change it by setting `load_from` in their configs.

- **How to use `RawFrameDataset` for training?**

  In MMAction2 1.x version, most of the configs take `VideoDataset` as the default dataset type, which is much more friendly to file storage. If you want to use `RawFrameDataset` instead, there are two steps to modify:

  - Dataset:
    modify dataset in `train_dataloader`/`val_dataloader`/`test_dataloader` from

    ```
        ...)
    ```

    to

    ```

```
        ...)
    ```

    to

    ```
        ...)
    ```

    remaining fields of `dataset` don't need to be modified. Please make sure that `filename_tmpl` is matching with your frame data, and you can refer to [config document](../user_guides/config.md) for more details about config file.

  - Transforms: delete `dict(type='DecordInit', **file_client_args)`, modify `dict(type='DecordDecode')` to `dict(type='RawFrameDecode', **file_client_args)` in `train_pipeline`/`val_pipeline`/`test_pipeline`, and please make sure that `file_client_args = dict(io_backend='disk')` has been defined in your config.

For more modifications about customizing datasets, please refer to [prepare dataset](../user_guides/prepare_dataset.md) and [customize dataset](../advanced_guides/customize_dataset.md).

## Testing

- **How to make predicted score normalized by softmax within \[0, 1\]?**

  change this in the config, make `model.cls_head.average_clips = 'prob'`.

- **What if the model is too large and the GPU memory can not fit even only one testing sample?**

  By default, the 3d models are tested with 10clips x 3crops, which are 30 views in total. For extremely large models, the GPU memory can not fit even only one testing sample (cuz there are 30 views). To handle this, you can set `max_testing_views=n` in `model['test_cfg']` of the config file. If so, n views will be used as a batch during forwarding to save GPU memory used.

# PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds

> [PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds](https://arxiv.org/abs/2103.14635)

We introduce Position Adaptive Convolution (PAConv), a generic convolution operation for 3D point cloud processing. The key of PAConv is to construct the convolution kernel by dynamically assembling basic weight matrices stored in Weight Bank, where the coefficients of these weight matrices are self-adaptively learned from point positions through ScoreNet. In this way, the kernel is built in a data-driven manner, endowing PAConv with more flexibility than 2D convolutions to better handle the irregular and unordered point cloud data. Besides, the complexity of the learning process is reduced by combining weight matrices instead of brutally predicting kernels from point positions.

Furthermore, different from the existing point convolution operators whose network architectures are often heavily engineered, we integrate our PAConv into classical MLP-based point cloud pipelines without changing network configurations. Even built on simple networks, our method still approaches or even surpasses the state-of-the-art models, and significantly improves baseline performance on both classification and segmentation tasks, yet with decent efficiency. Thorough ablation studies and visualizations are provided to understand PAConv.

<img src="https://user-images.githubusercontent.com/79644370/143881915-003d5f10-3999-474e-969a-c354cb738a11.png" width="800"/>
</div>

We implement PAConv and provide the result and checkpoints on S3DIS dataset.

**Notice**: The original PAConv paper used step learning rate schedule. We discovered that cosine schedule achieves slightly better results and adopt it in our implementations.

### S3DIS

|                              Method                               | Split  |   Lr schd   | Mem (GB) | Inf time (fps) | mIoU (Val set) |                                                                                                                                                                                                             Download                                                                                                                                                                                                             |

|    [PAConv (SSG)](./paconv_ssg_8xb8-cosine-150e_s3dis-seg.py)     | Area_5 | cosine 150e |   5.8    |                |     66.65      |           [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/paconv/paconv_ssg_8x8_cosine_150e_s3dis_seg-3d-13class/paconv_ssg_8x8_cosine_150e_s3dis_seg-3d-13class_20210729_200615-2147b2d1.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/paconv/paconv_ssg_8x8_cosine_150e_s3dis_seg-3d-13class/paconv_ssg_8x8_cosine_150e_s3dis_seg-3d-13class_20210729_200615.log.json)           |

| [PAConv\* (SSG)](./paconv_ssg-cuda_8xb8-cosine-200e_s3dis-seg.py) | Area_5 | cosine 200e |   3.8    |                |     65.33      | [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/paconv/paconv_cuda_ssg_8x8_cosine_200e_s3dis_seg-3d-13class/paconv_cuda_ssg_8x8_cosine_200e_s3dis_seg-3d-13class_20210802_171802-e5ea9bb9.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/paconv/paconv_cuda_ssg_8x8_cosine_200e_s3dis_seg-3d-13class/paconv_cuda_ssg_8x8_cosine_200e_s3dis_seg-3d-13class_20210802_171802.log.json) |

- PAConv\* stands for the CUDA implementation of PAConv operations. See the [paper](https://arxiv.org/pdf/2103.14635.pdf) appendix section D for more details. In our experiments, the training of PAConv\* is found to be very unstable. We achieved slightly lower mIoU than the result in the paper, but is consistent with the result obtained by running their [official code](https://github.com/CVMI-Lab/PAConv/tree/main/scene_seg). Besides, although the GPU memory consumption of PAConv\* is significantly lower than PAConv, its training and inference speed are actually slower (by ~10%).

Since PAConv testing adopts sliding patch inference which involves random point sampling, and the test script uses fixed random seeds while the random seeds of validation in training are not fixed, the test results may be slightly different from the results reported above.

```latex
@inproceedings{xu2021paconv,
  title={PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds},
  author={Xu, Mutian and Ding, Runyu and Zhao, Hengshuang and Qi, Xiaojuan},
  pages={3173--3182},
}
```

# 准备数据集

## CustomDataset

[`CustomDataset`](mmpretrain.datasets.CustomDataset) 是一个通用的数据集类，供您使用自己的数据集。目前 `CustomDataset` 支持以下两种方式组织你的数据集文件：

### 子文件夹方式

在这种格式下，您只需要重新组织您的数据集文件夹并将所有样本放在一个文件夹中，而无需创建任何标注文件。

对于监督任务（使用 `with_label=true`），我们使用子文件夹的名称作为类别名称，如下例所示，`class_x` 和 `class_y` 将被识别为类别名称。

```text
data_prefix/
├── class_x
│   ├── xxx.png
│   ├── xxy.png
│   └── ...
│       └── xxz.png
└── class_y
    ├── 123.png
    ├── nsdf3.png
    ├── ...
    └── asd932_.png
```

对于无监督任务（使用 `with_label=false`），我们直接加载指定文件夹下的所有样本文件：

```
├── folder_1
│   └── ...
├── 123.png
├── nsdf3.png
└── ...
```

假如你希望将之用于训练，那么配置文件中需要添加以下配置：

```python
    ...
    # 训练数据集配置
        data_prefix='path/to/data_prefix',
        with_label=True,  # 对于无监督任务，使用 False
        pipeline=...
    )
)
```

```{note}
如果要使用此格式，请不要指定 `ann_file`，或指定 `ann_file=''`。

data_prefix='path/to/data_prefix',
    )
)
```

```{note}

请注意，子文件夹格式需要对文件夹进行扫描，这可能会导致初始化速度变慢，尤其是对于大型数据集或慢速文件 IO。
```

### 标注文件方式

标注文件格式主要使用文本文件来保存类别信息，`data_prefix` 存放图片，`ann_file` 存放标注类别信息。

如下案例，dataset 目录如下：

在这种格式中，我们使用文本标注文件来存储图像文件路径和对应的类别索引。

对于监督任务（`with_label=true`），注释文件应在一行中包含一个样本的文件路径和类别索引，并用空格分隔，如下所示：

所有这些文件路径都可以是绝对路径，也可以是相对于 `data_prefix` 的相对路径。

```text
folder_1/xxx.png 0
folder_1/xxy.png 1
123.png 4
nsdf3.png 3
...
```

```{note}
类别的索引号从 0 开始。真实标签的值应在`[0, num_classes - 1]`范围内。

此外，请使用数据集设置中的 `classes` 字段来指定每个类别的名称
```

对于无监督任务（`with_label=false`），标注文件只需要在一行中包含一个样本的文件路径，如下：

```text
folder_1/xxx.png
folder_1/xxy.png
123.png
nsdf3.png
...
```

假设整个数据集文件夹如下：

```text
data_root
├── meta
│   ├── test.txt     # 测试数据集的标注文件
│   ├── train.txt    # 训练数据集的标注文件
│   └── val.txt      # 验证数据集的标注文件

├── train
│   ├── 123.png
│   ├── folder_1
│   │   ├── xxx.png

```text
data_root
├── meta

├── train
│   │   └── xxy.png
│   └── nsdf3.png
├── test
└── val
```

这是配置文件中的数据集设置的示例：

```python
# 训练数据设置
        data_root='path/to/data_root',  # `ann_flie` 和 `data_prefix` 共同的文件路径前缀
        ann_file='meta/train.txt',      # 相对于 `data_root` 的标注文件路径
        data_prefix='train',            # `ann_file` 中文件路径的前缀，相对于 `data_root`
        classes=['A', 'B', 'C', 'D', ...],  # 每个类别的名称
        pipeline=...,    # 处理数据集样本的一系列变换操作
    )
    ...
)
```

```{note}
有关如何使用 `CustomDataset` 的完整示例，请参阅[如何使用自定义数据集进行预训练](../notes/pretrain_custom_dataset.md)
```

## ImageNet

ImageNet 有多个版本，但最常用的一个是 [ILSVRC 2012](http://www.image-net.org/challenges/LSVRC/2012/)。 可以通过以下步骤使用它。

````{group-tab} MIM 下载

```

MIM支持使用一条命令行从 [OpenDataLab](https://opendatalab.com/) 下载并预处理 ImageNet 数据集。

_需要在 [OpenDataLab 官网](https://opendatalab.com/) 注册账号并命令行登录_。

```Bash
# 安装opendatalab库
# 登录到 OpenDataLab, 如果还没有注册，请到官网注册一个
odl login
# 使用 MIM 下载数据集, 最好在 $MMPreTrain 目录执行
mim download mmpretrain --dataset imagenet1k
```

````

````{group-tab} 从官网下载

1. 注册一个帐户并登录到[下载页面](http://www.image-net.org/download-images)。
2. 找到 ILSVRC2012 的下载链接，下载以下两个文件：
   - ILSVRC2012_img_train.tar (~138GB)
   - ILSVRC2012_img_val.tar (~6.3GB)
3. 解压已下载的图片。

````
`````

### ImageNet数据集目录结构

我们支持两种方式组织ImageNet数据集，子目录格式和文本注释文件格式。

#### 子文件夹格式

我们提供了一个样例，您可以从这个[链接](https://download.openmmlab.com/mmpretrain/datasets/imagenet_1k.zip)下载和解压。数据集的目录结构应如下所示：

```text
data/imagenet/
├── train/
│   ├── n01440764
│   │   ├── n01440764_10026.JPEG

```text
├── train/
│   │   ├── n01440764_10027.JPEG
│   │   ├── n01440764_10029.JPEG
│   │   ├── n01440764_10040.JPEG
│   │   ├── n01440764_10042.JPEG
│   │   ├── n01440764_10043.JPEG
│   │   └── n01440764_10048.JPEG
│   ├── ...
├── val/
│   │   ├── ILSVRC2012_val_00000293.JPEG
│   │   ├── ILSVRC2012_val_00002138.JPEG
│   │   ├── ILSVRC2012_val_00003014.JPEG
│   │   └── ...
│   ├── ...
```

#### 文本标注文件格式

您可以从[此链接](https://download.openmmlab.com/mmclassification/datasets/imagenet/meta/caffe_ilsvrc12.tar.gz)下载并解压元数据，然后组织文件夹如下：

```text
├── meta/
│   └── val.txt
├── train/

├── train/
│   ├── ...
├── val/
│   ├── ILSVRC2012_val_00000001.JPEG
│   ├── ILSVRC2012_val_00000002.JPEG
│   ├── ILSVRC2012_val_00000003.JPEG
│   ├── ILSVRC2012_val_00000004.JPEG
│   ├── ...
```

### 配置

当您的数据集以上述方式组织时，您可以使用具有以下配置的 [`ImageNet`](mmpretrain.datasets.ImageNet) 数据集：

```python
    ...
    # 训练数据集配置
        type='ImageNet',
        data_root='data/imagenet/',
        pipeline=...,
    )
)

    ...
    # 验证数据集配置
        split='val',
    )
)

```

## 支持的图像分类数据集

# 验证数据集配置
    )
)

```

| 数据集                                                                              | split                               | 主页                                                                               |
| ----------------------------------------------------------------------------------- | ----------------------------------- | ---------------------------------------------------------------------------------- |
| [`Calthch101`](mmpretrain.datasets.Caltech101)(data_root[, split, pipeline, ...])   | ["train", "test"]                   | [Caltech 101](https://data.caltech.edu/records/mzrjq-6wc02) 数据集                 |

| [`CIFAR10`](mmpretrain.datasets.CIFAR10)(data_root[, split, pipeline, ...])         | ["train", "test"]                   | [CIFAR10](https://www.cs.toronto.edu/~kriz/cifar.html) 数据集                      |
| [`CIFAR100`](mmpretrain.datasets.CIFAR100)(data_root[, split, pipeline, ...])       | ["train", "test"]                   | [CIFAR100](https://www.cs.toronto.edu/~kriz/cifar.html) 数据集                     |
| [`CUB`](mmpretrain.datasets.CUB)(data_root[, split, pipeline, ...])                 | ["train", "test"]                   | [CUB-200-2011](http://www.vision.caltech.edu/datasets/cub_200_2011/) 数据集        |
| [`DTD`](mmpretrain.datasets.DTD)(data_root[, split, pipeline, ...])                 | ["train", "val", "tranval", "test"] | [Describable Texture Dataset (DTD)](https://www.robots.ox.ac.uk/~vgg/data/dtd/) 数据集 |

| [`FashionMNIST`](mmpretrain.datasets.FashionMNIST) (data_root[, split, pipeline, ...]) | ["train", "test"]                   | [Fashion-MNIST](https://github.com/zalandoresearch/fashion-mnist) 数据集           |
| [`FGVCAircraft`](mmpretrain.datasets.FGVCAircraft)(data_root[, split, pipeline, ...]) | ["train", "val", "tranval", "test"] | [FGVC Aircraft](https://www.robots.ox.ac.uk/~vgg/data/fgvc-aircraft/) 数据集       |
| [`Flowers102`](mmpretrain.datasets.Flowers102)(data_root[, split, pipeline, ...])   | ["train", "val", "tranval", "test"] | [Oxford 102 Flower](https://www.robots.ox.ac.uk/~vgg/data/flowers/102/) 数据集     |
| [`Food101`](mmpretrain.datasets.Food101)(data_root[, split, pipeline, ...])         | ["train", "test"]                   | [Food101](https://data.vision.ee.ethz.ch/cvl/datasets_extra/food-101/) 数据集      |

| [`MNIST`](mmpretrain.datasets.MNIST) (data_root[, split, pipeline, ...])            | ["train", "test"]                   | [MNIST](http://yann.lecun.com/exdb/mnist/) 数据集                                  |
| [`OxfordIIITPet`](mmpretrain.datasets.OxfordIIITPet)(data_root[, split, pipeline, ...]) | ["tranval", test"]                  | [Oxford-IIIT Pets](https://www.robots.ox.ac.uk/~vgg/data/pets/) 数据集             |
| [`Places205`](mmpretrain.datasets.Places205)(data_root[, pipeline, ...])            | -                                   | [Places205](http://places.csail.mit.edu/downloadData.html) 数据集                  |
| [`StanfordCars`](mmpretrain.datasets.StanfordCars)(data_root[, split, pipeline, ...]) | ["train", "test"]                   | [StanfordCars](https://ai.stanford.edu/~jkrause/cars/car_dataset.html) 数据集      |

| [`SUN397`](mmpretrain.datasets.SUN397)(data_root[, split, pipeline, ...])           | ["train", "test"]                   | [SUN397](https://vision.princeton.edu/projects/2010/SUN/) 数据集                   |
| [`VOC`](mmpretrain.datasets.VOC)(data_root[, image_set_path, pipeline, ...])        | ["train", "val", "tranval", "test"] | [Pascal VOC](http://host.robots.ox.ac.uk/pascal/VOC/) 数据集                       |

有些数据集主页链接可能已经失效，您可以通过[OpenDataLab](https://opendatalab.com/)下载数据集，例如 [Stanford Cars](https://opendatalab.com/Stanford_Cars/download)数据集。

## OpenMMLab 2.0 标准数据集

为了统一不同任务的数据集接口，便于多任务的算法模型训练，OpenMMLab 制定了 **OpenMMLab 2.0 数据集格式规范**， 数据集标注文件需符合该规范，数据集基类基于该规范去读取与解析数据标注文件。如果用户提供的数据标注文件不符合规定格式，用户可以选择将其转化为规定格式，并使用 OpenMMLab 的算法库基于该数据标注文件进行算法训练和测试。

OpenMMLab 2.0 数据集格式规范规定，标注文件必须为 `json` 或 `yaml`，`yml` 或 `pickle`，`pkl` 格式；标注文件中存储的字典必须包含 `metainfo` 和 `data_list` 两个字段。其中 `metainfo` 是一个字典，里面包含数据集的元信息；`data_list` 是一个列表，列表中每个元素是一个字典，该字典定义了一个原始数据（raw data），每个原始数据包含一个或若干个训练/测试样本。

假设您要使用训练数据集，那么配置文件如下所示：

```

{
    'metainfo':
        {
            'classes': ('cat', 'dog'), # 'cat' 的类别序号为 0，'dog' 为 1。
            ...
        },
    'data_list':
        [
            {
                'img_path': "xxx/xxx_0.jpg",
                'gt_label': 0,
                ...
            },
            {
                'img_path': "xxx/xxx_1.jpg",
                'gt_label': 1,
                ...
            },
            ...
        ]
}
```

同时假设数据集存放路径如下：

```text
data
│   ├── train.json
│   └── ...
├── train
│   ├── xxx/xxx_0.jpg
│   ├── xxx/xxx_1.jpg
│   ├── ...
```

通过以下字典构建：

```python
dataset_cfg=dict(
    type='CustomDataset',
    ann_file='path/to/ann_file_path',
    data_prefix='path/to/images_folder',
    pipeline=transfrom_list)
```

## 其他数据集

MMPretrain 还支持更多其他的数据集，可以通过查阅[数据集文档](mmpretrain.datasets)获取它们的配置信息。

如果需要使用一些特殊格式的数据集，您需要实现您自己的数据集类，请参阅[添加新数据集](../advanced_guides/datasets.md)。

## 数据集包装

pipeline=transfrom_list)
```

## 其他数据集

## 数据集包装

MMEngine 中支持以下数据包装器，您可以参考 {external+mmengine:doc}`MMEngine 教程 <advanced_tutorials/basedataset>` 了解如何使用它。

- {external:py:class}`~mmengine.dataset.ConcatDataset`
- {external:py:class}`~mmengine.dataset.RepeatDataset`
- {external:py:class}`~mmengine.dataset.ClassBalancedDataset`

除上述之外，MMPretrain 还支持了[KFoldDataset](mmpretrain.datasets.KFoldDataset)，需用通过使用 `tools/kfold-cross-valid.py` 来使用它。

# UniFormerV2

[UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer](https://arxiv.org/abs/2211.09552)

Learning discriminative spatiotemporal representation is the key problem of video understanding. Recently, Vision Transformers (ViTs) have shown their power in learning long-term video dependency with self-attention. Unfortunately, they exhibit limitations in tackling local video redundancy, due to the blind global comparison among tokens. UniFormer has successfully alleviated this issue, by unifying convolution and self-attention as a relation aggregator in the transformer format. However, this model has to require a tiresome and complicated image-pretraining phrase, before being finetuned on videos. This blocks its wide usage in practice. On the contrary, open-sourced ViTs are readily available and well-pretrained with rich image supervision. Based on these observations, we propose a generic paradigm to build a powerful family of video networks, by arming the pretrained ViTs with efficient UniFormer designs. We call this family UniFormerV2, since it inherits the concise style of the UniFormer block. But it contains brand-new local and global relation aggregators, which allow for preferable accuracy-computation balance by seamlessly integrating advantages from both ViTs and UniFormer. Without any bells and whistles, our UniFormerV2 gets the state-of-the-art recognition performance on 8 popular video benchmarks, including scene-related Kinetics-400/600/700 and Moments in Time, temporal-related Something-Something V1/V2, untrimmed ActivityNet and HACS. In particular, it is the first model to achieve 90% top-1 accuracy on Kinetics-400, to our best knowledge.

<img src="https://raw.githubusercontent.com/OpenGVLab/UniFormerV2/main/img/framework.png"/>
</div>

| uniform sampling |   resolution   |        backbone        |     pretrain     | top1 acc | top5 acc | [reference](https://github.com/OpenGVLab/UniFormerV2/blob/main/MODEL_ZOO.md) top1 acc | [reference](https://github.com/OpenGVLab/UniFormerV2/blob/main/MODEL_ZOO.md) top5 acc | mm-Kinetics top1 acc | mm-Kinetics top5 acc | testing protocol | FLOPs | params |                                                         config                                                          |                                                                                                                          ckpt                                                                                                                          |                                                                                                                 log                                                                                                                 |

|        8         | short-side 320 |    UniFormerV2-B/16    |       clip       |    -     |    -     |                                         84.3                                          |                                         96.4                                          |         84.4         |         96.3         | 4 clips x 3 crop | 0.1T  |  115M  |         [config](/configs/recognition/uniformerv2/uniformerv2-base-p16-res224_clip_8xb32-u8_kinetics400-rgb.py)         |                 [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/uniformerv2-base-p16-res224_clip_8xb32-u8_kinetics400-rgb/uniformerv2-base-p16-res224_clip_8xb32-u8_kinetics400-rgb_20230313-e29fc968.pth)                 |                 [log](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/uniformerv2-base-p16-res224_clip_8xb32-u8_kinetics400-rgb/uniformerv2-base-p16-res224_clip_8xb32-u8_kinetics400-rgb.log)                 |

|        8         | short-side 320 |    UniFormerV2-B/16    | clip-kinetics710 |    -     |    -     |                                         85.6                                          |                                         97.0                                          |         85.8         |         97.1         | 4 clips x 3 crop | 0.1T  |  115M  | [config](/configs/recognition/uniformerv2/uniformerv2-base-p16-res224_clip-kinetics710-pre_8xb32-u8_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/uniformerv2-base-p16-res224_clip-kinetics710-pre_8xb32-u8_kinetics400-rgb/uniformerv2-base-p16-res224_clip-kinetics710-pre_8xb32-u8_kinetics400-rgb_20230313-75be0806.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/uniformerv2-base-p16-res224_clip-kinetics710-pre_8xb32-u8_kinetics400-rgb/uniformerv2-base-p16-res224_clip-kinetics710-pre_8xb32-u8_kinetics400-rgb.log) |

|        8         | short-side 320 |   UniFormerV2-L/14\*   | clip-kinetics710 |   88.7   |   98.1   |                                         88.8                                          |                                         98.1                                          |         88.7         |         98.1         | 4 clips x 3 crop | 0.7T  |  354M  |   [config](/configs/recognition/uniformerv2/uniformerv2-large-p14-res224_clip-kinetics710-pre_u8_kinetics400-rgb.py)    |                                  [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/kinetics400/uniformerv2-large-p14-res224_clip-kinetics710-pre_u8_kinetics400-rgb_20221219-972ea063.pth)                                   |                                                                                                                  -                                                                                                                  |

|        16        | short-side 320 |   UniFormerV2-L/14\*   | clip-kinetics710 |   89.0   |   98.2   |                                         89.1                                          |                                         98.2                                          |         89.0         |         98.2         | 4 clips x 3 crop | 1.3T  |  354M  |   [config](/configs/recognition/uniformerv2/uniformerv2-large-p14-res224_clip-kinetics710-pre_u16_kinetics400-rgb.py)   |                                  [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/kinetics400/uniformerv2-large-p14-res224_clip-kinetics710-pre_u16_kinetics400-rgb_20221219-6dc86d05.pth)                                  |                                                                                                                  -                                                                                                                  |

|        32        | short-side 320 |   UniFormerV2-L/14\*   | clip-kinetics710 |   89.3   |   98.2   |                                         89.3                                          |                                         98.2                                          |         89.4         |         98.2         | 2 clips x 3 crop | 2.7T  |  354M  |   [config](/configs/recognition/uniformerv2/uniformerv2-large-p14-res224_clip-kinetics710-pre_u32_kinetics400-rgb.py)   |                                  [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/kinetics400/uniformerv2-large-p14-res224_clip-kinetics710-pre_u32_kinetics400-rgb_20221219-56a46f64.pth)                                  |                                                                                                                  -                                                                                                                  |

|        32        | short-side 320 | UniFormerV2-L/14@336\* | clip-kinetics710 |   89.5   |   98.4   |                                         89.7                                          |                                         98.3                                          |         89.5         |         98.4         | 2 clips x 3 crop | 6.3T  |  354M  |   [config](/configs/recognition/uniformerv2/uniformerv2-large-p14-res336_clip-kinetics710-pre_u32_kinetics400-rgb.py)   |                                  [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/kinetics400/uniformerv2-large-p14-res336_clip-kinetics710-pre_u32_kinetics400-rgb_20221219-1dd7650f.pth)                                  |                                                                                                                  -                                                                                                                  |

### Kinetics-600

| uniform sampling | resolution |        backbone        |     pretrain     | top1 acc | top5 acc | [reference](https://github.com/OpenGVLab/UniFormerV2/blob/main/MODEL_ZOO.md) top1 acc | [reference](https://github.com/OpenGVLab/UniFormerV2/blob/main/MODEL_ZOO.md) top5 acc | mm-Kinetics top1 acc | mm-Kinetics top5 acc | testing protocol | FLOPs | params |                                                         config                                                          |                                                                                                                          ckpt                                                                                                                          |                                                                                                                 log                                                                                                                 |

|        8         |    Raw     |    UniFormerV2-B/16    | clip-kinetics710 |    -     |    -     |                                         86.1                                          |                                         97.2                                          |         86.4         |         97.3         | 4 clips x 3 crop | 0.1T  |  115M  | [config](/configs/recognition/uniformerv2/uniformerv2-base-p16-res224_clip-kinetics710-pre_8xb32-u8_kinetics600-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/uniformerv2-base-p16-res224_clip-kinetics710-pre_8xb32-u8_kinetics600-rgb/uniformerv2-base-p16-res224_clip-kinetics710-pre_8xb32-u8_kinetics600-rgb_20230313-544f06f0.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/uniformerv2-base-p16-res224_clip-kinetics710-pre_8xb32-u8_kinetics600-rgb/uniformerv2-base-p16-res224_clip-kinetics710-pre_8xb32-u8_kinetics600-rgb.log) |

|        8         |    Raw     |   UniFormerV2-L/14\*   | clip-kinetics710 |   89.0   |   98.3   |                                         89.0                                          |                                         98.2                                          |         87.5         |         98.0         | 4 clips x 3 crop | 0.7T  |  354M  |   [config](/configs/recognition/uniformerv2/uniformerv2-large-p14-res224_clip-kinetics710-pre_u8_kinetics600-rgb.py)    |                                  [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/kinetics600/uniformerv2-large-p14-res224_clip-kinetics710-pre_u8_kinetics600-rgb_20221219-cf88e4c2.pth)                                   |                                                                                                                  -                                                                                                                  |

|        16        |    Raw     |   UniFormerV2-L/14\*   | clip-kinetics710 |   89.4   |   98.3   |                                         89.4                                          |                                         98.3                                          |         87.8         |         98.0         | 4 clips x 3 crop | 1.3T  |  354M  |   [config](/configs/recognition/uniformerv2/uniformerv2-large-p14-res224_clip-kinetics710-pre_u16_kinetics600-rgb.py)   |                                  [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/kinetics600/uniformerv2-large-p14-res224_clip-kinetics710-pre_u16_kinetics600-rgb_20221219-38ff0e3e.pth)                                  |                                                                                                                  -                                                                                                                  |

|        32        |    Raw     |   UniFormerV2-L/14\*   | clip-kinetics710 |   89.2   |   98.3   |                                         89.5                                          |                                         98.3                                          |         87.7         |         98.1         | 2 clips x 3 crop | 2.7T  |  354M  |   [config](/configs/recognition/uniformerv2/uniformerv2-large-p14-res224_clip-kinetics710-pre_u32_kinetics600-rgb.py)   |                                  [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/kinetics600/uniformerv2-large-p14-res224_clip-kinetics710-pre_u32_kinetics600-rgb_20221219-d450d071.pth)                                  |                                                                                                                  -                                                                                                                  |

|        32        |    Raw     | UniFormerV2-L/14@336\* | clip-kinetics710 |   89.8   |   98.5   |                                         89.9                                          |                                         98.5                                          |         88.8         |         98.3         | 2 clips x 3 crop | 6.3T  |  354M  |   [config](/configs/recognition/uniformerv2/uniformerv2-large-p14-res336_clip-kinetics710-pre_u32_kinetics600-rgb.py)   |                                  [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/kinetics600/uniformerv2-large-p14-res336_clip-kinetics710-pre_u32_kinetics600-rgb_20221219-f984f5d2.pth)                                  |                                                                                                                  -                                                                                                                  |

### Kinetics-700

|        8         |    Raw     |    UniFormerV2-B/16    |       clip       |    -     |    -     |                                         75.8                                          |                                         92.8                                          |         75.9         |         92.9         | 4 clips x 3 crop | 0.1T  |  115M  | [config](/configs/recognition/uniformerv2/uniformerv2-base-p16-res224_clip-kinetics710-pre_8xb32-u8_kinetics700-rgb.py) |                 [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/uniformerv2-base-p16-res224_clip_8xb32-u8_kinetics700-rgb/uniformerv2-base-p16-res224_clip_8xb32-u8_kinetics700-rgb_20230313-f02e48ad.pth)                 |                 [log](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/uniformerv2-base-p16-res224_clip_8xb32-u8_kinetics700-rgb/uniformerv2-base-p16-res224_clip_8xb32-u8_kinetics700-rgb.log)                 |

|        8         |    Raw     |    UniFormerV2-B/16    | clip-kinetics710 |    -     |    -     |                                         76.3                                          |                                         92.7                                          |         76.3         |         92.9         | 4 clips x 3 crop | 0.1T  |  115M  | [config](/configs/recognition/uniformerv2/uniformerv2-base-p16-res224_clip-kinetics710-pre_8xb32-u8_kinetics700-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/uniformerv2-base-p16-res224_clip-kinetics710-pre_8xb32-u8_kinetics700-rgb/uniformerv2-base-p16-res224_clip-kinetics710-pre_8xb32-u8_kinetics700-rgb_20230313-69070837.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/uniformerv2-base-p16-res224_clip-kinetics710-pre_8xb32-u8_kinetics700-rgb/uniformerv2-base-p16-res224_clip-kinetics710-pre_8xb32-u8_kinetics700-rgb.log) |

|        8         |    Raw     |   UniFormerV2-L/14\*   | clip-kinetics710 |   80.8   |   95.2   |                                         80.8                                          |                                         95.4                                          |         79.4         |         94.8         | 4 clips x 3 crop | 0.7T  |  354M  |   [config](/configs/recognition/uniformerv2/uniformerv2-large-p14-res224_clip-kinetics710-pre_u8_kinetics700-rgb.py)    |                                  [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/kinetics700/uniformerv2-large-p14-res224_clip-kinetics710-pre_u8_kinetics700-rgb_20221219-bfb9f401.pth)                                   |                                                                                                                  -                                                                                                                  |

|        16        |    Raw     |   UniFormerV2-L/14\*   | clip-kinetics710 |   81.2   |   95.6   |                                         81.2                                          |                                         95.6                                          |         79.2         |         95.0         | 4 clips x 3 crop | 1.3T  |  354M  |   [config](/configs/recognition/uniformerv2/uniformerv2-large-p14-res224_clip-kinetics710-pre_u16_kinetics700-rgb.py)   |                                  [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/kinetics700/uniformerv2-large-p14-res224_clip-kinetics710-pre_u16_kinetics700-rgb_20221219-745209d2.pth)                                  |                                                                                                                  -                                                                                                                  |

|        32        |    Raw     |   UniFormerV2-L/14\*   | clip-kinetics710 |   81.4   |   95.7   |                                         81.5                                          |                                         95.7                                          |         79.8         |         95.3         | 2 clips x 3 crop | 2.7T  |  354M  |   [config](/configs/recognition/uniformerv2/uniformerv2-large-p14-res224_clip-kinetics710-pre_u32_kinetics700-rgb.py)   |                                  [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/kinetics700/uniformerv2-large-p14-res224_clip-kinetics710-pre_u32_kinetics700-rgb_20221219-eebe7056.pth)                                  |                                                                                                                  -                                                                                                                  |

|        32        |    Raw     | UniFormerV2-L/14@336\* | clip-kinetics710 |   82.1   |   96.0   |                                         82.1                                          |                                         96.1                                          |         80.6         |         95.6         | 2 clips x 3 crop | 6.3T  |  354M  |   [config](/configs/recognition/uniformerv2/uniformerv2-large-p14-res336_clip-kinetics710-pre_u32_kinetics700-rgb.py)   |                                  [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/kinetics700/uniformerv2-large-p14-res336_clip-kinetics710-pre_u32_kinetics700-rgb_20221219-95cf9046.pth)                                  |                                                                                                                  -                                                                                                                  |

### MiTv1

### MiTv1

| uniform sampling | resolution |        backbone        |           pretrain           | top1 acc | top5 acc | [reference](https://github.com/OpenGVLab/UniFormerV2/blob/main/MODEL_ZOO.md) top1 acc | [reference](https://github.com/OpenGVLab/UniFormerV2/blob/main/MODEL_ZOO.md) top5 acc | testing protocol | FLOPs | params |                                                              config                                                              |                                                                                                                                   ckpt                                                                                                                                   |                                                                                                                          log                                                                                                                          |

|        8         |    Raw     |    UniFormerV2-B/16    | clip-kinetics710-kinetics400 |   42.3   |   71.5   |                                         42.6                                          |                                         71.7                                          | 4 clips x 3 crop | 0.1T  |  115M  | [config](/configs/recognition/uniformerv2/uniformerv2-base-p16-res224_clip-kinetics710-kinetics-k400-pre_16xb32-u8_mitv1-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/uniformerv2-base-p16-res224_clip-kinetics710-kinetics-k400-pre_16xb32-u8_mitv1-rgb/uniformerv2-base-p16-res224_clip-kinetics710-kinetics-k400-pre_16xb32-u8_mitv1-rgb_20230313-a6f4a567.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/uniformerv2-base-p16-res224_clip-kinetics710-kinetics-k400-pre_16xb32-u8_mitv1-rgb/uniformerv2-base-p16-res224_clip-kinetics710-kinetics-k400-pre_16xb32-u8_mitv1-rgb.log) |

|        8         |    Raw     |   UniFormerV2-L/14\*   | clip-kinetics710-kinetics400 |   47.0   |   76.1   |                                         47.0                                          |                                         76.1                                          | 4 clips x 3 crop | 0.7T  |  354M  |    [config](/configs/recognition/uniformerv2/uniformerv2-large-p16-res224_clip-kinetics710-kinetics-k400-pre_u8_mitv1-rgb.py)    |                                          [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/mitv1/uniformerv2-large-p16-res224_clip-kinetics710-kinetics-k400-pre_u8_mitv1-rgb_20221219-882c0598.pth)                                           |                                                                                                                           -                                                                                                                           |

|        8         |    Raw     | UniFormerV2-L/14@336\* | clip-kinetics710-kinetics400 |   47.7   |   76.8   |                                         47.8                                          |                                         76.0                                          | 4 clips x 3 crop | 1.6T  |  354M  |    [config](/configs/recognition/uniformerv2/uniformerv2-large-p16-res336_clip-kinetics710-kinetics-k400-pre_u8_mitv1-rgb.py)    |                                          [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/mitv1/uniformerv2-large-p16-res336_clip-kinetics710-kinetics-k400-pre_u8_mitv1-rgb_20221219-9020986e.pth)                                           |                                                                                                                           -                                                                                                                           |

### Kinetics-710

| uniform sampling | resolution |        backbone        | pretrain | top1 acc | top5 acc |                    config                    |                    ckpt                    |                    log                     |
| :--------------: | :--------: | :--------------------: | :------: | :------: | :------: | :------------------------------------------: | :----------------------------------------: | :----------------------------------------: |

|        8         |    Raw     |   UniFormerV2-B/16\*   |   clip   |   78.9   |   94.2   | [config](/configs/recognition/uniformerv2/uniformerv2-base-p16-res224_clip-pre_u8_kinetics710-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/uniformerv2-base-p16-res224_clip-pre_u8_kinetics710-rgb/uniformerv2-base-p16-res224_clip-pre_u8_kinetics710-rgb_20230612-63cdbad9.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/uniformerv2-base-p16-res224_clip-pre_u8_kinetics710-rgb/uniformerv2-base-p16-res224_clip-pre_u8_kinetics710-rgb.log) |

|        8         |    Raw     |   UniFormerV2-L/14\*   |   clip   |    -     |    -     | [config](/configs/recognition/uniformerv2/uniformerv2-large-p14-res224_clip-pre_u8_kinetics710-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/kinetics710/uniformerv2-large-p14-res224_clip-pre_u8_kinetics710-rgb_20230612-d002a407.pth) |                     -                      |
|        8         |    Raw     | UniFormerV2-L/14@336\* |   clip   |    -     |    -     | [config](/configs/recognition/uniformerv2/uniformerv2-large-p14-res336_clip-pre_u8_kinetics710-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/uniformerv2/kinetics710/uniformerv2-large-p14-res336_clip-pre_u8_kinetics710-rgb_20230612-d723ddc1.pth) |                     -                      |

The models with * are ported from the repo [UniFormerV2](https://github.com/OpenGVLab/UniFormerV2/blob/main/MODEL_ZOO.md) and tested on our data. Due to computational limitations, we only support reliable training config for base model (i.e. UniFormerV2-B/16).

2. The values in `top1/5 acc` is tested on the same data list as the original repo, and the label map is provided by [UniFormerV2](https://drive.google.com/drive/folders/17VB-XdF3Kfr9ORmnGyXCxTMs86n0L4QL).

4. Since the original models for Kinetics-400/600/700 adopt different [label file](https://drive.google.com/drive/folders/17VB-XdF3Kfr9ORmnGyXCxTMs86n0L4QL), we simply map the weight according to the label name. New label map for Kinetics-400/600/700 can be found [here](/tools/data/kinetics).

5. Due to some differences between [SlowFast](https://github.com/facebookresearch/SlowFast) and MMAction2, there are some gaps between their performances.
6. Kinetics-710 is used for pretraining, which helps improve the performance on other datasets efficiently. You can find more details in the [paper](https://arxiv.org/abs/2211.09552). We also map the wegiht for Kinetics-710 checkpoints, you can find the label map [here](/tools/data/kinetics710/label_map_k710.txt).

- [preparing_kinetics](/tools/data/kinetics/README.md)
- [preparing_mit](/tools/data/mit/README.md)

## Test

```shell
```

Example: test UniFormerV2-B/16 model on Kinetics-400 dataset and dump the result to a pkl file.

```shell

```shell
```

```shell
python tools/test.py configs/recognition/uniformerv2/uniformerv2-base-p16-res224_clip-kinetics710-pre_u8_kinetics400-rgb.py \
```

```BibTeX
@article{Li2022UniFormerV2SL,
  title={UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer},
  author={Kunchang Li and Yali Wang and Yinan He and Yizhuo Li and Yi Wang and Limin Wang and Y. Qiao},
  volume={abs/2211.09552}
}
```

# PoseC3D

[Revisiting Skeleton-based Action Recognition](https://arxiv.org/abs/2104.13586)

Human skeleton, as a compact representation of human action, has received increasing attention in recent years. Many skeleton-based action recognition methods adopt graph convolutional networks (GCN) to extract features on top of human skeletons. Despite the positive results shown in previous works, GCN-based methods are subject to limitations in robustness, interoperability, and scalability. In this work, we propose PoseC3D, a new approach to skeleton-based action recognition, which relies on a 3D heatmap stack instead of a graph sequence as the base representation of human skeletons. Compared to GCN-based methods, PoseC3D is more effective in learning spatiotemporal features, more robust against pose estimation noises, and generalizes better in cross-dataset settings. Also, PoseC3D can handle multiple-person scenarios without additional computation cost, and its features can be easily integrated with other modalities at early fusion stages, which provides a great design space to further boost the performance. On four challenging datasets, PoseC3D consistently obtains superior performance, when used alone on skeletons and in combination with the RGB modality.

<img src="https://user-images.githubusercontent.com/34324155/142995620-21b5536c-8cda-48cd-9cb9-50b70cab7a89.png" width="800"/>
</div>

<table>
<thead>
  <tr>
    <td>
  <b> Pose Estimation Results </b>
  <br/>
  <br/>
  <br/>
    <td>
  <b> Keypoint Heatmap Volume Visualization </b>
  <br/>
  <br/>
  <br/>
    <td>

<br/>
  <br/>
    <td>
  <b> Limb Heatmap Volume Visualization </b>
  <br/>
  <br/>
  <br/>
  </tr>
</thead>
</table>

### FineGYM

| frame sampling strategy | pseudo heatmap | gpus |   backbone   | Mean Top-1 | testing protocol | FLOPs | params |                 config                 |                 ckpt                 |                 log                  |

| :---------------------: | :------------: | :--: | :----------: | :--------: | :--------------: | :---: | :----: | :------------------------------------: | :----------------------------------: | :----------------------------------: |
|       uniform 48        |    keypoint    |  8   | SlowOnly-R50 |    93.5    |     10 clips     | 20.6G |  2.0M  | [config](/configs/skeleton/posec3d/slowonly_r50_8xb16-u48-240e_gym-keypoint.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/posec3d/slowonly_r50_8xb16-u48-240e_gym-keypoint/slowonly_r50_8xb16-u48-240e_gym-keypoint_20220815-da338c58.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/posec3d/slowonly_r50_8xb16-u48-240e_gym-keypoint/slowonly_r50_8xb16-u48-240e_gym-keypoint.log) |

|       uniform 48        |      limb      |  8   | SlowOnly-R50 |    93.6    |     10 clips     | 20.6G |  2.0M  | [config](/configs/skeleton/posec3d/slowonly_r50_8xb16-u48-240e_gym-limb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/posec3d/slowonly_r50_8xb16-u48-240e_gym-limb/slowonly_r50_8xb16-u48-240e_gym-limb_20220815-2e6e3c5c.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/posec3d/slowonly_r50_8xb16-u48-240e_gym-limb/slowonly_r50_8xb16-u48-240e_gym-limb.log) |

### NTU60_XSub

| frame sampling strategy | pseudo heatmap | gpus |   backbone   | top1 acc | testing protocol | FLOPs | params |                 config                  |                 ckpt                  |                 log                  |
| :---------------------: | :------------: | :--: | :----------: | :------: | :--------------: | :---: | :----: | :-------------------------------------: | :-----------------------------------: | :----------------------------------: |

|       uniform 48        |    keypoint    |  8   | SlowOnly-R50 |   93.6   |     10 clips     | 20.6G |  2.0M  | [config](/configs/skeleton/posec3d/slowonly_r50_8xb16-u48-240e_ntu60-xsub-keypoint.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/posec3d/slowonly_r50_8xb16-u48-240e_ntu60-xsub-keypoint/slowonly_r50_8xb16-u48-240e_ntu60-xsub-keypoint_20220815-38db104b.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/posec3d/slowonly_r50_8xb16-u48-240e_ntu60-xsub-keypoint/slowonly_r50_8xb16-u48-240e_ntu60-xsub-keypoint.log) |

|       uniform 48        |      limb      |  8   | SlowOnly-R50 |   93.5   |     10 clips     | 20.6G |  2.0M  | [config](/configs/skeleton/posec3d/slowonly_r50_8xb16-u48-240e_ntu60-xsub-keypoint.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/posec3d/slowonly_r50_8xb16-u48-240e_ntu60-xsub-limb/slowonly_r50_8xb16-u48-240e_ntu60-xsub-limb_20220815-af2f119a.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/posec3d/slowonly_r50_8xb16-u48-240e_ntu60-xsub-limb/slowonly_r50_8xb16-u48-240e_ntu60-xsub-limb.log) |
|                         |     Fusion     |      |              |   94.0   |                  |       |        |                                         |                                       |                                      |

### UCF101

### UCF101

|       uniform 48        |    keypoint    |  8   | SlowOnly-R50 |   86.8   |     10 clips     | 14.6G |  3.1M  | [config](/configs/skeleton/posec3d/slowonly_kinetics400-pretrained-r50_8xb16-u48-120e_ucf101-split1-keypoint.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/posec3d/slowonly_kinetics400-pretrained-r50_8xb16-u48-120e_ucf101-split1-keypoint/slowonly_kinetics400-pretrained-r50_8xb16-u48-120e_ucf101-split1-keypoint_20220815-9972260d.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/posec3d/slowonly_kinetics400-pretrained-r50_8xb16-u48-120e_ucf101-split1-keypoint/slowonly_kinetics400-pretrained-r50_8xb16-u48-120e_ucf101-split1-keypoint.log) |

### HMDB51

|       uniform 48        |    keypoint    |  8   | SlowOnly-R50 |   69.6   |     10 clips     | 14.6G |  3.0M  | [config](/configs/skeleton/posec3d/slowonly_kinetics400-pretrained-r50_8xb16-u48-120e_hmdb51-split1-keypoint.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/posec3d/slowonly_kinetics400-pretrained-r50_8xb16-u48-120e_hmdb51-split1-keypoint/slowonly_kinetics400-pretrained-r50_8xb16-u48-120e_hmdb51-split1-keypoint_20220815-17eaa484.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/posec3d/slowonly_kinetics400-pretrained-r50_8xb16-u48-120e_hmdb51-split1-keypoint/slowonly_kinetics400-pretrained-r50_8xb16-u48-120e_hmdb51-split1-keypoint.log) |

1. The **gpus** indicates the number of gpu we used to get the checkpoint. It is noteworthy that the configs we provide are used for 8 gpus as default.
   e.g., lr=0.01 for 8 GPUs x 8 videos/gpu and lr=0.04 for 16 GPUs x 16 videos/gpu.
2. You can follow the guide in [Preparing Skeleton Dataset](/tools/data/skeleton/README.md) to obtain skeleton annotations used in the above configs.

## Train

```shell
```

Example: train PoseC3D model on FineGYM dataset in a deterministic option.

```shell
python tools/train.py configs/skeleton/posec3d/slowonly_r50_8xb16-u48-240e_gym-keypoint.py \
```

```shell
```

For training with your custom dataset, you can refer to [Custom Dataset Training](/configs/skeleton/posec3d/custom_dataset_training.md).

## Test

```shell
```

Example: test PoseC3D model on FineGYM dataset.

```shell
python tools/test.py configs/skeleton/posec3d/slowonly_r50_8xb16-u48-240e_gym-keypoint.py \
    checkpoints/SOME_CHECKPOINT.pth
```

```BibTeX

```

```BibTeX
}
```

# Mixed Precision Training

<details>

```bibtex
}
```

<details>

<summary align="right"><a href="http://openaccess.thecvf.com/content_CVPR_2019/html/Pavllo_3D_Human_Pose_Estimation_in_Video_With_Temporal_Convolutions_and_CVPR_2019_paper.html">VideoPose3D (CVPR'2019)</a></summary>

```bibtex
@inproceedings{pavllo20193d,
title={3d human pose estimation in video with temporal convolutions and semi-supervised training},
author={Pavllo, Dario and Feichtenhofer, Christoph and Grangier, David and Auli, Michael},
pages={7753--7762},
}
```

<details>

```bibtex
author = {Ionescu, Catalin and Papava, Dragos and Olaru, Vlad and Sminchisescu, Cristian},

```bibtex
title = {Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments},
volume = {36},
number = {7},
pages = {1325-1339},
month = {jul},
year = {2014}
}
```

Testing results on Human3.6M dataset with ground truth 2D detections, supervised training

| Arch                                                       | Receptive Field | MPJPE | P-MPJPE |                            ckpt                            |                            log                            |
| :--------------------------------------------------------- | :-------------: | :---: | :-----: | :--------------------------------------------------------: | :-------------------------------------------------------: |

| [VideoPose3D](/configs/body_3d_keypoint/pose_lift/h36m/pose-lift_videopose3d-27frm-supv_8xb128-80e_h36m.py) |       27        | 40.1  |  30.1   | [ckpt](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_27frames_fullconv_supervised-fe8fbba9_20210527.pth) | [log](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_27frames_fullconv_supervised_20210527.log.json) |
| [VideoPose3D](/configs/body_3d_keypoint/pose_lift/h36m/pose-lift_videopose3d-81frm-supv_8xb128-80e_h36m.py) |       81        | 39.1  |  29.3   | [ckpt](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_81frames_fullconv_supervised-1f2d1104_20210527.pth) | [log](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_81frames_fullconv_supervised_20210527.log.json) |

| [VideoPose3D](/configs/body_3d_keypoint/pose_lift/h36m/pose-lift_videopose3d-243frm-supv_8xb128-80e_h36m.py) |       243       |       |         | [ckpt](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_243frames_fullconv_supervised-880bea25_20210527.pth) | [log](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_243frames_fullconv_supervised_20210527.log.json) |

Testing results on Human3.6M dataset with CPN 2D detections<sup>1</sup>, supervised training

| [VideoPose3D](/configs/body_3d_keypoint/pose_lift/h36m/pose-lift_videopose3d-1frm-supv-cpn-ft_8xb128-80e_h36m.py) |        1        | 53.0  |  41.3   | [ckpt](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_1frame_fullconv_supervised_cpn_ft-5c3afaed_20210527.pth) | [log](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_1frame_fullconv_supervised_cpn_ft_20210527.log.json) |
| [VideoPose3D](/configs/body_3d_keypoint/pose_lift/h36m/pose-lift_videopose3d-243frm-supv-cpn-ft_8xb128-200e_h36m.py) |       243       |       |         | [ckpt](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_243frames_fullconv_supervised_cpn_ft-88f5abbb_20210527.pth) | [log](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_243frames_fullconv_supervised_cpn_ft_20210527.log.json) |

Testing results on Human3.6M dataset with ground truth 2D detections, semi-supervised training

| Training Data |                        Arch                         | Receptive Field | MPJPE | P-MPJPE | N-MPJPE |                        ckpt                         |                         log                         |
| :------------ | :-------------------------------------------------: | :-------------: | :---: | :-----: | :-----: | :-------------------------------------------------: | :-------------------------------------------------: |
| 10% S1        | [VideoPose3D](/configs/body_3d_keypoint/pose_lift/h36m/pose-lift_videopose3d-27frm-semi-supv_8xb64-200e_h36m.py) |       27        | 57.2  |  42.4   |  54.2   | [ckpt](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_27frames_fullconv_semi-supervised-54aef83b_20210527.pth) | [log](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_27frames_fullconv_semi-supervised_20210527.log.json) |

Testing results on Human3.6M dataset with CPN 2D detections<sup>1</sup>, semi-supervised training

| Training Data |              Arch              | Receptive Field | MPJPE | P-MPJPE | N-MPJPE |                              ckpt                              |                              log                              |
| :------------ | :----------------------------: | :-------------: | :---: | :-----: | :-----: | :------------------------------------------------------------: | :-----------------------------------------------------------: |
| 10% S1        | [VideoPose3D](/configs/xxx.py) |       27        | 67.3  |  50.4   |  63.6   | [ckpt](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_27frames_fullconv_semi-supervised_cpn_ft-71be9cde_20210527.pth) | [log](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_27frames_fullconv_semi-supervised_cpn_ft_20210527.log.json) |

<sup>1</sup> CPN 2D detections are provided by [official repo](https://github.com/facebookresearch/VideoPose3D/blob/master/DATASETS.md). The reformatted version used in this repository can be downloaded from [train_detection](https://download.openmmlab.com/mmpose/body3d/videopose/cpn_ft_h36m_dbb_train.npy) and [test_detection](https://download.openmmlab.com/mmpose/body3d/videopose/cpn_ft_h36m_dbb_test.npy).

# MMDetection3d 模型部署

- [MMDetection3d 模型部署](#mmdetection3d-模型部署)
  - [安装 mmdet3d](#安装-mmdet3d)

[MMDetection3d](https://github.com/open-mmlab/mmdetection3d)，又称 `mmdet3d`， 是一个基于 PyTorch 的目标检测开源工具箱, 下一代面向3D检测的平台。它是 [OpenMMLab](https://openmmlab.com/) 项目的一部分。

## 安装 mmdet3d

因为依赖的 codebase 不在 master 分支，所以要切到相应分支：

先安装前置依赖 mmcv/mmdet/mmseg，再安装 mmdet3d

```bash

cd -
```

成功后 `tools/check_env.py` 应能正常显示 mmdet3d 版本号。

```bash
..

cd -
```

```bash
..
..
```

## 模型转换

使用 `tools/deploy.py` 把 mmdet3d 转到相应后端，以 centerpoint onnxruntime 为例：

```bash

```

`work-dir` 应生成对应 onnx

```bash
..
```

## 模型推理

目前 mmdet3d 的 voxelize 预处理和后处理未转成 onnx 操作；C++ SDK 也未实现 voxelize 计算。调用方需参照对应 [python 实现](../../../mmdeploy/codebase/mmdet3d/deploy/voxel_detection_model.py) 完成。

## 模型支持列表

- 考虑到 ScatterND、动态 shape 等已知问题，请确保 trt >= 8.4

# Semantic FPN

> [Panoptic Feature Pyramid Networks](https://arxiv.org/abs/1901.02446)

<a href="https://github.com/facebookresearch/detectron2">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/decode_heads/fpn_head.py#L12">Code Snippet</a>

The recently introduced panoptic segmentation task has renewed our community's interest in unifying the tasks of instance segmentation (for thing classes) and semantic segmentation (for stuff classes). However, current state-of-the-art methods for this joint task use separate and dissimilar networks for instance and semantic segmentation, without performing any shared computation. In this work, we aim to unify these methods at the architectural level, designing a single network for both tasks. Our approach is to endow Mask R-CNN, a popular instance segmentation method, with a semantic segmentation branch using a shared Feature Pyramid Network (FPN) backbone. Surprisingly, this simple baseline not only remains effective for instance segmentation, but also yields a lightweight, top-performing method for semantic segmentation. In this work, we perform a detailed study of this minimally extended version of Mask R-CNN with FPN, which we refer to as Panoptic FPN, and show it is a robust and accurate baseline for both tasks. Given its effectiveness and conceptual simplicity, we hope our method can serve as a strong baseline and aid future research in panoptic segmentation.

<img src="https://user-images.githubusercontent.com/24582831/142902694-03ed2131-9104-467b-ace1-c74c62fb7177.png" width="60%"/>
</div>

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                    | download                                                                                                                                                                                                                                                                                                                           |

| FPN    | R-50     | 512x1024  |   80000 |      2.8 | 13.54          | V100   | 74.52 | 76.08         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/sem_fpn/fpn_r50_4xb2-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/sem_fpn/fpn_r50_512x1024_80k_cityscapes/fpn_r50_512x1024_80k_cityscapes_20200717_021437-94018a0d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/sem_fpn/fpn_r50_512x1024_80k_cityscapes/fpn_r50_512x1024_80k_cityscapes-20200717_021437.log.json)     |

| FPN    | R-101    | 512x1024  |   80000 |      3.9 | 10.29          | V100   | 75.80 | 77.40         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/sem_fpn/fpn_r101_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/sem_fpn/fpn_r101_512x1024_80k_cityscapes/fpn_r101_512x1024_80k_cityscapes_20200717_012416-c5800d4c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/sem_fpn/fpn_r101_512x1024_80k_cityscapes/fpn_r101_512x1024_80k_cityscapes-20200717_012416.log.json) |

### ADE20K

### ADE20K

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                | download                                                                                                                                                                                                                                                                                                           |

| FPN    | R-50     | 512x512   |  160000 |      4.9 | 55.77          | V100   | 37.49 | 39.09         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/sem_fpn/fpn_r50_4xb4-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/sem_fpn/fpn_r50_512x512_160k_ade20k/fpn_r50_512x512_160k_ade20k_20200718_131734-5b5a6ab9.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/sem_fpn/fpn_r50_512x512_160k_ade20k/fpn_r50_512x512_160k_ade20k-20200718_131734.log.json)     |

| FPN    | R-101    | 512x512   |  160000 |      5.9 | 40.58          | V100   | 39.35 | 40.72         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/sem_fpn/fpn_r101_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/sem_fpn/fpn_r101_512x512_160k_ade20k/fpn_r101_512x512_160k_ade20k_20200718_131734-306b5004.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/sem_fpn/fpn_r101_512x512_160k_ade20k/fpn_r101_512x512_160k_ade20k-20200718_131734.log.json) |

```bibtex
@inproceedings{kirillov2019panoptic,
  title={Panoptic feature pyramid networks},
  author={Kirillov, Alexander and Girshick, Ross and He, Kaiming and Doll{\'a}r, Piotr},
  pages={6399--6408},
}
```

```text
x: [[259 313 389 427 354 302]], y: [[542 462 417 459 507 582]], ornt: [u'c'], transcriptions: [u'PAUL']
x: [[400 478 494 436]], y: [[398 380 448 465]], ornt: [u'#'], transcriptions: [u'#']
```

This page provides the instructions for dataset preparation on existing benchmarks, include

- Multiple Object Tracking

  - [MOT Challenge](https://motchallenge.net/)
  - [CrowdHuman](https://www.crowdhuman.org/)

- Video Instance Segmentation

  - [YouTube-VIS](https://youtube-vos.org/dataset/vis/)

### 1. Download Datasets

Please download the datasets from the official websites. It is recommended to symlink the root of the datasets to `$MMDETECTION/data`.

#### 1.1 Multiple Object Tracking

- For the training and testing of multi object tracking task, one of the MOT Challenge datasets (e.g. MOT17, MOT20) are needed, CrowdHuman can be served as comlementary dataset.

- For users in China, the following datasets can be downloaded from [OpenDataLab](https://opendatalab.com/) with high speed:

  - [MOT17](https://opendatalab.com/MOT17/download)
  - [MOT20](https://opendatalab.com/MOT20/download)
  - [CrowdHuman](https://opendatalab.com/CrowdHuman/download)

- [MOT17](https://opendatalab.com/MOT17/download)

#### 1.2 Video Instance Segmentation

- For the training and testing of video instance segmetatioon task, only one of YouTube-VIS datasets (e.g. YouTube-VIS 2019, YouTube-VIS 2021) is needed.

- YouTube-VIS 2019 dataset can be download from [YouTubeVOS](https://codalab.lisn.upsaclay.fr/competitions/6064)

- YouTube-VIS 2021 dataset can be download from [YouTubeVOS](https://codalab.lisn.upsaclay.fr/competitions/7680)

#### 1.3 Data Structure

```
├── mmdet
├── tools
├── data
│   ├── coco
│   │
|   ├── MOT15/MOT16/MOT17/MOT20
|   |   ├── train
|   |   |   ├── MOT17-02-DPM

├── data
│   ├── coco
│   │
|   |   |   |   ├── det
|   │   │   │   ├── gt
|   │   │   │   ├── img1
|   │   │   │   ├── seqinfo.ini
│   │   │   ├── ......
|   |   ├── test
|   |   |   ├── MOT17-01-DPM
│   │
│   ├── crowdhuman
│   │   ├── annotation_train.odgt
│   │   ├── annotation_val.odgt
│   │   │   ├── Images
│   │   │   ├── CrowdHuman_train01.zip
│   │   │   ├── CrowdHuman_train02.zip
│   │   │   ├── CrowdHuman_train03.zip
│   │   ├── val
│   │   │   ├── CrowdHuman_val.zip
│   │
```

### 2. Convert Annotations

In this case, you need to convert the official annotations to coco style. We provide scripts and the usages are as following:

```shell
# MOT17

│   │
```

```shell
# MOT17
# The processing of other MOT Challenge dataset is the same as MOT17
python ./tools/dataset_converters/mot2coco.py -i ./data/MOT17/ -o ./data/MOT17/annotations --split-train --convert-det
python ./tools/dataset_converters/mot2reid.py -i ./data/MOT17/ -o ./data/MOT17/reid --val-split 0.2 --vis-threshold 0.3

# CrowdHuman
python ./tools/dataset_converters/crowdhuman2coco.py -i ./data/crowdhuman -o ./data/crowdhuman/annotations

# YouTube-VIS 2019
python ./tools/dataset_converters/youtubevis/youtubevis2coco.py -i ./data/youtube_vis_2019 -o ./data/youtube_vis_2019/annotations --version 2019

# YouTube-VIS 2021
python ./tools/dataset_converters/youtubevis/youtubevis2coco.py -i ./data/youtube_vis_2021 -o ./data/youtube_vis_2021/annotations --version 2021

```

```

The folder structure will be as following after your run these scripts:

```
├── mmtrack
├── tools
├── data
│   ├── coco
│   │
|   │   │   │   ├── gt
|   |   ├── test
|   |   ├── reid
│   │   │   ├── imgs
│   │   │   ├── meta
│   │

|   |   ├── reid
│   │
│   │   ├── val
│   │   │   ├── crowdhuman_train.json
│   │   │   ├── crowdhuman_val.json
│   │
│   ├── youtube_vis_2019
│   │   │── train
│   │   │   │── JPEGImages
│   │   │   │── ......
│   │   │── valid
│   │   │── test
│   │   │── train.json (the official annotation files)
│   │   │── valid.json (the official annotation files)
│   │   │── test.json (the official annotation files)
│   │   │── annotations (the converted annotation file)
│   │
│   ├── youtube_vis_2021

│   │
│   │   │   │── instances.json (the official annotation files)
│   │   │── test
```

#### The folder of annotations and reid in MOT15/MOT16/MOT17/MOT20

We take MOT17 dataset as examples, the other datasets share similar structure.

There are 8 JSON files in `data/MOT17/annotations`:

`train_cocoformat.json`: JSON file containing the annotations information of the training set in MOT17 dataset.

`train_detections.pkl`: Pickle file containing the public detections of the training set in MOT17 dataset.

`test_cocoformat.json`: JSON file containing the annotations information of the testing set in MOT17 dataset.

`test_detections.pkl`: Pickle file containing the public detections of the testing set in MOT17 dataset.

`half-train_cocoformat.json`, `half-train_detections.pkl`, `half-val_cocoformat.json`and `half-val_detections.pkl` share similar meaning with `train_cocoformat.json` and `train_detections.pkl`. The `half` means we split each video in the training set into half. The first half videos are denoted as `half-train` set, and the second half videos are denoted as`half-val` set.

The structure of `data/MOT17/reid` is as follows:

```
reid
├── imgs
│   ├── MOT17-02-FRCNN_000002
│   │   ├── 000000.jpg

```
reid
├── imgs
│   │   ├── 000001.jpg
│   │   ├── ...
│   ├── MOT17-02-FRCNN_000003
│   │   ├── ...
├── meta
│   ├── train_80.txt
│   ├── val_20.txt
```

The `80` in `train_80.txt` means the proportion of the training dataset to the whole ReID dataset is 80%. While the proportion of the validation dataset is 20%.

For training, we provide a annotation list `train_80.txt`. Each line of the list contains a filename and its corresponding ground-truth labels. The format is as follows:

```
MOT17-05-FRCNN_000110/000018.jpg 0
MOT17-13-FRCNN_000146/000014.jpg 1
MOT17-05-FRCNN_000088/000004.jpg 2
MOT17-02-FRCNN_000009/000081.jpg 3
```

`MOT17-05-FRCNN_000110` denotes the 110-th person in `MOT17-05-FRCNN` video.

For validation, The annotation list `val_20.txt` remains the same as format above.

```

Images in `reid/imgs` are cropped from raw images in `MOT17/train` by the corresponding `gt.txt`. The value of ground-truth labels should fall in range `[0, num_classes - 1]`.

#### The folder of annotations in crowdhuman

There are 2 JSON files in `data/crowdhuman/annotations`:

`crowdhuman_train.json`:  JSON file containing the annotations information of the training set in CrowdHuman dataset.
`crowdhuman_val.json`:  JSON file containing the annotations information of the validation set in CrowdHuman dataset.

#### The folder of annotations in youtube_vis_2019/youtube_vis2021

There are 3 JSON files in `data/youtube_vis_2019/annotations` or `data/youtube_vis_2021/annotations`:

`youtube_vis_2019_train.json`/`youtube_vis_2021_train.json`: JSON file containing the annotations information of the training set in youtube_vis_2019/youtube_vis2021 dataset.

`youtube_vis_2019_valid.json`/`youtube_vis_2021_valid.json`: JSON file containing the annotations information of the validation set in youtube_vis_2019/youtube_vis2021 dataset.

`youtube_vis_2019_test.json`/`youtube_vis_2021_test.json`: JSON file containing the annotations information of the testing set in youtube_vis_2019/youtube_vis2021 dataset.

# How to contribute to MMAction2

All kinds of contributions are welcome, including but not limited to the following.

- Fixes (typo, bugs)
- New features and components
- Add new project (Recommended) about video understanding algorithm with less restriction, refer to [here](../projectzoo.md) for details

## Workflow

1. Fork and pull the latest mmaction2
2. Checkout a new branch with a meaningful name (do not use main branch for PRs)
3. Commit your changes
4. Create a PR

```{note}
- If you plan to add some new features that involve large changes, it is encouraged to open an issue for discussion first.
- If you are the author of some papers and would like to include your method to mmaction2, please contact us. We will much appreciate your contribution.
```

### Python

```

### Python

- [flake8](http://flake8.pycqa.org/en/latest/): linter
- [yapf](https://github.com/google/yapf): formatter
- [isort](https://github.com/timothycrosley/isort): sort imports

Style configurations of yapf and isort can be found in [setup.cfg](https://github.com/open-mmlab/mmaction2/blob/main/setup.cfg).

The config for a pre-commit hook is stored in [.pre-commit-config](https://github.com/open-mmlab/mmaction2/blob/main/.pre-commit-config.yaml).

After you clone the repository, you will need to install initialize pre-commit hook.

```
```

From the repository folder

```shell
```

After this on every commit check code linters and formatter will be enforced.

> Before you create a PR, make sure that your code lints and is formatted by yapf.

### C++ and CUDA

```

We follow the [Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html).

# Cascade R-CNN

> [Cascade R-CNN: High Quality Object Detection and Instance Segmentation](https://arxiv.org/abs/1906.09756)

In object detection, the intersection over union (IoU) threshold is frequently used to define positives/negatives. The threshold used to train a detector defines its quality. While the commonly used threshold of 0.5 leads to noisy (low-quality) detections, detection performance frequently degrades for larger thresholds. This paradox of high-quality detection has two causes: 1) overfitting, due to vanishing positive samples for large thresholds, and 2) inference-time quality mismatch between detector and test hypotheses. A multi-stage object detection architecture, the Cascade R-CNN, composed of a sequence of detectors trained with increasing IoU thresholds, is proposed to address these problems. The detectors are trained sequentially, using the output of a detector as training set for the next. This resampling progressively improves hypotheses quality, guaranteeing a positive training set of equivalent size for all detectors and minimizing overfitting. The same cascade is applied at inference, to eliminate quality mismatches between hypotheses and detectors. An implementation of the Cascade R-CNN without bells or whistles achieves state-of-the-art performance on the COCO dataset, and significantly improves high-quality detection on generic and specific object detection datasets, including VOC, KITTI, CityPerson, and WiderFace. Finally, the Cascade R-CNN is generalized to instance segmentation, with nontrivial improvements over the Mask R-CNN.

<img src="https://user-images.githubusercontent.com/40661020/143872197-d99b90e4-4f05-4329-80a4-327ac862a051.png"/>
</div>

|    Backbone     |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                       Config                        |                                                                                                                                                                             Download                                                                                                                                                                              |

| :-------------: | :-----: | :-----: | :------: | :------------: | :----: | :-------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|    R-50-FPN     |  caffe  |   1x    |   4.2    |                |  40.4  |  [config](./cascade-rcnn_r50-caffe_fpn_1x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_r50_caffe_fpn_1x_coco/cascade_rcnn_r50_caffe_fpn_1x_coco_bbox_mAP-0.404_20200504_174853-b857be87.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_r50_caffe_fpn_1x_coco/cascade_rcnn_r50_caffe_fpn_1x_coco_20200504_174853.log.json)   |

|    R-50-FPN     | pytorch |   1x    |   4.4    |      16.1      |  40.3  |     [config](./cascade-rcnn_r50_fpn_1x_coco.py)     |                          [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_r50_fpn_1x_coco/cascade_rcnn_r50_fpn_1x_coco_20200316-3dc56deb.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_r50_fpn_1x_coco/cascade_rcnn_r50_fpn_1x_coco_20200316_214748.log.json)                          |
|    R-50-FPN     | pytorch |   20e   |    -     |       -        |  41.0  |    [config](./cascade-rcnn_r50_fpn_20e_coco.py)     |             [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_r50_fpn_20e_coco/cascade_rcnn_r50_fpn_20e_coco_bbox_mAP-0.41_20200504_175131-e9872a90.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_r50_fpn_20e_coco/cascade_rcnn_r50_fpn_20e_coco_20200504_175131.log.json)              |

|    R-101-FPN    |  caffe  |   1x    |   6.2    |                |  42.3  | [config](./cascade-rcnn_r101-caffe_fpn_1x_coco.py)  | [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_r101_caffe_fpn_1x_coco/cascade_rcnn_r101_caffe_fpn_1x_coco_bbox_mAP-0.423_20200504_175649-cab8dbd5.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_r101_caffe_fpn_1x_coco/cascade_rcnn_r101_caffe_fpn_1x_coco_20200504_175649.log.json) |
|    R-101-FPN    | pytorch |   1x    |   6.4    |      13.5      |  42.0  |    [config](./cascade-rcnn_r101_fpn_1x_coco.py)     |                        [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_r101_fpn_1x_coco/cascade_rcnn_r101_fpn_1x_coco_20200317-0b6a2fbf.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_r101_fpn_1x_coco/cascade_rcnn_r101_fpn_1x_coco_20200317_101744.log.json)                        |

|    R-101-FPN    | pytorch |   20e   |    -     |       -        |  42.5  |    [config](./cascade-rcnn_r101_fpn_20e_coco.py)    |           [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_r101_fpn_20e_coco/cascade_rcnn_r101_fpn_20e_coco_bbox_mAP-0.425_20200504_231812-5057dcc5.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_r101_fpn_20e_coco/cascade_rcnn_r101_fpn_20e_coco_20200504_231812.log.json)           |
| X-101-32x4d-FPN | pytorch |   1x    |   7.6    |      10.9      |  43.7  | [config](./cascade-rcnn_x101-32x4d_fpn_1x_coco.py)  |            [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_x101_32x4d_fpn_1x_coco/cascade_rcnn_x101_32x4d_fpn_1x_coco_20200316-95c2deb6.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_x101_32x4d_fpn_1x_coco/cascade_rcnn_x101_32x4d_fpn_1x_coco_20200316_055608.log.json)            |

| X-101-32x4d-FPN | pytorch |   20e   |   7.6    |                |  43.7  | [config](./cascade-rcnn_x101-32x4d_fpn_20e_coco.py) |      [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_x101_32x4d_fpn_20e_coco/cascade_rcnn_x101_32x4d_fpn_20e_coco_20200906_134608-9ae0a720.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_x101_32x4d_fpn_20e_coco/cascade_rcnn_x101_32x4d_fpn_20e_coco_20200906_134608.log.json)       |
| X-101-64x4d-FPN | pytorch |   1x    |   10.7   |                |  44.7  | [config](./cascade-rcnn_x101-64x4d_fpn_1x_coco.py)  |        [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_x101_64x4d_fpn_1x_coco/cascade_rcnn_x101_64x4d_fpn_1x_coco_20200515_075702-43ce6a30.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_x101_64x4d_fpn_1x_coco/cascade_rcnn_x101_64x4d_fpn_1x_coco_20200515_075702.log.json)         |

| X-101-64x4d-FPN | pytorch |   20e   |   10.7   |                |  44.5  | [config](./cascade-rcnn_x101_64x4d_fpn_20e_coco.py) |      [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_x101_64x4d_fpn_20e_coco/cascade_rcnn_x101_64x4d_fpn_20e_coco_20200509_224357-051557b1.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_x101_64x4d_fpn_20e_coco/cascade_rcnn_x101_64x4d_fpn_20e_coco_20200509_224357.log.json)       |

|    Backbone     |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                          Config                          |                                                                                                                                                                                               Download                                                                                                                                                                                                |

|    R-50-FPN     |  caffe  |   1x    |   5.9    |                |  41.2  |  36.0   |  [config](./cascade-mask-rcnn_r50-caffe_fpn_1x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r50_caffe_fpn_1x_coco/cascade_mask_rcnn_r50_caffe_fpn_1x_coco_bbox_mAP-0.412__segm_mAP-0.36_20200504_174659-5004b251.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r50_caffe_fpn_1x_coco/cascade_mask_rcnn_r50_caffe_fpn_1x_coco_20200504_174659.log.json)    |

|    R-50-FPN     | pytorch |   1x    |   6.0    |      11.2      |  41.2  |  35.9   |     [config](./cascade-mask-rcnn_r50_fpn_1x_coco.py)     |                                  [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r50_fpn_1x_coco/cascade_mask_rcnn_r50_fpn_1x_coco_20200203-9d4dcb24.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r50_fpn_1x_coco/cascade_mask_rcnn_r50_fpn_1x_coco_20200203_170449.log.json)                                  |

|    R-50-FPN     | pytorch |   20e   |    -     |       -        |  41.9  |  36.5   |    [config](./cascade-mask-rcnn_r50_fpn_20e_coco.py)     |             [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r50_fpn_20e_coco/cascade_mask_rcnn_r50_fpn_20e_coco_bbox_mAP-0.419__segm_mAP-0.365_20200504_174711-4af8e66e.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r50_fpn_20e_coco/cascade_mask_rcnn_r50_fpn_20e_coco_20200504_174711.log.json)             |

|    R-101-FPN    |  caffe  |   1x    |   7.8    |                |  43.2  |  37.6   | [config](./cascade-mask-rcnn_r101-caffe_fpn_1x_coco.py)  | [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r101_caffe_fpn_1x_coco/cascade_mask_rcnn_r101_caffe_fpn_1x_coco_bbox_mAP-0.432__segm_mAP-0.376_20200504_174813-5c1e9599.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r101_caffe_fpn_1x_coco/cascade_mask_rcnn_r101_caffe_fpn_1x_coco_20200504_174813.log.json) |

|    R-101-FPN    | pytorch |   1x    |   7.9    |      9.8       |  42.9  |  37.3   |    [config](./cascade-mask-rcnn_r101_fpn_1x_coco.py)     |                                [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r101_fpn_1x_coco/cascade_mask_rcnn_r101_fpn_1x_coco_20200203-befdf6ee.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r101_fpn_1x_coco/cascade_mask_rcnn_r101_fpn_1x_coco_20200203_092521.log.json)                                |

|    R-101-FPN    | pytorch |   20e   |    -     |       -        |  43.4  |  37.8   |    [config](./cascade-mask-rcnn_r101_fpn_20e_coco.py)    |           [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r101_fpn_20e_coco/cascade_mask_rcnn_r101_fpn_20e_coco_bbox_mAP-0.434__segm_mAP-0.378_20200504_174836-005947da.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r101_fpn_20e_coco/cascade_mask_rcnn_r101_fpn_20e_coco_20200504_174836.log.json)           |

| X-101-32x4d-FPN | pytorch |   1x    |   9.2    |      8.6       |  44.3  |  38.3   | [config](./cascade-mask-rcnn_x101-32x4d_fpn_1x_coco.py)  |                    [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_x101_32x4d_fpn_1x_coco/cascade_mask_rcnn_x101_32x4d_fpn_1x_coco_20200201-0f411b1f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_x101_32x4d_fpn_1x_coco/cascade_mask_rcnn_x101_32x4d_fpn_1x_coco_20200201_052416.log.json)                    |

| X-101-32x4d-FPN | pytorch |   20e   |   9.2    |       -        |  45.0  |  39.0   | [config](./cascade-mask-rcnn_x101-32x4d_fpn_20e_coco.py) |              [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_x101_32x4d_fpn_20e_coco/cascade_mask_rcnn_x101_32x4d_fpn_20e_coco_20200528_083917-ed1f4751.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_x101_32x4d_fpn_20e_coco/cascade_mask_rcnn_x101_32x4d_fpn_20e_coco_20200528_083917.log.json)               |

| X-101-64x4d-FPN | pytorch |   1x    |   12.2   |      6.7       |  45.3  |  39.2   | [config](./cascade-mask-rcnn_x101-64x4d_fpn_1x_coco.py)  |                    [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_x101_64x4d_fpn_1x_coco/cascade_mask_rcnn_x101_64x4d_fpn_1x_coco_20200203-9a2db89d.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_x101_64x4d_fpn_1x_coco/cascade_mask_rcnn_x101_64x4d_fpn_1x_coco_20200203_044059.log.json)                    |

| X-101-64x4d-FPN | pytorch |   20e   |   12.2   |                |  45.6  |  39.5   | [config](./cascade-mask-rcnn_x101-64x4d_fpn_20e_coco.py) |              [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_x101_64x4d_fpn_20e_coco/cascade_mask_rcnn_x101_64x4d_fpn_20e_coco_20200512_161033-bdb5126a.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_x101_64x4d_fpn_20e_coco/cascade_mask_rcnn_x101_64x4d_fpn_20e_coco_20200512_161033.log.json)               |

- The `20e` schedule in Cascade (Mask) R-CNN indicates decreasing the lr at 16 and 19 epochs, with a total of 20 epochs.

We also train some models with longer schedules and multi-scale training for Cascade Mask R-CNN. The users could finetune them for downstream tasks.

|    Backbone     |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                           Config                           |                                                                                                                                                                                                Download                                                                                                                                                                                                |

|    R-50-FPN     |  caffe  |   3x    |   5.7    |                |  44.0  |  38.1   | [config](./cascade-mask-rcnn_r50-caffe_fpn_ms-3x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r50_caffe_fpn_mstrain_3x_coco/cascade_mask_rcnn_r50_caffe_fpn_mstrain_3x_coco_20210707_002651-6e29b3a6.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r50_caffe_fpn_mstrain_3x_coco/cascade_mask_rcnn_r50_caffe_fpn_mstrain_3x_coco_20210707_002651.log.json)   |

|    R-50-FPN     | pytorch |   3x    |   5.9    |                |  44.3  |  38.5   |    [config](./cascade-mask-rcnn_r50_fpn_ms-3x_coco.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r50_fpn_mstrain_3x_coco/cascade_mask_rcnn_r50_fpn_mstrain_3x_coco_20210628_164719-5bdc3824.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r50_fpn_mstrain_3x_coco/cascade_mask_rcnn_r50_fpn_mstrain_3x_coco_20210628_164719.log.json)               |

|    R-101-FPN    |  caffe  |   3x    |   7.7    |                |  45.4  |  39.5   | [config](./cascade-mask-rcnn_r101-caffe_fpn_ms-3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r101_caffe_fpn_mstrain_3x_coco/cascade_mask_rcnn_r101_caffe_fpn_mstrain_3x_coco_20210707_002620-a5bd2389.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r101_caffe_fpn_mstrain_3x_coco/cascade_mask_rcnn_r101_caffe_fpn_mstrain_3x_coco_20210707_002620.log.json) |

|    R-101-FPN    | pytorch |   3x    |   7.8    |                |  45.5  |  39.6   |    [config](./cascade-mask-rcnn_r101_fpn_ms-3x_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r101_fpn_mstrain_3x_coco/cascade_mask_rcnn_r101_fpn_mstrain_3x_coco_20210628_165236-51a2d363.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r101_fpn_mstrain_3x_coco/cascade_mask_rcnn_r101_fpn_mstrain_3x_coco_20210628_165236.log.json)             |

| X-101-32x4d-FPN | pytorch |   3x    |   9.0    |                |  46.3  |  40.1   | [config](./cascade-mask-rcnn_x101-32x4d_fpn_ms-3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_x101_32x4d_fpn_mstrain_3x_coco/cascade_mask_rcnn_x101_32x4d_fpn_mstrain_3x_coco_20210706_225234-40773067.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_x101_32x4d_fpn_mstrain_3x_coco/cascade_mask_rcnn_x101_32x4d_fpn_mstrain_3x_coco_20210706_225234.log.json) |

| X-101-32x8d-FPN | pytorch |   3x    |   12.1   |                |  46.1  |  39.9   | [config](./cascade-mask-rcnn_x101-32x8d_fpn_ms-3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_x101_32x8d_fpn_mstrain_3x_coco/cascade_mask_rcnn_x101_32x8d_fpn_mstrain_3x_coco_20210719_180640-9ff7e76f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_x101_32x8d_fpn_mstrain_3x_coco/cascade_mask_rcnn_x101_32x8d_fpn_mstrain_3x_coco_20210719_180640.log.json) |

| X-101-64x4d-FPN | pytorch |   3x    |   12.0   |                |  46.6  |  40.3   | [config](./cascade-mask-rcnn_x101-64x4d_fpn_ms-3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_x101_64x4d_fpn_mstrain_3x_coco/cascade_mask_rcnn_x101_64x4d_fpn_mstrain_3x_coco_20210719_210311-d3e64ba0.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_x101_64x4d_fpn_mstrain_3x_coco/cascade_mask_rcnn_x101_64x4d_fpn_mstrain_3x_coco_20210719_210311.log.json) |

```latex
@article{Cai_2019,
   title={Cascade R-CNN: High Quality Object Detection and Instance Segmentation},
   ISSN={1939-3539},
   url={http://dx.doi.org/10.1109/tpami.2019.2956516},
   DOI={10.1109/tpami.2019.2956516},
   author={Cai, Zhaowei and Vasconcelos, Nuno},
   year={2019},

publisher={Institute of Electrical and Electronics Engineers (IEEE)},
   pages={1–1}
}
```

# 模型部署

[MMDeploy](https://github.com/open-mmlab/mmdeploy) 是 OpenMMLab 的部署仓库，负责包括 MMPretrain、MMDetection 等在内的各算法库的部署工作。
你可以从[这里](https://mmdeploy.readthedocs.io/zh_CN/1.x/04-supported-codebases/mmdet.html)获取 MMDeploy 对 MMDetection 部署支持的最新文档。

本文的结构如下：

-

## 安装

请参考[此处](https://mmdetection.readthedocs.io/en/latest/get_started.html)安装 mmdet。然后，按照[说明](https://mmdeploy.readthedocs.io/zh_CN/1.x/get_started.html#mmdeploy)安装 mmdeploy。

```{note}
```

## 模型转换

```{note}
```

## 模型转换

假设在安装步骤中，mmdetection 和 mmdeploy 代码库在同级目录下，并且当前的工作目录为 mmdetection 的根目录，那么以 [Faster R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/faster_rcnn/faster-rcnn_r50_fpn_1x_coco.py) 模型为例，你可以从[此处](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth)下载对应的 checkpoint，并使用以下代码将之转换为 onnx 模型：

```python

```

转换的关键之一是使用正确的配置文件。项目中已内置了各后端部署[配置文件](https://github.com/open-mmlab/mmdeploy/tree/1.x/configs/mmdet)。
文件的命名模式是：

```
```

其中：

- **{task}:** mmdet 中的任务

  mmdet 任务有2种：物体检测（detection）、实例分割（instance-seg）。例如，`RetinaNet`、`Faster R-CNN`、`DETR`等属于前者。`Mask R-CNN`、`SOLO`等属于后者。更多`模型-任务`的划分，请参考章节[模型支持列表](#模型支持列表)。

  **请务必**使用 `detection/detection_*.py` 转换检测模型，使用 `instance-seg/instance-seg_*.py` 转换实例分割模型。

**请务必**使用 `detection/detection_*.py` 转换检测模型，使用 `instance-seg/instance-seg_*.py` 转换实例分割模型。

在上例中，你也可以把 `Faster R-CNN` 转为其他后端模型。比如使用`detection_tensorrt-fp16_dynamic-320x320-1344x1344.py`，把模型转为 tensorrt-fp16 模型。

```{tip}
```

## 模型规范

上例中的`mmdeploy_models/mmdet/onnx`，结构如下：

```
```

重要的是：

## 模型推理

## 后端模型推理

```python

## 模型推理

## 后端模型推理

```python

```

## SDK 模型推理

```python
import cv2

        continue

```

continue

```

你可以参考[样例](https://github.com/open-mmlab/mmdeploy/tree/1.x/demo)学习其他语言接口的使用方法。

## 模型支持列表

请参考[这里](https://mmdeploy.readthedocs.io/zh_CN/1.x/04-supported-codebases/mmdet.html#id6)

# Visual-Attention-Network

> [Visual Attention Network](https://arxiv.org/abs/2202.09741)

While originally designed for natural language processing (NLP) tasks, the self-attention mechanism has recently taken various computer vision areas by storm. However, the 2D nature of images brings three challenges for applying self-attention in computer vision. (1) Treating images as 1D sequences neglects their 2D structures. (2) The quadratic complexity is too expensive for high-resolution images. (3) It only captures spatial adaptability but ignores channel adaptability. In this paper, we propose a novel large kernel attention (LKA) module to enable self-adaptive and long-range correlations in self-attention while avoiding the above issues. We further introduce a novel neural network based on LKA, namely Visual Attention Network (VAN). While extremely simple and efficient, VAN outperforms the state-of-the-art vision transformers and convolutional neural networks with a large margin in extensive experiments, including image classification, object detection, semantic segmentation, instance segmentation, etc.

<img src="https://user-images.githubusercontent.com/24734142/157409411-2f622ba7-553c-4702-91be-eba03f9ea04f.png" width="80%"/>
</div>

```python

predict = inference_model('van-tiny_3rdparty_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('van-tiny_3rdparty_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/van/van-tiny_8xb128_in1k.py https://download.openmmlab.com/mmclassification/v0/van/van-tiny_8xb128_in1k_20220501-385941af.pth
```

| Model                       |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |               Config               |                                          Download                                          |
| :-------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :--------------------------------: | :----------------------------------------------------------------------------------------: |

| `van-tiny_3rdparty_in1k`\*  | From scratch |    4.11    |   0.88    |   75.41   |   93.02   | [config](van-tiny_8xb128_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/van/van-tiny_8xb128_in1k_20220501-385941af.pth) |
| `van-small_3rdparty_in1k`\* | From scratch |   13.86    |   2.52    |   81.01   |   95.63   | [config](van-small_8xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/van/van-small_8xb128_in1k_20220501-17bc91aa.pth) |
| `van-base_3rdparty_in1k`\*  | From scratch |   26.58    |   5.03    |   82.80   |   96.21   | [config](van-base_8xb128_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/van/van-base_8xb128_in1k_20220501-6a4cc31b.pth) |
| `van-large_3rdparty_in1k`\* | From scratch |   44.77    |   8.99    |   83.86   |   96.73   | [config](van-large_8xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/van/van-large_8xb128_in1k_20220501-f212ba21.pth) |

*Models with * are converted from the [official repo](https://github.com/Visual-Attention-Network/VAN-Classification). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
}
```

# StyleGANv3 (NeurIPS'2021)

> [Alias-Free Generative Adversarial Networks](https://nvlabs-fi-cdn.nvidia.com/stylegan3/stylegan3-paper.pdf)

We observe that despite their hierarchical convolutional nature, the synthesis
process of typical generative adversarial networks depends on absolute pixel coordinates in an unhealthy manner. This manifests itself as, e.g., detail appearing to
be glued to image coordinates instead of the surfaces of depicted objects. We trace
the root cause to careless signal processing that causes aliasing in the generator
network. Interpreting all signals in the network as continuous, we derive generally
applicable, small architectural changes that guarantee that unwanted information
cannot leak into the hierarchical synthesis process. The resulting networks match
the FID of StyleGAN2 but differ dramatically in their internal representations, and

they are fully equivariant to translation and rotation even at subpixel scales. Our
results pave the way for generative models better suited for video and animation.

<img src="https://user-images.githubusercontent.com/22982797/150353023-8f7eeaea-8783-4ed4-98d5-67a226e00cff.png"/>
</div>

  <b> Results (compressed) from StyleGAN3 config-T converted by mmagic</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/22982797/150450502-c182834f-796f-4397-bd38-df1efe4a8a47.png" width="800"/>
</div>

We perform experiments on StyleGANv3 paper settings and also experimental settings.
For user convenience, we also offer the converted version of official weights.

### Paper Settings

</div>

|                                     Model                                     |      Dataset      |  Iter  |      FID50k       |                                      Download                                       |
| :---------------------------------------------------------------------------: | :---------------: | :----: | :---------------: | :---------------------------------------------------------------------------------: |

|   [stylegan3-t](./stylegan3-t_gamma32.8_8xb4-fp16-noaug_ffhq-1024x1024.py)    |  ffhq 1024x1024   | 490000 | 3.37<sup>\*</sup> | [ckpt](https://download.openmmlab.com/mmediting/stylegan3/stylegan3_t_noaug_fp16_gamma32.8_ffhq_1024_b4x8_best_fid_iter_490000_20220401_120733-4ff83434.pth) \| [log](https://download.openmmlab.com/mmediting/stylegan3/stylegan3_t_noaug_fp16_gamma32.8_ffhq_1024_b4x8_20220322_090417.log.json) |
| [stylegan3-t-ada](./stylegan3-t_ada-gamma6.6_8xb4-fp16_metfaces-1024x1024.py) | metface 1024x1024 | 130000 |       15.09       | [ckpt](https://download.openmmlab.com/mmediting/stylegan3/stylegan3_t_ada_fp16_gamma6.6_metfaces_1024_b4x8_best_fid_iter_130000_20220401_115101-f2ef498e.pth) \| [log](https://download.openmmlab.com/mmediting/stylegan3/stylegan3_t_ada_fp16_gamma6.6_metfaces_1024_b4x8_20220328_142211.log.json) |

### Experimental Settings

|                                     Model                                     |    Dataset     |  Iter  | FID50k |                                             Download                                              |
| :---------------------------------------------------------------------------: | :------------: | :----: | :----: | :-----------------------------------------------------------------------------------------------: |
|     [stylegan3-t](./stylegan3-t_gamma2.0_8xb4-fp16-noaug_ffhq-256x256.py)     |  ffhq 256x256  | 740000 |  4.51  | [ckpt](https://download.openmmlab.com/mmediting/stylegan3/stylegan3_t_noaug_fp16_gamma2.0_ffhq_256_b4x8_best_fid_iter_740000_20220401_122456-730e1fba.pth) \| [log](https://download.openmmlab.com/mmediting/stylegan3/stylegan3_t_noaug_fp16_gamma2.0_ffhq_256_b4x8_20220323_144815.log.json) |

| [stylegan3-r-ada](./stylegan3-r_ada-gamma3.3_8xb4-fp16_metfaces-1024x1024.py) | ffhq 1024x1024 |   -    |   -    |                                            [ckpt](<>)                                             |

### Converted Weights

|                                 Model                                  |    Dataset     |     Comment     | FID50k | EQ-T  | EQ-R  |                                       Download                                        |
| :--------------------------------------------------------------------: | :------------: | :-------------: | :----: | :---: | :---: | :-----------------------------------------------------------------------------------: |
|  [stylegan3-t](./stylegan3-t_cvt-official-rgb_8xb4_ffhqu-256x256.py)   | ffhqu 256x256  | official weight |  4.62  | 63.01 | 13.12 | [ckpt](https://download.openmmlab.com/mmediting/stylegan3/stylegan3_t_ffhqu_256_b4x8_cvt_official_rgb_20220329_235046-153df4c8.pth) |

|  [stylegan3-t](./stylegan3-t_cvt-official-rgb_8xb4_afhqv2-512x512.py)  | afhqv2 512x512 | official weight |  4.04  | 60.15 | 13.51 | [ckpt](https://download.openmmlab.com/mmediting/stylegan3/stylegan3_t_afhqv2_512_b4x8_cvt_official_rgb_20220329_235017-ee6b037a.pth) |
|  [stylegan3-t](./stylegan3-t_cvt-official-rgb_8xb4_ffhq-1024x1024.py)  | ffhq 1024x1024 | official weight |  2.79  | 61.21 | 13.82 | [ckpt](https://download.openmmlab.com/mmediting/stylegan3/stylegan3_t_ffhq_1024_b4x8_cvt_official_rgb_20220329_235113-db6c6580.pth) |
|  [stylegan3-r](./stylegan3-r_cvt-official-rgb_8xb4_ffhqu-256x256.py)   | ffhqu 256x256  | official weight |  4.50  | 66.65 | 40.48 | [ckpt](https://download.openmmlab.com/mmediting/stylegan3/stylegan3_r_ffhqu_256_b4x8_cvt_official_rgb_20220329_234909-4521d963.pth) |

| [stylegan3-r](./stylegan3-r_cvt-official-rgb_8xb4x8_afhqv2-512x512.py) | afhqv2 512x512 | official weight |  4.40  | 64.89 | 40.34 | [ckpt](https://download.openmmlab.com/mmediting/stylegan3/stylegan3_r_afhqv2_512_b4x8_cvt_official_rgb_20220329_234829-f2eaca72.pth) |
|  [stylegan3-r](./stylegan3-r_cvt-official-rgb_8xb4_ffhq-1024x1024.py)  | ffhq 1024x1024 | official weight |  3.07  | 64.76 | 46.62 | [ckpt](https://download.openmmlab.com/mmediting/stylegan3/stylegan3_r_ffhq_1024_b4x8_cvt_official_rgb_20220329_234933-ac0500a1.pth) |

## Interpolation

We provide a tool to generate video by walking through GAN's latent space.
Run this command to get the following video.

```bash
python apps/interpolate_sample.py configs/styleganv3/stylegan3_t_afhqv2_512_b4x8_official.py https://download.openmmlab.com/mmediting/stylegan3/stylegan3_t_afhqv2_512_b4x8_cvt_official.pkl --export-video --samples-path work_dirs/demos/ --endpoint 6 --interval 60 --space z --seed 2022 --sample-cfg truncation=0.8

```

https://user-images.githubusercontent.com/22982797/151506918-83da9ee3-0d63-4c5b-ad53-a41562b92075.mp4

## Equivarience Visualization && Evaluation

We also provide a tool to visualize the equivarience properties for StyleGAN3.
Run these commands to get the results below.

```bash
python tools/utils/equivariance_viz.py configs/styleganv3/stylegan3_r_ffhqu_256_b4x8_official.py https://download.openmmlab.com/mmediting/stylegan3/stylegan3_r_ffhqu_256_b4x8_cvt_official.pkl --translate_max 0.5 --transform rotate --seed 5432

python tools/utils/equivariance_viz.py configs/styleganv3/stylegan3_r_ffhqu_256_b4x8_official.py https://download.openmmlab.com/mmediting/stylegan3/stylegan3_r_ffhqu_256_b4x8_cvt_official.pkl --translate_max 0.25 --transform x_t --seed 5432

python tools/utils/equivariance_viz.py configs/styleganv3/stylegan3_r_ffhqu_256_b4x8_official.py https://download.openmmlab.com/mmediting/stylegan3/stylegan3_r_ffhqu_256_b4x8_cvt_official.pkl --translate_max 0.25 --transform y_t --seed 5432
```

https://user-images.githubusercontent.com/22982797/151504902-f3cbfef5-9014-4607-bbe1-deaf48ec6d55.mp4

https://user-images.githubusercontent.com/22982797/151504973-b96e1639-861d-434b-9d7c-411ebd4a653f.mp4

https://user-images.githubusercontent.com/22982797/151505099-cde4999e-aab1-42d4-a458-3bb069db3d32.mp4

If you want to get EQ-Metric for StyleGAN3, just add following codes into config.

```python
metrics = dict(
    eqv=dict(
        num_images=50000,
            compute_eqt_int=True, compute_eqt_frac=True, compute_eqr=True)))
```

And we highly recommend you to use [slurm_test.sh](../../tools/slurm_test.sh) script to accelerate evaluation time.

```latex

```

```latex
@inproceedings{Karras2021,
  author = {Tero Karras and Miika Aittala and Samuli Laine and Erik H\"ark\"onen and Janne Hellsten and Jaakko Lehtinen and Timo Aila},
  title = {Alias-Free Generative Adversarial Networks},
  booktitle = {Proc. NeurIPS},
}
```

}
```

<details>

```bibtex
```

Results on Human-Art validation dataset with detector having human AP of 56.2 on Human-Art validation dataset

| [rtmpose-t-coco](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-t_8xb256-420e_coco-256x192.py) |  256x192   | 0.161 |      0.283      |      0.154      | 0.221 |      0.373      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-tiny_simcc-coco_pt-aic-coco_420e-256x192-e613ba3f_20230127.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-tiny_simcc-coco_pt-aic-coco_420e-256x192-e613ba3f_20230127.json) |

| [rtmpose-t-humanart-coco](/configs/body_2d_keypoint/rtmpose/humanart/rtmpose-t_8xb256-420e_humanart-256x192.py) |  256x192   | 0.249 |      0.395      |      0.256      | 0.323 |      0.485      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-t_8xb256-420e_humanart-256x192-60b68c98_20230612.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-t_8xb256-420e_humanart-256x192-60b68c98_20230612.json) |
| [rtmpose-s-coco](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-s_8xb256-420e_coco-256x192.py) |  256x192   | 0.199 |      0.328      |      0.198      | 0.261 |      0.418      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_simcc-coco_pt-aic-coco_420e-256x192-8edcf0d7_20230127.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_simcc-coco_pt-aic-coco_420e-256x192-8edcf0d7_20230127.json) |

| [rtmpose-s-humanart-coco](/configs/body_2d_keypoint/rtmpose/humanart/rtmpose-s_8xb256-420e_humanart-256x192.py) |  256x192   | 0.311 |      0.462      |      0.323      | 0.381 |      0.540      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_8xb256-420e_humanart-256x192-5a3ac943_20230611.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_8xb256-420e_humanart-256x192-5a3ac943_20230611.json) |
| [rtmpose-m-coco](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-m_8xb256-420e_coco-256x192.py) |  256x192   | 0.239 |      0.372      |      0.243      | 0.302 |      0.455      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-coco_pt-aic-coco_420e-256x192-d8dd5ca4_20230127.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-coco_pt-aic-coco_420e-256x192-d8dd5ca4_20230127.json) |

| [rtmpose-m-humanart-coco](/configs/body_2d_keypoint/rtmpose/humanart/rtmpose-m_8xb256-420e_humanart-256x192.py) |  256x192   | 0.355 |      0.503      |      0.377      | 0.417 |      0.568      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_8xb256-420e_humanart-256x192-8430627b_20230611.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_8xb256-420e_humanart-256x192-8430627b_20230611.json) |
| [rtmpose-l-coco](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-l_8xb256-420e_coco-256x192.py) |  256x192   | 0.260 |      0.393      |      0.267      | 0.323 |      0.472      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-coco_pt-aic-coco_420e-256x192-1352a4d2_20230127.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-coco_pt-aic-coco_420e-256x192-1352a4d2_20230127.json) |

| [rtmpose-l-humanart-coco](/configs/body_2d_keypoint/rtmpose/humanart/rtmpose-l_8xb256-420e_humanart-256x192.py) |  256x192   | 0.378 |      0.521      |      0.399      | 0.442 |      0.584      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_8xb256-420e_humanart-256x192-389f2cb0_20230611.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_8xb256-420e_humanart-256x192-389f2cb0_20230611.json) |

Results on Human-Art validation dataset with ground-truth bounding-box

| [rtmpose-t-coco](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-t_8xb256-420e_coco-256x192.py) |  256x192   | 0.444 |      0.725      |      0.453      | 0.488 |      0.750      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-tiny_simcc-coco_pt-aic-coco_420e-256x192-e613ba3f_20230127.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-tiny_simcc-coco_pt-aic-coco_420e-256x192-e613ba3f_20230127.json) |
| [rtmpose-t-humanart-coco](/configs/body_2d_keypoint/rtmpose/humanart/rtmpose-t_8xb256-420e_humanart-256x192.py) |  256x192   | 0.655 |      0.872      |      0.720      | 0.693 |      0.890      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-t_8xb256-420e_humanart-256x192-60b68c98_20230612.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-t_8xb256-420e_humanart-256x192-60b68c98_20230612.json) |

| [rtmpose-s-coco](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-s_8xb256-420e_coco-256x192.py) |  256x192   | 0.480 |      0.739      |      0.498      | 0.521 |      0.763      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_simcc-coco_pt-aic-coco_420e-256x192-8edcf0d7_20230127.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_simcc-coco_pt-aic-coco_420e-256x192-8edcf0d7_20230127.json) |
| [rtmpose-s-humanart-coco](/configs/body_2d_keypoint/rtmpose/humanart/rtmpose-s_8xb256-420e_humanart-256x192.py) |  256x192   | 0.698 |      0.893      |      0.768      | 0.732 |      0.903      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_8xb256-420e_humanart-256x192-5a3ac943_20230611.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_8xb256-420e_humanart-256x192-5a3ac943_20230611.json) |

| [rtmpose-m-coco](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-m_8xb256-420e_coco-256x192.py) |  256x192   | 0.532 |      0.765      |      0.563      | 0.571 |      0.789      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-coco_pt-aic-coco_420e-256x192-d8dd5ca4_20230127.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-coco_pt-aic-coco_420e-256x192-d8dd5ca4_20230127.json) |
| [rtmpose-m-humanart-coco](/configs/body_2d_keypoint/rtmpose/humanart/rtmpose-m_8xb256-420e_humanart-256x192.py) |  256x192   | 0.728 |      0.895      |      0.791      | 0.759 |      0.906      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_8xb256-420e_humanart-256x192-8430627b_20230611.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_8xb256-420e_humanart-256x192-8430627b_20230611.json) |

| [rtmpose-l-coco](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-l_8xb256-420e_coco-256x192.py) |  256x192   | 0.564 |      0.789      |      0.602      | 0.599 |      0.808      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-coco_pt-aic-coco_420e-256x192-1352a4d2_20230127.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-coco_pt-aic-coco_420e-256x192-1352a4d2_20230127.json) |
| [rtmpose-l-humanart-coco](/configs/body_2d_keypoint/rtmpose/humanart/rtmpose-l_8xb256-420e_humanart-256x192.py) |  256x192   | 0.753 |      0.905      |      0.812      | 0.783 |      0.915      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_8xb256-420e_humanart-256x192-389f2cb0_20230611.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_8xb256-420e_humanart-256x192-389f2cb0_20230611.json) |

| [rtmpose-t-coco](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-t_8xb256-420e_coco-256x192.py) |  256x192   | 0.682 |      0.883      |      0.759      | 0.736 |      0.920      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-tiny_simcc-coco_pt-aic-coco_420e-256x192-e613ba3f_20230127.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-tiny_simcc-coco_pt-aic-coco_420e-256x192-e613ba3f_20230127.json) |
| [rtmpose-t-humanart-coco](/configs/body_2d_keypoint/rtmpose/humanart/rtmpose-t_8xb256-420e_humanart-256x192.py) |  256x192   | 0.665 |      0.875      |      0.739      | 0.721 |      0.916      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-t_8xb256-420e_humanart-256x192-60b68c98_20230612.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-t_8xb256-420e_humanart-256x192-60b68c98_20230612.json) |

| [rtmpose-s-coco](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-s_8xb256-420e_coco-256x192.py) |  256x192   | 0.716 |      0.892      |      0.789      | 0.768 |      0.929      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_simcc-coco_pt-aic-coco_420e-256x192-8edcf0d7_20230127.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_simcc-coco_pt-aic-coco_420e-256x192-8edcf0d7_20230127.json) |
| [rtmpose-s-humanart-coco](/configs/body_2d_keypoint/rtmpose/humanart/rtmpose-s_8xb256-420e_humanart-256x192.py) |  256x192   | 0.706 |      0.888      |      0.780      | 0.759 |      0.928      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_8xb256-420e_humanart-256x192-5a3ac943_20230611.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_8xb256-420e_humanart-256x192-5a3ac943_20230611.json) |

| [rtmpose-m-coco](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-m_8xb256-420e_coco-256x192.py) |  256x192   | 0.746 |      0.899      |      0.817      | 0.795 |      0.935      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-coco_pt-aic-coco_420e-256x192-d8dd5ca4_20230127.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-coco_pt-aic-coco_420e-256x192-d8dd5ca4_20230127.json) |
| [rtmpose-m-humanart-coco](/configs/body_2d_keypoint/rtmpose/humanart/rtmpose-m_8xb256-420e_humanart-256x192.py) |  256x192   | 0.725 |      0.892      |      0.795      | 0.775 |      0.929      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_8xb256-420e_humanart-256x192-8430627b_20230611.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_8xb256-420e_humanart-256x192-8430627b_20230611.json) |

| [rtmpose-l-coco](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-l_8xb256-420e_coco-256x192.py) |  256x192   | 0.758 |      0.906      |      0.826      | 0.806 |      0.942      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-coco_pt-aic-coco_420e-256x192-1352a4d2_20230127.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-coco_pt-aic-coco_420e-256x192-1352a4d2_20230127.json) |
| [rtmpose-l-humanart-coco](/configs/body_2d_keypoint/rtmpose/humanart/rtmpose-l_8xb256-420e_humanart-256x192.py) |  256x192   | 0.748 |      0.901      |      0.816      | 0.796 |      0.938      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_8xb256-420e_humanart-256x192-389f2cb0_20230611.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_8xb256-420e_humanart-256x192-389f2cb0_20230611.json) |

Results on COCO val2017 with ground-truth bounding box

| [rtmpose-t-humanart-coco](/configs/body_2d_keypoint/rtmpose/humanart/rtmpose-t_8xb256-420e_humanart-256x192.py) |  256x192   | 0.679 |      0.895      |      0.755      | 0.710 |      0.907      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-t_8xb256-420e_humanart-256x192-60b68c98_20230612.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-t_8xb256-420e_humanart-256x192-60b68c98_20230612.json) |

| [rtmpose-s-humanart-coco](/configs/body_2d_keypoint/rtmpose/humanart/rtmpose-s_8xb256-420e_humanart-256x192.py) |  256x192   | 0.725 |      0.916      |      0.798      | 0.753 |      0.925      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_8xb256-420e_humanart-256x192-5a3ac943_20230611.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_8xb256-420e_humanart-256x192-5a3ac943_20230611.json) |
| [rtmpose-m-humanart-coco](/configs/body_2d_keypoint/rtmpose/humanart/rtmpose-m_8xb256-420e_humanart-256x192.py) |  256x192   | 0.744 |      0.916      |      0.818      | 0.770 |      0.930      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_8xb256-420e_humanart-256x192-8430627b_20230611.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_8xb256-420e_humanart-256x192-8430627b_20230611.json) |

| [rtmpose-l-humanart-coco](/configs/body_2d_keypoint/rtmpose/humanart/rtmpose-l_8xb256-420e_humanart-256x192.py) |  256x192   | 0.770 |      0.927      |      0.840      | 0.794 |      0.939      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_8xb256-420e_humanart-256x192-389f2cb0_20230611.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_8xb256-420e_humanart-256x192-389f2cb0_20230611.json) |

# Panoptic Studio: A Massively Multiview System for Social Motion Capture

<details>
<summary align="right"><a href="https://openaccess.thecvf.com/content_iccv_2015/html/Joo_Panoptic_Studio_A_ICCV_2015_paper.html">CMU Panoptic (ICCV'2015)</a></summary>

```bibtex
@Article = {joo_iccv_2015,
author = {Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh},
title = {Panoptic Studio: A Massively Multiview System for Social Motion Capture},
booktitle = {ICCV},
year = {2015}
}
```

# GroupFisher Pruning for RTMPose

# Description

We try to apply a pruning algorithm to RTMPose models. In detail, we prune a RTMPose model to a smaller size as the same as a smaller RTMPose model, like pruning RTMPose-S to the size of RTMPose-T.
The expriments show that the pruned model have better performance(AP) than the RTMPose model with the similar size and inference speed.

Concretly, we select the RTMPose-S as the base model and prune it to the size of RTMPose-T, and use GroupFisher pruning algorithm which is able to determine the pruning structure automatically.
Furthermore, we provide two version of the pruned models including only using coco and using both of coco and ai-challenge datasets.

# Results and Models

We have three steps to apply GroupFisher to your model, including Prune, Finetune, Deploy.

Note: please use torch>=1.12, as we need fxtracer to parse the models automatically.

### Prune

```bash

### Prune

```bash
```

In the pruning config file. You have to fill some args as below.

```python
"""

importance, including ['flops','act']. The former uses flops, while the

"""
```

After the pruning process, you will get a checkpoint of the pruned model named flops\_{target_flop_ratio}.pth in your workdir.

```bash
```

There are also some args for you to fill in the config file as below.

```python
"""

```python
"""
"""
```

After finetuning, except a checkpoint of the best model, there is also a fix_subnet.json, which records the pruned model structure. It will be used when deploying.

### Test

```bash
```

### Deploy

For a pruned model, you only need to use the pruning deploy config to instead the pretrain config to deploy the pruned version of your model. If you are not familiar with mmdeploy, it's recommended to refer to [MMDeploy document](https://mmdeploy.readthedocs.io/en/latest/02-how-to-run/convert_model.html).

```bash

```bash
```

The deploy config has some args as below:

```python
"""
"""
```

The divisor is important for the actual inference speed, and we suggest you to test it in \[1,2,4,8,16,32\] to find the fastest divisor.

# TensorRT 支持情况

## 安装

### 安装TensorRT

请按照[安装指南](https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html#installing)安装TensorRT8。

**注意**:

- 此版本不支持`pip Wheel File Installation`。

- 我们强烈建议通过[tar包](https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html#installing-tar)的方式安装TensorRT。

- 安装完成后，最好通过以下方式将TensorRT环境变量添加到bashrc:

  ```bash
  cd ${TENSORRT_DIR} # 进入TensorRT根目录
  echo '# set env for TensorRT' >> ~/.bashrc
  echo "export TENSORRT_DIR=${TENSORRT_DIR}" >> ~/.bashrc
  echo 'export LD_LIBRARY_PATH=$TENSORRT_DIR/lib:$TENSORRT_DIR' >> ~/.bashrc
  source ~/.bashrc
  ```

### 构建自定义算子

OpenMMLab中创建了一些自定义算子来支持模型，自定义算子可以如下构建:

```bash
cd ${MMDEPLOY_DIR} # 进入TensorRT根目录
cmake -DMMDEPLOY_TARGET_BACKENDS=trt ..
```

如果你没有在默认路径下安装TensorRT，请在CMake中添加`-DTENSORRT_DIR`标志。

```bash
 cmake -DMMDEPLOY_TARGET_BACKENDS=trt -DTENSORRT_DIR=${TENSORRT_DIR} ..
 make -j$(nproc) && make install
```

## 转换模型

```

```bash
```

## 转换模型

请遵循[如何转换模型](../02-how-to-run/convert_model.md)中的教程。**注意**设备必须是`cuda` 设备。

### Int8 支持

由于TensorRT支持INT8模式，因此可以提供自定义数据集配置来校准模型。MMDetection的示例如下:

```python
# calibration_dataset.py

# 数据集设置，格式与OpenMMLab中的代码库相同
dataset_type = 'CalibrationDataset'
data_root = 'calibration/dataset/root'
    dict(

dict(type='RandomFlip'),
        ])
]
    val=dict(
        ann_file=data_root + 'val_annotations.json',
        ann_file=data_root + 'test_annotations.json',
```

使用此校准数据集转换您的模型:

```python
    ...
    --calib-dataset-cfg calibration_dataset.py
```

如果没有提供校准数据集，则使用模型配置中的数据集进行校准。

## FAQs

- 错误 `Cannot found TensorRT headers`或`Cannot found TensorRT libs`

  可以尝试在cmake时使用`-DTENSORRT_DIR`标志:

  ```bash
  cmake -DBUILD_TENSORRT_OPS=ON -DTENSORRT_DIR=${TENSORRT_DIR} ..
  ```

可以尝试在cmake时使用`-DTENSORRT_DIR`标志:

  ```bash
  ```

  请确保 `${TENSORRT_DIR}`中有库和头文件。

- 错误 `error: parameter check failed at: engine.cpp::setBindingDimensions::1046, condition: profileMinDims.d[i] <= dimensions.d[i]`

  在部署配置中有一个输入形状的限制:

  ```python
      # other configs
      model_inputs=[
          dict(
              input_shapes=dict(
                  input=dict(
                      min_shape=[1, 3, 320, 320],
                      opt_shape=[1, 3, 800, 1344],
                      max_shape=[1, 3, 1344, 1344])))
      ])
  ```

  `input` 张量的形状必须限制在`input_shapes["input"]["min_shape"]`和`input_shapes["input"]["max_shape"]`之间。

- 错误 `error: [TensorRT] INTERNAL ERROR: Assertion failed: cublasStatus == CUBLAS_STATUS_SUCCESS`

`input` 张量的形状必须限制在`input_shapes["input"]["min_shape"]`和`input_shapes["input"]["max_shape"]`之间。

  TRT 7.2.1切换到使用cuBLASLt(以前是cuBLAS)。cuBLASLt是SM版本>= 7.0的默认选择。但是，您可能需要CUDA-10.2补丁1(2020年8月26日发布)来解决一些cuBLASLt问题。如果不想升级，另一个选择是使用新的TacticSource API并禁用cuBLASLt策略。

  请阅读[本文](https://forums.developer.nvidia.com/t/matrixmultiply-failed-on-tensorrt-7-2-1/158187/4)了解详情。

- 在Jetson上安装mmdeploy

  我们在[这里](../01-how-to-build/jetsons.md)提供了一个Jetsons入门教程。

We use python files as configs, incorporate modular and inheritance design into our config system, which is convenient to conduct various experiments.
You can find all the provided configs under `$MMAction2/configs`. If you wish to inspect the config file,
you may run `python tools/analysis_tools/print_config.py /PATH/TO/CONFIG` to see the complete config.

- [Learn about Configs](#learn-about-configs)
  - [Modify config through script arguments](#modify-config-through-script-arguments)
  - [Config File Structure](#config-file-structure)
  - [Config File Naming Convention](#config-file-naming-convention)
    - [Config System for Action Recognition](#config-system-for-action-recognition)
    - [Config System for Spatio-Temporal Action Detection](#config-system-for-spatio-temporal-action-detection)
    - [Config System for Action localization](#config-system-for-action-localization)

- [Config System for Action localization](#config-system-for-action-localization)

- Update config keys of dict.

  Some config dicts are composed as a list in your config. For example, the training pipeline `train_pipeline` is normally a list
  e.g. `[dict(type='SampleFrames'), ...]`. If you want to change `'SampleFrames'` to `'DenseSampleFrames'` in the pipeline,
  you may specify `--cfg-options train_pipeline.0.type=DenseSampleFrames`.

you may specify `--cfg-options train_pipeline.0.type=DenseSampleFrames`.

  change this key, you may specify `--cfg-options model.data_preprocessor.mean="[128,128,128]"`. Note that the quotation mark " is necessary to support list/tuple data types.

There are 3 basic component types under `configs/_base_`, models, schedules, default_runtime.
Many methods could be easily constructed with one of each like TSN, I3D, SlowOnly, etc.

For easy understanding, we recommend contributors to inherit from exiting methods.
For example, if some modification is made based on TSN, users may first inherit the basic TSN structure by specifying `_base_ = ../tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py`, then modify the necessary fields in the config files.

If you are building an entirely new method that does not share the structure with any of the existing methods, you may create a folder under `configs/TASK`.

We follow the style below to name config files. Contributors are advised to follow the same style. The config file names are divided into several parts. Logically, different parts are concatenated by underscores `'_'`, and settings in the same part are concatenated by dashes `'-'`.

```

```
{algorithm info}_{module info}_{training info}_{data info}.py
```

- `{algorithm info}`:
  - `{model}`: model type, e.g. `tsn`, `i3d`, `swin`, `vit`, etc.
  - `[model setting]`: specific setting for some models, e.g. `base`, `p16`, `w877`, etc.
- `{module info}`:
  - `[pretained info]`: pretrained information, e.g. `kinetics400-pretrained`, `in1k-pre`, etc.
  - `{backbone}`: backbone type. e.g. `r50` (ResNet-50), etc.
  - `[backbone setting]`: specific setting for some backbones, e.g. `nl-dot-product`, `bnfrozen`, `nopool`, etc.
- `{training info}`:
  - `{gpu x batch_per_gpu]}`: GPUs and samples per GPU.
  - `{pipeline setting}`: frame sample setting, e.g. `dense`, `{clip_len}x{frame_interval}x{num_clips}`, `u48`, etc.
  - `{schedule}`: training schedule, e.g. `coslr-20e`.
- `{data info}`:
  - `{dataset}`: dataset name, e.g. `kinetics400`, `mmit`, etc.
  - `{modality}`: data modality, e.g. `rgb`, `flow`, `keypoint-2d`, etc.

### Config System for Action Recognition

We incorporate modular design into our config system,
which is convenient to conduct various experiments.

- An Example of TSN

  To help the users have a basic idea of a complete config structure and the modules in an action recognition system,
  we make brief comments on the config of TSN as the following.
  For more detailed usage and alternative for per parameter in each module, please refer to the API documentation.

  ```python
  # model settings
  model = dict(  # Config of the model
      type='Recognizer2D',  # Class name of the recognizer
      backbone=dict(  # Dict for backbone
          type='ResNet',  # Name of the backbone
          pretrained='torchvision://resnet50',  # The url/site of the pretrained model
          depth=50,  # Depth of ResNet model

type='ResNet',  # Name of the backbone
          norm_eval=False),  # Whether to set BN layers to eval mode when training
      cls_head=dict(  # Dict for classification head
          type='TSNHead',  # Name of classification head
          num_classes=400,  # Number of classes to be classified.
          in_channels=2048,  # The input channels of classification head.
          spatial_type='avg',  # Type of pooling in spatial dimension
          consensus=dict(type='AvgConsensus', dim=1),  # Config of consensus module
          dropout_ratio=0.4,  # Probability in dropout layer
          init_std=0.01, # Std value for linear layer initiation
          average_clips='prob'),  # Method to average multiple clip results
      data_preprocessor=dict(  # Dict for data preprocessor
          type='ActionDataPreprocessor',  # Name of data preprocessor

data_preprocessor=dict(  # Dict for data preprocessor
          mean=[123.675, 116.28, 103.53],  # Mean values of different channels to normalize
          std=[58.395, 57.12, 57.375],  # Std values of different channels to normalize
          format_shape='NCHW'),  # Final image shape format
      # model training and testing settings
      train_cfg=None,  # Config of training hyperparameters for TSN
      test_cfg=None)  # Config for testing hyperparameters for TSN.

  # dataset settings
  dataset_type = 'RawframeDataset'  # Type of dataset for training, validation and testing
  data_root = 'data/kinetics400/rawframes_train/'  # Root path to data for training
  data_root_val = 'data/kinetics400/rawframes_val/'  # Root path to data for validation and testing
  ann_file_train = 'data/kinetics400/kinetics400_train_list_rawframes.txt'  # Path to the annotation file for training

ann_file_train = 'data/kinetics400/kinetics400_train_list_rawframes.txt'  # Path to the annotation file for training
  ann_file_val = 'data/kinetics400/kinetics400_val_list_rawframes.txt'  # Path to the annotation file for validation
  ann_file_test = 'data/kinetics400/kinetics400_val_list_rawframes.txt'  # Path to the annotation file for testing

  train_pipeline = [  # Training data processing pipeline
      dict(  # Config of SampleFrames
          type='SampleFrames',  # Sample frames pipeline, sampling frames from video
          clip_len=1,  # Frames of each sampled output clip
          frame_interval=1,  # Temporal interval of adjacent sampled frames
          num_clips=3),  # Number of clips to be sampled
      dict(  # Config of RawFrameDecode
          type='RawFrameDecode'),  # Load and decode Frames pipeline, picking raw frames with given indices
      dict(  # Config of Resize
          type='Resize',  # Resize pipeline

type='RawFrameDecode'),  # Load and decode Frames pipeline, picking raw frames with given indices
          scale=(-1, 256)),  # The scale to resize images
      dict(  # Config of MultiScaleCrop
          type='MultiScaleCrop',  # Multi scale crop pipeline, cropping images with a list of randomly selected scales
          input_size=224,  # Input size of the network
          scales=(1, 0.875, 0.75, 0.66),  # Scales of width and height to be selected
          random_crop=False,  # Whether to randomly sample cropping bbox
          max_wh_scale_gap=1),  # Maximum gap of w and h scale levels
          scale=(224, 224),  # The scale to resize images
          keep_ratio=False),  # Whether to resize with changing the aspect ratio
      dict(  # Config of Flip
          type='Flip',  # Flip Pipeline

keep_ratio=False),  # Whether to resize with changing the aspect ratio
          flip_ratio=0.5),  # Probability of implementing flip
      dict(  # Config of FormatShape
          type='FormatShape',  # Format shape pipeline, Format final image shape to the given input_format
          input_format='NCHW'),  # Final image shape format
      dict(type='PackActionInputs')  # Config of PackActionInputs
  ]
  val_pipeline = [  # Validation data processing pipeline
          num_clips=3,  # Number of clips to be sampled
          test_mode=True),  # Whether to set test mode in sampling

num_clips=3,  # Number of clips to be sampled
      dict(  # Config of CenterCrop
          type='CenterCrop',  # Center crop pipeline, cropping the center area from images
          crop_size=224),  # The size to crop images
          type='Flip',  # Flip pipeline
          flip_ratio=0),  # Probability of implementing flip
  ]

input_format='NCHW'),  # Final image shape format
  ]
  test_pipeline = [  # Testing data processing pipeline
          num_clips=25,  # Number of clips to be sampled
      dict(  # Config of TenCrop
          type='TenCrop',  # Ten crop pipeline, cropping ten area from images

scale=(-1, 256)),  # The scale to resize images
  ]

  train_dataloader = dict(  # Config of train dataloader
      batch_size=32,  # Batch size of each single GPU during training
      num_workers=8,  # Workers to pre-fetch data for each single GPU during training
      persistent_workers=True,  # If `True`, the dataloader will not shut down the worker processes after an epoch end, which can accelerate training speed

persistent_workers=True,  # If `True`, the dataloader will not shut down the worker processes after an epoch end, which can accelerate training speed
      sampler=dict(
          type='DefaultSampler',  # DefaultSampler which supports both distributed and non-distributed training. Refer to https://github.com/open-mmlab/mmengine/blob/main/mmengine/dataset/sampler.py
          shuffle=True),  # Randomly shuffle the training data in each epoch
      dataset=dict(  # Config of train dataset
          ann_file=ann_file_train,  # Path of annotation file
          data_prefix=dict(img=data_root),  # Prefix of frame path
          pipeline=train_pipeline))
  val_dataloader = dict(  # Config of validation dataloader
      batch_size=1,  # Batch size of each single GPU during validation
      num_workers=8,  # Workers to pre-fetch data for each single GPU during validation

batch_size=1,  # Batch size of each single GPU during validation
      persistent_workers=True,  # If `True`, the dataloader will not shut down the worker processes after an epoch end
          type='DefaultSampler',
          shuffle=False),  # Not shuffle during validation and testing
      dataset=dict(  # Config of validation dataset
          ann_file=ann_file_val,  # Path of annotation file
          data_prefix=dict(img=data_root_val),  # Prefix of frame path
          test_mode=True))
  test_dataloader = dict(  # Config of test dataloader
      batch_size=32,  # Batch size of each single GPU during testing
      num_workers=8,  # Workers to pre-fetch data for each single GPU during testing

batch_size=32,  # Batch size of each single GPU during testing
      dataset=dict(  # Config of test dataset
          pipeline=test_pipeline,

  # evaluation settings
  val_evaluator = dict(type='AccMetric')  # Config of validation evaluator
  test_evaluator = val_evaluator  # Config of testing evaluator

  train_cfg = dict(  # Config of training loop
      type='EpochBasedTrainLoop',  # Name of training loop
      max_epochs=100,  # Total training epochs

train_cfg = dict(  # Config of training loop
      val_begin=1,  # The epoch that begins validating
      val_interval=1)  # Validation interval
  val_cfg = dict(  # Config of validation loop
      type='ValLoop')  # Name of validation loop
  test_cfg = dict( # Config of testing loop
      type='TestLoop')  # Name of testing loop

  # learning policy
  param_scheduler = [  # Parameter scheduler for updating optimizer parameters, support dict or list
      dict(type='MultiStepLR',  # Decays the learning rate once the number of epoch reaches one of the milestones
          begin=0,  # Step at which to start updating the learning rate
          end=100,  # Step at which to stop updating the learning rate
          by_epoch=True,  # Whether the scheduled learning rate is updated by epochs
          milestones=[40, 80],  # Steps to decay the learning rate

by_epoch=True,  # Whether the scheduled learning rate is updated by epochs
          gamma=0.1)]  # Multiplicative factor of learning rate decay

  # optimizer
  optim_wrapper = dict(  # Config of optimizer wrapper
      type='OptimWrapper',  # Name of optimizer wrapper, switch to AmpOptimWrapper to enable mixed precision training
      optimizer=dict(  # Config of optimizer. Support all kinds of optimizers in PyTorch. Refer to https://pytorch.org/docs/stable/optim.html#algorithms
          type='SGD',  # Name of optimizer
          lr=0.01,  # Learning rate
          momentum=0.9,  # Momentum factor
          weight_decay=0.0001),  # Weight decay
      clip_grad=dict(max_norm=40, norm_type=2))  # Config of gradient clip

  # runtime settings
  default_scope = 'mmaction'  # The default registry scope to find modules. Refer to https://mmengine.readthedocs.io/en/latest/tutorials/registry.html

  default_hooks = dict(  # Hooks to execute default actions like updating model parameters and saving checkpoints.
      runtime_info=dict(type='RuntimeInfoHook'),  # The hook to updates runtime information into message hub
      timer=dict(type='IterTimerHook'),  # The logger used to record time spent during iteration
      logger=dict(
          type='LoggerHook',  # The logger used to record logs during training/validation/testing phase
          interval=20,  # Interval to print the log
          ignore_last=False), # Ignore the log of last iterations in each epoch
      param_scheduler=dict(type='ParamSchedulerHook'),  # The hook to update some hyper-parameters in optimizer
      checkpoint=dict(
          type='CheckpointHook',  # The hook to save checkpoints periodically

          interval=3,  # The saving period
          save_best='auto',  # Specified metric to mearsure the best checkpoint during evaluation
          max_keep_ckpts=3),  # The maximum checkpoints to keep
      sampler_seed=dict(type='DistSamplerSeedHook'),  # Data-loading sampler for distributed training
      sync_buffers=dict(type='SyncBuffersHook'))  # Synchronize model buffers at the end of each epoch
  env_cfg = dict(  # Dict for setting environment
      cudnn_benchmark=False,  # Whether to enable cudnn benchmark
      mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0), # Parameters to setup multiprocessing
      dist_cfg=dict(backend='nccl')) # Parameters to setup distributed environment, the port can also be set

  log_processor = dict(
      type='LogProcessor',  # Log processor used to format log information
      window_size=20,  # Default smooth interval

      by_epoch=True)  # Whether to format logs with epoch type
  vis_backends = [  # List of visualization backends
      dict(type='LocalVisBackend')]  # Local visualization backend
  visualizer = dict(  # Config of visualizer
      type='ActionVisualizer',  # Name of visualizer
      vis_backends=vis_backends)
  log_level = 'INFO'  # The level of logging
  load_from = None  # Load model checkpoint as a pre-trained model from a given path. This will not resume training.
  resume = False  # Whether to resume from the checkpoint defined in `load_from`. If `load_from` is None, it will resume the latest checkpoint in the `work_dir`.
  ```

### Config System for Spatio-Temporal Action Detection

We incorporate modular design into our config system, which is convenient to conduct various experiments.

- An Example of FastRCNN

```

  To help the users have a basic idea of a complete config structure and the modules in a spatio-temporal action detection system,
  we make brief comments on the config of FastRCNN as the following.

  ```python
  # model setting
      type='FastRCNN',  # Class name of the detector
      _scope_='mmdet',  # The scope of current config
          type='ResNet3dSlowOnly',  # Name of the backbone
          depth=50, # Depth of ResNet model
          pretrained=None,   # The url/site of the pretrained model
          pretrained2d=False, # If the pretrained model is 2D

depth=50, # Depth of ResNet model
          lateral=False,  # If the backbone is with lateral connections
          num_stages=4, # Stages of ResNet model
          conv1_kernel=(1, 7, 7), # Conv1 kernel size
          conv1_stride_t=1, # Conv1 temporal stride
          pool1_stride_t=1, # Pool1 temporal stride
          spatial_strides=(1, 2, 2, 1)),  # The spatial stride for each ResNet stage
      roi_head=dict(  # Dict for roi_head
          type='AVARoIHead',  # Name of the roi_head
          bbox_roi_extractor=dict(  # Dict for bbox_roi_extractor
              type='SingleRoIExtractor3D',  # Name of the bbox_roi_extractor
              roi_layer_type='RoIAlign',  # Type of the RoI op
              output_size=8,  # Output feature size of the RoI op
              with_temporal_pool=True), # If temporal dim is pooled

roi_layer_type='RoIAlign',  # Type of the RoI op
          bbox_head=dict( # Dict for bbox_head
              type='BBoxHeadAVA', # Name of the bbox_head
              in_channels=2048, # Number of channels of the input feature
              num_classes=81, # Number of action classes + 1
              multilabel=True,  # If the dataset is multilabel
              dropout_ratio=0.5),  # The dropout ratio used
          format_shape='NCHW')),  # Final image shape format

std=[58.395, 57.12, 57.375],  # Std values of different channels to normalize
      train_cfg=dict(  # Training config of FastRCNN
          rcnn=dict(  # Dict for rcnn training config
              assigner=dict(  # Dict for assigner
                  type='MaxIoUAssignerAVA', # Name of the assigner
                  pos_iou_thr=0.9,  # IoU threshold for positive examples, > pos_iou_thr -> positive
                  neg_iou_thr=0.9,  # IoU threshold for negative examples, < neg_iou_thr -> negative
                  min_pos_iou=0.9), # Minimum acceptable IoU for positive examples
              sampler=dict( # Dict for sample
                  type='RandomSampler', # Name of the sampler
                  num=32, # Batch Size of the sampler
                  pos_fraction=1, # Positive bbox fraction of the sampler

type='RandomSampler', # Name of the sampler
                  neg_pos_ub=-1,  # Upper bound of the ratio of num negative to num positive
                  add_gt_as_proposals=True), # Add gt bboxes as proposals
              pos_weight=1.0)),  # Loss weight of positive examples
      test_cfg=dict(rcnn=None))  # Testing config of FastRCNN

  dataset_type = 'AVADataset' # Type of dataset for training, validation and testing
  data_root = 'data/ava/rawframes'  # Root path to data
  anno_root = 'data/ava/annotations'  # Root path to annotations

  ann_file_train = f'{anno_root}/ava_train_v2.1.csv'  # Path to the annotation file for training
  ann_file_val = f'{anno_root}/ava_val_v2.1.csv'  # Path to the annotation file for validation

ann_file_train = f'{anno_root}/ava_train_v2.1.csv'  # Path to the annotation file for training

  exclude_file_train = f'{anno_root}/ava_train_excluded_timestamps_v2.1.csv'  # Path to the exclude annotation file for training
  exclude_file_val = f'{anno_root}/ava_val_excluded_timestamps_v2.1.csv'  # Path to the exclude annotation file for validation

  label_file = f'{anno_root}/ava_action_list_v2.1_for_activitynet_2018.pbtxt'  # Path to the label file

  proposal_file_train = f'{anno_root}/ava_dense_proposals_train.FAIR.recall_93.9.pkl'  # Path to the human detection proposals for training examples
  proposal_file_val = f'{anno_root}/ava_dense_proposals_val.FAIR.recall_93.9.pkl'  # Path to the human detection proposals for validation examples

          type='AVASampleFrames',  # Sample frames pipeline, sampling frames from video
          clip_len=4,  # Frames of each sampled output clip
          frame_interval=16),  # Temporal interval of adjacent sampled frames
      dict(  # Config of RandomRescale
          type='RandomRescale',   # Randomly rescale the shortedge by a given range
          scale_range=(256, 320)),   # The shortedge size range of RandomRescale
      dict(  # Config of RandomCrop
          type='RandomCrop',   # Randomly crop a patch with the given size
          size=256),   # The size of the cropped patch

size=256),   # The size of the cropped patch
          input_format='NCTHW',  # Final image shape format
          collapse=True),   # Collapse the dim N if N == 1
      dict(type='PackActionInputs') # Pack input data
  ]

dict(  # Config of RawFrameDecode
  ]

          exclude_file=exclude_file_train,  # Path of exclude annotation file
          label_file=label_file,  # Path of label file
          proposal_file=proposal_file_train,  # Path of human detection proposals

proposal_file=proposal_file_train,  # Path of human detection proposals
      batch_size=1,  # Batch size of each single GPU during evaluation
      num_workers=8,  # Workers to pre-fetch data for each single GPU during evaluation
          exclude_file=exclude_file_val,  # Path of exclude annotation file

label_file=label_file,  # Path of label file
          proposal_file=proposal_file_val,  # Path of human detection proposals
  test_dataloader = val_dataloader  # Config of testing dataloader

  val_evaluator = dict(  # Config of validation evaluator
      type='AVAMetric',
      ann_file=ann_file_val,
      label_file=label_file,
      exclude_file=exclude_file_val)

      max_epochs=20,  # Total training epochs

val_interval=1)  # Validation interval

  param_scheduler = [ # Parameter scheduler for updating optimizer parameters, support dict or list
      dict(type='LinearLR',  # Decays the learning rate of each parameter group by linearly changing small multiplicative factor
          start_factor=0.1,  # The number we multiply learning rate in the first epoch
    	  begin=0,  # Step at which to start updating the learning rate
    	  end=5),  # Step at which to stop updating the learning rate

dict(type='MultiStepLR',  # Decays the learning rate once the number of epoch reaches one of the milestones
          end=20,  # Step at which to stop updating the learning rate
          milestones=[10, 15],  # Steps to decay the learning rate

          lr=0.2,  # Learning rate

type='SGD',  # Name of optimizer

interval=20,  # Interval to print the log

cudnn_benchmark=False,  # Whether to enable cudnn benchmark

  load_from = ('https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/'
               'slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb/'

load_from = ('https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/'
               'slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb_20220901-e7b65fad.pth')  # Load model checkpoint as a pre-trained model from a given path. This will not resume training.
  ```

### Config System for Action localization

- An Example of BMN

  To help the users have a basic idea of a complete config structure and the modules in an action localization system,
  we make brief comments on the config of BMN as the following.

To help the users have a basic idea of a complete config structure and the modules in an action localization system,
  For more detailed usage and alternative for per parameter in each module, please refer to the [API documentation](https://mmaction2.readthedocs.io/en/latest/api.html).

  ```python
      type='BMN',  # Class name of the localizer
      temporal_dim=100,  # Total frames selected for each video
      boundary_ratio=0.5,  # Ratio for determining video boundaries
      num_samples=32,  # Number of samples for each proposal
      num_samples_per_bin=3,  # Number of bin samples for each sample
      feat_dim=400,  # Dimension of feature
      soft_nms_alpha=0.4,  # Soft NMS alpha
      soft_nms_low_threshold=0.5,  # Soft NMS low threshold
      soft_nms_high_threshold=0.9,  # Soft NMS high threshold

soft_nms_alpha=0.4,  # Soft NMS alpha
      post_process_top_k=100)  # Top k proposals in post process

  dataset_type = 'ActivityNetDataset'  # Type of dataset for training, validation and testing
  data_root = 'data/activitynet_feature_cuhk/csv_mean_100/'  # Root path to data for training
  data_root_val = 'data/activitynet_feature_cuhk/csv_mean_100/'  # Root path to data for validation and testing
  ann_file_train = 'data/ActivityNet/anet_anno_train.json'  # Path to the annotation file for training
  ann_file_val = 'data/ActivityNet/anet_anno_val.json'  # Path to the annotation file for validation
  ann_file_test = 'data/ActivityNet/anet_anno_test.json'  # Path to the annotation file for testing

      dict(type='LoadLocalizationFeature'),  # Load localization feature pipeline

      dict(type='GenerateLocalizationLabels'),  # Generate localization labels pipeline
      dict(
          type='PackLocalizationInputs', # Pack localization data
          keys=('gt_bbox'), # Keys of input
          meta_keys=('video_name'))] # Meta keys of input
      dict(
          type='PackLocalizationInputs',  # Pack localization data
          keys=('gt_bbox'),   # Keys of input
          meta_keys=('video_name', 'duration_second', 'duration_frame',
                     'annotations', 'feature_frame'))]  # Meta keys of input

'annotations', 'feature_frame'))]  # Meta keys of input
      dict(
          keys=('gt_bbox'),  # Keys of input
      batch_size=8,  # Batch size of each single GPU during training

          data_prefix=dict(video=data_root),  # Prefix of video path

batch_size=1,  # Batch size of each single GPU during evaluation
          data_prefix=dict(video=data_root_val),  # Prefix of video path
      batch_size=1,  # Batch size of each single GPU during testing

batch_size=1,  # Batch size of each single GPU during testing

  work_dir = './work_dirs/bmn_400x100_2x8_9e_activitynet_feature/'  # Directory to save the model checkpoints and logs for the current experiments
    type='ANetMetric',
    metric_type='AR@AN',
    dump_config=dict(  # Config of localization output

        out=f'{work_dir}/results.json',  # Path to the output file
        output_format='json'))  # File format of the output file
  test_evaluator = val_evaluator   # Set test_evaluator as val_evaluator

  max_epochs = 9  # Total epochs to train the model
      max_epochs=max_epochs,  # Total training epochs
      type='ValLoop')  # Name of validating loop

type='TestLoop')  # Name of testing loop

      begin=0,  # Step at which to start updating the learning rate
      end=max_epochs,  # Step at which to stop updating the learning rate
      by_epoch=True,  # Whether the scheduled learning rate is updated by epochs
      milestones=[7, ],  # Steps to decay the learning rate
      gamma=0.1)]  # Multiplicative factor of parameter value decay

optimizer=dict(  # Config of optimizer. Support all kinds of optimizers in PyTorch. Refer to https://pytorch.org/docs/stable/optim.html#algorithms
          type='Adam',  # Name of optimizer
          lr=0.001,  # Learning rate

logger=dict(

env_cfg = dict(  # Dict for setting environment

vis_backends=vis_backends)
  ```

# Mask2Former for Video Instance Segmentation

We find Mask2Former also achieves state-of-the-art performance on video instance segmentation without modifying the architecture, the loss or even the training pipeline. In this report, we show universal image segmentation architectures trivially generalize to video segmentation by directly predicting 3D segmentation volumes. Specifically, Mask2Former sets a new state-of-the-art of 60.4 AP on YouTubeVIS-2019 and 52.6 AP on YouTubeVIS-2021. We believe Mask2Former is also capable of handling video semantic and panoptic segmentation, given its versatility in image segmentation. We hope this will make state-of-theart video segmentation research more accessible and bring more attention to designing universal image and video segmentation architectures.

  <img src="https://user-images.githubusercontent.com/46072190/188271377-164634a5-4d65-4161-8a69-2d0eaf2791f8.png"/>

</div>

```latex
}
```

## Results and models of Mask2Former on YouTube-VIS 2021 validation dataset

Note: Codalab has closed the evaluation portal of `YouTube-VIS 2019`, so we do not provide the results of `YouTube-VIS 2019` at present. If you want to evaluate the results of `YouTube-VIS 2021`, at present, you can submit the result to the evaluation portal of `YouTube-VIS 2022`. The value of `AP_S` is the result of `YouTube-VIS 2021`.

|          Method          | Backbone |  Style  | Lr schd | Mem (GB) | Inf time (fps) |  AP  |                                 Config                                  |                                                                                                                                                                                    Download                                                                                                                                                                                     |

|       Mask2Former        |   R-50   | pytorch |   8e    |   6.0    |       -        | 41.3 |           [config](mask2former_r50_8xb2-8e_youtubevis2021.py)           |        [model](https://download.openmmlab.com/mmdetection/v3.0/mask2former_vis/mask2former_r50_8xb2-8e_youtubevis2021/mask2former_r50_8xb2-8e_youtubevis2021_20230426_131833-5d215283.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/mask2former_vis/mask2former_r50_8xb2-8e_youtubevis2021/mask2former_r50_8xb2-8e_youtubevis2021_20230426_131833.json)         |

|       Mask2Former        |  R-101   | pytorch |   8e    |   7.5    |       -        | 42.3 |          [config](mask2former_r101_8xb2-8e_youtubevis2021.py)           |                             [model](https://download.openmmlab.com/mmdetection/v3.0/mask2former_vis/mask2former_r101_8xb2-8e_youtubevis2021/mask2former_r101_8xb2-8e_youtubevis2021_20220823_092747-8077d115.pth) \| [log](https://download.openmmlab.com/mmtracking/vis/mask2former/mask2former_r101_8xb2-8e_youtubevis2021_20220823_092747.json)                              |

| Mask2Former(200 queries) |  Swin-L  | pytorch |   8e    |   18.5   |       -        | 52.3 | [config](mask2former_swin-l-p4-w12-384-in21k_8xb2-8e_youtubevis2021.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/mask2former_vis/mask2former_swin-l-p4-w12-384-in21k_8xb2-8e_youtubevis2021/mask2former_swin-l-p4-w12-384-in21k_8xb2-8e_youtubevis2021_20220907_124752-48252603.pth) \| [log](https://download.openmmlab.com/mmtracking/vis/mask2former/mask2former_swin-l-p4-w12-384-in21k_8xb2-8e_youtubevis2021_20220907_124752.json) |

```shell
# Training Mask2Former on YouTube-VIS-2021 dataset with following command.
bash tools/dist_train.sh configs/mask2former_vis/mask2former_r50_8xb2-8e_youtubevis2021.py 8
```

```shell
bash tools/dist_test_tracking.sh configs/mask2former_vis/mask2former_r50_8xb2-8e_youtubevis2021.py --checkpoint ${CHECKPOINT_PATH}
```

```shell

```shell
python demo/mot_demo.py demo/demo_mot.mp4 configs/mask2former_vis/mask2former_r50_8xb2-8e_youtubevis2021.py  --checkpoint {CHECKPOINT_PATH} --out vis.mp4
```

# 如何加入技术交流QQ群？

- 打开QQ→点击群聊搜索→搜索群号637093648→输入问题答案：卷卷卷卷卷→进入群聊→准备接受图灵测试（bushi）
- 前往QQ搜索Pocky群：677104663(超多大佬)，问题答案：multi level intermediate representation

# 如何看作者b站直播？

- nihui的bilibili直播间：[水竹院落](https://live.bilibili.com/1264617)

# 编译

- ## 怎样下载完整源码？

   或者
   
   下载 [ncnn-xxxxx-full-source.zip](https://github.com/Tencent/ncnn/releases)

- ## 怎么交叉编译？cmake 工具链怎么设置啊？
  
   参见 https://github.com/Tencent/ncnn/wiki/how-to-build

   如上，下载完整源码。或者按提示执行: git submodule update --init

   或者自行编译安装，set(OpenCV_DIR {OpenCVConfig.cmake所在目录})

   set(ncnn_DIR {ncnnConfig.cmake所在目录})

- ## 找不到 Vulkan, 

   cmake版本 3.10，否则没有带 FindVulkan.cmake

   android-api >= 24

   macos 要先执行安装脚本

- ## 如何安装 vulkan sdk

- ## 找不到库（需要根据系统/编译器指定）

   需要链接openmp库 

   需要 vulkan-1.lib

需要链接openmp库 

   需要 glslang.lib OGLCompiler.lib SPIRV.lib OSDependent.lib

   find_library和target_like_libraries中增加 android 

   升级编译器 / libgcc_s libgcc

   升级 gcc

- ## 为啥自己编译的ncnn android库特别大？

   https://github.com/Tencent/ncnn/wiki/build-for-android.zh 以及见 如何裁剪更小的 ncnn 库

- ## ncnnoptimize和自定义层

   先ncnnoptimize再增加自定义层，避免ncnnoptimize不能处理自定义层保存。

- ## rtti/exceptions冲突

   产生原因是项目工程中使用的库配置不一样导致冲突，根据自己的实际情况分析是需要开启还是关闭。ncnn默认是ON，在重新编译ncnn时增加以下2个参数即可：

   - 开启：-DNCNN_DISABLE_RTTI=OFF -DNCNN_DISABLE_EXCEPTION=OFF
   - 关闭：-DNCNN_DISABLE_RTTI=ON -DNCNN_DISABLE_EXCEPTION=ON

   可能的情况：
   - 尝试升级 Android Studio 的 NDK 版本

- ## CMake 3.14.0 or higher is required.  You are running version 2.8.12.2
```shell
wget https://github.com/Kitware/CMake/releases/download/v3.18.2/cmake-3.18.2-Linux-x86_64.tar.gz
tar zxvf cmake-3.18.2-Linux-x86_64.tar.gz
mv cmake-3.18.2-Linux-x86_64 /opt/cmake-3.18.2
ln -sf /opt/cmake-3.18.2/bin/* /usr/bin/
```

# 怎样添加ncnn库到项目中？cmake方式怎么用？

编译ncnn，make install。linux/windows set/export ncnn_DIR 指向 install目录下包含ncnnConfig.cmake 的目录

- ## ios

- ## linux

- ## macos

# 转模型问题

- ## caffe

- ## ios

- ## linux

- ## macos

# 转模型问题

- ## caffe

- ## mxnet

   [通过MLIR将tensorflow2模型转换到ncnn](https://zhuanlan.zhihu.com/p/152535430) **@[nihui](https://www.zhihu.com/people/nihui-2)**

   onnx-simplifier 静态shape

- ## netron

- ## 怎么生成有固定 shape 信息的模型？

- ## why gpu能更快

- ## ncnnoptimize 怎么转成 fp16 模型

- ## ncnnoptimize 怎样查看模型的 FLOPS / 内存占用情况

- ## 怎么修改模型支持动态 shape？

- ## 如何将模型转换为代码内嵌到程序里？

   ncnn2mem

- ## 如何加密模型？

   https://zhuanlan.zhihu.com/p/268327784

- ## Linux下转的ncnn模型，Windows/MacOS/Android/.. 也能直接用吗？

   Yes，全平台通用

- ## 如何去掉后处理，再导出 onnx？

   检测：

   参考up的一篇文章<https://zhuanlan.zhihu.com/p/128974102>，步骤三就是去掉后处理,再导出onnx,其中去掉后处理可以是项目内测试时去掉后续步骤的结果。

- ## pytorch 有的层导不出 onnx 怎么办？

 方式一:

完全自定义的op，先改成能导出的（如 concat slice），转到 ncnn 后再修改 param

 方式一:

 方式二：

 可以使用PNNX来试试，参考以下文章大概说明:

   1. [Windows/Linux/macOS 编译 PNNX 步骤](https://zhuanlan.zhihu.com/p/431833958)

   2. [5分钟学会！用 PNNX 转换 TorchScript 模型到 ncnn 模型](https://zhuanlan.zhihu.com/p/427512763)

# 使用

   出现此类问题请先更新GPU驱动。Please upgrade your GPU driver if you encounter this crash or error.
   这里提供了一些品牌的GPU驱动下载网址.We have provided some drivers' download pages here.
   [Intel](https://downloadcenter.intel.com/product/80939/Graphics-Drivers)，[AMD](https://www.amd.com/en/support)，[Nvidia](https://www.nvidia.com/Download/index.aspx)

   文件路径 working dir

   param.bin 应该用 xxx.id.h 的枚举

   模型本身有问题

   你应该在 load_param / load_model 之前设置 net.opt.use_vulkan_compute = true;

- ## 多个blob输入，多个blob输出，怎么做？
   多次执行`ex.input()` 和 `ex.extract()`
```
ex.input("data1", in_1);

你应该在 load_param / load_model 之前设置 net.opt.use_vulkan_compute = true;

```
ex.input("data2", in_2);
ex.extract("output1", out_1);
ex.extract("output2", out_2);
```
- ## Extractor extract 多次会重复计算吗？

   不会

- ## 如何看每一层的耗时？

- ## 如何转换 cv::Mat CV_8UC3 BGR 图片

- ## 如何转换 float 数据为 ncnn::Mat

   首先，自己申请的内存需要自己管理，此时ncnn::Mat不会自动给你释放你传过来的float数据
   ``` c++
   std::vector<float> testData(60, 1.0);                                      // 利用std::vector<float>自己管理内存的申请和释放
   ncnn::Mat in1 = ncnn::Mat(60, (void*)testData.data()).reshape(4, 5, 3);    // 把float数据的指针转成void*传过去即可，甚至还可以指定维度(up说最好使用reshape用来解决channel gap)
   float* a = new float[60];                                                  // 自己new一块内存，后续需要自己释放
   ncnn::Mat in2 = ncnn::Mat(60, (void*)a).reshape(4, 5, 3).clone();          // 使用方法和上面相同，clone() to transfer data owner
   ```

ncnn::Mat in2 = ncnn::Mat(60, (void*)a).reshape(4, 5, 3).clone();          // 使用方法和上面相同，clone() to transfer data owner
   ```

- ## 如何初始化 ncnn::Mat 为全 0

   `mat.fill(0.f);`

- ## 如何查看／获取版本号

   cmake时会打印

   c_api.h ncnn_version()

   自己拼 1.0+yyyymmdd

- ## 如何转换 yuv 数据

   yuv420sp2rgb yuv420sp2rgb_nv12

   **[@metarutaiga](https://github.com/metarutaiga/xxYUV)**

- ## 如何 resize crop rotate 图片

   [efficient roi resize rotate](https://github.com/Tencent/ncnn/wiki/efficient-roi-resize-rotate)

- ## 如何人脸5点对齐

   get_affine_transform

   warpaffine_bilinear_c3

```c
// 计算变换矩阵 并且求逆变换
int type = 0;       // 0->区域外填充为v[0],v[1],v[2], -233->区域外不处理
unsigned int v = 0;
float tm[6];
float tm_inv[6];
// 人脸区域在原图上的坐标和宽高
float src_x = target->det.rect.x / target->det.w * pIveImageU8C3->u32Width;
float src_y = target->det.rect.y / target->det.h * pIveImageU8C3->u32Height;
float src_w = target->det.rect.w / target->det.w * pIveImageU8C3->u32Width;

float src_h = target->det.rect.h / target->det.h * pIveImageU8C3->u32Height;
float point_src[10] = {
src_x + src_w * target->attr.land[0][0], src_x + src_w * target->attr.land[0][1],
src_x + src_w * target->attr.land[1][0], src_x + src_w * target->attr.land[1][1],
src_x + src_w * target->attr.land[2][0], src_x + src_w * target->attr.land[2][1],
src_x + src_w * target->attr.land[3][0], src_x + src_w * target->attr.land[3][1],
src_x + src_w * target->attr.land[4][0], src_x + src_w * target->attr.land[4][1],
};
float point_dst[10] = { // +8 是因为我们处理112*112的图
30.2946f + 8.0f, 51.6963f,
65.5318f + 8.0f, 51.5014f,
48.0252f + 8.0f, 71.7366f,
33.5493f + 8.0f, 92.3655f,
62.7299f + 8.0f, 92.2041f,
};
// 第一种方式：先计算变换在求逆
AffineTrans::get_affine_transform(point_src, point_dst, 5, tm);
AffineTrans::invert_affine_transform(tm, tm_inv);

};
// 第二种方式：直接拿到求逆的结果
// AffineTrans::get_affine_transform(point_dst, point_src, 5, tm_inv);
// rgb 分离的，所以要单独处理
for(int c = 0; c < 3; c++)
{
    unsigned char* pSrc = malloc(xxx);
    unsigned char* pDst = malloc(xxx);
    ncnn::warpaffine_bilinear_c1(pSrc, SrcWidth, SrcHeight, SrcStride[c], pDst, DstWidth, DstHeight, DstStride[c], tm_inv, type, v);
}
// rgb packed则可以一次处理
ncnn::warpaffine_bilinear_c3(pSrc, SrcWidth, SrcHeight, SrcStride, pDst, DstWidth, DstHeight, DstStride, tm_inv, type, v);
```

- ## 如何获得中间层的blob输出
  
   ncnn::Mat output;
   
   ex.extract("your_blob_name", output);

- ## 为什么我使用GPU，但是GPU占用为0

   windows 10 任务管理器 - 性能选项卡 - GPU - 选择其中一个视图左上角的下拉箭头切换到 Compute_0 / Compute_1 / Cuda

   你还可以安装软件：GPU-Z 

- ## layer XYZ not exists or registered

   Your network contains some operations that are not implemented in ncnn.

   You may implement them as custom layer followed in how-to-implement-custom-layer-step-by-step.

   Or you could simply register them as no-op if you are sure those operations make no sense.

```
class Noop : public ncnn::Layer {};
DEFINE_LAYER_CREATOR(Noop)

net.register_custom_layer("LinearRegressionOutput", Noop_layer_creator);
net.register_custom_layer("MAERegressionOutput", Noop_layer_creator);
```

- ## network graph not ready

   You shall call Net::load_param() first, then Net::load_model().

   This error may also happens when Net::load_param() failed, but not properly handled.

   For more information about the ncnn model load api, see ncnn-load-model

- ## memory not 32-bit aligned at XYZ

For more information about the ncnn model load api, see ncnn-load-model

   The pointer passed to Net::load_param() or Net::load_model() is not 32bit aligned.

   In practice, the head pointer of std::vector is not guaranteed to be 32bit aligned.

   you can store your binary buffer in ncnn::Mat structure, its internal memory is aligned.

- ## crash on android with '__kmp_abort_process'

   This usually happens if you bundle multiple shared library with openmp linked

   It is actually an issue of the android ndk https://github.com/android/ndk/issues/1028

   On old android ndk, modify the link flags as

   -Wl,-Bstatic -lomp -Wl,-Bdynamic

   For recent ndk >= 21

   -fstatic-openmp

- ## dlopen failed: library "libomp.so" not found
   Newer android ndk defaults to dynamic openmp runtime

   modify the link flags as

   -fstatic-openmp -fopenmp

- ## crash when freeing a ncnn dynamic library(.dll/.so) built with openMP

Newer android ndk defaults to dynamic openmp runtime

   for optimal performance, the openmp threadpool spin waits for about a second prior to shutting down in case more work becomes available.

   If you unload a dynamic library that's in the process of spin-waiting, it will crash in the manner you see (most of the time).

   Just set OMP_WAIT_POLICY=passive in your environment, before calling loadlibrary. or Just wait a few seconds before calling freelibrary.

   You can also use the following method to set environment variables in your code:

   for msvc++:

      SetEnvironmentVariable(_T("OMP_WAIT_POLICY"), _T("passive"));

   for g++:

      setenv("OMP_WAIT_POLICY", "passive", 1)
   
      reference: https://stackoverflow.com/questions/34439956/vc-crash-when-freeing-a-dll-built-with-openmp

# 跑出来的结果对不上

for g++:

[ncnn-produce-wrong-result](https://github.com/Tencent/ncnn/wiki/FAQ-ncnn-produce-wrong-result)

- ## 如何打印 ncnn::Mat 的值？

```C++
{
    {
        {
            {
                printf("%f ", ptr[x]);
            }
            ptr += m.w;
        }
    }
}
```
In Android Studio, `printf` will not work, you can use `__android_log_print` instead. Example :
```C++
#include <android/log.h>  // Don't forget this

{
    {
        {

```C++

{
    {
        {
            {
                __android_log_print(ANDROID_LOG_DEBUG,"LOG_TAG","ncnn Mat is : %f", m.channel(q).row(y)[x]);
            }
        }
    }
}
```

- ## 如何可视化 ncnn::Mat 的值？

```
{

    {

        {
            {

for (int y=0; y<m.h; y++)
        {
            {
                if (v != v)
                {
                    sp[0] = 0;
                    sp[1] = 0;
                }

                sp += 3;
            }
        }
    }

    // tile
    {

    }

}
```

- ## 总是输出第一张图的结果

   复用 Extractor？！

}

}
```

- ## 启用fp16时的精度有差异

   net.opt.use_fp16_packed = false;

   net.opt.use_fp16_storage = false;

   net.opt.use_fp16_arithmetic = false;

   [ncnn-produce-wrong-result](https://github.com/Tencent/ncnn/wiki/FAQ-ncnn-produce-wrong-result)

# 如何跑得更快？内存占用更少？库体积更小？

- ## fp32 fp16

- ## 大小核绑定
   ncnn::set_cpu_powersave(int)绑定大核或小核
   注意windows系统不支持绑核。
   ncnn支持不同的模型运行在不同的核心。假设硬件平台有2个大核，4个小核，你想把netA运行在大核，netB运行在小核。
   0:全部
   1:小核
   2:大核
```
   {
   }

   {
   }
```

   }

   {
   }
```

   [openmp-best-practice.zh.md](https://github.com/Tencent/ncnn/blob/master/docs/how-to-use-and-FAQ/openmp-best-practice.zh.md)

- ## 查看 CPU 或 GPU 数量
   get_cpu_count
   
   get_gpu_count

- ## ncnnoptimize

   使用方式一：
    - ./ncnnoptimize ncnn.param ncnn.bin new.param new.bin flag
    <br/>注意这里的flag指的是fp32和fp16，其中0指的是fp32，1指的是fp16

   使用方式二：
    - ./ncnnoptimize ncnn.param ncnn.bin new.param new.bin flag cutstartname cutendname
    <br/>cutstartname：模型截取的起点
     <br/>cutendname：模型截取的终点

- ## 如何使用量化工具？

   [Post Training Quantization Tools](https://github.com/Tencent/ncnn/tree/master/tools/quantize)

- ## 如何设置线程数？

   opt.num_threads

- ## 如何降低CPU占用率？

   net.opt.openmp_blocktime = 0;
   
   OMP_WAIT_POLICY=passive

- ## 如何 batch inference？

```
   int max_batch_size = vkdev->info.compute_queue_count;

opt.num_threads

```
   
   ncnn::Mat inputs[1000];
   ncnn::Mat outputs[1000];
   
   #pragma omp parallel for num_threads(max_batch_size)
   for (int i=0; i<1000; i++)
   {
       ncnn::Extractor ex = net1.create_extractor();
       ex.input("data", inputs[i]);
       ex.extract("prob", outputs[i]);
   }
```

- ## partial graph inference

   先 extract 分类，判断后，再 extract bbox

- ## 如何启用 bf16s 加速？

```
net.opt.use_packing_layout = true;
net.opt.use_bf16_storage = true;
```

   [用bf16加速ncnn](https://zhuanlan.zhihu.com/p/112564372) **@[nihui](https://www.zhihu.com/people/nihui-2)**

   A53

- ## 如何裁剪更小的 ncnn 库？

   [build-minimal-library](https://github.com/Tencent/ncnn/wiki/build-minimal-library)

- ## net.opt sgemm winograd fp16_storage 各是有什么作用？

   对内存消耗的影响

- ## 如何解决显卡进入节能模式造成的一系列问题？

[build-minimal-library](https://github.com/Tencent/ncnn/wiki/build-minimal-library)

   对内存消耗的影响

   nVidia显卡（Intel和AMD估计也有）会在它认为的所谓空闲模式下，自动进入 `节能模式`，显存和核心频率就都会降低。
   
   简单来说就是如果你的计算任务是 `非连续的`，那么可能会让耗时看起来非常 `不均匀`，当期间有运算空闲间隔发生，显卡进入节能模式，则会在下一次冷启动时发生计算耗时远超正常耗时几倍的情况，如下日志所示：

   ```cpp
   //开始播放
   Total: 162ms, Diff: 0ms, GLTex2Mat: 7ms, calc: 152ms, Mat2GLTex: 3ms
   Total: 43ms, Diff: 0ms, GLTex2Mat: 3ms, calc: 35ms, Mat2GLTex: 2ms
   Total: 45ms, Diff: 0ms, GLTex2Mat: 3ms, calc: 37ms, Mat2GLTex: 3ms
   Total: 40ms, Diff: 0ms, GLTex2Mat: 3ms, calc: 32ms, Mat2GLTex: 4ms
   //暂停3秒
   //继续播放
   Total: 190ms, Diff: 0ms, GLTex2Mat: 9ms, calc: 177ms, Mat2GLTex: 3ms
   Total: 134ms, Diff: 0ms, GLTex2Mat: 5ms, calc: 110ms, Mat2GLTex: 18ms
   Total: 40ms, Diff: 0ms, GLTex2Mat: 3ms, calc: 34ms, Mat2GLTex: 2ms
   Total: 42ms, Diff: 0ms, GLTex2Mat: 3ms, calc: 36ms, Mat2GLTex: 2ms

Total: 40ms, Diff: 0ms, GLTex2Mat: 3ms, calc: 34ms, Mat2GLTex: 2ms
   Total: 47ms, Diff: 0ms, GLTex2Mat: 5ms, calc: 38ms, Mat2GLTex: 3ms
   ...
   ```

   在对时间不敏感的项目上，这个问题没什么大不了的，完全可以忽略，但是有些业务场景上必须精准推估下一帧及其未来几帧的从上传、计算到渲染的耗时情况，则这种现象将会给开发者打开些许困扰。

   ### 3种解决方法
   * 联系显卡厂商，让其更新驱动将你的应用加入到免节能模式的白名单。
     * 优点：你什么都不用改。缺点：沟通困难，很可能显卡厂商根本不理你。
   * [显卡控制面板] - [管理3D设置] - [电源管理模式]，改成：[最高性能优先]。
     * 优点：不用改代码。缺点：如果是部署端是小白用户，需要编写手册手把手教他。
   * 可以空闲（暂停）时定期灌一些心跳计算包的任务进去（放1x1小图）让GPU维持在高性能状态。
     * 优点：需要改代码。缺点：不低碳不环保。

# 白嫖项目

- ## nanodet

# 其他

- ## up主用的什么系统/编辑器/开发环境？

   | 软件类型     |   软件名称  |
   | ------------| ----------- |
   | 系统        | Fedora       |
   | 桌面环境     | KDE         |
   | 编辑器       | Kate        |
   | 画草图       | kolourpaint |
   | 画函数图像   | kmplot      |
   | bilibili直播 |  OBS         |

# TOFlow (IJCV'2019)

> [Video Enhancement with Task-Oriented Flow](https://arxiv.org/abs/1711.09078)

> **任务**: 视频插帧, 视频超分辨率

## 预训练模型测试结果

在 RGB 通道上评估。
评估指标 `PSNR / SSIM`。

|                         算法                         |                        预训练 SPyNet                         | Vimeo90k-triplet |      GPU 信息       |                         下载                          |
| :--------------------------------------------------: | :----------------------------------------------------------: | :--------------: | :-----------------: | :---------------------------------------------------: |

| [tof_vfi_spynet_chair_nobn_1xb1_vimeo90k](./tof_spynet-chair-wobn_1xb1_vimeo90k-triplet.py) | [spynet_chairs_final](https://download.openmmlab.com/mmediting/video_interpolators/toflow/pretrained_spynet_chair_20220321-4d82e91b.pth) | 33.3294 / 0.9465 | 1 (Tesla PG503-216) | [模型](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_chair_nobn_1xb1_vimeo90k_20220321-2fc9e258.pth) \| [日志](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_chair_nobn_1xb1_vimeo90k_20220321-2fc9e258.log.json) |

| [tof_vfi_spynet_kitti_nobn_1xb1_vimeo90k](./tof_spynet-kitti-wobn_1xb1_vimeo90k-triplet.py) | [spynet_chairs_final](https://download.openmmlab.com/mmediting/video_interpolators/toflow/pretrained_spynet_kitti_20220321-dbcc1cc1.pth) | 33.3339 / 0.9466 | 1 (Tesla PG503-216) | [模型](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_kitti_nobn_1xb1_vimeo90k_20220321-3f7ca4cd.pth) \| [日志](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_kitti_nobn_1xb1_vimeo90k_20220321-3f7ca4cd.log.json) |

| [tof_vfi_spynet_sintel_clean_nobn_1xb1_vimeo90k](./tof_spynet-sintel-wobn-clean_1xb1_vimeo90k-triplet.py) | [spynet_chairs_final](https://download.openmmlab.com/mmediting/video_interpolators/toflow/pretrained_spynet_sintel_clean_20220321-0756630b.pth) | 33.3170 / 0.9464 | 1 (Tesla PG503-216) | [模型](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_sintel_clean_nobn_1xb1_vimeo90k_20220321-6e52a6fd.pth) \| [日志](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_sintel_clean_nobn_1xb1_vimeo90k_20220321-6e52a6fd.log.json) |

| [tof_vfi_spynet_sintel_final_nobn_1xb1_vimeo90k](./tof_spynet-sintel-wobn-final_1xb1_vimeo90k-triplet.py) | [spynet_chairs_final](https://download.openmmlab.com/mmediting/video_interpolators/toflow/pretrained_spynet_sintel_final_20220321-5e89dcec.pth) | 33.3237 / 0.9465 | 1 (Tesla PG503-216) | [模型](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_sintel_final_nobn_1xb1_vimeo90k_20220321-8ab70dbb.pth) \| [日志](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_sintel_final_nobn_1xb1_vimeo90k_20220321-8ab70dbb.log.json) |

| [tof_vfi_spynet_pytoflow_nobn_1xb1_vimeo90k](./tof_spynet-pytoflow-wobn_1xb1_vimeo90k-triplet.py) | [spynet_chairs_final](https://download.openmmlab.com/mmediting/video_interpolators/toflow/pretrained_spynet_pytoflow_20220321-5bab842d.pth) | 33.3426 / 0.9467 | 1 (Tesla PG503-216) | [模型](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_pytoflow_nobn_1xb1_vimeo90k_20220321-5f4b243e.pth) \| [日志](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_pytoflow_nobn_1xb1_vimeo90k_20220321-5f4b243e.log.json) |

注: 由于 `batch_size=1` 预训练的 SPyNet 不包含 BN 层，这与 `https://github.com/Coldog2333/pytoflow` 一致.

## 快速开始

**训练**

<details>

TOF 的训练仅支持视频插帧任务。

```shell
# CPU上训练
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/tof/tof_spynet-chair-wobn_1xb1_vimeo90k-triplet.py

# 单个GPU上训练
python tools/train.py configs/tof/tof_spynet-chair-wobn_1xb1_vimeo90k-triplet.py

# 多个GPU上训练

# 单个GPU上训练

# 多个GPU上训练
./tools/dist_train.sh configs/tof/tof_spynet-chair-wobn_1xb1_vimeo90k-triplet.py 8
```

**测试**

<details>

TOF 的测试支持视频插帧和视频超分辨率两种任务。

**任务 1**: 视频插帧

```shell
# CPU上测试
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/tof/tof_spynet-chair-wobn_1xb1_vimeo90k-triplet.py https://download.openmmlab.com/mmediting/video_interpolators/toflow/pretrained_spynet_chair_20220321-4d82e91b.pth

# 单个GPU上测试
python tools/test.py configs/tof/tof_spynet-chair-wobn_1xb1_vimeo90k-triplet.py https://download.openmmlab.com/mmediting/video_interpolators/toflow/pretrained_spynet_chair_20220321-4d82e91b.pth

# 多个GPU上测试

# 多个GPU上测试
./tools/dist_test.sh configs/tof/tof_spynet-chair-wobn_1xb1_vimeo90k-triplet.py https://download.openmmlab.com/mmediting/video_interpolators/toflow/pretrained_spynet_chair_20220321-4d82e91b.pth 8
```

**任务 2**: 视频超分辨率

```shell
# CPU上测试
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/tof/tof_x4_official_vimeo90k.py https://download.openmmlab.com/mmediting/restorers/tof/tof_x4_vimeo90k_official-a569ff50.pth

# 单个GPU上测试
python tools/test.py configs/tof/tof_x4_official_vimeo90k.py https://download.openmmlab.com/mmediting/restorers/tof/tof_x4_vimeo90k_official-a569ff50.pth

# 多个GPU上测试
./tools/dist_test.sh configs/tof/tof_x4_official_vimeo90k.py https://download.openmmlab.com/mmediting/restorers/tof/tof_x4_vimeo90k_official-a569ff50.pth 8
```

```bibtex
  title={Video enhancement with task-oriented flow},

```bibtex
  author={Xue, Tianfan and Chen, Baian and Wu, Jiajun and Wei, Donglai and Freeman, William T},
  volume={127},
  number={8},
  pages={1106--1125},
}
```

# LFB

[Long-term feature banks for detailed video understanding](https://openaccess.thecvf.com/content_CVPR_2019/html/Wu_Long-Term_Feature_Banks_for_Detailed_Video_Understanding_CVPR_2019_paper.html)

To understand the world, we humans constantly need to relate the present to the past, and put events in context. In this paper, we enable existing video models to do the same. We propose a long-term feature bank---supportive information extracted over the entire span of a video---to augment state-of-the-art video models that otherwise would only view short clips of 2-5 seconds. Our experiments demonstrate that augmenting 3D convolutional networks with a long-term feature bank yields state-of-the-art results on three challenging video datasets: AVA, EPIC-Kitchens, and Charades.

<img src="https://user-images.githubusercontent.com/34324155/143016220-21d90fb3-fd9f-499c-820f-f6c421bda7aa.png" width="800"/>
</div>

### AVA2.1

| frame sampling strategy | resolution | gpus |               backbone               |   pretrain   |  mAP  | gpu_mem(M) |               config                |               ckpt                |               log                |
| :---------------------: | :--------: | :--: | :----------------------------------: | :----------: | :---: | :--------: | :---------------------------------: | :-------------------------------: | :------------------------------: |

|         4x16x1          |    raw     |  8   | SlowOnly ResNet50 (with Nonlocal LFB) | Kinetics-400 | 24.11 |    8620    | [config](/configs/detection/lfb/slowonly-lfb-nl_kinetics400-pretrained-r50_8xb12-4x16x1-20e_ava21-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/lfb/slowonly-lfb-nl_kinetics400-pretrained-r50_8xb12-4x16x1-20e_ava21-rgb/slowonly-lfb-nl_kinetics400-pretrained-r50_8xb12-4x16x1-20e_ava21-rgb_20220906-4c5b9f25.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/lfb/slowonly-lfb-nl_kinetics400-pretrained-r50_8xb12-4x16x1-20e_ava21-rgb/slowonly-lfb-nl_kinetics400-pretrained-r50_8xb12-4x16x1-20e_ava21-rgb.log) |

|         4x16x1          |    raw     |  8   |   SlowOnly ResNet50 (with Max LFB)   | Kinetics-400 | 22.15 |    8425    | [config](/configs/detection/lfb/slowonly-lfb-max_kinetics400-pretrained-r50_8xb12-4x16x1-20e_ava21-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/lfb/slowonly-lfb-max_kinetics400-pretrained-r50_8xb12-4x16x1-20e_ava21-rgb/slowonly-lfb-max_kinetics400-pretrained-r50_8xb12-4x16x1-20e_ava21-rgb_20220906-4963135b.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/lfb/slowonly-lfb-max_kinetics400-pretrained-r50_8xb12-4x16x1-20e_ava21-rgb/slowonly-lfb-max_kinetics400-pretrained-r50_8xb12-4x16x1-20e_ava21-rgb.log) |

Note:

According to the [Linear Scaling Rule](https://arxiv.org/abs/1706.02677), you may set the learning rate proportional to the batch size if you use different GPUs or videos per GPU,
2. We use `slowonly_r50_4x16x1` instead of `I3D-R50-NL` in the original paper as the backbone of LFB, but we have achieved the similar improvement: (ours: 20.1 -> 24.05 vs. author: 22.1 -> 25.8).
3. Because the long-term features are randomly sampled in testing, the test accuracy may have some differences.
4. Before train or test lfb, you need to infer feature bank with the [slowonly-lfb_ava-pretrained-r50_infer-4x16x1_ava21-rgb.py](/configs/detection/lfb/slowonly-lfb_ava-pretrained-r50_infer-4x16x1_ava21-rgb.py). For more details on infer feature bank, you can refer to [Train](#Train) part.
5. The ROIHead now supports single-label classification (i.e. the network outputs at most

   one-label per actor). This can be done by (a) setting multilabel=False during training and
   the test_cfg.rcnn.action_thr for testing.

## Train

### a. Infer long-term feature bank for training

Before train or test lfb, you need to infer long-term feature bank first. You can also dowonload long-term feature bank from [AVA_train_val_float32_lfb](https://download.openmmlab.com/mmaction/detection/lfb/AVA_train_val_float32_lfb.rar) or [AVA_train_val_float16_lfb](https://download.openmmlab.com/mmaction/detection/lfb/AVA_train_val_float16_lfb.rar), and then put them on `lfb_prefix_path`. In this case, you can skip this step.

Specifically, run the test on the training, validation, testing dataset with the config file [slowonly-lfb_ava-pretrained-r50_infer-4x16x1_ava21-rgb.py](/configs/detection/lfb/slowonly-lfb_ava-pretrained-r50_infer-4x16x1_ava21-rgb.py) (The config file will only infer the feature bank of training dataset and you need set `dataset_mode = 'val'` to infer the feature bank of validation dataset in the config file.), and the shared head [LFBInferHead](/mmaction/models/roi_heads/shared_heads/lfb_infer_head.py) will generate the feature bank.

A long-term feature bank file of AVA training and validation datasets with float32 precision occupies 3.3 GB. If store the features with float16 precision, the feature bank occupies 1.65 GB.

You can use the following command to infer feature bank of AVA training and validation dataset and the feature bank will be stored in `lfb_prefix_path/lfb_train.pkl` and `lfb_prefix_path/lfb_val.pkl`.

```shell

```shell
# set `dataset_mode = 'train'` in lfb_slowonly_r50_ava_infer.py
python tools/test.py configs/detection/lfb/slowonly-lfb-infer_r50_ava21-rgb.py \
    checkpoints/YOUR_BASELINE_CHECKPOINT.pth

# set `dataset_mode = 'val'` in lfb_slowonly_r50_ava_infer.py
```

We use [slowonly_r50_4x16x1 checkpoint](https://download.openmmlab.com/mmaction/detection/ava/slowonly_kinetics_pretrained_r50_4x16x1_20e_ava_rgb/slowonly_kinetics_pretrained_r50_4x16x1_20e_ava_rgb_20201217-40061d5f.pth) from [slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb](/configs/detection/slowonly/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb.py) to infer feature bank.

### b. Train LFB

```shell
```

```shell
```

Example: train LFB model on AVA with half-precision long-term feature bank.

```shell
python tools/train.py configs/detection/lfb/slowonly-lfb-nl_kinetics400-pretrained-r50_8xb12-4x16x1-20e_ava21-rgb.py \
  --seed 0 --deterministic
```

For more details and optional arguments infos, you can refer to the **Training** part in the [Training and Test Tutorial](/docs/en/user_guides/train_test.md).

## Test

### a. Infer long-term feature bank for testing

Before train or test lfb, you also need to infer long-term feature bank first. If you have generated the feature bank file, you can skip it.

The step is the same with **Infer long-term feature bank for training** part in [Train](#Train).

### b. Test LFB

```shell
```

```shell
```

Example: test LFB model on AVA with half-precision long-term feature bank and dump the result to a pkl file.

```shell
python tools/test.py configs/detection/lfb/slowonly-lfb-nl_kinetics400-pretrained-r50_8xb12-4x16x1-20e_ava21-rgb.py \
```

```BibTeX

booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
}
```

```BibTeX
@inproceedings{wu2019long,
  title={Long-term feature banks for detailed video understanding},
  author={Wu, Chao-Yuan and Feichtenhofer, Christoph and Fan, Haoqi and He, Kaiming and Krahenbuhl, Philipp and Girshick, Ross},
  pages={284--293},
}
```

# 高效评测

OpenCompass 支持自定义评测任务的任务划分器（`Partitioner`），实现评测任务的灵活切分；同时配合 `Runner` 控制任务执行的平台，如本机及集群。通过二者的组合，OpenCompass 可以将大评测任务分割到大量计算节点上运行，高效利用计算资源，从而大大加速评测流程。

默认情况下，OpenCompass 向用户隐藏了这些细节，并自动选择推荐的执行策略。但是，用户仍然可以根据自己需求定制其策略，只需向配置文件中添加 `infer` 和/或 `eval` 字段即可：

```python

        retry=5),
)

)
```

上面的例子演示了如何为推理和评估阶段配置执行策略。在推理阶段，任务将被划分成若干个子任务，每个子任务包含5000个样本，然后提交到 Slurm 集群进行执行，其中最多有64个任务并行运行。在评估阶段，每个单一的模型-数据集对形成一个任务，并在本地启动32个进程来计算指标。

以下章节将详细介绍里面涉及的模块。

)
```

## 任务划分 (Partitioner)

由于大语言模型的推理耗时长，评测的数据集量大，因此串行运行一次评测任务的时间开销往往很大。
OpenCompass 支持通过自定义评测任务的任务划分器（`Partitioner`），将大评测任务按不同策略划分为众多独立的小任务，通过并行运行充分利用计算资源。用户可以通过 `infer.partitioner` 及 `eval.partitioner` 配置推理和评测阶段的任务划分策略。下面，我们将会介绍 OpenCompass 中支持的所有划分策略。

该划分器会将每个模型和数据集的组合作为一个独立任务派发，为最基础的划分策略，并无任何额外参数。

```python

    # ...
)
```

该划分器目前不适用于评测阶段的任务（`OpenICLEvalTask`）。
```

该划分器会根据数据集的大小，乘上一个扩张系数，估算该数据集的推理成本（耗时）。然后会通过切分大数据集、合并小数据集的方式创建任务，尽可能保证各个子任务推理成本均等。

该划分器常用的参数如下：

```python

        max_task_size: int = 2000,  # 单个任务的最大长度

```python

        gen_task_coef: int = 20,  # 生成式任务的扩张系数
    ),
    # ...
)
```

`SizePartitioner` 在估算数据集推理成本时, 会根据推理任务的类型，选择不同的扩张系数。对于生成式任务，如使用 `GenInferencer` 的任务，会设置成比较大的 `gen_task_coef`；对于判别式任务，如使用 `PPLInferencer` 的任务，则会设置成 prompt 中 label 的数量。

```{note}
目前这种分割策略实现仍然比较粗糙，并未能准确反映生成式任务与判别式任务的计算量差距。我们也期待社区能提出更好的划分策略 ：）
```

## 运行后端 (Runner)

在多卡多机的集群环境下，我们若想实现多个任务的并行执行，通常需要依赖集群管理系统（如 Slurm）对任务进行分配和调度。OpenCompass 中，任务的分配和运行统一交由 Runner 负责。目前已经支持了 Slurm 和 PAI-DLC 两种调度后端，同时也保留了在本机直接启动任务的 `LocalRunner`。

`LocalRunner` 为最基本的运行器，可以将任务在本机并行运行。

```python

    # ...
        max_num_workers=16,  # 最大并行运行进程数

    # ...
        task=dict(type=OpenICLEvalTask),  # 待运行的任务
    )
)
```

```{note}
实际的运行任务数受到可用 GPU 资源和 `max_num_workers` 的限制。
```

`SlurmRunner` 会将任务提交到 Slurm 集群上运行。常用的配置字段如下：

```python

    # ...
        task=dict(type=OpenICLEvalTask),  # 待运行任务
        max_num_workers=16,  # 最大同时评测任务数
        retry=2,  # 任务失败的重试次数，可以避免意外发生的错误
    ),
)
```

`DLCRunner` 则可以将任务提交到 Alibaba Deep Learning Ceneter (DLC) 运行，该 Runner 依赖于 dlc。首先，先在环境内准备好 dlc：

```bash
cd ~
```

```bash
cd ~
```

根据提示填入相应信息，并得到 dlc 的配置文件（如 /user/.dlc/config），即完成了前期准备。之后，我们在配置文件按照格式指定 `DLCRunner` 的配置：

```python

    # ...
            bashrc_path="/user/.bashrc",  # 用于初始化运行环境的 bashrc 路径
            conda_env_name='opencompass',  # OpenCompass 的 conda 环境
            dlc_config_path="/user/.dlc/config",  # dlc 配置文件
            workspace_id='ws-xxx',  # DLC 工作空间 ID
            worker_image='xxx',  # 运行任务的 image url
        ),
    ),
)

```

## 任务 (Task)

任务（Task）是 OpenCompass 中的一个基础模块，本身是一个独立的脚本，用于执行计算密集的操作。每个任务都通过配置文件确定参数设置，且可以通过两种不同的方式执行：

),
    ),
)

```

1. 实例化一个任务对象，然后调用 `task.run()` 方法。
2. 调用 `get_command` 方法，并传入配置路径和包含 `{task_cmd}` 占位符的命令模板字符串（例如 `srun {task_cmd}`）。返回的命令字符串将是完整的命令，可以直接执行。

目前，OpenCompass 支持以下任务类型：

- `OpenICLInferTask`：基于 OpenICL 框架执行语言模型（LM）推断任务。
- `OpenICLEvalTask`：基于 OpenEval 框架执行语言模型（LM）评估任务。

未来，OpenCompass 将支持更多类型的任务。

# 2D Human Pose Estimation: New Benchmark and State of the Art Analysis

<details>

```bibtex
}
```

# SemanticKITTI 数据集

本页提供了有关在 MMDetection3D 中使用 SemanticKITTI 数据集的具体教程。

## 数据集准备

您可以在[这里](http://semantic-kitti.org/dataset.html#download)下载 SemanticKITTI 数据集并解压缩所有 zip 文件。

```
├── tools
├── data
│   │   │   ├── 00
│   │   │   ├── 01
│   │   │   ├── ..
│   │   │   ├── 22
```

SemanticKITTI 数据集包含 23 个序列，其中序列 \[0-7\] , \[9-10\] 作为训练集（约 19k 训练样本），序列 8 作为验证集（约 4k 验证样本），\[11-22\] 作为测试集 （约20k测试样本）。其中每个序列分别包含 velodyne 和 labels 两个文件夹分别存放激光雷达点云数据和分割标注 (其中高16位存放实例分割标注，低16位存放语义分割标注)。

### 创建 SemanticKITTI 数据集

我们提供了生成数据集信息的脚本，用于测试和训练。通过以下命令生成 `.pkl` 文件：

```bash
```

```
├── tools
├── data

```

```
├── tools
├── data
│   │   │   ├── 00
│   │   │   ├── 01
│   │   │   ├── ..
│   │   │   ├── 22
│   │   ├── semantickitti_infos_test.pkl
│   │   ├── semantickitti_infos_train.pkl
│   │   ├── semantickitti_infos_val.pkl
```

- `semantickitti_infos_train.pkl`: 训练数据集, 该字典包含两个键值: `metainfo` 和 `data_list`.
  `metainfo` 包含该数据集的基本信息。 `data_list` 是由字典组成的列表，每个字典（以下简称 `info`）包含了单个样本的所有详细信息。
  - info\['sample_id'\]：该样本在整个数据集的索引。
  - info\['lidar_points'\]：是一个字典，包含了激光雷达点相关的信息。
    - info\['lidar_points'\]\['lidar_path'\]：激光雷达点云数据的文件名。
    - info\['lidar_points'\]\['num_pts_feats'\]：点的特征维度
  - info\['pts_semantic_mask_pth'\]：三维语义分割的标注文件的文件路径

- info\['lidar_points'\]\['lidar_path'\]：激光雷达点云数据的文件名。

更多细节请参考 [semantickitti_converter.py](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/semantickitti_converter.py) 和 [update_infos_to_v2.py ](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/update_infos_to_v2.py) 。

下面展示了一个使用 SemanticKITTI 数据集进行 3D 语义分割的典型流程：

```python
    dict(
        load_dim=4,
        use_dim=4,
    dict(
        with_seg_3d=True,
        seg_3d_dtype='np.int32',
        seg_offset=2**16,
        dataset_type='semantickitti',
    dict(

seg_3d_dtype='np.int32',
    dict(
    dict(
        translation_std=[0.1, 0.1, 0.1],
    ),
]
```

- 数据增强:
  - `GlobalRotScaleTrans`：对输入点云数据进行旋转、缩放、平移。

## 评估

使用 8 个 GPU 以及 SemanticKITTI 指标评估的 MinkUNet 的示例如下：

```shell
bash tools/dist_test.sh configs/minkunet/minkunet_w32_8xb2-15e_semantickitti.py checkpoints/minkunet_w32_8xb2-15e_semantickitti_20230309_160710-7fa0a6f1.pth 8
```

## 度量指标

通常我们使用平均交并比 (mean Intersection over Union, mIoU) 作为 SemanticKITTI 语义分割任务的度量指标。

```

## 度量指标

以下是一个评估结果的样例:

| classes | car    | bicycle | motorcycle | truck  | bus    | person | bicyclist | motorcyclist | road   | parking | sidewalk | other-ground | building | fence  | vegetation | trunck | terrian | pole   | traffic-sign | miou   | acc    | acc_cls |
| ------- | ------ | ------- | ---------- | ------ | ------ | ------ | --------- | ------------ | ------ | ------- | -------- | ------------ | -------- | ------ | ---------- | ------ | ------- | ------ | ------------ | ------ | ------ | ------- |

| results | 0.9687 | 0.1908  | 0.6313     | 0.8580 | 0.6359 | 0.6818 | 0.8444    | 0.0002       | 0.9353 | 0.4854  | 0.8106   | 0.0024       | 0.9050   | 0.6111 | 0.8822     | 0.6605 | 0.7493  | 0.6442 | 0.4837       | 0.6306 | 0.9202 | 0.6924  |

# Preparing Kinetics-\[400/600/700\]

```BibTeX
  author = {Carreira, J. and Zisserman, Andrew},
  year = {2017},
  month = {07},
  pages = {4724-4733},
  title = {Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset},
  doi = {10.1109/CVPR.2017.502}
}
```

For basic dataset information, please refer to the official [website](https://deepmind.com/research/open-source/open-source-datasets/kinetics/).

:::{note}
Because of the expirations of some YouTube links, the sizes of kinetics dataset copies may be different. Here are the sizes of our kinetics dataset copies that used to train all checkpoints.

|   Dataset   | training videos | validation videos |
| :---------: | :-------------: | :---------------: |
| kinetics400 |     240436      |       19796       |
| Kinetics600 |     383393      |       27910       |
| Kinetics700 |     542357      |       34824       |

:::

:::

:::{note}
All experiments on Kinetics in MMAction2 are based on this version, we recommend users to try this version.
:::

MIM supports downloading from OpenDataLab and preprocessing Kinetics-400/600/700 dataset with one command line.

```Bash
odl login
# download and preprocess Kinetics-400 by MIM. Note that this might take a long time.
# download and preprocess Kinetics-600 by MIM. Note that this might take a long time.
# download and preprocess Kinetics-700 by MIM. Note that this might take a long time.

```

````

```

````

The scripts can be used for preparing kinetics400, kinetics600, kinetics700. To prepare different version of kinetics, you need to replace `${DATASET}` in the following examples with the specific dataset name. The choices of dataset names are `kinetics400`, `kinetics600` and `kinetics700`.
Before we start, please make sure that the directory is located at `$MMACTION2/tools/data/${DATASET}/`.

First of all, you can run the following script to prepare annotations by downloading from the official [website](https://deepmind.com/research/open-source/open-source-datasets/kinetics/).

```shell
```

Since some video urls are invalid, the number of video items in current official annotations are less than the original official ones.

```shell
```

So we provide an alternative way to download the older one as a reference.
Among these, the annotation files of Kinetics400 and Kinetics600 are from [official crawler](https://github.com/activitynet/ActivityNet/tree/199c9358907928a47cdfc81de4db788fddc2f91d/Crawler/Kinetics/data),
the annotation files of Kinetics700 are from [website](https://deepmind.com/research/open-source/open-source-datasets/kinetics/) downloaded in 05/02/2021.

```shell
bash download_backup_annotations.sh ${DATASET}
```

you can run the following script to prepare videos.

```shell
bash download_videos.sh ${DATASET}
```

```shell
```

**Important**: If you have already downloaded video dataset using the download script above,
you must replace all whitespaces in the class name for ease of processing by running

```shell
bash rename_classnames.sh ${DATASET}
```

For better decoding speed, you can resize the original videos into smaller sized, densely encoded version by:

```bash
python ../resize_videos.py ../../../data/${DATASET}/videos_train/ ../../../data/${DATASET}/videos_train_256p_dense_cache --dense --level 2
```

```bash
```

You can also download from [Academic Torrents](https://academictorrents.com/) ([kinetics400](https://academictorrents.com/details/184d11318372f70018cf9a72ef867e2fb9ce1d26) & [kinetics700](https://academictorrents.com/details/49f203189fb69ae96fb40a6d0e129949e1dfec98) with short edge 256 pixels are available) and [cvdfoundation/kinetics-dataset](https://github.com/cvdfoundation/kinetics-dataset) (Host by Common Visual Data Foundation and Kinetics400/Kinetics600/Kinetics-700-2020 are available)

```shell
mkdir /mnt/SSD/${DATASET}_extracted_train/
ln -s /mnt/SSD/${DATASET}_extracted_train/ ../../../data/${DATASET}/rawframes_train/
mkdir /mnt/SSD/${DATASET}_extracted_val/
ln -s /mnt/SSD/${DATASET}_extracted_val/ ../../../data/${DATASET}/rawframes_val/
```

```shell
bash extract_rgb_frames.sh ${DATASET}
```

```shell
```

```shell
bash extract_rgb_frames_opencv.sh ${DATASET}
```

```shell
bash extract_frames.sh ${DATASET}
```

The commands above can generate images with new short edge 256. If you want to generate images with short edge 320 (320p), or with fix size 340x256, you can change the args `--new-short 256` to `--new-short 320` or `--new-width 340 --new-height 256`.
More details can be found in [prepare dataset](/docs/en/user_guides/prepare_dataset.md).

you can run the follow scripts to generate file list in the format of videos and rawframes, respectively.

```shell
bash generate_videos_filelist.sh ${DATASET}
# execute the command below when rawframes are ready

```shell
bash generate_rawframes_filelist.sh ${DATASET}
```

````
`````

### Folder Structure

you can get the rawframes (RGB + Flow), videos and annotation files for Kinetics.

```
mmaction2
├── tools
├── data
│   ├── ${DATASET}
│   │   ├── ${DATASET}_train_list_videos.txt
│   │   ├── ${DATASET}_val_list_videos.txt
│   │   │   ├── abseiling
│   │   │   │   ├── 0wR5jVB-WPk_000417_000427.mp4
│   │   │   ├── ...
│   │   │   ├── wrapping_present
│   │   │   ├── ...
│   │   │   ├── zumba

│   │   │   ├── ...
│   │   │   ├── ...
│   │   ├── rawframes_train (optional)
│   │   ├── rawframes_val (optional)

```

# FAQ

To help users better understand the `projects/` folder and how to use it effectively, we've created this FAQ page. Here, users can find answers to common questions and learn more about various aspects of the `projects/` folder, such as its usage and contribution guidance.

## Q1: Why set up `projects/` folder?

Implementing new models and features into OpenMMLab's algorithm libraries could be troublesome due to the rigorous requirements on code quality, which could hinder the fast iteration of SOTA models and might discourage our members from sharing their latest outcomes here. And that's why we have this `projects/` folder now, where some experimental features, frameworks and models are placed, only needed to satisfy the minimum requirement on the code quality, and can be used as standalone libraries. Users are welcome to use them if they [use MMPose from source](https://mmpose.readthedocs.io/en/dev-1.x/installation.html#best-practices).

## Q2: Why should there be a checklist for a project?

## Q3: What kind of PR will be merged?

## Q4: Compared to other models in the core packages, why do the model implementations in projects have different training/testing commands?

Projects are organized independently from the core package, and therefore their modules cannot be directly imported by `train.py` and `test.py`. Each model implementation in projects should either use `mim` for training/testing as suggested in the example project or provide a custom `train.py`/`test.py`.

## Q5: How to debug a project with a debugger?

# Bottom-up Human Pose Estimation via Disentangled Keypoint Regression (DEKR)

<details>
<summary align="right"><a href="https://arxiv.org/abs/2104.02300">DEKR (CVPR'2021)</a></summary>

```bibtex
@inproceedings{geng2021bottom,
  title={Bottom-up human pose estimation via disentangled keypoint regression},
  author={Geng, Zigang and Sun, Ke and Xiao, Bin and Zhang, Zhaoxiang and Wang, Jingdong},
  pages={14676--14686},
}
```

DEKR is a popular 2D bottom-up pose estimation approach that simultaneously detects all the instances and regresses the offsets from the instance centers to joints.

year={2021}
}
```

In order to predict the offsets more accurately, the offsets of different joints are regressed using separated branches with deformable convolutional layers. Thus convolution kernels with different shapes are adopted to extract features for the corresponding joint.

<details>

```bibtex
}
```

<details>

```bibtex

| [pose_hrnetv2_w18](/configs/face_2d_keypoint/topdown_heatmap/coco_wholebody_face/td-hm_hrnetv2-w18_8xb32-60e_coco-wholebody-face-256x256.py) |  256x256   | 0.0569 | [ckpt](https://download.openmmlab.com/mmpose/face/hrnetv2/hrnetv2_w18_coco_wholebody_face_256x256-c1ca469b_20210909.pth) | [log](https://download.openmmlab.com/mmpose/face/hrnetv2/hrnetv2_w18_coco_wholebody_face_256x256_20210909.log.json) |

# 概述

欢迎来到 MMagic！ 在本节中，您将了解

- [MMagic是什么？](#mmagic-是什么)
- [为什么要使用 MMagic？](#为什么要使用-mmagic)
- [新手入门](#新手入门)
- [基础教程](#基础教程)
- [进阶教程](#进阶教程)

## MMagic 是什么？

MMagic (**M**ultimodal **A**dvanced, **G**enerative, and **I**ntelligent **C**reation) 是一个供专业人工智能研究人员和机器学习工程师去处理、编辑和生成图像与视频的开源 AIGC 工具箱。

MMagic 允许研究人员和工程师使用最先进的预训练模型，并且可以轻松训练和开发新的定制模型。

MMagic 支持各种基础生成模型，包括：

- 无条件生成对抗网络 (GANs)
- 条件生成对抗网络 (GANs)
- 内部学习
- 扩散模型
- 还有许多其他生成模型即将推出！

MMagic 支持各种应用程序，包括：

- 图文生成
- 图像翻译
- 3D 生成
- 图像超分辨率
- 视频超分辨率
- 视频插帧
- 图像补全
- 图像抠图
- 图像修复
- 图像上色
- 图像生成
- 还有许多其他应用程序即将推出！

        </object>
    </video>
</div>
</br>

## 为什么要使用 MMagic？

</object>
    </video>
</div>
</br>

## 新手入门

安装说明见[安装](install.md)。

## 基础教程

对于初学者，我们建议从 [基础教程](../user_guides/config.md) 学习 MMagic 的基本用法。

## 进阶教程

对于熟悉 MMagic 的用户，可能想了解 MMagic 的进阶实用，以及如何扩展算法库，如何使用多个算法库框架等高级用法，请参考[进阶教程](../advanced_guides/evaluator.md)。

## 开发指南

想要使用 MMagic 进行深度开发的用户，可以参考[开发指南](../howto/models.md)。

# 使用已有模型在标准数据集上进行推理

MMDetection 提供了许多预训练好的检测模型，可以在 [Model Zoo](https://mmdetection.readthedocs.io/zh_CN/latest/model_zoo.html) 查看具体有哪些模型。

推理具体指使用训练好的模型来检测图像上的目标，本文将会展示具体步骤。

在 MMDetection 中，一个模型被定义为一个[配置文件](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/config.html) 和对应被存储在 checkpoint 文件内的模型参数的集合。

首先，我们建议从 [RTMDet](https://github.com/open-mmlab/mmdetection/tree/main/configs/rtmdet) 开始，其 [配置](https://github.com/open-mmlab/mmdetection/blob/main/configs/rtmdet/rtmdet_l_8xb32-300e_coco.py) 文件和 [checkpoint](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet_l_8xb32-300e_coco/rtmdet_l_8xb32-300e_coco_20220719_112030-5a0be7c4.pth) 文件在此。
我们建议将 checkpoint 文件下载到 `checkpoints` 文件夹内。

## 推理的高层编程接口

MMDetection 为在图片上推理提供了 Python 的高层编程接口。下面是建立模型和在图像或视频上进行推理的例子。

```python
import cv2
from mmcv.transforms import Compose
from mmdet.registry import VISUALIZERS

# 指定模型的配置文件和 checkpoint 文件路径
config_file = 'configs/rtmdet/rtmdet_l_8xb32-300e_coco.py'
checkpoint_file = 'checkpoints/rtmdet_l_8xb32-300e_coco_20220719_112030-5a0be7c4.pth'

# 根据配置文件和 checkpoint 文件构建模型

# 初始化可视化工具
visualizer = VISUALIZERS.build(model.cfg.visualizer)
# 从 checkpoint 中加载 Dataset_meta，并将其传递给模型的 init_detector
visualizer.dataset_meta = model.dataset_meta

# 测试单张图片并展示结果
img = 'test.jpg'  # 或者 img = mmcv.imread(img)，这样图片仅会被读一次
result = inference_detector(model, img)

# 显示结果
img = mmcv.imread(img)
img = mmcv.imconvert(img, 'bgr', 'rgb')

visualizer.add_datasample(
    'result',
    img,
    data_sample=result,
    show=True)

# 测试视频并展示结果
# 构建测试 pipeline

    'result',
    img,

model.cfg.test_dataloader.dataset.pipeline[0].type = 'LoadImageFromNDArray'
test_pipeline = Compose(model.cfg.test_dataloader.dataset.pipeline)

# 可视化工具在第33行和35行已经初完成了初始化，如果直接在一个 jupyter nodebook 中运行这个 demo，
# 这里则不需要再创建一个可视化工具了。
# 初始化可视化工具

# 显示间隔 (ms), 0 表示暂停
wait_time = 1

cv2.namedWindow('video', 0)

for frame in track_iter_progress(video_reader):
    result = inference_detector(model, frame, test_pipeline=test_pipeline)
    visualizer.add_datasample(
        name='video',
        image=frame,
        data_sample=result,
        show=False)
    frame = visualizer.get_image()

    mmcv.imshow(frame, 'video', wait_time)

cv2.destroyAllWindows()
```

Jupyter notebook 上的演示样例在 [demo/inference_demo.ipynb](https://github.com/open-mmlab/mmdetection/blob/main/demo/inference_demo.ipynb) 。

注意: `inference_detector` 目前仅支持单张图片的推理。

## 演示样例

我们还提供了三个演示脚本，它们是使用高层编程接口实现的。[源码在此](https://github.com/open-mmlab/mmdetection/blob/main/demo) 。

### 图片样例

这是在单张图片上进行推理的脚本。

```shell
    [--weights ${WEIGHTS}] \
    [--pred-score-thr ${SCORE_THR}]
```

运行样例：

```shell
    configs/rtmdet/rtmdet_l_8xb32-300e_coco.py \
    --weights checkpoints/rtmdet_l_8xb32-300e_coco_20220719_112030-5a0be7c4.pth \
```

### 摄像头样例

这是使用摄像头实时图片的推理脚本。

```shell

configs/rtmdet/rtmdet_l_8xb32-300e_coco.py \
```

### 摄像头样例

```shell
python demo/webcam_demo.py \
    [--camera-id ${CAMERA-ID}] \
    [--score-thr ${SCORE_THR}]
```

运行样例：

```shell
    checkpoints/rtmdet_l_8xb32-300e_coco_20220719_112030-5a0be7c4.pth
```

### 视频样例

这是在视频样例上进行推理的脚本。

```shell
python demo/video_demo.py \
    ${VIDEO_FILE} \
    [--score-thr ${SCORE_THR}] \
    [--out ${OUT_FILE}] \
    [--show] \
    [--wait-time ${WAIT_TIME}]
```

运行样例：

```shell
python demo/video_demo.py demo/demo.mp4 \
    checkpoints/rtmdet_l_8xb32-300e_coco_20220719_112030-5a0be7c4.pth \
    --out result.mp4

运行样例：

```shell
```

### 视频样例，显卡加速版本

这是在视频样例上进行推理的脚本，使用显卡加速。

```shell
python demo/video_gpuaccel_demo.py \
     ${VIDEO_FILE} \
     ${CONFIG_FILE} \
     ${CHECKPOINT_FILE} \
     [--device ${GPU_ID}] \
     [--score-thr ${SCORE_THR}] \
     [--nvdecode] \
     [--out ${OUT_FILE}] \
     [--show] \
     [--wait-time ${WAIT_TIME}]

```

运行样例：

```shell
python demo/video_gpuaccel_demo.py demo/demo.mp4 \
    --nvdecode --out result.mp4
```

## 多模态算法的推理和验证

```

随着多模态视觉算法的不断发展，MMDetection 也完成了对这类算法的支持。这一小节我们通过 GLIP 算法和模型来演示如何使用对应多模态算法的 demo 和 eval 脚本。同时 MMDetection 也在 projects 下完成了 [gradio_demo 项目](../../../projects/gradio_demo/)，用户可以参照[文档](../../../projects/gradio_demo/README.md)在本地快速体验 MMDetection 中支持的各类图片输入的任务。

### 模型准备

首先需要安装多模态依赖：

```shell
# if source

# if wheel
```

MMDetection 已经集成了 glip 算法和模型，可以直接使用链接下载使用：

```shell
```

### 推理演示

下载完成后我们就可以利用 `demo` 下的多模态推理脚本完成推理：

```shell
python demo/image_demo.py demo/demo.jpg glip_tiny_a_mmdet-b3654169.pth --texts bench
```

demo 效果如下图所示：

```

### 推理演示

```shell
```

<img src="https://user-images.githubusercontent.com/17425982/234547841-266476c8-f987-4832-8642-34357be621c6.png" height="300"/>
</div>

如果想进行多种类型的识别，需要使用 `xx . xx .` 的格式在 `--texts` 字段后声明目标类型:

```shell
python demo/image_demo.py demo/demo.jpg glip_tiny_a_mmdet-b3654169.pth --texts 'bench . car .'
```

结果如下图所示：

<img src="https://user-images.githubusercontent.com/17425982/234548156-ef9bbc2e-7605-4867-abe6-048b8578893d.png" height="300"/>
</div>

推理脚本还支持输入一个句子作为 `--texts` 字段的输入：

```shell
python demo/image_demo.py demo/demo.jpg glip_tiny_a_mmdet-b3654169.pth --texts 'There are a lot of cars here.'
```

结果可以参考下图：

<img src="https://user-images.githubusercontent.com/17425982/234548490-d2e0a16d-1aad-4708-aea0-c829634fabbd.png" height="300"/>
</div>

### 验证演示

```

结果可以参考下图：

</div>

### 验证演示

MMDetection 支持后的 GLIP 算法对比官方版本没有精度上的损失， benchmark 如下所示：

| Model                   | official mAP | mmdet mAP |
| ----------------------- | :----------: | :-------: |
| glip_A_Swin_T_O365.yaml |     42.9     |   43.0    |
| glip_Swin_T_O365.yaml   |     44.9     |   44.9    |
| glip_Swin_L.yaml        |     51.4     |   51.3    |

用户可以使用 `test.py` 脚本对模型精度进行验证，使用如下所示：

```shell
# 1 gpu
python tools/test.py configs/glip/glip_atss_swin-t_fpn_dyhead_pretrain_obj365.py glip_tiny_a_mmdet-b3654169.pth

# 8 GPU
./tools/dist_test.sh configs/glip/glip_atss_swin-t_fpn_dyhead_pretrain_obj365.py glip_tiny_a_mmdet-b3654169.pth 8
```

验证结果大致如下：

```shell
Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.428
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.594

```

验证结果大致如下：

```shell
Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.466
Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.300
Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.477
Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.534
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.634
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.634
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.634
Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.473
Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.690
Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.789
```

# Restormer (CVPR'2022)

> [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://arxiv.org/abs/2111.09881)

> **Task**: Denoising, Deblurring, Deraining

Since convolutional neural networks (CNNs) perform well at learning generalizable image priors from large-scale data, these models have been extensively applied to image restoration and related tasks. Recently, another class of neural architectures, Transformers, have shown significant performance gains on natural language and high-level vision tasks. While the Transformer model mitigates the shortcomings of CNNs (i.e., limited receptive field and inadaptability to input content), its computational complexity grows quadratically with the spatial resolution, therefore making it infeasible to apply to most image restoration tasks involving high-resolution images. In this work, we propose an efficient Transformer model by making several key designs in the building blocks (multi-head attention and feed-forward network) such that it can capture long-range pixel interactions, while still remaining applicable to large images. Our model, named Restoration Transformer (Restormer), achieves state-of-the-art results on several image restoration tasks, including image deraining, single-image motion deblurring, defocus deblurring (single-image and dual-pixel data), and image denoising (Gaussian grayscale/color denoising, and real image denoising).

 <img src="https://user-images.githubusercontent.com/49083766/206964466-95de972a-ff92-4493-9097-73118590d78f.png" width="800"/>
</div >

### **Deraining**

Evaluated on Y channels. The metrics are `PSNR` / `SSIM` .

|                             Model                             | Dataset  | PSNR (Y) | SSIM (Y) | Training Resources |                                             Download                                              |
| :-----------------------------------------------------------: | :------: | :------: | :------: | :----------------: | :-----------------------------------------------------------------------------------------------: |
| [restormer_official_rain13k](./restormer_official_rain13k.py) | Rain100H | 31.4804  |  0.9056  |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_rain13k-2be7b550.pth) \| log |

| [restormer_official_rain13k](./restormer_official_rain13k.py) | Rain100L | 39.1023  |  0.9787  |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_rain13k-2be7b550.pth) \| log |
| [restormer_official_rain13k](./restormer_official_rain13k.py) | Test100  | 32.0287  |  0.9239  |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_rain13k-2be7b550.pth) \| log |
| [restormer_official_rain13k](./restormer_official_rain13k.py) | Test1200 | 33.2251  | /0.9272  |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_rain13k-2be7b550.pth) \| log |
| [restormer_official_rain13k](./restormer_official_rain13k.py) | Test2800 | 34.2170  |  0.9451  |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_rain13k-2be7b550.pth) \| log |

### **Motion Deblurring**

Evaluated on RGB channels for GoPro and HIDE, and Y channel for ReakBlur-J and ReakBlur-R. The metrics are `PSNR` / `SSIM` .

|                           Model                           |   Dataset    | PSNR/SSIM (RGB) | <br>PSNR/SSIM (Y) | Training Resources |                                     Download                                      |
| :-------------------------------------------------------: | :----------: | :-------------: | :---------------: | :----------------: | :-------------------------------------------------------------------------------: |
| [restormer_official_gopro](./restormer_official_gopro.py) | DatasetGoPro | 32.9295/0.9496  |         -         |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_gopro-db7363a0.pth) \| log |

| [restormer_official_gopro](./restormer_official_gopro.py) |     HIDE     | 31.2289/0.9345  |         -         |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_gopro-db7363a0.pth) \| log |
| [restormer_official_gopro](./restormer_official_gopro.py) |  RealBlur-J  |        -        |  28.4356/0.8681   |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_gopro-db7363a0.pth) \| log |
| [restormer_official_gopro](./restormer_official_gopro.py) |  RealBlur-R  |        -        |  35.9141/0.9707   |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_gopro-db7363a0.pth) \| log |

### **Defocus Deblurring**

Evaluated on RGB channels. The metrics are `PSNR` / `SSIM` / `MAE` / `LPIPS`.

|                                 Model                                 |    Dataset     |  PSNR   |  SSIM  |  MAE   | Training Resources |                                     Download                                     |
| :-------------------------------------------------------------------: | :------------: | :-----: | :----: | :----: | :----------------: | :------------------------------------------------------------------------------: |
| [restormer_official_dpdd-single](./restormer_official_dpdd-single.py) | Indoor Scenes  | 28.8681 | 0.8859 | 0.0251 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dpdd-single-6bc31582.pth) \| log |

| [restormer_official_dpdd-single](./restormer_official_dpdd-single.py) | Outdoor Scenes | 23.2410 | 0.7509 | 0.0499 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dpdd-single-6bc31582.pth) \| log |
| [restormer_official_dpdd-single](./restormer_official_dpdd-single.py) |    Combined    | 25.9805 | 0.8166 | 0.0378 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dpdd-single-6bc31582.pth) \| log |
|   [restormer_official_dpdd-dual](./restormer_official_dpdd-dual.py)   | Indoor Scenes  | 26.6160 | 0.8346 | 0.0354 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dpdd-dual-52c94c00.pth) \| log |

|   [restormer_official_dpdd-dual](./restormer_official_dpdd-dual.py)   | Outdoor Scenes | 26.6160 | 0.8346 | 0.0354 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dpdd-dual-52c94c00.pth) \| log |
|   [restormer_official_dpdd-dual](./restormer_official_dpdd-dual.py)   |    Combined    | 26.6160 | 0.8346 | 0.0354 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dpdd-dual-52c94c00.pth) \| log |

### **Gaussian Denoising**

**Test Grayscale Gaussian Noise**

Evaluated on grayscale images. The metrics are `PSNR` / `SSIM` .

**training a separate model for each noise level**

|                                   Model                                    | Dataset  | $\\sigma$ |  PSNR   |  SSIM  | Training Resources |                                    Download                                    |

| :------------------------------------------------------------------------: | :------: | :-------: | :-----: | :----: | :----------------: | :----------------------------------------------------------------------------: |
| [restormer_official_dfwb-gray-sigma15](./restormer_official_dfwb-gray-sigma15.py) |  Set12   |    15     | 34.0182 | 0.9160 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-sigma15-da74417f.pth) \| log |
| [restormer_official_dfwb-gray-sigma15](./restormer_official_dfwb-gray-sigma15.py) |  BSD68   |    15     | 32.4987 | 0.8940 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-sigma15-da74417f.pth) \| log |

| [restormer_official_dfwb-gray-sigma15](./restormer_official_dfwb-gray-sigma15.py) | Urban100 |    15     | 34.4336 | 0.9419 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-sigma15-da74417f.pth) \| log |
| [restormer_official_dfwb-gray-sigma25](./restormer_official_dfwb-gray-sigma25.py) |  Set12   |    25     | 31.7289 | 0.8811 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-sigma25-08010841.pth) \| log |
| [restormer_official_dfwb-gray-sigma25](./restormer_official_dfwb-gray-sigma25.py) |  BSD68   |    25     | 30.1613 | 0.8370 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-sigma25-08010841.pth) \| log |

| [restormer_official_dfwb-gray-sigma25](./restormer_official_dfwb-gray-sigma25.py) | Urban100 |    25     | 32.1162 | 0.9140 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-sigma25-08010841.pth) \| log |
| [restormer_official_dfwb-gray-sigma50](./restormer_official_dfwb-gray-sigma50.py) |  Set12   |    50     | 28.6269 | 0.8188 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-sigma50-ee852dfe.pth) \| log |
| [restormer_official_dfwb-gray-sigma50](./restormer_official_dfwb-gray-sigma50.py) |  BSD68   |    50     | 27.3266 | 0.7434 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-sigma50-ee852dfe.pth) \| log |

| [restormer_official_dfwb-gray-sigma50](./restormer_official_dfwb-gray-sigma50.py) | Urban100 |    50     | 28.9636 | 0.8571 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-sigma50-ee852dfe.pth) \| log |

**learning a single model to handle various noise levels**

| [restormer_official_dfwb-gray-sigma15](./restormer_official_dfwb-gray-sigma15.py) |  Set12   |    15     | 33.9642 | 0.9153 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-blind-5f094bcc.pth) \| log |
| [restormer_official_dfwb-gray-sigma15](./restormer_official_dfwb-gray-sigma15.py) |  BSD68   |    15     | 32.4994 | 0.8928 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-blind-5f094bcc.pth) \| log |
| [restormer_official_dfwb-gray-sigma15](./restormer_official_dfwb-gray-sigma15.py) | Urban100 |    15     | 34.3152 | 0.9409 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-blind-5f094bcc.pth) \| log |

| [restormer_official_dfwb-gray-sigma25](./restormer_official_dfwb-gray-sigma25.py) |  Set12   |    25     | 31.7106 | 0.8810 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-blind-5f094bcc.pth) \| log |
| [restormer_official_dfwb-gray-sigma25](./restormer_official_dfwb-gray-sigma25.py) |  BSD68   |    25     | 30.1486 | 0.8360 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-blind-5f094bcc.pth) \| log |
| [restormer_official_dfwb-gray-sigma25](./restormer_official_dfwb-gray-sigma25.py) | Urban100 |    25     | 32.0457 | 0.9131 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-blind-5f094bcc.pth) \| log |

| [restormer_official_dfwb-gray-sigma50](./restormer_official_dfwb-gray-sigma50.py) |  Set12   |    50     | 28.6614 | 0.8197 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-blind-5f094bcc.pth) \| log |
| [restormer_official_dfwb-gray-sigma50](./restormer_official_dfwb-gray-sigma50.py) |  BSD68   |    50     | 27.3537 | 0.7422 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-blind-5f094bcc.pth) \| log |
| [restormer_official_dfwb-gray-sigma50](./restormer_official_dfwb-gray-sigma50.py) | Urban100 |    50     | 28.9848 | 0.8571 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-blind-5f094bcc.pth) \| log |

**Test Color Gaussian Noise**

Evaluated on RGB channels. The metrics are `PSNR` / `SSIM` .

|                                  Model                                  | Dataset  | $\\sigma$ | PSNR (RGB) | SSIM (RGB) | Training Resources |                                  Download                                  |
| :---------------------------------------------------------------------: | :------: | :-------: | :--------: | :--------: | :----------------: | :------------------------------------------------------------------------: |
| [restormer_official_dfwb-color-sigma15](./restormer_official_dfwb-color-sigma15.py) |  CBSD68  |    15     |  34.3506   |   0.9352   |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma15-012ceb71.pth) \| log |

| [restormer_official_dfwb-color-sigma15](./restormer_official_dfwb-color-sigma15.py) | Kodak24  |    15     |  35.4900   |   0.9312   |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma15-012ceb71.pth) \| log |
| [restormer_official_dfwb-color-sigma15](./restormer_official_dfwb-color-sigma15.py) | McMaster |    15     |  35.6072   |   0.9352   |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma15-012ceb71.pth) \| log |
| [restormer_official_dfwb-color-sigma15](./restormer_official_dfwb-color-sigma15.py) | Urban100 |    15     |  35.1522   |   0.9530   |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma15-012ceb71.pth) \| log |

| [restormer_official_dfwb-color-sigma25](./restormer_official_dfwb-color-sigma25.py) |  CBSD68  |    25     |  31.7457   |   0.8942   |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma25-e307f222.pth) \| log |
| [restormer_official_dfwb-color-sigma25](./restormer_official_dfwb-color-sigma25.py) | Kodak24  |    25     |  33.0489   |   0.8943   |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma25-e307f222.pth) \| log |
| [restormer_official_dfwb-color-sigma25](./restormer_official_dfwb-color-sigma25.py) | McMaster |    25     |  33.3260   |   0.9066   |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma25-e307f222.pth) \| log |

| [restormer_official_dfwb-color-sigma25](./restormer_official_dfwb-color-sigma25.py) | Urban100 |    25     |  32.9670   |   0.9317   |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma25-e307f222.pth) \| log |
| [restormer_official_dfwb-color-sigma50](./restormer_official_dfwb-color-sigma50.py) |  CBSD68  |    50     |  28.5569   |   0.8127   |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma50-a991983d.pth) \| log |
| [restormer_official_dfwb-color-sigma50](./restormer_official_dfwb-color-sigma50.py) | Kodak24  |    50     |  30.0122   |   0.8238   |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma50-a991983d.pth) \| log |

| [restormer_official_dfwb-color-sigma50](./restormer_official_dfwb-color-sigma50.py) | McMaster |    50     |  30.2608   |   0.8515   |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma50-a991983d.pth) \| log |
| [restormer_official_dfwb-color-sigma50](./restormer_official_dfwb-color-sigma50.py) | Urban100 |    50     |  30.0230   |   0.8902   |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma50-a991983d.pth) \| log |

|                                  Model                                  | Dataset | $\\sigma$ | PSNR (RGB) | SSIM (RGB) |                                  Training Resources                                   | Download |

| :---------------------------------------------------------------------: | :-----: | :-------: | :--------: | :--------: | :-----------------------------------------------------------------------------------: | :------: |
| [restormer_official_dfwb-color-sigma15](./restormer_official_dfwb-color-sigma15.py) |   15    |  34.3422  |   0.9356   |     1      | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth) \| log |          |
| [restormer_official_dfwb-color-sigma15](./restormer_official_dfwb-color-sigma15.py) |   15    |  35.4544  |   0.9308   |     1      | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth) \| log |          |

| [restormer_official_dfwb-color-sigma15](./restormer_official_dfwb-color-sigma15.py) |   15    |  35.5473  |   0.9344   |     1      | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth) \| log |          |
| [restormer_official_dfwb-color-sigma15](./restormer_official_dfwb-color-sigma15.py) |   15    |  35.0754  |   0.9524   |     1      | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth) \| log |    1     |
| [restormer_official_dfwb-color-sigma25](./restormer_official_dfwb-color-sigma25.py) |   25    |  31.7391  |   0.8945   |     1      | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth) \| log |          |

| [restormer_official_dfwb-color-sigma25](./restormer_official_dfwb-color-sigma25.py) |   25    |  33.0380  |   0.8941   |     1      | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth) \| log |          |
| [restormer_official_dfwb-color-sigma25](./restormer_official_dfwb-color-sigma25.py) |   25    |  33.3040  |   0.9063   |     1      | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth) \| log |          |
| [restormer_official_dfwb-color-sigma25](./restormer_official_dfwb-color-sigma25.py) |   25    |  32.9165  |   0.9312   |     1      | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth) \| log |          |

| [restormer_official_dfwb-color-sigma50](./restormer_official_dfwb-color-sigma50.py) |   50    |  28.5582  |   0.8126   |     1      | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth) \| log |          |
| [restormer_official_dfwb-color-sigma50](./restormer_official_dfwb-color-sigma50.py) |   50    |  30.0074  |   0.8233   |     1      | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth) \| log |          |
| [restormer_official_dfwb-color-sigma50](./restormer_official_dfwb-color-sigma50.py) |   50    |  30.2671  |   0.8520   |     1      | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth) \| log |          |

| [restormer_official_dfwb-color-sigma50](./restormer_official_dfwb-color-sigma50.py) |   50    |  30.0172  |   0.8898   |     1      | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth) \| log |    1     |

### **Real Image Denoising**

|                          Model                          | Dataset |  PSNR   |  SSIM  | Training Resources |                                                Download                                                 |
| :-----------------------------------------------------: | :-----: | :-----: | :----: | :----------------: | :-----------------------------------------------------------------------------------------------------: |

| [restormer_official_sidd](./restormer_official_sidd.py) |  SIDD   | 40.0156 | 0.9225 |         1          | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_sidd-9e7025db.pth) \| log |

**Train**

You can refer to **Train a model** part in [train_test.md](/docs/en/user_guides/train_test.md#Train-a-model-in-MMagic).

**Test**

<details>

```shell
# cpu test
# Deraining
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/restormer/restormer_official_rain13k.py https://download.openmmlab.com/mmediting/restormer/restormer_official_rain13k-2be7b550.pth

# Motion Deblurring
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/restormer/restormer_official_gopro.py https://download.openmmlab.com/mmediting/restormer/restormer_official_gopro-db7363a0.pth

# Defocus Deblurring
# Single

# Single
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/restormer/restormer_official_dpdd-dual.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dpdd-single-6bc31582.pth
# Dual
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/restormer/restormer_official_dpdd-single.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dpdd-dual-52c94c00.pth

# Gaussian Denoising
# Test Grayscale Gaussian Noise
# sigma15
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/restormer/restormer_official_dfwb-gray-sigma15.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-sigma15-da74417f.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/restormer/restormer_official_dfwb-gray-sigma15.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-blind-5f094bcc.pth

# sigma25

# sigma25
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/restormer/restormer_official_dfwb-gray-sigma25.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-sigma25-08010841.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/restormer/restormer_official_dfwb-gray-sigma25.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-blind-5f094bcc.pth

# sigma50
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/restormer/restormer_official_dfwb-gray-sigma50.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-sigma50-ee852dfe.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/restormer/restormer_official_dfwb-gray-sigma50.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-blind-5f094bcc.pth

# Test Color Gaussian Noise
# sigma15

# sigma15
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/restormer/restormer_official_dfwb-color-sigma15.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma15-012ceb71.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/restormer/restormer_official_dfwb-color-sigma15.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth

# sigma25
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/restormer/restormer_official_dfwb-color-sigma25.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma25-e307f222.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/restormer/restormer_official_dfwb-color-sigma25.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth

# sigma50

# sigma50
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/restormer/restormer_official_dfwb-color-sigma50.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma50-a991983d.pth

CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/restormer/restormer_official_dfwb-color-sigma50.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth

python tools/test.py configs/restormer/restormer_official_rain13k.py https://download.openmmlab.com/mmediting/restormer/restormer_official_rain13k-2be7b550.pth

python tools/test.py configs/restormer/restormer_official_gopro.py https://download.openmmlab.com/mmediting/restormer/restormer_official_gopro-db7363a0.pth

# Single
python tools/test.py configs/restormer/restormer_official_dpdd-dual.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dpdd-single-6bc31582.pth
# Dual

# Single
# Dual
python tools/test.py configs/restormer/restormer_official_dpdd-single.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dpdd-dual-52c94c00.pth

# sigma15
python tools/test.py configs/restormer/restormer_official_dfwb-gray-sigma15.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-sigma15-da74417f.pth

python tools/test.py configs/restormer/restormer_official_dfwb-gray-sigma15.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-blind-5f094bcc.pth

# sigma25
python tools/test.py configs/restormer/restormer_official_dfwb-gray-sigma25.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-sigma25-08010841.pth

# sigma25

python tools/test.py configs/restormer/restormer_official_dfwb-gray-sigma25.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-blind-5f094bcc.pth

# sigma50
python tools/test.py configs/restormer/restormer_official_dfwb-gray-sigma50.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-sigma50-ee852dfe.pth

python tools/test.py configs/restormer/restormer_official_dfwb-gray-sigma50.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-blind-5f094bcc.pth

# sigma15
python tools/test.py configs/restormer/restormer_official_dfwb-color-sigma15.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma15-012ceb71.pth

# sigma15

python tools/test.py configs/restormer/restormer_official_dfwb-color-sigma15.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth

# sigma25
python tools/test.py configs/restormer/restormer_official_dfwb-color-sigma25.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma25-e307f222.pth

python tools/test.py configs/restormer/restormer_official_dfwb-color-sigma25.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth

# sigma50
python tools/test.py configs/restormer/restormer_official_dfwb-color-sigma50.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma50-a991983d.pth

# sigma50

python tools/test.py configs/restormer/restormer_official_dfwb-color-sigma50.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth

./tools/dist_test.sh configs/restormer/restormer_official_rain13k.py https://download.openmmlab.com/mmediting/restormer/restormer_official_rain13k-2be7b550.pth

./tools/dist_test.sh configs/restormer/restormer_official_gopro.py https://download.openmmlab.com/mmediting/restormer/restormer_official_gopro-db7363a0.pth

# Single
./tools/dist_test.sh configs/restormer/restormer_official_dpdd-dual.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dpdd-single-6bc31582.pth
# Dual

# Single
# Dual
./tools/dist_test.sh configs/restormer/restormer_official_dpdd-single.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dpdd-dual-52c94c00.pth

# sigma15
./tools/dist_test.sh configs/restormer/restormer_official_dfwb-gray-sigma15.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-sigma15-da74417f.pth

./tools/dist_test.sh configs/restormer/restormer_official_dfwb-gray-sigma15.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-blind-5f094bcc.pth

# sigma25
./tools/dist_test.sh configs/restormer/restormer_official_dfwb-gray-sigma25.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-sigma25-08010841.pth

# sigma25

./tools/dist_test.sh configs/restormer/restormer_official_dfwb-gray-sigma25.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-blind-5f094bcc.pth

# sigma50
./tools/dist_test.sh configs/restormer/restormer_official_dfwb-gray-sigma50.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-sigma50-ee852dfe.pth

./tools/dist_test.sh configs/restormer/restormer_official_dfwb-gray-sigma50.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-blind-5f094bcc.pth

# sigma15
./tools/dist_test.sh configs/restormer/restormer_official_dfwb-color-sigma15.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma15-012ceb71.pth

# sigma15

./tools/dist_test.sh configs/restormer/restormer_official_dfwb-color-sigma15.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth

# sigma25
./tools/dist_test.sh configs/restormer/restormer_official_dfwb-color-sigma25.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma25-e307f222.pth

./tools/dist_test.sh configs/restormer/restormer_official_dfwb-color-sigma25.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth

# sigma50
./tools/dist_test.sh configs/restormer/restormer_official_dfwb-color-sigma50.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma50-a991983d.pth

# sigma50

./tools/dist_test.sh configs/restormer/restormer_official_dfwb-color-sigma50.py https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth

```

```bibtex
    title={Restormer: Efficient Transformer for High-Resolution Image Restoration},
    author={Syed Waqas Zamir and Aditya Arora and Salman Khan and Munawar Hayat
            and Fahad Shahbaz Khan and Ming-Hsuan Yang},
    booktitle={CVPR},
}
```

# GN

> [Group Normalization](https://arxiv.org/abs/1803.08494)

Batch Normalization (BN) is a milestone technique in the development of deep learning, enabling various networks to train. However, normalizing along the batch dimension introduces problems --- BN's error increases rapidly when the batch size becomes smaller, caused by inaccurate batch statistics estimation. This limits BN's usage for training larger models and transferring features to computer vision tasks including detection, segmentation, and video, which require small batches constrained by memory consumption. In this paper, we present Group Normalization (GN) as a simple alternative to BN. GN divides the channels into groups and computes within each group the mean and variance for normalization. GN's computation is independent of batch sizes, and its accuracy is stable in a wide range of batch sizes. On ResNet-50 trained in ImageNet, GN has 10.6% lower error than its BN counterpart when using a batch size of 2; when using typical batch sizes, GN is comparably good with BN and outperforms other normalization variants. Moreover, GN can be naturally transferred from pre-training to fine-tuning. GN can outperform its BN-based counterparts for object detection and segmentation in COCO, and for video classification in Kinetics, showing that GN can effectively replace the powerful BN in a variety of tasks. GN can be easily implemented by a few lines of code in modern libraries.

<img src="https://user-images.githubusercontent.com/40661020/143889171-d4660307-3a4f-4ac2-a0ad-7d17c2f045b1.png" height="200"/>
</div>

|   Backbone    |   model    | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                         Config                          |                                                                                                                                                                  Download                                                                                                                                                                   |

| :-----------: | :--------: | :-----: | :------: | :------------: | :----: | :-----: | :-----------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| R-50-FPN (d)  | Mask R-CNN |   2x    |   7.1    |      11.0      |  40.2  |  36.4   |     [config](./mask-rcnn_r50_fpn_gn-all_2x_coco.py)     |                 [model](https://download.openmmlab.com/mmdetection/v2.0/gn/mask_rcnn_r50_fpn_gn-all_2x_coco/mask_rcnn_r50_fpn_gn-all_2x_coco_20200206-8eee02a6.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gn/mask_rcnn_r50_fpn_gn-all_2x_coco/mask_rcnn_r50_fpn_gn-all_2x_coco_20200206_050355.log.json)                 |

| R-50-FPN (d)  | Mask R-CNN |   3x    |   7.1    |       -        |  40.5  |  36.7   |     [config](./mask-rcnn_r50_fpn_gn-all_3x_coco.py)     |                 [model](https://download.openmmlab.com/mmdetection/v2.0/gn/mask_rcnn_r50_fpn_gn-all_3x_coco/mask_rcnn_r50_fpn_gn-all_3x_coco_20200214-8b23b1e5.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gn/mask_rcnn_r50_fpn_gn-all_3x_coco/mask_rcnn_r50_fpn_gn-all_3x_coco_20200214_063512.log.json)                 |
| R-101-FPN (d) | Mask R-CNN |   2x    |   9.9    |      9.0       |  41.9  |  37.6   |    [config](./mask-rcnn_r101_fpn_gn-all_2x_coco.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/gn/mask_rcnn_r101_fpn_gn-all_2x_coco/mask_rcnn_r101_fpn_gn-all_2x_coco_20200205-d96b1b50.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gn/mask_rcnn_r101_fpn_gn-all_2x_coco/mask_rcnn_r101_fpn_gn-all_2x_coco_20200205_234402.log.json)               |

| R-101-FPN (d) | Mask R-CNN |   3x    |   9.9    |                |  42.1  |  38.0   |    [config](./mask-rcnn_r101_fpn_gn-all_3x_coco.py)     |           [model](https://download.openmmlab.com/mmdetection/v2.0/gn/mask_rcnn_r101_fpn_gn-all_3x_coco/mask_rcnn_r101_fpn_gn-all_3x_coco_20200513_181609-0df864f4.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gn/mask_rcnn_r101_fpn_gn-all_3x_coco/mask_rcnn_r101_fpn_gn-all_3x_coco_20200513_181609.log.json)            |
| R-50-FPN (c)  | Mask R-CNN |   2x    |   7.1    |      10.9      |  40.0  |  36.1   | [config](./mask-rcnn_r50-contrib_fpn_gn-all_2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/gn/mask_rcnn_r50_fpn_gn-all_contrib_2x_coco/mask_rcnn_r50_fpn_gn-all_contrib_2x_coco_20200207-20d3e849.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gn/mask_rcnn_r50_fpn_gn-all_contrib_2x_coco/mask_rcnn_r50_fpn_gn-all_contrib_2x_coco_20200207_225832.log.json) |

| R-50-FPN (c)  | Mask R-CNN |   3x    |   7.1    |       -        |  40.1  |  36.2   | [config](./mask-rcnn_r50-contrib_fpn_gn-all_3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/gn/mask_rcnn_r50_fpn_gn-all_contrib_3x_coco/mask_rcnn_r50_fpn_gn-all_contrib_3x_coco_20200225-542aefbc.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gn/mask_rcnn_r50_fpn_gn-all_contrib_3x_coco/mask_rcnn_r50_fpn_gn-all_contrib_3x_coco_20200225_235135.log.json) |

- (d) means pretrained model converted from Detectron, and (c) means the contributed model pretrained by [@thangvubk](https://github.com/thangvubk).
- The `3x` schedule is epoch \[28, 34, 36\].
- **Memory, Train/Inf time is outdated.**

```latex
@inproceedings{wu2018group,
  title={Group Normalization},
  author={Wu, Yuxin and He, Kaiming},
}
```

# APCNet

> [Adaptive Pyramid Context Network for Semantic Segmentation](https://openaccess.thecvf.com/content_CVPR_2019/html/He_Adaptive_Pyramid_Context_Network_for_Semantic_Segmentation_CVPR_2019_paper.html)

<a href="https://github.com/Junjun2016/APCNet">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/decode_heads/apc_head.py#L111">Code Snippet</a>

Recent studies witnessed that context features can significantly improve the performance of deep semantic segmentation networks. Current context based segmentation methods differ with each other in how to construct context features and perform differently in practice. This paper firstly introduces three desirable properties of context features in segmentation task. Specially, we find that Global-guided Local Affinity (GLA) can play a vital role in constructing effective context features, while this property has been largely ignored in previous works. Based on this analysis, this paper proposes Adaptive Pyramid Context Network (APCNet)for semantic segmentation. APCNet adaptively constructs multi-scale contextual representations with multiple welldesigned Adaptive Context Modules (ACMs). Specifically, each ACM leverages a global image representation as a guidance to estimate the local affinity coefficients for each sub-region, and then calculates a context vector with these affinities. We empirically evaluate our APCNet on three semantic segmentation and scene parsing datasets, including PASCAL VOC 2012, Pascal-Context, and ADE20K dataset. Experimental results show that APCNet achieves state-ofthe-art performance on all three benchmarks, and obtains a new record 84.2% on PASCAL VOC 2012 test set without MS COCO pre-trained and any post-processing.

<img src="https://user-images.githubusercontent.com/24582831/142898638-e1c0c6ae-9270-448e-aa01-bbac3a236db5.png" width="70%"/>
</div>

| APCNet | R-50-D8  | 512x1024  |   40000 | 7.7      | 3.57           | V100   | 78.02 |         79.26 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/apcnet/apcnet_r50-d8_4xb2-40k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r50-d8_512x1024_40k_cityscapes/apcnet_r50-d8_512x1024_40k_cityscapes_20201214_115717-5e88fa33.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r50-d8_512x1024_40k_cityscapes/apcnet_r50-d8_512x1024_40k_cityscapes-20201214_115717.log.json)     |

| APCNet | R-101-D8 | 512x1024  |   40000 | 11.2     | 2.15           | V100   | 79.08 |         80.34 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/apcnet/apcnet_r101-d8_4xb2-40k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r101-d8_512x1024_40k_cityscapes/apcnet_r101-d8_512x1024_40k_cityscapes_20201214_115716-abc9d111.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r101-d8_512x1024_40k_cityscapes/apcnet_r101-d8_512x1024_40k_cityscapes-20201214_115716.log.json) |

| APCNet | R-50-D8  | 769x769   |   40000 | 8.7      | 1.52           | V100   | 77.89 |         79.75 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/apcnet/apcnet_r50-d8_4xb2-40k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r50-d8_769x769_40k_cityscapes/apcnet_r50-d8_769x769_40k_cityscapes_20201214_115717-2a2628d7.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r50-d8_769x769_40k_cityscapes/apcnet_r50-d8_769x769_40k_cityscapes-20201214_115717.log.json)         |

| APCNet | R-101-D8 | 769x769   |   40000 | 12.7     | 1.03           | V100   | 77.96 |         79.24 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/apcnet/apcnet_r101-d8_4xb2-40k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r101-d8_769x769_40k_cityscapes/apcnet_r101-d8_769x769_40k_cityscapes_20201214_115718-b650de90.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r101-d8_769x769_40k_cityscapes/apcnet_r101-d8_769x769_40k_cityscapes-20201214_115718.log.json)     |

| APCNet | R-50-D8  | 512x1024  |   80000 | -        | -              | V100   | 78.96 |         79.94 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/apcnet/apcnet_r50-d8_4xb2-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r50-d8_512x1024_80k_cityscapes/apcnet_r50-d8_512x1024_80k_cityscapes_20201214_115716-987f51e3.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r50-d8_512x1024_80k_cityscapes/apcnet_r50-d8_512x1024_80k_cityscapes-20201214_115716.log.json)     |

| APCNet | R-101-D8 | 512x1024  |   80000 | -        | -              | V100   | 79.64 |         80.61 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/apcnet/apcnet_r101-d8_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r101-d8_512x1024_80k_cityscapes/apcnet_r101-d8_512x1024_80k_cityscapes_20201214_115705-b1ff208a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r101-d8_512x1024_80k_cityscapes/apcnet_r101-d8_512x1024_80k_cityscapes-20201214_115705.log.json) |

| APCNet | R-50-D8  | 769x769   |   80000 | -        | -              | V100   | 78.79 |         80.35 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/apcnet/apcnet_r50-d8_4xb2-80k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r50-d8_769x769_80k_cityscapes/apcnet_r50-d8_769x769_80k_cityscapes_20201214_115718-7ea9fa12.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r50-d8_769x769_80k_cityscapes/apcnet_r50-d8_769x769_80k_cityscapes-20201214_115718.log.json)         |

| APCNet | R-101-D8 | 769x769   |   80000 | -        | -              | V100   | 78.45 |         79.91 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/apcnet/apcnet_r101-d8_4xb2-80k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r101-d8_769x769_80k_cityscapes/apcnet_r101-d8_769x769_80k_cityscapes_20201214_115716-a7fbc2ab.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r101-d8_769x769_80k_cityscapes/apcnet_r101-d8_769x769_80k_cityscapes-20201214_115716.log.json)     |

### ADE20K

| APCNet | R-50-D8  | 512x512   |   80000 | 10.1     | 19.61          | V100   | 42.20 |         43.30 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/apcnet/apcnet_r50-d8_4xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r50-d8_512x512_80k_ade20k/apcnet_r50-d8_512x512_80k_ade20k_20201214_115705-a8626293.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r50-d8_512x512_80k_ade20k/apcnet_r50-d8_512x512_80k_ade20k-20201214_115705.log.json)         |

| APCNet | R-101-D8 | 512x512   |   80000 | 13.6     | 13.10          | V100   | 45.54 |         46.65 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/apcnet/apcnet_r101-d8_4xb4-80k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r101-d8_512x512_80k_ade20k/apcnet_r101-d8_512x512_80k_ade20k_20201214_115704-c656c3fb.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r101-d8_512x512_80k_ade20k/apcnet_r101-d8_512x512_80k_ade20k-20201214_115704.log.json)     |

| APCNet | R-50-D8  | 512x512   |  160000 | -        | -              | V100   | 43.40 |         43.94 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/apcnet/apcnet_r50-d8_4xb4-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r50-d8_512x512_160k_ade20k/apcnet_r50-d8_512x512_160k_ade20k_20201214_115706-25fb92c2.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r50-d8_512x512_160k_ade20k/apcnet_r50-d8_512x512_160k_ade20k-20201214_115706.log.json)     |

| APCNet | R-101-D8 | 512x512   |  160000 | -        | -              | V100   | 45.41 |         46.63 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/apcnet/apcnet_r101-d8_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r101-d8_512x512_160k_ade20k/apcnet_r101-d8_512x512_160k_ade20k_20201214_115705-73f9a8d7.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/apcnet/apcnet_r101-d8_512x512_160k_ade20k/apcnet_r101-d8_512x512_160k_ade20k-20201214_115705.log.json) |

```bibtex
@InProceedings{He_2019_CVPR,
author = {He, Junjun and Deng, Zhongying and Zhou, Lei and Wang, Yali and Qiao, Yu},
title = {Adaptive Pyramid Context Network for Semantic Segmentation},
}
```

# Preparing LIVE1 Dataset

```bibtex
}
```

The test datasets can be download from [here](https://github.com/cszn/DnCNN/tree/master/testsets).

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── LIVE1
```

# Tutorial 2: Prepare datasets

In this section, we will detail how to prepare data and adopt the proper dataset in our repo for different methods.

We support multiple datasets of different tasks.
There are two ways to use datasets for training and testing models in MMagic:

1. Using downloaded datasets directly
2. Preprocessing downloaded datasets before using them.

The structure of this guide is as follows:

- [Tutorial 2: Prepare datasets](#tutorial-2-prepare-datasets)
  - [Download datasets](#download-datasets)
  - [Prepare datasets](#prepare-datasets)
  - [The overview of the datasets in MMagic](#the-overview-of-the-datasets-in-mmagic)

## Download datasets

You are supposed to download datasets from their homepage first.
Most datasets are available after downloaded, so you only need to make sure the folder structure is correct and further preparation is not necessary.

For example, you can simply prepare Vimeo90K-triplet datasets by downloading datasets from [homepage](http://toflow.csail.mit.edu/).

Some datasets need to be preprocessed before training or testing. We support many scripts to prepare datasets in [tools/dataset_converters](https://github.com/open-mmlab/mmagic/tree/main/tools/dataset_converters). And you can follow the tutorials of every dataset to run scripts.
For example, we recommend cropping the DIV2K images to sub-images. We provide a script to prepare cropped DIV2K dataset. You can run the following command:

```shell
```

## The overview of the datasets in MMagic

We support detailed tutorials and split them according to different tasks.

```

Please check our dataset zoo for data preparation of different tasks.

If you're interested in more details of datasets in MMagic, please check the [advanced guides](../howto/dataset.md).

# Dataset Visualization

## Introduce the dataset visualization tool

```bash
```

- `-o, --output-dir`: The output path for visualized images. If not specified, it will be set to `''`, which means not to save.
- **`-p, --phase`**: Phase of visualizing dataset，must be one of `['train', 'val', 'test']`. If not specified, it will be set to `'train'`.
- **`-n, --show-number`**: The number of samples to visualized. If not specified, display all images in the dataset.
- `--show-interval`: The interval of show (s).

- **`-m, --mode`**: The display mode, can be one of `['original', 'transformed', 'concat', 'pipeline']`. If not specified, it will be set to `'transformed'`.
- `-r, --rescale-factor`: The image rescale factor, which is useful if the output is too large or too small
  in the `original` mode.
- `-c, --channel-order`: The channel of the showing images, could be "BGR" or "RGB", If not specified, it will be set to 'BGR'.
- `--cfg-options` : Modifications to the configuration file, refer to [Learn about Configs](../user_guides/config.md).

```{note}
1. The `-m, --mode` is about display mode, display original pictures or transformed pictures or comparison pictures:
- "original" means show images load from disk;
- "transformed" means to show images after transformed;

- "concat" means show images stitched by "original" and "transformed" images;
- "pipeline" means show all the intermediate images throghout the pipeline.

2.  The `-r, --rescale-factor` option is set when the label information is too large or too small relative to the picture. For example, when visualizing the CIFAR dataset, since the resolution of the image is very small, `--rescale-factor` can be set to 10.
```

## How to visualize the original image

In **'original'** mode:

```shell
```

- `--phase val`: Visual validation set, can be simplified to `-p val`;
- `--output-dir tmp`: The visualization results are saved in the "tmp" folder, can be simplified to `-o tmp`;

```

- `--mode original`: Visualize the original image, can be simplified to `-m original`;
- `--show-number 100`: visualize 100 images, can be simplified to `-n 100`;
- `--rescale-factor`: the image is enlarged by 10 times, can be simplified to `-r 10`;
- `--channel-order RGB`: The channel order of the visualized image is "RGB", can be simplified to `-c RGB`.

## How to visualize the transformed images

In **'transformed'** mode:

```shell
```

```shell
```

## How to visualize the transformed images and original images together

In **'concat'** mode:

```shell
```

4. In **'pipeline'** mode：

```shell
```

```shell
```

```shell
```

We appreciate all contributions to improve MMRotate. Please refer to [CONTRIBUTING.md](https://github.com/open-mmlab/mmcv/blob/master/CONTRIBUTING.md) in MMCV for more details about the contributing guideline.

# Dataset Tools

## Animal Pose

<details>

```bibtex
}
```

}
```

For [Animal-Pose](https://sites.google.com/view/animal-pose/) dataset, the images and annotations can be downloaded from [official website](https://sites.google.com/view/animal-pose/). The script `tools/dataset_converters/parse_animalpose_dataset.py` converts raw annotations into the format compatible with MMPose. The pre-processed [annotation files](https://download.openmmlab.com/mmpose/datasets/animalpose_annotations.tar) are available. If you would like to generate the annotations by yourself, please follow:

1. Download the raw images and annotations and extract them under `$MMPOSE/data`. Make them look like this:

   ```text
   mmpose
   ├── mmpose
   ├── docs
   ├── tests
   ├── tools
   `── data
           │

├── mmpose
   ├── docs
   ├── tests
   ├── tools
   `── data
           │
           │
           │   │-- cat
           │   │-- cow
           │   │-- dog
           │   │-- horse
           │   │-- sheep
           │
           │   │-- cat
           │   │   │-- ...
           │   │-- cow
           │   │-- dog
           │   │-- horse
           │   │-- sheep
           │
           │   │-- cat
           │   │   │-- ...
           │   │-- cow
           │   │-- dog
           │   │-- horse

│   │-- cat
           │   │   │-- ...
           │   │-- cow
           │   │-- dog
           │   │-- horse
           │   │-- sheep
   ```

2. Run command

   ```bash
   ```

   The generated annotation files are put in `$MMPOSE/data/animalpose/annotations`.

## COFW

<details>

## COFW

<details>

```bibtex
}
```

For COFW data, please download from [COFW Dataset (Color Images)](https://data.caltech.edu/records/20099).
Move `COFW_train_color.mat` and `COFW_test_color.mat` to `$MMPOSE/data/cofw/` and make them look like:

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── cofw
```

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── cofw
```

Run `pip install h5py` first to install the dependency, then run the following script under `$MMPOSE`:

```bash
```

And you will get

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── cofw
        |── images
```

## DeepposeKit

<details>

```bibtex

```bibtex
}
```

}
```

For [Vinegar Fly](https://github.com/jgraving/DeepPoseKit-Data), [Desert Locust](https://github.com/jgraving/DeepPoseKit-Data), and [Grévy’s Zebra](https://github.com/jgraving/DeepPoseKit-Data) dataset, the annotations files can be downloaded from [DeepPoseKit-Data](https://github.com/jgraving/DeepPoseKit-Data). The script `tools/dataset_converters/parse_deepposekit_dataset.py` converts raw annotations into the format compatible with MMPose. The pre-processed annotation files are available at [vinegar_fly_annotations](https://download.openmmlab.com/mmpose/datasets/vinegar_fly_annotations.tar), [locust_annotations](https://download.openmmlab.com/mmpose/datasets/locust_annotations.tar), and [zebra_annotations](https://download.openmmlab.com/mmpose/datasets/zebra_annotations.tar). If you would like to generate the annotations by yourself, please follows:

   ```text
   mmpose
   ├── mmpose
   ├── docs
   ├── tests
   ├── tools
   `── data
       |
       |       |── fly
       |       |   |── ...
       |       |
       |       |   |── ...
       |       |
       |       `── zebra
       |           |── ...
       |
       │── fly
           `-- images
               │-- 0.jpg
               │-- 1.jpg
               │-- ...
   ```

       |           |── ...
       |
       │── fly
           `-- images
               │-- 0.jpg
               │-- 1.jpg
               │-- ...
   ```

   Note that the images can be downloaded from [vinegar_fly_images](https://download.openmmlab.com/mmpose/datasets/vinegar_fly_images.tar), [locust_images](https://download.openmmlab.com/mmpose/datasets/locust_images.tar), and [zebra_images](https://download.openmmlab.com/mmpose/datasets/zebra_images.tar).

   ```bash
   ```

   The generated annotation files are put in `$MMPOSE/data/fly/annotations`, `$MMPOSE/data/locust/annotations`, and `$MMPOSE/data/zebra/annotations`.

Since the official dataset does not provide the test set, we randomly select 90% images for training, and the rest (10%) for evaluation.

## Macaque

<details>

## Macaque

<details>

```bibtex
}
```

journal={bioRxiv},
}
```

For [MacaquePose](http://www2.ehub.kyoto-u.ac.jp/datasets/macaquepose/index.html) dataset, images and annotations can be downloaded from [download](http://www2.ehub.kyoto-u.ac.jp/datasets/macaquepose/index.html). The script `tools/dataset_converters/parse_macaquepose_dataset.py` converts raw annotations into the format compatible with MMPose. The pre-processed [macaque_annotations](https://download.openmmlab.com/mmpose/datasets/macaque_annotations.tar) are available. If you would like to generate the annotations by yourself, please follows:

   ```text
   mmpose
   ├── mmpose
   ├── docs
   ├── tests
   ├── tools
   `── data
           │-- images

mmpose
   ├── mmpose
   ├── docs
   ├── tests
   ├── tools
   `── data
           │-- images
           │   │-- ...
   ```

   ```bash
   ```

   The generated annotation files are put in `$MMPOSE/data/macaque/annotations`.

Since the official dataset does not provide the test set, we randomly select 12500 images for training, and the rest for evaluation.

<details>

```bibtex

```bibtex
}
```

For [Human3.6M](http://vision.imar.ro/human3.6m/description.php), please download from the official website and place the files under `$MMPOSE/data/h36m`.
Then run the [preprocessing script](/tools/dataset_converters/preprocess_h36m.py):

```bash
```

This will extract camera parameters and pose annotations at full framerate (50 FPS) and downsampled framerate (10 FPS). The processed data should have the following structure:

```

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    ├── h36m
        |   ├── fps50
        |   `── fps10
        `── images
            ├── S1

|       `── joint3d_stats.pkl
        `── images
            ├── S1
            |   |   ├── ...
            |   ├── ...
            ├── S5
            ├── S6
            ├── S7
            ├── S8
            ├── S9
            `── S11
```

After that, the annotations need to be transformed into COCO format which is compatible with MMPose. Please run:

```bash
```

## MPII

<details>

```bibtex

}
```

During training and inference for [MPII](http://human-pose.mpi-inf.mpg.de/), the prediction result will be saved as '.mat' format by default. We also provide a tool to convert this `.mat` to more readable `.json` format.

```shell
```

```shell
```

## Label Studio

<details>

```bibtex

<details>

```bibtex
  author={
}
```

For users of [Label Studio](https://github.com/heartexlabs/label-studio/), please follow the instructions in the [Label Studio to COCO document](./label_studio.md) to annotate and export the results as a Label Studio `.json` file. And save the `Code` from the `Labeling Interface` as an `.xml` file.

We provide a script to convert Label Studio `.json` annotation file to COCO `.json` format file. It can be used by running the following command:

```shell

```shell
```

```shell
```

# SNPE 支持情况

目前 mmdeploy 集成了 onnx2dlc 模型转换的 SDK 推理，但以下特性还不支持：

- GPU_FP16 模式
- DSP/AIP 量化
- 算子内部 profiling
- UDO 算子

# Tutorial 7: Useful tools

We provide lots of useful tools under `tools/` directory.

- [Tutorial 7: Useful tools](#tutorial-7-useful-tools)
  - [Get the FLOPs and params](#get-the-flops-and-params)
  - [Publish a model](#publish-a-model)
  - [Print full config](#print-full-config)

## Get the FLOPs and params

We provide a script adapted from [flops-counter.pytorch](https://github.com/sovrasov/flops-counter.pytorch) to compute the FLOPs and params of a given model.

```shell
```

```shell
python tools/analysis_tools/get_flops.py configs/resotorer/srresnet.py --shape 40 40
```

You will get the result like this.

```
Input shape: (3, 40, 40)
Flops: 4.07 GMac
Params: 1.52 M
```

```

```
```

**Note**: This tool is still experimental and we do not guarantee that the number is correct. You may well use the result for simple comparisons, but double check it before you adopt it in technical reports or papers.

(1) FLOPs are related to the input shape while parameters are not. The default input shape is (1, 3, 250, 250).
(2) Some operators are not counted in FLOPs like GN and custom operators.
You can add support for new operators by modifying [`mmcv/cnn/utils/flops_counter.py`](https://github.com/open-mmlab/mmcv/blob/master/mmcv/cnn/utils/flops_counter.py).

## Publish a model

3. compute the hash of the checkpoint file and append time and the hash id to the
   filename.

```shell

   filename.

```shell
```

E.g.,

```shell
python tools/model_converters/publish_model.py work_dirs/stylegan2/latest.pth stylegan2_c2_8xb4_ffhq-1024x1024.pth
```

The final output filename will be `stylegan2_c2_8xb4_ffhq-1024x1024_{time}-{hash id}.pth`.

## Print full config

MMGeneration incorporates config mechanism to set parameters used for training and testing models. With our [config](../user_guides/config.md) mechanism, users can easily conduct extensive experiments without hard coding. If you wish to inspect the config file, you may run `python tools/misc/print_config.py /PATH/TO/CONFIG` to see the complete config.

An Example:

```shell

```shell
python tools/misc/print_config.py configs/styleganv2/stylegan2_c2-PL_8xb4-fp16-partial-GD-no-scaler-800kiters_ffhq-256x256.py
```

# PSANet

> [PSANet: Point-wise Spatial Attention Network for Scene Parsing](https://openaccess.thecvf.com/content_ECCV_2018/papers/Hengshuang_Zhao_PSANet_Point-wise_Spatial_ECCV_2018_paper.pdf)

<a href="https://github.com/hszhao/PSANet">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/decode_heads/psa_head.py#L18">Code Snippet</a>

We notice information flow in convolutional neural networksis  restricted  inside  local  neighborhood  regions  due  to  the  physical  de-sign  of  convolutional  filters,  which  limits  the  overall  understanding  ofcomplex scenes. In this paper, we propose thepoint-wise  spatial  atten-tion network(PSANet) to relax the local neighborhood constraint. Eachposition on the feature map is connected to all the other ones througha self-adaptively learned attention mask. Moreover, information propa-gation in bi-direction for scene parsing is enabled. Information at otherpositions can be collected to help the prediction of the current positionand  vice  versa,  information  at  the  current  position  can  be  distributedto assist the prediction of other ones. Our proposed approach achievestop performance on various competitive scene parsing datasets, includ-ing  ADE20K,  PASCAL  VOC  2012  and  Cityscapes,  demonstrating  itseffectiveness and generality.

<img src="https://user-images.githubusercontent.com/24582831/142902367-0f29e8cb-5ac0-434b-98c4-b2af7c9c2e58.png" width="70%"/>
</div>

| PSANet | R-50-D8  | 512x1024  |   40000 | 7        | 3.17           | V100   | 77.63 |         79.04 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/psanet/psanet_r50-d8_4xb2-40k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r50-d8_512x1024_40k_cityscapes/psanet_r50-d8_512x1024_40k_cityscapes_20200606_103117-99fac37c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r50-d8_512x1024_40k_cityscapes/psanet_r50-d8_512x1024_40k_cityscapes_20200606_103117.log.json)     |

| PSANet | R-101-D8 | 512x1024  |   40000 | 10.5     | 2.20           | V100   | 79.14 |         80.19 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/psanet/psanet_r101-d8_4xb2-40k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r101-d8_512x1024_40k_cityscapes/psanet_r101-d8_512x1024_40k_cityscapes_20200606_001418-27b9cfa7.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r101-d8_512x1024_40k_cityscapes/psanet_r101-d8_512x1024_40k_cityscapes_20200606_001418.log.json) |

| PSANet | R-50-D8  | 769x769   |   40000 | 7.9      | 1.40           | V100   | 77.99 |         79.64 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/psanet/psanet_r50-d8_4xb2-40k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r50-d8_769x769_40k_cityscapes/psanet_r50-d8_769x769_40k_cityscapes_20200530_033717-d5365506.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r50-d8_769x769_40k_cityscapes/psanet_r50-d8_769x769_40k_cityscapes_20200530_033717.log.json)         |

| PSANet | R-101-D8 | 769x769   |   40000 | 11.9     | 0.98           | V100   | 78.43 |         80.26 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/psanet/psanet_r101-d8_4xb2-40k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r101-d8_769x769_40k_cityscapes/psanet_r101-d8_769x769_40k_cityscapes_20200530_035107-997da1e6.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r101-d8_769x769_40k_cityscapes/psanet_r101-d8_769x769_40k_cityscapes_20200530_035107.log.json)     |

| PSANet | R-50-D8  | 512x1024  |   80000 | -        | -              | V100   | 77.24 |         78.69 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/psanet/psanet_r50-d8_4xb2-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r50-d8_512x1024_80k_cityscapes/psanet_r50-d8_512x1024_80k_cityscapes_20200606_161842-ab60a24f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r50-d8_512x1024_80k_cityscapes/psanet_r50-d8_512x1024_80k_cityscapes_20200606_161842.log.json)     |

| PSANet | R-101-D8 | 512x1024  |   80000 | -        | -              | V100   | 79.31 |         80.53 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/psanet/psanet_r101-d8_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r101-d8_512x1024_80k_cityscapes/psanet_r101-d8_512x1024_80k_cityscapes_20200606_161823-0f73a169.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r101-d8_512x1024_80k_cityscapes/psanet_r101-d8_512x1024_80k_cityscapes_20200606_161823.log.json) |

| PSANet | R-50-D8  | 769x769   |   80000 | -        | -              | V100   | 79.31 |         80.91 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/psanet/psanet_r50-d8_4xb2-80k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r50-d8_769x769_80k_cityscapes/psanet_r50-d8_769x769_80k_cityscapes_20200606_225134-fe42f49e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r50-d8_769x769_80k_cityscapes/psanet_r50-d8_769x769_80k_cityscapes_20200606_225134.log.json)         |

| PSANet | R-101-D8 | 769x769   |   80000 | -        | -              | V100   | 79.69 |         80.89 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/psanet/psanet_r101-d8_4xb2-80k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r101-d8_769x769_80k_cityscapes/psanet_r101-d8_769x769_80k_cityscapes_20200606_214550-7665827b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r101-d8_769x769_80k_cityscapes/psanet_r101-d8_769x769_80k_cityscapes_20200606_214550.log.json)     |

### ADE20K

| PSANet | R-50-D8  | 512x512   |   80000 | 9        | 18.91          | V100   | 41.14 |         41.91 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/psanet/psanet_r50-d8_4xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r50-d8_512x512_80k_ade20k/psanet_r50-d8_512x512_80k_ade20k_20200614_144141-835e4b97.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r50-d8_512x512_80k_ade20k/psanet_r50-d8_512x512_80k_ade20k_20200614_144141.log.json)         |

| PSANet | R-101-D8 | 512x512   |   80000 | 12.5     | 13.13          | V100   | 43.80 |         44.75 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/psanet/psanet_r101-d8_4xb4-80k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r101-d8_512x512_80k_ade20k/psanet_r101-d8_512x512_80k_ade20k_20200614_185117-1fab60d4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r101-d8_512x512_80k_ade20k/psanet_r101-d8_512x512_80k_ade20k_20200614_185117.log.json)     |

| PSANet | R-50-D8  | 512x512   |  160000 | -        | -              | V100   | 41.67 |         42.95 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/psanet/psanet_r50-d8_4xb4-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r50-d8_512x512_160k_ade20k/psanet_r50-d8_512x512_160k_ade20k_20200615_161258-148077dd.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r50-d8_512x512_160k_ade20k/psanet_r50-d8_512x512_160k_ade20k_20200615_161258.log.json)     |

| PSANet | R-101-D8 | 512x512   |  160000 | -        | -              | V100   | 43.74 |         45.38 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/psanet/psanet_r101-d8_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r101-d8_512x512_160k_ade20k/psanet_r101-d8_512x512_160k_ade20k_20200615_161537-dbfa564c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r101-d8_512x512_160k_ade20k/psanet_r101-d8_512x512_160k_ade20k_20200615_161537.log.json) |

| PSANet | R-50-D8  | 512x512   |   20000 | 6.9      | 18.24          | V100   | 76.39 |         77.34 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/psanet/psanet_r50-d8_4xb4-20k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r50-d8_512x512_20k_voc12aug/psanet_r50-d8_512x512_20k_voc12aug_20200617_102413-2f1bbaa1.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r50-d8_512x512_20k_voc12aug/psanet_r50-d8_512x512_20k_voc12aug_20200617_102413.log.json)     |

| PSANet | R-101-D8 | 512x512   |   20000 | 10.4     | 12.63          | V100   | 77.91 |         79.30 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/psanet/psanet_r101-d8_4xb4-20k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r101-d8_512x512_20k_voc12aug/psanet_r101-d8_512x512_20k_voc12aug_20200617_110624-946fef11.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r101-d8_512x512_20k_voc12aug/psanet_r101-d8_512x512_20k_voc12aug_20200617_110624.log.json) |

| PSANet | R-50-D8  | 512x512   |   40000 | -        | -              | V100   | 76.30 |         77.35 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/psanet/psanet_r50-d8_4xb4-40k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r50-d8_512x512_40k_voc12aug/psanet_r50-d8_512x512_40k_voc12aug_20200613_161946-f596afb5.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r50-d8_512x512_40k_voc12aug/psanet_r50-d8_512x512_40k_voc12aug_20200613_161946.log.json)     |

| PSANet | R-101-D8 | 512x512   |   40000 | -        | -              | V100   | 77.73 |         79.05 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/psanet/psanet_r101-d8_4xb4-40k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r101-d8_512x512_40k_voc12aug/psanet_r101-d8_512x512_40k_voc12aug_20200613_161946-1f560f9e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/psanet/psanet_r101-d8_512x512_40k_voc12aug/psanet_r101-d8_512x512_40k_voc12aug_20200613_161946.log.json) |

```bibtex
@inproceedings{zhao2018psanet,
  title={Psanet: Point-wise spatial attention network for scene parsing},
  author={Zhao, Hengshuang and Zhang, Yi and Liu, Shu and Shi, Jianping and Change Loy, Chen and Lin, Dahua and Jia, Jiaya},
  pages={267--283},
}
```

# 第七章： TensorRT 自定义插件

## 介绍

在前面的模型部署入门系列文章中，我们介绍了部署一个 PyTorch 模型到推理后端，如 ONNXRuntime，这其中可能遇到很多工程性的问题。

有些可以通过创建 ONNX 节点来解决，该节点仍然使用后端原生的实现进行推理。而有些无法导出到后端的算法，可以通过重写代码改变算法的实现过程，同样可以导出到 ONNX ，达到一致的效果。以上两种方式一般可以处理绝大多数的部署问题，同时也不需要向推理框架引入新的内容，是我们进行模型部署时候的优先选择。

然而，仍然存在部分模型，模型中某些算子无法通过上述两种方式绕过问题，这时候，如何对特定后端实现对应代码就极为重要。这也是本文将介绍的第三种方式——**自定义插件**。

自定义插件是很多推理框架支持用户自定义算子的方式，以 MMDeploy 为例，它是一个支持多种推理后端的算法库。目前支持的后端有：

- ONNXRuntime
- TensorRT
- ncnn
- openvino
- PPLNN
  其中，前三种后端均实现了一些自定义的算子。例如 ONNXRuntime 中的调制可变性卷积，ncnn 中的topk 算子，TensorRT 中的 MultiLevelRoiAlign 。

介绍如何给后端自定义算子是一件相对复杂的事情，所以本文只针对其中一种后端 TensorRT，介绍自定义算子。如果读者对其他后端感兴趣，可以去他们的代码库查看，一般地，各个推理框架均有详细文档介绍如何添加客制化的算子实现。

## 在MMDeploy添加TensorRT插件

仍然以前面[教程二](./02_challenges.md)中的超分辨模型SRCNN为例。在教程二中，我们用 ONNXRuntime 作为后端，通过 PyTorch 的 symbolic 函数导出了一个支持动态 scale 的 ONNX 模型，这个模型可以直接用 ONNXRuntime 运行，这是因为 `NewInterpolate` 类导出的节点 `Resize` 就是ONNXRuntime支持的节点。下面我们尝试直接将教程二导出的 `srcnn3.onnx` 转换到TensorRT。

```python

```python
from mmdeploy.backend.tensorrt.utils import from_onnx

from_onnx(
    'srcnn3.onnx',
    'srcnn3',
    input_shapes=dict(
        input=dict(
            min_shape=[1, 3, 256, 256],
            opt_shape=[1, 3, 256, 256],
            max_shape=[1, 3, 256, 256]),
        factor=dict(
            min_shape=[4],
            opt_shape=[4],
            max_shape=[4])))
```

没有安装过MMDeploy的小伙伴可以先参考 [build](../01-how-to-build) 进行安装，安装完成后执行上述脚本，会有如下报错：

```shell
RuntimeError: Failed to parse onnx, In node 1 (importResize): UNSUPPORTED_NODE: Assertion failed: mode != "cubic" && "This version of TensorRT does not support cubic interpolation!"
```

报错的原因有以下两方面：

1. `srcnn3.onnx`文件中的 `Resize` 是 ONNX 原生节点。其插值方式之一 bicubic 并不被 TensorRT 支持（TensorRT 的 Resize Layer仅支持 nearest 和 bilinear 两种插值方式）。日志的错误信息也明确提示了这点；

```

2. 但即便将 "bicubic" 模式改为 "bilinear" ，转换仍然失败: `RuntimeError: Failed to parse onnx, In node 1 (importResize): UNSUPPORTED_NODE: Assertion failed: scales.is_weights() && Resize scales must be initializer!"`。这是因为 TensorRT 无法接受动态 scale 导致的。

### 创建ONNX节点

为解决上述问题，我们需要创建一个新的节点替换原生 Resize 节点，并且实现新节点对应的插件代码。

继续复用同样节点名的方式已经不可取，我们需要创建新的节点。改节点名称就叫 `Test::DynamicTRTResize`，这是种类C++的写法，`Test` 为域名，主要用于区分不同来源下的同名的节点，比如 `ONNX::` 和 `Test::`。当然了，ONNX本身也不存在 `DynamicTRTResize` 的节点名。

```python
import cv2
import os, requests

# Download checkpoint and test image
urls = ['https://download.openmmlab.com/mmediting/restorers/srcnn/srcnn_x4k915_1x16_1000k_div2k_20200608-4186f232.pth',

    'https://raw.githubusercontent.com/open-mmlab/mmagic/master/tests/data/face/000001.png']
names = ['srcnn.pth', 'face.png']
for url, name in zip(urls, names):
    if not os.path.exists(name):
        open(name, 'wb').write(requests.get(url).content)

class DynamicTRTResize(torch.autograd.Function):

    def symbolic(g, input, size_tensor, align_corners = False):
        """Symbolic function for creating onnx op."""
        return g.op(
            'Test::DynamicTRTResize',
            input,
            size_tensor,
            align_corners_i=align_corners)

    def forward(g, input, size_tensor, align_corners = False):
        """Run forward."""

input,

        size = [size_tensor.size(-2), size_tensor.size(-1)]
        return interpolate(
            input, size=size, mode='bicubic', align_corners=align_corners)

    def forward(self, x, size_tensor):
        x = DynamicTRTResize.apply(x, size_tensor)
        return out

out = self.relu(self.conv2(out))
        return out

factor = torch.rand([1, 1, 512, 512], dtype=torch.float)

cv2.imwrite("face_torch.png", torch_output)

dynamic_axes={
        'input': {
            3: 'width'
        },
        'factor': {
            0: 'batch1',
            2: 'height1',
            3: 'width1'
        },
            0: 'batch2',
            2: 'height2',
            3: 'width2'
        },
    }

    torch.onnx.export(
        model, (x, factor),
        "srcnn3.onnx",
        opset_version=11,
        input_names=['input', 'factor'],
        output_names=['output'],
        dynamic_axes=dynamic_axes)
```

执行上述脚本，我们导出成功了一个ONNX模型 `srcnn.onnx`。用[netron](https://netron.app/)打开这个模型可视化如下：

output_names=['output'],
```

  <img src="https://user-images.githubusercontent.com/28671653/241883709-e21d60d0-1b1d-4665-af14-9c1240484773.png"/>
</div>

直接将该模型转换成TensorRT模型也是不可行的，这是因为TensorRT还无法解析 `DynamicTRTResize` 节点。而想要解析该节点，我们必须为TensorRT添加c++代码，实现该插件。

### C++实现

因为MMDeploy中已经实现了Bicubic Interpolate算子，所以我们可以复用其中的CUDA部分代码，只针对TensorRT实现支持动态scale的插件即可。对CUDA编程感兴趣的小伙伴可以参考CUDA的[官方教程](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html)。因为 `csrc/backend_ops/tensorrt/bicubic_interpolate` 中有我们需要的CUDA代码，所以我们可以直接在该文件夹加添加TensorRT相关的trt_dynamic_resize.hpp和trt_dynamic_resize.cpp文件，在这两个文件中分别声明和实现插件就可以了。我们也可以新建文件夹 `csrc/backend_ops/tensorrt/dynamic_resize`，将这两个文件直接放到这个文件夹下。

对TensorRT 7+，要实现这样一个自定义插件，我们需要写两个类。

- `DynamicTRTResize`，继承自[nvinfer1::IPluginV2DynamicExt](https://docs.nvidia.com/deeplearning/tensorrt/api/c_api/classnvinfer1_1_1_i_plugin_v2_dynamic_ext.html)，完成插件的具体实现
- `DynamicTRTResizeCreator`，继承自[nvinfer1::IPluginCreator](https://docs.nvidia.com/deeplearning/tensorrt/api/c_api/classnvinfer1_1_1_i_plugin_creator.html)，是插件的工厂类，用于创建`DynamicTRTResize`插件的实例。

在MMDeploy中，由于有若干插件需要实现，所以我们在`mmdeploy/csrc/backend_ops/tensorrt/common/trt_plugin_base.hpp`中实现了`TRTPluginBase`和`TRTPluginCreatorBase`两个类，用于管理一些所有插件共有的属性方法。其中，`TRTPluginBase`是继承自`nvinfer1::IPluginV2DynamicExt`，而`TRTPluginCreatorBase`是继承自`nvinfer1::IPluginCreator`。这样，用户实现插件时只需继承这两个新的类即可。所以我们只需在`dynamic_resize`文件夹下.hpp文件中，引用`trt_plugin_base.hpp`头文件，然后实现类如下：

```cpp
class DynamicTRTResize : public TRTPluginBase{}
class DynamicTRTResizeCreator : public TRTPluginCreatorBase{}
```

在trt_dynamic_resize.hpp中，我们声明如下内容：

```cpp
#ifndef TRT_DYNAMIC_RESIZE_HPP
#define TRT_DYNAMIC_RESIZE_HPP

```

```cpp
#include <cublas_v2.h>

#include <memory>
#include <vector>

#include "trt_plugin_base.hpp"
namespace mmdeploy {
class DynamicTRTResize : public TRTPluginBase {
 public:
  DynamicTRTResize(const std::string &name, bool align_corners);

  DynamicTRTResize(const std::string name, const void *data, size_t length);

  DynamicTRTResize() = delete;

  // IPluginV2DynamicExt Methods
  nvinfer1::IPluginV2DynamicExt *clone() const TRT_NOEXCEPT override;
  nvinfer1::DimsExprs getOutputDimensions(int outputIndex, const nvinfer1::DimsExprs *inputs,
                                          int nbInputs, nvinfer1::IExprBuilder &exprBuilder)
      TRT_NOEXCEPT override;
  bool supportsFormatCombination(int pos, const nvinfer1::PluginTensorDesc *ioDesc, int nbInputs,

TRT_NOEXCEPT override;
                                 int nbOutputs) TRT_NOEXCEPT override;
  void configurePlugin(const nvinfer1::DynamicPluginTensorDesc *in, int nbInputs,
                       const nvinfer1::DynamicPluginTensorDesc *out,
                       int nbOutputs) TRT_NOEXCEPT override;
  size_t getWorkspaceSize(const nvinfer1::PluginTensorDesc *inputs, int nbInputs,
                          const nvinfer1::PluginTensorDesc *outputs,
                          int nbOutputs) const TRT_NOEXCEPT override;
  int enqueue(const nvinfer1::PluginTensorDesc *inputDesc,
              const nvinfer1::PluginTensorDesc *outputDesc, const void *const *inputs,
              void *const *outputs, void *workspace, cudaStream_t stream) TRT_NOEXCEPT override;

  // IPluginV2Ext Methods
  nvinfer1::DataType getOutputDataType(int index, const nvinfer1::DataType *inputTypes,

// IPluginV2Ext Methods
                                       int nbInputs) const TRT_NOEXCEPT override;

  // IPluginV2 Methods
  const char *getPluginType() const TRT_NOEXCEPT override;
  const char *getPluginVersion() const TRT_NOEXCEPT override;
  int getNbOutputs() const TRT_NOEXCEPT override;
  size_t getSerializationSize() const TRT_NOEXCEPT override;
  void serialize(void *buffer) const TRT_NOEXCEPT override;

 private:
  bool mAlignCorners;
};

class DynamicTRTResizeCreator : public TRTPluginCreatorBase {
 public:
  DynamicTRTResizeCreator();

  const char *getPluginName() const TRT_NOEXCEPT override;

  nvinfer1::IPluginV2 *createPlugin(const char *name, const nvinfer1::PluginFieldCollection *fc)

  nvinfer1::IPluginV2 *deserializePlugin(const char *name, const void *serialData,

                                         size_t serialLength) TRT_NOEXCEPT override;
};
}  // namespace mmdeploy
#endif  // TRT_DYNAMIC_RESIZE_HPP
```

在这样一份头文件中，DynamicTRTResize类进行了如下的套娃继承：

  <img src="https://user-images.githubusercontent.com/28671653/241883700-0bee87a0-6d6a-478b-8a71-983a4e47b670.png"/>
</div>

从上面的图片和代码中我们发现，插件类`DynamicTRTResize`中我们定义了私有变量`mAlignCorners`，该变量表示是否`align corners`。此外只要实现构造析构函数和TensoRT中三个基类的方法即可。其中构造函数有二，分别用于创建插件和反序列化插件。而基类方法中：

1. 基类`IPluginV2DynamicExt`的方法较为值得关注，`getOutputDimensions`获取输出张量的形状，`enqueue`真正负责执行我们的算法，内部一般会调用CUDA核函数。本文实现的插件直接调用MMDeploy已定义在`csrc/backend_ops/tensorrt/bicubic_interpolate`的核函数`bicubic_interpolate`。
2. 基类`IPluginV2Ext`的方法，我们只要实现获取输出数据类型的`getOutputDataType`即可。

3. 基类`IPluginV2`则是些获取插件类型和版本号的方法，此外则是序列化输入插件的参数的函数`serialize`和计算该参数的序列化后`buffer`大小的函数`getSerializationSize`，以及获取输出张量个数的方法`getNbOutputs`。还有部分公共方法被定义在`TRTPluginBase`类内了。

在插件工厂类 `DynamicTRTResizeCreator` 中，我们需要声明获取插件名称和版本的方法 `getPluginName` 和 `getPluginVersion`。同时我们还需要声明创建插件和反序列化插件的方法 `createPlugin` 和 `deserializePlugin`，前者调用 `DynamicTRTResize` 中创建插件的方法，后者调用反序列化插件的方法。

接下来，我们就实现上述声明吧。在.cpp文件中我们实现代码如下：

```cpp
// Copyright (c) OpenMMLab. All rights reserved
#include "trt_dynamic_resize.hpp"

#include <assert.h>

#include <chrono>

#include "trt_plugin_helper.hpp"
#include "trt_serialize.hpp"
// 引入CUDA核函数bicubic_interpolate在的头文件，会在enqueue中使用
#include "../bicubic_interpolate/trt_bicubic_interpolate_kernel.hpp"

namespace {
static const char *PLUGIN_VERSION{"1"};
static const char *PLUGIN_NAME{"DynamicTRTResize"};//插件名需和ONNX节点名一致，在转换TensorRT模型时被触发
}  // namespace

DynamicTRTResize::DynamicTRTResize(const std::string &name, bool align_corners)
    : TRTPluginBase(name), mAlignCorners(align_corners) {}

DynamicTRTResize::DynamicTRTResize(const std::string name, const void *data,
                                             size_t length)
    : TRTPluginBase(name) {
  deserialize_value(&data, &length, &mAlignCorners);
}

nvinfer1::IPluginV2DynamicExt *DynamicTRTResize::clone() const TRT_NOEXCEPT {
  DynamicTRTResize *plugin =
      new DynamicTRTResize(mLayerName, mAlignCorners);
  plugin->setPluginNamespace(getPluginNamespace());

  return plugin;
}

nvinfer1::DimsExprs DynamicTRTResize::getOutputDimensions(
    int outputIndex, const nvinfer1::DimsExprs *inputs, int nbInputs,
    nvinfer1::IExprBuilder &exprBuilder) TRT_NOEXCEPT {
  nvinfer1::DimsExprs ret;

int outputIndex, const nvinfer1::DimsExprs *inputs, int nbInputs,
  ret.nbDims = 4;
  // 输入张量有两个：input和size_tensor，后者只用于计算输出张量形状
  ret.d[0] = inputs[0].d[0];
  ret.d[1] = inputs[0].d[1];
  ret.d[2] = inputs[1].d[2];
  ret.d[3] = inputs[1].d[3];
  return ret;
}

bool DynamicTRTResize::supportsFormatCombination(int pos,
                                                      const nvinfer1::PluginTensorDesc *ioDesc,
                                                      int nbInputs, int nbOutputs) TRT_NOEXCEPT {
  if (pos == 0) {
    return (ioDesc[pos].type == nvinfer1::DataType::kFLOAT &&
            ioDesc[pos].format == nvinfer1::TensorFormat::kLINEAR);

  } else {
    return ioDesc[pos].type == ioDesc[0].type && ioDesc[pos].format == ioDesc[0].format;
  }
}

void DynamicTRTResize::configurePlugin(const nvinfer1::DynamicPluginTensorDesc *inputs,

} else {
  }
}

                                            int nbInputs,
                                            const nvinfer1::DynamicPluginTensorDesc *outputs,
                                            int nbOutputs) TRT_NOEXCEPT {}

size_t DynamicTRTResize::getWorkspaceSize(const nvinfer1::PluginTensorDesc *inputs,
                                               int nbInputs,
                                               const nvinfer1::PluginTensorDesc *outputs,
                                               int nbOutputs) const TRT_NOEXCEPT {
  return 0;
}

int DynamicTRTResize::enqueue(const nvinfer1::PluginTensorDesc *inputDesc,
                                   const nvinfer1::PluginTensorDesc *outputDesc,

return 0;
}

                                   const void *const *inputs, void *const *outputs, void *workSpace,
                                   cudaStream_t stream) TRT_NOEXCEPT {
  int batch = inputDesc[0].dims.d[0];
  int channels = inputDesc[0].dims.d[1];
  int height = inputDesc[0].dims.d[2];
  int width = inputDesc[0].dims.d[3];

  int height_out = outputDesc[0].dims.d[2];
  int width_out = outputDesc[0].dims.d[3];
  const void *x = inputs[0];
  void *output = outputs[0];

  // TODO: add fp16 support
  auto data_type = inputDesc[0].type;
  switch (data_type) {
    case nvinfer1::DataType::kFLOAT:
      bicubic_interpolate<float>((float *)x, (float *)output, batch, channels, height, width,
                                 height_out, width_out, mAlignCorners, stream);
      break;
    default:
      return 1;
      break;

height_out, width_out, mAlignCorners, stream);
      break;
    default:
      return 1;
      break;
  }

  return 0;
}

nvinfer1::DataType DynamicTRTResize::getOutputDataType(int index,
                                                            const nvinfer1::DataType *inputTypes,
                                                            int nbInputs) const TRT_NOEXCEPT {
  return inputTypes[0];
}

// IPluginV2 Methods
const char *DynamicTRTResize::getPluginType() const TRT_NOEXCEPT { return PLUGIN_NAME; }

const char *DynamicTRTResize::getPluginVersion() const TRT_NOEXCEPT { return PLUGIN_VERSION; }

int DynamicTRTResize::getNbOutputs() const TRT_NOEXCEPT { return 1; }

size_t DynamicTRTResize::getSerializationSize() const TRT_NOEXCEPT {
  return serialized_size(mAlignCorners);
}

void DynamicTRTResize::serialize(void *buffer) const TRT_NOEXCEPT {
  serialize_value(&buffer, mAlignCorners);
}

return serialized_size(mAlignCorners);
}

}

////////////////////// creator /////////////////////////////

DynamicTRTResizeCreator::DynamicTRTResizeCreator() {
  mPluginAttributes.clear();
  mPluginAttributes.emplace_back(nvinfer1::PluginField("align_corners"));
  mFC.nbFields = mPluginAttributes.size();
  mFC.fields = mPluginAttributes.data();
}

const char *DynamicTRTResizeCreator::getPluginName() const TRT_NOEXCEPT { return PLUGIN_NAME; }

const char *DynamicTRTResizeCreator::getPluginVersion() const TRT_NOEXCEPT {
  return PLUGIN_VERSION;
}

nvinfer1::IPluginV2 *DynamicTRTResizeCreator::createPlugin(
    const char *name, const nvinfer1::PluginFieldCollection *fc) TRT_NOEXCEPT {
  nvinfer1::Dims size{2, {1, 1}};
  bool align_corners = 1;

  for (int i = 0; i < fc->nbFields; i++) {
    if (fc->fields[i].data == nullptr) {
      continue;
    }

nvinfer1::Dims size{2, {1, 1}};

      continue;
    }
    std::string field_name(fc->fields[i].name);
    //获取align_corners值，用于创建插件DynamicTRTResize的实例
    if (field_name.compare("align_corners") == 0) {
      align_corners = static_cast<const int *>(fc->fields[i].data)[0];
    }
  }
  // 创建插件DynamicTRTResize实例并返回
  DynamicTRTResize *plugin = new DynamicTRTResize(name, align_corners);
}

nvinfer1::IPluginV2 *DynamicTRTResizeCreator::deserializePlugin(
    const char *name, const void *serialData, size_t serialLength) TRT_NOEXCEPT {
  auto plugin = new DynamicTRTResize(name, serialData, serialLength);
}
REGISTER_TENSORRT_PLUGIN(DynamicTRTResizeCreator);//真正注册了该插件
```

plugin->setPluginNamespace(getPluginNamespace());
}
```

然后，我们就对MMDeploy重新build一次TensorRT的动态库`build/lib/libmmdeploy_tensorrt_ops.so`。一般编译成功就表示已经注册算子了，但是我们需要进行一些测试以保证结果正确。

### 测试

我们用TensorRT的python api查看一下目前的插件列表：

```python
from mmdeploy.backend.tensorrt import load_tensorrt_plugin
load_tensorrt_plugin()
def get_plugin_names():
    return [pc.name for pc in trt.get_plugin_registry().plugin_creator_list]
print(get_plugin_names())
```

可以发现 'DynamicTRTResize' 在插件列表中。然后我们对这个插件进行功能测试，看推理结果是否和PyTroch结果一致，并且可以动态控制输出尺寸。

```python

engine = from_onnx(
    'srcnn3',
            min_shape=[1, 1, 256, 256],

min_shape=[1, 3, 256, 256],
            opt_shape=[1, 1, 512, 512],
            max_shape=[1, 1, 1024, 1024])))

from mmdeploy.backend.tensorrt import TRTWrapper

trt_model = TRTWrapper('srcnn3.engine', ['output'])

factor = torch.rand([1, 1, 768, 768], dtype=torch.float)
trt_output = trt_model.forward(dict(input=x.cuda(), factor=factor.cuda()))
torch_output = model.forward(x, factor)
assert np.allclose(
    trt_output['output'].cpu().numpy(),
    torch_output.cpu().detach(),
    rtol=1e-3,
    atol=1e-5)
```

对比 TensorRT 的输出结果和 PyTorch 的输出结果是否一致，程序如果不报错即可说明推理正确。此外，测试时我们使用和导出时不一样的尺寸，结果也和 PyTorch 一致，说明可以支持动态的尺寸。

## 总结

本篇教程我们主要讲述如何在 MMDeploy 代码库中添加一个自定义的 TensorRT 插件，整个过程不涉及太多更复杂的 CUDA 编程，相信小伙伴们学完可以自己实现想要的插件。

# Scratch

> [Rethinking ImageNet Pre-training](https://arxiv.org/abs/1811.08883)

We report competitive results on object detection and instance segmentation on the COCO dataset using standard models trained from random initialization. The results are no worse than their ImageNet pre-training counterparts even when using the hyper-parameters of the baseline system (Mask R-CNN) that were optimized for fine-tuning pre-trained models, with the sole exception of increasing the number of training iterations so the randomly initialized models may converge. Training from random initialization is surprisingly robust; our results hold even when: (i) using only 10% of the training data, (ii) for deeper and wider models, and (iii) for multiple tasks and metrics. Experiments show that ImageNet pre-training speeds up convergence early in training, but does not necessarily provide regularization or improve final target task accuracy. To push the envelope we demonstrate 50.9 AP on COCO object detection without using any external data---a result on par with the top COCO 2017 competition results that used ImageNet pre-training. These observations challenge the conventional wisdom of ImageNet pre-training for dependent tasks and we expect these discoveries will encourage people to rethink the current de facto paradigm of \`pre-training and fine-tuning' in computer vision.

<img src="https://user-images.githubusercontent.com/40661020/143974572-69c4f57d-0d6d-4f56-ba91-23f8a65a2a77.png" height="300"/>
</div>

|    Model     | Backbone |  Style  | Lr schd | box AP | mask AP |                          Config                           |                                                                                                                                                                                 Download                                                                                                                                                                                  |

| :----------: | :------: | :-----: | :-----: | :----: | :-----: | :-------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| Faster R-CNN | R-50-FPN | pytorch |   6x    |  40.7  |         | [config](./faster-rcnn_r50-scratch_fpn_gn-all_6x_coco.py) |     [model](https://download.openmmlab.com/mmdetection/v2.0/scratch/faster_rcnn_r50_fpn_gn-all_scratch_6x_coco/scratch_faster_rcnn_r50_fpn_gn_6x_bbox_mAP-0.407_20200201_193013-90813d01.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/scratch/faster_rcnn_r50_fpn_gn-all_scratch_6x_coco/scratch_faster_rcnn_r50_fpn_gn_6x_20200201_193013.log.json)     |

|  Mask R-CNN  | R-50-FPN | pytorch |   6x    |  41.2  |  37.4   |  [config](./mask-rcnn_r50-scratch_fpn_gn-all_6x_coco.py)  | [model](https://download.openmmlab.com/mmdetection/v2.0/scratch/mask_rcnn_r50_fpn_gn-all_scratch_6x_coco/scratch_mask_rcnn_r50_fpn_gn_6x_bbox_mAP-0.412__segm_mAP-0.374_20200201_193051-1e190a40.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/scratch/mask_rcnn_r50_fpn_gn-all_scratch_6x_coco/scratch_mask_rcnn_r50_fpn_gn_6x_20200201_193051.log.json) |

Note:

- The above models are trained with 16 GPUs.

```latex
@article{he2018rethinking,
  title={Rethinking imagenet pre-training},
  author={He, Kaiming and Girshick, Ross and Doll{\'a}r, Piotr},
  journal={arXiv preprint arXiv:1811.08883},
}
```

# ISIC-2016 Task1

This project support **`ISIC-2016 Task1 `**, and the dataset used in this project can be downloaded from [here](https://challenge.isic-archive.com/data/#2016).

The overarching goal of the challenge is to develop image analysis tools to enable the automated diagnosis of melanoma from dermoscopic images.

This challenge provides training data (~900 images) for participants to engage in all 3 components of lesion image analysis. A separate test dataset (~350 images) will be provided for participants to generate and submit automated results.

| [ISIC-2016 Task1](https://challenge.isic-archive.com/data/#2016) | full body         | segmentation | dermoscopy | 2            | 900/-/379-            | yes/-/yes              | 2016         | [CC-0](https://creativecommons.org/share-your-work/public-domain/cc0/) |

| background  |    900     |   82.08    |    -     |    -     |    379    |   81.98   |
| skin lesion |    900     |   17.92    |    -     |    -     |    379    |   18.02   |

Note:

Note:

![bac](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/dermoscopy/isic2016_task1/isic2016_task1.png)

All the commands below rely on the correct configuration of PYTHONPATH, which should point to the project's directory so that Python can locate the module files. In isic2016_task1/ root directory, run the following line to add the current directory to PYTHONPATH:

```shell
```

```shell
```

- download dataset from [here](https://challenge.isic-archive.com/data/#2016) and decompression data to path 'data/'.

```none
  ├── mmseg
  │   │   │   │   ├── isic2016_task1

```

```shell
```

```

```shell
```

```shell
```

```shell
```

## Results

### ISIC-2016 Task1

## Results

| fcn_unet_s5-d16 |   unet   |  512x512  |  0.01  |  -   |   -   |  [config](https://github.com/open-mmlab/mmsegmentation/tree/dev-1.x/projects/medical/2d_image/dermoscopy/isic2016_task1/configs/fcn-unet-s5-d16_unet_1xb16-0.01-20k_isic2016-task1-512x512.py)  |

| fcn_unet_s5-d16 |   unet   |  512x512  | 0.001  |  -   |   -   | [config](https://github.com/open-mmlab/mmsegmentation/tree/dev-1.x/projects/medical/2d_image/dermoscopy/isic2016_task1/configs/fcn-unet-s5-d16_unet_1xb16-0.001-20k_isic2016-task1-512x512.py)  |
| fcn_unet_s5-d16 |   unet   |  512x512  | 0.0001 |  -   |   -   | [config](https://github.com/open-mmlab/mmsegmentation/tree/dev-1.x/projects/medical/2d_image/dermoscopy/isic2016_task1/configs/fcn-unet-s5-d16_unet_1xb16-0.0001-20k_isic2016-task1-512x512.py) |

- [x] Training-time correctness

# Test

For image classification task and image retrieval task, you could test your model after training.

## Test with your PC

```shell
```

By default, MMPretrain prefers GPU to CPU. If you want to test a model on CPU, please empty `CUDA_VISIBLE_DEVICES` or set it to -1 to make GPU invisible to the program.

```bash
```
````

| `CHECKPOINT_FILE`                     | The path to the checkpoint file (It can be a http link, and you can find checkpoints [here](https://mmpretrain.readthedocs.io/en/latest/modelzoo_statistics.html)). |

| `--out OUT`                           | The path to save the file containing test results.                                                                                                                  |
| `--out-item OUT_ITEM`                 | To specify the content of the test results file, and it can be "pred" or "metrics". If "pred", save the outputs of the model for offline evaluation. If "metrics", save the evaluation metrics. Defaults to "pred". |

| `--tta`                               | Whether to enable the Test-Time-Aug (TTA). If the config file has `tta_pipeline` and `tta_model` fields, use them to determine the TTA transforms and how to merge the TTA results. Otherwise, use flip TTA by averaging classification score. |

## Test with multiple GPUs

```shell
```

```shell
```

| ARGS              | Description                                                                                                                                                         |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `CONFIG_FILE`     | The path to the config file.                                                                                                                                        |
| `CHECKPOINT_FILE` | The path to the checkpoint file (It can be a http link, and you can find checkpoints [here](https://mmpretrain.readthedocs.io/en/latest/modelzoo_statistics.html)). |

| `GPU_NUM`         | The number of GPUs to be used.                                                                                                                                      |
| `[PY_ARGS]`       | The other optional arguments of `tools/test.py`, see [here](#test-with-your-pc).                                                                                    |

```shell
```

```shell

```

```shell
```

## Test with multiple machines

```shell
```

```shell
```

```

If you run MMPretrain on a cluster managed with [slurm](https://slurm.schedmd.com/), you can use the script `tools/slurm_test.sh`.

```shell

```shell
```

| `PARTITION`       | The partition to use in your cluster.                                                                                                                               |

| `JOB_NAME`        | The name of your job, you can name it as you like.                                                                                                                  |

| `GPUS_PER_NODE` | The number of GPUs to be allocated per node.                                                               |

# NuScenes 数据集

本页提供了有关在 MMDetection3D 中使用 nuScenes 数据集的具体教程。

## 准备之前

您可以在[这里](https://www.nuscenes.org/download)下载 nuScenes 3D 检测数据 Full dataset (v1.0) 并解压缩所有 zip 文件。

如果您想进行 3D 语义分割任务，需要额外下载 nuScenes-lidarseg 数据标注，并将解压的文件放入 nuScenes 对应的文件夹下。

**注意**：nuScenes-lidarseg 中的 v1.0trainval(test)/categroy.json 会替换原先 Full dataset (v1.0) 原先的 v1.0trainval(test)/categroy.json，但是不会对 3D 目标检测任务造成影响。

在我们处理之前，文件夹结构应按如下方式组织。

```
├── tools
├── data
│   │   ├── maps
```

## 数据准备

我们通常需要通过特定样式来使用 `.pkl` 文件组织有用的数据信息。要为 nuScenes 准备这些文件，请运行以下命令：

```bash
```

处理后的文件夹结构应该如下。

```
├── tools
├── data
│   │   ├── maps

```

```
├── tools
├── data
│   │   ├── maps
```

- `nuscenes_database/xxxxx.bin`：训练数据集的每个 3D 包围框中包含的点云数据。
- `nuscenes_infos_train.pkl`：训练数据集，该字典包含了两个键值：`metainfo` 和 `data_list`。`metainfo` 包含数据集的基本信息，例如 `categories`, `dataset` 和 `info_version`。`data_list` 是由字典组成的列表，每个字典（以下简称 `info`）包含了单个样本的所有详细信息。
  - info\['sample_idx'\]：样本在整个数据集的索引。
  - info\['token'\]：样本数据标记。
  - info\['timestamp'\]：样本数据时间戳。
  - info\['ego2global'\]：自车到全局坐标的变换矩阵。（4x4 列表）
  - info\['lidar_points'\]：是一个字典，包含了所有与激光雷达点相关的信息。
    - info\['lidar_points'\]\['num_pts_feats'\]：点的特征维度。

- info\['lidar_points'\]：是一个字典，包含了所有与激光雷达点相关的信息。
    - info\['lidar_points'\]\['lidar2ego'\]：该激光雷达传感器到自车的变换矩阵。（4x4 列表）
  - info\['lidar_sweeps'\]：是一个列表，包含了扫描信息（没有标注的中间帧）。
    - info\['lidar_sweeps'\]\[i\]\['lidar_points'\]\['data_path'\]：第 i 次扫描的激光雷达数据的文件路径。
    - info\['lidar_sweeps'\]\[i\]\['lidar_points'\]\[lidar2ego''\]：当前激光雷达传感器到自车的变换矩阵。（4x4 列表）
    - info\['lidar_sweeps'\]\[i\]\['lidar_points'\]\['ego2global'\]：自车到全局坐标的变换矩阵。（4x4 列表）
    - info\['lidar_sweeps'\]\[i\]\['lidar2sensor'\]：从主激光雷达传感器到当前传感器（用于收集扫描数据）的变换矩阵。（4x4 列表）
    - info\['lidar_sweeps'\]\[i\]\['timestamp'\]：扫描数据的时间戳。
    - info\['lidar_sweeps'\]\[i\]\['sample_data_token'\]：扫描样本数据标记。
  - info\['images'\]：是一个字典，包含与每个相机对应的六个键值：`'CAM_FRONT'`, `'CAM_FRONT_RIGHT'`, `'CAM_FRONT_LEFT'`, `'CAM_BACK'`, `'CAM_BACK_LEFT'`, `'CAM_BACK_RIGHT'`。每个字典包含了对应相机的所有数据信息。

- info\['images'\]：是一个字典，包含与每个相机对应的六个键值：`'CAM_FRONT'`, `'CAM_FRONT_RIGHT'`, `'CAM_FRONT_LEFT'`, `'CAM_BACK'`, `'CAM_BACK_LEFT'`, `'CAM_BACK_RIGHT'`。每个字典包含了对应相机的所有数据信息。
    - info\['images'\]\['CAM_XXX'\]\['img_path'\]：图像的文件名。
    - info\['images'\]\['CAM_XXX'\]\['cam2img'\]：当 3D 点投影到图像平面时需要的内参信息相关的变换矩阵。（3x3 列表）
    - info\['images'\]\['CAM_XXX'\]\['sample_data_token'\]：图像样本数据标记。
    - info\['images'\]\['CAM_XXX'\]\['timestamp'\]：图像的时间戳。
    - info\['images'\]\['CAM_XXX'\]\['cam2ego'\]：该相机传感器到自车的变换矩阵。（4x4 列表）
    - info\['images'\]\['CAM_XXX'\]\['lidar2cam'\]：激光雷达传感器到该相机的变换矩阵。（4x4 列表）
  - info\['instances'\]：是一个字典组成的列表。每个字典包含单个实例的所有标注信息。对于其中的第 i 个实例，我们有：
    - info\['instances'\]\[i\]\['bbox_3d'\]：长度为 7 的列表，以 (x, y, z, l, w, h, yaw) 的顺序表示实例的 3D 边界框。
    - info\['instances'\]\[i\]\['bbox_label_3d'\]：整数表示实例的标签，-1 代表忽略。
    - info\['instances'\]\[i\]\['velocity'\]：3D 边界框的速度（由于不正确，没有垂直测量），大小为 (2, ) 的列表。
    - info\['instances'\]\[i\]\['num_lidar_pts'\]：每个 3D 边界框内包含的激光雷达点数。

- info\['instances'\]\[i\]\['velocity'\]：3D 边界框的速度（由于不正确，没有垂直测量），大小为 (2, ) 的列表。
    - info\['instances'\]\[i\]\['num_radar_pts'\]：每个 3D 边界框内包含的雷达点数。
    - info\['instances'\]\[i\]\['bbox_3d_isvalid'\]：每个包围框是否有效。一般情况下，我们只将包含至少一个激光雷达或雷达点的 3D 框作为有效框。
  - info\['cam_instances'\]：是一个字典，包含以下键值：`'CAM_FRONT'`, `'CAM_FRONT_RIGHT'`, `'CAM_FRONT_LEFT'`, `'CAM_BACK'`, `'CAM_BACK_LEFT'`, `'CAM_BACK_RIGHT'`。对于基于视觉的 3D 目标检测任务，我们将整个场景的 3D 标注划分至它们所属于的相应相机中。对于其中的第 i 个实例，我们有：
    - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['bbox_label'\]：实例标签。
    - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['bbox_label_3d'\]：实例标签。
    - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['bbox'\]：2D 边界框标注（3D 框投影的矩形框），顺序为 \[x1, y1, x2, y2\] 的列表。
    - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['center_2d'\]：3D 框投影到图像上的中心点，大小为 (2, ) 的列表。
    - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['depth'\]：3D 框投影中心的深度。

- info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['center_2d'\]：3D 框投影到图像上的中心点，大小为 (2, ) 的列表。
    - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['velocity'\]：3D 边界框的速度（由于不正确，没有垂直测量），大小为 (2, ) 的列表。
    - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['attr_label'\]：实例的属性标签。我们为属性分类维护了一个属性集合和映射。
    - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['bbox_3d'\]：长度为 7 的列表，以 (x, y, z, l, h, w, yaw) 的顺序表示实例的 3D 边界框。
  - info\['pts_semantic_mask_path'\]：激光雷达语义分割标注的文件名。

注意：

1. `instances` 和 `cam_instances` 中 `bbox_3d` 的区别。`bbox_3d` 都被转换到 MMDet3D 定义的坐标系下，`instances` 中的 `bbox_3d` 是在激光雷达坐标系下，而 `cam_instances` 是在相机坐标系下。注意它们 3D 框中表示的不同（'l, w, h' 和 'l, h, w'）。

2. 这里我们只解释训练信息文件中记录的数据。这同样适用于验证集和测试集（测试集的 `.pkl` 文件中不包含 `instances` 以及 `cam_instances`）。

获取 `nuscenes_infos_xxx.pkl` 的核心函数为 [\_fill_trainval_infos](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/nuscenes_converter.py#L146)。更多细节请参考 [nuscenes_converter.py](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/nuscenes_converter.py)。

## 训练流程

### 基于 LiDAR 的方法

nuScenes 上基于 LiDAR 的 3D 检测（包括多模态方法）的典型训练流程如下。

```python
    dict(
    dict(
    dict(

    dict(
]
```

与一般情况相比，nuScenes 有一个特定的 `'LoadPointsFromMultiSweeps'` 流水线来从连续帧加载点云。这是此设置中使用的常见做法。更多细节请参考 nuScenes [原始论文](https://arxiv.org/abs/1903.11027)。`'LoadPointsFromMultiSweeps'` 中的默认 `use_dim` 是 `[0, 1, 2, 4]`，其中前 3 个维度是指点坐标，最后一个是指时间戳差异。由于在拼接来自不同帧的点时使用点云的强度信息会产生噪声，因此默认情况下不使用点云的强度信息。

### 基于视觉的方法

nuScenes 上基于图像的 3D 检测的典型训练流水线如下。

```python
    dict(

```python
    dict(
    dict(type='mmdet.Resize', img_scale=(1600, 900), keep_ratio=True),
    dict(
        keys=[
        ]),
]
```

它遵循 2D 检测的一般流水线，但在一些细节上有所不同：

- 它使用单目流水线加载图像，其中包括额外的必需信息，如相机内参矩阵。
- 它需要加载 3D 标注。
- 一些数据增强技术需要调整，例如`RandomFlip3D`。目前我们不支持更多的增强方法，因为如何迁移和应用其他技术仍在探索中。

## 评估

使用 8 个 GPU 以及 nuScenes 指标评估的 PointPillars 的示例如下

```shell

## 评估

```shell
```

## 指标

NuScenes 提出了一个综合指标，即 nuScenes 检测分数（NDS），以评估不同的方法并设置基准测试。它由平均精度（mAP）、平均平移误差（ATE）、平均尺度误差（ASE）、平均方向误差（AOE）、平均速度误差（AVE）和平均属性误差（AAE）组成。更多细节请参考其[官方网站](https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Any)。

我们也采用这种方法对 nuScenes 进行评估。打印的评估结果示例如下：

```

```

## 测试和提交

使用 8 个 GPU 在 nuScenes 上测试 PointPillars 并生成对排行榜的提交的示例如下

你需要在对应的配置文件中的 `test_evaluator` 里修改 `jsonfile_prefix`。举个例子，添加 `test_evaluator = dict(type='NuScenesMetric', jsonfile_prefix='work_dirs/pp-nus/results_eval.json')` 或在测试命令后使用 `--cfg-options "test_evaluator.jsonfile_prefix=work_dirs/pp-nus/results_eval.json)`。

```shell

```shell
```

请注意，在[这里](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/datasets/nus-3d.py#L132)测试信息应更改为测试集而不是验证集。

生成 `work_dirs/pp-nus/results_eval.json` 后，您可以压缩并提交给 nuScenes 基准测试。更多信息请参考 [nuScenes 官方网站](https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Any)。

我们还可以使用我们开发的可视化工具将预测结果可视化。更多细节请参考[可视化文档](https://mmdetection3d.readthedocs.io/zh_CN/latest/useful_tools.html#id2)。

## 注意

### `NuScenesBox` 和我们的 `CameraInstanceBoxes` 之间的转换。

总的来说，`NuScenesBox` 和我们的 `CameraInstanceBoxes` 的主要区别主要体现在转向角（yaw）定义上。 `NuScenesBox` 定义了一个四元数或三个欧拉角的旋转，而我们的由于实际情况只定义了一个转向角（yaw），它需要我们在预处理和后处理中手动添加一些额外的旋转，例如[这里](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/datasets/nuscenes_mono_dataset.py#L673)。

另外，请注意，角点和位置的定义在 `NuScenesBox` 中是分离的。例如，在单目 3D 检测中，框位置的定义在其相机坐标中（有关汽车设置，请参阅其官方[插图](https://www.nuscenes.org/nuscenes#data-collection)），即与[我们的](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/core/bbox/structures/cam_box3d.py)一致。相比之下，它的角点是通过[惯例](https://github.com/nutonomy/nuscenes-devkit/blob/02e9200218977193a1058dd7234f935834378319/python-sdk/nuscenes/utils/data_classes.py#L527) 定义的，“x 向前， y 向左， z 向上”。它导致了与我们的 `CameraInstanceBoxes` 不同的维度和旋转定义理念。一个移除相似冲突的例子是 PR [#744](https://github.com/open-mmlab/mmdetection3d/pull/744)。同样的问题也存在于 LiDAR 系统中。为了解决它们，我们通常会在预处理和后处理中添加一些转换，以保证在整个训练和推理过程中框都在我们的坐标系系统里。

# Integral Human Pose Regression

<details>

```bibtex
}
```

booktitle={Proceedings of the European conference on computer vision (ECCV)},
}
```

State-of-the-art human pose estimation methods are based on heat map representation. In spite of the good performance, the representation has a few issues in nature, such as not differentiable and quantization error. This work shows that a simple integral operation relates and unifies the heat map representation and joint regression, thus avoiding the above issues. It is differentiable, efficient, and compatible with any heat map based methods. Its effectiveness is convincingly validated via comprehensive ablation experiments under various settings, specifically on 3D pose estimation, for the first time.

<img src="https://user-images.githubusercontent.com/13503330/189809459-026c7a9b-2260-4676-bc64-de92ce8784ce.png">
</div>

# Architecture of TurboMind

TurboMind is an inference engine that supports high throughput inference for conversational LLMs. It's based on NVIDIA's [FasterTransformer](https://github.com/NVIDIA/FasterTransformer). Major features of TurboMind include an efficient LLaMa implementation, the persistent batch inference model and an extendable KV cache manager.

## High level overview of TurboMind

```
  +--------------------+
  |        API         |
          |    ^
  request |    | stream callback
          v    |
  +--------------------+   fetch   +-------------------+
  |  Persistent Batch  | <-------> |  KV Cache Manager |
  +--------------------+   update  +-------------------+
             ^
             |
             v
+------------------------+
|  LLaMA implementation  |
| FT kernels & utilities |
```

## Persistent Batch

|
             v
```

You may recognize this feature as "continuous batching" in other repos. But during the concurrent development of the feature, we modeled the inference of a conversational LLM as a persistently running batch whose lifetime spans the entire serving process, hence the name "persistent batch". To put it simply

- The persistent batch as N pre-configured batch slots.
- Requests join the batch when there are free slots available. A batch slot is released and can be reused once the generation of the requested tokens is finished.
- __On cache-hits (see below), history tokens don't need to be decoded in every round of a conversation; generation of response tokens will start instantly.__
- The batch grows or shrinks automatically to minimize unnecessary computations.

## KV Cache Manager

The [KV cache manager](https://github.com/InternLM/lmdeploy/blob/main/src/turbomind/models/llama/LlamaCacheManager.h) of TurboMind is a memory-pool-liked object that also implements LRU policy so that it can be viewed as a form of __cache of KV caches__. It works in the following way

- All device memory required for KV cache is allocated by the manager. A fixed number of slots is pre-configured to match the memory size of the system. Each slot corresponds to the memory required by the KV cache of a single sequence. Allocation chunk-size can be configure to implement pre-allocate/on-demand style allocation policy (or something in-between).

- When space for the KV cache of a new sequence is requested but no free slots left in the pool, the least recently used sequence is evicted from the cache and its device memory is directly reused by the new sequence. However, this is not the end of the story.
- Fetching sequence currently resides in one of the slots resembles a _cache-hit_, the history KV cache is returned directly and no context decoding is needed.
- Victim (evicted) sequences are not erased entirely but converted to its most compact form, i.e. token IDs. When the same sequence id is fetched later (_cache-miss_) the token IDs will be decoded by FMHA backed context decoder and converted back to KV cache.
- The eviction and conversion are handled automatically inside TurboMind and thus transparent to the users. __From the user's aspect, system that use TurboMind has access to infinite device memory.__

## LLaMa implementation

Our implementation of the LLaMa family models is modified from Gpt-NeoX model in FasterTransformer. In addition to basic refactoring and modifications to support the LLaMa family, we made some improvements to enable high performance inference of conversational models, most importantly:

- To support fast context decoding in multi-round conversations. We replaced the attention implementation in context decoder with a [cutlass](https://github.com/NVIDIA/cutlass)-based FMHA implementation that supports mismatched Q/K lengths.
- We introduced indirect buffer pointers in both context FMHA and generation FMHA to support the discontinuity in KV cache within the batch.
- To support concurrent inference with persistent batch, new synchronization mechanism was designed to orchestrate the worker threads running in tensor parallel mode.

- To maximize the throughput, we implement INT8 KV cache support to increase the max batch size. It's effective because in real-world serving scenarios, KV cache costs more memory and consumes more memory bandwidth than weights or other activations.
- We resolved an NCCL hang issue when running multiple model instances in TP mode within a single process, NCCL APIs are now guarded by host-side synchronization barriers.

## API

TurboMind supports a Python API that enables streaming output and tensor parallel mode.

## API

The ability to use [tritonserver](https://github.com/triton-inference-server/server) for serving is also inherited from FasterTransformer. However, to support submitting concurrent requests into our persistent batch model, we no longer use sequence batching or dynamic batching as FasterTransformer does. The bookkeeping of request and sequence states are managed by TurboMind instead.

## Difference between FasterTransformer and TurboMind

Apart of the features described above, there are still many minor differences that we don't cover in this document. Notably, many capabilities of FT are dropped in TurboMind because of the difference in objectives (e.g. prefix prompt, beam search, context embedding, sparse GEMM, GPT/T5/other model families, etc)

## FAQ

### Supporting Huggingface models

## FAQ

For historical reasons, TurboMind's weight layout is based on [the original LLaMa implementation](https://github.com/facebookresearch/llama) (differ only by a transpose). The implementation in huggingface transformers uses a [different layout](https://github.com/huggingface/transformers/blob/45025d92f815675e483f32812caa28cce3a960e7/src/transformers/models/llama/convert_llama_weights_to_hf.py#L123C76-L123C76) for `W_q` and `W_k` which is handled in [deploy.py](https://github.com/InternLM/lmdeploy/blob/ff4648a1d09e5aec74cf70efef35bfaeeac552e0/lmdeploy/serve/turbomind/deploy.py#L398).

## Changelog

### V0.24.1 (5/1/2022)

- Fix `LayerDecayOptimizerConstructor` for MAE training ([#1539](https://github.com/open-mmlab/mmsegmentation/pull/1539), [#1540](https://github.com/open-mmlab/mmsegmentation/pull/1540))

### V0.24.0 (4/29/2022)

- Support MAE: Masked Autoencoders Are Scalable Vision Learners
- Support Resnet strikes back

- Support MAE: Masked Autoencoders Are Scalable Vision Learners ([#1307](https://github.com/open-mmlab/mmsegmentation/pull/1307), [#1523](https://github.com/open-mmlab/mmsegmentation/pull/1523))
- Support Resnet strikes back ([#1390](https://github.com/open-mmlab/mmsegmentation/pull/1390))
- Support extra dataloader settings in configs ([#1435](https://github.com/open-mmlab/mmsegmentation/pull/1435))

- Fix input previous results for the last cascade_decode_head ([#1450](https://github.com/open-mmlab/mmsegmentation/pull/1450))

- Fix validation loss logging ([#1494](https://github.com/open-mmlab/mmsegmentation/pull/1494))
- Fix the bug in binary_cross_entropy ([1527](https://github.com/open-mmlab/mmsegmentation/pull/1527))
- Support single channel prediction for Binary Cross Entropy Loss ([#1454](https://github.com/open-mmlab/mmsegmentation/pull/1454))
- Fix potential bugs in accuracy.py ([1496](https://github.com/open-mmlab/mmsegmentation/pull/1496))
- Avoid converting label ids twice by label map during evaluation ([1417](https://github.com/open-mmlab/mmsegmentation/pull/1417))
- Fix bug about label_map ([1445](https://github.com/open-mmlab/mmsegmentation/pull/1445))
- Fix image save path bug in Windows ([1423](https://github.com/open-mmlab/mmsegmentation/pull/1423))

- Fix MMSegmentation Colab demo ([1501](https://github.com/open-mmlab/mmsegmentation/pull/1501), [1452](https://github.com/open-mmlab/mmsegmentation/pull/1452))
- Migrate azure blob for beit checkpoints ([1503](https://github.com/open-mmlab/mmsegmentation/pull/1503))
- Fix bug in `tools/analyse_logs.py` caused by wrong plot_iter in some cases ([1428](https://github.com/open-mmlab/mmsegmentation/pull/1428))

- Merge BEiT and ConvNext's LR decay optimizer constructors ([#1438](https://github.com/open-mmlab/mmsegmentation/pull/1438))
- Register optimizer constructor with mmseg ([#1456](https://github.com/open-mmlab/mmsegmentation/pull/1456))
- Refactor transformer encode layer in ViT and BEiT backbone ([#1481](https://github.com/open-mmlab/mmsegmentation/pull/1481))

- Add `build_pos_embed` and `build_layers` for BEiT ([1517](https://github.com/open-mmlab/mmsegmentation/pull/1517))
- Add `with_cp` to mit and vit ([1431](https://github.com/open-mmlab/mmsegmentation/pull/1431))
- Fix inconsistent dtype of `seg_label` in stdc decode ([1463](https://github.com/open-mmlab/mmsegmentation/pull/1463))
- Delete random seed for training in `dist_train.sh` ([1519](https://github.com/open-mmlab/mmsegmentation/pull/1519))
- Revise high `workers_per_gpus` in config file ([#1506](https://github.com/open-mmlab/mmsegmentation/pull/1506))
- Add GPG keys and del mmcv version in Dockerfile ([1534](https://github.com/open-mmlab/mmsegmentation/pull/1534))
- Update checkpoint for model in deeplabv3plus ([#1487](https://github.com/open-mmlab/mmsegmentation/pull/1487))

- Add `DistSamplerSeedHook` to set epoch number to dataloader when runner is `EpochBasedRunner` ([1449](https://github.com/open-mmlab/mmsegmentation/pull/1449))
- Provide URLs of Swin Transformer pretrained models ([1389](https://github.com/open-mmlab/mmsegmentation/pull/1389))
- Updating Dockerfiles From Docker Directory and `get_started.md` to reach latest stable version of Python, PyTorch and MMCV ([1446](https://github.com/open-mmlab/mmsegmentation/pull/1446))

- Add more clearly statement of CPU training/inference ([1518](https://github.com/open-mmlab/mmsegmentation/pull/1518))

**Contributors**

- @jiangyitong made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1431
- @kahkeng made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1447

- @Nourollah made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1446
- @androbaza made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1452
- @Yzichen made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1445
- @whu-pzhang made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1423
- @panfeng-hover made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1417
- @Johnson-Wang made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1496
- @jere357 made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1460
- @mfernezir made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1494

- @donglixp made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1503
- @YuanLiuuuuuu made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1307
- @Dawn-bin made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1527

### V0.23.0 (4/1/2022)

- Support BEiT: BERT Pre-Training of Image Transformers
- Support K-Net: Towards Unified Image Segmentation
- Add `avg_non_ignore` of CELoss to support average loss over non-ignored elements
- Support dataset initialization with file client

- Support BEiT: BERT Pre-Training of Image Transformers ([#1404](https://github.com/open-mmlab/mmsegmentation/pull/1404))

- Support K-Net: Towards Unified Image Segmentation ([#1289](https://github.com/open-mmlab/mmsegmentation/pull/1289))
- Support dataset initialization with file client ([#1402](https://github.com/open-mmlab/mmsegmentation/pull/1402))
- Add class name function for STARE datasets ([#1376](https://github.com/open-mmlab/mmsegmentation/pull/1376))
- Support different seeds on different ranks when distributed training ([#1362](https://github.com/open-mmlab/mmsegmentation/pull/1362))
- Add `nlc2nchw2nlc` and `nchw2nlc2nchw` to simplify tensor with different dimension operation ([#1249](https://github.com/open-mmlab/mmsegmentation/pull/1249))

- Synchronize random seed for distributed sampler ([#1411](https://github.com/open-mmlab/mmsegmentation/pull/1411))

- Add script and documentation for multi-machine distributed training ([#1383](https://github.com/open-mmlab/mmsegmentation/pull/1383))

- Add `avg_non_ignore` of CELoss to support average loss over non-ignored elements ([#1409](https://github.com/open-mmlab/mmsegmentation/pull/1409))
- Fix some wrong URLs of models or logs in `./configs` ([#1336](https://github.com/open-mmlab/mmsegmentation/pull/1433))
- Add title and color theme arguments to plot function in `tools/confusion_matrix.py` ([#1401](https://github.com/open-mmlab/mmsegmentation/pull/1401))
- Fix outdated link in Colab demo ([#1392](https://github.com/open-mmlab/mmsegmentation/pull/1392))

- Fix typos ([#1424](https://github.com/open-mmlab/mmsegmentation/pull/1424), [#1405](https://github.com/open-mmlab/mmsegmentation/pull/1405), [#1371](https://github.com/open-mmlab/mmsegmentation/pull/1371), [#1366](https://github.com/open-mmlab/mmsegmentation/pull/1366), [#1363](https://github.com/open-mmlab/mmsegmentation/pull/1363))

- Add FAQ document ([#1420](https://github.com/open-mmlab/mmsegmentation/pull/1420))
- Fix the config name style description in official docs([#1414](https://github.com/open-mmlab/mmsegmentation/pull/1414))

- @kinglintianxia made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1371
- @CCODING04 made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1376
- @mob5566 made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1401

- @xiongnemo made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1392
- @Xiangxu-0103 made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1405

### V0.22.1 (3/9/2022)

- Fix the ZeroDivisionError that all pixels in one image is ignored. ([#1336](https://github.com/open-mmlab/mmsegmentation/pull/1336))

- Provide URLs of STDC, Segmenter and Twins pretrained models ([#1272](https://github.com/open-mmlab/mmsegmentation/pull/1357))

### V0.22 (3/04/2022)

- Support ConvNeXt: A ConvNet for the 2020s. Please use the latest MMClassification (0.21.0) to try it out.
- Support iSAID aerial Dataset.
- Officially Support inference on Windows OS.

- Support ConvNeXt: A ConvNet for the 2020s. ([#1216](https://github.com/open-mmlab/mmsegmentation/pull/1216))
- Support iSAID aerial Dataset. ([#1115](https://github.com/open-mmlab/mmsegmentation/pull/1115)
- Generating and plotting confusion matrix. ([#1301](https://github.com/open-mmlab/mmsegmentation/pull/1301))

- Refactor 4 decoder heads (ASPP, FCN, PSP, UPer): Split forward function into `_forward_feature` and `cls_seg`. ([#1299](https://github.com/open-mmlab/mmsegmentation/pull/1299))
- Add `min_size` arg in `Resize` to keep the shape after resize bigger than slide window. ([#1318](https://github.com/open-mmlab/mmsegmentation/pull/1318))
- Revise pre-commit-hooks. ([#1315](https://github.com/open-mmlab/mmsegmentation/pull/1315))
- Add win-ci. ([#1296](https://github.com/open-mmlab/mmsegmentation/pull/1296))

- Fix `mlp_ratio` type in Swin Transformer. ([#1274](https://github.com/open-mmlab/mmsegmentation/pull/1274))
- Fix path errors in `./demo` . ([#1269](https://github.com/open-mmlab/mmsegmentation/pull/1269))
- Fix bug in conversion of potsdam. ([#1279](https://github.com/open-mmlab/mmsegmentation/pull/1279))
- Make accuracy take into account `ignore_index`. ([#1259](https://github.com/open-mmlab/mmsegmentation/pull/1259))
- Add Pytorch HardSwish assertion in unit test. ([#1294](https://github.com/open-mmlab/mmsegmentation/pull/1294))
- Fix wrong palette value in vaihingen. ([#1292](https://github.com/open-mmlab/mmsegmentation/pull/1292))
- Fix the bug that SETR cannot load pretrain. ([#1293](https://github.com/open-mmlab/mmsegmentation/pull/1293))

- Update correct `In Collection` in metafile of each configs. ([#1239](https://github.com/open-mmlab/mmsegmentation/pull/1239))
- Upload completed STDC models. ([#1332](https://github.com/open-mmlab/mmsegmentation/pull/1332))
- Fix `DNLHead` exports onnx inference difference type Cast error. ([#1161](https://github.com/open-mmlab/mmsegmentation/pull/1332))

- @JiaYanhao made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1269
- @andife made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1281
- @SBCV made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1279
- @HJoonKwon made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1259
- @Tsingularity made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1290

- @Waterman0524 made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1115
- @MeowZheng made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1315
- @linfangjian01 made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1318

### V0.21.1 (2/9/2022)

- Fix typos in docs. ([#1263](https://github.com/open-mmlab/mmsegmentation/pull/1263))
- Fix repeating log by `setup_multi_processes`. ([#1267](https://github.com/open-mmlab/mmsegmentation/pull/1267))
- Upgrade isort in pre-commit hook. ([#1270](https://github.com/open-mmlab/mmsegmentation/pull/1270))

- Use MMCV load_state_dict func in ViT/Swin. ([#1272](https://github.com/open-mmlab/mmsegmentation/pull/1272))

- Add exception for PointRend for support CPU-only. ([#1271](https://github.com/open-mmlab/mmsegmentation/pull/1270))

### V0.21 (1/29/2022)

- Officially Support CPUs training and inference, please use the latest MMCV (1.4.4) to try it out.
- Support Segmenter: Transformer for Semantic Segmentation (ICCV'2021).
- Support ISPRS Potsdam and Vaihingen Dataset.
- Add Mosaic transform and `MultiImageMixDataset` class in `dataset_wrappers`.

- Support Segmenter: Transformer for Semantic Segmentation (ICCV'2021) ([#955](https://github.com/open-mmlab/mmsegmentation/pull/955))
- Support ISPRS Potsdam and Vaihingen Dataset ([#1097](https://github.com/open-mmlab/mmsegmentation/pull/1097), [#1171](https://github.com/open-mmlab/mmsegmentation/pull/1171))

- Add segformer‘s benchmark on cityscapes ([#1155](https://github.com/open-mmlab/mmsegmentation/pull/1155))
- Add auto resume ([#1172](https://github.com/open-mmlab/mmsegmentation/pull/1172))
- Add Mosaic transform and `MultiImageMixDataset` class in `dataset_wrappers` ([#1093](https://github.com/open-mmlab/mmsegmentation/pull/1093), [#1105](https://github.com/open-mmlab/mmsegmentation/pull/1105))
- Add log collector ([#1175](https://github.com/open-mmlab/mmsegmentation/pull/1175))

- New-style CPU training and inference ([#1251](https://github.com/open-mmlab/mmsegmentation/pull/1251))
- Add UNet benchmark with multiple losses supervision ([#1143](https://github.com/open-mmlab/mmsegmentation/pull/1143))

- Fix the model statistics in doc for readthedoc ([#1153](https://github.com/open-mmlab/mmsegmentation/pull/1153))
- Set random seed for `palette` if not given ([#1152](https://github.com/open-mmlab/mmsegmentation/pull/1152))
- Add `COCOStuffDataset` in `class_names.py` ([#1222](https://github.com/open-mmlab/mmsegmentation/pull/1222))
- Fix bug in non-distributed multi-gpu training/testing ([#1247](https://github.com/open-mmlab/mmsegmentation/pull/1247))
- Delete unnecessary lines of STDCHead ([#1231](https://github.com/open-mmlab/mmsegmentation/pull/1231))

- @jbwang1997 made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1152
- @BeaverCC made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1206

- @Echo-minn made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1214
- @rstrudel made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/955

### V0.20.2 (12/15/2021)

- Revise --option to --options to avoid BC-breaking. ([#1140](https://github.com/open-mmlab/mmsegmentation/pull/1140))

### V0.20.1 (12/14/2021)

- Change options to cfg-options ([#1129](https://github.com/open-mmlab/mmsegmentation/pull/1129))

- Fix `<!-- [ABSTRACT] -->` in metafile. ([#1127](https://github.com/open-mmlab/mmsegmentation/pull/1127))
- Fix correct `num_classes` of HRNet in `LoveDA` dataset ([#1136](https://github.com/open-mmlab/mmsegmentation/pull/1136))

### V0.20 (12/10/2021)

- Support Twins ([#989](https://github.com/open-mmlab/mmsegmentation/pull/989))

- Support a real-time segmentation model STDC ([#995](https://github.com/open-mmlab/mmsegmentation/pull/995))
- Support a widely-used segmentation model in lane detection ERFNet ([#960](https://github.com/open-mmlab/mmsegmentation/pull/960))
- Support A Remote Sensing Land-Cover Dataset LoveDA ([#1028](https://github.com/open-mmlab/mmsegmentation/pull/1028))
- Support focal loss ([#1024](https://github.com/open-mmlab/mmsegmentation/pull/1024))

- Add SETR cityscapes benchmark ([#1087](https://github.com/open-mmlab/mmsegmentation/pull/1087))
- Add BiSeNetV1 COCO-Stuff 164k benchmark ([#1019](https://github.com/open-mmlab/mmsegmentation/pull/1019))
- Add Cutout transform ([#1022](https://github.com/open-mmlab/mmsegmentation/pull/1022))

- Set a random seed when the user does not set a seed ([#1039](https://github.com/open-mmlab/mmsegmentation/pull/1039))
- Add CircleCI setup ([#1086](https://github.com/open-mmlab/mmsegmentation/pull/1086))
- Skip CI on ignoring given paths ([#1078](https://github.com/open-mmlab/mmsegmentation/pull/1078))
- Add abstract and image for every paper ([#1060](https://github.com/open-mmlab/mmsegmentation/pull/1060))

- Create a symbolic link on windows ([#1090](https://github.com/open-mmlab/mmsegmentation/pull/1090))
- Support video demo using trained model ([#1014](https://github.com/open-mmlab/mmsegmentation/pull/1014))

- Fix incorrectly loading init_cfg or pretrained models of several transformer models ([#999](https://github.com/open-mmlab/mmsegmentation/pull/999), [#1069](https://github.com/open-mmlab/mmsegmentation/pull/1069), [#1102](https://github.com/open-mmlab/mmsegmentation/pull/1102))
- Fix EfficientMultiheadAttention in SegFormer ([#1037](https://github.com/open-mmlab/mmsegmentation/pull/1037))
- Remove `fp16` folder in `configs` ([#1031](https://github.com/open-mmlab/mmsegmentation/pull/1031))

- Fix several typos in .yml file (Dice Metric [#1041](https://github.com/open-mmlab/mmsegmentation/pull/1041), ADE20K dataset [#1120](https://github.com/open-mmlab/mmsegmentation/pull/1120), Training Memory (GB) [#1083](https://github.com/open-mmlab/mmsegmentation/pull/1083))
- Fix test error when using `--show-dir` ([#1091](https://github.com/open-mmlab/mmsegmentation/pull/1091))
- Fix dist training infinite waiting issue ([#1035](https://github.com/open-mmlab/mmsegmentation/pull/1035))
- Change the upper version of mmcv to 1.5.0 ([#1096](https://github.com/open-mmlab/mmsegmentation/pull/1096))
- Fix symlink failure on Windows ([#1038](https://github.com/open-mmlab/mmsegmentation/pull/1038))
- Cancel previous runs that are not completed ([#1118](https://github.com/open-mmlab/mmsegmentation/pull/1118))

- Unified links of readthedocs in docs ([#1119](https://github.com/open-mmlab/mmsegmentation/pull/1119))

- @Junjue-Wang made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1028
- @ddebby made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1066
- @del-zhenwu made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1078
- @KangBK0120 made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1106
- @zergzzlun made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1091
- @fingertap made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1035
- @irvingzhang0512 made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1014

- @littleSunlxy made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/989
- @lkm2835
- @RockeyCoss
- @MengzhangLI
- @Junjun2016
- @xiexinch
- @xvjiarui

### V0.19 (11/02/2021)

- Support TIMMBackbone wrapper ([#998](https://github.com/open-mmlab/mmsegmentation/pull/998))
- Support custom hook ([#428](https://github.com/open-mmlab/mmsegmentation/pull/428))
- Add codespell pre-commit hook ([#920](https://github.com/open-mmlab/mmsegmentation/pull/920))
- Add FastFCN benchmark on ADE20K ([#972](https://github.com/open-mmlab/mmsegmentation/pull/972))

- Add codespell pre-commit hook and fix typos ([#920](https://github.com/open-mmlab/mmsegmentation/pull/920))

- Make inputs & channels smaller in unittests ([#1004](https://github.com/open-mmlab/mmsegmentation/pull/1004))
- Change `self.loss_decode` back to `dict` in Single Loss situation ([#1002](https://github.com/open-mmlab/mmsegmentation/pull/1002))

- Fix typo in usage example ([#1003](https://github.com/open-mmlab/mmsegmentation/pull/1003))
- Add contiguous after permutation in ViT ([#992](https://github.com/open-mmlab/mmsegmentation/pull/992))
- Fix the invalid link ([#985](https://github.com/open-mmlab/mmsegmentation/pull/985))

- Fix bug in CI with python 3.9 ([#994](https://github.com/open-mmlab/mmsegmentation/pull/994))
- Fix bug when loading class name form file in custom dataset ([#923](https://github.com/open-mmlab/mmsegmentation/pull/923))

- @ShoupingShan made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/923
- @RockeyCoss made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/954
- @HarborYuan made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/992
- @lkm2835 made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/1003
- @gszh made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/428
- @VVsssssk

### V0.18 (10/07/2021)

- Support three real-time segmentation models (ICNet [#884](https://github.com/open-mmlab/mmsegmentation/pull/884), BiSeNetV1 [#851](https://github.com/open-mmlab/mmsegmentation/pull/851), and BiSeNetV2 [#804](https://github.com/open-mmlab/mmsegmentation/pull/804))
- Support one efficient segmentation model (FastFCN [#885](https://github.com/open-mmlab/mmsegmentation/pull/885))
- Support one efficient non-local/self-attention based segmentation model (ISANet [#70](https://github.com/open-mmlab/mmsegmentation/pull/70))
- Support COCO-Stuff 10k and 164k datasets ([#625](https://github.com/open-mmlab/mmsegmentation/pull/625))
- Support evaluate concated dataset separately ([#833](https://github.com/open-mmlab/mmsegmentation/pull/833))

- Support loading GT for evaluation from multi-file backend ([#867](https://github.com/open-mmlab/mmsegmentation/pull/867))

- Auto-convert SyncBN to BN when training on DP automatly([#772](https://github.com/open-mmlab/mmsegmentation/pull/772))
- Refactor Swin-Transformer ([#800](https://github.com/open-mmlab/mmsegmentation/pull/800))

- Update mmcv installation in dockerfile ([#860](https://github.com/open-mmlab/mmsegmentation/pull/860))
- Fix number of iteration bug when resuming checkpoint in distributed train ([#866](https://github.com/open-mmlab/mmsegmentation/pull/866))
- Fix parsing parse in val_step ([#906](https://github.com/open-mmlab/mmsegmentation/pull/906))

### V0.17 (09/01/2021)

- Support SegFormer
- Support DPT
- Support Dark Zurich and Nighttime Driving datasets
- Support progressive evaluation

- Support SegFormer ([#599](https://github.com/open-mmlab/mmsegmentation/pull/599))
- Support DPT ([#605](https://github.com/open-mmlab/mmsegmentation/pull/605))
- Support Dark Zurich and Nighttime Driving datasets ([#815](https://github.com/open-mmlab/mmsegmentation/pull/815))
- Support progressive evaluation ([#709](https://github.com/open-mmlab/mmsegmentation/pull/709))

- Add multiscale_output interface and unittests for HRNet ([#830](https://github.com/open-mmlab/mmsegmentation/pull/830))
- Support inherit cityscapes dataset ([#750](https://github.com/open-mmlab/mmsegmentation/pull/750))
- Fix some typos in README.md ([#824](https://github.com/open-mmlab/mmsegmentation/pull/824))

- Delete convert function and add instruction to ViT/Swin README.md ([#791](https://github.com/open-mmlab/mmsegmentation/pull/791))
- Add vit/swin/mit convert weight scripts ([#783](https://github.com/open-mmlab/mmsegmentation/pull/783))
- Add copyright files ([#796](https://github.com/open-mmlab/mmsegmentation/pull/796))

- Fix invalid checkpoint link in inference_demo.ipynb ([#814](https://github.com/open-mmlab/mmsegmentation/pull/814))
- Ensure that items in dataset have the same order across multi machine ([#780](https://github.com/open-mmlab/mmsegmentation/pull/780))
- Fix the log error ([#766](https://github.com/open-mmlab/mmsegmentation/pull/766))

### V0.16 (08/04/2021)

- Support SegFormer backbone MiT

- Support md2yml pre-commit hook
- Support frozen stage for HRNet

- Support SegFormer backbone MiT ([#594](https://github.com/open-mmlab/mmsegmentation/pull/594))
- Support md2yml pre-commit hook ([#732](https://github.com/open-mmlab/mmsegmentation/pull/732))
- Support mim ([#717](https://github.com/open-mmlab/mmsegmentation/pull/717))
- Add mmseg2torchserve tool ([#552](https://github.com/open-mmlab/mmsegmentation/pull/552))

- Support hrnet frozen stage ([#743](https://github.com/open-mmlab/mmsegmentation/pull/743))
- Add template of reimplementation questions ([#741](https://github.com/open-mmlab/mmsegmentation/pull/741))
- Output pdf and epub formats for readthedocs ([#742](https://github.com/open-mmlab/mmsegmentation/pull/742))

- Refine the docstring of ResNet ([#723](https://github.com/open-mmlab/mmsegmentation/pull/723))
- Replace interpolate with resize ([#731](https://github.com/open-mmlab/mmsegmentation/pull/731))
- Update resource limit ([#700](https://github.com/open-mmlab/mmsegmentation/pull/700))
- Update config.md ([#678](https://github.com/open-mmlab/mmsegmentation/pull/678))

- Fix ATTENTION registry ([#729](https://github.com/open-mmlab/mmsegmentation/pull/729))
- Fix analyze log script ([#716](https://github.com/open-mmlab/mmsegmentation/pull/716))
- Fix doc api display ([#725](https://github.com/open-mmlab/mmsegmentation/pull/725))
- Fix patch_embed and pos_embed mismatch error ([#685](https://github.com/open-mmlab/mmsegmentation/pull/685))
- Fix efficient test for multi-node ([#707](https://github.com/open-mmlab/mmsegmentation/pull/707))

- Fix init_cfg in resnet backbone ([#697](https://github.com/open-mmlab/mmsegmentation/pull/697))
- Fix efficient test bug ([#702](https://github.com/open-mmlab/mmsegmentation/pull/702))
- Fix url error in config docs ([#680](https://github.com/open-mmlab/mmsegmentation/pull/680))
- Fix mmcv installation ([#676](https://github.com/open-mmlab/mmsegmentation/pull/676))
- Fix torch version ([#670](https://github.com/open-mmlab/mmsegmentation/pull/670))

@sshuair @xiexinch @Junjun2016 @mmeendez8 @xvjiarui @sennnnn @puhsu @BIGWangYuDong @keke1u @daavoo

### V0.15 (07/04/2021)

- Support ViT, SETR, and Swin-Transformer
- Unified parameter initialization

- Fix typo and links ([#608](https://github.com/open-mmlab/mmsegmentation/pull/608))
- Fix Dockerfile ([#607](https://github.com/open-mmlab/mmsegmentation/pull/607))

- Fix ViT init ([#609](https://github.com/open-mmlab/mmsegmentation/pull/609))
- Fix mmcv version compatible table ([#658](https://github.com/open-mmlab/mmsegmentation/pull/658))
- Fix model links of DMNEt ([#660](https://github.com/open-mmlab/mmsegmentation/pull/660))

- Support loading DeiT weights ([#538](https://github.com/open-mmlab/mmsegmentation/pull/538))
- Support SETR ([#531](https://github.com/open-mmlab/mmsegmentation/pull/531), [#635](https://github.com/open-mmlab/mmsegmentation/pull/635))
- Add config and models for ViT backbone with UperHead ([#520](https://github.com/open-mmlab/mmsegmentation/pull/531), [#635](https://github.com/open-mmlab/mmsegmentation/pull/520))
- Support Swin-Transformer ([#511](https://github.com/open-mmlab/mmsegmentation/pull/511))

- Add higher accuracy FastSCNN ([#606](https://github.com/open-mmlab/mmsegmentation/pull/606))
- Add Chinese documentation ([#666](https://github.com/open-mmlab/mmsegmentation/pull/666))

- Unified parameter initialization ([#567](https://github.com/open-mmlab/mmsegmentation/pull/567))
- Separate CUDA and CPU in  github action CI ([#602](https://github.com/open-mmlab/mmsegmentation/pull/602))
- Support persistent dataloader worker ([#646](https://github.com/open-mmlab/mmsegmentation/pull/646))
- Update meta file fields ([#661](https://github.com/open-mmlab/mmsegmentation/pull/661), [#664](https://github.com/open-mmlab/mmsegmentation/pull/664))

### V0.14 (06/02/2021)

- Support ONNX to TensorRT

- Fix ONNX to TensorRT verify ([#547](https://github.com/open-mmlab/mmsegmentation/pull/547))

- Fix save best for EvalHook ([#575](https://github.com/open-mmlab/mmsegmentation/pull/575))

- Support ONNX to TensorRT ([#542](https://github.com/open-mmlab/mmsegmentation/pull/542))
- Support output results for ADE20k ([#544](https://github.com/open-mmlab/mmsegmentation/pull/544))
- Support MIM ([#549](https://github.com/open-mmlab/mmsegmentation/pull/549))

- Add option for ViT output shape ([#530](https://github.com/open-mmlab/mmsegmentation/pull/530))
- Infer batch size using len(result) ([#532](https://github.com/open-mmlab/mmsegmentation/pull/532))
- Add compatible table between MMSeg and MMCV ([#558](https://github.com/open-mmlab/mmsegmentation/pull/558))

### V0.13 (05/05/2021)

- Support Pascal Context Class-59 dataset.
- Support Visual Transformer Backbone.
- Support mFscore metric.

- Fixed Colaboratory tutorial ([#451](https://github.com/open-mmlab/mmsegmentation/pull/451))
- Fixed mIoU calculation range ([#471](https://github.com/open-mmlab/mmsegmentation/pull/471))
- Fixed sem_fpn, unet README.md ([#492](https://github.com/open-mmlab/mmsegmentation/pull/492))
- Fixed `num_classes` in FCN for Pascal Context 60-class dataset ([#488](https://github.com/open-mmlab/mmsegmentation/pull/488))
- Fixed FP16 inference ([#497](https://github.com/open-mmlab/mmsegmentation/pull/497))

- Support dynamic export and visualize to pytorch2onnx ([#463](https://github.com/open-mmlab/mmsegmentation/pull/463))

- Support export to torchscript ([#469](https://github.com/open-mmlab/mmsegmentation/pull/469), [#499](https://github.com/open-mmlab/mmsegmentation/pull/499))
- Support Pascal Context Class-59 dataset ([#459](https://github.com/open-mmlab/mmsegmentation/pull/459))
- Support Visual Transformer backbone ([#465](https://github.com/open-mmlab/mmsegmentation/pull/465))
- Support UpSample Neck ([#512](https://github.com/open-mmlab/mmsegmentation/pull/512))
- Support mFscore metric ([#509](https://github.com/open-mmlab/mmsegmentation/pull/509))

- Add more CI for PyTorch ([#460](https://github.com/open-mmlab/mmsegmentation/pull/460))
- Add print model graph args for tools/print_config.py ([#451](https://github.com/open-mmlab/mmsegmentation/pull/451))

- Add cfg links in modelzoo README.md ([#468](https://github.com/open-mmlab/mmsegmentation/pull/469))
- Add BaseSegmentor import to segmentors/__init__.py ([#495](https://github.com/open-mmlab/mmsegmentation/pull/495))
- Add MMOCR, MMGeneration links ([#501](https://github.com/open-mmlab/mmsegmentation/pull/501), [#506](https://github.com/open-mmlab/mmsegmentation/pull/506))
- Add Chinese QR code ([#506](https://github.com/open-mmlab/mmsegmentation/pull/506))
- Use MMCV MODEL_REGISTRY ([#515](https://github.com/open-mmlab/mmsegmentation/pull/515))
- Add ONNX testing tools ([#498](https://github.com/open-mmlab/mmsegmentation/pull/498))
- Replace data_dict calling 'img' key to support MMDet3D ([#514](https://github.com/open-mmlab/mmsegmentation/pull/514))

- Support reading class_weight from file in loss function ([#513](https://github.com/open-mmlab/mmsegmentation/pull/513))
- Make tags as comment ([#505](https://github.com/open-mmlab/mmsegmentation/pull/505))
- Use MMCV EvalHook ([#438](https://github.com/open-mmlab/mmsegmentation/pull/438))

### V0.12 (04/03/2021)

- Support FCN-Dilate 6 model.
- Support Dice Loss.

- Fixed PhotoMetricDistortion Doc ([#388](https://github.com/open-mmlab/mmsegmentation/pull/388))
- Fixed install scripts ([#399](https://github.com/open-mmlab/mmsegmentation/pull/399))
- Fixed Dice Loss multi-class ([#417](https://github.com/open-mmlab/mmsegmentation/pull/417))

- Support Dice Loss ([#396](https://github.com/open-mmlab/mmsegmentation/pull/396))
- Add plot logs tool ([#426](https://github.com/open-mmlab/mmsegmentation/pull/426))

- Add opacity option to show_result ([#425](https://github.com/open-mmlab/mmsegmentation/pull/425))
- Speed up mIoU metric ([#430](https://github.com/open-mmlab/mmsegmentation/pull/430))

- Refactor unittest file structure ([#440](https://github.com/open-mmlab/mmsegmentation/pull/440))
- Fix typos in the repo ([#449](https://github.com/open-mmlab/mmsegmentation/pull/449))
- Include class-level metrics in the log ([#445](https://github.com/open-mmlab/mmsegmentation/pull/445))

### V0.11 (02/02/2021)

- Support memory efficient test, add more UNet models.

- Fixed TTA resize scale ([#334](https://github.com/open-mmlab/mmsegmentation/pull/334))
- Fixed CI for pip 20.3 ([#307](https://github.com/open-mmlab/mmsegmentation/pull/307))

- Fixed ADE20k test ([#359](https://github.com/open-mmlab/mmsegmentation/pull/359))

- Support memory efficient test ([#330](https://github.com/open-mmlab/mmsegmentation/pull/330))
- Add more UNet benchmarks ([#324](https://github.com/open-mmlab/mmsegmentation/pull/324))
- Support Lovasz Loss ([#351](https://github.com/open-mmlab/mmsegmentation/pull/351))

- Move train_cfg/test_cfg inside model ([#341](https://github.com/open-mmlab/mmsegmentation/pull/341))

### V0.10 (01/01/2021)

- Support MobileNetV3, DMNet, APCNet. Add models of ResNet18V1b, ResNet18V1c, ResNet50V1b.

- Fixed CPU TTA ([#276](https://github.com/open-mmlab/mmsegmentation/pull/276))

- Add ResNet18V1b, ResNet18V1c, ResNet50V1b, ResNet101V1b models ([#316](https://github.com/open-mmlab/mmsegmentation/pull/316))
- Support MobileNetV3 ([#268](https://github.com/open-mmlab/mmsegmentation/pull/268))
- Add 4 retinal vessel segmentation benchmark  ([#315](https://github.com/open-mmlab/mmsegmentation/pull/315))
- Support DMNet ([#313](https://github.com/open-mmlab/mmsegmentation/pull/313))
- Support APCNet ([#299](https://github.com/open-mmlab/mmsegmentation/pull/299))

- Refactor Documentation page ([#311](https://github.com/open-mmlab/mmsegmentation/pull/311))

- Support resize data augmentation according to original image size ([#291](https://github.com/open-mmlab/mmsegmentation/pull/291))

### V0.9 (30/11/2020)

- Support 4 medical dataset, UNet and CGNet.

- Support RandomRotate transform ([#215](https://github.com/open-mmlab/mmsegmentation/pull/215), [#260](https://github.com/open-mmlab/mmsegmentation/pull/260))
- Support RGB2Gray transform ([#227](https://github.com/open-mmlab/mmsegmentation/pull/227))
- Support Rerange transform ([#228](https://github.com/open-mmlab/mmsegmentation/pull/228))
- Support ignore_index for BCE loss ([#210](https://github.com/open-mmlab/mmsegmentation/pull/210))
- Add modelzoo statistics ([#263](https://github.com/open-mmlab/mmsegmentation/pull/263))

- Support Dice evaluation metric ([#225](https://github.com/open-mmlab/mmsegmentation/pull/225))
- Support Adjust Gamma transform ([#232](https://github.com/open-mmlab/mmsegmentation/pull/232))
- Support CLAHE transform ([#229](https://github.com/open-mmlab/mmsegmentation/pull/229))

- Fixed detail API link ([#267](https://github.com/open-mmlab/mmsegmentation/pull/267))

### V0.8 (03/11/2020)

- Support customize runner ([#118](https://github.com/open-mmlab/mmsegmentation/pull/118))
- Support UNet ([#161](https://github.com/open-mmlab/mmsegmentation/pull/162))
- Support CHASE_DB1, DRIVE, STARE, HRD ([#203](https://github.com/open-mmlab/mmsegmentation/pull/203))

- Support CGNet ([#223](https://github.com/open-mmlab/mmsegmentation/pull/223))

### V0.7 (07/10/2020)

- Support Pascal Context dataset and customizing class dataset.

- Fixed CPU inference ([#153](https://github.com/open-mmlab/mmsegmentation/pull/153))

- Add DeepLab OS16 models ([#154](https://github.com/open-mmlab/mmsegmentation/pull/154))
- Support Pascal Context dataset ([#133](https://github.com/open-mmlab/mmsegmentation/pull/133))
- Support customizing dataset classes ([#71](https://github.com/open-mmlab/mmsegmentation/pull/71))
- Support customizing dataset palette ([#157](https://github.com/open-mmlab/mmsegmentation/pull/157))

- Support 4D tensor output in ONNX ([#150](https://github.com/open-mmlab/mmsegmentation/pull/150))

- Remove redundancies in ONNX export ([#160](https://github.com/open-mmlab/mmsegmentation/pull/160))
- Migrate to MMCV DepthwiseSeparableConv ([#158](https://github.com/open-mmlab/mmsegmentation/pull/158))
- Migrate to MMCV collect_env ([#137](https://github.com/open-mmlab/mmsegmentation/pull/137))
- Use img_prefix and seg_prefix for loading ([#153](https://github.com/open-mmlab/mmsegmentation/pull/153))

### V0.6 (10/09/2020)

- Support new methods i.e. MobileNetV2, EMANet, DNL, PointRend, Semantic FPN, Fast-SCNN, ResNeSt.

- Fixed sliding inference ONNX export ([#90](https://github.com/open-mmlab/mmsegmentation/pull/90))

- Support MobileNet v2 ([#86](https://github.com/open-mmlab/mmsegmentation/pull/86))
- Support EMANet ([#34](https://github.com/open-mmlab/mmsegmentation/pull/34))

- Support DNL ([#37](https://github.com/open-mmlab/mmsegmentation/pull/37))
- Support PointRend ([#109](https://github.com/open-mmlab/mmsegmentation/pull/109))
- Support Semantic FPN ([#94](https://github.com/open-mmlab/mmsegmentation/pull/94))
- Support Fast-SCNN ([#58](https://github.com/open-mmlab/mmsegmentation/pull/58))
- Support ResNeSt backbone ([#47](https://github.com/open-mmlab/mmsegmentation/pull/47))
- Support ONNX export (experimental) ([#12](https://github.com/open-mmlab/mmsegmentation/pull/12))

- Support Upsample in ONNX ([#100](https://github.com/open-mmlab/mmsegmentation/pull/100))
- Support Windows install (experimental) ([#75](https://github.com/open-mmlab/mmsegmentation/pull/75))
- Add more OCRNet results ([#20](https://github.com/open-mmlab/mmsegmentation/pull/20))

- Add PyTorch 1.6 CI ([#64](https://github.com/open-mmlab/mmsegmentation/pull/64))
- Get version and githash automatically ([#55](https://github.com/open-mmlab/mmsegmentation/pull/55))

### v0.5.1 (11/08/2020)

- Support FP16 and more generalized OHEM

- Fixed Pascal VOC conversion script (#19)
- Fixed OHEM weight assign bug (#54)
- Fixed palette type when palette is not given (#27)

- Support FP16 (#21)
- Generalized OHEM (#54)

- Add load-from flag (#33)
- Fixed training tricks doc about different learning rates of model (#26)

# ANN

> [Asymmetric Non-local Neural Networks for Semantic Segmentation](https://arxiv.org/abs/1908.07678)

<a href="https://github.com/MendelXu/ANN">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/decode_heads/ann_head.py#L185">Code Snippet</a>

The non-local module works as a particularly useful technique for semantic segmentation while criticized for its prohibitive computation and GPU memory occupation. In this paper, we present Asymmetric Non-local Neural Network to semantic segmentation, which has two prominent components: Asymmetric Pyramid Non-local Block (APNB) and Asymmetric Fusion Non-local Block (AFNB). APNB leverages a pyramid sampling module into the non-local block to largely reduce the computation and memory consumption without sacrificing the performance. AFNB is adapted from APNB to fuse the features of different levels under a sufficient consideration of long range dependencies and thus considerably improves the performance. Extensive experiments on semantic segmentation benchmarks demonstrate the effectiveness and efficiency of our work. In particular, we report the state-of-the-art performance of 81.3 mIoU on the Cityscapes test set. For a 256x128 input, APNB is around 6 times faster than a non-local block on GPU while 28 times smaller in GPU running memory occupation. Code is available at: [this https URL](https://github.com/MendelXu/ANN).

<img src="https://user-images.githubusercontent.com/24582831/142898322-3bbd578c-e488-4bae-9c14-7598adac5cbd.png" width="70%"/>
</div>

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                   | download                                                                                                                                                                                                                                                                                                                               |

| ANN    | R-50-D8  | 512x1024  |   40000 | 6        | 3.71           | V100   | 77.40 |         78.57 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ann/ann_r50-d8_4xb2-40k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r50-d8_512x1024_40k_cityscapes/ann_r50-d8_512x1024_40k_cityscapes_20200605_095211-049fc292.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r50-d8_512x1024_40k_cityscapes/ann_r50-d8_512x1024_40k_cityscapes_20200605_095211.log.json)     |

| ANN    | R-101-D8 | 512x1024  |   40000 | 9.5      | 2.55           | V100   | 76.55 |         78.85 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ann/ann_r101-d8_4xb2-40k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r101-d8_512x1024_40k_cityscapes/ann_r101-d8_512x1024_40k_cityscapes_20200605_095243-adf6eece.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r101-d8_512x1024_40k_cityscapes/ann_r101-d8_512x1024_40k_cityscapes_20200605_095243.log.json) |

| ANN    | R-50-D8  | 769x769   |   40000 | 6.8      | 1.70           | V100   | 78.89 |         80.46 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ann/ann_r50-d8_4xb2-40k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r50-d8_769x769_40k_cityscapes/ann_r50-d8_769x769_40k_cityscapes_20200530_025712-2b46b04d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r50-d8_769x769_40k_cityscapes/ann_r50-d8_769x769_40k_cityscapes_20200530_025712.log.json)         |

| ANN    | R-101-D8 | 769x769   |   40000 | 10.7     | 1.15           | V100   | 79.32 |         80.94 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ann/ann_r101-d8_4xb2-40k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r101-d8_769x769_40k_cityscapes/ann_r101-d8_769x769_40k_cityscapes_20200530_025720-059bff28.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r101-d8_769x769_40k_cityscapes/ann_r101-d8_769x769_40k_cityscapes_20200530_025720.log.json)     |

| ANN    | R-50-D8  | 512x1024  |   80000 | -        | -              | V100   | 77.34 |         78.65 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ann/ann_r50-d8_4xb2-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r50-d8_512x1024_80k_cityscapes/ann_r50-d8_512x1024_80k_cityscapes_20200607_101911-5a9ad545.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r50-d8_512x1024_80k_cityscapes/ann_r50-d8_512x1024_80k_cityscapes_20200607_101911.log.json)     |

| ANN    | R-101-D8 | 512x1024  |   80000 | -        | -              | V100   | 77.14 |         78.81 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ann/ann_r101-d8_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r101-d8_512x1024_80k_cityscapes/ann_r101-d8_512x1024_80k_cityscapes_20200607_013728-aceccc6e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r101-d8_512x1024_80k_cityscapes/ann_r101-d8_512x1024_80k_cityscapes_20200607_013728.log.json) |

| ANN    | R-50-D8  | 769x769   |   80000 | -        | -              | V100   | 78.88 |         80.57 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ann/ann_r50-d8_4xb2-80k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r50-d8_769x769_80k_cityscapes/ann_r50-d8_769x769_80k_cityscapes_20200607_044426-cc7ff323.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r50-d8_769x769_80k_cityscapes/ann_r50-d8_769x769_80k_cityscapes_20200607_044426.log.json)         |

| ANN    | R-101-D8 | 769x769   |   80000 | -        | -              | V100   | 78.80 |         80.34 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ann/ann_r101-d8_4xb2-80k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r101-d8_769x769_80k_cityscapes/ann_r101-d8_769x769_80k_cityscapes_20200607_013713-a9d4be8d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r101-d8_769x769_80k_cityscapes/ann_r101-d8_769x769_80k_cityscapes_20200607_013713.log.json)     |

### ADE20K

### ADE20K

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                               | download                                                                                                                                                                                                                                                                                                               |

| ANN    | R-50-D8  | 512x512   |   80000 | 9.1      | 21.01          | V100   | 41.01 |         42.30 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ann/ann_r50-d8_4xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r50-d8_512x512_80k_ade20k/ann_r50-d8_512x512_80k_ade20k_20200615_014818-26f75e11.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r50-d8_512x512_80k_ade20k/ann_r50-d8_512x512_80k_ade20k_20200615_014818.log.json)         |

| ANN    | R-101-D8 | 512x512   |   80000 | 12.5     | 14.12          | V100   | 42.94 |         44.18 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ann/ann_r101-d8_4xb4-80k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r101-d8_512x512_80k_ade20k/ann_r101-d8_512x512_80k_ade20k_20200615_014818-c0153543.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r101-d8_512x512_80k_ade20k/ann_r101-d8_512x512_80k_ade20k_20200615_014818.log.json)     |

| ANN    | R-50-D8  | 512x512   |  160000 | -        | -              | V100   | 41.74 |         42.62 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ann/ann_r50-d8_4xb4-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r50-d8_512x512_160k_ade20k/ann_r50-d8_512x512_160k_ade20k_20200615_231733-892247bc.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r50-d8_512x512_160k_ade20k/ann_r50-d8_512x512_160k_ade20k_20200615_231733.log.json)     |

| ANN    | R-101-D8 | 512x512   |  160000 | -        | -              | V100   | 42.94 |         44.06 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ann/ann_r101-d8_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r101-d8_512x512_160k_ade20k/ann_r101-d8_512x512_160k_ade20k_20200615_231733-955eb1ec.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r101-d8_512x512_160k_ade20k/ann_r101-d8_512x512_160k_ade20k_20200615_231733.log.json) |

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                | download                                                                                                                                                                                                                                                                                                                   |

| ANN    | R-50-D8  | 512x512   |   20000 | 6        | 20.92          | V100   | 74.86 |         76.13 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ann/ann_r50-d8_4xb4-20k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r50-d8_512x512_20k_voc12aug/ann_r50-d8_512x512_20k_voc12aug_20200617_222246-dfcb1c62.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r50-d8_512x512_20k_voc12aug/ann_r50-d8_512x512_20k_voc12aug_20200617_222246.log.json)     |

| ANN    | R-101-D8 | 512x512   |   20000 | 9.5      | 13.94          | V100   | 77.47 |         78.70 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ann/ann_r101-d8_4xb4-20k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r101-d8_512x512_20k_voc12aug/ann_r101-d8_512x512_20k_voc12aug_20200617_222246-2fad0042.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r101-d8_512x512_20k_voc12aug/ann_r101-d8_512x512_20k_voc12aug_20200617_222246.log.json) |

| ANN    | R-50-D8  | 512x512   |   40000 | -        | -              | V100   | 76.56 |         77.51 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ann/ann_r50-d8_4xb4-40k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r50-d8_512x512_40k_voc12aug/ann_r50-d8_512x512_40k_voc12aug_20200613_231314-b5dac322.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r50-d8_512x512_40k_voc12aug/ann_r50-d8_512x512_40k_voc12aug_20200613_231314.log.json)     |

| ANN    | R-101-D8 | 512x512   |   40000 | -        | -              | V100   | 76.70 |         78.06 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ann/ann_r101-d8_4xb4-40k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r101-d8_512x512_40k_voc12aug/ann_r101-d8_512x512_40k_voc12aug_20200613_231314-bd205bbe.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r101-d8_512x512_40k_voc12aug/ann_r101-d8_512x512_40k_voc12aug_20200613_231314.log.json) |

```bibtex
@inproceedings{zhu2019asymmetric,
  title={Asymmetric non-local neural networks for semantic segmentation},
  author={Zhu, Zhen and Xu, Mengde and Bai, Song and Huang, Tengteng and Bai, Xiang},
  pages={593--602},
}
```

## Usage

```shell
```

```bash
mim train mmaction configs/examplenet_r50-in1k-pre_8xb32-1x1x3-100e_kinetics400-rgb.py
```

```bash
```

```bash
mim train mmaction configs/examplenet_r50-in1k-pre_8xb32-1x1x3-100e_kinetics400-rgb.py --launcher pytorch --gpus 8
```

```bash
mim train mmaction configs/examplenet_r50-in1k-pre_8xb32-1x1x3-100e_kinetics400-rgb.py --launcher slurm \
```

```bash
mim test mmaction configs/examplenet_r50-in1k-pre_8xb32-1x1x3-100e_kinetics400-rgb.py --checkpoint $CHECKPOINT
```

```bash
mim test mmaction configs/examplenet_r50-in1k-pre_8xb32-1x1x3-100e_kinetics400-rgb.py --checkpoint $CHECKPOINT --launcher pytorch --gpus 8
```

```bash

```

```bash
mim test mmaction configs/examplenet_r50-in1k-pre_8xb32-1x1x3-100e_kinetics400-rgb.py --checkpoint $CHECKPOINT --launcher slurm \
```

## Results

| frame sampling strategy | resolution | gpus | backbone | pretrain | top1 acc | top5 acc |  testing protocol  |                    config                     |                                   ckpt |                            log |
| :---------------------: | :--------: | :--: | :------: | :------: | :------: | :------: | :----------------: | :-------------------------------------------: | -------------------------------------: | -----------------------------: |

|          1x1x3          |  224x224   |  8   | ResNet50 | ImageNet |  72.83   |  90.65   | 25 clips x 10 crop | [config](./configs/examplenet_r50-in1k-pre_8xb32-1x1x3-100e_kinetics400-rgb.py) | [ckpt](https://example/checkpoint/url) | [log](https://example/log/url) |

```bibtex
  title={OpenMMLab's Next Generation Video Understanding Toolbox and Benchmark},
  author={MMAction2 Contributors},
  howpublished = {\url{https://github.com/open-mmlab/mmaction2}},
}
```

Here is a checklist of this project's progress, and you can ignore this part if you don't plan to contribute to MMAction2 projects.

    <!-- The code's design shall follow existing interfaces and convention. For example, each model component should be registered into `mmaction.registry.MODELS` and configurable via a config file. -->

- [ ] Converted checkpoint and results (Only for reproduction)

    <!-- If you are reproducing the result from a paper, train your model from scratch and verified that the final result can match the original result. Usually, ±0.1% is acceptable for the action recognition task on Kinetics400. -->

    <!-- Unit tests for the major module are required. [Example](https://github.com/open-mmlab/mmaction2/blob/main/tests/models/backbones/test_resnet.py) -->

- [ ] Code style

    <!-- It will used for MMAction2 to acquire your models. [Example](https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/swin/metafile.yml). In particular, you may have to refactor this README into a standard one. [Example](https://github.com/open-mmlab/mmaction2/blob/main/configs/recognition/swin/README.md) -->

# LLaVA

> [Visual Instruction Tuning](https://arxiv.org/abs/2304.08485)

Instruction tuning large language models (LLMs) using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field. In this paper, we present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding.Our early experiments show that LLaVA demonstrates impressive multimodel chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images/instructions, and yields a 85.1% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%. We make GPT-4 generated visual instruction tuning data, our model and code base publicly available.

<img src="https://github-production-user-asset-6210df.s3.amazonaws.com/26739999/246466979-c2f41b71-1de3-4da8-b20a-eaebe722c339.png" width="80%"/>
</div>

**Prepare the checkpoint**

According to the license of LLaMA, we cannot provide the merged checkpoint directly. Please use the below
script to download and get the merged the checkpoint.

```shell
python tools/model_converters/llava-delta2mmpre.py huggyllama/llama-7b liuhaotian/LLaVA-Lightning-7B-delta-v1-1 ./LLaVA-Lightning-7B-delta-v1-1.pth
```

```python

model = get_model('llava-7b-v1_caption', pretrained='MERGED_CHECKPOINT_PATH', device='cuda')
# {'pred_caption': 'In the image, there are two cats sitting on a blanket.'}
```

```

Test:

```shell
python tools/test.py configs/llava/llava-7b-v1_caption.py MERGED_CHECKPOINT_PATH
```

| Model                 | Params (M) |  BLEU-4  |  CIDER   |              Config              |        Download        |
| :-------------------- | :--------: | :------: | :------: | :------------------------------: | :--------------------: |
| `llava-7b-v1_caption` |  7045.82   | Upcoming | Upcoming | [config](llava-7b-v1_caption.py) | See the above tutorial |

```bibtex
@misc{liu2023llava,
      title={Visual Instruction Tuning},

```bibtex
      author={Liu, Haotian and Li, Chunyuan and Wu, Qingyang and Lee, Yong Jae},
      publisher={arXiv:2304.08485},
}
```

# MMDetection 模型部署

- [MMDetection 模型部署](#mmdetection-模型部署)
    - [安装 mmdet](#安装-mmdet)

[MMDetection](https://github.com/open-mmlab/mmdetection) ，又称 `mmdet`， 是一个基于 PyTorch 的目标检测开源工具箱。它是 [OpenMMLab](https://openmmlab.com/) 项目的一部分。

## 安装

### 安装 mmdet

请参考[官网安装指南](https://mmdetection.readthedocs.io/en/3.x/get_started.html)。

```shell

```shell
```

## 模型转换

你可以使用 [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/blob/main/tools/deploy.py) 把 mmdet 模型一键式转换为推理后端模型。

以下，我们将演示如何把 `Faster R-CNN` 转换为 onnx 模型。

```shell
# download faster r-cnn model from mmdet model zoo
mim download mmdet --config faster-rcnn_r50_fpn_1x_coco --dest .
# convert mmdet model to onnxruntime model with dynamic shape

    faster-rcnn_r50_fpn_1x_coco.py \
    faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
    demo/resources/det.jpg \
    --work-dir mmdeploy_models/mmdet/ort \
    --show \
```

转换的关键之一是使用正确的配置文件。项目中已内置了各后端部署[配置文件](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmdet)。
文件的命名模式是：

```
```

其中：

```{tip}
```

## 模型规范

上例中的`mmdeploy_models/mmdet/ort`，结构如下：

```
mmdeploy_models/mmdet/ort
```

重要的是：

## 模型推理

## 后端模型推理

```python

## 模型推理

## 后端模型推理

```python

deploy_cfg = 'configs/mmdet/detection/detection_onnxruntime_dynamic.py'
model_cfg = './faster-rcnn_r50_fpn_1x_coco.py'
backend_model = ['./mmdeploy_models/mmdet/ort/end2end.onnx']
image = './demo/resources/det.jpg'

```

## SDK 模型推理

```python
from mmdeploy_runtime import Detector
import cv2

img = cv2.imread('./demo/resources/det.jpg')

detector = Detector(model_path='./mmdeploy_models/mmdet/ort', device_name='cpu', device_id=0)

  [left, top, right, bottom], score = bbox[0:4].astype(int), bbox[4]
    continue

```

if score < 0.3:
    continue

```

## 模型支持列表

|                                                  Model                                                   |         Task          | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO |
| :------------------------------------------------------------------------------------------------------: | :-------------------: | :---------: | :------: | :--: | :---: | :------: |
|                 [ATSS](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/atss)                  |   Object Detection    |      Y      |    Y     |  N   |   N   |    Y     |

|                 [FCOS](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/fcos)                  |   Object Detection    |      Y      |    Y     |  Y   |   N   |    Y     |
|             [FoveaBox](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/foveabox)              |   Object Detection    |      Y      |    N     |  N   |   N   |    Y     |
|                 [FSAF](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/fsaf)                  |   Object Detection    |      Y      |    Y     |  Y   |   Y   |    Y     |
|            [RetinaNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/retinanet)             |   Object Detection    |      Y      |    Y     |  Y   |   Y   |    Y     |

|                  [SSD](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/ssd)                   |   Object Detection    |      Y      |    Y     |  Y   |   N   |    Y     |
|                [VFNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/vfnet)                 |   Object Detection    |      N      |    N     |  N   |   N   |    Y     |
|                [YOLOv3](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/yolo)                 |   Object Detection    |      Y      |    Y     |  Y   |   N   |    Y     |
|                [YOLOX](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/yolox)                 |   Object Detection    |      Y      |    Y     |  Y   |   N   |    Y     |

|         [Cascade R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/cascade_rcnn)         |   Object Detection    |      Y      |    Y     |  N   |   Y   |    Y     |
|          [Faster R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/faster_rcnn)          |   Object Detection    |      Y      |    Y     |  Y   |   Y   |    Y     |
|       [Faster R-CNN + DCN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/faster_rcnn)       |   Object Detection    |      Y      |    Y     |  Y   |   Y   |    Y     |
|                  [GFL](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/gfl)                   |   Object Detection    |      Y      |    Y     |  N   |   ?   |    Y     |

|            [RepPoints](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/reppoints)             |   Object Detection    |      N      |    Y     |  N   |   ?   |    Y     |
|                 [DETR](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/detr)                  |   Object Detection    |      Y      |    Y     |  N   |   ?   |    Y     |
|            [CenterNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/centernet)             |   Object Detection    |      Y      |    Y     |  N   |   ?   |    Y     |
|               [RTMDet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/rtmdet)                |   Object Detection    |      Y      |    Y     |  N   |   ?   |    Y     |

|      [Cascade Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/cascade_rcnn)       | Instance Segmentation |      Y      |    Y     |  N   |   N   |    Y     |
|            [Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/mask_rcnn)            | Instance Segmentation |      Y      |    Y     |  N   |   N   |    Y     |
|           [Swin Transformer](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/swin)            | Instance Segmentation |      Y      |    Y     |  N   |   N   |    Y     |
|                 [SOLO](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo)                  | Instance Segmentation |      Y      |    N     |  N   |   N   |    Y     |

|               [SOLOv2](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solov2)                | Instance Segmentation |      Y      |    N     |  N   |   N   |    Y     |
|         [Panoptic FPN](https://github.com/open-mmlab/mmdetection/tree/main/configs/panoptic_fpn)         | Panoptic Segmentation |      Y      |    Y     |  N   |   N   |    N     |
|           [MaskFormer](https://github.com/open-mmlab/mmdetection/tree/main/configs/maskformer)           | Panoptic Segmentation |      Y      |    Y     |  N   |   N   |    N     |
| [Mask2Former](https://github.com/open-mmlab/mmdetection/tree/main/configs/mask2former)[\*](#mask2former) | Panoptic Segmentation |      Y      |    Y     |  N   |   N   |    N     |

## 注意事项

- 强烈建议使用`TensorRT>=8.4`来转换基于 `transformer` 的模型.

## 注意事项

- <i id="mask2former">Mask2Former</i> 请使用 `TensorRT>=8.6.1` 以保证动态尺寸正常推理.

# Ubuntu Cross Build aarch64

mmdeploy chose ncnn as the inference backend for aarch64 embedded linux devices. There are two parts:

Host

- model conversion
- cross build SDK and demo for embedded devices

Device

- Run converted model

## 1. Model Convert on Host

Refer to the doc to install [mmdeploy](../01-how-to-build/) and [mmpretrain](https://github.com/open-mmlab/mmpretrain), and convert resnet18 for model package

```bash
export MODEL_CONFIG=/path/to/mmpretrain/configs/resnet/resnet18_8xb32_in1k.py
export MODEL_PATH=https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_8xb32_in1k_20210831-fbbb1da6.pth

# Convert resnet18
cd /path/to/mmdeploy
  configs/mmpretrain/classification_ncnn_static.py \
  $MODEL_CONFIG \
  $MODEL_PATH \
  tests/data/tiger.jpeg \
  --work-dir resnet18 \
  --device cpu \
  --dump-info
```

## 2. Cross Build on Host

It is recommended to compile directly with the script

```bash

$MODEL_PATH \
```

```bash
sh -x tools/scripts/ubuntu_cross_build_aarch64.sh
```

The following is the manual process corresponding to the script:

a) Install aarch64 build tools

```bash
sudo apt install -y gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
```

b) Cross build opencv and install to /tmp/ocv-aarch64

```bash
git clone https://github.com/opencv/opencv --depth=1 --branch=4.x --recursive
cd opencv/platforms/linux/
cmake ../../.. \
  -DCMAKE_INSTALL_PREFIX=/tmp/ocv-aarch64 \
  -DCMAKE_TOOLCHAIN_FILE=../aarch64-gnu.toolchain.cmake
make -j && make install
ls -alh /tmp/ocv-aarch64
..
```

c) Cross build ncnn and install to /tmp/ncnn-aarch64

```bash
git clone https://github.com/tencent/ncnn --branch 20221128 --depth=1
cmake .. \

..
```

```bash
cmake .. \
  -DCMAKE_TOOLCHAIN_FILE=../toolchains/aarch64-linux-gnu.toolchain.cmake \
  -DCMAKE_INSTALL_PREFIX=/tmp/ncnn-aarch64
ls -alh /tmp/ncnn-aarch64
..
```

d) Cross build mmdeploy

```bash
git submodule init
git submodule update
cmake .. \
  -DCMAKE_TOOLCHAIN_FILE=../cmake/toolchains/aarch64-linux-gnu.cmake \
  -DMMDEPLOY_TARGET_DEVICES="cpu" \
  -DMMDEPLOY_TARGET_BACKENDS="ncnn" \
  -Dncnn_DIR=/tmp/ncnn-aarch64/lib/cmake/ncnn \
  -DOpenCV_DIR=/tmp/ocv-aarch64/lib/cmake/opencv4
ls -lah install/bin/*
..
```

## 3. Execute on Device

Make sure that `--dump-info` is used during model conversion, so that the `resnet18` directory contains the files required by the SDK such as `pipeline.json`.

..
```

Copy the model folder(resnet18), executable(image_classification) file, test image(tests/data/tiger.jpeg) and prebuilt OpenCV(/tmp/ocv-aarch64) to the device.

```bash
./image_classification cpu ./resnet18  tiger.jpeg
..
label: 292, score: 0.9261
label: 282, score: 0.0726
label: 290, score: 0.0008
label: 281, score: 0.0002
label: 340, score: 0.0001
```

# Text Recognition

```{note}
```

|     Dataset      |                         images                          |                         annotation file                          |                          annotation file                          |
| :--------------: | :-----------------------------------------------------: | :--------------------------------------------------------------: | :---------------------------------------------------------------: |
|                  |                                                         |                             training                             |                               test                                |

|    coco_text     | [homepage](https://rrc.cvc.uab.es/?ch=5&com=downloads)  |                    [train_labels.json](#TODO)                    |                                 -                                 |
|    ICDAR2011     |        [homepage](https://rrc.cvc.uab.es/?ch=1)         |                                -                                 |                                 -                                 |
|     SynthAdd     | [SynthText_Add.zip](https://pan.baidu.com/s/1uV0LtoNmcxbO-0YA7Ch4dg)  (code:627x) | [train_labels.json](https://download.openmmlab.com/mmocr/data/1.x/recog/synthtext_add/train_labels.json) |                                 -                                 |

|     OpenVINO     | [Open Images](https://github.com/cvdfoundation/open-images-dataset) | [annotations](https://storage.openvinotoolkit.org/repositories/openvino_training_extensions/datasets/open_images_v5_text) | [annotations](https://storage.openvinotoolkit.org/repositories/openvino_training_extensions/datasets/open_images_v5_text) |
|      DeText      |        [homepage](https://rrc.cvc.uab.es/?ch=9)         |                                -                                 |                                 -                                 |
| Lecture Video DB | [homepage](https://cvit.iiit.ac.in/research/projects/cvit-projects/lecturevideodb) |                                -                                 |                                 -                                 |

|       LSVT       |        [homepage](https://rrc.cvc.uab.es/?ch=16)        |                                -                                 |                                 -                                 |
|      IMGUR       | [homepage](https://github.com/facebookresearch/IMGUR5K-Handwriting-Dataset) |                                -                                 |                                 -                                 |
|      KAIST       | [homepage](http://www.iapr-tc11.org/mediawiki/index.php/KAIST_Scene_Text_Database) |                                -                                 |                                 -                                 |
|       MTWI       | [homepage](https://tianchi.aliyun.com/competition/entrance/231685/information?lang=en-us) |                                -                                 |                                 -                                 |

|      ReCTS       |        [homepage](https://rrc.cvc.uab.es/?ch=12)        |                                -                                 |                                 -                                 |
|    IIIT-ILST     | [homepage](http://cvit.iiit.ac.in/research/projects/cvit-projects/iiit-ilst) |                                -                                 |                                 -                                 |
|     VinText      | [homepage](https://github.com/VinAIResearch/dict-guided) |                                -                                 |                                 -                                 |
|       BID        | [homepage](https://github.com/ricardobnjunior/Brazilian-Identity-Document-Dataset) |                                -                                 |                                 -                                 |

|       RCTW       |     [homepage](https://rctw.vlrlab.net/index.html)      |                                -                                 |                                 -                                 |
|     HierText     | [homepage](https://github.com/google-research-datasets/hiertext) |                                -                                 |                                 -                                 |
|       ArT        |        [homepage](https://rrc.cvc.uab.es/?ch=14)        |                                -                                 |                                 -                                 |

(\*) Since the official homepage is unavailable now, we provide an alternative for quick reference. However, we do not guarantee the correctness of the dataset.

  ```bash
  ```

# for the Default region name
  ```

- [icdar_2013](https://opendatalab.com/ICDAR_2013?source=OpenMMLab%20GitHub)
- [icdar_2015](https://opendatalab.com/ICDAR2015?source=OpenMMLab%20GitHub)
- [IIIT5K](https://opendatalab.com/IIIT_5K?source=OpenMMLab%20GitHub)
- [ct80](https://opendatalab.com/CUTE_80?source=OpenMMLab%20GitHub)
- [svt](https://opendatalab.com/SVT?source=OpenMMLab%20GitHub)
- [IAM](https://opendatalab.com/IAM_Handwriting?source=OpenMMLab%20GitHub)

- Step1: Download `Challenge1_Training_Task3_Images_GT.zip`, `Challenge1_Test_Task3_Images.zip`, and `Challenge1_Test_Task3_GT.txt` from [homepage](https://rrc.cvc.uab.es/?ch=1&com=downloads) `Task 1.3: Word Recognition (2013 edition)`.

  ```bash

  wget https://rrc.cvc.uab.es/downloads/Challenge1_Training_Task3_Images_GT.zip --no-check-certificate
  wget https://rrc.cvc.uab.es/downloads/Challenge1_Test_Task3_Images.zip --no-check-certificate
  wget https://rrc.cvc.uab.es/downloads/Challenge1_Test_Task3_GT.txt --no-check-certificate

  mkdir crops
  unzip -q Challenge1_Training_Task3_Images_GT.zip -d crops/train
  unzip -q Challenge1_Test_Task3_Images.zip -d crops/test

  mv Challenge1_Test_Task3_GT.txt annotations && mv crops/train/gt.txt annotations/Challenge1_Train_Task3_GT.txt
  ```

- Step2: Convert original annotations to `train_labels.json` and `test_labels.json` with the following command:

  ```bash
  python tools/dataset_converters/textrecog/ic11_converter.py PATH/TO/icdar2011
  ```

  ```text
  ├── icdar2011
  │   ├── crops
  │   ├── train_labels.json
  │   └── test_labels.json
  ```

## coco_text

- Step1: Download from [homepage](https://rrc.cvc.uab.es/?ch=5&com=downloads)

- Step2: Download [train_labels.json](https://download.openmmlab.com/mmocr/data/mixture/coco_text/train_labels.json)

- After running the above codes, the directory structure
  should be as follows:

  ```text

  ```text
  ├── coco_text
  │   └── train_words
  ```

## SynthAdd

- Step1: Download `SynthText_Add.zip` from [SynthAdd](https://pan.baidu.com/s/1uV0LtoNmcxbO-0YA7Ch4dg) (code:627x))

- Step2: Download [train_labels.json](https://download.openmmlab.com/mmocr/data/1.x/recog/synthtext_add/train_labels.json)

- Step3:

  ```bash
  mkdir SynthAdd && cd SynthAdd

  mv /path/to/SynthText_Add.zip .

  unzip SynthText_Add.zip

  mv /path/to/train_labels.json .

  # create soft link
  cd /path/to/mmocr/data/recog

  ln -s /path/to/SynthAdd SynthAdd

  ```

  ```text
  ├── SynthAdd
  │   └── SynthText_Add
  ```

## OpenVINO

- Step1 (optional): Install [AWS CLI](https://mmocr.readthedocs.io/en/latest/datasets/recog.html#install-aws-cli-optional).

│   ├── train_labels.json
  ```

- Step2: Download [Open Images](https://github.com/cvdfoundation/open-images-dataset#download-images-with-bounding-boxes-annotations) subsets `train_1`, `train_2`, `train_5`, `train_f`, and `validation` to `openvino/`.

  ```bash
  mkdir openvino && cd openvino

  # Download Open Images subsets
  for s in 1 2 5 f; do
    aws s3 --no-sign-request cp s3://open-images-dataset/tar/train_${s}.tar.gz .
  done
  aws s3 --no-sign-request cp s3://open-images-dataset/tar/validation.tar.gz .

  # Download annotations
    wget https://storage.openvinotoolkit.org/repositories/openvino_training_extensions/datasets/open_images_v5_text/text_spotting_openimages_v5_train_${s}.json
  done

for s in 1 2 5 f; do
  done
  wget https://storage.openvinotoolkit.org/repositories/openvino_training_extensions/datasets/open_images_v5_text/text_spotting_openimages_v5_validation.json

  # Extract images
  mkdir -p openimages_v5/val
    tar zxf train_${s}.tar.gz -C openimages_v5
  done
  tar zxf validation.tar.gz -C openimages_v5/val
  ```

- Step3: Generate `train_{1,2,5,f}_labels.json`, `val_labels.json` and crop images using 4 processes with the following command:

  ```bash
  python tools/dataset_converters/textrecog/openvino_converter.py /path/to/openvino 4
  ```

  ```text
  ├── OpenVINO
  │   ├── image_1
  │   ├── image_2
  │   ├── image_5
  │   ├── image_f
  │   ├── image_val
  │   ├── train_1_labels.json

should be as follows:

  ```text
  │   ├── train_2_labels.json
  │   ├── train_5_labels.json
  │   ├── train_f_labels.json
  │   └── val_labels.json
  ```

## DeText

  ```bash

wget https://rrc.cvc.uab.es/downloads/ch9_training_images.zip --no-check-certificate

  ```

- Step2: Generate `train_labels.json` and `test_labels.json` with following command:

  ```bash

```

  ```bash
  # Add --preserve-vertical to preserve vertical texts for training, otherwise
  # vertical images will be filtered and stored in PATH/TO/detext/ignores
  python tools/dataset_converters/textrecog/detext_converter.py PATH/TO/detext --nproc 4
  ```

  ```text
  ├── detext
  │   ├── crops
  │   ├── ignores
  ```

## NAF

- Step1: Download [labeled_images.tar.gz](https://github.com/herobd/NAF_dataset/releases/tag/v1.0) to `naf/`.

  ```bash
  mkdir naf && cd naf

  # Download NAF dataset
  wget https://github.com/herobd/NAF_dataset/releases/download/v1.0/labeled_images.tar.gz
  tar -zxf labeled_images.tar.gz

  mkdir annotations && mv labeled_images imgs

  git clone https://github.com/herobd/NAF_dataset.git

tar -zxf labeled_images.tar.gz

  mv NAF_dataset/train_valid_test_split.json annotations/ && mv NAF_dataset/groups annotations/

  rm -rf NAF_dataset && rm labeled_images.tar.gz
  ```

- Step2: Generate `train_labels.json`, `val_labels.json`, and `test_labels.json` with following command:

  ```bash
  # vertical images will be filtered and stored in PATH/TO/naf/ignores
  python tools/dataset_converters/textrecog/naf_converter.py PATH/TO/naf --nproc 4

  ```

  ```text
  ├── naf
  │   ├── crops
  │   ├── val_labels.json
  ```

This section is not fully tested yet.
```

```{note}

├── naf
  │   ├── crops
  ```

```

```{note}
The LV dataset has already provided cropped images and the corresponding annotations
```

  ```bash

  # For image
  mv IIIT-CVid/Crops ./

  # For annotation
  mv IIIT-CVid/train.txt train_labels.json && mv IIIT-CVid/val.txt val_label.txt && mv IIIT-CVid/test.txt test_labels.json

  ```

- Step2: Generate `train_labels.json`, `val.json`, and `test.json` with following command:

  ```bash
  python tools/dataset_converters/textdreog/lv_converter.py PATH/TO/lv
  ```

```

  ```bash
  ```

  ```text
  ├── lv
  │   ├── Crops
  ```

## LSVT

```

  ```bash

mkdir lsvt && cd lsvt

  ```

- Step2: Generate `train_labels.json` and `val_label.json` (optional) with the following command:

  ```bash
  # vertical images will be filtered and stored in PATH/TO/lsvt/ignores

  python tools/dataset_converters/textdrecog/lsvt_converter.py PATH/TO/lsvt --nproc 4
  ```

  ```text
  ├── lsvt
  │   ├── crops
  │   └── val_label.json (optional)
  ```

## IMGUR

```

  ```bash

```bash

  ```

- Step2: Generate `train_labels.json`, `val_label.txt` and `test_labels.json` and crop images with the following command:

  ```bash
  python tools/dataset_converters/textrecog/imgur_converter.py PATH/TO/imgur
  ```

  ```text
  ├── imgur
  │   ├── crops
  │   ├── test_labels.json
  │   └── val_label.json
  ```

## KAIST

```

```text
  ├── imgur
  │   ├── crops
  ```

## KAIST

```

- Step1: Download [KAIST_all.zip](http://www.iapr-tc11.org/mediawiki/index.php/KAIST_Scene_Text_Database) to `kaist/`.

  ```bash

  unzip -q KAIST_all.zip && rm KAIST_all.zip
  ```

  ```bash
  ```

- Step3: Generate `train_labels.json` and `val_label.json` (optional) with following command:

  ```bash

```bash
  # vertical images will be filtered and stored in PATH/TO/kaist/ignores
  python tools/dataset_converters/textrecog/kaist_converter.py PATH/TO/kaist --nproc 4
  ```

  ```text
  ├── kaist
  │   ├── crops
  ```

## MTWI

```

  ```bash

  ```

```bash

  ```

  ```bash
  # vertical images will be filtered and stored in PATH/TO/mtwi/ignores
  python tools/dataset_converters/textrecog/mtwi_converter.py PATH/TO/mtwi --nproc 4
  ```

  ```text
  ├── mtwi
  │   ├── crops
  ```

## ReCTS

```

```text
  ├── mtwi
  │   ├── crops
  ```

## ReCTS

```

  ```bash

  rm ReCTS.zip -f && rm -rf gt
  ```

  ```bash

# Annotations of ReCTS test split is not publicly available, split a validation
  # vertical images will be filtered and stored in PATH/TO/rects/ignores
  python tools/dataset_converters/textrecog/rects_converter.py PATH/TO/rects --nproc 4
  ```

  ```text
  ├── rects
  │   ├── crops
  ```

## ILST

```

- Step1: Download `IIIT-ILST.zip` from [onedrive link](https://iiitaphyd-my.sharepoint.com/:f:/g/personal/minesh_mathew_research_iiit_ac_in/EtLvCozBgaBIoqglF4M-lHABMgNcCDW9rJYKKWpeSQEElQ?e=zToXZP)

  ```bash

  ```bash

  ```

mv Telugu/*jpeg imgs/ && mv Telugu/*xml annotations/

  ```

- Step3: Generate `train_labels.json` and `val_label.json` (optional) and crop images using 4 processes with the following command (add `--preserve-vertical` if you wish to preserve the images containing vertical texts). Since the original dataset doesn't have a validation set, you may specify `--val-ratio` to split the dataset. E.g., if val-ratio is 0.2, then 20% of the data are left out as the validation set in this example.

  ```bash
  python tools/dataset_converters/textrecog/ilst_converter.py PATH/TO/IIIT-ILST --nproc 4
  ```

  ```text
  ├── IIIT-ILST
  │   ├── crops
  ```

## VinText

```

```text
  │   ├── crops
  ```

## VinText

```

  ```bash

  wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1UUQhNvzgpZy7zXBFQp0Qox-BBjunZ0ml' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1UUQhNvzgpZy7zXBFQp0Qox-BBjunZ0ml" -O vintext.zip && rm -rf /tmp/cookies.txt

unzip -q vintext.zip && rm vintext.zip

  mkdir imgs
  ```

- Step2: Generate `train_labels.json`, `test_labels.json`, `unseen_test_labels.json`,  and crop images using 4 processes with the following command (add `--preserve-vertical` if you wish to preserve the images containing vertical texts).

  ```bash
  python tools/dataset_converters/textrecog/vintext_converter.py PATH/TO/vietnamese --nproc 4
  ```

  ```text
  ├── vintext
  │   ├── crops
  │   └── unseen_test_labels.json
  ```

## BID

```text
  │   ├── crops
  ```

## BID

```

  ```bash
  # Rename

  cd BID

  ```

mv RG_Verso/*in.jpg imgs && mv RG_Verso/*txt annotations && rm -rf RG_Verso

  ```

- Step3: Generate `train_labels.json` and `val_label.json` (optional) and crop images using 4 processes with the following command (add `--preserve-vertical` if you wish to preserve the images containing vertical texts). Since the original dataset doesn't have a validation set, you may specify `--val-ratio` to split the dataset. E.g., if test-ratio is 0.2, then 20% of the data are left out as the validation set in this example.

  ```bash
  python tools/dataset_converters/textrecog/bid_converter.py PATH/TO/BID --nproc 4
  ```

  ```text
  ├── BID
  │   ├── crops
  ```

## RCTW

```

```text
  ├── BID
  │   ├── crops
  ```

## RCTW

```

- Step2: Generate `train_labels.json` and `val_label.json` (optional). Since the original dataset doesn't have a validation set, you may specify `--val-ratio` to split the dataset. E.g., if val-ratio is 0.2, then 20% of the data are left out as the validation set in this example.

  ```bash
  # Add --preserve-vertical to preserve vertical texts for training, otherwise vertical images will be filtered and stored in PATH/TO/rctw/ignores

# Add --preserve-vertical to preserve vertical texts for training, otherwise vertical images will be filtered and stored in PATH/TO/rctw/ignores
  python tools/dataset_converters/textrecog/rctw_converter.py PATH/TO/rctw --nproc 4
  ```

  ```text
  │── rctw
  │   ├── crops
  │   ├── ignores
  │   ├── train_labels.json
  │   └── val_label.json (optional)
  ```

```

  ```bash
  ```

  ```bash

```

  ```bash
  ```

  ```bash
  gzip -d annotations/train.json.gz
  gzip -d annotations/validation.json.gz
  mkdir imgs
  ```

- Step5: Generate `train_labels.json` and `val_label.json`. HierText includes different levels of annotation, including `paragraph`, `line`, and `word`. Check the original [paper](https://arxiv.org/pdf/2203.15143.pdf) for details. E.g. set `--level paragraph` to get paragraph-level annotation. Set `--level line` to get line-level annotation. set `--level word` to get word-level annotation.

  ```bash

```bash
  # Add --preserve-vertical to preserve vertical texts for training, otherwise vertical images will be filtered and stored in PATH/TO/HierText/ignores
  python tools/dataset_converters/textrecog/hiertext_converter.py PATH/TO/HierText --level word --nproc 4
  ```

  ```text
  │   ├── crops
  │   └── val_label.json
  ```

## ArT

```

  ```bash

  wget https://dataset-bj.cdn.bcebos.com/art/train_task2_images.tar.gz
  wget https://dataset-bj.cdn.bcebos.com/art/train_task2_labels.json

  # Extract

mkdir annotations

  # Extract
  tar -xf train_task2_images.tar.gz
  mv train_task2_images crops
  mv train_task2_labels.json annotations/

  ```

- Step2: Generate `train_labels.json` and `val_label.json` (optional). Since the test annotations are not publicly available, you may specify `--val-ratio` to split the dataset. E.g., if val-ratio is 0.2, then 20% of the data are left out as the validation set in this example.

  ```bash
  python tools/dataset_converters/textrecog/art_converter.py PATH/TO/art
  ```

  ```text
  │── art
  │   ├── crops

```

  ```text
  │── art
  │   ├── crops
  ```

  <img src="resources/mmocr-logo.png" width="500px"/>
    <sup>
      </a>
    </sup>
    <sup>
      </a>
    </sup>
  </div>

[![build](https://github.com/open-mmlab/mmocr/workflows/build/badge.svg)](https://github.com/open-mmlab/mmocr/actions)
[![docs](https://readthedocs.org/projects/mmocr/badge/?version=dev-1.x)](https://mmocr.readthedocs.io/en/dev-1.x/?badge=dev-1.x)
[![codecov](https://codecov.io/gh/open-mmlab/mmocr/branch/main/graph/badge.svg)](https://codecov.io/gh/open-mmlab/mmocr)

[![license](https://img.shields.io/github/license/open-mmlab/mmocr.svg)](https://github.com/open-mmlab/mmocr/blob/main/LICENSE)
[![PyPI](https://badge.fury.io/py/mmocr.svg)](https://pypi.org/project/mmocr/)
[![Average time to resolve an issue](https://isitmaintained.com/badge/resolution/open-mmlab/mmocr.svg)](https://github.com/open-mmlab/mmocr/issues)
[![Percentage of issues still open](https://isitmaintained.com/badge/open/open-mmlab/mmocr.svg)](https://github.com/open-mmlab/mmocr/issues)
<a href="https://console.tiyaro.ai/explore?q=mmocr&pub=mmocr"> <img src="https://tiyaro-public-docs.s3.us-west-2.amazonaws.com/assets/try_on_tiyaro_badge.svg"></a>

[📘文档](https://mmocr.readthedocs.io/zh_CN/dev-1.x/) |
[🛠️安装](https://mmocr.readthedocs.io/zh_CN/dev-1.x/get_started/install.html) |
[👀模型库](https://mmocr.readthedocs.io/zh_CN/dev-1.x/modelzoo.html) |

[🆕更新日志](https://mmocr.readthedocs.io/en/dev-1.x/notes/changelog.html) |
[🤔报告问题](https://github.com/open-mmlab/mmocr/issues/new/choose)

</div>

</div>

</div>

## 近期更新

**默认分支目前为 `main`，且分支上的代码已经切换到 v1.0.0 版本。旧版 `main` 分支（v0.6.3）的代码现存在 `0.x` 分支上。** 如果您一直在使用 `main` 分支，并遇到升级问题，请阅读 [迁移指南](https://mmocr.readthedocs.io/zh_CN/dev-1.x/migration/overview.html) 和 [分支说明](https://mmocr.readthedocs.io/zh_CN/dev-1.x/migration/branches.html) 。

最新的版本 v1.0.0 于 2023-04-06 发布。其相对于 1.0.0rc6 的主要更新如下：

1. Dataset Preparer 中支持了 SCUT-CTW1500, SynthText 和 MJSynth 数据集；
2. 更新了文档和 FAQ；
3. 升级文件后端；使用了 `backend_args` 替换 `file_client_args`;
4. 增加了 MMOCR 教程 notebook。

如果需要了解 MMOCR 1.0 相对于 0.x 的升级内容，请阅读 [MMOCR 1.x 更新汇总](https://mmocr.readthedocs.io/zh_CN/dev-1.x/migration/news.html)；或者阅读[更新日志](https://mmocr.readthedocs.io/zh_CN/dev-1.x/notes/changelog.html)以获取更多信息。

## 简介

MMOCR 是基于 PyTorch 和 mmdetection 的开源工具箱，专注于文本检测，文本识别以及相应的下游任务，如关键信息提取。 它是 OpenMMLab 项目的一部分。

主分支目前支持 **PyTorch 1.6 以上**的版本。

  <img src="https://user-images.githubusercontent.com/24622904/187838618-1fdc61c0-2d46-49f9-8502-976ffdf01f28.png"/>
</div>

### 主要特性

-**全流程**

该工具箱不仅支持文本检测和文本识别，还支持其下游任务，例如关键信息提取。

-**多种模型**

该工具箱支持用于文本检测，文本识别和关键信息提取的各种最新模型。

-**模块化设计**

MMOCR 的模块化设计使用户可以定义自己的优化器，数据预处理器，模型组件如主干模块，颈部模块和头部模块，以及损失函数。有关如何构建自定义模型的信息，请参考[概览](https://mmocr.readthedocs.io/zh_CN/dev-1.x/get_started/overview.html)。

-**众多实用工具**

该工具箱提供了一套全面的实用程序，可以帮助用户评估模型的性能。它包括可对图像，标注的真值以及预测结果进行可视化的可视化工具，以及用于在训练过程中评估模型的验证工具。它还包括数据转换器，演示了如何将用户自建的标注数据转换为 MMOCR 支持的标注文件。

## 安装

MMOCR 依赖 [PyTorch](https://pytorch.org/), [MMEngine](https://github.com/open-mmlab/mmengine), [MMCV](https://github.com/open-mmlab/mmcv) 和 [MMDetection](https://github.com/open-mmlab/mmdetection)，以下是安装的简要步骤。
更详细的安装指南请参考 [安装文档](https://mmocr.readthedocs.io/zh_CN/dev-1.x/get_started/install.html)。

```shell
conda create -n open-mmlab python=3.8 pytorch=1.10 cudatoolkit=11.3 torchvision -c pytorch -y
cd mmocr
```

## 快速入门

请参考[快速入门](https://mmocr.readthedocs.io/zh_CN/dev-1.x/get_started/quick_run.html)文档学习 MMOCR 的基本使用。

cd mmocr
```

## 快速入门

## [模型库](https://mmocr.readthedocs.io/zh_CN/dev-1.x/modelzoo.html)

支持的算法：

<summary>骨干网络</summary>

- [x] [oCLIP](configs/backbone/oclip/README.md) (ECCV'2022)

<summary>文字检测</summary>

- [x] [DBNet](configs/textdet/dbnet/README.md) (AAAI'2020) / [DBNet++](configs/textdet/dbnetpp/README.md) (TPAMI'2022)
- [x] [Mask R-CNN](configs/textdet/maskrcnn/README.md) (ICCV'2017)
- [x] [PANet](configs/textdet/panet/README.md) (ICCV'2019)
- [x] [PSENet](configs/textdet/psenet/README.md) (CVPR'2019)
- [x] [TextSnake](configs/textdet/textsnake/README.md) (ECCV'2018)
- [x] [DRRG](configs/textdet/drrg/README.md) (CVPR'2020)
- [x] [FCENet](configs/textdet/fcenet/README.md) (CVPR'2021)

<summary>文字识别</summary>

- [x] [ABINet](configs/textrecog/abinet/README.md) (CVPR'2021)
- [x] [ASTER](configs/textrecog/aster/README.md) (TPAMI'2018)
- [x] [CRNN](configs/textrecog/crnn/README.md) (TPAMI'2016)
- [x] [MASTER](configs/textrecog/master/README.md) (PR'2021)
- [x] [NRTR](configs/textrecog/nrtr/README.md) (ICDAR'2019)
- [x] [RobustScanner](configs/textrecog/robust_scanner/README.md) (ECCV'2020)
- [x] [SAR](configs/textrecog/sar/README.md) (AAAI'2019)
- [x] [SATRN](configs/textrecog/satrn/README.md) (CVPR'2020 Workshop on Text and Documents in the Deep Learning Era)
- [x] [SVTR](configs/textrecog/svtr/README.md) (IJCAI'2022)

<summary>关键信息提取</summary>

- [x] [SDMG-R](configs/kie/sdmgr/README.md) (ArXiv'2021)

<summary>端对端 OCR</summary>

- [x] [ABCNet](projects/ABCNet/README.md) (CVPR'2020)
- [x] [ABCNetV2](projects/ABCNet/README_V2.md) (TPAMI'2021)
- [x] [SPTS](projects/SPTS/README.md) (ACM MM'2022)

请点击[模型库](https://mmocr.readthedocs.io/zh_CN/dev-1.x/modelzoo.html)查看更多关于上述算法的详细信息。

## 社区项目

[这里](projects/README.md)有一些由社区用户支持和维护的基于 MMOCR 的 SOTA 模型和解决方案的实现。这些项目展示了基于 MMOCR 的研究和产品开发的最佳实践。
我们欢迎并感谢对 OpenMMLab 生态系统的所有贡献。

## 贡献指南

我们感谢所有的贡献者为改进和提升 MMOCR 所作出的努力。请参考[贡献指南](.github/CONTRIBUTING.md)来了解参与项目贡献的相关指引。

## 致谢

MMOCR 是一款由来自不同高校和企业的研发人员共同参与贡献的开源项目。我们感谢所有为项目提供算法复现和新功能支持的贡献者，以及提供宝贵反馈的用户。 我们希望此工具箱可以帮助大家来复现已有的方法和开发新的方法，从而为研究社区贡献力量。

## 引用

如果您发现此项目对您的研究有用，请考虑引用：

```bibtex
@article{mmocr2021,
    title={MMOCR:  A Comprehensive Toolbox for Text Detection, Recognition and Understanding},

## 引用

```bibtex
    author={Kuang, Zhanghui and Sun, Hongbin and Li, Zhizhong and Yue, Xiaoyu and Lin, Tsui Hin and Chen, Jianyong and Wei, Huaqiang and Zhu, Yiqin and Gao, Tong and Zhang, Wenwei and Chen, Kai and Zhang, Wayne and Lin, Dahua},
    journal= {arXiv preprint arXiv:2108.06543},
}
```

## 开源许可证

该项目采用 [Apache 2.0 license](LICENSE) 开源许可证。

扫描下方的二维码可关注 OpenMMLab 团队的 [知乎官方账号](https://www.zhihu.com/people/openmmlab)，加入 OpenMMLab 团队的 [官方交流 QQ 群](https://r.vansin.top/?r=join-qq)，或通过添加微信“Open小喵Lab”加入官方交流微信群。

<img src="https://raw.githubusercontent.com/open-mmlab/mmcv/master/docs/en/_static/zhihu_qrcode.jpg" height="400" />  <img src="https://cdn.vansin.top/OpenMMLab/q3.png" height="400" />  <img src="https://raw.githubusercontent.com/open-mmlab/mmcv/master/docs/en/_static/wechat_qrcode.jpg" height="400" />
</div>

# Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (ICLR'2016)

> [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://arxiv.org/abs/1511.06434)

In recent years, supervised learning with convolutional networks (CNNs) has seen huge adoption in computer vision applications. Comparatively, unsupervised learning with CNNs has received less attention. In this work we hope to help bridge the gap between the success of CNNs for supervised learning and unsupervised learning. We introduce a class of CNNs called deep convolutional generative adversarial networks (DCGANs), that have certain architectural constraints, and demonstrate that they are a strong candidate for unsupervised learning. Training on various image datasets, we show convincing evidence that our deep convolutional adversarial pair learns a hierarchy of representations from object parts to scenes in both the generator and discriminator. Additionally, we use the learned features for novel tasks - demonstrating their applicability as general image representations.

<img src="https://user-images.githubusercontent.com/28132635/143050281-60808c3f-81d0-4fae-9071-f4c297116b2f.JPG"/>
</div>

  <b> DCGAN 64x64, CelebA-Cropped</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/12726765/113991928-871f9b80-9885-11eb-920e-d389c603fed8.png" width="800"/>
</div>

|                                Model                                 |    Dataset     |           SWD            | MS-SSIM |                                        Download                                         |
| :------------------------------------------------------------------: | :------------: | :----------------------: | :-----: | :-------------------------------------------------------------------------------------: |

| [DCGAN 64x64](./dcgan_Glr4e-4_Dlr1e-4_1xb128-5kiters_mnist-64x64.py) | MNIST (64x64)  |  21.16, 4.4, 8.41/11.32  | 0.1395  | [model](https://download.openmmlab.com/mmediting/dcgan/dcgan_mnist-64_b128x1_Glr4e-4_Dlr1e-4_5k_20210512_163926-207a1eaf.pth) \| [log](https://download.openmmlab.com//mmgen/dcgan/dcgan_mnist-64_b128x1_Glr4e-4_Dlr1e-4_5k_20210512_163926-207a1eaf.json) |
|     [DCGAN 64x64](./dcgan_1xb128-300kiters_celeba-cropped-64.py)     | CelebA-Cropped |  8.93,10.53,50.32/23.26  | 0.2899  | [model](https://download.openmmlab.com/mmediting/dcgan/dcgan_celeba-cropped_64_b128x1_300kiter_20210408_161607-1f8a2277.pth) \| [log](https://download.openmmlab.com/mmediting/dcgan/dcgan_celeba-cropped_64_b128x1_300kiter_20210408_161607-1f8a2277.json) |

|     [DCGAN 64x64](./dcgan_1xb128-5epoches_lsun-bedroom-64x64.py)     |  LSUN-Bedroom  | 42.79, 34.55, 98.46/58.6 | 0.2095  | [model](https://download.openmmlab.com/mmediting/dcgan/dcgan_lsun-bedroom_64_b128x1_5e_20210408_161713-117c498b.pth) \| [log](https://download.openmmlab.com/mmediting/dcgan/dcgan_lsun-bedroom_64_b128x1_5e_20210408_161713-117c498b.json) |

```latex
@article{radford2015unsupervised,
  title={Unsupervised representation learning with deep convolutional generative adversarial networks},
  author={Radford, Alec and Metz, Luke and Chintala, Soumith},
  journal={arXiv preprint arXiv:1511.06434},
  url={https://arxiv.org/abs/1511.06434},
}
```

# Grid R-CNN

> [Grid R-CNN](https://arxiv.org/abs/1811.12030)

This paper proposes a novel object detection framework named Grid R-CNN, which adopts a grid guided localization mechanism for accurate object detection. Different from the traditional regression based methods, the Grid R-CNN captures the spatial information explicitly and enjoys the position sensitive property of fully convolutional architecture. Instead of using only two independent points, we design a multi-point supervision formulation to encode more clues in order to reduce the impact of inaccurate prediction of specific points. To take the full advantage of the correlation of points in a grid, we propose a two-stage information fusion strategy to fuse feature maps of neighbor grid points. The grid guided localization approach is easy to be extended to different state-of-the-art detection frameworks. Grid R-CNN leads to high quality object localization, and experiments demonstrate that it achieves a 4.1% AP gain at IoU=0.8 and a 10.0% AP gain at IoU=0.9 on COCO benchmark compared to Faster R-CNN with Res50 backbone and FPN architecture.

Grid R-CNN is a well-performed objection detection framework. It transforms the traditional box offset regression problem into a grid point estimation problem. With the guidance of the grid points, it can obtain high-quality localization results. However, the speed of Grid R-CNN is not so satisfactory. In this technical report we present Grid R-CNN Plus, a better and faster version of Grid R-CNN. We have made several updates that significantly speed up the framework and simultaneously improve the accuracy. On COCO dataset, the Res50-FPN based Grid R-CNN Plus detector achieves an mAP of 40.4%, outperforming the baseline on the same model by 3.0 points with similar inference time.

<img src="https://user-images.githubusercontent.com/40661020/143890379-5d9e6233-0533-48b4-88b9-bc33abbd9f82.png"/>
</div>

</div>

|  Backbone   | Lr schd | Mem (GB) | Inf time (fps) | box AP |                         Config                          |                                                                                                                                                                         Download                                                                                                                                                                          |

| :---------: | :-----: | :------: | :------------: | :----: | :-----------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|    R-50     |   2x    |   5.1    |      15.0      |  40.4  |    [config](./grid-rcnn_r50_fpn_gn-head_2x_coco.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/grid_rcnn/grid_rcnn_r50_fpn_gn-head_2x_coco/grid_rcnn_r50_fpn_gn-head_2x_coco_20200130-6cca8223.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/grid_rcnn/grid_rcnn_r50_fpn_gn-head_2x_coco/grid_rcnn_r50_fpn_gn-head_2x_coco_20200130_221140.log.json)               |

|    R-101    |   2x    |   7.0    |      12.6      |  41.5  |    [config](./grid-rcnn_r101_fpn_gn-head_2x_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/grid_rcnn/grid_rcnn_r101_fpn_gn-head_2x_coco/grid_rcnn_r101_fpn_gn-head_2x_coco_20200309-d6eca030.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/grid_rcnn/grid_rcnn_r101_fpn_gn-head_2x_coco/grid_rcnn_r101_fpn_gn-head_2x_coco_20200309_164224.log.json)             |
| X-101-32x4d |   2x    |   8.3    |      10.8      |  42.9  | [config](./grid-rcnn_x101-32x4d_fpn_gn-head_2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/grid_rcnn/grid_rcnn_x101_32x4d_fpn_gn-head_2x_coco/grid_rcnn_x101_32x4d_fpn_gn-head_2x_coco_20200130-d8f0e3ff.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/grid_rcnn/grid_rcnn_x101_32x4d_fpn_gn-head_2x_coco/grid_rcnn_x101_32x4d_fpn_gn-head_2x_coco_20200130_215413.log.json) |

| X-101-64x4d |   2x    |   11.3   |      7.7       |  43.0  | [config](./grid-rcnn_x101-64x4d_fpn_gn-head_2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/grid_rcnn/grid_rcnn_x101_64x4d_fpn_gn-head_2x_coco/grid_rcnn_x101_64x4d_fpn_gn-head_2x_coco_20200204-ec76a754.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/grid_rcnn/grid_rcnn_x101_64x4d_fpn_gn-head_2x_coco/grid_rcnn_x101_64x4d_fpn_gn-head_2x_coco_20200204_080641.log.json) |

- All models are trained with 8 GPUs instead of 32 GPUs in the original paper.
- The warming up lasts for 1 epoch and `2x` here indicates 25 epochs.

```latex
@inproceedings{lu2019grid,
  title={Grid r-cnn},
  author={Lu, Xin and Li, Buyu and Yue, Yuxin and Li, Quanquan and Yan, Junjie},
}

@article{lu2019grid,
  title={Grid R-CNN Plus: Faster and Better},

}

  journal={arXiv preprint arXiv:1906.05688},
}
```

[![docs](https://img.shields.io/badge/docs-latest-blue)](https://lmdeploy.readthedocs.io/en/latest/)

</div>

</div>

</p>

## News 🎉

- \[2023/08\] TurboMind supports Qwen-7B, dynamic NTK-RoPE scaling and dynamic logN scaling
- \[2023/08\] TurboMind supports Windows (tp=1)
- \[2023/08\] TurboMind supports 4-bit inference, 2.4x faster than FP16, the fastest open-source implementation🚀. Check [this](./docs/en/w4a16.md) guide for detailed info
- \[2023/08\] LMDeploy has launched on the [HuggingFace Hub](https://huggingface.co/lmdeploy), providing ready-to-use 4-bit models.

- \[2023/08\] LMDeploy supports 4-bit quantization using the [AWQ](https://arxiv.org/abs/2306.00978) algorithm.
- \[2023/07\] TurboMind supports Llama-2 70B with GQA.
- \[2023/07\] TurboMind supports Llama-2 7B/13B.
- \[2023/07\] TurboMind supports tensor-parallel inference of InternLM.

LMDeploy is a toolkit for compressing, deploying, and serving LLM, developed by the [MMRazor](https://github.com/open-mmlab/mmrazor) and [MMDeploy](https://github.com/open-mmlab/mmdeploy) teams. It has the following core features:

- **Efficient Inference Engine (TurboMind)**: Based on [FasterTransformer](https://github.com/NVIDIA/FasterTransformer), we have implemented an efficient inference engine - TurboMind, which supports the inference of LLaMA and its variant models on NVIDIA GPUs.

- **Interactive Inference Mode**: By caching the k/v of attention during multi-round dialogue processes, it remembers dialogue history, thus avoiding repetitive processing of historical sessions.

- **Multi-GPU Model Deployment and Quantization**: We provide comprehensive model deployment and quantification support, and have been validated at different scales.

- **Persistent Batch Inference**: Further optimization of model execution efficiency.

## Supported Models

`LMDeploy` has two inference backends, `Pytorch` and `TurboMind`.

> W4A16 inference requires Nvidia GPU with Ampere architecture or above.

|  Models  | Tensor Parallel | FP16 | KV INT8 | W4A16 | W8A8 |
| :------: | :-------------: | :--: | :-----: | :---: | :--: |
|  Llama   |       Yes       | Yes  |   Yes   |  Yes  |  No  |
|  Llama2  |       Yes       | Yes  |   Yes   |  Yes  |  No  |
| InternLM |       Yes       | Yes  |   Yes   |  Yes  |  No  |

|  Llama   |       Yes       | Yes  |   No    |  No   |  No  |
|  Llama2  |       Yes       | Yes  |   No    |  No   |  No  |
| InternLM |       Yes       | Yes  |   No    |  No   |  No  |

## Performance

**Case I**: output token throughput with fixed input token and output token number (1, 2048)

**Case II**: request throughput with real conversation data

Test Setting: LLaMA-7B, NVIDIA A100(80G)

The output token throughput of TurboMind exceeds 2000 tokens/s, which is about 5% - 15% higher than DeepSpeed overall and outperforms huggingface transformers by up to 2.3x.
And the request throughput of TurboMind is 30% higher than vLLM.

Install lmdeploy with pip ( python 3.8+) or [from source](./docs/en/build.md)

```shell
```

### Deploy InternLM

#### Get InternLM model

```shell
# 1. Download InternLM model

```shell
```

```shell

```

#### Inference by TurboMind

```shell
```

> When inferring with FP16 precision, the InternLM-7B model requires at least 15.7G of GPU memory overhead on TurboMind. <br />
> It is recommended to use NVIDIA cards such as 3090, V100, A100, etc.

> Disable GPU ECC can free up 10% memory, try `sudo nvidia-smi --ecc-config=0` and reboot system.

> Tensor parallel is available to perform inference on multiple GPUs. Add `--tp=<num_gpu>` on `chat` to enable runtime TP.

#### Serving with gradio

```shell
```

#### Serving with Restful API

Launch inference server by:

```shell
```

Then, you can communicate with it by command line,

```shell

```shell
```

or webui,

```shell
```

Refer to [restful_api.md](docs/en/restful_api.md) for more details.

#### Serving with Triton Inference Server

```shell
```

Then, you can communicate with the inference server by command line,

```shell
```

or webui,

```shell
```

```shell
```

or webui,

```shell
```

For the deployment of other supported models, such as LLaMA, LLaMA-2, vicuna and so on, you can find the guide from [here](docs/en/serving.md)

### Inference with PyTorch

For detailed instructions on Inference pytorch models, see [here](docs/en/pytorch.md).

#### Single GPU

```shell
python3 -m lmdeploy.pytorch.chat $NAME_OR_PATH_TO_HF_MODEL \
    --seed 0
```

#### Tensor Parallel with DeepSpeed

```shell
    --seed 0
```

You need to install deepspeed first to use this feature.

```
```

## Quantization

### Step 1. Obtain Quantization Parameters

    --seed 0
```

```
```

First, run the quantization script to obtain the quantization parameters.

> After execution, various parameters needed for quantization will be stored in `$WORK_DIR`; these will be used in the following steps..

```

```

--work_dir $WORK_DIR \             # Folder storing Pytorch format quantization statistics parameters and post-quantization weight

```

### Step 2. Actual Model Quantization

`LMDeploy` supports INT4 quantization of weights and INT8 quantization of KV Cache. Run the corresponding script according to your needs.

#### Weight INT4 Quantization

LMDeploy uses [AWQ](https://arxiv.org/abs/2306.00978) algorithm for model weight quantization

> Requires input from the $WORK_DIR of step 1, and the quantized weights will also be stored in this folder.

```
```

[Click here](./docs/zh_cn/w4a16.md) to view the test results for weight int4 usage.

#### KV Cache INT8 Quantization

```

[Click here](./docs/zh_cn/kv_int8.md) to view the usage method, implementation formula, and test results for kv int8.

> runtime Tensor Parallel for quantilized model is not available. Please setup `--tp` on `deploy` to enable static TP.

We appreciate all contributions to LMDeploy. Please refer to [CONTRIBUTING.md](.github/CONTRIBUTING.md) for the contributing guideline.

## License

### 启动服务

运行脚本

```shell
```

然后用户可以打开 swagger UI: `http://{server_ip}:{server_port}` 详细查看所有的 API 及其使用方法。
我们一共提供四个 restful api，其中三个仿照 OpenAI 的形式。不过，我们建议用户用我们提供的另一个 API: `generate`。
它有更好的性能，提供更多的参数让用户自定义修改。

### python

这是一个 python 示例，展示如何使用 `generate`。

```python

    pload = {

ignore_eos: bool = False) -> Iterable[List[str]]:
    pload = {
    }
        if chunk:

        512):
```

        512):
```

可以使用代码生成工具 [openapi-generator-cli](https://github.com/OpenAPITools/openapi-generator-cli) 将 `http://{server_ip}:{server_port}/openapi.json` 转成 java/rust/golang 客户端。
下面是一个使用示例：

```shell

$ ls rust/*

rust/src:
```

### cURL

cURL 也可以用于查看 API 的输出结果

查看模型列表：

```bash
```

使用 generate:

```bash

rust/src:
```

### cURL

查看模型列表：

```bash
```

```bash
  -d '{
  }'
```

```bash
  -d '{
  }'
```

```bash
  -d '{
  }'
```

restful api 服务可以通过客户端测试，例如

```shell
# restful_api_url 就是 api_server 产生的，比如 http://localhost:23333

-d '{
  }'
```

```shell
```

### webui

也可以直接用 webui 测试使用 restful-api。

```shell
# server_ip 和 server_port 是用来提供 gradio ui 访问服务的
# 例子: python -m lmdeploy.serve.gradio.app http://localhost:23333 localhost 6006 --restful_api True
```

### FAQ

1. 当返回结果结束原因为 `"finish_reason":"length"`，这表示回话长度超过最大值。
   请添加 `"renew_session": true` 到下一次请求中。

2. 当服务端显存 OOM 时，可以适当减小启动服务时的 `instance_num` 个数

3. 当同一个 `instance_id` 的请求给 `generate` 函数后，出现返回空字符串和负值的 `tokens`，应该是第二次问话没有设置 `sequence_start=false`

# Build for SNPE

It is quite simple to support snpe backend: Client/Server mode.

this mode

1. Can split `model convert` and `inference` environments;

- Inference irrelevant matters are done on host
- We can get the real running results of gpu/npu instead of cpu simulation values

2. Can cover cost-sensitive device, armv7/risc-v/mips chips meet product requirements, but often have limited support for Python;

3. Can simplify mmdeploy installation steps. If you only want to convert snpe model and test, you don't need to compile the .whl package.

## 1. Run inference server

Download the prebuilt snpe inference server package, `adb push` it to the phone and execute.
Note that **the phone must have a qcom chip**.

```bash
$ wget https://media.githubusercontent.com/media/tpoisonooo/mmdeploy_snpe_testdata/main/snpe-inference-server-1.59.tar.gz
...
$ sudo apt install adb
$ adb push snpe-inference-server-1.59.tar.gz  /data/local/tmp/

# decompress and execute
$ adb shell

...

$ adb shell
venus:/ $ cd /data/local/tmp
130|venus:/data/local/tmp $ tar xvf snpe-inference-server-1.59.tar.gz
...
130|venus:/data/local/tmp $ source export1.59.sh
130|venus:/data/local/tmp $ ./inference_server
...
  Server listening on [::]:60000
```

At this point the inference service should print all the ipv6 and ipv4 addresses of the device and listen on the port.

tips:

- If `adb devices` cannot find the device, may be:
  - Some cheap cables can only charge and cannot transmit data
  - or the "developer mode" of the phone is not turned on
- If you need to compile the binary by self, please refer to [NDK Cross Compiling snpe Inference Service](../appendix/cross_build_snpe_service.md)
- If a `segmentation fault` occurs when listening on a port, it may be because:
  - The port number is already occupied, use another port

## 2. Build mmdeploy

### 1) Environment

| Matters | Version            | Remarks                |
| ------- | ------------------ | ---------------------- |
| host OS | ubuntu18.04 x86_64 | snpe specified version |
| Python  | **3.6.0**          | snpe specified version |

### 2) Installation

Download [snpe-1.59 from the official website](https://developer.qualcomm.com/qfile/69652/snpe-1.59.0.zip)

```bash
$ unzip snpe-1.59.0.zip
$ export SNPE_ROOT=${PWD}/snpe-1.59.0.3230
$ export PYTHONPATH=${PWD}/service/snpe/client:${SNPE_ROOT}/lib/python:${PYTHONPATH}
$ export LD_LIBRARY_PATH=${SNPE_ROOT}/lib/x86_64-linux-clang:${LD_LIBRARY_PATH}
$ export PATH=${SNPE_ROOT}/bin/x86_64-linux-clang:${PATH}
$ python3 -m pip install -e .
```

## 3. Test the model

```

Take Resnet-18 as an example. First refer to [documentation to install mmpretrain](https://github.com/open-mmlab/mmpretrain/tree/main)  and  use `tools/deploy.py` to convert the model.

```bash
$ export MODEL_CONFIG=/path/to/mmpretrain/configs/resnet/resnet18_8xb16_cifar10.py
$ export MODEL_PATH=https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_b16x8_cifar10_20210528-bd6371c8.pth

# Convert the model
$ python3 tools/deploy.py  configs/mmpretrain/classification_snpe_static.py $MODEL_CONFIG  $MODEL_PATH   /path/to/test.png   --work-dir resnet18   --device cpu  --uri 10.0.0.1\:60000  --dump-info

# Test

# Test
$ python3 tools/test.py configs/mmpretrain/classification_snpe_static.py   $MODEL_CONFIG    --model reset18/end2end.dlc   --metrics accuracy precision f1_score recall  --uri 10.0.0.1\:60000
```

Note that `--uri` is required to specify the ip and port of the snpe inference service, ipv4 and ipv6 addresses can be used.

## 4. Build SDK with Android SDK

If you also need to compile mmdeploy SDK with Android NDK, please continue reading.

### 1) Download NDK and OpenCV package and setup environment

```bash
# Download android OCV
$ export OPENCV_VERSION=4.5.4
$ wget https://github.com/opencv/opencv/releases/download/${OPENCV_VERSION}/opencv-${OPENCV_VERSION}-android-sdk.zip
$ unzip opencv-${OPENCV_VERSION}-android-sdk.zip

$ export ANDROID_OCV_ROOT=`realpath opencv-${OPENCV_VERSION}-android-sdk`

# Download ndk r23b
$ wget https://dl.google.com/android/repository/android-ndk-r23b-linux.zip
$ unzip android-ndk-r23b-linux.zip

$ export ANDROID_NDK_ROOT=`realpath android-ndk-r23b`

```

### 2) Compile mmdeploy SDK and demo

```bash
$ mkdir build && cd build
$ cmake .. \
  -DMMDEPLOY_BUILD_SDK=ON \
  -DCMAKE_TOOLCHAIN_FILE=${ANDROID_NDK_ROOT}/build/cmake/android.toolchain.cmake \
  -DMMDEPLOY_TARGET_BACKENDS=snpe \
  -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-30  \
  -DANDROID_STL=c++_static  \
  -DOpenCV_DIR=${ANDROID_OCV_ROOT}/sdk/native/jni/abi-arm64-v8a \
  -DMMDEPLOY_BUILD_EXAMPLES=ON

  $ make && make install
  $ tree ./bin
./bin
├── image_classification
├── image_restorer
├── image_segmentation
├── mmdeploy_onnx2ncnn
├── object_detection
├── ocr
├── pose_detection
└── rotated_object_detection
```

| Options                       | Description                                                  |

├── ocr
```

| ----------------------------- | ------------------------------------------------------------ |
| CMAKE_TOOLCHAIN_FILE          | Load NDK parameters, mainly used to select compiler          |
| MMDEPLOY_TARGET_BACKENDS=snpe | Inference backend                                            |
| ANDROID_STL=c++\_static       | In case of NDK environment can not find suitable c++ library |
| MMDEPLOY_SHARED_LIBS=ON       | snpe does not provide static library                         |

[Here](../01-how-to-build/cmake_option.md) is all cmake build option description.

### 3) Run the demo

First make sure that`--dump-info`is used during convert model, so that the `resnet18` directory has the files required by the SDK such as `pipeline.json`.

`adb push` the model directory, executable file and .so to the device.

```bash
$ adb push resnet18  /data/local/tmp
$ adb push tests/data/tiger.jpeg /data/local/tmp/resnet18/

$ cd /path/to/install/
$ adb push lib /data/local/tmp
$ adb push bin/image_classification /data/local/tmp/resnet18/
```

Set up environment variable and execute the sample.

```bash
$ adb push /path/to/mmpretrain/demo/demo.JPEG /data/local/tmp
$ adb shell
venus:/ $ cd /data/local/tmp/resnet18
venus:/data/local/tmp/resnet18 $ export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/data/local/tmp/lib

venus:/data/local/tmp/resnet18 $ ./image_classification cpu ./  tiger.jpeg
..
label: 3, score: 0.3214
```

    <sup>
      </a>
    </sup>
    <sup>
      </a>
    </sup>
  </div>

[![Docs](https://img.shields.io/badge/docs-latest-blue)](https://mmpretrain.readthedocs.io/zh_CN/latest/)

[![license](https://img.shields.io/github/license/open-mmlab/mmpretrain.svg)](https://github.com/open-mmlab/mmpretrain/blob/main/LICENSE)
[![open issues](https://isitmaintained.com/badge/open/open-mmlab/mmpretrain.svg)](https://github.com/open-mmlab/mmpretrain/issues)
[![issue resolution](https://isitmaintained.com/badge/resolution/open-mmlab/mmpretrain.svg)](https://github.com/open-mmlab/mmpretrain/issues)

[📘 中文文档](https://mmpretrain.readthedocs.io/zh_CN/latest/) |
[🛠️ 安装教程](https://mmpretrain.readthedocs.io/zh_CN/latest/get_started.html) |
[👀 模型库](https://mmpretrain.readthedocs.io/zh_CN/latest/modelzoo_statistics.html) |
[🆕 更新日志](https://mmpretrain.readthedocs.io/zh_CN/latest/notes/changelog.html) |
[🤔 报告问题](https://github.com/open-mmlab/mmpretrain/issues/new/choose)

</div>

</div>

MMPreTrain 是一款基于 PyTorch 的开源深度学习预训练工具箱，是 [OpenMMLab](https://openmmlab.com/) 项目的成员之一

`主分支`代码目前支持 PyTorch 1.8 以上的版本。

### 主要特性

- 支持多样的主干网络与预训练模型
- 支持多种训练策略（有监督学习，无监督学习，多模态学习等）
- 提供多种训练技巧
- 大量的训练配置文件
- 高效率和高可扩展性
- 功能强大的工具箱，有助于模型分析和实验
- 支持多种开箱即用的推理任务

### 主要特性

- 提供多种训练技巧
  - 图像分类
  - 图像描述（Image Caption）
  - 视觉问答（Visual Question Answering）
  - 视觉定位（Visual Grounding）
  - 检索（图搜图，图搜文，文搜图）

## 更新日志

🌟 2023/7/28 发布了 v1.0.1 版本

修复部分 bug 和增强算法库功能。细节请参考 [更新日志](https://mmpretrain.readthedocs.io/zh_CN/latest/notes/changelog.html)。

🌟 2023/7/4 发布了 v1.0.0 版本

- 支持更多**多模态**算法的推理, 例如 [**LLaVA**](./configs/llava/), [**MiniGPT-4**](./configs/minigpt4), [**Otter**](./configs/otter/) 等。
- 支持约 **10 个多模态**数据集!
- 添加自监督学习算法 [**iTPN**](./configs/itpn/), [**SparK**](./configs/spark/)。

- 提供[新配置文件](./mmpretrain/configs/)和 [DeepSpeed/FSDP](./configs/mae/benchmarks/) 的样例。这是[新配置文件](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#a-pure-python-style-configuration-file-beta) 和 [DeepSpeed/FSDP with FlexibleRunner](https://mmengine.readthedocs.io/en/latest/api/generated/mmengine.runner.FlexibleRunner.html#mmengine.runner.FlexibleRunner) 的文档链接。

🌟 2023/5/22 发布了 v1.0.0rc8 版本

- 支持多种多模态算法和推理器。您可以通过 [gradio demo](https://github.com/open-mmlab/mmpretrain/tree/main/projects/gradio_demo) 探索这些功能!
- 新增 EVA-02，Dino-V2，ViT-SAM 和 GLIP 主干网络。
- 将 torchvision 变换注册到 MMPretrain，现在您可以轻松地将 torchvision 的数据增强集成到 MMPretrain 中。

之前版本更新内容

- 整和来自 MMSelfSup 的自监督学习算法，例如 `MAE`, `BEiT` 等
- 支持了 **RIFormer**，简单但有效的视觉主干网络，却移除了 token mixer
- 重构数据管道可视化
- 支持了 **LeViT**, **XCiT**, **ViG**, **ConvNeXt-V2**, **EVA**, **RevViT**, **EfficientnetV2**, **CLIP**, **TinyViT** 和 **MixMIM** 等骨干网络结构

这个版本引入一个全新的，可扩展性强的训练和测试引擎，但目前仍在开发中。欢迎根据 [文档](https://mmpretrain.readthedocs.io/zh_CN/latest/) 进行试用。

同时，新版本中存在一些与旧版本不兼容的修改。请查看 [迁移文档](https://mmpretrain.readthedocs.io/zh_CN/latest/migration.html) 来详细了解这些变动。

发布历史和更新细节请参考 [更新日志](https://mmpretrain.readthedocs.io/zh_CN/latest/notes/changelog.html)。

## 安装

以下是安装的简要步骤：

```shell
```

更详细的步骤请参考 [安装指南](https://mmpretrain.readthedocs.io/zh_CN/latest/get_started.html) 进行安装。

如果需要多模态模型，请使用如下方式安装额外的依赖：

```shell
```

## 基础教程

我们为新用户提供了一系列基础教程：

```

```shell
```

## 基础教程

- [学习配置文件](https://mmpretrain.readthedocs.io/zh_CN/latest/user_guides/config.html)
- [准备数据集](https://mmpretrain.readthedocs.io/zh_CN/latest/user_guides/dataset_prepare.html)
- [使用现有模型推理](https://mmpretrain.readthedocs.io/zh_CN/latest/user_guides/inference.html)
- [训练](https://mmpretrain.readthedocs.io/zh_CN/latest/user_guides/train.html)
- [测试](https://mmpretrain.readthedocs.io/zh_CN/latest/user_guides/test.html)
- [下游任务](https://mmpretrain.readthedocs.io/zh_CN/latest/user_guides/downstream.html)

关于更多的信息，请查阅我们的 [相关文档](https://mmpretrain.readthedocs.io/zh_CN/latest/)。

## 模型库

相关结果和模型可在 [模型库](https://mmpretrain.readthedocs.io/zh_CN/latest/modelzoo_statistics.html) 中获得。

  <b>概览</b>
</div>
  <tbody>
      <td>

  <b>概览</b>
</div>
  <tbody>
      <td>
        <b>支持的主干网络</b>
      </td>
      <td>
        <b>自监督学习</b>
      </td>
      <td>
        <b>多模态算法</b>
      </td>
      <td>
        <b>其它</b>
      </td>
    </tr>
      <td>
        <ul>

<li><a href="configs/shufflenet_v2">ShuffleNet V2</a></li>

<li><a href="configs/t2t_vit">T2T-ViT</a></li>

<li><a href="configs/hornet">HorNet</a></li>
        </ul>
      </td>
      <td>
        <ul>

<li><a href="configs/dinov2">DINO V2</a></li>
        </ul>
      </td>
      <td>
        <ul>

<li><a href="configs/beit">BEiT (ICLR'2022)</a></li>
        </ul>
      </td>
      <td>
        <ul>

<ul>
        </ul>
      </td>
      <td>
      图像检索任务：
        <ul>
        </ul>
      训练和测试 Tips:
        <ul>

<li><a href="https://arxiv.org/abs/1805.09501">AutoAug</a></li>
        </ul>
      </td>
  </tbody>
</table>

## 参与贡献

我们非常欢迎任何有助于提升 MMPreTrain 的贡献，请参考 [贡献指南](https://mmpretrain.readthedocs.io/zh_CN/latest/notes/contribution_guide.html) 来了解如何参与贡献。

## 致谢

MMPreTrain 是一款由不同学校和公司共同贡献的开源项目。我们感谢所有为项目提供算法复现和新功能支持的贡献者，以及提供宝贵反馈的用户。

## 引用

如果你在研究中使用了本项目的代码或者性能基准，请参考如下 bibtex 引用 MMPreTrain。

```BibTeX
}
```

## 许可证

该项目开源自 [Apache 2.0 license](LICENSE).

author={MMPreTrain Contributors},
}
```

## 许可证

</div>

# BEiT

> [BEiT: BERT Pre-Training of Image Transformers](https://arxiv.org/abs/2106.08254)

<a href="https://github.com/microsoft/unilm/tree/master/beit">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.23.0/mmseg/models/backbones/beit.py#1404">Code Snippet</a>

We introduce a self-supervised vision representation model BEiT, which stands for Bidirectional Encoder representation from Image Transformers. Following BERT developed in the natural language processing area, we propose a masked image modeling task to pretrain vision Transformers. Specifically, each image has two views in our pre-training, i.e, image patches (such as 16x16 pixels), and visual tokens (i.e., discrete tokens). We first "tokenize" the original image into visual tokens. Then we randomly mask some image patches and fed them into the backbone Transformer. The pre-training objective is to recover the original visual tokens based on the corrupted image patches. After pre-training BEiT, we directly fine-tune the model parameters on downstream tasks by appending task layers upon the pretrained encoder. Experimental results on image classification and semantic segmentation show that our model achieves competitive results with previous pre-training methods. For example, base-size BEiT achieves 83.2% top-1 accuracy on ImageNet-1K, significantly outperforming from-scratch DeiT training (81.8%) with the same setup. Moreover, large-size BEiT obtains 86.3% only using ImageNet-1K, even outperforming ViT-L with supervised pre-training on ImageNet-22K (85.2%). The code and pretrained models are available at [this https URL](https://github.com/microsoft/unilm/tree/master/beit).

<img src="https://user-images.githubusercontent.com/93248678/160155758-781c9a45-b1d7-4530-9015-88eca6645006.png" width="70%"/>
</div>

## Usage

To use other repositories' pre-trained models, it is necessary to convert keys.

We provide a script [`beit2mmseg.py`](../../tools/model_converters/beit2mmseg.py) in the tools directory to convert the key of models from [the official repo](https://github.com/microsoft/unilm/tree/master/beit/semantic_segmentation) to MMSegmentation style.

```shell
python tools/model_converters/beit2mmseg.py ${PRETRAIN_PATH} ${STORE_PATH}
```

E.g.

```shell
python tools/model_converters/beit2mmseg.py https://conversationhub.blob.core.windows.net/beit-share-public/beit/beit_base_patch16_224_pt22k_ft22k.pth pretrain/beit_base_patch16_224_pt22k_ft22k.pth
```

In our default setting, pretrained models could be defined below:

```

| pretrained models | original models                                                                                                             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------- |
| BEiT_base.pth     | ['BEiT_base'](https://conversationhub.blob.core.windows.net/beit-share-public/beit/beit_base_patch16_224_pt22k_ft22k.pth)   |
| BEiT_large.pth    | ['BEiT_large'](https://conversationhub.blob.core.windows.net/beit-share-public/beit/beit_large_patch16_224_pt22k_ft22k.pth) |

Verify the single-scale results of the model:

```shell
sh tools/dist_test.sh \
configs/beit/upernet_beit-large_fp16_8x1_640x640_160k_ade20k.py \
upernet_beit-large_fp16_8x1_640x640_160k_ade20k-8fc0dd5d.pth $GPUS --eval mIoU
```

```shell
```

Since relative position embedding requires the input length and width to be equal, the sliding window is adopted for multi-scale inference. So we set min_size=640, that is, the shortest edge is 640. So the multi-scale inference of config is performed separately, instead of '--aug-test'. For multi-scale inference:

```shell
configs/beit/upernet_beit-large_fp16_640x640_160k_ade20k_ms.py \
```

### ADE20K

```

### ADE20K

| Method  | Backbone | Crop Size | pretrain     | pretrain img size | Batch Size | Lr schd | Mem (GB) | Inf time (fps) | Device | mIoU  | mIoU(ms+flip) | config                                                                                                                           | download                                                                                                                                                                                                                                                                                                                                                 |

| UPerNet | BEiT-B   | 640x640   | ImageNet-22K | 224x224           | 16         | 160000  | 15.88    | 2.00           | V100   | 53.08 |         53.84 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/beit/beit-base_upernet_8xb2-160k_ade20k-640x640.py)      | [model](https://download.openmmlab.com/mmsegmentation/v0.5/beit/upernet_beit-base_8x2_640x640_160k_ade20k/upernet_beit-base_8x2_640x640_160k_ade20k-eead221d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/beit/upernet_beit-base_8x2_640x640_160k_ade20k/upernet_beit-base_8x2_640x640_160k_ade20k.log.json)                         |

| UPerNet | BEiT-L   | 640x640   | ImageNet-22K | 224x224           | 8          | 320000  | 22.64    | 0.96           | V100   | 56.33 |         56.84 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/beit/beit-large_upernet_8xb1-amp-160k_ade20k-640x640.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/beit/upernet_beit-large_fp16_8x1_640x640_160k_ade20k/upernet_beit-large_fp16_8x1_640x640_160k_ade20k-8fc0dd5d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/beit/upernet_beit-large_fp16_8x1_640x640_160k_ade20k/upernet_beit-large_fp16_8x1_640x640_160k_ade20k.log.json) |

```bibtex
@inproceedings{beit,
      title={{BEiT}: {BERT} Pre-Training of Image Transformers},
      author={Hangbo Bao and Li Dong and Songhao Piao and Furu Wei},
      booktitle={International Conference on Learning Representations},
      url={https://openreview.net/forum?id=p-BhZSz59o4}
}
```

> This is a README template for community `projects/`.

> All the fields in this README are **mandatory** for others to understand what you have achieved in this implementation. If you still feel unclear about the requirements, please read our [contribution guide](https://mmocr.readthedocs.io/en/dev-1.x/notes/contribution_guide.html), [projects FAQ](../faq.md), or approach us in [Discussions](https://github.com/open-mmlab/mmocr/discussions).

> Share any information you would like others to know. For example:
>
> Author: @xxx.
>
> This is an implementation of \[XXX\].

## Usage

> For a typical model, this section should contain the commands for training and testing. You are also suggested to dump your environment specification to env.yml by `conda env export > env.yml`.

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `example_project/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
# Linux
```

In MMOCR's root directory, run the following command to train the model:

```bash
mim train mmocr configs/dbnet_dummy-resnet_fpnc_1200e_icdar2015.py --work-dir work_dirs/dummy_mae/
```

```bash
mim train mmocr configs/dbnet_dummy-resnet_fpnc_1200e_icdar2015.py --work-dir work_dirs/dummy_mae/ --launcher pytorch --gpus 8
```

```bash
```

In MMOCR's root directory, run the following command to test the model:

```bash
mim test mmocr configs/dbnet_dummy-resnet_fpnc_1200e_icdar2015.py --work-dir work_dirs/dummy_mae/ --checkpoint ${CHECKPOINT_PATH}
```

## Results

> List the results as usually done in other model's README. [Example](https://github.com/open-mmlab/mmocr/blob/1.x/configs/textdet/dbnet/README.md#results-and-models)
>
> You should claim whether this is based on the pre-trained weights, which are converted from the official release; or it's a reproduced result obtained from retraining the model in this project.

|                              Method                               |  Backbone   | Pretrained Model |  Training set   |    Test set    | #epoch | Test size | Precision | Recall | Hmean  |         Download         |

| :---------------------------------------------------------------: | :---------: | :--------------: | :-------------: | :------------: | :----: | :-------: | :-------: | :----: | :----: | :----------------------: |
| [DBNet_dummy](configs/dbnet_dummy-resnet_fpnc_1200e_icdar2015.py) | DummyResNet |        -         | ICDAR2015 Train | ICDAR2015 Test |  1200  |    736    |  0.8853   | 0.7583 | 0.8169 | [model](<>) \| [log](<>) |

> You may remove this section if not applicable.

```bibtex
@software{MMOCR_Contributors_OpenMMLab_Text_Detection_2020,
author = {{MMOCR Contributors}},
license = {Apache-2.0},
month = {8},
title = {{OpenMMLab Text Detection, Recognition and Understanding Toolbox}},
url = {https://github.com/open-mmlab/mmocr},
version = {0.3.0},
}
```

Here is a checklist illustrating a usual development workflow of a successful project, and also serves as an overview of this project's progress.

}
```

> The PIC (person in charge) or contributors of this project should check all the items that they believe have been finished, which will further be verified by codebase maintainers via a PR.
>
> OpenMMLab's maintainer will review the code to ensure the project's quality. Reaching the first milestone means that this project suffices the minimum requirement of being merged into 'projects/'. But this project is only eligible to become a part of the core package upon attaining the last milestone.
>

>
> Note that keeping this section up-to-date is crucial not only for this project's developers but the entire community, since there might be some other contributors joining this project and deciding their starting point from this list. It also helps maintainers accurately estimate time and effort on further code polishing, if needed.
>
> A project does not necessarily have to be finished in a single PR, but it's essential for the project to at least reach the first milestone in its very first PR.

    > The code's design shall follow existing interfaces and convention. For example, each model component should be registered into `mmocr.registry.MODELS` and configurable via a config file.

    > Each major object should contain a docstring, describing its functionality and arguments. If you have adapted the code from other open-source projects, don't forget to cite the source project in docstring and make sure your behavior is not against its license. Typically, we do not accept any code snippet under GPL license. [A Short Guide to Open Source Licenses](https://medium.com/nationwide-technology/a-short-guide-to-open-source-licenses-cf5b1c329edd)

    > If you are reproducing the result from a paper, make sure your model's inference-time performance matches that in the original paper. The weights usually could be obtained by simply renaming the keys in the official pre-trained weights. This test could be skipped though, if you are able to prove the training-time correctness and check the second milestone.

    > As this template does.

    > If you are reproducing the result from a paper, checking this item means that you should have trained your model from scratch based on the original paper's specification and verified that the final result matches the report within a minor error range.

    > Ideally *all* the methods should have [type hints](https://www.pythontutorial.net/python-basics/python-type-hints/) and [docstrings](https://google.github.io/styleguide/pyguide.html#381-docstrings). [Example](https://github.com/open-mmlab/mmocr/blob/76637a290507f151215d299707c57cea5120976e/mmocr/utils/polygon_utils.py#L80-L96)

    > Unit tests for each module are required. [Example](https://github.com/open-mmlab/mmocr/blob/76637a290507f151215d299707c57cea5120976e/tests/test_utils/test_polygon_utils.py#L97-L106)

    > Refactor your code according to reviewer's comment.

    > It will be parsed by MIM and Inferencer. [Example](https://github.com/open-mmlab/mmocr/blob/1.x/configs/textdet/dbnet/metafile.yml)

  > In particular, you may have to refactor this README into a standard one. [Example](/configs/textdet/dbnet/README.md)

# StyleGANv1 (CVPR'2019)

> [A Style-Based Generator Architecture for Generative Adversarial Networks](https://openaccess.thecvf.com/content_CVPR_2019/html/Karras_A_Style-Based_Generator_Architecture_for_Generative_Adversarial_Networks_CVPR_2019_paper.html)

We propose an alternative generator architecture for generative adversarial networks, borrowing from style transfer literature. The new architecture leads to an automatically learned, unsupervised separation of high-level attributes (e.g., pose and identity when trained on human faces) and stochastic variation in the generated images (e.g., freckles, hair), and it enables intuitive, scale-specific control of the synthesis. The new generator improves the state-of-the-art in terms of traditional distribution quality metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation. To quantify interpolation quality and disentanglement, we propose two new, automated methods that are applicable to any generator architecture. Finally, we introduce a new, highly varied and high-quality dataset of human faces.

<img src="https://user-images.githubusercontent.com/28132635/143055313-f4988870-2963-4a2f-916e-0de0e04eb474.JPG"/>
</div>

  <b> Results (compressed) from StyleGANv1 trained by mmagic</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/12726765/113845642-4f9ee980-97c8-11eb-85c7-49d6d21dd46b.png" width="800"/>
</div>

|                                Model                                | Dataset | FID50k |  P&R50k_full  |                                                  Download                                                   |
| :-----------------------------------------------------------------: | :-----: | :----: | :-----------: | :---------------------------------------------------------------------------------------------------------: |

|  [styleganv1_ffhq_256](./styleganv1_ffhq-256x256_8xb4-25Mimgs.py)   |  FFHQ   | 6.090  | 70.228/27.050 | [model](https://download.openmmlab.com/mmediting/styleganv1/styleganv1_ffhq_256_g8_25Mimg_20210407_161748-0094da86.pth) |
| [styleganv1_ffhq_1024](./styleganv1_ffhq-1024x1024_8xb4-25Mimgs.py) |  FFHQ   | 4.056  | 70.302/36.869 | [model](https://download.openmmlab.com/mmediting/styleganv1/styleganv1_ffhq_1024_g8_25Mimg_20210407_161627-850a7234.pth) |

```latex
@inproceedings{karras2019style,
  title={A style-based generator architecture for generative adversarial networks},
  author={Karras, Tero and Laine, Samuli and Aila, Timo},
  pages={4401--4410},
  url={https://openaccess.thecvf.com/content_CVPR_2019/html/Karras_A_Style-Based_Generator_Architecture_for_Generative_Adversarial_Networks_CVPR_2019_paper.html},
}
```

# SimpleCopyPaste

> [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://arxiv.org/abs/2012.07177)

Building instance segmentation models that are data-efficient and can handle rare object categories is an important challenge in computer vision. Leveraging data augmentations is a promising direction towards addressing this challenge. Here, we perform a systematic study of the Copy-Paste augmentation (\[13, 12\]) for instance segmentation where we randomly paste objects onto an image. Prior studies on Copy-Paste relied on modeling the surrounding visual context for pasting the objects. However, we find that the simple mechanism of pasting objects randomly is good enough and can provide solid gains on top of strong baselines. Furthermore, we show Copy-Paste is additive with semi-supervised methods that leverage extra data through pseudo labeling (e.g. self-training). On COCO instance segmentation, we achieve 49.1 mask AP and 57.3 box AP, an improvement of +0.6 mask AP and +1.5 box AP over the previous state-of-the-art. We further demonstrate that Copy-Paste can lead to significant improvements on the LVIS benchmark. Our baseline model outperforms the LVIS 2020 Challenge winning entry by +3.6 mask AP on rare categories.

<img src="https://user-images.githubusercontent.com/40661020/161843866-c5b769da-58b2-4c1f-8078-db4a4ded3881.png"/>
</div>

### Mask R-CNN with Standard Scale Jittering (SSJ) and Simple Copy-Paste(SCP)

Standard Scale Jittering(SSJ) resizes and crops an image with a resize range of 0.8 to 1.25 of the original image size, and Simple Copy-Paste(SCP) selects a random subset of objects from one of the images and pastes them onto the other image.

| Backbone | Training schedule | Augmentation | batch size | box AP | mask AP |                                         Config                                         |                                                                                                                                                                                                                               Download                                                                                                                                                                                                                               |

|   R-50   |        90k        |     SSJ      |     64     |  43.3  |  39.0   |   [config](./mask-rcnn_r50_fpn_rpn-2conv_4conv1fc_syncbn-all_32xb2-ssj-90k_coco.py)    |           [model](https://download.openmmlab.com/mmdetection/v2.0/simple_copy_paste/mask_rcnn_r50_fpn_syncbn-all_rpn-2conv_ssj_32x2_90k_coco/mask_rcnn_r50_fpn_syncbn-all_rpn-2conv_ssj_32x2_90k_coco_20220316_181409-f79c84c5.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/simple_copy_paste/mask_rcnn_r50_fpn_syncbn-all_rpn-2conv_ssj_32x2_90k_coco/mask_rcnn_r50_fpn_syncbn-all_rpn-2conv_ssj_32x2_90k_coco_20220316_181409.log.json)           |

|   R-50   |        90k        |   SSJ+SCP    |     64     |  43.8  |  39.2   | [config](./mask-rcnn_r50_fpn_rpn-2conv_4conv1fc_syncbn-all_32xb2-ssj-scp-90k_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/simple_copy_paste/mask_rcnn_r50_fpn_syncbn-all_rpn-2conv_ssj_scp_32x2_90k_coco/mask_rcnn_r50_fpn_syncbn-all_rpn-2conv_ssj_scp_32x2_90k_coco_20220316_181307-6bc5726f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/simple_copy_paste/mask_rcnn_r50_fpn_syncbn-all_rpn-2conv_ssj_scp_32x2_90k_coco/mask_rcnn_r50_fpn_syncbn-all_rpn-2conv_ssj_scp_32x2_90k_coco_20220316_181307.log.json)   |

|   R-50   |       270k        |     SSJ      |     64     |  43.5  |  39.1   |   [config](./mask-rcnn_r50_fpn_rpn-2conv_4conv1fc_syncbn-all_32xb2-ssj-270k_coco.py)   |         [model](https://download.openmmlab.com/mmdetection/v2.0/simple_copy_paste/mask_rcnn_r50_fpn_syncbn-all_rpn-2conv_ssj_32x2_270k_coco/mask_rcnn_r50_fpn_syncbn-all_rpn-2conv_ssj_32x2_270k_coco_20220324_182940-33a100c5.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/simple_copy_paste/mask_rcnn_r50_fpn_syncbn-all_rpn-2conv_ssj_32x2_270k_coco/mask_rcnn_r50_fpn_syncbn-all_rpn-2conv_ssj_32x2_270k_coco_20220324_182940.log.json)         |

|   R-50   |       270k        |   SSJ+SCP    |     64     |  45.1  |  40.3   | [config](./mask-rcnn_r50_fpn_rpn-2conv_4conv1fc_syncbn-all_32xb2-ssj-scp-270k_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/simple_copy_paste/mask_rcnn_r50_fpn_syncbn-all_rpn-2conv_ssj_scp_32x2_270k_coco/mask_rcnn_r50_fpn_syncbn-all_rpn-2conv_ssj_scp_32x2_270k_coco_20220324_201229-80ee90b7.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/simple_copy_paste/mask_rcnn_r50_fpn_syncbn-all_rpn-2conv_ssj_scp_32x2_270k_coco/mask_rcnn_r50_fpn_syncbn-all_rpn-2conv_ssj_scp_32x2_270k_coco_20220324_201229.log.json) |

```latex
@inproceedings{ghiasi2021simple,
  title={Simple copy-paste is a strong data augmentation method for instance segmentation},
  author={Ghiasi, Golnaz and Cui, Yin and Srinivas, Aravind and Qian, Rui and Lin, Tsung-Yi and Cubuk, Ekin D and Le, Quoc V and Zoph, Barret},

author={Ghiasi, Golnaz and Cui, Yin and Srinivas, Aravind and Qian, Rui and Lin, Tsung-Yi and Cubuk, Ekin D and Le, Quoc V and Zoph, Barret},
  pages={2918--2928},
}
```

# Chest Image Dataset for Pneumothorax Segmentation

This project supports **`Chest Image Dataset for Pneumothorax Segmentation`**, which can be downloaded from [here](https://tianchi.aliyun.com/dataset/83075).

Pneumothorax can be caused by a blunt chest injury, damage from underlying lung disease, or most horrifying—it may occur for no obvious reason at all. On some occasions, a collapsed lung can be a life-threatening event.
Pneumothorax is usually diagnosed by a radiologist on a chest x-ray, and can sometimes be very difficult to confirm. An accurate AI algorithm to detect pneumothorax would be useful in a lot of clinical scenarios. AI could be used to triage chest radiographs for priority interpretation, or to provide a more confident diagnosis for non-radiologists.

The dataset is provided by the Society for Imaging Informatics in Medicine(SIIM), American College of Radiology (ACR),Society of Thoracic Radiology (STR) and MD.ai. You can develop a model to classify (and if present, segment) pneumothorax from a set of chest radiographic images. If successful, you could aid in the early recognition of pneumothoraces and save lives.

| Dataset name                                                          | Anatomical region | Task type    | Modality | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License                                                            |
| --------------------------------------------------------------------- | ----------------- | ------------ | -------- | ------------ | --------------------- | ---------------------- | ------------ | ------------------------------------------------------------------ |

| [pneumothorax segmentation](https://tianchi.aliyun.com/dataset/83075) | thorax            | segmentation | x_ray    | 2            | 12089/-/3205          | yes/-/no               | -            | [CC-BY-SA-NC 4.0](https://creativecommons.org/licenses/by-sa/4.0/) |

|      normal       |   12089    |   99.75    |    -     |    -     |     -     |     -     |
| pneumothorax area |    2669    |    0.25    |    -     |    -     |     -     |     -     |

Note:

![bac](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/x_ray/chest_image_pneum/chest_image_pneum_dataset.png)

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `chest_image_pneum/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
```

- download dataset from [here](https://tianchi.aliyun.com/dataset/83075) and decompress data to path `'data/'`.

```none
  ├── mmseg
  │   │   │   │   ├── chest_image_pneum

```

|      normal       |    9637    |   99.75    |   2410   |  99.74   |     -     |     -     |
| pneumothorax area |    2137    |    0.25    |   532    |   0.26   |     -     |     -     |

```shell
```

```shell
```

```shell
```

## Results

|     Method      | Backbone | Crop Size |   lr   | mIoU | mDice |                                         config                                         |         download         |

| :-------------: | :------: | :-------: | :----: | :--: | :---: | :------------------------------------------------------------------------------------: | :----------------------: |
| fcn_unet_s5-d16 |   unet   |  512x512  |  0.01  |  -   |   -   |  [config](./configs/fcn-unet-s5-d16_unet_1xb16-0.01-20k_chest-image-pneum-512x512.py)  | [model](<>) \| [log](<>) |
| fcn_unet_s5-d16 |   unet   |  512x512  | 0.001  |  -   |   -   | [config](./configs/fcn-unet-s5-d16_unet_1xb16-0.001-20k_chest-image-pneum-512x512.py)  | [model](<>) \| [log](<>) |
| fcn_unet_s5-d16 |   unet   |  512x512  | 0.0001 |  -   |   -   | [config](./configs/fcn-unet-s5-d16_unet_1xb16-0.0001-20k_chest-image-pneum-512x512.py) | [model](<>) \| [log](<>) |

# I3D

[Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://openaccess.thecvf.com/content_cvpr_2017/html/Carreira_Quo_Vadis_Action_CVPR_2017_paper.html)

[Non-local Neural Networks](https://openaccess.thecvf.com/content_cvpr_2018/html/Wang_Non-Local_Neural_Networks_CVPR_2018_paper.html)

The paucity of videos in current action classification datasets (UCF-101 and HMDB-51) has made it difficult to identify good video architectures, as most methods obtain similar performance on existing small-scale benchmarks. This paper re-evaluates state-of-the-art architectures in light of the new Kinetics Human Action Video dataset. Kinetics has two orders of magnitude more data, with 400 human action classes and over 400 clips per class, and is collected from realistic, challenging YouTube videos. We provide an analysis on how current architectures fare on the task of action classification on this dataset and how much performance improves on the smaller benchmark datasets after pre-training on Kinetics. We also introduce a new Two-Stream Inflated 3D ConvNet (I3D) that is based on 2D ConvNet inflation: filters and pooling kernels of very deep image classification ConvNets are expanded into 3D, making it possible to learn seamless spatio-temporal feature extractors from video while leveraging successful ImageNet architecture designs and even their parameters. We show that, after pre-training on Kinetics, I3D models considerably improve upon the state-of-the-art in action classification, reaching 80.9% on HMDB-51 and 98.0% on UCF-101.

<img src="https://user-images.githubusercontent.com/34324155/143043624-1944704a-5d3e-4a3f-b258-1505c49f6092.png" width="800"/>
</div>

| frame sampling strategy | resolution | gpus |           backbone            | pretrain | top1 acc | top5 acc | testing protocol  | FLOPs  | params |            config            |            ckpt             |            log             |
| :---------------------: | :--------: | :--: | :---------------------------: | :------: | :------: | :------: | :---------------: | :----: | :----: | :--------------------------: | :-------------------------: | :------------------------: |

|         32x2x1          |  224x224   |  8   | ResNet50 (NonLocalDotProduct) | ImageNet |  74.80   |  92.07   | 10 clips x 3 crop | 59.3G  | 35.4M  | [config](/configs/recognition/i3d/i3d_imagenet-pretrained-r50-nl-dot-product_8xb8-32x2x1-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/i3d/i3d_imagenet-pretrained-r50-nl-dot-product_8xb8-32x2x1-100e_kinetics400-rgb/i3d_imagenet-pretrained-r50-nl-dot-product_8xb8-32x2x1-100e_kinetics400-rgb_20220812-8e1f2148.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/i3d/i3d_imagenet-pretrained-r50-nl-dot-product_8xb8-32x2x1-100e_kinetics400-rgb/i3d_imagenet-pretrained-r50-nl-dot-product_8xb8-32x2x1-100e_kinetics400-rgb.log) |

|         32x2x1          |  224x224   |  8   | ResNet50 (NonLocalEmbedGauss) | ImageNet |  74.73   |  91.80   | 10 clips x 3 crop | 59.3G  | 35.4M  | [config](/configs/recognition/i3d/i3d_imagenet-pretrained-r50-nl-embedded-gaussian_8xb8-32x2x1-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/i3d/i3d_imagenet-pretrained-r50-nl-embedded-gaussian_8xb8-32x2x1-100e_kinetics400-rgb/i3d_imagenet-pretrained-r50-nl-embedded-gaussian_8xb8-32x2x1-100e_kinetics400-rgb_20220812-afd8f562.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/i3d/i3d_imagenet-pretrained-r50-nl-embedded-gaussian_8xb8-32x2x1-100e_kinetics400-rgb/i3d_imagenet-pretrained-r50-nl-embedded-gaussian_8xb8-32x2x1-100e_kinetics400-rgb.log) |

|         32x2x1          |  224x224   |  8   |   ResNet50 (NonLocalGauss)    | ImageNet |  73.97   |  91.33   | 10 clips x 3 crop |  56.5  | 31.7M  | [config](/configs/recognition/i3d/i3d_imagenet-pretrained-r50-nl-gaussian_8xb8-32x2x1-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/i3d/i3d_imagenet-pretrained-r50-nl-gaussian_8xb8-32x2x1-100e_kinetics400-rgb/i3d_imagenet-pretrained-r50-nl-gaussian_8xb8-32x2x1-100e_kinetics400-rgb_20220812-0c5cbf5a.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/i3d/i3d_imagenet-pretrained-r50-nl-gaussian_8xb8-32x2x1-100e_kinetics400-rgb/i3d_imagenet-pretrained-r50-nl-gaussian_8xb8-32x2x1-100e_kinetics400-rgb.log) |

|         32x2x1          |  224x224   |  8   |           ResNet50            | ImageNet |  73.47   |  91.27   | 10 clips x 3 crop | 43.5G  | 28.0M  | [config](/configs/recognition/i3d/i3d_imagenet-pretrained-r50_8xb8-32x2x1-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/i3d/i3d_imagenet-pretrained-r50_8xb8-32x2x1-100e_kinetics400-rgb/i3d_imagenet-pretrained-r50_8xb8-32x2x1-100e_kinetics400-rgb_20220812-e213c223.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/i3d/i3d_imagenet-pretrained-r50_8xb8-32x2x1-100e_kinetics400-rgb/i3d_imagenet-pretrained-r50_8xb8-32x2x1-100e_kinetics400-rgb.log) |

|      dense-32x2x1       |  224x224   |  8   |           ResNet50            | ImageNet |  73.77   |  91.35   | 10 clips x 3 crop | 43.5G  | 28.0M  | [config](/configs/recognition/i3d/i3d_imagenet-pretrained-r50_8xb8-dense-32x2x1-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/i3d/i3d_imagenet-pretrained-r50_8xb8-dense-32x2x1-100e_kinetics400-rgb/i3d_imagenet-pretrained-r50_8xb8-dense-32x2x1-100e_kinetics400-rgb_20220812-9f46003f.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/i3d/i3d_imagenet-pretrained-r50_8xb8-dense-32x2x1-100e_kinetics400-rgb/i3d_imagenet-pretrained-r50_8xb8-dense-32x2x1-100e_kinetics400-rgb.log) |

|         32x2x1          |  224x224   |  8   |       ResNet50 (Heavy)        | ImageNet |  76.21   |  92.48   | 10 clips x 3 crop | 166.3G | 33.0M  | [config](/configs/recognition/i3d/i3d_imagenet-pretrained-r50-heavy_8xb8-32x2x1-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/i3d/i3d_imagenet-pretrained-r50-heavy_8xb8-32x2x1-100e_kinetics400-rgb/i3d_imagenet-pretrained-r50-heavy_8xb8-32x2x1-100e_kinetics400-rgb_20220812-ed501b31.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/i3d/i3d_imagenet-pretrained-r50-heavy_8xb8-32x2x1-100e_kinetics400-rgb/i3d_imagenet-pretrained-r50-heavy_8xb8-32x2x1-100e_kinetics400-rgb.log) |

## Train

## Train

```shell
```

Example: train I3D model on Kinetics-400 dataset in a deterministic option with periodic validation.

```shell
python tools/train.py configs/recognition/i3d/i3d_imagenet-pretrained-r50_8xb8-32x2x1-100e_kinetics400-rgb.py \
```

## Test

```shell
```

Example: test I3D model on Kinetics-400 dataset and dump the result to a pkl file.

```shell
python tools/test.py configs/recognition/i3d/i3d_imagenet-pretrained-r50_8xb8-32x2x1-100e_kinetics400-rgb.py \

```shell
```

```BibTeX
}
```

```BibTeX
@article{NonLocal2018,
  author =   {Xiaolong Wang and Ross Girshick and Abhinav Gupta and Kaiming He},
  title =    {Non-local Neural Networks},
  journal =  {CVPR},
  year =     {2018}
}
```

# How to design your own data transforms

In this tutorial, we introduce the design of transforms pipeline in MMagic.

- [How to design your own data transforms](#how-to-design-your-own-data-transforms)
  - [Data pipelines in MMagic](#data-pipelines-in-mmagic)
    - [A simple example of data transform](#a-simple-example-of-data-transform)
    - [An example of BasicVSR](#an-example-of-basicvsr)
    - [An example of Pix2Pix](#an-example-of-pix2pix)
  - [Supported transforms in MMagic](#supported-transforms-in-mmagic)
    - [Data loading](#data-loading)
    - [Pre-processing](#pre-processing)
    - [Formatting](#formatting)
  - [Extend and use custom pipelines](#extend-and-use-custom-pipelines)
    - [A simple example of MyTransform](#a-simple-example-of-mytransform)
    - [An example of flipping](#an-example-of-flipping)

## Data pipelines in MMagic

- [A simple example of MyTransform](#a-simple-example-of-mytransform)

Following typical conventions, we use `Dataset` and `DataLoader` for data loading with multiple workers. `Dataset` returns a dict of data items corresponding the arguments of models' forward method.

The data preparation pipeline and the dataset is decomposed. Usually a dataset defines how to process the annotations and a data pipeline defines all the steps to prepare a data dict.

A pipeline consists of a sequence of operations. Each operation takes a dict as input and also output a dict for the next transform.

The operations are categorized into data loading, pre-processing, and formatting

In MMagic, all data transformations are inherited from `BaseTransform`.
The input and output types of transformations are both dict.

### A simple example of data transform

```python

```python
>>> from mmagic.transforms import LoadPairedImageFromFile
>>> transforms = LoadPairedImageFromFile(
>>>     key='pair',
>>>     domain_a='horse',
>>>     domain_b='zebra',
>>>     flag='color'),
>>> data_dict = {'pair_path': './data/pix2pix/facades/train/1.png'}
>>> data_dict = transforms(data_dict)
>>> print(data_dict.keys())
dict_keys(['pair_path', 'pair', 'pair_ori_shape', 'img_mask', 'img_photo', 'img_mask_path', 'img_photo_path', 'img_mask_ori_shape', 'img_photo_ori_shape'])
```

Generally, the last step of the transforms pipeline must be `PackInputs`.
`PackInputs` will pack the processed data into a dict containing two fields: `inputs` and `data_samples`.

```

`inputs` is the variable you want to use as the model's input, which can be the type of `torch.Tensor`, dict of `torch.Tensor`, or any type you want.
`data_samples` is a list of `DataSample`. Each `DataSample` contains groundtruth and necessary information for corresponding input.

### An example of BasicVSR

Here is a pipeline example for BasicVSR.

```python
    dict(type='LoadImageFromFile', key='img', channel_order='rgb'),
    dict(type='LoadImageFromFile', key='gt', channel_order='rgb'),
    dict(type='SetValues', dictionary=dict(scale=scale)),
    dict(type='PairedRandomCrop', gt_patch_size=256),
    dict(
        type='Flip',
        keys=['img', 'gt'],
        flip_ratio=0.5,
        direction='horizontal'),
    dict(

dict(type='PairedRandomCrop', gt_patch_size=256),
    dict(
    dict(
        type='Flip', keys=['img', 'gt'], flip_ratio=0.5, direction='vertical'),
    dict(type='RandomTransposeHW', keys=['img', 'gt'], transpose_ratio=0.5),
    dict(type='MirrorSequence', keys=['img', 'gt']),
    dict(type='PackInputs')
]

    dict(type='GenerateSegmentIndices', interval_list=[1]),
]

    dict(type='MirrorSequence', keys=['img']),
]
```

dict(type='LoadImageFromFile', key='gt', channel_order='rgb'),
]
```

For each operation, we list the related dict fields that are added/updated/removed, the dict fields marked by '\*' are optional.

### An example of Pix2Pix

Here is a pipeline example for Pix2Pix training on aerial2maps dataset.

```python
source_domain = 'aerial'
target_domain = 'map'

    dict(
        type='LoadPairedImageFromFile',
        io_backend='disk',
        key='pair',
        domain_a=domain_a,
        domain_b=domain_b,
        flag='color'),
    dict(
        type='TransformBroadcaster',
        mapping={'img': [f'img_{domain_a}', f'img_{domain_b}']},
        auto_remap=True,
        share_random_params=True,
            dict(
                type='mmagic.Resize', scale=(286, 286),
                interpolation='bicubic'),
            dict(type='mmagic.FixedCrop', crop_size=(256, 256))

transforms=[
            dict(
        ]),
    dict(
        keys=[f'img_{domain_a}', f'img_{domain_b}'],
    dict(
        type='PackInputs',
        keys=[f'img_{domain_a}', f'img_{domain_b}', 'pair'])
```

## Supported transforms in MMagic

### Data loading

   <thead>
      <tr>
         <th style="text-align:center">Transform</th>
         <th style="text-align:center">Modification of Results' keys</th>
      </tr>
   </thead>
   <tbody>
      <tr>
         <td>
            <code>LoadImageFromFile</code>
         </td>
         <td>
            - add: img, img_path, img_ori_shape, \*ori_img
         </td>
      </tr>
      <tr>
         <td>
            <code>RandomLoadResizeBg</code>
         </td>
         <td>
            - add: bg

</td>
      </tr>
      <tr>
         <td>
         </td>
         <td>
            - add: bg
         </td>
      </tr>
      <tr>
         <td>
            <code>LoadMask</code>
         </td>
         <td>
            - add: mask
         </td>
      </tr>
      <tr>
         <td>
            <code>GetSpatialDiscountMask</code>
         </td>
         <td>
            - add: discount_mask
         </td>
      </tr>
   </tbody>
</table>

### Pre-processing

   <thead>
      <tr>
      </tr>
   </thead>
   <tbody>
      <tr>
         <td>
            <code>Resize</code>
         </td >
         <td>
            - add: scale_factor, keep_ratio, interpolation, backend
            - update: specified by <code>keys</code>
         </td >
      </tr>
      <tr>
         <td>

<td>
         </td >
      </tr>
      <tr>
         <td>
            <code>MATLABLikeResize</code>
         </td >
         <td>
            - add: scale, output_shape
         </td >
      </tr>
      <tr>
         <td>
            <code>RandomRotation</code>
         </td >
         <td>
            - add: degrees
         <td>
      </tr>
      <tr>
         <td>
            <code>Flip</code>
         </td >
         <td>
            - add: flip, flip_direction
         </td >
      </tr>
      <tr>
         <td>
            <code>RandomAffine</code>
         </td >
         <td>
         </td >
      </tr>
      <tr>
         <td>

<tr>
         <td>
         </td >
         <td>
         </td >
      </tr>
      <tr>
         <td>
            <code>RandomJitter</code>
         </td >
         <td>
            - update: fg (img)
         </td >
      </tr>
      <tr>
         <td>
            <code>ColorJitter</code>
         </td >
         <td>
         </td >
      </tr>
      <tr>
         <td>
            <code>BinarizeImage</code>
         </td >
         <td>
         </td >
      </tr>
      <tr>
         <td>
            <code>RandomMaskDilation</code>
         </td >
         <td>
            - add: img_dilate_kernel_size
         <td>
      </tr>
      <tr>
         <td>
            <code>RandomTransposeHW</code>
         </td >
         <td>
            - add: transpose
      </tr>
      <tr>

<td>
      </tr>
      <tr>
         <td>
         </td >
         <td>
      </tr>
      <tr>
         <td>
            <code>RandomDownSampling</code>
         </td >
         <td>
            - update: scale, gt (img), lq (img)
         </td >
      </tr>
      <tr>
         <td>
            <code>RandomBlur</code>
         </td >
         <td>
         </td >
      </tr>
      <tr>
         <td>
            <code>RandomResize</code>
         </td >
         <td>
         </td >
      </tr>
      <tr>
         <td>
            <code>RandomNoise</code>
         </td >
         <td>
         </td >
      </tr>
      <tr>
         <td>
            <code>RandomJPEGCompression</code>
         </td >
         <td>

<td>
         </td >
      </tr>
      <tr>
         <td>
         </td >
         <td>
         </td >
      </tr>
      <tr>
         <td>
            <code>RandomVideoCompression</code>
         </td >
         <td>
         </td >
      </tr>
      <tr>
         <td>
            <code>DegradationsWithShuffle</code>
         </td >
         <td>
         </td >
      </tr>
      <tr>
         <td>
            <code>GenerateFrameIndices</code>
         </td >
         <td>
            - update: img_path (gt_path, lq_path)
         </td >
      </tr>
      <tr>
         <td>
            <code>GenerateFrameIndiceswithPadding</code>
         </td >
         <td>

</td >
      </tr>
      <tr>
         <td>
         </td >
         <td>
         </td >
      </tr>
      <tr>
         <td>
            <code>TemporalReverse</code>
         </td >
         <td>
            - add: reverse
         </td >
      </tr>
      <tr>
         <td>
            <code>GenerateSegmentIndices</code>
         </td >
         <td>
            - add: interval
         </td >
      </tr>
      <tr>
         <td>
            <code>MirrorSequence</code>
         </td >
         <td>
         </td >
      </tr>
      <tr>
         <td>
            <code>CopyValues</code>
         </td >
         <td>
            - add: specified by <code>dst_key</code>
         </td >
      </tr>
      <tr>

</tr>
      <tr>
         <td>
         </td >
         <td>
         </td >
      </tr>
      <tr>
         <td>
            <code>UnsharpMasking</code>
         </td >
         <td>
            - add: img_unsharp
         </td >
      </tr>
      <tr>
         <td>
            <code>Crop</code>
         </td >
         <td>
            - add: img_crop_bbox, crop_size
         </td >
      </tr>
      <tr>
         <td>
            <code>RandomResizedCrop</code>
         </td >
         <td>
            - add: img_crop_bbox
         </td >
      </tr>
      <tr>
         <td>
            <code>FixedCrop</code>
         </td >
         <td>
            - add: crop_size, crop_pos
         </td >
      </tr>
      <tr>
         <td>

</td >
         <td>
         </td >
      </tr>
      <tr>
         <td>
            <code>PairedRandomCrop</code>
         </td >
         <td>
            - update: gt (img), lq (img)
         </td >
      </tr>
      <tr>
         <td>
            <code>CropAroundCenter</code>
         </td >
         <td>
            - add: crop_bbox
            - update: fg (img), alpha (img), trimap (img), bg (img)
         </td >
      </tr>
      <tr>
         <td>
            <code>CropAroundUnknown</code>
         </td >
         <td>
         </td >
      </tr>
      <tr>
         <td>
            <code>CropAroundFg</code>
         </td >
         <td>
         </td >
      </tr>
      <tr>
         <td>

</td >
         <td>
         </td >
      </tr>
      <tr>
         <td>
            <code>ModCrop</code>
         </td >
         <td>
            - update: gt (img)
         </td >
      </tr>
      <tr>
         <td>
            <code>CropLike</code>
         </td >
         <td>
            - update: specified by <code>target_key</code>
         </td >
      </tr>
      <tr>
         <td>
            <code>GetMaskedImage</code>
         </td >
         <td>
            - add: masked_img
         </td >
      </tr>
      <tr>
         <td>
            <code>GenerateFacialHeatmap</code>
         </td >
         <td>
            - add: heatmap
         </td >
      </tr>
      <tr>
         <td>
            <code>GenerateCoordinateAndCell</code>
         </td >
         <td>
            - add: coord, cell
         </td >
      </tr>
      <tr>
         <td>

<code>GenerateCoordinateAndCell</code>
         </td >
         <td>
         </td >
      </tr>
      <tr>
         <td>
            <code>Normalize</code>
         </td >
         <td>
            - add: img_norm_cfg
         </td >
      </tr>
      <tr>
         <td>
            <code>RescaleToZeroOne</code>
         </td >
         <td>
         </td >
      </tr>
   </tbody>
</table>

   <thead>
      <tr>
      </tr>
   </thead>
   <tbody>
      <tr>
         <td>
            <code>ToTensor</code>
         </td>
         <td>
            update: specified by <code>keys</code>.
         </td>
      </tr>
      <tr>
         <td>

<tr>
         <td>
         </td>
         <td>
         </td>
      </tr>
      <tr>
         <td>
            <code>FormatTrimap</code>
         </td>
         <td>
            - update: trimap
         </td>
      </tr>
      <tr>
         <td>
            <code>PackInputs</code>
         </td>
         <td>
            - add: inputs, data_sample
            - remove: all other keys
         </td>
      </tr>
   </tbody>
</table>

## Extend and use custom pipelines

### A simple example of MyTransform

1. Write a new pipeline in a file, e.g., in `my_pipeline.py`. It takes a dict as input and returns a dict.

```python
from mmcv.transforms import BaseTransform
from mmagic.registry import TRANSFORMS

class MyTransform(BaseTransform):
    """Add your transform

    Args:
        p (float): Probability of shifts. Default 0.5.
    """

    Args:
    """

    def __init__(self, p=0.5):
        self.p = p

        if random.random() > self.p:
            results['dummy'] = True

    def __repr__(self):

        repr_str = self.__class__.__name__
        repr_str += (f'(p={self.p})')

        return repr_str
```

Make sure the import is relative to where your train script is located.

```python
    ...
    dict(type='MyTransform', p=0.2),
    ...
]
```

### An example of flipping

```python

```python

```

```python

```

```python
    ...
    ...
]
```

```

```python
    ...
    ...
]
```

Note that if you want to use `MyFlip` in config, you must ensure the file containing `MyFlip` is imported during the program run.

# 开始你的第一步（待翻译）

```{note}
```

```shell
```

```shell
```

# Posetrack: A benchmark for human pose estimation and tracking

<details>

```bibtex
}
```

## onnxruntime 自定义算子

- [ONNX Runtime Ops](#onnx-runtime-ops)
- [NMSRotated](#nmsrotated)
  - [Description](#description-2)
  - [Parameters](#parameters-2)
  - [Inputs](#inputs-2)
  - [Outputs](#outputs-2)
  - [Type Constraints](#type-constraints-2)
  - [RoIAlignRotated](#roialignrotated)

<dl>

<dl>
<dt><tt>grid</tt>: T</dt>
</dl>

<dl>
</dl>

Perform Modulated Deformable Convolution on input feature, read [Deformable ConvNets v2: More Deformable, Better Results](https://arxiv.org/abs/1811.11168?from=timeline) for detail.

| Type           | Parameter           | Description                                                                           |

| -------------- | ------------------- | ------------------------------------------------------------------------------------- |
| `list of ints` | `stride`            | The stride of the convolving kernel. (sH, sW)                                         |
| `list of ints` | `padding`           | Paddings on both sides of the input. (padH, padW)                                     |
| `list of ints` | `dilation`          | The spacing between kernel elements. (dH, dW)                                         |
| `int`          | `deformable_groups` | Groups of deformable offset.                                                          |
| `int`          | `groups`            | Split input into groups. `input_channel` should be divisible by the number of groups. |

<dl>

<dl>

<dd>Input bias; 1-D tensor of shape (output_channel).</dd>
</dl>

<dl>
</dl>

### NMSRotated

Non Max Suppression for rotated bboxes.

| Type    | Parameter       | Description                |
| ------- | --------------- | -------------------------- |
| `float` | `iou_threshold` | The IoU threshold for NMS. |

<dl>
<dd>Input feature; 2-D tensor of shape (N, 5), where N is the number of rotated bboxes, .</dd>
<dd>Input offset; 1-D tensor of shape (N, ), where N is the number of rotated bboxes.</dd>
</dl>

<dl>

</dl>

<dl>
<dd>Output feature; 1-D tensor of shape (K, ), where K is the number of keep bboxes.</dd>
</dl>

### RoIAlignRotated

Perform RoIAlignRotated on output feature, used in bbox_head of most two-stage rotated object detectors.

| Type    | Parameter        | Description                                                                                                                               |
| ------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |

| `int`   | `output_height`  | height of output roi                                                                                                                      |
| `int`   | `output_width`   | width of output roi                                                                                                                       |
| `float` | `spatial_scale`  | used to scale the input boxes                                                                                                             |
| `int`   | `sampling_ratio` | number of input samples to take for each output sample. `0` means to take samples densely for current models.                             |

| `int`   | `aligned`        | If `aligned=0`, use the legacy implementation in MMDetection. Else, align the results more perfectly.                                     |
| `int`   | `clockwise`      | If True, the angle in each proposal follows a clockwise fashion in image space, otherwise, the angle is counterclockwise. Default: False. |

<dl>
<dt><tt>rois</tt>: T</dt>
<dd>RoIs (Regions of Interest) to pool over; 2-D tensor of shape (num_rois, 6) given as [[batch_index, cx, cy, w, h, theta], ...]. The RoIs' coordinates are the coordinate system of input.</dd>
</dl>

<dl>
<dt><tt>feat</tt>: T</dt>

</dl>

<dl>
<dd>RoI pooled output, 4-D tensor of shape (num_rois, C, output_height, output_width). The r-th batch element feat[r-1] is a pooled feature map corresponding to the r-th RoI RoIs[r-1].<dd>
</dl>

- T:tensor(float32)

# Evaluator \[Coming Soon!\]

}
```

<details>

```bibtex
}
```

<details>

```bibtex

| [deeppose_mobilenetv2_rle_pretrained](/configs/body_2d_keypoint/topdown_regression/coco/td-reg_mobilenetv2_rle-pretrained-8xb64-210e_coco-256x192.py) |  256x192   | 0.593 |      0.836      |      0.660      | 0.644 |      0.877      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_regression/coco/td-reg_mobilenetv2_rle-pretrained-8xb64-210e_coco-256x192-39b73bd5_20220922.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_regression/coco/td-reg_mobilenetv2_rle-pretrained-8xb64-210e_coco-256x192-39b73bd5_20220922.log.json) |

# Test on TVM

| Model             | Codebase       |                                      Model config                                       |
| :---------------- | :------------- | :-------------------------------------------------------------------------------------: |
| RetinaNet         | MMDetection    |     [config](https://github.com/open-mmlab/mmdetection/tree/main/configs/retinanet)     |
| Faster R-CNN      | MMDetection    |    [config](https://github.com/open-mmlab/mmdetection/tree/main/configs/faster_rcnn)    |
| YOLOv3            | MMDetection    |       [config](https://github.com/open-mmlab/mmdetection/tree/main/configs/yolo)        |
| YOLOX             | MMDetection    |       [config](https://github.com/open-mmlab/mmdetection/tree/main/configs/yolox)       |
| Mask R-CNN        | MMDetection    |     [config](https://github.com/open-mmlab/mmdetection/tree/main/configs/mask_rcnn)     |

| SSD               | MMDetection    |        [config](https://github.com/open-mmlab/mmdetection/tree/main/configs/ssd)        |
| ResNet            | MMPretrain     |       [config](https://github.com/open-mmlab/mmpretrain/tree/main/configs/resnet)       |
| ResNeXt           | MMPretrain     |      [config](https://github.com/open-mmlab/mmpretrain/tree/main/configs/resnext)       |
| SE-ResNet         | MMPretrain     |      [config](https://github.com/open-mmlab/mmpretrain/tree/main/configs/seresnet)      |
| MobileNetV2       | MMPretrain     |    [config](https://github.com/open-mmlab/mmpretrain/tree/main/configs/mobilenet_v2)    |
| ShuffleNetV1      | MMPretrain     |   [config](https://github.com/open-mmlab/mmpretrain/tree/main/configs/shufflenet_v1)    |

| ShuffleNetV2      | MMPretrain     |   [config](https://github.com/open-mmlab/mmpretrain/tree/main/configs/shufflenet_v2)    |
| VisionTransformer | MMPretrain     | [config](https://github.com/open-mmlab/mmpretrain/tree/main/configs/vision_transformer) |
| FCN               | MMSegmentation |      [config](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/fcn)       |
| PSPNet            | MMSegmentation |     [config](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/pspnet)     |
| DeepLabV3         | MMSegmentation |   [config](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/deeplabv3)    |
| DeepLabV3+        | MMSegmentation | [config](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/deeplabv3plus)  |

| UNet              | MMSegmentation |      [config](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/unet)      |

The table above list the models that we have tested. Models not listed on the table might still be able to converted. Please have a try.

## Test

- Ubuntu 20.04
- tvm 0.9.0

|                                                        mmpretrain                                                         | metric | PyTorch |  TVM  |
| :-----------------------------------------------------------------------------------------------------------------------: | :----: | :-----: | :---: |
|           [ResNet-18](https://github.com/open-mmlab/mmpretrain/blob/main/configs/resnet/resnet18_8xb32_in1k.py)           | top-1  |  69.90  | 69.90 |

|      [ResNeXt-50](https://github.com/open-mmlab/mmpretrain/blob/main/configs/resnext/resnext50-32x4d_8xb32_in1k.py)       | top-1  |  77.90  | 77.90 |
| [ShuffleNet V2](https://github.com/open-mmlab/mmpretrain/blob/main/configs/shufflenet_v2/shufflenet-v2-1x_16xb64_in1k.py) | top-1  |  69.55  | 69.55 |
|    [MobileNet V2](https://github.com/open-mmlab/mmpretrain/tree/main/configs/mobilenet_v2/mobilenet-v2_8xb32_in1k.py)     | top-1  |  71.86  | 71.86 |

<!-- |     [Vision Transformer](https://github.com/open-mmlab/mmpretrain/blob/main/configs/vision_transformer/vit-base-p16_ft-64xb64_in1k-384.py)     | top-1  |  85.43  | 84.01 | -->

|                                       mmdet(\*)                                       | metric | PyTorch | TVM  |

| :-----------------------------------------------------------------------------------: | :----: | :-----: | :--: |
| [SSD](https://github.com/open-mmlab/mmdetection/tree/main/configs/ssd/ssd300_coco.py) | box AP |  25.5   | 25.5 |

\*: We only test model on ssd since dynamic shape is not supported for now.

|                                                             mmseg                                                             | metric | PyTorch |  TVM  |
| :---------------------------------------------------------------------------------------------------------------------------: | :----: | :-----: | :---: |
|     [FCN](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r50-d8_4xb2-40k_cityscapes-512x1024.py)      |  mIoU  |  72.25  | 72.36 |

| [PSPNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet/pspnet_r50-d8_4xb2-80k_cityscapes-512x1024.py) |  mIoU  |  78.55  | 77.90 |

# Microsoft coco: Common objects in context

<details>

```bibtex
}
```

### param is too old, please regenerate

Your model file is being the old format converted by an old caffe2ncnn tool.

Checkout the latest ncnn code, build it and regenerate param and model binary files, and that should work.

Make sure that your param file starts with the magic number 7767517.

you may find more info on [use-ncnn-with-alexnet](use-ncnn-with-alexnet)

If the original model is missing, you can try to manually fix the layer specific parameters in param file

1. **Softmax** append `1=1`

before
```
Softmax xxx 1 1 in out ...
```
after
```
Softmax xxx 1 1 in out ... 1=1
```

2. **Reduction** minus all axes value by 1 (except the leading array count) and append `5=1`

before
```
Reduction xxx 1 1 in out ... -23303=2,2,3 ...
```
after
```
Reduction xxx 1 1 in out ... -23303=2,1,2 ... 5=1
```

### find_blob_index_by_name XYZ failed

That means ncnn couldn't find the XYZ blob in the network. 

You shall call Extractor::input()/extract() by blob name instead of layer name.

```

For models loaded from binary param file or external memory, you shall call Extractor::input()/extract() by the enum defined in xxx.id.h because all the visible string literals have been stripped in binary form.

This error usually happens when the input layer is not properly converted.

You shall upgrade caffe prototxt/caffemodel before converting it to ncnn. Following snippet type shall be ok. 

```
layer {
  name: "data"
  type: "Input"
  top: "data"
  input_param { shape: { dim: 1 dim: 3 dim: 227 dim: 227 } }
}
```

you may find more info on [use-ncnn-with-alexnet](use-ncnn-with-alexnet).

### layer XYZ not exists or registered

Your network contains some operations that are not implemented in ncnn.

}
```

You may implement them as custom layer followed in [how-to-implement-custom-layer-step-by-step](how-to-implement-custom-layer-step-by-step).

Or you could simply register them as no-op if you are sure those operations make no sense.

```cpp

```

### fopen XYZ.param/XYZ.bin failed

File not found or not readable. Make sure that XYZ.param/XYZ.bin is accessible.

### network graph not ready

You shall call Net::load_param() first, then Net::load_model().

This error may also happens when Net::load_param() failed, but not properly handled.

For more information about the ncnn model load api, see [ncnn-load-model](ncnn-load-model)

### memory not 32-bit aligned at XYZ

The pointer passed to Net::load_param() or Net::load_model() is not 32bit aligned.

In practice, the head pointer of std::vector<unsigned char> is not guaranteed to be 32bit aligned.

you can store your binary buffer in ncnn::Mat structure, its internal memory is aligned.

### undefined reference to '__kmpc_XYZ_XYZ'

use clang for building android shared library

comment the following line in your Application.mk
```
NDK_TOOLCHAIN_VERSION := 4.9
```

### crash on android with '__kmp_abort_process'

This usually happens if you bundle multiple shared library with openmp linked

It is actually an issue of the android ndk https://github.com/android/ndk/issues/1028

On old android ndk, modify the link flags as

```
-Wl,-Bstatic -lomp -Wl,-Bdynamic
```

For recent ndk >= 21

```
-fstatic-openmp
```

### dlopen failed: library "libomp.so" not found

modify the link flags as

```
-fstatic-openmp -fopenmp
```

### crash when freeing a ncnn dynamic library(*.dll/*.so) built with openMP

for optimal performance, the openmp threadpool spin waits for about a second prior to shutting down in case more work becomes available. 

If you unload a dynamic library that's in the process of spin-waiting, it will crash in the manner you see (most of the time).

Just set OMP_WAIT_POLICY=passive in your environment, before calling loadlibrary. or Just wait a few seconds before calling freelibrary.

You can also use the following method to set environment variables in your code:

for msvc++:

```
SetEnvironmentVariable(_T("OMP_WAIT_POLICY"), _T("passive"));
```

for g++:

```
setenv("OMP_WAIT_POLICY", "passive", 1)
```

reference: https://stackoverflow.com/questions/34439956/vc-crash-when-freeing-a-dll-built-with-openmp

<details>

```bibtex
}
```

<details>

```bibtex

```bibtex
}
```

<details>

```bibtex
}
```

Results on CrowdPose test without multi-scale test

journal={arXiv preprint arXiv:1812.00324},
}
```

| [HRNet-w32](/configs/body_2d_keypoint/dekr/crowdpose/dekr_hrnet-w32_8xb10-300e_crowdpose-512x512.py) |  512x512   | 0.663 |      0.857      |      0.714      | 0.740  | 0.671  | 0.576  | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/dekr/crowdpose/dekr_hrnet-w32_8xb10-300e_crowdpose-512x512_147bae97-20221228.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/dekr/crowdpose/dekr_hrnet-w32_8xb10-300e_crowdpose-512x512_20221228.json) |
| [HRNet-w48](/configs/body_2d_keypoint/dekr/crowdpose/dekr_hrnet-w48_8xb5-300e_crowdpose-640x640.py) |  640x640   | 0.679 |      0.869      |      0.731      | 0.753  | 0.688  | 0.593  | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/dekr/crowdpose/dekr_hrnet-w48_8xb5-300e_crowdpose-640x640_4ea6031e-20230128.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/dekr/crowdpose/dekr_hrnet-w48_8xb5-300e_crowdpose-640x640_20230128.json) |

# Verify Dataset

In MMPretrain, we also provide a tool `tools/misc/verify_dataset.py` to check whether there exists **broken pictures** in the given dataset.

## Introduce the tool

```shell
python tools/print_config.py \
    [--out-path ${OUT-PATH}] \
    [--phase ${PHASE}] \
    [--num-process ${NUM-PROCESS}]
```

- `--out-path` : The path to save the verification result, if not set, defaults to 'brokenfiles.log'.
- `--phase` :  Phase of dataset to verify, accept "train" "test" and "val", if not set, defaults to "train".
- `--num-process` : number of process to use, if not set, defaults to 1.

## Example

```shell

## Example

```shell
python tools/misc/verify_dataset.py configs/t2t_vit/t2t-vit-t-14_8xb64_in1k.py --out-path broken_imgs.log --phase val --num-process 8
```

# 20分钟了解 MMAction2 框架设计

在本教程中，我们将通过一个视频动作识别的手把手教程来演示 `MMACTION2 1.0` 的整体架构。

本教程的目录如下:

- [20分钟了解 MMAction2 框架设计](#20分钟了解-mmaction2-框架设计)
  - [步骤0：准备数据](#步骤0准备数据)
  - [步骤1：构建一个数据流水线](#步骤1构建一个数据流水线)
  - [步骤2：构建一个数据集和数据加载器](#步骤2构建一个数据集和数据加载器)
  - [步骤3：构建一个识别器](#步骤3构建一个识别器)
  - [步骤4：构建一个评估指标](#步骤4构建一个评估指标)
  - [步骤5：使用本地 PyTorch 训练和测试](#步骤5使用本地-pytorch-训练和测试)
  - [步骤6：使用 MMEngine 训练和测试（推荐）](#步骤6使用-mmengine-训练和测试推荐)

首先，我们需要初始化注册表的 `scope` ，以确保每个模块都在 `mmaction` 范围下注册。有关注册表的更多详细信息，请参考[ MMEngine 教程](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/registry.html) 。

```python

```

## 步骤0：准备数据

请下载我们准备的[精简版 kinetics400](https://download.openmmlab.com/mmaction/kinetics400_tiny.zip) 数据集，并将其提取到 `$MMACTION2/data` 目录。

解压后的目录结构应如下所示:

```
mmaction2
├── data

```
mmaction2
├── data
│   │    │   ├── ...
│   │    └── val
│   │       ├── ...
```

以下是标注文件 `kinetics_tiny_train_video.txt` 中的一些示例:

```
```

文件中的每一行表示每一个视频的标注，其中第一项表示视频文件名(如 `D32_1gwq35E.mp4` )，第二项表示相应的标签(如 `D32_1gwq35E.mp4` 的标签是 `0` )。在这个数据集中，只有 `两个` 类别。

## 步骤1：构建一个数据流水线

```

为了实现 `解码`、`采样`、`调整大小`、`裁剪`、`格式化` 和 `打包` 视频数据和相应的标签，我们需要设计一个数据流水线来处理这些过程。具体来说，我们设计了7个 `Transform` 类来构建这个视频处理流水线。注意，OpenMMLab 中的所有`Transform` 类都必须继承自 `mmcv` 中的 `BaseTransform` 类，实现抽象方法 `transform`，并注册到 `TRANSFORMS` 注册表。有关数据转换的更多详细信息，请参阅[ MMEngine 教程](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/data_transform.html) 。

```python

container = decord.VideoReader(results['filename'])

            # 使测试期间的采样具有确定性

inds = bids + offset

self.r_size = (np.inf, r_size)

imgs = [img[y1:y2, x1:x2] for img in results['imgs']]

self.meta_keys = meta_keys

```

下面，我们提供了一个代码片段(使用标注文件中的 `D32_1gwq35E.mp4 0` )来演示如何使用数据流水线。

```python

]

dict(type='VideoCrop', c_size=224),
]

# 获取输入的信息

# 获取输入的标签
```

```
```

## 步骤2：构建一个数据集和数据加载器

```

```
```

OpenMMLab中的所有 `Dataset` 类都必须继承自 `mmengine` 中的 `BaseDataset` 类。我们可以通过覆盖 `load_data_list` 方法来定制注释加载过程。此外，我们可以通过覆盖 `get_data_info` 方法，向 `results` 字典添加更多字段，它将作为输入传给 `pipeline` 。有关 `BaseDataset` 类的更多详细信息，请参阅[ MMEngine 教程](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/basedataset.html) 。

```python

test_mode=False, modality='RGB', **kwargs):
                                           **kwargs)

```

```

接下来，我们将演示如何使用 dataset 和 dataloader 来索引数据。我们将使用 `Runner.build_dataloader` 方法来构造 dataloader。有关 dataloader 的更多详细信息，请参阅[ MMEngine 教程](https://mmengine.readthedocs.io/en/latest/tutorials/dataset.html#details-on-dataloader) 。

```python

]

dict(type='VideoSample', clip_len=16, num_clips=5, test_mode=True),
]

# 获取输入的信息

# 获取输入的信息

# 获取输入的标签

```

终端输出应该与[步骤1：构建一个数据流水线](#步骤1：构建一个数据流水线)中的输出相同。

## 步骤3：构建一个识别器

接下来，我们将构建 `recognizer`，它主要由三部分组成：用于批处理和规范化数据的 `data preprocessor`，用于特征提取的 `backbone` 和用于分类的 `cls_head` 。

`data_preprocessor` 的实现如下:

```python

            'mean',
            False)
            'std',

self.register_buffer(
            'mean',
            False)
            'std',
            False)

        batch_inputs = stack_batch(inputs)  # 批处理
        batch_inputs = (batch_inputs - self.mean) / self.std  # 归一化
```

以下是 data_preprocessor 的用法：将从[步骤2：构建一个数据集和数据加载器](#步骤2：构建一个数据集和数据加载器)中获得的 `batched_packed_results` 提供给 `data_preprocessor` 进行批处理和归一化。

```python

```

```
```

`backbone`、`cls_head` 和 `recognizer` 的实现如下:

```python

dict(type='Constant', layer='BatchNorm3d', val=1, bias=0)]

features = self.conv(self.maxpool(self.conv1(imgs)))

        else:

cls_scores = self.average_clip(cls_scores, num_views)

elif self.average_clips == 'score':

predictions = self.cls_head.predict(feats, data_samples)

        else:
```

`init_cfg` 用于模型权重初始化。有关模型权重初始化的更多信息，请参阅[ MMEngine 教程](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/initialize.html) 。上述模块的用法如下:

```python

cls_head=dict(

# 训练

# 验证
```

```shell

```

```shell

```

## 步骤4：构建一个评估指标

```

请注意，`OpenMMLab` 中的所有 `Metric` 类都必须继承自 `mmengine` 中的 `BaseMetric` 类，并实现抽象方法 `process` 和`compute_metrics`。有关评估的更多信息，请参阅[ MMEngine 教程](https://mmengine.readthedocs.io/en/latest/tutorials/evaluation.html) 。

```python

self.topk = topk

```

```python

return eval_results
```

```python

```

```shell
```

## 步骤5：使用本地 PyTorch 训练和测试

```python

    losses = []

```

## 步骤6：使用 MMEngine 训练和测试（推荐）

```

关于训练和测试的更多细节，你可以参考[ MMAction2 教程](https://mmaction2.readthedocs.io/en/latest/user_guides/train_test.html) 。有关 `Runner` 的更多信息，请参阅[ MMEngine 教程](https://mmengine.readthedocs.io/en/latest/tutorials/runner.html) 。

```python

```

Apart from `deploy.py`, there are other useful tools under the `tools/` directory.

This tool can be used to convert PyTorch model from OpenMMLab to ONNX.

### Usage

```bash
```

## extract

ONNX model with `Mark` nodes in it can be partitioned into multiple subgraphs. This tool can be used to extract the subgraph from the ONNX model.

### Usage

```bash
```

```

### Note

To support the model partition, you need to add Mark nodes in the ONNX model. The Mark node comes from the `@mark` decorator.
For example, if we have marked the `multiclass_nms` as below, we can set `end=multiclass_nms:input` to extract the subgraph before NMS.

```python

```python
```

This tool helps to convert an `ONNX` model to an `PPLNN` model.

### Usage

```bash
```

This tool can be used to convert ONNX to TensorRT engine.

### Usage

```bash
    --calib-file /path/to/file
```

This tool helps to convert an `ONNX` model to an `ncnn` model.

### Usage

```bash
```

This tool helps to test latency of models with PyTorch, TensorRT and other backends. Note that the pre- and post-processing is excluded when computing inference latency.

### Usage

```bash
```

### Example:

```shell

```shell
```

And the output look like this:

```text

```

This tool can be used to generate supported-backends markdown table.

### Usage

```shell
```

- `yml_file:` input yml config path
- `output:`  output markdown file path
- `--backends:` output backends list. If not specified, it will be set 'onnxruntime' 'tensorrt' 'torchscript' 'pplnn' 'openvino' 'ncnn'.

Generate backends markdown table from mmocr.yml

```shell
python tools/generate_md_table.py tests/regression/mmocr.yml tests/regression/mmocr.md --backends  onnxruntime tensorrt torchscript pplnn openvino ncnn
```

```shell
```

# 自定义优化器

在本教程中，我们将介绍一些构建优化器和学习率策略的方法，以用于你的任务。

- [自定义优化器](#自定义优化器)
  - [使用 optim_wrapper 构建优化器](#使用-optim_wrapper-构建优化器)
    - [使用 PyTorch 支持的优化器](#使用-pytorch-支持的优化器)
    - [参数化精细配置](#参数化精细配置)
    - [梯度裁剪](#梯度裁剪)
    - [梯度累积](#梯度累积)
  - [自定义参数策略](#自定义参数策略)
    - [自定义学习率策略](#自定义学习率策略)
    - [自定义动量策略](#自定义动量策略)
  - [添加新的优化器或构造器](#添加新的优化器或构造器)
    - [添加新的优化器](#添加新的优化器)
      - [1. 实现一个新的优化器](#1-实现一个新的优化器)
      - [2. 导入优化器](#2-导入优化器)
      - [3. 在配置文件中指定优化器](#3-在配置文件中指定优化器)
    - [添加新的优化器构造器](#添加新的优化器构造器)

## 使用 optim_wrapper 构建优化器

我们使用 `optim_wrapper` 字段来配置优化策略，其中包括选择优化器、参数逐个配置、梯度裁剪和梯度累积。一个简单的示例可以是：

```python
)
```

在上面的示例中，我们构建了一个学习率为 0.0003，权重衰减为 0.0001 的 SGD 优化器。

### 使用 PyTorch 支持的优化器

我们支持 PyTorch 实现的所有优化器。要使用不同的优化器，只需更改配置文件中的 `optimizer` 字段。例如，如果想使用 `torch.optim.Adam`，可以在配置文件中进行如下修改。

```python

```python
        lr=0.001,
)
```

首先，我们需要将 `type` 的值更改为 `torch.optim` 支持的期望优化器名称。然后，将该优化器的必要参数添加到 `optimizer` 字段中。上述配置将构建以下优化器：

```python
```

### 参数化精细配置

一些模型可能对优化有特定的参数设置，例如对于 BatchNorm 层不使用权重衰减，或者对不同网络层使用不同的学习率。为了对其进行细致配置，我们可以使用 `optim_wrapper` 中的 `paramwise_cfg` 参数。

- **为不同类型的参数设置不同的超参数倍数。**

  例如，我们可以在 `paramwise_cfg` 中设置 `norm_decay_mult=0.`，将归一化层的权重衰减设置为零。

  ```python
  ```

  还支持设置其他类型的参数，包括：

```python
  ```

  - `lr_mult`：所有参数的学习率乘数。
  - `decay_mult`：所有参数的权重衰减乘数。
  - `bias_lr_mult`：偏置项的学习率乘数（不包括归一化层的偏置项和可变形卷积层的偏移量）。默认为 1。
  - `bias_decay_mult`：偏置项的权重衰减乘数（不包括归一化层的偏置项和可变形卷积层的偏移量）。默认为 1。
  - `norm_decay_mult`：归一化层权重和偏置项的权重衰减乘数。默认为 1。
  - `dwconv_decay_mult`：深度卷积层的权重衰减乘数。默认为 1。
  - `bypass_duplicate`：是否跳过重复的参数。默认为 `False`。
  - `dcn_offset_lr_mult`：可变形卷积层的学习率乘数。默认为 1。

- **为特定参数设置不同的超参数倍数。**

  MMAction2 可以使用 `paramwise_cfg` 中的 `custom_keys` 来指定不同的参数使用不同的学习率或权重衰减。

  例如，要将 `backbone.layer0` 的所有学习率和权重衰减设置为 0，而保持 `backbone` 的其余部分与优化器相同，并将 `head` 的学习率设置为 0.001，可以使用以下配置：

  ```python

          }))
  ```

### 梯度裁剪

在训练过程中，损失函数可能接近悬崖区域，导致梯度爆炸。梯度裁剪有助于稳定训练过程。梯度裁剪的更多介绍可以在[这个页面](https://paperswithcode.com/method/gradient-clipping)找到。

目前，我们支持 `optim_wrapper` 中的 `clip_grad` 选项进行梯度裁剪，参考[PyTorch 文档](torch.nn.utils.clip_grad_norm_)。

以下是一个示例：

```python
    # norm_type: 使用的 p-范数的类型，这里 norm_type 为 2。
```

### 梯度累积

当计算资源有限时，批量大小只能设置为较小的值，这可能会影响模型的性能。可以使用梯度累积来解决这个问题。我们支持 `optim_wrapper` 中的 `accumulative_counts` 选项进行梯度累积。

以下是一个示例：

```python
```

表示在训练过程中，每 4 个迭代执行一次反向传播。上述示例等价于：

```python

```

```python
```

## 新增优化器或者优化器构造器

在训练中，优化参数（如学习率、动量等）通常不是固定的，而是随着迭代或周期的变化而变化。PyTorch 支持几种学习率策略，但对于复杂的策略可能不足够。在 MMAction2 中，我们提供 `param_scheduler` 来更好地控制不同参数的学习率策略。

### 配置学习率调整策略

调整学习率策略被广泛用于提高性能。我们支持大多数 PyTorch 学习率策略，包括 `ExponentialLR`、`LinearLR`、`StepLR`、`MultiStepLR` 等。

所有可用的学习率策略可以在[这里](https://mmaction2.readthedocs.io/en/latest/schedulers.html)找到，学习率策略的名称以 `LR` 结尾。

- **单一学习率策略**

  在大多数情况下，我们只使用一个学习策略以简化问题。例如，`MultiStepLR` 被用作 ResNet 的默认学习率策略。在这里，`param_scheduler` 是一个字典。

  ```python
  ```

  或者，我们想使用 `CosineAnnealingLR` 策略来衰减学习率：

  ```python

```python
  ```

  ```python
  ```

- **多个学习率策略**

  在某些训练案例中，为了提高准确性，会应用多个学习率策略。例如，在早期阶段，训练容易不稳定，预热是一种减少不稳定性的技术。学习率将从一个较小的值逐渐增加到预期值，通过预热进行衰减和其他策略进行衰减。

  在 MMAction2 中，通过将所需的策略组合成 `param_scheduler` 的列表即可实现预热策略。

  以下是一些示例：

  1. 在前 50 个迭代中进行线性预热。

  ```python
        # 线性预热
            by_epoch=False,  # 按迭代
            end=50),  # 仅在前 50 个迭代中进行预热
        # 主要的学习率策略
    ]
  ```

  2. 在前 10 个周期中进行线性预热，并在每个周期内按迭代更新学习率。

  ```python
        # 线性预热 [0, 10) 个周期

milestones=[8, 11],
    ]
  ```

  ```python
            end=10,
            convert_to_iter_based=True,  # 按迭代更新学习率
        ),
        # 在 10 个周期后使用 CosineAnnealing 策略
    ]
  ```

  注意，我们在这里使用 `begin` 和 `end` 参数来指定有效范围，该范围为 \[`begin`, `end`)。范围的单位由 `by_epoch` 参数定义。如果未指定，则 `begin` 为 0，`end` 为最大周期或迭代次数。

  如果所有策略的范围都不连续，则学习率将在忽略的范围内保持不变，否则所有有效的策略将按特定阶段的顺序执行，这与 PyTorch [`ChainedScheduler`](torch.optim.lr_scheduler.ChainedScheduler) 的行为相同。

### 自定义动量策略

我们支持使用动量策略根据学习率修改优化器的动量，这可以使损失以更快的方式收敛。使用方法与学习率策略相同。

所有可用的学习率策略可以在[这里](https://mmaction2.readthedocs.io/en/latest/schedulers.html)找到，动量策略的名称以 `Momentum` 结尾。

以下是一个示例：

```python
    # 学习率策略
    # 动量策略

以下是一个示例：

```python
    # 学习率策略
    # 动量策略
         begin=0,
         end=1000)
]
```

## 添加新的优化器或构造器

本部分将修改 MMAction2 源代码或向 MMAction2 框架中添加代码，初学者可以跳过此部分。

### 添加新的优化器

在学术研究和工业实践中，可能需要使用 MMAction2 未实现的优化方法，可以通过以下方法进行添加。

#### 1. 实现一个新的优化器

假设要添加一个名为 `MyOptimizer` 的优化器，它具有参数 `a`、`b` 和 `c`。需要在 `mmaction/engine/optimizers` 下创建一个新文件，并在文件中实现新的优化器，例如在 `mmaction/engine/optimizers/my_optimizer.py` 中：

```python

        ...

        ...
```

#### 2. 导入优化器

        ...

        ...
```

为了找到上述定义的模块，需要在运行时导入该模块。首先，在 `mmaction/engine/optimizers/__init__.py` 中导入该模块，将其添加到 `mmaction.engine` 包中。

```python
...
from .my_optimizer import MyOptimizer # MyOptimizer 可能是其他类名

```

在运行时，我们将自动导入 `mmaction.engine` 包，并同时注册 `MyOptimizer`。

然后，可以在配置文件的 `optim_wrapper.optimizer` 字段中使用 `MyOptimizer`。

```python
```

### 添加新的优化器构造器

一些模型可能对优化有一些特定的参数设置，例如所有 `BatchNorm` 层的不同权重衰减率。

尽管我们已经可以使用[优化器教程](#参数化精细配置)中的 `optim_wrapper.paramwise_cfg` 字段来配置各种特定参数的优化器设置，但可能仍无法满足需求。

```

当然，你可以修改它。默认情况下，我们使用 [`DefaultOptimWrapperConstructor`](mmengine.optim.DefaultOptimWrapperConstructor) 类来处理优化器的构造。在构造过程中，它根据 `paramwise_cfg` 对不同参数的优化器设置进行细致配置，这也可以作为新优化器构造器的模板。

你可以通过添加新的优化器构造器来覆盖这些行为。

```python

        ...

        ...
```

然后，导入它并几乎像[优化器教程](#添加新的优化器)中那样使用它。

1. 在 `mmaction/engine/optimizers/__init__.py` 中导入它，将其添加到 `mmaction.engine` 包中。

   ```python
   ...

```python
   ...

   ```

2. 在配置文件的 `optim_wrapper.constructor` 字段中使用 `MyOptimWrapperConstructor`。

   ```python
   )
   ```

# Preparing RSDD Dataset

```bibtex
@ARTICLE{RSDD2022,
author = {C. Xu, H. Su, J. Li, Y. Liu, L. Yao, L. Gao, W. Yan and T. Wang},
title = {RSDD-SAR: Rotated Ship Detection Dataset in SAR Images},
journal = {Journal of Radars},
year = {2022},
volume={11},
number={R22007},
pages={581},
}
```

## Download RSDD dataset

The RSDD dataset can be downloaded from [Google drive](https://drive.google.com/file/d/1PJxr7Tbr_ZAzuG8MNloDa4mLaRYCD3qc/view?usp=sharing).

```none
mmrotate
├── tools
├── data
│   ├── rsdd
│   │   ├── JPEGImages
│   │   ├── JPEGValidation
```

Please change `data_root` in `configs/_base_/datasets/rsdd.py` to `data/rsdd/`.

# ncnn GLSL extension

## rationale
Different GPUs support different features, some support fp16 as buffer storage type, some support fp16 as operand variable, some old GPUs only support fp32

When the GPU supports the `VK_KHR_16bit_storage` extension, in order to minimize the memory bandwidth consumption of the GPU, we will give priority to using fp16 as the storage type. Otherwise, we use `packHalf2x16` and `unpackHalf2x16` in GLSL 4.2 to compress 2 fp32 to uint, reducing read and write bandwidth.

Similarly, when the gpu supports the `VK_KHR_shader_float16_int8` extension, in order to speed up the calculation efficiency, we will give priority to using fp16 as the operation operand, which usually doubles the speed. Otherwise, we use fp32.

To ensure the widest compatibility, the following code for declaring descriptor binding and loading data will be written

```c
#if NCNN_fp16_storage // gpu supports 16bit storage
layout (binding = 0) buffer blob { f16vec4 blob_data[]; };

```c
#elif NCNN_fp16_packed // gpu supports GLSL 4.2
layout (binding = 0) buffer blob { uvec2 blob_data[]; };
#else // gpu only supports fp32
layout (binding = 0) buffer blob { vec4 blob_data[]; };
#endif

void main()
{
    const int i = int(gl_GlobalInvocationID.x);

#if NCNN_fp16_storage && NCNN_fp16_arithmetic // gpu supports 16bit storage and shader float16
    f16vec4 x = blob_data[i];
#elif NCNN_fp16_storage // gpu supports 16bit storage but no shader float16
    vec4 x = vec4(blob_data[i]);
#elif NCNN_fp16_packed && NCNN_fp16_arithmetic // gpu supports GLSL 4.2 and shader float16
    f16vec4 x = f16vec4(unpackFloat2x16(blob_data[i].x), unpackFloat2x16(blob_data[i].y));
    vec4 x = vec4(unpackHalf2x16(blob_data[i].x), unpackHalf2x16(blob_data[i].y));
    vec4 x = blob_data[i];
#endif
}
```

vec4 x = vec4(unpackHalf2x16(blob_data[i].x), unpackHalf2x16(blob_data[i].y));
#endif
}
```

As you can see, just declaring the buffer type and reading a value consumes a lot of lines of code, which is a maintenance nightmare. Therefore, ncnn adds more flexible data types and auxiliary functions to reduce the size of the code and improve readability, and will automatically expand to the most efficient implementation according to the feature level supported by the GPU.

The above code, by using the ncnn glsl extension, can be simplified to

```c
layout (binding = 0) buffer blob { sfpvec4 blob_data[]; };

{

    afpvec4 x = buffer_ld4(blob_data, i);
}
```

```c

{

}
```

The ncnn glsl extension provides the necessary data types for storage, computation, shared memory, and load, store, conversion functions for buffers and images. We also provide some buffer and image copy functions to prevent loss of precision when using fp16 as the intermediate data type, and to avoid unnecessary `unpackHalf2x16` and `packHalf2x16` pair.

# entrypoint for compiling GLSL

The gpu.h header in the ncnn library exposes 3 APIs for compiling glsl code into spir-v binary, they support ncnn glsl extension, these 3 functions accept opt switch to control the expansion form of ncnn glsl extension. The first two accept raw glsl code strings, and the last one is used to create ncnn's built-in shader.

```cpp
namespace ncnn {

// online spirv compilation

```cpp

NCNN_EXPORT int compile_spirv_module(const char* comp_string, const Option& opt, std::vector<uint32_t>& spirv);
NCNN_EXPORT int compile_spirv_module(const char* comp_data, int comp_data_size, const Option& opt, std::vector<uint32_t>& spirv);
NCNN_EXPORT int compile_spirv_module(int shader_type_index, const Option& opt, std::vector<uint32_t>& spirv);

} // namespace ncnn
```

## compile ncnn extended GLSL code directly

You can write shader code with ncnn glsl extension, compiled to spir-v using ncnn functions. The compiled product is a standard-compliant spir-v binary, which can be directly used to create a pipeline object in the vulkan api

```cpp
static const char my_glsl_data[] = R"(
#version 450

#if NCNN_fp16_storage
#extension GL_EXT_shader_16bit_storage: require
#endif
#if NCNN_fp16_arithmetic
#extension GL_EXT_shader_explicit_arithmetic_types_float16: require
#endif

#endif
#endif

layout (binding = 0) readonly buffer a_blob { sfpvec4 a_blob_data[]; };
layout (binding = 1) writeonly buffer b_blob { sfpvec4 b_blob_data[]; };

{

    afpvec4 v = buffer_ld4(a_blob_data, i);

    v = v + 123;

    buffer_st4(b_blob_data, i, v);
}
)";

Option opt;
 // you can control the extention behavior
 // even if the gpu supports 16bit storage
opt.use_fp16_storage = false;

std::vector<uint32_t> spirv;
ncnn::compile_spirv_module(my_glsl_data, sizeof(my_glsl_data) - 1, opt, spirv);

// To create pipeline object later
// ncnn::Pipeline pipeline(vkdev);
// pipeline.set_local_size_xyz(64, 1, 1);
// pipeline.create(spirv.data(), spirv.size() * 4, specializations);
```

## ncnn built-in shader

```

The shader index inside ncnn is exposed in the `layer_shader_type.h` header and can be used if needed

```cpp
#include "layer_shader_type.h"

int shader_type_index = LayerShaderType::convert_ycbcr;

int retc = compile_spirv_module(shader_type_index, opt, spirv);
```

# data types

## storage type

declare buffer data layout in descriptor binding

```c
layout (binding = 0) buffer top_blob { sfpvec4 top_blob_data[]; };
```

|storage type|fp32|fp16p|fp16s|
|sfp|float|float|float16_t|
|sfpvec2|vec2|uint|f16vec2|
|sfpvec4|vec4|uvec2|f16vec4|
|sfpvec8|mat2x4|uvec4|f16mat2x4|

## arithmetic type

declare local variable in glsl code

```c
{
    afpvec4 v = a * b;
}
```

|arithmetic type|fp32|fp16a|
|afp|float|float16_t|

```c
{
}
```

|afpvec2|vec2|f16vec2|
|afpvec4|vec4|f16vec4|
|afpvec8|mat2x4|f16mat2x4|

## local type

declare variable in shared local memory

```c
shared lfp tmp_a[8][4][2];
```

|local type|fp32|fp16p / fp16s|fp16s + fp16a|
|lfp|float|float|float16_t|
|lfpvec4|vec4|uvec2|f16vec4|

## image format and precision hint type

declare image format in descriptor binding

```c
layout (binding = 0) uniform unfp sampler3D bottom_blob_3d;
layout (binding = 1, imfmtc4) writeonly uniform unfp image3D top_blob_3d;
```

|format type|fp32|fp16p|fp16s|
|imfmt1|r32f|f32f|r16f|
|imfmt4|rgba32f|rgba16f|rgba16f|

|precision hint type|fp32|fp16p|fp16s|
|unfp|highp|mediump|mediump|

# buffer functions

- load typed value from src[offset]

```c
afp buffer_ld1(sfp src, int offset);

```c
afpvec2 buffer_ld2(sfpvec2 src, int offset);
afpvec4 buffer_ld4(sfpvec4 src, int offset);
afpvec8 buffer_ld8(sfpvec8 src, int offset);
```

- store typed value to dst[offset]

```c
void buffer_st1(sfp dst, int offset, afp v);
void buffer_st2(sfpvec2 dst, int offset, afpvec2 v);
void buffer_st4(sfpvec4 dst, int offset, afpvec4 v);
void buffer_st8(sfpvec8 dst, int offset, afpvec8 v);
```

- copy typed value from src[src_offset] to dst[dst_offset]

```c
void buffer_cp1(sfp dst, int dst_offset, sfp src, int src_offset);
void buffer_cp2(sfpvec2 dst, int dst_offset, sfpvec2 src, int src_offset);
void buffer_cp4(sfpvec4 dst, int dst_offset, sfpvec4 src, int src_offset);
void buffer_cp8(sfpvec4 dst, int dst_offset, sfpvec4 src, int src_offset);
```

```

- copy and pack value from src[src_offsets[0],src_offsets[1],...] to dst[dst_offset]

```c
void buffer_cp1to4(sfpvec4 dst, int dst_offset, sfp src, ivec4 src_offsets);
void buffer_cp1to8(sfpvec8 dst, int dst_offset, sfp src, ivec4 src_offsets_0, ivec4 src_offsets_1);
void buffer_cp4to8(sfpvec8 dst, int dst_offset, sfpvec4 src, ivec2 src_offsets);
```

- copy and unpack value from src[src_offset] to dst[dst_offsets[0],dst_offsets[1],...]

```c
void buffer_cp4to1(sfp dst, ivec4 dst_offsets, sfpvec4 src, int src_offset);
void buffer_cp8to1(sfp dst, ivec4 dst_offsets_0, ivec4 dst_offsets_1, sfpvec8 src, int src_offset);
void buffer_cp8to4(sfpvec4 dst, ivec2 dst_offsets, sfpvec8 src, int src_offset);
```

# image functions

- load typed value from src at pos

```c
afp image1d_ld1(sampler1D src, float pos);

```

```c
afp image2d_ld1(sampler2D src, vec2 pos);
afp image3d_ld1(sampler3D src, vec3 pos);
afpvec4 image1d_ld4(sampler1D src, float pos);
afpvec4 image2d_ld4(sampler2D src, vec2 pos);
afpvec4 image3d_ld4(sampler3D src, vec3 pos);
afpvec8 image1d_ld8(sampler1D src, float pos);
afpvec8 image2d_ld8(sampler2D src, vec2 pos);
afpvec8 image3d_ld8(sampler3D src, vec3 pos);
```

- store typed value to dst at pos

```c
void image1d_st1(image1D dst, int pos, afp v);
void image2d_st1(image2D dst, ivec2 pos, afp v);
void image3d_st1(image3D dst, ivec3 pos, afp v);
void image1d_st4(image1D dst, int pos, afpvec4 v);
void image2d_st4(image2D dst, ivec2 pos, afpvec4 v);
void image3d_st4(image3D dst, ivec3 pos, afpvec4 v);
void image1d_st8(image1D dst, int pos, afpvec8 v);

void image2d_st8(image2D dst, ivec2 pos, afpvec8 v);
void image3d_st8(image3D dst, ivec3 pos, afpvec8 v);
```

- copy typed value from src at src_pos to dst at dst_pos

```c
void image1d_cp1(image1D dst, int dst_pos, sampler1D src, float src_pos);
void image2d_cp1(image2D dst, ivec2 dst_pos, sampler2D src, vec2 src_pos);
void image3d_cp1(image3D dst, ivec3 dst_pos, sampler3D src, vec3 src_pos);
void image1d_cp4(image1D dst, int dst_pos, sampler1D src, float src_pos);
void image2d_cp4(image2D dst, ivec2 dst_pos, sampler2D src, vec2 src_pos);
void image3d_cp4(image3D dst, ivec3 dst_pos, sampler3D src, vec3 src_pos);
void image1d_cp8(image1D dst, int dst_pos, sampler1D src, float src_pos);
void image2d_cp8(image2D dst, ivec2 dst_pos, sampler2D src, vec2 src_pos);

void image3d_cp8(image3D dst, ivec3 dst_pos, sampler3D src, vec3 src_pos);
```

Note: Since image is an opaque data structure, no copy and pack/unpack functions are provided. To achieve this operation, you need to load first and then store.

# local data conversion functions

- storage buffer to local memory

```c
lfp sfp2lfp(sfp v);
lfpvec4 sfp2lfpvec4(sfpvec4 v);
```

- local memory to local variable

```c
afp lfp2afp(lfp v);
afpvec4 lfp2afpvec4(lfpvec4 v);
```

Note: The common usage of local memory is to read from global memory first, store it in local memory, and then read local variables from local memory for subsequent use. Therefore, only storage type to local type and local type to arithmetic type conversion functions are provided here.

# misc functions

- prefer specialization constant over push constant

```c
T psc(T x)
```

```c
T psc(T x)
```

Declare the same variable in specialization constant AND push constant section, then `psc(x)` will become a compile-time constant when specialization constant given non-zero or be dynamic via push constant otherwise. This is often used for tensor shape specialization. We can usually resolve all shape information and make them be compile-time constants for more aggressive shader optimization.

```c
layout (constant_id = 0) const int size = 0;

layout (push_constant) uniform parameter
{
    int size;
} p;

{
    const int s = psc(size);
}
```

# platform macros

judge if the current platform is moltenvk, for enabling some platform-specific workaround

```c
#if NCNN_moltenvk
// enable workaround for moltenvk
#endif
```

# option macros

enable glsl extension only if user enable some options

```c
#endif

#endif
```

```c
#endif
#endif
```

declare descriptor binding for image or buffer

```c
#if NCNN_image_shader
#else
layout (binding = 0) readonly buffer bottom_blob { sfpvec4 bottom_blob_data[]; };
#endif
```

|macro|defined by option|
|---|---|
|NCNN_fp16_packed|opt.use_fp16_packed|
|NCNN_fp16_storage|opt.use_fp16_storage|
|NCNN_fp16_arithmetic|opt.use_fp16_arithmetic|
|NCNN_int8_packed|opt.use_int8_packed|
|NCNN_int8_storage|opt.use_int8_storage|
|NCNN_int8_arithmetic|opt.use_int8_arithmetic|
|NCNN_image_shader|opt.use_image_storage|
|NCNN_shader_local_memory|opt.use_shader_local_memory|

# SegNeXt

> [SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation](https://arxiv.org/abs/2209.08575)

<a href="https://github.com/visual-attention-network/segnext">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/models/backbones/mscan.py#L328">Code Snippet</a>

We present SegNeXt, a simple convolutional network architecture for semantic segmentation. Recent transformer-based models have dominated the field of semantic segmentation due to the efficiency of self-attention in encoding spatial information. In this paper, we show that convolutional attention is a more efficient and effective way to encode contextual information than the self-attention mechanism in transformers. By re-examining the characteristics owned by successful segmentation models, we discover several key components leading to the performance improvement of segmentation models. This motivates us to design a novel convolutional attention network that uses cheap convolutional operations. Without bells and whistles, our SegNeXt significantly improves the performance of previous state-of-the-art methods on popular benchmarks, including ADE20K, Cityscapes, COCO-Stuff, Pascal VOC, Pascal Context, and iSAID. Notably, SegNeXt outperforms EfficientNet-L2 w/ NAS-FPN and achieves 90.6% mIoU on the Pascal VOC 2012 test leaderboard using only 1/10 parameters of it. On average, SegNeXt achieves about 2.0% mIoU improvements compared to the state-of-the-art methods on the ADE20K datasets with the same or fewer computations. Code is available at [this https URL](https://github.com/uyzhang/JSeg) (Jittor) and [this https URL](https://github.com/Visual-Attention-Network/SegNeXt) (Pytorch).

<img src="https://user-images.githubusercontent.com/24582831/215688018-5d4c8366-7793-4fdf-9397-960a09fac951.png" width="70%"/>
</div>

### ADE20K

| Method  | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device | mIoU  | mIoU(ms+flip) | config                                                                                                                              | download                                                                                                                                                                                                                                                                                                                                                                                   |

| SegNeXt | MSCAN-T  | 512x512   | 160000  | 17.88    | 52.38          | A100   | 41.50 | 42.59         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segnext/segnext_mscan-t_1xb16-adamw-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segnext/segnext_mscan-t_1x16_512x512_adamw_160k_ade20k/segnext_mscan-t_1x16_512x512_adamw_160k_ade20k_20230210_140244-05bd8466.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segnext/segnext_mscan-t_1x16_512x512_adamw_160k_ade20k/segnext_mscan-t_1x16_512x512_adamw_160k_ade20k_20230210_140244.log.json) |

| SegNeXt | MSCAN-S  | 512x512   | 160000  | 21.47    | 42.27          | A100   | 44.16 | 45.81         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segnext/segnext_mscan-s_1xb16-adamw-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segnext/segnext_mscan-s_1x16_512x512_adamw_160k_ade20k/segnext_mscan-s_1x16_512x512_adamw_160k_ade20k_20230214_113014-43013668.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segnext/segnext_mscan-s_1x16_512x512_adamw_160k_ade20k/segnext_mscan-s_1x16_512x512_adamw_160k_ade20k_20230214_113014.log.json) |

| SegNeXt | MSCAN-B  | 512x512   | 160000  | 31.03    | 35.15          | A100   | 48.03 | 49.68         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segnext/segnext_mscan-b_1xb16-adamw-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segnext/segnext_mscan-b_1x16_512x512_adamw_160k_ade20k/segnext_mscan-b_1x16_512x512_adamw_160k_ade20k_20230209_172053-b6f6c70c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segnext/segnext_mscan-b_1x16_512x512_adamw_160k_ade20k/segnext_mscan-b_1x16_512x512_adamw_160k_ade20k_20230209_172053.log.json) |

| SegNeXt | MSCAN-L  | 512x512   | 160000  | 43.32    | 22.91          | A100   | 50.99 | 52.10         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segnext/segnext_mscan-l_1xb16-adamw-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segnext/segnext_mscan-l_1x16_512x512_adamw_160k_ade20k/segnext_mscan-l_1x16_512x512_adamw_160k_ade20k_20230209_172055-19b14b63.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segnext/segnext_mscan-l_1x16_512x512_adamw_160k_ade20k/segnext_mscan-l_1x16_512x512_adamw_160k_ade20k_20230209_172055.log.json) |

Note:

Note:

- When we integrated SegNeXt into MMSegmentation, we modified some layers' names to make them more precise and concise without changing the model architecture. Therefore, the keys of pre-trained weights are different from the [original weights](https://cloud.tsinghua.edu.cn/d/c15b25a6745946618462/), but don't worry about these changes. we have converted them and uploaded the checkpoints, you might find URL of pre-trained checkpoints in config files and can use them directly for training.

- The total batch size is 16. We trained for SegNeXt with a single GPU as the performance degrades significantly when using`SyncBN` (mainly in `OverlapPatchEmbed` modules of `MSCAN`) of PyTorch 1.9.

- There will be subtle differences when model testing as Non-negative Matrix Factorization (NMF) in `LightHamHead` will be initialized randomly. To control this randomness, please set the random seed when model testing. You can modify [`./tools/test.py`](https://github.com/open-mmlab/mmsegmentation/blob/main/tools/test.py) like:

```python
    from mmengine.runner import seg_random_seed
    random_seed = xxx # set random seed recorded in training log
    set_random_seed(random_seed, deterministic=False)
    ...
```

    ...
```

- This model performance is sensitive to the seed values used, please refer to the log file for the specific settings of the seed. If you choose a different seed, the results might differ from the table results. Take SegNeXt Large for example, its results range from 49.60 to 51.0.

```bibtex
@article{guo2022segnext,
  title={SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation},
  author={Guo, Meng-Hao and Lu, Cheng-Ze and Hou, Qibin and Liu, Zhengning and Cheng, Ming-Ming and Hu, Shi-Min},
  journal={arXiv preprint arXiv:2209.08575},
}
```

# Welcome to Projects of MMAction2

In this folder, we welcome all contributions of deep-learning video understanding models from the community.

Here, these requirements, e.g., code standards, are not as strict as in the core package. Thus, developers from the community can implement their algorithms much more easily and efficiently in MMAction2. We appreciate all contributions from the community to make MMAction2 greater.

Here is an [example project](./example_project) about how to add your algorithms easily.

We also provide some documentation listed below:

- [Contribution Guide](https://mmaction2.readthedocs.io/en/latest/get_started/contribution_guide.html)

  The guides for new contributors about how to add your projects to MMAction2.

- [Discussions](https://github.com/open-mmlab/mmaction2/discussions)

# Test existing models on standard datasets

To evaluate a model's accuracy, one usually tests the model on some standard datasets, please refer to [dataset prepare guide](dataset_prepare.md) to prepare the dataset.

## Test existing models

We provide testing scripts for evaluating an existing model on the whole dataset (COCO, PASCAL VOC, Cityscapes, etc.).

- CPU

Choose the proper script to perform testing depending on the testing environment.

```shell
# Single-gpu testing
    [--show]

# CPU: disable GPUs and run single-gpu testing script
    [--show]

# Multi-gpu testing

    [--show]

```

`tools/dist_test.sh` also supports multi-node testing, but relies on PyTorch's [launch utility](https://pytorch.org/docs/stable/distributed.html#launch-utility).

- `RESULT_FILE`: Filename of the output results in pickle format. If not specified, the results will not be saved to a file.
- `--show`: If specified, detection results will be plotted on the images and shown in a new window. It is only applicable to single GPU testing and used for debugging and visualization. Please make sure that GUI is available in your environment. Otherwise, you may encounter an error like `cannot connect to X server`.

- `--show-dir`: If specified, detection results will be plotted on the images and saved to the specified directory. It is only applicable to single GPU testing and used for debugging and visualization. You do NOT need a GUI available in your environment for using this option.
- `--work-dir`: If specified, detection results containing evaluation metrics will be saved to the specified directory.
- `--cfg-options`:  If specified, the key-value pair optional cfg will be merged into config file

## Examples

Assuming that you have already downloaded the checkpoints to the directory `checkpoints/`.

1. Test RTMDet and visualize the results. Press any key for the next image.
   Config and checkpoint files are available [here](https://github.com/open-mmlab/mmdetection/tree/main/configs/rtmdet).

   ```shell
       --show
   ```

```shell
       --show
   ```

2. Test RTMDet and save the painted images for future visualization.

   ```shell
       --show-dir faster_rcnn_r50_fpn_1x_results
   ```

3. Test Faster R-CNN on PASCAL VOC (without saving the test results).
   Config and checkpoint files are available [here](../../../configs/pascal_voc).

   ```shell
   ```

4. Test Mask R-CNN with 8 GPUs, and evaluate.

configs/pascal_voc/faster-rcnn_r50_fpn_1x_voc0712.py \
   ```

   Config and checkpoint files are available [here](../../../configs/mask_rcnn).

   ```shell
       8 \
   ```

5. Test Mask R-CNN with 8 GPUs, and evaluate the metric **class-wise**.

   ```shell
       8 \
       --out results.pkl \
       --cfg-options test_evaluator.classwise=True
   ```

       8 \
   ```

6. Test Mask R-CNN on COCO test-dev with 8 GPUs, and generate JSON files for submitting to the official evaluation server.

   Replace the original test_evaluator and test_dataloader with test_evaluator and test_dataloader in the comment in [config](../../../configs/_base_/datasets/coco_instance.py) and run:

   ```shell
       8
   ```

   This command generates two JSON files `./work_dirs/coco_instance/test.bbox.json` and `./work_dirs/coco_instance/test.segm.json`.

8
   ```

7. Test Mask R-CNN on Cityscapes test with 8 GPUs, and generate txt and png files for submitting to the official evaluation server.
   Config and checkpoint files are available [here](../../../configs/cityscapes).

   Replace the original test_evaluator and test_dataloader with test_evaluator and test_dataloader in the comment in [config](../../../configs/_base_/datasets/cityscapes_instance.py) and run:

   ```shell
       8
   ```

   The generated png and txt would be under `./work_dirs/cityscapes_metric/` directory.

## Test without Ground Truth Annotations

8
   ```

MMDetection supports to test models without ground-truth annotations using `CocoDataset`. If your dataset format is not in COCO format, please convert them to COCO format. For example, if your dataset format is VOC, you can directly convert it to COCO format by the [script in tools.](../../../tools/dataset_converters/pascal_voc.py) If your dataset format is Cityscapes, you can directly convert it to COCO format by the [script in tools.](../../../tools/dataset_converters/cityscapes.py) The rest of the formats can be converted using [this script](../../../tools/dataset_converters/images2coco.py).

```shel
    ${OUT} \
```

arguments：

- `IMG_PATH`: The root path of images.
- `CLASSES`: The text file with a list of categories.

${IMG_PATH} \
    ${OUT} \
```

- `OUT`: The output annotation json file name. The save dir is in the same directory as `IMG_PATH`.
- `exclude-extensions`: The suffix of images to be excluded, such as 'png' and 'bmp'.

After the conversion is complete, you need to replace the original test_evaluator and test_dataloader with test_evaluator and test_dataloader in the comment in [config](../../../configs/_base_/datasets/coco_detection.py)(find which dataset in 'configs/_base_/datasets' the current config corresponds to) and run:

```shell
    [--show]

    [--show]

    [--show]

    [--show]
```

Assuming that the checkpoints in the [model zoo](https://mmdetection.readthedocs.io/en/latest/modelzoo_statistics.html) have been downloaded to the directory `checkpoints/`, we can test Mask R-CNN on COCO test-dev with 8 GPUs, and generate JSON files using the following command.

```sh
    8
```

This command generates two JSON files `./work_dirs/coco_instance/test.bbox.json` and `./work_dirs/coco_instance/test.segm.json`.

## Batch Inference

8
```

MMDetection supports inference with a single image or batched images in test mode. By default, we use single-image inference and you can use batch inference by modifying `samples_per_gpu` in the config of test data. You can do that either by modifying the config as below.

```shell
```

Or you can set it through `--cfg-options` as `--cfg-options test_dataloader.batch_size=2`

## Test Time Augmentation (TTA)

```

Test time augmentation (TTA) is a data augmentation strategy used during the test phase. It applies different augmentations, such as flipping and scaling, to the same image for model inference, and then merges the predictions of each augmented image to obtain more accurate predictions. To make it easier for users to use TTA, MMEngine provides [BaseTTAModel](https://mmengine.readthedocs.io/en/latest/api/generated/mmengine.model.BaseTTAModel.html#mmengine.model.BaseTTAModel) class, which allows users to implement different TTA strategies by simply extending the BaseTTAModel class according to their needs.

In MMDetection, we provides [DetTTAModel](../../../mmdet/models/test_time_augs/det_tta.py) class, which inherits from BaseTTAModel.

### Use case

Using TTA requires two steps. First, you need to add `tta_model` and `tta_pipeline` in the configuration file:

```shell

    dict(
        ], [
            dict(
       ]])]
```

       ]])]
```

Second, set `--tta` when running the test scripts as examples below:

```shell
    [--tta]

    [--tta]

    [--tta]
```

You can also modify the TTA config by yourself, such as adding scaling enhancement:

```shell

```shell

    dict(
        ], [
        ], [
            dict(
       ]])]
```

       ]])]
```

The above data augmentation pipeline will first perform 3 multi-scaling transformations on the image, followed by 2 flipping transformations (flipping and not flipping). Finally, the image is packaged into the final result using PackDetInputs.

Here are more TTA use cases for your reference:

For more advanced usage and data flow of TTA, please refer to [MMEngine](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/test_time_augmentation.html#data-flow). We will support instance segmentation TTA latter.

## 2D Face Keypoint Demo

We provide a demo script to test a single image or video with face detectors and top-down pose estimators. Assume that you have already installed [mmdet](https://github.com/open-mmlab/mmdetection) with version >= 3.0.

**Face Bounding Box Model Preparation:** The pre-trained face box estimation model can be found in [mmdet model zoo](/demo/docs/en/mmdet_modelzoo.md#face-bounding-box-detection-models).

### 2D Face Image Demo

```shell
```

[--show] [--device ${GPU_ID or CPU}] [--save-predictions] \
```

The pre-trained face keypoint estimation models can be found from [model zoo](https://mmpose.readthedocs.io/en/latest/model_zoo/face_2d_keypoint.html).
Take [aflw model](https://download.openmmlab.com/mmpose/face/hrnetv2/hrnetv2_w18_aflw_256x256-f2bbc62b_20210125.pth) as an example:

```shell
    demo/mmdetection_cfg/yolox-s_8xb8-300e_coco-face.py \
    https://download.openmmlab.com/mmpose/mmdet_pretrained/yolo-x_8xb8-300e_coco-face_13274d7c.pth \
    configs/face_2d_keypoint/topdown_heatmap/aflw/td-hm_hrnetv2-w18_8xb64-60e_aflw-256x256.py \
    https://download.openmmlab.com/mmpose/face/hrnetv2/hrnetv2_w18_aflw_256x256-f2bbc62b_20210125.pth \
    --input tests/data/cofw/001766.jpg \
```

https://download.openmmlab.com/mmpose/face/hrnetv2/hrnetv2_w18_aflw_256x256-f2bbc62b_20210125.pth \
```

<img src="https://user-images.githubusercontent.com/26127467/220538388-582ce90d-751a-40dd-ac06-3bc078b773a0.jpg" height="500px" alt><br>

```shell

    --draw-heatmap --output-root vis_results
```

```shell
    --show --draw-heatmap --device=cpu
```

### 2D Face Video Demo

```shell

```

```shell
    --input demo/resources/<demo_face.mp4> \
    --show --draw-heatmap --output-root vis_results
```

<img src="https://user-images.githubusercontent.com/26127467/220541430-6ade5a59-3d06-466a-a94d-00c82ff96a00.gif" height="500px" alt><br>

The original video can be downloaded from [Google Drive](https://drive.google.com/file/d/1kQt80t6w802b_vgVcmiV_QfcSJ3RWzmb/view?usp=sharing).

### 2D Face Pose Demo with Inferencer

```shell
python demo/inferencer_demo.py tests/data/wflw \
    --pose2d face --vis-out-dir vis_results/wflw --radius 1
```

This command infers all images located in `tests/data/wflw` and saves the visualization results in the `vis_results/wflw` directory.

<img src="https://user-images.githubusercontent.com/26127467/229793095-702f9d3b-461f-45bd-8535-d628e33bc907.jpg" alt="Image 1" width="400"/>

<img src="https://user-images.githubusercontent.com/26127467/229793121-9969f014-70da-40b5-8561-e21c3edd1aeb.jpg" alt="Image 2" width="400"/>

For 2D face keypoint estimation models, try to edit the config file. For example, set `model.test_cfg.flip_test=False` in line 90 of [aflw_hrnetv2](../../../configs/face_2d_keypoint/topdown_heatmap/aflw/td-hm_hrnetv2-w18_8xb64-60e_aflw-256x256.py).

# DyHead

> [Dynamic Head: Unifying Object Detection Heads with Attentions](https://arxiv.org/abs/2106.08322)

The complex nature of combining localization and classification in object detection has resulted in the flourished development of methods. Previous works tried to improve the performance in various object detection heads but failed to present a unified view. In this paper, we present a novel dynamic head framework to unify object detection heads with attentions. By coherently combining multiple self-attention mechanisms between feature levels for scale-awareness, among spatial locations for spatial-awareness, and within output channels for task-awareness, the proposed approach significantly improves the representation ability of object detection heads without any computational overhead. Further experiments demonstrate that the effectiveness and efficiency of the proposed dynamic head on the COCO benchmark. With a standard ResNeXt-101-DCN backbone, we largely improve the performance over popular object detectors and achieve a new state-of-the-art at 54.0 AP. Furthermore, with latest transformer backbone and extra data, we can push current best COCO result to a new record at 60.6 AP.

<img src="https://user-images.githubusercontent.com/42844407/149169448-fcafb6d0-b866-41cc-9422-94de9f1e1761.png" height="300"/>
</div>

| Method | Backbone |  Style  |   Setting    | Lr schd | Mem (GB) | Inf time (fps) | box AP |                      Config                      |                                                                                                                                                                                      Download                                                                                                                                                                                      |

|  ATSS  |   R-50   |  caffe  | reproduction |   1x    |   5.4    |      13.2      |  42.5  | [config](./atss_r50-caffe_fpn_dyhead_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/dyhead/atss_r50_fpn_dyhead_for_reproduction_1x_coco/atss_r50_fpn_dyhead_for_reproduction_4x4_1x_coco_20220107_213939-162888e6.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/dyhead/atss_r50_fpn_dyhead_for_reproduction_1x_coco/atss_r50_fpn_dyhead_for_reproduction_4x4_1x_coco_20220107_213939.log.json) |

|  ATSS  |   R-50   | pytorch |    simple    |   1x    |   4.9    |      13.7      |  43.3  |    [config](./atss_r50_fpn_dyhead_1x_coco.py)    |                               [model](https://download.openmmlab.com/mmdetection/v2.0/dyhead/atss_r50_fpn_dyhead_4x4_1x_coco/atss_r50_fpn_dyhead_4x4_1x_coco_20211219_023314-eaa620c6.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/dyhead/atss_r50_fpn_dyhead_4x4_1x_coco/atss_r50_fpn_dyhead_4x4_1x_coco_20211219_023314.log.json)                               |

- We trained the above models with 4 GPUs and 4 `samples_per_gpu`.
- The `reproduction` setting aims to reproduce the official implementation based on Detectron2.
- The `simple` setting serves as a minimum example to use DyHead in MMDetection. Specifically,
  - it adds `DyHead` to `neck` after `FPN`
  - it sets `stacked_convs=0` to `bbox_head`
- The `simple` setting achieves higher AP than the original implementation.

- it adds `DyHead` to `neck` after `FPN`
  We have not conduct ablation study between the two settings.
  `dict(type='Pad', size_divisor=128)` may further improve AP by prefer spatial alignment across pyramid levels, although large padding reduces efficiency.

We also trained the model with Swin-L backbone. Results are as below.

| Method | Backbone | Style |   Setting    | Lr schd | mstrain  | box AP |                         Config                          |                                                                                                                                                                                    Download                                                                                                                                                                                    |

|  ATSS  |  Swin-L  | caffe | reproduction |   2x    | 480~1200 |  56.2  | [config](./atss_swin-l-p4-w12_fpn_dyhead_ms-2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/dyhead/atss_swin-l-p4-w12_fpn_dyhead_mstrain_2x_coco/atss_swin-l-p4-w12_fpn_dyhead_mstrain_2x_coco_20220509_100315-bc5b6516.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/dyhead/atss_swin-l-p4-w12_fpn_dyhead_mstrain_2x_coco/atss_swin-l-p4-w12_fpn_dyhead_mstrain_2x_coco_20220509_100315.log.json) |

## Relation to Other Methods

- DyHead can be regarded as an improved [SEPC](https://arxiv.org/abs/2005.03101) with [DyReLU modules](https://arxiv.org/abs/2003.10027) and simplified [SE blocks](https://arxiv.org/abs/1709.01507).
- Xiyang Dai et al., the author team of DyHead, adopt it for [Dynamic DETR](https://openaccess.thecvf.com/content/ICCV2021/html/Dai_Dynamic_DETR_End-to-End_Object_Detection_With_Dynamic_Attention_ICCV_2021_paper.html).

The description of Dynamic Encoder in Sec. 3.2 will help you understand DyHead.

```latex
@inproceedings{DyHead_CVPR2021,
  author    = {Dai, Xiyang and Chen, Yinpeng and Xiao, Bin and Chen, Dongdong and Liu, Mengchen and Yuan, Lu and Zhang, Lei},
  title     = {Dynamic Head: Unifying Object Detection Heads With Attentions},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2021}
}
```

# Top-down integral-regression-based pose estimation

Top-down methods divide the task into two stages: object detection, followed by single-object pose estimation given object bounding boxes. At the 2nd stage, integral regression based methods use a simple integral operation relates and unifies the heatmap and joint regression differentiably, thus obtain the keypoint coordinates given the features extracted from the bounding box area, following the paradigm introduced in [Integral Human Pose Regression](https://arxiv.org/abs/1711.08229).

|        Model         | Input Size |  AP   |  AR   |                 Details and Download                  |
| :------------------: | :--------: | :---: | :---: | :---------------------------------------------------: |

| ResNet-50+Debias-IPR |  256x256   | 0.675 | 0.765 | [resnet_debias_coco.md](./coco/resnet_debias_coco.md) |
|    ResNet-50+DSNT    |  256x256   | 0.674 | 0.764 |   [resnet_dsnt_coco.md](./coco/resnet_dsnt_coco.md)   |
|    ResNet-50+IPR     |  256x256   | 0.633 | 0.730 |    [resnet_ipr_coco.md](./coco/resnet_ipr_coco.md)    |

# 新增数据增强

## 自定义数据增强

自定义数据增强必须继承 `BaseTransform` 并实现 `transform` 函数。这里我们使用一个简单的翻转变换作为示例：

```python

```

此外，新的类需要被导入。

```python
```

这样，我们就可以实例化一个 `MyFlip` 对象并使用它来处理数据字典。

```python

```

或者，我们可以在配置文件中的数据流程中使用 `MyFlip` 变换。

```python
    ...
    ...
]
```

需要注意，如果要在配置文件中使用 `MyFlip`，必须确保在运行时导入了包含 `MyFlip` 的文件。

<details>

```bibtex
}
```

<details>

```bibtex

```bibtex
}
```

Results on PoseTrack2018 val with ground-truth bounding boxes

| Arch                                                 | Input Size | Head | Shou | Elb  | Wri  | Hip  | Knee | Ankl | Total |                         ckpt                          |                         log                          |
| :--------------------------------------------------- | :--------: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :---: | :---------------------------------------------------: | :--------------------------------------------------: |

| [pose_hrnet_w32](/configs/body_2d_keypoint/topdown_heatmap/posetrack18/td-hm_hrnet-w32_8xb64-20e_posetrack18-256x192.py) |  256x192   | 86.2 | 89.0 | 84.5 | 79.2 | 82.3 | 82.5 | 78.7 | 83.4  | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_posetrack18_256x192-1ee951c4_20201028.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_posetrack18_256x192_20201028.log.json) |
| [pose_hrnet_w32](/configs/body_2d_keypoint/topdown_heatmap/posetrack18/td-hm_hrnet-w32_8xb64-20e_posetrack18-384x288.py) |  384x288   | 87.1 | 89.0 | 85.1 | 80.2 | 80.6 | 82.8 | 79.6 | 83.7  | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_posetrack18_384x288-806f00a3_20211130.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_posetrack18_384x288_20211130.log.json) |

| [pose_hrnet_w48](/configs/body_2d_keypoint/topdown_heatmap/posetrack18/td-hm_hrnet-w48_8xb64-20e_posetrack18-256x192.py) |  256x192   | 88.3 | 90.2 | 86.0 | 81.0 | 80.7 | 83.3 | 80.6 | 84.6  | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_posetrack18_256x192-b5d9b3f1_20211130.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_posetrack18_256x192_20211130.log.json) |
| [pose_hrnet_w48](/configs/body_2d_keypoint/topdown_heatmap/posetrack18/td-hm_hrnet-w48_8xb64-20e_posetrack18-384x288.py) |  384x288   | 87.8 | 90.0 | 86.2 | 81.3 | 81.0 | 83.4 | 80.9 | 84.6  | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_posetrack18_384x288-5fd6d3ff_20211130.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_posetrack18_384x288_20211130.log.json) |

The models are first pre-trained on COCO dataset, and then fine-tuned on PoseTrack18.

Results on PoseTrack2018 val with [MMDetection](https://github.com/open-mmlab/mmdetection) pre-trained [Cascade R-CNN](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_x101_64x4d_fpn_20e_coco/cascade_rcnn_x101_64x4d_fpn_20e_coco_20200509_224357-051557b1.pth) (X-101-64x4d-FPN) human detector

| [pose_hrnet_w32](/configs/body_2d_keypoint/topdown_heatmap/posetrack18/td-hm_hrnet-w32_8xb64-20e_posetrack18-256x192.py) |  256x192   | 78.0 | 82.9 | 79.5 | 73.8 | 76.9 | 76.6 | 70.2 | 76.9  | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_posetrack18_256x192-1ee951c4_20201028.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_posetrack18_256x192_20201028.log.json) |
| [pose_hrnet_w32](/configs/body_2d_keypoint/topdown_heatmap/posetrack18/td-hm_hrnet-w32_8xb64-20e_posetrack18-384x288.py) |  384x288   | 79.9 | 83.6 | 80.4 | 74.5 | 74.8 | 76.1 | 70.5 | 77.3  | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_posetrack18_384x288-806f00a3_20211130.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_posetrack18_384x288_20211130.log.json) |

| [pose_hrnet_w48](/configs/body_2d_keypoint/topdown_heatmap/posetrack18/td-hm_hrnet-w48_8xb64-20e_posetrack18-256x192.py) |  256x192   | 80.1 | 83.4 | 80.6 | 74.8 | 74.3 | 76.8 | 70.5 | 77.4  | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_posetrack18_256x192-b5d9b3f1_20211130.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_posetrack18_256x192_20211130.log.json) |
| [pose_hrnet_w48](/configs/body_2d_keypoint/topdown_heatmap/posetrack18/td-hm_hrnet-w48_8xb64-20e_posetrack18-384x288.py) |  384x288   | 80.2 | 83.8 | 80.9 | 75.2 | 74.7 | 76.7 | 71.7 | 77.8  | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_posetrack18_384x288-5fd6d3ff_20211130.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_posetrack18_384x288_20211130.log.json) |

# Build for Windows

- [Build for Windows](#build-for-windows)
      - [Install Dependencies for Model Converter](#install-dependencies-for-model-converter)
      - [Install Inference Engines for MMDeploy](#install-inference-engines-for-mmdeploy)
      - [Build Model Converter](#build-model-converter)
      - [Install Model Converter](#install-model-converter)
    - [Note](#note)

All the commands listed in the following chapters are verified on **Windows 10**.

1. Download and install [Visual Studio 2019](https://visualstudio.microsoft.com)

2. Add the path of `cmake` to the environment variable `PATH`, i.e., "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\CMake\\CMake\\bin"
3. Install cuda toolkit if NVIDIA gpu is available. You can refer to the official [guide](https://developer.nvidia.com/cuda-downloads).

#### Install Dependencies for Model Converter

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
    <td> Please install conda according to the official <a href="https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html">guide</a>. <br>

<tbody>
  <tr>
 After installation, open <code>anaconda powershell prompt</code> under the Start Menu <b>as the administrator</b>, because: <br>
1. <b>All the commands listed in the following text are verified in anaconda powershell </b><br>
2. <b>As an administrator, you can install the thirdparty libraries to the system path so as to simplify MMDeploy build command</b><br>
Note: if you are familiar with how cmake works, you can also use <code>anaconda powershell prompt</code> as an ordinary user.
    </td>
  </tr>
  <tr>
    <td>

<td>
pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html
    </td>
  </tr>
  <tr>
$env:cu_version="cu111"
$env:torch_version="torch1.8.0"
mim install "mmcv>=2.0.0rc1"
    </td>
  </tr>
</tbody>
</table>

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
    <td>

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
    <td>
    1. Find and download OpenCV 3+ for windows from <a href="https://github.com/opencv/opencv/releases">here</a>.<br>
    2. You can download the prebuilt package and install it to the target directory. Or you can build OpenCV from its source. <br>
    3. Find where <code>OpenCVConfig.cmake</code> locates in the installation directory. And export its path to the environment variable <code>PATH</code> like this,
<pre><code>$env:path = "\the\path\where\OpenCVConfig.cmake\locates;" + "$env:path"</code></pre>
    </td>
  </tr>
  <tr>
cd ppl.cv

cd ppl.cv
$env:PPLCV_DIR = "$pwd"
mkdir pplcv-build
cd pplcv-build
cmake .. -G "Visual Studio 16 2019" -T v142 -A x64 -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=install -DHPCC_USE_CUDA=ON -DPPLCV_USE_MSVC_STATIC_RUNTIME=OFF
cmake --build . --config Release -- /m
cmake --install . --config Release
cd ../..
   </td>
  </tr>
</tbody>
</table>

#### Install Inference Engines for MMDeploy

You can select your interested inference engines and do the installation by following the given commands.

**Currently, MMDeploy only verified ONNXRuntime and TensorRT for windows platform**.
As for the rest, MMDeploy will support them in the future.

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
    <td>
<pre><code>pip install onnxruntime==1.8.1</code></pre>
    2. Download the windows prebuilt binary package from <a href="https://github.com/microsoft/onnxruntime/releases/tag/v1.8.1">here</a>. Extract it and export environment variables as below:
Invoke-WebRequest -Uri https://github.com/microsoft/onnxruntime/releases/download/v1.8.1/onnxruntime-win-x64-1.8.1.zip -OutFile onnxruntime-win-x64-1.8.1.zip
Expand-Archive onnxruntime-win-x64-1.8.1.zip .
$env:ONNXRUNTIME_DIR = "$pwd\onnxruntime-win-x64-1.8.1"
$env:path = "$env:ONNXRUNTIME_DIR\lib;" + $env:path
    </td>
  </tr>
  <tr>
    <td>

    </td>
  </tr>
  <tr>
    <td>
   2. Here is an example of installing TensorRT 8.2 GA Update 2 for Windows x86_64 and CUDA 11.x that you can refer to. <br> First of all, click <a href="https://developer.nvidia.com/compute/machine-learning/tensorrt/secure/8.2.3.0/zip/TensorRT-8.2.3.0.Windows10.x86_64.cuda-11.4.cudnn8.2.zip">here</a> to download CUDA 11.x TensorRT 8.2.3.0 and then install it and other dependency like below:

Expand-Archive TensorRT-8.2.3.0.Windows10.x86_64.cuda-11.4.cudnn8.2.zip .
pip install $env:TENSORRT_DIR\python\tensorrt-8.2.3.0-cp37-none-win_amd64.whl
$env:TENSORRT_DIR = "$pwd\TensorRT-8.2.3.0"
   </td>
  </tr>
  <tr>
    <td>
In the above TensorRT's installation example, it requires cudnn8.2. Thus, you can download <a href="https://developer.nvidia.com/compute/machine-learning/cudnn/secure/8.2.1.32/11.3_06072021/cudnn-11.3-windows-x64-v8.2.1.32.zip">CUDA 11.x cuDNN 8.2</a><br>
    2. Extract the zip file and set the environment variables
cd \the\path\of\cudnn\zip\file
Expand-Archive cudnn-11.3-windows-x64-v8.2.1.32.zip .

2. Extract the zip file and set the environment variables
$env:CUDNN_DIR="$pwd\cuda"
$env:path = "$env:CUDNN_DIR\bin;" + $env:path
   </td>
  </tr>
  <tr>
    <td>TODO </td>
  </tr>
  <tr>
  </tr>
  <tr>
  </tr>
</tbody>
</table>

```powershell
cd \the\root\path\of\MMDeploy
$env:MMDEPLOY_DIR="$pwd"
```

If one of inference engines among ONNXRuntime, TensorRT and ncnn is selected, you have to build the corresponding custom ops.

mkdir build -ErrorAction SilentlyContinue
cd build
cmake .. -G "Visual Studio 16 2019" -A x64 -T v142 -DMMDEPLOY_TARGET_BACKENDS="ort" -DONNXRUNTIME_DIR="$env:ONNXRUNTIME_DIR"

cd build
```

cd build
cmake .. -G "Visual Studio 16 2019" -A x64 -T v142 -DMMDEPLOY_TARGET_BACKENDS="trt" -DTENSORRT_DIR="$env:TENSORRT_DIR" -DCUDNN_DIR="$env:CUDNN_DIR"
```

  TODO

cd $env:MMDEPLOY_DIR
```

**Note**

```

**Note**

  ```PowerShell
  cd $env:MMDEPLOY_DIR
  mkdir build -ErrorAction SilentlyContinue
  cd build
  cmake .. -G "Visual Studio 16 2019" -A x64 -T v142 `
      -DMMDEPLOY_BUILD_SDK=ON `
      -DMMDEPLOY_BUILD_EXAMPLES=ON `
      -DMMDEPLOY_BUILD_SDK_PYTHON_API=ON `
      -DMMDEPLOY_TARGET_DEVICES="cpu" `

-DMMDEPLOY_BUILD_SDK=ON `
      -DMMDEPLOY_TARGET_BACKENDS="ort" `
      -DONNXRUNTIME_DIR="$env:ONNXRUNTIME_DIR"

  cmake --build . --config Release -- /m
  cmake --install . --config Release
  ```

  cd build
    -DMMDEPLOY_BUILD_SDK=ON `
    -DMMDEPLOY_BUILD_EXAMPLES=ON `
    -DMMDEPLOY_BUILD_SDK_PYTHON_API=ON `
    -DMMDEPLOY_TARGET_DEVICES="cuda" `
    -DMMDEPLOY_TARGET_BACKENDS="trt" `
    -Dpplcv_DIR="$env:PPLCV_DIR/pplcv-build/install/lib/cmake/ppl" `
    -DTENSORRT_DIR="$env:TENSORRT_DIR" `
    -DCUDNN_DIR="$env:CUDNN_DIR"

  ```

### Note

-DTENSORRT_DIR="$env:TENSORRT_DIR" `

  ```

### Note

1. Release / Debug libraries can not be mixed. If MMDeploy is built with Release mode, all its dependent thirdparty libraries have to be built in Release mode too and vice versa.

# GCNet

> [GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond](https://arxiv.org/abs/1904.11492)

The Non-Local Network (NLNet) presents a pioneering approach for capturing long-range dependencies, via aggregating query-specific global context to each query position. However, through a rigorous empirical analysis, we have found that the global contexts modeled by non-local network are almost the same for different query positions within an image. In this paper, we take advantage of this finding to create a simplified network based on a query-independent formulation, which maintains the accuracy of NLNet but with significantly less computation. We further observe that this simplified design shares similar structure with Squeeze-Excitation Network (SENet). Hence we unify them into a three-step general framework for global context modeling. Within the general framework, we design a better instantiation, called the global context (GC) block, which is lightweight and can effectively model the global context. The lightweight property allows us to apply it for multiple layers in a backbone network to construct a global context network (GCNet), which generally outperforms both simplified NLNet and SENet on major benchmarks for various recognition tasks.

<img src="https://user-images.githubusercontent.com/40661020/143887418-c5d2c906-d6c9-415d-bce7-c5355c20a10a.png" height="300"/>
</div>

By [Yue Cao](http://yue-cao.me), [Jiarui Xu](http://jerryxu.net), [Stephen Lin](https://scholar.google.com/citations?user=c3PYmxUAAAAJ&hl=en), Fangyun Wei, [Han Hu](https://sites.google.com/site/hanhushomepage/).

We provide config files to reproduce the results in the paper for
["GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond"](https://arxiv.org/abs/1904.11492) on COCO object detection.

**GCNet** is initially described in [arxiv](https://arxiv.org/abs/1904.11492). Via absorbing advantages of Non-Local Networks (NLNet) and Squeeze-Excitation Networks (SENet),  GCNet provides a simple, fast and effective approach for global context modeling, which generally outperforms both NLNet and SENet on major benchmarks for various recognition tasks.

| Backbone  | Model |    Context     | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                         Config                          |                                                                                                                                                                       Download                                                                                                                                                                       |

| :-------: | :---: | :------------: | :-----: | :------: | :------------: | :----: | :-----: | :-----------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| R-50-FPN  | Mask  | GC(c3-c5, r16) |   1x    |   5.0    |                |  39.7  |  35.9   | [config](./mask-rcnn_r50-gcb-r16-c3-c5_fpn_1x_coco.py)  | [model](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r50_fpn_r16_gcb_c3-c5_1x_coco/mask_rcnn_r50_fpn_r16_gcb_c3-c5_1x_coco_20200515_211915-187da160.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r50_fpn_r16_gcb_c3-c5_1x_coco/mask_rcnn_r50_fpn_r16_gcb_c3-c5_1x_coco_20200515_211915.log.json) |

| R-50-FPN  | Mask  | GC(c3-c5, r4)  |   1x    |   5.1    |      15.0      |  39.9  |  36.0   |  [config](./mask-rcnn_r50-gcb-r4-c3-c5_fpn_1x_coco.py)  |      [model](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r50_fpn_r4_gcb_c3-c5_1x_coco/mask_rcnn_r50_fpn_r4_gcb_c3-c5_1x_coco_20200204-17235656.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r50_fpn_r4_gcb_c3-c5_1x_coco/mask_rcnn_r50_fpn_r4_gcb_c3-c5_1x_coco_20200204_024626.log.json)       |
| R-101-FPN | Mask  | GC(c3-c5, r16) |   1x    |   7.6    |      11.4      |  41.3  |  37.2   | [config](./mask-rcnn_r101-gcb-r16-c3-c5_fpn_1x_coco.py) |  [model](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r101_fpn_r16_gcb_c3-c5_1x_coco/mask_rcnn_r101_fpn_r16_gcb_c3-c5_1x_coco_20200205-e58ae947.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r101_fpn_r16_gcb_c3-c5_1x_coco/mask_rcnn_r101_fpn_r16_gcb_c3-c5_1x_coco_20200205_192835.log.json)   |

| R-101-FPN | Mask  | GC(c3-c5, r4)  |   1x    |   7.8    |      11.6      |  42.2  |  37.8   | [config](./mask-rcnn_r101-gcb-r4-c3-c5_fpn_1x_coco.py)  |    [model](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r101_fpn_r4_gcb_c3-c5_1x_coco/mask_rcnn_r101_fpn_r4_gcb_c3-c5_1x_coco_20200206-af22dc9d.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r101_fpn_r4_gcb_c3-c5_1x_coco/mask_rcnn_r101_fpn_r4_gcb_c3-c5_1x_coco_20200206_112128.log.json)     |

| Backbone  |      Model       |    Context     | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                                          Config                                          |                                                                                                                                                                                                                                                             Download                                                                                                                                                                                                                                                             |

| R-50-FPN  |       Mask       |       -        |   1x    |   4.4    |      16.6      |  38.4  |  34.6   |                     [config](./mask-rcnn_r50-syncbn_fpn_1x_coco.py)                      |                                                                                      [model](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r50_fpn_syncbn-backbone_1x_coco/mask_rcnn_r50_fpn_syncbn-backbone_1x_coco_20200202-bb3eb55c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r50_fpn_syncbn-backbone_1x_coco/mask_rcnn_r50_fpn_syncbn-backbone_1x_coco_20200202_214122.log.json)                                                                                       |

| R-50-FPN  |       Mask       | GC(c3-c5, r16) |   1x    |   5.0    |      15.5      |  40.4  |  36.2   |              [config](./mask-rcnn_r50-syncbn-gcb-r16-c3-c5_fpn_1x_coco.py)               |                                                          [model](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r50_fpn_syncbn-backbone_r16_gcb_c3-c5_1x_coco/mask_rcnn_r50_fpn_syncbn-backbone_r16_gcb_c3-c5_1x_coco_20200202-587b99aa.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r50_fpn_syncbn-backbone_r16_gcb_c3-c5_1x_coco/mask_rcnn_r50_fpn_syncbn-backbone_r16_gcb_c3-c5_1x_coco_20200202_174907.log.json)                                                           |

| R-50-FPN  |       Mask       | GC(c3-c5, r4)  |   1x    |   5.1    |      15.1      |  40.7  |  36.5   |               [config](./mask-rcnn_r50-syncbn-gcb-r4-c3-c5_fpn_1x_coco.py)               |                                                            [model](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r50_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco/mask_rcnn_r50_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco_20200202-50b90e5c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r50_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco/mask_rcnn_r50_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco_20200202_085547.log.json)                                                             |

| R-101-FPN |       Mask       |       -        |   1x    |   6.4    |      13.3      |  40.5  |  36.3   |                     [config](./mask-rcnn_r101-syncbn_fpn_1x_coco.py)                     |                                                                                    [model](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r101_fpn_syncbn-backbone_1x_coco/mask_rcnn_r101_fpn_syncbn-backbone_1x_coco_20200210-81658c8a.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r101_fpn_syncbn-backbone_1x_coco/mask_rcnn_r101_fpn_syncbn-backbone_1x_coco_20200210_220422.log.json)                                                                                     |

| R-101-FPN |       Mask       | GC(c3-c5, r16) |   1x    |   7.6    |      12.0      |  42.2  |  37.8   |              [config](./mask-rcnn_r101-syncbn-gcb-r16-c3-c5_fpn_1x_coco.py)              |                                                        [model](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r101_fpn_syncbn-backbone_r16_gcb_c3-c5_1x_coco/mask_rcnn_r101_fpn_syncbn-backbone_r16_gcb_c3-c5_1x_coco_20200207-945e77ca.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r101_fpn_syncbn-backbone_r16_gcb_c3-c5_1x_coco/mask_rcnn_r101_fpn_syncbn-backbone_r16_gcb_c3-c5_1x_coco_20200207_015330.log.json)                                                         |

| R-101-FPN |       Mask       | GC(c3-c5, r4)  |   1x    |   7.8    |      11.8      |  42.2  |  37.8   |              [config](./mask-rcnn_r101-syncbn-gcb-r4-c3-c5_fpn_1x_coco.py)               |                                                          [model](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r101_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco/mask_rcnn_r101_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco_20200206-8407a3f0.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_r101_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco/mask_rcnn_r101_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco_20200206_142508.log.json)                                                           |

| X-101-FPN |       Mask       |       -        |   1x    |   7.6    |      11.3      |  42.4  |  37.7   |                  [config](./mask-rcnn_x101-32x4d-syncbn_fpn_1x_coco.py)                  |                                                                        [model](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_x101_32x4d_fpn_syncbn-backbone_1x_coco/mask_rcnn_x101_32x4d_fpn_syncbn-backbone_1x_coco_20200211-7584841c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_x101_32x4d_fpn_syncbn-backbone_1x_coco/mask_rcnn_x101_32x4d_fpn_syncbn-backbone_1x_coco_20200211_054326.log.json)                                                                         |

| X-101-FPN |       Mask       | GC(c3-c5, r16) |   1x    |   8.8    |      9.8       |  43.5  |  38.6   |           [config](./mask-rcnn_x101-32x4d-syncbn-gcb-r16-c3-c5_fpn_1x_coco.py)           |                                            [model](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_x101_32x4d_fpn_syncbn-backbone_r16_gcb_c3-c5_1x_coco/mask_rcnn_x101_32x4d_fpn_syncbn-backbone_r16_gcb_c3-c5_1x_coco_20200211-cbed3d2c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_x101_32x4d_fpn_syncbn-backbone_r16_gcb_c3-c5_1x_coco/mask_rcnn_x101_32x4d_fpn_syncbn-backbone_r16_gcb_c3-c5_1x_coco_20200211_164715.log.json)                                             |

| X-101-FPN |       Mask       | GC(c3-c5, r4)  |   1x    |   9.0    |      9.7       |  43.9  |  39.0   |           [config](./mask-rcnn_x101-32x4d-syncbn-gcb-r4-c3-c5_fpn_1x_coco.py)            |                                              [model](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_x101_32x4d_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco/mask_rcnn_x101_32x4d_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco_20200212-68164964.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gcnet/mask_rcnn_x101_32x4d_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco/mask_rcnn_x101_32x4d_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco_20200212_070942.log.json)                                               |

| X-101-FPN |   Cascade Mask   |       -        |   1x    |   9.2    |      8.4       |  44.7  |  38.6   |              [config](./cascade-mask-rcnn_x101-32x4d-syncbn_fpn_1x_coco.py)              |                                                        [model](https://download.openmmlab.com/mmdetection/v2.0/gcnet/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_1x_coco/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_1x_coco_20200310-d5ad2a5e.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gcnet/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_1x_coco/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_1x_coco_20200310_115217.log.json)                                                         |

| X-101-FPN |   Cascade Mask   | GC(c3-c5, r16) |   1x    |   10.3   |      7.7       |  46.2  |  39.7   |       [config](./cascade-mask-rcnn_x101-32x4d-syncbn-r16-gcb-c3-c5_fpn_1x_coco.py)       |                            [model](https://download.openmmlab.com/mmdetection/v2.0/gcnet/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_r16_gcb_c3-c5_1x_coco/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_r16_gcb_c3-c5_1x_coco_20200211-10bf2463.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gcnet/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_r16_gcb_c3-c5_1x_coco/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_r16_gcb_c3-c5_1x_coco_20200211_184154.log.json)                             |

| X-101-FPN |   Cascade Mask   | GC(c3-c5, r4)  |   1x    |   10.6   |                |  46.4  |  40.1   |       [config](./cascade-mask-rcnn_x101-32x4d-syncbn-r4-gcb-c3-c5_fpn_1x_coco.py)        |                           [model](https://download.openmmlab.com/mmdetection/v2.0/gcnet/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco_20200703_180653-ed035291.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gcnet/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_r4_gcb_c3-c5_1x_coco_20200703_180653.log.json)                           |

| X-101-FPN | DCN Cascade Mask |       -        |   1x    |          |                |  47.5  |  40.9   |        [config](./cascade-mask-rcnn_x101-32x4d-syncbn-dconv-c3-c5_fpn_1x_coco.py)        |                             [model](https://download.openmmlab.com/mmdetection/v2.0/gcnet/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_dconv_c3-c5_1x_coco/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_dconv_c3-c5_1x_coco_20210615_211019-abbc39ea.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gcnet/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_dconv_c3-c5_1x_coco/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_dconv_c3-c5_1x_coco_20210615_211019.log.json)                             |

| X-101-FPN | DCN Cascade Mask | GC(c3-c5, r16) |   1x    |          |                |  48.0  |  41.3   | [config](./cascade-mask-rcnn_x101-32x4d-syncbn-dconv-c3-c5-r16-gcb-c3-c5_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/gcnet/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_dconv_c3-c5_r16_gcb_c3-c5_1x_coco/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_dconv_c3-c5_r16_gcb_c3-c5_1x_coco_20210615_215648-44aa598a.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gcnet/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_dconv_c3-c5_r16_gcb_c3-c5_1x_coco/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_dconv_c3-c5_r16_gcb_c3-c5_1x_coco_20210615_215648.log.json) |

| X-101-FPN | DCN Cascade Mask | GC(c3-c5, r4)  |   1x    |          |                |  47.9  |  41.1   | [config](./cascade-mask-rcnn_x101-32x4d-syncbn-dconv-c3-c5-r4-gcb-c3-c5_fpn_1x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/gcnet/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_dconv_c3-c5_r4_gcb_c3-c5_1x_coco/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_dconv_c3-c5_r4_gcb_c3-c5_1x_coco_20210615_161851-720338ec.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gcnet/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_dconv_c3-c5_r4_gcb_c3-c5_1x_coco/cascade_mask_rcnn_x101_32x4d_fpn_syncbn-backbone_dconv_c3-c5_r4_gcb_c3-c5_1x_coco_20210615_161851.log.json)   |

- The `SyncBN` is added in the backbone for all models in **Table 2**.
- `GC` denotes Global Context (GC) block is inserted after 1x1 conv of backbone.
- `DCN` denotes replace 3x3 conv with 3x3 Deformable Convolution in `c3-c5` stages of backbone.

- `r4` and `r16` denote ratio 4 and ratio 16 in GC block respectively.

```latex
@article{cao2019GCNet,
  title={GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond},
  author={Cao, Yue and Xu, Jiarui and Lin, Stephen and Wei, Fangyun and Hu, Han},
  journal={arXiv preprint arXiv:1904.11492},
}
```

# MobileNet V2

**MobileNet V2** is initially described in [the paper](https://arxiv.org/pdf/1801.04381.pdf), which improves the state of the art performance of mobile models on multiple tasks. MobileNetV2 is an improvement on V1. Its new ideas include Linear Bottleneck and Inverted Residuals, and is based on an inverted residual structure where the input and output of the residual block are thin bottleneck layers. The intermediate expansion layer uses lightweight depthwise convolutions to filter features as a source of non-linearity. The author of MobileNet V2 measure its performance on Imagenet classification, COCO object detection, and VOC image segmentation.

<img src="https://user-images.githubusercontent.com/26739999/142563365-7a9ea577-8f79-4c21-a750-ebcaad9bcc2f.png" width="60%"/>
</div>

<details>

</div>

<details>

<br>

</br>

```python

predict = inference_model('mobilenet-v2_8xb32_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('mobilenet-v2_8xb32_in1k', pretrained=True)
```

Train:

```shell
python tools/train.py configs/mobilenet_v2/mobilenet-v2_8xb32_in1k.py
```

Test:

```shell

Train:

```shell
```

Test:

```shell
python tools/test.py configs/mobilenet_v2/mobilenet-v2_8xb32_in1k.py https://download.openmmlab.com/mmclassification/v0/mobilenet_v2/mobilenet_v2_batch256_imagenet_20200708-3b2dc3af.pth
```

| Model                     |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                Config                |                                          Download                                          |
| :------------------------ | :----------: | :--------: | :-------: | :-------: | :-------: | :----------------------------------: | :----------------------------------------------------------------------------------------: |

| `mobilenet-v2_8xb32_in1k` | From scratch |    3.50    |   0.32    |   71.86   |   90.42   | [config](mobilenet-v2_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mobilenet_v2/mobilenet_v2_batch256_imagenet_20200708-3b2dc3af.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/mobilenet_v2/mobilenet_v2_batch256_imagenet_20200708-3b2dc3af.json) |

```bibtex
@INPROCEEDINGS{8578572,
  author={M. {Sandler} and A. {Howard} and M. {Zhu} and A. {Zhmoginov} and L. {Chen}},
  booktitle={2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  title={MobileNetV2: Inverted Residuals and Linear Bottlenecks},
  pages={4510-4520},
  doi={10.1109/CVPR.2018.00474}}
}
```

# Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction

> [Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction](https://arxiv.org/abs/2302.07817)

Modern methods for vision-centric autonomous driving perception widely adopt the bird's-eye-view (BEV) representation to describe a 3D scene. Despite its better efficiency than voxel representation, it has difficulty describing the fine-grained 3D structure of a scene with a single plane. To address this, we propose a tri-perspective view (TPV) representation which accompanies BEV with two additional perpendicular planes. We model each point in the 3D space by summing its projected features on the three planes. To lift image features to the 3D TPV space, we further propose a transformer-based TPV encoder (TPVFormer) to obtain the TPV features effectively. We employ the attention mechanism to aggregate the image features corresponding to each query in each TPV plane. Experiments show that our model trained with sparse supervision effectively predicts the semantic occupancy for all voxels. We demonstrate for the first time that using only camera inputs can achieve comparable performance with LiDAR-based methods on the LiDAR segmentation task on nuScenes. Code: https://github.com/wzzheng/TPVFormer.

<img src="https://github.com/traveller59/spconv/assets/72679458/8cc8caa6-b330-4f32-9599-3811dc5d7332" width="800"/>
</div>

We implement TPVFormer and provide the results and checkpoints on nuScenes dataset.

## Usage

1. Downloads the [pretrained backbone weights](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/tpvformer/tpvformer_8xb1-2x_nus-seg/tpvformer_pretrained_fcos3d_r101_dcn.pth) to checkpoints/

2. For example, to train TPVFormer on 8 GPUs, please use

```bash
bash tools/dist_train.sh projects/TPVFormer/config/tpvformer_8xb1-2x_nus-seg.py 8
```

```bash
```

In MMDetection3D's root directory, run the following command to test the model on 8 GPUs:

```bash
bash tools/dist_test.sh projects/TPVFormer/config/tpvformer_8xb1-2x_nus-seg.py  ${CHECKPOINT_PATH} 8
```

| Backbone                                                                                                                                         | Neck | Mem (GB) | Inf time (fps) | mIoU | Downloads                                                                                                                                                                                                                                                                                                             |

| [ResNet101 w/ DCN](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/fcos3d/fcos3d_r101-caffe-dcn_fpn_head-gn_8xb2-1x_nus-mono3d.py) | FPN  | 32.0     | -              | 68.9 | [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/tpvformer/tpvformer_8xb1-2x_nus-seg/tpvformer_8xb1-2x_nus-seg_20230411_150639-bd3844e2.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/tpvformer/tpvformer_8xb1-2x_nus-seg/tpvformer_8xb1-2x_nus-seg_20230411_150639.log) |

```latex
@article{huang2023tri,
    title={Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction},
    author={Huang, Yuanhui and Zheng, Wenzhao and Zhang, Yunpeng and Zhou, Jie and Lu, Jiwen },
    journal={arXiv preprint arXiv:2302.07817},
}
```

# Add New Datasets

## Add new custom dataset

Here we show how to develop a new custom dataset.

1. Create a new file `mmseg/datasets/example.py`

   ```python

           pass
   ```

2. Import the module in `mmseg/datasets/__init__.py`

   ```python
   ```

3. Use it by creating a new new dataset config file `configs/_base_/datasets/example_dataset.py`

   ```python
   ...
   ```

4. Add dataset meta information in `mmseg/utils/class_names.py`

   ```python
       return [
           ...
       ]

   ```python
       return [
           ...
       ]

       return [
           ...
       ]
       ...
   }
   ```

**Note:** If the new dataset does not satisfy the mmseg requirements, a data preprocessing script needs to be prepared in `tools/dataset_converters/`

## Customize datasets by reorganizing data

The simplest way is to convert your dataset to organize your data into folders.

An example of file structure is as followed.

```none
├── data
│   │   │   ├── val

│   │   │   ├── val
│   │   │   ├── val

```

A training pair will consist of the files with same suffix in img_dir/ann_dir.

Some datasets don't release the test set or don't release the ground truth of the test set, and we cannot evaluate models locally without the ground truth of the test set, so we set the validation set as the default test set in config files.

About how to build your own datasets or implement a new dataset class please refer to the [datasets guide](./datasets.md) for more detailed information.

**Note:** The annotations are images of shape (H, W), the value pixel should fall in range `[0, num_classes - 1]`.

You may use `'P'` mode of [pillow](https://pillow.readthedocs.io/en/stable/handbook/concepts.html#palette) to create your annotation image with color.

## Customize datasets by mixing dataset

MMSegmentation also supports to mix dataset for training.
Currently it supports to concat, repeat and multi-image mix datasets.

We use `RepeatDataset` as wrapper to repeat the dataset.
For example, suppose the original dataset is `Dataset_A`, to repeat it, the config looks like the following

```python
    times=N,
    dataset=dict(  # This is the original config of Dataset_A
        ...
    )
)
```

In case the dataset you want to concatenate is different, you can concatenate the dataset configs like the following.

```python

    )
)
```

```python
```

```python
    times=N,
        ...
    )
)
    ...
)
    ...
)
    times=M,
        ...
    )
)

)
    times=M,
        ...
    )
)

```

You can refer base dataset [tutorial](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/basedataset.html) from mmengine for more details

### Multi-image Mix Dataset

We use `MultiImageMixDataset` as a wrapper to mix images from multiple datasets.
`MultiImageMixDataset` can be used by multiple images mixed data augmentation like mosaic and mixup.

An example of using `MultiImageMixDataset` with `Mosaic` data augmentation:

```python

```python
]

        ]
    ),
)

```

# Ai challenger: A large-scale dataset for going deeper in image understanding

<details>

```bibtex
}
```

# 准备 Vimeo90K-triplet 数据集

```bibtex
  title={Video Enhancement with Task-Oriented Flow},
  journal={International Journal of Computer Vision (IJCV)},
}
```

训练集和测试集可以从 [此处](http://toflow.csail.mit.edu/) 下载。

Vimeo90K-triplet 数据集包含了如下所示的 `clip/sequence/img` 目录结构：

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── vimeo_triplet
│   │   ├── tri_testlist.txt
│   │   ├── tri_trainlist.txt
│   │   │   ├── 00001
│   │   │   │   ├── 0001
│   │   │   │   │   ├── im1.png
│   │   │   │   │   ├── im2.png
│   │   │   │   │   └── im3.png
│   │   │   │   ├── 0002
│   │   │   │   ├── 0003
│   │   │   ├── 00002
│   │   │   ├── ...
```

# Customize Models

We basically categorize model components into 5 types.

- backbone: usually an FCN network to extract feature maps, e.g., ResNet, MobileNet.
- neck: the component between backbones and heads, e.g., FPN, PAFPN.
- head: the component for specific tasks, e.g., bbox prediction and mask prediction.
- roi extractor: the part for extracting RoI features from feature maps, e.g., RoI Align.
- loss: the component in head for calculating losses, e.g., FocalLoss, L1Loss, and GHMLoss.

## Develop new components

### Add a new backbone

Here we show how to develop new components with an example of MobileNet.

#### 1. Define a new backbone (e.g. MobileNet)

Create a new file `mmdet/models/backbones/mobilenet.py`.

```python

        pass

        pass
```

#### 2. Import the module

        pass

        pass
```

You can either add the following line to `mmdet/models/backbones/__init__.py`

```python
```

```python
```

to the config file to avoid modifying the original code.

#### 3. Use the backbone in your config file

```python
    ...
        arg1=xxx,
    ...
```

### Add new necks

#### 1. Define a neck (e.g. PAFPN)

Create a new file `mmdet/models/necks/pafpn.py`.

```python

```python

                num_outs,
        pass

        pass
```

```python
```

```python
```

#### 3. Modify the config file

```python
```

### Add new heads

```python
```

Here we show how to develop a new head with the example of [Double Head R-CNN](https://arxiv.org/abs/1904.06493) as the following.

First, add a new bbox head in `mmdet/models/roi_heads/bbox_heads/double_bbox_head.py`.
Double Head R-CNN implements a new bbox head for object detection.
To implement a bbox head, basically we need to implement three functions of the new module as the following.

```python

                                          /-> cls
                                          \-> reg
                                          /-> cls
                                          \-> reg

norm_cfg: ConfigType = dict(type='BN'),
                         dict(
                     ]),

```

Second, implement a new RoI Head if it is necessary. We plan to inherit the new `DoubleHeadRoIHead` from `StandardRoIHead`. We can find that a `StandardRoIHead` already implements the following functions.

```python

```python

mask_head: ConfigType) -> None:

pos_inds: Optional[Tensor] = None,

```

Double Head's modification is mainly in the `bbox_forward` logic, and it inherits other logics from the `StandardRoIHead`. In the `mmdet/models/roi_heads/double_roi_head.py`, we implement the new RoI Head as the following:

```python

```

Last, the users need to add the module in
`mmdet/models/bbox_heads/__init__.py` and `mmdet/models/roi_heads/__init__.py` thus the corresponding registry could find and load them.

Alternatively, the users can add

```python
```

to the config file and achieve the same goal.

The config file of Double Head R-CNN is as the following

```python

```

Since MMDetection 2.0, the config system supports to inherit configs such that the users can focus on the modification.

```

The Double Head R-CNN mainly uses a new `DoubleHeadRoIHead` and a new `DoubleConvFCBBoxHead `, the arguments are set according to the `__init__` function of each module.

### Add new loss

Assume you want to add a new loss as `MyLoss`, for bounding box regression.
To add a new loss function, the users need implement it in `mmdet/models/losses/my_loss.py`.
The decorator `weighted_loss` enable the loss to be weighted for each element.

```python

                pred,
                target,
```

Then the users need to add it in the `mmdet/models/losses/__init__.py`.

```python

```

Alternatively, you can add

```python
```

```python

```

```python
```

To use it, modify the `loss_xxx` field.
Since MyLoss is for regression, you need to modify the `loss_bbox` field in the head.

```python
```

# MAE

<a href="https://github.com/facebookresearch/mae">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.24.0/mmseg/models/backbones/mae.py#L46">Code Snippet</a>

This paper shows that masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the input image and reconstruct the missing pixels. It is based on two core designs. First, we develop an asymmetric encoder-decoder architecture, with an encoder that operates only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask tokens. Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task. Coupling these two designs enables us to train large models efficiently and effectively: we accelerate training (by 3x or more) and improve accuracy. Our scalable approach allows for learning high-capacity models that generalize well: e.g., a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data. Transfer performance in downstream tasks outperforms supervised pre-training and shows promising scaling behavior.

<img src="https://user-images.githubusercontent.com/24582831/165456416-1cba54bf-b1b5-4bdf-ad86-d6390de7f342.png" width="70%"/>
</div>

## Usage

We provide a script [`beit2mmseg.py`](../../tools/model_converters/beit2mmseg.py) in the tools directory to convert the key of MAE model from [the official repo](https://github.com/facebookresearch/mae) to MMSegmentation style.

```shell
```

E.g.

```shell
python tools/model_converters/beit2mmseg.py https://dl.fbaipublicfiles.com/mae/pretrain/mae_pretrain_vit_base.pth pretrain/mae_pretrain_vit_base_mmcls.pth
```

```

| pretrained models               | original models                                                                                  |
| ------------------------------- | ------------------------------------------------------------------------------------------------ |
| mae_pretrain_vit_base_mmcls.pth | ['mae_pretrain_vit_base'](https://dl.fbaipublicfiles.com/mae/pretrain/mae_pretrain_vit_base.pth) |

```shell
configs/mae/upernet_mae-base_fp16_8x2_512x512_160k_ade20k.py \
upernet_mae-base_fp16_8x2_512x512_160k_ade20k_20220426_174752-f92a2975.pth $GPUS --eval mIoU
```

```shell
```

Since relative position embedding requires the input length and width to be equal, the sliding window is adopted for multi-scale inference. So we set min_size=512, that is, the shortest edge is 512. So the multi-scale inference of config is performed separately, instead of '--aug-test'. For multi-scale inference:

```shell
configs/mae/upernet_mae-base_fp16_512x512_160k_ade20k_ms.py \
```

### ADE20K

```

### ADE20K

| Method  | Backbone | Crop Size | pretrain    | pretrain img size | Batch Size | Lr schd | Mem (GB) | Inf time (fps) | Device | mIoU  | mIoU(ms+flip) | config                                                                                                                        | download                                                                                                                                                                                                                                                                                                                                                                       |

| UPerNet | ViT-B    | 512x512   | ImageNet-1K | 224x224           | 16         | 160000  | 9.96     | 7.14           | V100   | 48.13 |         48.70 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mae/mae-base_upernet_8xb2-amp-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mae/upernet_mae-base_fp16_8x2_512x512_160k_ade20k/upernet_mae-base_fp16_8x2_512x512_160k_ade20k_20220426_174752-f92a2975.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mae/upernet_mae-base_fp16_8x2_512x512_160k_ade20k/upernet_mae-base_fp16_8x2_512x512_160k_ade20k_20220426_174752.log.json) |

```bibtex
@article{he2021masked,
  title={Masked autoencoders are scalable vision learners},
  author={He, Kaiming and Chen, Xinlei and Xie, Saining and Li, Yanghao and Doll{\'a}r, Piotr and Girshick, Ross},
  journal={arXiv preprint arXiv:2111.06377},
}
```

# ESRGAN (ECCVW'2018)

> [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://arxiv.org/abs/1809.00219)

The Super-Resolution Generative Adversarial Network (SRGAN) is a seminal work that is capable of generating realistic textures during single image super-resolution. However, the hallucinated details are often accompanied with unpleasant artifacts. To further enhance the visual quality, we thoroughly study three key components of SRGAN - network architecture, adversarial loss and perceptual loss, and improve each of them to derive an Enhanced SRGAN (ESRGAN). In particular, we introduce the Residual-in-Residual Dense Block (RRDB) without batch normalization as the basic network building unit. Moreover, we borrow the idea from relativistic GAN to let the discriminator predict relative realness instead of the absolute value. Finally, we improve the perceptual loss by using the features before activation, which could provide stronger supervision for brightness consistency and texture recovery. Benefiting from these improvements, the proposed ESRGAN achieves consistently better visual quality with more realistic and natural textures than SRGAN and won the first place in the PIRM2018-SR Challenge.

 <img src="https://user-images.githubusercontent.com/7676947/144018578-6bb10830-b5fd-4d14-984e-4d7d85965c20.png" width="400"/>
</div >

|                                      Model                                      | Dataset |  PSNR   |  SSIM  | Training Resources |                                      Download                                       |
| :-----------------------------------------------------------------------------: | :-----: | :-----: | :----: | :----------------: | :---------------------------------------------------------------------------------: |

| [esrgan_psnr_x4c64b23g32_1x16_1000k_div2k](./esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py) |  Set5   | 30.6428 | 0.8559 |         1          | [model](https://download.openmmlab.com/mmediting/restorers/esrgan/esrgan_psnr_x4c64b23g32_1x16_1000k_div2k_20200420-bf5c993c.pth) |
| [esrgan_psnr_x4c64b23g32_1x16_1000k_div2k](./esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py) |  Set14  | 27.0543 | 0.7447 |         1          | [model](https://download.openmmlab.com/mmediting/restorers/esrgan/esrgan_psnr_x4c64b23g32_1x16_1000k_div2k_20200420-bf5c993c.pth) |
| [esrgan_psnr_x4c64b23g32_1x16_1000k_div2k](./esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py) |  DIV2K  | 29.3354 | 0.8263 |         1          | [model](https://download.openmmlab.com/mmediting/restorers/esrgan/esrgan_psnr_x4c64b23g32_1x16_1000k_div2k_20200420-bf5c993c.pth) |

| [esrgan_x4c64b23g32_1x16_400k_div2k](./esrgan_x4c64b23g32_1xb16-400k_div2k.py)  |  Set5   | 28.2700 | 0.7778 |         1          | [model](https://download.openmmlab.com/mmediting/restorers/esrgan/esrgan_x4c64b23g32_1x16_400k_div2k_20200508-f8ccaf3b.pth) |
| [esrgan_x4c64b23g32_1x16_400k_div2k](./esrgan_x4c64b23g32_1xb16-400k_div2k.py)  |  Set14  | 24.6328 | 0.6491 |         1          | [model](https://download.openmmlab.com/mmediting/restorers/esrgan/esrgan_x4c64b23g32_1x16_400k_div2k_20200508-f8ccaf3b.pth) |
| [esrgan_x4c64b23g32_1x16_400k_div2k](./esrgan_x4c64b23g32_1xb16-400k_div2k.py)  |  DIV2K  | 26.6531 | 0.7340 |         1          | [model](https://download.openmmlab.com/mmediting/restorers/esrgan/esrgan_x4c64b23g32_1x16_400k_div2k_20200508-f8ccaf3b.pth) |

**Train**

<details>

```shell
# cpu train

**Train**

<details>

```shell
# cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/esrgan/esrgan_x4c64b23g32_1xb16-400k_div2k.py

python tools/train.py configs/esrgan/esrgan_x4c64b23g32_1xb16-400k_div2k.py

./tools/dist_train.sh configs/esrgan/esrgan_x4c64b23g32_1xb16-400k_div2k.py 8
```

**Test**

<details>

```shell
# cpu test

**Test**

<details>

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/esrgan/esrgan_x4c64b23g32_1xb16-400k_div2k.py https://download.openmmlab.com/mmediting/restorers/esrgan/esrgan_x4c64b23g32_1x16_400k_div2k_20200508-f8ccaf3b.pth

python tools/test.py configs/esrgan/esrgan_x4c64b23g32_1xb16-400k_div2k.py https://download.openmmlab.com/mmediting/restorers/esrgan/esrgan_x4c64b23g32_1x16_400k_div2k_20200508-f8ccaf3b.pth

./tools/dist_test.sh configs/esrgan/esrgan_x4c64b23g32_1xb16-400k_div2k.py https://download.openmmlab.com/mmediting/restorers/esrgan/esrgan_x4c64b23g32_1x16_400k_div2k_20200508-f8ccaf3b.pth 8
```

```

```bibtex
@inproceedings{wang2018esrgan,
  title={Esrgan: Enhanced super-resolution generative adversarial networks},
  author={Wang, Xintao and Yu, Ke and Wu, Shixiang and Gu, Jinjin and Liu, Yihao and Dong, Chao and Qiao, Yu and Change Loy, Chen},
  booktitle={Proceedings of the European Conference on Computer Vision Workshops(ECCVW)},
  pages={0--0},
}
```

# 支持 RISC-V

MMDeploy 选择 ncnn 作为 RISC-V 平台下的推理后端。完整的部署过程包含两个步骤：

模型转换：在 Host 端将 PyTorch 模型转为 ncnn 模型。并将转换后的模型传到 device。

模型部署：在 Host 端以交叉编译方式编译 ncnn 和 MMDeploy。传到 Device 端进行推理。

## 1. 模型转换

a) 安装 MMDeploy

可参考 [BUILD 文档](./linux-x86_64.md)，安装 ncnn 推理引擎以及 MMDeploy。

b) 模型转换

以 Resnet-18 为例。先参照[文档](https://github.com/open-mmlab/mmpretrain)安装 mmpretrain，然后使用 `tools/deploy.py` 转换模型。

```bash

# 模型转换
```

## 2. 模型部署

a) 下载交叉编译工具链，设置环境变量

```bash
# 下载 Xuantie-900-gcc-linux-5.10.4-glibc-x86_64-V2.2.6-20220516.tar.gz
# https://occ.t-head.cn/community/download?id=4046947553902661632

--dump-info
```

## 2. 模型部署

```bash
tar xf Xuantie-900-gcc-linux-5.10.4-glibc-x86_64-V2.2.6-20220516.tar.gz
export RISCV_ROOT_PATH=`realpath Xuantie-900-gcc-linux-5.10.4-glibc-x86_64-V2.2.6`
```

b) 编译 ncnn & opencv

```bash
# ncnn
# 可参考 https://github.com/Tencent/ncnn/wiki/how-to-build#build-for-allwinner-d1

# opencv
git clone https://github.com/opencv/opencv.git
mkdir build_riscv && cd build_riscv
cmake .. \
 -DCMAKE_TOOLCHAIN_FILE=/path/to/mmdeploy/cmake/toolchains/riscv64-unknown-linux-gnu.cmake \
 -DCMAKE_INSTALL_PREFIX=install \
 -DBUILD_PERF_TESTS=OFF \
 -DBUILD_SHARED_LIBS=OFF \
 -DBUILD_TESTS=OFF \
 -DCMAKE_BUILD_TYPE=Release

```

c) 编译 mmdeploy SDK & demo

```bash
cmake .. \

-DBUILD_TESTS=OFF \

```

```bash
cmake .. \
  -DCMAKE_TOOLCHAIN_FILE=../cmake/toolchains/riscv64-unknown-linux-gnu.cmake \
  -DMMDEPLOY_SHARED_LIBS=OFF \
  -DMMDEPLOY_BUILD_EXAMPLES=ON \
  -Dncnn_DIR=${ncnn_DIR}/build-c906/install/lib/cmake/ncnn/ \
  -DMMDEPLOY_CODEBASES=all \
  -DOpenCV_DIR=${OpenCV_DIR}/build_riscv/install/lib/cmake/opencv4

```

执行 `make install` 之后, examples的可执行文件会保存在 `install/bin`

```
tree -L 1 install/bin/
.
├── ocr
```

d) 运行 demo

先确认测试模型用了 `--dump-info`，这样 `resnet18` 目录才有 `pipeline.json` 等 SDK 所需文件。

├── ocr
```

d) 运行 demo

把 dump 好的模型目录(resnet18)、可执行文件(image_classification)、测试图片(tests/data/tiger.jpeg)拷贝到设备中

```bash
```

# Log Collector

## Function

Automatically collect logs and write the result in a json file or markdown file.

If there are several `.log.json` files in one folder, Log Collector assumes that the `.log.json` files other than the first one are resume from the preceding `.log.json` file. Log Collector returns the result considering all `.log.json` files.

## Usage:

To use log collector, you need to write a config file to configure the log collector first.

example_config.py:

```python
# The work directory that contains folders that contains .log.json files.
work_dir = '../../work_dirs'
# The metric used to find the best evaluation.
metric = 'mIoU'

# **Don't specify the log_items and ignore_keywords at the same time.**
# Specify the log files we would like to collect in `log_items`.
# The folders specified should be the subdirectories of `work_dir`.
log_items = [
    'segformer_mit-b5_512x512_160k_ade20k_cnn_lr_with_warmup',

    'segformer_mit-b5_512x512_160k_ade20k_cnn_no_warmup_lr',
    'segformer_mit-b5_512x512_160k_ade20k_mit_trans_lr',
    'segformer_mit-b5_512x512_160k_ade20k_swin_trans_lr'
]
# Or specify `ignore_keywords`. The folders whose name contain one
# of the keywords in the `ignore_keywords` list(e.g., `'segformer'`)
# won't be collected.
# ignore_keywords = ['segformer']

# Other log items in .log.json that you want to collect.
# should not include metric.
other_info_keys = ["mAcc"]
# The output markdown file's name.
markdown_file ='markdowns/lr_in_trans.json.md'
# The output json file's name. (optional)
json_file = 'jsons/trans_in_cnn.json'
```

The structure of the work-dir directory should be like：

```text
├── work-dir
│   ├── folder1
│   │   ├── time1.log.json
│   │   ├── time2.log.json
│   │   ├── time3.log.json
│   │   ├── time4.log.json

```text
│   ├── folder2
│   │   ├── time5.log.json
│   │   ├── time6.log.json
│   │   ├── time7.log.json
│   │   ├── time8.log.json
```

Then , cd to the log collector folder.

Now you can run log_collector.py by using command:

```bash
python log_collector.py ./example_config.py
```

The output markdown file is like:

| exp_num |                         method                          | mIoU best | best index | mIoU last | last index | last iter num |
| :-----: | :-----------------------------------------------------: | :-------: | :--------: | :-------: | :--------: | :-----------: |
|    1    | segformer_mit-b5_512x512_160k_ade20k_cnn_lr_with_warmup |  0.2776   |     10     |  0.2776   |     10     |    160000     |

|    2    |  segformer_mit-b5_512x512_160k_ade20k_cnn_no_warmup_lr  |  0.2802   |     10     |  0.2802   |     10     |    160000     |
|    3    |    segformer_mit-b5_512x512_160k_ade20k_mit_trans_lr    |  0.4943   |     11     |  0.4943   |     11     |    160000     |
|    4    |   segformer_mit-b5_512x512_160k_ade20k_swin_trans_lr    |  0.4883   |     11     |  0.4883   |     11     |    160000     |

The output json file is like:

```json
[
    {
        "method": "segformer_mit-b5_512x512_160k_ade20k_cnn_lr_with_warmup",
        "metric_used": "mIoU",
        "last_iter": 160000,
        "last eval": {
            "eval_index": 10,
            "mIoU": 0.2776,
            "mAcc": 0.3779
        },
        "best eval": {
        }
    },
    {

"mIoU": 0.2776,
        },
        }
    },
    {
        "method": "segformer_mit-b5_512x512_160k_ade20k_cnn_no_warmup_lr",
            "mIoU": 0.2802,
            "mAcc": 0.3764
        },
        }
    },
    {
        "method": "segformer_mit-b5_512x512_160k_ade20k_mit_trans_lr",
            "eval_index": 11,
            "mIoU": 0.4943,
            "mAcc": 0.6097
        },
        }
    },
    {

"mIoU": 0.4943,
        },
        }
    },
    {
        "method": "segformer_mit-b5_512x512_160k_ade20k_swin_trans_lr",
            "mIoU": 0.4883,
            "mAcc": 0.6061
        },
        }
    }
]
```

    <b>OpenMMLab website</b>
    <sup>
        <i>HOT</i>
      </a>
    </sup>
    <b>OpenMMLab platform</b>
    <sup>
        <i>TRY IT OUT</i>
      </a>
    </sup>
  </div>

[📘Documentation](https://mmpose.readthedocs.io/en/latest/) |
[🛠️Installation](https://mmpose.readthedocs.io/en/latest/installation.html) |
[👀Model Zoo](https://mmpose.readthedocs.io/en/latest/model_zoo.html) |
[📜Papers](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/algorithms.html) |
[🆕Update News](https://mmpose.readthedocs.io/en/latest/notes/changelog.html) |
[🤔Reporting Issues](https://github.com/open-mmlab/mmpose/issues/new/choose) |

</div>

</div>

</div>

MMPose is an open-source toolbox for pose estimation based on PyTorch.

<br/>

<br/>

- **Support diverse tasks**

  We support a wide spectrum of mainstream pose analysis tasks in current research community, including 2d multi-person human pose estimation, 2d hand pose estimation, 2d face landmark detection, 133 keypoint whole-body human pose estimation, 3d human mesh recovery, fashion landmark detection and animal pose estimation.
  See [Demo](demo/docs/en) for more information.

- **Higher efficiency and higher accuracy**

  MMPose implements multiple state-of-the-art (SOTA) deep learning models, including both top-down & bottom-up approaches. We achieve faster training speed and higher accuracy than other popular codebases, such as [HRNet](https://github.com/leoxiaobin/deep-high-resolution-net.pytorch).
  See [benchmark.md](docs/en/notes/benchmark.md) for more information.

See [benchmark.md](docs/en/notes/benchmark.md) for more information.

- **Support for various datasets**

  The toolbox directly supports multiple popular and representative datasets, COCO, AIC, MPII, MPII-TRB, OCHuman etc.
  See [dataset_zoo](docs/en/dataset_zoo) for more information.

- **Well designed, tested and documented**

  We decompose MMPose into different components and one can easily construct a customized
  pose estimation framework by combining different modules.
  We provide detailed documentation and API reference, as well as unittests.

- We are glad to support 3 new datasets:

![image](https://github.com/open-mmlab/mmpose/assets/13503330/c9171dbb-7e7a-4c39-98e3-c92932182efb)

- Welcome to [*projects of MMPose*](/projects/README.md), where you can access to the latest features of MMPose, and share your ideas and codes with the community at once. Contribution to MMPose will be simple and smooth:

  - Provide an easy and agile way to integrate algorithms, features and applications into MMPose
  - Allow flexible code structure and style; only need a short code review process
  - Build individual projects with full power of MMPose but not bound up with heavy frameworks
  - Checkout new projects:
  - Become a contributors and make MMPose greater. Start your journey from the [example project](/projects/example_project/)

<br/>

- [Simple Keypoints](/projects/skps/)

<br/>

- 2023-07-04: MMPose [v1.1.0](https://github.com/open-mmlab/mmpose/releases/tag/v1.1.0) is officially released, with the main updates including:

  - Support new datasets: Human-Art, Animal Kingdom and LaPa.
  - Support new config type that is more user-friendly and flexible.
  - Improve RTMPose with better performance.
  - Migrate 3D pose estimation models on h36m.
  - Inference speedup and webcam inference with all demo scripts.

  Please refer to the [release notes](https://github.com/open-mmlab/mmpose/releases/tag/v1.1.0) for more updates brought by MMPose v1.1.0!

## 0.x / 1.x Migration

Please refer to the [release notes](https://github.com/open-mmlab/mmpose/releases/tag/v1.1.0) for more updates brought by MMPose v1.1.0!

MMPose v1.0.0 is a major update, including many API and config file changes. Currently, a part of the algorithms have been migrated to v1.0.0, and the remaining algorithms will be completed in subsequent versions. We will show the migration progress in the following list.

<summary><b>Migration Progress</b></summary>

| Algorithm                         |   Status    |

If your algorithm has not been migrated, you can continue to use the [0.x branch](https://github.com/open-mmlab/mmpose/tree/0.x) and [old documentation](https://mmpose.readthedocs.io/en/0.x/).

Please refer to [installation.md](https://mmpose.readthedocs.io/en/latest/installation.html) for more detailed installation and dataset preparation.

We provided a series of tutorials about the basic usage of MMPose for new users:

1. For the basic usage of MMPose:

   - [A 20-minute Tour to MMPose](https://mmpose.readthedocs.io/en/latest/guide_to_framework.html)
   - [Demos](https://mmpose.readthedocs.io/en/latest/demos.html)
   - [Inference](https://mmpose.readthedocs.io/en/latest/user_guides/inference.html)
   - [Configs](https://mmpose.readthedocs.io/en/latest/user_guides/configs.html)
   - [Prepare Datasets](https://mmpose.readthedocs.io/en/latest/user_guides/prepare_datasets.html)
   - [Train and Test](https://mmpose.readthedocs.io/en/latest/user_guides/train_and_test.html)

2. For developers who wish to develop based on MMPose:

   - [Learn about Codecs](https://mmpose.readthedocs.io/en/latest/advanced_guides/codecs.html)
   - [Dataflow in MMPose](https://mmpose.readthedocs.io/en/latest/advanced_guides/dataflow.html)

- [Learn about Codecs](https://mmpose.readthedocs.io/en/latest/advanced_guides/codecs.html)
   - [Implement New Models](https://mmpose.readthedocs.io/en/latest/advanced_guides/implement_new_models.html)
   - [Customize Datasets](https://mmpose.readthedocs.io/en/latest/advanced_guides/customize_datasets.html)
   - [Customize Data Transforms](https://mmpose.readthedocs.io/en/latest/advanced_guides/customize_transforms.html)
   - [Customize Optimizer](https://mmpose.readthedocs.io/en/latest/advanced_guides/customize_optimizer.html)
   - [Customize Logging](https://mmpose.readthedocs.io/en/latest/advanced_guides/customize_logging.html)
   - [How to Deploy](https://mmpose.readthedocs.io/en/latest/advanced_guides/how_to_deploy.html)
   - [Model Analysis](https://mmpose.readthedocs.io/en/latest/advanced_guides/model_analysis.html)

- [How to Deploy](https://mmpose.readthedocs.io/en/latest/advanced_guides/how_to_deploy.html)
   - [Migration Guide](https://mmpose.readthedocs.io/en/latest/migration.html)

3. For researchers and developers who are willing to contribute to MMPose:

   - [Contribution Guide](https://mmpose.readthedocs.io/en/latest/contribution_guide.html)

4. For some common issues, we provide a FAQ list:

   - [FAQ](https://mmpose.readthedocs.io/en/latest/faq.html)

Results and models are available in the **README.md** of each method's config directory.
A summary can be found in the [Model Zoo](https://mmpose.readthedocs.io/en/latest/model_zoo.html) page.

- [x] [DeepPose](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/algorithms.html#deeppose-cvpr-2014) (CVPR'2014)

- [x] [CPM](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#cpm-cvpr-2016) (CVPR'2016)
- [x] [Hourglass](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#hourglass-eccv-2016) (ECCV'2016)
- [x] [SimpleBaseline3D](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/algorithms.html#simplebaseline3d-iccv-2017) (ICCV'2017)
- [ ] [Associative Embedding](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/algorithms.html#associative-embedding-nips-2017) (NeurIPS'2017)
- [x] [SimpleBaseline2D](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/algorithms.html#simplebaseline2d-eccv-2018) (ECCV'2018)
- [x] [DSNT](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/algorithms.html#dsnt-2018) (ArXiv'2021)

- [x] [HRNet](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#hrnet-cvpr-2019) (CVPR'2019)
- [x] [IPR](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/algorithms.html#ipr-eccv-2018) (ECCV'2018)
- [x] [VideoPose3D](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/algorithms.html#videopose3d-cvpr-2019) (CVPR'2019)
- [x] [HRNetv2](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#hrnetv2-tpami-2019) (TPAMI'2019)
- [x] [MSPN](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#mspn-arxiv-2019) (ArXiv'2019)
- [x] [SCNet](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#scnet-cvpr-2020) (CVPR'2020)
- [ ] [HigherHRNet](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#higherhrnet-cvpr-2020) (CVPR'2020)

- [x] [RSN](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#rsn-eccv-2020) (ECCV'2020)
- [ ] [InterNet](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/algorithms.html#internet-eccv-2020) (ECCV'2020)
- [ ] [VoxelPose](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/algorithms.html#voxelpose-eccv-2020) (ECCV'2020)
- [x] [LiteHRNet](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#litehrnet-cvpr-2021) (CVPR'2021)
- [x] [ViPNAS](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#vipnas-cvpr-2021) (CVPR'2021)
- [x] [Debias-IPR](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/algorithms.html#debias-ipr-iccv-2021) (ICCV'2021)
- [x] [SimCC](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/algorithms.html#simcc-eccv-2022) (ECCV'2022)

<summary><b>Supported techniques:</b></summary>

- [x] [FPN](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/techniques.html#fpn-cvpr-2017) (CVPR'2017)
- [x] [FP16](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/techniques.html#fp16-arxiv-2017) (ArXiv'2017)
- [x] [Wingloss](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/techniques.html#wingloss-cvpr-2018) (CVPR'2018)
- [x] [AdaptiveWingloss](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/techniques.html#adaptivewingloss-iccv-2019) (ICCV'2019)
- [x] [DarkPose](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/techniques.html#darkpose-cvpr-2020) (CVPR'2020)
- [x] [UDP](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/techniques.html#udp-cvpr-2020) (CVPR'2020)

- [x] [Albumentations](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/techniques.html#albumentations-information-2020) (Information'2020)
- [x] [SoftWingloss](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/techniques.html#softwingloss-tip-2021) (TIP'2021)
- [x] [RLE](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/techniques.html#rle-iccv-2021) (ICCV'2021)

<summary><b>Supported datasets:</b></summary>

- [x] [AFLW](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#aflw-iccvw-2011) \[[homepage](https://www.tugraz.at/institute/icg/research/team-bischof/lrs/downloads/aflw/)\] (ICCVW'2011)
- [x] [sub-JHMDB](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#jhmdb-iccv-2013) \[[homepage](http://jhmdb.is.tue.mpg.de/dataset)\] (ICCV'2013)

- [x] [COFW](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#cofw-iccv-2013) \[[homepage](http://www.vision.caltech.edu/xpburgos/ICCV13/)\] (ICCV'2013)
- [x] [MPII](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#mpii-cvpr-2014) \[[homepage](http://human-pose.mpi-inf.mpg.de/)\] (CVPR'2014)
- [x] [Human3.6M](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#human3-6m-tpami-2014) \[[homepage](http://vision.imar.ro/human3.6m/description.php)\] (TPAMI'2014)
- [x] [COCO](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#coco-eccv-2014) \[[homepage](http://cocodataset.org/)\] (ECCV'2014)

- [x] [CMU Panoptic](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#cmu-panoptic-iccv-2015) \[[homepage](http://domedb.perception.cs.cmu.edu/)\] (ICCV'2015)
- [x] [DeepFashion](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#deepfashion-cvpr-2016) \[[homepage](http://mmlab.ie.cuhk.edu.hk/projects/DeepFashion/LandmarkDetection.html)\] (CVPR'2016)
- [x] [300W](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#300w-imavis-2016) \[[homepage](https://ibug.doc.ic.ac.uk/resources/300-W/)\] (IMAVIS'2016)
- [x] [RHD](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#rhd-iccv-2017) \[[homepage](https://lmb.informatik.uni-freiburg.de/resources/datasets/RenderedHandposeDataset.en.html)\] (ICCV'2017)

- [x] [CMU Panoptic HandDB](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#cmu-panoptic-handdb-cvpr-2017) \[[homepage](http://domedb.perception.cs.cmu.edu/handdb.html)\] (CVPR'2017)
- [x] [AI Challenger](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#ai-challenger-arxiv-2017) \[[homepage](https://github.com/AIChallenger/AI_Challenger_2017)\] (ArXiv'2017)
- [x] [MHP](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#mhp-acm-mm-2018) \[[homepage](https://lv-mhp.github.io/dataset)\] (ACM MM'2018)
- [x] [WFLW](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#wflw-cvpr-2018) \[[homepage](https://wywu.github.io/projects/LAB/WFLW.html)\] (CVPR'2018)
- [x] [PoseTrack18](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#posetrack18-cvpr-2018) \[[homepage](https://posetrack.net/users/download.php)\] (CVPR'2018)

- [x] [OCHuman](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#ochuman-cvpr-2019) \[[homepage](https://github.com/liruilong940607/OCHumanApi)\] (CVPR'2019)
- [x] [CrowdPose](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#crowdpose-cvpr-2019) \[[homepage](https://github.com/Jeff-sjtu/CrowdPose)\] (CVPR'2019)
- [x] [MPII-TRB](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#mpii-trb-iccv-2019) \[[homepage](https://github.com/kennymckormick/Triplet-Representation-of-human-Body)\] (ICCV'2019)
- [x] [FreiHand](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#freihand-iccv-2019) \[[homepage](https://lmb.informatik.uni-freiburg.de/projects/freihand/)\] (ICCV'2019)

- [x] [Animal-Pose](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#animal-pose-iccv-2019) \[[homepage](https://sites.google.com/view/animal-pose/)\] (ICCV'2019)
- [x] [OneHand10K](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#onehand10k-tcsvt-2019) \[[homepage](https://www.yangangwang.com/papers/WANG-MCC-2018-10.html)\] (TCSVT'2019)
- [x] [Vinegar Fly](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#vinegar-fly-nature-methods-2019) \[[homepage](https://github.com/jgraving/DeepPoseKit-Data)\] (Nature Methods'2019)
- [x] [Desert Locust](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#desert-locust-elife-2019) \[[homepage](https://github.com/jgraving/DeepPoseKit-Data)\] (Elife'2019)

- [x] [Grévy’s Zebra](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#grevys-zebra-elife-2019) \[[homepage](https://github.com/jgraving/DeepPoseKit-Data)\] (Elife'2019)
- [x] [ATRW](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#atrw-acm-mm-2020) \[[homepage](https://cvwc2019.github.io/challenge.html)\] (ACM MM'2020)
- [x] [Halpe](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#halpe-cvpr-2020) \[[homepage](https://github.com/Fang-Haoshu/Halpe-FullBody/)\] (CVPR'2020)
- [x] [COCO-WholeBody](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#coco-wholebody-eccv-2020) \[[homepage](https://github.com/jin-s13/COCO-WholeBody/)\] (ECCV'2020)

- [x] [MacaquePose](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#macaquepose-biorxiv-2020) \[[homepage](http://www.pri.kyoto-u.ac.jp/datasets/macaquepose/index.html)\] (bioRxiv'2020)
- [x] [InterHand2.6M](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#interhand2-6m-eccv-2020) \[[homepage](https://mks0601.github.io/InterHand2.6M/)\] (ECCV'2020)
- [x] [AP-10K](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#ap-10k-neurips-2021) \[[homepage](https://github.com/AlexTheBad/AP-10K)\] (NeurIPS'2021)
- [x] [Horse-10](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#horse-10-wacv-2021) \[[homepage](http://www.mackenziemathislab.org/horse10)\] (WACV'2021)

- [x] [Human-Art](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#human-art-cvpr-2023) \[[homepage](https://idea-research.github.io/HumanArt/)\] (CVPR'2023)
- [x] [LaPa](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/datasets.html#lapa-aaai-2020) \[[homepage](https://github.com/JDAI-CV/lapa-dataset)\] (AAAI'2020)

<summary><b>Supported backbones:</b></summary>

- [x] [AlexNet](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#alexnet-neurips-2012) (NeurIPS'2012)
- [x] [VGG](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#vgg-iclr-2015) (ICLR'2015)
- [x] [ResNet](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#resnet-cvpr-2016) (CVPR'2016)

- [x] [ResNext](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#resnext-cvpr-2017) (CVPR'2017)
- [x] [SEResNet](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#seresnet-cvpr-2018) (CVPR'2018)
- [x] [ShufflenetV1](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#shufflenetv1-cvpr-2018) (CVPR'2018)
- [x] [ShufflenetV2](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#shufflenetv2-eccv-2018) (ECCV'2018)
- [x] [MobilenetV2](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#mobilenetv2-cvpr-2018) (CVPR'2018)
- [x] [ResNetV1D](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#resnetv1d-cvpr-2019) (CVPR'2019)
- [x] [ResNeSt](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#resnest-arxiv-2020) (ArXiv'2020)

- [x] [HRFormer](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#hrformer-nips-2021) (NIPS'2021)
- [x] [PVT](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#pvt-iccv-2021) (ICCV'2021)
- [x] [PVTV2](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/backbones.html#pvtv2-cvmj-2022) (CVMJ'2022)

### Model Request

We will keep up with the latest progress of the community, and support more popular algorithms and frameworks. If you have any feature requests, please feel free to leave a comment in [MMPose Roadmap](https://github.com/open-mmlab/mmpose/issues/2258).

We appreciate all contributions to improve MMPose. Please refer to [CONTRIBUTING.md](https://mmpose.readthedocs.io/en/latest/contribution_guide.html) for the contributing guideline.

MMPose is an open source project that is contributed by researchers and engineers from various colleges and companies.
We appreciate all the contributors who implement their methods or add new features, as well as users who give valuable feedbacks.
We wish that the toolbox and benchmark could serve the growing research community by providing a flexible toolkit to reimplement existing methods and develop their own new models.

```bibtex
}
```

## License

title={OpenMMLab Pose Estimation Toolbox and Benchmark},
}
```

## License

- [MMDeploy](https://github.com/open-mmlab/mmdeploy): OpenMMLab Model Deployment Framework.

# DINOv2

> [DINOv2: Learning Robust Visual Features without Supervision](https://arxiv.org/abs/2304.07193)

The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could greatly simplify the use of images in any system by producing allpurpose visual features, i.e., features that work across image distributions and tasks without finetuning. This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources. We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size. Most of the technical contributions aim at accelerating and stabilizing the training at scale. In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature. In terms of models, we train a ViT model (Dosovitskiy et al., 2020) with 1B parameters and distill it into a series of smaller models that surpass the best available all-purpose features, OpenCLIP (Ilharco et al., 2021) on most of the benchmarks at image and pixel levels.

<img src="https://user-images.githubusercontent.com/36138628/234560516-b495795c-c75c-444c-a712-bb61a3de444e.png" width="70%"/>
</div>

```python

model = get_model('vit-small-p14_dinov2-pre_3rdparty', pretrained=True)
```

| Model                                 | Params (M) | Flops (G) |                     Config                     |                                              Download                                              |
| :------------------------------------ | :--------: | :-------: | :--------------------------------------------: | :------------------------------------------------------------------------------------------------: |

| `vit-small-p14_dinov2-pre_3rdparty`\* |   22.06    |   46.76   | [config](vit-small-p14_dinov2-pre_headless.py) | [model](https://download.openmmlab.com/mmpretrain/v1.0/dinov2/vit-small-p14_dinov2-pre_3rdparty_20230426-5641ca5a.pth) |
| `vit-base-p14_dinov2-pre_3rdparty`\*  |   86.58    |  152.00   | [config](vit-base-p14_dinov2-pre_headless.py)  | [model](https://download.openmmlab.com/mmpretrain/v1.0/dinov2/vit-base-p14_dinov2-pre_3rdparty_20230426-ba246503.pth) |
| `vit-large-p14_dinov2-pre_3rdparty`\* |   304.00   |  507.00   | [config](vit-large-p14_dinov2-pre_headless.py) | [model](https://download.openmmlab.com/mmpretrain/v1.0/dinov2/vit-large-p14_dinov2-pre_3rdparty_20230426-f3302d9e.pth) |
| `vit-giant-p14_dinov2-pre_3rdparty`\* |  1136.00   |  1784.00  | [config](vit-giant-p14_dinov2-pre_headless.py) | [model](https://download.openmmlab.com/mmpretrain/v1.0/dinov2/vit-giant-p14_dinov2-pre_3rdparty_20230426-2934a630.pth) |

*Models with * are converted from the [official repo](https://github.com/facebookresearch/dinov2). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@misc{oquab2023dinov2,
  title={DINOv2: Learning Robust Visual Features without Supervision},
  author={Oquab, Maxime and Darcet, Timothée and Moutakanni, Theo and Vo, Huy V. and Szafraniec, Marc and Khalidov, Vasil and Fernandez, Pierre and Haziza, Daniel and Massa, Francisco and El-Nouby, Alaaeldin and Howes, Russell and Huang, Po-Yao and Xu, Hu and Sharma, Vasu and Li, Shang-Wen and Galuba, Wojciech and Rabbat, Mike and Assran, Mido and Ballas, Nicolas and Synnaeve, Gabriel and Misra, Ishan and Jegou, Herve and Mairal, Julien and Labatut, Patrick and Joulin, Armand and Bojanowski, Piotr},
  journal={arXiv:2304.07193},
}
```

# GN + WS

> [Weight Standardization](https://arxiv.org/abs/1903.10520)

Batch Normalization (BN) has become an out-of-box technique to improve deep network training. However, its effectiveness is limited for micro-batch training, i.e., each GPU typically has only 1-2 images for training, which is inevitable for many computer vision tasks, e.g., object detection and semantic segmentation, constrained by memory consumption. To address this issue, we propose Weight Standardization (WS) and Batch-Channel Normalization (BCN) to bring two success factors of BN into micro-batch training: 1) the smoothing effects on the loss landscape and 2) the ability to avoid harmful elimination singularities along the training trajectory. WS standardizes the weights in convolutional layers to smooth the loss landscape by reducing the Lipschitz constants of the loss and the gradients; BCN combines batch and channel normalizations and leverages estimated statistics of the activations in convolutional layers to keep networks away from elimination singularities. We validate WS and BCN on comprehensive computer vision tasks, including image classification, object detection, instance segmentation, video recognition and semantic segmentation. All experimental results consistently show that WS and BCN improve micro-batch training significantly. Moreover, using WS and BCN with micro-batch training is even able to match or outperform the performances of BN with large-batch training.

<img src="https://user-images.githubusercontent.com/40661020/143889309-b1f3487d-abfe-4615-befc-79384bc8b65c.png"/ height="200">
</div>

Faster R-CNN

|    Backbone     |  Style  | Normalization | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                           Config                            |                                                                                                                                                                               Download                                                                                                                                                                                |

|    R-50-FPN     | pytorch |     GN+WS     |   1x    |   5.9    |      11.7      |  39.7  |    -    |    [config](./faster-rcnn_r50_fpn_gn-ws-all_1x_coco.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/faster_rcnn_r50_fpn_gn_ws-all_1x_coco/faster_rcnn_r50_fpn_gn_ws-all_1x_coco_20200130-613d9fe2.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/faster_rcnn_r50_fpn_gn_ws-all_1x_coco/faster_rcnn_r50_fpn_gn_ws-all_1x_coco_20200130_210936.log.json)               |

|    R-101-FPN    | pytorch |     GN+WS     |   1x    |   8.9    |      9.0       |  41.7  |    -    |    [config](./faster-rcnn_r101_fpn_gn-ws-all_1x_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/faster_rcnn_r101_fpn_gn_ws-all_1x_coco/faster_rcnn_r101_fpn_gn_ws-all_1x_coco_20200205-a93b0d75.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/faster_rcnn_r101_fpn_gn_ws-all_1x_coco/faster_rcnn_r101_fpn_gn_ws-all_1x_coco_20200205_232146.log.json)             |

| X-50-32x4d-FPN  | pytorch |     GN+WS     |   1x    |   7.0    |      10.3      |  40.7  |    -    | [config](./faster-rcnn_x50-32x4d_fpn_gn-ws-all_1x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/faster_rcnn_x50_32x4d_fpn_gn_ws-all_1x_coco/faster_rcnn_x50_32x4d_fpn_gn_ws-all_1x_coco_20200203-839c5d9d.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/faster_rcnn_x50_32x4d_fpn_gn_ws-all_1x_coco/faster_rcnn_x50_32x4d_fpn_gn_ws-all_1x_coco_20200203_220113.log.json)   |

| X-101-32x4d-FPN | pytorch |     GN+WS     |   1x    |   10.8   |      7.6       |  42.1  |    -    | [config](./faster-rcnn_x101-32x4d_fpn_gn-ws-all_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/faster_rcnn_x101_32x4d_fpn_gn_ws-all_1x_coco/faster_rcnn_x101_32x4d_fpn_gn_ws-all_1x_coco_20200212-27da1bc2.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/faster_rcnn_x101_32x4d_fpn_gn_ws-all_1x_coco/faster_rcnn_x101_32x4d_fpn_gn_ws-all_1x_coco_20200212_195302.log.json) |

Mask R-CNN

Mask R-CNN

|    Backbone     |  Style  | Normalization |  Lr schd  | Mem (GB) | Inf time (fps) | box AP | mask AP |                              Config                              |                                                                                                                                                                                         Download                                                                                                                                                                                          |

|    R-50-FPN     | pytorch |     GN+WS     |    2x     |   7.3    |      10.5      |  40.6  |  36.6   |        [config](./mask-rcnn_r50_fpn_gn-ws-all_2x_coco.py)        |                             [model](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/mask_rcnn_r50_fpn_gn_ws-all_2x_coco/mask_rcnn_r50_fpn_gn_ws-all_2x_coco_20200226-16acb762.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/mask_rcnn_r50_fpn_gn_ws-all_2x_coco/mask_rcnn_r50_fpn_gn_ws-all_2x_coco_20200226_062128.log.json)                             |

|    R-101-FPN    | pytorch |     GN+WS     |    2x     |   10.3   |      8.6       |  42.0  |  37.7   |       [config](./mask-rcnn_r101_fpn_gn-ws-all_2x_coco.py)        |                           [model](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/mask_rcnn_r101_fpn_gn_ws-all_2x_coco/mask_rcnn_r101_fpn_gn_ws-all_2x_coco_20200212-ea357cd9.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/mask_rcnn_r101_fpn_gn_ws-all_2x_coco/mask_rcnn_r101_fpn_gn_ws-all_2x_coco_20200212_213627.log.json)                           |

| X-50-32x4d-FPN  | pytorch |     GN+WS     |    2x     |   8.4    |      9.3       |  41.1  |  37.0   |     [config](./mask-rcnn_x50-32x4d_fpn_gn-ws-all_2x_coco.py)     |                 [model](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/mask_rcnn_x50_32x4d_fpn_gn_ws-all_2x_coco/mask_rcnn_x50_32x4d_fpn_gn_ws-all_2x_coco_20200216-649fdb6f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/mask_rcnn_x50_32x4d_fpn_gn_ws-all_2x_coco/mask_rcnn_x50_32x4d_fpn_gn_ws-all_2x_coco_20200216_201500.log.json)                 |

| X-101-32x4d-FPN | pytorch |     GN+WS     |    2x     |   12.2   |      7.1       |  42.1  |  37.9   |    [config](./mask-rcnn_x101-32x4d_fpn_gn-ws-all_2x_coco.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/mask_rcnn_x101_32x4d_fpn_gn_ws-all_2x_coco/mask_rcnn_x101_32x4d_fpn_gn_ws-all_2x_coco_20200319-33fb95b5.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/mask_rcnn_x101_32x4d_fpn_gn_ws-all_2x_coco/mask_rcnn_x101_32x4d_fpn_gn_ws-all_2x_coco_20200319_104101.log.json)               |

|    R-50-FPN     | pytorch |     GN+WS     | 20-23-24e |   7.3    |       -        |  41.1  |  37.1   |    [config](./mask-rcnn_r50_fpn_gn-ws-all_20-23-24e_coco.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/mask_rcnn_r50_fpn_gn_ws-all_20_23_24e_coco/mask_rcnn_r50_fpn_gn_ws-all_20_23_24e_coco_20200213-487d1283.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/mask_rcnn_r50_fpn_gn_ws-all_20_23_24e_coco/mask_rcnn_r50_fpn_gn_ws-all_20_23_24e_coco_20200213_035123.log.json)               |

|    R-101-FPN    | pytorch |     GN+WS     | 20-23-24e |   10.3   |       -        |  43.1  |  38.6   |    [config](./mask-rcnn_r101_fpn_gn-ws-all_20-23-24e_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/mask_rcnn_r101_fpn_gn_ws-all_20_23_24e_coco/mask_rcnn_r101_fpn_gn_ws-all_20_23_24e_coco_20200213-57b5a50f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/mask_rcnn_r101_fpn_gn_ws-all_20_23_24e_coco/mask_rcnn_r101_fpn_gn_ws-all_20_23_24e_coco_20200213_130142.log.json)             |

| X-50-32x4d-FPN  | pytorch |     GN+WS     | 20-23-24e |   8.4    |       -        |  42.1  |  38.0   | [config](./mask-rcnn_x50-32x4d_fpn_gn-ws-all_20-23-24e_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/mask_rcnn_x50_32x4d_fpn_gn_ws-all_20_23_24e_coco/mask_rcnn_x50_32x4d_fpn_gn_ws-all_20_23_24e_coco_20200226-969bcb2c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/mask_rcnn_x50_32x4d_fpn_gn_ws-all_20_23_24e_coco/mask_rcnn_x50_32x4d_fpn_gn_ws-all_20_23_24e_coco_20200226_093732.log.json)   |

| X-101-32x4d-FPN | pytorch |     GN+WS     | 20-23-24e |   12.2   |       -        |  42.7  |  38.5   | [config](./mask-rcnn_x101-32x4d_fpn_gn-ws-all_20-23-24e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/mask_rcnn_x101_32x4d_fpn_gn_ws-all_20_23_24e_coco/mask_rcnn_x101_32x4d_fpn_gn_ws-all_20_23_24e_coco_20200316-e6cd35ef.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gn%2Bws/mask_rcnn_x101_32x4d_fpn_gn_ws-all_20_23_24e_coco/mask_rcnn_x101_32x4d_fpn_gn_ws-all_20_23_24e_coco_20200316_013741.log.json) |

Note:

- GN+WS requires about 5% more memory than GN, and it is only 5% slower than GN.
- In the paper, a 20-23-24e lr schedule is used instead of 2x.
- The X-50-GN and X-101-GN pretrained models are also shared by the authors.

```latex
@article{weightstandardization,
  author    = {Siyuan Qiao and Huiyu Wang and Chenxi Liu and Wei Shen and Alan Yuille},
  title     = {Weight Standardization},

```latex
  journal   = {arXiv preprint arXiv:1903.10520},
}
```

# 为 CycleGAN 准备未配对数据集

```bibtex
}
```

您可以从[此处](https://people.eecs.berkeley.edu/~taesung_park/CycleGAN/datasets/)下载未配对的数据集。然后，您需要解压缩并移动相应的数据集以遵循如上所示的文件夹结构。数据集已经由原作者准备好了。

```text
mmagic
├── mmagic
├── tools
├── data
```

}
```

<details>

```bibtex
}
```

| [pose_hrnetv2_w18_dark](/configs/hand_2d_keypoint/topdown_heatmap/coco_wholebody_hand/td-hm_hrnetv2-w18_dark-8xb32-210e_coco-wholebody-hand-256x256.py) |  256x256   |  0.814  | 0.840 | 4.37 | [ckpt](https://download.openmmlab.com/mmpose/hand/dark/hrnetv2_w18_coco_wholebody_hand_256x256_dark-a9228c9c_20210908.pth) | [log](https://download.openmmlab.com/mmpose/hand/dark/hrnetv2_w18_coco_wholebody_hand_256x256_dark_20210908.log.json) |

# Associative embedding: End-to-end learning for joint detection and grouping (AE)

<details>
<summary align="right"><a href="https://arxiv.org/abs/1611.05424">Associative Embedding (NIPS'2017)</a></summary>

```bibtex
@inproceedings{newell2017associative,
  title={Associative embedding: End-to-end learning for joint detection and grouping},
  author={Newell, Alejandro and Huang, Zhiao and Deng, Jia},
  pages={2277--2287},
}
```

booktitle={Advances in neural information processing systems},
}
```

<img src="https://user-images.githubusercontent.com/15977946/146514181-84f22623-6b73-4656-89b8-9e7f551e9cc0.png">
</div>

# DINO

> [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://arxiv.org/abs/2203.03605)

We present DINO (DETR with Improved deNoising anchOr boxes), a state-of-the-art end-to-end object detector. DINO improves over previous DETR-like models in performance and efficiency by using a contrastive way for denoising training, a mixed query selection method for anchor initialization, and a look forward twice scheme for box prediction. DINO achieves 49.4AP in 12 epochs and 51.3AP in 24 epochs on COCO with a ResNet-50 backbone and multi-scale features, yielding a significant improvement of +6.0AP and +2.7AP, respectively, compared to DN-DETR, the previous best DETR-like model. DINO scales well in both model size and data size. Without bells and whistles, after pre-training on the Objects365 dataset with a SwinL backbone, DINO obtains the best results on both COCO val2017 (63.2AP) and test-dev (63.3AP). Compared to other models on the leaderboard, DINO significantly reduces its model size and pre-training data size while achieving better results.

<img src="https://user-images.githubusercontent.com/79644233/207820666-099e6a85-59c4-45d6-a687-91b5781d11cd.png"/>
</div>

| Backbone |    Model    | Lr schd | box AP |                     Config                      |                                                                                                                                                     Download                                                                                                                                                      |
| :------: | :---------: | :-----: | :----: | :---------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

|   R-50   | DINO-4scale |   12e   |  49.0  |  [config](./dino-4scale_r50_8xb2-12e_coco.py)   |    [model](https://download.openmmlab.com/mmdetection/v3.0/dino/dino-4scale_r50_8xb2-12e_coco/dino-4scale_r50_8xb2-12e_coco_20221202_182705-55b2bba2.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/dino/dino-4scale_r50_8xb2-12e_coco/dino-4scale_r50_8xb2-12e_coco_20221202_182705.log.json)     |
|  Swin-L  | DINO-5scale |   12e   |  57.2  | [config](./dino-5scale_swin-l_8xb2-12e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/dino/dino-5scale_swin-l_8xb2-12e_coco/dino-5scale_swin-l_8xb2-12e_coco_20230228_072924-a654145f.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/dino/dino-5scale_swin-l_8xb2-12e_coco/dino-5scale_swin-l_8xb2-12e_coco_20230228_072924.log) |

|  Swin-L  | DINO-5scale |   36e   |  58.4  | [config](./dino-5scale_swin-l_8xb2-36e_coco.py) |                                  [model](https://github.com/RistoranteRist/mmlab-weights/releases/download/dino-swinl/dino-5scale_swin-l_8xb2-36e_coco-5486e051.pth) \| [log](https://github.com/RistoranteRist/mmlab-weights/releases/download/dino-swinl/20230307_032359.log)                                   |

### NOTE

The performance is unstable. `DINO-4scale` with `R-50` may fluctuate about 0.4 mAP.

We provide the config files for DINO: [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://arxiv.org/abs/2203.03605).

```latex
@misc{zhang2022dino,
  title={DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection},
  author={Hao Zhang and Feng Li and Shilong Liu and Lei Zhang and Hang Su and Jun Zhu and Lionel M. Ni and Heung-Yeung Shum},
  eprint={2203.03605},

author={Hao Zhang and Feng Li and Shilong Liu and Lei Zhang and Hang Su and Jun Zhu and Lionel M. Ni and Heung-Yeung Shum},
  primaryClass={cs.CV}}
```

# PAA

> [Probabilistic Anchor Assignment with IoU Prediction for Object Detection](https://arxiv.org/abs/2007.08103)

In object detection, determining which anchors to assign as positive or negative samples, known as anchor assignment, has been revealed as a core procedure that can significantly affect a model's performance. In this paper we propose a novel anchor assignment strategy that adaptively separates anchors into positive and negative samples for a ground truth bounding box according to the model's learning status such that it is able to reason about the separation in a probabilistic manner. To do so we first calculate the scores of anchors conditioned on the model and fit a probability distribution to these scores. The model is then trained with anchors separated into positive and negative samples according to their probabilities. Moreover, we investigate the gap between the training and testing objectives and propose to predict the Intersection-over-Unions of detected boxes as a measure of localization quality to reduce the discrepancy. The combined score of classification and localization qualities serving as a box selection metric in non-maximum suppression well aligns with the proposed anchor assignment strategy and leads significant performance improvements. The proposed methods only add a single convolutional layer to RetinaNet baseline and does not require multiple anchors per location, so are efficient. Experimental results verify the effectiveness of the proposed methods. Especially, our models set new records for single-stage detectors on MS COCO test-dev dataset with various backbones.

<img src="https://user-images.githubusercontent.com/40661020/143968195-519a116a-de29-437e-b4c8-30aef43dcb15.png"/>
</div>

ECCV 2020 paper for Probabilistic Anchor Assignment with IoU
Prediction for Object Detection.

| Backbone  | Lr schd | Mem (GB) | Score voting | box AP |                 Config                 |                                                                                                                                               Download                                                                                                                                               |

| :-------: | :-----: | :------: | :----------: | :----: | :------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| R-50-FPN  |   12e   |   3.7    |     True     |  40.4  |   [config](./paa_r50_fpn_1x_coco.py)   |                     [model](https://download.openmmlab.com/mmdetection/v2.0/paa/paa_r50_fpn_1x_coco/paa_r50_fpn_1x_coco_20200821-936edec3.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/paa/paa_r50_fpn_1x_coco/paa_r50_fpn_1x_coco_20200821-936edec3.log.json)                      |

| R-50-FPN  |   12e   |   3.7    |    False     |  40.2  |                   -                    |                                                                                                                                                                                                                                                                                                      |
| R-50-FPN  |   18e   |   3.7    |     True     |  41.4  |  [config](./paa_r50_fpn_1.5x_coco.py)  |                 [model](https://download.openmmlab.com/mmdetection/v2.0/paa/paa_r50_fpn_1.5x_coco/paa_r50_fpn_1.5x_coco_20200823-805d6078.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/paa/paa_r50_fpn_1.5x_coco/paa_r50_fpn_1.5x_coco_20200823-805d6078.log.json)                  |

| R-50-FPN  |   18e   |   3.7    |    False     |  41.2  |                   -                    |                                                                                                                                                                                                                                                                                                      |
| R-50-FPN  |   24e   |   3.7    |     True     |  41.6  |   [config](./paa_r50_fpn_2x_coco.py)   |                     [model](https://download.openmmlab.com/mmdetection/v2.0/paa/paa_r50_fpn_2x_coco/paa_r50_fpn_2x_coco_20200821-c98bfc4e.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/paa/paa_r50_fpn_2x_coco/paa_r50_fpn_2x_coco_20200821-c98bfc4e.log.json)                      |

| R-50-FPN  |   36e   |   3.7    |     True     |  43.3  | [config](./paa_r50_fpn_ms-3x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/paa/paa_r50_fpn_mstrain_3x_coco/paa_r50_fpn_mstrain_3x_coco_20210121_145722-06a6880b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/paa/paa_r50_fpn_mstrain_3x_coco/paa_r50_fpn_mstrain_3x_coco_20210121_145722.log.json)   |
| R-101-FPN |   12e   |   6.2    |     True     |  42.6  |  [config](./paa_r101_fpn_1x_coco.py)   |                   [model](https://download.openmmlab.com/mmdetection/v2.0/paa/paa_r101_fpn_1x_coco/paa_r101_fpn_1x_coco_20200821-0a1825a4.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/paa/paa_r101_fpn_1x_coco/paa_r101_fpn_1x_coco_20200821-0a1825a4.log.json)                    |

| R-101-FPN |   12e   |   6.2    |    False     |  42.4  |                   -                    |                                                                                                                                                                                                                                                                                                      |
| R-101-FPN |   24e   |   6.2    |     True     |  43.5  |  [config](./paa_r101_fpn_2x_coco.py)   |                   [model](https://download.openmmlab.com/mmdetection/v2.0/paa/paa_r101_fpn_2x_coco/paa_r101_fpn_2x_coco_20200821-6829f96b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/paa/paa_r101_fpn_2x_coco/paa_r101_fpn_2x_coco_20200821-6829f96b.log.json)                    |

| R-101-FPN |   36e   |   6.2    |     True     |  45.1  | [config](./paa_r101_fpn_ms-3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/paa/paa_r101_fpn_mstrain_3x_coco/paa_r101_fpn_mstrain_3x_coco_20210122_084202-83250d22.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/paa/paa_r101_fpn_mstrain_3x_coco/paa_r101_fpn_mstrain_3x_coco_20210122_084202.log.json) |

**Note**:

1. We find that the performance is unstable with 1x setting and may fluctuate by about 0.2 mAP. We report the best results.

```latex
@inproceedings{paa-eccv2020,
  title={Probabilistic Anchor Assignment with IoU Prediction for Object Detection},
  author={Kim, Kang and Lee, Hee Seok},
  booktitle = {ECCV},
}
```

# 开始：安装和运行 MMSeg

## 预备知识

本教程中，我们将会演示如何使用 PyTorch 准备环境。

MMSegmentation 可以在 Linux, Windows 和 macOS 系统上运行，并且需要安装 Python 3.7+, CUDA 10.2+ 和 PyTorch 1.8+

**注意:**
如果您已经安装了 PyTorch, 可以跳过该部分，直接到[下一小节](##安装)。否则，您可以按照以下步骤操作。

**步骤 0.** 从[官方网站](https://docs.conda.io/en/latest/miniconda.html)下载并安装 Miniconda

**步骤 1.** 创建一个 conda 环境，并激活

```shell
```

**Step 2.** 参考 [official instructions](https://pytorch.org/get-started/locally/) 安装 PyTorch

在 GPU 平台上：

```shell
```

在 CPU 平台上

```shell
```

## 安装

我们建议用户遵循我们的最佳实践来安装 MMSegmentation 。但是整个过程是高度自定义的。更多信息请参见[自定义安装](##自定义安装)部分。

### 最佳实践

**步骤 0.** 使用 [MIM](https://github.com/open-mmlab/mim) 安装 [MMCV](https://github.com/open-mmlab/mmcv)

```shell
```

**步骤 1.** 安装 MMSegmentation

```shell
```

情况 a: 如果您想立刻开发和运行 mmsegmentation，您可通过源码安装：

```shell
# '-v' 表示详细模式，更多的输出
# '-e' 表示以可编辑模式安装工程，
# 因此对代码所做的任何修改都生效，无需重新安装
```

情况 b: 如果您把 mmsegmentation 作为依赖库或者第三方库，可以通过 pip 安装：

```shell
```

### 验证是否安装成功

为了验证 MMSegmentation 是否正确安装，我们提供了一些示例代码来运行一个推理 demo 。

**步骤 1.** 下载配置文件和模型文件

```shell
```

该下载过程可能需要花费几分钟，这取决于您的网络环境。当下载结束，您将看到以下两个文件在您当前工作目录：`pspnet_r50-d8_4xb2-40k_cityscapes-512x1024.py` 和 `pspnet_r50-d8_512x1024_40k_cityscapes_20200605_003338-2966598c.pth`

**步骤 2.** 验证推理 demo

选项 (a). 如果您通过源码安装了 mmsegmentation，运行以下命令即可：

```shell

```shell
```

您将在当前文件夹中看到一个新图像 `result.jpg`，其中所有目标都覆盖了分割 mask

选项 (b). 如果您通过 pip 安装 mmsegmentation, 打开您的 python 解释器，复制粘贴以下代码：

```python

# 根据配置文件和模型文件建立模型

# 在单张图像上测试并可视化
img = 'demo/demo.png'  # or img = mmcv.imread(img), 这样仅需下载一次
# 在新的窗口可视化结果
# 或者将可视化结果保存到图像文件夹中
# 您可以修改分割 map 的透明度 (0, 1].

# 在一段视频上测试并可视化分割结果
```

您可以修改上面的代码来测试单个图像或视频，这两个选项都可以验证安装是否成功。

### 自定义安装

当安装 PyTorch 的时候，您需要指定 CUDA 的版本， 如果您不确定选择哪个版本，请遵循我们的建议：

- 对于基于 Ampere 的 NVIDIA GPUs, 例如 GeForce 30 系列和 NVIDIA A100, 必须要求是 CUDA 11.
- 对于更老的 NVIDIA GPUs, CUDA 11 is backward compatible, but CUDA 10.2 提供了更好的兼容性，以及更加的轻量化

请确保 GPU 驱动满足最小的版本需求。详情请参考这个[表格](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cuda-major-component-versions__table-cuda-toolkit-driver-versions)

**注意:**

**注意:**
如果您按照我们的最佳实践，安装 CUDA 运行库就足够了，因为不需要 CUDA 代码在本地编译。 但是如果您希望从源码编译 MMCV 或者需要开发其他的 CUDA 算子，您需要从 NVIDIA 的[官网](https://developer.nvidia.com/cuda-downloads)安装完整的 CUDA 工具，同时它的版本需要与 PyTorch 的 CUDA 版本匹配。即 `conda install` 命令中指定的 cudatoolkit 版本。

#### 不使用 MIM 安装 MMCV

MMCV 包含 C++ 和 CUDA 扩展，因此与 PyTorch 的依赖方式比较复杂。MIM 自动解决了这种依赖关系，使安装更容易。然而，MIM 也并不是必须的。

为了使用 pip 而不是 MIM 安装 MMCV, 请参考 [MMCV 安装指南](https://mmcv.readthedocs.io/en/latest/get_started/installation.html). 这需要手动指定一个基于 PyTorch 版本及其 CUDA 版本的 find-url.

例如，以下命令可为 PyTorch 1.10.x and CUDA 11.3 安装 mmcv==2.0.0

```shell
```

#### 在仅有 CPU 的平台安装

MMSegmentation 可以在仅有 CPU 的版本上运行。在 CPU 模式，您可以训练（需要 MMCV 版本 >= 2.0.0），测试和推理模型。

#### 在 Google Colab 上安装

```

[Google Colab](https://research.google.com/) 通常已经安装了 PyTorch，因此我们仅需要通过以下命令安装 MMCV 和 MMSegmentation。

**步骤 1.** 使用 [MIM](https://github.com/open-mmlab/mim) 安装 [MMCV](https://github.com/open-mmlab/mmcv)

```shell
```

**Step 2.** 通过源码安装 MMSegmentation

```shell
```

**Step 3.** 验证

```python
# 示例输出: 1.0.0
```

**注意:**
在 Jupyter 中, 感叹号 `!` 用于调用外部可执行命令，`%cd` 是一个 [magic command](https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-cd) 可以改变当前 python 的工作目录。

### 通过 Docker 使用 MMSegmentation

**注意:**

我们提供了一个 [Dockerfile](https://github.com/open-mmlab/mmsegmentation/blob/master/docker/Dockerfile) 来建立映像。确保您的 [docker 版本](https://docs.docker.com/engine/install/) >=19.03.

```shell
# 通过 PyTorch 1.11, CUDA 11.3 建立映像
# 如果您使用其他版本，修改 Dockerfile 即可
```

运行：

```shell
```

### 可选依赖

#### 安装 GDAL

[GDAL](https://gdal.org/) 是一个用于栅格和矢量地理空间数据格式的转换库。安装 GDAL 可以读取复杂格式和极大的遥感图像。

```shell
```

## 问题解答

如果您在安装过程中遇到了其他问题，请第一时间查阅 [FAQ](notes/faq.md) 文件。如果没有找到答案，您也可以在 GitHub 上提出 [issue](https://github.com/open-mmlab/mmsegmentation/issues/new/choose)

# oCLIP

> [Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spotting](https://www.ecva.net/papers/eccv_2022/papers_ECCV/papers/136880282.pdf)

Recently, Vision-Language Pre-training (VLP) techniques have greatly benefited various vision-language tasks by jointly learning visual and textual representations, which intuitively helps in Optical Character Recognition (OCR) tasks due to the rich visual and textual information in scene text images. However, these methods cannot well cope with OCR tasks because of the difficulty in both instance-level text encoding and image-text pair acquisition (i.e. images and captured texts in them). This paper presents a weakly supervised pre-training method, oCLIP, which can acquire effective scene text representations by jointly learning and aligning visual and textual information. Our network consists of an image encoder and a character-aware text encoder that extract visual and textual features, respectively, as well as a visual-textual decoder that models the interaction among textual and visual features for learning effective scene text representations. With the learning of textual features, the pre-trained model can attend texts in images well with character awareness. Besides, these designs enable the learning from weakly annotated texts (i.e. partial texts in images without text bounding boxes) which mitigates the data annotation constraint greatly. Experiments over the weakly annotated images in ICDAR2019-LSVT show that our pre-trained model improves F-score by +2.5% and +4.8% while transferring its weights to other text detection and spotting networks, respectively. In addition, the proposed method outperforms existing pre-training techniques consistently across multiple public datasets (e.g., +3.2% and +1.3% for Total-Text and CTW1500).

<img src="https://user-images.githubusercontent.com/24622904/199475057-aa688422-518d-4d7a-86fc-1be0cc1b5dc6.png"/>
</div>

## Models

| Backbone  | Pre-train Data |                                       Model                                       |
| :-------: | :------------: | :-------------------------------------------------------------------------------: |
| ResNet-50 |   SynthText    | [Link](https://download.openmmlab.com/mmocr/backbone/resnet50-oclip-7ba0c533.pth) |

```{note}
The model is converted from the official [oCLIP](https://github.com/bytedance/oclip.git).
```

## Supported Text Detection Models

```{note}
```

|           | [DBNet](https://mmocr.readthedocs.io/en/dev-1.x/textdet_models.html#dbnet) | [DBNet++](https://mmocr.readthedocs.io/en/dev-1.x/textdet_models.html#dbnetpp) | [FCENet](https://mmocr.readthedocs.io/en/dev-1.x/textdet_models.html#fcenet) | [TextSnake](https://mmocr.readthedocs.io/en/dev-1.x/textdet_models.html#fcenet) | [PSENet](https://mmocr.readthedocs.io/en/dev-1.x/textdet_models.html#psenet) | [DRRG](https://mmocr.readthedocs.io/en/dev-1.x/textdet_models.html#drrg) | [Mask R-CNN](https://mmocr.readthedocs.io/en/dev-1.x/textdet_models.html#mask-r-cnn) |

| ICDAR2015 |                                     ✓                                      |                                       ✓                                        |                                      ✓                                       |                                                                                 |                                      ✓                                       |                                                                          |                                          ✓                                           |

|  CTW1500  |                                                                            |                                                                                |                                      ✓                                       |                                        ✓                                        |                                      ✓                                       |                                    ✓                                     |                                          ✓                                           |

```bibtex
@article{xue2022language,
  title={Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spotting},
  author={Xue, Chuhui and Zhang, Wenqing and Hao, Yu and Lu, Shijian and Torr, Philip and Bai, Song},
  journal={Proceedings of the European Conference on Computer Vision (ECCV)},
}
```

</div>

|      Model      | Input Size |  AP   |  AR   |                Details and Download                 |

| :-------------: | :--------: | :---: | :---: | :-------------------------------------------------: |
|    ViTPose-h    |  256x192   | 0.790 | 0.840 |      [vitpose_coco.md](./coco/vitpose_coco.md)      |
|  HRNet-w48+UDP  |  256x192   | 0.768 | 0.817 |    [hrnet_udp_coco.md](./coco/hrnet_udp_coco.md)    |
|   MSPN 4-stg    |  256x192   | 0.765 | 0.826 |         [mspn_coco.md](./coco/mspn_coco.md)         |
| HRNet-w48+Dark  |  256x192   | 0.764 | 0.814 |   [hrnet_dark_coco.md](./coco/hrnet_dark_coco.md)   |
|    HRNet-w48    |  256x192   | 0.756 | 0.809 |        [hrnet_coco.md](./coco/hrnet_coco.md)        |
|   HRFormer-B    |  256x192   | 0.754 | 0.807 |     [hrformer_coco.md](./coco/hrformer_coco.md)     |

|    RSN-50-3x    |  256x192   | 0.750 | 0.814 |          [rsn_coco.md](./coco/rsn_coco.md)          |
|    CSPNeXt-l    |  256x192   | 0.750 | 0.800 |  [cspnext_udp_coco.md](./coco/cspnext_udp_coco.md)  |
|    HRNet-w32    |  256x192   | 0.749 | 0.804 |        [hrnet_coco.md](./coco/hrnet_coco.md)        |
|     Swin-L      |  256x192   | 0.743 | 0.798 |         [swin_coco.md](./coco/swin_coco.md)         |
|    ViTPose-s    |  256x192   | 0.739 | 0.792 |      [vitpose_coco.md](./coco/vitpose_coco.md)      |
|   HRFormer-S    |  256x192   | 0.738 | 0.793 |     [hrformer_coco.md](./coco/hrformer_coco.md)     |
|     Swin-B      |  256x192   | 0.737 | 0.794 |         [swin_coco.md](./coco/swin_coco.md)         |
|  SEResNet-101   |  256x192   | 0.734 | 0.790 |     [seresnet_coco.md](./coco/seresnet_coco.md)     |

|    SCNet-101    |  256x192   | 0.733 | 0.789 |        [scnet_coco.md](./coco/scnet_coco.md)        |
| ResNet-101+Dark |  256x192   | 0.733 | 0.786 |  [resnet_dark_coco.md](./coco/resnet_dark_coco.md)  |
|    CSPNeXt-m    |  256x192   | 0.732 | 0.785 |  [cspnext_udp_coco.md](./coco/cspnext_udp_coco.md)  |
|  ResNetV1d-101  |  256x192   | 0.732 | 0.785 |    [resnetv1d_coco.md](./coco/resnetv1d_coco.md)    |
|   SEResNet-50   |  256x192   | 0.729 | 0.784 |     [seresnet_coco.md](./coco/seresnet_coco.md)     |
|    SCNet-50     |  256x192   | 0.728 | 0.784 |        [scnet_coco.md](./coco/scnet_coco.md)        |
|   ResNet-101    |  256x192   | 0.726 | 0.783 |       [resnet_coco.md](./coco/resnet_coco.md)       |
|   ResNeXt-101   |  256x192   | 0.726 | 0.781 |      [resnext_coco.md](./coco/resnext_coco.md)      |

|  HourglassNet   |  256x256   | 0.726 | 0.780 |    [hourglass_coco.md](./coco/hourglass_coco.md)    |
|   ResNeSt-101   |  256x192   | 0.725 | 0.781 |      [resnest_coco.md](./coco/resnest_coco.md)      |
|     RSN-50      |  256x192   | 0.724 | 0.790 |          [rsn_coco.md](./coco/rsn_coco.md)          |
|     Swin-T      |  256x192   | 0.724 | 0.782 |         [swin_coco.md](./coco/swin_coco.md)         |
|   MSPN 1-stg    |  256x192   | 0.723 | 0.788 |         [mspn_coco.md](./coco/mspn_coco.md)         |
|  ResNetV1d-50   |  256x192   | 0.722 | 0.777 |    [resnetv1d_coco.md](./coco/resnetv1d_coco.md)    |
|   ResNeSt-50    |  256x192   | 0.720 | 0.775 |      [resnest_coco.md](./coco/resnest_coco.md)      |
|    ResNet-50    |  256x192   | 0.718 | 0.774 |       [resnet_coco.md](./coco/resnet_coco.md)       |

|   ResNeXt-50    |  256x192   | 0.715 | 0.771 |      [resnext_coco.md](./coco/resnext_coco.md)      |
|      PVT-S      |  256x192   | 0.714 | 0.773 |          [pvt_coco.md](./coco/pvt_coco.md)          |
|    CSPNeXt-s    |  256x192   | 0.697 | 0.753 |  [cspnext_udp_coco.md](./coco/cspnext_udp_coco.md)  |
|  LiteHRNet-30   |  256x192   | 0.676 | 0.736 |    [litehrnet_coco.md](./coco/litehrnet_coco.md)    |
|  CSPNeXt-tiny   |  256x192   | 0.665 | 0.723 |  [cspnext_udp_coco.md](./coco/cspnext_udp_coco.md)  |
|  MobileNet-v2   |  256x192   | 0.648 | 0.709 |  [mobilenetv2_coco.md](./coco/mobilenetv2_coco.md)  |
|  LiteHRNet-18   |  256x192   | 0.642 | 0.705 |    [litehrnet_coco.md](./coco/litehrnet_coco.md)    |
|       CPM       |  256x192   | 0.627 | 0.689 |          [cpm_coco.md](./coco/cpm_coco.md)          |

|  ShuffleNet-v2  |  256x192   | 0.602 | 0.668 | [shufflenetv2_coco.md](./coco/shufflenetv2_coco.md) |
|  ShuffleNet-v1  |  256x192   | 0.587 | 0.654 | [shufflenetv1_coco.md](./coco/shufflenetv1_coco.md) |
|     AlexNet     |  256x192   | 0.448 | 0.521 |      [alexnet_coco.md](./coco/alexnet_coco.md)      |

### MPII Dataset

|     Model      | Input Size | PCKh@0.5 | PCKh@0.1 |                Details and Download                 |
| :------------: | :--------: | :------: | :------: | :-------------------------------------------------: |
| HRNet-w48+Dark |  256x256   |  0.905   |  0.360   |   [hrnet_dark_mpii.md](./mpii/hrnet_dark_mpii.md)   |
|   HRNet-w48    |  256x256   |  0.902   |  0.303   |     [hrnet_mpii.md](./mpii/cspnext_udp_mpii.md)     |
|   HRNet-w48    |  256x256   |  0.901   |  0.337   |        [hrnet_mpii.md](./mpii/hrnet_mpii.md)        |

|   HRNet-w32    |  256x256   |  0.900   |  0.334   |        [hrnet_mpii.md](./mpii/hrnet_mpii.md)        |
|  HourglassNet  |  256x256   |  0.889   |  0.317   |    [hourglass_mpii.md](./mpii/hourglass_mpii.md)    |
|   ResNet-152   |  256x256   |  0.889   |  0.303   |       [resnet_mpii.md](./mpii/resnet_mpii.md)       |
| ResNetV1d-152  |  256x256   |  0.888   |  0.300   |    [resnetv1d_mpii.md](./mpii/resnetv1d_mpii.md)    |
|    SCNet-50    |  256x256   |  0.888   |  0.290   |        [scnet_mpii.md](./mpii/scnet_mpii.md)        |
|  ResNeXt-152   |  256x256   |  0.887   |  0.294   |      [resnext_mpii.md](./mpii/resnext_mpii.md)      |
|  SEResNet-50   |  256x256   |  0.884   |  0.292   |     [seresnet_mpii.md](./mpii/seresnet_mpii.md)     |
|   ResNet-50    |  256x256   |  0.882   |  0.286   |       [resnet_mpii.md](./mpii/resnet_mpii.md)       |

|  ResNetV1d-50  |  256x256   |  0.881   |  0.290   |    [resnetv1d_mpii.md](./mpii/resnetv1d_mpii.md)    |
|      CPM       | 368x368\*  |  0.876   |  0.285   |          [cpm_mpii.md](./mpii/cpm_mpii.md)          |
|  LiteHRNet-30  |  256x256   |  0.869   |  0.271   |    [litehrnet_mpii.md](./mpii/litehrnet_mpii.md)    |
|  LiteHRNet-18  |  256x256   |  0.859   |  0.260   |    [litehrnet_mpii.md](./mpii/litehrnet_mpii.md)    |
|  MobileNet-v2  |  256x256   |  0.854   |  0.234   |  [mobilenetv2_mpii.md](./mpii/mobilenetv2_mpii.md)  |
| ShuffleNet-v2  |  256x256   |  0.828   |  0.205   | [shufflenetv2_mpii.md](./mpii/shufflenetv2_mpii.md) |
| ShuffleNet-v1  |  256x256   |  0.824   |  0.195   | [shufflenetv1_mpii.md](./mpii/shufflenetv1_mpii.md) |

### CrowdPose Dataset

|   Model    | Input Size |  AP   |  AR   |                    Details and Download                    |
| :--------: | :--------: | :---: | :---: | :--------------------------------------------------------: |
| HRNet-w32  |  256x192   | 0.675 | 0.816 |    [hrnet_crowdpose.md](./crowdpose/hrnet_crowdpose.md)    |
| CSPNeXt-m  |  256x192   | 0.662 | 0.755 | [hrnet_crowdpose.md](./crowdpose/cspnext_udp_crowdpose.md) |
| ResNet-101 |  256x192   | 0.647 | 0.800 |   [resnet_crowdpose.md](./crowdpose/resnet_crowdpose.md)   |
| HRNet-w32  |  256x192   | 0.637 | 0.785 |   [resnet_crowdpose.md](./crowdpose/resnet_crowdpose.md)   |

### AIC Dataset

Results on AIC val set with ground-truth bounding boxes.

|   Model    | Input Size |  AP   |  AR   |         Details and Download         |

| :--------: | :--------: | :---: | :---: | :----------------------------------: |
| HRNet-w32  |  256x192   | 0.323 | 0.366 |  [hrnet_aic.md](./aic/hrnet_aic.md)  |
| ResNet-101 |  256x192   | 0.294 | 0.337 | [resnet_aic.md](./aic/resnet_aic.md) |

### JHMDB Dataset

|   Model   | Input Size | PCK(norm. by person size) | PCK (norm. by torso size) |            Details and Download            |
| :-------: | :--------: | :-----------------------: | :-----------------------: | :----------------------------------------: |
| ResNet-50 |  256x256   |           96.0            |           80.1            | [resnet_jhmdb.md](./jhmdb/resnet_jhmdb.md) |
|    CPM    |  368x368   |           89.8            |           65.7            |    [cpm_jhmdb.md](./jhmdb/cpm_jhmdb.md)    |

### PoseTrack2018 Dataset

Results on PoseTrack2018 val with ground-truth bounding boxes.

|   Model   | Input Size |  AP  |                     Details and Download                     |
| :-------: | :--------: | :--: | :----------------------------------------------------------: |
| HRNet-w48 |  256x192   | 84.6 |  [hrnet_posetrack18.md](./posetrack18/hrnet_posetrack18.md)  |
| HRNet-w32 |  256x192   | 83.4 |  [hrnet_posetrack18.md](./posetrack18/hrnet_posetrack18.md)  |
| ResNet-50 |  256x192   | 81.2 | [resnet_posetrack18.md](./posetrack18/resnet_posetrack18.md) |

### Human-Art Dataset

|   Model   | Input Size |  AP   |  AR   |                 Details and Download                  |

| :-------: | :--------: | :---: | :---: | :---------------------------------------------------: |
| ViTPose-s |  256x192   | 0.381 | 0.448 | [vitpose_humanart.md](./humanart/vitpose_humanart.md) |
| ViTPose-b |  256x192   | 0.410 | 0.475 | [vitpose_humanart.md](./humanart/vitpose_humanart.md) |

| ViTPose-s |  256x192   | 0.738 | 0.768 | [vitpose_humanart.md](./humanart/vitpose_humanart.md) |
| ViTPose-b |  256x192   | 0.759 | 0.790 | [vitpose_humanart.md](./humanart/vitpose_humanart.md) |

# InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image

<details>
<summary align="right"><a href="https://link.springer.com/content/pdf/10.1007/978-3-030-58565-5_33.pdf">InterNet (ECCV'2020)</a></summary>

```bibtex
}
```

}
```

Analysis of hand-hand interactions is a crucial step towards better understanding human behavior. However, most researches in 3D hand pose estimation have focused on the isolated single hand case. Therefore, we firstly propose (1) a large-scale dataset, InterHand2.6M, and (2) a baseline network, InterNet, for 3D interacting hand pose estimation from a single RGB image. The proposed InterHand2.6M consists of 2.6 M labeled single and interacting hand frames under various poses from multiple subjects. Our InterNet simultaneously performs 3D single and interacting hand pose estimation. In our experiments, we demonstrate big gains in 3D interacting hand pose estimation accuracy when leveraging the interacting hand data in InterHand2.6M. We also report the accuracy of InterNet on InterHand2.6M, which serves as a strong baseline for this new dataset. Finally, we show 3D interacting hand pose estimation results from general images.

<img src="https://user-images.githubusercontent.com/15977946/146519040-dfd5662d-a769-4e63-bff8-cd70877e1acb.png">
</div>

# 快速运行

这个章节会介绍 MMOCR 的一些基本功能。我们假设你已经[从源码安装了 MMOCR](install.md#best-practices)。此外，你也可以通过[教程 Notebook](https://colab.research.google.com/github/open-mmlab/mmocr/blob/dev-1.x/demo/tutorial.ipynb)来了解如何在交互式环境下实现推理、训练和测试。

## 推理

在 MMOCR 的根目录下运行以下命令：

```shell
```

你可以看到弹出的预测结果，以及在控制台中打印出的推理结果。

</div>
<br/ >

```bash
# 识别结果
[...]}]}
```

```{note}
如果你在没有 GUI 的服务器上运行 MMOCR，或者通过没有开启 X11 转发的 SSH 隧道运行 MMOCR，你可能无法看到弹出的窗口。
```

对 MMOCR 中推理接口更为详细的说明，可以在[这里](../user_guides/inference.md)找到。

[...]}]}
```

```{note}
```

除了使用我们提供好的预训练模型，用户也可以在自己的数据集上训练流行模型。接下来我们以在迷你的 [ICDAR 2015](https://rrc.cvc.uab.es/?ch=4&com=downloads) 数据集上训练 DBNet 为例，带大家熟悉 MMOCR 的基本功能。

## 准备数据集

由于 OCR 任务的数据集种类多样，格式不一，不利于多数据集的切换和联合训练，因此 MMOCR 约定了一种[统一的数据格式](../user_guides/dataset_prepare.md)，并针对常用的 OCR 数据集提供了[一键式数据准备脚本](../user_guides/data_prepare/dataset_preparer.md)。通常，要在 MMOCR 中使用数据集，你只需要按照对应步骤运行指令即可。

```{note}
但我们亦深知，效率就是生命——尤其对想要快速上手 MMOCR 的你来说。
```

在这里，我们准备了一个用于演示的精简版 ICDAR 2015 数据集。下载我们预先准备好的[压缩包](https://download.openmmlab.com/mmocr/data/icdar2015/mini_icdar2015.tar.gz)，解压到 mmocr 的 `data/` 目录下，就能得到我们准备好的图片和标注文件。

```Bash
wget https://download.openmmlab.com/mmocr/data/icdar2015/mini_icdar2015.tar.gz
mkdir -p data/
tar xzvf mini_icdar2015.tar.gz -C data/
```

## 修改配置

准备好数据集后，我们接下来就需要通过修改配置的方式指定训练集的位置和训练参数。

```Bash
```

## 修改配置

在这个例子中，我们将会训练一个以 resnet18 作为骨干网络（backbone）的 DBNet。由于 MMOCR 已经有针对完整 ICDAR 2015 数据集的配置 （`configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py`），我们只需要在它的基础上作出一点修改。

我们首先需要修改数据集的路径。在这个配置中，大部分关键的配置文件都在 `_base_` 中被导入，如数据库的配置就来自 `configs/textdet/_base_/datasets/icdar2015.py`。打开该文件，把第一行 `icdar2015_textdet_data_root` 指向的路径替换：

```Python
icdar2015_textdet_data_root = 'data/mini_icdar2015'
```

另外，因为数据集尺寸缩小了，我们也要相应地减少训练的轮次到 400，缩短验证和储存权重的间隔到10轮，并放弃学习率衰减策略。直接把以下几行配置放入 `configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py`即可生效：

```Python
# 每 10 个 epoch 储存一次权重，且只保留最后一个权重
        interval=10,
        max_keep_ckpts=1,
    ))
# 设置最大 epoch 数为 400，每 10 个 epoch 运行一次验证

    ))
train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=400, val_interval=10)
# 令学习率为常量，即不进行学习率衰减
param_scheduler = [dict(type='ConstantLR', factor=1.0),]
```

这里，我们通过配置的继承 ({external+mmengine:doc}`MMEngine: Config <advanced_tutorials/config>`) 机制将基础配置中的相应参数直接进行了改写。原本的字段分布在 `configs/textdet/_base_/schedules/schedule_sgd_1200e.py` 和 `configs/textdet/_base_/default_runtime.py` 中，感兴趣的读者可以自行查看。

```{note}
关于配置文件更加详尽的说明，请参考[此处](../user_guides/config.md)。
```

## 可视化数据集

在正式开始训练前，我们还可以可视化一下经过训练过程中[数据变换（transforms）](../basic_concepts/transforms.md)后的图像。方法也很简单，把我们需要可视化的配置传入 [browse_dataset.py](/tools/analysis_tools/browse_dataset.py) 脚本即可：

```Bash
python tools/analysis_tools/browse_dataset.py configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py
```

数据变换后的图片和标签会在弹窗中逐张被展示出来。

```Bash
```

</center>

```{note}
有关该脚本更详细的指南，请参考[此处](../user_guides/useful_tools.md).
```

```{tip}
除了满足好奇心之外，可视化还可以帮助我们在训练前检查可能影响到模型表现的部分，如配置文件、数据集及数据变换中的问题。
```

## 训练

万事俱备，只欠东风。运行以下命令启动训练：

```Bash
python tools/train.py configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py
```

根据系统情况，MMOCR 会自动使用最佳的设备进行训练。如果有 GPU，则会默认在第一张卡启动单卡训练。当开始看到 loss 的输出，就说明你已经成功启动了训练。

```Bash

```Bash
```

```Bash
2022/08/22 18:42:22 - mmengine - INFO - Epoch(train) [1][5/7]  lr: 7.0000e-03  memory: 7730  data_time: 0.4496  loss_prob: 14.6061  loss_thr: 2.2904  loss_db: 0.9879  loss: 17.8843  time: 1.8666
2022/08/22 18:42:24 - mmengine - INFO - Exp name: dbnet_resnet18_fpnc_1200e_icdar2015
2022/08/22 18:42:28 - mmengine - INFO - Epoch(train) [2][5/7]  lr: 7.0000e-03  memory: 6695  data_time: 0.2052  loss_prob: 6.7840  loss_thr: 1.4114  loss_db: 0.9855  loss: 9.1809  time: 0.7506
2022/08/22 18:42:29 - mmengine - INFO - Exp name: dbnet_resnet18_fpnc_1200e_icdar2015
2022/08/22 18:42:33 - mmengine - INFO - Epoch(train) [3][5/7]  lr: 7.0000e-03  memory: 6690  data_time: 0.2101  loss_prob: 3.0700  loss_thr: 1.1800  loss_db: 0.9967  loss: 5.2468  time: 0.6244

2022/08/22 18:42:33 - mmengine - INFO - Exp name: dbnet_resnet18_fpnc_1200e_icdar2015
```

在不指定额外参数时，训练的权重默认会被保存到 `work_dirs/dbnet_resnet18_fpnc_1200e_icdar2015/` 下面，而日志则会保存在`work_dirs/dbnet_resnet18_fpnc_1200e_icdar2015/开始训练的时间戳/`里。接下来，我们只需要耐心等待模型训练完成即可。

```{note}
若需要了解训练的高级用法，如 CPU 训练、多卡训练及集群训练等，请查阅[训练与测试](../user_guides/train_test.md)。
```

## 测试

经过数十分钟的等待，模型顺利完成了400 epochs的训练。我们通过控制台的输出，观察到 DBNet 在最后一个 epoch 的表现最好，`hmean` 达到了 60.86（你可能会得到一个不太一样的结果）：

```Bash
08/22 19:24:52 - mmengine - INFO - Epoch(val) [400][100/100]  icdar/precision: 0.7285  icdar/recall: 0.5226  icdar/hmean: 0.6086
```

```{note}
它或许还没被训练到最优状态，但对于一个演示而言已经足够了。
```

然而，这个数值只反映了 DBNet 在迷你 ICDAR 2015 数据集上的性能。要想更加客观地评判它的检测能力，我们还要看看它在分布外数据集上的表现。例如，`tests/data/det_toy_dataset` 就是一个很小的真实数据集，我们可以用它来验证一下 DBNet 的实际性能。

```

```{note}
```

在测试前，我们同样需要对数据集的位置做一下修改。打开 `configs/textdet/_base_/datasets/icdar2015.py`，修改 `icdar2015_textdet_test` 的 `data_root` 为 `tests/data/det_toy_dataset`:

```Python
# ...
    # ...
    )
```

修改完毕，运行命令启动测试。

```Bash
python tools/test.py configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py work_dirs/dbnet_resnet18_fpnc_1200e_icdar2015/epoch_400.pth
```

得到输出：

```Bash
08/21 21:45:59 - mmengine - INFO - Epoch(test) [5/10]    memory: 8562
08/21 21:45:59 - mmengine - INFO - Epoch(test) [10/10]    eta: 0:00:00  time: 0.4893  data_time: 0.0191  memory: 283
08/21 21:45:59 - mmengine - INFO - Evaluating hmean-iou...

08/21 21:45:59 - mmengine - INFO - prediction score threshold: 0.30, recall: 0.6190, precision: 0.4815, hmean: 0.5417
08/21 21:45:59 - mmengine - INFO - prediction score threshold: 0.40, recall: 0.6190, precision: 0.5909, hmean: 0.6047
08/21 21:45:59 - mmengine - INFO - prediction score threshold: 0.50, recall: 0.6190, precision: 0.6842, hmean: 0.6500
08/21 21:45:59 - mmengine - INFO - prediction score threshold: 0.60, recall: 0.6190, precision: 0.7222, hmean: 0.6667
08/21 21:45:59 - mmengine - INFO - prediction score threshold: 0.70, recall: 0.3810, precision: 0.8889, hmean: 0.5333
08/21 21:45:59 - mmengine - INFO - prediction score threshold: 0.80, recall: 0.0000, precision: 0.0000, hmean: 0.0000

08/21 21:45:59 - mmengine - INFO - prediction score threshold: 0.90, recall: 0.0000, precision: 0.0000, hmean: 0.0000
08/21 21:45:59 - mmengine - INFO - Epoch(test) [10/10]  icdar/precision: 0.7222  icdar/recall: 0.6190  icdar/hmean: 0.6667
```

可以发现，模型在这个数据集上能达到的 hmean 为 0.6667，效果还是不错的。

```{note}
若需要了解测试的高级用法，如 CPU 测试、多卡测试及集群测试等，请查阅[训练与测试](../user_guides/train_test.md)。
```

## 可视化输出

为了对模型的输出有一个更直观的感受，我们还可以直接可视化它的预测输出。在 `test.py` 中，用户可以通过 `show` 参数打开弹窗可视化；也可以通过 `show-dir` 参数指定预测结果图导出的目录。

```Bash
python tools/test.py configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py work_dirs/dbnet_resnet18_fpnc_1200e_icdar2015/epoch_400.pth --show-dir imgs/
```

真实标签和预测值会在可视化结果中以平铺的方式展示。左图的绿框表示真实标签，右图的红框表示预测值。

    <img src="https://user-images.githubusercontent.com/22607038/187423562-6a85e209-4b12-46ee-8a41-5c67b1ba83f9.png"/><br>
</div>

```{note}

</div>

```{note}
有关更多可视化功能的介绍，请参阅[这里](../user_guides/visualization.md)。
```

# Add a Model

Currently, we support HF models, some model APIs, and some third-party models.

## Adding API Models

To add a new API-based model, you need to create a new file named `mymodel_api.py` under `opencompass/models` directory. In this file, you should inherit from `BaseAPIModel` and implement the `generate` method for inference and the `get_token_len` method to calculate the length of tokens. Once you have defined the model, you can modify the corresponding configuration file.

```python
from ..base_api import BaseAPIModel

class MyModelAPI(BaseAPIModel):

    is_api: bool = True

                 path: str,
                 max_seq_len: int = 2048,
                 query_per_second: int = 1,
                 retry: int = 2,
                 **kwargs):
        super().__init__(path=path,
                         max_seq_len=max_seq_len,
                         meta_template=meta_template,
                         query_per_second=query_per_second,

super().__init__(path=path,
                         retry=retry)
        ...

    def generate(
        self,
        inputs,
        max_out_len: int = 512,
        temperature: float = 0.7,
    ) -> List[str]:
        """Generate results given a list of inputs."""
        pass

    def get_token_len(self, prompt: str) -> int:
        """Get lengths of the tokenized string."""
        pass
```

## Adding Third-Party Models

pass

        pass
```

To add a new third-party model, you need to create a new file named `mymodel.py` under `opencompass/models` directory. In this file, you should inherit from `BaseModel` and implement the `generate` method for generative inference, the `get_ppl` method for discriminative inference, and the `get_token_len` method to calculate the length of tokens. Once you have defined the model, you can modify the corresponding configuration file.

```python
from ..base import BaseModel

class MyModel(BaseModel):

                 pkg_root: str,
                 ckpt_path: str,
                 tokenizer_only: bool = False,
                 meta_template: Optional[Dict] = None,
        ...

        """Get lengths of the tokenized strings."""

meta_template: Optional[Dict] = None,
        ...

        pass

    def generate(self, inputs: List[str], max_out_len: int) -> List[str]:
        """Generate results given a list of inputs. """
        pass

    def get_ppl(self,
                inputs: List[str],
                mask_length: Optional[List[int]] = None) -> List[float]:
        """Get perplexity scores given a list of inputs."""
        pass
```

# Migration of Data Settings

This section introduces the migration of data settings:

- [Migration of Data Settings](#migration-of-data-settings)
  - [Data pipelines](#data-pipelines)
  - [Dataloader](#dataloader)

## Data pipelines

We update data pipelines settings in MMagic 1.x. Important modifications are as following.

- Remove normalization and color space transforms operations. They are moved from datasets transforms pipelines to data_preprocessor.
- The original formatting transforms pipelines `Collect` and `ToTensor` are combined as `PackInputs`.
  More details of data pipelines are shown in [transform guides](../howto/transforms.md).

<thead>
  <tr>
<tbody>
<tr>

```python
    dict(type='LoadImageFromFile',  # Load images from files
        io_backend='disk',  # io backend
        key='lq',  # Keys in results to find corresponding path

dict(type='LoadImageFromFile',  # Load images from files
        flag='unchanged'),  # flag for reading images
        key='gt',  # Keys in results to find corresponding path
    dict(type='RescaleToZeroOne', keys=['lq', 'gt']),  # Rescale images from [0, 255] to [0, 1]
    dict(type='Normalize',  # Augmentation pipeline that normalize the input images
        keys=['lq', 'gt'],  # Images to be normalized
        mean=[0, 0, 0],  # Mean values
        std=[1, 1, 1],  # Standard variance
        to_rgb=True),  # Change to RGB channel
    dict(type='PairedRandomCrop', gt_patch_size=96),  # Paired random crop
    dict(type='Flip',  # Flip images
        keys=['lq', 'gt'],  # Images to be flipped

dict(type='PairedRandomCrop', gt_patch_size=96),  # Paired random crop
        flip_ratio=0.5,  # Flip ratio
        direction='horizontal'),  # Flip direction
        direction='vertical'),  # Flip direction
    dict(type='RandomTransposeHW',  # Random transpose h and w for images
        keys=['lq', 'gt'],  # Images to be transposed
        transpose_ratio=0.5  # Transpose ratio
        ),
    dict(type='Collect',  # Pipeline that decides which keys in the data should be passed to the model
        keys=['lq', 'gt'],  # Keys to pass to the model
        meta_keys=['lq_path', 'gt_path']), # Meta information keys. In training, meta information is not needed
    dict(type='ToTensor',  # Convert images to tensor
        keys=['lq', 'gt'])  # Images to be converted to Tensor

dict(type='ToTensor',  # Convert images to tensor
]
test_pipeline = [  # Test pipeline
    dict(
        type='LoadImageFromFile',  # Load images from files
    dict(
    dict(

keys=['lq', 'gt'],  # Images to be normalized
        meta_keys=['lq_path', 'gt_path']),  # Meta information keys
]
```

</td>

```python
        key='img',  # Keys in results to find corresponding path
        color_type='color',  # Color type of image
        channel_order='rgb',  # Channel order of image
        imdecode_backend='cv2'),  # decode backend

channel_order='rgb',  # Channel order of image
    dict(type='SetValues', dictionary=dict(scale=scale)),  # Set value to destination keys

flip_ratio=0.5,  # Flip ratio
        ),
    dict(type='PackInputs')  # The config of collecting data from current pipeline
]

color_type='color',  # Color type of image
]
```

</td>

</tr>
</thead>
</table>

## Dataloader

We update dataloader settings in MMagic 1.x. Important modifications are as following.

- The original `data` field is split to `train_dataloader`, `val_dataloader` and `test_dataloader`. This allows us to configure them in fine-grained. For example, you can specify different sampler and batch size during training and test.
- The `samples_per_gpu` is renamed to `batch_size`.
- The `workers_per_gpu` is renamed to `num_workers`.

<thead>
  <tr>
<tbody>
<tr>

```python
    # train
    samples_per_gpu=16,  # Batch size of a single GPU
    workers_per_gpu=4,  # Worker to pre-fetch data for each single GPU

<tbody>
<tr>

```python
    # train
    drop_last=True,  # Use drop_last in data_loader
        type='RepeatDataset',  # Repeated dataset for iter-based model
        times=1000,  # Repeated times for RepeatDataset
            type=train_dataset_type,  # Type of dataset
            lq_folder='data/DIV2K/DIV2K_train_LR_bicubic/X2_sub',  # Path for lq folder
            gt_folder='data/DIV2K/DIV2K_train_HR_sub',  # Path for gt folder
            ann_file='data/DIV2K/meta_info_DIV2K800sub_GT.txt',  # Path for annotation file
            pipeline=train_pipeline,  # See above for train_pipeline
            scale=scale)),  # Scale factor for upsampling
    # val
    val_samples_per_gpu=1,  # Batch size of a single GPU for validation

scale=scale)),  # Scale factor for upsampling
    # val
    val_workers_per_gpu=4,  # Worker to pre-fetch data for each single GPU for validation
    val=dict(
        type=val_dataset_type,  # Type of dataset
        lq_folder='data/val_set5/Set5_bicLRx2',  # Path for lq folder
        gt_folder='data/val_set5/Set5_mod12',  # Path for gt folder
        pipeline=test_pipeline,  # See above for test_pipeline
        scale=scale,  # Scale factor for upsampling
        filename_tmpl='{}'),  # filename template
    # test
        filename_tmpl='{}'))  # filename template
```

</td>

pipeline=test_pipeline,  # See above for test_pipeline
```

</td>

```python
dataset_type = 'BasicImageDataset'  # The type of dataset
data_root = 'data'  # Root path of data
    num_workers=4,  # The number of workers to pre-fetch data for each single GPU
    persistent_workers=False,  # Whether maintain the workers Dataset instances alive
    sampler=dict(type='InfiniteSampler', shuffle=True),  # The type of data sampler
        type=dataset_type,  # Type of dataset
        ann_file='meta_info_DIV2K800sub_GT.txt',  # Path of annotation file
        metainfo=dict(dataset_type='div2k', task_name='sisr'),
        data_root=data_root + '/DIV2K',  # Root path of data
        data_prefix=dict(  # Prefix of image path

metainfo=dict(dataset_type='div2k', task_name='sisr'),
            img='DIV2K_train_LR_bicubic/X2_sub', gt='DIV2K_train_HR_sub'),
        filename_tmpl=dict(img='{}', gt='{}'),  # Filename template
    drop_last=False,  # Whether drop the last incomplete batch
    sampler=dict(type='DefaultSampler', shuffle=False),  # The type of data sampler
    dataset=dict(  # Validation dataset config
        metainfo=dict(dataset_type='set5', task_name='sisr'),
        data_root=data_root + '/Set5',  # Root path of data
        data_prefix=dict(img='LRbicx2', gt='GTmod12'),  # Prefix of image path

metainfo=dict(dataset_type='set5', task_name='sisr'),
```

</td>

</tr>
</thead>
</table>

# 模型复杂度分析

## 计算 FLOPs 和参数数量（实验性的）

我们根据 [MMEngine](https://github.com/open-mmlab/mmengine/blob/main/mmengine/analysis/complexity_analysis.py) 提供了一个脚本用于计算给定模型的 FLOPs 和参数量。

```shell
```

所有参数说明：

- `config` : 配置文件的路径。
- `--shape`: 输入尺寸，支持单值或者双值， 如： `--shape 256`、`--shape 224 256`。默认为`224 224`。

示例：

```shell
python tools/analysis_tools/get_flops.py configs/resnet/resnet50_8xb32_in1k.py
```

你将获得如下结果：

```text
Input shape: (3, 224, 224)
Flops: 4.109G
Params: 25.557M
Activation: 11.114M
```

同时，你会得到每层的详细复杂度信息，如下所示：

```text
+--------------------------+----------------------+-----------+--------------+
| module                   | #parameters or shape | #flops    | #activations |
| model                    | 25.557M              | 4.109G    | 11.114M      |

|  backbone                |  23.508M             |  4.109G   |  11.114M     |
|   backbone.conv1         |   9.408K             |   0.118G  |   0.803M     |
|    backbone.conv1.weight |    (64, 3, 7, 7)     |           |              |
|   backbone.bn1           |   0.128K             |   1.606M  |   0          |
|    backbone.bn1.weight   |    (64,)             |           |              |
|    backbone.bn1.bias     |    (64,)             |           |              |
|   backbone.layer1        |   0.216M             |   0.677G  |   4.415M     |
|    backbone.layer1.0     |    75.008K           |    0.235G |    2.007M    |
|    backbone.layer1.1     |    70.4K             |    0.221G |    1.204M    |
|    backbone.layer1.2     |    70.4K             |    0.221G |    1.204M    |

|   backbone.layer2        |   1.22M              |   1.034G  |   3.111M     |
|    backbone.layer2.0     |    0.379M            |    0.375G |    1.305M    |
|    backbone.layer2.1     |    0.28M             |    0.22G  |    0.602M    |
|    backbone.layer2.2     |    0.28M             |    0.22G  |    0.602M    |
|    backbone.layer2.3     |    0.28M             |    0.22G  |    0.602M    |
|   backbone.layer3        |   7.098M             |   1.469G  |   2.158M     |
|    backbone.layer3.0     |    1.512M            |    0.374G |    0.652M    |
|    backbone.layer3.1     |    1.117M            |    0.219G |    0.301M    |
|    backbone.layer3.2     |    1.117M            |    0.219G |    0.301M    |
|    backbone.layer3.3     |    1.117M            |    0.219G |    0.301M    |

|    backbone.layer3.4     |    1.117M            |    0.219G |    0.301M    |
|    backbone.layer3.5     |    1.117M            |    0.219G |    0.301M    |
|   backbone.layer4        |   14.965M            |   0.81G   |   0.627M     |
|    backbone.layer4.0     |    6.04M             |    0.373G |    0.326M    |
|    backbone.layer4.1     |    4.463M            |    0.219G |    0.151M    |
|    backbone.layer4.2     |    4.463M            |    0.219G |    0.151M    |
|  head.fc                 |  2.049M              |           |              |
|   head.fc.weight         |   (1000, 2048)       |           |              |
|   head.fc.bias           |   (1000,)            |           |              |
|  neck.gap                |                      |  0.1M     |  0           |

```

警告

此工具仍处于试验阶段，我们不保证该数字正确无误。您最好将结果用于简单比较，但在技术报告或论文中采用该结果之前，请仔细检查。

- FLOPs 与输入的尺寸有关，而参数量与输入尺寸无关。默认输入尺寸为 (1, 3, 224, 224)
- 一些运算不会被计入 FLOPs 的统计中，例如某些自定义运算。详细信息请参考 [`mmengine.analysis.complexity_analysis._DEFAULT_SUPPORTED_FLOP_OPS`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/analysis/complexity_analysis.py)。
```

# VISION-Datasets

> VISION Datasets: A Benchmark for Vision-based InduStrial InspectiON

Despite progress in vision-based inspection algorithms, real-world industrial challenges – specifically in data availability, quality, and complex production requirements – often remain under-addressed. We introduce the VISION Datasets, a diverse collection of 14 industrial inspection datasets, uniquely poised to meet these challenges. Unlike previous datasets, VISION brings versatility to defect detection, offering annotation masks across all splits and catering to various detection methodologies. Our datasets also feature instance-segmentation annotation, enabling precise defect identification. With a total of 18k images encompassing 44 defect types, VISION strives to mirror a wide range of real-world production scenarios. By supporting two ongoing challenge competitions on the VISION Datasets, we hope to foster further advancements in vision-based industrial inspection. The datasets are available at https://huggingface.co/datasets/VISION-Workshop/VISION-Datasets.

<img src="https://github.com/open-mmlab/mmdetection/assets/17425982/1d8e81d1-0023-49ce-a855-a09149509fe7" width="70%"/>
</div>

At first, you should download the dataset from https://huggingface.co/datasets/VISION-Workshop/VISION-Datasets and organize it as follows:

```text
├── mmdet
├── tools
├── data
├── │── VISION-Datasets
├── │   ├── Cable.tar.gz
├── │   ├── Capacitor.tar.gz
├── │   ├── Casting.tar.gz
├── |   ├── Console.tar.gz
├── │   ├── Cylinder.tar.gz
├── │   ├── Electronics.tar.gz
├── │   ├── Groove.tar.gz
├── │   ├── Hemisphere.tar.gz
├── │   ├── Lens.tar.gz
├── │   ├── PCB_1.tar.gz
├── │   ├── PCB_2.tar.gz
├── |   ├── README.md
├── │   ├── Ring.tar.gz
├── │   ├── Screw.tar.gz
├── │   └── Wood.tar.gz
```

```

Then you can use the following command to save the following command as the `vision_unzip.sh` file and place it in the `mmdetection` root directory, and then run the script `bash vision_unzip.sh` to unzip it.

```shell
#!/usr/bin/env bash

for file in data/VISION-Datasets/*.tar.gz; do
    tar -xzvzf "$file" -C data/VISION-Datasets/
done
```

Finally, the file organization format is as follows:

```text
├── mmdet
├── tools
├── data
|   │── VISION-Datasets
|   │   ├── Cable.tar.gz
|   │   ├── Capacitor.tar.gz
|   │   ├── Casting.tar.gz
|   |   ├── Console.tar.gz
|   │   ├── Cylinder.tar.gz
|   │   ├── Electronics.tar.gz
|   │   ├── Groove.tar.gz
|   │   ├── Hemisphere.tar.gz
|   │   ├── Lens.tar.gz
|   │   ├── PCB_1.tar.gz
|   │   ├── PCB_2.tar.gz
|   |   ├── README.md
|   │   ├── Ring.tar.gz

|   │   ├── Screw.tar.gz
|   │   └── Wood.tar.gz
|   │   ├── Cable
|   │   |    |── train
|   │   |    |     |──  _annotations.coco.json # COCO format annotation
|   │   |    |     |──  000001.png # Images
|   │   |    |     |──  000002.png
|   │   |    |     |──  xxxxxx.png
|   │   |    |── val
|   │   |    |     |──  xxxxxx.png # Images
|   │   |    |── inference
|   │   |    |     |──  _annotations.coco.json # COCO format annotation with unlabeled image list only
...
```

## Models and Results

TODO

```latex
@article{vision-datasets,
  title         = {VISION Datasets: A Benchmark for Vision-based InduStrial InspectiON},

...
```

TODO

```latex
  author        = {Haoping Bai, Shancong Mou, Tatiana Likhomanenko, Ramazan Gokberk Cinbis, Oncel Tuzel, Ping Huang, Jiulong Shan, Jianjun Shi, Meng Cao},
  journal       = {arXiv preprint arXiv:2306.07890},
  year          = {2023},
}
```

# Distribution-aware coordinate representation for human pose estimation

<details>

```bibtex
}
```

While being the de facto standard coordinate representation for human pose estimation, heatmap has not been investigated in-depth. This work fills this gap. For the first time, we find that the process of decoding the predicted heatmaps into the final joint coordinates in the original image space is surprisingly significant for the performance. We further probe the design limitations of the standard coordinate decoding method, and propose a more principled distributionaware decoding method. Also, we improve the standard coordinate encoding process (i.e. transforming ground-truth coordinates to heatmaps) by generating unbiased/accurate heatmaps. Taking the two together, we formulate a novel Distribution-Aware coordinate Representation of Keypoints (DARK) method. Serving as a model-agnostic plug-in, DARK brings about significant performance boost to existing human pose estimation models. Extensive experiments show that DARK yields the best results on two common benchmarks, MPII and COCO. Besides, DARK achieves the 2nd place entry in the ICCV 2019 COCO Keypoints Challenge. The code is available online.

<img src="https://user-images.githubusercontent.com/15977946/146514732-1d53614b-e5b7-4a1c-a39f-6fc726217d81.png">
</div>

# Pan-Cancer Histology Dataset for Nuclei Instance Segmentation and Classification (PanNuke)

This project supports **`Pan-Cancer Histology Dataset for Nuclei Instance Segmentation and Classification (PanNuke)`**, which can be downloaded from [here](https://academictorrents.com/details/99f2c7b57b95500711e33f2ee4d14c9fd7c7366c).

Semi automatically generated nuclei instance segmentation and classification dataset with exhaustive nuclei labels across 19 different tissue types. The dataset consists of 481 visual fields, of which 312 are randomly sampled from more than 20K whole slide images at different magnifications, from multiple data sources. In total the dataset contains 205,343 labeled nuclei, each with an instance segmentation mask. Models trained on pannuke can aid in whole slide image tissue type segmentation, and generalise to new tissues. PanNuke demonstrates one of the first successfully semi-automatically generated datasets.

| Dataset Name                                                                             | Anatomical Region | Task Type    | Modality       | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License                                                         |
| ---------------------------------------------------------------------------------------- | ----------------- | ------------ | -------------- | ------------ | --------------------- | ---------------------- | ------------ | --------------------------------------------------------------- |
| [Pannuke](https://academictorrents.com/details/99f2c7b57b95500711e33f2ee4d14c9fd7c7366c) | full_body         | segmentation | histopathology | 6            | 7901/-/-              | yes/-/-                | 2019         | [CC-BY-NC 4.0](https://creativecommons.org/licenses/by-sa/4.0/) |

|        Class Name         | Num. Train | Pct. Train | Num. Val | Pct. Val | Num. Test | Pct. Test |

| :-----------------------: | :--------: | :--------: | :------: | :------: | :-------: | :-------: |
|        background         |    7901    |   83.32    |    -     |    -     |     -     |     -     |
|        neoplastic         |    4190    |    8.64    |    -     |    -     |     -     |     -     |
| non-neoplastic epithelial |    4126    |    1.77    |    -     |    -     |     -     |     -     |
|       inflammatory        |    6137    |    3.73    |    -     |    -     |     -     |     -     |
|        connective         |    232     |    0.07    |    -     |    -     |     -     |     -     |
|           dead            |    1528    |    2.47    |    -     |    -     |     -     |     -     |

Note:

Note:

![pannuke](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/histopathology/pannuke/pannuke_dataset.png?raw=true)

```
@inproceedings{gamper2019pannuke,
  title={PanNuke: an open pan-cancer histology dataset for nuclei instance segmentation and classification},
  author={Gamper, Jevgenij and Koohbanani, Navid Alemi and Benet, Ksenija and Khuram, Ali and Rajpoot, Nasir},
  booktitle={European Congress on Digital Pathology},
  pages={11--19},
}
```

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `pannuke/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
```

- download dataset from [here](https://academictorrents.com/details/99f2c7b57b95500711e33f2ee4d14c9fd7c7366c) and decompress data to path `'data/'`.

```none
  ├── mmseg
  │   │   │   │   ├── pannuke

```

|        background         |    6320    |   83.38    |   1581   |   83.1   |     -     |     -     |
|        neoplastic         |    3339    |    8.55    |   851    |   9.0    |     -     |     -     |
| non-neoplastic epithelial |    3293    |    1.77    |   833    |   1.76   |     -     |     -     |

|       inflammatory        |    4914    |    3.72    |   1223   |   3.76   |     -     |     -     |
|        connective         |    170     |    0.06    |    62    |   0.09   |     -     |     -     |
|           dead            |    1235    |    2.51    |   293    |   2.29   |     -     |     -     |

```shell
```

```shell
```

- [x] Finish the code

# Fast animal pose estimation using deep neural networks

<details>

```bibtex
}
```

}
```

<details>

```bibtex
}
```

}
```

| [deeppose_resnet_50](/configs/body_2d_keypoint/topdown_regression/mpii/td-reg_res50_8xb64-210e_mpii-256x256.py) |  256x256   | 0.826 |  0.180   | [ckpt](https://download.openmmlab.com/mmpose/top_down/deeppose/deeppose_res50_mpii_256x256-c63cd0b6_20210203.pth) | [log](https://download.openmmlab.com/mmpose/top_down/deeppose/deeppose_res50_mpii_256x256_20210203.log.json) |

| [deeppose_resnet_101](/configs/body_2d_keypoint/topdown_regression/mpii/td-reg_res101_8xb64-210e_mpii-256x256.py) |  256x256   | 0.841 |  0.200   | [ckpt](https://download.openmmlab.com/mmpose/top_down/deeppose/deeppose_res101_mpii_256x256-87516a90_20210205.pth) | [log](https://download.openmmlab.com/mmpose/top_down/deeppose/deeppose_res101_mpii_256x256_20210205.log.json) |
| [deeppose_resnet_152](/configs/body_2d_keypoint/topdown_regression/mpii/td-reg_res152_8xb64-210e_mpii-256x256.py) |  256x256   | 0.850 |  0.208   | [ckpt](https://download.openmmlab.com/mmpose/top_down/deeppose/deeppose_res152_mpii_256x256-15f5e6f9_20210205.pth) | [log](https://download.openmmlab.com/mmpose/top_down/deeppose/deeppose_res152_mpii_256x256_20210205.log.json) |

# Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

<details>
<summary align="right"><a href="https://arxiv.org/abs/2102.12122">PVT (ICCV'2021)</a></summary>

```bibtex
@inproceedings{wang2021pyramid,
  pages={568--578},
}
```

Although using convolutional neural networks (CNNs) as backbones achieves great
successes in computer vision, this work investigates a simple backbone network
useful for many dense prediction tasks without convolutions. Unlike the
recently-proposed Transformer model (e.g., ViT) that is specially designed for

image classification, we propose Pyramid Vision Transformer~(PVT), which overcomes
the difficulties of porting Transformer to various dense prediction tasks.
PVT has several merits compared to prior arts. (1) Different from ViT that
typically has low-resolution outputs and high computational and memory cost,
PVT can be not only trained on dense partitions of the image to achieve high
output resolution, which is important for dense predictions but also using a
progressive shrinking pyramid to reduce computations of large feature maps.
(2) PVT inherits the advantages from both CNN and Transformer, making it a
unified backbone in various vision tasks without convolutions by simply replacing
CNN backbones. (3) We validate PVT by conducting extensive experiments, showing

that it boosts the performance of many downstream tasks, e.g., object detection,
semantic, and instance segmentation. For example, with a comparable number of
parameters, RetinaNet+PVT achieves 40.4 AP on the COCO dataset, surpassing
RetinNet+ResNet50 (36.3 AP) by 4.1 absolute AP. We hope PVT could serve as an
alternative and useful backbone for pixel-level predictions and facilitate future
researches. Code is available at https://github.com/whai362/PVT .

<img src="https://user-images.githubusercontent.com/28900607/166134407-d455c57e-4ec3-4951-be9e-730cb9d9c213.png">
</div>

# RepLKNet

> [Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs](https://arxiv.org/abs/2203.06717)

We revisit large kernel design in modern convolutional neural networks (CNNs). Inspired by recent advances in vision transformers (ViTs), in this paper, we demonstrate that using a few large convolutional kernels instead of a stack of small kernels could be a more powerful paradigm. We suggested five guidelines, e.g., applying re-parameterized large depth-wise convolutions, to design efficient highperformance large-kernel CNNs. Following the guidelines, we propose RepLKNet, a pure CNN architecture whose kernel size is as large as 31×31, in contrast to commonly used 3×3. RepLKNet greatly closes the performance gap between CNNs and ViTs, e.g., achieving comparable or superior results than Swin Transformer on ImageNet and a few typical downstream tasks, with lower latency. RepLKNet also shows nice scalability to big data and large models, obtaining 87.8% top-1 accuracy on ImageNet and 56.0% mIoU on ADE20K, which is very competitive among the state-of-the-arts with similar model sizes. Our study further reveals that, in contrast to small-kernel CNNs, large kernel CNNs have much larger effective receptive fields and higher shape bias rather than texture bias.

<img src="https://user-images.githubusercontent.com/48375204/197546040-cdf078c3-7fbd-400f-8b27-01668c8dfebf.png" width="60%"/>
</div>

```python

model = get_model('replknet-31B_3rdparty_in1k', pretrained=True)
```

```python

```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/replknet/replknet-31B_32xb64_in1k.py https://download.openmmlab.com/mmclassification/v0/replknet/replknet-31B_3rdparty_in1k_20221118-fd08e268.pth
```

```bash
```

```bash

```bash
```

```python
from mmpretrain.models import RepLKNet

backbone = RepLKNet(arch='31B')
```

| Model                                          |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                   Config                    |                            Download                            |
| :--------------------------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :-----------------------------------------: | :------------------------------------------------------------: |

| `replknet-31B_3rdparty_in1k`\*                 | From scratch |   79.86    |   15.64   |   83.48   |   96.57   |    [config](replknet-31B_32xb64_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/replknet/replknet-31B_3rdparty_in1k_20221118-fd08e268.pth) |
| `replknet-31B_3rdparty_in1k-384px`\*           | From scratch |   79.86    |   45.95   |   84.84   |   97.34   | [config](replknet-31B_32xb64_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/replknet/replknet-31B_3rdparty_in1k-384px_20221118-03a170ce.pth) |
| `replknet-31B_in21k-pre_3rdparty_in1k`\*       | ImageNet-21k |   79.86    |   15.64   |   85.20   |   97.56   |    [config](replknet-31B_32xb64_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/replknet/replknet-31B_in21k-pre_3rdparty_in1k_20221118-54ed5c46.pth) |

| `replknet-31B_in21k-pre_3rdparty_in1k-384px`\* | ImageNet-21k |   79.86    |   45.95   |   85.99   |   97.75   | [config](replknet-31B_32xb64_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/replknet/replknet-31B_in21k-pre_3rdparty_in1k-384px_20221118-76c92b24.pth) |
| `replknet-31L_in21k-pre_3rdparty_in1k-384px`\* | ImageNet-21k |   172.67   |   97.24   |   86.63   |   98.00   | [config](replknet-31L_32xb64_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/replknet/replknet-31L_in21k-pre_3rdparty_in1k-384px_20221118-dc3fc07c.pth) |
| `replknet-XL_meg73m-pre_3rdparty_in1k-320px`\* |    MEG73M    |   335.44   |  129.57   |   87.57   |   98.39   | [config](replknet-XL_32xb64_in1k-320px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/replknet/replknet-XL_meg73m-pre_3rdparty_in1k-320px_20221118-88259b1d.pth) |

*Models with * are converted from the [official repo](https://github.com/DingXiaoH/RepLKNet-pytorch/blob/main/replknet.py). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@inproceedings{ding2022scaling,
  title={Scaling up your kernels to 31x31: Revisiting large kernel design in cnns},
  author={Ding, Xiaohan and Zhang, Xiangyu and Han, Jungong and Ding, Guiguang},
  pages={11963--11975},
}
```

# 准备 HVU

## 简介

```BibTeX
@article{Diba2019LargeSH,
  title={Large Scale Holistic Video Understanding},
  author={Ali Diba and M. Fayyaz and Vivek Sharma and Manohar Paluri and Jurgen Gall and R. Stiefelhagen and L. Gool},
}
```

请参照 [官方项目](https://github.com/holistic-video-understanding/HVU-Dataset/) 及 [原论文](https://arxiv.org/abs/1904.11451) 以获取数据集基本信息。
在开始之前，用户需确保当前目录为 `$MMACTION2/tools/data/hvu/`。

```shell
```

此外，用户可使用如下命令解析 HVU 的标签列表：

```shell
python parse_tag_list.py
```

## 2. 准备视频

```shell
```

## 3. 提取 RGB 帧和光流

```

用户可使用如下脚本同时抽取 RGB 帧和光流：

```shell
```

该脚本默认生成短边长度为 256 的帧，可参考 [数据准备](/docs/zh_cn/user_guides/prepare_dataset.md) 获得更多细节。

## 4. 生成文件列表

用户可以使用以下两个脚本分别为视频和帧文件夹生成文件列表：

```shell
# 为帧文件夹生成文件列表
```

## 5. 为每个 tag 种类生成文件列表

若用户需要为 HVU 数据集的每个 tag 种类训练识别模型，则需要进行此步骤。

步骤 4 中生成的文件列表包含不同类型的标签，仅支持使用 HVUDataset 进行涉及多个标签种类的多任务学习。加载数据的过程中需要使用 `LoadHVULabel` 类进行多类别标签的加载，训练过程中使用 `HVULoss` 作为损失函数。

如果用户仅需训练某一特定类别的标签，例如训练一识别模型用于识别 HVU 中 `action` 类别的标签，则建议使用如下脚本为特定标签种类生成文件列表。新生成的列表将只含有特定类别的标签，因此可使用 `VideoDataset` 或 `RawframeDataset` 进行加载。训训练过程中使用 `BCELossWithLogits` 作为损失函数。

以下脚本为类别为 ${category} 的标签生成文件列表，注意仅支持 HVU 数据集包含的 6 种标签类别: action, attribute, concept, event, object, scene。

```shell

```shell
python generate_sub_file_list.py path/to/filelist.json ${category}
```

对于类别 ${category}，生成的标签列表文件名中将使用 `hvu_${category}` 替代 `hvu`。例如，若原指定文件名为 `hvu_train.json`，则对于类别 action，生成的文件列表名为 `hvu_action_train.json`。

## 6. 目录结构

在完整完成 HVU 的数据处理后，将得到帧文件夹（RGB 帧和光流帧），视频以及标注文件。

在整个项目目录下（仅针对 HVU），完整目录结构如下所示：

```
mmaction2
├── tools
├── data
│   ├── hvu
│   │   ├── hvu_train_video.json
│   │   ├── hvu_val_video.json
│   │   ├── hvu_train.json
│   │   ├── hvu_val.json
│   │   │   ├── OLpWTpTC4P8_000570_000670.mp4
│   │   │   ├── xsPKW4tZZBc_002330_002430.mp4
│   │   │   ├── ...
│   │   ├── rawframes_train
│   │   ├── rawframes_val

```

关于 HVU 数据集上的训练与测试，请参照 [训练教程](/docs/zh_cn/user_guides/train_test.md)。

# EG3D (CVPR'2022)

> [Efficient geometry-aware 3D generative adversarial networks](https://openaccess.thecvf.com/content/CVPR2022/html/Chan_Efficient_Geometry-Aware_3D_Generative_Adversarial_Networks_CVPR_2022_paper.html)

> **Task**: 3D-aware Generation

Unsupervised generation of high-quality multi-view-consistent images and 3D shapes using only collections of single-view 2D photographs has been a long-standing challenge. Existing 3D GANs are either compute-intensive or make approximations that are not 3D-consistent; the former limits quality and resolution of the generated images and the latter adversely affects multi-view consistency and shape quality. In this work, we improve the computational efficiency and image quality of 3D GANs without overly relying on these approximations. We introduce an expressive hybrid explicit-implicit network architecture that, together with other design choices, synthesizes not only high-resolution multi-view-consistent images in real time but also produces high-quality 3D geometry. By decoupling feature generation and neural rendering, our framework is able to leverage state-of-the-art 2D CNN generators, such as StyleGAN2, and inherit their efficiency and expressiveness. We demonstrate state-of-the-art 3D-aware synthesis with FFHQ and AFHQ Cats, among other experiments.

<img src="https://user-images.githubusercontent.com/28132635/204269503-b66a6761-00e8-49ba-842f-65aae3110278.png"/>
</div>

|                            Model                            |    Dataset    |     Comment     | FID50k | FID50k-Camera |                                            Download                                            |
| :---------------------------------------------------------: | :-----------: | :-------------: | :----: | :-----------: | :--------------------------------------------------------------------------------------------: |
| [ShapeNet-Car](./eg3d_cvt-official-rgb_shapenet-128x128.py) | ShaperNet-Car | official weight | 5.6573 |    5.2325     | [model](https://download.openmmlab.com/mmediting/eg3d/eg3d_cvt-official-rgb_shapenet-128x128-85757f4d.pth) |

|       [AFHQ](./eg3d_cvt-official-rgb_afhq-512x512.py)       |     AFHQ      | official weight | 2.9134 |    6.4213     | [model](https://download.openmmlab.com/mmediting/eg3d/eg3d_cvt-official-rgb_afhq-512x512-ca1dd7c9.pth) |
|       [FFHQ](./eg3d_cvt-official-rgb_ffhq-512x512.py)       |     FFHQ      | official weight | 4.3076 |    6.4453     | [model](https://download.openmmlab.com/mmediting/eg3d/eg3d_cvt-official-rgb_ffhq-512x512-5a0ddcb6.pth) |

- `FID50k-Camera` denotes image generated with random sampled camera position.
- `FID50k` denotes image generated with camera position randomly sampled from the original dataset.

### Influence of FP16

All metrics are evaluated under FP32, and it's hard to determine how they will change if we use FP16.

For example, if we use FP16 at the super resolution module in [FFHQ model](./eg3d_cvt-official-rgb_ffhq-512x512.py), the output images will be slightly blurrier than the ones generated under FP32, but FID (**4.03**) will be better than FP32 ones.

## About generate images and videos with High-Level API

You can use the following command to generate sequence images with continuous changed camera position as input.

```shell
python demo/mmagic_inference_demo.py --model-name eg3d \
    --model-config configs/eg3d/eg3d_cvt-official-rgb_afhq-512x512.py \
    --model-ckpt https://download.openmmlab.com/mmediting/eg3d/eg3d_cvt-official-rgb_afhq-512x512-ca1dd7c9.pth \
    --result-out-dir eg3d_output \  # save images and videos to `eg3d_output`
    --interpolation camera \  # interpolation camera position only

--result-out-dir eg3d_output \  # save images and videos to `eg3d_output`
    --num-images 100  # generate 100 images during interpolation
```

The the following video will be saved to `eg3d_output`.

<video src="https://user-images.githubusercontent.com/28132635/204278664-b73b133b-9c3f-4a87-8750-133b7dedaebb.mp4"/>
</div>

To interpolate the camera position and style code at the same time, you can use the following command.

```shell
    --model-config configs/eg3d/eg3d_cvt-official-rgb_ffhq-512x512.py \
    --model-ckpt https://download.openmmlab.com/mmediting/eg3d/eg3d_cvt-official-rgb_ffhq-512x512-5a0ddcb6.pth \
    --interpolation both \  # interpolation camera and conditioning both

--interpolation both \  # interpolation camera and conditioning both
    --seed 233  # set random seed as 233
```

<video src="https://user-images.githubusercontent.com/28132635/205051392-e3e47ee3-bd18-4cd7-92ac-1cfc66014601.mp4"/>
</div>

If you only want to save video of depth map, you can use the following command:

```shell
    --model-config configs/eg3d/eg3d_cvt-official-rgb_shapenet-128x128.py \
    --model-ckpt https://download.openmmlab.com/mmediting/eg3d/eg3d_cvt-official-rgb_shapenet-128x128-85757f4d.pth \
    --num-images 100 \  # generate 100 images during interpolation
    --vis-mode depth  # only visualize depth image
```

--num-images 100 \  # generate 100 images during interpolation
```

<video src="https://user-images.githubusercontent.com/28132635/205051103-b0a0e540-c6b8-4f3c-a9ee-0e01ee9fd75b.mp4"/>
</div>

## How to prepare dataset

You should prepare your dataset follow the official [repo](https://github.com/NVlabs/eg3d/tree/main/dataset_preprocessing). Then preprocess the `dataset.json` with the following script:

```python
from argparse import ArgumentParser

from mmengine.fileio.io import load

    parser = ArgumentParser()
        'in-anno', type=str, help='Path to the official annotation file.')
        'out-anno', type=str, help='Path to MMagicing\'s annotation file.')

    anno = load(args.in_anno)
    label = anno['labels']

    anno_dict = {}
    for line in label:
        name, label = line

args = parser.parse_args()

        anno_dict[name] = label

    with open(args.out_anno, 'w') as file:
        json.dump(anno_dict, file)

    main()
```

```latex
@InProceedings{Chan_2022_CVPR,
    author    = {Chan, Eric R. and Lin, Connor Z. and Chan, Matthew A. and Nagano, Koki and Pan, Boxiao and De Mello, Shalini and Gallo, Orazio and Guibas, Leonidas J. and Tremblay, Jonathan and Khamis, Sameh and Karras, Tero and Wetzstein, Gordon},
    title     = {Efficient Geometry-Aware 3D Generative Adversarial Networks},
    pages     = {16123-16133}
}
```

# Contributing to MMSegmentation 1.x

1. fork and pull the latest MMSegmentation repository
2. checkout a new branch from 'dev-1.x' (do not use master branch for PRs)
3. commit your changes
4. create a PR

```{note}
If you plan to add some new features that involve large changes, it is encouraged to open an issue for discussion first.
```

### Python

Style configurations of yapf and isort can be found in [setup.cfg](./setup.cfg).

The config for a pre-commit hook is stored in [.pre-commit-config](./.pre-commit-config.yaml).

```shell
```

```shell
```

> A README.md template for releasing a project.
>
> All the fields in this README are **mandatory** for others to understand what you have achieved in this implementation.
> Please read our [Projects FAQ](../faq.md) if you still feel unclear about the requirements, or raise an [issue](https://github.com/open-mmlab/mmsegmentation/issues) to us!

>
>

Author： @xxx.

## Usage

> For a typical model, this section should contain the commands for training and testing.
> You are also suggested to dump your environment specification to env.yml by `conda env export > env.yml`.

- [MIM](https://github.com/open-mmlab/mim) v0.33 or higher

- [MMSegmentation](https://github.com/open-mmlab/mmsegmentation) v1.0.0rc2 or higher

```shell
```

```shell
mim train mmsegmentation configs/fcn_dummy-r50-d8_4xb2-40k_cityscapes-512x1024.py --work-dir work_dirs/dummy_resnet
```

```shell
mim train mmsegmentation configs/fcn_dummy-r50-d8_4xb2-40k_cityscapes-512x1024.py --work-dir work_dirs/dummy_resnet --launcher pytorch --gpus 8
```

```shell

```shell
```

```shell
mim test mmsegmentation configs/fcn_dummy-r50-d8_4xb2-40k_cityscapes-512x1024.py --work-dir work_dirs/dummy_resnet --checkpoint ${CHECKPOINT_PATH}
```

> List the results as usually done in other model's README. \[Example\](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/fcn#results-and-models
> You should claim whether this is based on the pre-trained weights, which are converted from the official release; or it's a reproduced result obtained from retraining the model in this project

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) |  mIoU | mIoU(ms+flip) | config                                                             | download                                                                                                                                                                                                                                                                                                                           |

| ------ | -------- | --------- | ------: | -------- | -------------- | ----: | ------------: | ------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| FCN    | R-50-D8  | 512x1024  |   40000 | 5.7      | 4.17           | 72.25 |         73.36 | [config](configs/fcn_dummy-r50-d8_4xb2-40k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50-d8_512x1024_40k_cityscapes/fcn_r50-d8_512x1024_40k_cityscapes_20200604_192608-efe53f0d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50-d8_512x1024_40k_cityscapes/fcn_r50-d8_512x1024_40k_cityscapes_20200604_192608.log.json) |

```bibtex
@misc{mmseg2020,
    title={{MMSegmentation}: OpenMMLab Semantic Segmentation Toolbox and Benchmark},
    author={MMSegmentation Contributors},
    howpublished = {\url{https://github.com/open-mmlab/mmsegmentation}},
}
```

> The code's design shall follow existing interfaces and convention. For example, each model component should be registered into `mmseg.registry.MODELS` and configurable via a config file.

> Each major object should contain a docstring, describing its functionality and arguments. If you have adapted the code from other open-source projects, don't forget to cite the source project in docstring and make sure your behavior is not against its license. Typically, we do not accept any code snippet under GPL license. [A Short Guide to Open Source Licenses](https://medium.com/nationwide-technology/a-short-guide-to-open-source-licenses-cf5b1c329edd)

- [ ] Test-time correctness

> If you are reproducing the result from a paper, make sure your model's inference-time performance matches that in the original paper. The weights usually could be obtained by simply renaming the keys in the official pre-trained weights. This test could be skipped though, if you are able to prove the training-time correctness and check the second milestone.

> As this template does.

> If you are reproducing the result from a paper, checking this item means that you should have trained your model from scratch based on the original paper's specification and verified that the final result matches the report within a minor error range.

> Ideally *all* the methods should have [type hints](https://www.pythontutorial.net/python-basics/python-type-hints/) and [docstrings](https://google.github.io/styleguide/pyguide.html#381-docstrings). [Example](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/utils/io.py#L9)

> Unit tests for each module are required. [Example](https://github.com/open-mmlab/mmsegmentation/blob/main/tests/test_utils/test_io.py#L14)

- [ ] Code polishing

> Refactor your code according to reviewer's comment.

> It will be parsed by MIM and Inferencer. [Example](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn.yml)

> In particular, you may have to refactor this README into a standard one. [Example](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/README.md)

# Post Training Quantization Tools

To support int8 model deployment on mobile devices,we provide the universal post training quantization tools which can convert the float32 model to int8 model.

## User Guide

Example with mobilenet, just need three steps.

### 1. Optimize model

```shell
./ncnnoptimize mobilenet.param mobilenet.bin mobilenet-opt.param mobilenet-opt.bin 0
```

### 2. Create the calibration table file

We suggest that using the verification dataset for calibration, which is more than 5000 images.

Some imagenet sample images here https://github.com/nihui/imagenet-sample-images

```shell
find images/ -type f > imagelist.txt
./ncnn2table mobilenet-opt.param mobilenet-opt.bin imagelist.txt mobilenet.table mean=[104,117,123] norm=[0.017,0.017,0.017] shape=[224,224,3] pixel=BGR thread=8 method=kl
```

* mean and norm are the values you passed to ```Mat::substract_mean_normalize()```
* shape is the blob shape of your model, [w,h] or [w,h,c]

>

```

>
    * if w and h both are given, image will be resized to exactly size.
    * if w and h both are zero or negative, image will not be resized.
    * if only h is zero or negative, image's width will scaled resize to w, keeping aspect ratio.
    * if only w is zero or negative, image's height will scaled resize to h

* pixel is the pixel format of your model, image pixels will be converted to this type before ```Extractor::input()```
* thread is the CPU thread count that could be used for parallel inference
* method is the post training quantization algorithm, kl and aciq are currently supported

If your model has multiple input nodes, you can use multiple list files and other parameters

```shell

```shell
./ncnn2table mobilenet-opt.param mobilenet-opt.bin imagelist-bgr.txt,imagelist-depth.txt mobilenet.table mean=[104,117,123],[128] norm=[0.017,0.017,0.017],[0.0078125] shape=[224,224,3],[224,224,1] pixel=BGR,GRAY thread=8 method=kl
```

### 3. Quantize model

```shell
./ncnn2int8 mobilenet-opt.param mobilenet-opt.bin mobilenet-int8.param mobilenet-int8.bin mobilenet.table
```

## use ncnn int8 inference

the ncnn library would use int8 inference automatically, nothing changed in your code

```cpp
ncnn::Net mobilenet;
mobilenet.load_param("mobilenet-int8.param");
mobilenet.load_model("mobilenet-int8.bin");
```

## mixed precision inference

Before quantize your model, comment the layer weight scale line in table file, then the layer will do the float32 inference

```
conv1_param_0 156.639840536

```

```
```

```
#conv1_param_0 156.639840536
```

# Feature pyramid networks for object detection

<details>
<summary align="right"><a href="https://openaccess.thecvf.com/content_cvpr_2017/html/Lin_Feature_Pyramid_Networks_CVPR_2017_paper.html">FPN (CVPR'2017)</a></summary>

```bibtex
@inproceedings{lin2017feature,
  title={Feature pyramid networks for object detection},
  author={Lin, Tsung-Yi and Doll{\'a}r, Piotr and Girshick, Ross and He, Kaiming and Hariharan, Bharath and Belongie, Serge},
  pages={2117--2125},
}
```

Feature pyramids are a basic component in recognition systems for detecting objects at different scales. But recent deep learning object detectors have avoided pyramid representations, in part because they are compute and memory intensive. In this paper, we exploit the inherent multi-scale, pyramidal hierarchy of deep convolutional networks to construct feature pyramids with marginal extra cost. A topdown architecture with lateral connections is developed for building high-level semantic feature maps at all scales. This architecture, called a Feature Pyramid Network (FPN), shows significant improvement as a generic feature extractor in several applications. Using FPN in a basic Faster R-CNN system, our method achieves state-of-the-art singlemodel results on the COCO detection benchmark without bells and whistles, surpassing all existing single-model entries including those from the COCO 2016 challenge winners. In addition, our method can run at 6 FPS on a GPU and thus is a practical and accurate solution to multi-scale object detection. Code will be made publicly available.

# 自定义数据集 (待更新)

## 支持新的数据格式

要支持新的数据格式，您可以将它们转换为现有的格式（DOTA 格式）。您可以选择离线（在通过脚本训练之前）或在线（实施新数据集并在训练时进行转换）进行转换。
在 MMRotate 中，我们建议将数据转换为 DOTA 格式并离线进行转换，如此您只需在数据转换后修改 config 的数据标注路径和类别即可。

### 将新数据格式重构为现有格式

最简单的方法是将数据集转换为现有数据集格式 (DOTA) 。

DOTA 格式的注解 txt 文件：

```text
...
```

每行代表一个对象，并将其记录为一个 10 维数组 `A` 。

- `A[0:8]`: 多边形的格式 `(x1, y1, x2, y2, x3, y3, x4, y4)` 。
- `A[8]`: 类别
- `A[9]`: 困难

在数据预处理之后，用户可以通过两个步骤来训练具有现有格式（例如 DOTA 格式）的自定义新数据集：

1. 修改配置文件以使用自定义数据集。
2. 检查自定义数据集的标注。

下面给出两个例子展示上述两个步骤，它使用一个自定义的 5 类 COCO 格式的数据集来训练一个现有的 Cascade Mask R-CNN R50-FPN 检测器。

#### 1. 修改配置文件以使用自定义数据集

配置文件的修改主要涉及两个方面:

1. `data` 部分。具体来说，您需要在 `data.train`, `data.val` 和 `data.test` 中显式添加 classes 字段。

2. `data` 属性变量。具体来说，特别是您需要在 `data.train`, `data.val` 和  `data.test` 中添加 classes 字段。

3. `model` 部分中的 ` num_classes`  属性变量。特别是将所有 num_classes 的默认值（例如 COCO 中的 80）覆盖到您的类别编号中。

在 `configs/my_custom_config.py` :

```python

```python

# 新配置继承了基础配置用于突出显示必要的修改

# 1. 数据集的设置

        # 注意将你的类名添加到字段 `classes`
    val=dict(

# 2. 模型设置

# 注意将你的类名添加到字段 `classes`

# 2. 模型设置
        # 显式将所有 `num_classes` 字段从 15 重写为 5。。
```

#### 2. 查看自定义数据集的标注

假设您的自定义数据集是 DOTA 格式，请确保您在自定义数据集中具有正确的标注：

- 配置文件中的 `classes` 字段应该与 txt 标注的 `A[8]` 保持完全相同的元素和相同的顺序。
  MMRotate 会自动的将 `categories` 中不连续的 `id` 映射到连续的标签索引中，所以在 `categories` 中 `name` 的字符串顺序会影响标签索引的顺序。同时，配置文件中 `classes` 的字符串顺序也会影响预测边界框可视化过程中的标签文本信息。

## 通过封装器自定义数据集

MMRotate 还支持许多数据集封装器对数据集进行混合或修改数据集的分布以进行训练。目前它支持三个数据集封装器，如下所示：

- `RepeatDataset`: 简单地重复整个数据集。
- `ClassBalancedDataset`: 以类平衡的方式重复数据集。
- `ConcatDataset`: 拼接数据集。

### 重复数据集

我们使用 `RepeatDataset` 作为封装器来重复这个数据集。例如，假设原始数据集是 `Dataset_A`，我们就重复一遍这个数据集。配置信息如下所示：

```python
        times=N,
        dataset=dict(  # 这是 Dataset_A 的原始配置信息

```python
        times=N,
            ...
        )
    )
```

### 类别平衡数据集

我们使用 `ClassBalancedDataset` 作为封装器，根据类别频率重复数据集。这个数据集的重复操作 `ClassBalancedDataset` 需要实例化函数 `self.get_cat_ids(idx)` 的支持。例如，`Dataset_A` 需要使用`oversample_thr=1e-3`，配置信息如下所示：

```python
            ...
        )
    )
```

### 拼接数据集

这里用三种方式对数据集进行拼接。

1. 如果要拼接的数据集属于同一类型且具有不同的标注文件，则可以通过如下所示的配置信息来拼接数据集：

   ````python
   )

````python
   )
   ```如果拼接后的数据集用于测试或评估，我们这种方式是可以支持对每个数据集分别进行评估。如果要测试的整个拼接数据集，如下所示您可以直接通过设置 separate_eval=False 来实现。

   ```python
   )
   ````

2. 如果您要拼接不同的数据集，您可以通过如下所示的方法对拼接数据集配置信息进行设置。

       train = [
       ],
       )
   ```如果拼接后的数据集用于测试或评估，这种方式还支持对每个数据集分别进行评估。

   ````

3. 我们也支持如下所示的方法对 `ConcatDataset` 进行明确的定义。

````

       val=dict(
   ```这种方式允许用户通过设置 `separate_eval=False` 将所有数据集转为单个数据集进行评估。

   ````

**笔记:**

1. 假设数据集在评估期间使用 `self.data_infos`，就要把选项设置为 `separate_eval=False`。因为 COCO 数据集不完全依赖 `self.data_infos` 进行评估，所以 COCO 数据集并不支持这种设置操作。没有在组合不同类型的数据集并对其进行整体评估的场景进行测试，因此我们不建议使用这样的操作。
2. 不支持评估 `ClassBalancedDataset` 和 `RepeatDataset`，所以也不支持评估这些类型的串联组合后的数据集。

一个更复杂的例子，分别将 `Dataset_A` 和 `Dataset_B` 重复 N 次和 M 次，然后将重复的数据集连接起来，如下所示。

```python
    times=N,
        ...
    )
)
    ...
)

times=N,
        ...
    )
)
    ...
)
    ...
)
    times=M,
        ...
    )
)
    train = [
    ],
)
```

```json
{
  "form": [
    {
      "id": 0,
      "text": "Registration No.",
      "box": [
          94,
          169,
          191,
          186
      ],
      "linking": [
          [
              0,
              1
          ]
      ],
      "label": "question",
      "words": [
          {
              "text": "Registration",
              "box": [
                  94,
                  169,
                  168,
                  186
              ]
          },
          {
              "text": "No.",
              "box": [
                  170,
                  169,
                  191,
                  183
              ]
          }
      ]
    },
    {
      "id": 1,
      "text": "533",
      "box": [
          209,
          169,
          236,
          182
      ],
      "label": "answer",
      "words": [
          {
              "box": [
                  209,
                  169,
                  236,

169,
          236,
          182
      ],
      "words": [
          {
              "box": [
                  209,
                  169,
                  236,
                  182
              ],
              "text": "533"
          }
      ],
          [
              0,
              1
          ]
      ]
    }
  ]
}
```

# TR3D: Towards Real-Time Indoor 3D Object Detection

> [TR3D: Towards Real-Time Indoor 3D Object Detection](https://arxiv.org/abs/2302.02858)

Recently, sparse 3D convolutions have changed 3D object detection. Performing on par with the voting-based approaches, 3D CNNs are memory-efficient and scale to large scenes better. However, there is still room for improvement. With a conscious, practice-oriented approach to problem-solving, we analyze the performance of such methods and localize the weaknesses. Applying modifications that resolve the found issues one by one, we end up with TR3D: a fast fully-convolutional 3D object detection model trained end-to-end, that achieves state-of-the-art results on the standard benchmarks, ScanNet v2, SUN RGB-D, and S3DIS. Moreover, to take advantage of both point cloud and RGB inputs, we introduce an early fusion of 2D and 3D features. We employ our fusion module to make conventional 3D object detection methods multimodal and demonstrate an impressive boost in performance. Our model with early feature fusion, which we refer to as TR3D+FF, outperforms existing 3D object detection approaches on the SUN RGB-D dataset. Overall, besides being accurate, both TR3D and TR3D+FF models are lightweight, memory-efficient, and fast, thereby marking another milestone on the way toward real-time 3D object detection.

<img src="https://user-images.githubusercontent.com/6030962/219644780-646516ec-a6c1-4ec5-9b8c-63bbc9702d05.png" width="800"/>
</div>

## Usage

Training and inference in this project were tested with `mmdet3d==1.1.0rc3`.

```bash
python tools/train.py projects/TR3D/configs/tr3d_1xb16_scannet-3d-18class.py
```

```bash
python tools/test.py projects/TR3D/configs/tr3d_1xb16_scannet-3d-18class.py ${CHECKPOINT_PATH}
```

```bash
```

|                          Backbone                          | Mem (GB) | Inf time (fps) |   AP@0.25   |   AP@0.5    |                                                                                                                                        Download                                                                                                                                         |
| :--------------------------------------------------------: | :------: | :------------: | :---------: | :---------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

| [MinkResNet34](./configs/tr3d_1xb16_scannet-3d-18class.py) |   8.6    |      23.7      | 72.9 (72.0) | 59.3 (57.4) | [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/tr3d/tr3d_1xb16_scannet-3d-18class/tr3d_1xb16_scannet-3d-18class.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/tr3d/tr3d_1xb16_scannet-3d-18class/tr3d_1xb16_scannet-3d-18class.log.json) |

| [MinkResNet34](./configs/tr3d_1xb16_sunrgbd-3d-10class.py) |   3.8    |      27.5      | 67.1 (66.3) | 50.4 (49.6) | [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/tr3d/tr3d_1xb16_sunrgbd-3d-10class/tr3d_1xb16_sunrgbd-3d-10class.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/tr3d/tr3d_1xb16_sunrgbd-3d-10class/tr3d_1xb16_sunrgbd-3d-10class.log.json) |

### S3DIS

### S3DIS

|                        Backbone                         | Mem (GB) | Inf time (fps) |   AP@0.25   |   AP@0.5    |                                                                                                                                  Download                                                                                                                                   |
| :-----------------------------------------------------: | :------: | :------------: | :---------: | :---------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

| [MinkResNet34](./configs/tr3d_1xb16_s3dis-3d-5class.py) |   15.2   |      21.0      | 74.5 (72.1) | 51.7 (47.6) | [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/tr3d/tr3d_1xb16_s3dis-3d-5class/tr3d_1xb16_s3dis-3d-5class.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/tr3d/tr3d_1xb16_s3dis-3d-5class/tr3d_1xb16_s3dis-3d-5class.log.json) |

**Note**

- We report the results across 5 train runs followed by 5 test runs. Median values are in round brackets.
- Inference time is given for a single NVidia GeForce RTX 4090 GPU.

```latex
@article{rukhovich2023tr3d,
  title={TR3D: Towards Real-Time Indoor 3D Object Detection},
  author={Rukhovich, Danila and Vorontsova, Anna and Konushin, Anton},
  journal={arXiv preprint arXiv:2302.02858},
}
```

year={2023}
}
```

# Animal Kingdom: A Large and Diverse Dataset for Animal Behavior Understanding

<details>

```bibtex
@InProceedings{Ng_2022_CVPR,
 }
```

# Lite-HRNet: A Lightweight High-Resolution Network

<details>

```bibtex
}
```

We present an efficient high-resolution network, Lite-HRNet, for human pose estimation. We start by simply applying the efficient shuffle block in ShuffleNet to HRNet (high-resolution network), yielding stronger performance over popular lightweight networks, such as MobileNet, ShuffleNet, and Small HRNet.

We find that the heavily-used pointwise (1x1) convolutions in shuffle blocks become the computational bottleneck. We introduce a lightweight unit, conditional channel weighting, to replace costly pointwise (1x1) convolutions in shuffle blocks. The complexity of channel weighting is linear w.r.t the number of channels and lower than the quadratic time complexity for pointwise convolutions. Our solution learns the weights from all the channels and over multiple resolutions that are readily available in the parallel branches in HRNet. It uses the weights as the bridge to exchange information across channels and resolutions, compensating the role played by the pointwise (1x1) convolution. Lite-HRNet demonstrates superior results on human pose estimation over popular lightweight networks. Moreover, Lite-HRNet can be easily applied to semantic segmentation task in the same lightweight manner.

<img src="https://user-images.githubusercontent.com/15977946/146520579-3e86e3af-a8b2-4071-94da-916e0ce90464.png">
</div>

# FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection

> [FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection](https://arxiv.org/abs/2112.00322)

Recently, promising applications in robotics and augmented reality have attracted considerable attention to 3D object detection from point clouds. In this paper, we present FCAF3D --- a first-in-class fully convolutional anchor-free indoor 3D object detection method. It is a simple yet effective method that uses a voxel representation of a point cloud and processes voxels with sparse convolutions. FCAF3D can handle large-scale scenes with minimal runtime through a single fully convolutional feed-forward pass. Existing 3D object detection methods make prior assumptions on the geometry of objects, and we argue that it limits their generalization ability. To eliminate prior assumptions, we propose a novel parametrization of oriented bounding boxes that allows obtaining better results in a purely data-driven way. The proposed method achieves state-of-the-art 3D object detection results in terms of mAP@0.5 on ScanNet V2 (+4.5), SUN RGB-D (+3.5), and S3DIS (+20.5) datasets.

<img src="https://user-images.githubusercontent.com/6030962/182842796-98c10576-d39c-4c2b-a15a-a04c9870919c.png" width="800"/>
</div>

We implement FCAF3D and provide the result and checkpoints on the ScanNet and SUN RGB-D dataset.

|                      Backbone                      | Mem (GB) | Inf time (fps) |   AP@0.25    |    AP@0.5    |                                                                                                                                                          Download                                                                                                                                                           |

| :------------------------------------------------: | :------: | :------------: | :----------: | :----------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| [MinkResNet34](./fcaf3d_8x2_scannet-3d-18class.py) |   10.5   |      15.7      | 69.7(70.7\*) | 55.2(56.0\*) | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/fcaf3d/fcaf3d_8x2_scannet-3d-18class/fcaf3d_8x2_scannet-3d-18class_20220805_084956.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/fcaf3d/fcaf3d_8x2_scannet-3d-18class/fcaf3d_8x2_scannet-3d-18class_20220805_084956.log.json) |

| [MinkResNet34](./fcaf3d_8x2_sunrgbd-3d-10class.py) |   6.3    |      17.9      | 63.8(63.8\*) | 47.3(48.2\*) | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/fcaf3d/fcaf3d_8x2_sunrgbd-3d-10class/fcaf3d_8x2_sunrgbd-3d-10class_20220805_165017.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/fcaf3d/fcaf3d_8x2_sunrgbd-3d-10class/fcaf3d_8x2_sunrgbd-3d-10class_20220805_165017.log.json) |

### S3DIS

|                     Backbone                     | Mem (GB) | Inf time (fps) |   AP@0.25    |    AP@0.5    |                                                                                                                                                    Download                                                                                                                                                     |

| :----------------------------------------------: | :------: | :------------: | :----------: | :----------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| [MinkResNet34](./fcaf3d_2xb8_s3dis-3d-5class.py) |   23.5   |      10.9      | 67.4(64.9\*) | 45.7(43.8\*) | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/fcaf3d/fcaf3d_8x2_s3dis-3d-5class/fcaf3d_8x2_s3dis-3d-5class_20220805_121957.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/fcaf3d/fcaf3d_8x2_s3dis-3d-5class/fcaf3d_8x2_s3dis-3d-5class_20220805_121957.log.json) |

**Note**

- We report the results across 5 train runs followed by 5 test runs. * means the results reported in the paper.

**Note**

- Inference time is given for a single NVidia RTX 4090 GPU. All models are trained on 2 GPUs.

```latex
@inproceedings{rukhovich2022fcaf3d,
  title={FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection},
}
```

# PVTv2: Improved Baselines with Pyramid Vision Transformer

<details>
<summary align="right"><a href="https://arxiv.org/abs/2106.13797">PVTV2 (CVMJ'2022)</a></summary>

```bibtex
@article{wang2022pvt,
  title={PVT v2: Improved baselines with Pyramid Vision Transformer},
  journal={Computational Visual Media},
  pages={1--10},
}
```

Transformer recently has presented encouraging progress in computer vision.
In this work, we present new baselines by improving the original Pyramid
Vision Transformer (PVTv1) by adding three designs, including (1) linear
complexity attention layer, (2) overlapping patch embedding, and (3)
convolutional feed-forward network. With these modifications, PVTv2 reduces

the computational complexity of PVTv1 to linear and achieves significant
improvements on fundamental vision tasks such as classification, detection,
and segmentation. Notably, the proposed PVTv2 achieves comparable or better
performances than recent works such as Swin Transformer. We hope this work
will facilitate state-of-the-art Transformer researches in computer vision.
Code is available at https://github.com/whai362/PVT .

# 依赖环境

MMPretrain 适用于 Linux、Windows 和 macOS。它需要 Python 3.7+、CUDA 10.2+ 和 PyTorch 1.8+。

```{note}
如果你对配置 PyTorch 环境已经很熟悉，并且已经完成了配置，可以直接进入[下一节](#安装)。
否则的话，请依照以下步骤完成配置。
```

**第 1 步** 从[官网](https://docs.conda.io/en/latest/miniconda.html)下载并安装 Miniconda。

```shell
```

**第 3 步** 按照[官方指南](https://pytorch.org/get-started/locally/)安装 PyTorch。例如：

在 GPU 平台：

```shell
```

以上命令会自动安装最新版的 PyTorch 与对应的 cudatoolkit，请检查它们是否与你的环境匹配。
```

在 CPU 平台：

```shell
```

# 安装

我们推荐用户按照我们的最佳实践来安装 MMPretrain。但除此之外，如果你想根据
你的习惯完成安装流程，也可以参见[自定义安装](#自定义安装)一节来获取更多信息。

## 最佳实践

根据具体需求，我们支持两种安装模式：

- [从源码安装（推荐）](#从源码安装)：希望基于 MMPretrain 框架开发自己的预训练任务，需要添加新的功能，比如新的模型或是数据集，或者使用我们提供的各种工具。
- [作为 Python 包安装](#作为-python-包安装)：只是希望调用 MMPretrain 的 API 接口，或者在自己的项目中导入 MMPretrain 中的模块。

### 从源码安装

这种情况下，从源码按如下方式安装 mmpretrain：

```shell
```

```{note}
`"-e"` 表示以可编辑形式安装，这样可以在不重新安装的情况下，让本地修改直接生效
```

直接使用 mim 安装即可。

```shell
```

```{note}
`mim` 是一个轻量级的命令行工具，可以根据 PyTorch 和 CUDA 版本为 OpenMMLab 算法库配置合适的环境。同时它也提供了一些对于深度学习实验很有帮助的功能。
```

## 安装多模态支持 (可选)

MMPretrain 中的多模态模型需要额外的依赖项，要安装这些依赖项，请在安装过程中添加 `[multimodal]` 参数，如下所示：

```shell
# 从源码安装

# 作为 Python 包安装
```

## 验证安装

为了验证 MMPretrain 的安装是否正确，我们提供了一些示例代码来执行模型推理。

如果你是**从源码安装**的 mmpretrain，那么直接运行以下命令进行验证：

```shell

```

## 验证安装

```shell
```

你可以看到命令行中输出了结果字典，包括 `pred_label`，`pred_score` 和 `pred_class` 三个字段。

如果你是**作为 Python 包安装**，那么可以打开你的 Python 解释器，并粘贴如下代码：

```python

model = get_model('resnet18_8xb32_in1k', device='cpu')  # 或者 device='cuda:0'
```

你会看到输出一个字典，包含预测的标签、得分及类别名。

```{note}
以上示例中，`resnet18_8xb32_in1k` 是模型名称。你可以使用 [`mmpretrain.list_models`](mmpretrain.apis.list_models) 接口来
浏览所有的模型，或者在[模型汇总](./modelzoo_statistics.md)页面进行查找。
```

## 自定义安装

### CUDA 版本

- 对于 Ampere 架构的 NVIDIA GPU，例如 GeForce 30 series 以及 NVIDIA A100，CUDA 11 是必需的。
- 对于更早的 NVIDIA GPU，CUDA 11 是向前兼容的，但 CUDA 10.2 能够提供更好的兼容性，也更加轻量。

请确保你的 GPU 驱动版本满足最低的版本需求，参阅[这张表](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cuda-major-component-versions__table-cuda-toolkit-driver-versions)。

```{note}
如果按照我们的最佳实践进行安装，CUDA 运行时库就足够了，因为我们提供相关 CUDA 代码的预编译，你不需要进行本地编译。
但如果你希望从源码进行 MMCV 的编译，或是进行其他 CUDA 算子的开发，那么就必须安装完整的 CUDA 工具链，参见
```

MMPretrain 可以仅在 CPU 环境中安装，在 CPU 模式下，你可以完成训练、测试和模型推理等所有操作。

参考 [Colab 教程](https://colab.research.google.com/github/mzr1996/mmclassification-tutorial/blob/master/1.x/MMClassification_tools.ipynb) 安装即可。

### 通过 Docker 使用 MMPretrain

MMPretrain 提供 [Dockerfile](https://github.com/open-mmlab/mmpretrain/blob/main/docker/Dockerfile)
用于构建镜像。请确保你的 [Docker 版本](https://docs.docker.com/engine/install/) >=19.03。

```shell
# 构建默认的 PyTorch 1.12.1，CUDA 11.3 版本镜像
# 如果你希望使用其他版本，请修改 Dockerfile
```

```shell
```

## 故障解决

如果你在安装过程中遇到了什么问题，请先查阅[常见问题](./notes/faq.md)。如果没有找到解决方法，可以在 GitHub
上[提出 issue](https://github.com/open-mmlab/mmpretrain/issues/new/choose)。

# Lyft 数据集

本页提供了有关在 MMDetection3D 中使用 Lyft 数据集的具体教程。

## 准备之前

您可以在[这里](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/data)下载 Lyft 3D 检测数据并解压缩所有 zip 文件。

在进行处理之前，文件夹结构应按如下方式组织：

```
├── tools
├── data
│   ├── lyft
```

```

其中 `v1.01-train` 和 `v1.01-test` 包含与 nuScenes 数据集相同的元文件，`.txt` 文件包含数据划分的信息。Lyft 不提供训练集和验证集的官方划分方案，因此 MMDetection3D 对不同场景下的不同类别的目标数量进行分析，并提供了一个数据集划分方案。`sample_submission.csv` 是用于提交到 Kaggle 评估服务器的基本文件。需要注意的是，我们遵循了 Lyft 最初的文件夹命名以实现更清楚的文件组织。请将下载下来的原始文件夹按照上述组织结构重新命名。

## 数据准备

组织 Lyft 数据集的方式和组织 nuScenes 的方式相同，首先会生成几乎具有相同结构的 `.pkl` 文件，接着需要重点关注这两个数据集之间的不同点，更多关于数据集信息文件结构的说明请参考 [nuScenes 教程](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/docs/zh_cn/advanced_guides/datasets/nuscenes_det.md)。

请通过运行下面的命令来生成 Lyft 的数据集信息文件：

```bash
```

```

请注意，上面的第二行命令用于修复损坏的 lidar 数据文件，更多细节请参考此处[讨论](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/discussion/110000)。

```
├── tools
├── data
│   ├── lyft
│   │   ├── lyft_infos_train.pkl
│   │   ├── lyft_infos_val.pkl
│   │   ├── lyft_infos_test.pkl
```

```

- `lyft_infos_train.pkl`：训练数据集信息，该字典包含两个关键字：`metainfo` 和 `data_list`。`metainfo` 包含数据集的基本信息，例如 `categories`, `dataset` 和 `info_version`。`data_list` 是由字典组成的列表，每个字典（以下简称 `info`）包含了单个样本的所有详细信息。
    - info\['lidar_points'\]\['ego2global'\]：自车到全局坐标的变换矩阵。（4x4 列表）

- info\['lidar_sweeps'\]：是一个列表，包含了扫描信息（没有标注的中间帧）。
    - info\['lidar_sweeps'\]\[i\]\['lidar_points'\]\[lidar2ego''\]：当前激光雷达传感器到自车在第 i 次扫描的变换矩阵。（4x4 列表）
    - info\['lidar_sweeps'\]\[i\]\['lidar_points'\]\['ego2global'\]：自车在第 i 次扫描到全局坐标的变换矩阵。（4x4 列表）
    - info\['lidar_sweeps'\]\[i\]\['lidar2sensor'\]：从当前帧主激光雷达到第 i 帧扫描激光雷达的变换矩阵。（4x4 列表）

- info\['images'\]\['CAM_XXX'\]\['sample_data_token'\]：图像样本数据标记。
    - info\['instances'\]\[i\]\['bbox_3d'\]：长度为 7 的列表，以 (x, y, z, l, w, h, yaw) 的顺序表示实例在激光雷达坐标系下的 3D 边界框。
    - info\['instances'\]\[i\]\['bbox_label_3d'\]：整数从 0 开始表示实例的标签，其中 -1 代表忽略该类别。

接下来将详细介绍 Lyft 数据集和 nuScenes 数据集之间的数据集信息文件中的不同点：

- `lyft_database/xxxxx.bin` 文件不存在：由于真实标注框的采样对实验的影响可以忽略不计，在 Lyft 数据集中不会提取该目录和相关的 `.bin` 文件。

- `lyft_infos_train.pkl`

  - info\['instances'\]\[i\]\['velocity'\] 不存在：Lyft 数据集中不存在速度评估信息。
  - info\['instances'\]\[i\]\['num_lidar_pts'\] 及 info\['instances'\]\[i\]\['num_radar_pts'\] 不存在。

这里仅介绍存储在训练数据文件的数据记录信息。这同样适用于验证集和测试集（没有实例）。

更多关于 `lyft_infos_xxx.pkl` 的结构信息请参考 [lyft_converter.py](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/lyft_converter.py)。

## 训练流程

Lyft 上基于 LiDAR 的 3D 检测（包括多模态方法）的训练流程与 nuScenes 几乎相同，如下所示：

```python
    dict(
    dict(
    dict(

    dict(
]
```

与 nuScenes 相似，在 Lyft 上进行训练的模型也需要 `LoadPointsFromMultiSweeps` 步骤来从连续帧中加载点云数据。另外，考虑到 Lyft 中所收集的激光雷达点的强度是无效的，因此将 `LoadPointsFromMultiSweeps` 中的 `use_dim` 默认值设置为 `[0, 1, 2, 4]`，其中前三个维度表示点的坐标，最后一个维度表示时间戳的差异。

## 评估

使用 8 个 GPU 以及 Lyft 指标评估的 PointPillars 的示例如下：

```shell
bash ./tools/dist_test.sh configs/pointpillars/pointpillars_hv_fpn_sbn-all_8xb2-2x_lyft-3d.py checkpoints/hv_pointpillars_fpn_sbn-all_2x8_2x_lyft-3d_20210517_202818-fc6904c3.pth 8
```

## 度量指标

```

## 度量指标

Lyft 提出了一个更加严格的用以评估所预测的 3D 检测框的度量指标。判断一个预测框是否是正类的基本评判标准和 KITTI 一样，如基于 3D 交并比进行评估，然而，Lyft 采用与 COCO 相似的方式来计算平均精度 -- 计算 3D 交并比在 0.5-0.95 之间的不同阈值下的平均精度。实际上，重叠部分大于 0.7 的 3D 交并比是一项对于 3D 检测方法比较严格的标准，因此整体的性能似乎会偏低。相比于其他数据集，Lyft 上不同类别的标注不平衡是导致最终结果偏低的另一个重要原因。更多关于度量指标的定义请参考[官方网址](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/overview/evaluation)。

这里将采用官方方法对 Lyft 进行评估，下面展示了一个评估结果的例子：

```
+mAPs@0.5:0.95------+--------------+
| class             | mAP@0.5:0.95 |
+-------------------+--------------+
| animal            | 0.0          |
| bicycle           | 0.099        |
| bus               | 0.177        |
| car               | 0.422        |
| emergency_vehicle | 0.0          |
| motorcycle        | 0.049        |
| other_vehicle     | 0.359        |

| pedestrian        | 0.066        |
| truck             | 0.176        |
| Overall           | 0.15         |
```

## 测试和提交

使用 8 个 GPU 在 Lyft 上测试 PointPillars 并生成对排行榜的提交的示例如下：

```shell
./tools/dist_test.sh configs/pointpillars/pointpillars_hv_fpn_sbn-all_8xb2-2x_lyft-3d.py work_dirs/pp-lyft/latest.pth 8 --cfg-options test_evaluator.jsonfile_prefix=work_dirs/pp-lyft/results_challenge  test_evaluator.csv_savepath=results/pp-lyft/results_challenge.csv
```

在生成 `work_dirs/pp-lyft/results_challenge.csv`，您可以将生成的文件提交到 Kaggle 评估服务器，请参考[官方网址](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles)获取更多细节。

同时还可以使用可视化工具将预测结果进行可视化，更多细节请参考[可视化文档](https://mmdetection3d.readthedocs.io/zh_CN/latest/useful_tools.html#visualization)。

<details>
<summary align="right">GCA (AAAI'2020)</summary>

```bibtex
}
```

<br/>

|                        算法                        |    SAD    |    MSE     |   GRAD    |   CONN    | GPU 信息 |                                                   下载                                                    |
| :------------------------------------------------: | :-------: | :--------: | :-------: | :-------: | :------: | :-------------------------------------------------------------------------------------------------------: |

|                    基线 (原文)                     |   40.62   |   0.0106   |   21.53   |   38.43   |    -     |                                                     -                                                     |
|                     GCA (原文)                     |   35.28   |   0.0091   |   16.92   |   32.53   |    -     |                                                     -                                                     |
| [基线 (复现)](./baseline_r34_4xb10-200k_comp1k.py) |   34.61   |   0.0083   |   16.21   |   32.12   |    4     | [模型](https://download.openmmlab.com/mmediting/mattors/gca/baseline_r34_4x10_200k_comp1k_SAD-34.61_20220620-96f85d56.pth) \| [日志](https://download.openmmlab.com/mmediting/mattors/gca/baseline_r34_4x10_200k_comp1k_SAD-34.61_20220620-96f85d56.log) |

|    [GCA (复现)](./gca_r34_4xb10-200k_comp1k.py)    | **33.38** | **0.0081** | **14.96** | **30.59** |    4     | [模型](https://download.openmmlab.com/mmediting/mattors/gca/gca_r34_4x10_200k_comp1k_SAD-33.38_20220615-65595f39.pth) \| [日志](https://download.openmmlab.com/mmediting/mattors/gca/gca_r34_4x10_200k_comp1k_SAD-33.38_20220615-65595f39.log) |

**其他结果**

|                                 算法                                 |  SAD  |  MSE   | GRAD  | CONN  | GPU 信息 |                                                  下载                                                   |
| :------------------------------------------------------------------: | :---: | :----: | :---: | :---: | :------: | :-----------------------------------------------------------------------------------------------------: |

| [基线 (使用 DIM 流水线)](./baseline_r34_4xb10-dimaug-200k_comp1k.py) | 49.95 | 0.0144 | 30.21 | 49.67 |    4     | [模型](https://download.openmmlab.com/mmediting/mattors/gca/baseline_dimaug_r34_4x10_200k_comp1k_SAD-49.95_20200626_231612-535c9a11.pth) \| [日志](https://download.openmmlab.com/mmediting/mattors/gca/baseline_dimaug_r34_4x10_200k_comp1k_20200626_231612.log.json) |
|    [GCA (使用 DIM 流水线)](./gca_r34_4xb10-dimaug-200k_comp1k.py)    | 49.42 | 0.0129 | 28.07 | 49.47 |    4     | [模型](https://download.openmmlab.com/mmediting/mattors/gca/gca_dimaug_r34_4x10_200k_comp1k_SAD-49.42_20200626_231422-8e9cc127.pth) \| [日志](https://download.openmmlab.com/mmediting/mattors/gca/gca_dimaug_r34_4x10_200k_comp1k_20200626_231422.log.json) |

## 快速开始

**训练**

<details>

```shell
# CPU上训练

# 单个GPU上训练

# 多个GPU上训练

```shell
# CPU上训练

# 单个GPU上训练

# 多个GPU上训练
```

**测试**

<details>

```shell
# CPU上测试

# 单个GPU上测试

# 多个GPU上测试

# 单个GPU上测试

# 多个GPU上测试
```

# 概述

本章向您介绍 MMDetection 的整体框架，并提供详细的教程链接。

## 什么是 MMDetection

![图片](https://user-images.githubusercontent.com/12907710/137271636-56ba1cd2-b110-4812-8221-b4c120320aa9.png)

MMDetection 是一个目标检测工具箱，包含了丰富的目标检测、实例分割、全景分割算法以及相关的组件和模块，下面是它的整体框架：

MMDetection 由 7 个主要部分组成，apis、structures、datasets、models、engine、evaluation 和 visualization。

- **apis** 为模型推理提供高级 API。
- **structures** 提供 bbox、mask 和 DetDataSample 等数据结构。
- **datasets** 支持用于目标检测、实例分割和全景分割的各种数据集。
  - **transforms** 包含各种数据增强变换。
  - **samplers** 定义了不同的数据加载器采样策略。
- **models** 是检测器最重要的部分，包含检测器的不同组件。
  - **detectors** 定义所有检测模型类。
  - **data_preprocessors** 用于预处理模型的输入数据。
  - **backbones** 包含各种骨干网络。
  - **necks** 包含各种模型颈部组件。
  - **dense_heads** 包含执行密集预测的各种检测头。
  - **roi_heads** 包含从 RoI 预测的各种检测头。
  - **seg_heads** 包含各种分割头。
  - **losses** 包含各种损失函数。
  - **task_modules** 为检测任务提供模块，例如 assigners、samplers、box coders 和 prior generators。
  - **layers** 提供了一些基本的神经网络层。
- **engine** 是运行时组件的一部分。

- **seg_heads** 包含各种分割头。
  - **runner** 为 [MMEngine 的执行器](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/runner.html)提供扩展。
  - **schedulers** 提供用于调整优化超参数的调度程序。
  - **optimizers** 提供优化器和优化器封装。
  - **hooks** 提供执行器的各种钩子。
- **evaluation** 为评估模型性能提供不同的指标。
- **visualization** 用于可视化检测结果。

## 如何使用本指南

以下是 MMDetection 的详细指南：

1. 安装说明见[开始你的第一步](get_started.md)。

2. MMDetection 的基本使用方法请参考以下教程。

   - [训练和测试](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/index.html#train-test)

   - [实用工具](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/index.html#useful-tools)

3. 参考以下教程深入了解：

   - [基础概念](https://mmdetection.readthedocs.io/zh_CN/latest/advanced_guides/index.html#basic-concepts)
   - [组件定制](https://mmdetection.readthedocs.io/zh_CN/latest/advanced_guides/index.html#component-customization)

4. 对于 MMDetection 2.x 版本的用户，我们提供了[迁移指南](./migration/migration.md)，帮助您完成新版本的适配。

# Retinal Images vessel Tree Extraction (RITE)

This project supports **`Retinal Images vessel Tree Extraction (RITE) `**, which can be downloaded from [here](https://opendatalab.com/RITE).

The RITE (Retinal Images vessel Tree Extraction) is a database that enables comparative studies on segmentation or classification of arteries and veins on retinal fundus images, which is established based on the public available DRIVE database (Digital Retinal Images for Vessel Extraction). RITE contains 40 sets of images, equally separated into a training subset and a test subset, the same as DRIVE. The two subsets are built from the corresponding two subsets in DRIVE. For each set, there is a fundus photograph, a vessel reference standard. The fundus photograph is inherited from DRIVE. For the training set, the vessel reference standard is a modified version of 1st_manual from DRIVE. For the test set, the vessel reference standard is 2nd_manual from DRIVE.

| Dataset Name                         | Anatomical Region | Task Type    | Modality           | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License                                                         |
| ------------------------------------ | ----------------- | ------------ | ------------------ | ------------ | --------------------- | ---------------------- | ------------ | --------------------------------------------------------------- |
| [Rite](https://opendatalab.com/RITE) | head_and_neck     | segmentation | fundus_photography | 2            | 20/-/20               | yes/-/yes              | 2013         | [CC-BY-NC 4.0](https://creativecommons.org/licenses/by-sa/4.0/) |

| background |     20     |   91.61    |    -     |    -     |    20     |   91.58   |
|   vessel   |     20     |    8.39    |    -     |    -     |    20     |   8.42    |

Note:

![rite](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/fundus_photography/rite/rite_dataset.png?raw=true)

```
@InProceedings{10.1007/978-3-642-40763-5_54,
	author={Hu, Qiao and Abr{\`a}moff, Michael D. and Garvin, Mona K.},
	title={Automated Separation of Binary Overlapping Trees in Low-Contrast Color Retinal Images},
	booktitle={Medical Image Computing and Computer-Assisted Intervention -- MICCAI 2013},
	year={2013},
	pages={436--443},
}

```

booktitle={Medical Image Computing and Computer-Assisted Intervention -- MICCAI 2013},
}

```

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `rite/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
```

- download dataset from [here](https://opendatalab.com/RITE) and decompress data to path `'data/'`.

```shell
```

```none
  ├── mmseg
  │   │   │   │   ├── rite

```

```shell
```

```shell
```

```

```shell
```

# Usage

**step 1.** Install Apache Ant.

Please check the Apache Ant existence using `ant -h` in the command line.

If there is no Apache Ant installed, please follow the command below.

```
sudo apt-get update
sudo apt-get install ant
```

Set environment variable

```
export ANT_HOME=/usr/share/ant
export PATH=${ANT_HOME}/bin:${PATH}
```

**step 2.** Build OpenCV jar package (PoseTracker only).

PoseTracker demo needs OpenCV Java, if you are interested in PoseTracker demo, you need to build OpenCV jar package first.

Using OpenCV-4.7.0 as example:

```
export OPENCV_VERSION=4.7.0
export JAVA_AWT_INCLUDE_PATH=${JAVA_HOME}
export JAVA_AWT_LIBRARY=${JAVA_HOME}
export JAVA_INCLUDE_PATH=${JAVA_HOME}/include
export JAVA_INCLUDE_PATH2=${JAVA_HOME}/include/darwin
export JAVA_JVM_LIBRARY=${JAVA_HOME}

wget https://github.com/opencv/opencv/archive/${OPENCV_VERSION}.zip
unzip ${OPENCV_VERSION}.zip && rm -rf ${OPENCV_VERSION}.zip && cd opencv-${OPENCV_VERSION}

cmake -DCMAKE_BUILD_TYPE=RELEASE -DBUILD_JAVA=ON ..
make -j8 && make install
```

**step 3.** Build MMDeploy Java API.

Please check the existence of `*.class` in `${MMDEPLOY_DIR}/csrc/mmdeploy/apis/java/mmdeploy`.

If there is no existence of `*.class`, please follow this [tutorial](../../csrc/mmdeploy/apis/java/README.md) to build Java class.

**step 4.** Build demo.

Use **ImageClassification** as example.

First, you should set your mmdeploy path, opencv path, model path and image path to `${MMDEPLOY_DIR}`, `${OPENCV_DIR}`, `${MODEL_PATH}` and `${IMAGE_PATH}`. And then follow the bash codes.

```bash
cd demo/java
export TASK=ImageClassification
export ARGS=cpu\ ${MODEL_PATH}\ ${IMAGE_PATH}

```bash
ant -DtaskName=${TASK} -DjarDir=${OPENCV_DIR}/build/bin -DlibDir=${OPENCV_DIR}/build/lib:${MMDEPLOY_DIR}/build/lib -Dcommand=${ARGS}
```

As for **PoseTracker**, you should execute:

```bash
export TASK=PoseTracker
export ARGS=cpu\ ${DET_MODEL_PATH}\  ${POSE_MODEL_PATH}\ ${VIDEO_PATH}
ant -DtaskName=${TASK} -DjarDir=${OPENCV_DIR}/build/bin -DlibDir=${OPENCV_DIR}/build/lib:${MMDEPLOY_DIR}/build/lib -Dcommand="${ARGS}"
```

# 基于 MMAction2 的生态项目

有许多研究工作和项目是基于 MMAction2 构建的。
我们列举了一些例子，展示了如何扩展 MMAction2 来适用于您自己的项目。
由于页面可能尚未完成，所以请随时通过提交PR来更新此页面。

## 作为扩展的项目

- [OTEAction2](https://github.com/openvinotoolkit/mmaction2)：用于动作识别的 OpenVINO 训练扩展。
- [PYSKL](https://github.com/kennymckormick/pyskl)：一个专注于基于骨骼点动作识别的工具箱。

## 论文相关的项目

还有一些与论文一起发布的项目。
其中一些论文发表在顶级会议（CVPR、ICCV 和 ECCV）上，其他一些也具有很高的影响力。
我们按照会议时间列出它们，方便社区参考。

- Video Swin Transformer，CVPR 2022 [\[论文\]](https://arxiv.org/abs/2106.13230)[\[github\]](https://github.com/SwinTransformer/Video-Swin-Transformer)
- Evidential Deep Learning for Open Set Action Recognition，ICCV 2021 Oral [\[论文\]](https://arxiv.org/abs/2107.10161)[\[github\]](https://github.com/Cogito2012/DEAR)
- Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspective，ICCV 2021 Oral [\[论文\]](https://arxiv.org/abs/2103.17263)[\[github\]](https://github.com/xvjiarui/VFS)

- MGSampler: An Explainable Sampling Strategy for Video Action Recognition，ICCV 2021 [\[论文\]](https://arxiv.org/abs/2104.09952)[\[github\]](https://github.com/MCG-NJU/MGSampler)
- MultiSports: A Multi-Person Video Dataset of Spatio-Temporally Localized Sports Actions，ICCV 2021 [\[论文\]](https://arxiv.org/abs/2105.07404)
- Long Short-Term Transformer for Online Action Detection，NeurIPS 2021 [\[论文\]](https://arxiv.org/abs/2107.03377)[\[github\]](https://github.com/amazon-research/long-short-term-transformer)

# MiniGPT-4

```sh
cd opencompass/multimodal/models/minigpt_4
git clone https://github.com/Vision-CAIR/MiniGPT-4.git
```

Then prepare the environement according to this [doc](https://github.com/Vision-CAIR/MiniGPT-4)

#### Slurm

```sh
cd $root
```

```sh
cd $root
```

# Cityscapes

> [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://arxiv.org/abs/1604.01685)

Visual understanding of complex urban street scenes is an enabling factor for a wide range of applications. Object detection has benefited enormously from large-scale datasets, especially in the context of deep learning. For semantic urban scene understanding, however, no current dataset adequately captures the complexity of real-world urban scenes.

To address this, we introduce Cityscapes, a benchmark suite and large-scale dataset to train and test approaches for pixel-level and instance-level semantic labeling. Cityscapes is comprised of a large, diverse set of stereo video sequences recorded in streets from 50 different cities. 5000 of these images have high quality pixel-level annotations; 20000 additional images have coarse annotations to enable methods that leverage large volumes of weakly-labeled data. Crucially, our effort exceeds previous attempts in terms of dataset size, annotation richness, scene variability, and complexity. Our accompanying empirical study provides an in-depth analysis of the dataset characteristics, as well as a performance evaluation of several state-of-the-art approaches based on our benchmark.

<img src="https://user-images.githubusercontent.com/40661020/143874154-db4484a5-9211-41f6-852a-b7f0a8c9ec26.png"/>
</div>

</div>

- All baselines were trained using 8 GPU with a batch size of 8 (1 images per GPU) using the [linear scaling rule](https://arxiv.org/abs/1706.02677) to scale the learning rate.
- All models were trained on `cityscapes_train`, and tested on `cityscapes_val`.
- 1x training schedule indicates 64 epochs which corresponds to slightly less than the 24k iterations reported in the original schedule from the [Mask R-CNN paper](https://arxiv.org/abs/1703.06870)
- COCO pre-trained weights are used to initialize.
- A conversion [script](../../tools/dataset_converters/cityscapes.py) is provided to convert Cityscapes into COCO format. Please refer to [install.md](../../docs/1_exist_data_model.md#prepare-datasets) for details.

- `CityscapesDataset` implemented three evaluation methods. `bbox` and `segm` are standard COCO bbox/mask AP. `cityscapes` is the cityscapes dataset official evaluation, which may be slightly higher than COCO.

| Backbone |  Style  | Lr schd |  Scale   | Mem (GB) | Inf time (fps) | box AP |                      Config                      |                                                                                                                          Download                                                                                                                           |
| :------: | :-----: | :-----: | :------: | :------: | :------------: | :----: | :----------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

| R-50-FPN | pytorch |   1x    | 800-1024 |   5.2    |       -        |  40.3  | [config](./faster-rcnn_r50_fpn_1x_cityscapes.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/cityscapes/faster_rcnn_r50_fpn_1x_cityscapes_20200502-829424c0.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cityscapes/faster_rcnn_r50_fpn_1x_cityscapes_20200502_114915.log.json) |

| Backbone |  Style  | Lr schd |  Scale   | Mem (GB) | Inf time (fps) | box AP | mask AP |                     Config                     |                                                                                                                                                            Download                                                                                                                                                            |

| :------: | :-----: | :-----: | :------: | :------: | :------------: | :----: | :-----: | :--------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| R-50-FPN | pytorch |   1x    | 800-1024 |   5.3    |       -        |  40.9  |  36.4   | [config](./mask-rcnn_r50_fpn_1x_cityscapes.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/cityscapes/mask_rcnn_r50_fpn_1x_cityscapes/mask_rcnn_r50_fpn_1x_cityscapes_20201211_133733-d2858245.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/cityscapes/mask_rcnn_r50_fpn_1x_cityscapes/mask_rcnn_r50_fpn_1x_cityscapes_20201211_133733.log.json) |

```latex
@inproceedings{Cordts2016Cityscapes,

```latex
   title={The Cityscapes Dataset for Semantic Urban Scene Understanding},
   author={Cordts, Marius and Omran, Mohamed and Ramos, Sebastian and Rehfeld, Timo and Enzweiler, Markus and Benenson, Rodrigo and Franke, Uwe and Roth, Stefan and Schiele, Bernt},
   booktitle={Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
   year={2016}
}
```

# TorchScript 支持情况

## Introduction of TorchScript

**TorchScript** a way to create serializable and optimizable models from PyTorch code. Any TorchScript program can be saved from a Python process and loaded in a process where there is no Python dependency. Check the [Introduction to TorchScript](https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html) for more details.

- Download libtorch from the official website [here](https://pytorch.org/get-started/locally/).

*Please note that only **Pre-cxx11 ABI** and **version 1.8.1+** on Linux platform are supported by now.*

For previous versions of libtorch, users can find through the [issue comment](https://github.com/pytorch/pytorch/issues/40961#issuecomment-1017317786). Libtorch1.8.1+cu111 as an example, extract it, expose `Torch_DIR` and add the lib path to `LD_LIBRARY_PATH` as below:

```bash

```bash

```

Note:

- If you want to save libtorch env variables to bashrc, you could run

  ```bash
  echo '# set env for libtorch' >> ~/.bashrc
  echo "export Torch_DIR=${Torch_DIR}" >> ~/.bashrc
  echo 'export LD_LIBRARY_PATH=$Torch_DIR/lib:$LD_LIBRARY_PATH' >> ~/.bashrc
  ```

```bash
cmake -DMMDEPLOY_TARGET_BACKENDS=torchscript -DTorch_DIR=${Torch_DIR} ..
```

## SDK backend

TorchScript SDK backend may be built by passing `-DMMDEPLOY_TORCHSCRIPT_SDK_BACKEND=ON` to `cmake`.

Notice that `libtorch` is sensitive to C++ ABI versions. On platforms defaulted to C++11 ABI (e.g. Ubuntu 16+) one may
pass `-DCMAKE_CXX_FLAGS="-D_GLIBCXX_USE_CXX11_ABI=0"` to `cmake` to use pre-C++11 ABI for building. In this case all
dependencies with ABI sensitive interfaces (e.g. OpenCV) must be built with pre-C++11 ABI.

## FAQs

- Error: `projects/thirdparty/libtorch/share/cmake/Caffe2/Caffe2Config.cmake:96 (message):Your installed Caffe2 version uses cuDNN but I cannot find the cuDNN libraries.  Please set the proper cuDNN prefixes and / or install cuDNN.`

  May export CUDNN_ROOT=/root/path/to/cudnn to resolve the build error.

# RegNet

> [Designing Network Design Spaces](https://arxiv.org/abs/2003.13678)

In this work, we present a new network design paradigm. Our goal is to help advance the understanding of network design and discover design principles that generalize across settings. Instead of focusing on designing individual network instances, we design network design spaces that parametrize populations of networks. The overall process is analogous to classic manual design of networks, but elevated to the design space level. Using our methodology we explore the structure aspect of network design and arrive at a low-dimensional design space consisting of simple, regular networks that we call RegNet. The core insight of the RegNet parametrization is surprisingly simple: widths and depths of good networks can be explained by a quantized linear function. We analyze the RegNet design space and arrive at interesting findings that do not match the current practice of network design. The RegNet design space provides simple and fast networks that work well across a wide range of flop regimes. Under comparable training settings and flops, the RegNet models outperform the popular EfficientNet models while being up to 5x faster on GPUs.

<img src="https://user-images.githubusercontent.com/40661020/143971942-da50f719-61e9-43bd-9468-0dbfbe80284e.png"/>
</div>

We implement RegNetX and RegNetY models in detection systems and provide their first results on Mask R-CNN, Faster R-CNN and RetinaNet.

The pre-trained models are converted from [model zoo of pycls](https://github.com/facebookresearch/pycls/blob/master/MODEL_ZOO.md).

## Usage

To use a regnet model, there are two steps to do:

1. Convert the model to ResNet-style supported by MMDetection
2. Modify backbone and neck in config accordingly

We already prepare models of FLOPs from 400M to 12G in our model zoo.

For more general usage, we also provide script `regnet2mmdet.py` in the tools directory to convert the key of models pretrained by [pycls](https://github.com/facebookresearch/pycls/) to
ResNet-style checkpoints used in MMDetection.

```bash

```bash
python -u tools/model_converters/regnet2mmdet.py ${PRETRAIN_PATH} ${STORE_PATH}
```

The users can modify the config's `depth` of backbone and corresponding keys in `arch` according to the configs in the [pycls model zoo](https://github.com/facebookresearch/pycls/blob/master/MODEL_ZOO.md).
The parameter `in_channels` in FPN can be found in the Figure 15 & 16 of the paper (`wi` in the legend).
This directory already provides some configs with their performance, using RegNetX from 800MF to 12GF level.
For other pre-trained models or self-implemented regnet models, the users are responsible to check these parameters by themselves.

**Note**: Although Fig. 15 & 16 also provide `w0`, `wa`, `wm`, `group_w`, and `bot_mul` for `arch`, they are quantized thus inaccurate, using them sometimes produces different backbone that does not match the key in the pre-trained model.

|                                       Backbone                                       |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                             Config                              |                                                                                                                                                                                          Download                                                                                                                                                                                          |

|                [R-50-FPN](../mask_rcnn/mask-rcnn_r50_fpn_1x_coco.py)                 | pytorch |   1x    |   4.4    |      12.0      |  38.2  |  34.7   |       [config](../mask_rcnn/mask-rcnn_r50_fpn_1x_coco.py)       |                                               [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_fpn_1x_coco/mask_rcnn_r50_fpn_1x_coco_20200205-d4b0c5d6.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_fpn_1x_coco/mask_rcnn_r50_fpn_1x_coco_20200205_050542.log.json)                                                |

|            [RegNetX-3.2GF-FPN](./mask-rcnn_regnetx-3.2GF_fpn_1x_coco.py)             | pytorch |   1x    |   5.0    |                |  40.3  |  36.6   |       [config](./mask-rcnn_regnetx-3.2GF_fpn_1x_coco.py)        |                           [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-3.2GF_fpn_1x_coco/mask_rcnn_regnetx-3.2GF_fpn_1x_coco_20200520_163141-2a9d1814.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-3.2GF_fpn_1x_coco/mask_rcnn_regnetx-3.2GF_fpn_1x_coco_20200520_163141.log.json)                           |

|             [RegNetX-4.0GF-FPN](./mask-rcnn_regnetx-4GF_fpn_1x_coco.py)              | pytorch |   1x    |   5.5    |                |  41.5  |  37.4   |        [config](./mask-rcnn_regnetx-4GF_fpn_1x_coco.py)         |                               [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-4GF_fpn_1x_coco/mask_rcnn_regnetx-4GF_fpn_1x_coco_20200517_180217-32e9c92d.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-4GF_fpn_1x_coco/mask_rcnn_regnetx-4GF_fpn_1x_coco_20200517_180217.log.json)                               |

|               [R-101-FPN](../mask_rcnn/mask-rcnn_r101_fpn_1x_coco.py)                | pytorch |   1x    |   6.4    |      10.3      |  40.0  |  36.1   |      [config](../mask_rcnn/mask-rcnn_r101_fpn_1x_coco.py)       |                                             [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r101_fpn_1x_coco/mask_rcnn_r101_fpn_1x_coco_20200204-1efe0ed5.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r101_fpn_1x_coco/mask_rcnn_r101_fpn_1x_coco_20200204_144809.log.json)                                              |

|            [RegNetX-6.4GF-FPN](./mask-rcnn_regnetx-6.4GF_fpn_1x_coco.py)             | pytorch |   1x    |   6.1    |                |  41.0  |  37.1   |       [config](./mask-rcnn_regnetx-6.4GF_fpn_1x_coco.py)        |                           [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-6.4GF_fpn_1x_coco/mask_rcnn_regnetx-6.4GF_fpn_1x_coco_20200517_180439-3a7aae83.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-6.4GF_fpn_1x_coco/mask_rcnn_regnetx-6.4GF_fpn_1x_coco_20200517_180439.log.json)                           |

|         [X-101-32x4d-FPN](../mask_rcnn/mask-rcnn_x101-32x4d_fpn_1x_coco.py)          | pytorch |   1x    |   7.6    |      9.4       |  41.9  |  37.5   |   [config](../mask_rcnn/mask-rcnn_x101-32x4d_fpn_1x_coco.py)    |                                 [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_x101_32x4d_fpn_1x_coco/mask_rcnn_x101_32x4d_fpn_1x_coco_20200205-478d0b67.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_x101_32x4d_fpn_1x_coco/mask_rcnn_x101_32x4d_fpn_1x_coco_20200205_034906.log.json)                                  |

|             [RegNetX-8.0GF-FPN](./mask-rcnn_regnetx-8GF_fpn_1x_coco.py)              | pytorch |   1x    |   6.4    |                |  41.7  |  37.5   |        [config](./mask-rcnn_regnetx-8GF_fpn_1x_coco.py)         |                               [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-8GF_fpn_1x_coco/mask_rcnn_regnetx-8GF_fpn_1x_coco_20200517_180515-09daa87e.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-8GF_fpn_1x_coco/mask_rcnn_regnetx-8GF_fpn_1x_coco_20200517_180515.log.json)                               |

|             [RegNetX-12GF-FPN](./mask-rcnn_regnetx-12GF_fpn_1x_coco.py)              | pytorch |   1x    |   7.4    |                |  42.2  |   38    |        [config](./mask-rcnn_regnetx-12GF_fpn_1x_coco.py)        |                             [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-12GF_fpn_1x_coco/mask_rcnn_regnetx-12GF_fpn_1x_coco_20200517_180552-b538bd8b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-12GF_fpn_1x_coco/mask_rcnn_regnetx-12GF_fpn_1x_coco_20200517_180552.log.json)                             |

| [RegNetX-3.2GF-FPN-DCN-C3-C5](./mask-rcnn_regnetx-3.2GF-mdconv-c3-c5_fpn_1x_coco.py) | pytorch |   1x    |   5.0    |                |  40.3  |  36.6   | [config](./mask-rcnn_regnetx-3.2GF-mdconv-c3-c5_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-3.2GF_fpn_mdconv_c3-c5_1x_coco/mask_rcnn_regnetx-3.2GF_fpn_mdconv_c3-c5_1x_coco_20200520_172726-75f40794.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-3.2GF_fpn_mdconv_c3-c5_1x_coco/mask_rcnn_regnetx-3.2GF_fpn_mdconv_c3-c5_1x_coco_20200520_172726.log.json) |

|                            Backbone                             |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                         Config                          |                                                                                                                                                                    Download                                                                                                                                                                    |

|    [R-50-FPN](../faster_rcnn/faster-rcnn_r50_fpn_1x_coco.py)    | pytorch |   1x    |   4.0    |      18.2      |  37.4  | [config](../faster_rcnn/faster-rcnn_r50_fpn_1x_coco.py) |                   [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130_204655.log.json)                    |

| [RegNetX-3.2GF-FPN](./faster-rcnn_regnetx-3.2GF_fpn_1x_coco.py) | pytorch |   1x    |   4.5    |                |  39.9  |  [config](./faster-rcnn_regnetx-3.2GF_fpn_1x_coco.py)   | [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/faster_rcnn_regnetx-3.2GF_fpn_1x_coco/faster_rcnn_regnetx-3.2GF_fpn_1x_coco_20200517_175927-126fd9bf.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/faster_rcnn_regnetx-3.2GF_fpn_1x_coco/faster_rcnn_regnetx-3.2GF_fpn_1x_coco_20200517_175927.log.json) |

| [RegNetX-3.2GF-FPN](./faster-rcnn_regnetx-3.2GF_fpn_2x_coco.py) | pytorch |   2x    |   4.5    |                |  41.1  |  [config](./faster-rcnn_regnetx-3.2GF_fpn_2x_coco.py)   | [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/faster_rcnn_regnetx-3.2GF_fpn_2x_coco/faster_rcnn_regnetx-3.2GF_fpn_2x_coco_20200520_223955-e2081918.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/faster_rcnn_regnetx-3.2GF_fpn_2x_coco/faster_rcnn_regnetx-3.2GF_fpn_2x_coco_20200520_223955.log.json) |

### RetinaNet

|                           Backbone                            |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                       Config                        |                                                                                                                                                                Download                                                                                                                                                                |

|     [R-50-FPN](../retinanet/retinanet_r50_fpn_1x_coco.py)     | pytorch |   1x    |   3.8    |      16.6      |  36.5  | [config](../retinanet/retinanet_r50_fpn_1x_coco.py) |                     [model](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r50_fpn_1x_coco/retinanet_r50_fpn_1x_coco_20200130-c2398f9e.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r50_fpn_1x_coco/retinanet_r50_fpn_1x_coco_20200130_002941.log.json)                      |

| [RegNetX-800MF-FPN](./retinanet_regnetx-800MF_fpn_1x_coco.py) | pytorch |   1x    |   2.5    |                |  35.6  | [config](./retinanet_regnetx-800MF_fpn_1x_coco.py)  | [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/retinanet_regnetx-800MF_fpn_1x_coco/retinanet_regnetx-800MF_fpn_1x_coco_20200517_191403-f6f91d10.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/retinanet_regnetx-800MF_fpn_1x_coco/retinanet_regnetx-800MF_fpn_1x_coco_20200517_191403.log.json) |

| [RegNetX-1.6GF-FPN](./retinanet_regnetx-1.6GF_fpn_1x_coco.py) | pytorch |   1x    |   3.3    |                |  37.3  | [config](./retinanet_regnetx-1.6GF_fpn_1x_coco.py)  | [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/retinanet_regnetx-1.6GF_fpn_1x_coco/retinanet_regnetx-1.6GF_fpn_1x_coco_20200517_191403-37009a9d.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/retinanet_regnetx-1.6GF_fpn_1x_coco/retinanet_regnetx-1.6GF_fpn_1x_coco_20200517_191403.log.json) |

| [RegNetX-3.2GF-FPN](./retinanet_regnetx-3.2GF_fpn_1x_coco.py) | pytorch |   1x    |   4.2    |                |  39.1  | [config](./retinanet_regnetx-3.2GF_fpn_1x_coco.py)  | [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/retinanet_regnetx-3.2GF_fpn_1x_coco/retinanet_regnetx-3.2GF_fpn_1x_coco_20200520_163141-cb1509e8.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/retinanet_regnetx-3.2GF_fpn_1x_coco/retinanet_regnetx-3.2GF_fpn_1x_coco_20200520_163141.log.json) |

### Pre-trained models

|      Method       |                                 Backbone                                 |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                            Config                             |                                                                                                                                                                                                Download                                                                                                                                                                                                |

|    Faster RCNN    |    [RegNetX-400MF-FPN](./faster-rcnn_regnetx-400MF_fpn_ms-3x_coco.py)    | pytorch |   3x    |   2.3    |                |  37.1  |    -    |    [config](./faster-rcnn_regnetx-400MF_fpn_ms-3x_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/faster_rcnn_regnetx-400MF_fpn_mstrain_3x_coco/faster_rcnn_regnetx-400MF_fpn_mstrain_3x_coco_20210526_095112-e1967c37.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/faster_rcnn_regnetx-400MF_fpn_mstrain_3x_coco/faster_rcnn_regnetx-400MF_fpn_mstrain_3x_coco_20210526_095112.log.json)             |

|    Faster RCNN    |    [RegNetX-800MF-FPN](./faster-rcnn_regnetx-800MF_fpn_ms-3x_coco.py)    | pytorch |   3x    |   2.8    |                |  38.8  |    -    |    [config](./faster-rcnn_regnetx-800MF_fpn_ms-3x_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/faster_rcnn_regnetx-800MF_fpn_mstrain_3x_coco/faster_rcnn_regnetx-800MF_fpn_mstrain_3x_coco_20210526_095118-a2c70b20.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/faster_rcnn_regnetx-800MF_fpn_mstrain_3x_coco/faster_rcnn_regnetx-800MF_fpn_mstrain_3x_coco_20210526_095118.log.json)             |

|    Faster RCNN    |    [RegNetX-1.6GF-FPN](./faster-rcnn_regnetx-1.6GF_fpn_ms-3x_coco.py)    | pytorch |   3x    |   3.4    |                |  40.5  |    -    |    [config](./faster-rcnn_regnetx-1.6GF_fpn_ms-3x_coco.py)    |                                     [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/faster_rcnn_regnetx-1.6GF_fpn_mstrain_3x_coco/faster_rcnn_regnetx-1_20210526_095325-94aa46cc.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/faster_rcnn_regnetx-1.6GF_fpn_mstrain_3x_coco/faster_rcnn_regnetx-1_20210526_095325.log.json)                                     |

|    Faster RCNN    |    [RegNetX-3.2GF-FPN](./faster-rcnn_regnetx-3.2GF_fpn_ms-3x_coco.py)    | pytorch |   3x    |   4.4    |                |  42.3  |    -    |    [config](./faster-rcnn_regnetx-3.2GF_fpn_ms-3x_coco.py)    |                                     [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/faster_rcnn_regnetx-3.2GF_fpn_mstrain_3x_coco/faster_rcnn_regnetx-3_20210526_095152-e16a5227.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/faster_rcnn_regnetx-3.2GF_fpn_mstrain_3x_coco/faster_rcnn_regnetx-3_20210526_095152.log.json)                                     |

|    Faster RCNN    |      [RegNetX-4GF-FPN](./faster-rcnn_regnetx-4GF_fpn_ms-3x_coco.py)      | pytorch |   3x    |   4.9    |                |  42.8  |    -    |     [config](./faster-rcnn_regnetx-4GF_fpn_ms-3x_coco.py)     |                 [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/faster_rcnn_regnetx-4GF_fpn_mstrain_3x_coco/faster_rcnn_regnetx-4GF_fpn_mstrain_3x_coco_20210526_095201-65eaf841.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/faster_rcnn_regnetx-4GF_fpn_mstrain_3x_coco/faster_rcnn_regnetx-4GF_fpn_mstrain_3x_coco_20210526_095201.log.json)                 |

|     Mask RCNN     |  [RegNetX-400MF-FPN](./mask-rcnn_regnetx-400MF_fpn_ms-poly-3x_coco.py)   | pytorch |   3x    |   2.5    |                |  37.6  |  34.4   |  [config](./mask-rcnn_regnetx-400MF_fpn_ms-poly-3x_coco.py)   |       [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-400MF_fpn_mstrain-poly_3x_coco/mask_rcnn_regnetx-400MF_fpn_mstrain-poly_3x_coco_20210601_235443-8aac57a4.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-400MF_fpn_mstrain-poly_3x_coco/mask_rcnn_regnetx-400MF_fpn_mstrain-poly_3x_coco_20210601_235443.log.json)       |

|     Mask RCNN     |  [RegNetX-800MF-FPN](./mask-rcnn_regnetx-800MF_fpn_ms-poly-3x_coco.py)   | pytorch |   3x    |   2.9    |                |  39.5  |  36.1   |  [config](./mask-rcnn_regnetx-800MF_fpn_ms-poly-3x_coco.py)   |       [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-800MF_fpn_mstrain-poly_3x_coco/mask_rcnn_regnetx-800MF_fpn_mstrain-poly_3x_coco_20210602_210641-715d51f5.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-800MF_fpn_mstrain-poly_3x_coco/mask_rcnn_regnetx-800MF_fpn_mstrain-poly_3x_coco_20210602_210641.log.json)       |

|     Mask RCNN     |  [RegNetX-1.6GF-FPN](./mask-rcnn_regnetx-1.6GF_fpn_ms-poly-3x_coco.py)   | pytorch |   3x    |   3.6    |                |  40.9  |  37.5   |  [config](./mask-rcnn_regnetx-1.6GF_fpn_ms-poly-3x_coco.py)   |                                    [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-1.6GF_fpn_mstrain-poly_3x_coco/mask_rcnn_regnetx-1_20210602_210641-6764cff5.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-1.6GF_fpn_mstrain-poly_3x_coco/mask_rcnn_regnetx-1_20210602_210641.log.json)                                    |

|     Mask RCNN     |     [RegNetX-3.2GF-FPN](./mask-rcnn_regnetx-3.2GF_fpn_ms-3x_coco.py)     | pytorch |   3x    |   5.0    |                |  43.1  |  38.7   |     [config](./mask-rcnn_regnetx-3.2GF_fpn_ms-3x_coco.py)     |                 [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-3.2GF_fpn_mstrain_3x_coco/mask_rcnn_regnetx-3.2GF_fpn_mstrain_3x_coco_20200521_202221-99879813.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-3.2GF_fpn_mstrain_3x_coco/mask_rcnn_regnetx-3.2GF_fpn_mstrain_3x_coco_20200521_202221.log.json)                 |

|     Mask RCNN     |    [RegNetX-4GF-FPN](./mask-rcnn_regnetx-4GF_fpn_ms-poly-3x_coco.py)     | pytorch |   3x    |   5.1    |                |  43.4  |  39.2   |   [config](./mask-rcnn_regnetx-4GF_fpn_ms-poly-3x_coco.py)    |           [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-4GF_fpn_mstrain-poly_3x_coco/mask_rcnn_regnetx-4GF_fpn_mstrain-poly_3x_coco_20210602_032621-00f0331c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/mask_rcnn_regnetx-4GF_fpn_mstrain-poly_3x_coco/mask_rcnn_regnetx-4GF_fpn_mstrain-poly_3x_coco_20210602_032621.log.json)           |

| Cascade Mask RCNN | [RegNetX-400MF-FPN](./cascade-mask-rcnn_regnetx-400MF_fpn_ms-3x_coco.py) | pytorch |   3x    |   4.3    |                |  41.6  |  36.4   | [config](./cascade-mask-rcnn_regnetx-400MF_fpn_ms-3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/cascade_mask_rcnn_regnetx-400MF_fpn_mstrain_3x_coco/cascade_mask_rcnn_regnetx-400MF_fpn_mstrain_3x_coco_20210715_211619-5142f449.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/cascade_mask_rcnn_regnetx-400MF_fpn_mstrain_3x_coco/cascade_mask_rcnn_regnetx-400MF_fpn_mstrain_3x_coco_20210715_211619.log.json) |

| Cascade Mask RCNN | [RegNetX-800MF-FPN](./cascade-mask-rcnn_regnetx-800MF_fpn_ms-3x_coco.py) | pytorch |   3x    |   4.8    |                |  42.8  |  37.6   | [config](./cascade-mask-rcnn_regnetx-800MF_fpn_ms-3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/cascade_mask_rcnn_regnetx-800MF_fpn_mstrain_3x_coco/cascade_mask_rcnn_regnetx-800MF_fpn_mstrain_3x_coco_20210715_211616-dcbd13f4.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/cascade_mask_rcnn_regnetx-800MF_fpn_mstrain_3x_coco/cascade_mask_rcnn_regnetx-800MF_fpn_mstrain_3x_coco_20210715_211616.log.json) |

| Cascade Mask RCNN | [RegNetX-1.6GF-FPN](./cascade-mask-rcnn_regnetx-1.6GF_fpn_ms-3x_coco.py) | pytorch |   3x    |   5.4    |                |  44.5  |  39.0   | [config](./cascade-mask-rcnn_regnetx-1.6GF_fpn_ms-3x_coco.py) |                         [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/cascade_mask_rcnn_regnetx-1.6GF_fpn_mstrain_3x_coco/cascade_mask_rcnn_regnetx-1_20210715_211616-75f29a61.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/cascade_mask_rcnn_regnetx-1.6GF_fpn_mstrain_3x_coco/cascade_mask_rcnn_regnetx-1_20210715_211616.log.json)                         |

| Cascade Mask RCNN | [RegNetX-3.2GF-FPN](./cascade-mask-rcnn_regnetx-3.2GF_fpn_ms-3x_coco.py) | pytorch |   3x    |   6.4    |                |  45.8  |  40.0   | [config](./cascade-mask-rcnn_regnetx-3.2GF_fpn_ms-3x_coco.py) |                         [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/cascade_mask_rcnn_regnetx-3.2GF_fpn_mstrain_3x_coco/cascade_mask_rcnn_regnetx-3_20210715_211616-b9c2c58b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/cascade_mask_rcnn_regnetx-3.2GF_fpn_mstrain_3x_coco/cascade_mask_rcnn_regnetx-3_20210715_211616.log.json)                         |

| Cascade Mask RCNN |   [RegNetX-4GF-FPN](./cascade-mask-rcnn_regnetx-4GF_fpn_ms-3x_coco.py)   | pytorch |   3x    |   6.9    |                |  45.8  |  40.0   |  [config](./cascade-mask-rcnn_regnetx-4GF_fpn_ms-3x_coco.py)  |     [model](https://download.openmmlab.com/mmdetection/v2.0/regnet/cascade_mask_rcnn_regnetx-4GF_fpn_mstrain_3x_coco/cascade_mask_rcnn_regnetx-4GF_fpn_mstrain_3x_coco_20210715_212034-cbb1be4c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/regnet/cascade_mask_rcnn_regnetx-4GF_fpn_mstrain_3x_coco/cascade_mask_rcnn_regnetx-4GF_fpn_mstrain_3x_coco_20210715_212034.log.json)     |

### Notice

1. The models are trained using a different weight decay, i.e., `weight_decay=5e-5` according to the setting in ImageNet training. This brings improvement of at least 0.7 AP absolute but does not improve the model using ResNet-50.

2. RetinaNets using RegNets are trained with learning rate 0.02 with gradient clip. We find that using learning rate 0.02 could improve the results by at least 0.7 AP absolute and gradient clip is necessary to stabilize the training. However, this does not improve the performance of ResNet-50-FPN RetinaNet.

```latex
@article{radosavovic2020designing,
    title={Designing Network Design Spaces},
    author={Ilija Radosavovic and Raj Prateek Kosaraju and Ross Girshick and Kaiming He and Piotr Dollár},
    eprint={2003.13678},
}
```

> [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://arxiv.org/pdf/2010.11929.pdf)

<a href="https://github.com/google-research/vision_transformer">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/backbones/vit.py#L98">Code Snippet</a>

<img src="https://user-images.githubusercontent.com/24582831/142903144-f80a12cc-8698-48ab-843c-49dedf558121.png" width="70%"/>
</div>

## Usage

We provide a script [`vit2mmseg.py`](../../tools/model_converters/vit2mmseg.py) in the tools directory to convert the key of models from [timm](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/vision_transformer.py) to MMSegmentation style.

```shell
python tools/model_converters/vit2mmseg.py ${PRETRAIN_PATH} ${STORE_PATH}
```

E.g.

```shell
python tools/model_converters/vit2mmseg.py https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-vitjx/jx_vit_base_p16_224-80ecf9dd.pth pretrain/jx_vit_base_p16_224-80ecf9dd.pth
```

### ADE20K

```

### ADE20K

| Method  | Backbone          | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                               | download                                                                                                                                                                                                                                                                                                                   |

| UPerNet | ViT-B + MLN       | 512x512   |   80000 | 9.20     | 6.94           | V100   | 47.71 |         49.51 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/vit/vit_vit-b16_mln_upernet_8xb2-80k_ade20k-512x512.py)      | [model](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_vit-b16_mln_512x512_80k_ade20k/upernet_vit-b16_mln_512x512_80k_ade20k_20210624_130547-0403cee1.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_vit-b16_mln_512x512_80k_ade20k/20210624_130547.log.json)                |

| UPerNet | ViT-B + MLN       | 512x512   |  160000 | 9.20     | 7.58           | V100   | 46.75 |         48.46 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/vit/vit_vit-b16_mln_upernet_8xb2-160k_ade20k-512x512.py)     | [model](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_vit-b16_mln_512x512_160k_ade20k/upernet_vit-b16_mln_512x512_160k_ade20k_20210624_130547-852fa768.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_vit-b16_mln_512x512_160k_ade20k/20210623_192432.log.json)             |

| UPerNet | ViT-B + LN + MLN  | 512x512   |  160000 | 9.21     | 6.82           | V100   | 47.73 |         49.95 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/vit/vit_vit-b16-ln_mln_upernet_8xb2-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_vit-b16_ln_mln_512x512_160k_ade20k/upernet_vit-b16_ln_mln_512x512_160k_ade20k_20210621_172828-f444c077.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_vit-b16_ln_mln_512x512_160k_ade20k/20210621_172828.log.json)    |

| UPerNet | DeiT-S            | 512x512   |   80000 | 4.68     | 29.85          | V100   | 42.96 |         43.79 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/vit/vit_deit-s16_upernet_8xb2-80k_ade20k-512x512.py)         | [model](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_deit-s16_512x512_80k_ade20k/upernet_deit-s16_512x512_80k_ade20k_20210624_095228-afc93ec2.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_deit-s16_512x512_80k_ade20k/20210624_095228.log.json)                         |

| UPerNet | DeiT-S            | 512x512   |  160000 | 4.68     | 29.19          | V100   | 42.87 |         43.79 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/vit/vit_deit-s16_upernet_8xb2-160k_ade20k-512x512.py)        | [model](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_deit-s16_512x512_160k_ade20k/upernet_deit-s16_512x512_160k_ade20k_20210621_160903-5110d916.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_deit-s16_512x512_160k_ade20k/20210621_160903.log.json)                      |

| UPerNet | DeiT-S + MLN      | 512x512   |  160000 | 5.69     | 11.18          | V100   | 43.82 |         45.07 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/vit/vit_deit-s16_mln_upernet_8xb2-160k_ade20k-512x512.py)    | [model](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_deit-s16_mln_512x512_160k_ade20k/upernet_deit-s16_mln_512x512_160k_ade20k_20210621_161021-fb9a5dfb.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_deit-s16_mln_512x512_160k_ade20k/20210621_161021.log.json)          |

| UPerNet | DeiT-S + LN + MLN | 512x512   |  160000 | 5.69     | 12.39          | V100   | 43.52 |         45.01 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/vit/vit_deit-s16-ln_mln_upernet_8xb2-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_deit-s16_ln_mln_512x512_160k_ade20k/upernet_deit-s16_ln_mln_512x512_160k_ade20k_20210621_161021-c0cd652f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_deit-s16_ln_mln_512x512_160k_ade20k/20210621_161021.log.json) |

| UPerNet | DeiT-B            | 512x512   |   80000 | 7.75     | 9.69           | V100   | 45.24 |         46.73 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/vit/vit_deit-b16_upernet_8xb2-80k_ade20k-512x512.py)         | [model](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_deit-b16_512x512_80k_ade20k/upernet_deit-b16_512x512_80k_ade20k_20210624_130529-1e090789.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_deit-b16_512x512_80k_ade20k/20210624_130529.log.json)                         |

| UPerNet | DeiT-B            | 512x512   |  160000 | 7.75     | 10.39          | V100   | 45.36 |         47.16 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/vit/vit_deit-b16_upernet_8xb2-160k_ade20k-512x512.py)        | [model](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_deit-b16_512x512_160k_ade20k/upernet_deit-b16_512x512_160k_ade20k_20210621_180100-828705d7.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_deit-b16_512x512_160k_ade20k/20210621_180100.log.json)                      |

| UPerNet | DeiT-B + MLN      | 512x512   |  160000 | 9.21     | 7.78           | V100   | 45.46 |         47.16 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/vit/vit_deit-b16_mln_upernet_8xb2-160k_ade20k-512x512.py)    | [model](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_deit-b16_mln_512x512_160k_ade20k/upernet_deit-b16_mln_512x512_160k_ade20k_20210621_191949-4e1450f3.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_deit-b16_mln_512x512_160k_ade20k/20210621_191949.log.json)          |

| UPerNet | DeiT-B + LN + MLN | 512x512   |  160000 | 9.21     | 7.75           | V100   | 45.37 |         47.23 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/vit/vit_deit-b16-ln_mln_upernet_8xb2-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_deit-b16_ln_mln_512x512_160k_ade20k/upernet_deit-b16_ln_mln_512x512_160k_ade20k_20210623_153535-8a959c14.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/vit/upernet_deit-b16_ln_mln_512x512_160k_ade20k/20210623_153535.log.json) |

```bibtex
@article{dosoViTskiy2020,
  author={DosoViTskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and  Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and Uszkoreit, Jakob and Houlsby, Neil},

# How to evaluate model

After converting a PyTorch model to a backend model, you may evaluate backend models with `tools/test.py`

Install MMDeploy according to [get-started](../get_started.md) instructions.
And convert the PyTorch model or ONNX model to the backend model by following the [guide](convert_model.md).

## Usage

```shell
${DEPLOY_CFG} \
${MODEL_CFG} \
--model ${BACKEND_MODEL_FILES} \
[--out ${OUTPUT_PKL_FILE}] \
[--format-only] \
[--metrics ${METRICS}] \
[--show] \
[--show-dir ${OUTPUT_IMAGE_DIR}] \
[--show-score-thr ${SHOW_SCORE_THR}] \
--device ${DEVICE} \
[--cfg-options ${CFG_OPTIONS}] \
[--metric-options ${METRIC_OPTIONS}]
[--log2file work_dirs/output.txt]
[--batch-size ${BATCH_SIZE}]
[--speed-test] \
[--warmup ${WARM_UP}] \
[--log-interval ${LOG_INTERVERL}] \
```

## Description of all arguments

- `deploy_cfg`: The config for deployment.
- `model_cfg`: The config of the model in OpenMMLab codebases.

```

- `--model`: The backend model file. For example, if we convert a model to TensorRT, we need to pass the model file with ".engine" suffix.
- `--out`:  The path to save output results in pickle format. (The results will be saved only if this argument is given)
- `--format-only`: Whether format the output results without evaluation or not. It is useful when you want to format the result to a specific format and submit it to the test server
- `--metrics`: The metrics to evaluate the model defined in OpenMMLab codebases. e.g. "segm", "proposal" for COCO in mmdet, "precision", "recall", "f1_score", "support" for single label dataset in mmpretrain.
- `--show`: Whether to show the evaluation result on the screen.

- `--show-dir`: The directory to save the evaluation result. (The results will be saved only if this argument is given)
- `--show-score-thr`: The threshold determining whether to show detection bounding boxes.
- `--device`: The device that the model runs on. Note that some backends restrict the device. For example, TensorRT must run on cuda.
- `--cfg-options`: Extra or overridden settings that will be merged into the current deploy config.
- `--metric-options`: Custom options for evaluation. The key-value pair in xxx=yyy
  format will be kwargs for dataset.evaluate() function.
- `--log2file`: log evaluation results (and speed) to file.
- `--batch-size`: the batch size for inference, which would override `samples_per_gpu` in data config. Default is `1`. Note that not all models support `batch_size>1`.
- `--speed-test`:  Whether to activate speed test.

- `--warmup`: warmup before counting inference elapse, require setting speed-test first.
- `--log-interval`: The interval between each log, require setting speed-test first.

\* Other arguments in `tools/test.py` are used for speed test. They have no concern with evaluation.

## Example

```shell
    {MMPRETRAIN_DIR}/configs/resnet/resnet50_b32x8_imagenet.py \
    --model model.onnx \
    --out out.pkl \
    --speed-test
```

## Note

- The performance of each model in [OpenMMLab](https://openmmlab.com/) codebases can be found in the document of each codebase.

# MMPose 模型部署

- [MMPose 模型部署](#mmpose-模型部署)
    - [安装 mmpose](#安装-mmpose)

[MMPose](https://github.com/open-mmlab/mmpose/tree/main)，又称 `mmpose`，是一个基于 PyTorch 的姿态估计的开源工具箱，是 [OpenMMLab](https://openmmlab.com/) 项目的成员之一。

## 安装

### 安装 mmpose

请参考[官网安装指南](https://mmpose.readthedocs.io/en/latest/installation.html#best-practices)。

```shell

```shell
```

## 模型转换

你可以使用 [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/tree/main/tools/deploy.py) 把 mmpose 模型一键式转换为推理后端模型。

以下，我们将演示如何把 `hrnet` 转换为 onnx 模型。

```shell

    --show
```

转换的关键之一是使用正确的配置文件。项目中已内置了各后端部署[配置文件](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmpose)。
文件的命名模式是：

```
```

其中：

在上例中，你也可以把 `hrnet` 转为其他后端模型。比如使用`pose-detection_tensorrt_static-256x192.py`，把模型转为 tensorrt 模型。

```{tip}
```

## 模型规范

```{tip}
```

## 模型规范

上例中的`mmdeploy_models/mmpose/ort`，结构如下：

```
```

重要的是：

## 模型推理

### 后端模型推理

```python

```

> TODO

## 模型支持列表

```

> TODO

## 模型支持列表

MMPose为姿态估计提供了大量可以从[模型库](https://mmpose.readthedocs.io/en/latest/model_zoo.html)中找到的预测训练模型。本指南将演示**如何执行推理**，或使用训练过的模型对提供的图像或视频运行姿态估计。

有关在标准数据集上测试现有模型的说明，请参阅本指南。

在MMPose，模型由配置文件定义，而其已计算好的参数存储在权重文件（checkpoint file）中。您可以在[模型库](https://mmpose.readthedocs.io/en/latest/model_zoo.html)中找到模型配置文件和相应的权重文件的URL。我们建议从使用HRNet模型的[配置文件](https://github.com/open-mmlab/mmpose/blob/main/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192.py)和[权重文件](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192-81c58e40_20220909.pth)开始。

## 推理器：统一的推理接口

MMPose提供了一个被称为`MMPoseInferencer`的、全面的推理API。这个API使得用户得以使用所有MMPose支持的模型来对图像和视频进行模型推理。此外，该API可以完成推理结果自动化，并方便用户保存预测结果。

### 基本用法

`MMPoseInferencer`可以在任何Python程序中被用来执行姿态估计任务。以下是在一个在Python Shell中使用预训练的人体姿态模型对给定图像进行推理的示例。

```python
from mmpose.apis import MMPoseInferencer

img_path = 'tests/data/coco/000000000785.jpg'   # 将img_path替换给你自己的路径

# 使用模型别名创建推断器

```python

inferencer = MMPoseInferencer('human')

# MMPoseInferencer采用了惰性推断方法，在给定输入时创建一个预测生成器
result_generator = inferencer(img_path, show=True)
result = next(result_generator)
```

如果一切正常，你将在一个新窗口中看到下图：

![inferencer_result_coco](https://user-images.githubusercontent.com/26127467/220008302-4a57fd44-0978-408e-8351-600e5513316a.jpg)

`result` 变量是一个包含两个键值 `'visualization'` 和 `'predictions'` 的字典。

- `'visualization'` 键对应的值是一个列表，该列表：
  - 包含可视化结果，例如输入图像、估计姿态的标记，以及可选的预测热图。
  - 如果没有指定 `return_vis` 参数，该列表将保持为空。
- `'predictions'` 键对应的值是：
  - 一个包含每个检测实例的预估关键点的列表。

`result` 字典的结构如下所示：

```python
result = {
    'visualization': [
        # 元素数量：batch_size（默认为1）
        vis_image_1,
        ...
    ],
    'predictions': [
        # 每张图像的姿态估计结果
        [
            # 每个检测到的实例的姿态信息
            # 元素数量：检测到的实例数

vis_image_1,
        ...
    ],
        [
            {'keypoints': ...,  # 实例 1
            'keypoint_scores': ...,
            ...
            },
            {'keypoints': ...,  # 实例 2
            ...
            },
        ]
    ...
    ]
}
```

还可以使用用于用于推断的**命令行界面工具**（CLI, command-line interface）: `demo/inferencer_demo.py`。这个工具允许用户使用以下命令使用相同的模型和输入执行推理：

```python
python demo/inferencer_demo.py 'tests/data/coco/000000000785.jpg' \
    --pose2d 'human' --show --pred-out-dir 'predictions'
```

预测结果将被保存在路径`predictions/000000000785.json`。作为一个API，`inferencer_demo.py`的输入参数与`MMPoseInferencer`的相同。前者能够处理一系列输入类型，包括以下内容：

- 图像路径

- 视频路径

- 文件夹路径（这会导致该文件夹中的所有图像都被推断出来）

- 表示图像的 numpy array (在命令行界面工具中未支持)

- 表示图像的 numpy array 列表 (在命令行界面工具中未支持)

- 摄像头（在这种情况下，输入参数应该设置为`webcam`或`webcam:{CAMERA_ID}`）

- 图像路径

- 视频路径

当输入对应于多个图像时，例如输入为**视频**或**文件夹**路径时，推理生成器必须被遍历，以便推理器对视频/文件夹中的所有帧/图像进行推理。以下是一个示例：

```python
folder_path = 'tests/data/coco'

result_generator = inferencer(folder_path, show=True)
results = [result for result in result_generator]
```

在这个示例中，`inferencer` 接受 `folder_path` 作为输入，并返回一个生成器对象（`result_generator`），用于生成推理结果。通过遍历 `result_generator` 并将每个结果存储在 `results` 列表中，您可以获得视频/文件夹中所有帧/图像的推理结果。

### 自定义姿态估计模型

`MMPoseInferencer`提供了几种可用于自定义所使用的模型的方法：

```python
# 使用模型别名构建推断器

# 使用模型配置名构建推断器
inferencer = MMPoseInferencer('td-hm_hrnet-w32_8xb64-210e_coco-256x192')

# 使用模型配置文件和权重文件的路径或 URL 构建推断器
inferencer = MMPoseInferencer(
    pose2d='configs/body_2d_keypoint/topdown_heatmap/coco/' \
           'td-hm_hrnet-w32_8xb64-210e_coco-256x192.py',

    pose2d_weights='https://download.openmmlab.com/mmpose/top_down/' \
                   'hrnet/hrnet_w32_coco_256x192-c78dce93_20200708.pth'
)
```

模型别名的完整列表可以在模型别名部分中找到。

此外，自顶向下的姿态估计器还需要一个对象检测模型。`MMPoseInferencer`能够推断用MMPose支持的数据集训练的模型的实例类型，然后构建必要的对象检测模型。用户也可以通过以下方式手动指定检测模型:

```python
# 通过别名指定检测模型
# 可用的别名包括“human”、“hand”、“face”、“animal”、
# 以及mmdet中定义的任何其他别名
    # 假设姿态估计器是在自定义数据集上训练的
    pose2d='custom_human_pose_estimator.py',
    pose2d_weights='custom_human_pose_estimator.pth',
    det_model='human'
)

# 使用模型配置名称指定检测模型
    pose2d='human',
    det_model='yolox_l_8x8_300e_coco',
    det_cat_ids=[0],  # 指定'human'类的类别id
)

# 使用模型配置文件和权重文件的路径或URL构建推断器

pose2d='human',
)

    det_model=f'{PATH_TO_MMDET}/configs/yolox/yolox_l_8x8_300e_coco.py',
    det_weights='https://download.openmmlab.com/mmdetection/v2.0/' \
                'yolox/yolox_l_8x8_300e_coco/' \
                'yolox_l_8x8_300e_coco_20211126_140236-d3bd2b23.pth',
)
```

### 转储结果

在执行姿态估计推理任务之后，您可能希望保存结果以供进一步分析或处理。本节将指导您将预测的关键点和可视化结果保存到本地。

要将预测保存在<mark>JSON文件</mark>中，在运行`MMPoseInferencer`的实例`inferencer`时使用`pred_out_dir`参数:

```python
result_generator = inferencer(img_path, pred_out_dir='predictions')
```

预测结果将以JSON格式保存在`predictions/`文件夹中，每个文件以相应的输入图像或视频的名称命名。

对于更高级的场景，还可以直接从`inferencer`返回的`result`字典中访问预测结果。其中，`predictions`包含输入图像或视频中每个单独实例的预测关键点列表。然后，您可以使用您喜欢的方法操作或存储这些结果。

请记住，如果你想将<mark>可视化图像</mark>和预测文件保存在一个文件夹中，你可以使用`out_dir`参数：

```python
result_generator = inferencer(img_path, out_dir='output')
```

在这种情况下，可视化图像将保存在`output/visualization/`文件夹中，而预测将存储在`output/forecasts/`文件夹中。

### 可视化

推理器`inferencer`可以自动对输入的图像或视频进行预测。可视化结果可以显示在一个新的窗口中，并保存在本地。

要在新窗口中查看可视化结果，请使用以下代码：

请注意：

- 如果输入视频来自网络摄像头，默认情况下将在新窗口中显示可视化结果，以此让用户看到输入

- 如果平台上没有GUI，这个步骤可能会卡住

要将可视化结果保存在本地，可以像这样指定`vis_out_dir`参数:

```python
result_generator = inferencer(img_path, vis_out_dir='vis_results')
```

输入图片或视频的可视化预测结果将保存在`vis_results/`文件夹中

在开头展示的滑雪图中，姿态的可视化估计结果由关键点（用实心圆描绘）和骨架（用线条表示）组成。这些视觉元素的默认大小可能不会产生令人满意的结果。用户可以使用`radius`和`thickness`参数来调整圆的大小和线的粗细，如下所示：

```python
result_generator = inferencer(img_path, show=True, radius=4, thickness=2)
```

### 推理器参数

```python
```

### 推理器参数

`MMPoseInferencer`提供了各种自定义姿态估计、可视化和保存预测结果的参数。下面是<mark>初始化</mark>推断器时可用的参数列表及对这些参数的描述：

| Argument         | Description                                                  |
| ---------------- | ------------------------------------------------------------ |
| `pose2d`         | 指定 2D 姿态估计模型的模型别名、配置文件名称或配置文件路径。 |
| `pose2d_weights` | 指定 2D 姿态估计模型权重文件的URL或本地路径。                |
| `pose3d`         | 指定 3D 姿态估计模型的模型别名、配置文件名称或配置文件路径。 |
| `pose3d_weights` | 指定 3D 姿态估计模型权重文件的URL或本地路径。                |
| `det_model`      | 指定对象检测模型的模型别名、配置文件名或配置文件路径。       |
| `det_weights`    | 指定对象检测模型权重文件的 URL 或本地路径。                  |
| `det_cat_ids`    | 指定与要检测的对象类对应的类别 id 列表。                     |
| `device`         | 执行推理的设备。如果为 `None`，推理器将选择最合适的一个。    |
| `scope`          | 定义模型模块的名称空间                                       |

推理器被设计用于可视化和保存预测。以下表格列出了在使用 `MMPoseInferencer` <mark>进行推断</mark>时可用的参数列表，以及它们与 2D 和 3D 推理器的兼容性：

| 参数                     | 描述                                                                                                                       | 2D  | 3D  |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------- | --- | --- |
| `show`                   | 控制是否在弹出窗口中显示图像或视频。                                                                                       | ✔️  | ✔️  |
| `radius`                 | 设置可视化关键点的半径。                                                                                                   | ✔️  | ✔️  |

| `thickness`              | 确定可视化链接的厚度。                                                                                                     | ✔️  | ✔️  |
| `kpt_thr`                | 设置关键点分数阈值。分数超过此阈值的关键点将被显示。                                                                       | ✔️  | ✔️  |
| `draw_bbox`              | 决定是否显示实例的边界框。                                                                                                 | ✔️  | ✔️  |
| `draw_heatmap`           | 决定是否绘制预测的热图。                                                                                                   | ✔️  | ❌  |
| `black_background`       | 决定是否在黑色背景上显示预估的姿势。                                                                                       | ✔️  | ❌  |

| `skeleton_style`         | 设置骨架样式。可选项包括 'mmpose'（默认）和 'openpose'。                                                                   | ✔️  | ❌  |
| `use_oks_tracking`       | 决定是否在追踪中使用OKS作为相似度测量。                                                                                    | ❌  | ✔️  |
| `tracking_thr`           | 设置追踪的相似度阈值。                                                                                                     | ❌  | ✔️  |
| `norm_pose_2d`           | 决定是否将边界框缩放至数据集的平均边界框尺寸，并将边界框移至数据集的平均边界框中心。                                       | ❌  | ✔️  |
| `rebase_keypoint_height` | 决定是否将最低关键点的高度置为 0。                                                                                         | ❌  | ✔️  |

| `return_vis`             | 决定是否在结果中包含可视化图像。                                                                                           | ✔️  | ✔️  |
| `vis_out_dir`            | 定义保存可视化图像的文件夹路径。如果未设置，将不保存可视化图像。                                                           | ✔️  | ✔️  |
| `return_datasample`      | 决定是否以 `PoseDataSample` 格式返回预测。                                                                                 | ✔️  | ✔️  |
| `pred_out_dir`           | 指定保存预测的文件夹路径。如果未设置，将不保存预测。                                                                       | ✔️  | ✔️  |
| `out_dir`                | 如果 `vis_out_dir` 或 `pred_out_dir` 未设置，它们将分别设置为 `f'{out_dir}/visualization'` 或 `f'{out_dir}/predictions'`。 | ✔️  | ✔️  |

### 模型别名

MMPose为常用模型提供了一组预定义的别名。在初始化 `MMPoseInferencer` 时，这些别名可以用作简略的表达方式，而不是指定完整的模型配置名称。下面是可用的模型别名及其对应的配置名称的列表：

### 模型别名

| 别名      | 配置文件名称                                       | 对应任务                        | 姿态估计模型  | 检测模型            |
| --------- | -------------------------------------------------- | ------------------------------- | ------------- | ------------------- |
| animal    | rtmpose-m_8xb64-210e_ap10k-256x256                 | Animal pose estimation          | RTMPose-m     | RTMDet-m            |
| human     | rtmpose-m_8xb256-420e_aic-coco-256x192             | Human pose estimation           | RTMPose-m     | RTMDet-m            |
| face      | rtmpose-m_8xb64-60e_wflw-256x256                   | Face keypoint detection         | RTMPose-m     | yolox-s             |
| hand      | rtmpose-m_8xb32-210e_coco-wholebody-hand-256x256   | Hand keypoint detection         | RTMPose-m     | ssdlite_mobilenetv2 |

| wholebody | rtmpose-m_8xb64-270e_coco-wholebody-256x192        | Human wholebody pose estimation | RTMPose-m     | RTMDet-m            |
| vitpose   | td-hm_ViTPose-base-simple_8xb64-210e_coco-256x192  | Human pose estimation           | ViTPose-base  | RTMDet-m            |
| vitpose-s | td-hm_ViTPose-small-simple_8xb64-210e_coco-256x192 | Human pose estimation           | ViTPose-small | RTMDet-m            |
| vitpose-b | td-hm_ViTPose-base-simple_8xb64-210e_coco-256x192  | Human pose estimation           | ViTPose-base  | RTMDet-m            |
| vitpose-l | td-hm_ViTPose-large-simple_8xb64-210e_coco-256x192 | Human pose estimation           | ViTPose-large | RTMDet-m            |
| vitpose-h | td-hm_ViTPose-huge-simple_8xb64-210e_coco-256x192  | Human pose estimation           | ViTPose-huge  | RTMDet-m            |

此外，用户可以使用命令行界面工具显示所有可用的别名，使用以下命令:

```shell
python demo/inferencer_demo.py --show-alias
```

# DNLNet

> [Disentangled Non-Local Neural Networks](https://arxiv.org/abs/2006.06668)

<a href="https://github.com/yinmh17/DNL-Semantic-Segmentation">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/decode_heads/dnl_head.py#L88">Code Snippet</a>

The non-local block is a popular module for strengthening the context modeling ability of a regular convolutional neural network. This paper first studies the non-local block in depth, where we find that its attention computation can be split into two terms, a whitened pairwise term accounting for the relationship between two pixels and a unary term representing the saliency of every pixel. We also observe that the two terms trained alone tend to model different visual clues, e.g. the whitened pairwise term learns within-region relationships while the unary term learns salient boundaries. However, the two terms are tightly coupled in the non-local block, which hinders the learning of each. Based on these findings, we present the disentangled non-local block, where the two terms are decoupled to facilitate learning for both terms. We demonstrate the effectiveness of the decoupled design on various tasks, such as semantic segmentation on Cityscapes, ADE20K and PASCAL Context, object detection on COCO, and action recognition on Kinetics.

<img src="https://user-images.githubusercontent.com/24582831/142900944-b8d93301-d2ce-488e-a461-b0813f96be49.png" width="70%"/>
</div>

## Results and models (in progress)

| DNLNet | R-50-D8  | 512x1024  |   40000 |      7.3 | 2.56           | V100   | 78.61 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dnlnet/dnl_r50-d8_4xb2-40k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r50-d8_512x1024_40k_cityscapes/dnl_r50-d8_512x1024_40k_cityscapes_20200904_233629-53d4ea93.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r50-d8_512x1024_40k_cityscapes/dnl_r50-d8_512x1024_40k_cityscapes-20200904_233629.log.json)     |

| DNLNet | R-101-D8 | 512x1024  |   40000 |     10.9 | 1.96           | V100   | 78.31 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dnlnet/dnl_r101-d8_4xb2-40k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r101-d8_512x1024_40k_cityscapes/dnl_r101-d8_512x1024_40k_cityscapes_20200904_233629-9928ffef.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r101-d8_512x1024_40k_cityscapes/dnl_r101-d8_512x1024_40k_cityscapes-20200904_233629.log.json) |

| DNLNet | R-50-D8  | 769x769   |   40000 |      9.2 | 1.50           | V100   | 78.44 | 80.27         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dnlnet/dnl_r50-d8_4xb2-40k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r50-d8_769x769_40k_cityscapes/dnl_r50-d8_769x769_40k_cityscapes_20200820_232206-0f283785.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r50-d8_769x769_40k_cityscapes/dnl_r50-d8_769x769_40k_cityscapes-20200820_232206.log.json)         |

| DNLNet | R-101-D8 | 769x769   |   40000 |     12.6 | 1.02           | V100   | 76.39 | 77.77         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dnlnet/dnl_r101-d8_4xb2-40k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r101-d8_769x769_40k_cityscapes/dnl_r101-d8_769x769_40k_cityscapes_20200820_171256-76c596df.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r101-d8_769x769_40k_cityscapes/dnl_r101-d8_769x769_40k_cityscapes-20200820_171256.log.json)     |

| DNLNet | R-50-D8  | 512x1024  |   80000 |        - | -              | V100   | 79.33 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dnlnet/dnl_r50-d8_4xb2-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r50-d8_512x1024_80k_cityscapes/dnl_r50-d8_512x1024_80k_cityscapes_20200904_233629-58b2f778.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r50-d8_512x1024_80k_cityscapes/dnl_r50-d8_512x1024_80k_cityscapes-20200904_233629.log.json)     |

| DNLNet | R-101-D8 | 512x1024  |   80000 |        - | -              | V100   | 80.41 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dnlnet/dnl_r101-d8_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r101-d8_512x1024_80k_cityscapes/dnl_r101-d8_512x1024_80k_cityscapes_20200904_233629-758e2dd4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r101-d8_512x1024_80k_cityscapes/dnl_r101-d8_512x1024_80k_cityscapes-20200904_233629.log.json) |

| DNLNet | R-50-D8  | 769x769   |   80000 |        - | -              | V100   | 79.36 | 80.70         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dnlnet/dnl_r50-d8_4xb2-80k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r50-d8_769x769_80k_cityscapes/dnl_r50-d8_769x769_80k_cityscapes_20200820_011925-366bc4c7.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r50-d8_769x769_80k_cityscapes/dnl_r50-d8_769x769_80k_cityscapes-20200820_011925.log.json)         |

| DNLNet | R-101-D8 | 769x769   |   80000 |        - | -              | V100   | 79.41 | 80.68         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dnlnet/dnl_r101-d8_4xb2-80k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r101-d8_769x769_80k_cityscapes/dnl_r101-d8_769x769_80k_cityscapes_20200821_051111-95ff84ab.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r101-d8_769x769_80k_cityscapes/dnl_r101-d8_769x769_80k_cityscapes-20200821_051111.log.json)     |

### ADE20K

### ADE20K

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                  | download                                                                                                                                                                                                                                                                                                                     |

| DNLNet | R-50-D8  | 512x512   |   80000 |      8.8 | 20.66          | V100   | 41.76 | 42.99         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dnlnet/dnl_r50-d8_4xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r50-d8_512x512_80k_ade20k/dnl_r50-d8_512x512_80k_ade20k_20200826_183354-1cf6e0c1.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r50-d8_512x512_80k_ade20k/dnl_r50-d8_512x512_80k_ade20k-20200826_183354.log.json)         |

| DNLNet | R-101-D8 | 512x512   |   80000 |     12.8 | 12.54          | V100   | 43.76 | 44.91         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dnlnet/dnl_r101-d8_4xb4-80k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r101-d8_512x512_80k_ade20k/dnl_r101-d8_512x512_80k_ade20k_20200826_183354-d820d6ea.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r101-d8_512x512_80k_ade20k/dnl_r101-d8_512x512_80k_ade20k-20200826_183354.log.json)     |

| DNLNet | R-50-D8  | 512x512   |  160000 |        - | -              | V100   | 41.87 | 43.01         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dnlnet/dnl_r50-d8_4xb4-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r50-d8_512x512_160k_ade20k/dnl_r50-d8_512x512_160k_ade20k_20200826_183350-37837798.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r50-d8_512x512_160k_ade20k/dnl_r50-d8_512x512_160k_ade20k-20200826_183350.log.json)     |

| DNLNet | R-101-D8 | 512x512   |  160000 |        - | -              | V100   | 44.25 | 45.78         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dnlnet/dnl_r101-d8_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r101-d8_512x512_160k_ade20k/dnl_r101-d8_512x512_160k_ade20k_20200826_183350-ed522c61.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dnlnet/dnl_r101-d8_512x512_160k_ade20k/dnl_r101-d8_512x512_160k_ade20k-20200826_183350.log.json) |

## Notes

This example is to reproduce ["Disentangled Non-Local Neural Networks"](https://arxiv.org/abs/2006.06668) for semantic segmentation. It is still in progress.

```bibtex
@misc{yin2020disentangled,
    title={Disentangled Non-Local Neural Networks},
    author={Minghao Yin and Zhuliang Yao and Yue Cao and Xiu Li and Zheng Zhang and Stephen Lin and Han Hu},
    booktitle={ECCV}
}
```

<details>

```bibtex
}
```

<details>

```bibtex

author={Wang, Yangang and Peng, Cong and Liu, Yebin},
}
```

| Arch                                                       | Input Size | PCK@0.2 |  AUC  |  EPE  |                            ckpt                            |                            log                            |
| :--------------------------------------------------------- | :--------: | :-----: | :---: | :---: | :--------------------------------------------------------: | :-------------------------------------------------------: |

| [pose_mobilenet_v2](/configs/hand_2d_keypoint/topdown_heatmap/onehand10k/td-hm_mobilenetv2_8xb64-210e_onehand10k-256x256.py) |  256x256   |  0.986  | 0.537 | 28.56 | [ckpt](https://download.openmmlab.com/mmpose/hand/mobilenetv2/mobilenetv2_onehand10k_256x256-f3a3d90e_20210330.pth) | [log](https://download.openmmlab.com/mmpose/hand/mobilenetv2/mobilenetv2_onehand10k_256x256_20210330.log.json) |

}
```

<details>

```bibtex
}
```

}
```

| [pose_cspnext_m](/configs/body_2d_keypoint/topdown_heatmap/crowdpose/cspnext-m_udp_8xb64-210e_crowpose-256x192.py) |  256x192   | 0.662 |      0.821      |      0.723      | 0.759  | 0.675  | 0.539  | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-m_udp-crowdpose_pt-in1k_210e-256x192-f591079f_20230123.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-m_udp-crowdpose_pt-in1k_210e-256x192-f591079f_20230123.json) |

# Darknet To NCNN Conversion Tools

This is a standalone darknet2ncnn converter without additional dependency.

Support yolov4, yolov4-tiny, yolov3, yolov3-tiny and enet-coco.cfg (EfficientNetB0-Yolov3).

Another conversion tool based on darknet can be found at: [darknet2ncnn](https://github.com/xiangweizeng/darknet2ncnn)

## Usage

```
Usage: darknet2ncnn [darknetcfg] [darknetweights] [ncnnparam] [ncnnbin] [merge_output]
        [darknetcfg]     .cfg file of input darknet model.
        [darknetweights] .weights file of input darknet model.
        [cnnparam]       .param file of output ncnn model.
        [ncnnbin]        .bin file of output ncnn model.
        [merge_output]   merge all output yolo layers into one, enabled by default.
```

## Example

### 1. Convert yolov4-tiny cfg and weights

[merge_output]   merge all output yolo layers into one, enabled by default.
```

## Example

Download pre-trained [yolov4-tiny.cfg](https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4-tiny.cfg) and [yolov4-tiny.weights](https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v4_pre/yolov4-tiny.weights) or with your own trained weight.

Convert cfg and weights:
```
./darknet2ncnn yolov4-tiny.cfg yolov4-tiny.weights yolov4-tiny.param yolov4-tiny.bin 1
```

If succeeded, the output would be:
```
Loading cfg...
WARNING: The ignore_thresh=0.700000 of yolo0 is too high. An alternative value 0.25 is written instead.
WARNING: The ignore_thresh=0.700000 of yolo1 is too high. An alternative value 0.25 is written instead.
Loading weights...
Converting model...
83 layers, 91 blobs generated.
NOTE: The input of darknet uses: mean_vals=0 and norm_vals=1/255.f.
NOTE: Remember to use ncnnoptimize for better performance.

```

### 2. Optimize graphic

```
./ncnnoptimize yolov4-tiny.param yolov4-tiny.bin yolov4-tiny-opt.param yolov4-tiny-opt.bin 0
```

### 3. Test

build examples/yolov4.cpp and test with:

```
./yolov4 dog.jpg
```

The result will be:

![](https://github.com/Tencent/ncnn/blob/master/tools/darknet/output.jpg)

## How to run with benchncnn

Set 2=0.3 for Yolov3DetectionOutput layer.

```json
{"fieldBBs": [{"poly_points": [[435, 1406], [466, 1406], [466, 1439], [435, 1439]], "type": "fieldCheckBox", "id": "f0", "isBlank": 1}, {"poly_points": [[435, 1444], [469, 1444], [469, 1478], [435, 1478]], "type": "fieldCheckBox", "id": "f1", "isBlank": 1}],
 "textBBs": [{"poly_points": [[1183, 1337], [2028, 1345], [2032, 1395], [1186, 1398]], "type": "text", "id": "t0"}, {"poly_points": [[492, 1336], [809, 1338], [809, 1379], [492, 1378]], "type": "text", "id": "t1"}, {"poly_points": [[512, 1375], [798, 1376], [798, 1405], [512, 1404]], "type": "textInst", "id": "t2"}], "imageFilename": "007182398_00026.jpg", "transcriptions": {"f0": "\u00bf\u00bf\u00bf \u00bf\u00bf\u00bf 18/1/49 \u00bf\u00bf\u00bf\u00bf\u00bf", "f1": "U.S. Navy 53rd. Naval Const. Batt.", "t0": "APPLICATION FOR HEADSTONE OR MARKER", "t1": "ORIGINAL"}}
```

# SCNet

> [SCNet: Training Inference Sample Consistency for Instance Segmentation](https://arxiv.org/abs/2012.10150)

Cascaded architectures have brought significant performance improvement in object detection and instance segmentation. However, there are lingering issues regarding the disparity in the Intersection-over-Union (IoU) distribution of the samples between training and inference. This disparity can potentially exacerbate detection accuracy. This paper proposes an architecture referred to as Sample Consistency Network (SCNet) to ensure that the IoU distribution of the samples at training time is close to that at inference time. Furthermore, SCNet incorporates feature relay and utilizes global contextual information to further reinforce the reciprocal relationships among classifying, detecting, and segmenting sub-tasks. Extensive experiments on the standard COCO dataset reveal the effectiveness of the proposed method over multiple evaluation metrics, including box AP, mask AP, and inference speed. In particular, while running 38% faster, the proposed SCNet improves the AP of the box and mask predictions by respectively 1.3 and 2.3 points compared to the strong Cascade Mask R-CNN baseline.

<img src="https://user-images.githubusercontent.com/40661020/143974840-8fed75f3-661e-4e2a-a210-acf4ab5f42a3.png"/>
</div>

## Dataset

SCNet requires COCO and [COCO-stuff](http://calvin.inf.ed.ac.uk/wp-content/uploads/data/cocostuffdataset/stuffthingmaps_trainval2017.zip) dataset for training. You need to download and extract it in the COCO dataset path.

```none
├── mmdet
├── tools
├── data
│   ├── coco
```

```

|    Backbone     |  Style  | Lr schd | Mem (GB) | Inf speed (fps) | box AP | mask AP | TTA box AP | TTA mask AP |                    Config                    |                                                                                                                                           Download                                                                                                                                           |

| :-------------: | :-----: | :-----: | :------: | :-------------: | :----: | :-----: | :--------: | :---------: | :------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|    R-50-FPN     | pytorch |   1x    |   7.0    |       6.2       |  43.5  |  39.2   |    44.8    |    40.9     |     [config](./scnet_r50_fpn_1x_coco.py)     |                 [model](https://download.openmmlab.com/mmdetection/v2.0/scnet/scnet_r50_fpn_1x_coco/scnet_r50_fpn_1x_coco-c3f09857.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/scnet/scnet_r50_fpn_1x_coco/scnet_r50_fpn_1x_coco_20210117_192725.log.json)                 |

|    R-50-FPN     | pytorch |   20e   |   7.0    |       6.2       |  44.5  |  40.0   |    45.8    |    41.5     |    [config](./scnet_r50_fpn_20e_coco.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/scnet/scnet_r50_fpn_20e_coco/scnet_r50_fpn_20e_coco-a569f645.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/scnet/scnet_r50_fpn_20e_coco/scnet_r50_fpn_20e_coco_20210116_060148.log.json)               |
|    R-101-FPN    | pytorch |   20e   |   8.9    |       5.8       |  45.8  |  40.9   |    47.3    |    42.7     |    [config](./scnet_r101_fpn_20e_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/scnet/scnet_r101_fpn_20e_coco/scnet_r101_fpn_20e_coco-294e312c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/scnet/scnet_r101_fpn_20e_coco/scnet_r101_fpn_20e_coco_20210118_175824.log.json)             |

| X-101-64x4d-FPN | pytorch |   20e   |   13.2   |       4.9       |  47.5  |  42.3   |    48.9    |    44.0     | [config](./scnet_x101-64x4d_fpn_20e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/scnet/scnet_x101_64x4d_fpn_20e_coco/scnet_x101_64x4d_fpn_20e_coco-fb09dec9.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/scnet/scnet_x101_64x4d_fpn_20e_coco/scnet_x101_64x4d_fpn_20e_coco_20210120_045959.log.json) |

### Notes

- Training hyper-parameters are identical to those of [HTC](https://github.com/open-mmlab/mmdetection/tree/main/configs/htc).
- TTA means Test Time Augmentation, which applies horizontal flip and multi-scale testing. Refer to [config](./scnet_r50_fpn_1x_coco.py).

We provide the code for reproducing experiment results of [SCNet](https://arxiv.org/abs/2012.10150).

```latex
@inproceedings{vu2019cascade,
  title={SCNet: Training Inference Sample Consistency for Instance Segmentation},

```latex
  author={Vu, Thang and Haeyong, Kang and Yoo, Chang D},
}
```

</div>

本项目将支持使用 MMPose 来生成骨架图片，用于姿态引导的 AI 图像生成。

当前已支持：

欢迎分享更多姿态引导的 AIGC 项目给我们！

## 快速上手

### 生成 Openpose 风格的骨架图片

#### Step 1: 准备

运行以下命令准备项目：

```shell
cd mmpose

```

#### Step 2: 生成骨架图片

运行以下命令生成骨架图片：

```shell
```

输入图片与生成骨架图片如下:

```

```shell
```

</div>

### 生成 MMPose 风格的骨架图片

**环境要求：**

- GCC >= 7.5

运行以下命令安装项目：

```shell
```

最终的文件结构如下：

```shell
```

#### Step 2: 生成姿态骨架图片

运行以下命令生成姿态骨架图片：

```shell
# 生成骨架图片
```

```

```shell
# 生成骨架图片
```

更多详细信息可以查看 [RTMPose](../rtmpose/README_CN.md)。

<img src="https://user-images.githubusercontent.com/13503330/222318807-a0a2a87a-cfc4-46cc-b647-e8f8bc44cfe3.jpg" width=450 /><img src='https://user-images.githubusercontent.com/13503330/222318943-6dba5f52-158a-427a-8222-03628addc051.jpg' width=450/>
</div>

### 使用 T2I-Adapter

T2I- Adapter 在线试玩请点击 [这里](https://huggingface.co/spaces/Adapter/T2I-Adapter)

</div>

## 结果展示

</div>

## 结果展示

</div>

# TorchServe 部署

为了使用 [`TorchServe`](https://pytorch.org/serve/) 部署一个 `MMPretrain` 模型，需要进行以下几步：

## 1. 转换 MMPretrain 模型至 TorchServe

```shell
```

```{note}
${MODEL_STORE} 需要是一个文件夹的绝对路径。
```

示例：

```shell
```

## 2. 构建 `mmpretrain-serve` docker 镜像

```shell
```

## 3. 运行 `mmpretrain-serve` 镜像

请参考官方文档 [基于 docker 运行 TorchServe](https://github.com/pytorch/serve/blob/master/docker/README.md#running-torchserve-in-a-production-docker-environment).

```

为了使镜像能够使用 GPU 资源，需要安装 [nvidia-docker](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html)。之后可以传递 `--gpus` 参数以在 GPU 上运。

示例：

```shell
--cpus 8 \
```

```{note}
`realpath ./checkpoints` 是 "./checkpoints" 的绝对路径，你可以将其替换为你保存 TorchServe 模型的目录的绝对路径。
```

参考 [该文档](https://github.com/pytorch/serve/blob/master/docs/rest_api.md) 了解关于推理 (8080)，管理 (8081) 和指标 (8082) 等 API 的信息。

## 4. 测试部署

```shell
```

您应该获得类似于以下内容的响应：

```json
{

## 4. 测试部署

```shell
```

```json
{
}
```

另外，你也可以使用 `test_torchserver.py` 来比较 TorchServe 和 PyTorch 的结果，并进行可视化。

```shell
```

示例：

```shell
```

# Preparing Multisports

```BibTeX
}
```

For basic dataset information, please refer to the official [project](https://deeperaction.github.io/datasets/multisports.html) and the [paper](https://arxiv.org/abs/2105.07404).
Before we start, please make sure that the directory is located at `$MMACTION2/tools/data/multisports/`.

First of all, you have to download annotations and videos to `$MMACTION2/data/multisports` on the official [website](https://github.com/MCG-NJU/MultiSports), please also download the Person Boxes and put it to `$MMACTION2/data/multisports`.

Before this step, please make sure the folder structure looks like:

```
mmaction2
├── tools
├── data
│   |   |   ├──...
│   |   ├── test
│   |   |   ├──...
```

Then, you can use the following command to uncompress.

```shell
```

## Step 3. Convert Annotations

you can run the following script to convert annotations and proposals as we need.

```shell
```

```

```shell
```

After the whole data process, you will get the videos and annotation files for MultiSports.

In the context of the whole project (for MultiSports only), the folder structure will look like:

```
mmaction2
├── tools
├── data

│   |   |   ├──...
│   |   ├── test
│   |   |   ├──...
```

We don't need the zip files under the project, you can handle them as you want.
For training and evaluating on MultiSports, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).

# Contributing to MMPreTrain

- [Contributing to MMPreTrain](#contributing-to-mmpretrain)
  - [Workflow](#workflow)
    - [C++ and CUDA](#c-and-cuda)
  - [Pre-commit Hook](#pre-commit-hook)

Thanks for your interest in contributing to MMPreTrain! All kinds of contributions are welcome, including but not limited to the following.

We recommend the potential contributors follow this workflow for contribution.

1. Fork and pull the latest MMPreTrain repository, follow [get started](https://mmpretrain.readthedocs.io/en/latest/get_started.html) to setup the environment.
2. Checkout a new branch (**do not use the master or dev branch** for PRs)

```bash
git checkout -b xxxx # xxxx is the name of new branch
```

3. Edit the related files follow the code style mentioned below

```bash
```

4. Use [pre-commit hook](https://pre-commit.com/) to check and format your changes.
5. Commit your changes
6. Create a PR with related information

### Python

Style configurations of yapf and isort can be found in [setup.cfg](https://github.com/open-mmlab/mmpretrain/blob/main/setup.cfg).

## Pre-commit Hook

The config for a pre-commit hook is stored in [.pre-commit-config](https://github.com/open-mmlab/mmpretrain/blob/main/.pre-commit-config.yaml).

```shell
```

```shell
```

# Preparing ActivityNet

```BibTeX
@article{Heilbron2015ActivityNetAL,
  title={ActivityNet: A large-scale video benchmark for human activity understanding},
  author={Fabian Caba Heilbron and Victor Escorcia and Bernard Ghanem and Juan Carlos Niebles},
  journal={2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages={961-970}
}
```

For basic dataset information, please refer to the official [website](http://activity-net.org/).
For action detection, you can either use the ActivityNet rescaled feature provided in this [repo](https://github.com/wzmsltw/BSN-boundary-sensitive-network#code-and-data-preparation) or extract feature with mmaction2 (which has better performance).
We release both pipeline.
Before we start, please make sure that current working directory is `$MMACTION2/tools/data/activitynet/`.

## Option 1: Use the ActivityNet rescaled feature provided in this [repo](https://github.com/wzmsltw/BSN-boundary-sensitive-network#code-and-data-preparation)

### Step 1. Download Annotations

First of all, you can run the following script to download annotation files.

```shell
bash download_feature_annotations.sh
```

### Step 2. Prepare Videos Features

Then, you can run the following script to download activitynet features.

```shell
bash download_features.sh
```

### Step 3. Process Annotation Files

Next, you can run the following script to process the downloaded annotation files for training and testing.
It first merges the two annotation files together and then separates the annoations by `train`, `val` and `test`.

```shell
python process_annotations.py
```

```shell
```

## Option 2: Extract ActivityNet feature using MMAction2 with all videos provided in official [website](http://activity-net.org/)

```shell
```

### Step 2. Prepare Videos

```shell
```

Since some videos in the ActivityNet dataset might be no longer available on YouTube, official [website](http://activity-net.org/) has made the full dataset available on Google and Baidu drives.

```

To accommodate missing data requests, you can fill in this [request form](https://docs.google.com/forms/d/e/1FAIpQLSeKaFq9ZfcmZ7W0B0PbEhfbTHY41GeEgwsa7WobJgGUhn4DTQ/viewform) provided in official [download page](http://activity-net.org/download.html) to have a 7-day-access to download the videos from the drive folders.

We also provide download steps for annotations from [BSN repo](https://github.com/wzmsltw/BSN-boundary-sensitive-network#code-and-data-preparation)

```shell
bash download_bsn_videos.sh
```

For this case, the downloading scripts update the annotation file after downloading to make sure every video in it exists.

### Step 3. Extract RGB and Flow

```shell
```

Use following scripts to extract both RGB and Flow.

```shell
```

The command above can generate images with new short edge 256. If you want to generate images with short edge 320 (320p), or with fix size 340x256, you can change the args `--new-short 256` to `--new-short 320` or `--new-width 340 --new-height 256`.
More details can be found in [prepare dataset](/docs/en/user_guides/prepare_dataset.md)

### Step 4. Generate File List for ActivityNet Finetuning

With extracted frames, you can generate video-level or clip-level lists of rawframes, which can be used for ActivityNet Finetuning.

```shell

```shell
python generate_rawframes_filelist.py
```

### Step 5. Finetune TSN models on ActivityNet

You can use ActivityNet configs in `configs/recognition/tsn` to finetune TSN models on ActivityNet.
You need to use Kinetics models for pretraining.
Both RGB models and Flow models are supported.

### Step 6. Extract ActivityNet Feature with finetuned ckpts

After finetuning TSN on ActivityNet, you can use it to extract both RGB and Flow feature.

```shell
python ../../misc/clip_feature_extraction.py tsn_extract_rgb_feat_config.py \
  /path/to/rgb_checkpoint.pth ../../../data/ActivityNet/rgb_tarin_feat.pkl \
  --video-list ../../../data/ActivityNet/anet_train_video.txt \
  --video-root ../../../data/ActivityNet/rawframes \
  --dump-score

--video-list ../../../data/ActivityNet/anet_train_video.txt \

  path/to/rgb_checkpoint.pth ../../../data/ActivityNet/rgb_val_feat.pkl \
  --video-list ../../../data/ActivityNet/anet_val_video.txt \

python ../../misc/clip_feature_extraction.py tsn_extract_flow_feat_config.py \
  /path/to/flow_checkpoint.pth ../../../data/ActivityNet/flow_tarin_feat.pkl \

  /path/to/flow_checkpoint.pth ../../../data/ActivityNet/flow_val_feat.pkl \
```

--video-list ../../../data/ActivityNet/anet_val_video.txt \
```

After feature extraction, you can use our post processing scripts to concat RGB and Flow feature, generate the `100-t X 400-d` feature for Action Detection.

```shell
python activitynet_feature_postprocessing.py --rgb ../../../data/ActivityNet/rgb_feat --flow ../../../data/ActivityNet/flow_feat --dest ../../../data/ActivityNet/mmaction_feat
```

## Final Step. Check Directory Structure

After the whole data pipeline for ActivityNet preparation,
you will get the features, videos, frames and annotation files.

In the context of the whole project (for ActivityNet only), the folder structure will look like:

```
mmaction2
├── tools
├── data
│   ├── ActivityNet

(if Option 1 used)
│   │   ├── anet_anno_{train,val,test,full}.json
│   │   ├── anet_anno_action.json
│   │   ├── video_info_new.csv
│   │   ├── activitynet_feature_cuhk

├── data

│   │   │   ├── csv_mean_100
│   │   │   │   ├── v___c8enCfzqw.csv
│   │   │   │   ├── v___dXUJsj3yo.csv
│   │   │   |   ├── ..

(if Option 2 used)
│   │   ├── anet_train_video.txt
│   │   ├── anet_val_video.txt
│   │   ├── anet_train_clip.txt
│   │   ├── anet_val_clip.txt
│   │   ├── activity_net.v1-3.min.json
│   │   ├── mmaction_feat
│   │   │   ├── v___c8enCfzqw.csv
│   │   │   ├── v___dXUJsj3yo.csv
│   │   │   ├── ..
│   │   │   ├── v___c8enCfzqw
│   │   │   │   ├── img_00000.jpg
│   │   │   │   ├── flow_x_00000.jpg
│   │   │   │   ├── flow_y_00000.jpg
│   │   │   │   ├── ..
│   │   │   ├── ..

```

For training and evaluating on ActivityNet, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).

# CFA

> [Beyond Bounding-Box: Convex-hull Feature Adaptation for Oriented and Densely Packed Object Detection.](https://openaccess.thecvf.com/content/CVPR2021/papers/Guo_Beyond_Bounding-Box_Convex-Hull_Feature_Adaptation_for_Oriented_and_Densely_Packed_CVPR_2021_paper.pdf)

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/cfa.png" width="800"/>
</div>

Detecting oriented and densely packed objects remains challenging for spatial feature aliasing caused by the intersection of reception fields between objects. In this paper, we propose a convex-hull feature adaptation (CFA) approach for configuring convolutional features in accordance with oriented and densely packed object layouts. CFA is rooted in convex-hull feature representation, which defines a set of dynamically predicted feature points guided by the convex intersection over union (CIoU) to bound the extent of objects. CFA pursues optimal feature assignment by constructing convex-hull sets and dynamically splitting positive or negative convex-hulls. By simultaneously considering overlapping convex-hulls and objects and penalizing convex-hulls shared by multiple objects, CFA alleviates spatial feature aliasing towards optimal feature adaptation. Experiments on DOTA and SKU110KR datasets show that CFA significantly outperforms the baseline approach, achieving new state-of-the-art detection performance.

| ResNet50 (1024,1024,200) | 69.63 | le135 |   1x    |   3.45   |      16.1      |  -  |     2      |                        [cfa-qbox_r50_fpn_1x_dota](./cfa-qbox_r50_fpn_1x_dota.py)                         |                                     [model](https://download.openmmlab.com/mmrotate/v0.1.0/cfa/cfa_r50_fpn_1x_dota_le135/cfa_r50_fpn_1x_dota_le135-aed1cbc6.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/cfa/cfa_r50_fpn_1x_dota_le135/cfa_r50_fpn_1x_dota_le135_20220205_144859.log.json)                                     |

| ResNet50 (1024,1024,200) | 73.45 |  oc   |   40e   |   3.45   |      16.1      |  -  |     2      |                       [cfa-qbox_r50_fpn_40e_dota](./cfa-qbox_r50_fpn_40e_dota.py)                        |                                         [model](https://download.openmmlab.com/mmrotate/v0.1.0/cfa/cfa_r50_fpn_40e_dota_oc/cfa_r50_fpn_40e_dota_oc-2f387232.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/cfa/cfa_r50_fpn_40e_dota_oc/cfa_r50_fpn_40e_dota_oc_20220209_171237.log.json)                                         |

```
@inproceedings{Guo_2021CVPR_CFA,
  author    = {Zonghao Guo, Chang Liu, Xiaosong Zhang, Jianbin Jiao, Xiangyang Ji and Qixiang Ye},
  title     = {Beyond Bounding-Box: Convex-hull Feature Adaptation for Oriented and Densely Packed Object Detection},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
}
```

# Use Mixed Datasets for Training

MMPose offers a convenient and versatile solution for training with mixed datasets through its `CombinedDataset` tool. Acting as a wrapper, it allows for the inclusion of multiple datasets and seamlessly reads and converts data from varying sources into a unified format for model training. The data processing pipeline utilizing `CombinedDataset` is illustrated in the following figure.

The following section will provide a detailed description of how to configure `CombinedDataset` with an example that combines the COCO and AI Challenger (AIC) datasets.

## COCO & AIC example

The COCO and AIC datasets are both human 2D pose datasets, but they differ in the number and order of keypoints. Here are two instances from the respective datasets.

Some keypoints, such as "left hand", are defined in both datasets, but they have different indices. Specifically, the index for the "left hand" keypoint is 9 in the COCO dataset and 5 in the AIC dataset. Furthermore, each dataset contains unique keypoints that are not present in the counterpart dataset. For instance, the facial keypoints (with indices 0~4) are only defined in the COCO dataset, whereas the "head top" (with index 12) and "neck" (with index 13) keypoints are exclusive to the AIC dataset. The relationship between the keypoints in both datasets is illustrated in the following Venn diagram.

Next, we will discuss two methods of mixing datasets.

- [Merge](#merge-aic-into-coco)
- [Combine](#combine-aic-and-coco)

### Merge AIC into COCO

If users aim to enhance their model's performance on the COCO dataset or other similar datasets, they can use the AIC dataset as an auxiliary source. To do so, they should select only the keypoints in AIC dataset that are shared with COCO datasets and ignore the rest. Moreover, the indices of these chosen keypoints in the AIC dataset should be transformed to match the corresponding indices in the COCO dataset.

In this scenario, no data conversion is required for the elements from the COCO dataset. To configure the COCO dataset, use the following code:

```python

```python
    pipeline=[], # Leave the `pipeline` empty, as no conversion is needed
)
```

For AIC dataset, the order of the keypoints needs to be transformed. MMPose provides a `KeypointConverter` transform to achieve this. Here's an example of how to configure the AIC sub dataset:

```python
        dict(
            num_keypoints=17,  # same as COCO dataset

'keypoint_train_images_20170902/'),
        dict(
            mapping=[  # includes index pairs for corresponding keypoints
                (0, 6),  # index 0 (in AIC) -> index 6 (in COCO)
                (1, 8),
                (2, 10),
                (3, 5),
                (4, 7),
                (5, 9),
                (6, 12),
                (7, 14),
                (8, 16),
                (9, 11),
                (10, 13),
                (11, 15),
            ])
    ],
)
```

By using the `KeypointConverter`, the indices of keypoints with indices 0 to 11 will be transformed to corresponding indices among 5 to 16. Meanwhile, the keypoints with indices 12 and 13 will be removed. For the target keypoints with indices 0 to 4, which are not defined in the `mapping` argument, they will be set as invisible and won't be used in training.

Once the sub datasets are configured, the `CombinedDataset` wrapper can be defined as follows:

```python
    # Since the combined dataset has the same data format as COCO,
    # it should use the same meta information for the dataset
    # The pipeline includes typical transforms, such as loading the
    # image and data augmentation
)
```

A complete, ready-to-use [config file](https://github.com/open-mmlab/mmpose/blob/dev-1.x/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-aic-256x192-merge.py) that merges the AIC dataset into the COCO dataset is also available. Users can refer to it for more details and use it as a template to build their own custom dataset.

### Combine AIC and COCO

The previously mentioned method discards some annotations in the AIC dataset. If users want to use all the information from both datasets, they can combine the two datasets. This means taking the union set of keypoints in both datasets.

In this scenario, both COCO and AIC datasets need to adjust the keypoint indices using `KeypointConverter`:

```python
        dict(
            num_keypoints=19,  # the size of union keypoint set
            mapping=[
                (0, 0),
                (1, 1),
                # omitted
                (16, 16),
            ])
    ])

mapping=[
                (0, 0),
                (1, 1),
                # omitted
                (16, 16),
            ])
    ])

        dict(
            mapping=[
                (0, 6),
                # omitted
                (12, 17),
                (13, 18),
            ])
    ],
)
```

mapping=[
                (0, 6),
                # omitted
                (12, 17),
                (13, 18),
            ])
    ],
)
```

To account for the fact that the combined dataset has 19 keypoints, which is different from either COCO or AIC dataset, a new dataset meta information file is needed to describe the new dataset. An example of such a file is [coco_aic.py](https://github.com/open-mmlab/mmpose/blob/dev-1.x/configs/_base_/datasets/coco_aic.py), which is based on [coco.py](https://github.com/open-mmlab/mmpose/blob/dev-1.x/configs/_base_/datasets/coco.py) but includes several updates:

- The paper information of AIC dataset has been added.
- The 'head_top' and 'neck' keypoints, which are unique in AIC, have been added to the `keypoint_info`.
- A skeleton link between 'head_top' and 'neck' has been added.
- The `joint_weights` and `sigmas` have been extended for the newly added keypoints.

Finally, the combined dataset can be configured as:

```python

```python
    # using new dataset meta information file
)
```

datasets=[dataset_coco, dataset_aic],
)
```

Additionally, the output channel number of the model should be adjusted as the number of keypoints changes. If the users aim to evaluate the model on the COCO dataset, a subset of model outputs must be chosen. This subset can be customized using the `output_keypoint_indices` argument in `test_cfg`. Users can refer to the [config file](https://github.com/open-mmlab/mmpose/blob/dev-1.x/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-aic-256x192-combine.py), which combines the COCO and AIC dataset, for more details and use it as a template to create their custom dataset.

---
title: ''
labels: ''
---

# Describe the feature

<details>
<summary align="right">PConv (ECCV'2018)</summary>

```bibtex
}
```

<br/>

|                          算法                          | 掩膜类型  | 分辨率  | 训练集容量 |    测试集     | l1 损失 |  PSNR  | SSIM  | GPU 信息 |                                       下载                                        |
| :----------------------------------------------------: | :-------: | :-----: | :--------: | :-----------: | :-----: | :----: | :---: | :------: | :-------------------------------------------------------------------------------: |

| [PConv_Stage1](./pconv_stage1_8xb12_places-256x256.py) | free-form | 256x256 |    500k    | Places365-val |    -    |   -    |   -   |    8     |                                         -                                         |
| [PConv_Stage2](./pconv_stage2_4xb2_places-256x256.py)  | free-form | 256x256 |    500k    | Places365-val |  8.776  | 22.762 | 0.801 |    4     | [模型](https://download.openmmlab.com/mmediting/inpainting/pconv/pconv_256x256_stage2_4x2_places_20200619-1ffed0e8.pth) \| [日志](https://download.openmmlab.com/mmediting/inpainting/pconv/pconv_256x256_stage2_4x2_places_20200619-1ffed0e8.log.json) |

|                         算法                          | 掩膜类型  | 分辨率  | 训练集容量 |   测试集   | l1 损失 |  PSNR  | SSIM  | GPU 信息 |                                         下载                                          |

| :---------------------------------------------------: | :-------: | :-----: | :--------: | :--------: | :-----: | :----: | :---: | :------: | :-----------------------------------------------------------------------------------: |
| [PConv_Stage1](./pconv_stage1_8xb1_celeba-256x256.py) | free-form | 256x256 |    500k    | CelebA-val |    -    |   -    |   -   |    8     |                                           -                                           |
| [PConv_Stage2](./pconv_stage2_4xb2_celeba-256x256.py) | free-form | 256x256 |    500k    | CelebA-val |  5.990  | 25.404 | 0.853 |    4     | [模型](https://download.openmmlab.com/mmediting/inpainting/pconv/pconv_256x256_stage2_4x2_celeba_20200619-860f8b95.pth) \| [日志](https://download.openmmlab.com/mmediting/inpainting/pconv/pconv_256x256_stage2_4x2_celeba_20200619-860f8b95.log.json) |

## 快速开始

**训练**

<details>

```shell
# CPU上训练

## 快速开始

**训练**

<details>

```shell
# CPU上训练

# 单个GPU上训练

# 多个GPU上训练
```

**测试**

<details>

```shell
# CPU上测试

# 单个GPU上测试

# 多个GPU上测试

# 多个GPU上测试
```

# PISA

> [Prime Sample Attention in Object Detection](https://arxiv.org/abs/1904.04821)

It is a common paradigm in object detection frameworks to treat all samples equally and target at maximizing the performance on average. In this work, we revisit this paradigm through a careful study on how different samples contribute to the overall performance measured in terms of mAP. Our study suggests that the samples in each mini-batch are neither independent nor equally important, and therefore a better classifier on average does not necessarily mean higher mAP. Motivated by this study, we propose the notion of Prime Samples, those that play a key role in driving the detection performance. We further develop a simple yet effective sampling and learning strategy called PrIme Sample Attention (PISA) that directs the focus of the training process towards such samples. Our experiments demonstrate that it is often more effective to focus on prime samples than hard samples when training a detector. Particularly, On the MSCOCO dataset, PISA outperforms the random sampling baseline and hard mining schemes, e.g., OHEM and Focal Loss, consistently by around 2% on both single-stage and two-stage detectors, even with a strong backbone ResNeXt-101.

<img src="https://user-images.githubusercontent.com/40661020/143970710-5cfd5960-fcf9-4e32-860a-acd46ce5d274.png"/>
</div>

| PISA |   Network    |    Backbone    | Lr schd | box AP | mask AP |                         Config                         |                                                                                                                                                              Download                                                                                                                                                              |

| :--: | :----------: | :------------: | :-----: | :----: | :-----: | :----------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|  ×   | Faster R-CNN |    R-50-FPN    |   1x    |  36.4  |         |                           -                            |                                                                                                                                                                                                                                                                                                                                    |

|  √   | Faster R-CNN |    R-50-FPN    |   1x    |  38.4  |         |    [config](./faster-rcnn_r50_fpn_pisa_1x_coco.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/pisa/pisa_faster_rcnn_r50_fpn_1x_coco/pisa_faster_rcnn_r50_fpn_1x_coco-dea93523.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pisa/pisa_faster_rcnn_r50_fpn_1x_coco/pisa_faster_rcnn_r50_fpn_1x_coco_20200506_185619.log.json)               |
|  ×   | Faster R-CNN | X101-32x4d-FPN |   1x    |  40.1  |         |                           -                            |                                                                                                                                                                                                                                                                                                                                    |

|  √   | Faster R-CNN | X101-32x4d-FPN |   1x    |  41.9  |         | [config](./faster-rcnn_x101-32x4d_fpn_pisa_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/pisa/pisa_faster_rcnn_x101_32x4d_fpn_1x_coco/pisa_faster_rcnn_x101_32x4d_fpn_1x_coco-e4accec4.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pisa/pisa_faster_rcnn_x101_32x4d_fpn_1x_coco/pisa_faster_rcnn_x101_32x4d_fpn_1x_coco_20200505_181503.log.json) |
|  ×   | Mask   R-CNN |    R-50-FPN    |   1x    |  37.3  |  34.2   |                           -                            |                                                                                                                                                                                                                                                                                                                                    |

|  √   | Mask   R-CNN |    R-50-FPN    |   1x    |  39.1  |  35.2   |     [config](./mask-rcnn_r50_fpn_pisa_1x_coco.py)      |                   [model](https://download.openmmlab.com/mmdetection/v2.0/pisa/pisa_mask_rcnn_r50_fpn_1x_coco/pisa_mask_rcnn_r50_fpn_1x_coco-dfcedba6.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pisa/pisa_mask_rcnn_r50_fpn_1x_coco/pisa_mask_rcnn_r50_fpn_1x_coco_20200508_150500.log.json)                   |
|  ×   | Mask   R-CNN | X101-32x4d-FPN |   1x    |  41.1  |  37.1   |                           -                            |                                                                                                                                                                                                                                                                                                                                    |

|  √   | Mask   R-CNN | X101-32x4d-FPN |   1x    |        |         |                                                        |                                                                                                                                                                                                                                                                                                                                    |
|  ×   |  RetinaNet   |    R-50-FPN    |   1x    |  35.6  |         |                           -                            |                                                                                                                                                                                                                                                                                                                                    |

|  √   |  RetinaNet   |    R-50-FPN    |   1x    |  36.9  |         |     [config](./retinanet-r50_fpn_pisa_1x_coco.py)      |                   [model](https://download.openmmlab.com/mmdetection/v2.0/pisa/pisa_retinanet_r50_fpn_1x_coco/pisa_retinanet_r50_fpn_1x_coco-76409952.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pisa/pisa_retinanet_r50_fpn_1x_coco/pisa_retinanet_r50_fpn_1x_coco_20200504_014311.log.json)                   |
|  ×   |  RetinaNet   | X101-32x4d-FPN |   1x    |  39.0  |         |                           -                            |                                                                                                                                                                                                                                                                                                                                    |

|  √   |  RetinaNet   | X101-32x4d-FPN |   1x    |  40.7  |         |  [config](./retinanet_x101-32x4d_fpn_pisa_1x_coco.py)  |     [model](https://download.openmmlab.com/mmdetection/v2.0/pisa/pisa_retinanet_x101_32x4d_fpn_1x_coco/pisa_retinanet_x101_32x4d_fpn_1x_coco-a0c13c73.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pisa/pisa_retinanet_x101_32x4d_fpn_1x_coco/pisa_retinanet_x101_32x4d_fpn_1x_coco_20200505_001404.log.json)     |
|  ×   |    SSD300    |     VGG16      |   1x    |  25.6  |         |                           -                            |                                                                                                                                                                                                                                                                                                                                    |

|  √   |    SSD300    |     VGG16      |   1x    |  27.6  |         |            [config](./ssd300_pisa_coco.py)             |                                               [model](https://download.openmmlab.com/mmdetection/v2.0/pisa/pisa_ssd300_coco/pisa_ssd300_coco-710e3ac9.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pisa/pisa_ssd300_coco/pisa_ssd300_coco_20200504_144325.log.json)                                               |
|  ×   |    SSD512    |     VGG16      |   1x    |  29.3  |         |                           -                            |                                                                                                                                                                                                                                                                                                                                    |

|  √   |    SSD512    |     VGG16      |   1x    |  31.8  |         |            [config](./ssd512_pisa_coco.py)             |                                               [model](https://download.openmmlab.com/mmdetection/v2.0/pisa/pisa_ssd512_coco/pisa_ssd512_coco-247addee.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/pisa/pisa_ssd512_coco/pisa_ssd512_coco_20200508_131030.log.json)                                               |

- In the original paper, all models are trained and tested on mmdet v1.x, thus results may not be exactly the same with this release on v2.0.
- It is noted PISA only modifies the training pipeline so the inference time remains the same with the baseline.

```latex
@inproceedings{cao2019prime,
  title={Prime sample attention in object detection},
  author={Cao, Yuhang and Chen, Kai and Loy, Chen Change and Lin, Dahua},
}
```

<details>

```bibtex
}
```

<details>

```bibtex
}
```

<details>

}
```

<details>

```bibtex
}
```

| [simcc_S-ViPNAS-MobileNetV3](/configs/body_2d_keypoint/simcc/coco/simcc_vipnas-mbv3_8xb64-210e_coco-256x192.py) |  256x192   | 0.695 |      0.883      |      0.772      | 0.755 |      0.927      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/simcc/coco/simcc_vipnas-mbv3_8xb64-210e_coco-256x192-719f3489_20220922.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/simcc/coco/simcc_vipnas-mbv3_8xb64-210e_coco-256x192-719f3489_20220922.log.json) |

# 准备 CelebA-HQ 数据集

```bibtex
@article{karras2017progressive,
  title={Progressive growing of gans for improved quality, stability, and variation},
  author={Karras, Tero and Aila, Timo and Laine, Samuli and Lehtinen, Jaakko},
  journal={arXiv preprint arXiv:1710.10196},
}
```

请按照[此处](https://github.com/tkarras/progressive_growing_of_gans#preparing-datasets-for-training)准备数据集。

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── CelebA-HQ
│   │   ├── train_256
|   |   ├── test_256
|   |   ├── train_celeba_img_list.txt
|   |   ├── val_celeba_img_list.txt

```

# Dataset Preparation (To be updated)

Please refer to [data preparation](https://github.com/open-mmlab/mmrotate/tree/main/tools/data) for dataset preparation.

# TimeSformer

[Is Space-Time Attention All You Need for Video Understanding?](https://arxiv.org/abs/2102.05095)

We present a convolution-free approach to video classification built exclusively on self-attention over space and time. Our method, named "TimeSformer," adapts the standard Transformer architecture to video by enabling spatiotemporal feature learning directly from a sequence of frame-level patches. Our experimental study compares different self-attention schemes and suggests that "divided attention," where temporal attention and spatial attention are separately applied within each block, leads to the best video classification accuracy among the design choices considered. Despite the radically new design, TimeSformer achieves state-of-the-art results on several action recognition benchmarks, including the best reported accuracy on Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks, our model is faster to train, it can achieve dramatically higher test efficiency (at a small drop in accuracy), and it can also be applied to much longer video clips (over one minute long).

<img src="https://user-images.githubusercontent.com/34324155/143018542-7f782ec9-dca2-495e-9043-c13ad941a25c.png" width="800"/>
</div>

| frame sampling strategy | resolution | gpus |        backbone         |   pretrain   | top1 acc | top5 acc | testing protocol | FLOPs | params |             config             |             ckpt             |             log             |
| :---------------------: | :--------: | :--: | :---------------------: | :----------: | :------: | :------: | :--------------: | :---: | :----: | :----------------------------: | :--------------------------: | :-------------------------: |

|         8x32x1          |  224x224   |  8   |   TimeSformer (divST)   | ImageNet-21K |  77.69   |  93.45   | 1 clip x 3 crop  | 196G  |  122M  | [config](/configs/recognition/timesformer/timesformer_divST_8xb8-8x32x1-15e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/timesformer/timesformer_divST_8xb8-8x32x1-15e_kinetics400-rgb/timesformer_divST_8xb8-8x32x1-15e_kinetics400-rgb_20220815-a4d0d01f.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/timesformer/timesformer_divST_8xb8-8x32x1-15e_kinetics400-rgb/timesformer_divST_8xb8-8x32x1-15e_kinetics400-rgb.log) |

|         8x32x1          |  224x224   |  8   |  TimeSformer (jointST)  | ImageNet-21K |  76.95   |  93.28   | 1 clip x 3 crop  | 180G  | 86.11M | [config](/configs/recognition/timesformer/timesformer_jointST_8xb8-8x32x1-15e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/timesformer/timesformer_jointST_8xb8-8x32x1-15e_kinetics400-rgb/timesformer_jointST_8xb8-8x32x1-15e_kinetics400-rgb_20220815-8022d1c0.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/timesformer/timesformer_jointST_8xb8-8x32x1-15e_kinetics400-rgb/timesformer_jointST_8xb8-8x32x1-15e_kinetics400-rgb.log) |

|         8x32x1          |  224x224   |  8   | TimeSformer (spaceOnly) | ImageNet-21K |  76.93   |  92.88   | 1 clip x 3 crop  | 141G  | 86.11M | [config](/configs/recognition/timesformer/timesformer_spaceOnly_8xb8-8x32x1-15e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/timesformer/timesformer_spaceOnly_8xb8-8x32x1-15e_kinetics400-rgb/timesformer_spaceOnly_8xb8-8x32x1-15e_kinetics400-rgb_20220815-78f05367.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/timesformer/timesformer_spaceOnly_8xb8-8x32x1-15e_kinetics400-rgb/timesformer_spaceOnly_8xb8-8x32x1-15e_kinetics400-rgb.log) |

2. We keep the test setting with the [original repo](https://github.com/facebookresearch/TimeSformer) (three crop x 1 clip).
3. The pretrained model `vit_base_patch16_224.pth` used by TimeSformer was converted from [vision_transformer](https://github.com/google-research/vision_transformer).

## Train

```shell
```

Example: train TimeSformer model on Kinetics-400 dataset in a deterministic option with periodic validation.

```shell
python tools/train.py configs/recognition/timesformer/timesformer_divST_8xb8-8x32x1-15e_kinetics400-rgb.py \
```

## Test

```shell

## Test

```shell
```

Example: test TimeSformer model on Kinetics-400 dataset and dump the result to a pkl file.

```shell
python tools/test.py configs/recognition/timesformer/timesformer_divST_8xb8-8x32x1-15e_kinetics400-rgb.py \
```

```BibTeX
@misc{bertasius2021spacetime,
    title   = {Is Space-Time Attention All You Need for Video Understanding?},
    author  = {Gedas Bertasius and Heng Wang and Lorenzo Torresani},
    year    = {2021},
    eprint  = {2102.05095},
    archivePrefix = {arXiv},
    primaryClass = {cs.CV}
}
```

# 数据变换迁移

## 简介

MMOCR 0.x 版本中，我们在 `mmocr/datasets/pipelines/xxx_transforms.py` 中实现了一系列的数据变换（Data Transforms）方法。然而，这些模块分散在各处，且缺乏规范统一的设计。因此，我们在 MMOCR 1.x 版本中对所有的数据增强模块进行了重构，并依照任务类型分别存放在 `mmocr/datasets/transforms` 目录下的 `ocr_transforms.py`，`textdet_transforms.py` 及 `textrecog_transforms.py` 中。其中，`ocr_transforms.py` 中实现了 OCR 相关任务通用的数据增强模块，而 `textdet_transforms.py` 和 `textrecog_transforms.py` 则分别实现了文本检测任务与文本识别任务相关的数据增强模组。

由于在重构过程中我们对部分模块进行了重命名、合并或拆分，使得新的调用接口与默认参数可能与旧版本存在不一致。因此，本文档将详细介绍如何对数据增强模块进行迁移，即，如何配置现有的数据变换来达到与旧版一致的行为。

## 配置迁移指南

### 数据格式化相关数据变换

1. `Collect` + `CustomFormatBundle` -> [`PackTextDetInputs`](mmocr.datasets.transforms.PackTextDetInputs)/[`PackTextRecogInputs`](mmocr.datasets.transforms.PackTextRecogInputs)

`PackxxxInputs` 同时囊括了 `Collect` 和 `CustomFormatBundle` 两个功能，且不再有 `key` 参数，而训练目标 target 的生成现在被转移至在 `loss` 中完成。

<thead>
  <tr>
    <th>MMOCR 0.x 配置</th>
    <th>MMOCR 1.x 配置</th>
  </tr>

```python
dict(

<thead>
  <tr>
  </tr>

```python
dict(
dict(
```

</td><td>

```python
dict(
```

</thead>
</table>

### 数据增强相关数据变换

   原有的 `ResizeOCR` 现在被拆分为三个独立的数据增强模块。

   `keep_aspect_ratio=False` 时，等价为 1.x 版本中的 `Resize`，其配置可按如下方式修改。

<thead>
  <tr>

原有的 `ResizeOCR` 现在被拆分为三个独立的数据增强模块。

<thead>
  <tr>
  </tr>

```python
dict(
```

</td><td>

```python
dict(
```

</thead>
</table>

`keep_aspect_ratio=True`，且 `max_width=None` 时。将图片的高缩放至固定值，并等比例缩放图像的宽。

<thead>
  <tr>
  </tr>

```python
dict(
```

</td><td>

```python
dict(
```

</thead>
</table>

keep_aspect_ratio=True)
```

</td><td>

```python
dict(
```

</thead>
</table>

`keep_aspect_ratio=True`，且 `max_width` 为固定值时。将图片的高缩放至固定值，并等比例缩放图像的宽。若缩放后的图像宽小于 `max_width`, 则将其填充至 `max_width`, 反之则将其裁剪至 `max_width`。即，输出图像的尺寸固定为 `(height, max_width)`。

<thead>
  <tr>
  </tr>

```python
dict(
```

</td><td>

```python
dict(
dict(
```

</thead>
</table>

dict(
```

</thead>
</table>

   随机旋转数据增强策略已被整合至 `RanomRotate`。该方法的默认行为与 0.x 版本中的 `RandomRotateTextDet` 保持一致。此时仅需指定最大旋转角度 `max_angle` 即可。

```{note}
  新旧版本 "max_angle" 的默认值不同，因此需要重新进行指定。
```

<thead>
  <tr>
  </tr>

```python
```

</td><td>

```python
```

</thead>
</table>

对于 `RandomRotatePolyInstances`，则需要指定参数 `use_canvas=True`。

<thead>
  <tr>
  </tr>

```python
dict(
  rotate_ratio=0.5, # 指定概率为0.5
```

</td><td>

```python

</tr>

```python
dict(
```

</td><td>

```python
# 用 RandomApply 对数据变换进行包装，并指定执行概率
dict(
  prob=0.5) # 设置执行概率为 0.5
```

</thead>
</table>

```{note}
在 0.x 版本中，部分数据增强方法通过定义一个内部变量 "xxx_ratio" 来指定执行概率，如 "rotate_ratio", "crop_ratio" 等。在 1.x 版本中，这些参数已被统一删除。现在，我们可以通过 "RandomApply" 来对不同的数据变换方法进行包装，并指定其执行概率。
```

   目前仅对方法名进行了更改，其他参数保持一致。

   新版本移除了 `crop_ratio` 以及 `instance_key`，并统一使用 `gt_polygons` 为目标进行裁剪。

<thead>
  <tr>
  </tr>

新版本移除了 `crop_ratio` 以及 `instance_key`，并统一使用 `gt_polygons` 为目标进行裁剪。

<thead>
  <tr>
  </tr>

```python
dict(
  crop_ratio=0.8, # 指定概率为 0.8
```

</td><td>

```python
dict(
  prob=0.8) # 设置执行概率为 0.8
```

</thead>
</table>

   新版本移除了 `instance_key` 和 `mask_type`，并统一使用 `gt_polygons` 为目标进行裁剪。

<thead>
  <tr>
  </tr>

```python
dict(
```

</td><td>

```python
dict(

<th>MMOCR 1.x 配置</th>
  </tr>

```python
dict(
```

</td><td>

```python
dict(
```

</thead>
</table>

   原有的 `EastRandomCrop` 内同时对图像进行了剪裁、缩放以及填充。在新版本中，我们可以通过组合三种数据增强策略来达到相同的效果。

<thead>
  <tr>
  </tr>

```python
dict(
```

</td><td>

```python
```

</thead>
</table>

```

</td><td>

```python
```

</thead>
</table>

   在新版本中，我们直接使用 MMEngine 中实现的 `RandomResize` 来代替原有的实现。

<thead>
  <tr>
  </tr>

```python
 dict(
  size=800,
```

</td><td>

```python
dict(
```

</thead>
</table>

```{note}

size=800,
```

</td><td>

```python
dict(
```

</thead>
</table>

```{note}
默认地，数据流水线会从当前 *scope* 的注册器中搜索对应的数据变换，如果不存在该数据变换，则将继续在上游库，如 MMCV 及 MMEngine 中进行搜索。例如，MMOCR 中并未实现 `RandomResize` 方法，但我们仍然可以在配置中直接引用该数据增强方法，因为程序将自动从上游的 MMCV 中搜索该方法。此外，用户也可以通过添加前缀的形式来指定 *scope*。例如，`mmengine.RandomResize` 将强制指定使用 MMCV 库中实现的 `RandomResize`，当上下游库中存在同名方法时，则可以通过这种形式强制使用特定的版本。另外需要注意的是，MMCV 中所有的数据变换方法都被注册至 MMEngine 中，因此我们使用 `mmengine.RandomResize` 而不是 `mmcv.RandomResize`。
```

   原有的 `SquareResizePad` 内部实现了两个分支，并依据概率 `pad_ratio` 随机使用其中的一个分支进行数据增强。具体而言，一个分支先对图像缩放再填充；另一个分支则直接对图像进行缩放。为增强不同模块的复用性，我们在 1.x 版本中将该方法拆分成了 `Resize` + `SourceImagePad` 的组合形式，并通过 MMCV 中的 `RandomChoice` 来控制分支。

<thead>
  <tr>

<thead>
  <tr>
  </tr>

```python
dict(
```

</td><td>

```python
dict(
    [
      dict(
      dict(
    ],
    [
      dict(
    ]
  ],
  prob=[0.4, 0.6]), # 两种组合的选用概率
```

</thead>
</table>

```{note}
在 1.x 版本中，随机选择包装器 "RandomChoice" 代替了 "OneOfWrapper"，可以从一系列数据变换组合中随机抽取一组并应用。
```

9. `RandomWrapper` -> [`mmegnine.RandomApply`](mmcv.transforms.RandomApply)

   在 1.x 版本中，`RandomWrapper` 包装器被替换为由 MMCV 实现的 `RandomApply`，用以指定数据变换的执行概率。其中概率 `p` 现在被命名为 `prob`。

<thead>
  <tr>
  </tr>

<thead>
  <tr>
  </tr>

```python
 dict(
  p=0.25,
  ])
```

</td><td>

```python
dict(
  ])
```

</thead>
</table>

10. `OneOfWrapper` -> [`mmegnine.RandomChoice`](mmcv.transforms.RandomChoice)

随机选择包装器现在被重命名为 `RandomChoice`，并且使用方法和原来完全一致。

原有的 `ScaleAspectJitter` 实现了多种不同的图像尺寸抖动数据增强策略，在新版本中，我们将其拆分为数个逻辑更加清晰的独立数据变化方法。

`resize_type='indep_sample_in_range'` 时，其等价于图像在指定范围内的随机缩放。

<thead>
  <tr>
  </tr>

```python
dict(

<thead>
  <tr>
  </tr>

```python
dict(
```

</td><td>

```python
 dict(
```

</thead>
</table>

`resize_type='long_short_bound'` 时，将图像缩放至指定大小，再对其长宽比进行抖动。这一逻辑现在由新的数据变换类 `BoundedScaleAspectJitter` 实现。

<thead>
  <tr>
  </tr>

```python
dict(
```

</td><td>

```python

aspect_ratio_range=(0.9, 1.1),
```

</td><td>

```python
dict(
```

</thead>
</table>

`resize_type='around_min_img_scale'` （默认参数）时，将图像的短边缩放至指定大小，再在指定范围内对长宽比进行抖动。最后，确保其边长能被 `scale_divisor` 整除。这一逻辑由新的数据变换类 `ShortScaleAspectJitter` 实现。

<thead>
  <tr>
  </tr>

```python
dict(
```

</td><td>

```python
dict(
```

keep_ratio=False)
```

</td><td>

```python
dict(
```

</thead>
</table>

We basically categorize model components into 6 types:

- encoder: Including voxel encoder and middle encoder used in voxel-based methods before backbone, e.g., `HardVFE` and `PointPillarsScatter`.
- backbone: Usually an FCN network to extract feature maps, e.g., `ResNet`, `SECOND`.
- neck: The component between backbones and heads, e.g., `FPN`, `SECONDFPN`.
- head: The component for specific tasks, e.g., `bbox prediction` and `mask prediction`.
- RoI extractor: The part for extracting RoI features from feature maps, e.g., `H3DRoIHead` and `PartAggregationROIHead`.
- loss: The component in heads for calculating losses, e.g., `FocalLoss`, `L1Loss`, and `GHMLoss`.

### Add a new encoder

Here we show how to develop new components with an example of HardVFE.

#### 1. Define a new voxel encoder (e.g. HardVFE: Voxel feature encoder used in HV-SECOND)

Create a new file `mmdet3d/models/voxel_encoders/voxel_encoder.py`.

```python

```python

        pass

        pass
```

You can either add the following line to `mmdet3d/models/voxel_encoders/__init__.py`:

```python
```

```python
```

#### 3. Use the voxel encoder in your config file

```python
    ...
        arg1=xxx,
    ...
)
```

```python
    ...
        arg1=xxx,
    ...
)
```

Here we show how to develop new components with an example of [SECOND](https://www.mdpi.com/1424-8220/18/10/3337) (Sparsely Embedded Convolutional Detection).

#### 1. Define a new backbone (e.g. SECOND)

Create a new file `mmdet3d/models/backbones/second.py`.

```python

        pass

        pass
```

You can either add the following line to `mmdet3d/models/backbones/__init__.py`:

```python
```

```python
```

```python
```

```python
```

```python
    ...
        arg1=xxx,
    ...
)
```

### Add a new neck

#### 1. Define a new neck (e.g. SECONDFPN)

Create a new file `mmdet3d/models/necks/second_fpn.py`.

```python

        pass

        pass
```

You can either add the following line to `mmdet3d/models/necks/__init__.py`:

```python
```

```python
```

#### 3. Use the neck in your config file

```python
    ...
    ...
)

```python
    ...
    ...
)
```

### Add a new head

Here we show how to develop a new head with the example of [PartA2 Head](https://arxiv.org/abs/1907.03670) as the following.

**Note**: Here the example of `PartA2 RoI Head` is used in the second stage. For one-stage heads, please refer to examples in `mmdet3d/models/dense_heads/`. They are more commonly used in 3D detection for autonomous driving due to its simplicity and high efficiency.

First, add a new bbox head in `mmdet3d/models/roi_heads/bbox_heads/parta2_bbox_head.py`.
`PartA2 RoI Head` implements a new bbox head for object detection.
To implement a bbox head, basically we need to implement two functions of the new module as the following. Sometimes other related functions like `loss` and `get_targets` are also required.

```python

```python

roi_feat_size=14,

        pass
```

Second, implement a new RoI Head if it is necessary. We plan to inherit the new `PartAggregationROIHead` from `Base3DRoIHead`. We can find that a `Base3DRoIHead` already implements the following functions.

```python

```python

        Args:

def init_bbox_head(self, bbox_roi_extractor: dict,

        Args:
        """

                ]

elif isinstance(self.train_cfg.assigner, list):
                ]

        pass
```

Double Head's modification is mainly in the bbox_forward logic, and it inherits other logics from the `Base3DRoIHead`.
In the `mmdet3d/models/roi_heads/part_aggregation_roi_head.py`, we implement the new RoI Head as the following:

```python

    Args:
    """

"""

        Args:

semantic_head: dict) -> None:

        Args:
        """

    @property

        Args:

**kwargs) -> InstanceList:

        Args:

        Returns:

              C >= 7.
        """

        ]

]

        Args:

voxel_dict (dict): Contains information of voxels.

        Returns:

              C >= 7.
        """
        ...

              C >= 7.
        """
        ...

        Args:

        Returns:
        """

dict[str, Tensor]: A dictionary of loss components
        """
            else:

sample_results = self._assign_and_sample(rpn_results_list,

```

Here we omit more details related to other functions. Please see the [code](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/models/roi_heads/part_aggregation_roi_head.py) for more details.

`mmdet3d/models/roi_heads/bbox_heads/__init__.py` and `mmdet3d/models/roi_heads/__init__.py` thus the corresponding registry could find and load them.

```python

```python
```

The config file of `PartAggregationROIHead` is as the following:

```python
    ...

gamma=2.0,

seg_in_channels=16,
    ...
)
```

loss_weight=1.0))),
    ...
)
```

The second stage of PartA2 Head mainly uses a new `PartAggregationROIHead` and a new
`PartA2BboxHead`, the arguments are set according to the `__init__` function of each module.

### Add a new loss

Assume you want to add a new loss as `MyLoss` for bounding box regression.
To add a new loss function, the users need to implement it in `mmdet3d/models/losses/my_loss.py`.
The decorator `weighted_loss` enables the loss to be weighted for each element.

```python

                pred,
                target,
```

Then the users need to add it in the `mmdet3d/models/losses/__init__.py`.

```python
```

return loss_bbox
```

```python
```

```python
```

To use it, users should modify the `loss_xxx` field.
Since `MyLoss` is for regression, you need to modify the `loss_bbox` field in the head.

```python
```

# MS R-CNN

> [Mask Scoring R-CNN](https://arxiv.org/abs/1903.00241)

Letting a deep network be aware of the quality of its own predictions is an interesting yet important problem. In the task of instance segmentation, the confidence of instance classification is used as mask quality score in most instance segmentation frameworks. However, the mask quality, quantified as the IoU between the instance mask and its ground truth, is usually not well correlated with classification score. In this paper, we study this problem and propose Mask Scoring R-CNN which contains a network block to learn the quality of the predicted instance masks. The proposed network block takes the instance feature and the corresponding predicted mask together to regress the mask IoU. The mask scoring strategy calibrates the misalignment between mask quality and mask score, and improves instance segmentation performance by prioritizing more accurate mask predictions during COCO AP evaluation. By extensive evaluations on the COCO dataset, Mask Scoring R-CNN brings consistent and noticeable gain with different models, and outperforms the state-of-the-art Mask R-CNN. We hope our simple and effective approach will provide a new direction for improving instance segmentation.

<img src="https://user-images.githubusercontent.com/40661020/143967239-3a95ae92-6443-4181-9cbc-dfe16e81b969.png"/>
</div>

|   Backbone   |  style  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                    Config                     |                                                                                                                                                                      Download                                                                                                                                                                       |

| :----------: | :-----: | :-----: | :------: | :------------: | :----: | :-----: | :-------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|   R-50-FPN   |  caffe  |   1x    |   4.5    |                |  38.2  |  36.0   | [config](./ms-rcnn_r50-caffe_fpn_1x_coco.py)  |                  [model](https://download.openmmlab.com/mmdetection/v2.0/ms_rcnn/ms_rcnn_r50_caffe_fpn_1x_coco/ms_rcnn_r50_caffe_fpn_1x_coco_20200702_180848-61c9355e.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/ms_rcnn/ms_rcnn_r50_caffe_fpn_1x_coco/ms_rcnn_r50_caffe_fpn_1x_coco_20200702_180848.log.json)                   |

|   R-50-FPN   |  caffe  |   2x    |    -     |       -        |  38.8  |  36.3   | [config](./ms-rcnn_r50-caffe_fpn_2x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/ms_rcnn/ms_rcnn_r50_caffe_fpn_2x_coco/ms_rcnn_r50_caffe_fpn_2x_coco_bbox_mAP-0.388__segm_mAP-0.363_20200506_004738-ee87b137.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/ms_rcnn/ms_rcnn_r50_caffe_fpn_2x_coco/ms_rcnn_r50_caffe_fpn_2x_coco_20200506_004738.log.json)   |
|  R-101-FPN   |  caffe  |   1x    |   6.5    |                |  40.4  |  37.6   | [config](./ms-rcnn_r101-caffe_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/ms_rcnn/ms_rcnn_r101_caffe_fpn_1x_coco/ms_rcnn_r101_caffe_fpn_1x_coco_bbox_mAP-0.404__segm_mAP-0.376_20200506_004755-b9b12a37.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/ms_rcnn/ms_rcnn_r101_caffe_fpn_1x_coco/ms_rcnn_r101_caffe_fpn_1x_coco_20200506_004755.log.json) |

|  R-101-FPN   |  caffe  |   2x    |    -     |       -        |  41.1  |  38.1   | [config](./ms-rcnn_r101-caffe_fpn_2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/ms_rcnn/ms_rcnn_r101_caffe_fpn_2x_coco/ms_rcnn_r101_caffe_fpn_2x_coco_bbox_mAP-0.411__segm_mAP-0.381_20200506_011134-5f3cc74f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/ms_rcnn/ms_rcnn_r101_caffe_fpn_2x_coco/ms_rcnn_r101_caffe_fpn_2x_coco_20200506_011134.log.json) |
| R-X101-32x4d | pytorch |   2x    |   7.9    |      11.0      |  41.8  |  38.7   | [config](./ms-rcnn_x101-32x4d_fpn_1x_coco.py) |                    [model](https://download.openmmlab.com/mmdetection/v2.0/ms_rcnn/ms_rcnn_x101_32x4d_fpn_1x_coco/ms_rcnn_x101_32x4d_fpn_1x_coco_20200206-81fd1740.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/ms_rcnn/ms_rcnn_x101_32x4d_fpn_1x_coco/ms_rcnn_x101_32x4d_fpn_1x_coco_20200206_100113.log.json)                    |

| R-X101-64x4d | pytorch |   1x    |   11.0   |      8.0       |  43.0  |  39.5   | [config](./ms-rcnn_x101-64x4d_fpn_1x_coco.py) |                    [model](https://download.openmmlab.com/mmdetection/v2.0/ms_rcnn/ms_rcnn_x101_64x4d_fpn_1x_coco/ms_rcnn_x101_64x4d_fpn_1x_coco_20200206-86ba88d2.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/ms_rcnn/ms_rcnn_x101_64x4d_fpn_1x_coco/ms_rcnn_x101_64x4d_fpn_1x_coco_20200206_091744.log.json)                    |
| R-X101-64x4d | pytorch |   2x    |   11.0   |      8.0       |  42.6  |  39.5   | [config](./ms-rcnn_x101-64x4d_fpn_2x_coco.py) |                    [model](https://download.openmmlab.com/mmdetection/v2.0/ms_rcnn/ms_rcnn_x101_64x4d_fpn_2x_coco/ms_rcnn_x101_64x4d_fpn_2x_coco_20200308-02a445e2.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/ms_rcnn/ms_rcnn_x101_64x4d_fpn_2x_coco/ms_rcnn_x101_64x4d_fpn_2x_coco_20200308_012247.log.json)                    |

```latex

```latex
@inproceedings{huang2019msrcnn,
    title={Mask Scoring R-CNN},
    author={Zhaojin Huang and Lichao Huang and Yongchao Gong and Chang Huang and Xinggang Wang},
}
```

# Shape Bias Tool Usage

Shape bias measures how a model relies the shapes, compared to texture, to sense the semantics in images. For more details,
we recommend interested readers to this [paper](https://arxiv.org/abs/2106.07411). MMPretrain provide an off-the-shelf toolbox to
obtain the shape bias of a classification model. You can following these steps below:

## Prepare the dataset

First you should download the [cue-conflict](https://github.com/bethgelab/model-vs-human/releases/download/v0.1/cue-conflict.tar.gz) to `data` folder,
and then unzip this dataset. After that, you `data` folder should have the following structure:

```text
data
├──cue-conflict
|      |──airplane
|      |──bear
|      ...
|      |── truck
```

## Modify the config for classification

```text
data
|      |──bear
|      ...
|      |── truck
```

We run the shape-bias tool on a ViT-base model with masked autoencoder pretraining. Its config file is `configs/mae/benchmarks/vit-base-p16_8xb128-coslr-100e_in1k.py`, and its checkpoint is downloaded from [this link](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-1600e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k_20220825-cf70aa21.pth). Replace the original test_pipeline, test_dataloader and test_evaluation with the following configurations:

```python
    dict(
        type='ResizeEdge',
        scale=256,
        edge='short',
        backend='pillow'),
]
    pin_memory=True,
    collate_fn=dict(type='default_collate'),

backend='pillow'),
]
        data_root='data/cue-conflict',
        _delete_=True),
    drop_last=False)
    type='mmpretrain.ShapeBiasMetric',
    csv_dir='work_dirs/shape_bias',
    model_name='mae')
```

Please note you should make custom modifications to the `csv_dir` and `model_name` above. I renamed my modified sample config file as `vit-base-p16_8xb128-coslr-100e_in1k_shape-bias.py` in the folder `configs/mae/benchmarks/`.

## Inference your model with above modified config file

Then you should inferece your model on the `cue-conflict` dataset with the your modified config file.

```shell
# For PyTorch
bash tools/dist_test.sh $CONFIG $CHECKPOINT
```

- `$CONFIG`: The path of your modified config file.
- `$CHECKPOINT`: The path or link of the checkpoint file.

```shell
# Example
bash tools/dist_test.sh configs/mae/benchmarks/vit-base-p16_8xb128-coslr-100e_in1k_shape-bias.py https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-1600e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k_20220825-cf70aa21.pth 1
```

After that, you should obtain a csv file in `csv_dir` folder, named `cue-conflict_model-name_session-1.csv`. Besides this file, you should also download these [csv files](https://github.com/bethgelab/model-vs-human/tree/master/raw-data/cue-conflict) to the
`csv_dir`.

## Plot shape bias

Then we can start to plot the shape bias:

```shell
python tools/analysis_tools/shape_bias.py --csv-dir $CSV_DIR --result-dir $RESULT_DIR --colors $RGB --markers o --plotting-names $YOUR_MODEL_NAME --model-names $YOUR_MODEL_NAME
```

- `--csv-dir $CSV_DIR`, the same directory to save these csv files.
- `--result-dir $RESULT_DIR`, the directory to output the result named `cue-conflict_shape-bias_matrixplot.pdf`.
- `--colors $RGB`, should be the RGB values, formatted in R G B, e.g. 100 100 100, and can be multiple RGB values, if you want to plot the shape bias of several models.
- `--plotting-names $YOUR_MODEL_NAME`, the name of the legend in the shape bias figure, and you can set it as your model name. If you want to plot several models, plotting_names can be multiple values.

- `model-names $YOUR_MODEL_NAME`, should be the same name specified in your config, and can be multiple names if you want to plot the shape bias of several models.

Please note, every three values for `--colors` corresponds to one value for `--model-names`. After all of above steps, you are expected to obtain the following figure.

<img src="https://github.com/open-mmlab/mmpretrain/assets/42371271/dc608d06-43eb-4860-bb70-486ed2a3f927" width="500" />
</div>

<details>

```bibtex
}
```

<details>
<summary align="right"><a href="https://arxiv.org/abs/2103.14030">Swin (ICCV'2021)</a></summary>

```bibtex
@inproceedings{liu2021swin,
  title={Swin transformer: Hierarchical vision transformer using shifted windows},
  author={Liu, Ze and Lin, Yutong and Cao, Yue and Hu, Han and Wei, Yixuan and Zhang, Zheng and Lin, Stephen and Guo, Baining},
  pages={10012--10022},

}
```

<details>

```bibtex
}
```

<details>

```bibtex

| [pose_swin_t](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_swin-t-p4-w7_8xb32-210e_coco-256x192.py) |  256x192   | 0.724 |      0.901      |      0.806      | 0.782 |      0.940      | [ckpt](https://download.openmmlab.com/mmpose/top_down/swin/swin_t_p4_w7_coco_256x192-eaefe010_20220503.pth) | [log](https://download.openmmlab.com/mmpose/top_down/swin/swin_t_p4_w7_coco_256x192_20220503.log.json) |
| [pose_swin_b](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_swin-b-p4-w7_8xb32-210e_coco-256x192.py) |  256x192   | 0.737 |      0.904      |      0.820      | 0.794 |      0.942      | [ckpt](https://download.openmmlab.com/mmpose/top_down/swin/swin_b_p4_w7_coco_256x192-7432be9e_20220705.pth) | [log](https://download.openmmlab.com/mmpose/top_down/swin/swin_b_p4_w7_coco_256x192_20220705.log.json) |

| [pose_swin_b](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_swin-b-p4-w7_8xb32-210e_coco-384x288.py) |  384x288   | 0.759 |      0.910      |      0.832      | 0.811 |      0.946      | [ckpt](https://download.openmmlab.com/mmpose/top_down/swin/swin_b_p4_w7_coco_384x288-3abf54f9_20220705.pth) | [log](https://download.openmmlab.com/mmpose/top_down/swin/swin_b_p4_w7_coco_384x288_20220705.log.json) |
| [pose_swin_l](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_swin-l-p4-w7_8xb32-210e_coco-256x192.py) |  256x192   | 0.743 |      0.906      |      0.821      | 0.798 |      0.943      | [ckpt](https://download.openmmlab.com/mmpose/top_down/swin/swin_l_p4_w7_coco_256x192-642a89db_20220705.pth) | [log](https://download.openmmlab.com/mmpose/top_down/swin/swin_l_p4_w7_coco_256x192_20220705.log.json) |

| [pose_swin_l](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_swin-l-p4-w7_8xb32-210e_coco-384x288.py) |  384x288   | 0.763 |      0.912      |      0.830      | 0.814 |      0.949      | [ckpt](https://download.openmmlab.com/mmpose/top_down/swin/swin_l_p4_w7_coco_384x288-c36b7845_20220705.pth) | [log](https://download.openmmlab.com/mmpose/top_down/swin/swin_l_p4_w7_coco_384x288_20220705.log.json) |

# breastCancerCellSegmentation

This project supports **`breastCancerCellSegmentation`**, which can be downloaded from [here](https://www.heywhale.com/mw/dataset/5e9e9b35ebb37f002c625423).

This dataset, with 58 H&E-stained histopathology images was used for breast cancer cell detection and associated real-world data.
Conventional histology uses a combination of hematoxylin and eosin stains, commonly referred to as H&E. These images are stained because most cells are inherently transparent with little or no intrinsic pigment.
Certain special stains selectively bind to specific components and can be used to identify biological structures such as cells.

| Dataset name                                                                                 | Anatomical region | Task type    | Modality       | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License                                                         |
| -------------------------------------------------------------------------------------------- | ----------------- | ------------ | -------------- | ------------ | --------------------- | ---------------------- | ------------ | --------------------------------------------------------------- |

| [breastCancerCellSegmentation](https://www.heywhale.com/mw/dataset/5e9e9b35ebb37f002c625423) | cell              | segmentation | histopathology | 2            | 58/-/-                | yes/-/-                | 2020         | [CC-BY-NC 4.0](https://creativecommons.org/licenses/by-sa/4.0/) |

|    Class Name    | Num. Train | Pct. Train | Num. Val | Pct. Val | Num. Test | Pct. Test |
| :--------------: | :--------: | :--------: | :------: | :------: | :-------: | :-------: |
|    background    |     58     |   98.37    |    -     |    -     |     -     |     -     |
| breastCancerCell |     58     |    1.63    |    -     |    -     |     -     |     -     |

Note:

![bac](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/histopathology/breastCancerCellSegmentation/breastCancerCellSegmentation_dataset.png)

## Usage

## Usage

- [MMSegmentation](https://github.com/open-mmlab/mmsegmentation) v1.0.0

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `breastCancerCellSegmentation/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
```

- Download dataset from [here](https://www.heywhale.com/mw/dataset/5e9e9b35ebb37f002c625423) and save it to the `data/` directory .

```

- Decompress data to path `data/`. This will create a new folder named `data/breastCancerCellSegmentation/`, which contains the original image data.

```none
  ├── mmseg
  │   │   │   │   ├── breastCancerCellSegmentation
  │   │   │   │   │   │   ├── breastCancerCellSegmentation
  |   │   │   │   │   │   │   ├── train.txt
  |   │   │   │   │   │   │   ├── val.txt
  |   │   │   │   │   │   │   ├── images
  |   │   │   │   │   │   │   |   ├── xxx.tif

|   │   │   │   │   │   │   ├── train.txt
  |   │   │   │   │   │   │   ├── masks
  |   │   │   │   │   │   │   |   ├── xxx.TIF

```

```shell
```

```shell
```

}
```

<details>

```bibtex
}
```

| [deeppose_resnet_50](/configs/hand_2d_keypoint/topdown_regression/onehand10k/td-reg_res50_8xb64-210e_onehand10k-256x256.py) |  256x256   |  0.990  | 0.485 | 34.21 | [ckpt](https://download.openmmlab.com/mmpose/hand/deeppose/deeppose_res50_onehand10k_256x256-cbddf43a_20210330.pth) | [log](https://download.openmmlab.com/mmpose/hand/deeppose/deeppose_res50_onehand10k_256x256_20210330.log.json) |

<details>

```bibtex
}
```

<details>

```bibtex
}
```

}
```

| [LiteHRNet-18](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_litehrnet-18_8xb64-210e_mpii-256x256.py) |  256x256   | 0.859 |  0.260   | [ckpt](https://download.openmmlab.com/mmpose/top_down/litehrnet/litehrnet18_mpii_256x256-cabd7984_20210623.pth) | [log](https://download.openmmlab.com/mmpose/top_down/litehrnet/litehrnet18_mpii_256x256_20210623.log.json) |

| [LiteHRNet-30](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_litehrnet-30_8xb64-210e_mpii-256x256.py) |  256x256   | 0.869 |  0.271   | [ckpt](https://download.openmmlab.com/mmpose/top_down/litehrnet/litehrnet30_mpii_256x256-faae8bd8_20210622.pth) | [log](https://download.openmmlab.com/mmpose/top_down/litehrnet/litehrnet30_mpii_256x256_20210622.log.json) |

# Configure Datasets

This tutorial mainly focuses on selecting datasets supported by OpenCompass and preparing their configs files. Please make sure you have downloaded the datasets following the steps in [Dataset Preparation](../get_started.md#dataset-preparation).

## Directory Structure of Dataset Configuration Files

First, let's introduce the structure under the `configs/datasets` directory in OpenCompass, as shown below:

```
├── apps
├── ARC_c
├── ...
├── CLUE_afqmc  # dataset
│   ├── CLUE_afqmc_gen_901306.py  # different version of config
├── ...
├── XLSum
├── Xsum
```

In the `configs/datasets` directory structure, we flatten all datasets directly, and there are multiple dataset configurations within the corresponding folders for each dataset.

```

The naming of the dataset configuration file is made up of `{dataset name}_{evaluation method}_{prompt version number}.py`. For example, `CLUE_afqmc/CLUE_afqmc_gen_db509b.py`, this configuration file is the `CLUE_afqmc` dataset under the Chinese universal ability, the corresponding evaluation method is `gen`, i.e., generative evaluation, and the corresponding prompt version number is `db509b`; similarly, `CLUE_afqmc_ppl_00b348.py` indicates that the evaluation method is `ppl`, i.e., discriminative evaluation, and the prompt version number is `00b348`.

In addition, files without a version number, such as: `CLUE_afqmc_gen.py`, point to the latest prompt configuration file of that evaluation method, which is usually the most accurate prompt.

## Dataset Selection

In each dataset configuration file, the dataset will be defined in the `{}_datasets` variable, such as `afqmc_datasets` in `CLUE_afqmc/CLUE_afqmc_gen_db509b.py`.

```python
    dict(
    ),
]
```

And `cmnli_datasets` in `CLUE_cmnli/CLUE_cmnli_ppl_b78ad4.py`.

```python
    dict(
]
```

split='train',
]
```

Take these two datasets as examples. If users want to evaluate these two datasets at the same time, they can create a new configuration file in the `configs` directory. We use the import mechanism in the `mmengine` configuration to build the part of the dataset parameters in the evaluation script, as shown below:

```python

```

Users can choose different abilities, different datasets and different evaluation methods configuration files to build the part of the dataset in the evaluation script according to their needs.

```

For information on how to start an evaluation task and how to evaluate self-built datasets, please refer to the relevant documents.

# MMagic 模型部署

- [MMagic 模型部署](#mmagic-模型部署)
    - [安装 mmagic](#安装-mmagic)
    - [超分任务模型转换](#超分任务模型转换)

[MMagic](https://github.com/open-mmlab/mmagic)，又称 `mmagic`，是基于 PyTorch 的开源图像和视频编辑工具箱。它是 [OpenMMLab](https://openmmlab.com/) 项目成员之一。

## 安装

### 安装 mmagic

请参考[官网安装指南](https://github.com/open-mmlab/mmagic/tree/main#installation)。

```shell

```shell
```

## 模型转换

你可以使用 [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/tree/main/tools/deploy.py) 把 mmagic 模型一键式转换为推理后端模型。
该工具的详细使用说明请参考[这里](https://github.com/open-mmlab/mmdeploy/tree/main/docs/zh_cn/02-how-to-run/convert_model.md#使用方法).

文件的命名模式是：

```
```

其中：

文件的命名模式是：

```
```

其中：

  mmagic 中任务有多种。目前，mmdeploy 支持其中的超分（super resolution）任务。关于`模型-任务`的划分，请参考章节[模型支持列表](#模型支持列表)。

  **请务必**使用对应的部署文件转换相关的模型。

### 超分任务模型转换

以下，我们将演示如何把 `ESRGAN` 转换为 onnx 模型。

```shell
# download esrgan model from mmagic model zoo
mim download mmagic --config esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k --dest .
# convert esrgan model to onnxruntime model with dynamic shape
  configs/mmagic/super-resolution/super-resolution_onnxruntime_dynamic.py \
  esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py \
  esrgan_psnr_x4c64b23g32_1x16_1000k_div2k_20200420-bf5c993c.pth \
  demo/resources/face.png \
  --work-dir mmdeploy_models/mmagic/ort \

esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py \
  --show \
```

你也可以把 `ESRGAN` 转为其他后端模型。比如使用`super-resolution/super-resolution_tensorrt-_dynamic-32x32-512x512.py`，把模型转为 tensorrt 模型。

```{tip}
```

## 模型规范

上例中的`mmdeploy_models/mmagic/ort`，结构如下：

```
mmdeploy_models/mmagic/ort
```

重要的是：

## 模型推理

### 后端模型推理

```python

## 模型推理

### 后端模型推理

```python

deploy_cfg = 'configs/mmagic/super-resolution/super-resolution_onnxruntime_dynamic.py'
model_cfg = 'esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py'
backend_model = ['./mmdeploy_models/mmagic/ort/end2end.onnx']
image = './demo/resources/face.png'

```

```python
from mmdeploy_runtime import Restorer
import cv2

img = cv2.imread('./demo/resources/face.png')

restorer = Restorer(model_path='./mmdeploy_models/mmagic/ort', device_name='cpu', device_id=0)

result = result[..., ::-1]
```

## 模型支持列表

| Model                                                                             | Task             | ONNX Runtime | TensorRT | ncnn | PPLNN | OpenVINO |

## 模型支持列表

| :-------------------------------------------------------------------------------- | :--------------- | :----------: | :------: | :--: | :---: | :------: |
| [SRCNN](https://github.com/open-mmlab/mmagic/tree/main/configs/srcnn)             | super-resolution |      Y       |    Y     |  Y   |   Y   |    Y     |
| [ESRGAN](https://github.com/open-mmlab/mmagic/tree/main/configs/esrgan)           | super-resolution |      Y       |    Y     |  Y   |   Y   |    Y     |
| [ESRGAN-PSNR](https://github.com/open-mmlab/mmagic/tree/main/configs/esrgan)      | super-resolution |      Y       |    Y     |  Y   |   Y   |    Y     |
| [SRGAN](https://github.com/open-mmlab/mmagic/tree/main/configs/srgan_resnet)      | super-resolution |      Y       |    Y     |  Y   |   Y   |    Y     |

| [SRResNet](https://github.com/open-mmlab/mmagic/tree/main/configs/srgan_resnet)   | super-resolution |      Y       |    Y     |  Y   |   Y   |    Y     |
| [Real-ESRGAN](https://github.com/open-mmlab/mmagic/tree/main/configs/real_esrgan) | super-resolution |      Y       |    Y     |  Y   |   Y   |    Y     |
| [EDSR](https://github.com/open-mmlab/mmagic/tree/main/configs/edsr)               | super-resolution |      Y       |    Y     |  Y   |   N   |    Y     |
| [RDN](https://github.com/open-mmlab/mmagic/tree/main/configs/rdn)                 | super-resolution |      Y       |    Y     |  Y   |   Y   |    Y     |

To utilize ViTPose, you'll need to have [MMPreTrain](https://github.com/open-mmlab/mmpretrain). To install the required version, run the following command:

```shell
mim install 'mmpretrain>=1.0.0'
```

<details>

<summary  align="right"><a  href="https://arxiv.org/abs/2204.12484">ViTPose (NeurIPS'2022)</a></summary>

```bibtex
  title={Vi{TP}ose: Simple Vision Transformer Baselines for Human Pose Estimation},
}
```

<details>

```bibtex

```bibtex
}
```

<details>

```bibtex
```

> With classic decoder

```

| [ViTPose-S-coco](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-small_8xb64-210e_coco-256x192.py) |  256x192   | 0.228 |      0.371      |      0.229      | 0.298 |      0.467      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-small_8xb64-210e_coco-256x192-62d7a712_20230314.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-small_8xb64-210e_coco-256x192-62d7a712_20230314.json) |

| [ViTPose-S-humanart-coco](configs/body_2d_keypoint/topdown_heatmap/humanart/td-hm_ViTPose-small_8xb64-210e_humanart-256x192.py) |  256x192   | 0.381 |      0.532      |      0.405      | 0.448 |      0.602      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-small_8xb64-210e_humanart-256x192-5cbe2bfc_20230611.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-small_8xb64-210e_humanart-256x192-5cbe2bfc_20230611.json) |

| [ViTPose-B-coco](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-base_8xb64-210e_coco-256x192.py) |  256x192   | 0.270 |      0.423      |      0.272      | 0.340 |      0.510      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-base_8xb64-210e_coco-256x192-216eae50_20230314.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-base_8xb64-210e_coco-256x192-216eae50_20230314.json) |

| [ViTPose-B-humanart-coco](configs/body_2d_keypoint/topdown_heatmap/humanart/td-hm_ViTPose-base_8xb64-210e_humanart-256x192.py) |  256x192   | 0.410 |      0.549      |      0.434      | 0.475 |      0.615      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-base_8xb64-210e_humanart-256x192-b417f546_20230611.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-base_8xb64-210e_humanart-256x192-b417f546_20230611.json) |

| [ViTPose-L-coco](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-base_8xb64-210e_coco-256x192.py) |  256x192   | 0.342 |      0.498      |      0.357      | 0.413 |      0.577      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-large_8xb64-210e_coco-256x192-53609f55_20230314.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-large_8xb64-210e_coco-256x192-53609f55_20230314.json) |

| [ViTPose-L-humanart-coco](configs/body_2d_keypoint/topdown_heatmap/humanart/td-hm_ViTPose-base_8xb64-210e_humanart-256x192.py) |  256x192   | 0.459 |      0.592      |      0.487      | 0.525 |      0.656      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-large_8xb64-210e_humanart-256x192-9aba9345_20230614.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-large_8xb64-210e_humanart-256x192-9aba9345_20230614.json) |

| [ViTPose-H-coco](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-huge_8xb64-210e_coco-256x192.py) |  256x192   | 0.377 |      0.541      |      0.391      | 0.447 |      0.615      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-huge_8xb64-210e_coco-256x192-e32adcd4_20230314.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-huge_8xb64-210e_coco-256x192-e32adcd4_20230314.json) |

| [ViTPose-H-humanart-coco](configs/body_2d_keypoint/topdown_heatmap/humanart/td-hm_ViTPose-huge_8xb64-210e_humanart-256x192.py) |  256x192   | 0.468 |      0.594      |      0.498      | 0.534 |      0.655      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-huge_8xb64-210e_humanart-256x192-603bb573_20230612.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-huge_8xb64-210e_humanart-256x192-603bb573_20230612.json) |

| [ViTPose-S-coco](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-small_8xb64-210e_coco-256x192.py) |  256x192   | 0.507 |      0.758      |      0.531      | 0.551 |      0.780      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-small_8xb64-210e_coco-256x192-62d7a712_20230314.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-small_8xb64-210e_coco-256x192-62d7a712_20230314.json) |

| [ViTPose-S-humanart-coco](configs/body_2d_keypoint/topdown_heatmap/humanart/td-hm_ViTPose-small_8xb64-210e_humanart-256x192.py) |  256x192   | 0.738 |      0.905      |      0.802      | 0.768 |      0.911      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-small_8xb64-210e_humanart-256x192-5cbe2bfc_20230611.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-small_8xb64-210e_humanart-256x192-5cbe2bfc_20230611.json) |

| [ViTPose-B-coco](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-base_8xb64-210e_coco-256x192.py) |  256x192   | 0.555 |      0.782      |      0.590      | 0.599 |      0.809      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-base_8xb64-210e_coco-256x192-216eae50_20230314.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-base_8xb64-210e_coco-256x192-216eae50_20230314.json) |

| [ViTPose-B-humanart-coco](configs/body_2d_keypoint/topdown_heatmap/humanart/td-hm_ViTPose-base_8xb64-210e_humanart-256x192.py) |  256x192   | 0.759 |      0.905      |      0.823      | 0.790 |      0.917      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-base_8xb64-210e_humanart-256x192-b417f546_20230611.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-base_8xb64-210e_humanart-256x192-b417f546_20230611.json) |

| [ViTPose-L-coco](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-base_8xb64-210e_coco-256x192.py) |  256x192   | 0.637 |      0.838      |      0.689      | 0.677 |      0.859      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-large_8xb64-210e_coco-256x192-53609f55_20230314.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-large_8xb64-210e_coco-256x192-53609f55_20230314.json) |

| [ViTPose-L-humanart-coco](configs/body_2d_keypoint/topdown_heatmap/humanart/td-hm_ViTPose-base_8xb64-210e_humanart-256x192.py) |  256x192   | 0.789 |      0.916      |      0.845      | 0.819 |      0.929      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-large_8xb64-210e_humanart-256x192-9aba9345_20230614.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-large_8xb64-210e_humanart-256x192-9aba9345_20230614.json) |

| [ViTPose-H-coco](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-huge_8xb64-210e_coco-256x192.py) |  256x192   | 0.665 |      0.860      |      0.715      | 0.701 |      0.871      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-huge_8xb64-210e_coco-256x192-e32adcd4_20230314.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-huge_8xb64-210e_coco-256x192-e32adcd4_20230314.json) |

| [ViTPose-H-humanart-coco](configs/body_2d_keypoint/topdown_heatmap/humanart/td-hm_ViTPose-huge_8xb64-210e_humanart-256x192.py) |  256x192   | 0.800 |      0.926      |      0.855      | 0.828 |      0.933      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-huge_8xb64-210e_humanart-256x192-603bb573_20230612.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-huge_8xb64-210e_humanart-256x192-603bb573_20230612.json) |

| [ViTPose-S-coco](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-small_8xb64-210e_coco-256x192.py) |  256x192   | 0.739 |      0.903      |      0.816      | 0.792 |      0.942      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-small_8xb64-210e_coco-256x192-62d7a712_20230314.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-small_8xb64-210e_coco-256x192-62d7a712_20230314.json) |

| [ViTPose-S-humanart-coco](configs/body_2d_keypoint/topdown_heatmap/humanart/td-hm_ViTPose-small_8xb64-210e_humanart-256x192.py) |  256x192   | 0.737 |      0.902      |      0.811      | 0.792 |      0.942      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-small_8xb64-210e_humanart-256x192-5cbe2bfc_20230611.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-small_8xb64-210e_humanart-256x192-5cbe2bfc_20230611.json) |

| [ViTPose-B-coco](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-base_8xb64-210e_coco-256x192.py) |  256x192   | 0.757 |      0.905      |      0.829      | 0.810 |      0.946      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-base_8xb64-210e_coco-256x192-216eae50_20230314.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-base_8xb64-210e_coco-256x192-216eae50_20230314.json) |

| [ViTPose-B-humanart-coco](configs/body_2d_keypoint/topdown_heatmap/humanart/td-hm_ViTPose-base_8xb64-210e_humanart-256x192.py) |  256x192   | 0.758 |      0.906      |      0.829      | 0.812 |      0.946      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-base_8xb64-210e_humanart-256x192-b417f546_20230611.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-base_8xb64-210e_humanart-256x192-b417f546_20230611.json) |

| [ViTPose-L-coco](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-large_8xb64-210e_coco-256x192.py) |  256x192   | 0.782 |      0.914      |      0.850      | 0.834 |      0.952      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-large_8xb64-210e_coco-256x192-53609f55_20230314.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-large_8xb64-210e_coco-256x192-53609f55_20230314.json) |

| [ViTPose-L-humanart-coco](configs/body_2d_keypoint/topdown_heatmap/humanart/td-hm_ViTPose-base_8xb64-210e_humanart-256x192.py) |  256x192   | 0.782 |      0.914      |      0.849      | 0.835 |      0.953      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-large_8xb64-210e_humanart-256x192-9aba9345_20230614.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-large_8xb64-210e_humanart-256x192-9aba9345_20230614.json) |

| [ViTPose-H-coco](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-huge_8xb64-210e_coco-256x192.py) |  256x192   | 0.788 |      0.917      |      0.855      | 0.839 |      0.954      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-huge_8xb64-210e_coco-256x192-e32adcd4_20230314.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-huge_8xb64-210e_coco-256x192-e32adcd4_20230314.json) |

| [ViTPose-H-humanart-coco](configs/body_2d_keypoint/topdown_heatmap/humanart/td-hm_ViTPose-huge_8xb64-210e_humanart-256x192.py) |  256x192   | 0.788 |      0.914      |      0.853      | 0.841 |      0.956      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-huge_8xb64-210e_humanart-256x192-603bb573_20230612.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_ViTPose-huge_8xb64-210e_humanart-256x192-603bb573_20230612.json) |

MMDetection provides hundreds of pre-trained detection models in [Model Zoo](https://mmdetection.readthedocs.io/en/latest/model_zoo.html).
This note will show how to inference, which means using trained models to detect objects on images.

In MMDetection, a model is defined by a [configuration file](https://mmdetection.readthedocs.io/en/latest/user_guides/config.html) and existing model parameters are saved in a checkpoint file.

To start with, we recommend [RTMDet](https://github.com/open-mmlab/mmdetection/tree/main/configs/rtmdet) with this [configuration file](https://github.com/open-mmlab/mmdetection/blob/main/configs/rtmdet/rtmdet_l_8xb32-300e_coco.py) and this [checkpoint file](https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet_l_8xb32-300e_coco/rtmdet_l_8xb32-300e_coco_20220719_112030-5a0be7c4.pth). It is recommended to download the checkpoint file to `checkpoints` directory.

## High-level APIs for inference

MMDetection provides high-level Python APIs for inference on images. Here is an example of building the model and inference on given images or videos.

```python
import cv2

# Build the model from a config file and a checkpoint file

# Init visualizer
# The dataset_meta is loaded from the checkpoint and
# then pass to the model in init_detector

# Test a single image and show the results

img = 'test.jpg'  # or img = mmcv.imread(img), which will only load it once

# Show the results

    'result',
    img,

# Test a video and show the results
# Build test pipeline

# visualizer has been created in line 31 and 34, if you run this demo in one notebook,
# you need not build the visualizer again.

# The interval of show (ms), 0 is block

video_reader = mmcv.VideoReader('video.mp4')

```

A notebook demo can be found in [demo/inference_demo.ipynb](https://github.com/open-mmlab/mmdetection/blob/main/demo/inference_demo.ipynb).

Note:  `inference_detector` only supports single-image inference for now.

## Demos

## Demos

We also provide three demo scripts, implemented with high-level APIs and supporting functionality codes.
Source codes are available [here](https://github.com/open-mmlab/mmdetection/blob/main/demo).

### Image demo

This script performs inference on a single image.

```shell
```

Examples:

```shell
```

#### Webcam demo

This is a live demo from a webcam.

```shell
```

Examples:

```shell

```shell
```

Examples:

```shell
```

### Video demo

This script performs inference on a video.

```shell
    [--show] \
```

Examples:

```shell
```

#### Video demo with GPU acceleration

This script performs inference on a video with GPU acceleration.

```shell

--out result.mp4
```

```shell
    [--nvdecode] \
    [--show] \
```

Examples:

```shell
```

## Multi-modal algorithm inference demo and evaluation

checkpoints/rtmdet_l_8xb32-300e_coco_20220719_112030-5a0be7c4.pth \
```

As multimodal vision algorithms continue to evolve, MMDetection has also supported such algorithms. This section demonstrates how to use the demo and eval scripts corresponding to multimodal algorithms using the GLIP algorithm and model as the example. Moreover, MMDetection integrated a [gradio_demo project](../../../projects/gradio_demo/), which allows developers to quickly play with all image input tasks in MMDetection on their local devices. Check the [document](../../../projects/gradio_demo/README.md) for more details.

### Preparation

Please first make sure that you have the correct dependencies installed:

```shell
# if source

# if wheel
```

```shell
# if source

# if wheel
```

MMDetection has already implemented GLIP algorithms and provided the weights, you can download directly from urls:

```shell
```

Once the model is successfully downloaded, you can use the `demo/image_demo.py` script to run the inference.

```shell
```

Demo result will be similar to this:

</div>

If users would like to detect multiple targets, please declare them in the format of `xx . xx .` after the `--texts`.

```shell

</div>

```shell
```

And the result will be like this one:

</div>

You can also use a sentence as the input prompt for the `--texts` field, for example:

```shell
```

The result will be similar to this:

</div>

The GLIP implementation in MMDetection does not have any performance degradation, our benchmark is as follows:

</div>

Users can use the test script we provided to run evaluation as well. Here is a basic example:

```shell
# 1 gpu

# 8 GPU
```

```shell

```shell
```

> **Task**: Video Interpolation, Video Super-Resolution

Many video enhancement algorithms rely on optical flow to register frames in a video sequence. Precise flow estimation is however intractable; and optical flow itself is often a sub-optimal representation for particular video processing tasks. In this paper, we propose task-oriented flow (TOFlow), a motion representation learned in a self-supervised, task-specific manner. We design a neural network with a trainable motion estimation component and a video processing component, and train them jointly to learn the task-oriented flow. For evaluation, we build Vimeo-90K, a large-scale, high-quality video dataset for low-level video processing. TOFlow outperforms traditional optical flow on standard benchmarks as well as our Vimeo-90K dataset in three video processing tasks: frame interpolation, video denoising/deblocking, and video super-resolution.

 <img src="https://user-images.githubusercontent.com/7676947/144035477-2480d580-1409-4a7c-88d5-c13a3dbd62ac.png" width="400"/>
</div >

Evaluated on Vimeo90k-triplet (RGB channels).

|                      Model                      |     Dataset      |                      Pretrained SPyNet                       |  PSNR   | Training Resources  |                      Download                       |
| :---------------------------------------------: | :--------------: | :----------------------------------------------------------: | :-----: | :-----------------: | :-------------------------------------------------: |

| [tof_vfi_spynet_chair_nobn_1xb1_vimeo90k](./tof_spynet-chair-wobn_1xb1_vimeo90k-triplet.py) | Vimeo90k-triplet | [spynet_chairs_final](https://download.openmmlab.com/mmediting/video_interpolators/toflow/pretrained_spynet_chair_20220321-4d82e91b.pth) | 33.3294 | 1 (Tesla PG503-216) | [model](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_chair_nobn_1xb1_vimeo90k_20220321-2fc9e258.pth) \| [log](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_chair_nobn_1xb1_vimeo90k_20220321-2fc9e258.log.json) |

| [tof_vfi_spynet_kitti_nobn_1xb1_vimeo90k](./tof_spynet-kitti-wobn_1xb1_vimeo90k-triplet.py) | Vimeo90k-triplet | [spynet_chairs_final](https://download.openmmlab.com/mmediting/video_interpolators/toflow/pretrained_spynet_kitti_20220321-dbcc1cc1.pth) | 33.3339 | 1 (Tesla PG503-216) | [model](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_kitti_nobn_1xb1_vimeo90k_20220321-3f7ca4cd.pth) \| [log](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_kitti_nobn_1xb1_vimeo90k_20220321-3f7ca4cd.log.json) |

| [tof_vfi_spynet_sintel_clean_nobn_1xb1_vimeo90k](./tof_spynet-sintel-wobn-clean_1xb1_vimeo90k-triplet.py) | Vimeo90k-triplet | [spynet_chairs_final](https://download.openmmlab.com/mmediting/video_interpolators/toflow/pretrained_spynet_sintel_clean_20220321-0756630b.pth) | 33.3170 | 1 (Tesla PG503-216) | [model](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_sintel_clean_nobn_1xb1_vimeo90k_20220321-6e52a6fd.pth) \| [log](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_sintel_clean_nobn_1xb1_vimeo90k_20220321-6e52a6fd.log.json) |

| [tof_vfi_spynet_sintel_final_nobn_1xb1_vimeo90k](./tof_spynet-sintel-wobn-final_1xb1_vimeo90k-triplet.py) | Vimeo90k-triplet | [spynet_chairs_final](https://download.openmmlab.com/mmediting/video_interpolators/toflow/pretrained_spynet_sintel_final_20220321-5e89dcec.pth) | 33.3237 | 1 (Tesla PG503-216) | [model](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_sintel_final_nobn_1xb1_vimeo90k_20220321-8ab70dbb.pth) \| [log](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_sintel_final_nobn_1xb1_vimeo90k_20220321-8ab70dbb.log.json) |

| [tof_vfi_spynet_pytoflow_nobn_1xb1_vimeo90k](./tof_spynet-pytoflow-wobn_1xb1_vimeo90k-triplet.py) | Vimeo90k-triplet | [spynet_chairs_final](https://download.openmmlab.com/mmediting/video_interpolators/toflow/pretrained_spynet_pytoflow_20220321-5bab842d.pth) | 33.3426 | 1 (Tesla PG503-216) | [model](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_pytoflow_nobn_1xb1_vimeo90k_20220321-5f4b243e.pth) \| [log](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_pytoflow_nobn_1xb1_vimeo90k_20220321-5f4b243e.log.json) |

|                      Model                       |     Dataset      |                      Pretrained SPyNet                       |  SSIM  | Training Resources  |                      Download                       |

| :----------------------------------------------: | :--------------: | :----------------------------------------------------------: | :----: | :-----------------: | :-------------------------------------------------: |
| [tof_vfi_spynet_chair_nobn_1xb1_vimeo90k](./tof_spynet-chair-wobn_1xb1_vimeo90k-triplet.py) | Vimeo90k-triplet | [spynet_chairs_final](https://download.openmmlab.com/mmediting/video_interpolators/toflow/pretrained_spynet_chair_20220321-4d82e91b.pth) | 0.9465 | 1 (Tesla PG503-216) | [model](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_chair_nobn_1xb1_vimeo90k_20220321-2fc9e258.pth) \| [log](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_chair_nobn_1xb1_vimeo90k_20220321-2fc9e258.log.json) |

| [tof_vfi_spynet_kitti_nobn_1xb1_vimeo90k](./tof_spynet-kitti-wobn_1xb1_vimeo90k-triplet.py) | Vimeo90k-triplet | [spynet_chairs_final](https://download.openmmlab.com/mmediting/video_interpolators/toflow/pretrained_spynet_kitti_20220321-dbcc1cc1.pth) | 0.9466 | 1 (Tesla PG503-216) | [model](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_kitti_nobn_1xb1_vimeo90k_20220321-3f7ca4cd.pth) \| [log](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_kitti_nobn_1xb1_vimeo90k_20220321-3f7ca4cd.log.json) |

| [tof_vfi_spynet_sintel_clean_nobn_1xb1_vimeo90k](./tof_spynet-sintel-wobn-clean_1xb1_vimeo90k-triplet.py) | Vimeo90k-triplet | [spynet_chairs_final](https://download.openmmlab.com/mmediting/video_interpolators/toflow/pretrained_spynet_sintel_clean_20220321-0756630b.pth) | 0.9464 | 1 (Tesla PG503-216) | [model](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_sintel_clean_nobn_1xb1_vimeo90k_20220321-6e52a6fd.pth) \| [log](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_sintel_clean_nobn_1xb1_vimeo90k_20220321-6e52a6fd.log.json) |

| [tof_vfi_spynet_sintel_final_nobn_1xb1_vimeo90k](./tof_spynet-sintel-wobn-final_1xb1_vimeo90k-triplet.py) | Vimeo90k-triplet | [spynet_chairs_final](https://download.openmmlab.com/mmediting/video_interpolators/toflow/pretrained_spynet_sintel_final_20220321-5e89dcec.pth) | 0.9465 | 1 (Tesla PG503-216) | [model](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_sintel_final_nobn_1xb1_vimeo90k_20220321-8ab70dbb.pth) \| [log](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_sintel_final_nobn_1xb1_vimeo90k_20220321-8ab70dbb.log.json) |

| [tof_vfi_spynet_pytoflow_nobn_1xb1_vimeo90k](./tof_spynet-pytoflow-wobn_1xb1_vimeo90k-triplet.py) | Vimeo90k-triplet | [spynet_chairs_final](https://download.openmmlab.com/mmediting/video_interpolators/toflow/pretrained_spynet_pytoflow_20220321-5bab842d.pth) | 0.9467 | 1 (Tesla PG503-216) | [model](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_pytoflow_nobn_1xb1_vimeo90k_20220321-5f4b243e.pth) \| [log](https://download.openmmlab.com/mmediting/video_interpolators/toflow/tof_vfi_spynet_pytoflow_nobn_1xb1_vimeo90k_20220321-5f4b243e.log.json) |

Note: These pretrained SPyNets don't contain BN layer since `batch_size=1`, which is consistent with `https://github.com/Coldog2333/pytoflow`.

|                           Model                           | Dataset  |       Vid4       | Training Resources |                                               Download                                                |
| :-------------------------------------------------------: | :------: | :--------------: | :----------------: | :---------------------------------------------------------------------------------------------------: |
| [tof_x4_vimeo90k_official](./tof_x4_official_vimeo90k.py) | vimeo90k | 24.4377 / 0.7433 |         -          | [model](https://download.openmmlab.com/mmediting/restorers/tof/tof_x4_vimeo90k_official-a569ff50.pth) |

**Train**

<details>

TOF only supports video interpolation task for training now.

```shell
# cpu train

```shell
# cpu train

```

**Test**

<details>

TOF supports two tasks for testing.

**Task 1**: Video Interpolation

```shell
# cpu test

```shell
# cpu test

```

**Task 2**: Video Super-Resolution

```shell
# cpu test

```

```shell
# cpu test

```

```bibtex

```bibtex
}
```

# Metric Calculation

In the evaluation phase, we typically select the corresponding evaluation metric strategy based on the characteristics of the dataset itself. The main criterion is the **type of standard answer**, generally including the following types:

- **Choice**: Common in classification tasks, judgment questions, and multiple-choice questions. Currently, this type of question dataset occupies the largest proportion, with datasets such as MMLU, CEval, etc. Accuracy is usually used as the evaluation standard-- `ACCEvaluator`.
- **Phrase**: Common in Q&A and reading comprehension tasks. This type of dataset mainly includes CLUE_CMRC, CLUE_DRCD, DROP datasets, etc. Matching rate is usually used as the evaluation standard--`EMEvaluator`.

- **Sentence**: Common in translation and generating pseudocode/command-line tasks, mainly including Flores, Summscreen, Govrepcrs, Iwdlt2017 datasets, etc. BLEU (Bilingual Evaluation Understudy) is usually used as the evaluation standard--`BleuEvaluator`.
- **Paragraph**: Common in text summary generation tasks, commonly used datasets mainly include Lcsts, TruthfulQA, Xsum datasets, etc. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is usually used as the evaluation standard--`RougeEvaluator`.
- **Code**: Common in code generation tasks, commonly used datasets mainly include Humaneval, MBPP datasets, etc. Execution pass rate and `pass@k` are usually used as the evaluation standard. At present, Opencompass supports `MBPPEvaluator` and `HumanEvaluator`.

There is also a type of **scoring-type** evaluation task without standard answers, such as judging whether the output of a model is toxic, which can directly use the related API service for scoring. At present, it supports `ToxicEvaluator`, and currently, the realtoxicityprompts dataset uses this evaluation method.

## Supported Evaluation Metrics

Currently, in OpenCompass, commonly used Evaluators are mainly located in the [`opencompass/openicl/icl_evaluator`](https://github.com/InternLM/opencompass/tree/main/opencompass/openicl/icl_evaluator) folder. There are also some dataset-specific indicators that are placed in parts of [`opencompass/datasets`](https://github.com/InternLM/opencompass/tree/main/opencompass/datasets). Below is a summary:

| Evaluation Strategy | Evaluation Metrics   | Common Postprocessing Method | Datasets                                                             |

| ------------------- | -------------------- | ---------------------------- | -------------------------------------------------------------------- |
| `ACCEvaluator`      | Accuracy             | `first_capital_postprocess`  | agieval, ARC, bbh, mmlu, ceval, commonsenseqa, crowspairs, hellaswag |
| `EMEvaluator`       | Match Rate           | None, dataset-specific       | drop, CLUE_CMRC, CLUE_DRCD                                           |
| `BleuEvaluator`     | BLEU                 | None, `flores`               | flores, iwslt2017, summscreen, govrepcrs                             |
| `RougeEvaluator`    | ROUGE                | None, dataset-specific       | lcsts, truthfulqa, Xsum, XLSum                                       |

| `HumanEvaluator`    | pass@k               | `humaneval_postprocess`      | humaneval_postprocess                                                |
| `MBPPEvaluator`     | Execution Pass Rate  | None                         | mbpp                                                                 |
| `ToxicEvaluator`    | PerspectiveAPI       | None                         | realtoxicityprompts                                                  |
| `AGIEvalEvaluator`  | Accuracy             | None                         | agieval                                                              |
| `AUCROCEvaluator`   | AUC-ROC              | None                         | jigsawmultilingual, civilcomments                                    |

| `MATHEvaluator`     | Accuracy             | `math_postprocess`           | math                                                                 |
| `MccEvaluator`      | Matthews Correlation | None                         | --                                                                   |
| `SquadEvaluator`    | F1-scores            | None                         | --                                                                   |

## How to Configure

The evaluation standard configuration is generally placed in the dataset configuration file, and the final xxdataset_eval_cfg will be passed to `dataset.infer_cfg` as an instantiation parameter.

Below is the definition of `govrepcrs_eval_cfg`, and you can refer to [configs/datasets/govrepcrs](https://github.com/InternLM/opencompass/tree/main/configs/datasets/govrepcrs).

```python

# Configuration of evaluation metrics
    evaluator=dict(type=BleuEvaluator),            # Use the common translator evaluator BleuEvaluator
    pred_role='BOT',                               # Accept 'BOT' role output
    pred_postprocessor=dict(type=general_cn_postprocess),      # Postprocessing of prediction results
    dataset_postprocessor=dict(type=general_cn_postprocess))   # Postprocessing of dataset standard answers

    dict(

dataset_postprocessor=dict(type=general_cn_postprocess))   # Postprocessing of dataset standard answers

    dict(
        type=GovRepcrsDataset,                 # Dataset class name
        path='./data/govrep/',                 # Dataset path
        abbr='GovRepcrs',                      # Dataset alias
        reader_cfg=govrepcrs_reader_cfg,       # Dataset reading configuration file, configure its reading split, column, etc.
        infer_cfg=govrepcrs_infer_cfg,         # Dataset inference configuration file, mainly related to prompt
        eval_cfg=govrepcrs_eval_cfg)           # Dataset result evaluation configuration file, evaluation standard, and preprocessing and postprocessing.
]
```

# ConvNeXt

> [A ConvNet for the 2020s](https://arxiv.org/abs/2201.03545v1)

**ConvNeXt** is initially described in [A ConvNet for the 2020s](https://arxiv.org/abs/2201.03545v1), which is a pure convolutional model (ConvNet), inspired by the design of Vision Transformers. The ConvNeXt has the pyramid structure and achieve competitive  performance on various vision tasks, with simplicity and efficiency.

<img src="https://user-images.githubusercontent.com/8370623/148624004-e9581042-ea4d-4e10-b3bd-42c92b02053b.png" width="100%"/>
</div>

<details>

<br>

</br>

```python

predict = inference_model('convnext-tiny_32xb128_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('convnext-tiny_32xb128_in1k', pretrained=True)
```

Train:

```shell
python tools/train.py configs/convnext/convnext-tiny_32xb128_in1k.py
```

Test:

```shell

Train:

```shell
```

Test:

```shell
python tools/test.py configs/convnext/convnext-tiny_32xb128_in1k.py https://download.openmmlab.com/mmclassification/v0/convnext/convnext-tiny_32xb128_in1k_20221207-998cf3e9.pth
```

| Model                              | Params (M) | Flops (G) |                  Config                   |                                                  Download                                                  |
| :--------------------------------- | :--------: | :-------: | :---------------------------------------: | :--------------------------------------------------------------------------------------------------------: |

| `convnext-base_3rdparty_in21k`\*   |   88.59    |   15.36   | [config](convnext-base_32xb128_in21k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-base_3rdparty_in21k_20220124-13b83eec.pth) |
| `convnext-large_3rdparty_in21k`\*  |   197.77   |   34.37   | [config](convnext-large_64xb64_in21k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-large_3rdparty_in21k_20220124-41b5a79f.pth) |
| `convnext-xlarge_3rdparty_in21k`\* |   350.20   |   60.93   | [config](convnext-xlarge_64xb64_in21k.py) | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-xlarge_3rdparty_in21k_20220124-f909bad7.pth) |

*Models with * are converted from the [official repo](https://github.com/facebookresearch/ConvNeXt). The config files of these models are only for inference. We haven't reproduce the training results.*

| `convnext-tiny_32xb128_in1k`                      | From scratch |   28.59    |   4.46    |   82.14   |   96.06   |    [config](convnext-tiny_32xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-tiny_32xb128_in1k_20221207-998cf3e9.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-tiny_32xb128_in1k_20221207-998cf3e9.json) |

| `convnext-tiny_32xb128-noema_in1k`                | From scratch |   28.59    |   4.46    |   81.95   |   95.89   |    [config](convnext-tiny_32xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-tiny_32xb128-noema_in1k_20221208-5d4509c7.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-tiny_32xb128_in1k_20221207-998cf3e9.json) |
| `convnext-tiny_in21k-pre_3rdparty_in1k`\*         | ImageNet-21k |   28.59    |   4.46    |   82.90   |   96.62   |    [config](convnext-tiny_32xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-tiny_in21k-pre_3rdparty_in1k_20221219-7501e534.pth) |

| `convnext-tiny_in21k-pre_3rdparty_in1k-384px`\*   | ImageNet-21k |   28.59    |   13.14   |   84.11   |   97.14   | [config](convnext-tiny_32xb128_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-tiny_in21k-pre_3rdparty_in1k-384px_20221219-c1182362.pth) |
| `convnext-small_32xb128_in1k`                     | From scratch |   50.22    |   8.69    |   83.16   |   96.56   |    [config](convnext-small_32xb128_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-small_32xb128_in1k_20221207-4ab7052c.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-small_32xb128_in1k_20221207-4ab7052c.json) |

| `convnext-small_32xb128-noema_in1k`               | From scratch |   50.22    |   8.69    |   83.21   |   96.48   |    [config](convnext-small_32xb128_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-small_32xb128-noema_in1k_20221208-4a618995.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-small_32xb128_in1k_20221207-4ab7052c.json) |
| `convnext-small_in21k-pre_3rdparty_in1k`\*        | ImageNet-21k |   50.22    |   8.69    |   84.59   |   97.41   |    [config](convnext-small_32xb128_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-small_in21k-pre_3rdparty_in1k_20221219-aeca4c93.pth) |

| `convnext-small_in21k-pre_3rdparty_in1k-384px`\*  | ImageNet-21k |   50.22    |   25.58   |   85.75   |   97.88   | [config](convnext-small_32xb128_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-small_in21k-pre_3rdparty_in1k-384px_20221219-96f0bb87.pth) |
| `convnext-base_32xb128_in1k`                      | From scratch |   88.59    |   15.36   |   83.66   |   96.74   |    [config](convnext-base_32xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-base_32xb128_in1k_20221207-fbdb5eb9.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-base_32xb128_in1k_20221207-fbdb5eb9.json) |

| `convnext-base_32xb128-noema_in1k`                | From scratch |   88.59    |   15.36   |   83.64   |   96.61   |    [config](convnext-base_32xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-base_32xb128-noema_in1k_20221208-f8182678.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-base_32xb128_in1k_20221207-fbdb5eb9.json) |
| `convnext-base_3rdparty_in1k`\*                   | From scratch |   88.59    |   15.36   |   83.85   |   96.74   |    [config](convnext-base_32xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-base_3rdparty_32xb128_in1k_20220124-d0915162.pth) |

| `convnext-base_3rdparty-noema_in1k`\*             | From scratch |   88.59    |   15.36   |   83.71   |   96.60   |    [config](convnext-base_32xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-base_3rdparty_32xb128-noema_in1k_20220222-dba4f95f.pth) |
| `convnext-base_3rdparty_in1k-384px`\*             | From scratch |   88.59    |   45.21   |   85.10   |   97.34   | [config](convnext-base_32xb128_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-base_3rdparty_in1k-384px_20221219-c8f1dc2b.pth) |
| `convnext-base_in21k-pre_3rdparty_in1k`\*         | ImageNet-21k |   88.59    |   15.36   |   85.81   |   97.86   |    [config](convnext-base_32xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-base_in21k-pre-3rdparty_32xb128_in1k_20220124-eb2d6ada.pth) |

| `convnext-base_in21k-pre-3rdparty_in1k-384px`\*   | From scratch |   88.59    |   45.21   |   86.82   |   98.25   | [config](convnext-base_32xb128_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-base_in21k-pre-3rdparty_in1k-384px_20221219-4570f792.pth) |
| `convnext-large_3rdparty_in1k`\*                  | From scratch |   197.77   |   34.37   |   84.30   |   96.89   |    [config](convnext-large_64xb64_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-large_3rdparty_64xb64_in1k_20220124-f8a0ded0.pth) |
| `convnext-large_3rdparty_in1k-384px`\*            | From scratch |   197.77   |  101.10   |   85.50   |   97.59   | [config](convnext-large_64xb64_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-large_3rdparty_in1k-384px_20221219-6dd29d10.pth) |

| `convnext-large_in21k-pre_3rdparty_in1k`\*        | ImageNet-21k |   197.77   |   34.37   |   86.61   |   98.04   |    [config](convnext-large_64xb64_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-large_in21k-pre-3rdparty_64xb64_in1k_20220124-2412403d.pth) |
| `convnext-large_in21k-pre-3rdparty_in1k-384px`\*  | From scratch |   197.77   |  101.10   |   87.46   |   98.37   | [config](convnext-large_64xb64_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-large_in21k-pre-3rdparty_in1k-384px_20221219-6d38dd66.pth) |
| `convnext-xlarge_in21k-pre_3rdparty_in1k`\*       | ImageNet-21k |   350.20   |   60.93   |   86.97   |   98.20   |    [config](convnext-xlarge_64xb64_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-xlarge_in21k-pre-3rdparty_64xb64_in1k_20220124-76b6863d.pth) |

| `convnext-xlarge_in21k-pre-3rdparty_in1k-384px`\* | From scratch |   350.20   |  179.20   |   87.76   |   98.55   | [config](convnext-xlarge_64xb64_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/convnext/convnext-xlarge_in21k-pre-3rdparty_in1k-384px_20221219-b161bc14.pth) |

```bibtex
@Article{liu2022convnet,
  author  = {Zhuang Liu and Hanzi Mao and Chao-Yuan Wu and Christoph Feichtenhofer and Trevor Darrell and Saining Xie},
  title   = {A ConvNet for the 2020s},
  journal = {arXiv preprint arXiv:2201.03545},
}
```

We use alexnet as an example

### prepare caffe prototxt and model

These files will usually generated when trained with caffe
```
train.prototxt
deploy.prototxt
snapshot_10000.caffemodel
```
deploy.prototxt and caffemodel file are enough for TEST phase

alexnet deploy.prototxt can be downloaded here

https://github.com/BVLC/caffe/tree/master/models/bvlc_alexnet

alexnet caffemodel can be downloaded here

http://dl.caffe.berkeleyvision.org/bvlc_alexnet.caffemodel

### convert to ncnn model

Convert old caffe prototxt and caffemodel to new ones using tools in caffe

because the ncnn convert tool needs the new format
```
upgrade_net_proto_text [old prototxt] [new prototxt]
upgrade_net_proto_binary [old caffemodel] [new caffemodel]
```

Use Input layer as input, set N dim as 1 since only one image can be processed each time
```
layer {
}
```

```
layer {
}
```
Use caffe2ncnn tool to convert caffe model to ncnn model
```
caffe2ncnn deploy.prototxt bvlc_alexnet.caffemodel alexnet.param alexnet.bin
```

### strip visible string

It is already enough for deploying with param and bin file only, but there are visible strings in param file, it may not be suitable to distribute plain neural network information in your APP.

You can use ncnn2mem tool to convert plain model file to binary representation. It will generate alexnet.param.bin and two static array code files.
```
ncnn2mem alexnet.param alexnet.bin alexnet.id.h alexnet.mem.h
```

### load model

Load param and bin file, the easy way
```cpp
ncnn::Net net;
net.load_param("alexnet.param");
net.load_model("alexnet.bin");
```
Load binary param.bin and bin file, no visible strings included, suitable for bundled as APP resource
```cpp

```
```cpp
net.load_param_bin("alexnet.param.bin");
```
Load network and model from external memory, no visible strings included, no external resource files bundled, the whole model is hardcoded in your program

You may use this way to load from android asset resource
```cpp
#include "alexnet.mem.h"
net.load_param(alexnet_param_bin);
net.load_model(alexnet_bin);
```
You can choose either way to load model. Loading from external memory is zero-copy, which means you must keep your memory buffer during processing

### unload model
```cpp
net.clear();
```

### input and output

ncnn Mat is the data structure for input and output data

Input image should be converted to Mat, and subtracted mean values and normalized when needed

```cpp
#include "mat.h"

```cpp
unsigned char* rgbdata;// data pointer to RGB image pixels
int w;// image width
int h;// image height
ncnn::Mat in = ncnn::Mat::from_pixels(rgbdata, ncnn::Mat::PIXEL_RGB, w, h);

const float mean_vals[3] = {104.f, 117.f, 123.f};
in.substract_mean_normalize(mean_vals, 0);
```
Execute the network inference and retrieve the result
```cpp
ncnn::Mat in;// input blob as above
ncnn::Mat out;
ncnn::Extractor ex = net.create_extractor();
ex.set_light_mode(true);
ex.input("data", in);
ex.extract("prob", out);
```
If you load model with binary param.bin file, you should use the enum value in alexnet.id.h file instead of the blob name
```cpp
#include "alexnet.id.h"

```cpp
ex.input(alexnet_param_id::BLOB_data, in);
ex.extract(alexnet_param_id::BLOB_prob, out);
```
Read the data in the output Mat. Iterate data to get all classification scores.
```cpp
ncnn::Mat out_flatterned = out.reshape(out.w * out.h * out.c);
std::vector<float> scores;
scores.resize(out_flatterned.w);
for (int j=0; j<out_flatterned.w; j++)
{
    scores[j] = out_flatterned[j];
}
```

### some tricks

Set multithreading thread number with Extractor
```cpp
ex.set_num_threads(4);
```
Convert image colorspace and resize image with Mat convenient function, these functions are well optimized

Support RGB2GRAY GRAY2RGB RGB2BGR etc, support scale up and scale down
```cpp

```cpp
int target_width = 227;// target resized width
int target_height = 227;// target resized height
ncnn::Mat in = ncnn::Mat::from_pixels_resize(rgbdata, ncnn::Mat::PIXEL_RGB2GRAY, w, h, target_width, target_height);
```
You can concat multiple model files into one, and load this single file from FILE* interface.

It should ease the distribution of param and model files.

> $ cat alexnet.param.bin alexnet.bin > alexnet-all.bin

```cpp
FILE* fp = fopen("alexnet-all.bin", "rb");
net.load_param_bin(fp);
net.load_model(fp);
fclose(fp);
```

>
> Please read our [Projects FAQ](../faq.md) if you still feel unclear about the requirements, or raise an [issue](https://github.com/open-mmlab/mmpose/issues) to us!

>
>

This project implements a top-down pose estimator with custom head and loss functions that have been seamlessly inherited from existing modules within MMPose.

## Usage

- [MMPose](https://github.com/open-mmlab/mmpose) v1.0.0rc0 or higher

```shell
```

Prepare the COCO dataset according to the [instruction](https://mmpose.readthedocs.io/en/dev-1.x/dataset_zoo/2d_body_keypoint.html#coco).

```shell
mim train mmpose configs/example-head-loss_hrnet-w32_8xb64-210e_coco-256x192.py
```

```shell

```shell
```

```shell
mim train mmpose configs/example-head-loss_hrnet-w32_8xb64-210e_coco-256x192.py --launcher pytorch --gpus 8
```

```shell
mim train mmpose configs/example-head-loss_hrnet-w32_8xb64-210e_coco-256x192.py --launcher slurm \
```

```shell
mim test mmpose configs/example-head-loss_hrnet-w32_8xb64-210e_coco-256x192.py $CHECKPOINT
```

```shell
mim test mmpose configs/example-head-loss_hrnet-w32_8xb64-210e_coco-256x192.py $CHECKPOINT --launcher pytorch --gpus 8
```

```shell
mim test mmpose configs/example-head-loss_hrnet-w32_8xb64-210e_coco-256x192.py $CHECKPOINT --launcher slurm \

```

```shell
```

## Results

> List the results as usually done in other model's README. Here is an [Example](https://github.com/open-mmlab/mmpose/blob/dev-1.x/configs/body_2d_keypoint/topdown_heatmap/coco/hrnet_coco.md).

|                             Model                             | Backbone  | Input Size |  AP   | AP<sup>50</sup> | AP<sup>75</sup> |  AR   | AR<sup>50</sup> |                             Download                              |

| :-----------------------------------------------------------: | :-------: | :--------: | :---: | :-------------: | :-------------: | :---: | :-------------: | :---------------------------------------------------------------: |
| [ExampleHead + ExampleLoss](./configs/example-head-loss_hrnet-w32_8xb64-210e_coco-256x192.py) | HRNet-w32 |  256x912   | 0.749 |      0.906      |      0.821      | 0.804 |      0.945      | [model](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192-81c58e40_20220909.pth) \| [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192_20220909.log) |

```bibtex
}
```

}
```

to MMPose projects.

    > The code's design shall follow existing interfaces and convention. For example, each model component should be registered into `mmpose.registry.MODELS` and configurable via a config file.

    > Each major class should contains a docstring, describing its functionality and arguments. If your code is copied or modified from other open-source projects, don't forget to cite the source project in docstring and make sure your behavior is not against its license. Typically, we do not accept any code snippet under GPL license. [A Short Guide to Open Source Licenses](https://medium.com/nationwide-technology/a-short-guide-to-open-source-licenses-cf5b1c329edd)

    > Ideally *all* the methods should have [type hints](https://www.pythontutorial.net/python-basics/python-type-hints/) and [docstrings](https://google.github.io/styleguide/pyguide.html#381-docstrings). [Example](https://github.com/open-mmlab/mmpose/blob/0fb7f22000197181dc0629f767dd99d881d23d76/mmpose/utils/tensor_utils.py#L53)

    > Unit tests for the major module are required. [Example](https://github.com/open-mmlab/mmpose/blob/dev-1.x/tests/test_models/test_heads/test_heatmap_heads/test_heatmap_head.py)

    > It will be parsed by MIM and Inferencer. [Example](https://github.com/open-mmlab/mmpose/blob/dev-1.x/configs/body_2d_keypoint/topdown_heatmap/coco/hrnet_coco.yml)

  - [ ] Move your modules into the core package following the codebase's file hierarchy structure.

    > In particular, you may have to refactor this README into a standard one. [Example](https://github.com/open-mmlab/mmpose/blob/dev-1.x/configs/body_2d_keypoint/topdown_heatmap/README.md)

  - [ ] Refactor your modules into the core package following the codebase's file hierarchy structure.

Associative Embedding is one of the most popular 2D bottom-up pose estimation approaches, that first detect all the keypoints and then group/associate them into person instances.

In order to group all the predicted keypoints to individuals, a tag is also predicted for each detected keypoint. Tags of the same person are similar, while tags of different people are different. Thus the keypoints can be grouped according to the tags.

</div>

# Core ML 支持情况

目前 mmdeploy 集成了 OpenMMLab 算法库中 Pytorch 模型到 Core ML模型的转换以及推理。

## 安装

转换 mmdet 中的模型，需要编译 libtorch 支持 nms 等自定义算子

```bash
cd ${PYTORCH_DIR}
cmake .. \
    -DDISABLE_SVE=ON # 低版本比如1.8.0需要加上这个参数
```

## 使用

```bash
    configs/mmdet/detection/detection_coreml_static-800x1344.py \
    /mmdetection_dir/configs/retinanet/retinanet_r18_fpn_1x_coco.py \
    /checkpoint/retinanet_r18_fpn_1x_coco_20220407_171055-614fd399.pth \
    /mmdetection_dir/demo/demo.jpg \
    --work-dir work_dir/retinanet \
```

```c
// v寄存器全部使用 %.4s
// 128-bit vreg matches %.4s
// a += b * c
    "fmla  %0.4s, %2.4s, %3.4s"
    : "0"(_a),
    :
);
```
```c
// v寄存器使用低64位  %.2s
// low 64-bit vreg matches %.2s
// a += b * c
    "fmla  %0.2s, %2.2s, %3.2s"
    : "0"(_a),
    :
);
```
```c
// v寄存器单路使用 %.s[0] %.s[1] %.s[2] %.s[3]
// 32-bit register matches %.s[0]
float32x4_t _a = vld1_f32(a);
float32x4_t _b = vld1_f32(b);
float32x4_t _c = vld1_f32(c);
    "fmla  %0.4s, %2.4s, %3.s[0]"
    "fmla  %0.4s, %2.4s, %3.s[1]"
    "fmla  %0.4s, %2.4s, %3.s[2]"
    "fmla  %0.4s, %2.4s, %3.s[3]"

    : "0"(_a),
    :
);
```

qwq

# 3D Body Keypoint Datasets

- [Human3.6M](#human36m) \[ [Homepage](http://vision.imar.ro/human3.6m/description.php) \]
- [CMU Panoptic](#cmu-panoptic) \[ [Homepage](http://domedb.perception.cs.cmu.edu/) \]
- [Campus/Shelf](#campus-and-shelf) \[ [Homepage](http://campar.in.tum.de/Chair/MultiHumanPose) \]

<details>

```bibtex

}
```

  <img src="https://user-images.githubusercontent.com/100993824/227866174-0355e7dd-6965-4d1a-8bb6-32e8dce4cefc.png" height="300px">
</div>

For [Human3.6M](http://vision.imar.ro/human3.6m/description.php), please download from the official website and run the [preprocessing script](/tools/dataset_converters/preprocess_h36m.py), which will extract camera parameters and pose annotations at full framerate (50 FPS) and downsampled framerate (10 FPS). The processed data should have the following structure:

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    ├── h36m

|   |   ├── ...
            |   ├── ...
            ├── S5
            ├── S6
            ├── S7
            ├── S8
            ├── S9
            `── S11
```

## CMU Panoptic

<details>

```bibtex
}
```

  <img src="https://user-images.githubusercontent.com/100993824/227866246-1af9441b-4f6c-4340-ae98-e40d6b4e8e55.png" height="300px">
</div>

Please follow [voxelpose-pytorch](https://github.com/microsoft/voxelpose-pytorch) to prepare this dataset.

</div>

1. Download the dataset by following the instructions in [panoptic-toolbox](https://github.com/CMU-Perceptual-Computing-Lab/panoptic-toolbox) and extract them under `$MMPOSE/data/panoptic`.

2. Only download those sequences that are needed. You can also just download a subset of camera views by specifying the number of views (HD_Video_Number) and changing the camera order in `./scripts/getData.sh`. The used sequences and camera views can be found in [VoxelPose](https://arxiv.org/abs/2004.06239). Note that the sequence "160906_band3" might not be available due to errors on the server of CMU Panoptic.

3. Note that we only use HD videos,  calibration data, and 3D Body Keypoint in the codes. You can comment out other irrelevant codes such as downloading 3D Face data in `./scripts/getData.sh`.

The directory tree should be like this:

```text
mmpose
├── mmpose
├── docs

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    ├── panoptic
        ├── 16060224_haggling1
        |   |   ├── hdImgs
        |   |   ├── hdvideos
        |   |   ├── hdPose3d_stage1_coco19
        |   |   ├── calibration_160224_haggling1.json
        ├── 160226_haggling1
            ├── ...
```

## Campus and Shelf

<details>
<summary align="right"><a href="http://campar.in.tum.de/pub/belagiannis2014cvpr/belagiannis2014cvpr.pdf">Campus and Shelf (CVPR'2014)</a></summary>

```bibtex
@inproceedings {belagian14multi,
    title = {{3D} Pictorial Structures for Multiple Human Pose Estimation},
    author = {Belagiannis, Vasileios and Amin, Sikandar and Andriluka, Mykhaylo and Schiele, Bernt and Navab
    Nassir and Ilic, Slobo
    booktitle = {IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR)},
    year = {2014},
    organization={IEEE}
}
```

booktitle = {IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR)},
}
```

  <img src="https://user-images.githubusercontent.com/100993824/227866373-ed8e6d1a-581b-4b88-93f5-2bc2caddbf6c.png" height="300px">
</div>

Please follow [voxelpose-pytorch](https://github.com/microsoft/voxelpose-pytorch) to prepare these two datasets.

1. Please download the datasets from the [official website](http://campar.in.tum.de/Chair/MultiHumanPose) and extract them under `$MMPOSE/data/campus` and `$MMPOSE/data/shelf`, respectively. The original data include images as well as the ground truth pose file `actorsGT.mat`.

2. We directly use the processed camera parameters from [voxelpose-pytorch](https://github.com/microsoft/voxelpose-pytorch). You can download them from this repository and place in under `$MMPOSE/data/campus/calibration_campus.json` and `$MMPOSE/data/shelf/calibration_shelf.json`, respectively.

3. Like [Voxelpose](https://github.com/microsoft/voxelpose-pytorch), due to the limited and incomplete annotations of the two datasets, we don't train the model using this dataset. Instead, we directly use the 2D pose estimator trained on COCO, and use independent 3D human poses from the CMU Panoptic dataset to train our 3D model. It lies in `${MMPOSE}/data/panoptic_training_pose.pkl`.

4. Like [Voxelpose](https://github.com/microsoft/voxelpose-pytorch), for testing, we first estimate 2D poses and generate 2D heatmaps for these two datasets. You can download the predicted poses from [voxelpose-pytorch](https://github.com/microsoft/voxelpose-pytorch) and place them in  `$MMPOSE/data/campus/pred_campus_maskrcnn_hrnet_coco.pkl` and `$MMPOSE/data/shelf/pred_shelf_maskrcnn_hrnet_coco.pkl`, respectively. You can also use the models trained on COCO dataset (like HigherHRNet) to generate 2D heatmaps directly.

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    ├── panoptic_training_pose.pkl
    ├── campus
    |   ├── Camera0
    |   |   |   ├── campus4-c0-00000.png
    |   |   |   ├── ...
    |   |   |   ├── campus4-c0-01999.png
    |   ...
    |   ├── Camera2
    |   |   |   ├── campus4-c2-00000.png
    |   |   |   ├── ...
    |   |   |   ├── campus4-c2-01999.png
    |   ├── calibration_campus.json

|   ...
    |   |   |   ├── ...
    |   ├── pred_campus_maskrcnn_hrnet_coco.pkl
    |   ├── actorsGT.mat
    ├── shelf
    |   |   |   ├── img_000000.png
    |   |   |   ├── ...
    |   |   |   ├── img_003199.png
    |   ...
    |   ├── Camera4
    |   |   |   ├── ...
    |   ├── calibration_shelf.json
    |   ├── pred_shelf_maskrcnn_hrnet_coco.pkl
```

# Customize Data Pipeline

In this tutorial, we will introduce some methods about how to build the data pipeline (i.e., data transformations) for your tasks.

- [Customize Data Pipeline](#customize-data-pipeline)
  - [Design of Data Pipeline](#design-of-data-pipeline)
  - [Modify the Training/Testing Pipeline](#modify-the-trainingtest-pipeline)
    - [Loading](#loading)
    - [Sampling Frames and Other Processing](#sampling-frames-and-other-processing)
  - [Add New Data Transforms](#add-new-data-transforms)

## Design of Data Pipeline

The data pipeline refers to the procedure of handling the data sample dict when indexing a sample from the dataset, and comprises a series of data transforms. Each data transform accepts a `dict` as input, processes it, and produces a `dict` as output for the subsequent data transform in the sequence.

Below is an example data pipeline for training SlowFast on Kinetics using `VideoDataset`. The pipeline initially employs [`decord`](https://github.com/dmlc/decord) to read the raw videos and randomly sample one video clip, which comprises `32` frames with a frame interval of `2`. Subsequently, it applies random resized crop and random horizontal flip to all frames before formatting the data shape as `NCTHW`, which is `(1, 3, 32, 224, 224)` in this example.

```python
    dict(type='DecordInit',),
    dict(type='SampleFrames', clip_len=32, frame_interval=2, num_clips=1),
    dict(type='RandomResizedCrop'),
    dict(type='FormatShape', input_format='NCTHW'),
]
```

dict(type='Resize', scale=(224, 224), keep_ratio=False),
]
```

A comprehensive list of all available data transforms in MMAction2 can be found in the [mmaction.datasets.transforms](mmaction.datasets.transforms).

## Modify the Training/Testing Pipeline

The data pipeline in MMAction2 is highly adaptable, as nearly every step of the data preprocessing can be configured from the config file. However, the wide array of options may be overwhelming for some users.

Below are some general practices and guidance for building a data pipeline for action recognition tasks.

### Loading

At the beginning of a data pipeline, it is customary to load videos. However, if the frames have already been extracted, you should utilize `RawFrameDecode` and modify the dataset type to `RawframeDataset`.

```python

```python
    dict(type='RawFrameDecode'),
]
```

If you need to load data from files with distinct formats (e.g., `pkl`, `bin`, etc.) or from specific locations, you may create a new loading transform and include it at the beginning of the data pipeline. Please refer to [Add New Data Transforms](#add-new-data-transforms) for more details.

### Sampling Frames and Other Processing

During training and testing, we may have different strategies to sample frames from the video.

For instance, when testing SlowFast, we uniformly sample multiple clips as follows:

```python
    ...
    dict(

```python
    ...
    dict(
        clip_len=32,
        frame_interval=2,
        num_clips=10,
    ...
]
```

In the above example, 10 video clips, each comprising 32 frames, will be uniformly sampled from each video. `test_mode=True` is employed to accomplish this, as opposed to random sampling during training.

Another example involves `TSN/TSM` models, which sample multiple segments from the video:

```python
    ...
    dict(type='SampleFrames', clip_len=1, frame_interval=1, num_clips=8),
    ...
]
```

Typically, the data augmentations in the data pipeline handles only video-level transforms, such as resizing or cropping, but not transforms like video normalization or mixup/cutmix. This is because we can do video normalization and mixup/cutmix on batched video data

to accelerate processing using GPUs. To configure video normalization and mixup/cutmix, please use the [mmaction.models.utils.data_preprocessor](mmaction.models.utils.data_preprocessor).

Formatting involves collecting training data from the data information dict and converting it into a format that is compatible with the model.

In most cases, you can simply employ [`PackActionInputs`](mmaction.datasets.transforms.PackActionInputs), and it will
convert the image in `NumPy Array` format to `PyTorch Tensor`, and pack the ground truth category information and
other meta information as a dict-like object [`ActionDataSample`](mmaction.structures.ActionDataSample).

```python
    ...
    dict(type='PackActionInputs'),
]
```

## Add New Data Transforms

1. To create a new data transform, write a new transform class in a python file named, for example, `my_transforms.py`. The data transform classes must inherit

]
```

   the [`mmcv.transforms.BaseTransform`](mmcv.transforms.BaseTransform) class and override the `transform` method which takes a `dict` as input and returns a `dict`. Finally, place `my_transforms.py` in the folder `mmaction/datasets/transforms/`.

   ```python
   from mmaction.datasets import TRANSFORMS

        def __init__(self, msg):
            self.msg = msg

           # Modify the data information dict `results`.
           print(msg, 'MMAction2.')
   ```

2. Import the new class in the `mmaction/datasets/transforms/__init__.py`.

   ```python
   ...
   from .my_transform import MyTransform

   __all__ = [

   ```

   ```python
   ...

   __all__ = [
       ..., 'MyTransform'
   ]
   ```

3. Use it in config files.

   ```python
       ...
       dict(type='MyTransform', msg='Hello!'),
       ...
   ]
   ```

# Learning Temporal Pose Estimation from Sparsely-Labeled Videos

<details>
<summary align="right"><a href="https://arxiv.org/abs/1906.04016">PoseWarper (NeurIPS'2019)</a></summary>

```bibtex
@inproceedings{NIPS2019_gberta,
title = {Learning Temporal Pose Estimation from Sparsely Labeled Videos},
author = {Bertasius, Gedas and Feichtenhofer, Christoph, and Tran, Du and Shi, Jianbo, and Torresani, Lorenzo},
booktitle = {Advances in Neural Information Processing Systems 33},
}
```

Modern approaches for multi-person pose estimation in video require large amounts of dense annotations. However, labeling every frame in a video is costly and labor intensive. To reduce the need for dense annotations, we propose a PoseWarper network that leverages training videos with sparse annotations (every k frames) to learn to perform dense temporal pose propagation and estimation. Given a pair of video frames---a labeled Frame A and an unlabeled Frame B---we train our model to predict human pose in Frame A using the features from Frame B by means of deformable convolutions to implicitly learn the pose warping between A and B. We demonstrate that we can leverage our trained PoseWarper for several applications. First, at inference time we can reverse the application direction of our network in order to propagate pose information from manually annotated frames to unlabeled frames. This makes it possible to generate pose annotations for the entire video given only a few manually-labeled frames. Compared to modern label propagation methods based on optical flow, our warping mechanism is much more compact (6M vs 39M parameters), and also more accurate (88.7% mAP vs 83.8% mAP). We also show that we can improve the accuracy of a pose estimator by training it on an augmented dataset obtained by adding our propagated poses to the original manual labels. Lastly, we can use our PoseWarper to aggregate temporal pose information from neighboring frames during inference. This allows our system to achieve state-of-the-art pose detection results on the PoseTrack2017 and PoseTrack2018 datasets.

<img src="https://user-images.githubusercontent.com/15977946/146521380-97f7c23c-4960-454c-bd93-426a9cca175c.png">
</div>

# Preparing Vid4 Dataset

```bibtex
}
```

The Vid4 dataset can be downloaded from [here](https://drive.google.com/file/d/1ZuvNNLgR85TV_whJoHM7uVb-XW1y70DW/view?usp=sharing). There are two degradations in the dataset.

1. BIx4 contains images downsampled by bicubic interpolation
2. BDx4 contains images blurred by Gaussian kernel with σ=1.6, followed by a subsampling every four pixels.

Note that we should prepare a annotation file (such as meta_info_Vid4_GT.txt) for Vid4 dataset as follows.

```text
```

For ToFlow, we should prepare directly upsampling dataset. We provide such a script:

```shell

```text
```

```shell
```

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── Vid4
│   │   ├── GT
│   │   ├── BDx4
│   │   ├── BIx4
```

# Ecosystem Projects based on MMAction2

There are many research works and projects built on MMAction2.
We list some of them as examples of how to extend MMAction2 for your own projects.

- [OTEAction2](https://github.com/openvinotoolkit/mmaction2): OpenVINO Training Extensions for Action Recognition.
- [PYSKL](https://github.com/kennymckormick/pyskl): A Toolbox Focusing on Skeleton-Based Action Recognition.

To make this list also a reference for the community to develop and compare new video understanding algorithms, we list them following the time order of top-tier conferences.
Methods already supported and maintained by MMAction2 are not listed.

- Video Swin Transformer, CVPR 2022. [\[paper\]](https://arxiv.org/abs/2106.13230)[\[github\]](https://github.com/SwinTransformer/Video-Swin-Transformer)
- Evidential Deep Learning for Open Set Action Recognition, ICCV 2021 Oral. [\[paper\]](https://arxiv.org/abs/2107.10161)[\[github\]](https://github.com/Cogito2012/DEAR)
- Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspective, ICCV 2021 Oral. [\[paper\]](https://arxiv.org/abs/2103.17263)[\[github\]](https://github.com/xvjiarui/VFS)
- MGSampler: An Explainable Sampling Strategy for Video Action Recognition, ICCV 2021. [\[paper\]](https://arxiv.org/abs/2104.09952)[\[github\]](https://github.com/MCG-NJU/MGSampler)
- MultiSports: A Multi-Person Video Dataset of Spatio-Temporally Localized Sports Actions, ICCV 2021. [\[paper\]](https://arxiv.org/abs/2105.07404)

- Long Short-Term Transformer for Online Action Detection, NeurIPS 2021 [\[paper\]](https://arxiv.org/abs/2107.03377)[\[github\]](https://github.com/amazon-research/long-short-term-transformer)

<details>

```bibtex
}
```

<details>

```bibtex
}
```

author={Wang, Xinyao and Bo, Liefeng and Fuxin, Li},
}
```

<details>

```bibtex
}
```

| [pose_hrnetv2_w18_awing](/configs/face_2d_keypoint/topdown_heatmap/wflw/td-hm_hrnetv2-w18_awing-8xb64-60e_wflw-256x256.py) |  256x256   |         4.02         |         6.94         |             3.97             |           4.78            |         4.59         |          3.87          |            4.28            | [ckpt](https://download.openmmlab.com/mmpose/face/hrnetv2/hrnetv2_w18_wflw_256x256_awing-5af5055c_20211212.pth) | [log](https://download.openmmlab.com/mmpose/face/hrnetv2/hrnetv2_w18_wflw_256x256_awing_20211212.log.json) |

# How to get partitioned ONNX models

MMDeploy supports exporting PyTorch models to partitioned onnx models. With this feature, users can define their partition policy and get partitioned onnx models at ease. In this tutorial, we will briefly introduce how to support partition a model step by step. In the example, we would break YOLOV3 model into two parts and extract the first part without the post-processing (such as anchor generating and NMS) in the onnx model.

## Step 1: Mark inputs/outpupts

To support the model partition, we need to add `Mark` nodes in the ONNX model. This could be done with mmdeploy's `@mark` decorator. Note that to make the `mark` work, the marking operation should be included in a rewriting function.

At first, we would mark the model input, which could be done by marking the input tensor `img` in the `forward` method of `BaseDetector` class, which is the parent class of all detector classes. Thus we name this marking point as `detector_forward` and mark the inputs as `input`. Since there could be three outputs for detectors such as `Mask RCNN`, the outputs are marked as  `dets`, `labels`, and `masks`. The following code shows the idea of adding mark functions and calling the mark functions in the rewrite. For source code, you could refer to [mmdeploy/codebase/mmdet/models/detectors/single_stage.py](https://github.com/open-mmlab/mmdeploy/blob/4fc8828af84281b62be143012cd9f9dafd1e7cc2/mmdeploy/codebase/mmdet/models/detectors/single_stage.py)

```python

@mark(
    ...

@mark(
    ...

    ...
```

Then, we have to mark the output feature of `YOLOV3Head`, which is the input argument `pred_maps` in `get_bboxes` method of `YOLOV3Head` class. We could add a internal function to only mark the `pred_maps` inside [`yolov3_head__get_bboxes`](https://github.com/open-mmlab/mmdeploy/blob/4fc8828af84281b62be143012cd9f9dafd1e7cc2/mmdeploy/codebase/mmdet/models/dense_heads/yolo_head.py#L16) function as following.

```python

```python

                            cfg=None,
    ...
```

Note that `pred_maps` is a list of `Tensor` and it has three elements. Thus, three `Mark` nodes with op name as `pred_maps.0`, `pred_maps.1`, `pred_maps.2` would be added in the onnx model.

## Step 2: Add partition config

After marking necessary nodes that would be used to split the model, we could add a deployment config file `configs/mmdet/detection/yolov3_partition_onnxruntime_static.py`. If you are not familiar with how to write config, you could check [write_config.md](../02-how-to-run/write_config.md).

In the config file, we need to add `partition_config`. The key part is `partition_cfg`, which contains elements of dict that designates the start nodes and end nodes of each model segments. Since we only want to keep `YOLOV3` without post-processing, we could set the `start` as `['detector_forward:input']`, and `end` as `['yolo_head:input']`. Note that `start` and `end` can have multiple marks.

```python

        dict(

        dict(
    ])

```

## Step 3: Get partitioned onnx models

Once we have marks of nodes and the deployment config with `parition_config` being set properly, we could use the [tool](../02-how-to-run/useful_tools.md) `torch2onnx` to export the model to onnx and get the partition onnx files.

```shell

```shell
```

After run the script above, we would have the partitioned onnx file `yolov3.onnx` in the `work-dir`. You can use the visualization tool [netron](https://netron.app/) to check the model structure.

With the partitioned onnx file, you could refer to [useful_tools.md](../02-how-to-run/useful_tools.md) to do the following procedures such as `mmdeploy_onnx2ncnn`, `onnx2tensorrt`.

# News

- **\[2023.08.10\]** OpenCompass is compatible with [LMDeploy](https://github.com/InternLM/lmdeploy). Now you can follow this [instruction](https://opencompass.readthedocs.io/en/latest/advanced_guides/evaluation_turbomind.html#) to evaluate the accelerated models provide by the **Turbomind**.
- **\[2023.08.10\]** We have supported [Qwen-7B](https://github.com/QwenLM/Qwen-7B) and [XVERSE-13B](https://github.com/xverse-ai/XVERSE-13B) ! Go to our [leaderboard](https://opencompass.org.cn/leaderboard-llm) for more results! More models are welcome to join OpenCompass.
- **\[2023.08.09\]** Several new datasets(**CMMLU, TydiQA, SQuAD2.0, DROP**) are updated on our [leaderboard](https://opencompass.org.cn/leaderboard-llm)! More datasets are welcomed to join OpenCompass.
- **\[2023.08.07\]** We have added a [script](tools/eval_mmbench.py) for users to evaluate the inference results of [MMBench](https://opencompass.org.cn/MMBench)-dev.

- **\[2023.08.05\]** We have supported [GPT-4](https://openai.com/gpt-4)! Go to our [leaderboard](https://opencompass.org.cn/leaderboard-llm) for more results! More models are welcome to join OpenCompass.
- **\[2023.07.27\]** We have supported [CMMLU](https://github.com/haonan-li/CMMLU)! More datasets are welcome to join OpenCompass.

This tutorial collects answers to any `How to xxx with MMDetection`. Feel free to update this doc if you meet new questions about `How to` and find the answers!

# Use backbone network through MMPretrain

The model registry in MMDet, MMPreTrain, MMSeg all inherit from the root registry in MMEngine. This allows these repositories to directly use the modules already implemented by each other. Therefore, users can use backbone networks from MMPretrain in MMDetection without implementing a network that already exists in MMPretrain.

## Use backbone network implemented in MMPretrain

Suppose you want to use `MobileNetV3-small` as the backbone network of `RetinaNet`, the example config is as the following.

```python
_base_ = [
    '../_base_/models/retinanet_r50_fpn.py',
]
# please install mmpretrain
# import mmpretrain.models to trigger register_module in mmpretrain

]
custom_imports = dict(imports=['mmpretrain.models'], allow_failed_imports=False)
pretrained = 'https://download.openmmlab.com/mmclassification/v0/mobilenet_v3/convert/mobilenet_v3_small-8427ecf0.pth'
        _delete_=True, # Delete the backbone field in _base_
        type='mmpretrain.MobileNetV3', # Using MobileNetV3 from mmpretrain
        arch='small',
        out_indices=(3, 8, 11), # Modify out_indices
            checkpoint=pretrained,
            prefix='backbone.')), # The pre-trained weights of backbone network in mmpretrain have prefix='backbone.'. The prefix in the keys will be removed so that these weights can be normally loaded.
    # Modify in_channels
    neck=dict(in_channels=[24, 48, 96], start_level=0))
```

# Modify in_channels
```

## Use backbone network in TIMM through MMPretrain

MMPretrain also provides a wrapper for the PyTorch Image Models (timm) backbone network, users can directly use the backbone network in timm through MMPretrain. Suppose you want to use [EfficientNet-B1](../../../configs/timm_example/retinanet_timm-efficientnet-b1_fpn_1x_coco.py) as the backbone network of RetinaNet, the example config is as the following.

```python
# https://github.com/open-mmlab/mmdetection/blob/main/configs/timm_example/retinanet_timm-efficientnet-b1_fpn_1x_coco.py

_base_ = [
]

]

        type='mmpretrain.TIMMBackbone', # Using timm from mmpretrain
        model_name='efficientnet_b1',
        features_only=True,
        pretrained=True,
        out_indices=(1, 2, 3, 4)), # Modify out_indices
    neck=dict(in_channels=[24, 40, 112, 320])) # Modify in_channels

```

`type='mmpretrain.TIMMBackbone'` means use the `TIMMBackbone` class from MMPretrain in MMDetection, and the model used is `EfficientNet-B1`, where `mmpretrain` means the MMPretrain repo and `TIMMBackbone` means the TIMMBackbone wrapper implemented in MMPretrain.

For the principle of the Hierarchy Registry, please refer to the [MMEngine document](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/config.md). For how to use other backbones in MMPretrain, you can refer to the [MMPretrain document](https://mmpretrain.readthedocs.io/en/latest/user_guides/config.html).

# Use Mosaic augmentation

If you want to use `Mosaic` in training, please make sure that you use `MultiImageMixDataset` at the same time. Taking the 'Faster R-CNN' algorithm as an example, you should modify the values of `train_pipeline` and `train_dataset` in the config as below:

```python
# Open configs/faster_rcnn/faster-rcnn_r50_fpn_1x_coco.py directly and add the following fields
img_scale=(1333, 800)

    dict(type='Mosaic', img_scale=img_scale, pad_val=114.0),
    dict(
        type='RandomAffine',
        scaling_ratio_range=(0.1, 2),

    dict(
        border=(-img_scale[0] // 2, -img_scale[1] // 2)), # The image will be enlarged by 4 times after Mosaic processing,so we use affine transformation to restore the image size.
]

    _delete_ = True, # remove unnecessary Settings
            dict(type='LoadAnnotations', with_bbox=True)
        ],
        filter_empty_gt=False,
    ),
    )

    train=train_dataset
    )
```

dict(type='LoadAnnotations', with_bbox=True)
        ],
    ),
    )

    )
```

# Unfreeze backbone network after freezing the backbone in the config

If you have freezed the backbone network in the config and want to unfreeze it after some epoches, you can write a hook function to do it.  Taking the Faster R-CNN with the resnet backbone as an example, you can freeze one stage of the backbone network and  add a `custom_hooks` in the config as below:

```python
_base_ = [
]
    # freeze one stage of the backbone network.
    backbone=dict(frozen_stages=1),
)
custom_hooks = [dict(type="UnfreezeBackboneEpochBasedHook", unfreeze_epoch=1)]
```

]
)
```

Meanwhile write the hook class `UnfreezeBackboneEpochBasedHook` in `mmdet/core/hook/unfreeze_backbone_epoch_based_hook.py`

```python

class UnfreezeBackboneEpochBasedHook(Hook):
    """Unfreeze backbone network Hook.

    Args:
        unfreeze_epoch (int): The epoch unfreezing the backbone network.
    """

    def __init__(self, unfreeze_epoch=1):
        self.unfreeze_epoch = unfreeze_epoch

    def before_train_epoch(self, runner):
        # Unfreeze the backbone network.
        # Only valid for resnet.
        if runner.epoch == self.unfreeze_epoch:
            model = runner.model
            if is_model_wrapper(model):
                model = model.module

# Only valid for resnet.
            backbone = model.backbone
            if backbone.frozen_stages >= 0:
                if backbone.deep_stem:
                    backbone.stem.train()
                    for param in backbone.stem.parameters():
                        param.requires_grad = True
                else:
                    backbone.norm1.train()
                    for m in [backbone.conv1, backbone.norm1]:
                        for param in m.parameters():
                            param.requires_grad = True

            for i in range(1, backbone.frozen_stages + 1):
                m = getattr(backbone, f'layer{i}')
                m.train()
                for param in m.parameters():
                    param.requires_grad = True
```

# Get the channels of a new backbone

m.train()
```

If you want to get the channels of a new backbone, you can build this backbone alone and input a pseudo image to get each stage output.

Take `ResNet` as an example:

```python
from mmdet.models import ResNet
self = ResNet(depth=18)
self.eval()
inputs = torch.rand(1, 3, 32, 32)
level_outputs = self.forward(inputs)
for level_out in level_outputs:
    print(tuple(level_out.shape))

```

Output of the above script is as below:

```python
(1, 64, 8, 8)
(1, 128, 4, 4)
(1, 256, 2, 2)
(1, 512, 1, 1)
```

Users can get the channels of the new backbone by Replacing the `ResNet(depth=18)` in this script with their customized backbone.

# Use Detectron2 Model in MMDetection

```

Users can use Detectron2Wrapper to run Detectron2's model in MMDetection. We provide examples of [Faster R-CNN](../../../configs/misc/d2_faster-rcnn_r50-caffe_fpn_ms-90k_coco.py),
[Mask R-CNN](../../../configs/misc/d2_mask-rcnn_r50-caffe_fpn_ms-90k_coco.py), and [RetinaNet](../../../configs/misc/d2_retinanet_r50-caffe_fpn_ms-90k_coco.py) in MMDetection.

The algorithm components in config file should be the same as those of in Detectron2. During setup, we will first initialize the default settings, which can be found in [Detectron2](https://github.com/facebookresearch/detectron2/blob/main/detectron2/config/defaults.py).
Then, the settings in config file will overwrite the default settings and the model will be built with these settings.

The input data will first convert to Detectron2's type and feed into Detectron2's model.
During inference the results calculate from Detectron2's model will reconvert back to the MMDetection's type.

## Use Detectron2's pre-trained weights

The weight initialization in `Detectron2Wrapper` will not use the logic of MMDetection. Users can set `model.d2_detector.weights=xxx` to load pre-trained weights.
For example, we can use `model.d2_detector.weights='detectron2://ImageNetPretrained/MSRA/R-50.pkl'` to load the pre-trained ResNet-50 or use
`model.d2_detector.weights='detectron2://COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_1x/137260431/model_final_a54504.pkl'` to load the pre-trained Mask R-CNN weights proposed in Detectron2.

**Note:** Detectron2's pretrained model cannot be loaded directly by using `load_from`, it should be first converted via `tools/model_converters/detectron2_to_mmdet.py`

For inference of released detectron2 checkpoints, users should first use `tools/model_converters/detectron2_to_mmdet.py` to convert Detectron2 checkpoint to MMDetection.

```shell
python tools/model_converters/detectron2_to_mmdet.py ${Detectron2 ckpt path} ${MMDetectron ckpt path}
```

# 2D Face Landmark Detection

2D face landmark detection (also referred to as face alignment) is defined as the task of detecting the face keypoints from an input image.

Normally, the input images are cropped face images, where the face locates at the center;
or the rough location (or the bounding box) of the hand is provided.

Please follow [DATA Preparation](/docs/en/dataset_zoo/2d_face_keypoint.md) to prepare data.

## Demo

Please follow [Demo](/demo/docs/en/2d_face_demo.md) to run demos.

<img src="https://user-images.githubusercontent.com/11788150/109144943-ccd44900-779c-11eb-9e9d-8682e7629654.gif" width="600px" alt><br>

# BLIP-2

> [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](http://arxiv.org/abs/2301.12597)

The cost of vision-and-language pre-training has become increasingly prohibitive due to end-toend training of large-scale models. This paper proposes BLIP-2, a generic and efficient pretraining strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models. BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pretrained in two stages. The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen language model. BLIP-2 achieves state-of-the-art performance on various visionlanguage tasks, despite having significantly fewer trainable parameters than existing methods. For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model’s emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.

<img src="https://user-images.githubusercontent.com/30762564/236385045-dc22a621-0a9c-4352-afa4-ca3888044850.png" width="70%"/>
</div>

```python

result = inference_model('blip2-opt2.7b_3rdparty-zeroshot_caption', 'demo/cat-dog.png')
# {'pred_caption': 'a dog and a cat sitting on a blanket'}
```

Test:

```shell
python tools/test.py configs/blip2/blip2_8xb32_retrieval.py https://download.openmmlab.com/mmclassification/v1/blip2/blip2_3rdparty_pretrain_20230505-f7ef4390.pth
```

```

| Model                                       | Params (M) | BLEU-4 | CIDER  |                   Config                   |                                           Download                                            |
| :------------------------------------------ | :--------: | :----: | :----: | :----------------------------------------: | :-------------------------------------------------------------------------------------------: |
| `blip2-opt2.7b_3rdparty-zeroshot_caption`\* |  3770.47   | 32.90  | 111.10 | [config](./blip2-opt2.7b_8xb32_caption.py) | [model](https://download.openmmlab.com/mmclassification/v1/blip2/blip2-opt2.7b_3rdparty_pretrain_20230505-b51db4e1.pth) |

| Model                                   | Params (M) | Accuracy |                 Config                 |                                                 Download                                                  |
| :-------------------------------------- | :--------: | :------: | :------------------------------------: | :-------------------------------------------------------------------------------------------------------: |
| `blip2-opt2.7b_3rdparty-zeroshot_vqa`\* |  3770.47   |  53.50   | [config](./blip2-opt2.7b_8xb16_vqa.py) | [model](https://download.openmmlab.com/mmclassification/v1/blip2/blip2-opt2.7b_3rdparty_pretrain_20230505-b51db4e1.pth) |

### Image-To-Text Retrieval on COCO

| Model                        | Params (M) | Recall@1 |                Config                |                                                    Download                                                     |

| :--------------------------- | :--------: | :------: | :----------------------------------: | :-------------------------------------------------------------------------------------------------------------: |
| `blip2_3rdparty_retrieval`\* |  1173.19   |  85.40   | [config](./blip2_8xb32_retrieval.py) | [model](https://download.openmmlab.com/mmclassification/v1/blip2/blip2_3rdparty_pretrain_20230505-f7ef4390.pth) |

*Models with * are converted from the [official repo](https://github.com/salesforce/LAVIS). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
    title={Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models},
    author={Li, Junnan and Li, Dongxu and Savarese, Silvio and Hoi, Steven},
    eprint={2301.12597},
}
```

# Data Structures and Elements

MMOCR uses {external+mmengine:doc}`MMEngine: Abstract Data Element <advanced_tutorials/data_element>` to encapsulate the data required for each task into `data_sample`. The base class has implemented basic add/delete/update/check functions and supports data migration between different devices, as well as dictionary-like and tensor-like operations, which also allows the interfaces of different algorithms to be unified.

Thanks to the unified data structures, the data flow between each module in the algorithm libraries, such as [`visualizer`](./visualizers.md), [`evaluator`](./evaluation.md), [`dataset`](./datasets.md), is greatly simplified. In MMOCR, we have the following conventions for different data types.

- **xxxData**: Single granularity data annotation or model output. Currently MMEngine has three built-in granularities of {external+mmengine:doc}`data elements <advanced_tutorials/data_element>`, including instance-level data (`InstanceData`), pixel-level data (`PixelData`) and image-level label data (`LabelData`). Among the tasks currently supported by MMOCR, text detection and key information extraction tasks use `InstanceData` to encapsulate the bounding boxes and the corresponding box label, while the text recognition task uses `LabelData` to encapsulate the text content.

- **xxxDataSample**: inherited from {external+mmengine:doc}`MMEngine: Base Data Element <advanced_tutorials/data_element>`, used to hold **all** annotation and prediction information that required by a single task. For example, [`TextDetDataSample`](mmocr.structures.textdet_data_sample.TextDetDataSample) for the text detection, [`TextRecogDataSample`](mmocr.structures.textrecog_data_sample.TextRecogDataSample) for text recognition, and [`KIEDataSample`](mmocr.structures.kie_data_sample.KIEDataSample) for the key information extraction task.

In the following, we will introduce the practical application of data elements **xxxData** and data samples **xxxDataSample** in MMOCR, respectively.

## Data Elements - xxxData

`InstanceData` and `LabelData` are the `BaseDataElement` defined in `MMEngine` to encapsulate different granularity of annotation data or model output. In MMOCR, we have used `InstanceData` and `LabelData` for encapsulating the data types actually used in OCR-related tasks.

In the **text detection** task, the detector concentrate on instance-level text samples, so we use `InstanceData` to encapsulate the data needed for this task. Typically, its required training annotation and prediction output contain rectangular or polygonal bounding boxes, as well as bounding box labels. Since the text detection task has only one positive sample class, "text", in MMOCR we use `0` to number this class by default. The following code example shows how to use the `InstanceData` to encapsulate the data used in the text detection task.

```python

# defining gt_instance for encapsulating the ground truth data

# defining pred_instance for encapsulating the prediction data
```

The conventions for the fields in `InstanceData` in MMOCR are shown in the table below. It is important to note that the length of each field in `InstanceData` must be equal to the number of instances `N` in the sample.

|             |                                    |                                                                                                                                                             |

| ----------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Field       | Type                               | Description                                                                                                                                                 |
| bboxes      | `torch.FloatTensor`                | Bounding boxes `[x1, y1, x2, y2]` with the shape `(N, 4)`.                                                                                                  |
| labels      | `torch.LongTensor`                 | Instance label with the shape `(N, )`. By default, MMOCR uses `0` to represent the "text" class.                                                            |

| polygons    | `list[np.array(dtype=np.float32)]` | Polygonal bounding boxes with the shape `(N, )`.                                                                                                            |
| scores      | `torch.Tensor`                     | Confidence scores of the predictions of bounding boxes. `(N, )`.                                                                                            |
| ignored     | `torch.BoolTensor`                 | Whether to ignore the current sample with the shape `(N, )`.                                                                                                |
| texts       | `list[str]`                        | The text content of each instance with the shape `(N, )`，used for e2e text spotting or KIE task.                                                           |

| text_scores | `torch.FloatTensor`                | Confidence score of the predictions of text contents with the shape `(N, )`，used for e2e text spotting task.                                               |
| edge_labels | `torch.IntTensor`                  | The node adjacency matrix with the shape `(N, N)`. In KIE, the optional values for the state between nodes are `-1` (ignored, not involved in loss calculation)，`0` (disconnected) and `1`(connected). |
| edge_scores | `torch.FloatTensor`                | The prediction confidence of each edge in the KIE task, with the shape `(N, N)`.                                                                            |

For **text recognition** tasks, both labeled content and predicted content are wrapped using `LabelData`.

```python

# defining gt_text for encapsulating the ground truth data

```python

# defining pred_text for encapsulating the prediction data
```

The conventions for the `LabelData` fields in MMOCR are shown in the following table.

|                |                    |                                                                                                                                                                          |
| -------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |

| Field          | Type               | Description                                                                                                                                                              |
| item           | `str`              | Text content.                                                                                                                                                            |
| score          | `list[float]`      | Confidence socre of the predicted text.                                                                                                                                  |
| indexes        | `torch.LongTensor` | A sequence of text characters encoded by [dictionary](../basic_concepts/models.md#dictionary) and containing all special characters except `<UNK>`.                      |

| padded_indexes | `torch.LongTensor` | If the length of indexes is less than the maximum sequence length and `pad_idx` exists, this field holds the encoded text sequence padded to the maximum sequence length of `max_seq_len`. |

## DataSample xxxDataSample

By defining a uniform data structure, we can easily encapsulate the annotation data and prediction results in a unified way, making data transfer between different modules of the code base easier. In MMOCR, we have designed three data structures based on the data needed in three tasks: [`TextDetDataSample`](mmocr.structures.textdet_data_sample.TextDetDataSample), [`TextRecogDataSample`](mmocr.structures.textrecog_data_sample.TextRecogDataSample), and [`KIEDataSample`](mmocr.structures.kie_data_sample.KIEDataSample). These data structures all inherit from {external+mmengine:doc}`MMEngine: Base Data Element <advanced_tutorials/data_element>`, which is used to hold all annotation and prediction information required by each task.

### Text Detection - TextDetDataSample

[TextDetDataSample](mmocr.structures.textdet_data_sample.TextDetDataSample) is used to encapsulate the data needed for the text detection task. It contains two main fields `gt_instances` and `pred_instances`, which are used to store the annotation information and prediction results respectively.

|                |                                 |                         |
| -------------- | ------------------------------- | ----------------------- |
| Field          | Type                            | Description             |
| gt_instances   | [`InstanceData`](#instancedata) | Annotation information. |
| pred_instances | [`InstanceData`](#instancedata) | Prediction results.     |

The fields of [`InstanceData`](#instancedata) that will be used are:

|          |                                    |                                                                                                  |

|          |                                    |                                                                                                  |
| -------- | ---------------------------------- | ------------------------------------------------------------------------------------------------ |
| Field    | Type                               | Description                                                                                      |
| bboxes   | `torch.FloatTensor`                | Bounding boxes `[x1, y1, x2, y2]` with the shape `(N, 4)`.                                       |
| labels   | `torch.LongTensor`                 | Instance label with the shape `(N, )`. By default, MMOCR uses `0` to represent the "text" class. |
| polygons | `list[np.array(dtype=np.float32)]` | Polygonal bounding boxes with the shape `(N, )`.                                                 |

| scores   | `torch.Tensor`                     | Confidence scores of the predictions of bounding boxes. `(N, )`.                                 |
| ignored  | `torch.BoolTensor`                 | Boolean flags with the shape `(N, )`, indicating whether to ignore the current sample.           |

Since text detection models usually only output one of the bboxes/polygons, we only need to make sure that one of these two is assigned a value.

The following sample code demonstrates the use of `TextDetDataSample`.

```python

# Define the ground truth data

# Define the prediction data
```

### Text Recognition - TextRecogDataSample

[`TextRecogDataSample`](mmocr.structures.textrecog_data_sample.TextRecogDataSample) is used to encapsulate the data for the text recognition task. It has two fields, `gt_text` and `pred_text` , which are used to store annotation information and prediction results, respectively.

|           |                                            |                     |
| --------- | ------------------------------------------ | ------------------- |

|           |                                            |                     |
| Field     | Type                                       | Description         |
| gt_text   | [`LabelData`](#text-recognition-labeldata) | Label information.  |
| pred_text | [`LabelData`](#text-recognition-labeldata) | Prediction results. |

The following sample code demonstrates the use of [`TextRecogDataSample`](mmocr.structures.textrecog_data_sample.TextRecogDataSample).

```python

```

```

The fields of `LabelData` that will be used are:

|                |                     |                                                                                                                                                                         |
| -------------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Field          | Type                | Description                                                                                                                                                             |

| item           | `list[str]`         | The text corresponding to the instance, of length (N, ), for end-to-end OCR tasks and KIE                                                                               |
| score          | `torch.FloatTensor` | Confidence of the text prediction, of length (N, ), for the end-to-end OCR task                                                                                         |
| indexes        | `torch.LongTensor`  | A sequence of text characters encoded by [dictionary](../basic_concepts/models.md#dictionary) and containing all special characters except `<UNK>`.                     |
| padded_indexes | `torch.LongTensor`  | If the length of indexes is less than the maximum sequence length and `pad_idx` exists, this field holds the encoded text sequence padded to the maximum sequence length of `max_seq_len`. |

### Key Information Extraction - KIEDataSample

[`KIEDataSample`](mmocr.structures.kie_data_sample.KIEDataSample) is used to encapsulate the data needed for the KIE task. It also contains two fields, `gt_instances` and `pred_instances`, which are used to store annotation information and prediction results respectively.

|                |                                                |                         |
| -------------- | ---------------------------------------------- | ----------------------- |
| Field          | Type                                           | Description             |
| gt_instances   | [`InstanceData`](#text-detection-instancedata) | Annotation information. |
| pred_instances | [`InstanceData`](#text-detection-instancedata) | Prediction results.     |

The [`InstanceData`](#text-detection-instancedata) fields that will be used by this task are shown in the following table.

|             |                     |                                                                                                                                                                            |
| ----------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Field       | Type                | Description                                                                                                                                                                |
| bboxes      | `torch.FloatTensor` | Bounding boxes `[x1, y1, x2, y2]` with the shape `(N, 4)`.                                                                                                                 |

| labels      | `torch.LongTensor`  | Instance label with the shape `(N, )`.                                                                                                                                     |
| texts       | `list[str]`         | The text content of each instance with the shape `(N, )`，used for e2e text spotting or KIE task.                                                                          |
| edge_labels | `torch.IntTensor`   | The node adjacency matrix with the shape `(N, N)`. In the KIE task, the optional values for the state between nodes are `-1` (ignored, not involved in loss calculation)，`0` (disconnected) and `1`(connected). |
| edge_scores | `torch.FloatTensor` | The prediction confidence of each edge in the KIE task, with the shape `(N, N)`.                                                                                           |

| scores      | `torch.FloatTensor` | The confidence scores for node label predictions, with the shape `(N,)`.                                                                                                   |

Since there is no unified standard for model implementation of KIE tasks, the design currently considers only [SDMGR](../../../configs/kie/sdmgr/README.md) model usage scenarios. Therefore, the design is subject to change as we support more KIE models.
```

The following sample code shows the use of [`KIEDataSample`](mmocr.structures.kie_data_sample.KIEDataSample).

```python

```

# 基于激光雷达的 3D 检测

基于激光雷达的 3D 检测是 MMDetection3D 支持的最基础的任务之一。它期望给定的模型以激光雷达采集的任意数量的特征点为输入，并为每一个感兴趣的目标预测 3D 框及类别标签。接下来，我们以 KITTI 数据集上的 PointPillars 为例，展示如何准备数据，在标准的 3D 检测基准上训练并测试模型，以及可视化并验证结果。

## 数据准备

首先，我们需要下载原始数据并按照[数据准备文档](https://mmdetection3d.readthedocs.io/zh_CN/dev-1.x/user_guides/dataset_prepare.html)中提供的标准方式重新组织数据。

由于不同数据集的原始数据有不同的组织方式，我们通常需要用 `.pkl` 文件收集有用的数据信息。因此，在准备好所有的原始数据之后，我们需要运行 `create_data.py` 中提供的脚本来为不同的数据集生成数据集信息。例如，对于 KITTI，我们需要运行如下命令：

```shell
```

随后，相关的目录结构将如下所示：

```
├── tools
├── data
│   ├── kitti
│   │   │   ├── velodyne_reduced
│   │   ├── kitti_gt_database

│   │   ├── kitti_infos_train.pkl
│   │   ├── kitti_infos_trainval.pkl
│   │   ├── kitti_infos_val.pkl
│   │   ├── kitti_infos_test.pkl
│   │   ├── kitti_dbinfos_train.pkl
```

## 训练

接着，我们将使用提供的配置文件训练 PointPillars。当您使用不同的 GPU 设置进行训练时，您可以按照这个[教程](https://mmdetection3d.readthedocs.io/en/dev-1.x/user_guides/train_test.html)的示例。假设我们在一台具有 8 块 GPU 的机器上使用分布式训练：

```shell
./tools/dist_train.sh configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py 8
```

注意，配置文件名中的 `8xb6` 是指训练用了 8 块 GPU，每块 GPU 上有 6 个数据样本。如果您的自定义设置不同于此，那么有时候您需要相应地调整学习率。基本规则可以参考[此处](https://arxiv.org/abs/1706.02677)。我们已经支持了使用 `--auto-scale-lr` 来自动缩放学习率。

## 定量评估

在训练期间，模型权重文件将会根据配置文件中的 `train_cfg = dict(val_interval=xxx)` 设置被周期性地评估。我们支持不同数据集的官方评估方案。对于 KITTI，将对 3 个类别使用交并比（IoU）阈值分别为 0.5/0.7 的平均精度（mAP）来评估模型。评估结果将会被打印到终端中，如下所示：

```

## 定量评估

```
bbox AP:98.1839, 89.7606, 88.7837
bev AP:89.6905, 87.4570, 85.4865
3d AP:87.4561, 76.7569, 74.1302
aos AP:97.70, 88.73, 87.34
bev AP:98.4400, 90.1218, 89.6270
3d AP:98.3329, 90.0209, 89.4035
```

此外，在训练完成后您也可以评估特定的模型权重文件。您可以简单地执行以下脚本：

```shell
./tools/dist_test.sh configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py work_dirs/pointpillars/latest.pth 8
```

## 测试与提交

```

```shell
```

## 测试与提交

如果您只想在在线基准上进行推理或测试模型性能，您需要在相应的评估器中指定 `submission_prefix`，例如，在配置文件中添加 `test_evaluator = dict(type='KittiMetric', ann_file=data_root + 'kitti_infos_test.pkl', format_only=True, pklfile_prefix='results/kitti-3class/kitti_results', submission_prefix='results/kitti-3class/kitti_results')`，然后可以得到结果文件。请确保配置文件中的[测试信息](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/datasets/kitti-3d-3class.py#L117)的 `data_prefix` 和 `ann_file` 由验证集相应地改为测试集。在生成结果后，您可以压缩文件夹并上传至 KITTI 评估服务器上。

## 定性评估

MMDetection3D 还提供了通用的可视化工具，以便于我们可以对训练好的模型预测的检测结果有一个直观的感受。您也可以在评估阶段通过设置 `--show` 来在线可视化检测结果，或者使用 `tools/misc/visualize_results.py` 来离线地进行可视化。此外，我们还提供了脚本 `tools/misc/browse_dataset.py` 用于可视化数据集而不做推理。更多的细节请参考[可视化文档](https://mmdetection3d.readthedocs.io/zh_CN/dev-1.x/user_guides/visualization.html)。

# 如何在 Jetson 模组上安装 MMDeploy

本教程将介绍如何在 NVIDIA Jetson 平台上安装 MMDeploy。该方法已经在以下 3 种 Jetson 模组上进行了验证：

- Jetson Nano
- Jetson TX2
- Jetson AGX Xavier

## 预备

首先需要在 Jetson 模组上安装 JetPack SDK。
此外，在利用 MMDeploy 的 Model Converter 转换 PyTorch 模型为 ONNX 模型时，需要创建一个装有 PyTorch 的环境。
最后，关于编译工具链，要求 CMake 和 GCC 的版本分别不低于 3.14 和 7.0。

### JetPack SDK

JetPack SDK 为构建硬件加速的边缘 AI 应用提供了一个全面的开发环境。
其支持所有的 Jetson 模组及开发套件。

主要有两种安装 JetPack SDK 的方式：

1. 使用 SD 卡镜像方式，直接将镜像刻录到 SD 卡上
2. 使用 NVIDIA SDK Manager 进行安装

你可以在 NVIDIA [官网](https://developer.nvidia.com/jetpack-sdk-50dp)上找到详细的安装指南。

这里我们选择 [JetPack 4.6.1](https://developer.nvidia.com/jetpack-sdk-461) 作为装配 Jetson 模组的首选。MMDeploy 已经在 JetPack 4.6 rev3 及以上版本，TensorRT 8.0.1.6 及以上版本进行了测试。更早的 JetPack 版本与 TensorRT 7.x 存在不兼容的情况。

### Conda

安装 [Archiconda](https://github.com/Archiconda/build-tools/releases) 而不是 Anaconda，因为后者不提供针对 Jetson 的 wheel 文件。

```shell
wget https://github.com/Archiconda/build-tools/releases/download/0.2.3/Archiconda3-0.2.3-Linux-aarch64.sh

```shell
bash Archiconda3-0.2.3-Linux-aarch64.sh -b

echo -e '\n# set environment variable for conda' >> ~/.bashrc
echo ". ~/archiconda3/etc/profile.d/conda.sh" >> ~/.bashrc
echo 'export PATH=$PATH:~/archiconda3/bin' >> ~/.bashrc

echo -e '\n# set environment variable for pip' >> ~/.bashrc
echo 'export OPENBLAS_CORETYPE=ARMV8' >> ~/.bashrc

conda --version
```

完成安装后需创建并启动一个 conda 环境。

```shell
# 得到默认安装的 python3 版本
export PYTHON_VERSION=`python3 --version | cut -d' ' -f 2 | cut -d'.' -f1,2`
conda create -y -n mmdeploy python=${PYTHON_VERSION}
```

```{note}
JetPack SDK 4+ 自带 python 3.6。我们强烈建议使用默认的 python 版本。尝试升级 python 可能会破坏 JetPack 环境。

如果必须安装更高版本的 python， 可以选择安装 JetPack 5+，其提供 python 3.8。
```

### PyTorch

```

```{note}

```

从[这里](https://forums.developer.nvidia.com/t/pytorch-for-jetson-version-1-10-now-available/72048)下载 Jetson 的 PyTorch wheel 文件并保存在本地目录 `/opt` 中。
此外，由于 torchvision 不提供针对 Jetson 平台的预编译包，因此需要从源码进行编译。

以 `torch 1.10.0` 和  `torchvision 0.11.1` 为例，可按以下方式进行安装：

```shell
# pytorch
wget https://nvidia.box.com/shared/static/fjtbno0vpo676a25cgvuqc1wty0fkkg6.whl -O torch-1.10.0-cp36-cp36m-linux_aarch64.whl
pip3 install torch-1.10.0-cp36-cp36m-linux_aarch64.whl
# torchvision
sudo apt-get install libjpeg-dev zlib1g-dev libpython3-dev libavcodec-dev libavformat-dev libswscale-dev libopenblas-dev -y
sudo rm -r torchvision
git clone https://github.com/pytorch/vision torchvision
cd torchvision
git checkout tags/v0.11.1 -b v0.11.1
export BUILD_VERSION=0.11.1
```

```

如果安装其他版本的 PyTorch 和 torchvision，需参考[这里](https://pypi.org/project/torchvision/)的表格以保证版本兼容性。

### CMake

这里我们使用 CMake 截至2022年4月的最新版本 v3.23.1。

```shell
# purge existing
sudo apt-get purge cmake
sudo snap remove cmake

# install prebuilt binary
export CMAKE_VER=3.23.1
export ARCH=aarch64
wget https://github.com/Kitware/CMake/releases/download/v${CMAKE_VER}/cmake-${CMAKE_VER}-linux-${ARCH}.sh
chmod +x cmake-${CMAKE_VER}-linux-${ARCH}.sh
sudo ./cmake-${CMAKE_VER}-linux-${ARCH}.sh --prefix=/usr --skip-license
cmake --version
```

## 安装依赖项

MMDeploy 中的 Model Converter 依赖于 [MMCV](https://github.com/open-mmlab/mmcv) 和推理引擎 [TensorRT](https://developer.nvidia.com/tensorrt)。

```

## 安装依赖项

同时， MMDeploy 的 C/C++ Inference SDK 依赖于 [spdlog](https://github.com/gabime/spdlog)， OpenCV， [ppl.cv](https://github.com/openppl-public/ppl.cv) 和 TensorRT 等。
因此，接下来我们将先介绍如何配置 TensorRT。
之后再分别展示安装 Model Converter 和 C/C++ Inference SDK 的步骤。

### 配置 TensorRT

JetPack SDK 自带 TensorRT。
但是为了能够在 Conda 环境中成功导入，我们需要将 TensorRT 拷贝进先前创建的 Conda 环境中。

```shell
cp -r /usr/lib/python${PYTHON_VERSION}/dist-packages/tensorrt* ~/archiconda3/envs/mmdeploy/lib/python${PYTHON_VERSION}/site-packages/
conda deactivate
python -c "import tensorrt; print(tensorrt.__version__)" # 将会打印出 TensorRT 版本

# 为之后编译 MMDeploy 设置环境变量
export TENSORRT_DIR=/usr/include/aarch64-linux-gnu

# 将 cuda 路径和 lib 路径写入到环境变量 `$PATH` 和 `$LD_LIBRARY_PATH` 中， 为之后编译 MMDeploy 做准备
export PATH=$PATH:/usr/local/cuda/bin

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64
```

你也可以通过添加以上环境变量至 `~/.bashrc` 使得它们永久化。

```shell
echo -e '\n# set environment variable for TensorRT' >> ~/.bashrc
echo 'export TENSORRT_DIR=/usr/include/aarch64-linux-gnu' >> ~/.bashrc

echo -e '\n# set environment variable for CUDA' >> ~/.bashrc
echo 'export PATH=$PATH:/usr/local/cuda/bin' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64' >> ~/.bashrc

```

### 安装 Model Converter 的依赖项

- 安装 [MMCV](https://github.com/open-mmlab/mmcv)

  MMCV 还未提供针对 Jetson 平台的预编译包，因此我们需要从源对其进行编译。

  ```shell
  sudo apt-get install -y libssl-dev
  git clone --branch 2.x https://github.com/open-mmlab/mmcv.git
  cd mmcv
  MMCV_WITH_OPS=1 pip install -e .
  ```

- 安装 ONNX

```shell
  cd mmcv
  ```

- 安装 ONNX

  不要安装最新的 ONNX，推荐的 ONNX 版本是 1.10.0。

  ```shell
  # 以下方式二选一
  python3 -m pip install onnx==1.10.0
  conda install -c conda-forge onnx
  ```

  如果安装时，出现如下的报错信息:

  ```
  CMake Error at CMakeLists.txt:299 (message):
        Protobuf compiler not found
  ```

  请根据如下指令，安装依赖项:

  ```shell
  sudo apt-get install protobuf-compiler libprotoc-dev
  ```

- 安装 ONNX Runtime [可选]

访问 [Jetson_Zoo#ONNX_Runtime](https://elinux.org/Jetson_Zoo#ONNX_Runtime) 找到对应版本的 ONNX Runtime，然后下载并安装。

示例：

```
# Download pip wheel from location mentioned above
$ wget https://nvidia.box.com/shared/static/jy7nqva7l88mq9i8bw3g3sklzf4kccn2.whl -O onnxruntime_gpu-1.10.0-cp36-cp36m-linux_aarch64.whl
# Install pip wheel
$ pip3 install onnxruntime_gpu-1.10.0-cp36-cp36m-linux_aarch64.whl
```

- 安装 h5py 和 pycuda

```

  Model Converter 使用 HDF5 存储 TensorRT INT8 量化的校准数据；需要 pycuda 拷贝显存

  ```shell
  sudo apt-get install -y pkg-config libhdf5-100 libhdf5-dev
  pip install versioned-hdf5 pycuda
  ```

### 安装 SDK 的依赖项

如果你不需要使用 MMDeploy C/C++ Inference SDK 则可以跳过本步骤。

- 安装 [spdlog](https://github.com/gabime/spdlog)

  “`spdlog` 是一个快速的，仅有头文件的 C++ 日志库。”

  ```shell
  sudo apt-get install -y libspdlog-dev
  ```

- 安装 [ppl.cv](https://github.com/openppl-public/ppl.cv)

  “`ppl.cv` 是 [OpenPPL](https://openppl.ai/home) 的高性能图像处理库。”

  ```shell
  git clone https://github.com/openppl-public/ppl.cv.git
  cd ppl.cv
  export PPLCV_DIR=$(pwd)
  echo -e '\n# set environment variable for ppl.cv' >> ~/.bashrc
  echo "export PPLCV_DIR=$(pwd)" >> ~/.bashrc
  ./build.sh cuda
  ```

## 安装 MMDeploy

```shell
git clone -b main --recursive https://github.com/open-mmlab/mmdeploy.git

echo "export PPLCV_DIR=$(pwd)" >> ~/.bashrc
  ```

```shell
```

### 安装 Model Converter

由于一些算子采用的是 OpenMMLab 代码库中的实现，并不被 TenorRT 支持，
因此我们需要自定义 TensorRT 插件，例如 `roi_align`， `scatternd` 等。
你可以从[这里](../06-custom-ops/tensorrt.md)找到完整的自定义插件列表。

```shell
# 编译 TensorRT 自定义算子
cmake .. -DMMDEPLOY_TARGET_BACKENDS="trt"

# 安装 model converter
# "-v" 表示显示详细安装信息
# "-e" 表示在可编辑模式下安装
# 因此任何针对代码的本地修改都可以在无需重装的情况下生效。
```

### 安装 C/C++ Inference SDK

1. 编译 SDK Libraries 和 Demos

   ```shell
   mkdir -p build && cd build
   cmake .. \
       -DMMDEPLOY_BUILD_SDK=ON \
       -DMMDEPLOY_BUILD_SDK_PYTHON_API=ON \
       -DMMDEPLOY_BUILD_EXAMPLES=ON \
       -DMMDEPLOY_TARGET_DEVICES="cuda;cpu" \

cmake .. \
       -DMMDEPLOY_TARGET_BACKENDS="trt" \
       -DMMDEPLOY_CODEBASES=all \
       -Dpplcv_DIR=${PPLCV_DIR}/cuda-build/install/lib/cmake/ppl
   make -j$(nproc) && make install
   ```

2. 运行 demo

   以目标检测为例:

   ```shell
   ./object_detection cuda ${directory/to/the/converted/models} ${path/to/an/image}
   ```

## Troubleshooting

### 安装

- `pip install` 报错 `Illegal instruction (core dumped)`

  ```shell
  echo '# set env for pip' >> ~/.bashrc
  echo 'export OPENBLAS_CORETYPE=ARMV8' >> ~/.bashrc
  ```

  如果上述方法仍无法解决问题，检查是否正在使用镜像文件。如果是的，可尝试：

  ```shell
  rm .condarc
  conda clean -i
  conda create -n xxx python=${PYTHON_VERSION}
  ```

### 执行

  ```

  ```shell
  ```

### 执行

- `#assertion/root/workspace/mmdeploy/csrc/backend_ops/tensorrt/batched_nms/trt_batched_nms.cpp,98` or `pre_top_k need to be reduced for devices with arch 7.2`

  1. 设置为 `MAX N` 模式并执行 `sudo nvpmodel -m 0 && sudo jetson_clocks`。
  2. 效仿 [mmdet pre_top_k](https://github.com/open-mmlab/mmdeploy/blob/34879e638cc2db511e798a376b9a4b9932660fe1/configs/mmdet/_base_/base_static.py#L13)，减少配置文件中 `pre_top_k` 的个数，例如 `1000`。
  3. 重新进行模型转换并重新运行 demo。

### FAQ

- 错误 `error: cannot import name 'ProcessGroup' from 'torch.distributed'.`
  - 请访问 [pytorch-for-jetson](https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048) 安装 pytorch 1.11 版本。

# PSENet

> [Shape robust text detection with progressive scale expansion network](https://arxiv.org/abs/1903.12473)

Scene text detection has witnessed rapid progress especially with the recent development of convolutional neural networks. However, there still exists two challenges which prevent the algorithm into industry applications. On the one hand, most of the state-of-art algorithms require quadrangle bounding box which is in-accurate to locate the texts with arbitrary shape. On the other hand, two text instances which are close to each other may lead to a false detection which covers both instances. Traditionally, the segmentation-based approach can relieve the first problem but usually fail to solve the second challenge. To address these two challenges, in this paper, we propose a novel Progressive Scale Expansion Network (PSENet), which can precisely detect text instances with arbitrary shapes. More specifically, PSENet generates the different scale of kernels for each text instance, and gradually expands the minimal scale kernel to the text instance with the complete shape. Due to the fact that there are large geometrical margins among the minimal scale kernels, our method is effective to split the close text instances, making it easier to use segmentation-based methods to detect arbitrary-shaped text instances. Extensive experiments on CTW1500, Total-Text, ICDAR 2015 and ICDAR 2017 MLT validate the effectiveness of PSENet. Notably, on CTW1500, a dataset full of long curve texts, PSENet achieves a F-measure of 74.3% at 27 FPS, and our best F-measure (82.2%) outperforms state-of-art algorithms by 6.6%. The code will be released in the future.

<img src="https://user-images.githubusercontent.com/22607038/142795864-9b455b10-8a19-45bb-aeaf-4b733f341afc.png"/>
</div>

|                 Method                  |                 Backbone                  | Pretrained Model | Training set  |   Test set   | #epochs | Test size | Precision | Recall | Hmean  |                  Download                  |

| [PSENet](/configs/textdet/psenet/psenet_resnet50_fpnf_600e_ctw1500.py) |                 ResNet50                  |        -         | CTW1500 Train | CTW1500 Test |   600   |   1280    |  0.7705   | 0.7883 | 0.7793 | [model](https://download.openmmlab.com/mmocr/textdet/psenet/psenet_resnet50_fpnf_600e_ctw1500/psenet_resnet50_fpnf_600e_ctw1500_20220825_221459-7f974ac8.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/psenet/psenet_resnet50_fpnf_600e_ctw1500/20220825_221459.log) |

| [PSENet_r50-oclip](/configs/textdet/psenet/psenet_resnet50-oclip_fpnf_600e_ctw1500.py) | [ResNet50-oCLIP](https://download.openmmlab.com/mmocr/backbone/resnet50-oclip-7ba0c533.pth) |        -         | CTW1500 Train | CTW1500 Test |   600   |   1280    |  0.8483   | 0.7636 | 0.8037 | [model](https://download.openmmlab.com/mmocr/textdet/psenet/psenet_resnet50-oclip_fpnf_600e_ctw1500/psenet_resnet50-oclip_fpnf_600e_ctw1500_20221101_140406-d431710d.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/psenet/psenet_resnet50-oclip_fpnf_600e_ctw1500/20221101_140406.log) |

|                  Method                  |                  Backbone                   | Pretrained Model | Training set | Test set  | #epochs | Test size | Precision | Recall | Hmean  |                  Download                   |

| :--------------------------------------: | :-----------------------------------------: | :--------------: | :----------: | :-------: | :-----: | :-------: | :-------: | :----: | :----: | :-----------------------------------------: |
| [PSENet](/configs/textdet/psenet/psenet_resnet50_fpnf_600e_icdar2015.py) |                  ResNet50                   |        -         |  IC15 Train  | IC15 Test |   600   |   2240    |  0.8396   | 0.7636 | 0.7998 | [model](https://download.openmmlab.com/mmocr/textdet/psenet/psenet_resnet50_fpnf_600e_icdar2015/psenet_resnet50_fpnf_600e_icdar2015_20220825_222709-b6741ec3.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/psenet/psenet_resnet50_fpnf_600e_icdar2015/20220825_222709.log) |

| [PSENet_r50-oclip](/configs/textdet/psenet/psenet_resnet50-oclip_fpnf_600e_icdar2015.py) | [ResNet50-oCLIP](https://download.openmmlab.com/mmocr/backbone/resnet50-oclip-7ba0c533.pth) |        -         |  IC15 Train  | IC15 Test |   600   |   2240    |  0.8895   | 0.8098 | 0.8478 | [model](https://download.openmmlab.com/mmocr/textdet/psenet/psenet_resnet50-oclip_fpnf_600e_icdar2015/psenet_resnet50-oclip_fpnf_600e_icdar2015_20221101_131357-2bdca389.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/psenet/psenet_resnet50-oclip_fpnf_600e_icdar2015/20221101_131357.log) |

```bibtex
@inproceedings{wang2019shape,
  title={Shape robust text detection with progressive scale expansion network},
  author={Wang, Wenhai and Xie, Enze and Li, Xiang and Hou, Wenbo and Lu, Tong and Yu, Gang and Shao, Shuai},
  pages={9336--9345},
}
```

# KITTI Dataset

This page provides specific tutorials about the usage of MMDetection3D for KITTI dataset.

You can download KITTI 3D detection data [HERE](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d) and unzip all zip files. Besides, the road planes could be downloaded from [HERE](https://download.openmmlab.com/mmdetection3d/data/train_planes.zip), which are optional for data augmentation during training for better performance. The road planes are generated by [AVOD](https://github.com/kujason/avod), you can see more details [HERE](https://github.com/kujason/avod/issues/19).

```
├── tools
├── data
│   ├── kitti

```
├── tools
├── data
│   ├── kitti
```

### Create KITTI dataset

To create KITTI point cloud data, we load the raw point cloud data and generate the relevant annotations including object labels and bounding boxes. We also generate all single training objects' point cloud in KITTI dataset and save them as `.bin` files in `data/kitti/kitti_gt_database`. Meanwhile, `.pkl` info files are also generated for training or validation. Subsequently, create KITTI data by running:

```bash

```bash

```

```

Note that if your local disk does not have enough space for saving converted data, you can change the `--out-dir` to anywhere else, and you need to remove the `--with-plane` flag if `planes` are not prepared.

The folder structure after processing should be as below

```
kitti
│   ├── calib
│   ├── calib
```

- `kitti_gt_database/xxxxx.bin`: point cloud data included in each 3D bounding box of the training dataset.

```

- `kitti_infos_train.pkl`: training dataset, a dict contains two keys: `metainfo` and `data_list`.
  - info\['images'\]: Information of images captured by multiple cameras. A dict contains five keys including: `CAM0`, `CAM1`, `CAM2`, `CAM3`, `R0_rect`.
    - info\['images'\]\['R0_rect'\]: Rectifying rotation matrix with shape (4, 4).
    - info\['images'\]\['CAM2'\]: Include some information about the `CAM2` camera sensor.

- info\['images'\]\['R0_rect'\]: Rectifying rotation matrix with shape (4, 4).
      - info\['images'\]\['CAM2'\]\['img_path'\]: The filename of the image.
      - info\['images'\]\['CAM2'\]\['height'\]: The height of the image.
      - info\['images'\]\['CAM2'\]\['width'\]: The width of the image.
      - info\['images'\]\['CAM2'\]\['cam2img'\]: Transformation matrix from camera to image with shape (4, 4).
      - info\['images'\]\['CAM2'\]\['lidar2cam'\]: Transformation matrix from lidar to camera with shape (4, 4).
      - info\['images'\]\['CAM2'\]\['lidar2img'\]: Transformation matrix from lidar to image with shape (4, 4).

- info\['lidar_points'\]\['lidar_path'\]: The filename of the lidar point cloud data.
    - info\['lidar_points'\]\['Tr_velo_to_cam'\]: Transformation from Velodyne coordinate to camera coordinate with shape (4, 4).
    - info\['lidar_points'\]\['Tr_imu_to_velo'\]: Transformation from IMU coordinate to Velodyne coordinate with shape (4, 4).
    - info\['instances'\]\[i\]\['bbox_3d'\]: List of 7 numbers representing the 3D bounding box of the instance, in (x, y, z, l, h, w, yaw) order.
    - info\['instances'\]\[i\]\['bbox_label'\]: An int indicate the 2D label of instance and the -1 indicating ignore.

- info\['instances'\]\[i\]\['bbox_label'\]: An int indicate the 2D label of instance and the -1 indicating ignore.
    - info\['instances'\]\[i\]\['bbox_label_3d'\]: An int indicate the 3D label of instance and the -1 indicating ignore.
    - info\['instances'\]\[i\]\['depth'\]: Projected center depth of the 3D bounding box with respect to the image plane.
    - info\['instances'\]\[i\]\['num_lidar_pts'\]: The number of LiDAR points in the 3D bounding box.
    - info\['instances'\]\[i\]\['center_2d'\]: Projected 2D center of the 3D bounding box.
    - info\['instances'\]\[i\]\['difficulty'\]: KITTI difficulty: 'Easy', 'Moderate', 'Hard'.
    - info\['instances'\]\[i\]\['truncated'\]: Float from 0 (non-truncated) to 1 (truncated), where truncated refers to the object leaving image boundaries.
    - info\['instances'\]\[i\]\['occluded'\]: Integer (0,1,2,3) indicating occlusion state: 0 = fully visible, 1 = partly occluded, 2 = largely occluded, 3 = unknown.

- info\['instances'\]\[i\]\['occluded'\]: Integer (0,1,2,3) indicating occlusion state: 0 = fully visible, 1 = partly occluded, 2 = largely occluded, 3 = unknown.
    - info\['instances'\]\[i\]\['group_ids'\]: Used for multi-part object.
  - info\['plane'\](optional): Road level information.

Please refer to [kitti_converter.py](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/kitti_converter.py) and [update_infos_to_v2.py ](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/update_infos_to_v2.py) for more details.

A typical train pipeline of 3D detection on KITTI is as below:

```python
    dict(
    dict(

load_dim=4, # x, y, z, intensity
    dict(
    dict(
    dict(
]
```

  - `ObjectNoise`: apply noise to each GT objects in the scene.

dict(
]
```

  - `RandomFlip3D`: randomly flip input point cloud horizontally or vertically.
  - `GlobalRotScaleTrans`: rotate input point cloud.

An example to evaluate PointPillars with 8 GPUs with kitti metrics is as follows:

```shell
```

## Metrics

KITTI evaluates 3D object detection performance using mean Average Precision (mAP) and Average Orientation Similarity (AOS), Please refer to its [official website](http://www.cvlibs.net/datasets/kitti/eval_3dobject.php) and [original paper](http://www.cvlibs.net/publications/Geiger2012CVPR.pdf) for more details.

We also adopt this approach for evaluation on KITTI. An example of printed evaluation results is as follows:

```
```

An example to test PointPillars on KITTI with 8 GPUs and generate a submission to the leaderboard is as follows:

- First, you need to modify the `test_dataloader` and `test_evaluator` dict in your config file, just like:

  ```python

load_eval_anns=False,
  ```

- And then, you can run the test script.

  ```shell
  ```

After generating `results/kitti-3class/kitti_results/xxxxx.txt` files, you can submit these files to KITTI benchmark. Please refer to the [KITTI official website](http://www.cvlibs.net/datasets/kitti/index.php) for more details.

* [AbsVal](#absval)
* [ArgMax](#argmax)
* [BatchNorm](#batchnorm)
* [Bias](#bias)
* [BinaryOp](#binaryop)
* [BNLL](#bnll)
* [Cast](#cast)
* [Clip](#clip)
* [Concat](#concat)
* [Convolution](#convolution)
* [Convolution1D](#convolution1d)
* [Convolution3D](#convolution3d)
* [ConvolutionDepthWise](#convolutiondepthwise)
* [ConvolutionDepthWise1D](#convolutiondepthwise1d)
* [ConvolutionDepthWise3D](#convolutiondepthwise3d)
* [CopyTo](#copyto)
* [Crop](#crop)
* [CumulativeSum](#cumulativesum)
* [Deconvolution](#deconvolution)
* [Deconvolution1D](#deconvolution1d)
* [Deconvolution3D](#deconvolution3d)
* [DeconvolutionDepthWise](#deconvolutiondepthwise)
* [DeconvolutionDepthWise1D](#deconvolutiondepthwise1d)
* [DeconvolutionDepthWise3D](#deconvolutiondepthwise3d)
* [DeformableConv2D](#deformableconv2d)
* [Dequantize](#dequantize)
* [Dropout](#dropout)
* [Eltwise](#eltwise)
* [ELU](#elu)
* [Exp](#exp)
* [Flatten](#flatten)
* [GELU](#gelu)
* [GLU](#glu)
* [Gemm](#gemm)

* [GridSample](#gridsample)
* [GroupNorm](#groupnorm)
* [GRU](#gru)
* [HardSigmoid](#hardsigmoid)
* [HardSwish](#hardswish)
* [InnerProduct](#innerproduct)
* [Input](#input)
* [InstanceNorm](#instancenorm)
* [Interp](#interp)
* [LayerNorm](#layernorm)
* [Log](#log)
* [LRN](#lrn)
* [LSTM](#lstm)
* [MemoryData](#memorydata)
* [Mish](#mish)
* [MultiHeadAttention](#multiheadattention)
* [MVN](#mvn)
* [Noop](#noop)
* [Normalize](#normalize)
* [Packing](#packing)
* [Padding](#padding)
* [Permute](#permute)
* [PixelShuffle](#pixelshuffle)
* [Pooling](#pooling)
* [Pooling1D](#pooling1d)
* [Pooling3D](#pooling3d)
* [Power](#power)
* [PReLU](#prelu)
* [Quantize](#quantize)
* [Reduction](#reduction)
* [ReLU](#relu)
* [Reorg](#reorg)
* [Requantize](#requantize)
* [Reshape](#reshape)
* [RNN](#rnn)
* [Scale](#scale)
* [SELU](#selu)

* [ShuffleChannel](#shufflechannel)
* [Sigmoid](#sigmoid)
* [Slice](#slice)
* [Softmax](#softmax)
* [Softplus](#softplus)
* [Split](#split)
* [Swish](#swish)
* [TanH](#tanh)
* [Threshold](#threshold)
* [Tile](#tile)
* [UnaryOp](#unaryop)

# AbsVal
```
y = abs(x)
```

* one_blob_only
* support_inplace

# ArgMax
```
y = argmax(x, out_max_val, topk)
```

| param id  | name          | type  | default   | description       |
| --------- | ------------- | ----- | --------- | ----------------- |
| 0         | out_max_val   | int   | 0         |                   |
| 1         | topk          | int   | 1         |                   |

# BatchNorm
```
y = (x - mean) / sqrt(var + eps) * slope + bias
```

```
```

| 0         | channels      | int   | 0         |                   |
| 1         | eps           | float | 0.f       |                   |

| weight        | type  | shape                 |
| ------------- | ----- | --------------------- |
| slope_data    | float | [channels]            |
| mean_data     | float | [channels]            |
| var_data      | float | [channels]            |
| bias_data     | float | [channels]            |

# Bias
```
y = x + bias
```

```
y = x + bias
```

| 0         | bias_data_size| int   | 0         |                   |

# BinaryOp
 This operation is used for binary computation, and the calculation rule depends on the [broadcasting rule](https://github.com/Tencent/ncnn/wiki/binaryop-broadcasting).
```
C = binaryop(A, B)
```
if with_scalar = 1:
- one_blob_only
- support_inplace

| 0         | op_type       | int   | 0         | Operation type as follows |

| 1         | with_scalar   | int   | 0         | with_scalar=0 B is a matrix, with_scalar=1 B is a scalar |
| 2         | b             | float | 0.f       | When B is a scalar, B = b |

Operation type:
- 0 = ADD
- 1 = SUB
- 2 = MUL
- 3 = DIV
- 4 = MAX
- 5 = MIN
- 6 = POW
- 7 = RSUB
- 8 = RDIV
- 9 = RPOW
- 10 = ATAN2
- 11 = RATAN2

# BNLL
```
y = log(1 + e^(-x)) , x > 0
y = log(1 + e^x),     x < 0
```

# Cast
```
y = cast(x)
```

* support_packing

| 0         | type_from     | int   | 0         |                   |
| 1         | type_to       | int   | 0         |                   |

Element type:
- 0 = auto
- 1 = float32
- 2 = float16
- 3 = int8

- 0 = auto
- 3 = int8
- 4 = bfloat16

# Clip
```
y = clamp(x, min, max)
```

| 0         | min           | float | -FLT_MAX  |                   |
| 1         | max           | float | FLT_MAX   |                   |

# Concat
```
y = concat(x0, x1, x2, ...) by axis
```

| 0         | axis          | int   | 0         |                   |

# Convolution
```
x2 = pad(x, pads, pad_value)
x3 = conv(x2, weight, kernel, stride, dilation) + bias
y = activation(x3, act_type, act_params)
```

```
```

| 0         | num_output    | int   | 0         |                   |
| 1         | kernel_w      | int   | 0         |                   |
| 2         | dilation_w    | int   | 1         |                   |
| 3         | stride_w      | int   | 1         |                   |
| 4         | pad_left      | int   | 0         |                   |
| 5         | bias_term     | int   | 0         |                   |
| 6         | weight_data_size| int | 0         |                   |
| 8         | int8_scale_term| int  | 0         |                   |
| 9         | activation_type| int  | 0         |                   |

| 10        | activation_params| array | [ ]    |                   |
| 11        | kernel_h      | int   | kernel_w  |                   |
| 12        | dilation_h    | int   | dilation_w |                  |
| 13        | stride_h      | int   | stride_w  |                   |
| 14        | pad_top       | int   | pad_left  |                   |
| 15        | pad_right     | int   | pad_left  |                   |
| 16        | pad_bottom    | int   | pad_top   |                   |
| 18        | pad_value     | float | 0.f       |                   |
| 19        | dynamic_weight| int   | 0         |                   |

| weight_data   | float/fp16/int8 | [kernel_w, kernel_h, num_input, num_output] |

| bias_data     | float | [num_output]          |
| weight_data_int8_scales| float | [num_output] |
| bottom_blob_int8_scales| float | [1]          |
| top_blob_int8_scales| float | [1]             |

# Convolution1D
```
x3 = conv1d(x2, weight, kernel, stride, dilation) + bias
```

| weight_data   | float/fp16/int8 | [kernel_w, num_input, num_output] |

# Convolution3D
```

```
x3 = conv3d(x2, weight, kernel, stride, dilation) + bias
```

| 17        | pad_behind    | int   | pad_front |                   |
| 21        | kernel_d      | int   | kernel_w  |                   |
| 22        | dilation_d    | int   | dilation_w |                  |
| 23        | stride_d      | int   | stride_w  |                   |

| 24        | pad_front     | int   | pad_left  |                   |

| weight_data   | float/fp16/int8 | [kernel_w, kernel_h, kernel_d, num_input, num_output] |

# ConvolutionDepthWise
```
x3 = conv(x2, weight, kernel, stride, dilation, group) + bias
```

| 7         | group         | int   | 1         |                   |

| weight_data   | float/fp16/int8 | [kernel_w, kernel_h, num_input / group, num_output / group, group] |
| weight_data_int8_scales| float | [group]      |

# ConvolutionDepthWise1D
```
x3 = conv1d(x2, weight, kernel, stride, dilation, group) + bias
```

```
```

| weight_data   | float/fp16/int8 | [kernel_w, num_input / group, num_output / group, group] |

# ConvolutionDepthWise3D
```
x3 = conv3d(x2, weight, kernel, stride, dilation, group) + bias
```

```

| weight_data   | float/fp16/int8 | [kernel_w, kernel_h, kernel_d, num_input / group, num_output / group, group] |

# CopyTo
```
self[offset] = src
```

| 0         | woffset       | int   | 0         |                   |
| 1         | hoffset       | int   | 0         |                   |
| 13        | doffset       | int   | 0         |                   |
| 2         | coffset       | int   | 0         |                   |
| 9         | starts        | array | [ ]       |                   |

| 11        | axes          | array | [ ]       |                   |

# Crop
```
y = crop(x)
```

| 2         | coffset       | int   | 1         |                   |
| 3         | outw          | int   | 1         |                   |
| 4         | outh          | int   | 0         |                   |
| 5         | outc          | int   | 0         |                   |
| 6         | woffset2      | int   | 0         |                   |
| 7         | hoffset2      | int   | 1         |                   |

| 8         | coffset2      | int   | 0         |                   |
| 10        | ends          | array | [ ]       |                   |

# CumulativeSum

If axis < 0, we use axis = x.dims + axis

It implements https://pytorch.org/docs/stable/generated/torch.cumsum.html

# Deconvolution
```
x2 = deconv(x, weight, kernel, stride, dilation) + bias
x3 = depad(x2, pads, pad_value)
```

```
```

| 18        | output_pad_right| int | 0         |                   |
| 19        | output_pad_bottom| int | output_pad_right |           |
| 20        | output_w      | int   | 0         |                   |
| 21        | output_h      | int   | output_w  |                   |

| weight_data   | float/fp16 | [kernel_w, kernel_h, num_input, num_output] |

# Deconvolution1D
```
x2 = deconv1d(x, weight, kernel, stride, dilation) + bias
```

| weight_data   | float/fp16 | [kernel_w, num_input, num_output] |

# Deconvolution3D
```
x2 = deconv3d(x, weight, kernel, stride, dilation) + bias

```
```

| 20        | output_pad_behind| int | output_pad_right |           |

| 25        | output_w      | int   | 0         |                   |
| 26        | output_h      | int   | output_w  |                   |
| 27        | output_d      | int   | output_w  |                   |

| weight_data   | float/fp16 | [kernel_w, kernel_h, kernel_d, num_input, num_output] |

# DeconvolutionDepthWise
```
x2 = deconv(x, weight, kernel, stride, dilation, group) + bias
```

```

| weight_data   | float/fp16 | [kernel_w, kernel_h, num_input / group, num_output / group, group] |

# DeconvolutionDepthWise1D
```
x2 = deconv1d(x, weight, kernel, stride, dilation, group) + bias
```

| weight_data   | float/fp16 | [kernel_w, num_input / group, num_output / group, group] |

# DeconvolutionDepthWise3D
```
x2 = deconv3d(x, weight, kernel, stride, dilation, group) + bias
```

| weight_data   | float/fp16 | [kernel_w, kernel_h, kernel_d, num_input / group, num_output / group, group] |

# DeformableConv2D
```
x2 = deformableconv2d(x, offset, mask, weight, kernel, stride, dilation) + bias
y = activation(x2, act_type, act_params)
```

# Dequantize
```
y = x * scale + bias
```

```
```

| 0         | scale_data_size| int  | 1         |                   |
| 1         | bias_data_size| int   | 0         |                   |

| scale_data    | float | [scale_data_size]     |
| bias_data     | float | [bias_data_size]      |

# Dropout
```
y = x * scale
```

| 0         | scale         | float | 1.f       |                   |

# Eltwise
```
y = elementwise_op(x0, x1, ...)

# Eltwise
```
```

| 0         | op_type       | int   | 0         |                   |
| 1         | coeffs        | array | [ ]       |                   |

- 0 = PROD
- 1 = SUM
- 2 = MAX

# ELU
```
if x < 0    y = (exp(x) - 1) * alpha
else        y = x
```

| 0         | alpha         | float | 0.1f      |                   |

# Exp
```
if base == -1   y = exp(shift + x * scale)
else            y = pow(base, (shift + x * scale))
```

# Exp
```
```

| 0         | base          | float | -1.f      |                   |
| 1         | scale         | float | 1.f       |                   |
| 2         | shift         | float | 0.f       |                   |

# Flatten
Reshape blob to 1 dimension

# GELU
```
if fast_gelu == 1   y = 0.5 * x * (1 + tanh(0.79788452 * (x + 0.044715 * x * x * x)));
else                y = 0.5 * x * erfc(-0.70710678 * x)
```

| 0         | fast_gelu     | int   | 0         | use approximation |

# GLU

# GLU

GLU(a,b)=a⊗σ(b)

where a is the first half of the input matrix and b is the second half.

axis specifies the dimension to split the input

# Gemm
```
a = transA ? transpose(x0) : x0
b = transb ? transpose(x1) : x1
c = x2
y = (gemm(a, b) + c * beta) * alpha
```

| 0         | alpha         | float | 1.f       |                   |
| 1         | beta          | float | 1.f       |                   |

| 2         | transA        | int   | 0         |                   |
| 3         | transb        | int   | 0         |                   |
| 4         | constantA     | int   | 0         |                   |
| 5         | constantB     | int   | 0         |                   |
| 6         | constantC     | int   | 0         |                   |
| 7         | constantM     | int   | 0         |                   |
| 8         | constantN     | int   | 0         |                   |
| 9         | constantK     | int   | 0         |                   |
| 10        | constant_broadcast_type_C | int | 0 |                 |
| 11        | output_N1M    | int   | 0         |                   |
| 12        | output_elempack | int | 0         |                   |
| 13        | output_elemtype | int | 0         |                   |

| 14        | output_transpose | int| 0         |                   |
| 20        | constant_TILE_M | int | 0         |                   |
| 21        | constant_TILE_N | int | 0         |                   |
| 22        | constant_TILE_K | int | 0         |                   |

| A_data        | float | [M, K] or [K, M]      |
| B_data        | float | [N, K] or [K, N]      |
| C_data        | float | [1], [M] or [N] or [1, M] or [N,1] or [N, M] |

# GridSample
```
Given an input and a flow-field grid, computes the output using input values and pixel locations from grid.

For each output location output[:, h2, w2], the size-2 vector grid[h2, w2, 2] specifies input pixel[:, h1, w1] locations x and y,

For each output location output[:, h2, w2], the size-2 vector grid[h2, w2, 2] specifies input pixel[:, h1, w1] locations x and y, 
which are used to interpolate the output value output[:, h2, w2]

This function is often used in conjunction with affine_grid() to build Spatial Transformer Networks .
```

| 0         | sample_type   | int   | 1         |                   |
| 1         | padding_mode  | int   | 1         |                   |
| 2         | align_corner  | int   | 0         |                   |

Sample type:
- 1 = Nearest
- 2 = Bilinear
- 3 = Bicubic

Padding mode:
- 1 = zeros
- 2 = border
- 3 = reflection

# GroupNorm
```
split x along channel axis into group x0, x1 ...
l2 normalize for each group x0, x1 ...
y = x * gamma + beta
```

- 2 = border

```
```

| 0         | group         | int   | 1         |                   |
| 1         | channels      | int   | 0         |                   |
| 2         | eps           | float | 0.001f    | x = x / sqrt(var + eps) |
| 3         | affine        | int   | 1         |                   |

| gamma_data    | float | [channels]            |
| beta_data     | float | [channels]            |

# GRU
Apply a single-layer GRU to a feature sequence of `T` timesteps. The input blob shape is `[w=input_size, h=T]` and the output blob shape is `[w=num_output, h=T]`.

```

# GRU

```
y = gru(x)
y0, hidden y1 = gru(x0, hidden x1)
```

* one_blob_only if bidirectional

| 0         | num_output    | int   | 0         | hidden size of output |
| 1         | weight_data_size| int | 0         | total size of weight matrix |
| 2         | direction     | int   | 0         | 0=forward, 1=reverse, 2=bidirectional |

| weight_xc_data| float/fp16/int8 | [input_size, num_output * 3, num_directions] |
| bias_c_data   | float/fp16/int8 | [num_output, 4, num_directions] |
| weight_hc_data| float/fp16/int8 | [num_output, num_output * 3, num_directions] |

Direction flag:

- 0 = forward only
- 1 = reverse only
- 2 = bidirectional

# HardSigmoid
```
y = clamp(x * alpha + beta, 0, 1)
```

| 0         | alpha         | float | 0.2f      |                   |
| 1         | beta          | float | 0.5f      |                   |

# HardSwish
```
y = x * clamp(x * alpha + beta, 0, 1)
```

# InnerProduct
```

```
x2 = innerproduct(x, weight) + bias
```

| 1         | bias_term     | int   | 0         |                   |
| 2         | weight_data_size| int | 0         |                   |

| weight_data   | float/fp16/int8 | [num_input, num_output] |

# Input
```
y = input
```

| 0         | w             | int   | 0         |                   |
| 1         | h             | int   | 0         |                   |
| 11        | d             | int   | 0         |                   |
| 2         | c             | int   | 0         |                   |

# InstanceNorm
```
split x along channel axis into instance x0, x1 ...
l2 normalize for each channel instance x0, x1 ...
```

```
```

| 1         | eps           | float | 0.001f    | x = x / sqrt(var + eps) |
| 2         | affine        | int   | 1         |                   |

# Interp
```
if dynamic_target_size == 0     y = resize(x) by fixed size or scale
else                            y = resize(x0, size(x1))
```

* one_blob_only if dynamic_target_size == 0

```

| 0         | resize_type   | int   | 0         |                   |
| 1         | height_scale  | float | 1.f       |                   |
| 2         | width_scale   | float | 1.f       |                   |
| 3         | output_height | int   | 0         |                   |
| 4         | output_width  | int   | 0         |                   |
| 5         | dynamic_target_size| int | 0      |                   |
| 6         | align_corner  | int   | 0         |                   |

Resize type:

# LayerNorm
```
split x along outmost axis into part x0, x1 ...
l2 normalize for each part x0, x1 ...
y = x * gamma + beta by elementwise
```

```
```

| 0         | affine_size   | int   | 0         |                   |

| gamma_data    | float | [affine_size]         |
| beta_data     | float | [affine_size]         |

# Log
```
if base == -1   y = log(shift + x * scale)
else            y = log(shift + x * scale) / log(base)
```

```

# LRN
```
if region_type == ACROSS_CHANNELS   square_sum = sum of channel window of local_size
if region_type == WITHIN_CHANNEL    square_sum = sum of spatial window of local_size
y = x * pow(bias + alpha * square_sum / (local_size * local_size), -beta)
```

| 0         | region_type   | int   | 0         |                   |

| 1         | local_size    | int   | 5         |                   |
| 2         | alpha         | float | 1.f       |                   |
| 3         | beta          | float | 0.75f     |                   |
| 4         | bias          | float | 1.f       |                   |

Region type:
- 0 = ACROSS_CHANNELS
- 1 = WITHIN_CHANNEL

# LSTM
Apply a single-layer LSTM to a feature sequence of `T` timesteps. The input blob shape is `[w=input_size, h=T]` and the output blob shape is `[w=num_output, h=T]`.

```
y = lstm(x)
y0, hidden y1, cell y2 = lstm(x0, hidden x1, cell x2)
```

| 0         | num_output    | int   | 0         | output size of output |

| 1         | weight_data_size| int | 0         | total size of IFOG weight matrix |
| 3         | hidden_size   | int   | num_output| hidden size       |

| weight_xc_data| float/fp16/int8 | [input_size, hidden_size * 4, num_directions] |
| bias_c_data   | float/fp16/int8 | [hidden_size, 4, num_directions] |
| weight_hc_data| float/fp16/int8 | [num_output, hidden_size * 4, num_directions] |
| weight_hr_data| float/fp16/int8 | [hidden_size, num_output, num_directions] |

# MemoryData
```
y = data
```

```
y = data
```

| data          | float | [w, h, d, c]          |

# Mish
```
y = x * tanh(log(exp(x) + 1))
```

# MultiHeadAttention
```
split q k v into num_head part q0, k0, v0, q1, k1, v1 ...
for each num_head part
    xq = affine(q) / (embed_dim / num_head)
    xk = affine(k)
    xv = affine(v)
    xqk = xq * xk

```
    xqk = xq * xk
    xqk = xqk + attn_mask if attn_mask exists
    softmax_inplace(xqk)
    xqkv = xqk * xv
    merge xqkv to out
y = affine(out)
```

| 0         | embed_dim     | int   | 0         |                   |
| 1         | num_heads     | int   | 1         |                   |
| 3         | kdim          | int   | embed_dim |                   |
| 4         | vdim          | int   | embed_dim |                   |
| 5         | attn_mask     | int   | 0         |                   |

| q_weight_data | float/fp16/int8 | [weight_data_size] |
| q_bias_data   | float | [embed_dim]           |
| k_weight_data | float/fp16/int8 | [embed_dim * kdim] |
| k_bias_data   | float | [embed_dim]           |
| v_weight_data | float/fp16/int8 | [embed_dim * vdim] |
| v_bias_data   | float | [embed_dim]           |
| out_weight_data| float/fp16/int8 | [weight_data_size] |
| out_bias_data | float | [embed_dim]           |

# MVN
```
if normalize_variance == 1 && across_channels == 1      y = (x - mean) / (sqrt(var) + eps) of whole blob
if normalize_variance == 1 && across_channels == 0      y = (x - mean) / (sqrt(var) + eps) of each channel
if normalize_variance == 0 && across_channels == 1      y = x - mean of whole blob
if normalize_variance == 0 && across_channels == 0      y = x - mean of each channel
```

```

| 0         | normalize_variance| int | 0       |                   |
| 1         | across_channels| int  | 0         |                   |
| 2         | eps           | float | 0.0001f   | x = x / (sqrt(var) + eps) |

# Noop
```
y = x
```

# Normalize
```
if across_spatial == 1 && across_channel == 1      x2 = normalize(x) of whole blob
if across_spatial == 1 && across_channel == 0      x2 = normalize(x) of each channel
if across_spatial == 0 && across_channel == 1      x2 = normalize(x) of each position
y = x2 * scale
```

```

| 0         | across_spatial| int   | 0         |                   |
| 1         | channel_shared| int   | 0         |                   |
| 2         | eps           | float | 0.0001f   | see eps mode      |
| 3         | scale_data_size| int  | 0         |                   |
| 4         | across_channel| int   | 0         |                   |
| 9         | eps_mode      | int   | 0         |                   |

Eps Mode:
- 0 = caffe/mxnet   x = x / sqrt(var + eps)
- 1 = pytorch       x = x / max(sqrt(var), eps)
- 2 = tensorflow    x = x / sqrt(max(var, eps))

# Packing
```
y = wrap_packing(x)
```

# Packing
```
```

| 0         | out_elempack  | int   | 1         |                   |
| 1         | use_padding   | int   | 0         |                   |
| 2         | cast_type_from| int   | 0         |                   |
| 3         | cast_type_to  | int   | 0         |                   |
| 4         | storage_type_from| int | 0        |                   |
| 5         | storage_type_to| int  | 0         |                   |

# Padding
```
y = pad(x, pads)
```

| param id  | name          | type | default   | description       |
| --------- | ------------- | ---- | --------- | ----------------- |
| 0         | top           | int  | 0         |                   |

| 1         | bottom        | int  | 0         |                   |
| 2         | left          | int  | 0         |                   |
| 3         | right         | int  | 0         |                   |
| 4         | type          | int  | 0         |                   |
| 5         | value         | float | 0         |                   |
| 6         | per_channel_pad_data_size| int | 0 |                 |
| 7         | front         | int  | stride_w  |                   |
| 8         | behind        | int  | pad_left  |                   |

| per_channel_pad_data| float | [per_channel_pad_data_size] |

Padding type:
- 0 = CONSTANT
- 1 = REPLICATE
- 2 = REFLECT

# Permute
```
y = reorder(x)
```

# Permute
```
```

| 0         | order_type    | int  | 0         |                   |

Order Type:
- 0 = WH WHC WHDC
- 1 = HW HWC HWDC
- 2 = WCH WDHC
- 3 = CWH DWHC
- 4 = HCW HDWC
- 5 = CHW DHWC
- 6 = WHCD
- 7 = HWCD
- 8 = WCHD
- 9 = CWHD
- 10 = HCWD
- 11 = CHWD
- 12 = WDCH
- 13 = DWCH
- 14 = WCDH
- 15 = CWDH
- 16 = DCWH
- 17 = CDWH
- 18 = HDCW
- 19 = DHCW
- 20 = HCDW
- 21 = CHDW
- 22 = DCHW
- 23 = CDHW

# PixelShuffle
```
if mode == 0    y = depth_to_space(x) where x channel order is sw-sh-outc
if mode == 1    y = depth_to_space(x) where x channel order is outc-sw-sh
```

```

| 0         | upscale_factor| int  | 1         |                   |
| 1         | mode          | int  | 0         |                   |

# Pooling
```
x2 = pad(x, pads)
x3 = pooling(x2, kernel, stride)
```

| --------- | --------------| ---- | --------- | ----------------- |
| 0         | pooling_type  | int  | 0         |                   |
| 1         | kernel_w      | int  | 0         |                   |
| 2         | stride_w      | int  | 1         |                   |
| 3         | pad_left      | int  | 0         |                   |
| 4         | global_pooling| int  | 0         |                   |
| 5         | pad_mode      | int  | 0         |                   |

| 6         | avgpool_count_include_pad| int | 0 |                 |
| 7         | adaptive_pooling| int | 0        |                   |
| 8         | out_w         | int  | 0         |                   |
| 11        | kernel_h      | int  | kernel_w  |                   |
| 12        | stride_h      | int  | stride_w  |                   |
| 13        | pad_top       | int  | pad_left  |                   |
| 14        | pad_right     | int  | pad_left  |                   |
| 15        | pad_bottom    | int  | pad_top   |                   |
| 18        | out_h         | int  | out_w     |                   |

Pooling type:
- 0 = MAX
- 1 = AVG

Pad mode:
- 0 = full padding
- 1 = valid padding
- 2 = tensorflow padding=SAME or onnx padding=SAME_UPPER
- 3 = onnx padding=SAME_LOWER

# Pooling1D
```
x3 = pooling1d(x2, kernel, stride)

```
```

- 0 = MAX
- 1 = AVG

Pad mode:

# Pooling3D
```
x3 = pooling3d(x2, kernel, stride)
```

| 16        | pad_behind    | int  | pad_front |                   |
| 21        | kernel_d      | int  | kernel_w  |                   |

| 22        | stride_d      | int  | stride_w  |                   |
| 23        | pad_front     | int  | pad_left  |                   |
| 28        | out_d         | int  | out_w     |                   |

- 0 = MAX
- 1 = AVG

Pad mode:

# Power
```
y = pow((shift + x * scale), power)
```

| 0         | power         | float | 1.f       |                   |

# PReLU
```
if x < 0    y = x * slope

# PReLU
```
else        y = x
```

| 0         | num_slope     | int   | 0         |                   |

| slope_data    | float | [num_slope]           |

# Quantize
```
y = float2int8(x * scale)
```

# Reduction
```
y = reduce_op(x * coeff)
```

| 0         | operation     | int   | 0         |                   |
| 1         | reduce_all    | int   | 1         |                   |
| 2         | coeff         | float | 1.f       |                   |
| 3         | axes          | array | [ ]       |                   |
| 4         | keepdims      | int   | 0         |                   |

- 0 = SUM
- 1 = ASUM
- 2 = SUMSQ
- 3 = MEAN
- 4 = MAX
- 5 = MIN
- 6 = PROD
- 7 = L1
- 8 = L2
- 9 = LogSum
- 10 = LogSumExp

# ReLU
```
else        y = x
```

- 0 = SUM
- 1 = ASUM
- 2 = SUMSQ
- 3 = MEAN
- 4 = MAX
- 5 = MIN
- 6 = PROD
- 7 = L1
- 8 = L2
- 9 = LogSum

# ReLU
```
else        y = x
```

| 0         | slope         | float | 0.f       |                   |

# Reorg
```
if mode == 0    y = space_to_depth(x) where x channel order is sw-sh-outc
if mode == 1    y = space_to_depth(x) where x channel order is outc-sw-sh
```

| 0         | stride        | int  | 1         |                   |

# Requantize
```
x2 = x * scale_in + bias
x3 = activation(x2)
y = float2int8(x3 * scale_out)
```

```
```

| 0         | scale_in_data_size| int | 1       |                   |
| 1         | scale_out_data_size| int | 1      |                   |
| 2         | bias_data_size| int   | 0         |                   |
| 3         | activation_type| int  | 0         |                   |
| 4         | activation_params| int | [ ]      |                   |

| scale_in_data | float | [scale_in_data_size]  |
| scale_out_data| float | [scale_out_data_size] |

# Reshape
```
if permute == 1     y = hwc2chw(reshape(chw2hwc(x)))

# Reshape
```
else                y = reshape(x)
```

| 0         | w             | int   | -233      |                   |
| 1         | h             | int   | -233      |                   |
| 11        | d             | int   | -233      |                   |
| 2         | c             | int   | -233      |                   |
| 3         | permute       | int   | 0         |                   |

Reshape flag:
- 0 = copy from bottom
- -1 = remaining
- -233 = drop this dim(default)

# RNN
Apply a single-layer RNN to a feature sequence of `T` timesteps. The input blob shape is `[w=input_size, h=T]` and the output blob shape is `[w=num_output, h=T]`.

```
y = rnn(x)

# RNN

```
y = rnn(x)
y0, hidden y1 = rnn(x0, hidden x1)
```

| weight_xc_data| float/fp16/int8 | [input_size, num_output, num_directions] |
| bias_c_data   | float/fp16/int8 | [num_output, 1, num_directions] |
| weight_hc_data| float/fp16/int8 | [num_output, num_output, num_directions] |

# Scale
```
if scale_data_size == -233  y = x0 * x1
else                        y = x * scale + bias
```

* one_blob_only if scale_data_size != -233

| 0         | scale_data_size| int  | 0         |                   |

| bias_data     | float | [scale_data_size]     |

# SELU
```
if x < 0    y = (exp(x) - 1.f) * alpha * lambda
else        y = x * lambda
```

# SELU
```
```

| 0         | alpha         | float | 1.67326324f|                  |
| 1         | lambda        | float | 1.050700987f|                 |

# ShuffleChannel
```
if reverse == 0     y = shufflechannel(x) by group
if reverse == 1     y = shufflechannel(x) by channel / group
```

| 0         | group         | int  | 1         |                   |
| 1         | reverse       | int  | 0         |                   |

# Sigmoid
```
y = 1 / (1 + exp(-x))
```

# Slice
```

# Sigmoid
```
```

# Slice
```
split x along axis into slices, each part slice size is based on slices array
```

| 0         | slices        | array | [ ]       |                   |
| 1         | axis          | int   | 0         |                   |

# Softmax
```
softmax(x, axis)
```

| 1         | fixbug0       | int   | 0         | hack for bug fix, should be 1 |

# Softplus
```
y = log(exp(x) + 1)
```

# Split
```
y0, y1 ... = x
```

# Swish

# Softplus
```
```

# Split
```
```

# Swish
```
y = x / (1 + exp(-x))
```

# TanH
```
y = tanh(x)
```

# Threshold
```
if x > threshold    y = 1
else                y = 0
```

| 0         | threshold     | float | 0.f       |                   |

# Tile
```
y = repeat tiles along axis for x
```

| 1         | tiles         | int   | 1         |                   |

| 2         | repeats       | array | [ ]       |                   |

# UnaryOp
```
y = unaryop(x)
```

- 0 = ABS
- 1 = NEG
- 2 = FLOOR
- 3 = CEIL
- 4 = SQUARE
- 5 = SQRT
- 6 = RSQ
- 7 = EXP
- 8 = LOG
- 9 = SIN
- 10 = COS
- 11 = TAN
- 12 = ASIN
- 13 = ACOS
- 14 = ATAN
- 15 = RECIPROCAL
- 16 = TANH
- 17 = LOG10
- 18 = ROUND
- 19 = TRUNC

# 使用 Docker 镜像

本文简述如何使用[Docker](https://docs.docker.com/get-docker/)安装mmdeploy

## 获取镜像

为了方便用户，mmdeploy在[Docker Hub](https://hub.docker.com/r/openmmlab/mmdeploy)上提供了多个版本的镜像，例如对于`mmdeploy==1.2.0`，
其镜像标签为`openmmlab/mmdeploy:ubuntu20.04-cuda11.3-mmdeploy1.2.0`，而最新的镜像标签为`openmmlab/mmdeploy:ubuntu20.04-cuda11.3-mmdeploy`。
镜像相关规格信息如下表所示：

|    Item     |   Version   |
| :---------: | :---------: |
|     OS      | Ubuntu20.04 |
|    CUDA     |    11.3     |
|    CUDNN    |     8.2     |
|   Python    |   3.8.10    |
|    Torch    |   1.10.0    |
| TorchVision |   0.11.0    |
| TorchScript |   1.10.0    |
|  TensorRT   |   8.2.3.0   |
| ONNXRuntime |    1.8.1    |
|  OpenVINO   |  2022.3.0   |
|    ncnn     |  20221128   |
|   openppl   |    0.8.1    |

用户可选择一个[镜像](https://hub.docker.com/r/openmmlab/mmdeploy/tags)并运行`docker pull`拉取镜像到本地：

```shell
export TAG=openmmlab/mmdeploy:ubuntu20.04-cuda11.3-mmdeploy
docker pull $TAG
```

## 构建镜像(可选)

```shell
```

如果已提供的镜像无法满足要求，用户可修改`docker/Release/Dockerfile`并在本地构建镜像。其中，构建参数`MMDEPLOY_VERSION`可以是[mmdeploy](https://github.com/open-mmlab/mmdeploy)项目的一个[标签](https://github.com/open-mmlab/mmdeploy/tags)或者分支。

```shell
export MMDEPLOY_VERSION=main
export TAG=mmdeploy-${MMDEPLOY_VERSION}
docker build docker/Release/ -t ${TAG} --build-arg MMDEPLOY_VERSION=${MMDEPLOY_VERSION}
```

## 运行 docker 容器

当拉取或构建 docker 镜像后，用户可使用 `docker run` 启动 docker 服务：

```shell
docker run --gpus=all -it --rm $TAG
```

## 常见问答

1. CUDA error: the provided PTX was compiled with an unsupported toolchain:

   如 [这里](https://forums.developer.nvidia.com/t/cuda-error-the-provided-ptx-was-compiled-with-an-unsupported-toolchain/185754)所说，更新 GPU 的驱动到您的GPU能使用的最新版本。

如 [这里](https://forums.developer.nvidia.com/t/cuda-error-the-provided-ptx-was-compiled-with-an-unsupported-toolchain/185754)所说，更新 GPU 的驱动到您的GPU能使用的最新版本。

2. docker: Error response from daemon: could not select device driver "" with capabilities: [gpu].

   ```shell
   # Add the package repositories
   distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
   curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
   curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

   sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
   sudo systemctl restart docker
   ```

# Core ML feature support

MMDeploy support convert Pytorch model to Core ML and inference.

To convert the model in mmdet, you need to compile libtorch to support custom operators such as nms.

```bash
cmake .. \
    -DDISABLE_SVE=ON # low version like 1.8.0 of pytorch need this option
```

## Usage

```bash
```

# SlowOnly

[Slowfast networks for video recognition](https://openaccess.thecvf.com/content_ICCV_2019/html/Feichtenhofer_SlowFast_Networks_for_Video_Recognition_ICCV_2019_paper.html)

</div>

### AVA2.1

| frame sampling strategy | gpus |                backbone                |   pretrain   |  mAP  |                  config                   |                  ckpt                   |                  log                   |
| :---------------------: | :--: | :------------------------------------: | :----------: | :---: | :---------------------------------------: | :-------------------------------------: | :------------------------------------: |

|         4x16x1          |  8   |           SlowOnly ResNet50            | Kinetics-400 | 20.72 | [config](/configs/detection/slowonly/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb_20220906-953ef5fe.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb.log) |

|         4x16x1          |  8   |           SlowOnly ResNet50            | Kinetics-700 | 22.77 | [config](/configs/detection/slowonly/slowonly_kinetics700-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics700-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb/slowonly_kinetics700-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb_20220906-b3b6d44e.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics700-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb/slowonly_kinetics700-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb.log) |

|         4x16x1          |  8   | SlowOnly ResNet50 (NonLocalEmbedGauss) | Kinetics-400 | 21.55 | [config](/configs/detection/slowonly/slowonly_kinetics400-pretrained-r50-nl_8xb16-4x16x1-20e_ava21-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r50-nl_8xb16-4x16x1-20e_ava21-rgb/slowonly_kinetics400-pretrained-r50-nl_8xb16-4x16x1-20e_ava21-rgb_20220906-5ae3f91b.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r50-nl_8xb16-4x16x1-20e_ava21-rgb/slowonly_kinetics400-pretrained-r50-nl_8xb16-4x16x1-20e_ava21-rgb.log) |

|          8x8x1          |  8   | SlowOnly ResNet50 (NonLocalEmbedGauss) | Kinetics-400 | 23.77 | [config](/configs/detection/slowonly/slowonly_kinetics400-pretrained-r50-nl_8xb16-8x8x1-20e_ava21-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r50-nl_8xb16-8x8x1-20e_ava21-rgb/slowonly_kinetics400-pretrained-r50-nl_8xb16-8x8x1-20e_ava21-rgb_20220906-9760eadb.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r50-nl_8xb16-8x8x1-20e_ava21-rgb/slowonly_kinetics400-pretrained-r50-nl_8xb16-8x8x1-20e_ava21-rgb.log) |

|          8x8x1          |  8   |           SlowOnly ResNet101           | Kinetics-400 | 24.83 | [config](/configs/detection/slowonly/slowonly_kinetics400-pretrained-r101_8xb16-8x8x1-20e_ava21-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r101_8xb16-8x8x1-20e_ava21-rgb/slowonly_kinetics400-pretrained-r101_8xb16-8x8x1-20e_ava21-rgb_20220906-43f16877.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r101_8xb16-8x8x1-20e_ava21-rgb/slowonly_kinetics400-pretrained-r101_8xb16-8x8x1-20e_ava21-rgb.log) |

### AVA2.2 (Trained on AVA-Kinetics)

Currently, we only use the training set of AVA-Kinetics and evaluate on the AVA2.2 validation dataset. The AVA-Kinetics validation dataset will be supported soon.

| frame sampling strategy | gpus |     backbone      |   pretrain   |  mAP  |                      config                      |                      ckpt                      |                      log                      |
| :---------------------: | :--: | :---------------: | :----------: | :---: | :----------------------------------------------: | :--------------------------------------------: | :-------------------------------------------: |

|         4x16x1          |  8   | SlowOnly ResNet50 | Kinetics-400 | 24.53 | [config](/configs/detection/slowonly/slowonly_k400-pre-r50_8xb8-4x16x1-10e_ava-kinetics-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_k400-pre-r50_8xb8-4x16x1-10e_ava-kinetics-rgb/slowonly_k400-pre-r50_8xb8-4x16x1-10e_ava-kinetics-rgb_20221205-33e3ca7c.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_k400-pre-r50_8xb8-4x16x1-10e_ava-kinetics-rgb/slowonly_k400-pre-r50_8xb8-4x16x1-10e_ava-kinetics-rgb.log) |

|         4x16x1          |  8   | SlowOnly ResNet50 | Kinetics-700 | 25.87 | [config](/configs/detection/slowonly/slowonly_k700-pre-r50_8xb8-4x16x1-10e_ava-kinetics-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_k700-pre-r50_8xb8-4x16x1-10e_ava-kinetics-rgb/slowonly_k700-pre-r50_8xb8-4x16x1-10e_ava-kinetics-rgb_20221205-a07e8c15.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_k700-pre-r50_8xb8-4x16x1-10e_ava-kinetics-rgb/slowonly_k700-pre-r50_8xb8-4x16x1-10e_ava-kinetics-rgb.log) |

|          8x8x1          |  8   | SlowOnly ResNet50 | Kinetics-400 | 26.10 | [config](/configs/detection/slowonly/slowonly_k400-pre-r50_8xb8-8x8x1-10e_ava-kinetics-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_k400-pre-r50_8xb8-8x8x1-10e_ava-kinetics-rgb/slowonly_k400-pre-r50_8xb8-8x8x1-10e_ava-kinetics-rgb_20221205-8f8dff3b.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_k400-pre-r50_8xb8-8x8x1-10e_ava-kinetics-rgb/slowonly_k400-pre-r50_8xb8-8x8x1-10e_ava-kinetics-rgb.log) |

|          8x8x1          |  8   | SlowOnly ResNet50 | Kinetics-700 | 27.82 | [config](/configs/detection/slowonly/slowonly_k700-pre-r50_8xb8-8x8x1-10e_ava-kinetics-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_k700-pre-r50_8xb8-8x8x1-10e_ava-kinetics-rgb/slowonly_k700-pre-r50_8xb8-8x8x1-10e_ava-kinetics-rgb_20221205-16a01c37.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_k700-pre-r50_8xb8-8x8x1-10e_ava-kinetics-rgb/slowonly_k700-pre-r50_8xb8-8x8x1-10e_ava-kinetics-rgb.log) |

### AVA2.2 (Trained on AVA-Kinetics with tricks)

We conduct ablation studies to show the improvements of training tricks using SlowOnly8x8 pretrained on the Kinetics700 dataset. The baseline is the last row in **AVA2.2 (Trained on AVA-Kinetics)**.

|         method         | frame sampling strategy | gpus |     backbone      |   pretrain   |  mAP  |                  config                  |                  ckpt                   |                  log                   |
| :--------------------: | :---------------------: | :--: | :---------------: | :----------: | :---: | :--------------------------------------: | :-------------------------------------: | :------------------------------------: |

|        baseline        |          8x8x1          |  8   | SlowOnly ResNet50 | Kinetics-700 | 27.82 | [config](/configs/detection/slowonly/slowonly_k700-pre-r50_8xb8-8x8x1-10e_ava-kinetics-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_k700-pre-r50_8xb8-8x8x1-10e_ava-kinetics-rgb/slowonly_k700-pre-r50_8xb8-8x8x1-10e_ava-kinetics-rgb_20221205-16a01c37.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_k700-pre-r50_8xb8-8x8x1-10e_ava-kinetics-rgb/slowonly_k700-pre-r50_8xb8-8x8x1-10e_ava-kinetics-rgb.log) |

|       + context        |          8x8x1          |  8   | SlowOnly ResNet50 | Kinetics-700 | 28.31 | [config](/configs/detection/slowonly/slowonly_k700-pre-r50-context_8xb8-8x8x1-10e_ava-kinetics-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_k700-pre-r50-context_8xb8-8x8x1-10e_ava-kinetics-rgb/slowonly_k700-pre-r50-context_8xb8-8x8x1-10e_ava-kinetics-rgb_20221205-5d514f8c.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_k700-pre-r50-context_8xb8-8x8x1-10e_ava-kinetics-rgb/slowonly_k700-pre-r50-context_8xb8-8x8x1-10e_ava-kinetics-rgb.log) |

| + temporal max pooling |          8x8x1          |  8   | SlowOnly ResNet50 | Kinetics-700 | 28.48 | [config](/configs/detection/slowonly/slowonly_k700-pre-r50-context-temporal-max_8xb8-8x8x1-10e_ava-kinetics-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_k700-pre-r50-context-temporal-max_8xb8-8x8x1-10e_ava-kinetics-rgb/slowonly_k700-pre-r50-context-temporal-max_8xb8-8x8x1-10e_ava-kinetics-rgb_20221205-5b5e71eb.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_k700-pre-r50-context-temporal-max_8xb8-8x8x1-10e_ava-kinetics-rgb/slowonly_k700-pre-r50-context-temporal-max_8xb8-8x8x1-10e_ava-kinetics-rgb.log) |

|    + nonlinear head    |          8x8x1          |  8   | SlowOnly ResNet50 | Kinetics-700 | 29.83 | [config](/configs/detection/slowonly/slowonly_k700-pre-r50-context-temporal-max-nl-head_8xb8-8x8x1-10e_ava-kinetics-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_k700-pre-r50-context-temporal-max-nl-head_8xb8-8x8x1-10e_ava-kinetics-rgb/slowonly_k700-pre-r50-context-temporal-max-nl-head_8xb8-8x8x1-10e_ava-kinetics-rgb_20221205-87624265.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_k700-pre-r50-context-temporal-max-nl-head_8xb8-8x8x1-10e_ava-kinetics-rgb/slowonly_k700-pre-r50-context-temporal-max-nl-head_8xb8-8x8x1-10e_ava-kinetics-rgb.log) |

|      + focal loss      |          8x8x1          |  8   | SlowOnly ResNet50 | Kinetics-700 | 30.33 | [config](/configs/detection/slowonly/slowonly_k700-pre-r50-context-temporal-max-nl-head_8xb8-8x8x1-focal-10e_ava-kinetics-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_k700-pre-r50-context-temporal-max-nl-head_8xb8-8x8x1-focal-10e_ava-kinetics-rgb/slowonly_k700-pre-r50-context-temporal-max-nl-head_8xb8-8x8x1-focal-10e_ava-kinetics-rgb_20221205-37aa8395.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_k700-pre-r50-context-temporal-max-nl-head_8xb8-8x8x1-focal-10e_ava-kinetics-rgb/slowonly_k700-pre-r50-context-temporal-max-nl-head_8xb8-8x8x1-focal-10e_ava-kinetics-rgb.log) |

|     + more frames      |         16x4x1          |  8   | SlowOnly ResNet50 | Kinetics-700 | 31.29 | [config](/configs/detection/slowonly/slowonly_k700-pre-r50_8xb8-16x4x1-10e-tricks_ava-kinetics-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_k700-pre-r50_8xb8-16x4x1-10e-tricks_ava-kinetics-rgb/slowonly_k700-pre-r50_8xb8-16x4x1-10e-tricks_ava-kinetics-rgb_20221205-dd652f81.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_k700-pre-r50_8xb8-16x4x1-10e-tricks_ava-kinetics-rgb/slowonly_k700-pre-r50_8xb8-16x4x1-10e-tricks_ava-kinetics-rgb.log) |

### MultiSports

| frame sampling strategy | gpus |     backbone      |   pretrain   | f-mAP | v-mAP@0.2 | v-mAP@0.5 | v-mAP@0.1:0.9 | gpu_mem(M) |               config               |               ckpt               |               log                |

| :---------------------: | :--: | :---------------: | :----------: | :---: | :-------: | :-------: | :-----------: | :--------: | :--------------------------------: | :------------------------------: | :------------------------------: |
|         4x16x1          |  8   | SlowOnly ResNet50 | Kinetics-400 | 26.40 |   15.48   |   10.62   |     9.65      |    8509    | [config](/configs/detection/slowonly/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb_20230320-a1ca5e76.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb.log) |

2. **+ context** indicates that using both RoI feature and global pooled feature for classification; **+ temporal max pooling** indicates that using max pooling in the temporal dimension for the feature; **nonlinear head** indicates that using a 2-layer mlp instead of a linear classifier.
3. MultiSports dataset utilizes frame-mAP(f-mAP) and video-mAP(v-map) to evaluate performance. Frame-mAP evaluates on detection results of each frame, and video-mAP uses 3D IoU to evaluate tube-level results under several thresholds. You could refer to the [competition page](https://codalab.lisn.upsaclay.fr/competitions/3736#learn_the_details-evaluation) for details.

- [AVA](/tools/data/ava/README.md)
- [AVA-Kinetics](/tools/data/ava_kinetics/README.md)
- [MultiSports](/tools/data/multisports/README.md)

## Train

```shell
```

Example: train the SlowOnly model on AVA2.1 in a deterministic option with periodic validation.

```shell
python tools/train.py configs/detection/slowonly/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb.py \
```

## Test

```shell
```

Example: test the SlowOnly model on AVA2.1 and dump the result to a pkl file.

```shell

```shell
```

```shell
python tools/test.py configs/detection/slowonly/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb.py \
```

```BibTeX
}
```

```BibTeX

booktitle={ICCV},
}
```

```BibTeX
}
```

```BibTeX
}
```

# Preparing SPMCS Dataset

```bibtex
}
```

The datasets can be download from [here](https://opendatalab.org.cn/SPMCS).

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── SPMCS
|   |   ├── GT
|   |   ├── BIx4
|   |   ├── BDx4
```

## 常用工具（待更新）

除了训练和测试的脚本，我们在 `tools/` 文件夹路径下还提供许多有用的工具。

### 计算参数量（params）和计算量（ FLOPs） (试验性)

我们基于 [flops-counter.pytorch](https://github.com/sovrasov/flops-counter.pytorch)
提供了一个用于计算给定模型参数量和计算量的脚本。

```shell
python tools/get_flops.py ${CONFIG_FILE} [--shape ${INPUT_SHAPE}]
```

您将得到如下的结果：

```none
Input shape: (3, 2048, 1024)
Flops: 1429.68 GMac
Params: 48.98 M
```

**注意**: 这个工具仍然是试验性的，我们无法保证数字是正确的。您可以拿这些结果做简单的实验的对照，在写技术文档报告或者论文前您需要再次确认一下。

(1) 计算量与输入的形状有关，而参数量与输入的形状无关，默认的输入形状是 (1, 3, 1280, 800)；
(2) 一些运算操作，如 GN 和其他定制的运算操作没有加入到计算量的计算中。

### 发布模型

在您上传一个模型到云服务器之前，您需要做以下几步：
(1) 将模型权重转成 CPU 张量；
(2) 删除记录优化器状态 (optimizer states)的相关信息；
(3) 计算检查点文件 (checkpoint file) 的哈希编码（hash id）并且将哈希编码加到文件名中。

```shell
python tools/publish_model.py ${INPUT_FILENAME} ${OUTPUT_FILENAME}
```

例如，

```shell
python tools/publish_model.py work_dirs/pspnet/latest.pth psp_r50_hszhao_200ep.pth
```

```shell
```

例如，

```shell
```

最终输出文件将是 `psp_r50_512x1024_40ki_cityscapes-{hash id}.pth`。

### 导出 ONNX (试验性)

我们提供了一个脚本来导出模型到 [ONNX](https://github.com/onnx/onnx) 格式。被转换的模型可以通过工具 [Netron](https://github.com/lutzroeder/netron)
来可视化。除此以外，我们同样支持对 PyTorch 和 ONNX 模型的输出结果做对比。

```bash
python tools/pytorch2onnx.py \
    --checkpoint ${CHECKPOINT_FILE} \
    --output-file ${ONNX_FILE} \
    --input-img ${INPUT_IMG} \
    --shape ${INPUT_SHAPE} \
    --rescale-shape ${RESCALE_SHAPE} \
    --show \
    --verify \
    --dynamic-export \
    --cfg-options \
      model.test_cfg.mode="whole"
```

各个参数的描述:

- `config` : 模型配置文件的路径
- `--checkpoint` : 模型检查点文件的路径
- `--output-file`: 输出的 ONNX 模型的路径。如果没有专门指定，它默认是 `tmp.onnx`
- `--input-img` : 用来转换和可视化的一张输入图像的路径
- `--shape`: 模型的输入张量的高和宽。如果没有专门指定，它将被设置成 `test_pipeline` 的 `img_scale`

- `--rescale-shape`: 改变输出的形状。设置这个值来避免 OOM，它仅在 `slide` 模式下可以用
- `--show`: 是否打印输出模型的结构。如果没有被专门指定，它将被设置成 `False`
- `--verify`: 是否验证一个输出模型的正确性 (correctness)。如果没有被专门指定，它将被设置成 `False`
- `--dynamic-export`: 是否导出形状变化的输入与输出的 ONNX 模型。如果没有被专门指定，它将被设置成 `False`
- `--cfg-options`: 更新配置选项

**注意**: 这个工具仍然是试验性的，目前一些自定义操作还没有被支持

### 评估 ONNX 模型

我们提供 `tools/deploy_test.py` 去评估不同后端的 ONNX 模型。

#### 先决条件

- 安装 onnx 和 onnxruntime-gpu

  ```shell
  pip install onnx onnxruntime-gpu
  ```

- 参考 [如何在 MMCV 里构建 tensorrt 插件](https://mmcv.readthedocs.io/en/latest/tensorrt_plugin.html#how-to-build-tensorrt-plugins-in-mmcv) 安装TensorRT (可选)

#### 使用方法

```bash
python tools/deploy_test.py \
    ${MODEL_FILE} \
    ${BACKEND} \
    --out ${OUTPUT_FILE} \
    --eval ${EVALUATION_METRICS} \
    --show \

#### 使用方法

```bash
    --show \
    --show-dir ${SHOW_DIRECTORY} \
    --eval-options ${EVALUATION_OPTIONS} \
    --opacity ${OPACITY} \
```

各个参数的描述:

- `config`: 模型配置文件的路径
- `model`: 被转换的模型文件的路径
- `backend`: 推理的后端，可选项：`onnxruntime`， `tensorrt`
- `--out`: 输出结果成 pickle 格式文件的路径
- `--format-only` : 不评估直接给输出结果的格式。通常用在当您想把结果输出成一些测试服务器需要的特定格式时。如果没有被专门指定，它将被设置成 `False`。 注意这个参数是用 `--eval` 来 **手动添加**
- `--eval`: 评估指标，取决于每个数据集的要求，例如 "mIoU" 是大多数据集的指标而 "cityscapes" 仅针对 Cityscapes 数据集。注意这个参数是用 `--format-only` 来 **手动添加**
- `--show`: 是否展示结果
- `--show-dir`: 涂上结果的图像被保存的文件夹的路径
- `--cfg-options`: 重写配置文件里的一些设置，`xxx=yyy` 格式的键值对将被覆盖到配置文件里
- `--eval-options`: 自定义的评估的选项， `xxx=yyy` 格式的键值对将成为  `dataset.evaluate()` 函数的参数变量
- `--opacity`: 涂上结果的分割图的透明度，范围在 (0, 1\] 之间

#### 结果和模型

#### 结果和模型

|    模型    |                    配置文件                     |   数据集   | 评价指标 | PyTorch | ONNXRuntime | TensorRT-fp32 | TensorRT-fp16 |
| :--------: | :---------------------------------------------: | :--------: | :------: | :-----: | :---------: | :-----------: | :-----------: |
|    FCN     |      fcn_r50-d8_512x1024_40k_cityscapes.py      | cityscapes |   mIoU   |  72.2   |    72.2     |     72.2      |     72.2      |
|   PSPNet   |    pspnet_r50-d8_512x1024_40k_cityscapes.py     | cityscapes |   mIoU   |  77.8   |    77.8     |     77.8      |     77.8      |
| deeplabv3  |   deeplabv3_r50-d8_512x1024_40k_cityscapes.py   | cityscapes |   mIoU   |  79.0   |    79.0     |     79.0      |     79.0      |

| deeplabv3+ | deeplabv3plus_r50-d8_512x1024_40k_cityscapes.py | cityscapes |   mIoU   |  79.6   |    79.5     |     79.5      |     79.5      |
|   PSPNet   |     pspnet_r50-d8_769x769_40k_cityscapes.py     | cityscapes |   mIoU   |  78.2   |    78.1     |               |               |
| deeplabv3  |   deeplabv3_r50-d8_769x769_40k_cityscapes.py    | cityscapes |   mIoU   |  78.5   |    78.3     |               |               |
| deeplabv3+ | deeplabv3plus_r50-d8_769x769_40k_cityscapes.py  | cityscapes |   mIoU   |  78.9   |    78.7     |               |               |

**注意**: TensorRT 仅在使用 `whole mode` 测试模式时的配置文件里可用。

### 导出 TorchScript (试验性)

我们同样提供一个脚本去把模型导出成 [TorchScript](https://pytorch.org/docs/stable/jit.html) 格式。您可以使用 pytorch C++ API [LibTorch](https://pytorch.org/docs/stable/cpp_index.html) 去推理训练好的模型。

被转换的模型能被像 [Netron](https://github.com/lutzroeder/netron) 的工具来可视化。此外，我们还支持 PyTorch 和 TorchScript 模型的输出结果的比较。

```shell
python tools/pytorch2torchscript.py \
    --output-file ${ONNX_FILE}
    --shape ${INPUT_SHAPE}
    --verify \
    --show
```

各个参数的描述:

- `config` : pytorch 模型的配置文件的路径
- `--checkpoint` : pytorch 模型的检查点文件的路径
- `--output-file`: TorchScript 模型输出的路径，如果没有被专门指定，它将被设置成 `tmp.pt`
- `--input-img` : 用来转换和可视化的输入图像的路径
- `--shape`: 模型的输入张量的宽和高。如果没有被专门指定，它将被设置成 `512 512`
- `--show`: 是否打印输出模型的追踪图 (traced graph)，如果没有被专门指定，它将被设置成 `False`
- `--verify`: 是否验证一个输出模型的正确性 (correctness)，如果没有被专门指定，它将被设置成 `False`

**注意**: 目前仅支持 PyTorch>=1.8.0 版本

**注意**: 这个工具仍然是试验性的，一些自定义操作符目前还不被支持

例子:

- 导出 PSPNet 在 cityscapes 数据集上的 pytorch 模型

  ```shell

例子:

  ```shell
  python tools/pytorch2torchscript.py configs/pspnet/pspnet_r50-d8_512x1024_40k_cityscapes.py \
  --checkpoint checkpoints/pspnet_r50-d8_512x1024_40k_cityscapes_20200605_003338-2966598c.pth \
  --output-file checkpoints/pspnet_r50-d8_512x1024_40k_cityscapes_20200605_003338-2966598c.pt \
  --shape 512 1024
  ```

### 导出 TensorRT (试验性)

一个导出 [ONNX](https://github.com/onnx/onnx) 模型成 [TensorRT](https://developer.nvidia.com/tensorrt) 格式的脚本

先决条件

- 按照 [ONNXRuntime in mmcv](https://mmcv.readthedocs.io/en/latest/deployment/onnxruntime_op.html) 和 [TensorRT plugin in mmcv](https://github.com/open-mmlab/mmcv/blob/master/docs/en/deployment/tensorrt_plugin.md) ，用 ONNXRuntime 自定义运算 (custom ops) 和 TensorRT 插件安装 `mmcv-full`
- 使用 [pytorch2onnx](#convert-to-onnx-experimental) 将模型从 PyTorch 转成 ONNX

使用方法

使用方法

```bash
python ${MMSEG_PATH}/tools/onnx2tensorrt.py \
    ${CFG_PATH} \
    --trt-file ${OUTPUT_TRT_PATH} \
    --min-shape ${MIN_SHAPE} \
    --max-shape ${MAX_SHAPE} \
    --show \
    --verify
```

各个参数的描述:

- `config` : 模型的配置文件
- `model` : 输入的 ONNX 模型的路径
- `--trt-file` : 输出的 TensorRT 引擎的路径
- `--max-shape` : 模型的输入的最大形状
- `--min-shape` : 模型的输入的最小形状
- `--fp16` : 做 fp16 模型转换
- `--workspace-size` : 在 GiB 里的最大工作空间大小 (Max workspace size)
- `--input-img` : 用来可视化的图像
- `--show` : 做结果的可视化
- `--dataset` : Palette provider, 默认为 `CityscapesDataset`
- `--verify` : 验证 ONNXRuntime 和 TensorRT 的输出
- `--verbose` : 当创建 TensorRT 引擎时，是否详细做信息日志。默认为 False

**注意**: 仅在全图测试模式 (whole mode) 下测试过

## 其他内容

### 打印完整的配置文件

`tools/print_config.py` 会逐字逐句的打印整个配置文件，展开所有的导入。

```shell
  ${CONFIG} \
  --graph \
  --cfg-options ${OPTIONS [OPTIONS...]} \
```

## 其他内容

```shell
  --graph \
```

各个参数的描述:

- `--graph` : 是否打印模型的图 (models graph)
- `--cfg-options`: 自定义替换配置文件的选项

### 对训练日志 (training logs) 画图

`tools/analyze_logs.py` 会画出给定的训练日志文件的 loss/mIoU 曲线，首先需要 `pip install seaborn` 安装依赖包。

```shell
python tools/analyze_logs.py xxx.log.json [--keys ${KEYS}] [--legend ${LEGEND}] [--backend ${BACKEND}] [--style ${STYLE}] [--out ${OUT_FILE}]
```

示例:

- 对 mIoU, mAcc, aAcc 指标画图

  ```shell
  python tools/analyze_logs.py log.json --keys mIoU mAcc aAcc --legend mIoU mAcc aAcc
  ```

- 对 loss 指标画图

  ```shell
  python tools/analyze_logs.py log.json --keys loss --legend loss
  ```

### 转换其他仓库的权重

`tools/model_converters/` 提供了若干个预训练权重转换脚本，支持将其他仓库的预训练权重的 key 转换为与 MMSegmentation 相匹配的 key。

#### ViT Swin MiT Transformer 模型

- ViT

```

- ViT

`tools/model_converters/vit2mmseg.py` 将 timm 预训练模型转换到 MMSegmentation。

```shell
python tools/model_converters/vit2mmseg.py ${SRC} ${DST}
```

- Swin

  `tools/model_converters/swin2mmseg.py` 将官方预训练模型转换到 MMSegmentation。

  ```shell
  python tools/model_converters/swin2mmseg.py ${SRC} ${DST}
  ```

- SegFormer

  `tools/model_converters/mit2mmseg.py` 将官方预训练模型转换到 MMSegmentation。

  ```shell
  python tools/model_converters/mit2mmseg.py ${SRC} ${DST}
  ```

## 模型服务

为了用 [`TorchServe`](https://pytorch.org/serve/) 服务 `MMSegmentation` 的模型 ， 您可以遵循如下流程:

### 1. 将 model 从　MMSegmentation 转换到 TorchServe

```shell
python tools/mmseg2torchserve.py ${CONFIG_FILE} ${CHECKPOINT_FILE} \
```

**注意**: ${MODEL_STORE} 需要设置为某个文件夹的绝对路径

### 2. 构建 `mmseg-serve` 容器镜像 (docker image)

```shell

```

```shell
docker build -t mmseg-serve:latest docker/serve/
```

### 3. 运行 `mmseg-serve`

请查阅官方文档: [使用容器运行 TorchServe](https://github.com/pytorch/serve/blob/master/docker/README.md#running-torchserve-in-a-production-docker-environment)

为了在 GPU 环境下使用, 您需要安装 [nvidia-docker](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html). 若在 CPU 环境下使用，您可以忽略添加 `--gpus` 参数。

示例:

```shell
--cpus 8 \
mmseg-serve:latest
```

阅读关于推理 (8080), 管理 (8081) 和指标 (8082) APIs 的 [文档](https://github.com/pytorch/serve/blob/072f5d088cce9bb64b2a18af065886c9b01b317b/docs/rest_api.md) 。

### 4. 测试部署

```shell

```

### 4. 测试部署

```shell
curl -O https://raw.githubusercontent.com/open-mmlab/mmsegmentation/master/resources/3dogs.jpg
curl http://127.0.0.1:8080/predictions/${MODEL_NAME} -T 3dogs.jpg -o 3dogs_mask.png
```

得到的响应将是一个 ".png" 的分割掩码.

您可以按照如下方法可视化输出:

```python
import matplotlib.pyplot as plt
plt.imshow(mmcv.imread("3dogs_mask.png", "grayscale"))
plt.show()
```

看到的东西将会和下图类似:

![3dogs_mask](../../resources/3dogs_mask.png)

然后您可以使用 `test_torchserve.py` 比较 torchserve 和 pytorch 的结果，并将它们可视化。

```shell
python tools/torchserve/test_torchserve.py ${IMAGE_FILE} ${CONFIG_FILE} ${CHECKPOINT_FILE} ${MODEL_NAME}
[--inference-addr ${INFERENCE_ADDR}] [--result-image ${RESULT_IMAGE}] [--device ${DEVICE}]
```

示例：

```shell
python tools/torchserve/test_torchserve.py \
demo/demo.png \

```

示例：

```shell
configs/fcn/fcn_r50-d8_512x1024_40k_cityscapes.py \
checkpoint/fcn_r50-d8_512x1024_40k_cityscapes_20200604_192608-efe53f0d.pth \
fcn
```

# \[WIP\] Deployment

> ## [Try the new MMDeploy to deploy your model](https://mmdeploy.readthedocs.io/)

## Convert to ONNX (experimental)

We provide a script to convert model to [ONNX](https://github.com/onnx/onnx) format. The converted model could be visualized by tools like [Netron](https://github.com/lutzroeder/netron). Besides, we also support comparing the output results between PyTorch and ONNX model.

```bash
    --show \
    --verify \
```

Description of arguments:

- `--checkpoint` : The path of a model checkpoint file.
- `--output-file`: The path of output ONNX model. If not specified, it will be set to `tmp.onnx`.

- `--input-img` : The path of an input image for conversion and visualize.
- `--shape`: The height and width of input tensor to the model. If not specified, it will be set to img_scale of test_pipeline.
- `--rescale-shape`: rescale shape of output, set this value to avoid OOM, only work on `slide` mode.
- `--show`: Determines whether to print the architecture of the exported model. If not specified, it will be set to `False`.
- `--verify`: Determines whether to verify the correctness of an exported model. If not specified, it will be set to `False`.
- `--dynamic-export`: Determines whether to export ONNX model with dynamic input and output shapes. If not specified, it will be set to `False`.
- `--cfg-options`:Update config options.

:::{note}

:::{note}
This tool is still experimental. Some customized operators are not supported for now.
:::

### Evaluate ONNX model

We provide `tools/deploy_test.py` to evaluate ONNX model with different backend.

- Install onnx and onnxruntime-gpu

  ```shell
  ```

- Install TensorRT following [how-to-build-tensorrt-plugins-in-mmcv](https://mmcv.readthedocs.io/en/latest/tensorrt_plugin.html#how-to-build-tensorrt-plugins-in-mmcv)(optional)

### Usage

```bash
    --show \

--out ${OUTPUT_FILE} \
    --show \
```

Description of all arguments

- `model`: The path of a converted model file.
- `backend`: Backend of the inference, options: `onnxruntime`, `tensorrt`.
- `--out`: The path of output result file in pickle format.
- `--format-only` : Format the output results without perform evaluation. It is useful when you want to format the result to a specific format and submit it to the test server. If not specified, it will be set to `False`. Note that this argument is **mutually exclusive** with `--eval`.
- `--eval`: Evaluation metrics, which depends on the dataset, e.g., "mIoU" for generic datasets, and "cityscapes" for Cityscapes. Note that this argument is **mutually exclusive** with `--format-only`.
- `--show`: Show results flag.

- `--show-dir`: Directory where painted images will be saved
- `--cfg-options`: Override some settings in the used config file, the key-value pair in `xxx=yyy` format will be merged into config file.
- `--eval-options`: Custom options for evaluation, the key-value pair in `xxx=yyy` format will be kwargs for `dataset.evaluate()` function
- `--opacity`: Opacity of painted segmentation map. In (0, 1\] range.

### Results and Models

|   Model    |                     Config                      |  Dataset   | Metric | PyTorch | ONNXRuntime | TensorRT-fp32 | TensorRT-fp16 |
| :--------: | :---------------------------------------------: | :--------: | :----: | :-----: | :---------: | :-----------: | :-----------: |
|    FCN     |      fcn_r50-d8_512x1024_40k_cityscapes.py      | cityscapes |  mIoU  |  72.2   |    72.2     |     72.2      |     72.2      |

|   PSPNet   |    pspnet_r50-d8_512x1024_40k_cityscapes.py     | cityscapes |  mIoU  |  77.8   |    77.8     |     77.8      |     77.8      |
| deeplabv3  |   deeplabv3_r50-d8_512x1024_40k_cityscapes.py   | cityscapes |  mIoU  |  79.0   |    79.0     |     79.0      |     79.0      |
| deeplabv3+ | deeplabv3plus_r50-d8_512x1024_40k_cityscapes.py | cityscapes |  mIoU  |  79.6   |    79.5     |     79.5      |     79.5      |
|   PSPNet   |     pspnet_r50-d8_769x769_40k_cityscapes.py     | cityscapes |  mIoU  |  78.2   |    78.1     |               |               |
| deeplabv3  |   deeplabv3_r50-d8_769x769_40k_cityscapes.py    | cityscapes |  mIoU  |  78.5   |    78.3     |               |               |
| deeplabv3+ | deeplabv3plus_r50-d8_769x769_40k_cityscapes.py  | cityscapes |  mIoU  |  78.9   |    78.7     |               |               |

:::{note}
TensorRT is only available on configs with `whole mode`.
:::

## Convert to TorchScript (experimental)

We also provide a script to convert model to [TorchScript](https://pytorch.org/docs/stable/jit.html) format. You can use the pytorch C++ API [LibTorch](https://pytorch.org/docs/stable/cpp_index.html) inference the trained model. The converted model could be visualized by tools like [Netron](https://github.com/lutzroeder/netron). Besides, we also support comparing the output results between PyTorch and TorchScript model.

```shell
    --verify \
    --show
```

- `config` : The path of a pytorch model config file.

--output-file ${ONNX_FILE}
    --verify \
    --show
```

- `--checkpoint` : The path of a pytorch model checkpoint file.
- `--output-file`: The path of output TorchScript model. If not specified, it will be set to `tmp.pt`.
- `--shape`: The height and width of input tensor to the model. If not specified, it will be set to `512 512`.
- `--show`: Determines whether to print the traced graph of the exported model. If not specified, it will be set to `False`.

:::{note}
It's only support PyTorch>=1.8.0 for now.
:::

:::{note}
:::

Examples:

- Convert the cityscapes PSPNet pytorch model.

:::

:::{note}
:::

Examples:

  ```shell
  ```

## Convert to TensorRT (experimental)

A script to convert [ONNX](https://github.com/onnx/onnx) model to [TensorRT](https://developer.nvidia.com/tensorrt) format.

Prerequisite

- install `mmcv-full` with ONNXRuntime custom ops and TensorRT plugins follow [ONNXRuntime in mmcv](https://mmcv.readthedocs.io/en/latest/deployment/onnxruntime_op.html) and [TensorRT plugin in mmcv](https://github.com/open-mmlab/mmcv/blob/master/docs/en/deployment/tensorrt_plugin.md).

- Use [pytorch2onnx](#convert-to-onnx-experimental) to convert the model from PyTorch to ONNX.

Usage

```bash
    --show \
    --verify
```

- `config` : Config file of the model.
- `model` : Path to the input ONNX model.
- `--trt-file` : Path to the output TensorRT engine.
- `--max-shape` : Maximum shape of model input.
- `--min-shape` : Minimum shape of model input.
- `--fp16` : Enable fp16 model conversion.
- `--workspace-size` : Max workspace size in GiB.
- `--input-img` : Image for visualize.
- `--show` : Enable result visualize.
- `--dataset` : Palette provider, `CityscapesDataset` as default.
- `--verify` : Verify the outputs of ONNXRuntime and TensorRT.

- `--verbose` : Whether to verbose logging messages while creating TensorRT engine. Defaults to False.

:::{note}
Only tested on whole mode.
:::

### blob内存是隐含共享的

ncnn的blob最初直接使用opencv的cv::Mat，后发现blob最多只支持三维，因此实现了类似的Mat
Mat的data每个通道内存16字节对齐，并且有原子的引用计数，a=b不复制数据，超级快
Mat支持直接引用外部的内存块，不复制数据，加快模型加载和输入输出

举个例子：split layer 将一个blob复制成n个，ncnn中实现为单纯的增加引用计数，没有任何数据复制

### 只运算一部分并保留中间结果

ncnn的net在解决分支依赖时是自上而下深度优先的，因此当网络有多个分支时，运算只会在需要结果的那个分支中进行，节约时间
当多个分支有重合部分时，运算其中一个分支后会自动保留其余分支所需的中间结果，隐含共享，以便运算其余分支时利用

举个例子：某网络结构为 A -> B -> C1 + C2，向ncnn索要C1结果时，运算过程是 A -> B -> C1，同时B结果引用计数加1自动保留，后面还需要C2结果时，只运算C2就足够了

### 开启轻模式省内存

每个layer都会产生blob，除了最后的结果和多分支中间结果，大部分blob都不值得保留，开启轻模式可以在运算后自动回收，省下内存

举个例子：某网络结构为 A -> B -> C，在轻模式下，向ncnn索要C结果时，A结果会在运算B时自动回收，而B结果会在运算C时自动回收，最后只保留C结果，后面再需要C结果会直接获得，满足绝大部分深度网络的使用方式

### 网络和运算是分开的

ncnn的net是网络模型，实际使用的是extractor，也就是同个net可以有很多个运算实例，而且运算实例互不影响，中间结果保留在extractor内部，在多线程使用时共用网络的结构和参数数据，初始化网络模型和参数只需要一遍

举个例子：全局静态的net实例，初始化一次后，就能不停地生成extractor使用

### openmp虽快但未必合适

ncnn中几乎所有运算都能用上openmp多线程加速，而且性能很赞
不过系统有时候会突然慢一下，比如手机太热自动降频，界面操作等等，ncnn耗时也会偶尔抖动变长，在计算耗时稳定性比较重要的时候建议关闭openmp，或者设置下extractor线程数

举个例子：手机自拍时，用ncnn进行人脸实时定位，如果耗时突然涨一下就会感觉到掉帧，而稳定的帧率体验更好

### NCNN_STDIO/NCNN_STRING禁用模型文件

ncnn支持加载自有的模型文件和模型内存，NCNN_STDIO控制是否需要支持加载模型文件，设成0能禁用这部分代码，从而减小库的体积，NCNN_STRING设成0能清除大部分可见的字符串和解析过程
模型内存加载时的参数数据是直接引用的，速度更快，通常在手机上使用这种方式

### 削减 ncnn 内置的层实现

cmake的时候，加参数 -DWITH_LAYER_xxx=OFF 就可以完全不编译对应的内置层，这样可以进一步减小库的体积

### 关于 ARM big.LITTLE 调度

调用set_cpu_powersave可以把ncnn运算线程控制在特定的cpu核心上，大核心速度快耗电多，小核心速度慢点但省电，大小一起用手机热得快

# 2D Animal Keypoint Detection

2D animal keypoint detection (animal pose estimation) aims to detect the key-point of different species, including rats,
dogs, macaques, and cheetah. It provides detailed behavioral analysis for neuroscience, medical and ecology applications.

Please follow [DATA Preparation](/docs/en/dataset_zoo/2d_animal_keypoint.md) to prepare data.

## Demo

Please follow [DEMO](/demo/docs/en/2d_animal_demo.md) to generate fancy demos.

<img src="https://user-images.githubusercontent.com/11788150/114201893-4446ec00-9989-11eb-808b-5718c47c7b23.gif" height="140px" alt><br>

<img src="https://user-images.githubusercontent.com/11788150/114205282-b5d46980-998c-11eb-9d6b-85ba47f81252.gif" height="140px" alt><br>

<img src="https://user-images.githubusercontent.com/11788150/114023530-944c8280-98a5-11eb-86b0-5f6d3e232af0.gif" height="140px" alt><br>

# DPT

> [Vision Transformer for Dense Prediction](https://arxiv.org/abs/2103.13413)

<a href="https://github.com/isl-org/DPT">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/decode_heads/dpt_head.py#L215">Code Snippet</a>

We introduce dense vision transformers, an architecture that leverages vision transformers in place of convolutional networks as a backbone for dense prediction tasks. We assemble tokens from various stages of the vision transformer into image-like representations at various resolutions and progressively combine them into full-resolution predictions using a convolutional decoder. The transformer backbone processes representations at a constant and relatively high resolution and has a global receptive field at every stage. These properties allow the dense vision transformer to provide finer-grained and more globally coherent predictions when compared to fully-convolutional networks. Our experiments show that this architecture yields substantial improvements on dense prediction tasks, especially when a large amount of training data is available. For monocular depth estimation, we observe an improvement of up to 28% in relative performance when compared to a state-of-the-art fully-convolutional network. When applied to semantic segmentation, dense vision transformers set a new state of the art on ADE20K with 49.02% mIoU. We further show that the architecture can be fine-tuned on smaller datasets such as NYUv2, KITTI, and Pascal Context where it also sets the new state of the art. Our models are available at [this https URL](https://github.com/isl-org/DPT).

<img src="https://user-images.githubusercontent.com/24582831/142901057-00aabea5-dab4-43d3-a14a-5f73eb5dd9b9.png" width="80%"/>
</div>

## Usage

```shell
```

E.g.

```shell
```

### ADE20K

```

### ADE20K

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                               | download                                                                                                                                                                                                                                                                                               |

| DPT    | ViT-B    | 512x512   |  160000 | 8.09     | 10.41          | V100   | 46.97 |         48.34 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dpt/dpt_vit-b16_8xb2-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/dpt/dpt_vit-b16_512x512_160k_ade20k/dpt_vit-b16_512x512_160k_ade20k-db31cf52.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/dpt/dpt_vit-b16_512x512_160k_ade20k/dpt_vit-b16_512x512_160k_ade20k-20210809_172025.log.json) |

```bibtex
  journal={arXiv preprint arXiv:2010.11929},
}

journal={arXiv preprint arXiv:2010.11929},
}

@article{Ranftl2021,
  author    = {Ren\'{e} Ranftl and Alexey Bochkovskiy and Vladlen Koltun},
  title     = {Vision Transformers for Dense Prediction},
  journal   = {ArXiv preprint},
  year      = {2021},
}
```

# Simple online and realtime tracking

This paper explores a pragmatic approach to multiple object tracking where the main focus is to associate objects efficiently for online and realtime applications. To this end, detection quality is identified as a key factor influencing tracking performance, where changing the detector can improve tracking by up to 18.9%. Despite only using a rudimentary combination of familiar techniques such as the Kalman Filter and Hungarian algorithm for the tracking components, this approach achieves an accuracy comparable to state-of-the-art online trackers. Furthermore, due to the simplicity of our tracking method, the tracker updates at a rate of 260 Hz which is over 20x faster than other state-of-the-art trackers.

  <img src="https://user-images.githubusercontent.com/99722489/176848133-d6621813-7b8f-4b25-96cd-2fbcc87983ce.png"/>
</div>

```latex

</div>

```latex
@inproceedings{bewley2016simple,
  title={Simple online and realtime tracking},
  author={Bewley, Alex and Ge, Zongyuan and Ott, Lionel and Ramos, Fabio and Upcroft, Ben},
  booktitle={2016 IEEE International Conference on Image Processing (ICIP)},
  pages={3464--3468},
}
```

| Method |      Detector      | ReID | Train Set  | Test Set | Public | Inf time (fps) | HOTA | MOTA | IDF1 |  FP   |  FN   | IDSw. |                                     Config                                     |                                                       Download                                                       |

| :----: | :----------------: | :--: | :--------: | :------: | :----: | :------------: | :--: | :--: | :--: | :---: | :---: | :---: | :----------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------: |
|  SORT  | R50-FasterRCNN-FPN |  -   | half-train | half-val |   N    |      18.6      | 52.0 | 62.0 | 57.8 | 15150 | 40410 | 5847  | [config](sort_faster-rcnn_r50_fpn_8xb2-4e_mot17halftrain_test-mot17halfval.py) | [detector](https://download.openmmlab.com/mmtracking/mot/faster_rcnn/faster-rcnn_r50_fpn_4e_mot17-half-64ee2ed4.pth) |

We implement SORT with independent detector models.
Note that, due to the influence of parameters such as learning rate in default configuration file,
we recommend using 8 GPUs for training in order to reproduce accuracy.

You can train the detector as follows.

# Training Faster R-CNN on mot17-half-train dataset with following command.
bash tools/dist_train.sh configs/sort/faster-rcnn_r50_fpn_8xb2-4e_mot17halftrain_test-mot17halfval.py 8
```

### 4.1 Example on MOTxx-halfval dataset

**4.1.1 use separate trained detector model to evaluating and testing**\*

# Example 1: Test on motXX-half-val set.
bash tools/dist_test_tracking.sh configs/sort/sort_faster-rcnn_r50_fpn_8xb2-4e_mot17halftrain_test-mot17halfval.py 8 --detector ${DETECTOR_CHECKPOINT_PATH}
```

**4.1.2 use video_baesd to evaluating and testing**

```
```

### 4.2 Example on MOTxx-test dataset

If you want to get the results of the [MOT Challenge](https://motchallenge.net/) test set,

```

please use the following command to generate result files that can be used for submission.
It will be stored in `./mot_17_test_res`, you can modify the saved path in `test_evaluator` of the config.

# Example 2: Test on motxx-test set
# The number after config file represents the number of GPUs used
bash tools/dist_test_tracking.sh configs/sort/sort_faster-rcnn_r50_fpn_8xb2-4e_mot17train_test-mot17test.py 8 --detector ${DETECTOR_CHECKPOINT_PATH}
```

```shell

```shell
python demo/mot_demo.py demo/demo_mot.mp4 configs/sort/sort_faster-rcnn_r50_fpn_8xb2-4e_mot17halftrain_test-mot17halfval.py --detector ${DETECTOR_CHECKPOINT_PATH}  --out mot.mp4
```

}
```

<details>

```bibtex
}
```

Results on DeepFashion2 val set

| Set                   |                        Arch                         | Input Size | PCK@0.2 |  AUC  | EPE  |                        ckpt                         |                         log                         |

| :-------------------- | :-------------------------------------------------: | :--------: | :-----: | :---: | :--: | :-------------------------------------------------: | :-------------------------------------------------: |
| short_sleeved_shirt   | [pose_resnet_50](/configs/fashion_2d_keypoint/topdown_heatmap/deepfashion2/td-hm_res50_6xb64-210e_deepfasion2-short-sleeved-shirt-256x192.py) |  256x192   |  0.988  | 0.703 | 10.2 | [ckpt](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_short_sleeved_shirt_256x192-21e1c5da_20221208.pth) | [log](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_short_sleeved_shirt_256x192_20221208.log.json) |

| long_sleeved_shirt    | [pose_resnet_50](/configs/fashion_2d_keypoint/topdown_heatmap/deepfashion2/td-hm_res50_8xb64-210e_deepfasion2-long-sleeved-shirt-256x192.py) |  256x192   |  0.973  | 0.587 | 16.6 | [ckpt](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_long_sleeved_shirt_256x192-8679e7e3_20221208.pth) | [log](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_long_sleeved_shirt_256x192_20221208.log.json) |
| short_sleeved_outwear | [pose_resnet_50](/configs/fashion_2d_keypoint/topdown_heatmap/deepfashion2/td-hm_res50_8xb64-210e_deepfasion2-short-sleeved-outwear-256x192.py) |  256x192   |  0.966  | 0.408 | 24.0 | [ckpt](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_short_sleeved_outwear_256x192-a04c1298_20221208.pth) | [log](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_short_sleeved_outwear_256x192_20221208.log.json) |

| long_sleeved_outwear  | [pose_resnet_50](/configs/fashion_2d_keypoint/topdown_heatmap/deepfashion2/td-hm_res50_8xb64-210e_deepfasion2-long-sleeved-outwear-256x192.py) |  256x192   |  0.987  | 0.517 | 18.1 | [ckpt](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_long_sleeved_outwear_256x192-31fbaecf_20221208.pth) | [log](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_long_sleeved_outwear_256x192_20221208.log.json) |
| vest                  | [pose_resnet_50](/configs/fashion_2d_keypoint/topdown_heatmap/deepfashion2/td-hm_res50_4xb64-210e_deepfasion2-vest-256x192.py) |  256x192   |  0.981  | 0.643 | 12.7 | [ckpt](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_vest_256x192-4c48d05c_20221208.pth) | [log](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_vest_256x192_20221208.log.json) |

| sling                 | [pose_resnet_50](/configs/fashion_2d_keypoint/topdown_heatmap/deepfashion2/td-hm_res50_4xb64-210e_deepfasion2-sling-256x192.py) |  256x192   |  0.940  | 0.557 | 21.6 | [ckpt](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_sling_256x192-ebb2b736_20221208.pth) | [log](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_sling_256x192_20221208.log.json) |
| shorts                | [pose_resnet_50](/configs/fashion_2d_keypoint/topdown_heatmap/deepfashion2/td-hm_res50_3xb64-210e_deepfasion2-shorts-256x192.py) |  256x192   |  0.975  | 0.682 | 12.4 | [ckpt](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_shorts_256x192-9ab23592_20221208.pth) | [log](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_shorts_256x192_20221208.log.json) |

| trousers              | [pose_resnet_50](/configs/fashion_2d_keypoint/topdown_heatmap/deepfashion2/td-hm_res50_2xb64-210e_deepfasion2-trousers-256x192.py) |  256x192   |  0.973  | 0.625 | 14.8 | [ckpt](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_trousers_256x192-3e632257_20221208.pth) | [log](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_trousers_256x192_20221208.log.json) |
| skirt                 | [pose_resnet_50](/configs/fashion_2d_keypoint/topdown_heatmap/deepfashion2/td-hm_res50_1xb64-210e_deepfasion2-skirt-256x192.py) |  256x192   |  0.952  | 0.653 | 16.6 | [ckpt](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_skirt_256x192-09573469_20221208.pth) | [log](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_skirt_256x192_20221208.log.json) |

| short_sleeved_dress   | [pose_resnet_50](/configs/fashion_2d_keypoint/topdown_heatmap/deepfashion2/td-hm_res50_4xb64-210e_deepfasion2-short-sleeved-dress-256x192.py) |  256x192   |  0.980  | 0.603 | 15.6 | [ckpt](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_short_sleeved_dress_256x192-1345b07a_20221208.pth) | [log](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_short_sleeved_dress_256x192_20221208.log.json) |
| long_sleeved_dress    | [pose_resnet_50](/configs/fashion_2d_keypoint/topdown_heatmap/deepfashion2/td-hm_res50_1xb64-210e_deepfasion2-long-sleeved-dress-256x192.py) |  256x192   |  0.976  | 0.518 | 20.1 | [ckpt](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_long_sleeved_dress_256x192-87bac74e_20221208.pth) | [log](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_long_sleeved_dress_256x192_20221208.log.json) |

| vest_dress            | [pose_resnet_50](/configs/fashion_2d_keypoint/topdown_heatmap/deepfashion2/td-hm_res50_1xb64-210e_deepfasion2-vest-dress-256x192.py) |  256x192   |  0.980  | 0.600 | 16.0 | [ckpt](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_vest_dress_256x192-fb3fbd6f_20221208.pth) | [log](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_vest_dress_256x192_20221208.log.json) |
| sling_dress           | [pose_resnet_50](/configs/fashion_2d_keypoint/topdown_heatmap/deepfashion2/td-hm_res50_4xb64-210e_deepfasion2-sling-dress-256x192.py) |  256x192   |  0.967  | 0.544 | 19.5 | [ckpt](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_sling_dress_256x192-8ebae0eb_20221208.pth) | [log](https://download.openmmlab.com/mmpose/fashion/resnet/res50_deepfashion2_sling_dress_256x192_20221208.log.json) |

# 自定义运行设定

## 实现自定义钩子

### Step 1: 创建一个新的钩子

MMEngine 已实现了训练和测试常用的[钩子](https://github.com/open-mmlab/mmengine/blob/main/docs/zh_cn/tutorials/hook.md),
当有定制化需求时, 可以按照如下示例实现适用于自身训练需求的钩子, 例如想修改一个超参数 `model.hyper_paramete` 的值, 让它随着训练迭代次数而变化:

```python

    """

        self.a = a
        self.b = b

                          runner,
        # 当模型被包在 wrapper 里时获取这个模型

cur_iter = runner.iter
```

### Step 2: 导入一个新的钩子

为了让上面定义的模块可以被执行的程序发现, 这个模块需要先被导入主命名空间 (main namespace) 里面,
假设 NewHook 在 `mmseg/engine/hooks/new_hook.py` 里面, 有两种方式去实现它:

- 修改 `mmseg/engine/hooks/__init__.py` 来导入它.
  新定义的模块应该在 `mmseg/engine/hooks/__init__.py` 里面导入, 这样注册器可以发现并添加这个新的模块:

```python

```

- 在配置文件里使用 custom_imports 来手动导入它.

```python
```

### Step 3: 修改配置文件

可以按照如下方式, 在训练或测试中配置并使用自定义的钩子. 不同钩子在同一位点的优先级可以参考[这里](https://github.com/open-mmlab/mmengine/blob/main/docs/zh_cn/tutorials/hook.md#%E5%86%85%E7%BD%AE%E9%92%A9%E5%AD%90), 自定义钩子如果没有指定优先, 默认是 `NORMAL`.

```python
]

```python
]
```

## 实现自定义优化器

### Step 1: 创建一个新的优化器

如果增加一个叫作 `MyOptimizer` 的优化器, 它有参数 `a`, `b` 和 `c`. 推荐在 `mmseg/engine/optimizers/my_optimizer.py` 文件中实现

```python

```

### Step 2: 导入一个新的优化器

假设 `MyOptimizer` 在 `mmseg/engine/optimizers/my_optimizer.py` 里面, 有两种方式去实现它:

- 修改 `mmseg/engine/optimizers/__init__.py` 来导入它.
  新定义的模块应该在 `mmseg/engine/optimizers/__init__.py` 里面导入, 这样注册器可以发现并添加这个新的模块:

```python
```

- 在配置文件里使用 `custom_imports` 来手动导入它.

```python
```

```

```python
```

随后需要修改配置文件 `optim_wrapper` 里的 `optimizer` 参数, 如果要使用你自己的优化器 `MyOptimizer`, 字段可以被修改成:

```python
```

## 实现自定义优化器封装构造器

### Step 1: 创建一个新的优化器封装构造器

构造器可以用来创建优化器, 优化器包, 以及自定义模型网络不同层的超参数. 一些模型的优化器可能会根据特定的参数而调整, 例如 BatchNorm 层的 weight decay. 使用者可以通过自定义优化器构造器来精细化设定不同参数的优化策略.

```python

```

默认的优化器构造器在[这里](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/optimizer/default_constructor.py#L19) 被实现, 它也可以用来作为新的优化器构造器的模板.

### Step 2: 导入一个新的优化器封装构造器

为了让上面定义的模块可以被执行的程序发现, 这个模块需要先被导入主命名空间 (main namespace) 里面, 假设 `MyOptimizerConstructor` 在 `mmseg/engine/optimizers/my_optimizer_constructor.py` 里面, 有两种方式去实现它:

```python
```

```python
```

```

```python
```

随后需要修改配置文件 `optim_wrapper`  里的 `constructor` 参数, 如果要使用你自己的优化器封装构造器 `MyOptimizerConstructor`, 字段可以被修改成:

```python
```

<details>
<summary align="right"><a href="https://arxiv.org/abs/1801.07372v2">DSNT (2018)</a></summary>

```bibtex
@article{nibali2018numerical,
  title={Numerical Coordinate Regression with Convolutional Neural Networks},
  author={Nibali, Aiden and He, Zhen and Morgan, Stuart and Prendergast, Luke},
  journal={arXiv preprint arXiv:1801.07372},
}
```

<details>

```bibtex
}
```

<details>

| [ipr_resnet_50_dsnt](/configs/body_2d_keypoint/integral_regression/coco/ipr_res50_dsnt-8xb64-210e_coco-256x256.py) |  256x256   | 0.674 |      0.870      |      0.744      | 0.764 |      0.928      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/integral_regression/coco/ipr_res50_dsnt-8xb64-210e_coco-256x256-441eedc0_20220913.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/integral_regression/coco/ipr_res50_dsnt-8xb64-210e_coco-256x256-441eedc0_20220913.log.json) |

The evaluation procedure would be executed at [ValLoop](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L300) and [TestLoop](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L373), users can evaluate model performance during training or using the test script with simple settings in the configuration file. The `ValLoop` and `TestLoop` are properties of [Runner](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/runner.py#L59), they will be built the first time they are called. To build the `ValLoop` successfully, the `val_dataloader` and `val_evaluator` must be set when building `Runner` since `dataloader` and `evaluator` are required parameters, and the same goes for `TestLoop`. For more information about the Runner's design, please refer to the [documentation](https://github.com/open-mmlab/mmengine/blob/main/docs/en/design/runner.md) of [MMEngine](https://github.com/open-mmlab/mmengine).

![test_step/val_step dataflow](https://user-images.githubusercontent.com/15952744/228828179-3269baa3-bebd-4c9a-9787-59e7d785fbcf.png)

In MMSegmentation, we write the settings of dataloader and metrics in the config files of datasets and the configuration of the evaluation loop in the `schedule_x` config files by default.

For example, in the ADE20K config file `configs/_base_/dataset/ade20k.py`, on lines 37 to 48, we configured the `val_dataloader`, on line 51, we select `IoUMetric` as the evaluator and set `mIoU` as the metric:

```python

```

img_path='images/validation',

```

To be able to evaluate the model during training, for example, we add the evaluation configuration to the file `configs/schedules/schedule_40k.py` on lines 15 to 16:

```python
```

With the above two settings, MMSegmentation evaluates the **mIoU** metric of the model once every 4000 iterations during the training of 40K iterations.

If we would like to test the model after training, we need to add the `test_dataloader`, `test_evaluator` and `test_cfg` configs to the config file.

```python

```python

```

In MMSegmentation, the settings of `test_dataloader` and `test_evaluator` are the same as the `ValLoop`'s dataloader and evaluator by default, we can modify these settings to meet our needs.

MMSegmentation implements [IoUMetric](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/evaluation/metrics/iou_metric.py) and [CityscapesMetric](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/evaluation/metrics/citys_metric.py) for evaluating the performance of models, based on the [BaseMetric](https://github.com/open-mmlab/mmengine/blob/main/mmengine/evaluator/metric.py) provided by [MMEngine](https://github.com/open-mmlab/mmengine). Please refer to [the documentation](https://mmengine.readthedocs.io/en/latest/tutorials/evaluation.html) for more details about the unified evaluation interface.

Here we briefly describe the arguments and the two main methods of `IoUMetric`.

The constructor of `IoUMetric` has some additional parameters besides the base `collect_device` and `prefix`.

The arguments of the constructor:

- ignore_index (int) - Index that will be ignored in evaluation. Default: 255.

- iou_metrics (list\[str\] | str) - Metrics to be calculated, the options includes 'mIoU', 'mDice' and 'mFscore'.
- nan_to_num (int, optional) - If specified, NaN values will be replaced by the numbers defined by the user. Default: None.
- beta (int) - Determines the weight of recall in the combined score. Default: 1.
- collect_device (str) - Device name used for collecting results from different ranks during distributed training. Must be 'cpu' or 'gpu'. Defaults to 'cpu'.
- prefix (str, optional) - The prefix that will be added in the metric names to disambiguate homonymous metrics of different evaluators. If the prefix is not provided in the argument, self.default_prefix will be used instead. Defaults to None.

`IoUMetric` implements the IoU metric calculation, the core two methods of `IoUMetric` are `process` and `compute_metrics`.

- `process` method processes one batch of data and data_samples.
- `compute_metrics` method computes the metrics from processed results.

- data_batch (Any) - A batch of data from the dataloader.
- data_samples (Sequence\[dict\]) - A batch of outputs from the model.

Returns:

This method doesn't have returns since the processed results would be stored in `self.results`, which will be used to compute the metrics when all batches have been processed.

- results (list) - The processed results of each batch.

Returns:

- Dict\[str, float\] - The computed metrics. The keys are the names of the metrics, and the values are corresponding results. The key mainly includes **aAcc**, **mIoU**, **mAcc**, **mDice**, **mFscore**, **mPrecision**, **mRecall**.

`CityscapesMetric` uses the official [CityscapesScripts](https://github.com/mcordts/cityscapesScripts) provided by Cityscapes to evaluate model performance.

### Usage

Before using it, please install the `cityscapesscripts` package first:

```shell
```

Since the `IoUMetric` is used as the default evaluator in MMSegmentation, if you would like to use `CityscapesMetric`, customizing the config file is required. In your customized config file, you should overwrite the default evaluator as follows.

```python
```

### Interface

- output_dir (str) - The directory for output prediction

```

- format_only (bool) - Only format result for results commit without perform evaluation. It is useful when you want to format the result to a specific format and submit it to the test server. Defaults to False.
- keep_results (bool) - Whether to keep the results. When `format_only` is True, `keep_results` must be True. Defaults to False.
- prefix (str, optional) - The prefix that will be added in the metric names to disambiguate homonymous metrics of different evaluators. If prefix is not provided in the argument, self.default_prefix will be used instead. Defaults to None.

This method would draw the masks on images and save the painted images to `work_dir`.

- data_batch (dict) - A batch of data from the dataloader.

Returns:

This method doesn't have returns, the annotations' path would be stored in `self.results`, which will be used to compute the metrics when all batches have been processed.

This method would call `cityscapesscripts.evaluation.evalPixelLevelSemanticLabeling` tool to calculate metrics.

- results (list) - Testing results of the dataset.

Returns:

- dict\[str: float\] - Cityscapes evaluation results.

# Preparing HACS Segments

```BibTeX
@inproceedings{zhao2019hacs,
  title={Hacs: Human action clips and segments dataset for recognition and temporal localization},
  author={Zhao, Hang and Torralba, Antonio and Torresani, Lorenzo and Yan, Zhicheng},
  pages={8668--8678},
}
```

### Step 0. Download Videos

Before we start preparing the dataset, please following the official [repository](https://github.com/hangzhaomit/HACS-dataset) to download videos from the HACS Segments dataset. You can submit a request for missing videos to the maintainer of the HACS dataset repository. But you can still prepare the dataset for MMAction2 if some videos are missing.

After you finish downloading the dataset, please move the dataset folder to `$MMACTION2/tools/data/hacs/` or use a soft link. The the folder structure should look like:

```
mmaction2

```
mmaction2
├── data
├── tools
│   ├── hacs
│   │   ├── slowonly_feature_infer.py
│   │   ├── ..
│   │   ├── data
│   │   │   ├── Applying_sunscreen
│   │   │   │   ├── v_0Ch__DqMPwA.mp4
│   │   │   │   ├── v_9CTDjFHl8WE.mp4
│   │   │   │   ├── ..

```

Before we start, make sure you are at `$MMACTION2/tools/data/hacs/`.

### Step 1. Extract Features

We extract features from the HACS videos using [SlowOnly ResNet50 8x8](/configs/recognition/slowonly/slowonly_imagenet-pretrained-r50_16xb16-4x16x1-steplr-150e_kinetics700-rgb.py) pretrained on Kinetics700 dataset. For each video, we uniformly sample 100 video clips and extract the 700-dimensional output (before softmax) as the feature, i.e., the feature shape is 100x700.

First, we generate a video list of the dataset:

```

```
python generate_list.py
```

It will generate an `hacs_data.txt` file located at `$MMACTION2/tools/data/hacs/` which looks like:

```
Horseback_riding/v_Sr2BSq_8FMw.mp4 0
Horseback_riding/v_EQb6OKoqz3Q.mp4 1
Horseback_riding/v_vYKUV8TRngg.mp4 2
Horseback_riding/v_Y8U0X1F-0ck.mp4 3
Horseback_riding/v_hnspbB7wNh0.mp4 4
Horseback_riding/v_HPhlhrT9IOk.mp4 5
```

Next we use the [slowonly_feature_infer.py](/tools/data/hacs/slowonly_feature_infer.py) config to extract features:

```
# number of GPUs to extract feature
NUM_GPUS=8

# download the pretraining checkpoint
wget https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_16xb16-8x8x1-steplr-150e_kinetics700-rgb/slowonly_imagenet-pretrained-r50_16xb16-8x8x1-steplr-150e_kinetics700-rgb_20221013-15b93b10.pth

bash ../mmaction2/tools/dist_test.sh \
    slowonly_feature_infer.py \

    slowonly_imagenet-pretrained-r50_16xb16-8x8x1-steplr-150e_kinetics700-rgb_20221013-15b93b10.pth \
    $NUM_GPUS --dump result.pkl
```

We will get a `result.pkl` that contains the 100x700 feature for each video. We re-write the features into csv format at `$MMACTION2/data/HACS/`:

```
# Make sure you are at $MMACTION2/tools/data/hacs/
python write_feature_csv.py
```

### Step 2. Prepare Annotations

We first download the original annotations from the official repository:

```
wget https://github.com/hangzhaomit/HACS-dataset/raw/master/HACS_v1.1.1.zip
unzip HACS_v1.1.1.zip
```

After unzipping, there should be an `HACS_v1.1.1` folder with an `HACS_segments_v1.1.1.json` file in it.

We generate `hacs_anno_train.json`,  `hacs_anno_val.json` and `hacs_anno_test.json` files at `$MMACTION2/data/HACS/`:

```
python3 generate_anotations.py
```

```
```

After the two steps finished, the folder structure of the HACS Segments dataset should look like:

```
mmaction2
├── data
│   ├── HACS
│   │   ├── hacs_anno_train.json
│   │   ├── hacs_anno_val.json
│   │   ├── hacs_anno_test.json
│   │   ├── slowonly_feature
│   │   │   ├── v_008gY2B8Pf4.csv
│   │   │   ├── v_0095rqic1n8.csv
├── tools

```

# RealBasicVSR (CVPR'2022)

> [RealBasicVSR: Investigating Tradeoffs in Real-World Video Super-Resolution](https://arxiv.org/abs/2111.12704)

The diversity and complexity of degradations in real-world video super-resolution (VSR) pose non-trivial challenges in inference and training. First, while long-term propagation leads to improved performance in cases of mild degradations, severe in-the-wild degradations could be exaggerated through propagation, impairing output quality. To balance the tradeoff between detail synthesis and artifact suppression, we found an image pre-cleaning stage indispensable to reduce noises and artifacts prior to propagation. Equipped with a carefully designed cleaning module, our RealBasicVSR outperforms existing methods in both quality and efficiency. Second, real-world VSR models are often trained with diverse degradations to improve generalizability, requiring increased batch size to produce a stable gradient. Inevitably, the increased computational burden results in various problems, including 1) speed-performance tradeoff and 2) batch-length tradeoff. To alleviate the first tradeoff, we propose a stochastic degradation scheme that reduces up to 40% of training time without sacrificing performance. We then analyze different training settings and suggest that employing longer sequences rather than larger batches during training allows more effective uses of temporal information, leading to more stable performance during inference. To facilitate fair comparisons, we propose the new VideoLQ dataset, which contains a large variety of real-world low-quality video sequences containing rich textures and patterns. Our dataset can serve as a common ground for benchmarking. Code, models, and the dataset will be made publicly available.

 <img src="https://user-images.githubusercontent.com/7676947/146704029-58bc4db4-267f-4158-8129-e49ab6652249.png" width="400"/>
</div >

Evaluated on Y channel. The code for computing NRQM, NIQE, and PI can be found [here](https://github.com/roimehrez/PIRM2018). MATLAB official code is used to compute BRISQUE.

|                               Model                                | Dataset | NRQM (Y) | NIQE (Y) | PI (Y) | BRISQUE (Y) |    Training Resources    |                                Download                                |
| :----------------------------------------------------------------: | :-----: | :------: | :------: | :----: | :---------: | :----------------------: | :--------------------------------------------------------------------: |

| [realbasicvsr_c64b20_1x30x8_lr5e-5_150k_reds](./realbasicvsr_c64b20-1x30x8_8xb1-lr5e-5-150k_reds.py) |  REDS   |  6.0477  |  3.7662  | 3.8593 |   29.030    | 8 (Tesla V100-SXM2-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/real_basicvsr/realbasicvsr_c64b20_1x30x8_lr5e-5_150k_reds_20211104-52f77c2c.pth)/[log](https://download.openmmlab.com/mmediting/restorers/real_basicvsr/realbasicvsr_c64b20_1x30x8_lr5e-5_150k_reds_20211104_183640.log.json) |
| [realbasicvsr_wogan-c64b20-2x30x8_8xb2-lr1e-4-300k_reds](./realbasicvsr_wogan-c64b20-2x30x8_8xb2-lr1e-4-300k_reds.py) |  REDS   |    -     |    -     |   -    |      -      | 8 (Tesla V100-SXM2-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/real_basicvsr/realbasicvsr_wogan_c64b20_2x30x8_lr1e-4_300k_reds_20211027-0e2ff207.pth)/[log](http://download.openmmlab.com/mmediting/restorers/real_basicvsr/realbasicvsr_wogan_c64b20_2x30x8_lr1e-4_300k_reds_20211027_114039.log.json) |

**Train**

**Train**

<details>

```shell
# cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/real_basicvsr/realbasicvsr_c64b20-1x30x8_8xb1-lr5e-5-150k_reds.py

python tools/train.py configs/real_basicvsr/realbasicvsr_c64b20-1x30x8_8xb1-lr5e-5-150k_reds.py

./tools/dist_train.sh configs/real_basicvsr/realbasicvsr_c64b20-1x30x8_8xb1-lr5e-5-150k_reds.py 8
```

**Test**

<details>

```shell
# cpu test

**Test**

<details>

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/real_basicvsr/realbasicvsr_c64b20-1x30x8_8xb1-lr5e-5-150k_reds.py https://download.openmmlab.com/mmediting/restorers/real_basicvsr/realbasicvsr_c64b20_1x30x8_lr5e-5_150k_reds_20211104-52f77c2c.pth

python tools/test.py python tools/test.py configs/real_basicvsr/realbasicvsr_c64b20-1x30x8_8xb1-lr5e-5-150k_reds.py https://download.openmmlab.com/mmediting/restorers/real_basicvsr/realbasicvsr_c64b20_1x30x8_lr5e-5_150k_reds_20211104-52f77c2c.pth

./tools/dist_test.sh configs/real_basicvsr/realbasicvsr_c64b20-1x30x8_8xb1-lr5e-5-150k_reds.py https://download.openmmlab.com/mmediting/restorers/real_basicvsr/realbasicvsr_c64b20_1x30x8_lr5e-5_150k_reds_20211104-52f77c2c.pth 8
```

```

```bibtex
@InProceedings{chan2022investigating,
  title = {RealBasicVSR: Investigating Tradeoffs in Real-World Video Super-Resolution},
}
```

# Oriented RepPoints

> [Oriented RepPoints for Aerial Object Detection](https://openaccess.thecvf.com/content/CVPR2022/papers/Li_Oriented_RepPoints_for_Aerial_Object_Detection_CVPR_2022_paper.pdf)

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/oriented_reppoints.png" width="800"/>
</div>

In contrast to the generic object, aerial targets are often non-axis aligned with arbitrary orientations having
the cluttered surroundings. Unlike the mainstreamed approaches regressing the bounding box orientations, this paper
proposes an effective adaptive points learning approach to aerial object detection by taking advantage of the adaptive
points representation, which is able to capture the geometric information of the arbitrary-oriented instances.
To this end, three oriented conversion functions are presented to facilitate the classification and localization

with accurate orientation. Moreover, we propose an effective quality assessment and sample assignment scheme for
adaptive points learning toward choosing the representative oriented reppoints samples during training, which is
able to capture the non-axis aligned features from adjacent objects or background noises. A spatial constraint is
introduced to penalize the outlier points for roust adaptive learning. Experimental results on four challenging
aerial datasets including DOTA, HRSC2016, UCAS-AOD and DIOR-R, demonstrate the efficacy of our proposed approach.

DOTA1.0

DOTA1.0

|         Backbone         |  mAP  | Angle | lr schd | Mem (GB) | Inf Time (fps) | Aug | Batch Size |                                                  Configs                                                  |                                                                                                                                                                                    Download                                                                                                                                                                                    |

| ResNet50 (1024,1024,200) | 71.94 | le135 |   1x    |   3.45   |      16.1      |  -  |     2      |          [oriented-reppoints-qbox_r50_fpn_1x_dota](./oriented-reppoints-qbox_r50_fpn_1x_dota.py)          |         [model](https://download.openmmlab.com/mmrotate/v0.1.0/orientedreppoints/oriented_reppoints_r50_fpn_1x_dota_le135/oriented_reppoints_r50_fpn_1x_dota_le135-ef072de9.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/orientedreppoints/oriented_reppoints_r50_fpn_1x_dota_le135/oriented_reppoints_r50_fpn_1x_dota_le135_20220505_092853.log.json)         |

| ResNet50 (1024,1024,200) | 75.21 | le135 |   40e   |   3.45   |      16.1      | ms  |     2      | [oriented-reppoints-qbox_r50_fpn_mstrain-40e_dota](./oriented-reppoints-qbox_r50_fpn_mstrain-40e_dota.py) | [model](https://download.openmmlab.com/mmrotate/v0.1.0/orientedreppoints/oriented_reppoints_r50_fpn_40e_dota_ms_le135/oriented_reppoints_r50_fpn_40e_dota_ms_le135-bb0323fd.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/orientedreppoints/oriented_reppoints_r50_fpn_40e_dota_ms_le135/oriented_reppoints_r50_fpn_40e_dota_ms_le135_20220516_145332.log.json) |

- Oriented RepPoints needs to install [MMCV](https://github.com/open-mmlab/mmcv) >= 1.5.3.
- `ms` means multi-scale image split online (768, 1280).

```
@inproceedings{li2022ori,
    title={Oriented RepPoints for Aerial Object Detection},
    author={Wentong Li, Yijie Chen, Kaixuan Hu, Jianke Zhu},

```
    booktitle={Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
}
```

# 2D Hand Keypoint Datasets

- [OneHand10K](#onehand10k) \[ [Homepage](https://www.yangangwang.com/papers/WANG-MCC-2018-10.html) \]
- [FreiHand](#freihand-dataset) \[ [Homepage](https://lmb.informatik.uni-freiburg.de/projects/freihand/) \]
- [CMU Panoptic HandDB](#cmu-panoptic-handdb) \[ [Homepage](http://domedb.perception.cs.cmu.edu/handdb.html) \]
- [RHD](#rhd-dataset) \[ [Homepage](https://lmb.informatik.uni-freiburg.de/resources/datasets/RenderedHandposeDataset.en.html) \]
- [COCO-WholeBody-Hand](#coco-wholebody-hand) \[ [Homepage](https://github.com/jin-s13/COCO-WholeBody/) \]

## OneHand10K

<details>

<details>

```bibtex
}
```

  <img src="https://user-images.githubusercontent.com/100993824/227771101-03a27bd8-ccc0-4eb9-a111-660f191a7a16.png" height="200px">
</div>

For [OneHand10K](https://www.yangangwang.com/papers/WANG-MCC-2018-10.html) data, please download from [OneHand10K Dataset](https://www.yangangwang.com/papers/WANG-MCC-2018-10.html).

</div>

Please download the annotation files from [onehand10k_annotations](https://download.openmmlab.com/mmpose/datasets/onehand10k_annotations.tar).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── onehand10k
        |   |── onehand10k_train.json
        |   |── onehand10k_test.json
        `── Train
        |   |── source
        |       |── 0.jpg
        |       |── 1.jpg
        |        ...
        `── Test
            |── source
                |── 0.jpg
                |── 1.jpg

```

## FreiHAND Dataset

<details>

|       |── 1.jpg
        |        ...
        `── Test
            |── source
                |── 0.jpg
                |── 1.jpg

```

<details>

```bibtex
}
```

</div>

}
```

</div>

For [FreiHAND](https://lmb.informatik.uni-freiburg.de/projects/freihand/) data, please download from [FreiHand Dataset](https://lmb.informatik.uni-freiburg.de/resources/datasets/FreihandDataset.en.html).
Since the official dataset does not provide validation set, we randomly split the training data into 8:1:1 for train/val/test.
Please download the annotation files from [freihand_annotations](https://download.openmmlab.com/mmpose/datasets/frei_annotations.tar).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── freihand
        |   |── freihand_train.json
        |   |── freihand_val.json
        |   |── freihand_test.json
        `── training
            |── rgb

`── data
            |── rgb
            |   |── 00000000.jpg
            |   |── 00000001.jpg
            |    ...
            |── mask
                |── 00000000.jpg
                |── 00000001.jpg
                 ...
```

## CMU Panoptic HandDB

<details>

```bibtex
}
```

}
```

</div>

For [CMU Panoptic HandDB](http://domedb.perception.cs.cmu.edu/handdb.html), please download from [CMU Panoptic HandDB](http://domedb.perception.cs.cmu.edu/handdb.html).
Following [Simon et al](https://arxiv.org/abs/1704.07809), panoptic images (hand143_panopticdb) and MPII & NZSL training sets (manual_train) are used for training, while MPII & NZSL test set (manual_test) for testing.
Please download the annotation files from [panoptic_annotations](https://download.openmmlab.com/mmpose/datasets/panoptic_annotations.tar).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── panoptic

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
        |   |── panoptic_train.json
        |   |── panoptic_test.json
        |
        `── hand143_panopticdb
        |   |── imgs
        |   |   |── 00000000.jpg
        |   |   |── 00000001.jpg
        |   |    ...
        |
        `── hand_labels
            |── manual_train
            |   |── 000015774_01_l.jpg
            |   |── 000015774_01_r.jpg
            |    ...
            |
            `── manual_test
                |── 000648952_02_l.jpg
                |── 000835470_01_l.jpg
                 ...
```

<details>

```bibtex

```bibtex
}
```

  <img src="https://user-images.githubusercontent.com/100993824/227771753-5df1d722-59bd-4815-b85f-64a5ef79bbf5.png" height="200px">
</div>

Please download the annotation files from [annotations](https://download.openmmlab.com/mmpose/datasets/interhand2.6m_annotations.zip).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
        |   |── all
        |
        `── images
        |   |── train
        |   |── val
        |   |── test
```

## RHD Dataset

<details>

```bibtex

```bibtex
}
```

  <img src="https://user-images.githubusercontent.com/100993824/227772014-f7406a2b-2e64-42fb-8081-200d40104553.png" height="200px">
</div>

For [RHD Dataset](https://lmb.informatik.uni-freiburg.de/resources/datasets/RenderedHandposeDataset.en.html), please download from [RHD Dataset](https://lmb.informatik.uni-freiburg.de/resources/datasets/RenderedHandposeDataset.en.html).
Please download the annotation files from [rhd_annotations](https://download.openmmlab.com/mmpose/datasets/rhd_annotations.zip).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── rhd
        |   |── rhd_train.json
        |   |── rhd_test.json
        |   |── color
        |   |   |── 00000.jpg
        |   |   |── 00001.jpg
        |   |── depth
        |   |── mask
        `── evaluation
        |   |── color
        |   |── depth
        |   |── mask
```

## COCO-WholeBody (Hand)

<details>

```bibtex

<details>

```bibtex
}
```

</div>

For [COCO-WholeBody](https://github.com/jin-s13/COCO-WholeBody/) dataset, images can be downloaded from [COCO download](http://cocodataset.org/#download), 2017 Train/Val is needed for COCO keypoints training and validation.

Download COCO-WholeBody annotations for COCO-WholeBody annotations for [Train](https://drive.google.com/file/d/1thErEToRbmM9uLNi1JXXfOsaS5VK2FXf/view?usp=sharing) / [Validation](https://drive.google.com/file/d/1N6VgwKnj8DeyGXCvp1eYgNbRmw6jdfrb/view?usp=sharing) (Google Drive).
Download person detection result of COCO val2017 from [OneDrive](https://1drv.ms/f/s!AhIXJn_J-blWzzDXoz5BeFl8sWM-) or [GoogleDrive](https://drive.google.com/drive/folders/1fRUDNUDxe9fjqcRZ2bnF_TKMlO0nB_dk?usp=sharing).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── coco
        │   │-- coco_wholebody_train_v1.0.json
        │   |-- coco_wholebody_val_v1.0.json

|   |-- COCO_val2017_detections_AP_H_56_person.json
        │   │-- ...
            │-- ...
```

Please also install the latest version of [Extended COCO API](https://github.com/jin-s13/xtcocoapi) to support COCO-WholeBody evaluation:

`pip install xtcocotools`

# Rotated ATSS

> [Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection](https://arxiv.org/abs/1912.02424)

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/atss.jpg" width="800"/>
</div>

Object detection has been dominated by anchor-based detectors for several years. Recently, anchor-free detectors have become popular due to the proposal of FPN and Focal Loss. In this paper, we first point out that the essential difference between anchor-based and anchor-free detection is actually how to define positive and negative training samples, which leads to the performance gap between them. If they adopt the same definition of positive and negative samples during training, there is no obvious difference in the final performance, no matter regressing from a box or a point. This shows that how to select positive and negative training samples is important for current object detectors. Then, we propose an Adaptive Training Sample Selection (ATSS) to automatically select positive and negative samples according to statistical characteristics of object. It significantly improves the performance of anchor-based and anchor-free detectors and bridges the gap between them. Finally, we discuss the necessity of tiling multiple anchors per location on the image to detect objects. Extensive experiments conducted on MS COCO support our aforementioned analysis and conclusions. With the newly introduced ATSS, we improve state-of-the-art detectors by a large margin to 50.7% AP without introducing any overhead.

DOTA1.0

Notes:

- `hbb` means the input of the assigner is the predicted box and the horizontal box that can surround the GT. `obb` means the input of the assigner is the predicted box and the GT. They can be switched by `assign_by_circumhbbox`  in `RotatedRetinaHead`.

```
@inproceedings{zhang2020bridging,
  title={Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection},
  author={Zhang, Shifeng and Chi, Cheng and Yao, Yongqiang and Lei, Zhen and Li, Stan Z},
  pages={9759--9768},
}
```

# 推理

在 OpenMMLab 中，所有的推理操作都被统一到了推理器 `Inferencer` 中。推理器被设计成为一个简洁易用的 API，它在不同的 OpenMMLab 库中都有着非常相似的接口。

MMOCR 中存在两种不同的推理器：

- **标准推理器**：MMOCR 中的每个基本任务都有一个标准推理器，即 `TextDetInferencer`（文本检测），`TextRecInferencer`（文本识别），`TextSpottingInferencer`（端到端 OCR） 和 `KIEInferencer`（关键信息提取）。它们具有非常相似的接口，具有标准的输入/输出协议，并且总体遵循 OpenMMLab 的设计。这些推理器也可以被串联在一起，以便对一系列任务进行推理。
- **MMOCRInferencer**：我们还提供了 `MMOCRInferencer`，一个专门为 MMOCR 设计的便捷推理接口。它封装和链接了 MMOCR 中的所有推理器，因此用户可以使用此推理器对图像执行一系列任务，并直接获得最终结果。*但是，它的接口与标准推理器有一些不同，并且为了简单起见，可能会牺牲一些标准的推理器功能。*

对于新用户，我们建议使用 **MMOCRInferencer** 来测试不同模型的组合。

如果你是开发人员并希望将模型集成到自己的项目中，我们建议使用**标准推理器**，因为它们更灵活且标准化，并具有完整的功能。

## 基础用法

目前，`MMOCRInferencer` 可以对以下任务进行推理：

- 文本检测
- 文本识别
- OCR（文本检测 + 文本识别）
- 关键信息提取（文本检测 + 文本识别 + 关键信息提取）
- *OCR（text spotting）*（即将推出）

为了便于使用，`MMOCRInferencer` 向用户提供了 Python 接口和命令行接口。例如，如果你想要对 demo/demo_text_ocr.jpg 进行 OCR 推理，使用 `DBNet` 作为文本检测模型，`CRNN` 作为文本识别模型，只需执行以下命令:

>>> # 读取模型
>>> # 进行推理并可视化结果
:::

:::{code-tab} bash 命令行
:::
::::

可视化结果将被显示在一个新窗口中：

</div>

```{note}
如果你在没有 GUI 的服务器上运行 MMOCR，或者是通过禁用 X11 转发的 SSH 隧道运行该指令，`show`  选项将不起作用。然而，你仍然可以通过设置 `out_dir` 和 `save_vis=True` 参数将可视化数据保存到文件。阅读 [储存结果](#储存结果) 了解详情。
```

根据初始化参数，`MMOCRInferencer`可以在不同模式下运行。例如，如果初始化时指定了 `det`、`rec` 和 `kie`，它可以在 KIE 模式下运行。

```

:::

:::

::::

可视化结果如下：

</div>
<br />

可以见到，MMOCRInferencer 的 Python 接口与命令行接口的使用方法非常相似。下文将以 Python 接口为例，介绍 MMOCRInferencer 的具体用法。关于命令行接口的更多信息，请参考 [命令行接口](#命令行接口)。

````

````{group-tab} 标准推理器

通常，OpenMMLab 中的所有标准推理器都具有非常相似的接口。下面的例子展示了如何使用 `TextDetInferencer` 对单个图像进行推理。

```python
>>> # 读取模型
>>> # 推理
```

可视化结果如图：

>>> # 读取模型
>>> # 推理
```

可视化结果如图：

</div>

````

`````

## 初始化

每个推理器必须使用一个模型进行初始化。初始化时，可以手动选择推理设备。

### 模型初始化

对于每个任务，`MMOCRInferencer` 需要两个参数 `xxx` 和 `xxx_weights` （例如 `det` 和 `det_weights`）以对模型进行初始化。此处将以`det`和`det_weights`为例来说明一些典型的初始化模型的方法。

- 要用 MMOCR 的预训练模型进行推理，只需要把它的名字传给参数 `det`，权重将自动从 OpenMMLab 的模型库中下载和加载。[此处](../modelzoo.md#权重)记录了 MMOCR 中可以通过该方法初始化的所有模型。

  ```python
  ```

- 要加载自定义的配置和权重，你可以把配置文件的路径传给 `det`，把权重的路径传给 `det_weights`。

  ```python
  ```

如果需要查看更多的初始化方法，请点击“标准推理器”选项卡。

````

```python
  ```

````

每个标准的 `Inferencer` 都接受两个参数，`model` 和 `weights` 。在 MMOCRInferencer 中，这两个参数分别对应 `xxx` 和 `xxx_weights` （例如 `det` 和 `det_weights`）。

- `model` 接受模型的名称或配置文件的路径作为输入。模型的名称从 [model-index.yml](https://github.com/open-mmlab/mmocr/blob/1.x/model-index.yml) 中的模型的元文件（[示例](https://github.com/open-mmlab/mmocr/blob/1.x/configs/textdet/dbnet/metafile.yml) ）中获取。你可以在[此处](../modelzoo.md#权重)找到可用权重的列表。

- `weights` 接受权重文件的路径。

<br />

此处列举了一些常见的初始化模型的方法。

- 你可以通过传递模型的名称给 `model` 来推理 MMOCR 的预训练模型。权重将会自动从 OpenMMLab 的模型库中下载并加载。

  ```python
  ```

  ```{note}
  模型与推理器的任务种类必须匹配。
  ```

  你可以通过将权重的路径或 URL 传递给 `weights` 来让推理器加载自定义的权重。

  ```python
  ```

```

  ```{note}
  ```

  ```python
  ```

- 如果有自定义的配置和权重，你可以将配置文件的路径传递给 `model`，将权重的路径传递给 `weights`。

  ```python
  ```

- 默认情况下，[MMEngine](https://github.com/open-mmlab/mmengine/) 会在训练模型时自动将配置文件转储到权重文件中。如果你有一个在 MMEngine 上训练的权重，你也可以将权重文件的路径传递给 `weights`，而不需要指定 `model`：

  ```python
  >>> # 如果无法在权重中找到配置文件，则会引发错误
  ```

- 传递配置文件到 `model` 而不指定 `weight` 则会产生一个随机初始化的模型。

````
`````

### 推理设备

每个推理器实例都会跟一个设备绑定。默认情况下，最佳设备是由 [MMEngine](https://github.com/open-mmlab/mmengine/) 自动决定的。你也可以通过指定 `device` 参数来改变设备。例如，你可以使用以下代码在 GPU 1上创建一个推理器。

```python
```

````

```python
```

````

```python
```

````

`````

如要在 CPU 上创建一个推理器：

```python
```

````

```python
```

````

`````

请参考 [torch.device](torch.device) 了解 `device` 参数支持的所有形式。

## 推理

当推理器初始化后，你可以直接传入要推理的原始数据，从返回值中获取推理结果。

### 输入

输入可以是以下任意一种格式：

- str: 图像的路径/URL。

  ```python
  ```

- array: 图像的 numpy 数组。它应该是 BGR 格式。

  ```python
  ```

>>> inferencer('demo/demo_text_ocr.jpg')
  ```

  ```python
  ```

- list: 基本类型的列表。列表中的每个元素都将单独处理。

  ```python
  >>> # 列表内混合类型也是允许的
  ```

- str: 目录的路径。目录中的所有图像都将被处理。

  ```python
  ```

````

输入可以是一个字典或者一个字典列表，其中每个字典包含以下键：

- `img` (str 或者 ndarray): 图像的路径或图像本身。如果 KIE 推理器在无可视模式下使用，则不需要此键。如果它是一个 numpy 数组，则应该是 BGR 顺序编码的图片。
- `img_shape` (tuple(int, int)): 图像的形状 (H, W)。仅在 KIE 推理器在无可视模式下使用且没有提供 `img` 时才需要。
- `instances` (list[dict]): 实例列表。

每个 `instance` 都应该包含以下键：

```python
{
    # 一个嵌套列表，其中包含 4 个数字，表示实例的边界框，顺序为 (x1, y1, x2, y2)

    # 文本列表
}
```

````
`````

    # 文本列表
}
```

````
`````

### 输出

默认情况下，每个推理器都以字典格式返回预测结果。

- `visualization` 包含可视化的预测结果。但默认情况下，它是一个空列表，除非 `return_vis=True`。

- `predictions` 包含以 json-可序列化格式返回的预测结果。如下所示，内容因任务类型而异。

  ```python
  {
        # 每个实例都对应于一个输入图像
        {
          'det_polygons': [...],  # 2d 列表，长度为 (N,)，格式为 [x1, y1, x2, y2, ...]
          'det_scores': [...],  # 浮点列表，长度为（N, ）
          'det_bboxes': [...],   # 2d 列表，形状为 (N, 4)，格式为 [min_x, min_y, max_x, max_y]
          'rec_texts': [...],  # 字符串列表，长度为（N, ）
          'rec_scores': [...],  # 浮点列表，长度为（N, ）
          'kie_labels': [...],  # 节点标签，长度为 (N, )
          'kie_scores': [...],  # 节点置信度，长度为 (N, )
          'kie_edge_scores': [...],  # 边预测置信度, 形状为 (N, N)
          'kie_edge_labels': [...]  # 边标签, 形状为 (N, N)
        },
        ...
      ],

'kie_scores': [...],  # 节点置信度，长度为 (N, )
        },
        ...
      ],
      ]
  }
  ```

  :::

  ````{group-tab} 标准推理器

  ::::{tabs}
  :::{code-tab} python TextDetInferencer

  {
        {
          'polygons': [...],  # 2d 列表，长度为 (N,)，格式为 [x1, y1, x2, y2, ...]
          'bboxes': [...],  # 2d 列表，形状为 (N, 4)，格式为 [min_x, min_y, max_x, max_y]
          'scores': [...]  # 浮点列表，长度为（N, ）
        },
        ...
      ]
      ]
  }
  :::

  :::{code-tab} python TextRecInferencer
  {
        {
          'text': '...',  # 字符串
          'scores': 0.1,  # 浮点
        },
        ...
      ]
      ]
  }

# 每个实例都对应于一个输入图像
        {
        },
        ...
      ]
      ]
  }
  :::

  :::{code-tab} python TextSpottingInferencer
  {
        {
          'texts': ['...',]  # 字符串列表，长度为（N, ）
        },
      ]
      ]
  }
  :::

  :::{code-tab} python KIEInferencer
  {
        {
          'labels': [...],  # 节点标签，长度为 (N, )
          'scores': [...],  # 节点置信度，长度为 (N, )
          'edge_scores': [...],  # 边预测置信度, 形状为 (N, N)
          'edge_labels': [...],  # 边标签, 形状为 (N, N)
        },
      ]

'scores': [...],  # 节点置信度，长度为 (N, )
        },
      ]
      ]
  }
  :::
  ::::
  ````

  `````

如果你想要从模型中获取原始输出，可以将 `return_datasamples` 设置为 `True` 来获取原始的 [DataSample](structures.md)，它将存储在 `predictions` 中。

### 储存结果

除了从返回值中获取预测结果，你还可以通过设置 `out_dir` 和 `save_pred`/`save_vis` 参数将预测结果和可视化结果导出到文件中。

```python
```

结果目录结构如下：

```text
outputs
├── preds
└── vis
```

文件名与对应的输入图像文件名相同。 如果输入图像是数组，则文件名将是从0开始的数字。

### 批量推理

你可以通过设置 `batch_size` 来自定义批量推理的批大小。 默认批大小为 1。

## API

这里列出了推理器详尽的参数列表。

### 批量推理

## API

| 参数          | 类型                                      | 默认值 | 描述                                                                                                                           |
| ------------- | ----------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------ |
| `det`         | str 或 [权重](../modelzoo.html#id2), 可选 | None   | 预训练的文本检测算法。它是配置文件的路径或者是 metafile 中定义的模型名称。                                                     |
| `det_weights` | str, 可选                                 | None   | det 模型的权重文件的路径。                                                                                                     |

| `rec`         | str 或 [权重](../modelzoo.html#id2), 可选 | None   | 预训练的文本识别算法。它是配置文件的路径或者是 metafile 中定义的模型名称。                                                     |
| `rec_weights` | str, 可选                                 | None   | rec 模型的权重文件的路径。                                                                                                     |
| `kie` \[1\]   | str 或 [权重](../modelzoo.html#id2), 可选 | None   | 预训练的关键信息提取算法。它是配置文件的路径或者是 metafile 中定义的模型名称。                                                 |
| `kie_weights` | str, 可选                                 | None   | kie 模型的权重文件的路径。                                                                                                     |

| `device`      | str, 可选                                 | None   | 推理使用的设备，接受 `torch.device` 允许的所有字符串。例如，'cuda:0' 或 'cpu'。如果为 None，将自动使用可用设备。 默认为 None。 |

\[1\]: 当同时指定了文本检测和识别模型时，`kie` 才会生效。

| 参数                 | 类型                    | 默认值     | 描述                                                                                           |
| -------------------- | ----------------------- | ---------- | ---------------------------------------------------------------------------------------------- |
| `inputs`             | str/list/tuple/np.array | **必需**   | 它可以是一个图片/文件夹的路径，一个 numpy 数组，或者是一个包含图片路径或 numpy 数组的列表/元组 |
| `return_datasamples` | bool                    | False      | 是否将结果作为 DataSample 返回。如果为 False，结果将被打包成一个字典。                         |

| `batch_size`         | int                     | 1          | 推理的批大小。                                                                                 |
| `det_batch_size`     | int, 可选               | None       | 推理的批大小 (文本检测模型)。如果不为 None，则覆盖 batch_size。                                    |
| `rec_batch_size`     | int, 可选               | None       | 推理的批大小 (文本识别模型)。如果不为 None，则覆盖 batch_size。                                    |
| `kie_batch_size`     | int, 可选               | None       | 推理的批大小 (关键信息提取模型)。如果不为 None，则覆盖 batch_size。                                |
| `return_vis`         | bool                    | False      | 是否返回可视化结果。                                                                           |

| `print_result`       | bool                    | False      | 是否将推理结果打印到控制台。                                                                   |
| `show`               | bool                    | False      | 是否在弹出窗口中显示可视化结果。                                                               |
| `wait_time`          | float                   | 0          | 弹窗展示可视化结果的时间间隔。                                                                 |
| `out_dir`            | str                     | `results/` | 结果的输出目录。                                                                               |
| `save_vis`           | bool                    | False      | 是否将可视化结果保存到 `out_dir`。                                                             |

| `save_pred`          | bool                    | False      | 是否将推理结果保存到 `out_dir`。                                                               |

```

```{group-tab} 标准推理器

| 参数      | 类型                                      | 默认值 | 描述                                                                                                                               |
| --------- | ----------------------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------- |
| `model`   | str 或 [权重](../modelzoo.html#id2), 可选 | None   | 路径到配置文件或者在 metafile 中定义的模型名称。                                                                                   |

| `weights` | str, 可选                                 | None   | 权重文件的路径。                                                                                                                   |
| `device`  | str, 可选                                 | None   | 推理使用的设备，接受 `torch.device` 允许的所有字符串。 例如，'cuda:0' 或 'cpu'。 如果为 None，则将自动使用可用设备。 默认为 None。 |

| 参数                 | 类型                    | 默认值     | 描述                                                                                |
| -------------------- | ----------------------- | ---------- | ----------------------------------------------------------------------------------- |
| `inputs`             | str/list/tuple/np.array | **必需**   | 可以是图像的路径/文件夹，np 数组或列表/元组（带有图像路径或 np 数组）               |

| `return_datasamples` | bool                    | False      | 是否将结果作为 DataSamples 返回。 如果为 False，则结果将被打包到一个 dict 中。      |
| `batch_size`         | int                     | 1          | 推理批大小。                                                                        |
| `progress_bar`       | bool                    | True       | 是否显示进度条。                                                                    |
| `return_vis`         | bool                    | False      | 是否返回可视化结果。                                                                |
| `print_result`       | bool                    | False      | 是否将推理结果打印到控制台。                                                        |
| `show`               | bool                    | False      | 是否在弹出窗口中显示可视化结果。                                                    |

| `wait_time`          | float                   | 0          | 弹窗展示可视化结果的时间间隔。                                                      |
| `draw_pred`          | bool                    | True       | 是否绘制预测的边界框。 *仅适用于 `TextDetInferencer` 和 `TextSpottingInferencer`。* |
| `out_dir`            | str                     | `results/` | 结果的输出目录。                                                                    |
| `save_vis`           | bool                    | False      | 是否将可视化结果保存到 `out_dir`。                                                  |
| `save_pred`          | bool                    | False      | 是否将推理结果保存到 `out_dir`。                                                    |

```
````

## 命令行接口

```{note}
该节仅适用于 `MMOCRInferencer`.
```

`MMOCRInferencer` 的命令行形式可以通过 `tools/infer.py` 调用，大致形式如下：

```bash

```
````

## 命令行接口

```{note}
```

```bash
```

其中，`INPUT_PATH` 为必须字段，内容应当为指向图片或文件目录的路径。其他参数与 Python 接口遵循的映射关系如下：

- 在命令行中调用参数时，需要在 Python 接口的参数前面加上两个`-`，然后把下划线`_`替换成连字符`-`。例如， `out_dir` 会变成 `--out-dir`。
- 对于布尔类型的参数，将参数放在命令中就相当于将其指定为 True。例如， `--show` 会将 `show` 参数指定为 True。

此外，命令行中默认不会回显推理结果，你可以通过 `--print-result` 参数来查看推理结果。

下面是一个例子：

```bash
```

运行该命令，可以得到如下结果：

```bash

```

# onnxruntime 支持情况

## ONNX Runtime 介绍

**ONNX Runtime** 是一个跨平台的推理和训练加速器，与许多流行的ML/DNN框架兼容。查看其[github](https://github.com/microsoft/onnxruntime)以获取更多信息。

## 安装

*请注意，目前Linux平台只支持 **onnxruntime>=1.8.1** 。*

### 安装ONNX Runtime python包

- CPU 版本

```bash
pip install onnxruntime==1.8.1 # 如果你想用cpu版本
```

- GPU 版本

```bash
pip install onnxruntime-gpu==1.8.1 # 如果你想用gpu版本
```

### 安装float16转换工具(可选)

如果你想用float16精度，请执行以下脚本安装工具:

```bash
```

## 构建自定义算子

### 下载ONNXRuntime库

从ONNX Runtime[发布版本](https://github.com/microsoft/onnxruntime/releases/tag/v1.8.1)下载`onnxruntime-linux-*.tgz`库，并解压，将onnxruntime所在路径添加到`ONNXRUNTIME_DIR`环境变量，最后将lib路径添加到`LD_LIBRARY_PATH`环境变量中，操作如下：

- CPU 版本

```bash

```

```

- GPU 版本

X64 GPU:

```bash

```

Arm GPU:

```bash

```

### 在Linux上构建

- CPU 版本

```bash
cd ${MMDEPLOY_DIR} # 进入MMDeploy根目录

```

- CPU 版本

```bash
```

- GPU 版本

```bash
```

## 如何转换模型

- 你可以按照教程[如何转换模型](../02-how-to-run/convert_model.md)的说明去做

## 如何添加新的自定义算子

## 提示

- 自定义算子不包含在ONNX Runtime[支持的算子列表](https://github.com/microsoft/onnxruntime/blob/master/docs/OperatorKernels.md)中。
- 自定义算子应该能够导出到ONNX。

#### 主要过程

以自定义操作符`roi_align`为例。

1. 在ONNX Runtime源目录`${MMDEPLOY_DIR}/csrc/backend_ops/onnxruntime/`中创建一个`roi_align`目录
2. 添加头文件和源文件到`roi_align`目录`${MMDEPLOY_DIR}/csrc/backend_ops/onnxruntime/roi_align/`

3. 将单元测试添加到`tests/test_ops/test_ops.py`中。
   查看[这里](../../../tests/test_ops/test_ops.py)的例子。

\**最后，欢迎发送为MMDeploy添加ONNX Runtime自定义算子的PR。* \*: nerd_face:

## 参考

- [如何将具有自定义op的Pytorch模型导出为ONNX并在ONNX Runtime运行](https://github.com/onnx/tutorials/blob/master/PyTorchCustomOperator/README.md)
- [如何在ONNX Runtime添加自定义算子/内核](https://onnxruntime.ai/docs/reference/operators/add-custom-op.html)

benchncnn can be used to test neural network inference performance

Only the network definition files (ncnn param) are required.

The large model binary files (ncnn bin) are not loaded but generated randomly for speed test.

If no model specified, it would benchmark default list. More model networks may be added later.

---
Build
```shell
# assume you have already build ncnn library successfully
# uncomment the following line in <ncnn-root-dir>/CMakeLists.txt with your favorite editor

# add_subdirectory(benchmark)

cd <ncnn-root-dir>/<your-build-dir>
make -j4

# you can find benchncnn binary in <ncnn-root-dir>/<your-build-dir>/benchmark
```

Usage
```shell
# copy all param files to the current directory
./benchncnn [loop count] [num threads] [powersave] [gpu device] [cooling down] [(key=value)...]
  param=model.param
  shape=[227,227,3],..
```
run benchncnn on android device
```shell
# for running on android device, upload to /data/local/tmp/ folder
adb push benchncnn /data/local/tmp/

param=model.param
```
```shell
adb push <ncnn-root-dir>/benchmark/*.param /data/local/tmp/
adb shell

# executed in android adb shell
cd /data/local/tmp/
```

Parameter

|param|options|default|
|loop count|1~N|4|
|num threads|1~N|max_cpu_count|
|powersave|0=all cores, 1=little cores only, 2=big cores only|0|
|gpu device|-1=cpu-only, 0=gpu0, 1=gpu1 ...|-1|
|cooling down|0=disable, 1=enable|1|
|param|ncnn model.param filepath|-|
|shape|model input shapes with, whc format|-|

Tips: Disable android UI server and set CPU and GPU to max frequency
```shell
# stopping android ui server, can be retarted later via adb shell start
adb root
adb shell stop

```shell
adb root

# set cpu performance mode
echo "performance" > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
echo "performance" > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
echo "performance" > /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor
echo "performance" > /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor
echo "performance" > /sys/devices/system/cpu/cpu4/cpufreq/scaling_governor
echo "performance" > /sys/devices/system/cpu/cpu5/cpufreq/scaling_governor

# set gpu performance mode (eg. RK3399)
echo "performance" > /sys/class/misc/mali0/device/devfreq/ff9a0000.gpu/governor

# set gpu performance mode (eg. Android Adreno)
echo 1 > /sys/class/kgsl/kgsl-3d0/force_clk_on
echo 10000000 > /sys/class/kgsl/kgsl-3d0/idle_timer
echo "performance" > /sys/class/kgsl/kgsl-3d0/devfreq/governor
echo <max freq> > /sys/class/kgsl/kgsl-3d0/gpuclk
```

```

---

Typical output (executed in android adb shell)

### NVIDIA Jetson AGX Orin (Cortex-A78AE 2.2 GHz x 12 + Ampere@1.3 GHz Tensor Cores 64)
```
i@orin:~/projects/ncnn/benchmark$ ./benchncnn 64 1 0 -1 0
loop_count = 64
num_threads = 1
gpu_device = -1
cooling_down = 0
          squeezenet  min =   11.66  max =   11.80  avg =   11.74
     squeezenet_int8  min =   12.24  max =   12.39  avg =   12.31
           mobilenet  min =   19.56  max =   19.73  avg =   19.65
      mobilenet_int8  min =   16.06  max =   16.25  avg =   16.14
        mobilenet_v2  min =   13.20  max =   13.41  avg =   13.29
        mobilenet_v3  min =   11.39  max =   11.57  avg =   11.48
          shufflenet  min =    8.07  max =    8.18  avg =    8.11
       shufflenet_v2  min =    8.41  max =    8.51  avg =    8.45

mobilenet_v3  min =   11.39  max =   11.57  avg =   11.48
             mnasnet  min =   12.74  max =   12.91  avg =   12.79
     proxylessnasnet  min =   15.18  max =   15.32  avg =   15.25
     efficientnet_b0  min =   26.86  max =   26.96  avg =   26.90
   efficientnetv2_b0  min =   35.99  max =   36.15  avg =   36.07
        regnety_400m  min =   16.81  max =   16.98  avg =   16.87
           blazeface  min =    4.25  max =    4.37  avg =    4.29
           googlenet  min =   48.73  max =   48.98  avg =   48.87
      googlenet_int8  min =   47.39  max =   47.60  avg =   47.49
            resnet18  min =   30.93  max =   31.24  avg =   31.08
       resnet18_int8  min =   55.44  max =   55.70  avg =   55.56
             alexnet  min =   44.19  max =   44.43  avg =   44.33
               vgg16  min =  173.94  max =  174.97  avg =  174.46

resnet18_int8  min =   55.44  max =   55.70  avg =   55.56
          vgg16_int8  min =  475.10  max =  479.37  avg =  477.33
            resnet50  min =   89.50  max =   90.11  avg =   89.80
       resnet50_int8  min =  106.77  max =  107.14  avg =  106.96
      squeezenet_ssd  min =   37.78  max =   38.35  avg =   37.93
 squeezenet_ssd_int8  min =   50.48  max =   50.88  avg =   50.74
       mobilenet_ssd  min =   45.62  max =   46.12  avg =   45.74
  mobilenet_ssd_int8  min =   37.77  max =   38.00  avg =   37.88
      mobilenet_yolo  min =   90.23  max =   90.49  avg =   90.35
  mobilenetv2_yolov3  min =   47.27  max =   47.48  avg =   47.33
         yolov4-tiny  min =   60.41  max =   60.75  avg =   60.57
           nanodet_m  min =   19.26  max =   19.43  avg =   19.35
    yolo-fastest-1.1  min =    8.16  max =    8.31  avg =    8.20

yolov4-tiny  min =   60.41  max =   60.75  avg =   60.57
      yolo-fastestv2  min =    8.26  max =    8.39  avg =    8.32
i@orin:~/projects/ncnn/benchmark$ ./benchncnn 64 2 0 -1 0
num_threads = 2
          squeezenet  min =    6.83  max =    6.98  avg =    6.90
     squeezenet_int8  min =    7.39  max =    7.50  avg =    7.45
           mobilenet  min =   10.40  max =   10.50  avg =   10.45
      mobilenet_int8  min =    8.92  max =    9.09  avg =    8.99
        mobilenet_v2  min =    7.67  max =    7.80  avg =    7.74
        mobilenet_v3  min =    6.86  max =    7.01  avg =    6.93
          shufflenet  min =    6.34  max =    6.44  avg =    6.39
       shufflenet_v2  min =    5.71  max =    5.83  avg =    5.76
             mnasnet  min =    7.47  max =    7.58  avg =    7.53

shufflenet  min =    6.34  max =    6.44  avg =    6.39
     proxylessnasnet  min =    8.73  max =    8.83  avg =    8.78
     efficientnet_b0  min =   14.93  max =   15.13  avg =   15.03
   efficientnetv2_b0  min =   20.17  max =   20.70  avg =   20.29
        regnety_400m  min =   12.50  max =   12.62  avg =   12.57
           blazeface  min =    2.95  max =    3.06  avg =    3.00
           googlenet  min =   26.25  max =   26.53  avg =   26.37
      googlenet_int8  min =   26.54  max =   26.79  avg =   26.66
            resnet18  min =   16.69  max =   16.90  avg =   16.80
       resnet18_int8  min =   29.70  max =   29.93  avg =   29.81
             alexnet  min =   22.96  max =   23.12  avg =   23.03
               vgg16  min =   88.39  max =   89.16  avg =   88.79
          vgg16_int8  min =  245.86  max =  247.55  avg =  246.62

alexnet  min =   22.96  max =   23.12  avg =   23.03
            resnet50  min =   46.55  max =   46.86  avg =   46.70
       resnet50_int8  min =   56.28  max =   56.63  avg =   56.43
      squeezenet_ssd  min =   23.65  max =   24.29  avg =   23.81
 squeezenet_ssd_int8  min =   30.86  max =   31.27  avg =   30.99
       mobilenet_ssd  min =   25.17  max =   25.31  avg =   25.24
  mobilenet_ssd_int8  min =   21.77  max =   21.97  avg =   21.84
      mobilenet_yolo  min =   48.03  max =   48.33  avg =   48.14
  mobilenetv2_yolov3  min =   26.58  max =   26.81  avg =   26.66
         yolov4-tiny  min =   35.31  max =   35.53  avg =   35.41
           nanodet_m  min =   12.93  max =   13.08  avg =   13.01
    yolo-fastest-1.1  min =    6.00  max =    6.10  avg =    6.04
      yolo-fastestv2  min =    6.46  max =    6.61  avg =    6.52

nanodet_m  min =   12.93  max =   13.08  avg =   13.01
i@orin:~/projects/ncnn/benchmark$ ./benchncnn 64 4 0 -1 0
          squeezenet  min =    4.54  max =    4.84  avg =    4.61
     squeezenet_int8  min =    4.96  max =    5.41  avg =    5.05
           mobilenet  min =    5.96  max =    6.23  avg =    6.04
      mobilenet_int8  min =    5.21  max =    5.50  avg =    5.30
        mobilenet_v2  min =    5.05  max =    5.26  avg =    5.15
        mobilenet_v3  min =    4.83  max =    5.14  avg =    4.90
          shufflenet  min =    5.11  max =    5.34  avg =    5.18
       shufflenet_v2  min =    4.13  max =    4.44  avg =    4.18
             mnasnet  min =    4.93  max =    5.27  avg =    5.01
     proxylessnasnet  min =    5.64  max =    5.89  avg =    5.72

shufflenet_v2  min =    4.13  max =    4.44  avg =    4.18
     efficientnet_b0  min =    9.47  max =   10.60  avg =    9.60
   efficientnetv2_b0  min =   12.67  max =   13.06  avg =   12.82
        regnety_400m  min =   10.27  max =   10.58  avg =   10.38
           blazeface  min =    2.05  max =    2.27  avg =    2.10
           googlenet  min =   15.57  max =   15.96  avg =   15.68
      googlenet_int8  min =   16.19  max =   16.65  avg =   16.32
            resnet18  min =   10.20  max =   11.76  avg =   10.35
       resnet18_int8  min =   16.89  max =   17.31  avg =   17.03
             alexnet  min =   13.13  max =   13.70  avg =   13.32
               vgg16  min =   51.03  max =   52.46  avg =   51.35
          vgg16_int8  min =  131.08  max =  139.44  avg =  133.78
            resnet50  min =   26.74  max =   28.32  avg =   26.91

vgg16  min =   51.03  max =   52.46  avg =   51.35
       resnet50_int8  min =   32.15  max =   32.74  avg =   32.38
      squeezenet_ssd  min =   16.58  max =   16.99  avg =   16.70
 squeezenet_ssd_int8  min =   20.22  max =   21.67  avg =   20.51
       mobilenet_ssd  min =   14.68  max =   16.07  avg =   14.83
  mobilenet_ssd_int8  min =   12.89  max =   13.27  avg =   13.01
      mobilenet_yolo  min =   28.44  max =   28.85  avg =   28.58
  mobilenetv2_yolov3  min =   17.21  max =   21.31  avg =   17.44
         yolov4-tiny  min =   23.68  max =   24.38  avg =   23.88
           nanodet_m  min =    8.76  max =    9.17  avg =    8.86
    yolo-fastest-1.1  min =    4.83  max =    5.04  avg =    4.88
      yolo-fastestv2  min =    4.93  max =    5.17  avg =    5.00
i@orin:~/projects/ncnn/benchmark$ ./benchncnn 64 8 0 -1 0

yolo-fastestv2  min =    4.93  max =    5.17  avg =    5.00
num_threads = 8
          squeezenet  min =    3.52  max =    4.28  avg =    3.65
     squeezenet_int8  min =    3.85  max =    4.11  avg =    3.93
           mobilenet  min =    3.78  max =    4.12  avg =    3.85
      mobilenet_int8  min =    3.57  max =    3.85  avg =    3.63
        mobilenet_v2  min =    4.14  max =    4.44  avg =    4.22
        mobilenet_v3  min =    3.89  max =    4.26  avg =    3.97
          shufflenet  min =    4.78  max =    4.95  avg =    4.84
       shufflenet_v2  min =    3.49  max =    3.84  avg =    3.54
             mnasnet  min =    3.94  max =    4.09  avg =    3.99
     proxylessnasnet  min =    4.41  max =    4.68  avg =    4.47
     efficientnet_b0  min =    7.01  max =    7.85  avg =    7.13
   efficientnetv2_b0  min =    9.22  max =    9.46  avg =    9.32

proxylessnasnet  min =    4.41  max =    4.68  avg =    4.47
        regnety_400m  min =    9.34  max =    9.66  avg =    9.44
           blazeface  min =    1.86  max =    1.98  avg =    1.89
           googlenet  min =   10.37  max =   10.76  avg =   10.48
      googlenet_int8  min =   11.03  max =   11.34  avg =   11.16
            resnet18  min =    6.83  max =    7.12  avg =    6.93
       resnet18_int8  min =   10.25  max =   11.50  avg =   10.42
             alexnet  min =    8.88  max =    9.71  avg =    9.01
               vgg16  min =   31.26  max =   31.97  avg =   31.44
          vgg16_int8  min =   71.31  max =   74.53  avg =   72.18
            resnet50  min =   16.43  max =   16.84  avg =   16.52
       resnet50_int8  min =   19.07  max =   20.28  avg =   19.42
      squeezenet_ssd  min =   13.50  max =   13.69  avg =   13.56

resnet50  min =   16.43  max =   16.84  avg =   16.52
 squeezenet_ssd_int8  min =   15.16  max =   16.06  avg =   15.30
       mobilenet_ssd  min =    9.73  max =   10.85  avg =    9.90
  mobilenet_ssd_int8  min =    9.27  max =    9.46  avg =    9.36
      mobilenet_yolo  min =   17.58  max =   17.79  avg =   17.67
  mobilenetv2_yolov3  min =   12.80  max =   13.50  avg =   12.90
         yolov4-tiny  min =   17.98  max =   21.31  avg =   18.24
           nanodet_m  min =    7.01  max =    7.18  avg =    7.09
    yolo-fastest-1.1  min =    4.76  max =    4.86  avg =    4.80
      yolo-fastestv2  min =    4.76  max =    4.88  avg =    4.82
i@orin:~/projects/ncnn/benchmark$ ./benchncnn 64 12 0 -1 0
num_threads = 12
          squeezenet  min =    3.50  max =    5.21  avg =    3.65

     squeezenet_int8  min =    3.97  max =    4.44  avg =    4.12
           mobilenet  min =    3.49  max =    7.73  avg =    3.78
      mobilenet_int8  min =    3.40  max =    3.86  avg =    3.49
        mobilenet_v2  min =    4.07  max =    4.39  avg =    4.17
        mobilenet_v3  min =    3.92  max =    4.17  avg =    4.03
          shufflenet  min =    5.08  max =    6.63  avg =    5.18
       shufflenet_v2  min =    3.64  max =    5.11  avg =    3.75
             mnasnet  min =    3.86  max =    4.16  avg =    3.95
     proxylessnasnet  min =    4.30  max =    5.39  avg =    4.38
     efficientnet_b0  min =    6.42  max =    9.19  avg =    6.61
   efficientnetv2_b0  min =    8.96  max =    9.43  avg =    9.12
        regnety_400m  min =   10.11  max =   10.89  avg =   10.27

efficientnet_b0  min =    6.42  max =    9.19  avg =    6.61
           blazeface  min =    1.93  max =    2.16  avg =    1.99
           googlenet  min =    9.72  max =   10.84  avg =   10.01
      googlenet_int8  min =   10.91  max =   13.03  avg =   11.17
            resnet18  min =    6.70  max =    7.27  avg =    6.92
       resnet18_int8  min =    9.62  max =   12.93  avg =   10.14
             alexnet  min =    7.21  max =    7.47  avg =    7.32
               vgg16  min =   29.61  max =   63.73  avg =   30.86
          vgg16_int8  min =   64.91  max =   75.06  avg =   68.72
            resnet50  min =   15.35  max =   16.28  avg =   15.73
       resnet50_int8  min =   17.47  max =   18.98  avg =   18.09
      squeezenet_ssd  min =   13.40  max =   28.74  avg =   14.07
 squeezenet_ssd_int8  min =   15.35  max =   16.77  avg =   15.67

resnet50_int8  min =   17.47  max =   18.98  avg =   18.09
       mobilenet_ssd  min =    9.51  max =   11.49  avg =    9.88
  mobilenet_ssd_int8  min =    9.43  max =   10.08  avg =    9.58
      mobilenet_yolo  min =   16.88  max =   17.45  avg =   17.09
  mobilenetv2_yolov3  min =   11.91  max =   31.90  avg =   12.50
         yolov4-tiny  min =   17.85  max =   18.87  avg =   18.36
           nanodet_m  min =    6.88  max =    7.64  avg =    7.06
    yolo-fastest-1.1  min =    5.02  max =    5.53  avg =    5.12
      yolo-fastestv2  min =    4.95  max =    5.60  avg =    5.05
i@orin:~/projects/ncnn/benchmark$ ./benchncnn 128 1 0 0 0
[0 NVIDIA Tegra Orin (nvgpu)]  queueC=2[8]  queueG=0[16]  queueT=1[2]
[0 NVIDIA Tegra Orin (nvgpu)]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=0
[0 NVIDIA Tegra Orin (nvgpu)]  fp16-p/s/a=1/1/1  int8-p/s/a=1/1/1

[0 NVIDIA Tegra Orin (nvgpu)]  subgroup=32  basic=1  vote=1  ballot=1  shuffle=1
loop_count = 128
          squeezenet  min =    2.13  max =    3.37  avg =    2.31
     squeezenet_int8  min =   12.31  max =   12.51  avg =   12.42
           mobilenet  min =    2.03  max =    2.73  avg =    2.23
      mobilenet_int8  min =   16.86  max =   17.91  avg =   16.99
        mobilenet_v2  min =    2.59  max =    3.59  avg =    2.91
        mobilenet_v3  min =    3.22  max =    4.23  avg =    3.71
          shufflenet  min =    2.57  max =    3.27  avg =    2.80
       shufflenet_v2  min =    3.20  max =    4.03  avg =    3.47
             mnasnet  min =    2.45  max =    3.06  avg =    2.69
     proxylessnasnet  min =    2.50  max =    3.14  avg =    2.72

shufflenet_v2  min =    3.20  max =    4.03  avg =    3.47
     efficientnet_b0  min =    4.23  max =    8.73  avg =    4.85
   efficientnetv2_b0  min =    8.15  max =    8.60  avg =    8.41
        regnety_400m  min =    3.25  max =    4.17  avg =    3.54
           blazeface  min =    1.29  max =    1.48  avg =    1.33
           googlenet  min =    4.95  max =   12.34  avg =    6.36
      googlenet_int8  min =   47.49  max =   47.78  avg =   47.61
            resnet18  min =    3.18  max =    9.49  avg =    4.04
       resnet18_int8  min =   55.57  max =   55.88  avg =   55.73
             alexnet  min =    3.22  max =   14.56  avg =    4.25
               vgg16  min =    6.82  max =   14.75  avg =    8.18
          vgg16_int8  min =  473.55  max =  479.07  avg =  476.22
            resnet50  min =    4.75  max =   15.06  avg =    6.08

vgg16  min =    6.82  max =   14.75  avg =    8.18
       resnet50_int8  min =  106.99  max =  107.48  avg =  107.22
      squeezenet_ssd  min =    6.87  max =    9.12  avg =    7.76
 squeezenet_ssd_int8  min =   50.87  max =   51.17  avg =   51.01
       mobilenet_ssd  min =    4.44  max =    6.22  avg =    5.23
  mobilenet_ssd_int8  min =   37.80  max =   38.03  avg =   37.92
      mobilenet_yolo  min =    5.41  max =    7.36  avg =    6.29
  mobilenetv2_yolov3  min =    7.20  max =    9.96  avg =    7.30
         yolov4-tiny  min =   16.48  max =   28.81  avg =   18.40
           nanodet_m  min =    5.75  max =    8.54  avg =    6.85
    yolo-fastest-1.1  min =    4.03  max =    4.75  avg =    4.35
      yolo-fastestv2  min =    4.27  max =    5.23  avg =    4.71
```

### AMD Ryzen Threadripper 3970X (Zen2 3.7 GHz ~ 4.5 GHz x 32)
```

yolo-fastestv2  min =    4.27  max =    5.23  avg =    4.71
```

```
i@s:~/qtang/ncnn/benchmark$ ../build-vulkan/benchmark/benchncnn 10 1 0 -1 0
loop_count = 10
          squeezenet  min =   11.73  max =   11.88  avg =   11.78
           mobilenet  min =   21.63  max =   21.73  avg =   21.68
        mobilenet_v2  min =   14.70  max =   14.95  avg =   14.82
        mobilenet_v3  min =   12.12  max =   12.17  avg =   12.15
          shufflenet  min =   14.08  max =   14.16  avg =   14.12
       shufflenet_v2  min =   25.99  max =   26.13  avg =   26.06
             mnasnet  min =   14.12  max =   14.17  avg =   14.14
     proxylessnasnet  min =   16.51  max =   16.71  avg =   16.61
     efficientnet_b0  min =   22.88  max =   22.97  avg =   22.93
        regnety_400m  min =   18.50  max =   18.61  avg =   18.56

proxylessnasnet  min =   16.51  max =   16.71  avg =   16.61
           blazeface  min =    6.18  max =    6.27  avg =    6.21
           googlenet  min =   58.42  max =   58.60  avg =   58.49
            resnet18  min =   61.13  max =   61.84  avg =   61.40
             alexnet  min =   50.82  max =   50.98  avg =   50.92
               vgg16  min =  217.19  max =  218.40  avg =  217.87
            resnet50  min =  126.84  max =  137.46  avg =  128.21
      squeezenet_ssd  min =  114.24  max =  114.57  avg =  114.47
       mobilenet_ssd  min =   51.60  max =   51.89  avg =   51.77
      mobilenet_yolo  min =  125.09  max =  126.33  avg =  125.83
  mobilenetv2_yolov3  min =   57.51  max =   57.79  avg =   57.65
         yolov4-tiny  min =   85.65  max =   85.97  avg =   85.79
```

### NVIDIA Quadro RTX 8000 (TU102 SM x 72 + Tensor Core x 576)
```

yolov4-tiny  min =   85.65  max =   85.97  avg =   85.79
```

```
i@s:~/qtang/ncnn/benchmark$ ../build-vulkan/benchmark/benchncnn 256 1 0 1 0
[0 Quadro RTX 8000]  queueC=2[8]  queueG=0[16]  queueT=1[2]
[0 Quadro RTX 8000]  bugsbn1=0  bugcopc=0  bugihfa=0
[0 Quadro RTX 8000]  fp16p=1  fp16s=1  fp16a=1  int8s=1  int8a=1
[0 Quadro RTX 8000]  subgroup=32  basic=1  vote=1  ballot=1  shuffle=1
[1 Quadro RTX 8000]  queueC=2[8]  queueG=0[16]  queueT=1[2]
[1 Quadro RTX 8000]  bugsbn1=0  bugcopc=0  bugihfa=0
[1 Quadro RTX 8000]  fp16p=1  fp16s=1  fp16a=1  int8s=1  int8a=1
[1 Quadro RTX 8000]  subgroup=32  basic=1  vote=1  ballot=1  shuffle=1
loop_count = 256
gpu_device = 1
          squeezenet  min =    0.84  max =    1.39  avg =    0.93
           mobilenet  min =    0.90  max =    2.30  avg =    0.91
        mobilenet_v2  min =    1.35  max =    9.59  avg =    1.46

squeezenet  min =    0.84  max =    1.39  avg =    0.93
        mobilenet_v3  min =    1.60  max =   77.94  avg =    2.12
          shufflenet  min =    0.86  max =    2.27  avg =    0.88
       shufflenet_v2  min =    1.25  max =    1.47  avg =    1.27
             mnasnet  min =    1.42  max =   20.77  avg =    1.72
     proxylessnasnet  min =    1.48  max =    1.67  avg =    1.49
     efficientnet_b0  min =    2.56  max =   12.86  avg =    2.77
        regnety_400m  min =    1.84  max =   14.98  avg =    2.42
           blazeface  min =    0.64  max =    0.90  avg =    0.65
           googlenet  min =    2.94  max =   76.82  avg =    3.45
            resnet18  min =    1.27  max =   10.56  avg =    1.56
             alexnet  min =    1.53  max =   71.76  avg =    1.96
               vgg16  min =    4.90  max =   78.12  avg =    5.80

resnet18  min =    1.27  max =   10.56  avg =    1.56
            resnet50  min =    3.00  max =   12.51  avg =    3.07
      squeezenet_ssd  min =    5.60  max =   97.09  avg =    6.50
       mobilenet_ssd  min =    2.40  max =   93.64  avg =    3.30
      mobilenet_yolo  min =    2.96  max =   19.15  avg =    3.25
  mobilenetv2_yolov3  min =    4.52  max =   66.96  avg =    5.32
         yolov4-tiny  min =    9.32  max =   72.92  avg =   14.01

```

### NVIDIA RTX3090 (GA102 SM x 82 + Tensor Core 328)
```
(base) i@t:~/wls/ncnn/benchmark$ ../build/benchmark/benchncnn 32 1 0 0 0
[0 GeForce RTX 3090]  queueC=2[8]  queueG=0[16]  queueT=1[2]
[0 GeForce RTX 3090]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=0
[0 GeForce RTX 3090]  fp16-p/s/a=1/1/1  int8-p/s/a=1/1/1
[0 GeForce RTX 3090]  subgroup=32  basic=1  vote=1  ballot=1  shuffle=1

[1 GeForce RTX 3090]  queueC=2[8]  queueG=0[16]  queueT=1[2]
[1 GeForce RTX 3090]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=0
[1 GeForce RTX 3090]  fp16-p/s/a=1/1/1  int8-p/s/a=1/1/1
[1 GeForce RTX 3090]  subgroup=32  basic=1  vote=1  ballot=1  shuffle=1
loop_count = 32
          squeezenet  min =    1.76  max =    2.74  avg =    1.80
     squeezenet_int8  min =   47.10  max =   47.75  avg =   47.21
           mobilenet  min =    4.77  max =    5.79  avg =    5.20
      mobilenet_int8  min =   64.19  max =   67.05  avg =   64.39
        mobilenet_v2  min =    2.44  max =   20.89  avg =    6.98
        mobilenet_v3  min =    2.75  max =    2.87  avg =    2.77
          shufflenet  min =    2.20  max =    2.62  avg =    2.46

mobilenet_v2  min =    2.44  max =   20.89  avg =    6.98
       shufflenet_v2  min =    5.10  max =    7.43  avg =    5.75
             mnasnet  min =    3.47  max =    3.50  avg =    3.48
     proxylessnasnet  min =    2.59  max =    9.08  avg =    7.28
     efficientnet_b0  min =    3.87  max =    4.65  avg =    3.91
   efficientnetv2_b0  min =   29.48  max =   41.90  avg =   30.14
        regnety_400m  min =    2.89  max =    2.99  avg =    2.91
           blazeface  min =    1.55  max =    2.14  avg =    1.60
           googlenet  min =    4.33  max =   17.89  avg =    6.05
      googlenet_int8  min =  174.46  max =  178.19  avg =  174.74
            resnet18  min =    2.14  max =   11.04  avg =    5.33
       resnet18_int8  min =  193.37  max =  193.83  avg =  193.55
             alexnet  min =    2.37  max =   15.99  avg =    4.50

resnet18  min =    2.14  max =   11.04  avg =    5.33
               vgg16  min =    4.55  max =   16.65  avg =    5.22
          vgg16_int8  min = 1538.76  max = 1544.81  avg = 1540.79
            resnet50  min =    4.13  max =   25.86  avg =    5.80
       resnet50_int8  min =  400.89  max =  401.72  avg =  401.29
      squeezenet_ssd  min =    6.95  max =    7.81  avg =    7.07
 squeezenet_ssd_int8  min =  158.51  max =  159.04  avg =  158.68
       mobilenet_ssd  min =    4.36  max =   18.98  avg =    9.40
  mobilenet_ssd_int8  min =  130.74  max =  130.92  avg =  130.83
      mobilenet_yolo  min =    3.96  max =   11.94  avg =    6.48
  mobilenetv2_yolov3  min =    6.07  max =    6.21  avg =    6.13
         yolov4-tiny  min =   13.01  max =   26.78  avg =   14.87

root@3090:~/Desktop/ncnn-20221128/build/benchmark$ ./benchncnn 100 10 2 0 0

yolov4-tiny  min =   13.01  max =   26.78  avg =   14.87

[0 NVIDIA GeForce RTX 3090]  queueC=2[8]  queueG=0[16]  queueT=1[2]
[0 NVIDIA GeForce RTX 3090]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=0
[0 NVIDIA GeForce RTX 3090]  fp16-p/s/a=1/1/1  int8-p/s/a=1/1/1
[0 NVIDIA GeForce RTX 3090]  subgroup=32  basic=1  vote=1  ballot=1  shuffle=1
loop_count = 100
num_threads = 10
powersave = 2
          squeezenet  min =    0.64  max =    0.66  avg =    0.65
     squeezenet_int8  min =    4.30  max =    4.93  avg =    4.45
           mobilenet  min =    0.60  max =    1.85  avg =    1.32
      mobilenet_int8  min =    3.08  max =    3.17  avg =    3.12
        mobilenet_v2  min =    1.40  max =    1.46  avg =    1.42
        mobilenet_v3  min =    1.22  max =    6.10  avg =    3.02
          shufflenet  min =    0.90  max =    0.97  avg =    0.92

mobilenet_v2  min =    1.40  max =    1.46  avg =    1.42
       shufflenet_v2  min =    1.06  max =    1.13  avg =    1.09
             mnasnet  min =    0.84  max =    0.98  avg =    0.91
     proxylessnasnet  min =    0.99  max =    3.01  avg =    2.45
     efficientnet_b0  min =    2.11  max =    2.85  avg =    2.16
   efficientnetv2_b0  min =    7.46  max =   28.58  avg =    8.55
        regnety_400m  min =    1.53  max =    1.75  avg =    1.59
           blazeface  min =    0.59  max =    0.94  avg =    0.63
           googlenet  min =    1.90  max =   12.22  avg =    2.63
      googlenet_int8  min =   17.45  max =   18.69  avg =   17.81
            resnet18  min =    0.90  max =   13.14  avg =    3.09
       resnet18_int8  min =   16.25  max =   17.34  avg =   16.50
             alexnet  min =    0.86  max =    4.77  avg =    2.59

resnet18  min =    0.90  max =   13.14  avg =    3.09
               vgg16  min =    1.38  max =   11.20  avg =    2.91
          vgg16_int8  min =   47.17  max =   49.02  avg =   47.57
            resnet50  min =    1.54  max =    2.16  avg =    1.64
       resnet50_int8  min =   22.90  max =   24.46  avg =   23.23
      squeezenet_ssd  min =    2.25  max =   10.91  avg =    4.12
 squeezenet_ssd_int8  min =   11.98  max =   14.54  avg =   12.31
       mobilenet_ssd  min =    1.46  max =    8.98  avg =    3.38
  mobilenet_ssd_int8  min =    6.13  max =    6.65  avg =    6.23
      mobilenet_yolo  min =    1.29  max =    1.43  avg =    1.34
  mobilenetv2_yolov3  min =    3.64  max =    6.66  avg =    3.77
         yolov4-tiny  min =    9.04  max =   11.65  avg =    9.54
           nanodet_m  min =    1.43  max =   11.90  avg =    3.16

mobilenetv2_yolov3  min =    3.64  max =    6.66  avg =    3.77
    yolo-fastest-1.1  min =    1.40  max =    1.82  avg =    1.57
      yolo-fastestv2  min =    1.36  max =    2.30  avg =    1.42
  vision_transformer  min =  202.71  max =  244.47  avg =  218.69
          FastestDet  min =    1.37  max =    5.37  avg =    2.77
```

### AMD Ryzen Embedded V1605B (Zen 2.0 GHz ~ 3.6 GHz x 4 + Radeon Vega 8 1.1GHz 8CU)
```
C:\Users\i\Desktop\benchmark>benchncnn.exe 32 1 0 -1 0
          squeezenet  min =   22.13  max =   24.07  avg =   22.88
     squeezenet_int8  min =   58.54  max =   62.21  avg =   59.55
           mobilenet  min =   40.99  max =   43.67  avg =   41.70
      mobilenet_int8  min =   98.06  max =  111.37  avg =  101.15

squeezenet_int8  min =   58.54  max =   62.21  avg =   59.55
        mobilenet_v2  min =   26.53  max =   28.96  avg =   27.81
        mobilenet_v3  min =   22.96  max =   25.25  avg =   23.30
          shufflenet  min =   20.17  max =   28.78  avg =   21.09
       shufflenet_v2  min =   19.06  max =   19.72  avg =   19.47
             mnasnet  min =   25.11  max =   39.53  avg =   27.54
     proxylessnasnet  min =   28.84  max =   35.16  avg =   30.03
     efficientnet_b0  min =   43.16  max =   46.03  avg =   43.65
   efficientnetv2_b0  min =   48.64  max =   52.07  avg =   49.62
        regnety_400m  min =   33.43  max =   35.87  avg =   33.97
           blazeface  min =    5.43  max =    6.04  avg =    5.56
           googlenet  min =   85.80  max =   90.93  avg =   87.65
      googlenet_int8  min =  214.37  max =  230.75  avg =  219.50

blazeface  min =    5.43  max =    6.04  avg =    5.56
            resnet18  min =   76.58  max =   80.38  avg =   77.34
       resnet18_int8  min =  231.16  max =  255.22  avg =  236.65
             alexnet  min =   60.69  max =   64.06  avg =   61.34
               vgg16  min =  286.45  max =  307.04  avg =  290.86
          vgg16_int8  min = 1797.58  max = 2079.73  avg = 1844.78
            resnet50  min =  198.27  max =  215.03  avg =  201.37
       resnet50_int8  min =  493.52  max =  499.67  avg =  496.95
      squeezenet_ssd  min =  189.97  max =  198.53  avg =  192.10
 squeezenet_ssd_int8  min =  198.81  max =  214.55  avg =  203.59
       mobilenet_ssd  min =   87.56  max =   92.72  avg =   89.03
  mobilenet_ssd_int8  min =  196.97  max =  209.51  avg =  201.95
      mobilenet_yolo  min =  206.87  max =  218.48  avg =  210.84

mobilenet_ssd  min =   87.56  max =   92.72  avg =   89.03
  mobilenetv2_yolov3  min =  102.72  max =  108.18  avg =  104.62
         yolov4-tiny  min =  117.97  max =  134.73  avg =  121.26

C:\Users\i\Desktop\benchmark>benchncnn.exe 32 2 0 -1 0
          squeezenet  min =   13.43  max =   14.35  avg =   13.62
     squeezenet_int8  min =   32.29  max =   50.76  avg =   33.56
           mobilenet  min =   23.42  max =   25.10  avg =   24.09
      mobilenet_int8  min =   51.99  max =   55.42  avg =   53.01
        mobilenet_v2  min =   15.45  max =   15.75  avg =   15.59
        mobilenet_v3  min =   14.32  max =   14.75  avg =   14.39
          shufflenet  min =   12.64  max =   12.83  avg =   12.69
       shufflenet_v2  min =   11.45  max =   12.44  avg =   11.60

mobilenet_v3  min =   14.32  max =   14.75  avg =   14.39
             mnasnet  min =   14.43  max =   20.45  avg =   15.11
     proxylessnasnet  min =   16.18  max =   16.38  avg =   16.24
     efficientnet_b0  min =   25.25  max =   28.42  avg =   26.59
   efficientnetv2_b0  min =   27.57  max =   32.05  avg =   30.04
        regnety_400m  min =   22.74  max =   24.75  avg =   23.31
           blazeface  min =    3.44  max =    3.83  avg =    3.62
           googlenet  min =   49.39  max =   66.76  avg =   53.76
      googlenet_int8  min =  113.89  max =  136.75  avg =  119.29
            resnet18  min =   43.77  max =   67.24  avg =   46.14
       resnet18_int8  min =  121.44  max =  148.01  avg =  126.95
             alexnet  min =   34.46  max =   37.38  avg =   35.50
               vgg16  min =  177.16  max =  207.25  avg =  184.19

resnet18_int8  min =  121.44  max =  148.01  avg =  126.95
          vgg16_int8  min =  951.86  max = 1155.60  avg =  990.51
            resnet50  min =  112.28  max =  137.18  avg =  115.64
       resnet50_int8  min =  260.69  max =  272.26  avg =  265.89
      squeezenet_ssd  min =  108.07  max =  121.66  avg =  110.35
 squeezenet_ssd_int8  min =  109.01  max =  126.86  avg =  111.96
       mobilenet_ssd  min =   49.60  max =   52.62  avg =   50.46
  mobilenet_ssd_int8  min =  104.22  max =  111.07  avg =  106.33
      mobilenet_yolo  min =  117.42  max =  136.73  avg =  122.92
  mobilenetv2_yolov3  min =   61.66  max =   65.22  avg =   63.01
         yolov4-tiny  min =   72.64  max =   77.09  avg =   74.30

C:\Users\i\Desktop\benchmark>benchncnn.exe 32 4 0 -1 0

yolov4-tiny  min =   72.64  max =   77.09  avg =   74.30

          squeezenet  min =    9.19  max =   14.82  avg =   11.15
     squeezenet_int8  min =   19.00  max =   40.30  avg =   24.80
           mobilenet  min =   18.02  max =   39.84  avg =   27.38
      mobilenet_int8  min =   28.04  max =   57.59  avg =   34.15
        mobilenet_v2  min =   10.26  max =   17.79  avg =   13.36
        mobilenet_v3  min =    8.87  max =   10.87  avg =    9.11
          shufflenet  min =    8.93  max =   11.96  avg =    9.34
       shufflenet_v2  min =    7.37  max =   13.10  avg =    8.72
             mnasnet  min =    9.24  max =   14.90  avg =   11.32
     proxylessnasnet  min =   10.21  max =   11.89  avg =   10.39
     efficientnet_b0  min =   16.22  max =   23.71  avg =   16.59
   efficientnetv2_b0  min =   17.44  max =   31.42  avg =   22.85

proxylessnasnet  min =   10.21  max =   11.89  avg =   10.39
        regnety_400m  min =   18.32  max =   24.02  avg =   18.90
           blazeface  min =    2.22  max =    2.81  avg =    2.30
           googlenet  min =   31.52  max =   51.80  avg =   42.11
      googlenet_int8  min =   65.47  max =  114.41  avg =   75.98
            resnet18  min =   28.90  max =   64.62  avg =   37.58
       resnet18_int8  min =   71.29  max =  136.67  avg =  103.03
             alexnet  min =   23.67  max =   34.01  avg =   29.78
               vgg16  min =  142.18  max =  211.00  avg =  170.46
          vgg16_int8  min =  531.36  max =  871.25  avg =  625.60
            resnet50  min =   69.23  max =  108.67  avg =   73.68
       resnet50_int8  min =  149.18  max =  309.88  avg =  168.68
      squeezenet_ssd  min =   68.83  max =   81.70  avg =   71.01

resnet50  min =   69.23  max =  108.67  avg =   73.68
 squeezenet_ssd_int8  min =   66.34  max =  118.16  avg =   74.34
       mobilenet_ssd  min =   29.96  max =   34.32  avg =   30.74
  mobilenet_ssd_int8  min =   56.87  max =   92.24  avg =   65.57
      mobilenet_yolo  min =   74.26  max =  113.91  avg =   81.28
  mobilenetv2_yolov3  min =   42.16  max =   63.49  avg =   45.34
         yolov4-tiny  min =   53.06  max =   69.84  avg =   55.81

C:\Users\i\Desktop\benchmark>benchncnn.exe 32 1 0 0 0
[0 AMD Radeon(TM) Vega 8 Graphics]  queueC=1[2]  queueG=0[1]  queueT=2[1]
[0 AMD Radeon(TM) Vega 8 Graphics]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=0
[0 AMD Radeon(TM) Vega 8 Graphics]  fp16-p/s/a=1/1/1  int8-p/s/a=1/1/1
[0 AMD Radeon(TM) Vega 8 Graphics]  subgroup=64  basic=1  vote=1  ballot=1  shuffle=1

          squeezenet  min =    6.78  max =    7.09  avg =    6.91
     squeezenet_int8  min =   58.93  max =   62.53  avg =   60.11
           mobilenet  min =    8.08  max =    8.39  avg =    8.25
      mobilenet_int8  min =   97.74  max =  116.77  avg =  100.17
        mobilenet_v2  min =    7.95  max =    8.27  avg =    8.14
        mobilenet_v3  min =    8.70  max =    9.70  avg =    9.02
          shufflenet  min =    6.36  max =    7.64  avg =    7.01
       shufflenet_v2  min =    7.04  max =    8.12  avg =    7.50
             mnasnet  min =    8.07  max =    9.08  avg =    8.38
     proxylessnasnet  min =    8.56  max =    9.66  avg =    8.81
     efficientnet_b0  min =   16.68  max =   18.00  avg =   17.30

mnasnet  min =    8.07  max =    9.08  avg =    8.38
   efficientnetv2_b0  min =  394.82  max =  404.88  avg =  401.05
        regnety_400m  min =   11.92  max =   12.17  avg =   12.03
           blazeface  min =    4.82  max =    6.50  avg =    5.42
           googlenet  min =   18.44  max =   19.66  avg =   19.18
      googlenet_int8  min =  213.41  max =  231.79  avg =  218.31
            resnet18  min =   14.27  max =   14.72  avg =   14.44
       resnet18_int8  min =  228.79  max =  249.65  avg =  236.06
             alexnet  min =   17.31  max =   18.31  avg =   17.69
               vgg16  min =  111.85  max =  123.35  avg =  112.98
          vgg16_int8  min = 1789.64  max = 1838.84  avg = 1826.05
            resnet50  min =   31.61  max =   32.86  avg =   32.12
       resnet50_int8  min =  483.57  max =  505.72  avg =  491.76

vgg16_int8  min = 1789.64  max = 1838.84  avg = 1826.05
      squeezenet_ssd  min =   99.66  max =  105.68  avg =  104.57
 squeezenet_ssd_int8  min =  200.48  max =  208.71  avg =  203.02
       mobilenet_ssd  min =   33.45  max =   35.64  avg =   34.75
  mobilenet_ssd_int8  min =  195.14  max =  205.35  avg =  200.18
      mobilenet_yolo  min =   59.20  max =   61.06  avg =   60.47
  mobilenetv2_yolov3  min =   31.48  max =   33.25  avg =   32.84
         yolov4-tiny  min =   93.75  max =   97.45  avg =   96.00
```

### Qualcomm SM8150-AC Snapdragon 855+ (Kyro485 2.96 GHz + 2.42 GHz x 3 + 1.80 GHz x 4 + Adreno 640)
```
OnePlus7T:/data/local/tmp $ ./benchncnn 8 4 2 -1 1
loop_count = 8
          squeezenet  min =    4.05  max =    4.15  avg =    4.09

```
     squeezenet_int8  min =    4.74  max =    4.82  avg =    4.79
           mobilenet  min =    5.64  max =    5.71  avg =    5.67
      mobilenet_int8  min =    4.66  max =    4.77  avg =    4.71
        mobilenet_v2  min =    4.79  max =    4.93  avg =    4.85
        mobilenet_v3  min =    4.47  max =    4.64  avg =    4.54
          shufflenet  min =    4.19  max =    4.35  avg =    4.26
       shufflenet_v2  min =    3.23  max =    3.36  avg =    3.29
             mnasnet  min =    4.43  max =    4.54  avg =    4.48
     proxylessnasnet  min =    5.12  max =    5.23  avg =    5.16
     efficientnet_b0  min =    7.82  max =    7.99  avg =    7.88
   efficientnetv2_b0  min =   12.97  max =   13.15  avg =   13.04
        regnety_400m  min =    8.03  max =    8.19  avg =    8.08

efficientnet_b0  min =    7.82  max =    7.99  avg =    7.88
           blazeface  min =    1.69  max =    1.89  avg =    1.77
           googlenet  min =   15.96  max =   16.14  avg =   16.04
      googlenet_int8  min =   17.26  max =   17.47  avg =   17.38
            resnet18  min =   11.56  max =   11.82  avg =   11.63
       resnet18_int8  min =   17.45  max =   17.80  avg =   17.54
             alexnet  min =   15.27  max =   15.41  avg =   15.34
               vgg16  min =   61.75  max =   63.38  avg =   62.65
          vgg16_int8  min =  138.96  max =  143.29  avg =  141.02
            resnet50  min =   27.51  max =   27.67  avg =   27.61
       resnet50_int8  min =   32.92  max =   33.07  avg =   32.98
      squeezenet_ssd  min =   15.17  max =   15.27  avg =   15.22
 squeezenet_ssd_int8  min =   19.06  max =   19.44  avg =   19.22

resnet50_int8  min =   32.92  max =   33.07  avg =   32.98
       mobilenet_ssd  min =   12.99  max =   13.18  avg =   13.07
  mobilenet_ssd_int8  min =   11.13  max =   11.29  avg =   11.21
      mobilenet_yolo  min =   27.02  max =   27.39  avg =   27.27
  mobilenetv2_yolov3  min =   17.94  max =   18.32  avg =   18.10
         yolov4-tiny  min =   25.17  max =   25.38  avg =   25.28
           nanodet_m  min =    7.58  max =    7.74  avg =    7.66
    yolo-fastest-1.1  min =    3.83  max =    4.00  avg =    3.89
      yolo-fastestv2  min =    3.77  max =    3.85  avg =    3.81

OnePlus7T:/data/local/tmp $ ./benchncnn 8 1 2 -1 1
          squeezenet  min =    9.05  max =    9.18  avg =    9.13
     squeezenet_int8  min =    9.56  max =    9.64  avg =    9.60

           mobilenet  min =   14.64  max =   14.81  avg =   14.72
      mobilenet_int8  min =   11.68  max =   11.77  avg =   11.72
        mobilenet_v2  min =   10.04  max =   10.11  avg =   10.07
        mobilenet_v3  min =    8.40  max =    8.68  avg =    8.51
          shufflenet  min =    6.60  max =    6.75  avg =    6.66
       shufflenet_v2  min =    5.75  max =    5.89  avg =    5.83
             mnasnet  min =    9.56  max =    9.68  avg =    9.61
     proxylessnasnet  min =   11.37  max =   11.50  avg =   11.43
     efficientnet_b0  min =   18.64  max =   18.79  avg =   18.71
   efficientnetv2_b0  min =   33.65  max =   33.74  avg =   33.70
        regnety_400m  min =   12.00  max =   12.08  avg =   12.04
           blazeface  min =    3.29  max =    3.62  avg =    3.41

efficientnetv2_b0  min =   33.65  max =   33.74  avg =   33.70
           googlenet  min =   40.32  max =   40.41  avg =   40.36
      googlenet_int8  min =   40.08  max =   40.38  avg =   40.22
            resnet18  min =   22.98  max =   23.21  avg =   23.06
       resnet18_int8  min =   44.84  max =   45.09  avg =   44.95
             alexnet  min =   35.10  max =   35.21  avg =   35.15
               vgg16  min =  135.30  max =  135.81  avg =  135.50
          vgg16_int8  min =  355.94  max =  356.66  avg =  356.33
            resnet50  min =   67.15  max =   67.38  avg =   67.26
       resnet50_int8  min =   83.66  max =   84.23  avg =   83.96
      squeezenet_ssd  min =   29.95  max =   30.08  avg =   30.02
 squeezenet_ssd_int8  min =   38.99  max =   39.84  avg =   39.29
       mobilenet_ssd  min =   33.58  max =   33.68  avg =   33.64

squeezenet_ssd  min =   29.95  max =   30.08  avg =   30.02
  mobilenet_ssd_int8  min =   27.19  max =   27.40  avg =   27.30
      mobilenet_yolo  min =   67.69  max =   67.90  avg =   67.79
  mobilenetv2_yolov3  min =   37.05  max =   37.25  avg =   37.17
         yolov4-tiny  min =   49.10  max =   49.39  avg =   49.20
           nanodet_m  min =   14.83  max =   15.03  avg =   14.92
    yolo-fastest-1.1  min =    6.01  max =    6.18  avg =    6.09
      yolo-fastestv2  min =    5.73  max =    5.84  avg =    5.80

OnePlus7T:/data/local/tmp $ ./benchncnn 8 1 2 0 1
[0 Adreno (TM) 640]  queueC=0[3]  queueG=0[3]  queueT=0[3]
[0 Adreno (TM) 640]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=1
[0 Adreno (TM) 640]  fp16-p/s/a=1/0/1  int8-p/s/a=1/0/0
[0 Adreno (TM) 640]  subgroup=64  basic=1  vote=1  ballot=0  shuffle=0

          squeezenet  min =    8.59  max =    9.51  avg =    9.09
           mobilenet  min =   13.04  max =   13.45  avg =   13.22
        mobilenet_v2  min =   10.68  max =   11.38  avg =   10.85
        mobilenet_v3  min =   11.86  max =   12.37  avg =   12.08
          shufflenet  min =    8.21  max =    8.40  avg =    8.25
       shufflenet_v2  min =    8.84  max =    9.13  avg =    8.97
             mnasnet  min =   11.32  max =   11.72  avg =   11.45
     proxylessnasnet  min =   12.27  max =   12.86  avg =   12.55
     efficientnet_b0  min =   22.64  max =   22.82  avg =   22.75
   efficientnetv2_b0  min =   32.32  max =   38.20  avg =   35.79
        regnety_400m  min =   15.35  max =   15.86  avg =   15.64
           blazeface  min =    2.82  max =    2.93  avg =    2.86

efficientnetv2_b0  min =   32.32  max =   38.20  avg =   35.79
           googlenet  min =   28.22  max =   28.34  avg =   28.26
            resnet18  min =   24.71  max =   24.96  avg =   24.82
             alexnet  min =   27.94  max =   28.10  avg =   28.01
               vgg16  min =  106.08  max =  106.53  avg =  106.30
            resnet50  min =   55.28  max =   56.03  avg =   55.68
      squeezenet_ssd  min =   29.77  max =   30.65  avg =   30.05
       mobilenet_ssd  min =   29.14  max =   29.39  avg =   29.25
      mobilenet_yolo  min =   49.78  max =   50.09  avg =   49.94
  mobilenetv2_yolov3  min =   31.11  max =   31.97  avg =   31.60
         yolov4-tiny  min =   46.22  max =   46.90  avg =   46.63
           nanodet_m  min =   15.96  max =   16.52  avg =   16.13
    yolo-fastest-1.1  min =    9.59  max =    9.66  avg =    9.61

yolov4-tiny  min =   46.22  max =   46.90  avg =   46.63
      yolo-fastestv2  min =    7.99  max =    8.23  avg =    8.13
```

### Qualcomm MSM6150 Snapdragon 675 (Kyro460 2.0GHz x 2 + Kyro460 1.7GHz x 6 + Adreno 612)
```
violet:/data/local/tmp/ncnn $ ./benchncnn 8 2 0
          squeezenet  min =   23.29  max =   24.65  avg =   23.95
     squeezenet_int8  min =   23.24  max =   61.55  avg =   31.20
           mobilenet  min =   31.60  max =   32.10  avg =   31.80
      mobilenet_int8  min =   30.35  max =   32.03  avg =   30.95
        mobilenet_v2  min =   25.92  max =   26.45  avg =   26.08
          shufflenet  min =   11.91  max =   12.11  avg =   12.00
             mnasnet  min =   21.38  max =   21.71  avg =   21.51
     proxylessnasnet  min =   25.53  max =   25.78  avg =   25.62

shufflenet  min =   11.91  max =   12.11  avg =   12.00
           googlenet  min =   93.62  max =  100.67  avg =   94.86
      googlenet_int8  min =   90.74  max =   91.06  avg =   90.87
            resnet18  min =   85.84  max =   87.37  avg =   86.50
       resnet18_int8  min =   77.88  max =   78.11  avg =   78.00
             alexnet  min =  196.33  max =  201.73  avg =  200.19
               vgg16  min =  560.71  max =  571.75  avg =  564.84
          vgg16_int8  min =  651.51  max =  652.68  avg =  652.12
            resnet50  min =  178.25  max =  179.86  avg =  178.77
       resnet50_int8  min =  181.07  max =  183.26  avg =  181.64
      squeezenet_ssd  min =   64.86  max =   68.39  avg =   66.05
 squeezenet_ssd_int8  min =   69.61  max =   70.37  avg =   69.93
       mobilenet_ssd  min =   65.92  max =   67.03  avg =   66.41

squeezenet_ssd  min =   64.86  max =   68.39  avg =   66.05
  mobilenet_ssd_int8  min =   61.54  max =   63.38  avg =   62.27
      mobilenet_yolo  min =  143.42  max =  146.69  avg =  144.33
    mobilenet_yolov3  min =  150.45  max =  152.30  avg =  151.36

violet:/data/local/tmp/ncnn $ ./benchncnn 8 1 0
          squeezenet  min =   36.04  max =   37.25  avg =   36.48
     squeezenet_int8  min =   37.82  max =   79.20  avg =   43.13
           mobilenet  min =   54.29  max =   54.73  avg =   54.41
      mobilenet_int8  min =   58.90  max =   60.11  avg =   59.39
        mobilenet_v2  min =   38.64  max =   40.22  avg =   38.97
          shufflenet  min =   18.05  max =   18.39  avg =   18.19
             mnasnet  min =   34.65  max =   34.98  avg =   34.79

mobilenet_v2  min =   38.64  max =   40.22  avg =   38.97
     proxylessnasnet  min =   42.61  max =   43.12  avg =   42.80
           googlenet  min =  164.74  max =  165.89  avg =  165.34
      googlenet_int8  min =  159.93  max =  160.38  avg =  160.12
            resnet18  min =  135.76  max =  137.93  avg =  136.98
       resnet18_int8  min =  140.22  max =  144.06  avg =  141.92
             alexnet  min =  391.01  max =  396.85  avg =  392.74
               vgg16  min = 1019.35  max = 1022.75  avg = 1021.26
          vgg16_int8  min = 1122.25  max = 1137.99  avg = 1124.78
            resnet50  min =  302.16  max =  304.22  avg =  303.05
       resnet50_int8  min =  318.35  max =  319.50  avg =  318.84
      squeezenet_ssd  min =   91.26  max =   94.86  avg =   92.39
 squeezenet_ssd_int8  min =  105.06  max =  106.17  avg =  105.56

resnet50_int8  min =  318.35  max =  319.50  avg =  318.84
       mobilenet_ssd  min =  105.01  max =  105.95  avg =  105.40
  mobilenet_ssd_int8  min =  119.93  max =  120.50  avg =  120.19
      mobilenet_yolo  min =  229.87  max =  230.76  avg =  230.21
    mobilenet_yolov3  min =  242.10  max =  242.91  avg =  242.47
```

### Kirin 970 (Cortex-A73 2.4GHz x 4 + Cortex-A53 1.8GHz x 4)
```
HWEML:/data/local/tmp/ncnnbench $ ./benchncnn 8 4 2 -1 1
[0 Mali-G72]  queueC=0[2]  queueG=0[2]  queueT=0[2]
[0 Mali-G72]  buglssc=0  bugsbn1=0  buglbia=0  bugihfa=1
[0 Mali-G72]  fp16p=1  fp16s=0  fp16a=1  int8s=0  int8a=0
          squeezenet  min =   24.38  max =   28.03  avg =   25.83
     squeezenet_int8  min =   21.79  max =   24.80  avg =   22.60

           mobilenet  min =   34.09  max =   36.88  avg =   35.93
      mobilenet_int8  min =   52.62  max =   61.70  avg =   55.38
        mobilenet_v2  min =   23.71  max =   25.70  avg =   24.49
        mobilenet_v3  min =   20.66  max =   25.68  avg =   23.07
          shufflenet  min =   17.89  max =   19.91  avg =   18.53
       shufflenet_v2  min =   13.73  max =   16.54  avg =   15.37
             mnasnet  min =   24.36  max =   27.14  avg =   25.58
     proxylessnasnet  min =   27.19  max =   29.70  avg =   28.59
     efficientnet_b0  min =   49.31  max =   50.26  avg =   49.70
        regnety_400m  min =   42.54  max =   51.22  avg =   46.71
           blazeface  min =    5.49  max =    7.67  avg =    6.27
           googlenet  min =   72.67  max =   81.22  avg =   75.92

regnety_400m  min =   42.54  max =   51.22  avg =   46.71
      googlenet_int8  min =   67.60  max =   74.50  avg =   71.21
            resnet18  min =   69.32  max =   81.59  avg =   73.45
       resnet18_int8  min =   60.92  max =   68.11  avg =   64.18
             alexnet  min =   60.90  max =   79.28  avg =   66.72
               vgg16  min =  337.01  max =  378.89  avg =  352.37
          vgg16_int8  min =  465.88  max =  505.19  avg =  489.76
            resnet50  min =  207.75  max =  220.74  avg =  214.42
       resnet50_int8  min =  165.67  max =  183.80  avg =  171.27
      squeezenet_ssd  min =   72.77  max =   84.45  avg =   79.09
 squeezenet_ssd_int8  min =   75.37  max =   86.58  avg =   78.70
       mobilenet_ssd  min =   88.88  max =   96.43  avg =   92.02
  mobilenet_ssd_int8  min =   89.04  max =  101.35  avg =   92.23

squeezenet_ssd_int8  min =   75.37  max =   86.58  avg =   78.70
      mobilenet_yolo  min =  189.73  max =  206.55  avg =  193.64
  mobilenetv2_yolov3  min =   99.08  max =  111.64  avg =  104.23

HWEML:/data/local/tmp/ncnnbench $ ./benchncnn 8 1 2 -1 1
          squeezenet  min =   73.47  max =   81.39  avg =   76.06
     squeezenet_int8  min =   62.63  max =   73.66  avg =   66.52
           mobilenet  min =  103.85  max =  112.83  avg =  108.98
      mobilenet_int8  min =  152.27  max =  161.26  avg =  157.17
        mobilenet_v2  min =   70.53  max =   87.26  avg =   76.67

mobilenet  min =  103.85  max =  112.83  avg =  108.98
        mobilenet_v3  min =   59.87  max =   68.59  avg =   63.08
          shufflenet  min =   36.69  max =   41.45  avg =   39.24
       shufflenet_v2  min =   33.97  max =   37.84  avg =   35.03
             mnasnet  min =   69.24  max =   79.73  avg =   74.20
     proxylessnasnet  min =   78.63  max =   88.57  avg =   81.83
     efficientnet_b0  min =  147.45  max =  159.07  avg =  152.09
        regnety_400m  min =   90.83  max =   98.51  avg =   93.82
           blazeface  min =   10.05  max =   11.59  avg =   10.78
           googlenet  min =  240.26  max =  277.71  avg =  259.61
      googlenet_int8  min =  214.64  max =  233.56  avg =  225.01
            resnet18  min =  245.62  max =  268.49  avg =  260.37
       resnet18_int8  min =  184.85  max =  194.91  avg =  190.60

googlenet_int8  min =  214.64  max =  233.56  avg =  225.01
             alexnet  min =  202.52  max =  241.12  avg =  211.51
               vgg16  min = 1632.98  max = 1769.05  avg = 1710.89
          vgg16_int8  min = 1237.01  max = 1316.40  avg = 1273.44
            resnet50  min =  558.41  max =  601.59  avg =  581.26
       resnet50_int8  min =  425.26  max =  445.19  avg =  436.22
      squeezenet_ssd  min =  228.50  max =  255.89  avg =  244.63
 squeezenet_ssd_int8  min =  166.97  max =  193.77  avg =  180.22
       mobilenet_ssd  min =  226.54  max =  246.62  avg =  235.75
  mobilenet_ssd_int8  min =  231.35  max =  249.63  avg =  241.29
      mobilenet_yolo  min =  469.71  max =  508.79  avg =  497.50
  mobilenetv2_yolov3  min =  242.88  max =  265.30  avg =  254.68

HWEML:/data/local/tmp/ncnnbench $ ./benchncnn 4 1 2 0 1

mobilenet_yolo  min =  469.71  max =  508.79  avg =  497.50

loop_count = 4
          squeezenet  min =   24.54  max =   25.75  avg =   25.16
           mobilenet  min =   22.03  max =   29.61  avg =   27.31
        mobilenet_v2  min =   20.15  max =   28.05  avg =   25.35
        mobilenet_v3  min =   34.26  max =   37.49  avg =   35.51
          shufflenet  min =   26.29  max =   27.68  avg =   26.86
       shufflenet_v2  min =   29.60  max =   32.08  avg =   31.27
             mnasnet  min =   25.85  max =   29.38  avg =   27.98
     proxylessnasnet  min =   23.64  max =   30.09  avg =   26.36

shufflenet_v2  min =   29.60  max =   32.08  avg =   31.27
     efficientnet_b0  min =   52.55  max =   58.51  avg =   55.56
        regnety_400m  min =   37.81  max =   43.22  avg =   40.30
           blazeface  min =    9.14  max =   10.93  avg =   10.08
           googlenet  min =   60.19  max =   62.84  avg =   61.51
            resnet18  min =   50.42  max =   52.93  avg =   51.70
             alexnet  min =  195.34  max =  196.98  avg =  196.14
               vgg16  min =  725.88  max =  751.20  avg =  739.99
            resnet50  min =  124.47  max =  125.93  avg =  125.02
      squeezenet_ssd  min =   91.79  max =   97.04  avg =   93.56
       mobilenet_ssd  min =   51.81  max =   59.31  avg =   54.09
      mobilenet_yolo  min =  124.67  max =  127.62  avg =  126.53
  mobilenetv2_yolov3  min =   53.11  max =   54.81  avg =   54.11
```

mobilenet_yolo  min =  124.67  max =  127.62  avg =  126.53
```

### Qualcomm MSM8998 Snapdragon 835 (Kyro 2.45GHz x 4 + Kyro 1.9GHz x 4 + Adreno 540)
```
taimen:/data/local/tmp/ncnnbench $ ./benchncnn 8 4 2 -1 0
[0 Adreno (TM) 540]  queueC=0[3]  queueG=0[3]  queueT=0[3]
[0 Adreno (TM) 540]  buglssc=0  bugsbn1=1  buglbia=0  bugihfa=0
[0 Adreno (TM) 540]  fp16p=1  fp16s=0  fp16a=0  int8s=0  int8a=0
          squeezenet  min =   28.46  max =   30.89  avg =   29.77
     squeezenet_int8  min =   30.32  max =   32.92  avg =   31.68
           mobilenet  min =   36.65  max =   38.37  avg =   37.32
      mobilenet_int8  min =   62.91  max =   66.71  avg =   64.49
        mobilenet_v2  min =   27.85  max =   31.21  avg =   29.41
        mobilenet_v3  min =   23.83  max =   26.40  avg =   24.79

mobilenet_int8  min =   62.91  max =   66.71  avg =   64.49
          shufflenet  min =   15.65  max =   16.88  avg =   16.27
       shufflenet_v2  min =   13.70  max =   14.49  avg =   14.08
             mnasnet  min =   25.04  max =   28.35  avg =   26.45
     proxylessnasnet  min =   27.49  max =   29.58  avg =   28.62
     efficientnet_b0  min =   48.43  max =   49.41  avg =   48.85
        regnety_400m  min =   42.48  max =   43.78  avg =   43.18
           blazeface  min =    4.39  max =    4.68  avg =    4.51
           googlenet  min =   75.98  max =   78.40  avg =   77.37
      googlenet_int8  min =   79.26  max =   83.20  avg =   80.55
            resnet18  min =   73.60  max =   76.97  avg =   75.63
       resnet18_int8  min =   62.93  max =   65.94  avg =   64.50
             alexnet  min =   64.18  max =   67.02  avg =   65.49

resnet18  min =   73.60  max =   76.97  avg =   75.63
               vgg16  min =  389.39  max =  399.13  avg =  394.09
          vgg16_int8  min =  509.06  max =  524.41  avg =  514.76
            resnet50  min =  188.21  max =  194.58  avg =  191.98
       resnet50_int8  min =  182.84  max =  187.22  avg =  184.23
      squeezenet_ssd  min =   77.69  max =   81.17  avg =   79.24
 squeezenet_ssd_int8  min =   81.71  max =   84.12  avg =   82.90
       mobilenet_ssd  min =   78.35  max =   81.50  avg =   79.82
  mobilenet_ssd_int8  min =   96.84  max =  100.97  avg =   98.42
      mobilenet_yolo  min =  167.32  max =  170.71  avg =  168.87
  mobilenetv2_yolov3  min =   97.00  max =  102.11  avg =   99.01

taimen:/data/local/tmp/ncnnbench $ ./benchncnn 8 1 2 -1 1

mobilenetv2_yolov3  min =   97.00  max =  102.11  avg =   99.01

          squeezenet  min =   67.25  max =   71.39  avg =   69.35
     squeezenet_int8  min =   62.12  max =   66.35  avg =   63.73
           mobilenet  min =  103.30  max =  110.39  avg =  107.13
      mobilenet_int8  min =  155.24  max =  161.42  avg =  157.82
        mobilenet_v2  min =   71.89  max =   74.73  avg =   73.48
        mobilenet_v3  min =   58.35  max =   63.43  avg =   60.68
          shufflenet  min =   35.96  max =   39.43  avg =   36.94
       shufflenet_v2  min =   35.53  max =   39.86  avg =   37.10
             mnasnet  min =   66.71  max =   74.00  avg =   68.65

shufflenet  min =   35.96  max =   39.43  avg =   36.94
     proxylessnasnet  min =   76.50  max =   82.20  avg =   78.57
     efficientnet_b0  min =  142.32  max =  152.17  avg =  146.14
        regnety_400m  min =   89.60  max =   98.27  avg =   92.62
           blazeface  min =   10.45  max =   12.81  avg =   11.07
           googlenet  min =  222.75  max =  233.61  avg =  228.38
      googlenet_int8  min =  206.70  max =  212.20  avg =  209.24
            resnet18  min =  210.86  max =  220.25  avg =  213.65
       resnet18_int8  min =  176.04  max =  183.58  avg =  178.71
             alexnet  min =  185.97  max =  195.91  avg =  191.40
               vgg16  min = 1176.82  max = 1200.64  avg = 1187.88
          vgg16_int8  min = 1086.52  max = 1105.00  avg = 1095.53
            resnet50  min =  517.48  max =  533.99  avg =  526.04

vgg16  min = 1176.82  max = 1200.64  avg = 1187.88
       resnet50_int8  min =  417.30  max =  435.81  avg =  422.36
      squeezenet_ssd  min =  164.88  max =  171.21  avg =  167.51
 squeezenet_ssd_int8  min =  164.78  max =  171.77  avg =  168.36
       mobilenet_ssd  min =  221.41  max =  229.13  avg =  226.18
  mobilenet_ssd_int8  min =  234.15  max =  245.91  avg =  239.01
      mobilenet_yolo  min =  471.34  max =  484.99  avg =  477.15
  mobilenetv2_yolov3  min =  249.14  max =  257.61  avg =  252.54

taimen:/data/local/tmp/ncnnbench $ ./benchncnn 8 1 2 0 1

          squeezenet  min =   18.74  max =   19.89  avg =   19.22
           mobilenet  min =   21.19  max =   25.61  avg =   22.94
        mobilenet_v2  min =   24.15  max =   34.68  avg =   30.12
        mobilenet_v3  min =   25.94  max =   33.15  avg =   30.09
          shufflenet  min =   25.05  max =   31.41  avg =   27.85
       shufflenet_v2  min =   28.82  max =   32.04  avg =   30.95
             mnasnet  min =   21.34  max =   27.69  avg =   24.17
     proxylessnasnet  min =   25.51  max =   30.03  avg =   28.01
     efficientnet_b0  min =   42.94  max =   47.44  avg =   45.28
        regnety_400m  min =   36.36  max =   55.73  avg =   41.82
           blazeface  min =   11.14  max =   13.11  avg =   12.20
           googlenet  min =   49.72  max =   56.92  avg =   51.79
            resnet18  min =   44.63  max =   47.37  avg =   45.86

blazeface  min =   11.14  max =   13.11  avg =   12.20
             alexnet  min =   42.83  max =   46.34  avg =   44.63
               vgg16  min =  568.82  max =  586.75  avg =  578.60
            resnet50  min =  108.63  max =  115.76  avg =  110.38
      squeezenet_ssd  min =   85.22  max =  104.73  avg =   93.14
       mobilenet_ssd  min =   49.91  max =   56.86  avg =   52.33
      mobilenet_yolo  min =   98.76  max =  109.37  avg =  102.27
  mobilenetv2_yolov3  min =   57.49  max =   61.15  avg =   58.74
```

### Qualcomm SDM660 Snapdragon 660 (Kyro260 2.2GHz x 4 + Kyro260 1.84GHz x 4 + Adreno 512)
```
lavender:/data/local/tmp/ncnnbench $ ./benchncnn 8 8 0 -1 1
[0 Adreno (TM) 512]  queueC=0[3]  queueG=0[3]  queueT=0[3]
[0 Adreno (TM) 512]  buglssc=0  bugsbn1=1  buglbia=0  bugihfa=0

```
[0 Adreno (TM) 512]  fp16p=1  fp16s=0  fp16a=0  int8s=0  int8a=0
          squeezenet  min =   29.05  max =   44.86  avg =   33.26
     squeezenet_int8  min =   35.47  max =   37.10  avg =   36.09
           mobilenet  min =   31.59  max =   33.47  avg =   32.33
      mobilenet_int8  min =   77.50  max =   91.15  avg =   82.98
        mobilenet_v2  min =   33.63  max =   35.43  avg =   34.54
        mobilenet_v3  min =   29.97  max =   49.80  avg =   34.81
          shufflenet  min =   28.52  max =   30.09  avg =   29.09
       shufflenet_v2  min =   19.15  max =   21.15  avg =   19.99
             mnasnet  min =   29.91  max =   35.11  avg =   31.46
     proxylessnasnet  min =   33.28  max =  117.09  avg =   55.22

shufflenet_v2  min =   19.15  max =   21.15  avg =   19.99
     efficientnet_b0  min =   52.29  max =   57.93  avg =   55.04
        regnety_400m  min =   96.05  max =  116.42  avg =  102.07
           blazeface  min =    7.98  max =   11.83  avg =    8.89
           googlenet  min =   76.88  max =  103.99  avg =   84.54
      googlenet_int8  min =   97.68  max =  118.56  avg =  104.92
            resnet18  min =   75.93  max =   89.31  avg =   80.00
       resnet18_int8  min =   73.27  max =   80.84  avg =   76.19
             alexnet  min =   90.94  max =  114.57  avg =   96.42
               vgg16  min =  381.30  max =  615.62  avg =  555.96
          vgg16_int8  min =  803.75  max = 1126.53  avg =  886.03
            resnet50  min =  257.38  max =  285.19  avg =  266.59
       resnet50_int8  min =  304.81  max =  338.01  avg =  314.84

vgg16_int8  min =  803.75  max = 1126.53  avg =  886.03
      squeezenet_ssd  min =  117.59  max =  145.79  avg =  123.79
 squeezenet_ssd_int8  min =  132.80  max =  163.00  avg =  149.99
       mobilenet_ssd  min =  103.98  max =  126.90  avg =  113.10
  mobilenet_ssd_int8  min =  167.86  max =  188.46  avg =  180.56
      mobilenet_yolo  min =  201.75  max =  263.92  avg =  240.17
  mobilenetv2_yolov3  min =  143.76  max =  167.77  avg =  151.94

lavender:/data/local/tmp/ncnnbench $ ./benchncnn 4 1 2 -1 1
          squeezenet  min =   69.75  max =   71.33  avg =   70.38

     squeezenet_int8  min =   67.12  max =   68.07  avg =   67.59
           mobilenet  min =  107.65  max =  110.48  avg =  108.82
      mobilenet_int8  min =  163.13  max =  164.74  avg =  164.24
        mobilenet_v2  min =   75.50  max =   77.36  avg =   76.38
        mobilenet_v3  min =   59.05  max =   59.36  avg =   59.23
          shufflenet  min =   38.33  max =   38.74  avg =   38.57
       shufflenet_v2  min =   37.43  max =   38.97  avg =   38.32
             mnasnet  min =   69.29  max =   73.20  avg =   70.73
     proxylessnasnet  min =   80.81  max =   82.66  avg =   81.52
     efficientnet_b0  min =  151.20  max =  152.38  avg =  151.72
        regnety_400m  min =   93.53  max =   94.53  avg =   94.19
           blazeface  min =   12.15  max =   12.82  avg =   12.46

efficientnet_b0  min =  151.20  max =  152.38  avg =  151.72
           googlenet  min =  239.63  max =  242.64  avg =  241.06
      googlenet_int8  min =  214.71  max =  216.53  avg =  215.79
            resnet18  min =  234.20  max =  238.74  avg =  236.90
       resnet18_int8  min =  181.57  max =  183.97  avg =  182.66
             alexnet  min =  205.94  max =  207.44  avg =  206.63
               vgg16  min = 1188.14  max = 1201.95  avg = 1196.93
          vgg16_int8  min = 1081.21  max = 1087.84  avg = 1085.17
            resnet50  min =  556.54  max =  566.68  avg =  561.21
       resnet50_int8  min =  433.19  max =  433.93  avg =  433.48
      squeezenet_ssd  min =  169.02  max =  170.54  avg =  169.73
 squeezenet_ssd_int8  min =  176.28  max =  177.90  avg =  176.87
       mobilenet_ssd  min =  228.15  max =  232.69  avg =  230.38

squeezenet_ssd  min =  169.02  max =  170.54  avg =  169.73
  mobilenet_ssd_int8  min =  236.97  max =  239.69  avg =  238.35
      mobilenet_yolo  min =  493.33  max =  506.34  avg =  499.79
  mobilenetv2_yolov3  min =  252.53  max =  261.58  avg =  256.30

lavender:/data/local/tmp/ncnnbench $ ./benchncnn 4 1 2 0 1
          squeezenet  min =   34.49  max =   34.65  avg =   34.55
           mobilenet  min =   54.45  max =   55.52  avg =   54.75
        mobilenet_v2  min =   39.32  max =   39.58  avg =   39.50
        mobilenet_v3  min =   36.13  max =   36.28  avg =   36.19

mobilenet  min =   54.45  max =   55.52  avg =   54.75
          shufflenet  min =   35.25  max =   35.42  avg =   35.31
       shufflenet_v2  min =   31.38  max =   31.70  avg =   31.53
             mnasnet  min =   40.95  max =   41.32  avg =   41.13
     proxylessnasnet  min =   43.81  max =   44.05  avg =   43.90
     efficientnet_b0  min =   68.34  max =   68.56  avg =   68.47
        regnety_400m  min =   53.89  max =   54.23  avg =   54.02
           blazeface  min =   19.82  max =   27.74  avg =   22.01
           googlenet  min =  119.46  max =  119.98  avg =  119.80
            resnet18  min =  115.56  max =  120.28  avg =  116.88
             alexnet  min =  102.06  max =  105.56  avg =  102.97
               vgg16  min = 1192.29  max = 1202.17  avg = 1197.03
            resnet50  min =  294.87  max =  298.79  avg =  296.05

alexnet  min =  102.06  max =  105.56  avg =  102.97
      squeezenet_ssd  min =  167.85  max =  168.42  avg =  168.09
       mobilenet_ssd  min =  120.30  max =  120.37  avg =  120.34
      mobilenet_yolo  min =  256.60  max =  260.21  avg =  257.54
  mobilenetv2_yolov3  min =  121.48  max =  125.22  avg =  122.53
```

### Qualcomm MSM8996 Pro Snapdragon 821 (Kyro 2.35GHz x 2 + Kyro 2.19GHz x 2)
```
natrium:/data/local/tmp # ./benchncnn 8 4 0 -1 1
          squeezenet  min =   18.46  max =   19.12  avg =   18.78
     squeezenet_int8  min =   16.69  max =   17.22  avg =   16.95
           mobilenet  min =   27.33  max =   28.74  avg =   27.88
      mobilenet_int8  min =   20.14  max =   20.71  avg =   20.46

squeezenet_int8  min =   16.69  max =   17.22  avg =   16.95
        mobilenet_v2  min =   21.94  max =   23.09  avg =   22.38
        mobilenet_v3  min =   18.81  max =   19.45  avg =   19.04
          shufflenet  min =   14.07  max =   14.75  avg =   14.29
       shufflenet_v2  min =   11.52  max =   11.92  avg =   11.71
             mnasnet  min =   20.41  max =   21.75  avg =   20.74
     proxylessnasnet  min =   22.99  max =   23.63  avg =   23.13
     efficientnet_b0  min =   34.74  max =   35.26  avg =   34.91
   efficientnetv2_b0  min =   41.16  max =   41.60  avg =   41.39
        regnety_400m  min =   44.27  max =   45.01  avg =   44.69
           blazeface  min =    4.25  max =    4.71  avg =    4.43
           googlenet  min =   54.88  max =   55.55  avg =   55.12
      googlenet_int8  min =   51.88  max =   52.72  avg =   52.25

blazeface  min =    4.25  max =    4.71  avg =    4.43
            resnet18  min =   44.33  max =   45.44  avg =   44.88
       resnet18_int8  min =   51.24  max =   51.94  avg =   51.54
             alexnet  min =   38.62  max =   39.31  avg =   38.88
               vgg16  min =  242.53  max =  244.23  avg =  243.16
          vgg16_int8  min =  183.15  max =  204.96  avg =  192.16
            resnet50  min =  122.14  max =  124.29  avg =  122.94
       resnet50_int8  min =  116.61  max =  118.47  avg =  117.56
      squeezenet_ssd  min =   47.92  max =   49.01  avg =   48.45
 squeezenet_ssd_int8  min =   43.21  max =   44.45  avg =   43.76
       mobilenet_ssd  min =   56.92  max =   58.21  avg =   57.56
  mobilenet_ssd_int8  min =   42.26  max =   42.92  avg =   42.48
      mobilenet_yolo  min =  126.20  max =  128.50  avg =  127.10

mobilenet_ssd  min =   56.92  max =   58.21  avg =   57.56
  mobilenetv2_yolov3  min =   75.49  max =   76.50  avg =   76.01
         yolov4-tiny  min =   94.24  max =   95.75  avg =   94.83
           nanodet_m  min =   31.30  max =   31.93  avg =   31.62
    yolo-fastest-1.1  min =   16.89  max =   17.56  avg =   17.23
      yolo-fastestv2  min =   12.97  max =   13.50  avg =   13.15

natrium:/data/local/tmp # ./benchncnn 4 1 0 -1 1
          squeezenet  min =   46.27  max =   46.60  avg =   46.45
     squeezenet_int8  min =   41.33  max =   41.73  avg =   41.56
           mobilenet  min =   80.89  max =   81.16  avg =   81.00
      mobilenet_int8  min =   60.33  max =   62.29  avg =   61.33
        mobilenet_v2  min =   51.78  max =   52.02  avg =   51.88

mobilenet  min =   80.89  max =   81.16  avg =   81.00
        mobilenet_v3  min =   43.71  max =   44.17  avg =   43.91
          shufflenet  min =   24.96  max =   25.08  avg =   25.02
       shufflenet_v2  min =   24.09  max =   24.26  avg =   24.17
             mnasnet  min =   51.28  max =   51.42  avg =   51.35
     proxylessnasnet  min =   59.25  max =   59.66  avg =   59.48
     efficientnet_b0  min =   92.16  max =   92.34  avg =   92.22
   efficientnetv2_b0  min =  112.27  max =  113.63  avg =  113.17
        regnety_400m  min =   68.59  max =   68.85  avg =   68.75
           blazeface  min =    7.36  max =    7.83  avg =    7.59
           googlenet  min =  151.15  max =  151.53  avg =  151.37
      googlenet_int8  min =  152.01  max =  158.63  avg =  154.18
            resnet18  min =  121.49  max =  121.90  avg =  121.77

googlenet  min =  151.15  max =  151.53  avg =  151.37
       resnet18_int8  min =  154.54  max =  166.73  avg =  161.30
             alexnet  min =   97.41  max =   97.74  avg =   97.62
               vgg16  min =  674.80  max =  675.86  avg =  675.38
          vgg16_int8  min =  593.42  max =  602.98  avg =  596.93
            resnet50  min =  360.44  max =  364.31  avg =  362.01
       resnet50_int8  min =  371.21  max =  386.24  avg =  381.53
      squeezenet_ssd  min =   97.72  max =   98.32  avg =   98.01
 squeezenet_ssd_int8  min =   98.33  max =   99.15  avg =   98.63
       mobilenet_ssd  min =  161.72  max =  161.89  avg =  161.79
  mobilenet_ssd_int8  min =  122.44  max =  123.38  avg =  123.00
      mobilenet_yolo  min =  367.34  max =  369.59  avg =  368.97
  mobilenetv2_yolov3  min =  190.09  max =  190.77  avg =  190.31

mobilenet_ssd_int8  min =  122.44  max =  123.38  avg =  123.00
         yolov4-tiny  min =  241.59  max =  242.29  avg =  241.81
           nanodet_m  min =   63.03  max =   63.22  avg =   63.12
    yolo-fastest-1.1  min =   29.06  max =   29.22  avg =   29.12
      yolo-fastestv2  min =   22.72  max =   22.80  avg =   22.77
```

### Qualcomm MSM8994 Snapdragon 810 (Cortex-A57 2.0GHz x 4 + Cortex-A53 1.55GHz x 4)
```
angler:/data/local/tmp $ ./benchncnn 8 8 0 -1 1
          squeezenet  min =   25.83  max =   29.17  avg =   27.69
     squeezenet_int8  min =   24.18  max =   26.31  avg =   25.18
           mobilenet  min =   33.94  max =   35.29  avg =   34.44
      mobilenet_int8  min =   24.99  max =   26.12  avg =   25.46

squeezenet_int8  min =   24.18  max =   26.31  avg =   25.18
        mobilenet_v2  min =   32.63  max =   34.44  avg =   33.56
        mobilenet_v3  min =   27.72  max =   30.14  avg =   29.35
          shufflenet  min =   23.23  max =   26.78  avg =   24.58
       shufflenet_v2  min =   21.04  max =   22.25  avg =   21.68
             mnasnet  min =   29.51  max =   31.26  avg =   30.27
     proxylessnasnet  min =   34.21  max =   37.55  avg =   35.20
     efficientnet_b0  min =   54.75  max =   60.45  avg =   56.38
   efficientnetv2_b0  min =   63.60  max =   67.51  avg =   64.81
        regnety_400m  min =   60.80  max =   72.33  avg =   68.27
           blazeface  min =    5.96  max =    7.22  avg =    6.41
           googlenet  min =   80.62  max =   94.46  avg =   86.50
      googlenet_int8  min =   69.05  max =   75.75  avg =   71.47

blazeface  min =    5.96  max =    7.22  avg =    6.41
            resnet18  min =   63.90  max =   75.96  avg =   69.64
       resnet18_int8  min =   46.43  max =   62.23  avg =   53.22
             alexnet  min =   82.67  max =   90.25  avg =   87.03
               vgg16  min =  562.23  max =  636.26  avg =  594.82
          vgg16_int8  min =  303.42  max =  358.03  avg =  325.60
            resnet50  min =  233.47  max =  279.99  avg =  248.49
       resnet50_int8  min =  170.11  max =  198.27  avg =  183.35
      squeezenet_ssd  min =   86.97  max =  112.21  avg =   96.84
 squeezenet_ssd_int8  min =   66.09  max =   77.00  avg =   70.57
       mobilenet_ssd  min =   76.95  max =  101.74  avg =   87.73
  mobilenet_ssd_int8  min =   53.27  max =   60.50  avg =   57.46
      mobilenet_yolo  min =  206.42  max =  260.06  avg =  227.84

mobilenet_ssd  min =   76.95  max =  101.74  avg =   87.73
  mobilenetv2_yolov3  min =  129.32  max =  147.76  avg =  138.90
         yolov4-tiny  min =  184.85  max =  213.03  avg =  203.52
           nanodet_m  min =   47.66  max =   60.55  avg =   53.00

angler:/data/local/tmp # ./benchncnn 4 4 2 -1 1
          squeezenet  min =   41.39  max =   47.64  avg =   43.08
     squeezenet_int8  min =   36.92  max =   37.59  avg =   37.24
           mobilenet  min =   59.04  max =   59.43  avg =   59.22
      mobilenet_int8  min =   44.67  max =   46.60  avg =   45.58
        mobilenet_v2  min =   43.38  max =   43.71  avg =   43.62
        mobilenet_v3  min =   37.57  max =   37.82  avg =   37.65
          shufflenet  min =   30.67  max =   30.86  avg =   30.76

mobilenet_v2  min =   43.38  max =   43.71  avg =   43.62
       shufflenet_v2  min =   27.80  max =   28.12  avg =   27.97
             mnasnet  min =   42.99  max =   46.41  avg =   44.21
     proxylessnasnet  min =   51.26  max =   53.52  avg =   52.04
     efficientnet_b0  min =   81.58  max =   82.30  avg =   82.03
   efficientnetv2_b0  min =   94.01  max =   94.48  avg =   94.27
        regnety_400m  min =   82.38  max =   83.86  avg =   82.95
           blazeface  min =   10.02  max =   10.42  avg =   10.18
           googlenet  min =  125.47  max =  126.72  avg =  125.92
      googlenet_int8  min =  109.92  max =  111.65  avg =  110.44
            resnet18  min =  110.14  max =  111.95  avg =  110.76
       resnet18_int8  min =   78.21  max =   79.65  avg =   79.07
             alexnet  min =   78.09  max =   80.34  avg =   78.87

resnet18  min =  110.14  max =  111.95  avg =  110.76
               vgg16  min =  486.69  max =  494.97  avg =  490.35
          vgg16_int8  min =  370.66  max =  377.64  avg =  373.78
            resnet50  min =  272.31  max =  278.64  avg =  274.10
       resnet50_int8  min =  215.57  max =  218.55  avg =  217.27
      squeezenet_ssd  min =  112.98  max =  114.75  avg =  113.60
 squeezenet_ssd_int8  min =   91.85  max =   94.82  avg =   93.13
       mobilenet_ssd  min =  115.18  max =  116.56  avg =  115.95
  mobilenet_ssd_int8  min =   90.95  max =   92.21  avg =   91.39
      mobilenet_yolo  min =  255.07  max =  259.01  avg =  256.18
  mobilenetv2_yolov3  min =  155.52  max =  156.58  avg =  156.09
         yolov4-tiny  min =  231.89  max =  234.14  avg =  232.97
           nanodet_m  min =   72.74  max =   74.71  avg =   73.52

mobilenetv2_yolov3  min =  155.52  max =  156.58  avg =  156.09
    yolo-fastest-1.1  min =   35.25  max =   36.51  avg =   35.77
      yolo-fastestv2  min =   29.94  max =   31.09  avg =   30.75

angler:/data/local/tmp # ./benchncnn 4 1 2 -1 1
          squeezenet  min =   70.83  max =   72.68  avg =   71.77
     squeezenet_int8  min =   59.27  max =   59.60  avg =   59.51
           mobilenet  min =  110.70  max =  112.72  avg =  111.48
      mobilenet_int8  min =   79.69  max =   80.01  avg =   79.81
        mobilenet_v2  min =   77.85  max =   78.19  avg =   78.03
        mobilenet_v3  min =   63.49  max =   63.92  avg =   63.73
          shufflenet  min =   41.43  max =   41.60  avg =   41.49
       shufflenet_v2  min =   37.49  max =   38.26  avg =   37.97

mobilenet_v3  min =   63.49  max =   63.92  avg =   63.73
             mnasnet  min =   73.91  max =   75.91  avg =   74.59
     proxylessnasnet  min =   94.13  max =   94.53  avg =   94.37
     efficientnet_b0  min =  161.91  max =  162.38  avg =  162.10
   efficientnetv2_b0  min =  179.33  max =  180.26  avg =  179.67
        regnety_400m  min =  100.35  max =  100.76  avg =  100.53
           blazeface  min =   12.57  max =   12.76  avg =   12.66
           googlenet  min =  232.77  max =  233.08  avg =  232.91
      googlenet_int8  min =  203.39  max =  205.25  avg =  204.77
            resnet18  min =  182.58  max =  183.17  avg =  182.91
       resnet18_int8  min =  150.40  max =  152.07  avg =  151.35
             alexnet  min =  147.27  max =  149.00  avg =  148.06
               vgg16  min =  986.93  max =  988.35  avg =  987.47

resnet18_int8  min =  150.40  max =  152.07  avg =  151.35
          vgg16_int8  min =  816.37  max =  819.93  avg =  817.79
            resnet50  min =  502.77  max =  510.88  avg =  508.53
       resnet50_int8  min =  393.33  max =  398.07  avg =  395.86
      squeezenet_ssd  min =  175.01  max =  175.61  avg =  175.32
 squeezenet_ssd_int8  min =  145.19  max =  145.94  avg =  145.66
       mobilenet_ssd  min =  231.04  max =  231.25  avg =  231.13
  mobilenet_ssd_int8  min =  159.81  max =  160.52  avg =  160.13
      mobilenet_yolo  min =  517.86  max =  523.71  avg =  521.85
  mobilenetv2_yolov3  min =  275.84  max =  279.16  avg =  277.13
         yolov4-tiny  min =  363.71  max =  366.14  avg =  364.56
           nanodet_m  min =   93.90  max =   95.09  avg =   94.40
    yolo-fastest-1.1  min =   45.94  max =   46.09  avg =   46.01

yolov4-tiny  min =  363.71  max =  366.14  avg =  364.56
      yolo-fastestv2  min =   38.23  max =   38.33  avg =   38.29

angler:/data/local/tmp $ ./benchncnn 4 1 2 0 1
[0 Adreno (TM) 430]  queueC=0[3]  queueG=0[3]  queueT=0[3]
[0 Adreno (TM) 430]  buglssc=0  bugsbn1=1  buglbia=0  bugihfa=0
[0 Adreno (TM) 430]  fp16p=1  fp16s=0  fp16a=0  int8s=0  int8a=0
          squeezenet  min =   39.49  max =   41.93  avg =   40.62
           mobilenet  min =   60.30  max =   61.81  avg =   60.88
        mobilenet_v2  min =   45.38  max =   47.10  avg =   45.88
        mobilenet_v3  min =   45.97  max =   47.39  avg =   46.69
          shufflenet  min =   29.12  max =   31.02  avg =   29.91
       shufflenet_v2  min =   47.58  max =   50.06  avg =   48.26

mobilenet_v3  min =   45.97  max =   47.39  avg =   46.69
             mnasnet  min =   47.84  max =   49.17  avg =   48.26
     proxylessnasnet  min =   49.51  max =   51.03  avg =   49.97
     efficientnet_b0  min =  100.56  max =  105.60  avg =  102.45
        regnety_400m  min =   59.67  max =   61.24  avg =   60.56
           blazeface  min =   13.87  max =   13.98  avg =   13.93
           googlenet  min =  131.26  max =  136.33  avg =  133.40
            resnet18  min =  116.38  max =  117.92  avg =  116.93
             alexnet  min =   72.59  max =   73.94  avg =   73.29
               vgg16  min = 1090.07  max = 1101.71  avg = 1096.34
            resnet50  min =  299.76  max =  300.78  avg =  300.40
      squeezenet_ssd  min =  181.95  max =  182.83  avg =  182.39
       mobilenet_ssd  min =  148.44  max =  151.07  avg =  149.75

resnet50  min =  299.76  max =  300.78  avg =  300.40
      mobilenet_yolo  min =  284.46  max =  285.74  avg =  285.39
  mobilenetv2_yolov3  min =  140.28  max =  148.62  avg =  144.83
```

### Qualcomm MSM8916 Snapdragon 410 (Cortex-A53 1.2GHz x 4)
```
HM2014812:/data/local/tmp # ./benchncnn 8 4 0 -1 1
          squeezenet  min =   65.45  max =   73.59  avg =   68.10
     squeezenet_int8  min =   59.39  max =   65.54  avg =   61.14
           mobilenet  min =   86.69  max =   94.10  avg =   90.03
      mobilenet_int8  min =   62.22  max =   69.67  avg =   64.13
        mobilenet_v2  min =   77.98  max =   89.53  avg =   82.00
        mobilenet_v3  min =   62.17  max =   68.31  avg =   63.90
          shufflenet  min =   47.52  max =   53.76  avg =   49.92

mobilenet_v2  min =   77.98  max =   89.53  avg =   82.00
       shufflenet_v2  min =   39.77  max =   46.08  avg =   40.66
             mnasnet  min =   69.27  max =   75.73  avg =   71.73
     proxylessnasnet  min =   78.72  max =   85.37  avg =   81.33
     efficientnet_b0  min =  126.62  max =  136.67  avg =  130.69
   efficientnetv2_b0  min =  143.24  max =  150.97  avg =  146.89
        regnety_400m  min =  108.79  max =  116.22  avg =  112.99
           blazeface  min =   14.85  max =   15.02  avg =   14.94
           googlenet  min =  180.91  max =  190.37  avg =  186.36
      googlenet_int8  min =  160.07  max =  170.86  avg =  165.05
            resnet18  min =  137.91  max =  155.37  avg =  144.99
       resnet18_int8  min =  104.34  max =  110.20  avg =  106.76
             alexnet  min =  105.30  max =  114.73  avg =  109.53

resnet18  min =  137.91  max =  155.37  avg =  144.99
               vgg16  min =  829.16  max =  942.94  avg =  853.28
          vgg16_int8  min =  515.61  max =  547.32  avg =  526.50
            resnet50  min =  380.46  max =  443.90  avg =  393.71
       resnet50_int8  min =  318.06  max =  327.13  avg =  323.23
      squeezenet_ssd  min =  178.22  max =  189.02  avg =  184.51
 squeezenet_ssd_int8  min =  153.75  max =  163.44  avg =  158.05
       mobilenet_ssd  min =  189.45  max =  195.17  avg =  193.10
  mobilenet_ssd_int8  min =  132.59  max =  139.63  avg =  137.23
      mobilenet_yolo  min =  404.52  max =  414.20  avg =  409.97
  mobilenetv2_yolov3  min =  271.33  max =  279.98  avg =  275.08
         yolov4-tiny  min =  349.36  max =  372.54  avg =  357.98
           nanodet_m  min =  103.01  max =  111.71  avg =  105.82

mobilenetv2_yolov3  min =  271.33  max =  279.98  avg =  275.08

HM2014812:/data/local/tmp # ./benchncnn 4 1 0 -1 1
          squeezenet  min =  147.48  max =  149.35  avg =  148.40
     squeezenet_int8  min =  143.20  max =  144.55  avg =  143.98
           mobilenet  min =  243.78  max =  244.33  avg =  244.08
      mobilenet_int8  min =  206.23  max =  207.13  avg =  206.55
        mobilenet_v2  min =  168.04  max =  170.37  avg =  169.06
        mobilenet_v3  min =  147.10  max =  147.91  avg =  147.55
          shufflenet  min =   88.47  max =   89.31  avg =   88.85
       shufflenet_v2  min =   84.47  max =   84.80  avg =   84.60
             mnasnet  min =  162.81  max =  163.93  avg =  163.22
     proxylessnasnet  min =  208.18  max =  209.15  avg =  208.61

shufflenet_v2  min =   84.47  max =   84.80  avg =   84.60
     efficientnet_b0  min =  370.06  max =  371.14  avg =  370.64
   efficientnetv2_b0  min =  418.28  max =  429.68  avg =  423.01
        regnety_400m  min =  216.42  max =  217.19  avg =  216.71
           blazeface  min =   27.63  max =   28.67  avg =   28.00
           googlenet  min =  525.25  max =  528.83  avg =  526.23
      googlenet_int8  min =  469.78  max =  472.51  avg =  470.76
            resnet18  min =  396.46  max =  399.66  avg =  397.57
       resnet18_int8  min =  324.07  max =  326.64  avg =  325.34
             alexnet  min =  362.44  max =  363.02  avg =  362.68
               vgg16  min = 2174.86  max = 2252.92  avg = 2215.62
          vgg16_int8  min = 1726.07  max = 1732.69  avg = 1729.18
            resnet50  min = 1136.96  max = 1142.94  avg = 1139.91

vgg16  min = 2174.86  max = 2252.92  avg = 2215.62
       resnet50_int8  min =  977.73  max =  983.64  avg =  980.71
      squeezenet_ssd  min =  350.46  max =  353.35  avg =  351.37
 squeezenet_ssd_int8  min =  333.91  max =  336.59  avg =  334.77
       mobilenet_ssd  min =  513.18  max =  519.05  avg =  516.22
  mobilenet_ssd_int8  min =  424.37  max =  426.89  avg =  426.03
      mobilenet_yolo  min = 1143.20  max = 1145.04  avg = 1144.31
  mobilenetv2_yolov3  min =  617.45  max =  619.30  avg =  618.37
         yolov4-tiny  min =  839.32  max =  847.57  avg =  844.61
           nanodet_m  min =  208.41  max =  211.31  avg =  210.03
```

### Raspberry Pi 3 Model B+ Broadcom BCM2837B0, Cortex-A53 (ARMv8) (1.4GHz x 4)
```
pi@raspberrypi:~/ncnn/build/benchmark $ ./benchncnn 4 4 0 -1 1

```

```
          squeezenet  min =   84.74  max =   85.60  avg =   85.22
     squeezenet_int8  min =   74.48  max =   74.80  avg =   74.68
           mobilenet  min =  107.84  max =  110.13  avg =  108.66
      mobilenet_int8  min =   66.91  max =   67.12  avg =   67.03
        mobilenet_v2  min =  110.64  max =  112.73  avg =  111.68
        mobilenet_v3  min =   85.78  max =   86.74  avg =   86.44
          shufflenet  min =   58.38  max =   60.32  avg =   59.33
       shufflenet_v2  min =   46.76  max =   47.53  avg =   47.19
             mnasnet  min =   95.53  max =   95.88  avg =   95.78
     proxylessnasnet  min =  102.24  max =  105.58  avg =  103.38
     efficientnet_b0  min =  134.87  max =  136.98  avg =  135.86

mnasnet  min =   95.53  max =   95.88  avg =   95.78
   efficientnetv2_b0  min =  146.62  max =  148.06  avg =  147.13
        regnety_400m  min =  118.60  max =  119.51  avg =  119.03
           blazeface  min =   15.42  max =   15.61  avg =   15.52
           googlenet  min =  223.78  max =  224.85  avg =  224.22
      googlenet_int8  min =  188.23  max =  190.15  avg =  189.21
            resnet18  min =  270.86  max =  272.66  avg =  271.93
       resnet18_int8  min =  159.57  max =  160.39  avg =  160.07
             alexnet  min =  157.79  max =  160.77  avg =  159.09
            resnet50  min =  583.57  max =  591.41  avg =  587.42
       resnet50_int8  min =  383.96  max =  401.37  avg =  391.87
      squeezenet_ssd  min =  247.90  max =  249.77  avg =  248.98
 squeezenet_ssd_int8  min =  191.65  max =  192.81  avg =  192.17

resnet50_int8  min =  383.96  max =  401.37  avg =  391.87
       mobilenet_ssd  min =  240.11  max =  241.02  avg =  240.62
  mobilenet_ssd_int8  min =  136.30  max =  137.26  avg =  136.73
      mobilenet_yolo  min =  523.59  max =  539.91  avg =  529.98
  mobilenetv2_yolov3  min =  356.44  max =  366.85  avg =  362.06
         yolov4-tiny  min =  410.25  max =  422.18  avg =  417.17
           nanodet_m  min =  114.98  max =  115.83  avg =  115.40
    yolo-fastest-1.1  min =   79.85  max =   80.83  avg =   80.28
      yolo-fastestv2  min =   62.36  max =   62.91  avg =   62.60
          FastestDet  min =   67.11  max =   68.51  avg =   67.98

pi@raspberrypi:~/ncnn/build/benchmark $ ./benchncnn 4 1 0 -1 1
          squeezenet  min =  125.34  max =  125.81  avg =  125.58

     squeezenet_int8  min =  135.56  max =  136.34  avg =  135.98
           mobilenet  min =  204.62  max =  207.06  avg =  205.65
      mobilenet_int8  min =  181.34  max =  182.46  avg =  181.91
        mobilenet_v2  min =  158.69  max =  158.94  avg =  158.80
        mobilenet_v3  min =  127.13  max =  127.31  avg =  127.23
          shufflenet  min =   84.64  max =   85.29  avg =   84.89
       shufflenet_v2  min =   74.28  max =   74.64  avg =   74.44
             mnasnet  min =  148.12  max =  148.65  avg =  148.42
     proxylessnasnet  min =  199.56  max =  201.99  avg =  200.42
     efficientnet_b0  min =  240.94  max =  241.75  avg =  241.27
   efficientnetv2_b0  min =  270.71  max =  270.90  avg =  270.83
        regnety_400m  min =  186.89  max =  187.08  avg =  187.01

efficientnet_b0  min =  240.94  max =  241.75  avg =  241.27
           blazeface  min =   22.75  max =   23.24  avg =   22.95
           googlenet  min =  450.64  max =  450.96  avg =  450.79
      googlenet_int8  min =  424.66  max =  426.83  avg =  425.78
            resnet18  min =  379.21  max =  380.01  avg =  379.57
       resnet18_int8  min =  312.23  max =  313.21  avg =  312.68
             alexnet  min =  270.13  max =  270.88  avg =  270.55
            resnet50  min =  977.51  max =  981.89  avg =  979.75
       resnet50_int8  min =  890.77  max =  896.89  avg =  893.83
      squeezenet_ssd  min =  331.52  max =  333.47  avg =  332.46
 squeezenet_ssd_int8  min =  317.71  max =  319.64  avg =  318.62
       mobilenet_ssd  min =  425.42  max =  426.52  avg =  425.93
  mobilenet_ssd_int8  min =  370.17  max =  370.90  avg =  370.66

squeezenet_ssd_int8  min =  317.71  max =  319.64  avg =  318.62
      mobilenet_yolo  min =  930.40  max =  932.24  avg =  931.46
  mobilenetv2_yolov3  min =  534.79  max =  543.56  avg =  539.20
         yolov4-tiny  min =  675.33  max =  676.83  avg =  676.14
           nanodet_m  min =  178.13  max =  178.98  avg =  178.64
    yolo-fastest-1.1  min =  100.83  max =  101.96  avg =  101.49
      yolo-fastestv2  min =   79.73  max =   79.94  avg =   79.84
          FastestDet  min =   89.09  max =   90.07  avg =   89.78
```

### Raspberry Pi 4 Model B Broadcom BCM2711B0, Cortex-A72 (ARMv8) (1.8GHz x 4)
```
pi@raspberrypi:~/ncnn/build/benchmark $ ./benchncnn 10 4 0 -1 1
          squeezenet  min =   46.28  max =   46.91  avg =   46.65

     squeezenet_int8  min =   42.18  max =   44.98  avg =   42.59
           mobilenet  min =   60.74  max =   61.79  avg =   61.17
      mobilenet_int8  min =   34.19  max =   34.55  avg =   34.37
        mobilenet_v2  min =   61.63  max =   62.02  avg =   61.88
        mobilenet_v3  min =   47.08  max =   48.40  avg =   47.53
          shufflenet  min =   32.91  max =   33.30  avg =   33.09
       shufflenet_v2  min =   24.37  max =   24.73  avg =   24.56
             mnasnet  min =   51.80  max =   52.14  avg =   51.98
     proxylessnasnet  min =   53.02  max =   53.58  avg =   53.32
     efficientnet_b0  min =   73.92  max =   74.44  avg =   74.19
   efficientnetv2_b0  min =   79.10  max =   79.60  avg =   79.34
        regnety_400m  min =   65.27  max =   66.12  avg =   65.70

efficientnet_b0  min =   73.92  max =   74.44  avg =   74.19
           blazeface  min =    8.62  max =    8.75  avg =    8.69
           googlenet  min =  113.74  max =  115.14  avg =  114.35
      googlenet_int8  min =  100.87  max =  101.71  avg =  101.25
            resnet18  min =  122.27  max =  125.39  avg =  123.12
       resnet18_int8  min =   82.19  max =   94.12  avg =   83.92
             alexnet  min =   75.75  max =   78.08  avg =   76.40
               vgg16  min =  541.66  max =  552.56  avg =  547.09
          vgg16_int8  min =  391.44  max =  395.73  avg =  394.23
            resnet50  min =  261.90  max =  263.91  avg =  262.83
       resnet50_int8  min =  195.60  max =  198.08  avg =  196.65
      squeezenet_ssd  min =  127.01  max =  129.85  avg =  127.61
 squeezenet_ssd_int8  min =  104.98  max =  107.67  avg =  105.47

resnet50_int8  min =  195.60  max =  198.08  avg =  196.65
       mobilenet_ssd  min =  120.43  max =  123.28  avg =  121.46
  mobilenet_ssd_int8  min =   70.70  max =   72.85  avg =   71.14
      mobilenet_yolo  min =  270.89  max =  273.42  avg =  272.33
  mobilenetv2_yolov3  min =  183.85  max =  185.73  avg =  184.88
         yolov4-tiny  min =  205.95  max =  209.90  avg =  207.22
           nanodet_m  min =   68.08  max =   68.69  avg =   68.38
    yolo-fastest-1.1  min =   47.97  max =   48.20  avg =   48.06
      yolo-fastestv2  min =   37.17  max =   37.69  avg =   37.47
  vision_transformer  min = 1872.31  max = 1964.95  avg = 1909.21
          FastestDet  min =   38.39  max =   39.17  avg =   38.69

pi@raspberrypi:~/ncnn/build/benchmark $ ./benchncnn 10 1 0 -1 1

FastestDet  min =   38.39  max =   39.17  avg =   38.69

          squeezenet  min =   73.35  max =   75.10  avg =   73.96
     squeezenet_int8  min =   69.17  max =   69.66  avg =   69.42
           mobilenet  min =  123.76  max =  125.35  avg =  124.32
      mobilenet_int8  min =   84.66  max =   85.24  avg =   84.82
        mobilenet_v2  min =   92.98  max =   94.05  avg =   93.48
        mobilenet_v3  min =   72.48  max =   73.14  avg =   72.81
          shufflenet  min =   47.17  max =   47.83  avg =   47.51
       shufflenet_v2  min =   41.62  max =   42.60  avg =   42.12
             mnasnet  min =   83.60  max =   84.35  avg =   83.98
     proxylessnasnet  min =   98.48  max =   99.33  avg =   98.78
     efficientnet_b0  min =  129.45  max =  130.02  avg =  129.73

mnasnet  min =   83.60  max =   84.35  avg =   83.98
   efficientnetv2_b0  min =  155.06  max =  156.70  avg =  155.76
        regnety_400m  min =  105.39  max =  106.03  avg =  105.70
           blazeface  min =   12.54  max =   12.84  avg =   12.65
           googlenet  min =  235.38  max =  236.34  avg =  235.94
      googlenet_int8  min =  209.63  max =  210.39  avg =  210.00
            resnet18  min =  190.80  max =  191.43  avg =  191.10
       resnet18_int8  min =  157.92  max =  158.97  avg =  158.50
             alexnet  min =  139.34  max =  139.44  avg =  139.40
               vgg16  min = 1066.58  max = 1079.30  avg = 1071.85
          vgg16_int8  min =  866.15  max =  873.75  avg =  869.84
            resnet50  min =  533.15  max =  535.12  avg =  534.11
       resnet50_int8  min =  423.72  max =  424.24  avg =  423.96

vgg16_int8  min =  866.15  max =  873.75  avg =  869.84
      squeezenet_ssd  min =  178.90  max =  179.53  avg =  179.30
 squeezenet_ssd_int8  min =  157.05  max =  159.06  avg =  157.89
       mobilenet_ssd  min =  250.71  max =  251.26  avg =  251.00
  mobilenet_ssd_int8  min =  170.21  max =  170.96  avg =  170.56
      mobilenet_yolo  min =  557.48  max =  560.08  avg =  558.80
  mobilenetv2_yolov3  min =  301.60  max =  307.98  avg =  306.52
         yolov4-tiny  min =  370.55  max =  375.69  avg =  372.99
           nanodet_m  min =  103.05  max =  103.74  avg =  103.45
    yolo-fastest-1.1  min =   56.58  max =   57.44  avg =   57.01
      yolo-fastestv2  min =   46.69  max =   47.34  avg =   47.03
  vision_transformer  min = 6605.19  max = 6606.66  avg = 6605.73
          FastestDet  min =   52.11  max =   52.97  avg =   52.61
```

vision_transformer  min = 6605.19  max = 6606.66  avg = 6605.73
```

### Raspberry Pi Zero 2 W Broadcom BCM2710A1, Cortex-A53 (ARMv8) (1.0GHz x 4)

```
          squeezenet  min =  119.52  max =  120.29  avg =  119.93
     squeezenet_int8  min =   96.32  max =   96.96  avg =   96.55
           mobilenet  min =  162.60  max =  165.49  avg =  163.19
      mobilenet_int8  min =   90.78  max =   91.39  avg =   91.03
        mobilenet_v2  min =  145.71  max =  148.83  avg =  147.39
        mobilenet_v3  min =  113.89  max =  151.95  avg =  119.04
          shufflenet  min =   72.72  max =   73.27  avg =   72.96
       shufflenet_v2  min =   63.64  max =   64.50  avg =   64.13
             mnasnet  min =  126.07  max =  126.93  avg =  126.53
     proxylessnasnet  min =  139.90  max =  140.84  avg =  140.35

shufflenet_v2  min =   63.64  max =   64.50  avg =   64.13
     efficientnet_b0  min =  201.88  max =  202.55  avg =  202.14
   efficientnetv2_b0  min =  227.22  max =  228.84  avg =  228.09
        regnety_400m  min =  156.49  max =  157.47  avg =  156.96
           blazeface  min =   22.79  max =   23.28  avg =   23.10
           googlenet  min =  323.74  max =  324.90  avg =  324.45
      googlenet_int8  min =  250.86  max =  252.82  avg =  251.63
            resnet18  min =  351.37  max =  355.67  avg =  353.45
       resnet18_int8  min =  194.83  max =  196.68  avg =  195.51
             alexnet  min =  271.18  max =  273.53  avg =  272.18
            resnet50  min =  777.44  max =  797.47  avg =  782.63
       resnet50_int8  min =  496.78  max =  498.86  avg =  497.57
      squeezenet_ssd  min =  376.10  max =  382.41  avg =  379.13

resnet50  min =  777.44  max =  797.47  avg =  782.63
 squeezenet_ssd_int8  min =  255.99  max =  257.57  avg =  256.78
       mobilenet_ssd  min =  338.64  max =  339.93  avg =  339.50
  mobilenet_ssd_int8  min =  190.24  max =  190.68  avg =  190.48
      mobilenet_yolo  min =  746.83  max =  748.14  avg =  747.53
  mobilenetv2_yolov3  min =  487.99  max =  491.18  avg =  489.37
         yolov4-tiny  min =  644.73  max =  652.24  avg =  646.64
           nanodet_m  min =  165.27  max =  167.12  avg =  166.27
    yolo-fastest-1.1  min =   98.74  max =  100.02  avg =   99.17
      yolo-fastestv2  min =   80.52  max =   81.86  avg =   81.29

          squeezenet  min =  240.53  max =  241.07  avg =  240.77

     squeezenet_int8  min =  212.63  max =  213.23  avg =  212.94
           mobilenet  min =  393.79  max =  394.04  avg =  393.94
      mobilenet_int8  min =  286.58  max =  286.95  avg =  286.75
        mobilenet_v2  min =  273.97  max =  274.51  avg =  274.23
        mobilenet_v3  min =  233.77  max =  234.59  avg =  234.20
          shufflenet  min =  133.05  max =  133.36  avg =  133.23
       shufflenet_v2  min =  128.86  max =  129.47  avg =  129.18
             mnasnet  min =  265.70  max =  266.17  avg =  265.93
     proxylessnasnet  min =  329.78  max =  330.54  avg =  330.13
     efficientnet_b0  min =  518.42  max =  519.38  avg =  519.00
   efficientnetv2_b0  min =  594.37  max =  595.17  avg =  594.74
        regnety_400m  min =  329.53  max =  330.44  avg =  329.87

efficientnet_b0  min =  518.42  max =  519.38  avg =  519.00
           blazeface  min =   42.24  max =   45.56  avg =   43.96
           googlenet  min =  780.05  max =  780.63  avg =  780.39
      googlenet_int8  min =  663.83  max =  664.43  avg =  664.15
            resnet18  min =  653.62  max =  657.59  avg =  654.69
       resnet18_int8  min =  479.03  max =  479.72  avg =  479.40
             alexnet  min =  687.99  max =  690.34  avg =  689.15
            resnet50  min = 1800.97  max = 1806.11  avg = 1802.79
       resnet50_int8  min = 1311.68  max = 1314.56  avg = 1313.15
      squeezenet_ssd  min =  563.63  max =  565.57  avg =  564.44
 squeezenet_ssd_int8  min =  481.24  max =  483.97  avg =  482.20
       mobilenet_ssd  min =  799.21  max =  829.10  avg =  803.56
  mobilenet_ssd_int8  min =  568.11  max =  568.88  avg =  568.42

squeezenet_ssd_int8  min =  481.24  max =  483.97  avg =  482.20
      mobilenet_yolo  min = 1815.60  max = 1816.44  avg = 1815.93
  mobilenetv2_yolov3  min =  951.34  max =  952.15  avg =  951.72
         yolov4-tiny  min = 1258.21  max = 1259.49  avg = 1258.66
           nanodet_m  min =  301.04  max =  304.09  avg =  301.70
    yolo-fastest-1.1  min =  155.04  max =  155.98  avg =  155.53
      yolo-fastestv2  min =  126.77  max =  127.40  avg =  127.05
```

### Banana Pi M2 Zero 2 AllWinner H2+, Cortex-A7 (ARMv7-A) (1.2GHz x 4)

```
          squeezenet  min =  230.97  max =  232.18  avg =  231.49
     squeezenet_int8  min =  171.12  max =  172.87  avg =  171.68
           mobilenet  min =  327.65  max =  340.92  avg =  329.88

squeezenet  min =  230.97  max =  232.18  avg =  231.49
      mobilenet_int8  min =  166.58  max =  169.55  avg =  167.47
        mobilenet_v2  min =  276.81  max =  278.67  avg =  277.55
        mobilenet_v3  min =  220.74  max =  225.14  avg =  222.08
          shufflenet  min =  147.97  max =  157.68  avg =  149.40
       shufflenet_v2  min =  146.56  max =  154.90  avg =  148.25
             mnasnet  min =  243.06  max =  244.47  avg =  243.80
     proxylessnasnet  min =  260.38  max =  261.47  avg =  260.66
     efficientnet_b0  min =  368.98  max =  371.03  avg =  369.96
   efficientnetv2_b0  min =  433.96  max =  459.25  avg =  437.52
        regnety_400m  min =  307.53  max =  312.29  avg =  308.68
           blazeface  min =   46.54  max =   47.35  avg =   46.98
           googlenet  min =  647.86  max =  669.20  avg =  651.19

regnety_400m  min =  307.53  max =  312.29  avg =  308.68
      googlenet_int8  min =  439.90  max =  442.35  avg =  441.38
            resnet18  min =  642.53  max =  856.58  avg =  698.28
       resnet18_int8  min =  352.10  max =  354.51  avg =  353.44
             alexnet  min =  593.16  max =  624.20  avg =  598.66
            resnet50  min = 1556.12  max = 1782.22  avg = 1606.86
       resnet50_int8  min =  911.63  max =  999.42  avg =  924.37
      squeezenet_ssd  min =  653.85  max =  658.07  avg =  655.19
 squeezenet_ssd_int8  min =  456.26  max =  467.76  avg =  459.87
       mobilenet_ssd  min =  671.93  max =  682.64  avg =  674.88
  mobilenet_ssd_int8  min =  347.18  max =  349.07  avg =  347.81
      mobilenet_yolo  min = 1471.16  max = 1492.65  avg = 1479.30
  mobilenetv2_yolov3  min =  895.90  max =  906.60  avg =  899.74

mobilenet_ssd_int8  min =  347.18  max =  349.07  avg =  347.81
         yolov4-tiny  min = 1178.53  max = 1205.79  avg = 1183.98
           nanodet_m  min =  358.89  max =  366.07  avg =  362.20
    yolo-fastest-1.1  min =  189.93  max =  192.18  avg =  190.91
      yolo-fastestv2  min =  158.60  max =  161.33  avg =  159.43

          squeezenet  min =  602.97  max =  604.97  avg =  603.46
     squeezenet_int8  min =  431.18  max =  432.42  avg =  431.77
           mobilenet  min =  971.52  max =  986.64  avg =  974.04
      mobilenet_int8  min =  556.74  max =  556.98  avg =  556.84
        mobilenet_v2  min =  682.85  max =  684.17  avg =  683.34
        mobilenet_v3  min =  585.10  max =  585.76  avg =  585.57

mobilenet_int8  min =  556.74  max =  556.98  avg =  556.84
          shufflenet  min =  340.64  max =  342.63  avg =  341.26
       shufflenet_v2  min =  322.41  max =  324.13  avg =  323.35
             mnasnet  min =  644.30  max =  645.93  avg =  644.71
     proxylessnasnet  min =  732.50  max =  733.30  avg =  732.96
     efficientnet_b0  min = 1084.70  max = 1094.98  avg = 1086.52
   efficientnetv2_b0  min = 1282.27  max = 1283.67  avg = 1282.60
        regnety_400m  min =  764.60  max =  768.54  avg =  765.30
           blazeface  min =  100.48  max =  106.28  avg =  103.33
           googlenet  min = 1878.69  max = 1883.96  avg = 1880.76
      googlenet_int8  min = 1274.31  max = 1296.02  avg = 1279.59
            resnet18  min = 1837.91  max = 1843.95  avg = 1839.17
       resnet18_int8  min = 1011.98  max = 1014.43  avg = 1013.01

googlenet_int8  min = 1274.31  max = 1296.02  avg = 1279.59
             alexnet  min = 1997.59  max = 2001.81  avg = 1999.42
            resnet50  min = 4844.31  max = 4857.05  avg = 4847.80
       resnet50_int8  min = 2792.59  max = 2810.08  avg = 2797.30
      squeezenet_ssd  min = 1438.96  max = 1443.31  avg = 1441.09
 squeezenet_ssd_int8  min = 1046.76  max = 1053.00  avg = 1049.22
       mobilenet_ssd  min = 2018.66  max = 2023.70  avg = 2019.67
  mobilenet_ssd_int8  min = 1129.16  max = 1130.62  avg = 1129.82
      mobilenet_yolo  min = 4724.90  max = 4728.57  avg = 4726.41
  mobilenetv2_yolov3  min = 2410.67  max = 2427.95  avg = 2413.89
         yolov4-tiny  min = 3177.27  max = 3185.52  avg = 3179.71
           nanodet_m  min =  761.38  max =  768.79  avg =  766.53
    yolo-fastest-1.1  min =  391.82  max =  393.32  avg =  392.39

yolov4-tiny  min = 3177.27  max = 3185.52  avg = 3179.71
      yolo-fastestv2  min =  316.93  max =  319.86  avg =  318.33
```

### Khadas VIM3, Amlogic A311D (Cortex-A73 2.2GHz x 4 + Cortex-A53 1.8GHz x 2)

```
vim3:/data/local/tmp # ./benchncnn 8 4 2 -1 1
          squeezenet  min =   30.98  max =   31.26  avg =   31.09
     squeezenet_int8  min =   24.70  max =   24.84  avg =   24.78
           mobilenet  min =   42.57  max =   43.37  avg =   42.96
      mobilenet_int8  min =   22.33  max =   22.52  avg =   22.44
        mobilenet_v2  min =   39.36  max =   39.77  avg =   39.56
        mobilenet_v3  min =   30.13  max =   30.45  avg =   30.28
          shufflenet  min =   21.62  max =   21.94  avg =   21.80

mobilenet_v2  min =   39.36  max =   39.77  avg =   39.56
       shufflenet_v2  min =   18.83  max =   19.24  avg =   19.05
             mnasnet  min =   33.54  max =   34.08  avg =   33.80
     proxylessnasnet  min =   35.81  max =   36.05  avg =   35.95
     efficientnet_b0  min =   53.82  max =   54.44  avg =   54.21
   efficientnetv2_b0  min =   62.20  max =   62.60  avg =   62.43
        regnety_400m  min =   48.82  max =   49.27  avg =   49.05
           blazeface  min =    6.34  max =    6.51  avg =    6.43
           googlenet  min =   81.96  max =   82.53  avg =   82.23
      googlenet_int8  min =   64.42  max =   65.00  avg =   64.77
            resnet18  min =   77.00  max =   77.83  avg =   77.46
       resnet18_int8  min =   48.91  max =   49.14  avg =   49.05
             alexnet  min =   60.43  max =   60.93  avg =   60.69

resnet18  min =   77.00  max =   77.83  avg =   77.46
               vgg16  min =  414.89  max =  423.00  avg =  418.75
          vgg16_int8  min =  245.58  max =  246.37  avg =  245.94
            resnet50  min =  185.53  max =  187.35  avg =  186.18
       resnet50_int8  min =  123.36  max =  124.75  avg =  124.17
      squeezenet_ssd  min =   85.87  max =   86.42  avg =   86.23
 squeezenet_ssd_int8  min =   64.90  max =   65.24  avg =   65.08
       mobilenet_ssd  min =   88.32  max =   90.02  avg =   89.10
  mobilenet_ssd_int8  min =   46.85  max =   47.18  avg =   46.98
      mobilenet_yolo  min =  192.33  max =  195.38  avg =  194.10
  mobilenetv2_yolov3  min =  127.33  max =  128.58  avg =  127.96
         yolov4-tiny  min =  150.44  max =  152.02  avg =  151.20
           nanodet_m  min =   54.22  max =   54.61  avg =   54.37

mobilenetv2_yolov3  min =  127.33  max =  128.58  avg =  127.96
    yolo-fastest-1.1  min =   28.13  max =   28.76  avg =   28.40
      yolo-fastestv2  min =   22.10  max =   22.26  avg =   22.19

vim3:/data/local/tmp # ./benchncnn 4 1 2 -1 1
          squeezenet  min =   68.25  max =   68.85  avg =   68.67
     squeezenet_int8  min =   51.92  max =   52.08  avg =   52.01
           mobilenet  min =  112.69  max =  113.72  avg =  113.33
      mobilenet_int8  min =   66.43  max =   66.89  avg =   66.68
        mobilenet_v2  min =   81.36  max =   81.77  avg =   81.62
        mobilenet_v3  min =   62.33  max =   63.39  avg =   62.94
          shufflenet  min =   37.84  max =   38.03  avg =   37.93
       shufflenet_v2  min =   37.33  max =   38.08  avg =   37.68

mobilenet_v3  min =   62.33  max =   63.39  avg =   62.94
             mnasnet  min =   73.83  max =   74.32  avg =   74.03
     proxylessnasnet  min =   85.19  max =   86.43  avg =   85.84
     efficientnet_b0  min =  138.68  max =  139.69  avg =  139.19
   efficientnetv2_b0  min =  167.53  max =  167.99  avg =  167.75
        regnety_400m  min =   94.78  max =   95.81  avg =   95.21
           blazeface  min =   11.22  max =   11.43  avg =   11.28
           googlenet  min =  229.35  max =  230.91  avg =  229.89
      googlenet_int8  min =  173.04  max =  173.48  avg =  173.24
            resnet18  min =  191.54  max =  193.78  avg =  192.49
       resnet18_int8  min =  132.97  max =  133.51  avg =  133.25
             alexnet  min =  140.31  max =  141.95  avg =  141.18
               vgg16  min = 1093.71  max = 1100.95  avg = 1097.64

resnet18_int8  min =  132.97  max =  133.51  avg =  133.25
          vgg16_int8  min =  734.44  max =  736.16  avg =  735.05
            resnet50  min =  530.38  max =  533.93  avg =  531.87
       resnet50_int8  min =  332.88  max =  334.22  avg =  333.71
      squeezenet_ssd  min =  159.08  max =  160.98  avg =  160.16
 squeezenet_ssd_int8  min =  126.97  max =  127.96  avg =  127.43
       mobilenet_ssd  min =  238.92  max =  241.14  avg =  239.70
  mobilenet_ssd_int8  min =  135.57  max =  136.02  avg =  135.78
      mobilenet_yolo  min =  539.59  max =  543.88  avg =  541.90
  mobilenetv2_yolov3  min =  281.32  max =  285.05  avg =  283.24
         yolov4-tiny  min =  381.99  max =  384.93  avg =  383.53
           nanodet_m  min =   98.32  max =   98.85  avg =   98.60
    yolo-fastest-1.1  min =   44.59  max =   44.95  avg =   44.80

yolov4-tiny  min =  381.99  max =  384.93  avg =  383.53
      yolo-fastestv2  min =   36.88  max =   37.11  avg =   36.98
```

### Rockchip RK3588 (Cortex-A76 2.4GHz x 4 + Cortex-A55 1.8GHz x 4)

```
rk3588_s:/data/local/tmp # ./benchncnn 8 4 2 -1 1
          squeezenet  min =    7.57  max =    7.68  avg =    7.60
     squeezenet_int8  min =    8.43  max =    8.52  avg =    8.46
           mobilenet  min =   11.01  max =   11.08  avg =   11.05
      mobilenet_int8  min =    8.89  max =    8.96  avg =    8.91
        mobilenet_v2  min =    8.73  max =    8.78  avg =    8.76
        mobilenet_v3  min =    7.90  max =    7.95  avg =    7.92
          shufflenet  min =    7.95  max =    8.02  avg =    7.99
       shufflenet_v2  min =    6.09  max =    6.13  avg =    6.11

mobilenet_v3  min =    7.90  max =    7.95  avg =    7.92
             mnasnet  min =    8.30  max =    8.35  avg =    8.33
     proxylessnasnet  min =    9.67  max =    9.72  avg =    9.69
     efficientnet_b0  min =   17.51  max =   17.60  avg =   17.56
   efficientnetv2_b0  min =   28.10  max =   28.17  avg =   28.14
        regnety_400m  min =   16.33  max =   16.39  avg =   16.35
           blazeface  min =    2.81  max =    2.89  avg =    2.83
           googlenet  min =   33.33  max =   33.41  avg =   33.37
      googlenet_int8  min =   33.62  max =   33.87  avg =   33.77
            resnet18  min =   18.83  max =   18.90  avg =   18.86
       resnet18_int8  min =   33.92  max =   34.10  avg =   34.00
             alexnet  min =   29.07  max =   29.11  avg =   29.09
               vgg16  min =  106.86  max =  107.40  avg =  107.06

resnet18_int8  min =   33.92  max =   34.10  avg =   34.00
          vgg16_int8  min =  283.66  max =  284.16  avg =  283.94
            resnet50  min =   53.70  max =   54.21  avg =   53.83
       resnet50_int8  min =   66.11  max =   66.24  avg =   66.15
      squeezenet_ssd  min =   34.88  max =   35.04  avg =   34.99
 squeezenet_ssd_int8  min =   43.25  max =   43.62  avg =   43.37
       mobilenet_ssd  min =   31.32  max =   31.42  avg =   31.37
  mobilenet_ssd_int8  min =   26.11  max =   26.18  avg =   26.13
      mobilenet_yolo  min =   58.89  max =   59.02  avg =   58.95
  mobilenetv2_yolov3  min =   37.53  max =   37.64  avg =   37.58
         yolov4-tiny  min =   52.95  max =   53.31  avg =   53.03
           nanodet_m  min =   16.06  max =   16.14  avg =   16.10
    yolo-fastest-1.1  min =    8.42  max =    8.47  avg =    8.45

yolov4-tiny  min =   52.95  max =   53.31  avg =   53.03
      yolo-fastestv2  min =    7.81  max =    7.88  avg =    7.84

rk3588_s:/data/local/tmp # ./benchncnn 8 1 2 -1 1
          squeezenet  min =   25.04  max =   25.14  avg =   25.07
     squeezenet_int8  min =   26.29  max =   26.38  avg =   26.33
           mobilenet  min =   41.17  max =   41.23  avg =   41.19
      mobilenet_int8  min =   32.51  max =   32.57  avg =   32.54
        mobilenet_v2  min =   27.27  max =   27.31  avg =   27.29
        mobilenet_v3  min =   22.49  max =   22.54  avg =   22.51
          shufflenet  min =   18.15  max =   18.22  avg =   18.18
       shufflenet_v2  min =   15.82  max =   15.86  avg =   15.85
             mnasnet  min =   26.45  max =   26.50  avg =   26.47

shufflenet  min =   18.15  max =   18.22  avg =   18.18
     proxylessnasnet  min =   31.60  max =   31.66  avg =   31.62
     efficientnet_b0  min =   55.53  max =   55.68  avg =   55.62
   efficientnetv2_b0  min =   96.84  max =   96.92  avg =   96.89
        regnety_400m  min =   33.66  max =   33.70  avg =   33.68
           blazeface  min =    8.80  max =    8.84  avg =    8.83
           googlenet  min =  116.89  max =  117.06  avg =  116.97
      googlenet_int8  min =  107.92  max =  108.03  avg =  107.98
            resnet18  min =   60.97  max =   61.18  avg =   61.05
       resnet18_int8  min =  118.95  max =  119.04  avg =  119.00
             alexnet  min =   93.49  max =   93.59  avg =   93.55
               vgg16  min =  333.81  max =  334.52  avg =  334.07
          vgg16_int8  min =  947.19  max =  947.55  avg =  947.35

alexnet  min =   93.49  max =   93.59  avg =   93.55
            resnet50  min =  186.95  max =  187.42  avg =  187.15
       resnet50_int8  min =  225.72  max =  225.86  avg =  225.75
      squeezenet_ssd  min =   93.29  max =   93.66  avg =   93.47
 squeezenet_ssd_int8  min =  120.22  max =  120.95  avg =  120.49
       mobilenet_ssd  min =  105.84  max =  105.90  avg =  105.87
  mobilenet_ssd_int8  min =   85.95  max =   86.04  avg =   86.01
      mobilenet_yolo  min =  194.22  max =  194.64  avg =  194.41
  mobilenetv2_yolov3  min =  103.63  max =  103.72  avg =  103.69
         yolov4-tiny  min =  136.59  max =  137.14  avg =  136.91
           nanodet_m  min =   41.40  max =   41.49  avg =   41.43
    yolo-fastest-1.1  min =   18.73  max =   18.80  avg =   18.77
      yolo-fastestv2  min =   18.25  max =   18.31  avg =   18.28

nanodet_m  min =   41.40  max =   41.49  avg =   41.43

rk3588_s:/data/local/tmp # ./benchncnn 8 4 1 -1 1
powersave = 1
          squeezenet  min =   25.54  max =   25.99  avg =   25.71
     squeezenet_int8  min =   30.88  max =   31.16  avg =   31.01
           mobilenet  min =   36.24  max =   62.95  avg =   39.89
      mobilenet_int8  min =   31.90  max =   32.37  avg =   32.06
        mobilenet_v2  min =   27.49  max =   27.82  avg =   27.64
        mobilenet_v3  min =   26.30  max =   26.69  avg =   26.45
          shufflenet  min =   25.49  max =   25.72  avg =   25.60
       shufflenet_v2  min =   21.59  max =   22.67  avg =   21.78
             mnasnet  min =   27.92  max =   28.10  avg =   28.00
     proxylessnasnet  min =   34.18  max =   34.42  avg =   34.28

shufflenet_v2  min =   21.59  max =   22.67  avg =   21.78
     efficientnet_b0  min =   57.37  max =   57.60  avg =   57.45
   efficientnetv2_b0  min =   83.50  max =   84.03  avg =   83.66
        regnety_400m  min =   50.83  max =   51.27  avg =   50.98
           blazeface  min =   14.07  max =   14.29  avg =   14.17
           googlenet  min =  100.60  max =  101.00  avg =  100.87
      googlenet_int8  min =  106.58  max =  107.14  avg =  106.71
            resnet18  min =   58.60  max =   59.62  avg =   59.00
       resnet18_int8  min =   84.90  max =   85.15  avg =   84.99
             alexnet  min =   86.06  max =   86.58  avg =   86.22
               vgg16  min =  308.42  max =  309.18  avg =  308.81
          vgg16_int8  min =  543.61  max =  545.09  avg =  544.40
            resnet50  min =  163.45  max =  164.44  avg =  163.92

vgg16  min =  308.42  max =  309.18  avg =  308.81
       resnet50_int8  min =  179.51  max =  180.16  avg =  179.83
      squeezenet_ssd  min =   96.32  max =   97.24  avg =   96.71
 squeezenet_ssd_int8  min =  116.48  max =  117.65  avg =  116.85
       mobilenet_ssd  min =   92.12  max =   93.09  avg =   92.55
  mobilenet_ssd_int8  min =   81.78  max =   82.42  avg =   81.95
      mobilenet_yolo  min =  174.95  max =  175.40  avg =  175.15
  mobilenetv2_yolov3  min =  110.63  max =  111.05  avg =  110.81
         yolov4-tiny  min =  163.37  max =  164.24  avg =  163.63
           nanodet_m  min =   52.96  max =   53.59  avg =   53.12
    yolo-fastest-1.1  min =   28.98  max =   29.33  avg =   29.20
      yolo-fastestv2  min =   23.52  max =   24.16  avg =   23.76

rk3588_s:/data/local/tmp # ./benchncnn 8 1 1 -1 1

yolo-fastest-1.1  min =   28.98  max =   29.33  avg =   29.20

          squeezenet  min =   83.46  max =   83.63  avg =   83.53
     squeezenet_int8  min =  101.39  max =  102.29  avg =  101.77
           mobilenet  min =  131.78  max =  132.25  avg =  131.87
      mobilenet_int8  min =  111.66  max =  112.60  avg =  111.94
        mobilenet_v2  min =   92.92  max =  227.19  avg =  132.44
        mobilenet_v3  min =   78.38  max =   78.64  avg =   78.49
          shufflenet  min =   62.98  max =   63.17  avg =   63.09
       shufflenet_v2  min =   56.85  max =   57.23  avg =   57.00
             mnasnet  min =   87.53  max =   87.71  avg =   87.60
     proxylessnasnet  min =  113.25  max =  114.10  avg =  113.58
     efficientnet_b0  min =  180.95  max =  181.16  avg =  181.07

mnasnet  min =   87.53  max =   87.71  avg =   87.60
   efficientnetv2_b0  min =  285.34  max =  285.62  avg =  285.51
        regnety_400m  min =  109.24  max =  109.36  avg =  109.31
           blazeface  min =   41.12  max =   41.53  avg =   41.23
           googlenet  min =  358.94  max =  359.55  avg =  359.24
      googlenet_int8  min =  371.32  max =  371.84  avg =  371.51
            resnet18  min =  209.97  max =  210.42  avg =  210.22
       resnet18_int8  min =  302.93  max =  303.51  avg =  303.26
             alexnet  min =  318.95  max =  321.70  avg =  319.40
               vgg16  min = 1126.11  max = 1127.83  avg = 1126.98
          vgg16_int8  min = 2026.90  max = 2034.04  avg = 2029.35
            resnet50  min =  602.90  max =  603.70  avg =  603.30
       resnet50_int8  min =  647.33  max =  649.41  avg =  648.65

vgg16_int8  min = 2026.90  max = 2034.04  avg = 2029.35
      squeezenet_ssd  min =  280.60  max =  281.50  avg =  281.02
 squeezenet_ssd_int8  min =  359.41  max =  362.07  avg =  360.66
       mobilenet_ssd  min =  319.11  max =  319.29  avg =  319.21
  mobilenet_ssd_int8  min =  272.16  max =  273.36  avg =  272.83
      mobilenet_yolo  min =  607.07  max =  607.38  avg =  607.21
  mobilenetv2_yolov3  min =  326.66  max =  326.95  avg =  326.80
         yolov4-tiny  min =  449.56  max =  450.45  avg =  450.04
           nanodet_m  min =  142.09  max =  142.54  avg =  142.32
    yolo-fastest-1.1  min =   63.74  max =   63.80  avg =   63.78
      yolo-fastestv2  min =   57.56  max =   58.17  avg =   57.97

rk3588_s:/data/local/tmp # ./benchncnn 8 1 2 0 0
[0 Mali-G610]  queueC=0[2]  queueG=0[2]  queueT=0[2]

yolo-fastestv2  min =   57.56  max =   58.17  avg =   57.97

[0 Mali-G610]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=0
[0 Mali-G610]  fp16-p/s/a=1/1/1  int8-p/s/a=1/1/1
[0 Mali-G610]  subgroup=16  basic=1  vote=1  ballot=1  shuffle=1
          squeezenet  min =    7.09  max =    7.20  avg =    7.13
           mobilenet  min =    9.16  max =    9.32  avg =    9.22
        mobilenet_v2  min =   10.18  max =   10.32  avg =   10.25
        mobilenet_v3  min =    8.01  max =    8.09  avg =    8.04
          shufflenet  min =    5.88  max =    5.93  avg =    5.89
       shufflenet_v2  min =    6.30  max =    6.33  avg =    6.31
             mnasnet  min =    7.91  max =    8.00  avg =    7.95
     proxylessnasnet  min =   11.20  max =   11.42  avg =   11.30

shufflenet_v2  min =    6.30  max =    6.33  avg =    6.31
        regnety_400m  min =   11.65  max =   11.84  avg =   11.74
           blazeface  min =    2.50  max =    2.59  avg =    2.53
           googlenet  min =   17.69  max =   17.78  avg =   17.74
            resnet18  min =   16.04  max =   16.39  avg =   16.25
             alexnet  min =   15.47  max =   15.66  avg =   15.56
               vgg16  min =   64.74  max =   65.42  avg =   65.04
            resnet50  min =   37.83  max =   38.31  avg =   38.12
      squeezenet_ssd  min =   23.14  max =   23.44  avg =   23.26
       mobilenet_ssd  min =   22.48  max =   23.01  avg =   22.74
      mobilenet_yolo  min =   40.08  max =   40.72  avg =   40.32
  mobilenetv2_yolov3  min =   31.88  max =   32.57  avg =   32.12
         yolov4-tiny  min =   49.64  max =   50.73  avg =   50.13

mobilenet_yolo  min =   40.08  max =   40.72  avg =   40.32
           nanodet_m  min =   10.60  max =   10.70  avg =   10.64
    yolo-fastest-1.1  min =    7.63  max =    7.66  avg =    7.64
      yolo-fastestv2  min =    6.99  max =    7.02  avg =    7.00
```

### Station-M3/ROC-RK3588S-PC, Rockchip RK3588S (Quad Core A76 2.4GHz + Quad Core A55 1.8GHz + Mali-G610) StationOS (Android)

```
roc_rk3588s_pc:/data/local/tmp # ./benchncnn 10 1 0 0 0
./benchncnn 10 1 0 0 0
[0 Mali-G610]  subgroup=16  basic/vote/ballot/shuffle=1/1/1/1
[0 Mali-G610]  fp16-matrix-16_8_8/16_8_16/16_16_16=0/0/0

          squeezenet  min =    7.83  max =   14.17  avg =    9.76
     squeezenet_int8  min =   13.41  max =   13.52  avg =   13.45
           mobilenet  min =    8.73  max =    9.68  avg =    9.07
      mobilenet_int8  min =   17.70  max =   17.89  avg =   17.80
        mobilenet_v2  min =   10.73  max =   21.20  avg =   18.93
        mobilenet_v3  min =    9.00  max =   13.36  avg =   10.64
          shufflenet  min =    7.79  max =    7.93  avg =    7.85
       shufflenet_v2  min =    8.01  max =    8.06  avg =    8.03
             mnasnet  min =    7.43  max =    8.71  avg =    8.28
     proxylessnasnet  min =   10.56  max =   12.07  avg =   11.70
     efficientnet_b0  min =    2.15  max =    2.19  avg =    2.17
   efficientnetv2_b0  min =    0.56  max =    0.62  avg =    0.57

proxylessnasnet  min =   10.56  max =   12.07  avg =   11.70
        regnety_400m  min =    1.65  max =    1.69  avg =    1.67
           blazeface  min =    0.76  max =    0.79  avg =    0.78
           googlenet  min =    1.53  max =    1.60  avg =    1.56
      googlenet_int8  min =   60.85  max =   61.01  avg =   60.93
            resnet18  min =    0.63  max =    0.82  avg =    0.65
       resnet18_int8  min =   64.60  max =   65.13  avg =   64.78
             alexnet  min =    0.35  max =    0.40  avg =    0.37
               vgg16  min =    0.54  max =    0.60  avg =    0.56
          vgg16_int8  min =  445.21  max =  562.09  avg =  537.10
            resnet50  min =    0.95  max =    0.97  avg =    0.96
       resnet50_int8  min =  113.02  max =  113.38  avg =  113.17
      squeezenet_ssd  min =    1.94  max =    2.00  avg =    1.96

resnet50  min =    0.95  max =    0.97  avg =    0.96
 squeezenet_ssd_int8  min =   52.09  max =   56.93  avg =   56.35
       mobilenet_ssd  min =    1.19  max =    1.26  avg =    1.21
  mobilenet_ssd_int8  min =   44.33  max =   44.87  avg =   44.66
      mobilenet_yolo  min =    1.05  max =    1.24  avg =    1.13
  mobilenetv2_yolov3  min =    1.18  max =    1.25  avg =    1.21
         yolov4-tiny  min =    0.78  max =    0.80  avg =    0.78
           nanodet_m  min =    3.43  max =    3.80  avg =    3.57
    yolo-fastest-1.1  min =    1.43  max =    1.50  avg =    1.47
      yolo-fastestv2  min =    2.03  max =    2.10  avg =    2.05
  vision_transformer  min =    0.32  max =    0.36  avg =    0.35
          FastestDet  min =    1.90  max =    1.95  avg =    1.93

roc_rk3588s_pc:/data/local/tmp # ./benchncnn 10 1 0 -1 0

vision_transformer  min =    0.32  max =    0.36  avg =    0.35

./benchncnn 10 1 0 -1 0
          squeezenet  min =   13.36  max =   13.50  avg =   13.40
     squeezenet_int8  min =   16.22  max =   16.34  avg =   16.30
           mobilenet  min =   22.41  max =   22.49  avg =   22.44
      mobilenet_int8  min =   17.76  max =   17.94  avg =   17.84
        mobilenet_v2  min =   17.60  max =   17.80  avg =   17.70
        mobilenet_v3  min =   13.55  max =   13.70  avg =   13.61
          shufflenet  min =    7.91  max =    7.95  avg =    7.93
       shufflenet_v2  min =    8.36  max =    8.40  avg =    8.38
             mnasnet  min =   14.50  max =   14.60  avg =   14.56
     proxylessnasnet  min =   16.99  max =   17.12  avg =   17.06

shufflenet_v2  min =    8.36  max =    8.40  avg =    8.38
     efficientnet_b0  min =   26.55  max =   26.78  avg =   26.62
   efficientnetv2_b0  min =   46.96  max =   47.44  avg =   47.30
        regnety_400m  min =   18.53  max =   18.63  avg =   18.58
           blazeface  min =    2.98  max =    3.02  avg =    3.00
           googlenet  min =   62.69  max =   63.14  avg =   62.90
      googlenet_int8  min =   60.86  max =   61.54  avg =   61.05
            resnet18  min =   30.34  max =   31.39  avg =   31.22
       resnet18_int8  min =   57.42  max =   57.67  avg =   57.56
             alexnet  min =   40.81  max =   40.87  avg =   40.84
               vgg16  min =  192.71  max =  195.20  avg =  194.26
          vgg16_int8  min =  450.95  max =  534.38  avg =  482.27
            resnet50  min =  105.11  max =  105.64  avg =  105.30

vgg16  min =  192.71  max =  195.20  avg =  194.26
       resnet50_int8  min =  105.94  max =  132.01  avg =  116.48
      squeezenet_ssd  min =   51.36  max =   51.59  avg =   51.51
 squeezenet_ssd_int8  min =   69.01  max =   69.83  avg =   69.37
       mobilenet_ssd  min =   53.19  max =   55.24  avg =   53.50
  mobilenet_ssd_int8  min =   44.49  max =   44.98  avg =   44.74
      mobilenet_yolo  min =  112.65  max =  113.28  avg =  112.94
  mobilenetv2_yolov3  min =   63.38  max =   63.83  avg =   63.55
         yolov4-tiny  min =   77.57  max =   78.20  avg =   77.90
           nanodet_m  min =   25.21  max =   25.81  avg =   25.58
    yolo-fastest-1.1  min =    8.76  max =    8.84  avg =    8.80
      yolo-fastestv2  min =    8.46  max =    8.53  avg =    8.50
  vision_transformer  min = 1499.53  max = 1501.32  avg = 1500.50

yolo-fastest-1.1  min =    8.76  max =    8.84  avg =    8.80
          FastestDet  min =    7.04  max =    7.08  avg =    7.06
```

### Station P2, Rockchip RK3568 (Cortex-A55 2.0GHz x 4)

```
./benchncnn 4 4 0 -1 1
          squeezenet  min =   26.02  max =   27.15  avg =   26.74
     squeezenet_int8  min =   44.69  max =   45.70  avg =   45.24
           mobilenet  min =   32.63  max =   33.49  avg =   33.10
      mobilenet_int8  min =   44.23  max =   45.86  avg =   44.99
        mobilenet_v2  min =   31.59  max =   32.02  avg =   31.86
        mobilenet_v3  min =   25.71  max =   26.44  avg =   26.10
          shufflenet  min =   22.12  max =   23.17  avg =   22.52
       shufflenet_v2  min =   17.84  max =   18.21  avg =   17.96

mobilenet_v3  min =   25.71  max =   26.44  avg =   26.10
             mnasnet  min =   28.26  max =   28.70  avg =   28.45
     proxylessnasnet  min =   31.96  max =   32.25  avg =   32.13
     efficientnet_b0  min =   53.17  max =   54.48  avg =   53.60
   efficientnetv2_b0  min =   70.08  max =   70.69  avg =   70.30
        regnety_400m  min =   40.80  max =   41.79  avg =   41.10
           blazeface  min =   10.79  max =   11.57  avg =   11.11
           googlenet  min =   83.66  max =   92.22  avg =   86.23
      googlenet_int8  min =  116.44  max =  118.34  avg =  117.08
            resnet18  min =   61.38  max =   62.52  avg =   61.94
       resnet18_int8  min =   95.58  max =   96.93  avg =   96.28
             alexnet  min =   69.90  max =   70.59  avg =   70.19
               vgg16  min =  334.24  max =  343.89  avg =  337.24

resnet18_int8  min =   95.58  max =   96.93  avg =   96.28
          vgg16_int8  min =  464.88  max =  474.71  avg =  468.29
            resnet50  min =  141.65  max =  146.23  avg =  143.78
       resnet50_int8  min =  230.36  max =  254.75  avg =  241.24
      squeezenet_ssd  min =   98.38  max =  104.60  avg =  100.50
 squeezenet_ssd_int8  min =  134.73  max =  137.88  avg =  136.12
       mobilenet_ssd  min =   77.48  max =   79.92  avg =   78.64
  mobilenet_ssd_int8  min =  101.44  max =  102.61  avg =  102.06
      mobilenet_yolo  min =  149.12  max =  150.14  avg =  149.76
  mobilenetv2_yolov3  min =  103.71  max =  107.81  avg =  105.69
         yolov4-tiny  min =  145.75  max =  149.35  avg =  147.09
           nanodet_m  min =   52.91  max =   54.06  avg =   53.53

./benchncnn 4 2 0 -1 1

yolov4-tiny  min =  145.75  max =  149.35  avg =  147.09

          squeezenet  min =   33.78  max =   34.38  avg =   34.16
     squeezenet_int8  min =   61.66  max =   62.11  avg =   61.85
           mobilenet  min =   46.53  max =   46.74  avg =   46.62
      mobilenet_int8  min =   71.06  max =   71.76  avg =   71.38
        mobilenet_v2  min =   39.05  max =   39.38  avg =   39.19
        mobilenet_v3  min =   32.20  max =   32.47  avg =   32.29
          shufflenet  min =   27.13  max =   27.40  avg =   27.27
       shufflenet_v2  min =   23.38  max =   23.92  avg =   23.62
             mnasnet  min =   35.51  max =   35.73  avg =   35.62
     proxylessnasnet  min =   42.98  max =   43.16  avg =   43.06
     efficientnet_b0  min =   75.34  max =   75.79  avg =   75.61

mnasnet  min =   35.51  max =   35.73  avg =   35.62
   efficientnetv2_b0  min =  107.34  max =  107.83  avg =  107.60
        regnety_400m  min =   47.91  max =   48.20  avg =   48.02
           blazeface  min =   16.38  max =   16.63  avg =   16.49
           googlenet  min =  124.27  max =  125.24  avg =  124.65
      googlenet_int8  min =  177.78  max =  178.39  avg =  178.06
            resnet18  min =   82.02  max =   82.70  avg =   82.38
       resnet18_int8  min =  148.06  max =  149.03  avg =  148.39
             alexnet  min =  105.20  max =  105.91  avg =  105.54
               vgg16  min =  459.65  max =  464.94  avg =  462.02
          vgg16_int8  min =  737.54  max =  750.64  avg =  742.90
            resnet50  min =  204.44  max =  205.20  avg =  204.84
       resnet50_int8  min =  364.47  max =  366.04  avg =  365.53

vgg16_int8  min =  737.54  max =  750.64  avg =  742.90
      squeezenet_ssd  min =  124.42  max =  128.01  avg =  125.80
 squeezenet_ssd_int8  min =  179.29  max =  183.83  avg =  181.43
       mobilenet_ssd  min =  113.85  max =  115.50  avg =  114.41
  mobilenet_ssd_int8  min =  161.35  max =  162.38  avg =  161.71
      mobilenet_yolo  min =  214.95  max =  216.62  avg =  215.72
  mobilenetv2_yolov3  min =  134.23  max =  136.26  avg =  135.07
         yolov4-tiny  min =  194.72  max =  195.49  avg =  195.18
           nanodet_m  min =   67.67  max =   68.09  avg =   67.90

./benchncnn 4 1 0 -1 1
          squeezenet  min =   54.31  max =   55.65  avg =   55.00
     squeezenet_int8  min =  103.96  max =  106.28  avg =  104.92

           mobilenet  min =   79.02  max =   79.46  avg =   79.25
      mobilenet_int8  min =  130.06  max =  130.61  avg =  130.36
        mobilenet_v2  min =   60.15  max =   60.66  avg =   60.31
        mobilenet_v3  min =   49.40  max =   49.57  avg =   49.49
          shufflenet  min =   39.39  max =   39.78  avg =   39.60
       shufflenet_v2  min =   35.48  max =   35.70  avg =   35.62
             mnasnet  min =   55.38  max =   56.10  avg =   55.71
     proxylessnasnet  min =   70.29  max =   70.48  avg =   70.35
     efficientnet_b0  min =  128.56  max =  129.96  avg =  129.26
   efficientnetv2_b0  min =  181.00  max =  181.56  avg =  181.24
        regnety_400m  min =   67.15  max =   69.62  avg =   67.95
           blazeface  min =   26.07  max =   26.58  avg =   26.33

efficientnetv2_b0  min =  181.00  max =  181.56  avg =  181.24
           googlenet  min =  219.19  max =  221.32  avg =  220.01
      googlenet_int8  min =  317.62  max =  319.40  avg =  318.37
            resnet18  min =  135.33  max =  136.94  avg =  135.88
       resnet18_int8  min =  264.69  max =  265.51  avg =  265.16
             alexnet  min =  190.54  max =  193.50  avg =  191.88
               vgg16  min =  790.99  max =  809.24  avg =  795.85
          vgg16_int8  min = 1354.48  max = 1358.89  avg = 1357.40
            resnet50  min =  358.08  max =  362.96  avg =  360.29
       resnet50_int8  min =  667.92  max =  670.40  avg =  668.78
      squeezenet_ssd  min =  193.15  max =  194.02  avg =  193.49
 squeezenet_ssd_int8  min =  291.42  max =  294.70  avg =  293.16
       mobilenet_ssd  min =  189.54  max =  190.28  avg =  189.97

squeezenet_ssd  min =  193.15  max =  194.02  avg =  193.49
  mobilenet_ssd_int8  min =  289.94  max =  290.40  avg =  290.28
      mobilenet_yolo  min =  370.37  max =  384.69  avg =  375.11
  mobilenetv2_yolov3  min =  210.93  max =  211.70  avg =  211.40
         yolov4-tiny  min =  309.11  max =  310.74  avg =  309.89
           nanodet_m  min =  100.42  max =  112.25  avg =  103.66
```

### Rock3A, Rockchip RK3568 (Cortex-A55 2.0GHz x 4) ubuntu 20.04

```
rock@rock3a:~/ncnn/build/benchmark$ ./benchncnn 8 4 0 -1 1
          squeezenet  min =   29.52  max =   30.30  avg =   29.76
     squeezenet_int8  min =   35.40  max =   36.19  avg =   35.88
           mobilenet  min =   34.47  max =   35.44  avg =   34.84

squeezenet  min =   29.52  max =   30.30  avg =   29.76
      mobilenet_int8  min =   34.19  max =   34.53  avg =   34.40
        mobilenet_v2  min =   35.75  max =   36.09  avg =   35.88
        mobilenet_v3  min =   28.12  max =   28.82  avg =   28.49
          shufflenet  min =   23.62  max =   24.08  avg =   23.84
       shufflenet_v2  min =   19.37  max =   19.64  avg =   19.52
             mnasnet  min =   30.84  max =   31.45  avg =   31.02
     proxylessnasnet  min =   35.73  max =   36.07  avg =   35.90
     efficientnet_b0  min =   48.16  max =   49.29  avg =   48.64
   efficientnetv2_b0  min =   66.62  max =   67.11  avg =   66.85
        regnety_400m  min =   41.11  max =   41.64  avg =   41.34
           blazeface  min =   12.38  max =   12.64  avg =   12.56
           googlenet  min =   86.73  max =   87.79  avg =   87.11

regnety_400m  min =   41.11  max =   41.64  avg =   41.34
      googlenet_int8  min =  101.42  max =  103.87  avg =  102.55
            resnet18  min =   64.85  max =   65.84  avg =   65.23
       resnet18_int8  min =   93.55  max =   94.54  avg =   94.03
             alexnet  min =   70.89  max =   73.58  avg =   71.57
               vgg16  min =  356.13  max =  358.52  avg =  357.15
          vgg16_int8  min =  521.92  max =  524.13  avg =  523.11
            resnet50  min =  147.65  max =  150.33  avg =  148.52
       resnet50_int8  min =  191.94  max =  192.73  avg =  192.30
      squeezenet_ssd  min =  104.32  max =  105.75  avg =  105.00
 squeezenet_ssd_int8  min =  125.97  max =  127.53  avg =  126.70
       mobilenet_ssd  min =   82.29  max =   82.65  avg =   82.47
  mobilenet_ssd_int8  min =   79.26  max =   80.93  avg =   79.72

squeezenet_ssd_int8  min =  125.97  max =  127.53  avg =  126.70
      mobilenet_yolo  min =  165.51  max =  165.86  avg =  165.72
  mobilenetv2_yolov3  min =  116.11  max =  116.83  avg =  116.43
         yolov4-tiny  min =  152.09  max =  153.39  avg =  152.60
           nanodet_m  min =   53.63  max =   54.14  avg =   53.92

rock@rock3a:~/ncnn/build/benchmark$ ./benchncnn 4 1 0 -1 1
          squeezenet  min =   62.47  max =   63.04  avg =   62.84
     squeezenet_int8  min =   67.23  max =   68.48  avg =   67.93
           mobilenet  min =   85.27  max =   85.69  avg =   85.49
      mobilenet_int8  min =   75.00  max =   75.48  avg =   75.26
        mobilenet_v2  min =   68.41  max =   69.09  avg =   68.76
        mobilenet_v3  min =   54.19  max =   54.52  avg =   54.34

mobilenet_int8  min =   75.00  max =   75.48  avg =   75.26
          shufflenet  min =   45.90  max =   46.30  avg =   46.09
       shufflenet_v2  min =   39.64  max =   40.07  avg =   39.91
             mnasnet  min =   62.16  max =   62.41  avg =   62.30
     proxylessnasnet  min =   80.79  max =   81.41  avg =   81.12
     efficientnet_b0  min =  113.47  max =  113.68  avg =  113.57
   efficientnetv2_b0  min =  167.30  max =  167.58  avg =  167.44
        regnety_400m  min =   72.12  max =   72.24  avg =   72.17
           blazeface  min =   31.89  max =   32.04  avg =   31.95
           googlenet  min =  224.27  max =  224.86  avg =  224.55
      googlenet_int8  min =  240.02  max =  240.93  avg =  240.45
            resnet18  min =  150.25  max =  150.69  avg =  150.47
       resnet18_int8  min =  226.70  max =  228.19  avg =  227.56

googlenet_int8  min =  240.02  max =  240.93  avg =  240.45
             alexnet  min =  197.44  max =  199.16  avg =  198.17
               vgg16  min =  859.80  max =  860.79  avg =  860.35
          vgg16_int8  min = 1409.66  max = 1411.92  avg = 1411.07
            resnet50  min =  381.04  max =  382.73  avg =  381.86
       resnet50_int8  min =  441.78  max =  445.00  avg =  443.29
      squeezenet_ssd  min =  208.14  max =  208.67  avg =  208.41
 squeezenet_ssd_int8  min =  248.82  max =  250.80  avg =  249.89
       mobilenet_ssd  min =  200.95  max =  201.21  avg =  201.06
  mobilenet_ssd_int8  min =  173.81  max =  174.54  avg =  174.28
      mobilenet_yolo  min =  394.65  max =  395.00  avg =  394.78
  mobilenetv2_yolov3  min =  231.80  max =  232.27  avg =  232.08
         yolov4-tiny  min =  321.31  max =  322.43  avg =  321.79

mobilenet_yolo  min =  394.65  max =  395.00  avg =  394.78
           nanodet_m  min =  103.81  max =  104.61  avg =  104.25
```

### Station-M2/ROC-RK3566-PC, Rockchip RK3566 (Cortex-A55 1.8GHz x 4 + Mali-G52) StationOS (Android)

```
rk3566_roc_pc:/data/local/tmp # ./benchncnn 10 1 0 0 0
[0 Mali-G52]  queueC=0[2]  queueG=0[2]  queueT=0[2]
[0 Mali-G52]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=1
[0 Mali-G52]  fp16-p/s/a=1/1/1  int8-p/s/a=1/1/1
[0 Mali-G52]  subgroup=8  basic/vote/ballot/shuffle=1/1/1/1
[0 Mali-G52]  fp16-matrix-16_8_8/16_8_16/16_16_16=0/0/0
          squeezenet  min =   43.67  max =   44.15  avg =   43.82
     squeezenet_int8  min =   62.72  max =   63.99  avg =   63.49

           mobilenet  min =   74.32  max =   74.82  avg =   74.58
      mobilenet_int8  min =   64.42  max =   65.43  avg =   64.89
        mobilenet_v2  min =   52.96  max =   53.23  avg =   53.09
        mobilenet_v3  min =   51.55  max =   53.12  avg =   51.96
          shufflenet  min =   40.73  max =   41.28  avg =   40.98
       shufflenet_v2  min =   41.56  max =   43.62  avg =   42.22
             mnasnet  min =   54.37  max =   54.63  avg =   54.52
     proxylessnasnet  min =   57.91  max =   59.38  avg =   58.36
     efficientnet_b0  min =   38.40  max =   40.29  avg =   39.06
   efficientnetv2_b0  min =   36.91  max =   38.45  avg =   37.72
        regnety_400m  min =   69.07  max =   69.98  avg =   69.40
           blazeface  min =   12.26  max =   13.08  avg =   12.57

efficientnetv2_b0  min =   36.91  max =   38.45  avg =   37.72
           googlenet  min =  147.08  max =  147.80  avg =  147.48
      googlenet_int8  min =  221.94  max =  225.99  avg =  223.12
            resnet18  min =  137.90  max =  138.50  avg =  138.19
       resnet18_int8  min =  187.84  max =  190.88  avg =  188.81
             alexnet  min =  167.56  max =  168.92  avg =  168.17
               vgg16  min =  713.42  max =  715.20  avg =  714.51
          vgg16_int8  min = 1279.97  max = 1302.95  avg = 1294.59
            resnet50  min =  369.74  max =  375.95  avg =  372.60
       resnet50_int8  min =  391.86  max =  397.49  avg =  395.17
      squeezenet_ssd  min =  155.18  max =  156.09  avg =  155.62
 squeezenet_ssd_int8  min =  218.83  max =  222.64  avg =  221.11
       mobilenet_ssd  min =  161.62  max =  163.22  avg =  162.27

squeezenet_ssd  min =  155.18  max =  156.09  avg =  155.62
  mobilenet_ssd_int8  min =  147.33  max =  149.16  avg =  148.23
      mobilenet_yolo  min =  344.09  max =  349.15  avg =  346.73
  mobilenetv2_yolov3  min =  168.72  max =  169.64  avg =  169.22
         yolov4-tiny  min =  239.44  max =  241.11  avg =  240.00
           nanodet_m  min =   88.06  max =   89.89  avg =   88.87
    yolo-fastest-1.1  min =   36.05  max =   37.86  avg =   36.47
      yolo-fastestv2  min =   34.80  max =   36.58  avg =   35.37
  vision_transformer  min =  356.42  max =  359.37  avg =  358.03
          FastestDet  min =   38.03  max =   38.52  avg =   38.24

rk3566_roc_pc:/data/local/tmp # ./benchncnn 10 1 0 -1 0

          squeezenet  min =   47.01  max =   48.12  avg =   47.62
     squeezenet_int8  min =   63.30  max =   64.10  avg =   63.74
           mobilenet  min =   70.24  max =   71.52  avg =   70.63
      mobilenet_int8  min =   63.90  max =   65.25  avg =   64.41
        mobilenet_v2  min =   55.75  max =   56.26  avg =   56.02
        mobilenet_v3  min =   45.56  max =   46.47  avg =   46.17
          shufflenet  min =   34.16  max =   35.16  avg =   34.64
       shufflenet_v2  min =   32.58  max =   33.86  avg =   33.25
             mnasnet  min =   52.43  max =   53.15  avg =   52.80
     proxylessnasnet  min =   65.55  max =   67.04  avg =   66.36
     efficientnet_b0  min =   82.52  max =   82.97  avg =   82.64
   efficientnetv2_b0  min =  148.90  max =  150.47  avg =  149.64

proxylessnasnet  min =   65.55  max =   67.04  avg =   66.36
        regnety_400m  min =   63.33  max =   64.29  avg =   63.70
           blazeface  min =   11.55  max =   12.35  avg =   11.77
           googlenet  min =  205.85  max =  208.74  avg =  207.17
      googlenet_int8  min =  222.72  max =  225.84  avg =  223.98
            resnet18  min =  134.19  max =  136.81  avg =  135.39
       resnet18_int8  min =  187.26  max =  189.45  avg =  188.36
             alexnet  min =  143.01  max =  144.97  avg =  143.42
               vgg16  min =  829.44  max =  839.46  avg =  835.37
          vgg16_int8  min = 1299.25  max = 1306.89  avg = 1301.71
            resnet50  min =  326.54  max =  330.21  avg =  328.27
       resnet50_int8  min =  391.67  max =  395.59  avg =  393.27
      squeezenet_ssd  min =  166.12  max =  168.33  avg =  167.08

resnet50  min =  326.54  max =  330.21  avg =  328.27
 squeezenet_ssd_int8  min =  221.82  max =  223.85  avg =  222.69
       mobilenet_ssd  min =  163.17  max =  166.55  avg =  164.11
  mobilenet_ssd_int8  min =  146.16  max =  148.20  avg =  147.41
      mobilenet_yolo  min =  335.15  max =  338.32  avg =  336.66
  mobilenetv2_yolov3  min =  193.18  max =  195.51  avg =  194.33
         yolov4-tiny  min =  288.82  max =  292.16  avg =  290.36
           nanodet_m  min =   98.31  max =  100.30  avg =   99.20
    yolo-fastest-1.1  min =   37.73  max =   38.97  avg =   38.40
      yolo-fastestv2  min =   36.21  max =   37.90  avg =   37.13
  vision_transformer  min = 7385.59  max = 7410.59  avg = 7402.20
          FastestDet  min =   34.55  max =   35.42  avg =   35.06
```

### Rockchip RK3399 (Cortex-A72 1.8GHz x 2 + Cortex-A53 1.5GHz x 4)
```

FastestDet  min =   34.55  max =   35.42  avg =   35.06
```

```
nanopc-t4:/data/local/tmp # ./benchncnn 8 2 2 -1 1
          squeezenet  min =   43.73  max =   44.30  avg =   43.97
     squeezenet_int8  min =   37.92  max =   38.39  avg =   38.09
           mobilenet  min =   64.28  max =   66.66  avg =   65.14
      mobilenet_int8  min =   43.17  max =   43.73  avg =   43.38
        mobilenet_v2  min =   51.30  max =   52.18  avg =   51.75
        mobilenet_v3  min =   41.51  max =   43.25  avg =   42.10
          shufflenet  min =   27.43  max =   28.27  avg =   27.75
       shufflenet_v2  min =   24.96  max =   25.79  avg =   25.55
             mnasnet  min =   45.44  max =   46.95  avg =   46.16
     proxylessnasnet  min =   51.98  max =   53.52  avg =   52.48
     efficientnet_b0  min =   83.79  max =   84.68  avg =   84.27

mnasnet  min =   45.44  max =   46.95  avg =   46.16
   efficientnetv2_b0  min =   97.89  max =   99.27  avg =   98.55
        regnety_400m  min =   65.15  max =   65.89  avg =   65.41
           blazeface  min =    8.74  max =    8.89  avg =    8.80
           googlenet  min =  131.46  max =  140.16  avg =  133.24
      googlenet_int8  min =  115.72  max =  118.34  avg =  116.60
            resnet18  min =  111.77  max =  113.18  avg =  112.37
       resnet18_int8  min =   84.27  max =   84.90  avg =   84.49
             alexnet  min =  105.74  max =  109.87  avg =  107.15
               vgg16  min =  619.88  max =  634.59  avg =  629.15
          vgg16_int8  min =  447.14  max =  451.09  avg =  448.53
            resnet50  min =  291.51  max =  296.55  avg =  293.08
       resnet50_int8  min =  224.09  max =  227.03  avg =  225.02

vgg16_int8  min =  447.14  max =  451.09  avg =  448.53
      squeezenet_ssd  min =  109.72  max =  112.09  avg =  110.78
 squeezenet_ssd_int8  min =   93.41  max =   94.83  avg =   93.97
       mobilenet_ssd  min =  131.30  max =  132.82  avg =  131.94
  mobilenet_ssd_int8  min =   87.52  max =   88.89  avg =   88.35
      mobilenet_yolo  min =  288.02  max =  289.84  avg =  288.61
  mobilenetv2_yolov3  min =  168.45  max =  170.94  avg =  169.79
         yolov4-tiny  min =  217.45  max =  226.39  avg =  219.76
           nanodet_m  min =   65.74  max =   66.84  avg =   66.49
    yolo-fastest-1.1  min =   32.91  max =   33.74  avg =   33.37
      yolo-fastestv2  min =   28.90  max =   37.31  avg =   30.27

nanopc-t4:/data/local/tmp # ./benchncnn 8 1 2 -1 1

yolo-fastestv2  min =   28.90  max =   37.31  avg =   30.27

          squeezenet  min =   71.35  max =   73.02  avg =   71.83
     squeezenet_int8  min =   60.39  max =   60.96  avg =   60.69
           mobilenet  min =  111.12  max =  113.02  avg =  111.99
      mobilenet_int8  min =   80.14  max =   81.59  avg =   81.00
        mobilenet_v2  min =   78.18  max =   80.89  avg =   79.18
        mobilenet_v3  min =   63.49  max =   64.26  avg =   63.90
          shufflenet  min =   38.90  max =   40.28  avg =   39.26
       shufflenet_v2  min =   37.72  max =   38.45  avg =   38.02
             mnasnet  min =   72.34  max =   73.59  avg =   72.87
     proxylessnasnet  min =   87.33  max =   89.70  avg =   88.45
     efficientnet_b0  min =  145.14  max =  146.77  avg =  145.93
   efficientnetv2_b0  min =  169.33  max =  171.16  avg =  170.16

proxylessnasnet  min =   87.33  max =   89.70  avg =   88.45
        regnety_400m  min =   99.08  max =   99.80  avg =   99.47
           blazeface  min =   12.28  max =   12.69  avg =   12.48
           googlenet  min =  228.18  max =  229.36  avg =  228.64
      googlenet_int8  min =  201.62  max =  203.71  avg =  202.25
            resnet18  min =  175.71  max =  180.53  avg =  176.85
       resnet18_int8  min =  151.42  max =  152.45  avg =  151.83
             alexnet  min =  160.81  max =  186.24  avg =  165.30
               vgg16  min = 1044.34  max = 1080.88  avg = 1062.34
          vgg16_int8  min =  844.53  max =  851.71  avg =  848.65
            resnet50  min =  503.25  max =  505.20  avg =  504.18
       resnet50_int8  min =  397.71  max =  400.19  avg =  398.63
      squeezenet_ssd  min =  162.98  max =  165.97  avg =  164.34

resnet50  min =  503.25  max =  505.20  avg =  504.18
 squeezenet_ssd_int8  min =  145.93  max =  148.59  avg =  146.94
       mobilenet_ssd  min =  226.54  max =  229.80  avg =  227.80
  mobilenet_ssd_int8  min =  159.97  max =  163.18  avg =  161.06
      mobilenet_yolo  min =  512.90  max =  517.47  avg =  515.06
  mobilenetv2_yolov3  min =  274.88  max =  280.24  avg =  276.36
         yolov4-tiny  min =  351.97  max =  358.70  avg =  355.60
           nanodet_m  min =   95.32  max =   97.83  avg =   96.28
    yolo-fastest-1.1  min =   43.47  max =   46.52  avg =   44.55
      yolo-fastestv2  min =   37.22  max =   37.63  avg =   37.45

nanopc-t4:/data/local/tmp # ./benchncnn 8 4 1 -1 1
          squeezenet  min =   48.11  max =   48.51  avg =   48.24

     squeezenet_int8  min =   43.19  max =   44.17  avg =   43.40
           mobilenet  min =   65.47  max =   66.40  avg =   65.68
      mobilenet_int8  min =   49.15  max =   51.65  avg =   49.76
        mobilenet_v2  min =   53.60  max =   54.19  avg =   53.87
        mobilenet_v3  min =   52.83  max =   92.92  avg =   66.25
          shufflenet  min =   35.71  max =   36.03  avg =   35.83
       shufflenet_v2  min =   31.88  max =   32.38  avg =   32.16
             mnasnet  min =   51.59  max =   54.01  avg =   52.30
     proxylessnasnet  min =   60.11  max =   60.40  avg =   60.24
     efficientnet_b0  min =   98.22  max =   99.40  avg =   98.56
   efficientnetv2_b0  min =  114.19  max =  123.90  avg =  115.89
        regnety_400m  min =   85.89  max =   86.20  avg =   86.03

efficientnet_b0  min =   98.22  max =   99.40  avg =   98.56
           blazeface  min =   11.23  max =   11.37  avg =   11.31
           googlenet  min =  142.25  max =  160.88  avg =  145.26
      googlenet_int8  min =  125.45  max =  128.50  avg =  125.96
            resnet18  min =  116.68  max =  118.26  avg =  117.00
       resnet18_int8  min =   88.43  max =   90.95  avg =   89.08
             alexnet  min =  150.91  max =  160.01  avg =  152.51
               vgg16  min =  674.91  max =  684.83  avg =  679.08
          vgg16_int8  min =  417.60  max =  422.52  avg =  419.60
            resnet50  min =  297.23  max =  299.37  avg =  298.03
       resnet50_int8  min =  243.99  max =  251.39  avg =  245.99
      squeezenet_ssd  min =  127.92  max =  128.53  avg =  128.17
 squeezenet_ssd_int8  min =  112.54  max =  114.63  avg =  113.19

resnet50_int8  min =  243.99  max =  251.39  avg =  245.99
       mobilenet_ssd  min =  136.43  max =  140.14  avg =  137.33
  mobilenet_ssd_int8  min =  102.14  max =  105.00  avg =  102.77
      mobilenet_yolo  min =  291.45  max =  294.04  avg =  292.63
  mobilenetv2_yolov3  min =  183.13  max =  187.00  avg =  184.05
         yolov4-tiny  min =  257.46  max =  268.76  avg =  260.49
           nanodet_m  min =   83.16  max =   91.03  avg =   84.77
    yolo-fastest-1.1  min =   43.53  max =   43.87  avg =   43.74
      yolo-fastestv2  min =   35.04  max =   35.54  avg =   35.17

nanopc-t4:/data/local/tmp # ./benchncnn 8 1 1 -1 1
          squeezenet  min =  129.63  max =  130.58  avg =  129.85
     squeezenet_int8  min =  124.10  max =  126.34  avg =  124.81

           mobilenet  min =  207.92  max =  208.72  avg =  208.41
      mobilenet_int8  min =  175.55  max =  176.11  avg =  175.84
        mobilenet_v2  min =  143.02  max =  143.56  avg =  143.25
        mobilenet_v3  min =  133.11  max =  134.05  avg =  133.33
          shufflenet  min =   77.97  max =   78.54  avg =   78.19
       shufflenet_v2  min =   75.59  max =   76.05  avg =   75.82
             mnasnet  min =  139.86  max =  141.77  avg =  140.19
     proxylessnasnet  min =  178.57  max =  179.57  avg =  179.03
     efficientnet_b0  min =  316.10  max =  317.82  avg =  316.86
   efficientnetv2_b0  min =  359.26  max =  362.03  avg =  360.31
        regnety_400m  min =  182.64  max =  183.03  avg =  182.82
           blazeface  min =   25.81  max =   26.53  avg =   26.20

efficientnetv2_b0  min =  359.26  max =  362.03  avg =  360.31
           googlenet  min =  448.45  max =  450.80  avg =  449.35
      googlenet_int8  min =  406.07  max =  410.65  avg =  408.04
            resnet18  min =  351.64  max =  362.12  avg =  354.19
       resnet18_int8  min =  298.10  max =  300.45  avg =  299.26
             alexnet  min =  586.92  max =  588.73  avg =  587.80
               vgg16  min = 2170.12  max = 2202.80  avg = 2183.32
          vgg16_int8  min = 1533.65  max = 1542.01  avg = 1537.33
            resnet50  min =  975.40  max =  977.79  avg =  976.61
       resnet50_int8  min =  851.59  max =  855.22  avg =  853.75
      squeezenet_ssd  min =  306.35  max =  307.54  avg =  306.96
 squeezenet_ssd_int8  min =  291.32  max =  292.87  avg =  292.18
       mobilenet_ssd  min =  423.70  max =  424.63  avg =  424.11

squeezenet_ssd  min =  306.35  max =  307.54  avg =  306.96
  mobilenet_ssd_int8  min =  358.62  max =  359.42  avg =  359.04
      mobilenet_yolo  min =  928.06  max =  929.25  avg =  928.55
  mobilenetv2_yolov3  min =  496.96  max =  499.29  avg =  497.73
         yolov4-tiny  min =  712.80  max =  714.15  avg =  713.55
           nanodet_m  min =  179.42  max =  180.60  avg =  179.75
    yolo-fastest-1.1  min =   88.06  max =   88.85  avg =   88.35
      yolo-fastestv2  min =   68.68  max =   69.83  avg =   69.08

nanopc-t4:/data/local/tmp # ./benchncnn 4 1 2 0 0
[0 Mali-T860]  queueC=0[2]  queueG=0[2]  queueT=0[2]
[0 Mali-T860]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=1
[0 Mali-T860]  fp16-p/s/a=1/0/1  int8-p/s/a=1/0/0
[0 Mali-T860]  subgroup=0  basic=0  vote=0  ballot=0  shuffle=0

          squeezenet  min =   24.57  max =   24.71  avg =   24.64
           mobilenet  min =   35.86  max =   36.14  avg =   36.04
        mobilenet_v2  min =   30.18  max =   30.19  avg =   30.19
        mobilenet_v3  min =   30.88  max =   31.12  avg =   31.01
          shufflenet  min =   33.90  max =   33.98  avg =   33.93
       shufflenet_v2  min =   29.10  max =   29.14  avg =   29.12
             mnasnet  min =   30.49  max =   30.59  avg =   30.53
     proxylessnasnet  min =   33.56  max =   33.61  avg =   33.59
     efficientnet_b0  min =   51.15  max =   51.54  avg =   51.38
   efficientnetv2_b0  min =   86.26  max =   87.36  avg =   86.91
        regnety_400m  min =   38.44  max =   38.54  avg =   38.49
           blazeface  min =    9.66  max =    9.74  avg =    9.70

efficientnetv2_b0  min =   86.26  max =   87.36  avg =   86.91
           googlenet  min =   80.62  max =   80.96  avg =   80.81
            resnet18  min =   74.07  max =   74.36  avg =   74.23
             alexnet  min =   76.84  max =   77.26  avg =   77.08
               vgg16  min =  300.71  max =  300.89  avg =  300.80
            resnet50  min =  175.96  max =  176.72  avg =  176.23
      squeezenet_ssd  min =   71.20  max =   71.38  avg =   71.32
       mobilenet_ssd  min =   76.99  max =   77.47  avg =   77.19
      mobilenet_yolo  min =  160.41  max =  160.84  avg =  160.62
  mobilenetv2_yolov3  min =   91.31  max =   91.37  avg =   91.35
         yolov4-tiny  min =  130.78  max =  131.54  avg =  131.16
           nanodet_m  min =   55.90  max =   56.03  avg =   55.96
    yolo-fastest-1.1  min =   25.50  max =   25.66  avg =   25.59

yolov4-tiny  min =  130.78  max =  131.54  avg =  131.16
      yolo-fastestv2  min =   24.94  max =   25.07  avg =   25.01
```

### OrangePi4 LTS, Rockchip RK3399 (Cortex-A72 1.8GHz x 2 + Cortex-A53 1.5GHz x 4)
Test Ubuntu 22.04 Gnome Desktop
```
orangepi@orangepi4-lts:~/ncnn/benchmark$ ./benchncnn 10 6 0 -1 0
num_threads = 6
          squeezenet  min =   40.89  max =   50.29  avg =   45.15
     squeezenet_int8  min =   40.36  max =   48.57  avg =   43.56
           mobilenet  min =   55.81  max =   67.35  avg =   59.81
      mobilenet_int8  min =   39.96  max =   45.10  avg =   42.09
        mobilenet_v2  min =   53.29  max =   64.12  avg =   57.40
        mobilenet_v3  min =   38.94  max =   51.11  avg =   43.06
          shufflenet  min =   27.32  max =   38.53  avg =   31.85

mobilenet_v2  min =   53.29  max =   64.12  avg =   57.40
       shufflenet_v2  min =   24.38  max =   31.17  avg =   28.32
             mnasnet  min =   47.02  max =   50.68  avg =   48.86
     proxylessnasnet  min =   52.31  max =   61.31  avg =   56.66
     efficientnet_b0  min =   68.14  max =   76.07  avg =   72.62
   efficientnetv2_b0  min =   77.23  max =   96.07  avg =   84.83
        regnety_400m  min =   60.81  max =   81.72  avg =   72.37
           blazeface  min =    7.24  max =    8.19  avg =    7.68
           googlenet  min =  122.99  max =  132.67  avg =  128.90
      googlenet_int8  min =  108.45  max =  121.17  avg =  115.37
            resnet18  min =  100.67  max =  115.30  avg =  107.65
       resnet18_int8  min =   80.17  max =   87.56  avg =   84.01
             alexnet  min =   71.00  max =   83.09  avg =   76.21

resnet18  min =  100.67  max =  115.30  avg =  107.65
               vgg16  min =  557.67  max =  606.30  avg =  581.12
          vgg16_int8  min =  369.93  max =  393.20  avg =  384.86
            resnet50  min =  254.25  max =  272.90  avg =  265.18
       resnet50_int8  min =  220.70  max =  231.50  avg =  225.03
      squeezenet_ssd  min =  118.91  max =  131.52  avg =  123.91
 squeezenet_ssd_int8  min =   98.25  max =  116.42  avg =  110.13
       mobilenet_ssd  min =  126.62  max =  134.13  avg =  129.56
  mobilenet_ssd_int8  min =   83.83  max =   91.61  avg =   86.75
      mobilenet_yolo  min =  281.19  max =  299.79  avg =  290.05
  mobilenetv2_yolov3  min =  180.37  max =  194.10  avg =  185.61
         yolov4-tiny  min =  215.28  max =  227.29  avg =  221.61
           nanodet_m  min =   64.63  max =   75.86  avg =   70.46

mobilenetv2_yolov3  min =  180.37  max =  194.10  avg =  185.61
    yolo-fastest-1.1  min =   39.54  max =   48.30  avg =   44.76
      yolo-fastestv2  min =   29.91  max =   53.15  avg =   37.32
  vision_transformer  min = 2520.25  max = 2595.28  avg = 2557.05
          FastestDet  min =   32.45  max =   47.38  avg =   40.55

orangepi@orangepi4-lts:~/ncnn/benchmark$ ./benchncnn 10 4 1 -1 0
          squeezenet  min =   48.90  max =   56.65  avg =   53.09
     squeezenet_int8  min =   48.09  max =   54.69  avg =   51.26
           mobilenet  min =   66.06  max =   79.73  avg =   73.96
      mobilenet_int8  min =   51.33  max =   58.30  avg =   54.71
        mobilenet_v2  min =   61.06  max =   88.93  avg =   71.48

mobilenet  min =   66.06  max =   79.73  avg =   73.96
        mobilenet_v3  min =   50.41  max =   65.40  avg =   56.51
          shufflenet  min =   38.11  max =   63.95  avg =   44.03
       shufflenet_v2  min =   33.27  max =   36.43  avg =   34.89
             mnasnet  min =   60.02  max =   72.71  avg =   64.57
     proxylessnasnet  min =   66.61  max =   73.25  avg =   70.65
     efficientnet_b0  min =   87.27  max =   94.97  avg =   91.00
   efficientnetv2_b0  min =   99.89  max =  112.09  avg =  106.13
        regnety_400m  min =   84.65  max =   92.78  avg =   89.51
           blazeface  min =    9.73  max =   11.45  avg =   10.85
           googlenet  min =  154.74  max =  164.25  avg =  159.33
      googlenet_int8  min =  140.29  max =  148.08  avg =  144.18
            resnet18  min =  131.51  max =  244.02  avg =  150.56

googlenet  min =  154.74  max =  164.25  avg =  159.33
       resnet18_int8  min =  102.11  max =  114.40  avg =  108.32
             alexnet  min =   81.13  max =   92.35  avg =   86.86
               vgg16  min =  649.91  max =  668.62  avg =  660.25
          vgg16_int8  min =  513.75  max =  523.77  avg =  518.17
            resnet50  min =  330.89  max =  378.23  avg =  344.07
       resnet50_int8  min =  280.38  max =  286.93  avg =  284.43
      squeezenet_ssd  min =  134.35  max =  146.97  avg =  141.17
 squeezenet_ssd_int8  min =  126.31  max =  137.29  avg =  130.73
       mobilenet_ssd  min =  146.83  max =  161.70  avg =  155.08
  mobilenet_ssd_int8  min =  105.74  max =  117.05  avg =  111.62
      mobilenet_yolo  min =  339.30  max =  352.16  avg =  345.22
  mobilenetv2_yolov3  min =  223.12  max =  234.18  avg =  229.81

mobilenet_ssd_int8  min =  105.74  max =  117.05  avg =  111.62
         yolov4-tiny  min =  267.30  max =  272.95  avg =  270.47
           nanodet_m  min =   78.72  max =   86.18  avg =   81.81
    yolo-fastest-1.1  min =   47.96  max =   55.08  avg =   51.81
      yolo-fastestv2  min =   38.01  max =   44.32  avg =   42.29
  vision_transformer  min = 3499.34  max = 3526.15  avg = 3514.43
          FastestDet  min =   40.14  max =   44.37  avg =   42.30

orangepi@orangepi4-lts:~/ncnn/benchmark$ ./benchncnn 10 2 2 -1 0
          squeezenet  min =   45.65  max =   46.72  avg =   46.15
     squeezenet_int8  min =   42.60  max =   43.01  avg =   42.76
           mobilenet  min =   69.35  max =   70.59  avg =   69.92

squeezenet  min =   45.65  max =   46.72  avg =   46.15
      mobilenet_int8  min =   46.08  max =   46.35  avg =   46.20
        mobilenet_v2  min =   57.47  max =   58.90  avg =   58.08
        mobilenet_v3  min =   44.72  max =   45.47  avg =   45.05
          shufflenet  min =   31.74  max =   32.16  avg =   31.97
       shufflenet_v2  min =   26.74  max =   26.98  avg =   26.86
             mnasnet  min =   50.47  max =   51.20  avg =   50.82
     proxylessnasnet  min =   57.31  max =   58.24  avg =   57.68
     efficientnet_b0  min =   79.61  max =   80.79  avg =   80.02
   efficientnetv2_b0  min =   92.67  max =   93.37  avg =   93.08
        regnety_400m  min =   67.08  max =   68.07  avg =   67.59
           blazeface  min =    8.56  max =    8.81  avg =    8.70
           googlenet  min =  136.82  max =  138.26  avg =  137.44

regnety_400m  min =   67.08  max =   68.07  avg =   67.59
      googlenet_int8  min =  121.96  max =  122.64  avg =  122.36
            resnet18  min =  118.04  max =  119.24  avg =  118.49
       resnet18_int8  min =   89.55  max =   92.11  avg =   90.38
             alexnet  min =   80.75  max =   82.34  avg =   81.24
               vgg16  min =  602.11  max =  628.12  avg =  612.26
          vgg16_int8  min =  481.31  max =  484.49  avg =  482.84
            resnet50  min =  307.31  max =  310.10  avg =  308.88
       resnet50_int8  min =  240.45  max =  243.43  avg =  241.76
      squeezenet_ssd  min =  119.65  max =  122.93  avg =  121.34
 squeezenet_ssd_int8  min =  102.71  max =  103.45  avg =  103.20
       mobilenet_ssd  min =  142.16  max =  143.58  avg =  142.54
  mobilenet_ssd_int8  min =   93.20  max =   93.81  avg =   93.41

squeezenet_ssd_int8  min =  102.71  max =  103.45  avg =  103.20
      mobilenet_yolo  min =  315.42  max =  318.06  avg =  317.00
  mobilenetv2_yolov3  min =  190.59  max =  191.74  avg =  190.96
         yolov4-tiny  min =  228.77  max =  230.49  avg =  229.78
           nanodet_m  min =   66.82  max =   67.23  avg =   67.02
    yolo-fastest-1.1  min =   38.20  max =   40.89  avg =   38.85
      yolo-fastestv2  min =   32.53  max =   33.48  avg =   33.03
  vision_transformer  min = 3372.17  max = 3516.54  avg = 3461.89
          FastestDet  min =   32.92  max =   35.55  avg =   33.62

```

### OrangePi5, Rockchip RK3588s (Quad Core A76 2.4GHz + Quad Core A55 1.8GHz)
```
orangepi@orangepi5:~/ncnn-master/benchmark$ ./benchncnn 10 8 0 -1 0

```
          squeezenet  min =    6.22  max =    6.69  avg =    6.37
     squeezenet_int8  min =    7.93  max =    8.32  avg =    8.07
           mobilenet  min =    9.08  max =   14.02  avg =    9.81
      mobilenet_int8  min =    7.89  max =    9.02  avg =    8.47
        mobilenet_v2  min =    7.77  max =    8.09  avg =    7.92
        mobilenet_v3  min =    6.87  max =    8.19  avg =    7.46
          shufflenet  min =    5.98  max =   10.21  avg =    7.23
       shufflenet_v2  min =    4.82  max =    5.04  avg =    4.93
             mnasnet  min =    6.15  max =    6.36  avg =    6.24
     proxylessnasnet  min =    9.50  max =   10.50  avg =    9.93
     efficientnet_b0  min =   11.46  max =   11.79  avg =   11.60
   efficientnetv2_b0  min =   18.61  max =   19.48  avg =   18.88

proxylessnasnet  min =    9.50  max =   10.50  avg =    9.93
        regnety_400m  min =   10.54  max =   12.44  avg =   10.86
           blazeface  min =    1.96  max =    5.35  avg =    2.58
           googlenet  min =   26.62  max =   32.59  avg =   29.96
      googlenet_int8  min =   28.27  max =   32.80  avg =   30.01
            resnet18  min =   15.52  max =   18.29  avg =   16.37
       resnet18_int8  min =   23.33  max =   26.89  avg =   24.99
             alexnet  min =   19.92  max =   22.75  avg =   21.06
               vgg16  min =  101.18  max =  122.44  avg =  107.45
          vgg16_int8  min =  164.69  max =  227.98  avg =  189.73
            resnet50  min =   42.96  max =   59.26  avg =   50.83
       resnet50_int8  min =   54.46  max =   66.72  avg =   61.37
      squeezenet_ssd  min =   24.39  max =   31.19  avg =   27.69

resnet50  min =   42.96  max =   59.26  avg =   50.83
 squeezenet_ssd_int8  min =   27.15  max =   41.55  avg =   33.68
       mobilenet_ssd  min =   22.26  max =   26.89  avg =   23.95
  mobilenet_ssd_int8  min =   21.18  max =   24.21  avg =   23.05
      mobilenet_yolo  min =   52.65  max =   65.53  avg =   58.47
  mobilenetv2_yolov3  min =   31.34  max =   45.15  avg =   34.63
         yolov4-tiny  min =   40.55  max =   49.32  avg =   43.85
           nanodet_m  min =   16.08  max =   19.51  avg =   17.58
    yolo-fastest-1.1  min =    6.48  max =    7.33  avg =    6.98
      yolo-fastestv2  min =    4.96  max =   11.66  avg =    7.30
  vision_transformer  min =  678.22  max =  815.73  avg =  729.16
          FastestDet  min =    4.95  max =   10.65  avg =    6.88

orangepi@orangepi5:~/ncnn-master/benchmark$ ./benchncnn 10 4 1 -1 0

FastestDet  min =    4.95  max =   10.65  avg =    6.88

          squeezenet  min =   10.91  max =   11.14  avg =   11.03
     squeezenet_int8  min =   14.26  max =   14.55  avg =   14.30
           mobilenet  min =   15.92  max =   16.26  avg =   16.11
      mobilenet_int8  min =   14.71  max =   15.22  avg =   14.91
        mobilenet_v2  min =   12.28  max =   12.49  avg =   12.37
        mobilenet_v3  min =   11.31  max =   11.72  avg =   11.46
          shufflenet  min =   10.10  max =   10.33  avg =   10.24
       shufflenet_v2  min =    9.38  max =    9.70  avg =    9.55
             mnasnet  min =   12.28  max =   12.80  avg =   12.44
     proxylessnasnet  min =   16.54  max =   16.66  avg =   16.60
     efficientnet_b0  min =   19.56  max =   20.66  avg =   19.86

mnasnet  min =   12.28  max =   12.80  avg =   12.44
   efficientnetv2_b0  min =   34.06  max =   34.65  avg =   34.41
        regnety_400m  min =   23.97  max =   24.69  avg =   24.20
           blazeface  min =    3.39  max =    3.56  avg =    3.48
           googlenet  min =   46.96  max =   47.90  avg =   47.56
      googlenet_int8  min =   49.56  max =   50.23  avg =   49.79
            resnet18  min =   28.44  max =   29.54  avg =   28.77
       resnet18_int8  min =   41.32  max =   42.44  avg =   41.67
             alexnet  min =   31.83  max =   32.77  avg =   32.32
               vgg16  min =  170.32  max =  178.30  avg =  173.22
          vgg16_int8  min =  282.55  max =  299.32  avg =  287.78
            resnet50  min =   78.00  max =   81.57  avg =   78.79
       resnet50_int8  min =   89.12  max =   92.31  avg =   90.92

vgg16_int8  min =  282.55  max =  299.32  avg =  287.78
      squeezenet_ssd  min =   38.07  max =   39.07  avg =   38.59
 squeezenet_ssd_int8  min =   50.98  max =   52.56  avg =   51.68
       mobilenet_ssd  min =   38.79  max =   39.67  avg =   39.34
  mobilenet_ssd_int8  min =   33.53  max =   35.26  avg =   34.66
      mobilenet_yolo  min =   90.50  max =   92.32  avg =   90.99
  mobilenetv2_yolov3  min =   51.38  max =   51.93  avg =   51.56
         yolov4-tiny  min =   75.65  max =   76.80  avg =   76.17
           nanodet_m  min =   21.33  max =   21.68  avg =   21.50
    yolo-fastest-1.1  min =   11.18  max =   12.06  avg =   11.36
      yolo-fastestv2  min =    9.87  max =   10.33  avg =   10.15
  vision_transformer  min = 1475.77  max = 1477.97  avg = 1476.77
          FastestDet  min =    9.39  max =    9.73  avg =    9.53

yolo-fastestv2  min =    9.87  max =   10.33  avg =   10.15

orangepi@orangepi5:~/ncnn-master/benchmark$ ./benchncnn 10 4 2 -1 0
          squeezenet  min =    3.59  max =    3.70  avg =    3.66
     squeezenet_int8  min =    4.32  max =    4.42  avg =    4.36
           mobilenet  min =    5.50  max =    5.55  avg =    5.53
      mobilenet_int8  min =    4.52  max =    4.60  avg =    4.56
        mobilenet_v2  min =    4.50  max =    4.60  avg =    4.54
        mobilenet_v3  min =    4.09  max =    4.28  avg =    4.15
          shufflenet  min =    3.49  max =    3.58  avg =    3.51
       shufflenet_v2  min =    2.91  max =    3.07  avg =    2.97
             mnasnet  min =    4.18  max =    4.25  avg =    4.21

shufflenet  min =    3.49  max =    3.58  avg =    3.51
     proxylessnasnet  min =    4.94  max =    5.00  avg =    4.97
     efficientnet_b0  min =    7.50  max =    7.54  avg =    7.52
   efficientnetv2_b0  min =   11.32  max =   11.41  avg =   11.37
        regnety_400m  min =    7.92  max =    8.01  avg =    7.95
           blazeface  min =    1.21  max =    1.31  avg =    1.24
           googlenet  min =   15.03  max =   15.17  avg =   15.10
      googlenet_int8  min =   15.48  max =   15.61  avg =   15.55
            resnet18  min =    9.91  max =    9.97  avg =    9.93
       resnet18_int8  min =   15.80  max =   16.00  avg =   15.89
             alexnet  min =   12.35  max =   12.64  avg =   12.48
               vgg16  min =   61.92  max =   65.62  avg =   62.93
          vgg16_int8  min =  129.94  max =  131.65  avg =  130.65

alexnet  min =   12.35  max =   12.64  avg =   12.48
            resnet50  min =   27.41  max =   27.62  avg =   27.52
       resnet50_int8  min =   33.01  max =   33.23  avg =   33.08
      squeezenet_ssd  min =   13.92  max =   14.27  avg =   14.02
 squeezenet_ssd_int8  min =   18.04  max =   18.40  avg =   18.15
       mobilenet_ssd  min =   13.69  max =   13.80  avg =   13.74
  mobilenet_ssd_int8  min =   10.95  max =   11.10  avg =   11.02
      mobilenet_yolo  min =   32.06  max =   32.30  avg =   32.17
  mobilenetv2_yolov3  min =   19.27  max =   20.68  avg =   19.97
         yolov4-tiny  min =   25.41  max =   29.51  avg =   27.76
           nanodet_m  min =    6.68  max =    6.73  avg =    6.70
    yolo-fastest-1.1  min =    3.77  max =    4.02  avg =    3.83
      yolo-fastestv2  min =    3.41  max =    3.65  avg =    3.48

nanodet_m  min =    6.68  max =    6.73  avg =    6.70
  vision_transformer  min =  548.32  max =  654.71  avg =  579.48
          FastestDet  min =    3.38  max =    3.46  avg =    3.42

```

### RDK X3 Module (Cortex-A53 1.5GHz x 4) aarch64
```
root@ubuntu:/home/sunrise/ncnn-master/benchmark# ../build-aarch64-linux-gnu/benchmark/benchncnn 10 4 0 -1 1
          squeezenet  min =   49.83  max =   50.57  avg =   50.08
     squeezenet_int8  min =   48.43  max =   49.18  avg =   48.67
           mobilenet  min =   68.37  max =   69.09  avg =   68.63
      mobilenet_int8  min =   58.19  max =   58.72  avg =   58.37
        mobilenet_v2  min =   58.76  max =   60.62  avg =   59.20
        mobilenet_v3  min =   49.75  max =   50.60  avg =   50.06

mobilenet_int8  min =   58.19  max =   58.72  avg =   58.37
          shufflenet  min =   37.17  max =   37.96  avg =   37.50
       shufflenet_v2  min =   32.08  max =   32.42  avg =   32.22
             mnasnet  min =   55.51  max =   57.02  avg =   55.90
     proxylessnasnet  min =   68.15  max =   69.53  avg =   68.78
     efficientnet_b0  min =   88.64  max =   90.16  avg =   89.43
   efficientnetv2_b0  min =  102.45  max =  103.42  avg =  102.92
        regnety_400m  min =   88.22  max =   89.09  avg =   88.62
           blazeface  min =    9.78  max =   10.15  avg =    9.93
           googlenet  min =  152.20  max =  153.92  avg =  153.28
      googlenet_int8  min =  141.80  max =  143.30  avg =  142.48
            resnet18  min =  116.70  max =  117.59  avg =  117.03
       resnet18_int8  min =  104.42  max =  105.85  avg =  104.94

googlenet_int8  min =  141.80  max =  143.30  avg =  142.48
             alexnet  min =   82.55  max =   83.23  avg =   82.82
               vgg16  min =  590.22  max =  598.18  avg =  594.35
          vgg16_int8  min =  504.56  max =  507.21  avg =  505.73
            resnet50  min =  307.36  max =  308.68  avg =  308.03
       resnet50_int8  min =  281.35  max =  283.87  avg =  282.30
      squeezenet_ssd  min =  124.93  max =  126.51  avg =  125.51
 squeezenet_ssd_int8  min =  118.07  max =  118.89  avg =  118.29
       mobilenet_ssd  min =  142.27  max =  142.57  avg =  142.44
  mobilenet_ssd_int8  min =  116.51  max =  117.60  avg =  117.04
      mobilenet_yolo  min =  314.64  max =  317.09  avg =  315.93
  mobilenetv2_yolov3  min =  204.55  max =  205.30  avg =  204.93
         yolov4-tiny  min =  246.69  max =  249.64  avg =  247.95

mobilenet_yolo  min =  314.64  max =  317.09  avg =  315.93
           nanodet_m  min =   77.73  max =   78.30  avg =   77.99
    yolo-fastest-1.1  min =   46.29  max =   47.52  avg =   46.93
      yolo-fastestv2  min =   36.55  max =   36.95  avg =   36.73
  vision_transformer  min = 3372.85  max = 3409.14  avg = 3377.75
          FastestDet  min =   38.23  max =   38.77  avg =   38.49

```

### NanoPi R2S, Rockchip RK3328 (Cortex-A53 1.3GHz x 4) Armbian focal (21.05.1) aarch64
```
root@nanopi-r2s:~/ncnn/build/benchmark# ./benchncnn 8 4 0
          squeezenet  min =   62.20  max =   62.81  avg =   62.49
     squeezenet_int8  min =   57.92  max =   71.46  avg =   59.76
           mobilenet  min =   82.88  max =   89.36  avg =   84.52

squeezenet  min =   62.20  max =   62.81  avg =   62.49
      mobilenet_int8  min =   57.16  max =   96.22  avg =   62.29
        mobilenet_v2  min =   73.68  max =   75.92  avg =   74.17
        mobilenet_v3  min =   59.57  max =   60.14  avg =   59.84
          shufflenet  min =   52.34  max =   52.70  avg =   52.53
       shufflenet_v2  min =   45.51  max =   45.92  avg =   45.73
             mnasnet  min =   67.75  max =   83.15  avg =   69.82
     proxylessnasnet  min =   81.70  max =   83.66  avg =   82.31
     efficientnet_b0  min =  121.10  max =  123.22  avg =  121.55
   efficientnetv2_b0  min =  138.93  max =  192.15  avg =  154.94
        regnety_400m  min =   99.62  max =  116.29  avg =  101.97
           blazeface  min =   18.80  max =   19.15  avg =   19.01
           googlenet  min =  176.36  max =  202.84  avg =  181.86

regnety_400m  min =   99.62  max =  116.29  avg =  101.97
      googlenet_int8  min =  155.50  max =  190.50  avg =  161.20
            resnet18  min =  165.79  max =  201.57  avg =  172.56
       resnet18_int8  min =  122.24  max =  160.53  avg =  134.24
             alexnet  min =  227.07  max =  238.09  avg =  232.19
          vgg16_int8  min =  522.14  max =  551.75  avg =  531.68
            resnet50  min =  378.30  max =  440.21  avg =  388.56
       resnet50_int8  min =  315.76  max =  373.97  avg =  329.88
      squeezenet_ssd  min =  175.37  max =  200.86  avg =  179.01
 squeezenet_ssd_int8  min =  134.71  max =  147.57  avg =  136.57
       mobilenet_ssd  min =  174.43  max =  212.11  avg =  180.61
  mobilenet_ssd_int8  min =  119.41  max =  153.75  avg =  124.21
      mobilenet_yolo  min =  366.27  max =  422.67  avg =  383.65

mobilenet_ssd  min =  174.43  max =  212.11  avg =  180.61
  mobilenetv2_yolov3  min =  238.56  max =  281.97  avg =  247.56
         yolov4-tiny  min =  311.45  max =  333.32  avg =  316.79
           nanodet_m  min =  114.15  max =  122.39  avg =  115.44

root@nanopi-r2s:~/ncnn/build/benchmark# ./benchncnn 8 2 0
          squeezenet  min =   89.02  max =   90.52  avg =   89.35
     squeezenet_int8  min =   81.19  max =   81.90  avg =   81.42
           mobilenet  min =  131.47  max =  134.39  avg =  132.34
      mobilenet_int8  min =  102.20  max =  103.03  avg =  102.66
        mobilenet_v2  min =  102.40  max =  108.12  avg =  103.91
        mobilenet_v3  min =   89.17  max =   90.10  avg =   89.53
          shufflenet  min =   65.74  max =   68.86  avg =   66.50

mobilenet_v2  min =  102.40  max =  108.12  avg =  103.91
       shufflenet_v2  min =   62.83  max =   64.41  avg =   63.25
             mnasnet  min =   98.01  max =   98.24  avg =   98.14
     proxylessnasnet  min =  121.10  max =  123.55  avg =  121.80
     efficientnet_b0  min =  187.79  max =  188.41  avg =  188.08
   efficientnetv2_b0  min =  211.96  max =  213.99  avg =  212.74
        regnety_400m  min =  124.98  max =  125.49  avg =  125.28
           blazeface  min =   24.91  max =   25.14  avg =   25.00
           googlenet  min =  278.47  max =  283.24  avg =  280.79
      googlenet_int8  min =  243.81  max =  247.82  avg =  245.30
            resnet18  min =  257.46  max =  259.29  avg =  258.29
       resnet18_int8  min =  187.18  max =  188.74  avg =  187.70
             alexnet  min =  384.52  max =  387.07  avg =  385.84

resnet18  min =  257.46  max =  259.29  avg =  258.29
          vgg16_int8  min =  897.26  max =  901.68  avg =  899.19
            resnet50  min =  618.85  max =  623.92  avg =  620.85
       resnet50_int8  min =  512.33  max =  514.93  avg =  513.64
      squeezenet_ssd  min =  211.21  max =  218.71  avg =  213.02
 squeezenet_ssd_int8  min =  193.32  max =  193.97  avg =  193.70
       mobilenet_ssd  min =  271.11  max =  275.58  avg =  272.06
  mobilenet_ssd_int8  min =  208.80  max =  209.59  avg =  209.05
      mobilenet_yolo  min =  570.55  max =  575.98  avg =  572.73
  mobilenetv2_yolov3  min =  329.04  max =  353.84  avg =  340.42
         yolov4-tiny  min =  435.16  max =  463.68  avg =  457.69
           nanodet_m  min =  155.70  max =  159.13  avg =  156.50
```

yolov4-tiny  min =  435.16  max =  463.68  avg =  457.69
```

### EAIDK 310, Rockchip RK3228H (Cortex-A53 1.3GHz x 4) fedora-28 aarch64
```
[openailab@MiWiFi-R1D-srv benchmark]$ ./benchncnn 8 4 0 -1 1
          squeezenet  min =   68.97  max =   71.42  avg =   69.65
     squeezenet_int8  min =   58.47  max =   59.58  avg =   58.77
           mobilenet  min =   90.87  max =  100.18  avg =   92.48
      mobilenet_int8  min =   59.46  max =   63.02  avg =   60.01
        mobilenet_v2  min =   82.92  max =  112.01  avg =   88.10
        mobilenet_v3  min =   66.65  max =   69.57  avg =   67.27
          shufflenet  min =   48.22  max =   48.49  avg =   48.34
       shufflenet_v2  min =   48.52  max =   52.88  avg =   49.17
             mnasnet  min =   75.63  max =   79.83  avg =   76.43

shufflenet  min =   48.22  max =   48.49  avg =   48.34
     proxylessnasnet  min =   84.73  max =   86.69  avg =   85.16
     efficientnet_b0  min =  125.69  max =  129.00  avg =  126.38
   efficientnetv2_b0  min =  144.44  max =  149.01  avg =  145.33
        regnety_400m  min =   99.69  max =  101.23  avg =  100.38
           blazeface  min =   15.84  max =   16.24  avg =   16.03
           googlenet  min =  194.64  max =  199.29  avg =  196.07
      googlenet_int8  min =  158.54  max =  165.64  avg =  160.25
            resnet18  min =  200.65  max =  221.60  avg =  204.30
       resnet18_int8  min =  122.69  max =  126.57  avg =  123.54
             alexnet  min =  175.54  max =  200.91  avg =  181.38
            resnet50  min =  428.75  max =  466.51  avg =  439.67
       resnet50_int8  min =  324.95  max =  347.47  avg =  329.74

alexnet  min =  175.54  max =  200.91  avg =  181.38
      squeezenet_ssd  min =  199.86  max =  207.51  avg =  201.99
 squeezenet_ssd_int8  min =  150.35  max =  176.92  avg =  154.60
       mobilenet_ssd  min =  186.50  max =  189.92  avg =  188.09
  mobilenet_ssd_int8  min =  123.55  max =  127.17  avg =  124.63
      mobilenet_yolo  min =  393.83  max =  414.09  avg =  398.57
  mobilenetv2_yolov3  min =  263.49  max =  273.11  avg =  266.11
         yolov4-tiny  min =  342.33  max =  363.69  avg =  346.34
           nanodet_m  min =  119.66  max =  127.29  avg =  121.26
    yolo-fastest-1.1  min =   61.87  max =   90.26  avg =   65.77
      yolo-fastestv2  min =   48.48  max =   50.82  avg =   48.93

[openailab@MiWiFi-R1D-srv benchmark]$ ./benchncnn 4 1 0 -1 1

          squeezenet  min =  152.15  max =  152.67  avg =  152.43
     squeezenet_int8  min =  143.22  max =  144.24  avg =  143.61
           mobilenet  min =  237.77  max =  239.69  avg =  238.47
      mobilenet_int8  min =  199.91  max =  201.35  avg =  200.50
        mobilenet_v2  min =  169.67  max =  170.18  avg =  169.93
        mobilenet_v3  min =  150.06  max =  151.17  avg =  150.78
          shufflenet  min =   91.78  max =   92.38  avg =   92.06
       shufflenet_v2  min =  100.86  max =  101.75  avg =  101.50
             mnasnet  min =  165.10  max =  166.74  avg =  166.24
     proxylessnasnet  min =  218.42  max =  220.55  avg =  219.12
     efficientnet_b0  min =  348.00  max =  349.03  avg =  348.49
   efficientnetv2_b0  min =  404.06  max =  406.16  avg =  405.00
        regnety_400m  min =  209.48  max =  211.36  avg =  210.44

efficientnet_b0  min =  348.00  max =  349.03  avg =  348.49
           blazeface  min =   31.31  max =   32.61  avg =   32.00
           googlenet  min =  510.38  max =  512.43  avg =  511.25
      googlenet_int8  min =  454.38  max =  456.19  avg =  455.02
            resnet18  min =  407.78  max =  409.45  avg =  408.34
       resnet18_int8  min =  357.01  max =  360.72  avg =  358.74
             alexnet  min =  504.12  max =  506.74  avg =  505.08
            resnet50  min = 1115.42  max = 1121.91  avg = 1118.67
       resnet50_int8  min =  973.38  max =  976.26  avg =  975.21
      squeezenet_ssd  min =  361.52  max =  363.69  avg =  362.38
 squeezenet_ssd_int8  min =  333.81  max =  337.16  avg =  335.24
       mobilenet_ssd  min =  477.43  max =  478.36  avg =  477.82
  mobilenet_ssd_int8  min =  409.33  max =  409.67  avg =  409.52

squeezenet_ssd_int8  min =  333.81  max =  337.16  avg =  335.24
      mobilenet_yolo  min = 1048.79  max = 1057.72  avg = 1053.80
  mobilenetv2_yolov3  min =  567.04  max =  571.44  avg =  569.04
         yolov4-tiny  min =  788.40  max =  790.74  avg =  789.12
           nanodet_m  min =  253.68  max =  254.59  avg =  254.16
    yolo-fastest-1.1  min =  102.44  max =  103.11  avg =  102.67
      yolo-fastestv2  min =   82.19  max =   82.43  avg =   82.35
```

### NVIDIA Jetson Nano
```
[0 NVIDIA Tegra X1 (nvgpu)]  queueC=0[16]  queueG=0[16]  queueT=0[16]
[0 NVIDIA Tegra X1 (nvgpu)]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=0
[0 NVIDIA Tegra X1 (nvgpu)]  fp16-p/s/a=1/1/1  int8-p/s/a=1/1/1
[0 NVIDIA Tegra X1 (nvgpu)]  subgroup=32  basic=1  vote=1  ballot=1  shuffle=1

          squeezenet  min =   12.15  max =   26.48  avg =   18.11
     squeezenet_int8  min =   27.60  max =   42.50  avg =   29.89
           mobilenet  min =   16.07  max =   16.10  avg =   16.09
      mobilenet_int8  min =   30.65  max =   32.15  avg =   31.07
        mobilenet_v2  min =   12.87  max =   13.15  avg =   12.99
        mobilenet_v3  min =   13.32  max =   16.65  avg =   14.57
          shufflenet  min =   14.21  max =   14.34  avg =   14.29
       shufflenet_v2  min =   13.03  max =   21.97  avg =   19.02
             mnasnet  min =   13.33  max =   13.64  avg =   13.49
     proxylessnasnet  min =   14.65  max =   14.91  avg =   14.76
     efficientnet_b0  min =   21.26  max =   21.41  avg =   21.35
   efficientnetv2_b0  min =   54.66  max =   60.81  avg =   57.16

proxylessnasnet  min =   14.65  max =   14.91  avg =   14.76
        regnety_400m  min =   17.91  max =   18.08  avg =   18.01
           blazeface  min =    6.87  max =    7.03  avg =    6.94
           googlenet  min =   43.30  max =   43.54  avg =   43.43
      googlenet_int8  min =   80.07  max =   84.28  avg =   81.10
            resnet18  min =   43.89  max =   44.06  avg =   43.98
       resnet18_int8  min =   60.70  max =   63.43  avg =   61.60
             alexnet  min =   74.21  max =   75.20  avg =   74.45
               vgg16  min =  310.39  max =  310.65  avg =  310.52
          vgg16_int8  min =  293.15  max =  297.28  avg =  294.93
            resnet50  min =   93.03  max =   93.22  avg =   93.12
       resnet50_int8  min =  158.54  max =  161.25  avg =  159.56
      squeezenet_ssd  min =   55.88  max =   57.43  avg =   56.46

resnet50  min =   93.03  max =   93.22  avg =   93.12
 squeezenet_ssd_int8  min =   72.42  max =   73.25  avg =   72.73
       mobilenet_ssd  min =   35.38  max =   37.57  avg =   36.63
  mobilenet_ssd_int8  min =   62.92  max =   64.97  avg =   63.63
      mobilenet_yolo  min =   76.56  max =   80.44  avg =   78.05
  mobilenetv2_yolov3  min =   46.35  max =   48.14  avg =   47.26
         yolov4-tiny  min =   95.38  max =   97.55  avg =   96.45
           nanodet_m  min =   22.82  max =   26.01  avg =   24.48
    yolo-fastest-1.1  min =   20.23  max =   25.51  avg =   21.52
      yolo-fastestv2  min =   20.67  max =   20.82  avg =   20.75

          squeezenet  min =   12.00  max =   15.41  avg =   13.55
     squeezenet_int8  min =   78.76  max =   79.14  avg =   78.91
           mobilenet  min =   16.03  max =   16.25  avg =   16.15
      mobilenet_int8  min =  107.58  max =  107.68  avg =  107.61
        mobilenet_v2  min =   12.84  max =   13.13  avg =   12.99
        mobilenet_v3  min =   13.29  max =   16.64  avg =   14.38
          shufflenet  min =   14.23  max =   14.54  avg =   14.34
       shufflenet_v2  min =   12.94  max =   13.21  avg =   13.02
             mnasnet  min =   13.42  max =   13.66  avg =   13.53
     proxylessnasnet  min =   14.64  max =   14.94  avg =   14.76

shufflenet_v2  min =   12.94  max =   13.21  avg =   13.02
     efficientnet_b0  min =   21.28  max =   21.51  avg =   21.36
   efficientnetv2_b0  min =   74.32  max =   78.50  avg =   77.79
        regnety_400m  min =   17.94  max =   18.26  avg =   18.07
           blazeface  min =    6.83  max =    6.94  avg =    6.89
           googlenet  min =   43.45  max =   43.63  avg =   43.52
      googlenet_int8  min =  255.68  max =  256.33  avg =  255.92
            resnet18  min =   43.96  max =   44.06  avg =   44.01
       resnet18_int8  min =  192.01  max =  192.64  avg =  192.33
             alexnet  min =   74.04  max =   74.23  avg =   74.14
               vgg16  min =  310.32  max =  310.64  avg =  310.44
          vgg16_int8  min = 1003.05  max = 1004.27  avg = 1003.66
            resnet50  min =   93.05  max =   93.34  avg =   93.21

vgg16  min =  310.32  max =  310.64  avg =  310.44
       resnet50_int8  min =  516.27  max =  517.12  avg =  516.69
      squeezenet_ssd  min =   56.67  max =   56.86  avg =   56.73
 squeezenet_ssd_int8  min =  182.96  max =  184.26  avg =  183.71
       mobilenet_ssd  min =   35.61  max =   35.70  avg =   35.65
  mobilenet_ssd_int8  min =  217.02  max =  217.50  avg =  217.23
      mobilenet_yolo  min =   78.10  max =   78.36  avg =   78.20
  mobilenetv2_yolov3  min =   49.86  max =   57.83  avg =   53.18
         yolov4-tiny  min =   96.76  max =   96.86  avg =   96.82
           nanodet_m  min =   25.26  max =   25.36  avg =   25.31
    yolo-fastest-1.1  min =   21.55  max =   24.22  avg =   23.78
      yolo-fastestv2  min =   20.80  max =   21.01  avg =   20.90

yolo-fastest-1.1  min =   21.55  max =   24.22  avg =   23.78

          squeezenet  min =   30.03  max =   31.41  avg =   30.59
     squeezenet_int8  min =   27.32  max =   27.76  avg =   27.50
           mobilenet  min =   41.74  max =   42.57  avg =   42.05
      mobilenet_int8  min =   30.48  max =   31.57  avg =   30.85
        mobilenet_v2  min =   33.49  max =   34.18  avg =   33.83
        mobilenet_v3  min =   30.59  max =   30.96  avg =   30.79
          shufflenet  min =   21.07  max =   31.68  avg =   22.53
       shufflenet_v2  min =   19.55  max =   20.01  avg =   19.71
             mnasnet  min =   31.70  max =   32.26  avg =   31.93
     proxylessnasnet  min =   36.90  max =   38.55  avg =   37.27
     efficientnet_b0  min =   68.42  max =   77.60  avg =   70.60

mnasnet  min =   31.70  max =   32.26  avg =   31.93
   efficientnetv2_b0  min =   73.72  max =   81.05  avg =   75.31
        regnety_400m  min =   56.67  max =   66.82  avg =   58.24
           blazeface  min =    6.55  max =    6.96  avg =    6.74
           googlenet  min =   92.74  max =   94.22  avg =   93.12
      googlenet_int8  min =   80.86  max =   87.28  avg =   82.41
            resnet18  min =   83.10  max =   84.30  avg =   83.44
       resnet18_int8  min =   59.40  max =   65.86  avg =   60.70
             alexnet  min =   89.21  max =   92.45  avg =   89.98
               vgg16  min =  445.72  max =  451.09  avg =  447.39
          vgg16_int8  min =  292.81  max =  295.55  avg =  294.34
            resnet50  min =  203.42  max =  204.45  avg =  204.08
       resnet50_int8  min =  157.87  max =  160.30  avg =  158.67

vgg16_int8  min =  292.81  max =  295.55  avg =  294.34
      squeezenet_ssd  min =   85.60  max =   87.24  avg =   86.18
 squeezenet_ssd_int8  min =   73.10  max =   85.64  avg =   74.94
       mobilenet_ssd  min =   86.75  max =   96.51  avg =   88.49
  mobilenet_ssd_int8  min =   63.40  max =   71.57  avg =   64.97
      mobilenet_yolo  min =  193.84  max =  195.24  avg =  194.62
  mobilenetv2_yolov3  min =  115.80  max =  117.27  avg =  116.27
         yolov4-tiny  min =  156.30  max =  158.26  avg =  156.81
           nanodet_m  min =   46.64  max =   47.97  avg =   47.12
    yolo-fastest-1.1  min =   25.78  max =   27.86  avg =   26.29
      yolo-fastestv2  min =   20.54  max =   30.73  avg =   22.18

yolo-fastestv2  min =   20.54  max =   30.73  avg =   22.18

          squeezenet  min =   85.91  max =   86.86  avg =   86.14
     squeezenet_int8  min =   77.57  max =   78.10  avg =   77.69
           mobilenet  min =  137.43  max =  138.03  avg =  137.63
      mobilenet_int8  min =  108.06  max =  108.21  avg =  108.13
        mobilenet_v2  min =   93.81  max =   94.70  avg =   93.99
        mobilenet_v3  min =   81.77  max =   82.49  avg =   81.99
          shufflenet  min =   47.84  max =   48.46  avg =   48.17
       shufflenet_v2  min =   47.93  max =   48.23  avg =   48.09
             mnasnet  min =   91.73  max =   92.55  avg =   91.98
     proxylessnasnet  min =  115.41  max =  115.75  avg =  115.56
     efficientnet_b0  min =  225.64  max =  226.21  avg =  225.94
   efficientnetv2_b0  min =  239.71  max =  240.20  avg =  239.89

proxylessnasnet  min =  115.41  max =  115.75  avg =  115.56
        regnety_400m  min =  118.46  max =  118.84  avg =  118.61
           blazeface  min =   15.58  max =   17.14  avg =   16.21
           googlenet  min =  286.85  max =  287.51  avg =  287.11
      googlenet_int8  min =  256.44  max =  256.74  avg =  256.53
            resnet18  min =  221.27  max =  221.93  avg =  221.60
       resnet18_int8  min =  189.95  max =  191.34  avg =  190.74
             alexnet  min =  284.30  max =  285.40  avg =  284.87
               vgg16  min = 1241.51  max = 1244.53  avg = 1242.90
          vgg16_int8  min = 1003.92  max = 1004.47  avg = 1004.29
            resnet50  min =  624.43  max =  625.34  avg =  624.84
       resnet50_int8  min =  516.64  max =  517.26  avg =  516.99
      squeezenet_ssd  min =  190.21  max =  191.35  avg =  190.71

resnet50  min =  624.43  max =  625.34  avg =  624.84
 squeezenet_ssd_int8  min =  182.97  max =  184.19  avg =  183.38
       mobilenet_ssd  min =  275.60  max =  276.17  avg =  275.90
  mobilenet_ssd_int8  min =  216.67  max =  217.58  avg =  216.94
      mobilenet_yolo  min =  616.16  max =  617.45  avg =  616.71
  mobilenetv2_yolov3  min =  324.88  max =  325.73  avg =  325.19
         yolov4-tiny  min =  421.01  max =  423.52  avg =  422.14
           nanodet_m  min =  117.39  max =  117.75  avg =  117.54
    yolo-fastest-1.1  min =   54.55  max =   55.61  avg =   54.87
      yolo-fastestv2  min =   44.40  max =   44.78  avg =   44.57
```

### Rockchip RK3288-CG.W (Cortex-A17 1.8GHz x 4)
```
WW_Tinker_Board:/data/local/tmp # ./benchncnn 8 4 0 -1 1

```

```
          squeezenet  min =   56.61  max =   56.80  avg =   56.69
     squeezenet_int8  min =   40.63  max =   41.05  avg =   40.89
           mobilenet  min =   83.91  max =   84.59  avg =   84.23
      mobilenet_int8  min =   36.15  max =   36.44  avg =   36.25
        mobilenet_v2  min =   71.12  max =   71.73  avg =   71.54
        mobilenet_v3  min =   56.08  max =   56.56  avg =   56.28
          shufflenet  min =   37.39  max =   37.75  avg =   37.55
       shufflenet_v2  min =   35.19  max =   35.52  avg =   35.34
             mnasnet  min =   62.08  max =   62.36  avg =   62.24
     proxylessnasnet  min =   66.98  max =   67.38  avg =   67.16
     efficientnet_b0  min =  109.95  max =  110.71  avg =  110.15
   efficientnetv2_b0  min =  122.56  max =  123.31  avg =  122.94

proxylessnasnet  min =   66.98  max =   67.38  avg =   67.16
        regnety_400m  min =   88.84  max =   89.19  avg =   88.99
           blazeface  min =   11.79  max =   11.92  avg =   11.85
           googlenet  min =  162.56  max =  165.39  avg =  163.19
      googlenet_int8  min =  110.35  max =  110.91  avg =  110.60
            resnet18  min =  172.39  max =  173.99  avg =  173.24
       resnet18_int8  min =   84.00  max =   84.40  avg =   84.19
             alexnet  min =  156.71  max =  158.23  avg =  157.59
               vgg16  min =  956.95  max =  964.32  avg =  960.60
          vgg16_int8  min =  388.10  max =  389.52  avg =  388.68
            resnet50  min =  403.05  max =  404.80  avg =  404.01
       resnet50_int8  min =  205.12  max =  207.42  avg =  206.19
      squeezenet_ssd  min =  163.61  max =  165.79  avg =  164.93

resnet50  min =  403.05  max =  404.80  avg =  404.01
 squeezenet_ssd_int8  min =  125.88  max =  126.35  avg =  126.12
       mobilenet_ssd  min =  175.97  max =  176.86  avg =  176.39
  mobilenet_ssd_int8  min =   76.90  max =   77.74  avg =   77.35
      mobilenet_yolo  min =  385.59  max =  387.19  avg =  386.60
  mobilenetv2_yolov3  min =  234.88  max =  236.22  avg =  235.66
         yolov4-tiny  min =  307.44  max =  310.64  avg =  308.54
           nanodet_m  min =   92.54  max =   93.15  avg =   92.82
    yolo-fastest-1.1  min =   46.69  max =   47.02  avg =   46.83
      yolo-fastestv2  min =   38.37  max =   38.68  avg =   38.54

WW_Tinker_Board:/data/local/tmp # ./benchncnn 4 1 0 -1 1
          squeezenet  min =  138.27  max =  138.57  avg =  138.41

     squeezenet_int8  min =   85.97  max =   86.23  avg =   86.05
           mobilenet  min =  234.90  max =  235.08  avg =  235.00
      mobilenet_int8  min =   99.92  max =  100.45  avg =  100.12
        mobilenet_v2  min =  157.76  max =  157.99  avg =  157.86
        mobilenet_v3  min =  130.05  max =  130.23  avg =  130.17
          shufflenet  min =   74.48  max =   74.62  avg =   74.55
       shufflenet_v2  min =   74.05  max =   74.25  avg =   74.13
             mnasnet  min =  150.74  max =  151.03  avg =  150.87
     proxylessnasnet  min =  171.09  max =  171.23  avg =  171.16
     efficientnet_b0  min =  306.85  max =  307.02  avg =  306.97
   efficientnetv2_b0  min =  347.40  max =  347.87  avg =  347.64
        regnety_400m  min =  190.26  max =  190.33  avg =  190.29

efficientnet_b0  min =  306.85  max =  307.02  avg =  306.97
           blazeface  min =   25.25  max =   25.68  avg =   25.47
           googlenet  min =  432.09  max =  432.48  avg =  432.32
      googlenet_int8  min =  275.55  max =  276.07  avg =  275.88
            resnet18  min =  355.11  max =  358.56  avg =  356.90
       resnet18_int8  min =  205.80  max =  206.68  avg =  206.26
             alexnet  min =  330.09  max =  330.29  avg =  330.15
               vgg16  min = 2122.95  max = 2124.45  avg = 2123.68
          vgg16_int8  min = 1048.53  max = 1049.29  avg = 1048.86
            resnet50  min = 1047.27  max = 1048.33  avg = 1047.63
       resnet50_int8  min =  517.75  max =  519.28  avg =  518.81
      squeezenet_ssd  min =  304.69  max =  305.75  avg =  305.16
 squeezenet_ssd_int8  min =  219.16  max =  219.94  avg =  219.45

resnet50_int8  min =  517.75  max =  519.28  avg =  518.81
       mobilenet_ssd  min =  483.73  max =  484.12  avg =  484.01
  mobilenet_ssd_int8  min =  208.89  max =  209.19  avg =  209.09
      mobilenet_yolo  min = 1092.75  max = 1093.70  avg = 1093.13
  mobilenetv2_yolov3  min =  560.66  max =  560.92  avg =  560.77
         yolov4-tiny  min =  704.69  max =  705.38  avg =  705.12
           nanodet_m  min =  187.13  max =  187.57  avg =  187.39
    yolo-fastest-1.1  min =   83.05  max =   83.11  avg =   83.08
      yolo-fastestv2  min =   72.19  max =   72.23  avg =   72.21

WW_Tinker_Board:/data/local/tmp # ./benchncnn 4 1 0 0 0
[0 Mali-T760]  queueC=0[2]  queueG=0[2]  queueT=0[2]
[0 Mali-T760]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=1
[0 Mali-T760]  fp16-p/s/a=1/0/1  int8-p/s/a=1/0/0

[0 Mali-T760]  subgroup=0  basic=0  vote=0  ballot=0  shuffle=0
          squeezenet  min =   41.78  max =   41.82  avg =   41.79
           mobilenet  min =   62.67  max =   62.80  avg =   62.74
        mobilenet_v2  min =   51.08  max =   51.26  avg =   51.17
        mobilenet_v3  min =   51.43  max =   51.70  avg =   51.51
          shufflenet  min =   56.83  max =   56.94  avg =   56.87
       shufflenet_v2  min =   48.46  max =   48.63  avg =   48.53
             mnasnet  min =   52.31  max =   52.63  avg =   52.42
     proxylessnasnet  min =   57.33  max =   57.46  avg =   57.41
     efficientnet_b0  min =   87.52  max =   87.80  avg =   87.62
   efficientnetv2_b0  min =  123.83  max =  124.67  avg =  124.34

proxylessnasnet  min =   57.33  max =   57.46  avg =   57.41
        regnety_400m  min =   65.52  max =   65.81  avg =   65.64
           blazeface  min =   14.56  max =   14.73  avg =   14.62
           googlenet  min =  138.52  max =  139.39  avg =  138.89
            resnet18  min =  124.45  max =  124.81  avg =  124.58
             alexnet  min =  130.46  max =  130.68  avg =  130.54
```
### HiSilicon Hi3519V101 (Cortex-A17 1.2GHz x 1)
```
root@Hi3519:/ncnn-benchmark # taskset 2 ./benchncnn 8 1 0
      squeezenet  min =  272.97  max =  275.84  avg =  274.85
 squeezenet-int8  min =  200.87  max =  202.47  avg =  201.74
       mobilenet  min =  480.90  max =  482.16  avg =  481.64
    mobilenet_v2  min =  350.01  max =  352.39  avg =  350.81
      shufflenet  min =  152.40  max =  153.17  avg =  152.80

mobilenet  min =  480.90  max =  482.16  avg =  481.64
       googlenet  min = 1096.65  max = 1101.35  avg = 1099.21
        resnet18  min =  983.92  max =  987.00  avg =  985.25
         alexnet  min = 1140.30  max = 1141.55  avg = 1140.92
  squeezenet-ssd  min =  574.62  max =  580.12  avg =  577.23
   mobilenet-ssd  min =  960.26  max =  969.13  avg =  965.93
  mobilenet-yolo  min = 1867.78  max = 1880.08  avg = 1873.89
```

### iPhone 5S (Apple A7 1.3GHz x 2)
```
iPhone:~ root# ./benchncnn 8 2 0 -1
[0 Apple A7 GPU]  queueC=0[8]  queueT=0[8]  memU=1  memDL=1  memHV=1
[0 Apple A7 GPU]  fp16p=1  fp16s=0  fp16a=0  int8s=0  int8a=0
          squeezenet  min =   49.21  max =   50.40  avg =   49.74
     squeezenet_int8  min =   54.73  max =   57.39  avg =   56.70

           mobilenet  min =   79.03  max =   80.00  avg =   79.44
      mobilenet_int8  min =  109.95  max =  112.69  avg =  111.38
        mobilenet_v2  min =   57.34  max =   57.88  avg =   57.47
        mobilenet_v3  min =   52.66  max =   53.73  avg =   53.12
          shufflenet  min =   32.78  max =   36.12  avg =   35.12
       shufflenet_v2  min =   31.25  max =   32.10  avg =   31.61
             mnasnet  min =   54.58  max =   56.12  avg =   55.44
     proxylessnasnet  min =   69.52  max =   72.42  avg =   70.40
           googlenet  min =  192.82  max =  194.20  avg =  193.35
      googlenet_int8  min =  235.43  max =  244.71  avg =  239.64
            resnet18  min =  164.33  max =  167.27  avg =  165.51
       resnet18_int8  min =  176.16  max =  179.73  avg =  178.60

googlenet_int8  min =  235.43  max =  244.71  avg =  239.64
             alexnet  min =  224.50  max =  228.21  avg =  226.51
               vgg16  min = 4262.28  max = 4400.29  avg = 4300.34
          vgg16_int8  min = 2835.84  max = 2955.22  avg = 2890.26
            resnet50  min =  542.66  max = 1344.49  avg =  737.05
       resnet50_int8  min =  426.08  max =  435.34  avg =  431.87
      squeezenet_ssd  min =  129.03  max =  131.44  avg =  129.99
 squeezenet_ssd_int8  min =  155.52  max =  161.42  avg =  158.51
       mobilenet_ssd  min =  168.18  max =  170.17  avg =  169.42
  mobilenet_ssd_int8  min =  205.78  max =  212.07  avg =  209.66
      mobilenet_yolo  min =  347.32  max =  363.15  avg =  355.72
  mobilenetv2_yolov3  min =  193.11  max =  196.64  avg =  194.31

iPhone:~ root# ./benchncnn 4 1 0 -1

mobilenet_yolo  min =  347.32  max =  363.15  avg =  355.72

          squeezenet  min =   86.36  max =   86.81  avg =   86.57
     squeezenet_int8  min =   99.62  max =  100.07  avg =   99.83
           mobilenet  min =  143.11  max =  146.50  avg =  145.38
      mobilenet_int8  min =  202.25  max =  203.32  avg =  203.02
        mobilenet_v2  min =   97.56  max =   98.55  avg =   98.09
        mobilenet_v3  min =   87.45  max =   87.68  avg =   87.52
          shufflenet  min =   54.01  max =   54.13  avg =   54.08
       shufflenet_v2  min =   48.11  max =   48.65  avg =   48.36
             mnasnet  min =   95.02  max =   95.77  avg =   95.25

shufflenet  min =   54.01  max =   54.13  avg =   54.08
     proxylessnasnet  min =  123.91  max =  124.61  avg =  124.18
           googlenet  min =  344.23  max =  348.95  avg =  345.97
      googlenet_int8  min =  420.30  max =  420.99  avg =  420.65
            resnet18  min =  300.44  max =  301.36  avg =  300.99
       resnet18_int8  min =  308.60  max =  310.52  avg =  309.70
             alexnet  min =  423.92  max =  429.84  avg =  427.24
               vgg16  min = 4787.59  max = 5015.23  avg = 4900.43
          vgg16_int8  min = 3560.59  max = 3722.75  avg = 3639.88
            resnet50  min =  797.88  max = 1294.57  avg =  985.63
       resnet50_int8  min =  751.15  max =  760.25  avg =  757.89
      squeezenet_ssd  min =  193.75  max =  196.13  avg =  195.29
 squeezenet_ssd_int8  min =  243.78  max =  245.19  avg =  244.74

resnet50_int8  min =  751.15  max =  760.25  avg =  757.89
       mobilenet_ssd  min =  299.69  max =  307.22  avg =  305.12
  mobilenet_ssd_int8  min =  385.91  max =  389.82  avg =  388.48
      mobilenet_yolo  min =  657.00  max =  659.31  avg =  658.08
  mobilenetv2_yolov3  min =  335.59  max =  342.22  avg =  339.37

iPhone:~ root# ./benchncnn 4 1 0 0
          squeezenet  min =  260.18  max =  262.55  avg =  261.09
           mobilenet  min =  288.73  max =  291.83  avg =  289.67
        mobilenet_v2  min =  265.72  max =  267.05  avg =  266.14
        mobilenet_v3  min =  255.86  max =  257.35  avg =  256.43

mobilenet  min =  288.73  max =  291.83  avg =  289.67
          shufflenet  min =  236.66  max =  239.49  avg =  237.98
       shufflenet_v2  min =  244.92  max =  247.75  avg =  246.22
             mnasnet  min =  254.75  max =  256.48  avg =  255.85
     proxylessnasnet  min =  281.42  max =  282.62  avg =  282.11
           googlenet  min =  745.36  max =  764.91  avg =  754.16
            resnet18  min =  721.26  max =  741.98  avg =  734.78
             alexnet  min =  521.43  max =  530.95  avg =  527.01
            resnet50  min = 1494.86  max = 1505.79  avg = 1501.49
      squeezenet_ssd  min = 1096.45  max = 1102.84  avg = 1098.55
       mobilenet_ssd  min =  639.50  max =  641.81  avg =  640.83
      mobilenet_yolo  min = 1445.16  max = 1450.94  avg = 1447.42
  mobilenetv2_yolov3  min = 1047.24  max = 1060.97  avg = 1052.86
```

mobilenet_yolo  min = 1445.16  max = 1450.94  avg = 1447.42
```

### Freescale i.MX7 Dual (Cortex A7 1.0GHz x 2)
```
imx7d_pico:/data/local/tmp $ ./benchncnn 8 2 0 -1 1
          squeezenet  min =  220.10  max =  226.46  avg =  222.89
     squeezenet_int8  min =  159.26  max =  165.25  avg =  161.71
           mobilenet  min =  366.92  max =  373.78  avg =  371.55
      mobilenet_int8  min =  223.14  max =  229.66  avg =  225.66
        mobilenet_v2  min =  252.32  max =  259.41  avg =  255.54
        mobilenet_v3  min =  214.05  max =  222.24  avg =  217.53
          shufflenet  min =  137.02  max =  144.79  avg =  138.85
       shufflenet_v2  min =  134.89  max =  140.75  avg =  137.18
             mnasnet  min =  250.64  max =  256.75  avg =  253.33
     proxylessnasnet  min =  285.35  max =  291.43  avg =  288.37

shufflenet_v2  min =  134.89  max =  140.75  avg =  137.18
     efficientnet_b0  min =  430.47  max =  436.63  avg =  434.75
        regnety_400m  min =  317.69  max =  325.77  avg =  321.24
           blazeface  min =   42.93  max =   43.30  avg =   43.14
           googlenet  min =  721.84  max =  728.40  avg =  724.23
      googlenet_int8  min =  504.07  max =  511.06  avg =  507.39
            resnet18  min =  645.61  max =  653.08  avg =  648.51
       resnet18_int8  min =  370.84  max =  514.38  avg =  392.80
             alexnet  min =  783.64  max =  794.83  avg =  786.95
      squeezenet_ssd  min =  508.71  max =  513.70  avg =  511.29
 squeezenet_ssd_int8  min =  402.85  max =  409.32  avg =  406.45
       mobilenet_ssd  min =  763.70  max =  771.52  avg =  767.61
  mobilenet_ssd_int8  min =  457.99  max =  460.85  avg =  459.76

squeezenet_ssd_int8  min =  402.85  max =  409.32  avg =  406.45
      mobilenet_yolo  min = 1730.90  max = 1746.52  avg = 1741.26
  mobilenetv2_yolov3  min =  884.00  max =  892.97  avg =  889.38
         yolov4-tiny  min = 1181.20  max = 1218.20  avg = 1202.28
           nanodet_m  min =  331.53  max =  339.89  avg =  334.62

imx7d_pico:/data/local/tmp $ ./benchncnn 4 1 0 -1 1
          squeezenet  min =  408.39  max =  410.27  avg =  408.95
     squeezenet_int8  min =  290.25  max =  290.95  avg =  290.61
           mobilenet  min =  707.10  max =  711.64  avg =  708.47
      mobilenet_int8  min =  434.95  max =  436.16  avg =  435.66
        mobilenet_v2  min =  466.52  max =  467.41  avg =  466.96
        mobilenet_v3  min =  407.03  max =  408.29  avg =  407.56

mobilenet_int8  min =  434.95  max =  436.16  avg =  435.66
          shufflenet  min =  240.65  max =  241.07  avg =  240.85
       shufflenet_v2  min =  229.27  max =  235.66  avg =  231.51
             mnasnet  min =  471.21  max =  471.48  avg =  471.35
     proxylessnasnet  min =  544.74  max =  547.62  avg =  546.20
     efficientnet_b0  min =  824.09  max =  824.44  avg =  824.20
        regnety_400m  min =  570.20  max =  571.73  avg =  570.82
           blazeface  min =   76.46  max =   77.05  avg =   76.81
           googlenet  min = 1368.82  max = 1369.99  avg = 1369.33
      googlenet_int8  min =  945.51  max =  946.61  avg =  945.91
            resnet18  min = 1237.79  max = 1257.12  avg = 1246.80
       resnet18_int8  min =  705.09  max =  706.72  avg =  705.63
             alexnet  min = 1516.35  max = 1522.82  avg = 1519.52

resnet18  min = 1237.79  max = 1257.12  avg = 1246.80
      squeezenet_ssd  min =  906.97  max =  908.48  avg =  907.68
 squeezenet_ssd_int8  min =  727.15  max =  728.16  avg =  727.77
       mobilenet_ssd  min = 1475.19  max = 1478.52  avg = 1476.81
  mobilenet_ssd_int8  min =  883.88  max =  890.68  avg =  885.90
      mobilenet_yolo  min = 3408.43  max = 3418.63  avg = 3412.52
  mobilenetv2_yolov3  min = 1685.18  max = 1695.89  avg = 1689.23
         yolov4-tiny  min = 2168.24  max = 2183.24  avg = 2175.93
           nanodet_m  min =  561.56  max =  562.05  avg =  561.72
```

### Z7-Lite 7020 XC7Z020CLG400-2 (Cortex-A9 766MHz x 2)
```
root@petalinux_hdmi:~# LD_LIBRARY_PATH=. ./benchncnn 8 2 0 -1 1
          squeezenet  min =  389.18  max =  390.13  avg =  389.60

     squeezenet_int8  min =  254.33  max =  255.24  avg =  254.85
           mobilenet  min =  623.71  max =  625.01  avg =  624.46
      mobilenet_int8  min =  240.40  max =  241.03  avg =  240.87
        mobilenet_v2  min =  450.00  max =  450.89  avg =  450.40
        mobilenet_v3  min =  362.99  max =  363.66  avg =  363.28
          shufflenet  min =  212.20  max =  213.28  avg =  212.84
       shufflenet_v2  min =  210.26  max =  212.64  avg =  211.53
             mnasnet  min =  408.67  max =  409.64  avg =  409.17
     proxylessnasnet  min =  449.86  max =  450.94  avg =  450.45
     efficientnet_b0  min =  737.40  max =  739.58  avg =  738.32
   efficientnetv2_b0  min =  848.58  max =  849.74  avg =  849.24
        regnety_400m  min =  501.32  max =  503.02  avg =  501.87

efficientnet_b0  min =  737.40  max =  739.58  avg =  738.32
           blazeface  min =   70.89  max =   72.22  avg =   71.61
      squeezenet_ssd  min =  978.55  max =  979.86  avg =  979.22
 squeezenet_ssd_int8  min =  691.90  max =  694.18  avg =  692.73
       mobilenet_ssd  min = 1353.12  max = 1354.13  avg = 1353.53
  mobilenet_ssd_int8  min =  496.26  max =  497.29  avg =  496.61
           nanodet_m  min =  542.04  max =  546.29  avg =  544.73
    yolo-fastest-1.1  min =  282.75  max =  286.11  avg =  284.24
      yolo-fastestv2  min =  230.91  max =  232.74  avg =  231.56

root@petalinux_hdmi:~# LD_LIBRARY_PATH=. ./benchncnn 4 1 0 -1 1
          squeezenet  min =  637.19  max =  639.33  avg =  637.82
     squeezenet_int8  min =  390.31  max =  391.63  avg =  390.94

           mobilenet  min = 1085.54  max = 1085.96  avg = 1085.71
      mobilenet_int8  min =  437.28  max =  437.65  avg =  437.44
        mobilenet_v2  min =  716.03  max =  716.75  avg =  716.35
        mobilenet_v3  min =  587.83  max =  588.55  avg =  588.21
          shufflenet  min =  331.28  max =  331.97  avg =  331.63
       shufflenet_v2  min =  331.03  max =  333.19  avg =  331.76
             mnasnet  min =  682.68  max =  683.11  avg =  682.82
     proxylessnasnet  min =  763.89  max =  764.80  avg =  764.35
     efficientnet_b0  min = 1288.61  max = 1289.10  avg = 1288.81
   efficientnetv2_b0  min = 1499.12  max = 1500.11  avg = 1499.65
        regnety_400m  min =  852.03  max =  853.16  avg =  852.68
           blazeface  min =  109.40  max =  111.51  avg =  110.41

efficientnetv2_b0  min = 1499.12  max = 1500.11  avg = 1499.65
      squeezenet_ssd  min = 1493.25  max = 1497.00  avg = 1494.87
 squeezenet_ssd_int8  min = 1016.77  max = 1019.31  avg = 1017.99
       mobilenet_ssd  min = 2379.20  max = 2379.83  avg = 2379.64
  mobilenet_ssd_int8  min =  881.70  max =  881.89  avg =  881.83
           nanodet_m  min =  831.13  max =  832.58  avg =  831.87
    yolo-fastest-1.1  min =  466.80  max =  469.90  avg =  468.79
      yolo-fastestv2  min =  352.07  max =  355.20  avg =  353.36
```

### Loongson 2K1000 (GS264 1.0GHz x 2)
```
root@ls2k:~/ncnn/build/benchmark# ./benchncnn 10 2 0 -1 1
          squeezenet  min =  184.33  max =  184.94  avg =  184.65
     squeezenet_int8  min =  201.42  max =  201.99  avg =  201.72

           mobilenet  min =  277.17  max =  278.04  avg =  277.66
      mobilenet_int8  min =  234.61  max =  235.17  avg =  234.81
        mobilenet_v2  min =  223.10  max =  274.92  avg =  228.71
        mobilenet_v3  min =  185.79  max =  201.76  avg =  187.60
          shufflenet  min =  129.78  max =  131.09  avg =  130.28
       shufflenet_v2  min =  115.86  max =  116.77  avg =  116.42
             mnasnet  min =  213.92  max =  214.72  avg =  214.26
     proxylessnasnet  min =  240.05  max =  242.02  avg =  240.86
     efficientnet_b0  min =  347.52  max =  348.53  avg =  348.13
   efficientnetv2_b0  min =  382.78  max =  479.58  avg =  398.18
        regnety_400m  min =  270.00  max =  312.84  avg =  274.66
           blazeface  min =   37.60  max =   38.02  avg =   37.79

efficientnetv2_b0  min =  382.78  max =  479.58  avg =  398.18
           googlenet  min =  659.55  max =  693.17  avg =  666.17
      googlenet_int8  min =  678.26  max =  718.39  avg =  682.79
            resnet18  min =  499.75  max =  766.88  avg =  532.49
       resnet18_int8  min =  500.38  max =  533.97  avg =  504.56
             alexnet  min =  508.49  max =  542.94  avg =  516.13
               vgg16  min = 2654.06  max = 3082.44  avg = 2762.51
          vgg16_int8  min = 2628.96  max = 2665.35  avg = 2647.12
            resnet50  min = 1256.97  max = 1417.45  avg = 1283.04
       resnet50_int8  min = 1232.55  max = 1276.94  avg = 1244.59
      squeezenet_ssd  min =  538.83  max =  588.03  avg =  553.44
 squeezenet_ssd_int8  min =  501.67  max =  532.61  avg =  505.72
       mobilenet_ssd  min =  571.14  max =  600.93  avg =  578.22

squeezenet_ssd  min =  538.83  max =  588.03  avg =  553.44
  mobilenet_ssd_int8  min =  478.67  max =  515.39  avg =  483.06
      mobilenet_yolo  min = 1644.48  max = 1729.17  avg = 1669.18
  mobilenetv2_yolov3  min =  752.22  max =  792.40  avg =  760.10
         yolov4-tiny  min =  994.48  max = 1096.10  avg = 1016.49
           nanodet_m  min =  299.12  max =  343.99  avg =  303.98
    yolo-fastest-1.1  min =  141.56  max =  142.93  avg =  142.04
      yolo-fastestv2  min =  125.66  max =  168.88  avg =  130.28

root@ls2k:~/ncnn/build/benchmark# ./benchncnn 4 1 0 -1 1
          squeezenet  min =  295.48  max =  296.42  avg =  295.98
     squeezenet_int8  min =  334.05  max =  336.31  avg =  335.35
           mobilenet  min =  476.33  max =  479.00  avg =  477.41

squeezenet  min =  295.48  max =  296.42  avg =  295.98
      mobilenet_int8  min =  446.03  max =  448.21  avg =  446.73
        mobilenet_v2  min =  343.26  max =  343.97  avg =  343.69
        mobilenet_v3  min =  296.84  max =  297.31  avg =  297.11
          shufflenet  min =  202.31  max =  203.96  avg =  202.79
       shufflenet_v2  min =  181.69  max =  182.42  avg =  182.08
             mnasnet  min =  353.73  max =  354.12  avg =  353.99
     proxylessnasnet  min =  404.49  max =  405.00  avg =  404.75
     efficientnet_b0  min =  592.54  max =  593.81  avg =  593.14
   efficientnetv2_b0  min =  649.91  max =  651.49  avg =  650.54
        regnety_400m  min =  425.96  max =  426.33  avg =  426.12
           blazeface  min =   59.74  max =   60.19  avg =   59.90
           googlenet  min = 1120.13  max = 1217.54  avg = 1146.27

regnety_400m  min =  425.96  max =  426.33  avg =  426.12
      googlenet_int8  min = 1205.17  max = 1213.43  avg = 1208.13
            resnet18  min =  803.07  max =  997.37  avg =  856.09
       resnet18_int8  min =  911.74  max =  916.16  avg =  913.31
             alexnet  min =  883.47  max =  903.08  avg =  889.06
               vgg16  min = 4425.52  max = 4587.36  avg = 4467.61
          vgg16_int8  min = 4896.90  max = 4993.15  avg = 4924.44
            resnet50  min = 2163.22  max = 2169.90  avg = 2167.49
       resnet50_int8  min = 2202.87  max = 2218.00  avg = 2210.51
      squeezenet_ssd  min =  831.06  max =  926.94  avg =  856.24
 squeezenet_ssd_int8  min =  800.52  max =  803.28  avg =  801.72
       mobilenet_ssd  min =  979.74  max =  980.82  avg =  980.22
  mobilenet_ssd_int8  min =  893.79  max =  895.41  avg =  894.51

squeezenet_ssd_int8  min =  800.52  max =  803.28  avg =  801.72
      mobilenet_yolo  min = 2578.17  max = 2586.30  avg = 2582.55
  mobilenetv2_yolov3  min = 1190.77  max = 1207.67  avg = 1196.06
         yolov4-tiny  min = 1558.29  max = 1570.18  avg = 1561.52
           nanodet_m  min =  442.90  max =  444.27  avg =  443.72
    yolo-fastest-1.1  min =  203.60  max =  208.43  avg =  205.20
      yolo-fastestv2  min =  184.61  max =  185.05  avg =  184.75
```

### Loongson 2K1000LA (LA264 1.0GHz * 2)
```
root@ls2kla:~/ncnn/build/benchmark# ./benchncnn 10 2 0 -1 1
          squeezenet  min =  151.11  max =  162.36  avg =  153.30
     squeezenet_int8  min =  195.32  max =  198.63  avg =  196.12
           mobilenet  min =  279.27  max =  283.42  avg =  280.40

squeezenet  min =  151.11  max =  162.36  avg =  153.30
      mobilenet_int8  min =  264.78  max =  268.41  avg =  265.76
        mobilenet_v2  min =  204.39  max =  207.69  avg =  205.77
        mobilenet_v3  min =  171.32  max =  187.07  avg =  173.15
          shufflenet  min =  147.43  max =  150.72  avg =  147.89
       shufflenet_v2  min =  169.42  max =  172.58  avg =  170.35
             mnasnet  min =  204.87  max =  208.01  avg =  205.63
     proxylessnasnet  min =  226.79  max =  237.74  avg =  229.02
     efficientnet_b0  min =  302.30  max =  310.91  avg =  303.87
   efficientnetv2_b0  min =  327.65  max =  361.15  avg =  334.45
        regnety_400m  min =  264.08  max =  278.49  avg =  266.35
           blazeface  min =   31.80  max =   39.18  avg =   32.88
           googlenet  min =  562.95  max =  578.42  avg =  566.28

regnety_400m  min =  264.08  max =  278.49  avg =  266.35
      googlenet_int8  min =  598.16  max =  613.56  avg =  601.68
            resnet18  min =  466.73  max =  472.08  avg =  469.58
       resnet18_int8  min =  489.69  max =  493.74  avg =  491.63
             alexnet  min =  381.35  max =  388.12  avg =  384.78
               vgg16  min = 2321.29  max = 2345.89  avg = 2330.29
          vgg16_int8  min = 2562.86  max = 2568.06  avg = 2565.68
            resnet50  min = 1219.09  max = 1225.67  avg = 1221.36
       resnet50_int8  min = 1263.44  max = 1266.74  avg = 1265.09
      squeezenet_ssd  min =  433.23  max =  441.06  avg =  437.07
 squeezenet_ssd_int8  min =  438.69  max =  443.17  avg =  440.81
       mobilenet_ssd  min =  587.37  max =  598.57  avg =  589.99
  mobilenet_ssd_int8  min =  539.62  max =  552.57  avg =  542.87

squeezenet_ssd_int8  min =  438.69  max =  443.17  avg =  440.81
      mobilenet_yolo  min = 1485.30  max = 1491.17  avg = 1487.81
  mobilenetv2_yolov3  min =  711.57  max =  722.91  avg =  715.07
         yolov4-tiny  min =  954.76  max =  961.66  avg =  957.28
           nanodet_m  min =  364.22  max =  369.32  avg =  365.94
    yolo-fastest-1.1  min =  154.81  max =  160.45  avg =  156.23
      yolo-fastestv2  min =  157.39  max =  168.82  avg =  159.51
  vision_transformer  min = 18926.46  max = 18980.43  avg = 18951.29
          FastestDet  min =  168.81  max =  176.77  avg =  170.26

root@ls2kla:~/ncnn/build/benchmark# ./benchncnn 4 1 0 -1 1
          squeezenet  min =  272.76  max =  280.89  avg =  275.29
     squeezenet_int8  min =  352.02  max =  353.25  avg =  352.40

           mobilenet  min =  519.09  max =  519.68  avg =  519.34
      mobilenet_int8  min =  509.85  max =  510.23  avg =  510.04
        mobilenet_v2  min =  352.06  max =  352.74  avg =  352.37
        mobilenet_v3  min =  295.13  max =  295.70  avg =  295.39
          shufflenet  min =  241.58  max =  241.94  avg =  241.73
       shufflenet_v2  min =  282.88  max =  283.39  avg =  283.18
             mnasnet  min =  357.74  max =  358.21  avg =  357.98
     proxylessnasnet  min =  403.26  max =  411.69  avg =  406.02
     efficientnet_b0  min =  546.11  max =  546.88  avg =  546.53
   efficientnetv2_b0  min =  596.83  max =  597.05  avg =  596.93
        regnety_400m  min =  441.94  max =  442.02  avg =  441.98
           blazeface  min =   54.08  max =   54.59  avg =   54.38

efficientnetv2_b0  min =  596.83  max =  597.05  avg =  596.93
           googlenet  min = 1042.19  max = 1048.03  avg = 1044.40
      googlenet_int8  min = 1118.22  max = 1121.18  avg = 1119.79
            resnet18  min =  838.79  max =  839.81  avg =  839.43
       resnet18_int8  min =  939.62  max =  940.72  avg =  940.23
             alexnet  min =  729.36  max =  740.65  avg =  734.19
               vgg16  min = 4326.68  max = 4335.10  avg = 4330.97
          vgg16_int8  min = 4896.71  max = 4909.63  avg = 4905.14
            resnet50  min = 2277.36  max = 2280.34  avg = 2279.14
       resnet50_int8  min = 2399.07  max = 2402.21  avg = 2400.78
      squeezenet_ssd  min =  751.49  max =  753.79  avg =  752.20
 squeezenet_ssd_int8  min =  771.01  max =  774.08  avg =  771.91
       mobilenet_ssd  min = 1063.41  max = 1065.65  avg = 1064.16

squeezenet_ssd  min =  751.49  max =  753.79  avg =  752.20
  mobilenet_ssd_int8  min = 1031.59  max = 1033.03  avg = 1032.09
      mobilenet_yolo  min = 2585.33  max = 2586.65  avg = 2586.11
  mobilenetv2_yolov3  min = 1246.35  max = 1248.43  avg = 1247.32
         yolov4-tiny  min = 1639.13  max = 1642.47  avg = 1640.87
           nanodet_m  min =  606.40  max =  607.14  avg =  606.86
    yolo-fastest-1.1  min =  242.15  max =  244.64  avg =  243.43
      yolo-fastestv2  min =  246.92  max =  247.84  avg =  247.27
  vision_transformer  min = 36607.51  max = 36870.44  avg = 36724.88
          FastestDet  min =  266.96  max =  268.86  avg =  267.94
```

### Loongson 3A3000 (GS464E 1.45GHz * 4)
```
root@3A3K:~/Desktop/ncnn-20221128/build/benchmark$ ./benchncnn 5 4 2 -1 0
loop_count = 5

```
          squeezenet  min =   88.82  max =  116.74  avg =   94.92
     squeezenet_int8  min =  140.62  max =  162.48  avg =  146.32
           mobilenet  min =  144.80  max =  244.58  avg =  172.14
      mobilenet_int8  min =  265.21  max =  293.89  avg =  281.80
        mobilenet_v2  min =  109.80  max =  156.74  avg =  120.48
        mobilenet_v3  min =   90.18  max =   93.25  avg =   91.50
          shufflenet  min =   56.64  max =  216.12  avg =  100.68
       shufflenet_v2  min =   45.70  max =  142.00  avg =   65.20
             mnasnet  min =  106.99  max =  229.11  avg =  134.22
     proxylessnasnet  min =  123.68  max =  261.01  avg =  155.97
     efficientnet_b0  min =  160.98  max =  191.14  avg =  171.55
   efficientnetv2_b0  min =  162.75  max =  187.67  avg =  176.19

proxylessnasnet  min =  123.68  max =  261.01  avg =  155.97
        regnety_400m  min =  135.06  max =  174.12  avg =  151.30
           blazeface  min =   15.26  max =   43.81  avg =   23.91
           googlenet  min =  327.16  max =  386.02  avg =  350.25
      googlenet_int8  min =  500.45  max =  637.39  avg =  540.62
            resnet18  min =  254.45  max =  421.56  avg =  304.48
       resnet18_int8  min =  385.14  max =  559.01  avg =  439.74
             alexnet  min =  179.19  max =  220.91  avg =  190.63
               vgg16  min = 1563.99  max = 1645.01  avg = 1619.63
          vgg16_int8  min = 1436.00  max = 1530.45  avg = 1473.00
            resnet50  min =  702.35  max =  833.23  avg =  764.14
       resnet50_int8  min = 1099.40  max = 1208.84  avg = 1154.51
      squeezenet_ssd  min =  191.40  max =  270.10  avg =  218.75

resnet50  min =  702.35  max =  833.23  avg =  764.14
 squeezenet_ssd_int8  min =  304.51  max =  387.51  avg =  344.98
       mobilenet_ssd  min =  315.77  max =  417.37  avg =  344.40
  mobilenet_ssd_int8  min =  554.28  max =  656.07  avg =  580.72
      mobilenet_yolo  min =  806.48  max =  851.22  avg =  825.50
  mobilenetv2_yolov3  min =  382.38  max =  503.38  avg =  421.03
         yolov4-tiny  min =  502.87  max =  620.30  avg =  550.08
           nanodet_m  min =  126.00  max =  314.03  avg =  184.93
    yolo-fastest-1.1  min =   64.68  max =  189.47  avg =  110.89
      yolo-fastestv2  min =   69.03  max =  116.31  avg =   82.36
  vision_transformer  min = 14737.56  max = 15012.35  avg = 14890.56
          FastestDet  min =   84.30  max =  139.87  avg =  102.23
```

### Loongson 3A4000 (GS464V 1.8GHz * 4 with MSA128)
```

vision_transformer  min = 14737.56  max = 15012.35  avg = 14890.56
```

```
root@3A4K:~/Desktop/ncnn-20221128/build/benchmark$ ./benchncnn 10 4 2 -1 0
          squeezenet  min =   17.04  max =   39.86  avg =   20.39
     squeezenet_int8  min =   21.77  max =   25.93  avg =   23.02
           mobilenet  min =   26.34  max =   97.11  avg =   38.24
      mobilenet_int8  min =   32.93  max =   33.31  avg =   33.07
        mobilenet_v2  min =   19.40  max =   19.91  avg =   19.63
        mobilenet_v3  min =   16.48  max =   45.31  avg =   19.68
          shufflenet  min =   12.23  max =  116.79  avg =   22.86
       shufflenet_v2  min =   11.14  max =   11.59  avg =   11.37
             mnasnet  min =   18.33  max =   51.66  avg =   24.52

shufflenet  min =   12.23  max =  116.79  avg =   22.86
     proxylessnasnet  min =   22.03  max =   22.46  avg =   22.19
     efficientnet_b0  min =   34.94  max =  129.52  avg =   45.76
   efficientnetv2_b0  min =   38.58  max =   67.86  avg =   41.84
        regnety_400m  min =   35.53  max =   38.59  avg =   36.14
           blazeface  min =    4.08  max =    4.34  avg =    4.17
           googlenet  min =   72.60  max =  100.31  avg =   76.25
      googlenet_int8  min =   82.09  max =  107.09  avg =   86.78
            resnet18  min =   53.99  max =  100.21  avg =   63.52
       resnet18_int8  min =   57.20  max =   77.00  avg =   60.47
             alexnet  min =   61.95  max =   80.86  avg =   65.01
               vgg16  min =  329.58  max =  438.99  avg =  360.40
          vgg16_int8  min =  293.27  max =  366.16  avg =  311.23

alexnet  min =   61.95  max =   80.86  avg =   65.01
            resnet50  min =  138.06  max =  260.50  avg =  169.27
       resnet50_int8  min =  154.06  max =  244.31  avg =  173.37
      squeezenet_ssd  min =   60.44  max =   97.92  avg =   65.41
 squeezenet_ssd_int8  min =   55.34  max =  136.72  avg =   68.15
       mobilenet_ssd  min =   57.97  max =  139.16  avg =   69.27
  mobilenet_ssd_int8  min =   66.66  max =   89.91  avg =   71.00
      mobilenet_yolo  min =  169.38  max =  711.10  avg =  242.62
  mobilenetv2_yolov3  min =   75.61  max =   97.83  avg =   80.23
         yolov4-tiny  min =  110.52  max =  143.67  avg =  118.53
           nanodet_m  min =   24.04  max =   92.81  avg =   32.45
    yolo-fastest-1.1  min =   10.97  max =   32.77  avg =   15.05
      yolo-fastestv2  min =   11.54  max =   12.09  avg =   11.84

nanodet_m  min =   24.04  max =   92.81  avg =   32.45
  vision_transformer  min = 4193.41  max = 4274.03  avg = 4213.64
          FastestDet  min =   12.54  max =   13.01  avg =   12.78
```

Test on UOS V20 E1050

```
uos@uos-PC:~/ncnn/benchmark$ ./benchncnn 10 4 2 -1 0
          squeezenet  min =   25.28  max =   38.19  avg =   27.81
     squeezenet_int8  min =   21.61  max =   22.13  avg =   21.85
           mobilenet  min =   44.77  max =   69.54  avg =   55.37
      mobilenet_int8  min =   32.96  max =   44.00  avg =   36.08
        mobilenet_v2  min =   29.21  max =   52.70  avg =   35.47
        mobilenet_v3  min =   24.62  max =   27.32  avg =   25.18

mobilenet_int8  min =   32.96  max =   44.00  avg =   36.08
          shufflenet  min =   18.90  max =   49.70  avg =   22.95
       shufflenet_v2  min =   15.87  max =   22.38  avg =   17.67
             mnasnet  min =   29.08  max =   69.37  avg =   35.53
     proxylessnasnet  min =   33.30  max =   94.15  avg =   42.81
     efficientnet_b0  min =   49.34  max =   61.22  avg =   52.01
   efficientnetv2_b0  min =   57.89  max =   72.55  avg =   60.72
        regnety_400m  min =   50.65  max =   74.16  avg =   57.56
           blazeface  min =    4.97  max =    5.33  avg =    5.11
           googlenet  min =  101.45  max =  119.73  avg =  106.85
      googlenet_int8  min =   83.94  max =   99.75  avg =   87.36
            resnet18  min =   81.65  max =   99.76  avg =   85.96
       resnet18_int8  min =   58.60  max =   75.88  avg =   60.62

googlenet_int8  min =   83.94  max =   99.75  avg =   87.36
             alexnet  min =   77.05  max =  208.05  avg =  120.39
               vgg16  min =  427.51  max =  676.57  avg =  531.53
          vgg16_int8  min =  326.59  max =  487.96  avg =  417.74
            resnet50  min =  221.51  max =  580.11  avg =  305.64
       resnet50_int8  min =  158.00  max =  190.71  avg =  167.50
      squeezenet_ssd  min =   98.87  max =  135.55  avg =  115.54
 squeezenet_ssd_int8  min =   66.33  max =  361.40  avg =  148.19
       mobilenet_ssd  min =   94.12  max =  340.16  avg =  184.85
  mobilenet_ssd_int8  min =   88.26  max =  150.47  avg =  112.35
      mobilenet_yolo  min =  252.07  max =  510.61  avg =  327.21
  mobilenetv2_yolov3  min =  115.31  max =  183.63  avg =  147.28
         yolov4-tiny  min =  153.92  max =  259.18  avg =  196.70

mobilenet_yolo  min =  252.07  max =  510.61  avg =  327.21
           nanodet_m  min =   34.95  max =   66.15  avg =   46.41
    yolo-fastest-1.1  min =   15.34  max =   15.94  avg =   15.62
      yolo-fastestv2  min =   15.53  max =   16.06  avg =   15.80
  vision_transformer  min = 4200.48  max = 5853.43  avg = 4555.42
          FastestDet  min =   16.73  max =   18.72  avg =   17.08

uos@uos-PC:~/ncnn/benchmark$ ./benchncnn 10 4 1 -1 0
          squeezenet  min =   25.93  max =   47.61  avg =   28.45
     squeezenet_int8  min =   21.84  max =   27.09  avg =   22.84
           mobilenet  min =   44.61  max =   83.44  avg =   52.52
      mobilenet_int8  min =   32.91  max =   45.99  avg =   34.52
        mobilenet_v2  min =   29.44  max =   37.14  avg =   30.43

mobilenet  min =   44.61  max =   83.44  avg =   52.52
        mobilenet_v3  min =   24.54  max =   42.68  avg =   27.25
          shufflenet  min =   17.16  max =   42.10  avg =   20.08
       shufflenet_v2  min =   15.99  max =   16.43  avg =   16.29
             mnasnet  min =   29.14  max =   43.37  avg =   30.79
     proxylessnasnet  min =   33.15  max =   34.12  avg =   33.52
     efficientnet_b0  min =   49.35  max =   87.75  avg =   54.03
   efficientnetv2_b0  min =   57.69  max =   84.67  avg =   64.12
        regnety_400m  min =   50.55  max =   75.35  avg =   55.31
           blazeface  min =    5.01  max =    5.16  avg =    5.05
           googlenet  min =  101.51  max =  116.33  avg =  105.38
      googlenet_int8  min =   84.34  max =  102.58  avg =   89.89
            resnet18  min =   80.58  max =   94.47  avg =   86.27

googlenet  min =  101.51  max =  116.33  avg =  105.38
       resnet18_int8  min =   59.00  max =   76.66  avg =   62.15
             alexnet  min =   91.72  max =  117.98  avg =  102.20
               vgg16  min =  435.57  max =  453.90  avg =  441.39
          vgg16_int8  min =  308.39  max =  332.69  avg =  321.09
            resnet50  min =  219.93  max =  249.30  avg =  231.93
       resnet50_int8  min =  156.78  max =  179.34  avg =  163.43
      squeezenet_ssd  min =  109.48  max =  153.84  avg =  123.75
 squeezenet_ssd_int8  min =   74.33  max =  117.03  avg =   93.81
       mobilenet_ssd  min =   94.91  max =  161.38  avg =  127.78
  mobilenet_ssd_int8  min =   82.35  max =  112.79  avg =   91.86
      mobilenet_yolo  min =  252.05  max =  285.16  avg =  266.33
  mobilenetv2_yolov3  min =  113.98  max =  173.83  avg =  139.60

mobilenet_ssd_int8  min =   82.35  max =  112.79  avg =   91.86
         yolov4-tiny  min =  150.06  max =  210.96  avg =  164.94
           nanodet_m  min =   34.62  max =   67.81  avg =   48.43
    yolo-fastest-1.1  min =   15.78  max =   16.09  avg =   15.93
      yolo-fastestv2  min =   15.54  max =   32.82  avg =   17.62
  vision_transformer  min = 4202.89  max = 5573.15  avg = 4426.38
          FastestDet  min =   16.39  max =   17.06  avg =   16.75

uos@uos-PC:~/ncnn/benchmark$ ./benchncnn 10 4 0 -1 0
          squeezenet  min =   25.98  max =   36.75  avg =   28.86
     squeezenet_int8  min =   22.04  max =   30.86  avg =   23.28
           mobilenet  min =   44.82  max =   60.73  avg =   46.72
      mobilenet_int8  min =   33.00  max =   48.45  avg =   34.70

squeezenet_int8  min =   22.04  max =   30.86  avg =   23.28
        mobilenet_v2  min =   29.53  max =   56.78  avg =   33.98
        mobilenet_v3  min =   24.69  max =   45.60  avg =   28.13
          shufflenet  min =   17.25  max =   24.72  avg =   18.18
       shufflenet_v2  min =   16.00  max =   31.27  avg =   17.62
             mnasnet  min =   28.95  max =   44.73  avg =   32.58
     proxylessnasnet  min =   32.99  max =   45.42  avg =   34.66
     efficientnet_b0  min =   49.71  max =   53.47  avg =   50.25
   efficientnetv2_b0  min =   57.51  max =   78.56  avg =   61.47
        regnety_400m  min =   50.18  max =   71.85  avg =   54.77
           blazeface  min =    4.98  max =    9.36  avg =    5.48
           googlenet  min =  101.25  max =  121.71  avg =  105.71
      googlenet_int8  min =   82.97  max =  111.81  avg =   89.49

blazeface  min =    4.98  max =    9.36  avg =    5.48
            resnet18  min =   75.66  max =   87.19  avg =   78.72
       resnet18_int8  min =   58.92  max =  108.67  avg =   76.70
             alexnet  min =   79.12  max =  144.22  avg =  101.91
               vgg16  min =  430.14  max =  460.46  avg =  444.56
          vgg16_int8  min =  308.08  max =  350.15  avg =  324.86
            resnet50  min =  219.60  max =  258.59  avg =  237.46
       resnet50_int8  min =  156.54  max =  180.28  avg =  163.11
      squeezenet_ssd  min =   77.71  max =  137.36  avg =  119.68
 squeezenet_ssd_int8  min =   78.88  max =  113.64  avg =   95.83
       mobilenet_ssd  min =   94.82  max =  156.99  avg =  119.67
  mobilenet_ssd_int8  min =   77.17  max =   98.29  avg =   86.90
      mobilenet_yolo  min =  252.29  max =  295.62  avg =  265.58

mobilenet_ssd  min =   94.82  max =  156.99  avg =  119.67
  mobilenetv2_yolov3  min =  114.28  max =  159.82  avg =  140.03
         yolov4-tiny  min =  150.99  max =  203.07  avg =  165.18
           nanodet_m  min =   34.48  max =   71.56  avg =   49.84
    yolo-fastest-1.1  min =   15.36  max =   30.00  avg =   17.11
      yolo-fastestv2  min =   15.42  max =   26.96  avg =   16.78
  vision_transformer  min = 4187.60  max = 4319.84  avg = 4220.05
          FastestDet  min =   16.30  max =   24.88  avg =   17.49

```

### Loongson 3A5000 (LA464 2.5GHz * 4)
```
root@3A5K:~/Desktop/ncnn-20230223/build/benchmark$ ./benchncnn 10 4 2 -1 0
          squeezenet  min =   11.97  max =   19.38  avg =   13.61
     squeezenet_int8  min =   14.96  max =   15.36  avg =   15.12

           mobilenet  min =   20.14  max =   27.50  avg =   21.12
      mobilenet_int8  min =   25.28  max =   35.06  avg =   27.37
        mobilenet_v2  min =   12.82  max =   13.20  avg =   12.98
        mobilenet_v3  min =   11.39  max =   25.03  avg =   12.86
          shufflenet  min =    7.35  max =    7.50  avg =    7.40
       shufflenet_v2  min =    7.12  max =    7.23  avg =    7.18
             mnasnet  min =   12.85  max =   21.69  avg =   13.83
     proxylessnasnet  min =   15.35  max =   15.79  avg =   15.43
     efficientnet_b0  min =   24.20  max =   24.46  avg =   24.30
   efficientnetv2_b0  min =   26.80  max =   42.43  avg =   29.25
        regnety_400m  min =   22.85  max =   38.30  avg =   24.51
           blazeface  min =    2.57  max =    2.67  avg =    2.60

efficientnetv2_b0  min =   26.80  max =   42.43  avg =   29.25
           googlenet  min =   49.09  max =   85.91  avg =   67.57
      googlenet_int8  min =   64.89  max =   95.28  avg =   76.41
            resnet18  min =   42.43  max =   62.39  avg =   52.38
       resnet18_int8  min =   47.96  max =   68.69  avg =   56.75
             alexnet  min =   46.01  max =   59.26  avg =   49.20
               vgg16  min =  246.82  max =  261.80  avg =  252.81
          vgg16_int8  min =  247.13  max =  256.81  avg =  252.37
            resnet50  min =  102.17  max =  138.16  avg =  117.65
       resnet50_int8  min =  115.09  max =  151.30  avg =  129.13
      squeezenet_ssd  min =   43.62  max =   70.64  avg =   53.89
 squeezenet_ssd_int8  min =   38.66  max =   60.12  avg =   47.66
       mobilenet_ssd  min =   42.67  max =   68.78  avg =   53.95

squeezenet_ssd  min =   43.62  max =   70.64  avg =   53.89
  mobilenet_ssd_int8  min =   56.29  max =   68.31  avg =   59.86
      mobilenet_yolo  min =  129.04  max =  188.26  avg =  149.64
  mobilenetv2_yolov3  min =   61.80  max =   71.41  avg =   66.43
         yolov4-tiny  min =   88.64  max =  108.17  avg =   95.48
           nanodet_m  min =   16.24  max =   16.57  avg =   16.34
    yolo-fastest-1.1  min =    6.98  max =    7.16  avg =    7.05
      yolo-fastestv2  min =    6.95  max =    7.29  avg =    7.08
  vision_transformer  min = 2910.63  max = 3109.29  avg = 2949.04
          FastestDet  min =    7.66  max =    7.90  avg =    7.80
```

### Loongson 3A6000 (LA664 2.5GHz * 4+4)

```
~/ncnn/build/benchmark$ ./benchncnn 10 8 2 -1 0

```

```
          squeezenet  min =    7.12  max =    7.20  avg =    7.16
     squeezenet_int8  min =    8.93  max =    9.20  avg =    8.98
           mobilenet  min =   11.81  max =   11.88  avg =   11.84
      mobilenet_int8  min =   14.25  max =   14.33  avg =   14.28
        mobilenet_v2  min =    8.06  max =    8.16  avg =    8.08
        mobilenet_v3  min =    6.84  max =    6.90  avg =    6.87
          shufflenet  min =    5.38  max =    5.44  avg =    5.39
       shufflenet_v2  min =    5.20  max =    5.22  avg =    5.20
             mnasnet  min =    8.06  max =    8.10  avg =    8.07
     proxylessnasnet  min =    8.94  max =    9.09  avg =    8.99
     efficientnet_b0  min =   13.43  max =   13.65  avg =   13.48
   efficientnetv2_b0  min =   16.06  max =   16.18  avg =   16.11

proxylessnasnet  min =    8.94  max =    9.09  avg =    8.99
        regnety_400m  min =   18.11  max =   18.18  avg =   18.14
           blazeface  min =    1.59  max =    1.61  avg =    1.60
           googlenet  min =   26.08  max =   26.24  avg =   26.17
      googlenet_int8  min =   31.25  max =   31.42  avg =   31.34
            resnet18  min =   19.65  max =   19.73  avg =   19.69
       resnet18_int8  min =   25.55  max =   25.66  avg =   25.60
             alexnet  min =   19.56  max =   19.81  avg =   19.67
               vgg16  min =  115.32  max =  116.38  avg =  115.99
          vgg16_int8  min =  135.94  max =  136.73  avg =  136.34
            resnet50  min =   56.46  max =   56.96  avg =   56.81
       resnet50_int8  min =   66.13  max =   66.40  avg =   66.27
      squeezenet_ssd  min =   22.84  max =   22.99  avg =   22.89

resnet50  min =   56.46  max =   56.96  avg =   56.81
 squeezenet_ssd_int8  min =   22.34  max =   22.76  avg =   22.54
       mobilenet_ssd  min =   24.67  max =   24.75  avg =   24.71
  mobilenet_ssd_int8  min =   29.32  max =   29.37  avg =   29.34
      mobilenet_yolo  min =   82.82  max =   84.02  avg =   83.40
  mobilenetv2_yolov3  min =   30.31  max =   30.45  avg =   30.38
         yolov4-tiny  min =   42.49  max =   42.74  avg =   42.62
           nanodet_m  min =   11.00  max =   11.08  avg =   11.02
    yolo-fastest-1.1  min =    5.28  max =    5.40  avg =    5.31
      yolo-fastestv2  min =    5.09  max =    5.10  avg =    5.10
  vision_transformer  min =  869.40  max =  898.18  avg =  874.07
          FastestDet  min =    5.28  max =    5.37  avg =    5.31
```

### Phytium FT-2000/4 (FTC663 armv8 2.2GHz x 4)
Test on Kylin OS V10

FastestDet  min =    5.28  max =    5.37  avg =    5.31
```

```
mobtgzhang@mobtgzhang-PC:~/ncnn/benchmark$ ./benchncnn 10 1 0 -1 0
          squeezenet  min =   40.92  max =   43.43  avg =   41.34
     squeezenet_int8  min =   35.48  max =   36.07  avg =   35.75
           mobilenet  min =   72.23  max =   72.53  avg =   72.39
      mobilenet_int8  min =   48.10  max =   48.59  avg =   48.31
        mobilenet_v2  min =   47.94  max =   48.45  avg =   48.13
        mobilenet_v3  min =   37.95  max =   39.59  avg =   38.41
          shufflenet  min =   21.51  max =   21.84  avg =   21.64
       shufflenet_v2  min =   21.10  max =   21.45  avg =   21.26
             mnasnet  min =   44.53  max =   45.15  avg =   44.74
     proxylessnasnet  min =   53.02  max =   53.62  avg =   53.21

shufflenet_v2  min =   21.10  max =   21.45  avg =   21.26
     efficientnet_b0  min =   79.81  max =   80.51  avg =   80.15
   efficientnetv2_b0  min =   92.55  max =  103.10  avg =   97.53
        regnety_400m  min =   58.52  max =   70.04  avg =   64.20
           blazeface  min =    6.06  max =    9.85  avg =    6.88
           googlenet  min =  146.49  max =  162.69  avg =  152.98
      googlenet_int8  min =  127.38  max =  132.11  avg =  128.51
            resnet18  min =  107.79  max =  108.83  avg =  108.37
       resnet18_int8  min =   97.28  max =   99.03  avg =   97.73
             alexnet  min =   89.95  max =   91.63  avg =   90.28
               vgg16  min =  642.27  max =  647.16  avg =  644.09
          vgg16_int8  min =  567.03  max =  574.11  avg =  568.74
            resnet50  min =  329.12  max =  331.79  avg =  330.10

vgg16  min =  642.27  max =  647.16  avg =  644.09
       resnet50_int8  min =  252.48  max =  253.65  avg =  252.93
      squeezenet_ssd  min =   96.46  max =   96.95  avg =   96.69
 squeezenet_ssd_int8  min =   92.35  max =   93.24  avg =   92.72
       mobilenet_ssd  min =  149.14  max =  150.56  avg =  149.40
  mobilenet_ssd_int8  min =   97.56  max =   98.03  avg =   97.82
      mobilenet_yolo  min =  339.71  max =  340.60  avg =  339.89
  mobilenetv2_yolov3  min =  174.53  max =  175.80  avg =  175.01
         yolov4-tiny  min =  213.72  max =  214.94  avg =  214.08
           nanodet_m  min =   49.95  max =   50.47  avg =   50.19
    yolo-fastest-1.1  min =   23.80  max =   24.42  avg =   23.91
      yolo-fastestv2  min =   19.78  max =   19.95  avg =   19.84
  vision_transformer  min = 3927.51  max = 4025.76  avg = 3947.06

yolo-fastest-1.1  min =   23.80  max =   24.42  avg =   23.91
          FastestDet  min =   21.78  max =   22.17  avg =   21.88

mobtgzhang@mobtgzhang-PC:~/ncnn/benchmark$ ./benchncnn 10 4 1 -1 0
          squeezenet  min =   70.80  max =   76.55  avg =   72.49
     squeezenet_int8  min =  110.36  max =  133.06  avg =  114.23
           mobilenet  min =   77.97  max =   85.73  avg =   79.98
      mobilenet_int8  min =   80.05  max =   84.09  avg =   81.76
        mobilenet_v2  min =  101.07  max =  192.92  avg =  139.32
        mobilenet_v3  min =  108.60  max =  129.37  avg =  113.80
          shufflenet  min =  160.96  max =  188.96  avg =  168.62
       shufflenet_v2  min =   96.20  max =  190.31  avg =  119.77

mobilenet_v3  min =  108.60  max =  129.37  avg =  113.80
             mnasnet  min =   97.34  max =  104.00  avg =   99.85
     proxylessnasnet  min =  112.58  max =  276.49  avg =  145.74
     efficientnet_b0  min =  171.01  max =  238.15  avg =  195.53
   efficientnetv2_b0  min =  235.31  max =  299.00  avg =  254.12
        regnety_400m  min = 1059.87  max = 1173.49  avg = 1084.13
           blazeface  min =   58.69  max =   64.83  avg =   60.83
           googlenet  min =  190.47  max =  257.76  avg =  207.71
      googlenet_int8  min =  285.67  max =  327.20  avg =  300.87
            resnet18  min =  111.87  max =  118.36  avg =  114.48
       resnet18_int8  min =  143.08  max =  147.98  avg =  144.93
             alexnet  min =   72.83  max =   76.52  avg =   74.01
               vgg16  min =  390.35  max =  406.58  avg =  397.19

resnet18_int8  min =  143.08  max =  147.98  avg =  144.93
          vgg16_int8  min =  358.54  max =  369.89  avg =  364.31
            resnet50  min =  275.57  max =  300.14  avg =  283.21
       resnet50_int8  min =  315.18  max =  371.22  avg =  328.43
      squeezenet_ssd  min =  170.14  max =  200.18  avg =  175.23
 squeezenet_ssd_int8  min =  259.01  max =  271.23  avg =  263.35
       mobilenet_ssd  min =  166.85  max =  170.64  avg =  168.74
  mobilenet_ssd_int8  min =  191.71  max =  195.91  avg =  193.44
      mobilenet_yolo  min =  960.70  max = 1080.81  avg =  983.68
  mobilenetv2_yolov3  min =  187.72  max =  207.92  avg =  192.60
         yolov4-tiny  min =  172.72  max =  177.62  avg =  174.63
           nanodet_m  min =  128.79  max =  137.31  avg =  131.04
    yolo-fastest-1.1  min =  132.39  max =  148.06  avg =  137.90

yolov4-tiny  min =  172.72  max =  177.62  avg =  174.63
      yolo-fastestv2  min =  130.97  max =  137.73  avg =  133.53
  vision_transformer  min = 2229.10  max = 2392.59  avg = 2304.21
          FastestDet  min =  119.98  max =  126.26  avg =  122.40

mobtgzhang@mobtgzhang-PC:~/ncnn/benchmark$ ./benchncnn 10 4 2 -1 0
          squeezenet  min =   70.93  max =   75.55  avg =   72.93
     squeezenet_int8  min =  109.65  max =  153.48  avg =  124.20
           mobilenet  min =   78.02  max =   85.80  avg =   81.97
      mobilenet_int8  min =   80.34  max =   89.31  avg =   83.20
        mobilenet_v2  min =   99.51  max =  110.36  avg =  102.54
        mobilenet_v3  min =  109.04  max =  116.28  avg =  111.75

mobilenet_int8  min =   80.34  max =   89.31  avg =   83.20
          shufflenet  min =  160.04  max =  166.21  avg =  163.59
       shufflenet_v2  min =   88.90  max =   91.82  avg =   90.24
             mnasnet  min =   97.02  max =  103.09  avg =   98.70
     proxylessnasnet  min =  111.21  max =  117.47  avg =  113.97
     efficientnet_b0  min =  167.99  max =  175.35  avg =  171.26
   efficientnetv2_b0  min =  228.59  max =  245.97  avg =  232.79
        regnety_400m  min = 1049.34  max = 1085.18  avg = 1064.68
           blazeface  min =   59.35  max =   64.91  avg =   60.35
           googlenet  min =  187.87  max =  195.29  avg =  190.56
      googlenet_int8  min =  283.22  max =  301.69  avg =  287.66
            resnet18  min =  111.48  max =  116.76  avg =  112.88
       resnet18_int8  min =  142.41  max =  148.79  avg =  145.14

googlenet_int8  min =  283.22  max =  301.69  avg =  287.66
             alexnet  min =   72.59  max =   75.37  avg =   73.62
               vgg16  min =  389.61  max =  452.95  avg =  424.36
          vgg16_int8  min =  365.57  max =  465.13  avg =  422.84
            resnet50  min =  283.07  max =  411.14  avg =  332.88
       resnet50_int8  min =  323.21  max =  381.13  avg =  340.59
      squeezenet_ssd  min =  178.21  max =  252.82  avg =  211.62
 squeezenet_ssd_int8  min =  263.82  max =  372.38  avg =  284.38
       mobilenet_ssd  min =  166.29  max =  281.36  avg =  195.16
  mobilenet_ssd_int8  min =  194.00  max =  220.95  avg =  204.07
      mobilenet_yolo  min =  964.99  max = 1027.13  avg =  989.45
  mobilenetv2_yolov3  min =  218.58  max =  512.86  avg =  265.12
         yolov4-tiny  min =  172.20  max =  177.27  avg =  174.14

mobilenet_yolo  min =  964.99  max = 1027.13  avg =  989.45
           nanodet_m  min =  128.78  max =  222.66  avg =  150.88
    yolo-fastest-1.1  min =  132.52  max =  196.41  avg =  149.03
      yolo-fastestv2  min =  131.39  max =  138.72  avg =  134.96
  vision_transformer  min = 2243.31  max = 2659.56  avg = 2395.76
          FastestDet  min =  119.44  max =  126.07  avg =  122.27

```

### Phytium FT-2000+/64 (FTC662 armv8 2.4GHz x 8)
```
[root@bogon benchmark]# ./benchncnn 10 1 0 -1 0
          squeezenet  min =   57.60  max =   59.78  avg =   58.51
     squeezenet_int8  min =   47.05  max =   47.89  avg =   47.40
           mobilenet  min =   91.08  max =   95.16  avg =   91.89
      mobilenet_int8  min =   60.27  max =   61.17  avg =   60.74

squeezenet_int8  min =   47.05  max =   47.89  avg =   47.40
        mobilenet_v2  min =   63.38  max =   68.12  avg =   66.96
        mobilenet_v3  min =   53.34  max =   54.71  avg =   54.01
          shufflenet  min =   37.87  max =   41.78  avg =   39.37
       shufflenet_v2  min =   35.89  max =   37.30  avg =   36.40
             mnasnet  min =   59.57  max =   63.23  avg =   60.25
     proxylessnasnet  min =   71.24  max =   71.93  avg =   71.51
     efficientnet_b0  min =  134.34  max =  141.14  avg =  137.74
   efficientnetv2_b0  min =  143.82  max =  145.63  avg =  144.36
        regnety_400m  min =   76.96  max =   77.66  avg =   77.27
           blazeface  min =   11.57  max =   11.90  avg =   11.70
           googlenet  min =  188.10  max =  191.27  avg =  189.02
      googlenet_int8  min =  167.54  max =  169.63  avg =  168.38

blazeface  min =   11.57  max =   11.90  avg =   11.70
            resnet18  min =  144.76  max =  163.39  avg =  154.95
       resnet18_int8  min =  124.14  max =  129.84  avg =  127.83
             alexnet  min =  198.22  max =  208.86  avg =  205.35
               vgg16  min =  848.10  max =  891.00  avg =  859.94
          vgg16_int8  min =  686.54  max =  742.77  avg =  704.74
            resnet50  min =  413.45  max =  428.84  avg =  417.81
       resnet50_int8  min =  306.32  max =  324.27  avg =  316.47
      squeezenet_ssd  min =  147.62  max =  149.58  avg =  148.48
 squeezenet_ssd_int8  min =  116.18  max =  134.86  avg =  126.93
       mobilenet_ssd  min =  188.49  max =  191.97  avg =  189.48
  mobilenet_ssd_int8  min =  120.28  max =  121.36  avg =  120.83
      mobilenet_yolo  min =  421.79  max =  425.68  avg =  423.51

mobilenet_ssd  min =  188.49  max =  191.97  avg =  189.48
  mobilenetv2_yolov3  min =  222.86  max =  225.58  avg =  224.01
         yolov4-tiny  min =  303.77  max =  310.70  avg =  307.45
           nanodet_m  min =   80.87  max =   82.11  avg =   81.35

[root@bogon benchmark]# ./benchncnn 10 8 0 -1 0
          squeezenet  min =   14.53  max =   14.92  avg =   14.68
     squeezenet_int8  min =   11.67  max =   11.89  avg =   11.82
           mobilenet  min =   17.60  max =   20.05  avg =   18.34
      mobilenet_int8  min =    9.94  max =   10.22  avg =   10.08
        mobilenet_v2  min =   18.46  max =   19.18  avg =   18.81
        mobilenet_v3  min =   16.30  max =   16.71  avg =   16.45
          shufflenet  min =   14.65  max =   14.93  avg =   14.78

mobilenet_v2  min =   18.46  max =   19.18  avg =   18.81
       shufflenet_v2  min =   11.23  max =   11.56  avg =   11.35
             mnasnet  min =   15.65  max =   16.08  avg =   15.92
     proxylessnasnet  min =   18.78  max =   21.72  avg =   19.68
     efficientnet_b0  min =   29.16  max =   29.62  avg =   29.37
   efficientnetv2_b0  min =   33.28  max =   35.48  avg =   34.23
        regnety_400m  min =   44.90  max =   47.36  avg =   46.32
           blazeface  min =    4.23  max =    4.43  avg =    4.30
           googlenet  min =   42.11  max =   42.98  avg =   42.38
      googlenet_int8  min =   33.24  max =   38.21  avg =   34.10
            resnet18  min =   33.27  max =   34.00  avg =   33.57
       resnet18_int8  min =   23.66  max =   24.78  avg =   24.24
             alexnet  min =   35.78  max =   37.68  avg =   36.46

resnet18  min =   33.27  max =   34.00  avg =   33.57
               vgg16  min =  219.60  max =  235.79  avg =  222.11
          vgg16_int8  min =  128.64  max =  135.19  avg =  130.73
            resnet50  min =   84.15  max =   85.48  avg =   84.66
       resnet50_int8  min =   58.87  max =   61.98  avg =   59.85
      squeezenet_ssd  min =   47.60  max =   50.24  avg =   48.54
 squeezenet_ssd_int8  min =   36.42  max =   37.89  avg =   36.99
       mobilenet_ssd  min =   39.37  max =   42.63  avg =   41.06
  mobilenet_ssd_int8  min =   21.59  max =   22.05  avg =   21.83
      mobilenet_yolo  min =   83.16  max =   88.75  avg =   85.29
  mobilenetv2_yolov3  min =   58.13  max =   59.50  avg =   58.62
         yolov4-tiny  min =   74.18  max =   76.56  avg =   75.13
           nanodet_m  min =   25.16  max =   31.45  avg =   26.71

mobilenetv2_yolov3  min =   58.13  max =   59.50  avg =   58.62

root@FT2K:~/Desktop/ncnn-20221128/build/benchmark$ ./benchncnn 10 4 2 -1 0
          squeezenet  min =   14.19  max =   21.46  avg =   15.16
     squeezenet_int8  min =   11.63  max =   12.08  avg =   11.91
           mobilenet  min =   20.52  max =   37.00  avg =   23.66
      mobilenet_int8  min =   13.38  max =   25.95  avg =   15.01
        mobilenet_v2  min =   15.80  max =   16.59  avg =   16.12
        mobilenet_v3  min =   13.38  max =   17.62  avg =   14.21
          shufflenet  min =   10.62  max =   11.10  avg =   10.85
       shufflenet_v2  min =    9.09  max =   12.30  avg =    9.66
             mnasnet  min =   14.85  max =   15.67  avg =   15.14

shufflenet  min =   10.62  max =   11.10  avg =   10.85
     proxylessnasnet  min =   16.83  max =   17.10  avg =   16.98
     efficientnet_b0  min =   24.59  max =   26.40  avg =   25.06
   efficientnetv2_b0  min =   30.25  max =   34.46  avg =   31.42
        regnety_400m  min =   32.37  max =   41.10  avg =   35.17
           blazeface  min =    3.00  max =    3.56  avg =    3.18
           googlenet  min =   49.52  max =   64.98  avg =   56.29
      googlenet_int8  min =   38.65  max =   52.51  avg =   43.90
            resnet18  min =   42.81  max =   53.94  avg =   45.38
       resnet18_int8  min =   32.53  max =   53.62  avg =   37.26
             alexnet  min =   33.92  max =   47.88  avg =   37.12
               vgg16  min =  214.19  max =  228.96  avg =  220.16
          vgg16_int8  min =  164.22  max =  224.51  avg =  180.15

alexnet  min =   33.92  max =   47.88  avg =   37.12
            resnet50  min =  106.90  max =  189.61  avg =  133.34
       resnet50_int8  min =   79.62  max =   94.41  avg =   83.56
      squeezenet_ssd  min =   48.00  max =   49.11  avg =   48.43
 squeezenet_ssd_int8  min =   33.59  max =   47.60  avg =   37.57
       mobilenet_ssd  min =   43.97  max =   58.84  avg =   49.64
  mobilenet_ssd_int8  min =   27.94  max =   32.89  avg =   29.56
      mobilenet_yolo  min =  107.29  max =  118.80  avg =  114.24
  mobilenetv2_yolov3  min =   63.44  max =  106.75  avg =   70.69
         yolov4-tiny  min =   89.93  max =  155.39  avg =  101.90
           nanodet_m  min =   20.34  max =   28.67  avg =   21.44
    yolo-fastest-1.1  min =   11.74  max =   12.24  avg =   11.96
      yolo-fastestv2  min =    9.81  max =    9.98  avg =    9.91

nanodet_m  min =   20.34  max =   28.67  avg =   21.44
  vision_transformer  min = 1617.60  max = 1634.13  avg = 1625.87
          FastestDet  min =   10.19  max =   10.55  avg =   10.36
```
### HUAWEI KunPeng 920 2251K (x8 cores)
test on UOS 1050
```
          squeezenet  min =   12.11  max =   12.40  avg =   12.25
     squeezenet_int8  min =   14.24  max =   14.50  avg =   14.36
           mobilenet  min =   20.52  max =   21.11  avg =   20.63
      mobilenet_int8  min =   18.29  max =   18.63  avg =   18.45
        mobilenet_v2  min =   13.73  max =   13.90  avg =   13.79
        mobilenet_v3  min =   11.37  max =   11.49  avg =   11.41

mobilenet_int8  min =   18.29  max =   18.63  avg =   18.45
          shufflenet  min =    7.90  max =    7.96  avg =    7.92
       shufflenet_v2  min =    8.09  max =    8.13  avg =    8.11
             mnasnet  min =   13.26  max =   13.44  avg =   13.30
     proxylessnasnet  min =   16.19  max =   16.39  avg =   16.26
     efficientnet_b0  min =   34.92  max =   35.22  avg =   35.04
   efficientnetv2_b0  min =   43.82  max =   44.39  avg =   43.94
        regnety_400m  min =   17.55  max =   18.02  avg =   17.65
           blazeface  min =    3.05  max =    3.08  avg =    3.07
           googlenet  min =   58.65  max =   59.26  avg =   58.89
      googlenet_int8  min =   60.55  max =   63.00  avg =   61.96
            resnet18  min =   34.27  max =   35.43  avg =   34.84
       resnet18_int8  min =   60.79  max =   62.15  avg =   61.47

googlenet_int8  min =   60.55  max =   63.00  avg =   61.96
             alexnet  min =   42.01  max =   44.43  avg =   43.36
               vgg16  min =  174.46  max =  177.33  avg =  175.57
          vgg16_int8  min =  453.93  max =  457.03  avg =  454.79
            resnet50  min =   95.36  max =   96.27  avg =   95.55
       resnet50_int8  min =  119.77  max =  121.26  avg =  120.46
      squeezenet_ssd  min =   39.05  max =   39.69  avg =   39.20
 squeezenet_ssd_int8  min =   55.06  max =   56.23  avg =   55.72
       mobilenet_ssd  min =   45.20  max =   45.96  avg =   45.49
  mobilenet_ssd_int8  min =   39.40  max =   40.13  avg =   39.76
      mobilenet_yolo  min =   98.86  max =   99.85  avg =   99.34
  mobilenetv2_yolov3  min =   51.17  max =   52.89  avg =   51.89
         yolov4-tiny  min =   66.43  max =   67.23  avg =   66.70

mobilenet_yolo  min =   98.86  max =   99.85  avg =   99.34
           nanodet_m  min =   20.59  max =   20.79  avg =   20.71
    yolo-fastest-1.1  min =    7.90  max =    7.99  avg =    7.93
      yolo-fastestv2  min =    7.45  max =    7.49  avg =    7.47
  vision_transformer  min = 1586.33  max = 1595.34  avg = 1589.76
          FastestDet  min =    7.45  max =    7.52  avg =    7.47

mobtgzhang@mobtgzhang-PC:~/ncnn/benchmark$ ./benchncnn 10 8 0 -1 0
          squeezenet  min =    2.93  max =    3.10  avg =    3.00
     squeezenet_int8  min =    3.47  max =    3.56  avg =    3.52
           mobilenet  min =    3.89  max =    4.04  avg =    3.94
      mobilenet_int8  min =    3.29  max =    3.39  avg =    3.33

squeezenet_int8  min =    3.47  max =    3.56  avg =    3.52
        mobilenet_v2  min =    3.95  max =    4.08  avg =    3.98
        mobilenet_v3  min =    3.45  max =    3.59  avg =    3.49
          shufflenet  min =    3.42  max =    4.66  avg =    3.62
       shufflenet_v2  min =    2.60  max =    2.94  avg =    2.68
             mnasnet  min =    3.46  max =    3.57  avg =    3.52
     proxylessnasnet  min =    3.94  max =   12.34  avg =    4.88
     efficientnet_b0  min =    7.31  max =    7.60  avg =    7.38
   efficientnetv2_b0  min =    9.01  max =    9.22  avg =    9.08
        regnety_400m  min =    8.56  max =    9.36  avg =    8.70
           blazeface  min =    1.36  max =    3.52  avg =    1.60
           googlenet  min =   11.80  max =   12.02  avg =   11.93
      googlenet_int8  min =   11.87  max =   23.09  avg =   13.16

blazeface  min =    1.36  max =    3.52  avg =    1.60
            resnet18  min =    7.27  max =    7.64  avg =    7.38
       resnet18_int8  min =   11.02  max =   11.73  avg =   11.20
             alexnet  min =    9.05  max =    9.35  avg =    9.17
               vgg16  min =   44.13  max =   50.84  avg =   46.89
          vgg16_int8  min =   75.15  max =   80.73  avg =   77.52
            resnet50  min =   18.72  max =   27.49  avg =   19.96
       resnet50_int8  min =   22.72  max =   36.80  avg =   26.78
      squeezenet_ssd  min =   13.96  max =   27.42  avg =   15.62
 squeezenet_ssd_int8  min =   15.01  max =   29.53  avg =   19.51
       mobilenet_ssd  min =    9.37  max =   13.34  avg =   10.44
  mobilenet_ssd_int8  min =    8.07  max =   24.28  avg =    9.83
      mobilenet_yolo  min =   22.06  max =   24.89  avg =   22.91

mobilenet_ssd  min =    9.37  max =   13.34  avg =   10.44
  mobilenetv2_yolov3  min =   14.41  max =   15.97  avg =   14.78
         yolov4-tiny  min =   20.71  max =   23.96  avg =   21.42
           nanodet_m  min =    6.37  max =    6.59  avg =    6.45
    yolo-fastest-1.1  min =    4.27  max =    4.52  avg =    4.34
      yolo-fastestv2  min =    3.53  max =    3.63  avg =    3.58
  vision_transformer  min =  435.60  max =  523.43  avg =  479.70
          FastestDet  min =    3.54  max =    7.95  avg =    5.24

          squeezenet  min =    4.04  max =    4.22  avg =    4.09
     squeezenet_int8  min =    4.64  max =    4.76  avg =    4.69

           mobilenet  min =    6.04  max =    6.06  avg =    6.05
      mobilenet_int8  min =    5.23  max =    5.32  avg =    5.25
        mobilenet_v2  min =    5.00  max =    5.03  avg =    5.01
        mobilenet_v3  min =    4.49  max =    4.69  avg =    4.52
          shufflenet  min =    3.90  max =    3.94  avg =    3.91
       shufflenet_v2  min =    3.27  max =    3.48  avg =    3.33
             mnasnet  min =    4.80  max =    4.83  avg =    4.82
     proxylessnasnet  min =    5.20  max =    5.28  avg =    5.23
     efficientnet_b0  min =   10.53  max =   11.06  avg =   10.68
   efficientnetv2_b0  min =   13.18  max =   13.37  avg =   13.25
        regnety_400m  min =    9.20  max =    9.25  avg =    9.22
           blazeface  min =    1.43  max =    1.45  avg =    1.44

efficientnetv2_b0  min =   13.18  max =   13.37  avg =   13.25
           googlenet  min =   17.63  max =   17.78  avg =   17.71
      googlenet_int8  min =   17.63  max =   18.03  avg =   17.85
            resnet18  min =   10.34  max =   10.59  avg =   10.40
       resnet18_int8  min =   17.93  max =   18.84  avg =   18.25
             alexnet  min =   13.28  max =   13.37  avg =   13.31
               vgg16  min =   55.41  max =   56.60  avg =   55.70
          vgg16_int8  min =  123.71  max =  125.34  avg =  124.48
            resnet50  min =   27.82  max =   28.22  avg =   27.95
       resnet50_int8  min =   34.50  max =   34.89  avg =   34.70
      squeezenet_ssd  min =   14.67  max =   15.19  avg =   14.85
 squeezenet_ssd_int8  min =   19.76  max =   20.32  avg =   19.87
       mobilenet_ssd  min =   13.15  max =   13.38  avg =   13.21

squeezenet_ssd  min =   14.67  max =   15.19  avg =   14.85
  mobilenet_ssd_int8  min =   11.52  max =   11.70  avg =   11.60
      mobilenet_yolo  min =   30.95  max =   31.28  avg =   31.05
  mobilenetv2_yolov3  min =   20.04  max =   20.36  avg =   20.16
         yolov4-tiny  min =   25.61  max =   26.73  avg =   25.80
           nanodet_m  min =    7.93  max =    7.97  avg =    7.95
    yolo-fastest-1.1  min =    4.52  max =    4.59  avg =    4.53
      yolo-fastestv2  min =    3.74  max =    3.88  avg =    3.77
  vision_transformer  min =  546.94  max =  726.81  avg =  698.27
          FastestDet  min =    3.59  max =    3.61  avg =    3.60
```

### HUAWEI KunPeng 920 3211K (x24 cores)
test on ubuntu 22.04
```
(base) mobtgzhang@mobtgzhang-PC:~/ncnn/benchmark$ ./benchncnn 10 1 0 -1 0

```
          squeezenet  min =   12.11  max =   12.20  avg =   12.14
     squeezenet_int8  min =   14.34  max =   14.46  avg =   14.41
           mobilenet  min =   20.27  max =   20.36  avg =   20.31
      mobilenet_int8  min =   17.45  max =   17.74  avg =   17.58
        mobilenet_v2  min =   13.72  max =   13.87  avg =   13.78
        mobilenet_v3  min =   11.51  max =   11.69  avg =   11.61
          shufflenet  min =    8.07  max =    8.36  avg =    8.20
       shufflenet_v2  min =    8.13  max =    8.17  avg =    8.14
             mnasnet  min =   13.34  max =   13.45  avg =   13.41
     proxylessnasnet  min =   16.22  max =   16.35  avg =   16.29
     efficientnet_b0  min =   34.69  max =   35.14  avg =   34.82
   efficientnetv2_b0  min =   44.54  max =   44.68  avg =   44.61

proxylessnasnet  min =   16.22  max =   16.35  avg =   16.29
        regnety_400m  min =   18.06  max =   18.15  avg =   18.10
           blazeface  min =    3.06  max =    3.22  avg =    3.12
           googlenet  min =   56.80  max =   57.60  avg =   57.08
      googlenet_int8  min =   58.64  max =   59.98  avg =   59.42
            resnet18  min =   35.02  max =   35.35  avg =   35.10
       resnet18_int8  min =   61.13  max =   61.68  avg =   61.33
             alexnet  min =   42.56  max =   43.05  avg =   42.69
               vgg16  min =  186.32  max =  188.73  avg =  187.20
          vgg16_int8  min =  459.01  max =  461.48  avg =  460.29
            resnet50  min =   97.59  max =   98.32  avg =   97.83
       resnet50_int8  min =  118.67  max =  120.45  avg =  119.78
      squeezenet_ssd  min =   39.62  max =   39.95  avg =   39.81

resnet50  min =   97.59  max =   98.32  avg =   97.83
 squeezenet_ssd_int8  min =   56.72  max =   57.63  avg =   57.00
       mobilenet_ssd  min =   45.44  max =   45.82  avg =   45.63
  mobilenet_ssd_int8  min =   38.99  max =   40.08  avg =   39.39
      mobilenet_yolo  min =   98.71  max =   99.27  avg =   98.94
  mobilenetv2_yolov3  min =   51.50  max =   52.41  avg =   51.87
         yolov4-tiny  min =   68.02  max =   68.43  avg =   68.24
           nanodet_m  min =   20.49  max =   20.64  avg =   20.59
    yolo-fastest-1.1  min =    8.17  max =    8.45  avg =    8.23
      yolo-fastestv2  min =    7.73  max =    8.06  avg =    7.87
  vision_transformer  min = 1620.65  max = 1630.45  avg = 1625.64
          FastestDet  min =    7.65  max =    7.77  avg =    7.69
(base) mobtgzhang@mobtgzhang-PC:~/ncnn/benchmark$ ./benchncnn 10 2 0 -1 0

FastestDet  min =    7.65  max =    7.77  avg =    7.69
          squeezenet  min =    6.77  max =    6.85  avg =    6.81
     squeezenet_int8  min =    7.98  max =    8.07  avg =    8.03
           mobilenet  min =   10.70  max =   10.78  avg =   10.73
      mobilenet_int8  min =    9.21  max =    9.36  avg =    9.28
        mobilenet_v2  min =    7.91  max =    7.99  avg =    7.94
        mobilenet_v3  min =    6.72  max =    6.92  avg =    6.78
          shufflenet  min =    5.34  max =    5.55  avg =    5.38
       shufflenet_v2  min =    5.12  max =    5.15  avg =    5.14
             mnasnet  min =    7.74  max =    7.86  avg =    7.80
     proxylessnasnet  min =    9.00  max =    9.03  avg =    9.02
     efficientnet_b0  min =   18.51  max =   18.58  avg =   18.54

mnasnet  min =    7.74  max =    7.86  avg =    7.80
   efficientnetv2_b0  min =   23.68  max =   23.83  avg =   23.74
        regnety_400m  min =   12.65  max =   12.68  avg =   12.66
           blazeface  min =    1.99  max =    2.14  avg =    2.03
           googlenet  min =   30.83  max =   31.29  avg =   30.91
      googlenet_int8  min =   31.97  max =   33.12  avg =   32.45
            resnet18  min =   18.81  max =   18.87  avg =   18.84
       resnet18_int8  min =   32.80  max =   32.99  avg =   32.90
             alexnet  min =   22.88  max =   23.16  avg =   22.94
               vgg16  min =  100.58  max =  101.12  avg =  100.90
          vgg16_int8  min =  235.81  max =  237.97  avg =  236.20
            resnet50  min =   51.12  max =   51.43  avg =   51.28
       resnet50_int8  min =   62.46  max =   63.02  avg =   62.72

vgg16_int8  min =  235.81  max =  237.97  avg =  236.20
      squeezenet_ssd  min =   23.26  max =   23.73  avg =   23.38
 squeezenet_ssd_int8  min =   31.91  max =   32.30  avg =   32.13
       mobilenet_ssd  min =   24.73  max =   24.95  avg =   24.84
  mobilenet_ssd_int8  min =   20.99  max =   21.52  avg =   21.21
      mobilenet_yolo  min =   54.91  max =   55.70  avg =   55.15
  mobilenetv2_yolov3  min =   30.18  max =   30.52  avg =   30.31
         yolov4-tiny  min =   40.46  max =   40.61  avg =   40.55
           nanodet_m  min =   12.56  max =   12.72  avg =   12.62
    yolo-fastest-1.1  min =    6.00  max =    6.15  avg =    6.04
      yolo-fastestv2  min =    5.32  max =    5.59  avg =    5.43
  vision_transformer  min =  894.51  max =  896.28  avg =  895.57
          FastestDet  min =    5.33  max =    5.42  avg =    5.36

yolo-fastestv2  min =    5.32  max =    5.59  avg =    5.43
(base) mobtgzhang@mobtgzhang-PC:~/ncnn/benchmark$ ./benchncnn 10 4 0 -1 0
          squeezenet  min =    4.18  max =    4.35  avg =    4.22
     squeezenet_int8  min =    4.85  max =    4.98  avg =    4.89
           mobilenet  min =    5.80  max =    5.95  avg =    5.89
      mobilenet_int8  min =    4.86  max =    4.94  avg =    4.89
        mobilenet_v2  min =    4.66  max =    4.73  avg =    4.69
        mobilenet_v3  min =    4.46  max =    4.50  avg =    4.48
          shufflenet  min =    4.01  max =    4.17  avg =    4.04
       shufflenet_v2  min =    3.39  max =    3.41  avg =    3.39
             mnasnet  min =    4.81  max =    4.93  avg =    4.85

shufflenet  min =    4.01  max =    4.17  avg =    4.04
     proxylessnasnet  min =    5.47  max =    5.54  avg =    5.49
     efficientnet_b0  min =   10.49  max =   10.55  avg =   10.52
   efficientnetv2_b0  min =   13.67  max =   13.77  avg =   13.72
        regnety_400m  min =   10.20  max =   10.24  avg =   10.21
           blazeface  min =    1.52  max =    1.58  avg =    1.54
           googlenet  min =   17.65  max =   17.69  avg =   17.68
      googlenet_int8  min =   18.14  max =   18.27  avg =   18.19
            resnet18  min =   10.52  max =   10.63  avg =   10.57
       resnet18_int8  min =   17.42  max =   17.53  avg =   17.49
             alexnet  min =   13.12  max =   13.20  avg =   13.16
               vgg16  min =   55.24  max =   55.45  avg =   55.35
          vgg16_int8  min =  123.46  max =  124.23  avg =  123.75

alexnet  min =   13.12  max =   13.20  avg =   13.16
            resnet50  min =   28.31  max =   28.57  avg =   28.39
       resnet50_int8  min =   34.10  max =   34.39  avg =   34.23
      squeezenet_ssd  min =   14.85  max =   14.96  avg =   14.91
 squeezenet_ssd_int8  min =   19.71  max =   19.88  avg =   19.82
       mobilenet_ssd  min =   13.49  max =   13.58  avg =   13.52
  mobilenet_ssd_int8  min =   11.60  max =   11.70  avg =   11.66
      mobilenet_yolo  min =   31.74  max =   31.96  avg =   31.81
  mobilenetv2_yolov3  min =   17.87  max =   18.03  avg =   17.93
         yolov4-tiny  min =   25.63  max =   25.78  avg =   25.72
           nanodet_m  min =    8.16  max =    8.22  avg =    8.20
    yolo-fastest-1.1  min =    4.72  max =    4.86  avg =    4.75
      yolo-fastestv2  min =    3.98  max =    4.15  avg =    4.00

nanodet_m  min =    8.16  max =    8.22  avg =    8.20
  vision_transformer  min =  501.18  max =  503.51  avg =  502.12
          FastestDet  min =    3.74  max =    3.76  avg =    3.75
(base) mobtgzhang@mobtgzhang-PC:~/ncnn/benchmark$ ./benchncnn 10 8 0 -1 0
          squeezenet  min =    2.91  max =    3.10  avg =    2.97
     squeezenet_int8  min =    3.42  max =    3.74  avg =    3.51
           mobilenet  min =    3.57  max =    3.70  avg =    3.61
      mobilenet_int8  min =    3.06  max =    3.14  avg =    3.10
        mobilenet_v2  min =    3.73  max =    3.75  avg =    3.75
        mobilenet_v3  min =    3.50  max =    3.66  avg =    3.56
          shufflenet  min =    3.63  max =    3.65  avg =    3.64

mobilenet_v2  min =    3.73  max =    3.75  avg =    3.75
       shufflenet_v2  min =    2.85  max =    3.02  avg =    2.95
             mnasnet  min =    3.60  max =    3.67  avg =    3.62
     proxylessnasnet  min =    4.00  max =    4.08  avg =    4.03
     efficientnet_b0  min =    7.31  max =    7.34  avg =    7.33
   efficientnetv2_b0  min =    9.44  max =    9.51  avg =    9.47
        regnety_400m  min =    9.76  max =   10.07  avg =    9.90
           blazeface  min =    1.56  max =    1.75  avg =    1.61
           googlenet  min =   11.22  max =   11.28  avg =   11.25
      googlenet_int8  min =   11.40  max =   12.82  avg =   11.76
            resnet18  min =    6.83  max =    6.96  avg =    6.90
       resnet18_int8  min =   10.28  max =   10.38  avg =   10.33
             alexnet  min =    8.75  max =    8.88  avg =    8.80

resnet18  min =    6.83  max =    6.96  avg =    6.90
               vgg16  min =   36.00  max =   36.72  avg =   36.29
          vgg16_int8  min =   67.38  max =   67.72  avg =   67.54
            resnet50  min =   17.63  max =   17.82  avg =   17.68
       resnet50_int8  min =   20.05  max =   20.21  avg =   20.15
      squeezenet_ssd  min =   11.18  max =   11.45  avg =   11.26
 squeezenet_ssd_int8  min =   14.09  max =   14.23  avg =   14.18
       mobilenet_ssd  min =    8.60  max =    8.69  avg =    8.64
  mobilenet_ssd_int8  min =    7.75  max =    7.87  avg =    7.81
      mobilenet_yolo  min =   21.97  max =   22.25  avg =   22.09
  mobilenetv2_yolov3  min =   14.04  max =   14.18  avg =   14.12
         yolov4-tiny  min =   19.66  max =   19.93  avg =   19.81
           nanodet_m  min =    6.52  max =    6.67  avg =    6.57

mobilenetv2_yolov3  min =   14.04  max =   14.18  avg =   14.12
    yolo-fastest-1.1  min =    4.61  max =    4.76  avg =    4.66
      yolo-fastestv2  min =    3.78  max =    3.91  avg =    3.82
  vision_transformer  min =  323.01  max =  327.38  avg =  323.75
          FastestDet  min =    3.50  max =    3.54  avg =    3.51
(base) mobtgzhang@mobtgzhang-PC:~/ncnn/benchmark$ ./benchncnn 10 16 0 -1 0
num_threads = 16
          squeezenet  min =    3.00  max =    3.25  avg =    3.08
     squeezenet_int8  min =    4.13  max =    4.47  avg =    4.21
           mobilenet  min =    3.27  max =    3.42  avg =    3.34
      mobilenet_int8  min =    3.49  max =    3.58  avg =    3.56
        mobilenet_v2  min =    3.86  max =    4.10  avg =    3.97

mobilenet  min =    3.27  max =    3.42  avg =    3.34
        mobilenet_v3  min =    3.72  max =    3.80  avg =    3.76
          shufflenet  min =    4.67  max =    4.78  avg =    4.72
       shufflenet_v2  min =    3.16  max =    3.24  avg =    3.20
             mnasnet  min =    3.51  max =    3.65  avg =    3.57
     proxylessnasnet  min =    4.08  max =    4.35  avg =    4.15
     efficientnet_b0  min =    7.51  max =    7.80  avg =    7.63
   efficientnetv2_b0  min =    8.92  max =    9.39  avg =    9.05
        regnety_400m  min =   14.80  max =   15.05  avg =   14.89
           blazeface  min =    2.14  max =    2.28  avg =    2.20
           googlenet  min =    9.91  max =   10.00  avg =    9.96
      googlenet_int8  min =   11.51  max =   11.65  avg =   11.60
            resnet18  min =    6.39  max =    6.56  avg =    6.46

googlenet  min =    9.91  max =   10.00  avg =    9.96
       resnet18_int8  min =    9.76  max =    9.91  avg =    9.84
             alexnet  min =    6.99  max =    7.10  avg =    7.04
               vgg16  min =   27.52  max =   28.64  avg =   27.88
          vgg16_int8  min =   45.64  max =   45.93  avg =   45.78
            resnet50  min =   13.96  max =   14.17  avg =   14.07
       resnet50_int8  min =   16.82  max =   16.93  avg =   16.89
      squeezenet_ssd  min =   11.11  max =   11.54  avg =   11.23
 squeezenet_ssd_int8  min =   13.77  max =   14.00  avg =   13.88
       mobilenet_ssd  min =    8.21  max =    8.46  avg =    8.35
  mobilenet_ssd_int8  min =    8.87  max =    9.03  avg =    8.94
      mobilenet_yolo  min =   30.77  max =   31.35  avg =   31.08
  mobilenetv2_yolov3  min =   12.11  max =   13.10  avg =   12.43

mobilenet_ssd_int8  min =    8.87  max =    9.03  avg =    8.94
         yolov4-tiny  min =   18.25  max =   18.68  avg =   18.41
           nanodet_m  min =    6.55  max =    6.68  avg =    6.59
    yolo-fastest-1.1  min =    6.00  max =    6.22  avg =    6.09
      yolo-fastestv2  min =    4.86  max =    5.01  avg =    4.94
  vision_transformer  min =  218.18  max =  220.49  avg =  218.79
          FastestDet  min =    5.01  max =    5.14  avg =    5.07
(base) mobtgzhang@mobtgzhang-PC:~/ncnn/benchmark$ ./benchncnn 10 24 0 -1 0
num_threads = 24
          squeezenet  min =    3.52  max =    3.96  avg =    3.70
     squeezenet_int8  min =    5.49  max =    5.83  avg =    5.65
           mobilenet  min =    3.42  max =    3.83  avg =    3.55

squeezenet  min =    3.52  max =    3.96  avg =    3.70
      mobilenet_int8  min =    3.69  max =   45.17  avg =   11.59
        mobilenet_v2  min =    4.63  max =    5.44  avg =    4.84
        mobilenet_v3  min =    4.51  max =    4.89  avg =    4.68
          shufflenet  min =    6.21  max =    6.52  avg =    6.36
       shufflenet_v2  min =    3.98  max =   17.54  avg =    5.45
             mnasnet  min =    4.28  max =    4.56  avg =    4.39
     proxylessnasnet  min =    4.76  max =    5.13  avg =    4.92
     efficientnet_b0  min =    7.45  max =  111.76  avg =   22.59
   efficientnetv2_b0  min =   10.87  max =   33.13  avg =   13.51
        regnety_400m  min =   20.97  max =   21.73  avg =   21.46
           blazeface  min =    2.56  max =    2.82  avg =    2.67
           googlenet  min =   10.54  max =  105.87  avg =   21.85

regnety_400m  min =   20.97  max =   21.73  avg =   21.46
      googlenet_int8  min =   14.21  max =   77.02  avg =   22.23
            resnet18  min =    7.08  max =    7.51  avg =    7.31
       resnet18_int8  min =   11.25  max =   50.66  avg =   19.14
             alexnet  min =    7.13  max =    8.67  avg =    7.44
               vgg16  min =   27.59  max =   35.35  avg =   29.12
          vgg16_int8  min =   44.43  max =   51.76  avg =   46.90
            resnet50  min =   15.16  max =  105.98  avg =   24.91
       resnet50_int8  min =   19.82  max =   20.50  avg =   20.16
      squeezenet_ssd  min =   13.03  max =   13.69  avg =   13.40
 squeezenet_ssd_int8  min =   17.62  max =  187.55  avg =   39.92
       mobilenet_ssd  min =    8.83  max =   71.97  avg =   15.37
  mobilenet_ssd_int8  min =   10.22  max =   49.61  avg =   15.26

squeezenet_ssd_int8  min =   17.62  max =  187.55  avg =   39.92
      mobilenet_yolo  min =   35.19  max =   46.43  avg =   36.93
  mobilenetv2_yolov3  min =   12.96  max =   15.57  avg =   13.41
         yolov4-tiny  min =   19.22  max =   21.43  avg =   19.89
           nanodet_m  min =    7.71  max =    8.74  avg =    8.09
    yolo-fastest-1.1  min =    6.71  max =   78.72  avg =   14.16
      yolo-fastestv2  min =    5.72  max =    6.08  avg =    5.88
  vision_transformer  min =  192.16  max =  221.86  avg =  202.73
          FastestDet  min =    5.13  max =    5.47  avg =    5.30
```

### Intel Atom x5-Z8350
```
nihui@nihui-ROCK-Pi-X:~/ncnn/build/benchmark$ ./benchncnn 20 4 0 -1 1
loop_count = 20
          squeezenet  min =   50.22  max =   50.53  avg =   50.32

     squeezenet_int8  min =   77.92  max =   78.37  avg =   78.07
           mobilenet  min =   80.12  max =   81.53  avg =   80.35
      mobilenet_int8  min =  120.54  max =  124.10  avg =  120.84
        mobilenet_v2  min =   56.62  max =   60.12  avg =   58.37
        mobilenet_v3  min =   50.19  max =   50.41  avg =   50.27
          shufflenet  min =   37.96  max =   38.28  avg =   38.10
       shufflenet_v2  min =   35.28  max =   35.59  avg =   35.45
             mnasnet  min =   54.91  max =   55.10  avg =   55.01
     proxylessnasnet  min =   62.25  max =   62.59  avg =   62.40
     efficientnet_b0  min =  101.92  max =  105.73  avg =  102.27
   efficientnetv2_b0  min =  115.48  max =  117.25  avg =  115.89
        regnety_400m  min =   79.66  max =   81.70  avg =   79.95

efficientnet_b0  min =  101.92  max =  105.73  avg =  102.27
           blazeface  min =   10.43  max =   10.60  avg =   10.49
           googlenet  min =  170.41  max =  173.44  avg =  170.68
      googlenet_int8  min =  253.06  max =  257.34  avg =  253.57
            resnet18  min =  127.19  max =  130.69  avg =  127.65
       resnet18_int8  min =  200.54  max =  204.25  avg =  200.88
             alexnet  min =  104.89  max =  110.89  avg =  105.56
               vgg16  min =  653.78  max =  661.34  avg =  655.44
          vgg16_int8  min =  974.72  max = 1006.48  avg =  978.76
            resnet50  min =  367.63  max =  371.74  avg =  368.27
       resnet50_int8  min =  574.94  max =  584.08  avg =  576.18
      squeezenet_ssd  min =  115.35  max =  116.47  avg =  115.62
 squeezenet_ssd_int8  min =  169.95  max =  170.75  avg =  170.26

resnet50_int8  min =  574.94  max =  584.08  avg =  576.18
       mobilenet_ssd  min =  167.00  max =  172.02  avg =  168.95
  mobilenet_ssd_int8  min =  244.91  max =  248.30  avg =  245.27
      mobilenet_yolo  min =  382.80  max =  393.23  avg =  385.79
  mobilenetv2_yolov3  min =  208.23  max =  211.54  avg =  209.64
         yolov4-tiny  min =  251.10  max =  263.77  avg =  256.37
           nanodet_m  min =   84.48  max =   84.95  avg =   84.70
    yolo-fastest-1.1  min =   44.11  max =   45.15  avg =   44.26
      yolo-fastestv2  min =   37.95  max =   38.52  avg =   38.34

nihui@nihui-ROCK-Pi-X:~/ncnn/build/benchmark$ ./benchncnn 10 1 0 -1 1
          squeezenet  min =  130.52  max =  131.08  avg =  130.64

     squeezenet_int8  min =  231.03  max =  231.38  avg =  231.19
           mobilenet  min =  231.40  max =  231.74  avg =  231.61
      mobilenet_int8  min =  409.74  max =  410.02  avg =  409.85
        mobilenet_v2  min =  150.23  max =  150.72  avg =  150.47
        mobilenet_v3  min =  119.08  max =  119.34  avg =  119.20
          shufflenet  min =   72.62  max =   72.81  avg =   72.73
       shufflenet_v2  min =   73.63  max =   73.71  avg =   73.68
             mnasnet  min =  140.87  max =  141.09  avg =  140.98
     proxylessnasnet  min =  166.39  max =  166.75  avg =  166.54
     efficientnet_b0  min =  280.55  max =  281.30  avg =  280.77
   efficientnetv2_b0  min =  321.05  max =  321.24  avg =  321.16
        regnety_400m  min =  183.78  max =  184.64  avg =  183.91

efficientnet_b0  min =  280.55  max =  281.30  avg =  280.77
           blazeface  min =   18.94  max =   19.08  avg =   19.01
           googlenet  min =  453.56  max =  454.71  avg =  454.15
      googlenet_int8  min =  791.40  max =  791.93  avg =  791.61
            resnet18  min =  365.87  max =  366.40  avg =  366.15
       resnet18_int8  min =  652.86  max =  653.39  avg =  653.09
             alexnet  min =  289.15  max =  290.25  avg =  289.65
               vgg16  min = 1887.16  max = 1887.73  avg = 1887.41
          vgg16_int8  min = 3211.44  max = 3213.39  avg = 3212.55
            resnet50  min = 1060.37  max = 1061.40  avg = 1060.80
       resnet50_int8  min = 1869.41  max = 1870.59  avg = 1870.17
      squeezenet_ssd  min =  277.23  max =  277.83  avg =  277.50
 squeezenet_ssd_int8  min =  455.54  max =  458.06  avg =  456.28

resnet50_int8  min = 1869.41  max = 1870.59  avg = 1870.17
       mobilenet_ssd  min =  478.03  max =  478.83  avg =  478.32
  mobilenet_ssd_int8  min =  822.61  max =  822.96  avg =  822.79
      mobilenet_yolo  min = 1136.89  max = 1138.51  avg = 1137.74
  mobilenetv2_yolov3  min =  551.81  max =  552.53  avg =  552.14
         yolov4-tiny  min =  685.49  max =  686.15  avg =  685.79
           nanodet_m  min =  181.21  max =  181.52  avg =  181.32
    yolo-fastest-1.1  min =   82.21  max =   82.68  avg =   82.30
      yolo-fastestv2  min =   67.62  max =   68.36  avg =   68.10

root@nihui-ROCK-Pi-X:/home/nihui/osd/ncnn/build/benchmark# ./benchncnn 10 1 0 0 0
[0 Intel(R) HD Graphics (CHV)]  queueC=0[1]  queueG=0[1]  queueT=0[1]
[0 Intel(R) HD Graphics (CHV)]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=0

[0 Intel(R) HD Graphics (CHV)]  fp16-p/s/a=1/1/1  int8-p/s/a=1/1/1
[0 Intel(R) HD Graphics (CHV)]  subgroup=32  basic=1  vote=1  ballot=1  shuffle=1
          squeezenet  min =   29.14  max =   29.76  avg =   29.45
           mobilenet  min =   36.19  max =   37.03  avg =   36.52
        mobilenet_v2  min =   30.39  max =   31.62  avg =   30.76
        mobilenet_v3  min =   31.60  max =   32.25  avg =   31.92
          shufflenet  min =   22.47  max =   23.19  avg =   22.70
       shufflenet_v2  min =   22.30  max =   24.16  avg =   23.12
             mnasnet  min =   29.40  max =   30.23  avg =   29.84
     proxylessnasnet  min =   31.00  max =   31.91  avg =   31.41
     efficientnet_b0  min =   58.03  max =   58.74  avg =   58.42

mnasnet  min =   29.40  max =   30.23  avg =   29.84
   efficientnetv2_b0  min =  131.17  max =  191.61  avg =  161.37
        regnety_400m  min =   40.30  max =   42.27  avg =   41.04
           blazeface  min =   15.06  max =   15.96  avg =   15.48
           googlenet  min =   85.37  max =   86.49  avg =   85.84
            resnet18  min =   93.87  max =   95.00  avg =   94.53
             alexnet  min =  110.96  max =  120.83  avg =  115.14
               vgg16  min =  798.75  max =  812.60  avg =  804.93
            resnet50  min =  213.12  max =  214.81  avg =  213.79
      squeezenet_ssd  min =  124.48  max =  125.18  avg =  124.87
       mobilenet_ssd  min =   84.04  max =   84.70  avg =   84.49
      mobilenet_yolo  min =  186.52  max =  189.61  avg =  188.53
  mobilenetv2_yolov3  min =  102.07  max =  102.97  avg =  102.39

mobilenet_ssd  min =   84.04  max =   84.70  avg =   84.49
         yolov4-tiny  min =  212.49  max =  214.75  avg =  213.77
           nanodet_m  min =   42.97  max =   45.58  avg =   44.05
    yolo-fastest-1.1  min =   27.14  max =   32.53  avg =   28.76
      yolo-fastestv2  min =   20.73  max =   25.90  avg =   22.97
```

### Intel Celeron N5105
```
          squeezenet  min =   18.06  max =   18.21  avg =   18.12
     squeezenet_int8  min =   24.55  max =   25.16  avg =   24.69
           mobilenet  min =   32.22  max =   32.70  avg =   32.40
      mobilenet_int8  min =   40.52  max =   40.59  avg =   40.54
        mobilenet_v2  min =   22.54  max =   22.71  avg =   22.65
        mobilenet_v3  min =   17.86  max =   19.02  avg =   18.09

mobilenet_int8  min =   40.52  max =   40.59  avg =   40.54
          shufflenet  min =   11.23  max =   11.30  avg =   11.28
       shufflenet_v2  min =   11.04  max =   11.19  avg =   11.13
             mnasnet  min =   19.93  max =   20.09  avg =   20.01
     proxylessnasnet  min =   21.91  max =   22.00  avg =   21.95
     efficientnet_b0  min =   33.29  max =   33.66  avg =   33.50
   efficientnetv2_b0  min =   40.16  max =   40.63  avg =   40.34
        regnety_400m  min =   27.38  max =   27.59  avg =   27.50
           blazeface  min =    3.01  max =    3.11  avg =    3.04
           googlenet  min =   64.78  max =   65.16  avg =   65.01
      googlenet_int8  min =   80.11  max =   80.79  avg =   80.46
            resnet18  min =   53.91  max =   54.28  avg =   54.07
       resnet18_int8  min =   63.95  max =   64.20  avg =   64.06

googlenet_int8  min =   80.11  max =   80.79  avg =   80.46
             alexnet  min =   51.84  max =   52.17  avg =   52.00
               vgg16  min =  322.01  max =  324.34  avg =  322.72
          vgg16_int8  min =  323.83  max =  324.17  avg =  324.02
            resnet50  min =  152.66  max =  153.33  avg =  153.03
       resnet50_int8  min =  193.40  max =  194.55  avg =  194.03
      squeezenet_ssd  min =   44.07  max =   44.51  avg =   44.37
 squeezenet_ssd_int8  min =   51.08  max =   52.26  avg =   51.60
       mobilenet_ssd  min =   67.73  max =   68.21  avg =   67.98
  mobilenet_ssd_int8  min =   82.41  max =   82.70  avg =   82.55
      mobilenet_yolo  min =  157.38  max =  159.44  avg =  158.23
  mobilenetv2_yolov3  min =   83.35  max =   83.68  avg =   83.55
         yolov4-tiny  min =  107.25  max =  107.72  avg =  107.50

mobilenet_yolo  min =  157.38  max =  159.44  avg =  158.23
           nanodet_m  min =   26.93  max =   27.24  avg =   27.09
    yolo-fastest-1.1  min =   12.47  max =   12.71  avg =   12.61
      yolo-fastestv2  min =   10.65  max =   10.95  avg =   10.81

          squeezenet  min =   54.43  max =   54.48  avg =   54.46
     squeezenet_int8  min =   79.32  max =   79.64  avg =   79.43
           mobilenet  min =  105.92  max =  106.12  avg =  106.03
      mobilenet_int8  min =  152.24  max =  152.28  avg =  152.26
        mobilenet_v2  min =   62.44  max =   62.83  avg =   62.57
        mobilenet_v3  min =   49.47  max =   49.55  avg =   49.50
          shufflenet  min =   27.32  max =   27.37  avg =   27.34

mobilenet_v2  min =   62.44  max =   62.83  avg =   62.57
       shufflenet_v2  min =   29.85  max =   30.00  avg =   29.93
             mnasnet  min =   59.83  max =   60.09  avg =   59.98
     proxylessnasnet  min =   66.66  max =   66.84  avg =   66.76
     efficientnet_b0  min =  104.00  max =  104.19  avg =  104.08
   efficientnetv2_b0  min =  128.05  max =  128.39  avg =  128.21
        regnety_400m  min =   77.95  max =   78.03  avg =   78.00
           blazeface  min =    6.66  max =    6.77  avg =    6.70
           googlenet  min =  195.32  max =  195.75  avg =  195.52
      googlenet_int8  min =  275.81  max =  276.25  avg =  275.98
            resnet18  min =  160.94  max =  161.17  avg =  161.03
       resnet18_int8  min =  223.88  max =  224.12  avg =  224.03
             alexnet  min =  120.96  max =  121.16  avg =  121.05

resnet18  min =  160.94  max =  161.17  avg =  161.03
               vgg16  min =  852.50  max =  853.66  avg =  853.04
          vgg16_int8  min = 1081.07  max = 1083.31  avg = 1082.18
            resnet50  min =  497.54  max =  497.85  avg =  497.67
       resnet50_int8  min =  681.79  max =  682.60  avg =  682.29
      squeezenet_ssd  min =  101.81  max =  102.49  avg =  102.13
 squeezenet_ssd_int8  min =  147.77  max =  148.52  avg =  148.04
       mobilenet_ssd  min =  215.63  max =  216.07  avg =  215.91
  mobilenet_ssd_int8  min =  305.65  max =  305.97  avg =  305.78
      mobilenet_yolo  min =  494.99  max =  495.41  avg =  495.16
  mobilenetv2_yolov3  min =  233.51  max =  234.26  avg =  233.84
         yolov4-tiny  min =  287.26  max =  287.89  avg =  287.50
           nanodet_m  min =   70.48  max =   70.73  avg =   70.61

mobilenetv2_yolov3  min =  233.51  max =  234.26  avg =  233.84
    yolo-fastest-1.1  min =   27.32  max =   27.36  avg =   27.34
      yolo-fastestv2  min =   23.51  max =   23.85  avg =   23.76

[0 Intel(R) UHD Graphics (JSL)]  queueC=0[1]  queueG=0[1]  queueT=0[1]
[0 Intel(R) UHD Graphics (JSL)]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=0
[0 Intel(R) UHD Graphics (JSL)]  fp16-p/s/a=1/1/1  int8-p/s/a=1/1/1
[0 Intel(R) UHD Graphics (JSL)]  subgroup=32  basic=1  vote=1  ballot=1  shuffle=1
          squeezenet  min =   14.71  max =   15.37  avg =   14.90
           mobilenet  min =   15.38  max =   16.34  avg =   16.07
        mobilenet_v2  min =   13.58  max =   14.52  avg =   14.23
        mobilenet_v3  min =   14.95  max =   15.81  avg =   15.20

mobilenet  min =   15.38  max =   16.34  avg =   16.07
          shufflenet  min =   11.93  max =   12.73  avg =   12.31
       shufflenet_v2  min =   14.47  max =   14.74  avg =   14.60
             mnasnet  min =   15.32  max =   17.13  avg =   15.95
     proxylessnasnet  min =   15.34  max =   16.25  avg =   15.66
     efficientnet_b0  min =   26.02  max =   26.19  avg =   26.11
   efficientnetv2_b0  min =   75.92  max =   76.18  avg =   76.07
        regnety_400m  min =   17.79  max =   18.00  avg =   17.91
           blazeface  min =    5.03  max =    5.96  avg =    5.65
           googlenet  min =   35.20  max =   35.40  avg =   35.32
            resnet18  min =   35.49  max =   35.61  avg =   35.56
             alexnet  min =   40.93  max =   41.25  avg =   41.11
               vgg16  min =  220.66  max =  222.18  avg =  221.42

resnet18  min =   35.49  max =   35.61  avg =   35.56
            resnet50  min =   78.10  max =   78.48  avg =   78.28
      squeezenet_ssd  min =   46.90  max =   47.46  avg =   47.26
       mobilenet_ssd  min =   33.33  max =   33.54  avg =   33.44
      mobilenet_yolo  min =   67.54  max =   67.77  avg =   67.64
  mobilenetv2_yolov3  min =   38.98  max =   39.69  avg =   39.37
         yolov4-tiny  min =   68.01  max =   69.74  avg =   68.86
           nanodet_m  min =   17.41  max =   18.13  avg =   17.78
    yolo-fastest-1.1  min =   13.91  max =   14.18  avg =   14.03
      yolo-fastestv2  min =   15.94  max =   16.02  avg =   15.97
```

### nVIDIA RTX2060 of Notebook
```
C:\Users\ai\AppData\Local\Temp\benchmark>benchncnn.exe 64 1 0 0 0
[0 GeForce RTX 2060]  queueC=2[8]  queueG=0[16]  queueT=1[2]

```

```
[0 GeForce RTX 2060]  buglssc=0  bugihfa=0
[0 GeForce RTX 2060]  fp16p=1  fp16s=1  fp16a=1  int8s=1  int8a=1
          squeezenet  min =    2.14  max =    2.93  avg =    2.26
           mobilenet  min =    2.08  max =    2.53  avg =    2.22
        mobilenet_v2  min =    2.81  max =    4.03  avg =    3.05
        mobilenet_v3  min =    2.90  max =    3.53  avg =    3.08
          shufflenet  min =    1.94  max =    4.27  avg =    2.55
       shufflenet_v2  min =    2.34  max =    2.97  avg =    2.49
             mnasnet  min =    2.11  max =    2.86  avg =    2.37
     proxylessnasnet  min =    2.27  max =    3.25  avg =    2.49
           googlenet  min =    4.34  max =    6.79  avg =    5.25

mnasnet  min =    2.11  max =    2.86  avg =    2.37
            resnet18  min =    2.60  max =    4.36  avg =    2.90
             alexnet  min =    2.79  max =    4.70  avg =    3.04
               vgg16  min =   11.40  max =   14.32  avg =   12.42
            resnet50  min =    5.26  max =    5.86  avg =    5.51
      squeezenet_ssd  min =    5.58  max =    7.94  avg =    6.56
       mobilenet_ssd  min =    3.47  max =    5.29  avg =    3.77
      mobilenet_yolo  min =    5.49  max =    6.19  avg =    5.70
  mobilenetv2_yolov3  min =    3.69  max =    5.14  avg =    3.91
```

### nVIDIA RTX2080 of Desktop
```
E:\projects\framework\ncnn\benchmark>benchncnn.exe 4096 1 0 0 0
[0 GeForce RTX 2080]  queueC=2[8]  queueG=0[16]  queueT=1[2]
[0 GeForce RTX 2080]  buglssc=0  bugihfa=0
[0 GeForce RTX 2080]  fp16p=1  fp16s=1  fp16a=1  int8s=1  int8a=1

loop_count = 4096
          squeezenet  min =    1.39  max =   16.70  avg =    1.49
           mobilenet  min =    1.32  max =    2.55  avg =    1.42
        mobilenet_v2  min =    1.88  max =    5.02  avg =    2.00
        mobilenet_v3  min =    2.31  max =    3.58  avg =    2.45
          shufflenet  min =    1.45  max =    2.65  avg =    1.55
       shufflenet_v2  min =    1.90  max =    3.21  avg =    2.03
             mnasnet  min =    1.95  max =    3.17  avg =    2.09
     proxylessnasnet  min =    2.02  max =    2.95  avg =    2.16
           googlenet  min =    3.81  max =    5.91  avg =    4.05
            resnet18  min =    2.10  max =    3.28  avg =    2.24
             alexnet  min =    2.15  max =    3.35  avg =    2.30

googlenet  min =    3.81  max =    5.91  avg =    4.05
               vgg16  min =    7.33  max =   11.12  avg =    7.80
            resnet50  min =    4.21  max =    6.70  avg =    4.49
      squeezenet_ssd  min =    4.58  max =    6.86  avg =    4.88
       mobilenet_ssd  min =    2.90  max =    4.52  avg =    3.09
      mobilenet_yolo  min =    4.15  max =    6.09  avg =    4.40
  mobilenetv2_yolov3  min =    3.04  max =    9.13  avg =    3.28
```

### NVIDIA Jetson AGX Xavier (Carmel 2.2 GHz x 8 + Volta Tensor Cores 64)
```
i@ubuntu:~/projects/ncnn/benchmark$ ./benchncnn 32 1 0 -1 0
          squeezenet  min =   22.31  max =   23.29  avg =   22.68
     squeezenet_int8  min =   47.64  max =   52.88  avg =   49.72

           mobilenet  min =   37.50  max =   38.45  avg =   37.85
      mobilenet_int8  min =   89.14  max =   92.38  avg =   90.95
        mobilenet_v2  min =   24.31  max =   25.53  avg =   24.68
        mobilenet_v3  min =   20.20  max =   21.21  avg =   20.56
          shufflenet  min =   14.85  max =   15.64  avg =   15.15
       shufflenet_v2  min =   14.34  max =   16.11  avg =   14.86
             mnasnet  min =   23.42  max =   23.86  avg =   23.56
     proxylessnasnet  min =   27.44  max =   28.83  avg =   27.83
     efficientnet_b0  min =   34.57  max =   37.84  avg =   35.13
   efficientnetv2_b0  min =   65.16  max =   68.67  avg =   66.76
        regnety_400m  min =   33.86  max =   34.49  avg =   34.17
           blazeface  min =   11.86  max =   14.15  avg =   12.52

efficientnetv2_b0  min =   65.16  max =   68.67  avg =   66.76
           googlenet  min =   83.19  max =   89.84  avg =   85.14
      googlenet_int8  min =  146.74  max =  155.25  avg =  151.14
            resnet18  min =   50.46  max =   57.80  avg =   53.40
       resnet18_int8  min =  108.43  max =  116.14  avg =  110.78
             alexnet  min =   56.59  max =   64.93  avg =   59.51
               vgg16  min =  266.78  max =  272.16  avg =  269.14
          vgg16_int8  min =  538.71  max =  551.55  avg =  544.78
            resnet50  min =  169.11  max =  172.26  avg =  170.51
       resnet50_int8  min =  370.55  max =  384.36  avg =  377.75
      squeezenet_ssd  min =   58.51  max =   67.88  avg =   62.78
 squeezenet_ssd_int8  min =   95.34  max =  106.49  avg =   97.99
       mobilenet_ssd  min =   83.52  max =   86.84  avg =   84.86

squeezenet_ssd  min =   58.51  max =   67.88  avg =   62.78
  mobilenet_ssd_int8  min =  172.70  max =  181.84  avg =  176.25
      mobilenet_yolo  min =  165.26  max =  167.74  avg =  166.51
  mobilenetv2_yolov3  min =   88.11  max =   90.29  avg =   89.19
         yolov4-tiny  min =  105.44  max =  109.24  avg =  107.07
           nanodet_m  min =   33.60  max =   37.02  avg =   34.39
    yolo-fastest-1.1  min =   13.56  max =   14.22  avg =   13.75
      yolo-fastestv2  min =   13.76  max =   14.59  avg =   14.02
i@ubuntu:~/projects/ncnn/benchmark$ ./benchncnn 32 2 0 -1 0
          squeezenet  min =   13.05  max =   13.76  avg =   13.36
     squeezenet_int8  min =   26.08  max =   28.09  avg =   26.69
           mobilenet  min =   20.61  max =   21.21  avg =   20.81

squeezenet  min =   13.05  max =   13.76  avg =   13.36
      mobilenet_int8  min =   44.72  max =   47.33  avg =   45.76
        mobilenet_v2  min =   14.67  max =   15.23  avg =   14.86
        mobilenet_v3  min =   12.59  max =   15.50  avg =   13.36
          shufflenet  min =   12.74  max =   14.14  avg =   13.31
       shufflenet_v2  min =   10.05  max =   10.89  avg =   10.40
             mnasnet  min =   14.02  max =   14.75  avg =   14.19
     proxylessnasnet  min =   16.05  max =   16.94  avg =   16.31
     efficientnet_b0  min =   20.47  max =   23.05  avg =   20.81
   efficientnetv2_b0  min =   37.51  max =   41.53  avg =   39.19
        regnety_400m  min =   25.21  max =   25.73  avg =   25.39
           blazeface  min =    7.30  max =    8.44  avg =    7.43
           googlenet  min =   42.52  max =   47.38  avg =   44.39

regnety_400m  min =   25.21  max =   25.73  avg =   25.39
      googlenet_int8  min =   76.38  max =   81.63  avg =   77.93
            resnet18  min =   26.76  max =   28.72  avg =   27.22
       resnet18_int8  min =   55.97  max =   61.57  avg =   57.26
             alexnet  min =   29.29  max =   33.20  avg =   31.03
               vgg16  min =  134.32  max =  138.65  avg =  136.05
          vgg16_int8  min =  267.70  max =  281.71  avg =  272.79
            resnet50  min =   87.22  max =   88.75  avg =   87.65
       resnet50_int8  min =  183.80  max =  192.17  avg =  187.25
      squeezenet_ssd  min =   35.80  max =   39.00  avg =   37.32
 squeezenet_ssd_int8  min =   53.56  max =   60.43  avg =   55.58
       mobilenet_ssd  min =   44.17  max =   48.30  avg =   44.70
  mobilenet_ssd_int8  min =   90.32  max =   94.09  avg =   92.27

squeezenet_ssd_int8  min =   53.56  max =   60.43  avg =   55.58
      mobilenet_yolo  min =   87.50  max =   89.63  avg =   88.33
  mobilenetv2_yolov3  min =   49.76  max =   51.58  avg =   50.44
         yolov4-tiny  min =   61.17  max =   64.41  avg =   62.15
           nanodet_m  min =   21.43  max =   22.47  avg =   21.82
    yolo-fastest-1.1  min =   10.90  max =   12.63  avg =   11.12
      yolo-fastestv2  min =   10.61  max =   11.11  avg =   10.82
i@ubuntu:~/projects/ncnn/benchmark$ ./benchncnn 32 4 0 -1 0
          squeezenet  min =    8.06  max =    8.79  avg =    8.39
     squeezenet_int8  min =   14.96  max =   16.64  avg =   15.37
           mobilenet  min =   11.24  max =   11.91  avg =   11.48
      mobilenet_int8  min =   23.63  max =   24.75  avg =   23.81

squeezenet_int8  min =   14.96  max =   16.64  avg =   15.37
        mobilenet_v2  min =    9.27  max =    9.97  avg =    9.44
        mobilenet_v3  min =    8.81  max =   10.06  avg =    9.07
          shufflenet  min =   11.22  max =   11.53  avg =   11.37
       shufflenet_v2  min =    7.81  max =    8.17  avg =    7.97
             mnasnet  min =    9.40  max =   10.49  avg =   10.06
     proxylessnasnet  min =   10.53  max =   10.73  avg =   10.62
     efficientnet_b0  min =   13.55  max =   15.14  avg =   13.80
   efficientnetv2_b0  min =   19.83  max =   21.95  avg =   21.09
        regnety_400m  min =   21.80  max =   22.91  avg =   22.13
           blazeface  min =    5.17  max =    6.27  avg =    5.31
           googlenet  min =   22.67  max =   25.35  avg =   23.10
      googlenet_int8  min =   43.19  max =   45.68  avg =   43.72

blazeface  min =    5.17  max =    6.27  avg =    5.31
            resnet18  min =   15.19  max =   16.14  avg =   15.42
       resnet18_int8  min =   31.22  max =   34.76  avg =   31.81
             alexnet  min =   15.20  max =   17.65  avg =   15.56
               vgg16  min =   70.76  max =   73.21  avg =   71.70
          vgg16_int8  min =  137.94  max =  143.50  avg =  139.54
            resnet50  min =   47.15  max =   47.91  avg =   47.40
       resnet50_int8  min =   99.80  max =  102.94  avg =  100.29
      squeezenet_ssd  min =   22.10  max =   24.11  avg =   22.46
 squeezenet_ssd_int8  min =   33.21  max =   35.98  avg =   33.98
       mobilenet_ssd  min =   25.09  max =   26.81  avg =   25.50
  mobilenet_ssd_int8  min =   48.15  max =   50.96  avg =   49.49
      mobilenet_yolo  min =   48.63  max =   49.02  avg =   48.84

mobilenet_ssd  min =   25.09  max =   26.81  avg =   25.50
  mobilenetv2_yolov3  min =   30.93  max =   31.41  avg =   31.13
         yolov4-tiny  min =   38.43  max =   41.20  avg =   39.28
           nanodet_m  min =   14.95  max =   15.74  avg =   15.35
    yolo-fastest-1.1  min =    8.89  max =    9.18  avg =    9.01
      yolo-fastestv2  min =    8.36  max =    9.28  avg =    8.50
i@ubuntu:~/projects/ncnn/benchmark$ ./benchncnn 32 8 0 -1 0
          squeezenet  min =    6.52  max =   74.10  avg =   12.94
     squeezenet_int8  min =   10.44  max =   18.81  avg =   12.15
           mobilenet  min =    7.49  max =   14.63  avg =    8.67
      mobilenet_int8  min =   13.80  max =   15.89  avg =   14.53
        mobilenet_v2  min =    8.15  max =   11.42  avg =    8.78

mobilenet  min =    7.49  max =   14.63  avg =    8.67
        mobilenet_v3  min =    7.60  max =   10.92  avg =    8.38
          shufflenet  min =   11.51  max =   19.48  avg =   12.97
       shufflenet_v2  min =    7.06  max =   15.58  avg =    9.48
             mnasnet  min =    7.77  max =   15.12  avg =    8.68
     proxylessnasnet  min =    8.54  max =   42.73  avg =   10.00
     efficientnet_b0  min =   11.11  max =   12.86  avg =   11.89
   efficientnetv2_b0  min =   17.17  max =   29.03  avg =   20.48
        regnety_400m  min =   22.41  max =   36.72  avg =   25.49
           blazeface  min =    4.93  max =   11.62  avg =    6.13
           googlenet  min =   17.02  max =   31.61  avg =   19.92
      googlenet_int8  min =   27.70  max =   35.49  avg =   29.18
            resnet18  min =    9.74  max =   18.78  avg =   11.40

googlenet  min =   17.02  max =   31.61  avg =   19.92
       resnet18_int8  min =   18.52  max =   24.70  avg =   19.32
             alexnet  min =   10.70  max =   15.41  avg =   11.39
               vgg16  min =   40.80  max =   54.47  avg =   42.72
          vgg16_int8  min =   74.71  max =   79.66  avg =   76.37
            resnet50  min =   28.21  max =   36.62  avg =   29.41
       resnet50_int8  min =   54.53  max =   76.02  avg =   56.81
      squeezenet_ssd  min =   19.01  max =   30.68  avg =   24.89
 squeezenet_ssd_int8  min =   27.61  max =   35.87  avg =   29.22
       mobilenet_ssd  min =   17.35  max =   22.87  avg =   18.55
  mobilenet_ssd_int8  min =   29.92  max =   36.35  avg =   31.15
      mobilenet_yolo  min =   31.63  max =   55.61  avg =   34.31
  mobilenetv2_yolov3  min =   23.75  max =   35.45  avg =   25.68

mobilenet_ssd_int8  min =   29.92  max =   36.35  avg =   31.15
         yolov4-tiny  min =   29.23  max =   70.12  avg =   31.94
           nanodet_m  min =   13.00  max =   21.72  avg =   15.39
    yolo-fastest-1.1  min =    9.72  max =   17.94  avg =   11.45
      yolo-fastestv2  min =    9.16  max =   16.35  avg =   11.08
i@ubuntu:~/projects/ncnn/benchmark$ ./benchncnn 128 1 0 0 0
[0 NVIDIA Tegra Xavier (nvgpu)]  queueC=2[8]  queueG=0[16]  queueT=1[1]
[0 NVIDIA Tegra Xavier (nvgpu)]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=0
[0 NVIDIA Tegra Xavier (nvgpu)]  fp16-p/s/a=1/1/1  int8-p/s/a=1/1/1
[0 NVIDIA Tegra Xavier (nvgpu)]  subgroup=32  basic=1  vote=1  ballot=1  shuffle=1
          squeezenet  min =    4.85  max =   19.65  avg =    6.83

     squeezenet_int8  min =   46.38  max =   49.70  avg =   47.22
           mobilenet  min =    5.62  max =    6.61  avg =    6.33
      mobilenet_int8  min =   87.42  max =   92.95  avg =   90.52
        mobilenet_v2  min =    5.96  max =    7.53  avg =    6.50
        mobilenet_v3  min =    6.77  max =    7.83  avg =    7.01
          shufflenet  min =   10.58  max =   18.46  avg =   13.68
       shufflenet_v2  min =   20.06  max =   21.09  avg =   20.37
             mnasnet  min =    6.49  max =   26.49  avg =    8.26
     proxylessnasnet  min =    6.75  max =   27.37  avg =    7.88
     efficientnet_b0  min =   12.11  max =   48.35  avg =   14.63
   efficientnetv2_b0  min =   24.61  max =   69.68  avg =   34.33
        regnety_400m  min =    9.02  max =   34.40  avg =   10.84

efficientnet_b0  min =   12.11  max =   48.35  avg =   14.63
           blazeface  min =    7.55  max =    8.10  avg =    7.78
           googlenet  min =   12.57  max =   65.14  avg =   18.91
      googlenet_int8  min =  145.74  max =  155.87  avg =  151.06
            resnet18  min =    8.88  max =   30.48  avg =    9.34
       resnet18_int8  min =  109.19  max =  116.78  avg =  111.52
             alexnet  min =    9.06  max =   54.53  avg =   19.04
               vgg16  min =   18.12  max =   37.31  avg =   19.65
          vgg16_int8  min =  530.60  max =  551.58  avg =  542.33
            resnet50  min =   11.62  max =   20.64  avg =   12.17
       resnet50_int8  min =  374.83  max =  384.79  avg =  379.50
      squeezenet_ssd  min =   14.01  max =   55.88  avg =   23.64
 squeezenet_ssd_int8  min =   89.86  max =   95.80  avg =   92.18

resnet50_int8  min =  374.83  max =  384.79  avg =  379.50
       mobilenet_ssd  min =   13.20  max =   13.61  avg =   13.37
  mobilenet_ssd_int8  min =  170.17  max =  181.48  avg =  174.93
      mobilenet_yolo  min =   11.78  max =   20.42  avg =   13.34
  mobilenetv2_yolov3  min =   18.08  max =   62.94  avg =   26.70
         yolov4-tiny  min =   26.44  max =   34.83  avg =   31.83
           nanodet_m  min =    7.93  max =    9.91  avg =    9.01
    yolo-fastest-1.1  min =    6.03  max =   20.85  avg =    8.42
      yolo-fastestv2  min =    9.01  max =   20.60  avg =   12.51
```

### MacBook Pro (13-inch, M1, 2020)
```
MacBook-Pro benchmark % ./benchncnn 10 1 0 -1 0
          squeezenet  min =    4.80  max =    5.05  avg =    4.86

```
     squeezenet_int8  min =    4.02  max =    4.13  avg =    4.04
           mobilenet  min =    9.09  max =    9.41  avg =    9.22
      mobilenet_int8  min =    4.65  max =    4.76  avg =    4.70
        mobilenet_v2  min =    5.64  max =    5.83  avg =    5.73
        mobilenet_v3  min =    4.64  max =    4.85  avg =    4.76
          shufflenet  min =    3.48  max =    3.63  avg =    3.56
       shufflenet_v2  min =    3.69  max =    3.81  avg =    3.73
             mnasnet  min =    5.67  max =    5.94  avg =    5.77
     proxylessnasnet  min =    7.03  max =    7.28  avg =    7.20
     efficientnet_b0  min =    9.13  max =    9.53  avg =    9.28
   efficientnetv2_b0  min =   17.37  max =   18.47  avg =   17.63
        regnety_400m  min =    7.64  max =    8.08  avg =    7.72

efficientnet_b0  min =    9.13  max =    9.53  avg =    9.28
           blazeface  min =    1.80  max =    1.89  avg =    1.83
           googlenet  min =   25.71  max =   25.90  avg =   25.81
      googlenet_int8  min =   16.89  max =   17.10  avg =   16.97
            resnet18  min =   17.16  max =   17.28  avg =   17.20
       resnet18_int8  min =   15.55  max =   15.75  avg =   15.64
             alexnet  min =   30.60  max =   31.11  avg =   30.69
               vgg16  min =   73.41  max =   75.37  avg =   73.91
          vgg16_int8  min =  103.81  max =  105.15  avg =  104.19
            resnet50  min =   43.47  max =   44.24  avg =   43.68
       resnet50_int8  min =   30.37  max =   35.25  avg =   31.61
      squeezenet_ssd  min =   20.97  max =   21.21  avg =   21.12
 squeezenet_ssd_int8  min =   19.34  max =   19.54  avg =   19.42

resnet50_int8  min =   30.37  max =   35.25  avg =   31.61
       mobilenet_ssd  min =   22.18  max =   22.58  avg =   22.28
  mobilenet_ssd_int8  min =   13.27  max =   15.31  avg =   14.05
      mobilenet_yolo  min =   40.78  max =   41.04  avg =   40.89
  mobilenetv2_yolov3  min =   20.87  max =   21.92  avg =   21.02
         yolov4-tiny  min =   30.73  max =   32.37  avg =   31.29
           nanodet_m  min =    8.54  max =    8.86  avg =    8.65

MacBook-Pro benchmark % ./benchncnn 10 8 0 0 0
[0 Apple M1]  queueC=0[1]  queueG=0[1]  queueT=0[1]
[0 Apple M1]  bugsbn1=0  bugbilz=151  bugcopc=0  bugihfa=0
[0 Apple M1]  fp16-p/s/a=1/1/1  int8-p/s/a=1/1/1
[0 Apple M1]  subgroup=32  basic=1  vote=1  ballot=1  shuffle=1

          squeezenet  min =    1.86  max =    2.22  avg =    2.01
     squeezenet_int8  min =    2.38  max =    8.40  avg =    5.13
           mobilenet  min =    2.50  max =    2.91  avg =    2.64
      mobilenet_int8  min =    2.29  max =    5.26  avg =    3.54
        mobilenet_v2  min =    2.93  max =    3.12  avg =    2.98
        mobilenet_v3  min =    3.36  max =    3.61  avg =    3.48
          shufflenet  min =    1.99  max =    2.54  avg =    2.18
       shufflenet_v2  min =    2.35  max =    2.84  avg =    2.52
             mnasnet  min =    2.81  max =    3.33  avg =    2.92
     proxylessnasnet  min =    3.21  max =    3.62  avg =    3.36
     efficientnet_b0  min =    4.74  max =    5.73  avg =    5.07
   efficientnetv2_b0  min =   12.04  max =   13.04  avg =   12.61

proxylessnasnet  min =    3.21  max =    3.62  avg =    3.36
        regnety_400m  min =    3.86  max =    4.04  avg =    3.98
           blazeface  min =    0.98  max =    1.11  avg =    1.03
           googlenet  min =    4.86  max =    5.38  avg =    5.02
      googlenet_int8  min =    9.43  max =   15.72  avg =   10.44
            resnet18  min =    3.92  max =    4.59  avg =    4.24
       resnet18_int8  min =    6.83  max =    7.57  avg =    7.35
             alexnet  min =    7.49  max =    7.87  avg =    7.65
               vgg16  min =   34.10  max =   35.29  avg =   34.60
          vgg16_int8  min =   40.09  max =   44.66  avg =   41.95
            resnet50  min =    7.22  max =    7.83  avg =    7.42
       resnet50_int8  min =   14.52  max =   20.56  avg =   15.78
      squeezenet_ssd  min =    8.52  max =   13.79  avg =    9.98

resnet50  min =    7.22  max =    7.83  avg =    7.42
 squeezenet_ssd_int8  min =   12.38  max =   15.44  avg =   13.37
       mobilenet_ssd  min =    4.83  max =    6.00  avg =    5.31
  mobilenet_ssd_int8  min =    7.26  max =   13.12  avg =    9.01
      mobilenet_yolo  min =    7.22  max =    8.66  avg =    7.99
  mobilenetv2_yolov3  min =    7.46  max =    8.06  avg =    7.80
         yolov4-tiny  min =   12.17  max =   13.95  avg =   12.82
           nanodet_m  min =    3.54  max =    4.78  avg =    3.86
```

### Ingenic T40XP Xburst2 Core X2 1.4Ghz (without MSA)
```
          squeezenet  min =  921.23  max =  944.03  avg =  930.71
     squeezenet_int8  min = 3280.89  max = 3404.83  avg = 3359.68
           mobilenet  min = 1277.61  max = 1298.51  avg = 1284.38

squeezenet  min =  921.23  max =  944.03  avg =  930.71
      mobilenet_int8  min = 4342.67  max = 4350.21  avg = 4345.85
        mobilenet_v2  min =  780.92  max =  783.93  avg =  782.79
        mobilenet_v3  min =  650.59  max =  655.08  avg =  652.06
          shufflenet  min =  352.75  max =  353.69  avg =  353.24
       shufflenet_v2  min =  362.82  max =  364.08  avg =  363.38
             mnasnet  min =  790.45  max =  791.89  avg =  790.99
     proxylessnasnet  min =  868.71  max =  870.47  avg =  869.17
     efficientnet_b0  min = 1491.44  max = 1492.36  avg = 1491.95
   efficientnetv2_b0  min = 2135.04  max = 2148.02  avg = 2139.99
        regnety_400m  min = 1000.53  max = 1005.29  avg = 1001.81
           blazeface  min =  102.72  max =  104.18  avg =  103.51
           googlenet  min = 3652.89  max = 3705.40  avg = 3675.43

regnety_400m  min = 1000.53  max = 1005.29  avg = 1001.81
      googlenet_int8  min = 8067.30  max = 8070.22  avg = 8069.21
```
### MacBook Pro (15-inch, 2019) - 2.6GHz six cores Intel Core i7 && Radeon Pro 555X 4GB && Intel UHD Graphics 630 1536MB
```

➜  benchmark git:(master) ✗ ./benchncnn 10 1 0 -1
          squeezenet  min =   14.68  max =   17.06  avg =   15.55
     squeezenet_int8  min =   51.64  max =   57.85  avg =   54.01
           mobilenet  min =   20.74  max =   25.38  avg =   22.77
      mobilenet_int8  min =   66.84  max =   91.01  avg =   75.69
        mobilenet_v2  min =   14.04  max =   20.06  avg =   16.36
        mobilenet_v3  min =   11.89  max =   16.22  avg =   13.58
          shufflenet  min =   13.74  max =   17.10  avg =   15.02

mobilenet_v2  min =   14.04  max =   20.06  avg =   16.36
       shufflenet_v2  min =   12.73  max =   14.36  avg =   13.53
             mnasnet  min =   11.05  max =   17.79  avg =   13.82
     proxylessnasnet  min =   12.60  max =   27.38  avg =   17.55
     efficientnet_b0  min =   23.73  max =   26.82  avg =   25.45
   efficientnetv2_b0  min =   27.03  max =   33.89  avg =   30.78
        regnety_400m  min =   13.81  max =   21.50  avg =   15.40
           blazeface  min =    3.72  max =    4.98  avg =    4.43
           googlenet  min =   65.88  max =   76.62  avg =   69.40
      googlenet_int8  min =  192.07  max =  227.85  avg =  203.81
            resnet18  min =   79.45  max =   90.41  avg =   85.32
       resnet18_int8  min =  201.71  max =  222.31  avg =  207.39
             alexnet  min =   70.67  max =   80.13  avg =   74.43

resnet18  min =   79.45  max =   90.41  avg =   85.32
               vgg16  min =  233.74  max =  261.62  avg =  250.99
          vgg16_int8  min = 1722.78  max = 1997.14  avg = 1772.71
            resnet50  min =  130.39  max =  135.31  avg =  133.27
       resnet50_int8  min =  439.69  max =  483.78  avg =  461.33
      squeezenet_ssd  min =  108.54  max =  122.15  avg =  115.02
 squeezenet_ssd_int8  min =  175.58  max =  185.09  avg =  181.33
       mobilenet_ssd  min =   51.89  max =   59.32  avg =   54.30
  mobilenet_ssd_int8  min =  140.15  max =  192.10  avg =  164.47
      mobilenet_yolo  min =  117.37  max =  131.89  avg =  126.34
  mobilenetv2_yolov3  min =   57.57  max =   72.29  avg =   64.92
         yolov4-tiny  min =  114.45  max =  123.15  avg =  116.91
           nanodet_m  min =   25.65  max =   33.27  avg =   28.75

mobilenetv2_yolov3  min =   57.57  max =   72.29  avg =   64.92

➜  benchmark git:(master) ✗ ./benchncnn 10 1 0 0
[0 AMD Radeon Pro 555X]  queueC=0[1]  queueG=0[1]  queueT=0[1]
[0 AMD Radeon Pro 555X]  bugsbn1=0  bugbilz=196  bugcopc=0  bugihfa=0
[0 AMD Radeon Pro 555X]  fp16-p/s/a=1/1/1  int8-p/s/a=1/1/1
[0 AMD Radeon Pro 555X]  subgroup=64  basic=0  vote=0  ballot=0  shuffle=0
[1 Intel(R) UHD Graphics 630]  queueC=0[1]  queueG=0[1]  queueT=0[1]
[1 Intel(R) UHD Graphics 630]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=0
[1 Intel(R) UHD Graphics 630]  fp16-p/s/a=1/1/1  int8-p/s/a=1/1/1
[1 Intel(R) UHD Graphics 630]  subgroup=32  basic=0  vote=0  ballot=0  shuffle=0
          squeezenet  min =    6.66  max =    7.30  avg =    6.91

     squeezenet_int8  min =   49.97  max =   60.92  avg =   53.86
           mobilenet  min =    6.99  max =    7.48  avg =    7.17
      mobilenet_int8  min =   70.46  max =   83.20  avg =   79.33
        mobilenet_v2  min =    9.56  max =   10.87  avg =   10.34
        mobilenet_v3  min =   11.48  max =   12.20  avg =   11.94
          shufflenet  min =    4.52  max =    5.25  avg =    4.96
       shufflenet_v2  min =    7.29  max =    9.65  avg =    7.99
             mnasnet  min =    9.82  max =   11.88  avg =   10.62
     proxylessnasnet  min =    7.85  max =    8.41  avg =    8.07
     efficientnet_b0  min =   17.34  max =   17.85  avg =   17.56
   efficientnetv2_b0  min =   21.95  max =   24.10  avg =   23.15
        regnety_400m  min =   13.54  max =   14.83  avg =   14.11

efficientnet_b0  min =   17.34  max =   17.85  avg =   17.56
           blazeface  min =    3.26  max =    6.59  avg =    5.50
           googlenet  min =   17.62  max =   19.47  avg =   18.27
      googlenet_int8  min =  198.88  max =  247.97  avg =  223.31
            resnet18  min =   11.10  max =   12.01  avg =   11.59
       resnet18_int8  min =  225.56  max =  259.39  avg =  238.97
             alexnet  min =   17.66  max =   19.19  avg =   18.24
               vgg16  min =   53.20  max =   54.88  avg =   53.73
          vgg16_int8  min = 1747.52  max = 2130.08  avg = 1880.42
            resnet50  min =   27.38  max =   28.84  avg =   28.34
       resnet50_int8  min =  461.86  max =  579.83  avg =  528.15
      squeezenet_ssd  min =   19.99  max =   20.98  avg =   20.50
 squeezenet_ssd_int8  min =  185.20  max =  209.66  avg =  196.81

resnet50_int8  min =  461.86  max =  579.83  avg =  528.15
       mobilenet_ssd  min =   12.81  max =   14.21  avg =   13.48
  mobilenet_ssd_int8  min =  139.29  max =  168.38  avg =  148.20
      mobilenet_yolo  min =   19.50  max =   20.51  avg =   19.97
  mobilenetv2_yolov3  min =   15.95  max =   19.28  avg =   16.85
         yolov4-tiny  min =   21.43  max =   23.42  avg =   22.28
           nanodet_m  min =    7.95  max =    9.23  avg =    8.48

➜  benchmark git:(master) ✗ ./benchncnn 10 1 0 1
[0 AMD Radeon Pro 555X]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=0

          squeezenet  min =   11.06  max =   13.22  avg =   12.09
     squeezenet_int8  min =   54.87  max =   64.55  avg =   59.84
           mobilenet  min =   13.65  max =   16.70  avg =   14.81
      mobilenet_int8  min =   72.36  max =   93.58  avg =   86.40
        mobilenet_v2  min =   11.88  max =   15.90  avg =   13.47
        mobilenet_v3  min =   12.68  max =   16.16  avg =   14.56
          shufflenet  min =   13.87  max =   16.68  avg =   14.93
       shufflenet_v2  min =   11.73  max =   13.65  avg =   12.87

mobilenet_v3  min =   12.68  max =   16.16  avg =   14.56
             mnasnet  min =   12.71  max =   15.56  avg =   14.22
     proxylessnasnet  min =   14.03  max =   17.28  avg =   15.37
     efficientnet_b0  min =   17.50  max =   21.46  avg =   19.30
   efficientnetv2_b0  min =   35.47  max =   38.58  avg =   36.89
        regnety_400m  min =   16.00  max =   19.45  avg =   17.48
           blazeface  min =    6.08  max =    7.18  avg =    6.39
           googlenet  min =   23.35  max =   29.68  avg =   25.77
      googlenet_int8  min =  198.49  max =  254.38  avg =  222.77
            resnet18  min =   21.85  max =   28.10  avg =   24.70
       resnet18_int8  min =  211.21  max =  279.55  avg =  222.64
             alexnet  min =   24.45  max =   30.47  avg =   26.87
               vgg16  min =  115.20  max =  117.76  avg =  116.48

resnet18_int8  min =  211.21  max =  279.55  avg =  222.64
          vgg16_int8  min = 1715.92  max = 1960.02  avg = 1800.21
            resnet50  min =   45.65  max =   46.25  avg =   46.05
       resnet50_int8  min =  448.13  max =  555.53  avg =  485.47
      squeezenet_ssd  min =   28.43  max =   33.26  avg =   29.85
 squeezenet_ssd_int8  min =  180.91  max =  202.51  avg =  190.84
       mobilenet_ssd  min =   21.03  max =   26.93  avg =   23.48
  mobilenet_ssd_int8  min =  154.41  max =  184.64  avg =  165.04
      mobilenet_yolo  min =   37.04  max =   38.64  avg =   37.52
  mobilenetv2_yolov3  min =   24.98  max =   30.03  avg =   27.70
         yolov4-tiny  min =   39.29  max =   50.25  avg =   44.18
           nanodet_m  min =   15.97  max =   20.27  avg =   17.93
```

### Sunway SW421 (sw_64 1.7GHz * 4)
```

yolov4-tiny  min =   39.29  max =   50.25  avg =   44.18
```

```
root@SW421:~/Desktop/ncnn-20220420/ncnn-20220420/build/benchmark$ ./benchncnn
          squeezenet  min =  943.61  max =  966.98  avg =  955.24
     squeezenet_int8  min =  654.75  max =  731.28  avg =  674.87
           mobilenet  min = 1584.87  max = 1612.88  avg = 1597.47
      mobilenet_int8  min = 1198.21  max = 1204.82  avg = 1201.61
        mobilenet_v2  min =  733.94  max =  754.79  avg =  744.48
        mobilenet_v3  min =  665.26  max =  683.81  avg =  675.18
          shufflenet  min =  401.53  max =  435.21  avg =  420.32
       shufflenet_v2  min =  294.65  max =  316.50  avg =  309.08
             mnasnet  min =  671.22  max =  808.46  avg =  713.01
     proxylessnasnet  min =  686.12  max =  698.13  avg =  692.29

shufflenet_v2  min =  294.65  max =  316.50  avg =  309.08
     efficientnet_b0  min = 1151.75  max = 1184.86  avg = 1161.33
   efficientnetv2_b0  min = 1372.05  max = 1395.22  avg = 1379.47
        regnety_400m  min =  933.93  max =  949.42  avg =  942.43
           blazeface  min =  104.72  max =  136.77  avg =  112.86
           googlenet  min = 2574.02  max = 4330.81  avg = 3015.56
      googlenet_int8  min = 2136.42  max = 2183.61  avg = 2166.45
            resnet18  min = 2511.12  max = 2537.42  avg = 2526.08
       resnet18_int8  min = 2003.84  max = 2027.50  avg = 2012.48
             alexnet  min =  668.28  max =  686.35  avg =  673.95
               vgg16  min = 24863.92  max = 24967.94  avg = 24907.39
          vgg16_int8  min = 18735.54  max = 18926.83  avg = 18859.32
            resnet50  min = 9896.47  max = 9981.13  avg = 9929.77

vgg16_int8  min = 18735.54  max = 18926.83  avg = 18859.32
       resnet50_int8  min = 6971.01  max = 7085.29  avg = 7017.88
      squeezenet_ssd  min = 1798.23  max = 1814.25  avg = 1806.57
 squeezenet_ssd_int8  min = 1586.11  max = 1606.83  avg = 1596.75
       mobilenet_ssd  min = 3995.54  max = 4018.27  avg = 4002.78
  mobilenet_ssd_int8  min = 2753.65  max = 2766.06  avg = 2760.04
      mobilenet_yolo  min = 10892.22  max = 10978.84  avg = 10921.00
  mobilenetv2_yolov3  min = 3600.80  max = 3607.72  avg = 3603.18
         yolov4-tiny  min = 5565.82  max = 5582.22  avg = 5571.78
           nanodet_m  min = 1182.97  max = 1220.47  avg = 1199.30
    yolo-fastest-1.1  min =  340.63  max =  360.95  avg =  349.15
      yolo-fastestv2  min =  255.47  max =  281.79  avg =  268.82
```

### Sunway SW831 (sw_64 2.5GHz * 8)
```
root@SW831:~/Desktop/ncnn_20221128/build/benchmark$ ./benchncnn 5 8 2 -1 0

yolo-fastestv2  min =  255.47  max =  281.79  avg =  268.82
```

```
          squeezenet  min =  343.27  max =  420.86  avg =  364.97
     squeezenet_int8  min =  237.91  max =  251.71  avg =  243.84
           mobilenet  min =  607.80  max =  696.04  avg =  646.61
      mobilenet_int8  min =  428.37  max =  499.32  avg =  460.21
        mobilenet_v2  min =  291.29  max =  381.93  avg =  311.76
        mobilenet_v3  min =  262.01  max =  287.93  avg =  277.29
          shufflenet  min =  144.89  max =  169.10  avg =  150.84
       shufflenet_v2  min =  121.44  max =  139.62  avg =  126.96
             mnasnet  min =  265.59  max =  353.84  avg =  288.79
     proxylessnasnet  min =  272.08  max =  293.19  avg =  284.61
     efficientnet_b0  min =  445.40  max =  508.36  avg =  467.84

mnasnet  min =  265.59  max =  353.84  avg =  288.79
   efficientnetv2_b0  min =  550.57  max =  619.16  avg =  581.85
        regnety_400m  min =  374.02  max =  460.64  avg =  394.49
           blazeface  min =   39.93  max =   59.19  avg =   44.14
           googlenet  min =  941.35  max = 1014.23  avg =  976.37
      googlenet_int8  min =  770.66  max =  827.44  avg =  797.93
            resnet18  min =  815.02  max =  895.13  avg =  843.57
       resnet18_int8  min =  701.10  max =  776.40  avg =  729.49
             alexnet  min =  216.74  max =  273.39  avg =  228.99
               vgg16  min = 8645.55  max = 8699.60  avg = 8681.61
          vgg16_int8  min = 6786.91  max = 6930.90  avg = 6854.29
            resnet50  min = 3624.02  max = 3698.91  avg = 3652.31
       resnet50_int8  min = 2537.92  max = 2618.10  avg = 2567.88

vgg16_int8  min = 6786.91  max = 6930.90  avg = 6854.29
      squeezenet_ssd  min =  635.25  max =  693.23  avg =  663.56
 squeezenet_ssd_int8  min =  577.37  max =  641.12  avg =  603.34
       mobilenet_ssd  min = 1529.35  max = 1711.54  avg = 1582.10
  mobilenet_ssd_int8  min =  982.65  max = 1042.82  avg = 1016.62
      mobilenet_yolo  min = 4053.62  max = 4124.84  avg = 4094.38
  mobilenetv2_yolov3  min = 1367.81  max = 1527.79  avg = 1433.04
         yolov4-tiny  min = 1943.20  max = 2028.02  avg = 1978.31
           nanodet_m  min =  433.66  max =  498.83  avg =  457.77
    yolo-fastest-1.1  min =  140.07  max =  284.35  avg =  192.46
      yolo-fastestv2  min =  123.91  max =  225.70  avg =  152.54
  vision_transformer  min = 2470.70  max = 2509.73  avg = 2486.40
          FastestDet  min =  145.30  max =  163.43  avg =  154.35
```

vision_transformer  min = 2470.70  max = 2509.73  avg = 2486.40
```

### AXERA AX620A (Cortex-A7 1.0GHz * 4)
```
/root/axera # ./benchncnn 4 1 0 -1 0
          squeezenet  min =  530.57  max =  533.11  avg =  532.22
     squeezenet_int8  min =  359.74  max =  360.02  avg =  359.86
           mobilenet  min =  920.12  max =  921.04  avg =  920.52
      mobilenet_int8  min =  532.60  max =  533.08  avg =  532.81
        mobilenet_v2  min =  608.81  max =  609.49  avg =  609.18
        mobilenet_v3  min =  531.43  max =  532.34  avg =  531.90
          shufflenet  min =  297.91  max =  300.08  avg =  299.06
       shufflenet_v2  min =  288.44  max =  289.30  avg =  288.79
             mnasnet  min =  590.29  max =  590.99  avg =  590.63
     proxylessnasnet  min =  678.22  max =  679.22  avg =  678.63

shufflenet_v2  min =  288.44  max =  289.30  avg =  288.79
     efficientnet_b0  min = 1041.41  max = 1043.79  avg = 1042.61
   efficientnetv2_b0  min = 1222.41  max = 1223.63  avg = 1222.91
        regnety_400m  min =  723.83  max =  725.37  avg =  724.64
           blazeface  min =   86.77  max =   87.21  avg =   86.92
           googlenet  min = 1740.32  max = 1741.44  avg = 1740.81
      googlenet_int8  min = 1167.95  max = 1169.18  avg = 1168.54
            resnet18  min = 1584.41  max = 1585.36  avg = 1584.97
       resnet18_int8  min =  915.78  max =  918.77  avg =  917.16
             alexnet  min = 1811.30  max = 1812.86  avg = 1812.07
            resnet50  min = 4516.48  max = 4523.48  avg = 4519.03
       resnet50_int8  min = 2573.18  max = 2574.29  avg = 2573.69
      squeezenet_ssd  min = 1191.79  max = 1193.71  avg = 1193.02

resnet50  min = 4516.48  max = 4523.48  avg = 4519.03
 squeezenet_ssd_int8  min =  862.36  max =  863.69  avg =  862.83
       mobilenet_ssd  min = 1950.48  max = 1950.98  avg = 1950.65
  mobilenet_ssd_int8  min = 1081.70  max = 1082.64  avg = 1082.20
      mobilenet_yolo  min = 4629.22  max = 4630.23  avg = 4629.69
  mobilenetv2_yolov3  min = 2233.05  max = 2234.14  avg = 2233.42
         yolov4-tiny  min = 2942.58  max = 2946.55  avg = 2944.81
           nanodet_m  min =  692.19  max =  693.36  avg =  692.79
    yolo-fastest-1.1  min =  333.62  max =  334.43  avg =  334.00
      yolo-fastestv2  min =  256.41  max =  257.32  avg =  256.83

/root/axera # ./benchncnn 4 4 0 -1 0
          squeezenet  min =  150.38  max =  179.83  avg =  157.90

     squeezenet_int8  min =  106.97  max =  107.43  avg =  107.22
           mobilenet  min =  248.92  max =  273.98  avg =  255.72
      mobilenet_int8  min =  139.49  max =  139.65  avg =  139.60
        mobilenet_v2  min =  174.67  max =  204.35  avg =  182.30
        mobilenet_v3  min =  152.17  max =  152.54  avg =  152.30
          shufflenet  min =   98.74  max =  125.99  avg =  105.74
       shufflenet_v2  min =  103.44  max =  103.88  avg =  103.65
             mnasnet  min =  167.63  max =  197.54  avg =  175.28
     proxylessnasnet  min =  186.02  max =  186.32  avg =  186.15
     efficientnet_b0  min =  284.35  max =  318.17  avg =  292.90
   efficientnetv2_b0  min =  329.56  max =  359.71  avg =  337.22
        regnety_400m  min =  246.91  max =  277.08  avg =  254.71

efficientnet_b0  min =  284.35  max =  318.17  avg =  292.90
           blazeface  min =   30.95  max =   31.31  avg =   31.16
           googlenet  min =  474.87  max =  504.38  avg =  489.43
      googlenet_int8  min =  322.06  max =  331.97  avg =  324.57
            resnet18  min =  440.03  max =  475.28  avg =  456.70
       resnet18_int8  min =  252.01  max =  280.64  avg =  259.22
             alexnet  min =  453.16  max =  478.80  avg =  465.88
            resnet50  min = 1214.70  max = 1252.42  avg = 1229.22
       resnet50_int8  min =  684.53  max =  715.65  avg =  706.14
      squeezenet_ssd  min =  358.84  max =  393.45  avg =  367.77
 squeezenet_ssd_int8  min =  281.56  max =  312.86  avg =  289.85
       mobilenet_ssd  min =  519.11  max =  559.14  avg =  538.41
  mobilenet_ssd_int8  min =  284.58  max =  310.02  avg =  291.02

squeezenet_ssd_int8  min =  281.56  max =  312.86  avg =  289.85
      mobilenet_yolo  min = 1238.87  max = 1284.74  avg = 1260.51
  mobilenetv2_yolov3  min =  624.42  max =  665.81  avg =  642.15
         yolov4-tiny  min =  826.46  max =  852.97  avg =  844.88
           nanodet_m  min =  246.76  max =  279.09  avg =  255.04
    yolo-fastest-1.1  min =  116.12  max =  116.95  avg =  116.50
      yolo-fastestv2  min =   91.08  max =  102.93  avg =   94.41
```

### AMD Ryzen 5700g (Zen3 3.8 GHz ~ 4.6 GHz x 8)
test in wsl2 with ubuntu 20.04
```
$ ./benchncnn  10 1 0 -1 0
          squeezenet  min =    6.53  max =    7.05  avg =    6.77
     squeezenet_int8  min =   17.72  max =   17.86  avg =   17.79
           mobilenet  min =   11.43  max =   11.98  avg =   11.64

squeezenet  min =    6.53  max =    7.05  avg =    6.77
      mobilenet_int8  min =   22.91  max =   24.48  avg =   23.26
        mobilenet_v2  min =    8.28  max =    9.29  avg =    8.66
        mobilenet_v3  min =    6.86  max =    6.98  avg =    6.94
          shufflenet  min =    3.75  max =    4.64  avg =    3.91
       shufflenet_v2  min =    5.08  max =    5.80  avg =    5.22
             mnasnet  min =    7.54  max =    8.60  avg =    7.81
     proxylessnasnet  min =    9.18  max =   10.33  avg =    9.41
     efficientnet_b0  min =   22.57  max =   23.67  avg =   22.93
   efficientnetv2_b0  min =   21.23  max =   22.08  avg =   21.45
        regnety_400m  min =   10.56  max =   10.80  avg =   10.63
           blazeface  min =    1.08  max =    1.17  avg =    1.11
           googlenet  min =   27.91  max =   29.51  avg =   28.28

regnety_400m  min =   10.56  max =   10.80  avg =   10.63
      googlenet_int8  min =   71.00  max =   86.86  avg =   72.74
            resnet18  min =   20.11  max =   20.56  avg =   20.26
       resnet18_int8  min =   63.80  max =   65.13  avg =   64.19
             alexnet  min =   20.64  max =   24.25  avg =   21.65
               vgg16  min =  119.99  max =  125.45  avg =  121.59
          vgg16_int8  min =  268.11  max =  270.41  avg =  269.15
            resnet50  min =   55.42  max =   56.29  avg =   55.70
       resnet50_int8  min =  126.73  max =  132.37  avg =  128.72
      squeezenet_ssd  min =   28.41  max =   30.30  avg =   29.20
 squeezenet_ssd_int8  min =   41.12  max =   42.53  avg =   41.52
       mobilenet_ssd  min =   24.15  max =   24.91  avg =   24.33
  mobilenet_ssd_int8  min =   46.06  max =   59.19  avg =   49.87

squeezenet_ssd_int8  min =   41.12  max =   42.53  avg =   41.52
      mobilenet_yolo  min =   67.58  max =   73.19  avg =   68.99
  mobilenetv2_yolov3  min =   29.44  max =   30.46  avg =   29.78
         yolov4-tiny  min =   41.89  max =   43.47  avg =   42.37
           nanodet_m  min =   11.23  max =   11.47  avg =   11.36
    yolo-fastest-1.1  min =    3.86  max =    4.64  avg =    4.04
      yolo-fastestv2  min =    3.43  max =    3.99  avg =    3.56
  vision_transformer  min = 1590.86  max = 1593.97  avg = 1591.91

$ ./benchncnn  10 16 0 -1 0
          squeezenet  min =    2.94  max =    4.66  avg =    3.31
     squeezenet_int8  min =    3.53  max =    5.26  avg =    3.92
           mobilenet  min =    3.96  max =    5.30  avg =    4.21

squeezenet  min =    2.94  max =    4.66  avg =    3.31
      mobilenet_int8  min =    4.27  max =    4.56  avg =    4.35
        mobilenet_v2  min =    3.63  max =    4.20  avg =    3.82
        mobilenet_v3  min =    3.25  max =    4.79  avg =    3.58
          shufflenet  min =    2.98  max =    3.59  avg =    3.12
       shufflenet_v2  min =    2.62  max =    5.93  avg =    3.04
             mnasnet  min =    3.09  max =    3.49  avg =    3.28
     proxylessnasnet  min =    3.57  max =    4.18  avg =    3.76
     efficientnet_b0  min =    5.98  max =    6.48  avg =    6.18
   efficientnetv2_b0  min =    6.96  max =    7.48  avg =    7.13
        regnety_400m  min =    8.71  max =   11.89  avg =    9.61
           blazeface  min =    0.86  max =    0.96  avg =    0.89
           googlenet  min =   10.75  max =   11.33  avg =   11.00

regnety_400m  min =    8.71  max =   11.89  avg =    9.61
      googlenet_int8  min =   12.75  max =   15.47  avg =   13.50
            resnet18  min =    8.92  max =   16.08  avg =   10.08
       resnet18_int8  min =   10.55  max =   10.99  avg =   10.69
             alexnet  min =    9.95  max =   10.45  avg =   10.17
               vgg16  min =   52.28  max =   53.69  avg =   52.89
          vgg16_int8  min =   44.90  max =   47.90  avg =   45.61
            resnet50  min =   17.80  max =   21.43  avg =   18.66
       resnet50_int8  min =   21.80  max =   25.42  avg =   22.75
      squeezenet_ssd  min =   14.49  max =   16.36  avg =   14.90
 squeezenet_ssd_int8  min =   10.02  max =   10.49  avg =   10.28
       mobilenet_ssd  min =    7.20  max =    7.86  avg =    7.51
  mobilenet_ssd_int8  min =    8.51  max =   10.90  avg =    9.09

squeezenet_ssd_int8  min =   10.02  max =   10.49  avg =   10.28
      mobilenet_yolo  min =   35.67  max =   44.84  avg =   37.33
  mobilenetv2_yolov3  min =   12.72  max =   17.16  avg =   13.67
         yolov4-tiny  min =   20.81  max =   22.11  avg =   21.33
           nanodet_m  min =    5.13  max =   42.12  avg =    9.07
    yolo-fastest-1.1  min =    3.05  max =    4.72  avg =    3.39
      yolo-fastestv2  min =    3.33  max =    3.73  avg =    3.44
  vision_transformer  min =  214.91  max =  229.91  avg =  220.82
```

### Intel Celeron M 420 (Yonah 1.60 GHz x 1)

Tested on `Debian GNU/Linux 11 (bullseye) i686` with `cmake -DNCNN_RUNTIME_CPU=OFF -DNCNN_AVX=OFF -DNCNN_AVX2=OFF -DNCNN_AVX512=OFF-DNCNN_BUILD_TESTS=ON ..`.

```
mouri@Mouri-Laptop-2:~/ncnn/benchmark$ ./../build/benchmark/benchncnn

```
          squeezenet  min =  289.23  max =  301.83  avg =  292.90
     squeezenet_int8  min =  442.82  max =  457.21  avg =  446.89
           mobilenet  min =  549.62  max =  561.20  avg =  554.78
      mobilenet_int8  min =  823.92  max =  837.70  avg =  830.52
        mobilenet_v2  min =  341.72  max =  353.77  avg =  345.34
        mobilenet_v3  min =  267.68  max =  282.08  avg =  273.10
          shufflenet  min =  151.66  max =  153.02  avg =  152.24
       shufflenet_v2  min =  161.54  max =  163.38  avg =  162.13
             mnasnet  min =  322.66  max =  336.91  avg =  326.86
     proxylessnasnet  min =  356.63  max =  368.79  avg =  360.66
     efficientnet_b0  min =  489.92  max =  505.11  avg =  497.32
   efficientnetv2_b0  min =  618.16  max =  632.02  avg =  622.82

proxylessnasnet  min =  356.63  max =  368.79  avg =  360.66
        regnety_400m  min =  414.83  max =  428.42  avg =  419.28
           blazeface  min =   38.56  max =   40.05  avg =   39.05
           googlenet  min = 1022.54  max = 1037.53  avg = 1029.48
      googlenet_int8  min = 1493.35  max = 1495.46  avg = 1494.31
            resnet18  min =  803.32  max =  818.27  avg =  812.49
       resnet18_int8  min = 1188.26  max = 1200.88  avg = 1192.56
             alexnet  min =  613.78  max =  623.88  avg =  619.99
               vgg16  min = 4465.44  max = 4478.12  avg = 4474.16
          vgg16_int8  min = 6042.40  max = 6114.37  avg = 6077.07
            resnet50  min = 2517.75  max = 2528.42  avg = 2522.83
       resnet50_int8  min = 3746.28  max = 3771.09  avg = 3756.88
      squeezenet_ssd  min =  585.56  max =  636.01  avg =  602.62

resnet50  min = 2517.75  max = 2528.42  avg = 2522.83
 squeezenet_ssd_int8  min =  822.43  max =  968.77  avg =  862.33
       mobilenet_ssd  min = 1116.98  max = 1139.17  avg = 1127.65
  mobilenet_ssd_int8  min = 1665.03  max = 1670.55  avg = 1668.37
      mobilenet_yolo  min = 2638.61  max = 2666.54  avg = 2652.26
  mobilenetv2_yolov3  min = 1248.56  max = 1255.98  avg = 1251.22
         yolov4-tiny  min = 1507.31  max = 1525.56  avg = 1514.66
           nanodet_m  min =  386.41  max =  400.63  avg =  391.21
    yolo-fastest-1.1  min =  159.97  max =  164.53  avg =  161.41
      yolo-fastestv2  min =  134.29  max =  135.47  avg =  134.70
  vision_transformer  min = 22201.32  max = 22510.75  avg = 22315.09
          FastestDet  min =  146.94  max =  148.50  avg =  147.44
```
### VisionFive2 , JH7110 (SiFive-U74(RV64GC) 1.5GHz x 4) riscv64

FastestDet  min =  146.94  max =  148.50  avg =  147.44
```
Test on Deepin V23
```
deepin@deepin-riscv:~/ncnn/benchmark$ ./benchncnn 10 4 0 -1 0
          squeezenet  min =  152.80  max =  295.81  avg =  205.79
     squeezenet_int8  min = 1407.77  max = 1490.49  avg = 1444.99
           mobilenet  min =  262.57  max =  337.06  avg =  296.57
      mobilenet_int8  min = 2166.73  max = 2294.66  avg = 2217.37
        mobilenet_v2  min =  179.47  max =  262.96  avg =  201.42
        mobilenet_v3  min =  169.87  max =  290.39  avg =  188.40
          shufflenet  min =   93.05  max =  150.10  avg =  103.12
       shufflenet_v2  min =   86.26  max =  140.20  avg =   95.41
             mnasnet  min =  194.68  max =  276.12  avg =  227.29
     proxylessnasnet  min =  231.28  max =  319.04  avg =  256.31

shufflenet_v2  min =   86.26  max =  140.20  avg =   95.41
     efficientnet_b0  min =  356.40  max =  455.36  avg =  398.40
   efficientnetv2_b0  min =  350.56  max =  464.91  avg =  390.02
        regnety_400m  min =  232.63  max =  319.92  avg =  262.74
           blazeface  min =   26.10  max =   26.79  avg =   26.39
           googlenet  min =  529.74  max =  610.94  avg =  579.42
      googlenet_int8  min = 4142.42  max = 4287.84  avg = 4209.89
            resnet18  min =  412.50  max =  503.50  avg =  455.87
       resnet18_int8  min = 4322.51  max = 4477.35  avg = 4398.36
             alexnet  min =  303.05  max =  420.22  avg =  344.30
               vgg16  min = 2598.99  max = 2714.96  avg = 2658.85
          vgg16_int8  min = 39161.25  max = 39798.05  avg = 39502.86
            resnet50  min = 1326.47  max = 1395.62  avg = 1353.32

vgg16  min = 2598.99  max = 2714.96  avg = 2658.85
       resnet50_int8  min = 12792.62  max = 13524.54  avg = 13000.57
      squeezenet_ssd  min =  362.38  max =  444.80  avg =  398.51
 squeezenet_ssd_int8  min = 3049.12  max = 3205.56  avg = 3110.10
       mobilenet_ssd  min =  544.07  max =  666.15  avg =  598.36
  mobilenet_ssd_int8  min = 4853.38  max = 5043.51  avg = 4961.78
      mobilenet_yolo  min = 1485.06  max = 1638.32  avg = 1536.13
  mobilenetv2_yolov3  min =  661.66  max =  745.29  avg =  701.72
         yolov4-tiny  min =  861.67  max =  952.40  avg =  910.70
           nanodet_m  min =  213.38  max =  294.39  avg =  241.32
    yolo-fastest-1.1  min =  105.56  max =  188.51  avg =  147.26
      yolo-fastestv2  min =   91.16  max =  184.20  avg =  131.51
  vision_transformer  min = 15491.20  max = 15800.66  avg = 15714.61

yolo-fastest-1.1  min =  105.56  max =  188.51  avg =  147.26
          FastestDet  min =  109.37  max =  197.13  avg =  140.97
deepin@deepin-riscv:~/ncnn/benchmark$ ./benchncnn 10 4 1 -1 0
          squeezenet  min =  176.87  max =  381.36  avg =  230.00
     squeezenet_int8  min = 1371.59  max = 1422.99  avg = 1397.98
           mobilenet  min =  258.24  max =  316.70  avg =  278.97
      mobilenet_int8  min = 2078.32  max = 2151.96  avg = 2107.99
        mobilenet_v2  min =  177.17  max =  234.22  avg =  186.59
        mobilenet_v3  min =  168.33  max =  246.14  avg =  186.58
          shufflenet  min =   91.85  max =  122.44  avg =   99.00
       shufflenet_v2  min =   84.77  max =  142.90  avg =   91.09

mobilenet_v3  min =  168.33  max =  246.14  avg =  186.58
             mnasnet  min =  191.80  max =  246.91  avg =  206.91
     proxylessnasnet  min =  228.68  max =  290.25  avg =  242.97
     efficientnet_b0  min =  346.73  max =  446.28  avg =  380.96
   efficientnetv2_b0  min =  347.51  max =  563.90  avg =  430.75
        regnety_400m  min =  230.00  max =  313.97  avg =  248.80
           blazeface  min =   25.92  max =   57.47  avg =   31.94
           googlenet  min =  516.39  max =  578.23  avg =  545.93
      googlenet_int8  min = 3964.04  max = 4514.07  avg = 4104.93
            resnet18  min =  409.70  max =  471.53  avg =  437.14
       resnet18_int8  min = 4190.57  max = 4376.97  avg = 4232.26
             alexnet  min =  309.28  max =  359.44  avg =  328.25
               vgg16  min = 2520.93  max = 2728.60  avg = 2585.15

resnet18_int8  min = 4190.57  max = 4376.97  avg = 4232.26
          vgg16_int8  min = 37396.47  max = 39027.31  avg = 37782.73
            resnet50  min = 1260.35  max = 1338.68  avg = 1280.79
       resnet50_int8  min = 12158.23  max = 12272.77  avg = 12221.79
      squeezenet_ssd  min =  360.82  max =  429.28  avg =  378.91
 squeezenet_ssd_int8  min = 2922.44  max = 3073.98  avg = 2988.48
       mobilenet_ssd  min =  533.02  max =  605.98  avg =  561.60
  mobilenet_ssd_int8  min = 4638.71  max = 5526.65  avg = 4856.56
      mobilenet_yolo  min = 1414.40  max = 1499.87  avg = 1447.08
  mobilenetv2_yolov3  min =  636.87  max =  709.35  avg =  668.95
         yolov4-tiny  min =  842.87  max =  888.76  avg =  867.95
           nanodet_m  min =  212.61  max =  282.76  avg =  231.15
    yolo-fastest-1.1  min =  105.64  max =  209.95  avg =  140.39

yolov4-tiny  min =  842.87  max =  888.76  avg =  867.95
      yolo-fastestv2  min =   91.57  max =  199.97  avg =  127.63
  vision_transformer  min = 14926.32  max = 15952.46  avg = 15278.62
          FastestDet  min =  109.82  max =  181.18  avg =  129.30
```
### T-Head TH1520 (C910V, 1.848 GHz x 4 + BXM-4-64 PowerVR)

Tested on `Linux anolis-riscv 5.10.112-00579-g8e3db308d5a5 #23 SMP PREEMPT Fri Aug 12 10:17:32 CST 2022 riscv64 riscv64 riscv64 GNU/Linux`

```
[root@anolis-riscv benchmark]# ./benchncnn
syscall error -1
          squeezenet  min =  187.88  max =  188.82  avg =  188.13
     squeezenet_int8  min = 2388.26  max = 2446.92  avg = 2411.46
           mobilenet  min =  321.46  max =  323.34  avg =  322.19

squeezenet  min =  187.88  max =  188.82  avg =  188.13
      mobilenet_int8  min = 2318.93  max = 2458.55  avg = 2400.99
        mobilenet_v2  min =  214.01  max =  216.00  avg =  215.35
        mobilenet_v3  min =  247.71  max =  248.18  avg =  247.96
          shufflenet  min =  155.58  max =  155.85  avg =  155.67
       shufflenet_v2  min =   99.50  max =   99.75  avg =   99.63
             mnasnet  min =  261.46  max =  263.83  avg =  262.53
     proxylessnasnet  min =  315.40  max =  316.89  avg =  316.28
     efficientnet_b0  min =  484.97  max =  486.16  avg =  485.55
   efficientnetv2_b0  min =  453.03  max =  453.40  avg =  453.21
        regnety_400m  min =  314.09  max =  315.33  avg =  314.77
           blazeface  min =   46.14  max =   46.69  avg =   46.39
           googlenet  min =  650.99  max =  653.60  avg =  651.69

regnety_400m  min =  314.09  max =  315.33  avg =  314.77
      googlenet_int8  min = 5435.11  max = 6391.98  avg = 6012.81
            resnet18  min =  505.48  max =  506.70  avg =  506.06
       resnet18_int8  min = 5053.33  max = 6599.94  avg = 6001.86
             alexnet  min =  403.68  max =  404.60  avg =  404.23
               vgg16  min = 2731.55  max = 2746.48  avg = 2738.82
```

test on `Beaglev-ahead(Linux ahead 5.10.113-ahead #2023.08.02.13.12+2c2096a98 SMP PREEMPT Wed Aug 2 13:13:02 UTC 2 riscv64 GNU/Linux)`

```
debian@ahead:~/ncnn/build/benchmark$ sudo ./benchncnn 10 1 0 0 0
[0 PowerVR B-Series BXM-4-64]  queueC=0[2]  queueG=0[2]  queueT=0[2]
[0 PowerVR B-Series BXM-4-64]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=0
[0 PowerVR B-Series BXM-4-64]  fp16-p/s/a=1/1/1  int8-p/s/a=1/1/1

[0 PowerVR B-Series BXM-4-64]  subgroup=1  basic/vote/ballot/shuffle=1/1/1/1
[0 PowerVR B-Series BXM-4-64]  fp16-matrix-16_8_8/16_8_16/16_16_16=0/0/0
          squeezenet  min =  287.88  max =  296.84  avg =  295.68
     squeezenet_int8  min = 2289.46  max = 2320.97  avg = 2306.60
           mobilenet  min =  584.32  max =  588.48  avg =  587.41
      mobilenet_int8  min = 2487.91  max = 2492.12  avg = 2489.64
        mobilenet_v2  min =  380.02  max =  386.67  avg =  385.75
        mobilenet_v3  min =  314.73  max =  328.84  avg =  325.76
          shufflenet  min =  146.96  max =  158.29  avg =  156.38
       shufflenet_v2  min =  203.94  max =  211.77  avg =  210.82
             mnasnet  min =  395.80  max =  404.95  avg =  403.80

shufflenet  min =  146.96  max =  158.29  avg =  156.38
     proxylessnasnet  min =  447.74  max =  456.89  avg =  454.87
     efficientnet_b0  min =  532.23  max =  543.05  avg =  538.53
   efficientnetv2_b0  min =  659.43  max =  681.64  avg =  669.13
        regnety_400m  min =  393.16  max =  407.27  avg =  403.81
           blazeface  min =   50.41  max =   61.83  avg =   56.92
           googlenet  min =  890.79  max =  898.09  avg =  896.25
      googlenet_int8  min = 4713.76  max = 5296.61  avg = 5044.39
            resnet18  min =  814.16  max =  824.53  avg =  820.35
       resnet18_int8  min = 4800.73  max = 6015.34  avg = 5765.47
             alexnet  min =  453.80  max =  465.51  avg =  462.11
               vgg16  min = 4016.26  max = 4027.30  avg = 4021.94
          vgg16_int8  min = 55069.69  max = 64814.86  avg = 59096.20

alexnet  min =  453.80  max =  465.51  avg =  462.11
            resnet50  min = 2494.42  max = 2502.38  avg = 2500.28
       resnet50_int8  min = 15366.90  max = 17179.36  avg = 16701.20
      squeezenet_ssd  min =  724.36  max =  738.28  avg =  730.44
 squeezenet_ssd_int8  min = 4550.62  max = 5235.87  avg = 4684.19
       mobilenet_ssd  min = 1207.04  max = 1218.80  avg = 1212.86
  mobilenet_ssd_int8  min = 6019.61  max = 6349.35  avg = 6184.49
      mobilenet_yolo  min = 2736.28  max = 2747.06  avg = 2743.21
  mobilenetv2_yolov3  min = 1339.16  max = 1349.46  avg = 1344.81
         yolov4-tiny  min = 1457.05  max = 1459.04  avg = 1457.81
           nanodet_m  min =  443.40  max =  444.58  avg =  444.00
    yolo-fastest-1.1  min =  240.39  max =  248.05  avg =  247.04
      yolo-fastestv2  min =  162.71  max =  173.30  avg =  169.39

nanodet_m  min =  443.40  max =  444.58  avg =  444.00
  vision_transformer  min = 17148.14  max = 17250.66  avg = 17202.60
          FastestDet  min =  199.71  max =  200.38  avg =  199.90
```

### Rockchip RK3588 (Quad Core A76 2.4GHz + Quad Core A55 1.8GHz)
test in ROCK5 MODEL B

```
rock@rock-5b:~/ncnn/build/benchmark$ ./benchncnn  10 1 0 -1 0
          squeezenet  min =   15.22  max =   16.03  avg =   15.70
     squeezenet_int8  min =   16.77  max =   16.96  avg =   16.86
           mobilenet  min =   23.07  max =   23.58  avg =   23.36
      mobilenet_int8  min =   18.58  max =   18.90  avg =   18.72
        mobilenet_v2  min =   18.74  max =   19.10  avg =   18.96
        mobilenet_v3  min =   14.40  max =   14.65  avg =   14.50

mobilenet_int8  min =   18.58  max =   18.90  avg =   18.72
          shufflenet  min =    9.74  max =    9.88  avg =    9.84
       shufflenet_v2  min =    9.44  max =    9.55  avg =    9.50
             mnasnet  min =   14.73  max =   15.03  avg =   14.87
     proxylessnasnet  min =   18.37  max =   18.59  avg =   18.46
     efficientnet_b0  min =   29.11  max =   30.18  avg =   29.63
   efficientnetv2_b0  min =   46.40  max =   46.95  avg =   46.76
        regnety_400m  min =   19.18  max =   19.39  avg =   19.28
           blazeface  min =    5.16  max =    5.23  avg =    5.20
           googlenet  min =   64.64  max =   65.33  avg =   65.00
      googlenet_int8  min =   61.86  max =   63.41  avg =   62.42
            resnet18  min =   42.00  max =   43.34  avg =   42.48
       resnet18_int8  min =   67.22  max =   67.80  avg =   67.45

googlenet_int8  min =   61.86  max =   63.41  avg =   62.42
             alexnet  min =   57.65  max =   58.21  avg =   58.01
               vgg16  min =  192.35  max =  193.36  avg =  192.84
          vgg16_int8  min =  570.86  max =  578.81  avg =  574.50
            resnet50  min =  107.86  max =  109.52  avg =  108.70
       resnet50_int8  min =  134.41  max =  135.86  avg =  135.18
      squeezenet_ssd  min =   40.85  max =   41.24  avg =   41.02
 squeezenet_ssd_int8  min =   52.23  max =   53.70  avg =   52.54
       mobilenet_ssd  min =   45.11  max =   45.50  avg =   45.32
  mobilenet_ssd_int8  min =   36.53  max =   36.63  avg =   36.59
      mobilenet_yolo  min =   95.18  max =   96.79  avg =   95.90
  mobilenetv2_yolov3  min =   65.50  max =   65.88  avg =   65.72
         yolov4-tiny  min =   86.13  max =   88.84  avg =   87.29

mobilenet_yolo  min =   95.18  max =   96.79  avg =   95.90
           nanodet_m  min =   22.57  max =   22.87  avg =   22.74
    yolo-fastest-1.1  min =    9.23  max =    9.35  avg =    9.29
      yolo-fastestv2  min =    8.62  max =    8.83  avg =    8.73
  vision_transformer  min = 3077.54  max = 3396.13  avg = 3339.58
          FastestDet  min =    9.11  max =    9.30  avg =    9.20

rock@rock-5b:~/ncnn/build/benchmark$ ./benchncnn  10 8 0 -1 0
          squeezenet  min =   10.02  max =   11.01  avg =   10.43
     squeezenet_int8  min =   11.78  max =   13.77  avg =   12.55
           mobilenet  min =   12.75  max =   13.58  avg =   13.12
      mobilenet_int8  min =   12.23  max =   14.29  avg =   13.54
        mobilenet_v2  min =   12.76  max =   14.27  avg =   13.40

mobilenet  min =   12.75  max =   13.58  avg =   13.12
        mobilenet_v3  min =    9.51  max =    9.81  avg =    9.71
          shufflenet  min =    7.06  max =    7.23  avg =    7.13
       shufflenet_v2  min =    6.21  max =    7.32  avg =    6.38
             mnasnet  min =    9.32  max =   12.49  avg =   10.75
     proxylessnasnet  min =   13.79  max =   15.51  avg =   14.70
     efficientnet_b0  min =   16.59  max =   17.99  avg =   17.08
   efficientnetv2_b0  min =   28.26  max =   32.26  avg =   30.52
        regnety_400m  min =   13.43  max =   15.00  avg =   13.72
           blazeface  min =    3.87  max =    7.38  avg =    5.65
           googlenet  min =   29.18  max =   44.00  avg =   36.31
      googlenet_int8  min =   31.14  max =   37.48  avg =   34.58
            resnet18  min =   21.47  max =   24.40  avg =   22.35

googlenet  min =   29.18  max =   44.00  avg =   36.31
       resnet18_int8  min =   26.68  max =   29.89  avg =   28.45
             alexnet  min =   29.35  max =   38.09  avg =   31.65
               vgg16  min =  112.37  max =  122.94  avg =  117.05
          vgg16_int8  min =  161.08  max =  215.29  avg =  176.89
            resnet50  min =   54.54  max =   57.50  avg =   55.71
       resnet50_int8  min =   54.76  max =   65.05  avg =   60.59
      squeezenet_ssd  min =   26.21  max =   35.05  avg =   30.76
 squeezenet_ssd_int8  min =   33.34  max =   40.88  avg =   36.19
       mobilenet_ssd  min =   26.71  max =   28.85  avg =   27.88
  mobilenet_ssd_int8  min =   22.03  max =   25.31  avg =   24.21
      mobilenet_yolo  min =   60.51  max =   74.65  avg =   65.45
  mobilenetv2_yolov3  min =   37.27  max =   44.13  avg =   41.20

mobilenet_ssd_int8  min =   22.03  max =   25.31  avg =   24.21
         yolov4-tiny  min =   49.84  max =   58.12  avg =   53.93
           nanodet_m  min =   16.54  max =   22.41  avg =   20.60
    yolo-fastest-1.1  min =    8.49  max =   13.50  avg =    9.91
      yolo-fastestv2  min =    6.28  max =   11.22  avg =    8.00
  vision_transformer  min =  968.62  max = 1063.47  avg = 1019.12
          FastestDet  min =    6.14  max =   11.92  avg =    7.85

rock@rock-5b:~/ncnn/build/benchmark$ ./benchncnn 10 4 2 -1 0
          squeezenet  min =    6.78  max =    7.27  avg =    7.07
     squeezenet_int8  min =    4.58  max =    4.73  avg =    4.63
           mobilenet  min =    5.67  max =    5.78  avg =    5.72
      mobilenet_int8  min =    5.01  max =    5.20  avg =    5.15

squeezenet_int8  min =    4.58  max =    4.73  avg =    4.63
        mobilenet_v2  min =    5.44  max =    5.76  avg =    5.50
        mobilenet_v3  min =    4.67  max =    5.03  avg =    4.74
          shufflenet  min =    4.22  max =    4.30  avg =    4.27
       shufflenet_v2  min =    3.48  max =    3.60  avg =    3.53
             mnasnet  min =    4.52  max =    4.83  avg =    4.61
     proxylessnasnet  min =    5.44  max =    6.01  avg =    5.56
     efficientnet_b0  min =    8.33  max =    8.52  avg =    8.41
   efficientnetv2_b0  min =   12.95  max =   13.08  avg =   13.02
        regnety_400m  min =    8.60  max =    8.73  avg =    8.66
           blazeface  min =    1.86  max =    1.95  avg =    1.90
           googlenet  min =   16.58  max =   16.85  avg =   16.65
      googlenet_int8  min =   16.99  max =   17.13  avg =   17.06

blazeface  min =    1.86  max =    1.95  avg =    1.90
            resnet18  min =   14.98  max =   15.30  avg =   15.08
       resnet18_int8  min =   20.10  max =   20.22  avg =   20.15
             alexnet  min =   19.78  max =   20.21  avg =   19.87
               vgg16  min =   66.35  max =   94.16  avg =   75.24
          vgg16_int8  min =  131.02  max =  131.98  avg =  131.51
            resnet50  min =   28.07  max =   28.78  avg =   28.28
       resnet50_int8  min =   33.56  max =   35.53  avg =   33.84
      squeezenet_ssd  min =   16.40  max =   16.80  avg =   16.49
 squeezenet_ssd_int8  min =   18.64  max =   19.00  avg =   18.76
       mobilenet_ssd  min =   13.66  max =   13.78  avg =   13.72
  mobilenet_ssd_int8  min =   11.23  max =   11.42  avg =   11.33
      mobilenet_yolo  min =   30.76  max =   31.03  avg =   30.86

mobilenet_ssd  min =   13.66  max =   13.78  avg =   13.72
  mobilenetv2_yolov3  min =   19.28  max =   21.07  avg =   20.30
         yolov4-tiny  min =   33.44  max =   37.68  avg =   34.70
           nanodet_m  min =    8.28  max =    8.55  avg =    8.38
    yolo-fastest-1.1  min =    4.30  max =    4.40  avg =    4.34
      yolo-fastestv2  min =    4.07  max =    4.18  avg =    4.13
  vision_transformer  min =  815.67  max =  819.27  avg =  817.49
          FastestDet  min =    4.34  max =    7.47  avg =    5.18
```

### AWS c5.4xlarge Instance (Intel Xeon Platinum 8124M @ 3.399GHz, Ubuntu 20.04.6 LTS x86_64)

icpc (ICC) 2021.9.0 20230302

```
root@ip-172-31-3-216:/opt/ncnn-icc/benchmark# ../build/benchmark/benchncnn 4 8 0 -1

```
          squeezenet  min =    3.23  max =    3.25  avg =    3.24
     squeezenet_int8  min =    5.97  max =    6.00  avg =    5.98
           mobilenet  min =    3.44  max =    3.56  avg =    3.51
      mobilenet_int8  min =    4.53  max =    4.63  avg =    4.59
        mobilenet_v2  min =    3.82  max =    3.95  avg =    3.88
        mobilenet_v3  min =    3.49  max =    3.53  avg =    3.51
          shufflenet  min =    3.56  max =    3.58  avg =    3.57
       shufflenet_v2  min =    3.47  max =    3.52  avg =    3.48
             mnasnet  min =    3.69  max =    4.07  avg =    3.94
     proxylessnasnet  min =    3.75  max =    3.81  avg =    3.78
     efficientnet_b0  min =    4.80  max =    4.85  avg =    4.82
   efficientnetv2_b0  min =    6.51  max =    6.65  avg =    6.59

proxylessnasnet  min =    3.75  max =    3.81  avg =    3.78
        regnety_400m  min =    8.20  max =   10.79  avg =    9.04
           blazeface  min =    1.16  max =    1.19  avg =    1.18
           googlenet  min =    9.92  max =   11.07  avg =   10.56
      googlenet_int8  min =   21.50  max =   21.67  avg =   21.59
            resnet18  min =    6.47  max =    6.57  avg =    6.52
       resnet18_int8  min =   19.01  max =   19.17  avg =   19.05
             alexnet  min =    5.20  max =    5.28  avg =    5.25
               vgg16  min =   35.74  max =   35.88  avg =   35.79
          vgg16_int8  min =   38.20  max =   38.51  avg =   38.36
            resnet50  min =   14.02  max =   14.12  avg =   14.05
       resnet50_int8  min =   27.92  max =   28.12  avg =   28.03
      squeezenet_ssd  min =    9.51  max =    9.70  avg =    9.60

resnet50  min =   14.02  max =   14.12  avg =   14.05
 squeezenet_ssd_int8  min =   12.91  max =   13.06  avg =   12.97
       mobilenet_ssd  min =    6.55  max =    6.65  avg =    6.59
  mobilenet_ssd_int8  min =    9.16  max =    9.23  avg =    9.20
      mobilenet_yolo  min =   17.02  max =   17.22  avg =   17.13
  mobilenetv2_yolov3  min =   12.67  max =   12.78  avg =   12.71
         yolov4-tiny  min =   23.42  max =   23.49  avg =   23.46
           nanodet_m  min =    7.27  max =    7.30  avg =    7.28
    yolo-fastest-1.1  min =    4.05  max =    4.08  avg =    4.06
      yolo-fastestv2  min =    4.12  max =    4.15  avg =    4.13
  vision_transformer  min =  135.25  max =  136.36  avg =  135.71
          FastestDet  min =    4.12  max =    4.21  avg =    4.16
```

Intel(R) oneAPI DPC++/C++ Compiler 2023.1.0 (2023.1.0.20230320)

```

FastestDet  min =    4.12  max =    4.21  avg =    4.16
```

```
root@ip-172-31-3-216:/opt/ncnn-icx/benchmark# ../build/benchmark/benchncnn 4 8 0 -1
          squeezenet  min =    3.13  max =    3.18  avg =    3.16
     squeezenet_int8  min =    4.04  max =    4.07  avg =    4.06
           mobilenet  min =    2.99  max =    3.05  avg =    3.03
      mobilenet_int8  min =    3.68  max =    3.77  avg =    3.73
        mobilenet_v2  min =    3.88  max =    3.92  avg =    3.89
        mobilenet_v3  min =    3.60  max =    3.70  avg =    3.64
          shufflenet  min =    3.52  max =    3.54  avg =    3.53
       shufflenet_v2  min =    3.61  max =    3.64  avg =    3.63
             mnasnet  min =    3.51  max =    3.53  avg =    3.52
     proxylessnasnet  min =    3.73  max =    3.78  avg =    3.75

shufflenet_v2  min =    3.61  max =    3.64  avg =    3.63
     efficientnet_b0  min =    4.86  max =    4.95  avg =    4.91
   efficientnetv2_b0  min =    6.84  max =    6.97  avg =    6.91
        regnety_400m  min =    7.83  max =    7.89  avg =    7.86
           blazeface  min =    1.10  max =    1.13  avg =    1.11
           googlenet  min =    9.80  max =    9.89  avg =    9.83
      googlenet_int8  min =   11.32  max =   11.42  avg =   11.37
            resnet18  min =    6.68  max =    6.74  avg =    6.72
       resnet18_int8  min =    8.86  max =    8.92  avg =    8.90
             alexnet  min =    5.21  max =    5.25  avg =    5.22
               vgg16  min =   35.77  max =   35.92  avg =   35.88
          vgg16_int8  min =   29.64  max =   29.79  avg =   29.75
            resnet50  min =   14.11  max =   14.31  avg =   14.22

vgg16  min =   35.77  max =   35.92  avg =   35.88
       resnet50_int8  min =   17.73  max =   18.01  avg =   17.86
      squeezenet_ssd  min =    9.57  max =    9.65  avg =    9.61
 squeezenet_ssd_int8  min =    9.57  max =    9.67  avg =    9.63
       mobilenet_ssd  min =    6.56  max =    6.61  avg =    6.59
  mobilenet_ssd_int8  min =    7.51  max =    7.72  avg =    7.58
      mobilenet_yolo  min =   16.89  max =   17.08  avg =   17.00
  mobilenetv2_yolov3  min =   13.79  max =   13.93  avg =   13.86
         yolov4-tiny  min =   24.91  max =   25.08  avg =   24.98
           nanodet_m  min =    7.42  max =    7.47  avg =    7.44
    yolo-fastest-1.1  min =    4.02  max =    4.09  avg =    4.07
      yolo-fastestv2  min =    4.02  max =    4.04  avg =    4.03
  vision_transformer  min =  135.54  max =  136.68  avg =  136.21

yolo-fastest-1.1  min =    4.02  max =    4.09  avg =    4.07
          FastestDet  min =    4.06  max =    4.10  avg =    4.08
```

### Hyper-V Linux Guest with GPU-PV enabled (Intel Core i7-11800H, NVIDIA GeForce RTX 3070 Laptop GPU)

- Host OS: Microsoft Windows 11 Enterprise (10.0.22621.1635)
- Guest OS: openSUSE Tumbleweed x86_64 20230507
- Mesa 3D source tree: https://gitlab.freedesktop.org/mesa/mesa/-/tree/ce6430067613e3e64cabf79918a3d96122b0c4c4
- Mesa 3D configuration command
  > meson --prefix="${PWD}/build/install" -D gallium-drivers=swrast,d3d12 -D vulkan-drivers=swrast,microsoft-experimental build/
- ncnn configuration command
  > cmake -DNCNN_VULKAN=ON -DNCNN_BUILD_TESTS=ON ..

```

```
mouri@MouriVM-openSUSE:~/Workspace/ncnn/benchmark> VK_ICD_FILENAMES=/home/mouri/Workspace/mesa/build/install/share/vulkan/icd.d/dzn_icd.x86_64.json ./../build/benchmark/benchncnn 10 1 0 0 0
WARNING: dzn is not a conformant Vulkan implementation, testing use only.
[0 Microsoft Direct3D12 (NVIDIA GeForce RTX 3070 Laptop GPU)]  queueC=1[8]  queueG=0[4]  queueT=2[1]
[0 Microsoft Direct3D12 (NVIDIA GeForce RTX 3070 Laptop GPU)]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=0
[0 Microsoft Direct3D12 (NVIDIA GeForce RTX 3070 Laptop GPU)]  fp16-p/s/a=1/1/1  int8-p/s/a=1/0/0
[0 Microsoft Direct3D12 (NVIDIA GeForce RTX 3070 Laptop GPU)]  subgroup=32  basic=1  vote=1  ballot=1  shuffle=1
[1 Microsoft Direct3D12 (Intel(R) UHD Graphics)]  queueC=1[8]  queueG=0[4]  queueT=2[1]

[1 Microsoft Direct3D12 (Intel(R) UHD Graphics)]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=0
[1 Microsoft Direct3D12 (Intel(R) UHD Graphics)]  fp16-p/s/a=1/1/1  int8-p/s/a=1/0/0
[1 Microsoft Direct3D12 (Intel(R) UHD Graphics)]  subgroup=16  basic=1  vote=1  ballot=1  shuffle=1
          squeezenet  min =   52.30  max =   65.51  avg =   56.65
     squeezenet_int8  min =   14.53  max =   15.55  avg =   14.88
           mobilenet  min =   37.42  max =   52.07  avg =   42.48
      mobilenet_int8  min =   19.01  max =   19.82  avg =   19.46
        mobilenet_v2  min =   55.34  max =   73.39  avg =   63.94
        mobilenet_v3  min =   97.02  max =  123.14  avg =  109.90

mobilenet_int8  min =   19.01  max =   19.82  avg =   19.46
          shufflenet  min =   72.75  max =  100.26  avg =   88.26
       shufflenet_v2  min =   93.34  max =  119.64  avg =  105.76
             mnasnet  min =   63.49  max =   74.11  avg =   69.05
     proxylessnasnet  min =   65.87  max =   83.87  avg =   76.33
     efficientnet_b0  min =  162.86  max =  210.51  avg =  184.03
   efficientnetv2_b0  min =  200.88  max =  220.40  avg =  210.85
        regnety_400m  min =  106.92  max =  134.68  avg =  123.04
           blazeface  min =   58.64  max =   66.50  avg =   60.54
           googlenet  min =  117.34  max =  145.28  avg =  134.84
      googlenet_int8  min =   62.50  max =   65.07  avg =   63.44
            resnet18  min =   67.30  max =   92.40  avg =   80.23
       resnet18_int8  min =   56.09  max =   58.40  avg =   56.97

googlenet_int8  min =   62.50  max =   65.07  avg =   63.44
             alexnet  min =   29.94  max =   47.51  avg =   38.83
               vgg16  min =   59.72  max =   73.08  avg =   65.46
          vgg16_int8  min =  136.35  max =  148.39  avg =  143.96
            resnet50  min =  115.92  max =  152.34  avg =  129.64
       resnet50_int8  min =   93.86  max =  101.51  avg =   97.96
      squeezenet_ssd  min =  139.82  max =  149.15  avg =  144.78
 squeezenet_ssd_int8  min =   32.09  max =   35.96  avg =   33.41
       mobilenet_ssd  min =   88.14  max =  102.62  avg =   97.79
  mobilenet_ssd_int8  min =   33.93  max =   36.42  avg =   34.41
      mobilenet_yolo  min =   52.22  max =   65.25  avg =   58.81
  mobilenetv2_yolov3  min =   75.09  max =   94.12  avg =   85.23
         yolov4-tiny  min =   73.27  max =   88.69  avg =   81.44

mobilenet_yolo  min =   52.22  max =   65.25  avg =   58.81
           nanodet_m  min =  110.98  max =  150.70  avg =  127.60
    yolo-fastest-1.1  min =  104.72  max =  135.40  avg =  116.92
      yolo-fastestv2  min =  113.84  max =  142.19  avg =  128.24
  vision_transformer  min =  412.19  max =  474.25  avg =  444.15
          FastestDet  min =   96.31  max =  131.51  avg =  117.27
mouri@MouriVM-openSUSE:~/Workspace/ncnn/benchmark> VK_ICD_FILENAMES=/home/mouri/Workspace/mesa/build/install/share/vulkan/icd.d/dzn_icd.x86_64.json ./../build/benchmark/benchncnn 10 1 0 1 0

          squeezenet  min =   36.86  max =   62.04  avg =   44.48
     squeezenet_int8  min =   15.31  max =   16.14  avg =   15.63
           mobilenet  min =   30.79  max =   34.67  avg =   32.95
      mobilenet_int8  min =   19.23  max =   19.72  avg =   19.42
        mobilenet_v2  min =   36.56  max =   40.53  avg =   38.20
        mobilenet_v3  min =   52.11  max =   61.72  avg =   56.58
          shufflenet  min =   41.50  max =   74.61  avg =   49.24
       shufflenet_v2  min =   44.49  max =   52.30  avg =   49.04
             mnasnet  min =   35.66  max =   43.45  avg =   37.98
     proxylessnasnet  min =   41.27  max =   47.63  avg =   43.63
     efficientnet_b0  min =   67.66  max =   80.88  avg =   73.64
   efficientnetv2_b0  min =  111.10  max =  156.52  avg =  126.70

proxylessnasnet  min =   41.27  max =   47.63  avg =   43.63
        regnety_400m  min =   62.66  max =   89.16  avg =   68.99
           blazeface  min =   24.86  max =   33.52  avg =   26.91
           googlenet  min =   70.55  max =   84.22  avg =   75.19
      googlenet_int8  min =   58.78  max =   64.81  avg =   62.99
            resnet18  min =   44.17  max =   49.37  avg =   46.73
       resnet18_int8  min =   59.99  max =   66.91  avg =   62.35
             alexnet  min =   41.54  max =   57.16  avg =   44.30
               vgg16  min =  138.74  max =  165.03  avg =  146.90
          vgg16_int8  min =  135.36  max =  165.89  avg =  142.61
            resnet50  min =   97.46  max =  107.18  avg =  100.89
       resnet50_int8  min =   92.90  max =  100.45  avg =   95.91
      squeezenet_ssd  min =   72.27  max =   90.71  avg =   76.09

resnet50  min =   97.46  max =  107.18  avg =  100.89
 squeezenet_ssd_int8  min =   34.66  max =   40.46  avg =   36.58
       mobilenet_ssd  min =   59.90  max =   68.74  avg =   62.40
  mobilenet_ssd_int8  min =   37.02  max =   38.59  avg =   37.82
      mobilenet_yolo  min =   73.19  max =   80.40  avg =   76.42
  mobilenetv2_yolov3  min =   58.56  max =   66.71  avg =   62.02
         yolov4-tiny  min =   63.75  max =   84.29  avg =   69.54
           nanodet_m  min =   54.66  max =   67.89  avg =   60.82
    yolo-fastest-1.1  min =   40.89  max =   51.03  avg =   43.15
      yolo-fastestv2  min =   50.43  max =   77.46  avg =   60.66
  vision_transformer  min = 1330.82  max = 1388.73  avg = 1354.10
          FastestDet  min =   85.75  max =  112.67  avg =   98.62

yolo-fastestv2  min =   50.43  max =   77.46  avg =   60.66
mouri@MouriVM-openSUSE:~/Workspace/ncnn/benchmark> VK_ICD_FILENAMES=/home/mouri/Workspace/mesa/build/install/share/vulkan/icd.d/dzn_icd.x86_64.json ./../build/benchmark/benchncnn 10 1 0 -1 0
          squeezenet  min =    6.30  max =   10.16  avg =    8.21
     squeezenet_int8  min =   14.53  max =   14.94  avg =   14.67
           mobilenet  min =   10.71  max =   11.26  avg =   10.91
      mobilenet_int8  min =   17.66  max =   18.46  avg =   17.91
        mobilenet_v2  min =    7.74  max =    8.05  avg =    7.89
        mobilenet_v3  min =    6.25  max =    6.70  avg =    6.38
          shufflenet  min =    3.78  max =    7.87  avg =    5.37
       shufflenet_v2  min =    4.19  max =    7.83  avg =    5.25

mobilenet_v3  min =    6.25  max =    6.70  avg =    6.38
             mnasnet  min =    7.29  max =    7.61  avg =    7.44
     proxylessnasnet  min =    8.10  max =    8.43  avg =    8.24
     efficientnet_b0  min =   11.77  max =   12.66  avg =   12.06
   efficientnetv2_b0  min =   13.80  max =   15.02  avg =   14.11
        regnety_400m  min =   10.09  max =   10.26  avg =   10.17
           blazeface  min =    1.24  max =    4.02  avg =    2.45
           googlenet  min =   24.05  max =   25.78  avg =   24.64
      googlenet_int8  min =   58.75  max =   62.45  avg =   59.54
            resnet18  min =   20.31  max =   21.48  avg =   20.74
       resnet18_int8  min =   53.82  max =   55.27  avg =   54.43
             alexnet  min =   17.37  max =   18.69  avg =   17.66
               vgg16  min =  114.49  max =  117.62  avg =  115.96

resnet18_int8  min =   53.82  max =   55.27  avg =   54.43
          vgg16_int8  min =  133.82  max =  144.40  avg =  137.07
            resnet50  min =   54.40  max =   58.74  avg =   55.54
       resnet50_int8  min =   92.95  max =  104.71  avg =   99.18
      squeezenet_ssd  min =   17.30  max =   18.65  avg =   17.71
 squeezenet_ssd_int8  min =   32.27  max =   33.88  avg =   32.82
       mobilenet_ssd  min =   24.01  max =   25.94  avg =   25.02
  mobilenet_ssd_int8  min =   34.68  max =   36.09  avg =   35.43
      mobilenet_yolo  min =   53.32  max =   63.48  avg =   56.58
  mobilenetv2_yolov3  min =   30.06  max =   34.24  avg =   31.46
         yolov4-tiny  min =   41.49  max =   43.55  avg =   42.50
           nanodet_m  min =   10.24  max =   11.08  avg =   10.43
    yolo-fastest-1.1  min =    3.85  max =    8.34  avg =    5.40

yolov4-tiny  min =   41.49  max =   43.55  avg =   42.50
      yolo-fastestv2  min =    4.33  max =    7.61  avg =    6.01
  vision_transformer  min =  556.38  max =  599.49  avg =  567.98
          FastestDet  min =    4.20  max =   11.37  avg =    6.51
mouri@MouriVM-openSUSE:~/Workspace/ncnn/benchmark>
```

### Hyper-V Linux Guest with GPU-PV enabled (Intel Core i7-7700K, NVIDIA GeForce GTX 1050 Ti)

- Host OS: Microsoft Windows 10 Enterprise LTSC 2021 (10.0.19044.2846)

```
[0 Microsoft Direct3D12 (NVIDIA GeForce GTX 1050 Ti)]  queueC=1[8]  queueG=0[4]  queueT=2[1]
[0 Microsoft Direct3D12 (NVIDIA GeForce GTX 1050 Ti)]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=0
[0 Microsoft Direct3D12 (NVIDIA GeForce GTX 1050 Ti)]  fp16-p/s/a=1/0/0  int8-p/s/a=1/0/0
[0 Microsoft Direct3D12 (NVIDIA GeForce GTX 1050 Ti)]  subgroup=32  basic=1  vote=1  ballot=1  shuffle=1

          squeezenet  min =   53.80  max =   64.22  avg =   59.91
     squeezenet_int8  min =   23.21  max =   25.98  avg =   24.44
           mobilenet  min =   47.63  max =   55.22  avg =   49.79
      mobilenet_int8  min =   23.27  max =   25.05  avg =   23.77
        mobilenet_v2  min =   58.17  max =   83.14  avg =   68.48
        mobilenet_v3  min =   92.14  max =  114.74  avg =  101.66
          shufflenet  min =   75.96  max =  106.54  avg =   89.64
       shufflenet_v2  min =   90.66  max =  114.69  avg =  103.25
             mnasnet  min =   58.40  max =   85.74  avg =   67.75
     proxylessnasnet  min =   66.73  max =   84.82  avg =   77.73
     efficientnet_b0  min =  134.28  max =  164.39  avg =  155.40
   efficientnetv2_b0  min =  171.97  max =  220.43  avg =  198.26

proxylessnasnet  min =   66.73  max =   84.82  avg =   77.73
        regnety_400m  min =  124.15  max =  145.61  avg =  135.99
           blazeface  min =   53.18  max =   72.10  avg =   60.21
           googlenet  min =  119.34  max =  159.93  avg =  134.71
      googlenet_int8  min =   96.71  max =  102.44  avg =   98.57
            resnet18  min =   68.14  max =   89.99  avg =   80.76
       resnet18_int8  min =   88.07  max =  108.62  avg =   91.09
             alexnet  min =   44.12  max =   51.57  avg =   48.09
               vgg16  min =   88.49  max =   99.87  avg =   93.42
          vgg16_int8  min =  196.17  max =  211.99  avg =  201.27
            resnet50  min =  115.36  max =  138.65  avg =  125.57
       resnet50_int8  min =  138.15  max =  148.55  avg =  141.08
      squeezenet_ssd  min =  138.42  max =  168.49  avg =  155.66

resnet50  min =  115.36  max =  138.65  avg =  125.57
 squeezenet_ssd_int8  min =   46.01  max =   47.83  avg =   46.85
       mobilenet_ssd  min =   82.39  max =  134.74  avg =  101.22
  mobilenet_ssd_int8  min =   45.53  max =   46.67  avg =   45.96
      mobilenet_yolo  min =   70.39  max =   87.83  avg =   80.01
  mobilenetv2_yolov3  min =   75.71  max =   90.59  avg =   84.04
         yolov4-tiny  min =   72.16  max =   87.76  avg =   76.81
           nanodet_m  min =   98.27  max =  129.60  avg =  112.34
    yolo-fastest-1.1  min =  101.01  max =  118.45  avg =  106.47
      yolo-fastestv2  min =  109.89  max =  137.23  avg =  123.97
  vision_transformer  min =  688.60  max =  750.54  avg =  723.30
          FastestDet  min =  104.16  max =  139.23  avg =  123.75

yolo-fastestv2  min =  109.89  max =  137.23  avg =  123.97
          squeezenet  min =    8.90  max =    9.48  avg =    9.15
     squeezenet_int8  min =   22.54  max =   24.13  avg =   22.85
           mobilenet  min =   14.85  max =   16.15  avg =   15.18
      mobilenet_int8  min =   23.56  max =   23.98  avg =   23.74
        mobilenet_v2  min =   11.03  max =   11.73  avg =   11.22
        mobilenet_v3  min =    8.61  max =    9.29  avg =    8.79
          shufflenet  min =    5.26  max =    5.96  avg =    5.42
       shufflenet_v2  min =    5.56  max =    7.06  avg =    5.82

mobilenet_v3  min =    8.61  max =    9.29  avg =    8.79
             mnasnet  min =   10.46  max =   11.04  avg =   10.68
     proxylessnasnet  min =   12.18  max =   12.55  avg =   12.33
     efficientnet_b0  min =   22.46  max =   23.15  avg =   22.86
   efficientnetv2_b0  min =   23.33  max =   23.80  avg =   23.55
        regnety_400m  min =   13.03  max =   14.25  avg =   13.28
           blazeface  min =    1.49  max =    1.95  avg =    1.61
           googlenet  min =   35.26  max =   46.31  avg =   39.63
      googlenet_int8  min =   96.25  max =   98.15  avg =   96.93
            resnet18  min =   29.34  max =   31.00  avg =   29.92
       resnet18_int8  min =   87.84  max =   89.85  avg =   88.73
             alexnet  min =   22.91  max =   23.87  avg =   23.18
               vgg16  min =  151.26  max =  174.79  avg =  155.94

resnet18_int8  min =   87.84  max =   89.85  avg =   88.73
          vgg16_int8  min =  193.66  max =  210.63  avg =  199.14
            resnet50  min =   74.89  max =   77.27  avg =   75.91
       resnet50_int8  min =  136.59  max =  162.13  avg =  141.22
      squeezenet_ssd  min =   24.48  max =   34.00  avg =   26.19
 squeezenet_ssd_int8  min =   46.31  max =   48.87  avg =   47.09
       mobilenet_ssd  min =   31.56  max =   34.45  avg =   32.50
  mobilenet_ssd_int8  min =   45.15  max =   46.53  avg =   45.93
      mobilenet_yolo  min =   72.09  max =   78.05  avg =   74.31
  mobilenetv2_yolov3  min =   40.44  max =   41.54  avg =   40.86
         yolov4-tiny  min =   56.73  max =   60.59  avg =   57.93
           nanodet_m  min =   13.22  max =   19.28  avg =   14.65
    yolo-fastest-1.1  min =    5.47  max =    5.70  avg =    5.58

yolov4-tiny  min =   56.73  max =   60.59  avg =   57.93
      yolo-fastestv2  min =    5.68  max =    7.20  avg =    5.88
  vision_transformer  min =  600.83  max =  666.35  avg =  617.33
          FastestDet  min =    6.05  max =    6.72  avg =    6.23
```

# PoseTracker-Android-Prototype

PoseTracker Android Demo Prototype, which is based on [mmdeploy](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x)

Please refer to [Original Repository](https://github.com/hanrui1sensetime/PoseTracker-Android-Prototype).

# Fast R-CNN

> [Fast R-CNN](https://arxiv.org/abs/1504.08083)

This paper proposes a Fast Region-based Convolutional Network method (Fast R-CNN) for object detection. Fast R-CNN builds on previous work to efficiently classify object proposals using deep convolutional networks. Compared to previous work, Fast R-CNN employs several innovations to improve training and testing speed while also increasing detection accuracy. Fast R-CNN trains the very deep VGG16 network 9x faster than R-CNN, is 213x faster at test-time, and achieves a higher mAP on PASCAL VOC 2012. Compared to SPPnet, Fast R-CNN trains VGG16 3x faster, tests 10x faster, and is more accurate.

<img src="https://user-images.githubusercontent.com/40661020/143882189-6258c05c-f2a1-4320-9282-7e2f2d502eb2.png"/>
</div>

Before training the Fast R-CNN, users should first train an [RPN](../rpn/README.md), and use the RPN to extract the region proposals.

</div>

The region proposals can be obtained by setting `DumpProposals` pseudo metric. The dumped results is a `dict(file_name: pred_instance)`.
The `pred_instance` is an `InstanceData` containing the sorted boxes and scores predicted by RPN. We provide example of dumping proposals in [RPN config](../rpn/rpn_r50_fpn_1x_coco.py).

- First, it should be obtained the region proposals in both training and validation (or testing) set.
  change the type of `test_evaluator` to `DumpProposals` in the RPN config to get the region proposals as below:

  The config of get training image region proposals can be set as below:

  ```python
  # For training set
          ann_file='data/coco/annotations/instances_train2017.json',
          data_prefix=dict(img='val2017/')))

      _delete_=True,
      type='DumpProposals',
      output_dir='data/coco/proposals/',
      proposals_file='rpn_r50_fpn_1x_train2017.pkl')
  test_evaluator = val_dataloader
  ```

  The config of get validation image region proposals can be set as below:

  ```python
  # For validation set
    type='DumpProposals',
    output_dir='data/coco/proposals/',
    proposals_file='rpn_r50_fpn_1x_val2017.pkl')
  ```

  Extract the region proposals command can be set as below:

  ```bash
  ./tools/dist_test.sh \
      configs/rpn_r50_fpn_1x_coco.py \
      checkpoints/rpn_r50_fpn_1x_coco_20200218-5525fa2e.pth \
      8
  ```

```bash
      8
  ```

  Users can refer to [test tutorial](https://mmdetection.readthedocs.io/en/latest/user_guides/test.html) for more details.

- Then, modify the path of `proposal_file` in the dataset and using `ProposalBroadcaster` to process both ground truth bounding boxes and region proposals in pipelines.
  An example of Fast R-CNN important setting can be seen as below:

  ```python
      dict(
          type='LoadImageFromFile',
          backend_args={{_base_.backend_args}}),
      dict(type='LoadProposals', num_max_proposals=2000),
      dict(type='LoadAnnotations', with_bbox=True),
      dict(
          type='ProposalBroadcaster',
          transforms=[
              dict(type='Resize', scale=(1333, 800), keep_ratio=True),
              dict(type='RandomFlip', prob=0.5),
          ]),
      dict(type='PackDetInputs')
  ]

          ]),
  ]
  test_pipeline = [
      dict(
      dict(type='LoadProposals', num_max_proposals=None),
      dict(
          ]),
      dict(
          type='PackDetInputs',
          meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',
                     'scale_factor'))
  ]
          proposal_file='proposals/rpn_r50_fpn_1x_train2017.pkl',
          proposal_file='proposals/rpn_r50_fpn_1x_val2017.pkl',
          pipeline=test_pipeline))

pipeline=train_pipeline))
  ```

- Finally, users can start training the Fast R-CNN.

```latex
@inproceedings{girshick2015fast,
  title={Fast r-cnn},
  author={Girshick, Ross},
  year={2015}
}
```

Apart from training/testing scripts, We provide lots of useful tools under the `tools/` directory.

## Useful Tools Link

- [Useful Tools](#useful-tools)
  - [Useful Tools Link](#useful-tools-link)
  - [Model Conversion](#model-conversion)
    - [Prepare a model for publishing](#prepare-a-model-for-publishing)
  - [Miscellaneous](#miscellaneous)
    - [Evaluating a metric](#evaluating-a-metric)
    - [Print the entire config](#print-the-entire-config)
    - [Check videos](#check-videos)
    - [Multi-Stream Fusion](#multi-stream-fusion)

`tools/deployment/publish_model.py` helps users to prepare their model for publishing.

Before you upload a model to AWS, you may want to:

(1) convert model weights to CPU tensors.
(2) delete the optimizer states.
(3) compute the hash of the checkpoint file and append the hash id to the filename.

```shell

```shell
```

E.g.,

```shell
```

The final output filename will be `tsn_r50_8xb32-1x1x3-100e_kinetics400-rgb-{hash id}.pth`.

`tools/analysis_tools/eval_metric.py` evaluates certain metrics of the results saved in a file according to a config file.

The saved result file is created on `tools/test.py` by setting the arguments `--out ${RESULT_FILE}` to indicate the result file,
which stores the final output of the whole model.

```shell

```shell
```

`tools/analysis_tools/print_config.py` prints the whole config verbatim, expanding all its imports.

```shell
```

### Check videos

`tools/analysis_tools/check_videos.py` uses specified video encoder to iterate all samples that are specified by the input configuration file, looks for invalid videos (corrupted or missing), and saves the corresponding file path to the output file. Please note that after deleting invalid videos, users need to regenerate the video file list.

```shell

```shell
```

### Multi-Stream Fusion

`tools/analysis_tools/report_accuracy.py` uses the dumped results (by setting `--dump res.pkl` when testing) to fuse the multi-stream prediction scores, i.e., late fusion.

```shell
```

Take joint-bone fusion as an example, which is a general practice in the task of skeleton-based action recognition.

```shell
```

```
```

}
```

<details>

```bibtex
}
```

| [pose_cspnext_t_udp](/configs/body_2d_keypoint/topdown_heatmap/coco/cspnext-tiny_udp_8xb256-210e_coco-256x192.py) |  256x192   | 0.665 |      0.874      |      0.723      | 0.723 |      0.917      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-tiny_udp-coco_pt-in1k_210e-256x192-0908dd2d_20230123.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-tiny_udp-coco_pt-in1k_210e-256x192-0908dd2d_20230123.json) |
| [pose_cspnext_s_udp](/configs/body_2d_keypoint/topdown_heatmap/coco/cspnext-s_udp_8xb256-210e_coco-256x192.py) |  256x192   | 0.697 |      0.886      |      0.776      | 0.753 |      0.929      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-s_udp-coco_pt-in1k_210e-256x192-92dbfc1d_20230123.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-s_udp-coco_pt-in1k_210e-256x192-92dbfc1d_20230123.json) |

| [pose_cspnext_m_udp](/configs/body_2d_keypoint/topdown_heatmap/coco/cspnext-m_udp_8xb256-210e_coco-256x192.py) |  256x192   | 0.732 |      0.896      |      0.806      | 0.785 |      0.937      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-m_udp-coco_pt-in1k_210e-256x192-95f5967e_20230123.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-m_udp-coco_pt-in1k_210e-256x192-95f5967e_20230123.json) |
| [pose_cspnext_l_udp](/configs/body_2d_keypoint/topdown_heatmap/coco/cspnext-l_udp_8xb256-210e_coco-256x192.py) |  256x192   | 0.750 |      0.904      |      0.822      | 0.800 |      0.941      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-l_udp-coco_pt-in1k_210e-256x192-661cdd8c_20230123.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-l_udp-coco_pt-in1k_210e-256x192-661cdd8c_20230123.json) |

| [pose_cspnext_t_udp_aic_coco](/configs/body_2d_keypoint/topdown_heatmap/coco/cspnext-tiny_udp_8xb256-210e_aic-coco-256x192.py) |  256x192   | 0.655 |      0.884      |      0.731      | 0.689 |      0.890      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-tiny_udp-aic-coco_210e-256x192-cbed682d_20230130.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-tiny_udp-aic-coco_210e-256x192-cbed682d_20230130.json) |
| [pose_cspnext_s_udp_aic_coco](/configs/body_2d_keypoint/topdown_heatmap/coco/cspnext-s_udp_8xb256-210e_aic-coco-256x192.py) |  256x192   | 0.700 |      0.905      |      0.783      | 0.733 |      0.918      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-s_udp-aic-coco_210e-256x192-92f5a029_20230130.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-s_udp-aic-coco_210e-256x192-92f5a029_20230130.json) |

| [pose_cspnext_m_udp_aic_coco](/configs/body_2d_keypoint/topdown_heatmap/coco/cspnext-m_udp_8xb256-210e_aic-coco-256x192.py) |  256x192   | 0.748 |      0.925      |      0.818      | 0.777 |      0.933      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-m_udp-aic-coco_210e-256x192-f2f7d6f6_20230130.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-m_udp-aic-coco_210e-256x192-f2f7d6f6_20230130.json) |
| [pose_cspnext_l_udp_aic_coco](/configs/body_2d_keypoint/topdown_heatmap/coco/cspnext-l_udp_8xb256-210e_aic-coco-256x192.py) |  256x192   | 0.772 |      0.936      |      0.839      | 0.799 |      0.943      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-l_udp-aic-coco_210e-256x192-273b7631_20230130.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/cspnext-l_udp-aic-coco_210e-256x192-273b7631_20230130.json) |

Note that, UDP also adopts the unbiased encoding/decoding algorithm of [DARK](https://mmpose.readthedocs.io/en/latest/model_zoo_papers/techniques.html#darkpose-cvpr-2020).

Flip test and detector is not used in the result of aic-coco training.

# LiDAR-Based 3D Semantic Segmentation

LiDAR-based 3D semantic segmentation is one of the most basic tasks supported in MMDetection3D.
It expects the given model to take any number of points with features collected by LiDAR as input, and predict the semantic labels for each input point.
Next, taking PointNet++ (SSG) on the ScanNet dataset as an example, we will show how to prepare data, train and test a model on a standard 3D semantic segmentation benchmark, and how to visualize and validate the results.

To begin with, we need to download the raw data from ScanNet's [official website](http://kaldir.vc.in.tum.de/scannet_benchmark/documentation).

Due to different ways of organizing the raw data in different datasets, we typically need to collect the useful data information with a .pkl or .json file.

So after getting all the raw data ready, we can follow the instructions presented in [ScanNet README doc](https://github.com/open-mmlab/mmdetection3d/blob/master/data/scannet/README.md/) to generate data infos.

Afterwards, the related folder structure should be as follows:

```
├── tools
├── data
│   │   ├── scannet_instance_data
│   │   ├── instance_mask
│   │   ├── seg_info
│   │   │   ├── train_label_weight.npy
│   │   │   ├── train_resampled_scene_idxs.npy
│   │   │   ├── val_label_weight.npy
│   │   │   ├── val_resampled_scene_idxs.npy

│   │   ├── scannet_infos_train.pkl
│   │   ├── scannet_infos_val.pkl
│   │   ├── scannet_infos_test.pkl
```

Then let us train a model with provided configs for PointNet++ (SSG).
You can basically follow this [tutorial](https://mmdetection3d.readthedocs.io/en/latest/1_exist_data_model.html#inference-with-existing-models) for sample scripts when training with different GPU settings.
Suppose we use 2 GPUs on a single machine with distributed training:

```
./tools/dist_train.sh configs/pointnet2/pointnet2_ssg_2xb16-cosine-200e_scannet-seg.py 2
```

Note that `2xb16` in the config name refers to the training is completed with 2 GPUs and 16 samples on each GPU.
If your customized setting is different from this, sometimes you need to adjust the learning rate accordingly.

A basic rule can be referred to [here](https://arxiv.org/abs/1706.02677).

## Quantitative Evaluation

During training, the model checkpoints will be evaluated regularly according to the setting of `train_cfg = dict(val_interval=xxx)` in the config.
We support official evaluation protocols for different datasets.
For ScanNet, the model will be evaluated with mean Intersection over Union (mIoU) over all 20 categories.
The evaluation results will be printed in the command like:

```
+---------+--------+--------+---------+--------+--------+--------+--------+--------+--------+-----------+---------+---------+--------+---------+--------------+----------------+--------+--------+---------+----------------+--------+--------+---------+

| classes | wall   | floor  | cabinet | bed    | chair  | sofa   | table  | door   | window | bookshelf | picture | counter | desk   | curtain | refrigerator | showercurtrain | toilet | sink   | bathtub | otherfurniture | miou   | acc    | acc_cls |
| results | 0.7257 | 0.9373 | 0.4625  | 0.6613 | 0.7707 | 0.5562 | 0.5864 | 0.4010 | 0.4558 | 0.7011    | 0.2500  | 0.4645  | 0.4540 | 0.5399  | 0.2802       | 0.3488         | 0.7359 | 0.4971 | 0.6922  | 0.3681         | 0.5444 | 0.8118 | 0.6695  |

```

In addition, you can also evaluate a specific model checkpoint after training is finished. Simply run scripts like the following:

```
./tools/dist_test.sh configs/pointnet2/pointnet2_ssg_16x2_cosine_200e_scannet-seg.py work_dirs/pointnet2_ssg/latest.pth 8
```

If you would like to only conduct inference or test the model performance on the online benchmark,
you should change `ann_file='scannet_infos_val.pkl'` to `ann_file='scannet_infos_test.pkl'` in the
ScanNet dataset's [config](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/datasets/scannet-seg.py#L129). Remember to
specify the `submission_prefix` in the `test_evaluator`,
e.g., adding `test_evaluator = dict(type='SegMetric', submission_prefix=work_dirs/pointnet2_ssg/test_submission`) or just add `--cfg-options test_evaluator.submission_prefix=work_dirs/pointnet2_ssg/test_submission` in the end of command.

## Qualitative Validation

MMDetection3D also provides versatile tools for visualization such that you can use `tools/misc/visualize_results.py` with results pkl file for offline visualization of add `--show` in the end of test command to do the online visualization.
Besides, we also provide scripts `tools/misc/browse_dataset.py` to visualize the dataset without inference.
Please refer more details in the [doc for visualization](https://mmdetection3d.readthedocs.io/en/latest/useful_tools.html#visualization).

</div>

### COCO-WholeBody Dataset

|        Model        | Input Size | Whole AP | Whole AR |                              Details and Download                               |

| :-----------------: | :--------: | :------: | :------: | :-----------------------------------------------------------------------------: |
|   HRNet-w48+Dark+   |  384x288   |  0.661   |  0.743   |  [hrnet_dark_coco-wholebody.md](./coco-wholebody/hrnet_dark_coco-wholebody.md)  |
|   HRNet-w32+Dark    |  256x192   |  0.582   |  0.671   |  [hrnet_dark_coco-wholebody.md](./coco-wholebody/hrnet_dark_coco-wholebody.md)  |
|      HRNet-w48      |  256x192   |  0.579   |  0.681   |       [hrnet_coco-wholebody.md](./coco-wholebody/hrnet_coco-wholebody.md)       |
|      CSPNeXt-m      |  256x192   |  0.567   |  0.641   | [cspnext_udp_coco-wholebody.md](./coco-wholebody/cspnext_udp_coco-wholebody.md) |
|     ResNet-152      |  256x192   |  0.548   |  0.661   |      [resnet_coco-wholebody.md](./coco-wholebody/resnet_coco-wholebody.md)      |

|      HRNet-w32      |  256x192   |  0.536   |  0.636   |       [hrnet_coco-wholebody.md](./coco-wholebody/hrnet_coco-wholebody.md)       |
|     ResNet-101      |  256x192   |  0.531   |  0.645   |      [resnet_coco-wholebody.md](./coco-wholebody/resnet_coco-wholebody.md)      |
| S-ViPNAS-Res50+Dark |  256x192   |  0.528   |  0.632   | [vipnas_dark_coco-wholebody.md](./coco-wholebody/vipnas_dark_coco-wholebody.md) |
|      ResNet-50      |  256x192   |  0.521   |  0.633   |      [resnet_coco-wholebody.md](./coco-wholebody/resnet_coco-wholebody.md)      |
|   S-ViPNAS-Res50    |  256x192   |  0.495   |  0.607   |      [vipnas_coco-wholebody.md](./coco-wholebody/vipnas_coco-wholebody.md)      |

# NCNN增加自定义层

## 举例

这里举个例子添加自定义层次 如Relu6，即 std::min(6.f, std::max(0.f, val))

```
Input            input   0 1 input
Convolution      conv2d  1 1 input conv2d 0=32 1=1 2=1 3=1 4=0 5=0 6=768
Relu6            relu6   1 1 conv2d relu6
Pooling          maxpool 1 1 relu6 maxpool 0=0 1=3 2=2 3=-233 4=0
```

## 定义源码h文件：src/layer/relu6.h

```CPP
#ifndef LAYER_RELU6_H
#define LAYER_RELU6_H

#include "layer.h"

class Relu6 : public Layer
{
public:
    Relu6();

    virtual int forward_inplace(Mat& bottom_top_blob, const Option& opt) const;
};

#endif // LAYER_RELU6_H
```

## 定义源码CPP文件：src/layer/relu6.cpp

```CPP
#include "relu6.h"

#include <math.h>

Relu6::Relu6()
{
    one_blob_only = true;
    support_inplace = true;
}

int Relu6::forward_inplace(Mat& bottom_top_blob, const Option& opt) const
{
        int w = bottom_top_blob.w;
        int h = bottom_top_blob.h;
        int channels = bottom_top_blob.c;
        int size = w * h;

{

        #pragma omp parallel for num_threads(opt.num_threads)
        for (int q=0; q < channels; q++)
        {
            float* ptr = bottom_top_blob.channel(q);

            for (int i=0; i<size; i++)
            {
                ptr[i] = std::min(6.f, std::max(0.f, ptr[i]));
            }
        }

        return 0;
}

```

## 修改 src/CMakeLists.txt 注册Relu6

```CPP
ncnn_add_layer(GroupNorm)
ncnn_add_layer(LayerNorm)
ncnn_add_layer(Relu6)
```

## 定义测试用例CPP文件 tests/test_relu6.cpp 

```CPP
#include "layer/relu6.h"
#include "testutil.h"

static int test_relu6(const ncnn::Mat& a)
{
    ncnn::ParamDict pd;

    std::vector<ncnn::Mat> weights(0);

    int ret = test_layer<ncnn::Relu6>("Relu6", pd, weights, a);
    if (ret != 0)
    {

{

    {
        fprintf(stderr, "test_relu6 failed a.dims=%d a=(%d %d %d)\n", a.dims, a.w, a.h, a.c);
    }

    return ret;
}

static int test_relu6_0()
{
    return 0
           || test_relu6(RandomMat(5, 7, 24))
           || test_relu6(RandomMat(7, 9, 12))
           || test_relu6(RandomMat(3, 5, 13));
}

static int test_relu6_1()
{
    return 0
           || test_relu6(RandomMat(15, 24))
           || test_relu6(RandomMat(17, 12))
           || test_relu6(RandomMat(19, 15));
}

static int test_relu6_2()
{
    return 0
           || test_relu6(RandomMat(128))
           || test_relu6(RandomMat(124))
           || test_relu6(RandomMat(127));
}

int main()
{
    SRAND(7767517);

    return 0
           || test_relu6_0()
           || test_relu6_1()
           || test_relu6_2();
}

```

|| test_relu6(RandomMat(127));
}

int main()
{

    return 0
}

```

## 修改tests/CMakeLists.txt 注册Relu6测试用例

```CPP
ncnn_add_layer_test(LSTM)
ncnn_add_layer_test(Yolov3DetectionOutput)
ncnn_add_layer_test(Relu6)
```

## 编译

```
按原NCNN步骤编译
```

## 单元测试

```
./test_relu6
```

}
```

</div>

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
        |   |── all
        |
        `── images
        |   |── train
        |   |── val

> [Deep Face Super-Resolution with Iterative Collaboration between Attentive Recovery and Landmark Estimation](https://arxiv.org/abs/2003.13063)

Recent works based on deep learning and facial priors have succeeded in super-resolving severely degraded facial images. However, the prior knowledge is not fully exploited in existing methods, since facial priors such as landmark and component maps are always estimated by low-resolution or coarsely super-resolved images, which may be inaccurate and thus affect the recovery performance. In this paper, we propose a deep face super-resolution (FSR) method with iterative collaboration between two recurrent networks which focus on facial image recovery and landmark estimation respectively. In each recurrent step, the recovery branch utilizes the prior knowledge of landmarks to yield higher-quality images which facilitate more accurate landmark estimation in turn. Therefore, the iterative information interaction between two processes boosts the performance of each other progressively. Moreover, a new attentive fusion module is designed to strengthen the guidance of landmark maps, where facial components are generated individually and aggregated attentively for better restoration. Quantitative and qualitative experimental results show the proposed method significantly outperforms state-of-the-art FSR methods in recovering high-quality face images.

 <img src="https://user-images.githubusercontent.com/7676947/144017838-63e31123-1b59-4743-86bb-737bd32a9209.png" width="400"/>
</div >

In the log data of `dic_gan_x8c48b6_g4_150k_CelebAHQ`, DICGAN is verified on the first 9 pictures of the test set of CelebA-HQ, so `PSNR` and `SSIM` shown in the follow table is different from the log data.

`Training Resources`: Training Resourcesrmation during training.

|                                    Model                                     | Dataset  | scale |  PSNR   |  SSIM  | Training Resources  |                                    Download                                     |

| :--------------------------------------------------------------------------: | :------: | :---: | :-----: | :----: | :-----------------: | :-----------------------------------------------------------------------------: |
|     [dic_x8c48b6_g4_150k_CelebAHQ](./dic_x8c48b6_4xb2-150k_celeba-hq.py)     | CelebAHQ |  x8   | 25.2319 | 0.7422 | 4 (Tesla PG503-216) | [model](https://download.openmmlab.com/mmediting/restorers/dic/dic_x8c48b6_g4_150k_CelebAHQ_20210611-5d3439ca.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/dic/dic_x8c48b6_g4_150k_CelebAHQ_20210611-5d3439ca.log.json) |
| [dic_gan_x8c48b6_g4_500k_CelebAHQ](./dic_gan-x8c48b6_4xb2-500k_celeba-hq.py) | CelebAHQ |  x8   | 23.6241 | 0.6721 | 4 (Tesla PG503-216) | [model](https://download.openmmlab.com/mmediting/restorers/dic/dic_gan_x8c48b6_g4_500k_CelebAHQ_20210625-3b89a358.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/dic/dic_gan_x8c48b6_g4_500k_CelebAHQ_20210625-3b89a358.log.json) |

**Train**

<details>

```shell
# cpu train

```

**Test**

<details>

```shell
# cpu test

```

```bibtex
}
```

# Overview

This chapter will introduce you to the overall framework of MMPose and provide links to detailed tutorials.

## What is MMPose

MMPose is a Pytorch-based pose estimation open-source toolkit, a member of the [OpenMMLab Project](https://github.com/open-mmlab). It contains a rich set of algorithms for 2d multi-person human pose estimation, 2d hand pose estimation, 2d face landmark detection, 133 keypoint whole-body human pose estimation, fashion landmark detection and animal pose estimation as well as related components and modules, below is its overall framework.

MMPose consists of **8** main components:

- **apis** provides high-level APIs for model inference
- **structures** provides data structures like bbox, keypoint and PoseDataSample
- **datasets** supports various datasets for pose estimation

  - **transforms** contains a lot of useful data augmentation transforms
- **codecs** provides pose encoders and decoders: an encoder encodes poses (mostly keypoints) into learning targets (e.g. heatmaps), and a decoder decodes model outputs into pose predictions
- **models** provides all components of pose estimation models in a modular structure
  - **pose_estimators** defines all pose estimation model classes
  - **data_preprocessors** is for preprocessing the input data of the model
  - **backbones** provides a collection of backbone networks
  - **necks** contains various neck modules
  - **heads** contains various prediction heads that perform pose estimation
  - **losses** contains various loss functions
- **engine** provides runtime components related to pose estimation

- **heads** contains various prediction heads that perform pose estimation
  - **hooks** provides various hooks of the runner
- **evaluation** provides metrics for evaluating model performance
- **visualization** is for visualizing skeletons, heatmaps and other information

## How to Use this Guide

We have prepared detailed guidelines for all types of users:

1. For installation instrunctions:

   - [Installation](./installation.md)

2. For the basic usage of MMPose:

   - [A 20-minute Tour to MMPose](./guide_to_framework.md)
   - [Inference](./user_guides/inference.md)
   - [Configs](./user_guides/configs.md)
   - [Prepare Datasets](./user_guides/prepare_datasets.md)
   - [Train and Test](./user_guides/train_and_test.md)

3. For developers who wish to develop based on MMPose:

   - [Learn about Codecs](./advanced_guides/codecs.md)

- [Train and Test](./user_guides/train_and_test.md)

   - [Dataflow in MMPose](./advanced_guides/dataflow.md)
   - [Implement New Models](./advanced_guides/implement_new_models.md)
   - [Customize Datasets](./advanced_guides/customize_datasets.md)
   - [Customize Data Transforms](./advanced_guides/customize_transforms.md)
   - [Customize Optimizer](./advanced_guides/customize_optimizer.md)
   - [Customize Logging](./advanced_guides/customize_logging.md)
   - [How to Deploy](./advanced_guides/how_to_deploy.md)
   - [Model Analysis](./advanced_guides/model_analysis.md)
   - [Migration Guide](./migration.md)

4. For researchers and developers who are willing to contribute to MMPose:

   - [Contribution Guide](./contribution_guide.md)

5. For some common issues, we provide a FAQ list:

## current param load api
### Cons
#### long and awful code
#### three functions
#### not extensible
#### no default value
#### no variable length array
```
MyLayer  mylayer 1 1 in out 100 1.250000
```
```
binary 100
binary 1.250000
```
```cpp
#if NCNN_STDIO
#if NCNN_STRING
int MyLayer::load_param(FILE* paramfp)
{
    int nscan = fscanf(paramfp, "%d %f", &a, &b);
    if (nscan != 2)
    {
        fprintf(stderr, "MyLayer load_param failed %d\n", nscan);
        return -1;
    }

    return 0;
}
#endif // NCNN_STRING
int MyLayer::load_param_bin(FILE* paramfp)
{
    fread(&a, sizeof(int), 1, paramfp);

    fread(&b, sizeof(float), 1, paramfp);

    return 0;
}
#endif // NCNN_STDIO

int MyLayer::load_param(const unsigned char*& mem)
{
    a = *(int*)(mem);
    mem += 4;

    b = *(float*)(mem);
    mem += 4;

    return 0;
}
```

## new param load api proposed
### Pros
#### clean and simple api
#### default value
#### extensible
#### variable length array
```
7767517

b = *(float*)(mem);
    mem += 4;

    return 0;
}
```

### Pros
```
7767517
MyLayer  mylayer 1 1 in out 0=100 1=1.250000 -23303=5,0.1,0.2,0.4,0.8,1.0
```
```
binary 0xDD857600(magic)

binary 0
binary 100
binary 1
binary -23303
binary 5
binary 0.1
binary 0.2
binary 0.4
binary 0.8
binary 1.0
binary -233(EOP)
```
```cpp
int MyLayer::load_param(const ParamDict& pd)
{
    // pd.get( param id (seq), default value );
    a = pd.get(0, 100);
    b = pd.get(1, 1.25f);

    // get default value for c if not specified in param file
    c = pd.get(2, 0.001);

    // get array
    d = pd.get(3, Mat(len, array));
    return 0;
}
```

# BiSeNetV2

> [Bisenet v2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation](https://arxiv.org/abs/2004.02147)

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.18.0/mmseg/models/backbones/bisenetv2.py#L545">Code Snippet</a>

The low-level details and high-level semantics are both essential to the semantic segmentation task. However, to speed up the model inference, current approaches almost always sacrifice the low-level details, which leads to a considerable accuracy decrease. We propose to treat these spatial details and categorical semantics separately to achieve high accuracy and high efficiency for realtime semantic segmentation. To this end, we propose an efficient and effective architecture with a good trade-off between speed and accuracy, termed Bilateral Segmentation Network (BiSeNet V2). This architecture involves: (i) a Detail Branch, with wide channels and shallow layers to capture low-level details and generate high-resolution feature representation; (ii) a Semantic Branch, with narrow channels and deep layers to obtain high-level semantic context. The Semantic Branch is lightweight due to reducing the channel capacity and a fast-downsampling strategy. Furthermore, we design a Guided Aggregation Layer to enhance mutual connections and fuse both types of feature representation. Besides, a booster training strategy is designed to improve the segmentation performance without any extra inference cost. Extensive quantitative and qualitative evaluations demonstrate that the proposed architecture performs favourably against a few state-of-the-art real-time semantic segmentation approaches. Specifically, for a 2,048x1,024 input, we achieve 72.6% Mean IoU on the Cityscapes test set with a speed of 156 FPS on one NVIDIA GeForce GTX 1080 Ti card, which is significantly faster than existing methods, yet we achieve better segmentation accuracy.

<img src="https://user-images.githubusercontent.com/24582831/142898966-ec4a81da-b4b0-41ee-b083-1d964582c18a.png" width="70%"/>
</div>

| Method    | Backbone         | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                  | download                                                                                                                                                                                                                                                                                                                                                                                               |

| BiSeNetV2 | BiSeNetV2        | 1024x1024 |  160000 | 7.64     | 31.77          | V100   | 73.21 |         75.74 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/bisenetv2/bisenetv2_fcn_4xb4-160k_cityscapes-1024x1024.py)      | [model](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv2/bisenetv2_fcn_4x4_1024x1024_160k_cityscapes/bisenetv2_fcn_4x4_1024x1024_160k_cityscapes_20210902_015551-bcf10f09.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv2/bisenetv2_fcn_4x4_1024x1024_160k_cityscapes/bisenetv2_fcn_4x4_1024x1024_160k_cityscapes_20210902_015551.log.json)                     |

| BiSeNetV2 | BiSeNetV2 (OHEM) | 1024x1024 |  160000 | 7.64     | -              | V100   | 73.57 |         75.80 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/bisenetv2/bisenetv2_fcn_4xb4-ohem-160k_cityscapes-1024x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv2/bisenetv2_fcn_ohem_4x4_1024x1024_160k_cityscapes/bisenetv2_fcn_ohem_4x4_1024x1024_160k_cityscapes_20210902_112947-5f8103b4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv2/bisenetv2_fcn_ohem_4x4_1024x1024_160k_cityscapes/bisenetv2_fcn_ohem_4x4_1024x1024_160k_cityscapes_20210902_112947.log.json) |

| BiSeNetV2 | BiSeNetV2 (4x8)  | 1024x1024 |  160000 | 15.05    | -              | V100   | 75.76 |         77.79 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/bisenetv2/bisenetv2_fcn_4xb8-160k_cityscapes-1024x1024.py)      | [model](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv2/bisenetv2_fcn_4x8_1024x1024_160k_cityscapes/bisenetv2_fcn_4x8_1024x1024_160k_cityscapes_20210903_000032-e1a2eed6.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv2/bisenetv2_fcn_4x8_1024x1024_160k_cityscapes/bisenetv2_fcn_4x8_1024x1024_160k_cityscapes_20210903_000032.log.json)                     |

| BiSeNetV2 | BiSeNetV2 (FP16) | 1024x1024 |  160000 | 5.77     | 36.65          | V100   | 73.07 |         75.13 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/bisenetv2/bisenetv2_fcn_4xb4-amp-160k_cityscapes-1024x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv2/bisenetv2_fcn_fp16_4x4_1024x1024_160k_cityscapes/bisenetv2_fcn_fp16_4x4_1024x1024_160k_cityscapes_20210902_045942-b979777b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv2/bisenetv2_fcn_fp16_4x4_1024x1024_160k_cityscapes/bisenetv2_fcn_fp16_4x4_1024x1024_160k_cityscapes_20210902_045942.log.json) |

Note:

- `OHEM` means Online Hard Example Mining (OHEM) is adopted in training.
- `4x8` means 4 GPUs with 8 samples per GPU in training.

```bibtex
@article{yu2021bisenet,
  title={Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation},

```bibtex
  author={Yu, Changqian and Gao, Changxin and Wang, Jingbo and Yu, Gang and Shen, Chunhua and Sang, Nong},
  pages={1--18},
}
```

# FreeAnchor for 3D Object Detection

Modern CNN-based object detectors assign anchors for ground-truth objects under the restriction of object-anchor Intersection-over-Unit (IoU). In this study, we propose a learning-to-match approach to break IoU restriction, allowing objects to match anchors in a flexible manner. Our approach, referred to as FreeAnchor, updates hand-crafted anchor assignment to “free" anchor matching by formulating detector training as a maximum likelihood estimation (MLE) procedure. FreeAnchor targets at learning features which best explain a class of objects in terms of both classification and localization. FreeAnchor is implemented by optimizing detection customized likelihood and can be fused with CNN-based detectors in a plug-and-play manner. Experiments on COCO demonstrate that FreeAnchor consistently outperforms the counterparts with significant margins.

<img src="https://user-images.githubusercontent.com/36950400/143866685-e3ac08bb-cd0c-4ada-ba8a-18e03cccdd0f.png" width="600"/>
</div>

We implement FreeAnchor in 3D detection systems and provide their first results with PointPillars on nuScenes dataset.
With the implemented `FreeAnchor3DHead`, a PointPillar detector with a big backbone (e.g., RegNet-3.2GF) achieves top performance
on the nuScenes benchmark.

## Usage

As in the [baseline config](pointpillars_hv_fpn_head-free-anchor_sbn-all_8xb4-2x_nus-3d.py), we only need to replace the head of an existing one-stage detector to use FreeAnchor head.
Since the config is inherit from a common detector head, `_delete_=True` is necessary to avoid conflicts.
The hyperparameters are specifically tuned according to the original paper.

```python
_base_ = [
    '../_base_/models/pointpillars_hv_fpn_lyft.py',
    '../_base_/datasets/nus-3d.py', '../_base_/schedules/schedule-2x.py',

```python
_base_ = [
    '../_base_/default_runtime.py'
]

    pts_bbox_head=dict(
        type='FreeAnchor3DHead',
        num_classes=10,
        pre_anchor_topk=25,
        bbox_thr=0.5,
        gamma=2.0,
        alpha=0.5,
            ranges=[[-50, -50, -1.8, 50, 50, -1.8]],
            scales=[1, 2, 4],
            sizes=[
                [2.5981, 0.8660, 1.],  # 1.5 / sqrt(3)
                [1.7321, 0.5774, 1.],  # 1 / sqrt(3)
                [1., 1., 1.],
                [0.4, 0.4, 1],
            ],
            custom_values=[0, 0],
            reshape_out=True),
        assigner_per_size=False,

[0.4, 0.4, 1],
            ],
        dir_offset=-0.7854,  # -pi / 4
        bbox_coder=dict(type='DeltaXYZWLHRBBoxCoder', code_size=9),
        loss_bbox=dict(type='SmoothL1Loss', beta=1.0 / 9.0, loss_weight=0.8),
            type='CrossEntropyLoss', use_sigmoid=False, loss_weight=0.2)),
    train_cfg = dict(
        pts=dict(code_weight=[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.25, 0.25])))
```

# model training and testing settings
```

|                                                    Backbone                                                     | FreeAnchor | Lr schd | Mem (GB) | Inf time (fps) |  mAP  |  NDS  |                                                                                                                                                                                                                                                                    Download                                                                                                                                                                                                                                                                    |

|                      [FPN](../pointpillars/pointpillars_hv_fpn_sbn-all_8xb4-2x_nus-3d.py)                       |     ✗      |   2x    |   17.1   |                | 40.0  | 53.3  |                                                                        [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d_20200620_230405-2fa62f3d.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d_20200620_230405.log.json)                                                                        |

|                     [FPN](./pointpillars_hv_fpn_head-free-anchor_sbn-all_8xb4-2x_nus-3d.py)                     |     ✓      |   2x    |   16.3   |                | 43.82 | 54.86 |                                                 [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/free_anchor/hv_pointpillars_fpn_sbn-all_free-anchor_4x8_2x_nus-3d/hv_pointpillars_fpn_sbn-all_free-anchor_4x8_2x_nus-3d_20210816_163441-ae0897e7.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/free_anchor/hv_pointpillars_fpn_sbn-all_free-anchor_4x8_2x_nus-3d/hv_pointpillars_fpn_sbn-all_free-anchor_4x8_2x_nus-3d_20210816_163441.log.json)                                                 |

|            [RegNetX-400MF-FPN](../regnet/pointpillars_hv_regnet-400mf_fpn_sbn-all_8xb4-2x_nus-3d.py)            |     ✗      |   2x    |   17.3   |                | 44.8  | 56.4  |                                                    [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/regnet/hv_pointpillars_regnet-400mf_fpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_regnet-400mf_fpn_sbn-all_4x8_2x_nus-3d_20200620_230239-c694dce7.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/regnet/hv_pointpillars_regnet-400mf_fpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_regnet-400mf_fpn_sbn-all_4x8_2x_nus-3d_20200620_230239.log.json)                                                    |

|       [RegNetX-400MF-FPN](./pointpillars_hv_regnet-400mf_fpn_head-free-anchor_sbn-all_8xb4-2x_nus-3d.py)        |     ✓      |   2x    |   17.6   |                | 48.3  | 58.65 |                       [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/free_anchor/hv_pointpillars_regnet-400mf_fpn_sbn-all_free-anchor_4x8_2x_nus-3d/hv_pointpillars_regnet-400mf_fpn_sbn-all_free-anchor_4x8_2x_nus-3d_20210827_213939-a2dd3fff.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/free_anchor/hv_pointpillars_regnet-400mf_fpn_sbn-all_free-anchor_4x8_2x_nus-3d/hv_pointpillars_regnet-400mf_fpn_sbn-all_free-anchor_4x8_2x_nus-3d_20210827_213939.log.json)                       |

|       [RegNetX-1.6GF-FPN](./pointpillars_hv_regnet-1.6gf_fpn_head-free-anchor_sbn-all_8xb4-2x_nus-3d.py)        |     ✓      |   2x    |   24.3   |                | 52.04 | 61.49 |                       [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/free_anchor/hv_pointpillars_regnet-1.6gf_fpn_sbn-all_free-anchor_4x8_2x_nus-3d/hv_pointpillars_regnet-1.6gf_fpn_sbn-all_free-anchor_4x8_2x_nus-3d_20210828_025608-bfbd506e.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/free_anchor/hv_pointpillars_regnet-1.6gf_fpn_sbn-all_free-anchor_4x8_2x_nus-3d/hv_pointpillars_regnet-1.6gf_fpn_sbn-all_free-anchor_4x8_2x_nus-3d_20210828_025608.log.json)                       |

| [RegNetX-1.6GF-FPN](./pointpillars_hv_regnet-1.6gf_fpn_head-free-anchor_sbn-all_8xb4-strong-aug-3x_nus-3d.py)\* |     ✓      |   3x    |   24.4   |                | 52.69 | 62.45 | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/free_anchor/hv_pointpillars_regnet-1.6gf_fpn_sbn-all_free-anchor_strong-aug_4x8_3x_nus-3d/hv_pointpillars_regnet-1.6gf_fpn_sbn-all_free-anchor_strong-aug_4x8_3x_nus-3d_20210827_184909-14d2dbd1.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/free_anchor/hv_pointpillars_regnet-1.6gf_fpn_sbn-all_free-anchor_strong-aug_4x8_3x_nus-3d/hv_pointpillars_regnet-1.6gf_fpn_sbn-all_free-anchor_strong-aug_4x8_3x_nus-3d_20210827_184909.log.json) |

|       [RegNetX-3.2GF-FPN](./pointpillars_hv_regnet-3.2gf_fpn_head-free-anchor_sbn-all_8xb4-2x_nus-3d.py)        |     ✓      |   2x    |   29.4   |                | 52.4  | 61.94 |                       [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/free_anchor/hv_pointpillars_regnet-3.2gf_fpn_sbn-all_free-anchor_4x8_2x_nus-3d/hv_pointpillars_regnet-3.2gf_fpn_sbn-all_free-anchor_4x8_2x_nus-3d_20210827_181237-e385c35a.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/free_anchor/hv_pointpillars_regnet-3.2gf_fpn_sbn-all_free-anchor_4x8_2x_nus-3d/hv_pointpillars_regnet-3.2gf_fpn_sbn-all_free-anchor_4x8_2x_nus-3d_20210827_181237.log.json)                       |

| [RegNetX-3.2GF-FPN](./pointpillars_hv_regnet-3.2gf_fpn_head-free-anchor_sbn-all_8xb4-strong-aug-3x_nus-3d.py)\* |     ✓      |   3x    |   29.2   |                | 54.23 | 63.41 | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/free_anchor/hv_pointpillars_regnet-3.2gf_fpn_sbn-all_free-anchor_strong-aug_4x8_3x_nus-3d/hv_pointpillars_regnet-3.2gf_fpn_sbn-all_free-anchor_strong-aug_4x8_3x_nus-3d_20210828_030816-06708918.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/free_anchor/hv_pointpillars_regnet-3.2gf_fpn_sbn-all_free-anchor_strong-aug_4x8_3x_nus-3d/hv_pointpillars_regnet-3.2gf_fpn_sbn-all_free-anchor_strong-aug_4x8_3x_nus-3d_20210828_030816.log.json) |

**Note**: Models noted by `*` means it is trained using stronger augmentation with vertical flip under bird-eye-view, global translation, and larger range of global rotation.

```latex

```latex
}
```

<details>

```bibtex
}
```

<details>

```bibtex

}
```

<details>

```bibtex
}
```

volume={29},
}
```

| [pose_hrnetv2_w18_udp](/configs/hand_2d_keypoint/topdown_heatmap/onehand10k/td-hm_hrnetv2-w18_udp-8xb64-210e_onehand10k-256x256.py) |  256x256   |  0.990  | 0.571 | 23.88 | [ckpt](https://download.openmmlab.com/mmpose/hand/udp/hrnetv2_w18_onehand10k_256x256_udp-0d1b515d_20210330.pth) | [log](https://download.openmmlab.com/mmpose/hand/udp/hrnetv2_w18_onehand10k_256x256_udp_20210330.log.json) |

# BiSeNetV1

> [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://arxiv.org/abs/1808.00897)

<a href="https://github.com/ycszen/TorchSeg/tree/master/model/bisenet">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.18.0/mmseg/models/backbones/bisenetv1.py#L266">Code Snippet</a>

Semantic segmentation requires both rich spatial information and sizeable receptive field. However, modern approaches usually compromise spatial resolution to achieve real-time inference speed, which leads to poor performance. In this paper, we address this dilemma with a novel Bilateral Segmentation Network (BiSeNet). We first design a Spatial Path with a small stride to preserve the spatial information and generate high-resolution features. Meanwhile, a Context Path with a fast downsampling strategy is employed to obtain sufficient receptive field. On top of the two paths, we introduce a new Feature Fusion Module to combine features efficiently. The proposed architecture makes a right balance between the speed and segmentation performance on Cityscapes, CamVid, and COCO-Stuff datasets. Specifically, for a 2048x1024 input, we achieve 68.4% Mean IOU on the Cityscapes test dataset with speed of 105 FPS on one NVIDIA Titan XP card, which is significantly faster than the existing methods with comparable performance.

<img src="https://user-images.githubusercontent.com/24582831/142898839-a0a78148-848a-41b2-8682-b1f61ac004ba.png" width="70%"/>
</div>

| Method    | Backbone               | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                          | download                                                                                                                                                                                                                                                                                                                                                                                                                               |

| BiSeNetV1 | R-18-D32 (No Pretrain) | 1024x1024 |  160000 | 5.69     | 31.77          | V100   | 74.44 | 77.05         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/bisenetv1/bisenetv1_r18-d32_4xb4-160k_cityscapes-1024x1024.py)          | [model](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r18-d32_4x4_1024x1024_160k_cityscapes/bisenetv1_r18-d32_4x4_1024x1024_160k_cityscapes_20210922_172239-c55e78e2.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r18-d32_4x4_1024x1024_160k_cityscapes/bisenetv1_r18-d32_4x4_1024x1024_160k_cityscapes_20210922_172239.log.json)                                     |

| BiSeNetV1 | R-18-D32               | 1024x1024 |  160000 | 5.69     | 31.77          | V100   | 74.37 | 76.91         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/bisenetv1/bisenetv1_r18-d32-in1k-pre_4xb4-160k_cityscapes-1024x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r18-d32_in1k-pre_4x4_1024x1024_160k_cityscapes/bisenetv1_r18-d32_in1k-pre_4x4_1024x1024_160k_cityscapes_20210905_220251-8ba80eff.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r18-d32_in1k-pre_4x4_1024x1024_160k_cityscapes/bisenetv1_r18-d32_in1k-pre_4x4_1024x1024_160k_cityscapes_20210905_220251.log.json) |

| BiSeNetV1 | R-18-D32 (4x8)         | 1024x1024 |  160000 | 11.17    | 31.77          | V100   | 75.16 | 77.24         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/bisenetv1/bisenetv1_r18-d32-in1k-pre_4xb8-160k_cityscapes-1024x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r18-d32_in1k-pre_4x8_1024x1024_160k_cityscapes/bisenetv1_r18-d32_in1k-pre_4x8_1024x1024_160k_cityscapes_20210905_220322-bb8db75f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r18-d32_in1k-pre_4x8_1024x1024_160k_cityscapes/bisenetv1_r18-d32_in1k-pre_4x8_1024x1024_160k_cityscapes_20210905_220322.log.json) |

| BiSeNetV1 | R-50-D32 (No Pretrain) | 1024x1024 |  160000 | 15.39    | 7.71           | V100   | 76.92 | 78.87         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/bisenetv1/bisenetv1_r50-d32_4xb4-160k_cityscapes-1024x1024.py)          | [model](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r50-d32_4x4_1024x1024_160k_cityscapes/bisenetv1_r50-d32_4x4_1024x1024_160k_cityscapes_20210923_222639-7b28a2a6.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r50-d32_4x4_1024x1024_160k_cityscapes/bisenetv1_r50-d32_4x4_1024x1024_160k_cityscapes_20210923_222639.log.json)                                     |

| BiSeNetV1 | R-50-D32               | 1024x1024 |  160000 | 15.39    | 7.71           | V100   | 77.68 | 79.57         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/bisenetv1/bisenetv1_r50-d32-in1k-pre_4xb4-160k_cityscapes-1024x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r50-d32_in1k-pre_4x4_1024x1024_160k_cityscapes/bisenetv1_r50-d32_in1k-pre_4x4_1024x1024_160k_cityscapes_20210917_234628-8b304447.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r50-d32_in1k-pre_4x4_1024x1024_160k_cityscapes/bisenetv1_r50-d32_in1k-pre_4x4_1024x1024_160k_cityscapes_20210917_234628.log.json) |

| Method    | Backbone                | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                             | download                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |

| BiSeNetV1 | R-18-D32 (No Pretrain)  | 512x512   |  160000 | -        | -              | V100   | 25.45 | 26.15         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/bisenetv1/bisenetv1_r18-d32_4xb4-160k_coco-stuff164k-512x512.py)           | [model](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r18-d32_lr5e-3_4x4_512x512_160k_coco-stuff164k/bisenetv1_r18-d32_lr5e-3_4x4_512x512_160k_coco-stuff164k_20211022_054328-046aa2f2.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r18-d32_lr5e-3_4x4_512x512_160k_coco-stuff164k/bisenetv1_r18-d32_lr5e-3_4x4_512x512_160k_coco-stuff164k_20211022_054328.log.json)                                         |

| BiSeNetV1 | R-18-D32                | 512x512   |  160000 | 6.33     | 74.24          | V100   | 28.55 | 29.26         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/bisenetv1/bisenetv1_r18-d32-in1k-pre_4xb4-160k_coco-stuff164k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r18-d32_in1k-pre_lr5e-3_4x4_512x512_160k_coco-stuff164k/bisenetv1_r18-d32_in1k-pre_lr5e-3_4x4_512x512_160k_coco-stuff164k_20211023_013100-f700dbf7.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r18-d32_in1k-pre_lr5e-3_4x4_512x512_160k_coco-stuff164k/bisenetv1_r18-d32_in1k-pre_lr5e-3_4x4_512x512_160k_coco-stuff164k_20211023_013100.log.json)     |

| BiSeNetV1 | R-50-D32 (No Pretrain)  | 512x512   |  160000 | -        | -              | V100   | 29.82 | 30.33         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/bisenetv1/bisenetv1_r50-d32_4xb4-160k_coco-stuff164k-512x512.py)           | [model](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r50-d32_lr5e-3_4x4_512x512_160k_coco-stuff164k/bisenetv1_r50-d32_lr5e-3_4x4_512x512_160k_coco-stuff164k_20211101_040616-d2bb0df4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r50-d32_lr5e-3_4x4_512x512_160k_coco-stuff164k/bisenetv1_r50-d32_lr5e-3_4x4_512x512_160k_coco-stuff164k_20211101_040616.log.json)                                         |

| BiSeNetV1 | R-50-D32                | 512x512   |  160000 | 9.28     | 32.60          | V100   | 34.88 | 35.37         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/bisenetv1/bisenetv1_r50-d32-in1k-pre_4xb4-160k_coco-stuff164k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r50-d32_in1k-pre_lr5e-3_4x4_512x512_160k_coco-stuff164k/bisenetv1_r50-d32_in1k-pre_lr5e-3_4x4_512x512_160k_coco-stuff164k_20211101_181932-66747911.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r50-d32_in1k-pre_lr5e-3_4x4_512x512_160k_coco-stuff164k/bisenetv1_r50-d32_in1k-pre_lr5e-3_4x4_512x512_160k_coco-stuff164k_20211101_181932.log.json)     |

| BiSeNetV1 | R-101-D32 (No Pretrain) | 512x512   |  160000 | -        | -              | V100   | 31.14 | 31.76         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/bisenetv1/bisenetv1_r50-d32-in1k-pre_4xb4-160k_coco-stuff164k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r101-d32_lr5e-3_4x4_512x512_160k_coco-stuff164k/bisenetv1_r101-d32_lr5e-3_4x4_512x512_160k_coco-stuff164k_20211102_164147-c6b32c3b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r101-d32_lr5e-3_4x4_512x512_160k_coco-stuff164k/bisenetv1_r101-d32_lr5e-3_4x4_512x512_160k_coco-stuff164k_20211102_164147.log.json)                                     |

| BiSeNetV1 | R-101-D32               | 512x512   |  160000 | 10.36    | 25.25          | V100   | 37.38 | 37.99         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/bisenetv1/bisenetv1_r101-d32-in1k-pre_4xb4-160k_coco-stuff164k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r101-d32_in1k-pre_lr5e-3_4x4_512x512_160k_coco-stuff164k/bisenetv1_r101-d32_in1k-pre_lr5e-3_4x4_512x512_160k_coco-stuff164k_20211101_225220-28c8f092.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/bisenetv1/bisenetv1_r101-d32_in1k-pre_lr5e-3_4x4_512x512_160k_coco-stuff164k/bisenetv1_r101-d32_in1k-pre_lr5e-3_4x4_512x512_160k_coco-stuff164k_20211101_225220.log.json) |

Note:

- `4x8`: Using 4 GPUs with 8 samples per GPU in training.
- For BiSeNetV1 on Cityscapes dataset, default setting is 4 GPUs with 4 samples per GPU in training.
- `No Pretrain` means the model is trained from scratch.

```bibtex

```bibtex
@inproceedings{yu2018bisenet,
  title={Bisenet: Bilateral segmentation network for real-time semantic segmentation},
  author={Yu, Changqian and Wang, Jingbo and Peng, Chao and Gao, Changxin and Yu, Gang and Sang, Nong},
  pages={325--341},
}
```

# TSM

[TSM: Temporal Shift Module for Efficient Video Understanding](https://openaccess.thecvf.com/content_ICCV_2019/html/Lin_TSM_Temporal_Shift_Module_for_Efficient_Video_Understanding_ICCV_2019_paper.html)

The explosive growth in video streaming gives rise to challenges on performing video understanding at high accuracy and low computation cost. Conventional 2D CNNs are computationally cheap but cannot capture temporal relationships; 3D CNN based methods can achieve good performance but are computationally intensive, making it expensive to deploy. In this paper, we propose a generic and effective Temporal Shift Module (TSM) that enjoys both high efficiency and high performance. Specifically, it can achieve the performance of 3D CNN but maintain 2D CNN's complexity. TSM shifts part of the channels along the temporal dimension; thus facilitate information exchanged among neighboring frames. It can be inserted into 2D CNNs to achieve temporal modeling at zero computation and zero parameters. We also extended TSM to online setting, which enables real-time low-latency online video recognition and video object detection. TSM is accurate and efficient: it ranks the first place on the Something-Something leaderboard upon publication; on Jetson Nano and Galaxy Note8, it achieves a low latency of 13ms and 35ms for online video recognition.

<img src="https://user-images.githubusercontent.com/34324155/143019083-abc0de39-9ea1-4175-be5c-073c90de64c3.png" width="800"/>
</div>

| frame sampling strategy | resolution | gpus |           backbone            | pretrain | top1 acc | top5 acc |  testing protocol  | FLOPs  | params |            config            |                       ckpt |                        log |
| :---------------------: | :--------: | :--: | :---------------------------: | :------: | :------: | :------: | :----------------: | :----: | :----: | :--------------------------: | -------------------------: | -------------------------: |

|          1x1x8          |  224x224   |  8   |           ResNet50            | ImageNet |  73.18   |  90.56   | 8 clips x 10 crop  | 32.88G | 23.87M | [config](/configs/recognition/tsm/tsm_imagenet-pretrained-r50_8xb16-1x1x8-50e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-r50_8xb16-1x1x8-50e_kinetics400-rgb/tsm_imagenet-pretrained-r50_8xb16-1x1x8-50e_kinetics400-rgb_20220831-64d69186.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-r50_8xb16-1x1x8-50e_kinetics400-rgb/tsm_imagenet-pretrained-r50_8xb16-1x1x8-50e_kinetics400-rgb.log) |

|          1x1x8          |  224x224   |  8   |           ResNet50            | ImageNet |  73.22   |  90.22   | 8 clips x 10 crop  | 32.88G | 23.87M | [config](/configs/recognition/tsm/tsm_imagenet-pretrained-r50_8xb16-1x1x8-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-r50_8xb16-1x1x8-100e_kinetics400-rgb/tsm_imagenet-pretrained-r50_8xb16-1x1x8-100e_kinetics400-rgb_20220831-a6db1e5d.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-r50_8xb16-1x1x8-100e_kinetics400-rgb/tsm_imagenet-pretrained-r50_8xb16-1x1x8-100e_kinetics400-rgb.log) |

|         1x1x16          |  224x224   |  8   |           ResNet50            | ImageNet |  75.12   |  91.55   | 16 clips x 10 crop | 65.75G | 23.87M | [config](/configs/recognition/tsm/tsm_imagenet-pretrained-r50_8xb16-1x1x16-50e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-r50_8xb16-1x1x16-50e_kinetics400-rgb/tsm_imagenet-pretrained-r50_8xb16-1x1x16-50e_kinetics400-rgb_20220831-042b1748.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-r50_8xb16-1x1x16-50e_kinetics400-rgb/tsm_imagenet-pretrained-r50_8xb16-1x1x16-50e_kinetics400-rgb.log) |

|      1x1x8 (dense)      |  224x224   |  8   |           ResNet50            | ImageNet |  73.38   |  90.78   | 8 clips x 10 crop  | 32.88G | 23.87M | [config](/configs/recognition/tsm/tsm_imagenet-pretrained-r50_8xb16-dense-1x1x8-50e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-r50_8xb16-dense-1x1x8-50e_kinetics400-rgb/tsm_imagenet-pretrained-r50_8xb16-dense-1x1x8-50e_kinetics400-rgb_20220831-f55d3c2b.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-r50_8xb16-dense-1x1x8-50e_kinetics400-rgb/tsm_imagenet-pretrained-r50_8xb16-dense-1x1x8-50e_kinetics400-rgb.log) |

|          1x1x8          |  224x224   |  8   | ResNet50 (NonLocalDotProduct) | ImageNet |  74.49   |  91.15   | 8 clips x 10 crop  | 61.30G | 31.68M | [config](/configs/recognition/tsm/tsm_imagenet-pretrained-r50-nl-dot-product_8xb16-1x1x8-50e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-r50-nl-dot-product_8xb16-1x1x8-50e_kinetics400-rgb/tsm_imagenet-pretrained-r50-nl-dot-product_8xb16-1x1x8-50e_kinetics400-rgb_20220831-108bfde5.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-r50-nl-dot-product_8xb16-1x1x8-50e_kinetics400-rgb/tsm_imagenet-pretrained-r50-nl-dot-product_8xb16-1x1x8-50e_kinetics400-rgb.log) |

|          1x1x8          |  224x224   |  8   |   ResNet50 (NonLocalGauss)    | ImageNet |  73.66   |  90.99   | 8 clips x 10 crop  | 59.06G | 28.00M | [config](/configs/recognition/tsm/tsm_imagenet-pretrained-r50-nl-gaussian_8xb16-1x1x8-50e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-r50-nl-gaussian_8xb16-1x1x8-50e_kinetics400-rgb/tsm_imagenet-pretrained-r50-nl-gaussian_8xb16-1x1x8-50e_kinetics400-rgb_20220831-7e54dacf.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-r50-nl-gaussian_8xb16-1x1x8-50e_kinetics400-rgb/tsm_imagenet-pretrained-r50-nl-gaussian_8xb16-1x1x8-50e_kinetics400-rgb.log) |

|          1x1x8          |  224x224   |  8   | ResNet50 (NonLocalEmbedGauss) | ImageNet |  74.34   |  91.23   | 8 clips x 10 crop  | 61.30G | 31.68M | [config](/configs/recognition/tsm/tsm_imagenet-pretrained-r50-nl-embedded-gaussian_8xb16-1x1x8-50e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-r50-nl-embedded-gaussian_8xb16-1x1x8-50e_kinetics400-rgb/tsm_imagenet-pretrained-r50-nl-embedded-gaussian_8xb16-1x1x8-50e_kinetics400-rgb_20220831-35eddb57.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-r50-nl-embedded-gaussian_8xb16-1x1x8-50e_kinetics400-rgb/tsm_imagenet-pretrained-r50-nl-embedded-gaussian_8xb16-1x1x8-50e_kinetics400-rgb.log) |

|          1x1x8          |  224x224   |  8   |          MobileNetV2          | ImageNet |  68.71   |  88.32   |  8 clips x 3 crop  | 3.269G | 2.736M | [config](/configs/recognition/tsm/tsm_imagenet-pretrained-mobilenetv2_8xb16-1x1x8-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-mobilenetv2_8xb16-1x1x8-100e_kinetics400-rgb/tsm_imagenet-pretrained-mobilenetv2_8xb16-1x1x8-100e_kinetics400-rgb_20230414-401127fd.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-mobilenetv2_8xb16-1x1x8-100e_kinetics400-rgb/tsm_imagenet-pretrained-mobilenetv2_8xb16-1x1x8-100e_kinetics400-rgb.log) |

### Something-something V2

| frame sampling strategy | resolution | gpus | backbone  | pretrain | top1 acc | top5 acc | testing protocol  | FLOPs  | params |               config                |               ckpt                |                log                |

| :---------------------: | :--------: | :--: | :-------: | :------: | :------: | :------: | :---------------: | :----: | :----: | :---------------------------------: | :-------------------------------: | :-------------------------------: |
|          1x1x8          |  224x224   |  8   | ResNet50  | ImageNet |  62.72   |  87.70   | 8 clips x 3 crop  | 32.88G | 23.87M | [config](/configs/recognition/tsm/tsm_imagenet-pretrained-r50_8xb16-1x1x8-50e_sthv2-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-r50_8xb16-1x1x8-50e_sthv2-rgb/tsm_imagenet-pretrained-r50_8xb16-1x1x8-50e_sthv2-rgb_20230317-be0fc26e.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-r50_8xb16-1x1x8-50e_sthv2-rgb/tsm_imagenet-pretrained-r50_8xb16-1x1x8-50e_sthv2-rgb.log) |

|         1x1x16          |  224x224   |  8   | ResNet50  | ImageNet |  64.16   |  88.61   | 16 clips x 3 crop | 65.75G | 23.87M | [config](/configs/recognition/tsm/tsm_imagenet-pretrained-r50_8xb16-1x1x16-50e_sthv2-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-r50_8xb16-1x1x16-50e_sthv2-rgb/tsm_imagenet-pretrained-r50_8xb16-1x1x16-50e_sthv2-rgb_20230317-ec6696ad.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-r50_8xb16-1x1x16-50e_sthv2-rgb/tsm_imagenet-pretrained-r50_8xb16-1x1x16-50e_sthv2-rgb.log) |

|          1x1x8          |  224x224   |  8   | ResNet101 | ImageNet |  63.70   |  88.28   | 8 clips x 3 crop  | 62.66G | 42.86M | [config](/configs/recognition/tsm/tsm_imagenet-pretrained-r101_8xb16-1x1x8-50e_sthv2-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-r101_8xb16-1x1x8-50e_sthv2-rgb/tsm_imagenet-pretrained-r101_8xb16-1x1x8-50e_sthv2-rgb_20230320-efcc0d1b.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tsm/tsm_imagenet-pretrained-r101_8xb16-1x1x8-50e_sthv2-rgb/tsm_imagenet-pretrained-r101_8xb16-1x1x8-50e_sthv2-rgb.log) |

## Train

```shell
```

Example: train TSM model on Kinetics-400 dataset in a deterministic option.

```shell
python tools/train.py configs/recognition/tsm/tsm_imagenet-pretrained-r50_8xb16-1x1x8-50e_kinetics400-rgb.py \

```shell
     --seed=0 --deterministic
```

## Test

```shell
```

Example: test TSM model on Kinetics-400 dataset and dump the result to a pkl file.

```shell
python tools/test.py configs/recognition/tsm/tsm_imagenet-pretrained-r50_8xb16-1x1x8-50e_kinetics400-rgb.py \
```

```BibTeX
@inproceedings{lin2019tsm,

```

```BibTeX
  title={TSM: Temporal Shift Module for Efficient Video Understanding},
  author={Lin, Ji and Gan, Chuang and Han, Song},
}
```

```BibTeX
@article{Nonlocal2018,
}
```

## **v1.0.0rc1 (14/10/2022)**

- Release RTMPose, a high-performance real-time pose estimation algorithm with cross-platform deployment and inference support. See details at the [project page](/projects/rtmpose/)
- Support several new algorithms: ViTPose (arXiv'2022), CID (CVPR'2022), DEKR (CVPR'2021)
- Add Inferencer, a convenient inference interface that perform pose estimation and visualization on images, videos and webcam streams with only one line of code
- Introduce *Project*, a new form for rapid and easy implementation of new algorithms and features in MMPose, which is more handy for community contributors

- Support RTMPose ([#1971](https://github.com/open-mmlab/mmpose/pull/1971), [#2024](https://github.com/open-mmlab/mmpose/pull/2024), [#2028](https://github.com/open-mmlab/mmpose/pull/2028), [#2030](https://github.com/open-mmlab/mmpose/pull/2030), [#2040](https://github.com/open-mmlab/mmpose/pull/2040), [#2057](https://github.com/open-mmlab/mmpose/pull/2057))
- Support Inferencer ([#1969](https://github.com/open-mmlab/mmpose/pull/1969))
- Support ViTPose ([#1876](https://github.com/open-mmlab/mmpose/pull/1876), [#2056](https://github.com/open-mmlab/mmpose/pull/2056), [#2058](https://github.com/open-mmlab/mmpose/pull/2058)， [#2065](https://github.com/open-mmlab/mmpose/pull/2065))
- Support CID ([#1907](https://github.com/open-mmlab/mmpose/pull/1907))

- Support DEKR ([#1834](https://github.com/open-mmlab/mmpose/pull/1834), [#1901](https://github.com/open-mmlab/mmpose/pull/1901))
- Support training with multiple datasets ([#1767](https://github.com/open-mmlab/mmpose/pull/1767), [#1930](https://github.com/open-mmlab/mmpose/pull/1930), [#1938](https://github.com/open-mmlab/mmpose/pull/1938), [#2025](https://github.com/open-mmlab/mmpose/pull/2025))
- Add *project* to allow rapid and easy implementation of new models and features ([#1914](https://github.com/open-mmlab/mmpose/pull/1914))

- Improve documentation quality ([#1846](https://github.com/open-mmlab/mmpose/pull/1846), [#1858](https://github.com/open-mmlab/mmpose/pull/1858), [#1872](https://github.com/open-mmlab/mmpose/pull/1872), [#1899](https://github.com/open-mmlab/mmpose/pull/1899), [#1925](https://github.com/open-mmlab/mmpose/pull/1925), [#1945](https://github.com/open-mmlab/mmpose/pull/1945), [#1952](https://github.com/open-mmlab/mmpose/pull/1952), [#1990](https://github.com/open-mmlab/mmpose/pull/1990), [#2023](https://github.com/open-mmlab/mmpose/pull/2023), [#2042](https://github.com/open-mmlab/mmpose/pull/2042))
- Support visualizing keypoint indices ([#2051](https://github.com/open-mmlab/mmpose/pull/2051))
- Support OpenPose style visualization ([#2055](https://github.com/open-mmlab/mmpose/pull/2055))

- Accelerate image transpose in data pipelines with tensor operation ([#1976](https://github.com/open-mmlab/mmpose/pull/1976))
- Support auto-import modules from registry ([#1961](https://github.com/open-mmlab/mmpose/pull/1961))
- Support keypoint partition metric ([#1944](https://github.com/open-mmlab/mmpose/pull/1944))
- Support SimCC 1D-heatmap visualization ([#1912](https://github.com/open-mmlab/mmpose/pull/1912))
- Support saving predictions and data metainfo in demos ([#1814](https://github.com/open-mmlab/mmpose/pull/1814), [#1879](https://github.com/open-mmlab/mmpose/pull/1879))
- Support SimCC with DARK ([#1870](https://github.com/open-mmlab/mmpose/pull/1870))
- Remove Gaussian blur for offset maps in UDP-regress ([#1815](https://github.com/open-mmlab/mmpose/pull/1815))

- Refactor encoding interface of Codec for better extendibility and easier configuration ([#1781](https://github.com/open-mmlab/mmpose/pull/1781))
- Support evaluating CocoMetric without annotation file ([#1722](https://github.com/open-mmlab/mmpose/pull/1722))
- Improve unit tests ([#1765](https://github.com/open-mmlab/mmpose/pull/1765))

- Fix repeated warnings from different ranks ([#2053](https://github.com/open-mmlab/mmpose/pull/2053))
- Avoid frequent scope switching when using mmdet inference api ([#2039](https://github.com/open-mmlab/mmpose/pull/2039))
- Remove EMA parameters and message hub data when publishing model checkpoints ([#2036](https://github.com/open-mmlab/mmpose/pull/2036))

- Fix metainfo copying in dataset class ([#2017](https://github.com/open-mmlab/mmpose/pull/2017))
- Fix top-down demo bug when there is no object detected ([#2007](https://github.com/open-mmlab/mmpose/pull/2007))
- Fix config errors ([#1882](https://github.com/open-mmlab/mmpose/pull/1882), [#1906](https://github.com/open-mmlab/mmpose/pull/1906), [#1995](https://github.com/open-mmlab/mmpose/pull/1995))
- Fix image demo failure when GUI is unavailable ([#1968](https://github.com/open-mmlab/mmpose/pull/1968))
- Fix bug in AdaptiveWingLoss ([#1953](https://github.com/open-mmlab/mmpose/pull/1953))
- Fix incorrect importing of RepeatDataset which is deprecated ([#1943](https://github.com/open-mmlab/mmpose/pull/1943))

- Fix bug in bottom-up datasets that ignores images without instances ([#1752](https://github.com/open-mmlab/mmpose/pull/1752), [#1936](https://github.com/open-mmlab/mmpose/pull/1936))
- Fix upstream dependency issues ([#1867](https://github.com/open-mmlab/mmpose/pull/1867), [#1921](https://github.com/open-mmlab/mmpose/pull/1921))
- Fix evaluation issues and update results ([#1763](https://github.com/open-mmlab/mmpose/pull/1763), [#1773](https://github.com/open-mmlab/mmpose/pull/1773), [#1780](https://github.com/open-mmlab/mmpose/pull/1780), [#1850](https://github.com/open-mmlab/mmpose/pull/1850), [#1868](https://github.com/open-mmlab/mmpose/pull/1868))
- Fix local registry missing warnings ([#1849](https://github.com/open-mmlab/mmpose/pull/1849))
- Remove deprecated scripts for model deployment ([#1845](https://github.com/open-mmlab/mmpose/pull/1845))

- Fix a bug in input transformation in BaseHead ([#1843](https://github.com/open-mmlab/mmpose/pull/1843))
- Fix an interface mismatch with MMDetection in webcam demo ([#1813](https://github.com/open-mmlab/mmpose/pull/1813))
- Fix a bug in heatmap visualization that causes incorrect scale ([#1800](https://github.com/open-mmlab/mmpose/pull/1800))
- Add model metafiles ([#1768](https://github.com/open-mmlab/mmpose/pull/1768))

## **v1.0.0rc0 (14/10/2022)**

- Support 4 light-weight pose estimation algorithms: [SimCC](https://doi.org/10.48550/arxiv.2107.03332) (ECCV'2022), [Debias-IPR](https://openaccess.thecvf.com/content/ICCV2021/papers/Gu_Removing_the_Bias_of_Integral_Pose_Regression_ICCV_2021_paper.pdf) (ICCV'2021), [IPR](https://arxiv.org/abs/1711.08229) (ECCV'2018), and [DSNT](https://arxiv.org/abs/1801.07372v2) (ArXiv'2018) ([#1628](https://github.com/open-mmlab/mmpose/pull/1628))

**Migrations**

- Add Webcam API in MMPose 1.0 ([#1638](https://github.com/open-mmlab/mmpose/pull/1638), [#1662](https://github.com/open-mmlab/mmpose/pull/1662)) @Ben-Louis
- Add codec for Associative Embedding (beta) ([#1603](https://github.com/open-mmlab/mmpose/pull/1603)) @ly015

- Add a colab tutorial for MMPose 1.0 ([#1660](https://github.com/open-mmlab/mmpose/pull/1660)) @Tau-J

- Add model index in config folder ([#1710](https://github.com/open-mmlab/mmpose/pull/1710), [#1709](https://github.com/open-mmlab/mmpose/pull/1709), [#1627](https://github.com/open-mmlab/mmpose/pull/1627)) @ly015, @Tau-J, @Ben-Louis
- Update and improve documentation ([#1692](https://github.com/open-mmlab/mmpose/pull/1692), [#1656](https://github.com/open-mmlab/mmpose/pull/1656), [#1681](https://github.com/open-mmlab/mmpose/pull/1681), [#1677](https://github.com/open-mmlab/mmpose/pull/1677), [#1664](https://github.com/open-mmlab/mmpose/pull/1664), [#1659](https://github.com/open-mmlab/mmpose/pull/1659)) @Tau-J, @Ben-Louis, @liqikai9
- Improve config structures and formats ([#1651](https://github.com/open-mmlab/mmpose/pull/1651)) @liqikai9

- Update mmengine version requirements ([#1715](https://github.com/open-mmlab/mmpose/pull/1715)) @Ben-Louis

- Update dependencies of pre-commit hooks ([#1705](https://github.com/open-mmlab/mmpose/pull/1705)) @Ben-Louis
- Fix mmcv version in DockerFile ([#1704](https://github.com/open-mmlab/mmpose/pull/1704))
- Fix a bug in setting dataset metainfo in configs ([#1684](https://github.com/open-mmlab/mmpose/pull/1684)) @ly015
- Fix a bug in UDP training ([#1682](https://github.com/open-mmlab/mmpose/pull/1682)) @liqikai9
- Fix a bug in Dark decoding ([#1676](https://github.com/open-mmlab/mmpose/pull/1676)) @liqikai9
- Fix bugs in visualization ([#1671](https://github.com/open-mmlab/mmpose/pull/1671), [#1668](https://github.com/open-mmlab/mmpose/pull/1668), [#1657](https://github.com/open-mmlab/mmpose/pull/1657)) @liqikai9, @Ben-Louis
- Fix incorrect flops calculation ([#1669](https://github.com/open-mmlab/mmpose/pull/1669)) @liqikai9

- Fix `tensor.tile` compatibility issue for pytorch 1.6 ([#1658](https://github.com/open-mmlab/mmpose/pull/1658)) @ly015
- Fix compatibility with `MultilevelPixelData` ([#1647](https://github.com/open-mmlab/mmpose/pull/1647)) @liqikai9

## **v1.0.0beta (1/09/2022)**

We are excited to announce the release of MMPose 1.0.0beta.
MMPose 1.0.0beta is the first version of MMPose 1.x, a part of the OpenMMLab 2.0 projects.

- **New engines**. MMPose 1.x is based on [MMEngine](https://github.com/open-mmlab/mmengine), which provides a general and powerful runner that allows more flexible customizations and significantly simplifies the entrypoints of high-level interfaces.

- **Unified interfaces**. As a part of the OpenMMLab 2.0 projects, MMPose 1.x unifies and refactors the interfaces and internal logics of train, testing, datasets, models, evaluation, and visualization. All the OpenMMLab 2.0 projects share the same design in those interfaces and logics to allow the emergence of multi-task/modality algorithms.

- **More documentation and tutorials**. We add a bunch of documentation and tutorials to help users get started more smoothly. Read it [here](https://mmpose.readthedocs.io/en/latest/).

## **v0.28.1 (28/07/2022)**

This release is meant to fix the compatibility with the latest mmcv v1.6.1

## **v0.28.0 (06/07/2022)**

- Support [TCFormer](https://openaccess.thecvf.com/content/CVPR2022/html/Zeng_Not_All_Tokens_Are_Equal_Human-Centric_Visual_Analysis_via_Token_CVPR_2022_paper.html) backbone, CVPR'2022 ([#1447](https://github.com/open-mmlab/mmpose/pull/1447), [#1452](https://github.com/open-mmlab/mmpose/pull/1452)) @zengwang430521

- Add [RLE](https://arxiv.org/abs/2107.11291) models on COCO dataset ([#1424](https://github.com/open-mmlab/mmpose/pull/1424)) @Indigo6, @Ben-Louis, @ly015

- Update swin models with better performance ([#1467](https://github.com/open-mmlab/mmpose/pull/1434)) @jin-s13

- Support layer decay optimizer constructor and learning rate decay optimizer constructor ([#1423](https://github.com/open-mmlab/mmpose/pull/1423)) @jin-s13

- Improve documentation quality ([#1416](https://github.com/open-mmlab/mmpose/pull/1416), [#1421](https://github.com/open-mmlab/mmpose/pull/1421), [#1423](https://github.com/open-mmlab/mmpose/pull/1423), [#1426](https://github.com/open-mmlab/mmpose/pull/1426), [#1458](https://github.com/open-mmlab/mmpose/pull/1458), [#1463](https://github.com/open-mmlab/mmpose/pull/1463)) @ly015, @liqikai9

- Support installation by [mim](https://github.com/open-mmlab/mim) ([#1425](https://github.com/open-mmlab/mmpose/pull/1425)) @liqikai9

- Support PAVI logger ([#1434](https://github.com/open-mmlab/mmpose/pull/1434)) @EvelynWang-0423

- Add progress bar for some demos ([#1454](https://github.com/open-mmlab/mmpose/pull/1454)) @liqikai9

- Webcam API supports quick device setting in terminal commands ([#1466](https://github.com/open-mmlab/mmpose/pull/1466)) @ly015

- Rename `custom_hooks_config` to `custom_hooks` in configs to align with the documentation ([#1427](https://github.com/open-mmlab/mmpose/pull/1427)) @ly015

- Fix deadlock issue in Webcam API ([#1430](https://github.com/open-mmlab/mmpose/pull/1430)) @ly015

- Fix smoother configs in video 3D demo ([#1457](https://github.com/open-mmlab/mmpose/pull/1457)) @ly015

## **v0.27.0 (07/06/2022)**

- Support hand gesture recognition

  - Try the demo for gesture recognition

  - Learn more about the algorithm, dataset and experiment results

- Major upgrade to the Webcam API

  - Tutorials (EN|zh_CN)
  - [API Reference](https://mmpose.readthedocs.io/en/latest/api.html#mmpose-apis-webcam)
  - Demo

- Support gesture recognition algorithm [MTUT](https://openaccess.thecvf.com/content_CVPR_2019/html/Abavisani_Improving_the_Performance_of_Unimodal_Dynamic_Hand-Gesture_Recognition_With_Multimodal_CVPR_2019_paper.html) CVPR'2019 and dataset [NVGesture](https://openaccess.thecvf.com/content_cvpr_2016/html/Molchanov_Online_Detection_and_CVPR_2016_paper.html) CVPR'2016 ([#1380](https://github.com/open-mmlab/mmpose/pull/1380)) @Ben-Louis

- Upgrade Webcam API and related documents ([#1393](https://github.com/open-mmlab/mmpose/pull/1393), [#1404](https://github.com/open-mmlab/mmpose/pull/1404), [#1413](https://github.com/open-mmlab/mmpose/pull/1413)) @ly015

- Support exporting COCO inference result without the annotation file ([#1368](https://github.com/open-mmlab/mmpose/pull/1368)) @liqikai9

- Replace markdownlint with mdformat in CI to avoid the dependence on ruby [#1382](https://github.com/open-mmlab/mmpose/pull/1382) @ly015

- Improve documentation quality ([#1385](https://github.com/open-mmlab/mmpose/pull/1385), [#1394](https://github.com/open-mmlab/mmpose/pull/1394), [#1395](https://github.com/open-mmlab/mmpose/pull/1395), [#1408](https://github.com/open-mmlab/mmpose/pull/1408)) @chubei-oppen, @ly015, @liqikai9

- Fix xywh->xyxy bbox conversion in dataset sanity check ([#1367](https://github.com/open-mmlab/mmpose/pull/1367)) @jin-s13

- Fix a bug in two-stage 3D keypoint demo ([#1373](https://github.com/open-mmlab/mmpose/pull/1373)) @ly015

- Fix out-dated settings in PVT configs ([#1376](https://github.com/open-mmlab/mmpose/pull/1376)) @ly015

- Fix myst settings for document compiling ([#1381](https://github.com/open-mmlab/mmpose/pull/1381)) @ly015

- Fix a bug in bbox transform ([#1384](https://github.com/open-mmlab/mmpose/pull/1384)) @ly015

- Fix inaccurate description of `min_keypoints` in tracking apis ([#1398](https://github.com/open-mmlab/mmpose/pull/1398)) @pallgeuer

- Fix warning with `torch.meshgrid` ([#1402](https://github.com/open-mmlab/mmpose/pull/1402)) @pallgeuer

- Remove redundant transformer modules from `mmpose.datasets.backbones.utils` ([#1405](https://github.com/open-mmlab/mmpose/pull/1405)) @ly015

## **v0.26.0 (05/05/2022)**

- Support [RLE (Residual Log-likelihood Estimation)](https://arxiv.org/abs/2107.11291), ICCV'2021 ([#1259](https://github.com/open-mmlab/mmpose/pull/1259)) @Indigo6, @ly015

- Support [Swin Transformer](https://arxiv.org/abs/2103.14030), ICCV'2021 ([#1300](https://github.com/open-mmlab/mmpose/pull/1300)) @yumendecc, @ly015

- Support [PVT](https://arxiv.org/abs/2102.12122), ICCV'2021 and [PVTv2](https://arxiv.org/abs/2106.13797), CVMJ'2022 ([#1343](https://github.com/open-mmlab/mmpose/pull/1343)) @zengwang430521

- Speed up inference and reduce CPU usage by optimizing the pre-processing pipeline ([#1320](https://github.com/open-mmlab/mmpose/pull/1320)) @chenxinfeng4, @liqikai9

- Support [FPN](https://openaccess.thecvf.com/content_cvpr_2017/html/Lin_Feature_Pyramid_Networks_CVPR_2017_paper.html), CVPR'2017 ([#1300](https://github.com/open-mmlab/mmpose/pull/1300)) @yumendecc, @ly015

- Video demo supports models that requires multi-frame inputs ([#1300](https://github.com/open-mmlab/mmpose/pull/1300)) @liqikai9, @jin-s13

- Update benchmark regression list ([#1328](https://github.com/open-mmlab/mmpose/pull/1328)) @ly015, @liqikai9

- Remove unnecessary warnings in `TopDownPoseTrack18VideoDataset` ([#1335](https://github.com/open-mmlab/mmpose/pull/1335)) @liqikai9

- Improve documentation quality ([#1313](https://github.com/open-mmlab/mmpose/pull/1313), [#1305](https://github.com/open-mmlab/mmpose/pull/1305)) @Ben-Louis, @ly015

- Update deprecating settings in configs ([#1317](https://github.com/open-mmlab/mmpose/pull/1317)) @ly015

- Fix a bug in human skeleton grouping that may skip the matching process unexpectedly when `ignore_to_much` is True ([#1341](https://github.com/open-mmlab/mmpose/pull/1341)) @daixinghome

- Fix a GPG key error that leads to CI failure ([#1354](https://github.com/open-mmlab/mmpose/pull/1354)) @ly015

- Fix bugs in distributed training script ([#1338](https://github.com/open-mmlab/mmpose/pull/1338), [#1298](https://github.com/open-mmlab/mmpose/pull/1298)) @ly015

- Fix an upstream bug in xtoccotools that causes incorrect AP(M) results ([#1308](https://github.com/open-mmlab/mmpose/pull/1308)) @jin-s13, @ly015

- Fix indentiation errors in the colab tutorial ([#1298](https://github.com/open-mmlab/mmpose/pull/1298)) @YuanZi1501040205

- Fix incompatible model weight initialization with other OpenMMLab codebases ([#1329](https://github.com/open-mmlab/mmpose/pull/1329)) @274869388

- Fix HRNet FP16 checkpoints download URL ([#1309](https://github.com/open-mmlab/mmpose/pull/1309)) @YinAoXiong

- Fix typos in `body3d_two_stage_video_demo.py` ([#1295](https://github.com/open-mmlab/mmpose/pull/1295)) @mucozcan

- Refactor bbox processing in datasets and pipelines ([#1311](https://github.com/open-mmlab/mmpose/pull/1311)) @ly015, @Ben-Louis

- The bbox format conversion (xywh to center-scale) and random translation are moved from the dataset to the pipeline. The comparison between new and old version is as below:

v0.26.0v0.25.0Dataset
(e.g. [TopDownCOCODataset](https://github.com/open-mmlab/mmpose/blob/master/mmpose/datasets/datasets/top_down/topdown_coco_dataset.py))

... # Data sample only contains bbox rec.append({    'bbox': obj\['clean_bbox\]\[:4\],    ... })

</td>

<td  valign='top'>

... # Convert bbox from xywh to center-scale center, scale = self.\_xywh2cs(\*obj\['clean_bbox'\]\[:4\]) # Data sample contains center and scale rec.append({    'bbox': obj\['clean_bbox\]\[:4\],    'center': center,    'scale': scale,    ... })

</td>

</tr>

<tr>

<th>Pipeline Config

(e.g. [HRNet+COCO](https://github.com/open-mmlab/mmpose/blob/master/configs/body/2d_kpt_sview_rgb_img/topdown_heatmap/coco/hrnet_w32_coco_256x192.py))</th>

<td valign='top'>

... train_pipeline = \[    dict(type='LoadImageFromFile'),    # Convert bbox from xywh to center-scale    dict(type='TopDownGetBboxCenterScale', padding=1.25),    # Randomly shift bbox center    dict(type='TopDownRandomShiftBboxCenter', shift_factor=0.16, prob=0.3),    ... \]

</td>

... train_pipeline = \[    dict(type='LoadImageFromFile'),    ... \]

</td>

</tr>

<tr>

<th>Advantage</th>

<li>Simpler data sample content</li>

<li>Flexible bbox format conversion and augmentation</li>

<li>Apply bbox random translation every epoch (instead of only applying once at the annotation loading)

</td>

<td valign='top'>-</td>

</tr>

<tr>

<th>BC Breaking</th>

</td>

</tr>

<tr>

<td valign='top'>The method `_xywh2cs` of dataset base classes (e.g. [Kpt2dSviewRgbImgTopDownDataset](https://github.com/open-mmlab/mmpose/blob/master/mmpose/datasets/datasets/base/kpt_2d_sview_rgb_img_top_down_dataset.py)) will be deprecated in the future. Custom datasets will need modifications to move the bbox format conversion to pipelines.</td>

</tr>

</tbody>

</table>

## **v0.25.0 (02/04/2022)**

- Support Shelf and Campus datasets with pre-trained VoxelPose models, ["3D Pictorial Structures for Multiple Human Pose Estimation"](http://campar.in.tum.de/pub/belagiannis2014cvpr/belagiannis2014cvpr.pdf), CVPR'2014 ([#1225](https://github.com/open-mmlab/mmpose/pull/1225)) @liqikai9, @wusize

- Add `Smoother` module for temporal smoothing of the pose estimation with configurable filters ([#1127](https://github.com/open-mmlab/mmpose/pull/1127)) @ailingzengzzz, @ly015

- Support SmoothNet for pose smoothing, ["SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videos"](https://arxiv.org/abs/2112.13715), arXiv'2021 ([#1279](https://github.com/open-mmlab/mmpose/pull/1279)) @ailingzengzzz, @ly015

- Add multiview 3D pose estimation demo ([#1270](https://github.com/open-mmlab/mmpose/pull/1270)) @wusize

- Support multi-machine distributed training ([#1248](https://github.com/open-mmlab/mmpose/pull/1248)) @ly015

- Update HRFormer configs and checkpoints with relative position bias ([#1245](https://github.com/open-mmlab/mmpose/pull/1245)) @zengwang430521

- Support using different random seed for each distributed node ([#1257](https://github.com/open-mmlab/mmpose/pull/1257), [#1229](https://github.com/open-mmlab/mmpose/pull/1229)) @ly015

- Improve documentation quality ([#1275](https://github.com/open-mmlab/mmpose/pull/1275), [#1255](https://github.com/open-mmlab/mmpose/pull/1255), [#1258](https://github.com/open-mmlab/mmpose/pull/1258), [#1249](https://github.com/open-mmlab/mmpose/pull/1249), [#1247](https://github.com/open-mmlab/mmpose/pull/1247), [#1240](https://github.com/open-mmlab/mmpose/pull/1240), [#1235](https://github.com/open-mmlab/mmpose/pull/1235)) @ly015, @jin-s13, @YoniChechik

- Fix keypoint index in RHD dataset meta information ([#1265](https://github.com/open-mmlab/mmpose/pull/1265)) @liqikai9

- Fix pre-commit hook unexpected behavior on Windows ([#1282](https://github.com/open-mmlab/mmpose/pull/1282)) @liqikai9

- Remove python-dev installation in CI ([#1276](https://github.com/open-mmlab/mmpose/pull/1276)) @ly015

- Unify hyphens in argument names in tools and demos ([#1271](https://github.com/open-mmlab/mmpose/pull/1271)) @ly015

- Fix ambiguous channel size in `channel_shuffle` that may cause exporting failure (#1242) @PINTO0309

- Fix a bug in Webcam API that causes single-class detectors fail ([#1239](https://github.com/open-mmlab/mmpose/pull/1239)) @674106399

- Fix the issue that `custom_hook` can not be set in configs ([#1236](https://github.com/open-mmlab/mmpose/pull/1236)) @bladrome

- Fix incompatible MMCV version in DockerFile ([#raykindle](https://github.com/open-mmlab/mmpose/pull/raykindle))

- Skip invisible joints in visualization ([#1228](https://github.com/open-mmlab/mmpose/pull/1228)) @womeier

## **v0.24.0 (07/03/2022)**

- Support HRFormer ["HRFormer: High-Resolution Vision Transformer for Dense Predict"](https://proceedings.neurips.cc/paper/2021/hash/3bbfdde8842a5c44a0323518eec97cbe-Abstract.html), NeurIPS'2021 ([#1203](https://github.com/open-mmlab/mmpose/pull/1203)) @zengwang430521

- Support Windows installation with pip ([#1213](https://github.com/open-mmlab/mmpose/pull/1213)) @jin-s13, @ly015

- Add WebcamAPI documents ([#1187](https://github.com/open-mmlab/mmpose/pull/1187)) @ly015

- Support CPU training with mmcv \< v1.4.4 ([#1161](https://github.com/open-mmlab/mmpose/pull/1161)) @EasonQYS, @ly015

- Add "Valentine Magic" demo with WebcamAPI ([#1189](https://github.com/open-mmlab/mmpose/pull/1189), [#1191](https://github.com/open-mmlab/mmpose/pull/1191)) @liqikai9

- Refactor multi-view 3D pose estimation framework towards better modularization and expansibility ([#1196](https://github.com/open-mmlab/mmpose/pull/1196)) @wusize

- Add WebcamAPI documents and tutorials ([#1187](https://github.com/open-mmlab/mmpose/pull/1187)) @ly015

- Refactor dataset evaluation interface to align with other OpenMMLab codebases ([#1209](https://github.com/open-mmlab/mmpose/pull/1209)) @ly015

- Add deprecation message for deploy tools since [MMDeploy](https://github.com/open-mmlab/mmdeploy) has supported MMPose ([#1207](https://github.com/open-mmlab/mmpose/pull/1207)) @QwQ2000

- Improve documentation quality ([#1206](https://github.com/open-mmlab/mmpose/pull/1206), [#1161](https://github.com/open-mmlab/mmpose/pull/1161)) @ly015

- Switch to OpenMMLab official pre-commit-hook for copyright check ([#1214](https://github.com/open-mmlab/mmpose/pull/1214)) @ly015

- Fix hard-coded data collating and scattering in inference ([#1175](https://github.com/open-mmlab/mmpose/pull/1175)) @ly015

- Fix model configs on JHMDB dataset ([#1188](https://github.com/open-mmlab/mmpose/pull/1188)) @jin-s13

- Fix area calculation in pose tracking inference ([#1197](https://github.com/open-mmlab/mmpose/pull/1197)) @pallgeuer

- Fix registry scope conflict of module wrapper ([#1204](https://github.com/open-mmlab/mmpose/pull/1204)) @ly015

- Update MMCV installation in CI and documents ([#1205](https://github.com/open-mmlab/mmpose/pull/1205))

- Fix incorrect color channel order in visualization functions ([#1212](https://github.com/open-mmlab/mmpose/pull/1212)) @ly015

## **v0.23.0 (11/02/2022)**

- Add [MMPose Webcam API](https://github.com/open-mmlab/mmpose/tree/master/tools/webcam): A simple yet powerful tools to develop interactive webcam applications with MMPose functions. ([#1178](https://github.com/open-mmlab/mmpose/pull/1178), [#1173](https://github.com/open-mmlab/mmpose/pull/1173), [#1173](https://github.com/open-mmlab/mmpose/pull/1173), [#1143](https://github.com/open-mmlab/mmpose/pull/1143), [#1094](https://github.com/open-mmlab/mmpose/pull/1094), [#1133](https://github.com/open-mmlab/mmpose/pull/1133), [#1098](https://github.com/open-mmlab/mmpose/pull/1098), [#1160](https://github.com/open-mmlab/mmpose/pull/1160)) @ly015, @jin-s13, @liqikai9, @wusize, @luminxu, @zengwang430521 @mzr1996

- Support ConcatDataset ([#1139](https://github.com/open-mmlab/mmpose/pull/1139)) @Canwang-sjtu

- Support CPU training and testing ([#1157](https://github.com/open-mmlab/mmpose/pull/1157)) @ly015

- Add multi-processing configurations to speed up distributed training and testing ([#1146](https://github.com/open-mmlab/mmpose/pull/1146)) @ly015

- Add default runtime config ([#1145](https://github.com/open-mmlab/mmpose/pull/1145))

- Upgrade isort in pre-commit hook ([#1179](https://github.com/open-mmlab/mmpose/pull/1179)) @liqikai9

- Update README and documents ([#1171](https://github.com/open-mmlab/mmpose/pull/1171), [#1167](https://github.com/open-mmlab/mmpose/pull/1167), [#1153](https://github.com/open-mmlab/mmpose/pull/1153), [#1149](https://github.com/open-mmlab/mmpose/pull/1149), [#1148](https://github.com/open-mmlab/mmpose/pull/1148), [#1147](https://github.com/open-mmlab/mmpose/pull/1147), [#1140](https://github.com/open-mmlab/mmpose/pull/1140)) @jin-s13, @wusize, @TommyZihao, @ly015

- Fix undeterministic behavior in pre-commit hooks ([#1136](https://github.com/open-mmlab/mmpose/pull/1136)) @jin-s13

- Deprecate the support for "python setup.py test" ([#1179](https://github.com/open-mmlab/mmpose/pull/1179)) @ly015

- Fix incompatible settings with MMCV on HSigmoid default parameters ([#1132](https://github.com/open-mmlab/mmpose/pull/1132)) @ly015

- Fix albumentation installation ([#1184](https://github.com/open-mmlab/mmpose/pull/1184)) @BIGWangYuDong

## **v0.22.0 (04/01/2022)**

- Support VoxelPose ["VoxelPose: Towards Multi-Camera 3D Human Pose Estimation in Wild Environment"](https://arxiv.org/abs/2004.06239), ECCV'2020 ([#1050](https://github.com/open-mmlab/mmpose/pull/1050)) @wusize

- Support Soft Wing loss ["Structure-Coherent Deep Feature Learning for Robust Face Alignment"](https://linchunze.github.io/papers/TIP21_Structure_coherent_FA.pdf), TIP'2021 ([#1077](https://github.com/open-mmlab/mmpose/pull/1077)) @jin-s13

- Support Adaptive Wing loss ["Adaptive Wing Loss for Robust Face Alignment via Heatmap Regression"](https://arxiv.org/abs/1904.07399), ICCV'2019 ([#1072](https://github.com/open-mmlab/mmpose/pull/1072)) @jin-s13

- Add LiteHRNet-18 Checkpoints trained on COCO. ([#1120](https://github.com/open-mmlab/mmpose/pull/1120)) @jin-s13

- Improve documentation quality ([#1115](https://github.com/open-mmlab/mmpose/pull/1115), [#1111](https://github.com/open-mmlab/mmpose/pull/1111), [#1105](https://github.com/open-mmlab/mmpose/pull/1105), [#1087](https://github.com/open-mmlab/mmpose/pull/1087), [#1086](https://github.com/open-mmlab/mmpose/pull/1086), [#1085](https://github.com/open-mmlab/mmpose/pull/1085), [#1084](https://github.com/open-mmlab/mmpose/pull/1084), [#1083](https://github.com/open-mmlab/mmpose/pull/1083), [#1124](https://github.com/open-mmlab/mmpose/pull/1124), [#1070](https://github.com/open-mmlab/mmpose/pull/1070), [#1068](https://github.com/open-mmlab/mmpose/pull/1068)) @jin-s13, @liqikai9, @ly015

- Support CircleCI ([#1074](https://github.com/open-mmlab/mmpose/pull/1074)) @ly015

- Skip unit tests in CI when only document files were changed ([#1074](https://github.com/open-mmlab/mmpose/pull/1074), [#1041](https://github.com/open-mmlab/mmpose/pull/1041)) @QwQ2000, @ly015

- Support file_client_args in LoadImageFromFile ([#1076](https://github.com/open-mmlab/mmpose/pull/1076)) @jin-s13

- Fix a bug in Dark UDP postprocessing that causes error when the channel number is large. ([#1079](https://github.com/open-mmlab/mmpose/pull/1079), [#1116](https://github.com/open-mmlab/mmpose/pull/1116)) @X00123, @jin-s13

- Fix hard-coded `sigmas` in bottom-up image demo ([#1107](https://github.com/open-mmlab/mmpose/pull/1107), [#1101](https://github.com/open-mmlab/mmpose/pull/1101)) @chenxinfeng4, @liqikai9

- Fix unstable checks in unit tests ([#1112](https://github.com/open-mmlab/mmpose/pull/1112)) @ly015

- Do not destroy NULL windows if `args.show==False` in demo scripts ([#1104](https://github.com/open-mmlab/mmpose/pull/1104)) @bladrome

## **v0.21.0 (06/12/2021)**

- Support ["Learning Temporal Pose Estimation from Sparsely-Labeled Videos"](https://arxiv.org/abs/1906.04016), NeurIPS'2019 ([#932](https://github.com/open-mmlab/mmpose/pull/932), [#1006](https://github.com/open-mmlab/mmpose/pull/1006), [#1036](https://github.com/open-mmlab/mmpose/pull/1036), [#1060](https://github.com/open-mmlab/mmpose/pull/1060)) @liqikai9

- Add ViPNAS-MobileNetV3 models ([#1025](https://github.com/open-mmlab/mmpose/pull/1025)) @luminxu, @jin-s13

- Add inference speed benchmark ([#1028](https://github.com/open-mmlab/mmpose/pull/1028), [#1034](https://github.com/open-mmlab/mmpose/pull/1034), [#1044](https://github.com/open-mmlab/mmpose/pull/1044)) @liqikai9

- Support ["Learning Temporal Pose Estimation from Sparsely-Labeled Videos"](https://arxiv.org/abs/1906.04016), NeurIPS'2019 ([#932](https://github.com/open-mmlab/mmpose/pull/932), [#1006](https://github.com/open-mmlab/mmpose/pull/1006), [#1036](https://github.com/open-mmlab/mmpose/pull/1036)) @liqikai9

- Add light-weight top-down models for whole-body keypoint detection ([#1009](https://github.com/open-mmlab/mmpose/pull/1009), [#1020](https://github.com/open-mmlab/mmpose/pull/1020), [#1055](https://github.com/open-mmlab/mmpose/pull/1055)) @luminxu, @ly015

- Add HRNet checkpoints with various settings on PoseTrack18 ([#1035](https://github.com/open-mmlab/mmpose/pull/1035)) @liqikai9

- Update model metafile format ([#1001](https://github.com/open-mmlab/mmpose/pull/1001)) @ly015

- Support minus output feature index in mobilenet_v3 ([#1005](https://github.com/open-mmlab/mmpose/pull/1005)) @luminxu

- Improve documentation quality ([#1018](https://github.com/open-mmlab/mmpose/pull/1018), [#1026](https://github.com/open-mmlab/mmpose/pull/1026), [#1027](https://github.com/open-mmlab/mmpose/pull/1027), [#1031](https://github.com/open-mmlab/mmpose/pull/1031), [#1038](https://github.com/open-mmlab/mmpose/pull/1038), [#1046](https://github.com/open-mmlab/mmpose/pull/1046), [#1056](https://github.com/open-mmlab/mmpose/pull/1056), [#1057](https://github.com/open-mmlab/mmpose/pull/1057)) @edybk, @luminxu, @ly015, @jin-s13

- Set default random seed in training initialization ([#1030](https://github.com/open-mmlab/mmpose/pull/1030)) @ly015

- Skip CI when only specific files changed ([#1041](https://github.com/open-mmlab/mmpose/pull/1041), [#1059](https://github.com/open-mmlab/mmpose/pull/1059)) @QwQ2000, @ly015

- Automatically cancel uncompleted action runs when new commit arrives ([#1053](https://github.com/open-mmlab/mmpose/pull/1053)) @ly015

- Update pose tracking demo to be compatible with latest mmtracking ([#1014](https://github.com/open-mmlab/mmpose/pull/1014)) @jin-s13

- Fix symlink creation failure when installed in Windows environments ([#1039](https://github.com/open-mmlab/mmpose/pull/1039)) @QwQ2000

- Fix AP-10K dataset sigmas ([#1040](https://github.com/open-mmlab/mmpose/pull/1040)) @jin-s13

## **v0.20.0 (01/11/2021)**

- Add AP-10K dataset for animal pose estimation ([#987](https://github.com/open-mmlab/mmpose/pull/987)) @Annbless, @AlexTheBad, @jin-s13, @ly015

- Support TorchServe ([#979](https://github.com/open-mmlab/mmpose/pull/979)) @ly015

- Add HRNetv2 checkpoints on 300W and COFW datasets ([#980](https://github.com/open-mmlab/mmpose/pull/980)) @jin-s13

- Fix some deprecated or risky settings in configs ([#963](https://github.com/open-mmlab/mmpose/pull/963), [#976](https://github.com/open-mmlab/mmpose/pull/976), [#992](https://github.com/open-mmlab/mmpose/pull/992)) @jin-s13, @wusize

- Fix issues of default arguments of training and testing scripts ([#970](https://github.com/open-mmlab/mmpose/pull/970), [#985](https://github.com/open-mmlab/mmpose/pull/985)) @liqikai9, @wusize

- Fix heatmap and tag size mismatch in bottom-up with UDP ([#994](https://github.com/open-mmlab/mmpose/pull/994)) @wusize

- Fix python3.9 installation in CI ([#983](https://github.com/open-mmlab/mmpose/pull/983)) @ly015

- Fix model zoo document integrity issue ([#990](https://github.com/open-mmlab/mmpose/pull/990)) @jin-s13

- Support non-square input shape for bottom-up ([#991](https://github.com/open-mmlab/mmpose/pull/991)) @wusize

- Add image and video resources for demo ([#971](https://github.com/open-mmlab/mmpose/pull/971)) @liqikai9

- Use CUDA docker images to accelerate CI ([#973](https://github.com/open-mmlab/mmpose/pull/973)) @ly015

- Add codespell hook and fix detected typos ([#977](https://github.com/open-mmlab/mmpose/pull/977)) @ly015

## **v0.19.0 (08/10/2021)**

- Add models for Associative Embedding with Hourglass network backbone ([#906](https://github.com/open-mmlab/mmpose/pull/906), [#955](https://github.com/open-mmlab/mmpose/pull/955)) @jin-s13, @luminxu

- Support COCO-Wholebody-Face and COCO-Wholebody-Hand datasets ([#813](https://github.com/open-mmlab/mmpose/pull/813)) @jin-s13, @innerlee, @luminxu

- Upgrade dataset interface ([#901](https://github.com/open-mmlab/mmpose/pull/901), [#924](https://github.com/open-mmlab/mmpose/pull/924)) @jin-s13, @innerlee, @ly015, @liqikai9

- New style of documentation ([#945](https://github.com/open-mmlab/mmpose/pull/945)) @ly015

- Add pseudo-labeling tool to generate COCO style keypoint annotations with given bounding boxes ([#928](https://github.com/open-mmlab/mmpose/pull/928)) @soltkreig

- Fix segmentation parsing in Macaque dataset preprocessing ([#948](https://github.com/open-mmlab/mmpose/pull/948)) @jin-s13

- Fix dependencies that may lead to CI failure in downstream projects ([#936](https://github.com/open-mmlab/mmpose/pull/936), [#953](https://github.com/open-mmlab/mmpose/pull/953)) @RangiLyu, @ly015

- Fix keypoint order in Human3.6M dataset ([#940](https://github.com/open-mmlab/mmpose/pull/940)) @ttxskk

- Fix unstable image loading for Interhand2.6M ([#913](https://github.com/open-mmlab/mmpose/pull/913)) @zengwang430521

- Improve demo usability and stability ([#908](https://github.com/open-mmlab/mmpose/pull/908), [#934](https://github.com/open-mmlab/mmpose/pull/934)) @ly015

- Standardize model metafile format ([#941](https://github.com/open-mmlab/mmpose/pull/941)) @ly015

- Support `persistent_worker` and several other arguments in configs ([#946](https://github.com/open-mmlab/mmpose/pull/946)) @jin-s13

- Use MMCV root model registry to enable cross-project module building ([#935](https://github.com/open-mmlab/mmpose/pull/935)) @RangiLyu

- Improve the document quality ([#916](https://github.com/open-mmlab/mmpose/pull/916), [#909](https://github.com/open-mmlab/mmpose/pull/909), [#942](https://github.com/open-mmlab/mmpose/pull/942), [#913](https://github.com/open-mmlab/mmpose/pull/913), [#956](https://github.com/open-mmlab/mmpose/pull/956)) @jin-s13, @ly015, @bit-scientist, @zengwang430521

- Improve pull request template ([#952](https://github.com/open-mmlab/mmpose/pull/952), [#954](https://github.com/open-mmlab/mmpose/pull/954)) @ly015

- Upgrade dataset interface ([#901](https://github.com/open-mmlab/mmpose/pull/901)) @jin-s13, @innerlee, @ly015

## **v0.18.0 (01/09/2021)**

- Fix redundant model weight loading in pytorch-to-onnx conversion ([#850](https://github.com/open-mmlab/mmpose/pull/850)) @ly015

- Fix a bug in update_model_index.py that may cause pre-commit hook failure([#866](https://github.com/open-mmlab/mmpose/pull/866)) @ly015

- Fix a bug in interhand_3d_head ([#890](https://github.com/open-mmlab/mmpose/pull/890)) @zengwang430521

- Fix pose tracking demo failure caused by out-of-date configs ([#891](https://github.com/open-mmlab/mmpose/pull/891))

- Add automatic benchmark regression tools ([#849](https://github.com/open-mmlab/mmpose/pull/849), [#880](https://github.com/open-mmlab/mmpose/pull/880), [#885](https://github.com/open-mmlab/mmpose/pull/885)) @liqikai9, @ly015

- Add copyright information and checking hook ([#872](https://github.com/open-mmlab/mmpose/pull/872))

- Add PR template ([#875](https://github.com/open-mmlab/mmpose/pull/875)) @ly015

- Add citation information ([#876](https://github.com/open-mmlab/mmpose/pull/876)) @ly015

- Add python3.9 in CI ([#877](https://github.com/open-mmlab/mmpose/pull/877), [#883](https://github.com/open-mmlab/mmpose/pull/883)) @ly015

- Improve the quality of the documents ([#845](https://github.com/open-mmlab/mmpose/pull/845), [#845](https://github.com/open-mmlab/mmpose/pull/845), [#848](https://github.com/open-mmlab/mmpose/pull/848), [#867](https://github.com/open-mmlab/mmpose/pull/867), [#870](https://github.com/open-mmlab/mmpose/pull/870), [#873](https://github.com/open-mmlab/mmpose/pull/873), [#896](https://github.com/open-mmlab/mmpose/pull/896)) @jin-s13, @ly015, @zhiqwang

## **v0.17.0 (06/08/2021)**

1. Support ["Lite-HRNet: A Lightweight High-Resolution Network"](https://arxiv.org/abs/2104.06403) CVPR'2021 ([#733](https://github.com/open-mmlab/mmpose/pull/733),[#800](https://github.com/open-mmlab/mmpose/pull/800)) @jin-s13

2. Add 3d body mesh demo ([#771](https://github.com/open-mmlab/mmpose/pull/771)) @zengwang430521

3. Add Chinese documentation ([#787](https://github.com/open-mmlab/mmpose/pull/787), [#798](https://github.com/open-mmlab/mmpose/pull/798), [#799](https://github.com/open-mmlab/mmpose/pull/799), [#802](https://github.com/open-mmlab/mmpose/pull/802), [#804](https://github.com/open-mmlab/mmpose/pull/804), [#805](https://github.com/open-mmlab/mmpose/pull/805), [#815](https://github.com/open-mmlab/mmpose/pull/815), [#816](https://github.com/open-mmlab/mmpose/pull/816), [#817](https://github.com/open-mmlab/mmpose/pull/817), [#819](https://github.com/open-mmlab/mmpose/pull/819), [#839](https://github.com/open-mmlab/mmpose/pull/839)) @ly015, @luminxu, @jin-s13, @liqikai9, @zengwang430521

4. Add Colab Tutorial ([#834](https://github.com/open-mmlab/mmpose/pull/834)) @ly015

- Support ["Lite-HRNet: A Lightweight High-Resolution Network"](https://arxiv.org/abs/2104.06403) CVPR'2021 ([#733](https://github.com/open-mmlab/mmpose/pull/733),[#800](https://github.com/open-mmlab/mmpose/pull/800)) @jin-s13

- Add 3d body mesh demo ([#771](https://github.com/open-mmlab/mmpose/pull/771)) @zengwang430521

- Add Chinese documentation ([#787](https://github.com/open-mmlab/mmpose/pull/787), [#798](https://github.com/open-mmlab/mmpose/pull/798), [#799](https://github.com/open-mmlab/mmpose/pull/799), [#802](https://github.com/open-mmlab/mmpose/pull/802), [#804](https://github.com/open-mmlab/mmpose/pull/804), [#805](https://github.com/open-mmlab/mmpose/pull/805), [#815](https://github.com/open-mmlab/mmpose/pull/815), [#816](https://github.com/open-mmlab/mmpose/pull/816), [#817](https://github.com/open-mmlab/mmpose/pull/817), [#819](https://github.com/open-mmlab/mmpose/pull/819), [#839](https://github.com/open-mmlab/mmpose/pull/839)) @ly015, @luminxu, @jin-s13, @liqikai9, @zengwang430521

- Add Colab Tutorial ([#834](https://github.com/open-mmlab/mmpose/pull/834)) @ly015

- Support training for InterHand v1.0 dataset ([#761](https://github.com/open-mmlab/mmpose/pull/761)) @zengwang430521

- Fix mpii pckh@0.1 index ([#773](https://github.com/open-mmlab/mmpose/pull/773)) @jin-s13

- Fix multi-node distributed test ([#818](https://github.com/open-mmlab/mmpose/pull/818)) @ly015

- Fix docstring and init_weights error of ShuffleNetV1 ([#814](https://github.com/open-mmlab/mmpose/pull/814)) @Junjun2016

- Fix imshow_bbox error when input bboxes is empty ([#796](https://github.com/open-mmlab/mmpose/pull/796)) @ly015

- Fix model zoo doc generation ([#778](https://github.com/open-mmlab/mmpose/pull/778)) @ly015

- Fix typo ([#767](https://github.com/open-mmlab/mmpose/pull/767)), ([#780](https://github.com/open-mmlab/mmpose/pull/780), [#782](https://github.com/open-mmlab/mmpose/pull/782)) @ly015, @jin-s13

- Use MMCV EvalHook ([#686](https://github.com/open-mmlab/mmpose/pull/686)) @ly015

- Add pytest.ini and fix docstring ([#812](https://github.com/open-mmlab/mmpose/pull/812)) @jin-s13

- Update MSELoss ([#829](https://github.com/open-mmlab/mmpose/pull/829)) @Ezra-Yu

- Move process_mmdet_results into inference.py ([#831](https://github.com/open-mmlab/mmpose/pull/831)) @ly015

- Update resource limit ([#783](https://github.com/open-mmlab/mmpose/pull/783)) @jin-s13

- Use COCO 2D pose model in 3D demo examples ([#785](https://github.com/open-mmlab/mmpose/pull/785)) @ly015

- Change model zoo titles in the doc from center-aligned to left-aligned ([#792](https://github.com/open-mmlab/mmpose/pull/792), [#797](https://github.com/open-mmlab/mmpose/pull/797)) @ly015

- Support MIM ([#706](https://github.com/open-mmlab/mmpose/pull/706), [#794](https://github.com/open-mmlab/mmpose/pull/794)) @ly015

- Update out-of-date configs ([#827](https://github.com/open-mmlab/mmpose/pull/827)) @jin-s13

- Remove opencv-python-headless dependency by albumentations ([#833](https://github.com/open-mmlab/mmpose/pull/833)) @ly015

- Update QQ QR code in README_CN.md ([#832](https://github.com/open-mmlab/mmpose/pull/832)) @ly015

## **v0.16.0 (02/07/2021)**

1. Support ["ViPNAS: Efficient Video Pose Estimation via Neural Architecture Search"](https://arxiv.org/abs/2105.10154) CVPR'2021 ([#742](https://github.com/open-mmlab/mmpose/pull/742),[#755](https://github.com/open-mmlab/mmpose/pull/755)).

2. Support MPI-INF-3DHP dataset ([#683](https://github.com/open-mmlab/mmpose/pull/683),[#746](https://github.com/open-mmlab/mmpose/pull/746),[#751](https://github.com/open-mmlab/mmpose/pull/751)).

3. Add webcam demo tool ([#729](https://github.com/open-mmlab/mmpose/pull/729))

4. Add 3d body and hand pose estimation demo ([#704](https://github.com/open-mmlab/mmpose/pull/704), [#727](https://github.com/open-mmlab/mmpose/pull/727)).

- Support ["ViPNAS: Efficient Video Pose Estimation via Neural Architecture Search"](https://arxiv.org/abs/2105.10154) CVPR'2021 ([#742](https://github.com/open-mmlab/mmpose/pull/742),[#755](https://github.com/open-mmlab/mmpose/pull/755))

- Support MPI-INF-3DHP dataset ([#683](https://github.com/open-mmlab/mmpose/pull/683),[#746](https://github.com/open-mmlab/mmpose/pull/746),[#751](https://github.com/open-mmlab/mmpose/pull/751))

- Support Webcam demo ([#729](https://github.com/open-mmlab/mmpose/pull/729))

- Support Interhand 3d demo ([#704](https://github.com/open-mmlab/mmpose/pull/704))

- Support 3d pose video demo ([#727](https://github.com/open-mmlab/mmpose/pull/727))

- Support H36m dataset for 2d pose estimation ([#709](https://github.com/open-mmlab/mmpose/pull/709), [#735](https://github.com/open-mmlab/mmpose/pull/735))

- Add scripts to generate mim metafile ([#749](https://github.com/open-mmlab/mmpose/pull/749))

- Fix typos ([#692](https://github.com/open-mmlab/mmpose/pull/692),[#696](https://github.com/open-mmlab/mmpose/pull/696),[#697](https://github.com/open-mmlab/mmpose/pull/697),[#698](https://github.com/open-mmlab/mmpose/pull/698),[#712](https://github.com/open-mmlab/mmpose/pull/712),[#718](https://github.com/open-mmlab/mmpose/pull/718),[#728](https://github.com/open-mmlab/mmpose/pull/728))

- Change model download links from `http` to `https` ([#716](https://github.com/open-mmlab/mmpose/pull/716))

- Switch to MMCV MODEL_REGISTRY ([#669](https://github.com/open-mmlab/mmpose/pull/669))

- Refactor MeshMixDataset ([#752](https://github.com/open-mmlab/mmpose/pull/752))

- Rename 'GaussianHeatMap' to 'GaussianHeatmap' ([#745](https://github.com/open-mmlab/mmpose/pull/745))

- Update out-of-date configs ([#734](https://github.com/open-mmlab/mmpose/pull/734))

- Improve compatibility for breaking changes ([#731](https://github.com/open-mmlab/mmpose/pull/731))

- Enable to control radius and thickness in visualization ([#722](https://github.com/open-mmlab/mmpose/pull/722))

- Add regex dependency ([#720](https://github.com/open-mmlab/mmpose/pull/720))

## **v0.15.0 (02/06/2021)**

1. Support 3d video pose estimation (VideoPose3D).

2. Support 3d hand pose estimation (InterNet).

3. Improve presentation of modelzoo.

- Support "InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image" (ECCV‘20) ([#624](https://github.com/open-mmlab/mmpose/pull/624))

- Support "3D human pose estimation in video with temporal convolutions and semi-supervised training" (CVPR'19)  ([#602](https://github.com/open-mmlab/mmpose/pull/602), [#681](https://github.com/open-mmlab/mmpose/pull/681))

- Support 3d pose estimation demo ([#653](https://github.com/open-mmlab/mmpose/pull/653), [#670](https://github.com/open-mmlab/mmpose/pull/670))

- Support bottom-up whole-body pose estimation ([#689](https://github.com/open-mmlab/mmpose/pull/689))

- Support mmcli ([#634](https://github.com/open-mmlab/mmpose/pull/634))

- Fix opencv compatibility ([#635](https://github.com/open-mmlab/mmpose/pull/635))

- Fix demo with UDP ([#637](https://github.com/open-mmlab/mmpose/pull/637))

- Fix bottom-up model onnx conversion ([#680](https://github.com/open-mmlab/mmpose/pull/680))

- Fix `GPU_IDS` in distributed training ([#668](https://github.com/open-mmlab/mmpose/pull/668))

- Fix MANIFEST.in ([#641](https://github.com/open-mmlab/mmpose/pull/641), [#657](https://github.com/open-mmlab/mmpose/pull/657))

- Fix docs ([#643](https://github.com/open-mmlab/mmpose/pull/643),[#684](https://github.com/open-mmlab/mmpose/pull/684),[#688](https://github.com/open-mmlab/mmpose/pull/688),[#690](https://github.com/open-mmlab/mmpose/pull/690),[#692](https://github.com/open-mmlab/mmpose/pull/692))

- Reorganize configs by tasks, algorithms, datasets, and techniques ([#647](https://github.com/open-mmlab/mmpose/pull/647))

- Rename heads and detectors ([#667](https://github.com/open-mmlab/mmpose/pull/667))

- Add `radius` and `thickness` parameters in visualization ([#638](https://github.com/open-mmlab/mmpose/pull/638))

- Add `trans_prob` parameter in `TopDownRandomTranslation` ([#650](https://github.com/open-mmlab/mmpose/pull/650))

- Switch to `MMCV MODEL_REGISTRY` ([#669](https://github.com/open-mmlab/mmpose/pull/669))

- Update dependencies ([#674](https://github.com/open-mmlab/mmpose/pull/674), [#676](https://github.com/open-mmlab/mmpose/pull/676))

## **v0.14.0 (06/05/2021)**

1. Support animal pose estimation with 7 popular datasets.

2. Support "A simple yet effective baseline for 3d human pose estimation" (ICCV'17).

- Support "A simple yet effective baseline for 3d human pose estimation" (ICCV'17)  ([#554](https://github.com/open-mmlab/mmpose/pull/554),[#558](https://github.com/open-mmlab/mmpose/pull/558),[#566](https://github.com/open-mmlab/mmpose/pull/566),[#570](https://github.com/open-mmlab/mmpose/pull/570),[#589](https://github.com/open-mmlab/mmpose/pull/589))

- Support animal pose estimation ([#559](https://github.com/open-mmlab/mmpose/pull/559),[#561](https://github.com/open-mmlab/mmpose/pull/561),[#563](https://github.com/open-mmlab/mmpose/pull/563),[#571](https://github.com/open-mmlab/mmpose/pull/571),[#603](https://github.com/open-mmlab/mmpose/pull/603),[#605](https://github.com/open-mmlab/mmpose/pull/605))

- Support Horse-10 dataset ([#561](https://github.com/open-mmlab/mmpose/pull/561)), MacaquePose dataset ([#561](https://github.com/open-mmlab/mmpose/pull/561)), Vinegar Fly dataset ([#561](https://github.com/open-mmlab/mmpose/pull/561)), Desert Locust dataset ([#561](https://github.com/open-mmlab/mmpose/pull/561)), Grevy's Zebra dataset ([#561](https://github.com/open-mmlab/mmpose/pull/561)), ATRW dataset ([#571](https://github.com/open-mmlab/mmpose/pull/571)), and Animal-Pose dataset ([#603](https://github.com/open-mmlab/mmpose/pull/603))

- Support bottom-up pose tracking demo ([#574](https://github.com/open-mmlab/mmpose/pull/574))

- Support FP16 training ([#584](https://github.com/open-mmlab/mmpose/pull/584),[#616](https://github.com/open-mmlab/mmpose/pull/616),[#626](https://github.com/open-mmlab/mmpose/pull/626))

- Support NMS for bottom-up ([#609](https://github.com/open-mmlab/mmpose/pull/609))

- Fix bugs in the top-down demo, when there are no people in the images ([#569](https://github.com/open-mmlab/mmpose/pull/569)).

- Fix the links in the doc ([#612](https://github.com/open-mmlab/mmpose/pull/612))

- Speed up top-down inference ([#560](https://github.com/open-mmlab/mmpose/pull/560))

- Update github CI ([#562](https://github.com/open-mmlab/mmpose/pull/562), [#564](https://github.com/open-mmlab/mmpose/pull/564))

- Update Readme ([#578](https://github.com/open-mmlab/mmpose/pull/578),[#579](https://github.com/open-mmlab/mmpose/pull/579),[#580](https://github.com/open-mmlab/mmpose/pull/580),[#592](https://github.com/open-mmlab/mmpose/pull/592),[#599](https://github.com/open-mmlab/mmpose/pull/599),[#600](https://github.com/open-mmlab/mmpose/pull/600),[#607](https://github.com/open-mmlab/mmpose/pull/607))

- Update Faq ([#587](https://github.com/open-mmlab/mmpose/pull/587), [#610](https://github.com/open-mmlab/mmpose/pull/610))

## **v0.13.0 (31/03/2021)**

1. Support Wingloss.

2. Support RHD hand dataset.

- Support Wingloss ([#482](https://github.com/open-mmlab/mmpose/pull/482))

- Support RHD hand dataset ([#523](https://github.com/open-mmlab/mmpose/pull/523), [#551](https://github.com/open-mmlab/mmpose/pull/551))

- Support Human3.6m dataset for 3d keypoint detection ([#518](https://github.com/open-mmlab/mmpose/pull/518), [#527](https://github.com/open-mmlab/mmpose/pull/527))

- Support TCN model for 3d keypoint detection ([#521](https://github.com/open-mmlab/mmpose/pull/521), [#522](https://github.com/open-mmlab/mmpose/pull/522))

- Support Interhand3D model for 3d hand detection ([#536](https://github.com/open-mmlab/mmpose/pull/536))

- Support Multi-task detector ([#480](https://github.com/open-mmlab/mmpose/pull/480))

- Fix PCKh@0.1 calculation ([#516](https://github.com/open-mmlab/mmpose/pull/516))

- Fix unittest ([#529](https://github.com/open-mmlab/mmpose/pull/529))

- Fix circular importing ([#542](https://github.com/open-mmlab/mmpose/pull/542))

- Fix bugs in bottom-up keypoint score ([#548](https://github.com/open-mmlab/mmpose/pull/548))

- Update config & checkpoints ([#525](https://github.com/open-mmlab/mmpose/pull/525), [#546](https://github.com/open-mmlab/mmpose/pull/546))

- Fix typos ([#514](https://github.com/open-mmlab/mmpose/pull/514), [#519](https://github.com/open-mmlab/mmpose/pull/519), [#532](https://github.com/open-mmlab/mmpose/pull/532), [#537](https://github.com/open-mmlab/mmpose/pull/537), )

- Speed up post processing ([#535](https://github.com/open-mmlab/mmpose/pull/535))

- Update mmcv version dependency ([#544](https://github.com/open-mmlab/mmpose/pull/544))

## **v0.12.0 (28/02/2021)**

1. Support DeepPose algorithm.

- Support DeepPose algorithm ([#446](https://github.com/open-mmlab/mmpose/pull/446), [#461](https://github.com/open-mmlab/mmpose/pull/461))

- Support interhand3d dataset ([#468](https://github.com/open-mmlab/mmpose/pull/468))

- Support Albumentation pipeline ([#469](https://github.com/open-mmlab/mmpose/pull/469))

- Support PhotometricDistortion pipeline ([#485](https://github.com/open-mmlab/mmpose/pull/485))

- Set seed option for training ([#493](https://github.com/open-mmlab/mmpose/pull/493))

- Add demos for face keypoint detection ([#502](https://github.com/open-mmlab/mmpose/pull/502))

- Change channel order according to configs ([#504](https://github.com/open-mmlab/mmpose/pull/504))

- Fix `num_factors` in UDP encoding ([#495](https://github.com/open-mmlab/mmpose/pull/495))

- Fix configs ([#456](https://github.com/open-mmlab/mmpose/pull/456))

- Refactor configs for wholebody pose estimation ([#487](https://github.com/open-mmlab/mmpose/pull/487), [#491](https://github.com/open-mmlab/mmpose/pull/491))

- Rename `decode` function for heads ([#481](https://github.com/open-mmlab/mmpose/pull/481))

- Update config & checkpoints ([#453](https://github.com/open-mmlab/mmpose/pull/453),[#484](https://github.com/open-mmlab/mmpose/pull/484),[#487](https://github.com/open-mmlab/mmpose/pull/487))

- Add README in Chinese ([#462](https://github.com/open-mmlab/mmpose/pull/462))

- Add tutorials about configs  ([#465](https://github.com/open-mmlab/mmpose/pull/465))

- Add demo videos for various tasks ([#499](https://github.com/open-mmlab/mmpose/pull/499), [#503](https://github.com/open-mmlab/mmpose/pull/503))

- Update docs about MMPose installation ([#467](https://github.com/open-mmlab/mmpose/pull/467), [#505](https://github.com/open-mmlab/mmpose/pull/505))

- Rename `stat.py` to `stats.py` ([#483](https://github.com/open-mmlab/mmpose/pull/483))

- Fix typos ([#463](https://github.com/open-mmlab/mmpose/pull/463), [#464](https://github.com/open-mmlab/mmpose/pull/464), [#477](https://github.com/open-mmlab/mmpose/pull/477), [#481](https://github.com/open-mmlab/mmpose/pull/481))

- latex to bibtex ([#471](https://github.com/open-mmlab/mmpose/pull/471))

- Update FAQ ([#466](https://github.com/open-mmlab/mmpose/pull/466))

## **v0.11.0 (31/01/2021)**

1. Support fashion landmark detection.

2. Support face keypoint detection.

3. Support pose tracking with MMTracking.

- Support fashion landmark detection (DeepFashion) ([#413](https://github.com/open-mmlab/mmpose/pull/413))

- Support face keypoint detection (300W, AFLW, COFW, WFLW) ([#367](https://github.com/open-mmlab/mmpose/pull/367))

- Support pose tracking demo with MMTracking ([#427](https://github.com/open-mmlab/mmpose/pull/427))

- Support face demo ([#443](https://github.com/open-mmlab/mmpose/pull/443))

- Support AIC dataset for bottom-up methods ([#438](https://github.com/open-mmlab/mmpose/pull/438), [#449](https://github.com/open-mmlab/mmpose/pull/449))

- Fix multi-batch training ([#434](https://github.com/open-mmlab/mmpose/pull/434))

- Fix sigmas in AIC dataset ([#441](https://github.com/open-mmlab/mmpose/pull/441))

- Fix config file ([#420](https://github.com/open-mmlab/mmpose/pull/420))

- Refactor Heads ([#382](https://github.com/open-mmlab/mmpose/pull/382))

- Update readme ([#409](https://github.com/open-mmlab/mmpose/pull/409), [#412](https://github.com/open-mmlab/mmpose/pull/412), [#415](https://github.com/open-mmlab/mmpose/pull/415), [#416](https://github.com/open-mmlab/mmpose/pull/416), [#419](https://github.com/open-mmlab/mmpose/pull/419), [#421](https://github.com/open-mmlab/mmpose/pull/421), [#422](https://github.com/open-mmlab/mmpose/pull/422), [#424](https://github.com/open-mmlab/mmpose/pull/424), [#425](https://github.com/open-mmlab/mmpose/pull/425), [#435](https://github.com/open-mmlab/mmpose/pull/435), [#436](https://github.com/open-mmlab/mmpose/pull/436), [#437](https://github.com/open-mmlab/mmpose/pull/437), [#444](https://github.com/open-mmlab/mmpose/pull/444), [#445](https://github.com/open-mmlab/mmpose/pull/445))

- Add GAP (global average pooling) neck ([#414](https://github.com/open-mmlab/mmpose/pull/414))

- Speed up ([#411](https://github.com/open-mmlab/mmpose/pull/411), [#423](https://github.com/open-mmlab/mmpose/pull/423))

- Support COCO test-dev test ([#433](https://github.com/open-mmlab/mmpose/pull/433))

## **v0.10.0 (31/12/2020)**

1. Support more human pose estimation methods.

   1. [UDP](https://arxiv.org/abs/1911.07524)

2. Support pose tracking.

3. Support multi-batch inference.

4. Add some useful tools, including `analyze_logs`, `get_flops`, `print_config`.

5. Support more backbone networks.

   1. [ResNest](https://arxiv.org/pdf/2004.08955.pdf)
   2. [VGG](https://arxiv.org/abs/1409.1556)

- Support UDP ([#353](https://github.com/open-mmlab/mmpose/pull/353), [#371](https://github.com/open-mmlab/mmpose/pull/371), [#402](https://github.com/open-mmlab/mmpose/pull/402))

- Support multi-batch inference ([#390](https://github.com/open-mmlab/mmpose/pull/390))

- Support MHP dataset ([#386](https://github.com/open-mmlab/mmpose/pull/386))

- Support pose tracking demo ([#380](https://github.com/open-mmlab/mmpose/pull/380))

- Support mpii-trb demo ([#372](https://github.com/open-mmlab/mmpose/pull/372))

- Support mobilenet for hand pose estimation ([#377](https://github.com/open-mmlab/mmpose/pull/377))

- Support ResNest backbone ([#370](https://github.com/open-mmlab/mmpose/pull/370))

- Support VGG backbone ([#370](https://github.com/open-mmlab/mmpose/pull/370))

- Add some useful tools, including `analyze_logs`, `get_flops`, `print_config` ([#324](https://github.com/open-mmlab/mmpose/pull/324))

- Fix bugs in pck evaluation ([#328](https://github.com/open-mmlab/mmpose/pull/328))

- Fix model download links in README ([#396](https://github.com/open-mmlab/mmpose/pull/396), [#397](https://github.com/open-mmlab/mmpose/pull/397))

- Fix CrowdPose annotations and update benchmarks ([#384](https://github.com/open-mmlab/mmpose/pull/384))

- Fix modelzoo stat ([#354](https://github.com/open-mmlab/mmpose/pull/354), [#360](https://github.com/open-mmlab/mmpose/pull/360), [#362](https://github.com/open-mmlab/mmpose/pull/362))

- Fix config files for aic datasets ([#340](https://github.com/open-mmlab/mmpose/pull/340))

- Rename `image_thr` to `det_bbox_thr` for top-down methods.

- Organize the readme files ([#398](https://github.com/open-mmlab/mmpose/pull/398), [#399](https://github.com/open-mmlab/mmpose/pull/399), [#400](https://github.com/open-mmlab/mmpose/pull/400))

- Check linting for markdown ([#379](https://github.com/open-mmlab/mmpose/pull/379))

- Add faq.md ([#350](https://github.com/open-mmlab/mmpose/pull/350))

- Remove PyTorch 1.4 in CI ([#338](https://github.com/open-mmlab/mmpose/pull/338))

- Add pypi badge in readme ([#329](https://github.com/open-mmlab/mmpose/pull/329))

## **v0.9.0 (30/11/2020)**

   1. [MSPN](https://arxiv.org/abs/1901.00148)
   2. [RSN](https://arxiv.org/abs/2003.04030)

2. Support video pose estimation datasets.

   1. [sub-JHMDB](http://jhmdb.is.tue.mpg.de/dataset)

1. [MSPN](https://arxiv.org/abs/1901.00148)

3. Support Onnx model conversion.

- Support MSPN ([#278](https://github.com/open-mmlab/mmpose/pull/278))

- Support RSN ([#221](https://github.com/open-mmlab/mmpose/pull/221), [#318](https://github.com/open-mmlab/mmpose/pull/318))

- Support new post-processing method for MSPN & RSN ([#288](https://github.com/open-mmlab/mmpose/pull/288))

- Support sub-JHMDB dataset ([#292](https://github.com/open-mmlab/mmpose/pull/292))

- Support urls for pre-trained models in config files ([#232](https://github.com/open-mmlab/mmpose/pull/232))

- Support Onnx ([#305](https://github.com/open-mmlab/mmpose/pull/305))

- Fix model download links in README ([#255](https://github.com/open-mmlab/mmpose/pull/255), [#315](https://github.com/open-mmlab/mmpose/pull/315))

- `post_process=True|False` and `unbiased_decoding=True|False` are deprecated, use `post_process=None|default|unbiased` etc. instead ([#288](https://github.com/open-mmlab/mmpose/pull/288))

- Enrich the model zoo ([#256](https://github.com/open-mmlab/mmpose/pull/256), [#320](https://github.com/open-mmlab/mmpose/pull/320))

- Set the default map_location as 'cpu' to reduce gpu memory cost ([#227](https://github.com/open-mmlab/mmpose/pull/227))

- Support return heatmaps and backbone features for bottom-up models ([#229](https://github.com/open-mmlab/mmpose/pull/229))

- Upgrade mmcv maximum & minimum version ([#269](https://github.com/open-mmlab/mmpose/pull/269), [#313](https://github.com/open-mmlab/mmpose/pull/313))

- Automatically add modelzoo statistics to readthedocs ([#252](https://github.com/open-mmlab/mmpose/pull/252))

- Improve README ([#226](https://github.com/open-mmlab/mmpose/pull/226), [#257](https://github.com/open-mmlab/mmpose/pull/257), [#264](https://github.com/open-mmlab/mmpose/pull/264), [#280](https://github.com/open-mmlab/mmpose/pull/280), [#296](https://github.com/open-mmlab/mmpose/pull/296))

- Support PyTorch 1.7 in CI ([#274](https://github.com/open-mmlab/mmpose/pull/274))

- Add docs/tutorials for running demos ([#263](https://github.com/open-mmlab/mmpose/pull/263))

## **v0.8.0 (31/10/2020)**

1. Support more human pose estimation datasets.

   1. [CrowdPose](https://github.com/Jeff-sjtu/CrowdPose)
   2. [PoseTrack18](https://posetrack.net/)

2. Support more 2D hand keypoint estimation datasets.

   1. [InterHand2.6](https://github.com/facebookresearch/InterHand2.6M)

3. Support adversarial training for 3D human shape recovery.

4. Support multi-stage losses.

5. Support mpii demo.

- Support [CrowdPose](https://github.com/Jeff-sjtu/CrowdPose) dataset ([#195](https://github.com/open-mmlab/mmpose/pull/195))

- Support [PoseTrack18](https://posetrack.net/) dataset ([#220](https://github.com/open-mmlab/mmpose/pull/220))

- Support [InterHand2.6](https://github.com/facebookresearch/InterHand2.6M) dataset ([#202](https://github.com/open-mmlab/mmpose/pull/202))

- Support adversarial training for 3D human shape recovery ([#192](https://github.com/open-mmlab/mmpose/pull/192))

- Support multi-stage losses ([#204](https://github.com/open-mmlab/mmpose/pull/204))

- Fix config files ([#190](https://github.com/open-mmlab/mmpose/pull/190))

- Add mpii demo ([#216](https://github.com/open-mmlab/mmpose/pull/216))

- Improve README ([#181](https://github.com/open-mmlab/mmpose/pull/181), [#183](https://github.com/open-mmlab/mmpose/pull/183), [#208](https://github.com/open-mmlab/mmpose/pull/208))

- Support return heatmaps and backbone features ([#196](https://github.com/open-mmlab/mmpose/pull/196), [#212](https://github.com/open-mmlab/mmpose/pull/212))

- Support different return formats of mmdetection models ([#217](https://github.com/open-mmlab/mmpose/pull/217))

## **v0.7.0 (30/9/2020)**

1. Support HMR for 3D human shape recovery.

2. Support WholeBody human pose estimation.

   1. [COCO-WholeBody](https://github.com/jin-s13/COCO-WholeBody)

3. Support more 2D hand keypoint estimation datasets.

   1. [Frei-hand](https://lmb.informatik.uni-freiburg.de/projects/freihand/)

1. [COCO-WholeBody](https://github.com/jin-s13/COCO-WholeBody)

   2. [CMU Panoptic HandDB](http://domedb.perception.cs.cmu.edu/handdb.html)

4. Add more popular backbones & enrich the [modelzoo](https://mmpose.readthedocs.io/en/latest/model_zoo.html)

   1. ShuffleNetv2

5. Support hand demo and whole-body demo.

- Support HMR for 3D human shape recovery ([#157](https://github.com/open-mmlab/mmpose/pull/157), [#160](https://github.com/open-mmlab/mmpose/pull/160), [#161](https://github.com/open-mmlab/mmpose/pull/161), [#162](https://github.com/open-mmlab/mmpose/pull/162))

- Support [COCO-WholeBody](https://github.com/jin-s13/COCO-WholeBody) dataset ([#133](https://github.com/open-mmlab/mmpose/pull/133))

- Support [Frei-hand](https://lmb.informatik.uni-freiburg.de/projects/freihand/) dataset ([#125](https://github.com/open-mmlab/mmpose/pull/125))

- Support [CMU Panoptic HandDB](http://domedb.perception.cs.cmu.edu/handdb.html) dataset ([#144](https://github.com/open-mmlab/mmpose/pull/144))

- Support H36M dataset ([#159](https://github.com/open-mmlab/mmpose/pull/159))

- Support ShuffleNetv2 ([#139](https://github.com/open-mmlab/mmpose/pull/139))

- Support saving best models based on key indicator ([#127](https://github.com/open-mmlab/mmpose/pull/127))

- Fix typos in docs ([#121](https://github.com/open-mmlab/mmpose/pull/121))

- Fix assertion ([#142](https://github.com/open-mmlab/mmpose/pull/142))

- Add tools to transform .mat format to .json format ([#126](https://github.com/open-mmlab/mmpose/pull/126))

- Add hand demo ([#115](https://github.com/open-mmlab/mmpose/pull/115))

- Add whole-body demo ([#163](https://github.com/open-mmlab/mmpose/pull/163))

- Reuse mmcv utility function and update version files ([#135](https://github.com/open-mmlab/mmpose/pull/135), [#137](https://github.com/open-mmlab/mmpose/pull/137))

- Enrich the modelzoo ([#147](https://github.com/open-mmlab/mmpose/pull/147), [#169](https://github.com/open-mmlab/mmpose/pull/169))

- Improve docs ([#174](https://github.com/open-mmlab/mmpose/pull/174), [#175](https://github.com/open-mmlab/mmpose/pull/175), [#178](https://github.com/open-mmlab/mmpose/pull/178))

- Improve README ([#176](https://github.com/open-mmlab/mmpose/pull/176))

- Improve version.py ([#173](https://github.com/open-mmlab/mmpose/pull/173))

## **v0.6.0 (31/8/2020)**

1. Add more popular backbones & enrich the [modelzoo](https://mmpose.readthedocs.io/en/latest/model_zoo.html)

   1. ResNext
   2. SEResNet
   3. ResNetV1D
   4. MobileNetv2
   5. ShuffleNetv1
   6. CPM (Convolutional Pose Machine)

2. Add more popular datasets:

   1. [AIChallenger](https://arxiv.org/abs/1711.06475?context=cs.CV)
   2. [MPII](http://human-pose.mpi-inf.mpg.de/)
   3. [MPII-TRB](https://github.com/kennymckormick/Triplet-Representation-of-human-Body)
   4. [OCHuman](http://www.liruilong.cn/projects/pose2seg/index.html)

3. Support 2d hand keypoint estimation.

   1. [OneHand10K](https://www.yangangwang.com/papers/WANG-MCC-2018-10.html)

4. Support bottom-up inference.

- Support [OneHand10K](https://www.yangangwang.com/papers/WANG-MCC-2018-10.html) dataset ([#52](https://github.com/open-mmlab/mmpose/pull/52))

- Support [MPII](http://human-pose.mpi-inf.mpg.de/) dataset ([#55](https://github.com/open-mmlab/mmpose/pull/55))

- Support [MPII-TRB](https://github.com/kennymckormick/Triplet-Representation-of-human-Body) dataset ([#19](https://github.com/open-mmlab/mmpose/pull/19), [#47](https://github.com/open-mmlab/mmpose/pull/47), [#48](https://github.com/open-mmlab/mmpose/pull/48))

- Support [OCHuman](http://www.liruilong.cn/projects/pose2seg/index.html) dataset ([#70](https://github.com/open-mmlab/mmpose/pull/70))

- Support [AIChallenger](https://arxiv.org/abs/1711.06475?context=cs.CV) dataset ([#87](https://github.com/open-mmlab/mmpose/pull/87))

- Support multiple backbones ([#26](https://github.com/open-mmlab/mmpose/pull/26))

- Support CPM model ([#56](https://github.com/open-mmlab/mmpose/pull/56))

- Fix configs for MPII & MPII-TRB datasets ([#93](https://github.com/open-mmlab/mmpose/pull/93))

- Fix the bug of missing `test_pipeline` in configs ([#14](https://github.com/open-mmlab/mmpose/pull/14))

- Fix typos ([#27](https://github.com/open-mmlab/mmpose/pull/27), [#28](https://github.com/open-mmlab/mmpose/pull/28), [#50](https://github.com/open-mmlab/mmpose/pull/50), [#53](https://github.com/open-mmlab/mmpose/pull/53), [#63](https://github.com/open-mmlab/mmpose/pull/63))

- Update benchmark ([#93](https://github.com/open-mmlab/mmpose/pull/93))

- Add Dockerfile ([#44](https://github.com/open-mmlab/mmpose/pull/44))

- Improve unittest coverage and minor fix ([#18](https://github.com/open-mmlab/mmpose/pull/18))

- Support CPUs for train/val/demo ([#34](https://github.com/open-mmlab/mmpose/pull/34))

- Support bottom-up demo ([#69](https://github.com/open-mmlab/mmpose/pull/69))

- Add tools to publish model ([#62](https://github.com/open-mmlab/mmpose/pull/62))

- Enrich the modelzoo ([#64](https://github.com/open-mmlab/mmpose/pull/64), [#68](https://github.com/open-mmlab/mmpose/pull/68), [#82](https://github.com/open-mmlab/mmpose/pull/82))

## **v0.5.0 (21/7/2020)**

- MMPose is released.

**Main Features**

- Support both top-down and bottom-up pose estimation approaches.

- Achieve higher training efficiency and higher accuracy than other popular codebases (e.g. AlphaPose, HRNet)

- Support various backbone models: ResNet, HRNet, SCNet, Houglass and HigherHRNet.

}
```

<details>

```bibtex
}
```

<details>

```bibtex

<details>

```bibtex
}
```

| [pose_resnet_50_dark](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_res50_dark-8xb64-210e_coco-256x192.py) |  256x192   | 0.724 |      0.897      |      0.797      | 0.777 |      0.934      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res50_dark-8xb64-210e_coco-256x192-c129dcb6_20220926.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res50_dark-8xb64-210e_coco-256x192_20220926.log) |

| [pose_resnet_50_dark](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_res50_dark-8xb64-210e_coco-384x288.py) |  384x288   | 0.735 |      0.902      |      0.801      | 0.786 |      0.938      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res50_dark-8xb64-210e_coco-384x288-8b90b538_20220926.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res50_dark-8xb64-210e_coco-384x288_20220926.log) |
| [pose_resnet_101_dark](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_res101_dark-8xb64-210e_coco-256x192.py) |  256x192   | 0.733 |      0.900      |      0.810      | 0.786 |      0.938      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res101_dark-8xb64-210e_coco-256x192-528ec248_20220926.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res101_dark-8xb64-210e_coco-256x192_20220926.log) |

| [pose_resnet_101_dark](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_res101_dark-8xb64-210e_coco-384x288.py) |  384x288   | 0.749 |      0.905      |      0.818      | 0.799 |      0.940      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res101_dark-8xb64-210e_coco-384x288-487d40a4_20220926.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res101_dark-8xb64-210e_coco-384x288_20220926.log) |
| [pose_resnet_152_dark](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_res152_dark-8xb32-210e_coco-256x192.py) |  256x192   | 0.743 |      0.906      |      0.819      | 0.796 |      0.943      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res152_dark-8xb32-210e_coco-256x192-f754df5f_20221031.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res152_dark-8xb32-210e_coco-256x192_20221031.log) |

| [pose_resnet_152_dark](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_res152_dark-8xb32-210e_coco-384x288.py) |  384x288   | 0.755 |      0.907      |      0.825      | 0.805 |      0.943      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res152_dark-8xb32-210e_coco-384x288-329f8454_20221031.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_res152_dark-8xb32-210e_coco-384x288_20221031.log) |

</div>

| frame sampling strategy |    scheduler     | resolution | gpus |          backbone          | pretrain | top1 acc | top5 acc | testing protocol  | FLOPs  | params |          config          |          ckpt          |          log           |
| :---------------------: | :--------------: | :--------: | :--: | :------------------------: | :------: | :------: | :------: | :---------------: | :----: | :----: | :----------------------: | :--------------------: | :--------------------: |

|         4x16x1          |  Linear+Cosine   |  224x224   |  8   |          ResNet50          |   None   |  72.97   |  90.88   | 10 clips x 3 crop | 27.38G | 32.45M | [config](/configs/recognition/slowonly/slowonly_r50_8xb16-4x16x1-256e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_r50_8xb16-4x16x1-256e_kinetics400-rgb/slowonly_r50_4x16x1_256e_8xb16_kinetics400-rgb_20220901-f6a40d08.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_r50_8xb16-4x16x1-256e_kinetics400-rgb/slowonly_r50_4x16x1_256e_8xb16_kinetics400-rgb.log) |

|          8x8x1          |  Linear+Cosine   |  224x224   |  8   |          ResNet50          |   None   |  75.15   |  92.11   | 10 clips x 3 crop | 54.75G | 32.45M | [config](/configs/recognition/slowonly/slowonly_r50_8xb16-8x8x1-256e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_r50_8xb16-8x8x1-256e_kinetics400-rgb/slowonly_r50_8xb16-8x8x1-256e_kinetics400-rgb_20220901-2132fc87.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_r50_8xb16-8x8x1-256e_kinetics400-rgb/slowonly_r50_8xb16-8x8x1-256e_kinetics400-rgb.log) |

|          8x8x1          |  Linear+Cosine   |  224x224   |  8   |         ResNet101          |   None   |  76.59   |  92.80   | 10 clips x 3 crop |  112G  | 60.36M | [config](/configs/recognition/slowonly/slowonly_r101_8xb16-8x8x1-196e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_r101_8xb16-8x8x1-196e_kinetics400-rgb/slowonly_r101_8xb16-8x8x1-196e_kinetics400-rgb_20220901-e6281431.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_r101_8xb16-8x8x1-196e_kinetics400-rgb/slowonly_r101_8xb16-8x8x1-196e_kinetics400-rgb.log) |

|         4x16x1          | Linear+MultiStep |  224x224   |  8   |          ResNet50          | ImageNet |  75.12   |  91.72   | 10 clips x 3 crop | 27.38G | 32.45M | [config](/configs/recognition/slowonly/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb_20220901-e7b65fad.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb.log) |

|          8x8x1          | Linear+MultiStep |  224x224   |  8   |          ResNet50          | ImageNet |  76.45   |  92.55   | 10 clips x 3 crop | 54.75G | 32.45M | [config](/configs/recognition/slowonly/slowonly_imagenet-pretrained-r50_8xb16-8x8x1-steplr-150e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_8xb16-8x8x1-steplr-150e_kinetics400-rgb/slowonly_imagenet-pretrained-r50_8xb16-8x8x1-steplr-150e_kinetics400-rgb_20220901-df42dc84.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_8xb16-8x8x1-steplr-150e_kinetics400-rgb/slowonly_imagenet-pretrained-r50_8xb16-8x8x1-steplr-150e_kinetics400-rgb.log) |

|         4x16x1          | Linear+MultiStep |  224x224   |  8   | ResNet50 (NonLocalEmbedGauss) | ImageNet |  75.07   |  91.69   | 10 clips x 3 crop | 43.23G | 39.81M | [config](/configs/recognition/slowonly/slowonly_r50-in1k-pre-nl-embedded-gaussian_8xb16-4x16x1-steplr-150e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_r50-in1k-pre-nl-embedded-gaussian_8xb16-4x16x1-steplr-150e_kinetics400-rgb/slowonly_r50-in1k-pre-nl-embedded-gaussian_8xb16-4x16x1-steplr-150e_kinetics400-rgb_20220901-cf739c75.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_r50-in1k-pre-nl-embedded-gaussian_8xb16-4x16x1-steplr-150e_kinetics400-rgb/slowonly_r50-in1k-pre-nl-embedded-gaussian_8xb16-4x16x1-steplr-150e_kinetics400-rgb.log) |

|          8x8x1          | Linear+MultiStep |  224x224   |  8   | ResNet50 (NonLocalEmbedGauss) | ImageNet |  76.65   |  92.47   | 10 clips x 3 crop | 96.66G | 39.81M | [config](/configs/recognition/slowonly/slowonly_r50-in1k-pre-nl-embedded-gaussian_8xb16-8x8x1-steplr-150e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_r50-in1k-pre-nl-embedded-gaussian_8xb16-8x8x1-steplr-150e_kinetics400-rgb/slowonly_r50-in1k-pre-nl-embedded-gaussian_8xb16-8x8x1-steplr-150e_kinetics400-rgb_20220901-df42dc84.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_r50-in1k-pre-nl-embedded-gaussian_8xb16-8x8x1-steplr-150e_kinetics400-rgb/slowonly_r50-in1k-pre-nl-embedded-gaussian_8xb16-8x8x1-steplr-150e_kinetics400-rgb.log) |

| frame sampling strategy |    scheduler     | resolution | gpus | backbone | pretrain | top1 acc | top5 acc | testing protocol  | FLOPs  | params |             config             |             ckpt             |             log              |
| :---------------------: | :--------------: | :--------: | :--: | :------: | :------: | :------: | :------: | :---------------: | :----: | :----: | :----------------------------: | :--------------------------: | :--------------------------: |

|         4x16x1          | Linear+MultiStep |  224x224   | 8x2  | ResNet50 | ImageNet |  65.52   |  86.39   | 10 clips x 3 crop | 27.38G | 32.45M | [config](/configs/recognition/slowonly/slowonly_imagenet-pretrained-r50_16xb16-4x16x1-steplr-150e_kinetics700-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_16xb16-4x16x1-steplr-150e_kinetics700-rgb/slowonly_imagenet-pretrained-r50_16xb16-4x16x1-steplr-150e_kinetics700-rgb_20221013-98b1b0a7.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_16xb16-4x16x1-steplr-150e_kinetics700-rgb/slowonly_imagenet-pretrained-r50_16xb16-4x16x1-steplr-150e_kinetics700-rgb.log) |

|          8x8x1          | Linear+MultiStep |  224x224   | 8x2  | ResNet50 | ImageNet |  67.67   |  87.80   | 10 clips x 3 crop | 54.75G | 32.45M | [config](/configs/recognition/slowonly/slowonly_imagenet-pretrained-r50_16xb16-8x8x1-steplr-150e_kinetics700-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_16xb16-8x8x1-steplr-150e_kinetics700-rgb/slowonly_imagenet-pretrained-r50_16xb16-8x8x1-steplr-150e_kinetics700-rgb_20221013-15b93b10.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_16xb16-8x8x1-steplr-150e_kinetics700-rgb/slowonly_imagenet-pretrained-r50_16xb16-8x8x1-steplr-150e_kinetics700-rgb.log) |

|          8x8x1          | Linear+MultiStep |  224x224   | 8x4  | ResNet50 | ImageNet |  72.39   |  90.60   | 10 clips x 3 crop | 54.75G | 32.45M | [config](/configs/recognition/slowonly/slowonly_imagenet-pretrained-r50_32xb8-8x8x1-steplr-150e_kinetics710-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_32xb8-8x8x1-steplr-150e_kinetics710-rgb/slowonly_imagenet-pretrained-r50_32xb8-8x8x1-steplr-150e_kinetics710-rgb_20230612-12ce977c.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_32xb8-8x8x1-steplr-150e_kinetics710-rgb/slowonly_imagenet-pretrained-r50_32xb8-8x8x1-steplr-150e_kinetics710-rgb.log) |

## Train

```shell
```

Example: train SlowOnly model on Kinetics-400 dataset in a deterministic option with periodic validation.

```shell
python tools/train.py configs/recognition/slowonly/slowonly_r50_8xb16-4x16x1-256e_kinetics400-rgb.py \
```

## Test

```shell
```

Example: test SlowOnly model on Kinetics-400 dataset and dump the result to a pkl file.

```shell
python tools/test.py configs/recognition/slowonly/slowonly_r50_8xb16-4x16x1-256e_kinetics400-rgb.py \

```shell
```

```BibTeX
}
```

<details>

```bibtex
}
```

<details>

```bibtex

author={Wu, Wayne and Qian, Chen and Yang, Shuo and Wang, Quan and Cai, Yici and Zhou, Qiang},
}
```

| Arch                                                           | Input Size | NME  |                              ckpt                              |                              log                               |
| :------------------------------------------------------------- | :--------: | :--: | :------------------------------------------------------------: | :------------------------------------------------------------: |

| [pose_rtmpose_m](/configs/face_2d_keypoint/rtmpose/wflw/rtmpose-m_8xb64-60e_wflw-256x256.py) |  256x256   | 4.01 | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-wflw_pt-aic-coco_60e-256x256-dc1dcdcf_20230228.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-wflw_pt-aic-coco_60e-256x256-dc1dcdcf_20230228.json) |

# ResNet strikes back

> [ResNet strikes back: An improved training procedure in timm](https://arxiv.org/abs/2110.00476)

The influential Residual Networks designed by He et al. remain the gold-standard architecture in numerous scientific publications. They typically serve as the default architecture in studies, or as baselines when new architectures are proposed. Yet there has been significant progress on best practices for training neural networks since the inception of the ResNet architecture in 2015. Novel optimization & dataaugmentation have increased the effectiveness of the training recipes.

In this paper, we re-evaluate the performance of the vanilla ResNet-50 when trained with a procedure that integrates such advances. We share competitive training settings and pre-trained models in the timm open-source library, with the hope that they will serve as better baselines for future work. For instance, with our more demanding training setting, a vanilla ResNet-50 reaches 80.4% top-1 accuracy at resolution 224×224 on ImageNet-val without extra data or distillation. We also report the performance achieved with popular models with our training procedure.

<img src="https://user-images.githubusercontent.com/12907710/149324625-4546a5a7-704f-406c-982f-0376a20d03d8.png"/>
</div>

</div>

|       Method       | Backbone | Lr schd | Mem (GB) | Inf time (fps) |   box AP    |   mask AP   |                          Config                          |                                                                                                                                                                                                   Download                                                                                                                                                                                                   |

|    Faster R-CNN    | R-50 rsb |   1x    |   3.9    |       -        | 40.8 (+3.4) |      -      |    [Config](./faster-rcnn_r50-rsb-pre_fpn_1x_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/resnet_strikes_back/faster_rcnn_r50_fpn_rsb-pretrain_1x_coco/faster_rcnn_r50_fpn_rsb-pretrain_1x_coco_20220113_162229-32ae82a9.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/resnet_strikes_back/faster_rcnn_r50_fpn_rsb-pretrain_1x_coco/faster_rcnn_r50_fpn_rsb-pretrain_1x_coco_20220113_162229.log.json)             |

|     Mask R-CNN     | R-50 rsb |   1x    |   4.5    |       -        | 41.2 (+3.0) | 38.2 (+3.0) |     [Config](./mask-rcnn_r50-rsb-pre_fpn_1x_coco.py)     |                 [model](https://download.openmmlab.com/mmdetection/v2.0/resnet_strikes_back/mask_rcnn_r50_fpn_rsb-pretrain_1x_coco/mask_rcnn_r50_fpn_rsb-pretrain_1x_coco_20220113_174054-06ce8ba0.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/resnet_strikes_back/mask_rcnn_r50_fpn_rsb-pretrain_1x_coco/mask_rcnn_r50_fpn_rsb-pretrain_1x_coco_20220113_174054.log.json)                 |

| Cascade Mask R-CNN | R-50 rsb |   1x    |   6.2    |       -        | 44.8 (+3.6) | 39.9 (+3.6) | [Config](./cascade-mask-rcnn_r50-rsb-pre_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/resnet_strikes_back/cascade_mask_rcnn_r50_fpn_rsb-pretrain_1x_coco/cascade_mask_rcnn_r50_fpn_rsb-pretrain_1x_coco_20220113_193636-8b9ad50f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/resnet_strikes_back/cascade_mask_rcnn_r50_fpn_rsb-pretrain_1x_coco/cascade_mask_rcnn_r50_fpn_rsb-pretrain_1x_coco_20220113_193636.log.json) |

|     RetinaNet      | R-50 rsb |   1x    |   3.8    |       -        | 39.0 (+2.5) |      -      |     [Config](./retinanet_r50-rsb-pre_fpn_1x_coco.py)     |                 [model](https://download.openmmlab.com/mmdetection/v2.0/resnet_strikes_back/retinanet_r50_fpn_rsb-pretrain_1x_coco/retinanet_r50_fpn_rsb-pretrain_1x_coco_20220113_175432-bd24aae9.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/resnet_strikes_back/retinanet_r50_fpn_rsb-pretrain_1x_coco/retinanet_r50_fpn_rsb-pretrain_1x_coco_20220113_175432.log.json)                 |

- 'rsb' is short for 'resnet strikes back'
- We have done some grid searches on learning rate and weight decay and get these optimal hyper-parameters.

```latex
title={Resnet strikes back: An improved training procedure in timm},
author={Ross Wightman, Hugo Touvron, Hervé Jégou},
journal={arXiv preprint arXiv:2110.00476},
}
```

# MobileViT

> [MobileViT Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://arxiv.org/abs/2110.02178)

**MobileViT** aims at introducing a light-weight network, which takes the advantages of both ViTs and CNNs, uses the `InvertedResidual` blocks in [MobileNetV2](../mobilenet_v2/README.md) and `MobileViTBlock` which refers to [ViT](../vision_transformer/README.md) transformer blocks to build a standard 5-stage model structure.

The MobileViTBlock reckons transformers as convolutions to perform a global representation, meanwhile conbined with original convolution layers for local representation to build a block with global receptive field. This is different from ViT, which adds an extra class token and position embeddings for learning relative relationship. Without any position embeddings, MobileViT can benfit from multi-scale inputs during training.

Also, this paper puts forward a strategy for multi-scale training to dynamically adjust batch size based on the image size to both improve training efficiency and final performance.

It is also proven effective in downstream tasks such as object detection and segmentation.

<img src="https://user-images.githubusercontent.com/42952108/193229983-822bf025-89a6-4d95-b6be-76b7f1a62f2c.png" width="70%"/>
</div>

<details>

<br>

Light-weight convolutional neural networks (CNNs) are the de-facto for mobile vision tasks. Their spatial inductive biases allow them to learn representations with fewer parameters across different vision tasks. However, these networks are spatially local. To learn global representations, self-attention-based vision trans-formers (ViTs) have been adopted. Unlike CNNs, ViTs are heavy-weight. In this paper, we ask the following question: is it possible to combine the strengths of CNNs and ViTs to build a light-weight and low latency network for mobile vision tasks? Towards this end, we introduce MobileViT, a light-weight and general-purpose vision transformer for mobile devices. MobileViT presents a different perspective for the global processing of information with transformers, i.e., transformers as convolutions. Our results show that MobileViT significantly outperforms CNN- and ViT-based networks across different tasks and datasets. On the ImageNet-1k dataset, MobileViT achieves top-1 accuracy of 78.4% with about 6 million parameters, which is 3.2% and 6.2% more accurate than MobileNetv3 (CNN-based) and DeIT (ViT-based) for a similar number of parameters. On the MS-COCO object detection task, MobileViT is 5.7% more accurate than MobileNetv3 for a similar number of parameters.

</br>

```python

predict = inference_model('mobilevit-small_3rdparty_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('mobilevit-small_3rdparty_in1k', pretrained=True)
```

Test:

```shell
python tools/test.py configs/mobilevit/mobilevit-small_8xb128_in1k.py https://download.openmmlab.com/mmclassification/v0/mobilevit/mobilevit-small_3rdparty_in1k_20221018-cb4f741c.pth
```

```

| `mobilevit-small_3rdparty_in1k`\*   | From scratch |    5.58    |   2.03    |   78.25   |   94.09   |  [config](mobilevit-small_8xb128_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/mobilevit/mobilevit-small_3rdparty_in1k_20221018-cb4f741c.pth) |

| `mobilevit-xsmall_3rdparty_in1k`\*  | From scratch |    2.32    |   1.05    |   74.75   |   92.32   | [config](mobilevit-xsmall_8xb128_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/mobilevit/mobilevit-xsmall_3rdparty_in1k_20221018-be39a6e7.pth) |
| `mobilevit-xxsmall_3rdparty_in1k`\* | From scratch |    1.27    |   0.42    |   69.02   |   88.91   | [config](mobilevit-xxsmall_8xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mobilevit/mobilevit-xxsmall_3rdparty_in1k_20221018-77835605.pth) |

*Models with * are converted from the [official repo](https://github.com/apple/ml-cvnets). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@article{mehta2021mobilevit,
  title={MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer},
  author={Mehta, Sachin and Rastegari, Mohammad},
  journal={arXiv preprint arXiv:2110.02178},
}

author={Mehta, Sachin and Rastegari, Mohammad},
}
```

<details>

```bibtex
}
```

<details>

```bibtex

| [cpm](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_cpm_8xb64-210e_coco-256x192.py) |  256x192   | 0.627 |      0.862      |      0.709      | 0.689 |      0.906      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_cpm_8xb64-210e_coco-256x192-0e978875_20220920.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_cpm_8xb64-210e_coco-256x192_20220920.log) |
| [cpm](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_cpm_8xb32-210e_coco-384x288.py) |  384x288   | 0.652 |      0.865      |      0.730      | 0.710 |      0.907      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_cpm_8xb32-210e_coco-384x288-165487b8_20221011.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_cpm_8xb32-210e_coco-384x288_20221011.log) |

# Tutorial 1: Learn about Configs in MMagic

You can learn about the usage of our config system according to the following tutorials.

- [Tutorial 1: Learn about Configs in MMagic](#tutorial-1-learn-about-configs-in-mmagic)
  - [Config file structure](#config-file-structure)
  - [Config name style](#config-name-style)
  - [An example of EDSR](#an-example-of-edsr)
    - [Model config](#model-config)
    - [Data config](#data-config)
      - [Data pipeline](#data-pipeline)
      - [Dataloader](#dataloader)
    - [Evaluation config](#evaluation-config)
    - [Training and testing config](#training-and-testing-config)

- [Data pipeline](#data-pipeline)
    - [Optimization config](#optimization-config)
    - [Hook config](#hook-config)
    - [Runtime config](#runtime-config)
  - [An example of StyleGAN2](#an-example-of-stylegan2)
    - [Model config](#model-config-1)
    - [Dataset and evaluator config](#dataset-and-evaluator-config)
    - [Training and testing config](#training-and-testing-config-1)
    - [Optimization config](#optimization-config-1)
    - [Hook config](#hook-config-1)
    - [Runtime config](#runtime-config-1)
  - [Other examples](#other-examples)
    - [An example of config system for inpainting](#an-example-of-config-system-for-inpainting)
    - [An example of config system for matting](#an-example-of-config-system-for-matting)
    - [An example of config system for restoration](#an-example-of-config-system-for-restoration)

- [An example of config system for matting](#an-example-of-config-system-for-matting)

  For example, `--cfg-options test_cfg.use_ema=False` changes the default sampling model to the original generator,
  and  `--cfg-options train_dataloader.batch_size=8` changes the batch size of train dataloader.

  Some config dicts are composed as a list in your config.
  For example, the training pipeline `train_dataloader.dataset.pipeline` is normally a list

  you may specify `--cfg-options train_dataloader.dataset.pipeline.0.type=LoadImageFromWebcam`.
  The training pipeline `train_pipeline` is normally a list
  e.g. `[dict(type='LoadImageFromFile'), ...]`. If you want to change `'LoadImageFromFile'` to `'LoadMask'` in the pipeline,
  you may specify `--cfg-options train_pipeline.0.type=LoadMask`.

you may specify `--cfg-options train_pipeline.0.type=LoadMask`.

  If the value to be updated is a list or a tuple. You can set `--cfg-options key="[a,b]"` or `--cfg-options key=a,b`. It also allows nested list/tuple values, e.g., `--cfg-options key="[(a,b),(c,d)]"`. Note that the quotation mark " is necessary to support list/tuple data types, and that **NO** white space is allowed inside the quotation marks in the specified value.

## Config file structure

There are 3 basic component types under `config/_base_`: datasets, models and default_runtime.
Many methods could be easily constructed with one of each like AOT-GAN, EDVR, GLEAN, StyleGAN2, CycleGAN, SinGAN, etc.
Configs consisting of components from `_base_` are called _primitive_.

For example, if some modification is made base on BasicVSR,
user may first inherit the basic BasicVSR structure by specifying `_base_ = ../basicvsr/basicvsr_reds4.py`,
then modify the necessary fields in the config files.
If some modification is made base on StyleGAN2,
user may first inherit the basic StyleGAN2 structure by specifying `_base_ = ../styleganv2/stylegan2_c2_ffhq_256_b4x8_800k.py`,

If you are building an entirely new method that does not share the structure with any of the existing methods,
you may create a folder `xxx` under `configs`,

Please refer to [MMEngine](https://github.com/open-mmlab/mmengine/blob/main/docs/en/advanced_tutorials/config.md) for detailed documentation.

```
{model}_[module setting]_{training schedule}_{dataset}
```

```
```

- `{model}`: model type like `stylegan`, `dcgan`, `basicvsr`, `dim`, etc.
  Settings referred in the original paper are included in this field as well (e.g., `Stylegan2-config-f`, `edvrm` of `edvrm_8xb4-600k_reds`.)
- `[module setting]`: specific setting for some modules, including Encoder, Decoder, Generator, Discriminator, Normalization, loss, Activation, etc. E.g. `c64n7` of `basicvsr-pp_c64n7_8xb1-600k_reds4`, learning rate `Glr4e-4_Dlr1e-4` for dcgan, `gamma32.8` for stylegan3, `woReLUInplace` in sagan. In this section, information from different submodules (e.g., generator and discriminator) are connected with `_`.

- `{training_scheduler}`: specific setting for training, including batch_size, schedule, etc. For example, learning rate (e.g., `lr1e-3`), number of gpu and batch size is used (e.g., `8xb32`), and total iterations (e.g., `160kiter`) or number of images shown in the discriminator (e.g., `12Mimgs`).
- `{dataset}`: dataset name and data size info like `celeba-256x256` of `deepfillv1_4xb4_celeba-256x256`, `reds4` of `basicvsr_2xb4_reds4`, `ffhq`, `lsun-car`, `celeba-hq`.

## An example of EDSR

To help the users have a basic idea of a complete config,
we make a brief comments on the [config of the EDSR model](https://github.com/open-mmlab/mmagic/blob/main/configs/edsr/edsr_x2c64b16_g1_300k_div2k.py) we implemented as the following.
For more detailed usage and the corresponding alternative for each modules,
please refer to the API documentation and the [tutorial in MMEngine](https://github.com/open-mmlab/mmengine/blob/main/docs/en/advanced_tutorials/config.md).

In MMagic's config, we use model fields to set up a model.

```python
    type='BaseEditModel',  # Name of the model
    generator=dict(  # Config of the generator
        type='EDSRNet',  # Type of the generator
        in_channels=3,  # Channel number of inputs
        out_channels=3,  # Channel number of outputs
        mid_channels=64,  # Channel number of intermediate features
        num_blocks=16,  # Block number in the trunk network
        upscale_factor=scale, # Upsampling factor
        res_scale=1,  # Used to scale the residual in residual block
        rgb_mean=(0.4488, 0.4371, 0.4040),  # Image mean in RGB orders
        rgb_std=(1.0, 1.0, 1.0)),  # Image std in RGB orders
    pixel_loss=dict(type='L1Loss', loss_weight=1.0, reduction='mean')  # Config for pixel loss

rgb_std=(1.0, 1.0, 1.0)),  # Image std in RGB orders
    train_cfg=dict(),  # Config of training model.
    test_cfg=dict(),  # Config of testing model.
    data_preprocessor=dict(  # The Config to build data preprocessor
        type='DataPreprocessor', mean=[0., 0., 0.], std=[255., 255.,
                                                             255.]))
```

### Data config

[Dataloaders](https://pytorch.org/docs/stable/data.html?highlight=data%20loader#torch.utils.data.DataLoader) are required for the training, validation, and testing of the [runner](https://mmengine.readthedocs.io/en/latest/tutorials/runner.html).
Dataset and data pipeline need to be set to build the dataloader. Due to the complexity of this part, we use intermediate variables to simplify the writing of dataloader configs.

#### Data pipeline

```python

```python
        key='img',  # Keys in results to find the corresponding path
        key='gt',  # Keys in results to find the corresponding path

        ),
    dict(type='PackInputs')  # The config of collecting data from the current pipeline
]

key='img',  # Keys in results to find corresponding path
]
```

#### Dataloader

```python

persistent_workers=False,  # Whether maintain the workers Dataset instances alive

```

### Evaluation config

[Evaluators](https://mmengine.readthedocs.io/en/latest/tutorials/evaluation.html) are used to compute the metrics of the trained model on the validation and testing datasets.
The config of evaluators consists of one or a list of metric configs:

```python
    dict(type='MAE'),  # The name of metrics to evaluate

```python
    dict(type='SSIM', crop_border=scale),  # The name of metrics to evaluate
]
test_evaluator = val_evaluator # The config for testing evaluator
```

Users can set the maximum training iteration and validation intervals with these fields.

```python
    type='IterBasedTrainLoop',  # The name of train loop type
    max_iters=300000,  # The number of total iterations
    val_interval=5000,  # The number of validation interval iterations
)
```

)
```

`optim_wrapper` is the field to configure optimization related settings.
The optimizer wrapper not only provides the functions of the optimizer, but also supports functions such as gradient clipping, mixed precision training, etc. Find more in [optimizer wrapper tutorial](https://mmengine.readthedocs.io/en/latest/tutorials/optim_wrapper.html).

```python
    dict(
        optimizer=dict(type='Adam', lr=0.00001),
    )
```

`param_scheduler` is a field that configures methods of adjusting optimization hyper-parameters such as learning rate and momentum.
Users can combine multiple schedulers to create a desired parameter adjustment strategy.

Find more in [parameter scheduler tutorial](https://mmengine.readthedocs.io/en/latest/tutorials/param_scheduler.html).

```python
    type='MultiStepLR', by_epoch=False, milestones=[200000], gamma=0.5)
```

Users can attach hooks to training, validation, and testing loops to insert some operations during running. There are two different hook fields, one is `default_hooks` and the other is `custom_hooks`.

`default_hooks` is a dict of hook configs. `default_hooks` are the hooks must required at runtime. They have default priority which should not be modified. If not set, runner will use the default values. To disable a default hook, users can set its config to `None`.

```python

```python
    ),
)
```

`custom_hooks` is a list of hook configs. Users can develop there own hooks and insert them in this field.

```python
custom_hooks = [dict(type='BasicVisualizationHook', interval=1)] # Config of visualization hook
```

```python
default_scope = 'mmagic' # Used to set registries location

```python
)
```

## An example of StyleGAN2

Taking [Stylegan2 at 1024x1024 scale](https://github.com/open-mmlab/mmagic/blob/main/configs//styleganv2/stylegan2_c2_8xb4-fp16-global-800kiters_quicktest-ffhq-256x256.py) as an example,
we introduce each field in the config according to different function modules.

In addition to neural network components such as generator, discriminator etc, it also requires `data_preprocessor`, `loss_config`, and some of them contains `ema_config`.
`data_preprocessor` is responsible for processing a batch of data output by dataloader.
`loss_config` is responsible for weight of loss terms.
`ema_config` is responsible for exponential moving average (EMA) operation for generator.

```python
    type='StyleGAN2',  # The name of the model
    data_preprocessor=dict(type='DataPreprocessor'),  # The config of data preprocessor, usually includs image normalization and padding
    generator=dict(  # The config for generator
        type='StyleGANv2Generator',  # The name of the generator
        out_size=1024,  # The output resolution of the generator
        style_channels=512),  # The number of style channels of the generator

out_size=1024,  # The output resolution of the generator
    discriminator=dict(  # The config for discriminator
        type='StyleGAN2Discriminator',  # The name of the discriminator
        in_size=1024),  # The input resolution of the discriminator
    ema_config=dict(  # The config for EMA
        type='ExponentialMovingAverage',  # Specific the type of Average model
        interval=1,  # The interval of EMA operation
        momentum=0.9977843871238888),  # The momentum of EMA operation
    loss_config=dict(  # The config for loss terms
        r1_loss_weight=80.0,  # The weight for r1 gradient penalty
        r1_interval=16,  # The interval of r1 gradient penalty
        norm_mode='HWC',  # The normalization mode for r1 gradient penalty
        g_reg_interval=4,  # The interval for generator's regularization
        g_reg_weight=8.0,  # The weight for generator's regularization

g_reg_interval=4,  # The interval for generator's regularization
        pl_batch_shrink=2))  # The factor of shrinking the batch size in path length regularization
```

```python
dataset_type = 'BasicImageDataset'  # Dataset type, this will be used to define the dataset
data_root = './data/ffhq/'  # Root path of data

train_pipeline = [  # Training data process pipeline

    dict(type='LoadImageFromFile', key='img'),  # First pipeline to load images from file path
    dict(type='Flip', keys=['img'], direction='horizontal'),  # Argumentation pipeline that flip the images
    dict(type='PackInputs', keys=['img'])  # The last pipeline that formats the annotation data (if have) and decides which keys in the data should be packed into data_samples
]
]
train_dataloader = dict(  # The config of train dataloader
    batch_size=4,  # Batch size of a single GPU

]
    num_workers=8,  # Worker to pre-fetch data for each single GPU
    persistent_workers=True,  # If ``True``, the dataloader will not shutdown the worker processes after an epoch end, which can accelerate training speed.
    sampler=dict(  # The config of training data sampler
        type='InfiniteSampler',  # InfiniteSampler for iteratiion-based training. Refers to https://github.com/open-mmlab/mmengine/blob/fe0eb0a5bbc8bf816d5649bfdd34908c258eb245/mmengine/dataset/sampler.py#L107
        shuffle=True),  # Whether randomly shuffle the training data
    dataset=dict(  # The config of the training dataset
val_dataloader = dict(  # The config of validation dataloader

    dataset=dict(  # The config of the validation dataset
    sampler=dict(  # The config of validatioin data sampler
        type='DefaultSampler',  # DefaultSampler which supports both distributed and non-distributed training. Refer to https://github.com/open-mmlab/mmengine/blob/fe0eb0a5bbc8bf816d5649bfdd34908c258eb245/mmengine/dataset/sampler.py#L14
        shuffle=False),  # Whether randomly shuffle the validation data
    persistent_workers=True)
test_dataloader = val_dataloader  # The config of the testing dataloader
```

```

```python
val_evaluator = dict(  # The config for validation evaluator
    type='Evaluator',  # The type of evaluation
    metrics=[  # The config for metrics
        dict(
            type='FrechetInceptionDistance',
            sample_model='ema'),
        dict(type='PrecisionAndRecall', fake_nums=50000, prefix='PR-50K'),
        dict(type='PerceptualPathLength', fake_nums=50000, prefix='ppl-w')
    ])
test_evaluator = val_evaluator  # The config for testing evaluator
```

])
```

```python
train_cfg = dict(  # The config for training
    by_epoch=False,  # Set `by_epoch` as False to use iteration-based training
    val_begin=1,  # Which iteration to start the validation
    val_interval=10000,  # Validation intervals
    max_iters=800002)  # Maximum training iterations
val_cfg = dict(type='MultiValLoop')  # The validation loop type
test_cfg = dict(type='MultiTestLoop')  # The testing loop type
```

```

```python
        optimizer=dict(type='Adam', lr=0.0016, betas=(0, 0.9919919678228657))),
        optimizer=dict(
            type='Adam',
            lr=0.0018823529411764706,
            betas=(0, 0.9905854573074332))))
```

`param_scheduler` is a field that configures methods of adjusting optimization hyperparameters such as learning rate and momentum.

Since StyleGAN2 do not use parameter scheduler, we use config in [CycleGAN](https://github.com/open-mmlab/mmagic/blob/main/configs/cyclegan/cyclegan_lsgan-id0-resnet-in_1xb1-250kiters_summer2winter.py) as an example:

```python
# parameter scheduler in CycleGAN config
    type='LinearLrInterval',  # The type of scheduler
    interval=400,  # The interval to update the learning rate
    by_epoch=False,  # The scheduler is called by iteration
    start_factor=0.0002,  # The number we multiply parameter value in the first iteration
    end_factor=0,  # The number we multiply parameter value at the end of linear changing process.
    begin=40000,  # The start iteration of the scheduler
    end=80000)  # The end iteration of the scheduler
```

begin=40000,  # The start iteration of the scheduler
```

```python
    logger=dict(type='LoggerHook', interval=100, log_metric_by_epoch=False),
        save_best='FID-Full-50k/fid'))
```

interval=10000,
```

```python
    dict(
        vis_kwargs_list=dict(type='GAN', name='fake_img'))
]
```

```python
default_scope = 'mmagic'  # The default registry scope to find modules. Refer to https://mmengine.readthedocs.io/en/latest/tutorials/registry.html

# config for environment
    cudnn_benchmark=True,  # whether to enable cudnn benchmark.
    mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),  # set multi process parameters.
    dist_cfg=dict(backend='nccl'),  # set distributed parameters.
)

dist_cfg=dict(backend='nccl'),  # set distributed parameters.
)

    type='LogProcessor',  # log processor to process runtime logs
    by_epoch=False)  # print log by iteration
load_from = None  # load model checkpoint as a pre-trained model for a given path
resume = False  # Whether to resume from the checkpoint define in `load_from`. If `load_from` is `None`, it will resume the latest checkpoint in `work_dir`
```

## Other examples

### An example of config system for inpainting

To help the users have a basic idea of a complete config and the modules in a inpainting system,
we make brief comments on the config of Global&Local as the following.
For more detailed usage and the corresponding alternative for each modules, please refer to the API documentation.

```python
    type='GLInpaintor', # The name of inpaintor

```python
        type='DataPreprocessor', # The name of data preprocessor
        mean=[127.5], # Mean value used in data normalization
        std=[127.5], # Std value used in data normalization
    ),
    encdec=dict(
        type='GLEncoderDecoder', # The name of encoder-decoder
        encoder=dict(type='GLEncoder', norm_cfg=dict(type='SyncBN')), # The config of encoder
        decoder=dict(type='GLDecoder', norm_cfg=dict(type='SyncBN')), # The config of decoder
        dilation_neck=dict(
            type='GLDilationNeck', norm_cfg=dict(type='SyncBN'))), # The config of dilation neck
    disc=dict(
        type='GLDiscs', # The name of discriminator
        global_disc_cfg=dict(
            in_channels=3, # The input channel of discriminator
            max_channels=512, # The maximum middle channel in discriminator

global_disc_cfg=dict(
            fc_in_channels=512 * 4 * 4, # The input channel of last fc layer
            fc_out_channels=1024, # The output channel of last fc channel
            num_convs=6, # The number of convs used in discriminator
            norm_cfg=dict(type='SyncBN') # The config of norm layer
        ),
        local_disc_cfg=dict(
            num_convs=5, # The number of convs used in discriminator
        ),
    ),
    loss_gan=dict(

num_convs=5, # The number of convs used in discriminator
        ),
    ),
        type='GANLoss', # The name of GAN loss
        gan_type='vanilla', # The type of GAN loss
        loss_weight=0.001 # The weight of GAN loss
    ),
    loss_l1_hole=dict(
        type='L1Loss', # The type of l1 loss
        loss_weight=1.0 # The weight of l1 loss
    ))

    type='IterBasedTrainLoop',# The name of train loop type
    max_iters=500002, # The number of total iterations
    val_interval=50000, # The number of validation interval iterations
)
val_cfg = dict(type='ValLoop') # The name of validation loop type
test_cfg = dict(type='TestLoop') # The name of test loop type

    dict(type='MAE', mask_key='mask', scaling=100), # The name of metrics to evaluate
    dict(type='PSNR'), # The name of metrics to evaluate

    dict(type='SSIM'), # The name of metrics to evaluate
]

input_shape = (256, 256) # The shape of input image

    dict(type='LoadImageFromFile', key='gt'), # The config of loading image
    dict(
        type='LoadMask', # The type of loading mask pipeline
        mask_mode='bbox', # The type of mask
        mask_config=dict(
            max_bbox_shape=(128, 128), # The shape of bbox
            max_bbox_delta=40, # The changing delta of bbox height and width
            min_margin=20,  # The minimum margin from bbox to the image border
            img_shape=input_shape)),  # The input image shape
    dict(
        type='Crop', # The type of crop pipeline
        keys=['gt'],  # The keys of images to be cropped
        crop_size=(384, 384),  # The size of cropped patch

dict(
        random_crop=True,  # Whether to use random crop
    ),
    dict(
        type='Resize',  # The type of resizing pipeline
        keys=['gt'],  # They keys of images to be resized
        scale=input_shape,  # The scale of resizing function
        keep_ratio=False,  # Whether to keep ratio during resizing
    ),
    dict(
        type='Normalize',  # The type of normalizing pipeline
        keys=['gt_img'],  # The keys of images to be normed
        mean=[127.5] * 3,  # Mean value used in normalization
        std=[127.5] * 3,  # Std value used in normalization
        to_rgb=False),  # Whether to transfer image channels to rgb
    dict(type='GetMaskedImage'), # The config of getting masked image pipeline
    dict(type='PackInputs'), # The config of collecting data from the current pipeline
]

dict(type='GetMaskedImage'), # The config of getting masked image pipeline
]

test_pipeline = train_pipeline  # Constructing testing/validation pipeline

dataset_type = 'BasicImageDataset' # The type of dataset
data_root = 'data/places'  # Root path of data

    batch_size=12, # Batch size of a single GPU
    num_workers=4, # The number of workers to pre-fetch data for each single GPU
    persistent_workers=False, # Whether maintain the workers Dataset instances alive
    sampler=dict(type='InfiniteSampler', shuffle=False), # The type of data sampler
        type=dataset_type, # Type of dataset
        data_root=data_root, # Root path of data
        data_prefix=dict(gt='data_large'), # Prefix of image path
        ann_file='meta/places365_train_challenge.txt', # Path of annotation file
        test_mode=False,

data_prefix=dict(gt='data_large'), # Prefix of image path
    ))

    batch_size=1, # Batch size of a single GPU
    drop_last=False, # Whether drop the last incomplete batch
    sampler=dict(type='DefaultSampler', shuffle=False), # The type of data sampler
    dataset=dict( # Validation dataset config
        data_prefix=dict(gt='val_large'), # Prefix of image path
        ann_file='meta/places365_val.txt', # Path of annotation file
    ))

ann_file='meta/places365_val.txt', # Path of annotation file
    ))

model_wrapper_cfg = dict(type='MMSeparateDistributedDataParallel') # The name of model wrapper

optim_wrapper = dict( # Config used to build optimizer, support all the optimizers in PyTorch whose arguments are also the same as those in PyTorch
        type='OptimWrapper', optimizer=dict(type='Adam', lr=0.0004)),
    disc=dict(type='OptimWrapper', optimizer=dict(type='Adam', lr=0.0004)))

save_dir = './work_dirs' # Directory to save the model checkpoints and logs for the current experiments
exp_name = 'gl_places'  # The experiment name

default_hooks = dict( # Used to build default hooks

    logger=dict(type='LoggerHook', interval=100), # Config to register logger hook
    checkpoint=dict( # Config to set the checkpoint hook
        interval=50000,
        out_dir=save_dir),
)

env_cfg = dict( # Parameters to setup distributed training, the port can also be set
)

vis_backends = [dict(type='LocalVisBackend')] # The name of visualization backend
visualizer = dict( # Config used to build visualizer

type='ConcatImageVisualizer',
custom_hooks = [dict(type='BasicVisualizationHook', interval=1)] # Used to build custom hooks

log_processor = dict(type='LogProcessor', by_epoch=False) # Used to build log processor

load_from = None # load models as a pre-trained model from a given path. This will not resume training.
resume = False # Resume checkpoints from a given path, the training will be resumed from the epoch when the checkpoint's is saved.

find_unused_parameters = False  # Whether to set find unused parameters in ddp
```

### An example of config system for matting

```

To help the users have a basic idea of a complete config, we make a brief comments on the config of the original DIM model we implemented as the following. For more detailed usage and the corresponding alternative for each modules, please refer to the API documentation.

```python
    type='DIM',  # The name of model (we call mattor).
        proc_inputs='normalize',
        proc_trimap='rescale_to_zero_one',
        proc_gt='rescale_to_zero_one',
    ),
    backbone=dict(  # The config of the backbone.
        type='SimpleEncoderDecoder',  # The type of the backbone.
        encoder=dict(  # The config of the encoder.

),
            type='VGG16'),  # The type of the encoder.
        decoder=dict(  # The config of the decoder.
            type='PlainDecoder')),  # The type of the decoder.
    pretrained='./weights/vgg_state_dict.pth',  # The pretrained weight of the encoder to be loaded.
    loss_alpha=dict(  # The config of the alpha loss.
        type='CharbonnierLoss',  # The type of the loss for predicted alpha matte.
        loss_weight=0.5),  # The weight of the alpha loss.
    loss_comp=dict(  # The config of the composition loss.
        type='CharbonnierCompLoss',  # The type of the composition loss.
        loss_weight=0.5), # The weight of the composition loss.
    train_cfg=dict(  # Config of training DIM model.
        train_backbone=True,  # In DIM stage1, backbone is trained.

loss_weight=0.5), # The weight of the composition loss.
        train_refiner=False),  # In DIM stage1, refiner is not trained.
    test_cfg=dict(  # Config of testing DIM model.
        refine=False,  # Whether use refiner output as output, in stage1, we don't use it.
        resize_method='pad',
        resize_mode='reflect',
        size_divisor=32,
    ),
)

dataset_type = 'AdobeComp1kDataset'  # Dataset type, this will be used to define the dataset.
data_root = 'data/adobe_composition-1k'  # Root path of data.

train_pipeline = [  # Training data processing pipeline.
    dict(
        type='LoadImageFromFile',  # Load alpha matte from file.
        key='alpha',  # Key of alpha matte in annotation file. The pipeline will read alpha matte from path `alpha_path`.

dict(
        color_type='grayscale'),  # Load as grayscale image which has shape (height, width).
    dict(
        type='LoadImageFromFile',  # Load image from file.
        key='fg'),  # Key of image to load. The pipeline will read fg from path `fg_path`.
    dict(
        key='bg'),  # Key of image to load. The pipeline will read bg from path `bg_path`.
    dict(
        key='merged'),  # Key of image to load. The pipeline will read merged from path `merged_path`.
    dict(
        type='CropAroundUnknown',  # Crop images around unknown area (semi-transparent area).
        keys=['alpha', 'merged', 'fg', 'bg'],  # Images to crop.

dict(
        crop_sizes=[320, 480, 640]),  # Candidate crop size.
    dict(
        type='Flip',  # Augmentation pipeline that flips the images.
        keys=['alpha', 'merged', 'fg', 'bg']),  # Images to be flipped.
    dict(
        type='Resize',  # Augmentation pipeline that resizes the images.
        keys=['alpha', 'merged', 'fg', 'bg'],  # Images to be resized.
        scale=(320, 320),  # Target size.
        keep_ratio=False),  # Whether to keep the ratio between height and width.
    dict(
        type='GenerateTrimap',  # Generate trimap from alpha matte.
        kernel_size=(1, 30)),  # Kernel size range of the erode/dilate kernel.
    dict(type='PackInputs'),  # The config of collecting data from the current pipeline
]
    dict(
        type='LoadImageFromFile',  # Load alpha matte.

dict(type='PackInputs'),  # The config of collecting data from the current pipeline
]
    dict(
        save_original_img=True),
    dict(
        type='LoadImageFromFile',  # Load image from file
        key='trimap',  # Key of image to load. The pipeline will read trimap from path `trimap_path`.
        color_type='grayscale',  # Load as grayscale image which has shape (height, width).
        save_original_img=True),  # Save a copy of trimap for calculating metrics. It will be saved with key `ori_trimap`
    dict(

dict(
]

        data_root=data_root,  # Root path of data
        ann_file='training_list.json',  # Path of annotation file
    ))

test_mode=False,
    ))

        ann_file='test_list.json',  # Path of annotation file
    ))

    dict(type='SAD'),  # The name of metrics to evaluate
    dict(type='MattingMSE'),  # The name of metrics to evaluate
    dict(type='GradientError'),  # The name of metrics to evaluate

dict(type='SAD'),  # The name of metrics to evaluate
    dict(type='ConnectivityError'),  # The name of metrics to evaluate
]

    max_iters=1_000_000,  # The number of total iterations
    val_interval=40000,  # The number of validation interval iterations
)

    dict(
    )

default_scope = 'mmagic'  # Used to set registries location

)

save_dir = './work_dirs'  # Directory to save the model checkpoints and logs for the current experiments.

        interval=40000,  # The save interval is 40000 iterations.
        by_epoch=False,  # Count by iterations.
)

)

)

log_processor = dict(type='LogProcessor', by_epoch=False)  # Used to build log processor

```

### An example of config system for restoration

To help the users have a basic idea of a complete config, we make a brief comments on the config of the EDSR model we implemented as the following. For more detailed usage and the corresponding alternative for each modules, please refer to the API documentation.

```python
exp_name = 'edsr_x2c64b16_1x16_300k_div2k'  # The experiment name

```python
work_dir = f'./work_dirs/{experiment_name}'
save_dir = './work_dirs/'

load_from = None  # based on pre-trained x2 model

scale = 2  # Scale factor for upsampling

                                                             255.]))

        ),

dict(type='RandomTransposeHW',  # Random transpose h and w for images
        ),
]
    dict(type='ToTensor', keys=['img', 'gt']),  # Convert images to tensor

]

ann_file='meta_info_DIV2K800sub_GT.txt',  # Path of annotation file

type=dataset_type,  # Type of dataset

]

    dict(
    )

    dict(
    )

    ),
)

logger=dict(type='LoggerHook', interval=100),  # Config to register logger hook
)

)

```

# CSL

> [Arbitrary-Oriented Object Detection with Circular Smooth Label](https://link.springer.com/chapter/10.1007/978-3-030-58598-3_40)

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/csl.jpg" width="800"/>
</div>

Arbitrary-oriented object detection has recently attracted increasing attention in vision for their importance
in aerial imagery, scene text, and face etc. In this paper, we show that existing regression-based rotation detectors
suffer the problem of discontinuous boundaries, which is directly caused by angular periodicity or corner ordering.
By a careful study, we find the root cause is that the ideal predictions are beyond the defined range. We design a
new rotation detection baseline, to address the boundary problem by transforming angular prediction from a regression
problem to a classification task with little accuracy loss, whereby high-precision angle classification is devised in

contrast to previous works using coarse-granularity in rotation detection. We also propose a circular smooth label (CSL)
technique to handle the periodicity of the angle and increase the error tolerance to adjacent angles. We further
introduce four window functions in CSL and explore the effect of different window radius sizes on detection performance.
Extensive experiments and visual analysis on two large-scale public datasets for aerial images i.e. DOTA, HRSC2016,
as well as scene text dataset ICDAR2015 and MLT, show the effectiveness of our approach.

DOTA1.0

DOTA1.0

|         Backbone         |  mAP  | Angle | Window func. | Omega | lr schd | Mem (GB) | Inf Time (fps) | Aug | Batch Size |                                                              Configs                                                              |                                                                                                                                                                                                      Download                                                                                                                                                                                                      |

| ResNet50 (1024,1024,200) | 68.42 | le90  |      -       |   -   |   1x    |   3.38   |      17.8      |  -  |     2      |        [rotated-retinanet-rbox-le90_r50_fpn_1x_dota](../rotated_retinanet/rotated-retinanet-rbox-le90_r50_fpn_1x_dota.py)         |                       [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le90/rotated_retinanet_obb_r50_fpn_1x_dota_le90-c0097bc4.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le90/rotated_retinanet_obb_r50_fpn_1x_dota_le90_20220128_130740.log.json)                       |

| ResNet50 (1024,1024,200) | 68.79 | le90  |      -       |   -   |   1x    |   2.36   |      25.9      |  -  |     2      |    [rotated-retinanet-rbox-le90_r50_fpn_amp-1x_dota](../rotated_retinanet/rotated-retinanet-rbox-le90_r50_fpn_amp-1x_dota.py)     |             [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_fp16_1x_dota_le90/rotated_retinanet_obb_r50_fpn_fp16_1x_dota_le90-01de71b5.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_fp16_1x_dota_le90/rotated_retinanet_obb_r50_fpn_fp16_1x_dota_le90_20220303_183714.log.json)             |

| ResNet50 (1024,1024,200) | 69.51 | le90  |   Gaussian   |   4   |   1x    |   2.60   |      24.0      |  -  |     2      | [rotated-retinanet-rbox-le90_r50_fpn_csl-gaussian_amp-1x_dota](./rotated-retinanet-rbox-le90_r50_fpn_csl-gaussian_amp-1x_dota.py) | [model](https://download.openmmlab.com/mmrotate/v0.1.0/csl/rotated_retinanet_obb_csl_gaussian_r50_fpn_fp16_1x_dota_le90/rotated_retinanet_obb_csl_gaussian_r50_fpn_fp16_1x_dota_le90-b4271aed.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/csl/rotated_retinanet_obb_csl_gaussian_r50_fpn_fp16_1x_dota_le90/rotated_retinanet_obb_csl_gaussian_r50_fpn_fp16_1x_dota_le90_20220321_010033.log.json) |

```
@inproceedings{yang2020arbitrary,
    title={Arbitrary-Oriented Object Detection with Circular Smooth Label},
    author={Yang, Xue and Yan, Junchi},
    booktitle={European Conference on Computer Vision},
    pages={677--694},
}
```

# 下游任务

## 检测

我们使用 MMDetection 进行图像检测。首先确保您已经安装了 [MIM](https://github.com/open-mmlab/mim)，这也是 OpenMMLab 的一个项目。

```shell
```

此外，请参考 MMDetection 的[安装](https://mmdetection.readthedocs.io/en/dev-3.x/get_started.html)和[数据准备](https://mmdetection.readthedocs.io/en/dev-3.x/user_guides/dataset_prepare.html)

### 训练

安装完后，您可以使用如下的简单命令运行 MMDetection。

```shell

```

- `${CONFIG}`：直接用 MMDetection 中的配置文件路径即可。对于一些算法，我们有一些修改过的配置文件，
  可以在相应算法文件夹下的 `benchmarks` 文件夹中找到。另外，您也可以从头开始编写配置文件。
- `${PRETRAIN}`：预训练模型文件
- `${GPUS}`：使用多少 GPU 进行训练，对于检测任务，我们默认使用 8 个 GPU。

例子：

```

例子：

```shell
```

### 测试

在训练之后，您可以运行如下命令测试您的模型。

```shell

```

备注：

- `${CHECKPOINT}`：您想测试的训练好的检测模型。

例子：

```shell
configs/benchmarks/mmdetection/coco/mask-rcnn_r50_fpn_ms-1x_coco.py \

```

备注：

例子：

```shell
```

## 分割

我们使用 MMSegmentation 进行图像分割。首先确保您已经安装了 [MIM](https://github.com/open-mmlab/mim)，这也是 OpenMMLab 的一个项目。

```shell
```

此外，请参考 MMSegmentation 的[安装](https://mmsegmentation.readthedocs.io/en/dev-1.x/get_started.html)和[数据准备](https://mmsegmentation.readthedocs.io/en/dev-1.x/user_guides/2_dataset_prepare.html)。

### 训练

在安装完后，可以使用如下简单命令运行 MMSegmentation。

```shell

```

备注：

```

备注：

- `${CONFIG}`：直接用 MMSegmentation 中的配置文件路径即可。对于一些算法，我们有一些修改过的配置文件，

例子：

```shell
```

### 测试

```shell

```

备注：

- `${CHECKPOINT}`：您想测试的训练好的分割模型。

例子：

```shell
bash ./tools/benchmarks/mmsegmentation/mim_dist_test.sh \

```

备注：

例子：

```shell
```

<img src="https://user-images.githubusercontent.com/22607038/142795605-dfdd5f69-e9cd-4b69-9c6b-6d8bded18e89.png"/>
</div>

| [MaskRCNN](/configs/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_ctw1500.py) |                     -                     |        -         | CTW1500 Train | CTW1500 Test |   160   |   1600    |  0.7165   | 0.7776 | 0.7458 | [model](https://download.openmmlab.com/mmocr/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_ctw1500/mask-rcnn_resnet50_fpn_160e_ctw1500_20220826_154755-ce68ee8e.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_ctw1500/20220826_154755.log) |

| [MaskRCNN_r50-oclip](/configs/textdet/maskrcnn/mask-rcnn_resnet50-oclip_fpn_160e_ctw1500.py) | [ResNet50-oCLIP](https://download.openmmlab.com/mmocr/backbone/resnet50-oclip-7ba0c533.pth) |        -         | CTW1500 Train | CTW1500 Test |   160   |   1600    |   0.753   | 0.7593 | 0.7562 | [model](https://download.openmmlab.com/mmocr/textdet/maskrcnn/mask-rcnn_resnet50-oclip_fpn_160e_ctw1500/mask-rcnn_resnet50-oclip_fpn_160e_ctw1500_20221101_154448-6e9e991c.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/maskrcnn/mask-rcnn_resnet50-oclip_fpn_160e_ctw1500/20221101_154448.log) |

|                 Method                 |                 BackBone                 | Pretrained Model |  Training set   |    Test set    | #epochs | Test size | Precision | Recall | Hmean  |                 Download                 |

| :------------------------------------: | :--------------------------------------: | :--------------: | :-------------: | :------------: | :-----: | :-------: | :-------: | :----: | :----: | :--------------------------------------: |
| [MaskRCNN](/configs/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_icdar2015.py) |                 ResNet50                 |        -         | ICDAR2015 Train | ICDAR2015 Test |   160   |   1920    |  0.8644   | 0.7766 | 0.8182 | [model](https://download.openmmlab.com/mmocr/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_icdar2015/mask-rcnn_resnet50_fpn_160e_icdar2015_20220826_154808-ff5c30bf.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_icdar2015/20220826_154808.log) |

| [MaskRCNN_r50-oclip](/configs/textdet/maskrcnn/mask-rcnn_resnet50-oclip_fpn_160e_icdar2015.py) | [ResNet50-oCLIP](https://download.openmmlab.com/mmocr/backbone/resnet50-oclip-7ba0c533.pth) |        -         | ICDAR2015 Train | ICDAR2015 Test |   160   |   1920    |  0.8695   | 0.8339 | 0.8513 | [model](https://download.openmmlab.com/mmocr/textdet/maskrcnn/mask-rcnn_resnet50-oclip_fpn_160e_icdar2015/mask-rcnn_resnet50-oclip_fpn_160e_icdar2015_20221101_131357-a19f7802.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/maskrcnn/mask-rcnn_resnet50-oclip_fpn_160e_icdar2015/20221101_131357.log) |

```bibtex
@INPROCEEDINGS{8237584,
  author={K. {He} and G. {Gkioxari} and P. {Dollár} and R. {Girshick}},
  booktitle={2017 IEEE International Conference on Computer Vision (ICCV)},
  title={Mask R-CNN},
  pages={2980-2988},
  doi={10.1109/ICCV.2017.322}}
```

# Prompt Template

In language model evaluation, we often construct prompts from the original dataset according to certain rules to enable the model to answer questions as required.

Typically, we place instructions at the beginning of the prompt, followed by several in-context examples, and finally, we include the question. For example:

```text
Solve the following questions.
1+1=?
2
3+9=?
12
5+6=?
```

Extensive experiments have shown that even with the same original test questions, different ways of constructing the prompt can affect the model's performance. Factors that may influence this include:

- The composition of the prompt itself, including instructions, in-context examples, and the format of the question.
- The selection of in-context examples, including the number and method of selection.
- The manner in which the prompt is used. Should the model complete the prompt based on the given context, or should it choose the best prompt among the candidate prompts?

OpenCompass defines the prompt construction strategy in the `infer_cfg` section of the dataset configuration. A typical `infer_cfg` is shown below:

```python
infer_cfg = dict(
    ice_template=dict(  # Template used to construct In Context Examples (ice).
        template='{question}\n{answer}'
    ),
    prompt_template=dict(  # Template used to construct the main prompt.
        template='Solve the following questions.\n</E>{question}\n{answer}',
        ice_token="</E>"
    ),
    retriever=dict(type=FixKRetriever),  # Definition of how to retrieve in-context examples.
    inferencer=dict(type=GenInferencer, fix_id_list=[0, 1]),  # Method used to generate predictions.
)
```

inferencer=dict(type=GenInferencer, fix_id_list=[0, 1]),  # Method used to generate predictions.
)
```

In this document, we will mainly introduce the definitions of `ice_template`, `prompt_template`, and `inferencer`. For information on the `retriever`, please refer to other documents.

Let's start by introducing the basic syntax of the prompt.

## String-Based Prompt

String-based prompt is a classic form of template. Consider the following template:

```python
prompt_template=dict(
    type=PromptTemplate,
    template="{anything}\nQuestion: {question}\nAnswer: {answer}"
)
```

At runtime, the fields within the `{}` will be replaced with corresponding fields from the data sample. If a field does not exist in the data sample, it will be kept as is in the output.

For example, let's consider a data example as follows:

```python
example = {
    'question': '1+1=?',
    'answer': '2',  # Assume the answer is in the reader_cfg.output_column
    'irrelevant_infos': 'blabla',
}
```

```python
example = {
}
```

After filling in the template, the result will be:

```text
{anything}
Question: 1+1=?
Answer:
```

As you can see, the actual answer for the question, represented by the field `answer`, does not appear in the generated result. This is because OpenCompass will mask fields that are written in `reader_cfg.output_column` to prevent answer leakage. For detailed explanations on `reader_cfg`, please refer to the relevant documentation on dataset configuration.

## Dialogue-Based Prompt

In practical testing, making models perform simple completions may not effectively test the performance of chat-based models. Therefore, we prefer prompts that take the form of dialogues. Additionally, different models have varying definitions of dialogue formats. Hence, we need prompts generated from the dataset to be more versatile, and the specific prompts required by each model can be generated during testing.

To achieve this, OpenCompass extends the string-based prompt to dialogue-based prompt. Dialogue-based prompt is more flexible, as it can combine with different [meta_templates](./meta_template.md) on the model side to generate prompts in various dialogue formats. It is applicable to both base and chat models, but their definitions are relatively complex.

Now, let's assume we have a data sample as follows:

```python
example = {

```python
example = {
    'irrelavent_infos': 'blabla',
}
```

Next, let's showcase a few examples:

````{tab} Single-round Dialogue
```python
    template=dict(
        round=[
            dict(role="HUMAN", prompt="Question: {question}"),
            dict(role="BOT", prompt="Answer: {answer}"),
        ]
    )
)
```

The intermediate result obtained by OpenCompass after filling the data into the template is:

```python
    dict(role='HUMAN', prompt='Question: 1+1=?'),
    dict(role='BOT', prompt='Answer: '),
])
```

````

````{tab} Multi-round Dialogue
```python
        round=[
            dict(role="HUMAN", prompt="Question: 2+2=?"),
            dict(role="BOT", prompt="Answer: 4"),

```python
        round=[
            dict(role="HUMAN", prompt="Question: 3+3=?"),
            dict(role="BOT", prompt="Answer: 6"),
        ]
    )
)
```

```python
    dict(role='HUMAN', prompt='Question: 2+2=?'),
    dict(role='BOT', prompt='Answer: 4'),
    dict(role='HUMAN', prompt='Question: 3+3=?'),
    dict(role='BOT', prompt='Answer: 6'),
])
```
````

````{tab} Dialogue with sys instruction

```python
        begin=[

dict(role='BOT', prompt='Answer: '),
])
```
````

```python
        begin=[
            dict(role='SYSTEM', fallback_role='HUMAN', prompt='Solve the following questions.'),
        ],
        round=[
        ]
    )
)
```

```python
    dict(role='SYSTEM', fallback_role='HUMAN', prompt='Solve the following questions.'),
])
```

])
```

During the processing of a specific meta template, if the definition includes the SYSTEM role, the template designated for the SYSTEM role will be used for processing. On the other hand, if the SYSTEM role is not defined, the template assigned to the fallback_role role will be utilized, which, in this example, corresponds to the HUMAN role.

````

`````

In dialogue-based templates, prompts are organized in the form of conversations between different roles (`role`). In the current predefined dataset configuration of OpenCompass, some commonly used roles in a prompt include:

- `HUMAN`: Represents a human, usually the one asking questions.
- `BOT`: Represents the language model, usually the one providing answers.

- `SYSTEM`: Represents the system, typically used at the beginning of prompts to give instructions.

Furthermore, unlike string-based templates, the prompts generated by dialogue-based templates are transformed into an intermediate structure called PromptList. This structure will be further combined with the model-side [meta_templates](./meta_template.md) to assemble the final prompt. If no meta template is specified, the prompts in the PromptList will be directly concatenated into a single string.

```{note}

```{note}
The content within the PromptList in the example above is not the final input to the model and depends on the processing of the meta template. One potential source of misunderstanding is that in generative evaluations, the prompt of the last `BOT` role, `Answer: `, **will not** be inputted to the model. This is because API models generally cannot customize the initial part of model-generated responses. Therefore, this setting ensures consistency in the evaluation behavior between language models and API models. For more information, please refer to the documentation on [meta template](./meta_template.md).
```

<details>
<summary>Expand the complete parameter descriptions</summary>

- `begin`, `end`: (list, optional) The beginning and end of the prompt, typically containing system-level instructions. Each item inside can be **a dictionary or a string**.

- `round`: (list) The format of the dialogue in the template. Each item in the list must be a dictionary.

Each dictionary has the following parameters:

- `role` (str): The role name participating in the dialogue. It is used to associate with the names in meta_template but does not affect the actual generated prompt.

- `fallback_role` (str): The default role name to use in case the associated role is not found in the meta_template. Defaults to None.

- `prompt` (str): The dialogue content for the role.

## Prompt Templates and `inferencer`

Once we understand the basic definition of prompt templates, we also need to organize them according to the type of `inferencer`.

OpenCompass mainly supports two types of inferencers: `GenInferencer` and `PPLInferencer`, corresponding to two different inference methods.

`GenInferencer` corresponds to generative inference. During inference, the model is asked to continue generating text based on the input prompt. In this case, the `template` represents a single template for each sentence, for example:

````{group-tab} String-based Prompt
```python
    template='Solve the following questions.\n{question}\n{answer}'
)
```
````

````{group-tab} Dialogue-Based Prompt
```python
        begin=[
        ],
        round=[
            dict(role="HUMAN", prompt="{question}"),
            dict(role="BOT", prompt="{answer}"),
        ]
    )
)
```
````

`````

],
        round=[
        ]
    )
)
```
````

`````

Then, the model's inference result will be a continuation of the concatenated string.

For `PPLInferencer`, it corresponds to discriminative inference. During inference, the model is asked to compute the perplexity (PPL) for each input string and select the item with the lowest perplexity as the model's inference result. In this case, `template` is a `dict` representing the template for each sentence, for example:

```python
        "A": "Question: Which is true?\nA. {A}\nB. {B}\nC. {C}\nAnswer: A",
        "B": "Question: Which is true?\nA. {A}\nB. {B}\nC. {C}\nAnswer: B",
        "C": "Question: Which is true?\nA. {A}\nB. {B}\nC. {C}\nAnswer: C",

"B": "Question: Which is true?\nA. {A}\nB. {B}\nC. {C}\nAnswer: B",
        "UNK": "Question: Which is true?\nA. {A}\nB. {B}\nC. {C}\nAnswer: None of them is true.",
    )
)
```
````

```python
        "A": dict(
            round=[
                dict(role="HUMAN", prompt="Question: Which is true?\nA. {A}\nB. {B}\nC. {C}"),
                dict(role="BOT", prompt="Answer: A"),
            ]
        ),
        "B": dict(
            round=[
                dict(role="BOT", prompt="Answer: B"),
            ]
        ),
        "C": dict(
            round=[
                dict(role="BOT", prompt="Answer: C"),

),
        "C": dict(
            round=[
            ]
        ),
        "UNK": dict(
            round=[
                dict(role="BOT", prompt="Answer: None of them is true."),
            ]
        ),
    )
)
```
````

`````

In this case, the model's inference result will be one of the four keys in the `template` ("A" / "B" / "C" / "UNK").

## `ice_template` and `prompt_template`

In OpenCompass, for 0-shot evaluation, we usually only need to define the `prompt_template` field to complete prompt construction. However, for few-shot evaluation, we also need to define the `ice_template` field, which manages the prompt templates corresponding to the in-context examples during context learning.

Both `ice_template` and `prompt_template` follow the same syntax and rules. The complete prompt construction process can be represented using the following pseudo-code:

```python
def build_prompt():
    ice = ice_template.format(*ice_example)
    prompt = prompt_template.replace(prompt_template.ice_token, ice).format(*prompt_example)
    return prompt
```

Now, let's assume there are two training data (ex1, ex2) and one testing data (ex3):

```python
ex1 = {
    'question': '2+2=?',
    'answer': '4',
}
ex2 = {
    'question': '3+3=?',
    'answer': '6',
}
ex3 = {
}
```

Next, let's take a look at the actual effects of different prompt construction methods:

Template configurations are as follows:

```python
infer_cfg=dict(

```python
    ice_template=dict(
    ),
        template='Solve the following questions.\n</E>{question}\n{answer}'
        ice_token='</E>',
    )
)
```

The resulting strings are as follows:

```text
2+2=?
4
3+3=?
6
1+1=?

```

````

```python
        template=dict(
            round=[
                dict(role="HUMAN", prompt="{question}"),
                dict(role="BOT", prompt="{answer}"),
            ]
        )
    ),
            begin=[

dict(role="BOT", prompt="{answer}"),
            ]
        )
    ),
            begin=[
                dict(role='SYSTEM', fallback_role='HUMAN', prompt='Solve the following questions.'),
                '</E>',
            ],
            round=[
            ],
        ),
    )
)
```

The intermediate results obtained by OpenCompass after filling the data into the templates are as follows:

```python
    dict(role='HUMAN', prompt='3+3=?'),
    dict(role='BOT', prompt='6'),
    dict(role='BOT', prompt=''),
])
```
````

`````

### Abbreviated Usage

dict(role='HUMAN', prompt='3+3=?'),
])
```
````

`````

It is worth noting that, for the sake of simplicity in the configuration file, the `prompt_template` field can be omitted. When the `prompt_template` field is omitted, the `ice_template` will be used as the `prompt_template` as well, to assemble the complete prompt. The following two `infer_cfg` configurations are equivalent:

  <thead>
  <tr>
      <th>Complete Form</th>
      <th>Abbreviated Form</th>
  <tbody>
  <tr>
  <td>

```python
        template="Q: {question}\nA: {answer}",
    ),
        template="</E>Q: {question}\nA: {answer}",
        ice_token="</E>",
    ),
    # ...
)
```

</td>
  <td>

```python

type=PromptTemplate,
    ),
    # ...
)
```

</td>
  <td>

```python
    ),
    # ...
)
```

</td>
  </tr>
  </thead>
  </table>

More generally, even in the case of 0-shot learning (i.e., when `retriever` is `ZeroRetriver`), this mechanism still applies. Therefore, the following configuration is also valid:

```python
datasets = [
    dict(
        infer_cfg=dict(
            ice_template=dict(
                type=PromptTemplate,
                template="Q: {question}\nA: {answer}",
            ),
            retriever=dict(type=ZeroRetriever),
            inferencer=dict(type=GenInferencer),
        )
    ),
]
```

## Usage Suggestion

),
        )
    ),
]
```

It is suggested to use the [Prompt Viewer](../tools.md) tool to visualize the completed prompts, confirm the correctness of the templates, and ensure that the results meet expectations.

# SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation

> [SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation](https://arxiv.org/abs/2002.10111)

Estimating 3D orientation and translation of objects is essential for infrastructure-less autonomous navigation and driving. In case of monocular vision, successful methods have been mainly based on two ingredients: (i) a network generating 2D region proposals, (ii) a R-CNN structure predicting 3D object pose by utilizing the acquired regions of interest. We argue that the 2D detection network is redundant and introduces non-negligible noise for 3D detection. Hence, we propose a novel 3D object detection method, named SMOKE, in this paper that predicts a 3D bounding box for each detected object by combining a single keypoint estimate with regressed 3D variables. As a second contribution, we propose a multi-step disentangling approach for constructing the 3D bounding box, which significantly improves both training convergence and detection accuracy. In contrast to previous 3D detection techniques, our method does not require complicated pre/post-processing, extra data, and a refinement stage. Despite of its structural simplicity, our proposed SMOKE network outperforms all existing monocular 3D detection methods on the KITTI dataset, giving the best state-of-the-art result on both 3D object detection and Bird's eye view evaluation.

<img src="https://user-images.githubusercontent.com/79644370/143886681-52cb72b9-6635-4624-a728-1c243b046517.png" width="800"/>
</div>

We implement SMOKE and provide the results and checkpoints on KITTI dataset.

### KITTI

|                           Backbone                            | Lr schd | Mem (GB) | Inf time (fps) |  mAP  |                                                                                                                                                         Download                                                                                                                                                         |

| :-----------------------------------------------------------: | :-----: | :------: | :------------: | :---: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| [DLA34](./smoke_dla34_dlaneck_gn-all_4xb8-6x_kitti-mono3d.py) |   6x    |   9.64   |                | 13.85 | [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/smoke/smoke_dla34_pytorch_dlaneck_gn-all_8x4_6x_kitti-mono3d_20210929_015553-d46d9bb0.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/smoke/smoke_dla34_pytorch_dlaneck_gn-all_8x4_6x_kitti-mono3d_20210929_015553.log.json) |

Note: mAP represents Car moderate 3D strict AP11 results.

Detailed performance on KITTI 3D detection (3D/BEV) is as follows, evaluated by AP11 metric:

|            |     Easy      |   Moderate    |     Hard      |
| ---------- | :-----------: | :-----------: | :-----------: |
| Car        | 16.92 / 22.97 | 13.85 / 18.32 | 11.90 / 15.88 |
| Pedestrian | 11.13 / 12.61 | 11.10 / 11.32 | 10.67 / 11.14 |
| Cyclist    | 0.99  / 1.47  |  0.54 / 0.65  |  0.55 / 0.67  |

```latex
@inproceedings{liu2020smoke,
  title={Smoke: Single-stage monocular 3d object detection via keypoint estimation},
  author={Liu, Zechen and Wu, Zizhang and T{\'o}th, Roland},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops},
  pages={996--997},
}
```

# MMOCR Deployment

- [MMOCR Deployment](#mmocr-deployment)
    - [Install mmocr](#install-mmocr)
    - [Convert text detection model](#convert-text-detection-model)
    - [Convert text recognition model](#convert-text-recognition-model)
      - [Text detection SDK model inference](#text-detection-sdk-model-inference)
      - [Text Recognition SDK model inference](#text-recognition-sdk-model-inference)
  - [Reminder](#reminder)

[MMOCR](https://github.com/open-mmlab/mmocr/tree/main) aka `mmocr` is an open-source toolbox based on PyTorch and mmdetection for text detection, text recognition, and the corresponding downstream tasks including key information extraction. It is a part of the [OpenMMLab](https://openmmlab.com/) project.

### Install mmocr

Please follow the [installation guide](https://mmocr.readthedocs.io/en/latest/get_started/install.html) to install mmocr.

```shell
```

You can use [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/tree/main/tools/deploy.py) to convert mmocr models to the specified backend models. Its detailed usage can be learned from [here](https://github.com/open-mmlab/mmdeploy/tree/main/docs/en/02-how-to-run/convert_model.md#usage).

When using `tools/deploy.py`, it is crucial to specify the correct deployment config. We've already provided builtin deployment config [files](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmocr) of all supported backends for mmocr, under which the config file path follows the pattern:

```
```

- **{task}:** task in mmocr.

```
```

  MMDeploy supports models of two tasks of mmocr, one is `text detection` and the other is `text-recogntion`.

  **DO REMEMBER TO USE** the corresponding deployment config file when trying to convert models of different tasks.

In the next two chapters, we will task `dbnet` model from `text detection` task and `crnn` model from `text recognition` task respectively as examples, showing how to convert them to onnx model that can be inferred by ONNX Runtime.

### Convert text detection model

```shell
# download dbnet model from mmocr model zoo

```shell
mim download mmocr --config dbnet_resnet18_fpnc_1200e_icdar2015 --dest .
# convert mmocr model to onnxruntime model with dynamic shape
    configs/mmocr/text-detection/text-detection_onnxruntime_dynamic.py \
    dbnet_resnet18_fpnc_1200e_icdar2015.py \
    dbnet_resnet18_fpnc_1200e_icdar2015_20220825_221614-7c0e94f2.pth \
    demo/resources/text_det.jpg \
    --work-dir mmdeploy_models/mmocr/dbnet/ort \
    --show \
```

### Convert text recognition model

```shell
# download crnn model from mmocr model zoo
mim download mmocr --config crnn_mini-vgg_5e_mj --dest .
    configs/mmocr/text-recognition/text-recognition_onnxruntime_dynamic.py \
    crnn_mini-vgg_5e_mj.py \
    crnn_mini-vgg_5e_mj_20220826_224120-8afbedbb.pth \

    demo/resources/text_recog.jpg \
    --work-dir mmdeploy_models/mmocr/crnn/ort \
    --show \
```

You can also convert the above models to other backend models by changing the deployment config file `*_onnxruntime_dynamic.py` to [others](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmocr), e.g., converting `dbnet` to tensorrt-fp32 model by `text-detection/text-detection_tensorrt-_dynamic-320x320-2240x2240.py`.

```{tip}
When converting mmocr models to tensorrt models, --device should be set to "cuda"
```

```

The converted model locates in the working directory like `mmdeploy_models/mmocr/dbnet/ort` in the previous example. It includes:

```
mmdeploy_models/mmocr/dbnet/ort
```

in which,

The whole package **mmdeploy_models/mmocr/dbnet/ort** is defined as **mmdeploy SDK model**, i.e., **mmdeploy SDK model** includes both backend model and inference meta information.

Take the previous converted `end2end.onnx` mode of `dbnet` as an example, you can use the following code to inference the model and visualize the results.

```python

```python

deploy_cfg = 'configs/mmocr/text-detection/text-detection_onnxruntime_dynamic.py'
model_cfg = 'dbnet_resnet18_fpnc_1200e_icdar2015.py'
backend_model = ['./mmdeploy_models/mmocr/dbnet/ort/end2end.onnx']
image = './demo/resources/text_det.jpg'

    output_file='output_ocr.png')
```

**Tip**:

Map 'deploy_cfg', 'model_cfg', 'backend_model' and 'image' to corresponding arguments in chapter [convert text recognition model](#convert-text-recognition-model), you will get the ONNX Runtime inference results of `crnn` onnx model.

Given the above SDK models of `dbnet` and `crnn`, you can also perform SDK model inference like following,

#### Text detection SDK model inference

```python
import cv2
from mmdeploy_runtime import TextDetector

img = cv2.imread('demo/resources/text_det.jpg')
# create text detector
detector = TextDetector(

```python
import cv2

    model_path='mmdeploy_models/mmocr/dbnet/ort',
    device_name='cpu',
    device_id=0)
bboxes = detector(img)
# draw detected bbox into the input image
if len(bboxes) > 0:
    pts = ((bboxes[:, 0:8] + 0.5).reshape(len(bboxes), -1,
                                          2).astype(int))
    cv2.polylines(img, pts, True, (0, 255, 0), 2)
    cv2.imwrite('output_ocr.png', img)
```

#### Text Recognition SDK model inference

```python
import cv2
from mmdeploy_runtime import TextRecognizer

img = cv2.imread('demo/resources/text_recog.jpg')
# create text recognizer
recognizer = TextRecognizer(
  model_path='mmdeploy_models/mmocr/crnn/ort',
  device_name='cpu',
  device_id=0
)
texts = recognizer(img)
# print the result
print(texts)
```

)
```

| Model                                                                                | Task             | TorchScript | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO |
| :----------------------------------------------------------------------------------- | :--------------- | :---------: | :---------: | :------: | :--: | :---: | :------: |
| [DBNet](https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/dbnet)         | text-detection   |      Y      |      Y      |    Y     |  Y   |   Y   |    Y     |

| [DBNetpp](https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/dbnetpp)     | text-detection   |      N      |      Y      |    Y     |  ?   |   ?   |    Y     |
| [PSENet](https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/psenet)       | text-detection   |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |
| [PANet](https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/panet)         | text-detection   |      Y      |      Y      |    Y     |  Y   |   N   |    Y     |
| [TextSnake](https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/textsnake) | text-detection   |      Y      |      Y      |    Y     |  ?   |   ?   |    ?     |

| [MaskRCNN](https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/maskrcnn)   | text-detection   |      Y      |      Y      |    Y     |  ?   |   ?   |    ?     |
| [CRNN](https://github.com/open-mmlab/mmocr/blob/main/configs/textrecog/crnn)         | text-recognition |      Y      |      Y      |    Y     |  Y   |   Y   |    N     |
| [SAR](https://github.com/open-mmlab/mmocr/blob/main/configs/textrecog/sar)           | text-recognition |      N      |      Y      |    Y     |  N   |   N   |    N     |
| [SATRN](https://github.com/open-mmlab/mmocr/blob/main/configs/textrecog/satrn)       | text-recognition |      Y      |      Y      |    Y     |  N   |   N   |    N     |

| [ABINet](https://github.com/open-mmlab/mmocr/blob/main/configs/textrecog/abinet)     | text-recognition |      Y      |      Y      |    Y     |  ?   |   ?   |    ?     |

- ABINet for TensorRT require pytorch1.10+ and TensorRT 8.4+.

- SAR uses `valid_ratio` inside network inference, which causes performance drops. When the `valid_ratio`s between
  testing image and the image for conversion are quite different, the gap would be enlarged.

- For TensorRT backend, users have to choose the right config. For example, CRNN only accepts 1 channel input. Here is a recommendation table:

  | Model    | Config                                                     |
  | :------- | :--------------------------------------------------------- |

| Model    | Config                                                     |
  | MaskRCNN | text-detection_mrcnn_tensorrt_dynamic-320x320-2240x2240.py |
  | CRNN     | text-recognition_tensorrt_dynamic-1x32x32-1x32x640.py      |
  | SATRN    | text-recognition_tensorrt_dynamic-32x32-32x640.py          |
  | SAR      | text-recognition_tensorrt_dynamic-48x64-48x640.py          |
  | ABINet   | text-recognition_tensorrt_static-32x128.py                 |

# FCN

> [Fully Convolutional Networks for Semantic Segmentation](https://arxiv.org/abs/1411.4038)

<a href="https://github.com/BVLC/caffe/wiki/Model-Zoo#fcn">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/decode_heads/fcn_head.py#L11">Code Snippet</a>

Convolutional networks are powerful visual models that yield hierarchies of features. We show that convolutional networks by themselves, trained end-to-end, pixels-to-pixels, exceed the state-of-the-art in semantic segmentation. Our key insight is to build "fully convolutional" networks that take input of arbitrary size and produce correspondingly-sized output with efficient inference and learning. We define and detail the space of fully convolutional networks, explain their application to spatially dense prediction tasks, and draw connections to prior models. We adapt contemporary classification networks (AlexNet, the VGG net, and GoogLeNet) into fully convolutional networks and transfer their learned representations by fine-tuning to the segmentation task. We then define a novel architecture that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations. Our fully convolutional network achieves state-of-the-art segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012), NYUDv2, and SIFT Flow, while inference takes one third of a second for a typical image.

<img src="https://user-images.githubusercontent.com/24582831/142901525-fd0d2bf4-9a47-4143-85f5-3cee8849eaa4.png" width="70%"/>
</div>

| Method     | Backbone   | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device   |  mIoU | mIoU(ms+flip) | config                                                                                                                        | download                                                                                                                                                                                                                                                                                                                                                   |

| FCN        | R-50-D8    | 512x1024  |   40000 | 5.7      | 4.17           | V100     | 72.25 |         73.36 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r50-d8_4xb2-40k_cityscapes-512x1024.py)       | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50-d8_512x1024_40k_cityscapes/fcn_r50-d8_512x1024_40k_cityscapes_20200604_192608-efe53f0d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50-d8_512x1024_40k_cityscapes/fcn_r50-d8_512x1024_40k_cityscapes_20200604_192608.log.json)                         |

| FCN        | R-101-D8   | 512x1024  |   40000 | 9.2      | 2.66           | V100     | 75.45 |         76.58 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r101-d8_4xb2-40k_cityscapes-512x1024.py)      | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_512x1024_40k_cityscapes/fcn_r101-d8_512x1024_40k_cityscapes_20200604_181852-a883d3a1.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_512x1024_40k_cityscapes/fcn_r101-d8_512x1024_40k_cityscapes_20200604_181852.log.json)                     |

| FCN        | R-50-D8    | 769x769   |   40000 | 6.5      | 1.80           | V100     | 71.47 |         72.54 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r50-d8_4xb2-40k_cityscapes-769x769.py)        | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50-d8_769x769_40k_cityscapes/fcn_r50-d8_769x769_40k_cityscapes_20200606_113104-977b5d02.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50-d8_769x769_40k_cityscapes/fcn_r50-d8_769x769_40k_cityscapes_20200606_113104.log.json)                             |

| FCN        | R-101-D8   | 769x769   |   40000 | 10.4     | 1.19           | V100     | 73.93 |         75.14 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r101-d8_4xb2-40k_cityscapes-769x769.py)       | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_769x769_40k_cityscapes/fcn_r101-d8_769x769_40k_cityscapes_20200606_113208-7d4ab69c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_769x769_40k_cityscapes/fcn_r101-d8_769x769_40k_cityscapes_20200606_113208.log.json)                         |

| FCN        | R-18-D8    | 512x1024  |   80000 | 1.7      | 14.65          | V100     | 71.11 |         72.91 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r18-d8_4xb2-80k_cityscapes-512x1024.py)       | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r18-d8_512x1024_80k_cityscapes/fcn_r18-d8_512x1024_80k_cityscapes_20201225_021327-6c50f8b4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r18-d8_512x1024_80k_cityscapes/fcn_r18-d8_512x1024_80k_cityscapes-20201225_021327.log.json)                         |

| FCN        | R-50-D8    | 512x1024  |   80000 | -        |                | V100     | 73.61 |         74.24 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r50-d8_4xb2-80k_cityscapes-512x1024.py)       | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50-d8_512x1024_80k_cityscapes/fcn_r50-d8_512x1024_80k_cityscapes_20200606_113019-03aa804d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50-d8_512x1024_80k_cityscapes/fcn_r50-d8_512x1024_80k_cityscapes_20200606_113019.log.json)                         |

| FCN        | R-101-D8   | 512x1024  |   80000 | -        | -              | V100     | 75.13 |         75.94 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r101-d8_4xb2-80k_cityscapes-512x1024.py)      | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_512x1024_80k_cityscapes/fcn_r101-d8_512x1024_80k_cityscapes_20200606_113038-3fb937eb.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_512x1024_80k_cityscapes/fcn_r101-d8_512x1024_80k_cityscapes_20200606_113038.log.json)                     |

| FCN (FP16) | R-101-D8   | 512x1024  |   80000 | 5.37     | 8.64           | V100     | 76.80 |             - | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r101-d8_4xb2-amp-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_fp16_512x1024_80k_cityscapes/fcn_r101-d8_fp16_512x1024_80k_cityscapes_20200717_230921-fb13e883.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_fp16_512x1024_80k_cityscapes/fcn_r101-d8_fp16_512x1024_80k_cityscapes_20200717_230921.log.json) |

| FCN        | R-18-D8    | 769x769   |   80000 | 1.9      | 6.40           | V100     | 70.80 |         73.16 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r18-d8_4xb2-80k_cityscapes-769x769.py)        | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r18-d8_769x769_80k_cityscapes/fcn_r18-d8_769x769_80k_cityscapes_20201225_021451-9739d1b8.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r18-d8_769x769_80k_cityscapes/fcn_r18-d8_769x769_80k_cityscapes-20201225_021451.log.json)                             |

| FCN        | R-50-D8    | 769x769   |   80000 | -        | -              | V100     | 72.64 |         73.32 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r50-d8_4xb2-80k_cityscapes-769x769.py)        | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50-d8_769x769_80k_cityscapes/fcn_r50-d8_769x769_80k_cityscapes_20200606_195749-f5caeabc.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50-d8_769x769_80k_cityscapes/fcn_r50-d8_769x769_80k_cityscapes_20200606_195749.log.json)                             |

| FCN        | R-101-D8   | 769x769   |   80000 | -        | -              | V100     | 75.52 |         76.61 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r101-d8_4xb2-80k_cityscapes-769x769.py)       | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_769x769_80k_cityscapes/fcn_r101-d8_769x769_80k_cityscapes_20200606_214354-45cbac68.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_769x769_80k_cityscapes/fcn_r101-d8_769x769_80k_cityscapes_20200606_214354.log.json)                         |

| FCN        | R-18b-D8   | 512x1024  |   80000 | 1.6      | 16.74          | V100     | 70.24 |         72.77 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r18b-d8_4xb2-80k_cityscapes-512x1024.py)      | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r18b-d8_512x1024_80k_cityscapes/fcn_r18b-d8_512x1024_80k_cityscapes_20201225_230143-92c0f445.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r18b-d8_512x1024_80k_cityscapes/fcn_r18b-d8_512x1024_80k_cityscapes-20201225_230143.log.json)                     |

| FCN        | R-50b-D8   | 512x1024  |   80000 | 5.6      | 4.20           | V100     | 75.65 |         77.59 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r50b-d8_4xb2-80k_cityscapes-512x1024.py)      | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50b-d8_512x1024_80k_cityscapes/fcn_r50b-d8_512x1024_80k_cityscapes_20201225_094221-82957416.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50b-d8_512x1024_80k_cityscapes/fcn_r50b-d8_512x1024_80k_cityscapes-20201225_094221.log.json)                     |

| FCN        | R-101b-D8  | 512x1024  |   80000 | 9.1      | 2.73           | V100     | 77.37 |         78.77 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r101b-d8_4xb2-80k_cityscapes-512x1024.py)     | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101b-d8_512x1024_80k_cityscapes/fcn_r101b-d8_512x1024_80k_cityscapes_20201226_160213-4543858f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101b-d8_512x1024_80k_cityscapes/fcn_r101b-d8_512x1024_80k_cityscapes-20201226_160213.log.json)                 |

| FCN        | R-18b-D8   | 769x769   |   80000 | 1.7      | 6.70           | V100     | 69.66 |         72.07 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r18b-d8_4xb2-80k_cityscapes-769x769.py)       | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r18b-d8_769x769_80k_cityscapes/fcn_r18b-d8_769x769_80k_cityscapes_20201226_004430-32d504e5.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r18b-d8_769x769_80k_cityscapes/fcn_r18b-d8_769x769_80k_cityscapes-20201226_004430.log.json)                         |

| FCN        | R-50b-D8   | 769x769   |   80000 | 6.3      | 1.82           | V100     | 73.83 |         76.60 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r50b-d8_4xb2-80k_cityscapes-769x769.py)       | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50b-d8_769x769_80k_cityscapes/fcn_r50b-d8_769x769_80k_cityscapes_20201225_094223-94552d38.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50b-d8_769x769_80k_cityscapes/fcn_r50b-d8_769x769_80k_cityscapes-20201225_094223.log.json)                         |

| FCN        | R-101b-D8  | 769x769   |   80000 | 10.3     | 1.15           | V100     | 77.02 |         78.67 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r101b-d8_4xb2-80k_cityscapes-769x769.py)      | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101b-d8_769x769_80k_cityscapes/fcn_r101b-d8_769x769_80k_cityscapes_20201226_170012-82be37e2.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101b-d8_769x769_80k_cityscapes/fcn_r101b-d8_769x769_80k_cityscapes-20201226_170012.log.json)                     |

| FCN (D6)   | R-50-D16   | 512x1024  |   40000 | 3.4      | 10.22          | TITAN Xp | 77.06 |         78.85 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn-d6_r50-d16_4xb2-40k_cityscapes-512x1024.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r50-d16_512x1024_40k_cityscapes/fcn_d6_r50-d16_512x1024_40k_cityscapes_20210305_130133-98d5d1bc.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r50-d16_512x1024_40k_cityscapes/fcn_d6_r50-d16_512x1024_40k_cityscapes-20210305_130133.log.json)         |

| FCN (D6)   | R-50-D16   | 512x1024  |   80000 | -        | 10.35          | TITAN Xp | 77.27 |         78.88 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn-d6_r50-d16_4xb2-80k_cityscapes-512x1024.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r50-d16_512x1024_80k_cityscapes/fcn_d6_r50-d16_512x1024_80k_cityscapes_20210306_115604-133c292f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r50-d16_512x1024_80k_cityscapes/fcn_d6_r50-d16_512x1024_80k_cityscapes-20210306_115604.log.json)         |

| FCN (D6)   | R-50-D16   | 769x769   |   40000 | 3.7      | 4.17           | TITAN Xp | 76.82 |         78.22 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn-d6_r50-d16_4xb2-40k_cityscapes-769x769.py)    | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r50-d16_769x769_40k_cityscapes/fcn_d6_r50-d16_769x769_40k_cityscapes_20210305_185744-1aab18ed.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r50-d16_769x769_40k_cityscapes/fcn_d6_r50-d16_769x769_40k_cityscapes-20210305_185744.log.json)             |

| FCN (D6)   | R-50-D16   | 769x769   |   80000 | -        | 4.15           | TITAN Xp | 77.04 |         78.40 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn-d6_r50-d16_4xb2-80k_cityscapes-769x769.py)    | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r50-d16_769x769_80k_cityscapes/fcn_d6_r50-d16_769x769_80k_cityscapes_20210305_200413-109d88eb.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r50-d16_769x769_80k_cityscapes/fcn_d6_r50-d16_769x769_80k_cityscapes-20210305_200413.log.json)             |

| FCN (D6)   | R-101-D16  | 512x1024  |   40000 | 4.5      | 8.04           | TITAN Xp | 77.36 |         79.18 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn-d6_r101-d16_4xb2-40k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r101-d16_512x1024_40k_cityscapes/fcn_d6_r101-d16_512x1024_40k_cityscapes_20210305_130337-9cf2b450.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r101-d16_512x1024_40k_cityscapes/fcn_d6_r101-d16_512x1024_40k_cityscapes-20210305_130337.log.json)     |

| FCN (D6)   | R-101-D16  | 512x1024  |   80000 | -        | 8.26           | TITAN Xp | 78.46 |         80.42 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn-d6_r101-d16_4xb2-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r101-d16_512x1024_80k_cityscapes/fcn_d6_r101-d16_512x1024_80k_cityscapes_20210308_102747-cb336445.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r101-d16_512x1024_80k_cityscapes/fcn_d6_r101-d16_512x1024_80k_cityscapes-20210308_102747.log.json)     |

| FCN (D6)   | R-101-D16  | 769x769   |   40000 | 5.0      | 3.12           | TITAN Xp | 77.28 |         78.95 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn-d6_r101-d16_4xb2-40k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r101-d16_769x769_40k_cityscapes/fcn_d6_r101-d16_769x769_40k_cityscapes_20210308_102453-60b114e9.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r101-d16_769x769_40k_cityscapes/fcn_d6_r101-d16_769x769_40k_cityscapes-20210308_102453.log.json)         |

| FCN (D6)   | R-101-D16  | 769x769   |   80000 | -        | 3.21           | TITAN Xp | 78.06 |         79.58 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn-d6_r101-d16_4xb2-80k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r101-d16_769x769_80k_cityscapes/fcn_d6_r101-d16_769x769_80k_cityscapes_20210306_120016-e33adc4f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r101-d16_769x769_80k_cityscapes/fcn_d6_r101-d16_769x769_80k_cityscapes-20210306_120016.log.json)         |

| FCN (D6)   | R-50b-D16  | 512x1024  |   80000 | 3.2      | 10.16          | TITAN Xp | 76.99 |         79.03 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn-d6_r50b-d16_4xb2-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r50b-d16_512x1024_80k_cityscapes/fcn_d6_r50b-d16_512x1024_80k_cityscapes_20210311_125550-6a0b62e9.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r50b_d16_512x1024_80k_cityscapes/fcn_d6_r50b_d16_512x1024_80k_cityscapes-20210311_125550.log.json)     |

| FCN (D6)   | R-50b-D16  | 769x769   |   80000 | 3.6      | 4.17           | TITAN Xp | 76.86 |         78.52 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn-d6_r50b-d16_4xb2-80k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r50b-d16_769x769_80k_cityscapes/fcn_d6_r50b-d16_769x769_80k_cityscapes_20210311_131012-d665f231.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r50b_d16_769x769_80k_cityscapes/fcn_d6_r50b_d16_769x769_80k_cityscapes-20210311_131012.log.json)         |

| FCN (D6)   | R-101b-D16 | 512x1024  |   80000 | 4.3      | 8.46           | TITAN Xp | 77.72 |         79.53 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn-d6_r101b-d16_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r101b-d16_512x1024_80k_cityscapes/fcn_d6_r101b-d16_512x1024_80k_cityscapes_20210311_144305-3f2eb5b4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r101b_d16_512x1024_80k_cityscapes/fcn_d6_r101b_d16_512x1024_80k_cityscapes-20210311_144305.log.json) |

| FCN (D6)   | R-101b-D16 | 769x769   |   80000 | 4.8      | 3.32           | TITAN Xp | 77.34 |         78.91 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn-d6_r101b-d16_4xb2-80k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r101b-d16_769x769_80k_cityscapes/fcn_d6_r101b-d16_769x769_80k_cityscapes_20210311_154527-c4d8bfbc.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_d6_r101b_d16_769x769_80k_cityscapes/fcn_d6_r101b_d16_769x769_80k_cityscapes-20210311_154527.log.json)     |

### ADE20K

| FCN    | R-50-D8  | 512x512   |   80000 | 8.5      | 23.49          | V100   | 35.94 |         37.94 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r50-d8_4xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50-d8_512x512_80k_ade20k/fcn_r50-d8_512x512_80k_ade20k_20200614_144016-f8ac5082.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50-d8_512x512_80k_ade20k/fcn_r50-d8_512x512_80k_ade20k_20200614_144016.log.json)         |

| FCN    | R-101-D8 | 512x512   |   80000 | 12       | 14.78          | V100   | 39.61 |         40.83 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r101-d8_4xb4-80k_ade20k-512x512.pyy) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_512x512_80k_ade20k/fcn_r101-d8_512x512_80k_ade20k_20200615_014143-bc1809f7.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_512x512_80k_ade20k/fcn_r101-d8_512x512_80k_ade20k_20200615_014143.log.json)     |

| FCN    | R-50-D8  | 512x512   |  160000 | -        | -              | V100   | 36.10 |         38.08 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r50-d8_4xb4-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50-d8_512x512_160k_ade20k/fcn_r50-d8_512x512_160k_ade20k_20200615_100713-4edbc3b4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50-d8_512x512_160k_ade20k/fcn_r50-d8_512x512_160k_ade20k_20200615_100713.log.json)     |

| FCN    | R-101-D8 | 512x512   |  160000 | -        | -              | V100   | 39.91 |         41.40 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r101-d8_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_512x512_160k_ade20k/fcn_r101-d8_512x512_160k_ade20k_20200615_105816-fd192bd5.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_512x512_160k_ade20k/fcn_r101-d8_512x512_160k_ade20k_20200615_105816.log.json) |

| FCN    | R-50-D8  | 512x512   |   20000 | 5.7      | 23.28          | V100   | 67.08 |         69.94 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r50-d8_4xb4-20k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50-d8_512x512_20k_voc12aug/fcn_r50-d8_512x512_20k_voc12aug_20200617_010715-52dc5306.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50-d8_512x512_20k_voc12aug/fcn_r50-d8_512x512_20k_voc12aug_20200617_010715.log.json)     |

| FCN    | R-101-D8 | 512x512   |   20000 | 9.2      | 14.81          | V100   | 71.16 |         73.57 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r101-d8_4xb4-20k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_512x512_20k_voc12aug/fcn_r101-d8_512x512_20k_voc12aug_20200617_010842-0bb4e798.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_512x512_20k_voc12aug/fcn_r101-d8_512x512_20k_voc12aug_20200617_010842.log.json) |

| FCN    | R-50-D8  | 512x512   |   40000 | -        | -              | V100   | 66.97 |         69.04 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r50-d8_4xb4-40k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50-d8_512x512_40k_voc12aug/fcn_r50-d8_512x512_40k_voc12aug_20200613_161222-5e2dbf40.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r50-d8_512x512_40k_voc12aug/fcn_r50-d8_512x512_40k_voc12aug_20200613_161222.log.json)     |

| FCN    | R-101-D8 | 512x512   |   40000 | -        | -              | V100   | 69.91 |         72.38 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r101-d8_4xb4-40k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_512x512_40k_voc12aug/fcn_r101-d8_512x512_40k_voc12aug_20200613_161240-4c8bcefd.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_512x512_40k_voc12aug/fcn_r101-d8_512x512_40k_voc12aug_20200613_161240.log.json) |

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                      | download                                                                                                                                                                                                                                                                                                                                           |

| FCN    | R-101-D8 | 480x480   |   40000 | -        | 9.93           | V100   | 44.43 |         45.63 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r101-d8_4xb4-40k_pascal-context-480x480.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_480x480_40k_pascal_context/fcn_r101-d8_480x480_40k_pascal_context_20210421_154757-b5e97937.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_480x480_40k_pascal_context/fcn_r101-d8_480x480_40k_pascal_context-20210421_154757.log.json) |

| FCN    | R-101-D8 | 480x480   |   80000 | -        | -              | V100   | 44.13 |         45.26 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r101-d8_4xb4-80k_pascal-context-480x480.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_480x480_80k_pascal_context/fcn_r101-d8_480x480_80k_pascal_context_20210421_163310-4711813f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_480x480_80k_pascal_context/fcn_r101-d8_480x480_80k_pascal_context-20210421_163310.log.json) |

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                         | download                                                                                                                                                                                                                                                                                                                                                       |

| FCN    | R-101-D8 | 480x480   |   40000 | -        | -              | V100   | 48.42 |          50.4 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r101-d8_4xb4-40k_pascal-context-59-480x480.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_480x480_40k_pascal_context_59/fcn_r101-d8_480x480_40k_pascal_context_59_20210415_230724-8cf83682.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_480x480_40k_pascal_context_59/fcn_r101-d8_480x480_40k_pascal_context_59-20210415_230724.log.json) |

| FCN    | R-101-D8 | 480x480   |   80000 | -        | -              | V100   | 49.35 |         51.38 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn_r101-d8_4xb4-80k_pascal-context-59-480x480.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_480x480_80k_pascal_context_59/fcn_r101-d8_480x480_80k_pascal_context_59_20210416_110804-9a6f2c94.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/fcn/fcn_r101-d8_480x480_80k_pascal_context_59/fcn_r101-d8_480x480_80k_pascal_context_59-20210416_110804.log.json) |

Note:

- `FCN D6` means dilation rate of convolution operator in FCN is 6.

```bibtex
@article{shelhamer2017fully,
  title={Fully convolutional networks for semantic segmentation},
  author={Shelhamer, Evan and Long, Jonathan and Darrell, Trevor},

author={Shelhamer, Evan and Long, Jonathan and Darrell, Trevor},
  volume={39},
  pages={640--651},
  publisher={IEEE Trans Pattern Anal Mach Intell}
}
```

# Guided Diffusion (NeurIPS'2021)

> [Diffusion Models Beat GANs on Image Synthesis](https://papers.nips.cc/paper/2021/file/49ad23d1ec9fa4bd8d77d02681df5cfa-Paper.pdf)

> **Task**: Image Generation

We show that diffusion models can achieve image sample quality superior to the current state-of-the-art generative models. We achieve this on unconditional image synthesis by finding a better architecture through a series of ablations. For conditional image synthesis, we further improve sample quality with classifier guidance: a simple, compute-efficient method for trading off diversity for fidelity using gradients from a classifier. We achieve an FID of 2.97 on ImageNet 128x128, 4.59 on ImageNet 256x256, and 7.72 on ImageNet 512x512, and we match BigGAN-deep even with as few as 25 forward passes per sample, all while maintaining better coverage of the distribution. Finally, we find that classifier guidance combines well with upsampling diffusion models, further improving FID to 3.94 on ImageNet 256x256 and 3.85 on ImageNet 512x512.

 <img src="https://user-images.githubusercontent.com/22982797/204706276-e340c545-3ec6-48bf-be21-58ed44e8a4df.jpg" width="400"/>
</div >

  <b>hamster, classifier-guidance samplings with CGS=1.0</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/22982797/212831070-470034df-0a9f-4a75-8ab8-97d39bc1806c.png" width="400"/>
</div>

**ImageNet**

|                               Model                                |     Dataset      | Scheduler | Steps | CGS | Time Consuming(A100) | FID-Full-50K |                               Download                                |
| :----------------------------------------------------------------: | :--------------: | :-------: | :---: | :-: | :------------------: | :----------: | :-------------------------------------------------------------------: |

| [adm_ddim250_8xb32_imagenet-64x64](./adm_ddim250_8xb32_imagenet-64x64.py) |  ImageNet 64x64  |   DDIM    |  250  |  -  |          1h          |    3.2284    | [ckpt](https://download.openmmlab.com/mmediting/guided_diffusion/adm-u-cvt-rgb_8xb32_imagenet-64x64-7ff0080b.pth) |
| [adm-g_ddim25_8xb32_imagenet-64x64](configs/guided_diffusion/adm-g_ddim25_8xb32_imagenet-64x64.py) |  ImageNet 64x64  |   DDIM    |  25   | 1.0 |          2h          |    3.7566    | [ckpt](https://download.openmmlab.com/mmediting/guided_diffusion/adm-g_8xb32_imagenet-64x64-2c0fbeda.pth) |
| [adm_ddim250_8xb32_imagenet-256x256](configs/guided_diffusion/adm_ddim250_8xb32_imagenet-256x256.py) | ImageNet 256x256 |   DDIM    |  250  |  -  |          -           |      -       | [ckpt](https://download.openmmlab.com/mmediting/guided_diffusion/adm_8xb32_imagenet-256x256-f94735fe.pth) |

| [adm-g_ddim25_8xb32_imagenet-256x256](configs/guided_diffusion/adm-g_ddim25_8xb32_imagenet-256x256.py) | ImageNet 256x256 |   DDIM    |  25   | 1.0 |          -           |      -       | [ckpt](https://download.openmmlab.com/mmediting/guided_diffusion/adm-g_8xb32_imagenet-256x256-aec3fc9f.pth) |
| [adm_ddim250_8xb32_imagenet-512x512](configs/guided_diffusion/adm_ddim250_8xb32_imagenet-512x512.py) | ImageNet 512x512 |   DDIM    |  250  |  -  |          -           |      -       | [ckpt](https://download.openmmlab.com/mmediting/guided_diffusion/adm-u_8xb32_imagenet-512x512-60b381cb.pth) |
| [adm-g_ddim25_8xb32_imagenet-512x512](configs/guided_diffusion/adm-g_ddim25_8xb32_imagenet-512x512.py) | ImageNet 512x512 |   DDIM    |  25   | 1.0 |          -           |      -       | [ckpt](https://download.openmmlab.com/mmediting/guided_diffusion/adm-g_8xb32_imagenet-512x512-23cf0b58.pth) |

**infer**

<details>
<summary>Infer Instructions</summary>

**infer**

<details>

You can run adm as follows:

```python
from mmengine import Config, MODELS
from torchvision.utils import save_image

init_default_scope('mmagic')

# sampling without classifier guidance, CGS=1.0
config = 'configs/guided_diffusion/adm-g_ddim25_8xb32_imagenet-64x64.py'
ckpt_path = 'https://download.openmmlab.com/mmediting/guided_diffusion/adm-g_8xb32_imagenet-64x64-2c0fbeda.pth'  # noqa

model_cfg = Config.fromfile(config).model
model_cfg.pretrained_cfgs = dict(unet=dict(ckpt_path=ckpt_path, prefix='unet'),
                                 classifier=dict(ckpt_path=ckpt_path, prefix='classifier'))
model = MODELS.build(model_cfg).cuda().eval()

samples = model.infer(
            init_image=None,
            batch_size=4,
            num_inference_steps=25,
            labels=333,
            classifier_scale=1.0,
            show_progress=True)['samples']

init_image=None,

# sampling without classifier guidance
config = 'configs/guided_diffusion/adm_ddim250_8xb32_imagenet-64x64.py'
ckpt_path = 'https://download.openmmlab.com/mmediting/guided_diffusion/adm-u-cvt-rgb_8xb32_imagenet-64x64-7ff0080b.pth'  # noqa

model_cfg.pretrained_cfgs = dict(unet=dict(ckpt_path=ckpt_path, prefix='unet'))

            num_inference_steps=250,
            labels=None,
            classifier_scale=0.0,
```

**Test**

<details>

```shell
# cpu test

```

**Test**

<details>

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/guided_diffusion/adm-u_ddim250_8xb32_imagenet-64x64.py https://download.openmmlab.com/mmgen/guided_diffusion/adm-u-cvt-rgb_8xb32_imagenet-64x64-7ff0080b.pth

python tools/test.py configs/guided_diffusion/adm-u_ddim250_8xb32_imagenet-64x64.py https://download.openmmlab.com/mmgen/guided_diffusion/adm-u-cvt-rgb_8xb32_imagenet-64x64-7ff0080b.pth

./tools/dist_test.sh configs/guided_diffusion/adm-u_ddim250_8xb32_imagenet-64x64.py https://download.openmmlab.com/mmgen/guided_diffusion/adm-u-cvt-rgb_8xb32_imagenet-64x64-7ff0080b.pth 8
```

```bibtex

```

```bibtex
@article{PrafullaDhariwal2021DiffusionMB,
  title={Diffusion Models Beat GANs on Image Synthesis},
  author={Prafulla Dhariwal and Alex Nichol},
  journal={arXiv: Learning},
}
```

# LiDAR-Based 3D Detection

LiDAR-based 3D detection is one of the most basic tasks supported in MMDetection3D.
It expects the given model to take any number of points with features collected by LiDAR as input, and predict the 3D bounding boxes and category labels for each object of interest.
Next, taking PointPillars on the KITTI dataset as an example, we will show how to prepare data, train and test a model on a standard 3D detection benchmark, and how to visualize and validate the results.

To begin with, we need to download the raw data and reorganize the data in a standard way presented in the [doc for data preparation](https://mmdetection3d.readthedocs.io/en/dev-1.x/user_guides/dataset_prepare.html).
Note that for KITTI, we need extra `.txt` files for data splits.

Due to different ways of organizing the raw data in different datasets, we typically need to collect the useful data information with a `.pkl` file.

So after getting all the raw data ready, we need to run the scripts provided in the `create_data.py` for different datasets to generate data infos.
For example, for KITTI we need to run:

```shell
```

```
├── tools
├── data
│   ├── kitti

```

Then let us train a model with provided configs for PointPillars.
You can basically follow the examples provided in this [tutorial](https://mmdetection3d.readthedocs.io/en/dev-1.x/user_guides/train_test.html) when training with different GPU settings.
Suppose we use 8 GPUs on a single machine with distributed training:

```shell
```

Note that `8xb6` in the config name refers to the training is completed with 8 GPUs and 6 samples on each GPU.

A basic rule can be referred to [here](https://arxiv.org/abs/1706.02677). We have supported `--auto-scale-lr` to
enable automatically scaling LR.

For KITTI, the model will be evaluated with mean average precision (mAP) with Intersection over Union (IoU) thresholds 0.5/0.7 for 3 categories respectively.

```

```

```shell
```

you need to specify the `submission_prefix` for corresponding evaluator,
e.g., add `test_evaluator = dict(type='KittiMetric', ann_file=data_root + 'kitti_infos_test.pkl', format_only=True, pklfile_prefix='results/kitti-3class/kitti_results', submission_prefix='results/kitti-3class/kitti_results')` in the configuration then you can get the results file.

Please guarantee the `data_prefix` and `ann_file` in [info for testing](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/datasets/kitti-3d-3class.py#L117) in the config corresponds to the test set instead of validation set.
After generating the results, you can basically compress the folder and upload to the KITTI evaluation server.

MMDetection3D also provides versatile tools for visualization such that we can have an intuitive feeling of the detection results predicted by our trained models.
You can either set the `--show` option to visualize the detection results online during evaluation,
or using `tools/misc/visualize_results.py` for offline visualization.
Please refer more details in the [doc for visualization](https://mmdetection3d.readthedocs.io/en/dev-1.x/user_guides/visualization.html).

# Objects are Different: Flexible Monocular 3D Object Detection

> [Objects are Different: Flexible Monocular 3D Object Detection](https://arxiv.org/abs/2104.02323)

The precise localization of 3D objects from a single image without depth information is a highly challenging problem. Most existing methods adopt the same approach for all objects regardless of their diverse distributions, leading to limited performance for truncated objects. In this paper, we propose a flexible framework for monocular 3D object detection which explicitly decouples the truncated objects and adaptively combines multiple approaches for object depth estimation. Specifically, we decouple the edge of the feature map for predicting long-tail truncated objects so that the optimization of normal objects is not influenced. Furthermore, we formulate the object depth estimation as an uncertainty-guided ensemble of directly regressed object depth and solved depths from different groups of keypoints. Experiments demonstrate that our method outperforms the state-of-the-art method by relatively 27% for the moderate level and 30% for the hard level in the test set of KITTI benchmark while maintaining real-time efficiency.

<img src="https://user-images.githubusercontent.com/36950400/153138824-d54a7a47-773f-42f9-8a51-b0a71078593e.png" width="800"/>
</div>

We implement MonoFlex and provide the results and checkpoints on KITTI dataset.

### KITTI

|                                Backbone                                 | Lr schd | Mem (GB) | Inf time (fps) |  mAP  |                                                                                                                                                               Download                                                                                                                                                               |

| :---------------------------------------------------------------------: | :-----: | :------: | :------------: | :---: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| [DLA34](./monoflex_dla34_pytorch_dlaneck_gn-all_2x4_6x_kitti-mono3d.py) |   6x    |   9.64   |                | 21.86 | [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/monoflex/monoflex_dla34_pytorch_dlaneck_gn-all_2x4_6x_kitti-mono3d_20211228_027553-d46d9bb0.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/monoflex/monoflex_dla34_pytorch_dlaneck_gn-all_2x4_6x_kitti-mono3d_20211228_027553.log.json) |

Detailed performance on KITTI 3D detection (3D/BEV) is as follows, evaluated by AP11 and AP40 metric:

| Car (AP11) | 28.02 / 36.11 | 21.86 / 29.46 | 19.01 / 24.83 |
| Car (AP40) | 23.22 / 32.74 | 17.18 / 24.02 | 15.13 / 20.67 |

Note: mAP represents Car moderate 3D strict AP11 / AP40 results. Because of the limited data for pedestrians and cyclists, the detection performance for these two classes is usually unstable. Therefore, we only list car detection results here. In addition, the AP11 result may fluctuate in a larger range (~1 AP), so AP40 is a more recommended metric for reference due to its much better stability.

```latex
@InProceedings{MonoFlex,
    author    = {Zhang, Yunpeng and Lu, Jiwen and Zhou, Jie},

```latex
    title     = {Objects Are Different: Flexible Monocular 3D Object Detection},
    year      = {2021},
    pages     = {3289-3298}
}
```

# DeepLabV3+

> [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://arxiv.org/abs/1802.02611)

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/decode_heads/sep_aspp_head.py#L30">Code Snippet</a>

Spatial pyramid pooling module or encode-decoder structure are used in deep neural networks for semantic segmentation task. The former networks are able to encode multi-scale contextual information by probing the incoming features with filters or pooling operations at multiple rates and multiple effective fields-of-view, while the latter networks can capture sharper object boundaries by gradually recovering the spatial information. In this work, we propose to combine the advantages from both methods. Specifically, our proposed model, DeepLabv3+, extends DeepLabv3 by adding a simple yet effective decoder module to refine the segmentation results especially along object boundaries. We further explore the Xception model and apply the depthwise separable convolution to both Atrous Spatial Pyramid Pooling and decoder modules, resulting in a faster and stronger encoder-decoder network. We demonstrate the effectiveness of the proposed model on PASCAL VOC 2012 and Cityscapes datasets, achieving the test set performance of 89.0% and 82.1% without any post-processing. Our paper is accompanied with a publicly available reference implementation of the proposed models in Tensorflow at [this https URL](https://github.com/tensorflow/models/tree/master/research/deeplab).

<img src="https://user-images.githubusercontent.com/24582831/142900680-3e2c3098-8341-4760-bbfd-b1d7d29968ea.png" width="70%"/>
</div>

| Method            | Backbone        | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                                      | download                                                                                                                                                                                                                                                                                                                                                                                                                       |

| DeepLabV3+        | R-50-D8         | 512x1024  |   40000 | 7.5      | 3.94           | V100   | 79.61 |         81.01 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r50-d8_4xb2-40k_cityscapes-512x1024.py)                 | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_512x1024_40k_cityscapes/deeplabv3plus_r50-d8_512x1024_40k_cityscapes_20200605_094610-d222ffcd.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_512x1024_40k_cityscapes/deeplabv3plus_r50-d8_512x1024_40k_cityscapes_20200605_094610.log.json)                                 |

| DeepLabV3+        | R-101-D8        | 512x1024  |   40000 | 11       | 2.60           | V100   | 80.21 |         81.82 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r101-d8_4xb2-40k_cityscapes-512x1024.py)                | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_512x1024_40k_cityscapes/deeplabv3plus_r101-d8_512x1024_40k_cityscapes_20200605_094614-3769eecf.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_512x1024_40k_cityscapes/deeplabv3plus_r101-d8_512x1024_40k_cityscapes_20200605_094614.log.json)                             |

| DeepLabV3+        | R-50-D8         | 769x769   |   40000 | 8.5      | 1.72           | V100   | 78.97 |         80.46 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r50-d8_4xb2-40k_cityscapes-769x769.py)                  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_769x769_40k_cityscapes/deeplabv3plus_r50-d8_769x769_40k_cityscapes_20200606_114143-1dcb0e3c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_769x769_40k_cityscapes/deeplabv3plus_r50-d8_769x769_40k_cityscapes_20200606_114143.log.json)                                     |

| DeepLabV3+        | R-101-D8        | 769x769   |   40000 | 12.5     | 1.15           | V100   | 79.46 |         80.50 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r101-d8_4xb2-40k_cityscapes-769x769.py)                 | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_769x769_40k_cityscapes/deeplabv3plus_r101-d8_769x769_40k_cityscapes_20200606_114304-ff414b9e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_769x769_40k_cityscapes/deeplabv3plus_r101-d8_769x769_40k_cityscapes_20200606_114304.log.json)                                 |

| DeepLabV3+        | R-18-D8         | 512x1024  |   80000 | 2.2      | 14.27          | V100   | 76.89 |         78.76 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r18-d8_4xb2-80k_cityscapes-512x1024.py)                 | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r18-d8_512x1024_80k_cityscapes/deeplabv3plus_r18-d8_512x1024_80k_cityscapes_20201226_080942-cff257fe.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r18-d8_512x1024_80k_cityscapes/deeplabv3plus_r18-d8_512x1024_80k_cityscapes-20201226_080942.log.json)                                 |

| DeepLabV3+        | R-50-D8         | 512x1024  |   80000 | -        | -              | V100   | 80.09 |         81.13 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r50-d8_4xb2-80k_cityscapes-512x1024.py)                 | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_512x1024_80k_cityscapes/deeplabv3plus_r50-d8_512x1024_80k_cityscapes_20200606_114049-f9fb496d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_512x1024_80k_cityscapes/deeplabv3plus_r50-d8_512x1024_80k_cityscapes_20200606_114049.log.json)                                 |

| DeepLabV3+        | R-101-D8        | 512x1024  |   80000 | -        | -              | V100   | 80.97 |         82.03 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r101-d8_4xb2-80k_cityscapes-512x1024.py)                | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_512x1024_80k_cityscapes/deeplabv3plus_r101-d8_512x1024_80k_cityscapes_20200606_114143-068fcfe9.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_512x1024_80k_cityscapes/deeplabv3plus_r101-d8_512x1024_80k_cityscapes_20200606_114143.log.json)                             |

| DeepLabV3+ (FP16) | R-101-D8        | 512x1024  |   80000 | 6.35     | 7.87           | V100   | 80.46 |             - | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r101-d8_4xb2-amp-80k_cityscapes-512x1024.py)            | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_fp16_512x1024_80k_cityscapes/deeplabv3plus_r101-d8_fp16_512x1024_80k_cityscapes_20200717_230920-f1104f4b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_fp16_512x1024_80k_cityscapes/deeplabv3plus_r101-d8_fp16_512x1024_80k_cityscapes_20200717_230920.log.json)         |

| DeepLabV3+        | R-18-D8         | 769x769   |   80000 | 2.5      | 5.74           | V100   | 76.26 |         77.91 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r18-d8_4xb2-80k_cityscapes-769x769.py)                  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r18-d8_769x769_80k_cityscapes/deeplabv3plus_r18-d8_769x769_80k_cityscapes_20201226_083346-f326e06a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r18-d8_769x769_80k_cityscapes/deeplabv3plus_r18-d8_769x769_80k_cityscapes-20201226_083346.log.json)                                     |

| DeepLabV3+        | R-50-D8         | 769x769   |   80000 | -        | -              | V100   | 79.83 |         81.48 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r50-d8_4xb2-80k_cityscapes-769x769.py)                  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_769x769_80k_cityscapes/deeplabv3plus_r50-d8_769x769_80k_cityscapes_20200606_210233-0e9dfdc4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_769x769_80k_cityscapes/deeplabv3plus_r50-d8_769x769_80k_cityscapes_20200606_210233.log.json)                                     |

| DeepLabV3+        | R-101-D8        | 769x769   |   80000 | -        | -              | V100   | 80.65 |         81.47 | [config<sup>\[1\]</sup>](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r101-d8_4xb2-80k_cityscapes-769x769.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_769x769_80k_cityscapes/deeplabv3plus_r101-d8_769x769_80k_cityscapes_20220406_154720-dfcc0b68.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_769x769_80k_cityscapes/deeplabv3plus_r101-d8_769x769_80k_cityscapes_20220406_154720.log.json)                                 |

| DeepLabV3+        | R-101-D16-MG124 | 512x1024  |   40000 | 5.8      | 7.48           | V100   | 79.09 |         80.36 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/ddeeplabv3plus_r101-d16-mg124_4xb2-40k_cityscapes-512x1024.py)        | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d16-mg124_512x1024_40k_cityscapes/deeplabv3plus_r101-d16-mg124_512x1024_40k_cityscapes_20200908_005644-cf9ce186.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d16-mg124_512x1024_40k_cityscapes/deeplabv3plus_r101-d16-mg124_512x1024_40k_cityscapes-20200908_005644.log.json) |

| DeepLabV3+        | R-101-D16-MG124 | 512x1024  |   80000 | 9.9      | -              | V100   | 79.90 |         81.33 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r101-d16-mg124_4xb2-80k_cityscapes-512x1024.py)         | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d16-mg124_512x1024_80k_cityscapes/deeplabv3plus_r101-d16-mg124_512x1024_80k_cityscapes_20200908_005644-ee6158e0.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d16-mg124_512x1024_80k_cityscapes/deeplabv3plus_r101-d16-mg124_512x1024_80k_cityscapes-20200908_005644.log.json) |

| DeepLabV3+        | R-18b-D8        | 512x1024  |   80000 | 2.1      | 14.95          | V100   | 75.87 |         77.52 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r18b-d8_4xb2-80k_cityscapes-512x1024.py)                | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r18b-d8_512x1024_80k_cityscapes/deeplabv3plus_r18b-d8_512x1024_80k_cityscapes_20201226_090828-e451abd9.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r18b-d8_512x1024_80k_cityscapes/deeplabv3plus_r18b-d8_512x1024_80k_cityscapes-20201226_090828.log.json)                             |

| DeepLabV3+        | R-50b-D8        | 512x1024  |   80000 | 7.4      | 3.94           | V100   | 80.28 |         81.44 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r50b-d8_4xb2-80k_cityscapes-512x1024.py)                | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50b-d8_512x1024_80k_cityscapes/deeplabv3plus_r50b-d8_512x1024_80k_cityscapes_20201225_213645-a97e4e43.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50b-d8_512x1024_80k_cityscapes/deeplabv3plus_r50b-d8_512x1024_80k_cityscapes-20201225_213645.log.json)                             |

| DeepLabV3+        | R-101b-D8       | 512x1024  |   80000 | 10.9     | 2.60           | V100   | 80.16 |         81.41 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r101b-d8_4xb2-80k_cityscapes-512x1024.py)               | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101b-d8_512x1024_80k_cityscapes/deeplabv3plus_r101b-d8_512x1024_80k_cityscapes_20201226_190843-9c3c93a4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101b-d8_512x1024_80k_cityscapes/deeplabv3plus_r101b-d8_512x1024_80k_cityscapes-20201226_190843.log.json)                         |

| DeepLabV3+        | R-18b-D8        | 769x769   |   80000 | 2.4      | 5.96           | V100   | 76.36 |         78.24 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r18b-d8_4xb2-80k_cityscapes-769x769.py)                 | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r18b-d8_769x769_80k_cityscapes/deeplabv3plus_r18b-d8_769x769_80k_cityscapes_20201226_151312-2c868aff.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r18b-d8_769x769_80k_cityscapes/deeplabv3plus_r18b-d8_769x769_80k_cityscapes-20201226_151312.log.json)                                 |

| DeepLabV3+        | R-50b-D8        | 769x769   |   80000 | 8.4      | 1.72           | V100   | 79.41 |         80.56 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r50b-d8_4xb2-80k_cityscapes-769x769.py)                 | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50b-d8_769x769_80k_cityscapes/deeplabv3plus_r50b-d8_769x769_80k_cityscapes_20201225_224655-8b596d1c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50b-d8_769x769_80k_cityscapes/deeplabv3plus_r50b-d8_769x769_80k_cityscapes-20201225_224655.log.json)                                 |

| DeepLabV3+        | R-101b-D8       | 769x769   |   80000 | 12.3     | 1.10           | V100   | 79.88 |         81.46 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r101b-d8_4xb2-80k_cityscapes-769x769.py)                | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101b-d8_769x769_80k_cityscapes/deeplabv3plus_r101b-d8_769x769_80k_cityscapes_20201226_205041-227cdf7c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101b-d8_769x769_80k_cityscapes/deeplabv3plus_r101b-d8_769x769_80k_cityscapes-20201226_205041.log.json)                             |

\[1\] The training of the model is sensitive to random seed, and the seed to train it is 1111.

### ADE20K

### ADE20K

| Method     | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                   | download                                                                                                                                                                                                                                                                                                                                                                           |

| DeepLabV3+ | R-50-D8  | 512x512   |   80000 | 10.6     | 21.01          | V100   | 42.72 |         43.75 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r50-d8_4xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_512x512_80k_ade20k/deeplabv3plus_r50-d8_512x512_80k_ade20k_20200614_185028-bf1400d8.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_512x512_80k_ade20k/deeplabv3plus_r50-d8_512x512_80k_ade20k_20200614_185028.log.json)         |

| DeepLabV3+ | R-101-D8 | 512x512   |   80000 | 14.1     | 14.16          | V100   | 44.60 |         46.06 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r101-d8_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_512x512_80k_ade20k/deeplabv3plus_r101-d8_512x512_80k_ade20k_20200615_014139-d5730af7.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_512x512_80k_ade20k/deeplabv3plus_r101-d8_512x512_80k_ade20k_20200615_014139.log.json)     |

| DeepLabV3+ | R-50-D8  | 512x512   |  160000 | -        | -              | V100   | 43.95 |         44.93 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r50-d8_4xb4-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_512x512_160k_ade20k/deeplabv3plus_r50-d8_512x512_160k_ade20k_20200615_124504-6135c7e0.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_512x512_160k_ade20k/deeplabv3plus_r50-d8_512x512_160k_ade20k_20200615_124504.log.json)     |

| DeepLabV3+ | R-101-D8 | 512x512   |  160000 | -        | -              | V100   | 45.47 |         46.35 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r101-d8_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_512x512_160k_ade20k/deeplabv3plus_r101-d8_512x512_160k_ade20k_20200615_123232-38ed86bb.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_512x512_160k_ade20k/deeplabv3plus_r101-d8_512x512_160k_ade20k_20200615_123232.log.json) |

| Method     | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                    | download                                                                                                                                                                                                                                                                                                                                                                               |

| DeepLabV3+ | R-50-D8  | 512x512   |   20000 | 7.6      | 21             | V100   | 75.93 |         77.50 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r50-d8_4xb4-20k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_512x512_20k_voc12aug/deeplabv3plus_r50-d8_512x512_20k_voc12aug_20200617_102323-aad58ef1.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_512x512_20k_voc12aug/deeplabv3plus_r50-d8_512x512_20k_voc12aug_20200617_102323.log.json)     |

| DeepLabV3+ | R-101-D8 | 512x512   |   20000 | 11       | 13.88          | V100   | 77.22 |         78.59 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r101-d8_4xb4-20k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_512x512_20k_voc12aug/deeplabv3plus_r101-d8_512x512_20k_voc12aug_20200617_102345-c7ff3d56.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_512x512_20k_voc12aug/deeplabv3plus_r101-d8_512x512_20k_voc12aug_20200617_102345.log.json) |

| DeepLabV3+ | R-50-D8  | 512x512   |   40000 | -        | -              | V100   | 76.81 |         77.57 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r50-d8_4xb4-40k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_512x512_40k_voc12aug/deeplabv3plus_r50-d8_512x512_40k_voc12aug_20200613_161759-e1b43aa9.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_512x512_40k_voc12aug/deeplabv3plus_r50-d8_512x512_40k_voc12aug_20200613_161759.log.json)     |

| DeepLabV3+ | R-101-D8 | 512x512   |   40000 | -        | -              | V100   | 78.62 |         79.53 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r101-d8_4xb4-40k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_512x512_40k_voc12aug/deeplabv3plus_r101-d8_512x512_40k_voc12aug_20200613_205333-faf03387.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_512x512_40k_voc12aug/deeplabv3plus_r101-d8_512x512_40k_voc12aug_20200613_205333.log.json) |

| Method     | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                         | download                                                                                                                                                                                                                                                                                                                                                                                                       |

| DeepLabV3+ | R-101-D8 | 480x480   |   40000 | -        | 9.09           | V100   | 47.30 |         48.47 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r50-d8_4xb4-40k_pascal-context-480x480.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_480x480_40k_pascal_context/deeplabv3plus_r101-d8_480x480_40k_pascal_context_20200911_165459-d3c8a29e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_480x480_40k_pascal_context/deeplabv3plus_r101-d8_480x480_40k_pascal_context-20200911_165459.log.json) |

| DeepLabV3+ | R-101-D8 | 480x480   |   80000 | -        | -              | V100   | 47.23 |         48.26 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r50-d8_4xb4-80k_pascal-context-480x480.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_480x480_80k_pascal_context/deeplabv3plus_r101-d8_480x480_80k_pascal_context_20200911_155322-145d3ee8.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_480x480_80k_pascal_context/deeplabv3plus_r101-d8_480x480_80k_pascal_context-20200911_155322.log.json) |

| Method     | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                             | download                                                                                                                                                                                                                                                                                                                                                                                                                   |

| DeepLabV3+ | R-101-D8 | 480x480   |   40000 | -        | -              | V100   | 52.86 |         54.54 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r101-d8_4xb4-40k_pascal-context-59-480x480.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_480x480_40k_pascal_context_59/deeplabv3plus_r101-d8_480x480_40k_pascal_context_59_20210416_111233-ed937f15.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_480x480_40k_pascal_context_59/deeplabv3plus_r101-d8_480x480_40k_pascal_context_59-20210416_111233.log.json) |

| DeepLabV3+ | R-101-D8 | 480x480   |   80000 | -        | -              | V100   |  53.2 |         54.67 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r101-d8_4xb4-80k_pascal-context-59-480x480.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_480x480_80k_pascal_context_59/deeplabv3plus_r101-d8_480x480_80k_pascal_context_59_20210416_111127-7ca0331d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_480x480_80k_pascal_context_59/deeplabv3plus_r101-d8_480x480_80k_pascal_context_59-20210416_111127.log.json) |

### LoveDA

### LoveDA

| Method     | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                  | download                                                                                                                                                                                                                                                                                                                                                                       |

| DeepLabV3+ | R-18-D8  | 512x512   |   80000 | 1.93     | 25.57          | V100   | 50.28 |         50.47 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r18-d8_4xb4-80k_loveda-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r18-d8_512x512_80k_loveda/deeplabv3plus_r18-d8_512x512_80k_loveda_20211104_132800-ce0fa0ca.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r18-d8_512x512_80k_loveda/deeplabv3plus_r18-d8_512x512_80k_loveda_20211104_132800.log.json)     |

| DeepLabV3+ | R-50-D8  | 512x512   |   80000 | 7.37     | 6.00           | V100   | 50.99 |         50.65 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r50-d8_4xb4-80k_loveda-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_512x512_80k_loveda/deeplabv3plus_r50-d8_512x512_80k_loveda_20211105_080442-f0720392.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_512x512_80k_loveda/deeplabv3plus_r50-d8_512x512_80k_loveda_20211105_080442.log.json)     |

| DeepLabV3+ | R-101-D8 | 512x512   |   80000 | 10.84    | 4.33           | V100   | 51.47 |         51.32 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r101-d8_4xb4-80k_loveda-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_512x512_80k_loveda/deeplabv3plus_r101-d8_512x512_80k_loveda_20211105_110759-4c1f297e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_512x512_80k_loveda/deeplabv3plus_r101-d8_512x512_80k_loveda_20211105_110759.log.json) |

| DeepLabV3+ | R-18-D8  | 512x512   |   80000 | 1.91     | 81.68          | V100   | 77.09 |         78.44 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r18-d8_4xb4-80k_potsdam-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r18-d8_512x512_80k_potsdam/deeplabv3plus_r18-d8_512x512_80k_potsdam_20211219_020601-75fd5bc3.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r18-d8_512x512_80k_potsdam/deeplabv3plus_r18-d8_512x512_80k_potsdam_20211219_020601.log.json)     |

| DeepLabV3+ | R-50-D8  | 512x512   |   80000 | 7.36     | 26.44          | V100   | 78.33 |         79.27 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r50-d8_4xb4-80k_potsdam-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_512x512_80k_potsdam/deeplabv3plus_r50-d8_512x512_80k_potsdam_20211219_031508-7e7a2b24.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_512x512_80k_potsdam/deeplabv3plus_r50-d8_512x512_80k_potsdam_20211219_031508.log.json)     |

| DeepLabV3+ | R-101-D8 | 512x512   |   80000 | 10.83    | 17.56          | V100   |  78.7 |         79.47 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r101-d8_4xb4-80k_potsdam-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_512x512_80k_potsdam/deeplabv3plus_r101-d8_512x512_80k_potsdam_20211219_031508-8b112708.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_512x512_80k_potsdam/deeplabv3plus_r101-d8_512x512_80k_potsdam_20211219_031508.log.json) |

| Method     | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                     | download                                                                                                                                                                                                                                                                                                                                                                                                   |

| DeepLabV3+ | R-18-D8  | 512x512   |   80000 | 1.91     | 72.79          | V100   | 72.50 |         74.13 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r18-d8_4xb4-80k_vaihingen-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r18-d8_4x4_512x512_80k_vaihingen/deeplabv3plus_r18-d8_4x4_512x512_80k_vaihingen_20211231_230805-7626a263.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r18-d8_4x4_512x512_80k_vaihingen/deeplabv3plus_r18-d8_4x4_512x512_80k_vaihingen_20211231_230805.log.json)     |

| DeepLabV3+ | R-50-D8  | 512x512   |   80000 | 7.36     | 26.91          | V100   | 73.97 |         75.05 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r50-d8_4xb4-80k_vaihingen-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_4x4_512x512_80k_vaihingen/deeplabv3plus_r50-d8_4x4_512x512_80k_vaihingen_20211231_230816-5040938d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_4x4_512x512_80k_vaihingen/deeplabv3plus_r50-d8_4x4_512x512_80k_vaihingen_20211231_230816.log.json)     |

| DeepLabV3+ | R-101-D8 | 512x512   |   80000 | 10.83    | 18.59          | V100   | 73.06 |         74.14 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r101-d8_4xb4-80k_vaihingen-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_4x4_512x512_80k_vaihingen/deeplabv3plus_r101-d8_4x4_512x512_80k_vaihingen_20211231_230816-8a095afa.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r101-d8_4x4_512x512_80k_vaihingen/deeplabv3plus_r101-d8_4x4_512x512_80k_vaihingen_20211231_230816.log.json) |

### iSAID

### iSAID

| Method     | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                | download                                                                                                                                                                                                                                                                                                                                                                               |

| DeepLabV3+ | R-18-D8  | 896x896   |   80000 | 6.19     | 24.81          | V100   | 61.35 |         62.61 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r18-d8_4xb4-80k_isaid-896x896.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r18-d8_4x4_896x896_80k_isaid/deeplabv3plus_r18-d8_4x4_896x896_80k_isaid_20220110_180526-7059991d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r18-d8_4x4_896x896_80k_isaid/deeplabv3plus_r18-d8_4x4_896x896_80k_isaid_20220110_180526.log.json) |

| DeepLabV3+ | R-50-D8  | 896x896   |   80000 | 21.45    | 8.42           | V100   | 67.06 |         68.02 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r50-d8_4xb4-80k_isaid-896x896.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_4x4_896x896_80k_isaid/deeplabv3plus_r50-d8_4x4_896x896_80k_isaid_20220110_180526-598be439.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_4x4_896x896_80k_isaid/deeplabv3plus_r50-d8_4x4_896x896_80k_isaid_20220110_180526.log.json) |

### Mapillary Vistas v1.2

| Method     | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                            | download                                                                                                                                                                                                                                                                                                                                                                                                                               |

| DeepLabV3+ | R-50-D8  | 1280x1280 |  300000 | 24.04    | 17.92          | A100   | 47.35 |             - | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus/deeplabv3plus_r50-d8_4xb2-300k_mapillay_v1_65-1280x1280.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_4xb2-300k_mapillay_v1_65-1280x1280/deeplabv3plus_r50-d8_4xb2-300k_mapillay_v1_65-1280x1280_20230301_110504-655f8e43.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/deeplabv3plus/deeplabv3plus_r50-d8_4xb2-300k_mapillay_v1_65-1280x1280/deeplabv3plus_r50-d8_4xb2-300k_mapillay_v1_65-1280x1280_20230301_110504.json) |

Note:

- `D-8`/`D-16` here corresponding to the output stride 8/16 setting for DeepLab series.
- `MG-124` stands for multi-grid dilation in the last stage of ResNet.

```bibtex
@inproceedings{deeplabv3plus2018,
  title={Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation},
  author={Liang-Chieh Chen and Yukun Zhu and George Papandreou and Florian Schroff and Hartwig Adam},
}
```

# 3D human pose estimation in video with temporal convolutions and semi-supervised training

<details>

```bibtex
  title={3d human pose estimation in video with temporal convolutions and semi-supervised training},
  author={Pavllo, Dario and Feichtenhofer, Christoph and Grangier, David and Auli, Michael},
  pages={7753--7762},
}
```

}
```

In this work, we demonstrate that 3D poses in video can be effectively estimated with a fully convolutional model based on dilated temporal convolutions over 2D keypoints. We also introduce back-projection, a simple and effective semi-supervised training method that leverages unlabeled video data. We start with predicted 2D keypoints for unlabeled video, then estimate 3D poses and finally back-project to the input 2D keypoints. In the supervised setting, our fully-convolutional model outperforms the previous best result from the literature by 6 mm mean per-joint position error on Human3.6M, corresponding to an error reduction of 11%, and the model also shows significant improvements on HumanEva-I. Moreover, experiments with back-projection show that it comfortably outperforms previous state-of-the-art results in semi-supervised settings where labeled data is scarce.

<img src="https://user-images.githubusercontent.com/15977946/146525184-34a14153-5cab-4fe0-8500-018c99a5f647.png">
</div>

# Tokens-to-Token ViT

> [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://arxiv.org/abs/2101.11986)

Transformers, which are popular for language modeling, have been explored for solving vision tasks recently, e.g., the Vision Transformer (ViT) for image classification. The ViT model splits each image into a sequence of tokens with fixed length and then applies multiple Transformer layers to model their global relation for classification. However, ViT achieves inferior performance to CNNs when trained from scratch on a midsize dataset like ImageNet. We find it is because: 1) the simple tokenization of input images fails to model the important local structure such as edges and lines among neighboring pixels, leading to low training sample efficiency; 2) the redundant attention backbone design of ViT leads to limited feature richness for fixed computation budgets and limited training samples. To overcome such limitations, we propose a new Tokens-To-Token Vision Transformer (T2T-ViT), which incorporates 1) a layer-wise Tokens-to-Token (T2T) transformation to progressively structurize the image to tokens by recursively aggregating neighboring Tokens into one Token (Tokens-to-Token), such that local structure represented by surrounding tokens can be modeled and tokens length can be reduced; 2) an efficient backbone with a deep-narrow structure for vision transformer motivated by CNN architecture design after empirical study. Notably, T2T-ViT reduces the parameter count and MACs of vanilla ViT by half, while achieving more than 3.0% improvement when trained from scratch on ImageNet. It also outperforms ResNets and achieves comparable performance with MobileNets by directly training on ImageNet. For example, T2T-ViT with comparable size to ResNet50 (21.5M parameters) can achieve 83.3% top1 accuracy in image resolution 384×384 on ImageNet.

<img src="https://user-images.githubusercontent.com/26739999/142578381-e9040610-05d9-457c-8bf5-01c2fa94add2.png" width="60%"/>
</div>

```python

predict = inference_model('t2t-vit-t-14_8xb64_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('t2t-vit-t-14_8xb64_in1k', pretrained=True)
```

Train:

```shell
python tools/train.py configs/t2t_vit/t2t-vit-t-14_8xb64_in1k.py
```

Test:

```shell

Train:

```shell
```

Test:

```shell
python tools/test.py configs/t2t_vit/t2t-vit-t-14_8xb64_in1k.py https://download.openmmlab.com/mmclassification/v0/t2t-vit/t2t-vit-t-14_8xb64_in1k_20211220-f7378dd5.pth
```

| `t2t-vit-t-14_8xb64_in1k` | From scratch |   21.47    |   4.34    |   81.83   |   95.84   | [config](t2t-vit-t-14_8xb64_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/t2t-vit/t2t-vit-t-14_8xb64_in1k_20211220-f7378dd5.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/t2t-vit/t2t-vit-t-14_8xb64_in1k_20211220-f7378dd5.json) |
| `t2t-vit-t-19_8xb64_in1k` | From scratch |   39.08    |   7.80    |   82.63   |   96.18   | [config](t2t-vit-t-19_8xb64_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/t2t-vit/t2t-vit-t-19_8xb64_in1k_20211214-7f5e3aaf.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/t2t-vit/t2t-vit-t-19_8xb64_in1k_20211214-7f5e3aaf.json) |

| `t2t-vit-t-24_8xb64_in1k` | From scratch |   64.00    |   12.69   |   82.71   |   96.09   | [config](t2t-vit-t-24_8xb64_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/t2t-vit/t2t-vit-t-24_8xb64_in1k_20211214-b2a68ae3.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/t2t-vit/t2t-vit-t-24_8xb64_in1k_20211214-b2a68ae3.json) |

```bibtex
@article{yuan2021tokens,
  title={Tokens-to-token vit: Training vision transformers from scratch on imagenet},
  author={Yuan, Li and Chen, Yunpeng and Wang, Tao and Yu, Weihao and Shi, Yujun and Tay, Francis EH and Feng, Jiashi and Yan, Shuicheng},
  journal={arXiv preprint arXiv:2101.11986},
}
```

# Overview

This section introduce the following contents in terms of migration from MMEditing 0.x

  - [New dependencies](#new-dependencies)
  - [Overall structures](#overall-structures)
  - [Other config settings](#other-config-settings)

## New dependencies

MMagic 1.x depends on some new packages, you can prepare a new clean environment and install again according to the [install tutorial](../get_started/install.md).

## Overall structures

We refactor overall structures in MMagic 1.x as following.

- The  `core` in the old versions of MMEdit is split into `engine`, `evaluation`, `structures`, and `visualization`
- The `pipelines` of `datasets` in the old versions of MMEdit is refactored to `transforms`
- The `models` in MMagic 1.x is refactored to six parts: `archs`, `base_models`, `data_preprocessors`, `editors`, `diffusion_schedulers` and `losses`.

## Other config settings

We rename config file to new template: `{model_settings}_{module_setting}_{training_setting}_{datasets_info}`.

More details of config are shown in [config guides](../user_guides/config.md).

# Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments

<details>

```bibtex
}
```

# 贡献指南

OpenMMLab 欢迎所有人参与我们项目的共建。本文档将指导您如何通过拉取请求为 OpenMMLab 项目作出贡献。

## 什么是拉取请求？

`拉取请求` (Pull Request), [GitHub 官方文档](https://docs.github.com/en/github/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests)定义如下。

```
拉取请求是一种通知机制。你修改了他人的代码，将你的修改通知原来作者，希望他合并你的修改。
```

## 基本的工作流：

1. 获取最新的代码库
2. 从最新的 `dev-1.x` 分支创建分支进行开发
3. 提交修改 ([不要忘记使用 pre-commit hooks!](#3-提交你的修改))
4. 推送你的修改并创建一个 `拉取请求`
5. 讨论、审核代码
6. 将开发分支合并到 `dev-1.x` 分支

## 具体步骤

### 1. 获取最新的代码库

- 当你第一次提 PR 时

  复刻 OpenMMLab 原代码库，点击 GitHub 页面右上角的 **Fork** 按钮即可
  ![avatar](https://user-images.githubusercontent.com/22607038/195038780-06a46340-8376-4bde-a07f-2577f231a204.png)

  克隆复刻的代码库到本地

  ```bash
  git clone git@github.com:XXX/mmocr.git
  ```

  添加原代码库为上游代码库

  ```bash
  git remote add upstream git@github.com:open-mmlab/mmocr
  ```

- 从第二个 PR 起

  检出本地代码库的主分支，然后从最新的原代码库的主分支拉取更新。这里假设你正基于 `dev-1.x` 开发。

  ```bash
  ```

```

- 从第二个 PR 起

  ```bash
  ```

### 2. 从 `dev-1.x` 分支创建一个新的开发分支

```bash
```

```{tip}
为了保证提交历史清晰可读，我们强烈推荐您先切换到 `dev-1.x` 分支，再创建新的分支。
```

### 3. 提交你的修改

- 如果你是第一次尝试贡献，请在 MMOCR 的目录下安装并初始化 pre-commit hooks。

  ```bash
  ```

- 提交修改。在每次提交前，pre-commit hooks 都会被触发并规范化你的代码格式。

  ```bash
  # coding
  ```

  ```{note}
  有时你的文件可能会在提交时被 pre-commit hooks 自动修改。这时请重新添加并提交修改后的文件。
  ```

### 4. 推送你的修改到复刻的代码库，并创建一个拉取请求

- 推送当前分支到远端复刻的代码库

  ```bash
  ```

- 创建一个拉取请求

  ![avatar](https://user-images.githubusercontent.com/22607038/195053564-71bd3cb4-b8d4-4ed9-9075-051e138b7fd4.png)

  ```bash
  ```

- 创建一个拉取请求

- 修改拉取请求信息模板，描述修改原因和修改内容。还可以在 PR 描述中，手动关联到相关的议题 (issue),（更多细节，请参考[官方文档](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue)）。

- 另外，如果你正在往 `dev-1.x` 分支提交代码，你还需要在创建 PR 的界面中将基础分支改为 `dev-1.x`，因为现在默认的基础分支是 `main`。

  ![avatar](https://user-images.githubusercontent.com/22607038/195045928-f3ceedc8-0162-46a7-ae1a-7e22829fe189.png)

- 你同样可以把 PR 关联给相关人员进行评审。

### 5. 讨论并评审你的代码

- 根据评审人员的意见修改代码，并推送修改

### 6. `拉取请求`合并之后删除该分支

- 在 PR 合并之后，你就可以删除该分支了。

  ```bash
  git branch -d branchname # 删除本地分支
  git push origin --delete branchname # 删除远程分支
  ```

## PR 规范

2. 一个 PR 对应一个短期分支

3. 粒度要细，一个PR只做一件事情，避免超大的PR

```

## PR 规范

To meet diverse requirements, MMOCR supports training and testing models on various devices, including PCs, work stations, computation clusters, etc.

## Single GPU Training and Testing

`tools/train.py` provides the basic training service. MMOCR recommends using GPUs for model training and testing, but it still enables CPU-Only training and testing. For example, the following commands demonstrate how to train a DBNet model using a single GPU or CPU.

```bash
# Train the specified MMOCR model by calling tools/train.py

# Training
# Example 1: Training DBNet with CPU

# Example 2: Specify to train DBNet with gpu:0, specify the working directory as dbnet/, and turn on mixed precision (amp) training

```

```{note}
If multiple GPUs are available, you can specify a certain GPU, e.g. the third one, by setting CUDA_VISIBLE_DEVICES=3.
```

The following table lists all the arguments supported by `train.py`. Args without the `--` prefix are mandatory, while others are optional.

| ARGS            | Type | Description                                                                 |
| --------------- | ---- | --------------------------------------------------------------------------- |
| config          | str  | (required) Path to config.                                                  |
| --work-dir      | str  | Specify the working directory for the training logs and models checkpoints. |

| --resume        | bool | Whether to resume training from the latest checkpoint.                      |
| --amp           | bool | Whether to use automatic mixture precision for training.                    |
| --auto-scale-lr | bool | Whether to use automatic learning rate scaling.                             |
| --cfg-options   | str  | Override some settings in the configs. [Example](<>)                        |
| --launcher      | str  | Option for launcher，\['none', 'pytorch', 'slurm', 'mpi'\].                 |
| --local_rank    | int  | Rank of local machine，used for distributed training，defaults to 0。       |
| --tta           | bool | Whether to use test time augmentation.                                      |

### Test

### Test

`tools/test.py` provides the basic testing service, which is used in a similar way to the training script. For example, the following command demonstrates test a DBNet model on a single GPU or CPU.

```bash
# Test a pretrained MMOCR model by calling tools/test.py

# Test
# Example 1: Testing DBNet with CPU

# Example 2: Testing DBNet on gpu:0
```

The following table lists all the arguments supported by `test.py`. Args without the `--` prefix are mandatory, while others are optional.

```

| ARGS          | Type  | Description                                                          |
| ------------- | ----- | -------------------------------------------------------------------- |
| config        | str   | (required) Path to config.                                           |
| checkpoint    | str   | (required) The model to be tested.                                   |
| --work-dir    | str   | Specify the working directory for the logs.                          |
| --save-preds  | bool  | Whether to save the predictions to a pkl file.                       |
| --show        | bool  | Whether to visualize the predictions.                                |
| --show-dir    | str   | Path to save the visualization results.                              |

| --wait-time   | float | Interval of visualization (s), defaults to 2.                        |
| --cfg-options | str   | Override some settings in the configs. [Example](<>)                 |
| --launcher    | str   | Option for launcher，\['none', 'pytorch', 'slurm', 'mpi'\].          |
| --local_rank  | int   | Rank of local machine，used for distributed training，defaults to 0. |

## Training and Testing with Multiple GPUs

For large models, distributed training or testing significantly improves the efficiency. For this purpose, MMOCR provides distributed scripts `tools/dist_train.sh` and `tools/dist_test.sh` implemented based on [MMDistributedDataParallel](mmengine.model.wrappers.MMDistributedDataParallel).

```bash
# Training

```bash
# Training

# Testing
```

The following table lists the arguments supported by `dist_*.sh`.

| ARGS            | Type | Description                                                                                   |
| --------------- | ---- | --------------------------------------------------------------------------------------------- |
| NNODES          | int  | The number of nodes. Defaults to 1.                                                           |
| NODE_RANK       | int  | The rank of current node. Defaults to 0.                                                      |

| PORT            | int  | The master port that will be used by rank 0 node, ranging from 0 to 65535. Defaults to 29500. |
| MASTER_ADDR     | str  | The address of rank 0 node. Defaults to "127.0.0.1".                                          |
| CONFIG_FILE     | str  | (required) The path to config.                                                                |
| CHECKPOINT_FILE | str  | (required，only used in dist_test.sh)The path to checkpoint to be tested.                     |
| GPU_NUM         | int  | (required) The number of GPUs to be used per node.                                            |
| \[PY_ARGS\]     | str  | Arguments to be parsed by tools/train.py and tools/test.py.                                   |

These two scripts enable training and testing on **single-machine multi-GPU** or **multi-machine multi-GPU**. See the following example for usage.

### Single-machine Multi-GPU

The following commands demonstrate how to train and test with a specified number of GPUs on a **single machine** with multiple GPUs.

1. **Training**

   Training DBNet using 4 GPUs on a single machine.

   ```bash
   ```

2. **Testing**

   Testing DBNet using 4 GPUs on a single machine.

   ```bash
   ```

### Launching Multiple Tasks on Single Machine

```bash
   ```

For a workstation equipped with multiple GPUs, the user can launch multiple tasks simultaneously by specifying the GPU IDs. For example, the following command demonstrates how to test DBNet with GPU `[0, 1, 2, 3]` and train CRNN on GPU `[4, 5, 6, 7]`.

```bash
# Specify gpu:0,1,2,3 for testing and assign port number 29500

# Specify gpu:4,5,6,7 for training and assign port number 29501
```

```{note}

```

```{note}
`dist_train.sh` sets `MASTER_PORT` to `29500` by default. When other processes already occupy this port, the program will get a runtime error `RuntimeError: Address already in use`. In this case, you need to set `MASTER_PORT` to another free port number in the range of `(0~65535)`.
```

### Multi-machine Multi-GPU Training and Testing

You can launch a task on multiple machines connected to the same network. MMOCR relies on `torch.distributed` package for distributed training. Find more information at PyTorch’s [launch utility](https://pytorch.org/docs/stable/distributed.html#launch-utility).

   The following command demonstrates how to train DBNet on two machines with a total of 4 GPUs.

   ```bash
   # Say that you want to launch the training job on two machines

   ```bash
   # On the first machine:
   NNODES=2 NODE_RANK=0 PORT=29500 MASTER_ADDR=10.140.0.169 tools/dist_train.sh configs/textdet/dbnet/dbnet_r50dcnv2_fpnc_1200e_icdar2015.py 2
   # On the second machine:
   ```

   The following command demonstrates how to test DBNet on two machines with a total of 4 GPUs.

   ```bash
   # Say that you want to launch the testing job on two machines

NNODES=2 NODE_RANK=0 PORT=29500 MASTER_ADDR=10.140.0.169 tools/dist_test.sh configs/textdet/dbnet/dbnet_r50dcnv2_fpnc_1200e_icdar2015.py dbnet_r50.pth 2
   ```

   ```{note}
   The speed of the network could be the bottleneck of training.
   ```

## Training and Testing with Slurm Cluster

If you run MMOCR on a cluster managed with [Slurm](https://slurm.schedmd.com/), you can use the script `tools/slurm_train.sh` and `tools/slurm_test.sh`.

```bash
# tools/slurm_train.sh provides scripts for submitting training tasks on clusters managed by the slurm

# tools/slurm_test.sh provides scripts for submitting testing tasks on clusters managed by the slurm
```

| ARGS            | Type | Description                                                                                                 |
| --------------- | ---- | ----------------------------------------------------------------------------------------------------------- |
| GPUS            | int  | The number of GPUs to be used by this task. Defaults to 8.                                                  |

| GPUS_PER_NODE   | int  | The number of GPUs to be allocated per node. Defaults to 8.                                                 |
| CPUS_PER_TASK   | int  | The number of CPUs to be allocated per task. Defaults to 5.                                                 |
| SRUN_ARGS       | str  | Arguments to be parsed by srun. Available options can be found [here](https://slurm.schedmd.com/srun.html). |
| PARTITION       | str  | (required) Specify the partition on cluster.                                                                |
| JOB_NAME        | str  | (required) Name of the submitted job.                                                                       |
| WORK_DIR        | str  | (required) Specify the working directory for saving the logs and checkpoints.                               |

| CHECKPOINT_FILE | str  | (required，only used in slurm_test.sh)Path to the checkpoint to be tested.                                  |
| PY_ARGS         | str  | Arguments to be parsed by `tools/train.py` and `tools/test.py`.                                             |

These scripts enable training and testing on slurm clusters, see the following examples.

1. Training

   Here is an example of using 1 GPU to train a DBNet model on the `dev` partition.

   ```bash
   # Example: Request 1 GPU resource on dev partition for DBNet training task
   GPUS=1 GPUS_PER_NODE=1 CPUS_PER_TASK=5 tools/slurm_train.sh dev db_r50 configs/textdet/dbnet/dbnet_r50dcnv2_fpnc_1200e_icdar2015.py work_dir
   ```

2. Testing

   Similarly, the following example requests 1 GPU for testing.

   ```bash

```

2. Testing

   ```bash
   # Example: Request 1 GPU resource on dev partition for DBNet testing task
   GPUS=1 GPUS_PER_NODE=1 CPUS_PER_TASK=5 tools/slurm_test.sh dev db_r50 configs/textdet/dbnet/dbnet_r50dcnv2_fpnc_1200e_icdar2015.py dbnet_r50.pth work_dir
   ```

## Advanced Tips

### Resume Training from a Checkpoint

`tools/train.py` allows users to resume training from a checkpoint by specifying the `--resume` parameter, where it will automatically resume training from the latest saved checkpoint.

```bash
# Example: Resuming training from the latest checkpoint
```

By default, the program will automatically resume training from the last successfully saved checkpoint in the last training session, i.e. `latest.pth`. However,

```python
# Example: Set the path of the checkpoint you want to load in the configuration file

```python
```

### Mixed Precision Training

Mixed precision training offers significant computational speedup by performing operations in half-precision format, while storing minimal information in single-precision to retain as much information as possible in critical parts of the network. In MMOCR, the users can enable the automatic mixed precision training by simply add `--amp`.

```bash
# Example: Using automatic mixed precision training
```

The following table shows the support of each algorithm in MMOCR for automatic mixed precision training.

|               | Whether support AMP |               Description               |
| ------------- | :-----------------: | :-------------------------------------: |

|               |   Text Detection    |                                         |
| DBNet         |          Y          |                                         |
| DBNetpp       |          Y          |                                         |
| DRRG          |          N          | roi_align_rotated does not support fp16 |
| FCENet        |          N          |      BCELoss does not support fp16      |
| Mask R-CNN    |          Y          |                                         |
| PANet         |          Y          |                                         |
| PSENet        |          Y          |                                         |
| TextSnake     |          N          |                                         |
|               |  Text Recognition   |                                         |

| ABINet        |          Y          |                                         |
| CRNN          |          Y          |                                         |
| MASTER        |          Y          |                                         |
| NRTR          |          Y          |                                         |
| RobustScanner |          Y          |                                         |
| SAR           |          Y          |                                         |
| SATRN         |          Y          |                                         |

### Automatic Learning Rate Scaling

MMOCR sets default initial learning rates for each model in the configuration file. However, these initial learning rates may not be applicable when the user uses a different `batch_size` than our preset `base_batch_size`. Therefore, we provide a tool to automatically scale the learning rate, which can be called by adding the `--auto-scale-lr`.

```bash
# Example: Using automatic learning rate scaling
```

### Visualize the Predictions

`tools/test.py` provides the visualization interface to facilitate the qualitative analysis of the OCR models.

![Detection](../../../demo/resources/det_vis.png)

(Green boxes are GTs, while red boxes are predictions)

</div>

![Recognition](../../../demo/resources/rec_vis.png)

</div>

(Green font is the GT, red font is the prediction)

</div>

![KIE](../../../demo/resources/kie_vis.png)

(From left to right: original image, text detection and recognition result, text classification result, relationship)

</div>

```bash
# Example 1: Show the visualization results per 2 seconds

# Example 2: For systems that do not support graphical interfaces (such as computing clusters, etc.), the visualization results can be dumped in the specified path
```

The visualization-related parameters in `tools/test.py` are described as follows.

```

| ARGS        | Type  | Description                                   |
| ----------- | ----- | --------------------------------------------- |
| --show      | bool  | Whether to show the visualization results.    |
| --show-dir  | str   | Path to save the visualization results.       |
| --wait-time | float | Interval of visualization (s), defaults to 2. |

### Test Time Augmentation

Test time augmentation (TTA) is a technique that is used to improve the performance of a model by performing data augmentation on the input image at test time. It is a simple yet effective method to improve the performance of a model. In MMOCR, we support TTA in the following ways:

```{note}
TTA is only supported for text recognition models.
```

```bash
```

# 数据集验证

在 MMPretrain 中，`tools/misc/verify_dataset.py` 脚本会检查数据集的所有图片，查看是否有**已经损坏**的图片。

## 工具介绍

```shell
```

**所有参数说明**:

- `--out-path` : 输出结果路径，默认为 ‘brokenfiles.log’。
- `--phase` :  检查哪个阶段的数据集，可用值为 “train” 、”test” 或者 “val”， 默认为 “train”。
- `--num-process` : 指定的进程数，默认为 1。
- `--cfg-options`: 额外的配置选项，会被合入配置文件，参考[教程 1：如何编写配置文件](https://mmpretrain.readthedocs.io/zh_CN/latest/tutorials/config.html)。

## 示例：

```shell
```

## Customize datasets by reorganizing data to COCO format

The simplest way to use the custom dataset is to convert your annotation format to COCO dataset format.

The annotation JSON files in COCO format have the following necessary keys:

```python
'images': [
    {
        'file_name': '000000001268.jpg',
        'height': 427,
        'width': 640,
        'id': 1268
    },
    ...
],
'annotations': [
    {
        'segmentation': [[426.36,
            ...
            424.34,
            223.3]],
        'keypoints': [0,0,0,
            0,0,0,
            0,0,0,
            427,220,2,
            443,222,2,
            414,228,2,
            449,232,2,
            408,248,1,
            454,261,2,
            0,0,0,
            0,0,0,
            411,287,2,
            431,287,2,
            0,0,0,
            458,265,2,
            0,0,0,
            466,300,1],
        'num_keypoints': 10,
        'area': 3894.5826,
        'iscrowd': 0,

431,287,2,
            0,0,0,
            0,0,0,
        'image_id': 1268,
        'bbox': [402.34, 205.02, 65.26, 88.45],
        'category_id': 1,
        'id': 215218
    },
    ...
],
'categories': [
    {'id': 1, 'name': 'person'},
 ]
```

There are three necessary keys in the json file:

- `images`: contains a list of images with their information like `file_name`, `height`, `width`, and `id`.
- `annotations`: contains the list of instance annotations.
- `categories`: contains the category name ('person') and its ID (1).

If the annotations have been organized in COCO format, there is no need to create a new dataset class. You can use `CocoDataset` class alternatively.

## Create a custom dataset_info config file for the dataset

Add a new dataset info config file that contains the metainfo about the dataset.

```

```
configs/_base_/datasets/custom.py
```

An example of the dataset config is as follows.

`keypoint_info` contains the information about each keypoint.

1. `name`: the keypoint name. The keypoint name must be unique.
2. `id`: the keypoint id.
3. `color`: (\[B, G, R\]) is used for keypoint visualization.
4. `type`: 'upper' or 'lower', will be used in data augmentation.
5. `swap`: indicates the 'swap pair' (also known as 'flip pair'). When applying image horizontal flip, the left part will become the right part. We need to flip the keypoints accordingly.

`skeleton_info` contains information about the keypoint connectivity, which is used for visualization.

`joint_weights` assigns different loss weights to different keypoints.

`sigmas` is used to calculate the OKS score. You can read [keypoints-eval](https://cocodataset.org/#keypoints-eval) to learn more about it.

Here is an simplified example of dataset_info config file ([full text](/configs/_base_/datasets/coco.py)).

```
    dataset_name='coco',
        author='Lin, Tsung-Yi and Maire, Michael and '
        'Belongie, Serge and Hays, James and '
        'Perona, Pietro and Ramanan, Deva and '
        r'Doll{\'a}r, Piotr and Zitnick, C Lawrence',
        title='Microsoft coco: Common objects in context',
        container='European conference on computer vision',
        homepage='http://cocodataset.org/',
    ),
        0:
        dict(name='nose', id=0, color=[51, 153, 255], type='upper', swap=''),

year='2014',
    ),
        0:
        1:
        dict(
            name='left_eye',
            id=1,
            color=[51, 153, 255],
            type='upper',
            swap='right_eye'),
        ...
        16:
        dict(
            id=16,
            swap='left_ankle')
    },
        0:
        dict(link=('left_ankle', 'left_knee'), id=0, color=[0, 255, 0]),
        ...
        18:
        dict(
            link=('right_ear', 'right_shoulder'), id=18, color=[51, 153, 255])
    },
        1., 1., 1., 1., 1., 1., 1., 1.2, 1.2, 1.5, 1.5, 1., 1., 1.2, 1.2, 1.5,
        1.5
    ],
    sigmas=[
        0.026, 0.025, 0.025, 0.035, 0.035, 0.079, 0.079, 0.072, 0.072, 0.062,

1., 1., 1., 1., 1., 1., 1., 1.2, 1.2, 1.5, 1.5, 1., 1., 1.2, 1.2, 1.5,
        1.5
    ],
    sigmas=[
        0.062, 0.107, 0.107, 0.087, 0.087, 0.089, 0.089
    ])
```

## Create a custom dataset class

If the annotations are not organized in COCO format, you need to create a custom dataset class by the following steps:

1. First create a package inside the `mmpose/datasets/datasets` folder.

2. Create a class definition of your dataset in the package folder and register it in the registry with a name. Without a name, it will keep giving the error. `KeyError: 'XXXXX is not in the dataset registry'`

   ```
   from mmengine.dataset import BaseDataset
   from mmpose.registry import DATASETS

   @DATASETS.register_module(name='MyCustomDataset')
   class MyCustomDataset(BaseDataset):
   ```

```

   ```

   You can refer to [this doc](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/basedataset.html) on how to build customed dataset class with `mmengine.BaseDataset`.

3. Make sure you have updated the `__init__.py` of your package folder

4. Make sure you have updated the `__init__.py` of the dataset package folder.

## Create a custom training config file

Create a custom training config file as per your need and the model/architecture you want to use in the configs folder. You may modify an existing config file to use the new custom dataset.

```python
...

```python
...

        ...),
    )

        ...),
    )

        data_root='root/of/your/test/data',
        ann_file='path/to/your/test/json',
        data_prefix=dict(img='path/to/your/test/img'),

        ...),
    )
...
```

Make sure you have provided all the paths correctly.

## Dataset Wrappers

The following dataset wrappers are supported in [MMEngine](https://github.com/open-mmlab/mmengine), you can refer to [MMEngine tutorial](https://mmengine.readthedocs.io/en/latest) to learn how to use it.

- [ConcatDataset](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/basedataset.html#concatdataset)
- [RepeatDataset](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/basedataset.html#repeatdataset)

### CombinedDataset

MMPose provides `CombinedDataset` to combine multiple datasets with different annotations. A combined dataset can be defined in config files as:

```python
dataset_1 = dict(

```python
    type='dataset_type_1',
    data_root='root/of/your/dataset1',
    data_prefix=dict(img_path='path/to/your/img'),
    ann_file='annotations/train.json',
        # the converter transforms convert data into a unified format
        converter_transform_1
    ])

dataset_2 = dict(
    type='dataset_type_2',
    data_root='root/of/your/dataset2',
        converter_transform_2
    ])

shared_pipeline = [
    LoadImage(),
    ParseImage(),
]

combined_dataset = dict(
    metainfo=dict(from_file='path/to/your/metainfo'),
    datasets=[dataset_1, dataset_2],
    pipeline=shared_pipeline,
)
```

]

)
```

- **MetaInfo of combined dataset** determines the annotation format. Either metainfo of a sub-dataset or a customed dataset metainfo is valid here. To custom a dataset metainfo, please refer to [Create a custom dataset_info config file for the dataset](#create-a-custom-datasetinfo-config-file-for-the-dataset).

- **Converter transforms of sub-datasets** are applied when there exist mismatches of annotation format between sub-datasets and the combined dataset. For example, the number and order of keypoints might be different in the combined dataset and the sub-datasets. Then `KeypointConverter` can be used to unify the keypoints number and order.

- More details about `CombinedDataset` and `KeypointConverter` can be found in Advanced Guides-[Training with Mixed Datasets](../user_guides/mixed_datasets.md).

In our design, a complete model is defined as a top-level module which contains several model components based on their functionalities.

- model: a top-level module defines the type of the task, such as `ImageClassifier` for image classification, `MAE` for self-supervised leanrning, `ImageToImageRetriever` for image retrieval.
- backbone: usually a feature extraction network that records the major differences between models, e.g., `ResNet`, `MobileNet`.
- neck: the component between backbone and head, e.g., `GlobalAveragePooling`.
- head: the component for specific tasks, e.g., `ClsHead`, `ContrastiveHead`.
- loss: the component in the head for calculating losses, e.g., `CrossEntropyLoss`, `LabelSmoothLoss`.
- target_generator: the component for self-supervised leanrning task specifically, e.g., `VQKD`, `HOGGenerator`.

## Add a new model

Generally, for image classification and retrieval tasks, the pipelines are consistent. However, the pipelines are different from each self-supervised leanrning algorithms, like `MAE` and `BEiT`. Thus, in this section, we will explain how to add your self-supervised learning algorithm.

### Add a new self-supervised learning algorithm

1. Create a new file `mmpretrain/models/selfsup/new_algorithm.py` and implement `NewAlgorithm` in it.

   ```python
   from mmpretrain.registry import MODELS
   from .base import BaseSelfSupvisor

   class NewAlgorithm(BaseSelfSupvisor):

       def __init__(self, backbone, neck=None, head=None, init_cfg=None):
           super().__init__(init_cfg)
           pass

       # ``extract_feat`` function is defined in BaseSelfSupvisor, you could
       # overwrite it if needed

super().__init__(init_cfg)
           pass

       def extract_feat(self, inputs, **kwargs):
           pass

       # the core function to compute the loss
       def loss(self, inputs, data_samples, **kwargs):
           pass

   ```

2. Import the new algorithm module in `mmpretrain/models/selfsup/__init__.py`

   ```python
   ...
   from .new_algorithm import NewAlgorithm

   __all__ = [
       ...,
       'NewAlgorithm',
       ...
   ]
   ```

3. Use it in your config file.

   ```python
       type='NewAlgorithm',
       backbone=...,
       neck=...,
       head=...,
       ...
   )
   ```

## Add a new backbone

Here we present how to develop a new backbone component by an example of `ResNet_CIFAR`.

backbone=...,
       neck=...,
       head=...,
       ...
   )
   ```

As the input size of CIFAR is 32x32, which is much smaller than the default size of 224x224 in ImageNet, this backbone replaces the `kernel_size=7, stride=2` to `kernel_size=3, stride=1` and removes the MaxPooling after the stem layer to avoid forwarding small feature maps to residual blocks.

The easiest way is to inherit from `ResNet` and only modify the stem layer.

1. Create a new file `mmpretrain/models/backbones/resnet_cifar.py`.

   ```python

   from .resnet import ResNet

   class ResNet_CIFAR(ResNet):

       """ResNet backbone for CIFAR.

       short description of the backbone

       Args:
           depth(int): Network depth, from {18, 34, 50, 101, 152}.
           ...
       """

"""ResNet backbone for CIFAR.

       Args:
           ...
       """

       def __init__(self, depth, deep_stem, **kwargs):
           # call ResNet init
           super(ResNet_CIFAR, self).__init__(depth, deep_stem=deep_stem, **kwargs)
           # other specific initializations
           assert not self.deep_stem, 'ResNet_CIFAR do not support deep_stem'

       def _make_stem_layer(self, in_channels, base_channels):
           # override the ResNet method to modify the network structure
           self.conv1 = build_conv_layer(
               self.conv_cfg,
               in_channels,
               base_channels,
               kernel_size=3,
               stride=1,
               padding=1,
               bias=False)
           self.norm1_name, norm1 = build_norm_layer(
               self.norm_cfg, base_channels, postfix=1)

stride=1,
           self.add_module(self.norm1_name, norm1)
           self.relu = nn.ReLU(inplace=True)

       def forward(self, x):
           # Customize the forward method if needed.
           x = self.conv1(x)
           x = self.norm1(x)
           x = self.relu(x)
           outs = []
           for i, layer_name in enumerate(self.res_layers):
               res_layer = getattr(self, layer_name)
               x = res_layer(x)
               if i in self.out_indices:
                   outs.append(x)
           # The return value needs to be a tuple with multi-scale outputs from different depths.
           # If you don't need multi-scale features, just wrap the output as a one-item tuple.
           return tuple(outs)

       def init_weights(self):

# If you don't need multi-scale features, just wrap the output as a one-item tuple.

           # Customize the weight initialization method if needed.
           super().init_weights()

           # Disable the weight initialization if loading a pretrained model.
           if self.init_cfg is not None and self.init_cfg['type'] == 'Pretrained':
               return

           # Usually, we recommend using `init_cfg` to specify weight initialization methods
           # of convolution, linear, or normalization layers. If you have some special needs,
           # do these extra weight initialization here.
           ...
   ```

```{note}
Replace original registry names from `BACKBONES`, `NECKS`, `HEADS` and `LOSSES` to `MODELS` in OpenMMLab 2.0 design.
```

2. Import the new backbone module in `mmpretrain/models/backbones/__init__.py`.

   ```python
   ...
   from .resnet_cifar import ResNet_CIFAR

   __all__ = [

```

   ```python
   ...

   __all__ = [
       ..., 'ResNet_CIFAR'
   ]
   ```

3. Modify the correlated settings in your config file.

   ```python
       ...
           type='ResNet_CIFAR',
           depth=18,
           ...),
       ...
   ```

### Add a new backbone for self-supervised learning

For some self-supervised learning algorithms, the backbones are kind of different, such as `MAE`, `BEiT`, etc. Their backbones need to deal with `mask` in order to extract features from visible tokens.

Take [MAEViT](mmpretrain.models.selfsup.MAEViT) as an example, we need to overwrite `forward` function to compute with `mask`. We also defines `init_weights` to initialize parameters and `random_masking` to generate mask for `MAE` pre-training.

```python
class MAEViT(VisionTransformer):

```python
    """Vision Transformer for MAE pre-training"""

    def __init__(mask_ratio, **kwargs) -> None:
        # position embedding is not learnable during pretraining
        self.pos_embed.requires_grad = False
        self.mask_ratio = mask_ratio
        self.num_patches = self.patch_resolution[0] * self.patch_resolution[1]

    def init_weights(self) -> None:
        """Initialize position embedding, patch embedding and cls token."""
        super().init_weights()
        # define what if needed
        pass

    def random_masking(
        self,
        x: torch.Tensor,
        mask_ratio: float = 0.75
    ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
        """Generate the mask for MAE Pre-training."""
        pass

    def forward(
        self,
        mask: Optional[bool] = True

pass

        self,
        """Generate features for masked images.

        The function supports two kind of forward behaviors. If the ``mask`` is
        ``True``, the function will generate mask to masking some patches
        randomly and get the hidden features for visible patches, which means
        the function will be executed as masked imagemodeling pre-training;
        if the ``mask`` is ``None`` or ``False``, the forward function will
        call ``super().forward()``, which extract features from images without
        mask.
        """
        if mask is None or False:
            return super().forward(x)

        else:
            B = x.shape[0]
            x = self.patch_embed(x)[0]
            # add pos embed w/o cls token
            x = x + self.pos_embed[:, 1:, :]

            # masking: length -> length * mask_ratio

x = self.patch_embed(x)[0]

            x, mask, ids_restore = self.random_masking(x, self.mask_ratio)

            # append cls token
            cls_token = self.cls_token + self.pos_embed[:, :1, :]
            cls_tokens = cls_token.expand(B, -1, -1)
            x = torch.cat((cls_tokens, x), dim=1)

            for _, layer in enumerate(self.layers):
                x = layer(x)
            # Use final norm
            x = self.norm1(x)

            return (x, mask, ids_restore)

```

## Add a new neck

Here we take `GlobalAveragePooling` as an example. It is a very simple neck without any arguments.
To add a new neck, we mainly implement the `forward` function, which applies some operations on the output from the backbone and forwards the results to the head.

1. Create a new file in `mmpretrain/models/necks/gap.py`.

   ```python

   ```python

   class GlobalAveragePooling(nn.Module):

       def __init__(self):
           self.gap = nn.AdaptiveAvgPool2d((1, 1))

       def forward(self, inputs):
           # we regard inputs as tensor for simplicity
           outs = self.gap(inputs)
           outs = outs.view(inputs.size(0), -1)
           return outs
   ```

2. Import the new neck module in `mmpretrain/models/necks/__init__.py`.

   ```python
   ...
   from .gap import GlobalAveragePooling

   __all__ = [
       ..., 'GlobalAveragePooling'
   ]
   ```

   ```python
       neck=dict(type='GlobalAveragePooling'),
   )
   ```

## Add a new head

### Based on ClsHead

Here we present how to develop a new head by the example of simplified `VisionTransformerClsHead` as the following.

)
   ```

To implement a new head, we need to implement a `pre_logits` method for processes before the final classification head and a `forward` method.

:::{admonition} Why do we need the `pre_logits` method?
:class: note

In classification tasks, we usually use a linear layer to do the final classification. And sometimes, we need
to obtain the feature before the final classification, which is the output of the `pre_logits` method.
:::

1. Create a new file in `mmpretrain/models/heads/vit_head.py`.

   ```python

   from .cls_head import ClsHead

   class VisionTransformerClsHead(ClsHead):

       def __init__(self, num_classes, in_channels, hidden_dim, **kwargs):

           self.in_channels = in_channels
           self.num_classes = num_classes
           self.hidden_dim = hidden_dim

           self.fc1 = nn.Linear(in_channels, hidden_dim)
           self.act = nn.Tanh()
           self.fc2 = nn.Linear(hidden_dim, num_classes)

       def pre_logits(self, feats):
           # The output of the backbone is usually a tuple from multiple depths,
           # and for classification, we only need the final output.
           feat = feats[-1]

           # The final output of VisionTransformer is a tuple of patch tokens and
           # classification tokens. We need classification tokens here.
           _, cls_token = feat

           # Do all works except the final classification linear layer.
           return self.act(self.fc1(cls_token))

_, cls_token = feat

       def forward(self, feats):
           pre_logits = self.pre_logits(feats)

           # The final classification linear layer.
           cls_score = self.fc2(pre_logits)
           return cls_score
   ```

2. Import the module in `mmpretrain/models/heads/__init__.py`.

   ```python
   ...
   from .vit_head import VisionTransformerClsHead

   __all__ = [
       ..., 'VisionTransformerClsHead'
   ]
   ```

   ```python
       head=dict(
           type='VisionTransformerClsHead',
           ...,
       ))
   ```

### Based on BaseModule

Here is an example of `MAEPretrainHead`, which is based on `BaseModule` and implemented for mask image modeling task. It is required to implement `loss` function to generate loss, but the other helper functions are optional.

```python

from mmpretrain.registry import MODELS

class MAEPretrainHead(BaseModule):
    """Head for MAE Pre-training."""

                 loss: dict,
                 norm_pix: bool = False,
                 patch_size: int = 16) -> None:
        self.norm_pix = norm_pix
        self.patch_size = patch_size
        self.loss_module = MODELS.build(loss)

    def patchify(self, imgs: torch.Tensor) -> torch.Tensor:
        """Split images into non-overlapped patches."""
        p = self.patch_size
        assert imgs.shape[2] == imgs.shape[3] and imgs.shape[2] % p == 0

        h = w = imgs.shape[2] // p
        x = imgs.reshape(shape=(imgs.shape[0], 3, h, p, w, p))
        x = torch.einsum('nchpwq->nhwpqc', x)
        x = x.reshape(shape=(imgs.shape[0], h * w, p**2 * 3))
        return x

x = imgs.reshape(shape=(imgs.shape[0], 3, h, p, w, p))
        return x

    def construct_target(self, target: torch.Tensor) -> torch.Tensor:
        """Construct the reconstruction target."""
        target = self.patchify(target)
        if self.norm_pix:
            # normalize the target image
            mean = target.mean(dim=-1, keepdim=True)
            var = target.var(dim=-1, keepdim=True)
            target = (target - mean) / (var + 1.e-6)**.5

        return target

    def loss(self, pred: torch.Tensor, target: torch.Tensor,
             mask: torch.Tensor) -> torch.Tensor:
        """Generate loss."""
        target = self.construct_target(target)
        loss = self.loss_module(pred, target, mask)

```

After implementation, the following step is the same as the step-2 and step-3 in [Based on ClsHead](#based-on-clshead)

loss = self.loss_module(pred, target, mask)

```

## Add a new loss

To add a new loss function, we mainly implement the `forward` function in the loss module. We should register the loss module as `MODELS` as well.
In addition, it is helpful to leverage the decorator `weighted_loss` to weight the loss for each element.
Assuming that we want to mimic a probabilistic distribution generated from another classification model, we implement an L1Loss to fulfill the purpose as below.

1. Create a new file in `mmpretrain/models/losses/l1_loss.py`.

   ```python
   import torch

   from .utils import weighted_loss

   @weighted_loss
   def l1_loss(pred, target):
       assert pred.size() == target.size() and target.numel() > 0
       loss = torch.abs(pred - target)

       return loss

   class L1Loss(nn.Module):

       def __init__(self, reduction='mean', loss_weight=1.0):
           super(L1Loss, self).__init__()
           self.reduction = reduction
           self.loss_weight = loss_weight

       def forward(self,
                   pred,
                   target,
                   weight=None,
                   avg_factor=None,
                   reduction_override=None):
           assert reduction_override in (None, 'none', 'mean', 'sum')
           reduction = (
               reduction_override if reduction_override else self.reduction)
           loss = self.loss_weight * l1_loss(
               pred, target, weight, reduction=reduction, avg_factor=avg_factor)
           return loss
   ```

loss = self.loss_weight * l1_loss(
   ```

2. Import the module in `mmpretrain/models/losses/__init__.py`.

   ```python
   ...
   from .l1_loss import L1Loss

   __all__ = [
       ..., 'L1Loss'
   ]
   ```

3. Modify loss field in the head configs.

   ```python
           loss=dict(type='L1Loss', loss_weight=1.0),
       ))
   ```

Finally, we can combine all the new model components in a config file to create a new model for best practices. Because `ResNet_CIFAR` is not a ViT-based backbone, we do not implement `VisionTransformerClsHead` here.

```python
    type='ImageClassifier',
        type='ResNet_CIFAR',
        depth=18,
        out_indices=(3, ),
        style='pytorch'),
        type='LinearClsHead',

depth=18,
        loss=dict(type='L1Loss', loss_weight=1.0),
        topk=(1, 5),
    ))

```

```{tip}
For convenience, the same model components could inherit from existing config files, refers to [Learn about configs](../user_guides/config.md) for more details.
```

# TRN

[Temporal Relational Reasoning in Videos](https://openaccess.thecvf.com/content_ECCV_2018/html/Bolei_Zhou_Temporal_Relational_Reasoning_ECCV_2018_paper.html)

Temporal relational reasoning, the ability to link meaningful transformations of objects or entities over time, is a fundamental property of intelligent species. In this paper, we introduce an effective and interpretable network module, the Temporal Relation Network (TRN), designed to learn and reason about temporal dependencies between video frames at multiple time scales. We evaluate TRN-equipped networks on activity recognition tasks using three recent video datasets - Something-Something, Jester, and Charades - which fundamentally depend on temporal relational reasoning. Our results demonstrate that the proposed TRN gives convolutional neural networks a remarkable capacity to discover temporal relations in videos. Through only sparsely sampled video frames, TRN-equipped networks can accurately predict human-object interactions in the Something-Something dataset and identify various human gestures on the Jester dataset with very competitive performance. TRN-equipped networks also outperform two-stream networks and 3D convolution networks in recognizing daily activities in the Charades dataset. Further analyses show that the models learn intuitive and interpretable visual common sense knowledge in videos.

<img src="https://user-images.githubusercontent.com/34324155/143018998-d2120c3d-a9a7-4e4c-90b1-1e5ff1fd5f06.png" width="800"/>
</div>

### Something-Something V1

| frame sampling strategy | resolution | gpus | backbone | pretrain | top1 acc (efficient/accurate) | top5 acc (efficient/accurate) |  testing protocol  | FLOPs  | params |        config         |        ckpt         |         log         |
| :---------------------: | :--------: | :--: | :------: | :------: | :---------------------------: | :---------------------------: | :----------------: | :----: | :----: | :-------------------: | :-----------------: | :-----------------: |

|          1x1x8          |  224x224   |  8   | ResNet50 | ImageNet |         31.60 / 33.65         |         60.15 / 62.22         | 16 clips x 10 crop | 42.94G | 26.64M | [config](/configs/recognition/trn/trn_imagenet-pretrained-r50_8xb16-1x1x8-50e_sthv1-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/trn/trn_imagenet-pretrained-r50_8xb16-1x1x8-50e_sthv1-rgb/trn_imagenet-pretrained-r50_8xb16-1x1x8-50e_sthv1-rgb_20220815-e13db2e9.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/trn/trn_imagenet-pretrained-r50_8xb16-1x1x8-50e_sthv1-rgb/trn_imagenet-pretrained-r50_8xb16-1x1x8-50e_sthv1-rgb.log) |

|          1x1x8          |  224x224   |  8   | ResNet50 | ImageNet |         47.65 / 51.20         |         76.27 / 78.42         | 16 clips x 10 crop | 42.94G | 26.64M | [config](/configs/recognition/trn/trn_imagenet-pretrained-r50_8xb16-1x1x8-50e_sthv2-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/trn/trn_imagenet-pretrained-r50_8xb16-1x1x8-50e_sthv2-rgb/trn_imagenet-pretrained-r50_8xb16-1x1x8-50e_sthv2-rgb_20220815-e01617db.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/trn/trn_imagenet-pretrained-r50_8xb16-1x1x8-50e_sthv2-rgb/trn_imagenet-pretrained-r50_8xb16-1x1x8-50e_sthv2-rgb.log) |

2. There are two kinds of test settings for Something-Something dataset, efficient setting (center crop only) and accurate setting (three crop and `twice_sample`).
3. In the original [repository](https://github.com/zhoubolei/TRN-pytorch), the author augments data with random flipping on something-something dataset, but the augmentation method may be wrong due to the direct actions, such as `push left to right`. So, we replaced `flip` with `flip with label mapping`, and change the testing method `TenCrop`, which has five flipped crops, to `Twice Sample & ThreeCrop`.

4. We use `ResNet50` instead of `BNInception` as the backbone of TRN. When Training `TRN-ResNet50` on sthv1 dataset in the original repository, we get top1 (top5) accuracy 30.542 (58.627) vs. ours 31.81 (60.47).

For more details on data preparation, you can refer to [Something-something V1](/tools/data/sthv1/README.md) and [Something-something V2](/tools/data/sthv2/README.md).

## Train

```shell
```

Example: train TRN model on sthv1 dataset in a deterministic option with periodic validation.

```shell
python tools/train.py configs/recognition/trn/trn_imagenet-pretrained-r50_8xb16-1x1x8-50e_sthv1-rgb.py \
```

## Test

```shell

## Test

```shell
```

Example: test TRN model on sthv1 dataset and dump the result to a pkl file.

```shell
python tools/test.py configs/recognition/trn/trn_imagenet-pretrained-r50_8xb16-1x1x8-50e_sthv1-rgb.py \
```

```BibTeX
@article{zhou2017temporalrelation,
    title = {Temporal Relational Reasoning in Videos},
    author = {Zhou, Bolei and Andonian, Alex and Oliva, Aude and Torralba, Antonio},
    journal={European Conference on Computer Vision},
    year={2018}
}
```

# Rotated FCOS

</div>

We propose a fully convolutional one-stage object detector (FCOS) to solve object detection in a per-pixel prediction
fashion, analogue to semantic segmentation. Almost all state-of-the-art object detectors such as RetinaNet, SSD, YOLOv3,
and Faster R-CNN rely on pre-defined anchor boxes. In contrast, our proposed detector FCOS is anchor box free, as well
as proposal free. By eliminating the predefined set of anchor boxes, FCOS completely avoids the complicated computation
related to anchor boxes such as calculating overlapping during training. More importantly, we also avoid all
hyper-parameters related to anchor boxes, which are often very sensitive to the final detection performance. With the

only post-processing non-maximum suppression (NMS), FCOS with ResNeXt-64x4d-101 achieves 44.7% in AP with single-model
and single-scale testing, surpassing previous one-stage detectors with the advantage of being much simpler. For the
first time, we demonstrate a much simpler and flexible detection framework achieving improved detection accuracy. We
hope that the proposed FCOS framework can serve as a simple and strong alternative for many other instance-level tasks.

DOTA1.0

DOTA1.0

|         Backbone         |  mAP  | Angle | Separate Angle | Tricks | lr schd | Mem (GB) | Inf Time (fps) | Aug | Batch Size |                                                     Configs                                                     |                                                                                                                                                                                   Download                                                                                                                                                                                   |

| ResNet50 (1024,1024,200) | 70.70 | le90  |       Y        |   Y    |   1x    |   4.18   |      26.4      |  -  |     2      |              [rotated-fcos-hbox-le90_r50_fpn_1x_dota](./rotated-fcos-hbox-le90_r50_fpn_1x_dota.py)              |       [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_sep_angle_r50_fpn_1x_dota_le90/rotated_fcos_sep_angle_r50_fpn_1x_dota_le90-0be71a0c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_sep_angle_r50_fpn_1x_dota_le90/rotated_fcos_sep_angle_r50_fpn_1x_dota_le90_20220409_023250.log.json)       |

| ResNet50 (1024,1024,200) | 71.28 | le90  |       N        |   Y    |   1x    |   4.18   |      25.9      |  -  |     2      |                   [rotated-fcos-le90_r50_fpn_1x_dota](./rotated-fcos-le90_r50_fpn_1x_dota.py)                   |                           [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_r50_fpn_1x_dota_le90/rotated_fcos_r50_fpn_1x_dota_le90-d87568ed.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_r50_fpn_1x_dota_le90/rotated_fcos_r50_fpn_1x_dota_le90_20220413_163526.log.json)                           |

| ResNet50 (1024,1024,200) | 71.76 | le90  |       Y        |   Y    |   1x    |   4.23   |      25.7      |  -  |     2      | [rotated-fcos-hbox-le90_r50_fpn_csl-gaussian_1x_dota](./rotated-fcos-hbox-le90_r50_fpn_csl-gaussian_1x_dota.py) | [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_csl_gaussian_r50_fpn_1x_dota_le90/rotated_fcos_csl_gaussian_r50_fpn_1x_dota_le90-4e044ad2.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_csl_gaussian_r50_fpn_1x_dota_le90/rotated_fcos_csl_gaussian_r50_fpn_1x_dota_le90_20220409_080616.log.json) |

| ResNet50 (1024,1024,200) | 71.89 | le90  |       N        |   Y    |   1x    |   4.18   |      26.2      |  -  |     2      |               [rotated-fcos-le90_r50_fpn_kld_1x_dota](./rotated-fcos-le90_r50_fpn_kld_1x_dota.py)               |                   [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_kld_r50_fpn_1x_dota_le90/rotated_fcos_kld_r50_fpn_1x_dota_le90-ecafdb2b.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_kld_r50_fpn_1x_dota_le90/rotated_fcos_kld_r50_fpn_1x_dota_le90_20220409_202939.log.json)                   |

- `Rotated IoU Loss` need mmcv version 1.5.0 or above.
- `Separate Angle` means angle loss is calculated separately.
  At this time bbox loss uses horizontal bbox loss such as `IoULoss`, `GIoULoss`.

At this time bbox loss uses horizontal bbox loss such as `IoULoss`, `GIoULoss`.
- Inf time was tested on a single RTX3090.

```
}
```

# ncnn
python wrapper of ncnn with [pybind11](https://github.com/pybind/pybind11), only support python3.x now.

Install from pip
==================

ncnn is available as wheel packages for macOS, Windows and Linux distributions, you can install with pip:

```
python -m pip install -U pip
python -m pip install -U ncnn
```

# Build from source

If you want to build ncnn with some options not as default, or just like to build everything yourself, it is not difficult to build ncnn from source.

**On Unix (Linux, OS X)**

* A compiler with C++11 support
* CMake >= 3.4

**On Mac**

**On Windows**

* Visual Studio 2015 or higher

## Build
1. clone ncnn and init submodule.
```bash
cd /pathto/ncnn
git submodule init && git submodule update
```
2. build.
```bash
cd build
cmake -DNCNN_PYTHON=ON ..
make
```

## Install
```bash
pip install .
```

```bash
```
2. build.
```bash
cd build
make
```

## Install
```bash
```

if you use conda or miniconda, you can also install as following:
```bash
python3 setup.py install
```

## Tests
**test**
```bash
cd /pathto/ncnn/python
python3 tests/test.py
```

**benchmark**

```bash
python3 tests/benchmark.py
```

## Numpy
**ncnn.Mat->numpy.array, with no memory copy**

```bash
mat = ncnn.Mat(...)
mat_np = np.array(mat)
```

**numpy.array->ncnn.Mat, with no memory copy**
```bash
mat_np = np.array(...)
mat = ncnn.Mat(mat_np)
```

# Model Zoo
install requirements
```bash
```
then you can import ncnn.model_zoo and get model list as follow:
```bash
import ncnn
import ncnn.model_zoo as model_zoo

print(model_zoo.get_model_list())
```
models now in model zoo are as list below:
```bash
mobilenet_yolov2

```bash

```
```bash
mobilenetv2_yolov3
yolov4_tiny
yolov4
yolov5s
yolact
mobilenet_ssd
squeezenet_ssd
mobilenetv2_ssdlite
mobilenetv3_ssdlite
faster_rcnn
peleenet_ssd
retinaface
rfcn
shufflenetv2
simplepose
nanodet
```
all model in model zoo has example in ncnn/python/examples folder

# Custom Layer

custom layer demo is in ncnn/python/ncnn/model_zoo/yolov5.py:23

# SABL

> [Side-Aware Boundary Localization for More Precise Object Detection](https://arxiv.org/abs/1912.04260)

Current object detection frameworks mainly rely on bounding box regression to localize objects. Despite the remarkable progress in recent years, the precision of bounding box regression remains unsatisfactory, hence limiting performance in object detection. We observe that precise localization requires careful placement of each side of the bounding box. However, the mainstream approach, which focuses on predicting centers and sizes, is not the most effective way to accomplish this task, especially when there exists displacements with large variance between the anchors and the targets. In this paper, we propose an alternative approach, named as Side-Aware Boundary Localization (SABL), where each side of the bounding box is respectively localized with a dedicated network branch. To tackle the difficulty of precise localization in the presence of displacements with large variance, we further propose a two-step localization scheme, which first predicts a range of movement through bucket prediction and then pinpoints the precise position within the predicted bucket. We test the proposed method on both two-stage and single-stage detection frameworks. Replacing the standard bounding box regression branch with the proposed design leads to significant improvements on Faster R-CNN, RetinaNet, and Cascade R-CNN, by 3.0%, 1.7%, and 0.9%, respectively.

<img src="https://user-images.githubusercontent.com/40661020/143973698-3dfaea91-4415-4818-9781-5017183e7489.png"/>
</div>

Single-scale testing (1333x800) is adopted in all results.

|       Method       | Backbone  | Lr schd | ms-train | box AP |                      Config                       |                                                                                                                                  Download                                                                                                                                   |

| :----------------: | :-------: | :-----: | :------: | :----: | :-----------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| SABL Faster R-CNN  | R-50-FPN  |   1x    |    N     |  39.9  |  [config](./sabl-faster-rcnn_r50_fpn_1x_coco.py)  |    [model](https://download.openmmlab.com/mmdetection/v2.0/sabl/sabl_faster_rcnn_r50_fpn_1x_coco/sabl_faster_rcnn_r50_fpn_1x_coco-e867595b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/sabl/sabl_faster_rcnn_r50_fpn_1x_coco/20200830_130324.log.json)    |

| SABL Faster R-CNN  | R-101-FPN |   1x    |    N     |  41.7  | [config](./sabl-faster-rcnn_r101_fpn_1x_coco.py)  |  [model](https://download.openmmlab.com/mmdetection/v2.0/sabl/sabl_faster_rcnn_r101_fpn_1x_coco/sabl_faster_rcnn_r101_fpn_1x_coco-f804c6c1.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/sabl/sabl_faster_rcnn_r101_fpn_1x_coco/20200830_183949.log.json)   |
| SABL Cascade R-CNN | R-50-FPN  |   1x    |    N     |  41.6  | [config](./sabl-cascade-rcnn_r50_fpn_1x_coco.py)  |  [model](https://download.openmmlab.com/mmdetection/v2.0/sabl/sabl_cascade_rcnn_r50_fpn_1x_coco/sabl_cascade_rcnn_r50_fpn_1x_coco-e1748e5e.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/sabl/sabl_cascade_rcnn_r50_fpn_1x_coco/20200831_033726.log.json)   |

| SABL Cascade R-CNN | R-101-FPN |   1x    |    N     |  43.0  | [config](./sabl-cascade-rcnn_r101_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/sabl/sabl_cascade_rcnn_r101_fpn_1x_coco/sabl_cascade_rcnn_r101_fpn_1x_coco-2b83e87c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/sabl/sabl_cascade_rcnn_r101_fpn_1x_coco/20200831_141745.log.json) |

|     Method     | Backbone  | GN  | Lr schd |  ms-train   | box AP |                            Config                            |                                                                                                                                                   Download                                                                                                                                                   |

| :------------: | :-------: | :-: | :-----: | :---------: | :----: | :----------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| SABL RetinaNet | R-50-FPN  |  N  |   1x    |      N      |  37.7  |        [config](./sabl-retinanet_r50_fpn_1x_coco.py)         |                       [model](https://download.openmmlab.com/mmdetection/v2.0/sabl/sabl_retinanet_r50_fpn_1x_coco/sabl_retinanet_r50_fpn_1x_coco-6c54fd4f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/sabl/sabl_retinanet_r50_fpn_1x_coco/20200830_053451.log.json)                        |

| SABL RetinaNet | R-50-FPN  |  Y  |   1x    |      N      |  38.8  |       [config](./sabl-retinanet_r50-gn_fpn_1x_coco.py)       |                   [model](https://download.openmmlab.com/mmdetection/v2.0/sabl/sabl_retinanet_r50_fpn_gn_1x_coco/sabl_retinanet_r50_fpn_gn_1x_coco-e16dfcf1.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/sabl/sabl_retinanet_r50_fpn_gn_1x_coco/20200831_141955.log.json)                   |
| SABL RetinaNet | R-101-FPN |  N  |   1x    |      N      |  39.7  |        [config](./sabl-retinanet_r101_fpn_1x_coco.py)        |                      [model](https://download.openmmlab.com/mmdetection/v2.0/sabl/sabl_retinanet_r101_fpn_1x_coco/sabl_retinanet_r101_fpn_1x_coco-42026904.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/sabl/sabl_retinanet_r101_fpn_1x_coco/20200831_034256.log.json)                      |

| SABL RetinaNet | R-101-FPN |  Y  |   1x    |      N      |  40.5  |      [config](./sabl-retinanet_r101-gn_fpn_1x_coco.py)       |                 [model](https://download.openmmlab.com/mmdetection/v2.0/sabl/sabl_retinanet_r101_fpn_gn_1x_coco/sabl_retinanet_r101_fpn_gn_1x_coco-40a893e8.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/sabl/sabl_retinanet_r101_fpn_gn_1x_coco/20200830_201422.log.json)                  |
| SABL RetinaNet | R-101-FPN |  Y  |   2x    | Y (640~800) |  42.9  | [config](./sabl-retinanet_r101-gn_fpn_ms-640-800-2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/sabl/sabl_retinanet_r101_fpn_gn_2x_ms_640_800_coco/sabl_retinanet_r101_fpn_gn_2x_ms_640_800_coco-1e63382c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/sabl/sabl_retinanet_r101_fpn_gn_2x_ms_640_800_coco/20200830_144807.log.json) |

| SABL RetinaNet | R-101-FPN |  Y  |   2x    | Y (480~960) |  43.6  | [config](./sabl-retinanet_r101-gn_fpn_ms-480-960-2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/sabl/sabl_retinanet_r101_fpn_gn_2x_ms_480_960_coco/sabl_retinanet_r101_fpn_gn_2x_ms_480_960_coco-5342f857.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/sabl/sabl_retinanet_r101_fpn_gn_2x_ms_480_960_coco/20200830_164537.log.json) |

We provide config files to reproduce the object detection results in the ECCV 2020 Spotlight paper for [Side-Aware Boundary Localization for More Precise Object Detection](https://arxiv.org/abs/1912.04260).

```latex
@inproceedings{Wang_2020_ECCV,
    title = {Side-Aware Boundary Localization for More Precise Object Detection},
    author = {Jiaqi Wang and Wenwei Zhang and Yuhang Cao and Kai Chen and Jiangmiao Pang and Tao Gong and Jianping Shi and Chen Change Loy and Dahua Lin},
    booktitle = {ECCV},
}
```

# 准备 Moments in Time

## 简介

```BibTeX
@article{monfortmoments,
    title={Moments in Time Dataset: one million videos for event understanding},
    author={Monfort, Mathew and Andonian, Alex and Zhou, Bolei and Ramakrishnan, Kandan and Bargal, Sarah Adel and Yan, Tom and Brown, Lisa and Fan, Quanfu and Gutfruend, Dan and Vondrick, Carl and others},
    journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
    issn={0162-8828},
    pages={1--8},
    numpages={8},
    doi={10.1109/TPAMI.2019.2901464},
}
```

用户可以参照数据集 [官网](http://moments.csail.mit.edu/)，获取数据集相关的基本信息。
在准备数据集前，请确保命令行当前路径为 `$MMACTION2/tools/data/mit/`。

## 步骤 1. 准备标注文件和视频文件

首先，用户需要访问[官网](http://moments.csail.mit.edu/)，填写申请表来下载数据集。
在得到下载链接后，用户可以使用 `bash preprocess_data.sh` 来准备标注文件和视频。
请注意此脚本并没有下载标注和视频文件，用户需要根据脚本文件中的注释，提前下载好数据集，并放/软链接到合适的位置。

为加快视频解码速度，用户需要缩小原视频的尺寸，可使用以下命令获取密集编码版视频：

```shell

```shell
python ../resize_videos.py ../../../data/mit/videos/ ../../../data/mit/videos_256p_dense_cache --dense --level 2
```

## Step 2. 抽取帧和光流

如果用户有大量的 SSD 存储空间，则推荐将抽取的帧存储至 I/O 性能更优秀的 SSD 上。
用户可使用以下命令为 SSD 建立软链接。

```shell
# 执行这两行指令进行抽取（假设 SSD 挂载在 "/mnt/SSD/"上）
mkdir /mnt/SSD/mit_extracted/
ln -s /mnt/SSD/mit_extracted/ ../../../data/mit/rawframes
```

```shell
```

```shell
```

```shell
```

## 步骤 3. 生成文件列表

```shell
```

```shell
```

```shell
```

## 步骤 4. 检查目录结构

在完成 Moments in Time 数据集准备流程后，用户可以得到 Moments in Time 的 RGB 帧 + 光流文件，视频文件以及标注文件。

在整个 MMAction2 文件夹下，Moments in Time 的文件结构如下：

```
mmaction2
├── data
│   └── mit
│       ├── annotations
│       │   ├── license.txt
│       │   ├── moments_categories.txt
│       │   ├── README.txt
│       │   ├── trainingSet.csv
│       │   └── validationSet.csv
│       ├── mit_train_rawframe_anno.txt
│       ├── mit_train_video_anno.txt
│       ├── mit_val_rawframe_anno.txt
│       ├── mit_val_video_anno.txt
│       ├── rawframes
│       │   ├── training
│       │   │   ├── adult+female+singing
│       │   │   │   ├── 0P3XG_vf91c_35
│       │   │   │   │   ├── flow_x_00001.jpg
│       │   │   │   │   ├── flow_x_00002.jpg
│       │   │   │   │   ├── ...

│       │   │   │   │   ├── flow_y_00001.jpg
│       │   │   │   │   ├── flow_y_00002.jpg
│       │   │   │   │   ├── img_00001.jpg
│       │   │   │   │   └── img_00002.jpg
│       │   │   │   └── yt-zxQfALnTdfc_56
│       │   │   └── yawning
│       │   │       ├── _8zmP1e-EjU_2
│       │   │       │   ├── ...
│       │   └── validation
│       │   │       ├── ...
│       └── videos
│           ├── training
│           │   ├── adult+female+singing
│           │   │   ├── 0P3XG_vf91c_35.mp4
│           │   │   ├── ...
│           │   │   └── yt-zxQfALnTdfc_56.mp4
│           │   └── yawning
│           │       ├── ...
│           └── validation
│           │   ├── ...
└── mmaction
└── ...

```

│           │       ├── ...
│           │   ├── ...
└── ...

```

关于对 Moments in Times 进行训练和验证，可以参照 [训练教程](/docs/zh_cn/user_guides/train_test.md)。

# Build from Script

Through user investigation, we know that most users are already familiar with python and torch before using mmdeploy. Therefore we provide scripts to simplify mmdeploy installation.

Assuming you already have

- python3 -m pip (`conda` or `pyenv`)
- nvcc (depends on inference backend)
- torch (not compulsory)

run this script to install mmdeploy + ncnn backend, `nproc` is not compulsory.

```bash
$ python3 tools/scripts/build_ubuntu_x64_ncnn.py
..
```

A sudo password may be required during this time, and the script will try its best to build and install mmdeploy SDK and demo:

- Detect host OS version, `make` job number, whether use `root` and try to fix `python3 -m pip`
- Find the necessary basic tools, such as g++-7, cmake, wget, etc.
- Compile necessary dependencies, such as pyncnn, protobuf

The script will also try to avoid affecting host environment:

- The dependencies of source code compilation are placed in the `mmdeploy-dep` directory at the same level as mmdeploy
- The script would not modify variables such as PATH, LD_LIBRARY_PATH, PYTHONPATH, etc.
- The environment variables that need to be modified will be printed, **please pay attention to the final output**

The script will eventually execute `python3 tools/check_env.py`, the successful installation should display the version number of the corresponding backend and `ops_is_available: True`, for example:

```bash
$ python3 tools/check_env.py
..
2022-09-13 14:49:13,767 - mmdeploy - INFO - **********Backend information**********
2022-09-13 14:49:14,116 - mmdeploy - INFO - onnxruntime: 1.8.0	ops_is_avaliable : True

..
2022-09-13 14:49:14,131 - mmdeploy - INFO - tensorrt: 8.4.1.5	ops_is_avaliable : True
2022-09-13 14:49:14,139 - mmdeploy - INFO - ncnn: 1.0.20220901	ops_is_avaliable : True
2022-09-13 14:49:14,150 - mmdeploy - INFO - pplnn_is_avaliable: True
..
```

Here is the verified installation script. If you want mmdeploy to support multiple backends at the same time, you can execute each script once:

|             script              |     OS version      |
| :-----------------------------: | :-----------------: |
|    build_ubuntu_x64_ncnn.py     |     18.04/20.04     |
|     build_ubuntu_x64_ort.py     |     18.04/20.04     |
|    build_ubuntu_x64_pplnn.py    |     18.04/20.04     |
| build_ubuntu_x64_torchscript.py |     18.04/20.04     |
|     build_ubuntu_x64_tvm.py     |     18.04/20.04     |

|  build_jetson_orin_python38.sh  | JetPack5.0 L4T 34.1 |

# 学习配置文件 (待更新)

我们在配置文件中支持了继承和模块化，这便于进行各种实验。
mmrotate 是建立在 [mmdet](https://github.com/open-mmlab/mmdetection) 之上的，
因此强烈建议学习 [mmdet](https://mmdetection.readthedocs.io/en/latest/) 的基本知识。

当运行 `tools/train.py` 或者 `tools/test.py` 时，可以通过 `--cfg-options` 来修改配置。

- 更新字典链的配置

  可以按照原始配置文件中的 dict 键顺序地指定配置预选项。
  例如，使用 `--cfg-options model.backbone.norm_eval=False` 将模型主干网络中的所有 BN 模块都改为 `train` 模式。

  在配置文件里，一些字典型的配置被包含在列表中。例如，数据训练流程 `data.train.pipeline` 通常是一个列表，比如  `[dict(type='LoadImageFromFile'), ...]`。 如果需要将 `'LoadImageFromFile'` 改成 `'LoadImageFromWebcam'` ，需要写成下述形式： `--cfg-options data.train.pipeline.0.type=LoadImageFromWebcam`。

  如果要更新的值是列表或元组。例如，配置文件通常设置 `workflow=[('train', 1)]`，如果需要改变这个键，可以通过 `--cfg-options workflow="[(train,1),(val,1)]"` 来重新设置。需要注意，引号 " 是支持列表或元组数据类型所必需的，并且在指定值的引号内**不允许**有空格。

```text

```text
```

`{xxx}` 是被要求的文件 `[yyy]` 是可选的。

- `{model}`： 模型种类，例如 `rotated_faster_rcnn`, `rotated_retinanet` 等。
- `[model setting]`： 特定的模型，例如 `hbb` for `rotated_retinanet` 等。
- `{backbone}`： 主干网络种类例如 `r50` (ResNet-50), `swin_tiny` (SWIN-tiny) 。
- `{neck}`： Neck 模型的种类包括 `fpn`,  `refpn`。
- `[norm_setting]`： 默认使用 `bn` (Batch Normalization)，其他指定可以有 `gn` (Group Normalization)， `syncbn` (Synchronized Batch Normalization) 等。 `gn-head`/`gn-neck` 表示 GN 仅应用于网络的 Head 或 Neck， `gn-all` 表示 GN 用于整个模型， 例如主干网络、Neck 和 Head。
- `[misc]`： 模型中各式各样的设置/插件，例如 `dconv`、 `gcb`、 `attention`、`albu`、 `mstrain` 等。
- `[gpu x batch_per_gpu]`： GPU 数量和每个 GPU 的样本数，默认使用 `1xb2`。
- `{dataset}`：数据集，例如 `dota`。
- `{angle version}`：旋转定义方式，例如 `oc`, `le135` 或者 `le90`。

## RotatedRetinaNet 配置文件示例

为了帮助用户对 MMRotate 检测系统中的完整配置和模块有一个基本的了解

我们对使用 ResNet50 和 FPN 的 RotatedRetinaNet 的配置文件进行简要注释说明。更详细的用法和各个模块对应的替代方案，请参考 API 文档。

```python
angle_version = 'oc'  # 旋转定义方式
    type='RotatedRetinaNet',  # 检测器(detector)名称
        type='ResNet',  # # 主干网络的类别
        depth=50,  # 主干网络的深度
        num_stages=4,  # 主干网络阶段(stages)的数目
        out_indices=(0, 1, 2, 3),  # 每个阶段产生的特征图输出的索引
        frozen_stages=1,  # 第一个阶段的权重被冻结
        zero_init_residual=False,  # 是否对残差块(resblocks)中的最后一个归一化层使用零初始化(zero init)让它们表现为自身
        style='pytorch',  # 主干网络的风格，'pytorch' 意思是步长为2的层为 3x3 卷积， 'caffe' 意思是步长为2的层为 1x1 卷积。

        type='FPN',  # 检测器的 neck 是 FPN， 我们同样支持 'ReFPN'
        start_level=1,  # 用于构建特征金字塔的主干网络起始输入层索引值
        add_extra_convs='on_input',  # 决定是否在原始特征图之上添加卷积层
        num_outs=5),  # 决定输出多少个尺度的特征图(scales)
        type='RotatedRetinaHead',# bbox_head 的类型是 'RRetinaHead'
        num_classes=15,  # 分类的类别数量
        in_channels=256,  # bbox head 输入通道数
        stacked_convs=4,  # head 卷积层的层数
        assign_by_circumhbbox='oc',  # obb2hbb 的旋转定义方式

stacked_convs=4,  # head 卷积层的层数
            type='RotatedAnchorGenerator',  # 锚点生成器类别
            octave_base_scale=4,  # RetinaNet 用于生成锚点的超参数，特征图 anchor 的基本尺度。值越大，所有 anchor 的尺度都会变大。
            scales_per_octave=3,  #  RetinaNet 用于生成锚点的超参数，每个特征图有3个尺度
            ratios=[1.0, 0.5, 2.0],  # 高度和宽度之间的比率
            strides=[8, 16, 32, 64, 128]),  # 锚生成器的步幅。这与 FPN 特征步幅一致。如果未设置 base_sizes，则当前步幅值将被视为 base_sizes。
            type='DeltaXYWHAOBBoxCoder',  # 框编码器的类别
            angle_range='oc',  # 框编码器的旋转定义方式
            norm_factor=None,  # 框编码器的范数
            edge_swap=False,  # 设置是否启用框编码器的边缘交换
            proj_xy=False,  # 设置是否启用框编码器的投影
            target_means=(0.0, 0.0, 0.0, 0.0, 0.0),  # 用于编码和解码框的目标均值
            target_stds=(1.0, 1.0, 1.0, 1.0, 1.0)),  # 用于编码和解码框的标准差

proj_xy=False,  # 设置是否启用框编码器的投影
            type='FocalLoss',  # 分类分支的损失函数类型
            use_sigmoid=True,  #  是否使用 sigmoid
            gamma=2.0,  # Focal Loss 用于解决难易不均衡的参数 gamma
            alpha=0.25,  # Focal Loss 用于解决样本数量不均衡的参数 alpha
            loss_weight=1.0),  # 分类分支的损失权重
            type='L1Loss',  # 回归分支的损失类型
    train_cfg=dict(  # 训练超参数的配置
        assigner=dict(  # 分配器(assigner)的配置
            type='MaxIoUAssigner',  # 分配器的类型
            pos_iou_thr=0.5,  # IoU >= 0.5(阈值) 被视为正样本
            neg_iou_thr=0.4,  # IoU < 0.4(阈值) 被视为负样本
            min_pos_iou=0,  # 将框作为正样本的最小 IoU 阈值
            ignore_iof_thr=-1,  # 忽略 bbox 的 IoF 阈值
            iou_calculator=dict(type='RBboxOverlaps2D')),  # IoU 的计算器类型

min_pos_iou=0,  # 将框作为正样本的最小 IoU 阈值
        allowed_border=-1,  # 填充有效锚点(anchor)后允许的边框
        pos_weight=-1,  # 训练期间正样本的权重
        debug=False),  # 是否设置调试(debug)模式
    test_cfg=dict(  # 测试超参数的配置
        nms_pre=2000,  # NMS 前的 box 数
        min_bbox_size=0,  # box 允许的最小尺寸
        score_thr=0.05,  # bbox 的分数阈值
        nms=dict(iou_thr=0.1), # NMS 的阈值
        max_per_img=2000))  # 每张图像的最大检测次数
dataset_type = 'DOTADataset'  # 数据集类型，这将被用来定义数据集
data_root = '../datasets/split_1024_dota1_0/'  # 数据的根路径
img_norm_cfg = dict(  # 图像归一化配置，用来归一化输入的图像
    mean=[123.675, 116.28, 103.53],  # 预训练里用于预训练主干网络模型的平均值
    std=[58.395, 57.12, 57.375],  # 预训练里用于预训练主干网络模型的标准差
    to_rgb=True)  # 预训练里用于预训练主干网络的图像的通道顺序
    dict(type='LoadImageFromFile'),  # 第 1 个流程，从文件路径里加载图像
    dict(type='LoadAnnotations',  # 第 2 个流程，对于当前图像，加载它的注释信息

to_rgb=True)  # 预训练里用于预训练主干网络的图像的通道顺序
         with_bbox=True),  # 是否加载标注框(bounding box)， 目标检测需要设置为 True
    dict(type='RResize',  # 变化图像和其注释大小的数据增广的流程
         img_scale=(1024, 1024)),  # 图像的最大规模
    dict(type='RRandomFlip',  # 翻转图像和其注释大小的数据增广的流程
         flip_ratio=0.5,  # 翻转图像的概率
         version='oc'),  # 定义旋转的方式
    dict(
        type='Normalize',  # 归一化当前图像的数据增广的流程
        mean=[123.675, 116.28, 103.53],  # 这些键与 img_norm_cfg 一致，
        std=[58.395, 57.12, 57.375],  # 因为 img_norm_cfg 被用作参数
    dict(type='Pad',  # 填充当前图像到指定大小的数据增广的流程
         size_divisor=32),  # 填充图像可以被当前值整除
    dict(type='DefaultFormatBundle'),  # 流程里收集数据的默认格式包
    dict(type='Collect',  # 决定数据中哪些键应该传递给检测器的流程
]
test_pipeline = [ # 测试流程

dict(type='Collect',  # 决定数据中哪些键应该传递给检测器的流程
]
    dict(
        type='MultiScaleFlipAug',  # 封装测试时数据增广(test time augmentations)
        img_scale=(1024, 1024),  # 决定测试时可改变图像的最大规模。用于改变图像大小的流程
        flip=False,  # 测试时是否翻转图像
            dict(type='RResize'),  # 使用改变图像大小的数据增广
            dict(
                type='Normalize',  # 归一化配置项，值来自 img_norm_cfg
            dict(type='Pad',  # 将配置传递给可被 32 整除的图像
            dict(type='DefaultFormatBundle'),  # 用于在管道中收集数据的默认格式包
            dict(type='Collect',  # 收集测试时必须的键的收集流程
        ])
]
    samples_per_gpu=2,  # 单个 GPU 的 Batch size
    workers_per_gpu=2,  # 单个 GPU 分配的数据加载线程数

dict(type='Collect',  # 收集测试时必须的键的收集流程
        ])
]
    train=dict(  # 训练数据集配置
        type='DOTADataset',  # 数据集的类别
        ann_file=
        '../datasets/split_1024_dota1_0/trainval/annfiles/',  # 注释文件路径
        '../datasets/split_1024_dota1_0/trainval/images/',  # 图片路径前缀
        pipeline=[  # 流程, 这是由之前创建的 train_pipeline 传递的
            dict(

        ],
    val=dict(  # 验证数据集的配置
        ann_file=
            dict(
                    dict(

                ])
        ],
    test=dict(  # 测试数据集配置，修改测试开发/测试(test-dev/test)提交的 ann_file
        ann_file=
        pipeline=[  # 由之前创建的 test_pipeline 传递的流程
            dict(
                    dict(

img_scale=(1024, 1024),
                    dict(
                ])
        ],
evaluation = dict(  # evaluation hook 的配置
    interval=12,  # 验证的间隔
    metric='mAP')  # 验证期间使用的指标
optimizer = dict(  # 用于构建优化器的配置文件
    type='SGD',  # 优化器类型
    lr=0.0025,  # 优化器的学习率
    momentum=0.9,  # 动量(Momentum)
    weight_decay=0.0001)  # SGD 的衰减权重(weight decay)
optimizer_config = dict(  # optimizer hook 的配置文件

momentum=0.9,  # 动量(Momentum)
lr_config = dict(  # 学习率调整配置，用于注册 LrUpdater hook
    policy='step',  # 调度流程(scheduler)的策略
    warmup='linear',  # 预热(warmup)策略，也支持 `exp` 和 `constant`
    warmup_iters=500,  # 预热的迭代次数
    warmup_ratio=0.3333333333333333,  # 用于预热的起始学习率的比率
    step=[8, 11])  # 衰减学习率的起止回合数
    type='EpochBasedRunner',  # 将使用的 runner 的类别 (例如 IterBasedRunner 或 EpochBasedRunner)
    max_epochs=12) # runner 总回合(epoch)数， 对于 IterBasedRunner 使用 `max_iters`
checkpoint_config = dict(  # checkpoint hook 的配置文件
    interval=12)  # 保存的间隔是 12
log_config = dict(  # register logger hook 的配置文件
    interval=50,  # 打印日志的间隔
    hooks=[
        # dict(type='TensorboardLoggerHook')  # 同样支持 Tensorboard 日志
    ])  # 用于记录训练过程的记录器(logger)

interval=50,  # 打印日志的间隔
    hooks=[
dist_params = dict(backend='nccl')  # 用于设置分布式训练的参数，端口也同样可被设置
log_level = 'INFO'  # 日志的级别
load_from = None  # 从一个给定路径里加载模型作为预训练模型，它并不会消耗训练时间
resume_from = None  # 从给定路径里恢复检查点(checkpoints)，训练模式将从检查点保存的轮次开始恢复训练。
workflow = [('train', 1)]  # runner 的工作流程，[('train', 1)] 表示只有一个工作流且工作流仅执行一次。根据 total_epochs 工作流训练 12 个回合(epoch)。
work_dir = './work_dirs/rotated_retinanet_hbb_r50_fpn_1x_dota_oc'  # 用于保存当前实验的模型检查点(checkpoints)和日志的目录
```

## 常见问题 (FAQ)

配置文件里会使用一些中间变量，例如数据集里的 `train_pipeline`/`test_pipeline`。
值得注意的是，在修改子配置中的中间变量时，需要再次将中间变量传递到相应的字段中。
例如，我们想使用离线多尺度策略 (multi scale strategy)来训练 RoI-Trans。 `train_pipeline` 是我们想要修改的中间变量。

```python

```python

    dict(
    dict(
]

dict(type='DefaultFormatBundle'),
]
    val=dict(
```

同样的，如果我们想从 `SyncBN` 切换到 `BN` 或者 `MMSyncBN`，我们需要修改配置文件里的每一个 `norm_cfg`。

```python
    ...)
```

# 自定义训练优化策略

在我们的算法库中，已经提供了通用数据集（如ImageNet，CIFAR）的[默认训练策略配置](https://github.com/open-mmlab/mmpretrain/blob/main/configs/_base_/schedules)。如果想要在这些数据集上继续提升模型性能，或者在不同数据集和方法上进行新的尝试，我们通常需要修改这些默认的策略。

在本教程中，我们将介绍如何在运行自定义训练时，通过修改配置文件进行构造优化器、参数化精细配置、梯度裁剪、梯度累计以及定制动量调整策略等。同时也会通过模板简单介绍如何自定义开发优化器和构造器。

## 配置训练优化策略

我们通过 `optim_wrapper` 来配置主要的优化策略，包括优化器的选择，混合精度训练的选择，参数化精细配置，梯度裁剪以及梯度累计。接下来将分别介绍这些内容。

### 构造 PyTorch 内置优化器

MMPretrain 支持 PyTorch 实现的所有优化器，仅需在配置文件中，指定优化器封装需要的 `optimizer` 字段。

如果要使用 [`SGD`](torch.optim.SGD)，则修改如下。这里要注意所有优化相关的配置都需要封装在 `optim_wrapper` 配置里。

```python
)
```

```{note}
配置文件中的 'type' 不是构造时的参数，而是 PyTorch 内置优化器的类名。
更多优化器选择可以参考{external+torch:ref}`PyTorch 支持的优化器列表<optim:algorithms>`。
```

要修改模型的学习率，只需要在优化器的配置中修改 `lr` 即可。
要配置其他参数，可直接根据 [PyTorch API 文档](torch.optim) 进行。

```

例如，如果想使用 [`Adam`](torch.optim.Adam) 并设置参数为 `torch.optim.Adam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False)`。
则需要进行如下修改：

```python
        lr=0.001,
)
```

考虑到对于单精度训练来说，优化器封装的默认类型就是 `OptimWrapper`，我们在这里可以直接省略，因此配置文件可以进一步简化为：

```python
        lr=0.001,
        amsgrad=False))
```
````

### 混合精度训练

如果我们想要使用混合精度训练（Automactic Mixed Precision），我们只需简单地将 `optim_wrapper` 的类型改为 `AmpOptimWrapper`。

```python

eps=1e-08,
```
````

### 混合精度训练

```python
optim_wrapper = dict(type='AmpOptimWrapper', optimizer=...)
```

另外，为了方便，我们同时在启动训练脚本 `tools/train.py` 中提供了 `--amp` 参数作为开启混合精度训练的开关，更多细节可以参考[训练教程](../user_guides/train.md)。

在一些模型中，不同的优化策略需要适应特定的参数，例如不在 BatchNorm 层使用权重衰减，或者在不同层使用不同的学习率等等。
我们需要用到 `optim_wrapper` 中的 `paramwise_cfg` 参数来进行精细化配置。

- **为不同类型的参数设置超参乘子**

  例如，我们可以在 `paramwise_cfg` 配置中设置 `norm_decay_mult=0.` 来改变归一化层权重和偏移的衰减为0。

  ```python
  ```

  支持更多类型的参数配置，参考以下列表：

  - `bias_lr_mult`：偏置的学习率系数（不包括正则化层的偏置以及可变形卷积的 offset），默认值为 1
  - `bias_decay_mult`：偏置的权值衰减系数（不包括正则化层的偏置以及可变形卷积的 offset），默认值为 1
  - `norm_decay_mult`：正则化层权重和偏置的权值衰减系数，默认值为 1
  - `flat_decay_mult`: 一维参数的权值衰减系数，默认值为 1

- `bias_decay_mult`：偏置的权值衰减系数（不包括正则化层的偏置以及可变形卷积的 offset），默认值为 1
  - `dwconv_decay_mult`：Depth-wise 卷积的权值衰减系数，默认值为 1
  - `bypass_duplicate`：是否跳过重复的参数，默认为 `False`
  - `dcn_offset_lr_mult`：可变形卷积（Deformable Convolution）的学习率系数，默认值为 1

- **为特定参数设置超参乘子**

  MMPretrain 通过 `paramwise_cfg` 的 `custom_keys` 参数来配置特定参数的超参乘子。

  例如，我们可以通过以下配置来设置所有 `backbone.layer0` 层的学习率和权重衰减为0， `backbone` 的其余层和优化器保持一致，另外 `head` 层的学习率为0.001.

  ```python
          }))
  ```

### 梯度裁剪

在训练过程中，损失函数可能接近于一些异常陡峭的区域，从而导致梯度爆炸。而梯度裁剪可以帮助稳定训练过程，更多介绍可以参见[该页面](https://paperswithcode.com/method/gradient-clipping)。

'head': dict(lr_mult=0.1)
          }))
  ```

### 梯度裁剪

目前我们支持在 `optim_wrapper` 字段中添加 `clip_grad` 参数来进行梯度裁剪，更详细的参数可参考 [PyTorch 文档](torch.nn.utils.clip_grad_norm_)。

用例如下：

```python
    # norm_type: 使用的范数类型，此处使用范数2。
```

### 梯度累计

计算资源缺乏缺乏时，每个训练批次的大小（batch size）只能设置为较小的值，这可能会影响模型的性能。

可以使用梯度累计来规避这一问题。我们支持在 `optim_wrapper` 字段中添加 `accumulative_counts` 参数来进行梯度累计。

用例如下：

```python
```

表示训练时，每 4 个 iter 执行一次反向传播。由于此时单张 GPU 上的批次大小为 64，也就等价于单张 GPU 上一次迭代的批次大小为 256，也即：

```python

accumulative_counts=4)
```

```python
```

## 配置参数优化策略

在训练过程中，优化参数例如学习率、动量，通常不会是固定不变，而是随着训练进程的变化而调整。PyTorch 支持一些学习率调整的调度器，但是不足以完成复杂的策略。在 MMPretrain 中，我们提供 `param_scheduler` 来更好地控制不同优化参数的策略。

深度学习研究中，广泛应用学习率衰减来提高网络的性能。我们支持大多数 PyTorch 学习率调度器， 其中包括 `ExponentialLR`, `LinearLR`, `StepLR`, `MultiStepLR` 等等。

- **单个学习率策略**

  多数情况下，我们使用单一学习率策略，这里 `param_scheduler` 会是一个字典。比如在默认的 ResNet 网络训练中，我们使用阶梯式的学习率衰减策略 [`MultiStepLR`](mmengine.optim.MultiStepLR)，配置文件为：

  ```python
  ```

  或者我们想使用 [`CosineAnnealingLR`](mmengine.optim.CosineAnnealingLR) 来进行学习率衰减：

  ```python

gamma=0.1)
  ```

  ```python
  ```

  然而在一些其他情况下，为了提高模型的精度，通常会使用多种学习率策略。例如，在训练的早期阶段，网络容易不稳定，而学习率的预热就是为了减少这种不稳定性。

  整个学习过程中，学习率将会通过预热从一个很小的值逐步提高到预定值，再会通过其他的策略进一步调整。

  在 MMPretrain 中，我们同样使用 `param_scheduler` ，将多种学习策略写成列表就可以完成上述预热策略的组合。

  例如：

  1. 在前50次迭代中逐**迭代次数**地**线性**预热

  ```python
        # 逐迭代次数，线性预热
            by_epoch=False,  # 逐迭代次数
            end=50),  # 只预热50次迭代次数
        # 主要的学习率策略
    ]
  ```

  2. 在前10轮迭代中逐**迭代次数**地**线性**预热

  ```python
        # 在前10轮迭代中，逐迭代次数，线性预热

    ]
  ```

  ```python
            end=10,
            convert_to_iter_based=True,  # 逐迭代次数更新学习率.
        ),
        # 在 10 轮次后，通过余弦退火衰减
    ]
  ```

  注意这里增加了 `begin` 和 `end` 参数，这两个参数指定了调度器的**生效区间**。生效区间通常只在多个调度器组合时才需要去设置，使用单个调度器时可以忽略。当指定了 `begin` 和 `end` 参数时，表示该调度器只在 [begin, end) 区间内生效，其单位是由 `by_epoch` 参数决定。在组合不同调度器时，各调度器的 `by_epoch` 参数不必相同。如果没有指定的情况下，`begin` 为 0， `end` 为最大迭代轮次或者最大迭代次数。

  如果相邻两个调度器的生效区间没有紧邻，而是有一段区间没有被覆盖，那么这段区间的学习率维持不变。而如果两个调度器的生效区间发生了重叠，则对多组调度器叠加使用，学习率的调整会按照调度器配置文件中的顺序触发（行为与 PyTorch 中 [`ChainedScheduler`](torch.optim.lr_scheduler.ChainedScheduler) 一致）。

  ```{tip}
  为了避免学习率曲线与预期不符， 配置完成后，可以使用 MMPretrain 提供的 [学习率可视化工具](../useful_tools/scheduler_visualization.md) 画出对应学习率调整曲线。
  ```

### 配置动量调整策略

```{tip}
  ```

MMPretrain 支持动量调度器根据学习率修改优化器的动量，从而使损失函数收敛更快。用法和学习率调度器一致。

我们支持的动量策略和详细的使用细节可以参考[这里](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/momentum_scheduler.py)。我们只将调度器中的 `LR` 替换为了 `Momentum`，动量策略可以直接追加 `param_scheduler` 列表中。

这里是一个用例：

```python
    # 学习率策略
    # 动量策略
         begin=0,
         end=1000)
]
```

```{note}
本部分将修改 MMPretrain 源码或者向 MMPretrain 框架添加代码，初学者可跳过。
```

### 新增优化器

在学术研究和工业实践中，可能需要使用 MMPretrain 未实现的优化方法，可以通过以下方法添加。

1. 定义一个新的优化器

   一个自定义的优化器可根据如下规则进行定制：

   假设我们想添加一个名为 `MyOptimzer` 的优化器，其拥有参数 `a`, `b` 和 `c`。
   可以创建一个名为 `mmpretrain/engine/optimizer` 的文件夹，并在目录下的一个文件，如 `mmpretrain/engine/optimizer/my_optimizer.py` 中实现该自定义优化器：

   ```python

假设我们想添加一个名为 `MyOptimzer` 的优化器，其拥有参数 `a`, `b` 和 `c`。

   ```python
   from mmpretrain.registry import OPTIMIZERS
   from torch.optim import Optimizer

   @OPTIMIZERS.register_module()
   class MyOptimizer(Optimizer):

       def __init__(self, a, b, c):
           ...

       def step(self, closure=None):
           ...
   ```

2. 注册优化器

   要注册上面定义的上述模块，首先需要将此模块导入到主命名空间中。有两种方法可以实现它。

   修改 `mmpretrain/engine/optimizers/__init__.py`，将其导入至 `mmpretrain.engine` 包。

   ```python
   # 在 mmpretrain/engine/optimizers/__init__.py 中
   ...
   from .my_optimizer import MyOptimizer # MyOptimizer 是我们自定义的优化器的名字

   __all__ = [..., 'MyOptimizer']
   ```

   在运行过程中，我们会自动导入 `mmpretrain.engine` 包并同时注册 `MyOptimizer`。

3. 在配置文件中指定优化器

   之后，用户便可在配置文件的 `optim_wrapper.optimizer` 域中使用 `MyOptimizer`：

   ```python

```

   ```python
       optimizer=dict(type='MyOptimizer', a=a_value, b=b_value, c=c_value))
   ```

### 新增优化器构造器

某些模型可能具有一些特定于参数的设置以进行优化，例如为所有 BatchNorm 层设置不同的权重衰减。

尽管我们已经可以使用 [`optim_wrapper.paramwise_cfg` 字段](#参数化精细配置)来配置特定参数的优化设置，但可能仍然无法覆盖你的需求。

当然你可以在此基础上进行修改。我们默认使用 [`DefaultOptimWrapperConstructor`](mmengine.optim.DefaultOptimWrapperConstructor) 来构造优化器。在构造过程中，通过 `paramwise_cfg` 来精细化配置不同设置。这个默认构造器可以作为新优化器构造器实现的模板。

我们可以新增一个优化器构造器来覆盖这些行为。

```python
# 在 mmpretrain/engine/optimizers/my_optim_constructor.py 中
from mmpretrain.registry import OPTIM_WRAPPER_CONSTRUCTORS

        ...

        ...

        ...
```

这是一个已实现的 [OptimWrapperConstructor](mmpretrain.engine.optimizers.LearningRateDecayOptimWrapperConstructor) 具体例子。

接下来类似 [新增优化器教程](#新增优化器) 来导入并使用新的优化器构造器。

1. 修改 `mmpretrain/engine/optimizers/__init__.py`，将其导入至 `mmpretrain.engine` 包。

   ```python
   ...

   ```

2. 在配置文件的 `optim_wrapper.constructor` 字段中使用 `MyOptimWrapperConstructor` 。

   ```python
   )
   ```

> **任务**: 视频插帧

<details>
<summary align="right"><a href="https://aaai.org/ojs/index.php/AAAI/article/view/6693/6547">CAIN (AAAI'2020)</a></summary>

```bibtex
}
```

<br/>

学习率调整策略是等间隔调整策略。

|                                  算法                                   | vimeo-90k-triplet |         GPU 信息         |                                           下载                                           |

| :---------------------------------------------------------------------: | :---------------: | :----------------------: | :--------------------------------------------------------------------------------------: |
| [cain_b5_g1b32_vimeo90k_triplet](./cain_g1b32_1xb5_vimeo90k-triplet.py) | 34.6010 / 0.9578  | 1 (Tesla V100-SXM2-32GB) | [模型](https://download.openmmlab.com/mmediting/video_interpolators/cain/cain_b5_g1b32_vimeo90k_triplet_20220530-3520b00c.pth)/[日志](https://download.openmmlab.com/mmediting/video_interpolators/cain/cain_b5_g1b32_vimeo90k_triplet_20220530-3520b00c.log.json) |

## 快速开始

**训练**

<details>

```shell
# CPU上训练

# 单个GPU上训练

# 多个GPU上训练
```

# 单个GPU上训练

# 多个GPU上训练
```

**测试**

<details>

```shell
# CPU上测试

# 单个GPU上测试

# 多个GPU上测试
```

```

# SlowFast

</div>

### AVA2.1

| frame sampling strategy | gpus |             backbone             |   pretrain   |  mAP  |                   config                    |                   ckpt                    |                   log                    |
| :---------------------: | :--: | :------------------------------: | :----------: | :---: | :-----------------------------------------: | :---------------------------------------: | :--------------------------------------: |

|         4x16x1          |  8   |        SlowFast ResNet50         | Kinetics-400 | 24.32 | [config](/configs/detection/slowfast/slowfast_kinetics400-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowfast/slowfast_kinetics400-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb/slowfast_kinetics400-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb_20220906-5180ea3c.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowfast/slowfast_kinetics400-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb/slowfast_kinetics400-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb.log) |

|         4x16x1          |  8   | SlowFast ResNet50 (with context) | Kinetics-400 | 25.34 | [config](/configs/detection/slowfast/slowfast_kinetics400-pretrained-r50-context_8xb16-4x16x1-20e_ava21-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowfast/slowfast_kinetics400-pretrained-r50-context_8xb16-4x16x1-20e_ava21-rgb/slowfast_kinetics400-pretrained-r50-context_8xb16-4x16x1-20e_ava21-rgb_20220906-5bb4f6f2.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowfast/slowfast_kinetics400-pretrained-r50-context_8xb16-4x16x1-20e_ava21-rgb/slowfast_kinetics400-pretrained-r50-context_8xb16-4x16x1-20e_ava21-rgb.log) |

|          8x8x1          |  8   |        SlowFast ResNet50         | Kinetics-400 | 25.80 | [config](/configs/detection/slowfast/slowfast_kinetics400-pretrained-r50_8xb8-8x8x1-20e_ava21-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowfast/slowfast_kinetics400-pretrained-r50_8xb8-8x8x1-20e_ava21-rgb/slowfast_kinetics400-pretrained-r50_8xb8-8x8x1-20e_ava21-rgb_20220906-39133ec7.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowfast/slowfast_kinetics400-pretrained-r50_8xb8-8x8x1-20e_ava21-rgb/slowfast_kinetics400-pretrained-r50_8xb8-8x8x1-20e_ava21-rgb.log) |

### AVA2.2

| frame sampling strategy | gpus |                 backbone                  |   pretrain   |  mAP  |                  config                  |                  ckpt                  |                  log                  |

| :---------------------: | :--: | :---------------------------------------: | :----------: | :---: | :--------------------------------------: | :------------------------------------: | :-----------------------------------: |
|          8x8x1          |  8   |             SlowFast ResNet50             | Kinetics-400 | 25.90 | [config](/configs/detection/slowfast/slowfast_kinetics400-pretrained-r50_8xb6-8x8x1-cosine-10e_ava22-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowfast/slowfast_kinetics400-pretrained-r50_8xb6-8x8x1-cosine-10e_ava22-rgb/slowfast_kinetics400-pretrained-r50_8xb6-8x8x1-cosine-10e_ava22-rgb_20220906-d934a48f.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowfast/slowfast_kinetics400-pretrained-r50_8xb6-8x8x1-cosine-10e_ava22-rgb/slowfast_kinetics400-pretrained-r50_8xb6-8x8x1-cosine-10e_ava22-rgb.log) |

|          8x8x1          |  8   |     SlowFast ResNet50 (temporal-max)      | Kinetics-400 | 26.41 | [config](/configs/detection/slowfast/slowfast_kinetics400-pretrained-r50-temporal-max_8xb6-8x8x1-cosine-10e_ava22-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowfast/slowfast_kinetics400-pretrained-r50-temporal-max_8xb6-8x8x1-cosine-10e_ava22-rgb/slowfast_kinetics400-pretrained-r50-temporal-max_8xb6-8x8x1-cosine-10e_ava22-rgb_20220906-13a9078e.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowfast/slowfast_kinetics400-pretrained-r50-temporal-max_8xb6-8x8x1-cosine-10e_ava22-rgb/slowfast_kinetics400-pretrained-r50-temporal-max_8xb6-8x8x1-cosine-10e_ava22-rgb.log) |

|          8x8x1          |  8   | SlowFast ResNet50 (temporal-max, focal loss) | Kinetics-400 | 26.65 | [config](/configs/detection/slowfast/slowfast_r50-k400-pre-temporal-max-focal-alpha3-gamma1_8xb6-8x8x1-cosine-10e_ava22-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowfast/slowfast_r50-k400-pre-temporal-max-focal-alpha3-gamma1_8xb6-8x8x1-cosine-10e_ava22-rgb/slowfast_r50-k400-pre-temporal-max-focal-alpha3-gamma1_8xb6-8x8x1-cosine-10e_ava22-rgb_20220906-dd59e26f.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowfast/slowfast_r50-k400-pre-temporal-max-focal-alpha3-gamma1_8xb6-8x8x1-cosine-10e_ava22-rgb/slowfast_r50-k400-pre-temporal-max-focal-alpha3-gamma1_8xb6-8x8x1-cosine-10e_ava22-rgb.log) |

|         4x16x1          |  8   | SlowFast ResNet50 | Kinetics-400 | 36.88 |   22.83   |   16.9    |     14.74     |   18618    | [config](/configs/detection/slowfast/slowfast_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/slowfast/slowfast_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb/slowfast_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb_20230320-af666368.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/slowfast/slowfast_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb/slowfast_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb.log) |

2. **with context** indicates that using both RoI feature and global pooled feature for classification; **temporal-max** indicates that using max pooling in the temporal dimension for the feature.

## Train

```shell
```

Example: train the SlowFast model on AVA2.1 in a deterministic option with periodic validation.

```shell
python tools/train.py configs/detection/slowfast/slowfast_kinetics400-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb.py \
```

## Test

```shell
```

Example: test the SlowFast model on AVA2.1 and dump the result to a pkl file.

```shell

```shell
```

```shell
python tools/test.py configs/detection/slowfast/slowfast_kinetics400-pretrained-r50_8xb16-4x16x1-20e_ava21-rgb.py \
```

```BibTeX
}
```

```BibTeX

}
```

```BibTeX
}
```

# Human-Art: A Versatile Human-Centric Dataset Bridging Natural and Artificial Scenes

<details>

```bibtex
```

# Wandb Feature Map Visualization

MMSegmentation 1.x provides backend support for Weights & Biases to facilitate visualization and management of project code results.

## Wandb Configuration

Install Weights & Biases following [official instructions](https://docs.wandb.ai/quickstart) e.g.

```shell
pip install wandb
wandb login
```

Add `WandbVisBackend` in `vis_backend` of `visualizer` in `default_runtime.py` config file:

```python
vis_backends=[dict(type='LocalVisBackend'),
              dict(type='TensorboardVisBackend'),
              dict(type='WandbVisBackend')]
```

## Examining feature map visualization in Wandb

`SegLocalVisualizer` is child class inherits from `Visualizer` in MMEngine and works for MMSegmentation visualization, for more details about `Visualizer` please refer to [visualization tutorial](https://github.com/open-mmlab/mmengine/blob/main/docs/en/advanced_tutorials/visualization.md) in MMEngine.

Here is an example about `SegLocalVisualizer`, first you may download example data below by following commands:

</div>

```shell

wget https://download.openmmlab.com/mmsegmentation/v0.5/ann/ann_r50-d8_512x1024_40k_cityscapes/ann_r50-d8_512x1024_40k_cityscapes_20200605_095211-049fc292.pth

```

```python
from typing import Type

from mmengine.model import revert_sync_batchnorm

from mmseg.apis import inference_model, init_model
from mmseg.utils import register_all_modules

class Recorder:
    """record the forward output feature map and save to data_buffer."""

        self.data_buffer = list()

    def __enter__(self, ):
        self._data_buffer = list()

    def record_data_hook(self, model: nn.Module, input: Type, output: Type):
        self.data_buffer.append(output)

    def __exit__(self, *args, **kwargs):
        pass

def visualize(args, model, recorder, result):
    seg_visualizer = SegLocalVisualizer(
        vis_backends=[dict(type='WandbVisBackend')],
        save_dir='temp_dir',
        alpha=0.5)

    seg_visualizer.dataset_meta = dict(
        classes=model.dataset_meta['classes'],
        palette=model.dataset_meta['palette'])

    image = mmcv.imread(args.img, 'color')

    seg_visualizer.add_datasample(
        name='predict',
        image=image,
        draw_pred=True,
        wait_time=0,
        out_file=None,

    # add feature map to wandb visualizer
    for i in range(len(recorder.data_buffer)):
        feature = recorder.data_buffer[i][0]  # remove the batch
        drawn_img = seg_visualizer.draw_featmap(
            feature, image, channel_reduction='select_max')
        seg_visualizer.add_image(f'feature_map{i}', drawn_img)

    if args.gt_mask:
        sem_seg = mmcv.imread(args.gt_mask, 'unchanged')

seg_visualizer.add_image(f'feature_map{i}', drawn_img)

        sem_seg = torch.from_numpy(sem_seg)
        gt_mask = dict(data=sem_seg)
        gt_mask = PixelData(**gt_mask)
        data_sample = SegDataSample()
        data_sample.gt_sem_seg = gt_mask

        seg_visualizer.add_datasample(
            name='gt_mask',
            image=image,
            data_sample=data_sample,
            draw_gt=True,
            draw_pred=False,
            wait_time=0,
            out_file=None,
            show=False)

    seg_visualizer.add_image('image', image)

    parser = ArgumentParser(
        description='Draw the Feature Map During Inference')
    parser.add_argument('img', help='Image file')
    parser.add_argument('config', help='Config file')
    parser.add_argument('checkpoint', help='Checkpoint file')
    parser.add_argument('--gt_mask', default=None, help='Path of gt mask file')

parser.add_argument('config', help='Config file')
    parser.add_argument('--out-file', default=None, help='Path to output file')
        '--device', default='cuda:0', help='Device used for inference')
        '--opacity',
        type=float,
        default=0.5,
        help='Opacity of painted segmentation map. In (0, 1] range.')
        '--title', default='result', help='The image identifier.')

    register_all_modules()

    # build the model from a config file and a checkpoint file
    model = init_model(args.config, args.checkpoint, device=args.device)
    if args.device == 'cpu':
        model = revert_sync_batchnorm(model)

    # show all named module in the model and use it in source list below

if args.device == 'cpu':

    for name, module in model.named_modules():
        print(name)

    source = [
        'decode_head.fusion.stages.0.query_project.activate',
        'decode_head.context.stages.0.key_project.activate',
        'decode_head.context.bottleneck.activate'
    ]
    source = dict.fromkeys(source)

    count = 0
    recorder = Recorder()
    # registry the forward hook
        if name in source:
            count += 1
            module.register_forward_hook(recorder.record_data_hook)
            if count == len(source):
                break

    with recorder:
        # test a single image, and record feature map to data_buffer
        result = inference_model(model, args.img)

    visualize(args, model, recorder, result)

    main()

```

result = inference_model(model, args.img)

    main()

```

Save the above code as feature_map_visual.py and execute the  following code in terminal

```shell
python feature_map_visual.py ${image} ${config} ${checkpoint} [optional args]
```

e.g

```shell
python feature_map_visual.py \
aachen_000000_000019_leftImg8bit.png \
configs/ann/ann_r50-d8_4xb2-40k_cityscapes-512x1024.py \
ann_r50-d8_512x1024_40k_cityscapes_20200605_095211-049fc292.pth \
--gt_mask aachen_000000_000019_gtFine_labelTrainIds.png
```

The visualized image result and its corresponding feature map will appear in the wandb account.

<img src="https://user-images.githubusercontent.com/76149310/217520321-647f5bf9-eef2-446d-a9e8-5ca7b621d500.png">
</div>

<details>
<summary align="right">ESRGAN (ECCVW'2018)</summary>

```bibtex
}
```

<br/>

|                                  算法                                   |       Set5        |      Set14       |      DIV2K       | GPU 信息 |                                   下载                                   |

| :---------------------------------------------------------------------: | :---------------: | :--------------: | :--------------: | :------: | :----------------------------------------------------------------------: |
| [esrgan_psnr_x4c64b23g32_1x16_1000k_div2k](./esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py) | 30.6428 / 0.8559  | 27.0543 / 0.7447 | 29.3354 / 0.8263 |    1     | [模型](https://download.openmmlab.com/mmediting/restorers/esrgan/esrgan_psnr_x4c64b23g32_1x16_1000k_div2k_20200420-bf5c993c.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/esrgan/esrgan_psnr_x4c64b23g32_1x16_1000k_div2k_20200420_112550.log.json) |

| [esrgan_x4c64b23g32_1x16_400k_div2k](./esrgan_x4c64b23g32_1xb16-400k_div2k.py) | 28.2700 /  0.7778 | 24.6328 / 0.6491 | 26.6531 / 0.7340 |    1     | [模型](https://download.openmmlab.com/mmediting/restorers/esrgan/esrgan_x4c64b23g32_1x16_400k_div2k_20200508-f8ccaf3b.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/esrgan/esrgan_x4c64b23g32_1x16_400k_div2k_20200508_191042.log.json) |

## 快速开始

**训练**

<details>

```shell
# CPU上训练

# 单个GPU上训练

# 多个GPU上训练
```

**测试**

<details>

```shell
# CPU上测试

```

**测试**

<details>

```shell
# CPU上测试

# 单个GPU上测试

# 多个GPU上测试
```

# 一键式脚本安装

通过用户调研，我们得知多数使用者在了解 mmdeploy 前，已经熟知 python 和 torch 用法。因此我们提供脚本简化 mmdeploy 安装。

假设您已经准备好

- python3 -m pip（必须，conda 或 pyenv 皆可）
- nvcc（取决于推理后端）
- torch（非必须，可延后安装）

运行这个脚本来安装 mmdeploy + ncnn backend，`nproc` 可以不指定。

```bash
..
```

期间可能需要 sudo 密码，脚本会尽最大努力完成 mmdeploy SDK 和 demo：

- 检测系统版本、make 使用的 job 个数、是否 root 用户，也会自动修复 pip 问题
- 寻找必须的基础工具，如 g++-7、cmake、wget 等
- 编译必须的依赖，如 pyncnn、 protobuf

脚本也会尽量避免影响 host 环境：

- 源码编译的依赖，都放在与 mmdeploy 同级的 `mmdeploy-dep` 目录中
- 不会主动修改 PATH、LD_LIBRARY_PATH、PYTHONPATH 等变量
- 会打印需要修改的环境变量，**需要注意最终的输出信息**

脚本最终会执行 `python3 tools/check_env.py`，安装成功应显示对应 backend 的版本号和 `ops_is_available: True`，例如：

```bash
..

..
..
```

这是已验证的安装脚本。如果想让 mmdeploy 同时支持多种 backend，每个脚本执行一次即可：

# PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection

> [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://arxiv.org/abs/1912.13192)

3D object detection has been receiving increasing attention from both industry and academia thanks to its wide applications in various fields such as autonomous driving and robotics. LiDAR sensors are widely adopted in autonomous driving vehicles and robots for capturing 3D scene information as sparse and irregular point clouds, which provide vital cues for 3D scene perception and understanding. In this paper, we propose to achieve high performance 3D object detection by designing novel point-voxel integrated networks to learn better 3D features from irregular point clouds.

<img src="https://user-images.githubusercontent.com/88368822/202114244-ccf52f56-b8c9-4f1b-9cc2-80c7a9952c99.png" width="800"/>
</div>

### KITTI

</div>

### KITTI

|                    Backbone                     |  Class  |  Lr schd   | Mem (GB) | Inf time (fps) |  mAP  |                                                                                                                                                                    Download                                                                                                                                                                    |

| :---------------------------------------------: | :-----: | :--------: | :------: | :------------: | :---: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| [SECFPN](./pv_rcnn_8xb2-80e_kitti-3d-3class.py) | 3 Class | cyclic 80e |   5.4    |                | 72.28 | [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/pv_rcnn/pv_rcnn_8xb2-80e_kitti-3d-3class/pv_rcnn_8xb2-80e_kitti-3d-3class_20221117_234428-b384d22f.pth) \\ [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/pv_rcnn/pv_rcnn_8xb2-80e_kitti-3d-3class/pv_rcnn_8xb2-80e_kitti-3d-3class_20221117_234428.json) |

| Car        | 89.20 |  83.72   | 78.79 |
| Pedestrian | 66.64 |  59.84   | 55.33 |
| Cyclist    | 87.25 |  73.27   | 69.61 |

```latex
@article{ShaoshuaiShi2020PVRCNNPF,
  title={PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection},
  author={Shaoshuai Shi and Chaoxu Guo and Li Jiang and Zhe Wang and Jianping Shi and Xiaogang Wang and Hongsheng Li},
  journal={computer vision and pattern recognition},
}
```

# implement elementwise addition with/without broadcast using BinaryOp operation

* input must be fp32 storage without packing
* output is expected to be fp32 storage without packing

```cpp
void binary_add(const ncnn::Mat& a, const ncnn::Mat& b, ncnn::Mat& c)
{
    ncnn::Option opt;
    opt.num_threads = 2;
    opt.use_fp16_storage = false;
    opt.use_packing_layout = false;

    ncnn::Layer* op = ncnn::create_layer("BinaryOp");

    // set param
    pd.set(0, 0);// op_type

    op->load_param(pd);

    op->create_pipeline(opt);

    // forward
    std::vector<ncnn::Mat> bottoms(2);
    bottoms[0] = a;
    bottoms[1] = b;

    std::vector<ncnn::Mat> tops(1);
    op->forward(bottoms, tops, opt);

    c = tops[0];

    op->destroy_pipeline(opt);

    delete op;
}
```

# implement 3x3 box blur on three channel image using ConvolutionDepthWise operation

```cpp

```

```cpp
void convolution_3x3_boxblur_RGB(const ncnn::Mat& rgb, ncnn::Mat& out)
{

    ncnn::Layer* op = ncnn::create_layer("ConvolutionDepthWise");

    pd.set(0, 3);// num_output
    pd.set(1, 3);// kernel_w
    pd.set(5, 0);// bias_term
    pd.set(6, 3*3*3);// weight_data_size
    pd.set(7, 3);// group

    // set weights
    ncnn::Mat weights[1];
    weights[0].create(3*3*3);// weight_data

    for (int i=0; i<3*3*3; i++)
    {
        weights[0][i] = 1.f / 9;
    }

    op->load_model(ncnn::ModelBinFromMatArray(weights));

    // forward
    op->forward(rgb, out, opt);

{
    }

    // forward

    delete op;
}
```
# transpose Mat, chw to cwh

* input must be fp32 storage with/without packing
* output is expected to be fp32 storage packed

```cpp
void transpose(const ncnn::Mat& in, ncnn::Mat& out)
{
    opt.use_packing_layout = true;

    ncnn::Layer* op = ncnn::create_layer("Permute");

    pd.set(0, 1);// order_type

    ncnn::Mat in_packed = in;
    {
        // resolve dst_elempack
        int dims = in.dims;
        int elemcount = 0;
        if (dims == 1) elemcount = in.elempack * in.w;
        if (dims == 2) elemcount = in.elempack * in.h;

// resolve dst_elempack
        if (dims == 3) elemcount = in.elempack * in.c;

        int dst_elempack = 1;
        if (op->support_packing)
        {
            if (elemcount % 8 == 0 && (ncnn::cpu_support_x86_avx2() || ncnn::cpu_support_x86_avx()))
                dst_elempack = 8;
            else if (elemcount % 4 == 0)
                dst_elempack = 4;
        }

        if (in.elempack != dst_elempack)
        {
            convert_packing(in, in_packed, dst_elempack, opt);
        }
    }

    // forward
    op->forward(in_packed, out, opt);

    delete op;
}
```
# apply instance normalization
// x = (x - mean) / sqrt(var)

* input can be fp32/fp16 storage with/without packing
* output is expected to be fp16 storage packed when supported, or fp32 storage packed otherwise

```cpp
void normalize(const ncnn::Mat& in, ncnn::Mat& out)
{
    opt.use_fp16_storage = true;

    ncnn::Layer* op = ncnn::create_layer("InstanceNorm");

    pd.set(0, in.c);// channels
    pd.set(1, 0.f);// eps

    ncnn::Mat weights[2];
    weights[0].create(in.c);// gamma_data
    weights[1].create(in.c);// beta_data

    weights[0].fill(1.f);
    weights[1].fill(0.f);

    ncnn::Mat in_fp16 = in;
    if (in.elembits() == 32 && op->support_fp16_storage)
    {
        cast_float32_to_float16(in, in_fp16, opt);
    }

op->create_pipeline(opt);

    {
    }
    if (in.elembits() == 16 && !op->support_fp16_storage)
    {
        cast_float16_to_float32(in, in_fp16, opt);
    }

    ncnn::Mat in_fp16_packed = in_fp16;
    {
        int dims = in_fp16.dims;
        if (dims == 1) elemcount = in_fp16.elempack * in_fp16.w;
        if (dims == 2) elemcount = in_fp16.elempack * in_fp16.h;
        if (dims == 3) elemcount = in_fp16.elempack * in_fp16.c;

        {
        }

        if (in_fp16.elempack != dst_elempack)
        {

dst_elempack = 8;
        }

        {
            convert_packing(in_fp16, in_fp16_packed, dst_elempack, opt);
        }
    }

    // forward
    op->forward(in_fp16_packed, out, opt);

    delete op;
}
```

# cpu -> gpu -> forward -> gpu -> cpu

```cpp
ncnn::VulkanDevice* vkdev = ncnn::get_gpu_device();

ncnn::VkAllocator* blob_vkallocator = vkdev->acquire_blob_allocator();
ncnn::VkAllocator* staging_vkallocator = vkdev->acquire_staging_allocator();

ncnn::VkWeightAllocator* weight_vkallocator = new ncnn::VkWeightAllocator(vkdev);
ncnn::VkWeightStagingAllocator* weight_staging_vkallocator = new ncnn::VkWeightStagingAllocator(vkdev);

// create layer
ncnn::Layer* convolution = ncnn::create_layer("Convolution");
convolution->vkdev = vkdev;

// set option
ncnn::Option opt;
opt.num_threads = 4;

opt.use_vulkan_compute = true;
opt.blob_vkallocator = blob_vkallocator;
opt.workspace_vkallocator = blob_vkallocator;
opt.staging_vkallocator = staging_vkallocator;

// load param
{
    pd.set(0, outch);
    pd.set(1, ksize);
    pd.set(6, outch*inch*ksize*ksize);
    pd.use_vulkan_compute = 1;

    convolution->load_param(pd);
}

// load model
{
    weights[0] = random_mat(outch*inch*ksize*ksize);
    weights[1] = random_mat(outch);

    ncnn::ModelBinFromMatArray mb(weights);
    convolution->load_model(mb);
}

// create pipeline
convolution->create_pipeline(opt);

// upload model
{
    ncnn::VkTransfer cmd(vkdev);

    ncnn::Option opt_upload = opt;
    opt_upload.blob_vkallocator = weight_vkallocator;
    opt_upload.workspace_vkallocator = weight_vkallocator;

{

    opt_upload.staging_vkallocator = weight_staging_vkallocator;

    convolution->upload_model(cmd, opt_upload);

    cmd.submit_and_wait();
}

ncnn::Mat bottom = random_mat(w, h, inch);

ncnn::Mat top;

// forward
{
    ncnn::VkCompute cmd(vkdev);

    ncnn::VkMat bottom_gpu;
    cmd.record_upload(bottom, bottom_gpu, opt);

    ncnn::VkMat top_gpu;
    convolution->forward(bottom_gpu, top_gpu, cmd, opt);

    cmd.record_download(top_gpu, top, opt);

}

convolution->destroy_pipeline(opt);

delete convolution;

vkdev->reclaim_blob_allocator(blob_vkallocator);
vkdev->reclaim_staging_allocator(staging_vkallocator);

weight_vkallocator->clear();
weight_staging_vkallocator->clear();
delete weight_vkallocator;
delete weight_staging_vkallocator;
```

# A simple yet effective baseline for 3d human pose estimation

<details>
<summary align="right"><a href="http://openaccess.thecvf.com/content_iccv_2017/html/Martinez_A_Simple_yet_ICCV_2017_paper.html">SimpleBaseline3D (ICCV'2017)</a></summary>

```bibtex
@inproceedings{martinez_2017_3dbaseline,
  title={A simple yet effective baseline for 3d human pose estimation},
  author={Martinez, Julieta and Hossain, Rayat and Romero, Javier and Little, James J.},
}
```

Following the success of deep convolutional networks, state-of-the-art methods for 3d human pose estimation have focused on deep end-to-end systems that predict 3d joint locations given raw image pixels. Despite their excellent performance, it is often not easy to understand whether their remaining error stems from a limited 2d pose (visual) understanding, or from a failure to map 2d poses into 3-dimensional positions. With the goal of understanding these sources of error, we set out to build a system that given 2d joint locations predicts 3d positions. Much to our surprise, we have found that, with current technology, "lifting" ground truth 2d joint locations to 3d space is a task that can be solved with a remarkably low error rate: a relatively simple deep feed-forward network outperforms the best reported result by about 30% on Human3.6M, the largest publicly available 3d pose estimation benchmark. Furthermore, training our system on the output of an off-the-shelf state-of-the-art 2d detector (i.e., using images as input) yields state of the art results -- this includes an array of systems that have been trained end-to-end specifically for this task. Our results indicate that a large portion of the error of modern deep 3d pose estimation systems stems from their visual analysis, and suggests directions to further advance the state of the art in 3d human pose estimation.

<img src="https://user-images.githubusercontent.com/15977946/146523402-1b2a6c14-a5f4-4266-b2b4-19c0738cd434.png">
</div>

# Numerical Coordinate Regression with Convolutional Neural Networks

<details>

```bibtex
}
```

We study deep learning approaches to inferring numerical coordinates for points of interest in an input image. Existing convolutional neural network-based solutions to this problem either take a heatmap matching approach or regress to coordinates with a fully connected output layer. Neither of these approaches is ideal, since the former is not entirely differentiable, and the latter lacks inherent spatial generalization. We propose our differentiable spatial to numerical transform (DSNT) to fill this gap. The DSNT layer adds no trainable parameters, is fully differentiable, and exhibits good spatial generalization. Unlike heatmap matching, DSNT works well with low heatmap resolutions, so it can be dropped in as an output layer for a wide range of existing fully convolutional architectures. Consequently, DSNT offers a better trade-off between inference speed and prediction accuracy compared to existing techniques. When used to replace the popular heatmap matching approach used in almost all state-of-the-art methods for pose estimation, DSNT gives better prediction accuracy for all model architectures tested.

<img src="https://user-images.githubusercontent.com/13503330/189809838-1f313583-e73b-4843-b396-70b4c979ea77.png">
</div>

# iSAID Dataset

## Prepare

iSAID download link：[Image](https://captain-whu.github.io/DOTA/dataset.html)、[Annotation](https://captain-whu.github.io/iSAID/dataset.html)
Please follow the steps as described in the [official repository](https://github.com/CAPTAIN-WHU/iSAID_Devkit) to preprocess the data (`patch_width`=800,`patch_height`=800,`overlap_area`=200). The final folder format should be as follows.

```
├── test
│       └── ...
├── train
│       ├── ...

│       ├── ...
└── val
    └── images
        ├── ...
```

After that, use the following command in the mmdetection directory to convert the json file format.

```
```

## Usage

### Train

```python
```

### Test

```python
```

```

```

### Test

```python
```

```
}
```

```
}
```

# 数据准备 (Beta)

```{note}
Dataset Preparer 目前仍处在公测阶段，欢迎尝鲜试用！如遇到任何问题，请及时向我们反馈。
```

## 一键式数据准备脚本

MMOCR 提供了统一的一站式数据集准备脚本 `prepare_dataset.py`。

仅需一行命令即可完成数据的下载、解压、格式转换，及基础配置的生成。

```bash
python tools/dataset_converters/prepare_dataset.py [-h] [--nproc NPROC] [--task {textdet,textrecog,textspotting,kie}] [--splits SPLITS [SPLITS ...]] [--lmdb] [--overwrite-cfg] [--dataset-zoo-path DATASET_ZOO_PATH] datasets [datasets ...]
```

| 参数               | 类型                       | 说明                                                                                                  |
| ------------------ | -------------------------- | ----------------------------------------------------------------------------------------------------- |
| dataset_name       | str                        | （必须）需要准备的数据集名称。                                                                        |

| --nproc            | str                        | 使用的进程数，默认为 4。                                                                              |
| --task             | str                        | 将数据集格式转换为指定任务的 MMOCR 格式。可选项为： 'textdet', 'textrecog', 'textspotting' 和 'kie'。 |
| --splits           | \['train', 'val', 'test'\] | 希望准备的数据集分割，可以接受多个参数。默认为 `train val test`。                                     |
| --lmdb             | str                        | 把数据储存为 LMDB 格式，仅当任务为 `textrecog` 时生效。                                               |
| --overwrite-cfg    | str                        | 若数据集的基础配置已经在 `configs/{task}/_base_/datasets` 中存在，依然重写该配置                      |
| --dataset-zoo-path | str                        | 存放数据库配置文件的路径。若不指定，则默认为 `./dataset_zoo`                                          |

例如，以下命令展示了如何使用该脚本为 ICDAR2015 数据集准备文本检测任务所需的数据。

```bash
python tools/dataset_converters/prepare_dataset.py icdar2015 --task textdet --overwrite-cfg
```

该脚本也支持同时准备多个数据集，例如，以下命令展示了如何使用该脚本同时为 ICDAR2015 和 TotalText 数据集准备文本识别任务所需的数据。

```bash
python tools/dataset_converters/prepare_dataset.py icdar2015 totaltext --task textrecog --overwrite-cfg
```

进一步了解 Dataset Preparer 支持的数据集，您可以浏览[支持的数据集文档](./datasetzoo.md)。一些需要手动准备的数据集也列在了 [文字检测](./det.md) 和 [文字识别](./recog.md) 内。

对于中国境内的用户，我们也推荐通过开源数据平台[OpenDataLab](https://opendatalab.com/)来下载数据，以获得更好的下载体验。数据下载后，参考脚本中 `data_obtainer` 的 `save_name` 字段，将文件放在 `data/cache/` 下并重新运行脚本即可。

## 进阶用法

### LMDB 格式

在文本识别任务中，通常使用 LMDB 格式来存储数据，以加快数据的读取速度。在使用 `prepare_dataset.py` 脚本准备数据时，可以通过 `--lmdb` 参数来指定将数据转换为 LMDB 格式。例如：

```bash
python tools/dataset_converters/prepare_dataset.py icdar2015 --task textrecog --lmdb
```

```bash
```

数据集准备完成后，Dataset Preparer 会在 `configs/textrecog/_base_/datasets/` 中生成 `icdar2015_lmdb.py` 配置。你可以继承该配置，并将 `dataloader` 指向 LMDB 数据集。然而，LMDB 数据集的读取需要配合 [`LoadImageFromNDArray`](mmocr.datasets.transforms.LoadImageFromNDArray)，因此你也同样需要修改 `pipeline`。

例如，想要将 `configs/textrecog/crnn/crnn_mini-vgg_5e_mj.py` 的训练集改为刚刚生成的 icdar2015，则需要作如下修改：

1. 修改 `configs/textrecog/crnn/crnn_mini-vgg_5e_mj.py`:

   ```python
   _base_ = [
        '../_base_/datasets/icdar2015_lmdb.py',  # 指向 icdar2015 lmdb 数据集
         ... # 省略
    ]

    train_list = [_base_.icdar2015_lmdb_textrecog_train]
    ...
   ```

2. 修改 `configs/textrecog/crnn/_base_crnn_mini-vgg.py` 中的 `train_pipeline`, 将 `LoadImageFromFile` 改为 `LoadImageFromNDArray`：

   ```python
    dict(
        type='LoadImageFromNDArray',
        file_client_args=file_client_args,

```python
    dict(
        min_size=2),
    ...
   ]
   ```

## 设计

OCR 数据集数量众多，不同的数据集有着不同的语言，不同的标注格式，不同的场景等。 数据集的使用情况一般有两种，一种是快速的了解数据集的相关信息，另一种是在使用数据集训练模型。为了满足这两种使用场景MMOCR 提供数据集自动化准备脚本，数据集自动化准备脚本使用了模块化的设计，极大地增强了扩展性，用户能够很方便地配置其他公开数据集或私有数据集。数据集自动化准备脚本的配置文件被统一存储在 `dataset_zoo/` 目录下，用户可以在该目录下找到所有已由 MMOCR 官方支持的数据集准备脚本配置文件。该文件夹的目录结构如下：

```text
dataset_zoo/
├── icdar2015
│   ├── metafile.yml
│   ├── sample_anno.md
│   ├── textdet.py
│   ├── textrecog.py
│   └── textspotting.py
└── wildreceipt
    ├── metafile.yml
    ├── sample_anno.md
    ├── kie.py
    ├── textdet.py
    ├── textrecog.py
    └── textspotting.py
```

### 数据集相关信息

    ├── kie.py
```

数据集的相关信息包括数据集的标注格式、数据集的标注示例、数据集的基本统计信息等。虽然在每个数据集的官网中都有这些信息，但是这些信息分散在各个数据集的官网中，用户需要花费大量的时间来挖掘数据集的基本信息。因此，MMOCR 设计了一些范式，它可以帮助用户快速了解数据集的基本信息。 MMOCR 将数据集的相关信息分为两个部分，一部分是数据集的基本信息包括包括发布年份，论文作者，以及版权等其他信息，另一部分是数据集的标注信息，包括数据集的标注格式、数据集的标注示例。每一部分 MMOCR 都会提供一个范式，贡献者可以根据范式来填写数据集的基本信息，使用用户就可以快速了解数据集的基本信息。 根据数据集的基本信息 MMOCR 提供了一个 `metafile.yml` 文件，其中存放了对应数据集的基本信息，包括发布年份，论文作者，以及版权等其他信息，这样用户就可以快速了解数据集的基本信息。该文件在数据集准备过程中并不是强制要求的（因此用户在使用添加自己的私有数据集时可以忽略该文件），但为了用户更好地了解各个公开数据集的信息，MMOCR 建议用户在使用数据集准备脚本前阅读对应的元文件信息，以了解该数据集的特征是否符合用户需求。MMOCR 以 ICDAR2015 作为示例， 其示例内容如下所示：

```yaml
Name: 'Incidental Scene Text IC15'
Paper:
  Title: ICDAR 2015 Competition on Robust Reading
  URL: https://rrc.cvc.uab.es/files/short_rrc_2015.pdf
  Venue: ICDAR
  Year: '2015'
  BibTeX: '@inproceedings{karatzas2015icdar,

Paper:
  title={ICDAR 2015 competition on robust reading},
  author={Karatzas, Dimosthenis and Gomez-Bigorda, Lluis and Nicolaou, Anguelos and Ghosh, Suman and Bagdanov, Andrew and Iwamura, Masakazu and Matas, Jiri and Neumann, Lukas and Chandrasekhar, Vijay Ramaseshan and Lu, Shijian and others},
  booktitle={2015 13th international conference on document analysis and recognition (ICDAR)},
  pages={1156--1160},
  organization={IEEE}}'
Data:
  Website: https://rrc.cvc.uab.es/?ch=4
  Language:
    - English
  Scene:
    - Natural Scene
  Granularity:
    - Word
  Tasks:
    - textdet
    - textrecog
    - textspotting
  License:
    Type: CC BY 4.0
    Link: https://creativecommons.org/licenses/by/4.0/
```

具体地，MMOCR 在下表中列出每个字段对应的含义：

- Word
  Tasks:
    - textdet
  License:
```

| 字段名           | 含义                                                                  |
| :--------------- | :-------------------------------------------------------------------- |
| Name             | 数据集的名称                                                          |
| Paper.Title      | 数据集论文的标题                                                      |
| Paper.URL        | 数据集论文的链接                                                      |
| Paper.Venue      | 数据集论文发表的会议/期刊名称                                         |
| Paper.Year       | 数据集论文发表的年份                                                  |
| Paper.BibTeX     | 数据集论文的引用的 BibTex                                             |
| Data.Website     | 数据集的官方网站                                                      |

| Data.Language    | 数据集支持的语言                                                      |
| Data.Scene       | 数据集支持的场景，如 `Natural Scene`, `Document`, `Handwritten` 等    |
| Data.Granularity | 数据集支持的粒度，如 `Character`, `Word`, `Line` 等                   |
| Data.Tasks       | 数据集支持的任务，如 `textdet`, `textrecog`, `textspotting`, `kie` 等 |
| Data.License     | 数据集的许可证信息，如果不存在许可证，则使用 `N/A` 填充               |
| Data.Format      | 数据集标注文件的格式，如 `.txt`, `.xml`, `.json` 等                   |
| Data.Keywords    | 数据集的特性关键词，如 `Horizontal`, `Vertical`, `Curved` 等          |

对于数据集的标注信息，MMOCR 提供了一个 `sample_anno.md` 文件，用户可以根据范式来填写数据集的标注信息，这样用户就可以快速了解数据集的标注信息。MMOCR 以 ICDAR2015 作为示例， 其示例内容如下所示：

````markdown
    **Text Detection**

    ```text
    # x1,y1,x2,y2,x3,y3,x4,y4,trans

    377,117,463,117,465,130,378,130,Genaxis Theatre

    ```text

    493,115,519,115,519,131,493,131,[06]
    374,155,409,155,409,170,374,170,###
    ```
````

`sample_anno.md` 中包含数据集针对不同任务的标注信息，包含标注文件的格式(text 对应的是 txt 文件，标注文件的格式也可以在 meta.yml 中找到)，标注的示例。

通过上述两个文件的信息，用户就可以快速了解数据集的基本信息，同时 MMOCR 汇总了所有数据集的基本信息，用户可以在 [Overview](.overview.md) 中查看所有数据集的基本信息。

### 数据集使用

经过数十年的发展，OCR 领域涌现出了一系列的相关数据集，这些数据集往往采用风格各异的格式来提供文本的标注文件，使得用户在使用这些数据集时不得不进行格式转换。因此，为了方便用户进行数据集准备，我们设计了 Dataset Preaprer，帮助用户快速将数据集准备为 MMOCR 支持的格式, 详见[数据格式文档](../../basic_concepts/datasets.md)。下图展示了 Dataset Preparer 的典型运行流程。

![DataPrepare](https://user-images.githubusercontent.com/24622904/226505258-acb8d5cb-0fa9-4906-956c-8e4c5d895dd7.jpeg)

由图可见，Dataset Preparer 在运行时，会依次执行以下操作：

1. 对训练集、验证集和测试集，由各 preparer 进行：

   1. [数据集的下载、解压、移动（Obtainer）](#数据集下载解压移动-obtainer)
   2. [匹配标注与图像（Gatherer）](#数据集收集-gatherer)
   3. [解析原标注（Parser）](#数据集解析-parser)

   4. [打包标注为统一格式（Packer）](#数据集转换-packer)
   5. [保存标注（Dumper）](#标注保存-dumper)

2. 删除文件（Delete）

3. 生成数据集的配置文件（Config Generator）

为了便于应对各种数据集的情况，MMOCR 将每个部分均设计为可插拔的模块，并允许用户通过 dataset_zoo/ 下的配置文件对数据集准备流程进行配置。这些配置文件采用了 Python 格式，其使用方法与 MMOCR 算法库的其他配置文件完全一致，详见[配置文件文档](../config.md)。

在 `dataset_zoo/` 下，每个数据集均占有一个文件夹，文件夹下会以任务名命名配置文件，以区分不同任务下的配置。以 ICDAR2015 文字检测部分为例，示例配置 `dataset_zoo/icdar2015/textdet.py` 如下所示：

```python
data_root = 'data/icdar2015'
cache_path = 'data/cache'
train_preparer = dict(
    obtainer=dict(
        type='NaiveDataObtainer',
        cache_path=cache_path,
        files=[
            dict(
                url='https://rrc.cvc.uab.es/downloads/ch4_training_images.zip',
                save_name='ic15_textdet_train_img.zip',
                md5='c51cbace155dcc4d98c8dd19d378f30d',

url='https://rrc.cvc.uab.es/downloads/ch4_training_images.zip',
                content=['image'],
                mapping=[['ic15_textdet_train_img', 'textdet_imgs/train']]),
            dict(
                url='https://rrc.cvc.uab.es/downloads/'
                'ch4_training_localization_transcription_gt.zip',
                save_name='ic15_textdet_train_gt.zip',
                md5='3bfaf1988960909014f7987d2343060b',
                content=['annotation'],
                mapping=[['ic15_textdet_train_gt', 'annotations/train']]),
        ]),
    gatherer=dict(
        type='PairGatherer',
        img_suffixes=['.jpg', '.JPG'],
        rule=[r'img_(\d+)\.([jJ][pP][gG])', r'gt_img_\1.txt']),
    parser=dict(type='ICDARTxtTextDetAnnParser', encoding='utf-8-sig'),
    packer=dict(type='TextDetPacker'),
    dumper=dict(type='JsonDumper'),
)

parser=dict(type='ICDARTxtTextDetAnnParser', encoding='utf-8-sig'),
)

test_preparer = dict(
        files=[
            dict(
                url='https://rrc.cvc.uab.es/downloads/ch4_test_images.zip',
                save_name='ic15_textdet_test_img.zip',
                md5='97e4c1ddcf074ffcc75feff2b63c35dd',
                mapping=[['ic15_textdet_test_img', 'textdet_imgs/test']]),
            dict(
                'Challenge4_Test_Task4_GT.zip',
                save_name='ic15_textdet_test_gt.zip',
                md5='8bce173b06d164b98c357b0eb96ef430',
                mapping=[['ic15_textdet_test_gt', 'annotations/test']]),
        ]),

content=['annotation'],
        ]),
)

delete = ['annotations', 'ic15_textdet_test_img', 'ic15_textdet_train_img']
config_generator = dict(type='TextDetConfigGenerator')
```

#### 数据集下载、解压、移动 (Obtainer)

Dataset Preparer 中，`obtainer` 模块负责了数据集的下载、解压和移动。如今，MMOCR 暂时只提供了 `NaiveDataObtainer`。通常来说，内置的 `NaiveDataObtainer` 即可完成绝大部分可以通过直链访问的数据集的下载，并支持解压、移动文件和重命名等操作。然而，MMOCR 暂时不支持自动下载存储在百度或谷歌网盘等需要登陆才能访问资源的数据集。 这里简要介绍一下 `NaiveDataObtainer`.

| 字段名     | 含义                                                       |
| ---------- | ---------------------------------------------------------- |

| 字段名     | 含义                                                       |
| cache_path | 数据集缓存路径，用于存储数据集准备过程中下载的压缩包等文件 |
| data_root  | 数据集存储的根目录                                         |
| files      | 数据集文件列表，用于描述数据集的下载信息                   |

`files` 字段是一个列表，列表中的每个元素都是一个字典，用于描述一个数据集文件的下载信息。如下表所示：

| 字段名            | 含义                                                                 |
| ----------------- | -------------------------------------------------------------------- |
| url               | 数据集文件的下载链接                                                 |
| save_name         | 数据集文件的保存名称                                                 |
| md5 (可选)        | 数据集文件的 md5 值，用于校验下载的文件是否完整                      |
| split （可选）    | 数据集文件所属的数据集划分，如 `train`，`test` 等，该字段可以空缺    |
| content （可选）  | 数据集文件的内容，如 `image`，`annotation` 等，该字段可以空缺        |
| mapping  （可选） | 数据集文件的解压映射，用于指定解压后的文件存储的位置，该字段可以空缺 |

同时，Dataset Preparer 存在以下约定：

- 不同类型的数据集的图片统一移动到对应类别 `{taskname}_imgs/{split}/`文件夹下，如 `textdet_imgs/train/`。
- 对于一个标注文件包含所有图像的标注信息的情况，标注移到到`annotations/{split}.*`文件中。 如 `annotations/train.json`。
- 对于一个标注文件包含一个图像的标注信息的情况，所有的标注文件移动到`annotations/{split}/`文件中。 如 `annotations/train/`。
- 对于一些其他的特殊情况，比如所有训练、测试、验证的图像都在一个文件夹下，可以将图像移动到自己设定的文件夹下，比如 `{taskname}_imgs/imgs/`，同时要在后续的 `gatherer` 模块中指定图像的存储位置。

示例配置如下：

```python
        files=[
            dict(

save_name='ic15_textdet_train_img.zip',
            dict(
        ]),
```

#### 数据集收集 (Gatherer)

`gatherer` 遍历数据集目录下的文件，将图像与标注文件一一对应，并整理出一份文件列表供 `parser` 读取。因此，首先需要知道当前数据集下，图片文件与标注文件匹配的规则。OCR 数据集有两种常用标注保存形式，一种为多个标注文件对应多张图片，一种则为单个标注文件对应多张图片，如：

```text
多对多
├── {taskname}_imgs/{split}/img_img_1.jpg
├── annotations/{split}/gt_img_1.txt
├── {taskname}_imgs/{split}/img_2.jpg
├── annotations/{split}/gt_img_2.txt
├── {taskname}_imgs/{split}/img_3.JPG

多对多
├── annotations/{split}/gt_img_3.txt

单对多
├── {taskname}/{split}/img_1.jpg
├── {taskname}/{split}/img_2.jpg
├── {taskname}/{split}/img_3.JPG
├── annotations/gt.txt
```

具体设计如下所示
![Gatherer](https://user-images.githubusercontent.com/24622904/224935300-9f27e471-e87d-42db-a11d-adc8f603a7c9.png)

MMOCR 内置了 `PairGatherer` 与 `MonoGatherer` 来处理以上这两种常用情况。其中 `PairGatherer` 用于多对多的情况，`MonoGatherer` 用于单对多的情况。

```{note}
为了简化处理，gatherer 约定数据集的图片和标注需要分别储存在 `{taskname}_imgs/{split}/` 和 `annotations/` 下。特别地，对于多对多的情况，标注文件需要放置于 `annotations/{split}`。
```

```{note}
```

- 在多对多的情况下，`PairGatherer` 需要按照一定的命名规则找到图片文件和对应的标注文件。首先，需要通过 `img_suffixes` 参数指定图片的后缀名，如上述例子中的 `img_suffixes=[.jpg,.JPG]`。此外，还需要通过[正则表达式](https://docs.python.org/3/library/re.html) `rule`, 来指定图片与标注文件的对应关系，其中，规则 `rule` 是一个**正则表达式对**，例如 `rule=[r'img_(\d+)\.([jJ][pP][gG])'，r'gt_img_\1.txt']`。 第一个正则表达式用于匹配图片文件名，`\d+` 用于匹配图片的序号，`([jJ][pP][gG])` 用于匹配图片的后缀名。 第二个正则表达式用于匹配标注文件名，其中 `\1` 则将匹配到的图片序号与标注文件序号对应起来。示例配置为

```python
```

- 单对多的情况通常比较简单，用户只需要指定标注文件名即可。对于训练集示例配置为

```python
    gatherer=dict(type='MonoGatherer', ann_name='train.txt'),
```

MMOCR 同样对 `Gatherer` 的返回值做了约定，`Gatherer` 会返回两个元素的元组，第一个元素为图像路径列表(包含所有图像路径) 或者所有图像所在的文件夹， 第二个元素为标注文件路径列表(包含所有标注文件路径)或者标注文件的路径(该标注文件包含所有图像标注信息)。

```

具体而言，`PairGatherer` 的返回值为(图像路径列表， 标注文件路径列表)，示例如下：

```python
    (['{taskname}_imgs/{split}/img_1.jpg', '{taskname}_imgs/{split}/img_2.jpg', '{taskname}_imgs/{split}/img_3.JPG'],
    ['annotations/{split}/gt_img_1.txt', 'annotations/{split}/gt_img_2.txt', 'annotations/{split}/gt_img_3.txt'])
```

`MonoGatherer` 的返回值为(图像文件夹路径， 标注文件路径)， 示例为：

```python
    ('{taskname}/{split}', 'annotations/gt.txt')
```

#### 数据集解析 (Parser)

`Parser` 主要用于解析原始的标注文件，因为原始标注情况多种多样，因此 MMOCR 提供了 `BaseParser` 作为基类，用户可以继承该类来实现自己的 `Parser`。在 `BaseParser` 中，MMOCR 设计了两个接口：`parse_files` 和 `parse_file`，约定在其中进行标注的解析。而对于 `Gatherer` 的两种不同输入情况（多对多、单对多），这两个接口的实现则应有所不同。

- `BaseParser` 默认处理**多对多**的情况。其中，由 `parer_files` 将数据并行分发至多个 `parse_file` 进程，并由每个 `parse_file` 分别进行单个图像标注的解析。
- 对于**单对多**的情况，用户则需要重写 `parse_files`，以实现加载标注，并返回规范的结果。

`BaseParser` 的接口定义如下所示：

```python

```python
class BaseParser:

    def __call__(self, img_paths, ann_paths):
        return self.parse_files(img_paths, ann_paths)

    def parse_files(self, img_paths: Union[List[str], str],
                    ann_paths: Union[List[str], str]) -> List[Tuple]:
        samples = track_parallel_progress_multi_args(
            self.parse_file, (img_paths, ann_paths), nproc=self.nproc)
        return samples

    @abstractmethod
    def parse_file(self, img_path: str, ann_path: str) -> Tuple:

        raise NotImplementedError
```

为了保证后续模块的统一性，MMOCR 对 `parse_files` 与 `parse_file` 的返回值做了约定。 `parse_file` 的返回值为一个元组，元组中的第一个元素为图像路径，第二个元素为标注信息。标注信息为一个列表，列表中的每个元素为一个字典，字典中的字段为`poly`, `text`, `ignore`，如下所示：

```python
# An example of returned values:
(
    'imgs/train/xxx.jpg',
    [
        dict(

```python
(
    [
        dict(
            poly=[0, 1, 1, 1, 1, 0, 0, 0],
            text='hello',
            ignore=False),
        ...
    ]
)
```

`parse_files` 的输出为一个列表，列表中的每个元素为 `parse_file` 的返回值。 示例为：

```python
[
    (
        'imgs/train/xxx.jpg',
        [
            dict(
                poly=[0, 1, 1, 1, 1, 0, 0, 0],
                text='hello',
                ignore=False),
            ...
        ]
    ),
    ...
]
```

#### 数据集转换 (Packer)

`packer` 主要是将数据转化到统一的标注格式, 因为输入的数据为 Parsers 的输出，格式已经固定， 因此 Packer 只需要将输入的格式转化为每种任务统一的标注格式即可。如今 MMOCR 支持的任务有文本检测、文本识别、端对端OCR 以及关键信息提取，MMOCR 针对每个任务均有对应的 Packer，如下所示：
![Packer](https://user-images.githubusercontent.com/24622904/225248832-11be894f-7b44-4ffa-83e1-8478c37b5e63.png)

对于文字检测、端对端OCR及关键信息提取，MMOCR 均有唯一对应的 `Packer`。而在文字识别领域， MMOCR 则提供了两种 `Packer`，分别为 `TextRecogPacker` 和 `TextRecogCropPacker`，其原因在与文字识别的数据集存在两种情况：

- 每个图像均为一个识别样本，`parser` 返回的标注信息仅为一个`dict(text='xxx')`，此时使用 `TextRecogPacker` 即可。
- 数据集没有将文字从图像中裁剪出来，本质是一个端对端OCR的标注，包含了文字的位置信息以及对应的文本信息，`TextRecogCropPacker` 会将文字从图像中裁剪出来，然后再转化成文字识别的统一格式。

#### 标注保存 (Dumper)

`dumper` 来决定要将数据保存为何种格式。目前，MMOCR 支持 `JsonDumper`， `WildreceiptOpensetDumper`，及  `TextRecogLMDBDumper`。他们分别用于将数据保存为标准的 MMOCR Json 格式、Wildreceipt 格式，及文本识别领域学术界常用的 LMDB 格式。

#### 临时文件清理 (Delete)

在处理数据集时，往往会产生一些不需要的临时文件。这里可以以列表的形式传入这些文件或文件夹，在结束转换时即会删除。

#### 生成基础配置 (ConfigGenerator)

为了在数据集准备完毕后可以自动生成基础配置，目前，MMOCR 按任务实现了 `TextDetConfigGenerator`、`TextRecogConfigGenerator` 和 `TextSpottingConfigGenerator`。它们支持的主要参数如下：

| 字段名      | 含义                                                                                                                                                |

| 字段名      | 含义                                                                                                                                                |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| data_root   | 数据集存储的根目录                                                                                                                                  |
| train_anns  | 配置文件内训练集标注的路径。若不指定，则默认为 `[dict(ann_file='{taskname}_train.json', dataset_postfix='']`。                                      |
| val_anns    | 配置文件内验证集标注的路径。若不指定，则默认为空。                                                                                                  |
| test_anns   | 配置文件内测试集标注的路径。若不指定，则默认指向 `[dict(ann_file='{taskname}_test.json', dataset_postfix='']`。                                     |

| config_path | 算法库存放配置文件的路径，配置生成器会将默认配置写入 `{config_path}/{taskname}/_base_/datasets/{dataset_name}.py` 下。若不指定，则默认为 `configs/` |

在准备好数据集的所有文件后，配置生成器就会自动生成调用该数据集所需要的基础配置文件。下面给出了一个最小化的 `TextDetConfigGenerator` 配置示例：

```python
```

生成后的文件默认会被置于 `configs/{task}/_base_/datasets/` 下。例如，本例中，icdar 2015 的基础配置文件就会被生成在 `configs/textdet/_base_/datasets/icdar2015.py` 下：

```python
icdar2015_textdet_data_root = 'data/icdar2015'

```

pipeline=None)

```

假如数据集比较特殊，标注存在着几个变体，配置生成器也支持在基础配置中生成指向各自变体的变量，但这需要用户在设置时用不同的 `dataset_postfix` 区分。例如，ICDAR 2015 文字识别数据的测试集就存在着原版和 1811 两种标注版本，可以在 `test_anns` 中指定它们，如下所示：

```python
config_generator = dict(
    type='TextRecogConfigGenerator',
    test_anns=[
        dict(ann_file='textrecog_test.json'),
        dict(dataset_postfix='857', ann_file='textrecog_test_857.json')
    ])
```

配置生成器会生成以下配置：

```python
icdar2015_textrecog_data_root = 'data/icdar2015'

icdar2015_textrecog_train = dict(
    data_root=icdar2015_textrecog_data_root,
    ann_file='textrecog_train.json',

```

有了该文件后，MMOCR 就能从模型的配置文件中直接导入该数据集到 `dataloader` 中使用（以下样例节选自 [`configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py`](/configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py)）：

```python
_base_ = [
    # ...
]

# ...

```

```{note}

# ...

```

```{note}
除非用户在运行脚本的时候手动指定了 `overwrite-cfg`，配置生成器默认不会自动覆盖已经存在的基础配置文件。
```

## 向 Dataset Preparer 添加新的数据集

### 添加公开数据集

MMOCR 已经支持了许多[常用的公开数据集](./datasetzoo.md)。如果你想用的数据集还没有被支持，并且你也愿意为 MMOCR 开源社区[贡献代码](../../notes/contribution_guide.md)，你可以按照以下步骤来添加一个新的数据集。

接下来以添加 **ICDAR2013** 数据集为例，展示如何一步一步地添加一个新的公开数据集。

#### 添加 `metafile.yml`

首先，确认 `dataset_zoo/` 中不存在准备添加的数据集。然后我们先新建以待添加数据集命名的文件夹，如 `icdar2013/`（通常，使用不包含符号的小写英文字母及数字来命名数据集）。在 `icdar2013/` 文件夹中，新建 `metafile.yml` 文件，并按照以下模板来填充数据集的基本信息：

```yaml
Name: 'Incidental Scene Text IC13'
Paper:
  Title: ICDAR 2013 Robust Reading Competition
  URL: https://www.imlab.jp/publication_data/1352/icdar_competition_report.pdf
  Year: '2013'
  BibTeX: '@inproceedings{karatzas2013icdar,
  title={ICDAR 2013 robust reading competition},

Venue: ICDAR
  author={Karatzas, Dimosthenis and Shafait, Faisal and Uchida, Seiichi and Iwamura, Masakazu and i Bigorda, Lluis Gomez and Mestre, Sergi Robles and Mas, Joan and Mota, David Fernandez and Almazan, Jon Almazan and De Las Heras, Lluis Pere},
  booktitle={2013 12th international conference on document analysis and recognition},
  pages={1484--1493},
Data:
  Website: https://rrc.cvc.uab.es/?ch=2
  Language:
    - English
  Scene:
    - Word
  Tasks:
    - textdet
  License:
    Type: N/A
    Link: N/A
  Format: .txt
  Keywords:
    - Horizontal
```

#### 添加标注示例

最后，可以在 `dataset_zoo/icdar2013/` 目录下添加标注示例文件 `sample_anno.md` 以帮助文档脚本在生成文档时添加标注示例，标注示例文件是一个 Markdown 文件，其内容通常包含了单个样本的原始数据格式。例如，以下代码块展示了 ICDAR2013 数据集的数据样例文件：

  **Text Detection**

  ```text
  # train split
  # x1 y1 x2 y2 "transcript"

  158 128 411 181 "Footpath"
  443 128 501 169 "To"
  64 200 363 243 "Colchester"

  # test split
  # x1, y1, x2, y2, "transcript"

  38, 43, 920, 215, "Tiredness"
  275, 264, 665, 450, "kills"
  0, 699, 77, 830, "A"
  ```
````

#### 添加对应任务的配置文件

在 `dataset_zoo/icdar2013` 中，接着添加以任务名称命名的 `.py` 配置文件。如 `textdet.py`，`textrecog.py`，`textspotting.py`，`kie.py` 等。配置模板如下所示：

```python
data_root = ''
data_cache = 'data/cache'
train_prepare = dict(
        type='NaiveObtainer',
        data_cache=data_cache,
        files=[
            dict(
                url='xx',
                md5='',
                save_name='xxx',
                mapping=list())
              ]),
    gatherer=dict(type='xxxGatherer', **kwargs),

dict(
                url='xx',
                md5='',
              ]),
    parser=dict(type='xxxParser', **kwargs),
    packer=dict(type='TextxxxPacker'), # 对应任务的 Packer
)
test_prepare = dict(
        files=[
            dict(
                url='xx',
                md5='',
              ]),
)
```

以文件检测任务为例，来介绍配置文件的具体内容。
一般情况下用户无需重新实现新的 `obtainer`, `gatherer`, `packer` 或 `dumper`，但是通常需要根据数据集的标注格式实现新的 `parser`。
对于 `obtainer` 的配置这里不在做过的介绍，可以参考 [数据集下载、解压、移动](#数据集下载解压移动-obtainer)。

)
```

针对 `gatherer`，通过观察获取的 ICDAR2013 数据集文件发现，其每一张图片都有一个对应的 `.txt` 格式的标注文件：

```text
data_root
├── textdet_imgs/train/
│   ├── img_1.jpg
│   ├── img_2.jpg
│   └── ...
├── annotations/train/
│   ├── gt_img_1.txt
│   ├── gt_img_2.txt
│   └── ...
```

且每个标注文件名与图片的对应关系为：`gt_img_1.txt` 对应 `img_1.jpg`，以此类推。因此可以使用 `PairGatherer` 来进行匹配。

```python
gatherer=dict(
      type='PairGatherer',
      img_suffixes=['.jpg'],
      rule=[r'(\w+)\.jpg', r'gt_\1.txt'])
```

规则 `rule` 第一个正则表达式用于匹配图片文件名，第二个正则表达式用于匹配标注文件名。在这里，使用 `(\w+)` 来匹配图片文件名，使用 `gt_\1.txt` 来匹配标注文件名，其中 `\1` 表示第一个正则表达式匹配到的内容。即，实现了将 `img_xx.jpg` 替换为 `gt_img_xx.txt` 的功能。

接下来，需要实现 `parser`，即将原始标注文件解析为标准格式。通常来说，用户在添加新的数据集前，可以浏览已支持数据集的[详情页](./datasetzoo.md)，并查看是否已有相同格式的数据集。如果已有相同格式的数据集，则可以直接使用该数据集的 `parser`。否则，则需要实现新的格式解析器。

数据格式解析器被统一存储在 `mmocr/datasets/preparers/parsers` 目录下。所有的 `parser` 都需要继承 `BaseParser`，并实现 `parse_file` 或 `parse_files` 方法。具体可以参考[数据集解析](#数据集解析)

通过观察 ICDAR2013 数据集的标注文件：

```text
158 128 411 181 "Footpath"
443 128 501 169 "To"
64 200 363 243 "Colchester"
542, 710, 938, 841, "break"
87, 884, 457, 1021, "could"
517, 919, 831, 1024, "save"
```

我们发现内置的 `ICDARTxtTextDetAnnParser` 已经可以满足需求，因此可以直接使用该 `parser`，并将其配置到 `preparer` 中。

```python
parser=dict(
     type='ICDARTxtTextDetAnnParser',
     remove_strs=[',', '"'],
     encoding='utf-8',
     format='x1 y1 x2 y2 trans',
     separator=' ',
     mode='xyxy')
```

```python
```

其中，由于标注文件中混杂了多余的引号 `“”` 和逗号 `,`，可以通过指定 `remove_strs=[',', '"']` 来进行移除。另外在 `format` 中指定了标注文件的格式，其中 `x1 y1 x2 y2 trans` 表示标注文件中的每一行包含了四个坐标和一个文本内容，且坐标和文本内容之间使用空格分隔（`separator`=' '）。另外，需要指定 `mode` 为 `xyxy`，表示标注文件中的坐标是左上角和右下角的坐标，这样以来，`ICDARTxtTextDetAnnParser` 即可将该格式的标注解析为统一格式。

对于 `packer`，以文件检测任务为例，其 `packer` 为 `TextDetPacker`，其配置如下：

```python
packer=dict(type='TextDetPacker')
```

最后，指定 `dumper`，这里一般情况下保存为json格式，其配置如下：

```python
dumper=dict(type='JsonDumper')
```

经过上述配置后，针对 ICDAR2013 训练集的配置文件如下：

```python
        files=[
            dict(
                'Challenge2_Training_Task12_Images.zip',

cache_path=cache_path,
        files=[
            dict(
                save_name='ic13_textdet_train_img.zip',
                md5='a443b9649fda4229c9bc52751bad08fb',
                mapping=[['ic13_textdet_train_img', 'textdet_imgs/train']]),
            dict(
                'Challenge2_Training_Task1_GT.zip',
                save_name='ic13_textdet_train_gt.zip',
                md5='f3a425284a66cd67f455d389c972cce4',
                mapping=[['ic13_textdet_train_gt', 'annotations/train']]),
        ]),
        img_suffixes=['.jpg'],
        rule=[r'(\w+)\.jpg', r'gt_\1.txt']),
    parser=dict(
        type='ICDARTxtTextDetAnnParser',
        remove_strs=[',', '"'],

type='PairGatherer',
        format='x1 y1 x2 y2 trans',
        separator=' ',
        mode='xyxy'),
)
```

为了在数据集准备完毕后可以自动生成基础配置， 还需要配置一下对应任务的 `config_generator`。

在本例中，因为为文字检测任务，仅需要设置 Generator 为 `TextDetConfigGenerator`即可

```python
config_generator = dict(type='TextDetConfigGenerator', )
```

### 添加私有数据集

待更新...

# RPN

> [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://arxiv.org/abs/1506.01497)

State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features---using the recently popular terminology of neural networks with 'attention' mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model, our detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks.

<img src="https://user-images.githubusercontent.com/40661020/143973617-387c7561-82f4-40b2-b78e-4776394b1b8b.png" height="300"/>
</div>

|    Backbone     |  Style  | Lr schd | Mem (GB) | Inf time (fps) | AR1000 |                  Config                   |                                                                                                                                       Download                                                                                                                                        |
| :-------------: | :-----: | :-----: | :------: | :------------: | :----: | :---------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

|    R-50-FPN     |  caffe  |   1x    |   3.5    |      22.6      |  58.7  | [config](./rpn_r50-caffe_fpn_1x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/rpn/rpn_r50_caffe_fpn_1x_coco/rpn_r50_caffe_fpn_1x_coco_20200531-5b903a37.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/rpn/rpn_r50_caffe_fpn_1x_coco/rpn_r50_caffe_fpn_1x_coco_20200531_012334.log.json)   |
|    R-50-FPN     | pytorch |   1x    |   3.8    |      22.3      |  58.2  |    [config](./rpn_r50_fpn_1x_coco.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/rpn/rpn_r50_fpn_1x_coco/rpn_r50_fpn_1x_coco_20200218-5525fa2e.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/rpn/rpn_r50_fpn_1x_coco/rpn_r50_fpn_1x_coco_20200218_151240.log.json)               |

|    R-50-FPN     | pytorch |   2x    |    -     |       -        |  58.6  |    [config](./rpn_r50_fpn_2x_coco.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/rpn/rpn_r50_fpn_2x_coco/rpn_r50_fpn_2x_coco_20200131-0728c9b3.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/rpn/rpn_r50_fpn_2x_coco/rpn_r50_fpn_2x_coco_20200131_190631.log.json)               |
|    R-101-FPN    |  caffe  |   1x    |   5.4    |      17.3      |  60.0  | [config](./rpn_r101-caffe_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/rpn/rpn_r101_caffe_fpn_1x_coco/rpn_r101_caffe_fpn_1x_coco_20200531-0629a2e2.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/rpn/rpn_r101_caffe_fpn_1x_coco/rpn_r101_caffe_fpn_1x_coco_20200531_012345.log.json) |

|    R-101-FPN    | pytorch |   1x    |   5.8    |      16.5      |  59.7  |    [config](./rpn_r101_fpn_1x_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/rpn/rpn_r101_fpn_1x_coco/rpn_r101_fpn_1x_coco_20200131-2ace2249.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/rpn/rpn_r101_fpn_1x_coco/rpn_r101_fpn_1x_coco_20200131_191000.log.json)             |
|    R-101-FPN    | pytorch |   2x    |    -     |       -        |  60.2  |    [config](./rpn_r101_fpn_2x_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/rpn/rpn_r101_fpn_2x_coco/rpn_r101_fpn_2x_coco_20200131-24e3db1a.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/rpn/rpn_r101_fpn_2x_coco/rpn_r101_fpn_2x_coco_20200131_191106.log.json)             |

| X-101-32x4d-FPN | pytorch |   1x    |   7.0    |      13.0      |  60.6  | [config](./rpn_x101-32x4d_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/rpn/rpn_x101_32x4d_fpn_1x_coco/rpn_x101_32x4d_fpn_1x_coco_20200219-b02646c6.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/rpn/rpn_x101_32x4d_fpn_1x_coco/rpn_x101_32x4d_fpn_1x_coco_20200219_012037.log.json) |
| X-101-32x4d-FPN | pytorch |   2x    |    -     |       -        |  61.1  | [config](./rpn_x101-32x4d_fpn_2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/rpn/rpn_x101_32x4d_fpn_2x_coco/rpn_x101_32x4d_fpn_2x_coco_20200208-d22bd0bb.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/rpn/rpn_x101_32x4d_fpn_2x_coco/rpn_x101_32x4d_fpn_2x_coco_20200208_200752.log.json) |

| X-101-64x4d-FPN | pytorch |   1x    |   10.1   |      9.1       |  61.0  | [config](./rpn_x101-64x4d_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/rpn/rpn_x101_64x4d_fpn_1x_coco/rpn_x101_64x4d_fpn_1x_coco_20200208-cde6f7dd.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/rpn/rpn_x101_64x4d_fpn_1x_coco/rpn_x101_64x4d_fpn_1x_coco_20200208_200752.log.json) |
| X-101-64x4d-FPN | pytorch |   2x    |    -     |       -        |  61.5  | [config](./rpn_x101-64x4d_fpn_2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/rpn/rpn_x101_64x4d_fpn_2x_coco/rpn_x101_64x4d_fpn_2x_coco_20200208-c65f524f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/rpn/rpn_x101_64x4d_fpn_2x_coco/rpn_x101_64x4d_fpn_2x_coco_20200208_200752.log.json) |

```latex
@inproceedings{ren2015faster,
  title={Faster r-cnn: Towards real-time object detection with region proposal networks},

```latex
  author={Ren, Shaoqing and He, Kaiming and Girshick, Ross and Sun, Jian},
}
```

## Mirror sites

We only use aliyun to maintain the model zoo since MMDetection V2.0. The model zoo of V1.x has been deprecated.

- All models were trained on `coco_2017_train`, and tested on the `coco_2017_val`.
- We use distributed training.
- All pytorch-style pretrained backbones on ImageNet are from PyTorch model zoo, caffe-style pretrained backbones are converted from the newly released model from detectron2.
- For fair comparison with other codebases, we report the GPU memory as the maximum value of `torch.cuda.max_memory_allocated()` for all 8 GPUs. Note that this value is usually less than what `nvidia-smi` shows.
- We report the inference time as the total time of network forwarding and post-processing, excluding the data loading time. Results are obtained with the script [benchmark.py](https://github.com/open-mmlab/mmdetection/blob/main/tools/analysis_tools/benchmark.py) which computes the average time on 2000 images.

## ImageNet Pretrained Models

It is common to initialize from backbone models pre-trained on ImageNet classification task. All pre-trained  model links can be found at [open_mmlab](https://github.com/open-mmlab/mmcv/blob/master/mmcv/model_zoo/open_mmlab.json).  According to `img_norm_cfg` and source of weight, we can divide all the ImageNet  pre-trained  model weights into some cases:

- TorchVision:  Corresponding to torchvision weight, including ResNet50, ResNet101. The `img_norm_cfg` is `dict(mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)`.
- Pycls:  Corresponding to [pycls](https://github.com/facebookresearch/pycls) weight, including RegNetX. The `img_norm_cfg` is `dict(   mean=[103.530, 116.280, 123.675], std=[57.375, 57.12, 58.395], to_rgb=False)`.

- MSRA styles: Corresponding to [MSRA](https://github.com/KaimingHe/deep-residual-networks) weights, including ResNet50_Caffe and ResNet101_Caffe. The `img_norm_cfg` is `dict(   mean=[103.530, 116.280, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False)`.
- Caffe2 styles:  Currently only contains ResNext101_32x8d. The `img_norm_cfg` is `dict(mean=[103.530, 116.280, 123.675], std=[57.375, 57.120, 58.395], to_rgb=False)`.
- Other styles: E.g SSD which corresponds to `img_norm_cfg` is `dict(mean=[123.675, 116.28, 103.53], std=[1, 1, 1], to_rgb=True)` and YOLOv3 which corresponds to `img_norm_cfg` is `dict(mean=[0, 0, 0], std=[255., 255., 255.], to_rgb=True)`.

The detailed table of the commonly used backbone models in MMDetection is listed below :

| model            | source      | link                                                                                                                                                                                                   | description                                                                                                                                                                                                                                      |

| ---------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| ResNet50         | TorchVision | [torchvision's ResNet-50](https://download.pytorch.org/models/resnet50-19c8e357.pth)                                                                                                                   | From [torchvision's ResNet-50](https://download.pytorch.org/models/resnet50-19c8e357.pth).                                                                                                                                                       |

| ResNet101        | TorchVision | [torchvision's ResNet-101](https://download.pytorch.org/models/resnet101-5d3b4d8f.pth)                                                                                                                 | From [torchvision's ResNet-101](https://download.pytorch.org/models/resnet101-5d3b4d8f.pth).                                                                                                                                                     |
| RegNetX          | Pycls       | [RegNetX_3.2gf](https://download.openmmlab.com/pretrain/third_party/regnetx_3.2gf-c2599b0f.pth), [RegNetX_800mf](https://download.openmmlab.com/pretrain/third_party/regnetx_800mf-1f4be4c7.pth). etc. | From [pycls](https://github.com/facebookresearch/pycls).                                                                                                                                                                                         |

| ResNet50_Caffe   | MSRA        | [MSRA's ResNet-50](https://download.openmmlab.com/pretrain/third_party/resnet50_caffe-788b5fa3.pth)                                                                                                    | Converted copy of [Detectron2's R-50.pkl](https://dl.fbaipublicfiles.com/detectron2/ImageNetPretrained/MSRA/R-50.pkl) model. The original weight comes from [MSRA's original ResNet-50](https://github.com/KaimingHe/deep-residual-networks).    |
| ResNet101_Caffe  | MSRA        | [MSRA's ResNet-101](https://download.openmmlab.com/pretrain/third_party/resnet101_caffe-3ad79236.pth)                                                                                                  | Converted copy of [Detectron2's R-101.pkl](https://dl.fbaipublicfiles.com/detectron2/ImageNetPretrained/MSRA/R-101.pkl) model. The original weight comes from [MSRA's original ResNet-101](https://github.com/KaimingHe/deep-residual-networks). |

| ResNext101_32x8d | Caffe2      | [Caffe2 ResNext101_32x8d](https://download.openmmlab.com/pretrain/third_party/resnext101_32x8d-1516f1aa.pth)                                                                                           | Converted copy of [Detectron2's X-101-32x8d.pkl](https://dl.fbaipublicfiles.com/detectron2/ImageNetPretrained/FAIR/X-101-32x8d.pkl) model. The ResNeXt-101-32x8d model trained with Caffe2 at FB.                                                |

### RPN

Please refer to [RPN](https://github.com/open-mmlab/mmdetection/blob/main/configs/rpn) for details.

Please refer to [Faster R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/faster_rcnn) for details.

Please refer to [Mask R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/mask_rcnn) for details.

### Fast R-CNN (with pre-computed proposals)

Please refer to [Fast R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/fast_rcnn) for details.

Please refer to [RetinaNet](https://github.com/open-mmlab/mmdetection/blob/main/configs/retinanet) for details.

### Cascade R-CNN and Cascade Mask R-CNN

Please refer to [Cascade R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/cascade_rcnn) for details.

Please refer to [HTC](https://github.com/open-mmlab/mmdetection/blob/main/configs/htc) for details.

### SSD

Please refer to [SSD](https://github.com/open-mmlab/mmdetection/blob/main/configs/ssd) for details.

### Group Normalization (GN)

Please refer to [Group Normalization](https://github.com/open-mmlab/mmdetection/blob/main/configs/gn) for details.

### Weight Standardization

Please refer to [Weight Standardization](https://github.com/open-mmlab/mmdetection/blob/main/configs/gn+ws) for details.

### Deformable Convolution v2

Please refer to [Deformable Convolutional Networks](https://github.com/open-mmlab/mmdetection/blob/main/configs/dcn) for details.

### CARAFE: Content-Aware ReAssembly of FEatures

Please refer to [CARAFE](https://github.com/open-mmlab/mmdetection/blob/main/configs/carafe) for details.

### Instaboost

Please refer to [Instaboost](https://github.com/open-mmlab/mmdetection/blob/main/configs/instaboost) for details.

### Libra R-CNN

Please refer to [Libra R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/libra_rcnn) for details.

### Guided Anchoring

Please refer to [Guided Anchoring](https://github.com/open-mmlab/mmdetection/blob/main/configs/guided_anchoring) for details.

### FCOS

Please refer to [FCOS](https://github.com/open-mmlab/mmdetection/blob/main/configs/fcos) for details.

### FoveaBox

Please refer to [FoveaBox](https://github.com/open-mmlab/mmdetection/blob/main/configs/foveabox) for details.

### RepPoints

Please refer to [RepPoints](https://github.com/open-mmlab/mmdetection/blob/main/configs/reppoints) for details.

### FreeAnchor

Please refer to [FreeAnchor](https://github.com/open-mmlab/mmdetection/blob/main/configs/free_anchor) for details.

### Grid R-CNN (plus)

Please refer to [Grid R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/grid_rcnn) for details.

### GHM

### GHM

Please refer to [GHM](https://github.com/open-mmlab/mmdetection/blob/main/configs/ghm) for details.

### GCNet

Please refer to [GCNet](https://github.com/open-mmlab/mmdetection/blob/main/configs/gcnet) for details.

### HRNet

Please refer to [HRNet](https://github.com/open-mmlab/mmdetection/blob/main/configs/hrnet) for details.

### Mask Scoring R-CNN

Please refer to [Mask Scoring R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/ms_rcnn) for details.

### Train from Scratch

Please refer to [Rethinking ImageNet Pre-training](https://github.com/open-mmlab/mmdetection/blob/main/configs/scratch) for details.

### NAS-FPN

Please refer to [NAS-FPN](https://github.com/open-mmlab/mmdetection/blob/main/configs/nas_fpn) for details.

### ATSS

### ATSS

Please refer to [ATSS](https://github.com/open-mmlab/mmdetection/blob/main/configs/atss) for details.

### FSAF

Please refer to [FSAF](https://github.com/open-mmlab/mmdetection/blob/main/configs/fsaf) for details.

Please refer to [RegNet](https://github.com/open-mmlab/mmdetection/blob/main/configs/regnet) for details.

### Res2Net

Please refer to [Res2Net](https://github.com/open-mmlab/mmdetection/blob/main/configs/res2net) for details.

### GRoIE

Please refer to [GRoIE](https://github.com/open-mmlab/mmdetection/blob/main/configs/groie) for details.

### Dynamic R-CNN

Please refer to [Dynamic R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/dynamic_rcnn) for details.

Please refer to [PointRend](https://github.com/open-mmlab/mmdetection/blob/main/configs/point_rend) for details.

### DetectoRS

Please refer to [DetectoRS](https://github.com/open-mmlab/mmdetection/blob/main/configs/detectors) for details.

### Generalized Focal Loss

Please refer to [Generalized Focal Loss](https://github.com/open-mmlab/mmdetection/blob/main/configs/gfl) for details.

### CornerNet

Please refer to [CornerNet](https://github.com/open-mmlab/mmdetection/blob/main/configs/cornernet) for details.

### YOLOv3

Please refer to [YOLOv3](https://github.com/open-mmlab/mmdetection/blob/main/configs/yolo) for details.

### PAA

Please refer to [PAA](https://github.com/open-mmlab/mmdetection/blob/main/configs/paa) for details.

### SABL

Please refer to [SABL](https://github.com/open-mmlab/mmdetection/blob/main/configs/sabl) for details.

### CentripetalNet

### SABL

Please refer to [CentripetalNet](https://github.com/open-mmlab/mmdetection/blob/main/configs/centripetalnet) for details.

Please refer to [ResNeSt](https://github.com/open-mmlab/mmdetection/blob/main/configs/resnest) for details.

### DETR

Please refer to [DETR](https://github.com/open-mmlab/mmdetection/blob/main/configs/detr) for details.

### Deformable DETR

Please refer to [Deformable DETR](https://github.com/open-mmlab/mmdetection/blob/main/configs/deformable_detr) for details.

### AutoAssign

Please refer to [AutoAssign](https://github.com/open-mmlab/mmdetection/blob/main/configs/autoassign) for details.

### YOLOF

Please refer to [YOLOF](https://github.com/open-mmlab/mmdetection/blob/main/configs/yolof) for details.

### Seesaw Loss

### YOLOF

Please refer to [Seesaw Loss](https://github.com/open-mmlab/mmdetection/blob/main/configs/seesaw_loss) for details.

### CenterNet

Please refer to [CenterNet](https://github.com/open-mmlab/mmdetection/blob/main/configs/centernet) for details.

### YOLOX

Please refer to [YOLOX](https://github.com/open-mmlab/mmdetection/blob/main/configs/yolox) for details.

### PVT

Please refer to [PVT](https://github.com/open-mmlab/mmdetection/blob/main/configs/pvt) for details.

### SOLO

Please refer to [SOLO](https://github.com/open-mmlab/mmdetection/blob/main/configs/solo) for details.

### QueryInst

Please refer to [QueryInst](https://github.com/open-mmlab/mmdetection/blob/main/configs/queryinst) for details.

### PanopticFPN

Please refer to [PanopticFPN](https://github.com/open-mmlab/mmdetection/blob/main/configs/panoptic_fpn) for details.

### MaskFormer

Please refer to [MaskFormer](https://github.com/open-mmlab/mmdetection/blob/main/configs/maskformer) for details.

### DyHead

Please refer to [DyHead](https://github.com/open-mmlab/mmdetection/blob/main/configs/dyhead) for details.

### Mask2Former

Please refer to [Mask2Former](https://github.com/open-mmlab/mmdetection/blob/main/configs/mask2former) for details.

### Efficientnet

Please refer to [Efficientnet](https://github.com/open-mmlab/mmdetection/blob/main/configs/efficientnet) for details.

### Other datasets

We also benchmark some methods on [PASCAL VOC](https://github.com/open-mmlab/mmdetection/blob/main/configs/pascal_voc), [Cityscapes](https://github.com/open-mmlab/mmdetection/blob/main/configs/cityscapes), [OpenImages](https://github.com/open-mmlab/mmdetection/blob/main/configs/openimages) and [WIDER FACE](https://github.com/open-mmlab/mmdetection/blob/main/configs/wider_face).

We also train [Faster R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/faster_rcnn) and [Mask R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/mask_rcnn) using ResNet-50 and [RegNetX-3.2G](https://github.com/open-mmlab/mmdetection/blob/main/configs/regnet) with multi-scale training and longer schedules. These models serve as strong pre-trained models for downstream tasks for convenience.

### Training Speed benchmark

We provide [analyze_logs.py](https://github.com/open-mmlab/mmdetection/blob/main/tools/analysis_tools/analyze_logs.py) to get average time of iteration in training. You can find examples in [Log Analysis](https://mmdetection.readthedocs.io/en/latest/useful_tools.html#log-analysis).

We compare the training speed of Mask R-CNN with some other popular frameworks (The data is copied from [detectron2](https://github.com/facebookresearch/detectron2/blob/main/docs/notes/benchmarks.md/)).
For mmdetection, we benchmark with [mask-rcnn_r50-caffe_fpn_poly-1x_coco_v1.py](https://github.com/open-mmlab/mmdetection/blob/main/configs/mask_rcnn/mask-rcnn_r50-caffe_fpn_poly-1x_coco_v1.py), which should have the same setting with [mask_rcnn_R_50_FPN_noaug_1x.yaml](https://github.com/facebookresearch/detectron2/blob/main/configs/Detectron1-Comparisons/mask_rcnn_R_50_FPN_noaug_1x.yaml) of detectron2.

We also provide the [checkpoint](https://download.openmmlab.com/mmdetection/v2.0/benchmark/mask_rcnn_r50_caffe_fpn_poly_1x_coco_no_aug/mask_rcnn_r50_caffe_fpn_poly_1x_coco_no_aug_compare_20200518-10127928.pth) and [training log](https://download.openmmlab.com/mmdetection/v2.0/benchmark/mask_rcnn_r50_caffe_fpn_poly_1x_coco_no_aug/mask_rcnn_r50_caffe_fpn_poly_1x_coco_no_aug_20200518_105755.log.json) for reference. The throughput is computed as the average throughput in iterations 100-500 to skip GPU warmup time.

| Implementation                                                                         | Throughput (img/s) |
| -------------------------------------------------------------------------------------- | ------------------ |
| [Detectron2](https://github.com/facebookresearch/detectron2)                           | 62                 |
| [MMDetection](https://github.com/open-mmlab/mmdetection)                               | 61                 |

| [maskrcnn-benchmark](https://github.com/facebookresearch/maskrcnn-benchmark/)          | 53                 |
| [tensorpack](https://github.com/tensorpack/tensorpack/tree/master/examples/FasterRCNN) | 50                 |
| [simpledet](https://github.com/TuSimple/simpledet/)                                    | 39                 |
| [Detectron](https://github.com/facebookresearch/Detectron)                             | 19                 |
| [matterport/Mask_RCNN](https://github.com/matterport/Mask_RCNN/)                       | 14                 |

### Inference Speed Benchmark

We provide [benchmark.py](https://github.com/open-mmlab/mmdetection/blob/main/tools/analysis_tools/benchmark.py) to benchmark the inference latency.

The script benchmarkes the model with 2000 images and calculates the average time ignoring first 5 times. You can change the output log interval (defaults: 50) by setting `LOG-INTERVAL`.

```shell
python tools/benchmark.py ${CONFIG} ${CHECKPOINT} [--log-interval $[LOG-INTERVAL]] [--fuse-conv-bn]
```

The latency of all models in our model zoo is benchmarked without setting `fuse-conv-bn`, you can get a lower latency by setting it.

## Comparison with Detectron2

We compare mmdetection with [Detectron2](https://github.com/facebookresearch/detectron2.git) in terms of speed and performance.
We use the commit id [185c27e](https://github.com/facebookresearch/detectron2/tree/185c27e4b4d2d4c68b5627b3765420c6d7f5a659)(30/4/2020) of detectron.

For fair comparison, we install and run both frameworks on the same machine.

- PyTorch 1.4
- CUDA 10.1

### Performance

- CUDA 10.1

| Type                                                                                                                            | Lr schd | Detectron2                                                                                                                           | mmdetection | Download                                                                                                                                                                                                                                                                                                                                                         |

| [Faster R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/faster_rcnn/faster-rcnn_r50-caffe_fpn_ms-1x_coco.py) | 1x      | [37.9](https://github.com/facebookresearch/detectron2/blob/main/configs/COCO-Detection/faster_rcnn_R_50_FPN_1x.yaml)                 | 38.0        | [model](https://download.openmmlab.com/mmdetection/v2.0/benchmark/faster_rcnn_r50_caffe_fpn_mstrain_1x_coco/faster_rcnn_r50_caffe_fpn_mstrain_1x_coco-5324cff8.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/benchmark/faster_rcnn_r50_caffe_fpn_mstrain_1x_coco/faster_rcnn_r50_caffe_fpn_mstrain_1x_coco_20200429_234554.log.json)             |

| [Mask R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/mask_rcnn/mask-rcnn_r50-caffe_fpn_ms-poly-1x_coco.py)  | 1x      | [38.6 & 35.2](https://github.com/facebookresearch/detectron2/blob/main/configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_1x.yaml) | 38.8 & 35.4 | [model](https://download.openmmlab.com/mmdetection/v2.0/benchmark/mask_rcnn_r50_caffe_fpn_mstrain-poly_1x_coco/mask_rcnn_r50_caffe_fpn_mstrain-poly_1x_coco-dbecf295.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/benchmark/mask_rcnn_r50_caffe_fpn_mstrain-poly_1x_coco/mask_rcnn_r50_caffe_fpn_mstrain-poly_1x_coco_20200430_054239.log.json) |

| [Retinanet](https://github.com/open-mmlab/mmdetection/blob/main/configs/retinanet/retinanet_r50-caffe_fpn_ms-1x_coco.py)        | 1x      | [36.5](https://github.com/facebookresearch/detectron2/blob/master/configs/COCO-Detection/retinanet_R_50_FPN_1x.yaml)                 | 37.0        | [model](https://download.openmmlab.com/mmdetection/v2.0/benchmark/retinanet_r50_caffe_fpn_mstrain_1x_coco/retinanet_r50_caffe_fpn_mstrain_1x_coco-586977a0.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/benchmark/retinanet_r50_caffe_fpn_mstrain_1x_coco/retinanet_r50_caffe_fpn_mstrain_1x_coco_20200430_014748.log.json)                     |

### Training Speed

The training speed is measure with s/iter. The lower, the better.

| Type         | Detectron2 | mmdetection |
| ------------ | ---------- | ----------- |
| Faster R-CNN | 0.210      | 0.216       |
| Mask R-CNN   | 0.261      | 0.265       |
| Retinanet    | 0.200      | 0.205       |

### Inference Speed

The inference speed is measured with fps (img/s) on a single GPU, the higher, the better.
To be consistent with Detectron2, we report the pure inference speed (without the time of data loading).
For Mask R-CNN, we exclude the time of RLE encoding in post-processing.
We also include the officially reported speed in the parentheses, which is slightly higher
than the results tested on our server due to differences of hardwares.

| Type         | Detectron2  | mmdetection |
| ------------ | ----------- | ----------- |
| Faster R-CNN | 25.6 (26.3) | 22.2        |
| Mask R-CNN   | 22.5 (23.3) | 19.6        |
| Retinanet    | 17.8 (18.2) | 20.6        |

### Training memory

| Faster R-CNN | 3.0        | 3.8         |
| Mask R-CNN   | 3.4        | 3.9         |
| Retinanet    | 3.9        | 3.4         |

Community users are welcome to contribute to this project directory by performing inference speed tests on different hardware devices.

Currently tested:

- CPU
- GPU
  - AGX Orin
  - Orin NX
- ARM

## Body 2d (17 Keypoints)

- Numbers displayed in the table are inference latencies in millisecond(ms).

## WholeBody 2d (133 Keypoints)

- Data from different community users are separated by `|`.

## How To Test Speed

```shell
```

```shell

../test_images \
```

```shell

```

In this document, we show how to prepare the training data and train models required for this project.

# Hand detection

We use multiple hand pose estimation datasets to generate a hand detection dataset. The circumscribed rectangle of hand key points of is used as the detection bounding box of the hand. In our demo, we use 4 datasets supported from [MMPose](https://github.com/open-mmlab/mmpose): [FreiHAND Dataset](https://mmpose.readthedocs.io/en/latest/dataset_zoo/2d_hand_keypoint.html#freihand-dataset), [OneHand10K](https://mmpose.readthedocs.io/en/latest/dataset_zoo/2d_hand_keypoint.html#onehand10k), [RHD Dataset](https://mmpose.readthedocs.io/en/latest/dataset_zoo/2d_hand_keypoint.html#rhd-dataset) and [Halpe](https://mmpose.readthedocs.io/en/latest/dataset_zoo/2d_wholebody_keypoint.html#halpe). You can find instructions for preparing each dataset from the corresponding link.

To train the hand detection model, you need to install [MMDet](https://github.com/open-mmlab/mmdetection) and move (or link) the above datasets to `$MMDet/data/`. The folder structure should look like this:

```
├── docs
├── tests
├── tools
|── data
    |-- freihand
       │-- annotations
       │-- ..
    |-- onehand10k
       │-- ..
    |-- rhd
       │-- ..
    │-- halpe
       |-- hico_20160224_det
          │-- images
          |-- ..
       │-- ..
```

We provide a [parse_pose.py](/projects/gesture_recognition/parse_pose.py) file to convert the annotation files of the above pose datasets to a COCO-style detection annotation. Suppose you are at `$MMDet/data`, run the following command and it will generate `hand_det_train.json` and `hand_det_val.json` at `$MMDet/data/hand_det/`

```
python3 $MMAction/projects/gesture_recognition/parse_pose.py
```

```
```

The training annotation file combines the above four data sets, and the validation annotation file just uses the OneHand10K validation for a quick verification. You can also add more hand detection datasets to improve performance. Now we are done with data preparation.

## Training and inference

We provide a [config](/projects/gesture_recognition/configs/rtmdet_nano_320-8xb32_multi-dataset-hand.py) to train a [RTMDet](https://arxiv.org/abs/2212.07784) detection model. Suppose you are at `$MMDet`, you can run the follow command to train the hand detection model with 8 GPUs:

```bash
bash tools/dist_train.sh $MMAction/projects/gesture_recognition/configs/rtmdet_nano_320-8xb32_multi-dataset-hand.py 8
```

```bash
```

To see the detection result for a single image, we can use `$MMDet/demo/image_demo.py`. The follow command will do inference on a single [image](/projects/gesture_recognition/demo/hand_det.jpg) (from a video in the [jester dataset](/tools/data/jester)) and the output should be similar to [this image](/projects/gesture_recognition/demo/hand_det_out.jpg).

```bash
python3 $MMDet/demo/image_demo.py $MMAction/projects/gesture_recognition/demo/hand_det.jpg PATH_TO_HAND_DET_CHECKPOINT --out-dir='.'
```

# Pose estimation

We directly use the pose estimation model from MMPose. Please refer to [RTMPose](https://github.com/open-mmlab/mmpose/tree/main/configs/hand_2d_keypoint/rtmpose) for details.

# Gesture recognition

We use the [jester dataset](/tools/data/jester)) to train a skeleton based gesture recognition model. Please follow the link to prepare this dataset (in frames).

Once we have the jester dataset, we provide the [extract_keypoint.py](/projects/gesture_recognition/extract_keypoint.py) to extract the hand keypoints for all video frames in the dataset. This step requires the hand detection model and the pose estimation model in the above two stages. Here is an example to extract the keypoints for the dataset. You may need to modify the path to the dataset, configs or checkpoints according to your system.

```bash
ROOT_TO_JESTER='20bn-jester-v1'
POSE_CONFIG='rtmpose-m_8xb32-210e_coco-wholebody-hand-256x256.py'
POSE_CKPT='rtmdet-nano_8xb32-300e_multi-dataset-hand-320x320_20230524-f6ffed6a.pth'
DET_CONFIG='rtmdet-nano_8xb32-300e_multi-dataset-hand-320x320.py'
DET_CKPT='hand-cocktail5-4e-4-bs256-210e-b74fb594_20230320.pth'

python3 -u extract_keypoint.py $ROOT_TO_JESTER \
    --pose_config $POSE_CONFIG --pose_ckpt $POSE_CKPT \
    --det_config $DET_CONFIG --det-ckpt $DET_CKPT
```

The program will generate a `jester.pkl` file in your current directory. Then move this file to `$MMAction`. We will use this file for skeleton based gesture recognition training.

We provide a [config](/projects/gesture_recognition/configs/stgcnpp_8xb16-joint-u100-16e_jester-keypoint-2d.py) to train a STGCN++ model. Suppose you are at `$MMAction`, you can run the follow command to train the model with 8 GPUs:

```bash
bash tools/dist_train.sh $MMAction/projects/gesture_recognition/configs/stgcnpp_8xb16-joint-u100-80e_jester-keypoint-2d.py 8
```

# 准备 NTIRE21 decompression 数据集

```bibtex
}
```

测试数据集可以从其[主页](https://github.com/RenYang-home/NTIRE21_VEnh)下载。

请按照[主页](https://github.com/RenYang-home/NTIRE21_VEnh)教程生成数据集。

```text
mmagic
├── mmagic
├── tools
├── data
|   |   ├── GT
|   |   |   ├── 001
|   |   |   ├── ...
|   |   |   ├── 010
|   |   ├── LQ
|   |   |   ├── 001
|   |   |   ├── ...
|   |   |   ├── 010
|   |   ├── GT
|   |   ├── LQ
|   |   ├── GT
|   |   ├── LQ
```

# MMSegmentation Deployment

- [MMSegmentation Deployment](#mmsegmentation-deployment)
    - [Install mmseg](#install-mmseg)

[MMSegmentation](https://github.com/open-mmlab/mmsegmentation/tree/main) aka `mmseg` is an open source semantic segmentation toolbox based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com/) project.

### Install mmseg

Please follow the [installation guide](https://mmsegmentation.readthedocs.io/en/latest/get_started.html) to install mmseg.

```shell

You can use [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/tree/main/tools/deploy.py) to convert mmseg models to the specified backend models. Its detailed usage can be learned from [here](https://github.com/open-mmlab/mmdeploy/tree/main/docs/en/02-how-to-run/convert_model.md#usage).

The command below shows an example about converting `unet` model to onnx model that can be inferred by ONNX Runtime.

```shell

    --show \
```

It is crucial to specify the correct deployment config during model conversion. We've already provided builtin deployment config [files](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmseg) of all supported backends for mmsegmentation. The config filename pattern is:

```
```

Therefore, in the above example, you can also convert `unet` to other backend models by changing the deployment config file `segmentation_onnxruntime_dynamic.py` to [others](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmseg), e.g., converting to tensorrt-fp16 model by `segmentation_tensorrt-fp16_dynamic-512x1024-2048x2048.py`.

```{tip}
When converting mmseg models to tensorrt models, --device should be set to "cuda"
```

The converted model locates in the working directory like `mmdeploy_models/mmseg/ort` in the previous example. It includes:

```

```
```

in which,

The whole package **mmdeploy_models/mmseg/ort** is defined as **mmdeploy SDK model**, i.e., **mmdeploy SDK model** includes both backend model and inference meta information.

```python

```

```python
import cv2

```

```python
import cv2

  color_seg[seg == label, :] = color
```

color_seg[seg == label, :] = color
```

- Only `whole` inference mode is supported for all mmseg models.

- <i id="static_shape">PSPNet, Fast-SCNN</i> only support static shape, because [nn.AdaptiveAvgPool2d](https://github.com/open-mmlab/mmsegmentation/blob/0c87f7a0c9099844eff8e90fa3db5b0d0ca02fee/mmseg/models/decode_heads/psp_head.py#L38) is not supported by most inference backends.

- For models that only supports static shape, you should use the deployment config file of static shape such as `configs/mmseg/segmentation_tensorrt_static-1024x2048.py`.

- For users prefer deployed models generate probability feature map, put `codebase_config = dict(with_argmax=False)` in deploy configs.

# 训练与测试

- [训练与测试](#训练与测试)
    - [使用单个 GPU 进行训练](#使用单个-gpu-进行训练)
    - [使用多个 GPU 进行训练](#使用多个-gpu-进行训练)
    - [使用多台机器进行训练](#使用多台机器进行训练)
      - [同一网络中的多台机器](#同一网络中的多台机器)
      - [使用 slurm 管理的多台机器](#使用-slurm-管理的多台机器)
    - [使用单个 GPU 进行测试](#使用单个-gpu-进行测试)
    - [使用多个 GPU 进行测试](#使用多个-gpu-进行测试)
    - [使用多台机器进行测试](#使用多台机器进行测试)
      - [同一网络中的多台机器](#同一网络中的多台机器-1)
      - [使用 slurm 管理的多台机器](#使用-slurm-管理的多台机器-1)

## 训练

### 使用单个 GPU 进行训练

您可以使用 `tools/train.py` 在一台带有 CPU 和 GPU(可选) 的单机上训练模型。

下面是脚本的完整用法：

```shell
```

默认情况下，MMAction2 更倾向于使用 GPU 而不是 CPU 进行训练。如果您想在 CPU 上训练模型，请清空 `CUDA_VISIBLE_DEVICES` 或将其设置为 -1 以使 GPU 对程序不可见。

```bash
```
````

| 参数                                  | 描述                                                                                                                                                                |

| `--work-dir WORK_DIR`                 | 保存日志和权重的目标文件夹。默认为与配置文件相同名称的文件夹，位于 `./work_dirs` 下。                                                                               |
| `--resume [RESUME]`                   | 恢复训练。如果指定了路径，则从该路径恢复，如果未指定，则尝试从最新的权重自动恢复。                                                                                  |
| `--amp`                               | 启用自动混合精度训练。                                                                                                                                              |

| `--no-validate`                       | **不建议使用**。在训练期间禁用权重评估。                                                                                                                            |
| `--auto-scale-lr`                     | 根据实际批次大小和原始批次大小自动缩放学习率。                                                                                                                      |
| `--seed`                              | 随机种子。                                                                                                                                                          |
| `--diff-rank-seed`                    | 是否为不同的 rank 设置不同的种子。                                                                                                                                  |

| `--deterministic`                     | 是否为 CUDNN 后端设置确定性选项。                                                                                                                                   |
| `--cfg-options CFG_OPTIONS`           | 覆盖使用的配置中的某些设置，xxx=yyy 格式的键值对将合并到配置文件中。如果要覆盖的值是一个列表，则应采用 `key="[a,b]"` 或 `key=a,b` 的形式。该参数还允许嵌套的列表/元组值，例如 `key="[(a,b),(c,d)]"`。请注意，引号是必需的，且不允许有空格。 |
| `--launcher {none,pytorch,slurm,mpi}` | 作业启动器的选项。默认为 `none`。                                                                                                                                   |

### 使用多个 GPU 进行训练

我们提供了一个 shell 脚本使用 `torch.distributed.launch` 来启动多个 GPU 的训练任务。

```shell
```

```shell
```

| 参数       | 描述                                                                    |
| ---------- | ----------------------------------------------------------------------- |
| `CONFIG`   | 配置文件的路径。                                                        |
| `GPUS`     | 要使用的 GPU 数量。                                                     |
| `[PYARGS]` | `tools/train.py` 的其他可选参数，请参见[这里](#使用单个-gpu-进行训练)。 |

您还可以通过环境变量来指定启动器的其他参数。例如，使用以下命令将启动器的通信端口更改为 29666：

```shell
```

如果您想启动多个训练作业并使用不同的 GPU，可以通过指定不同的端口和可见设备来启动它们。

```shell
```

### 使用多台机器进行训练

#### 同一网络中的多台机器

如果您使用以太网连接的多台机器启动训练作业，可以运行以下命令：

在第一台机器上：

```

在第一台机器上：

```shell
```

在第二台机器上：

```shell
```

需要指定以下额外的环境变量来训练或测试多台机器上的模型：

| ENV_VARS      | 描述                                                             |
| `NNODES`      | 机器的总数。默认为 1。                                           |
| `NODE_RANK`   | 本地机器的索引。默认为 0。                                       |
| `PORT`        | 通信端口，在所有机器上应该保持一致。默认为 29500。               |
| `MASTER_ADDR` | 主机器的 IP 地址，在所有机器上应该保持一致。默认为 `127.0.0.1`。 |

通常，如果您没有高速网络（如 InfiniBand），则速度会比较慢。

#### 使用 slurm 管理的多台机器

如果您在使用 [slurm](https://slurm.schedmd.com/) 管理的集群上运行 MMAction2，可以使用脚本 `slurm_train.sh`。

```shell
```

下面是该脚本的参数描述。

| 参数        | 描述                                                                    |
| ----------- | ----------------------------------------------------------------------- |
| `PARTITION` | 集群中要使用的分区。                                                    |
| `JOB_NAME`  | 作业的名称，您可以自定义。                                              |
| `CONFIG`    | 配置文件的路径。                                                        |
| `[PYARGS]`  | `tools/train.py` 的其他可选参数，请参见[这里](#使用单个-gpu-进行训练)。 |

下面列出了可用于配置 slurm 作业的环境变量。

| ENV_VARS        | 描述                                                                             |

| --------------- | -------------------------------------------------------------------------------- |
| `GPUS`          | 要使用的 GPU 数量。默认为 8。                                                    |
| `GPUS_PER_NODE` | 每个节点要分配的 GPU 数量。默认为 8。                                            |
| `CPUS_PER_TASK` | 每个任务要分配的 CPU 数量（通常一个 GPU 对应一个任务）。默认为 5。               |
| `SRUN_ARGS`     | `srun` 的其他参数。可用选项可在[这里](https://slurm.schedmd.com/srun.html)找到。 |

## 测试

### 使用单个 GPU 进行测试

您可以使用 `tools/test.py` 在一台带有 CPU 和可选 GPU 的单机上测试模型。

```shell
```

默认情况下，MMAction2 更倾向于使用 GPU 而不是 CPU 进行测试。如果您想在 CPU 上测试模型，请清空 `CUDA_VISIBLE_DEVICES` 或将其设置为 -1 以使 GPU 对程序不可见。

```bash

```

```bash
```
````

| 参数                                  | 描述                                                                                                                                                                |

| `CHECKPOINT_FILE`                     | 权重文件的路径（可以是 HTTP 链接）。                                                                                                                                |
| `--work-dir WORK_DIR`                 | 保存包含评估指标的文件的目录。默认为与配置文件相同名称的文件夹，位于 `./work_dirs` 下。                                                                             |
| `--dump DUMP`                         | 存储模型的所有输出以进行离线评估的路径。                                                                                                                            |

| `--show-dir SHOW_DIR`                 | 保存结果可视化图片的目录。                                                                                                                                          |
| `--show`                              | 在窗口中可视化预测结果。                                                                                                                                            |
| `--interval INTERVAL`                 | 可视化的样本间隔。默认为 1。                                                                                                                                        |
| `--wait-time WAIT_TIME`               | 每个窗口的显示时间（单位：秒）。默认为 2。                                                                                                                          |

### 使用多个 GPU 进行测试

我们提供了一个 shell 脚本使用 `torch.distributed.launch` 来启动多个 GPU 的测试任务。

```shell
```

| 参数         | 描述                                                                   |
| ------------ | ---------------------------------------------------------------------- |
| `CONFIG`     | 配置文件的路径。                                                       |
| `CHECKPOINT` | 权重文件的路径（可以是 HTTP 链接）。                                   |
| `GPUS`       | 要使用的 GPU 数量。                                                    |

| `[PYARGS]`   | `tools/test.py` 的其他可选参数，请参见[这里](#使用单个-gpu-进行测试)。 |

```shell
```

如果您想启动多个测试作业并使用不同的 GPU，可以通过指定不同的端口和可见设备来启动它们。

```shell
```

### 使用多台机器进行测试

如果您使用以太网连接的多台机器进行测试作业，可以运行以下命令：

在第一台机器上：

```shell
```

在第二台机器上：

```shell
```

```

在第二台机器上：

```shell
```

与单台机器上的多个 GPU 相比，您需要指定一些额外的环境变量：

如果您在使用 [slurm](https://slurm.schedmd.com/) 管理的集群上运行 MMAction2，可以使用脚本 `slurm_test.sh`。

```shell
```

| 参数         | 描述                                                                   |

```

| 参数         | 描述                                                                   |
| `PARTITION`  | 集群中要使用的分区。                                                   |
| `JOB_NAME`   | 作业的名称，您可以自定义。                                             |

# Transformer in Transformer

> [Transformer in Transformer](https://arxiv.org/abs/2103.00112)

Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 16×16) as "visual sentences" and present to further divide them into smaller patches (e.g., 4×4) as "visual words". The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, e.g., we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% higher than that of the state-of-the-art visual transformer with similar computational cost.

<img src="https://user-images.githubusercontent.com/26739999/142578661-298d92a1-2e25-4910-a312-085587be6b65.png" width="80%"/>
</div>

```python

predict = inference_model('tnt-small-p16_3rdparty_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('tnt-small-p16_3rdparty_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/tnt/tnt-s-p16_16xb64_in1k.py https://download.openmmlab.com/mmclassification/v0/tnt/tnt-small-p16_3rdparty_in1k_20210903-c56ee7df.pth
```

| Model                           |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |               Config               |                                        Download                                        |
| :------------------------------ | :----------: | :--------: | :-------: | :-------: | :-------: | :--------------------------------: | :------------------------------------------------------------------------------------: |

| `tnt-small-p16_3rdparty_in1k`\* | From scratch |   23.76    |   3.36    |   81.52   |   95.73   | [config](tnt-s-p16_16xb64_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/tnt/tnt-small-p16_3rdparty_in1k_20210903-c56ee7df.pth) |

*Models with * are converted from the [official repo](https://github.com/contrastive/pytorch-image-models/blob/809271b0f3e5d9be4e11c0c5cec1dbba8b5e2c60/timm/models/tnt.py#L144). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@misc{han2021transformer,
      title={Transformer in Transformer},
      author={Kai Han and An Xiao and Enhua Wu and Jianyuan Guo and Chunjing Xu and Yunhe Wang},
      eprint={2103.00112},
}
```

  ```
  ```

  ```
  ```

  2. Install mmcv-full following the [installation instruction](https://mmcv.readthedocs.io/en/latest/#installation).

## Data

2. Install mmcv-full following the [installation instruction](https://mmcv.readthedocs.io/en/latest/#installation).

## Data

  One can choose to use gt bounding boxes to evaluate models, by setting `bbox_file=None''` in `val_dataloader.dataset` in config. Or one can use detected boxes to evaluate

  Set the environment variables `MASTER_PORT=XXX`. For example,

  `MASTER_PORT=29517 GPUS=16 GPUS_PER_NODE=8 CPUS_PER_TASK=2 ./tools/slurm_train.sh Test res50 configs/body_2d_keypoint/topdown_regression/coco/td-reg_res50_8xb64-210e_coco-256x192.py work_dirs/res50_coco_256x192`

  Use `TensorboardLoggerHook` in `log_config` like

  ```python
  log_config=dict(interval=20, hooks=[dict(type='TensorboardLoggerHook')])
  ```

  Use smaller log interval. For example, change `interval=50` to `interval=1` in the config.

  For top-down models, try to edit the config file. For example,

  1. set `flip_test=False` in `init_cfg` in the config file.
  2. use faster human bounding box detector, see [MMDetection](https://mmdetection.readthedocs.io/zh_CN/3.x/model_zoo.html).

# DCNv2

> [Deformable ConvNets v2: More Deformable, Better Results](https://arxiv.org/abs/1811.11168)

The superior performance of Deformable Convolutional Networks arises from its ability to adapt to the geometric variations of objects. Through an examination of its adaptive behavior, we observe that while the spatial support for its neural features conforms more closely than regular ConvNets to object structure, this support may nevertheless extend well beyond the region of interest, causing features to be influenced by irrelevant image content. To address this problem, we present a reformulation of Deformable ConvNets that improves its ability to focus on pertinent image regions, through increased modeling power and stronger training. The modeling power is enhanced through a more comprehensive integration of deformable convolution within the network, and by introducing a modulation mechanism that expands the scope of deformation modeling. To effectively harness this enriched modeling capability, we guide network training via a proposed feature mimicking scheme that helps the network to learn features that reflect the object focus and classification power of RCNN features. With the proposed contributions, this new version of Deformable ConvNets yields significant performance gains over the original model and produces leading results on the COCO benchmark for object detection and instance segmentation.

|     Backbone      | Model  |  Style  |     Conv      |  Pool  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                             Config                             |                                                                                                                                                                                 Download                                                                                                                                                                                  |

|     R-50-FPN      | Faster | pytorch | mdconv(c3-c5) |   -    |   1x    |   4.1    |      17.6      |  41.4  |         |    [config](./faster-rcnn_r50-mdconv-c3-c5_fpn_1x_coco.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/dcn/faster_rcnn_r50_fpn_mdconv_c3-c5_1x_coco/faster_rcnn_r50_fpn_mdconv_c3-c5_1x_coco_20200130-d099253b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/dcn/faster_rcnn_r50_fpn_mdconv_c3-c5_1x_coco/faster_rcnn_r50_fpn_mdconv_c3-c5_1x_coco_20200130_222144.log.json)               |

| \*R-50-FPN (dg=4) | Faster | pytorch | mdconv(c3-c5) |   -    |   1x    |   4.2    |      17.4      |  41.5  |         | [config](./faster-rcnn_r50-mdconv-group4-c3-c5_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/dcn/faster_rcnn_r50_fpn_mdconv_c3-c5_group4_1x_coco/faster_rcnn_r50_fpn_mdconv_c3-c5_group4_1x_coco_20200130-01262257.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/dcn/faster_rcnn_r50_fpn_mdconv_c3-c5_group4_1x_coco/faster_rcnn_r50_fpn_mdconv_c3-c5_group4_1x_coco_20200130_222058.log.json) |

|     R-50-FPN      | Faster | pytorch |       -       | mdpool |   1x    |   5.8    |      16.6      |  38.7  |         |       [config](./faster-rcnn_r50_fpn_mdpool_1x_coco.py)        |                           [model](https://download.openmmlab.com/mmdetection/v2.0/dcn/faster_rcnn_r50_fpn_mdpool_1x_coco/faster_rcnn_r50_fpn_mdpool_1x_coco_20200307-c0df27ff.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/dcn/faster_rcnn_r50_fpn_mdpool_1x_coco/faster_rcnn_r50_fpn_mdpool_1x_coco_20200307_203304.log.json)                           |

|     R-50-FPN      |  Mask  | pytorch | mdconv(c3-c5) |   -    |   1x    |   4.5    |      15.1      |  41.5  |  37.1   |     [config](./mask-rcnn_r50-mdconv-c3-c5_fpn_1x_coco.py)      |                   [model](https://download.openmmlab.com/mmdetection/v2.0/dcn/mask_rcnn_r50_fpn_mdconv_c3-c5_1x_coco/mask_rcnn_r50_fpn_mdconv_c3-c5_1x_coco_20200203-ad97591f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/dcn/mask_rcnn_r50_fpn_mdconv_c3-c5_1x_coco/mask_rcnn_r50_fpn_mdconv_c3-c5_1x_coco_20200203_063443.log.json)                   |

|  R-50-FPN (FP16)  |  Mask  | pytorch | mdconv(c3-c5) |   -    |   1x    |   3.1    |                |  42.0  |  37.6   |   [config](./mask-rcnn_r50-mdconv-c3-c5_fpn_amp-1x_coco.py)    |    [model](https://download.openmmlab.com/mmdetection/v2.0/fp16/mask_rcnn_r50_fpn_fp16_mdconv_c3-c5_1x_coco/mask_rcnn_r50_fpn_fp16_mdconv_c3-c5_1x_coco_20210520_180434-cf8fefa5.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/fp16/mask_rcnn_r50_fpn_fp16_mdconv_c3-c5_1x_coco/mask_rcnn_r50_fpn_fp16_mdconv_c3-c5_1x_coco_20210520_180434.log.json)     |

- `mdconv` denotes modulated deformable convolution, `c3-c5` means adding dconv in resnet stage 3 to 5. `mdpool` denotes modulated deformable roi pooling.
- The dcn ops are modified from https://github.com/chengdazhi/Deformable-Convolution-V2-PyTorch, which should be more memory efficient and slightly faster.

- (\*) For R-50-FPN (dg=4), dg is short for deformable_group. This model is trained and tested on Amazon EC2 p3dn.24xlarge instance.

```latex
@article{zhu2018deformable,
  title={Deformable ConvNets v2: More Deformable, Better Results},
  author={Zhu, Xizhou and Hu, Han and Lin, Stephen and Dai, Jifeng},
  journal={arXiv preprint arXiv:1811.11168},
}
```

|   Model   | Input Size | Whole AP | Whole AR |                          Details and Download                           |
| :-------: | :--------: | :------: | :------: | :---------------------------------------------------------------------: |
| RTMPose-m |  256x192   |  0.582   |  0.674   | [rtmpose_coco-wholebody.md](./coco-wholebody/rtmpose_coco-wholebody.md) |
| RTMPose-l |  256x192   |  0.611   |  0.700   | [rtmpose_coco-wholebody.md](./coco-wholebody/rtmpose_coco-wholebody.md) |
| RTMPose-l |  384x288   |  0.648   |  0.730   | [rtmpose_coco-wholebody.md](./coco-wholebody/rtmpose_coco-wholebody.md) |

<details>

```bibtex
}
```

<details>

```bibtex

}
```

Results on Human3.6M dataset with ground truth 2D detections

| Arch                                                            | MPJPE | P-MPJPE |                              ckpt                               |                              log                               |
| :-------------------------------------------------------------- | :---: | :-----: | :-------------------------------------------------------------: | :------------------------------------------------------------: |

| [SimpleBaseline3D-tcn<sup>1</sup>](/configs/body_3d_keypoint/pose_lift/h36m/pose-lift_simplebaseline3d_8xb64-200e_h36m.py) | 43.4  |  34.3   | [ckpt](https://download.openmmlab.com/mmpose/body3d/simple_baseline/simple3Dbaseline_h36m-f0ad73a4_20210419.pth) | [log](https://download.openmmlab.com/mmpose/body3d/simple_baseline/20210415_065056.log.json) |

<sup>1</sup> Differing from the original paper, we didn't apply the `max-norm constraint` because we found this led to a better convergence and performance.

# ByteTrack: Multi-Object Tracking by Associating Every Detection Box

Multi-object tracking (MOT) aims at estimating bounding boxes and identities of objects in videos. Most methods obtain identities by associating detection boxes whose scores are higher than a threshold. The objects with low detection scores, e.g. occluded objects, are simply thrown away, which brings non-negligible true object missing and fragmented trajectories. To solve this problem, we present a simple, effective and generic association method, tracking by associating every detection box instead of only the high score ones. For the low score detection boxes, we utilize their similarities with tracklets to recover true objects and filter out the background detections. When applied to 9 different state-of-the-art trackers, our method achieves consistent improvement on IDF1 score ranging from 1 to 10 points. To put forwards the state-of-the-art performance of MOT, we design a simple and strong tracker, named ByteTrack. For the first time, we achieve 80.3 MOTA, 77.3 IDF1 and 63.1 HOTA on the test set of MOT17 with 30 FPS running speed on a single V100 GPU.

  <img src="https://user-images.githubusercontent.com/26813582/147467498-b8d16d8c-8472-4830-8bac-b107c49f7c6f.png"/>
</div>

```latex
@inproceedings{zhang2021bytetrack,
  title={ByteTrack: Multi-Object Tracking by Associating Every Detection Box},
  author={Zhang, Yifu and Sun, Peize and Jiang, Yi and Yu, Dongdong and Yuan, Zehuan and Luo, Ping and Liu, Wenyu and Wang, Xinggang},
  journal={arXiv preprint arXiv:2110.06864},
}
```

Please note that the performance on `MOT17-half-val` is comparable with the performance reported in the manuscript, while the performance on `MOT17-test` is lower than the performance reported in the manuscript.

The reason is that ByteTrack tunes customized hyper-parameters (e.g., image resolution and the high threshold of detection score) for each video in `MOT17-test` set, while we use unified parameters.

|  Method   | Detector |           Train Set           |    Test Set    | Public | Inf time (fps) | HOTA | MOTA | IDF1 |  FP   |  FN   | IDSw. |                                         Config                                          |                                                                                                                                                           Download                                                                                                                                                           |

| ByteTrack | YOLOX-X  | CrowdHuman + MOT17-half-train | MOT17-half-val |   N    |       -        | 67.5 | 78.6 | 78.5 | 12852 | 21060 |  672  | [config](bytetrack_yolox_x_8xb4-amp-80e_crowdhuman-mot17halftrain_test-mot17halfval.py) | [model](https://download.openmmlab.com/mmtracking/mot/bytetrack/bytetrack_yolox_x/bytetrack_yolox_x_crowdhuman_mot17-private-half_20211218_205500-1985c9f0.pth) \| [log](https://download.openmmlab.com/mmtracking/mot/bytetrack/bytetrack_yolox_x/bytetrack_yolox_x_crowdhuman_mot17-private-half_20211218_205500.log.json) |

| ByteTrack | YOLOX-X  | CrowdHuman + MOT17-half-train |   MOT17-test   |   N    |       -        | 61.7 | 78.1 | 74.8 | 36705 | 85032 | 2049  |  [config](bytetrack_yolox_x_8xb4-amp-80e_crowdhuman-mot17halftrain_test-mot17test.py)   | [model](https://download.openmmlab.com/mmtracking/mot/bytetrack/bytetrack_yolox_x/bytetrack_yolox_x_crowdhuman_mot17-private-half_20211218_205500-1985c9f0.pth) \| [log](https://download.openmmlab.com/mmtracking/mot/bytetrack/bytetrack_yolox_x/bytetrack_yolox_x_crowdhuman_mot17-private-half_20211218_205500.log.json) |

## Results and models on MOT20

Since there are only 4 videos in `MOT20-train`, ByteTrack is validated on `MOT17-train` rather than `MOT20-half-train`.

Please note that the MOTA on `MOT20-test` is slightly lower than that reported in the manuscript, because we don't tune the threshold for each video.

|  Method   | Detector |        Train Set         |  Test Set   | Public | Inf time (fps) | HOTA | MOTA | IDF1 |   FP   |   FN   | IDSw. |                                      Config                                      |                                                                                                                                                      Download                                                                                                                                                      |

| ByteTrack | YOLOX-X  | CrowdHuman + MOT20-train | MOT17-train |   N    |       -        | 57.3 | 64.9 | 71.8 | 33,747 | 83,385 | 1,263 | [config](bytetrack_yolox_x_8xb4-amp-80e_crowdhuman-mot20train_test-mot20test.py) | [model](https://download.openmmlab.com/mmtracking/mot/bytetrack/bytetrack_yolox_x/bytetrack_yolox_x_crowdhuman_mot20-private_20220506_101040-9ce38a60.pth) \| [log](https://download.openmmlab.com/mmtracking/mot/bytetrack/bytetrack_yolox_x/bytetrack_yolox_x_crowdhuman_mot20-private_20220506_101040.log.json) |

| ByteTrack | YOLOX-X  | CrowdHuman + MOT20-train | MOT20-test  |   N    |       -        | 61.5 | 77.0 | 75.4 | 33,083 | 84,433 | 1,345 | [config](bytetrack_yolox_x_8xb4-amp-80e_crowdhuman-mot20train_test-mot20test.py) | [model](https://download.openmmlab.com/mmtracking/mot/bytetrack/bytetrack_yolox_x/bytetrack_yolox_x_crowdhuman_mot20-private_20220506_101040-9ce38a60.pth) \| [log](https://download.openmmlab.com/mmtracking/mot/bytetrack/bytetrack_yolox_x/bytetrack_yolox_x_crowdhuman_mot20-private_20220506_101040.log.json) |

**3.1 Joint training and tracking**

Some algorithm like ByteTrack, OCSORT don't need reid model, so we provide joint training and tracking for convenient.

```shell
# Training Bytetrack on crowdhuman and mot17-half-train dataset with following command
# The number after config file represents the number of GPUs used. Here we use 8 GPUs
bash tools/dist_train.sh configs/bytetrack/bytetrack_yolox_x_8xb4-80e_crowdhuman-mot17halftrain_test-mot17halfval.py 8
```

**3.2 Separate training and tracking**

Of course, we provide train detector independently like SORT, DeepSORT, StrongSORT. Then use this detector to track.

```shell

```

```shell
bash tools/dist_train.sh configs/bytetrack/yolox_x_8xb4-amp-80e_crowdhuman-mot17halftrain_test-mot17halfval.py 8
```

**4.1.1 use joint trained detector to evaluating and testing**

```shell
bash tools/dist_test_tracking.sh configs/bytetrack/bytetrack_yolox_x_8xb4-amp-80e_crowdhuman-mot17halftrain_test-mot17halfval.py 8 --checkpoint ${CHECKPOINT_FILE}
```

```shell
```

**4.1.2 use separate trained detector to evaluating and testing**

```shell
bash tools/dist_test_tracking.sh configs/bytetrack/bytetrack_yolox_x_8xb4-amp-80e_crowdhuman-mot17halftrain_test-mot17halfval.py 8 --detector ${CHECKPOINT_FILE}
```

**4.1.3 use video_baesd to evaluating and testing**

```
```

#### 4.2 Example on MOTxx-test dataset

```
```

If you want to get the results of the [MOT Challenge](https://motchallenge.net/) test set, please use the following command to generate result files that can be used for submission. It will be stored in `./mot_17_test_res`, you can modify the saved path in `test_evaluator` of the config.

```shell
bash tools/dist_test_tracking.sh configs/bytetrack/bytetrack_yolox_x_8xb4-amp-80e_crowdhuman-mot17halftrain_test-mot17test.py 8 --checkpoint ${CHECKPOINT_FILE}
```

```shell

```shell
python demo/mot_demo.py demo/demo_mot.mp4 configs/bytetrack/bytetrack_yolox_x_8xb4-amp-80e_crowdhuman-mot17halftrain_test-mot17halfval.py --checkpoint ${CHECKPOINT_FILE} --out mot.mp4
```

# Learning delicate local representations for multi-person pose estimation

<details>

```bibtex
}
```

In this paper, we propose a novel method called Residual Steps Network (RSN). RSN aggregates features with the same spatial size (Intra-level features) efficiently to obtain delicate local representations, which retain rich low-level spatial information and result in precise keypoint localization. Additionally, we observe the output features contribute differently to final performance. To tackle this problem, we propose an efficient attention mechanism - Pose Refine Machine (PRM) to make a trade-off between local and global representations in output features and further refine the keypoint locations. Our approach won the 1st place of COCO Keypoint Challenge 2019 and achieves state-of-the-art results on both COCO and MPII benchmarks, without using extra training data and pretrained model. Our single model achieves 78.6 on COCO test-dev, 93.0 on MPII test dataset. Ensembled models achieve 79.2 on COCO test-dev, 77.1 on COCO test-challenge dataset.

<img src="https://user-images.githubusercontent.com/15977946/146522226-5041d16e-41cb-4d31-ae53-dbe21314a697.png">
</div>

# Dataset

In MMOCR, all the datasets are processed via different Dataset classes based on [mmengine.BaseDataset](mmengine.dataset.BaseDataset). Dataset classes are responsible for loading the data and performing initial parsing, then fed to [data pipeline](./transforms.md) for data preprocessing, augmentation, formatting, etc.

</div>

In this tutorial, we will introduce some common interfaces of the Dataset class, and the usage of Dataset implementations in MMOCR as well as the annotation types they support.

```{tip}

</div>

```{tip}
Dataset class supports some advanced features, such as lazy initialization and data serialization, and takes advantage of various dataset wrappers to perform data concatenation, repeating, and category balancing. These content will not be covered in this tutorial, but you can read {external+mmengine:doc}`MMEngine: BaseDataset <advanced_tutorials/basedataset>` for more details.
```

## Common Interfaces

Now, let's look at a concrete example and learn some typical interfaces of a Dataset class.

```

`OCRDataset` is a widely used Dataset implementation in MMOCR, and is suggested as a default Dataset type in MMOCR as its associated annotation format is flexible enough to support *all* the OCR tasks ([more info](#ocrdataset)). Now we will instantiate an `OCRDataset` object wherein the toy dataset in `tests/data/det_toy_dataset` will be loaded.

```python

    dict(
    dict(
    ),
    dict(

),
    dict(
]

```

Let's peek the size of this dataset:

```python

10
```

Typically, a Dataset class loads and stores two types of information: (1) **meta information**: Some meta descriptors of the dataset's property, such as available object categories in this dataset. (2) **annotation**: The path to images, and their labels. We can access the meta information in `dataset.metainfo`:

```python

```

```python

```

As for the annotations, we can access them via `dataset.get_data_info(idx)`, which returns a dictionary containing the information of the `idx`-th sample in the dataset that is initially parsed, but not yet processed by [data pipeline](./transforms.md).

```python

                ...,

```

On the other hand, we can get the sample fully processed by data pipeline via `dataset[idx]` or `dataset.__getitem__(idx)`, which is directly feedable to models and perform a full train/test cycle. It has two fields:

- `inputs`: The image after data augmentation;
- `data_samples`: The [DataSample](./structures.md) that contains the augmented annotations, and meta information appended by some data transforms to keep track of some key properties of this sample.

```python

ignored: tensor([ True, False,  True])
```

## Dataset Classes and Annotation Formats

Each Dataset implementation can only load datasets in a specific annotation format. Here lists all supported Dataset classes and their compatible annotation formats, as well as an example config that showcases how to use them in practice.

```{note}
If you are not familiar with the config system, you may find [Dataset Configuration](../user_guides/dataset_prepare.md#dataset-configuration) helpful.
```

```{note}
```

Usually, there are many different types of annotations in OCR datasets, and the formats often vary between different subtasks, such as text detection and text recognition. These differences can result in the need for different data loading code when using different datasets, increasing the learning and maintenance costs for users.

In MMOCR, we propose a unified dataset format that can adapt to all three subtasks of OCR: text detection, text recognition, and text spotting. This design maximizes the uniformity of the dataset, allows for the reuse of data annotations across different tasks, and makes dataset management more convenient. Considering that popular dataset formats are still inconsistent, MMOCR provides [Dataset Preparer](../user_guides/data_prepare/dataset_preparer.md) to help users convert their datasets to MMOCR format. We also strongly encourage researchers to develop their own datasets based on this data format.

#### Annotation Format

This annotation file is a `.json` file that stores a `dict`, containing both `metainfo` and `data_list`, where the former includes basic information about the dataset and the latter consists of the label item of each target instance. Here presents an extensive list of all the fields in the annotation file, but some fields are used in a subset of tasks and can be ignored in other tasks.

```python
{
    {
      "dataset_type": "TextDetDataset",  # Options: TextDetDataset/TextRecogDataset/TextSpotterDataset
      "task_name": "textdet",  #  Options: textdet/textspotter/textrecog
      "category": [{"id": 0, "name": "text"}]  # Used in textdet/textspotter
    },
    [
      {
        "instances":  # multiple instances in one image
        [
          {
            "bbox": [0, 0, 10, 20],  # in textdet/textspotter, [x1, y1, x2, y2].

"width": 640,
        [
          {
            "bbox_label": 0,  # The object category, always 0 (text) in MMOCR
            "polygon": [0, 0, 0, 10, 10, 20, 20, 0], # in textdet/textspotter. [x1, y1, x2, y2, ....]
            "text": "mmocr",  # in textspotter/textrecog
            "ignore": False # in textspotter/textdet. Whether to ignore this sample during training
          },
          #...
        ],
      }
      #... multiple images
    ]
}
```

#### Example Config

},
          #...
        ],
      }
    ]
}
```

Here is a part of config example where we make `train_dataloader` use `OCRDataset` to load the ICDAR2015 dataset for a text detection model. Keep in mind that `OCRDataset` can load any OCR datasets prepared by Dataset Preparer regardless of its task. That is, you can use it for text recognition and text spotting, but you still have to modify the transform types in `pipeline` according to the needs of different tasks.

```python
    dict(
    dict(
    ),
    dict(
]

meta_keys=('img_path', 'ori_shape', 'img_shape'))
]

```

Reading images or labels from files can be slow when data are excessive, e.g. on a scale of millions. Besides, in academia, most of the scene text recognition datasets are stored in lmdb format, including images and labels. ([Example](https://github.com/clovaai/deep-text-recognition-benchmark))

To get closer to the mainstream practice and enhance the data storage efficiency, MMOCR supports loading images and labels from lmdb datasets via `RecogLMDBDataset`.

MMOCR requires the following keys for LMDB datasets:

- `num_samples`: The parameter describing the data volume of the dataset.
- The keys of images and labels are in the
  format of `image-000000001` and `label-000000001`, respectively. The index starts from 1.

MMOCR has a toy LMDB dataset in `tests/data/rec_toy_dataset/imgs.lmdb`.
You can get a sense of the format with the following code snippet.

```python
>>>

```

Here is a part of config example where we make `train_dataloader` use `RecogLMDBDataset` to load the toy dataset. Since `RecogLMDBDataset` loads images as numpy arrays, don't forget to use `LoadImageFromNDArray` instead of `LoadImageFromFile` in the pipeline for successful loading.

```python
    dict(
    dict(

```python
    dict(
    dict(
    ),
    dict(
]

```

Prior to MMOCR 1.0, MMOCR 0.x takes text files as input for text recognition. These formats has been deprecated in MMOCR 1.0, and this class could be removed anytime in the future. [More info](../migration/dataset.md)

Text files can either be in `txt` format or `jsonl` format. The simple `.txt` annotations separate image name and word annotation by a blank space, which cannot handle the case when spaces are included in a text instance.

```text
```

```json
```

Here is a part of config example where we use `RecogTextDataset` to load the old txt labels in training, and the old jsonl labels in testing.

```python
    dict(
    dict(
    ),
    dict(

dict(
    ),
    dict(
]

type='RecogTextDataset',

```

Prior to MMOCR 1.0, MMOCR 0.x takes COCO-like format annotations as input for text detection. These formats has been deprecated in MMOCR 1.0, and this class could be removed anytime in the future. [More info](../migration/dataset.md)

```json
{
    {
      "id": 1,
    }
  ],
    {
      "id": 1,

}
  ],
    {
      "id": 1,
      ],
    }
  ]
}
```

Here is a part of config example where we make `train_dataloader` use `IcdarDataset` to load the old labels.

```python
    dict(
    dict(
    ),
    dict(
]

```

It's customized for [WildReceipt](https://mmocr.readthedocs.io/en/dev-1.x/user_guides/data_prepare/datasetzoo.html#wildreceipt) dataset only.

```json
{
    [
      {
        "label": 1
      },
      {
      }
    ], //...
}

// Open Set
{

"label": 1
      },
      {
      }
    ], //...
}

// Open Set
{
    [
      {
        "edge": 1
      },
      {
        "edge": 1
      }
    ]
}
```

Please refer to [SDMGR's config](https://github.com/open-mmlab/mmocr/blob/f30c16ce96bd2393570c04eeb9cf48a7916315cc/configs/kie/sdmgr/sdmgr_novisual_60e_wildreceipt.py) for more details.

We provide demo scripts to inference a given video or a folder that contains continuous images. The source codes are available [here](https://github.com/open-mmlab/mmdetection/tree/tracking/demo).

Note that if you use a folder as the input, the image names there must be  **sortable** , which means we can re-order the images according to the numbers contained in the filenames. We now only support reading the images whose filenames end with `.jpg`, `.jpeg` and `.png`.

## Inference MOT models

This script can inference an input video / images with a multiple object tracking or video instance segmentation model.

```shell
python demo/mot_demo.py \
    ${INPUTS}
    [--checkpoint ${CHECKPOINT_FILE}] \
    [--detector ${DETECTOR_FILE}] \
    [--reid ${REID_FILE}] \
    [--out ${OUTPUT}] \
    [--show]
```

The `INPUT` and `OUTPUT` support both _mp4 video_ format and the _folder_ format.

[--score-thr ${SCORE_THR}] \
    [--show]
```

**Important:** For `DeepSORT`, `SORT`, `StrongSORT`, they need load the weight of the `reid` and the weight of the `detector` separately. Therefore, we use `--detector` and `--reid` to load weights. Other algorithms such as `ByteTrack`, `OCSORT` `QDTrack` `MaskTrackRCNN` and `Mask2Former` use `--checkpoint` to load weights.

- `CHECKPOINT_FILE`: The checkpoint is optional.
- `DETECTOR_FILE`: The detector is optional.
- `REID_FILE`: The reid is optional.
- `SCORE_THR`: The threshold of score to filter bboxes.
- `DEVICE`: The device for inference. Options are `cpu` or `cuda:0`, etc.

**Examples of running mot model:**

```shell
# Example 1: do not specify --checkpoint to use --detector
    demo/demo_mot.mp4 \
    configs/sort/sort_faster-rcnn_r50_fpn_8xb2-4e_mot17halftrain_test-mot17halfval.py \
    --detector \
    https://download.openmmlab.com/mmtracking/mot/faster_rcnn/faster-rcnn_r50_fpn_4e_mot17-half-64ee2ed4.pth \
    --out mot.mp4

# Example 2: use --checkpoint
    configs/qdtrack/qdtrack_faster-rcnn_r50_fpn_8xb2-4e_mot17halftrain_test-mot17halfval.py \
    --checkpoint https://download.openmmlab.com/mmtracking/mot/qdtrack/mot_dataset/qdtrack_faster-rcnn_r50_fpn_4e_mot17_20220315_145635-76f295ef.pth \
```

After decades of development, the OCR community has produced a series of related datasets that often provide annotations of text in a variety of styles, making it necessary for users to convert these datasets to the required format when using them. MMOCR supports dozens of commonly used text-related datasets and provides a [data preparation script](./data_prepare/dataset_preparer.md) to help users prepare the datasets with only one command.

In this section, we will introduce a typical process of preparing a dataset for MMOCR:

1. [Download datasets and convert its format to the suggested one](#downloading-datasets-and-converting-format)
2. [Modify the config file](#dataset-configuration)

However, the first step is not necessary if you already have a dataset in the format that MMOCR supports. You can read [Dataset Classes](../basic_concepts/datasets.md#dataset-classes-and-annotation-formats) for more details.

## Downloading Datasets and Converting Format

As an example of the data preparation steps, you can use the following command to prepare the ICDAR 2015 dataset for text detection task.

```shell
```

Then, the dataset has been downloaded and converted to MMOCR format, and the file directory structure is as follows:

```text
│   ├── test
│   └── train
```

Once your dataset has been prepared, you can use the [browse_dataset.py](./useful_tools.md#dataset-visualization-tool) to visualize the dataset and check if the annotations are correct.

```bash
```

## Dataset Configuration

### Single Dataset Training

```bash
```

When training or evaluating a model on new datasets, we need to write the dataset config where the image path, annotation path, and image prefix are set. The path `configs/xxx/_base_/datasets/` is pre-configured with the commonly used datasets in MMOCR (if you use `prepare_dataset.py` to prepare dataset, this config will be generated automatically), here we take the ICDAR 2015 dataset as an example (see `configs/textdet/_base_/datasets/icdar2015.py`).

```Python
icdar2015_textdet_data_root = 'data/icdar2015' # dataset root path

# Train set config
    data_root=icdar2015_textdet_data_root,               # dataset root path
    ann_file='textdet_train.json',                       # name of annotation

type='OCRDataset',
    filter_cfg=dict(filter_empty_gt=True, min_size=32),  # filtering empty images
# Test set config
```

After configuring the dataset, we can import it in the corresponding model configs. For example, to train the "DBNet_R18" model on the ICDAR 2015 dataset.

```Python
_base_ = [
    '../_base_/datasets/icdar2015.py',  # import the dataset config
]

icdar2015_textdet_train = _base_.icdar2015_textdet_train            # specify the training set

]

icdar2015_textdet_train.pipeline = _base_.train_pipeline   # specify the training pipeline
icdar2015_textdet_test = _base_.icdar2015_textdet_test              # specify the testing set
icdar2015_textdet_test.pipeline = _base_.test_pipeline     # specify the testing pipeline

    dataset=icdar2015_textdet_train)    # specify the dataset in train_dataloader

    dataset=icdar2015_textdet_test)    # specify the dataset in val_dataloader

```

### Multi-dataset Training

dataset=icdar2015_textdet_test)    # specify the dataset in val_dataloader

```

In addition, [`ConcatDataset`](mmocr.datasets.ConcatDataset) enables users to train or test the model on a combination of multiple datasets. You just need to set the dataset type in the dataloader to `ConcatDataset` in the configuration file and specify the corresponding list of datasets.

```Python
```

For example, the following configuration uses the MJSynth dataset for training and 6 academic datasets (CUTE80, IIIT5K, SVT, SVTP, ICDAR2013, ICDAR2015) for testing.

```Python
_base_ = [ # Import all dataset configurations you want to use

]

# List of training datasets
# List of testing datasets
]

# Use ConcatDataset to combine the datasets in the list

# ICNet

> [ICNet for Real-time Semantic Segmentation on High-resolution Images](https://arxiv.org/abs/1704.08545)

<a href="https://github.com/hszhao/ICNet">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.18.0/mmseg/models/necks/ic_neck.py#L77">Code Snippet</a>

We focus on the challenging task of real-time semantic segmentation in this paper. It finds many practical applications and yet is with fundamental difficulty of reducing a large portion of computation for pixel-wise label inference. We propose an image cascade network (ICNet) that incorporates multi-resolution branches under proper label guidance to address this challenge. We provide in-depth analysis of our framework and introduce the cascade feature fusion unit to quickly achieve high-quality segmentation. Our system yields real-time inference on a single GPU card with decent quality results evaluated on challenging datasets like Cityscapes, CamVid and COCO-Stuff.

<img src="https://user-images.githubusercontent.com/24582831/142901772-4570455d-7b27-44ae-a690-47dd9fde8445.png" width="70%"/>

</div>

| Method           | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                | download                                                                                                                                                                                                                                                                                                                                                                               |

| ICNet            | R-18-D8  | 832x832   |   80000 | 1.70     | 27.12          | V100   | 68.14 |         70.16 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/icnet/icnet_r18-d8_4xb2-80k_cityscapes-832x832.py)            | [model](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r18-d8_832x832_80k_cityscapes/icnet_r18-d8_832x832_80k_cityscapes_20210925_225521-2e36638d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r18-d8_832x832_80k_cityscapes/icnet_r18-d8_832x832_80k_cityscapes_20210925_225521.log.json)                                             |

| ICNet            | R-18-D8  | 832x832   |  160000 | -        | -              | V100   | 71.64 |         74.18 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/icnet/icnet_r18-d8_4xb2-160k_cityscapes-832x832.py)           | [model](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r18-d8_832x832_160k_cityscapes/icnet_r18-d8_832x832_160k_cityscapes_20210925_230153-2c6eb6e0.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r18-d8_832x832_160k_cityscapes/icnet_r18-d8_832x832_160k_cityscapes_20210925_230153.log.json)                                         |

| ICNet (in1k-pre) | R-18-D8  | 832x832   |   80000 | -        | -              | V100   | 72.51 |         74.78 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/icnet/icnet_r18-d8-in1k-pre_4xb2-80k_cityscapes-832x832.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r18-d8_in1k-pre_832x832_80k_cityscapes/icnet_r18-d8_in1k-pre_832x832_80k_cityscapes_20210925_230354-1cbe3022.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r18-d8_in1k-pre_832x832_80k_cityscapes/icnet_r18-d8_in1k-pre_832x832_80k_cityscapes_20210925_230354.log.json)         |

| ICNet (in1k-pre) | R-18-D8  | 832x832   |  160000 | -        | -              | V100   | 74.43 |         76.72 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/icnet/icnet_r18-d8-in1k-pre_4xb2-160k_cityscapes-832x832.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r18-d8_in1k-pre_832x832_160k_cityscapes/icnet_r18-d8_in1k-pre_832x832_160k_cityscapes_20210926_052702-619c8ae1.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r18-d8_in1k-pre_832x832_160k_cityscapes/icnet_r18-d8_in1k-pre_832x832_160k_cityscapes_20210926_052702.log.json)     |

| ICNet            | R-50-D8  | 832x832   |   80000 | 2.53     | 20.08          | V100   | 68.91 |         69.72 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/icnet/icnet_r50-d8_4xb2-80k_cityscapes-832x832.py)            | [model](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r50-d8_832x832_80k_cityscapes/icnet_r50-d8_832x832_80k_cityscapes_20210926_044625-c6407341.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r50-d8_832x832_80k_cityscapes/icnet_r50-d8_832x832_80k_cityscapes_20210926_044625.log.json)                                             |

| ICNet            | R-50-D8  | 832x832   |  160000 | -        | -              | V100   | 73.82 |         75.67 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/icnet/icnet_r50-d8_4xb2-160k_cityscapes-832x832.py)           | [model](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r50-d8_832x832_160k_cityscapes/icnet_r50-d8_832x832_160k_cityscapes_20210925_232612-a95f0d4e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r50-d8_832x832_160k_cityscapes/icnet_r50-d8_832x832_160k_cityscapes_20210925_232612.log.json)                                         |

| ICNet (in1k-pre) | R-50-D8  | 832x832   |   80000 | -        | -              | V100   | 74.58 |         76.41 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/icnet/icnet_r50-d8-in1k-pre_4xb2-80k_cityscapes-832x832.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r50-d8_in1k-pre_832x832_80k_cityscapes/icnet_r50-d8_in1k-pre_832x832_80k_cityscapes_20210926_032943-1743dc7b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r50-d8_in1k-pre_832x832_80k_cityscapes/icnet_r50-d8_in1k-pre_832x832_80k_cityscapes_20210926_032943.log.json)         |

| ICNet (in1k-pre) | R-50-D8  | 832x832   |  160000 | -        | -              | V100   | 76.29 |         78.09 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/icnet/icnet_r50-d8-in1k-pre_4xb2-160k_cityscapes-832x832.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r50-d8_in1k-pre_832x832_160k_cityscapes/icnet_r50-d8_in1k-pre_832x832_160k_cityscapes_20210926_042715-ce310aea.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r50-d8_in1k-pre_832x832_160k_cityscapes/icnet_r50-d8_in1k-pre_832x832_160k_cityscapes_20210926_042715.log.json)     |

| ICNet            | R-101-D8 | 832x832   |   80000 | 3.08     | 16.95          | V100   | 70.28 |         71.95 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/icnet/icnet_r101-d8_4xb2-80k_cityscapes-832x832.py)           | [model](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r101-d8_832x832_80k_cityscapes/icnet_r101-d8_832x832_80k_cityscapes_20210926_072447-b52f936e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r101-d8_832x832_80k_cityscapes/icnet_r101-d8_832x832_80k_cityscapes_20210926_072447.log.json)                                         |

| ICNet            | R-101-D8 | 832x832   |  160000 | -        | -              | V100   | 73.80 |         76.10 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/icnet/icnet_r101-d8_4xb2-160k_cityscapes-832x832.py)          | [model](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r101-d8_832x832_160k_cityscapes/icnet_r101-d8_832x832_160k_cityscapes_20210926_092350-3a1ebf1a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r101-d8_832x832_160k_cityscapes/icnet_r101-d8_832x832_160k_cityscapes_20210926_092350.log.json)                                     |

| ICNet (in1k-pre) | R-101-D8 | 832x832   |   80000 | -        | -              | V100   | 75.57 |         77.86 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/icnet/icnet_r101-d8-in1k-pre_4xb2-80k_cityscapes-832x832.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r101-d8_in1k-pre_832x832_80k_cityscapes/icnet_r101-d8_in1k-pre_832x832_80k_cityscapes_20210926_020414-7ceb12c5.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r101-d8_in1k-pre_832x832_80k_cityscapes/icnet_r101-d8_in1k-pre_832x832_80k_cityscapes_20210926_020414.log.json)     |

| ICNet (in1k-pre) | R-101-D8 | 832x832   |  160000 | -        | -              | V100   | 76.15 |         77.98 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/icnet/icnet_r101-d8-in1k-pre_4xb2-160k_cityscapes-832x832.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r101-d8_in1k-pre_832x832_160k_cityscapes/icnet_r101-d8_in1k-pre_832x832_160k_cityscapes_20210925_232612-9484ae8a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/icnet/icnet_r101-d8_in1k-pre_832x832_160k_cityscapes/icnet_r101-d8_in1k-pre_832x832_160k_cityscapes_20210925_232612.log.json) |

Note: `in1k-pre` means pretrained model is used.

@inproceedings{zhao2018icnet,
  title={Icnet for real-time semantic segmentation on high-resolution images},
  author={Zhao, Hengshuang and Qi, Xiaojuan and Shen, Xiaoyong and Shi, Jianping and Jia, Jiaya},

author={Zhao, Hengshuang and Qi, Xiaojuan and Shen, Xiaoyong and Shi, Jianping and Jia, Jiaya},
  pages={405--420},
}
```

# RobustScanner

> [RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition](https://arxiv.org/abs/2007.07542)

The attention-based encoder-decoder framework has recently achieved impressive results for scene text recognition, and many variants have emerged with improvements in recognition quality. However, it performs poorly on contextless texts (e.g., random character sequences) which is unacceptable in most of real application scenarios. In this paper, we first deeply investigate the decoding process of the decoder. We empirically find that a representative character-level sequence decoder utilizes not only context information but also positional information. Contextual information, which the existing approaches heavily rely on, causes the problem of attention drift. To suppress such side-effect, we propose a novel position enhancement branch, and dynamically fuse its outputs with those of the decoder attention module for scene text recognition. Specifically, it contains a position aware module to enable the encoder to output feature vectors encoding their own spatial positions, and an attention module to estimate glimpses using the positional clue (i.e., the current decoding time step) only. The dynamic fusion is conducted for more robust feature via an element-wise gate mechanism. Theoretically, our proposed method, dubbed \\emph{RobustScanner}, decodes individual characters with dynamic ratio between context and positional clues, and utilizes more positional ones when the decoding sequences with scarce context, and thus is robust and practical. Empirically, it has achieved new state-of-the-art results on popular regular and irregular text recognition benchmarks while without much performance drop on contextless benchmarks, validating its robustness in both contextual and contextless application scenarios.

<img src="https://user-images.githubusercontent.com/22607038/142798010-eee8795e-8cda-4a7f-a81d-ff9c94af58dc.png"/>
</div>

## Dataset

|                               Methods                                | GPUs |        | Regular Text |           |     |           | Irregular Text |        |                               download                                |

| :------------------------------------------------------------------: | :--: | :----: | :----------: | :-------: | :-: | :-------: | :------------: | :----: | :-------------------------------------------------------------------: |
|                                                                      |      | IIIT5K |     SVT      | IC13-1015 |     | IC15-2077 |      SVTP      |  CT80  |                                                                       |
| [RobustScanner](/configs/textrecog/robust_scanner/robustscanner_resnet31_5e_st-sub_mj-sub_sa_real.py) |  4   | 0.9510 |    0.9011    |  0.9320   |     |  0.7578   |     0.8078     | 0.8750 | [model](https://download.openmmlab.com/mmocr/textrecog/robust_scanner/robustscanner_resnet31_5e_st-sub_mj-sub_sa_real/robustscanner_resnet31_5e_st-sub_mj-sub_sa_real_20220915_152447-7fc35929.pth) \| [log](https://download.openmmlab.com/mmocr/textrecog/robust_scanner/robustscanner_resnet31_5e_st-sub_mj-sub_sa_real/20220915_152447.log) |

| [RobustScanner-TTA](/configs/textrecog/robust_scanner/robustscanner_resnet31_5e_st-sub_mj-sub_sa_real.py) |  4   | 0.9487 |    0.9011    |  0.9261   |     |  0.7805   |     0.8124     | 0.8819 |                                                                       |

<a id="1">\[1\]</a> Li, Hui and Wang, Peng and Shen, Chunhua and Zhang, Guyu. Show, attend and read: A simple and strong baseline for irregular text recognition. In AAAI 2019.

```bibtex
@inproceedings{yue2020robustscanner,
  title={RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition},
  author={Yue, Xiaoyu and Kuang, Zhanghui and Lin, Chenhao and Sun, Hongbin and Zhang, Wayne},
  booktitle={European Conference on Computer Vision},
}
```

# 准备 DPDD 数据集

```bibtex
}
```

测试数据集可以从 [此处](https://drive.google.com/file/d/1dDWUQ_D93XGtcywoUcZE1HOXCV4EuLyw/) 下载。

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── DPDD
```

# Serving a model

## Serving [LLaMA-2](https://github.com/facebookresearch/llama)

You can download [llama-2 models from huggingface](https://huggingface.co/meta-llama) and serve them like below:

<summary><b>7B</b></summary>

```shell
python3 -m lmdeploy.serve.turbomind.deploy llama2 /path/to/llama-2-7b-chat-hf
```

<summary><b>13B</b></summary>

```shell
python3 -m lmdeploy.serve.turbomind.deploy llama2 /path/to/llama-2-13b-chat-hf --tp 2
```

<summary><b>70B</b></summary>

```shell
python3 -m lmdeploy.serve.turbomind.deploy llama2 /path/to/llama-2-70b-chat-hf --tp 8
```

## Serving [LLaMA](https://github.com/facebookresearch/llama)

```

Weights for the LLaMA models can be obtained from by filling out [this form](https://docs.google.com/forms/d/e/1FAIpQLSfqNECQnMkycAp2jP4Z9TFX0cGR4uf7b_fBxjY_OjhJILlKGA/viewform)

```shell
python3 -m lmdeploy.serve.turbomind.deploy llama /path/to/llama-7b llama \
    --tokenizer_path /path/to/tokenizer/model
```

```shell
python3 -m lmdeploy.serve.turbomind.deploy llama /path/to/llama-13b llama \
    --tokenizer_path /path/to/tokenizer/model --tp 2
```

<summary><b>30B</b></summary>

```shell
python3 -m lmdeploy.serve.turbomind.deploy llama /path/to/llama-30b llama \
    --tokenizer_path /path/to/tokenizer/model --tp 4
```

```shell
```

<summary><b>65B</b></summary>

```shell
python3 -m lmdeploy.serve.turbomind.deploy llama /path/to/llama-65b llama \
    --tokenizer_path /path/to/tokenizer/model --tp 8
```

### Serving [Vicuna](https://lmsys.org/blog/2023-03-30-vicuna/)

```shell
python3 -m pip install fschat
python3 -m fastchat.model.apply_delta \
  --base-model-path /path/to/llama-7b \
  --target-model-path /path/to/vicuna-7b \
  --delta-path lmsys/vicuna-7b-delta-v1.1

python3 -m lmdeploy.serve.turbomind.deploy vicuna /path/to/vicuna-7b
```

```shell

```

```shell
  --base-model-path /path/to/llama-13b \
  --target-model-path /path/to/vicuna-13b \
  --delta-path lmsys/vicuna-13b-delta-v1.1

python3 -m lmdeploy.serve.turbomind.deploy vicuna /path/to/vicuna-13b
```

# Very Deep Convolutional Networks for Large-Scale Image Recognition

<details>

```bibtex
}
```

journal={arXiv preprint arXiv:1409.1556},
}
```

<img src="https://user-images.githubusercontent.com/15977946/146540877-b0fc7823-efe2-4b5c-8168-3b3a9d7e93ac.png">
</div>

# BYOL

> [Bootstrap your own latent: A new approach to self-supervised Learning](https://arxiv.org/abs/2006.07733)

**B**ootstrap **Y**our **O**wn **L**atent (BYOL) is a new approach to self-supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other. From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view. At the same time, we update the target network with a slow-moving average of the online network.

<img src="https://user-images.githubusercontent.com/36138628/149720208-5ffbee78-1437-44c7-9ddb-b8caab60d2c3.png" width="800" />
</div>

```python

</div>

```python

predict = inference_model('resnet50_byol-pre_8xb512-linear-coslr-90e_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('byol_resnet50_16xb256-coslr-200e_in1k', pretrained=True)
```

Train:

```shell
python tools/train.py configs/byol/byol_resnet50_16xb256-coslr-200e_in1k.py
```

Test:

```shell

Train:

```shell
```

Test:

```shell
python tools/test.py configs/byol/benchmarks/resnet50_8xb512-linear-coslr-90e_in1k.py https://download.openmmlab.com/mmselfsup/1.x/byol/byol_resnet50_16xb256-coslr-200e_in1k/resnet50_linear-8xb512-coslr-90e_in1k/resnet50_linear-8xb512-coslr-90e_in1k_20220825-7596c6f5.pth
```

| Model                                   | Params (M) | Flops (G) |                       Config                       |                                           Download                                           |
| :-------------------------------------- | :--------: | :-------: | :------------------------------------------------: | :------------------------------------------------------------------------------------------: |

| `byol_resnet50_16xb256-coslr-200e_in1k` |   68.02    |   4.11    | [config](byol_resnet50_16xb256-coslr-200e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/byol/byol_resnet50_16xb256-coslr-200e_in1k/byol_resnet50_16xb256-coslr-200e_in1k_20220825-de817331.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/byol/byol_resnet50_16xb256-coslr-200e_in1k/byol_resnet50_16xb256-coslr-200e_in1k_20220825-de817331.json) |

| `resnet50_byol-pre_8xb512-linear-coslr-90e_in1k` | [BYOL](https://download.openmmlab.com/mmselfsup/1.x/byol/byol_resnet50_16xb256-coslr-200e_in1k/byol_resnet50_16xb256-coslr-200e_in1k_20220825-de817331.pth) |   25.56    |   4.11    |   71.80   | [config](benchmarks/resnet50_8xb512-linear-coslr-90e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/byol/byol_resnet50_16xb256-coslr-200e_in1k/resnet50_linear-8xb512-coslr-90e_in1k/resnet50_linear-8xb512-coslr-90e_in1k_20220825-7596c6f5.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/byol/byol_resnet50_16xb256-coslr-200e_in1k/resnet50_linear-8xb512-coslr-90e_in1k/resnet50_linear-8xb512-coslr-90e_in1k_20220825-7596c6f5.json) |

```bibtex
@inproceedings{grill2020bootstrap,
  title={Bootstrap your own latent: A new approach to self-supervised learning},

```bibtex
  author={Grill, Jean-Bastien and Strub, Florian and Altch{\'e}, Florent and Tallec, Corentin and Richemond, Pierre H and Buchatskaya, Elena and Doersch, Carl and Pires, Bernardo Avila and Guo, Zhaohan Daniel and Azar, Mohammad Gheshlaghi and others},
}
```

<details>

```bibtex
}
```

<details>

```bibtex

```bibtex
}
```

<details>

```bibtex
}
```

| [pose_hrnet_w32_udp](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_udp-8xb64-210e_coco-256x192.py) |  256x192   | 0.762 |      0.907      |      0.829      | 0.810 |      0.942      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_udp-8xb64-210e_coco-256x192-73ede547_20220914.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_udp-8xb64-210e_coco-256x192_20220914.log) |

| [pose_hrnet_w32_udp](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_udp-8xb64-210e_coco-384x288.py) |  384x288   | 0.768 |      0.909      |      0.832      | 0.815 |      0.945      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_udp-8xb64-210e_coco-384x288-9a3f7c85_20220914.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_udp-8xb64-210e_coco-384x288_20220914.log) |

| [pose_hrnet_w48_udp](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_udp-8xb32-210e_coco-256x192.py) |  256x192   | 0.768 |      0.908      |      0.833      | 0.817 |      0.945      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_udp-8xb32-210e_coco-256x192-3feaef8f_20220913.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_udp-8xb32-210e_coco-256x192_20220913.log) |

| [pose_hrnet_w48_udp](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_udp-8xb32-210e_coco-384x288.py) |  384x288   | 0.773 |      0.911      |      0.836      | 0.821 |      0.946      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_udp-8xb32-210e_coco-384x288-70d7ab01_20220913.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_udp-8xb32-210e_coco-384x288_20220913.log) |

| [pose_hrnet_w32_udp_regress](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_udp-regress-8xb64-210e_coco-256x192.py) |  256x192   | 0.759 |      0.907      |      0.827      | 0.813 |      0.943      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_udp-regress-8xb64-210e_coco-256x192-9c0b77b4_20220926.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_udp-regress-8xb64-210e_coco-256x192_20220226.log) |

## net.param
### example
```
7767517
3 3
Input         input    0 1 data 0=4 1=4 2=1
InnerProduct  ip       1 1 data fc 0=10 1=1 2=80
Softmax       softmax  1 1 fc prob 0=0
```
### overview
```
[magic]
```
* magic number : 7767517
```
[layer count] [blob count]
```
* layer count : count of the layer line follows, should be exactly the count of all layer names
* blob count : count of all blobs, usually greater than or equals to the layer count
### layer line
```
[layer type] [layer name] [input count] [output count] [input blobs] [output blobs] [layer specific params]
```
* layer type : type name, such as Convolution Softmax etc
* layer name : name of this layer, must be unique among all layer names
* input count : count of the blobs this layer needs as input
* output count : count of the blobs this layer produces as output
* input blobs : name list of all the input blob names, separated by space, must be unique among input blob names of all layers

* output blobs : name list of all the output blob names, separated by space, must be unique among output blob names of all layers
* layer specific params : key=value pair list, separated by space
### layer param
```
0=1 1=2.5 -23303=2,2.0,3.0
```
key index should be unique in each layer line, pair can be omitted if the default value used

the meaning of existing param key index can be looked up at [operation-param-weight-table](operation-param-weight-table)

* integer or float key : index 0 ~ 19
* integer value : int
* float value : float
* integer array or float array key : -23300 minus index 0 ~ 19
* integer array value : [array size],int,int,...,int
* float array value : [array size],float,float,...,float

## net.bin
```
  +---------+---------+---------+---------+---------+---------+

## net.bin
```
  | weight1 | weight2 | weight3 | weight4 | ....... | weightN |
  ^         ^         ^         ^
  0x0      0x80      0x140     0x1C0
```
the model binary is the concatenation of all weight data, each weight buffer is aligned by 32bit

### weight buffer
```
[flag] (optional)
[raw data]
[padding] (optional)
```
* flag : unsigned int,  little-endian, indicating the weight storage type, 0 => float32, 0x01306B47 => float16, otherwise => quantized int8, may be omitted if the layer implementation forced the storage type explicitly
* raw data : raw weight data, little-endian, float32 data or float16 data or quantized table and indexes depending on the storage type flag

* padding : padding space for 32bit alignment, may be omitted if already aligned

<details>

```bibtex
}
```

<details>

```bibtex
}
```

| [LiteHRNet-18](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_litehrnet-18_8xb64-210e_coco-256x192.py) |  256x192   | 0.642 |      0.867      |      0.719      | 0.705 |      0.911      | [ckpt](https://download.openmmlab.com/mmpose/top_down/litehrnet/litehrnet18_coco_256x192-6bace359_20211230.pth) | [log](https://download.openmmlab.com/mmpose/top_down/litehrnet/litehrnet18_coco_256x192_20211230.log.json) |
| [LiteHRNet-18](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_litehrnet-18_8xb32-210e_coco-384x288.py) |  384x288   | 0.676 |      0.876      |      0.746      | 0.735 |      0.919      | [ckpt](https://download.openmmlab.com/mmpose/top_down/litehrnet/litehrnet18_coco_384x288-8d4dac48_20211230.pth) | [log](https://download.openmmlab.com/mmpose/top_down/litehrnet/litehrnet18_coco_384x288_20211230.log.json) |

| [LiteHRNet-30](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_litehrnet-30_8xb64-210e_coco-256x192.py) |  256x192   | 0.676 |      0.880      |      0.756      | 0.736 |      0.922      | [ckpt](https://download.openmmlab.com/mmpose/top_down/litehrnet/litehrnet30_coco_256x192-4176555b_20210626.pth) | [log](https://download.openmmlab.com/mmpose/top_down/litehrnet/litehrnet30_coco_256x192_20210626.log.json) |
| [LiteHRNet-30](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_litehrnet-30_8xb32-210e_coco-384x288.py) |  384x288   | 0.700 |      0.883      |      0.776      | 0.758 |      0.926      | [ckpt](https://download.openmmlab.com/mmpose/top_down/litehrnet/litehrnet30_coco_384x288-a3aef5c4_20210626.pth) | [log](https://download.openmmlab.com/mmpose/top_down/litehrnet/litehrnet30_coco_384x288_20210626.log.json) |

# 模型评测

```{note}
阅读此文档前，建议您先了解 {external+mmengine:doc}`MMEngine: 模型精度评测基本概念 <tutorials/evaluation>`。
```

## 评测指标

MMOCR 基于 {external+mmengine:doc}`MMEngine: BaseMetric <design/evaluation>` 基类实现了常用的文本检测、文本识别以及关键信息抽取任务的评测指标，用户可以通过修改配置文件中的 `val_evaluator` 与 `test_evaluator` 字段来便捷地指定验证与测试阶段采用的评测方法。例如，以下配置展示了如何在文本检测算法中使用 `HmeanIOUMetric` 来评测模型性能。

```python
# 文本检测任务中通常使用 HmeanIOUMetric 来评测模型性能
val_evaluator = [dict(type='HmeanIOUMetric')]

# 此外，MMOCR 也支持相同任务下的多种指标组合评测，如同时使用 WordMetric 及 CharMetric
]
```

```{tip}
更多评测相关配置请参考[评测配置教程](../user_guides/config.md#评测配置)。
```

如下表所示，MMOCR 目前针对文本检测、识别、及关键信息抽取等任务共内置了 5 种评测指标，分别为 `HmeanIOUMetric`，`WordMetric`，`CharMetric`，`OneMinusNEDMetric`，和 `F1Metric`。

```

```{tip}
```

|                                         |              |                                                   |                                                                       |
| --------------------------------------- | ------------ | ------------------------------------------------- | --------------------------------------------------------------------- |
| 评测指标                                | 任务类型     | 输入字段                                          | 输出字段                                                              |
| [HmeanIOUMetric](#hmeanioumetric)       | 文本检测     | `pred_polygons`<br>`pred_scores`<br>`gt_polygons` | `recall`<br>`precision`<br>`hmean`                                    |

| [WordMetric](#wordmetric)               | 文本识别     | `pred_text`<br>`gt_text`                          | `word_acc`<br>`word_acc_ignore_case`<br>`word_acc_ignore_case_symbol` |
| [CharMetric](#charmetric)               | 文本识别     | `pred_text`<br>`gt_text`                          | `char_recall`<br>`char_precision`                                     |
| [OneMinusNEDMetric](#oneminusnedmetric) | 文本识别     | `pred_text`<br>`gt_text`                          | `1-N.E.D`                                                             |
| [F1Metric](#f1metric)                   | 关键信息抽取 | `pred_labels`<br>`gt_labels`                      | `macro_f1`<br>`micro_f1`                                              |

通常来说，每一类任务所采用的评测标准是约定俗成的，用户一般无须深入了解或手动修改评测方法的内部实现。然而，为了方便用户实现更加定制化的需求，本文档将进一步介绍了 MMOCR 内置评测算法的具体实现策略，以及可配置参数。

[HmeanIOUMetric](mmocr.evaluation.metrics.hmean_iou_metric.HmeanIOUMetric) 是文本检测任务中应用最广泛的评测指标之一，因其计算了检测精度（Precision）与召回率（Recall）之间的调和平均数（Harmonic mean, H-mean），故得名 `HmeanIOUMetric`。记精度为 *P*，召回率为 *R*，则 `HmeanIOUMetric` 可由下式计算得到：

```{math}
```

另外，由于其等价于 {math}`\beta = 1` 时的 F-score (又称 F-measure 或 F-metric)，`HmeanIOUMetric` 有时也被写作 `F1Metric` 或 `f1-score` 等：

```{math}
```

在 MMOCR 的设计中，`HmeanIOUMetric` 的计算可以概括为以下几个步骤：

1. 过滤无效的预测边界盒

   - 依据置信度阈值 `pred_score_thrs` 过滤掉得分较低的预测边界盒
   - 依据 `ignore_precision_thr` 阈值过滤掉与 `ignored` 样本重合度过高的预测边界盒

```

   值得注意的是，`pred_score_thrs` 默认将**自动搜索**一定范围内的**最佳阈值**，用户也可以通过手动修改配置文件来自定义搜索范围：

   ```python
   # HmeanIOUMetric 默认以 0.1 为步长搜索 [0.3, 0.9] 范围内的最佳得分阈值
   ```

2. 计算 IoU 矩阵

   - 在数据处理阶段，`HmeanIOUMetric` 会计算并维护一个 {math}`M \times N` 的 IoU 矩阵 `iou_metric`，以方便后续的边界盒配对步骤。其中，M 和 N 分别为标签边界盒与过滤后预测边界盒的数量。由此，该矩阵的每个元素都存放了第 m 个标签边界盒与第 n 个预测边界盒之间的交并比（IoU）。

3. 基于相应的配对策略统计能被准确匹配的 GT 样本数

   尽管 `HmeanIOUMetric` 可以由固定的公式计算取得，不同的任务或算法库内部的具体实现仍可能存在一些细微差别。这些差异主要体现在采用不同的策略来匹配真实与预测边界盒，从而导致最终得分的差距。目前，MMOCR 内部的 `HmeanIOUMetric` 共支持两种不同的匹配策略，即 `vanilla` 与 `max_matching`。如下所示，用户可以通过修改配置文件来指定不同的匹配策略。

   - `vanilla` 匹配策略

- `vanilla` 匹配策略

     `HmeanIOUMetric` 默认采用 `vanilla` 匹配策略，该实现与 MMOCR 0.x 版本中的 `hmean-iou` 及 ICDAR 系列**官方文本检测竞赛的评测标准保持一致**，采用先到先得的匹配方式对标签边界盒（Ground-truth bbox）与预测边界盒（Predicted bbox）进行配对。

     ```python
     # 不指定 strategy 时，HmeanIOUMetric 默认采用 'vanilla' 匹配策略
     ```

   - `max_matching` 匹配策略

     针对现有匹配机制中的不完善之处，MMOCR 算法库实现了一套更高效的匹配策略，用以最大化匹配数目。

     ```python
     # 指定采用 'max_matching' 匹配策略
     ```

   ```{note}
   我们建议面向学术研究的开发用户采用默认的 `vanilla` 匹配策略，以保证与其他论文的对比结果保持一致。而面向工业应用的开发用户则可以采用 `max_matching` 匹配策略，以获得精准的结果。
   ```

4. 根据上文介绍的 `HmeanIOUMetric` 公式计算最终的评测得分

[WordMetric](mmocr.evaluation.metrics.recog_metric.WordMetric) 实现了**单词级别**的文本识别评测指标，并内置了 `exact`，`ignore_case`，及 `ignore_case_symbol` 三种文本匹配模式，用户可以在配置文件中修改 `mode` 字段来自由组合输出一种或多种文本匹配模式下的 `WordMetric` 得分。

```python
# 在文本识别任务中使用 WordMetric 评测

```python
]
```

- `exact`：全匹配模式，即，预测与标签完全一致才能被记录为正确样本。
- `ignore_case`：忽略大小写的匹配模式。
- `ignore_case_symbol`：忽略大小写及符号的匹配模式，这也是大部分学术论文中报告的文本识别准确率；MMOCR 报告的识别模型性能默认采用该匹配模式。

假设真实标签为 `MMOCR!`，模型的输出结果为 `mmocr`，则三种匹配模式下的 `WordMetric` 得分分别为：`{'exact': 0, 'ignore_case': 0, 'ignore_case_symbol': 1}`。

[CharMetric](mmocr.evaluation.metrics.recog_metric.CharMetric) 实现了**不区分大小写**的**字符级别**的文本识别评测指标。

```python
# 在文本识别任务中使用 CharMetric 评测
```

具体而言，`CharMetric` 会输出两个评测评测指标，即字符精度 `char_precision` 和字符召回率 `char_recall`。设正确预测的字符（True Positive）数量为 {math}`\sigma_{tp}`，则精度 *P* 和召回率 *R* 可由下式计算取得：

```{math}
```

其中，{math}`\sigma_{gt}` 与 {math}`\sigma_{pred}` 分别为标签文本与预测文本所包含的字符总数。

例如，假设标签文本为 "MM**O**CR"，预测文本为 "mm**0**cR**1**"，则使用 `CharMetric` 评测指标的得分为：

```

```{math}
```

[`OneMinusNEDMetric(1-N.E.D)`](mmocr.evaluation.metrics.recog_metric.OneMinusNEDMetric) 常用于中文或英文**文本行级别**标注的文本识别评测，不同于全匹配的评测标准要求预测与真实样本完全一致，该评测指标使用归一化的[编辑距离](https://en.wikipedia.org/wiki/Edit_distance)（Edit Distance，又名莱温斯坦距离 Levenshtein Distance）来测量预测文本与真实文本之间的差异性，从而在评测长文本样本时能够更好地区分出模型的性能差异。假设真实和预测文本分别为 {math}`s_i` 和 {math}`\hat{s_i}`，其长度分别为 {math}`l_{i}` 和 {math}`\hat{l_i}`，则 `OneMinusNEDMetric` 得分可由下式计算得到：

```{math}
```

其中，*N* 是样本总数，{math}`D(s_1, s_2)` 为两个字符串之间的编辑距离。

例如，假设真实标签为 "OpenMMLabMMOCR"，模型 A 的预测结果为 "0penMMLabMMOCR", 模型 B 的预测结果为 "uvwxyz"，则采用全匹配和 `OneMinusNEDMetric` 评测指标的结果分别为:

|        |        |            |
| ------ | ------ | ---------- |
|        | 全匹配 | 1 - N.E.D. |

|        |        |            |
| 模型 A | 0      | 0.92857    |
| 模型 B | 0      | 0          |

由上表可以发现，尽管模型 A 仅预测错了一个字母，而模型 B 全部预测错误，在使用全匹配的评测指标时，这两个模型的得分都为0；而使用 `OneMinuesNEDMetric` 的评测指标则能够更好地区分模型在**长文本**上的性能差异。

[F1Metric](mmocr.evaluation.metrics.f_metric.F1Metric) 实现了针对 KIE 任务的 F1-Metric 评测指标，并提供了 `micro` 和 `macro` 两种评测模式。

```python
]
```

- `micro` 模式：依据 True Positive，False Negative，及 False Positive 总数来计算全局 F1-Metric 得分。

- `macro` 模式：依据类别标签计算每一类的 F1-Metric，并求平均值。

### 自定义评测指标

对于追求更高定制化功能的用户，MMOCR 也支持自定义实现不同类型的评测指标。一般来说，用户只需要新建自定义评测指标类 `CustomizedMetric` 并继承 {external+mmengine:doc}`MMEngine: BaseMetric <design/evaluation>`，然后分别重写数据格式处理方法 `process` 以及指标计算方法 `compute_metrics`。最后，将其加入 `METRICS` 注册器即可实现任意定制化的评测指标。

```python

```python

    """ process 接收两个参数，分别为 data_batch 存放真实标签信息，以及 predictions
        存放预测结果。process 方法负责将标签信息转换并存放至 self.results 变量中
    """
        pass

    """ compute_metric 使用经过 process 方法处理过的标签数据计算最终评测得分
    """
        pass
```

```{note}
更多内容可参见 {external+mmengine:doc}`MMEngine 文档: BaseMetric <design/evaluation>`。
```

# 准备 Something-Something V1

## 简介

```
}
```

用户可参考该数据集的 [官网](https://20bn.com/datasets/something-something/v1)，以获取数据集相关的基本信息。
在数据集准备前，请确保命令行当前路径为 `$MMACTION2/tools/data/sthv1/`。

由于 Something-Something V1 的官方网站已经失效，用户需要通过第三方源下载原始数据集。下载好的标注文件需要放在 `$MMACTION2/data/sthv1/annotations` 文件夹下。

## 步骤 2. 准备 RGB 帧

官方数据集并未提供原始视频文件，只提供了对原视频文件进行抽取得到的 RGB 帧，用户可在第三方源直接下载视频帧。

将下载好的压缩文件放在 `$MMACTION2/data/sthv1/` 文件夹下，并使用以下脚本进行解压。

```shell
cd $MMACTION2/data/sthv1/

```shell
cat 20bn-something-something-v1-?? | tar zx
cd $MMACTION2/tools/data/sthv1/
```

如果用户只想使用 RGB 帧，则可以跳过中间步骤至步骤 5 以直接生成视频帧的文件列表。
由于官网的 JPG 文件名形如 "%05d.jpg" （比如，"00001.jpg"），需要在配置文件的 `data.train`, `data.val` 和 `data.test` 处添加 `"filename_tmpl='{:05}.jpg'"` 代码，以修改文件名模板。

```
    val=dict(

pipeline=val_pipeline),
```

## 步骤 3. 抽取光流

如果用户只想使用原 RGB 帧加载训练，则该部分是 **可选项**。

```shell
mkdir /mnt/SSD/sthv1_extracted/
ln -s /mnt/SSD/sthv1_extracted/ ../../../data/sthv1/rawframes
```

如果想抽取光流，则可以运行以下脚本从 RGB 帧中抽取出光流。

```shell
```

## 步骤 4: 编码视频

如果用户只想使用 RGB 帧加载训练，则该部分是 **可选项**。

用户可以运行以下命令进行视频编码。

```shell
```

## 步骤 5. 生成文件列表

```shell
```

## 步骤 6. 检查文件夹结构

```

```shell
```

在完成所有 Something-Something V1 数据集准备流程后，

在整个 MMAction2 文件夹下，Something-Something V1 的文件结构如下：

```
mmaction2
├── tools
├── data
│   ├── sthv1
│   │   ├── sthv1_{train,val}_list_rawframes.txt
│   │   ├── sthv1_{train,val}_list_videos.txt
│   |   |   ├──...
│   |   |   ├── 1
│   |   |   ├── 2
│   |   |   ├── ...

```

│   |   |   ├── 2
│   |   |   ├── ...

```

关于对 Something-Something V1 进行训练和验证，请参考 [训练和测试教程](/docs/en/user_guides/train_test.md)。

# Designing Network Design Spaces

<img src="https://user-images.githubusercontent.com/79644370/144025148-b73002cb-3c82-42e4-8da4-65df97aead9c.png" width="800"/>
</div>

We implement RegNetX models in 3D detection systems and provide their first results with PointPillars on nuScenes and Lyft dataset.

The pre-trained modles are converted from [model zoo of pycls](https://github.com/facebookresearch/pycls/blob/master/MODEL_ZOO.md) and maintained in [mmcv](https://github.com/open-mmlab/mmcv).

## Usage

We already prepare models of FLOPs from 800M to 12G in our model zoo.

|                                        Backbone                                         | Lr schd | Mem (GB) | Inf time (fps) |  mAP  | NDS  |                                                                                                                                                                                                                       Download                                                                                                                                                                                                                       |

|       [SECFPN](../pointpillars/pointpillars_hv_secfpn_sbn-all_8xb4-2x_nus-3d.py)        |   2x    |   16.4   |                | 35.17 | 49.7 |                     [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_secfpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_secfpn_sbn-all_4x8_2x_nus-3d_20200620_230725-0817d270.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_secfpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_secfpn_sbn-all_4x8_2x_nus-3d_20200620_230725.log.json)                     |

| [RegNetX-400MF-SECFPN](./pointpillars_hv_regnet-400mf_secfpn_sbn-all_8xb4-2x_nus-3d.py) |   2x    |   16.4   |                | 41.2  | 55.2 | [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/regnet/hv_pointpillars_regnet-400mf_secfpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_regnet-400mf_secfpn_sbn-all_4x8_2x_nus-3d_20200620_230334-53044f32.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/regnet/hv_pointpillars_regnet-400mf_secfpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_regnet-400mf_secfpn_sbn-all_4x8_2x_nus-3d_20200620_230334.log.json) |

|          [FPN](../pointpillars/pointpillars_hv_fpn_sbn-all_8xb4-2x_nus-3d.py)           |   2x    |   17.1   |                | 40.0  | 53.3 |                           [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d_20200620_230405-2fa62f3d.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d_20200620_230405.log.json)                           |

|    [RegNetX-400MF-FPN](./pointpillars_hv_regnet-400mf_fpn_sbn-all_8xb4-2x_nus-3d.py)    |   2x    |   17.3   |                | 44.8  | 56.4 |       [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/regnet/hv_pointpillars_regnet-400mf_fpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_regnet-400mf_fpn_sbn-all_4x8_2x_nus-3d_20200620_230239-c694dce7.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/regnet/hv_pointpillars_regnet-400mf_fpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_regnet-400mf_fpn_sbn-all_4x8_2x_nus-3d_20200620_230239.log.json)       |

|    [RegNetX-1.6gF-FPN](./pointpillars_hv_regnet-1.6gf_fpn_sbn-all_8xb4-2x_nus-3d.py)    |   2x    |   24.0   |                | 48.2  | 59.3 |       [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/regnet/hv_pointpillars_regnet-1.6gf_fpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_regnet-1.6gf_fpn_sbn-all_4x8_2x_nus-3d_20200629_050311-dcd4e090.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/regnet/hv_pointpillars_regnet-1.6gf_fpn_sbn-all_4x8_2x_nus-3d/hv_pointpillars_regnet-1.6gf_fpn_sbn-all_4x8_2x_nus-3d_20200629_050311.log.json)       |

### Lyft

### Lyft

|                                        Backbone                                         | Lr schd | Mem (GB) | Inf time (fps) | Private Score | Public Score |                                                                                                                                                                                                                         Download                                                                                                                                                                                                                         |

|       [SECFPN](../pointpillars/pointpillars_hv_secfpn_sbn-all_8xb2-2x_lyft-3d.py)       |   2x    |   12.2   |                |     13.9      |     14.1     |                     [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_secfpn_sbn-all_2x8_2x_lyft-3d/hv_pointpillars_secfpn_sbn-all_2x8_2x_lyft-3d_20210517_204807-2518e3de.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_secfpn_sbn-all_2x8_2x_lyft-3d/hv_pointpillars_secfpn_sbn-all_2x8_2x_lyft-3d_20210517_204807.log.json)                     |

| [RegNetX-400MF-SECFPN](./hv_pointpillars_regnet-400mf_secfpn_sbn-all_4x8_2x_lyft-3d.py) |   2x    |   15.9   |                |     14.9      |     15.1     | [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/regnet/hv_pointpillars_regnet-400mf_secfpn_sbn-all_2x8_2x_lyft-3d/hv_pointpillars_regnet-400mf_secfpn_sbn-all_2x8_2x_lyft-3d_20210524_092151-42513826.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/regnet/hv_pointpillars_regnet-400mf_secfpn_sbn-all_2x8_2x_lyft-3d/hv_pointpillars_regnet-400mf_secfpn_sbn-all_2x8_2x_lyft-3d_20210524_092151.log.json) |

|          [FPN](../pointpillars/pointpillars_hv_fpn_sbn-all_8xb2-2x_lyft-3d.py)          |   2x    |   9.2    |                |     14.9      |     15.1     |                           [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_fpn_sbn-all_2x8_2x_lyft-3d/hv_pointpillars_fpn_sbn-all_2x8_2x_lyft-3d_20210517_202818-fc6904c3.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_fpn_sbn-all_2x8_2x_lyft-3d/hv_pointpillars_fpn_sbn-all_2x8_2x_lyft-3d_20210517_202818.log.json)                           |

|    [RegNetX-400MF-FPN](./hv_pointpillars_regnet-400mf_fpn_sbn-all_4x8_2x_lyft-3d.py)    |   2x    |   13.0   |                |     16.0      |     16.1     |       [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/regnet/hv_pointpillars_regnet-400mf_fpn_sbn-all_2x8_2x_lyft-3d/hv_pointpillars_regnet-400mf_fpn_sbn-all_2x8_2x_lyft-3d_20210521_115618-823dcf18.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/regnet/hv_pointpillars_regnet-400mf_fpn_sbn-all_2x8_2x_lyft-3d/hv_pointpillars_regnet-400mf_fpn_sbn-all_2x8_2x_lyft-3d_20210521_115618.log.json)       |

```latex
}
```

# Models

We usually define a neural network in a deep learning task as a model, and this model is the core of an algorithm. [MMEngine](https://github.com/open-mmlab/mmengine) abstracts a unified model [BaseModel](https://github.com/open-mmlab/mmengine/blob/main/mmengine/model/base_model/base_model.py#L16) to standardize the interfaces for training, testing and other processes. All models implemented by MMSegmentation inherit from `BaseModel`, and in MMSegmentation we implemented forward and added some functions for the semantic segmentation algorithm.

## Common components

### Segmentor

In MMSegmentation, we abstract the network architecture as a **Segmentor**, it is a model that contains all components of a network. We have already implemented **EncoderDecoder** and **CascadeEncoderDecoder**, which typically consist of **Data preprocessor**, **Backbone**, **Decode head** and **Auxiliary head**.

### Data preprocessor

**Data preprocessor** is the part that copies data to the target device and preprocesses the data into the model input format.

**Backbone** is the part that transforms an image to feature maps, such as a **ResNet-50** without the last fully connected layer.

### Neck

**Neck** is the part that connects the backbone and heads. It performs some refinements or reconfigurations on the raw feature maps produced by the backbone. An example is **Feature Pyramid Network (FPN)**.

### Decode head

**Decode head** is the part that transforms the feature maps into a segmentation mask, such as **PSPNet**.

### Auxiliary head

**Auxiliary head** is an optional component that transforms the feature maps into segmentation masks which only used for computing auxiliary losses.

## Basic interfaces

MMSegmentation wraps `BaseModel` and implements the [BaseSegmentor](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/models/segmentors/base.py#L15) class, which mainly provides the interfaces `forward`, `train_step`, `val_step` and `test_step`. The following will introduce these interfaces in detail.

### forward

![EncoderDecoder dataflow](https://user-images.githubusercontent.com/15952744/228827860-c0e34875-d370-4736-84f0-9560c26c9576.png)
![CascadeEncoderDecoder dataflow](https://user-images.githubusercontent.com/15952744/228827987-aa214507-0c6d-4a08-8ce4-679b2b200b79.png)

The `forward` method returns losses or predictions of training, validation, testing, and a simple inference process.

The method should accept three modes: "tensor", "predict" and "loss":

- "tensor": Forward the whole network and return the tensor or tuple of tensor without any post-processing, same as a common `nn.Module`.
- "predict": Forward and return the predictions, which are fully processed to a list of `SegDataSample`.
- "loss": Forward and return a `dict` of losses according to the given inputs and data samples.

**Note:** [SegDataSample](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/structures/seg_data_sample.py) is a data structure interface of MMSegmentation, it is used as an interface between different components. `SegDataSample` implements the abstract data element `mmengine.structures.BaseDataElement`, please refer to [the SegDataSample documentation](https://mmsegmentation.readthedocs.io/en/1.x/advanced_guides/structures.html) and [data element documentation](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/data_element.html) in [MMEngine](https://github.com/open-mmlab/mmengine) for more information.

Note that this method doesn't handle either backpropagation or optimizer updating, which are done in the method `train_step`.

- inputs (torch.Tensor) - The input tensor with shape (N, C, ...) in general.

- data_sample (list\[[SegDataSample](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/structures/seg_data_sample.py)\]) - The seg data samples. It usually includes information such as `metainfo` and `gt_sem_seg`. Default to None.
- mode (str) - Return what kind of value. Defaults to 'tensor'.

Returns:

- `dict` or `list`:
  - If `mode == "loss"`, return a `dict` of loss tensor used for backward and logging.
  - If `mode == "predict"`, return a `list` of `SegDataSample`, the inference results will be incrementally added to the `data_sample` parameter passed to the forward method, each `SegDataSample` contains the following keys:
    - pred_sem_seg (`PixelData`): Prediction of semantic segmentation.
    - seg_logits (`PixelData`): Predicted logits of semantic segmentation before normalization.

- pred_sem_seg (`PixelData`): Prediction of semantic segmentation.
  - If `mode == "tensor"`, return a `tensor` or `tuple of tensor` or `dict` of `tensor` for custom use.

### prediction modes

We briefly describe the fields of the model's configuration in [the config documentation](../user_guides/1_config.md), here we elaborate on the `model.test_cfg` field. `model.test_cfg` is used to control forward behavior, the `forward` method in `"predict"` mode can run in two modes:

- `whole_inference`: If `cfg.model.test_cfg.mode == 'whole'`, model will inference with full images.

  An `whole_inference` mode example config:

  ```python
    type='EncoderDecoder'
    ...
    test_cfg=dict(mode='whole')
  )
  ```

An `whole_inference` mode example config:

  ```python
    ...
  )
  ```

- `slide_inference`: If `cfg.model.test_cfg.mode == 'slide'`, model will inference by sliding-window. **Note:** if you select the `slide` mode, `cfg.model.test_cfg.stride` and `cfg.model.test_cfg.crop_size` should also be specified.

  An `slide_inference` mode example config:

  ```python
    ...
    test_cfg=dict(mode='slide', crop_size=256, stride=170)
  )
  ```

### train_step

The `train_step` method calls the forward interface of the `loss` mode to get the loss `dict`. The `BaseModel` class implements the default model training process including preprocessing, model forward propagation, loss calculation, optimization, and back-propagation.

- data (dict or tuple or list) - Data sampled from the dataset. In MMSegmentation, the data dict contains `inputs` and `data_samples` two fields.
- optim_wrapper (OptimWrapper) - OptimWrapper instance used to update model parameters.

**Note:** [OptimWrapper](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/optimizer/optimizer_wrapper.py#L17) provides a common interface for updating parameters, please refer to optimizer wrapper [documentation](https://mmengine.readthedocs.io/en/latest/tutorials/optim_wrapper.html) in [MMEngine](https://github.com/open-mmlab/mmengine) for more information.

Returns:

- Dict\[str, `torch.Tensor`\]: A `dict` of tensor for logging.

![train_step dataflow](https://user-images.githubusercontent.com/15952744/228828089-a9ae1225-958d-4cf7-99af-9af8576f7ef7.png)

### val_step

The `val_step` method calls the forward interface of the `predict` mode and returns the prediction result, which is further passed to the process interface of the evaluator and the `after_val_iter` interface of the Hook.

- data (`dict` or `tuple` or `list`) - Data sampled from the dataset. In MMSegmentation, the data dict contains `inputs` and `data_samples` two fields.

Returns:

- `list` - The predictions of given data.

### test_step

The `BaseModel` implements `test_step` the same as `val_step`.

## Data Preprocessor

The [SegDataPreProcessor](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/models/data_preprocessor.py#L13) implemented by MMSegmentation inherits from the [BaseDataPreprocessor](https://github.com/open-mmlab/mmengine/blob/main/mmengine/model/base_model/data_preprocessor.py#L18) implemented by [MMEngine](https://github.com/open-mmlab/mmengine) and provides the functions of data preprocessing and copying data to the target device.

The runner carries the model to the specified device during the construction stage, while the data is carried to the specified device by the [SegDataPreProcessor](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/models/data_preprocessor.py#L13) in `train_step`, `val_step`, and `test_step`, and the processed data is further passed to the model.

The parameters of the `SegDataPreProcessor` constructor:

- mean (Sequence\[Number\], optional) - The pixel mean of R, G, B channels. Defaults to None.
- std (Sequence\[Number\], optional) - The pixel standard deviation of R, G, B channels. Defaults to None.
- size (tuple, optional) - Fixed padding size.
- size_divisor (int, optional) - The divisor of padded size.
- pad_val (float, optional) - Padding value. Default: 0.
- seg_pad_val (float, optional) - Padding value of segmentation map. Default: 255.
- bgr_to_rgb (bool) - whether to convert image from BGR to RGB. Defaults to False.
- rgb_to_bgr (bool) - whether to convert image from RGB to BGR. Defaults to False.
- batch_augments (list\[dict\], optional) - Batch-level augmentations. Default to None.

The data will be processed as follows:

- Collate and move data to the target device.
- Pad inputs to the input size with defined `pad_val`, and pad seg map with defined `seg_pad_val`.
- Stack inputs to batch_inputs.

- Convert inputs from bgr to rgb if the shape of input is (3, H, W).
- Normalize image with defined std and mean.
- Do batch augmentations like Mixup and Cutmix during training.

The parameters of the `forward` method:

- data (dict) - data sampled from dataloader.
- training (bool) - Whether to enable training time augmentation.

The returns of the `forward` method:

- Dict: Data in the same format as the model input.

# Swin-Transformer V2

> [Swin Transformer V2: Scaling Up Capacity and Resolution](https://arxiv.org/abs/2111.09883)

**Swin Transformer V2** is a work on the scale up visual model based on [Swin Transformer](https://github.com/open-mmlab/mmpretrain/tree/main/configs/swin_transformer). In the visual field, We can not increase the performance by just simply scaling up the visual model like NLP models. The possible reasons mentioned in the article are:

- Training instability when increasing the vision model
- Migrating the model trained at low resolution to a larger scale resolution task
- Too mush GPU memory

To solve it, The following method improvements are proposed in the paper:

- post normalization: layer normalization after self-attention layer and MLP block
- scaled cosine attention approach: use cosine similarity to calculate the relationship between token pairs
- log-spaced continuous position bias: redefine relative position encoding

<img src="https://user-images.githubusercontent.com/42952108/180748696-ee7ed23d-7fee-4ccf-9eb5-f117db228a42.png" width="100%"/>
</div>

<details>

<summary>Show the detailed Abstract</summary>

<br>

Large-scale NLP models have been shown to significantly improve the performance on language tasks with no signs of saturation. They also demonstrate amazing few-shot capabilities like that of human beings. This paper aims to explore large-scale models in computer vision. We tackle three major issues in training and application of large vision models, including training instability, resolution gaps between pre-training and fine-tuning, and hunger on labelled data. Three main techniques are proposed: 1) a residual-post-norm method combined with cosine attention to improve training stability; 2) A log-spaced continuous position bias method to effectively transfer models pre-trained using low-resolution images to downstream tasks with high-resolution inputs; 3) A self-supervised pre-training method, SimMIM, to reduce the needs of vast labeled images. Through these techniques, this paper successfully trained a 3 billion-parameter Swin Transformer V2 model, which is the largest dense vision model to date, and makes it capable of training with images of up to 1,536×1,536 resolution. It set new performance records on 4 representative vision tasks, including ImageNet-V2 image classification, COCO object detection, ADE20K semantic segmentation, and Kinetics-400 video action classification. Also note our training is much more efficient than that in Google's billion-level visual models, which consumes 40 times less labelled data and 40 times less training time.

</br>

```python

predict = inference_model('swinv2-tiny-w8_3rdparty_in1k-256px', 'demo/bird.JPEG')
```

```python

model = get_model('swinv2-tiny-w8_3rdparty_in1k-256px', pretrained=True)
```

Test:

```shell
python tools/test.py configs/swin_transformer_v2/swinv2-tiny-w8_16xb64_in1k-256px.py https://download.openmmlab.com/mmclassification/v0/swin-v2/swinv2-tiny-w8_3rdparty_in1k-256px_20220803-e318968f.pth
```

```

| Model                                     | Params (M) | Flops (G) |                      Config                      |                                           Download                                           |
| :---------------------------------------- | :--------: | :-------: | :----------------------------------------------: | :------------------------------------------------------------------------------------------: |
| `swinv2-base-w12_3rdparty_in21k-192px`\*  |   87.92    |   8.51    | [config](swinv2-base-w12_8xb128_in21k-192px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/swin-v2/pretrain/swinv2-base-w12_3rdparty_in21k-192px_20220803-f7dc9763.pth) |

| `swinv2-large-w12_3rdparty_in21k-192px`\* |   196.74   |   19.04   | [config](swinv2-large-w12_8xb128_in21k-192px.py) | [model](https://download.openmmlab.com/mmclassification/v0/swin-v2/pretrain/swinv2-large-w12_3rdparty_in21k-192px_20220803-d9073fee.pth) |

*Models with * are converted from the [official repo](https://github.com/microsoft/Swin-Transformer). The config files of these models are only for inference. We haven't reproduce the training results.*

| `swinv2-tiny-w8_3rdparty_in1k-256px`\*            | From scratch |   28.35    |   4.35    |   81.76   |   95.87   |   [config](swinv2-tiny-w8_16xb64_in1k-256px.py)    | [model](https://download.openmmlab.com/mmclassification/v0/swin-v2/swinv2-tiny-w8_3rdparty_in1k-256px_20220803-e318968f.pth) |
| `swinv2-tiny-w16_3rdparty_in1k-256px`\*           | From scratch |   28.35    |   4.40    |   82.81   |   96.23   |   [config](swinv2-tiny-w16_16xb64_in1k-256px.py)   | [model](https://download.openmmlab.com/mmclassification/v0/swin-v2/swinv2-tiny-w16_3rdparty_in1k-256px_20220803-9651cdd7.pth) |
| `swinv2-small-w8_3rdparty_in1k-256px`\*           | From scratch |   49.73    |   8.45    |   83.74   |   96.60   |   [config](swinv2-small-w8_16xb64_in1k-256px.py)   | [model](https://download.openmmlab.com/mmclassification/v0/swin-v2/swinv2-small-w8_3rdparty_in1k-256px_20220803-b01a4332.pth) |

| `swinv2-small-w16_3rdparty_in1k-256px`\*          | From scratch |   49.73    |   8.57    |   84.13   |   96.83   |  [config](swinv2-small-w16_16xb64_in1k-256px.py)   | [model](https://download.openmmlab.com/mmclassification/v0/swin-v2/swinv2-small-w16_3rdparty_in1k-256px_20220803-b707d206.pth) |
| `swinv2-base-w8_3rdparty_in1k-256px`\*            | From scratch |   87.92    |   14.99   |   84.20   |   96.86   |   [config](swinv2-base-w8_16xb64_in1k-256px.py)    | [model](https://download.openmmlab.com/mmclassification/v0/swin-v2/swinv2-base-w8_3rdparty_in1k-256px_20220803-8ff28f2b.pth) |
| `swinv2-base-w16_3rdparty_in1k-256px`\*           | From scratch |   87.92    |   15.14   |   84.60   |   97.05   |   [config](swinv2-base-w16_16xb64_in1k-256px.py)   | [model](https://download.openmmlab.com/mmclassification/v0/swin-v2/swinv2-base-w16_3rdparty_in1k-256px_20220803-5a1886b7.pth) |

| `swinv2-base-w16_in21k-pre_3rdparty_in1k-256px`\* | ImageNet-21k |   87.92    |   15.14   |   86.17   |   97.88   | [config](swinv2-base-w16_in21k-pre_16xb64_in1k-256px.py) | [model](https://download.openmmlab.com/mmclassification/v0/swin-v2/swinv2-base-w16_in21k-pre_3rdparty_in1k-256px_20220803-8d7aa8ad.pth) |
| `swinv2-base-w24_in21k-pre_3rdparty_in1k-384px`\* | ImageNet-21k |   87.92    |   34.07   |   87.14   |   98.23   | [config](swinv2-base-w24_in21k-pre_16xb64_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/swin-v2/swinv2-base-w24_in21k-pre_3rdparty_in1k-384px_20220803-44eb70f8.pth) |
| `swinv2-large-w16_in21k-pre_3rdparty_in1k-256px`\* | ImageNet-21k |   196.75   |   33.86   |   86.93   |   98.06   | [config](swinv2-large-w16_in21k-pre_16xb64_in1k-256px.py) | [model](https://download.openmmlab.com/mmclassification/v0/swin-v2/swinv2-large-w16_in21k-pre_3rdparty_in1k-256px_20220803-c40cbed7.pth) |

| `swinv2-large-w24_in21k-pre_3rdparty_in1k-384px`\* | ImageNet-21k |   196.75   |   76.20   |   87.59   |   98.27   | [config](swinv2-large-w24_in21k-pre_16xb64_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/swin-v2/swinv2-large-w24_in21k-pre_3rdparty_in1k-384px_20220803-3b36c165.pth) |

```bibtex
@article{https://doi.org/10.48550/arxiv.2111.09883,
  doi = {10.48550/ARXIV.2111.09883},
  url = {https://arxiv.org/abs/2111.09883},
  author = {Liu, Ze and Hu, Han and Lin, Yutong and Yao, Zhuliang and Xie, Zhenda and Wei, Yixuan and Ning, Jia and Cao, Yue and Zhang, Zheng and Dong, Li and Wei, Furu and Guo, Baining},

author = {Liu, Ze and Hu, Han and Lin, Yutong and Yao, Zhuliang and Xie, Zhenda and Wei, Yixuan and Ning, Jia and Cao, Yue and Zhang, Zheng and Dong, Li and Wei, Furu and Guo, Baining},
  title = {Swin Transformer V2: Scaling Up Capacity and Resolution},
  year = {2021},
}
```

</div>

|      Model       | Input Size |  AP   |  AR   |                   Details and Download                    |

| :--------------: | :--------: | :---: | :---: | :-------------------------------------------------------: |
|  ResNet-152+RLE  |  256x192   | 0.731 | 0.805 |      [resnet_rle_coco.md](./coco/resnet_rle_coco.md)      |
|  ResNet-101+RLE  |  256x192   | 0.722 | 0.768 |      [resnet_rle_coco.md](./coco/resnet_rle_coco.md)      |
|  ResNet-50+RLE   |  256x192   | 0.706 | 0.768 |      [resnet_rle_coco.md](./coco/resnet_rle_coco.md)      |
| MobileNet-v2+RLE |  256x192   | 0.593 | 0.644 | [mobilenetv2_rle_coco.md](./coco/mobilenetv2_rle_coco.md) |
|    ResNet-152    |  256x192   | 0.584 | 0.688 |          [resnet_coco.md](./coco/resnet_coco.md)          |
|    ResNet-101    |  256x192   | 0.562 | 0.670 |          [resnet_coco.md](./coco/resnet_coco.md)          |

|    ResNet-50     |  256x192   | 0.528 | 0.639 |          [resnet_coco.md](./coco/resnet_coco.md)          |

|     Model     | Input Size | PCKh@0.5 | PCKh@0.1 |              Details and Download               |
| :-----------: | :--------: | :------: | :------: | :---------------------------------------------: |
| ResNet-50+RLE |  256x256   |  0.861   |  0.277   | [resnet_rle_mpii.md](./mpii/resnet_rle_mpii.md) |
|  ResNet-152   |  256x256   |  0.850   |  0.208   |     [resnet_mpii.md](./mpii/resnet_mpii.md)     |
|  ResNet-101   |  256x256   |  0.841   |  0.200   |     [resnet_mpii.md](./mpii/resnet_mpii.md)     |
|   ResNet-50   |  256x256   |  0.826   |  0.180   |     [resnet_mpii.md](./mpii/resnet_mpii.md)     |

# 从 MMAction2 0.x 迁移

MMAction2 1.x 引入了一些重构和修改，包括一些向后不兼容的更改。我们提供这个教程，帮助您从 MMAction2 0.x 迁移您的项目。

## 新的依赖项

MMAction2 1.x 依赖于以下库。建议您准备一个新的运行环境，并根据[安装教程](./get_started/installation.md)进行安装。

1. [MMEngine](https://github.com/open-mmlab/mmengine)：MMEngine 是引入于 OpenMMLab 2.0 架构中的用于训练深度学习模型的基础库。
2. [MMCV](https://github.com/open-mmlab/mmcv)：MMCV 是用于计算机视觉的基础库。MMAction2 1.x 需要 `mmcv>=2.0.0`，它比 `mmcv-full==2.0.0` 更紧凑和高效。

## 配置文件

在 MMAction2 1.x 中，我们重构了配置文件的结构。旧风格的配置文件将不兼容。

在本节中，我们将介绍配置文件的所有更改。我们假设您已经熟悉[配置文件](./user_guides/config.md)。

### 模型设置

`model.backbone` 和 `model.neck` 没有更改。对于 `model.cls_head`，我们将 `average_clips` 移到其中，原本设置在 `model.test_cfg` 中。

### 数据设置

#### **`data`** 中的更改

- 原始的 `data` 字段被拆分为 `train_dataloader`、`val_dataloader` 和 `test_dataloader`。这样可以对它们进行细粒度的配置。例如，您可以在训练和测试过程中指定不同的采样器和批大小。
- `videos_per_gpu` 改名为 `batch_size`。
- `workers_per_gpu` 改名为 `num_workers`。

<tr>
<td>旧版本</td>
<td>

```python

<tr>
<td>

```python
)
```

</td>
<tr>
<td>新版本</td>
<td>

```python
    sampler=dict(type='DefaultSampler', shuffle=True)  # 必要
)

    sampler=dict(type='DefaultSampler', shuffle=False)  # 必要
)

```

</td>
</tr>
</table>

#### **`pipeline`** 中的更改

- 原来的格式化变换 **`ToTensor`**、**`Collect`** 被合并为 `PackActionInputs`。
- 我们不建议在数据集流水线中进行 **`Normalize`**。请从流水线中移除它，并在 `model.data_preprocessor` 字段中设置。

<tr>
<td>

```python

    dict(type='DecordInit'),

<tr>
<td>

```python

    dict(
    dict(type='Collect', keys=['imgs', 'label'], meta_keys=[]),
    dict(type='ToTensor', keys=['imgs', 'label'])
]
```

</td>
<tr>
<td>

```python
model.data_preprocessor = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=False)

]
```

</td>
<tr>
<td>

```python

    dict(type='SampleFrames', clip_len=1, frame_interval=1, num_clips=5),
    dict(
]
```

</td>
</tr>
</table>

#### **`evaluation`** 中的更改

- **`evaluation`** 字段被拆分为 `val_evaluator` 和 `test_evaluator`。不再支持 `interval` 和 `save_best` 参数。
- `interval` 移到 `train_cfg.val_interval`，`save_best` 移到 `default_hooks.checkpoint.save_best`。

- 'mean_average_precision'、'mean_class_accuracy'、'mmit_mean_average_precision'、'top_k_accuracy' 被合并为 `AccMetric`，您可以使用 `metric_list` 指定要计算的指标。
- `AVAMetric` 用于评估 AVA 数据集。
- `ANetMetric` 用于评估 ActivityNet 数据集。

<tr>
<td>

```python
    interval=5,
    metrics=['top_k_accuracy', 'mean_class_accuracy'])
```

</td>
<tr>
<td>

```python
    type='AccMetric',
    metric_list=('top_k_accuracy', 'mean_class_accuracy'))
```

</td>
</tr>
</table>

### 学习率策略设置

#### **`optimizer`** 和 **`optimizer_config`** 中的更改

- 现在我们使用 `optim_wrapper` 字段来配置优化过程。`optimizer` 成为 `optim_wrapper` 的子字段。
- `paramwise_cfg` 也是 `optim_wrapper` 的子字段，与 `optimizer` 平行。

- 现在已删除 `optimizer_config`，其中的所有配置都移动到 `optim_wrapper`。
- `grad_clip` 改名为 `clip_grad`。

<tr>
<td>

```python
    type='AdamW',
    lr=0.0015,
    weight_decay=0.3,
        norm_decay_mult=0.0,
        bias_decay_mult=0.0,
    ))

optimizer_config = dict(grad_clip=dict(max_norm=1.0))
```

</td>
<tr>
<td>

```python
    optimizer=dict(type='AdamW', lr=0.0015, weight_decay=0.3),
    ),
    clip_gard=dict(max_norm=1.0),
)
```

</td>
</tr>
</table>

#### **`lr_config`** 中的更改

- 删除了 `lr_config` 字段，我们使用新的 `param_scheduler` 来替代它。
- 删除了与 warmup 相关的参数，因为我们使用策略组合来实现这个功能。

新的组合机制非常灵活，您可以使用它来设计多种学习率/动量曲线。

)
```

</td>
</tr>
</table>

<tr>
<td>

```python
    policy='CosineAnnealing',
    min_lr=0,
    warmup_iters=5,
    warmup_ratio=0.01,
    warmup_by_epoch=True)
```

</td>
<tr>
<td>

```python
    # 学习率预热
    dict(
        start_factor=0.01,
        end=5,
        # 在每个迭代后更新学习率。
    # 主要的学习率策略
    dict(type='CosineAnnealingLR', by_epoch=True, begin=5),
]
```

</td>
</tr>
</table>

#### **`runner`** 中的更改

原始 `runner` 字段中的大多数配置已移至 `train_cfg`、`val_cfg` 和 `test_cfg`，用于配置训练、验证和测试的循环。

<tr>
<td>

```python
runner = dict(type='EpochBasedRunner', max_epochs=100)
```

</td>
<tr>
<td>

```python

<tr>
<td>

```python
```

</td>
<tr>
<td>

```python
# `val_interval` 是原 `evaluation.interval`。
train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=100, val_begin=1, val_interval=1)
val_cfg = dict(type='ValLoop')   # 使用默认验证循环。
test_cfg = dict(type='TestLoop')  # 使用默认测试循环。
```

</td>
</tr>
</table>

事实上，在 OpenMMLab 2.0 中，我们引入了 `Loop` 来控制训练、验证和测试的行为。`Runner` 的功能也发生了变化。您可以在[MMEngine 教程](https://mmengine.readthedocs.io/en/latest/tutorials/runner.html)中找到更多详细信息。

### 运行时设置

#### **`checkpoint_config`** 和 **`log_config`** 中的更改

`checkpoint_config` 移到 `default_hooks.checkpoint`，`log_config` 移到 `default_hooks.logger`。我们将许多钩子的设置从脚本代码中移动到运行时配置的 `default_hooks` 字段中。

```python
    # 更新运行时信息，如当前迭代和学习率。
    runtime_info=dict(type='RuntimeInfoHook'),

    # 记录每个迭代的时间。

    # 每 100 次迭代打印日志。

```python

    # 启用参数策略器。

    # 每个 epoch 保存一次权重，并自动保存最佳权重。
    checkpoint=dict(type='CheckpointHook', interval=1, save_best='auto'),

    # 在每个 epoch 结束时同步模型缓冲区。
    sync_buffers=dict(type='SyncBuffersHook')
)
```

此外，我们将原来的 logger 拆分为 logger 和 visualizer。logger 用于记录信息，visualizer 用于在不同的后端（如终端、TensorBoard 和 Wandb）中显示 logger。

<tr>
<td>

```python
    hooks=[
    ])
```

</td>
<tr>
<td>

```python
    ...
)

dict(type='TensorboardLoggerHook'),
    ])
```

</td>
<tr>
<td>

```python
    ...
)

    type='ActionVisualizer',
    vis_backends=[dict(type='LocalVisBackend'), dict(type='TensorboardVisBackend')],
)
```

</td>
</tr>
</table>

#### **`load_from`** 和 **`resume_from`** 中的更改

- 删除了 `resume_from`。现在我们使用 `resume` 和 `load_from` 来替代它。
  - 如果 `resume=True` 并且 `load_from` 不为 None，则从 `load_from` 中的权重恢复训练。
  - 如果 `resume=True` 并且 `load_from` 为 None，则尝试从工作目录中的最新权重恢复。
  - 如果 `resume=False` 并且 `load_from` 不为 None，则只加载权重文件，不恢复训练。
  - 如果 `resume=False` 并且 `load_from` 为 None，则既不加载也不恢复。

#### **`dist_params`** 中的更改

`dist_params` 字段现在是 `env_cfg` 的子字段。`env_cfg` 中还有一些新的配置。

```python
    # 是否启用 cudnn benchmark

    # 设置多进程参数

    # 设置分布式参数
)
```

# 是否启用 cudnn benchmark

    # 设置多进程参数

    # 设置分布式参数
)
```

#### **`workflow`** 中的更改

删除了与 `workflow` 相关的功能。

#### 新字段 **`visualizer`**

visualizer 是 OpenMMLab 2.0 架构中的新设计。我们在 runner 中使用一个 visualizer 实例来处理结果和日志的可视化，并保存到不同的后端，如终端、TensorBoard 和 Wandb。

```python
        # 取消下面一行的注释，将日志和可视化结果保存到 TensorBoard。
        # dict(type='TensorboardVisBackend')
    ]
)
```

#### 新字段 **`default_scope`**

所有注册表在不同包中的定义已移动到 `mmaction.registry` 包中。

## Packages

### `mmaction.apis`

文档可以在[这里](mmaction.apis)找到。

|          函数          |                     更改                     |
| :--------------------: | :------------------------------------------: |
|   `init_recognizer`    |                   无需更改                   |

| `inference_recognizer` |                   无需更改                   |
|     `train_model`      |      删除，使用 `runner.train` 进行训练      |
|    `multi_gpu_test`    |      删除，使用 `runner.test` 进行测试       |
|   `single_gpu_test`    |      删除，使用 `runner.test` 进行测试       |
|   `set_random_seed`    | 删除，使用 `mmengine.runner.set_random_seed` |
|   `init_random_seed`   | 删除，使用 `mmengine.dist.sync_random_seed`  |

### `mmaction.core`

`mmaction.core` 包已被重命名为 [`mmaction.engine`](mmaction.engine)。

|     子包     |                           更改                            |
| :----------: | :-------------------------------------------------------: |
| `evaluation` |         删除，使用 `mmaction.evaluation` 中的指标         |
|   `hooks`    |              移动到 `mmaction.engine.hooks`               |
| `optimizer`  |            移动到 `mmaction.engine.optimizers`            |

|   `utils`    | 删除，分布式环境相关的函数可以在 `mmengine.dist` 包中找到 |

### `mmaction.datasets`

文档可以在[这里](mmaction.datasets)找到。

#### [`BaseActionDataset`](mmaction.datasets.BaseActionDataset) 中的更改：

|          方法          |                    更改                     |
| :--------------------: | :-----------------------------------------: |
| `prepare_train_frames` |           由 `get_data_info` 替换           |
| `preprare_test_frames` |           由 `get_data_info` 替换           |
|       `evaluate`       |  删除，使用 `mmengine.evaluator.Evaluator`  |
|     `dump_results`     | 删除，使用 `mmengine.evaluator.DumpResults` |
|   `load_annotations`   |           替换为 `load_data_list`           |

现在，您可以编写一个继承自 `BaseActionDataset` 的新 Dataset 类，并仅重写 `load_data_list`。要加载更多的数据信息，您可以像 `RawframeDataset` 和 `AVADataset` 那样重写 `get_data_info`。

`mmaction.datasets.pipelines` 被重命名为 `mmaction.datasets.transforms`，`mmaction.datasets.pipelines.augmentations` 被重命名为 `mmaction.datasets.pipelines.processing`。

### `mmaction.models`

文档可以在[这里](mmaction.models)找到。所有 **backbones**、**necks** 和 **losses** 的接口没有更改。

[`BaseRecognizer`](mmaction.models.BaseRecognizer) 中的更改：

|      方法       |                                                              更改                                                              |
| :-------------: | :----------------------------------------------------------------------------------------------------------------------------: |
| `extract_feat`  | 增强的方法，现在支持三个阶段（`backbone`、`neck`、`head`）的输出特征，并且可以处理不同的模式，如 `train_mode` 和 `test_mode`。 |
|    `forward`    |         现在只接受三个参数：`inputs`、`data_samples` 和 `mode`。详细信息请参阅[文档](mmaction.models.BaseRecognizer)。         |

| `forward_train` |                                                       已替换为 `loss`。                                                        |
| `forward_test`  |                                                      已替换为 `predict`。                                                      |
|  `train_step`   |                `optimizer` 参数被替换为 `optim_wrapper`，它接受 [`OptimWrapper`](mmengine.optim.OptimWrapper)。                |
|   `val_step`    |                                    原 `val_step` 与 `train_step` 相同，现在调用 `predict`。                                    |
|   `test_step`   |                                                  新方法，与 `val_step` 相同。                                                  |

[BaseHead](mmaction.models.BaseHead) 中的更改：

|   方法    |                                                                              更改                                                                              |
| :-------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| `forward` |                                                                            无需更改                                                                            |
|  `loss`   | 接受 `feats` 和 `data_samples`，而不是 `cls_score` 和 `labels` 来计算损失。`data_samples` 是 [ActionDataSample](mmaction.structures.ActionDataSample) 的列表。 |

| `predict` |                                                        接受 `feats` 和 `data_samples` 来预测分类分数。                                                         |

### `mmaction.utils`

|          函数           |                            更改                            |
| :---------------------: | :--------------------------------------------------------: |
|      `collect_env`      |                          无需更改                          |
|    `get_root_logger`    |    删除，使用 `mmengine.MMLogger.get_current_instance`     |
| `setup_multi_processes` | 删除，使用 `mmengine.utils.dl_utils.setup_multi_processes` |

### 其他更改

- 我们将所有注册器的定义从各个包移动到了 `mmaction.registry` 。

The table below lists the models that are guaranteed to be exportable to other backends.

### Note

- Tag:

# MViT V2

> [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](http://openaccess.thecvf.com//content/CVPR2022/papers/Li_MViTv2_Improved_Multiscale_Vision_Transformers_for_Classification_and_Detection_CVPR_2022_paper.pdf)

In this paper, we study Multiscale Vision Transformers (MViTv2) as a unified architecture for image and video
classification, as well as object detection. We present an improved version of MViT that incorporates
decomposed relative positional embeddings and residual pooling connections. We instantiate this architecture
in five sizes and evaluate it for ImageNet classification, COCO detection and Kinetics video recognition where
it outperforms prior work. We further compare MViTv2s' pooling attention to window attention mechanisms where
it outperforms the latter in accuracy/compute. Without bells-and-whistles, MViTv2 has state-of-the-art

performance in 3 domains: 88.8% accuracy on ImageNet classification, 58.7 boxAP on COCO object detection as
well as 86.1% on Kinetics-400 video classification.

<img src="https://user-images.githubusercontent.com/26739999/180376227-755243fa-158e-4068-940a-416036519665.png" width="50%"/>
</div>

```python

predict = inference_model('mvitv2-tiny_3rdparty_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('mvitv2-tiny_3rdparty_in1k', pretrained=True)
```

```

Test:

```shell
python tools/test.py configs/mvit/mvitv2-tiny_8xb256_in1k.py https://download.openmmlab.com/mmclassification/v0/mvit/mvitv2-tiny_3rdparty_in1k_20220722-db7beeef.pth
```

| `mvitv2-tiny_3rdparty_in1k`\*  | From scratch |   24.17    |   4.70    |   82.33   |   96.15   | [config](mvitv2-tiny_8xb256_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/mvit/mvitv2-tiny_3rdparty_in1k_20220722-db7beeef.pth) |
| `mvitv2-small_3rdparty_in1k`\* | From scratch |   34.87    |   7.00    |   83.63   |   96.51   | [config](mvitv2-small_8xb256_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mvit/mvitv2-small_3rdparty_in1k_20220722-986bd741.pth) |
| `mvitv2-base_3rdparty_in1k`\*  | From scratch |   51.47    |   10.16   |   84.34   |   96.86   | [config](mvitv2-base_8xb256_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/mvit/mvitv2-base_3rdparty_in1k_20220722-9c4f0a17.pth) |

| `mvitv2-large_3rdparty_in1k`\* | From scratch |   217.99   |   43.87   |   85.25   |   97.14   | [config](mvitv2-large_8xb256_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mvit/mvitv2-large_3rdparty_in1k_20220722-2b57b983.pth) |

*Models with * are converted from the [official repo](https://github.com/facebookresearch/mvit). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@inproceedings{li2021improved,
  title={MViTv2: Improved multiscale vision transformers for classification and detection},
  author={Li, Yanghao and Wu, Chao-Yuan and Fan, Haoqi and Mangalam, Karttikeya and Xiong, Bo and Malik, Jitendra and Feichtenhofer, Christoph},
}
```

# Chest X-ray Images with Pneumothorax Masks

This project support **`Chest X-ray Images with Pneumothorax Masks `**, and the dataset used in this project can be downloaded from [here](https://www.kaggle.com/datasets/vbookshelf/pneumothorax-chest-xray-images-and-masks).

A pneumothorax (noo-moe-THOR-aks) is a collapsed lung. A pneumothorax occurs when air leaks into the space between your lung and chest wall. This air pushes on the outside of your lung and makes it collapse. Pneumothorax can be a complete lung collapse or a collapse of only a portion of the lung.

A pneumothorax can be caused by a blunt or penetrating chest injury, certain medical procedures, or damage from underlying lung disease. Or it may occur for no obvious reason. Symptoms usually include sudden chest pain and shortness of breath. On some occasions, a collapsed lung can be a life-threatening event.

Treatment for a pneumothorax usually involves inserting a needle or chest tube between the ribs to remove the excess air. However, a small pneumothorax may heal on its own.

| Dataset Name                                                                                                                      | Anatomical Region | Task type    | Modality | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release date | License                                                         |
| --------------------------------------------------------------------------------------------------------------------------------- | ----------------- | ------------ | -------- | ------------ | --------------------- | ---------------------- | ------------ | --------------------------------------------------------------- |

| [Chest-x-ray-images-with-pneumothorax-masks](https://www.kaggle.com/datasets/vbookshelf/pneumothorax-chest-xray-images-and-masks) | throax            | segmentation | x_ray    | 2            | 10675/-/1372          | yes/-/yes              | 2020         | [CC-BY-NC 4.0](https://creativecommons.org/licenses/by-sa/4.0/) |

|  background  |   10675    |    99.7    |    -     |    -     |   1372    |   99.71   |
| pneumothroax |    2379    |    0.3     |    -     |    -     |    290    |   0.29    |

![chest_x_ray_images_with_pneumothorax_masks](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/x_ray/chest_x_ray_images_with_pneumothorax_masks/chest_x_ray_images_with_pneumothorax_masks_dataset.png?raw=true)

All the commands below rely on the correct configuration of PYTHONPATH, which should point to the project's directory so that Python can locate the module files. In chest_x_ray_images_with_pneumothorax_masks/ root directory, run the following line to add the current directory to PYTHONPATH:

```shell
```

- download dataset from [here](https://www.kaggle.com/datasets/vbookshelf/pneumothorax-chest-xray-images-and-masks) and decompression data to path 'data/'.

```none
  ├── mmseg
  │   │   │   │   ├── chest_x_ray_images_with_pneumothorax_masks

│   │   │   │   │   │   │   ├── train
```

```shell
```

```shell
```

```shell
```

# Add New Modules

We can customize all the components introduced at [the model documentation](./models.md), such as **backbone**, **head**, **loss function** and **data preprocessor**.

### Add new backbones

Here we show how to develop a new backbone with an example of MobileNet.

1. Create a new file `mmseg/models/backbones/mobilenet.py`.

   ```python

           pass

           pass

           pass
   ```

2. Import the module in `mmseg/models/backbones/__init__.py`.

   ```python
   ```

   ```python
       ...
           arg1=xxx,

```

   ```python
       ...
           arg1=xxx,
       ...
   ```

In MMSegmentation, we provide a [BaseDecodeHead](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/models/decode_heads/decode_head.py#L17) for developing all segmentation heads.
All newly implemented decode heads should be derived from it.
Here we show how to develop a new head with the example of [PSPNet](https://arxiv.org/abs/1612.01105) as the following.

First, add a new decode head in `mmseg/models/decode_heads/psp_head.py`.
PSPNet implements a decode head for segmentation decode.
To implement a decode head, we need to implement three functions of the new module as the following.

```python

```python

        pass

        pass
```

Next, the users need to add the module in the `mmseg/models/decode_heads/__init__.py`, thus the corresponding registry could find and load them.

To config file of PSPNet is as the following

```python
        depth=50,

strides=(1, 2, 1, 1),

```

Assume you want to add a new loss as `MyLoss` for segmentation decode.
To add a new loss function, the users need to implement it in `mmseg/models/losses/my_loss.py`.

```python

                pred,
                target,
```

loss = self.loss_weight * my_loss(
```

Then the users need to add it in the `mmseg/models/losses/__init__.py`.

```python

```

Then you need to modify the `loss_decode` field in the head.
`loss_weight` could be used to balance multiple losses.

```python
```

### Add new data preprocessor

In MMSegmentation 1.x versions, we use [SegDataPreProcessor](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/models/data_preprocessor.py#L13) to copy data to the target device and preprocess the data into the model input format as default. Here we show how to develop a new data preprocessor.

1. Create a new file `mmseg/models/my_datapreprocessor.py`.

   ```python

   ```python

           pass
   ```

2. Import your data preprocessor in `mmseg/models/__init__.py`

   ```python
   ```

   ```python
       ...
   )
   ```

## Develop new segmentors

```

   ```python
       ...
   )
   ```

The segmentor is an algorithmic architecture in which users can customize their algorithms by adding customized components and defining the logic of algorithm execution. Please refer to [the model document](./models.md) for more details.

Since the [BaseSegmentor](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/models/segmentors/base.py#L15) in MMSegmentation unifies three modes for a forward process, to develop a new segmentor, users need to overwrite `loss`, `predict` and `_forward` methods corresponding to the `loss`, `predict` and `tensor` modes.

Here we show how to develop a new segmentor.

1. Create a new file `mmseg/models/segmentors/my_segmentor.py`.

   ```python

   ```python

            pass

            pass

processing."""
            pass

            """
            pass
   ```

2. Import your segmentor in `mmseg/models/segmentors/__init__.py`.

   ```python
   ```

   ```python
       ...
   )
   ```

# CARAFE

> [CARAFE: Content-Aware ReAssembly of FEatures](https://arxiv.org/abs/1905.02188)

Feature upsampling is a key operation in a number of modern convolutional network architectures, e.g. feature pyramids. Its design is critical for dense prediction tasks such as object detection and semantic/instance segmentation. In this work, we propose Content-Aware ReAssembly of FEatures (CARAFE), a universal, lightweight and highly effective operator to fulfill this goal. CARAFE has several appealing properties: (1) Large field of view. Unlike previous works (e.g. bilinear interpolation) that only exploit sub-pixel neighborhood, CARAFE can aggregate contextual information within a large receptive field. (2) Content-aware handling. Instead of using a fixed kernel for all samples (e.g. deconvolution), CARAFE enables instance-specific content-aware handling, which generates adaptive kernels on-the-fly. (3) Lightweight and fast to compute. CARAFE introduces little computational overhead and can be readily integrated into modern network architectures. We conduct comprehensive evaluations on standard benchmarks in object detection, instance/semantic segmentation and inpainting. CARAFE shows consistent and substantial gains across all the tasks (1.2%, 1.3%, 1.8%, 1.1db respectively) with negligible computational overhead. It has great potential to serve as a strong building block for future research. It has great potential to serve as a strong building block for future research.

<img src="https://user-images.githubusercontent.com/40661020/143872016-48225685-0e59-49cf-bd65-a50ee04ca8a2.png"/>
</div>

The results on COCO 2017 val is shown in the below table.

|         Method         | Backbone |  Style  | Lr schd | Test Proposal Num | Inf time (fps) | Box AP | Mask AP |                      Config                       |                                                                                                                                                                         Download                                                                                                                                                                          |

| Faster R-CNN w/ CARAFE | R-50-FPN | pytorch |   1x    |       1000        |      16.5      |  38.6  |  38.6   | [config](./faster-rcnn_r50_fpn-carafe_1x_coco.py) |     [model](https://download.openmmlab.com/mmdetection/v2.0/carafe/faster_rcnn_r50_fpn_carafe_1x_coco/faster_rcnn_r50_fpn_carafe_1x_coco_bbox_mAP-0.386_20200504_175733-385a75b7.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/carafe/faster_rcnn_r50_fpn_carafe_1x_coco/faster_rcnn_r50_fpn_carafe_1x_coco_20200504_175733.log.json)     |

|  Mask R-CNN w/ CARAFE  | R-50-FPN | pytorch |   1x    |       1000        |      14.0      |  39.3  |  35.8   |  [config](./mask-rcnn_r50_fpn-carafe_1x_coco.py)  | [model](https://download.openmmlab.com/mmdetection/v2.0/carafe/mask_rcnn_r50_fpn_carafe_1x_coco/mask_rcnn_r50_fpn_carafe_1x_coco_bbox_mAP-0.393__segm_mAP-0.358_20200503_135957-8687f195.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/carafe/mask_rcnn_r50_fpn_carafe_1x_coco/mask_rcnn_r50_fpn_carafe_1x_coco_20200503_135957.log.json) |

|           -            |    -     |    -    |    -    |       2000        |                |        |         |                                                   |                                                                                                                                                                                                                                                                                                                                                           |

## Implementation

The CUDA implementation of CARAFE can be find at https://github.com/myownskyW7/CARAFE.

We provide config files to reproduce the object detection & instance segmentation results in the ICCV 2019 Oral paper for [CARAFE: Content-Aware ReAssembly of FEatures](https://arxiv.org/abs/1905.02188).

```latex
@inproceedings{Wang_2019_ICCV,
    title = {CARAFE: Content-Aware ReAssembly of FEatures},

```latex
    author = {Wang, Jiaqi and Chen, Kai and Xu, Rui and Liu, Ziwei and Loy, Chen Change and Lin, Dahua},
}
```

# Projects FAQ

Implementing new models and features into OpenMMLab's algorithm libraries could be troublesome due to the rigorous requirements on code quality, which could hinder the fast iteration of SOTA models and might discourage our members from sharing their latest outcomes here. And that's why we have this `projects/` folder now, where some experimental features, frameworks and models are placed, only needed to satisfy the minimum requirement on the code quality, and can be used as standalone libraries. Users are welcome to use them if they [use MMOCR from source](https://mmocr.readthedocs.io/en/dev-1.x/get_started/install.html#best-practices).

This checklist is crucial not only for this project's developers but the entire community, since there might be some other contributors joining this project and deciding their starting point from this list. It also helps maintainers accurately estimate time and effort on further code polishing, if needed.

Reaching the first milestone means that this project suffices the minimum requirement of being merged into `projects/`. That is, the very first PR of a project must have all the terms in the first milestone checked. We do not have any extra requirements on the project's following PRs, so they can be a minor bug fix or update, and do not have to achieve one milestone at once. But keep in mind that this project is only eligible to become a part of the core package upon attaining the last milestone.

<details>

```bibtex
}
```

    <sup>
      </a>
    </sup>
    <sup>
      </a>
    </sup>
  </div>

[📘Documentation](https://mmocr.readthedocs.io/en/dev-1.x/) |
[🛠️Installation](https://mmocr.readthedocs.io/en/dev-1.x/get_started/install.html) |
[👀Model Zoo](https://mmocr.readthedocs.io/en/dev-1.x/modelzoo.html) |

[🆕Update News](https://mmocr.readthedocs.io/en/dev-1.x/notes/changelog.html) |
[🤔Reporting Issues](https://github.com/open-mmlab/mmocr/issues/new/choose)

</div>

</div>

</div>

## Latest Updates

<img src="https://user-images.githubusercontent.com/25839884/219026120-ba71e48b-6e94-4bd4-b4e9-b7d175b5e362.png" width="3%" alt="" /></a>
</div>

**The default branch is now `main` and the code on the branch has been upgraded to v1.0.0. The old `main` branch (v0.6.3) code now exists on the `0.x` branch.** If you have been using the `main` branch and encounter upgrade issues, please read the [Migration Guide](https://mmocr.readthedocs.io/en/dev-1.x/migration/overview.html) and notes on [Branches](https://mmocr.readthedocs.io/en/dev-1.x/migration/branches.html) .

v1.0.0 was released in 2023-04-06. Major updates from 1.0.0rc6 include:

1. Support for SCUT-CTW1500, SynthText, and MJSynth datasets in Dataset Preparer
2. Updated FAQ and documentation
3. Deprecation of file_client_args in favor of backend_args
4. Added a new MMOCR tutorial notebook

To know more about the updates in MMOCR 1.0, please refer to [What's New in MMOCR 1.x](https://mmocr.readthedocs.io/en/dev-1.x/migration/news.html), or
Read [Changelog](https://mmocr.readthedocs.io/en/dev-1.x/notes/changelog.html) for more details!

MMOCR is an open-source toolbox based on PyTorch and mmdetection for text detection, text recognition, and the corresponding downstream tasks including key information extraction. It is part of the [OpenMMLab](https://openmmlab.com/) project.

The main branch works with **PyTorch 1.6+**.

</div>

### Major Features

- **Comprehensive Pipeline**

</div>

  The toolbox supports not only text detection and text recognition, but also their downstream tasks such as key information extraction.

- **Multiple Models**

  The toolbox supports a wide variety of state-of-the-art models for text detection, text recognition and key information extraction.

  The modular design of MMOCR enables users to define their own optimizers, data preprocessors, and model components such as backbones, necks and heads as well as losses. Please refer to [Overview](https://mmocr.readthedocs.io/en/dev-1.x/get_started/overview.html) for how to construct a customized model.

- **Numerous Utilities**

  The toolbox provides a comprehensive set of utilities which can help users assess the performance of models. It includes visualizers which allow visualization of images, ground truths as well as predicted bounding boxes, and a validation tool for evaluating checkpoints during training.  It also includes data converters to demonstrate how to convert your own data to the annotation files which the toolbox supports.

MMOCR depends on [PyTorch](https://pytorch.org/), [MMEngine](https://github.com/open-mmlab/mmengine), [MMCV](https://github.com/open-mmlab/mmcv) and [MMDetection](https://github.com/open-mmlab/mmdetection).
Please refer to [Install Guide](https://mmocr.readthedocs.io/en/dev-1.x/get_started/install.html) for more detailed instruction.

```shell

```shell
cd mmocr
```

Please see [Quick Run](https://mmocr.readthedocs.io/en/dev-1.x/get_started/quick_run.html) for the basic usage of MMOCR.

## [Model Zoo](https://mmocr.readthedocs.io/en/dev-1.x/modelzoo.html)

Supported algorithms:

<summary>BackBone</summary>

<summary>Text Detection</summary>

<summary>Text Recognition</summary>

<summary>Key Information Extraction</summary>

<summary>Text Spotting</summary>

Please refer to [model_zoo](https://mmocr.readthedocs.io/en/dev-1.x/modelzoo.html) for more details.

[Here](projects/README.md) are some implementations of SOTA models and solutions built on MMOCR, which are supported and maintained by community users. These projects demonstrate the best practices based on MMOCR for research and product development. We welcome and appreciate all the contributions to OpenMMLab ecosystem.

We appreciate all contributions to improve MMOCR. Please refer to [CONTRIBUTING.md](.github/CONTRIBUTING.md) for the contributing guidelines.

MMOCR is an open-source project that is contributed by researchers and engineers from various colleges and companies. We appreciate all the contributors who implement their methods or add new features, as well as users who give valuable feedbacks.

We hope the toolbox and benchmark could serve the growing research community by providing a flexible toolkit to reimplement existing methods and develop their own new OCR methods.

```bibtex
}
```

## License

## OpenMMLab Family

- [MMEngine](https://github.com/open-mmlab/mmengine): OpenMMLab foundational library for training deep learning models

## Welcome to the OpenMMLab community

Scan the QR code below to follow the OpenMMLab team's [**Zhihu Official Account**](https://www.zhihu.com/people/openmmlab) and join the OpenMMLab team's [**QQ Group**](https://jq.qq.com/?_wv=1027&k=aCvMxdr3), or join the official communication WeChat group by adding the WeChat, or join our [**Slack**](https://join.slack.com/t/mmocrworkspace/shared_invite/zt-1ifqhfla8-yKnLO_aKhVA2h71OrK8GZw)

</div>

We will provide you with the OpenMMLab community

</div>

- 📢 share the latest core technologies of AI frameworks
- 💻 Explaining PyTorch common module source Code
- 📰 News related to the release of OpenMMLab
- 🚀 Introduction of cutting-edge algorithms developed by OpenMMLab
  🏃 Get the more efficient answer and feedback
- 🔥 Provide a platform for communication with developers from all walks of life

The OpenMMLab community looks forward to your participation! 👬

# Contributing guidance

Welcome to the MMagic community, we are committed to building a Multimodal Advanced, Generative, and Intelligent Creation Toolbox.

This section introduces following contents:

- [Contributing guidance](#contributing-guidance)
  - [Pull Request Workflow](#pull-request-workflow)
    - [1. Fork and clone](#1-fork-and-clone)
    - [2. Configure pre-commit](#2-configure-pre-commit)
    - [3. Create a development branch](#3-create-a-development-branch)
    - [4. Commit the code and pass the unit test](#4-commit-the-code-and-pass-the-unit-test)
    - [5. Push the code to remote](#5-push-the-code-to-remote)
    - [6. Create a Pull Request](#6-create-a-pull-request)
    - [7. Resolve conflicts](#7-resolve-conflicts)
  - [Guidance](#guidance)
    - [Unit test](#unit-test)
    - [Document rendering](#document-rendering)
  - [PR Specs](#pr-specs)

- [Unit test](#unit-test)

All kinds of contributions are welcomed, including but not limited to

```shell
```

After that, you should ddd official repository as the upstream repository

```bash
```

```bash
```

```{note}

```

```{note}
Here's a brief introduction to origin and upstream. When we use "git clone", we create an "origin" remote by default, which points to the repository cloned from. As for "upstream", we add it ourselves to point to the target repository. Of course, if you don't like the name "upstream", you could name it as you wish. Usually, we'll push the code to "origin". If the pushed code conflicts with the latest code in official("upstream"), we should pull the latest code from upstream to resolve the conflicts, and then push to "origin" again. The posted Pull Request will be updated automatically.
```

```

You should configure [pre-commit](https://pre-commit.com/#intro) in the local development environment to make sure the code style matches that of OpenMMLab. **Note**: The following code should be executed under the mmagic directory.

```shell
```

```shell
```

```{note}
Chinese users may fail to download the pre-commit hooks due to the network issue. In this case, you could download these hooks from gitee by setting the .pre-commit-config-zh-cn.yaml

```

If we want to commit our code bypassing the pre-commit hook, we can use the `--no-verify` option(**only for temporarily commit**.

```shell
```

After configuring the pre-commit, we should create a branch based on the main branch to develop the new feature or fix the bug. The proposed branch name is `username/pr_name`

```shell
```

```shell
```

In subsequent development, if the main branch of the local repository is behind the main branch of "upstream", we need to pull the upstream for synchronization, and then execute the above command:

```shell
```

- mmagic introduces mypy to do static type checking to increase the robustness of the code. Therefore, we need to add Type Hints to our code and pass the mypy check. If you are not familiar with Type Hints, you can refer to [this tutorial](https://docs.python.org/3/library/typing.html).

  ```shell

  ```

pytest tests/test_runner/test_runner.py
  ```

```shell
```

**note**

(a) The Pull Request description should contain the reason for the change, the content of the change, and the impact of the change, and be associated with the relevant Issue (see [documentation](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue)

mmagic will run unit test for the posted Pull Request on different platforms (Linux, Window, Mac), based on different versions of Python, PyTorch, CUDA to make sure the code is correct. We can see the specific test information by clicking `Details` in the above image so that we can modify the code.

If your local branch conflicts with the latest main branch of "upstream", you'll need to resolove them. There are two ways to do this:

```shell
```

or

```shell
```

#### Unit test

We should make sure the committed code will not decrease the coverage of unit test, we could run the following command to check the coverage of unit test:

```shell
```

```shell
```

```shell
make html
```

Style configurations of yapf and isort can be found in [setup.cfg](../../../setup.cfg).

The config for a pre-commit hook is stored in [.pre-commit-config](../../../.pre-commit-config.yaml).

# 检测器鲁棒性检查

## 介绍

我们提供了在 [Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming](https://arxiv.org/abs/1907.07484) 中定义的「图像损坏基准测试」上测试目标检测和实例分割模型的工具。
此页面提供了如何使用该基准测试的基本教程。

```latex
}
```

## 关于基准测试

要将结果提交到基准测试，请访问[基准测试主页](https://github.com/bethgelab/robust-detection-benchmark)

基准测试是仿照 [imagenet-c 基准测试](https://github.com/hendrycks/robustness)，由 Dan Hendrycks 和 Thomas Dietterich 在[Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://arxiv.org/abs/1903.12261)(ICLR 2019)中发表。

图像损坏变换功能包含在此库中，但可以使用以下方法单独安装：

```shell
```

与 imagenet-c 相比，我们必须进行一些更改以处理任意大小的图像和灰度图像。
我们还修改了“运动模糊”和“雪”损坏，以解除对于 linux 特定库的依赖，
否则必须单独安装这些库。有关详细信息，请参阅 [imagecorruptions](https://github.com/bethgelab/imagecorruptions)。

我们提供了一个测试脚本来评估模型在基准测试中提供的各种损坏变换组合下的性能。

### 在数据集上测试

- [x] 单张 GPU 测试
- [ ] 多张 GPU 测试
- [ ] 可视化检测结果

您可以使用以下命令在基准测试中使用 15 种损坏变换来测试模型性能。

```shell
python tools/analysis_tools/test_robustness.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [--eval ${EVAL_METRICS}]
```

也可以选择其它不同类型的损坏变换。

```shell
# noise
python tools/analysis_tools/test_robustness.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [--eval ${EVAL_METRICS}] --corruptions noise

# blur
python tools/analysis_tools/test_robustness.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [--eval ${EVAL_METRICS}] --corruptions blur

# wetaher

# blur

# wetaher
python tools/analysis_tools/test_robustness.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [--eval ${EVAL_METRICS}] --corruptions weather

# digital
python tools/analysis_tools/test_robustness.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [--eval ${EVAL_METRICS}] --corruptions digital
```

或者使用一组自定义的损坏变换，例如：

```shell
```

最后，我们也可以选择施加在图像上的损坏变换的严重程度。
严重程度从 1 到 5 逐级增强，0 表示不对图像施加损坏变换，即原始图像数据。

```shell

```shell

```

## 模型测试结果

下表是各模型在 COCO 2017val 上的测试结果。

由于对图像的损坏变换存在随机性，测试结果可能略有不同。

# benchmark
op

# naive C with openmp
for for for

# unroll, first try
h

# register allocation
kernels

# unroll, second try
simd

# neon intrinsics
optional

# naive neon assembly with pld
asm

# pipeline optimize, first try
more register load mla

# pipeline optimize, second try
interleave load mla

# pipeline optimize, third try
loop tail

# usual practice, load/save
233

# usual practice, unroll
233

# usual practice, save register
233

# CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmark

<details>

```bibtex
}
```

# 学习配置文件

为了管理深度学习实验的各种设置，我们使用配置文件来记录所有这些配置。这种配置文件系统具有模块化和继承特性，更多细节可以在{external+mmengine:doc}`MMEngine 中的教程 <advanced_tutorials/config>`。

MMPretrain 主要使用 python 文件作为配置文件，所有配置文件都放置在 [`configs`](https://github.com/open-mmlab/mmpretrain/tree/main/configs) 文件夹下，目录结构如下所示：

```text
MMPretrain/
    ├── configs/
    │   ├── _base_/                       # primitive configuration folder
    │   │   ├── datasets/                      # primitive datasets
    │   │   ├── models/                        # primitive models
    │   │   ├── schedules/                     # primitive schedules
    │   │   └── default_runtime.py             # primitive runtime setting
    │   ├── beit/                         # BEiT Algorithms Folder
    │   ├── mae/                          # MAE Algorithms Folder
    │   ├── mocov2/                       # MoCoV2 Algorithms Folder
    │   ├── resnet/                       # ResNet Algorithms Folder
    │   ├── swin_transformer/             # Swin Algorithms Folder

│   ├── resnet/                       # ResNet Algorithms Folder
    │   ├── vision_transformer/           # ViT Algorithms Folder
    │   ├── ...
    └── ...
```

可以使用 `python tools/misc/print_config.py /PATH/TO/CONFIG` 命令来查看完整的配置信息，从而方便检查所对应的配置文件。

本文主要讲解 MMPretrain 配置文件的命名和结构，以及如何基于已有的配置文件修改，并以 [ResNet50 配置文件](https://github.com/open-mmlab/mmpretrain/blob/main/configs/resnet/resnet50_8xb32_in1k.py) 逐行解释。

## 配置文件结构

在 `configs/_base_` 文件夹下有 4 个基本组件类型，分别是：

- [模型(model)](https://github.com/open-mmlab/mmpretrain/tree/main/configs/_base_/models)
- [数据(data)](https://github.com/open-mmlab/mmpretrain/tree/main/configs/_base_/datasets)
- [训练策略(schedule)](https://github.com/open-mmlab/mmpretrain/tree/main/configs/_base_/schedules)
- [运行设置(runtime)](https://github.com/open-mmlab/mmpretrain/blob/main/configs/_base_/default_runtime.py)

你可以通过继承一些基本配置文件轻松构建自己的训练配置文件。我们称这些被继承的配置文件为 _原始配置文件_，如 `_base_` 文件夹中的文件一般仅作为原始配置文件。

下面使用 [ResNet50 配置文件](https://github.com/open-mmlab/mmpretrain/blob/main/configs/resnet/resnet50_8xb32_in1k.py) 作为案例进行说明并注释每一行含义。

```python
_base_ = [                                    # 此配置文件将继承所有 `_base_` 中的配置
    '../_base_/models/resnet50.py',           # 模型配置
    '../_base_/datasets/imagenet_bs32.py',    # 数据配置
    '../_base_/schedules/imagenet_bs256.py',  # 训练策略配置
    '../_base_/default_runtime.py'            # 默认运行设置
]
```

我们将在下面分别解释这四个原始配置文件。

### 模型配置

模型原始配置文件包含一个 `model` 字典数据结构，主要包括网络结构、损失函数等信息：

- `type`：算法类型，我们支持了多种任务
  - 对于图像分类任务，通常为 `ImageClassifier`，更多细节请参考 [API 文档](mmpretrain.models.classifiers)。
  - 对于自监督任务，有多种类型的算法，例如 `MoCoV2`, `BEiT`, `MAE` 等。更多细节请参考 [API 文档](mmpretrain.models.selfsup)。
  - 对于图像检索任务，通常为 `ImageToImageRetriever`，更多细节请参考 [API 文档](mmpretrain.models.retrievers).

- 对于自监督任务，有多种类型的算法，例如 `MoCoV2`, `BEiT`, `MAE` 等。更多细节请参考 [API 文档](mmpretrain.models.selfsup)。

通常，我们使用 **`type`字段** 来指定组件的类，并使用其他字段来传递类的初始化参数。{external+mmengine:doc}`注册器教程 <advanced_tutorials/registry>` 对其进行了详细描述。

这里我们以 [`ImageClassifier`](mmpretrain.models.classifiers.ImageClassifier) 的配置字段为例，对初始化参数进行说明：

- `backbone`： 主干网络设置，主干网络为主要的特征提取网络，比如 `ResNet`, `Swin Transformer`, `Vision Transformer` 等等。更多可用选项请参考 [API 文档](mmpretrain.models.backbones)。
  - 对于自监督学习，有些主干网络需要重新实现，您可以在 [API 文档](mmpretrain.models.selfsup) 中获取更多细节。
- `neck`： 颈网络设置，颈网络主要是连接主干网和头网络的中间部分，比如 `GlobalAveragePooling` 等，更多可用选项请参考 [API 文档](mmpretrain.models.necks)。
- `head`： 头网络设置，头网络主要是与具体任务关联的部件，如图像分类、自监督训练等，更多可用选项请参考 [API 文档](mmpretrain.models.heads)。
  - `loss`： 损失函数设置， 支持 `CrossEntropyLoss`, `LabelSmoothLoss`, `PixelReconstructionLoss` 等，更多可用选项参考 [API 文档](mmpretrain.models.losses)。

- `loss`： 损失函数设置， 支持 `CrossEntropyLoss`, `LabelSmoothLoss`, `PixelReconstructionLoss` 等，更多可用选项参考 [API 文档](mmpretrain.models.losses)。
- `data_preprocessor`: 图像输入的预处理模块，输入在进入模型前的预处理操作，例如 `ClsDataPreprocessor`, 有关详细信息，请参阅 [API 文档](mmpretrain.models.utils.data_preprocessor)。
- `train_cfg`: `ImageClassifier` 的额外训练配置。在 `ImageClassifier` 中，我们使用这一参数指定批数据增强设置，比如 `Mixup` 和 `CutMix`。详见[文档](mmpretrain.models.utils.batch_augments)。

以下是 ResNet50 的模型配置['configs/_base_/models/resnet50.py'](https://github.com/open-mmlab/mmpretrain/blob/main/configs/_base_/models/resnet50.py)：

```python
    type='ImageClassifier',     # 主模型类型（对于图像分类任务，使用 `ImageClassifier`）
        type='ResNet',          # 主干网络类型
        # 除了 `type` 之外的所有字段都来自 `ResNet` 类的 __init__ 方法
        # 可查阅 https://mmpretrain.readthedocs.io/zh_CN/latest/api/generated/mmpretrain.models.backbones.ResNet.html
        depth=50,
        num_stages=4,           # 主干网络状态(stages)的数目，这些状态产生的特征图作为后续的 head 的输入。

depth=50,
        out_indices=(3, ),      # 输出的特征图输出索引。
        frozen_stages=-1,       # 冻结主干网的层数
    neck=dict(type='GlobalAveragePooling'),    # 颈网络类型
        type='LinearClsHead',         # 分类颈网络类型
        # 除了 `type` 之外的所有字段都来自 `LinearClsHead` 类的 __init__ 方法
        # 可查阅 https://mmpretrain.readthedocs.io/zh_CN/latest/api/generated/mmpretrain.models.heads.LinearClsHead.html
        loss=dict(type='CrossEntropyLoss', loss_weight=1.0), # 损失函数配置信息
        topk=(1, 5),                 # 评估指标，Top-k 准确率， 这里为 top1 与 top5 准确率
    ))
```

### 数据

数据原始配置文件主要包括预处理设置、dataloader 以及 评估器等设置：

- `data_preprocessor`: 模型输入预处理配置，与 `model.data_preprocessor` 相同，但优先级更低。
- `train_evaluator | val_evaluator | test_evaluator`: 构建评估器，参考 [API 文档](mmpretrain.evaluation)。

- `train_dataloader | val_dataloader | test_dataloader`: 构建 dataloader
  - `batch_size`: 每个 GPU 的 batch size
  - `num_workers`: 每个 GPU 的线程数
  - `sampler`: 采样器配置
  - `dataset`: 数据集配置
    - `type`:  数据集类型， MMPretrain 支持 `ImageNet`、 `Cifar` 等数据集 ，参考 [API 文档](mmpretrain.datasets)
    - `pipeline`:  数据处理流水线，参考相关教程文档 [如何设计数据处理流水线](../advanced_guides/pipeline.md)

以下是 ResNet50 的数据配置 ['configs/_base_/datasets/imagenet_bs32.py'](https://github.com/open-mmlab/mmpretrain/blob/main/configs/_base_/datasets/imagenet_bs32.py)：

```python
dataset_type = 'ImageNet'
# 预处理配置
data_preprocessor = dict(
    # 输入的图片数据通道以 'RGB' 顺序
    mean=[123.675, 116.28, 103.53],    # 输入图像归一化的 RGB 通道均值
    std=[58.395, 57.12, 57.375],       # 输入图像归一化的 RGB 通道标准差
    to_rgb=True,                       # 是否将通道翻转，从 BGR 转为 RGB 或者 RGB 转为 BGR
)

std=[58.395, 57.12, 57.375],       # 输入图像归一化的 RGB 通道标准差
)

    dict(type='LoadImageFromFile'),     # 读取图像
    dict(type='RandomResizedCrop', scale=224),     # 随机放缩裁剪
    dict(type='RandomFlip', prob=0.5, direction='horizontal'),   # 随机水平翻转
    dict(type='PackInputs'),         # 准备图像以及标签
]

    dict(type='ResizeEdge', scale=256, edge='short'),  # 缩放短边尺寸至 256px
    dict(type='CenterCrop', crop_size=224),     # 中心裁剪
    dict(type='PackInputs'),                 # 准备图像以及标签
]

# 构造训练集 dataloader
    batch_size=32,                     # 每张 GPU 的 batchsize
    num_workers=5,                     # 每个 GPU 的线程数
    dataset=dict(                      # 训练数据集
        data_root='data/imagenet',
        ann_file='meta/train.txt',

dataset=dict(                      # 训练数据集
    sampler=dict(type='DefaultSampler', shuffle=True),   # 默认采样器
    persistent_workers=True,                             # 是否保持进程，可以缩短每个 epoch 的准备时间
)

# 构造验证集 dataloader
    num_workers=5,
        ann_file='meta/val.txt',
        data_prefix='val',
)
# 验证集评估设置，使用准确率为指标， 这里使用 topk1 以及 top5 准确率
val_evaluator = dict(type='Accuracy', topk=(1, 5))

test_dataloader = val_dataloader  # test dataloader 配置，这里直接与 val_dataloader 相同
test_evaluator = val_evaluator    # 测试集的评估配置，这里直接与 val_evaluator 相同
```

```{note}

```

```{note}
预处理配置（`data_preprocessor`）既可以作为 `model` 的一个子字段，也可以定义在外部的 `data_preprocessor` 字段，
同时配置时，优先使用 `model.data_preprocessor` 的配置。
```

### 训练策略

训练策略原始配置文件主要包括预优化器设置和训练、验证及测试的循环控制器(LOOP)：

- `optim_wrapper`: 优化器装饰器配置信息，我们使用优化器装饰配置优化进程。
  - `optimizer`: 支持 `pytorch` 所有的优化器，参考相关 {external+mmengine:doc}`MMEngine <tutorials/optim_wrapper>` 文档。
  - `paramwise_cfg`: 根据参数的类型或名称设置不同的优化参数，参考相关 [学习策略文档](../advanced_guides/schedule.md) 文档。
  - `accumulative_counts`: 积累几个反向传播后再优化参数，你可以用它通过小批量来模拟大批量。
- `param_scheduler` : 学习率策略，你可以指定训练期间的学习率和动量曲线。有关详细信息，请参阅 MMEngine 中的 {external+mmengine:doc}`文档 <tutorials/param_scheduler>`。
- `train_cfg | val_cfg | test_cfg`: 训练、验证以及测试的循环执行器配置，请参考相关的{external+mmengine:doc}`MMEngine 文档 <design/runner>`。

以下是 ResNet50 的训练策略配置['configs/_base_/schedules/imagenet_bs256.py'](https://github.com/open-mmlab/mmpretrain/blob/main/configs/_base_/schedules/imagenet_bs256.py)：

```python
    # 使用 SGD 优化器来优化参数
    optimizer=dict(type='SGD', lr=0.1, momentum=0.9, weight_decay=0.0001))

# 学习率参数的调整策略
# 'MultiStepLR' 表示使用多步策略来调度学习率（LR）。
    type='MultiStepLR', by_epoch=True, milestones=[30, 60, 90], gamma=0.1)

# 训练的配置，迭代 100 个 epoch，每一个训练 epoch 后都做验证集评估
# 'by_epoch=True' 默认使用 `EpochBaseLoop`,  'by_epoch=False' 默认使用 `IterBaseLoop`
train_cfg = dict(by_epoch=True, max_epochs=100, val_interval=1)
# 使用默认的验证循环控制器
val_cfg = dict()
# 使用默认的测试循环控制器
test_cfg = dict()

# 通过默认策略自动缩放学习率，此策略适用于总批次大小 256
# 如果你使用不同的总批量大小，比如 512 并启用自动学习率缩放
# 我们将学习率扩大到 2 倍
auto_scale_lr = dict(base_batch_size=256)
```

### 运行设置

本部分主要包括保存权重策略、日志配置、训练参数、断点权重路径和工作目录等等。

```

### 运行设置

以下是几乎所有算法都使用的运行配置['configs/_base_/default_runtime.py'](https://github.com/open-mmlab/mmpretrain/blob/main//configs/_base_/default_runtime.py)：

```python
# 默认所有注册器使用的域
default_scope = 'mmpretrain'

# 配置默认的 hook

    # 每 100 次迭代打印一次日志。

    # 启用默认参数调度 hook。

    # 每个 epoch 保存检查点。

    # 验证结果可视化，默认不启用，设置 True 时启用。
)

# 配置环境
   # 是否开启 cudnn benchmark

    # 设置多进程参数

# 验证结果可视化，默认不启用，设置 True 时启用。
)

# 配置环境

    # 设置多进程参数

    # 设置分布式参数
)

# 设置可视化工具
vis_backends = [dict(type='LocalVisBackend')] # 使用磁盘(HDD)后端
    type='UniversalVisualizer', vis_backends=vis_backends, name='visualizer')

# 设置日志级别

# 从哪个检查点加载

# 是否从加载的检查点恢复训练
```

## 继承并修改配置文件

为了精简代码、更快的修改配置文件以及便于理解，我们建议继承现有方法。

对于在同一算法文件夹下的所有配置文件，MMPretrain 推荐只存在 **一个** 对应的 _原始配置_ 文件。
所有其他的配置文件都应该继承 _原始配置_ 文件，这样就能保证配置文件的最大继承深度为 3。

```

例如，如果在 ResNet 的基础上做了一些修改，用户首先可以通过指定 `_base_ = './resnet50_8xb32_in1k.py'`（相对于你的配置文件的路径），来继承基础的 ResNet 结构、数据集以及其他训练配置信息，然后修改配置文件中的必要参数以完成继承。如想在基础 resnet50 的基础上使用 `CutMix` 训练增强，将训练轮数由 100 改为 300 和修改学习率衰减轮数，同时修改数据集路径，可以建立新的配置文件 `configs/resnet/resnet50_8xb32-300e_in1k.py`， 文件中写入以下内容：

```python
# 在 'configs/resnet/' 创建此文件
_base_ = './resnet50_8xb32_in1k.py'

# 模型在之前的基础上使用 CutMix 训练增强
        augments=dict(type='CutMix', alpha=1.0)
    )
)

# 优化策略在之前基础上训练更多个 epoch
train_cfg = dict(max_epochs=300, val_interval=10)  # 训练 300 个 epoch，每 10 个 epoch 评估一次
param_scheduler = dict(step=[150, 200, 250])   # 学习率调整也有所变动

# 使用自己的数据集目录
    dataset=dict(data_root='mydata/imagenet/train'),
)

)
    batch_size=64,                  # 验证时没有反向传播，可以使用更大的 batchsize
    dataset=dict(data_root='mydata/imagenet/val'),
)
    batch_size=64,                  # 测试时没有反向传播，可以使用更大的 batchsize
)
```

用一些中间变量，中间变量让配置文件更加清晰，也更容易修改。

例如数据集里的 `train_pipeline` / `test_pipeline` 是作为数据流水线的中间变量。我们首先要定义它们，然后将它们传递到 `train_dataloader` / `test_dataloader` 中。如果想修改训练或测试时输入图片的大小，就需要修改 `train_pipeline` / `test_pipeline` 这些中间变量。

```python
bgr_mean = [103.53, 116.28, 123.675]
    dict(type='RandomResizedCrop', scale=224, backend='pillow', interpolation='bicubic'),
    dict(type='RandomFlip', prob=0.5, direction='horizontal'),
    dict(
        type='RandAugment',

dict(type='RandomResizedCrop', scale=224, backend='pillow', interpolation='bicubic'),
    dict(
        policies='timm_increasing',
        num_policies=2,
        total_level=10,
        magnitude_level=6,
        magnitude_std=0.5,
        hparams=dict(pad_val=[round(x) for x in bgr_mean], interpolation='bicubic')),
    dict(type='PackInputs'),
]

    dict(type='ResizeEdge', scale=236, edge='short', backend='pillow', interpolation='bicubic'),
]

val_dataloader = dict(dataset=dict(pipeline=val_pipeline))
test_dataloader = dict(dataset=dict(pipeline=val_pipeline))
```

```

有时，您需要设置 `_delete_=True` 去忽略基础配置文件里的一些域内容。可以查看 {external+mmengine:doc}`MMEngine 文档 <advanced_tutorials/config>` 进一步了解该设计。

以下是一个简单应用案例。 如果在上述 ResNet50 案例中 使用余弦调度 ,使用继承并直接修改会报 `get unexcepected keyword 'step'` 错，因为基础配置文件 `param_scheduler` 域信息的 `'step'` 字段被保留下来了，需要加入 `_delete_=True` 去忽略基础配置文件里的 `param_scheduler` 相关域内容：

```python
_base_ = '../../configs/resnet/resnet50_8xb32_in1k.py'

# 学习率调整策略
param_scheduler = dict(type='CosineAnnealingLR', by_epoch=True, _delete_=True)
```

### 引用基础配置文件里的变量

有时，您可以引用 `_base_` 配置信息的一些域内容，这样可以避免重复定义。可以查看 {external+mmengine:doc}`MMEngine 文档 <advanced_tutorials/config>` 进一步了解该设计。

```

以下是一个简单应用案例，在训练数据预处理流水线中使用 `auto augment` 数据增强，参考配置文件 [`configs/resnest/resnest50_32xb64_in1k.py`](https://github.com/open-mmlab/mmpretrain/blob/main/configs/resnest/resnest50_32xb64_in1k.py)。 在定义 `train_pipeline` 时，可以直接在 `_base_` 中加入定义 auto augment 数据增强的文件命名，再通过 `{{_base_.auto_increasing_policies}}` 引用变量：

```python
_base_ = [
    '../_base_/models/resnest50.py', '../_base_/datasets/imagenet_bs64.py',
    '../_base_/default_runtime.py', './_randaug_policies.py',
]

    dict(
        policies={{_base_.policies}},    # 这里使用了 _base_ 里的 `policies` 参数。
        magnitude_level=12),
    dict(type='EfficientNetRandomCrop', scale=224, backend='pillow'),

num_policies=2,
    dict(type='ColorJitter', brightness=0.4, contrast=0.4, saturation=0.4),
    dict(
        type='Lighting',
        eigval=EIGVAL,
        eigvec=EIGVEC,
        alphastd=0.1,
        to_rgb=False),
]

```

## 通过命令行参数修改配置信息

当用户使用脚本 "tools/train.py" 或者 "tools/test.py" 提交任务，以及使用一些工具脚本时，可以通过指定 `--cfg-options` 参数来直接修改所使用的配置文件内容。

- 更新配置文件内的字典

  可以按照原始配置文件中字典的键的顺序指定配置选项。
  例如，`--cfg-options model.backbone.norm_eval=False` 将主干网络中的所有 BN 模块更改为 `train` 模式。

- 更新配置文件内列表的键

  一些配置字典在配置文件中会形成一个列表。例如，训练流水线 `data.train.pipeline` 通常是一个列表。

可以按照原始配置文件中字典的键的顺序指定配置选项。

  例如，`[dict(type='LoadImageFromFile'), dict(type='TopDownRandomFlip', flip_prob=0.5), ...]` 。如果要将流水线中的 `'flip_prob=0.5'` 更改为 `'flip_prob=0.0'`，您可以这样指定 `--cfg-options data.train.pipeline.1.flip_prob=0.0` 。

  当配置文件中需要更新的是一个列表或者元组，例如，配置文件通常会设置 `val_evaluator = dict(type='Accuracy', topk=(1, 5))`，用户如果想更改 `topk`，
  需要指定 `--cfg-options val_evaluator.topk="(1,3)"`。注意这里的引号 " 对于列表以及元组数据类型的修改是必要的，
  并且 **不允许** 引号内所指定的值的书写存在空格。

# Welcome to Projects of MMPreTrain

In this folder, we welcome all contribution of vision deep-learning backbone from community.

Here, these requirements, e.g. code standards, are not that strict as in core package. Thus, developers from the community can implement their algorithms much more easily and efficiently in MMPreTrain. We appreciate all contributions from community to make MMPreTrain greater.

- [New Model Guide](https://mmpretrain.readthedocs.io/en/latest/advanced_guides/modules.html)

- [Contribution Guide](https://mmpretrain.readthedocs.io/en/latest/notes/contribution_guide.html)

  The guides for new contributors about how to add your projects to MMPreTrain.

- [Discussions](https://github.com/open-mmlab/mmpretrain/discussions)

  Welcome to start discussion!

# Confusion Matrix

MMPretrain provides `tools/analysis_tools/confusion_matrix.py` tool to calculate and visualize the confusion matrix. For an introduction to the confusion matrix, see [link](https://en.wikipedia.org/wiki/Confusion_matrix).

## Command-line Usage

```shell
    [--show] \
```

- `--show`: If or not to show the matplotlib visualization result of the confusion matrix, the default is `False`.
- `--show-path`: If `show` is True, the path where the results are saved is visualized.
- `--include-values`: Whether to add values to the visualization results.
- `--cmap`: The color map used for visualization results, `cmap`, which defaults to `viridis`.

* `--cfg-options`: Modifications to the configuration file, refer to [Learn about Configs](../user_guides/config.md).

**Examples of use**:

```shell
    --show
```

**output image**:

## **Basic Usage**

```python

```

## **Use with Evalutor**

```python
... ]

```

We appreciate all contributions to improve MMDetection3D. Please refer to [CONTRIBUTING.md](https://github.com/open-mmlab/mmcv/blob/master/CONTRIBUTING.md) in MMCV for more details about the contributing guideline.

# TPN

[Temporal Pyramid Network for Action Recognition](https://openaccess.thecvf.com/content_CVPR_2020/html/Yang_Temporal_Pyramid_Network_for_Action_Recognition_CVPR_2020_paper.html)

Visual tempo characterizes the dynamics and the temporal scale of an action. Modeling such visual tempos of different actions facilitates their recognition. Previous works often capture the visual tempo through sampling raw videos at multiple rates and constructing an input-level frame pyramid, which usually requires a costly multi-branch network to handle. In this work we propose a generic Temporal Pyramid Network (TPN) at the feature-level, which can be flexibly integrated into 2D or 3D backbone networks in a plug-and-play manner. Two essential components of TPN, the source of features and the fusion of features, form a feature hierarchy for the backbone so that it can capture action instances at various tempos. TPN also shows consistent improvements over other challenging baselines on several action recognition datasets. Specifically, when equipped with TPN, the 3D ResNet-50 with dense sampling obtains a 2% gain on the validation set of Kinetics-400. A further analysis also reveals that TPN gains most of its improvements on action classes that have large variances in their visual tempos, validating the effectiveness of TPN.

<img src="https://user-images.githubusercontent.com/34324155/143018779-1d2a398f-dbd3-405a-87e5-e188b61fcc86.png" width="800"/>
</div>

| frame sampling strategy |   resolution   | gpus | backbone | pretrain | top1 acc | top5 acc |   reference top1 acc    |   reference top5 acc    | testing protocol  | inference time(video/s) | gpu_mem(M) |    config    |    ckpt    |    log    |
| :---------------------: | :------------: | :--: | :------: | :------: | :------: | :------: | :---------------------: | :---------------------: | :---------------: | :---------------------: | :--------: | :----------: | :--------: | :-------: |

|          8x8x1          | short-side 320 | 8x2  | ResNet50 |   None   |  74.20   |  91.48   |            x            |            x            | 10 clips x 3 crop |            x            |    6916    | [config](/configs/recognition/tpn/tpn-slowonly_r50_8xb8-8x8x1-150e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tpn/tpn-slowonly_r50_8xb8-8x8x1-150e_kinetics400-rgb/tpn-slowonly_r50_8xb8-8x8x1-150e_kinetics400-rgb_20220913-97d0835d.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tpn/tpn-slowonly_r50_8xb8-8x8x1-150e_kinetics400-rgb/tpn-slowonly_r50_8xb8-8x8x1-150e_kinetics400-rgb.log) |

|          8x8x1          | short-side 320 |  8   | ResNet50 | ImageNet |  76.74   |  92.57   | [75.49](https://github.com/decisionforce/TPN/blob/master/MODELZOO.md) | [92.05](https://github.com/decisionforce/TPN/blob/master/MODELZOO.md) | 10 clips x 3 crop |            x            |    6916    | [config](/configs/recognition/tpn/tpn-slowonly_imagenet-pretrained-r50_8xb8-8x8x1-150e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tpn/tpn_imagenet_pretrained_slowonly_r50_8x8x1_150e_kinetics_rgb/tpn-slowonly_imagenet-pretrained-r50_8xb8-8x8x1-150e_kinetics400-rgb_20220913-fed3f4c1.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tpn/tpn_imagenet_pretrained_slowonly_r50_8x8x1_150e_kinetics_rgb/tpn_imagenet_pretrained_slowonly_r50_8x8x1_150e_kinetics_rgb.log) |

| frame sampling strategy | resolution | gpus | backbone | pretrain | top1 acc | top5 acc | reference top1 acc | reference top5 acc | testing protocol | inference time(video/s) | gpu_mem(M) |      config       |      ckpt       |      log       |
| :---------------------: | :--------: | :--: | :------: | :------: | :------: | :------: | :----------------: | :----------------: | :--------------: | :---------------------: | :--------: | :---------------: | :-------------: | :------------: |

|          1x1x8          | height 100 | 8x6  | ResNet50 |   TSM    |  51.87   |  79.67   |         x          |         x          | 8 clips x 3 crop |            x            |    8828    | [config](/configs/recognition/tpn/tpn-tsm_imagenet-pretrained-r50_8xb8-1x1x8-150e_sthv1-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tpn/tpn-tsm_imagenet-pretrained-r50_8xb8-1x1x8-150e_sthv1-rgb/tpn-tsm_imagenet-pretrained-r50_8xb8-1x1x8-150e_sthv1-rgb_20230221-940a3615.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tpn/tpn-tsm_imagenet-pretrained-r50_8xb8-1x1x8-150e_sthv1-rgb/tpn-tsm_imagenet-pretrained-r50_8xb8-1x1x8-150e_sthv1-rgb.log) |

:::{note}

:::{note}

2. The values in columns named after "reference" are the results got by testing the checkpoint released on the original repo and codes, using the same dataset with ours.

3. The validation set of Kinetics400 we used consists of 19796 videos. These videos are available at [Kinetics400-Validation](https://mycuhk-my.sharepoint.com/:u:/g/personal/1155136485_link_cuhk_edu_hk/EbXw2WX94J1Hunyt3MWNDJUBz-nHvQYhO9pvKqm6g39PMA?e=a9QldB). The corresponding [data list](https://download.openmmlab.com/mmaction/v1.0/dataset/k400_val/kinetics_val_list.txt) (each line is of the format 'video_id, num_frames, label_index') and the [label map](https://download.openmmlab.com/mmaction/v1.0/dataset/k400_val/kinetics_class2ind.txt) are also available.

:::

- [Something-something V1](/tools/data/sthv1/README.md)

## Train

## Train

```shell
```

Example: train TPN model on Kinetics-400 dataset in a deterministic option with periodic validation.

```shell
python tools/train.py configs/recognition/tpn/tpn-slowonly_r50_8xb8-8x8x1-150e_kinetics400-rgb.py \
    --work-dir work_dirs/tpn-slowonly_r50_8xb8-8x8x1-150e_kinetics400-rgb [--validate --seed 0 --deterministic]
```

## Test

```shell
```

Example: test TPN model on Kinetics-400 dataset and dump the result to a json file.

```shell

```shell
```

```shell
python tools/test.py configs/recognition/tpn/tpn-slowonly_r50_8xb8-8x8x1-150e_kinetics400-rgb.py \
```

```BibTeX
@inproceedings{yang2020tpn,
  title={Temporal Pyramid Network for Action Recognition},
  author={Yang, Ceyuan and Xu, Yinghao and Shi, Jianping and Dai, Bo and Zhou, Bolei},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
}
```

> **任务**: 图像超分辨率, 图像去噪, JPEG压缩伪影移除

<details>
<summary align="right">SwinIR (ICCVW'2021)</summary>

```bibtex
}
```

<br/>

在 Y 通道上进行评估，在评估之前裁剪每个边界中的 `scale` 像素。

|                                算法                                 | Set5 PSNR | Set14 PSNR | DIV2K PSNR | Set5 SSIM | Set14 SSIM | DIV2K SSIM | GPU 信息 |                                下载                                 |

| :-----------------------------------------------------------------: | :-------: | :--------: | :--------: | :-------: | :--------: | :--------: | :------: | :-----------------------------------------------------------------: |
| [swinir_x2s48w8d6e180_8xb4-lr2e-4-500k_div2k](./swinir_x2s48w8d6e180_8xb4-lr2e-4-500k_div2k.py) |  38.3240  |  34.1174   |  37.8921   |  0.9626   |   0.9230   |   0.9481   |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x2s48w8d6e180_8xb4-lr2e-4-500k_div2k-ed2d419e.pth) \| log |
| [swinir_x3s48w8d6e180_8xb4-lr2e-4-500k_div2k](./swinir_x3s48w8d6e180_8xb4-lr2e-4-500k_div2k.py) |  34.8640  |  30.7669   |  34.1397   |  0.9317   |   0.8508   |   0.8917   |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x3s48w8d6e180_8xb4-lr2e-4-500k_div2k-926950f1.pth) \| log |

| [swinir_x4s48w8d6e180_8xb4-lr2e-4-500k_div2k](./swinir_x4s48w8d6e180_8xb4-lr2e-4-500k_div2k.py) |  32.7315  |  28.9065   |  32.0953   |  0.9029   |   0.7915   |   0.8418   |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x4s48w8d6e180_8xb4-lr2e-4-500k_div2k-88e4903d.pth) \| log |
| [swinir_x2s64w8d6e180_8xb4-lr2e-4-500k_df2k](./swinir_x2s64w8d6e180_8xb4-lr2e-4-500k_df2k.py) |  38.3971  |  34.4149   |  37.9473   |  0.9629   |   0.9252   |   0.9488   |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x2s64w8d6e180_8xb4-lr2e-4-500k_df2k-69e15fb6.pth) \| log |
| [swinir_x3s64w8d6e180_8xb4-lr2e-4-500k_df2k](./swinir_x3s64w8d6e180_8xb4-lr2e-4-500k_df2k.py) |  34.9335  |  30.9258   |  34.2830   |  0.9323   |   0.8540   |   0.8939   |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x3s64w8d6e180_8xb4-lr2e-4-500k_df2k-d6982f7b.pth) \| log |

| [swinir_x4s64w8d6e180_8xb4-lr2e-4-500k_df2k](./swinir_x4s64w8d6e180_8xb4-lr2e-4-500k_df2k.py) |  32.9214  |  29.0792   |  32.3021   |  0.9053   |   0.7953   |   0.8451   |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x4s64w8d6e180_8xb4-lr2e-4-500k_df2k-0502d775.pth) \| log |

| [swinir_x2s64w8d4e60_8xb4-lr2e-4-500k_div2k](./swinir_x2s64w8d4e60_8xb4-lr2e-4-500k_div2k.py) |  38.1289  |  33.8404   |  37.5844   |  0.9617   |   0.9207   |   0.9459   |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x2s64w8d4e60_8xb4-lr2e-4-500k_div2k-131d3f64.pth) \| log |
| [swinir_x3s64w8d4e60_8xb4-lr2e-4-500k_div2k](./swinir_x3s64w8d4e60_8xb4-lr2e-4-500k_div2k.py) |  34.6037  |  30.5340   |  33.8394   |  0.9293   |   0.8468   |   0.8867   |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x3s64w8d4e60_8xb4-lr2e-4-500k_div2k-309cb239.pth) \| log |
| [swinir_x4s64w8d4e60_8xb4-lr2e-4-500k_div2k](./swinir_x4s64w8d4e60_8xb4-lr2e-4-500k_div2k.py) |  32.4343  |  28.7441   |  31.8636   |  0.8984   |   0.7861   |   0.8353   |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_x4s64w8d4e60_8xb4-lr2e-4-500k_div2k-d6622d03.pth) \| log |

在 Y 通道上进行评估。
我们使用 NIQE 作为指标。

|                                         算法                                          | RealSRSet+5images NIQE | GPU 信息 |                                         下载                                          |
| :-----------------------------------------------------------------------------------: | :--------------------: | :------: | :-----------------------------------------------------------------------------------: |
| [swinir_gan-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost](./swinir_gan-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py) |         5.7975         |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_gan-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-os-c6425057.pth) \| log |

| [swinir_psnr-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost](./swinir_psnr-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py) |         7.2738         |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_psnr-x2s64w8d6e180_8xb4-lr1e-4-600k_df2k-os-6f0c425f.pth) \| log |
| [swinir_gan-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost](./swinir_gan-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py) |         5.2329         |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_gan-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-os-36960d18.pth) \| log |
| [swinir_psnr-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost](./swinir_psnr-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-ost.py) |         7.7460         |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_psnr-x4s64w8d6e180_8xb4-lr1e-4-600k_df2k-os-a016a72f.pth) \| log |

| [swinir_gan-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-ost](./swinir_gan-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-ost.py) |         5.1464         |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_gan-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-os-9f1599b5.pth) \| log |
| [swinir_psnr-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-ost](./swinir_psnr-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-ost.py) |         7.6378         |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_psnr-x4s64w8d9e240_8xb4-lr1e-4-600k_df2k-os-25f1722a.pth) \| log |

在灰度图上进行评估。
我们使用 PSNR 作为指标。

|                                      算法                                       | Set12 PSNR | BSD68 PSNR | Urban100 PSNR | GPU 信息 |                                       下载                                       |

| :-----------------------------------------------------------------------------: | :--------: | :--------: | :-----------: | :------: | :------------------------------------------------------------------------------: |
| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN15](./swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN15.py) |  33.9731   |  32.5203   |    34.3424    |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN15-6782691b.pth) \| log |
| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN25](./swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN25.py) |  31.6434   |  30.1377   |    31.9493    |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN25-d0d8d4da.pth) \| log |

| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN50](./swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN50.py) |  28.5651   |  27.3157   |    28.6626    |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-grayDN50-54c9968a.pth) \| log |

|                                  算法                                   | CBSD68 PSNR | Kodak24 PSNR | McMaster PSNR | Urban100 PSNR | GPU 信息 |                                   下载                                   |
| :---------------------------------------------------------------------: | :---------: | :----------: | :-----------: | :-----------: | :------: | :----------------------------------------------------------------------: |

| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15](./swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15.py) |   34.4136   |   35.3555    |    35.6205    |    35.1836    |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN15-c74a2cee.pth) \| log |
| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25](./swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25.py) |   31.7626   |   32.9003    |    33.3198    |    32.9458    |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN25-df2b1c0c.pth) \| log |
| [swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50](./swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50.py) |   28.5346   |   29.8058    |    30.2027    |    29.8832    |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s128w8d6e180_8xb1-lr2e-4-1600k_dfwb-colorDN50-e369874c.pth) \| log |

我们使用 PSNR 和 SSIM 作为指标。

|                                   算法                                    | Classic5 PSNR | Classic5 SSIM | LIVE1 PSNR | LIVE1 SSIM | GPU 信息 |                                   下载                                    |
| :-----------------------------------------------------------------------: | :-----------: | :-----------: | :--------: | :--------: | :------: | :-----------------------------------------------------------------------: |
| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR10](./swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR10.py) |    30.2746    |    0.8254     |  29.8611   |   0.8292   |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR10-da93c8e9.pth) \| log |

| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR20](./swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR20.py) |    32.5331    |    0.8753     |  32.2667   |   0.8914   |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR20-d47367b1.pth) \| log |
| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR30](./swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR30.py) |    33.7504    |    0.8966     |  33.7001   |   0.9179   |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR30-52c083cf.pth) \| log |
| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR40](./swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR40.py) |    34.5377    |    0.9087     |  34.6846   |   0.9322   |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-grayCAR40-803e8d9b.pth) \| log |

| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR10](./swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR10.py) |    30.1019    |    0.8217     |  28.0676   |   0.8094   |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR10-09aafadc.pth) \| log |

| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR20](./swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR20.py) |    32.3489    |    0.8727     |  30.3489   |   0.8745   |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR20-b8a42b5e.pth) \| log |
| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR30](./swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR30.py) |    33.6028    |    0.8949     |  31.8235   |   0.9023   |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR30-e9fe6859.pth) \| log |
| [swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR40](./swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR40.py) |    34.4344    |    0.9076     |  32.7610   |   0.9179   |    8     | [model](https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR40-5b77a6e6.pth) \| log |

## 快速开始

**训练**

<details>

## 快速开始

**训练**

<details>

```shell
# CPU上训练

# color

# color

# 单个GPU上训练

# color

# color

# 多个GPU上训练

# color
```

**测试**

<details>

```shell
# CPU上测试

<details>

```shell
# CPU上测试

# 单个GPU上测试

# 多GPU测试

./tools/dist_test.sh configs/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR40.py https://download.openmmlab.com/mmediting/swinir/swinir_s126w7d6e180_8xb1-lr2e-4-1600k_dfwb-colorCAR30-e9fe6859.pth

```

# Preparing DF2K_OST Dataset

```bibtex
  title={Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data},
}
```

- The DIV2K dataset can be downloaded from [here](https://data.vision.ee.ethz.ch/cvl/DIV2K/) (We use the training set only).
- The Flickr2K dataset can be downloaded [here](https://cv.snu.ac.kr/research/EDSR/Flickr2K.tar) (We use the training set only).
- The OST dataset can be downloaded [here](https://openmmlab.oss-cn-hangzhou.aliyuncs.com/datasets/OST_dataset.zip) (We use the training set only).

Please first put all the images into the `GT` folder (naming does not need to be in order):

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── df2k_ost
│   │   ├── GT

```text
mmagic
├── mmagic
├── tools
├── data
│   │   ├── GT
│   │   │   ├── ...
...
```

For faster IO, we recommend to crop the images to sub-images. We provide such a script:

```shell
python tools/dataset_converters/df2k_ost/preprocess_df2k_ost_dataset.py --data-root ./data/df2k_ost
```

The generated data is stored under `df2k_ost` and the data structure is as follows, where `_sub` indicates the sub-images.

```text
mmagic
├── mmagic
├── tools
├── data
│   │   ├── GT
│   │   ├── GT_sub
│   │   ├── meta_info_df2k_ost.txt
...
```

```text

```text
```

Note that `preprocess_df2k_ost_dataset.py` will generate default annotation files.

## Prepare LMDB dataset for DF2K_OST

```shell
python tools/dataset_converters/df2k_ost/preprocess_df2k_ost_dataset.py --data-root ./data/df2k_ost --make-lmdb
```

Gesture recognition aims to recognize the hand gestures in the video, such as thumbs up.

Please follow [DATA Preparation](/docs/en/dataset_zoo/2d_hand_gesture.md) to prepare data.

## Demo

Please follow [Demo](/demo/docs/en/gesture_recognition_demo.md) to run the demo.

<img src="https://user-images.githubusercontent.com/15977946/172213082-afb9d71a-f2df-4509-932c-e47dc61ec7d7.gif" width="600px" alt>

  <img src="resources/mmseg-logo.png" width="600"/>
    <sup>
      </a>
    </sup>
    <sup>
      </a>
    </sup>
  </div>

[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/mmsegmentation)](https://pypi.org/project/mmsegmentation/)
[![PyPI](https://img.shields.io/pypi/v/mmsegmentation)](https://pypi.org/project/mmsegmentation)
[![docs](https://img.shields.io/badge/docs-latest-blue)](https://mmsegmentation.readthedocs.io/en/latest/)
[![badge](https://github.com/open-mmlab/mmsegmentation/workflows/build/badge.svg)](https://github.com/open-mmlab/mmsegmentation/actions)

[![codecov](https://codecov.io/gh/open-mmlab/mmsegmentation/branch/master/graph/badge.svg)](https://codecov.io/gh/open-mmlab/mmsegmentation)
[![license](https://img.shields.io/github/license/open-mmlab/mmsegmentation.svg)](https://github.com/open-mmlab/mmsegmentation/blob/main/LICENSE)
[![issue resolution](https://isitmaintained.com/badge/resolution/open-mmlab/mmsegmentation.svg)](https://github.com/open-mmlab/mmsegmentation/issues)
[![open issues](https://isitmaintained.com/badge/open/open-mmlab/mmsegmentation.svg)](https://github.com/open-mmlab/mmsegmentation/issues)

Documentation: <https://mmsegmentation.readthedocs.io/en/latest/>

</div>

</div>

<img src="https://user-images.githubusercontent.com/25839884/218346637-d30c8a0f-3eba-4699-8131-512fb06d46db.png" width="3%" alt="" /></a>

</div>

MMSegmentation is an open source semantic segmentation toolbox based on PyTorch.
It is a part of the OpenMMLab project.

The [main](https://github.com/open-mmlab/mmsegmentation/tree/main) branch works with PyTorch 1.6+.

### 🎉 Introducing MMSegmentation v1.0.0 🎉

We are thrilled to announce the official release of MMSegmentation's latest version! For this new release, the [main](https://github.com/open-mmlab/mmsegmentation/tree/main) branch serves as the primary branch, while the development branch is [dev-1.x](https://github.com/open-mmlab/mmsegmentation/tree/dev-1.x). The stable branch for the previous release remains as the [0.x](https://github.com/open-mmlab/mmsegmentation/tree/0.x) branch. Please note that the [master](https://github.com/open-mmlab/mmsegmentation/tree/master) branch will only be maintained for a limited time before being removed. We encourage you to be mindful of branch selection and updates during use. Thank you for your unwavering support and enthusiasm, and let's work together to make MMSegmentation even more robust and powerful! 💪

MMSegmentation v1.x brings remarkable improvements over the 0.x release, offering a more flexible and feature-packed experience. To utilize the new features in v1.x, we kindly invite you to consult our detailed [📚 migration guide](https://mmsegmentation.readthedocs.io/en/latest/migration/interface.html), which will help you seamlessly transition your projects. Your support is invaluable, and we eagerly await your feedback!

![demo image](resources/seg_demo.gif)

- **Unified Benchmark**

  We provide a unified benchmark toolbox for various semantic segmentation methods.

  We decompose the semantic segmentation framework into different components and one can easily construct a customized semantic segmentation framework by combining different modules.

- **Support of multiple methods out of box**

  The toolbox directly supports popular and contemporary semantic segmentation frameworks, *e.g.* PSPNet, DeepLabV3, PSANet, DeepLabV3+, etc.

  The training speed is faster than or comparable to other codebases.

v1.1.1 was released on 07/24/2023.
Please refer to [changelog.md](docs/en/notes/changelog.md) for details and release history.

- Support 24 medical image datasets in [projects](./projects/medical/).
- Add GDAL backend and support remote sensing datasets [LEVIR-CD](https://github.com/open-mmlab/mmsegmentation/pull/2903).
- Support [DDRNet](https://github.com/open-mmlab/mmsegmentation/pull/2855).

Please refer to [get_started.md](docs/en/get_started.md#installation) for installation and [dataset_prepare.md](docs/en/user_guides/2_dataset_prepare.md#prepare-datasets) for dataset preparation.

Please see [Overview](docs/en/overview.md) for the general introduction of MMSegmentation.

Please see [user guides](https://mmsegmentation.readthedocs.io/en/latest/user_guides/index.html#) for the basic usage of MMSegmentation.
There are also [advanced tutorials](https://mmsegmentation.readthedocs.io/en/latest/advanced_guides/index.html) for in-depth understanding of mmseg design and implementation .

A Colab tutorial is also provided. You may preview the notebook [here](demo/MMSegmentation_Tutorial.ipynb) or directly [run](https://colab.research.google.com/github/open-mmlab/mmsegmentation/blob/main/demo/MMSegmentation_Tutorial.ipynb) on Colab.

To migrate from MMSegmentation 0.x, please refer to [migration](docs/en/migration).

## Tutorial

<details>
<summary>Get Started</summary>

- [MMSeg overview](docs/en/overview.md)
- [MMSeg Installation](docs/en/get_started.md)
- [FAQ](docs/en/notes/faq.md)

<details>
<summary>MMSeg Basic Tutorial</summary>

<details>

- [Tutorial 1: Learn about Configs](docs/en/user_guides/1_config.md)
- [Tutorial 2: Prepare datasets](docs/en/user_guides/2_dataset_prepare.md)
- [Tutorial 3: Inference with existing models](docs/en/user_guides/3_inference.md)
- [Tutorial 4: Train and test with existing models](docs/en/user_guides/4_train_test.md)
- [Tutorial 5: Model deployment](docs/en/user_guides/5_deployment.md)
- [Useful Tools](docs/en/user_guides/useful_tools.md)
- [Feature Map Visualization](docs/en/user_guides/visualization_feature_map.md)
- [Visualization](docs/en/user_guides/visualization.md)

<details>
<summary>MMSeg Detail Tutorial</summary>

- [MMSeg Dataset](docs/en/advanced_guides/datasets.md)
- [MMSeg Models](docs/en/advanced_guides/models.md)
- [MMSeg Dataset Structures](docs/en/advanced_guides/structures.md)

- [MMSeg Data Transforms](docs/en/advanced_guides/transforms.md)
- [MMSeg Dataflow](docs/en/advanced_guides/data_flow.md)
- [MMSeg Training Engine](docs/en/advanced_guides/engine.md)
- [MMSeg Evaluation](docs/en/advanced_guides/evaluation.md)

<details>
<summary>MMSeg Development Tutorial</summary>

- [Add New Datasets](docs/en/advanced_guides/add_datasets.md)
- [Add New Metrics](docs/en/advanced_guides/add_metrics.md)
- [Add New Modules](docs/en/advanced_guides/add_models.md)
- [Add New Data Transforms](docs/en/advanced_guides/add_transforms.md)
- [Customize Runtime Settings](docs/en/advanced_guides/customize_runtime.md)
- [Training Tricks](docs/en/advanced_guides/training_tricks.md)
- [Contribute code to MMSeg](.github/CONTRIBUTING.md)

- [Contribute a standard dataset in projects](docs/zh_cn/advanced_guides/contribute_dataset.md)
- [NPU (HUAWEI Ascend)](docs/en/device/npu.md)
- [0.x → 1.x migration](docs/en/migration/interface.md)，[0.x → 1.x package](docs/en/migration/package.md)

## Benchmark and model zoo

<summary>Supported backbones:</summary>

- [x] ResNet (CVPR'2016)
- [x] ResNeXt (CVPR'2017)
- [x] [HRNet (CVPR'2019)](configs/hrnet)
- [x] [ResNeSt (ArXiv'2020)](configs/resnest)
- [x] [MobileNetV2 (CVPR'2018)](configs/mobilenet_v2)
- [x] [MobileNetV3 (ICCV'2019)](configs/mobilenet_v3)
- [x] [Vision Transformer (ICLR'2021)](configs/vit)
- [x] [Swin Transformer (ICCV'2021)](configs/swin)

- [x] [Twins (NeurIPS'2021)](configs/twins)
- [x] [BEiT (ICLR'2022)](configs/beit)
- [x] [ConvNeXt (CVPR'2022)](configs/convnext)
- [x] [MAE (CVPR'2022)](configs/mae)
- [x] [PoolFormer (CVPR'2022)](configs/poolformer)
- [x] [SegNeXt (NeurIPS'2022)](configs/segnext)

<summary>Supported methods:</summary>

- [x] [FCN (CVPR'2015/TPAMI'2017)](configs/fcn)
- [x] [ERFNet (T-ITS'2017)](configs/erfnet)
- [x] [UNet (MICCAI'2016/Nat. Methods'2019)](configs/unet)
- [x] [PSPNet (CVPR'2017)](configs/pspnet)
- [x] [DeepLabV3 (ArXiv'2017)](configs/deeplabv3)
- [x] [BiSeNetV1 (ECCV'2018)](configs/bisenetv1)
- [x] [PSANet (ECCV'2018)](configs/psanet)
- [x] [DeepLabV3+ (CVPR'2018)](configs/deeplabv3plus)
- [x] [UPerNet (ECCV'2018)](configs/upernet)
- [x] [ICNet (ECCV'2018)](configs/icnet)

- [x] [NonLocal Net (CVPR'2018)](configs/nonlocal_net)
- [x] [EncNet (CVPR'2018)](configs/encnet)
- [x] [Semantic FPN (CVPR'2019)](configs/sem_fpn)
- [x] [DANet (CVPR'2019)](configs/danet)
- [x] [APCNet (CVPR'2019)](configs/apcnet)
- [x] [EMANet (ICCV'2019)](configs/emanet)
- [x] [CCNet (ICCV'2019)](configs/ccnet)
- [x] [DMNet (ICCV'2019)](configs/dmnet)
- [x] [ANN (ICCV'2019)](configs/ann)
- [x] [GCNet (ICCVW'2019/TPAMI'2020)](configs/gcnet)
- [x] [FastFCN (ArXiv'2019)](configs/fastfcn)
- [x] [Fast-SCNN (ArXiv'2019)](configs/fastscnn)
- [x] [ISANet (ArXiv'2019/IJCV'2021)](configs/isanet)
- [x] [OCRNet (ECCV'2020)](configs/ocrnet)
- [x] [DNLNet (ECCV'2020)](configs/dnlnet)
- [x] [PointRend (CVPR'2020)](configs/point_rend)
- [x] [CGNet (TIP'2020)](configs/cgnet)

- [x] [BiSeNetV2 (IJCV'2021)](configs/bisenetv2)
- [x] [STDC (CVPR'2021)](configs/stdc)
- [x] [SETR (CVPR'2021)](configs/setr)
- [x] [DPT (ArXiv'2021)](configs/dpt)
- [x] [Segmenter (ICCV'2021)](configs/segmenter)
- [x] [SegFormer (NeurIPS'2021)](configs/segformer)
- [x] [K-Net (NeurIPS'2021)](configs/knet)
- [x] [MaskFormer (NeurIPS'2021)](configs/maskformer)
- [x] [Mask2Former (CVPR'2022)](configs/mask2former)
- [x] [PIDNet (ArXiv'2022)](configs/pidnet)
- [x] [DDRNet (T-ITS'2022)](configs/ddrnet)

<summary>Supported datasets:</summary>

- [x] [Cityscapes](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/en/user_guides/2_dataset_prepare.md#cityscapes)

- [x] [PASCAL VOC](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/en/user_guides/2_dataset_prepare.md#pascal-voc)
- [x] [ADE20K](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/en/user_guides/2_dataset_prepare.md#ade20k)
- [x] [Pascal Context](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/en/user_guides/2_dataset_prepare.md#pascal-context)
- [x] [COCO-Stuff 10k](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/en/user_guides/2_dataset_prepare.md#coco-stuff-10k)
- [x] [COCO-Stuff 164k](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/en/user_guides/2_dataset_prepare.md#coco-stuff-164k)
- [x] [CHASE_DB1](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/en/user_guides/2_dataset_prepare.md#chase-db1)

- [x] [DRIVE](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/en/user_guides/2_dataset_prepare.md#drive)
- [x] [HRF](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/en/user_guides/2_dataset_prepare.md#hrf)
- [x] [STARE](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/en/user_guides/2_dataset_prepare.md#stare)
- [x] [Dark Zurich](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/en/user_guides/2_dataset_prepare.md#dark-zurich)
- [x] [Nighttime Driving](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/en/user_guides/2_dataset_prepare.md#nighttime-driving)
- [x] [LoveDA](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/en/user_guides/2_dataset_prepare.md#loveda)
- [x] [Potsdam](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/en/user_guides/2_dataset_prepare.md#isprs-potsdam)

- [x] [Vaihingen](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/en/user_guides/2_dataset_prepare.md#isprs-vaihingen)
- [x] [iSAID](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/en/user_guides/2_dataset_prepare.md#isaid)
- [x] [Mapillary Vistas](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/en/user_guides/2_dataset_prepare.md#mapillary-vistas-datasets)
- [x] [LEVIR-CD](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/en/user_guides/2_dataset_prepare.md#levir-cd)
- [x] [BDD100K](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/en/user_guides/2_dataset_prepare.md#bdd100K)

[Here](projects/README.md) are some implementations of SOTA models and solutions built on MMSegmentation, which are supported and maintained by community users. These projects demonstrate the best practices based on MMSegmentation for research and product development. We welcome and appreciate all the contributions to OpenMMLab ecosystem.

We appreciate all contributions to improve MMSegmentation. Please refer to [CONTRIBUTING.md](.github/CONTRIBUTING.md) for the contributing guideline.

MMSegmentation is an open source project that welcome any contribution and feedback.
We wish that the toolbox and benchmark could serve the growing research
community by providing a flexible as well as standardized toolkit to reimplement existing methods
and develop their own new semantic segmentation methods.

```bibtex
}
```

## License

# Preparing SIDD Dataset

```bibtex
}
```

The train datasets can be download from [here](https://drive.google.com/file/d/1UHjWZzLPGweA9ZczmV8lFSRcIxqiOVJw/). The validation datasets can be download from [here](https://drive.google.com/file/d/1Fw6Ey1R-nCHN9WEpxv0MnMqxij-ECQYJ/). The test datasets can be download from [here](https://drive.google.com/file/d/11vfqV-lqousZTuAit1Qkqghiv_taY0KZ/).

For test datasets, we need to export images from mat file. We provide such a script:

```shell
```

```text
mmagic
├── mmagic
├── tools

```

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── SIDD
|   |   |   ├── gt
|   |   ├── val
|   |   ├── test
|   |   |   ├── gt
```

# TurboMind

TurboMind 是一款关于 LLM 推理的高效推理引擎，基于英伟达的 [FasterTransformer](https://github.com/NVIDIA/FasterTransformer) 研发而成。它的主要功能包括：LLaMa 结构模型的支持，persistent batch 推理模式和可扩展的 KV 缓存管理器。

## TurboMind 结构

```
  |        API         |
          |    ^
    请 求  |    | 流式回调
          v    |
  +--------------------+    获取   +-------------------+
  |  Persistent Batch  | <-------> |  KV Cache 管理器 |
  +--------------------+    更新   +-------------------+
             ^
             |
             v
|      LLaMa推理实现      |
```

你也许在别的项目中看到这项机制的另一个名字： `continuous batching` 。在开发这个功能时，我们将对话式 LLM 的推理建模为一个持续运行的 batch ，其生命周期跨越整个服务过程，故将其命名为 `persistent batch` 。简单来说是这样实现的：

- 该功能会预先准备好 N 个 batch slots。
- 当有空闲 slots 时， 请求就会加入到 batch 中。当请求对应的 tokens 都生成完毕后，对应的 batch slot 会立刻被释放，接收新的请求。

- **当一个 sequence 命中缓存时（见下文），它的历史 token 不必在每轮中都进行解码，所以它的 token 生成过程会即刻开始**。
- 整个 batch 会自动扩缩容来避免不必要的计算。

## KV 缓存管理器

TurboMind 的 [KV 缓存管理器](https://github.com/InternLM/lmdeploy/blob/main/src/turbomind/models/llama/LlamaCacheManager.h) 是一个内存池类型的对象，并且在其中加入了 LRU 的实现，这样整个管理器可以被看作是一个 **KV 缓存的缓存**。大致工作方式如下：

- KV 缓存由管理器分配。管理器会根据预先配置好的 slot 数量开辟空间。每个 slot 对应于一个 sequence 所需的 KV 缓存。分配的内存块大小可通过配置来实现预分配或者按需分配（或介于两者之间）。
- 当有新的请求，但是缓存池中没有空闲 slot时，根据 LRU 机制，管理器会踢除最近使用最少的 sequence，把它占据的 slot 分给新的请求。不仅仅如此，
- sequence获取到了slot，类似缓存命中。它在缓存中的历史KV会被直接返回，而不用在进行context decoding 。
- 被踢除的 sequences 不会被完全的删除，而是会被转换成最简洁的形式，例如 token IDs 。当之后获取到相同的 sequence id 时 (即 _cache-miss_ 状态)，这些 token IDs 将被 FMHA 的 context decoder 解码并被转回 KV 缓存。
- 踢除和转换均由 TurboMind 内部自动管理所以对用户来说是透明的。__从用户的使用角度来看，使用了 TurboMind 的系统就像是可以访问无限的设备内存__。

## TurboMind 的 LLaMa 实现

我们对 LLaMa 系列模型的实现是从 FasterTransformer 中的 Gpt-NeX 模型修改而来的。除了对 LLaMa 系列进行基本重构和修改外，我们还做了一些改进以实现会话模型的高性能推理，其中最重要的是：

- 支持多轮对话中的快速文本解码。我们用基于 [cutlass](https://github.com/NVIDIA/cutlass) 的 FMHA 实现替代了 context decoder 中的注意力机制实现，从而支持了 Q/K 长度不匹配的情况。
- 我们在 context FMHA 和 generation FMHA 中都加入了间接缓冲指针，支持 batch 中不连续的 KV 缓存。
- 为了支持 persistent batch 的并发推理，我们设计了新的同步机制来协调在张量并型模式下的工作线程。
- 我们实现了 INT8 KV cache，降低了内存开销，提高了批处理大小和系统吞吐量。这在实际场景中非常有用，因为相比权重和其他激活，KV cache 会消耗更多的内存和内存带宽。
- 我们解决了单个进程内多个模型实例在 TP 模式下运行时 NCCL 卡住的问题。NCCL APIs 现由 host 端的同步 barriers 保护。

## API

TurboMind 的 Python API 支持流式结果返回和张量并行模式。

同时 TurboMind 也继承了 FasterTransformer 能够注册为 [Triton Inference Server](https://github.com/triton-inference-server/server) 推理后端的能力。但是为了支持 persistent batch 中的并发请求，我们不再像 FasterTransformer 那样使用 sequence batching 或者 dynamic batching 。相反，TurboMind 负责记录和管理请求序列的状态。

## TurboMind 和 FasterTransformer 的区别

除了上文中提到的功能外，TurboMind 相较于 FasterTransformer 还有不少差别。譬如不少 FasterTransformer 的功能在 TurboMind 中都被去掉了，这其中包括前缀提示词、 beam search 、上下文 embedding、稀疏化 GEMM 操作和对应 GPT 或 T5 等结构的模型的支持等等。

## FAQ

### 对 Huggingface 模型的支持

因为历史因素， TurboMind 的权重设计是基于 [LLaMa 的官方实现](https://github.com/facebookresearch/llama) 完成的，两者只相差一个转置操作。但是 Huggingface 版本的实现却是[另一种形式](https://github.com/huggingface/transformers/blob/45025d92f815675e483f32812caa28cce3a960e7/src/transformers/models/llama/convert_llama_weights_to_hf.py#L123C76-L123C76)，两种权重实现方式在 `W_q` 和 `W_k` 上的区别我们在 [deploy.py](https://github.com/InternLM/lmdeploy/blob/ff4648a1d09e5aec74cf70efef35bfaeeac552e0/lmdeploy/serve/turbomind/deploy.py#L398) 进行了适配处理，用户可前往查看。

Optimization related configuration is now all managed by `optim_wrapper` which usually has three fields: `optimizer`, `paramwise_cfg`, `clip_grad`, refer to [OptimWrapper](https://mmengine.readthedocs.io/en/latest/tutorials/optim_wrapper.md) for more detail. See the example below, where `Adamw` is used as an `optimizer`, the learning rate of the backbone is reduced by a factor of 10, and gradient clipping is added.

```python
        eps=1e-8,

        },

```

        },

```

### Customize optimizer supported by Pytorch

We already support to use all the optimizers implemented by PyTorch, and the only modification is to change the `optimizer` field in `optim_wrapper` field of config files. For example, if you want to use `ADAM` (note that the performance could drop a lot), the modification could be as the following.

```python
```

A customized optimizer could be defined as following.

You need to create a new directory named `mmdet/engine/optimizers`. And then implement the new optimizer in a file, e.g., in `mmdet/engine/optimizers/my_optimizer.py`:

```python

```

- Modify `mmdet/engine/optimizers/__init__.py` to import it.

  The newly defined module should be imported in `mmdet/engine/optimizers/__init__.py` so that the registry will find the new module and add it:

```python

The newly defined module should be imported in `mmdet/engine/optimizers/__init__.py` so that the registry will find the new module and add it:

```python
```

- Use `custom_imports` in the config to manually import it

```python
```

The module `mmdet.engine.optimizers.my_optimizer` will be imported at the beginning of the program and the class `MyOptimizer` is then automatically registered.
Note that only the package containing the class `MyOptimizer` should be imported.
`mmdet.engine.optimizers.my_optimizer.MyOptimizer` **cannot** be imported directly.

Actually users can use a totally different file directory structure using this importing method, as long as the module root can be located in `PYTHONPATH`.

```python
```

To use your own optimizer, the field can be changed to

```python
```

```python

```python

```

  ```python
  ```

  If your config inherits the base config which already sets the `optim_wrapper`, you might need `_delete_=True` to override the unnecessary settings. See the [config documentation](../user_guides/config.md) for more details.

          begin=0,
          end=8,
      dict(
          T_max=12,
          begin=8,
          end=20,
  ]
  ```

By default we use step learning rate with 1x schedule, this calls [MultiStepLR](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py#L139) in MMEngine.
We support many other learning rate schedule [here](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py), such as `CosineAnnealingLR` and `PolyLR` schedule. Here are some examples

  ```python
      dict(
          begin=0,
          end=8,
  ```

```python
      dict(
          begin=0,
          end=8,
  ```

- ConsineAnnealing schedule:

  ```python
      dict(
          T_max=8,
          begin=0,
          end=8,

  ```

By default, `EpochBasedTrainLoop` is used in `train_cfg` and validation is done after every train epoch, as follows.

```python
```

Actually, both [`IterBasedTrainLoop`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L183%5D) and [`EpochBasedTrainLoop`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L18) support dynamical interval, see the following example.

```python

```python

```

MMEngine provides many useful [hooks](https://mmengine.readthedocs.io/en/latest/tutorials/hooks.html), but there are some occasions when the users might need to implement a new hook. MMDetection supports customized hooks in training in v3.0 . Thus the users could implement a hook directly in mmdet or their mmdet-based codebases and use the hook by only modifying the config in training.

Here we give an example of creating a new hook in mmdet and using it in training.

```python

                          runner,

                         runner,
```

runner,
```

Depending on the functionality of the hook, the users need to specify what the hook will do at each stage of the training in `before_run`, `after_run`, `before_train`, `after_train` , `before_train_epoch`, `after_train_epoch`, `before_train_iter`, and `after_train_iter`.  There are more points where hooks can be inserted, refer to [base hook class](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/hook.py#L9) for more detail.

Then we need to make `MyHook` imported. Assuming the file is in `mmdet/engine/hooks/my_hook.py` there are two ways to do that:

- Modify `mmdet/engine/hooks/__init__.py` to import it.

  The newly defined module should be imported in `mmdet/engine/hooks/__init__.py` so that the registry will find the new module and add it:

```python
```

```python
```

```python
]
```

You can also set the priority of the hook by adding key `priority` to `'NORMAL'` or `'HIGHEST'` as below

```python
]
```

### Use hooks implemented in MMDetection

]
```

If the hook is already implemented in MMDectection, you can directly modify the config to use the hook as below

#### Example: `NumClassCheckHook`

We implement a customized hook named [NumClassCheckHook](../../../mmdet/engine/hooks/num_class_check_hook.py) to check whether the `num_classes` in head matches the length of `classes` in the metainfo of `dataset`.

We set it in [default_runtime.py](../../../configs/_base_/default_runtime.py).

```python
```

- `IterTimerHook`: A hook that logs 'data_time' for loading data and 'time' for a model train step.

- `LoggerHook`: A hook that Collect logs from different components of `Runner` and write them to terminal, JSON file, tensorboard and wandb .etc.
- `ParamSchedulerHook`: A hook to update some hyper-parameters in optimizer, e.g., learning rate and momentum.
- `DetVisualizationHook`: A hook used to visualize validation and testing process prediction results.

`IterTimerHook`, `ParamSchedulerHook` and `DistSamplerSeedHook` are simple and no need to be modified usually, so here we reveals how what we can do with `LoggerHook`, `CheckpointHook` and `DetVisualizationHook`.

Except saving checkpoints periodically, [`CheckpointHook`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L19) provides other options such as `max_keep_ckpts`, `save_optimizer` and etc. The users could set `max_keep_ckpts` to only save small number of checkpoints or decide whether to store state dict of optimizer by `save_optimizer`. More details of the arguments are [here](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L19)

```python
```

The `LoggerHook` enables to set intervals. And the detail usages can be found in the [docstring](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/logger_hook.py#L18).

```python
```

```python
```

`DetVisualizationHook` use `DetLocalVisualizer` to visualize prediction results, and `DetLocalVisualizer` current supports different backends, e.g., `TensorboardVisBackend` and `WandbVisBackend` (see [docstring](https://github.com/open-mmlab/mmengine/blob/main/mmengine/visualization/vis_backend.py) for more detail). The users could add multi backbends to do visualization, as follows.

```python

```

# How to write config

This tutorial describes how to write a config for model conversion and deployment. A deployment config includes `onnx config`, `codebase config`, `backend config`.

- [How to write config](#how-to-write-config)
  - [1. How to write onnx config](#1-how-to-write-onnx-config)
    - [Description of onnx config arguments](#description-of-onnx-config-arguments)
    - [Example](#example)
    - [If you need to use dynamic axes](#if-you-need-to-use-dynamic-axes)
      - [Example](#example-1)
  - [2. How to write codebase config](#2-how-to-write-codebase-config)
    - [Description of codebase config arguments](#description-of-codebase-config-arguments)
      - [Example](#example-2)
  - [3. How to write backend config](#3-how-to-write-backend-config)
    - [Example](#example-3)
  - [4. A complete example of mmpretrain on TensorRT](#4-a-complete-example-of-mmpretrain-on-tensorrt)

- [3. How to write backend config](#3-how-to-write-backend-config)
  - [5. The name rules of our deployment config](#5-the-name-rules-of-our-deployment-config)
    - [Example](#example-4)
  - [6. How to write model config](#6-how-to-write-model-config)

## 1. How to write onnx config

Onnx config to describe how to export a model from pytorch to onnx.

### Description of onnx config arguments

- `type`: Type of config dict. Default is `onnx`.
- `export_params`: If specified, all parameters will be exported. Set this to False if you want to export an untrained model.

- `keep_initializers_as_inputs`: If True, all the initializers (typically corresponding to parameters) in the exported graph will also be added as inputs to the graph. If False, then initializers are not added as inputs to the graph, and only the non-parameter inputs are added as inputs.
- `opset_version`: Opset_version is 11 by default.
- `save_file`: Output onnx file.
- `input_names`: Names to assign to the input nodes of the graph.
- `output_names`: Names to assign to the output nodes of the graph.
- `input_shape`: The height and width of input tensor to the model.

```python

type='onnx',
```

### If you need to use dynamic axes

If the dynamic shape of inputs and outputs is required, you need to add dynamic_axes dict in onnx config.

- `dynamic_axes`: Describe the dimensional information about input and output.

```python
        'input': {
            3: 'width'
        },
        'dets': {
        },
        },
    }
```

## 2. How to write codebase config

Codebase config part contains information like codebase type and task type.

### Description of codebase config arguments

},
    }
```

- `type`: Model's codebase, including `mmpretrain`, `mmdet`, `mmseg`, `mmocr`, `mmagic`.
- `task`: Model's task type, referring to [List of tasks in all codebases](#list-of-tasks-in-all-codebases).

```python
```

## 3. How to write backend config

The backend config is mainly used to specify the backend on which model runs and provide the information needed when the model runs on the backend , referring to [ONNX Runtime](../05-supported-backends/onnxruntime.md), [TensorRT](../05-supported-backends/tensorrt.md), [ncnn](../05-supported-backends/ncnn.md), [PPLNN](../05-supported-backends/pplnn.md).

- `type`: Model's backend, including `onnxruntime`, `ncnn`, `pplnn`, `tensorrt`, `openvino`.

```python

```python
        dict(
    ])
```

## 4. A complete example of mmpretrain on TensorRT

Here we provide a complete deployment config from mmpretrain on TensorRT.

```python

        dict(

max_workspace_size=1 << 30),
        dict(

        'input': {
            3: 'width'
        },
            0: 'batch'
        }
    },
```

## 5. The name rules of our deployment config

There is a specific naming convention for the filename of deployment config files.

```bash
```

- `task name`: Model's task type.

```bash
```

- `backend name`: Backend's name. Note if you use the quantization function, you need to indicate the quantization type. Just like `tensorrt-int8`.
- `dynamic or static`: Dynamic or static export. Note if the backend needs explicit shape information, you need to add a description of input size with `height x width` format. Just like `dynamic-512x1024-2048x2048`, it means that the min input shape is `512x1024` and the max input shape is `2048x2048`.

```bash
```

## 6. How to write model config

```bash
```

According to model's codebase, write the model config file. Model's config file is used to initialize the model, referring to [MMPretrain](https://github.com/open-mmlab/mmpretrain/blob/main/docs/en/user_guides/config.md), [MMDetection](https://github.com/open-mmlab/mmdetection/blob/3.x/docs/en/user_guides/config.md), [MMSegmentation](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/en/user_guides/1_config.md), [MMOCR](https://github.com/open-mmlab/mmocr/blob/main/docs/en/user_guides/config.md), [MMagic](https://github.com/open-mmlab/mmagic/blob/main/docs/en/user_guides/config.md).

# Shufflenet V1

> [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://openaccess.thecvf.com/content_cvpr_2018/html/Zhang_ShuffleNet_An_Extremely_CVPR_2018_paper.html)

We introduce an extremely computation-efficient CNN architecture named ShuffleNet, which is designed specially for mobile devices with very limited computing power (e.g., 10-150 MFLOPs). The new architecture utilizes two new operations, pointwise group convolution and channel shuffle, to greatly reduce computation cost while maintaining accuracy. Experiments on ImageNet classification and MS COCO object detection demonstrate the superior performance of ShuffleNet over other structures, e.g. lower top-1 error (absolute 7.8%) than recent MobileNet on ImageNet classification task, under the computation budget of 40 MFLOPs. On an ARM-based mobile device, ShuffleNet achieves ~13x actual speedup over AlexNet while maintaining comparable accuracy.

<img src="https://user-images.githubusercontent.com/26739999/142575730-dc2f616d-80df-4fb1-93e1-77ebb2b835cf.png" width="70%"/>
</div>

```python

predict = inference_model('shufflenet-v1-1x_16xb64_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('shufflenet-v1-1x_16xb64_in1k', pretrained=True)
```

Train:

```shell
python tools/train.py configs/shufflenet_v1/shufflenet-v1-1x_16xb64_in1k.py
```

Test:

```shell

Train:

```shell
```

Test:

```shell
python tools/test.py configs/shufflenet_v1/shufflenet-v1-1x_16xb64_in1k.py https://download.openmmlab.com/mmclassification/v0/shufflenet_v1/shufflenet_v1_batch1024_imagenet_20200804-5d6cec73.pth
```

| Model                          |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                  Config                   |                                     Download                                     |
| :----------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :---------------------------------------: | :------------------------------------------------------------------------------: |

| `shufflenet-v1-1x_16xb64_in1k` | From scratch |    1.87    |   0.15    |   68.13   |   87.81   | [config](shufflenet-v1-1x_16xb64_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/shufflenet_v1/shufflenet_v1_batch1024_imagenet_20200804-5d6cec73.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/shufflenet_v1/shufflenet_v1_batch1024_imagenet_20200804-5d6cec73.json) |

```bibtex
}
```

# 数据集

在 MMSegmentation 算法库中, 所有 Dataset 类的功能有两个: 加载[预处理](../user_guides/2_dataset_prepare.md) 之后的数据集的信息, 和将数据送入[数据集变换流水线](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/datasets/basesegdataset.py#L141) 中, 进行[数据变换操作](./transforms.md). 加载的数据集信息包括两类: 元信息 (meta information), 数据集本身的信息, 例如数据集总共的类别, 和它们对应调色盘信息: 数据信息 (data information) 是指每组数据中图片和对应标签的路径. 下文中介绍了 MMSegmentation 1.x 中数据集的常用接口, 和 mmseg 数据集基类中数据信息加载与修改数据集类别的逻辑, 以及数据集与数据变换流水线 (pipeline) 的关系.

## 常用接口

以 Cityscapes 为例, 介绍数据集常用接口. 如需运行以下示例, 请在当前工作目录下的 `data` 目录下载并[预处理](../user_guides/2_dataset_prepare.md#cityscapes) Cityscapes 数据集.

实例化 Cityscapes 训练数据集:

```python
from mmseg.datasets import CityscapesDataset

init_default_scope('mmseg')

data_root = 'data/cityscapes/'
data_prefix=dict(img_path='leftImg8bit/train', seg_map_path='gtFine/train')

    dict(type='RandomCrop', crop_size=(512, 1024), cat_max_ratio=0.75),
]

dataset = CityscapesDataset(data_root=data_root, data_prefix=data_prefix, test_mode=False, pipeline=train_pipeline)
```

查看训练数据集长度:

```python
print(len(dataset))

2975
```

获取数据信息, 数据信息的类型是一个字典, 包括 `'img_path'` 字段的存放图片的路径和 `'seg_map_path'` 字段存放分割标注的路径, 以及标签重映射的字段 `'label_map'` 和 `'reduce_zero_label'`(主要功能在下文中介绍), 还有存放已加载标签字段 `'seg_fields'`, 和当前样本的索引字段 `'sample_idx'`.

```python
# 获取数据集中第一组样本的数据信息
print(dataset.get_data_info(0))

{'img_path': 'data/cityscapes/leftImg8bit/train/aachen/aachen_000000_000019_leftImg8bit.png',
 'seg_map_path': 'data/cityscapes/gtFine/train/aachen/aachen_000000_000019_gtFine_labelTrainIds.png',
 'label_map': None,

'seg_map_path': 'data/cityscapes/gtFine/train/aachen/aachen_000000_000019_gtFine_labelTrainIds.png',
 'reduce_zero_label': False,
 'seg_fields': [],
 'sample_idx': 0}
```

获取数据集元信息, MMSegmentation 的数据集元信息的类型同样是一个字典, 包括 `'classes'` 字段存放数据集类别, `'palette'` 存放数据集类别对应的可视化时调色盘的颜色, 以及标签重映射的字段 `'label_map'` 和 `'reduce_zero_label'`.

```python
print(dataset.metainfo)

{'classes': ('road',
  'sidewalk',
  'building',
  'wall',
  'fence',
  'pole',
  'traffic light',
  'traffic sign',
  'vegetation',
  'terrain',
  'sky',
  'person',
  'rider',
  'car',
  'truck',
  'bus',
  'train',
  'motorcycle',
  'bicycle'),
 'palette': [[128, 64, 128],
  [244, 35, 232],
  [70, 70, 70],
  [102, 102, 156],
  [190, 153, 153],
  [153, 153, 153],
  [250, 170, 30],
  [220, 220, 0],
  [107, 142, 35],
  [152, 251, 152],
  [70, 130, 180],
  [220, 20, 60],
  [255, 0, 0],
  [0, 0, 142],
  [0, 0, 70],
  [0, 60, 100],
  [0, 80, 100],
  [0, 0, 230],
  [119, 11, 32]],

[107, 142, 35],
  [0, 0, 70],
 'reduce_zero_label': False}
```

数据集 `__getitem__` 方法的返回值, 是经过数据增强的样本数据的输出, 同样也是一个字典, 包括两个字段, `'inputs'` 字段是当前样本经过数据增强操作的图像, 类型为 torch.Tensor, `'data_samples'` 字段存放的数据类型是 MMSegmentation 1.x 新添加的数据结构 [`Segdatasample`](./structures.md), 其中`gt_sem_seg` 字段是经过数据增强的标签数据.

```python
print(dataset[0])

{'inputs': tensor([[[131, 130, 130,  ...,  23,  23,  23],
          [132, 132, 132,  ...,  23,  22,  23],
          [134, 133, 133,  ...,  23,  23,  23],
          ...,
          [ 66,  67,  67,  ...,  71,  71,  71],
          [ 66,  67,  66,  ...,  68,  68,  68],
          [ 67,  67,  66,  ...,  70,  70,  70]],

         [[143, 143, 142,  ...,  28,  28,  29],
          [145, 145, 145,  ...,  28,  28,  29],
          [145, 145, 145,  ...,  27,  28,  29],
          ...,

[[143, 143, 142,  ...,  28,  28,  29],
          ...,
          [ 75,  75,  76,  ...,  80,  81,  81],
          [ 75,  76,  75,  ...,  80,  80,  80],
          [ 77,  76,  76,  ...,  82,  82,  82]],

         [[126, 125, 126,  ...,  21,  21,  22],
          [127, 127, 128,  ...,  21,  21,  22],
          [127, 127, 126,  ...,  21,  21,  22],
          ...,
          [ 63,  63,  64,  ...,  69,  69,  70],
          [ 64,  65,  64,  ...,  69,  69,  69],
          [ 65,  66,  66,  ...,  72,  71,  71]]], dtype=torch.uint8),
 'data_samples': <SegDataSample(

     META INFORMATION
     img_path: 'data/cityscapes/leftImg8bit/train/aachen/aachen_000000_000019_leftImg8bit.png'
     seg_map_path: 'data/cityscapes/gtFine/train/aachen/aachen_000000_000019_gtFine_labelTrainIds.png'
     img_shape: (512, 1024, 3)
     flip_direction: None
     ori_shape: (1024, 2048)
     flip: False

img_shape: (512, 1024, 3)

     DATA FIELDS
     gt_sem_seg: <PixelData(

             META INFORMATION

             DATA FIELDS
             data: tensor([[[2, 2, 2,  ..., 8, 8, 8],
                          [2, 2, 2,  ..., 8, 8, 8],
                          ...,
                          [0, 0, 0,  ..., 0, 0, 0],
                          [0, 0, 0,  ..., 0, 0, 0]]])
         )>
     _gt_sem_seg: <PixelData(

                          ...,

...,
         )>
 )}
```

## BaseSegDataset

由于 MMSegmentation 中的所有数据集的基本功能均包括(1) 加载[数据集预处理](../user_guides/2_dataset_prepare.md) 之后的数据信息和 (2) 将数据送入数据变换流水线中进行数据变换, 因此在 MMSegmentation 中将其中的共同接口抽象成 [`BaseSegDataset`](https://mmsegmentation.readthedocs.io/zh_CN/latest/api.html?highlight=BaseSegDataset#mmseg.datasets.BaseSegDataset)，它继承自 [MMEngine 的 `BaseDataset`](https://github.com/open-mmlab/mmengine/blob/main/docs/en/advanced_tutorials/basedataset.md), 遵循 OpenMMLab 数据集初始化统一流程, 支持高效的内部数据存储格式, 支持数据集拼接、数据集重复采样等功能.
在 MMSegmentation BaseSegDataset 中重新定义了**数据信息加载方法**（`load_data_list`）和并新增了 `get_label_map` 方法用来**修改数据集的类别信息**.

### 数据信息加载

### 数据信息加载

数据信息加载的内容是样本数据的图片路径和标签路径, 具体实现在 MMSegmentation 的 BaseSegDataset 的 [`load_data_list`](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/datasets/basesegdataset.py#L231) 中.
主要有两种获取图片和标签的路径方法, 如果当数据集目录按以下目录结构组织, [`load_data_list`](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/datasets/basesegdataset.py#L231)) 会根据数据路径和后缀来解析.

```
├── data
│   │   │   ├── val
│   │   │   ├── val
```

例如 ADE20k 数据集结构如下所示:

```
├── ade
│   ├── ADEChallengeData2016

│   │   │   ├── val
```

```
├── ade
│   │   │   │   ├── ADE_train_00000001.png
│   │   │   │── validation
│   │   │   │   ├── ADE_val_00000001.png
│   │   │   │   ├── ADE_train_00000001.jpg
│   │   │   │   ├── ADE_val_00000001.jpg
```

实例化 ADE20k 数据集时，输入图片和标签的路径和后缀:

```python
from mmseg.datasets import ADE20KDataset

ADE20KDataset(data_root = 'data/ade/ADEChallengeData2016',
    data_prefix=dict(img_path='images/training', seg_map_path='annotations/training'),
    img_suffix='.jpg',
    seg_map_suffix='.png',
    reduce_zero_label=True）
```

如果数据集有标注文件, 实例化数据集时会根据输入的数据集标注文件加载数据信息. 例如, PascalContext 数据集实例, 输入标注文件的内容为:

```python
2008_000008
...
```

实例化时需要定义 `ann_file`

```python

seg_map_suffix='.png',
```

```python
...
```

```python
PascalContextDataset(data_root='data/VOCdevkit/VOC2010/',
    data_prefix=dict(img_path='JPEGImages', seg_map_path='SegmentationClassContext'),
    ann_file='ImageSets/SegmentationContext/train.txt')
```

### 数据集类别修改

- 通过输入 metainfo 修改
  `BaseSegDataset` 的子类元信息在数据集实现时定义为类变量，例如 Cityscapes 的 `METAINFO` 变量:

```python
class CityscapesDataset(BaseSegDataset):
    """Cityscapes dataset.

    The ``img_suffix`` is fixed to '_leftImg8bit.png' and ``seg_map_suffix`` is
    fixed to '_gtFine_labelTrainIds.png' for Cityscapes dataset.
    """
    METAINFO = dict(
        classes=('road', 'sidewalk', 'building', 'wall', 'fence', 'pole',
                 'traffic light', 'traffic sign', 'vegetation', 'terrain',
                 'sky', 'person', 'rider', 'car', 'truck', 'bus', 'train',

'traffic light', 'traffic sign', 'vegetation', 'terrain',
                 'motorcycle', 'bicycle'),
        palette=[[128, 64, 128], [244, 35, 232], [70, 70, 70], [102, 102, 156],
                 [190, 153, 153], [153, 153, 153], [250, 170,
                                                    30], [220, 220, 0],
                 [107, 142, 35], [152, 251, 152], [70, 130, 180],
                 [220, 20, 60], [255, 0, 0], [0, 0, 142], [0, 0, 70],
                 [0, 60, 100], [0, 80, 100], [0, 0, 230], [119, 11, 32]])

```

这里的 `'classes'` 中定义了 Cityscapes 数据集标签中的类别名, 如果训练时只关注几个交通工具类别, **忽略其他类别**,
在实例化 Cityscapes 数据集时通过定义 `metainfo` 输入参数的 classes 的字段来修改数据集的元信息:

```python

# metainfo 中只保留以下 classes

```python

metainfo=dict(classes=( 'car', 'truck', 'bus', 'train', 'motorcycle', 'bicycle'))
dataset = CityscapesDataset(data_root=data_root, data_prefix=data_prefix, metainfo=metainfo)

{'classes': ('car', 'truck', 'bus', 'train', 'motorcycle', 'bicycle'),
 'palette': [[0, 0, 142],
  [0, 0, 70],
  [119, 11, 32],
  [128, 64, 128],
  [255, 0, 0]],
 # 类别索引为 255 的像素，在计算损失时会被忽略
 'label_map': {0: 255,
  1: 255,
  2: 255,
  3: 255,
  4: 255,
  5: 255,
  6: 255,
  7: 255,
  8: 255,
  9: 255,
  10: 255,
  11: 255,
  12: 255,
  13: 0,
  14: 1,
  15: 2,

# 类别索引为 255 的像素，在计算损失时会被忽略
  1: 255,
  2: 255,
  3: 255,
  4: 255,
  5: 255,
  6: 255,
  7: 255,
  8: 255,
  9: 255,
  10: 255,
  11: 255,
  12: 255,
  13: 0,
  14: 1,
  15: 2,
  16: 3,
  17: 4,
  18: 5},
```

可以看到, 数据集元信息的类别和默认 Cityscapes 不同. 并且, 定义了标签重映射的字段 `label_map` 用来修改每个分割掩膜上的像素的类别索引, 分割标签类别会根据 `label_map`, 将类别重映射, [具体实现](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/datasets/basesegdataset.py#L151):

```python
gt_semantic_seg_copy = gt_semantic_seg.copy()
for old_id, new_id in results['label_map'].items():
    gt_semantic_seg[gt_semantic_seg_copy == old_id] = new_id
```

- 通过 `reduce_zero_label` 修改
  对于常见的忽略 0 号标签的场景, `BaseSegDataset` 的子类中可以用 `reduce_zero_label` 输入参数来控制。`reduce_zero_label` (默认为 `False`)
  用来控制是否将标签 0 忽略, 当该参数为 `True` 时(最常见的应用是 ADE20k 数据集), 对分割标签中第 0 个类别对应的类别索引改为 255 (MMSegmentation 模型中计算损失时, 默认忽略 255), 其他类别对应的类别索引减一:

```python
gt_semantic_seg[gt_semantic_seg == 0] = 255

用来控制是否将标签 0 忽略, 当该参数为 `True` 时(最常见的应用是 ADE20k 数据集), 对分割标签中第 0 个类别对应的类别索引改为 255 (MMSegmentation 模型中计算损失时, 默认忽略 255), 其他类别对应的类别索引减一:

```python
gt_semantic_seg = gt_semantic_seg - 1
gt_semantic_seg[gt_semantic_seg == 254] = 255
```

## 数据集与数据变换流水线

在常用接口的例子中可以看到, 输入的参数中定义了数据变换流水线参数 `pipeline`, 数据集 `__getitem__` 方法返回经过数据变换的值.
当数据集输入参数没有定义 pipeline, 返回值和 `get_data_info` 方法返回值相同, 例如:

```python

dataset = CityscapesDataset(data_root=data_root, data_prefix=data_prefix, test_mode=False)

```

# ScanNet 数据集

MMDetection3D 支持在 ScanNet 数据集上进行 3D 目标检测\\语义分割 任务。本页提供了有关在 MMDetection3D 中使用 ScanNet 数据集的具体教程。

## 数据集准备

请参考 ScanNet 的[指南](https://github.com/open-mmlab/mmdetection3d/blob/master/data/scannet/README.md)以查看总体流程。

### 提取 ScanNet 点云数据

通过提取 ScanNet 数据，我们加载原始点云文件，并生成包括语义标签、实例标签和真实物体包围框在内的相关标注。

```shell
```

数据处理之前的文件目录结构如下：

```
├── tools
├── data
```

在 `scans` 文件夹下总共有 1201 个训练样本文件夹和 312 个验证样本文件夹，其中存有未处理的点云数据和相关的标注。比如说，在文件夹 `scene0001_01` 下文件是这样组织的：

- `scene0001_01_vh_clean_2.ply`：存有每个顶点坐标和颜色的网格文件。网格的顶点被直接用作未处理的点云数据。
- `scene0001_01.aggregation.json`：包含物体 ID、分割部分 ID、标签的标注文件。
- `scene0001_01_vh_clean_2.0.010000.segs.json`：包含分割部分 ID 和顶点的分割标注文件。
- `scene0001_01.txt`：包括对齐矩阵等的元文件。

- `scene0001_01_vh_clean_2.labels.ply`：包含每个顶点类别的标注文件。

通过运行 `python batch_load_scannet_data.py` 来提取 ScanNet 数据的处理过程主要包含以下几步：

- 从原始文件中提取出点云、实例标签、语义标签和包围框标签文件。
- 下采样原始点云并过滤掉不合法的类别。
- 保存处理后的点云数据和相关的标注文件。

`load_scannet_data.py` 中的核心函数 `export` 如下：

```python
           agg_file,
           seg_file,

    # 标签映射文件：./data/scannet/meta_data/scannetv2-labels.combined.tsv
    # 该标签映射文件中有多种标签标准，比如 'nyu40id'
    # 加载原始点云数据，特征包括6维：XYZRGB

    # 加载场景坐标轴对齐矩阵：一个 4x4 的变换矩阵
    # 将传感器坐标系下的原始点转化到另一个坐标系下

# 加载原始点云数据，特征包括6维：XYZRGB

    # 该坐标系与房屋的两边平行（也就是与坐标轴平行）
    # 测试集的数据没有对齐矩阵
                float(x)
            ]
            break

    # 对网格顶点进行全局的对齐
    # 同种类坐标下的原始点云，每一行的数据是 [x, y, z, 1]
    # 将原始网格顶点转换为对齐后的顶点
                                           axis=1)

    # 加载语义与实例标签

aligned_mesh_vertices = np.concatenate([pts[:, 0:3], mesh_vertices[:, 3:]],
                                           axis=1)

        # 每个物体都有一个语义标签，并且包含几个分割部分
        # 很多点属于同一分割部分
                # 每个点都有一个语义标签
            shape=(num_verts), dtype=np.uint32)  # 0：未标注的
                # object_id 从 1 开始计数，比如 1,2,3,.,,,.NUM_INSTANCES

for object_id, segs in object_id_to_segs.items():
                # 每个点都属于一个物体
        # 包围框格式为 [x, y, z, dx, dy, dz, label_id]
        # [x, y, z] 是包围框的重力中心, [dx, dy, dz] 是与坐标轴平行的
        # [label_id] 是 'nyu40id' 标准下的语义标签
        # 注意：因为三维包围框是与坐标轴平行的，所以旋转角是 0
    ...

object_id_to_label_id, instance_ids)
    ...

```

在从每个场景的扫描文件提取数据后，如果原始点云点数过多，可以将其下采样（比如到 50000 个点），但在三维语义分割任务中，点云不会被下采样。此外，在 `nyu40id` 标准之外的不合法语义标签或者可选的 `DONOT CARE` 类别标签应被过滤。最终，点云文件、语义标签、实例标签和真实物体的集合应被存储于 `.npy` 文件中。

### 提取 ScanNet RGB 色彩数据（可选的）

通过提取 ScanNet RGB 色彩数据，对于每个场景我们加载 RGB 图像与配套 4x4 位姿矩阵、单个 4x4 相机内参矩阵的集合。请注意，这一步是可选的，除非要运行多视图物体检测，否则可以略去这步。

```shell
```

1201 个训练样本，312 个验证样本和 100 个测试样本中的每一个都包含一个单独的 `.sens` 文件。比如说，对于场景 `0001_01` 我们有 `data/scannet/scans/scene0001_01/0001_01.sens`。对于这个场景所有图像和位姿数据都被提取至 `data/scannet/posed_images/scene0001_01`。具体来说，该文件夹下会有 300 个 xxxxx.jpg 格式的图像数据，300 个 xxxxx.txt 格式的相机位姿数据和一个单独的 `intrinsic.txt` 内参文件。通常来说，一个场景包含数千张图像。默认情况下，我们只会提取其中的 300 张，从而只占用少于 100 Gb 的空间。要想提取更多图像，请使用 `--max-images-per-scene` 参数。

### 创建数据集

```shell

### 创建数据集

```shell
```

上述提取的点云文件，语义类别标注文件，和物体实例标注文件被进一步以 `.bin` 格式保存。与此同时 `.pkl` 格式的文件被生成并用于训练和验证。获取数据信息的核心函数 `process_single_scene` 如下：

```python

    # 分别以 .bin 格式保存点云文件，语义类别标注文件和物体实例标注文件
    # 获取 info['pts_path']，info['pts_instance_mask_path'] 和 info['pts_semantic_mask_path']
    ...

    # 获取标注
        # 包围框的形状为 [k, 6 + class]

unaligned_box = unaligned_box_label[:, :-1]
            ])
            # 为了向后兼容，默认的参数名赋予了与坐标轴平行的包围框
            # 我们同时保存了对应的与坐标轴不平行的包围框的信息

annotations['index'] = np.arange(
            ])
```

如上数据处理后，文件目录结构应如下：

```
scannet
├── scans
├── points

```

- `points/xxxxx.bin`：下采样后，未与坐标轴平行（即没有对齐）的点云。因为 ScanNet 3D 检测任务将与坐标轴平行的点云作为输入，而 ScanNet 3D 语义分割任务将对齐前的点云作为输入，我们选择存储对齐前的点云和它们的对齐矩阵。请注意：在 3D 检测的预处理流程 [`GlobalAlignment`](https://github.com/open-mmlab/mmdetection3d/blob/9f0b01caf6aefed861ef4c3eb197c09362d26b32/mmdet3d/datasets/pipelines/transforms_3d.py#L423) 后，点云就都是与坐标轴平行的了。
- `instance_mask/xxxxx.bin`：每个点的实例标签，值的范围为：\[0, NUM_INSTANCES\]，其中 0 表示没有标注。
- `semantic_mask/xxxxx.bin`：每个点的语义标签，值的范围为：\[1, 40\], 也就是 `nyu40id` 的标准。请注意：在训练流程 `PointSegClassMapping` 中，`nyu40id` 的 ID 会被映射到训练 ID。
  - `train_label_weight.npy`：每一语义类别的权重系数。因为 ScanNet 中属于不同类的点的数量相差很大，一个常见的操作是在计算损失时对不同类别进行加权 (label re-weighting) 以得到更好的分割性能。

  - `train_resampled_scene_idxs.npy`：每一个场景 (房间) 的重采样标签。在训练过程中，我们依据每个场景的点的数量，会对其进行不同次数的重采样，以保证训练数据均衡。
- `posed_images/scenexxxx_xx`：`.jpg` 图像的集合，还包含 `.txt` 格式的 4x4 相机姿态和单个 `.txt` 格式的相机内参矩阵文件。
- `scannet_infos_train.pkl`：训练集的数据信息，每个场景的具体信息如下：
  - info\['lidar_points'\]：字典包含与激光雷达点相关的信息。
    - info\['lidar_points'\]\['axis_align_matrix'\]：用于对齐坐标轴的变换矩阵。
  - info\['pts_semantic_mask_path'\]：语义分割标注的文件名。
  - info\['pts_instance_mask_path'\]：实例分割标注的文件名。
  - info\['instances'\]：字典组成的列表，每个字典包含一个实例的所有标注信息。对于其中的第 i 个实例，我们有：
    - info\['instances'\]\[i\]\['bbox_3d'\]：长度为 6 的列表，以 (x, y, z, l, w, h) 的顺序表示深度坐标系下与坐标轴平行的 3D 边界框。
    - info\[instances\]\[i\]\['bbox_label_3d'\]：3D 边界框的标签。

- info\['instances'\]\[i\]\['bbox_3d'\]：长度为 6 的列表，以 (x, y, z, l, w, h) 的顺序表示深度坐标系下与坐标轴平行的 3D 边界框。
- `scannet_infos_val.pkl`：验证集上的数据信息，与 `scannet_infos_train.pkl` 格式完全一致。
- `scannet_infos_test.pkl`：测试集上的数据信息，与 `scannet_infos_train.pkl` 格式几乎完全一致，除了缺少标注。

## 训练流程

ScanNet 进行 **3D 目标检测**的一种典型数据载入流程如下所示：

```python
    dict(
    dict(
    dict(
    dict(

dict(type='PointSample', num_points=40000),
    dict(
    dict(
    dict(
        keys=[
        ])
]
```

- `GlobalAlignment`：输入的点云在施加了坐标轴平行的矩阵后应被转换为与坐标轴平行的形式。
- `PointSegClassMapping`：训练中，只有合法的类别 ID 才会被映射到类别标签，比如 \[0, 18)。
- 数据增强:
  - `PointSample`：下采样输入点云。
  - `RandomFlip3D`：随机左右或前后翻转点云。
  - `GlobalRotScaleTrans`：旋转输入点云，对于 ScanNet 角度通常落入 \[-5, 5\]（度）的范围；并放缩输入点云，对于 ScanNet 比例通常为 1.0（即不做缩放）；最后平移输入点云，对于 ScanNet 通常位移量为 0（即不做位移）。

ScanNet 进行 **3D 语义分割**的一种典型数据载入流程如下所示：

```python
    dict(

```python
    dict(
    dict(
    dict(
    dict(
]
```

- `PointSegClassMapping`：在训练过程中，只有被使用的类别的序号会被映射到类似 \[0, 20) 范围内的类别标签。其余的类别序号会被转换为 `ignore_index` 所制定的忽略标签，在本例中是 `20`。

dict(type='Pack3DDetInputs', keys=['points', 'pts_semantic_mask'])
]
```

- `IndoorPatchPointSample`：从输入点云中裁剪一个含有固定数量点的小块 (patch)。`block_size` 指定了裁剪块的边长，在 ScanNet 上这个数值一般设置为 `1.5`。

## 评估指标

- **目标检测**：通常使用全类平均精度（mAP）来评估 ScanNet 的 3D 检测任务的性能，比如 `mAP@0.25` 和 `mAP@0.5`。具体来说，评估时调用一个通用的计算 3D 物体检测多个类别的精度和召回率的函数。更多细节请参考 [indoor_eval](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/evaluation/functional/indoor_eval.py)。

  **注意**：与在章节`提取 ScanNet 数据`中介绍的那样，所有真实物体的三维包围框是与坐标轴平行的，也就是说旋转角为 0。因此，预测包围框的网络接受的包围框旋转角监督也是 0，且在后处理阶段我们使用适用于与坐标轴平行的包围框的非极大值抑制（NMS），该过程不会考虑包围框的旋转。

**注意**：与在章节`提取 ScanNet 数据`中介绍的那样，所有真实物体的三维包围框是与坐标轴平行的，也就是说旋转角为 0。因此，预测包围框的网络接受的包围框旋转角监督也是 0，且在后处理阶段我们使用适用于与坐标轴平行的包围框的非极大值抑制（NMS），该过程不会考虑包围框的旋转。

- **语义分割**：通常使用平均交并比 (mean Intersection over Union, mIoU) 来评估 ScanNet 的 3D 语义分割任务的性能。具体而言，我们先计算所有类别的 IoU，然后取平均值作为 mIoU。更多实现细节请参考 [seg_eval.py](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/evaluation/functional/seg_eval.py)。

## 在测试集上测试并提交结果

默认情况下，MMDet3D 的代码是在训练集上进行模型训练，然后在验证集上进行模型测试。

如果你也想在在线基准上测试模型的性能（仅支持语义分割），请在测试命令中加上 `--format-only` 的标记，同时也要将 ScanNet 数据集[配置文件](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/_base_/datasets/scannet-seg.py#L126)中的 `ann_file=data_root + 'scannet_infos_val.pkl'` 改成 `ann_file=data_root + 'scannet_infos_test.pkl'`。

请记得通过 `txt_prefix` 来指定想要保存测试结果的文件夹名称。

以 PointNet++ (SSG) 在 ScanNet 上的测试为例，你可以运行以下命令来完成测试结果的保存：

```

```
```

在保存测试结果后，你可以将该文件夹压缩，然后提交到 [ScanNet 在线测试服务器](http://kaldir.vc.in.tum.de/scannet_benchmark/semantic_label_3d)上进行验证。

MMAction2 在[模型库](../modelzoo.md)中提供了预训练的视频理解模型。本文将展示如何使用现有模型对给定的视频进行推理。

关于如何在标准数据集上测试现有模型，请参考这个[指南](./train_test.md#test)。

## 对给定视频进行推理

MMAction2 提供了用于对给定视频进行推理的高级 Python API：

- [init_recognizer](mmaction.apis.init_recognizer): 使用配置文件和权重文件初始化一个识别器
- [inference_recognizer](mmaction.apis.inference_recognizer): 对给定视频进行推理

下面是一个使用 Kinitics-400 预训练权重构建模型并对给定视频进行推理的示例。

```{note}
如果您将 mmaction2 用作第三方包，您需要下载示例中的配置文件和演示视频。

运行 'mim download mmaction2 --config tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb --dest .' 下载所需的配置文件。

运行 'wget https://github.com/open-mmlab/mmaction2/blob/main/demo/demo.mp4' 下载所需的演示视频。
```

```python

```

```python

checkpoint_path = 'https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb/tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb_20220906-2692d16c.pth' # 可以是本地路径
img_path = 'demo/demo.mp4'   # 您可以指定自己的图片路径

# 从配置文件和权重文件中构建模型
model = init_recognizer(config_path, checkpoint_path, device="cpu")  # device 可以是 'cuda:0'
# 对单个视频进行测试
```

`result` 是一个包含 `pred_scores` 的字典。

示例中的动作识别演示可以在[demo/demo.py](https://github.com/open-mmlab/mmaction2/blob/main/demo/demo.py)中找到。

# 支持 SNPE

mmdeploy 集成 snpe 的方式简单且有效： Client/Server 模式。

这种模式

1. 能剥离`模型转换`和`推理`环境：

- 推理无关事项在算力更高的设备上完成；
- 对于推理计算，能拿到 gpu/npu 真实运行结果，而非 cpu 模拟数值。

2. 能覆盖成本敏感的设备。armv7/risc-v/mips 芯片满足产品需求，但往往对 Python 支持有限；

3. 能简化 mmdeploy 安装步骤。如果只想转 snpe 模型测试精度，不需要编译 .whl 包。

## 一、运行推理服务

下载预编译 snpe 推理服务包， `adb push` 到手机、执行。
注意**手机要有 qcom 芯片**。

```bash
...

# 解压运行
$ adb shell
...
130|venus:/data/local/tmp $ ./inference_server  60000
...
```

此时推理服务应打印设备所有 ipv6 和 ipv4 地址，并监听端口。

tips:

- 如果 `adb devices` 找不到设备，可能因为：
  - 有些廉价线只能充电、不能传输数据
  - 或者没有打开手机的“开发者模式”

...
```

tips:

- 如果需要自己编译，可参照 [NDK 交叉编译 snpe 推理服务](../appendix/cross_build_snpe_service.md)
- 如果监听端口时  `segmentation fault`，可能是因为：
  - 端口号已占用，换一个端口

## 二、安装 mmdeploy

1. 环境要求

| 事项    | 版本               | 备注          |
| ------- | ------------------ | ------------- |
| host OS | ubuntu18.04 x86_64 | snpe 指定版本 |
| Python  | **3.6.0**          | snpe 指定版本 |

2. 安装

[官网下载 snpe-1.59](https://developer.qualcomm.com/qfile/69652/snpe-1.59.0.zip)，解压设置环境变量

```bash
```

tips:

```

tips:

- 如果网络不好，[这个 .tar.gz](https://github.com/tpoisonooo/mmdeploy_snpe_testdata/blob/main/snpe-1.59.tar.gz) 仅减小官方包体积，没有修改原始内容。

## 三、测试模型

以 Resnet-18 为例。先参照[文档](https://github.com/open-mmlab/mmpretrain/tree/main)安装 mmpretrain，然后使用 `tools/deploy.py` 转换模型。

```bash

# 模型转换
$ python3 tools/deploy.py  configs/mmpretrain/classification_snpe_static.py $MODEL_CONFIG  $MODEL_PATH   /path/to/test.png   --work-dir resnet18   --device cpu  --uri 192.168.1.1\:60000  --dump-info

# 精度测试

# 精度测试
$ python3 tools/test.py configs/mmpretrain/classification_snpe_static.py   $MODEL_CONFIG    --model reset18/end2end.dlc   --metrics accuracy precision f1_score recall  --uri 192.168.1.1\:60000
```

注意需要 `--uri` 指明 snpe 推理服务的 ip 和端口号，可以使用 ipv4 和 ipv6 地址。

## 四、Android NDK 编译 SDK

如果你还需要用 Android NDK 编译 mmdeploy SDK，请继续阅读本章节。

### 1. 下载 OCV、NDK，设置环境变量

```bash
# 下载 android OCV

# 下载 ndk r23b

```

### 2. 编译 mmdeploy SDK 和 demo

```bash
$ cmake .. \

```

```bash
$ cmake .. \
  -DANDROID_STL=c++_static \

./bin
├── ocr
```

选项说明

| 选项                          | 说明                                  |
| ----------------------------- | ------------------------------------- |
| CMAKE_TOOLCHAIN_FILE          | 加载 NDK 参数，主要用于选择编译器版本 |
| MMDEPLOY_TARGET_BACKENDS=snpe | 使用 snpe 推理                        |

| ANDROID_STL=c++\_static       | 避免 NDK 环境找不到合适的 c++ lib     |
| MMDEPLOY_SHARED_LIBS=ON       | 官方 snpe 没有提供静态库              |

[这里](../01-how-to-build/cmake_option.md)是完整的编译选项说明

## 3. 运行 demo

把 dump 好的模型目录、可执行文件和 lib 都 `adb push` 到设备里

```bash

```

设置环境变量，执行样例

```bash
$ adb shell

..

..
```

}
```

<details>

```bibtex
}
```

| [pose_resnet_50](/configs/body_2d_keypoint/topdown_heatmap/crowdpose/td-hm_res50_8xb64-210e_crowdpose-256x192.py) |  256x192   | 0.637 |      0.808      |      0.692      | 0.738  | 0.650  | 0.506  | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnet/res50_crowdpose_256x192-c6a526b6_20201227.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnet/res50_crowdpose_256x192_20201227.log.json) |
| [pose_resnet_101](/configs/body_2d_keypoint/topdown_heatmap/crowdpose/td-hm_res101_8xb64-210e_crowdpose-256x192.py) |  256x192   | 0.647 |      0.810      |      0.703      | 0.745  | 0.658  | 0.521  | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnet/res101_crowdpose_256x192-8f5870f4_20201227.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnet/res101_crowdpose_256x192_20201227.log.json) |

| [pose_resnet_101](/configs/body_2d_keypoint/topdown_heatmap/crowdpose/td-hm_res101_8xb64-210e_crowdpose-320x256.py) |  320x256   | 0.661 |      0.821      |      0.714      | 0.759  | 0.672  | 0.534  | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnet/res101_crowdpose_320x256-c88c512a_20201227.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnet/res101_crowdpose_320x256_20201227.log.json) |
| [pose_resnet_152](/configs/body_2d_keypoint/topdown_heatmap/crowdpose/td-hm_res152_8xb64-210e_crowdpose-256x192.py) |  256x192   | 0.656 |      0.818      |      0.712      | 0.754  | 0.666  | 0.533  | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnet/res152_crowdpose_256x192-dbd49aba_20201227.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnet/res152_crowdpose_256x192_20201227.log.json) |

> **任务**: 图像去噪，图像去模糊，图像去雨

</div >

## **各个任务下模型的测试结果**

### **图像去雨**

所有数据集均在Y通道上进行测试，测试指标为PSNR和SSIM。

|                方法                | Rain100H<br>PSNR/SSIM (Y) | Rain100L<br>PSNR/SSIM (Y) | Test100<br>PSNR/SSIM (Y) | Test1200<br>PSNR/SSIM (Y) | Test2800<br>PSNR/SSIM (Y) | GPU信息 |                下载                 |
| :--------------------------------: | :-----------------------: | :-----------------------: | :----------------------: | :-----------------------: | :-----------------------: | :-----: | :---------------------------------: |

| [restormer_official_rain13k](./restormer_official_rain13k.py) |      31.4804/0.9056       |      39.1023/0.9787       |      32.0287/0.9239      |      33.2251/0.9272       |      34.2170/0.9451       |    1    | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_rain13k-2be7b550.pth) \| log |

### **图像去模糊**

Gopro和HIDE数据集上使用RGB通道测试，ReakBlur-J 和 ReakBlur-R数据集使用Y通道测试。测试指标为PSNR和SSIM。

|                      方法                      | GoPro<br>PSNR/SSIM (RGB) | HIDE<br>PSNR/SSIM (RGB) | RealBlur-J<br>PSNR/SSIM (Y) | RealBlur-R<br>PSNR/SSIM (Y) | GPU信息 |                      下载                      |
| :--------------------------------------------: | :----------------------: | :---------------------: | :-------------------------: | :-------------------------: | :-----: | :--------------------------------------------: |

| [restormer_official_gopro](./restormer_official_gopro.py) |      32.9295/0.9496      |     31.2289/0.9345      |       28.4356/0.8681        |       35.9141/0.9707        |    1    | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_gopro-db7363a0.pth) \| log |

### **图像去失焦模糊**

所有指标均在RGB通道上进行测试。测试指标为PSNR、SSIM、MAE和LPIPS.

|  方法  | 室内场景图像的PSNR | 室内场景图像的SSIM | 室内场景图像的MAE | 室内场景图像的LPIPS | 室外场景图像的PSNR | 室外场景图像的SSIM | 室外场景图像的MAE | 室外场景图像的LPIPS | 所有图像平均PSNR | 所有图像平均SSIM | 所有图像平均MAE | 所有图像平均LPIPS | GPU 信息 |  下载   |
| :----: | :-------------: | :-------------: | :------------: | :--------------: | :-------------: | :-------------: | :------------: | :--------------: | :------------: | :-------------: | :------------: | :--------------: | :------: | :-----: |

| [restormer_official_dpdd-single](./restormer_official_dpdd-single.py) |     28.8681     |     0.8859      |     0.0251     |        -         |     23.2410     |     0.7509      |     0.0499     |        -         |    25.9805     |     0.8166      |     0.0378     |        -         |    1     | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dpdd-single-6bc31582.pth) \| log |
| [restormer_official_dpdd-dual](./restormer_official_dpdd-dual.py) |     26.6160     |     0.8346      |     0.0354     |        -         |     26.6160     |     0.8346      |     0.0354     |        -         |    26.6160     |     0.8346      |     0.0354     |        -         |    1     | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dpdd-dual-52c94c00.pth) \| log |

### **图像高斯噪声去除**

**灰度图的高斯噪声**

使用PSNR和SSIM指标对数据集上的灰度图进行测试。

|                              方法                               | $\\sigma$ | Set12<br>PSNR/SSIM | BSD68<br>PSNR/SSIM | Urban100<br>PSNR/SSIM | GPU信息 |                               下载                               |
| :-------------------------------------------------------------: | :-------: | :----------------: | :----------------: | :-------------------: | :-----: | :--------------------------------------------------------------: |
| [restormer_official_dfwb-gray-sigma15](./restormer_official_dfwb-gray-sigma15.py) |    15     |   34.0182/0.9160   |   32.4987/0.8940   |    34.4336/0.9419     |    1    | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-sigma15-da74417f.pth) \| log |

| [restormer_official_dfwb-gray-sigma25](./restormer_official_dfwb-gray-sigma25.py) |    25     |   31.7289/0.8811   |   30.1613/0.8370   |    32.1162/0.9140     |    1    | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-sigma25-08010841.pth) \| log |
| [restormer_official_dfwb-gray-sigma50](./restormer_official_dfwb-gray-sigma50.py) |    50     |   28.6269/0.8188   |   27.3266/0.7434   |    28.9636/0.8571     |    1    | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-sigma50-ee852dfe.pth) \| log |
|                                                                 |           |                    |                    |                       |         |                                                                  |

| [restormer_official_dfwb-gray-sigma15](./restormer_official_dfwb-gray-sigma15.py) |    15     |   33.9642/0.9153   |   32.4994/0.8928   |    34.3152/0.9409     |    1    | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-blind-5f094bcc.pth) \| log |
| [restormer_official_dfwb-gray-sigma25](./restormer_official_dfwb-gray-sigma25.py) |    25     |   31.7106/0.8810   |   30.1486/0.8360   |    32.0457/0.9131     |    1    | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-blind-5f094bcc.pth) \| log |
| [restormer_official_dfwb-gray-sigma50](./restormer_official_dfwb-gray-sigma50.py) |    50     |   28.6614/0.8197   |   27.3537/0.7422   |    28.9848/0.8571     |    1    | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-gray-blind-5f094bcc.pth) \| log |

> 上面三行代表每个噪声等级训练一个单独的模型，下面三行代表学习一个单一的模型来处理各种噪音水平。

**彩色图像的高斯噪声**

所有指标均在RGB通道上进行测试，测试指标为PSNR和SSIM。

|                  方法                   | $\\sigma$ | CBSD68<br>PSNR/SSIM (RGB) | Kodak24<br>PSNR/SSIM (RGB) | McMaster<br>PSNR/SSIM (RGB) | Urban100<br>PSNR/SSIM (RGB) | GPU信息 |                   下载                   |
| :-------------------------------------: | :-------: | :-----------------------: | :------------------------: | :-------------------------: | :-------------------------: | :-----: | :--------------------------------------: |
| [restormer_official_dfwb-color-sigma15](./restormer_official_dfwb-color-sigma15.py) |    15     |      34.3506/0.9352       |       35.4900/0.9312       |       35.6072/0.9352        |       35.1522/0.9530        |    1    | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma15-012ceb71.pth) \| log |

| [restormer_official_dfwb-color-sigma25](./restormer_official_dfwb-color-sigma25.py) |    25     |      31.7457/0.8942       |       33.0489/0.8943       |       33.3260/0.9066        |       32.9670/0.9317        |    1    | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma25-e307f222.pth) \| log |
| [restormer_official_dfwb-color-sigma50](./restormer_official_dfwb-color-sigma50.py) |    50     |      28.5569/0.8127       |       30.0122/0.8238       |       30.2608/0.8515        |       30.0230/0.8902        |    1    | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-sigma50-a991983d.pth) \| log |
|                                         |           |                           |                            |                             |                             |         |                                          |

| [restormer_official_dfwb-color-sigma15](./restormer_official_dfwb-color-sigma15.py) |    15     |      34.3422/0.9356       |       35.4544/0.9308       |       35.5473/0.9344        |       35.0754/0.9524        |    1    | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth) \| log |
| [restormer_official_dfwb-color-sigma25](./restormer_official_dfwb-color-sigma25.py) |    25     |      31.7391/0.8945       |       33.0380/0.8941       |       33.3040/0.9063        |       32.9165/0.9312        |    1    | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth) \| log |

| [restormer_official_dfwb-color-sigma50](./restormer_official_dfwb-color-sigma50.py) |    50     |      28.5582/0.8126       |       30.0074/0.8233       |       30.2671/0.8520        |       30.0172/0.8898        |    1    | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_dfwb-color-blind-dfd03c9f.pth) \| log |

### **真实场景图像去噪**

|                          方法                           | SIDD<br>PSNR/SSIM | GPU信息 |                                                  下载                                                   |
| :-----------------------------------------------------: | :---------------: | :-----: | :-----------------------------------------------------------------------------------------------------: |

| [restormer_official_sidd](./restormer_official_sidd.py) |  40.0156/0.9225   |    1    | [model](https://download.openmmlab.com/mmediting/restormer/restormer_official_sidd-9e7025db.pth) \| log |

## 使用方法

**训练**

可以参考 [train_test.md](/docs/zh_cn/user_guides/train_test.md) 中的 **Train a model** 部分。

**测试**

<details>

```shell
# cpu test

# sigma50

```

---
title: ''
---

If you feel we have helped you, give us a STAR! :satisfied:

2. Reimplement a model in the model zoo on other datasets (e.g., custom datasets)
3. Reimplement a custom model but all the components are implemented in MMSegmentation

- For cases 1 & 3, please follow the steps in the following sections thus we could help to quickly identify the issue.
- For cases 2 & 4, please understand that we are not able to do much help here because we usually do not know the full code, and the users should be responsible for the code they write.

- One suggestion for cases 2 & 4 is that the users should first check whether the bug lies in the self-implemented code or the original code. For example, users can first make sure that the same model runs well on supported datasets. If you still need help, please describe what you have done and what you obtain in the issue, and follow the steps in the following sections, and try as clear as possible so that we can better help you.

A clear and concise description of the problem you meet and what you have done.

```
```

```

```
```

```
```

3. Did you make any modifications to the code or config? Did you understand what you have modified?

1. Please run `PYTHONPATH=${PWD}:$PYTHONPATH python mmseg/utils/collect_env.py` to collect the necessary environment information and paste it here.
2. You may add an addition that may be helpful for locating the problem, such as

```
```

```
```

|   Model   | Input Size |  NME   |                                  Details and Download                                  |
| :-------: | :--------: | :----: | :------------------------------------------------------------------------------------: |
| RTMPose-m |  256x256   | 0.0466 | [rtmpose_coco_wholebody_face.md](./coco_wholebody_face/rtmpose_coco_wholebody_face.md) |

|   Model   | Input Size | NME  |           Details and Download            |
| :-------: | :--------: | :--: | :---------------------------------------: |
| RTMPose-m |  256x256   | 4.01 | [rtmpose_wflw.md](./wflw/rtmpose_wflw.md) |

### LaPa Dataset

Results on LaPa dataset

| RTMPose-m |  256x256   | 1.29 | [rtmpose_lapa.md](./lapa/rtmpose_lapa.md) |

# AVA Annotation Explained

In this section, we explain the annotation format of AVA in details:

```
mmaction2
├── data
│   ├── ava
```

## The proposals generated by human detectors

In the annotation folder, `ava_dense_proposals_[train/val/test].FAIR.recall_93.9.pkl` are human proposals generated by a human detector. They are used in training, validation and testing respectively. Take `ava_dense_proposals_train.FAIR.recall_93.9.pkl` as an example. It is a dictionary of size 203626. The key consists of the `videoID` and the `timestamp`. For example, the key `-5KQ66BBWC4,0902` means the values are the detection results for the frame at the 902-nd second in the video `-5KQ66BBWC4`. The values in the dictionary are numpy arrays with shape $$N \\times 5$$ , $$N$$ is the number of detected human bounding boxes in the corresponding frame. The format of bounding box is $$\[x_1, y_1, x_2, y_2, score\], 0 \\le x_1, y_1, x_2, w_2, score \\le 1$$. $$(x_1, y_1)$$ indicates the top-left corner of the bounding box, $$(x_2, y_2)$$ indicates the bottom-right corner of the bounding box; $$(0, 0)$$ indicates the top-left corner of the image, while $$(1, 1)$$ indicates the bottom-right corner of the image.

## The ground-truth labels for spatio-temporal action detection

In the annotation folder, `ava_[train/val]_v[2.1/2.2].csv` are ground-truth labels for spatio-temporal action detection, which are used during training & validation. Take `ava_train_v2.1.csv` as an example, it is a csv file with 837318 lines, each line is the annotation for a human instance in one frame. For example, the first line in `ava_train_v2.1.csv` is `'-5KQ66BBWC4,0902,0.077,0.151,0.283,0.811,80,1'`: the first two items `-5KQ66BBWC4` and `0902` indicate that it corresponds to the 902-nd second in the video `-5KQ66BBWC4`. The next four items ($$\[0.077(x_1), 0.151(y_1), 0.283(x_2), 0.811(y_2)\]$$) indicates the location of the bounding box, the bbox format is the same as human proposals. The next item `80` is the action label. The last item `1` is the ID of this bounding box.

## Excluded timestamps

`ava_[train/val]_excludes_timestamps_v[2.1/2.2].csv` contains excluded timestamps which are not used during training or validation. The format is `video_id, second_idx` .

## Label map

`ava_action_list_v[2.1/2.2]_for_activitynet_[2018/2019].pbtxt` contains the label map of the AVA dataset, which maps the action name to the label index.

# 分支迁移

在早期阶段，MMOCR 有三个分支：`main`、`1.x` 和 `dev-1.x`。随着 MMOCR 1.0.0 正式版的发布，我们也重命名了其中一些分支，下面提供了新旧分支的对照。

- `main` 分支包括了 MMOCR 0.x（例如 v0.6.3）的代码。现在已经被重命名为 `0.x`。
- `1.x` 包含了 MMOCR 1.x（例如 1.0.0rc6）的代码。现在它是 `main` 分支的别名，会在 2023 的年中删除。
- `dev-1.x` 是 MMOCR 1.x 的开发分支。现在保持不变。

有关分支的更多信息，请查看[分支](../notes/branches.md)。

## 升级 `main` 分支时解决冲突

对于希望从旧 `main` 分支（包含 MMOCR 0.x 代码）升级的用户，代码可能会导致冲突。要避免这些冲突，请按照以下步骤操作：

1. 请 commit 在 `main` 上的所有更改（若有），并备份您当前的 `main` 分支。

   ```bash
   git checkout main
   git add --all
   git commit -m 'backup'
   git checkout -b main_backup
   ```

2. 从远程存储库获取最新更改。

   ```bash
   git remote add openmmlab git@github.com:open-mmlab/mmocr.git
   git fetch openmmlab
   ```

3. 通过运行 `git reset --hard openmmlab/main` 将 `main` 分支重置为远程存储库上的最新 `main` 分支。

   ```bash
   git reset --hard openmmlab/main
   ```

按照这些步骤，您可以成功升级您的 `main` 分支。

## ncnn 自定义算子

- [ncnn Ops](#ncnn-ops)
  - [Expand](#expand)
  - [Gather](#gather)
  - [Shape](#shape)
  - [TopK](#topk)

### Expand

Broadcast the input blob following the given shape and the broadcast rule of ncnn.

Expand has no parameters.

<dl>

### Expand

<dl>
<dt><tt>inputs[0]</tt>: ncnn.Mat</dt>
<dd>bottom_blobs[0]; An ncnn.Mat of input data.</dd>
<dt><tt>inputs[1]</tt>: ncnn.Mat</dt>
<dd>bottom_blobs[1]; An 1-dim ncnn.Mat. A valid shape of ncnn.Mat.</dd>
</dl>

<dl>
<dd>top_blob; The blob of ncnn.Mat which expanded by given shape and broadcast rule of ncnn.</dd>
</dl>

- ncnn.Mat: Mat(float32)

### Gather

Given the data and indice blob, gather entries of the axis dimension of data indexed by indices.

| Type  | Parameter | Description                            |
| ----- | --------- | -------------------------------------- |
| `int` | `axis`    | Which axis to gather on. Default is 0. |

<dl>

<dl>
<dd>bottom_blobs[1]; An 1-dim ncnn.Mat of indices on given axis.</dd>
</dl>

<dl>
<dd>top_blob; The blob of ncnn.Mat which gathered by given data and indice blob.</dd>
</dl>

### Shape

Get the shape of the ncnn blobs.

Shape has no parameters.

<dl>
<dd>bottom_blob; An ncnn.Mat of input data.</dd>
</dl>

<dl>
<dd>top_blob; 1-D ncnn.Mat of shape (bottom_blob.dims,), `bottom_blob.dims` is the input blob dimensions.</dd>
</dl>

### TopK

</dl>

### TopK

Get the indices and value(optional) of largest or smallest k data among the axis. This op will map to onnx op `TopK`, `ArgMax`, and `ArgMin`.

| Type  | Parameter   | Description                                                                                                                                                                |
| ----- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `int` | `axis`      | The axis of data which topk calculate on. Default is -1, indicates the last dimension.                                                                                     |

| `int` | `largest`   | The binary value which indicates the TopK operator selects the largest or smallest K values. Default is 1, the TopK selects the largest K values.                          |
| `int` | `sorted`    | The binary value of whether returning sorted topk value or not. If not, the topk returns topk values in any order. Default is 1, this operator returns sorted topk values. |
| `int` | `keep_dims` | The binary value of whether keep the reduced dimension or not. Default is 1, each output blob has the same dimension as input blob.                                        |

<dl>
<dd>bottom_blob[0]; An ncnn.Mat of input data.</dd>
<dt><tt>inputs[1] (optional)</tt>: ncnn.Mat</dt>

<dl>
<dd>bottom_blob[1]; An optional ncnn.Mat. A blob of K in TopK. If this blob not exist, K is 1.</dd>
</dl>

<dl>
<dd>top_blob[0]; If outputs has only 1 blob, outputs[0] is the indice blob of topk, if outputs has 2 blobs, outputs[0] is the value blob of topk. This blob is ncnn.Mat format with the shape of bottom_blob[0] or reduced shape of bottom_blob[0].</dd>
<dd>top_blob[1] (optional); If outputs has 2 blobs, outputs[1] is the value blob of topk. This blob is ncnn.Mat format with the shape of bottom_blob[0] or reduced shape of bottom_blob[0].</dd>
</dl>

# Model Zoo

- We report the inference time as the total time of network forwarding and post-processing, excluding the data loading time. Results are obtained with the script [benchmark.py](https://github.com/open-mmlab/mmdetection/blob/master/tools/analysis_tools/benchmark.py) which computes the average time on 2000 images.

### SECOND

Please refer to [SECOND](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/second) for details. We provide SECOND baselines on KITTI and Waymo datasets.

Please refer to [PointPillars](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/pointpillars) for details. We provide pointpillars baselines on KITTI, nuScenes, Lyft, and Waymo datasets.

Please refer to [Part-A2](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/parta2) for details.

Please refer to [VoteNet](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/votenet) for details. We provide VoteNet baselines on ScanNet and SUNRGBD datasets.

Please refer to [Dynamic Voxelization](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/dynamic_voxelization) for details.

### MVXNet

Please refer to [MVXNet](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/mvxnet) for details.

### MVXNet

Please refer to [RegNet](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/regnet) for details. We provide pointpillars baselines with RegNetX backbones on nuScenes and Lyft datasets currently.

We also support baseline models on [nuImages dataset](https://www.nuscenes.org/nuimages). Please refer to [nuImages](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/nuimages) for details. We report Mask R-CNN, Cascade Mask R-CNN and HTC results currently.

### H3DNet

Please refer to [H3DNet](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/h3dnet) for details.

### 3DSSD

Please refer to [3DSSD](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/3dssd) for details.

Please refer to [CenterPoint](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/centerpoint) for details.

### SSN

Please refer to [SSN](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/ssn) for details. We provide pointpillars with shape-aware grouping heads used in SSN on the nuScenes and Lyft datasets currently.

Please refer to [ImVoteNet](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/imvotenet) for details. We provide ImVoteNet baselines on SUNRGBD dataset.

### FCOS3D

Please refer to [FCOS3D](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/fcos3d) for details. We provide FCOS3D baselines on the nuScenes dataset.

Please refer to [PointNet++](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/pointnet2) for details. We provide PointNet++ baselines on ScanNet and S3DIS datasets.

Please refer to [Group-Free-3D](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/groupfree3d) for details. We provide Group-Free-3D baselines on ScanNet dataset.

Please refer to [ImVoxelNet](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/imvoxelnet) for details. We provide ImVoxelNet baselines on KITTI dataset.

### PAConv

Please refer to [PAConv](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/paconv) for details. We provide PAConv baselines on S3DIS dataset.

### DGCNN

Please refer to [DGCNN](https://github.com/open-mmlab/mmdetection3d/tree/v1.0.0.dev0/configs/dgcnn) for details. We provide DGCNN baselines on S3DIS dataset.

### SMOKE

### DGCNN

### SMOKE

Please refer to [SMOKE](https://github.com/open-mmlab/mmdetection3d/tree/v1.0.0.dev0/configs/smoke) for details. We provide SMOKE baselines on KITTI dataset.

### PGD

Please refer to [PGD](https://github.com/open-mmlab/mmdetection3d/tree/v1.0.0.dev0/configs/pgd) for details. We provide PGD baselines on KITTI and nuScenes dataset.

Please refer to [PointRCNN](https://github.com/open-mmlab/mmdetection3d/tree/v1.0.0.dev0/configs/point_rcnn) for details. We provide PointRCNN baselines on KITTI dataset.

Please refer to [MonoFlex](https://github.com/open-mmlab/mmdetection3d/tree/v1.0.0.dev0/configs/monoflex) for details. We provide MonoFlex baselines on KITTI dataset.

### SA-SSD

### SA-SSD

Please refer to [SA-SSD](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/sassd) for details. We provide SA-SSD baselines on the KITTI dataset.

### FCAF3D

Please refer to [FCAF3D](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/fcaf3d) for details. We provide FCAF3D baselines on the ScanNet, S3DIS, and SUN RGB-D datasets.

Please refer to [PV-RCNN](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/pv_rcnn) for details. We provide PV-RCNN baselines on the KITTI dataset.

Please refer to [BEVFusion](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/projects/BEVFusion) for details. We provide BEVFusion baselines on the NuScenes dataset.

Please refer to [CenterFormer](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/projects/CenterFormer) for details. We provide CenterFormer baselines on the Waymo dataset.

### TR3D

Please refer to [TR3D](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/projects/TR3D) for details. We provide TR3D baselines on the ScanNet, SUN RGB-D and S3DIS dataset.

### DETR3D

Please refer to [DETR3D](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/projects/DETR3D) for details. We provide DETR3D baselines on the nuScenes dataset.

### PETR

Please refer to [PETR](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/projects/PETR) for details. We provide PETR baselines on the nuScenes dataset.

### PETR

Please refer to [TPVFormer](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/projects/TPVFormer) for details. We provide TPVFormer baselines on the nuScenes dataset.

Please refer to [Mixed Precision (FP16) Training on PointPillars](https://github.com/open-mmlab/mmdetection3d/tree/v1.0.0.dev0/configs/pointpillars/hv_pointpillars_fpn_sbn-all_fp16_2x8_2x_nus-3d.py) for details.

    <sup>
      </a>
    </sup>
    <sup>
      </a>
    </sup>
  </div>
</div>

[![docs](https://img.shields.io/badge/docs-latest-blue)](https://mmdeploy.readthedocs.io/zh_CN/latest/)

## MMDeploy 1.x 版本

全新的 MMDeploy 1.x 已发布，该版本适配OpenMMLab 2.0 生态体系，使用时务必**对齐版本**。
MMDeploy 代码库默认分支从`master`切换至`main`。 MMDeploy 0.x (`master`)将逐步废弃，新特性将只添加到 MMDeploy 1.x (`main`)。

| mmdeploy | mmengine |   mmcv   |  mmdet   | mmpretrain and others |
| :------: | :------: | :------: | :------: | :-------------------: |
|  0.x.y   |    -     | \<=1.x.y | \<=2.x.y |         0.x.y         |

|  1.x.y   |  0.x.y   |  2.x.y   |  3.x.y   |         1.x.y         |

[硬件模型库](https://platform.openmmlab.com/deploee/) 使用 MMDeploy 1.x 版本转换了 2300 个 onnx/ncnn/trt/openvino 模型，可免费搜索下载。系统内置真实的服务端/嵌入式硬件，用户可以在线完成模型转和速度测试。

## 介绍

MMDeploy 是 [OpenMMLab](https://openmmlab.com/) 模型部署工具箱，**为各算法库提供统一的部署体验**。基于 MMDeploy，开发者可以轻松从训练 repo 生成指定硬件所需 SDK，省去大量适配时间。

## 架构简析

  <img src="resources/introduction.png"/>
</div>

## 特性简介

### 支持超多 OpenMMLab 算法库

- [mmpretrain](docs/zh_cn/04-supported-codebases/mmpretrain.md)
- [mmdet](docs/zh_cn/04-supported-codebases/mmdet.md)
- [mmseg](docs/zh_cn/04-supported-codebases/mmseg.md)
- [mmagic](docs/zh_cn/04-supported-codebases/mmagic.md)
- [mmocr](docs/zh_cn/04-supported-codebases/mmocr.md)
- [mmpose](docs/zh_cn/04-supported-codebases/mmpose.md)
- [mmdet3d](docs/zh_cn/04-supported-codebases/mmdet3d.md)

- [mmrotate](docs/zh_cn/04-supported-codebases/mmrotate.md)
- [mmaction2](docs/zh_cn/04-supported-codebases/mmaction2.md)

### 支持多种推理后端

支持的设备平台和推理引擎如下表所示。benchmark请参考[这里](docs/zh_cn/03-benchmark/benchmark.md)

<table>
  <tr>
  </tr>
  <tr>
    <td>

<sub><a href="https://github.com/open-mmlab/mmdeploy/actions/workflows/backend-pplnn.yml"><img src="https://img.shields.io/github/actions/workflow/status/open-mmlab/mmdeploy/backend-pplnn.yml"></a></sub> <sub>pplnn</sub> <br>
        <sub><a href="https://github.com/open-mmlab/mmdeploy/actions/workflows/backend-torchscript.yml"><img src="https://img.shields.io/github/actions/workflow/status/open-mmlab/mmdeploy/backend-torchscript.yml"></a></sub> <sub>LibTorch</sub> <br>
    </td>
    <td>

<sub><img src="https://img.shields.io/badge/build-no%20status-lightgrey"></sub> <sub>TVM</sub> <br>
    </td>
    <td>
    </td>
        -
    </td>
        -
    </td>
  </tr>

<tr>
    <td>
    </td>
        -
    </td>
        -
    </td>

</td>
        -
    </td>
        -
    </td>
    </td>
  </tr>

<tr>
    <td>
    </td>
        -
    </td>
        -
    </td>
        -
    </td>
  </tr>

<tr>
    <td>

    </td>
  </tr>

<tr>
        -
    </td>
        -
    </td>
        -
    </td>
    <td>
    </td>
  </tr>
</table>
</div>

### SDK 可高度定制化

- Transform 数据预处理
- Net 推理
- Module 后处理

## [中文文档](https://mmdeploy.readthedocs.io/zh_CN/latest/)

- [快速上手](docs/zh_cn/get_started.md)
- [编译](docs/zh_cn/01-how-to-build/build_from_source.md)
  - [一键式脚本安装](docs/zh_cn/01-how-to-build/build_from_script.md)

  - [Build from Docker](docs/zh_cn/01-how-to-build/build_from_docker.md)
  - [Build for Linux](docs/zh_cn/01-how-to-build/linux-x86_64.md)
  - [Build for macOS](docs/zh_cn/01-how-to-build/macos-arm64.md)
  - [Build for Win10](docs/zh_cn/01-how-to-build/windows.md)
  - [Build for Android](docs/zh_cn/01-how-to-build/android.md)
  - [Build for Jetson](docs/zh_cn/01-how-to-build/jetsons.md)
  - [Build for SNPE](docs/zh_cn/01-how-to-build/snpe.md)
  - [Cross Build for aarch64](docs/zh_cn/01-how-to-build/cross_build_ncnn_aarch64.md)
- 使用
  - [把模型转换到推理 Backend](docs/zh_cn/02-how-to-run/convert_model.md)
  - [配置转换参数](docs/zh_cn/02-how-to-run/write_config.md)
  - [量化](docs/zh_cn/02-how-to-run/quantize_model.md)
  - [测试转换完成的模型](docs/zh_cn/02-how-to-run/profile_model.md)
  - [工具集介绍](docs/zh_cn/02-how-to-run/useful_tools.md)
- 开发指南

- [量化](docs/zh_cn/02-how-to-run/quantize_model.md)
- 开发指南
  - [软件架构](docs/zh_cn/07-developer-guide/architecture.md)
  - [支持新模型](docs/zh_cn/07-developer-guide/support_new_model.md)
  - [增加推理 backend](docs/zh_cn/07-developer-guide/support_new_backend.md)
  - [模型分块](docs/zh_cn/07-developer-guide/partition_model.md)
  - [测试重写模型](docs/zh_cn/07-developer-guide/test_rewritten_models.md)
  - [backend 算子测试](docs/zh_cn/07-developer-guide/add_backend_ops_unittest.md)
  - [回归测试](docs/zh_cn/07-developer-guide/regression_test.md)
- 各 backend 自定义算子列表
  - [ncnn](docs/zh_cn/06-custom-ops/ncnn.md)
  - [onnxruntime](docs/zh_cn/06-custom-ops/onnxruntime.md)
  - [tensorrt](docs/zh_cn/06-custom-ops/tensorrt.md)
- [FAQ](docs/zh_cn/faq.md)
- [贡献者手册](.github/CONTRIBUTING.md)

## 新人解说

- [01 术语解释、加载第一个模型](docs/zh_cn/tutorial/01_introduction_to_model_deployment.md)

## 新人解说

- [02 部署常见问题](docs/zh_cn/tutorial/02_challenges.md)
- [03 torch转onnx](docs/zh_cn/tutorial/03_pytorch2onnx.md)
- [04 让torch支持更多onnx算子](docs/zh_cn/tutorial/04_onnx_custom_op.md)
- [05 调试onnx模型](docs/zh_cn/tutorial/05_onnx_model_editing.md)

## 基准与模型库

基准和支持的模型列表可以在[基准](https://mmdeploy.readthedocs.io/zh_CN/latest/03-benchmark/benchmark.html)和[模型列表](https://mmdeploy.readthedocs.io/en/latest/03-benchmark/supported_models.html)中获得。

## 贡献指南

我们感谢所有的贡献者为改进和提升 MMDeploy 所作出的努力。请参考[贡献指南](.github/CONTRIBUTING.md)来了解参与项目贡献的相关指引。

## 致谢

- [OpenPPL](https://github.com/openppl-public/ppl.nn): 高性能推理框架底层库
- [OpenVINO](https://github.com/openvinotoolkit/openvino): AI 推理优化和部署框架
- [ncnn](https://github.com/tencent/ncnn): 为手机端极致优化的高性能神经网络前向计算框架

## 引用

如果您在研究中使用了本项目的代码或者性能基准，请参考如下 bibtex 引用 MMDeploy:

```BibTeX

## 引用

```BibTeX
}
```

## 开源许可证

- [MMPretrain](https://github.com/open-mmlab/mmpretrain): OpenMMLab 深度学习预训练工具箱
- [MMYOLO](https://github.com/open-mmlab/mmyolo): OpenMMLab YOLO 系列工具箱和基准测试

扫描下方的二维码可关注 OpenMMLab 团队的 [知乎官方账号](https://www.zhihu.com/people/openmmlab)，扫描下方微信二维码添加喵喵好友，进入 MMDeploy 微信交流社群。【加好友申请格式：研究方向+地区+学校/公司+姓名】

  <img src="https://user-images.githubusercontent.com/25839884/205870927-39f4946d-8751-4219-a4c0-740117558fd7.jpg" height="400" />
  <img src="https://github.com/open-mmlab/mmdeploy/assets/62195058/a8f116ca-b567-42ce-b70e-38526a81c9a3" height="400" />
</div>

</div>

干货满满 📘，等您来撩 💗，OpenMMLab 社区期待您的加入 👬

# Res2Net

> [Res2Net: A New Multi-scale Backbone Architecture](https://arxiv.org/abs/1904.01169)

Representing features at multiple scales is of great importance for numerous vision tasks. Recent advances in backbone convolutional neural networks (CNNs) continually demonstrate stronger multi-scale representation ability, leading to consistent performance gains on a wide range of applications. However, most existing methods represent the multi-scale features in a layer-wise manner. In this paper, we propose a novel building block for CNNs, namely Res2Net, by constructing hierarchical residual-like connections within one single residual block. The Res2Net represents multi-scale features at a granular level and increases the range of receptive fields for each network layer. The proposed Res2Net block can be plugged into the state-of-the-art backbone CNN models, e.g., ResNet, ResNeXt, and DLA. We evaluate the Res2Net block on all these models and demonstrate consistent performance gains over baseline models on widely-used datasets, e.g., CIFAR-100 and ImageNet. Further ablation studies and experimental results on representative computer vision tasks, i.e., object detection, class activation mapping, and salient object detection, further verify the superiority of the Res2Net over the state-of-the-art baseline methods.

<img src="https://user-images.githubusercontent.com/26739999/142573547-cde68abf-287b-46db-a848-5cffe3068faf.png" width="50%"/>
</div>

```python

predict = inference_model('res2net50-w14-s8_3rdparty_8xb32_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('res2net50-w14-s8_3rdparty_8xb32_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/res2net/res2net50-w14-s8_8xb32_in1k.py https://download.openmmlab.com/mmclassification/v0/res2net/res2net50-w14-s8_3rdparty_8xb32_in1k_20210927-bc967bf1.pth
```

| `res2net50-w14-s8_3rdparty_8xb32_in1k`\*  | From scratch |   25.06    |   4.22    |   78.14   |   93.85   | [config](res2net50-w14-s8_8xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/res2net/res2net50-w14-s8_3rdparty_8xb32_in1k_20210927-bc967bf1.pth) |
| `res2net50-w26-s8_3rdparty_8xb32_in1k`\*  | From scratch |   48.40    |   8.39    |   79.20   |   94.36   | [config](res2net50-w26-s8_8xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/res2net/res2net50-w26-s8_3rdparty_8xb32_in1k_20210927-f547a94b.pth) |
| `res2net101-w26-s4_3rdparty_8xb32_in1k`\* | From scratch |   45.21    |   8.12    |   79.19   |   94.44   | [config](res2net101-w26-s4_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/res2net/res2net101-w26-s4_3rdparty_8xb32_in1k_20210927-870b6c36.pth) |

*Models with * are converted from the [official repo](https://github.com/Res2Net/Res2Net-PretrainedModels/blob/master/res2net.py#L181). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@article{gao2019res2net,
  title={Res2Net: A New Multi-scale Backbone Architecture},
  author={Gao, Shang-Hua and Cheng, Ming-Ming and Zhao, Kai and Zhang, Xin-Yu and Yang, Ming-Hsuan and Torr, Philip},
  journal={IEEE TPAMI},
  doi={10.1109/TPAMI.2019.2938758},
}
```

<details>

```bibtex
}
```

| [rtmpose-m](/configs/animal_2d_keypoint/rtmpose/ap10k/rtmpose-m_8xb64-210e_ap10k-256x256.py) |  256x256   | 0.722 |      0.939      |      0.788      |     0.569      |     0.728      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-ap10k_pt-aic-coco_210e-256x256-7a041aa1_20230206.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-ap10k_pt-aic-coco_210e-256x256-7a041aa1_20230206.json) |

# Frequently Asked Questions (To be updated)

We list some common troubles faced by many users and their corresponding solutions here. Feel free to enrich the list if you find any frequent issues and have ways to help others to solve them. If the contents here do not cover your issue, please create an issue using the [provided templates](https://github.com/open-mmlab/mmdetection/blob/master/.github/ISSUE_TEMPLATE/error-report.md/) and make sure you fill in all required information in the template.

## MMCV Installation

- Compatibility issue between MMCV and MMDetection; "ConvWS is already registered in conv layer"; "AssertionError: MMCV==xxx is used but incompatible. Please install mmcv>=xxx, \<=xxx."

Compatible MMCV, MMDetection and MMRotate versions are shown as below. Please install the correct version of them to avoid installation issues.

| MMRotate | MMEngine                    | MMCV                       | MMDetection                 |

| -------- | --------------------------- | -------------------------- | --------------------------- |
| dev-1.x  | 0.1.0 \<= mmengine \< 1.0.0 | 2.0.0rc2 \<= mmcv \< 2.1.0 | 3.0.0rc5 \<= mmdet \< 3.1.0 |
| 1.0.0rc1 | 0.1.0 \<= mmengine \< 1.0.0 | 2.0.0rc2 \<= mmcv \< 2.1.0 | 3.0.0rc5 \<= mmdet \< 3.1.0 |
| 1.0.0rc0 | 0.1.0 \<= mmengine \< 1.0.0 | 2.0.0rc2 \<= mmcv \< 2.1.0 | 3.0.0rc2 \<= mmdet \< 3.1.0 |

**Note:**

1. If you want to install mmrotate-v0.x, the compatible MMRotate and MMCV versions table can be found at [here](https://mmrotate.readthedocs.io/en/stable/faq.html#installation). Please choose the correct version of MMCV to avoid installation issues.

You may need to set `TORCH_CUDA_ARCH_LIST` to reinstall MMCV.
     i.e. run `TORCH_CUDA_ARCH_LIST=7.0 pip install mmcv-full` to build MMCV for Volta GPUs.

i.e. run `python mmdet/utils/collect_env.py` to see if `"MMCV Compiler"`/`"MMCV CUDA Compiler"` is the same as `"GCC"`/`"CUDA_HOME"`.

- "setuptools.sandbox.UnpickleableException: DistutilsSetupError("each element of 'ext_modules' option must be an Extension instance or 2-tuple")"

2. You may also need to check the compatibility between the `setuptools`, `Cython`, and `PyTorch` in your environment.

  2. Check whether PyTorch is correctly installed and could use CUDA op, e.g. type the following command in your terminal.

     ```shell
     ```

     ```shell
     ```

3. If Pytorch is correctly installed, check whether MMCV is correctly installed.

     ```shell
     ```

## E2CNN

- "ImportError: cannot import name 'container_bacs' from 'torch.\_six'"

  1. This is because `container_abcs` has been removed since PyTorch 1.9.

  2. Replace

     ```shell
     from torch.six import container_abcs
     ```

     in `python3.7/site-packages/e2cnn/nn/modules/module_list.py` with

     ```shell
     TORCH_MAJOR = int(torch.__version__.split('.')[0])
     TORCH_MINOR = int(torch.__version__.split('.')[1])
     if TORCH_MAJOR ==1 and TORCH_MINOR < 8:
         from torch.six import container_abcs
     else:

TORCH_MINOR = int(torch.__version__.split('.')[1])
     else:
         import collections.abs as container_abcs
     ```

  3. Or downgrade the version of Pytorch.

2. Reduce the learning rate: the learning rate might be too large due to some reasons, e.g., change of batch size. You can rescale them to the value that could stably train the model.
  4. Add gradient clipping: some models requires gradient clipping to stabilize the training process. The default of `grad_clip` is `None`, you can add gradient clippint to avoid gradients that are too large, i.e., set `optimizer_config=dict(_delete_=True, grad_clip=dict(max_norm=35, norm_type=2))` in your config file. If your config does not inherits from any basic config that contains `optimizer_config=dict(grad_clip=None)`, you can simply add `optimizer_config=dict(grad_clip=dict(max_norm=35, norm_type=2))`.

  1. There are some scenarios when there are large amounts of ground truth boxes, which may cause OOM during target assignment. You can set `gpu_assign_thr=N` in the config of assigner thus the assigner will calculate box overlaps through CPU when there are more than N GT boxes.
  3. Try mixed precision training by setting `fp16 = dict(loss_scale='dynamic')` in the config file.

  2. You can set `find_unused_parameters = True` in the config to solve the above problems or find those unused parameters manually.

2. You can set `find_unused_parameters = True` in the config to solve the above problems or find those unused parameters manually.

<details>

```bibtex
}
```

<details>

```bibtex

```bibtex
}
```

| [pose_scnet_50](/configs/face_2d_keypoint/topdown_heatmap/coco_wholebody_face/td-hm_scnet50_8xb32-60e_coco-wholebody-face-256x256.py) |  256x256   | 0.0567 | [ckpt](https://download.openmmlab.com/mmpose/face/scnet/scnet50_coco_wholebody_face_256x256-a0183f5f_20210909.pth) | [log](https://download.openmmlab.com/mmpose/face/scnet/scnet50_coco_wholebody_face_256x256_20210909.log.json) |

```
```

2. Checkout a new branch from `dev-1.x` branch, depending on the version of the codebase you want to contribute to.

6. Merge your branch to `dev-1.x` branch

  ```bash
  ```

  ```bash
  ```

  ```bash
  ```

```bash
  ```

### 2. Checkout a new branch from `dev-1.x` branch

```bash
```

```{tip}
```

  ```bash
  ```

  ```bash
  # coding
  ```

  ```{note}
  ```

  ```bash

```

  ```bash
  ```

- Specifically, if you are contributing to `dev-1.x`, you will have to change the base branch of the PR to `dev-1.x` in the PR page, since the default base branch is `main`.

![avatar](https://user-images.githubusercontent.com/22607038/195045928-f3ceedc8-0162-46a7-ae1a-7e22829fe189.png)

### 6.  Merge your branch to `dev-1.x` branch and delete the branch

  ```bash
  ```

## PR Specs

- Bad: Support Faster R-CNN

We use python files as our config system. You can find all the provided configs under $MMDetection/configs.

We incorporate modular and inheritance design into our config system,
If you wish to inspect the config file,
you may run `python tools/misc/print_config.py /PATH/TO/CONFIG` to see the complete config.

## A brief description of a complete config

A complete config usually contains the following primary fields:

- `model`: the basic config of model, which may contain `data_preprocessor`, modules (e.g., `detector`, `motion`),`train_cfg`, `test_cfg`, etc.
- `train_dataloader`: the config of training dataloader, which usually contains `batch_size`, `num_workers`, `sampler`, `dataset`, etc.
- `val_dataloader`: the config of validation dataloader, which is similar with `train_dataloader`.
- `test_dataloader`: the config of testing dataloader, which is similar with `train_dataloader`.

- `val_evaluator`: the config of validation evaluator. For example,`type='MOTChallengeMetrics'` for MOT task on the MOTChallenge benchmarks.
- `test_evaluator`: the config of testing evaluator, which is similar with `val_evaluator`.
- `train_cfg`: the config of training loop. For example, `type='EpochBasedTrainLoop'`.
- `val_cfg`: the config of validation loop. For example, `type='VideoValLoop'`.
- `test_cfg`: the config of testing loop. For example, `type='VideoTestLoop'`.
- `default_hooks`: the config of default hooks, which may include hooks for timer, logger, param_scheduler, checkpoint, sampler_seed, visualization, etc.
- `vis_backends`: the config of visualization backends, which uses `type='LocalVisBackend'` as default.

- `visualizer`: the config of visualizer.  `type='TrackLocalVisualizer'` for MOT tasks.
- `param_scheduler`: the config of parameter scheduler, which usually sets the learning rate scheduler.
- `optim_wrapper`: the config of optimizer wrapper, which contains optimization-related information, for example optimizer, gradient clipping, etc.
- `load_from`: load models as a pre-trained model from a given path.
- `resume`: If `True`, resume checkpoints from `load_from`, and the training will be resumed from the epoch when the checkpoint is saved.

When submitting jobs using `tools/train.py` or `tools/test_tracking.py`,
you may specify `--cfg-options` to in-place modify the config.
We present several examples as follows.
For more details, please refer to [MMEngine](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/config.md).

- **Update config keys of dict chains.**

  For example, `--cfg-options model.detector.backbone.norm_eval=False` changes the all BN modules in model backbones to train mode.

- **Update keys inside a list of configs.**

  For example, the testing pipeline `test_dataloader.dataset.pipeline` is normally a list e.g. `[dict(type='LoadImageFromFile'), ...]`.
  If you want to change `LoadImageFromFile` to `LoadImageFromWebcam` in the pipeline,
  you may specify `--cfg-options test_dataloader.dataset.pipeline.0.type=LoadImageFromWebcam`.

- **Update values of list/tuples.**

  Maybe the value to be updated is a list or a tuple.

you may specify `--cfg-options test_dataloader.dataset.pipeline.0.type=LoadImageFromWebcam`.

  For example, you can change the key `mean` of `data_preprocessor` by specifying `--cfg-options model.data_preprocessor.mean=[0,0,0]`.
  Note that **NO** white space is allowed inside the specified value.

There are 3 basic component types under `config/_base_`, i.e., dataset, model and default_runtime.
Many methods could be easily constructed with one of each like SORT, DeepSORT.
The configs that are composed by components from `_base_` are called *primitive*.

For all configs under the same folder, it is recommended to have only **one** *primitive* config.
All other configs should inherit from the *primitive* config.
In this way, the maximum of inheritance level is 3.

For example, if some modification is made base on Faster R-CNN,
user may first inherit the basic Faster R-CNN structure
by specifying `_base_ = ../_base_/models/faster-rcnn_r50-dc5.py`,

you may create a folder `method_name` under `configs`.

Please refer to [MMEngine](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/config.md) for detailed documentation.

```shell
{method}_{module}_{train_cfg}_{train_data}_{test_data}
```

- `{method}`: method name, like `sort`.

```shell
```

- `{module}`: basic modules of the method, like `faster-rcnn_r50_fpn`.
- `{train_cfg}`: training config which usually contains batch size, epochs, etc, like `8xb4-80e`.
- `{train_data}`: training data, like `mot17halftrain`.
- `{test_data}`: testing data, like `test-mot17halfval`.

## FAQ

**Ignore some fields in the base configs**

You may refer to [MMEngine](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/config.md) for simple illustration.

## Tracking Data Structure Introduction

### Advantages and new features

In mmdetection tracking task, we employ videos to organize the dataset and use
TrackDataSample to descirbe dataset info.

- Based on video organization, we provide transform `UniformRefFrameSample` to sample key frames and ref frames and use `TransformBroadcaster` for for clip training.
- TrackDataSample can be viewd as a wrapper of multiple DetDataSample to some extent. It contains a property `video_data_samples` which is a list of DetDataSample, each of which corresponds to a single frame. In addition, it's metainfo includes key_frames_inds and ref_frames_inds to apply clip training way.
- Thanks to video-based data organization, the entire video can be directly tested. This way is more concise and intuitive. We also provide image_based test method, if your GPU mmemory cannot fit the entire video.

### TODO

### TODO

- Some algorithms like StrongSORT, Mask2Former can not support video_based testing. These algorithms pose a challenge to GPU memory. we will optimize this problem in the future.
- Now we do not support joint training of video_based dataset like MOT Challenge Dataset and image_based dataset like Crowdhuman for the algorithm QDTrack. we will optimize this problem in the future.

# DaViT

> [DaViT: Dual Attention Vision Transformers](https://arxiv.org/abs/2204.03645v1)

In this work, we introduce Dual Attention Vision Transformers (DaViT), a simple yet effective vision transformer architecture that is able to capture global context while maintaining computational efficiency. We propose approaching the problem from an orthogonal angle: exploiting self-attention mechanisms with both "spatial tokens" and "channel tokens". With spatial tokens, the spatial dimension defines the token scope, and the channel dimension defines the token feature dimension. With channel tokens, we have the inverse: the channel dimension defines the token scope, and the spatial dimension defines the token feature dimension. We further group tokens along the sequence direction for both spatial and channel tokens to maintain the linear complexity of the entire model. We show that these two self-attentions complement each other: (i) since each channel token contains an abstract representation of the entire image, the channel attention naturally captures global interactions and representations by taking all spatial positions into account when computing attention scores between channels; (ii) the spatial attention refines the local representations by performing fine-grained interactions across spatial locations, which in turn helps the global information modeling in channel attention. Extensive experiments show our DaViT achieves state-of-the-art performance on four different tasks with efficient computations. Without extra data, DaViT-Tiny, DaViT-Small, and DaViT-Base achieve 82.8%, 84.2%, and 84.6% top-1 accuracy on ImageNet-1K with 28.3M, 49.7M, and 87.9M parameters, respectively. When we further scale up DaViT with 1.5B weakly supervised image and text pairs, DaViT-Gaint reaches 90.4% top-1 accuracy on ImageNet-1K.

<img src="https://user-images.githubusercontent.com/24734142/196125065-e232409b-f710-4729-b657-4e5f9158f2d1.png" width="90%"/>
</div>

```python

predict = inference_model('davit-tiny_3rdparty_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('davit-tiny_3rdparty_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/davit/davit-tiny_4xb256_in1k.py https://download.openmmlab.com/mmclassification/v0/davit/davit-tiny_3rdparty_in1k_20221116-700fdf7d.pth
```

| `davit-tiny_3rdparty_in1k`\*  | From scratch |   28.36    |   4.54    |   82.24   |   96.13   | [config](davit-tiny_4xb256_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/davit/davit-tiny_3rdparty_in1k_20221116-700fdf7d.pth) |
| `davit-small_3rdparty_in1k`\* | From scratch |   49.75    |   8.80    |   83.61   |   96.75   | [config](davit-small_4xb256_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/davit/davit-small_3rdparty_in1k_20221116-51a849a6.pth) |
| `davit-base_3rdparty_in1k`\*  | From scratch |   87.95    |   15.51   |   84.09   |   96.82   | [config](davit-base_4xb256_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/davit/davit-base_3rdparty_in1k_20221116-19e0d956.pth) |

*Models with * are converted from the [official repo](https://github.com/dingmyu/davit/blob/main/mmdet/mmdet/models/backbones/davit.py#L355). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@inproceedings{ding2022davit,
    title={DaViT: Dual Attention Vision Transformer},
    author={Ding, Mingyu and Xiao, Bin and Codella, Noel and Luo, Ping and Wang, Jingdong and Yuan, Lu},
    booktitle={ECCV},
}
```

# How to use prebuilt package on Windows10

- [How to use prebuilt package on Windows10](#how-to-use-prebuilt-package-on-windows10)
  - [Prerequisite](#prerequisite)
  - [Model Convert](#model-convert)
  - [Troubleshooting](#troubleshooting)

- [ONNXRuntime](#onnxruntime-2)

This tutorial takes `mmdeploy-1.2.0-windows-amd64.zip` and `mmdeploy-1.2.0-windows-amd64-cuda11.3.zip` as examples to show how to use the prebuilt packages. The former support onnxruntime cpu inference, the latter support onnxruntime-gpu and tensorrt inference.

The directory structure of the prebuilt package is as follows, where the `dist` folder is about model converter, and the `sdk` folder is related to model inference.

```
.
├── lib
```

In order to use the prebuilt package, you need to install some third-party dependent libraries.

├── lib
```

1. Follow the [get_started](../get_started.md) documentation to create a virtual python environment and install pytorch, torchvision and mmcv. To use the C interface of the SDK, you need to install [vs2019+](https://visualstudio.microsoft.com/), [OpenCV](https://github.com/opencv/opencv/releases).

   :point_right: It is recommended to use `pip` instead of `conda` to install pytorch and torchvision

2. Clone the mmdeploy repository

   ```bash
   ```

   :point_right: The main purpose here is to use the configs, so there is no need to compile `mmdeploy`.

3. Install mmpretrain

   ```bash
   ```

   ```bash
   ```

4. Prepare a PyTorch model as our example

   Download the pth [resnet18_8xb32_in1k_20210831-fbbb1da6.pth](https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_8xb32_in1k_20210831-fbbb1da6.pth). The corresponding config of the model is [resnet18_8xb32_in1k.py](https://github.com/open-mmlab/mmpretrain/blob/main/configs/resnet/resnet18_8xb32_in1k.py)

After the above work is done, the structure of the current working directory should be:

```
.
```

In order to use `ONNX Runtime` backend, you should also do the following steps.

5. Install `mmdeploy` (Model Converter) and `mmdeploy_runtime` (SDK Python API).

   ```bash
   ```

   ```bash
   ```

   :point_right: If you have installed it before, please uninstall it first.

6. Install onnxruntime package

   ```
   ```

7. Download [`onnxruntime`](https://github.com/microsoft/onnxruntime/releases/tag/v1.8.1), and add environment variable.

   As shown in the figure, add the lib directory of onnxruntime to the `PATH`.

   :exclamation: Restart powershell to make the environment variables setting take effect. You can check whether the settings are in effect by `echo $env:PATH`.

8. Download SDK C/cpp Library mmdeploy-1.2.0-windows-amd64.zip

In order to use `TensorRT` backend, you should also do the following steps.

   ```bash
   ```

6. Install TensorRT related package and set environment variables

   - TensorRT 8.2.3.0

   Add the runtime libraries of TensorRT and cuDNN to the `PATH`. You can refer to the path setting of onnxruntime. Don't forget to install python package of TensorRT.

   :exclamation: Restart powershell to make the environment variables setting take effect. You can check whether the settings are in effect by echo `$env:PATH`.

:exclamation: Restart powershell to make the environment variables setting take effect. You can check whether the settings are in effect by echo `$env:PATH`.

   :exclamation: It is recommended to add only one version of the TensorRT/cuDNN runtime libraries to the `PATH`. It is better not to copy the runtime libraries of TensorRT/cuDNN to the cuda directory in `C:\`.

7. Install pycuda by `pip install pycuda`

8. Download SDK C/cpp Library mmdeploy-1.2.0-windows-amd64-cuda11.3.zip

## Model Convert

The following describes how to use the prebuilt package to do model conversion based on the previous downloaded pth.

After preparation work, the structure of the current working directory should be：

```
..
```

Model conversion can be performed like below:

```python

```

```python

```

The structure of the converted model directory:

```bash
`-- onnx
    `-- resnet
```

```bash
`-- onnx
    `-- resnet
```

After installation of mmdeploy-tensorrt prebuilt package, the structure of the current working directory should be：

```
..
```

```python
import os

import os

```

```
`-- trt
    `-- resnet

```

```
`-- trt
    `-- resnet
```

You can obtain two model folders after model conversion.

```
```

The structure of current working directory：

```
.
```

:exclamation: It should be emphasized that `inference_model` is not for deployment, but shields the difference of backend inference api(`TensorRT`, `ONNX Runtime` etc.). The main purpose of this api is to check whether the converted model can be inferred normally.

```python

```python

```

```python

```

The following describes how to use the SDK's Python API for inference

```bash

```

```bash
```

```bash
```

### C SDK

The following describes how to use the SDK's C API for inference

1. Add environment variables

   Refer to the README.md in sdk folder

2. Build examples

3. Inference：

   It is recommended to use `CMD` here.

   Under `mmdeploy-1.2.0-windows-amd64\\example\\cpp\\build\\Release` directory：

   ```
   ```

```
   ```

3. Inference

   Under `mmdeploy-1.2.0-windows-amd64-cuda11.3\\example\\cpp\\build\\Release` directory

   ```
   ```

If you encounter problems, please refer to [FAQ](../faq.md)

# RoI Trans

> [Learning RoI Transformer for Oriented Object Detection in Aerial Images](https://openaccess.thecvf.com/content_CVPR_2019/papers/Ding_Learning_RoI_Transformer_for_Oriented_Object_Detection_in_Aerial_Images_CVPR_2019_paper.pdf)

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/roi_trans.png" width="800"/>
</div>

Object detection in aerial images is an active yet challenging task in computer vision because of the bird’s-eye view perspective, the highly complex backgrounds, and the variant appearances of objects. Especially when detecting densely packed objects in aerial images, methods relying on horizontal proposals for common object detection often introduce mismatches between the Region of Interests (RoIs) and objects. This leads to the common misalignment between the final object classification confidence and localization accuracy. In this paper, we propose a RoI Transformer to address these problems. The core idea of RoI Transformer is to apply spatial transformations on RoIs and learn the transformation parameters under the supervision of oriented bounding box (OBB) annotations. RoI Transformer is with lightweight and can be easily embedded into detectors for oriented object detection. Simply apply the RoI Transformer to light-head RCNN has achieved state-of-the-art performances on two common and challenging aerial datasets, i.e., DOTA and HRSC2016, with a neglectable reduction to detection speed. Our RoI Transformer exceeds the deformable Position Sensitive RoI pooling when oriented bounding-box annotations are available. Extensive experiments have also validated the flexibility and effectiveness of our RoI Transformer

DOTA1.0

|         Backbone          |  mAP  | Angle | lr schd | Mem (GB) | Inf Time (fps) |  Aug  | Batch Size |                                                    Configs                                                     |                                                                                                                                                                              Download                                                                                                                                                                              |

| ResNet50 (1024,1024,200)  | 73.40 | le90  |   1x    |   8.46   |      16.5      |   -   |     2      | [rotated-faster-rcnn-le90_r50_fpn_1x_dota](../rotated_faster_rcnn/rotated-faster-rcnn-le90_r50_fpn_1x_dota.py) | [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_faster_rcnn/rotated_faster_rcnn_r50_fpn_1x_dota_le90/rotated_faster_rcnn_r50_fpn_1x_dota_le90-0393aa5c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_faster_rcnn/rotated_faster_rcnn_r50_fpn_1x_dota_le90/rotated_faster_rcnn_r50_fpn_1x_dota_le90_20220131_082156.log.json) |

| ResNet50 (1024,1024,200)  | 75.75 | le90  |   1x    |   7.56   |      19.3      |   -   |     2      |                 [roi-trans-le90_r50_fpn_amp-1x_dota](./roi-trans-le90_r50_fpn_amp-1x_dota.py)                  |                     [model](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_r50_fpn_fp16_1x_dota_le90/roi_trans_r50_fpn_fp16_1x_dota_le90-62eb88b1.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_r50_fpn_fp16_1x_dota_le90/roi_trans_r50_fpn_fp16_1x_dota_le90_20220303_193513.log.json)                     |

| ResNet50 (1024,1024,200)  | 76.08 | le90  |   1x    |   8.67   |      14.4      |   -   |     2      |                     [roi-trans-le90_r50_fpn_1x_dota](./roi-trans-le90_r50_fpn_1x_dota.py)                      |                               [model](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_r50_fpn_1x_dota_le90/roi_trans_r50_fpn_1x_dota_le90-d1f0b77a.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_r50_fpn_1x_dota_le90/roi_trans_r50_fpn_1x_dota_le90_20220130_132727.log.json)                               |

| Swin-tiny (1024,1024,200) | 77.51 | le90  |   1x    |          |      10.9      |   -   |     2      |               [roi-trans-le90_swin-tiny_fpn_1x_dota](./roi-trans-le90_swin-tiny_fpn_1x_dota.py)                |                   [model](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_swin_tiny_fpn_1x_dota_le90/roi_trans_swin_tiny_fpn_1x_dota_le90-ddeee9ae.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_swin_tiny_fpn_1x_dota_le90/roi_trans_swin_tiny_fpn_1x_dota_le90_20220131_083622.log.json)                   |

| ResNet50 (1024,1024,500)  | 79.66 | le90  |   1x    |          |      14.4      | MS+RR |     2      |               [roi_trans_r50_fpn_1x_dota_ms_rr_le90](./roi_trans_r50_fpn_1x_dota_ms_rr_le90.py)                |                   [model](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_r50_fpn_1x_dota_ms_rr_le90/roi_trans_r50_fpn_1x_dota_ms_rr_le90-fa99496f.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_r50_fpn_1x_dota_ms_rr_le90/roi_trans_r50_fpn_1x_dota_ms_rr_le90_20220205_171729.log.json)                   |

Notes:

```
@InProceedings{ding2018learning,
	author = {Ding, Jian and Xue, Nan and Long, Yang and Xia, Gui-Song and Lu, Qikai},
	title = {Learning RoI Transformer for Oriented Object Detection in Aerial Images},
	booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
	pages={2849--2858},

title = {Learning RoI Transformer for Oriented Object Detection in Aerial Images},
	year = {2019}
}
```

# Tutorial 4: Train and test in MMagic

In this section, we introduce how to test and train models in MMagic.

In this section, we provide the following guides:

- [Tutorial 4: Train and test in MMagic](#tutorial-4-train-and-test-in-mmagic)
  - [Test a model in MMagic](#test-a-model-in-mmagic)
    - [Test with a single GPUs](#test-with-a-single-gpus)
    - [Test with Slurm](#test-with-slurm)
    - [Test with specific metrics](#test-with-specific-metrics)
  - [Train a model in MMagic](#train-a-model-in-mmagic)
    - [Train with a single GPU](#train-with-a-single-gpu)
    - [Train with multiple nodes](#train-with-multiple-nodes)
    - [Train with multiple GPUs](#train-with-multiple-gpus)
    - [Train with Slurm](#train-with-slurm)
    - [Optional arguments](#optional-arguments)
  - [Train with specific evaluation metrics](#train-with-specific-evaluation-metrics)

- [Train with Slurm](#train-with-slurm)

Users need to [prepare dataset](../user_guides/dataset_prepare.md) first to enable training and testing models in MMagic.

## Test a model in MMagic

### Test with a single GPUs

You can use the following commands to test a pre-trained model with single GPUs.

```shell
python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE}
```

```shell
python tools/test.py configs/example_config.py work_dirs/example_exp/example_model_20200202.pth
```

MMagic supports testing with multiple GPUs,
which can largely save your time in testing models.
You can use the following commands to test a pre-trained model with multiple GPUs.

```shell
./tools/dist_test.sh ${CONFIG_FILE} ${CHECKPOINT_FILE} ${GPU_NUM}
```

```shell

```shell
```

```shell
./tools/dist_test.sh configs/example_config.py work_dirs/example_exp/example_model_20200202.pth
```

### Test with Slurm

If you run MMagic on a cluster managed with [slurm](https://slurm.schedmd.com/), you can use the script `slurm_test.sh`. (This script also supports single machine testing.)

```shell
```

Here is an example of using 8 GPUs to test an example model on the 'dev' partition with the job name 'test'.

```shell
```

You can check [slurm_test.sh](../../../tools/slurm_test.sh) for full arguments and environment variables.

### Test with specific metrics

```

MMagic provides various **evaluation metrics**, i.e., MS-SSIM, SWD, IS, FID, Precision&Recall, PPL, Equivarience, TransFID, TransIS, etc.
We have provided unified evaluation scripts in [tools/test.py](https://github.com/open-mmlab/mmagic/tree/main/tools/test.py) for all models.
If users want to evaluate their models with some metrics, you can add the `metrics` into your config file like this:

```python
# at the end of the configs/styleganv2/stylegan2_c2_ffhq_256_b4x8_800k.py
metrics = [
    dict(
        sample_model='ema'),
    dict(type='PrecisionAndRecall', fake_nums=50000, prefix='PR-50K'),
]
```

sample_model='ema'),
]
```

As above, `metrics` consist of multiple metric dictionaries. Each metric will contain `type` to indicate the category of the metric. `fake_nums` denotes the number of images generated by the model. Some metrics will output a dictionary of results, you can also set `prefix`  to specify the prefix of the results.
If you set the prefix of FID as `FID-Full-50k`, then an example of output may be

```bash
FID-Full-50k/fid: 3.6561  FID-Full-50k/mean: 0.4263  FID-Full-50k/cov: 3.2298
```

Then users can test models with the command below:

```shell
bash tools/dist_test.sh ${CONFIG_FILE} ${CKPT_FILE}
```

If you are in slurm environment, please switch to the [tools/slurm_test.sh](https://github.com/open-mmlab/mmagic/tree/main/tools/slurm_test.sh) by using the following commands:

```shell

```

```shell
sh slurm_test.sh ${PLATFORM} ${JOBNAME} ${CONFIG_FILE} ${CKPT_FILE}
```

## Train a model in MMagic

MMagic supports multiple ways of training:

1. [Train with a single GPU](#train-with-a-single-gpu)
2. [Train with multiple GPUs](#train-with-multiple-gpus)
3. [Train with multiple nodes](#train-with-multiple-nodes)
4. [Train with Slurm](#train-with-slurm)

Specifically, all outputs (log files and checkpoints) will be saved to the working directory,
which is specified by `work_dir` in the config file.

### Train with a single GPU

```shell
CUDA_VISIBLE=0 python tools/train.py configs/example_config.py --work-dir work_dirs/example
```

### Train with multiple nodes

To launch distributed training on multiple machines, which can be accessed via IPs, run the following commands:

```

```shell
```

```shell
```

To speed up network communication, high speed network hardware, such as Infiniband, is recommended.
Please refer to [PyTorch docs](https://pytorch.org/docs/1.11/distributed.html#launch-utility) for more information.

```shell
```

### Train with Slurm

If you run MMagic on a cluster managed with [slurm](https://slurm.schedmd.com/), you can use the script `slurm_train.sh`. (This script also supports single machine training.)

```shell

```shell
```

Here is an example of using 8 GPUs to train an inpainting model on the dev partition.

```shell
```

You can check [slurm_train.sh](https://github.com/open-mmlab/mmagic/blob/master/tools/slurm_train.sh) for full arguments and environment variables.

### Optional arguments

- `--amp`: This argument is used for fixed-precision training.
- `--resume`: This argument is used for auto resume if the training is aborted.

## Train with specific evaluation metrics

Benefit from the `mmengine`'s `Runner`. We can evaluate model during training in a simple way as below.

```python
# define metrics
metrics = [
    dict(

```python
metrics = [
    dict(
        inception_style='StyleGAN')
]

# define dataloader
    batch_size=128,
        type='BasicImageDataset',
        data_root='data/celeba-cropped/',
            dict(type='LoadImageFromFile', key='img'),
            dict(type='Resize', scale=(64, 64)),
            dict(type='PackInputs')
        ]),

# define val interval
train_cfg = dict(by_epoch=False, val_begin=1, val_interval=10000)

# define val loop and evaluator
val_cfg = dict(type='MultiValLoop')
val_evaluator = dict(type='Evaluator', metrics=metrics)
```

```

You can set `val_begin` and `val_interval` to adjust when to begin validation and interval of validation.

For details of metrics, refer to [metrics' guide](./metrics.md).

<details>

```bibtex
}
```

<details>

```bibtex
}
```

Results on LaPa val set

author={Liu, Yinglu and Shi, Hailin and Shen, Hao and Si, Yue and Wang, Xiaobo and Mei, Tao},
}
```

| [pose_rtmpose_m](/configs/face_2d_keypoint/rtmpose/lapa/rtmpose-m_8xb64-120e_lapa-256x256.py) |  256x256   | 1.29 | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-lapa_pt-aic-coco_120e-256x256-762b1ae2_20230422.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-lapa_pt-aic-coco_120e-256x256-762b1ae2_20230422.json) |

# ConvNeXt

</div>

DOTA1.0

|          Backbone          |  mAP  | Angle | lr schd | Mem (GB) | Inf Time (fps) | Aug | Batch Size |                                                                        Configs                                                                        |                                                                                                                                                                                                                   Download                                                                                                                                                                                                                   |

|  ResNet50 (1024,1024,200)  | 70.22 | le90  |   1x    |   3.35   |      16.9      |  -  |     2      |              [rotated-retinanet-rbox-le90_r50_fpn_kld-stable_1x_dota](../kld/rotated-retinanet-rbox-le90_r50_fpn_kld-stable_1x_dota.py)               |                            [model](https://download.openmmlab.com/mmrotate/v0.1.0/kld/rotated_retinanet_obb_kld_stable_r50_fpn_1x_dota_le90/rotated_retinanet_obb_kld_stable_r50_fpn_1x_dota_le90-31193e00.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kld/rotated_retinanet_obb_kld_stable_r50_fpn_1x_dota_le90/rotated_retinanet_obb_kld_stable_r50_fpn_1x_dota_le90_20220402_225531.log.json)                            |

|  ResNet50 (1024,1024,200)  | 71.30 | le90  |   1x    |   3.61   |      16.9      |  -  |     2      |        [rotated-retinanet-rbox-le90_r50_fpn_kld-stable_adamw-1x_dota](../kld/rotated-retinanet-rbox-le90_r50_fpn_kld-stable_adamw-1x_dota.py)         |                [model](https://download.openmmlab.com/mmrotate/v0.1.0/kld/rotated_retinanet_obb_kld_stable_r50_adamw_fpn_1x_dota_le90/rotated_retinanet_obb_kld_stable_r50_adamw_fpn_1x_dota_le90-474d9955.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kld/rotated_retinanet_obb_kld_stable_r50_adamw_fpn_1x_dota_le90/rotated_retinanet_obb_kld_stable_r50_adamw_fpn_1x_dota_le90_20220608_003758.log.json)                |

| ConvNeXt-T (1024,1024,200) | 74.49 | le90  |   1x    |   6.12   |      7.9       |  -  |     2      | [rotated-retinanet-rbox-le90_convnext-tiny_fpn_kld-stable_adamw-1x_dota](./rotated-retinanet-rbox-le90_convnext-tiny_fpn_kld-stable_adamw-1x_dota.py) | [model](https://download.openmmlab.com/mmrotate/v0.1.0/convnext/rotated_retinanet_obb_kld_stable_convnext_adamw_fpn_1x_dota_le90/rotated_retinanet_obb_kld_stable_convnext_adamw_fpn_1x_dota_le90-388184f6.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/convnext/rotated_retinanet_obb_kld_stable_convnext_adamw_fpn_1x_dota_le90/rotated_retinanet_obb_kld_stable_convnext_adamw_fpn_1x_dota_le90_20220608_191712.log.json) |

**Note**:

```shell
pip install "mmcls>=1.0.0rc0"
```

- The performance may be unstable according to mmdetection's experience.

```shell
```

```bibtex
}
```

# ACRN

[Actor-centric relation network](https://openaccess.thecvf.com/content_ECCV_2018/html/Chen_Sun_Actor-centric_Relation_Network_ECCV_2018_paper.html)

Current state-of-the-art approaches for spatio-temporal action localization rely on detections at the frame level and model temporal context with 3D ConvNets. Here, we go one step further and model spatio-temporal relations to capture the interactions between human actors, relevant objects and scene elements essential to differentiate similar human actions. Our approach is weakly supervised and mines the relevant elements automatically with an actor-centric relational network (ACRN). ACRN computes and accumulates pair-wise relation information from actor and global scene features, and generates relation features for action classification. It is implemented as neural networks and can be trained jointly with an existing action detection system. We show that ACRN outperforms alternative approaches which capture relation information, and that the proposed framework improves upon the state-of-the-art performance on JHMDB and AVA. A visualization of the learned relation features confirms that our approach is able to attend to the relevant relations for each action.

<img src="https://user-images.githubusercontent.com/34324155/142996406-09ac1b09-2a9e-478c-9035-5fe7a80bc80b.png" width="800"/>
</div>

### AVA2.1

|          8x8x1          |  8   | SlowFast ResNet50 | Kinetics-400 | 27.65 | [config](/configs/detection/acrn/slowfast-acrn_kinetics400-pretrained-r50_8xb8-8x8x1-cosine-10e_ava21-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/acrn/slowfast-acrn_kinetics400-pretrained-r50_8xb8-8x8x1-cosine-10e_ava21-rgb/slowfast-acrn_kinetics400-pretrained-r50_8xb8-8x8x1-cosine-10e_ava21-rgb_20220906-0dae1a90.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/acrn/slowfast-acrn_kinetics400-pretrained-r50_8xb8-8x8x1-cosine-10e_ava21-rgb/slowfast-acrn_kinetics400-pretrained-r50_8xb8-8x8x1-cosine-10e_ava21-rgb.log) |

### AVA2.2

|          8x8x1          |  8   | SlowFast ResNet50 | Kinetics-400 | 27.71 | [config](/configs/detection/acrn/slowfast-acrn_kinetics400-pretrained-r50_8xb8-8x8x1-cosine-10e_ava22-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/detection/acrn/slowfast-acrn_kinetics400-pretrained-r50_8xb8-8x8x1-cosine-10e_ava22-rgb/slowfast-acrn_kinetics400-pretrained-r50_8xb8-8x8x1-cosine-10e_ava22-rgb_20220906-66ec24a2.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/detection/acrn/slowfast-acrn_kinetics400-pretrained-r50_8xb8-8x8x1-cosine-10e_ava22-rgb/slowfast-acrn_kinetics400-pretrained-r50_8xb8-8x8x1-cosine-10e_ava22-rgb.log) |

For more details on data preparation, you can refer to [AVA](/tools/data/ava/README.md).

## Train

```shell
```

Example: train ACRN with SlowFast backbone on AVA2.1 in a deterministic option with periodic validation.

```shell
python tools/train.py configs/detection/acrn/slowfast-acrn_kinetics400-pretrained-r50_8xb8-8x8x1-cosine-10e_ava21-rgb.py \
```

## Test

--seed 0 --deterministic
```

## Test

```shell
```

Example: test ACRN with SlowFast backbone on AVA2.1 and dump the result to a pkl file.

```shell
python tools/test.py configs/detection/acrn/slowfast-acrn_kinetics400-pretrained-r50_8xb8-8x8x1-cosine-10e_ava21-rgb.py \
```

```BibTeX
@inproceedings{sun2018actor,
  title={Actor-centric relation network},
  author={Sun, Chen and Shrivastava, Abhinav and Vondrick, Carl and Murphy, Kevin and Sukthankar, Rahul and Schmid, Cordelia},

  pages={318--334},
}
```

# ReDet

> [ReDet: A Rotation-equivariant Detector for Aerial Object Detection](https://openaccess.thecvf.com/content/CVPR2021/papers/Han_ReDet_A_Rotation-Equivariant_Detector_for_Aerial_Object_Detection_CVPR_2021_paper.pdf)

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/redet.png" width="800"/>
</div>

Recently, object detection in aerial images has gained much attention in computer vision. Different from objects in natural images, aerial objects are often distributed with arbitrary orientation. Therefore, the detector requires more parameters to encode the orientation information, which are often highly redundant and inefficient. Moreover, as ordinary CNNs do not explicitly model the orientation variation, large amounts of rotation augmented data is needed to train an accurate object detector. In this paper, we propose a Rotation-equivariant Detector (ReDet) to address these issues, which explicitly encodes rotation equivariance and rotation invariance. More precisely, we incorporate rotation-equivariant networks into the detector to extract rotation-equivariant features, which can accurately predict the orientation and lead to a huge reduction of model size. Based on the rotation-equivariant features, we also present Rotation-invariant RoI Align (RiRoI Align), which adaptively extracts rotation-invariant features from equivariant features according to the orientation of RoI. Extensive experiments on several challenging aerial image datasets DOTA-v1.0, DOTA-v1.5 and HRSC2016, show that our method can achieve state-of-the-art performance on the task of aerial object detection. Compared with previous best results, our ReDet gains 1.2, 3.5 and 2.6 mAP on DOTA-v1.0, DOTA-v1.5 and HRSC2016 respectively while reducing the number of parameters by 60% (313 Mb vs. 121 Mb).

DOTA1.0

|          Backbone          |  mAP  | Angle | lr schd | Mem (GB) | Inf Time (fps) |  Aug  | Batch Size |                                                    Configs                                                     |                                                                                                                                                                              Download                                                                                                                                                                              |

|  ResNet50 (1024,1024,200)  | 73.40 | le90  |   1x    |   8.46   |      16.5      |   -   |     2      | [rotated-faster-rcnn-le90_r50_fpn_1x_dota](../rotated_faster_rcnn/rotated-faster-rcnn-le90_r50_fpn_1x_dota.py) | [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_faster_rcnn/rotated_faster_rcnn_r50_fpn_1x_dota_le90/rotated_faster_rcnn_r50_fpn_1x_dota_le90-0393aa5c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_faster_rcnn/rotated_faster_rcnn_r50_fpn_1x_dota_le90/rotated_faster_rcnn_r50_fpn_1x_dota_le90_20220131_082156.log.json) |

| ReResNet50 (1024,1024,200) | 75.99 | le90  |   1x    |   7.71   |      13.3      |   -   |     2      |                  [redet-le90_re50_refpn_amp-1x_dota](./redet-le90_re50_refpn_amp-1x_dota.py)                   |                           [model](https://download.openmmlab.com/mmrotate/v0.1.0/redet/redet_re50_refpn_fp16_1x_dota_le90/redet_re50_refpn_fp16_1x_dota_le90-1e34da2d.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/redet/redet_re50_refpn_fp16_1x_dota_le90/redet_re50_refpn_fp16_1x_dota_le90_20220303_194725.log.json)                           |

| ReResNet50 (1024,1024,200) | 76.68 | le90  |   1x    |   9.32   |      10.9      |   -   |     2      |                      [redet-le90_re50_refpn_1x_dota](./redet-le90_re50_refpn_1x_dota.py)                       |                                         [model](https://download.openmmlab.com/mmrotate/v0.1.0/redet/redet_re50_fpn_1x_dota_le90/redet_re50_fpn_1x_dota_le90-724ab2da.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/redet/redet_re50_fpn_1x_dota_le90/redet_re50_fpn_1x_dota_le90_20220130_132751.log.json)                                         |

| ReResNet50 (1024,1024,500) | 79.87 | le90  |   1x    |          |      10.9      | MS+RR |     2      |                [redet-le90_re50_refpn_rr-1x_dota-ms](./redet-le90_re50_refpn_rr-1x_dota-ms.py)                 |                             [model](https://download.openmmlab.com/mmrotate/v0.1.0/redet/redet_re50_fpn_1x_dota_ms_rr_le90/redet_re50_fpn_1x_dota_ms_rr_le90-fc9217b5.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/redet/redet_re50_fpn_1x_dota_ms_rr_le90/redet_re50_fpn_1x_dota_ms_rr_le90_20220206_105343.log.json)                             |

HRSC

HRSC

|       Backbone       |  mAP  | AP50  | AP75  | Angle | lr schd | Mem (GB) | Inf Time (fps) | Aug | Batch Size |                               Configs                               |                                                                                                                                          Download                                                                                                                                          |
| :------------------: | :---: | :---: | :---: | :---: | :-----: | :------: | :------------: | :-: | :--------: | :-----------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

| ReResNet50 (800,512) | 72.31 | 90.40 | 89.50 | le90  |   3x    |   2.30   |      17.5      |  -  |     2      | [redet-le90_re50_refpn_3x_hrsc](./redet-le90_re50_refpn_3x_hrsc.py) | [model](https://download.openmmlab.com/mmrotate/v0.1.0/redet/redet_re50_refpn_3x_hrsc_le90/redet_re50_refpn_3x_hrsc_le90-241e217b.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/redet/redet_re50_refpn_3x_hrsc_le90/redet_re50_refpn_3x_hrsc_le90_20220411_113223.log.json) |

Notes:

- Please download pretrained weight of ReResNet from [ReDet](https://github.com/csuhan/ReDet), and put it on `work_dirs/pretrain`. BTW, it is normal for `missing keys in source state_dict: xxx.filter ` to appear in the log. Don't worry!
- Please use distributed training, there are some bugs when using `train.py`.

```
@inproceedings{han2021redet,
  title={Redet: A rotation-equivariant detector for aerial object detection},

```
  author={Han, Jiaming and Ding, Jian and Xue, Nan and Xia, Gui-Song},
  pages={2786--2795},
}

```

# 可视化

MMDetection3D 提供了 `Det3DLocalVisualizer` 用来在训练及测试阶段可视化和存储模型的状态以及结果，其具有以下特性：

1. 支持多模态数据和多任务的基本绘图界面。
2. 支持多个后端（如 local，TensorBoard），将训练状态（如 `loss`，`lr`）或模型评估指标写入指定的一个或多个后端中。
3. 支持多模态数据真实标签的可视化，3D 检测结果的跨模态可视化。

## 基本绘制界面

继承自 `DetLocalVisualizer`，`Det3DLocalVisualizer` 提供了在 2D 图像上绘制常见目标的界面，例如绘制检测框、点、文本、线、圆、多边形、二进制掩码等。关于 2D 绘制的更多细节，请参考 MMDetection 中的[可视化文档](https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/visualization.html)。这里我们介绍 3D 绘制界面。

### 在图像上绘制点云

通过使用 `draw_points_on_image`，我们支持在图像上绘制点云。

```python
from mmengine import load

from mmdet3d.visualization import Det3DLocalVisualizer

info_file = load('demo/data/kitti/000008.pkl')
points = np.fromfile('demo/data/kitti/000008.bin', dtype=np.float32)
points = points.reshape(-1, 4)[:, :3]
lidar2img = np.array(info_file['data_list'][0]['images']['CAM2']['lidar2img'], dtype=np.float32)

visualizer = Det3DLocalVisualizer()
img = mmcv.imread('demo/data/kitti/000008.png')

visualizer.set_image(img)
visualizer.draw_points_on_image(points, lidar2img)
visualizer.show()
```

![points_on_image](../../../resources/points_on_image.png)

### 在点云上绘制 3D 框

通过使用 `draw_bboxes_3d`，我们支持在点云上绘制 3D 框。

```python

from mmdet3d.structures import LiDARInstance3DBoxes

points = points.reshape(-1, 4)
# set point cloud in visualizer
visualizer.set_points(points)
bboxes_3d = LiDARInstance3DBoxes(
    torch.tensor([[8.7314, -1.8559, -1.5997, 4.2000, 3.4800, 1.8900,
                   -1.5808]]))
# Draw 3D bboxes
visualizer.draw_bboxes_3d(bboxes_3d)
```

torch.tensor([[8.7314, -1.8559, -1.5997, 4.2000, 3.4800, 1.8900,
```

![mono3d](../../../resources/pcd.png)

### 在图像上绘制投影的 3D 框

通过使用 `draw_proj_bboxes_3d`，我们支持在图像上绘制投影的 3D 框。

```python

from mmdet3d.structures import CameraInstance3DBoxes

cam2img = np.array(info_file['data_list'][0]['images']['CAM2']['cam2img'], dtype=np.float32)
bboxes_3d = []
for instance in info_file['data_list'][0]['instances']:
    bboxes_3d.append(instance['bbox_3d'])
gt_bboxes_3d = np.array(bboxes_3d, dtype=np.float32)
gt_bboxes_3d = CameraInstance3DBoxes(gt_bboxes_3d)
input_meta = {'cam2img': cam2img}

# project 3D bboxes to image
visualizer.draw_proj_bboxes_3d(gt_bboxes_3d, input_meta)
```

### 绘制 BEV 视角的框

通过使用 `draw_bev_bboxes`，我们支持绘制 BEV 视角下的框。

```python

# set bev image in visualizer
visualizer.set_bev_image()
# draw bev bboxes
visualizer.draw_bev_bboxes(gt_bboxes_3d, edge_colors='orange')
```

### 绘制 3D 分割掩码

```

通过使用 `draw_seg_mask`，我们支持通过逐点着色来绘制分割掩码。

```python

points = np.fromfile('demo/data/sunrgbd/000017.bin', dtype=np.float32)
points = points.reshape(-1, 3)
mask = np.random.rand(points.shape[0], 3)
points_with_mask = np.concatenate((points, mask), axis=-1)
# Draw 3D points with mask
visualizer.set_points(points, pcd_mode=2, vis_mode='add')
visualizer.draw_seg_mask(points_with_mask)
```

## 结果

如果想要可视化训练模型的预测结果，你可以运行如下指令：

```bash
python tools/test.py ${CONFIG_FILE} ${CKPT_PATH} --show --show-dir ${SHOW_DIR}
```

运行该指令后，绘制的结果（包括输入数据和网络输出在输入上的可视化）将会被保存在 `${SHOW_DIR}` 中。

```

## 结果

```bash
```

运行该指令后，你将在 `${SHOW_DIR}` 中获得输入数据，网络输出和真是标签在输入上的可视化（如在多模态检测任务和基于视觉的检测任务中的 `***_gt.png` 和 `***_pred.png`）。当启用 `show` 时，[Open3D](http://www.open3d.org/) 将会用于在线可视化结果。如果你是在没有 GUI 的远程服务器上测试时，在线可视化是不被支持的。你可以从远程服务器中下载 `results.pkl`，并在本地机器上离线可视化预测结果。

使用 `Open3D` 后端离线可视化结果，你可以运行如下指令：

```bash
python tools/misc/visualize_results.py ${CONFIG_FILE} --result ${RESULTS_PATH} --show-dir ${SHOW_DIR}
```

![](../../../resources/open3d_visual.gif)

这需要在远程服务器中能够推理并生成结果，然后用户在主机中使用 GUI 打开。

## 数据集

我们也提供了脚本来可视化数据集而无需推理。你可以使用 `tools/misc/browse_dataset.py` 来在线可视化加载的数据的真实标签，并保存在硬盘中。目前我们支持所有数据集的单模态 3D 检测和 3D 分割，KITTI 和 SUN RGB-D 的多模态 3D 检测，以及 nuScenes 的单目 3D 检测。如果想要浏览 KITTI 数据集，你可以运行如下指令：

```shell
python tools/misc/browse_dataset.py configs/_base_/datasets/kitti-3d-3class.py --task lidar_det --output-dir ${OUTPUT_DIR}

```shell
```

**注意**：一旦指定了 `--output-dir`，当在 open3d 窗口中按下 `_ESC_` 时，用户指定的视图图像将会被保存下来。如果你想要对点云进行缩放操作以观察更多细节， 你可以在命令中指定 `--show-interval=0`。

为了验证数据的一致性和数据增强的效果，你可以加上 `--aug` 来可视化数据增强后的数据，指令如下所示：

```shell
python tools/misc/browse_dataset.py configs/_base_/datasets/kitti-3d-3class.py --task det --aug --output-dir ${OUTPUT_DIR}
```

如果你想显示带有投影的 3D 边界框的 2D 图像，你需要一个支持多模态数据加载的配置文件，并将 `--task` 参数改为 `multi-modality_det`。示例如下：

```shell
python tools/misc/browse_dataset.py configs/mvxnet/mvxnet_fpn_dv_second_secfpn_8xb2-80e_kitti-3d-3class.py --task multi-modality_det --output-dir ${OUTPUT_DIR}
```

![](../../../resources/browse_dataset_multi_modality.png)

你可以使用不同的配置浏览不同的数据集，例如在 3D 语义分割任务中可视化 ScanNet 数据集：

```shell
python tools/misc/browse_dataset.py configs/_base_/datasets/scannet-seg.py --task lidar_seg --output-dir ${OUTPUT_DIR}
```

```shell
```

![](../../../resources/browse_dataset_seg.png)

在单目 3D 检测任务中浏览 nuScenes 数据集：

```shell
python tools/misc/browse_dataset.py configs/_base_/datasets/nus-mono3d.py --task mono_det --output-dir ${OUTPUT_DIR}
```

![](../../../resources/browse_dataset_mono.png)

# Training Tricks

MMSegmentation support following training tricks out of box.

## Different Learning Rate(LR) for Backbone and Heads

In semantic segmentation, some methods make the LR of heads larger than backbone to achieve better performance or faster convergence.

In MMSegmentation, you may add following lines to config to make the LR of heads 10 times of backbone.

```python
```

With this modification, the LR of any parameter group with `'head'` in name will be multiplied by 10.
You may refer to [MMEngine documentation](https://mmengine.readthedocs.io/en/latest/tutorials/optim_wrapper.html#advanced-usages) for further details.

## Online Hard Example Mining (OHEM)

We implement pixel sampler for training sampling, like OHEM (Online Hard Example Mining),
which is used for remove the "easy" examples for model training.

Here is an example config of training PSPNet with OHEM enabled.

```python
_base_ = './pspnet_r50-d8_4xb2-40k_cityscapes-512x1024.py'
```

In this way, only pixels with confidence score under 0.7 are used to train. And we keep at least 100000 pixels during training. If `thresh` is not specified, pixels of top `min_kept` loss will be selected.

## Class Balanced Loss

For dataset that is not balanced in classes distribution, you may change the loss weight of each class.
Here is an example for cityscapes dataset.

```python

```python
            # DeepLab used this class weight for cityscapes
```

`class_weight` will be passed into `CrossEntropyLoss` as `weight` argument. Please refer to [PyTorch Doc](https://pytorch.org/docs/stable/nn.html?highlight=crossentropy#torch.nn.CrossEntropyLoss) for details.

## Multiple Losses

For loss calculation, we support multiple losses training concurrently. Here is an example config of training `unet` on `DRIVE` dataset, whose loss function is `1:3` weighted sum of `CrossEntropyLoss` and `DiceLoss`:

```python

```python
    ]),
    ]),
)
```

In this way, `loss_weight` and `loss_name` will be weight and name in training log of corresponding loss, respectively.

Note: If you want this loss item to be included into the backward graph, `loss_` must be the prefix of the name.

# MaskFeat Pre-training with Video

- [MaskFeat Pre-training with Video](#maskfeat-pre-training-with-video)
  - [Description](#description)
  - [Usage](#usage)
    - [Setup Environment](#setup-environment)
    - [Data Preparation](#data-preparation)
    - [Pre-training Commands](#pre-training-commands)
      - [On Local Single GPU](#on-local-single-gpu)
      - [On Multiple GPUs](#on-multiple-gpus)
      - [On Multiple GPUs with Slurm](#on-multiple-gpus-with-slurm)
    - [Downstream Tasks Commands](#downstream-tasks-commands)
      - [On Multiple GPUs](#on-multiple-gpus-1)
      - [On Multiple GPUs with Slurm](#on-multiple-gpus-with-slurm-1)
  - [Results](#results)
  - [Citation](#citation)
  - [Checklist](#checklist)

Author: @fangyixiao18

This is the implementation of **MaskFeat** with video dataset, like Kinetics400.

## Usage

## Usage

<!-- For a typical model, this section should contain the commands for dataset prepareation, pre-training, downstream tasks. You are also suggested to dump your environment specification to env.yml by `conda env export > env.yml`. -->

Requirements:

- MMPretrain >= 1.0.0rc0
- MMAction2 >= 1.0.0rc3

Besides, to process the video data, we apply transforms in MMAction2. The instruction to install MMAction2 can be found in [Get Started documentation](https://mmaction2.readthedocs.io/en/1.x/get_started.html).

You can refer to the [documentation](https://mmaction2.readthedocs.io/en/1.x/user_guides/2_data_prepare.html) in MMAction2.

### Pre-training Commands

At first, you need to add the current folder to `PYTHONPATH`, so that Python can find your model files. In `projects/maskfeat_video/` root directory, please run command below to add it.

```shell
```

Then run the following commands to train the model:

#### On Local Single GPU

```bash

mim train mmpretrain configs/maskfeat_mvit-small_8xb32-amp-coslr-300e_k400.py \
    --work-dir work_dirs/selfsup/maskfeat_mvit-small_8xb32-amp-coslr-300e_k400/

# train with scripts
python tools/train.py configs/maskfeat_mvit-small_8xb32-amp-coslr-300e_k400.py \
```

#### On Multiple GPUs

```bash

--work-dir work_dirs/selfsup/maskfeat_mvit-small_8xb32-amp-coslr-300e_k400/
```

```bash
    --work-dir work_dirs/selfsup/maskfeat_mvit-small_8xb32-amp-coslr-300e_k400/ \

bash tools/dist_train.sh configs/maskfeat_mvit-small_8xb32-amp-coslr-300e_k400.py 8
```

Note:

- CONFIG: the config files under the directory `configs/`
- WORK_DIR: the working directory to save configs, logs, and checkpoints

#### On Multiple GPUs with Slurm

```bash
mim train mmpretrain configs/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400.py \
    --work-dir work_dirs/selfsup/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400/ \

--work-dir work_dirs/selfsup/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400/ \

GPUS_PER_NODE=8 GPUS=16 bash tools/slurm_train.sh ${PARTITION} maskfeat-video \
    configs/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400.py \
    --work-dir work_dirs/selfsup/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400/
```

Note:

- PARTITION: the slurm partition you are using

### Downstream Tasks Commands

To evaluate the **MaskFeat MViT** pretrained with MMPretrain, we recommend to run MMAction2:

```bash
# command example for train
mim train mmaction2 ${CONFIG} \
    --launcher pytorch -gpus 8 \
    --cfg-options model.backbone.init_cfg.type=Pretrained \
    model.backbone.init_cfg.checkpoint=${CHECKPOINT} \

--work-dir ${WORK_DIR} \
    model.backbone.init_cfg.prefix="backbone." \
    ${PY_ARGS}
    [optional args]

mim train mmaction2 configs/mvit-small_ft-8xb8-coslr-100e_k400.py \
    --work-dir work_dirs/benchmarks/maskfeat/training_maskfeat-mvit-k400/ \
    model.backbone.init_cfg.checkpoint=https://download.openmmlab.com/mmselfsup/1.x/maskfeat/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400_20230131-87d60b6f.pth \
    $PY_ARGS

# command example for test
mim test mmaction2 configs/mvit-small_ft-8xb16-coslr-100e_k400.py \

model.backbone.init_cfg.prefix="backbone." \
    $PY_ARGS

  --checkpoint https://download.openmmlab.com/mmselfsup/1.x/maskfeat/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400/mvit-small_ft-8xb16-coslr-100e_k400/mvit-small_ft-8xb16-coslr-100e_k400_20230131-5e8303f5.pth \
  --work-dir work_dirs/benchmarks/maskfeat/maskfeat-mvit-k400/test/ \
  --launcher pytorch --gpus 8
```

```bash
    model.backbone.init_cfg.checkpoint=$CHECKPOINT \
    $PY_ARGS

mim test mmaction2 ${CONFIG} \

model.backbone.init_cfg.checkpoint=$CHECKPOINT \
    $PY_ARGS

    --checkpoint https://download.openmmlab.com/mmselfsup/1.x/maskfeat/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400/mvit-small_ft-8xb16-coslr-100e_k400/mvit-small_ft-8xb16-coslr-100e_k400_20230131-5e8303f5.pth
    $PY_ARGS
```

Note:

- CHECKPOINT: the pretrained checkpoint of MMPretrain saved in working directory, like `$WORK_DIR/epoch_300.pth`
- PY_ARGS: other optional args

## Results

## Results

<!-- You should claim whether this is based on the pre-trained weights, which are converted from the official release; or it's a reproduced result obtained from retraining the model in this project. -->

The Fine-tuning results are based on Kinetics400(K400) dataset.

Due to the version of K400 dataset, our pretraining, fine-tuning and the final test results are based on MMAction2 version, which is a little different from PySlowFast version.

<thead>
  <tr>
	    <th>Algorithm</th>
	    <th>Backbone</th>
	    <th>Epoch</th>
      <th>Batch Size</th>
      <th>Fine-tuning</th>
      <th>Pretrain Links</th>
      <th>Fine-tuning Links</th>
	</tr>
  </thead>
  <tbody>
  <tr>
      <td>MaskFeat</td>
	    <td>MViT-small</td>
	    <td>300</td>
      <td>512</td>
      <td>81.8</td>

<th>Fine-tuning Links</th>
	</tr>
  </thead>
  <tbody>
  <tr>
      <td><a href='https://github.com/open-mmlab/mmselfsup/blob/dev-1.x/projects/maskfeat_video/configs/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400.py'>config</a> | <a href='https://download.openmmlab.com/mmselfsup/1.x/maskfeat/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400_20230131-87d60b6f.pth'>model</a> | <a href='https://download.openmmlab.com/mmselfsup/1.x/maskfeat/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400_20230118_114151.json'>log</a></td>

<td><a href='https://github.com/open-mmlab/mmselfsup/blob/dev-1.x/projects/maskfeat_video/configs/mvit-small_ft-8xb16-coslr-100e_k400.py'>config</a> | <a href='https://download.openmmlab.com/mmselfsup/1.x/maskfeat/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400/mvit-small_ft-8xb16-coslr-100e_k400/mvit-small_ft-8xb16-coslr-100e_k400_20230131-5e8303f5.pth'>model</a> | <a href='https://download.openmmlab.com/mmselfsup/1.x/maskfeat/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400/mvit-small_ft-8xb16-coslr-100e_k400/mvit-small_ft-8xb16-coslr-100e_k400_20230121_142927.json'>log</a></td>
	</tr>
</tbody>
</table>

Remarks:

- We converted the pretrained model from PySlowFast and run fine-tuning with MMAction2, based on MMAction2 version of K400, we got `81.5` test accuracy. The pretrained model from MMPretrain got `81.8`, as provided above.

- We also tested our model on [other version](https://github.com/facebookresearch/video-nonlocal-net/blob/main/DATASET.md) of K400, we got `82.1` test accuracy.
- Some other details can be found in [MMAction2 MViT page](https://github.com/open-mmlab/mmaction2/tree/dev-1.x/configs/recognition/mvit).

```bibtex
}
```

<!--The PIC (person in charge) or contributors of this project should check all the items that they believe have been finished, which will further be verified by codebase maintainers via a PR.

    <!-- The code's design shall follow existing interfaces and convention. For example, each model component should be registered into `MMPretrain.registry.MODELS` and configurable via a config file. -->

- [x] Inference correctness

    <!-- If you are reproducing the result from a paper, make sure your model's inference-time feature vectors or losses matches that from the original codes. The weights usually could be obtained by simply renaming the keys in the official pre-trained weights. This test could be skipped though, if you are able to prove the training-time correctness and check the second milestone. -->

    <!-- If you are reproducing the result from a paper, checking this item means that you should have trained your model from scratch based on the original paper's specification and verified that the final result. Due to the pretrain-downstream pipeline of self-supervised learning, this item requires at least one downstream result matches the report within a minor error range. -->

    <!-- Ideally *all* the methods should have [type hints](https://www.pythontutorial.net/python-basics/python-type-hints/) and [docstrings](https://google.github.io/styleguide/pyguide.html#381-docstrings). [Example](https://github.com/open-mmlab/mmpretrain/blob/main/mmpretrain/models/selfsup/mae.py) -->

    <!-- Unit tests for each module are required. [Example](https://github.com/open-mmlab/mmpretrain/blob/main/tests/test_models/test_selfsup/test_mae.py) -->

    <!-- It will be parsed by MIM and Inferencer. [Example](https://github.com/open-mmlab/mmpretrain/blob/main/configs/mae/metafile.yml). In particular, you may have to refactor this README into a standard one. [Example](https://github.com/open-mmlab/mmpretrain/blob/main/configs/mae/README.md) -->

- [ ] Refactor and Move your modules into the core package following the codebase's file hierarchy structure.

## 1. Abstract

## 2. Steps

- install dsdl:

  install by pip:

  ```

## 2. Steps

  ```
  ```

  install by source code:

  ```
  git clone https://github.com/opendatalab/dsdl-sdk.git -b schema-dsdl
  cd dsdl-sdk
  ```

- install mmdet and pytorch:
  please refer this [installation documents](https://mmdetection.readthedocs.io/en/latest/get_started.html).

- train:

  ```
  ```

  ```
  ```

## 3. Test Results

- detection task:

  |  Datasets  |                                                                                         Model                                                                                          | box AP |           Config            |

| :--------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :----: | :-------------------------: |
  |  VOC07+12  |             [model](https://download.openmmlab.com/mmdetection/v2.0/pascal_voc/faster_rcnn_r50_fpn_1x_voc0712/faster_rcnn_r50_fpn_1x_voc0712_20220320_192712-54bef0f3.pth)             | 80.3\* |   [config](./voc0712.py)    |
  |    COCO    |                   [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth)                   |  37.4  |     [config](./coco.py)     |
  | Objects365 |       [model](https://download.openmmlab.com/mmdetection/v2.0/objects365/faster_rcnn_r50_fpn_16x4_1x_obj365v2/faster_rcnn_r50_fpn_16x4_1x_obj365v2_20221220_175040-5910b015.pth)       |  19.8  | [config](./objects365v2.py) |

| OpenImages | [model](https://download.openmmlab.com/mmdetection/v2.0/openimages/faster_rcnn_r50_fpn_32x2_cas_1x_openimages/faster_rcnn_r50_fpn_32x2_cas_1x_openimages_20220306_202424-98c630e5.pth) | 59.9\* | [config](./openimagesv6.py) |

  \*: box AP in voc metric and openimages metric, actually means AP_50.

- instance segmentation task:

  | Datasets |                                                                    Model                                                                     | box AP | mask AP |            Config            |
  | :------: | :------------------------------------------------------------------------------------------------------------------------------------------: | :----: | :-----: | :--------------------------: |
  |   COCO   | [model](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_fpn_1x_coco/mask_rcnn_r50_fpn_1x_coco_20200205-d4b0c5d6.pth) |  38.1  |  34.7   | [config](./coco_instance.py) |

# Branches

This documentation aims to provide a comprehensive understanding of the purpose and features of each branch in MMOCR.

## Branch Overview

The `main` branch serves as the default branch for the MMOCR project. It contains the latest stable version of MMOCR, currently housing the code for MMOCR 1.x (e.g. v1.0.0). The `main` branch ensures users have access to the most recent and reliable version of the software.

The `dev-1.x` branch is dedicated to the development of the next major version of MMOCR. This branch will routinely undergo reliance tests, and the passing commits will be squashed in a release and published to the `main` branch. By having a separate development branch, the project can continue to evolve without impacting the stability of the `main` branch. **All the PRs should be merged into the `dev-1.x` branch.**

The `0.x` branch serves as an archive for MMOCR 0.x (e.g. v0.6.3). This branch will no longer actively receive updates or improvements, but it remains accessible for historical reference or for users who have not yet upgraded to MMOCR 1.x.

### 3. `1.x`

It's an alias of `main` branch, which is intended for a smooth transition from the compatibility period. It will be removed in mid 2023.

```{note}
The branches mapping has been changed in 2023.04.06. For the legacy branches mapping and the guide for migration, please refer to the [branch migration guide](../migration/branches.md).
```

## supported platform

* Y = known work
* ? = shall work, not confirmed
* / = not applied

|    |windows|linux|android|mac|ios|
|---|---|---|---|---|---|
|intel|Y|Y|?|?|/|
|amd|Y|Y|/|?|/|
|nvidia|Y|Y|?|/|/|
|qcom|/|/|Y|/|/|
|apple|/|/|/|?|Y|
|arm|/|?|?|/|/|

## enable vulkan compute support
```
$ sudo dnf install vulkan-devel
$ cmake -DNCNN_VULKAN=ON ..
```

## enable vulkan compute inference
```cpp
net.opt.use_vulkan_compute = 1;
```

## proper allocator usage
```cpp
ncnn::VkAllocator* blob_vkallocator = vkdev.acquire_blob_allocator();
ncnn::VkAllocator* staging_vkallocator = vkdev.acquire_blob_allocator();

net.opt.blob_vkallocator = blob_vkallocator;
net.opt.workspace_vkallocator = blob_vkallocator;
net.opt.staging_vkallocator = staging_vkallocator;

// ....

// after inference
vkdev.reclaim_blob_allocator(blob_vkallocator);
vkdev.reclaim_staging_allocator(staging_vkallocator);
```

## select gpu device
```cpp
// get gpu count
int gpu_count = ncnn::get_gpu_count();

```

```cpp

// set specified vulkan device before loading param and model
net.set_vulkan_device(0); // use device-0
net.set_vulkan_device(1); // use device-1
```

## zero-copy on unified memory device
```cpp
ncnn::VkMat blob_gpu;
ncnn::Mat mapped = blob_gpu.mapped();

// use mapped.data directly
```

## hybrid cpu/gpu inference
```cpp
ncnn::Extractor ex_cpu = net.create_extractor();
ncnn::Extractor ex_gpu = net.create_extractor();
ex_cpu.set_vulkan_compute(false);
ex_gpu.set_vulkan_compute(true);

#pragma omp parallel sections
{
    #pragma omp section
    {
        ex_cpu.input();
        ex_cpu.extract();
    }
    {
        ex_gpu.input();
        ex_gpu.extract();
    }
}
```

## zero-copy gpu inference chaining
```cpp
ncnn::Extractor ex1 = net1.create_extractor();

}
    {
    }
}
```

```cpp
ncnn::Extractor ex2 = net2.create_extractor();

ncnn::VkCompute cmd(&vkdev);

ncnn::VkMat conv1;
ncnn::VkMat conv2;
ncnn::VkMat conv3;

ex1.input("conv1", conv1);
ex1.extract("conv2", conv2, cmd);

ex2.input("conv2", conv2);
ex2.extract("conv3", conv3, cmd);

cmd.submit();

cmd.wait();

```

## batch inference
```cpp
int max_batch_size = vkdev->info.compute_queue_count;

ncnn::Mat inputs[1000];
ncnn::Mat outputs[1000];

#pragma omp parallel for num_threads(max_batch_size)
for (int i=0; i<1000; i++)
{
    ncnn::Extractor ex = net1.create_extractor();
    ex.input("data", inputs[i]);
    ex.extract("prob", outputs[i]);
}
```

## control storage and arithmetic precision

disable all lower-precision optimizations, get full fp32 precision

```cpp
net.opt.use_fp16_packed = false;

}
```

```cpp
net.opt.use_fp16_storage = false;
net.opt.use_fp16_arithmetic = false;
net.opt.use_int8_storage = false;
net.opt.use_int8_arithmetic = false;
```

## debugging tips
```cpp
#define ENABLE_VALIDATION_LAYER 1 // modify to 1 in gpu.cpp
```

## add vulkan compute support to layer
1. add vulkan shader in src/layer/shader/

2. upload model weight data in Layer::upload_model()

3. setup pipeline in Layer::create_pipeline()

4. destroy pipeline in Layer::destroy_pipeline()

5. record command in Layer::forward()

## add optimized shader path
1. add vulkan shader in src/layer/shader/ named XXX_abc.comp

2. create pipeline with "XXX_abc"

3. record command using XXX_abc pipeline

## low-level op api
1. create layer

2. load param and load model

3. upload model

4. create pipeline

5. new command

6. record

6. record

7. submit and wait

# PaStaNet: Toward Human Activity Knowledge Engine

<details>
<summary align="right"><a href="https://arxiv.org/abs/2004.00945">Halpe (CVPR'2020)</a></summary>

```bibtex
@inproceedings{li2020pastanet,
  title={PaStaNet: Toward Human Activity Knowledge Engine},
  author={Li, Yong-Lu and Xu, Liang and Liu, Xinpeng and Huang, Xijie and Xu, Yue and Wang, Shiyi and Fang, Hao-Shu and Ma, Ze and Chen, Mingyang and Lu, Cewu},
}
```

# Seesaw Loss

> [Seesaw Loss for Long-Tailed Instance Segmentation](https://arxiv.org/abs/2008.10032)

Instance segmentation has witnessed a remarkable progress on class-balanced benchmarks. However, they fail to perform as accurately in real-world scenarios, where the category distribution of objects naturally comes with a long tail. Instances of head classes dominate a long-tailed dataset and they serve as negative samples of tail categories. The overwhelming gradients of negative samples on tail classes lead to a biased learning process for classifiers. Consequently, objects of tail categories are more likely to be misclassified as backgrounds or head categories. To tackle this problem, we propose Seesaw Loss to dynamically re-balance gradients of positive and negative samples for each category, with two complementary factors, i.e., mitigation factor and compensation factor. The mitigation factor reduces punishments to tail categories w.r.t. the ratio of cumulative training instances between different categories. Meanwhile, the compensation factor increases the penalty of misclassified instances to avoid false positives of tail categories. We conduct extensive experiments on Seesaw Loss with mainstream frameworks and different data sampling strategies. With a simple end-to-end training pipeline, Seesaw Loss obtains significant gains over Cross-Entropy Loss, and achieves state-of-the-art performance on LVIS dataset without bells and whistles.

<img src="https://user-images.githubusercontent.com/40661020/143974715-d181abe5-d0a2-40d3-a2bd-17d8c60b89b8.png"/>
</div>

- Please setup [LVIS dataset](../lvis/README.md) for MMDetection.

- RFS indicates to use oversample strategy [here](../../docs/tutorials/customipredataset.md#class-balanced-dataset) with oversample threshold `1e-3`.

## Results and models of Seasaw Loss on LVIS v1 dataset

|       Method       | Backbone  |  Style  | Lr schd | Data Sampler | Norm Mask | box AP | mask AP |                                           Config                                           |                                                                                                                                                              Download                                                                                                                                                              |

|     Mask R-CNN     | R-50-FPN  | pytorch |   2x    |    random    |     N     |  25.6  |  25.0   |             [config](./mask-rcnn_r50_fpn_seesaw-loss_random-ms-2x_lvis-v1.py)              |                          [model](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/mask_rcnn_r50_fpn_random_seesaw_loss_mstrain_2x_lvis_v1-a698dd3d.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/mask_rcnn_r50_fpn_random_seesaw_loss_mstrain_2x_lvis_v1.log.json)                          |

|     Mask R-CNN     | R-50-FPN  | pytorch |   2x    |    random    |     Y     |  25.6  |  25.4   |       [config](./mask-rcnn_r50_fpn_seesaw-loss-normed-mask_random-ms-2x_lvis-v1.py)        |              [model](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/mask_rcnn_r50_fpn_random_seesaw_loss_normed_mask_mstrain_2x_lvis_v1-a1c11314.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/mask_rcnn_r50_fpn_random_seesaw_loss_normed_mask_mstrain_2x_lvis_v1.log.json)              |

|     Mask R-CNN     | R-101-FPN | pytorch |   2x    |    random    |     N     |  27.4  |  26.7   |             [config](./mask-rcnn_r101_fpn_seesaw-loss_random-ms-2x_lvis-v1.py)             |                         [model](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/mask_rcnn_r101_fpn_random_seesaw_loss_mstrain_2x_lvis_v1-8e6e6dd5.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/mask_rcnn_r101_fpn_random_seesaw_loss_mstrain_2x_lvis_v1.log.json)                         |

|     Mask R-CNN     | R-101-FPN | pytorch |   2x    |    random    |     Y     |  27.2  |  27.3   |       [config](./mask-rcnn_r101_fpn_seesaw-loss-normed-mask_random-ms-2x_lvis-v1.py)       |             [model](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/mask_rcnn_r101_fpn_random_seesaw_loss_normed_mask_mstrain_2x_lvis_v1-a0b59c42.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/mask_rcnn_r101_fpn_random_seesaw_loss_normed_mask_mstrain_2x_lvis_v1.log.json)             |

|     Mask R-CNN     | R-50-FPN  | pytorch |   2x    |     RFS      |     N     |  27.6  |  26.4   |           [config](./mask-rcnn_r50_fpn_seesaw-loss_sample1e-3-ms-2x_lvis-v1.py)            |                      [model](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/mask_rcnn_r50_fpn_sample1e-3_seesaw_loss_mstrain_2x_lvis_v1-392a804b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/mask_rcnn_r50_fpn_sample1e-3_seesaw_loss_mstrain_2x_lvis_v1.log.json)                      |

|     Mask R-CNN     | R-50-FPN  | pytorch |   2x    |     RFS      |     Y     |  27.6  |  26.8   |     [config](./mask-rcnn_r50_fpn_seesaw-loss-normed-mask_sample1e-3-ms-2x_lvis-v1.py)      |          [model](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/mask_rcnn_r50_fpn_sample1e-3_seesaw_loss_normed_mask_mstrain_2x_lvis_v1-cd0f6a12.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/mask_rcnn_r50_fpn_sample1e-3_seesaw_loss_normed_mask_mstrain_2x_lvis_v1.log.json)          |

|     Mask R-CNN     | R-101-FPN | pytorch |   2x    |     RFS      |     N     |  28.9  |  27.6   |           [config](./mask-rcnn_r101_fpn_seesaw-loss_sample1e-3-ms-2x_lvis-v1.py)           |                     [model](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/mask_rcnn_r101_fpn_sample1e-3_seesaw_loss_mstrain_2x_lvis_v1-e68eb464.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/mask_rcnn_r101_fpn_sample1e-3_seesaw_loss_mstrain_2x_lvis_v1.log.json)                     |

|     Mask R-CNN     | R-101-FPN | pytorch |   2x    |     RFS      |     Y     |  28.9  |  28.2   |     [config](./mask-rcnn_r101_fpn_seesaw-loss-normed-mask_sample1e-3-ms-2x_lvis-v1.py)     |         [model](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/mask_rcnn_r101_fpn_sample1e-3_seesaw_loss_normed_mask_mstrain_2x_lvis_v1-1d817139.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/mask_rcnn_r101_fpn_sample1e-3_seesaw_loss_normed_mask_mstrain_2x_lvis_v1.log.json)         |

| Cascade Mask R-CNN | R-101-FPN | pytorch |   2x    |    random    |     N     |  33.1  |  29.2   |         [config](./cascade-mask-rcnn_r101_fpn_seesaw-loss_random-ms-2x_lvis-v1.py)         |                 [model](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/cascade_mask_rcnn_r101_fpn_random_seesaw_loss_mstrain_2x_lvis_v1-71e2215e.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/cascade_mask_rcnn_r101_fpn_random_seesaw_loss_mstrain_2x_lvis_v1.log.json)                 |

| Cascade Mask R-CNN | R-101-FPN | pytorch |   2x    |    random    |     Y     |  33.0  |  30.0   |   [config](./cascade-mask-rcnn_r101_fpn_seesaw-loss-normed-mask_random-ms-2x_lvis-v1.py)   |     [model](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/cascade_mask_rcnn_r101_fpn_random_seesaw_loss_normed_mask_mstrain_2x_lvis_v1-8b5a6745.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/cascade_mask_rcnn_r101_fpn_random_seesaw_loss_normed_mask_mstrain_2x_lvis_v1.log.json)     |

| Cascade Mask R-CNN | R-101-FPN | pytorch |   2x    |     RFS      |     N     |  30.0  |  29.3   |       [config](./cascade-mask-rcnn_r101_fpn_seesaw-loss_sample1e-3-ms-2x_lvis-v1.py)       |             [model](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/cascade_mask_rcnn_r101_fpn_sample1e-3_seesaw_loss_mstrain_2x_lvis_v1-5d8ca2a4.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/cascade_mask_rcnn_r101_fpn_sample1e-3_seesaw_loss_mstrain_2x_lvis_v1.log.json)             |

| Cascade Mask R-CNN | R-101-FPN | pytorch |   2x    |     RFS      |     Y     |  32.8  |  30.1   | [config](./cascade-mask-rcnn_r101_fpn_seesaw-loss-normed-mask_sample1e-3-ms-2x_lvis-v1.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/cascade_mask_rcnn_r101_fpn_sample1e-3_seesaw_loss_normed_mask_mstrain_2x_lvis_v1-c8551505.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/seesaw_loss/cascade_mask_rcnn_r101_fpn_sample1e-3_seesaw_loss_normed_mask_mstrain_2x_lvis_v1.log.json) |

We provide config files to reproduce the instance segmentation performance in the CVPR 2021 paper for [Seesaw Loss for Long-Tailed Instance Segmentation](https://arxiv.org/abs/2008.10032).

```latex
@inproceedings{wang2021seesaw,
  title={Seesaw Loss for Long-Tailed Instance Segmentation},
  author={Jiaqi Wang and Wenwei Zhang and Yuhang Zang and Yuhang Cao and Jiangmiao Pang and Tao Gong and Kai Chen and Ziwei Liu and Chen Change Loy and Dahua Lin},

author={Jiaqi Wang and Wenwei Zhang and Yuhang Zang and Yuhang Cao and Jiangmiao Pang and Tao Gong and Kai Chen and Ziwei Liu and Chen Change Loy and Dahua Lin},
  booktitle={Proceedings of the {IEEE} Conference on Computer Vision and Pattern Recognition},
}
```

---
name: "\U0001F4C8 quantization"
about: best wishes for your low bit quantization has a low accuracy loss...\(^▽^)/...2333... 
---

## expectation | 诉求 | 期待する
1. speed 
2. precision

1. model.param and model.bin

# 准备 Denoising 数据集

```bibtex
}
```

测试数据集(Set12, BSD68, CBSD68, Kodak, McMaster, Urban100)可以从 [此处](https://drive.google.com/file/d/1P_-RAvltEoEhfT-9GrWRdpEi6NSswTs8/) 下载。

```text
mmagic
├── mmagic
├── tools
├── data
```

## Pre-trained Detection Models

### 人体 Bounding Box 检测模型

MMDetection 提供了基于 COCO 的包括 `person` 在内的 80 个类别的预训练模型，用户可前往 [MMDetection Model Zoo](https://mmdetection.readthedocs.io/zh_CN/3.x/model_zoo.html) 下载并将其用作人体 bounding box 识别模型。

### 手部 Bounding Box 检测模型

对于手部 bounding box 检测模型，我们提供了一个通过 MMDetection 基于 OneHand10K 数据库训练的模型。

#### 基于 OneHand10K 测试集的测试结果

| Arch                                                              | Box AP |                               ckpt                                |                               log                                |
| :---------------------------------------------------------------- | :----: | :---------------------------------------------------------------: | :--------------------------------------------------------------: |

| [Cascade_R-CNN X-101-64x4d-FPN-1class](/demo/mmdetection_cfg/cascade_rcnn_x101_64x4d_fpn_1class.py) | 0.817  | [ckpt](https://download.openmmlab.com/mmpose/mmdet_pretrained/cascade_rcnn_x101_64x4d_fpn_20e_onehand10k-dac19597_20201030.pth) | [log](https://download.openmmlab.com/mmpose/mmdet_pretrained/cascade_rcnn_x101_64x4d_fpn_20e_onehand10k_20201030.log.json) |

### 脸部 Bounding Box 检测模型

对于脸部 bounding box 检测模型，我们提供了一个通过 MMDetection 基于 COCO-Face 数据库训练的 YOLOX 检测器。

#### 基于 COCO-face 测试集的测试结果

| Arch                                                            | Box AP |                                                  ckpt                                                  |
| :-------------------------------------------------------------- | :----: | :----------------------------------------------------------------------------------------------------: |

| [YOLOX-s](/demo/mmdetection_cfg/yolox-s_8xb8-300e_coco-face.py) | 0.408  | [ckpt](https://download.openmmlab.com/mmpose/mmdet_pretrained/yolo-x_8xb8-300e_coco-face_13274d7c.pth) |

### 动物 Bounding Box 检测模型

#### COCO animals

COCO 数据集内包括了 10 种常见的 `animal` 类型：

(14: 'bird', 15: 'cat', 16: 'dog', 17: 'horse', 18: 'sheep', 19: 'cow', 20: 'elephant', 21: 'bear', 22: 'zebra', 23: 'giraffe') 。

用户如果需要使用以上类别的动物检测模型，可以前往 [MMDetection Model Zoo](https://mmdetection.readthedocs.io/zh_CN/3.x/model_zoo.html) 下载。

#### 基于 MacaquePose 测试集的测试结果

| [Faster_R-CNN_Res50-FPN-1class](/demo/mmdetection_cfg/faster_rcnn_r50_fpn_1class.py) | 0.840  | [ckpt](https://download.openmmlab.com/mmpose/mmdet_pretrained/faster_rcnn_r50_fpn_1x_macaque-f64f2812_20210409.pth) | [log](https://download.openmmlab.com/mmpose/mmdet_pretrained/faster_rcnn_r50_fpn_1x_macaque_20210409.log.json) |
| [Cascade_R-CNN X-101-64x4d-FPN-1class](/demo/mmdetection_cfg/cascade_rcnn_x101_64x4d_fpn_1class.py) | 0.879  | [ckpt](https://download.openmmlab.com/mmpose/mmdet_pretrained/cascade_rcnn_x101_64x4d_fpn_20e_macaque-e45e36f5_20210409.pth) | [log](https://download.openmmlab.com/mmpose/mmdet_pretrained/cascade_rcnn_x101_64x4d_fpn_20e_macaque_20210409.log.json) |

# Customize Models (To be updated)

- backbone: usually an FCN network to extract feature maps, e.g., ResNet, Swin.
- neck: the component between backbones and heads, e.g., FPN, ReFPN.
- head: the component for specific tasks, e.g., bbox prediction.
- roi extractor: the part for extracting RoI features from feature maps, e.g., RoI Align Rotated.
- loss: the component in head for calculating losses, e.g., FocalLoss, GWDLoss, and KFIoULoss.

Create a new file `mmrotate/models/backbones/mobilenet.py`.

```python

from mmrotate.models.builder import ROTATED_BACKBONES

@ROTATED_BACKBONES.register_module()

        pass

        pass

        pass
```

You can either add the following line to `mmrotate/models/backbones/__init__.py`

```python
```

```python
    imports=['mmrotate.models.backbones.mobilenet'],
```

```python
    ...
        arg1=xxx,
    ...
```

Create a new file `mmrotate/models/necks/pafpn.py`.

```python
from mmrotate.models.builder import ROTATED_NECKS

@ROTATED_NECKS.register_module()

```python

                num_outs,
        pass

        pass
```

You can either add the following line to `mmrotate/models/necks/__init__.py`,

```python
```

```python
    imports=['mmrotate.models.necks.pafpn.py'],
```

```python
```

```python
```

First, add a new bbox head in `mmrotate/models/roi_heads/bbox_heads/double_bbox_head.py`.

```python
from mmrotate.models.builder import ROTATED_HEADS
from mmrotate.models.roi_heads.bbox_heads.bbox_head import BBoxHead

@ROTATED_HEADS.register_module()

                                      /-> cls
                  /-> shared convs ->
                                      \-> reg
    roi features

r"""Bbox head used in Double-Head R-CNN

                                      /-> cls
                                      \-> reg
                                      /-> cls
                  \-> shared fc    ->
                                      \-> reg

                 num_convs=0,
                 num_fcs=0,
                 conv_out_channels=1024,
                 fc_out_channels=1024,
                 conv_cfg=None,
                 norm_cfg=dict(type='BN'),
        super(DoubleConvFCBBoxHead, self).__init__(**kwargs)

    def forward(self, x_cls, x_reg):

```

```python

```python

from mmdet.core import bbox2result, bbox2roi, build_assigner, build_sampler
from mmrotate.models.builder import ROTATED_HEADS, build_head, build_roi_extractor
from mmrotate.models.roi_heads.base_roi_head import BaseRoIHead
from mmrotate.models.roi_heads.test_mixins import BBoxTestMixin, MaskTestMixin

class StandardRoIHead(BaseRoIHead, BBoxTestMixin, MaskTestMixin):
    """Simplest base roi head including one bbox head and one mask head.
    """

    def init_bbox_head(self, bbox_roi_extractor, bbox_head):

    def forward_dummy(self, x, proposals):

    def forward_train(self,
                      x,
                      img_metas,
                      proposal_list,
                      gt_bboxes,
                      gt_labels,
                      gt_bboxes_ignore=None,
                      gt_masks=None):

    def _bbox_forward(self, x, rois):

gt_bboxes,

    def _bbox_forward_train(self, x, sampling_results, gt_bboxes, gt_labels,
                            img_metas):

    def simple_test(self,
                    x,
                    proposal_list,
                    proposals=None,
                    rescale=False):
        """Test without augmentation."""

```

Double Head's modification is mainly in the bbox_forward logic, and it inherits other logics from the `StandardRoIHead`.
In the `mmrotate/models/roi_heads/double_roi_head.py`, we implement the new RoI Head as the following:

```python
from mmrotate.models.roi_heads.standard_roi_head import StandardRoIHead

    """RoI head for Double Head RCNN

    https://arxiv.org/abs/1904.06493
    """

    def __init__(self, reg_roi_scale_factor, **kwargs):
        super(DoubleHeadRoIHead, self).__init__(**kwargs)

            rois,

if self.with_shared_head:

```

`mmrotate/models/bbox_heads/__init__.py` and `mmrotate/models/roi_heads/__init__.py` thus the corresponding registry could find and load them.

```python
    imports=['mmrotate.models.roi_heads.double_roi_head', 'mmrotate.models.bbox_heads.double_bbox_head'])
```

```

To add a new loss function, the users need implement it in `mmrotate/models/losses/my_loss.py`.

```python

from mmrotate.models.builder import ROTATED_LOSSES

@ROTATED_LOSSES.register_module()

                pred,
                target,

self.loss_weight = loss_weight

                pred,
                target,
```

Then the users need to add it in the `mmrotate/models/losses/__init__.py`.

```python

```

```python
    imports=['mmrotate.models.losses.my_loss'])
```

```python

```

```python
```

# 教程2：准备数据集

我们建议将数据集根目录符号链接到 `$MMSEGMENTATION/data`。
如果您的目录结构不同，您可能需要更改配置文件中相应的路径。
对于中国境内的用户，我们也推荐通过开源数据平台 [OpenDataLab](https://opendatalab.com/) 来下载dsdl标准数据，以获得更好的下载和使用体验，这里有一个下载dsdl数据集并进行训练的案例[DSDLReadme](../../../configs/dsdl/README.md)，欢迎尝试。

```none
├── mmseg
├── tools
├── data
│   │   │   ├── val
│   │   │   ├── val
│   │   │   ├── JPEGImages
│   │   │   ├── SegmentationClass
│   │   │   │   ├── Segmentation
│   │   ├── VOC2010
│   │   │   ├── SegmentationClassContext
│   │   │   │   ├── SegmentationContext
│   │   │   │   │   ├── train.txt
│   │   │   │   │   ├── val.txt
│   │   │   ├── trainval_merged.json
│   │   ├── VOCaug
│   │   │   ├── dataset
│   │   │   │   ├── cls
│   ├── ade
│   │   ├── ADEChallengeData2016

│   ├── ade
│   │   │   ├── annotations
│   │   │   │   ├── training
│   │   │   │   ├── validation
│   │   │   ├── images
│   ├── coco_stuff10k
│   │   │   ├── train2014
│   │   │   ├── test2014
│   │   ├── imagesLists
│   │   │   ├── test.txt
│   │   │   ├── all.txt
│   ├── coco_stuff164k
│   │   │   ├── train2017
│   │   │   ├── val2017
│   ├── CHASE_DB1
│   ├── DRIVE

│   ├── DRIVE
│   ├── HRF
│   ├── STARE
|   ├── dark_zurich
|   │   ├── gps
|   │   │   ├── val
|   │   │   └── val_ref
|   │   ├── gt
|   │   │   └── val
|   │   ├── LICENSE.txt
|   │   ├── lists_file_names
|   │   │   ├── val_filenames.txt
|   │   │   └── val_ref_filenames.txt
|   │   ├── README.md
|   │   └── rgb_anon
|   │   |   ├── val
|   │   |   └── val_ref

|   │   |   ├── val
|   ├── NighttimeDrivingTest
|   |   ├── gtCoarse_daytime_trainvaltest
|   |   │   └── test
|   |   │       └── night
|   |   └── leftImg8bit
|   |   |   └── test
|   |   |       └── night
│   ├── loveDA
│   │   │   ├── val
│   │   │   ├── test
│   │   │   ├── val
│   ├── potsdam
│   │   │   ├── val
│   │   │   ├── val
│   ├── vaihingen
│   │   │   ├── val
│   │   │   ├── val
│   ├── iSAID
│   │   │   ├── val
│   │   │   ├── val
│   ├── synapse

│   │   │   ├── val
│   ├── iSAID
│   │   │   ├── val
│   │   │   ├── val
│   │   │   ├── val
│   │   │   ├── val
│   ├── REFUGE
│   ├── mapillary
│   │   │   ├── v1.2
|   │   │   │   ├── instances
|   │   │   │   ├── labels
|   │   │   │   └── panoptic
│   │   │   ├── v2.0
|   │   │   │   ├── panoptic
|   │   │   │   └── polygons
│   │   ├── validation
|   │   │   ├── v1.2

│   ├── bdd100k
│   │   │   └── 10k
|   │   │   │   ├── test
|   │   │   │   └── val
│   │   └── labels
│   │   │   └── sem_seg
|   │   │   │   ├── colormaps
|   │   │   │   │   ├──train
|   │   │   │   │   └──val
|   │   │   │   ├── masks
|   │   │   │   ├── polygons
|   │   │   │   │   ├──sem_seg_train.json
|   │   │   │   │   └──sem_seg_val.json
|   │   │   │   └── rles
```

## Cityscapes

```

Cityscapes [官方网站](https://www.cityscapes-dataset.com/)可以下载 Cityscapes 数据集，按照官网要求注册并登陆后，数据可以在[这里](https://www.cityscapes-dataset.com/downloads/)找到。

按照惯例，`**labelTrainIds.png` 用于 cityscapes 训练。
我们提供了一个基于 [cityscapesscripts](https://github.com/mcordts/cityscapesScripts) 的[脚本](https://github.com/open-mmlab/mmsegmentation/blob/1.x/tools/dataset_converters/cityscapes.py)用于生成 `**labelTrainIds.png`。

```shell
# --nproc 表示 8 个转换进程，也可以省略。
python tools/dataset_converters/cityscapes.py data/cityscapes --nproc 8
```

## Pascal VOC

Pascal VOC 2012 可从[此处](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar)下载。
此外，Pascal VOC 数据集的最新工作通常利用额外的增强数据，可以在[这里](http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/semantic_contours/benchmark.tgz)找到。

如果您想使用增强的 VOC 数据集，请运行以下命令将增强数据的标注转换为正确的格式。

```shell

```shell
python tools/dataset_converters/voc_aug.py data/VOCdevkit data/VOCdevkit/VOCaug --nproc 8
```

请参考[拼接数据集文档](../advanced_guides/add_datasets.md#拼接数据集)及 [voc_aug 配置示例](../../../configs/_base_/datasets/pascal_voc12_aug.py)以详细了解如何将它们拼接并合并训练。

## ADE20K

ADE20K 的训练和验证集可以从这个[链接](http://data.csail.mit.edu/places/ADEchallenge/ADEChallengeData2016.zip)下载。
如果需要下载测试数据集，可以在[官网](http://host.robots.ox.ac.uk/)注册后，下载[测试集](http://host.robots.ox.ac.uk:8080/eval/downloads/VOC2010test.tar)。

## Pascal Context

Pascal Context 的训练和验证集可以从[此处](http://host.robots.ox.ac.uk/pascal/VOC/voc2010/VOCtrainval_03-May-2010.tar)下载。注册后，您也可以从[此处](http://host.robots.ox.ac.uk:8080/eval/downloads/VOC2010test.tar)下载测试集。

从原始数据集中抽出部分数据作为验证集，您可以从[此处](https://codalabuser.blob.core.windows.net/public/trainval_merged.json)下载 trainval_merged.json 文件。

请先安装 [Detail](https://github.com/zhanghang1989/detail-api) 工具然后运行以下命令将标注转换为正确的格式。

```shell
python tools/dataset_converters/pascal_context.py data/VOCdevkit data/VOCdevkit/VOC2010/trainval_merged.json
```

## COCO Stuff 10k

数据可以通过 wget 在[这里](http://calvin.inf.ed.ac.uk/wp-content/uploads/data/cocostuffdataset/cocostuff-10k-v1.1.zip)下载。

对于 COCO Stuff 10k 数据集，请运行以下命令下载并转换数据集。

```shell
# 下载
mkdir coco_stuff10k && cd coco_stuff10k
wget http://calvin.inf.ed.ac.uk/wp-content/uploads/data/cocostuffdataset/cocostuff-10k-v1.1.zip

# 解压
unzip cocostuff-10k-v1.1.zip

python tools/dataset_converters/coco_stuff10k.py /path/to/coco_stuff10k --nproc 8
```

# 解压

```

按照惯例，`/path/to/coco_stuff164k/annotations/*2014/*_labelTrainIds.png` 中的 mask 标注用于 COCO Stuff 10k 的训练和测试。

## COCO Stuff 164k

对于 COCO Stuff 164k 数据集，请运行以下命令下载并转换增强的数据集。

```shell
# 下载
mkdir coco_stuff164k && cd coco_stuff164k
wget http://images.cocodataset.org/zips/train2017.zip
wget http://images.cocodataset.org/zips/val2017.zip
wget http://calvin.inf.ed.ac.uk/wp-content/uploads/data/cocostuffdataset/stuffthingmaps_trainval2017.zip

# 解压
unzip train2017.zip -d images/
unzip val2017.zip -d images/
unzip stuffthingmaps_trainval2017.zip -d annotations/

python tools/dataset_converters/coco_stuff164k.py /path/to/coco_stuff164k --nproc 8
```

按照惯例，`/path/to/coco_stuff164k/annotations/*2017/*_labelTrainIds.png` 中的 mask 标注用于 COCO Stuff 164k 的训练和测试。

```

此数据集的详细信息可在[此处](https://github.com/nightrome/cocostuff#downloads)找到。

## CHASE DB1

CHASE DB1 的训练和验证集可以从[此处](https://staffnet.kingston.ac.uk/~ku15565/CHASE_DB1/assets/CHASEDB1.zip)下载。

请运行以下命令，准备 CHASE DB1 数据集：

```shell
python tools/dataset_converters/chase_db1.py /path/to/CHASEDB1.zip
```

该脚本将自动调整数据集目录结构，使其满足 MMSegmentation 数据集加载要求。

## DRIVE

按照[官网](https://drive.grand-challenge.org/)要求，注册并登陆后，便可以下载 DRIVE 的训练和验证数据集。

要将 DRIVE 数据集转换为 MMSegmentation 的格式，请运行以下命令：

```shell
python tools/dataset_converters/drive.py /path/to/training.zip /path/to/test.zip
```

## HRF

```shell
```

## HRF

请下载 [health.zip](https://www5.cs.fau.de/fileadmin/research/datasets/fundus-images/healthy.zip)、[glaucoma.zip](https://www5.cs.fau.de/fileadmin/research/datasets/fundus-images/glaucoma.zip)、[diabetic_retinopathy.zip](https://www5.cs.fau.de/fileadmin/research/datasets/fundus-images/diabetic_retinopathy.zip)、[healthy_manualsegm.zip](https://www5.cs.fau.de/fileadmin/research/datasets/fundus-images/healthy_manualsegm.zip)、[glaucoma_manualsegm.zip](https://www5.cs.fau.de/fileadmin/research/datasets/fundus-images/glaucoma_manualsegm.zip) 和 [diabetic_retinopathy_manualsegm.zip](https://www5.cs.fau.de/fileadmin/research/datasets/fundus-images/diabetic_retinopathy_manualsegm.zip)，无需解压，可以直接运行以下命令，准备 HRF 数据集：

```shell

```shell
python tools/dataset_converters/hrf.py /path/to/healthy.zip /path/to/healthy_manualsegm.zip /path/to/glaucoma.zip /path/to/glaucoma_manualsegm.zip /path/to/diabetic_retinopathy.zip /path/to/diabetic_retinopathy_manualsegm.zip
```

## STARE

请下载 [stare images.tar](http://cecas.clemson.edu/~ahoover/stare/probing/stare-images.tar)、[labels-ah.tar](http://cecas.clemson.edu/~ahoover/stare/probing/labels-ah.tar) 和 [labels-vk.tar](http://cecas.clemson.edu/~ahoover/stare/probing/labels-vk.tar)，无需解压，可以直接运行以下命令，准备 STARE 数据集：

```shell
python tools/dataset_converters/stare.py /path/to/stare-images.tar /path/to/labels-ah.tar /path/to/labels-vk.tar
```

## Dark Zurich

由于我们只支持在此数据集上的模型测试，因此您只需要下载并解压[验证数据集](https://data.vision.ee.ethz.ch/csakarid/shared/GCMA_UIoU/Dark_Zurich_val_anon.zip)。

## Nighttime Driving

由于我们只支持在此数据集上的模型测试，因此您只需要下载并解压[验证数据集](http://data.vision.ee.ethz.ch/daid/NighttimeDriving/NighttimeDrivingTest.zip)。

## LoveDA

数据可以从[此处](https://drive.google.com/drive/folders/1ibYV0qwn4yuuh068Rnc-w4tPi0U0c-ti?usp=sharing)下载 LaveDA 数据集。

或者可以从 [zenodo](https://zenodo.org/record/5706578#.YZvN7SYRXdF) 下载。下载后，无需解压，直接运行以下命令：

```shell
# 下载 Train.zip
wget https://zenodo.org/record/5706578/files/Train.zip
# 下载 Val.zip
wget https://zenodo.org/record/5706578/files/Val.zip
# 下载 Test.zip
wget https://zenodo.org/record/5706578/files/Test.zip
```

请对于 LoveDA 数据集，请运行以下命令调整数据集目录。

```shell
python tools/dataset_converters/loveda.py /path/to/loveDA
```

可将模型对 LoveDA 的测试集的预测结果上传至到数据集[测试服务器](https://codalab.lisn.upsaclay.fr/competitions/421)，查看评测结果。

有关 LoveDA 的更多详细信息，可查看[此处](https://github.com/Junjue-Wang/LoveDA).

## ISPRS Potsdam

```

[Potsdam](https://www.isprs.org/education/benchmarks/UrbanSemLab/2d-sem-label-potsdam.aspx) 城市语义分割数据集用于 2D 语义分割竞赛 —— Potsdam。

数据集可以在竞赛[主页](https://www.isprs.org/education/benchmarks/UrbanSemLab/default.aspx)上请求获得。
这里也提供了[BaiduNetdisk](https://pan.baidu.com/s/1K-cLVZnd1X7d8c26FQ-nGg?pwd=mseg)，提取码：mseg、 [Google Drive](https://drive.google.com/drive/folders/1w3EJuyUGet6_qmLwGAWZ9vw5ogeG0zLz?usp=sharing)以及[OpenDataLab](https://opendatalab.com/ISPRS_Potsdam/download)。
实验中需要下载 '2_Ortho_RGB.zip' 和 '5_Labels_all_noBoundary.zip'。

对于 Potsdam 数据集，请运行以下命令调整数据集目录。

```shell
python tools/dataset_converters/potsdam.py /path/to/potsdam
```

在我们的默认设置中，将生成 3456 张图像用于训练和 2016 张图像用于验证。

## ISPRS Vaihingen

```shell
```

[Vaihingen](https://www.isprs.org/education/benchmarks/UrbanSemLab/2d-sem-label-vaihingen.aspx) 城市语义分割数据集用于 2D 语义分割竞赛 —— Vaihingen。

这里也提供了[BaiduNetdisk](https://pan.baidu.com/s/109D3WLrLafsuYtLeerLiiA?pwd=mseg)，提取码：mseg 、 [Google Drive](https://drive.google.com/drive/folders/1w3NhvLVA2myVZqOn2pbiDXngNC7NTP_t?usp=sharing)。
实验中需要下载 'ISPRS_semantic_labeling_Vaihingen.zip' 和 'ISPRS_semantic_labeling_Vaihingen_ground_truth_eroded_COMPLETE.zip'。

对于 Vaihingen 数据集，请运行以下命令调整数据集目录。

```shell
python tools/dataset_converters/vaihingen.py /path/to/vaihingen
```

在我们的默认设置（`clip_size`=512, `stride_size`=256）中，将生成 344 张图像用于训练和 398 张图像用于验证。

## iSAID

iSAID 数据集可从 [DOTA-v1.0](https://captain-whu.github.io/DOTA/dataset.html) 下载训练/验证/测试数据集的图像数据，

```

## iSAID

并从 [iSAID](https://captain-whu.github.io/iSAID/dataset.html)下载训练/验证数据集的标注数据。

该数据集是航空图像实例分割和语义分割任务的大规模数据集。

下载 iSAID 数据集后，您可能需要按照以下结构进行数据集准备。

```none
├── data
│   ├── iSAID
│   │   │   │   ├── part1.zip
│   │   │   │   ├── part2.zip
│   │   │   │   ├── part3.zip
│   │   │   ├── Semantic_masks
│   │   │   │   ├── images.zip
│   │   ├── val
│   │   ├── test
```

```shell
python tools/dataset_converters/isaid.py /path/to/iSAID
```

在我们的默认设置（`patch_width`=896, `patch_height`=896, `overlap_area`=384）中，将生成 33978 张图像用于训练和 11644 张图像用于验证。

## LIP(Look Into Person) dataset

```

该数据集可以从[此页面](https://lip.sysuhcp.com/overview.php)下载。

请运行以下命令来解压数据集。

```shell
unzip LIP.zip
cd LIP
unzip TrainVal_images.zip
unzip TrainVal_parsing_annotations.zip
cd TrainVal_parsing_annotations
mv train_segmentations ../
mv val_segmentations ../
cd ..
```

LIP 数据集的内容包括：

```none
├── data
│   ├── LIP
│   │   ├── train_images
│   │   │   ├── 1000_1234574.jpg
│   │   │   ├── ...
│   │   ├── train_segmentations
│   │   │   ├── 1000_1234574.png
│   │   │   ├── ...
│   │   ├── val_images
│   │   │   ├── 100034_483681.jpg
│   │   │   ├── ...
│   │   ├── val_segmentations
│   │   │   ├── 100034_483681.png
│   │   │   ├── ...
```

## Synapse dataset

此数据集可以从[此页面](https://www.synapse.org/#!Synapse:syn3193805/wiki/)下载。

│   │   │   ├── ...
│   │   │   ├── ...
```

遵循 [TransUNet](https://arxiv.org/abs/2102.04306) 的数据准备设定，将原始训练集（30 次扫描）拆分为新的训练集（18 次扫描）和验证集（12 次扫描）。请运行以下命令来准备数据集。

```shell
unzip RawData.zip
cd ./RawData/Training
```

然后创建 `train.txt` 和 `val.txt` 以拆分数据集。

根据 TransUnet，以下是数据集的划分。

train.txt

```none
img0005.nii.gz
img0006.nii.gz
img0007.nii.gz
img0009.nii.gz
img0010.nii.gz
img0021.nii.gz
img0023.nii.gz
img0024.nii.gz
img0026.nii.gz
img0027.nii.gz
img0028.nii.gz
img0030.nii.gz
img0031.nii.gz
img0033.nii.gz
img0034.nii.gz
img0037.nii.gz
img0039.nii.gz
img0040.nii.gz
```

val.txt

```none
img0008.nii.gz
img0022.nii.gz
img0038.nii.gz
img0036.nii.gz
img0032.nii.gz
img0002.nii.gz
img0029.nii.gz
img0003.nii.gz
img0001.nii.gz
img0004.nii.gz
img0025.nii.gz
img0035.nii.gz
```

synapse 数据集的内容包括：

```none
├── Training
│   ├── img

```

```none
│   ├── img
│   │   ├── img0001.nii.gz
│   │   ├── img0002.nii.gz
│   │   ├── ...
│   ├── label
│   │   ├── label0001.nii.gz
│   │   ├── label0002.nii.gz
│   │   ├── ...
```

然后，使用此命令转换 synapse 数据集。

```shell
python tools/dataset_converters/synapse.py --dataset-path /path/to/synapse
```

注意，MMSegmentation 的默认评估指标（例如 mean dice value）是在 2D 切片图像上计算的，这与 [TransUNet](https://arxiv.org/abs/2102.04306) 等一些论文中的 3D 扫描结果是不同的。

## REFUGE

在 [REFUGE Challenge](https://refuge.grand-challenge.org) 官网上注册并下载 [REFUGE 数据集](https://refuge.grand-challenge.org/REFUGE2Download)。

然后，解压 `REFUGE2.zip`，原始数据集的内容包括：

```none
├── REFUGE2
│   ├── REFUGE2
│   │   ├── Annotation-Training400.zip
│   │   ├── REFUGE-Test400.zip
│   │   ├── REFUGE-Test-GT.zip
│   │   ├── REFUGE-Training400.zip

```none
│   │   ├── REFUGE-Validation400.zip
│   │   ├── REFUGE-Validation400-GT.zip
│   ├── __MACOSX
```

请运行以下命令转换 REFUGE 数据集：

```shell
python tools/convert_datasets/refuge.py --raw_data_root=/path/to/refuge/REFUGE2/REFUGE2
```

脚本会将目录结构转换如下：

```none
```

包含 400 张用于训练的图像、400 张用于验证的图像和 400 张用于测试的图像，这与 REFUGE 2018 数据集相同。

## Mapillary Vistas Datasets

- Mapillary Vistas [官方网站](https://www.mapillary.com/dataset/vistas) 可以下载 Mapillary Vistas 数据集，按照官网要求注册并登陆后，数据可以在[这里](https://www.mapillary.com/dataset/vistas)找到。

- Mapillary Vistas 数据集使用 8-bit with color-palette 来存储标签。不需要进行转换操作。

- 假设您已将数据集 zip 文件放在 `mmsegmentation/data/mapillary` 中

- 请运行以下命令来解压数据集。

  ```bash
  cd data/mapillary
  unzip An-ZjB1Zm61yAZG0ozTymz8I8NqI4x0MrYrh26dq7kPgfu8vf9ImrdaOAVOFYbJ2pNAgUnVGBmbue9lTgdBOb5BbKXIpFs0fpYWqACbrQDChAA2fdX0zS9PcHu7fY8c-FOvyBVxPNYNFQuM.zip
  ```

- 解压后，您将获得类似于此结构的 Mapillary Vistas 数据集。语义分割 mask 标签在 `labels` 文件夹中。

  ```none
  ├── mmseg
  ├── tools
  ├── data
  |   │   │   ├── v1.2

  ```

- 您可以在配置中使用 `MapillaryDataset_v1` 和 `Mapillary Dataset_v2` 设置数据集版本。
  在此处 [V1.2](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/_base_/datasets/mapillary_v1.py) 和 [V2.0](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/_base_/datasets/mapillary_v2.py) 查看 Mapillary Vistas 数据集配置文件

## LEVIR-CD

[LEVIR-CD](https://justchenhao.github.io/LEVIR/) 大规模遥感建筑变化检测数据集。

数据集可以在[主页](https://justchenhao.github.io/LEVIR/)上请求获得。

数据集的补充版本可以在[主页](https://github.com/S2Looking/Dataset)上请求获得。

请下载数据集的补充版本，然后解压 `LEVIR-CD+.zip`，数据集的内容包括：

```none
│   ├── LEVIR-CD+
│   │   │   ├── A
│   │   │   ├── B
│   │   │   ├── label
│   │   ├── test

```none
│   │   │   ├── A
│   │   │   ├── B
│   │   ├── test
│   │   │   ├── A
│   │   │   ├── B
```

对于 LEVIR-CD 数据集，请运行以下命令无重叠裁剪影像：

```shell
python tools/dataset_converters/levircd.py --dataset-path /path/to/LEVIR-CD+ --out_dir /path/to/LEVIR-CD
```

裁剪后的影像大小为256x256，与原论文保持一致。

## BDD100K

- 可以从[官方网站](https://bdd-data.berkeley.edu/) 下载 BDD100K数据集（语义分割任务主要是10K数据集），按照官网要求注册并登陆后，数据可以在[这里](https://bdd-data.berkeley.edu/portal.html#download)找到。

- 图像数据对应的名称是是`10K Images`, 语义分割标注对应的名称是`Segmentation`

- 下载后，可以使用以下代码进行解压

  ```bash
  unzip ~/bdd100k_images_10k.zip -d ~/mmsegmentation/data/
  unzip ~/bdd100k_sem_seg_labels_trainval.zip -d ~/mmsegmentation/data/
  ```

就可以得到以下文件结构了：

```none
├── mmseg
├── tools
├── data
│   │   │   └── 10k

```

```none
├── mmseg
├── tools
├── data
│   │   │   └── 10k
```

# Protobuf 类问题解决方法

## 问题分析

protobuf 有关的报错，一般都是两个原因：

1. 需要的 pb 没安装/`FindProtobuf.cmake`不存在，最终 `find_package` 失败
2. 系统不止一套 pb，导致 bin/lib/include 三者不匹配

如果你遇到了这些报错，都可以通过本文档解决：

1. Linux 编译 `caffe2ncnn` 时报 `Protobuf not found`
2. 编译 `caffe2ncnn` 时报 protoc 和 protobuf.so 版本不匹配

## （推荐）通用处理办法

这个办法包治百病，**不管什么情况一定生效**

1. 编译下载 protobuf，以 3.20.0 版本为例

```bash
$ wget https://github.com/protocolbuffers/protobuf/releases/download/v3.20.0/protobuf-cpp-3.20.0.tar.gz
$ tar xvf protobuf-cpp-3.20.0.tar.gz
$ cd protobuf-3.20.0/
$ ./configure --prefix=/path/to/install
$ make && make install
```
注意需要 `--prefix`，不要装到系统里。能遇到这些错，说明本来系统环境就有问题，再给系统环境装 lib 就更乱了。

2. 修改 cmake

找到报错的 CMakeLists.txt，在 `find_package` 前插入 protobuf 路径。

```bash
# 加入下面 1 行
list(APPEND CMAKE_PREFIX_PATH "/path/to/install")

find_package(Protobuf REQUIRED)
...
```

3. 调整 cmake 选项

`cmake ..` 时，额外加入选项 `-DProtobuf_PROTOC_EXECUTABLE=/path/to/install/bin/protoc`

```bash
$ cd /path/to/ncnn/build
$ rm -rf CMakeCache
# 加入新选项

...
```

```bash
# 加入新选项
$ cmake .. -DProtobuf_PROTOC_EXECUTABLE=/path/to/install/bin/protoc 
$ ...
```

## （不推荐）自己改环境变量

### 一、遇到 `Protobuf not found`

是因为 protobuf 未安装或环境变量未设置

1. 安装 protobuf

Ubuntu 系统尝试以下命令
```bash
$ sudo apt-get install libprotobuf-dev protobuf-compiler
```

CentOS 尝试
```bash
$ sudo yum install protobuf-devel.x86_64 protobuf-compiler.x86_64
```

2. 然后设置 C++ 环境

在 LD_LIBRARY_PATH 增加参数

```bash
$ export LD_LIBRARY_PATH=${YOUR_PROTOBUF_LIB_PATH}:$LD_LIBRARY_PATH
```

### 二、遇到 protoc 和 protobuf.so 版本不匹配

1. 先看 protoc 需要的 so 版本号
```bash
$ ldd `whereis protoc| awk '{print $2}'` | grep libprotobuf.so
```

例如是 libprotobuf.so.10

2. 然后搜这个文件所在的路径
```bash
$ cd / && find . -type f | grep libprotobuf.so.10
```

假设在`/home/user/mydir`

3. 设置 protobuf.so 的搜索目录
```bash

```

```bash
```

```bash
$ export LD_LIBRARY_PATH=/home/user/mydir:$LD_LIBRARY_PATH
```

### 三、行走江湖必备
关于环境变量设置、工具和技巧，强烈建议学习下 https://missing.csail.mit.edu/

# 模型

我们通常将深度学习任务中的神经网络定义为模型，这个模型即是算法的核心。[MMEngine](https://github.com/open-mmlab/mmengine) 抽象出了一个统一模型 [BaseModel](https://github.com/open-mmlab/mmengine/blob/main/mmengine/model/base_model/base_model.py#L16) 以标准化训练、测试和其他过程。MMSegmentation 实现的所有模型都继承自 `BaseModel`，并且在 MMSegmention 中，我们实现了前向传播并为语义分割算法添加了一些功能。

## 常用组件

### 分割器（Segmentor）

在 MMSegmentation 中，我们将网络架构抽象为**分割器**，它是一个包含网络所有组件的模型。我们已经实现了**编码器解码器（EncoderDecoder）**和**级联编码器解码器（CascadeEncoderDecoder）**，它们通常由**数据预处理器**、**骨干网络**、**解码头**和**辅助头**组成。

### 数据预处理器（Data preprocessor）

**数据预处理器**是将数据复制到目标设备并将数据预处理为模型输入格式的部分。

**主干网络**是将图像转换为特征图的部分，例如没有最后全连接层的 **ResNet-50**。

### 颈部（Neck）

**颈部**是连接主干网络和头的部分。它对主干网络生成的原始特征图进行一些改进或重新配置。例如 **Feature Pyramid Network（FPN）**。

### 解码头（Decode head）

**解码头**是将特征图转换为分割掩膜的部分，例如 **PSPNet**。

### 辅助头（Auxiliary head）

**辅助头**是一个可选组件，它将特征图转换为仅用于计算辅助损失的分割掩膜。

## 基本接口

## 基本接口

MMSegmentation 封装 `BaseModel` 并实现了 [BaseSegmentor](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/models/segmentors/base.py#L15) 类，主要提供 `forward`、`train_step`、`val_step` 和 `test_step` 接口。接下来将详细介绍这些接口。

<center>
  <img src='https://user-images.githubusercontent.com/15952744/228827860-c0e34875-d370-4736-84f0-9560c26c9576.png' />
  <center>编码器解码器数据流</center>
</center>

<center>
  <center><img src='https://user-images.githubusercontent.com/15952744/228827987-aa214507-0c6d-4a08-8ce4-679b2b200b79.png' /></center>
  <center>级联编码器解码器数据流</center>
</center>

`forward` 方法返回训练、验证、测试和简单推理过程的损失或预测。

该方法应接受三种模式：“tensor”、“predict” 和 “loss”：

- “tensor”：前向推理整个网络并返回张量或张量数组，无需任何后处理，与常见的 `nn.Module` 相同。
- “predict”：前向推理并返回预测值，这些预测值将被完全处理到 `SegDataSample` 列表中。
- “loss”：前向推理并根据给定的输入和数据样本返回损失的`字典`。

**注：**[SegDataSample](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/structures/seg_data_sample.py) 是 MMSegmentation 的数据结构接口，用作不同组件之间的接口。`SegDataSample` 实现了抽象数据元素 `mmengine.structures.BaseDataElement`，请参阅 [MMMEngine](https://github.com/open-mmlab/mmengine) 中的 [SegDataSample 文档](https://mmsegmentation.readthedocs.io/zh_CN/1.x/advanced_guides/structures.html)和[数据元素文档](https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/data_element.html)了解更多信息。

注意，此方法不处理在 `train_step` 方法中完成的反向传播或优化器更新。

参数：

- inputs（torch.Tensor）- 通常为形状是（N, C, ...) 的输入张量。
- data_sample（list\[[SegDataSample](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/structures/seg_data_sample.py)\]) - 分割数据样本。它通常包括 `metainfo` 和 `gt_sem_seg` 等信息。默认值为 None。
- mode (str) - 返回什么类型的值。默认为 'tensor'。

返回值：

返回值：

- `dict` 或 `list`：
  - 如果 `mode == "loss"`，则返回用于反向过程和日志记录的损失张量`字典`。
  - 如果 `mode == "predict"`，则返回 `SegDataSample` 的`列表`，推理结果将被递增地添加到传递给 forward 方法的 `data_sample` 参数中，每个 `SegDataSeample` 包含以下关键词：
    - pred_sm_seg (`PixelData`)：语义分割的预测。
    - seg_logits (`PixelData`)：标准化前语义分割的预测指标。
  - 如果 `mode == "tensor"`，则返回`张量`或`张量数组`的`字典`以供自定义使用。

### 预测模式

我们在[配置文档](../user_guides/1_config.md)中简要描述了模型配置的字段，这里我们详细介绍 `model.test_cfg` 字段。`model.test_cfg` 用于控制前向行为，`"predict"` 模式下的 `forward` 方法可以在两种模式下运行：

- `whole_inference`：如果 `cfg.model.test_cfg.mode == 'whole'`，则模型将使用完整图像进行推理。

  `whole_inference` 模式的一个示例配置：

  ```python
    ...
  )
  ```

- `slide_inference`：如果 `cfg.model.test_cfg.mode == ‘slide’`，则模型将通过滑动窗口进行推理。**注意：** 如果选择 `slide` 模式，还应指定 `cfg.model.test_cfg.stride` 和 `cfg.model.test_cfg.crop_size`。

  `slide_inference` 模式的一个示例配置：

  ```python

`slide_inference` 模式的一个示例配置：

  ```python
    ...
  )
  ```

`train_step` 方法调用 `loss` 模式的前向接口以获得损失`字典`。`BaseModel` 类实现默认的模型训练过程，包括预处理、模型前向传播、损失计算、优化和反向传播。

参数：

- data (dict or tuple or list) - 从数据集采样的数据。在 MMSegmentation 中，数据字典包含 `inputs` 和 `data_samples` 两个字段。
- optim_wrapper (OptimWrapper) - 用于更新模型参数的 OptimWrapper 实例。

**注：**[OptimWrapper](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/optimizer/optimizer_wrapper.py#L17) 提供了一个用于更新参数的通用接口，请参阅 [MMMEngine](https://github.com/open-mmlab/mmengine) 中的优化器封装[文档](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/optim_wrapper.html)了解更多信息。

返回值：

-Dict\[str, `torch.Tensor`\]：用于记录日志的张量的`字典`。

<center>
  <img src='https://user-images.githubusercontent.com/15952744/228828089-a9ae1225-958d-4cf7-99af-9af8576f7ef7.png' />
  <center>train_step 数据流</center>
</center>

<center>
</center>

`val_step` 方法调用 `predict` 模式的前向接口并返回预测结果，预测结果将进一步被传递给评测器的进程接口和钩子的 `after_val_inter` 接口。

参数：

- data (`dict` or `tuple` or `list`) - 从数据集中采样的数据。在 MMSegmentation 中，数据字典包含 `inputs` 和 `data_samples` 两个字段。

返回值：

- `list` - 给定数据的预测结果。

<center>
  <img src='https://user-images.githubusercontent.com/15952744/228828179-3269baa3-bebd-4c9a-9787-59e7d785fbcf.png' />
  <center>test_step/val_step 数据流</center>
</center>

`BaseModel` 中 `test_step` 与 `val_step` 的实现相同。

## 数据预处理器（Data Preprocessor）

<center>test_step/val_step 数据流</center>
</center>

MMSegmentation 实现的 [SegDataPreProcessor](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/models/data_preprocessor.py#L13) 继承自由 [MMEngine](https://github.com/open-mmlab/mmengine) 实现的 [BaseDataPreprocessor](https://github.com/open-mmlab/mmengine/blob/main/mmengine/model/base_model/data_preprocessor.py#L18)，提供数据预处理和将数据复制到目标设备的功能。

Runner 在构建阶段将模型传送到指定的设备，而 [SegDataPreProcessor](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/models/data_preprocessor.py#L13) 在 `train_step`、`val_step` 和 `test_step` 中将数据传送到指定设备，之后处理后的数据将被进一步传递给模型。

`SegDataPreProcessor` 构造函数的参数：

- mean (Sequence\[Number\], 可选) - R、G、B 通道的像素平均值。默认为 None。
- std (Sequence\[Number\], 可选) - R、G、B 通道的像素标准差。默认为 None。
- size (tuple, 可选) - 固定的填充大小。
- size_divisor (int, 可选) - 填充大小的除法因子。
- pad_val (float, 可选) - 填充值。默认值：0。
- seg_pad_val (float, 可选) - 分割图的填充值。默认值：255。

- bgr_to_rgb (bool) - 是否将图像从 BGR 转换为 RGB。默认为 False。
- rgb_to_bgr (bool) - 是否将图像从 RGB 转换为 BGR。默认为 False。
- batch_augments (list\[dict\], 可选) - 批量化的数据增强。默认值为 None。

数据将按如下方式处理：

- 收集数据并将其移动到目标设备。
- 用定义的 `pad_val` 将输入填充到输入大小，并用定义的 `seg_Pad_val` 填充分割图。
- 将输入堆栈到 batch_input。
- 如果输入的形状为 (3, H, W)，则将输入从 BGR 转换为 RGB。
- 使用定义的标准差和平均值标准化图像。
- 在训练期间进行如 Mixup 和 Cutmix 的批量化数据增强。

`forward` 方法的参数：

- data (dict) - 从数据加载器采样的数据。
- training (bool) - 是否启用训练时数据增强。

`forward` 方法的返回值：

- Dict：与模型输入格式相同的数据。

# 使用ONNXRuntime进行RTMPose推理

本示例展示了如何在Python中用ONNXRuntime推理RTMPose模型。

## 准备

### 1. 安装onnxruntime推理引擎.

选择以下方式之一来安装onnxruntime。

- CPU版本

```bash
```

- GPU版本

```bash
```

### 2. 将模型转换为onnx文件

- 安装`mim`工具

```bash
```

- 下载`mmpose`模型

```bash
```

- 克隆`mmdeploy`仓库

```bash

```bash
```

```bash
```

```bash
```

- 将模型转换为onnx文件

```bash
```

## 运行

### 安装依赖

```bash
```

### 用法：

```bash
```

### 参数解释

- `ONNX_FILE`: onnx文件的路径
- `IMAGE_FILE`: 图像文件的路径
- `DEVICE`: 运行模型的设备，默认为\`cpu'
- `SAVE_PATH`: 保存输出图像的路径，默认为 "output.jpg"

<details>

```bibtex
}
```

<details>

```bibtex

```bibtex
}
```

The model is trained on COFW train.

| [pose_hrnetv2_w18](/configs/face_2d_keypoint/topdown_heatmap/cofw/td-hm_hrnetv2-w18_8xb64-60e_cofw-256x256.py) |  256x256   | 3.48 | [ckpt](https://download.openmmlab.com/mmpose/face/hrnetv2/hrnetv2_w18_cofw_256x256-49243ab8_20211019.pth) | [log](https://download.openmmlab.com/mmpose/face/hrnetv2/hrnetv2_w18_cofw_256x256_20211019.log.json) |

> [Learning Texture Transformer Network for Image Super-Resolution](https://arxiv.org/abs/2006.04139)

We study on image super-resolution (SR), which aims to recover realistic textures from a low-resolution (LR) image. Recent progress has been made by taking high-resolution images as references (Ref), so that relevant textures can be transferred to LR images. However, existing SR approaches neglect to use attention mechanisms to transfer high-resolution (HR) textures from Ref images, which limits these approaches in challenging cases. In this paper, we propose a novel Texture Transformer Network for Image Super-Resolution (TTSR), in which the LR and Ref images are formulated as queries and keys in a transformer, respectively. TTSR consists of four closely-related modules optimized for image generation tasks, including a learnable texture extractor by DNN, a relevance embedding module, a hard-attention module for texture transfer, and a soft-attention module for texture synthesis. Such a design encourages joint feature learning across LR and Ref images, in which deep feature correspondences can be discovered by attention, and thus accurate texture features can be transferred. The proposed texture transformer can be further stacked in a cross-scale way, which enables texture recovery from different levels (e.g., from 1x to 4x magnification). Extensive experiments show that TTSR achieves significant improvements over state-of-the-art approaches on both quantitative and qualitative evaluations.

 <img src="https://user-images.githubusercontent.com/7676947/144035689-e5afa799-f469-40a0-aa94-0b84a46726a1.png" width="400"/>
</div >

Evaluated on CUFED dataset (RGB channels), `scale` pixels in each border are cropped before evaluation.

|                                   Model                                    | Dataset | scale |  PSNR   |  SSIM  | Training Resources |                                      Download                                       |
| :------------------------------------------------------------------------: | :-----: | :---: | :-----: | :----: | :----------------: | :---------------------------------------------------------------------------------: |

| [ttsr-rec_x4_c64b16_g1_200k_CUFED](./ttsr-rec_x4c64b16_1xb9-200k_CUFED.py) |  CUFED  |  x4   | 25.2433 | 0.7491 |    1 (TITAN Xp)    | [model](https://download.openmmlab.com/mmediting/restorers/ttsr/ttsr-rec_x4_c64b16_g1_200k_CUFED_20210525-b0dba584.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/ttsr/ttsr-rec_x4_c64b16_g1_200k_CUFED_20210525-b0dba584.log.json) |
| [ttsr-gan_x4_c64b16_g1_500k_CUFED](./ttsr-gan_x4c64b16_1xb9-500k_CUFED.py) |  CUFED  |  x4   | 24.6075 | 0.7234 |    1 (TITAN Xp)    | [model](https://download.openmmlab.com/mmediting/restorers/ttsr/ttsr-gan_x4_c64b16_g1_500k_CUFED_20210626-2ab28ca0.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/ttsr/ttsr-gan_x4_c64b16_g1_500k_CUFED_20210626-2ab28ca0.log.json) |

**Train**

<details>

```shell
# cpu train

**Train**

<details>

```shell
# cpu train

```

**Test**

<details>

```shell
# cpu test

```

```bibtex
}
```

# Test Results Submission

## Panoptic segmentation test results submission

The following sections introduce how to produce the prediction results of panoptic segmentation models on the COCO test-dev set and submit the predictions to [COCO evaluation server](https://competitions.codalab.org/competitions/19507).

- Download [COCO test dataset images](http://images.cocodataset.org/zips/test2017.zip), [testing image info](http://images.cocodataset.org/annotations/image_info_test2017.zip), and [panoptic train/val annotations](http://images.cocodataset.org/annotations/panoptic_annotations_trainval2017.zip), then unzip them, put 'test2017' to `data/coco/`, put json files and annotation files to `data/coco/annotations/`.

```shell
# suppose data/coco/ does not exist
mkdir -pv data/coco/

# download test2017
wget -P data/coco/ http://images.cocodataset.org/zips/test2017.zip
wget -P data/coco/ http://images.cocodataset.org/annotations/image_info_test2017.zip

wget -P data/coco/ http://images.cocodataset.org/annotations/panoptic_annotations_trainval2017.zip

# unzip them
unzip data/coco/test2017.zip -d data/coco/
unzip data/coco/image_info_test2017.zip -d data/coco/
unzip data/coco/panoptic_annotations_trainval2017.zip -d data/coco/

# remove zip files (optional)
rm -rf data/coco/test2017.zip data/coco/image_info_test2017.zip data/coco/panoptic_annotations_trainval2017.zip
```

- Run the following code to update category information in testing image info. Since the attribute `isthing` is missing in category information of 'image_info_test-dev2017.json', we need to update it with the category information in 'panoptic_val2017.json'.

```shell
python tools/misc/gen_coco_panoptic_test_info.py data/coco/annotations
```

```shell
```

After completing the above preparations, your directory structure of `data` should be like this:

```text
data
`-- coco
    |-- annotations
    |   |-- image_info_test-dev2017.json
    |   |-- image_info_test2017.json
    |   |-- panoptic_image_info_test-dev2017.json
    |   |-- panoptic_train2017.json
    |   |-- panoptic_train2017.zip
    |   |-- panoptic_val2017.json
    |   `-- panoptic_val2017.zip
    `-- test2017
```

### Inference on coco test-dev

To do inference on coco test-dev, we should update the setting of `test_dataloder` and `test_evaluator` first. There two ways to do this: 1. update them in config file; 2. update them in command line.

#### Update them in config file

The relevant settings are provided at the end of `configs/_base_/datasets/coco_panoptic.py`, as below.

```python

```python
        ann_file='annotations/panoptic_image_info_test-dev2017.json',
    type='CocoPanopticMetric',
    ann_file=data_root + 'annotations/panoptic_image_info_test-dev2017.json',
    outfile_prefix='./work_dirs/coco_panoptic/test')
```

Any of the following way can be used to update the setting for inference on coco test-dev set.

Case 1: Directly uncomment the setting in `configs/_base_/datasets/coco_panoptic.py`.

Case 2: Copy the following setting to the config file you used now.

```python

```python
        data_prefix=dict(img='test2017/', _delete_=True)))
```

Then infer on coco test-dev et by the following command.

```shell
    ${CHECKPOINT_FILE}
```

#### Update them in command line

The command for update of the related settings and inference on coco test-dev are as below.

```shell
# test with single gpu
CUDA_VISIBLE_DEVICES=0 python tools/test.py \
    test_dataloader.dataset.ann_file=annotations/panoptic_image_info_test-dev2017.json \
    test_dataloader.dataset.data_prefix.img=test2017 \

    test_dataloader.dataset.data_prefix._delete_=True \
    test_evaluator.format_only=True \
    test_evaluator.ann_file=data/coco/annotations/panoptic_image_info_test-dev2017.json \
    test_evaluator.outfile_prefix=${WORK_DIR}/results

# test with four gpus
CUDA_VISIBLE_DEVICES=0,1,3,4 bash tools/dist_test.sh \
    8 \  # eights gpus

# test with slurm

test_evaluator.format_only=True \

GPUS=8 tools/slurm_test.sh \
    ${Partition} \
    ${JOB_NAME} \
```

Example

Suppose we perform inference on `test2017` using pretrained MaskFormer with ResNet-50 backbone.

```shell
    configs/maskformer/maskformer_r50_mstrain_16x1_75e_coco.py \

```shell
    checkpoints/maskformer_r50_mstrain_16x1_75e_coco_20220221_141956-bc2699cb.pth \
    test_evaluator.outfile_prefix=work_dirs/maskformer/results
```

### Rename files and zip results

test_evaluator.ann_file=data/coco/annotations/panoptic_image_info_test-dev2017.json \
```

After inference, the panoptic segmentation results (a json file and a directory where the masks are stored) will be in `WORK_DIR`. We should rename them according to the naming convention described on [COCO's Website](https://cocodataset.org/#upload). Finally, we need to compress the json and the directory where the masks are stored into a zip file, and rename the zip file according to the naming convention. Note that the zip file should **directly** contains the above two files.

The commands to rename files and zip results:

```shell
# In WORK_DIR, we have panoptic segmentation results: 'panoptic' and 'results.panoptic.json'.
cd ${WORK_DIR}

# replace '[algorithm_name]' with the name of algorithm you used.
mv ./panoptic ./panoptic_test-dev2017_[algorithm_name]_results

mv ./results.panoptic.json ./panoptic_test-dev2017_[algorithm_name]_results.json
zip panoptic_test-dev2017_[algorithm_name]_results.zip -ur panoptic_test-dev2017_[algorithm_name]_results panoptic_test-dev2017_[algorithm_name]_results.json
```

# ConvNeXt V2

> [Co-designing and Scaling ConvNets with Masked Autoencoders](http://arxiv.org/abs/2301.00808)

Driven by improved architectures and better representation learning frameworks, the field of visual recognition has enjoyed rapid modernization and performance boost in the early 2020s. For example, modern ConvNets, represented by ConvNeXt, have demonstrated strong performance in various scenarios. While these models were originally designed for supervised learning with ImageNet labels, they can also potentially benefit from self-supervised learning techniques such as masked autoencoders (MAE). However, we found that simply combining these two approaches leads to subpar performance. In this paper, we propose a fully convolutional masked autoencoder framework and a new Global Response Normalization (GRN) layer that can be added to the ConvNeXt architecture to enhance inter-channel feature competition. This co-design of self-supervised learning techniques and architectural improvement results in a new model family called ConvNeXt V2, which significantly improves the performance of pure ConvNets on various recognition benchmarks, including ImageNet classification, COCO detection, and ADE20K segmentation. We also provide pre-trained ConvNeXt V2 models of various sizes, ranging from an efficient 3.7M-parameter Atto model with 76.7% top-1 accuracy on ImageNet, to a 650M Huge model that achieves a state-of-the-art 88.9% accuracy using only public training data.

<img src="https://user-images.githubusercontent.com/26739999/210496285-f235083f-218f-4153-8e21-c8a64481a2f5.png" width="50%"/>
</div>

```python

predict = inference_model('convnext-v2-atto_fcmae-pre_3rdparty_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('convnext-v2-atto_3rdparty-fcmae_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/convnext_v2/convnext-v2-atto_32xb32_in1k.py https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-atto_fcmae-pre_3rdparty_in1k_20230104-23765f83.pth
```

| Model                                     | Params (M) | Flops (G) |                   Config                   |                                              Download                                              |
| :---------------------------------------- | :--------: | :-------: | :----------------------------------------: | :------------------------------------------------------------------------------------------------: |

| `convnext-v2-atto_3rdparty-fcmae_in1k`\*  |    3.71    |   0.55    | [config](convnext-v2-atto_32xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-atto_3rdparty-fcmae_in1k_20230104-07514db4.pth) |
| `convnext-v2-femto_3rdparty-fcmae_in1k`\* |    5.23    |   0.78    | [config](convnext-v2-femto_32xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-femto_3rdparty-fcmae_in1k_20230104-adbe2082.pth) |
| `convnext-v2-pico_3rdparty-fcmae_in1k`\*  |    9.07    |   1.37    | [config](convnext-v2-pico_32xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-pico_3rdparty-fcmae_in1k_20230104-147b1b59.pth) |
| `convnext-v2-nano_3rdparty-fcmae_in1k`\*  |   15.62    |   2.45    | [config](convnext-v2-nano_32xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-nano_3rdparty-fcmae_in1k_20230104-3dd1f29e.pth) |

| `convnext-v2-tiny_3rdparty-fcmae_in1k`\*  |   28.64    |   4.47    | [config](convnext-v2-tiny_32xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-tiny_3rdparty-fcmae_in1k_20230104-80513adc.pth) |
| `convnext-v2-base_3rdparty-fcmae_in1k`\*  |   88.72    |   15.38   | [config](convnext-v2-base_32xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-base_3rdparty-fcmae_in1k_20230104-8a798eaf.pth) |
| `convnext-v2-large_3rdparty-fcmae_in1k`\* |   197.96   |   34.40   | [config](convnext-v2-large_32xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-large_3rdparty-fcmae_in1k_20230104-bf38df92.pth) |
| `convnext-v2-huge_3rdparty-fcmae_in1k`\*  |   660.29   |  115.00   | [config](convnext-v2-huge_32xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-huge_3rdparty-fcmae_in1k_20230104-fe43ae6c.pth) |

*Models with * are converted from the [official repo](https://github.com/facebookresearch/ConvNeXt-V2). The config files of these models are only for inference. We haven't reproduce the training results.*

| Model                                           |      Pretrain      | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                      Config                      |                      Download                      |
| :---------------------------------------------- | :----------------: | :--------: | :-------: | :-------: | :-------: | :----------------------------------------------: | :------------------------------------------------: |

| `convnext-v2-atto_fcmae-pre_3rdparty_in1k`\*    |       FCMAE        |    3.71    |   0.55    |   76.64   |   93.04   |    [config](convnext-v2-atto_32xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-atto_fcmae-pre_3rdparty_in1k_20230104-23765f83.pth) |
| `convnext-v2-femto_fcmae-pre_3rdparty_in1k`\*   |       FCMAE        |    5.23    |   0.78    |   78.48   |   93.98   |    [config](convnext-v2-femto_32xb32_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-femto_fcmae-pre_3rdparty_in1k_20230104-92a75d75.pth) |
| `convnext-v2-pico_fcmae-pre_3rdparty_in1k`\*    |       FCMAE        |    9.07    |   1.37    |   80.31   |   95.08   |    [config](convnext-v2-pico_32xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-pico_fcmae-pre_3rdparty_in1k_20230104-d20263ca.pth) |

| `convnext-v2-nano_fcmae-pre_3rdparty_in1k`\*    |       FCMAE        |   15.62    |   2.45    |   81.86   |   95.75   |    [config](convnext-v2-nano_32xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-nano_fcmae-pre_3rdparty_in1k_20230104-fe1aaaf2.pth) |
| `convnext-v2-nano_fcmae-in21k-pre_3rdparty_in1k`\* | FCMAE ImageNet-21k |   15.62    |   2.45    |   82.04   |   96.16   |    [config](convnext-v2-nano_32xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-nano_fcmae-in21k-pre_3rdparty_in1k_20230104-91fa8ae2.pth) |
| `convnext-v2-tiny_fcmae-pre_3rdparty_in1k`\*    |       FCMAE        |   28.64    |   4.47    |   82.94   |   96.29   |    [config](convnext-v2-tiny_32xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-tiny_fcmae-pre_3rdparty_in1k_20230104-471a86de.pth) |

| `convnext-v2-tiny_fcmae-in21k-pre_3rdparty_in1k`\* | FCMAE ImageNet-21k |   28.64    |   4.47    |   83.89   |   96.96   |    [config](convnext-v2-tiny_32xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-tiny_fcmae-in21k-pre_3rdparty_in1k_20230104-8cc8b8f2.pth) |
| `convnext-v2-nano_fcmae-in21k-pre_3rdparty_in1k-384px`\* | FCMAE ImageNet-21k |   15.62    |   7.21    |   83.36   |   96.75   | [config](convnext-v2-nano_32xb32_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-nano_fcmae-in21k-pre_3rdparty_in1k-384px_20230104-f951ae87.pth) |
| `convnext-v2-tiny_fcmae-in21k-pre_3rdparty_in1k-384px`\* | FCMAE ImageNet-21k |   28.64    |   13.14   |   85.09   |   97.63   | [config](convnext-v2-tiny_32xb32_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-tiny_fcmae-in21k-pre_3rdparty_in1k-384px_20230104-d8579f84.pth) |

| `convnext-v2-base_fcmae-pre_3rdparty_in1k`\*    |       FCMAE        |   88.72    |   15.38   |   84.87   |   97.08   |    [config](convnext-v2-base_32xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-base_fcmae-pre_3rdparty_in1k_20230104-00a70fa4.pth) |
| `convnext-v2-base_fcmae-in21k-pre_3rdparty_in1k`\* | FCMAE ImageNet-21k |   88.72    |   15.38   |   86.74   |   98.02   |    [config](convnext-v2-base_32xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-base_fcmae-in21k-pre_3rdparty_in1k_20230104-c48d16a5.pth) |
| `convnext-v2-large_fcmae-pre_3rdparty_in1k`\*   |       FCMAE        |   197.96   |   34.40   |   85.76   |   97.59   |    [config](convnext-v2-large_32xb32_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-large_fcmae-pre_3rdparty_in1k_20230104-ef393013.pth) |

| `convnext-v2-large_fcmae-in21k-pre_3rdparty_in1k`\* | FCMAE ImageNet-21k |   197.96   |   34.40   |   87.26   |   98.24   |    [config](convnext-v2-large_32xb32_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-large_fcmae-in21k-pre_3rdparty_in1k_20230104-d9c4dc0c.pth) |
| `convnext-v2-base_fcmae-in21k-pre_3rdparty_in1k-384px`\* | FCMAE ImageNet-21k |   88.72    |   45.21   |   87.63   |   98.42   | [config](convnext-v2-base_32xb32_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-base_fcmae-in21k-pre_3rdparty_in1k-384px_20230104-379425cc.pth) |
| `convnext-v2-large_fcmae-in21k-pre_3rdparty_in1k-384px`\* | FCMAE ImageNet-21k |   197.96   |  101.10   |   88.18   |   98.52   | [config](convnext-v2-large_32xb32_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-large_fcmae-in21k-pre_3rdparty_in1k-384px_20230104-9139a1f3.pth) |

| `convnext-v2-huge_fcmae-pre_3rdparty_in1k`\*    |       FCMAE        |   660.29   |  115.00   |   86.25   |   97.75   |    [config](convnext-v2-huge_32xb32_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-huge_fcmae-pre_3rdparty_in1k_20230104-f795e5b8.pth) |
| `convnext-v2-huge_fcmae-in21k-pre_3rdparty_in1k-384px`\* | FCMAE ImageNet-21k |   660.29   |  337.96   |   88.68   |   98.73   | [config](convnext-v2-huge_32xb32_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-huge_fcmae-in21k-pre_3rdparty_in1k-384px_20230104-02a4eb35.pth) |
| `convnext-v2-huge_fcmae-in21k-pre_3rdparty_in1k-512px`\* | FCMAE ImageNet-21k |   660.29   |  600.81   |   88.86   |   98.74   | [config](convnext-v2-huge_32xb32_in1k-512px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/convnext-v2/convnext-v2-huge_fcmae-in21k-pre_3rdparty_in1k-512px_20230104-ce32e63c.pth) |

```bibtex
@article{Woo2023ConvNeXtV2,
  title={ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders},
  author={Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon and Saining Xie},
  journal={arXiv preprint arXiv:2301.00808},
}
```

# 变更日志

## v1.0.1 (26/05/2023)

**新功能和改进**

- 支持 StableDiffusion tomesd 加速. [#1801](https://github.com/open-mmlab/mmagic/pull/1801)
- 支持所有 inpainting/matting/image restoration 模型的 inferencer. [#1833](https://github.com/open-mmlab/mmagic/pull/1833), [#1873](https://github.com/open-mmlab/mmagic/pull/1873)
- 支持 animated drawings. [#1837](https://github.com/open-mmlab/mmagic/pull/1837)
- 支持 Style-Based Global Appearance Flow for Virtual Try-On at projects. [#1786](https://github.com/open-mmlab/mmagic/pull/1786)
- 支持 tokenizer wrapper 和 EmbeddingLayerWithFixe. [#1846](https://github.com/open-mmlab/mmagic/pull/1846)

**Bug 修复**

- 修复安装依赖. [#1819](https://github.com/open-mmlab/mmagic/pull/1819)
- 修复 inst-colorization PackInputs. [#1828](https://github.com/open-mmlab/mmagic/pull/1828), [#1827](https://github.com/open-mmlab/mmagic/pull/1827)
- 修复 pip install 时 inferencer 无法使用的问题. [#1875](https://github.com/open-mmlab/mmagic/pull/1875)

## v1.0.0 (25/04/2023)

![mmagic-log](https://user-images.githubusercontent.com/49083766/233557648-9034f5a0-c85d-4092-b700-3a28072251b6.png)

自从 MMEditing 诞生以来，它一直是许多图像超分辨率、编辑和生成任务的首选算法库，帮助多个研究团队取得 10 余 项国际顶级赛事的胜利，支撑了 100 多个 GitHub 生态项目。经过 OpenMMLab 2.0 框架的迭代更新以及与 MMGeneration 的合并，MMEditing 已经成为了一个支持基于 GAN 和 CNN 的底层视觉算法的强大工具。

而今天，MMEditing 将更加拥抱生成式 AI（Generative AI），正式更名为 **MMagic**（**M**ultimodal **A**dvanced, **G**enerative, and **I**ntelligent **C**reation），致力于打造更先进、更全面的 AIGC 开源算法库。

在 MMagic 中，我们已经支持了 53+ 模型，分布于 Stable Diffusion 的微调、图文生成、图像及视频修复、超分辨率、编辑和生成等多种任务。配合 [MMEngine](https://github.com/open-mmlab/mmengine) 出色的训练与实验管理支持，MMagic 将为广大研究者与 AIGC 爱好者们提供更加快捷灵活的实验支持，助力你的 AIGC 探索之旅。使用 MMagic，体验更多生成的魔力！让我们一起开启超越编辑的新纪元！ More than Editing, Unlock the Magic!

**主要更新**

**1. 新算法**

  - GLIDE
  - EG3D
  - NAFNet
  - SwinIR

  ![de87f16f-bf6d-4a61-8406-5ecdbb9167b6](https://user-images.githubusercontent.com/49083766/233558077-2005e603-c5a8-49af-930f-e7a465ca818b.png)

  MMagic 支持通过 ControlNet 与多帧渲染法实现长视频的生成。
  prompt keywords: a handsome man, silver hair, smiling, play basketball

  https://user-images.githubusercontent.com/12782558/227149757-fd054d32-554f-45d5-9f09-319184866d85.mp4

  prompt keywords: a girl, black hair, white pants, smiling, play basketball

  https://user-images.githubusercontent.com/49083766/233559964-bd5127bd-52f6-44b6-a089-9d7adfbc2430.mp4

  prompt keywords: a handsome man

  https://user-images.githubusercontent.com/12782558/227152129-d70d5f76-a6fc-4d23-97d1-a94abd08f95a.mp4

prompt keywords: a handsome man

- SAM + MMagic = Generate Anything！
  当下流行的 SAM（Segment Anything Model）也可以为 MMagic 提供更多加持！想制作自己的动画，可以移步至 [OpenMMLab PlayGround](https://github.com/open-mmlab/playground/blob/main/mmediting_sam/README.md)！

  https://user-images.githubusercontent.com/49083766/233562228-f39fc675-326c-4ae8-986a-c942059effd0.mp4

为了提升你的“施法”效率，我们对“魔术回路”做了以下升级:

- 通过 OpenMMLab 2.0 框架的 MMEngine 和 MMCV， MMagic 将编辑框架分解为不同的组件，并且可以通过组合不同的模块轻松地构建自定义的编辑器模型。我们可以像搭建“乐高”一样定义训练流程，提供丰富的组件和策略。在 MMagic 中，你可以使用不同的 APIs 完全控制训练流程.

- 支持 53+ 算法，232+ 配置，213+ 模型权重，26+ 损失函数，and 20+ 评价指标.

- 支持 controlnet 动画生成以及 Gradio gui. [点击查看.](https://github.com/open-mmlab/mmagic/tree/main/configs/controlnet_animation)
- 支持 Inferencer 和 Demo，使用High-level Inference APIs. [点击查看.](https://github.com/open-mmlab/mmagic/tree/main/demo)
- 支持 Inpainting 推理的 Gradio gui. [点击查看.](https://github.com/open-mmlab/mmagic/blob/main/demo/gradio-demo.py)
- 支持可视化图像/视频质量比较工具. [点击查看.](https://github.com/open-mmlab/mmagic/tree/main/tools/gui)
- 开启 projects，助力社区更快向算法库中添加新算法. [点击查看.](https://github.com/open-mmlab/mmagic/tree/main/projects)
- 完善数据集的预处理脚本和使用说明文档. [点击查看.](https://github.com/open-mmlab/mmagic/tree/main/tools/dataset_converters)

## v1.0.0rc7 (07/04/2023)

**主要更新**

我们很高兴发布 MMEditing 1.0.0rc7 版本。 此版本支持了 MMEditing 和 MMGeneration 的 51+ 模型，226+ configs 和 212+ checkpoints。以下是此次版本发布的重点新功能

**主要更新**

- 支持了 DiffuserWrapper.
- 支持了 ControlNet 的推理与训练.
- 支持了 PyTorch 2.0.

- 支持了 DiffuserWrapper. [#1692](https://github.com/open-mmlab/mmagic/pull/1692)
- 支持了 ControlNet 的推理与训练. [#1744](https://github.com/open-mmlab/mmagic/pull/1744)
- 支持了 PyTorch 2.0 (使用 'inductor' 后端成功编译 33+ 模型) [#1742](https://github.com/open-mmlab/mmagic/pull/1742).
- 支持了图像超分和视频超分的 inferencer. [#1662](https://github.com/open-mmlab/mmagic/pull/1662), [#1720](https://github.com/open-mmlab/mmagic/pull/1720)
- 重构 get_flops 脚本. [#1675](https://github.com/open-mmlab/mmagic/pull/1675)
- 重构数据集的 dataset_converters 脚本和使用文档. [#1690](https://github.com/open-mmlab/mmagic/pull/1690)
- 迁移 stylegan 算子到 MMCV 中. [#1383](https://github.com/open-mmlab/mmagic/pull/1383)

**Bug 修复**

- 修复 disco inferencer. [#1673](https://github.com/open-mmlab/mmagic/pull/1673)

**Bug 修复**

- 修复 nafnet optimizer 配置. [#1716](https://github.com/open-mmlab/mmagic/pull/1716)
- 修复 tof typo. [#1711](https://github.com/open-mmlab/mmagic/pull/1711)

**贡献者**

@LeoXing1996, @Z-Fran, @plyfager, @zengyh1900, @liuwenran, @ryanxingql, @HAOCHENYE, @VongolaWu

## v1.0.0rc6 (02/03/2023)

**主要更新**

我们很高兴发布 MMEditing 1.0.0rc6 版本。 此版本支持了 MMEditing 和 MMGeneration 的 50+ 模型，222+ configs 和 209+ checkpoints。以下是此次版本发布的重点新功能

- 支持了 Inpainting 任务推理的 Gradio gui.
- 支持了图像上色、图像翻译和 GAN 模型的 inferencer.

- 重构了 FileIO. [#1572](https://github.com/open-mmlab/mmagic/pull/1572)
- 重构了 registry. [#1621](https://github.com/open-mmlab/mmagic/pull/1621)
- 重构了 Random degradations. [#1583](https://github.com/open-mmlab/mmagic/pull/1583)

- 重构了 DataSample, DataPreprocessor, Metric 和 Loop. [#1656](https://github.com/open-mmlab/mmagic/pull/1656)
- 使用 mmengine.basemodule 替换 nn.module. [#1491](https://github.com/open-mmlab/mmagic/pull/1491)
- 重构了算法库主页. [#1609](https://github.com/open-mmlab/mmagic/pull/1609)
- 支持了 Inpainting 任务推理的 Gradio gui. [#1601](https://github.com/open-mmlab/mmagic/pull/1601)
- 支持了图像上色的 inferencer. [#1588](https://github.com/open-mmlab/mmagic/pull/1588)
- 支持了图像翻译和所有 GAN 模型的 inferencer. [#1650](https://github.com/open-mmlab/mmagic/pull/1650)
- 支持了 GAN 模型的 inferencer. [#1653](https://github.com/open-mmlab/mmagic/pull/1653), [#1659](https://github.com/open-mmlab/mmagic/pull/1659)
- 新增 Print config 工具. [#1590](https://github.com/open-mmlab/mmagic/pull/1590)
- 改进 type hints. [#1604](https://github.com/open-mmlab/mmagic/pull/1604)

- 更新 metrics 和 datasets 的中文文档. [#1568](https://github.com/open-mmlab/mmagic/pull/1568), [#1638](https://github.com/open-mmlab/mmagic/pull/1638)
- 更新 BigGAN 和 Disco-Diffusion 的中文文档. [#1620](https://github.com/open-mmlab/mmagic/pull/1620)
- 更新 Guided-Diffusion 的 Evaluation 和 README. [#1547](https://github.com/open-mmlab/mmagic/pull/1547)

**Bug 修复**

- 修复 EMA `momentum`. [#1581](https://github.com/open-mmlab/mmagic/pull/1581)
- 修复 RandomNoise 的输出类型. [#1585](https://github.com/open-mmlab/mmagic/pull/1585)
- 修复 pytorch2onnx 工具. [#1629](https://github.com/open-mmlab/mmagic/pull/1629)
- 修复 API 文档. [#1641](https://github.com/open-mmlab/mmagic/pull/1641), [#1642](https://github.com/open-mmlab/mmagic/pull/1642)
- 修复 RealESRGAN 加载 EMA 参数. [#1647](https://github.com/open-mmlab/mmagic/pull/1647)

- 修复 dataset_converters 脚本的 arg passing bug. [#1648](https://github.com/open-mmlab/mmagic/pull/1648)

**贡献者**

@plyfager, @LeoXing1996, @Z-Fran, @zengyh1900, @VongolaWu, @liuwenran, @austinmw, @dienachtderwelt, @liangzelong, @i-aki-y, @xiaomile, @Li-Qingyun, @vansin, @Luo-Yihang, @ydengbi, @ruoningYu, @triple-Mu

## v1.0.0rc5 (04/01/2023)

**主要更新**

我们很高兴发布 MMEditing 1.0.0rc5 版本。 此版本支持了 MMEditing 和 MMGeneration 的 49+ 模型，180+ configs 和 177+ checkpoints。以下是此次版本发布的重点新功能

- 支持了 Restormer 算法.
- 支持了 GLIDE 算法.
- 支持了 SwinIR 算法.
- 支持了 Stable Diffusion 算法.

- 新增 Disco notebook. (#1507)
- 优化测试 requirements 和 CI. (#1514)
- 自动生成文档 summary 和 API docstring. (#1517)
- 开启 projects. (#1526)
- 支持 mscoco dataset. (#1520)
- 改进中文文档. (#1532)
- 添加 Type hints. (#1481)
- 更新模型权重下载链接. (#1554)
- 更新部署指南. (#1551)

**Bug 修复**

- 修复文档链接检查. (#1522)
- 修复 ssim bug. (#1515)
- 修复 realesrgan 的 `extract_gt_data`. (#1542)

**Bug 修复**

- 修复算法索引. (#1559)
- F修复 disco-diffusion 的 config 路径. (#1553)
- Fix text2image inferencer. (#1523)

**贡献者**

@plyfager, @LeoXing1996, @Z-Fran, @zengyh1900, @VongolaWu, @liuwenran, @AlexZou14, @lvhan028, @xiaomile, @ldr426, @austin273, @whu-lee, @willaty, @curiosity654, @Zdafeng, @Taited

## v1.0.0rc4 (05/12/2022)

**主要更新**

我们很高兴发布 MMEditing 1.0.0rc4 版本。 此版本支持了 MMEditing 和 MMGeneration 的 45+ 模型，176+ configs 和 175+ checkpoints。以下是此次版本发布的重点新功能

- 支持了 High-level APIs.
- 支持了 diffusion 算法.
- 支持了 Text2Image 任务.
- 支持了 3D-Aware Generation.

- 支持和重构了 High-level APIs. (#1410)
- 支持了 disco-diffusion 文生图算法. (#1234, #1504)
- 支持了 EG3D 算法. (#1482, #1493, #1494, #1499)
- 支持了 NAFNet 算法. (#1369)

**Bug 修复**

- 修复 srgan 的训练配置. (#1441)
- 修复 cain 的 config. (#1404)
- 修复 rdn 和 srcnn 的训练配置. (#1392)

**贡献者**

**Bug 修复**

**贡献者**

@plyfager, @LeoXing1996, @Z-Fran, @zengyh1900, @VongolaWu, @gaoyang07, @ChangjianZhao, @zxczrx123, @jackghosts, @liuwenran, @CCODING04, @RoseZhao929, @shaocongliu, @liangzelong.

## v1.0.0rc3 (10/11/2022)

**主要更新**

我们很高兴发布 MMEditing 1.0.0rc3 版本。 此版本支持了 MMEditing 和 MMGeneration 的 43+ 模型，170+ configs 和 169+ checkpoints。以下是此次版本发布的重点新功能

- 将 `mmdet` 和 `clip` 改为可选安装需求.

- 支持 `mmdet` 的 `try_import`. (#1408)
- 支持 `flip` 的 `try_import`. (#1420)
- 更新 `.gitignore`. ($1416)
- 设置 `inception_utils` 的 `real_feat` 为 cpu 变量. (#1415)
- 更新 StyleGAN2 和 PEGAN 的 README 和 configs.  (#1418)
- 改进 API 文档的渲染. (#1373)

**Bug 修复**

- 修复 ESRGAN 的 config 和预训练模型加载. (#1407)
- 修复 LSGAN 的 config. (#1409)
- 修复 CAIN 的 config. (#1404)

**贡献者**

@Z-Fran, @zengyh1900, @plyfager, @LeoXing1996, @ruoningYu.

## v1.0.0rc2 (02/11/2022)

**贡献者**

**主要更新**

我们很高兴发布 MMEditing 1.0.0rc2 版本。 此版本支持了 MMEditing 和 MMGeneration 的 43+ 模型，170+ configs 和 169+ checkpoints。以下是此次版本发布的重点新功能

- 基于 patch 和 slider 的 图像和视频可视化质量比较工具.
- 支持了图像上色算法.

- 支持了质量比较工具. (#1303)
- 支持了 instance aware colorization 上色算法. (#1370)
- 支持使用不同采样模型的 multi-metrics. (#1171)
- 改进代码实现
  - 重构 evaluation metrics. (#1164)
  - 在 PGGAN 的 `forward` 中保存 gt 图像. (#1332)
  - 改进 `preprocess_div2k_dataset.py` 脚本的默认参数. (#1380)
  - 支持在 visualizer 中裁剪像素值. (#1365)
  - 支持了 SinGAN 数据集和 SinGAN demo. (#1363)
  - 支持 GenDataPreprocessor 中返回 int 和 float 数据类型. (#1385)
- 改进文档
  - 更新菜单切换. (#1162)
  - 修复 TTSR README. (#1325)

**Bug 修复**

- 修复 PPL bug. (#1172)
- 修复 RDN `number of channels` 参数. (#1328)
- 修复 demo 的 exceptions 类型. (#1372)
- 修复 realesrgan ema. (#1341)

- 改进文档

**Bug 修复**

- 改进 assertion 检查 `GenerateFacialHeatmap` 的数据类型为 `np.float32`. (#1310)
- 修复 `unpaired_dataset.py` 的采样方式. (#1308)
- 修复视频超分模型的 pytorch2onnx 脚本. (#1300)
- 修复错误的 config 配置. (#1167,#1200,#1236,#1293,#1302,#1304,#1319,#1331,#1336,#1349,#1352,#1353,#1358,#1364,#1367,#1384,#1386,#1391,#1392,#1393)

**贡献者**

@LeoXing1996, @Z-Fran, @zengyh1900, @plyfager, @ryanxingql, @ruoningYu, @gaoyang07.

## v1.0.0rc1(23/9/2022)

MMEditing 1.0.0rc1 已经合并了 MMGeneration 1.x。

- 支持 42+ 算法, 169+ 配置文件 and 168+ 预训练模型参数文件.
- 支持 26+ loss functions, 20+ metrics.
- 支持 tensorboard, wandb.
- 支持 unconditional GANs, conditional GANs, image2image translation 以及 internal learning.

## v1.0.0rc0(31/8/2022)

MMEditing 1.0.0rc0 是 MMEditing 1.x 的第一个版本，是 OpenMMLab 2.0 项目的一部分。

基于新的[训练引擎](https://github.com/open-mmlab/mmengine), MMEditing 1.x 统一了数据、模型、评测和可视化的接口。

该版本存在有一些 BC-breaking 的修改。 请在[迁移指南](https://mmagic.readthedocs.io/zh_CN/latest/migration/overview.html)中查看更多细节。

We appreciate all contributions to improve MMDetection. Please refer to [CONTRIBUTING.md](https://github.com/open-mmlab/mmcv/blob/master/CONTRIBUTING.md) in MMCV for more details about the contributing guideline.

We compare our results with some popular frameworks and official releases in terms of speed and accuracy.

## Comparison Rules

Here we compare our MMPose repo with other pose estimation toolboxes in the same data and model settings.

To ensure the fairness of the comparison, the comparison experiments were conducted under the same hardware environment and using the same dataset.
For each model setting, we kept the same data pre-processing methods to make sure the same feature input.
In addition, we also used Memcached, a distributed memory-caching system, to load the data in all the compared toolboxes.
This minimizes the IO time during benchmark.

The time we measured is the average training time for an iteration, including data processing and model training.

### Results on COCO val2017 with detector having human AP of 56.4 on COCO val2017 dataset

We demonstrate the superiority of our MMPose framework in terms of speed and accuracy on the standard COCO keypoint detection benchmark.
The mAP (the mean average precision) is used as the evaluation metric.

| Model      | Input size | MMPose (s/iter) | HRNet (s/iter) | MMPose (mAP) | HRNet (mAP) |
| :--------- | :--------: | :-------------: | :------------: | :----------: | :---------: |
| resnet_50  |  256x192   |    **0.28**     |      0.64      |  **0.718**   |    0.704    |
| resnet_50  |  384x288   |    **0.81**     |      1.24      |  **0.731**   |    0.722    |
| resnet_101 |  256x192   |    **0.36**     |      0.84      |  **0.726**   |    0.714    |
| resnet_101 |  384x288   |    **0.79**     |      1.53      |  **0.748**   |    0.736    |

| resnet_152 |  256x192   |    **0.49**     |      1.00      |  **0.735**   |    0.720    |
| resnet_152 |  384x288   |    **0.96**     |      1.65      |  **0.750**   |    0.743    |
| hrnet_w32  |  256x192   |    **0.54**     |      1.31      |  **0.746**   |    0.744    |
| hrnet_w32  |  384x288   |    **0.76**     |      2.00      |  **0.760**   |    0.758    |
| hrnet_w48  |  256x192   |    **0.66**     |      1.55      |  **0.756**   |    0.751    |
| hrnet_w48  |  384x288   |    **1.23**     |      2.20      |  **0.767**   |    0.763    |

## Hardware

## Software Environment

- CUDA 10.1

# STDC

> [Rethinking BiSeNet For Real-time Semantic Segmentation](https://arxiv.org/abs/2104.13188)

<a href="https://github.com/MichaelFan01/STDC-Seg">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.20.0/mmseg/models/backbones/stdc.py#L394">Code Snippet</a>

BiSeNet has been proved to be a popular two-stream network for real-time segmentation. However, its principle of adding an extra path to encode spatial information is time-consuming, and the backbones borrowed from pretrained tasks, e.g., image classification, may be inefficient for image segmentation due to the deficiency of task-specific design. To handle these problems, we propose a novel and efficient structure named Short-Term Dense Concatenate network (STDC network) by removing structure redundancy. Specifically, we gradually reduce the dimension of feature maps and use the aggregation of them for image representation, which forms the basic module of STDC network. In the decoder, we propose a Detail Aggregation module by integrating the learning of spatial information into low-level layers in single-stream manner. Finally, the low-level features and deep features are fused to predict the final segmentation results. Extensive experiments on Cityscapes and CamVid dataset demonstrate the effectiveness of our method by achieving promising trade-off between segmentation accuracy and inference speed. On Cityscapes, we achieve 71.9% mIoU on the test set with a speed of 250.4 FPS on NVIDIA GTX 1080Ti, which is 45.2% faster than the latest methods, and achieve 76.8% mIoU with 97.0 FPS while inferring on higher resolution images.

<img src="https://user-images.githubusercontent.com/24582831/143640374-d0709587-edb2-4821-bb60-340035f6ad8f.png" width="60%"/>
</div>

## Usage

We have provided [ImageNet Pretrained STDCNet Weights](https://drive.google.com/drive/folders/1wROFwRt8qWHD4jSo8Zu1gp1d6oYJ3ns1) models converted from [official repo](https://github.com/MichaelFan01/STDC-Seg).

If you want to convert keys on your own to use official repositories' pre-trained models, we also provide a script [`stdc2mmseg.py`](../../tools/model_converters/stdc2mmseg.py) in the tools directory to convert the key of models from [the official repo](https://github.com/MichaelFan01/STDC-Seg) to MMSegmentation style.

```shell
python tools/model_converters/stdc2mmseg.py ${PRETRAIN_PATH} ${STORE_PATH} ${STDC_TYPE}
```

E.g.

```shell
python tools/model_converters/stdc2mmseg.py ./STDCNet813M_73.91.tar ./pretrained/stdc1.pth STDC1

```

E.g.

```shell

python tools/model_converters/stdc2mmseg.py ./STDCNet1446_76.47.tar ./pretrained/stdc2.pth STDC2
```

| Method | Backbone             | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                        | download                                                                                                                                                                                                                                                                                                                                             |

| STDC   | STDC1 (No Pretrain)  | 512x1024  |   80000 | 7.15     | 23.06          | V100   | 71.82 | 73.89         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/stdc/stdc1_4xb12-80k_cityscapes-512x1024.py)          | [model](https://download.openmmlab.com/mmsegmentation/v0.5/stdc/stdc1_512x1024_80k_cityscapes/stdc1_512x1024_80k_cityscapes_20220224_073048-74e6920a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/stdc/stdc1_512x1024_80k_cityscapes/stdc1_512x1024_80k_cityscapes_20220224_073048.log.json)                                     |

| STDC   | STDC1                | 512x1024  |   80000 | -        | -              | V100   | 74.94 | 76.97         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/stdc/stdc1_in1k-pre_4xb12-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/stdc/stdc1_in1k-pre_512x1024_80k_cityscapes/stdc1_in1k-pre_512x1024_80k_cityscapes_20220224_141648-3d4c2981.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/stdc/stdc1_in1k-pre_512x1024_80k_cityscapes/stdc1_in1k-pre_512x1024_80k_cityscapes_20220224_141648.log.json) |

| STDC   | STDC2  (No Pretrain) | 512x1024  |   80000 | 8.27     | 23.71          | V100   | 73.15 | 76.13         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/stdc/stdc2_4xb12-80k_cityscapes-512x1024.py)          | [model](https://download.openmmlab.com/mmsegmentation/v0.5/stdc/stdc2_512x1024_80k_cityscapes/stdc2_512x1024_80k_cityscapes_20220222_132015-fb1e3a1a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/stdc/stdc2_512x1024_80k_cityscapes/stdc2_512x1024_80k_cityscapes_20220222_132015.log.json)                                     |

| STDC   | STDC2                | 512x1024  |   80000 | -        | -              | V100   | 76.67 | 78.67         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/stdc/stdc2_in1k-pre_4xb12-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/stdc/stdc2_in1k-pre_512x1024_80k_cityscapes/stdc2_in1k-pre_512x1024_80k_cityscapes_20220224_073048-1f8f0f6c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/stdc/stdc2_in1k-pre_512x1024_80k_cityscapes/stdc2_in1k-pre_512x1024_80k_cityscapes_20220224_073048.log.json) |

Note:

- For STDC on Cityscapes dataset, default setting is 4 GPUs with 12 samples per GPU in training.
- The FPS is for reference only. The environment is also different from paper setting, whose input size is `512x1024` and `768x1536`, i.e., 50% and 75% of our input size, respectively and using TensorRT.

- The parameter `fusion_kernel` in `STDCHead` is not learnable. In official repo, `find_unused_parameters=True` is set [here](https://github.com/MichaelFan01/STDC-Seg/blob/59ff37fbd693b99972c76fcefe97caa14aeb619f/train.py#L220). You may check it by printing model parameters of original repo on your own.

```bibtex
@inproceedings{fan2021rethinking,
  title={Rethinking BiSeNet For Real-time Semantic Segmentation},
  author={Fan, Mingyuan and Lai, Shenqi and Huang, Junshi and Wei, Xiaoming and Chai, Zhenhua and Luo, Junfeng and Wei, Xiaolin},
  pages={9716--9725},
}
```

## PCIE

- python2
- python3

查看是否有瀚博推理卡：`lspci -d:0100`

1. 环境准备

   ```bash
   ```

2. driver安装

   ```bash
   ```

3. 查看安装

   ```bash
   # 1.查看deb包是否安装成功

   #output
   ……

   # 2.查看驱动是否已加载到内核

   #output
   ```

4. 升级驱动

   ```bash
   ```

5. 卸载驱动

   ```bash
   ```

```bash

```

4. 升级驱动

   ```bash
   ```

5. 卸载驱动

   ```bash
   ```

```bash
```

## SDK

### step.1

```bash
```

### step.2

```bash

```

# Learn about Visualization

## Local Visualization

This section will present how to visualize the detection/tracking results with local visualizer.

If you want to draw prediction results, you can turn this feature on by setting `draw=True` in `TrackVisualizationHook` as follows.

default_hooks = dict(visualization=dict(type='TrackVisualizationHook', draw=True))
```

Specifically, the `TrackVisualizationHook` has the following arguments:

- `draw`: whether to draw prediction results. If it is False, it means that no drawing will be done. Defaults to False.
- `interval`: The interval of visualization. Defaults to 30.
- `score_thr`: The threshold to visualize the bboxes and masks. Defaults to 0.3.
- `show`: Whether to display the drawn image. Default to False.
- `wait_time`: The interval of show (s). Defaults to 0.
- `test_out_dir`: directory where painted images will be saved in testing process.

- `backend_args`: Arguments to instantiate a file client. Defaults to `None`.

In the `TrackVisualizationHook`, `TrackLocalVisualizer` will be called to implement visualization for MOT and VIS tasks.
We will present the details below.
You can refer to MMEngine for more details about [Visualization](https://github.com/open-mmlab/mmengine/blob/main/docs/en/advanced_tutorials/visualization.md) and [Hook](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/hook.md).

#### Tracking Visualization

We realize the tracking visualization with class `TrackLocalVisualizer`.
You can call it as follows.

```python
visualizer = dict(type='TrackLocalVisualizer')
```

It has the following arguments:

- `name`: Name of the instance. Defaults to 'visualizer'.
- `image`: The origin image to draw. The format should be RGB. Defaults to None.

```

- `vis_backends`: Visual backend config list. Defaults to None.
- `save_dir`: Save file dir for all storage backends. If it is None, the backend storage will not save any data.
- `line_width`: The linewidth of lines. Defaults to 3.
- `alpha`: The transparency of bboxes or mask. Defaults to 0.8.

Here is a visualization example of DeepSORT:

![test_img_89](https://user-images.githubusercontent.com/99722489/186062929-6d0e4663-0d8e-4045-9ec8-67e0e41da876.png)

# DetectoRS

> [DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution](https://arxiv.org/abs/2006.02334)

Many modern object detectors demonstrate outstanding performances by using the mechanism of looking and thinking twice. In this paper, we explore this mechanism in the backbone design for object detection. At the macro level, we propose Recursive Feature Pyramid, which incorporates extra feedback connections from Feature Pyramid Networks into the bottom-up backbone layers. At the micro level, we propose Switchable Atrous Convolution, which convolves the features with different atrous rates and gathers the results using switch functions. Combining them results in DetectoRS, which significantly improves the performances of object detection. On COCO test-dev, DetectoRS achieves state-of-the-art 55.7% box AP for object detection, 48.5% mask AP for instance segmentation, and 50.0% PQ for panoptic segmentation.

<img src="https://user-images.githubusercontent.com/40661020/143877901-24451581-2c50-4a54-b000-c4cb111e29ad.png"/>
</div>

DetectoRS requires COCO and [COCO-stuff](http://calvin.inf.ed.ac.uk/wp-content/uploads/data/cocostuffdataset/stuffthingmaps_trainval2017.zip) dataset for training. You need to download and extract it in the COCO dataset path.

```none
├── mmdet
├── tools
├── data
│   ├── coco
```

DetectoRS includes two major components:

- Recursive Feature Pyramid (RFP).
- Switchable Atrous Convolution (SAC).

They can be used independently.
Combining them together results in DetectoRS.
The results on COCO 2017 val are shown in the below table.

|  Method   |      Detector       | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                      Config                       |                                                                                                                                                         Download                                                                                                                                                         |

| :-------: | :-----------------: | :-----: | :------: | :------------: | :----: | :-----: | :-----------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|    RFP    | Cascade + ResNet-50 |   1x    |   7.5    |       -        |  44.8  |         |    [config](./cascade-rcnn_r50-rfp_1x_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/detectors/cascade_rcnn_r50_rfp_1x_coco/cascade_rcnn_r50_rfp_1x_coco-8cf51bfd.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/detectors/cascade_rcnn_r50_rfp_1x_coco/cascade_rcnn_r50_rfp_1x_coco_20200624_104126.log.json)             |

|    SAC    | Cascade + ResNet-50 |   1x    |   5.6    |       -        |  45.0  |         |    [config](./cascade-rcnn_r50-sac_1x_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/detectors/cascade_rcnn_r50_sac_1x_coco/cascade_rcnn_r50_sac_1x_coco-24bfda62.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/detectors/cascade_rcnn_r50_sac_1x_coco/cascade_rcnn_r50_sac_1x_coco_20200624_104402.log.json)             |
| DetectoRS | Cascade + ResNet-50 |   1x    |   9.9    |       -        |  47.4  |         | [config](./detectors_cascade-rcnn_r50_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/detectors/detectors_cascade_rcnn_r50_1x_coco/detectors_cascade_rcnn_r50_1x_coco-32a10ba0.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/detectors/detectors_cascade_rcnn_r50_1x_coco/detectors_cascade_rcnn_r50_1x_coco_20200706_001203.log.json) |

|    RFP    |   HTC + ResNet-50   |   1x    |   11.2   |       -        |  46.6  |  40.9   |        [config](./htc_r50-rfp_1x_coco.py)         |                               [model](https://download.openmmlab.com/mmdetection/v2.0/detectors/htc_r50_rfp_1x_coco/htc_r50_rfp_1x_coco-8ff87c51.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/detectors/htc_r50_rfp_1x_coco/htc_r50_rfp_1x_coco_20200624_103053.log.json)                               |
|    SAC    |   HTC + ResNet-50   |   1x    |   9.3    |       -        |  46.4  |  40.9   |        [config](./htc_r50-sac_1x_coco.py)         |                               [model](https://download.openmmlab.com/mmdetection/v2.0/detectors/htc_r50_sac_1x_coco/htc_r50_sac_1x_coco-bfa60c54.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/detectors/htc_r50_sac_1x_coco/htc_r50_sac_1x_coco_20200624_103111.log.json)                               |

| DetectoRS |   HTC + ResNet-50   |   1x    |   13.6   |       -        |  49.1  |  42.6   |     [config](./detectors_htc-r50_1x_coco.py)      |                   [model](https://download.openmmlab.com/mmdetection/v2.0/detectors/detectors_htc_r50_1x_coco/detectors_htc_r50_1x_coco-329b1453.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/detectors/detectors_htc_r50_1x_coco/detectors_htc_r50_1x_coco_20200624_103659.log.json)                   |
| DetectoRS |  HTC + ResNet-101   |   20e   |   19.6   |                |  50.5  |  43.9   |    [config](./detectors_htc-r101_20e_coco.py)     |       [model](https://download.openmmlab.com/mmdetection/v2.0/detectors/detectors_htc_r101_20e_coco/detectors_htc_r101_20e_coco_20210419_203638-348d533b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/detectors/detectors_htc_r101_20e_coco/detectors_htc_r101_20e_coco_20210419_203638.log.json)       |

*Note*: This is a re-implementation based on MMDetection-V2.

The original implementation is based on MMDetection-V1.

We provide the config files for [DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution](https://arxiv.org/pdf/2006.02334.pdf).

```latex
@article{qiao2020detectors,
  title={DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution},
  author={Qiao, Siyuan and Chen, Liang-Chieh and Yuille, Alan},
  journal={arXiv preprint arXiv:2006.02334},
}
```

# Timm Example

> [PyTorch Image Models](https://github.com/rwightman/pytorch-image-models)

Py**T**orch **Im**age **M**odels (`timm`) is a collection of image models, layers, utilities, optimizers, schedulers, data-loaders / augmentations, and reference training / validation scripts that aim to pull together a wide variety of SOTA models with ability to reproduce ImageNet training results.

<!--
<img src="" height="400" />
</div>
-->

|    Backbone     |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                          Config                           | Download |
| :-------------: | :-----: | :-----: | :------: | :------------: | :----: | :-------------------------------------------------------: | :------: |
|      R-50       | pytorch |   1x    |          |                |        |   [config](./retinanet_timm-tv-resnet50_fpn_1x_coco.py)   |          |

| EfficientNet-B1 |    -    |   1x    |          |                |        | [config](./retinanet_timm-efficientnet-b1_fpn_1x_coco.py) |          |

## Usage

### Install additional requirements

MMDetection supports timm backbones via `TIMMBackbone`, a wrapper class in MMPretrain.
Thus, you need to install `mmpretrain` in addition to timm.
If you have already installed requirements for mmdet, run

```shell
pip install 'dataclasses; python_version<"3.7"'
pip install timm
```

See [this document](https://mmpretrain.readthedocs.io/en/latest/get_started.html#installation) for the details of MMPretrain installation.

### Edit config

- See example configs for basic usage.

```

- See the documents of [timm feature extraction](https://rwightman.github.io/pytorch-image-models/feature_extraction/#multi-scale-feature-maps-feature-pyramid) and [TIMMBackbone](https://mmpretrain.readthedocs.io/en/latest/api/generated/mmpretrain.models.backbones.TIMMBackbone.html#mmpretrain.models.backbones.TIMMBackbone) for details.
- Which feature map is output depends on the backbone.
  Please check `backbone out_channels` and `backbone out_strides` in your log, and modify `model.neck.in_channels` and `model.backbone.out_indices` if necessary.
- If you use Vision Transformer models that do not support `features_only=True`, add `custom_hooks = []` to your config to disable `NumClassCheckHook`.

```latex
@misc{rw2019timm,
  author = {Ross Wightman},

```latex
  title = {PyTorch Image Models},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  doi = {10.5281/zenodo.4414861},
  howpublished = {\url{https://github.com/rwightman/pytorch-image-models}}
}
```

# TensorRT Support

### Install TensorRT

Please install TensorRT 8 follow [install-guide](https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html#installing).

**Note**:

- `pip Wheel File Installation` is not supported yet in this repo.

- We strongly suggest you install TensorRT through [tar file](https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html#installing-tar)

- After installation, you'd better add TensorRT environment variables to bashrc by:

  ```bash
  cd ${TENSORRT_DIR} # To TensorRT root directory
  ```

### Build custom ops

Some custom ops are created to support models in OpenMMLab, and the custom ops can be built as follow:

```bash

```bash
```

If you haven't installed TensorRT in the default path, Please add `-DTENSORRT_DIR` flag in CMake.

```bash
```

Please follow the tutorial in [How to convert model](../02-how-to-run/convert_model.md). **Note** that the device must be `cuda` device.

### Int8 Support

Since TensorRT supports INT8 mode, a custom dataset config can be given to calibrate the model. Following is an example for MMDetection:

```python

# dataset settings, same format as the codebase in OpenMMLab

```python

    dict(
        ])
]
    val=dict(

    val=dict(
```

Convert your model with this calibration dataset:

```python
    ...
```

If the calibration dataset is not given, the data will be calibrated with the dataset in model config.

## FAQs

- Error `Cannot found TensorRT headers` or `Cannot found TensorRT libs`

  Try cmake with flag `-DTENSORRT_DIR`:

  ```bash
  ```

  Please make sure there are libs and headers in `${TENSORRT_DIR}`.

```bash
  ```

- Error `error: parameter check failed at: engine.cpp::setBindingDimensions::1046, condition: profileMinDims.d[i] <= dimensions.d[i]`

  There is an input shape limit in deployment config:

  ```python
          dict(
      ])
  ```

  The shape of the tensor `input` must be limited between `input_shapes["input"]["min_shape"]` and `input_shapes["input"]["max_shape"]`.

- Error `error: [TensorRT] INTERNAL ERROR: Assertion failed: cublasStatus == CUBLAS_STATUS_SUCCESS`

  TRT 7.2.1 switches to use cuBLASLt (previously it was cuBLAS). cuBLASLt is the default choice for SM version >= 7.0. However, you may need CUDA-10.2 Patch 1 (Released Aug 26, 2020) to resolve some cuBLASLt issues. Another option is to use the new TacticSource API and disable cuBLASLt tactics if you don't want to upgrade.

  Read [this](https://forums.developer.nvidia.com/t/matrixmultiply-failed-on-tensorrt-7-2-1/158187/4) for detail.

- Install mmdeploy on Jetson

  We provide a tutorial to get start on Jetsons [here](../01-how-to-build/jetsons.md).

# Changelog v2.x

### v2.25.0 (31/5/2022)

- Support dedicated `WandbLogger` hook
- Support [ConvNeXt](configs/convnext), [DDOD](configs/ddod), [SOLOv2](configs/solov2)
- Support [Mask2Former](configs/mask2former) for instance segmentation
- Rename [config files of Mask2Former](configs/mask2former)

#### Backwards incompatible changes

- Rename [config files of Mask2Former](configs/mask2former) (#7571)

    <thead>
        </tr>
    </thead>
    <th>

  - `mask2former_xxx_coco.py` represents config files for **panoptic segmentation**.

  </th>
    <th>

  - `mask2former_xxx_coco.py` represents config files for **instance segmentation**.
  - `mask2former_xxx_coco-panoptic.py` represents config files for **panoptic segmentation**.

- `mask2former_xxx_coco-panoptic.py` represents config files for **panoptic segmentation**.

- Support [ConvNeXt](https://arxiv.org/abs/2201.03545) (#7281)
- Support [DDOD](https://arxiv.org/abs/2107.02963) (#7279)
- Support [SOLOv2](https://arxiv.org/abs/2003.10152) (#7441)
- Support [Mask2Former](https://arxiv.org/abs/2112.01527) for instance segmentation (#7571, #8032)

- Enable YOLOX training on different devices (#7912)
- Fix the log plot error when evaluation with `interval != 1` (#7784)
- Fix RuntimeError of HTC (#8083)

- Support dedicated `WandbLogger` hook (#7459)

  Users can set

  ```python
  cfg.log_config.hooks = [
    dict(type='MMDetWandbHook',
         init_kwargs={'project': 'MMDetection-tutorial'},
         interval=10,
         log_checkpoint=True,
         log_checkpoint_metadata=True,
         num_eval_images=10)]
  ```

init_kwargs={'project': 'MMDetection-tutorial'},
  ```

  in the config to use `MMDetWandbHook`. Example can be found in this [colab tutorial](https://colab.research.google.com/drive/1RCSXHZwDZvakFh3eo9RuNrJbCGqD0dru?usp=sharing#scrollTo=WTEdPDRaBz2C)

- Add `AvoidOOM` to avoid OOM (#7434, #8091)

  Try to use `AvoidCUDAOOM` to avoid GPU out of memory. It will first retry after calling `torch.cuda.empty_cache()`. If it still fails, it will then retry by converting the type of inputs to FP16 format. If it still fails, it will try to copy inputs from GPUs to CPUs to continue computing. Try AvoidOOM in code to make the code continue to run when GPU memory runs out:

  ```python
  from mmdet.utils import AvoidCUDAOOM

  output = AvoidCUDAOOM.retry_if_cuda_oom(some_function)(input1, input2)
  ```

```python

  ```

  Users can also try `AvoidCUDAOOM` as a decorator to make the code continue to run when GPU memory runs out:

  ```python

  @AvoidCUDAOOM.retry_if_cuda_oom
  def function(*args, **kwargs):
      ...
      return xxx
  ```

- Support reading `gpu_collect` from `cfg.evaluation.gpu_collect` (#7672)

- Speedup the Video Inference by Accelerating data-loading Stage (#7832)

- Support replacing the `${key}` with the value of `cfg.key` (#7492)

- Accelerate result analysis in `analyze_result.py`. The evaluation time is speedup by 10 ~ 15 times and only tasks 10 ~ 15 minutes now. (#7891)

- Support to set `block_dilations` in `DilatedEncoder` (#7812)

- Support panoptic segmentation result analysis (#7922)

- Release DyHead with Swin-Large backbone (#7733)

- Documentations updating and adding

  - Fix wrong default type of `act_cfg` in `SwinTransformer` (#7794)
  - Fix text errors in the tutorials (#7959)
  - Rewrite the [installation guide](docs/en/get_started.md) (#7897)
  - [Useful hooks](docs/en/tutorials/useful_hooks.md) (#7810)
  - Fix heading anchor in documentation  (#8006)
  - Replace `markdownlint` with `mdformat` for avoiding installing ruby (#8009)

Thanks @ZwwWayne, @DarthThomas, @solyaH, @LutingWang, @chenxinfeng4, @Czm369, @Chenastron, @chhluo, @austinmw, @Shanyaliux @hellock, @Y-M-Y, @jbwang1997, @hhaAndroid, @Irvingao, @zhanggefan, @BIGWangYuDong, @Keiku, @PeterVennerstrom, @ayulockin

### v2.24.0 (26/4/2022)

- Support [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://arxiv.org/abs/2012.07177)
- Support automatically scaling LR according to GPU number and samples per GPU
- Support Class Aware Sampler that improves performance on OpenImages Dataset

- Support [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://arxiv.org/abs/2012.07177), see [example configs](configs/simple_copy_paste/mask_rcnn_r50_fpn_syncbn-all_rpn-2conv_ssj_scp_32x2_270k_coco.py) (#7501)

- Support Class Aware Sampler, users can set

  ```python
  data=dict(train_dataloader=dict(class_aware_sampler=dict(num_sample_class=1))))
  ```

  in the config to use `ClassAwareSampler`. Examples can be found in [the configs of OpenImages Dataset](https://github.com/open-mmlab/mmdetection/tree/main/configs/openimages/faster_rcnn_r50_fpn_32x2_cas_1x_openimages.py).  (#7436)

- Support automatically scaling LR according to GPU number and samples per GPU. (#7482)
  In each config, there is a corresponding config of auto-scaling LR as below,

  ```python
  auto_scale_lr = dict(enable=True, base_batch_size=N)
  ```

  where `N` is the batch size used for the current learning rate in the config (also equals to `samples_per_gpu` * gpu number to train this config).
  By default, we set `enable=False` so that the original usages will not be affected. Users can set `enable=True` in each config or add `--auto-scale-lr` after the command line to enable this feature and should check the correctness of `base_batch_size` in customized configs.

- Support setting dataloader arguments in config and add functions to handle config compatibility. (#7668)

    <thead>
            <td>v2.23.0</td>
            <td>v2.24.0</td>
        </tr>
    </thead>

    <thead>
        </tr>
    </thead>
    <th>

  ```python
      samples_per_gpu=64, workers_per_gpu=4,
  )
  ```

  </th>
    <th>

  ```python
      train_dataloader=dict(samples_per_gpu=64, workers_per_gpu=4),
      val_dataloader=dict(samples_per_gpu=8, workers_per_gpu=2),
      test_dataloader=dict(samples_per_gpu=8, workers_per_gpu=2),
  )

      samples_per_gpu=64,  # only works for train_dataloader

)

      workers_per_gpu=4,  # only works for train_dataloader
  )
  ```

- Support memory profile hook. Users can use it to monitor the memory usages during training as below (#7560)

  ```python
  custom_hooks = [
      dict(type='MemoryProfilerHook', interval=50)
  ]
  ```

- Support to run on PyTorch with MLU chip (#7578)

- Support re-spliting data batch with tag (#7641)

- Support the `DiceCost` used by [K-Net](https://arxiv.org/abs/2106.14855) in `MaskHungarianAssigner` (#7716)

- Support splitting COCO data for Semi-supervised object detection (#7431)

- Support Pathlib for Config.fromfile (#7685)

- Support to use file client in OpenImages dataset (#7433)

- Add a probability parameter to Mosaic transformation (#7371)

- Support specifying interpolation mode in `Resize` pipeline (#7585)

- Avoid invalid bbox after deform_sampling (#7567)
- Fix the issue that argument color_theme does not take effect when exporting confusion matrix (#7701)
- Fix the `end_level` in Necks, which should be the index of the end input backbone level (#7502)
- Fix the bug that `mix_results` may be None in `MultiImageMixDataset` (#7530)
- Fix the bug in ResNet plugin when two plugins are used (#7797)

- Enhance `load_json_logs` of analyze_logs.py for resumed training logs (#7732)

- Add argument `out_file` in image_demo.py (#7676)
- Allow mixed precision training with `SimOTAAssigner` (#7516)
- Updated INF to 100000.0 to be the same as that in the official YOLOX (#7778)
- Add documentations of:
  - how to get channels of a new backbone (#7642)
  - how to unfreeze the backbone network (#7570)
  - how to train fast_rcnn model (#7549)
  - proposals in Deformable DETR (#7690)
  - from-scratch install script in get_started.md (#7575)
- Release pre-trained models of
  - [Mask2Former](configs/mask2former) (#7595, #7709)
  - RetinaNet with ResNet-18 and release models (#7387)
  - RetinaNet with EfficientNet backbone (#7646)

- RetinaNet with ResNet-18 and release models (#7387)

Thanks @jovialio, @zhangsanfeng2022, @HarryZJ, @jamiechoi1995, @nestiank, @PeterH0323, @RangeKing, @Y-M-Y, @mattcasey02, @weiji14, @Yulv-git, @xiefeifeihu, @FANG-MING, @meng976537406, @nijkah, @sudz123, @CCODING04, @SheffieldCao, @Czm369, @BIGWangYuDong, @zytx121, @jbwang1997, @chhluo, @jshilong, @RangiLyu, @hhaAndroid, @ZwwWayne

### v2.23.0 (28/3/2022)

- Support Mask2Former: [Masked-attention Mask Transformer for Universal Image Segmentation](https://arxiv.org/abs/2112.01527)
- Support EfficientNet: [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://arxiv.org/abs/1905.11946)
- Support setting data root through environment variable `MMDET_DATASETS`, users don't have to modify the corresponding path in config files anymore.

- Find a good recipe for fine-tuning high precision ResNet backbone pre-trained by Torchvision.

- Support [Mask2Former](configs/mask2former)(#6938)(#7466)(#7471)
- Support [EfficientNet](configs/efficientnet) (#7514)
- Support setting data root through environment variable `MMDET_DATASETS`, users don't have to modify the corresponding path in config files anymore. (#7386)
- Support setting different seeds to different ranks (#7432)
- Update the `dist_train.sh` so that the script can be used to support launching multi-node training on machines without slurm (#7415)
- Find a good recipe for fine-tuning high precision ResNet backbone pre-trained by Torchvision (#7489)

- Fix bug in VOC unit test which removes the data directory (#7270)
- Adjust the order of `get_classes` and `FileClient` (#7276)

- Force the inputs of `get_bboxes` in yolox_head to float32 (#7324)
- Fix misplaced arguments in LoadPanopticAnnotations (#7388)
- Fix reduction=mean in CELoss. (#7449)
- Update unit test of CrossEntropyCost (#7537)
- Fix memory leaking in panpotic segmentation evaluation (#7538)
- Fix the bug of shape broadcast in YOLOv3 (#7551)

- Add Chinese version of onnx2tensorrt.md (#7219)
- Update colab tutorials (#7310)
- Update information about Localization Distillation (#7350)
- Add Chinese version of `finetune.md` (#7178)
- Update YOLOX log for non square input (#7235)
- Add `nproc` in `coco_panoptic.py` for panoptic quality computing (#7315)
- Allow to set channel_order in LoadImageFromFile (#7258)
- Take point sample related functions out of mask_point_head (#7353)
- Add instance evaluation for coco_panoptic (#7313)

- Enhance the robustness of analyze_logs.py (#7407)
- Supplementary notes of sync_random_seed (#7440)
- Update docstring of cross entropy loss (#7472)
- Update pascal voc result (#7503)
- We create How-to documentation to record any questions about How to xxx. In this version, we added
  - How to use Mosaic augmentation (#7507)
  - How to use backbone in mmcls (#7438)
  - How to produce and submit the prediction results of panoptic segmentation models on COCO test-dev set (#7430))

- How to produce and submit the prediction results of panoptic segmentation models on COCO test-dev set (#7430))

Thanks @ZwwWayne, @haofanwang, @shinya7y, @chhluo, @yangrisheng, @triple-Mu, @jbwang1997, @HikariTJU, @imflash217, @274869388, @zytx121, @matrixgame2018, @jamiechoi1995, @BIGWangYuDong, @JingweiZhang12, @Xiangxu-0103, @hhaAndroid, @jshilong, @osbm, @ceroytres, @bunge-bedstraw-herb, @Youth-Got, @daavoo, @jiangyitong, @RangiLyu, @CCODING04, @yarkable

### v2.22.0 (24/2/2022)

- Support MaskFormer: [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://arxiv.org/abs/2107.06278) (#7212)
- Support DyHead: [Dynamic Head: Unifying Object Detection Heads with Attentions](https://arxiv.org/abs/2106.08322) (#6823)

- Release a good recipe of using ResNet in object detectors pre-trained by [ResNet Strikes Back](https://arxiv.org/abs/2110.00476), which consistently brings about 3~4 mAP improvements over RetinaNet, Faster/Mask/Cascade Mask R-CNN (#7001)
- Support [Open Images Dataset](https://storage.googleapis.com/openimages/web/index.html) (#6331)
- Support TIMM backbone: [PyTorch Image Models](https://github.com/rwightman/pytorch-image-models) (#7020)

- Support [MaskFormer](configs/maskformer) (#7212)
- Support [DyHead](configs/dyhead) (#6823)
- Support [ResNet Strikes Back](configs/resnet_strikes_back) (#7001)
- Support [OpenImages Dataset](configs/openimages) (#6331)
- Support [TIMM backbone](configs/timm_example) (#7020)
- Support visualization for Panoptic Segmentation (#7041)

#### Breaking Changes

In order to support the visualization for Panoptic Segmentation, the `num_classes` can not be `None` when using the `get_palette` function to determine whether to use the panoptic palette.

- Fix bug for the best checkpoints can not be saved when the `key_score` is None (#7101)
- Fix MixUp transform filter boxes failing case (#7080)
- Add missing properties in SABLHead (#7091)
- Fix bug when NaNs exist in confusion matrix (#7147)
- Fix PALETTE AttributeError in downstream task (#7230)

- Speed up SimOTA matching (#7098)
- Add Chinese translation of `docs_zh-CN/tutorials/init_cfg.md` (#7188)

Thanks @ZwwWayne, @hhaAndroid, @RangiLyu, @AronLin, @BIGWangYuDong, @jbwang1997, @zytx121, @chhluo, @shinya7y, @LuooChen, @dvansa, @siatwangmin, @del-zhenwu, @vikashranjan26, @haofanwang, @jamiechoi1995, @HJoonKwon, @yarkable, @zhijian-liu, @RangeKing

### v2.21.0 (8/2/2022)

To standardize the contents in config READMEs and meta files of OpenMMLab projects, the READMEs and meta files in each config directory have been significantly changed. The template will be released in the future, for now, you can refer to the examples of README for [algorithm](https://github.com/open-mmlab/mmdetection/blob/master/configs/faster_rcnn/README.md), [dataset](https://github.com/open-mmlab/mmdetection/blob/master/configs/deepfashion/README.md) and [backbone](https://github.com/open-mmlab/mmdetection/blob/master/configs/regnet/README.md). To align with the standard, the configs in dcn are put into to two directories named `dcn` and `dcnv2`.

- Allow to customize colors of different classes during visualization (#6716)
- Support CPU training (#7016)
- Add download script of COCO, LVIS, and VOC dataset (#7015)

- Fix weight conversion issue of RetinaNet with Swin-S (#6973)

- Update `__repr__` of `Compose` (#6951)
- Fix BadZipFile Error when build docker (#6966)
- Fix bug in non-distributed multi-gpu training/testing (#7019)
- Fix bbox clamp in PyTorch 1.10 (#7074)
- Relax the requirement of PALETTE in dataset wrappers (#7085)
- Keep the same weights before reassign in the PAA head (#7032)
- Update code demo in doc (#7092)

- Speed-up training by allow to set variables of multi-processing (#6974, #7036)
- Add links of Chinese tutorials in readme (#6897)
- Disable cv2 multiprocessing by default for acceleration (#6867)
- Deprecate the support for "python setup.py test" (#6998)
- Re-organize metafiles and config readmes (#7051)
- Fix None grad problem during training TOOD by adding `SigmoidGeometricMean` (#7090)

A total of 26 developers contributed to this release.
Thanks @del-zhenwu, @zimoqingfeng, @srishilesh, @imyhxy, @jenhaoyang, @jliu-ac, @kimnamu, @ShengliLiu, @garvan2021, @ciusji, @DIYer22, @kimnamu, @q3394101, @zhouzaida, @gaotongxiao, @topsy404, @AntoAndGar, @jbwang1997, @nijkah, @ZwwWayne, @Czm369, @jshilong, @RangiLyu, @BIGWangYuDong, @hhaAndroid, @AronLin

### v2.20.0 (27/12/2021)

- Support [TOOD](configs/tood/README.md): Task-aligned One-stage Object Detection (ICCV 2021 Oral) (#6746)
- Support resuming from the latest checkpoint automatically (#6727)

- Fix wrong bbox `loss_weight` of the PAA head (#6744)
- Fix the padding value of `gt_semantic_seg` in batch collating (#6837)
- Fix test error of lvis when using `classwise` (#6845)
- Avoid BC-breaking of `get_local_path`  (#6719)

- Fix bug in `sync_norm_hook` when the BN layer does not exist (#6852)
- Use pycocotools directly no matter what platform it is (#6838)

- Add unit test for SimOTA with no valid bbox (#6770)
- Use precommit to check readme (#6802)
- Support selecting GPU-ids in non-distributed testing time (#6781)

Thanks @ZwwWayne, @Czm369, @jshilong, @RangiLyu, @BIGWangYuDong, @hhaAndroid, @jamiechoi1995, @AronLin, @Keiku, @gkagkos, @fcakyon, @www516717402, @vansin, @zactodd, @kimnamu, @jenhaoyang

### v2.19.1 (14/12/2021)

- Release [YOLOX](configs/yolox/README.md) COCO pretrained models (#6698)

- Fix DCN initialization in DenseHead (#6625)
- Fix initialization of ConvFCHead (#6624)

- Fix PseudoSampler in RCNN (#6622)
- Fix weight initialization in Swin and PVT (#6663)
- Fix dtype bug in BaseDenseHead (#6767)
- Fix SimOTA with no valid bbox (#6733)

- Add an example of combining swin and one-stage models (#6621)
- Add `get_ann_info` to dataset_wrappers (#6526)
- Support keeping image ratio in the multi-scale training of YOLOX (#6732)
- Support `bbox_clip_border` for the augmentations of YOLOX (#6730)

#### Documents

- Update metafile (#6717)
- Add mmhuman3d in readme (#6699)
- Update FAQ docs (#6587)
- Add doc for `detect_anomalous_params` (#6697)

Thanks @ZwwWayne, @LJoson, @Czm369, @jshilong, @ZCMax, @RangiLyu, @BIGWangYuDong, @hhaAndroid, @zhaoxin111, @GT9505, @shinya7y

### v2.19.0 (29/11/2021)

- Support [Label Assignment Distillation](https://arxiv.org/abs/2108.10520)
- Support `persistent_workers` for Pytorch >= 1.7
- Align accuracy to the updated official YOLOX

- Support [Label Assignment Distillation](https://arxiv.org/abs/2108.10520) (#6342)
- Support `persistent_workers` for Pytorch >= 1.7 (#6435)

- Fix repeatedly output warning message (#6584)
- Avoid infinite GPU waiting in dist training (#6501)
- Fix SSD512 config error (#6574)
- Fix MMDetection model to ONNX command (#6558)

- Refactor configs of FP16 models (#6592)
- Align accuracy to the updated official YOLOX (#6443)
- Speed up training and reduce memory cost when using PhotoMetricDistortion. (#6442)

- Make OHEM work with seesaw loss (#6514)

- Update README.md (#6567)

Thanks @FloydHsiu, @RangiLyu, @ZwwWayne, @AndreaPi, @st9007a, @hachreak, @BIGWangYuDong, @hhaAndroid, @AronLin, @chhluo, @vealocia, @HarborYuan, @st9007a, @jshilong

### v2.18.1 (15/11/2021)

- Release [QueryInst](http://arxiv.org/abs/2105.01928) pre-trained weights (#6460)
- Support plot confusion matrix (#6344)

- Fix aug test error when the number of prediction bboxes is 0 (#6398)
- Fix SpatialReductionAttention in PVT (#6488)

- Fix wrong use of `trunc_normal_init` in PVT and Swin-Transformer (#6432)

- Save the printed AP information of COCO API to logger (#6505)
- Always map location to cpu when load checkpoint (#6405)
- Set a random seed when the user does not set a seed (#6457)

- Chinese version of [Corruption Benchmarking](robustness_benchmarking.md) (#6375)
- Fix config path in docs (#6396)
- Update GRoIE readme (#6401)

Thanks @st9007a, @hachreak, @HarborYuan, @vealocia, @chhluo, @AndreaPi, @AronLin, @BIGWangYuDong, @hhaAndroid, @RangiLyu, @ZwwWayne

### v2.18.0 (27/10/2021)

- Support [QueryInst](http://arxiv.org/abs/2105.01928) (#6050)

- Refactor dense heads to decouple onnx export logics from `get_bboxes` and speed up inference (#5317, #6003, #6369, #6268, #6315)

- Support infinite sampler (#5996)

- Fix init_weight in fcn_mask_head (#6378)
- Fix type error in imshow_bboxes of RPN (#6386)
- Fix broken colab link in MMDetection Tutorial (#6382)
- Make sure the device and dtype of scale_factor are the same as bboxes (#6374)
- Remove sampling hardcode (#6317)
- Fix RandomAffine bbox coordinate recorrection (#6293)
- Fix init bug of final cls/reg layer in convfc head (#6279)
- Fix img_shape broken in auto_augment (#6259)
- Fix kwargs parameter missing error in two_stage (#6256)

- Unify the interface of stuff head and panoptic head (#6308)
- Polish readme (#6243)

- Add code-spell pre-commit hook and fix a typo (#6306)
- Fix typo (#6245, #6190)
- Fix sampler unit test (#6284)
- Fix `forward_dummy` of YOLACT to enable `get_flops` (#6079)
- Fix link error in the config documentation (#6252)
- Adjust the order to beautify the document (#6195)

#### Refactors

- Refactor one-stage get_bboxes logic (#5317)
- Refactor ONNX export of One-Stage models (#6003, #6369)
- Refactor dense_head and speedup (#6268)
- Migrate to use prior_generator in training of dense heads (#6315)

A total of 18 developers contributed to this release.
Thanks @Boyden, @onnkeat, @st9007a, @vealocia, @yhcao6, @DapangpangX, @yellowdolphin, @cclauss, @kennymckormick,
@pingguokiller, @collinzrj, @AndreaPi, @AronLin, @BIGWangYuDong, @hhaAndroid, @jshilong, @RangiLyu, @ZwwWayne

### v2.17.0 (28/9/2021)

- Support [PVT](https://arxiv.org/abs/2102.12122) and [PVTv2](https://arxiv.org/abs/2106.13797)
- Support [SOLO](https://arxiv.org/abs/1912.04488)
- Support large scale jittering and New Mask R-CNN baselines
- Speed up `YOLOv3` inference

- Support [PVT](https://arxiv.org/abs/2102.12122) and [PVTv2](https://arxiv.org/abs/2106.13797) (#5780)
- Support [SOLO](https://arxiv.org/abs/1912.04488) (#5832)
- Support large scale jittering and New Mask R-CNN baselines (#6132)
- Add a general data structure for the results of models (#5508)
- Added a base class for one-stage instance segmentation (#5904)
- Speed up `YOLOv3` inference (#5991)
- Release Swin Transformer pre-trained models (#6100)
- Support mixed precision training in `YOLOX` (#5983)
- Support `val` workflow in `YOLACT` (#5986)

- Add script to test `torchserve` (#5936)
- Support `onnxsim` with dynamic input shape (#6117)

- Fix the function naming errors in `model_wrappers` (#5975)
- Fix regression loss bug when the input is an empty tensor (#5976)
- Fix scores not contiguous error in `centernet_head` (#6016)
- Fix missing parameters bug in `imshow_bboxes` (#6034)
- Fix bug in `aug_test` of `HTC` when the length of `det_bboxes` is 0 (#6088)
- Fix empty proposal errors in the training of some two-stage models (#5941)
- Fix `dynamic_axes` parameter error in `ONNX` dynamic shape export (#6104)
- Fix `dynamic_shape` bug of `SyncRandomSizeHook` (#6144)
- Fix the Swin Transformer config link error in the configuration (#6172)

- Add filter rules in `Mosaic` transform (#5897)

- Add size divisor in get flops to avoid some potential bugs (#6076)
- Add Chinese translation of `docs_zh-CN/tutorials/customize_dataset.md` (#5915)
- Add Chinese translation of `conventions.md` (#5825)
- Add description of the output of data pipeline (#5886)
- Add dataset information in the README file for `PanopticFPN` (#5996)
- Add `extra_repr` for `DropBlock` layer to get details in the model printing (#6140)
- Fix CI out of memory and add PyTorch1.9 Python3.9 unit tests (#5862)
- Fix download links error of some model (#6069)
- Improve the generalization of XML dataset (#5943)
- Polish assertion error messages (#6017)
- Remove `opencv-python-headless` dependency by `albumentations` (#5868)
- Check dtype in transform unit tests (#5969)

- Replace the default theme of documentation with PyTorch Sphinx Theme (#6146)
- Update the paper and code fields in the metafile (#6043)
- Support to customize padding value of segmentation map (#6152)
- Support to resize multiple segmentation maps (#5747)

A total of 24 developers contributed to this release.
Thanks @morkovka1337, @HarborYuan, @guillaumefrd, @guigarfr, @www516717402, @gaotongxiao, @ypwhs, @MartaYang, @shinya7y, @justiceeem, @zhaojinjian0000, @VVsssssk, @aravind-anantha, @wangbo-zhao, @czczup, @whai362, @czczup, @marijnl, @AronLin, @BIGWangYuDong, @hhaAndroid, @jshilong, @RangiLyu, @ZwwWayne

### v2.16.0 (30/8/2021)

- Support [Panoptic FPN](https://arxiv.org/abs/1901.02446) and [Swin Transformer](https://arxiv.org/abs/2103.14030)

- Support [Panoptic FPN](https://arxiv.org/abs/1901.02446) and release models (#5577, #5902)
- Support Swin Transformer backbone (#5748)
- Release RetinaNet models pre-trained with multi-scale 3x schedule (#5636)
- Add script to convert unlabeled image list to coco format (#5643)
- Add hook to check whether the loss value is valid (#5674)
- Add YOLO anchor optimizing tool (#5644)
- Support export onnx models without post process. (#5851)
- Support classwise evaluation in CocoPanopticDataset (#5896)
- Adapt browse_dataset for concatenated datasets. (#5935)
- Add `PatchEmbed` and `PatchMerging` with `AdaptivePadding` (#5952)

- Fix unit tests of YOLOX (#5859)
- Fix lose randomness in `imshow_det_bboxes` (#5845)
- Make output result of `ImageToTensor` contiguous (#5756)

- Fix inference bug when calling `regress_by_class` in RoIHead in some cases (#5884)
- Fix bug in CIoU loss where alpha should not have gradient. (#5835)
- Fix the bug that `multiscale_output` is defined but not used in HRNet (#5887)
- Set the priority of EvalHook to LOW. (#5882)
- Fix a YOLOX bug when applying bbox rescaling in test mode (#5899)
- Fix mosaic coordinate error (#5947)
- Fix dtype of bbox in RandomAffine. (#5930)

- Add Chinese version of `data_pipeline` and  (#5662)
- Support to remove state dicts of EMA when publishing models. (#5858)
- Refactor the loss function in HTC and SCNet (#5881)
- Use warnings instead of logger.warning (#5540)
- Use legacy coordinate in metric of VOC (#5627)
- Add Chinese version of customize_losses (#5826)
- Add Chinese version of model_zoo (#5827)

Thanks @ypwhs, @zywvvd, @collinzrj, @OceanPang, @ddonatien, @@haotian-liu, @viibridges, @Muyun99, @guigarfr, @zhaojinjian0000, @jbwang1997,@wangbo-zhao, @xvjiarui, @RangiLyu, @jshilong, @AronLin, @BIGWangYuDong, @hhaAndroid, @ZwwWayne

### v2.15.1 (11/8/2021)

- Support [YOLOX](https://arxiv.org/abs/2107.08430)

- Support [YOLOX](https://arxiv.org/abs/2107.08430)(#5756, #5758, #5760, #5767, #5770, #5774, #5777, #5808, #5828, #5848)

- Update correct SSD models. (#5789)
- Fix casting error in mask structure (#5820)
- Fix MMCV deployment documentation links. (#5790)

- Use dynamic MMCV download link in TorchServe dockerfile (#5779)

- Rename the function `upsample_like` to `interpolate_as` for more general usage (#5788)

Thanks @HAOCHENYE, @xiaohu2015, @HsLOL, @zhiqwang, @Adamdad, @shinya7y, @Johnson-Wang, @RangiLyu, @jshilong, @mmeendez8, @AronLin, @BIGWangYuDong, @hhaAndroid, @ZwwWayne

### v2.15.0 (02/8/2021)

- Support adding [MIM](https://github.com/open-mmlab/mim) dependencies during pip installation
- Support MobileNetV2 for SSD-Lite and YOLOv3
- Support Chinese Documentation

- Add function `upsample_like` (#5732)
- Support to output pdf and epub format documentation (#5738)
- Support and release Cascade Mask R-CNN 3x pre-trained models (#5645)
- Add `ignore_index` to CrossEntropyLoss (#5646)

- Support adding [MIM](https://github.com/open-mmlab/mim) dependencies during pip installation (#5676)
- Add MobileNetV2 config and models for YOLOv3 (#5510)
- Support COCO Panoptic Dataset (#5231)
- Support ONNX export of cascade models (#5486)
- Support DropBlock with RetinaNet (#5544)
- Support MobileNetV2 SSD-Lite (#5526)

- Fix the device of label in multiclass_nms (#5673)
- Fix error of backbone initialization from pre-trained checkpoint in config file (#5603, #5550)
- Fix download links of RegNet pretrained weights (#5655)
- Fix two-stage runtime error given empty proposal (#5559)
- Fix flops count error in DETR (#5654)
- Fix unittest for `NumClassCheckHook` when it is not used. (#5626)
- Fix description bug of using custom dataset (#5546)

- Fix bug of `multiclass_nms` that returns the global indices (#5592)
- Fix `valid_mask` logic error in RPNHead (#5562)
- Fix unit test error of pretrained configs (#5561)
- Fix typo error in anchor_head.py (#5555)
- Fix bug when using dataset wrappers (#5552)
- Fix a typo error in demo/MMDet_Tutorial.ipynb (#5511)
- Fixing crash in `get_root_logger` when `cfg.log_level` is not None (#5521)
- Fix docker version (#5502)
- Fix optimizer parameter error when using `IterBasedRunner` (#5490)

- Add unit tests for MMTracking (#5620)
- Add Chinese translation of documentation (#5718, #5618, #5558, #5423, #5593, #5421, #5408. #5369, #5419, #5530, #5531)
- Update resource limit (#5697)
- Update docstring for InstaBoost (#5640)
- Support key `reduction_override` in all loss functions (#5515)

- Use repeatdataset to accelerate CenterNet training (#5509)
- Remove unnecessary code in autoassign (#5519)
- Add documentation about `init_cfg` (#5273)

Thanks @OceanPang, @AronLin, @hellock, @Outsider565, @RangiLyu, @ElectronicElephant, @likyoo, @BIGWangYuDong, @hhaAndroid, @noobying, @yyz561, @likyoo,
@zeakey, @ZwwWayne, @ChenyangLiu, @johnson-magic, @qingswu, @BuxianChen

### v2.14.0 (29/6/2021)

- Add `simple_test` to dense heads to improve the consistency of single-stage and two-stage detectors
- Revert the `test_mixins` to single image test to improve efficiency and readability
- Add Faster R-CNN and Mask R-CNN config using multi-scale training with 3x schedule

- Support pretrained models from MoCo v2 and SwAV (#5286)

- Add Faster R-CNN and Mask R-CNN config using multi-scale training with 3x schedule (#5179, #5233)
- Add `reduction_override` in MSELoss (#5437)
- Stable support of exporting DETR to ONNX with dynamic shapes and batch inference (#5168)
- Stable support of exporting PointRend to ONNX with dynamic shapes and batch inference (#5440)

- Fix size mismatch bug in `multiclass_nms` (#4980)
- Fix the import path of `MultiScaleDeformableAttention` (#5338)
- Fix errors in config of GCNet ResNext101 models (#5360)
- Fix Grid-RCNN error when there is no bbox result (#5357)
- Fix errors in `onnx_export` of bbox_head when setting reg_class_agnostic (#5468)
- Fix type error of AutoAssign in the document (#5478)
- Fix web links ending with `.md` (#5315)

- Add `simple_test` to dense heads to improve the consistency of single-stage and two-stage detectors (#5264)
- Add support for mask diagonal flip in TTA (#5403)
- Revert the `test_mixins` to single image test to improve efficiency and readability (#5249)
- Make YOLOv3 Neck more flexible (#5218)
- Refactor SSD to make it more general (#5291)
- Refactor `anchor_generator` and `point_generator` (#5349)
- Allow to configure out the `mask_head` of the HTC algorithm (#5389)
- Delete deprecated warning in FPN (#5311)
- Move `model.pretrained` to `model.backbone.init_cfg` (#5370)
- Make deployment tools more friendly to use (#5280)
- Clarify installation documentation (#5316)
- Add ImageNet Pretrained Models docs (#5268)

- Add FAQ about training loss=nan solution and COCO AP or AR =-1 (# 5312, #5313)
- Change all weight links of http to https (#5328)

### v2.13.0 (01/6/2021)

- Support new methods: [CenterNet](https://arxiv.org/abs/1904.07850), [Seesaw Loss](https://arxiv.org/abs/2008.10032), [MobileNetV2](https://arxiv.org/abs/1801.04381)

- Support paper [Objects as Points](https://arxiv.org/abs/1904.07850) (#4602)
- Support paper [Seesaw Loss for Long-Tailed Instance Segmentation (CVPR 2021)](https://arxiv.org/abs/2008.10032) (#5128)
- Support [MobileNetV2](https://arxiv.org/abs/1801.04381) backbone and inverted residual block (#5122)
- Support [MIM](https://github.com/open-mmlab/mim) (#5143)
- ONNX exportation with dynamic shapes of CornerNet (#5136)
- Add `mask_soft` config option to allow non-binary masks (#4615)

- Add PWC metafile (#5135)

- Fix YOLOv3 FP16 training error (#5172)
- Fix Cacscade R-CNN TTA test error when `det_bboxes` length is 0  (#5221)
- Fix `iou_thr` variable naming errors in VOC recall calculation function (#5195)
- Fix Faster R-CNN performance dropped in ONNX Runtime (#5197)
- Fix DETR dict changed error when using python 3.8 during iteration  (#5226)

- Refactor ONNX export of two stage detector (#5205)
- Replace MMDetection's EvalHook with MMCV's EvalHook for consistency  (#4806)
- Update RoI extractor for ONNX (#5194)
- Use better parameter initialization in YOLOv3 head for higher performance (#5181)
- Release new DCN models of Mask R-CNN by mixed-precision training (#5201)
- Update YOLOv3 model weights (#5229)

- Add DetectoRS ResNet-101 model weights (#4960)
- Discard bboxes with sizes equals to `min_bbox_size` (#5011)
- Remove duplicated code in DETR head (#5129)
- Remove unnecessary object in class definition (#5180)
- Fix doc link (#5192)

### v2.12.0 (01/5/2021)

- Support new methods: [AutoAssign](https://arxiv.org/abs/2007.03496), [YOLOF](https://arxiv.org/abs/2103.09460), and [Deformable DETR](https://arxiv.org/abs/2010.04159)
- Stable support of exporting models to ONNX with batched images and dynamic shape (#5039)

MMDetection is going through big refactoring for more general and convenient usages during the releases from v2.12.0 to v2.15.0 (maybe longer).

- MMCV version. MMDetection v2.12.0 relies on the newest features in MMCV 1.3.3, including `BaseModule` for unified parameter initialization, model registry, and the CUDA operator `MultiScaleDeformableAttn` for [Deformable DETR](https://arxiv.org/abs/2010.04159). Note that MMCV 1.3.2 already contains all the features used by MMDet but has known issues. Therefore, we recommend users skip MMCV v1.3.2 and use v1.3.3, though v1.3.2 might work for most cases.

- Unified model initialization (#4750). To unify the parameter initialization in OpenMMLab projects, MMCV supports `BaseModule` that accepts `init_cfg` to allow the modules' parameters initialized in a flexible and unified manner. Now the users need to explicitly call `model.init_weights()` in the training script to initialize the model (as in [here](https://github.com/open-mmlab/mmdetection/blob/master/tools/train.py#L162), previously this was handled by the detector. The models in MMDetection have been re-benchmarked to ensure accuracy based on PR #4750. __The downstream projects should update their code accordingly to use MMDetection v2.12.0__.

- Unified model registry (#5059). To easily use backbones implemented in other OpenMMLab projects, MMDetection migrates to inherit the model registry created in MMCV (#760). In this way, as long as the backbone is supported in an OpenMMLab project and that project also uses the registry in MMCV, users can use that backbone in MMDetection by simply modifying the config without copying the code of that backbone into MMDetection.
- Mask AP evaluation (#4898). Previous versions calculate the areas of masks through the bounding boxes when calculating the mask AP of small, medium, and large instances. To indeed use the areas of masks, we pop the key `bbox` during mask AP calculation. This change does not affect the overall mask AP evaluation and aligns the mask AP of similar models in other projects like Detectron2.

- Support paper [AutoAssign: Differentiable Label Assignment for Dense Object Detection](https://arxiv.org/abs/2007.03496) (#4295)

- Support paper [You Only Look One-level Feature](https://arxiv.org/abs/2103.09460) (#4295)
- Support paper [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://arxiv.org/abs/2010.04159) (#4778)
- Support calculating IoU with FP16 tensor in `bbox_overlaps` to save memory and keep speed (#4889)
- Add `__repr__` in custom dataset to count the number of instances (#4756)
- Add windows support by updating requirements.txt (#5052)
- Stable support of exporting models to ONNX with batched images and dynamic shape, including SSD, FSAF,FCOS, YOLOv3, RetinaNet, Faster R-CNN, and Mask R-CNN (#5039)

- Use MMCV `MODEL_REGISTRY` (#5059)
- Unified parameter initialization for more flexible usage (#4750)
- Rename variable names and fix docstring in anchor head (#4883)

- Support training with empty GT in Cascade RPN (#4928)
- Add more details of usage of `test_robustness` in documentation (#4917)
- Changing to use `pycocotools` instead of `mmpycocotools` to fully support Detectron2 and MMDetection in one environment (#4939)
- Update torch serve dockerfile to support dockers of more versions (#4954)
- Add check for training with single class dataset (#4973)
- Refactor transformer and DETR Head (#4763)
- Update FPG model zoo (#5079)
- More accurate mask AP of small/medium/large instances (#4898)

- Fix bug in mean_ap.py when calculating mAP by 11 points (#4875)
- Fix error when key `meta` is not in old checkpoints (#4936)
- Fix hanging bug when training with empty GT in VFNet, GFL, and FCOS by changing the place of `reduce_mean` (#4923, #4978, #5058)

- Fix asyncronized inference error and provide related demo (#4941)
- Fix IoU losses dimensionality unmatch error (#4982)
- Fix torch.randperm whtn using PyTorch 1.8 (#5014)
- Fix empty bbox error in `mask_head` when using CARAFE (#5062)
- Fix `supplement_mask` bug when there are zero-size RoIs (#5065)
- Fix testing with empty rois in RoI Heads (#5081)

### v2.11.0 (01/4/2021)

__Highlights__

- Support new method: [Localization Distillation for Object Detection](https://arxiv.org/pdf/2102.12252.pdf)
- Support Pytorch2ONNX with batch inference and dynamic shape

__New Features__

- Support [Localization Distillation for Object Detection](https://arxiv.org/pdf/2102.12252.pdf) (#4758)

- Support Pytorch2ONNX with batch inference and dynamic shape for Faster-RCNN and mainstream one-stage detectors (#4796)

__Improvements__

- Support batch inference in head of RetinaNet (#4699)
- Add batch dimension in second stage of Faster-RCNN (#4785)
- Support batch inference in bbox coder (#4721)
- Add check for `ann_ids` in `COCODataset` to ensure it is unique (#4789)
- support for showing the FPN results (#4716)
- support dynamic shape for grid_anchor (#4684)
- Move pycocotools version check to when it is used (#4880)

__Bug Fixes__

- Fix a bug of TridentNet when doing the batch inference (#4717)
- Fix a bug of Pytorch2ONNX in FASF (#4735)
- Fix a bug when show the image with float type (#4732)

### v2.10.0 (01/03/2021)

- Support new methods: [FPG](https://arxiv.org/abs/2004.03580)
- Support ONNX2TensorRT for SSD, FSAF, FCOS, YOLOv3, and Faster R-CNN.

- Support ONNX2TensorRT for SSD, FSAF, FCOS, YOLOv3, and Faster R-CNN (#4569)
- Support [Feature Pyramid Grids (FPG)](https://arxiv.org/abs/2004.03580) (#4645)
- Support video demo (#4420)
- Add seed option for sampler (#4665)
- Support to customize type of runner (#4570, #4669)
- Support synchronizing BN buffer in `EvalHook` (#4582)
- Add script for GIF demo (#4573)

- Fix ConfigDict AttributeError and add Colab link (#4643)
- Avoid crash in empty gt training of GFL head (#4631)
- Fix `iou_thrs` bug in RPN evaluation (#4581)
- Fix syntax error of config when upgrading model version (#4584)

- Refactor unit test file structures (#4600)

- Refactor nms config (#4636)
- Get loading pipeline by checking the class directly rather than through config strings (#4619)
- Add doctests for mask target generation and mask structures (#4614)
- Use deep copy when copying pipeline arguments (#4621)
- Update documentations (#4642, #4650, #4620, #4630)
- Remove redundant code calling `import_modules_from_strings` (#4601)
- Clean deprecated FP16 API (#4571)
- Check whether `CLASSES` is correctly initialized in the initialization of `XMLDataset` (#4555)
- Support batch inference in the inference API (#4462, #4526)
- Clean deprecated warning and fix 'meta' error (#4695)

### v2.9.0 (01/02/2021)

- Support new methods: [SCNet](https://arxiv.org/abs/2012.10150), [Sparse R-CNN](https://arxiv.org/abs/2011.12450)

- Move `train_cfg` and `test_cfg` into model in configs
- Support to visualize results based on prediction quality

- Support [SCNet](https://arxiv.org/abs/2012.10150) (#4356)
- Support [Sparse R-CNN](https://arxiv.org/abs/2011.12450) (#4219)
- Support evaluate mAP by multiple IoUs (#4398)
- Support concatenate dataset for testing (#4452)
- Support to visualize results based on prediction quality (#4441)
- Add ONNX simplify option to Pytorch2ONNX script (#4468)
- Add hook for checking compatibility of class numbers in heads and datasets (#4508)

- Fix CPU inference bug of Cascade RPN (#4410)
- Fix NMS error of CornerNet when there is no prediction box (#4409)
- Fix TypeError in CornerNet inference (#4411)
- Fix bug of PAA when training with background images (#4391)

- Fix the error that the window data is not destroyed when `out_file is not None` and `show==False` (#4442)
- Fix order of NMS `score_factor` that will decrease the performance of YOLOv3 (#4473)
- Fix bug in HTC TTA when the number of detection boxes is 0 (#4516)
- Fix resize error in mask data structures (#4520)

- Allow to customize classes in LVIS dataset (#4382)
- Add tutorials for building new models with existing datasets (#4396)
- Add CPU compatibility information in documentation (#4405)
- Add documentation of deprecated `ImageToTensor` for batch inference (#4408)
- Add more details in documentation for customizing dataset (#4430)
- Switch `imshow_det_bboxes` visualization backend from OpenCV to Matplotlib (#4389)
- Deprecate `ImageToTensor` in `image_demo.py` (#4400)

- Move train_cfg/test_cfg into model (#4347, #4489)
- Update docstring for `reg_decoded_bbox` option in bbox heads (#4467)
- Update dataset information in documentation (#4525)
- Release pre-trained R50 and R101 PAA detectors with multi-scale 3x training schedules (#4495)
- Add guidance for speed benchmark (#4537)

### v2.8.0 (04/01/2021)

- Support new methods: [Cascade RPN](https://arxiv.org/abs/1909.06720), [TridentNet](https://arxiv.org/abs/1901.01892)

- Support [Cascade RPN](https://arxiv.org/abs/1909.06720) (#1900)
- Support [TridentNet](https://arxiv.org/abs/1901.01892) (#3313)

- Fix bug of show result in async_benchmark (#4367)
- Fix scale factor in MaskTestMixin (#4366)
- Fix but when returning indices in `multiclass_nms` (#4362)

- Fix bug of empirical attention in resnext backbone error (#4300)
- Fix bug of `img_norm_cfg` in FCOS-HRNet models with updated performance and models (#4250)
- Fix invalid checkpoint and log in Mask R-CNN models on Cityscapes dataset (#4287)
- Fix bug in distributed sampler when dataset is too small (#4257)
- Fix bug of 'PAFPN has no attribute extra_convs_on_inputs' (#4235)

- Update model url from aws to aliyun (#4349)
- Update ATSS for PyTorch 1.6+ (#4359)
- Update script to install ruby in pre-commit installation (#4360)
- Delete deprecated `mmdet.ops` (#4325)
- Refactor hungarian assigner for more general usage in Sparse R-CNN (#4259)
- Handle scipy import in DETR to reduce package dependencies (#4339)

- Update documentation of usages for config options after MMCV (1.2.3) supports overriding list in config (#4326)
- Update pre-train models of faster rcnn trained on COCO subsets (#4307)
- Avoid zero or too small value for beta in Dynamic R-CNN (#4303)
- Add doccumentation for Pytorch2ONNX (#4271)
- Add deprecated warning FPN arguments (#4264)
- Support returning indices of kept bboxes when using nms (#4251)
- Update type and device requirements when creating tensors `GFLHead` (#4210)
- Update device requirements when creating tensors in `CrossEntropyLoss` (#4224)

### v2.7.0 (30/11/2020)

- Support new method: [DETR](https://arxiv.org/abs/2005.12872), [ResNest](https://arxiv.org/abs/2004.08955), Faster R-CNN DC5.
- Support YOLO, Mask R-CNN, and Cascade R-CNN models exportable to ONNX.

- Support [DETR](https://arxiv.org/abs/2005.12872) (#4201, #4206)
- Support to link the best checkpoint in training (#3773)
- Support to override config through options in inference.py (#4175)
- Support YOLO, Mask R-CNN, and Cascade R-CNN models exportable to ONNX (#4087, #4083)
- Support [ResNeSt](https://arxiv.org/abs/2004.08955) backbone (#2959)
- Support unclip border bbox regression (#4076)
- Add tpfp func in evaluating AP (#4069)
- Support mixed precision training of SSD detector with other backbones (#4081)
- Add Faster R-CNN DC5 models (#4043)

- Fix bug of `gpu_id` in distributed training mode (#4163)
- Support Albumentations with version higher than 0.5 (#4032)
- Fix num_classes bug in faster rcnn config (#4088)

- Update code in docs/2_new_data_model.md (#4041)

- Ensure DCN offset to have similar type as features in VFNet (#4198)
- Add config links in README files of models (#4190)
- Add tutorials for loss conventions (#3818)
- Add solution to installation issues in 30-series GPUs (#4176)
- Update docker version in get_started.md (#4145)
- Add model statistics and polish some titles in configs README (#4140)
- Clamp neg probability in FreeAnchor (#4082)
- Speed up expanding large images (#4089)
- Fix Pytorch 1.7 incompatibility issues (#4103)
- Update trouble shooting page to resolve segmentation fault (#4055)
- Update aLRP-Loss in project page (#4078)
- Clean duplicated `reduce_mean` function (#4056)
- Refactor Q&A (#4045)

### v2.6.0 (1/11/2020)

- Support new method: [VarifocalNet](https://arxiv.org/abs/2008.13367).
- Refactored documentation with more tutorials.

- Support GIoU calculation in `BboxOverlaps2D`, and re-implement `giou_loss` using `bbox_overlaps` (#3936)
- Support random sampling in CPU mode (#3948)
- Support VarifocalNet (#3666, #4024)

- Fix SABL validating bug in Cascade R-CNN (#3913)
- Avoid division by zero in PAA head when num_pos=0 (#3938)
- Fix temporary directory bug of multi-node testing error (#4034, #4017)
- Fix `--show-dir` option in test script (#4025)
- Fix GA-RetinaNet r50 model url (#3983)
- Update code in docs and fix broken urls (#3947)

- Refactor pytorch2onnx API into `mmdet.core.export` and use `generate_inputs_and_wrap_model` for pytorch2onnx (#3857, #3912)

- Update RPN upgrade scripts for v2.5.0 compatibility (#3986)
- Use mmcv `tensor2imgs` (#4010)
- Update test robustness (#4000)
- Update trouble shooting page (#3994)
- Accelerate PAA training speed (#3985)
- Support batch_size > 1 in validation (#3966)
- Use RoIAlign implemented in MMCV for inference in CPU mode (#3930)
- Documentation refactoring (#4031)

### v2.5.0 (5/10/2020)

- Support new methods: [YOLACT](https://arxiv.org/abs/1904.02689), [CentripetalNet](https://arxiv.org/abs/2003.09119).
- Add more documentations for easier and more clear usage.

__FP16 related methods are imported from mmcv instead of mmdet. (#3766, #3822)__

Mixed precision training utils in `mmdet.core.fp16` are moved to `mmcv.runner`, including `force_fp32`, `auto_fp16`, `wrap_fp16_model`, and `Fp16OptimizerHook`. A deprecation warning will be raised if users attempt to import those methods from `mmdet.core.fp16`, and will be finally removed in V2.10.0.

__\[0, N-1\] represents foreground classes and N indicates background classes for all models. (#3221)__
Before v2.5.0, the background label for RPN is 0, and N for other heads. Now the behavior is consistent for all models. Thus `self.background_labels` in `dense_heads` is removed and all heads use `self.num_classes` to indicate the class index of background labels.

This change has no effect on the pre-trained models in the v2.x model zoo, but will affect the training of all models with RPN heads. Two-stage detectors whose RPN head uses softmax will be affected because the order of categories is changed.

**Only call `get_subset_by_classes` when `test_mode=True` and `self.filter_empty_gt=True` (#3695)**
Function `get_subset_by_classes` in dataset is refactored and only filters out images when `test_mode=True` and `self.filter_empty_gt=True`.

In the original implementation, `get_subset_by_classes` is not related to the flag `self.filter_empty_gt` and will only be called when the classes is set during initialization no matter `test_mode` is `True` or `False`. This brings ambiguous behavior and potential bugs in many cases. After v2.5.0, if `filter_empty_gt=False`, no matter whether the classes are specified in a dataset, the dataset will use all the images in the annotations. If `filter_empty_gt=True` and `test_mode=True`, no matter whether the classes are specified, the dataset will call \`\`get_subset_by_classes\` to check the images and filter out images containing no GT boxes. Therefore, the users should be responsible for the data filtering/cleaning process for the test dataset.

- Test time augmentation for single stage detectors (#3844, #3638)

- Support to show the name of experiments during training (#3764)
- Add `Shear`, `Rotate`, `Translate` Augmentation (#3656, #3619, #3687)
- Add image-only transformations including `Constrast`, `Equalize`, `Color`, and `Brightness`. (#3643)
- Support [YOLACT](https://arxiv.org/abs/1904.02689) (#3456)
- Support [CentripetalNet](https://arxiv.org/abs/2003.09119) (#3390)
- Support PyTorch 1.6 in docker (#3905)

- Fix the bug of training ATSS when there is no ground truth boxes (#3702)
- Fix the bug of using Focal Loss when there is `num_pos` is 0 (#3702)
- Fix the label index mapping in dataset browser (#3708)
- Fix Mask R-CNN training stuck problem when their is no positive rois (#3713)
- Fix the bug of `self.rpn_head.test_cfg` in `RPNTestMixin` by using `self.rpn_head` in rpn head (#3808)
- Fix deprecated `Conv2d` from mmcv.ops (#3791)
- Fix device bug in RepPoints (#3836)

- Fix SABL validating bug (#3849)
- Use `https://download.openmmlab.com/mmcv/dist/index.html` for installing MMCV (#3840)
- Fix nonzero in NMS for PyTorch 1.6.0 (#3867)
- Fix the API change bug of PAA (#3883)
- Fix typo in bbox_flip (#3886)
- Fix cv2 import error of ligGL.so.1 in Dockerfile (#3891)

- Change to use `mmcv.utils.collect_env` for collecting environment information to avoid duplicate codes (#3779)
- Update checkpoint file names to v2.0 models in documentation (#3795)
- Update tutorials for changing runtime settings (#3778), modifying loss (#3777)
- Improve the function of `simple_test_bboxes` in SABL (#3853)
- Convert mask to bool before using it as img's index for robustness and speedup (#3870)
- Improve documentation of modules and dataset customization (#3821)

### v2.4.0 (5/9/2020)

- Fix lots of issues/bugs and reorganize the trouble shooting page
- Support new methods [SABL](https://arxiv.org/abs/1912.04260), [YOLOv3](https://arxiv.org/abs/1804.02767), and [PAA Assign](https://arxiv.org/abs/2007.08103)
- Start to publish `mmdet` package to PyPI since v2.3.0

__Backwards Incompatible Changes__

- Support Batch Inference (#3564, #3686, #3705): Since v2.4.0, MMDetection could inference model with multiple images in a single GPU.
  This change influences all the test APIs in MMDetection and downstream codebases. To help the users migrate their code, we use `replace_ImageToTensor` (#3686) to convert legacy test data pipelines during dataset initialization.

- Support RandomFlip with horizontal/vertical/diagonal direction (#3608): Since v2.4.0, MMDetection supports horizontal/vertical/diagonal flip in the data augmentation. This influences bounding box, mask, and image transformations in data augmentation process and the process that will map those data back to the original format.
- Migrate to use `mmlvis` and `mmpycocotools` for COCO and LVIS dataset (#3727). The APIs are fully compatible with the original `lvis` and `pycocotools`. Users need to uninstall the existing pycocotools and lvis packages in their environment first and install `mmlvis` & `mmpycocotools`.

- Fix default mean/std for onnx (#3491)
- Fix coco evaluation and add metric items (#3497)
- Fix typo for install.md (#3516)
- Fix atss when sampler per gpu is 1 (#3528)
- Fix import of fuse_conv_bn (#3529)
- Fix bug of gaussian_target, update unittest of heatmap (#3543)
- Fixed VOC2012 evaluate (#3553)
- Fix scale factor bug of rescale (#3566)

- Fix with_xxx_attributes in base detector (#3567)
- Fix boxes scaling when number is 0 (#3575)
- Fix rfp check when neck config is a list (#3591)
- Fix import of fuse conv bn in benchmark.py (#3606)
- Fix webcam demo (#3634)
- Fix typo and itemize issues in tutorial (#3658)
- Fix error in distributed training when some levels of FPN are not assigned with bounding boxes (#3670)
- Fix the width and height orders of stride in valid flag generation (#3685)
- Fix weight initialization bug in Res2Net DCN (#3714)
- Fix bug in OHEMSampler (#3677)

- Support Cutout augmentation (#3521)
- Support evaluation on multiple datasets through ConcatDataset (#3522)
- Support [PAA assign](https://arxiv.org/abs/2007.08103) #(3547)
- Support eval metric with pickle results (#3607)

- Support [YOLOv3](https://arxiv.org/abs/1804.02767) (#3083)
- Support [SABL](https://arxiv.org/abs/1912.04260) (#3603)
- Support to publish to Pypi in github-action (#3510)
- Support custom imports (#3641)

- Refactor common issues in documentation (#3530)
- Add pytorch 1.6 to CI config (#3532)
- Add config to runner meta (#3534)
- Add eval-option flag for testing (#3537)
- Add init_eval to evaluation hook (#3550)
- Add include_bkg in ClassBalancedDataset (#3577)
- Using config's loading in inference_detector (#3611)
- Add ATSS ResNet-101 models in model zoo (#3639)
- Update urls to download.openmmlab.com (#3665)
- Support non-mask training for CocoDataset (#3711)

### v2.3.0 (5/8/2020)

- The CUDA/C++ operators have been moved to `mmcv.ops`. For backward compatibility `mmdet.ops` is kept as warppers of `mmcv.ops`.
- Support new methods [CornerNet](https://arxiv.org/abs/1808.01244), [DIOU](https://arxiv.org/abs/1911.08287)/[CIOU](https://arxiv.org/abs/2005.03572) loss, and new dataset: [LVIS V1](https://arxiv.org/abs/1908.03195)
- Provide more detailed colab training tutorials and more complete documentation.
- Support to convert RetinaNet from Pytorch to ONNX.

- Fix the model initialization bug of DetectoRS (#3187)
- Fix the bug of module names in NASFCOSHead (#3205)
- Fix the filename bug in publish_model.py (#3237)
- Fix the dimensionality bug when `inside_flags.any()` is `False` in dense heads (#3242)

- Fix the bug of forgetting to pass flip directions in `MultiScaleFlipAug` (#3262)
- Fixed the bug caused by default value of `stem_channels` (#3333)
- Fix the bug of model checkpoint loading for CPU inference (#3318, #3316)
- Fix topk bug when box number is smaller than the expected topk number in ATSSAssigner (#3361)
- Fix the gt priority bug in center_region_assigner.py (#3208)
- Fix NaN issue of iou calculation in iou_loss.py (#3394)
- Fix the bug that `iou_thrs` is not actually used during evaluation in coco.py (#3407)
- Fix test-time augmentation of RepPoints (#3435)
- Fix runtimeError caused by incontiguous tensor in Res2Net+DCN (#3412)

- Support [CornerNet](https://arxiv.org/abs/1808.01244) (#3036)

- Support [DIOU](https://arxiv.org/abs/1911.08287)/[CIOU](https://arxiv.org/abs/2005.03572) loss (#3151)
- Support [LVIS V1](https://arxiv.org/abs/1908.03195) dataset (#)
- Support customized hooks in training (#3395)
- Support fp16 training of generalized focal loss (#3410)
- Support to convert RetinaNet from Pytorch to ONNX (#3075)

- Support to process ignore boxes in ATSS assigner (#3082)
- Allow to crop images without ground truth in `RandomCrop` (#3153)
- Enable the the `Accuracy` module to set threshold (#3155)
- Refactoring unit tests (#3206)
- Unify the training settings of `to_float32` and `norm_cfg` in RegNets configs (#3210)
- Add colab training tutorials for beginners (#3213, #3273)
- Move CUDA/C++ operators into `mmcv.ops` and keep `mmdet.ops` as warppers for backward compatibility (#3232)(#3457)

- Update installation scripts in documentation (#3290) and dockerfile (#3320)
- Support to set image resize backend (#3392)
- Remove git hash in version file (#3466)
- Check mmcv version to force version compatibility (#3460)

### v2.2.0 (1/7/2020)

- Support new methods: [DetectoRS](https://arxiv.org/abs/2006.02334), [PointRend](https://arxiv.org/abs/1912.08193), [Generalized Focal Loss](https://arxiv.org/abs/2006.04388), [Dynamic R-CNN](https://arxiv.org/abs/2004.06002)

- Fix FreeAnchor when no gt in image (#3176)
- Clean up deprecated usage of `register_module()` (#3092, #3161)
- Fix pretrain bug in NAS FCOS (#3145)
- Fix `num_classes` in SSD (#3142)
- Fix FCOS warmup (#3119)
- Fix `rstrip` in `tools/publish_model.py`

- Fix `flip_ratio` default value in RandomFLip pipeline (#3106)
- Fix cityscapes eval with ms_rcnn (#3112)
- Fix RPN softmax (#3056)
- Fix filename of LVIS@v0.5 (#2998)
- Fix nan loss by filtering out-of-frame gt_bboxes in COCO (#2999)
- Fix bug in FSAF (#3018)
- Add FocalLoss `num_classes` check (#2964)
- Fix PISA Loss when there are no gts (#2992)
- Avoid nan in `iou_calculator` (#2975)
- Prevent possible bugs in loading and transforms caused by shallow copy (#2967)

- Add DetectoRS (#3064)
- Support Generalize Focal Loss (#3097)
- Support PointRend (#2752)
- Support Dynamic R-CNN (#3040)
- Add DeepFashion dataset (#2968)
- Implement FCOS training tricks (#2935)
- Use BaseDenseHead as base class for anchor-base heads (#2963)
- Add `with_cp` for BasicBlock (#2891)
- Add `stem_channels` argument for ResNet (#2954)

- Add anchor free base head (#2867)
- Migrate to github action (#3137)
- Add docstring for datasets, pipelines, core modules and methods (#3130, #3125, #3120)
- Add VOC benchmark (#3060)
- Add `concat` mode in GRoI (#3098)
- Remove cmd arg `autorescale-lr` (#3080)
- Use `len(data['img_metas'])` to indicate `num_samples` (#3073, #3053)
- Switch to EpochBasedRunner (#2976)

### v2.1.0 (8/6/2020)

- Support new backbones: [RegNetX](https://arxiv.org/abs/2003.13678), [Res2Net](https://arxiv.org/abs/1904.01169)
- Support new methods: [NASFCOS](https://arxiv.org/abs/1906.04423), [PISA](https://arxiv.org/abs/1904.04821), [GRoIE](https://arxiv.org/abs/2004.13665)
- Support new dataset: [LVIS](https://arxiv.org/abs/1908.03195)

- Change the CLI argument `--validate` to `--no-validate` to enable validation after training epochs by default. (#2651)
- Add missing cython to docker file (#2713)
- Fix bug in nms cpu implementation (#2754)
- Fix bug when showing mask results (#2763)
- Fix gcc requirement (#2806)
- Fix bug in async test (#2820)
- Fix mask encoding-decoding bugs in test API (#2824)
- Fix bug in test time augmentation (#2858, #2921, #2944)
- Fix a typo in comment of apis/train (#2877)
- Fix the bug of returning None when no gt bboxes are in the original image in `RandomCrop`. Fix the bug that misses to handle `gt_bboxes_ignore`, `gt_label_ignore`, and `gt_masks_ignore` in `RandomCrop`, `MinIoURandomCrop` and `Expand` modules. (#2810)
- Fix bug of `base_channels` of regnet (#2917)
- Fix the bug of logger when loading pre-trained weights in base detector (#2936)

- Add IoU models (#2666)

- Add colab demo for inference
- Support class agnostic nms (#2553)
- Add benchmark gathering scripts for development only (#2676)
- Add mmdet-based project links (#2736, #2767, #2895)
- Add config dump in training (#2779)
- Add ClassBalancedDataset (#2721)
- Add res2net backbone (#2237)
- Support RegNetX models (#2710)
- Use `mmcv.FileClient` to support different storage backends (#2712)
- Code Release: Prime Sample Attention in Object Detection (CVPR 2020) (#2626)
- Implement NASFCOS (#2682)
- Add class weight in CrossEntropyLoss (#2797)
- Support LVIS dataset (#2088)
- Support GRoIE (#2584)

- Allow different x and y strides in anchor heads. (#2629)
- Make FSAF loss more robust to no gt (#2680)

- Compute pure inference time instead (#2657) and update inference speed (#2730)
- Avoided the possibility that a patch with 0 area is cropped. (#2704)
- Add warnings when deprecated `imgs_per_gpu` is used. (#2700)
- Add a mask rcnn example for config (#2645)
- Update model zoo (#2762, #2866, #2876, #2879, #2831)
- Add `ori_filename` to img_metas and use it in test show-dir (#2612)
- Use `img_fields` to handle multiple images during image transform (#2800)
- Add upsample_cfg support in FPN (#2787)
- Add `['img']` as default `img_fields` for back compatibility (#2809)
- Rename the pretrained model from `open-mmlab://resnet50_caffe` and `open-mmlab://resnet50_caffe_bgr` to `open-mmlab://detectron/resnet50_caffe` and `open-mmlab://detectron2/resnet50_caffe`. (#2832)

- Added sleep(2) in test.py to reduce hanging problem (#2847)
- Support `c10::half` in CARAFE (#2890)
- Improve documentations (#2918, #2714)
- Use optimizer constructor in mmcv and clean the original implementation in `mmdet.core.optimizer` (#2947)

### v2.0.0 (6/5/2020)

In this release, we made lots of major refactoring and modifications.

1. __Faster speed__. We optimize the training and inference speed for common models, achieving up to 30% speedup for training and 25% for inference. Please refer to [model zoo](model_zoo.md#comparison-with-detectron2) for details.

2. __Higher performance__. We change some default hyperparameters with no additional cost, which leads to a gain of performance for most models. Please refer to [compatibility](compatibility.md#training-hyperparameters) for details.

3. __More documentation and tutorials__. We add a bunch of documentation and tutorials to help users get started more smoothly. Read it [here](https://mmdetection.readthedocs.io/en/latest/).

4. __Support PyTorch 1.5__. The support for 1.1 and 1.2 is dropped, and we switch to some new APIs.

5. __Better configuration system__. Inheritance is supported to reduce the redundancy of configs.

6. __Better modular design__. Towards the goal of simplicity and flexibility, we simplify some encapsulation while add more other configurable modules like BBoxCoder, IoUCalculator, OptimizerConstructor, RoIHead. Target computation is also included in heads and the call hierarchy is simpler.

7. Support new methods: [FSAF](https://arxiv.org/abs/1903.00621) and PAFPN (part of [PAFPN](https://arxiv.org/abs/1803.01534)).

__Breaking Changes__
Models training with MMDetection 1.x are not fully compatible with 2.0, please refer to the [compatibility doc](compatibility.md) for the details and how to migrate to the new version.

- Unify cuda and cpp API for custom ops. (#2277)
- New config files with inheritance. (#2216)
- Encapsulate the second stage into RoI heads. (#1999)
- Refactor GCNet/EmpericalAttention into plugins. (#2345)
- Set low quality match as an option in IoU-based bbox assigners. (#2375)
- Change the codebase's coordinate system. (#2380)
- Refactor the category order in heads. 0 means the first positive class instead of background now. (#2374)
- Add bbox sampler and assigner registry. (#2419)
- Speed up the inference of RPN. (#2420)
- Add `train_cfg` and `test_cfg` as class members in all anchor heads. (#2422)

- Merge target computation methods into heads. (#2429)
- Add bbox coder to support different bbox encoding and losses. (#2480)
- Unify the API for regression loss. (#2156)
- Refactor Anchor Generator. (#2474)
- Make `lr` an optional argument for optimizers. (#2509)
- Migrate to modules and methods in MMCV. (#2502, #2511, #2569, #2572)
- Support PyTorch 1.5. (#2524)
- Drop the support for Python 3.5 and use F-string in the codebase. (#2531)

- Fix the scale factors for resized images without keep the aspect ratio. (#2039)
- Check if max_num > 0 before slicing in NMS. (#2486)
- Fix Deformable RoIPool when there is no instance. (#2490)
- Fix the default value of assigned labels. (#2536)
- Fix the evaluation of Cityscapes. (#2578)

- Add deep_stem and avg_down option to ResNet, i.e., support ResNetV1d. (#2252)
- Add L1 loss. (#2376)
- Support both polygon and bitmap for instance masks. (#2353, #2540)
- Support CPU mode for inference. (#2385)
- Add optimizer constructor for complicated configuration of optimizers. (#2397, #2488)
- Implement PAFPN. (#2392)
- Support empty tensor input for some modules. (#2280)
- Support for custom dataset classes without overriding it. (#2408, #2443)
- Support to train subsets of coco dataset. (#2340)
- Add iou_calculator to potentially support more IoU calculation methods. (2405)
- Support class wise mean AP (was removed in the last version). (#2459)
- Add option to save the testing result images. (#2414)
- Support MomentumUpdaterHook. (#2571)
- Add a demo to inference a single image. (#2605)

### v1.1.0 (24/2/2020)

- Dataset evaluation is rewritten with a unified api, which is used by both evaluation hooks and test scripts.
- Support new methods: [CARAFE](https://arxiv.org/abs/1905.02188).

- The new MMDDP inherits from the official DDP, thus the `__init__` api is changed to be the same as official DDP.
- The `mask_head` field in HTC config files is modified.
- The evaluation and testing script is updated.
- In all transforms, instance masks are stored as a numpy array shaped (n, h, w) instead of a list of (h, w) arrays, where n is the number of instances.

- Fix IOU assigners when ignore_iof_thr > 0 and there is no pred boxes. (#2135)
- Fix mAP evaluation when there are no ignored boxes. (#2116)
- Fix the empty RoI input for Deformable RoI Pooling. (#2099)

- Fix the dataset settings for multiple workflows. (#2103)
- Fix the warning related to `torch.uint8` in PyTorch 1.4. (#2105)
- Fix the inference demo on devices other than gpu:0. (#2098)
- Fix Dockerfile. (#2097)
- Fix the bug that `pad_val` is unused in Pad transform. (#2093)
- Fix the albumentation transform when there is no ground truth bbox. (#2032)

- Use torch instead of numpy for random sampling. (#2094)
- Migrate to the new MMDDP implementation in MMCV v0.3. (#2090)
- Add meta information in logs. (#2086)
- Rewrite Soft NMS with pytorch extension and remove cython as a dependency. (#2056)
- Rewrite dataset evaluation. (#2042, #2087, #2114, #2128)
- Use numpy array for masks in transforms. (#2030)

- Implement "CARAFE: Content-Aware ReAssembly of FEatures". (#1583)

- Add `worker_init_fn()` in data_loader when seed is set. (#2066, #2111)
- Add logging utils. (#2035)

### v1.0.0 (30/1/2020)

This release mainly improves the code quality and add more docstrings.

- Documentation is online now: <https://mmdetection.readthedocs.io>.
- Support new models: [ATSS](https://arxiv.org/abs/1912.02424).
- DCN is now available with the api `build_conv_layer` and `ConvModule` like the normal conv layer.
- A tool to collect environment information is available for trouble shooting.

- Fix the incompatibility of the latest numpy and pycocotools. (#2024)
- Fix the case when distributed package is unavailable, e.g., on Windows. (#1985)
- Fix the dimension issue for `refine_bboxes()`. (#1962)

- Fix the typo when `seg_prefix` is a list. (#1906)
- Add segmentation map cropping to RandomCrop. (#1880)
- Fix the return value of `ga_shape_target_single()`. (#1853)
- Fix the loaded shape of empty proposals. (#1819)
- Fix the mask data type when using albumentation. (#1818)

- Enhance AssignResult and SamplingResult. (#1995)
- Add ability to overwrite existing module in Registry. (#1982)
- Reorganize requirements and make albumentations and imagecorruptions optional. (#1969)
- Check NaN in `SSDHead`. (#1935)
- Encapsulate the DCN in ResNe(X)t into a ConvModule & Conv_layers. (#1894)
- Refactoring for mAP evaluation and support multiprocessing and logging. (#1889)
- Init the root logger before constructing Runner to log more information. (#1865)
- Split `SegResizeFlipPadRescale` into different existing transforms. (#1852)

- Move `init_dist()` to MMCV. (#1851)
- Documentation and docstring improvements. (#1971, #1938, #1869, #1838)
- Fix the color of the same class for mask visualization. (#1834)
- Remove the option `keep_all_stages` in HTC and Cascade R-CNN. (#1806)

- Add two test-time options `crop_mask` and `rle_mask_encode` for mask heads. (#2013)
- Support loading grayscale images as single channel. (#1975)
- Implement "Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection". (#1872)
- Add sphinx generated docs. (#1859, #1864)
- Add GN support for flops computation. (#1850)
- Collect env info for trouble shooting. (#1812)

### v1.0rc1 (13/12/2019)

The RC1 release mainly focuses on improving the user experience, and fixing bugs.

- Support new models: [FoveaBox](https://arxiv.org/abs/1904.03797), [RepPoints](https://arxiv.org/abs/1904.11490) and [FreeAnchor](https://arxiv.org/abs/1909.02466).
- Add a Dockerfile.
- Add a jupyter notebook demo and a webcam demo.
- Setup the code style and CI.
- Add lots of docstrings and unit tests.
- Fix lots of bugs.

- There was a bug for computing COCO-style mAP w.r.t different scales (AP_s, AP_m, AP_l), introduced by #621. (#1679)

- Fix a sampling interval bug in Libra R-CNN. (#1800)
- Fix the learning rate in SSD300 WIDER FACE. (#1781)
- Fix the scaling issue when `keep_ratio=False`. (#1730)
- Fix typos. (#1721, #1492, #1242, #1108, #1107)
- Fix the shuffle argument in `build_dataloader`. (#1693)
- Clip the proposal when computing mask targets. (#1688)

- Fix the "index out of range" bug for samplers in some corner cases. (#1610, #1404)
- Fix the NMS issue on devices other than GPU:0. (#1603)
- Fix SSD Head and GHM Loss on CPU. (#1578)
- Fix the OOM error when there are too many gt bboxes. (#1575)
- Fix the wrong keyword argument `nms_cfg` in HTC. (#1573)
- Process masks and semantic segmentation in Expand and MinIoUCrop transforms. (#1550, #1361)
- Fix a scale bug in the Non Local op. (#1528)
- Fix a bug in transforms when `gt_bboxes_ignore` is None. (#1498)
- Fix a bug when `img_prefix` is None. (#1497)
- Pass the device argument to `grid_anchors` and `valid_flags`. (#1478)
- Fix the data pipeline for test_robustness. (#1476)
- Fix the argument type of deformable pooling. (#1390)
- Fix the coco_eval when there are only two classes. (#1376)

- Fix a bug in Modulated DeformableConv when deformable_group>1. (#1359)
- Fix the mask cropping in RandomCrop. (#1333)
- Fix zero outputs in DeformConv when not running on cuda:0. (#1326)
- Fix the type issue in Expand. (#1288)
- Fix the inference API. (#1255)
- Fix the inplace operation in Expand. (#1249)
- Fix the from-scratch training config. (#1196)
- Fix inplace add in RoIExtractor which cause an error in PyTorch 1.2. (#1160)
- Fix FCOS when input images has no positive sample. (#1136)
- Fix recursive imports. (#1099)

- Print the config file and mmdet version in the log. (#1721)
- Lint the code before compiling in travis CI. (#1715)
- Add a probability argument for the `Expand` transform. (#1651)
- Update the PyTorch and CUDA version in the docker file. (#1615)

- Raise a warning when specifying `--validate` in non-distributed training. (#1624, #1651)
- Beautify the mAP printing. (#1614)
- Add pre-commit hook. (#1536)
- Add the argument `in_channels` to backbones. (#1475)
- Add lots of docstrings and unit tests, thanks to [@Erotemic](https://github.com/Erotemic). (#1603, #1517, #1506, #1505, #1491, #1479, #1477, #1475, #1474)
- Add support for multi-node distributed test when there is no shared storage. (#1399)
- Optimize Dockerfile to reduce the image size. (#1306)
- Update new results of HRNet. (#1284, #1182)
- Add an argument `no_norm_on_lateral` in FPN. (#1240)
- Test the compiling in CI. (#1235)
- Move docs to a separate folder. (#1233)
- Add a jupyter notebook demo. (#1158)
- Support different type of dataset for training. (#1133)

- Use int64_t instead of long in cuda kernels. (#1131)
- Support unsquare RoIs for bbox and mask heads. (#1128)
- Manually add type promotion to make compatible to PyTorch 1.2. (#1114)
- Allowing validation dataset for computing validation loss. (#1093)
- Use `.scalar_type()` instead of `.type()` to suppress some warnings. (#1070)

- Add an option `--with_ap` to compute the AP for each class. (#1549)
- Implement "FreeAnchor: Learning to Match Anchors for Visual Object Detection". (#1391)
- Support [Albumentations](https://github.com/albumentations-team/albumentations) for augmentations in the data pipeline. (#1354)
- Implement "FoveaBox: Beyond Anchor-based Object Detector". (#1339)
- Support horizontal and vertical flipping. (#1273, #1115)

- Implement "RepPoints: Point Set Representation for Object Detection". (#1265)
- Add test-time augmentation to HTC and Cascade R-CNN. (#1251)
- Add a COCO result analysis tool. (#1228)
- Add Dockerfile. (#1168)
- Add a webcam demo. (#1155, #1150)
- Add FLOPs counter. (#1127)
- Allow arbitrary layer order for ConvModule. (#1078)

### v1.0rc0 (27/07/2019)

- Implement lots of new methods and components (Mixed Precision Training, HTC, Libra R-CNN, Guided Anchoring, Empirical Attention, Mask Scoring R-CNN, Grid R-CNN (Plus), GHM, GCNet, FCOS, HRNet, Weight Standardization, etc.). Thank all collaborators!
- Support two additional datasets: WIDER FACE and Cityscapes.
- Refactoring for loss APIs and make it more flexible to adopt different losses and related hyper-parameters.
- Speed up multi-gpu testing.
- Integrate all compiling and installing in a single script.

### v0.6.0 (14/04/2019)

- Up to 30% speedup compared to the model zoo.
- Support both PyTorch stable and nightly version.
- Replace NMS and SigmoidFocalLoss with Pytorch CUDA extensions.

### v0.6rc0(06/02/2019)

- Migrate to PyTorch 1.0.

### v0.5.7 (06/02/2019)

- Add support for Deformable ConvNet v2. (Many thanks to the authors and [@chengdazhi](https://github.com/chengdazhi))
- This is the last release based on PyTorch 0.4.1.

### v0.5.6 (17/01/2019)

- Add support for Group Normalization.
- Unify RPNHead and single stage heads (RetinaHead, SSDHead) with AnchorHead.

### v0.5.5 (22/12/2018)

- Add SSD for COCO and PASCAL VOC.
- Add ResNeXt backbones and detection models.
- Refactoring for Samplers/Assigners and add OHEM.
- Add VOC dataset and evaluation scripts.

### v0.5.4 (27/11/2018)

- Add SingleStageDetector and RetinaNet.

### v0.5.3 (26/11/2018)

- Add Cascade R-CNN and Cascade Mask R-CNN.
- Add support for Soft-NMS in config files.

### v0.5.2 (21/10/2018)

- Add support for custom datasets.
- Add a script to convert PASCAL VOC annotations to the expected format.

### v0.5.1 (20/10/2018)

- Add BBoxAssigner and BBoxSampler, the `train_cfg` field in config files are restructured.
- `ConvFCRoIHead` / `SharedFCRoIHead` are renamed to `ConvFCBBoxHead` / `SharedFCBBoxHead` for consistency.

  <img src="docs/en/_static/image/logo.svg" width="500px"/>
  <br />
  <br />

[![docs](https://readthedocs.org/projects/opencompass/badge)](https://opencompass.readthedocs.io/en)

[📘Documentation](https://opencompass.readthedocs.io/en/latest/) |
[🛠️Installation](https://opencompass.readthedocs.io/en/latest/get_started.html#installation) |

</div>

</div>

</p>

## 🧭	Welcome

to **OpenCompass**!

Just like a compass guides us on our journey, OpenCompass will guide you through the complex landscape of evaluating large language models. With its powerful algorithms and intuitive interface, OpenCompass makes it easy to assess the quality and effectiveness of your NLP models.

> **🔥 Attention**<br />
> We launch the OpenCompass Collabration project, welcome to support diverse evaluation benchmarks into OpenCompass!
> Clike [Issue](https://github.com/InternLM/opencompass/issues/248) for more information.
> Let's work together to build a more powerful OpenCompass toolkit!

- **\[2023.08.25\]**  [**TigerBot**](https://github.com/TigerResearch/TigerBot) team adpots OpenCompass to evaluate their models systematically. We deeply appreciate the community's dedication to transparency and reproducibility in LLM evaluation.
- **\[2023.08.21\]** [**Lagent**](https://github.com/InternLM/lagent) has been released, which is a lightweight framework for building LLM-based agents. We are working with Lagent team to support the evaluation of general tool-use capability, stay tuned! 🔥🔥🔥.

- **\[2023.08.18\]** We have supported evaluation for **multi-modality learning**, include **MMBench, SEED-Bench, COCO-Caption, Flickr-30K, OCR-VQA, ScienceQA** and so on. Leaderboard is on the road. Feel free to try multi-modality evaluation with  OpenCompass ! 🔥🔥🔥.
- **\[2023.08.18\]** [Dataset card](https://opencompass.org.cn/dataset-detail/MMLU) is now online. Welcome new evaluation benchmark  OpenCompass !
- **\[2023.08.11\]** [Model comparison](https://opencompass.org.cn/model-compare/GPT-4,ChatGPT,LLaMA-2-70B,LLaMA-65B) is now online. We hope this feature offers deeper insights!
- **\[2023.08.11\]** We have supported [LEval](https://github.com/OpenLMLab/LEval).

> [More](docs/en/notes/news.md)

## ✨ Introduction

OpenCompass is a one-stop platform for large model evaluation, aiming to provide a fair, open, and reproducible benchmark for large model evaluation. Its main features includes:

- **Comprehensive support for models and datasets**: Pre-support for 20+ HuggingFace and API models, a model evaluation scheme of 50+ datasets with about 300,000 questions, comprehensively evaluating the capabilities of the models in five dimensions.

- **Efficient distributed evaluation**: One line command to implement task division and distributed evaluation, completing the full evaluation of billion-scale models in just a few hours.

- **Diversified evaluation paradigms**: Support for zero-shot, few-shot, and chain-of-thought evaluations, combined with standard or dialogue type prompt templates, to easily stimulate the maximum performance of various models.

- **Modular design with high extensibility**: Want to add new models or datasets, customize an advanced task division strategy, or even support a new cluster management system? Everything about OpenCompass can be easily expanded!

- **Experiment management and reporting mechanism**: Use config files to fully record each experiment, support real-time reporting of results.

## 📊 Leaderboard

We provide [OpenCompass Leaderbaord](https://opencompass.org.cn/rank) for community to rank all public models and API models. If you would like to join the evaluation, please provide the model repository URL or a standard API interface to the email address `opencompass@pjlab.org.cn`.

## 📖 Dataset Support

  <tbody>
      <td>
        <b>Language</b>
      </td>
      <td>
        <b>Knowledge</b>
      </td>
      <td>
        <b>Reasoning</b>
      </td>
      <td>

      <td>
      </td>
      <td>
      </td>
      <td>
      </td>
      <td>
        <b>Comprehensive Examination</b>
      </td>
      <td>
        <b>Understanding</b>
      </td>
    </tr>
      <td>
<summary><b>Word Definition</b></summary>

- WiC

<summary><b>Idiom Learning</b></summary>

- CHID

<summary><b>Semantic Similarity</b></summary>

- AFQMC
- BUSTM

<summary><b>Coreference Resolution</b></summary>

- CLUEWSC
- WSC

<summary><b>Translation</b></summary>

- Flores

      </td>
      <td>
<summary><b>Knowledge Question Answering</b></summary>

- BoolQ

- Flores

      </td>
      <td>

- BoolQ

<summary><b>Multi-language Question Answering</b></summary>

- TyDi-QA

      </td>
      <td>
<summary><b>Textual Entailment</b></summary>

- CMNLI
- OCNLI
- OCNLI_FC
- AX-b
- AX-g
- CB
- RTE

<summary><b>Commonsense Reasoning</b></summary>

- StoryCloze-CN (coming soon)
- COPA
- ReCoRD
- PIQA
- SIQA

<summary><b>Mathematical Reasoning</b></summary>

- MATH
- GSM8K

<summary><b>Theorem Application</b></summary>

<summary><b>Code</b></summary>

- MBPP

<summary><b>Comprehensive Reasoning</b></summary>

- BBH

      </td>
      <td>

- MBPP

- BBH

      </td>
      <td>
<summary><b>Junior High, High School, University, Professional Examinations</b></summary>

- CEval
- AGIEval
- MMLU
- CMMLU
- ARC

      </td>
      <td>
<summary><b>Reading Comprehension</b></summary>

- C3
- CMRC
- DRCD
- MultiRC
- RACE

<summary><b>Content Summary</b></summary>

- CSL
- LCSTS
- XSum

<summary><b>Content Analysis</b></summary>

- EPRSTMT
- LAMBADA
- TNEWS

      </td>
    </tr>
</td>
    </tr>
  </tbody>
</table>

## 📖 Model Support

  <tbody>
      <td>
        <b>Open-source Models</b>
      </td>
      <td>
        <b>API Models</b>

  <tbody>
      <td>
      </td>
      <td>
      </td>
      <!-- <td>
        <b>Custom Models</b>
      </td> -->
    </tr>
      <td>

- InternLM
- LLaMA
- Vicuna
- Alpaca
- Baichuan
- WizardLM
- MPT
- Falcon
- TigerBot
- MOSS
- ...

</td>
<td>

- OpenAI
- Claude (coming soon)
- PaLM (coming soon)
- ……

</td>

<!--
- GLM
- ...

</td> -->

</tr>
  </tbody>
</table>

Below are the steps for quick installation and datasets preparation.

```Python
# Download dataset to data/ folder

```

Some third-party features, like Humaneval and Llama, may require additional steps to work properly, for detailed steps please refer to the [Installation Guide](https://opencompass.readthedocs.io/en/latest/get_started.html).

## 🏗️ ️Evaluation

Make sure you have installed OpenCompass correctly and prepared your datasets according to the above steps. Please read the [Quick Start](https://opencompass.readthedocs.io/en/latest/get_started.html#quick-start) to learn how to run an evaluation task.

For more tutorials, please check our [Documentation](https://opencompass.readthedocs.io/en/latest/index.html).

## 🔜 Roadmap

- [ ] Subjective Evaluation

  - [ ] Release CompassAreana
  - [ ] Subjective evaluation dataset.
- [ ] Long-context
  - [ ] Long-context evaluation with extensive datasets.
  - [ ] Long-context leaderboard.
- [ ] Coding
  - [ ] Coding evaluation leaderdboard.
  - [ ] Non-python language evaluation service.
- [ ] Agent
  - [ ] Support various agenet framework.
  - [ ] Evaluation of tool use of the LLMs.
- [ ] Robustness
  - [ ] Support various attack method

## 👷‍♂️ Contributing

We appreciate all contributions to improve OpenCompass. Please refer to the [contributing guideline](https://opencompass.readthedocs.io/en/latest/notes/contribution_guide.html) for the best practice.

## 🤝 Acknowledgements

Some code in this project is cited and modified from [OpenICL](https://github.com/Shark-NLP/OpenICL).

Some datasets and prompt implementations are modified from [chain-of-thought-hub](https://github.com/FranxYao/chain-of-thought-hub) and [instruct-eval](https://github.com/declare-lab/instruct-eval).

```bibtex
}
```

# Disco Diffusion (2022)

> [Disco Diffusion](https://github.com/alembics/disco-diffusion)

> **任务**: 图文生成, 图像到图像的翻译, 扩散模型

## 摘要

Disco Diffusion(DD)是一个 Google Colab 笔记本，它利用一种叫做 CLIP-Guided Diffusion 的人工智能图像生成技术，让你从文本输入中创造出引人注目的精美图像。

由 Somnai 创建，由 Gandamu 改进，并建立在 RiversHaveWings、nshepperd 和许多其他人的工作之上。更多细节见[Credits](#credits)。

<thead>
  <tr>
    <td>

<img src="https://user-images.githubusercontent.com/22982797/204526957-ac30547e-5a44-417a-aaa2-6b357b4a139c.png" width="400"/>
    <td>

<img src="https://user-images.githubusercontent.com/22982797/215749979-1ea973c4-3e76-4204-9fa0-b0adf3e942b6.png" width="400"/>
    <td>

<img src="https://user-images.githubusercontent.com/22982797/215757871-d38e1b78-fee0-4351-be61-5b1e782d1e6e.png" width="400"/>
  </tr>
</thead>
</table>

## 模型与结果

  </tr>
</thead>
</table>

## 模型与结果

我们已经转换了几个 `unet` 的权重，并提供相关的配置文件。在[Tutorial](#tutorials)中可以看到更多关于不同 `unet` 的细节。

|                                               模型                                               |  数据集  |                                               下载                                               |
| :----------------------------------------------------------------------------------------------: | :------: | :----------------------------------------------------------------------------------------------: |
| [512x512_diffusion_uncond_finetune_008100](./disco-diffusion_adm-u-finetuned_imagenet-512x512.py) | ImageNet | [model](https://download.openmmlab.com/mmediting/synthesizers/disco/adm-u_finetuned_imagenet-512x512-ab471d70.pth) |

|        [256x256_diffusion_uncond](./disco-diffusion_adm-u-finetuned_imagenet-256x256.py)         | ImageNet |                                           [model](<>)                                            |
|             [portrait_generator_v001](./disco-diffusion_portrait-generator-v001.py)              | unknown  | [model](https://download.openmmlab.com/mmediting/synthesizers/disco/adm-u-cvt-rgb_portrait-v001-f4a3f3bc.pth) |

|             模型             |     下载     |
| :--------------------------: | :----------: |
|  pixelartdiffusion_expanded  | Coming soon! |
| pixel_art_diffusion_hard_256 | Coming soon! |
| pixel_art_diffusion_soft_256 | Coming soon! |
|     pixelartdiffusion4k      | Coming soon! |
|    watercolordiffusion_2     | Coming soon! |
|     watercolordiffusion      | Coming soon! |
|      PulpSciFiDiffusion      | Coming soon! |

## 待办列表

- [x] 图文生成
- [x] 图像到图像的翻译
- [x] Imagenet, portrait 扩散模型
- \[\] 像素艺术，水彩，科幻小说的扩散模型
- \[\] 支持图像提示

## 待办列表

- [x] 图文生成
- \[\] 支持视频生成
- \[\] 支持更快的采样器(plms，dpm-solver等)

我们很欢迎社区用户支持这些项目和任何其他有趣的工作!

运行以下代码，你可以使用文本生成图像。

```python
from mmagic.utils import register_all_modules

disco = MODELS.build(
    Config.fromfile('configs/disco_diffusion/disco-baseline.py').model).cuda().eval()
text_prompts = {
    0: [
        "A beautiful painting of a singular lighthouse, shining its light across a tumultuous sea of blood by greg rutkowski and thomas kinkade, Trending on artstation.",
        "yellow color scheme"
    ]
}
image = disco.infer(
    height=768,
    width=1280,
    text_prompts=text_prompts,
    show_progress=True,
    num_inference_steps=250,
    eta=0.8)['samples']

"yellow color scheme"
    ]
}
save_image(image, "image.png")

```

## 教程

考虑到`disco-diffusion`包含许多可调整的参数，我们为用户提供了一个[jupyter-notebook](./tutorials.ipynb)/[colab](https://githubtocolab.com/open-mmlab/mmagic/blob/main/configs/disco_diffusion/tutorials.ipynb)的教程，展示了不同参数的含义，并给出相应的调整结果。
请参考[Disco Sheet](https://docs.google.com/document/d/1l8s7uS2dGqjztYSjPpzlmXLjl5PM3IGkRWI3IiCuK7g/edit)。

## 鸣谢

Since our adaptation of disco-diffusion are heavily influenced by disco [colab](https://colab.research.google.com/github/alembics/disco-diffusion/blob/main/Disco_Diffusion.ipynb#scrollTo=License), here we copy the credits below.

<details>
<summary>鸣谢</summary>

<details>
Original notebook by Katherine Crowson (https://github.com/crowsonkb, https://twitter.com/RiversHaveWings). It uses either OpenAI's 256x256 unconditional ImageNet or Katherine Crowson's fine-tuned 512x512 diffusion model (https://github.com/openai/guided-diffusion), together with CLIP (https://github.com/openai/CLIP) to connect text prompts with images.

Modified by Daniel Russell (https://github.com/russelldc, https://twitter.com/danielrussruss) to include (hopefully) optimal params for quick generations in 15-100 timesteps rather than 1000, as well as more robust augmentations.

Further improvements from Dango233 and nshepperd helped improve the quality of diffusion in general, and especially so for shorter runs like this notebook aims to achieve.

Vark added code to load in multiple Clip models at once, which all prompts are evaluated against, which may greatly improve accuracy.

The latest zoom, pan, rotation, and keyframes features were taken from Chigozie Nri's VQGAN Zoom Notebook (https://github.com/chigozienri, https://twitter.com/chigozienri)

Advanced DangoCutn Cutout method is also from Dango223.

\--

Disco:

Somnai (https://twitter.com/Somnai_dreams) added Diffusion Animation techniques, QoL improvements and various implementations of tech and techniques, mostly listed in the changelog below.

3D animation implementation added by Adam Letts (https://twitter.com/gandamu_ml) in collaboration with Somnai. Creation of disco.py and ongoing maintenance.

Turbo feature by Chris Allen (https://twitter.com/zippy731)

Improvements to ability to run on local systems, Windows support, and dependency installation by HostsServer (https://twitter.com/HostsServer)

VR Mode by Tom Mason (https://twitter.com/nin_artificial)

Horizontal and Vertical symmetry functionality by nshepperd. Symmetry transformation_steps by huemin (https://twitter.com/huemin_art). Symmetry integration into Disco Diffusion by Dmitrii Tochilkin (https://twitter.com/cut_pow).

Warp and custom model support by Alex Spirin (https://twitter.com/devdef).

Pixel Art Diffusion, Watercolor Diffusion, and Pulp SciFi Diffusion models from KaliYuga (https://twitter.com/KaliYuga_ai). Follow KaliYuga's Twitter for the latest models and for notebooks with specialized settings.

Integration of OpenCLIP models and initiation of integration of KaliYuga models by Palmweaver / Chris Scalf (https://twitter.com/ChrisScalf11)

Integrated portrait_generator_v001 from Felipe3DArtist (https://twitter.com/Felipe3DArtist)

```bibtex
@misc{github,
  author={alembics},
  title={disco-diffusion},
  url={https://github.com/alembics/disco-diffusion},
}
```

# How to Get the Complete Config

We also provide the `print_config.py` tools to print the complete configuration of the given experiment.
You can check each item of the config before the training by using the following command.

`tools/misc/print_config.py` prints the whole config verbatim, expanding all its imports.

```shell
```

```shell
# Print a complete config

# Save the complete config to a independent config file.
```

# Build for Jetson

In this chapter, we introduce how to install MMDeploy on NVIDIA Jetson platforms, which we have verified on the following modules:

Hardware recommendation:

- [Seeed reComputer built with Jetson Nano module](https://www.seeedstudio.com/Jetson-10-1-A0-p-5336.html)
- [Seeed reComputer built with Jetson Xavier NX module](https://www.seeedstudio.com/Jetson-20-1-H1-p-5328.html)

- To equip a Jetson device, JetPack SDK is a must.
- The Model Converter of MMDeploy requires an environment with PyTorch for converting PyTorch models to ONNX models.
- Regarding the toolchain, CMake and GCC has to be upgraded to no less than 3.14 and 7.0 respectively.

JetPack SDK provides a full development environment for hardware-accelerated AI-at-the-edge development.
All Jetson modules and developer kits are supported by JetPack SDK.

There are two major installation methods including,

1. SD Card Image Method
2. NVIDIA SDK Manager Method

You can find a very detailed installation guide from NVIDIA [official website](https://developer.nvidia.com/jetpack-sdk-461).

```{note}
Please select the option to install "Jetson SDK Components" when using NVIDIA SDK Manager as this includes CUDA and TensorRT which are needed for this guide.
```

Here we have chosen [JetPack 4.6.1](https://developer.nvidia.com/jetpack-sdk-461) as our best practice on setting up Jetson platforms. MMDeploy has been tested on JetPack 4.6 (rev.3) and above and TensorRT 8.0.1.6 and above. Earlier JetPack versions has incompatibilities with TensorRT 7.x

### Conda

Install [Archiconda](https://github.com/Archiconda/build-tools/releases) instead of Anaconda because the latter does not provide the wheel built for Jetson.

```shell

### Conda

```shell

```

After the installation, create a conda environment and activate it.

```shell
# get the version of python3 installed by default
```

```{note}

```

```{note}
JetPack SDK 4+ provides Python 3.6. We strongly recommend using the default Python. Trying to upgrade it will probably ruin the JetPack environment.

If a higher-version of Python is necessary, you can install JetPack 5+, in which the Python version is 3.8.
```

Download the PyTorch wheel for Jetson from [here](https://forums.developer.nvidia.com/t/pytorch-for-jetson-version-1-11-now-available/72048) and save it to the `/home/username` directory. Build torchvision from source as there is no prebuilt torchvision for Jetson platforms.

Take `torch 1.10.0` and  `torchvision 0.11.1` for example. You can install them as below:

```shell
# pytorch

```shell
# pytorch

sudo apt-get install libjpeg-dev zlib1g-dev libpython3-dev libavcodec-dev libavformat-dev libswscale-dev libopenblas-base libopenmpi-dev  libopenblas-dev -y
git clone --branch v0.11.1 https://github.com/pytorch/vision torchvision
```

```{note}
It takes about 30 minutes to install torchvision on a Jetson Nano. So, please be patient until the installation is complete.
```

If you install other versions of PyTorch and torchvision, make sure the versions are compatible. Refer to the compatibility chart listed [here](https://pypi.org/project/torchvision/).

### CMake

We use the latest cmake v3.23.1 released in April 2022.

```shell
sudo apt-get purge cmake -y

### CMake

```shell

```

The Model Converter of MMDeploy on Jetson platforms depends on [MMCV](https://github.com/open-mmlab/mmcv) and the inference engine [TensorRT](https://developer.nvidia.com/tensorrt).
While MMDeploy C/C++ Inference SDK relies on [spdlog](https://github.com/gabime/spdlog), OpenCV and [ppl.cv](https://github.com/openppl-public/ppl.cv) and so on, as well as TensorRT.
Thus, in the following sections, we will describe how to prepare TensorRT.

And then, we will present the way to install dependencies of Model Converter and C/C++ Inference SDK respectively.

### Prepare TensorRT

TensorRT is already packed into JetPack SDK. But In order to import it successfully in conda environment, we need to copy the tensorrt package to the conda environment created before.

```shell
python -c "import tensorrt; print(tensorrt.__version__)" # Will print the version of TensorRT

# set environment variable for building mmdeploy later on

# append cuda path and libraries to PATH and LD_LIBRARY_PATH, which is also used for building mmdeploy later on.

# this is not needed if you use NVIDIA SDK Manager with "Jetson SDK Components" for installing JetPack.
# this is only needed if you install JetPack using SD Card Image Method.
```

You can also make the above environment variables permanent by adding them to `~/.bashrc`.

```shell

```

#### Install MMCV

[MMCV](https://github.com/open-mmlab/mmcv) has not provided prebuilt package for Jetson platforms, so we have to build it from source.

```shell
sudo apt-get install -y libssl-dev
git clone --branch 2.x https://github.com/open-mmlab/mmcv.git
cd mmcv
MMCV_WITH_OPS=1 pip install -e .
```

```{note}
It takes about 1 hour 40 minutes to install MMCV on a Jetson Nano. So, please be patient until the installation is complete.
```

#### Install ONNX

Don't install the latest ONNX. The recommended ONNX version is 1.10.0.

```shell
# Execute one of the following commands
pip install onnx==1.10.0

```

```shell
conda install -c conda-forge onnx
```

If the installation failed and showed the following error:

```
CMake Error at CMakeLists.txt:299 (message):
      Protobuf compiler not found
```

Please install dependencies:

```shell
sudo apt-get install protobuf-compiler libprotoc-dev
```

#### Install ONNX Runtime [Optional]

Go to [Jetson_Zoo#ONNX_Runtime](https://elinux.org/Jetson_Zoo#ONNX_Runtime) to find **right version** of onnx runtime. Then download and install package.

```

```

#### Install h5py and pycuda

```

Model Converter employs HDF5 to save the calibration data for TensorRT INT8 quantization and needs `pycuda` to copy device memory.

```shell
sudo apt-get install -y pkg-config libhdf5-100 libhdf5-dev
pip install versioned-hdf5 pycuda
```

```{note}
It takes about 6 minutes to install versioned-hdf5 on a Jetson Nano. So, please be patient until the installation is complete.
```

### Install Dependencies for C/C++ Inference SDK

#### Install spdlog

[spdlog](https://github.com/gabime/spdlog) is a very fast, header-only/compiled, C++ logging library

```shell
sudo apt-get install -y libspdlog-dev
```

#### Install ppl.cv

[ppl.cv](https://github.com/openppl-public/ppl.cv) is a high-performance image processing library of [openPPL](https://openppl.ai/home)

```shell
cd ppl.cv

```shell
cd ppl.cv
echo -e '\n# set environment variable for ppl.cv' >> ~/.bashrc
```

```{note}
It takes about 15 minutes to install ppl.cv on a Jetson Nano. So, please be patient until the installation is complete.
```

## Install MMDeploy

```shell
```

Since some operators adopted by OpenMMLab codebases are not supported by TensorRT, we build the custom TensorRT plugins to make it up, such as `roi_align`, `scatternd`, etc.
You can find a full list of custom plugins from [here](../06-custom-ops/tensorrt.md).

```shell
# build TensorRT custom operators

# install model converter

```shell

# thus any local modifications made to the code will take effect without re-installation.
```

```{note}
It takes about 5 minutes to install model converter on a Jetson Nano. So, please be patient until the installation is complete.
```

### Install C/C++ Inference SDK

Build SDK Libraries and its demo as below:

```shell
cmake .. \
    -DMMDEPLOY_BUILD_SDK=ON \
    -DMMDEPLOY_BUILD_SDK_PYTHON_API=ON \
    -DMMDEPLOY_BUILD_EXAMPLES=ON \
    -DMMDEPLOY_TARGET_DEVICES="cuda;cpu" \
    -DMMDEPLOY_TARGET_BACKENDS="trt" \
    -DMMDEPLOY_CODEBASES=all \
    -Dpplcv_DIR=${PPLCV_DIR}/cuda-build/install/lib/cmake/ppl
```

```{note}

-DMMDEPLOY_TARGET_BACKENDS="trt" \
```

```{note}
It takes about 9 minutes to build SDK libraries on a Jetson Nano. So, please be patient until the installation is complete.
```

### Run a Demo

#### Object Detection demo

Before running this demo, you need to convert model files to be able to use with this SDK.

1. Install [MMDetection](https://github.com/open-mmlab/mmdetection) which is needed for model conversion

MMDetection is an open source object detection toolbox based on PyTorch

```shell
git clone -b 3.x https://github.com/open-mmlab/mmdetection.git
pip install -r requirements/build.txt
```

2. Follow [this document](../02-how-to-run/convert_model.md) on how to convert model files.

```

For this example, we have used [retinanet_r18_fpn_1x_coco.py](https://github.com/open-mmlab/mmdetection/blob/3.x/configs/retinanet/retinanet_r18_fpn_1x_coco.py) as the model config, and [this file](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r18_fpn_1x_coco/retinanet_r18_fpn_1x_coco_20220407_171055-614fd399.pth) as the corresponding checkpoint file. Also for deploy config, we have used [detection_tensorrt_dynamic-320x320-1344x1344.py](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmdet/detection/detection_tensorrt_dynamic-320x320-1344x1344.py)

```shell
    $PATH_TO_MMDET/configs/retinanet/retinanet_r18_fpn_1x_coco.py \

```shell
    retinanet_r18_fpn_1x_coco_20220407_171055-614fd399.pth \
    --show \
```

3. Finally run inference on an image

<div align=center><img width=650 src="https://files.seeedstudio.com/wiki/open-mmlab/source_image.jpg"/></div>

```shell
./object_detection cuda ${directory/to/the/converted/models} ${path/to/an/image}
```

<div align=center><img width=650 src="https://files.seeedstudio.com/wiki/open-mmlab/output_detection.png"/></div>

The above inference is done on a [Seeed reComputer built with Jetson Nano module](https://www.seeedstudio.com/Jetson-10-1-A0-p-5336.html)

- `pip install` throws an error like `Illegal instruction (core dumped)`

  Check if you are using any mirror, if you did, try this:

  ```shell
  ```

### Runtime

  1. Set `MAX N` mode and perform `sudo nvpmodel -m 0 && sudo jetson_clocks`.
  2. Reduce the number of `pre_top_k` in deploy config file like [mmdet pre_top_k](https://github.com/open-mmlab/mmdeploy/blob/34879e638cc2db511e798a376b9a4b9932660fe1/configs/mmdet/_base_/base_static.py#L13) does, e.g., `1000`.
  3. Convert the model again and try SDK demo again.

### FAQ

- Error `error: cannot import name 'ProcessGroup' from 'torch.distributed'.`

3. Convert the model again and try SDK demo again.

### FAQ

  - Please try install pytorch 1.11 version from [pytorch-for-jetson](https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048).

# ORVS (Online Retinal image for Vessel Segmentation (ORVS))

This project supports **`ORVS (Online Retinal image for Vessel Segmentation (ORVS))`**, which can be downloaded from [here](https://opendatalab.org.cn/ORVS).

The ORVS dataset is a newly established collaboration between the Department of Computer Science and the Department of Vision Science at the University of Calgary. The dataset contains 49 images collected from a clinic in Calgary, Canada, consisting of 42 training images and 7 testing images. All images were obtained using a Zeiss Visucam 200 with a 30-degree field of view (FOV). The image size is 1444×1444 pixels with 24 bits per pixel. The images are stored in JPEG format with low compression, which is common in ophthalmic practice. All images were manually traced by an expert who has been working in the field of retinal image analysis and has been trained to mark all pixels belonging to retinal vessels. The Windows Paint 3D tool was used for manual image annotation.

| Dataset name                                           | Anatomical region | Task type    | Modality           | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License |
| ------------------------------------------------------ | ----------------- | ------------ | ------------------ | ------------ | --------------------- | ---------------------- | ------------ | ------- |
| [Bactteria detection](https://opendatalab.org.cn/ORVS) | bacteria          | segmentation | fundus photography | 2            | 130/-/72              | yes/-/yes              | 2020         | -       |

| background |    130     |   94.83    |    -     |    -     |    72     |   94.25   |
|   vessel   |    130     |    5.17    |    -     |    -     |    72     |   5.75    |

Note:

![bac](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/fundus_photography/orvs/ORVS_dataset.png)

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `orvs/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell

```shell
```

- Clone this [repository](https://github.com/AbdullahSarhan/ICPRVessels), then move `Vessels-Datasets` to `data/`.

```none
  ├── mmseg
  │   │   │   │   ├── orvs

```

```shell
```

```shell
```

```

```shell
```

```
@inproceedings{sarhan2021transfer,
  title={Transfer learning through weighted loss function and group normalization for vessel segmentation from retinal images},
  author={Sarhan, Abdullah and Rokne, Jon and Alhajj, Reda and Crichton, Andrew},

author={Sarhan, Abdullah and Rokne, Jon and Alhajj, Reda and Crichton, Andrew},
  pages={9211--9218},
}
```

# 准备视频检索数据集

## 简介

```BibTeX
@inproceedings{xu2016msr,
      title={Msr-vtt: A large video description dataset for bridging video and language},
      author={Xu, Jun and Mei, Tao and Yao, Ting and Rui, Yong},
      booktitle={CVPR},
      pages={5288--5296},
      year={2016}
}
```

在数据集准备前，请确保命令行当前路径为 `$MMACTION2/tools/data/video_retrieval/`。

## 准备 MSRVTT 数据集

用户可参考该数据集的[官网](https://www.microsoft.com/en-us/research/publication/msr-vtt-a-large-video-description-dataset-for-bridging-video-and-language/)，以获取数据集相关的基本信息。运行下面的命令准备 MSRVTT 数据集：

```shell
bash prepare_msrvtt.sh
```

完场上述准备步骤后，文件目录如下：

```
mmaction2
├── tools
├── data
│   ├── video_retrieval
│   │   └── msrvtt
│   │       ├── train_9k.json
│   │       ├── train_7k.json
│   │       ├── test_JSFUSION.json
│   │       └─── videos
│   │           ├── video0.mp4
│   │           ├── video1.mp4
│   │           ├── ...
│   │           └── video9999.mp4
```

# How to design your own loss functions

`losses` are registered as `LOSSES` in `MMagic`.
Customizing losses is similar to customizing any other model.
This section is mainly for clarifying the design of loss modules in MMagic.
Importantly, when writing your own loss modules, you should follow the same design,
so that the new loss module can be adopted in our framework without extra effort.

This guides includes:

- [How to design your own loss functions](#how-to-design-your-own-loss-functions)
  - [Introduction to supported losses](#introduction-to-supported-losses)
  - [Design a new loss function](#design-a-new-loss-function)
    - [An example of MSELoss](#an-example-of-mseloss)
    - [An example of DiscShiftLoss](#an-example-of-discshiftloss)
    - [An example of GANWithCustomizedLoss](#an-example-of-ganwithcustomizedloss)
  - [Available losses](#available-losses)
    - [regular losses](#regular-losses)
    - [losses components](#losses-components)

- [Available losses](#available-losses)

## Introduction to supported losses

For convenient usage, you can directly use default loss calculation process we set for concrete algorithms like lsgan, biggan, styleganv2 etc.
Take `stylegan2` as an example, we use R1 gradient penalty and generator path length regularization as configurable losses, and users can adjust
related arguments like `r1_loss_weight` and `g_reg_weight`.

```python
loss_config = dict(
    r1_loss_weight=10. / 2. * d_reg_interval,
    r1_interval=d_reg_interval,
    norm_mode='HWC',
    g_reg_interval=g_reg_interval,
    g_reg_weight=2. * g_reg_interval,
    pl_batch_shrink=2)

    type='StyleGAN2',
    xxx,
    loss_config=loss_config)
```

## Design a new loss function

### An example of MSELoss

g_reg_weight=2. * g_reg_interval,

    xxx,
```

In general, to implement a loss module, we will write a function implementation and then wrap it with a class implementation. Take the MSELoss as an example:

```python
def mse_loss(pred, target):
    return F.mse_loss(pred, target, reduction='none')

class MSELoss(nn.Module):

    def __init__(self, loss_weight=1.0, reduction='mean', sample_wise=False):
        # codes can be found in ``mmagic/models/losses/pixelwise_loss.py``

    def forward(self, pred, target, weight=None, **kwargs):
```

Given the definition of the loss, we can now use the loss by simply defining it in the configuration file:

```python
```

```

```python
```

Note that `pixel_loss` above must be defined in the model. Please refer to `customize_models` for more details. Similar to model customization, in order to use your customized loss, you need to import the loss in `mmagic/models/losses/__init__.py` after writing it.

### An example of DiscShiftLoss

In general, to implement a loss module, we will write a function implementation and then wrap it with a class implementation.
However, in `MMagic`, we provide another unified interface `data_info` for users to define the mapping between the input argument and data items.

```python
def disc_shift_loss(pred):
    return pred**2

```python

        # codes can be found in ``mmagic/models/losses/disc_auxiliary_loss.py``

```

The goal of this design for loss modules is to allow for using it automatically in the generative models (`MODELS`), without other complex codes to define the mapping between data and keyword arguments. Thus, different from other frameworks in `OpenMMLab`, our loss modules contain a special keyword, `data_info`, which is a dictionary defining the mapping between the input arguments and data from the generative models. Taking the `DiscShiftLoss` as an example, when writing the config file, users may use this loss as follows:

```python

```python
    data_info=dict(pred='disc_pred_real')
```

The information in `data_info` tells the module to use the `disc_pred_real` data as the input tensor for `pred` arguments. Once the `data_info` is not `None`, our loss module will automatically build up the computational graph.

```python

if len(args) == 1:
            else:
            }

loss_input_dict = {
            }
        else:

```

As shown in this part of codes, once users set the `data_info`, the loss module will receive a dictionary containing all of the necessary data and modules, which is provided by the `MODELS` in the training procedure. If this dictionary is given as a non-keyword argument, it should be offered as the first argument. If you are using a keyword argument, please name it as `outputs_dict`.

### An example of GANWithCustomizedLoss

To build the computational graph, the generative models have to provide a dictionary containing all kinds of data. Having a close look at any generative model, you will find that we collect all kinds of features and modules into a dictionary. We provide a customized `GANWithCustomizedLoss` here to show the process.

```python

        # ...
        else:

        else:

if not isinstance(self.disc_auxiliary_losses, nn.ModuleList):
        else:

        else:

        # ...

iteration=curr_iter,

        # ...

        # gan loss

losses_dict['loss_disc_real'] = self.gan_loss(

                    continue

                else:

```

Here, the `_get_disc_loss` will help to combine all kinds of losses automatically.

loss, log_var = self.parse_losses(losses_dict)

```

Therefore, as long as users design the loss module with the same rules, any kind of loss can be inserted in the training of generative models,
without other modifications in the code of models. What you only need to do is just defining the `data_info` in the config files.

## Available losses

We list available losses with examples in configs as follows.

### regular losses

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
<td>

```python
# dic gan
)

```

</td>

</tr>
  <tr>
<td>
</td>

</tr>
  <tr>

<tr>
    <td>

```python
# ttsr

```

</td>
  </tr>
</tbody>
</table>

### losses components

For `GANWithCustomizedLoss`, we provide several components to build customized loss.

### 源码安装

- 确保物理机环境的 gcc 版本不低于 9，可以通过`gcc --version`确认。
- 安装编译和运行依赖包：
  ```shell
  ```
- 安装 [nccl](https://docs.nvidia.com/deeplearning/nccl/install-guide/index.html),设置环境变量
  ```shell
  ```
- rapidjson 安装
- openmpi 安装, 推荐从源码安装:
  ```shell
  ```
- lmdeploy 编译安装:
  ```shell
  ```

# Prompt 概括

提示词 (prompt) 是 LLM 的输入，用于让 LLM 往后续写内容或计算困惑度 (ppl)，提示词的选取会对被评测模型的精度产生重大影响。如何将数据集转换为一系列的提示词的过程是由模板 (template) 来定义的。

在 OpenCompass 中，我们将 template 拆分为两部分：数据侧的 template 和模型侧的 template。在测评模型时，数据会先后经过数据和模型侧的 template，最终转化为模型所需的输入。

数据侧的 template 被称为 [prompt_template](./prompt_template.md)，它表示了把数据集的字段转化成提示词的过程。

模型侧的 template 被称为 [meta_template](./meta_template.md)，它表示了模型将这些提示词转化为自身期望的输入的过程。

我们另外还提供了一些 [思维链](./chain_of_thought.md) 的 prompt 示例。

# MMPretrain Gradio Demo

Here is a gradio demo for MMPretrain supported inference tasks.

- Image Classifiation
- Image-To-Image Retrieval
- Text-To-Image Retrieval (require multi-modality support)
- Image Caption (require multi-modality support)
- Visual Question Answering (require multi-modality support)
- Visual Grounding (require multi-modality support)

## Preview

To run the demo, you need to install MMPretrain at first. And please install with the extra multi-modality

```shell
# At the MMPretrain root folder
```

```shell
```

## Start

```shell
# At the project folder
python launch.py
```

```shell
```

## Start

```shell
```

And to share it to others, please set `share=True` in the `demo.launch()`.

# BEiT

We introduce a self-supervised vision representation model BEiT, which stands for Bidirectional Encoder representation from Image Transformers. Following BERT developed in the natural language processing area, we propose a masked image modeling task to pretrain vision Transformers. Specifically, each image has two views in our pre-training, i.e, image patches (such as 16x16 pixels), and visual tokens (i.e., discrete tokens). We first "tokenize" the original image into visual tokens. Then we randomly mask some image patches and fed them into the backbone Transformer. The pre-training objective is to recover the original visual tokens based on the corrupted image patches. After pre-training BEiT, we directly fine-tune the model parameters on downstream tasks by appending task layers upon the pretrained encoder. Experimental results on image classification and semantic segmentation show that our model achieves competitive results with previous pre-training methods. For example, base-size BEiT achieves 83.2% top-1 accuracy on ImageNet-1K, significantly outperforming from-scratch DeiT training (81.8%) with the same setup. Moreover, large-size BEiT obtains 86.3% only using ImageNet-1K, even outperforming ViT-L with supervised pre-training on ImageNet-22K (85.2%).

<img src="https://user-images.githubusercontent.com/36138628/203688351-adac7146-4e71-4ab6-8958-5cfe643a2dc5.png" width="70%"/>
</div>

```python

predict = inference_model('beit-base-p16_beit-pre_8xb128-coslr-100e_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('beit_beit-base-p16_8xb256-amp-coslr-300e_in1k', pretrained=True)
```

Train:

```shell

```

Train:

```shell
python tools/train.py configs/beit/beit_beit-base-p16_8xb256-amp-coslr-300e_in1k.py
```

Test:

```shell
python tools/test.py configs/beit/benchmarks/beit-base-p16_8xb128-coslr-100e_in1k.py https://download.openmmlab.com/mmselfsup/1.x/beit/beit_vit-base-p16_8xb256-amp-coslr-300e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k_20221128-0ca393e9.pth
```

| `beit_beit-base-p16_8xb256-amp-coslr-300e_in1k` |   86.53    |   17.58   | [config](beit_beit-base-p16_8xb256-amp-coslr-300e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/beit/beit_vit-base-p16_8xb256-amp-coslr-300e_in1k/beit_vit-base-p16_8xb256-amp-coslr-300e_in1k_20221128-ab79e626.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/beit/beit_vit-base-p16_8xb256-amp-coslr-300e_in1k/beit_vit-base-p16_8xb256-amp-coslr-300e_in1k_20221128-ab79e626.json) |

| `beit-base-p16_beit-pre_8xb128-coslr-100e_in1k` | [BEIT](https://download.openmmlab.com/mmselfsup/1.x/beit/beit_vit-base-p16_8xb256-amp-coslr-300e_in1k/beit_vit-base-p16_8xb256-amp-coslr-300e_in1k_20221128-ab79e626.pth) |   86.53    |   17.58   |   83.10   |    N/A    | [config](benchmarks/beit-base-p16_8xb128-coslr-100e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/beit/beit_vit-base-p16_8xb256-amp-coslr-300e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k_20221128-0ca393e9.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/beit/beit_vit-base-p16_8xb256-amp-coslr-300e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k_20221128-0ca393e9.json) |

| `beit-base-p16_beit-in21k-pre_3rdparty_in1k`\* |             BEIT ImageNet-21k              |   86.53    |   17.58   |   85.28   |   97.59   | [config](benchmarks/beit-base-p16_8xb64_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/beit/beit-base_3rdparty_in1k_20221114-c0a4df23.pth) |

*Models with * are converted from the [official repo](https://github.com/microsoft/unilm/tree/master/beit). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@inproceedings{bao2022beit,
    title={{BE}iT: {BERT} Pre-Training of Image Transformers},
    author={Hangbo Bao and Li Dong and Songhao Piao and Furu Wei},
    booktitle={International Conference on Learning Representations},
}
```

- [ ] If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects.

# Dynamic R-CNN

> [Dynamic R-CNN: Towards High Quality Object Detection via Dynamic Training](https://arxiv.org/abs/2004.06002)

Although two-stage object detectors have continuously advanced the state-of-the-art performance in recent years, the training process itself is far from crystal. In this work, we first point out the inconsistency problem between the fixed network settings and the dynamic training procedure, which greatly affects the performance. For example, the fixed label assignment strategy and regression loss function cannot fit the distribution change of proposals and thus are harmful to training high quality detectors. Consequently, we propose Dynamic R-CNN to adjust the label assignment criteria (IoU threshold) and the shape of regression loss function (parameters of SmoothL1 Loss) automatically based on the statistics of proposals during training. This dynamic design makes better use of the training samples and pushes the detector to fit more high quality samples. Specifically, our method improves upon ResNet-50-FPN baseline with 1.9% AP and 5.5% AP90 on the MS COCO dataset with no extra overhead.

<img src="https://user-images.githubusercontent.com/40661020/143879518-842f5bec-9f65-4454-93a1-9b3b0c42ec3c.png"/>
</div>

| Backbone |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                   Config                    |                                                                                                                                      Download                                                                                                                                      |
| :------: | :-----: | :-----: | :------: | :------------: | :----: | :-----------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

|   R-50   | pytorch |   1x    |   3.8    |                |  38.9  | [config](./dynamic-rcnn_r50_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/dynamic_rcnn/dynamic_rcnn_r50_fpn_1x/dynamic_rcnn_r50_fpn_1x-62a3f276.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/dynamic_rcnn/dynamic_rcnn_r50_fpn_1x/dynamic_rcnn_r50_fpn_1x_20200618_095048.log.json) |

```latex
@article{DynamicRCNN,
    author = {Hongkai Zhang and Hong Chang and Bingpeng Ma and Naiyan Wang and Xilin Chen},
    title = {Dynamic {R-CNN}: Towards High Quality Object Detection via Dynamic Training},
    journal = {arXiv preprint arXiv:2004.06002},
}
```

## Design of Data pipelines

In the [new dataset tutorial](./datasets.md), we know that the dataset class use the `load_data_list` method
to initialize the entire dataset, and we save the information of every sample to a dict.

Usually, to save memory usage, we only load image paths and labels in the `load_data_list`, and load full
image content when we use them. Moreover, we may want to do some random data augmentation during picking
samples when training. Almost all data loading, pre-processing, and formatting operations can be configured in
MMPretrain by the **data pipeline**.

The data pipeline means how to process the sample dict when indexing a sample from the dataset. And it
consists of a sequence of data transforms. Each data transform takes a dict as input, processes it, and outputs a
dict for the next data transform.

Here is a data pipeline example for ResNet-50 training on ImageNet.

```python

```python
    dict(type='RandomResizedCrop', scale=224),
]
```

All available data transforms in MMPretrain can be found in the [data transforms docs](mmpretrain.datasets.transforms).

## Modify the training/test pipeline

The data pipeline in MMPretrain is pretty flexible. You can control almost every step of the data
preprocessing from the config file, but on the other hand, you may be confused facing so many options.

Here is a common practice and guidance for image classification tasks.

At the beginning of a data pipeline, we usually need to load image data from the file path.
[`LoadImageFromFile`](mmcv.transforms.LoadImageFromFile) is commonly used to do this task.

```python

```python
    ...
]
```

If you want to load data from files with special formats or special locations, you can [implement a new loading
transform](#add-new-data-transforms) and add it at the beginning of the data pipeline.

### Augmentation and other processing

During training, we usually need to do data augmentation to avoid overfitting. During the test, we also need to do
some data processing like resizing and cropping. These data transforms will be placed after the loading process.

Here is a simple data augmentation recipe example. It will randomly resize and crop the input image to the
specified scale, and randomly flip the image horizontally with probability.

```python
    ...
    ...
]
```

```python
    ...
    ...
]
```

Here is a heavy data augmentation recipe example used in [Swin-Transformer](../papers/swin_transformer.md)
training. To align with the official implementation, it specified `pillow` as the resize backend and `bicubic`
as the resize algorithm. Moreover, it added [`RandAugment`](mmpretrain.datasets.transforms.RandAugment) and
[`RandomErasing`](mmpretrain.datasets.transforms.RandomErasing) as extra data augmentation method.

This configuration specified every detail of the data augmentation, and you can simply copy it to your own
config file to apply the data augmentations of the Swin-Transformer.

```python
bgr_std = [57.375, 57.12, 58.395]

    ...

```python

    ...
    dict(
        magnitude_level=9,
        hparams=dict(
            pad_val=[round(x) for x in bgr_mean], interpolation='bicubic')),
    dict(
        type='RandomErasing',
        erase_prob=0.25,
        mode='rand',
        min_area_ratio=0.02,
        max_area_ratio=1 / 3,
        fill_color=bgr_mean,
        fill_std=bgr_std),
    ...
]
```

```{note}
Usually, the data augmentation part in the data pipeline handles only image-wise transforms, but not transforms
like image normalization or mixup/cutmix. It's because we can do image normalization and mixup/cutmix on batch data

to accelerate. To configure image normalization and mixup/cutmix, please use the [data preprocessor](mmpretrain.models.utils.data_preprocessor).
```

The formatting is to collect training data from the data information dict and convert these data to
model-friendly format.

In most cases, you can simply use [`PackInputs`](mmpretrain.datasets.transforms.PackInputs), and it will
convert the image in NumPy array format to PyTorch tensor, and pack the ground truth categories information and
other meta information as a [`DataSample`](mmpretrain.structures.DataSample).

```python
    ...
]
```

## Add new data transforms

1. Write a new data transform in any file, e.g., `my_transform.py`, and place it in
   the folder `mmpretrain/datasets/transforms/`. The data transform class needs to inherit

   the [`mmcv.transforms.BaseTransform`](mmcv.transforms.BaseTransform) class and override
   the `transform` method which takes a dict as input and returns a dict.

   ```python
   from mmpretrain.registry import TRANSFORMS

   ```

2. Import the new class in the `mmpretrain/datasets/transforms/__init__.py`.

   ```python
   ...

   __all__ = [
   ]
   ```

   ```python
       ...
       ...
   ]
   ```

## Pipeline visualization

..., 'MyTransform'
   ]
   ```

   ```python
       ...
       ...
   ]
   ```

After designing data pipelines, you can use the [visualization tools](../useful_tools/dataset_visualization.md) to view the performance.

# How to support new models

We provide several tools to support model conversion.

## Function Rewriter

The PyTorch neural network is written in python that eases the development of the algorithm. But the use of Python control flow and third-party libraries make it difficult to export the network to an intermediate representation. We provide a 'monkey patch' tool to rewrite the unsupported function to another one that can be exported. Here is an example:

```python
from mmdeploy.core import FUNCTION_REWRITER

    func_name='torch.Tensor.repeat', backend='tensorrt')
def repeat_static(input, *size):
    origin_func = ctx.origin_func
    if input.dim() == 1 and len(size) == 1:
        return origin_func(input.unsqueeze(0), *([1] + list(size))).squeeze(0)
    else:
        return origin_func(input, *size)
```

It is easy to use the function rewriter. Just add a decorator with arguments:

else:
```

- `func_name` is the function to override. It can be either a PyTorch function or a custom function. Methods in modules can also be overridden by this tool.
- `backend` is the inference engine. The function will be overridden when the model is exported to this engine. If it is not given, this rewrite will be the default rewrite. The default rewrite will be used if the rewrite of the given backend does not exist.

The arguments are the same as the original function, except a context `ctx` as the first argument. The context provides some useful information such as the deployment config `ctx.cfg` and the original function (which has been overridden) `ctx.origin_func`.

## Module Rewriter

If you want to replace a whole module with another one, we have another rewriter as follows:

```python
@MODULE_REWRITER.register_rewrite_module(

```python
    'mmagic.models.backbones.sr_backbones.SRCNN', backend='tensorrt')
class SRCNNWrapper(nn.Module):

                 module,
                 cfg,
                 channels=(3, 64, 32, 3),
                 kernel_sizes=(9, 1, 5),
        super(SRCNNWrapper, self).__init__()

        self._module = module

        module.img_upsampler = nn.Upsample(
            scale_factor=module.upscale_factor,

        return self._module(*args, **kwargs)

    def init_weights(self, *args, **kwargs):
        """Initialize weights."""
        return self._module.init_weights(*args, **kwargs)
```

return self._module(*args, **kwargs)

```

Just like function rewriter, add a decorator with arguments:

- `module_type` the module class to rewrite.

All instances of the module in the network will be replaced with instances of this new class. The original module and the deployment config will be passed as the first two arguments.

## Custom Symbolic

The mappings between PyTorch and ONNX are defined in PyTorch with symbolic functions. The custom symbolic function can help us to bypass some ONNX nodes which are unsupported by inference engine.

```python

```python
@SYMBOLIC_REWRITER.register_symbolic('squeeze', is_pytorch=True)
def squeeze_default(g, self, dim=None):
    if dim is None:
        dims = []
        for i, size in enumerate(self.type().sizes()):
            if size == 1:
                dims.append(i)
    else:
        dims = [sym_help._get_const(dim, 'i', 'dim')]
    return g.op('Squeeze', self, axes_i=dims)
```

The decorator arguments:

- `func_name` The function name to add symbolic. Use full path if it is a custom `torch.autograd.Function`. Or just a name if it is a PyTorch built-in function.
- `is_pytorch` True if the function is a PyTorch built-in function.

- `arg_descriptors` the descriptors of the symbolic function arguments. Will be feed to `torch.onnx.symbolic_helper._parse_arg`.

Just like function rewriter, there is a context `ctx` as the first argument. The context provides some useful information such as the deployment config `ctx.cfg` and the original function (which has been overridden) `ctx.origin_func`. Note that the `ctx.origin_func` can be used only when `is_pytorch==False`.

}
```

<details>

```bibtex
}
```

}
```

| Set  |                           Arch                            | Input Size | PCK@0.2 |  AUC  | EPE  |                           ckpt                            |                           log                            |
| :--- | :-------------------------------------------------------: | :--------: | :-----: | :---: | :--: | :-------------------------------------------------------: | :------------------------------------------------------: |
| test | [pose_resnet_50](/configs/hand_2d_keypoint/topdown_heatmap/freihand2d/td-hm_res50_8xb64-100e_freihand2d-224x224.py) |  224x224   |  0.999  | 0.868 | 3.27 | [ckpt](https://download.openmmlab.com/mmpose/hand/resnet/res50_freihand_224x224-ff0799bc_20200914.pth) | [log](https://download.openmmlab.com/mmpose/hand/resnet/res50_freihand_224x224_20200914.log.json) |

# Preparing HRSID Dataset

```bibtex
@ARTICLE{HRSID_2020,
author={Wei, Shunjun and Zeng, Xiangfeng and Qu, Qizhe and Wang, Mou and Su, Hao and Shi, Jun},
journal={IEEE Access},
title={HRSID: A High-Resolution SAR Images Dataset for Ship Detection and Instance Segmentation},
year={2020},
volume={8},
pages={120234-120254},
}
```

## Download HRSID dataset

The HRSID dataset can be downloaded from [Google drive](https://drive.google.com/file/d/1BZTU8Gyg20wqHXtBPFzRazn_lEdvhsbE/view).

```none
mmrotate
├── tools
├── data
│   ├── HRSID_JPG
```

Please change `data_root` in `configs/_base_/datasets/hrisd.py` to `data/HRSID_JPG/`.

# LLaVA

```sh
cd opencompass/multimodal/models/llava
git clone https://github.com/haotian-liu/LLaVA.git
```

Then prepare the environement according to the [install instruction](https://github.com/haotian-liu/LLaVA/tree/main#install)

# Preparing Something-Something V1

```BibTeX
}
```

For basic dataset information, you can refer to the dataset [paper](https://arxiv.org/pdf/1706.04261.pdf).
Before we start, please make sure that the directory is located at `$MMACTION2/tools/data/sthv1/`.

Since the official [website](https://20bn.com/datasets/something-something/v1) of Something-Something V1 is currently unavailable, you can download the annotations from third-part source to `$MMACTION2/data/sthv1/` .

Since the official dataset doesn't provide the original video data and only extracted RGB frames are available, you have to directly download RGB frames.

You can download all compressed file parts from third-part source  to `$MMACTION2/data/sthv1/` and use the following command to uncompress.

```shell
```

For users who only want to use RGB frames, you can skip to step 5 to generate file lists in the format of rawframes.

```

Since the prefix of official JPGs is "%05d.jpg" (e.g., "00001.jpg"), users need to add `"filename_tmpl='{:05}.jpg'"` to the dict of `data.train`, `data.val` and `data.test` in the config files related with sthv1 like this:

```
    val=dict(

```

```shell
```

```shell
```

```

```shell
```

```shell
```

```shell
```

## Step 6. Check Directory Structure

After the whole data process for Something-Something V1 preparation,
you will get the rawframes (RGB + Flow), and annotation files for Something-Something V1.

In the context of the whole project (for Something-Something V1 only), the folder structure will look like:

```
mmaction2
├── tools
├── data
│   ├── sthv1

```
mmaction2
├── tools
├── data
│   ├── sthv1
│   |   |   ├──...
│   |   |   ├── 1
│   |   |   ├── 2
│   |   |   ├── ...

```

For training and evaluating on Something-Something V1, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).

# Data Transforms

In this tutorial, we introduce the design of transforms pipeline in MMSegmentation.

- [Data Transforms](#data-transforms)
  - [Design of Data pipelines](#design-of-data-pipelines)

Following typical conventions, we use `Dataset` and `DataLoader` for data loading with multiple workers. `Dataset` returns a dict of data items corresponding the arguments of models' forward method. Since the data in semantic segmentation may not be the same size, we introduce a new `DataContainer` type in MMCV to help collect and distribute data of different size. See [here](https://github.com/open-mmlab/mmcv/blob/master/mmcv/parallel/data_container.py) for more details.

In 1.x version of MMSegmentation, all data transformations are inherited from [`BaseTransform`](https://github.com/open-mmlab/mmcv/blob/2.x/mmcv/transforms/base.py#L6).

The input and output types of transformations are both dict. A simple example is as follows:

```python
>>> from mmseg.datasets.transforms import LoadAnnotations
>>> transforms = LoadAnnotations()
>>> img_path = './data/cityscapes/leftImg8bit/train/aachen/aachen_000000_000019_leftImg8bit.png.png'
>>> gt_path = './data/cityscapes/gtFine/train/aachen/aachen_000015_000019_gtFine_instanceTrainIds.png'
>>> results = dict(
>>>     img_path=img_path,
>>>     seg_map_path=gt_path,
>>>     reduce_zero_label=False,
>>>     seg_fields=[])
>>> data_dict = transforms(results)
dict_keys(['img_path', 'seg_map_path', 'reduce_zero_label', 'seg_fields', 'gt_seg_map'])
```

```

The data preparation pipeline and the dataset are decomposed. Usually a dataset defines how to process the annotations and a data pipeline defines all the steps to prepare a data dict. A pipeline consists of a sequence of operations. Each operation takes a dict as input and also outputs a dict for the next transform.

The operations are categorized into data loading, pre-processing, formatting and test-time augmentation.

Here is a pipeline example for PSPNet:

```python
    dict(

dict(
]
    dict(type='Resize', scale=(2048, 1024), keep_ratio=True),
    # does not need to resize data transform
]
```

For each operation, we list the related dict fields that are `added`/`updated`/`removed`. Before pipelines, the information we can directly obtain from the datasets are `img_path` and `seg_map_path`.

`LoadImageFromFile`: Load an image from file.

- add: `img`, `img_shape`, `ori_shape`

`LoadAnnotations`: Load semantic segmentation maps provided by dataset.

- add: `seg_fields`, `gt_seg_map`

`RandomResize`: Random resize image & segmentation map.

- add: `scale`, `scale_factor`, `keep_ratio`
- update: `img`, `img_shape`, `gt_seg_map`

`Resize`: Resize image & segmentation map.

- update: `img`, `gt_seg_map`, `img_shape`

`RandomCrop`: Random crop image & segmentation map.

`RandomFlip`: Flip the image & segmentation map.

- add: `flip`, `flip_direction`
- update: `img`, `gt_seg_map`

`PhotoMetricDistortion`: Apply photometric distortion to image sequentially, every transformation is applied with a probability of 0.5. The position of random contrast is in second or second to last(mode 0 or 1 below, respectively).

```
1. random brightness

```
2. random contrast (mode 0)
3. convert color from BGR to HSV
4. random saturation
5. random hue
6. convert color from HSV to BGR
7. random contrast (mode 1)
```

- update: `img`

`PackSegInputs`: Pack the inputs data for the semantic segmentation.

- add: `inputs`, `data_sample`
- remove: keys specified by `meta_keys` (merged into the metainfo of data_sample), all other keys

## ONNX Runtime Ops

### Human Bounding Box Detection Models

For human bounding box detection models, please download from [MMDetection Model Zoo](https://mmdetection.readthedocs.io/en/3.x/model_zoo.html).
MMDetection provides 80-class COCO-pretrained models, which already includes the `person` category.

### Hand Bounding Box Detection Models

For hand bounding box detection, we simply train our hand box models on OneHand10K dataset using MMDetection.

#### Hand detection results on OneHand10K test set

### Face Bounding Box Detection Models

For face bounding box detection, we train a YOLOX detector on COCO-face data using MMDetection.

#### Face detection results on COCO-face test set

### Animal Bounding Box Detection Models

In COCO dataset, there are 80 object categories, including 10 common `animal` categories (14: 'bird', 15: 'cat', 16: 'dog', 17: 'horse', 18: 'sheep', 19: 'cow', 20: 'elephant', 21: 'bear', 22: 'zebra', 23: 'giraffe')
For animals in the categories, please download from [MMDetection Model Zoo](https://mmdetection.readthedocs.io/en/3.x/model_zoo.html).

#### Macaque detection results on MacaquePose test set

> [Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data](https://arxiv.org/abs/2107.10833)

Though many attempts have been made in blind super-resolution to restore low-resolution images with unknown and complex degradations, they are still far from addressing general real-world degraded images. In this work, we extend the powerful ESRGAN to a practical restoration application (namely, Real-ESRGAN), which is trained with pure synthetic data. Specifically, a high-order degradation modeling process is introduced to better simulate complex real-world degradations. We also consider the common ringing and overshoot artifacts in the synthesis process. In addition, we employ a U-Net discriminator with spectral normalization to increase discriminator capability and stabilize the training dynamics. Extensive comparisons have shown its superior visual performance than prior works on various real datasets. We also provide efficient implementations to synthesize training pairs on the fly.

 <img src="https://user-images.githubusercontent.com/7676947/144034533-f81430df-351b-490c-9e00-733465edf3ee.png" width="400"/>
</div >

Evaluated on Set5 dataset with RGB channels. The metrics are `PSNR` and `SSIM`.

|                                    Model                                     | Dataset  |  PSNR   |  SSIM  |    Training Resources    |                                    Download                                     |
| :--------------------------------------------------------------------------: | :------: | :-----: | :----: | :----------------------: | :-----------------------------------------------------------------------------: |

| [realesrnet_c64b23g32_12x4_lr2e-4_1000k_df2k_ost](./realesrnet_c64b23g32_4xb12-lr2e-4-1000k_df2k-ost.py) | df2k_ost | 28.0297 | 0.8236 | 4 (Tesla V100-SXM2-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/real_esrgan/realesrnet_c64b23g32_12x4_lr2e-4_1000k_df2k_ost_20210816-4ae3b5a4.pth)/log |
| [realesrgan_c64b23g32_12x4_lr1e-4_400k_df2k_ost](./realesrgan_c64b23g32_4xb12-lr1e-4-400k_df2k-ost.py) | df2k_ost | 26.2204 | 0.7655 | 4 (Tesla V100-SXM2-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/real_esrgan/realesrgan_c64b23g32_12x4_lr1e-4_400k_df2k_ost_20211010-34798885.pth) /[log](https://download.openmmlab.com/mmediting/restorers/real_esrgan/realesrgan_c64b23g32_12x4_lr1e-4_400k_df2k_ost_20210922_142838.log.json) |

**Train**

<details>

```shell
# cpu train

**Train**

<details>

```shell
# cpu train

```

**Test**

<details>

```shell
# cpu test

**Test**

<details>

```shell
# cpu test

```

```

```bibtex
}
```

# Preparing RealSRSet Dataset

```bibtex
}
```

The datasets RealSRSet can be download from [here](https://github.com/cszn/BSRGAN/tree/main/testsets/RealSRSet).

The datasets RealSRSet+5images can be download from [here](https://github.com/JingyunLiang/SwinIR/releases/download/v0.0/RealSRSet+5images.zip).

```text
mmagic
├── mmagic
├── tools
├── data
```

# Demo

## Outline

- [Modify configs through script arguments](#modify-config-through-script-arguments): Tricks to directly modify configs through script arguments.
- [Video demo](#video-demo): A demo script to predict the recognition result using a single video.
- [Video GradCAM Demo](#video-gradcam-demo): A demo script to visualize GradCAM results using a single video.
- [Webcam demo](#webcam-demo): A demo script to implement real-time action recognition from a web camera.
- [Skeleton-based Action Recognition Demo](#skeleton-based-action-recognition-demo): A demo script to predict the skeleton-based action recognition result using a single video.
- [SpatioTemporal Action Detection Video Demo](#spatiotemporal-action-detection-video-demo): A demo script to predict the spatiotemporal action detection result using a single video.

- [SpatioTemporal Action Detection ONNX Video Demo](#spatiotemporal-action-detection-onnx-video-demo): A demo script to predict the SpatioTemporal Action Detection result using the onnx file instead of building the PyTorch models.
- [Inferencer Demo](#inferencer): A demo script to implement fast predict for video analysis tasks based on unified inferencer interface.

## Modify configs through script arguments

When running demos using our provided scripts, you may specify `--cfg-options` to in-place modify the config.

For example, `--cfg-options model.backbone.norm_eval=False` changes the all BN modules in model backbones to `train` mode.

  you may specify `--cfg-options train_dataloader.dataset.pipeline.0.type=DenseSampleFrames`.

## Video demo

MMAction2 provides a demo script to predict the recognition result using a single video. In order to get predict results in range `[0, 1]`, make sure to set `model['test_cfg'] = dict(average_clips='prob')` in config file.

```shell
python demo/demo.py ${CONFIG_FILE} ${CHECKPOINT_FILE} ${VIDEO_FILE} ${LABEL_FILE} \
    [--device ${DEVICE_TYPE}] [--fps ${FPS}] [--font-scale ${FONT_SCALE}] [--font-color ${FONT_COLOR}] \
    [--target-resolution ${TARGET_RESOLUTION}] [--out-filename ${OUT_FILE}]
```

- `--use-frames`: If specified, the demo will take rawframes as input. Otherwise, it will take a video as input.
- `DEVICE_TYPE`: Type of device to run the demo. Allowed values are cuda device like `cuda:0` or `cpu`. If not specified, it will be set to `cuda:0`.

- `FPS`: FPS value of the output video when using rawframes as input. If not specified, it will be set to 30.
- `FONT_SCALE`: Font scale of the text added in the video. If not specified, it will be None.
- `FONT_COLOR`: Font color of the text added in the video. If not specified, it will be `white`.
- `TARGET_RESOLUTION`: Resolution(desired_width, desired_height) for resizing the frames before output when using a video as input. If not specified, it will be None and the frames are resized by keeping the existing aspect ratio.
- `OUT_FILE`: Path to the output file which can be a video format or gif format. If not specified, it will be set to `None` and does not generate the output file.

Examples:

Assume that you are located at `$MMACTION2` and have already downloaded the checkpoints to the directory `checkpoints/`,

Examples:

or use checkpoint url from `configs/` to directly load corresponding checkpoint, which will be automatically saved in `$HOME/.cache/torch/checkpoints`.

1. Recognize a video file as input by using a TSN model on cuda by default.

   ```shell
   # The demo.mp4 and label_map_k400.txt are both from Kinetics-400
   python demo/demo.py demo/demo_configs/tsn_r50_1x1x8_video_infer.py \
       checkpoints/tsn_r50_8xb32-1x1x8-100e_kinetics400-rgb_20220818-2692d16c.pth \
       demo/demo.mp4 tools/data/kinetics/label_map_k400.txt
   ```

2. Recognize a video file as input by using a TSN model on cuda by default, loading checkpoint from url.

   ```shell

```shell
       https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_r50_8xb32-1x1x8-100e_kinetics400-rgb/tsn_r50_8xb32-1x1x8-100e_kinetics400-rgb_20220818-2692d16c.pth \
   ```

3. Recognize a video file as input by using a TSN model and then generate an mp4 file.

   ```shell
       demo/demo.mp4 tools/data/kinetics/label_map_k400.txt --out-filename demo/demo_out.mp4
   ```

## Video GradCAM Demo

MMAction2 provides a demo script to visualize GradCAM results using a single video.

```shell

```

```shell
python tools/visualizations/vis_cam.py ${CONFIG_FILE} ${CHECKPOINT_FILE} ${VIDEO_FILE} [--use-frames] \
    [--device ${DEVICE_TYPE}] [--target-layer-name ${TARGET_LAYER_NAME}] [--fps {FPS}] \
    [--target-resolution ${TARGET_RESOLUTION}] [--resize-algorithm {RESIZE_ALGORITHM}] [--out-filename {OUT_FILE}]
```

- `TARGET_LAYER_NAME`: Layer name to generate GradCAM localization map.
- `RESIZE_ALGORITHM`: Resize algorithm used for resizing. If not specified, it will be set to `bilinear`.

Examples:

1. Get GradCAM results of a I3D model, using a video file as input and then generate an gif file with 10 fps.

   ```shell

   ```shell
   python tools/visualizations/vis_cam.py demo/demo_configs/i3d_r50_32x2x1_video_infer.py \
       checkpoints/i3d_imagenet-pretrained-r50_8xb8-32x2x1-100e_kinetics400-rgb_20220812-e213c223.pth demo/demo.mp4 \
       --target-layer-name backbone/layer4/1/relu --fps 10 \
       --out-filename demo/demo_gradcam.gif
   ```

2. Get GradCAM results of a TSN model, using a video file as input and then generate an gif file, loading checkpoint from url.

   ```shell
   python tools/visualizations/vis_cam.py demo/demo_configs/tsn_r50_1x1x8_video_infer.py \
       https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-dense-1x1x5-100e_kinetics400-rgb/tsn_imagenet-pretrained-r50_8xb32-dense-1x1x5-100e_kinetics400-rgb_20220906-dcbc6e01.pth \

demo/demo.mp4 --target-layer-name backbone/layer4/1/relu --out-filename demo/demo_gradcam_tsn.gif
   ```

## Webcam demo

We provide a demo script to implement real-time action recognition from web camera. In order to get predict results in range `[0, 1]`, make sure to set `model.cls_head.average_clips='prob'` in config file.

```shell
python demo/webcam_demo.py ${CONFIG_FILE} ${CHECKPOINT_FILE} ${LABEL_FILE} \
    [--device ${DEVICE_TYPE}] [--camera-id ${CAMERA_ID}] [--threshold ${THRESHOLD}] \
    [--average-size ${AVERAGE_SIZE}] [--drawing-fps ${DRAWING_FPS}] [--inference-fps ${INFERENCE_FPS}]
```

- `CAMERA_ID`: ID of camera device If not specified, it will be set to 0.

- `THRESHOLD`: Threshold of prediction score for action recognition. Only label with score higher than the threshold will be shown. If not specified, it will be set to 0.
- `AVERAGE_SIZE`: Number of latest clips to be averaged for prediction. If not specified, it will be set to 1.
- `DRAWING_FPS`: Upper bound FPS value of the output drawing. If not specified, it will be set to 20.
- `INFERENCE_FPS`: Upper bound FPS value of the output drawing. If not specified, it will be set to 4.

If your hardware is good enough, increasing the value of `DRAWING_FPS` and `INFERENCE_FPS` will get a better experience.

Examples:

1. Recognize the action from web camera as input by using a TSN model on cpu, averaging the score per 5 times
   and outputting result labels with score higher than 0.2.

   ```shell
   python demo/webcam_demo.py demo/demo_configs/tsn_r50_1x1x8_video_infer.py \
     checkpoints/tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb_20220906-2692d16c.pth tools/data/kinetics/label_map_k400.txt --average-size 5 \
     --threshold 0.2 --device cpu
   ```

2. Recognize the action from web camera as input by using a TSN model on cpu, averaging the score per 5 times
   and outputting result labels with score higher than 0.2, loading checkpoint from url.

   ```shell

and outputting result labels with score higher than 0.2, loading checkpoint from url.

   ```shell
     https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb/tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb_20220906-2692d16c.pth \
     tools/data/kinetics/label_map_k400.txt --average-size 5 --threshold 0.2 --device cpu
   ```

3. Recognize the action from web camera as input by using a I3D model on gpu by default, averaging the score per 5 times

   ```shell
   python demo/webcam_demo.py demo/demo_configs/i3d_r50_32x2x1_video_infer.py \
     checkpoints/i3d_imagenet-pretrained-r50_8xb8-32x2x1-100e_kinetics400-rgb_20220812-e213c223.pth tools/data/kinetics/label_map_k400.txt \
     --average-size 5 --threshold 0.2
   ```

checkpoints/i3d_imagenet-pretrained-r50_8xb8-32x2x1-100e_kinetics400-rgb_20220812-e213c223.pth tools/data/kinetics/label_map_k400.txt \
   ```

Considering the efficiency difference for users' hardware, Some modifications might be done to suit the case.
Users can change:

- `SampleFrames` step (especially the number of `clip_len` and `num_clips`) of `test_pipeline` in the config file, like `--cfg-options test_pipeline.0.num_clips=3`.
- Change to the suitable Crop methods like `TenCrop`, `ThreeCrop`, `CenterCrop`, etc. in `test_pipeline` of the config file, like `--cfg-options test_pipeline.4.type=CenterCrop`.
- Change the number of `--average-size`. The smaller, the faster.

## Skeleton-based Action Recognition Demo

MMAction2 provides a demo script to predict the skeleton-based action recognition result using a single video.

```shell
python demo/demo_skeleton.py ${VIDEO_FILE} ${OUT_FILENAME} \

```shell
    [--config ${SKELETON_BASED_ACTION_RECOGNITION_CONFIG_FILE}] \
    [--checkpoint ${SKELETON_BASED_ACTION_RECOGNITION_CHECKPOINT}] \
    [--det-config ${HUMAN_DETECTION_CONFIG_FILE}] \
    [--det-checkpoint ${HUMAN_DETECTION_CHECKPOINT}] \
    [--det-score-thr ${HUMAN_DETECTION_SCORE_THRESHOLD}] \
    [--det-cat-id ${HUMAN_DETECTION_CATEGORY_ID}] \
    [--pose-config ${HUMAN_POSE_ESTIMATION_CONFIG_FILE}] \
    [--pose-checkpoint ${HUMAN_POSE_ESTIMATION_CHECKPOINT}] \
    [--label-map ${LABEL_MAP}] \
    [--short-side] ${SHORT_SIDE}
```

- `SKELETON_BASED_ACTION_RECOGNITION_CONFIG_FILE`: The skeleton-based action recognition config file path.
- `SKELETON_BASED_ACTION_RECOGNITION_CHECKPOINT`: The skeleton-based action recognition checkpoint path or url.

- `HUMAN_DETECTION_CONFIG_FILE`: The human detection config file path.
- `HUMAN_DETECTION_CHECKPOINT`: The human detection checkpoint path or url.
- `HUMAN_DETECTION_SCORE_THRE`: The score threshold for human detection. Defaults to 0.9.
- `HUMAN_DETECTION_CATEGORY_ID`: The category id for human detection. Defaults to 0.
- `HUMAN_POSE_ESTIMATION_CONFIG_FILE`: The human pose estimation config file path (trained on COCO-Keypoint).
- `HUMAN_POSE_ESTIMATION_CHECKPOINT`: The human pose estimation checkpoint path or url (trained on COCO-Keypoint).
- `LABEL_MAP`: The label map used. Defaults to `'tools/data/skeleton/label_map_ntu60.txt'`.
- `DEVICE`: Type of device to run the demo. Allowed values are cuda device like `'cuda:0'` or `'cpu'`. Defaults to `'cuda:0'`.
- `SHORT_SIDE`: The short side used for frame extraction. Defaults to 480.

Examples:

Examples:

Assume that you are located at `$MMACTION2` .

1. Use the Faster-RCNN as the human detector, HRNetw32 as the pose estimator, PoseC3D-NTURGB+D-60-XSub-Keypoint as the skeleton-based action recognizer.

```shell
python demo/demo_skeleton.py demo/demo_skeleton.mp4 demo/demo_skeleton_out.mp4 \
    --config configs/skeleton/posec3d/slowonly_r50_8xb16-u48-240e_ntu60-xsub-keypoint.py \
    --checkpoint https://download.openmmlab.com/mmaction/skeleton/posec3d/slowonly_r50_u48_240e_ntu60_xsub_keypoint/slowonly_r50_u48_240e_ntu60_xsub_keypoint-f3adabf1.pth \
    --det-config demo/demo_configs/faster-rcnn_r50_fpn_2x_coco_infer.py \
    --det-checkpoint http://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_2x_coco/faster_rcnn_r50_fpn_2x_coco_bbox_mAP-0.384_20200504_210434-a5d8aa15.pth \
    --det-score-thr 0.9 \
    --det-cat-id 0 \

--det-score-thr 0.9 \
    --pose-config demo/demo_configs/td-hm_hrnet-w32_8xb64-210e_coco-256x192_infer.py \
    --pose-checkpoint https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_coco_256x192-c78dce93_20200708.pth \
    --label-map tools/data/skeleton/label_map_ntu60.txt
```

2. Use the Faster-RCNN as the human detector, HRNetw32 as the pose estimator, STGCN-NTURGB+D-60-XSub-Keypoint as the skeleton-based action recognizer.

```shell
    --config configs/skeleton/stgcn/stgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py \
    --checkpoint https://download.openmmlab.com/mmaction/v1.0/skeleton/stgcn/stgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d/stgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d_20221129-484a394a.pth \

--det-config demo/demo_configs/faster-rcnn_r50_fpn_2x_coco_infer.py \
```

## SpatioTemporal Action Detection Video Demo

MMAction2 provides a demo script to predict the SpatioTemporal Action Detection result using a single video.

```shell
python demo/demo_spatiotemporal_det.py --video ${VIDEO_FILE} \
    [--out-filename ${OUTPUT_FILENAME}] \
    [--config ${SPATIOTEMPORAL_ACTION_DETECTION_CONFIG_FILE}] \
    [--checkpoint ${SPATIOTEMPORAL_ACTION_DETECTION_CHECKPOINT}] \

[--out-filename ${OUTPUT_FILENAME}] \
    [--action-score-thr ${ACTION_DETECTION_SCORE_THRESHOLD}] \
    [--short-side] ${SHORT_SIDE} \
    [--predict-stepsize ${PREDICT_STEPSIZE}] \
    [--output-stepsize ${OUTPUT_STEPSIZE}] \
    [--output-fps ${OUTPUT_FPS}]
```

- `OUTPUT_FILENAME`: Path to the output file which is a video format. Defaults to `demo/stdet_demo.mp4`.
- `SPATIOTEMPORAL_ACTION_DETECTION_CONFIG_FILE`: The spatiotemporal action detection config file path.

- `SPATIOTEMPORAL_ACTION_DETECTION_CHECKPOINT`: The spatiotemporal action detection checkpoint URL.
- `HUMAN_DETECTION_CHECKPOINT`: The human detection checkpoint URL.
- `HUMAN_DETECTION_SCORE_THRESHOLD`: The score threshold for human detection. Defaults to 0.9.
- `ACTION_DETECTION_SCORE_THRESHOLD`: The score threshold for action detection. Defaults to 0.5.
- `LABEL_MAP`: The label map used. Defaults to `tools/data/ava/label_map.txt`.
- `DEVICE`: Type of device to run the demo. Allowed values are cuda device like `cuda:0` or `cpu`.  Defaults to `cuda:0`.
- `SHORT_SIDE`: The short side used for frame extraction. Defaults to 256.
- `PREDICT_STEPSIZE`: Make a prediction per N frames.  Defaults to 8.

- `OUTPUT_STEPSIZE`: Output 1 frame per N frames in the input video. Note that `PREDICT_STEPSIZE % OUTPUT_STEPSIZE == 0`. Defaults to 4.
- `OUTPUT_FPS`: The FPS of demo video output. Defaults to 6.

Examples:

1. Use the Faster RCNN as the human detector, SlowOnly-8x8-R101 as the action detector. Making predictions per 8 frames, and output 1 frame per 4 frames to the output video. The FPS of the output video is 4.

```shell
python demo/demo_spatiotemporal_det.py demo/demo.mp4 demo/demo_spatiotemporal_det.mp4 \
    --config configs/detection/slowonly/slowonly_kinetics400-pretrained-r101_8xb16-8x8x1-20e_ava21-rgb.py \

    --checkpoint https://download.openmmlab.com/mmaction/detection/ava/slowonly_omnisource_pretrained_r101_8x8x1_20e_ava_rgb/slowonly_omnisource_pretrained_r101_8x8x1_20e_ava_rgb_20201217-16378594.pth \
    --action-score-thr 0.5 \
    --label-map tools/data/ava/label_map.txt \
    --predict-stepsize 8 \
    --output-stepsize 4 \
    --output-fps 6
```

## SpatioTemporal Action Detection ONNX Video Demo

--label-map tools/data/ava/label_map.txt \
```

MMAction2 provides a demo script to predict the SpatioTemporal Action Detection result using the onnx file instead of building the PyTorch models.

```shell
python demo/demo_spatiotemporal_det_onnx.py --video ${VIDEO_FILE} \
    [--onnx-file ${SPATIOTEMPORAL_ACTION_DETECTION_ONNX_FILE}] \

[--label-map ${LABEL_MAP}] \
```

- `SPATIOTEMPORAL_ACTION_DETECTION_ONNX_FILE`: The spatiotemporal action detection onnx file.

Examples:

1. Export an onnx file given the config file and checkpoint.

```shell
python tools/deployment/export_onnx_stdet.py \

Examples:

```shell
    configs/detection/slowonly/slowonly_kinetics400-pretrained-r101_8xb16-8x8x1-20e_ava21-rgb.py \
    https://download.openmmlab.com/mmaction/detection/ava/slowonly_omnisource_pretrained_r101_8x8x1_20e_ava_rgb/slowonly_omnisource_pretrained_r101_8x8x1_20e_ava_rgb_20201217-16378594.pth \
    --output_file slowonly_kinetics400-pretrained-r101_8xb16-8x8x1-20e_ava21-rgb.onnx \
    --num_frames 8
```

2. Use the Faster RCNN as the human detector, the generated `slowonly_kinetics400-pretrained-r101_8xb16-8x8x1-20e_ava21-rgb.onnx` file as the action detector. Making predictions per 8 frames, and output 1 frame per 4 frames to the output video. The FPS of the output video is 4.

```shell
python demo/demo_spatiotemporal_det_onnx.py demo/demo.mp4 demo/demo_spatiotemporal_det.mp4 \

```shell
    --onnx-file slowonly_kinetics400-pretrained-r101_8xb16-8x8x1-20e_ava21-rgb.onnx \
```

MMAction2 provides a demo script to implement fast prediction for video analysis tasks based on unified inferencer interface, currently only supports action recognition task.

```shell
python demo/demo.py ${INPUTS} \
    [--vid-out-dir ${VID_OUT_DIR}] \

```shell
    [--rec ${RECOG_TASK}] \
    [--rec-weights ${RECOG_WEIGHTS}] \
    [--label-file ${LABEL_FILE}] \
    [--device ${DEVICE_TYPE}] \
    [--batch-size ${BATCH_SIZE}] \
    [--print-result ${PRINT_RESULT}] \
    [--pred-out-file ${PRED_OUT_FILE} ]
```

- `--show`: If specified, the demo will display the video in a popup window.
- `--print-result`: If specified, the demo will print the inference results'
- `VID_OUT_DIR`: Output directory of saved videos. Defaults to None, means not to save videos.
- `RECOG_TASK`: Type of Action Recognition algorithm. It could be the path to the config file, the model name or alias defined in metafile.
- `RECOG_WEIGHTS`: Path to the custom checkpoint file of the selected recog model. If it is not specified and "rec" is a model name of metafile, the weights will be loaded from metafile.

- `LABEL_FILE`: Label file for dataset the algorithm pretrained on. Defaults to None, means don't show label in result.
- `DEVICE_TYPE`: Type of device to run the demo. Allowed values are cuda device like `cuda:0` or `cpu`. Defaults to `cuda:0`.
- `BATCH_SIZE`: The batch size used in inference. Defaults to 1.
- `PRED_OUT_FILE`: File path to save the inference results. Defaults to None, means not to save prediction results.

Examples:

Assume that you are located at `$MMACTION2`.

1. Recognize a video file as input by using a TSN model, loading checkpoint from metafile.

   ```shell
   python demo/demo_inferencer.py demo/demo.mp4 \
       --rec tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb \

# The demo.mp4 and label_map_k400.txt are both from Kinetics-400
       --label-file tools/data/kinetics/label_map_k400.txt
   ```

2. Recognize a video file as input by using a TSN model, using model alias in metafile.

   ```shell
       --rec tsn \
   ```

3. Recognize a video file as input by using a TSN model, and then save visulization video.

   ```shell
       --vid-out-dir demo_out \
       --rec tsn \
   ```

# t-分布随机邻域嵌入（t-SNE）可视化

## t-分布随机邻域嵌入可视化工具介绍

MMPretrain 提供 `tools/visualization/vis_tsne.py` 工具来用t-SNE可视化图像的特征嵌入。请使用 `pip install scikit-learn` 安装 `sklearn` 来计算t-SNE。

**命令**：

```bash
    CONFIG \
    [--show]
```

- `CONFIG`: t-SNE 配置文件的路径。
- `--checkpoint CHECKPOINT`: 模型权重文件的路径。
- `--work-dir WORK_DIR`: 保存日志和可视化图像的目录。

[--n-iter-without-progress N_ITER_WITHOUT_PROGRESS]
```

- `--test-cfg TEST_CFG`: 用来加载 test_dataloader 配置的 t-SNE 配置文件的路径。
- `--vis-stage {backbone,neck,pre_logits}`: 模型可视化的阶段。
- `--class-idx CLASS_IDX [CLASS_IDX ...]`: 用来计算 t-SNE 的类别。
- `--max-num-class MAX_NUM_CLASS`: 前 N 个被应用 t-SNE 算法的类别，默认为20。
- `--max-num-samples MAX_NUM_SAMPLES`: 每个类别中最大的样本数，值越高需要的计算时间越长，默认为100。
- `--cfg-options CFG_OPTIONS [CFG_OPTIONS ...]`: 覆盖被使用的配置中的一些设定，形如 xxx=yyy 格式的关键字-值对会被合并到配置文件中。如果被覆盖的值是一个列表，它应该形如 key="[a,b]" 或者 key=a,b 。它还允许嵌套的列表/元组值，例如 key="[(a,b),(c,d)]" 。注意引号是必需的，而且不允许有空格。
- `--device DEVICE`: 用于推理的设备。
- `--legend`: 显示所有类别的图例。
- `--show`: 在图形窗口中显示结果。
- `--n-components N_COMPONENTS`: 结果的维数。
- `--perplexity PERPLEXITY`: 复杂度与其他流形学习算法中使用的最近邻的数量有关。
- `--early-exaggeration EARLY_EXAGGERATION`: 控制原空间中的自然聚类在嵌入空间中的紧密程度以及它们之间的空间大小。

- `--learning-rate LEARNING_RATE`: t-SNE 的学习率通常在[10.0, 1000.0]的范围内。如果学习率太高，数据可能看起来像一个球，其中任何一点与它最近的邻居近似等距。如果学习率太低，大多数点可能看起来被压缩在一个几乎没有异常值的密集点云中。
- `--n-iter N_ITER`: 优化的最大迭代次数。应该至少为250。
- `--n-iter-without-progress N_ITER_WITHOUT_PROGRESS`: 在我们中止优化之前，最大的没有进展的迭代次数。
- `--init INIT`: 初始化方法。

## 如何可视化分类模型的t-SNE（如 ResNet）

以下是在CIFAR-10数据集上训练的 ResNet-18 和 ResNet-50 模型上运行 t-SNE 可视化的两个样例：

```shell

```

## 如何可视化自监督模型的t-SNE（如 MAE）

以下是在ImageNet数据集上训练的 MAE-ViT-base 模型上运行 t-SNE 可视化的一个样例。输入数据来自 ImageNet 验证集。MAE和一些自监督预训练算法配置中没有 test_dataloader 信息。在分析这些自监督算法时，你需要在配置中添加 test_dataloader 信息，或者使用 `--test-cfg` 字段来指定一个配置文件。

```shell
```

# AP-10K: A Benchmark for Animal Pose Estimation in the Wild

<details>

```bibtex
}
```

---
name: Error report
about: Create a report to help us improve
title: ''
labels: ''
---

Thanks for your error report and we appreciate it a lot.

2. The bug has not been fixed in the latest version.

**Describe the bug**
A clear and concise description of what the bug is.

   ```none
   A placeholder for the command.
   ```

2. Did you make any modifications on the code or config? Did you understand what you have modified?

3. What dataset did you use?

1. Please run `python mmseg/utils/collect_env.py` to collect necessary environment information and paste it here.
   - How you installed PyTorch \[e.g., pip, conda, source\]
   - Other environment variables that may be related (such as `$PATH`, `$LD_LIBRARY_PATH`, `$PYTHONPATH`, etc.)

- How you installed PyTorch \[e.g., pip, conda, source\]

**Error traceback**

If applicable, paste the error trackback here.

```none
A placeholder for trackback.
```

**Bug fix**

3. If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMPreTrain.

# 学习配置文件

OpenCompass 使用 OpenMMLab 新式风格的配置文件。如果你之前熟悉 OpenMMLab 风格的配置文件，可以直接阅读
[纯 Python 风格的配置文件（Beta）](https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/config.html#python-beta)
了解新式配置文件与原配置文件的区别。如果你之前没有接触过 OpenMMLab 风格的配置文件，
下面我将会用一个简单的例子来介绍配置文件的使用。请确保你安装了最新版本的 MMEngine，以支持新式风格的配置文件。

## 基本格式

OpenCompass 的配置文件都是 Python 格式的，遵从基本的 Python 语法，通过定义变量的形式指定每个配置项。
比如在定义模型时，我们使用如下配置：

```python
# model_cfg.py

models = [
    dict(
        type=HuggingFaceCausalLM,
        path='huggyllama/llama-7b',
        model_kwargs=dict(device_map='auto'),
        tokenizer_path='huggyllama/llama-7b',
        tokenizer_kwargs=dict(padding_side='left', truncation_side='left'),
        max_seq_len=2048,
        max_out_len=50,
        run_cfg=dict(num_gpus=8, num_procs=1),
    )
]
```

当读取配置文件时，使用 MMEngine 中的 `Config.fromfile` 进行解析。

```python
>>> from mmengine.config import Config
>>> cfg = Config.fromfile('./model_cfg.py')

)
]
```

```python
>>> print(cfg.models[0])
{'type': HuggingFaceCausalLM, 'path': 'huggyllama/llama-7b', 'model_kwargs': {'device_map': 'auto'}, ...}
```

## 继承机制

OpenCompass 的配置文件使用了 Python 的 import 机制进行配置文件的继承。需要注意的是，
我们需要在继承配置文件时使用 `read_base` 上下文管理器。

```python
# inherit.py

    from .model_cfg import models  # model_cfg.py 中的 models 被继承到本配置文件
```

使用 `Config.fromfile` 解析配置文件：

```python
>>> cfg = Config.fromfile('./inherit.py')
```

## 评测配置文件示例

```python
# configs/llama7b.py

    from .datasets.piqa.piqa_ppl import piqa_datasets

```

```python

    from .datasets.siqa.siqa_gen import siqa_datasets

# 将需要评测的数据集拼接成 datasets 字段
datasets = [*piqa_datasets, *siqa_datasets]

# 使用 HuggingFaceCausalLM 评测 HuggingFace 中 AutoModelForCausalLM 支持的模型

models = [
    dict(
        # 以下参数为 HuggingFaceCausalLM 的初始化参数
        # 以下参数为各类模型都必须设定的参数，非 HuggingFaceCausalLM 的初始化参数
        abbr='llama-7b',            # 模型简称，用于结果展示
        max_out_len=100,            # 最长生成 token 数
        batch_size=16,              # 批次大小
        run_cfg=dict(num_gpus=1),   # 运行配置，用于指定资源需求
    )
]
```

## 数据集配置文件示例

max_out_len=100,            # 最长生成 token 数
    )
]
```

以上示例配置文件中，我们直接以继承的方式获取了数据集相关的配置。接下来，
我们会以 PIQA 数据集配置文件为示例，展示数据集配置文件中各个字段的含义。
如果你不打算修改模型测试的 prompt，或者添加新的数据集，则可以跳过这一节的介绍。

PIQA 数据集 [配置文件](https://github.com/InternLM/opencompass/blob/main/configs/datasets/piqa/piqa_ppl_1cf9f0.py)
如下，这是一个基于 PPL（困惑度）进行评测的配置，并且不使用上下文学习方法（In-Context Learning）。

```python
from opencompass.openicl.icl_inferencer import PPLInferencer
from opencompass.openicl.icl_evaluator import AccEvaluator
from opencompass.datasets import HFDataset

# 读取配置
# 加载后的数据集通常以字典形式组织样本，分别指定样本中用于组成 prompt 的输入字段，和作为答案的输出字段
piqa_reader_cfg = dict(
    input_columns=['goal', 'sol1', 'sol2'],
    output_column='label',
    test_split='validation',
)

# 推理配置
piqa_infer_cfg = dict(

)

# 推理配置
    # Prompt 生成配置
        # Prompt 模板，模板形式与后续指定的 inferencer 类型相匹配
        # 这里为了计算 PPL，需要指定每个答案对应的 Prompt 模板
        template={
            0: 'The following makes sense: \nQ: {goal}\nA: {sol1}\n',
            1: 'The following makes sense: \nQ: {goal}\nA: {sol2}\n'
        }),
    # 上下文样本配置，此处指定 `ZeroRetriever`，即不使用上下文样本
    # 推理方式配置
    #   - PPLInferencer 使用 PPL（困惑度）获取答案
    #   - GenInferencer 使用模型的生成结果获取答案
    inferencer=dict(type=PPLInferencer))

# 评估配置，使用 Accuracy 作为评估指标
piqa_eval_cfg = dict(evaluator=dict(type=AccEvaluator))

# 数据集配置，以上各个变量均为此配置的参数
# 为一个列表，用于指定一个数据集各个评测子集的配置。
piqa_datasets = [
    dict(
        path='piqa',
        reader_cfg=piqa_reader_cfg,
        infer_cfg=piqa_infer_cfg,

    dict(
        eval_cfg=piqa_eval_cfg)
]
```

其中 **Prompt 生成配置** 的详细配置方式，可以参见 [Prompt 模板](../prompt/prompt_template.md)。

## 进阶评测配置

在 OpenCompass 中，我们支持了任务划分器（Partitioner）、运行后端（Runner）等配置项，
用于更加灵活、高效的利用计算资源。

默认情况下，我们会使用基于样本数的方式对推理任务进行划分，你可以在启动任务时使用
`--max-partition-size` 指定进行任务划分的样本数阈值。同时，我们默认使用本地资源进行推理和评估任务，
如果你希望使用 Slurm 集群资源，可以在启动任务时使用 `--slurm` 参数和 `--partition` 参数指定 slurm 运行后端。

进一步地，如果以上功能无法满足你的任务划分和运行后端配置需求，你可以在配置文件中进行更详细的配置。
参见[高效评测](./evaluation.md)。

# SimSiam

> [Exploring simple siamese representation learning](https://arxiv.org/abs/2011.10566)

Siamese networks have become a common structure in various recent models for unsupervised visual representation learning. These models maximize the similarity between two augmentations of one image, subject to certain conditions for avoiding collapsing solutions. In this paper, we report surprising empirical results that simple Siamese networks can learn meaningful representations even using none of the following: (i) negative sample pairs, (ii) large batches, (iii) momentum encoders. Our experiments show that collapsing solutions do exist for the loss and structure, but a stop-gradient operation plays an essential role in preventing collapsing. We provide a hypothesis on the implication of stop-gradient, and further show proof-of-concept experiments verifying it. Our “SimSiam” method achieves competitive results on ImageNet and downstream tasks. We hope this simple baseline will motivate people to rethink the roles of Siamese architectures for unsupervised representation learning.

<img  src="https://user-images.githubusercontent.com/36138628/149724180-bc7bac6a-fcb8-421e-b8f1-9550c624d154.png" width="500" />
</div>

```python

predict = inference_model('resnet50_simsiam-100e-pre_8xb512-linear-coslr-90e_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('simsiam_resnet50_8xb32-coslr-100e_in1k', pretrained=True)
```

Train:

```shell

```

Train:

```shell
python tools/train.py configs/simsiam/simsiam_resnet50_8xb32-coslr-100e_in1k.py
```

Test:

```shell
python tools/test.py configs/simsiam/benchmarks/resnet50_8xb512-linear-coslr-90e_in1k.py https://download.openmmlab.com/mmselfsup/1.x/simsiam/simsiam_resnet50_8xb32-coslr-100e_in1k/resnet50_linear-8xb512-coslr-90e_in1k/resnet50_linear-8xb512-coslr-90e_in1k_20220825-f53ba400.pth
```

| `simsiam_resnet50_8xb32-coslr-100e_in1k` |   38.20    |   4.11    | [config](simsiam_resnet50_8xb32-coslr-100e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/simsiam/simsiam_resnet50_8xb32-coslr-100e_in1k/simsiam_resnet50_8xb32-coslr-100e_in1k_20220825-d07cb2e6.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/simsiam/simsiam_resnet50_8xb32-coslr-100e_in1k/simsiam_resnet50_8xb32-coslr-100e_in1k_20220825-d07cb2e6.json) |

| `simsiam_resnet50_8xb32-coslr-200e_in1k` |   38.20    |   4.11    | [config](simsiam_resnet50_8xb32-coslr-200e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/simsiam/simsiam_resnet50_8xb32-coslr-200e_in1k/simsiam_resnet50_8xb32-coslr-200e_in1k_20220825-efe91299.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/simsiam/simsiam_resnet50_8xb32-coslr-200e_in1k/simsiam_resnet50_8xb32-coslr-200e_in1k_20220825-efe91299.json) |

| `resnet50_simsiam-100e-pre_8xb512-linear-coslr-90e_in1k` | [SIMSIAM 100-Epochs](https://download.openmmlab.com/mmselfsup/1.x/simsiam/simsiam_resnet50_8xb32-coslr-100e_in1k/simsiam_resnet50_8xb32-coslr-100e_in1k_20220825-d07cb2e6.pth) |   25.56    |   4.11    |   68.30   | [config](benchmarks/resnet50_8xb512-linear-coslr-90e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/simsiam/simsiam_resnet50_8xb32-coslr-100e_in1k/resnet50_linear-8xb512-coslr-90e_in1k/resnet50_linear-8xb512-coslr-90e_in1k_20220825-f53ba400.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/simsiam/simsiam_resnet50_8xb32-coslr-100e_in1k/resnet50_linear-8xb512-coslr-90e_in1k/resnet50_linear-8xb512-coslr-90e_in1k_20220825-f53ba400.json) |

| `resnet50_simsiam-200e-pre_8xb512-linear-coslr-90e_in1k` | [SIMSIAM 200-Epochs](https://download.openmmlab.com/mmselfsup/1.x/simsiam/simsiam_resnet50_8xb32-coslr-200e_in1k/simsiam_resnet50_8xb32-coslr-200e_in1k_20220825-efe91299.pth) |   25.56    |   4.11    |   69.80   | [config](benchmarks/resnet50_8xb512-linear-coslr-90e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/simsiam/simsiam_resnet50_8xb32-coslr-200e_in1k/resnet50_linear-8xb512-coslr-90e_in1k/resnet50_linear-8xb512-coslr-90e_in1k_20220825-519b5135.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/simsiam/simsiam_resnet50_8xb32-coslr-200e_in1k/resnet50_linear-8xb512-coslr-90e_in1k/resnet50_linear-8xb512-coslr-90e_in1k_20220825-519b5135.json) |

```bibtex
@inproceedings{chen2021exploring,
  title={Exploring simple siamese representation learning},
  author={Chen, Xinlei and He, Kaiming},
}
```

# SimCLR

> [A simple framework for contrastive learning of visual representations](https://arxiv.org/abs/2002.05709)

This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and (3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50.

<img  src="https://user-images.githubusercontent.com/36138628/149723851-cf5f309e-d891-454d-90c0-e5337e5a11ed.png" width="400" />
</div>

```python

predict = inference_model('resnet50_simclr-200e-pre_8xb512-linear-coslr-90e_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('simclr_resnet50_16xb256-coslr-200e_in1k', pretrained=True)
```

Train:

```shell

```

Train:

```shell
python tools/train.py configs/simclr/simclr_resnet50_16xb256-coslr-200e_in1k.py
```

Test:

```shell
python tools/test.py configs/simclr/benchmarks/resnet50_8xb512-linear-coslr-90e_in1k.py https://download.openmmlab.com/mmselfsup/1.x/simclr/simclr_resnet50_16xb256-coslr-200e_in1k/resnet50_linear-8xb512-coslr-90e_in1k/resnet50_linear-8xb512-coslr-90e_in1k_20220825-f12c0457.pth
```

| Model                                     | Params (M) | Flops (G) |                        Config                        |                                         Download                                         |

| :---------------------------------------- | :--------: | :-------: | :--------------------------------------------------: | :--------------------------------------------------------------------------------------: |
| `simclr_resnet50_16xb256-coslr-200e_in1k` |   27.97    |   4.11    | [config](simclr_resnet50_16xb256-coslr-200e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/simclr/simclr_resnet50_16xb256-coslr-200e_in1k/simclr_resnet50_16xb256-coslr-200e_in1k_20220825-4d9cce50.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/simclr/simclr_resnet50_16xb256-coslr-200e_in1k/simclr_resnet50_16xb256-coslr-200e_in1k_20220825-4d9cce50.json) |

| `simclr_resnet50_16xb256-coslr-800e_in1k` |   27.97    |   4.11    | [config](simclr_resnet50_16xb256-coslr-800e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/simclr/simclr_resnet50_16xb256-coslr-800e_in1k/simclr_resnet50_16xb256-coslr-800e_in1k_20220825-85fcc4de.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/simclr/simclr_resnet50_16xb256-coslr-800e_in1k/simclr_resnet50_16xb256-coslr-800e_in1k_20220825-85fcc4de.json) |

| `resnet50_simclr-200e-pre_8xb512-linear-coslr-90e_in1k` | [SIMCLR 200-Epochs](https://download.openmmlab.com/mmselfsup/1.x/simclr/simclr_resnet50_16xb256-coslr-200e_in1k/simclr_resnet50_16xb256-coslr-200e_in1k_20220825-4d9cce50.pth) |   25.56    |   4.11    |   66.90   | [config](benchmarks/resnet50_8xb512-linear-coslr-90e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/simclr/simclr_resnet50_16xb256-coslr-200e_in1k/resnet50_linear-8xb512-coslr-90e_in1k/resnet50_linear-8xb512-coslr-90e_in1k_20220825-f12c0457.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/simclr/simclr_resnet50_16xb256-coslr-200e_in1k/resnet50_linear-8xb512-coslr-90e_in1k/resnet50_linear-8xb512-coslr-90e_in1k_20220825-f12c0457.json) |

| `resnet50_simclr-800e-pre_8xb512-linear-coslr-90e_in1k` | [SIMCLR 800-Epochs](https://download.openmmlab.com/mmselfsup/1.x/simclr/simclr_resnet50_16xb256-coslr-800e_in1k/simclr_resnet50_16xb256-coslr-800e_in1k_20220825-85fcc4de.pth) |   25.56    |   4.11    |   69.20   | [config](benchmarks/resnet50_8xb512-linear-coslr-90e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/simclr/simclr_resnet50_16xb256-coslr-800e_in1k/resnet50_linear-8xb512-coslr-90e_in1k/resnet50_linear-8xb512-coslr-90e_in1k_20220825-b80ae1e5.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/simclr/simclr_resnet50_16xb256-coslr-800e_in1k/resnet50_linear-8xb512-coslr-90e_in1k/resnet50_linear-8xb512-coslr-90e_in1k_20220825-b80ae1e5.json) |

```bibtex
@inproceedings{chen2020simple,
  title={A simple framework for contrastive learning of visual representations},
  author={Chen, Ting and Kornblith, Simon and Norouzi, Mohammad and Hinton, Geoffrey},
  booktitle={ICML},
}

author={Chen, Ting and Kornblith, Simon and Norouzi, Mohammad and Hinton, Geoffrey},
}
```

# Strong Baselines

We train Mask R-CNN with large-scale jitter and longer schedule as strong baselines.
The modifications follow those in [Detectron2](https://github.com/facebookresearch/detectron2/tree/master/configs/new_baselines).

| Backbone |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                                       Config                                       |         Download         |
| :------: | :-----: | :-----: | :------: | :------------: | :----: | :-----: | :--------------------------------------------------------------------------------: | :----------------------: |
| R-50-FPN | pytorch |   50e   |          |                |        |         |    [config](./mask-rcnn_r50_fpn_rpn-2conv_4conv1fc_syncbn-all_lsj-50e_coco.py)     | [model](<>) \| [log](<>) |

| R-50-FPN | pytorch |  100e   |          |                |        |         |    [config](./mask-rcnn_r50_fpn_rpn-2conv_4conv1fc_syncbn-all_lsj-100e_coco.py)    | [model](<>) \| [log](<>) |
| R-50-FPN |  caffe  |  100e   |          |                |  44.7  |  40.4   | [config](./mask-rcnn_r50-caffe_fpn_rpn-2conv_4conv1fc_syncbn-all_lsj-100e_coco.py) | [model](<>) \| [log](<>) |
| R-50-FPN |  caffe  |  400e   |          |                |        |         | [config](./mask-rcnn_r50-caffe_fpn_rpn-2conv_4conv1fc_syncbn-all_lsj-400e_coco.py) | [model](<>) \| [log](<>) |

## Notice

When using large-scale jittering, there are sometimes empty proposals in the box and mask heads during training.

## Notice

This requires MMSyncBN that allows empty tensors. Therefore, please use mmcv-full>=1.3.14 to train models supported in this directory.

### how to enable ncnn vulkan capability

follow [the build and install instruction](https://github.com/Tencent/ncnn/blob/master/docs/how-to-build/how-to-build.md)

make sure you have installed vulkan sdk from [lunarg vulkan sdk website](https://vulkan.lunarg.com/sdk/home)

Usually, you can enable the vulkan compute inference feature by adding only one line of code to your application.

```cpp
// enable vulkan compute feature before loading
```

### does my graphics device support vulkan

Some platforms have been tested and known working. In theory, if your platform support vulkan api, either 1.0 or 1.1, it shall work.

|apple|/|/|/|Y|Y|
|arm|/|?|Y|/|/|

You can search [the vulkan database](https://vulkan.gpuinfo.org) to see if your device supports vulkan.

Some old buggy drivers may produce wrong result, that are blacklisted in ncnn and treated as non-vulkan capable device.
You could check if your device and driver have this issue with  [my conformance test here](vulkan-conformance-test).
Most of these systems are android with version lower than 8.1.

### why using vulkan over cuda/opencl/metal

In the beginning, I had no GPGPU programming experience, and I had to learn one.

vulkan is considered more portable and well supported by venders and the cross-platform low-overhead graphics api. As a contrast, cuda is only available on nvidia device, metal is only available on macos and ios, while loading opencl library is banned in android 7.0+ and does not work on ios.

### I got errors like "vkCreateComputePipelines failed -1000012000" or random stalls or crashes

Upgrade your vulkan driver.

[intel https://downloadcenter.intel.com/product/80939/Graphics-Drivers](https://downloadcenter.intel.com/product/80939/Graphics-Drivers)

[amd https://www.amd.com/en/support](https://www.amd.com/en/support)

[nvidia https://www.nvidia.com/Download/index.aspx](https://www.nvidia.com/Download/index.aspx)

### how to use ncnn vulkan on android

minimum android ndk version: android-ndk-r18b

minimum sdk platform api version: android-24

link your jni project with libvulkan.so

[The squeezencnn example](https://github.com/Tencent/ncnn/tree/master/examples/squeezencnn) have equipped gpu inference, you could take it as reference.

### how to use ncnn vulkan on ios

setup vulkan sdk (https://vulkan.lunarg.com/sdk/home#mac)

metal only works on real device with arm64 cpu (iPhone 5s and later)

link your project with MoltenVK framework and Metal

### what about the layers without vulkan support

These layers have vulkan support currently

AbsVal, BatchNorm, BinaryOp, Cast, Clip, Concat, Convolution, ConvolutionDepthWise, Crop, Deconvolution, DeconvolutionDepthWise, Dropout, Eltwise, Flatten, HardSigmoid, InnerProduct, Interp, LRN, Packing, Padding, Permute, Pooling(pad SAME not supported), PReLU, PriorBox, ReLU, Reorg, Reshape, Scale, ShuffleChannel, Sigmoid, Softmax, TanH, UnaryOp

For these layers without vulkan support, ncnn inference engine will automatically fallback to cpu path.

Thus, it is usually not a serious issue if your network only has some special head layers like SSD or YOLO. All examples in ncnn are known working properly with vulkan enabled.

### my model runs slower on gpu than cpu

The current vulkan inference implementation is far from the preferred state. Many handful optimization techniques are planned, such as winograd convolution, operator fusion, fp16 storage and arithmetic etc.

It is common that your model runs slower on gpu than cpu on arm devices like mobile phones, since we have quite good arm optimization in ncnn ;)

### vulkan device not found / extra high cpu utility while vulkan is enabled on nvidia gpu

There are severel reasons could lead to this outcome. First please check your driver status with `nvidia-smi`. If you have correctly installed your driver, you should see something like this:

```bash
$ nvidia-smi
Sat Mar 06 19:53:16 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 451.48       Driver Version: 451.48       CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+

| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1060   WDDM  | 00000000:02:00.0 Off |                  N/A |
| N/A   31C    P8     5W /  N/A |     90MiB /  6144MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |

|=============================================================================|
|  No running processes found                                                 |
```

If `nvidia-smi` crashes or cannot be found, please reinstall your graphics driver.

If ncnn *is* utilizing the Tesla GPU, you can see your program in the `Processes` block at the bottom. In that case, it's likely some operators are not yet supported in Vulkan, and have fallbacked to the CPU, thus leading to a low utilization of the GPU.

If you *couldn't* find your process running, plase check the active driver model, which can be found to the right of your device name. For Geforce and Titan GPUs, the default driver model is WDDM (Windows Desktop Driver Model), which supports both rendering graphics as well as computing. But for Tesla GPUs, without configuration, the driver model is defualted to TCC ([Tesla Computing Cluster](https://docs.nvidia.com/gameworks/content/developertools/desktop/tesla_compute_cluster.htm)). NVIDIA's TCC driver does not support Vulkan, so you need to use the following command to set the driver model back to WDDM, to use Vulkan:

```bash
$ nvidia-smi -g 0 -dm 0
```

The number following `-g` is the GPU ID (which can be found to the left of your device name in `nvidia-smi` output); and `-dm` stands for driver model, 0 refers to WDDM and 1 means TCC.

# 为 OpenCompass 做贡献

- [为 OpenCompass 做贡献](#为-opencompass-做贡献)
  - [什么是拉取请求？](#什么是拉取请求)
  - [基本的工作流：](#基本的工作流)
  - [具体步骤](#具体步骤)
    - [1. 获取最新的代码库](#1-获取最新的代码库)
    - [2. 从 `main` 分支创建一个新的开发分支](#2-从-main-分支创建一个新的开发分支)
    - [3. 提交你的修改](#3-提交你的修改)
    - [4. 推送你的修改到复刻的代码库，并创建一个拉取请求](#4-推送你的修改到复刻的代码库并创建一个拉取请求)
    - [5. 讨论并评审你的代码](#5-讨论并评审你的代码)
    - [6. `拉取请求`合并之后删除该分支](#6-拉取请求合并之后删除该分支)
  - [代码风格](#代码风格)

感谢你对于OpenCompass的贡献！我们欢迎各种形式的贡献，包括但不限于以下几点。

- 修改错别字或修复bug
- 添加文档或将文档翻译成其它语言
- 添加新功能和组件

```
```

## 基本的工作流：

2. 从最新的 `main` 分支创建分支进行开发
5. 讨论、审核代码
6. 将开发分支合并到 `main` 分支

## 具体步骤

```

## 基本的工作流：

5. 讨论、审核代码

## 具体步骤

  复刻 OpenCompass 原代码库，点击 GitHub 页面右上角的 **Fork** 按钮即可

  ```bash
  ```

  ```bash
  ```

- 从第二个 PR 起

  检出本地代码库的主分支，然后从最新的原代码库的主分支拉取更新。

  ```bash
  ```

### 2. 从 `main` 分支创建一个新的开发分支

```bash
```

- 如果你是第一次尝试贡献，请在 OpenCompass 的目录下安装并初始化 pre-commit hooks。

  ```bash
  ```

  ````{tip}
  对于中国地区的用户，由于网络原因，安装 pre-commit hook 可能会失败。可以尝试以下命令切换为国内镜像源：
  ```bash
  pre-commit install -c .pre-commit-config-zh-cn.yaml

  ```

  ````{tip}
  ```bash
  pre-commit run –all-files -c .pre-commit-config-zh-cn.yaml
  ```
  ````

  ```bash
  # coding
  ```

  ```{note}
  ```

  ```bash
  ```

- 创建一个拉取请求

  ```bash

  ```bash
  ```

## 代码风格

### Python

我们采用[PEP8](https://www.python.org/dev/peps/pep-0008/)作为首选的代码风格。

我们使用以下工具进行linting和格式化：

- [flake8](https://github.com/PyCQA/flake8): 一个围绕一些linter工具的封装器。
- [isort](https://github.com/timothycrosley/isort): 一个用于排序Python导入的实用程序。
- [yapf](https://github.com/google/yapf): 一个Python文件的格式化器。
- [codespell](https://github.com/codespell-project/codespell): 一个Python实用程序，用于修复文本文件中常见的拼写错误。
- [mdformat](https://github.com/executablebooks/mdformat): mdformat是一个有明确定义的Markdown格式化程序，可以用来在Markdown文件中强制执行一致的样式。
- [docformatter](https://github.com/myint/docformatter): 一个格式化docstring的工具。

yapf和isort的样式配置可以在[setup.cfg](https://github.com/OpenCompass/blob/main/setup.cfg)中找到。

# Preparing GLEAN Dataset

```bibtex
}
```

## Preparing cat_train dataset

1. Download [cat dataset](http://dl.yf.io/lsun/objects/cat.zip) from [LSUN homepage](https://www.yf.io/p/lsun)

2. Download [cat_train/meta_info_LSUNcat_GT.txt](https://github.com/ckkelvinchan/GLEAN/blob/main/data/cat_train/meta_info_LSUNcat_GT.txt) from [GLEAN homepage](https://github.com/ckkelvinchan/GLEAN)

3. Export and downsample images

Export images from lmdb file and resize the input images to the designated size. We provide such a script:

```shell

```shell
python tools/dataset_converters/glean/preprocess_cat_train_dataset.py --lmdb-path .data/cat --meta-file-path ./data/cat_train/meta_info_LSUNcat_GT.txt --out-dir ./data/cat_train
```

The generated data is stored under `cat_train` and the folder structure is as follows.

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── cat_train
│   │   ├── GT
│   │   ├── BIx8_down
│   │   ├── BIx16_down
│   │   ├── meta_info_LSUNcat_GT.txt
...
```

## Preparing cat_test dataset

1. Download [CAT dataset](https://archive.org/download/CAT_DATASET/CAT_DATASET_02.zip) from [here](https://archive.org/details/CAT_DATASET).

2. Download [cat_test/meta_info_Cat100_GT.txt](https://github.com/ckkelvinchan/GLEAN/blob/main/data/cat_test/meta_info_Cat100_GT.txt) from [GLEAN homepage](https://github.com/ckkelvinchan/GLEAN)

3. Downsample images

Resize the input images to the designated size. We provide such a script:

```shell
python tools/dataset_converters/glean/preprocess_cat_test_dataset.py --data-path ./data/CAT_03 --meta-file-path ./data/cat_test/meta_info_Cat100_GT.txt --out-dir ./data/cat_test
```

The generated data is stored under `cat_test` and the folder structure is as follows.

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── cat_test
│   │   ├── GT
│   │   ├── meta_info_Cat100_GT.txt
...
```

## Preparing FFHQ dataset

1. Download [FFHQ dataset (images1024x1024)](https://drive.google.com/drive/folders/1tZUcXDBeOibC6jcMCtgRRz67pzrAHeHL) from it's [homepage](https://github.com/NVlabs/ffhq-dataset)

Then you can refactor the folder structure looks like:

```text
ffhq
├── images
|   ├── 00000.png
|   ├── 00001.png
|   ├── ...
|   ├── 69999.png
```

```text
ffhq
├── images
|   ├── ...
```

2. Download [ffhq/meta_info_FFHQ_GT.txt](https://github.com/ckkelvinchan/GLEAN/blob/main/data/FFHQ/meta_info_FFHQ_GT.txt) from [GLEAN homepage](https://github.com/ckkelvinchan/GLEAN)

```shell
python tools/dataset_converters/glean/preprocess_ffhq_celebahq_dataset.py --data-root ./data/ffhq/images
```

The generated data is stored under `ffhq` and the folder structure is as follows.

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── ffhq
|   |   ├── images
|   |   ├── BIx16_down
|   |   ├── meta_info_FFHQ_GT.txt
...
```

## Preparing CelebA-HQ dataset

1. Preparing datasets following it's [homepage](https://github.com/tkarras/progressive_growing_of_gans)

...
```

```text
CelebA-HQ
├── GT
|   ├── ...
|   ├── 30000.png
```

2. Download [CelebA-HQ/meta_info_CelebAHQ_val100_GT.txt](https://github.com/ckkelvinchan/GLEAN/blob/main/data/CelebA-HQ/meta_info_CelebAHQ_val100_GT.txt) from [GLEAN homepage](https://github.com/ckkelvinchan/GLEAN)

```shell
python tools/dataset_converters/glean/preprocess_ffhq_celebahq_dataset.py --data-root ./data/CelebA-HQ/GT
```

The generated data is stored under `CelebA-HQ` and the folder structure is as follows.

```text
mmagic
├── mmagic
├── tools
├── configsdata
├── data
|   ├── CelebA-HQ
|   |   ├── GT

```text
mmagic
├── mmagic
├── tools
├── data
|   |   ├── GT
|   |   ├── meta_info_CelebAHQ_val100_GT.txt
...
```

## Preparing FFHQ_CelebAHQ dataset

We merge FFHQ(`ffhq/images`) and CelebA-HQ(`CelebA-HQ/GT`) to generate FFHQ_CelebAHQ dataset.

The folder structure should looks like:

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── FFHQ_CelebAHQ
|   |   ├── GT
...
```

[`CustomDataset`](mmpretrain.datasets.CustomDataset) is a general dataset class for you to use your own datasets. To use `CustomDataset`, you need to organize your dataset files according to the following two formats:

### Subfolder Format

In this format, you only need to re-organize your dataset folder and place all samples in one folder without
creating any annotation files.

For supervised tasks (with `with_label=True`), we use the name of sub-folders as the categories names, as
shown in the below example, `class_x` and `class_y` will be recognized as the categories names.

```text
│   └── ...
    ├── ...
```

For unsupervised tasks (with `with_label=False`), we directly load all sample files under the specified folder:

```text
│   └── ...

```

```text
│   └── ...
└── ...
```

Assume you want to use it as the training dataset, and the below is the configurations in your config file.

```python
    ...
    # Training dataset configurations
        with_label=True,   # or False for unsupervised tasks
    )
)
```

```{note}
If you want to use this format, do not specify `ann_file`, or specify `ann_file=''`.

And please note that the subfolder format requires a folder scanning which may cause a slower initialization,
especially for large datasets or slow file IO.
```

### Text Annotation File Format

```

In this format, we use a text annotation file to store image file paths and the corespondding category
indices.

For supervised tasks (with `with_label=True`), the annotation file should include the file path and the
category index of one sample in one line and split them by a space, as below:

All these file paths can be absolute paths, or paths relative to the `data_prefix`.

```text
123.png 4
...
```

```{note}
The index numbers of categories start from 0. And the value of ground-truth labels should fall in range `[0, num_classes - 1]`.

In addition, please use the `classes` field in the dataset settings to specify the name of every category.
```

```

For unsupervised tasks (with `with_label=False`), the annotation file only need to include the file path of
one sample in one line, as below:

```text
123.png
nsdf3.png
...
```

Assume the entire dataset folder is as below:

```text
data_root
├── meta
│   ├── test.txt     # The annotation file for the test dataset
│   ├── train.txt    # The annotation file for the training dataset
│   └── val.txt      # The annotation file for the validation dataset.
├── train
├── test
└── val
```

Here is an example dataset settings in config files:

```python
# Training dataloader configurations
        data_root='path/to/data_root',  # The common prefix of both `ann_flie` and `data_prefix`.

        ann_file='meta/train.txt',      # The path of annotation file relative to the data_root.
        data_prefix='train',            # The prefix of file paths in the `ann_file`, relative to the data_root.
        with_label=True,                # or False for unsupervised tasks
        classes=['A', 'B', 'C', 'D', ...],  # The name of every category.
        pipeline=...,    # The transformations to process the dataset samples.
    )
    ...
)
```

```{note}
For a complete example about how to use the `CustomDataset`, please see [How to Pretrain with Custom Dataset](../notes/pretrain_custom_dataset.md)
```

ImageNet has multiple versions, but the most commonly used one is [ILSVRC 2012](http://www.image-net.org/challenges/LSVRC/2012/). It can be accessed with the following steps.

MIM supports downloading from [OpenDataLab](https://opendatalab.com/) and preprocessing ImageNet dataset with one command line.

_You need to register an account at [OpenDataLab official website](https://opendatalab.com/) and login by CLI._

```Bash
# log in OpenDataLab, register if you don't have an account.
odl login
# download and preprocess by MIM, better to execute in $MMPreTrain directory.
```

````

1. Register an account and login to the [download page](http://www.image-net.org/download-images).
2. Find download links for ILSVRC2012 and download the following two files

3. Untar the downloaded files

````

`````

### The Directory Structrue of the ImageNet dataset

We support two ways of organizing the ImageNet dataset: Subfolder Format and Text Annotation File Format.

#### Subfolder Format

We have provided a sample, which you can download and extract from this [link](https://download.openmmlab.com/mmpretrain/datasets/imagenet_1k.zip). The directory structure of the dataset should be as below:

```text
├── train/
│   ├── ...
├── val/

│   ├── ...
├── val/
│   │   └── ...
│   ├── ...
```

#### Text Annotation File Format

You can download and untar the meta data from this [link](https://download.openmmlab.com/mmclassification/datasets/imagenet/meta/caffe_ilsvrc12.tar.gz). And re-organize the dataset as below:

```text
├── meta/
├── train/
│   ├── ...
├── val/

│   ├── ...
├── val/
│   ├── ...
```

### Configuration

Once your dataset is organized in the way described above, you can use the [`ImageNet`](mmpretrain.datasets.ImageNet) dataset with the below configurations:

```python
    ...
    )
)

    ...
    # Validation dataset configurations
    )
)

```

## Supported Image Classification Datasets

type='ImageNet',
    )
)

```

| Datasets                                                                           | split                               | HomePage                                                                            |
| ---------------------------------------------------------------------------------- | :---------------------------------- | ----------------------------------------------------------------------------------- |
| [`Calthch101`](mmpretrain.datasets.Caltech101)(data_root[, split, pipeline, ...])  | ["train", "test"]                   | [Caltech 101](https://data.caltech.edu/records/mzrjq-6wc02) Dataset.                |

| [`CIFAR10`](mmpretrain.datasets.CIFAR10)(data_root[, split, pipeline, ...])        | ["train", "test"]                   | [CIFAR10](https://www.cs.toronto.edu/~kriz/cifar.html) Dataset.                     |
| [`CIFAR100`](mmpretrain.datasets.CIFAR100)(data_root[, split, pipeline, ...])      | ["train", "test"]                   | [CIFAR100](https://www.cs.toronto.edu/~kriz/cifar.html) Dataset.                    |
| [`CUB`](mmpretrain.datasets.CUB)(data_root[, split, pipeline, ...])                | ["train", "test"]                   | [CUB-200-2011](http://www.vision.caltech.edu/datasets/cub_200_2011/) Dataset.       |
| [`DTD`](mmpretrain.datasets.DTD)(data_root[, split, pipeline, ...])                | ["train", "val", "tranval", "test"] | [Describable Texture Dataset (DTD)](https://www.robots.ox.ac.uk/~vgg/data/dtd/) Dataset. |

| [`FashionMNIST`](mmpretrain.datasets.FashionMNIST) (data_root[, split, pipeline, ...]) | ["train", "test"]                   | [Fashion-MNIST](https://github.com/zalandoresearch/fashion-mnist) Dataset.          |
| [`FGVCAircraft`](mmpretrain.datasets.FGVCAircraft)(data_root[, split, pipeline, ...]) | ["train", "val", "tranval", "test"] | [FGVC Aircraft](https://www.robots.ox.ac.uk/~vgg/data/fgvc-aircraft/) Dataset.      |
| [`Flowers102`](mmpretrain.datasets.Flowers102)(data_root[, split, pipeline, ...])  | ["train", "val", "tranval", "test"] | [Oxford 102 Flower](https://www.robots.ox.ac.uk/~vgg/data/flowers/102/) Dataset.    |
| [`Food101`](mmpretrain.datasets.Food101)(data_root[, split, pipeline, ...])        | ["train", "test"]                   | [Food101](https://data.vision.ee.ethz.ch/cvl/datasets_extra/food-101/) Dataset.     |

| [`MNIST`](mmpretrain.datasets.MNIST) (data_root[, split, pipeline, ...])           | ["train", "test"]                   | [MNIST](http://yann.lecun.com/exdb/mnist/) Dataset.                                 |
| [`OxfordIIITPet`](mmpretrain.datasets.OxfordIIITPet)(data_root[, split, pipeline, ...]) | ["tranval", test"]                  | [Oxford-IIIT Pets](https://www.robots.ox.ac.uk/~vgg/data/pets/) Dataset.            |
| [`Places205`](mmpretrain.datasets.Places205)(data_root[, pipeline, ...])           | -                                   | [Places205](http://places.csail.mit.edu/downloadData.html) Dataset.                 |
| [`StanfordCars`](mmpretrain.datasets.StanfordCars)(data_root[, split, pipeline, ...]) | ["train", "test"]                   | [Stanford Cars](https://ai.stanford.edu/~jkrause/cars/car_dataset.html) Dataset.    |

| [`SUN397`](mmpretrain.datasets.SUN397)(data_root[, split, pipeline, ...])          | ["train", "test"]                   | [SUN397](https://vision.princeton.edu/projects/2010/SUN/) Dataset.                  |
| [`VOC`](mmpretrain.datasets.VOC)(data_root[, image_set_path, pipeline, ...])       | ["train", "val", "tranval", "test"] | [Pascal VOC](http://host.robots.ox.ac.uk/pascal/VOC/) Dataset.                      |

Some dataset homepage links may be unavailable, and you can download datasets through [OpenDataLab](https://opendatalab.com/), such as [Stanford Cars](https://opendatalab.com/Stanford_Cars/download).

## OpenMMLab 2.0 Standard Dataset

In order to facilitate the training of multi-task algorithm models, we unify the dataset interfaces of different tasks. OpenMMLab has formulated the **OpenMMLab 2.0 Dataset Format Specification**. When starting a trainning task, the users can choose to convert their dataset annotation into the specified format, and use the algorithm library of OpenMMLab to perform algorithm training and testing based on the data annotation file.

The OpenMMLab 2.0 Dataset Format Specification stipulates that the annotation file must be in `json` or `yaml`, `yml`, `pickle` or `pkl` format; the dictionary stored in the annotation file must contain `metainfo` and `data_list` fields, The value of `metainfo` is a dictionary, which contains the meta information of the dataset; and the value of `data_list` is a list, each element in the list is a dictionary, the dictionary defines a raw data, each raw data contains a or several training/testing samples.

The following is an example of a JSON annotation file (in this example each raw data contains only one train/test sample):

```
{
        {
            'classes': ('cat', 'dog'), # the category index of 'cat' is 0 and 'dog' is 1.
            ...
        },
        [
            {
                ...
            },
            {
                ...
            },
            ...
        ]
}
```

Assume you want to use the training dataset and the dataset is stored as the below structure:

```text
data
├── train
│   ├── ...
```

Build from the following dictionaries:

```python
    ...
        type='BaseDataset',
        data_root='data',
        ann_file='annotations/train.json',

```python
    ...
        data_prefix='train/',
    )
)
```

## Other Datasets

To find more datasets supported by MMPretrain, and get more configurations of the above datasets, please see the [dataset documentation](mmpretrain.datasets).

To implement your own dataset class for some special formats, please see the [Adding New Dataset](../advanced_guides/datasets.md).

The following datawrappers are supported in MMEngine, you can refer to {external+mmengine:doc}`MMEngine tutorial <advanced_tutorials/basedataset>` to learn how to use it.

The MMPretrain also support [KFoldDataset](mmpretrain.datasets.KFoldDataset), please use it with `tools/kfold-cross-valid.py`.

# RAVIR: A Dataset and Methodology for the Semantic Segmentation and Quantitative Analysis of Retinal Arteries and Veins in Infrared Reflectance Imaging

This project support **`RAVIR: A Dataset and Methodology for the Semantic Segmentation and Quantitative Analysis of Retinal Arteries and Veins in Infrared Reflectance Imaging`**, and the dataset used in this project can be downloaded from [here](https://ravir.grand-challenge.org/).

The retinal vasculature provides important clues in the diagnosis and monitoring of systemic diseases including hypertension and diabetes. The microvascular system is of primary involvement in such conditions, and the retina is the only anatomical site where the microvasculature can be directly observed. The objective assessment of retinal vessels has long been considered a surrogate biomarker for systemic vascular diseases, and with recent advancements in retinal imaging and computer vision technologies, this topic has become the subject of renewed attention. In this paper, we present a novel dataset, dubbed RAVIR, for the semantic segmentation of Retinal Arteries and Veins in Infrared Reflectance (IR) imaging. It enables the creation of deep learning-based models that distinguish extracted vessel type without extensive post-processing.

| Dataset name                                | Anatomical region | Task type    | Modality                     | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License                                                         |
| ------------------------------------------- | ----------------- | ------------ | ---------------------------- | ------------ | --------------------- | ---------------------- | ------------ | --------------------------------------------------------------- |
| [Ravir](https://ravir.grand-challenge.org/) | eye               | segmentation | infrared reflectance imaging | 3            | 23/-/19               | yes/-/-                | 2022         | [CC-BY-NC 4.0](https://creativecommons.org/licenses/by-sa/4.0/) |

| background |     23     |   87.22    |    -     |    -     |     -     |     -     |
|   artery   |     23     |    5.45    |    -     |    -     |     -     |     -     |
|    vein    |     23     |    7.33    |    -     |    -     |     -     |     -     |

Note:

![bac](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/infrared_reflectance_imaging/ravir/ravir_dataset.png)

```bibtex
@article{hatamizadeh2022ravir,
  title={RAVIR: A dataset and methodology for the semantic segmentation and quantitative analysis of retinal arteries and veins in infrared reflectance imaging},

  author={Hatamizadeh, Ali and Hosseini, Hamid and Patel, Niraj and Choi, Jinseo and Pole, Cameron C and Hoeferlin, Cory M and Schwartz, Steven D and Terzopoulos, Demetri},
  journal={IEEE Journal of Biomedical and Health Informatics},
  pages={3272--3283},
}
```

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `ravir/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
```

- download dataset from [here](https://ravir.grand-challenge.org/) and decompression data to path `'data/ravir/'`.

- run script `"python ../../tools/split_seg_dataset.py --data_root data/ravir"` to split dataset and generate `train.txt`, `val.txt` and `test.txt`. If the label of official validation set and test set can't be obtained, we generate `train.txt` and `val.txt` from the training set randomly.

```none
  ├── mmseg
  │   │   │   ├── infrared_reflectance_imaging
  │   │   │   │   ├── ravir

```

| background |     18     |   87.41    |    5     |  86.53   |     -     |     -     |

|   artery   |     18     |    5.44    |    5     |   5.50   |     -     |     -     |
|    vein    |     18     |    7.15    |    5     |   7.97   |     -     |     -     |

```shell
```

```shell
```

```

## Results

### Ravir

|     Method      | Backbone | Crop Size |   lr   |                                   config                                   |
| :-------------: | :------: | :-------: | :----: | :------------------------------------------------------------------------: |
| fcn_unet_s5-d16 |   unet   |  512x512  |  0.01  |  [config](./configs/fcn-unet-s5-d16_unet_1xb16-0.01-20k_ravir-512x512.py)  |
| fcn_unet_s5-d16 |   unet   |  512x512  | 0.001  | [config](./configs/fcn-unet-s5-d16_unet_1xb16-0.001-20k_ravir-512x512.py)  |

| fcn_unet_s5-d16 |   unet   |  512x512  | 0.0001 | [config](./configs/fcn-unet-s5-d16_unet_1xb16-0.0001-20k_ravir-512x512.py) |

# CTRGCN Project

[Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://arxiv.org/abs/2107.12213)

Graph convolutional networks (GCNs) have been widely used and achieved remarkable results in skeleton-based action recognition. In GCNs, graph topology dominates feature aggregation and therefore is the key to extracting representative features. In this work, we propose a novel Channel-wise Topology Refinement Graph Convolution (CTR-GC) to dynamically learn different topologies and effectively aggregate joint features in different channels for skeleton-based action recognition. The proposed CTR-GC models channel-wise topologies through learning a shared topology as a generic prior for all channels and refining it with channel-specific correlations for each channel. Our refinement method introduces few extra parameters and significantly reduces the difficulty of modeling channel-wise topologies. Furthermore, via reformulating graph convolutions into a unified form, we find that CTR-GC relaxes strict constraints of graph convolutions, leading to stronger representation capability. Combining CTR-GC with temporal modeling modules, we develop a powerful graph convolutional network named CTR-GCN which notably outperforms state-of-the-art methods on the NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets.

<img src="https://user-images.githubusercontent.com/58767402/223147561-9158fd51-8963-47c9-9338-de70470820cc.png" width="800"/>
</div>

## Usage

Assume that you are located at `$MMACTION2/projects/ctrgcn`.

```shell
```

```

```shell
```

```bash
mim train mmaction configs/ctrgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py
```

```bash
mim train mmaction configs/ctrgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py --launcher pytorch --gpus 8
```

```bash
mim train mmaction configs/ctrgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py --launcher slurm \
```

```bash

```

```bash
mim test mmaction configs/ctrgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py --checkpoint $CHECKPOINT
```

```bash
mim test mmaction configs/ctrgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py --checkpoint $CHECKPOINT --launcher pytorch --gpus 8
```

```bash
mim test mmaction configs/ctrgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py --checkpoint $CHECKPOINT --launcher slurm \
```

## Results

|       uniform 100       |  joint   |  8   |  CTRGCN  |   89.6   |     10 clips     | [config](./configs/ctrgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/projects/ctrgcn/ctrgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d/ctrgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d_20230308-7aba454e.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/projects/ctrgcn/ctrgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d/ctrgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.log) |

|       uniform 100       |  joint   |  8   |  CTRGCN  |   89.0   |     10 clips     | [config](./configs/ctrgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/projects/ctrgcn/ctrgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d/ctrgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d_20230308-950dca0a.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/projects/ctrgcn/ctrgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d/ctrgcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d.log) |

```bibtex
@inproceedings{chen2021channel,
  title={Channel-wise topology refinement graph convolution for skeleton-based action recognition},
  author={Chen, Yuxin and Zhang, Ziqi and Yuan, Chunfeng and Li, Bing and Deng, Ying and Hu, Weiming},
  pages={13359--13368},
}
```

# 分布式训练的迁移

我们已经将[MMGeneration 1.x](https://github.com/open-mmlab/mmgeneration/tree/1.x)合并至MMagic。以下是针对MMGeneration中分布式训练的迁移事项。

在0.x版中，MMGeneration使用`DDPWrapper`和`DynamicRunner`来训练对应的静态和动态模型（例如PGGAN和StyleGANv2），但在1.x 版中，我们使用MMEngine提供的`MMSeparateDistributedDataParallel`来实现分布式训练。

如下是配置前后对比：

    <thead>
      <tr>
        <th> 0.x版中的静态模型 </th>
        <th> 1.x版中的静态模型 </th>
      </tr>
    </thead>
<tbody>
<tr>

```python
# 使用DDPWrapper
use_ddp_wrapper = True
find_unused_parameters = False

    is_dynamic_ddp=False)
```

</td>

```python
model_wrapper_cfg = dict(
    type='MMSeparateDistributedDataParallel',
    broadcast_buffers=False,
    find_unused_parameters=False)
```

</td>
</tr>
</tbody>
</table>

<thead>
  <tr>
    <th> 0.x版中的动态模型 </th>
    <th> 1.x版中的动态模型 </th>
  </tr>
</thead>
<tbody>
<tr>

```python
use_ddp_wrapper = False

</tr>
</tbody>
</table>

<thead>
  <tr>
  </tr>
</thead>
<tbody>
<tr>

```python

# 使用DynamicRunner
    is_dynamic_ddp=True)
```

</td>

```python
    find_unused_parameters=True) # 针对动态模型，设置`find_unused_parameters`标志为True
```

</td>

</tr>
</tbody>
</table>

首先，非常感谢大家对 ncnn 组件的关注
为了方便大家使用 ncnn 组件，up主特意写了这篇使用指北，以烂大街的 alexnet 作为例子

### 准备caffe网络和模型

caffe 的网络和模型通常是搞深度学习的研究者训练出来的，一般来说训练完会有
```
```
部署的时候只需要 TEST 过程，所以有 deploy.prototxt 和 caffemodel 就足够了

alexnet 的 deploy.prototxt 可以在这里下载

alexnet 的 caffemodel 可以在这里下载

### 转换ncnn网络和模型

caffe 自带了工具可以把老版本的 caffe 网络和模型转换为新版（ncnn的工具只认识新版
```
upgrade_net_proto_text [老prototxt] [新prototxt]
upgrade_net_proto_binary [老caffemodel] [新caffemodel]
```
输入层改用 Input，因为每次只需要做一个图片，所以第一个 dim 设为 1
```
layer {
}
```
使用 caffe2ncnn 工具转换为 ncnn 的网络描述和模型
```
```
### 去除可见字符串

有 param 和 bin 文件其实已经可以用了，但是 param 描述文件是明文的，如果放在 APP 分发出去容易被窥探到网络结构（说得好像不明文就看不到一样

```
```

使用 ncnn2mem 工具转换为二进制描述文件和内存模型，生成 alexnet.param.bin 和两个静态数组的代码文件
```
```
### 加载模型

直接加载 param 和 bin，适合快速验证效果使用
```cpp
```
加载二进制的 param.bin 和 bin，没有可见字符串，适合 APP 分发模型资源
```cpp
```
从内存引用加载网络和模型，没有可见字符串，模型数据全在代码里头，没有任何外部文件
另外，android apk 打包的资源文件读出来也是内存块
```cpp
```
以上三种都可以加载模型，其中内存引用方式加载是 zero-copy 的，所以使用 net 模型的来源内存块必须存在

### 卸载模型
```cpp
```

### 输入和输出

ncnn 用自己的数据结构 Mat 来存放输入和输出数据
输入图像的数据要转换为 Mat，依需要减去均值和乘系数
```cpp

```

### 卸载模型
```cpp
```

### 输入和输出

```cpp

```
执行前向网络，获得计算结果
```cpp
```
如果是二进制的 param.bin 方式，没有可见字符串，利用 alexnet.id.h 的枚举来代替 blob 的名字
```cpp
```

```
获取 Mat 中的输出数据，Mat 内部的数据通常是三维的，c / h / w，遍历所有获得全部分类的分数
```cpp
{
}
```
### 某些使用技巧

Extractor 有个多线程加速的开关，设置线程数能加快计算
```cpp
```
Mat 转换图像的时候可以顺便转换颜色和缩放大小，这些顺带的操作也是有优化的
支持 RGB2GRAY GRAY2RGB RGB2BGR 等常用转换，支持缩小和放大
```cpp
```

```
Net 有从 FILE* 文件描述加载的接口，可以利用这点把多个网络和模型文件合并为一个，分发时能方便些，内存引用就无所谓了

```cpp
```

# ChineseCLIP

> [Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese](https://arxiv.org/abs/2211.01335)

The tremendous success of CLIP (Radford et al., 2021) has promoted the research and application of contrastive learning for vision-language pretraining. In this work, we construct a large-scale dataset of image-text pairs in Chinese, where most data are retrieved from publicly available datasets, and we pretrain Chinese CLIP models on the new dataset. We develop 5 Chinese CLIP models of multiple sizes, spanning from 77 to 958 million parameters. Furthermore, we propose a two-stage pretraining method, where the model is first trained with the image encoder frozen and then trained with all parameters being optimized, to achieve enhanced model performance. Our comprehensive experiments demonstrate that Chinese CLIP can achieve the state-of-the-art performance on MUGE, Flickr30K-CN, and COCO-CN in the setups of zero-shot learning and finetuning, and it is able to achieve competitive performance in zero-shot image classification based on the evaluation on the ELEVATER benchmark (Li et al., 2022). We have released our codes, models, and demos in https://github.com/OFA-Sys/Chinese-CLIP

<img src="https://github.com/open-mmlab/mmpretrain/assets/36138628/4d05e51f-d834-4ef5-bbf0-0e2f80fea461" width="80%"/>
</div>

**Use the model for zero-shot classification**

```python
from mmpretrain import ImageClassificationInferencer

inferencer = ImageClassificationInferencer(
    'cn-clip_resnet50_zeroshot-cls_cifar100',
    pretrained=True,
    classes=['鸟', '狗', '猫', '蛇'],
    text_prototype=['鸟', '狗', '猫', '蛇'],
)

prediction = inferencer('./demo/bird.JPEG')[0]
print('Results:', prediction['pred_class'])
```

Test:

```shell
python tools/test.py configs/chinese_clip/cn-clip_resnet50_zeroshot-cls_cifar100.py https://download.openmmlab.com/mmpretrain/v1.0/chinese_clip/cn-clip_resnet50_3rdparty_20230519-6a2b3eb2.pth
```

```

### Image Classification on CIFAR100

| Model                                           | Params (M) | Top-1 (%) |                          Config                          |                                    Download                                    |
| :---------------------------------------------- | :--------: | :-------: | :------------------------------------------------------: | :----------------------------------------------------------------------------: |
| `cn-clip_resnet50_zeroshot-cls_cifar100`\*      |   77.00    |   40.70   |   [config](cn-clip_resnet50_zeroshot-cls_cifar100.py)    | [model](https://download.openmmlab.com/mmpretrain/v1.0/chinese_clip/cn-clip_resnet50_3rdparty_20230519-6a2b3eb2.pth) |

| `cn-clip_vit-base-p16_zeroshot-cls_cifar100`\*  |   188.00   |   64.50   | [config](cn-clip_vit-base-p16_zeroshot-cls_cifar100.py)  | [model](https://download.openmmlab.com/mmpretrain/v1.0/chinese_clip/cn-clip_vit-base-p16_3rdparty_20230519-37fbc59e.pth) |
| `cn-clip_vit-large-p14_zeroshot-cls_cifar100`\* |   406.00   |   74.80   | [config](cn-clip_vit-large-p14_zeroshot-cls_cifar100.py) | [model](https://download.openmmlab.com/mmpretrain/v1.0/chinese_clip/cn-clip_vit-large-p14_3rdparty_20230519-3f844503.pth) |
| `cn-clip_vit-huge-p14_zeroshot-cls_cifar100`\*  |   958.00   |   79.10   | [config](cn-clip_vit-huge-p14_zeroshot-cls_cifar100.py)  | [model](https://download.openmmlab.com/mmpretrain/v1.0/chinese_clip/cn-clip_vit-huge-p14_3rdparty_20230519-e4f49b00.pth) |

*Models with * are converted from the [official repo](https://github.com/OFA-Sys/Chinese-CLIP). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex

```bibtex
@article{chinese-clip,
  title={Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese},
  author={Yang, An and Pan, Junshu and Lin, Junyang and Men, Rui and Zhang, Yichang and Zhou, Jingren and Zhou, Chang},
  journal={arXiv preprint arXiv:2211.01335},
}
```

# Introducing the Segment Anything Model (SAM) Inference Demo!

Welcome to the Segment Anything (SA) Inference Demo, a user-friendly implementation based on the original Segment Anything project. Our demo allows you to experience the power and versatility of the Segment Anything Model (SAM) through an easy-to-use API.

With this inference demo, you can explore the capabilities of the Segment Anything Model and witness its effectiveness in various tasks and image distributions. For more information on the original project, dataset, and model, please visit the official website at https://segment-anything.com.

- Python 3.10
- PyTorch 1.13
- MMEngine >= v0.7.2
- MMCV >= v2.0.0

We assume that you have already installed PyTorch. If not, please follow the instructions on the [PyTorch website](https://pytorch.org/).

**1. Install MMEngine & MMCV**

```shell
```

**2. Install MMPretrain**

```shell
```

```shell
pip install git+https://github.com/open-mmlab/mmpretrain.git@dev
```

**3. Install MMSegmentation**

```shell
pip install mmsegmentation
```

### Usage

Open the `sam_image_demo.ipynb` notebook and follow the instructions to run the demo.

# VideoSwin

[Video Swin Transformer](https://openaccess.thecvf.com/content/CVPR2022/html/Liu_Video_Swin_Transformer_CVPR_2022_paper.html)

The vision community is witnessing a modeling shift from CNNs to Transformers, where pure Transformer architectures have attained top accuracy on the major video recognition benchmarks. These video models are all built on Transformer layers that globally connect patches across the spatial and temporal dimensions. In this paper, we instead advocate an inductive bias of locality in video Transformers, which leads to a better speed-accuracy trade-off compared to previous approaches which compute self-attention globally even with spatial-temporal factorization. The locality of the proposed video architecture is realized by adapting the Swin Transformer designed for the image domain, while continuing to leverage the power of pre-trained image models. Our approach achieves state-of-the-art accuracy on a broad range of video recognition benchmarks, including on action recognition (84.9 top-1 accuracy on Kinetics-400 and 85.9 top-1 accuracy on Kinetics-600 with ~20xless pre-training data and ~3xsmaller model size) and temporal modeling (69.6 top-1 accuracy on Something-Something v2).

<img src="https://user-images.githubusercontent.com/58767402/191190475-3aecf940-c254-47fa-96a7-df2d2b3bae68.png" width="800"/>
</div>

| frame sampling strategy | resolution | gpus | backbone |   pretrain   | top1 acc | top5 acc |      reference top1 acc      |      reference top1 acc      | testing protocol | FLOPs | params |      config      |      ckpt      |      log       |
| :---------------------: | :--------: | :--: | :------: | :----------: | :------: | :------: | :--------------------------: | :--------------------------: | :--------------: | :---: | :----: | :--------------: | :------------: | :------------: |

|         32x2x1          |  224x224   |  8   |  Swin-T  | ImageNet-1k  |  78.90   |  93.77   | 78.84 \[[VideoSwin](https://github.com/SwinTransformer/Video-Swin-Transformer)\] | 93.76 \[[VideoSwin](https://github.com/SwinTransformer/Video-Swin-Transformer)\] | 4 clips x 3 crop |  88G  | 28.2M  | [config](/configs/recognition/swin/swin-tiny-p244-w877_in1k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/swin/swin-tiny-p244-w877_in1k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb/swin-tiny-p244-w877_in1k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb_20220930-241016b2.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/swin/swin-tiny-p244-w877_in1k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb/swin-tiny-p244-w877_in1k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb.log) |

|         32x2x1          |  224x224   |  8   |  Swin-S  | ImageNet-1k  |  80.54   |  94.46   | 80.58 \[[VideoSwin](https://github.com/SwinTransformer/Video-Swin-Transformer)\] | 94.45 \[[VideoSwin](https://github.com/SwinTransformer/Video-Swin-Transformer)\] | 4 clips x 3 crop | 166G  | 49.8M  | [config](/configs/recognition/swin/swin-small-p244-w877_in1k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/swin/swin-small-p244-w877_in1k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb/swin-small-p244-w877_in1k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb_20220930-e91ab986.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/swin/swin-small-p244-w877_in1k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb/swin-small-p244-w877_in1k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb.log) |

|         32x2x1          |  224x224   |  8   |  Swin-B  | ImageNet-1k  |  80.57   |  94.49   | 80.55 \[[VideoSwin](https://github.com/SwinTransformer/Video-Swin-Transformer)\] | 94.66 \[[VideoSwin](https://github.com/SwinTransformer/Video-Swin-Transformer)\] | 4 clips x 3 crop | 282G  | 88.0M  | [config](/configs/recognition/swin/swin-base-p244-w877_in1k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/swin/swin-base-p244-w877_in1k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb/swin-base-p244-w877_in1k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb_20220930-182ec6cc.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/swin/swin-base-p244-w877_in1k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb/swin-base-p244-w877_in1k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb.log) |

|         32x2x1          |  224x224   |  8   |  Swin-L  | ImageNet-22k |  83.46   |  95.91   |            83.1\*            |            95.9\*            | 4 clips x 3 crop | 604G  |  197M  | [config](/configs/recognition/swin/swin-large-p244-w877_in22k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/swin/swin-large-p244-w877_in22k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb/swin-large-p244-w877_in22k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb_20220930-78ad8b11.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/swin/swin-large-p244-w877_in22k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb/swin-large-p244-w877_in22k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb.log) |

| frame sampling strategy | resolution | gpus | backbone |   pretrain   | top1 acc | top5 acc | testing protocol | FLOPs | params |               config                |               ckpt                |               log                |

| :---------------------: | :--------: | :--: | :------: | :----------: | :------: | :------: | :--------------: | :---: | :----: | :---------------------------------: | :-------------------------------: | :------------------------------: |
|         32x2x1          |  224x224   |  16  |  Swin-L  | ImageNet-22k |  75.92   |  92.72   | 4 clips x 3 crop | 604G  |  197M  | [config](/configs/recognition/swin/swin-large-p244-w877_in22k-pre_16xb8-amp-32x2x1-30e_kinetics700-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/swin/swin-large-p244-w877_in22k-pre_16xb8-amp-32x2x1-30e_kinetics700-rgb/swin-large-p244-w877_in22k-pre_16xb8-amp-32x2x1-30e_kinetics700-rgb_20220930-f8d74db7.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/swin/swin-large-p244-w877_in22k-pre_16xb8-amp-32x2x1-30e_kinetics700-rgb/swin-large-p244-w877_in22k-pre_16xb8-amp-32x2x1-30e_kinetics700-rgb.log) |

| frame sampling strategy | resolution | gpus | backbone |  pretrain   | top1 acc | top5 acc | testing protocol | FLOPs | params |               config                |               ckpt                |                log                |
| :---------------------: | :--------: | :--: | :------: | :---------: | :------: | :------: | :--------------: | :---: | :----: | :---------------------------------: | :-------------------------------: | :-------------------------------: |

|         32x2x1          |  224x224   |  32  |  Swin-S  | ImageNet-1k |  76.90   |  92.96   | 4 clips x 3 crop | 604G  |  197M  | [config](/configs/recognition/swin/swin-small-p244-w877_in1k-pre_32xb4-amp-32x2x1-30e_kinetics710-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/swin/swin-small-p244-w877_in1k-pre_32xb4-amp-32x2x1-30e_kinetics710-rgb/swin-small-p244-w877_in1k-pre_32xb4-amp-32x2x1-30e_kinetics710-rgb_20230612-8e082ff1.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/swin/swin-small-p244-w877_in1k-pre_32xb4-amp-32x2x1-30e_kinetics710-rgb/swin-small-p244-w877_in1k-pre_32xb4-amp-32x2x1-30e_kinetics710-rgb.log) |

2. The values in columns named after "reference" are the results got by testing on our dataset, using the checkpoints provided by the author with same model settings. `*` means that the numbers are copied from the paper.

4. Pre-trained image models can be downloaded from [Swin Transformer for ImageNet Classification](https://github.com/microsoft/Swin-Transformer#main-results-on-imagenet-with-pretrained-models).

For more details on data preparation, you can refer to [Kinetics](/tools/data/kinetics/README.md).

## Train

```shell
```

## Train

```shell
```

Example: train VideoSwin model on Kinetics-400 dataset in a deterministic option with periodic validation.

```shell
python tools/train.py configs/recognition/swin/swin-tiny-p244-w877_in1k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb.py \
```

## Test

```shell
```

Example: test VideoSwin model on Kinetics-400 dataset and dump the result to a pkl file.

```shell
python tools/test.py configs/recognition/swin/swin-tiny-p244-w877_in1k-pre_8xb8-amp-32x2x1-30e_kinetics400-rgb.py \
```

```shell
```

```BibTeX
@inproceedings{liu2022video,
  title={Video swin transformer},
  author={Liu, Ze and Ning, Jia and Cao, Yue and Wei, Yixuan and Zhang, Zheng and Lin, Stephen and Hu, Han},
  pages={3202--3211},
}
```

# Instaboost

> [Instaboost: Boosting instance segmentation via probability map guided copy-pasting](https://arxiv.org/abs/1908.07801)

Instance segmentation requires a large number of training samples to achieve satisfactory performance and benefits from proper data augmentation. To enlarge the training set and increase the diversity, previous methods have investigated using data annotation from other domain (e.g. bbox, point) in a weakly supervised mechanism. In this paper, we present a simple, efficient and effective method to augment the training set using the existing instance mask annotations. Exploiting the pixel redundancy of the background, we are able to improve the performance of Mask R-CNN for 1.7 mAP on COCO dataset and 3.3 mAP on Pascal VOC dataset by simply introducing random jittering to objects. Furthermore, we propose a location probability map based approach to explore the feasible locations that objects can be placed based on local appearance similarity. With the guidance of such map, we boost the performance of R101-Mask R-CNN on instance segmentation from 35.7 mAP to 37.9 mAP without modifying the backbone or network structure. Our method is simple to implement and does not increase the computational complexity. It can be integrated into the training pipeline of any instance segmentation model without affecting the training and inference efficiency.

<img src="https://user-images.githubusercontent.com/40661020/143894053-ecfe8406-1a05-461b-953c-efeef22d7a60.png" height="300"/>
</div>

Configs in this directory is the implementation for ICCV2019 paper "InstaBoost: Boosting Instance Segmentation Via Probability Map Guided Copy-Pasting" and provided by the authors of the paper. InstaBoost is a data augmentation method for object detection and instance segmentation. The paper has been released on [`arXiv`](https://arxiv.org/abs/1908.07801).

## Usage

You need to install `instaboostfast` before using it.

```shell
pip install instaboostfast
```

The code and more details can be found [here](https://github.com/GothicAi/Instaboost).

### Integration with MMDetection

```shell
```

InstaBoost have been already integrated in the data pipeline, thus all you need is to add or change **InstaBoost** configurations after **LoadImageFromFile**. We have provided examples like [this](mask_rcnn_r50_fpn_instaboost_4x#L121). You can refer to [`InstaBoostConfig`](https://github.com/GothicAi/InstaBoost-pypi#instaboostconfig) for more details.

- All models were trained on `coco_2017_train` and tested on `coco_2017_val` for convenience of evaluation and comparison. In the paper, the results are obtained from `test-dev`.
- To balance accuracy and training time when using InstaBoost, models released in this page are all trained for 48 Epochs. Other training and testing configs strictly follow the original framework.

- For results and models in MMDetection V1.x, please refer to [Instaboost](https://github.com/GothicAi/Instaboost).

|    Network    |    Backbone     | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                           Config                            |                                                                                                                                                                                    Download                                                                                                                                                                                    |

|  Mask R-CNN   |    R-50-FPN     |   4x    |   4.4    |      17.5      |  40.6  |  36.6   |     [config](./mask-rcnn_r50_fpn_instaboost-4x_coco.py)     |                  [model](https://download.openmmlab.com/mmdetection/v2.0/instaboost/mask_rcnn_r50_fpn_instaboost_4x_coco/mask_rcnn_r50_fpn_instaboost_4x_coco_20200307-d025f83a.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/instaboost/mask_rcnn_r50_fpn_instaboost_4x_coco/mask_rcnn_r50_fpn_instaboost_4x_coco_20200307_223635.log.json)                   |

|  Mask R-CNN   |    R-101-FPN    |   4x    |   6.4    |                |  42.5  |  38.0   |    [config](./mask-rcnn_r101_fpn_instaboost-4x_coco.py)     |             [model](https://download.openmmlab.com/mmdetection/v2.0/instaboost/mask_rcnn_r101_fpn_instaboost_4x_coco/mask_rcnn_r101_fpn_instaboost_4x_coco_20200703_235738-f23f3a5f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/instaboost/mask_rcnn_r101_fpn_instaboost_4x_coco/mask_rcnn_r101_fpn_instaboost_4x_coco_20200703_235738.log.json)             |

|  Mask R-CNN   | X-101-64x4d-FPN |   4x    |   10.7   |                |  44.7  |  39.7   | [config](./mask-rcnn_x101-64x4d_fpn_instaboost-4x_coco.py)  | [model](https://download.openmmlab.com/mmdetection/v2.0/instaboost/mask_rcnn_x101_64x4d_fpn_instaboost_4x_coco/mask_rcnn_x101_64x4d_fpn_instaboost_4x_coco_20200515_080947-8ed58c1b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/instaboost/mask_rcnn_x101_64x4d_fpn_instaboost_4x_coco/mask_rcnn_x101_64x4d_fpn_instaboost_4x_coco_20200515_080947.log.json) |

| Cascade R-CNN |    R-101-FPN    |   4x    |   6.0    |      12.0      |  43.7  |  38.0   | [config](./cascade-mask-rcnn_r50_fpn_instaboost-4x_coco.py) |  [model](https://download.openmmlab.com/mmdetection/v2.0/instaboost/cascade_mask_rcnn_r50_fpn_instaboost_4x_coco/cascade_mask_rcnn_r50_fpn_instaboost_4x_coco_20200307-c19d98d9.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/instaboost/cascade_mask_rcnn_r50_fpn_instaboost_4x_coco/cascade_mask_rcnn_r50_fpn_instaboost_4x_coco_20200307_223646.log.json)   |

```latex
@inproceedings{fang2019instaboost,
  title={Instaboost: Boosting instance segmentation via probability map guided copy-pasting},
  author={Fang, Hao-Shu and Sun, Jianhua and Wang, Runzhong and Gou, Minghao and Li, Yong-Lu and Lu, Cewu},
  pages={682--691},
}
```

# FAQ

## General

**Q1** I'm getting the warning like `unexpected key in source state_dict: fc.weight, fc.bias`, is there something wrong?

**A** It's not an error. It occurs because the backbone network is pretrained on image classification tasks, where the last fc layer is required to generate the classification output. However, the fc layer is no longer needed when the backbone network is used to extract features in downstream tasks, and therefore these weights can be safely skipped when loading the checkpoint.

**Q2** MMOCR terminates with an error: `shapely.errors.TopologicalError: The operation 'GEOSIntersection_r' could not be performed. Likely cause is invalidity of the geometry`. How could I fix it?

**A** This error occurs because of some invalid polygons (e.g., polygons with self-intersections) existing in the dataset or generated by some non-rigorous data transforms. These polygons can be fixed by adding `FixInvalidPolygon` transform after the transform likely to introduce invalid polygons. For example, a common practice is to append it after `LoadOCRAnnotations` in both train and test pipeline. The resulting pipeline should look like:

```python
    ...
    dict(
    ),
    dict(type='FixInvalidPolygon', min_poly_points=4),
    ...
]
```

dict(
    ),
    ...
]
```

In practice, we find that Totaltext contains some invalid polygons and using `FixInvalidPolygon` is a must. [Here](https://github.com/open-mmlab/mmocr/blob/27b6a68586b9a040678fe083bcf60662ae1b9261/configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_totaltext.py) is an example config.

**Q3** Getting `libpng warning: iCCP: known incorrect sRGB profile` when loading images with `cv2` backend.

**A** This is a warning from `libpng` and it is safe to ignore. It is caused by the `icc` profile in the image. You can use `pillow` backend to avoid this warning:

```python
    dict(
        imdecode_backend='pillow'),
    ...
]
```

**Q1** What are the steps to train text recognition models with my own dictionary?

dict(
    ...
]
```

**A** In MMOCR 1.0, you only need to modify the config and point `Dictionary` to your custom dict file. For example, if you want to train SAR model (https://github.com/open-mmlab/mmocr/blob/75c06d34bbc01d3d11dfd7afc098b6cdeee82579/configs/textrecog/sar/sar_resnet31_parallel-decoder_5e_st-sub_mj-sub_sa_real.py) with your own dictionary placed at `/my/dict.txt`, you can modify `dictionary.dict_file` term in [base config](https://github.com/open-mmlab/mmocr/blob/75c06d34bbc01d3d11dfd7afc098b6cdeee82579/configs/textrecog/sar/_base_sar_resnet31_parallel-decoder.py#L1) to:

```python
dictionary = dict(
    type='Dictionary',
    dict_file='/my/dict.txt',
    with_start=True,
    with_end=True,
    same_start_end=True,
    with_padding=True,
    with_unknown=True)
```

```python
```

Now you are good to go. You can also find more information in [Dictionary API](https://mmocr.readthedocs.io/en/dev-1.x/api/generated/mmocr.models.common.Dictionary.html#mmocr.models.common.Dictionary).

**Q2** How to properly visualize non-English characters?

**A** You can customize `font_families` or `font_properties` in visualizer. For example, to visualize Korean:

`configs/textrecog/_base_/default_runtime.py`:

```python
    type='TextRecogLocalVisualizer',
    name='visualizer',
    font_families='NanumGothic', # new feature
    vis_backends=vis_backends)
```

It's also fine to pass the font path to visualizer:

```python
    font_properties='path/to/font_file',

```

```python
```

# 准备模型

要在 OpenCompass 中支持新模型的评测，有以下几种方式：

1. 基于 HuggingFace 的模型
2. 基于 API 的模型
3. 自定义模型

## 基于 HuggingFace 的模型

在 OpenCompass 中，我们支持直接从 Huggingface 的 `AutoModel.from_pretrained` 和
`AutoModelForCausalLM.from_pretrained` 接口构建评测模型。如果需要评测的模型符合 HuggingFace 模型通常的生成接口，
则不需要编写代码，直接在配置文件中指定相关配置即可。

如下，为一个示例的 HuggingFace 模型配置文件：

```python
# 使用 `HuggingFace` 评测 HuggingFace 中 AutoModel 支持的模型
# 使用 `HuggingFaceCausalLM` 评测 HuggingFace 中 AutoModelForCausalLM 支持的模型

models = [
    dict(
        # 以下参数为 `HuggingFaceCausalLM` 的初始化参数
        batch_padding=False,
        # 以下参数为各类模型都有的参数，非 `HuggingFaceCausalLM` 的初始化参数

batch_padding=False,
    )
]
```

对以上一些参数的说明：

- `batch_padding=False`：如为 False，会对一个批次的样本进行逐一推理；如为 True，则会对一个批次的样本进行填充，
  组成一个 batch 进行推理。对于部分模型，这样的填充可能导致意料之外的结果；如果评测的模型支持样本填充，
  则可以将该参数设为 True，以加速推理。
- `padding_side='left'`：在左侧进行填充，因为不是所有模型都支持填充，在右侧进行填充可能会干扰模型的输出。
- `truncation_side='left'`：在左侧进行截断，评测输入的 prompt 通常包括上下文样本 prompt 和输入 prompt 两部分，
  如果截断右侧的输入 prompt，可能导致生成模型的输入和预期格式不符，因此如有必要，应对左侧进行截断。

在评测时，OpenCompass 会使用配置文件中的 `type` 与各个初始化参数实例化用于评测的模型，
其他参数则用于推理及总结等过程中，与模型相关的配置。例如上述配置文件，我们会在评测时进行如下实例化过程：

```python
model = HuggingFaceCausalLM(
    path='huggyllama/llama-7b',
    tokenizer_path='huggyllama/llama-7b',
    tokenizer_kwargs=dict(padding_side='left', truncation_side='left'),
    max_seq_len=2048,
)
```

## 基于 API 的模型

path='huggyllama/llama-7b',
)
```

OpenCompass 目前支持以下基于 API 的模型推理：

- OpenAI（`opencompass.models.OpenAI`）
- Coming soon

以下，我们以 OpenAI 的配置文件为例，模型如何在配置文件中使用基于 API 的模型。

```python
from opencompass.models import OpenAI

models = [
    dict(
        type=OpenAI,                             # 使用 OpenAI 模型
        # 以下为 `OpenAI` 初始化参数
        path='gpt-4',                            # 指定模型类型
        key='YOUR_OPENAI_KEY',                   # OpenAI API Key
        max_seq_len=2048,                        # 最大输入长度
        # 以下参数为各类模型都有的参数，非 `OpenAI` 的初始化参数
        abbr='GPT-4',                            # 模型简称
        run_cfg=dict(num_gpus=0),                # 资源需求（不需要 GPU）
        max_out_len=512,                         # 最长生成长度
        batch_size=1,                            # 批次大小
    ),
]
```

## 自定义模型

max_out_len=512,                         # 最长生成长度
    ),
]
```

## 自定义模型

如果以上方式无法支持你的模型评测需求，请参考 [支持新模型](../advanced_guides/new_model.md) 在 OpenCompass 中增添新的模型支持。

# DDOD

> [Disentangle Your Dense Object Detector](https://arxiv.org/pdf/2107.02963.pdf)

Deep learning-based dense object detectors have achieved great success in the past few years and have been applied to numerous multimedia applications such as video understanding. However, the current training pipeline for dense detectors is compromised to lots of conjunctions that may not hold. In this paper, we investigate three such important conjunctions: 1) only samples assigned as positive in classification head are used to train the regression head; 2) classification and regression share the same input feature and computational fields defined by the parallel head architecture; and 3) samples distributed in different feature pyramid layers are treated equally when computing the loss. We first carry out a series of pilot experiments to show disentangling such conjunctions can lead to persistent performance improvement. Then, based on these findings, we propose Disentangled Dense Object Detector(DDOD), in which simple and effective disentanglement mechanisms are designed and integrated into the current state-of-the-art dense object detectors. Extensive experiments on MS COCO benchmark show that our approach can lead to 2.0 mAP, 2.4 mAP and 2.2 mAP absolute improvements on RetinaNet, FCOS, and ATSS baselines with negligible extra overhead. Notably, our best model reaches 55.0 mAP on the COCO test-dev set and 93.5 AP on the hard subset of WIDER FACE, achieving new state-of-the-art performance on these two competitive benchmarks. Code is available at https://github.com/zehuichen123/DDOD.

<img src="https://user-images.githubusercontent.com/17425982/159212920-2e99d433-82c9-46cf-8f3a-32fdf3c566f5.png"/>
</div>

|   Model   | Backbone |  Style  | Lr schd | Mem (GB) | box AP |               Config                |                                                                                                                                Download                                                                                                                                |
| :-------: | :------: | :-----: | :-----: | :------: | :----: | :---------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

| DDOD-ATSS |   R-50   | pytorch |   1x    |   3.4    |  41.7  | [config](./ddod_r50_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/ddod/ddod_r50_fpn_1x_coco/ddod_r50_fpn_1x_coco_20220523_223737-29b2fc67.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/ddod/ddod_r50_fpn_1x_coco/ddod_r50_fpn_1x_coco_20220523_223737.log.json) |

```latex
@inproceedings{chen2021disentangle,
title={Disentangle Your Dense Object Detector},
author={Chen, Zehui and Yang, Chenhongyi and Li, Qiaofei and Zhao, Feng and Zha, Zheng-Jun and Wu, Feng},
booktitle={Proceedings of the 29th ACM International Conference on Multimedia},
pages={4939--4948},
}
```

## Usage

Please refer to [Get Started](https://mmpretrain.readthedocs.io/en/latest/get_started.html) to install
MMPreTrain.

```shell
```

Prepare the ImageNet-2012 dataset according to the [instruction](https://mmpretrain.readthedocs.io/en/latest/user_guides/dataset_prepare.html#imagenet).

```bash
mim train mmpretrain configs/examplenet_8xb32_in1k.py
```

```bash

```bash
```

```bash
mim train mmpretrain configs/examplenet_8xb32_in1k.py --launcher pytorch --gpus 8
```

```bash
mim train mmpretrain configs/examplenet_8xb32_in1k.py --launcher slurm \
```

```bash
mim test mmpretrain configs/examplenet_8xb32_in1k.py --checkpoint $CHECKPOINT
```

```bash
mim test mmpretrain configs/examplenet_8xb32_in1k.py --checkpoint $CHECKPOINT --launcher pytorch --gpus 8
```

```bash
mim test mmpretrain configs/examplenet_8xb32_in1k.py --checkpoint $CHECKPOINT --launcher slurm \
```

## Results

```bash
```

## Results

|       Model        |   Pretrain   | Top-1 (%) | Top-5 (%) |                 Config                  |                Download                |
| :----------------: | :----------: | :-------: | :-------: | :-------------------------------------: | :------------------------------------: |
|  ExampleNet-tiny   | From scratch |   82.33   |   96.15   | [config](./mvitv2-tiny_8xb256_in1k.py)  | [model](MODEL-LINK) \| [log](LOG-LINK) |
| ExampleNet-small\* | From scratch |   83.63   |   96.51   | [config](./mvitv2-small_8xb256_in1k.py) |          [model](MODEL-LINK)           |
| ExampleNet-base\*  | From scratch |   84.34   |   96.86   | [config](./mvitv2-base_8xb256_in1k.py)  |          [model](MODEL-LINK)           |

```BibTeX
}
```

to MMPreTrain projects.

    <!-- The code's design shall follow existing interfaces and convention. For example, each model component should be registered into `mmpretrain.registry.MODELS` and configurable via a config file. -->

    <!-- If you are reproducing the result from a paper, train your model from scratch and verified that the final result can match the original result. Usually, ±0.1% is acceptable for the image classification task on ImageNet-1k. -->

    <!-- Unit tests for the major module are required. [Example](https://github.com/open-mmlab/mmpretrain/blob/main/tests/test_models/test_backbones/test_vision_transformer.py) -->

    <!-- It will used for MMPreTrain to acquire your models. [Example](https://github.com/open-mmlab/mmpretrain/blob/main/configs/mvit/metafile.yml). In particular, you may have to refactor this README into a standard one. [Example](https://github.com/open-mmlab/mmpretrain/blob/main/configs/swin_transformer/README.md) -->

# DETR

> [End-to-End Object Detection with Transformers](https://arxiv.org/abs/2005.12872)

We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the detection pipeline, effectively removing the need for many hand-designed components like a non-maximum suppression procedure or anchor generation that explicitly encode our prior knowledge about the task. The main ingredients of the new framework, called DEtection TRansformer or DETR, are a set-based global loss that forces unique predictions via bipartite matching, and a transformer encoder-decoder architecture. Given a fixed small set of learned object queries, DETR reasons about the relations of the objects and the global image context to directly output the final set of predictions in parallel. The new model is conceptually simple and does not require a specialized library, unlike many other modern detectors. DETR demonstrates accuracy and run-time performance on par with the well-established and highly-optimized Faster RCNN baseline on the challenging COCO object detection dataset. Moreover, DETR can be easily generalized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive baselines.

<img src="https://user-images.githubusercontent.com/40661020/143878072-0a7434e4-416b-4315-aeea-a8297f4d6453.png"/>
</div>

| Backbone | Model | Lr schd | Mem (GB) | Inf time (fps) | box AP |                 Config                 |                                                                                                                                      Download                                                                                                                                      |
| :------: | :---: | :-----: | :------: | :------------: | :----: | :------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

|   R-50   | DETR  |  150e   |   7.9    |                |  39.9  | [config](./detr_r50_8xb2-150e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/detr/detr_r50_8xb2-150e_coco/detr_r50_8xb2-150e_coco_20221023_153551-436d03e8.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/detr/detr_r50_8xb2-150e_coco/detr_r50_8xb2-150e_coco_20221023_153551.log.json) |

We provide the config files for DETR: [End-to-End Object Detection with Transformers](https://arxiv.org/abs/2005.12872).

```latex
@inproceedings{detr,
  author    = {Nicolas Carion and
               Francisco Massa and
               Gabriel Synnaeve and
               Nicolas Usunier and
               Alexander Kirillov and
               Sergey Zagoruyko},
  title     = {End-to-End Object Detection with Transformers},
  year      = {2020}
}
```

<details>

```bibtex
}

```

<details>

```bibtex

<details>

```bibtex
}
```

| Arch                                                     | Input Size | Mean / w. flip | Mean@0.1 |                           ckpt                           |                           log                            |
| :------------------------------------------------------- | :--------: | :------------: | :------: | :------------------------------------------------------: | :------------------------------------------------------: |

| [rtmpose-m](/configs/body_2d_keypoint/rtmpose/mpii/rtmpose-m_8xb64-210e_mpii-256x256.py) |  256x256   |     0.907      |  0.348   | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-mpii_pt-aic-coco_210e-256x256-ec4dbec8_20230206.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-mpii_pt-aic-coco_210e-256x256-ec4dbec8_20230206.json) |

To support a new data format, you can either convert them to existing formats (COCO format or PASCAL format) or directly convert them to the middle format. You could also choose to convert them offline (before training by a script) or online (implement a new dataset and do the conversion at training). In MMDetection, we recommend to convert the data into COCO formats and do the conversion offline, thus you only need to modify the config's data annotation paths and classes after the conversion of your data.

The simplest way is to convert your dataset to existing dataset formats (COCO or PASCAL VOC).

The annotation JSON files in COCO format has the following necessary keys:

```python
    {
        'file_name': 'COCO_val2014_000000001268.jpg',
        'id': 1268
    },
    ...
],

    {

```python
    {
        'id': 1268
    },
    ...
],

    {
        'segmentation': [[192.81,
            247.09,
            ...
            219.03,
            249.06]],  # If you have mask labels, and it is in polygon XY point coordinate format, you need to ensure that at least 3 point coordinates are included. Otherwise, it is an invalid polygon.
        'area': 1035.749,
        'bbox': [192.81, 224.8, 74.73, 33.43],
        'category_id': 16,
        'id': 42986
    },
    ...
],

    {'id': 0, 'name': 'car'},
 ]
```

There are three necessary keys in the JSON file:

- `categories`: contains the list of categories names and their ID.

After the data pre-processing, there are two steps for users to train the customized new dataset with existing format (e.g. COCO format):

1. The `data` field. Specifically, you need to explicitly add the `metainfo=dict(classes=classes)` fields in `train_dataloader.dataset`, `val_dataloader.dataset` and `test_dataloader.dataset` and `classes` must be a tuple type.

```python

_base_ = './cascade_mask_rcnn_r50_fpn_1x_coco.py'

data_root='path/to/your/'

        # explicitly add your class names to the field `metainfo`

        metainfo=dict(classes=classes),
        ann_file='train/annotation_data',
        data_prefix=dict(img='train/image_data')
        )
    )

        ann_file='val/annotation_data',
        data_prefix=dict(img='val/image_data')
        )
    )

num_workers=2,
        ann_file='test/annotation_data',
        data_prefix=dict(img='test/image_data')
        )
    )

# explicitly over-write all the `num_classes` field from default 80 to 5.
            dict(
                # explicitly over-write all the `num_classes` field from default 80 to 5.
                num_classes=5),
            dict(
            dict(

num_classes=5),
            dict(
                num_classes=5)],
    # explicitly over-write all the `num_classes` field from default 80 to 5.
    mask_head=dict(num_classes=5)))
```

Assuming your customized dataset is COCO format, make sure you have the correct annotations in the customized dataset:

1. The length for `categories` field in annotations should exactly equal the tuple length of `classes` fields in your config, meaning the number of classes (e.g. 5 in this example).

2. The `classes` fields in your config file should have exactly the same elements and the same order with the `name` in `categories` of annotations. MMDetection automatically maps the uncontinuous `id` in `categories` to the continuous label indices, so the string order of `name` in `categories` field affects the order of label indices. Meanwhile, the string order of `classes` in config affects the label text during visualization of predicted bounding boxes.
3. The `category_id` in `annotations` field should be valid, i.e., all values in `category_id` should belong to `id` in `categories`.

Here is a valid example of annotations:

```python

    {
            247.09,
            ...
            219.03,
            249.06]],  # if you have mask labels

```python

    {
            247.09,
            ...
            219.03,
    },
    ...
],

# MMDetection automatically maps the uncontinuous `id` to the continuous label indices.
    {'id': 1, 'name': 'a'}, {'id': 3, 'name': 'b'}, {'id': 4, 'name': 'c'}, {'id': 16, 'name': 'd'}, {'id': 17, 'name': 'e'},
 ]
```

We use this way to support CityScapes dataset. The script is in [cityscapes.py](../../../tools/dataset_converters/cityscapes.py) and we also provide the finetuning [configs](../../../configs/cityscapes).

**Note**

1. For instance segmentation datasets, **MMDetection only supports evaluating mask AP of dataset in COCO format for now**.

**Note**

2. It is recommended to convert the data offline before training, thus you can still use `CocoDataset` and only need to modify the path of annotations and the training classes.

### Reorganize new data format to middle format

It is also fine if you do not want to convert the annotation format to COCO or PASCAL format.
Actually, we define a simple annotation format in MMEninge's [BaseDataset](https://github.com/open-mmlab/mmengine/blob/main/mmengine/dataset/base_dataset.py#L116) and all existing datasets are
processed to be compatible with it, either online or offline.

The annotation of the dataset must be in `json` or `yaml`, `yml` or `pickle`, `pkl` format; the dictionary stored in the annotation file must contain two fields `metainfo` and `data_list`.  The `metainfo` is a dictionary, which contains the metadata of the dataset, such as class information; `data_list` is a list, each element in the list is a dictionary, the dictionary defines the raw data of one image, and each raw data contains a or several training/testing samples.

```python
{
        {
            'classes': ('person', 'bicycle', 'car', 'motorcycle'),
            ...
        },
        [
            {
                "img_path": "xxx/xxx_1.jpg",
                "height": 604,
                "width": 640,
                "instances":
                [
                  {
                    "bbox": [0, 0, 10, 20],
                    "bbox_label": 1,

                [
                  {
                    "ignore_flag": 0
                  },
                  {
                    "bbox": [10, 10, 110, 120],
                    "bbox_label": 2,
                  }
                ]
              },
            {
                "img_path": "xxx/xxx_2.jpg",
                "height": 320,
                "width": 460,
                [
                  {
                    "bbox": [10, 0, 20, 20],
                    "bbox_label": 3,
                    "ignore_flag": 1,
                  }
                ]
              },
            ...
        ]
}
```

Some datasets may provide annotations like crowd/difficult/ignored bboxes, we use `ignore_flag`to cover them.

}
                ]
              },
            ...
        ]
}
```

After obtaining the above standard data annotation format, you can directly use [BaseDetDataset](../../../mmdet/datasets/base_det_dataset.py#L13) of MMDetection in the configuration , without conversion.

### An example of customized dataset

Assume the annotation is in a new format in text files.
The bounding boxes annotations are stored in text file `annotation.txt` as the following

```
#
000001.jpg
1280 720
2
10 20 40 60 1
20 40 50 60 2
#
000002.jpg
1280 720
3
50 20 40 60 2
20 40 30 45 2
30 40 50 60 3
```

We can create a new dataset in `mmdet/datasets/my_dataset.py` to load the data.

```python

from mmdet.base_det_dataset import BaseDetDataset
from mmdet.registry import DATASETS

class MyDataset(BaseDetDataset):

```python

       'classes': ('person', 'bicycle', 'car', 'motorcycle'),
        'palette': [(220, 20, 60), (119, 11, 32), (0, 0, 142), (0, 0, 230)]
    }

    def load_data_list(self, ann_file):
        ann_list = mmengine.list_from_file(ann_file)

        data_infos = []
        for i, ann_line in enumerate(ann_list):
            if ann_line != '#':
                continue

            img_shape = ann_list[i + 2].split(' ')
            width = int(img_shape[0])
            height = int(img_shape[1])
            bbox_number = int(ann_list[i + 3])

            instances = []
            for anns in ann_list[i + 4:i + 4 + bbox_number]:
                instance = {}
                instance['bbox'] = [float(ann) for ann in anns.split(' ')[:4]]
                instance['bbox_label']=int(anns[4])

instance = {}
 				instances.append(instance)

            data_infos.append(
                dict(
                    img_path=ann_list[i + 1],
                    img_id=i,
                    width=width,
                    height=height,
                    instances=instances
                ))

        return data_infos
```

Then in the config, to use `MyDataset` you can modify the config as the following

```python
    type='MyDataset',
    ann_file = 'image_list.txt',
)
```

MMEngine also supports many dataset wrappers to mix the dataset or modify the dataset distribution for training.

For detailed usage, see [MMEngine Dataset Wrapper](#TODO).

## Modify Dataset Classes

With existing dataset types, we can modify the metainfo of them to train subset of the annotations.
For example, if you want to train only three classes of the current dataset,
you can modify the classes of dataset.
The dataset will filter out the ground truth boxes of other classes automatically.

```python
classes = ('person', 'bicycle', 'car')
        metainfo=dict(classes=classes))
    )
    )
    )
```

**Note**:

)
    )
    )
```

**Note**:

- Before MMDetection v2.5.0, the dataset will filter out the empty GT images automatically if the classes are set and there is no way to disable that through config. This is an undesirable behavior and introduces confusion because if the classes are not set, the dataset only filter the empty GT images when `filter_empty_gt=True` and `test_mode=False`. After MMDetection v2.5.0, we decouple the image filtering process and the classes modification, i.e., the dataset will only filter empty GT images when `filter_cfg=dict(filter_empty_gt=True)` and `test_mode=False`, no matter whether the classes are set. Thus, setting the classes only influences the annotations of classes used for training and users could decide whether to filter empty GT images by themselves.

- When directly using `BaseDataset` in MMEngine or `BaseDetDataset` in MMDetection, users cannot filter images without GT by modifying the configuration, but it can be solved in an offline way.
- Please remember to modify the `num_classes` in the head when specifying `classes` in dataset. We implemented [NumClassCheckHook](../../../mmdet/engine/hooks/num_class_check_hook.py) to check whether the numbers are consistent since v2.9.0(after PR#4508).

## COCO Panoptic Dataset

Now we support COCO Panoptic Dataset, the format of panoptic annotations is different from COCO format.
Both the foreground and the background will exist in the annotation file.
The annotation json files in COCO Panoptic format has the following necessary keys:

```python
    {
        'id': 1268
    },
    ...
]

    {
        'filename': '000000001268.jpg',

'height': 427,
        'id': 1268
    },
    ...
]

    {
        'segments_info': [
            {
                'id':8345037,  # One-to-one correspondence with the id in the annotation map.
                'category_id': 51,
                'iscrowd': 0,
                'bbox': (x1, y1, w, h),  # The bbox of the background is the outer rectangle of its mask.
                'area': 24315
            },
            ...
        ]
    },
    ...
]

'categories': [  # including both foreground categories and background categories
    {'id': 0, 'name': 'person'},
    ...
 ]
```

Moreover, the `seg` must be set to the path of the panoptic annotation images.

```python
dataset_type = 'CocoPanopticDataset'

```python

            img='train/image_data/', seg='train/panoptic/image_annotation_data/')
    )
)
            img='val/image_data/', seg='val/panoptic/image_annotation_data/')
    )
)
            img='test/image_data/', seg='test/panoptic/image_annotation_data/')
    )
)
```

# 准备 HACS Segments

## 简介

```BibTeX
}
```

### 0. 下载视频

在我们开始准备数据集之前，请按照[官方代码库](https://github.com/hangzhaomit/HACS-dataset)的指令下载HACS Segments数据集中的视频。如果有视频缺失，您可以向HACS数据集存储库的维护者提交请求以获取缺失的视频。但是如果一些视频缺失，您仍然可以为MMAction2准备数据集。

在下载完数据集后，请将数据集文件夹移动到(或者使用软链接)`$MMACTION2/tools/data/hacs/`。文件夹结构应该如下所示：

```
mmaction2
├── data
├── tools
│   ├── hacs
│   │   ├── ..
│   │   ├── data
│   │   │   │   ├── ..

```

在开始之前，请确保您位于`$MMACTION2/tools/data/hacs/`路径下。

### 1. 提取特征

│   │   │   │   ├── ..

```

### 1. 提取特征

以下是使用[SlowOnly ResNet50 8x8](/configs/recognition/slowonly/slowonly_imagenet-pretrained-r50_16xb16-4x16x1-steplr-150e_kinetics700-rgb.py)在Kinetics700数据集上预训练的模型，从HACS视频中提取特征。对于每个视频，我们均匀采样100个视频片段，并提取700维输出（softmax之前）作为特征，即特征形状为100x700。

首先，我们使用如下命令生成数据集的视频列表：

```
```

这将生成一个位于`$MMACTION2/tools/data/hacs/`的`hacs_data.txt`文件，其内容格式如下：

```
```

接下来，我们使用[slowonly_feature_infer.py](/tools/data/hacs/slowonly_feature_infer.py) 配置文件来提取特征：

```
# 指定提取特征的GPU数量

# 下载预训练模型权重

```

```

```

我们将得到一个名为 `result.pkl` 的文件，其中包含每个视频的大小为100x700的特征。我们将特征重写为csv格式，并保存在 `$MMACTION2/data/HACS/` 目录下。

```
＃确保您位于$ $MMACTION2/tools/data/hacs/
```

### 2. 准备标注文件

我们首先从官方仓库下载标注文件：

```
```

```

```
```

解压缩后，应该有一个名为`HACS_v1.1.1`的文件夹，其中包含一个名为`HACS_segments_v1.1.1.json`的文件。

我们在`$MMACTION2/data/HACS/`目录下生成`hacs_anno_train.json`、`hacs_anno_val.json`和`hacs_anno_test.json`文件：

```
```

完成这两个步骤后，HACS Segments数据集的文件夹结构应该如下所示：

```
mmaction2
├── data
│   ├── HACS
├── tools

```

# Dataset Preparer (Beta)

```{note}
Dataset Preparer is still in beta version and might not be stable enough. You are welcome to try it out and report any issues to us.
```

## One-click data preparation script

MMOCR provides a unified one-stop data preparation script `prepare_dataset.py`.

Only one line of command is needed to complete the data download, decompression, format conversion, and basic configure generation.

```bash
```

| ARGS               | Type | Description                                                                                                                               |

```

| ------------------ | ---- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| dataset_name       | str  | (required) dataset name.                                                                                                                  |
| --nproc            | int  | Number of processes to be used. Defaults to 4.                                                                                            |
| --task             | str  | Convert the dataset to the format of a specified task supported by MMOCR. options are: 'textdet', 'textrecog', 'textspotting', and 'kie'. |

| --splits           | str  | Splits of the dataset to be prepared. Multiple splits can be accepted. Defaults to `train val test`.                                      |
| --lmdb             | str  | Store the data in LMDB format. Only valid when the task is `textrecog`.                                                                   |
| --overwrite-cfg    | str  | Whether to overwrite the dataset config file if it already exists in `configs/{task}/_base_/datasets`.                                    |
| --dataset-zoo-path | str  | Path to the dataset config file. If not specified, the default path is `./dataset_zoo`.                                                   |

For example, the following command shows how to use the script to prepare the ICDAR2015 dataset for text detection task.

```bash

```bash
```

Also, the script supports preparing multiple datasets at the same time. For example, the following command shows how to prepare the ICDAR2015 and TotalText datasets for text recognition task.

```bash
```

To check the supported datasets of Dataset Preparer, please refer to [Dataset Zoo](./datasetzoo.md). Some of other datasets that need to be prepared manually are listed in [Text Detection](./det.md) and [Text Recognition](./recog.md).

For users in China, more datasets can be downloaded from the opensource dataset platform: [OpenDataLab](https://opendatalab.com/). After downloading the data, you can place the files listed in `data_obtainer.save_name` in `data/cache` and rerun the script.

## Advanced Usage

### LMDB Format

In text recognition tasks, we usually use LMDB format to store data to speed up data loading. When using the `prepare_dataset.py` script to prepare data, you can store data to the LMDB format by the `--lmdb` parameter. For example:

```bash
```

```bash
```

As soon as the dataset is prepared, Dataset Preparer will generate `icdar2015_lmdb.py` in the `configs/textrecog/_base_/datasets/` directory. You can inherit this file and point the `dataloader` to the LMDB dataset. Moreover, the LMDB dataset needs to be loaded by [`LoadImageFromNDArray`](mmocr.datasets.transforms.LoadImageFromNDArray), thus you also need to modify `pipeline`.

For example, if we want to change the training set of `configs/textrecog/crnn/crnn_mini-vgg_5e_mj.py` to icdar2015 generated before, we need to perform the following modifications:

1. Modify `configs/textrecog/crnn/crnn_mini-vgg_5e_mj.py`:

   ```python
   _base_ = [
        '../_base_/datasets/icdar2015_lmdb.py',  # point to icdar2015 lmdb dataset
         ...
    ]

    ...
   ```

_base_ = [
         ...
    ]

    ...
   ```

2. Modify `train_pipeline` in `configs/textrecog/crnn/_base_crnn_mini-vgg.py`, change `LoadImageFromFile` to `LoadImageFromNDArray`:

   ```python
    dict(
    ...
   ]
   ```

## Design

type='LoadImageFromNDArray',
    ...
   ]
   ```

## Design

There are many OCR datasets with different languages, annotation formats, and scenarios. There are generally two ways to use these datasets: to quickly understand the relevant information about the dataset, or to use it to train models. To meet these two usage scenarios, MMOCR provides dataset automatic preparation scripts. The dataset automatic preparation script uses modular design, which greatly enhances scalability, and allows users to easily configure other public or private datasets. The configuration files for the dataset automatic preparation script are uniformly stored in the `dataset_zoo/` directory. Users can find all the configuration files for the dataset preparation scripts officially supported by MMOCR in this directory. The directory structure of this folder is as follows:

```text

```text
    ├── kie.py
```

### Dataset-related Information

The relevant information of a dataset includes the annotation format, annotation examples, and basic statistical information of the dataset. Although this information can be found on the official website of each dataset, it is scattered across various websites, and users need to spend a lot of time to discover the basic information of the dataset. Therefore, MMOCR has designed some paradigms to help users quickly understand the basic information of the dataset. MMOCR divides the relevant information of the dataset into two parts. One part is the basic information of the dataset, including the year of publication, the authors of the paper, and copyright information, etc. The other part is the annotation information of the dataset, including the annotation format and annotation examples. MMOCR provides a paradigm for each part, and contributors can fill in the basic information of the dataset according to the paradigm. This way, users can quickly understand the basic information of the dataset. Based on the basic information of the dataset, MMOCR provides a `metafile.yml` file, which contains the basic information of the corresponding dataset, including the year of publication, the authors of the paper, and copyright information, etc. In this way, users can quickly understand the basic information of the dataset. This file is not mandatory during the dataset preparation process (so users can ignore it when adding their own private datasets), but to better understand the information of various public datasets, MMOCR recommends that users read the corresponding metafile information before using the dataset preparation script to understand whether the characteristics of the dataset meet the user's needs. MMOCR uses ICDAR2015 as an example, and its sample content is shown below:

```yaml
Paper:
Data:
  Language:
    - English
  Scene:
    - Word
  Tasks:
    - textdet
  License:
```

- Natural Scene
    - Word
  Tasks:
    - textdet
  License:
```

Specifically, MMOCR lists the meaning of each field in the following table:

| Field Name       | Meaning                                                                                                  |
| :--------------- | :------------------------------------------------------------------------------------------------------- |
| Name             | The name of the dataset                                                                                  |
| Paper.Title      | The title of the paper for the dataset                                                                   |
| Paper.URL        | The URL of the paper for the dataset                                                                     |

| Paper.Venue      | The venue of the paper for the dataset                                                                   |
| Paper.Year       | The year of publication for the paper                                                                    |
| Paper.BibTeX     | The BibTeX citation of the paper for the dataset                                                         |
| Data.Website     | The official website of the dataset                                                                      |
| Data.Language    | The supported languages of the dataset                                                                   |
| Data.Scene       | The supported scenes of the dataset, such as `Natural Scene`, `Document`, `Handwritten`, etc.            |

| Data.Granularity | The supported granularities of the dataset, such as `Character`, `Word`, `Line`, etc.                    |
| Data.Tasks       | The supported tasks of the dataset, such as `textdet`, `textrecog`, `textspotting`, `kie`, etc.          |
| Data.License     | License information for the dataset. Use `N/A` if no license exists.                                     |
| Data.Format      | File format of the annotation files, such as `.txt`, `.xml`, `.json`, etc.                               |
| Data.Keywords    | Keywords describing the characteristics of the dataset, such as `Horizontal`, `Vertical`, `Curved`, etc. |

For the annotation information of the dataset, MMOCR provides a `sample_anno.md` file, which users can use as a template to fill in the annotation information of the dataset, so that users can quickly understand the annotation information of the dataset. MMOCR uses ICDAR2015 as an example, and the sample content is as follows:

    ```text

````

`sample_anno.md` provides annotation information for different tasks of the dataset, including the format of the annotation files (text corresponds to `txt` files, and the format of the annotation files can also be found in `meta.yml`), and examples of the annotations.

With the information in these two files, users can quickly understand the basic information of the dataset. Additionally, MMOCR has summarized the basic information of all datasets, and users can view the basic information of all datasets in the [Overview](.overview.md).

### Dataset Usage

After decades of development, the OCR field has seen a series of related datasets emerge, often providing text annotation files in various styles, making it necessary for users to perform format conversion when using these datasets. Therefore, to facilitate dataset preparation for users, we have designed the Dataset Preparer to help users quickly prepare datasets in the format supported by MMOCR. For details, please refer to the [Dataset Format](../../basic_concepts/datasets.md) document. The following figure shows a typical workflow for running the Dataset Preparer.

![workflow](https://user-images.githubusercontent.com/87774050/233025618-aa3c3ad6-c595-49a3-b080-a6284748c0c1.jpg)

The figure shows that when running the Dataset Preparer, the following operations will be performed in sequence:

1. For the training set, validation set, and test set, the preparers will perform:
   1. [Dataset download, extraction, and movement (Obtainer)](#Dataset-download-extraction-and-movement-obtainer)
   2. [Matching annotations with images (Gatherer)](#dataset-collection-gatherer)
   3. [Parsing original annotations (Parser)](#dataset-parsing-parser)
   4. [Packing annotations into a unified format (Packer)](#dataset-conversion-packer)
   5. [Saving annotations (Dumper)](#annotation-saving-dumper)
2. Delete files (Delete)
3. Generate the configuration file for the data set (Config Generator).

5. [Saving annotations (Dumper)](#annotation-saving-dumper)

To handle various types of datasets, MMOCR has designed each component as a plug-and-play module, and allows users to configure the dataset preparation process through configuration files located in `dataset_zoo/`. These configuration files are in Python format and can be used in the same way as other configuration files in MMOCR, as described in the [Configuration File documentation](../config.md).

In `dataset_zoo/`, each dataset has its own folder, and the configuration files are named after the task to distinguish different configurations under different tasks. Taking the text detection part of ICDAR2015 as an example, the sample configuration file `dataset_zoo/icdar2015/textdet.py` is shown below:

```python

```python
        files=[
            dict(
            dict(
        ]),

mapping=[['ic15_textdet_train_gt', 'annotations/train']]),
        ]),
)

        files=[
            dict(
            dict(

dict(
        ]),
)

```

#### Dataset download extraction and movement (Obtainer)

)

```

The `obtainer` module in Dataset Preparer is responsible for downloading, extracting, and moving the dataset. Currently, MMOCR only provides the `NaiveDataObtainer`. Generally speaking, the built-in `NaiveDataObtainer` is sufficient for downloading most datasets that can be accessed through direct links, and supports operations such as extraction, moving files, and renaming. However, MMOCR currently does not support automatically downloading datasets stored in resources that require login, such as Baidu or Google Drive. Here is a brief introduction to the `NaiveDataObtainer`.

| Field Name | Meaning                                                                                      |
| ---------- | -------------------------------------------------------------------------------------------- |

| cache_path | Dataset cache path, used to store the compressed files downloaded during dataset preparation |
| data_root  | Root directory where the dataset is stored                                                   |
| files      | Dataset file list, used to describe the download information of the dataset                  |

The `files` field is a list, and each element in the list is a dictionary used to describe the download information of a dataset file. The table below shows the meaning of each field:

| Field Name         | Meaning                                                                                                                                    |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ |

| url                | Download link for the dataset file                                                                                                         |
| save_name          | Name used to save the dataset file                                                                                                         |
| md5 (optional)     | MD5 hash of the dataset file, used to check if the downloaded file is complete                                                             |
| split (optional)   | Dataset split the file belongs to, such as `train`, `test`, etc., this field can be omitted                                                |
| content (optional) | Content of the dataset file, such as `image`, `annotation`, etc., this field can be omitted                                                |

| mapping (optional) | Decompression mapping of the dataset file, used to specify the storage location of the file after decompression, this field can be omitted |

The Dataset Preparer follows the following conventions:

- Images of different types of datasets are moved to the corresponding category `{taskname}_imgs/{split}/` folder, such as `textdet_imgs/train/`.
- For a annotation file containing annotation information for all images, the annotations are moved to `annotations/{split}.*` file, such as `annotations/train.json`.
- For a annotation file containing annotation information for one image, all annotation files are moved to `annotations/{split}/` folder, such as `annotations/train/`.

- For some other special cases, such as all training, testing, and validation images are in one folder, the images can be moved to a self-set folder, such as `{taskname}_imgs/imgs/`, and the image storage location should be specified in the subsequent `gatherer` module.

An example configuration is as follows:

```python
        files=[
            dict(
            dict(

content=['image'],
            dict(
        ]),
```

#### Dataset collection (Gatherer)

        ]),
```

The `gatherer` module traverses the files in the dataset directory, matches image files with their corresponding annotation files, and organizes a file list for the `parser` module to read. Therefore, it is necessary to know the matching rules between image files and annotation files in the current dataset. There are two commonly used annotation storage formats for OCR datasets: one is multiple annotation files corresponding to multiple images, and the other is a single annotation file corresponding to multiple images, for example:

```text
Many-to-Many

One-to-Many

```

Specific design is as follows:

MMOCR has built-in `PairGatherer` and `MonoGatherer` to handle the two common cases mentioned above. `PairGatherer` is used for many-to-many situations, while `MonoGatherer` is used for one-to-many situations.

```{note}
To simplify processing, the gatherer assumes that the dataset's images and annotations are stored separately in `{taskname}_imgs/{split}/` and `annotations/`, respectively. In particular, for many-to-many situations, the annotation file needs to be placed in `annotations/{split}`.
```

```

- In the many-to-many case, `PairGatherer` needs to find the image files and corresponding annotation files according to a certain naming convention. First, the suffix of the image needs to be specified by the `img_suffixes` parameter, as in the example above `img_suffixes=[.jpg,.JPG]`. In addition, a pair of [regular expressions](https://docs.python.org/3/library/re.html) `rule` is used to specify the correspondence between the image and annotation files. For example, `rule=[r'img_(\d+)\.([jJ][pP][gG])'，r'gt_img_\1.txt']`. The first regular expression is used to match the image file name, `\d+` is used to match the image sequence number, and `([jJ][pP][gG])` is used to match the image suffix. The second regular expression is used to match the annotation file name, where `\1` associates the matched image sequence number with the annotation file sequence number. An example configuration is:

```python

```python
```

For the case of one-to-many, it is usually simple, and the user only needs to specify the annotation file name. For example, for the training set configuration:

```python
```

MMOCR has also made conventions on the return value of `Gatherer`. `Gatherer` returns a tuple with two elements. The first element is a list of image paths (including all image paths) or the folder containing all images. The second element is a list of annotation file paths (including all annotation file paths) or the path of the annotation file (the annotation file contains all image annotation information). Specifically, the return value of `PairGatherer` is (list of image paths, list of annotation file paths), as shown below:

```python

```python
```

`MonoGatherer` returns a tuple containing the path to the image directory and the path to the annotation file, as follows:

```python
```

#### Dataset parsing (Parser)

`Parser` is mainly used to parse the original annotation files. Since the original annotation formats vary greatly, MMOCR provides `BaseParser` as a base class, which users can inherit to implement their own `Parser`. In `BaseParser`, MMOCR has designed two interfaces: `parse_files` and `parse_file`, where the annotation parsing is conventionally carried out. For the two different input situations of `Gatherer` (many-to-many, one-to-many), the implementations of these two interfaces should be different.

- `BaseParser` by default handles the many-to-many situation. Among them, `parse_files` distributes the data in parallel to multiple `parse_file` processes, and each `parse_file` parses the annotation of a single image separately.
- For the one-to-many situation, the user needs to override `parse_files` to implement loading the annotation and returning standardized results.

The interface of `BaseParser` is defined as follows:

```python
    	return self.parse_files(img_paths, ann_paths)

```

return samples

```

In order to ensure the uniformity of subsequent modules, MMOCR has made conventions for the return values of `parse_files` and `parse_file`. The return value of `parse_file` is a tuple, the first element of which is the image path, and the second element is the annotation information. The annotation information is a list, each element of which is a dictionary with the fields `poly`, `text`, and `ignore`, as shown below:

```python
(
    [
        dict(
        ...
    ]
)
```

The output of `parse_files` is a list, and each element in the list is the return value of `parse_file`. An example is:

```python
[
    (
        [
            dict(

```python
[
    (
        [
            dict(
            ...
        ]
    ),
    ...
]
```

#### Dataset Conversion (Packer)

`Packer` is mainly used to convert data into a unified annotation format, because the input data is the output of parsers and the format has been fixed. Therefore, the packer only needs to convert the input format into a unified annotation format for each task. Currently, MMOCR supports tasks such as text detection, text recognition, end-to-end OCR, and key information extraction, and MMOCR has a corresponding packer for each task, as shown below:

For text detection, end-to-end OCR, and key information extraction, MMOCR has a unique corresponding `Packer`. However, for text recognition, MMOCR provides two `Packer` options: `TextRecogPacker` and `TextRecogCropPacker`, due to the existence of two types of datasets:

- Each image is a recognition sample, and the annotation information returned by the `parser` is only a `dict(text='xxx')`. In this case, `TextRecogPacker` can be used.
- The dataset does not crop text from the image, and it essentially contains end-to-end OCR annotations that include the position information of the text and the corresponding text information. `TextRecogCropPacker` will crop the text from the image and then convert it into the unified format for text recognition.

#### Annotation Saving (Dumper)

The `dumper` module is used to determine what format the data should be saved in. Currently, MMOCR supports `JsonDumper`, `WildreceiptOpensetDumper`, and `TextRecogLMDBDumper`. They are used to save data in the standard MMOCR JSON format, the Wildreceipt format, and the LMDB format commonly used in the academic community for text recognition, respectively.

#### Delete files (Delete)

When processing a dataset, temporary files that are not needed may be generated. Here, a list of such files or folders can be passed in, which will be deleted when the conversion is finished.

#### Generate the configuration file for the dataset (ConfigGenerator)

In order to automatically generate basic configuration files after preparing the dataset, MMOCR has implemented `TextDetConfigGenerator`, `TextRecogConfigGenerator`, and `TextSpottingConfigGenerator` for each task. The main parameters supported by these generators are as follows:

| Field Name  | Meaning                                                                                                                                                                                       |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| data_root   | Root directory where the dataset is stored.                                                                                                                                                   |
| train_anns  | Path to the training set annotations in the configuration file. If not specified, it defaults to `[dict(ann_file='{taskname}_train.json', dataset_postfix='']`.                               |

| val_anns    | Path to the validation set annotations in the configuration file. If not specified, it defaults to an empty string.                                                                           |
| test_anns   | Path to the test set annotations in the configuration file. If not specified, it defaults to `[dict(ann_file='{taskname}_test.json', dataset_postfix='']`.                                    |
| config_path | Path to the directory where the configuration files for the algorithm are stored. The configuration generator will write the default configuration to `{config_path}/{taskname}/_base_/datasets/{dataset_name}.py`. If not specified, it defaults to `configs/`. |

After preparing all the files for the dataset, the configuration generator will automatically generate the basic configuration files required to call the dataset. Below is a minimal example of a `TextDetConfigGenerator` configuration:

```python
```

```python
```

The generated file will be placed by default under `configs/{task}/_base_/datasets/`. In this example, the basic configuration file for the ICDAR 2015 dataset will be generated at `configs/textdet/_base_/datasets/icdar2015.py`.

```python

```

```

If the dataset is special and there are several variants of the annotations, the configuration generator also supports generating variables pointing to each variant in the base configuration. However, this requires users to differentiate them by using different `dataset_postfix` when setting up. For example, the ICDAR 2015 text recognition dataset has two annotation versions for the test set, the original version and the 1811 version, which can be specified in `test_anns` as follows:

```python
    ])
```

The configuration generator will generate the following configurations:

```python

dict(dataset_postfix='857', ann_file='textrecog_test_857.json')
    ])
```

```python

```

```

With this file, MMOCR can directly import this dataset into the `dataloader` from the model configuration file (the following sample is excerpted from [`configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py`](/configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py)):

```python
_base_ = [
    # ...
]

# ...

```

```{note}

# ...

```

```{note}
By default, the configuration generator does not overwrite existing base configuration files unless the user manually specifies `overwrite-cfg` when running the script.
```

## Adding a new dataset to Dataset Preparer

### Adding Public Datasets

MMOCR has already supported many [commonly used public datasets](./datasetzoo.md). If the dataset you want to use has not been supported yet and you are willing to [contribute to the MMOCR](../../notes/contribution_guide.md) open-source community, you can follow the steps below to add a new dataset.

In the following example, we will show you how to add the **ICDAR2013** dataset step by step.

#### Adding `metafile.yml`

First, make sure that the dataset you want to add does not already exist in `dataset_zoo/`. Then, create a new folder named after the dataset you want to add, such as `icdar2013/` (usually, use lowercase alphanumeric characters without symbols to name the dataset). In the `icdar2013/` folder, create a `metafile.yml` file and fill in the basic information of the dataset according to the following template:

```yaml
Paper:

Data:
  Language:
    - English
  Scene:
    - Word
  Tasks:
    - textdet
  License:
    Type: N/A
    Link: N/A
  Keywords:
```

#### Add Annotation Examples

- Word
  Tasks:
    - textdet
  License:
    Type: N/A
    Link: N/A
  Keywords:
```

Finally, you can add an annotation example file `sample_anno.md` under the `dataset_zoo/icdar2013/` directory to help the documentation script add annotation examples when generating documentation. The annotation example file is a Markdown file that typically contains the raw data format of a single sample. For example, the following code block shows a sample data file for the ICDAR2013 dataset:

  ```text

````

#### Add configuration files for corresponding tasks

# test split

````

In the `dataset_zoo/icdar2013` directory, add a `.py` configuration file named after the task. For example, `textdet.py`, `textrecog.py`, `textspotting.py`, `kie.py`, etc. The configuration template is shown below:

```python
        files=[
            dict(
                url='xx',
                md5='',
              ]),
    packer=dict(type='TextxxxPacker'), # Packer for the task
)

packer=dict(type='TextxxxPacker'), # Packer for the task
)
        files=[
            dict(
                url='xx',
                md5='',
              ]),
)
```

Taking the file detection task as an example, let's introduce the specific content of the configuration file. In general, users do not need to implement new `obtainer`, `gatherer`, `packer`, or `dumper`, but usually need to implement a new `parser` according to the annotation format of the dataset.

Regarding the configuration of `obtainer`, we will not go into detail here, and you can refer to [Data set download, extraction, and movement (Obtainer)](#Dataset-download-extraction-and-movement-obtainer).

For the `gatherer`, by observing the obtained ICDAR2013 dataset files, we found that each image has a corresponding `.txt` format annotation file:

```text
data_root
│   └── ...
│   └── ...
```

Moreover, the name of each annotation file corresponds to the image: `gt_img_1.txt` corresponds to `img_1.jpg`, and so on. Therefore, `PairGatherer` can be used to match them.

```python
```

```python
```

The first regular expression in the rule is used to match the image file name, and the second regular expression is used to match the annotation file name. Here, `(\w+)` is used to match the image file name, and `gt_\1.txt` is used to match the annotation file name, where `\1` represents the content matched by the first regular expression. That is, it replaces `img_xx.jpg` with `gt_img_xx.txt`.

Next, you need to implement a `parser` to parse the original annotation files into a standard format. Usually, before adding a new dataset, users can browse the [details page](./datasetzoo.md) of the supported datasets and check if there is a dataset with the same format. If there is, you can use the parser of that dataset directly. Otherwise, you need to implement a new format parser.

Data format parsers are stored in the `mmocr/datasets/preparers/parsers` directory. All parsers need to inherit from `BaseParser` and implement the `parse_file` or `parse_files` method. For more information, please refer to [Parsing original annotations (Parser)](#dataset-parsing-parser).

By observing the annotation files of the ICDAR2013 dataset:

```text
```

We found that the built-in `ICDARTxtTextDetAnnParser` already meets the requirements, so we can directly use this parser and configure it in the `preparer`.

```python
```

```python
```

In the configuration for the `ICDARTxtTextDetAnnParser`, `remove_strs=[',', '"']` is specified to remove extra quotes and commas in the annotation files. In the `format` section, `x1 y1 x2 y2 trans` indicates that each line in the annotation file contains four coordinates and a text content separated by spaces (`separator`=' '). Also, `mode` is set to `xyxy`, which means that the coordinates in the annotation file are the coordinates of the top-left and bottom-right corners, so that `ICDARTxtTextDetAnnParser` can parse the annotations into a unified format.

For the `packer`, taking the file detection task as an example, its `packer` is `TextDetPacker`, and its configuration is as follows:

```python
```

```python
```

Finally, specify the `dumper`, which is generally saved in json format. Its configuration is as follows:

```python
```

After the above configuration, the configuration file for the ICDAR2013 training set is as follows:

```python
        files=[
            dict(
            dict(

            dict(
        ]),
)
```

remove_strs=[',', '"'],
)
```

To automatically generate the basic configuration after the dataset is prepared, you also need to configure the corresponding task's `config_generator`.

In this example, since it is a text detection task, you only need to set the generator to `TextDetConfigGenerator`.

```python
```

### Use DataPreparer to prepare customized dataset

\[Coming Soon\]

# RegNet

<img src="https://user-images.githubusercontent.com/26739999/142572813-5dad3317-9d58-4177-971f-d346e01fb3c4.png" width=60%/>
</div>

```python

predict = inference_model('regnetx-400mf_8xb128_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('regnetx-400mf_8xb128_in1k', pretrained=True)
```

Train:

```shell
python tools/train.py configs/regnet/regnetx-400mf_8xb128_in1k.py
```

Test:

```shell

Train:

```shell
```

Test:

```shell
python tools/test.py configs/regnet/regnetx-400mf_8xb128_in1k.py https://download.openmmlab.com/mmclassification/v0/regnet/regnetx-400mf_8xb128_in1k_20211213-89bfc226.pth
```

| Model                       |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                 Config                 |                                        Download                                        |
| :-------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :------------------------------------: | :------------------------------------------------------------------------------------: |

| `regnetx-400mf_8xb128_in1k` | From scratch |    5.16    |   0.41    |   72.56   |   90.78   | [config](regnetx-400mf_8xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/regnet/regnetx-400mf_8xb128_in1k_20211213-89bfc226.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/regnet/regnetx-400mf_8xb128_in1k_20211208_143316.json) |
| `regnetx-800mf_8xb128_in1k` | From scratch |    7.26    |   0.81    |   74.76   |   92.32   | [config](regnetx-800mf_8xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/regnet/regnetx-800mf_8xb128_in1k_20211213-222b0f11.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/regnet/regnetx-800mf_8xb128_in1k_20211207_143037.log.json) |

| `regnetx-1.6gf_8xb128_in1k` | From scratch |    9.19    |   1.63    |   76.84   |   93.31   | [config](regnetx-1.6gf_8xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/regnet/regnetx-1.6gf_8xb128_in1k_20211213-d1b89758.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/regnet/regnetx-1.6gf_8xb128_in1k_20211208_143018.log.json) |
| `regnetx-3.2gf_8xb64_in1k`  | From scratch |    3.21    |   1.53    |   78.09   |   94.08   | [config](regnetx-3.2gf_8xb64_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/regnet/regnetx-3.2gf_8xb64_in1k_20211213-1fdd82ae.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/regnet/regnetx-3.2gf_8xb64_in1k_20211208_142720.log.json) |

| `regnetx-4.0gf_8xb64_in1k`  | From scratch |   22.12    |   4.00    |   78.60   |   94.17   | [config](regnetx-4.0gf_8xb64_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/regnet/regnetx-4.0gf_8xb64_in1k_20211213-efed675c.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/regnet/regnetx-4.0gf_8xb64_in1k_20211207_150431.log.json) |
| `regnetx-6.4gf_8xb64_in1k`  | From scratch |   26.21    |   6.51    |   79.38   |   94.65   | [config](regnetx-6.4gf_8xb64_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/regnet/regnetx-6.4gf_8xb64_in1k_20211215-5c6089da.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/regnet/regnetx-6.4gf_8xb64_in1k_20211213_172748.log.json) |

| `regnetx-8.0gf_8xb64_in1k`  | From scratch |   39.57    |   8.03    |   79.12   |   94.51   | [config](regnetx-8.0gf_8xb64_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/regnet/regnetx-8.0gf_8xb64_in1k_20211213-9a9fcc76.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/regnet/regnetx-8.0gf_8xb64_in1k_20211208_103250.log.json) |
| `regnetx-12gf_8xb64_in1k`   | From scratch |   46.11    |   12.15   |   79.67   |   95.03   |  [config](regnetx-12gf_8xb64_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/regnet/regnetx-12gf_8xb64_in1k_20211213-5df8c2f8.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/regnet/regnetx-12gf_8xb64_in1k_20211208_143713.log.json) |

```bibtex

title={Designing Network Design Spaces},
}
```

### caffemodel should be row-major

`caffe2ncnn` tool assumes the caffemodel is row-major (produced by c++ caffe train command).

The kernel 3x3 weights should be stored as
```
a b c
d e f
g h i
```

However, matlab caffe produced col-major caffemodel.

You have to transpose all the kernel weights by yourself or re-training using c++ caffe train command.

Besides, you may interest in https://github.com/conanhujinming/matcaffe2caffe

### check input is RGB or BGR

If your caffemodel is trained using c++ caffe and opencv, then the input image should be BGR order.

If your model is trained using matlab caffe or pytorch or mxnet or tensorflow, the input image would probably be RGB order.

The channel order can be changed on-the-fly through proper pixel type enum
```
// construct RGB blob from rgb image
ncnn::Mat in_rgb = ncnn::Mat::from_pixels(rgb_data, ncnn::Mat::PIXEL_RGB, w, h);

// construct BGR blob from bgr image

```

ncnn::Mat in_bgr = ncnn::Mat::from_pixels(bgr_data, ncnn::Mat::PIXEL_BGR, w, h);

// construct BGR blob from rgb image
ncnn::Mat in_bgr = ncnn::Mat::from_pixels(rgb_data, ncnn::Mat::PIXEL_RGB2BGR, w, h);

// construct RGB blob from bgr image
ncnn::Mat in_rgb = ncnn::Mat::from_pixels(bgr_data, ncnn::Mat::PIXEL_BGR2RGB, w, h);
```

### image decoding

JPEG(`.jpg`,`.jpeg`) is loss compression, people may get different pixel value for same image on same position. 

`.bmp` images are recommended instead.

### interpolation / resizing

There are several image resizing methods, which may generate different result for same input image.

Even we specify same interpolation method, different frameworks/libraries and their various versions may also introduce difference.

A good practice is feed same size image as the input layer expected, e.g. read a 224x244 bmp image when input layer need 224x224 size.

### Mat::from_pixels/from_pixels_resize assume that the pixel data is continuous

You shall pass continuous pixel buffer to from_pixels family.

If your image is an opencv submat from an image roi, call clone() to get a continuous one.
```
cv::Mat image;// the image
cv::Rect facerect;// the face rectangle

cv::Mat faceimage = image(facerect).clone();// get a continuous sub image

ncnn::Mat in = ncnn::Mat::from_pixels(faceimage.data, ncnn::Mat::PIXEL_BGR, faceimage.cols, faceimage.rows);
```

### pre process
Apply pre process according to your training configuration

Different model has different pre process config, you may find the following transform config in Data layer section
```
transform_param {

```
    mean_value: 103.94
    mean_value: 116.78
    mean_value: 123.68
    scale: 0.017
}
```
Then the corresponding code for ncnn pre process is
```cpp
const float mean_vals[3] = { 103.94f, 116.78f, 123.68f };
const float norm_vals[3] = { 0.017f, 0.017f, 0.017f };
in.substract_mean_normalize(mean_vals, norm_vals);
```

Mean file is not supported currently

So you have to pre process the input data by yourself (use opencv or something)
```
    mean_file: "imagenet_mean.binaryproto"
}
```

For pytorch or mxnet-gluon
```python
transforms.ToTensor(),
transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),
```
```cpp
// R' = (R / 255 - 0.485) / 0.229 = (R - 0.485 * 255) / 0.229 / 255

```
```cpp
// G' = (G / 255 - 0.456) / 0.224 = (G - 0.456 * 255) / 0.224 / 255
// B' = (B / 255 - 0.406) / 0.225 = (B - 0.406 * 255) / 0.225 / 255
const float mean_vals[3] = {0.485f*255.f, 0.456f*255.f, 0.406f*255.f};
const float norm_vals[3] = {1/0.229f/255.f, 1/0.224f/255.f, 1/0.225f/255.f};
```

### use the desired blob
The blob names for input and extract are differ among models.

For example, squeezenet v1.1 use "data" as input blob and "prob" as output blob while mobilenet-ssd use "data" as input blob and "detection_out" as output blob.

Some models may need multiple input or produce multiple output.

```cpp

ex.input("data", in);// change "data" to yours

```cpp

ex.input("mask", mask);// change "mask" to yours

ex.extract("output1", out1);// change "output1" to yours
ex.extract("output2", out2);// change "output2" to yours
```

### blob may have channel gap
Each channel pointer is aligned by 128bit in ncnn Mat structure.

blob may have gaps between channels if (width x height) can not divided exactly by 4

Prefer using ncnn::Mat::from_pixels or ncnn::Mat::from_pixels_resize for constructing input blob from image data

If you do need a continuous blob buffer, reshape the output.
```cpp
// out is the output blob extracted
ncnn::Mat flattened_out = out.reshape(out.w * out.h * out.c);

// plain array, C-H-W
const float* outptr = flattened_out;
```

### create new Extractor for each image

```

The `ncnn::Extractor` object is stateful, if you reuse for different input, you will always get exact the same result cached inside.

Always create new Extractor to process images in loop unless you do know how the stateful Extractor works.
```cpp
for (int i=0; i<count; i++)
{
    // always create Extractor
    // it's cheap and almost instantly !

    // use
    ex.input(your_data[i]);
}
```

### use proper loading api

If you want to load plain param file buffer, you shall use Net::load_param_mem instead of Net::load_param.

```cpp

// param_buffer is the content buffe of XYZ.param file
net.load_param_mem(param_buffer);
```

### disable fp16

```cpp

```

Some models may overflow fp16, resulting in a nan result.

So try to turn off fp16 lower-precision optimizations, and the precision will be improved to fp32 to investigate and solve the overflow problem caused by this.

You can set it as follows
```cpp

```

# Shufflenet V2

> [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://openaccess.thecvf.com/content_ECCV_2018/papers/Ningning_Light-weight_CNN_Architecture_ECCV_2018_paper.pdf)

Currently, the neural network architecture design is mostly guided by the *indirect* metric of computation complexity, i.e., FLOPs. However, the *direct* metric, e.g., speed, also depends on the other factors such as memory access cost and platform characterics. Thus, this work proposes to evaluate the direct metric on the target platform, beyond only considering FLOPs. Based on a series of controlled experiments, this work derives several practical *guidelines* for efficient network design. Accordingly, a new architecture is presented, called *ShuffleNet V2*. Comprehensive ablation experiments verify that our model is the state-of-the-art in terms of speed and accuracy tradeoff.

<img src="https://user-images.githubusercontent.com/26739999/142576336-e0db2866-3add-44e6-a792-14d4f11bd983.png" width="80%"/>
</div>

```python

predict = inference_model('shufflenet-v2-1x_16xb64_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('shufflenet-v2-1x_16xb64_in1k', pretrained=True)
```

Train:

```shell
python tools/train.py configs/shufflenet_v2/shufflenet-v2-1x_16xb64_in1k.py
```

Test:

```shell

Train:

```shell
```

Test:

```shell
python tools/test.py configs/shufflenet_v2/shufflenet-v2-1x_16xb64_in1k.py https://download.openmmlab.com/mmclassification/v0/shufflenet_v2/shufflenet_v2_batch1024_imagenet_20200812-5bf4721e.pth
```

| `shufflenet-v2-1x_16xb64_in1k` | From scratch |    2.28    |   0.15    |   69.55   |   88.92   | [config](shufflenet-v2-1x_16xb64_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/shufflenet_v2/shufflenet_v2_batch1024_imagenet_20200812-5bf4721e.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/shufflenet_v2/shufflenet_v2_batch1024_imagenet_20200812-5bf4721e.json) |

```bibtex
}
```

# 将配置文件从 MMDetection 2.x 迁移至 3.x

MMDetection 3.x 的配置文件与 2.x 相比有较大变化，这篇文档将介绍如何将 2.x 的配置文件迁移到 3.x。

在前面的[配置文件教程](../user_guides/config.md)中，我们以 Mask R-CNN 为例介绍了 MMDetection 3.x 的配置文件结构，这里我们将按同样的结构介绍如何将 2.x 的配置文件迁移至 3.x。

## 模型配置

模型的配置与 2.x 相比并没有太大变化，对于模型的 backbone，neck，head，以及 train_cfg 和 test_cfg，它们的参数与 2.x 版本的参数保持一致。

不同的是，我们在 3.x 版本的模型中新增了 `DataPreprocessor` 模块。
`DataPreprocessor` 模块的配置位于 `model.data_preprocessor` 中，它用于对输入数据进行预处理，例如对输入图像进行归一化，将不同大小的图片进行 padding 从而组成 batch，将图像从内存中读取到显存中等。这部分配置取代了原本存在于 train_pipeline 和 test_pipeline 中的 `Normalize` 和 `Pad`。

<tr>
<td>原配置</td>
<td>

```python
# 图像归一化参数
    ...,
    dict(type='Pad', size_divisor=32),  # 图像 padding 到 32 的倍数
    ...
]
```

</td>
<tr>
<td>新配置</td>
<td>

```python

dict(type='Pad', size_divisor=32),  # 图像 padding 到 32 的倍数
    ...
]
```

</td>
<tr>
<td>

```python
        # 图像归一化参数
        # 图像 padding 参数
        pad_mask=True,  # 在实例分割中，需要将 mask 也进行 padding
        pad_size_divisor=32)  # 图像 padding 到 32 的倍数
)
```

</td>
</tr>
</table>

## 数据集和评测器配置

数据集和评测部分的配置相比 2.x 版本有较大的变化。我们将从 Dataloader 和 Dataset，Data transform pipeline，以及评测器配置三个方面介绍如何将 2.x 版本的配置迁移到 3.x 版本。

### Dataloader 和 Dataset 配置

在新版本中，我们将数据加载的设置与 PyTorch 官方的 DataLoader 保持一致，这样可以使用户更容易理解和上手。
我们将训练、验证和测试的数据加载设置分别放在 `train_dataloader`，`val_dataloader` 和 `test_dataloader` 中，用户可以分别对这些 dataloader 设置不同的参数，其输入参数与 [PyTorch 的 Dataloader](https://pytorch.org/docs/stable/data.html?highlight=dataloader#torch.utils.data.DataLoader) 所需要的参数基本一致。

通过这种方式，我们将 2.x 版本中不可配置的 `sampler`，`batch_sampler`，`persistent_workers` 等参数都放到了配置文件中，使得用户可以更加灵活地设置数据加载的参数。

用户可以通过 `train_dataloader.dataset`，`val_dataloader.dataset` 和 `test_dataloader.dataset` 来设置数据集的配置，它们分别对应 2.x 版本中的 `data.train`，`data.val` 和 `data.test`。

<tr>
<td>

```python
    val=dict(
```

</td>
<tr>
<td>

```python

ann_file=data_root + 'annotations/instances_val2017.json',
```

</td>
<tr>
<td>

```python
    persistent_workers=True,  # 避免每次迭代后 dataloader 重新创建子进程
    sampler=dict(type='DefaultSampler', shuffle=True),  # 默认的 sampler，同时支持分布式训练和非分布式训练
    batch_sampler=dict(type='AspectRatioBatchSampler'),  # 默认的 batch_sampler，用于保证 batch 中的图片具有相似的长宽比，从而可以更好地利用显存
# 在 3.x 版本中可以独立配置验证和测试的 dataloader

test_dataloader = val_dataloader  # 测试 dataloader 的配置与验证 dataloader 的配置相同，这里省略
```

</td>
</tr>
</table>

### Data transform pipeline 配置

上文中提到，我们将图像 normalize 和 padding 的配置从 `train_pipeline` 和 `test_pipeline` 中独立出来，放到了 `model.data_preprocessor` 中，因此在 3.x 版本的 pipeline 中，我们不再需要 `Normalize` 和 `Pad` 这两个 transform。

同时，我们也对负责数据格式打包的 transform 进行了重构，将 `Collect` 和 `DefaultFormatBundle` 这两个 transform 合并为了 `PackDetInputs`，它负责将 data pipeline 中的数据打包成模型的输入格式，关于输入格式的转换，详见[数据流文档](../advanced_guides/data_flow.md)。

下面以 Mask R-CNN 1x 的 train_pipeline 为例，介绍如何将 2.x 版本的配置迁移到 3.x 版本：

<tr>

<tr>
<td>

```python
]
```

</td>
<tr>
<td>

```python
]
```

</td>
</tr>
</table>

dict(type='Resize', scale=(1333, 800), keep_ratio=True),
]
```

</td>
</tr>
</table>

对于 test_pipeline，除了将 `Normalize` 和 `Pad` 这两个 transform 去掉之外，我们也将测试时的数据增强（TTA）与普通的测试流程分开，移除了 `MultiScaleFlipAug`。关于新版的 TTA 如何使用，详见[TTA 文档](../advanced_guides/tta.md)。

下面同样以 Mask R-CNN 1x 的 test_pipeline 为例，介绍如何将 2.x 版本的配置迁移到 3.x 版本：

<tr>
<td>

```python
    dict(
        ])
]
```

</td>
<tr>
<td>

```python

dict(type='ImageToTensor', keys=['img']),
        ])
]
```

</td>
<tr>
<td>

```python
    dict(
]
```

</td>
</tr>
</table>

除此之外，我们还对一些数据增强进行了重构，下表列出了 2.x 版本中的 transform 与 3.x 版本中的 transform 的对应关系：

<thead>
  <tr>
    <th>原配置</th>
    <th>新配置</th>
  </tr>
</thead>
<tbody>
  <tr>
<td>

```python
```

</td>

<td>

```python
```

</td>
  </tr>
  <tr>
<td>

```python
dict(

scale=(1333, 800),
```

</td>
  </tr>
  <tr>
<td>

```python
dict(
```

</td>
<td>

```python
dict(
    scale=[
```

</td>
  </tr>
  <tr>
<td>

```python
dict(
```

</td>
<td>

```python
dict(
    scales=[
```

</td>
  </tr>

<tr>
<td>

```python
dict(type='RandomFlip',
     flip_ratio=0.5)
```

</td>
<td>

```python

keep_ratio=True)
```

</td>
  </tr>

<tr>
<td>

```python
```

</td>
<td>

```python
     prob=0.5)
```

</td>
  </tr>

</tbody>
</table>

### 评测器配置

在 3.x 版本中，模型精度评测不再与数据集绑定，而是通过评测器（Evaluator）来完成。
评测器配置分为 val_evaluator 和 test_evaluator 两部分，其中 val_evaluator 用于验证集评测，test_evaluator 用于测试集评测，对应 2.x 版本中的 evaluation 字段。
下表列出了 2.x 版本与 3.x 版本中的评测器的对应关系：

<thead>
  <tr>
    <th>评测指标名称</th>
  </tr>
</thead>
<tbody>
  <tr>
<td>

```python
    val=dict(
```

</td>

<td>

```python
```

</td>
  </tr>
  <tr>

```python
```

</td>
  </tr>
  <tr>
<td>

```python
    val=dict(
```

</td>
<td>

```python
```

</td>
  </tr>
  <tr>
<td>

```python
    val=dict(

hierarchy_file=data_root +
```

</td>
<td>

```python
```

</td>
  </tr>

<tr>
<td>

```python
    val=dict(
```

</td>
<td>

```python
    dict(

pipeline=test_pipeline))
```

</td>
<td>

```python
    dict(
    dict(
]
```

</td>
  </tr>

</tbody>
</table>

## 训练和测试的配置

<tr>
<td>

```python
    type='EpochBasedRunner',  # 训练循环的类型
    max_epochs=12)  # 最大训练轮次
evaluation = dict(interval=2)  # 验证间隔。每 2 个 epoch 验证一次
```

</td>
<tr>
<td>

```python

<td>

```python
    val_interval=2)  # 验证间隔。每 2 个 epoch 验证一次
```

</td>
</tr>
</table>

## 优化相关配置

优化器以及梯度裁剪的配置都移至 optim_wrapper 字段中。下表列出了 2.x 版本与 3.x 版本中的优化器配置的对应关系：

<tr>
<td>

```python
    type='SGD',  # 随机梯度下降优化器
    lr=0.02,  # 基础学习率
    momentum=0.9,  # 带动量的随机梯度下降
    weight_decay=0.0001)  # 权重衰减
optimizer_config = dict(grad_clip=None)  # 梯度裁剪的配置，设置为 None 关闭梯度裁剪
```

</td>
<tr>
<td>

```python

optimizer=dict(  # 优化器配置。支持 PyTorch 的各种优化器。请参考 https://pytorch.org/docs/stable/optim.html#algorithms
    )
```

</td>
</tr>
</table>

学习率的配置也从 lr_config 字段中移至 param_scheduler 字段中。param_scheduler 的配置更贴近 PyTorch 的学习率调整策略，更加灵活。下表列出了 2.x 版本与 3.x 版本中的学习率配置的对应关系：

<tr>
<td>

```python
    policy='step',  # 在训练过程中使用 multi step 学习率策略
    warmup='linear',  # 使用线性学习率预热
    warmup_iters=500,  # 到第 500 个 iteration 结束预热
    warmup_ratio=0.001,  # 学习率预热的系数
    step=[8, 11],  # 在哪几个 epoch 进行学习率衰减
    gamma=0.1)  # 学习率衰减系数
```

</td>
<tr>
<td>

```python
    dict(

gamma=0.1)  # 学习率衰减系数
```

</td>
<tr>
<td>

```python
    dict(
    dict(
]
```

</td>
</tr>
</table>

关于其他的学习率调整策略的迁移，请参考 MMEngine 的[学习率迁移文档](https://mmengine.readthedocs.io/zh_CN/latest/migration/param_scheduler.html)。

## 其他配置的迁移

### 保存 checkpoint 的配置

<thead>
  <tr>
    <th>功能</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>设置保存间隔</td>
<td>

```python
```

</td>

<td>

```python

<th>原配置</th>
  </tr>
</thead>
<tbody>
  <tr>
<td>

```python
```

</td>

<td>

```python
```

</td>
  </tr>

<tr>
    <td>保存最佳模型</td>
<td>

```python
```

</td>
<td>

```python
```

</td>
  </tr>

<tr>
    <td>只保留最新的几个模型</td>
<td>

```python
```

</td>
<td>

```python
```

</td>
  </tr>

</tbody>
</table>

### 日志的配置

3.x 版本中，日志的打印和可视化由 MMEngine 中的 logger 和 visualizer 分别完成。下表列出了 2.x 版本与 3.x 版本中的日志配置的对应关系：

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
    <td>设置日志打印间隔</td>
<td>

```python
    interval=50)
```

</td>

<td>

```python
    logger=dict(
        type='LoggerHook',
        interval=50))
# 可选： 配置日志打印数值的平滑窗口大小
    window_size=50)
```

</td>
  </tr>

<tr>
    <td>使用 TensorBoard 或 WandB 可视化日志</td>
<td>

```python
    hooks=[
             },
    ])
```

</td>
<td>

```python

interval=50,
    ])
```

</td>
<td>

```python
         })
]
```

</td>
  </tr>

</tbody>
</table>

关于可视化相关的教程，请参考 MMDetection 的[可视化教程](../user_guides/visualization.md)。

### Runtime 的配置

3.x 版本中 runtime 的配置字段有所调整，具体的对应关系如下：

<thead>
  <tr>
  </tr>
</thead>
<tbody>

<tr>
<td>

```python

```

</td>
<td>

```python

```python

```

</td>
<td>

```python
```

</td>
  </tr>

</tbody>
</table>

> **Task**: Text2Image, Image2Image

Disco Diffusion (DD) is a Google Colab Notebook which leverages an AI Image generating technique called CLIP-Guided Diffusion to allow you to create compelling and beautiful images from text inputs.

Created by Somnai, augmented by Gandamu, and building on the work of RiversHaveWings, nshepperd, and many others. See more details in [Credits](#credits).

<thead>
  <tr>
    <td>

    <td>

    <td>

<td>

    <td>

  </tr>
</thead>
</table>

We have converted several `unet` weights and offer related configs. See more details of different `unet` in [Tutorial](#tutorials).

|                                             Model                                              | Dataset  |                                              Download                                              |
| :--------------------------------------------------------------------------------------------: | :------: | :------------------------------------------------------------------------------------------------: |

|       [256x256_diffusion_uncond](./disco-diffusion_adm-u-finetuned_imagenet-256x256.py)        | ImageNet |                                            [model](<>)                                             |
|            [portrait_generator_v001](./disco-diffusion_portrait-generator-v001.py)             | unknown  | [model](https://download.openmmlab.com/mmediting/synthesizers/disco/adm-u-cvt-rgb_portrait-v001-f4a3f3bc.pth) |

|            Model             |   Download   |

## To-do List

- [x] Text2Image
- [x] Image2Image
- [x] Imagenet, portrait diffusion models
- [ ] pixelart, watercolor, sci-fiction diffusion models
- [ ] image prompt
- [ ] video generation
- [ ] faster sampler(plms, dpm-solver etc.)

We really welcome community users supporting these items and any other interesting stuffs!

Running the following codes, you can get a text-generated image.

```python

```python

    0: [
    ]
}

```

height=768,

```

Considering that `disco-diffusion` contains many adjustable parameters, we provide users with a [jupyter-notebook](./tutorials.ipynb) / [colab](https://githubtocolab.com/open-mmlab/mmagic/blob/main/configs/disco_diffusion/tutorials.ipynb) tutorial that exhibits the meaning of different parameters, and gives results corresponding to adjustment.
Refer to [Disco Sheet](https://docs.google.com/document/d/1l8s7uS2dGqjztYSjPpzlmXLjl5PM3IGkRWI3IiCuK7g/edit).

## Credits

<details>
<summary>Credits</summary>

<details>

# PSC

> [Phase-Shifting Coder: Predicting Accurate Orientation in Oriented Object Detection](https://arxiv.org/abs/2211.06368)

With the vigorous development of computer vision, oriented object detection has gradually been featured. In this paper, a novel differentiable angle coder named phase-shifting coder (PSC) is proposed to accurately predict the orientation of objects, along with a dual-frequency version PSCD. By mapping rotational periodicity of different cycles into phase of different frequencies, we provide a unified framework for various periodic fuzzy problems in oriented object detection. Upon such framework, common problems in oriented object detection such as boundary discontinuity and square-like problems are elegantly solved in a unified form. Visual analysis and experiments on three datasets prove the effectiveness and the potentiality of our approach. When facing scenarios requiring high-quality bounding boxes, the proposed methods are expected to give a competitive performance.

DOTA1.0

DOTA1.0

|         Backbone         | AP50  | AP75  | Angle | Dual-freqency | lr schd | Mem (GB) | Inf Time (fps) | Aug | Batch Size |                                                              Configs                                                              |                                                                                                                                                                                          Download                                                                                                                                                                                          |

| ResNet50 (1024,1024,200) | 71.14 | 40.28 | le90  |     True      |   1x    |   4.29   |      21.9      |  -  |     2      |              [rotated-fcos-hbox-le90_r50_fpn_psc-dual_1x_dota](./rotated-fcos-hbox-le90_r50_fpn_psc-dual_1x_dota.py)              |                   [model](https://download.openmmlab.com/mmrotate/v1.0/psc/rotated-fcos-hbox-le90_r50_fpn_psc-dual_1x_dota/rotated-fcos-hbox-le90_r50_fpn_psc-dual_1x_dota-326e276b.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/psc/rotated-fcos-hbox-le90_r50_fpn_psc-dual_1x_dota/rotated-fcos-hbox-le90_r50_fpn_psc-dual_1x_dota-20221115_233457.json)                   |

| ResNet50 (1024,1024,200) | 71.92 | 45.84 | le90  |     True      |   1x    |   2.26   |      20.1      |  -  |     2      | [rotated-retinanet-rbox-le90_r50_fpn_psc-dual_amp-1x_dota (note1)](./rotated-retinanet-rbox-le90_r50_fpn_psc-dual_amp-1x_dota.py) | [model](https://download.openmmlab.com/mmrotate/v1.0/psc/rotated-retinanet-rbox-le90_r50_fpn_psc-dual_amp-1x_dota/rotated-retinanet-rbox-le90_r50_fpn_psc-dual_amp-1x_dota-951713be.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/psc/rotated-retinanet-rbox-le90_r50_fpn_psc-dual_amp-1x_dota/rotated-retinanet-rbox-le90_r50_fpn_psc-dual_amp-1x_dota-20221121_131433.json) |

note1: Trained with --cfg-options load_from=rotated_retinanet_obb_r50_fpn_1x_dota_le90-c0097bc4.pth

HRSC

HRSC

|      Backbone      | AP50  | AP75  | Angle | Dual-freqency | lr schd | Mem (GB) | Inf Time (fps) | Aug | Batch Size |                                                    Configs                                                    |                                                                                                                                                                              Download                                                                                                                                                                              |

| ResNet50 (800,512) | 90.10 | 79.39 | le90  |       -       |   6x    |   1.46   |      37.0      | RR  |     2      |      [rotated-fcos-hbox-le90_r50_fpn_psc_rr-6x_hrsc](./rotated-fcos-hbox-le90_r50_fpn_psc_rr-6x_hrsc.py)      |           [model](https://download.openmmlab.com/mmrotate/v1.0/psc/rotated-fcos-hbox-le90_r50_fpn_psc_rr-6x_hrsc/rotated-fcos-hbox-le90_r50_fpn_psc_rr-6x_hrsc-3da09c7a.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/psc/rotated-fcos-hbox-le90_r50_fpn_psc_rr-6x_hrsc/rotated-fcos-hbox-le90_r50_fpn_psc_rr-6x_hrsc-20221114_193627.json)           |

| ResNet50 (800,512) | 84.86 | 60.07 | le90  |       -       |   6x    |   1.44   |      35.7      | RR  |     2      | [rotated-retinanet-rbox-le90_r50_fpn_psc_rr-6x_hrsc](./rotated-retinanet-rbox-le90_r50_fpn_psc_rr-6x_hrsc.py) | [model](https://download.openmmlab.com/mmrotate/v1.0/psc/rotated-retinanet-rbox-le90_r50_fpn_psc_rr-6x_hrsc/rotated-retinanet-rbox-le90_r50_fpn_psc_rr-6x_hrsc-d2e78a2d.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/psc/rotated-retinanet-rbox-le90_r50_fpn_psc_rr-6x_hrsc/rotated-retinanet-rbox-le90_r50_fpn_psc_rr-6x_hrsc-20221119_190110.json) |

```
@misc{Yu2022PSC,
doi = {10.48550/ARXIV.2211.06368},
url = {https://arxiv.org/abs/2211.06368},
author = {Yu, Yi and Da, Feipeng},
title = {Phase-Shifting Coder: Predicting Accurate Orientation in Oriented Object Detection},
publisher = {arXiv},
copyright = {arXiv.org perpetual, non-exclusive license}
}
```

**New Features & Improvements**

- Support tomesd for StableDiffusion speed-up. [#1801](https://github.com/open-mmlab/mmagic/pull/1801)
- Support all inpainting/matting/image restoration models inferencer. [#1833](https://github.com/open-mmlab/mmagic/pull/1833), [#1873](https://github.com/open-mmlab/mmagic/pull/1873)
- Support animated drawings at projects. [#1837](https://github.com/open-mmlab/mmagic/pull/1837)
- Support Style-Based Global Appearance Flow for Virtual Try-On at projects. [#1786](https://github.com/open-mmlab/mmagic/pull/1786)
- Support tokenizer wrapper and support EmbeddingLayerWithFixe. [#1846](https://github.com/open-mmlab/mmagic/pull/1846)

- Fix install requirements. [#1819](https://github.com/open-mmlab/mmagic/pull/1819)
- Fix inst-colorization PackInputs. [#1828](https://github.com/open-mmlab/mmagic/pull/1828), [#1827](https://github.com/open-mmlab/mmagic/pull/1827)

- Fix inferencer in pip-install. [#1875](https://github.com/open-mmlab/mmagic/pull/1875)

**New Contributors**

- @XDUWQ made their first contribution in https://github.com/open-mmlab/mmagic/pull/1830
- @FerryHuang made their first contribution in https://github.com/open-mmlab/mmagic/pull/1786
- @bobo0810 made their first contribution in https://github.com/open-mmlab/mmagic/pull/1851
- @jercylew made their first contribution in https://github.com/open-mmlab/mmagic/pull/1874

Since its inception, MMEditing has been the preferred algorithm library for many super-resolution, editing, and generation tasks, helping research teams win more than 10 top international competitions and supporting over 100 GitHub ecosystem projects. After iterative updates with OpenMMLab 2.0 framework and merged with MMGeneration, MMEditing has become a powerful tool that supports low-level algorithms based on both GAN and CNN.

Today, MMEditing embraces Generative AI and transforms into a more advanced and comprehensive AIGC toolkit: **MMagic** (**M**ultimodal **A**dvanced, **G**enerative, and **I**ntelligent **C**reation).

In MMagic, we have supported 53+ models in multiple tasks such as fine-tuning for stable diffusion, text-to-image, image and video restoration, super-resolution, editing and generation. With excellent training and experiment management support from [MMEngine](https://github.com/open-mmlab/mmengine), MMagic will provide more agile and flexible experimental support for researchers and AIGC enthusiasts, and help you on your AIGC exploration journey. With MMagic, experience more magic in generation! Let's open a new era beyond editing together. More than Editing, Unlock the Magic!

  - GLIDE

  - GLIDE
  - EG3D
  - NAFNet
  - SwinIR

  MMagic supports the generation of long videos in various styles through ControlNet and MultiFrame Render.

  SAM (Segment Anything Model) is a popular model these days and can also provide more support for MMagic! If you want to create your own animation, you can go to [OpenMMLab PlayGround](https://github.com/open-mmlab/playground/blob/main/mmediting_sam/README.md).

To improve your "spellcasting" efficiency, we have made the following adjustments to the "magic circuit":

- By using MMEngine and MMCV of OpenMMLab 2.0 framework, We decompose the editing framework into different modules and one can easily construct a customized editor framework by combining different modules. We can define the training process just like playing with Legos and provide rich components and strategies. In MMagic, you can complete controls on the training process with different levels of APIs.

- Support 53+ algorithms, 232+ configs, 213+ checkpoints, 26+ loss functions, and 20+ metrics.
- Support controlnet animation and Gradio gui. [Click to view.](https://github.com/open-mmlab/mmagic/tree/main/configs/controlnet_animation)
- Support Inferencer and Demo using High-level Inference APIs. [Click to view.](https://github.com/open-mmlab/mmagic/tree/main/demo)

- Support Gradio gui of Inpainting inference. [Click to view.](https://github.com/open-mmlab/mmagic/blob/main/demo/gradio-demo.py)
- Support qualitative comparison tools. [Click to view.](https://github.com/open-mmlab/mmagic/tree/main/tools/gui)
- Enable projects. [Click to view.](https://github.com/open-mmlab/mmagic/tree/main/projects)
- Improve converters scripts and documents for datasets. [Click to view.](https://github.com/open-mmlab/mmagic/tree/main/tools/dataset_converters)

We are excited to announce the release of MMEditing 1.0.0rc7. This release supports 51+ models, 226+ configs and 212+ checkpoints in MMGeneration and MMEditing. We highlight the following new features

- Support DiffuserWrapper
- Support ControlNet (training and inference).
- Support PyTorch 2.0.

- Support DiffuserWrapper. [#1692](https://github.com/open-mmlab/mmagic/pull/1692)
- Support ControlNet (training and inference). [#1744](https://github.com/open-mmlab/mmagic/pull/1744)
- Support PyTorch 2.0 (successfully compile 33+ models on 'inductor' backend). [#1742](https://github.com/open-mmlab/mmagic/pull/1742)
- Support Image Super-Resolution and Video Super-Resolution models inferencer. [#1662](https://github.com/open-mmlab/mmagic/pull/1662), [#1720](https://github.com/open-mmlab/mmagic/pull/1720)
- Refactor tools/get_flops script. [#1675](https://github.com/open-mmlab/mmagic/pull/1675)
- Refactor dataset_converters and documents for datasets. [#1690](https://github.com/open-mmlab/mmagic/pull/1690)
- Move stylegan ops to MMCV. [#1383](https://github.com/open-mmlab/mmagic/pull/1383)

- Fix disco inferencer. [#1673](https://github.com/open-mmlab/mmagic/pull/1673)
- Fix nafnet optimizer config. [#1716](https://github.com/open-mmlab/mmagic/pull/1716)
- Fix tof typo. [#1711](https://github.com/open-mmlab/mmagic/pull/1711)

Thanks @LeoXing1996, @Z-Fran, @plyfager, @zengyh1900, @liuwenran, @ryanxingql, @HAOCHENYE, @VongolaWu

- @HAOCHENYE made their first contribution in https://github.com/open-mmlab/mmagic/pull/1712

We are excited to announce the release of MMEditing 1.0.0rc6. This release supports 50+ models, 222+ configs and 209+ checkpoints in MMGeneration and MMEditing. We highlight the following new features

- Support Gradio gui of Inpainting inference.
- Support Colorization, Translationin and GAN models inferencer.

- Refactor FileIO. [#1572](https://github.com/open-mmlab/mmagic/pull/1572)
- Refactor registry. [#1621](https://github.com/open-mmlab/mmagic/pull/1621)
- Refactor Random degradations. [#1583](https://github.com/open-mmlab/mmagic/pull/1583)
- Refactor DataSample, DataPreprocessor, Metric and Loop. [#1656](https://github.com/open-mmlab/mmagic/pull/1656)
- Use mmengine.basemodule instead of nn.module. [#1491](https://github.com/open-mmlab/mmagic/pull/1491)
- Refactor Main Page. [#1609](https://github.com/open-mmlab/mmagic/pull/1609)
- Support Gradio gui of Inpainting inference. [#1601](https://github.com/open-mmlab/mmagic/pull/1601)
- Support Colorization inferencer. [#1588](https://github.com/open-mmlab/mmagic/pull/1588)
- Support Translation models inferencer. [#1650](https://github.com/open-mmlab/mmagic/pull/1650)

- Support GAN models inferencer. [#1653](https://github.com/open-mmlab/mmagic/pull/1653), [#1659](https://github.com/open-mmlab/mmagic/pull/1659)
- Print config tool. [#1590](https://github.com/open-mmlab/mmagic/pull/1590)
- Improve type hints. [#1604](https://github.com/open-mmlab/mmagic/pull/1604)
- Update Chinese documents of metrics and datasets. [#1568](https://github.com/open-mmlab/mmagic/pull/1568), [#1638](https://github.com/open-mmlab/mmagic/pull/1638)
- Update Chinese documents of BigGAN and Disco-Diffusion. [#1620](https://github.com/open-mmlab/mmagic/pull/1620)
- Update Evaluation and README of Guided-Diffusion. [#1547](https://github.com/open-mmlab/mmagic/pull/1547)

- Fix the meaning of `momentum` in EMA. [#1581](https://github.com/open-mmlab/mmagic/pull/1581)

- Fix output dtype of RandomNoise. [#1585](https://github.com/open-mmlab/mmagic/pull/1585)
- Fix pytorch2onnx tool. [#1629](https://github.com/open-mmlab/mmagic/pull/1629)
- Fix API documents. [#1641](https://github.com/open-mmlab/mmagic/pull/1641), [#1642](https://github.com/open-mmlab/mmagic/pull/1642)
- Fix loading RealESRGAN EMA weights. [#1647](https://github.com/open-mmlab/mmagic/pull/1647)
- Fix arg passing bug of dataset_converters scripts. [#1648](https://github.com/open-mmlab/mmagic/pull/1648)

A total of 17 developers contributed to this release.
Thanks @plyfager, @LeoXing1996, @Z-Fran, @zengyh1900, @VongolaWu, @liuwenran, @austinmw, @dienachtderwelt, @liangzelong, @i-aki-y, @xiaomile, @Li-Qingyun, @vansin, @Luo-Yihang, @ydengbi, @ruoningYu, @triple-Mu

- @dienachtderwelt made their first contribution in https://github.com/open-mmlab/mmagic/pull/1578
- @i-aki-y made their first contribution in https://github.com/open-mmlab/mmagic/pull/1590
- @triple-Mu made their first contribution in https://github.com/open-mmlab/mmagic/pull/1618
- @Li-Qingyun made their first contribution in https://github.com/open-mmlab/mmagic/pull/1640
- @Luo-Yihang made their first contribution in https://github.com/open-mmlab/mmagic/pull/1648
- @ydengbi made their first contribution in https://github.com/open-mmlab/mmagic/pull/1557

We are excited to announce the release of MMEditing 1.0.0rc5. This release supports 49+ models, 180+ configs and 177+ checkpoints in MMGeneration and MMEditing. We highlight the following new features

- Support Restormer.
- Support GLIDE.
- Support SwinIR.
- Support Stable Diffusion.

- Disco notebook. (#1507)

- Revise test requirements and CI. (#1514)
- Recursive generate summary and docstring. (#1517)
- Enable projects. (#1526)
- Support mscoco dataset. (#1520)
- Improve Chinese documents. (#1532)
- Type hints. (#1481)
- Update download link of checkpoints. (#1554)
- Update deployment guide. (#1551)

- Fix documentation link checker. (#1522)
- Fix ssim first channel bug. (#1515)
- Fix extract_gt_data of realesrgan. (#1542)
- Fix model index. (#1559)
- Fix config path in disco-diffusion. (#1553)

Thanks @plyfager, @LeoXing1996, @Z-Fran, @zengyh1900, @VongolaWu, @liuwenran, @AlexZou14, @lvhan028, @xiaomile, @ldr426, @austin273, @whu-lee, @willaty, @curiosity654, @Zdafeng, @Taited

- @xiaomile made their first contribution in https://github.com/open-mmlab/mmagic/pull/1481
- @ldr426 made their first contribution in https://github.com/open-mmlab/mmagic/pull/1542
- @austin273 made their first contribution in https://github.com/open-mmlab/mmagic/pull/1553
- @whu-lee made their first contribution in https://github.com/open-mmlab/mmagic/pull/1539
- @willaty made their first contribution in https://github.com/open-mmlab/mmagic/pull/1541
- @curiosity654 made their first contribution in https://github.com/open-mmlab/mmagic/pull/1556
- @Zdafeng made their first contribution in https://github.com/open-mmlab/mmagic/pull/1476
- @Taited made their first contribution in https://github.com/open-mmlab/mmagic/pull/1534

We are excited to announce the release of MMEditing 1.0.0rc4. This release supports 45+ models, 176+ configs and 175+ checkpoints in MMGeneration and MMEditing. We highlight the following new features

- Support High-level APIs.
- Support diffusion models.
- Support Text2Image Task.
- Support 3D-Aware Generation.

- Refactor High-level APIs. (#1410)
- Support disco-diffusion text-2-image. (#1234, #1504)
- Support EG3D. (#1482, #1493, #1494, #1499)
- Support NAFNet model. (#1369)

- fix srgan train config. (#1441)
- fix cain config. (#1404)
- fix rdn and srcnn train configs. (#1392)

Thanks @plyfager, @LeoXing1996, @Z-Fran, @zengyh1900, @VongolaWu, @gaoyang07, @ChangjianZhao, @zxczrx123, @jackghosts, @liuwenran, @CCODING04, @RoseZhao929, @shaocongliu, @liangzelong.

- @gaoyang07 made their first contribution in https://github.com/open-mmlab/mmagic/pull/1372
- @ChangjianZhao made their first contribution in https://github.com/open-mmlab/mmagic/pull/1461
- @zxczrx123 made their first contribution in https://github.com/open-mmlab/mmagic/pull/1462
- @jackghosts made their first contribution in https://github.com/open-mmlab/mmagic/pull/1463
- @liuwenran made their first contribution in https://github.com/open-mmlab/mmagic/pull/1410
- @CCODING04 made their first contribution in https://github.com/open-mmlab/mmagic/pull/783

- @RoseZhao929 made their first contribution in https://github.com/open-mmlab/mmagic/pull/1474
- @shaocongliu made their first contribution in https://github.com/open-mmlab/mmagic/pull/1470
- @liangzelong made their first contribution in https://github.com/open-mmlab/mmagic/pull/1488

We are excited to announce the release of MMEditing 1.0.0rc3. This release supports 43+ models, 170+ configs and 169+ checkpoints in MMGeneration and MMEditing. We highlight the following new features

- convert `mmdet` and `clip` to optional requirements.

- Support `try_import` for `mmdet`. (#1408)
- Support `try_import` for `flip`. (#1420)
- Update `.gitignore`. ($1416)
- Set `real_feat` to cpu in `inception_utils`. (#1415)

- Modify README and configs of StyleGAN2 and PEGAN (#1418)
- Improve the rendering of Docs-API (#1373)

- Revise config and pretrain model loading in ESRGAN (#1407)
- Revise config of LSGAN (#1409)
- Revise config of CAIN (#1404)

We are excited to announce the release of MMEditing 1.0.0rc2. This release supports 43+ models, 170+ configs and 169+ checkpoints in MMGeneration and MMEditing. We highlight the following new features

- patch-based and slider-based image and video comparison viewer.
- image colorization.

- Support qualitative comparison tools. (#1303)

- Support instance aware colorization. (#1370)
- Support multi-metrics with different sample-model. (#1171)
- Improve the implementation
  - refactoring evaluation metrics. (#1164)
  - Save gt images in PGGAN's `forward`. (#1332)
  - Improve type and change default number of `preprocess_div2k_dataset.py`. (#1380)
  - Support pixel value clip in visualizer. (#1365)
  - Support SinGAN Dataset and SinGAN demo. (#1363)
  - Avoid cast int and float in GenDataPreprocessor. (#1385)
- Improve the documentation
  - Update a menu switcher. (#1162)
  - Fix TTSR's README. (#1325)

- Fix PPL bug. (#1172)
- Fix RDN number of channels. (#1328)
- Fix types of exceptions in demos. (#1372)
- Fix realesrgan ema. (#1341)
- Improve the assertion to ensuer `GenerateFacialHeatmap` as `np.float32`. (#1310)

- Fix sampling behavior of `unpaired_dataset.py` and  urls in cyclegan's README. (#1308)
- Fix vsr models in pytorch2onnx. (#1300)
- Fix incorrect settings in configs. (#1167,#1200,#1236,#1293,#1302,#1304,#1319,#1331,#1336,#1349,#1352,#1353,#1358,#1364,#1367,#1384,#1386,#1391,#1392,#1393)

Thanks @LeoXing1996, @Z-Fran, @zengyh1900, @plyfager, @ryanxingql, @ruoningYu, @gaoyang07.

MMEditing 1.0.0rc1 has merged MMGeneration 1.x.

- Support 42+ algorithms, 169+ configs and 168+ checkpoints.
- Support 26+ loss functions, 20+ metrics.
- Support tensorboard, wandb.

- Support unconditional GANs, conditional GANs, image2image translation and internal learning.

MMEditing 1.0.0rc0 is the first version of MMEditing 1.x, a part of the OpenMMLab 2.0 projects.

Built upon the new [training engine](https://github.com/open-mmlab/mmengine), MMEditing 1.x unifies the interfaces of dataset, models, evaluation, and visualization.

And there are some BC-breaking changes. Please check [the migration tutorial](https://mmagic.readthedocs.io/en/latest/migration/overview.html) for more details.

<details>
<summary align="right">DeepFillv2 (CVPR'2019)</summary>

```bibtex
}
```

<br/>

|                       算法                        | 掩膜类型  | 分辨率  | 训练集容量 |    测试集     | l1 损失 |  PSNR  | SSIM  | GPU 信息 |                                          下载                                          |
| :-----------------------------------------------: | :-------: | :-----: | :--------: | :-----------: | :-----: | :----: | :---: | :------: | :------------------------------------------------------------------------------------: |

| [DeepFillv2](./deepfillv2_8xb2_places-256x256.py) | free-form | 256x256 |    100k    | Places365-val |  8.635  | 22.398 | 0.815 |    8     | [模型](https://download.openmmlab.com/mmediting/inpainting/deepfillv2/deepfillv2_256x256_8x2_places_20200619-10d15793.pth) \| [日志](https://download.openmmlab.com/mmediting/inpainting/deepfillv2/deepfillv2_256x256_8x2_places_20200619-10d15793.log.json) |

|                       算法                        | 掩膜类型  | 分辨率  | 训练集容量 |   测试集   | l1 损失 |  PSNR  | SSIM  | GPU 信息 |                                           下载                                            |
| :-----------------------------------------------: | :-------: | :-----: | :--------: | :--------: | :-----: | :----: | :---: | :------: | :---------------------------------------------------------------------------------------: |

| [DeepFillv2](./deepfillv2_8xb2_celeba-256x256.py) | free-form | 256x256 |    20k     | CelebA-val |  5.411  | 25.721 | 0.871 |    8     | [模型](https://download.openmmlab.com/mmediting/inpainting/deepfillv2/deepfillv2_256x256_8x2_celeba_20200619-c96e5f12.pth) \| [日志](https://download.openmmlab.com/mmediting/inpainting/deepfillv2/deepfillv2_256x256_8x2_celeba_20200619-c96e5f12.log.json) |

## 快速开始

**训练**

<details>

```shell
# CPU上训练

# 单个GPU上训练

# 多个GPU上训练
```

**测试**

<details>

```shell
# CPU上测试

```

**测试**

<details>

```shell
# CPU上测试

# 单个GPU上测试

# 多个GPU上测试
```

# 如何给 MMPose 贡献代码

欢迎加入 MMPose 社区，我们致力于打造最前沿的计算机视觉基础库，我们欢迎任何形式的贡献，包括但不限于：

- **修复错误**
  1. 如果提交的代码改动较大，我们鼓励你先开一个 issue 并正确描述现象、原因和复现方式，讨论后确认修复方案。
  2. 修复错误并补充相应的单元测试，提交 PR 。
- **新增功能或组件**
  1. 如果新功能或模块涉及较大的代码改动，我们建议先提交 issue，与我们确认功能的必要性。
  2. 实现新增功能并添加单元测试，提交 PR 。
- **文档补充或翻译**
  - 如果发现文档有错误或不完善的地方，欢迎直接提交 PR 。

```{note}
- 如果你希望向 MMPose 1.0 贡献代码，请从 dev-1.x 上创建新分支，并提交 PR 到 dev-1.x 分支上。
- 如果你是论文作者，并希望将你的方法加入到 MMPose 中，欢迎联系我们，我们将非常感谢你的贡献。
- 如果你希望尽快将你的项目分享到 MMPose 开源社区，欢迎将 PR 提到 Projects 目录下，该目录下的项目将简化 Review 流程并尽快合入。
- 如果你希望加入 MMPose 的维护者，欢迎联系我们，我们将邀请你加入 MMPose 的维护者群。
```

## 准备工作

PR 操作所使用的命令都是用 Git 去实现的，该章节将介绍如何进行 Git 配置与 GitHub 绑定。

### Git 配置

首先，你需要在本地安装 Git，然后配置你的 Git 用户名和邮箱：

```Shell
# 在命令提示符（cmd）或终端（terminal）中输入以下命令，查看 Git 版本
git --version
```

然后，你需要检查自己的 Git Config 是否正确配置，如果 `user.name` 和 `user.email` 为空，你需要配置你的 Git 用户名和邮箱：

```Shell
# 在命令提示符（cmd）或终端（terminal）中输入以下命令，查看 Git 配置
git config --global --list
# 设置 Git 用户名和邮箱
git config --global user.name "这里填入你的用户名"

```Shell
git config --global user.email "这里填入你的邮箱"
```

## PR 流程

如果你对 PR 流程不熟悉，接下来将会从零开始，一步一步地教你如何提交 PR。如果你想深入了解 PR 开发模式，可以参考 [GitHub 官方文档](https://docs.github.com/cn/github/collaborating-with-issues-and-pull-requests/about-pull-requests)。

### 1. Fork 项目

当你第一次提交 PR 时，需要先 Fork 项目到自己的 GitHub 账号下。点击项目右上角的 Fork 按钮，将项目 Fork 到自己的 GitHub 账号下。

![](https://user-images.githubusercontent.com/13503330/223318144-a49c6cef-b1fb-45b8-aa2b-0833d0e3fd5c.png)

接着，你需要将你的 Fork 仓库 Clone 到本地，然后添加官方仓库作为远程仓库：

```Shell

# Clone 你的 Fork 仓库到本地
git clone https://github.com/username/mmpose.git

# 添加官方仓库作为远程仓库
cd mmpose
git remote add upstream https://github.com/open-mmlab/mmpose.git
```

在终端中输入以下命令，查看远程仓库是否成功添加：

```Shell
git remote -v
```

如果出现以下信息，说明你已经成功添加了远程仓库：

```Shell
origin	https://github.com/{username}/mmpose.git (fetch)

```

```Shell
```

```Shell
origin	https://github.com/{username}/mmpose.git (push)
upstream	https://github.com/open-mmlab/mmpose.git (fetch)
upstream	https://github.com/open-mmlab/mmpose.git (push)
```

```{note}
```

### 2. 配置 pre-commit

在本地开发环境中，我们使用 pre-commit 来检查代码风格，以确保代码风格的统一。在提交代码前，你需要先安装 pre-commit：

```Shell

# 在 mmpose 根目录下安装 pre-commit
```

```Shell
```

```Shell

```

```Shell
```

![](https://user-images.githubusercontent.com/57566630/202368856-0465a90d-8fce-4345-918e-67b8b9c82614.png)

```{note}
如果你是中国大陆用户，由于网络原因，可能会出现 pre-commit 安装失败的情况。

这时你可以使用清华源来安装 pre-commit：
pip install -U pre-commit -i https://pypi.tuna.tsinghua.edu.cn/simple

或者使用国内镜像来安装 pre-commit：
pip install -U pre-commit -i https://pypi.mirrors.ustc.edu.cn/simple
```

如果安装过程被中断，可以重复执行上述命令，直到安装成功。

如果你提交的代码中有不符合规范的地方，pre-commit 会发出警告，并自动修复部分错误。

![](https://user-images.githubusercontent.com/57566630/202369176-67642454-0025-4023-a095-263529107aa3.png)

### 3. 创建开发分支

安装完 pre-commit 之后，我们需要基于 dev 分支创建一个新的开发分支，建议以 `username/pr_name` 的形式命名，例如：

```Shell
git checkout -b username/refactor_contributing_doc
```

在后续的开发中，如果本地仓库的 dev 分支落后于官方仓库的 dev 分支，需要先拉取 upstream 的 dev 分支，然后 rebase 到本地的开发分支上：

```Shell

```Shell
```

```Shell
git checkout username/refactor_contributing_doc
git fetch upstream
git rebase upstream/dev-1.x
```

在 rebase 时，如果出现冲突，需要手动解决冲突，然后执行 `git add` 命令，再执行 `git rebase --continue` 命令，直到 rebase 完成。

### 4. 提交代码并在本地通过单元测试

在本地开发完成后，我们需要在本地通过单元测试，然后提交代码。

```shell
# 运行单元测试
pytest tests/

# 提交代码
git add .
git commit -m "commit message"
```

### 5. 推送代码到远程仓库

在本地开发完成后，我们需要将代码推送到远程仓库。

```Shell
git push origin username/refactor_contributing_doc
```

### 6. 提交 Pull Request (PR)

#### (1) 在 GitHub 上创建 PR

![](https://user-images.githubusercontent.com/13503330/223321382-e6068e18-1d91-4458-8328-b1c7c907b3b2.png)

#### (2) 在 PR 中根据指引修改描述，添加必要的信息

![](https://user-images.githubusercontent.com/13503330/223322447-94ad4b8c-21bf-4ca7-b3d6-0568cace6eee.png)

```{note}
- 在 PR branch 左侧选择 `dev` 分支，否则 PR 会被拒绝。
- 如果你是第一次向 OpenMMLab 提交 PR，需要签署 CLA。
```

```{note}
```

![](https://user-images.githubusercontent.com/57566630/167307569-a794b967-6e28-4eac-a942-00deb657815f.png)

## 代码风格

### Python

我们采用[PEP8](https://www.python.org/dev/peps/pep-0008/)作为代码风格。

使用下面的工具来对代码进行整理和格式化：

- [flake8](http://flake8.pycqa.org/en/latest/)：代码提示
- [isort](https://github.com/timothycrosley/isort)：import 排序
- [yapf](https://github.com/google/yapf)：格式化工具
- [codespell](https://github.com/codespell-project/codespell)： 单词拼写检查
- [mdformat](https://github.com/executablebooks/mdformat): markdown 文件格式化工具
- [docformatter](https://github.com/myint/docformatter): docstring 格式化工具

`yapf`和`isort`的样式配置可以在[setup.cfg](/setup.cfg)中找到。

我们使用[pre-commit hook](https://pre-commit.com/)来：

- 检查和格式化 `flake8`、`yapf`、`isort`、`trailing whitespaces`
- 修复 `end-of-files`

- 在每次提交时自动排序 `requirments.txt`

`pre-commit`的配置存储在[.pre-commit-config](/.pre-commit-config.yaml)中。

```{note}
在你创建PR之前，请确保你的代码格式符合规范，且经过了 yapf 格式化。
```

### C++与CUDA

遵循[Google C++风格指南](https://google.github.io/styleguide/cppguide.html)

<details>
<summary align="right">IndexNet (ICCV'2019)</summary>

```bibtex
}
```

<br/>

|                          算法                           |   SAD    |    MSE    |   GRAD   |   CONN   | GPU 信息 |                                                   下载                                                   |
| :-----------------------------------------------------: | :------: | :-------: | :------: | :------: | :------: | :------------------------------------------------------------------------------------------------------: |

|                     M2O DINs (原文)                     |   45.8   |   0.013   |   25.9   | **43.7** |    -     |                                                    -                                                     |
| [M2O DINs (复现)](./indexnet_mobv2_1xb16-78k_comp1k.py) | **45.6** | **0.012** | **25.5** |   44.8   |    1     | [模型](https://download.openmmlab.com/mmediting/mattors/indexnet/indexnet_mobv2_1x16_78k_comp1k_SAD-45.6_20200618_173817-26dd258d.pth) \| [日志](https://download.openmmlab.com/mmediting/mattors/indexnet/indexnet_mobv2_1x16_78k_comp1k_20200618_173817.log.json) |

**其他结果**

**其他结果**

|                                   算法                                    | SAD  |  MSE  | GRAD | CONN | GPU 信息 |                                                  下载                                                  |
| :-----------------------------------------------------------------------: | :--: | :---: | :--: | :--: | :------: | :----------------------------------------------------------------------------------------------------: |
| [M2O DINs (使用 DIM 流水线)](./indexnet_mobv2-dimaug_1xb16-78k_comp1k.py) | 50.1 | 0.016 | 30.8 | 49.5 |    1     | [模型](https://download.openmmlab.com/mmediting/mattors/indexnet/indexnet_dimaug_mobv2_1x16_78k_comp1k_SAD-50.1_20200626_231857-af359436.pth) \| [日志](https://download.openmmlab.com/mmediting/mattors/indexnet/indexnet_dimaug_mobv2_1x16_78k_comp1k_20200626_231857.log.json) |

## 快速开始

**训练**

<details>

```shell
# CPU上训练

## 快速开始

**训练**

<details>

```shell
# CPU上训练

# 单个GPU上训练

# 多个GPU上训练
```

**测试**

<details>

```shell
# CPU上测试

# 单个GPU上测试

# 多个GPU上测试

# 多个GPU上测试
```

# 使用 MMDetection 和 Label-Studio 进行半自动化目标检测标注

标注数据是一个费时费力的任务，本文介绍了如何使用 MMDetection 中的 RTMDet 算法联合 Label-Studio 软件进行半自动化标注。具体来说，使用 RTMDet 预测图片生成标注，然后使用 Label-Studio 进行微调标注，社区用户可以参考此流程和方法，将其应用到其他领域。

- RTMDet：RTMDet 是 OpenMMLab 自研的高精度单阶段的目标检测算法，开源于 MMDetection 目标检测工具箱中，其开源协议为 Apache 2.0，工业界的用户可以不受限的免费使用。
- [Label Studio](https://github.com/heartexlabs/label-studio) 是一款优秀的标注软件，覆盖图像分类、目标检测、分割等领域数据集标注的功能。

本文将使用[喵喵数据集](https://download.openmmlab.com/mmyolo/data/cat_dataset.zip)的图片，进行半自动化标注。

## 环境配置

首先需要创建一个虚拟环境，然后安装 PyTorch 和 MMCV。在本文中，我们将指定 PyTorch 和 MMCV 的版本。接下来安装 MMDetection、Label-Studio 和 label-studio-ml-backend，具体步骤如下：

创建虚拟环境：

```shell
```

安装 PyTorch

```shell

```shell
# OSX
```

安装 MMCV

```shell
# 安装 mmcv 的过程中会自动安装 mmengine
```

安装 MMDetection

```shell
```

安装 Label-Studio 和 label-studio-ml-backend

```shell
# 安装 label-studio 需要一段时间,如果找不到版本请使用官方源
```

下载rtmdet权重

```shell

```shell
```

```shell
```

## 启动服务

启动 RTMDet 后端推理服务：

```shell

# device=cpu 为使用 CPU 推理，如果使用 GPU 推理，将 cpu 替换为 cuda:0
```

此时，RTMDet 后端推理服务已经启动，后续在 Label-Studio Web 系统中配置 http://localhost:8003 后端推理服务即可。

现在启动 Label-Studio 网页服务：

```shell
```

```shell
```

打开浏览器访问 [http://localhost:8080/](http://localhost:8080/) 即可看到 Label-Studio 的界面。

我们注册一个用户，然后创建一个 RTMDet-Semiautomatic-Label 项目。

我们通过下面的方式下载好示例的喵喵图片，点击 Data Import 导入需要标注的猫图片。

```shell

```

然后选择 Object Detection With Bounding Boxes 模板

```shell
airplane
apple
backpack
banana
bear
bed
bench
bicycle
bird
boat
book
bottle
bowl
broccoli
bus
cake
car
carrot
cat
chair

```shell
airplane
apple
backpack
banana
bear
bed
bench
bicycle
bird
boat
book
bottle
bowl
broccoli
bus
cake
car
carrot
cat
chair
clock
couch
cow
cup
dog
donut
elephant
fork
frisbee
giraffe
handbag
horse
hot_dog
keyboard
kite
knife
laptop
microwave
mouse
orange
oven
person
pizza
remote
sandwich
scissors
sheep
sink
skis
snowboard
spoon
stop_sign
suitcase
surfboard
tie
toaster
toilet
train
truck
tv
umbrella
vase
zebra
```

然后将上述类别复制添加到 Label-Studio，然后点击 Save。

然后在设置中点击 Add Model 添加 RTMDet 后端推理服务。

点击 Validate and Save，然后点击 Start Labeling。

看到如下 Connected 就说明后端推理服务添加成功。

## 开始半自动化标注

点击 Label 开始标注

我们可以看到 RTMDet 后端推理服务已经成功返回了预测结果并显示在图片上，我们可以发现这个喵喵预测的框有点大。

我们手工拖动框，修正一下框的位置，得到以下修正过后的标注，然后点击 Submit，本张图片就标注完毕了。

我们 submit 完毕所有图片后，点击 exprot 导出 COCO 格式的数据集，就能把标注好的数据集的压缩包导出来了。

用 vscode 打开解压后的文件夹，可以看到标注好的数据集，包含了图片和 json 格式的标注文件。

到此半自动化标注就完成了，我们可以用这个数据集在 MMDetection 训练精度更高的模型了，训练出更好的模型，然后再用这个模型继续半自动化标注新采集的图片，这样就可以不断迭代，扩充高质量数据集，提高模型的精度。

## 使用 MMYOLO 作为后端推理服务

如果想在 MMYOLO 中使用 Label-Studio，可以参考在启动后端推理服务时，将 config_file 和 checkpoint_file 替换为 MMYOLO 的配置文件和权重文件即可。

```shell

```shell

```

旋转目标检测和实例分割还在支持中，敬请期待。

# MMOCR 模型部署

- [MMOCR 模型部署](#mmocr-模型部署)
    - [安装 mmocr](#安装-mmocr)
    - [文字检测任务模型转换](#文字检测任务模型转换)
    - [文字识别任务模型转换](#文字识别任务模型转换)
      - [文字检测 SDK 模型推理](#文字检测-sdk-模型推理)
      - [文字识别 SDK 模型推理](#文字识别-sdk-模型推理)

[MMOCR](https://github.com/open-mmlab/mmocr/tree/main)，又称 `mmocr`，是基于 PyTorch 和 mmdetection 的开源工具箱，专注于文本检测，文本识别以及相应的下游任务，如关键信息提取。 它是 [OpenMMLab](https://openmmlab.com/) 项目的一部分。

## 安装

### 安装 mmocr

请参考[官网安装指南](https://mmocr.readthedocs.io/en/latest/get_started/install.html).

```shell
```

## 模型转换

你可以使用 [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/tree/main/tools/deploy.py) 把 mmocr 模型一键式转换为推理后端模型。

转换的关键之一是使用正确的配置文件。项目中已内置了各后端部署[配置文件](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmocr)。
文件的命名模式是：

```
```

其中：

- **{task}:** mmocr 中的任务

  mmdeploy 支持 mmocr 中的文字检测（text detection）、文字识别（text recognition）任务中的模型。关于`模型-任务`的划分，请参考章节[模型支持列表](#模型支持列表)。

在接下来来的两个章节，我们将分别演示文字检测任务中的`dbnet`模型，和文字识别任务中的`crnn`模型转换 onnx 模型的方法。

### 文字检测任务模型转换

```shell

    --show \
```

### 文字识别任务模型转换

```shell
    --show \

crnn_mini-vgg_5e_mj.py \
    --show \
```

你也可以把它们转为其他后端模型。比如使用`text-detection/text-detection_tensorrt-_dynamic-320x320-2240x2240.py`，把 `dbnet` 模型转为 tensorrt 模型。

```{tip}
```

## 模型规范

[文字检测任务模型转换](#文字检测任务模型转换)例子中的`mmdeploy_models/mmocr/dbnet/ort`，结构如下：

```
```

重要的是：

## 模型推理

### 后端模型推理

```python

## 模型推理

### 后端模型推理

```python

```

**提示：**

在这个脚本中，把'deploy_cfg', 'model_cfg', 'backend_model' and 'image' 替换为[文字识别任务模型转换](#文字识别任务模型转换)中对应的参数，
就可以推理 `crnn` onnx 模型了。

#### 文字检测 SDK 模型推理

你也可以参考如下代码，对 `dbnet` SDK model 进行推理：

```python
import cv2

```

#### 文字识别 SDK 模型推理

```python
import cv2

)
```

## 模型支持列表

## 模型支持列表

## 注意事项

- ABINet 在 TensorRT 后端要求使用 pytorch1.10+， TensorRT 8.4+。

- SAR 在网络推广中使用 `valid_ratio`，这会让导出的 ONNX 文件精度下降。当测试图片的 `valid_ratio`s 和转换图片的值差异很大，这种下降就会越多。

- 对于 TensorRT 后端，用户需要使用正确的配置文件。比如 CRNN 只接受单通道输入。下面是一个示例表格:

### 如何提交代码

#### 一、fork 分支
在浏览器中打开 [ncnn](https://github.com/tencent/ncnn), `fork` 到自己的 repositories，例如
```
https://github.com/user/ncnn
```

clone 项目到本地，添加官方 remote 并 fetch:
```
$ git clone https://github.com/user/ncnn && cd ncnn
$ git remote add tencent https://github.com/tencent/ncnn
$ git fetch tencent
```
对于 `git clone` 下来的项目，它现在有两个 remote，分别是 origin 和 tencent：

```
$ git remote -v
origin   https://github.com/user/ncnn (fetch)
origin   https://github.com/user/ncnn (push)
tencent  https://github.com/Tencent/ncnn (fetch)
tencent  https://github.com/Tencent/ncnn (push)
```
origin 指向你 fork 的仓库地址；remote 即官方 repo。可以基于不同的 remote 创建和提交分支。

例如切换到官方 master 分支，并基于此创建自己的分支（命名尽量言简意赅。一个分支只做一件事，方便 review 和 revert）
```
$ git checkout tencent/master
$ git checkout -b add-conv-int8
```

或创建分支时指定基于官方 master 分支：
```
$ git checkout -b fix-typo-in-document tencent/master
```

```
```

```
```

> `git fetch` 是从远程获取最新代码到本地。如果是第二次 pr ncnn，直接从  `git fetch tencent` 开始即可，不需要 `git remote add tencent`，也不需要修改 `github.com/user/ncnn`。

#### 二、代码习惯
为了增加沟通效率，reviewer 一般要求 contributor 遵从以下规则

* `if-else`和花括号`{`中间需要换行
* 不能随意增删空行
* tab 替换为 4 个空格
* 为了保证平台兼容性，目前不使用`c++11`，`src`目录下尽量避免使用`template`
* 若是新增功能或平台，`test`目录需有对应测试用例
* 文档放到`doc`对应目录下，中文用`.zh.md`做后缀；英文直接用`.md`后缀

开发完成后提交到自己的 repository
```
$ git commit -a
$ git push origin add-conv-int8
```
推荐使用 [`commitizen`](https://pypi.org/project/commitizen/) 或 [`gitlint`](https://jorisroovers.com/gitlint/) 等工具格式化 commit message，方便事后检索海量提交记录

#### 三、代码提交
浏览器中打开 [ncnn pulls](https://github.com/Tencent/ncnn/pulls) ，此时应有此分支 pr 提示，点击 `Compare & pull request`

* 标题**必须**是英文。未完成的分支应以 `WIP:` 开头，例如 `WIP: add conv int8`
* 正文宜包含以下内容，中英不限
    * 内容概述和实现方式
    * 功能或性能测试
    * 测试结果

    * 功能或性能测试
    * 测试结果

CI 已集成了自动格式化，restyled-io 会在 pr 的同时生成 `Restyled add conv int8`，需要 merge 自动 restyled 的分支，例如
```
$ git checkout add-conv-int8
$ git merge tencent/restyled/pull-2078
```
回到浏览器签署  CLA，所有 CI 测试通过后通知 reviewer merge 此分支。

#### 四、彩蛋
留下个人 qq 号会触发隐藏事件。

# 2D Fashion Landmark Detection

2D fashion landmark detection (also referred to as fashion alignment) aims to detect the key-point located at the functional region of clothes, for example the neckline and the cuff.

Please follow [DATA Preparation](/docs/en/dataset_zoo/2d_fashion_landmark.md) to prepare data.

# Quick Run

This chapter will introduce you to the fundamental functionalities of MMAction2. We assume that you have [installed MMAction2 from source](installation.md#best-practices).

- [Quick Run](#quick-run)
  - [Inference](#inference)
  - [Prepare a Dataset](#prepare-a-dataset)
  - [Modify the Config](#modify-the-config)
    - [Modify Dataset](#modify-dataset)
    - [Modify Runtime Config](#modify-runtime-config)
    - [Modify Model Config](#modify-model-config)
  - [Browse the Dataset](#browse-the-dataset)

Run the following command in the root directory of MMAction2:

```shell
python demo/demo_inferencer.py  demo/demo.mp4 \
    --rec tsn --print-result \
    --label-file tools/data/kinetics/label_map_k400.txt
```

You should be able to see a pop-up video and the inference result printed out in the console.

--label-file tools/data/kinetics/label_map_k400.txt
```

    <img src="https://user-images.githubusercontent.com/33249023/227216933-29b84ac7-ca0e-408d-b4d2-5a2e5a7357bf.gif" height="250"/>
</div>
<br />

```bash
{'predictions': [{'rec_labels': [[6]], 'rec_scores': [[...]]}]}
```

```{note}
If you are running MMAction2 on a server without a GUI or via an SSH tunnel with X11 forwarding disabled, you may not see the pop-up window.
```

A detailed description of MMAction2's inference interface can be found [here](https://github.com/open-mmlab/mmaction2/tree/main/demo/README.md#inferencer).

```

In addition to using our well-provided pre-trained models, you can also train models on your own datasets. In the next section, we will take you through the basic functions of MMAction2 by training TSN on the tiny [Kinetics](https://download.openmmlab.com/mmaction/kinetics400_tiny.zip) dataset as an example.

## Prepare a Dataset

Since the variety of video dataset formats are not conducive to switching datasets, MMAction2 proposes a uniform [data format](../user_guides/2_data_prepare.md), and provides [dataset preparer](../user_guides/data_prepare/dataset_preparer.md) for commonly used video datasets. Usually, to use those datasets in MMAction2, you just need to follow the steps to get them ready for use.

```{note}
But here, efficiency means everything.
```

```{note}
```

To get started, please download our pre-prepared [kinetics400_tiny.zip](https://download.openmmlab.com/mmaction/kinetics400_tiny.zip) and extract it to the `data/` directory in the root directory of MMAction2. This will provide you with the necessary videos and annotation file.

```Bash
wget https://download.openmmlab.com/mmaction/kinetics400_tiny.zip
unzip kinetics400_tiny.zip -d data/
```

## Modify the Config

After preparing the dataset, the next step is to modify the config file to specify the location of the training set and training parameters.

In this example, we will train a TSN using resnet50 as its backbone. Since MMAction2 already has a config file for the full Kinetics400 dataset (`configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py`), we just need to make some modifications on top of it.

### Modify Dataset

We first need to modify the path to the dataset. Open `configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py` and replace keys as followed:

```Python
data_root = 'data/kinetics400_tiny/train'
data_root_val = 'data/kinetics400_tiny/val'
ann_file_train = 'data/kinetics400_tiny/kinetics_tiny_train_video.txt'
ann_file_val = 'data/kinetics400_tiny/kinetics_tiny_val_video.txt'
```

### Modify Runtime Config

Additionally, due to the reduced size of the dataset, we recommend decreasing the training batch size to 4 and the number of training epochs to 10 accordingly. Furthermore, we suggest shortening the validation and weight storage intervals to 1 round each, and modifying the learning rate decay strategy. Modify the corresponding keys in  `configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py` as following lines to take effect.

```python
# set training batch size to 4

```python
train_dataloader['batch_size'] = 4

# Save checkpoints every epoch, and only keep the latest checkpoint
    checkpoint=dict(type='CheckpointHook', interval=1, max_keep_ckpts=1))
# Set the maximum number of epochs to 10, and validate the model every 1 epochs
# adjust learning rate schedule according to 10 epochs
    dict(
        begin=0,
        end=10,
        milestones=[4, 8],
]
```

### Modify Model Config

    dict(
        begin=0,
        end=10,
]
```

Further, due to the small size of tiny Kinetics dataset, it is recommended to load a pre-trained model on the original Kinetics dataset. Additionally, the model needs to be modified according to the actual number of classes. Please directly add the following lines to `configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py`.

```python
    cls_head=dict(num_classes=2))
```

```

Here, we have rewritten the corresponding parameters in the base configuration directly through the inheritance ({external+mmengine:doc}`MMEngine: Config <advanced_tutorials/config>`) mechanism of the config. The original fields are distributed in `configs/_base_/models/tsn_r50.py`, `configs/_base_/schedules/sgd_100e.py` and `configs/_base_/default_runtime.py`.

```{note}
For a more detailed description of config, please refer to [here](../user_guides/1_config.md).
```

## Browse the Dataset

Before we start the training, we can also visualize the frames processed by training-time data transforms. It's quite simple: pass the config file we need to visualize into the [browse_dataset.py](https://github.com/open-mmlab/mmaction2/tree/main/tools/analysis_tools/browse_dataset.py) script.

```Bash
    configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py \
    browse_out --mode pipeline
```

```Bash
```

The transformed videos will be saved to `browse_out` folder.

    <img src="https://user-images.githubusercontent.com/33249023/227452030-81895695-8a9b-45be-922a-3d9d86baf65d.gif" height="250"/>
</center>

```{note}
For details on the parameters and usage of this script, please refer to [here](../user_guides/useful_tools.md).
```

```{tip}
In addition to satisfying our curiosity, visualization can also help us check the parts that may affect the model's performance before training, such as problems in configs, datasets and data transforms.
```

we can further visualize the learning rate schedule to make sure that the config is as expected by following script:

```Bash

```

```Bash
python tools/visualizations/vis_scheduler.py configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py
```

The training learning rate schedule will be displayed in a pop-up window.

    <img src="https://user-images.githubusercontent.com/33249023/227502329-6fd44259-e23b-46e0-8e19-29f9b664f4e2.png" height="250"/>
</center>

```{note}
The learning rate is auto scaled according to the actual batchsize.
```

Start the training by running the following command:

```Bash
python tools/train.py configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py
```

```

```Bash
```

Depending on the system environment, MMAction2 will automatically use the best device for training. If a GPU is available, a single GPU training will be started by default. When you start to see the output of the losses, you have successfully started the training.

```Bash
03/24 16:36:15 - mmengine - INFO - Exp name: tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb_20230324_163608
03/24 16:36:15 - mmengine - INFO - Epoch(train)  [1][8/8]  lr: 1.5625e-04  eta: 0:00:15  time: 0.2151  data_time: 0.0845  memory: 1314  grad_norm: 8.5647  loss: 0.7267  top1_acc: 0.0000  top5_acc: 1.0000  loss_cls: 0.7267
03/24 16:36:16 - mmengine - INFO - Exp name: tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb_20230324_163608

03/24 16:36:16 - mmengine - INFO - Epoch(train)  [2][8/8]  lr: 1.5625e-04  eta: 0:00:12  time: 0.1979  data_time: 0.0717  memory: 1314  grad_norm: 8.4709  loss: 0.7130  top1_acc: 0.0000  top5_acc: 1.0000  loss_cls: 0.7130
03/24 16:36:18 - mmengine - INFO - Exp name: tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb_20230324_163608
03/24 16:36:18 - mmengine - INFO - Epoch(train)  [3][8/8]  lr: 1.5625e-04  eta: 0:00:10  time: 0.1691  data_time: 0.0478  memory: 1314  grad_norm: 8.2910  loss: 0.6900  top1_acc: 0.5000  top5_acc: 1.0000  loss_cls: 0.6900
03/24 16:36:18 - mmengine - INFO - Saving checkpoint at 3 epochs
03/24 16:36:19 - mmengine - INFO - Epoch(val) [3][1/1]  acc/top1: 0.9000  acc/top5: 1.0000  acc/mean1: 0.9000data_time: 1.2716  time: 1.3658

03/24 16:36:20 - mmengine - INFO - The best checkpoint with 0.9000 acc/top1 at 3 epoch is saved to best_acc/top1_epoch_3.pth.
```

Without extra configurations, model weights will be saved to `work_dirs/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb/`, while the logs will be stored in `work_dirs/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb/TIMESTAMP/`. Next, we just need to wait with some patience for training to finish.

```{note}
For advanced usage of training, such as CPU training, multi-GPU training, and cluster training, please refer to [Training and Testing](../user_guides/train_test.md).
```

## Testing

After 10 epochs, we observe that TSN performs best in the 6th epoch, with `acc/top1` reaching 1.0000:

```Bash

```

## Testing

```Bash
03/24 16:36:25 - mmengine - INFO - Epoch(val) [6][1/1]  acc/top1: 1.0000  acc/top5: 1.0000  acc/mean1: 1.0000data_time: 1.0210  time: 1.1091
```

```{note}
The result is pretty high due to pre-trained on original Kinetics400, you may see a different result.
```

However, this value only reflects the validation performance of TSN on the mini Kinetics dataset, While test results are usually higher due to more augmentation in test pipeline.

Start testing:

```Bash
    work_dirs/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb/best_acc/top1_epoch_6.pth
```

And get the outputs like:

```Bash
03/24 17:00:59 - mmengine - INFO - Epoch(test) [10/10]  acc/top1: 1.0000  acc/top5: 1.0000  acc/mean1: 0.9000data_time: 0.0420  time: 1.0795
```

```

```Bash
```

The model achieves an top1-accuracy of 1.0000 on this dataset.

```{note}
For advanced usage of testing, such as CPU testing, multi-GPU testing, and cluster testing, please refer to [Training and Testing](../user_guides/train_test.md).
```

# TOOD

> [TOOD: Task-aligned One-stage Object Detection](https://arxiv.org/abs/2108.07755)

One-stage object detection is commonly implemented by optimizing two sub-tasks: object classification and localization, using heads with two parallel branches, which might lead to a certain level of spatial misalignment in predictions between the two tasks. In this work, we propose a Task-aligned One-stage Object Detection (TOOD) that explicitly aligns the two tasks in a learning-based manner. First, we design a novel Task-aligned Head (T-Head) which offers a better balance between learning task-interactive and task-specific features, as well as a greater flexibility to learn the alignment via a task-aligned predictor. Second, we propose Task Alignment Learning (TAL) to explicitly pull closer (or even unify) the optimal anchors for the two tasks during training via a designed sample assignment scheme and a task-aligned loss. Extensive experiments are conducted on MS-COCO, where TOOD achieves a 51.1 AP at single-model single-scale testing. This surpasses the recent one-stage detectors by a large margin, such as ATSS (47.7 AP), GFL (48.2 AP), and PAA (49.0 AP), with fewer parameters and FLOPs. Qualitative results also demonstrate the effectiveness of TOOD for better aligning the tasks of object classification and localization.

<img src="https://user-images.githubusercontent.com/12907710/145400075-e08191f5-8afa-4335-9b3b-27926fc9a26e.png"/>
</div>

|     Backbone      |  Style  | Anchor Type  | Lr schd | Multi-scale Training | Mem (GB) | Inf time (fps) | box AP |                          Config                           |                                                                                                                                                                       Download                                                                                                                                                                        |

|       R-50        | pytorch | Anchor-free  |   1x    |          N           |   4.1    |                |  42.4  |            [config](./tood_r50_fpn_1x_coco.py)            |                                           [model](https://download.openmmlab.com/mmdetection/v2.0/tood/tood_r50_fpn_1x_coco/tood_r50_fpn_1x_coco_20211210_103425-20e20746.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/tood/tood_r50_fpn_1x_coco/tood_r50_fpn_1x_coco_20211210_103425.log)                                           |

|       R-50        | pytorch | Anchor-based |   1x    |          N           |   4.1    |                |  42.4  |     [config](./tood_r50_fpn_anchor-based_1x_coco.py)      |                 [model](https://download.openmmlab.com/mmdetection/v2.0/tood/tood_r50_fpn_anchor_based_1x_coco/tood_r50_fpn_anchor_based_1x_coco_20211214_100105-b776c134.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/tood/tood_r50_fpn_anchor_based_1x_coco/tood_r50_fpn_anchor_based_1x_coco_20211214_100105.log)                 |

|       R-50        | pytorch | Anchor-free  |   2x    |          Y           |   4.1    |                |  44.5  |          [config](./tood_r50_fpn_ms-2x_coco.py)           |                           [model](https://download.openmmlab.com/mmdetection/v2.0/tood/tood_r50_fpn_mstrain_2x_coco/tood_r50_fpn_mstrain_2x_coco_20211210_144231-3b23174c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/tood/tood_r50_fpn_mstrain_2x_coco/tood_r50_fpn_mstrain_2x_coco_20211210_144231.log)                           |

|       R-101       | pytorch | Anchor-free  |   2x    |          Y           |   6.0    |                |  46.1  |          [config](./tood_r101_fpn_ms-2x_coco.py)          |                         [model](https://download.openmmlab.com/mmdetection/v2.0/tood/tood_r101_fpn_mstrain_2x_coco/tood_r101_fpn_mstrain_2x_coco_20211210_144232-a18f53c8.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/tood/tood_r101_fpn_mstrain_2x_coco/tood_r101_fpn_mstrain_2x_coco_20211210_144232.log)                         |

|    R-101-dcnv2    | pytorch | Anchor-free  |   2x    |          Y           |   6.2    |                |  49.3  |    [config](./tood_r101-dconv-c3-c5_fpn_ms-2x_coco.py)    | [model](https://download.openmmlab.com/mmdetection/v2.0/tood/tood_r101_fpn_dconv_c3-c5_mstrain_2x_coco/tood_r101_fpn_dconv_c3-c5_mstrain_2x_coco_20211210_213728-4a824142.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/tood/tood_r101_fpn_dconv_c3-c5_mstrain_2x_coco/tood_r101_fpn_dconv_c3-c5_mstrain_2x_coco_20211210_213728.log) |

|    X-101-64x4d    | pytorch | Anchor-free  |   2x    |          Y           |   10.2   |                |  47.6  |       [config](./tood_x101-64x4d_fpn_ms-2x_coco.py)       |             [model](https://download.openmmlab.com/mmdetection/v2.0/tood/tood_x101_64x4d_fpn_mstrain_2x_coco/tood_x101_64x4d_fpn_mstrain_2x_coco_20211211_003519-a4f36113.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/tood/tood_x101_64x4d_fpn_mstrain_2x_coco/tood_x101_64x4d_fpn_mstrain_2x_coco_20211211_003519.log)             |

| X-101-64x4d-dcnv2 | pytorch | Anchor-free  |   2x    |          Y           |          |                |        | [config](./tood_x101-64x4d-dconv-c4-c5_fpn_ms-2x_coco.py) |                                                                                                                                                               [model](<>) \| [log](<>)                                                                                                                                                                |

\[1\] *1x and 2x mean the model is trained for 90K and 180K iterations, respectively.* \
\[2\] *All results are obtained with a single model and without any test time data augmentation such as multi-scale, flipping and etc..* \
\[3\] *`dcnv2` denotes deformable convolutional networks v2.* \\

```latex
@inproceedings{feng2021tood,
    title={TOOD: Task-aligned One-stage Object Detection},

```latex
    author={Feng, Chengjian and Zhong, Yujie and Gao, Yu and Scott, Matthew R and Huang, Weilin},
    booktitle={ICCV},
}
```

</div>

|   Model   | Input Size | PCK@0.2 |  AUC  |  EPE  |                   Details and Download                    |
| :-------: | :--------: | :-----: | :---: | :---: | :-------------------------------------------------------: |

| ResNet-50 |  256x256   |  0.990  | 0.485 | 34.21 | [resnet_onehand10k.md](./onehand10k/resnet_onehand10k.md) |

|   Model   | Input Size | PCK@0.2 |  AUC  | EPE  |            Details and Download            |
| :-------: | :--------: | :-----: | :---: | :--: | :----------------------------------------: |
| ResNet-50 |  256x256   |  0.988  | 0.865 | 3.32 | [resnet_rhd2d.md](./rhd2d/resnet_rhd2d.md) |

# InterHand2.6M: A dataset and baseline for 3D interacting hand pose estimation from a single RGB image

<details>

```bibtex
@article{moon2020interhand2,
  title={InterHand2.6M: A dataset and baseline for 3D interacting hand pose estimation from a single RGB image},
  author={Moon, Gyeongsik and Yu, Shoou-I and Wen, He and Shiratori, Takaaki and Lee, Kyoung Mu},
  journal={arXiv preprint arXiv:2008.09309},
}
```

# Model Zoo Statistics

- Number of papers: 47

  - ALGORITHM: 36
  - BACKBONE: 11

- Number of checkpoints: 612

  - \[ALGORITHM\] [ANN](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ann) (16 ckpts)

  - \[ALGORITHM\] [APCNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/apcnet) (12 ckpts)

  - \[BACKBONE\] [BEiT](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/beit) (2 ckpts)

  - \[ALGORITHM\] [BiSeNetV1](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/bisenetv1) (11 ckpts)

  - \[ALGORITHM\] [BiSeNetV2](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/bisenetv2) (4 ckpts)

  - \[ALGORITHM\] [CCNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ccnet) (16 ckpts)

  - \[ALGORITHM\] [CGNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/cgnet) (2 ckpts)

  - \[BACKBONE\] [ConvNeXt](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/convnext) (6 ckpts)

- \[BACKBONE\] [ConvNeXt](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/convnext) (6 ckpts)

  - \[ALGORITHM\] [DANet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/danet) (16 ckpts)

  - \[ALGORITHM\] [DeepLabV3](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3) (41 ckpts)

  - \[ALGORITHM\] [DeepLabV3+](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/deeplabv3plus) (42 ckpts)

  - \[ALGORITHM\] [DMNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dmnet) (12 ckpts)

  - \[ALGORITHM\] [DNLNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dnlnet) (12 ckpts)

  - \[ALGORITHM\] [DPT](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/dpt) (1 ckpts)

  - \[ALGORITHM\] [EMANet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/emanet) (4 ckpts)

  - \[ALGORITHM\] [EncNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/encnet) (12 ckpts)

- \[ALGORITHM\] [EncNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/encnet) (12 ckpts)

  - \[ALGORITHM\] [ERFNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/erfnet) (1 ckpts)

  - \[ALGORITHM\] [FastFCN](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fastfcn) (12 ckpts)

  - \[ALGORITHM\] [Fast-SCNN](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fastscnn) (1 ckpts)

  - \[ALGORITHM\] [FCN](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn) (41 ckpts)

  - \[ALGORITHM\] [GCNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/gcnet) (16 ckpts)

  - \[BACKBONE\] [HRNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet) (37 ckpts)

  - \[ALGORITHM\] [ICNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/icnet) (12 ckpts)

  - \[ALGORITHM\] [ISANet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/isanet) (16 ckpts)

- \[ALGORITHM\] [ISANet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/isanet) (16 ckpts)

  - \[ALGORITHM\] [K-Net](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/knet) (7 ckpts)

  - \[BACKBONE\] [MAE](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mae) (1 ckpts)

  - \[ALGORITHM\] [Mask2Former](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mask2former) (13 ckpts)

  - \[ALGORITHM\] [MaskFormer](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/maskformer) (4 ckpts)

  - \[BACKBONE\] [MobileNetV2](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mobilenet_v2) (8 ckpts)

  - \[BACKBONE\] [MobileNetV3](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mobilenet_v3) (4 ckpts)

  - \[ALGORITHM\] [NonLocal Net](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/nonlocal_net) (16 ckpts)

- \[ALGORITHM\] [NonLocal Net](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/nonlocal_net) (16 ckpts)

  - \[ALGORITHM\] [OCRNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet) (24 ckpts)

  - \[ALGORITHM\] [PointRend](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/point_rend) (4 ckpts)

  - \[BACKBONE\] [PoolFormer](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/poolformer) (5 ckpts)

  - \[ALGORITHM\] [PSANet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/psanet) (16 ckpts)

  - \[ALGORITHM\] [PSPNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pspnet) (54 ckpts)

  - \[BACKBONE\] [ResNeSt](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/resnest) (8 ckpts)

  - \[ALGORITHM\] [SegFormer](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segformer) (13 ckpts)

- \[ALGORITHM\] [SegFormer](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segformer) (13 ckpts)

  - \[ALGORITHM\] [Segmenter](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segmenter) (5 ckpts)

  - \[ALGORITHM\] [Semantic FPN](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/sem_fpn) (4 ckpts)

  - \[ALGORITHM\] [SETR](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/setr) (7 ckpts)

  - \[ALGORITHM\] [STDC](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/stdc) (4 ckpts)

  - \[BACKBONE\] [Swin Transformer](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/swin) (6 ckpts)

  - \[BACKBONE\] [Twins](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/twins) (12 ckpts)

  - \[ALGORITHM\] [UNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet) (25 ckpts)

  - \[ALGORITHM\] [UPerNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/upernet) (16 ckpts)

- \[ALGORITHM\] [UPerNet](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/upernet) (16 ckpts)

  - \[BACKBONE\] [Vision Transformer](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/vit) (11 ckpts)

# DeepFillv1 (CVPR'2018)

> [Generative Image Inpainting with Contextual Attention](https://arxiv.org/abs/1801.07892)

Recent deep learning based approaches have shown promising results for the challenging task of inpainting large missing regions in an image. These methods can generate visually plausible image structures and textures, but often create distorted structures or blurry textures inconsistent with surrounding areas. This is mainly due to ineffectiveness of convolutional neural networks in explicitly borrowing or copying information from distant spatial locations. On the other hand, traditional texture and patch synthesis approaches are particularly suitable when it needs to borrow textures from the surrounding regions. Motivated by these observations, we propose a new deep generative model-based approach which can not only synthesize novel image structures but also explicitly utilize surrounding image features as references during network training to make better predictions. The model is a feed-forward, fully convolutional neural network which can process images with multiple holes at arbitrary locations and with variable sizes during the test time. Experiments on multiple datasets including faces (CelebA, CelebA-HQ), textures (DTD) and natural images (ImageNet, Places2) demonstrate that our proposed approach generates higher-quality inpainting results than existing ones.

 <img src="https://user-images.githubusercontent.com/12726765/144174665-9675931f-e448-4475-a659-99b65e7d4a64.png" width="400"/>
</div >

|                       Model                       |  Mask Type  | Resolution | Train Iters |    Dataset    | l1 error |  PSNR  | SSIM  | Training Resources |                               Download                                |
| :-----------------------------------------------: | :---------: | :--------: | :---------: | :-----------: | :------: | :----: | :---: | :----------------: | :-------------------------------------------------------------------: |

| [DeepFillv1](./deepfillv1_8xb2_places-256x256.py) | square bbox |  256x256   |    3500k    | Places365-val |  11.019  | 23.429 | 0.862 |         8          | [model](https://download.openmmlab.com/mmediting/inpainting/deepfillv1/deepfillv1_256x256_8x2_places_20200619-c00a0e21.pth) \| [log](https://download.openmmlab.com/mmediting/inpainting/deepfillv1/deepfillv1_256x256_8x2_places_20200619-c00a0e21.log.json) |

|                       Model                       |  Mask Type  | Resolution | Train Iters |  Dataset   | l1 error |  PSNR  | SSIM  | Training Resources |                                 Download                                 |
| :-----------------------------------------------: | :---------: | :--------: | :---------: | :--------: | :------: | :----: | :---: | :----------------: | :----------------------------------------------------------------------: |

| [DeepFillv1](./deepfillv1_4xb4_celeba-256x256.py) | square bbox |  256x256   |    1500k    | CelebA-val |  6.677   | 26.878 | 0.911 |         4          | [model](https://download.openmmlab.com/mmediting/inpainting/deepfillv1/deepfillv1_256x256_4x4_celeba_20200619-dd51a855.pth) \| [log](https://download.openmmlab.com/mmediting/inpainting/deepfillv1/deepfillv1_256x256_4x4_celeba_20200619-dd51a855.log.json) |

**Train**

<details>

```shell
# cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/deepfillv1/deepfillv1_8xb2_places-256x256.py

python tools/train.py configs/deepfillv1/deepfillv1_8xb2_places-256x256.py

./tools/dist_train.sh configs/deepfillv1/deepfillv1_8xb2_places-256x256.py 8
```

```

**Test**

<details>

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/deepfillv1/deepfillv1_8xb2_places-256x256.py https://download.openmmlab.com/mmediting/inpainting/deepfillv1/deepfillv1_256x256_8x2_places_20200619-c00a0e21.pth

python tools/test.py configs/deepfillv1/deepfillv1_8xb2_places-256x256.py https://download.openmmlab.com/mmediting/inpainting/deepfillv1/deepfillv1_256x256_8x2_places_20200619-c00a0e21.pth

./tools/dist_test.sh configs/deepfillv1/deepfillv1_8xb2_places-256x256.py https://download.openmmlab.com/mmediting/inpainting/deepfillv1/deepfillv1_256x256_8x2_places_20200619-c00a0e21.pth 8
```

```bibtex
@inproceedings{yu2018generative,
  title={Generative image inpainting with contextual attention},
  pages={5505--5514},
}
```

# Open Images Dataset

> [Open Images Dataset](https://arxiv.org/abs/1811.00982)

#### Open Images v6

[Open Images](https://storage.googleapis.com/openimages/web/index.html) is a dataset of ~9M images annotated with image-level labels,
object bounding boxes, object segmentation masks, visual relationships,
and localized narratives:

- It contains a total of 16M bounding boxes for 600 object classes on
  1.9M images, making it the largest existing dataset with object location
  annotations. The boxes have been largely manually drawn by professional
  annotators to ensure accuracy and consistency. The images are very diverse
  and often contain complex scenes with several objects (8.3 per image on
  average).

- Open Images also offers visual relationship annotations, indicating pairs
  of objects in particular relations (e.g. "woman playing guitar", "beer on

average).

  table"), object properties (e.g. "table is wooden"), and human actions (e.g.
  "woman is jumping"). In total it has 3.3M annotations from 1,466 distinct
  relationship triplets.

- In V5 we added segmentation masks for 2.8M object instances in 350 classes.
  Segmentation masks mark the outline of objects, which characterizes their
  spatial extent to a much higher level of detail.

- In V6 we added 675k localized narratives: multimodal descriptions of images
  consisting of synchronized voice, text, and mouse traces over the objects being
  described. (Note we originally launched localized narratives only on train in V6,
  but since July 2020 we also have validation and test covered.)

- Finally, the dataset is annotated with 59.9M image-level labels spanning 19,957
  classes.

but since July 2020 we also have validation and test covered.)

  classes.

We believe that having a single dataset with unified annotations for image
classification, object detection, visual relationship detection, instance
segmentation, and multimodal image descriptions will enable to study these
tasks jointly and stimulate progress towards genuine scene understanding.

<img src="https://user-images.githubusercontent.com/48282753/147199750-23e17230-c0cf-49a0-a13c-0d014d49107e.png" height="400"/>
</div>

#### Open Images Challenge 2019

[Open Images Challenges 2019](https://storage.googleapis.com/openimages/web/challenge2019.html) is based on the V5 release of the Open
Images dataset. The images of the dataset are very varied and
often contain complex scenes with several objects (explore the dataset).

```
@article{OpenImages,

```
  author = {Alina Kuznetsova and Hassan Rom and Neil Alldrin and Jasper Uijlings and Ivan Krasin and Jordi Pont-Tuset and Shahab Kamali and Stefan Popov and Matteo Malloci and Alexander Kolesnikov and Tom Duerig and Vittorio Ferrari},
  title = {The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale},
  year = {2020},
  journal = {IJCV}
}
```

1. You need to download and extract Open Images dataset.

2. The Open Images dataset does not have image metas (width and height of the image),
   which will be used during training and testing (evaluation). We suggest to get test image metas before
   training/testing by using `tools/misc/get_image_metas.py`.

   **Usage**

   ```shell
   python tools/misc/get_image_metas.py ${CONFIG} \

training/testing by using `tools/misc/get_image_metas.py`.

   **Usage**

   ```shell
   --dataset ${DATASET TYPE} \  # train or val or test
   --out ${OUTPUT FILE NAME}
   ```

3. The directory should be like this:

   ```none
   mmdetection
   ├── mmdet
   ├── tools
   ├── data
   │   ├── OpenImages
   │   │   ├── annotations
   │   │   │   ├── bbox_labels_600_hierarchy.json
   │   │   │   ├── class-descriptions-boxable.csv
   │   │   │   ├── oidv6-train-annotations-bbox.scv
   │   │   │   ├── validation-annotations-bbox.csv
   │   │   │   ├── validation-annotations-human-imagelabels-boxable.csv
   │   │   │   ├── validation-image-metas.pkl      # get from script
   │   │   ├── challenge2019
   │   │   │   ├── challenge-2019-train-detection-bbox.txt
   │   │   │   ├── challenge-2019-validation-detection-bbox.txt
   │   │   │   ├── class_label_tree.np
   │   │   │   ├── class_sample_train.pkl

│   │   │   ├── challenge-2019-validation-detection-bbox.txt
   │   │   │   ├── challenge-2019-validation-detection-human-imagelabels.csv       # download from official website
   │   │   │   ├── challenge-2019-validation-metas.pkl     # get from script
   │   │   ├── OpenImages
   │   │   │   ├── train           # training images
   │   │   │   ├── test            # testing images
   │   │   │   ├── validation      # validation images
   ```

**Note**:

1. The training and validation images of Open Images Challenge dataset are based on
   Open Images v6, but the test images are different.
2. The Open Images Challenges annotations are obtained from [TSD](https://github.com/Sense-X/TSD).
   You can also download the annotations from [official website](https://storage.googleapis.com/openimages/web/challenge2019_downloads.html),

You can also download the annotations from [official website](https://storage.googleapis.com/openimages/web/challenge2019_downloads.html),
   and set data.train.type=OpenImagesDataset, data.val.type=OpenImagesDataset, and data.test.type=OpenImagesDataset in the config
3. If users do not want to use `validation-annotations-human-imagelabels-boxable.csv` and `challenge-2019-validation-detection-human-imagelabels.csv`
   users can set `test_dataloader.dataset.image_level_ann_file=None` and `test_dataloader.dataset.image_level_ann_file=None` in the config.
   Please note that loading image-levels label is the default of Open Images evaluation metric.
   More details please refer to the [official website](https://storage.googleapis.com/openimages/web/evaluation.html)

More details please refer to the [official website](https://storage.googleapis.com/openimages/web/evaluation.html)

|         Architecture          | Backbone |  Style  | Lr schd |       Sampler       | Mem (GB) | Inf time (fps) | box AP |                                Config                                |                                                                                                                                                                                                      Download                                                                                                                                                                                                      |

|         Faster R-CNN          |   R-50   | pytorch |   1x    |    Group Sampler    |   7.7    |       -        |  51.6  |        [config](./faster-rcnn_r50_fpn_32xb2-1x_openimages.py)        |                             [model](https://download.openmmlab.com/mmdetection/v2.0/openimages/faster_rcnn_r50_fpn_32x2_1x_openimages/faster_rcnn_r50_fpn_32x2_1x_openimages_20211130_231159-e87ab7ce.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/openimages/faster_rcnn_r50_fpn_32x2_1x_openimages/faster_rcnn_r50_fpn_32x2_1x_openimages_20211130_231159.log.json)                             |

|         Faster R-CNN          |   R-50   | pytorch |   1x    | Class Aware Sampler |   7.7    |       -        |  60.0  |      [config](./faster-rcnn_r50_fpn_32xb2-cas-1x_openimages.py)      |                     [model](https://download.openmmlab.com/mmdetection/v2.0/openimages/faster_rcnn_r50_fpn_32x2_cas_1x_openimages/faster_rcnn_r50_fpn_32x2_cas_1x_openimages_20220306_202424-98c630e5.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/openimages/faster_rcnn_r50_fpn_32x2_cas_1x_openimages/faster_rcnn_r50_fpn_32x2_cas_1x_openimages_20220306_202424.log.json)                     |

| Faster R-CNN (Challenge 2019) |   R-50   | pytorch |   1x    |    Group Sampler    |   7.7    |       -        |  54.9  |   [config](./faster-rcnn_r50_fpn_32xb2-1x_openimages-challenge.py)   |         [model](https://download.openmmlab.com/mmdetection/v2.0/openimages/faster_rcnn_r50_fpn_32x2_1x_openimages_challenge/faster_rcnn_r50_fpn_32x2_1x_openimages_challenge_20220114_045100-0e79e5df.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/openimages/faster_rcnn_r50_fpn_32x2_1x_openimages_challenge/faster_rcnn_r50_fpn_32x2_1x_openimages_challenge_20220114_045100.log.json)         |

| Faster R-CNN (Challenge 2019) |   R-50   | pytorch |   1x    | Class Aware Sampler |   7.1    |       -        |  65.0  | [config](./faster-rcnn_r50_fpn_32xb2-cas-1x_openimages-challenge.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/openimages/faster_rcnn_r50_fpn_32x2_cas_1x_openimages_challenge/faster_rcnn_r50_fpn_32x2_cas_1x_openimages_challenge_20220221_192021-34c402d9.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/openimages/faster_rcnn_r50_fpn_32x2_cas_1x_openimages_challenge/faster_rcnn_r50_fpn_32x2_cas_1x_openimages_challenge_20220221_192021.log.json) |

|           Retinanet           |   R-50   | pytorch |   1x    |    Group Sampler    |   6.6    |       -        |  61.5  |         [config](./retinanet_r50_fpn_32xb2-1x_openimages.py)         |                                 [model](https://download.openmmlab.com/mmdetection/v2.0/openimages/retinanet_r50_fpn_32x2_1x_openimages/retinanet_r50_fpn_32x2_1x_openimages_20211223_071954-d2ae5462.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/openimages/retinanet_r50_fpn_32x2_1x_openimages/retinanet_r50_fpn_32x2_1x_openimages_20211223_071954.log.json)                                 |

|              SSD              |  VGG16   | pytorch |   36e   |    Group Sampler    |   10.8   |       -        |  35.4  |              [config](./ssd300_32xb8-36e_openimages.py)              |                                                     [model](https://download.openmmlab.com/mmdetection/v2.0/openimages/ssd300_32x8_36e_openimages/ssd300_32x8_36e_openimages_20211224_000232-dce93846.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/openimages/ssd300_32x8_36e_openimages/ssd300_32x8_36e_openimages_20211224_000232.log.json)                                                     |

- 'cas' is short for 'Class Aware Sampler'

### Results of consider image level labels

|           Architecture            |       Sampler       | Consider Image Level Labels | box AP |
| :-------------------------------: | :-----------------: | :-------------------------: | :----: |
| Faster R-CNN r50 (Challenge 2019) |    Group Sampler    |             w/o             | 62.19  |

| Faster R-CNN r50 (Challenge 2019) |    Group Sampler    |             w/              | 54.87  |
| Faster R-CNN r50 (Challenge 2019) | Class Aware Sampler |             w/o             | 71.77  |
| Faster R-CNN r50 (Challenge 2019) | Class Aware Sampler |             w/              | 64.98  |

}
```

<details>

```bibtex
}
```

pages={5167--5176},
}
```

| [pose_resnet_50](/configs/body_2d_keypoint/topdown_heatmap/posetrack18/td-hm_res50_8xb64-20e_posetrack18-256x192.py) |  256x192   | 86.5 | 87.7 | 82.5 | 75.8 | 80.1 | 78.8 | 74.2 | 81.2  | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnet/res50_posetrack18_256x192-a62807c7_20201028.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnet/res50_posetrack18_256x192_20201028.log.json) |

# OCRNet

> [Object-Contextual Representations for Semantic Segmentation](https://arxiv.org/abs/1909.11065)

<a href="https://github.com/openseg-group/OCNet.pytorch">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/decode_heads/ocr_head.py#L86">Code Snippet</a>

In this paper, we address the problem of semantic segmentation and focus on the context aggregation strategy for robust segmentation. Our motivation is that the label of a pixel is the category of the object that the pixel belongs to. We present a simple yet effective approach, object-contextual representations, characterizing a pixel by exploiting the representation of the corresponding object class. First, we construct object regions based on a feature map supervised by the ground-truth segmentation, and then compute the object region representations. Second, we compute the representation similarity between each pixel and each object region, and augment the representation of each pixel with an object contextual representation, which is a weighted aggregation of all the object region representations according to their similarities with the pixel. We empirically demonstrate that the proposed approach achieves competitive performance on six challenging semantic segmentation benchmarks: Cityscapes, ADE20K, LIP, PASCAL VOC 2012, PASCAL-Context and COCO-Stuff. Notably, we achieved the \\nth{2} place on the Cityscapes leader-board with a single model.

<img src="https://user-images.githubusercontent.com/24582831/142902197-b06b1e04-57ab-44ac-adc8-cea6695bb236.png" width="70%"/>
</div>

#### HRNet backbone

| Method | Backbone           | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                        | download                                                                                                                                                                                                                                                                                                                                                         |

| OCRNet | HRNetV2p-W18-Small | 512x1024  |   40000 | 3.5      | 10.45          | A100   | 76.61 |         78.01 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_hr18s_4xb2-40k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18s_4xb2-40k_cityscapes-512x1024/ocrnet_hr18s_4xb2-40k_cityscapes-512x1024_20230227_145026-6c052a14.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18s_4xb2-40k_cityscapes-512x1024/ocrnet_hr18s_4xb2-40k_cityscapes-512x1024_20230227_145026.json) |

| OCRNet | HRNetV2p-W18       | 512x1024  |   40000 | 4.7      | 7.50           | V100   | 77.72 |         79.49 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_hr18_4xb2-40k_cityscapes-512x1024.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18_512x1024_40k_cityscapes/ocrnet_hr18_512x1024_40k_cityscapes_20200601_033320-401c5bdd.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18_512x1024_40k_cityscapes/ocrnet_hr18_512x1024_40k_cityscapes_20200601_033320.log.json)                     |

| OCRNet | HRNetV2p-W48       | 512x1024  |   40000 | 8        | 4.22           | V100   | 80.58 |         81.79 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_hr48_4xb2-40k_cityscapes-512x1024.pyy)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr48_512x1024_40k_cityscapes/ocrnet_hr48_512x1024_40k_cityscapes_20200601_033336-55b32491.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr48_512x1024_40k_cityscapes/ocrnet_hr48_512x1024_40k_cityscapes_20200601_033336.log.json)                     |

| OCRNet | HRNetV2p-W18-Small | 512x1024  |   80000 | -        | -              | V100   | 77.16 |         78.66 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_hr18s_4xb2-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18s_512x1024_80k_cityscapes/ocrnet_hr18s_512x1024_80k_cityscapes_20200601_222735-55979e63.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18s_512x1024_80k_cityscapes/ocrnet_hr18s_512x1024_80k_cityscapes_20200601_222735.log.json)                 |

| OCRNet | HRNetV2p-W18       | 512x1024  |   80000 | -        | -              | V100   | 78.57 |         80.46 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_hr18_4xb2-80k_cityscapes-512x1024.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18_512x1024_80k_cityscapes/ocrnet_hr18_512x1024_80k_cityscapes_20200614_230521-c2e1dd4a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18_512x1024_80k_cityscapes/ocrnet_hr18_512x1024_80k_cityscapes_20200614_230521.log.json)                     |

| OCRNet | HRNetV2p-W48       | 512x1024  |   80000 | -        | -              | V100   | 80.70 |         81.87 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_hr48_4xb2-80k_cityscapes-512x1024.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr48_512x1024_80k_cityscapes/ocrnet_hr48_512x1024_80k_cityscapes_20200601_222752-9076bcdf.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr48_512x1024_80k_cityscapes/ocrnet_hr48_512x1024_80k_cityscapes_20200601_222752.log.json)                     |

| OCRNet | HRNetV2p-W18-Small | 512x1024  |  160000 | -        | -              | V100   | 78.45 |         79.97 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_hr18s_4xb2-160k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18s_512x1024_160k_cityscapes/ocrnet_hr18s_512x1024_160k_cityscapes_20200602_191005-f4a7af28.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18s_512x1024_160k_cityscapes/ocrnet_hr18s_512x1024_160k_cityscapes_20200602_191005.log.json)             |

| OCRNet | HRNetV2p-W18       | 512x1024  |  160000 | -        | -              | V100   | 79.47 |         80.91 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_hr18_4xb2-160k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18_512x1024_160k_cityscapes/ocrnet_hr18_512x1024_160k_cityscapes_20200602_191001-b9172d0c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18_512x1024_160k_cityscapes/ocrnet_hr18_512x1024_160k_cityscapes_20200602_191001.log.json)                 |

| OCRNet | HRNetV2p-W48       | 512x1024  |  160000 | -        | -              | V100   | 81.35 |         82.70 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_hr48_4xb2-160k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr48_512x1024_160k_cityscapes/ocrnet_hr48_512x1024_160k_cityscapes_20200602_191037-dfbf1b0c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr48_512x1024_160k_cityscapes/ocrnet_hr48_512x1024_160k_cityscapes_20200602_191037.log.json)                 |

#### ResNet backbone

| Method | Backbone | Crop Size | Batch Size | Lr schd | Mem (GB) | Inf time (fps) | Device | mIoU  | mIoU(ms+flip) | config                                                                                                                         | download                                                                                                                                                                                                                                                                                                                                                                 |

| OCRNet | R-101-D8 | 512x1024  | 8          | 40000   | -        | -              | V100   | 80.09 |             - | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_r101-d8_4xb2-40k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_r101-d8_512x1024_40k_b8_cityscapes/ocrnet_r101-d8_512x1024_40k_b8_cityscapes_20200717_110721-02ac0f13.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_r101-d8_512x1024_40k_b8_cityscapes/ocrnet_r101-d8_512x1024_40k_b8_cityscapes_20200717_110721.log.json)     |

| OCRNet | R-101-D8 | 512x1024  | 16         | 40000   | 8.8      | 3.02           | V100   | 80.30 |             - | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_r101-d8_8xb2-40k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_r101-d8_512x1024_40k_b16_cityscapes/ocrnet_r101-d8_512x1024_40k_b16_cityscapes_20200723_193726-db500f80.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_r101-d8_512x1024_40k_b16_cityscapes/ocrnet_r101-d8_512x1024_40k_b16_cityscapes_20200723_193726.log.json) |

| OCRNet | R-101-D8 | 512x1024  | 16         | 80000   | 8.8      | 3.02           | V100   | 80.81 |             - | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_r101-d8_8xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_r101-d8_512x1024_80k_b16_cityscapes/ocrnet_r101-d8_512x1024_80k_b16_cityscapes_20200723_192421-78688424.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_r101-d8_512x1024_80k_b16_cityscapes/ocrnet_r101-d8_512x1024_80k_b16_cityscapes_20200723_192421.log.json) |

### ADE20K

### ADE20K

| Method | Backbone           | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                  | download                                                                                                                                                                                                                                                                                                                         |

| OCRNet | HRNetV2p-W18-Small | 512x512   |   80000 | 6.7      | 28.98          | V100   | 35.06 |         35.80 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_hr18s_4xb4-80k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18s_512x512_80k_ade20k/ocrnet_hr18s_512x512_80k_ade20k_20200615_055600-e80b62af.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18s_512x512_80k_ade20k/ocrnet_hr18s_512x512_80k_ade20k_20200615_055600.log.json)     |

| OCRNet | HRNetV2p-W18       | 512x512   |   80000 | 7.9      | 18.93          | V100   | 37.79 |         39.16 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_hr18_4xb4-80k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18_512x512_80k_ade20k/ocrnet_hr18_512x512_80k_ade20k_20200615_053157-d173d83b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18_512x512_80k_ade20k/ocrnet_hr18_512x512_80k_ade20k_20200615_053157.log.json)         |

| OCRNet | HRNetV2p-W48       | 512x512   |   80000 | 11.2     | 16.99          | V100   | 43.00 |         44.30 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_hr48_4xb4-80k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr48_512x512_80k_ade20k/ocrnet_hr48_512x512_80k_ade20k_20200615_021518-d168c2d1.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr48_512x512_80k_ade20k/ocrnet_hr48_512x512_80k_ade20k_20200615_021518.log.json)         |

| OCRNet | HRNetV2p-W18-Small | 512x512   |  160000 | -        | -              | V100   | 37.19 |         38.40 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_hr18s_4xb4-80k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18s_512x512_160k_ade20k/ocrnet_hr18s_512x512_160k_ade20k_20200615_184505-8e913058.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18s_512x512_160k_ade20k/ocrnet_hr18s_512x512_160k_ade20k_20200615_184505.log.json) |

| OCRNet | HRNetV2p-W18       | 512x512   |  160000 | -        | -              | V100   | 39.32 |         40.80 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_hr18_4xb4-80k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18_512x512_160k_ade20k/ocrnet_hr18_512x512_160k_ade20k_20200615_200940-d8fcd9d1.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18_512x512_160k_ade20k/ocrnet_hr18_512x512_160k_ade20k_20200615_200940.log.json)     |

| OCRNet | HRNetV2p-W48       | 512x512   |  160000 | -        | -              | V100   | 43.25 |         44.88 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_hr48_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr48_512x512_160k_ade20k/ocrnet_hr48_512x512_160k_ade20k_20200615_184705-a073726d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr48_512x512_160k_ade20k/ocrnet_hr48_512x512_160k_ade20k_20200615_184705.log.json)     |

| Method | Backbone           | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                    | download                                                                                                                                                                                                                                                                                                                             |

| OCRNet | HRNetV2p-W18-Small | 512x512   |   20000 | 3.5      | 31.55          | V100   | 71.70 |         73.84 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_hr18s_4xb4-20k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18s_512x512_20k_voc12aug/ocrnet_hr18s_512x512_20k_voc12aug_20200617_233913-02b04fcb.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18s_512x512_20k_voc12aug/ocrnet_hr18s_512x512_20k_voc12aug_20200617_233913.log.json) |

| OCRNet | HRNetV2p-W18       | 512x512   |   20000 | 4.7      | 19.91          | V100   | 74.75 |         77.11 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_hr18_4xb4-20k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18_512x512_20k_voc12aug/ocrnet_hr18_512x512_20k_voc12aug_20200617_233932-8954cbb7.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18_512x512_20k_voc12aug/ocrnet_hr18_512x512_20k_voc12aug_20200617_233932.log.json)     |

| OCRNet | HRNetV2p-W48       | 512x512   |   20000 | 8.1      | 17.83          | V100   | 77.72 |         79.87 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_hr48_4xb4-20k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr48_512x512_20k_voc12aug/ocrnet_hr48_512x512_20k_voc12aug_20200617_233932-9e82080a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr48_512x512_20k_voc12aug/ocrnet_hr48_512x512_20k_voc12aug_20200617_233932.log.json)     |

| OCRNet | HRNetV2p-W18-Small | 512x512   |   40000 | -        | -              | V100   | 72.76 |         74.60 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_hr18s_4xb4-40k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18s_512x512_40k_voc12aug/ocrnet_hr18s_512x512_40k_voc12aug_20200614_002025-42b587ac.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18s_512x512_40k_voc12aug/ocrnet_hr18s_512x512_40k_voc12aug_20200614_002025.log.json) |

| OCRNet | HRNetV2p-W18       | 512x512   |   40000 | -        | -              | V100   | 74.98 |         77.40 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_hr18_4xb4-40k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18_512x512_40k_voc12aug/ocrnet_hr18_512x512_40k_voc12aug_20200614_015958-714302be.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr18_512x512_40k_voc12aug/ocrnet_hr18_512x512_40k_voc12aug_20200614_015958.log.json)     |

| OCRNet | HRNetV2p-W48       | 512x512   |   40000 | -        | -              | V100   | 77.14 |         79.71 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ocrnet/ocrnet_hr48_4xb4-40k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr48_512x512_40k_voc12aug/ocrnet_hr48_512x512_40k_voc12aug_20200614_015958-255bc5ce.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ocrnet/ocrnet_hr48_512x512_40k_voc12aug/ocrnet_hr48_512x512_40k_voc12aug_20200614_015958.log.json)     |

```bibtex
@article{YuanW18,
  title={Ocnet: Object context network for scene parsing},
  author={Yuhui Yuan and Jingdong Wang},
  booktitle={arXiv preprint arXiv:1809.00916},
}

@article{YuanCW20,
  title={Object-Contextual Representations for Semantic Segmentation},
  author={Yuhui Yuan and Xilin Chen and Jingdong Wang},
}
```

# MMPose 0.X 兼容性说明

MMPose 1.0 经过了大规模重构并解决了许多遗留问题，对于 0.x 版本的大部分代码 MMPose 1.0 将不兼容。

## 数据变换

### 平移、旋转和缩放

旧版的数据变换方法 `TopDownRandomShiftBboxCenter` 和 `TopDownGetRandomScaleRotation`，将被合并为 `RandomBBoxTransform`：

```Python

    Args:
            0.3

Defaults to 0.16
            0.3
            to 1.0
            to 80.0
            to 0.6
    """

```

### 标签生成

scale_prob: float = 1.0,
```

### 标签生成

旧版用于训练标签生成的方法 `TopDownGenerateTarget` 、`TopDownGenerateTargetRegression`、`BottomUpGenerateHeatmapTarget`、`BottomUpGenerateTarget` 等将被合并为 `GenerateTarget`，而实际的生成方法由[编解码器](./user_guides/codecs.md) 提供：

```Python

    Args:

the results, e.g. ``'heatmaps'`` or ``'keypoint_weights'``. See

    Args:
            supported
    """

use_dataset_keypoint_weights (bool): Whether use the keypoint weights
    """

```

### 数据归一化

旧版的数据归一化操作 `NormalizeTensor` 和 `ToTensor` 方法将由 **DataPreprocessor** 模块替代，不再作为流水线的一部分，而是作为模块加入到模型前向传播中。

## 模型兼容

我们对 model zoo 提供的模型权重进行了兼容性处理，确保相同的模型权重测试精度能够与 0.x 版本保持同等水平，但由于在这两个版本中存在大量处理细节的差异，推理结果可能会产生轻微的不同（精度误差小于 0.05%）。

对于使用 0.x 版本训练保存的模型权重，我们在预测头中提供了 `_load_state_dict_pre_hook()` 方法来将旧版的权重字典替换为新版，如果你希望将在旧版上开发的模型兼容到新版，可以参考我们的实现。

```Python

    ## omitted

```

### Heatmap-based 方法

## omitted

```

对于基于SimpleBaseline方法的模型，主要需要注意最后一层卷积层的兼容：

```Python

        return

            continue
        #

# "final_layer").
        #
        #
                else:

k_new = 'conv_layers.' + '.'.join(k_parts[1:])
                else:
            else:
                k_new = k
        else:
            k_new = k

```

### RLE-based 方法

对于基于 RLE 的模型，由于新版的 `loss` 模块更名为 `loss_module`，且 flow 模型归属在 `loss` 模块下，因此需要对权重字典中 `loss` 字段进行更改：

```Python

        return

keys = list(state_dict.keys())
        else:
            k_new = _k

```

# step1 create a new empty class
```cpp
// mylayer.h
using namespace ncnn;

// a new layer type called MyLayer
class MyLayer : public Layer
{
};

// mylayer.cpp
#include "mylayer.h"
DEFINE_LAYER_CREATOR(MyLayer)
```

# step2 declare layer parameters and weights
```cpp

{
private:
    int channels;// new code
    float gamma;// new code
    Mat weight;// new code
};

```

# step3 implement load functions for parameters and weights
```cpp

{
public:
    virtual int load_param(const ParamDict& pd);// new code
    virtual int load_model(const ModelBin& mb);// new code

private:
    int channels;
    float eps;
    Mat gamma_data;
};

// new routine for loading parameters

private:
    float eps;
};

{
    // details about the relations with param file
    // https://github.com/Tencent/ncnn/wiki/param-and-model-file-structure
    //
    channels = pd.get(0, 0);// parse 0=<int value> entry, default value 0
    eps = pd.get(1, 0.001f);// parse 1=<float value> entry, default value 0.001f

    return 0;// return zero if success
}

// new routine for loading weights
int MyLayer::load_model(const ModelBin& mb)
{
    // details about the relations with model file
    //
    // read weights with length of channels * sizeof(float)
    // the second argument explains as follows
    // 0 judge the value type automatically, you may get float or float16 or uint8 etc

//
    //   depends on the model storage and the supporting target hardware
    // 1 read float values anyway
    // 2 read float16 values anyway
    // 3 read uint8 values anyway
    gamma_data = mb.load(channels, 1);
    if (gamma_data.empty())
        return -100;// return non-zero on error, -100 indicates out-of-memory

}
```

# step4 determine forward behavior
```cpp

{
public:
    MyLayer();// new code
    virtual int load_param(const ParamDict& pd);
    virtual int load_model(const ModelBin& mb);

private:
    float eps;
};

virtual int load_model(const ModelBin& mb);

private:
    float eps;
};

// new routine for setting forward behavior
MyLayer::MyLayer()
{
    // one input and one output
    // typical one_blob_only type: Convolution, Pooling, ReLU, Softmax ...
    // typical non-one_blob_only type: Eltwise, Split, Concat, Slice ...

    // do not change the blob size, modify data in-place
    // typical support_inplace type: ReLU, Sigmoid ...
    // typical non-support_inplace type: Convolution, Pooling ...
}

{
    channels = pd.get(0, 0);
    eps = pd.get(1, 0.001f);

    // you could alter the behavior based on loaded parameter
    // if (eps == 0.001f)
    // {
    //     one_blob_only = false;
    //     support_inplace = false;
    // }

    return 0;
}

// you could alter the behavior based on loaded parameter
    // {
    // }

    return 0;
}

{
        return -100;

    // you could alter the behavior based on loaded weight
    // if (gamma_data[0] == 0.f)
    // {
    // }

    return 0;
}
```

# step5 choose proper interface based on forward behavior
```cpp
// The base class Layer defines four interfaces for each forward behavior combination

// 1
virtual int forward(const std::vector<Mat>& bottom_blobs, std::vector<Mat>& top_blobs, const Option& opt) const;

// 2
virtual int forward(const Mat& bottom_blob, Mat& top_blob, const Option& opt) const;

// 3
virtual int forward_inplace(std::vector<Mat>& bottom_top_blobs, const Option& opt) const;

// 4

// 2

// 3

// 4
virtual int forward_inplace(Mat& bottom_top_blob, const Option& opt) const;
```
**must** = layer must implement this function

**optional** = layer may implement this function for optimal performance

sometimes the graph inference path cannot call forward_inplace directly due to data sharing, in this situation the non-inplace forward routine will be used, which deep-copy the input blob and call inplace forward on it if the optional routine is not implemented. Thus, you could avoid this deep-copy by process input to output on-the-fly.

|one_blob_only|support_inplace|1|2|3|4|
|false|false|must| | | |
|false|true|optional| |must| |
|true|false| |must| | |
|true|true| |optional| |must|

# step6 implement forward function
```cpp

```cpp

{
public:
    MyLayer();

    virtual int forward(const Mat& bottom_blob, Mat& top_blob, const Option& opt) const;// new code, optional
    virtual int forward_inplace(Mat& bottom_top_blob, const Option& opt) const;// new code

private:
    float eps;
};

{
}

{

    return 0;
}

{

{

    return 0;
}

{

    return 0;
}

// optional new routine for layer forward function, non-inplace version
int MyLayer::forward(const Mat& bottom_blob, Mat& top_blob, const Option& opt) const
{
    // check input dims, return non-zero on error
    if (bottom_blob.c != channels)
        return -1;

    // x = (x + eps) * gamma_per_channel

    int w = bottom_blob.w;
    int h = bottom_blob.h;
    size_t elemsize = bottom_blob.elemsize;
    int size = w * h;

    top_blob.create(w, h, channels, elemsize, opt.blob_allocator);
    if (top_blob.empty())

    #pragma omp parallel for num_threads(opt.num_threads)
    for (int q=0; q<channels; q++)
    {
        const float* ptr = bottom_blob.channel(q);

    {
        float* outptr = top_blob.channel(q);
        const float gamma = gamma_data[q];

        for (int i=0; i<size; i++)
        {
            outptr[i] = (ptr[i] + eps) * gamma ;
        }
    }

    return 0;
}

// new routine for layer forward function
int MyLayer::forward_inplace(Mat& bottom_top_blob, const Option& opt) const
{
    if (bottom_top_blob.c != channels)
        return -1;

    int w = bottom_top_blob.w;
    int h = bottom_top_blob.h;

    {
        float* ptr = bottom_top_blob.channel(q);

        {

for (int q=0; q<channels; q++)
    {

        {
            ptr[i] = (ptr[i] + eps) * gamma ;
        }
    }

    return 0;
}
```

# step7 integrate with ncnn library
you may probably need to modify caffe2ncnn or mxnet2ncnn etc. to write your layer specific parameters and weights into ncnn param and model file

the param and model file structure [param-and-model-file-structure](param-and-model-file-structure)

```
// example param file content
MyLayer          mylayer 1 1 conv2d mylayer0
Pooling          maxpool 1 1 mylayer0 maxpool 0=0 1=3 2=2 3=-233 4=0
```

```cpp

// register custom layer before load param and model
// the layer creator function signature is always XYZ_layer_creator, which defined in DEFINE_LAYER_CREATOR macro

```

```cpp

net.register_custom_layer("MyLayer", MyLayer_layer_creator);

net.load_param("model.param");
net.load_model("model.bin");
```

# 快速运行

本章将介绍 MMAction2 的基本功能。我们假设你已经[源码安装 MMAction2](installation.md#best-practices)。

- [快速运行](#快速运行)
  - [推理](#推理)
  - [修改配置](#修改配置)
    - [修改数据集](#修改数据集)
    - [修改运行配置](#修改运行配置)
    - [修改模型配置](#修改模型配置)

## 推理

在 MMAction2 的根目录下执行如下命令:

```shell
```

您应该能够看到弹出的视频窗口，和在控制台中打印的推断结果。

</div>
<br />

```bash
# 推理结果
```

```{note}
如果您在没有 GUI 的服务器上运行 MMAction2，或者通过禁用 X11 转发的 SSH 隧道运行 MMAction2，则可能不会看到弹出窗口。
```

关于 MMAction2 推理接口的详细描述可以在[这里](/demo/README.md#inferencer)找到.

```

```{note}
```

除了使用我们提供的预训练模型，您还可以在自己的数据集上训练模型。在下一节中，我们将通过在精简版 [Kinetics](https://download.openmmlab.com/mmaction/kinetics400_tiny.zip) 数据集上训练 TSN 为例，带您了解 MMAction2 的基本功能。

## 准备数据集

由于视频数据集格式的多样性不利于数据集的切换，MMAction2 提出了统一的[数据格式](../user_guides/prepare_dataset.md) ，并为常用的视频数据集提供了[数据集准备指南](../user_guides/data_prepare/dataset_prepare.md)。通常，要在 MMAction2 中使用这些数据集，你只需要按照步骤进行准备。

```{笔记}
但在这里，效率意味着一切。
```

首先，请下载我们预先准备好的 [kinetics400_tiny.zip](https://download.openmmlab.com/mmaction/kinetics400_tiny.zip) ，并将其解压到 MMAction2 根目录下的 `data/` 目录。这将为您提供必要的视频和注释文件。

```Bash
```

## 修改配置

准备好数据集之后，下一步是修改配置文件，以指定训练集和训练参数的位置。

```Bash
```

## 修改配置

在本例中，我们将使用 resnet50 作为主干网络来训练 TSN。由于 MMAction2 已经有了完整的 Kinetics400 数据集的配置文件 (`configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py`)，我们只需要在其基础上进行一些修改。

### 修改数据集

我们首先需要修改数据集的路径。打开 `configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py` ，按如下替换关键字:

```Python
```

### 修改运行配置

此外，由于数据集的大小减少，我们建议将训练批大小减少到4个，训练epoch的数量相应减少到10个。此外，我们建议将验证和权值存储间隔缩短为1轮，并修改学习率衰减策略。修改 `configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py` 中对应的关键字，如下所示生效。

```python
# 设置训练批大小为 4

```python

# 每轮都保存权重，并且只保留最新的权重
# 将最大 epoch 数设置为 10，并每 1 个 epoch验证模型
#根据 10 个 epoch调整学习率调度
    dict(
        begin=0,
        end=10,
]
```

### 修改模型配置

此外，由于精简版 Kinetics 数据集规模较小，建议加载原始 Kinetics 数据集上的预训练模型。此外，模型需要根据实际类别数进行修改。请直接将以下代码添加到 `configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py` 中。

```python
```

```

在这里，我们直接通过继承 ({external+mmengine:doc} `MMEngine: Config <advanced_tutorials/ Config>`) 机制重写了基本配置中的相应参数。原始字段分布在 `configs/_base_/models/tsn_r50.py`、`configs/_base_/schedules/sgd_100e.py` 和 `configs/_base_/default_runtime.py`中。

```{note}
关于配置的更详细的描述，请参考[这里](../user_guides/config.md)。
```

## 浏览数据集

在开始训练之前，我们还可以将训练时数据转换处理的帧可视化。这很简单：传递我们需要可视化的配置文件到 [browse_dataset.py](/tools/analysis_tools/browse_dataset.py)脚本中。

```Bash
```

转换后的视频将被保存到 `browse_out` 文件夹中。

</center>

```{note}
有关该脚本的参数和使用方法的详细信息，请参考[这里](../user_guides/useful_tools.md)。
```

```{tip}
除了满足我们的好奇心，可视化还可以帮助我们在训练前检查可能影响模型性能的部分，例如配置、数据集和数据转换中的问题。
```

我们可以通过以下脚本进一步可视化学习率调度，以确保配置符合预期:

```Bash

</center>

```{note}
```

```{tip}
```

```Bash
```

训练学习率时间表将显示在弹出窗口中。

</center>

```{note}
学习率根据实际批数据大小自动缩放。
```

## 训练

运行如下命令启动训练:

```Bash
```

根据系统环境，MMAction2 将自动使用最佳设备进行训练。如果有GPU，则默认启动单个GPU训练。当你开始看到 loss 的输出时，就说明你已经成功启动了训练。

```Bash

```Bash

```

在没有额外配置的情况下，模型权重将被保存到 `work_dirs/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb/`，而日志将被存储到 `work_dirs/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb/`。接下来，我们只需要耐心等待训练完成。

```{note}
训练的高级用法，如 CPU 训练、多卡训练及集群训练，请参考[training and Testing](../user_guides/train_test.md)
```

## 测试

```{note}
```

## 测试

经过 10 个 epoch 后，我们观察到 TSN 在第 6 个 epoch 表现最好，`acc/top1` 达到1.0000:

```Bash
```

```{note}
由于在原始 Kinetics400 上进行了预训练，结果非常高，您可能会看到不同的结果
```

然而，该值仅反映了 TSN 在精简版 Kinetics 数据集上的验证性能，而测试结果通常更高，因为在测试数据流水线中增加了更多的数据增强。

开始测试：

```Bash
```

并得到如下输出:

```Bash
```

该模型在该数据集上实现了 1.000 的 top1 准确率。

```{note}
测试的高级用法，如CPU测试、多gpu测试、集群测试，请参考[Training and testing](../user_guides/train_test.md)
```

# MViT V2

[MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](http://openaccess.thecvf.com//content/CVPR2022/papers/Li_MViTv2_Improved_Multiscale_Vision_Transformers_for_Classification_and_Detection_CVPR_2022_paper.pdf)

In this paper, we study Multiscale Vision Transformers (MViTv2) as a unified architecture for image and video classification, as well as object detection. We present an improved version of MViT that incorporates decomposed relative positional embeddings and residual pooling connections. We instantiate this architecture in five sizes and evaluate it for ImageNet classification, COCO detection and Kinetics video recognition where it outperforms prior work. We further compare MViTv2s' pooling attention to window attention mechanisms where it outperforms the latter in accuracy/compute. Without bells-and-whistles, MViTv2 has state-of-the-art performance in 3 domains: 88.8% accuracy on ImageNet classification, 58.7 boxAP on COCO object detection as well as 86.1% on Kinetics-400 video classification.

<img src="https://user-images.githubusercontent.com/33249023/196627033-03a4e9b1-082e-42ee-a2a0-77f874fe632a.png" width="50%"/>
</div>

1. Models with * in `Inference results` are ported from the repo [SlowFast](https://github.com/facebookresearch/SlowFast/) and tested on our data, and models in `Training results` are trained in MMAction2 on our data.
2. The values in columns named after `reference` are copied from paper, and `reference*` are results using [SlowFast](https://github.com/facebookresearch/SlowFast/) repo and trained on our data.

4. MaskFeat fine-tuning experiment is based on pretrain model from [MMSelfSup](https://github.com/open-mmlab/mmselfsup/tree/main/projects/maskfeat_video), and the corresponding reference result is based on pretrain model from [SlowFast](https://github.com/facebookresearch/SlowFast/).
5. Due to the different versions of Kinetics-400, our training results are different from paper.
6. Due to the training efficiency, we currently only provide MViT-small training results, we don't ensure other config files' training accuracy and welcome you to contribute your reproduction results.

7. We use `repeat augment` in MViT training configs following [SlowFast](https://github.com/facebookresearch/SlowFast/). [Repeat augment](https://arxiv.org/pdf/1901.09335.pdf) takes multiple times of data augment for one video, this way can improve the generalization of the model and relieve the IO stress of loading videos. And please note that the actual batch size is `num_repeats` times of `batch_size` in `train_dataloader`.

### Inference results

#### Kinetics-400

| frame sampling strategy | resolution |  backbone  |   pretrain   | top1 acc | top5 acc |        reference top1 acc        |        reference top5 acc        | testing protocol | FLOPs | params |        config        |        ckpt        |

| :---------------------: | :--------: | :--------: | :----------: | :------: | :------: | :------------------------------: | :------------------------------: | :--------------: | :---: | :----: | :------------------: | :----------------: |
|         16x4x1          |  224x224   | MViTv2-S\* | From scratch |   81.1   |   94.7   | [81.0](https://github.com/facebookresearch/SlowFast/blob/main/projects/mvitv2/README.md) | [94.6](https://github.com/facebookresearch/SlowFast/blob/main/projects/mvitv2/README.md) | 5 clips x 1 crop |  64G  | 34.5M  | [config](/configs/recognition/mvit/mvit-small-p244_32xb16-16x4x1-200e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/mvit/converted/mvit-small-p244_16x4x1_kinetics400-rgb_20221021-9ebaaeed.pth) |

|         32x3x1          |  224x224   | MViTv2-B\* | From scratch |   82.6   |   95.8   | [82.9](https://github.com/facebookresearch/SlowFast/blob/main/projects/mvitv2/README.md) | [95.7](https://github.com/facebookresearch/SlowFast/blob/main/projects/mvitv2/README.md) | 5 clips x 1 crop | 225G  | 51.2M  | [config](/configs/recognition/mvit/mvit-base-p244_32x3x1_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/mvit/converted/mvit-base-p244_32x3x1_kinetics400-rgb_20221021-f392cd2d.pth) |

|         40x3x1          |  312x312   | MViTv2-L\* | From scratch |   85.4   |   96.2   | [86.1](https://github.com/facebookresearch/SlowFast/blob/main/projects/mvitv2/README.md) | [97.0](https://github.com/facebookresearch/SlowFast/blob/main/projects/mvitv2/README.md) | 5 clips x 3 crop | 2828G |  213M  | [config](/configs/recognition/mvit/mvit-large-p244_40x3x1_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/mvit/converted/mvit-large-p244_40x3x1_kinetics400-rgb_20221021-11fe1f97.pth) |

#### Something-Something V2

|       uniform 16        |  224x224   | MViTv2-S\* |     K400     |   68.1   |   91.0   | [68.2](https://github.com/facebookresearch/SlowFast/blob/main/projects/mvitv2/README.md) | [91.4](https://github.com/facebookresearch/SlowFast/blob/main/projects/mvitv2/README.md) | 1 clips x 3 crop |  64G  | 34.4M  | [config](/configs/recognition/mvit/mvit-small-p244_k400-pre_16xb16-u16-100e_sthv2-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/mvit/converted/mvit-small-p244_u16_sthv2-rgb_20221021-65ecae7d.pth) |

|       uniform 32        |  224x224   | MViTv2-B\* |     K400     |   70.8   |   92.7   | [70.5](https://github.com/facebookresearch/SlowFast/blob/main/projects/mvitv2/README.md) | [92.7](https://github.com/facebookresearch/SlowFast/blob/main/projects/mvitv2/README.md) | 1 clips x 3 crop | 225G  | 51.1M  | [config](/configs/recognition/mvit/mvit-base-p244_u32_sthv2-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/mvit/converted/mvit-base-p244_u32_sthv2-rgb_20221021-d5de5da6.pth) |

|       uniform 40        |  312x312   | MViTv2-L\* | IN21K + K400 |   73.2   |   94.0   | [73.3](https://github.com/facebookresearch/SlowFast/blob/main/projects/mvitv2/README.md) | [94.0](https://github.com/facebookresearch/SlowFast/blob/main/projects/mvitv2/README.md) | 1 clips x 3 crop | 2828G |  213M  | [config](/configs/recognition/mvit/mvit-large-p244_u40_sthv2-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/mvit/converted/mvit-large-p244_u40_sthv2-rgb_20221021-61696e07.pth) |

### Training results

| frame sampling strategy | resolution | backbone |   pretrain    | top1 acc | top5 acc |     reference\* top1 acc      |      reference\* top5 acc      | testing protocol  | FLOPs | params |      config      |      ckpt      |      log      |

|         16x4x1          |  224x224   | MViTv2-S | From scratch  |   80.6   |   94.7   | [80.8](https://github.com/facebookresearch/SlowFast/blob/main/projects/mvitv2/README.md) | [94.6](https://github.com/facebookresearch/SlowFast/blob/main/projects/mvitv2/README.md) | 5 clips x 1 crop  |  64G  | 34.5M  | [config](/configs/recognition/mvit/mvit-small-p244_32xb16-16x4x1-200e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/mvit/mvit-small-p244_32xb16-16x4x1-200e_kinetics400-rgb/mvit-small-p244_32xb16-16x4x1-200e_kinetics400-rgb_20230201-23284ff3.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/mvit/mvit-small-p244_32xb16-16x4x1-200e_kinetics400-rgb/mvit-small-p244_32xb16-16x4x1-200e_kinetics400-rgb.log) |

|         16x4x1          |  224x224   | MViTv2-S | K400 MaskFeat |   81.8   |   95.2   | [81.5](https://github.com/facebookresearch/SlowFast/blob/main/projects/maskfeat/README.md) | [94.9](https://github.com/facebookresearch/SlowFast/blob/main/projects/maskfeat/README.md) | 10 clips x 1 crop |  71G  | 36.4M  | [config](/configs/recognition/mvit/mvit-small-p244_k400-maskfeat-pre_8xb32-16x4x1-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/mvit/mvit-small-p244_k400-maskfeat-pre_8xb32-16x4x1-100e_kinetics400-rgb/mvit-small-p244_k400-maskfeat-pre_8xb32-16x4x1-100e_kinetics400-rgb_20230201-5bced1d0.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/mvit/mvit-small-p244_k400-maskfeat-pre_8xb32-16x4x1-100e_kinetics400-rgb/mvit-small-p244_k400-maskfeat-pre_8xb32-16x4x1-100e_kinetics400-rgb.log) |

the corresponding result without repeat augment is as follows:

| frame sampling strategy | resolution | backbone |   pretrain   | top1 acc | top5 acc |                 reference\* top1 acc                 |                 reference\* top5 acc                 | testing protocol | FLOPs | params |
| :---------------------: | :--------: | :------: | :----------: | :------: | :------: | :--------------------------------------------------: | :--------------------------------------------------: | :--------------: | :---: | :----: |
|         16x4x1          |  224x224   | MViTv2-S | From scratch |   79.4   |   93.9   | [80.8](https://github.com/facebookresearch/SlowFast/blob/main/projects/mvitv2/README.md) | [94.6](https://github.com/facebookresearch/SlowFast/blob/main/projects/mvitv2/README.md) | 5 clips x 1 crop |  64G  | 34.5M  |

| frame sampling strategy | resolution | backbone | pretrain | top1 acc | top5 acc |      reference top1 acc       |       reference top5 acc       | testing protocol | FLOPs | params |       config       |       ckpt       |       log       |
| :---------------------: | :--------: | :------: | :------: | :------: | :------: | :---------------------------: | :----------------------------: | :--------------: | :---: | :----: | :----------------: | :--------------: | :-------------: |

|       uniform 16        |  224x224   | MViTv2-S |   K400   |   68.2   |   91.3   | [68.2](https://github.com/facebookresearch/SlowFast/blob/main/projects/mvitv2/README.md) | [91.4](https://github.com/facebookresearch/SlowFast/blob/main/projects/mvitv2/README.md) | 1 clips x 3 crop |  64G  | 34.4M  | [config](/configs/recognition/mvit/mvit-small-p244_k400-pre_16xb16-u16-100e_sthv2-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/mvit/mvit-small-p244_k400-pre_16xb16-u16-100e_sthv2-rgb/mvit-small-p244_k400-pre_16xb16-u16-100e_sthv2-rgb_20230201-4065c1b9.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/mvit/mvit-small-p244_k400-pre_16xb16-u16-100e_sthv2-rgb/mvit-small-p244_k400-pre_16xb16-u16-100e_sthv2-rgb.log) |

## Test

```shell

## Test

```shell
```

Example: test MViT model on Kinetics-400 dataset and dump the result to a pkl file.

```shell
python tools/test.py configs/recognition/mvit/mvit-small-p244_16x4x1_kinetics400-rgb.py \
```

```bibtex
}
```

# FAQ

There is probably a cached mmengine in your local host. Try to install its latest version.

```shell
```

It may have been caused by the following reasons.

1. You haven't installed lmdeploy's precompiled package. `_turbomind` is the pybind package of c++ turbomind, which involves compilation. It is recommended that you install the precompiled one.

```shell
```

2. If you have installed it and still encounter this issue, it is probably because you are executing turbomind-related command in the root directory of lmdeploy source code. Switching to another directory will fix it

## Libs

Make sure you have install lmdeploy (>=v0.0.5) through `pip install lmdeploy`.

If the issue still exists after lmdeploy installation, add the path of `libnccl.so.2` to environment variable LD_LIBRARY_PATH.

```shell

```shell
# Get the location of nvidia-nccl-cu11 package
# insert the path of "libnccl.so.2" to LD_LIBRARY_PATH
```

It's probably due to a low-version cuda toolkit. LMDeploy runtime requires a minimum CUDA version of 11.2

## Turbomind Inference

## Pytorch Inference

## Serve

# Preparing REDS Dataset

```bibtex
}
```

- Training dataset: [REDS dataset](https://seungjunnah.github.io/Datasets/reds.html).
- Validation dataset: [REDS dataset](https://seungjunnah.github.io/Datasets/reds.html) and Vid4.

Note that we merge train and val datasets in REDS for easy switching between REDS4 partition (used in EDVR) and the official validation partition.
The original val dataset (clip names from 000 to 029) are modified to avoid conflicts with training dataset (total 240 clips). Specifically, the clip names are changed to 240, 241, ... 269.

You can prepare the REDS dataset by running:

```shell
python tools/dataset_converters/reds/preprocess_reds_dataset.py --root-path ./data/REDS
```

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── REDS
│   │   │   ├── 000
│   │   │   ├── 001
│   │   │   ├── ...
│   │   │   ├── X4
│   ├── REDS4
│   │   ├── GT
```

## Prepare LMDB dataset for REDS

```shell

│   ├── REDS4
│   │   ├── GT
```

```shell
```

## Crop to sub-images

MMagic also supports cropping REDS images to sub-images for faster IO. We provide such a script:

```shell
```

The generated data is stored under `REDS` and the data structure is as follows, where `_sub` indicates the sub-images.

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── REDS
│   │   │   ├── 000
│   │   │   ├── 001
│   │   │   ├── ...
│   │   │   ├── ...
│   │   │   ├── ...
│   │   │   ├── X4

│   │   │   ├── ...
│   │   │   ├── ...
│   │   │   ├── X4
│   │   │   ├── ...
│   │   │   ├── ...
```

Note that by default `preprocess_reds_dataset.py` does not make lmdb and annotation file for the cropped dataset. You may need to modify the scripts a little bit for such operations.

<!--
Title (above): please place [branch_name] at the beginning if you are targeting a branch other than master. *Do not target stable*.
It is recommended to use conventional commit format, see conventionalcommits.org, but not required.
-->

<!-- Include relevant issues or PRs here, describe what changed and why -->

## Suggested changelog entry:

<!-- Fill in the below block with the expected RestructuredText entry. Delete if no entry needed;
     but do not delete header or rst block if an entry is needed! Will be collected via a script. -->

```rst

```

<!-- If the upgrade guide needs updating, note that here too -->

# Vision-Based 3D Detection

Vision-based 3D detection refers to the 3D detection solutions based on vision-only input, such as monocular, binocular, and multi-view image based 3D detection.
Currently, we only support monocular and multi-view 3D detection methods. Other approaches should be also compatible with our framework and will be supported in the future.

It expects the given model to take any number of images as input, and predict the 3D bounding boxes and category labels for each object of interest.
Taking FCOS3D on the nuScenes dataset as an example, we will show how to prepare data, train and test a model on a standard 3D detection benchmark, and how to visualize and validate the results.

To begin with, we need to download the raw data and reorganize the data in a standard way presented in the [doc for data preparation](https://mmdetection3d.readthedocs.io/en/latest/data_preparation.html).

For example, for nuScenes we need to run:

```
```

```
├── tools
├── data
│   │   ├── maps
│   │   ├── nuscenes_infos_trainval.pkl
```

```

Then let us train a model with provided configs for FCOS3D. The basic script is the same as other models.
You can basically follow the examples provided in this [tutorial](https://mmdetection3d.readthedocs.io/en/latest/1_exist_data_model.html#inference-with-existing-models) when training with different GPU settings.

```
./tools/dist_train.sh configs/fcos3d/fcos3d_r101-caffe-dcn_fpn_head-gn_8xb2-1x_nus-mono3d.py 8
```

Note that `8xb2` in the config name refers to the training is completed with 8 GPUs and 2 data samples on each GPU.
If your customized setting is different from this, you should add `--auto-scale-lr` to enable automatically scaling learning rate. A basic rule can be referred to [here](https://arxiv.org/abs/1706.02677).

We can also achieve better performance with finetuned FCOS3D by running:

```
./tools/dist_train.sh configs/fcos3d/fcos3d_r101-caffe-dcn_fpn_head-gn_8xb2-1x_nus-mono3d_finetune.py 8
```

After training a baseline model with the previous script,
please remember to modify the path [here](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/fcos3d/fcos3d_r101-caffe-dcn_fpn_head-gn_8xb2-1x_nus-mono3d_finetune.py#L8) correspondingly.

Due to the output format is the same as 3D detection based on other modalities, the evaluation methods are also the same.

For nuScenes, the model will be evaluated with distance-based mean AP (mAP) and NuScenes Detection Score (NDS) for 10 categories respectively.

```

```

```

```
./tools/dist_test.sh configs/fcos3d/fcos3d_r101-caffe-dcn_fpn_head-gn_8xb2-1x_nus-mono3d.py work_dirs/fcos3d/latest.pth 8
```

you just need to specify the `jsonfile_prefix` for corresponding evaluator,
e.g., add `test_evaluator = dict(type='NuscenesMetric', jsonfile_prefix=work_dirs/fcos3d/test_submission)` in the configuration then you can get the results file.

Please guarantee the `data_prefix` and `ann_file` in [info for testing](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/datasets/nus-mono3d.py#L93) in the config corresponds to the test set instead of validation set.

After generating the results, you can basically compress the folder and upload to the evalAI evaluation server for nuScenes 3D detection challenge.

You can either set the `--eval-options 'show=True' 'out_dir=${SHOW_DIR}'` option to visualize the detection results online during evaluation,

Note that currently we only support the visualization on images for vision-only methods.
The visualization in the perspective view and bird-eye-view (BEV) will be integrated in the future.

# Scripts for developing MMagic

- [1. Check UT](#1-check-ut)
- [2. Test all the models](#2-test-all-the-models)
- [3. Train all the models](#3-train-all-the-models)
  - [3.1 Train for debugging](#31-train-for-debugging)
  - [3.2 Train for FP32](#32-train-for-fp32)
  - [3.3 Train for FP16](#33-train-for-fp16)
- [4. Monitor your training](#4-monitor-your-training)
- [5. Train with a list of models](#5-train-with-a-list-of-models)
- [6. Train with skipping a list of models](#6-train-with-skipping-a-list-of-models)
- [7. Train failed or canceled jobs](#7-train-failed-or-canceled-jobs)
- [8. Deterministic training](#8-deterministic-training)
- [9. Automatically check links](#9-automatically-check-links)
- [10. Calculate flops](#10-calculate-flops)
- [11. Update model idnex](#11-update-model-index)

## 1. Check UT

Please check your UT by the following scripts:

```python
cd mmagic/
python .dev_script/update_ut.py
```

Then, you will find some redundant UT, missing UT and blank UT.

```python
cd mmagic/
```

Please create UTs according to your package code implementation.

## 2. Test all the models

Please follow these steps to test all the models in MMagic:

First, you will need download all the pre-trained checkpoints by:

```shell
python .dev_scripts/download_models.py
```

Then, you can start testing all the benchmarks by：

```shell
python .dev_scripts/test_benchmark.py
```

## 3. Train all the models

### 3.1 Train for debugging

In order to test all the pipelines of training, visualization, etc., you may want to set the total iterations of all the models as less steps (e.g., 100 steps) for quick evaluation. You can use the following steps:

First, since our datasets are stored in ceph, you need to create ceph configs.

```shell
# create configs
python .dev_scripts/create_ceph_configs.py \

```shell
        --target-dir configs_ceph_debug \
        --gpus-per-job 2 \
        --iters 100 \
        --save-dir-prefix work_dirs/benchmark_debug \
        --work-dir-prefix work_dirs/benchmark_debug
```

If you only want to update a specific config file, you can specify it by `--test-file configs/aot_gan/aot-gan_smpgan_4xb4_places-512x512.py`.

Here, `--target-dir` denotes the path of new created configs, `--gpus-per-job` denotes the numbers of gpus used for each job, `--iters` denotes the total iterations of each model, `--save-dir-prefix` and `--work-dir-prefix` denote the working directory, where you can find the working logging.

Then, you will need to submit all the jobs by running `train_benchmark.py`.

```shell
python .dev_scripts/train_benchmark.py mm_lol \
    --config-dir configs_ceph_debug \
    --run \
    --gpus-per-job 2 \

```shell
    --run \
    --job-name debug \
    --work-dir work_dirs/benchmark_debug \
    --resume \
    --quotatype=auto
```

Here, you will specify the configs files used for training by `--config-dir`, submit all the jobs to run by set `--run`. You can set the prefix name of the submitted jobs by `--job-name`, specify the working directory by `--work-dir`. We suggest using `--resume` to enable auto resume during training and `--quotatype=auto` to fully exploit all the computing resources.

### 3.2 Train for FP32

If you want to train all the models with FP32 (i.e, regular settings as the same with `configs/`),
you can follow these steps:

```shell
# create configs for fp32
        --target-dir configs_ceph_fp32 \
        --gpus-per-job 4 \
        --save-dir-prefix work_dirs/benchmark_fp32 \

```shell
        --work-dir-prefix work_dirs/benchmark_fp32 \
```

Then, submit the jobs to run by slurm:

```shell
    --config-dir configs_ceph_fp32 \
    --run \
    --resume \
    --gpus-per-job 4 \
    --job-name fp32 \
    --work-dir work_dirs/benchmark_fp32 \
```

### 3.3 Train for FP16

You will also need to train the models with AMP (i.e., FP16), you can use the following steps to achieve this:

```shell
        --target-dir configs_ceph_amp \
        --save-dir-prefix work_dirs/benchmark_amp \
        --work-dir-prefix work_dirs/benchmark_amp
```

Then, submit the jobs to run:

```shell

--save-dir-prefix work_dirs/benchmark_amp \
```

```shell
    --config-dir configs_ceph_amp \
    --run \
    --resume \
    --amp \
    --job-name amp \
    --work-dir work_dirs/benchmark_amp \
```

## 4. Monitor your training

After you submitting jobs following [3-Train-all-the-models](#3-train-all-the-models), you will find a `xxx.log` file.
This log file list all the job name of job id you have submitted. With this log file, you can monitor your training by running `.dev_scripts/job_watcher.py`.

For example, you can run

```shell
python .dev_scripts/job_watcher.py --work-dir work_dirs/benchmark_fp32/ --log 20220923-140317.log
```

Then, you will find `20220923-140317.csv`, which reports the status and recent log of each job.

## 5. Train with a list of models

```

If you only need to run some of the models, you can list all the models' name in a file, and specify the models when using `train_benchmark.py`.

```shell
    --run \
    --resume \
    --quotatype=auto \
    --rerun \
    --rerun-list 20220923-140317.log \
```

Specifically, you need to enable `--rerun`, and specify the list of models to rerun by `--rerun-list`

## 6. Train with skipping a list of models

If you want to train all the models while skipping some models, you can also list all the models' name in a file, and specify the models when running `train_benchmark.py`.

```shell

```shell
    --run \
    --resume \
    --skip \
    --skip-list 20220923-140317.log \
```

Specifically, you need to enable `--skip`, and specify the list of models to skip by `--skip-list`

## 7. Train failed or canceled jobs

If you want to rerun failed or canceled jobs in the last run, you can combine `--rerun` flag with `--rerun-failure` and `--rerun-cancel` flags.

For example, the log file of the last run is `train-20221009-211904.log`, and now you want to rerun the failed jobs. You can use the following command:

```bash
    --job-name RERUN \
    --rerun train-20221009-211904.log \
    --rerun-fail \
    --run
```

```bash
    --run
```

We can combine `--rerun-fail` and `--rerun-cancel` with flag `---models` to rerun a **subset** of failed or canceled model.

```bash
    --models sagan \  # only rerun 'sagan' models in all failed tasks
    --run
```

Specifically, `--rerun-fail` and `--rerun-cancel` can be used together to rerun both failed and cancaled jobs.

## 8. `deterministic` training

Set `torch.backends.cudnn.deterministic = True` and `torch.backends.cudnn.benchmark = False` can remove randomness operation in Pytorch training. You can add `--deterministic` flag when start your benchmark training to remove the influence of randomness operation.

```shell

```shell
python .dev_scripts/train_benchmark.py mm_lol --job-name xzn --models pix2pix --cpus-per-job 16 --run --deterministic
```

## 9. Automatically check links

Use the following script to check whether the links in documentations are valid:

```shell
python .dev_scripts/doc_link_checker.py --target docs/zh_cn
python .dev_scripts/doc_link_checker.py --target README_zh-CN.md
python .dev_scripts/doc_link_checker.py --target docs/en
python .dev_scripts/doc_link_checker.py --target README.md
```

You can specify the `--target` by a file or a directory.

**Notes:** DO NOT use it in CI, because requiring too many http requirements by CI will cause 503 and CI will propabaly fail.

## 10. Calculate flops

To summarize the flops of different models, you can run the following commands:

```bash
python .dev_scripts/benchmark_valid_flop.py --flops --flops-str
```

## 11. Update model index

To update model-index according to `README.md`, please run the following commands,

```bash

```bash
```

```bash
python .dev_scripts/update_model_index.py
```

# 概述

## 什么是 MMAction2

MMAction2 是一个基于 PyTorch 的开源工具包，支持了大量的视频理解模型，包括**行为识别、基于骨架的行为识别、时空行为检测和时序动作定位**等多个主要方向。它还支持了大多数流行的学术数据集，并提供了许多实用工具帮助用户对数据集和模型进行多方面的探索和调试。它具有以下特点：

**全流程，多模型**：MMAction2 支持各种视频理解任务，实现了最先进的行为识别、定位、检测模型。

**模块化设计**：MMAction2 的模块化设计使用户可以根据需要定义和重用模型中的模块。

**实用工具众多**：MMAction2 提供了一系列的分析工具，如可视化器、验证脚本、评估器等，以帮助用户进行故障排除、微调或比较模型。

**由 OpenMMLab 强力驱动**：与家族内的其它算法库一样，MMAction2 遵循着 OpenMMLab 严谨的开发准则和接口约定，极大地降低了用户切换各算法库时的学习成本。同时，MMAction2 也可以非常便捷地与家族内其他算法库跨库联动，从而满足用户跨领域研究和落地的需求。

    <p style="text-align: center;">行为识别</p></td>
    <p style="text-align: center;">基于骨架的行为识别</p></td>

<p style="text-align: center;">基于骨架的行为识别</p></td>
    <p style="text-align: center;">时空动作检测</p></td>

## 如何使用文档

<summary><b> MMAction2 的基础用法</b></summary>

- [安装](installation.md)
- [快速运行](quick_run.md)
- [利用现有模型进行推理](../user_guides/inference.md)

<summary><b>关于在已支持的数据集上进行训练</b></summary>

- [了解配置文件](../user_guides/config.md)
- [准备数据集](../user_guides/prepare_dataset.md)
- [训练与测试](../user_guides/train_test.md)

<summary><b>关于使用过程中的常见问题</b></summary>

- [常见问题解答](faq.md)
- [有用的工具](../useful_tools.md)

<summary><b>关于 MMAction2 的框架设计</b></summary>

- [20分钟 MMAction2 框架指南](guide_to_framework.md)
- [MMAction2 中的数据流](../advanced_guides/dataflow.md)

<summary><b>关于自定义训练的高级用法</b></summary>

- [自定义模型](../advanced_guides/customize_models.md)
- [自定义数据集](../advanced_guides/customize_dataset.md)
- [自定义数据管道](../advanced_guides/customize_pipeline.md)
- [自定义优化器](../advanced_guides/customize_optimizer.md)
- [自定义日志记录](../advanced_guides/customize_logging.md)

<summary><b>关于支持的模型库和数据集</b></summary>

- [模型库](../modelzoo_statistics.md)
- [数据集](../datasetzoo_statistics.md)

<summary><b>关于从 MMAction2 0.x 迁移</b></summary>

- [从 MMAction2 0.x 迁移](../migration.md)

<summary><b>对于希望加入开源社区，向 MMAction2 贡献代码的研究者和开发者</b></summary>

- [如何为 MMAction2 做出贡献](contribution_guide.md)

# 准备 Paris Street View 数据集

```bibtex
}
```

请从[此处](https://github.com/pathak22/context-encoder/issues/24)获取数据集。

```text
mmagic
├── mmagic
├── tools
├── data
|   |   ├── val

```

# SparK

> [Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling](https://arxiv.org/abs/2301.03580)

We identify and overcome two key obstacles in extending the success of BERT-style pre-training, or the masked image modeling, to convolutional networks (convnets): (i) convolution operation cannot handle irregular, random-masked input images; (ii) the single-scale nature of BERT pre-training is inconsistent with convnet's hierarchical structure. For (i), we treat unmasked pixels as sparse voxels of 3D point clouds and use sparse convolution to encode. This is the first use of sparse convolution for 2D masked modeling. For (ii), we develop a hierarchical decoder to reconstruct images from multi-scale encoded features. Our method called Sparse masKed modeling (SparK) is general: it can be used directly on any convolutional model without backbone modifications. We validate it on both classical (ResNet) and modern (ConvNeXt) models: on three downstream tasks, it surpasses both state-of-the-art contrastive learning and transformer-based masked modeling by similarly large margins (around +1.0%). Improvements on object detection and instance segmentation are more substantial (up to +3.5%), verifying the strong transferability of features learned. We also find its favorable scaling behavior by observing more gains on larger models. All this evidence reveals a promising future of generative pre-training on convnets. Codes and models are released at https://github.com/keyu-tian/SparK.

<img src="https://github.com/open-mmlab/mmpretrain/assets/36138628/b93e8d6f-ec1e-4f27-b986-da470fabe7df" width="80%"/>
</div>

```python

predict = inference_model('resnet50_spark-pre_300e_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('spark_sparse-resnet50_800e_in1k', pretrained=True)
```

Train:

```shell
python tools/train.py configs/spark/spark_sparse-resnet50_8xb512-amp-coslr-800e_in1k.py
```

Test:

```shell

Train:

```shell
```

Test:

```shell
python tools/test.py configs/spark/benchmarks/resnet50_8xb256-coslr-300e_in1k.py https://download.openmmlab.com/mmpretrain/v1.0/spark/spark_sparse-resnet50_8xb512-amp-coslr-800e_in1k/resnet50_8xb256-coslr-300e_in1k/resnet50_8xb256-coslr-300e_in1k_20230612-f86aab51.pth
```

| Model                                    | Params (M) | Flops (G) |                                Config                                 |                                 Download                                 |
| :--------------------------------------- | :--------: | :-------: | :-------------------------------------------------------------------: | :----------------------------------------------------------------------: |

| `spark_sparse-resnet50_800e_in1k`        |   37.97    |   4.10    |     [config](spark_sparse-resnet50_8xb512-amp-coslr-800e_in1k.py)     | [model](https://download.openmmlab.com/mmpretrain/v1.0/spark/spark_sparse-resnet50_8xb512-amp-coslr-800e_in1k/spark_sparse-resnet50_8xb512-amp-coslr-800e_in1k_20230612-e403c28f.pth) \| [log](https://download.openmmlab.com/mmpretrain/v1.0/spark/spark_sparse-resnet50_8xb512-amp-coslr-800e_in1k/spark_sparse-resnet50_8xb512-amp-coslr-800e_in1k_20230612-e403c28f.json) |

| `spark_sparse-convnextv2-tiny_800e_in1k` |   39.73    |   4.47    | [config](spark_sparse-convnextv2-tiny_16xb256-amp-coslr-800e_in1k.py) | [model](https://download.openmmlab.com/mmpretrain/v1.0/spark/spark_sparse-convnextv2-tiny_16xb256-amp-coslr-800e_in1k/spark_sparse-convnextv2-tiny_16xb256-amp-coslr-800e_in1k_20230612-b0ea712e.pth) \| [log](https://download.openmmlab.com/mmpretrain/v1.0/spark/spark_sparse-convnextv2-tiny_16xb256-amp-coslr-800e_in1k/spark_sparse-convnextv2-tiny_16xb256-amp-coslr-800e_in1k_20230612-b0ea712e.json) |

| Model                                 |                  Pretrain                  | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                  Config                   |                  Download                   |

| :------------------------------------ | :----------------------------------------: | :--------: | :-------: | :-------: | :-------: | :---------------------------------------: | :-----------------------------------------: |
| `resnet50_spark-pre_300e_in1k`        | [SPARK](https://download.openmmlab.com/mmpretrain/v1.0/spark/spark_sparse-resnet50_8xb512-amp-coslr-800e_in1k/spark_sparse-resnet50_8xb512-amp-coslr-800e_in1k_20230612-e403c28f.pth) |   23.52    |   1.31    |   80.10   |   94.90   | [config](benchmarks/resnet50_8xb256-coslr-300e_in1k.py) | [model](https://download.openmmlab.com/mmpretrain/v1.0/spark/spark_sparse-resnet50_8xb512-amp-coslr-800e_in1k/resnet50_8xb256-coslr-300e_in1k/resnet50_8xb256-coslr-300e_in1k_20230612-f86aab51.pth) \| [log](https://download.openmmlab.com/mmpretrain/v1.0/spark/spark_sparse-resnet50_8xb512-amp-coslr-800e_in1k/resnet50_8xb256-coslr-300e_in1k/resnet50_8xb256-coslr-300e_in1k_20230612-f86aab51.json) |

| `convnextv2-tiny_spark-pre_300e_in1k` | [SPARK](https://download.openmmlab.com/mmpretrain/v1.0/spark/spark_sparse-convnextv2-tiny_16xb256-amp-coslr-800e_in1k/spark_sparse-convnextv2-tiny_16xb256-amp-coslr-800e_in1k_20230612-b0ea712e.pth) |   28.64    |   4.47    |   82.80   |   96.30   | [config](benchmarks/convnextv2-tiny_8xb256-coslr-300e_in1k.py) | [model](https://download.openmmlab.com/mmpretrain/v1.0/spark//spark_sparse-convnextv2-tiny_16xb256-amp-coslr-800e_in1k/convnextv2-tiny_8xb256-coslr-300e_in1k/convnextv2-tiny_8xb256-coslr-300e_in1k_20230612-ffc78743.pth) \| [log](https://download.openmmlab.com/mmpretrain/v1.0/spark//spark_sparse-convnextv2-tiny_16xb256-amp-coslr-800e_in1k/convnextv2-tiny_8xb256-coslr-300e_in1k/convnextv2-tiny_8xb256-coslr-300e_in1k_20230612-ffc78743.json) |

```bibtex
@Article{tian2023designing,
  author  = {Keyu Tian and Yi Jiang and Qishuai Diao and Chen Lin and Liwei Wang and Zehuan Yuan},

```bibtex
  title   = {Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling},
  journal = {arXiv:2301.03580},
  year    = {2023},
}
```

# SASM

> [Shape-Adaptive Selection and Measurement for Oriented Object Detection](https://www.aaai.org/AAAI22Papers/AAAI-2171.HouL.pdf)

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/sasm.jpg" width="800"/>
</div>

The development of detection methods for oriented object detection remains a challenging task. A considerable obstacle
is the wide variation in the shape (e.g., aspect ratio) of objects. Sample selection in general object detection has been
widely studied as it plays a crucial role in the performance of the detection method and has achieved great progress.
However, existing sample selection strategies still overlook some issues: (1) most of them ignore the object shape information;
(2) they do not make a potential distinction between selected positive samples; and (3) some of them can only be applied

to either anchor-free or anchor-based methods and cannot be used for both of them simultaneously. In this paper, we
propose novel flexible shape-adaptive selection (SA-S) and shape-adaptive measurement (SA-M) strategies for oriented
object detection, which comprise an SA-S strategy for sample selection and SA-M strategy for the quality estimation of
positive samples. Specifically, the SA-S strategy dynamically selects samples according to the shape information and
characteristics distribution of objects. The SA-M strategy measures the localization potential and adds quality information
on the selected positive samples. The experimental results on both anchor-free and anchor-based baselines and four publicly
available oriented datasets (DOTA, HRSC2016, UCASAOD, and ICDAR2015) demonstrate the effectiveness of the proposed method

DOTA1.0

DOTA1.0

#### RepPoints

| ResNet50 (1024,1024,200) | 66.45 |  oc   |   1x    |   3.53   |      15.3      |  -  |     2      |             [sasm-reppoints-qbox_r50_fpn_1x_dota](./sasm-reppoints-qbox_r50_fpn_1x_dota.py)              |                    [model](https://download.openmmlab.com/mmrotate/v0.1.0/sasm/sasm_reppoints_r50_fpn_1x_dota_oc/sasm_reppoints_r50_fpn_1x_dota_oc-6d9edded.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/sasm/sasm_reppoints_r50_fpn_1x_dota_oc/sasm_reppoints_r50_fpn_1x_dota_oc_20220205_144938.log.json)                    |

```
@inproceedings{hou2022shape,
    title={Shape-Adaptive Selection and Measurement for Oriented Object Detection},
    author={Hou, Liping and Lu, Ke and Xue, Jian and Li, Yuqiu},
}

```

# 2D Human Whole-Body Pose Estimation

2D human whole-body pose estimation aims to localize dense landmarks on the entire human body including face, hands, body, and feet.

Existing approaches can be categorized into top-down and bottom-up approaches.

Top-down methods divide the task into two stages: human detection and whole-body pose estimation. They perform human detection first, followed by single-person whole-body pose estimation given human bounding boxes.

Bottom-up approaches (e.g. AE) first detect all the whole-body keypoints and then group/associate them into person instances.

Please follow [DATA Preparation](/docs/en/dataset_zoo/2d_wholebody_keypoint.md) to prepare data.

## Demo

Please follow [Demo](/demo/docs/en/2d_wholebody_pose_demo.md) to run demos.

<img src="https://user-images.githubusercontent.com/9464825/95552839-00a61080-0a40-11eb-818c-b8dad7307217.gif" width="600px" alt><br>

# 实用工具 (待更新)

除了训练和测试脚本，我们在 `tools/` 文件夹内还提供了一些有用的工具。

## 日志分析

`tools/analysis_tools/analyze_logs.py` 通过给定的日志文件绘制 loss/mAP 曲线。 需首先执行 `pip install seaborn` 安装依赖。

```shell
```

示例:

- 绘制某次执行的分类损失

  ```shell
  ```

- 绘制某次执行的分类和回归损失，同时将图像保存到 pdf 文件

  ```shell
  ```

- 在同一张图像中比较两次执行的 mAP

  ```shell
  ```

- 计算平均训练速度

  ```shell

```

  ```shell
  ```

- 计算平均训练速度

  ```shell
  ```

  预计输出如下

  ```text
  ```

## 可视化

### 可视化数据集

`tools/misc/browse_dataset.py` 帮助用户浏览检测的数据集（包括图像和检测框的标注），或将图像保存到指定目录。

```shell
```

## 模型部署

为了使用 [`TorchServe`](https://pytorch.org/serve/) 部署一个 `MMRotate` 模型，需要进行以下几步：

### 1. 转换 MMRotate 模型至 TorchServe

```shell

```

## 模型部署

```shell
```

示例:

```shell

```

**Note**: ${MODEL_STORE} 需要是一个文件夹的绝对路径。

### 2. 构建 `mmrotate-serve` docker 镜像

```shell
```

### 3. 运行 `mmrotate-serve` 镜像

```

```shell
```

为了使镜像能够使用 GPU 资源，需要安装 [nvidia-docker](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html)。之后可以传递 `--gpus` 参数以在 GPU 上运行。

示例：

```shell
--cpus 8 \
```

### 4. 测试部署

```shell
```

### 4. 测试部署

```shell
```

```json
[
  {
    "bbox": [
    ],
  },
  {
    "bbox": [
    ],
  },
  {
    "bbox": [
    ],
  },
  {
    "bbox": [

0.007036328315734863
    ],
  },
  {
    "bbox": [
    ],
  }
]
```

```shell
```

示例：

```shell
```

## 模型复杂度

`tools/analysis_tools/get_flops.py` 是改编自 [flops-counter.pytorch](https://github.com/sovrasov/flops-counter.pytorch) 的脚本，用于计算给定模型的 FLOPs 和参数量。

```shell

```

## 模型复杂度

```shell
```

预计输出如下

```text
```

**注意**: 此工具仍处于实验阶段，我们并不能保证计算结果是绝对正确的。你可以将结果用于简单的比较，
但在技术报告或论文中采用之前请仔细检查

1. FLOPs 与输入大小相关，但参数量与其无关。默认输入大小是(1, 3, 1024, 1024)。
2. 一些算子例如 DCN 或自定义算子并未包含在 FLOPs 计算中，所以 S<sup>2</sup>A-Net 和基于 RepPoints 的模型的 FLOPs 计算是错误的。
   详细信息请查看 [`mmcv.cnn.get_model_complexity_info()`](https://github.com/open-mmlab/mmcv/blob/master/mmcv/cnn/utils/flops_counter.py)。
3. 两阶段检测器的 FLOPs 取决于候选的数量。

### 准备发布模型

`tools/model_converters/publish_model.py` 帮助用户准备他们将发布的模型。

在将模型上传到 AWS 之前，你可能需要

1. 将模型权重转换至 CPU
2. 删除优化器的状态
3. 计算权重文件的哈希值并附加到文件名后

```shell

### 准备发布模型

```shell
```

例如,

```shell
```

最终输出的文件名是 `rotated_faster_rcnn_r50_fpn_1x_dota_le90_20190801-{hash id}.pth`。

## 基准测试

### FPS 基准

`tools/analysis_tools/benchmark.py` 帮助用户计算 FPS。 FPS 值包括模型前向传播和后处理。为了得到更准确的数值，目前只支持单 GPU 分布式启动。

```shell
```

示例: 假设你已经下载了 `Rotated Faster R-CNN` 模型权重到 `checkpoints/` 文件夹

```shell

[--max-iter ${MAX_ITER}] \
```

```shell
```

## 杂项

### 打印完整配置文件

`tools/misc/print_config.py` 输出整个配置文件并整合其所有导入。

```shell
```

## 混淆矩阵

混淆矩阵是预测结果的概要

`tools/analysis_tools/confusion_matrix.py` 可以分析预测结果并绘制混淆矩阵。

首先执行 `tools/test.py` 将检测结果保存为 `.pkl` 文件。

之后执行

```
```

你会得到一个类似于下图的混淆矩阵:

之后执行

```
```

# 在标准数据集上训练预定义的模型

MMDetection 也为训练检测模型提供了开盖即食的工具。本节将展示在标准数据集（比如 COCO）上如何训练一个预定义的模型。

### 数据集

训练需要准备好数据集，细节请参考 [数据集准备](#%E6%95%B0%E6%8D%AE%E9%9B%86%E5%87%86%E5%A4%87) 。

**注意**：
目前，`configs/cityscapes` 文件夹下的配置文件都是使用 COCO 预训练权值进行初始化的。如果网络连接不可用或者速度很慢，你可以提前下载现存的模型。否则可能在训练的开始会有错误发生。

### 学习率自动缩放

**注意**：在配置文件中的学习率是在 8 块 GPU，每块 GPU 有 2 张图像（批大小为 8\*2=16）的情况下设置的。其已经设置在 `config/_base_/schedules/schedule_1x.py` 中的 `auto_scale_lr.base_batch_size`。学习率会基于批次大小为 `16`时的值进行自动缩放。同时，为了不影响其他基于 mmdet 的 codebase，启用自动缩放标志 `auto_scale_lr.enable` 默认设置为 `False`。

如果要启用此功能，需在命令添加参数 `--auto-scale-lr`。并且在启动命令之前，请检查下即将使用的配置文件的名称，因为配置名称指示默认的批处理大小。
在默认情况下，批次大小是 `8 x 2 = 16`，例如：`faster_rcnn_r50_caffe_fpn_90k_coco.py` 或者 `pisa_faster_rcnn_x101_32x4d_fpn_1x_coco.py`；若不是默认批次，你可以在配置文件看到像 `_NxM_` 字样的，例如：`cornernet_hourglass104_mstest_32x3_210e_coco.py` 的批次大小是 `32 x 3 = 96`, 或者 `scnet_x101_64x4d_fpn_8x1_20e_coco.py` 的批次大小是 `8 x 1 = 8`。

**请记住：如果使用不是默认批次大小为 `16`的配置文件，请检查配置文件中的底部，会有 `auto_scale_lr.base_batch_size`。如果找不到，可以在其继承的 `_base_=[xxx]` 文件中找到。另外，如果想使用自动缩放学习率的功能，请不要修改这些值。**

学习率自动缩放基本用法如下：

```shell
```

执行命令之后，会根据机器的GPU数量和训练的批次大小对学习率进行自动缩放，缩放方式详见 [线性扩展规则](https://arxiv.org/abs/1706.02677) ，比如：在 4 块 GPU 并且每张 GPU 上有 2 张图片的情况下 `lr=0.01`，那么在 16 块 GPU 并且每张 GPU 上有 4 张图片的情况下, LR 会自动缩放至 `lr=0.08`。

如果不启用该功能，则需要根据 [线性扩展规则](https://arxiv.org/abs/1706.02677) 来手动计算并修改配置文件里面 `optimizer.lr` 的值。

### 使用单 GPU 训练

我们提供了 `tools/train.py` 来开启在单张 GPU 上的训练任务。基本使用如下：

```shell
```

在训练期间，日志文件和 checkpoint 文件将会被保存在工作目录下，它需要通过配置文件中的 `work_dir` 或者 CLI 参数中的 `--work-dir` 来指定。

默认情况下，模型将在每轮训练之后在 validation 集上进行测试，测试的频率可以通过设置配置文件来指定：

```python
# 每 12 轮迭代进行一次测试评估
```

这个工具接受以下参数：

- `--work-dir ${WORK_DIR}`: 覆盖工作目录.
- `--resume`：自动从work_dir中的最新检查点恢复.

```python
```

- `--resume ${CHECKPOINT_FILE}`: 从某个 checkpoint 文件继续训练.
- `--cfg-options 'Key=value'`: 覆盖使用的配置文件中的其他设置.

**注意**：
`resume` 和 `load-from` 的区别：

`resume` 既加载了模型的权重和优化器的状态，也会继承指定 checkpoint 的迭代次数，不会重新开始训练。`load-from` 则是只加载模型的权重，它的训练是从头开始的，经常被用于微调模型。其中load-from需要写入配置文件中，而resume作为命令行参数传入。

### 使用 CPU 训练

使用 CPU 训练的流程和使用单 GPU 训练的流程一致，我们仅需要在训练流程开始前禁用 GPU。

```shell
```

之后运行单 GPU 训练脚本即可。

**注意**：

我们不推荐用户使用 CPU 进行训练，这太过缓慢。我们支持这个功能是为了方便用户在没有 GPU 的机器上进行调试。

### 在多 GPU 上训练

我们提供了 `tools/dist_train.sh` 来开启在多 GPU 上的训练。基本使用如下：

```shell
```

可选参数和单 GPU 训练的可选参数一致。

#### 同时启动多个任务

如果你想在一台机器上启动多个任务的话，比如在一个有 8 块 GPU 的机器上启动 2 个需要 4 块GPU的任务，你需要给不同的训练任务指定不同的端口（默认为 29500）来避免冲突。

如果你使用 `dist_train.sh` 来启动训练任务，你可以使用命令来设置端口。

```shell

```

```shell
```

### 使用多台机器训练

如果您想使用由 ethernet 连接起来的多台机器， 您可以使用以下命令:

在第一台机器上:

```shell
```

在第二台机器上:

```shell
```

但是，如果您不使用高速网路连接这几台机器的话，训练将会非常慢。

### 使用 Slurm 来管理任务

Slurm 是一个常见的计算集群调度系统。在 Slurm 管理的集群上，你可以使用 `slurm.sh` 来开启训练任务。它既支持单节点训练也支持多节点训练。

基本使用如下：

```shell
```

以下是在一个名称为 _dev_ 的 Slurm 分区上，使用 16 块 GPU 来训练 Mask R-CNN 的例子，并且将 `work-dir` 设置在了某些共享文件系统下。

```shell
```

```shell
GPUS=16 ./tools/slurm_train.sh dev mask_r50_1x configs/mask_rcnn_r50_fpn_1x_coco.py /nfs/xxxx/mask_rcnn_r50_fpn_1x
```

你可以查看 [源码](https://github.com/open-mmlab/mmdetection/blob/main/tools/slurm_train.sh) 来检查全部的参数和环境变量.

在使用 Slurm 时，端口需要以下方的某个方法之一来设置。

1. 通过 `--options` 来设置端口。我们非常建议用这种方法，因为它无需改变原始的配置文件。

   ```shell
   ```

2. 修改配置文件来设置不同的交流端口。

   在 `config1.py` 中，设置：

   ```python
   ```

   在 `config2.py` 中，设置：

   ```python

```

   ```python
   ```

   ```python
   ```

   然后你可以使用 `config1.py` 和 `config2.py` 来启动两个任务了。

   ```shell
   ```

# 在自定义数据集上进行训练

通过本文档，你将会知道如何使用自定义数据集对预先定义好的模型进行推理，测试以及训练。我们使用 [balloon dataset](https://github.com/matterport/Mask_RCNN/tree/master/samples/balloon) 作为例子来描述整个过程。

基本步骤如下：

1. 准备自定义数据集
2. 准备配置文件
3. 在自定义数据集上进行训练，测试和推理。

## 准备自定义数据集

MMDetection 一共支持三种形式应用新数据集：

1. 将数据集重新组织为 COCO 格式。
2. 将数据集重新组织为一个中间格式。
3. 实现一个新的数据集。

我们通常建议使用前面两种方法，因为它们通常来说比第三种方法要简单。

在本文档中，我们展示一个例子来说明如何将数据转化为 COCO 格式。

**注意**：在 MMDetection 3.0 之后，数据集和指标已经解耦（除了 CityScapes）。因此，用户在验证阶段使用任意的评价指标来评价模型在任意数据集上的性能。比如，用 VOC 评价指标来评价模型在 COCO 数据集的性能，或者同时使用 VOC 评价指标和 COCO 评价指标来评价模型在 OpenImages 数据集上的性能。

### COCO标注格式

用于实例分割的 COCO 数据集格式如下所示，其中的键（key）都是必要的，参考[这里](https://cocodataset.org/#format-data)来获取更多细节。

```json
{
}

image = {
    "id": int,
}

    "id": int,
    "bbox": [x,y,width,height], # (x, y) 为 bbox 左上角的坐标
}

    "id": int,
}]
```

现在假设我们使用 balloon dataset。

"bbox": [x,y,width,height], # (x, y) 为 bbox 左上角的坐标
}

    "id": int,
}]
```

下载了数据集之后，我们需要实现一个函数将标注格式转化为 COCO 格式。然后我们就可以使用已经实现的 `CocoDataset` 类来加载数据并进行训练以及评测。

如果你浏览过新数据集，你会发现格式如下：

```json
     1000,
     994,
     1003,
     1023,
     1050,
     1089,
     1134,
     1190,
     1265,
     1321,
     1361,
     1403,
     1428,
     1442,
     1445,
     1441,
     1427,
     1400,
     1361,
     1316,
     1269,
     1228,
     1198,
     1207,
     1210,
     1190,
     1177,
     1172,
     1174,
     1170,
     1153,
     1127,
     1104,
     1061,
     1032,
     1020],
     899,
     841,
     787,
     738,
     700,
     663,
     638,

1172,
     1174,
     1170,
     1153,
     1127,
     1104,
     1061,
     1032,
     1020],
     899,
     841,
     787,
     738,
     700,
     663,
     638,
     621,
     619,
     643,
     672,
     720,
     765,
     800,
     860,
     896,
     942,
     990,
     1035,
     1079,
     1112,
     1129,
     1134,
     1144,
     1153,
     1166,
     1166,
     1150,
     1136,
     1129,
     1122,
     1112,
     1084,
     1037,
     989,
     963],
```

标注文件时是 JSON 格式的，其中所有键（key）组成了一张图片的所有标注。

其中将 balloon dataset 转化为 COCO 格式的代码如下所示。

```python

    images = []

data_infos = load(ann_file)

    images = []

data_anno = dict(

            'id': 0,
        }])

convert_balloon_to_coco(ann_file='data/balloon/val/via_region_data.json',
```

使用如上的函数，用户可以成功将标注文件转化为 JSON 格式，之后可以使用 `CocoDataset` 对模型进行训练，并用 `CocoMetric` 评测。

## 准备配置文件

第二步需要准备一个配置文件来成功加载数据集。假设我们想要用 balloon dataset 来训练配备了 FPN 的 Mask R-CNN ，如下是我们的配置文件。假设配置文件命名为 `mask-rcnn_r50-caffe_fpn_ms-poly-1x_balloon.py`，相应保存路径为 `configs/balloon/`，配置文件内容如下所示。详细的配置文件方法可以参考[学习配置文件 — MMDetection 3.0.0 文档](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/config.html#base)。

```python
# 新配置继承了基本配置，并做了必要的修改

# 我们还需要更改 head 中的 num_classes 以匹配数据集中的类别数

# 修改数据集相关配置
    ]
}

    ]
}

# 修改评价指标相关配置

# 使用预训练的 Mask R-CNN 模型权重来做初始化，可以提高模型性能

```

## 训练一个新的模型

为了使用新的配置方法来对模型进行训练，你只需要运行如下命令。

```shell

```

```shell
```

参考 [在标准数据集上训练预定义的模型](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/train.html#id1) 来获取更多详细的使用方法。

## 测试以及推理

为了测试训练完毕的模型，你只需要运行如下命令。

```shell
```

参考 [测试现有模型](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/test.html) 来获取更多详细的使用方法。

# Dataset

Dataset classes in MMSegmentation have two functions: (1) load data information after [data preparation](../user_guides/2_dataset_prepare.md)
and (2) send data into [dataset transform pipeline](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/datasets/basesegdataset.py#L141) to do [data augmentation](./transforms.md).
There are 2 kinds of loaded information: (1) meta information which is original dataset information such as categories (classes) of dataset and their corresponding palette information, (2) data information which includes
the path of dataset images and labels.
The tutorial includes some main interfaces in MMSegmentation 1.x dataset class: methods of loading data information and modifying dataset classes in base dataset class, and the relationship between dataset and the data transform pipeline.

## Main Interfaces

Take Cityscapes as an example, if you want to run the example, please download and [preprocess](../user_guides/2_dataset_prepare.md#cityscapes)
Cityscapes dataset in `data` directory, before running the demo code:

Instantiate Cityscapes training dataset:

```python

]

```

Get the length of training set:

```python

2975
```

]

```

```python

2975
```

Get data information: The type of data information is `dict` which includes several keys:

- `'img_path'`: path of images
- `'seg_map_path'`: path of segmentation labels
- `'seg_fields'`: saving label fields
- `'sample_idx'`: the  index of the current sample

There are also `'label_map'` and `'reduce_zero_label'` whose functions would be introduced in the next section.

```python
# Acquire data information of first sample in dataset

```

```

Get dataset meta information: the type of MMSegmentation meta information is also `dict`, which includes `'classes'` field for dataset classes and `'palette'` field for corresponding colors in visualization, and has `'label_map'` field and `'reduce_zero_label'` filed.

```python

  'wall',
  'fence',
  'pole',
  'sky',
  'person',
  'rider',
  'car',
  'truck',
  'bus',
  'train',

[70, 70, 70],
  [0, 0, 70],
```

The return value of dataset `__getitem__` method is the output of data samples after data augmentation, whose type is also `dict`. It has two fields: `'inputs'` corresponding to images after data augmentation,
and `'data_samples'` corresponding to [`SegDataSample`](./structures.md) which is new data structures in MMSegmentation 1.x,
and `gt_sem_seg` of `SegDataSample` has labels after data augmentation operations.

```python

          ...,

```python

          ...,

          ...,

          ...,

...,

                          ...,
         )>

...,
         )>

                          ...,
         )>
 )}
```

As mentioned above, dataset classes have the same functions, we implemented  [`BaseSegDataset`](https://mmsegmentation.readthedocs.io/en/latest/api.html?highlight=BaseSegDataset#mmseg.datasets.BaseSegDataset) to reues the common functions.

It inherits [`BaseDataset` of MMEngine](https://github.com/open-mmlab/mmengine/blob/main/docs/en/advanced_tutorials/basedataset.md) and follows unified initialization process of OpenMMLab. It supports the highly effective interior storing format, some functions like
dataset concatenation and repeatedly sampling. In MMSegmentation `BaseSegDataset`, the **method of loading data information** (`load_data_list`) is redefined and adds new `get_label_map` method to **modify dataset classes information**.

### Loading Dataset Information

The loaded data information includes the path of images samples and annotations samples, the detailed implementation could be found in
[`load_data_list`](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/datasets/basesegdataset.py#L231) of `BaseSegDataset` in MMSegmentation.
There are two main methods to acquire the path of images and labels:

1. Load file paths according to the dirictory and suffix of input images and annotations

If the dataset directory structure is organized as below, the [`load_data_list`](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/datasets/basesegdataset.py#L231) can parse dataset directory Structure:

```
├── data
│   │   │   ├── val
│   │   │   ├── val
```

Here is an example pf ADE20K, and below the directory structure of the dataset:

```
├── ade

```
├── ade
```

```python

    reduce_zero_label=True)
```

2. Load file paths from annotation file

Dataset also can load an annotation file which includes the data sample paths of dataset.
Take PascalContext dataset instance as an example, its input annotation file is:

```python
...
```

```python
...
```

It needs to define `ann_file` when instantiation:

```python
```

### Modification of Dataset Classes

- Use `metainfo` input argument

Meta information is defined as class variables, such as `METAINFO` variable of Cityscapes:

```python

    """

"""

```

Here `'classes'` defines class names of Cityscapes dataset annotations, if users only concern some classes about vehicles and **ignore other classes**,

```

the meta information of dataset could be modified by defined input argument `metainfo` when instantiating Cityscapes dataset:

```python

# metainfo only keep classes below:

  [0, 0, 70],

[0, 60, 100],
 # pixels whose label index are 255 would be ignored when calculating loss
  1: 255,
  2: 255,
  3: 255,
  4: 255,
  5: 255,
  6: 255,
  7: 255,
  8: 255,
  9: 255,
  10: 255,
  11: 255,
  12: 255,
  13: 0,
  14: 1,
  15: 2,
  16: 3,
  17: 4,
  18: 5},
```

Meta information is different from default setting of Cityscapes dataset. Moreover, `label_map` field is also defined, which is used for modifying label index of each pixel on segmentation mask.
The segmentation label would re-map class information by `label_map`, [here](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/datasets/basesegdataset.py#L151) is detailed implementation:

```python
```

- Using `reduce_zero_label` input argument

To ignore label 0 (such as ADE20K dataset), we can use `reduce_zero_label` (default to `False`) argument of BaseSegDataset and its subclasses.
When `reduce_zero_label` is `True`, label 0 in segmentation annotations would be set as 255 (models of MMSegmentation would ignore label 255 in calculating loss) and indices of other labels will minus 1:

```python
```

## Dataset and Data Transform Pipeline

```python
```

If the argument `pipeline` is defined, the return value of `__getitem__` method is after data argument.
If dataset input argument does not define pipeline, it is the same as return value of `get_data_info` method.

```python

```

# 3D Pictorial Structures for Multiple Human Pose Estimation

<details>

```bibtex
    Nassir and Ilic, Slobodan},
}
```

# SETR

> [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://arxiv.org/abs/2012.15840)

<a href="https://github.com/fudan-zvg/SETR">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/decode_heads/setr_up_head.py#L11">Code Snippet</a>

Most recent semantic segmentation methods adopt a fully-convolutional network (FCN) with an encoder-decoder architecture. The encoder progressively reduces the spatial resolution and learns more abstract/semantic visual concepts with larger receptive fields. Since context modeling is critical for segmentation, the latest efforts have been focused on increasing the receptive field, through either dilated/atrous convolutions or inserting attention modules. However, the encoder-decoder based FCN architecture remains unchanged. In this paper, we aim to provide an alternative perspective by treating semantic segmentation as a sequence-to-sequence prediction task. Specifically, we deploy a pure transformer (ie, without convolution and resolution reduction) to encode an image as a sequence of patches. With the global context modeled in every layer of the transformer, this encoder can be combined with a simple decoder to provide a powerful segmentation model, termed SEgmentation TRansformer (SETR). Extensive experiments show that SETR achieves new state of the art on ADE20K (50.28% mIoU), Pascal Context (55.83% mIoU) and competitive results on Cityscapes. Particularly, we achieve the first position in the highly competitive ADE20K test server leaderboard on the day of submission.

<img src="https://user-images.githubusercontent.com/24582831/142902777-ee2d34b7-a631-4fa7-ad68-118ff5716afe.png" width="80%"/>
</div>

```None
This head has two version head.
```

## Usage

You can download the pretrain from [here](https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-vitjx/jx_vit_large_p16_384-b3be5167.pth). Then you can convert its keys with the script `vit2mmseg.py` in the tools directory.

```shell
```

E.g.

```shell
python tools/model_converters/vit2mmseg.py \
jx_vit_large_p16_384-b3be5167.pth pretrain/vit_large_p16.pth
```

This script convert the model from `PRETRAIN_PATH` and store the converted model in `STORE_PATH`.

### ADE20K

```

### ADE20K

| Method     | Backbone | Crop Size | Batch Size | Lr schd | Mem (GB) | Inf time (fps) | Device | mIoU  | mIoU(ms+flip) | config                                                                                                                     | download                                                                                                                                                                                                                                                                                                                             |

| SETR Naive | ViT-L    | 512x512   | 16         | 160000  | 18.40    | 4.72           | V100   | 48.28 |         49.56 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/setr/setr_vit-l_naive_8xb2-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/setr/setr_naive_512x512_160k_b16_ade20k/setr_naive_512x512_160k_b16_ade20k_20210619_191258-061f24f5.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/setr/setr_naive_512x512_160k_b16_ade20k/setr_naive_512x512_160k_b16_ade20k_20210619_191258.log.json) |

| SETR PUP   | ViT-L    | 512x512   | 16         | 160000  | 19.54    | 4.50           | V100   | 48.24 |         49.99 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/setr/setr_vit-l_pup_8xb2-160k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/setr/setr_pup_512x512_160k_b16_ade20k/setr_pup_512x512_160k_b16_ade20k_20210619_191343-7e0ce826.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/setr/setr_pup_512x512_160k_b16_ade20k/setr_pup_512x512_160k_b16_ade20k_20210619_191343.log.json)         |

| SETR MLA   | ViT-L    | 512x512   | 8          | 160000  | 10.96    | -              | V100   | 47.34 |         49.05 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/setr/setr_vit-l-mla_8xb1-160k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/setr/setr_mla_512x512_160k_b8_ade20k/setr_mla_512x512_160k_b8_ade20k_20210619_191118-c6d21df0.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/setr/setr_mla_512x512_160k_b8_ade20k/setr_mla_512x512_160k_b8_ade20k_20210619_191118.log.json)             |

| SETR MLA   | ViT-L    | 512x512   | 16         | 160000  | 17.30    | 5.25           | V100   | 47.39 |         49.37 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/setr/setr_vit-l_mla_8xb2-160k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/setr/setr_mla_512x512_160k_b16_ade20k/setr_mla_512x512_160k_b16_ade20k_20210619_191057-f9741de7.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/setr/setr_mla_512x512_160k_b16_ade20k/setr_mla_512x512_160k_b16_ade20k_20210619_191057.log.json)         |

| Method     | Backbone | Crop Size | Batch Size | Lr schd | Mem (GB) | Inf time (fps) | Device | mIoU  | mIoU(ms+flip) | config                                                                                                                        | download                                                                                                                                                                                                                                                                                                                                                                                 |

| SETR Naive | ViT-L    | 768x768   | 8          | 80000   | 24.06    | 0.39           | V100   | 78.10 |         80.22 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/setr/setr_vit-l_naive_8xb1-80k_cityscapes-768x768.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/setr/setr_naive_vit-large_8x1_768x768_80k_cityscapes/setr_naive_vit-large_8x1_768x768_80k_cityscapes_20211123_000505-20728e80.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/setr/setr_naive_vit-large_8x1_768x768_80k_cityscapes/setr_naive_vit-large_8x1_768x768_80k_cityscapes_20211123_000505.log.json) |

| SETR PUP   | ViT-L    | 768x768   | 8          | 80000   | 27.96    | 0.37           | V100   | 79.21 |         81.02 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/setr/setr_vit-l_pup_8xb1-80k_cityscapes-768x768.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/setr/setr_pup_vit-large_8x1_768x768_80k_cityscapes/setr_pup_vit-large_8x1_768x768_80k_cityscapes_20211122_155115-f6f37b8f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/setr/setr_pup_vit-large_8x1_768x768_80k_cityscapes/setr_pup_vit-large_8x1_768x768_80k_cityscapes_20211122_155115.log.json)         |

| SETR MLA   | ViT-L    | 768x768   | 8          | 80000   | 24.10    | 0.41           | V100   | 77.00 |         79.59 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/setr/setr_vit-l_mla_8xb1-80k_cityscapes-768x768.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/setr/setr_mla_vit-large_8x1_768x768_80k_cityscapes/setr_mla_vit-large_8x1_768x768_80k_cityscapes_20211119_101003-7f8dccbe.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/setr/setr_mla_vit-large_8x1_768x768_80k_cityscapes/setr_mla_vit-large_8x1_768x768_80k_cityscapes_20211119_101003.log.json)         |

```bibtex
@article{zheng2020rethinking,
  title={Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers},
  author={Zheng, Sixiao and Lu, Jiachen and Zhao, Hengshuang and Zhu, Xiatian and Luo, Zekun and Wang, Yabiao and Fu, Yanwei and Feng, Jianfeng and Xiang, Tao and Torr, Philip HS and others},

author={Zheng, Sixiao and Lu, Jiachen and Zhao, Hengshuang and Zhu, Xiatian and Luo, Zekun and Wang, Yabiao and Fu, Yanwei and Feng, Jianfeng and Xiang, Tao and Torr, Philip HS and others},
  journal={arXiv preprint arXiv:2012.15840},
}
```

# Flamingo

> [Flamingo: a Visual Language Model for Few-Shot Learning](https://arxiv.org/abs/2204.14198)

Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bridge powerful pretrained vision-only and language-only models, (ii) handle sequences of arbitrarily interleaved visual and textual data, and (iii) seamlessly ingest images or videos as inputs. Thanks to their flexibility, Flamingo models can be trained on large-scale multimodal web corpora containing arbitrarily interleaved text and images, which is key to endow them with in-context few-shot learning capabilities. We perform a thorough evaluation of our models, exploring and measuring their ability to rapidly adapt to a variety of image and video tasks. These include open-ended tasks such as visual question-answering, where the model is prompted with a question which it has to answer; captioning tasks, which evaluate the ability to describe a scene or an event; and close-ended tasks such as multiple-choice visual question-answering. For tasks lying anywhere on this spectrum, a single Flamingo model can achieve a new state of the art with few-shot learning, simply by prompting the model with task-specific examples. On numerous benchmarks, Flamingo outperforms models fine-tuned on thousands of times more task-specific data.

<img src="https://user-images.githubusercontent.com/26739999/236371424-3b9d2e16-3966-4c64-8b87-e33fd6348824.png" width="80%"/>
</div>

```python

result = inference_model('flamingo_3rdparty-zeroshot_caption', 'demo/cat-dog.png')
# {'pred_caption': 'A dog and a cat are looking at each other. '}
```

Test:

```shell
python tools/test.py configs/flamingo/flamingo_zeroshot_caption.py https://download.openmmlab.com/mmclassification/v1/flamingo/openflamingo-9b-adapter_20230505-554310c8.pth
```

```

| Model                                  | Params (G) | CIDER |                 Config                 |                                                   Download                                                    |
| :------------------------------------- | :--------: | :---: | :------------------------------------: | :-----------------------------------------------------------------------------------------------------------: |
| `flamingo_3rdparty-zeroshot_caption`\* |   8.220    | 65.50 | [config](flamingo_zeroshot_caption.py) | [model](https://download.openmmlab.com/mmclassification/v1/flamingo/openflamingo-9b-adapter_20230505-554310c8.pth) |

*Models with * are converted from the [openflamingo](https://github.com/mlfoundations/open_flamingo). The config files of these models are only for inference. We haven't reproduce the training results.*

| Model                              | Params (G) | Accuracy |               Config               |                                                      Download                                                      |
| :--------------------------------- | :--------: | :------: | :--------------------------------: | :----------------------------------------------------------------------------------------------------------------: |
| `flamingo_3rdparty-zeroshot_vqa`\* |    8.22    |  43.50   | [config](flamingo_zeroshot_vqa.py) | [model](https://download.openmmlab.com/mmclassification/v1/flamingo/openflamingo-9b-adapter_20230505-554310c8.pth) |

```bibtex
@article{Alayrac2022FlamingoAV,

```bibtex
  title={Flamingo: a Visual Language Model for Few-Shot Learning},
  author={Jean-Baptiste Alayrac and Jeff Donahue and Pauline Luc and Antoine Miech and Iain Barr and Yana Hasson and Karel Lenc and Arthur Mensch and Katie Millican and Malcolm Reynolds and Roman Ring and Eliza Rutherford and Serkan Cabi and Tengda Han and Zhitao Gong and Sina Samangooei and Marianne Monteiro and Jacob Menick and Sebastian Borgeaud and Andy Brock and Aida Nematzadeh and Sahand Sharifzadeh and Mikolaj Binkowski and Ricardo Barreira and Oriol Vinyals and Andrew Zisserman and Karen Simonyan},
  volume={abs/2204.14198}
}
```

```bibtex
@software{anas_awadalla_2023_7733589,

journal={ArXiv},
}
```

```bibtex
  author = {Awadalla, Anas and Gao, Irena and Gardner, Joshua and Hessel, Jack and Hanafy, Yusuf and Zhu, Wanrong and Marathe, Kalyani and Bitton, Yonatan and Gadre, Samir and Jitsev, Jenia and Kornblith, Simon and Koh, Pang Wei and Ilharco, Gabriel and Wortsman, Mitchell and Schmidt, Ludwig},
  title = {OpenFlamingo},
  month        = mar,
  year         = 2023,
  publisher    = {Zenodo},
  version      = {v0.1.1},
  doi          = {10.5281/zenodo.7733589},
  url          = {https://doi.org/10.5281/zenodo.7733589}
}
```

<details>

```bibtex
}
```

<details>

```bibtex

| [pose_resnext_50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnext50_8xb64-210e_coco-256x192.py) |  256x192   | 0.715 |      0.897      |      0.791      | 0.771 |      0.935      | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnext/resnext50_coco_256x192-dcff15f6_20200727.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnext/resnext50_coco_256x192_20200727.log.json) |
| [pose_resnext_50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnext50_8xb64-210e_coco-384x288.py) |  384x288   | 0.724 |      0.899      |      0.794      | 0.777 |      0.936      | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnext/resnext50_coco_384x288-412c848f_20200727.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnext/resnext50_coco_384x288_20200727.log.json) |

| [pose_resnext_101](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnext101_8xb64-210e_coco-256x192.py) |  256x192   | 0.726 |      0.900      |      0.801      | 0.781 |      0.939      | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnext/resnext101_coco_256x192-c7eba365_20200727.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnext/resnext101_coco_256x192_20200727.log.json) |
| [pose_resnext_101](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnext101_8xb32-210e_coco-384x288.py) |  384x288   | 0.744 |      0.903      |      0.815      | 0.794 |      0.939      | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnext/resnext101_coco_384x288-f5eabcd6_20200727.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnext/resnext101_coco_384x288_20200727.log.json) |

| [pose_resnext_152](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnext152_8xb32-210e_coco-256x192.py) |  256x192   | 0.730 |      0.903      |      0.808      | 0.785 |      0.940      | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnext/resnext152_coco_256x192-102449aa_20200727.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnext/resnext152_coco_256x192_20200727.log.json) |
| [pose_resnext_152](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnext152_8xb48-210e_coco-384x288.py) |  384x288   | 0.742 |      0.904      |      0.810      | 0.794 |      0.940      | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnext/resnext152_coco_384x288-806176df_20200727.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnext/resnext152_coco_384x288_20200727.log.json) |

# 2D Face Keypoint Datasets

- [300W](#300w-dataset) \[ [Homepage](https://ibug.doc.ic.ac.uk/resources/300-W/) \]
- [WFLW](#wflw-dataset) \[ [Homepage](https://wywu.github.io/projects/LAB/WFLW.html) \]
- [AFLW](#aflw-dataset) \[ [Homepage](https://www.tugraz.at/institute/icg/research/team-bischof/lrs/downloads/aflw/) \]
- [COFW](#cofw-dataset) \[ [Homepage](http://www.vision.caltech.edu/xpburgos/ICCV13/) \]
- [COCO-WholeBody-Face](#coco-wholebody-face) \[ [Homepage](https://github.com/jin-s13/COCO-WholeBody/) \]
- [LaPa](#lapa-dataset) \[ [Homepage](https://github.com/JDAI-CV/lapa-dataset) \]

## 300W Dataset

<details>
<summary align="right"><a href="https://www.sciencedirect.com/science/article/pii/S0262885616000147">300W (IMAVIS'2016)</a></summary>

```bibtex

<details>

```bibtex
@article{sagonas2016300,
  title={300 faces in-the-wild challenge: Database and results},
  author={Sagonas, Christos and Antonakos, Epameinondas and Tzimiropoulos, Georgios and Zafeiriou, Stefanos and Pantic, Maja},
  journal={Image and vision computing},
  volume={47},
  pages={3--18},
  publisher={Elsevier}
}
```

  <img src="https://user-images.githubusercontent.com/100993824/227780043-e557acc3-5966-48ba-ac9e-b6c5916f1e55.jpg" height="200px">
</div>

For 300W data, please download images from [300W Dataset](https://ibug.doc.ic.ac.uk/resources/300-W/).
Please download the annotation files from [300w_annotations](https://download.openmmlab.com/mmpose/datasets/300w_annotations.tar).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── 300w
        |   |── face_landmarks_300w_train.json
        |   |── face_landmarks_300w_valid.json
        |   |── face_landmarks_300w_valid_common.json
        |   |── face_landmarks_300w_valid_challenge.json
        |   |── face_landmarks_300w_test.json
        `── images
            |── afw
            |   |── 1051618982_1.jpg
            |   |── 111076519_1.jpg
            |    ...
            |── helen
            |   |── trainset
            |   |   |── 100032540_1.jpg
            |   |   |── 100040721_1.jpg
            |   |    ...
            |   |── testset
            |   |   |── 296814969_3.jpg
            |   |   |── 2968560214_1.jpg
            |   |    ...

|   |   |── 100040721_1.jpg
            |   |    ...
            |   |    ...
            |── ibug
            |   |── image_003_1.jpg
            |   |── image_004_1.jpg
            |    ...
            |── lfpw
            |   |   |── image_0001.png
            |   |   |── image_0002.png
            |   |    ...
            |   |    ...
            `── Test
                |── 01_Indoor
                |   |── indoor_001.png
                |   |── indoor_002.png
                |    ...
                `── 02_Outdoor
                    |── outdoor_001.png
                    |── outdoor_002.png
                     ...
```

## WFLW Dataset

<details>

`── 02_Outdoor
                     ...
```

<details>

```bibtex
}
```

  <img src="https://user-images.githubusercontent.com/100993824/227785100-4b3d1e39-0d64-47c0-9e55-c947ce70866e.png" height="200px">
</div>

For WFLW data, please download images from [WFLW Dataset](https://wywu.github.io/projects/LAB/WFLW.html).

</div>

Please download the annotation files from [wflw_annotations](https://download.openmmlab.com/mmpose/datasets/wflw_annotations.tar).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── wflw
        |   |── face_landmarks_wflw_train.json
        |   |── face_landmarks_wflw_test.json
        |   |── face_landmarks_wflw_test_blur.json
        |   |── face_landmarks_wflw_test_occlusion.json
        |   |── face_landmarks_wflw_test_expression.json
        |   |── face_landmarks_wflw_test_largepose.json
        |   |── face_landmarks_wflw_test_illumination.json
        |   |── face_landmarks_wflw_test_makeup.json
        |
        `── images
            |── 0--Parade
            |   |── 0_Parade_marchingband_1_1015.jpg

|   |── face_landmarks_wflw_test_makeup.json
        |
        `── images
            |   |── 0_Parade_marchingband_1_1031.jpg
            |    ...
            |── 1--Handshaking
            |   |── 1_Handshaking_Handshaking_1_105.jpg
            |   |── 1_Handshaking_Handshaking_1_107.jpg
            |    ...
            ...
```

## AFLW Dataset

<details>

```bibtex
}

}
```

For AFLW data, please download images from [AFLW Dataset](https://www.tugraz.at/institute/icg/research/team-bischof/lrs/downloads/aflw/).
Please download the annotation files from [aflw_annotations](https://download.openmmlab.com/mmpose/datasets/aflw_annotations.tar).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── aflw
        |   |── face_landmarks_aflw_train.json
        |   |── face_landmarks_aflw_test_frontal.json
        |   |── face_landmarks_aflw_test.json
        `── images
            |── flickr
                |── 0
                |   |── image00002.jpg
                |   |── image00013.jpg
                |    ...
                |── 2
                |   |── image00004.jpg

|── 0
                |    ...
                |── 2
                |   |── image00006.jpg
                |    ...
                `── 3
                    |── image00032.jpg
                    |── image00035.jpg
                     ...
```

## COFW Dataset

<details>

```bibtex
}
```

}
```

  <img src="https://user-images.githubusercontent.com/100993824/227786792-06604943-c062-4bcd-bb2d-a2f78d80115b.png" height="200px">
</div>

For COFW data, please download from [COFW Dataset (Color Images)](http://www.vision.caltech.edu/xpburgos/ICCV13/Data/COFW_color.zip).
Move `COFW_train_color.mat` and `COFW_test_color.mat` to `data/cofw/` and make them look like:

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── cofw
```

Run the following script under `{MMPose}/data`

`python tools/dataset_converters/parse_cofw_dataset.py`

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── cofw

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── cofw
        |── images
```

## COCO-WholeBody (Face)

<details>

```bibtex
}
```

}
```

  <img src="https://user-images.githubusercontent.com/100993824/227787217-2f226dc0-e5d7-4d0b-9ab8-68b53a5467c2.png" height="200px">
</div>

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── coco
        │   │-- ...
            │-- ...

```

## LaPa

<details>

## LaPa

<details>

```bibtex
}
```

  <img src="https://github.com/lucia123/lapa-dataset/raw/master/sample.png" height="200px">
</div>

For [LaPa](https://github.com/JDAI-CV/lapa-dataset) dataset, images can be downloaded from [their github page](https://github.com/JDAI-CV/lapa-dataset).

Download and extract them under $MMPOSE/data, and use our `tools/dataset_converters/lapa2coco.py` to make them look like this:

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── LaPa

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── LaPa
        │   │-- lapa_train.json
        │   |-- lapa_val.json
        │   |-- lapa_test.json
        |   |-- lapa_trainval.json
        │-- train
        │   │-- labels
        │   │-- landmarks
        │-- val
        `-- test

```

# Faster R-CNN

<img src="https://user-images.githubusercontent.com/40661020/143881188-ab87720f-5059-4b4e-a928-b540fb8fb84d.png" height="300"/>
</div>

|    Backbone     |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                      Config                       |                                                                                                                                                                          Download                                                                                                                                                                           |

| :-------------: | :-----: | :-----: | :------: | :------------: | :----: | :-----------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|     R-50-C4     |  caffe  |   1x    |    -     |       -        |  35.6  |  [config](./faster-rcnn_r50-caffe_c4-1x_coco.py)  |            [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_c4_1x_coco/faster_rcnn_r50_caffe_c4_1x_coco_20220316_150152-3f885b85.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_c4_1x_coco/faster_rcnn_r50_caffe_c4_1x_coco_20220316_150152.log.json)             |

|    R-50-DC5     |  caffe  |   1x    |    -     |       -        |  37.2  | [config](./faster-rcnn_r50-caffe-dc5_1x_coco.py)  |          [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_dc5_1x_coco/faster_rcnn_r50_caffe_dc5_1x_coco_20201030_151909-531f0f43.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_dc5_1x_coco/faster_rcnn_r50_caffe_dc5_1x_coco_20201030_151909.log.json)           |
|    R-50-FPN     |  caffe  |   1x    |   3.8    |                |  37.8  | [config](./faster-rcnn_r50-caffe_fpn_1x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_fpn_1x_coco/faster_rcnn_r50_caffe_fpn_1x_coco_bbox_mAP-0.378_20200504_180032-c5925ee5.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_fpn_1x_coco/faster_rcnn_r50_caffe_fpn_1x_coco_20200504_180032.log.json)   |

|    R-50-FPN     | pytorch |   1x    |   4.0    |      21.4      |  37.4  |    [config](./faster-rcnn_r50_fpn_1x_coco.py)     |                          [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130_204655.log.json)                          |
| R-50-FPN (FP16) | pytorch |   1x    |   3.4    |      28.8      |  37.5  |  [config](./faster-rcnn_r50_fpn_amp-1x_coco.py)   |                       [model](https://download.openmmlab.com/mmdetection/v2.0/fp16/faster_rcnn_r50_fpn_fp16_1x_coco/faster_rcnn_r50_fpn_fp16_1x_coco_20200204-d4dc1471.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/fp16/faster_rcnn_r50_fpn_fp16_1x_coco/faster_rcnn_r50_fpn_fp16_1x_coco_20200204_143530.log.json)                       |

|    R-50-FPN     | pytorch |   2x    |    -     |       -        |  38.4  |    [config](./faster-rcnn_r50_fpn_2x_coco.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_2x_coco/faster_rcnn_r50_fpn_2x_coco_bbox_mAP-0.384_20200504_210434-a5d8aa15.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_2x_coco/faster_rcnn_r50_fpn_2x_coco_20200504_210434.log.json)               |
|    R-101-FPN    |  caffe  |   1x    |   5.7    |                |  39.8  | [config](./faster-rcnn_r101-caffe_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r101_caffe_fpn_1x_coco/faster_rcnn_r101_caffe_fpn_1x_coco_bbox_mAP-0.398_20200504_180057-b269e9dd.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r101_caffe_fpn_1x_coco/faster_rcnn_r101_caffe_fpn_1x_coco_20200504_180057.log.json) |

|    R-101-FPN    | pytorch |   1x    |   6.0    |      15.6      |  39.4  |    [config](./faster-rcnn_r101_fpn_1x_coco.py)    |                        [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r101_fpn_1x_coco/faster_rcnn_r101_fpn_1x_coco_20200130-f513f705.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r101_fpn_1x_coco/faster_rcnn_r101_fpn_1x_coco_20200130_204655.log.json)                        |
|    R-101-FPN    | pytorch |   2x    |    -     |       -        |  39.8  |    [config](./faster-rcnn_r101_fpn_2x_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r101_fpn_2x_coco/faster_rcnn_r101_fpn_2x_coco_bbox_mAP-0.398_20200504_210455-1d2dac9c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r101_fpn_2x_coco/faster_rcnn_r101_fpn_2x_coco_20200504_210455.log.json)             |

| X-101-32x4d-FPN | pytorch |   1x    |   7.2    |      13.8      |  41.2  | [config](./faster-rcnn_x101-32x4d_fpn_1x_coco.py) |            [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_x101_32x4d_fpn_1x_coco/faster_rcnn_x101_32x4d_fpn_1x_coco_20200203-cff10310.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_x101_32x4d_fpn_1x_coco/faster_rcnn_x101_32x4d_fpn_1x_coco_20200203_000520.log.json)            |
| X-101-32x4d-FPN | pytorch |   2x    |    -     |       -        |  41.2  | [config](./faster-rcnn_x101-32x4d_fpn_2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_x101_32x4d_fpn_2x_coco/faster_rcnn_x101_32x4d_fpn_2x_coco_bbox_mAP-0.412_20200506_041400-64a12c0b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_x101_32x4d_fpn_2x_coco/faster_rcnn_x101_32x4d_fpn_2x_coco_20200506_041400.log.json) |

| X-101-64x4d-FPN | pytorch |   1x    |   10.3   |      9.4       |  42.1  | [config](./faster-rcnn_x101-64x4d_fpn_1x_coco.py) |            [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_x101_64x4d_fpn_1x_coco/faster_rcnn_x101_64x4d_fpn_1x_coco_20200204-833ee192.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_x101_64x4d_fpn_1x_coco/faster_rcnn_x101_64x4d_fpn_1x_coco_20200204_134340.log.json)            |
| X-101-64x4d-FPN | pytorch |   2x    |    -     |       -        |  41.6  | [config](./faster-rcnn_x101-64x4d_fpn_2x_coco.py) |        [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_x101_64x4d_fpn_2x_coco/faster_rcnn_x101_64x4d_fpn_2x_coco_20200512_161033-5961fa95.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_x101_64x4d_fpn_2x_coco/faster_rcnn_x101_64x4d_fpn_2x_coco_20200512_161033.log.json)         |

## Different regression loss

We trained with R-50-FPN pytorch style backbone for 1x schedule.

| Backbone |   Loss type    | Mem (GB) | Inf time (fps) | box AP |                         Config                         |                                                                                                                                                             Download                                                                                                                                                             |
| :------: | :------------: | :------: | :------------: | :----: | :----------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

| R-50-FPN |     L1Loss     |   4.0    |      21.4      |  37.4  |       [config](./faster-rcnn_r50_fpn_1x_coco.py)       |            [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130_204655.log.json)             |
| R-50-FPN |    IoULoss     |          |                |  37.9  |     [config](./faster-rcnn_r50_fpn_iou_1x_coco.py)     | [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_iou_1x_coco/faster_rcnn_r50_fpn_iou_1x_coco_20200506_095954-938e81f0.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_iou_1x_coco/faster_rcnn_r50_fpn_iou_1x_coco_20200506_095954.log.json) |

| R-50-FPN |    GIoULoss    |          |                |  37.6  |    [config](./faster-rcnn_r50_fpn_giou_1x_coco.py)     |            [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_giou_1x_coco-0eada910.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_giou_1x_coco_20200505_161120.log.json)            |
| R-50-FPN | BoundedIoULoss |          |                |  37.4  | [config](./faster-rcnn_r50_fpn_bounded-iou_1x_coco.py) |     [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_bounded_iou_1x_coco-98ad993b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_bounded_iou_1x_coco_20200505_160738.log.json)     |

|                           Backbone                            |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                        Config                        |                                                                                                                                                                                        Download                                                                                                                                                                                         |

|      [R-50-C4](./faster-rcnn_r50-caffe-c4_ms-1x_coco.py)      |  caffe  |   1x    |    -     |                |  35.9  |  [config](./faster-rcnn_r50-caffe-c4_ms-1x_coco.py)  |          [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_c4_mstrain_1x_coco/faster_rcnn_r50_caffe_c4_mstrain_1x_coco_20220316_150527-db276fed.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_c4_mstrain_1x_coco/faster_rcnn_r50_caffe_c4_mstrain_1x_coco_20220316_150527.log.json)           |

|     [R-50-DC5](./faster-rcnn_r50-caffe-dc5_ms-1x_coco.py)     |  caffe  |   1x    |    -     |                |  37.4  | [config](./faster-rcnn_r50-caffe-dc5_ms-1x_coco.py)  |        [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_dc5_mstrain_1x_coco/faster_rcnn_r50_caffe_dc5_mstrain_1x_coco_20201028_233851-b33d21b9.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_dc5_mstrain_1x_coco/faster_rcnn_r50_caffe_dc5_mstrain_1x_coco_20201028_233851.log.json)         |

|     [R-50-DC5](./faster-rcnn_r50-caffe-dc5_ms-3x_coco.py)     |  caffe  |   3x    |    -     |                |  38.7  | [config](./faster-rcnn_r50-caffe-dc5_ms-3x_coco.py)  |        [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_dc5_mstrain_3x_coco/faster_rcnn_r50_caffe_dc5_mstrain_3x_coco_20201028_002107-34a53b2c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_dc5_mstrain_3x_coco/faster_rcnn_r50_caffe_dc5_mstrain_3x_coco_20201028_002107.log.json)         |

|     [R-50-FPN](./faster-rcnn_r50-caffe_fpn_ms-2x_coco.py)     |  caffe  |   2x    |   3.7    |                |  39.7  | [config](./faster-rcnn_r50-caffe_fpn_ms-2x_coco.py)  | [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_fpn_mstrain_2x_coco/faster_rcnn_r50_caffe_fpn_mstrain_2x_coco_bbox_mAP-0.397_20200504_231813-10b2de58.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_fpn_mstrain_2x_coco/faster_rcnn_r50_caffe_fpn_mstrain_2x_coco_20200504_231813.log.json) |

|     [R-50-FPN](./faster-rcnn_r50-caffe_fpn_ms-3x_coco.py)     |  caffe  |   3x    |   3.7    |                |  39.9  | [config](./faster-rcnn_r50-caffe_fpn_ms-3x_coco.py)  |        [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_fpn_mstrain_3x_coco/faster_rcnn_r50_caffe_fpn_mstrain_3x_coco_20210526_095054-1f77628b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_caffe_fpn_mstrain_3x_coco/faster_rcnn_r50_caffe_fpn_mstrain_3x_coco_20210526_095054.log.json)         |

|        [R-50-FPN](./faster-rcnn_r50_fpn_ms-3x_coco.py)        | pytorch |   3x    |   3.9    |                |  40.3  |    [config](./faster-rcnn_r50_fpn_ms-3x_coco.py)     |                    [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_mstrain_3x_coco/faster_rcnn_r50_fpn_mstrain_3x_coco_20210524_110822-e10bd31c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_mstrain_3x_coco/faster_rcnn_r50_fpn_mstrain_3x_coco_20210524_110822.log.json)                     |

|    [R-101-FPN](./faster-rcnn_r101-caffe_fpn_ms-3x_coco.py)    |  caffe  |   3x    |   5.6    |                |  42.0  | [config](./faster-rcnn_r101-caffe_fpn_ms-3x_coco.py) |      [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r101_caffe_fpn_mstrain_3x_coco/faster_rcnn_r101_caffe_fpn_mstrain_3x_coco_20210526_095742-a7ae426d.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r101_caffe_fpn_mstrain_3x_coco/faster_rcnn_r101_caffe_fpn_mstrain_3x_coco_20210526_095742.log.json)       |

|       [R-101-FPN](./faster-rcnn_r101_fpn_ms-3x_coco.py)       | pytorch |   3x    |   5.8    |                |  41.8  |    [config](./faster-rcnn_r101_fpn_ms-3x_coco.py)    |                  [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r101_fpn_mstrain_3x_coco/faster_rcnn_r101_fpn_mstrain_3x_coco_20210524_110822-4d4d2ca8.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r101_fpn_mstrain_3x_coco/faster_rcnn_r101_fpn_mstrain_3x_coco_20210524_110822.log.json)                   |

| [X-101-32x4d-FPN](./faster-rcnn_x101-32x4d_fpn_ms-3x_coco.py) | pytorch |   3x    |   7.0    |                |  42.5  | [config](./faster-rcnn_x101-32x4d_fpn_ms-3x_coco.py) |      [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_x101_32x4d_fpn_mstrain_3x_coco/faster_rcnn_x101_32x4d_fpn_mstrain_3x_coco_20210524_124151-16b9b260.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_x101_32x4d_fpn_mstrain_3x_coco/faster_rcnn_x101_32x4d_fpn_mstrain_3x_coco_20210524_124151.log.json)       |

| [X-101-32x8d-FPN](./faster-rcnn_x101-32x8d_fpn_ms-3x_coco.py) | pytorch |   3x    |   10.1   |                |  42.4  | [config](./faster-rcnn_x101-32x8d_fpn_ms-3x_coco.py) |      [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_x101_32x8d_fpn_mstrain_3x_coco/faster_rcnn_x101_32x8d_fpn_mstrain_3x_coco_20210604_182954-002e082a.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_x101_32x8d_fpn_mstrain_3x_coco/faster_rcnn_x101_32x8d_fpn_mstrain_3x_coco_20210604_182954.log.json)       |

| [X-101-64x4d-FPN](./faster-rcnn_x101-64x4d_fpn_ms-3x_coco.py) | pytorch |   3x    |   10.0   |                |  43.1  | [config](./faster-rcnn_x101-64x4d_fpn_ms-3x_coco.py) |      [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_x101_64x4d_fpn_mstrain_3x_coco/faster_rcnn_x101_64x4d_fpn_mstrain_3x_coco_20210524_124528-26c63de6.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_x101_64x4d_fpn_mstrain_3x_coco/faster_rcnn_x101_64x4d_fpn_mstrain_3x_coco_20210524_124528.log.json)       |

We further finetune some pre-trained models on the COCO subsets, which only contain only a few of the 80 categories.

| Backbone                                                                 | Style | Class name         | Pre-traind model                                               | Mem (GB) | box AP | Config                                                                 | Download                                                                                                                                                                                                                                                                                                                                                                                     |

| [R-50-FPN](./faster-rcnn_r50-caffe_fpn_ms-1x_coco-person.py)             | caffe | person             | [R-50-FPN-Caffe-3x](./faster-rcnn_r50-caffe_fpn_ms-3x_coco.py) | 3.7      | 55.8   | [config](./faster-rcnn_r50-caffe_fpn_ms-1x_coco-person.py)             | [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco-person/faster_rcnn_r50_fpn_1x_coco-person_20201216_175929-d022e227.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco-person/faster_rcnn_r50_fpn_1x_coco-person_20201216_175929.log.json)                                                 |

| [R-50-FPN](./faster-rcnn_r50-caffe_fpn_ms-1x_coco-person-bicycle-car.py) | caffe | person-bicycle-car | [R-50-FPN-Caffe-3x](./faster-rcnn_r50-caffe_fpn_ms-3x_coco.py) | 3.7      | 44.1   | [config](./faster-rcnn_r50-caffe_fpn_ms-1x_coco-person-bicycle-car.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco-person-bicycle-car/faster_rcnn_r50_fpn_1x_coco-person-bicycle-car_20201216_173117-6eda6d92.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco-person-bicycle-car/faster_rcnn_r50_fpn_1x_coco-person-bicycle-car_20201216_173117.log.json) |

## Torchvision New Receipe (TNR)

Torchvision released its high-precision ResNet models. The training details can be found on the [Pytorch website](https://pytorch.org/blog/how-to-train-state-of-the-art-models-using-torchvision-latest-primitives/). Here, we have done grid searches on learning rate and weight decay and found the optimal hyper-parameter on the detection task.

|                       Backbone                       |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                       Config                       |                                                                                                                                                                               Download                                                                                                                                                                               |

| [R-50-TNR](./faster-rcnn_r50-tnr-pre_fpn_1x_coco.py) | pytorch |   1x    |    -     |                |  40.2  | [config](./faster-rcnn_r50-tnr-pre_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_tnr-pretrain_1x_coco/faster_rcnn_r50_fpn_tnr-pretrain_1x_coco_20220320_085147-efedfda4.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_tnr-pretrain_1x_coco/faster_rcnn_r50_fpn_tnr-pretrain_1x_coco_20220320_085147.log.json) |

```latex
}
```

# Mobilenetv2: Inverted residuals and linear bottlenecks

<details>

```bibtex
}
```

In this paper we describe a new mobile architecture, mbox{MobileNetV2}, that improves the state of the art performance of mobile models on multiple tasks and benchmarks as well as across a spectrum of different model sizes. We also describe efficient ways of applying these mobile models to object detection in a novel framework we call mbox{SSDLite}. Additionally, we demonstrate how to build mobile semantic segmentation models through a reduced form of mbox{DeepLabv3} which we call Mobile mbox{DeepLabv3}. is based on an inverted residual structure where the shortcut connections are between the thin bottleneck layers. The intermediate expansion layer uses lightweight depthwise convolutions to filter features as a source of non-linearity. Additionally, we find that it is important to remove non-linearities in the narrow layers in order to maintain representational power. We demonstrate that this improves performance and provide an intuition that led to this design. Finally, our approach allows decoupling of the input/output domains from the expressiveness of the transformation, which provides a convenient framework for further analysis. We measure our performance on mbox{ImageNet}~cite{Russakovsky:2015:ILS:2846547.2846559} classification, COCO object detection cite{COCO}, VOC image segmentation cite{PASCAL}. We evaluate the trade-offs between accuracy, and number of operations measured by multiply-adds (MAdd), as well as actual latency, and the number of parameters.

<img src="https://user-images.githubusercontent.com/15977946/146537871-75c22a93-72ef-4a24-9ce9-0c2823381961.png">
</div>

<details>

```bibtex
}
```

# Preparing Vimeo90K Dataset

```bibtex
}
```

The training and test datasets can be downloaded from [here](http://toflow.csail.mit.edu/).

Then you can rename the directory `vimeo_septuplet/sequences` to  `vimeo90k/GT`. The Vimeo90K dataset has a `clip/sequence/img` folder structure:

```text
vimeo90k
├── GT
│   ├── 00001
│   │   ├── 0001
│   │   │   ├── im1.png
│   │   │   ├── im2.png
│   │   │   ├── ...
│   │   ├── 0002
│   │   ├── 0003
│   │   ├── ...
│   ├── 00002
│   ├── ...
├── sep_trainlist.txt
├── sep_testlist.txt
```

To generate the downsampling images BIx4 and BDx4 and prepare the annotation file, you need to run the following command:

│   ├── 00002
│   ├── ...
```

```shell
python tools/dataset_converters/vimeo90k/preprocess_vimeo90k_dataset.py --data-root ./data/vimeo90k
```

```text
mmagic
├── mmagic
├── tools
├── data
│   │   ├── GT
│   │   │   ├── ...
│   │   ├── BIx4
│   │   ├── BDx4
│   │   ├── meta_info_Vimeo90K_test_GT.txt
│   │   ├── meta_info_Vimeo90K_train_GT.txt
```

## Prepare LMDB dataset for Vimeo90K

```shell

```

```shell
python tools/dataset_converters/vimeo90k/preprocess_vimeo90k_dataset.py --data-root ./data/vimeo90k --train_list ./data/vimeo90k/sep_trainlist.txt --gt-path ./data/vimeo90k/GT --lq-path ./data/Vimeo90k/BIx4  --make-lmdb
```

# S2ANet

> [Align Deep Features for Oriented Object Detection](https://ieeexplore.ieee.org/document/9377550)

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/s2a.png" width="800"/>
</div>

The past decade has witnessed significant progress on detecting objects in aerial images that are often distributed with large-scale variations and arbitrary orientations. However, most of existing methods rely on heuristically defined anchors with different scales, angles, and aspect ratios, and usually suffer from severe misalignment between anchor boxes (ABs) and axis-aligned convolutional features, which lead to the common inconsistency between the classification score and localization accuracy. To address this issue, we propose a single-shot alignment network (S²A-Net) consisting of two modules: a feature alignment module (FAM) and an oriented detection module (ODM). The FAM can generate high-quality anchors with an anchor refinement network and adaptively align the convolutional features according to the ABs with a novel alignment convolution. The ODM first adopts active rotating filters to encode the orientation information and then produces orientation-sensitive and orientation-invariant features to alleviate the inconsistency between classification score and localization accuracy. Besides, we further explore the approach to detect objects in large-size images, which leads to a better trade-off between speed and accuracy. Extensive experiments demonstrate that our method can achieve the state-of-the-art performance on two commonly used aerial objects' data sets (i.e., DOTA and HRSC2016) while keeping high efficiency.

DOTA1.0

|         Backbone         |  mAP  | Angle | lr schd | Mem (GB) | Inf Time (fps) | Aug | Batch Size |                                                       Configs                                                        |                                                                                                                                                                                  Download                                                                                                                                                                                  |

| ResNet50 (1024,1024,200) | 69.79 | le135 |   1x    |   3.38   |      17.2      |  -  |     2      | [rotated-retinanet-rbox-le135_r50_fpn_1x_dota](../rotated_retinanet/rotated-retinanet-rbox-le135_r50_fpn_1x_dota.py) | [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le135/rotated_retinanet_obb_r50_fpn_1x_dota_le135-e4131166.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le135/rotated_retinanet_obb_r50_fpn_1x_dota_le135_20220128_130755.log.json) |

| ResNet50 (1024,1024,200) | 73.91 | le135 |   1x    |   3.14   |      15.5      |  -  |     2      |                          [s2anet-le135_r50_fpn_1x_dota](./s2anet-le135_r50_fpn_1x_dota.py)                           |                                          [model](https://download.openmmlab.com/mmrotate/v0.1.0/s2anet/s2anet_r50_fpn_1x_dota_le135/s2anet_r50_fpn_1x_dota_le135-5dfcf396.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/s2anet/s2anet_r50_fpn_1x_dota_le135/s2anet_r50_fpn_1x_dota_le135_20220124_163529.log.json)                                          |

| ResNet50 (1024,1024,200) | 74.19 | le135 |   1x    |   2.17   |      17.4      |  -  |     2      |                      [s2anet-le135_r50_fpn_amp-1x_dota](./s2anet-le135_r50_fpn_amp-1x_dota.py)                       |                                [model](https://download.openmmlab.com/mmrotate/v0.1.0/s2anet/s2anet_r50_fpn_fp16_1x_dota_le135/s2anet_r50_fpn_fp16_1x_dota_le135-5cac515c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/s2anet/s2anet_r50_fpn_fp16_1x_dota_le135/s2anet_r50_fpn_fp16_1x_dota_le135_20220303_194910.log.json)                                |

HRSC

|      Backbone      |  mAP  | Angle | lr schd | Mem (GB) | Inf Time (fps) | Aug | Batch Size |                              Configs                              | Download |
| :----------------: | :---: | :---: | :-----: | :------: | :------------: | :-: | :--------: | :---------------------------------------------------------------: | :------: |

| ResNet50 (800,800) | 89.75 | le90  |   3x    |          |                |  -  |     2      | [s2anet_r50_fpn_3x_hrsc_le135](./s2anet_r50_fpn_3x_hrsc_le135.py) |          |

```
@article{han2021align,
  title={Align deep features for oriented object detection},
  author={Han, Jiaming and Ding, Jian and Li, Jie and Xia, Gui-Song},
  journal={IEEE Transactions on Geoscience and Remote Sensing},
}
```

# 高级训练设置

## 恢复训练

恢复训练是指从之前某次训练保存下来的状态开始继续训练，这里的状态包括模型的权重、优化器和优化器参数调整策略的状态。

### 自动恢复训练

用户可以在训练命令最后加上 `--resume` 恢复训练，程序会自动从 `work_dirs` 中加载最新的权重文件恢复训练。如果 `work_dir` 中有最新的 `checkpoint`（例如该训练在上一次训练时被中断），则会从该 `checkpoint` 恢复训练，否则（例如上一次训练还没来得及保存 `checkpoint` 或者启动了新的训练任务）会重新开始训练。

下面是一个恢复训练的示例:

```shell
```

### 指定 Checkpoint 恢复训练

你也可以对 `--resume` 指定 `checkpoint` 路径，MMPose 会自动读取该 `checkpoint` 并从中恢复训练，命令如下：

```shell
```

如果你希望手动在配置文件中指定 `checkpoint` 路径，除了设置 `resume=True`，还需要设置 `load_from` 参数。需要注意的是，如果只设置了 `load_from` 而没有设置 `resume=True`，则只会加载 `checkpoint` 中的权重并重新开始训练，而不是接着之前的状态继续训练。

下面的例子与上面指定 `--resume` 参数的例子等价：

```python

```python
    ## 内容省略 ##
    )
```

## 自动混合精度（AMP）训练

混合精度训练在不改变模型、不降低模型训练精度的前提下，可以缩短训练时间，降低存储需求，因而能支持更大的 batch size、更大模型和尺寸更大的输入的训练。

如果要开启自动混合精度（AMP）训练，在训练命令最后加上 --amp 即可， 命令如下：

```shell
```

具体例子如下：

```shell
```

## 设置随机种子

如果想要在训练时指定随机种子，可以使用以下命令：

```shell
    ${CONFIG} \                               # 配置文件路径
    --cfg-options randomness.seed=2023 \      # 设置随机种子为 2023
    [randomness.diff_rank_seed=True] \        # 根据 rank 来设置不同的种子。
    [randomness.deterministic=True]           # 把 cuDNN 后端确定性选项设置为 True
# [] 代表可选参数，实际输入命令行时，不用输入 []
```

randomness 有三个参数可设置，具体含义如下：

- `randomness.seed=2023` ，设置随机种子为 `2023`。

[randomness.deterministic=True]           # 把 cuDNN 后端确定性选项设置为 True
```

- `randomness.diff_rank_seed=True`，根据 `rank` 来设置不同的种子，`diff_rank_seed` 默认为 `False`。

- `randomness.deterministic=True`，把 `cuDNN` 后端确定性选项设置为 `True`，即把 `torch.backends.cudnn.deterministic` 设为 `True`，把 `torch.backends.cudnn.benchmark` 设为 `False`。`deterministic` 默认为 `False`。更多细节见 [Pytorch Randomness](https://pytorch.org/docs/stable/notes/randomness.html)。

如果你希望手动在配置文件中指定随机种子，可以在配置文件中设置 `random_seed` 参数，具体如下：

```python
randomness = dict(seed=2023)
    ## 内容省略 ##
    )
```

## 使用 Tensorboard 可视化训练过程

安装 Tensorboard 环境

```shell
pip install tensorboard
```

在 config 文件中添加 tensorboard 配置

```python
visualizer = dict(vis_backends=[dict(type='LocalVisBackend'),dict(type='TensorboardVisBackend')])
```

```shell
```

```python
```

运行训练命令后，tensorboard 文件会生成在可视化文件夹 `work_dir/${CONFIG}/${TIMESTAMP}/vis_data` 下，运行下面的命令就可以在网页链接使用 tensorboard 查看 loss、学习率和精度等信息。

```shell
tensorboard --logdir work_dir/${CONFIG}/${TIMESTAMP}/vis_data
```

# 代码结构变动

MMOCR 为了兼顾文本检测、识别和关键信息提取等任务，在初版设计时存在许多欠缺考虑的地方。在本次 1.0 版本的升级中，MMOCR 同步提出了新的模型架构，旨在尽量与 OpenMMLab 整体的设计对齐，且在算法库内部达成结构上的统一。虽然本次升级并非完全后向兼容，但所有的变动都是有迹可循的。因此，我们在本章节总结出了开发者可能会关心的改动，供有需要的用户参考。

## 整体改动

MMOCR 0.x 存在着对模块功能边界定义不清晰的问题。在 MMOCR 1.0 中，我们重构了模型模块的设计，并定义了它们的模块边界。

- 考虑到方向差异过大，MMOCR 1.0 中取消了对命名实体识别的支持。

- 模型中计算损失（loss）的部分模块被抽象化为 Module Loss，转换原始标注为损失目标（loss target）的功能也被包括在内。另一个模块抽象 Postprocessor 则负责在预测时解码模型原始输出为对应任务的 `DataSample`。

- 所有模型的输入简化为包含图像原始特征的 `inputs` 和图片元信息的 `List[DataSample]`。输出格式也得到统一，训练时是包含 loss 的字典，测试时的输出为包含预测结果的对应任务的 [`DataSample`](<>)。

- Module Loss 来源于 0.x 版本中实现与单个模型强相关的 `XXLoss` 类，它们在 1.0 中均被统一重命名为`XXModuleLoss`的形式（如`DBLoss` 被重命名为 `DBModuleLoss`）, `head` 传入的 loss 配置参数名也从 `loss` 改为 `module_loss`。

- 与模型实现无关的通用损失类名称保持 `XXLoss` 的形式，并放置于 `mmocr/models/common/losses` 下，如 [`MaskedBCELoss`](mmocr.models.common.losses.MaskedBCELoss)。

- `mmocr/models/common/losses` 下的改动：0.x 中 `DiceLoss` 被重名为 [`MaskedDiceLoss`](mmocr.models.common.losses.MaskedDiceLoss)。`FocalLoss` 被移除。

- 增加了起源于 label converter 的 Dictionary 模块，它会在文本识别和关键信息提取任务中被用到。

## 文本检测

### 关键改动（太长不看版）

- 旧版的模型权重仍然适用于新版，但需要将权重字典 `state_dict` 中以 `bbox_head` 开头的字段重命名为 `det_head`。

- 计算 target 有关的变换 `XXTargets` 被转移到了 `XXModuleLoss` 中。

- 原本继承链为 `mmdet.BaseDetector->SingleStageDetector->SingleStageTextDetector`，现在改为直接继承自 `BaseDetector`, 中间的 `SingleStageDetector` 被删除。

- `bbox_head` 改名为 `det_head`。

- `train_cfg`、`test_cfg`和`pretrained`字段被移除。

- `forward_train()` 与 `simple_test()` 分别被重构为 `loss()` 与 `predict()` 方法。其中 `simple_test()` 中负责将模型原始输出拆分并输入 `head.get_bounary()` 的部分被整合进了 `BaseTextDetPostProcessor` 中。

- `TextDetectorMixin` 中只实现了 `show_result()`方法，实现与 `TextDetLocalVisualizer` 重合，因此已经被移除。

### Head

### Head

- `HeadMixin` 为`XXXHead` 在 0.x 版本中必须继承的基类，现在被 `BaseTextDetHead` 代替。里面的 `get_boundary()` 和 `resize_boundary()` 方法被重写为 `BaseTextDetPostProcessor` 的 `__call__()` 和 `rescale()` 方法。

- 文本检测中特有的数据变换 `XXXTargets` 全部移动到 `XXXModuleLoss._get_target_single` 中，与生成 target 相关的配置不再在数据流水线（pipeline）中设置，转而在 `XXXLoss` 中被配置。例如，`DBNetTargets` 的实现被移动到 `DBModuleLoss._get_target_single()`中，而用户可以通过设置 `DBModuleLoss` 的初始化参数来控制损失目标的生成。

- 原本的 `XXXPostprocessor.__call__()` 中的逻辑转移到重构后的 `XXXPostprocessor.get_text_instances()` 。

- `BasePostprocessor` 重构为 `BaseTextDetPostProcessor`，此基类会将模型输出的预测结果拆分并逐个进行处理，并支持根据 `scale_factor` 自动缩放输出的多边形（polygon）或界定框（bounding box）。

## 文本识别

- 由于字典序发生了变化，且存在部分模型架构上的 bug 被修复，旧版的识别模型权重已经不再能直接应用于 1.0 中，我们将会在后续为有需要的用户推出迁移脚本教程。

- 0.x 版本中的 SegOCR 支持暂时移除，TPS-CRNN 会在后续版本中被支持。

- 测试时增强（test time augmentation）在此版本中暂未支持，但将会在后续版本中更新。

- Label converter 模块被移除，里面的功能被拆分至 Dictionary, ModuleLoss 和 Postprocessor 模块中。

- 统一模型中对 `max_seq_len` 的定义为模型的原始输出长度。

- 原有的 label converter 存在拼写错误 (label convertor)，我们通过删除掉这个类规避了这个问题。

- 负责对字符/字符串与数字索引互相转换的部分被提取至 [`Dictionary`](mmocr.models.common.Dictionary) 类中。

- 在旧版本中，不同的 label converter 会有不一样的特殊字符集和字符序。在 0.x 版本中，字符序如下：

  | Converter                       | 字符序                                    |
  | ------------------------------- | ----------------------------------------- |
  | `AttnConvertor`, `ABIConvertor` | `<UKN>`, `<BOS/EOS>`, `<PAD>`, characters |
  | `CTCConvertor`                  | `<BLK>`, `<UKN>`, characters              |

在 1.0 中，我们不再以任务为边界设计不同的字典和字符序，取而代之的是统一了字符序的 Dictionary，其字符序为 characters, \<BOS/EOS>, \<PAD>, \<UKN>。`CTCConvertor` 中 \<BLK> 被等价替换为 \<PAD>。

- `label_convertor` 中原本支持三种方式初始化字典：`dict_type`、`dict_file` 和 `dict_list`，现在在 `Dictionary` 中被简化为 `dict_file` 一种。同时，我们也把原本在 `dict_type` 中支持的字典格式转化为现在 `dicts/` 目录下的预设字典文件。对应映射如下：

  | MMOCR 0.x: `dict_type` | MMOCR 1.0: 字典路径                    |

- `label_converter` 中 `str2tensor()` 的实现被转移到 `ModuleLoss.get_targets()` 中。下面的表格列出了旧版与新版方法实现的对应关系。注意，新旧版的实现并非完全一致。

  | MMOCR 0.x                                                 | MMOCR 1.0                               | 备注                                         |

| MMOCR 0.x                                                 | MMOCR 1.0                               | 备注                                         |
  | --------------------------------------------------------- | --------------------------------------- | -------------------------------------------- |
  | `ABIConvertor.str2tensor()`, `AttnConvertor.str2tensor()` | `BaseTextRecogModuleLoss.get_targets()` | 原本两个类中的实现存在的差异在新版本中被统一 |
  | `CTCConvertor.str2tensor()`                               | `CTCModuleLoss.get_targets()`           |                                              |

- `label_converter` 中 `tensor2idx()` 的实现被转移到 `Postprocessor.get_single_prediction()` 中。下面的表格列出了旧版与新版方法实现的对应关系。注意，新旧版的实现并非完全一致。

  | MMOCR 0.x                                                 | MMOCR 1.0                                        |

| --------------------------------------------------------- | ------------------------------------------------ |

## 关键信息提取

- 由于模型的输入发生了变化，旧版模型的权重已经不再能直接应用于 1.0 中。

- 读取数据的部分被简化到 `WildReceiptDataset` 中。

- 对节点和边作额外处理的部分被转移到了 `LoadKIEAnnotation` 中。

- 使用字典对文本进行转化的部分被转移到了 `SDMGRHead.convert_text()` 中，使用 `Dictionary` 实现。

- 计算文本框之间关系的部分`compute_relation()` 被转移到 `SDMGRHead.compute_relations()` 中，在模型内进行。

- 评估模型表现的部分被简化为 `F1Metric`。

- `OpensetKIEDataset` 中处理模型边输出的部分被整理到 `SDMGRPostProcessor`中。

### SDMGR

- `show_result()` 被整合到 `KIEVisualizer` 中。

- `forward_test()` 中对输出进行后处理的部分被整理到 `SDMGRPostProcessor`中。

## Utils 变动

原本散布在各处的功能函数现已被统一归类在 `mmocr/utils/` 下。以下为该目录下各文件的作用域：

### SDMGR

## Utils 变动

- bbox_utils.py：四边界定框（bounding box）有关的功能函数。
- check_argument.py：检查参数类型的功能函数。
- collect_env.py：收集运行环境的功能函数。
- data_converter_utils.py：用于数据集转换的功能函数。
- fileio.py：输入/输出有关的功能函数。
- img_utils.py：处理图片的功能函数。
- mask_utils.py：与掩码有关的功能函数。
- ocr.py：用于 MMOCR 推理的功能函数。
- parsers.py：解码文件的功能函数。
- polygon_utils.py：多边形的功能函数。
- setup_env.py：存放初始化 MMOCR 的功能函数。
- string_utils.py：存放字符串的功能函数。
- typing.py：存放 MMOCR 中常用数据类型的缩写。

<details>

```bibtex
}
```

<details>

```bibtex

| [pose_hrnetv2_w18](/configs/hand_2d_keypoint/topdown_heatmap/onehand10k/td-hm_hrnetv2-w18_8xb64-210e_onehand10k-256x256.py) |  256x256   |  0.990  | 0.567 | 24.26 | [ckpt](https://download.openmmlab.com/mmpose/hand/hrnetv2/hrnetv2_w18_onehand10k_256x256-30bc9c6b_20210330.pth) | [log](https://download.openmmlab.com/mmpose/hand/hrnetv2/hrnetv2_w18_onehand10k_256x256_20210330.log.json) |

We provide a demo script to test a single image or video with hand detectors and top-down pose estimators. Assume that you have already installed [mmdet](https://github.com/open-mmlab/mmdetection) with version >= 3.0.

**Hand Box Model Preparation:** The pre-trained hand box estimation model can be found in [mmdet model zoo](/demo/docs/en/mmdet_modelzoo.md#hand-bounding-box-detection-models).

### 2D Hand Image Demo

```shell
```

```

The pre-trained hand pose estimation model can be downloaded from [model zoo](https://mmpose.readthedocs.io/en/latest/model_zoo/hand_2d_keypoint.html).
Take [onehand10k model](https://download.openmmlab.com/mmpose/hand/hrnetv2/hrnetv2_w18_onehand10k_256x256-30bc9c6b_20210330.pth) as an example:

```shell

https://download.openmmlab.com/mmpose/hand/hrnetv2/hrnetv2_w18_onehand10k_256x256-30bc9c6b_20210330.pth \
```

```shell

```

```shell
```

### 2D Hand Keypoints Video Demo

--input tests/data/onehand10k/9.jpg \
```

```shell
```

--output-root vis_results --show --draw-heatmap
```

The original video can be downloaded from [Github](https://raw.githubusercontent.com/open-mmlab/mmpose/master/tests/data/nvgesture/sk_color.avi).

### 2D Hand Keypoints Demo with Inferencer

```shell
```

This command infers all images located in `tests/data/onehand10k` and saves the visualization results in the `vis_results/onehand10k` directory.

--bbox-thr 0.5 --kpt-thr 0.05
```

For 2D hand keypoint estimation models, try to edit the config file. For example, set `model.test_cfg.flip_test=False` in [onehand10k_hrnetv2](../../configs/hand_2d_keypoint/topdown_heatmap/onehand10k/td-hm_hrnetv2-w18_8xb64-210e_onehand10k-256x256.py#90).

# Shufflenet v2: Practical guidelines for efficient cnn architecture design

<details>

```bibtex
}
```

}
```

Current network architecture design is mostly guided by the indirect metric of computation complexity, i.e., FLOPs. However, the direct metric, such as speed, also depends on the other factors such as memory access cost and platform characterics. Taking these factors into account, this work proposes practical guidelines for efficient network de- sign. Accordingly, a new architecture called ShuffleNet V2 is presented. Comprehensive experiments verify that it is the state-of-the-art in both speed and accuracy.

<img src="https://user-images.githubusercontent.com/15977946/146540752-993df527-7ec3-475c-ba12-3791bf8b3a0c.png">
</div>

1. Set up the OpenCompass environment:

   ```bash
   ```

   If you want to customize the PyTorch version or related CUDA version, please refer to the [official documentation](https://pytorch.org/get-started/locally/) to set up the PyTorch environment. Note that OpenCompass requires `pytorch>=1.13`.

2. Install OpenCompass:

   ```bash
   ```

3. Install humaneval (Optional)

   If you want to **evaluate your models coding ability on the humaneval dataset**, follow this step.

   <details>
   <summary><b>click to show the details</b></summary>

   ```bash
   cd ..
   ```

```bash
   cd ..
   ```

   Please read the comments in `human_eval/execution.py` **lines 48-57** to understand the potential risks of executing the model generation code. If you accept these risks, uncomment **line 58** to enable code execution evaluation.

4. Install Llama (Optional)

   If you want to **evaluate Llama / Llama-2 / Llama-2-chat with its official implementation**, follow this step.

   <details>

   ```bash
   cd llama
   cd ..
   ```

   You can find example configs in `configs/models`. ([example](https://github.com/InternLM/opencompass/blob/eb4822a94d624a4e16db03adeb7a59bbd10c2012/configs/models/llama2_7b_chat.py))

The datasets supported by OpenCompass mainly include two parts:

1. Huggingface datasets: The [Huggingface Datasets](https://huggingface.co/datasets) provide a large number of datasets, which will **automatically download** when running with this option.
2. Custom dataset: OpenCompass also provides some Chinese custom **self-built** datasets. Please run the following command to **manually download and extract** them.

Run the following commands to download and place the datasets in the `${OpenCompass}/data` directory can complete dataset preparation.

```bash
# Run in the OpenCompass directory
```

OpenCompass has supported most of the datasets commonly used for performance comparison, please refer to `configs/dataset` for the specific list of supported datasets.

# Quick Start

```

The evaluation of OpenCompass relies on configuration files which must contain fields **`datasets`** and **`models`**.
The configurations specify the models and datasets to evaluate using **"run.py"**.

We will demonstrate some basic features of OpenCompass through evaluating pretrained models [OPT-125M](https://huggingface.co/facebook/opt-125m) and [OPT-350M](https://huggingface.co/facebook/opt-350m) on both [SIQA](https://huggingface.co/datasets/social_i_qa) and [Winograd](https://huggingface.co/datasets/winogrande) benchmark tasks with their config file located at [configs/eval_demo.py](https://github.com/InternLM/opencompass/blob/main/configs/eval_demo.py).

Before running this experiment, please make sure you have installed OpenCompass locally and it should run successfully under one _GTX-1660-6G_ GPU.
For larger parameterized models like Llama-7B, refer to other examples provided in the [configs directory](https://github.com/InternLM/opencompass/tree/main/configs).

Since OpenCompass launches evaluation processes in parallel by default, we can start the evaluation for the first run and check if there is any prblem. In debugging mode, the tasks will be executed sequentially and the status will be printed in real time.

```bash
```

If everything is fine, you should see "Starting inference process" on screen:

```bash
```

Then you can press `ctrl+c` to interrupt the program, and then run the following command to start the parallel evaluation:

```bash

```

```bash
```

Now let's go over the configuration file and the launch options used in this case.

## Explanations

### Dataset list - `datasets`

Below is the configuration snippet related to datasets in `configs/eval_demo.py`:

```python
from mmengine.config import read_base  # Use mmengine.read_base() to load base configs

    # Read the required dataset configurations directly from the preset dataset configurations
    from .datasets.winograd.winograd_ppl import winograd_datasets   # Load Winograd's configuration, which uses perplexity-based inference
    from .datasets.siqa.siqa_gen import siqa_datasets               # Load SIQA's configuration, which uses generation-based inference

from .datasets.siqa.siqa_gen import siqa_datasets               # Load SIQA's configuration, which uses generation-based inference

datasets = [*siqa_datasets, *winograd_datasets]   # Concatenate the datasets to be evaluated into the datasets field
```

Various dataset configurations are available in [configs/datasets](https://github.com/InternLM/OpenCompass/blob/main/configs/datasets).
Some datasets have two types of configuration files within their folders named `ppl` and `gen`, representing different evaluation methods. Specifically, `ppl` represents discriminative evaluation, while `gen` stands for generative evaluation.

[configs/datasets/collections](https://github.com/InternLM/OpenCompass/blob/main/configs/datasets/collections) contains various collections of datasets for comprehensive evaluation purposes.

You can find more information from [Dataset Preparation](./user_guides/dataset_prepare.md).

### Model list - `models`

OpenCompass supports directly specifying the list of models to be tested in the configuration. For HuggingFace models, users usually do not need to modify the code. The following is the relevant configuration snippet:

```python
# Evaluate models supported by HuggingFace's `AutoModelForCausalLM` using `HuggingFaceCausalLM`

# OPT-350M
       # Initialization parameters for `HuggingFaceCausalLM`
       # Below are common parameters for all models, not specific to HuggingFaceCausalLM

proxies=None,
       abbr='opt350m',               # Model abbreviation for result display
       max_seq_len=2048,             # The maximum length of the entire sequence
       max_out_len=100,              # Maximum number of generated tokens
       batch_size=64,                # batchsize
       run_cfg=dict(num_gpus=1),     # Run configuration for specifying resource requirements
    )

# OPT-125M

padding_side='left',
       abbr='opt125m',                # Model abbreviation for result display
       max_seq_len=2048,              # The maximum length of the entire sequence
       max_out_len=100,               # Maximum number of generated tokens
       batch_size=128,                # batchsize
       run_cfg=dict(num_gpus=1),      # Run configuration for specifying resource requirements
    )

```

The pretrained models 'facebook/opt-350m' and 'facebook/opt-125m' will be automatically downloaded from HuggingFace during the first run.

More information about model configuration can be found in [Prepare Models](./user_guides/models.md).

### Launch Evaluation

When the config file is ready, we can start the task in **debug mode** to check for any exceptions in model loading, dataset reading, or incorrect cache usage.

```shell
```

However, in `--debug` mode, tasks are executed sequentially. After confirming that everything is correct, you
can disable the `--debug` mode to fully utilize multiple GPUs.

```shell
```

Here are some parameters related to evaluation that can help you configure more efficient inference tasks based on your environment:

- `-w outputs/demo`: Directory to save evaluation logs and results.
- `-r`: Restart the previous (interrupted) evaluation.
- `--mode all`: Specify a specific stage of the task.
  - all: Perform a complete evaluation, including inference and evaluation.

  - infer: Perform inference on each dataset.
  - eval: Perform evaluation based on the inference results.
  - viz: Display evaluation results only.
- `--max-partition-size 2000`: Dataset partition size. Some datasets may be large, and using this parameter can split them into multiple sub-tasks to efficiently utilize resources. However, if the partition is too fine, the overall speed may be slower due to longer model loading times.
- `--max-num-workers 32`: Maximum number of parallel tasks. In distributed environments such as Slurm, this parameter specifies the maximum number of submitted tasks. In a local environment, it specifies the maximum number of tasks executed in parallel. Note that the actual number of parallel tasks depends on the available GPU resources and may not be equal to this number.

If you are not performing the evaluation on your local machine but using a Slurm cluster, you can specify the following parameters:

- `--slurm`: Submit tasks using Slurm on the cluster.
- `--partition(-p) my_part`: Slurm cluster partition.
- `--retry 2`: Number of retries for failed tasks.

```{tip}
The entry also supports submitting tasks to Alibaba Deep Learning Center (DLC), and more customized evaluation strategies. Please refer to [Launching an Evaluation Task](./user_guides/experimentation.md#launching-an-evaluation-task) for details.
```

## Obtaining Evaluation Results

After the evaluation is complete, the evaluation results table will be printed as follows:

```text
```

```

All run outputs will be directed to `outputs/demo/` directory with following structure:

```text
├── 20230220_183030     # one experiment pre folder
│   ├── configs         # Dumped config files for record. Multiple configs may be kept if different experiments have been re-run on the same experiment folder
│   ├── logs            # log files for both inference and evaluation stages
│   │   ├── eval
│   ├── predictions   # Prediction results for each task
│   ├── results       # Evaluation results for each task
│   └── summary       # Summarized evaluation results for a single experiment
├── ...
```

The summarization process can be further customized in configuration and output the averaged score of some benchmarks (MMLU, C-Eval, etc.).

├── ...
```

More information about obtaining evaluation results can be found in [Results Summary](./user_guides/summarizer.md).

## Additional Tutorials

To learn more about using OpenCompass, explore the following tutorials:

- [Prepare Datasets](./user_guides/dataset_prepare.md)
- [Prepare Models](./user_guides/models.md)
- [Task Execution and Monitoring](./user_guides/experimentation.md)
- [Understand Prompts](./prompt/overview.md)
- [Results Summary](./user_guides/summarizer.md)
- [Learn about Config](./user_guides/config.md)

# Tutorial 8: Deploy models in MMagic

The deployment of OpenMMLab codebases, including MMClassification, MMDetection, MMagic and so on are supported by [MMDeploy](https://github.com/open-mmlab/mmdeploy).
The latest deployment guide for MMagic can be found from [here](https://mmdeploy.readthedocs.io/en/latest/04-supported-codebases/mmagic.html).

- [Tutorial 8: Deploy models in MMagic](#tutorial-8-deploy-models-in-mmagic)

- [Backend model inference](#backend-model-inference)

Please follow the [guide](../get_started/install.md) to install mmagic. And then install mmdeploy from source by following [this](https://mmdeploy.readthedocs.io/en/latest/get_started.html#installation) guide.

```{note}
```

Suppose MMagic and mmdeploy repositories are in the same directory, and the working directory is the root path of MMagic.

Take [ESRGAN](../../../configs/esrgan/esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py) model as an example.

You can download its checkpoint from [here](https://download.openmmlab.com/MMagic/restorers/esrgan/esrgan_psnr_x4c64b23g32_1x16_1000k_div2k_20200420-bf5c993c.pth), and then convert it to onnx model as follows:

```python

```

It is crucial to specify the correct deployment config during model conversion.MMDeploy has already provided builtin deployment config [files](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmagic) of all supported backends for mmagic, under which the config file path follows the pattern:

```
```

- **{task}:** task in mmagic.

Therefore, in the above example, you can also convert `ESRGAN` to other backend models by changing the deployment config file, e.g., converting to tensorrt-fp16 model by `super-resolution_tensorrt-fp16_dynamic-32x32-512x512.py`.

```{tip}
When converting mmagic models to tensorrt models, --device should be set to "cuda"
```

The converted model locates in the working directory like `mmdeploy_models/mmagic/onnx` in the previous example. It includes:

```
```

in which,

```
```

in which,

The whole package **mmdeploy_models/mmagic/onnx** is defined as **mmdeploy SDK model**, i.e., **mmdeploy SDK model** includes both backend model and inference meta information.

```python

```

```python
import cv2

output_file='output_restorer.bmp')
```

```python
import cv2

```

Besides python API, MMDeploy SDK also provides other FFI (Foreign Function Interface), such as C, C++, C#, Java and so on. You can learn their usage from [demos](https://github.com/open-mmlab/mmdeploy/tree/main/demo).

Please refer to [here](https://mmdeploy.readthedocs.io/en/latest/04-supported-codebases/mmagic.html#supported-models) for the supported model list.

# 准备 GoPro 数据集

```bibtex
}
```

训练数据集可以从 [此处](https://drive.google.com/file/d/1zgALzrLCC_tcXKu_iHQTHukKUVT1aodI/) 下载。测试数据集可以从 [此处](https://drive.google.com/file/d/1k6DTSHu4saUgrGTYkkZXTptILyG9RRll/) 下载。

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── GoPro
|   |   |   ├── input
|   |   |   ├── target
|   |   ├── test
```

# Empirical Attention

> [An Empirical Study of Spatial Attention Mechanisms in Deep Networks](https://arxiv.org/abs/1904.05873)

Attention mechanisms have become a popular component in deep neural networks, yet there has been little examination of how different influencing factors and methods for computing attention from these factors affect performance. Toward a better general understanding of attention mechanisms, we present an empirical study that ablates various spatial attention elements within a generalized attention formulation, encompassing the dominant Transformer attention as well as the prevalent deformable convolution and dynamic convolution modules. Conducted on a variety of applications, the study yields significant findings about spatial attention in deep networks, some of which run counter to conventional understanding. For example, we find that the query and key content comparison in Transformer attention is negligible for self-attention, but vital for encoder-decoder attention. A proper combination of deformable convolution with key content only saliency achieves the best accuracy-efficiency tradeoff in self-attention. Our results suggest that there exists much room for improvement in the design of attention mechanisms.

<img src="https://user-images.githubusercontent.com/40661020/143879619-f1817da9-1573-45c9-891d-cfe55ad54911.png"/>
</div>

| Backbone | Attention Component | DCN | Lr schd | Mem (GB) | Inf time (fps) | box AP |                         Config                          |                                                                                                                                                                                               Download                                                                                                                                                                                                |

|   R-50   |        1111         |  N  |   1x    |   8.0    |      13.8      |  40.0  |   [config](./faster-rcnn_r50-attn1111_fpn_1x_coco.py)   |         [model](https://download.openmmlab.com/mmdetection/v2.0/empirical_attention/faster_rcnn_r50_fpn_attention_1111_1x_coco/faster_rcnn_r50_fpn_attention_1111_1x_coco_20200130-403cccba.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/empirical_attention/faster_rcnn_r50_fpn_attention_1111_1x_coco/faster_rcnn_r50_fpn_attention_1111_1x_coco_20200130_210344.log.json)         |

|   R-50   |        0010         |  N  |   1x    |   4.2    |      18.4      |  39.1  |   [config](./faster-rcnn_r50-attn0010_fpn_1x_coco.py)   |         [model](https://download.openmmlab.com/mmdetection/v2.0/empirical_attention/faster_rcnn_r50_fpn_attention_0010_1x_coco/faster_rcnn_r50_fpn_attention_0010_1x_coco_20200130-7cb0c14d.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/empirical_attention/faster_rcnn_r50_fpn_attention_0010_1x_coco/faster_rcnn_r50_fpn_attention_0010_1x_coco_20200130_210125.log.json)         |

|   R-50   |        1111         |  Y  |   1x    |   8.0    |      12.7      |  42.1  | [config](./faster-rcnn_r50-attn1111-dcn_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/empirical_attention/faster_rcnn_r50_fpn_attention_1111_dcn_1x_coco/faster_rcnn_r50_fpn_attention_1111_dcn_1x_coco_20200130-8b2523a6.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/empirical_attention/faster_rcnn_r50_fpn_attention_1111_dcn_1x_coco/faster_rcnn_r50_fpn_attention_1111_dcn_1x_coco_20200130_204442.log.json) |

|   R-50   |        0010         |  Y  |   1x    |   4.2    |      17.1      |  42.0  | [config](./faster-rcnn_r50-attn0010-dcn_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/empirical_attention/faster_rcnn_r50_fpn_attention_0010_dcn_1x_coco/faster_rcnn_r50_fpn_attention_0010_dcn_1x_coco_20200130-1a2e831d.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/empirical_attention/faster_rcnn_r50_fpn_attention_0010_dcn_1x_coco/faster_rcnn_r50_fpn_attention_0010_dcn_1x_coco_20200130_210410.log.json) |

```latex
@article{zhu2019empirical,
  title={An Empirical Study of Spatial Attention Mechanisms in Deep Networks},
  author={Zhu, Xizhou and Cheng, Dazhi and Zhang, Zheng and Lin, Stephen and Dai, Jifeng},
  journal={arXiv preprint arXiv:1904.05873},
}
```

# 代码评测服务

我们支持评测多编程语言的数据集，类似 [humaneval-x](https://huggingface.co/datasets/THUDM/humaneval-x). 在启动之前需要确保你已经启动了代码评测服务，代码评测服务可参考[code-evaluator](https://github.com/Ezra-Yu/code-evaluator)项目。

## 启动代码评测服务

确保您已经安装了 docker，然后构建一个镜像并运行一个容器服务。

构建 Docker 镜像：

```shell
```

获取镜像后，使用以下命令创建容器：

```shell
# 输出日志格式

# 在后台运行程序

# 使用不同的端口
```

确保您能够访问服务，检查以下命令(如果在本地主机中运行服务，就跳过这个操作)：

```shell
```

```note
如果运算节点不能连接到评估服务，也可直接运行 `python run.py xxx...`，代码生成结果会保存在 'outputs' 文件夹下，迁移后直接使用 [code-evaluator](https://github.com/Ezra-Yu/code-evaluator) 评测得到结果（不需要考虑后面 eval_cfg 的配置）。
```

## 配置文件

```

```note
```

## 配置文件

我么已经给了 huamaneval-x 在 codegeex2 上评估的[配置文件](https://github.com/InternLM/opencompass/blob/main/configs/eval_codegeex2.py)。

其中数据集以及相关后处理的配置文件为这个[链接](https://github.com/InternLM/opencompass/tree/main/configs/datasets/humanevalx)， 需要注意 `humanevalx_eval_cfg_dict` 中的
`evaluator` 字段。

```python

input_columns=['prompt'], output_column='task_id', train_split='test')

}

    dict(

abbr=f'humanevalx-{lang}',
]
```

> [MaskFormer: Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://arxiv.org/abs/2107.06278)

<a href="https://github.com/facebookresearch/MaskFormer/">Official Repo</a>

<a href="https://github.com/open-mmlab/mmdetection/blob/dev-3.x/mmdet/models/dense_heads/maskformer_head.py#L21">Code Snippet</a>

Modern approaches typically formulate semantic segmentation as a per-pixel classification task, while instance-level segmentation is handled with an alternative mask classification. Our key insight: mask classification is sufficiently general to solve both semantic- and instance-level segmentation tasks in a unified manner using the exact same model, loss, and training procedure. Following this observation, we propose MaskFormer, a simple mask classification model which predicts a set of binary masks, each associated with a single global class label prediction. Overall, the proposed mask classification-based method simplifies the landscape of effective approaches to semantic and panoptic segmentation tasks and shows excellent empirical results. In particular, we observe that MaskFormer outperforms per-pixel classification baselines when the number of classes is large. Our mask classification-based method outperforms both current state-of-the-art semantic (55.6 mIoU on ADE20K) and panoptic segmentation (52.7 PQ on COCO) models.

<img src="https://user-images.githubusercontent.com/24582831/199215459-ea507126-aafe-4823-8eb1-ae6487509d5c.png" width="90%"/>
</div>

### Usage

- MaskFormer model needs to install [MMDetection](https://github.com/open-mmlab/mmdetection) first.

```shell
```

### ADE20K

</div>

### Usage

```shell
```

### ADE20K

| Method     | Backbone  | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device | mIoU  | mIoU(ms+flip) | config                                                                                                                                    | download                                                                                                                                                                                                                                                                                                                                                                                                     |

| MaskFormer | R-50-D32  | 512x512   | 160000  | 3.29     | A100           | 42.20  | 44.29 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/maskformer/maskformer_r50-d32_8xb2-160k_ade20k-512x512.py)        | [model](https://download.openmmlab.com/mmsegmentation/v0.5/maskformer/maskformer_r50-d32_8xb2-160k_ade20k-512x512/maskformer_r50-d32_8xb2-160k_ade20k-512x512_20221030_182724-3a9cfe45.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/maskformer/maskformer_r50-d32_8xb2-160k_ade20k-512x512/maskformer_r50-d32_8xb2-160k_ade20k-512x512_20221030_182724.json)                             |

| MaskFormer | R-101-D32 | 512x512   | 160000  | 4.12     | A100           | 34.90  | 45.11 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/maskformer/maskformer_r101-d32_8xb2-160k_ade20k-512x512.py)       | [model](https://download.openmmlab.com/mmsegmentation/v0.5/maskformer/maskformer_r101-d32_8xb2-160k_ade20k-512x512/maskformer_r101-d32_8xb2-160k_ade20k-512x512_20221031_223053-84adbfcb.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/maskformer/maskformer_r101-d32_8xb2-160k_ade20k-512x512/maskformer_r101-d32_8xb2-160k_ade20k-512x512_20221031_223053.json)                         |

| MaskFormer | Swin-T    | 512x512   | 160000  | 3.73     | A100           | 40.53  | 46.69 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/maskformer/maskformer_swin-t_upernet_8xb2-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/maskformer/maskformer_swin-t_upernet_8xb2-160k_ade20k-512x512/maskformer_swin-t_upernet_8xb2-160k_ade20k-512x512_20221114_232813-f14e7ce0.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/maskformer/maskformer_swin-t_upernet_8xb2-160k_ade20k-512x512/maskformer_swin-t_upernet_8xb2-160k_ade20k-512x512_20221114_232813.json) |

| MaskFormer | Swin-S    | 512x512   | 160000  | 5.33     | A100           | 26.98  | 49.36 | -             | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/maskformer/maskformer_swin-s_upernet_8xb2-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/maskformer/maskformer_swin-s_upernet_8xb2-160k_ade20k-512x512/maskformer_swin-s_upernet_8xb2-160k_ade20k-512x512_20221115_114710-723512c7.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/maskformer/maskformer_swin-s_upernet_8xb2-160k_ade20k-512x512/maskformer_swin-s_upernet_8xb2-160k_ade20k-512x512_20221115_114710.json) |

Note:

- All experiments of MaskFormer are implemented with 8 V100 (32G) GPUs with 2 samplers per GPU.
- The results of MaskFormer are relatively not stable.  The accuracy (mIoU) of model with `R-101-D32` is from 44.7 to 46.0, and with `Swin-S` is from 49.0 to 49.8.

```bibtex
@article{cheng2021per,
  title={Per-pixel classification is not all you need for semantic segmentation},
  author={Cheng, Bowen and Schwing, Alex and Kirillov, Alexander},
  pages={17864--17875},
}
```

# 训练 & 测试 (待更新)

## 测试一个模型

- 单个 GPU
- 单个节点多个 GPU
- 多个节点多个 GPU

您可以使用以下命令来推理数据集。

```shell
# 单个 GPU

# 多个 GPU

# slurm 环境中多个节点
```

例子:

在 DOTA-1.0 数据集推理 RotatedRetinaNet 并生成压缩文件用于在线[提交](https://captain-whu.github.io/DOTA/evaluation.html) (首先请修改 [data_root](https://github.com/open-mmlab/mmrotate/tree/main/configs/_base_/datasets/dotav1.py))。

```shell
```

或者

```shell

```

或者

```shell
```

您可以修改 [data_root](https://github.com/open-mmlab/mmrotate/tree/main/configs/_base_/datasets/dotav1.py) 中测试集的路径为验证集或训练集路径用于离线的验证。

```shell
```

或者

```shell
```

您也可以可视化结果。

```shell
```

## 训练一个模型

### 单 GPU 训练

```shell
```

checkpoints/SOME_CHECKPOINT.pth \
```

## 训练一个模型

### 单 GPU 训练

```shell
```

如果您想在命令行中指定工作路径，您可以增加参数 `--work_dir ${YOUR_WORK_DIR}`。

### 多 GPU 训练

```shell
```

可选参数包括:

- `--no-validate` (**不建议**): 默认情况下代码将在训练期间进行评估。通过设置 `--no-validate` 关闭训练期间进行评估。
- `--resume-from ${CHECKPOINT_FILE}`: 从以前的检查点恢复训练。

`resume-from` 和 `load-from` 的不同点：

`resume-from` 读取模型的权重和优化器的状态，并且 epoch 也会继承于指定的检查点。通常用于恢复意外中断的训练过程。
`load-from` 只读取模型的权重并且训练的 epoch 会从 0 开始。通常用于微调。

在第一台机器上:

```shell
```

在第二台机器上:

```shell
```

```

在第二台机器上:

```shell
```

如果您在 [slurm](https://slurm.schedmd.com/) 管理的集群上运行 MMRotate，您可以使用脚本 `slurm_train.sh` (此脚本还支持单机训练)。

```shell
```

如果您有多台机器联网，您可以参考 PyTorch [launch utility](https://pytorch.org/docs/stable/distributed_deprecated.html#launch-utility)。
如果您没有像 InfiniBand 这样的高速网络，训练速度通常会很慢。

如果您在一台机器上启动多个作业，如在一台机器上使用 8 张 GPU 训练 2 个作业，每个作业使用 4 张 GPU ，您需要为每个作业指定不同的端口号(默认为 29500 )进而避免通讯冲突。

如果您使用 `dist_train.sh` 启动训练，您可以在命令行中指定端口号。

```shell
```

如果您通过 Slurm 启动训练，您需要修改配置文件(通常是配置文件底部的第 6 行)进而设置不同的通讯端口。

```python

```

```python
```

```python
```

之后您可以使用 `config1.py` 和 `config2.py` 开启两个作业。

```shell
```

# VarifocalNet

> [VarifocalNet: An IoU-aware Dense Object Detector](https://arxiv.org/abs/2008.13367)

Accurately ranking the vast number of candidate detections is crucial for dense object detectors to achieve high performance. Prior work uses the classification score or a combination of classification and predicted localization scores to rank candidates. However, neither option results in a reliable ranking, thus degrading detection performance. In this paper, we propose to learn an Iou-aware Classification Score (IACS) as a joint representation of object presence confidence and localization accuracy. We show that dense object detectors can achieve a more accurate ranking of candidate detections based on the IACS. We design a new loss function, named Varifocal Loss, to train a dense object detector to predict the IACS, and propose a new star-shaped bounding box feature representation for IACS prediction and bounding box refinement. Combining these two new components and a bounding box refinement branch, we build an IoU-aware dense object detector based on the FCOS+ATSS architecture, that we call VarifocalNet or VFNet for short. Extensive experiments on MS COCO show that our VFNet consistently surpasses the strong baseline by ∼2.0 AP with different backbones. Our best model VFNet-X-1200 with Res2Net-101-DCN achieves a single-model single-scale AP of 55.1 on COCO test-dev, which is state-of-the-art among various object detectors.

<img src="https://user-images.githubusercontent.com/9102141/97464778-4b9ab000-197c-11eb-9283-ab2907ee0252.png"/>
</div>

**VarifocalNet (VFNet)** learns to predict the IoU-aware classification score which mixes the object presence confidence and localization accuracy together as the detection score for a bounding box. The learning is supervised by the proposed Varifocal Loss (VFL), based on a new star-shaped bounding box feature representation (the features at nine yellow sampling points). Given the new representation, the object localization accuracy is further improved by refining the initially regressed bounding box. The full paper is available at: [https://arxiv.org/abs/2008.13367](https://arxiv.org/abs/2008.13367).

|  Backbone   |  Style  | DCN | MS train | Lr schd | Inf time (fps) | box AP (val) | box AP (test-dev) |                           Config                            |                                                                                                                                                                               Download                                                                                                                                                                               |

|    R-50     | pytorch |  N  |    N     |   1x    |       -        |     41.6     |       41.6        |            [config](./vfnet_r50_fpn_1x_coco.py)             |                                                          [model](https://download.openmmlab.com/mmdetection/v2.0/vfnet/vfnet_r50_fpn_1x_coco/vfnet_r50_fpn_1x_coco_20201027-38db6f58.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/vfnet/vfnet_r50_fpn_1x_coco/vfnet_r50_fpn_1x_coco.json)                                                           |

|    R-50     | pytorch |  N  |    Y     |   2x    |       -        |     44.5     |       44.8        |           [config](./vfnet_r50_fpn_ms-2x_coco.py)           |                                          [model](https://download.openmmlab.com/mmdetection/v2.0/vfnet/vfnet_r50_fpn_mstrain_2x_coco/vfnet_r50_fpn_mstrain_2x_coco_20201027-7cc75bd2.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/vfnet/vfnet_r50_fpn_mstrain_2x_coco/vfnet_r50_fpn_mstrain_2x_coco.json)                                           |

|    R-50     | pytorch |  Y  |    Y     |   2x    |       -        |     47.8     |       48.0        |    [config](./vfnet_r50-mdconv-c3-c5_fpn_ms-2x_coco.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/vfnet/vfnet_r50_fpn_mdconv_c3-c5_mstrain_2x_coco/vfnet_r50_fpn_mdconv_c3-c5_mstrain_2x_coco_20201027pth-6879c318.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/vfnet/vfnet_r50_fpn_mdconv_c3-c5_mstrain_2x_coco/vfnet_r50_fpn_mdconv_c3-c5_mstrain_2x_coco.json)               |

|    R-101    | pytorch |  N  |    N     |   1x    |       -        |     43.0     |       43.6        |            [config](./vfnet_r101_fpn_1x_coco.py)            |                                                       [model](https://download.openmmlab.com/mmdetection/v2.0/vfnet/vfnet_r101_fpn_1x_coco/vfnet_r101_fpn_1x_coco_20201027pth-c831ece7.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/vfnet/vfnet_r101_fpn_1x_coco/vfnet_r101_fpn_1x_coco.json)                                                       |

|    R-101    | pytorch |  N  |    Y     |   2x    |       -        |     46.2     |       46.7        |          [config](./vfnet_r101_fpn_ms-2x_coco.py)           |                                       [model](https://download.openmmlab.com/mmdetection/v2.0/vfnet/vfnet_r101_fpn_mstrain_2x_coco/vfnet_r101_fpn_mstrain_2x_coco_20201027pth-4a5d53f1.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/vfnet/vfnet_r101_fpn_mstrain_2x_coco/vfnet_r101_fpn_mstrain_2x_coco.json)                                       |

|    R-101    | pytorch |  Y  |    Y     |   2x    |       -        |     49.0     |       49.2        |    [config](./vfnet_r101-mdconv-c3-c5_fpn_ms-2x_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/vfnet/vfnet_r101_fpn_mdconv_c3-c5_mstrain_2x_coco/vfnet_r101_fpn_mdconv_c3-c5_mstrain_2x_coco_20201027pth-7729adb5.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/vfnet/vfnet_r101_fpn_mdconv_c3-c5_mstrain_2x_coco/vfnet_r101_fpn_mdconv_c3-c5_mstrain_2x_coco.json)             |

| X-101-32x4d | pytorch |  Y  |    Y     |   2x    |       -        |     49.7     |       50.0        | [config](./vfnet_x101-32x4d-mdconv-c3-c5_fpn_ms-2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/vfnet/vfnet_x101_32x4d_fpn_mdconv_c3-c5_mstrain_2x_coco/vfnet_x101_32x4d_fpn_mdconv_c3-c5_mstrain_2x_coco_20201027pth-d300a6fc.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/vfnet/vfnet_x101_32x4d_fpn_mdconv_c3-c5_mstrain_2x_coco/vfnet_x101_32x4d_fpn_mdconv_c3-c5_mstrain_2x_coco.json) |

| X-101-64x4d | pytorch |  Y  |    Y     |   2x    |       -        |     50.4     |       50.8        | [config](./vfnet_x101-64x4d-mdconv-c3-c5_fpn_ms-2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/vfnet/vfnet_x101_64x4d_fpn_mdconv_c3-c5_mstrain_2x_coco/vfnet_x101_64x4d_fpn_mdconv_c3-c5_mstrain_2x_coco_20201027pth-b5f6da5e.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/vfnet/vfnet_x101_64x4d_fpn_mdconv_c3-c5_mstrain_2x_coco/vfnet_x101_64x4d_fpn_mdconv_c3-c5_mstrain_2x_coco.json) |

- The MS-train scale range is 1333x\[480:960\] (`range` mode) and the inference scale keeps 1333x800.
- Inference time will be updated soon.
- More results and pre-trained models can be found in [VarifocalNet-Github](https://github.com/hyz-xmaster/VarifocalNet)

```latex
@article{zhang2020varifocalnet,
  title={VarifocalNet: An IoU-aware Dense Object Detector},

```latex
  author={Zhang, Haoyang and Wang, Ying and Dayoub, Feras and S{\"u}nderhauf, Niko},
  journal={arXiv preprint arXiv:2008.13367},
}
```

}
```

<details>

```bibtex
}
```

| [pose_resnet50](/configs/hand_2d_keypoint/topdown_heatmap/rhd2d/td-hm_res50_8xb64-210e_rhd2d-256x256.py) |  256x256   |  0.991  | 0.898 | 2.32 | [ckpt](https://download.openmmlab.com/mmpose/hand/resnet/res50_rhd2d_256x256-5dc7e4cc_20210330.pth) | [log](https://download.openmmlab.com/mmpose/hand/resnet/res50_rhd2d_256x256_20210330.log.json) |

本教程收集了任何如何使用 MMDetection 进行 xxx 的答案。 如果您遇到有关`如何做`的问题及答案，请随时更新此文档！

## 使用 MMPretrain 的骨干网络

MMDet、MMPretrain、MMSeg 中的模型注册表都继承自 MMEngine 中的根注册表，允许这些存储库直接使用彼此已经实现的模块。 因此用户可以在 MMDetection 中使用来自 MMPretrain 的骨干网络，而无需实现MMPretrain 中已经存在的网络。

### 使用在 MMPretrain 中实现的骨干网络

假设想将 `MobileNetV3-small` 作为 `RetinaNet` 的骨干网络，则配置文件如下。

```python
_base_ = [
]
        _delete_=True, # 将 _base_ 中关于 backbone 的字段删除
        type='mmpretrain.MobileNetV3', # 使用 mmpretrain 中的 MobileNetV3

        out_indices=(3, 8, 11), # 修改 out_indices
            prefix='backbone.')), # mmpretrain 中骨干网络的预训练权重含义 prefix='backbone.'，为了正常加载权重，需要把这个 prefix 去掉。
    # 修改 in_channels
```

### 通过 MMPretrain 使用 TIMM 中实现的骨干网络

由于 MMPretrain 提供了 Py**T**orch **Im**age **M**odels (`timm`) 骨干网络的封装，用户也可以通过 MMPretrain 直接使用 `timm` 中的骨干网络。假设想将 [`EfficientNet-B1`](../../../configs/timm_example/retinanet_timm-efficientnet-b1_fpn_1x_coco.py) 作为 `RetinaNet` 的骨干网络，则配置文件如下。

```python
# https://github.com/open-mmlab/mmdetection/blob/main/configs/timm_example/retinanet_timm_efficientnet_b1_fpn_1x_coco.py
_base_ = [

```python
_base_ = [
]

        type='mmpretrain.TIMMBackbone', # 使用 mmpretrain 中 timm 骨干网络
        out_indices=(1, 2, 3, 4)), # 修改 out_indices
    neck=dict(in_channels=[24, 40, 112, 320])) # 修改 in_channels

```

out_indices=(1, 2, 3, 4)), # 修改 out_indices

```

`type='mmpretrain.TIMMBackbone'` 表示在 MMDetection 中使用 MMPretrain 中的 `TIMMBackbone` 类，并且使用的模型为` EfficientNet-B1`，其中 `mmpretrain` 表示 MMPretrain 库，而 `TIMMBackbone ` 表示 MMPretrain 中实现的 TIMMBackbone 包装器。

关于层次注册器的具体原理可以参考 [MMEngine 文档](https://mmengine.readthedocs.io/zh_cn/latest/tutorials/config.md#跨项目继承配置文件)，关于如何使用 MMPretrain 中的其他 backbone，可以参考 [MMPretrain 文档](https://mmpretrain.readthedocs.io/en/latest/user_guides/config.html)。

## 使用马赛克数据增强

如果你想在训练中使用 `Mosaic`，那么请确保你同时使用 `MultiImageMixDataset`。以 `Faster R-CNN` 算法为例，你可以通过如下做法实现：

```python
# 直接打开 configs/faster_rcnn/faster-rcnn_r50_fpn_1x_coco.py ,增添如下字段

    dict(

    dict(
        border=(-img_scale[0] // 2, -img_scale[1] // 2)), # 图像经过马赛克处理后会放大4倍，所以我们使用仿射变换来恢复图像的大小。
    dict(type='PackDetInputs'))
]

    _delete_ = True, # 删除不必要的设置
        ],
    ),
    )

    )
```

## 在配置文件中冻结骨干网络后在训练中解冻骨干网络

],
    ),
    )

    )
```

如果你在配置文件中已经冻结了骨干网络并希望在几个训练周期后解冻它，你可以通过 hook 来实现这个功能。以用 ResNet 为骨干网络的 Faster R-CNN 为例，你可以冻结一个骨干网络的一个层并在配置文件中添加如下 `custom_hooks`:

```python
_base_ = [
]
)
```

同时在 `mmdet/core/hook/unfreeze_backbone_epoch_based_hook.py` 当中书写 `UnfreezeBackboneEpochBasedHook` 类

```python

    Args:

    Args:
    """

            if is_module_wrapper(model):
                else:

for param in backbone.stem.parameters():
                else:

                m.train()
```

## 获得新的骨干网络的通道数

如果你想获得一个新骨干网络的通道数，你可以单独构建这个骨干网络并输入一个伪造的图片来获取每一个阶段的输出。

以 `ResNet` 为例：

```python

```

以上脚本的输出为:

```python
```

```

以上脚本的输出为:

```python
```

用户可以通过将脚本中的 `ResNet(depth=18)` 替换为自己的骨干网络配置来得到新的骨干网络的通道数。

# MMDetection 中训练 Detectron2 的模型

用户可以使用 `Detectron2Wrapper` 从而在 MMDetection 中使用 Detectron2 的模型。
我们提供了 [Faster R-CNN](../../../configs/misc/d2_faster-rcnn_r50-caffe_fpn_ms-90k_coco.py),
[Mask R-CNN](../../../configs/misc/d2_mask-rcnn_r50-caffe_fpn_ms-90k_coco.py) 和 [RetinaNet](../../../configs/misc/d2_retinanet_r50-caffe_fpn_ms-90k_coco.py) 的示例来在 MMDetection 中训练/测试 Detectron2 的模型。

使用过程中需要注意配置文件中算法组件要和 Detectron2 中的相同。模型初始化时，我们首先初始化 [Detectron2](https://github.com/facebookresearch/detectron2/blob/main/detectron2/config/defaults.py) 的默认设置，然后配置文件中的设置将覆盖默认设置，模型将基于更新过的设置来建立。
输入数据首先转换成 Detectron2 的类型并输入进 Detectron2 的模型中。在推理阶段，Detectron2 的模型结果将会转换回 MMDetection 的类型。

## 使用 Detectron2 的预训练权重

`Detectron2Wrapper` 中的权重初始化将不使用 MMDetection 的逻辑。用户可以设置 `model.d2_detector.weights=xxx` 来加载预训练的权重。
例如，我们可以使用 `model.d2_detector.weights='detectron2://ImageNetPretrained/MSRA/R-50.pkl'` 来加载 ResNet-50 的预训练权重，或者使用
`model.d2_detector.weights='detectron2://COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_1x/137260431/model_final_a54504.pkl'` 来加载 Detectron2 中提出的预训练的Mask R-CNN权重。

**注意：** 不能直接使用 `load_from` 来加载 Detectron2 的预训练模型，但可以通过 `tools/model_converters/detectron2_to_mmdet.py` 先对该预训练模型进行转换。

在测试时，用户应该首先使用 `tools/model_converters/detectron2_to_mmdet.py` 将 Detectron2 的预训练权重转换为 MMDetection 可读取的结构。

```shell
python tools/model_converters/detectron2_to_mmdet.py ${Detectron2 ckpt path} ${MMDetectron ckpt path}。
```

This is an implementation of [DiffusionDet](https://github.com/ShoufaChen/DiffusionDet) based on [MMDetection](https://github.com/open-mmlab/mmdetection/tree/main), [MMCV](https://github.com/open-mmlab/mmcv), and [MMEngine](https://github.com/open-mmlab/mmengine).

<center>
<img src="https://user-images.githubusercontent.com/48282753/211472911-c84d658a-952b-4608-8b91-9ac932cbf2e2.png">
</center>

## Usage

### Comparison of results

1. Download the [DiffusionDet released model](https://github.com/ShoufaChen/DiffusionDet#models).

2. Convert model from DiffusionDet version to MMDetection version. We give a [sample script](model_converters/diffusiondet_resnet_to_mmdet.py)

   to convert `DiffusionDet-resnet50` model. Users can download the corresponding models from [here](https://github.com/ShoufaChen/DiffusionDet/releases/download/v0.1/diffdet_coco_res50.pth).

   ```shell
   python projects/DiffusionDet/model_converters/diffusiondet_resnet_to_mmdet.py ${DiffusionDet ckpt path} ${MMDetectron ckpt path}
   ```

3. Testing the model in MMDetection.

   ```shell
   python tools/test.py projects/DiffusionDet/configs/diffusiondet_r50_fpn_500-proposals_1-step_crop-ms-480-800-450k_coco.py ${CHECKPOINT_PATH}
   ```

**Note:** During inference time, DiffusionDet will randomly generate noisy boxes,
which may affect the AP results. If users want to get the same result every inference time, setting seed is a good way.

We give a table to compare the inference results on `ResNet50-500-proposals` between DiffusionDet and MMDetection.

|                                                         Config                                                          | Step |    AP     |
| :---------------------------------------------------------------------------------------------------------------------: | :--: | :-------: |
| [DiffusionDet](https://github.com/ShoufaChen/DiffusionDet/blob/main/configs/diffdet.coco.res50.yaml) (released results) |  1   |   45.5    |
|      [DiffusionDet](https://github.com/ShoufaChen/DiffusionDet/blob/main/configs/diffdet.coco.res50.yaml) (seed=0)      |  1   |   45.66   |
|         [MMDetection](configs/diffusiondet_r50_fpn_500-proposals_1-step_crop-ms-480-800-450k_coco.py) (seed=0)          |  1   |   45.7    |

|       [MMDetection](configs/diffusiondet_r50_fpn_500-proposals_1-step_crop-ms-480-800-450k_coco.py) (random seed)       |  1   | 45.6~45.8 |
| [DiffusionDet](https://github.com/ShoufaChen/DiffusionDet/blob/main/configs/diffdet.coco.res50.yaml) (released results) |  4   |   46.1    |
|      [DiffusionDet](https://github.com/ShoufaChen/DiffusionDet/blob/main/configs/diffdet.coco.res50.yaml) (seed=0)      |  4   |   46.38   |
|         [MMDetection](configs/diffusiondet_r50_fpn_500-proposals_1-step_crop-ms-480-800-450k_coco.py) (seed=0)          |  4   |   46.4    |
|       [MMDetection](configs/diffusiondet_r50_fpn_500-proposals_1-step_crop-ms-480-800-450k_coco.py) (random seed)       |  4   | 46.2~46.4 |

- `seed=0` means hard set seed before generating random boxes.
  ```python
  # hard set seed=0 before generating random boxes
  seed = 0

  ```python
  seed = 0
  random.seed(seed)
  torch.manual_seed(seed)
  # torch.cuda.manual_seed(seed)
  torch.cuda.manual_seed_all(seed)
  ...
  noise_bboxes_raw = torch.randn(
      (self.num_proposals, 4),
      device=device)
  ...
  ```
- `random seed` means do not hard set seed before generating random boxes.

```bash
python tools/train.py projects/DiffusionDet/configs/diffusiondet_r50_fpn_500-proposals_1-step_crop-ms-480-800-450k_coco.py
```

```bash
python -m torch.distributed.launch --nnodes=1 --node_rank=0 --nproc_per_node=${NUM_GPUS} --master_port=29506 --master_addr="127.0.0.1" tools/train.py projects/DiffusionDet/configs/diffusiondet_r50_fpn_500-proposals_1-step_crop-ms-480-800-450k_coco.py
```

```

```bash
# for 1 step inference
# test command
python tools/test.py projects/DiffusionDet/configs/diffusiondet_r50_fpn_500-proposals_1-step_crop-ms-480-800-450k_coco.py ${CHECKPOINT_PATH}

# for 4 steps inference

python tools/test.py projects/DiffusionDet/configs/diffusiondet_r50_fpn_500-proposals_1-step_crop-ms-480-800-450k_coco.py ${CHECKPOINT_PATH} --cfg-options model.bbox_head.sampling_timesteps=4
```

**Note:** There is no difference between 1 step or 4 steps (or other multi-step) during training. Users can set different steps during inference through `--cfg-options model.bbox_head.sampling_timesteps=${STEPS}`, but larger `sampling_timesteps` will affect the inference time.

## Results

Here we provide the baseline version of DiffusionDet with ResNet50 backbone.

## Results

To find more variants, please visit the [official model zoo](https://github.com/ShoufaChen/DiffusionDet#models).

| Backbone |  Style  | Lr schd | AP (Step=1) | AP (Step=4) |                                           Config                                           |                                                                                                                                                                                                                                      Download                                                                                                                                                                                                                                      |

|   R-50   | PyTorch |  450k   |    44.5     |    46.2     | [config](./configs/diffusiondet_r50_fpn_500-proposals_1-step_crop-ms-480-800-450k_coco.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/diffusiondet/diffusiondet_r50_fpn_500-proposals_1-step_crop-ms-480-800-450k_coco/diffusiondet_r50_fpn_500-proposals_1-step_crop-ms-480-800-450k_coco_20230215_090925-7d6ed504.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/diffusiondet/diffusiondet_r50_fpn_500-proposals_1-step_crop-ms-480-800-450k_coco/diffusiondet_r50_fpn_500-proposals_1-step_crop-ms-480-800-450k_coco_20230215_090925.log.json) |

## License

DiffusionDet is under the [CC-BY-NC 4.0 license](https://github.com/ShoufaChen/DiffusionDet/blob/main/LICENSE). Users should be careful about adopting these features in any commercial matters.

If you find DiffusionDet is useful in your research or applications, please consider giving a star 🌟 to the [official repository](https://github.com/ShoufaChen/DiffusionDet) and citing DiffusionDet by the following BibTeX entry.

```BibTeX
@article{chen2022diffusiondet,
      title={DiffusionDet: Diffusion Model for Object Detection},
      author={Chen, Shoufa and Sun, Peize and Song, Yibing and Luo, Ping},
      journal={arXiv preprint arXiv:2211.09788},
      year={2022}
}
```

# Training a ReID Model

You may want to train a ReID model for multiple object tracking or other applications. We support ReID model training in MMDetection, which is built upon [MMPretrain](https://github.com/open-mmlab/mmpretrain).

### 2. Dataset Preparation

This section will show how to train a ReID model on standard datasets i.e. MOT17.

We need to download datasets following docs. We use [ReIDDataset](mmdet/datasets/reid_dataset.py) to maintain standard datasets. In this case, you need to convert the official dataset to this style. We provide scripts and the usages as follow:

```python
python tools/dataset_converters/mot2reid.py -i ./data/MOT17/ -o ./data/MOT17/reid --val-split 0.2 --vis-threshold 0.3
```

- `--val-split`: Proportion of the validation dataset to the whole ReID dataset.

```

- `--vis-threshold`: Threshold of visibility for each person.

The directory of the converted datasets is as follows:

```
MOT17
├── train
├── test
├── reid
│   ├── imgs
│   │   ├── MOT17-02-FRCNN_000002
│   │   │   ├── 000000.jpg
│   │   │   ├── 000001.jpg
│   │   │   ├── ...
│   │   ├── MOT17-02-FRCNN_000003
│   │   │   ├── ...
│   ├── meta
│   │   ├── train_80.txt
│   │   ├── val_20.txt
```

Note: `80` in `train_80.txt` means the proportion of the training dataset to the whole ReID dataset is eighty percent. While the proportion of the validation dataset is twenty percent.

For training, we provide a annotation list `train_80.txt`. Each line of the list constraints a filename and its corresponding ground-truth labels. The format is as follows:

```

```
```

Note: Images in `MOT17/reid/imgs` are cropped from raw images in `MOT17/train` by the corresponding `gt.txt`. The value of ground-truth labels should fall in range `[0, num_classes - 1]`.

#### Training on a single GPU

```shell
python tools/train.py configs/reid/reid_r50_8xb32-6e_mot17train80_test-mot17val20.py
```

#### Training on multiple GPUs

```shell
bash tools/dist_train.sh configs/reid/reid_r50_8xb32-6e_mot17train80_test-mot17val20.py 8
```

### 4. Customize Dataset

This section will show how to train a ReID model on customize datasets.

### 4.1 Dataset Preparation

```

You need to convert your customize datasets to existing dataset format.

#### An example of customized dataset

Assume we are going to implement a `Filelist` dataset, which takes filelists for both training and testing. The directory of the dataset is as follows:

```
Filelist
├── imgs
│   ├── person1
│   │   ├── ...
│   ├── person2
│   │   ├── ...
├── meta
```

The format of annotation list is as follows:

```
person1/000000.jpg 0
person1/000001.jpg 0
person2/000000.jpg 1
person2/000001.jpg 1
```

You can directly use [ReIDDataset](mmdet/datasets/reid_dataset.py). In this case, you only need to modify the config as follows:

```python
# modify the path of annotation files and the image path prefix

```python
        data_prefix='data/Filelist/imgs',
        ann_file='data/Filelist/meta/train.txt'),
    val=dict(
        ann_file='data/Filelist/meta/val.txt'),
)
# modify the number of classes, assume your training set has 100 classes
model = dict(reid=dict(head=dict(num_classes=100)))
```

### 4.2 Training

The training stage is the same as `Standard Dataset`.

# 准备 THUMOS'14

## 简介

```BibTex
@misc{THUMOS14,
    author = {Jiang, Y.-G. and Liu, J. and Roshan Zamir, A. and Toderici, G. and Laptev,
    I. and Shah, M. and Sukthankar, R.},
    title = {{THUMOS} Challenge: Action Recognition with a Large
    Number of Classes},
    howpublished = "\url{http://crcv.ucf.edu/THUMOS14/}",
    Year = {2014}
}
```

用户可以参照数据集 [官网](https://www.crcv.ucf.edu/THUMOS14/download.html)，获取数据集相关的基本信息。
在准备数据集前，请确保命令行当前路径为 `$MMACTION2/tools/data/thumos14/`。

首先，用户可使用以下命令下载标注文件。

```shell
cd $MMACTION2/tools/data/thumos14/
```

## 步骤 2. 下载视频

之后，用户可使用以下指令下载视频

```shell
```

## 步骤 3. 抽取帧和光流

```shell

```shell
mkdir /mnt/SSD/thumos14_extracted/
ln -s /mnt/SSD/thumos14_extracted/ ../data/thumos14/rawframes/
```

```shell
```

```shell
```

```shell
bash extract_frames.sh tvl1
```

如果用户不使用 SSN 模型，则该部分是 **可选项**。

可使用运行以下脚本下载预先计算的候选标签。

```shell
bash fetch_tag_proposals.sh
```

## 步骤 5. 去规范化候选文件

可运行以下脚本，来根据本地原始帧的实际数量，去规范化预先计算的候选标签。

```shell
bash denormalize_proposal_file.sh
```

## 步骤 6. 检查目录结构

```

```shell
```

在完成 THUMOS'14 数据集准备流程后，用户可以得到 THUMOS'14 的 RGB 帧 + 光流文件，视频文件以及标注文件。

在整个 MMAction2 文件夹下，THUMOS'14 的文件结构如下：

```
mmaction2
├── tools
├── data
│   ├── thumos14
│   │   ├── proposals
│   │   |   ├── thumos14_tag_val_normalized_proposal_list.txt
│   │   |   ├── thumos14_tag_test_normalized_proposal_list.txt
│   │   ├── annotations_val
│   │   ├── annotations_test
│   │   │   ├── val
│   │   │   |   ├── video_validation_0000001.mp4
│   │   |   ├── test
│   │   │   |   ├── video_test_0000001.mp4
│   │   │   ├── val
│   │   │   |   ├── video_validation_0000001
|   │   │   |   │   ├── img_00001.jpg
|   │   │   |   │   ├── img_00002.jpg
|   │   │   |   │   ├── ...

│   │   │   ├── val
|   │   │   |   │   ├── flow_x_00001.jpg
|   │   │   |   │   ├── flow_x_00002.jpg
|   │   │   |   │   ├── flow_y_00001.jpg
|   │   │   |   │   ├── flow_y_00002.jpg
│   │   │   |   ├── video_test_0000001
```

关于对 THUMOS'14 进行训练和验证，可以参照 [训练教程](/docs/zh_cn/user_guides/train_test.md)。

<details>
<summary align="right">TDAN (CVPR'2020)</summary>

```bibtex
}
```

<br/>

在 RGB 通道上进行评估，在评估之前裁剪每个边界中的8像素。

|                            算法                            |   Vid4 (BIx4)   | SPMCS-30 (BIx4) |   Vid4 (BDx4)   | SPMCS-30 (BDx4) |         GPU 信息         |                            下载                            |
| :--------------------------------------------------------: | :-------------: | :-------------: | :-------------: | :-------------: | :----------------------: | :--------------------------------------------------------: |

| [tdan_x4_1xb16-lr1e-4-400k_vimeo90k-bi](./tdan_x4_8xb16-lr1e-4-400k_vimeo90k-bi.py) |        -        |        -        |        -        |        -        | 8 (Tesla V100-SXM2-32GB) |                             -                              |
| [tdan_x4_1xb16-lr1e-4-400k_vimeo90k-bd](./tdan_x4_8xb16-lr1e-4-400k_vimeo90k-bd.py) |        -        |        -        |        -        |        -        | 8 (Tesla V100-SXM2-32GB) |                             -                              |
| [tdan_x4ft_1xb16-lr5e-5-400k_vimeo90k-bi](./tdan_x4ft_8xb16-lr5e-5-400k_vimeo90k-bi.py) | **26.49/0.792** | **30.42/0.856** |   25.93/0.772   |   29.69/0.842   | 8 (Tesla V100-SXM2-32GB) | [模型](https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bix4_20210528-739979d9.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bix4_20210528_135616.log.json) |

| [tdan_x4ft_1xb16-lr5e-5-800k_vimeo90k-bd](./tdan_x4ft_8xb16-lr5e-5-800k_vimeo90k-bd.py) |   25.80/0.784   |   29.56/0.851   | **26.87/0.815** | **30.77/0.868** | 8 (Tesla V100-SXM2-32GB) | [模型](https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bdx4_20210528-c53ab844.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/tdan/tdan_vimeo90k_bdx4_20210528_122401.log.json) |

## 快速开始

**训练**

<details>

TDAN 训练有两个阶段。

**阶段 1**: 以更大的学习率训练 (1e-4)

```shell
# CPU上训练

# 单个GPU上训练

# 多个GPU上训练
./tools/dist_train.sh configs/tdan/tdan_x4_1xb16-lr1e-4-400k_vimeo90k-bi.py 8
```

**阶段 2**: 以较小的学习率进行微调 (5e-5)

```shell
# CPU上训练

# 单个GPU上训练

```

```shell
# CPU上训练

# 单个GPU上训练

# 多个GPU上训练
```

**测试**

<details>

```shell
# CPU上测试

# 单个GPU上测试

# 多个GPU上测试

# 单个GPU上测试

# 多个GPU上测试
```

## <a href='https://mmsegmentation.readthedocs.io/en/latest/'>English</a>

## <a href='https://mmsegmentation.readthedocs.io/zh_CN/latest/'>简体中文</a>

# 准备 LIVE1 数据集

```bibtex
}
```

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── LIVE1
```

The runtime configurations include many helpful functionalities, like checkpoint saving, logger configuration,
etc. In this tutorial, we will introduce how to configure these functionalities.

## Save Checkpoint

The checkpoint saving functionality is a default hook during training. And you can configure it in the
`default_hooks.checkpoint` field.

```{note}
The hook mechanism is widely used in all OpenMMLab libraries. Through hooks, you can plug in many
functionalities without modifying the main execution logic of the runner.

A detailed introduction of hooks can be found in {external+mmengine:doc}`Hooks <tutorials/hook>`.
```

**The default settings**

```python
    ...
    ...
)
```

Here are some usual arguments, and all available arguments can be found in the [CheckpointHook](mmengine.hooks.CheckpointHook).

    ...
)
```

- **`interval`** (int): The saving period. If use -1, it will never save checkpoints.
- **`by_epoch`** (bool): Whether the **`interval`** is by epoch or by iteration. Defaults to `True`.
- **`out_dir`** (str): The root directory to save checkpoints. If not specified, the checkpoints will be saved in the work directory. If specified, the checkpoints will be saved in the sub-folder of the **`out_dir`**.
- **`max_keep_ckpts`** (int): The maximum checkpoints to keep. In some cases, we want only the latest few checkpoints and would like to delete old ones to save disk space. Defaults to -1, which means unlimited.
- **`save_best`** (str, List[str]): If specified, it will save the checkpoint with the best evaluation result.

  Usually, you can simply use `save_best="auto"` to automatically select the evaluation metric.

And if you want more advanced configuration, please refer to the [CheckpointHook docs](tutorials/hook.md#checkpointhook).

## Load Checkpoint / Resume Training

In config files, you can specify the loading and resuming functionality as below:

```python
# load from which checkpoint

# whether to resume training from the loaded checkpoint
```

The `load_from` field can be either a local path or an HTTP path. And you can resume training from the checkpoint by
specify `resume=True`.

```{tip}
You can also enable auto resuming from the latest checkpoint by specifying `load_from=None` and `resume=True`.
Runner will find the latest checkpoint from the work directory automatically.
```

```

If you are training models by our `tools/train.py` script, you can also use `--resume` argument to resume
training without modifying the config file manually.

```bash
# Automatically resume from the latest checkpoint.

# Resume from the specified checkpoint.
```

## Randomness Configuration

In the `randomness` field, we provide some options to make the experiment as reproducible as possible.

By default, we won't specify seed in the config file, and in every experiment, the program will generate a random seed.

**Default settings:**

```python
```

```python
```

To make the experiment more reproducible, you can specify a seed and set `deterministic=True`. The influence
of the `deterministic` option can be found [here](https://pytorch.org/docs/stable/notes/randomness.html#cuda-convolution-benchmarking).

## Log Configuration

The log configuration relates to multiple fields.

In the `log_level` field, you can specify the global logging level. See {external+python:ref}`Logging Levels<levels>` for a list of levels.

```python
```

In the `default_hooks.logger` field, you can specify the logging interval during training and testing. And all
available arguments can be found in the [LoggerHook docs](tutorials/hook.md#loggerhook).

```python
    ...
    # print log every 100 iterations.
    ...
)
```

```python
    ...
    ...
)
```

In the `log_processor` field, you can specify the log smooth method. Usually, we use a window with length of 10
to smooth the log and output the mean value of all information. If you want to specify the smooth method of
some information finely, see the {external+mmengine:doc}`LogProcessor docs <advanced_tutorials/logging>`.

```python
# The default setting, which will smooth the values in training log by a 10-length window.
```

In the `visualizer` field, you can specify multiple backends to save the log information, such as TensorBoard
and WandB. More details can be found in the [Visualizer section](#visualizer).

## Custom Hooks

Many above functionalities are implemented by hooks, and you can also plug-in other custom hooks by modifying

`custom_hooks` field. Here are some hooks in MMEngine and MMPretrain that you can use directly, such as:

- ......

For example, EMA (Exponential Moving Average) is widely used in the model training, and you can enable it as
below:

```python
]
```

## Visualize Validation

The validation visualization functionality is a default hook during validation. And you can configure it in the
`default_hooks.visualization` field.

By default, we disabled it, and you can enable it by specifying `enable=True`. And more arguments can be found in

the [VisualizationHook docs](mmpretrain.engine.hooks.VisualizationHook).

```python
    ...
    ...
)
```

This hook will select some images in the validation dataset, and tag the prediction results on these images
during every validation process. You can use it to watch the varying of model performance on actual images
during training.

In addition, if the images in your validation dataset are small (\<100), you can rescale them before
visualization by specifying `rescale_factor=2.` or higher.

The visualizer is used to record all kinds of information during training and test, including logs, images and
scalars. By default, the recorded information will be saved at the `vis_data` folder under the work directory.

```python
    ]
)
```

Usually, the most useful function is to save the log and scalars like `loss` to different backends.
For example, to save them to TensorBoard, simply set them as below:

```python
    ]
)
```

Or save them to WandB as below:

```python
    ]
)
```

In the `env_cfg` field, you can configure some low-level parameters, like cuDNN, multi-process, and distributed
communication.

]
)
```

**Please make sure you understand the meaning of these parameters before modifying them.**

```python
    # whether to enable cudnn benchmark

    # set multi-process parameters

    # set distributed parameters
)
```

}
```

<details>

```bibtex
}
```

| [pose_resnet_50](/configs/hand_2d_keypoint/topdown_heatmap/onehand10k/td-hm_res50_8xb32-210e_onehand10k-256x256.py) |  256x256   |  0.989  | 0.555 | 25.16 | [ckpt](https://download.openmmlab.com/mmpose/hand/resnet/res50_onehand10k_256x256-739c8639_20210330.pth) | [log](https://download.openmmlab.com/mmpose/hand/resnet/res50_onehand10k_256x256_20210330.log.json) |

To manage various configurations in a deep-learning experiment, we use a kind of config file to record all of
these configurations. This config system has a modular and inheritance design, and more details can be found in
{external+mmengine:doc}`the tutorial in MMEngine <advanced_tutorials/config>`.

Usually, we use python files as config file. All configuration files are placed under the [`configs`](https://github.com/open-mmlab/mmpretrain/tree/main/configs) folder, and the directory structure is as follows:

```text

│   │   └── default_runtime.py             # primitive runtime setting
    │   ├── ...
    └── ...
```

This article mainly explains the structure of configuration files, and how to modify it based on the existing configuration files. We will take [ResNet50 config file](https://github.com/open-mmlab/mmpretrain/blob/main/configs/resnet/resnet50_8xb32_in1k.py) as an example and explain it line by line.

## Config Structure

There are four kinds of basic component files in the `configs/_base_` folders, namely：

- [models](https://github.com/open-mmlab/mmpretrain/tree/main/configs/_base_/models)
- [datasets](https://github.com/open-mmlab/mmpretrain/tree/main/configs/_base_/datasets)
- [schedules](https://github.com/open-mmlab/mmpretrain/tree/main/configs/_base_/schedules)
- [runtime](https://github.com/open-mmlab/mmpretrain/blob/main/configs/_base_/default_runtime.py)

We call all the config files in the `_base_` folder as _primitive_ config files. You can easily build your training config file by inheriting some primitive config files.

For easy understanding, we use [ResNet50 config file](https://github.com/open-mmlab/mmpretrain/blob/main/configs/resnet/resnet50_8xb32_in1k.py) as an example and comment on each line.

```python
_base_ = [                                    # This config file will inherit all config files in `_base_`.

```python
    '../_base_/default_runtime.py'            # runtime settings
]
```

We will explain the four primitive config files separately below.

### Model settings

This primitive config file includes a dict variable `model`, which mainly includes information such as network structure and loss function:

- `type`: The type of model to build, we support several tasks.
  - For image classification tasks, it's usually `ImageClassifier` You can find more details in the [API documentation](mmpretrain.models.classifiers).

- For image classification tasks, it's usually `ImageClassifier` You can find more details in the [API documentation](mmpretrain.models.classifiers).
  - For self-supervised leanrning, there are several `SelfSupervisors`, such as `MoCoV2`, `BEiT`, `MAE`, etc. You can find more details in the [API documentation](mmpretrain.models.selfsup).
  - For image retrieval tasks, it's usually `ImageToImageRetriever` You can find more details in the [API documentation](mmpretrain.models.retrievers).

Usually, we use the **`type` field** to specify the class of the component and use other fields to pass
the initialization arguments of the class. The {external+mmengine:doc}`registry tutorial <advanced_tutorials/registry>` describes it in detail.

Here, we use the config fields of [`ImageClassifier`](mmpretrain.models.classifiers.ImageClassifier) as an example to
describe the initialization arguments as below:

- `backbone`: The settings of the backbone. The backbone is the main network to extract features of the inputs, like `ResNet`, `Swin Transformer`, `Vision Transformer` etc. All available backbones can be found in the [API documentation](mmpretrain.models.backbones).
  - For self-supervised leanrning, some of the backbones are re-implemented, you can find more details in the [API documentation](mmpretrain.models.selfsup).
- `neck`: The settings of the neck. The neck is the intermediate module to connect the backbone and the head, like `GlobalAveragePooling`. All available necks can be found in the [API documentation](mmpretrain.models.necks).

- `head`: The settings of the task head. The head is the task-related component to do a specified task, like image classification or self-supervised training. All available heads can be found in the [API documentation](mmpretrain.models.heads).
  - `loss`: The loss function to optimize, like `CrossEntropyLoss`, `LabelSmoothLoss`, `PixelReconstructionLoss` and etc. All available losses can be found in the [API documentation](mmpretrain.models.losses).
- `data_preprocessor`: The component before the model forwarding to preprocess the inputs. See the [documentation](mmpretrain.models.utils.data_preprocessor) for more details.
- `train_cfg`: The extra settings of `ImageClassifier` during training. In `ImageClassifier`, we mainly use it to specify batch augmentation settings, like `Mixup` and `CutMix`. See the [documentation](mmpretrain.models.utils.batch_augments) for more details.

Following is the model primitive config of the ResNet50 config file in [`configs/_base_/models/resnet50.py`](https://github.com/open-mmlab/mmpretrain/blob/main/configs/_base_/models/resnet50.py):

```python
    type='ImageClassifier',     # The type of the main model (here is for image classification task).
        type='ResNet',          # The type of the backbone module.
        # All fields except `type` come from the __init__ method of class `ResNet`
        # and you can find them from https://mmpretrain.readthedocs.io/en/latest/api/generated/mmpretrain.models.backbones.ResNet.html
        depth=50,
    neck=dict(type='GlobalAveragePooling'),    # The type of the neck module.
        type='LinearClsHead',     # The type of the classification head module.

style='pytorch'),
        # All fields except `type` come from the __init__ method of class `LinearClsHead`
        # and you can find them from https://mmpretrain.readthedocs.io/en/latest/api/generated/mmpretrain.models.heads.LinearClsHead.html
        loss=dict(type='CrossEntropyLoss', loss_weight=1.0),
    ))
```

### Data settings

This primitive config file includes information to construct the dataloader and evaluator:

- `data_preprocessor`: Model input preprocessing configuration, same as `model.data_preprocessor` but with lower priority.
- `train_evaluator | val_evaluator | test_evaluator`: To build the evaluator or metrics, refer to the [tutorial](mmpretrain.evaluation).

- `train_dataloader | val_dataloader | test_dataloader`: The settings of dataloaders
  - `batch_size`: The batch size of each GPU.
  - `num_workers`: The number of workers to fetch data of each GPU.
  - `sampler`: The settings of the sampler.
  - `persistent_workers`: Whether to persistent workers after finishing one epoch.
  - `dataset`: The settings of the dataset.
    - `type`: The type of the dataset, we support `CustomDataset`, `ImageNet` and many other datasets, refer to [documentation](mmpretrain.datasets).
    - `pipeline`: The data transform pipeline. You can find how to design a pipeline in [this tutorial](https://mmpretrain.readthedocs.io/en/latest/tutorials/data_pipeline.html).

- `pipeline`: The data transform pipeline. You can find how to design a pipeline in [this tutorial](https://mmpretrain.readthedocs.io/en/latest/tutorials/data_pipeline.html).

Following is the data primitive config of the ResNet50 config in [`configs/_base_/datasets/imagenet_bs32.py`](https://github.com/open-mmlab/mmpretrain/blob/main/configs/_base_/datasets/imagenet_bs32.py)：

```python
# preprocessing configuration
    # Input image data channels in 'RGB' order
    mean=[123.675, 116.28, 103.53],    # Input image normalized channel mean in RGB order
    std=[58.395, 57.12, 57.375],       # Input image normalized channel std in RGB order
    to_rgb=True,                       # Whether to flip the channel from BGR to RGB or RGB to BGR
)

    dict(type='LoadImageFromFile'),     # read image
    dict(type='RandomResizedCrop', scale=224),     # Random scaling and cropping

)

    dict(type='RandomFlip', prob=0.5, direction='horizontal'),   # random horizontal flip
    dict(type='PackInputs'),         # prepare images and labels
]

    dict(type='ResizeEdge', scale=256, edge='short'),  # Scale the short side to 256
    dict(type='CenterCrop', crop_size=224),     # center crop
    dict(type='PackInputs'),                 # prepare images and labels
]

# Construct training set dataloader
    batch_size=32,                     # batchsize per GPU
    num_workers=5,                     # Number of workers to fetch data per GPU
    dataset=dict(                      # training dataset

dataset=dict(                      # training dataset
    sampler=dict(type='DefaultSampler', shuffle=True),   # default sampler
    persistent_workers=True,                             # Whether to keep the process, can shorten the preparation time of each epoch
)

# Construct the validation set dataloader
)
# The settings of the evaluation metrics for validation. We use the top1 and top5 accuracy here.

persistent_workers=True,
)

test_dataloader = val_dataloader  # The settings of the dataloader for the test dataset, which is the same as val_dataloader
test_evaluator = val_evaluator    # The settings of the evaluation metrics for test, which is the same as val_evaluator
```

```{note}
The data preprocessor can be defined either in the subfield of `model`, or a using the `data_preprocessor` definition here, if both of them exist, use the `model.data_preprocessor` configuration.
```

### Schedule settings

This primitive config file mainly contains training strategy settings and the settings of training, val and
test loops:

- `optim_wrapper`: The settings of the optimizer wrapper. We use the optimizer wrapper to customize the
  optimization process.

  - `optimizer`: Supports all `pytorch` optimizers, refers to the relevant {external+mmengine:doc}`MMEngine documentation <tutorials/optim_wrapper>`.
  - `paramwise_cfg`: To set different optimization arguments according to the parameters' type or name, refer to the relevant [learning policy documentation](../advanced_guides/schedule.md).
  - `accumulative_counts`: Optimize parameters after several backward steps instead of one backward step. You
    can use it to simulate large batch size by small batch size.
- `param_scheduler`: Optimizer parameters policy. You can use it to specify learning rate and momentum curves during training. See the {external+mmengine:doc}`documentation <tutorials/param_scheduler>` in MMEngine for more details.

- `train_cfg | val_cfg | test_cfg`: The settings of the training, validation and test loops, refer to the relevant {external+mmengine:doc}`MMEngine documentation <design/runner>`.

Following is the schedule primitive config of the ResNet50 config in [`configs/_base_/datasets/imagenet_bs32.py`](https://github.com/open-mmlab/mmpretrain/blob/main/configs/_base_/datasets/imagenet_bs32.py)：

```python
    # Use SGD optimizer to optimize parameters.

# The tuning strategy of the learning rate.
# The 'MultiStepLR' means to use multiple steps policy to schedule the learning rate (LR).

# Training configuration, iterate 100 epochs, and perform validation after every training epoch.
# 'by_epoch=True' means to use `EpochBaseTrainLoop`, 'by_epoch=False' means to use IterBaseTrainLoop.

# Use the default val loop settings.
# Use the default test loop settings.

# This schedule is for the total batch size 256.
# If you use a different total batch size, like 512 and enable auto learning rate scaling.
# We will scale up the learning rate to 2 times.
```

### Runtime settings

This part mainly includes saving the checkpoint strategy, log configuration, training parameters, breakpoint weight path, working directory, etc.

Here is the runtime primitive config file ['configs/_base_/default_runtime.py'](https://github.com/open-mmlab/mmpretrain/blob/main/configs/_base_/default_runtime.py) file used by almost all configs:

```python

```python
# defaults to use registries in mmpretrain

# configure default hooks
    # record the time of every iteration.

    # enable the parameter scheduler.

    # save checkpoint per epoch.

    # set sampler seed in a distributed environment.

    # validation results visualization, set True to enable it.
)

# configure environment

)

cudnn_benchmark=False,

)

# set visualizer
vis_backends = [dict(type='LocalVisBackend')]  # use local HDD backend

# set log level

```

## Inherit and Modify Config File

For easy understanding, we recommend contributors inherit from existing config files. But do not abuse the
inheritance. Usually, for all config files, we recommend the maximum inheritance level is 3.

For example, if your config file is based on ResNet with some other modification, you can first inherit the
basic ResNet structure, dataset and other training settings by specifying `_base_ ='./resnet50_8xb32_in1k.py'`

(The path relative to your config file), and then modify the necessary parameters in the config file. A more
specific example, now we want to use almost all configs in `configs/resnet/resnet50_8xb32_in1k.py`, but using
`CutMix` train batch augment and changing the number of training epochs from 100 to 300, modify when to decay
the learning rate, and modify the dataset path, you can create a new config file
`configs/resnet/resnet50_8xb32-300e_in1k.py` with content as below:

```python
# create this file under 'configs/resnet/' folder

# using CutMix batch augment
    )
)

# trains more epochs
train_cfg = dict(max_epochs=300, val_interval=10)  # Train for 300 epochs, evaluate every 10 epochs

    )
)

param_scheduler = dict(step=[150, 200, 250])   # The learning rate adjustment has also changed

# Use your own dataset directory
)
    batch_size=64,                  # No back-propagation during validation, larger batch size can be used
)
    batch_size=64,                  # No back-propagation during test, larger batch size can be used
)
```

Some intermediate variables are used in the configuration file. The intermediate variables make the configuration file clearer and easier to modify.

)
```

For example, `train_pipeline` / `test_pipeline` is the intermediate variable of the data pipeline. We first need to define `train_pipeline` / `test_pipeline`, and then pass them to `train_dataloader` / `test_dataloader`. If you want to modify the size of the input image during training and testing, you need to modify the intermediate variables of `train_pipeline` / `test_pipeline`.

```python
bgr_mean = [103.53, 116.28, 123.675]  # mean in BGR order
    dict(

dict(type='RandomFlip', prob=0.5, direction='horizontal'),
    dict(
]

]

```

```

Sometimes, you need to set `_delete_=True` to ignore some domain content in the basic configuration file. You can refer to the {external+mmengine:doc}`documentation in MMEngine <advanced_tutorials/config>` for more instructions.

The following is an example. If you want to use cosine schedule in the above ResNet50 case, just using inheritance and directly modifying it will report `get unexpected keyword 'step'` error, because the `'step'` field of the basic config in `param_scheduler` domain information is reserved, and you need to add `_delete_ =True` to ignore the content of `param_scheduler` related fields in the basic configuration file:

```python

# the learning rate scheduler
```

```python

```

### Use some fields in the base configs

Sometimes, you may refer to some fields in the `_base_` config, to avoid duplication of definitions. You can refer to {external+mmengine:doc}`MMEngine <advanced_tutorials/config>` for some more instructions.

The following is an example of using auto augment in the training data preprocessing pipeline, refer to [`configs/resnest/resnest50_32xb64_in1k.py`](https://github.com/open-mmlab/mmpretrain/blob/main/configs/resnest/resnest50_32xb64_in1k.py). When defining `train_pipeline`, just add the definition file name of auto augment to `_base_`, and then use `_base_.auto_increasing_policies` to reference the variables in the primitive config:

```python
_base_ = [

```python
_base_ = [
]

    dict(
        policies=_base_.policies, # This uses the `policies` parameter in the primitive config.
    dict(
]

```

## Modify config in command

eigvec=EIGVEC,
]

```

When you use the script "tools/train.py" or "tools/test.py" to submit tasks or use some other tools, they can directly modify the content of the configuration file used by specifying the `--cfg-options` argument.

  e.g. `[dict(type='LoadImageFromFile'), dict(type='TopDownRandomFlip', flip_prob=0.5), ...]`. If you want to change `'flip_prob=0.5'` to `'flip_prob=0.0'` in the pipeline,
  you may specify `--cfg-options data.train.pipeline.1.flip_prob=0.0`.

  If the value to be updated is a list or a tuple. For example, the config file normally sets `val_evaluator = dict(type='Accuracy', topk=(1, 5))`. If you want to change the field `topk`, you may specify `--cfg-options val_evaluator.topk="(1,3)"`. Note that the quotation mark " is necessary to support list/tuple data types and that **NO** white space is allowed inside the quotation marks in the specified value.

    <sup>
      </a>
    </sup>
    <sup>
      </a>
    </sup>
  </div>

[🆕Update News](https://mmdetection.readthedocs.io/en/latest/notes/changelog.html) |
[🚀Ongoing Projects](https://github.com/open-mmlab/mmdetection/projects) |
[🤔Reporting Issues](https://github.com/open-mmlab/mmdetection/issues/new/choose)

</div>

</div>

</div>

</div>

MMDetection is an open source object detection toolbox based on PyTorch. It is

</div>

a part of the [OpenMMLab](https://openmmlab.com/) project.

  We decompose the detection framework into different components and one can easily construct a customized object detection framework by combining different modules.

- **Support of multiple tasks out of box**

  The toolbox directly supports multiple detection tasks such as **object detection**, **instance segmentation**, **panoptic segmentation**, and **semi-supervised object detection**.

  All basic bbox and mask operations run on GPUs. The training speed is faster than or comparable to other codebases, including [Detectron2](https://github.com/facebookresearch/detectron2), [maskrcnn-benchmark](https://github.com/facebookresearch/maskrcnn-benchmark) and [SimpleDet](https://github.com/TuSimple/simpledet).

- **State of the art**

  The toolbox stems from the codebase developed by the *MMDet* team, who won [COCO Detection Challenge](http://cocodataset.org/#detection-leaderboard) in 2018, and we keep pushing it forward.
  The newly released [RTMDet](configs/rtmdet) also obtains new state-of-the-art results on real-time instance segmentation and rotated object detection tasks and the best parameter-accuracy trade-off on object detection.

Apart from MMDetection, we also released [MMEngine](https://github.com/open-mmlab/mmengine) for model training and [MMCV](https://github.com/open-mmlab/mmcv) for computer vision research, which are heavily depended on by this toolbox.

We are excited to announce our latest work on real-time object recognition tasks, **RTMDet**, a family of fully convolutional single-stage detectors. RTMDet not only achieves the best parameter-accuracy trade-off on object detection from tiny to extra-large model sizes but also obtains new state-of-the-art performance on instance segmentation and rotated object detection tasks. Details can be found in the [technical report](https://arxiv.org/abs/2212.07784). Pre-trained models are [here](configs/rtmdet).

</div>

**v3.1.0** was released in 30/6/2023:

- Support [ViTDet](projects/ViTDet)

- Supports inference and evaluation of multimodal algorithms [GLIP](configs/glip) and [XDecoder](projects/XDecoder), and also supports datasets such as COCO semantic segmentation, COCO Caption, ADE20k general segmentation, and RefCOCO. GLIP fine-tuning will be supported in the future.

Please refer to [Installation](https://mmdetection.readthedocs.io/en/latest/get_started.html) for installation instructions.

Please see [Overview](https://mmdetection.readthedocs.io/en/latest/get_started.html) for the general introduction of MMDetection.

For detailed user guides and advanced guides, please refer to our [documentation](https://mmdetection.readthedocs.io/en/latest/):

  <details>

  <details>

  - [Train & Test](https://mmdetection.readthedocs.io/en/latest/user_guides/index.html#train-test)
    - [Learn about Configs](https://mmdetection.readthedocs.io/en/latest/user_guides/config.html)
    - [Inference with existing models](https://mmdetection.readthedocs.io/en/latest/user_guides/inference.html)
    - [Dataset Prepare](https://mmdetection.readthedocs.io/en/latest/user_guides/dataset_prepare.html)
    - [Test existing models on standard datasets](https://mmdetection.readthedocs.io/en/latest/user_guides/test.html)
    - [Train predefined models on standard datasets](https://mmdetection.readthedocs.io/en/latest/user_guides/train.html)
    - [Train with customized datasets](https://mmdetection.readthedocs.io/en/latest/user_guides/train.html#train-with-customized-datasets)

- [Train with customized datasets](https://mmdetection.readthedocs.io/en/latest/user_guides/train.html#train-with-customized-datasets)
    - [Train with customized models and standard datasets](https://mmdetection.readthedocs.io/en/latest/user_guides/new_model.html)
    - [Finetuning Models](https://mmdetection.readthedocs.io/en/latest/user_guides/finetune.html)
    - [Test Results Submission](https://mmdetection.readthedocs.io/en/latest/user_guides/test_results_submission.html)
    - [Weight initialization](https://mmdetection.readthedocs.io/en/latest/user_guides/init_cfg.html)
    - [Use a single stage detector as RPN](https://mmdetection.readthedocs.io/en/latest/user_guides/single_stage_as_rpn.html)
    - [Semi-supervised Object Detection](https://mmdetection.readthedocs.io/en/latest/user_guides/semi_det.html)
  - [Useful Tools](https://mmdetection.readthedocs.io/en/latest/user_guides/index.html#useful-tools)

  <details>

- [Useful Tools](https://mmdetection.readthedocs.io/en/latest/user_guides/index.html#useful-tools)

  <details>

  - [Basic Concepts](https://mmdetection.readthedocs.io/en/latest/advanced_guides/index.html#basic-concepts)
  - [Component Customization](https://mmdetection.readthedocs.io/en/latest/advanced_guides/index.html#component-customization)
  - [How to](https://mmdetection.readthedocs.io/en/latest/advanced_guides/index.html#how-to)

We also provide object detection colab tutorial [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](demo/MMDet_Tutorial.ipynb) and instance segmentation colab tutorial [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](demo/MMDet_InstanceSeg_Tutorial.ipynb).

To migrate from MMDetection 2.x, please refer to [migration](https://mmdetection.readthedocs.io/en/latest/migration.html).

</div>
  <tbody>
      <td>
      </td>
      <td>
      </td>
      <td>
      </td>
      <td>
      </td>
    </tr>
      <td>
        <ul>

<li><a href="configs/rpn">RPN (NeurIPS'2015)</a></li>

<li><a href="configs/tridentnet">TridentNet (ICCV'2019)</a></li>

<li><a href="configs/autoassign">AutoAssign (ArXiv'2020)</a></li>

<li><a href="configs/ddod">DDOD (ACM MM'2021)</a></li>
      </ul>
      </td>
      <td>
        <ul>

<li><a href="configs/cascade_rcnn">Cascade Mask R-CNN (CVPR'2018)</a></li>

<li><a href="configs/mask2former">Mask2Former (ArXiv'2021)</a></li>
        </ul>
      </td>
      <td>
        <ul>
        </ul>
      </td>
      <td>
        </ul>
        <ul>
        <ul>

<ul>
        <ul>
        </ul>
        </ul>
        </ul>
        <ul>
        <ul>
        </ul>
        </ul>
        <ul>
        <ul>
        </ul>
        </ul>
      </ul>
      </td>
    </tr>
</td>
    </tr>
  </tbody>
</table>

</div>
  <tbody>
      <td>
      </td>

</tr>
  </tbody>
</table>

</div>
  <tbody>
      <td>
      </td>
      <td>
      </td>
      <td>
      </td>
      <td>
      </td>
    </tr>
      <td>
      <ul>

<li><a href="configs/res2net">Res2Net (TPAMI'2020)</a></li>
      </ul>
      </td>
      <td>
      <ul>

<li><a href="configs/nas_fpn">NAS-FPN (CVPR'2019)</a></li>
      </ul>
      </td>
      <td>
        <ul>
        </ul>
      </td>
      <td>
        <ul>
          <li><a href="configs/faster_rcnn/faster-rcnn_r50_fpn_ohem_1x_coco.py">OHEM (CVPR'2016)</a></li>

<li><a href="configs/dcn">DCN (ICCV'2017)</a></li>
        </ul>
      </td>
    </tr>
</td>
    </tr>
  </tbody>
</table>

Some other methods are also supported in [projects using MMDetection](./docs/en/notes/projects.md).

## FAQ

## FAQ

We appreciate all contributions to improve MMDetection. Ongoing projects can be found in out [GitHub Projects](https://github.com/open-mmlab/mmdetection/projects). Welcome community users to participate in these projects. Please refer to [CONTRIBUTING.md](.github/CONTRIBUTING.md) for the contributing guideline.

MMDetection is an open source project that is contributed by researchers and engineers from various colleges and companies. We appreciate all the contributors who implement their methods or add new features, as well as users who give valuable feedbacks.
We wish that the toolbox and benchmark could serve the growing research community by providing a flexible toolkit to reimplement existing methods and develop their own new detectors.

```

```
}
```

## License

<details>

```bibtex
}
```

<details>

```bibtex

author={Buslaev, Alexander and Iglovikov, Vladimir I and Khvedchenya, Eugene and Parinov, Alex and Druzhinin, Mikhail and Kalinin, Alexandr A},
}
```

<details>

```bibtex
}
```

| [coarsedropout](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_coarsedropout-8xb64-210e_coco-256x192.py) |  256x192   | 0.753 |      0.908      |      0.822      | 0.805 |      0.944      | [ckpt](https://download.openmmlab.com/mmpose/top_down/augmentation/hrnet_w32_coco_256x192_coarsedropout-0f16a0ce_20210320.pth) | [log](https://download.openmmlab.com/mmpose/top_down/augmentation/hrnet_w32_coco_256x192_coarsedropout_20210320.log.json) |
| [gridmask](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_gridmask-8xb64-210e_coco-256x192.py) |  256x192   | 0.752 |      0.906      |      0.825      | 0.804 |      0.943      | [ckpt](https://download.openmmlab.com/mmpose/top_down/augmentation/hrnet_w32_coco_256x192_gridmask-868180df_20210320.pth) | [log](https://download.openmmlab.com/mmpose/top_down/augmentation/hrnet_w32_coco_256x192_gridmask_20210320.log.json) |

| [photometric](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_photometric-8xb64-210e_coco-256x192.py) |  256x192   | 0.754 |      0.908      |      0.825      | 0.805 |      0.943      | [ckpt](https://download.openmmlab.com/mmpose/top_down/augmentation/hrnet_w32_coco_256x192_photometric-308cf591_20210320.pth) | [log](https://download.openmmlab.com/mmpose/top_down/augmentation/hrnet_w32_coco_256x192_photometric_20210320.log.json) |

# MMagic projects

The goal of this ecosystem is to support, accelerate, and aid in your exploration with MMagic for AIGC such as image, video, 3D content generation, editing and processing.

Here is an [example project](../../../projects/example_project) about how to add your projects to MMagic.

You can copy and create your own project from the [example project](../../../projects/example_project).

# DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations

<details>

```bibtex
}
```

<details>

```bibtex

```bibtex
 }
```

---
title: ''
labels: ''
---

2. I have read the [FAQ documentation](https://mmdetection.readthedocs.io/en/latest/faq.html) but cannot get the expected help.
3. The bug has not been fixed in the latest version.

```none
```

```none
```

# Migration

We introduce some modifications in MMPretrain 1.x, and some of them are BC-breacking. To migrate your projects from **MMClassification 0.x** or **MMSelfSup 0.x** smoothly, please read this tutorial.

- [Migration](#migration)
- [General change of config](#general-change-of-config)
  - [Schedule settings](#schedule-settings)
  - [Runtime settings](#runtime-settings)
  - [Other changes](#other-changes)
- [Migration from MMClassification 0.x](#migration-from-mmclassification-0x)
  - [Config files](#config-files)
    - [Model settings](#model-settings)
    - [Data settings](#data-settings)
  - [Packages](#packages)
    - [`mmpretrain.apis`](#mmpretrainapis)
    - [`mmpretrain.core`](#mmpretraincore)
    - [`mmpretrain.datasets`](#mmpretraindatasets)
    - [`mmpretrain.models`](#mmpretrainmodels)
    - [`mmpretrain.utils`](#mmpretrainutils)
- [Migration from MMSelfSup 0.x](#migration-from-mmselfsup-0x)
  - [Config](#config)

- [`mmpretrain.models`](#mmpretrainmodels)
    - [Dataset settings](#dataset-settings)
    - [Model settings](#model-settings-1)
  - [Package](#package)

MMPretrain 1.x has new package dependencies, and a new environment should be created for MMPretrain 1.x even if you already have a well-rounded MMClassification 0.x or MMSelfSup 0.x environment. Please refer to the [installation tutorial](./get_started.md) for the required package installation or install the packages manually.
```

1. [MMEngine](https://github.com/open-mmlab/mmengine): MMEngine is the core the OpenMMLab 2.0 architecture,
   and we have split many compentents unrelated to computer vision from MMCV to MMEngine.
2. [MMCV](https://github.com/open-mmlab/mmcv): The computer vision package of OpenMMLab. This is not a new

and we have split many compentents unrelated to computer vision from MMCV to MMEngine.
   dependency, but it should be upgraded to version `2.0.0rc1` or above.
3. [rich](https://github.com/Textualize/rich): A terminal formatting package, and we use it to enhance some
   outputs in the terminal.
4. [einops](https://github.com/arogozhnikov/einops): Operators for Einstein notations.

# General change of config

In this section, we introduce the general difference between old version(**MMClassification 0.x** or **MMSelfSup 0.x**) and **MMPretrain 1.x**.

## Schedule settings

| MMCls or MMSelfSup 0.x | MMPretrain 1.x  | Remark                                                                                                                          |

| ---------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| optimizer_config       | /               | It has been **removed**.                                                                                                        |
| /                      | optim_wrapper   | The `optim_wrapper` provides a common interface for updating parameters.                                                        |
| lr_config              | param_scheduler | The `param_scheduler` is a list to set learning rate or other parameters, which is more flexible.                               |

| runner                 | train_cfg       | The loop setting (`EpochBasedTrainLoop`, `IterBasedTrainLoop`) in `train_cfg` controls the work flow of the algorithm training. |

Changes in **`optimizer`** and **`optimizer_config`**:

- Now we use `optim_wrapper` field to specify all configurations related to optimization process. The
  `optimizer` has become a subfield of `optim_wrapper`.
- The `paramwise_cfg` field is also a subfield of `optim_wrapper`, instead of `optimizer`.
- The `optimizer_config` field has been removed, and all configurations has been moved to `optim_wrapper`.
- The `grad_clip` field has been renamed to `clip_grad`.

<tr>
<td>Original</td>
<td>

```python

<tr>
<td>

```python
    ))

```

</td>
<tr>
<td>New</td>
<td>

```python
    ),
    clip_grad=dict(max_norm=1.0),
)
```

</td>
</tr>
</table>

Changes in **`lr_config`**:

- The `lr_config` field has been removed and replaced by the new `param_scheduler`.
- The `warmup` related arguments have also been removed since we use a combination of schedulers to implement this
  functionality.

The new scheduler combination mechanism is highly flexible and enables the design of various learning rate/momentum curves.

functionality.

For more details, see the {external+mmengine:doc}`parameter schedulers tutorial <tutorials/param_scheduler>`.

<tr>
<td>

```python
    min_lr=0,
```

</td>
<tr>
<td>

```python
    # warmup
    dict(
        end=5,
        # Update the learning rate after every iters.
    # main learning rate scheduler
]
```

</td>
</tr>
</table>

Changes in **`runner`**:

]
```

</td>
</tr>
</table>

Most of the configurations that were originally in the `runner` field have been moved to `train_cfg`, `val_cfg`, and `test_cfg`.
These fields are used to configure the loop for training, validation, and testing.

<tr>
<td>

```python
```

</td>
<tr>
<td>

```python
# The `val_interval` is the original `evaluation.interval`.
val_cfg = dict()   # Use the default validation loop.
test_cfg = dict()  # Use the default test loop.
```

</td>
</tr>
</table>

In OpenMMLab 2.0, we introduced `Loop` to control the behaviors in training, validation and testing. As a result, the functionalities of `Runner` have also been changed.

```

</td>
</tr>
</table>

More details can be found in the {external+mmengine:doc}`MMEngine tutorials <design/runner>`.

## Runtime settings

Changes in **`checkpoint_config`** and **`log_config`**:

The `checkpoint_config` has been moved to `default_hooks.checkpoint`, and `log_config` has been moved to
`default_hooks.logger`. Additionally, many hook settings that were previously included in the script code have
been moved to the `default_hooks` field in the runtime configuration.

```python
    # record the time of every iterations.

# print log every 100 iterations.

    # save checkpoint per epoch, and automatically save the best checkpoint.

    # set sampler seed in distributed evrionment.

)
```

In OpenMMLab 2.0, we have split the original logger into logger and visualizer. The logger is used to record
information, while the visualizer is used to display the logger in different backends such as terminal,
TensorBoard, and Wandb.

<tr>
<td>

```python
    hooks=[

<tr>
<td>

```python
    hooks=[
    ])
```

</td>
<tr>
<td>

```python
    ...
)

)
```

</td>
</tr>
</table>

Changes in **`load_from`** and **`resume_from`**:

- The `resume_from` is removed. And we use `resume` and `load_from` to replace it.
  - If `resume=True` and `load_from` is not None, resume training from the checkpoint in `load_from`.
  - If `resume=True` and `load_from` is None, try to resume from the latest checkpoint in the work directory.
  - If `resume=False` and `load_from` is not None, only load the checkpoint, not resume training.

- If `resume=False` and `load_from` is not None, only load the checkpoint, not resume training.
  - If `resume=False` and `load_from` is None, do not load nor resume.

the `resume_from` field has been removed, and we use `resume` and `load_from` instead.

- If `resume=True` and `load_from` is not None, training is resumed from the checkpoint in `load_from`.
- If `resume=True` and `load_from` is None, the latest checkpoint in the work directory is used for resuming.
- If `resume=False` and `load_from` is not None, only the checkpoint is loaded without resuming training.
- If `resume=False` and `load_from` is None, neither checkpoint is loaded nor is training resumed.

Changes in **`dist_params`**: The `dist_params` field has become a subfield of `env_cfg` now.
Additionally, some new configurations have been added to `env_cfg`.

```python

    # set multi process parameters

```python

)
```

Changes in **`workflow`**: `workflow` related functionalities are removed.

New field **`visualizer`**: The visualizer is a new design in OpenMMLab 2.0 architecture. The runner uses an
instance of the visualizer to handle result and log visualization, as well as to save to different backends.
For more information, please refer to the {external+mmengine:doc}`MMEngine tutorial <advanced_tutorials/visualization>`.

```python
        # Uncomment the below line to save the log and visualization results to TensorBoard.

    ]
)
```

New field **`default_scope`**: The start point to search module for all registries. The `default_scope` in MMPretrain is `mmpretrain`. See {external+mmengine:doc}`the registry tutorial <advanced_tutorials/registry>` for more details.

## Other changes

We moved the definition of all registries in different packages to the `mmpretrain.registry` package.

# Migration from MMClassification 0.x

## Config files

In MMPretrain 1.x, we refactored the structure of configuration files, and the original files are not usable.

In this section, we will introduce all changes of the configuration files. And we assume you already have
ideas of the [config files](./user_guides/config.md).

No changes in `model.backbone`, `model.neck` and `model.head` fields.

Changes in **`model.train_cfg`**:

- `BatchMixup` is renamed to [`Mixup`](mmpretrain.models.utils.batch_augments.Mixup).
- `BatchCutMix` is renamed to [`CutMix`](mmpretrain.models.utils.batch_augments.CutMix).
- `BatchResizeMix` is renamed to [`ResizeMix`](mmpretrain.models.utils.batch_augments.ResizeMix).
- The `prob` argument is removed from all augments settings, and you can use the `probs` field in `train_cfg` to
  specify probabilities of every augemnts. If no `probs` field, randomly choose one by the same probability.

<tr>
<td>

```python
    ...
    train_cfg=dict(augments=[
        dict(type='BatchMixup', alpha=0.8, num_classes=1000, prob=0.5),

<tr>
<td>

```python
    ...
        dict(type='BatchCutMix', alpha=1.0, num_classes=1000, prob=0.5)
    ]
)
```

</td>
<tr>
<td>

```python
    ...
        dict(type='Mixup', alpha=0.8), dict(type='CutMix', alpha=1.0)]
)
```

</td>
</tr>
</table>

Changes in **`data`**:

- The original `data` field is splited to `train_dataloader`, `val_dataloader` and
  `test_dataloader`. This allows us to configure them in fine-grained. For example,
  you can specify different sampler and batch size during training and test.

<tr>
<td>

```python
    samples_per_gpu=32,

<tr>
<td>

```python
)
```

</td>
<tr>
<td>

```python
    sampler=dict(type='DefaultSampler', shuffle=True)  # necessary
)

    sampler=dict(type='DefaultSampler', shuffle=False)  # necessary
)

```

</td>
</tr>
</table>

Changes in **`pipeline`**:

- The original formatting transforms **`ToTensor`**, **`ImageToTensor`** and **`Collect`** are combined as [`PackInputs`](mmpretrain.datasets.transforms.PackInputs).
- We don't recommend to do **`Normalize`** in the dataset pipeline. Please remove it from pipelines and set it in the `data_preprocessor` field.

- The argument `flip_prob` in [**`RandomFlip`**](mmcv.transforms.RandomFlip) is renamed to `prob`.
- The argument `size` in [**`RandomCrop`**](mmpretrain.datasets.transforms.RandomCrop) is renamed to `crop_size`.
- The argument `size` in [**`RandomResizedCrop`**](mmpretrain.datasets.transforms.RandomResizedCrop) is renamed to `scale`.
- The argument `size` in [**`Resize`**](mmcv.transforms.Resize) is renamed to `scale`. And `Resize` won't support size like `(256, -1)`, please use [`ResizeEdge`](mmpretrain.datasets.transforms.ResizeEdge) to replace it.
- The argument `policies` in [**`AutoAugment`**](mmpretrain.datasets.transforms.AutoAugment) and [**`RandAugment`**](mmpretrain.datasets.transforms.RandAugment) supports using string to specify preset policies. `AutoAugment` supports "imagenet" and `RandAugment` supports "timm_increasing".

- **`RandomResizedCrop`** and **`CenterCrop`** won't supports `efficientnet_style`, and please use [`EfficientNetRandomCrop`](mmpretrain.datasets.transforms.EfficientNetRandomCrop) and [`EfficientNetCenterCrop`](mmpretrain.datasets.transforms.EfficientNetCenterCrop) to replace them.

```{note}
We move some work of data transforms to the data preprocessor, like normalization, see [the documentation](mmpretrain.models.utils.data_preprocessor) for
more details.
```

<tr>
<td>

```python

    dict(type='RandomResizedCrop', size=224),
    dict(type='RandomFlip', flip_prob=0.5, direction='horizontal'),
    dict(type='ImageToTensor', keys=['img']),
    dict(type='ToTensor', keys=['gt_label']),
    dict(type='Collect', keys=['img', 'gt_label'])
]
```

</td>

dict(type='Normalize', **img_norm_cfg),
]
```

</td>
<tr>
<td>

```python

]
```

</td>
</tr>
</table>

Changes in **`evaluation`**:

- The **`evaluation`** field is splited to `val_evaluator` and `test_evaluator`. And it won't supports `interval` and `save_best` arguments.
  The `interval` is moved to `train_cfg.val_interval`, see [the schedule settings](./user_guides/config.md#schedule-settings) and the `save_best`
  is moved to `default_hooks.checkpoint.save_best`, see [the runtime settings](./user_guides/config.md#runtime-settings).

is moved to `default_hooks.checkpoint.save_best`, see [the runtime settings](./user_guides/config.md#runtime-settings).
- The 'accuracy' metric is renamed to [`Accuracy`](mmpretrain.evaluation.Accuracy).
- The 'precision', 'recall', 'f1-score' and 'support' are combined as [`SingleLabelMetric`](mmpretrain.evaluation.SingleLabelMetric), and use `items` argument to specify to calculate which metric.
- The 'mAP' is renamed to [`AveragePrecision`](mmpretrain.evaluation.AveragePrecision).
- The 'CP', 'CR', 'CF1', 'OP', 'OR', 'OF1' are combined as [`MultiLabelMetric`](mmpretrain.evaluation.MultiLabelMetric), and use `items` and `average` arguments to specify to calculate which metric.

<tr>
<td>

```python
    metric='accuracy',
    metric_options=dict(topk=(1, 5))
)
```

</td>
<tr>
<td>

```python
```

</td>
</tr>
<tr>

metric_options=dict(topk=(1, 5))
)
```

</td>
<tr>
<td>

```python
```

</td>
</tr>
<tr>
<td>

```python
    metric=['mAP', 'CP', 'OP', 'CR', 'OR', 'CF1', 'OF1'],
    metric_options=dict(thr=0.5),
)
```

</td>
<tr>
<td>

```python
    dict(type='AveragePrecision'),
    dict(type='MultiLabelMetric',
        items=['precision', 'recall', 'f1-score'],
        average='both',
        thr=0.5),
]
```

</td>
</tr>
</table>

### `mmpretrain.apis`

The documentation can be found [here](mmpretrain.apis).

|       Function       | Changes                                                                                                                                          |

| :------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------- |
|     `init_model`     | No changes                                                                                                                                       |
|  `inference_model`   | No changes. But we recommend to use [`mmpretrain.ImageClassificationInferencer`](mmpretrain.apis.ImageClassificationInferencer) instead.         |
|    `train_model`     | Removed, use `runner.train` to train.                                                                                                            |

|   `multi_gpu_test`   | Removed, use `runner.test` to test.                                                                                                              |
|  `single_gpu_test`   | Removed, use `runner.test` to test.                                                                                                              |
| `show_result_pyplot` | Removed, use [`mmpretrain.ImageClassificationInferencer`](mmpretrain.apis.ImageClassificationInferencer) to inference model and show the result. |
|  `set_random_seed`   | Removed, use `mmengine.runner.set_random_seed`.                                                                                                  |

|  `init_random_seed`  | Removed, use `mmengine.dist.sync_random_seed`.                                                                                                   |

### `mmpretrain.core`

The `mmpretrain.core` package is renamed to [`mmpretrain.engine`](mmpretrain.engine).

|   Sub package   | Changes                                                                                                                           |
| :-------------: | :-------------------------------------------------------------------------------------------------------------------------------- |
|  `evaluation`   | Removed, use the metrics in [`mmpretrain.evaluation`](mmpretrain.evaluation).                                                     |

|     `hook`      | Moved to [`mmpretrain.engine.hooks`](mmpretrain.engine.hooks)                                                                     |
|  `optimizers`   | Moved to [`mmpretrain.engine.optimizers`](mmpretrain.engine.optimizers)                                                           |
|     `utils`     | Removed, the distributed environment related functions can be found in the [`mmengine.dist`](api/dist) package.                   |
| `visualization` | Removed, the related functionalities are implemented in [`mmengine.visualization.Visualizer`](mmengine.visualization.Visualizer). |

The `MMClsWandbHook` in `hooks` package is waiting for implementation.

The `CosineAnnealingCooldownLrUpdaterHook` in `hooks` package is removed, and we support this functionality by

the combination of parameter schedulers, see [the tutorial](./advanced_guides/schedule.md).

### `mmpretrain.datasets`

The documentation can be found [here](mmpretrain.datasets).

|                                       Dataset class                                       | Changes                                                                                                         |
| :---------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------- |
|                   [`CustomDataset`](mmpretrain.datasets.CustomDataset)                    | Add `data_root` argument as the common prefix of `data_prefix` and `ann_file` and support to load unlabeled data. |

|                        [`ImageNet`](mmpretrain.datasets.ImageNet)                         | Same as `CustomDataset`.                                                                                        |
|                     [`ImageNet21k`](mmpretrain.datasets.ImageNet21k)                      | Same as `CustomDataset`.                                                                                        |
|   [`CIFAR10`](mmpretrain.datasets.CIFAR10) & [`CIFAR100`](mmpretrain.datasets.CIFAR100)   | The `test_mode` argument is a required argument now.                                                            |
| [`MNIST`](mmpretrain.datasets.MNIST) & [`FashionMNIST`](mmpretrain.datasets.FashionMNIST) | The `test_mode` argument is a required argument now.                                                            |

|                             [`VOC`](mmpretrain.datasets.VOC)                              | Requires `data_root`, `image_set_path` and `test_mode` now.                                                     |
|                             [`CUB`](mmpretrain.datasets.CUB)                              | Requires `data_root` and `test_mode` now.                                                                       |

The `mmpretrain.datasets.pipelines` is renamed to `mmpretrain.datasets.transforms`.

|         Transform class         | Changes                                                                                                                                                                   |
| :-----------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |

|       `LoadImageFromFile`       | Removed, use [`mmcv.transforms.LoadImageFromFile`](mmcv.transforms.LoadImageFromFile).                                                                                    |
|          `RandomFlip`           | Removed, use [`mmcv.transforms.RandomFlip`](mmcv.transforms.RandomFlip). The argument `flip_prob` is renamed to `prob`.                                                   |
|          `RandomCrop`           | The argument `size` is renamed to `crop_size`.                                                                                                                            |
|       `RandomResizedCrop`       | The argument `size` is renamed to `scale`. The argument `scale` is renamed to `crop_ratio_range`. Won't support `efficientnet_style`, use [`EfficientNetRandomCrop`](mmpretrain.datasets.transforms.EfficientNetRandomCrop). |

|          `CenterCrop`           | Removed, use [`mmcv.transforms.CenterCrop`](mmcv.transforms.CenterCrop). Won't support `efficientnet_style`, use [`EfficientNetCenterCrop`](mmpretrain.datasets.transforms.EfficientNetCenterCrop). |
|            `Resize`             | Removed, use [`mmcv.transforms.Resize`](mmcv.transforms.Resize). The argument `size` is renamed to `scale`. Won't support size like `(256, -1)`, use [`ResizeEdge`](mmpretrain.datasets.transforms.ResizeEdge). |
| `AutoAugment` & `RandomAugment` | The argument `policies` supports using string to specify preset policies.                                                                                                 |
|            `Compose`            | Removed, use [`mmcv.transforms.Compose`](mmcv.transforms.Compose).                                                                                                        |

### `mmpretrain.models`

The documentation can be found [here](mmpretrain.models). The interface of all **backbones**, **necks** and **losses** didn't change.

Changes in [`ImageClassifier`](mmpretrain.models.classifiers.ImageClassifier):

| Method of classifiers | Changes                                                                                                                                                                 |
| :-------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|    `extract_feat`     | No changes                                                                                                                                                              |

|       `forward`       | Now only accepts three arguments: `inputs`, `data_samples` and `mode`. See [the documentation](mmpretrain.models.classifiers.ImageClassifier.forward) for more details. |
|    `forward_train`    | Replaced by `loss`.                                                                                                                                                     |
|     `simple_test`     | Replaced by `predict`.                                                                                                                                                  |
|     `train_step`      | The `optimizer` argument is replaced by `optim_wrapper` and it accepts [`OptimWrapper`](mmengine.optim.OptimWrapper).                                                   |

|      `val_step`       | The original `val_step` is the same as `train_step`, now it calls `predict`.                                                                                            |
|      `test_step`      | New method, and it's the same as `val_step`.                                                                                                                            |

Changes in [heads](mmpretrain.models.heads):

| Method of heads | Changes                                                                                                                                                |
| :-------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------- |

|  `pre_logits`   | No changes                                                                                                                                             |
| `forward_train` | Replaced by `loss`.                                                                                                                                    |
|  `simple_test`  | Replaced by `predict`.                                                                                                                                 |
|     `loss`      | It accepts `data_samples` instead of `gt_labels` to calculate loss. The `data_samples` is a list of [ClsDataSample](mmpretrain.structures.DataSample). |

|    `forward`    | New method, and it returns the output of the classification head without any post-processs like softmax or sigmoid.                                    |

### `mmpretrain.utils`

|           Function           | Changes                                                                                                         |
| :--------------------------: | :-------------------------------------------------------------------------------------------------------------- |
|        `collect_env`         | No changes                                                                                                      |
|      `get_root_logger`       | Removed, use [`mmengine.logging.MMLogger.get_current_instance`](mmengine.logging.MMLogger.get_current_instance) |

|       `load_json_log`        | The output format changed.                                                                                      |
|   `setup_multi_processes`    | Removed, use [`mmengine.utils.dl_utils.set_multi_processing`](mmengine.utils.dl_utils.set_multi_processing).    |
| `wrap_non_distributed_model` | Removed, we auto wrap the model in the runner.                                                                  |
|   `wrap_distributed_model`   | Removed, we auto wrap the model in the runner.                                                                  |
|     `auto_select_device`     | Removed, we auto select the device in the runner.                                                               |

# Migration from MMSelfSup 0.x

## Config

## Config

This section illustrates the changes of our config files in the `_base_` folder, which includes three parts

- Datasets: `configs/_base_/datasets`
- Models: `configs/_base_/models`
- Schedules: `configs/_base_/schedules`

### Dataset settings

In **MMSelfSup 0.x**, we use key `data` to summarize all information, such as `samples_per_gpu`, `train`, `val`, etc.

In **MMPretrain 1.x**, we separate `train_dataloader`, `val_dataloader` to summarize information correspodingly and the key `data` has been **removed**.

<tr>
<td>

```python
    samples_per_gpu=32,  # total 32*8(gpu)=256
        data_source=dict(
            type=data_source,
            data_prefix='data/imagenet/train',

workers_per_gpu=4,
            ann_file='data/imagenet/meta/train.txt',
        ),
        num_views=[1, 1],
        pipelines=[train_pipeline1, train_pipeline2],
        prefetch=prefetch,
    ),
    val=...)
```

</td>

<tr>
<td>

```python
        data_prefix=dict(img_path='train/'),
val_dataloader = ...
```

</td>
</tr>
</table>

```

</td>
</tr>
</table>

Besides, we **remove** the key of `data_source` to keep the pipeline format consistent with that in other OpenMMLab projects. Please refer to [Config](user_guides/config.md) for more details.

Take MAE as an example of `pipeline`:

```python
    dict(
        type='RandomResizedCrop',
        scale=224,
        crop_ratio_range=(0.2, 1.0),
        backend='pillow',
        interpolation='bicubic'),
]
```

In the config of models, there are two main different parts from MMSeflSup 0.x.

]
```

1. There is a new key called `data_preprocessor`, which is responsible for preprocessing the data, like normalization, channel conversion, etc. For example:

```python
    bgr_to_rgb=True)
    type='MAE',
        mean=[127.5, 127.5, 127.5],
        std=[127.5, 127.5, 127.5],
    backbone=...,
    neck=...,
    head=...,
    init_cfg=...)
```

2. There is a new key `loss` in `head` in MMPretrain 1.x, to determine the loss function of the algorithm. For example:

```python
    neck=...,
        type='MAEPretrainHead',
        norm_pix=True,
        patch_size=16,

```python
    neck=...,
        loss=dict(type='MAEReconstructionLoss')),
```

## Package

The table below records the general modification of the folders and files.

| MMSelfSup 0.x            | MMPretrain 1.x      | Remark                                                                                                                                                        |
| ------------------------ | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| apis                     | apis                | The high level APIs are updated.                                                                                                                              |

| core                     | engine              | The `core` folder has been renamed to `engine`, which includes `hooks`, `opimizers`. ([API link](mmpretrain.engine))                                          |
| datasets                 | datasets            | The datasets is implemented according to different datasets, such as ImageNet, Places205. ([API link](mmpretrain.datasets))                                   |
| datasets/data_sources    | /                   | The `data_sources` has been **removed** and the directory of `datasets` now is consistent with other OpenMMLab projects.                                      |
| datasets/pipelines       | datasets/transforms | The `pipelines` folder has been renamed to `transforms`. ([API link](mmpretrain.datasets.transforms))                                                         |

| /                        | evaluation          | The `evaluation` is created for some evaluation functions or classes. ([API link](mmpretrain.evaluation))                                                     |
| models/algorithms        | selfsup             | The algorithms are moved to `selfsup` folder. ([API link](mmpretrain.models.selfsup))                                                                         |
| models/backbones         | selfsup             | The re-implemented backbones are moved to corresponding self-supervised learning algorithm `.py` files. ([API link](mmpretrain.models.selfsup))               |
| models/target_generators | selfsup             | The target generators are moved to corresponding self-supervised learning algorithm `.py` files. ([API link](mmpretrain.models.selfsup))                      |

| /                        | models/losses       | The `losses` folder is created to provide different loss implementations, which is from `heads`. ([API link](mmpretrain.models.losses))                       |
| /                        | structures          | The `structures` folder is for the implementation of data structures. In MMPretrain, we implement a new data structure, `DataSample`,  to pass and receive data throughout the training/val process. ([API link](mmpretrain.structures)) |
| /                        | visualization       | The `visualization` folder contains the visualizer, which is responsible for some visualization tasks like visualizing data augmentation. ([API link](mmpretrain.visualization)) |

# Oriented R-CNN

> [Oriented R-CNN for Object Detection](https://openaccess.thecvf.com/content/ICCV2021/papers/Xie_Oriented_R-CNN_for_Object_Detection_ICCV_2021_paper.pdf)

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/oriented_rcnn.png" width="800"/>
</div>

</div>

Current state-of-the-art two-stage detectors generate oriented proposals through time-consuming schemes. This diminishes the detectors’ speed, thereby becoming the computational bottleneck in advanced oriented object detection systems. This work proposes an effective and simple oriented object detection framework, termed Oriented R-CNN, which is a general two-stage oriented detector with promising accuracy and efficiency. To be specific, in the first stage, we propose an oriented Region Proposal Network (oriented RPN) that directly generates high-quality oriented proposals in a nearly cost-free manner. The second stage is oriented R-CNN head for refining oriented Regions of Interest (oriented RoIs) and recognizing them.

DOTA1.0

| ResNet50 (1024,1024,200) | 75.63 | le90  |   1x    |   7.37   |      21.2      |  -  |     2      |             [oriented-rcnn-le90_r50_fpn_amp-1x_dota](./oriented-rcnn-le90_r50_fpn_amp-1x_dota.py)              |         [model](https://download.openmmlab.com/mmrotate/v0.1.0/oriented_rcnn/oriented_rcnn_r50_fpn_fp16_1x_dota_le90/oriented_rcnn_r50_fpn_fp16_1x_dota_le90-57c88621.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/oriented_rcnn/oriented_rcnn_r50_fpn_fp16_1x_dota_le90/oriented_rcnn_r50_fpn_fp16_1x_dota_le90_20220303_195049.log.json)         |

| ResNet50 (1024,1024,200) | 75.69 | le90  |   1x    |   8.46   |      16.2      |  -  |     2      |                 [oriented-rcnn-le90_r50_fpn_1x_dota](./oriented-rcnn-le90_r50_fpn_1x_dota.py)                  |                   [model](https://download.openmmlab.com/mmrotate/v0.1.0/oriented_rcnn/oriented_rcnn_r50_fpn_1x_dota_le90/oriented_rcnn_r50_fpn_1x_dota_le90-6d2b2ce0.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/oriented_rcnn/oriented_rcnn_r50_fpn_1x_dota_le90/oriented_rcnn_r50_fpn_1x_dota_le90_20220127_100150.log.json)                   |

```
@InProceedings{Xie_2021_ICCV,
  author = {Xie, Xingxing and Cheng, Gong and Wang, Jiabao and Yao, Xiwen and Han, Junwei},
  title = {Oriented R-CNN for Object Detection},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  month = {October},
  pages = {3520-3529} }
```

# BDD100K Dataset

Support **`BDD100K Dataset`**

Author: CastleDream

This project implements **`BDD100K Dataset`**

Preparing `BDD100K Dataset` dataset following [BDD100K Dataset Preparing Guide](https://github.com/open-mmlab/mmsegmentation/tree/main/projects/mapillary_dataset/docs/en/user_guides/2_dataset_prepare.md#bdd100k)

```none
mmsegmentation/data
└── bdd100k
    ├── images
    │   └── 10k
    │       ├── test [2000 entries exceeds filelimit, not opening dir]
    │       ├── train [7000 entries exceeds filelimit, not opening dir]
    │       └── val [1000 entries exceeds filelimit, not opening dir]
    └── labels
        └── sem_seg
            ├── colormaps
            │   ├── train [7000 entries exceeds filelimit, not opening dir]
            │   └── val [1000 entries exceeds filelimit, not opening dir]
            ├── masks

│   └── val [1000 entries exceeds filelimit, not opening dir]
            ├── masks
            ├── polygons
            │   ├── sem_seg_train.json
            │   └── sem_seg_val.json
            └── rles
                ├── sem_seg_train.json
                └── sem_seg_val.json
```

```bash
!python tools/train.py projects/bdd100k_dataset/configs/pspnet_r50-d8_4xb2-80k_bdd100k-512x1024.py\
--work-dir your_work_dir
```

## Thanks

- [\[Datasets\] Add Mapillary Vistas Datasets to MMSeg Core Package. #2576](https://github.com/open-mmlab/mmsegmentation/pull/2576/files)
- [\[Feature\] Support CIHP dataset #1493](https://github.com/open-mmlab/mmsegmentation/pull/1493/files)

State-of-the-art image inpainting approaches can suffer from generating distorted structures and blurry textures in high-resolution images (e.g., 512x512). The challenges mainly drive from (1) image content reasoning from distant contexts, and (2) fine-grained texture synthesis for a large missing region. To overcome these two challenges, we propose an enhanced GAN-based model, named Aggregated COntextual-Transformation GAN (AOT-GAN), for high-resolution image inpainting. Specifically, to enhance context reasoning, we construct the generator of AOT-GAN by stacking multiple layers of a proposed AOT block. The AOT blocks aggregate contextual transformations from various receptive fields, allowing to capture both informative distant image contexts and rich patterns of interest for context reasoning. For improving texture synthesis, we enhance the discriminator of AOT-GAN by training it with a tailored mask-prediction task. Such a training objective forces the discriminator to distinguish the detailed appearances of real and synthesized patches, and in turn, facilitates the generator to synthesize clear textures. Extensive comparisons on Places2, the most challenging benchmark with 1.8 million high-resolution images of 365 complex scenes, show that our model outperforms the state-of-the-art by a significant margin in terms of FID with 38.60% relative improvement. A user study including more than 30 subjects further validates the superiority of AOT-GAN. We further evaluate the proposed AOT-GAN in practical applications, e.g., logo removal, face editing, and object removal. Results show that our model achieves promising completions in the real world. We release code and models in [this https URL](https://github.com/researchmm/AOT-GAN-for-Inpainting).

</div >

|                   Model                    |       Dataset       |     Mask Type      | Resolution | Train Iters |   Test Set    | l1 error | PSNR  | SSIM  |   Training Resources    |                    Download                    |
| :----------------------------------------: | :-----------------: | :----------------: | :--------: | :---------: | :-----------: | :------: | :---: | :---: | :---------------------: | :--------------------------------------------: |

| [AOT-GAN](./aot-gan_smpgan_4xb4_places-512x512.py) | Places365-Challenge | free-form (50-60%) |  512x512   |    500k     | Places365-val |   7.07   | 19.01 | 0.682 | 4 (GeForce GTX 1080 Ti) | [model](https://download.openmmlab.com/mmediting/inpainting/aot_gan/AOT-GAN_512x512_4x12_places_20220509-6641441b.pth) \| [log](https://download.openmmlab.com/mmediting/inpainting/aot_gan/AOT-GAN_512x512_4x12_places_20220509-6641441b.json) |

More results for different mask area:

| Metric          | Mask Area | Paper Results | Reimplemented Results |
| :-------------- | :-------- | :------------ | :-------------------- |
| L1 (10^-2)      | 1 – 10%   | 0.55          | 0.54                  |
| (lower better)  | 10 – 20%  | 1.19          | 1.47                  |
|                 | 20 – 30%  | 2.11          | 2.79                  |
|                 | 30 – 40%  | 3.20          | 4.38                  |

|                 | 40 – 50%  | 4.51          | 6.28                  |
|                 | 50 – 60%  | 7.07          | 10.16                 |
| PSNR            | 1 – 10%   | 34.79         | inf                   |
| (higher better) | 10 – 20%  | 29.49         | 31.22                 |
|                 | 20 – 30%  | 26.03         | 27.65                 |
|                 | 30 – 40%  | 23.58         | 25.06                 |
|                 | 40 – 50%  | 21.65         | 23.01                 |
|                 | 50 – 60%  | 19.01         | 20.05                 |
| SSIM            | 1 – 10%   | 0.976         | 0.982                 |
| (higher better) | 10 – 20%  | 0.940         | 0.951                 |
|                 | 20 – 30%  | 0.890         | 0.911                 |

|                 | 30 – 40%  | 0.835         | 0.866                 |
|                 | 40 – 50%  | 0.773         | 0.815                 |
|                 | 50 – 60%  | 0.682         | 0.739                 |

**Train**

<details>

```shell
# cpu train

```

```

**Test**

<details>

```shell
# cpu test

```

```

```bibtex
}
```

# 快速开始

大多数 ML 模型除了推理外，需要对输入数据进行一些预处理，并对输出进行一些后处理步骤，以获得结构化输出。 MMDeploy sdk 提供了常见的预处理和后处理步骤。 当您使用 MMDeploy 进行模型转换后，您可以直接使用mmdeploy sdk 进行推理。

## 模型转换

可参考 [convert model](../02-how-to-run/convert_model.md) 获得更多信息.

转模型时通过增加 `--dump-info` 参数得到如下的目录结构(tensorrt)。 如果转换为其他后端，结构会略有不同。其中两个图片为不同后端推理结果

```bash
├── pipeline.json
├── end2end.engine
├── output_pytorch.jpg
└── output_tensorrt.jpg
```

和SDK相关的文件有：

- deploy.json    // 模型信息.
- pipeline.json  // pipeline信息，包括前处理、模型以及后处理.
- end2end.engine // 模型文件

SDK 可以直接读取模型目录，也可以读取相关文件打包成 zip 压缩包。 要读取 zip 文件，sdk 在编译时要设置 `-DMMDEPLOY_ZIP_MODEL=ON`

## SDK 推理

一般来讲，模型推理包含以下三个部分。

- 创建 pipeline
- 读取数据
- 模型推理

以下使用 `classifier` 作为例子来展示三个步骤.

### 创建 pipeline

#### 从硬盘中加载模型

```cpp

std::string model_path = "/data/resnet"; // or "/data/resnet.zip" if build with `-DMMDEPLOY_ZIP_MODEL=ON`
mmdeploy_model_t model;

mmdeploy_classifier_t classifier{};

mmdeploy_classifier_create(model, "cpu", 0, &classifier);
```

#### 从内存中加载模型

```cpp
std::string model_path = "/data/resnet.zip"
std::ifstream ifs(model_path, std::ios::binary); // /path/to/zipmodel
ifs.seekg(0, std::ios::end);
auto size = ifs.tellg();
ifs.seekg(0, std::ios::beg);
std::string str(size, '\0'); // binary data, should decrypt if it's encrypted
ifs.read(str.data(), size);

mmdeploy_model_create(str.data(), size, &model);

```

### 读取数据

```cpp
cv::Mat img = cv::imread(image_path);
```

### 模型推理

```cpp
mmdeploy_classification_t* res{};
int* res_count{};
mmdeploy_classifier_apply(classifier, &mat, 1, &res, &res_count);
```

# Migration from MMAction2 0.x

MMAction2 1.x introduced major refactorings and modifications including some BC-breaking changes. We provide this tutorial to help you migrate your projects from MMAction2 0.x smoothly.

MMAction2 1.x depends on the following packages. You are recommended to prepare a new clean environment and install them according to [install tutorial](./get_started/installation.md)

1. [MMEngine](https://github.com/open-mmlab/mmengine): MMEngine is a foundational library for training deep learning model introduced in OpenMMLab 2.0 architecture.
2. [MMCV](https://github.com/open-mmlab/mmcv): MMCV is a foundational library for computer vision. MMAction2 1.x requires `mmcv>=2.0.0` which is more compact and efficient than `mmcv-full==2.0.0`.

## Configuration files

In MMAction2 1.x, we refactored the structure of configuration files. The configuration files with the old style will be incompatible.

In this section, we will introduce all changes of the configuration files. And we assume you are already familiar with the [config files](./user_guides/config.md).

No changes in `model.backbone` and `model.neck`. For `model.cls_head`, we move the `average_clips` inside it, which is originally set in `model.test_cfg`.

#### Changes in **`data`**

- The `videos_per_gpu` is renamed to `batch_size`.

<tr>
<td>

```python

<tr>
<td>

```python
)
```

</td>
<tr>
<td>

```python
)

)

```

</td>
</tr>
</table>

#### Changes in **`pipeline`**

- The original formatting transforms **`ToTensor`**, **`Collect`** are combined as `PackActionInputs`.
- We don't recommend to do **`Normalize`** in the dataset pipeline. Please remove it from pipelines and set it in the `model.data_preprocessor` field.

<tr>
<td>

<tr>
<td>

```python

    dict(
]
```

</td>
<tr>
<td>

```python

</td>
<tr>
<td>

```python

    dict(
]
```

</td>
</tr>
</table>

#### Changes in **`evaluation`**

- The **`evaluation`** field is splited to `val_evaluator` and `test_evaluator`. And it won't support `interval` and `save_best` arguments.

```

</td>
</tr>
</table>

- The `interval` is moved to `train_cfg.val_interval` and the `save_best` is moved to `default_hooks.checkpoint.save_best`.
- The 'mean_average_precision', 'mean_class_accuracy', 'mmit_mean_average_precision', 'top_k_accuracy' are combined as `AccMetric`, and you could use `metric_list` to specify which metric to calculate.
- The `AVAMetric` is used to evaluate AVA Dataset.
- The `ANetMetric` is used to evaluate ActivityNet Dataset.

<tr>
<td>

```python
```

</td>
<tr>
<td>

```python
```

</td>
</tr>
</table>

</td>
<tr>
<td>

```python
```

</td>
</tr>
</table>

#### Changes in **`optimizer`** and **`optimizer_config`**

- Now we use `optim_wrapper` field to configure the optimization process. And the
  `optimizer` becomes a sub field of `optim_wrapper`.
- `paramwise_cfg` is also a sub field of `optim_wrapper` parallel to `optimizer`.
- `optimizer_config` is removed now, and all configurations of it are moved to `optim_wrapper`.
- `grad_clip` is renamed to `clip_grad`.

<tr>
<td>

```python
    ))

```

</td>
<tr>
<td>

```python

norm_decay_mult=0.0,
    ))

```

</td>
<tr>
<td>

```python
    ),
)
```

</td>
</tr>
</table>

#### Changes in **`lr_config`**

- The `warmup` related arguments are removed, since we use schedulers combination to implement this

The new schedulers combination mechanism is very flexible, and you can use it to design many kinds of learning
rate / momentum curves.

<tr>
<td>

```python
    min_lr=0,
```

</td>

<td>

```python
    min_lr=0,
```

</td>
<tr>
<td>

```python
    # warmup
    dict(
        end=5,
]
```

</td>
</tr>
</table>

#### Changes in **`runner`**

Most configuration in the original `runner` field is moved to `train_cfg`, `val_cfg` and `test_cfg`, which
configure the loop in training, validation and test.

<tr>
<td>

```python
```

</td>
<tr>
<td>

```python

<tr>
<td>

```python
```

</td>
<tr>
<td>

```python
val_cfg = dict(type='ValLoop')   # Use the default validation loop.
test_cfg = dict(type='TestLoop')  # Use the default test loop.
```

</td>
</tr>
</table>

In fact, in OpenMMLab 2.0, we introduced `Loop` to control the behaviors in training, validation and test. And
the functionalities of `Runner` are also changed. You can find more details in the [MMEngine tutorials](https://mmengine.readthedocs.io/en/latest/tutorials/runner.html).

#### Changes in **`checkpoint_config`** and **`log_config`**

The `checkpoint_config` are moved to `default_hooks.checkpoint` and the `log_config` are moved to `default_hooks.logger`.

And we move many hooks settings from the script code to the `default_hooks` field in the runtime configuration.

```python
    # update runtime information, e.g. current iter and lr.

    # set sampler seed in distributed environment.

checkpoint=dict(type='CheckpointHook', interval=1, save_best='auto'),

    # synchronize model buffers at the end of each epoch.
)
```

In addition, we splited the original logger to logger and visualizer. The logger is used to record
information and the visualizer is used to show the logger in different backends, like terminal, TensorBoard
and Wandb.

<tr>
<td>

```python
    hooks=[
    ])
```

</td>
<tr>
<td>

```python
    ...
)

)
```

</td>
</tr>
</table>

)

)
```

</td>
</tr>
</table>

#### Changes in **`load_from`** and **`resume_from`**

#### Changes in **`dist_params`**

The `dist_params` field is a sub field of `env_cfg` now. And there are some new configurations in the `env_cfg`.

```python

```python

)
```

#### Changes in **`workflow`**

`Workflow` related functionalities are removed.

#### New field **`visualizer`**

The visualizer is a new design in OpenMMLab 2.0 architecture. We use a visualizer instance in the runner to handle results & log visualization and save to different backends.

```python
    ]
)
```

#### New field **`default_scope`**

# Uncomment the below line to save the log and visualization results to TensorBoard.
    ]
)
```

The start point to search module for all registries. The `default_scope` in MMAction2 is `mmaction`. See [the registry tutorial](https://mmengine.readthedocs.io/en/latest/tutorials/registry.html) for more details.

The documentation can be found [here](mmaction.apis).

|        Function        |                     Changes                     |
| :--------------------: | :---------------------------------------------: |
|   `init_recognizer`    |                   No changes                    |
| `inference_recognizer` |                   No changes                    |
|     `train_model`      |      Removed, use `runner.train` to train.      |
|    `multi_gpu_test`    |       Removed, use `runner.test` to test.       |

|   `single_gpu_test`    |       Removed, use `runner.test` to test.       |
|   `set_random_seed`    | Removed, use `mmengine.runner.set_random_seed`. |
|   `init_random_seed`   | Removed, use `mmengine.dist.sync_random_seed`.  |

The `mmaction.core` package is renamed to [`mmaction.engine`](mmaction.engine).

| Sub package  |                                               Changes                                               |
| :----------: | :-------------------------------------------------------------------------------------------------: |
| `evaluation` |                         Removed, use the metrics in `mmaction.evaluation`.                          |
|   `hooks`    |                                  Moved to `mmaction.engine.hooks`                                   |

| `optimizer`  |                                Moved to `mmaction.engine.optimizers`                                |
|   `utils`    | Removed, the distributed environment related functions can be found in the `mmengine.dist` package. |

The documentation can be found [here](mmaction.datasets)

#### Changes in [`BaseActionDataset`](mmaction.datasets.BaseActionDataset):

|         Method         |                    Changes                    |
| :--------------------: | :-------------------------------------------: |
| `prepare_train_frames` |          Replaced by `get_data_info`          |
| `preprare_test_frames` |          Replaced by `get_data_info`          |
|       `evaluate`       |  Removed, use `mmengine.evaluator.Evaluator`  |
|     `dump_results`     | Removed, use `mmengine.evaluator.DumpResults` |

|   `load_annotations`   |         Replaced by `load_data_list`          |

Now, you can write a new Dataset class inherited from `BaseActionDataset` and overwrite `load_data_list` only. To load more data information, you could overwrite `get_data_info` like `RawframeDataset` and `AVADataset`.
The `mmaction.datasets.pipelines` is renamed to `mmaction.datasets.transforms` and the `mmaction.datasets.pipelines.augmentations` is renamed to `mmaction.datasets.pipelines.processing`.

The documentation can be found [here](mmaction.models). The interface of all **backbones**, **necks** and **losses** didn't change.

#### Changes in [`BaseRecognizer`](mmaction.models.BaseRecognizer):

|     Method      |                                                                                Changes                                                                                 |
| :-------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| `extract_feat`  | Enhanced method, which now supports output features of three stages (`backbone`, `neck`, `head`) and can handle different modes, such as `train_mode` and `test_mode`. |
|    `forward`    |            Now only accepts three arguments: `inputs`, `data_samples` and `mode`. See [the documentation](mmaction.models.BaseRecognizer) for more details.            |

| `forward_train` |                                                                          Replaced by `loss`.                                                                           |
| `forward_test`  |                                                                         Replaced by `predict`.                                                                         |
|  `train_step`   |                         The `optimizer` argument is replaced by `optim_wrapper` and it accepts [`OptimWrapper`](mmengine.optim.OptimWrapper).                          |
|   `val_step`    |                                              The original `val_step` is the same as `train_step`, now it calls `predict`.                                              |

|   `test_step`   |                                                              New method, and it's the same as `val_step`.                                                              |

#### Changes in [BaseHead](mmaction.models.BaseHead):

|  Method   |                                                                                        Changes                                                                                         |
| :-------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

| `forward` |                                                                                       No changes                                                                                       |
|  `loss`   | It accepts `feats` and `data_samples` instead of `cls_score` and `labels` to calculate loss. The `data_samples` is a list of [ActionDataSample](mmaction.structures.ActionDataSample). |
| `predict` |                                                  New method. It accepts `feats` and `data_samples` to predict classification scores.                                                   |

|        Function         |                            Changes                            |

| :---------------------: | :-----------------------------------------------------------: |
|      `collect_env`      |                          No changes                           |
|    `get_root_logger`    |     Removed, use `mmengine.MMLogger.get_current_instance`     |
| `setup_multi_processes` | Removed, use `mmengine.utils.dl_utils.setup_multi_processes`. |

### Other changes

- We moved the definition of all registries in different packages to the `mmaction.registry` package.

# Win10 下构建方式

- [Win10 下构建方式](#win10-下构建方式)

## 源码安装

下述安装方式，均是在 **Windows 10** 下进行，使用 **PowerShell Preview** 版本。

1. 下载并安装 [Visual Studio 2019](https://visualstudio.microsoft.com) 。安装时请勾选 "使用C++的桌面开发, "Windows 10 SDK <br>
2. 把 cmake 路径加入到环境变量 PATH 中, "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\CMake\\CMake\\bin" <br>
3. 如果系统中配置了 NVIDIA 显卡，根据[官网教程](https://developer.nvidia.com\/cuda-downloads)，下载并安装 cuda toolkit。<br>

### 安装依赖包

### 安装依赖包

<thead>
  <tr>
    <th>安装方法 </th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>请参考 <a href="https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html">这里</a> 安装 conda。安装完毕后，打开系统开始菜单，<b>以管理员的身份打开 anaconda powershell prompt</b>。 因为，<br>
<b>1. 下文中的安装命令均是在 anaconda powershell 中测试验证的。</b><br>
<b>2. 使用管理员权限，可以把第三方库安装到系统目录。能够简化 MMDeploy 编译命令。</b><br>
<b>说明：如果你对 cmake 工作原理很熟悉，也可以使用普通用户权限打开 anaconda powershell prompt</b>。
</td>
  </tr>
  <tr>
    <td> 安装 PyTorch，要求版本是 torch>=1.8.0。可查看<a href="https://pytorch.org/">官网</a>获取更多详细的安装教程。请确保 PyTorch 要求的 CUDA 版本和您主机的 CUDA 版本是一致<br>
    </td>
  </tr>

    </td>
  </tr>
  <tr>
    <td>参考如下命令安装 mmcv。更多安装方式，可查看 <a href="https://github.com/open-mmlab/mmcv/tree/2.x#installation">mmcv 官网</a><br>
$env:torch_version="torch1.8"
    </td>
  </tr>
</tbody>
</table>

<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
    <td>OpenCV </td>
    <td>
    1. 从<a href="https://github.com/opencv/opencv/releases">这里</a>下载 OpenCV 3+。
    2. 您可以下载并安装 OpenCV 预编译包到指定的目录下。也可以选择源码编译安装的方式
    3. 在安装目录中，找到 <code>OpenCVConfig.cmake</code>，并把它的路径添加到环境变量 <code>PATH</code> 中。像这样：
    </td>

3. 在安装目录中，找到 <code>OpenCVConfig.cmake</code>，并把它的路径添加到环境变量 <code>PATH</code> 中。像这样：
    </td>
  </tr>
  <tr>
cd ppl.cv
cmake .. -G "Visual Studio 16 2019" -T v142 -A x64 -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=install -DPPLCV_USE_CUDA=ON -DPPLCV_USE_MSVC_STATIC_RUNTIME=OFF
cd ../..
   </td>
  </tr>
</tbody>
</table>

**目前，在 Windows 平台下，MMDeploy 支持 ONNXRuntime 和 TensorRT 两种推理引擎**。其他推理引擎尚未进行验证，或者验证未通过。后续将陆续予以支持

<thead>
  <tr>

<thead>
  <tr>
    <th>推理引擎 </th>
    <th>依赖包</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>
    2. 从<a href="https://github.com/microsoft/onnxruntime/releases/tag/v1.8.1">这里</a>下载 onnxruntime 的预编译二进制包，解压并配置环境变量
    </td>
  </tr>
  <tr>

    </td>
  </tr>
  <tr>
    <td>
    1. 登录 <a href="https://www.nvidia.com/">NVIDIA 官网</a>，从<a href="https://developer.nvidia.com/nvidia-tensorrt-download">这里</a>选取并下载 TensorRT tar 包。要保证它和您机器的 CPU 架构以及 CUDA 版本是匹配的。您可以参考这份 <a href="https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html#installing-tar">指南</a> 安装 TensorRT。<br>
    2. 这里也有一份 TensorRT 8.2 GA Update 2 在 Windows x86_64 和 CUDA 11.x 下的安装示例，供您参考。首先，点击<a href="https://developer.nvidia.com/compute/machine-learning/tensorrt/secure/8.2.3.0/zip/TensorRT-8.2.3.0.Windows10.x86_64.cuda-11.4.cudnn8.2.zip">此处</a>下载 CUDA 11.x TensorRT 8.2.3.0。然后，根据如下命令，安装并配置 TensorRT 以及相关依赖。

   </td>
  </tr>
  <tr>
    <td>cudnn </td>
    <td>
    1. 从 <a href="https://developer.nvidia.com/rdp/cudnn-archive">cuDNN Archive</a> 中选择和您环境中 CPU 架构、CUDA 版本以及 TensorRT 版本配套的 cuDNN。以前文 TensorRT 安装说明为例，它需要 cudnn8.2。因此，可以下载 <a href="https://developer.nvidia.com/compute/machine-learning/cudnn/secure/8.2.1.32/11.3_06072021/cudnn-11.3-windows-x64-v8.2.1.32.zip">CUDA 11.x cuDNN 8.2</a><br>
   </td>
  </tr>
  <tr>

   </td>
  </tr>
  <tr>
    <td> TODO </td>
  </tr>
  <tr>
  </tr>
  <tr>
  </tr>
</tbody>
</table>

```

如果您选择了 ONNXRuntime，TensorRT 和 ncnn 任一种推理后端，您需要编译对应的自定义算子库。

cd build
```

cd build

```

cd build
```

  TODO

```

**注意**

下文展示2个构建SDK的样例，分别用 ONNXRuntime 和 TensorRT 作为推理引擎。您可以参考它们，并结合前文 SDK 的编译选项说明，激活其他的推理引擎。

  cd build

  cd build

  ```

  mkdir build
  cd build

-DMMDEPLOY_TARGET_BACKENDS="trt" `

  ```

### 注意事项

1. Release / Debug 库不能混用。MMDeploy 要是编译 Release 版本，所有第三方依赖都要是 Release 版本。反之亦然。

# OVERVIEW

This chapter introduces you to the framework of MMDetection, and provides links to detailed tutorials about MMDetection.

## What is MMDetection

![image](https://user-images.githubusercontent.com/12907710/137271636-56ba1cd2-b110-4812-8221-b4c120320aa9.png)

MMDetection is an object detection toolbox that contains a rich set of object detection, instance segmentation, and panoptic segmentation methods as well as related components and modules, and below is its whole framework:

MMDetection consists of 7 main parts, apis, structures, datasets, models, engine, evaluation and visualization.

- **structures** provides data structures like bbox, mask, and DetDataSample.
- **datasets** supports various dataset for object detection, instance segmentation, and panoptic segmentation.
  - **samplers** defines different data loader sampling strategy.

- **transforms** contains a lot of useful data augmentation transforms.
- **models** is the most vital part for detectors and contains different components of a detector.
  - **detectors** defines all of the detection model classes.
  - **data_preprocessors** is for preprocessing the input data of the model.
  - **backbones** contains various backbone networks.
  - **necks** contains various neck components.
  - **dense_heads** contains various detection heads that perform dense predictions.
  - **roi_heads** contains various detection heads that predict from RoIs.
  - **seg_heads** contains various segmentation heads.
  - **task_modules** provides modules for detection tasks. E.g. assigners, samplers, box coders, and prior generators.
  - **layers** provides some basic neural network layers.
- **engine** is a part for runtime components.

- **layers** provides some basic neural network layers.
  - **runner** provides extensions for [MMEngine's runner](https://mmengine.readthedocs.io/en/latest/tutorials/runner.html).
  - **schedulers** provides schedulers for adjusting optimization hyperparameters.
- **visualization** is for visualizing detection results.

Here is a detailed step-by-step guide to learn more about MMDetection:

1. For installation instructions, please see [get_started](get_started.md).

2. Refer to the below tutorials for the basic usage of MMDetection.

   - [Train and Test](https://mmdetection.readthedocs.io/en/latest/user_guides/index.html#train-test)

   - [Useful Tools](https://mmdetection.readthedocs.io/en/latest/user_guides/index.html#useful-tools)

3. Refer to the below tutorials to dive deeper:

   - [Basic Concepts](https://mmdetection.readthedocs.io/en/latest/advanced_guides/index.html#basic-concepts)
   - [Component Customization](https://mmdetection.readthedocs.io/en/latest/advanced_guides/index.html#component-customization)

4. For users of MMDetection 2.x version, we provide a guide to help you adapt to the new version. You can find it in the [migration guide](./migration/migration.md).

}
```

<details>

```bibtex
}
```

}
```

| [pose_hrnetv2_w18_dark](/configs/hand_2d_keypoint/topdown_heatmap/onehand10k/td-hm_hrnetv2-w18_dark-8xb64-210e_onehand10k-256x256.py) |  256x256   |  0.990  | 0.572 | 23.96 | [ckpt](https://download.openmmlab.com/mmpose/hand/dark/hrnetv2_w18_onehand10k_256x256_dark-a2f80c64_20210330.pth) | [log](https://download.openmmlab.com/mmpose/hand/dark/hrnetv2_w18_onehand10k_256x256_dark_20210330.log.json) |

# Preparing THUMOS'14

```BibTeX
}
```

For basic dataset information, you can refer to the dataset [website](https://www.crcv.ucf.edu/THUMOS14/download.html).
Before we start, please make sure that the directory is located at `$MMACTION2/tools/data/thumos14/`.

First of all, run the following script to prepare annotations.

```shell
```

```shell
```

```

```shell
```

```shell
```

```shell

```shell
```

```shell
```

```shell
```

## Step 4. Fetch File List

This part is **optional** if you do not use SSN model.

You can run the follow script to fetch pre-computed tag proposals.

```shell
```

## Step 5. Denormalize Proposal File

You can run the follow script to denormalize pre-computed tag proposals according to
actual number of local rawframes.

```shell
```

```shell
```

After the whole data process for THUMOS'14 preparation,
you will get the rawframes (RGB + Flow), videos and annotation files for THUMOS'14.

In the context of the whole project (for THUMOS'14 only), the folder structure will look like:

```
mmaction2
├── tools
├── data
│   │   │   ├── val
│   │   │   ├── val

│   │   │   ├── val
```

For training and evaluating on THUMOS'14, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).

# ASTER

> [ASTER: An Attentional Scene Text Recognizer with Flexible Rectification](https://ieeexplore.ieee.org/abstract/document/8395027/)

A challenging aspect of scene text recognition is to handle text with distortions or irregular layout. In particular, perspective text and curved text are common in natural scenes and are difficult to recognize. In this work, we introduce ASTER, an end-to-end neural network model that comprises a rectification network and a recognition network. The rectification network adaptively transforms an input image into a new one, rectifying the text in it. It is powered by a flexible Thin-Plate Spline transformation which handles a variety of text irregularities and is trained without human annotations. The recognition network is an attentional sequence-to-sequence model that predicts a character sequence directly from the rectified image. The whole model is trained end to end, requiring only images and their groundtruth text. Through extensive experiments, we verify the effectiveness of the rectification and demonstrate the state-of-the-art recognition performance of ASTER. Furthermore, we demonstrate that ASTER is a powerful component in end-to-end recognition systems, for its ability to enhance the detector.

<img src="https://user-images.githubusercontent.com/65173622/207841597-fcd596cf-20eb-42db-9108-21e586dd9109.png"/>
</div>

## Dataset

|                             Methods                              | Backbone |        | Regular Text |           |     |           | Irregular Text |        |                               download                                |

| :--------------------------------------------------------------: | :------: | :----: | :----------: | :-------: | :-: | :-------: | :------------: | :----: | :-------------------------------------------------------------------: |
|                                                                  |          | IIIT5K |     SVT      | IC13-1015 |     | IC15-2077 |      SVTP      |  CT80  |                                                                       |
|   [ASTER](/configs/textrecog/aster/aster_resnet45_6e_st_mj.py)   | ResNet45 | 0.9357 |    0.8949    |  0.9281   |     |  0.7665   |     0.8062     | 0.8507 | [model](https://download.openmmlab.com/mmocr/textrecog/aster/aster_resnet45_6e_st_mj/aster_resnet45_6e_st_mj-cc56eca4.pth) \| [log](https://download.openmmlab.com/mmocr/textrecog/aster/aster_resnet45_6e_st_mj/20221214_232605.log) |

| [ASTER-TTA](/configs/textrecog/aster/aster_resnet45_6e_st_mj.py) | ResNet45 | 0.9337 |    0.8949    |  0.9251   |     |  0.7925   |     0.8109     | 0.8507 |                                                                       |

```bibtex
@article{shi2018aster,
  title={Aster: An attentional scene text recognizer with flexible rectification},
  author={Shi, Baoguang and Yang, Mingkun and Wang, Xinggang and Lyu, Pengyuan and Yao, Cong and Bai, Xiang},
  volume={41},
  number={9},
  pages={2035--2048},
}
```

# Simple Keypoints

Author： @2120140200@mail.nankai.edu.cn

It is a simple keypoints detector model. The model predict a score heatmap and an encoded location map.
The result in wflw achieves 3.94 NME.

## Usage

```shell
```

```shell

```shell
mim train mmpose configs/td-hm_hrnetv2-w18_skps-1xb64-80e_wflw-256x256.py
```

```shell
mim train mmpose configs/td-hm_hrnetv2-w18_skps-1xb64-80e_wflw-256x256.py --launcher pytorch --gpus 8
```

```shell
mim train mmpose configs/td-hm_hrnetv2-w18_skps-1xb64-80e_wflw-256x256.py --launcher slurm \
```

```shell
mim test mmpose configs/td-hm_hrnetv2-w18_skps-1xb64-80e_wflw-256x256.py -C $CHECKPOINT
```

```shell
mim test mmpose configs/td-hm_hrnetv2-w18_skps-1xb64-80e_wflw-256x256.py -C $CHECKPOINT --launcher pytorch --gpus 8
```

```shell
mim test mmpose configs/td-hm_hrnetv2-w18_skps-1xb64-80e_wflw-256x256.py -C $CHECKPOINT --launcher slurm \

```

```shell
```

## Results

WFLW

| [skps](./configs/td-hm_hrnetv2-w18_skps-1xb64-80e_wflw-256x256.py) |  256x256   |         3.88         |         6.60         |             3.81             |           4.57            |         4.44         |          3.75          |            4.13            | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/skps/best_NME_epoch_80.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/skps/20230522_142437.log) |

COFW

| [skps](./configs/td-hm_hrnetv2-w18_skps-1xb16-160e_cofw-256x256.py) |  256x256   | 3.20 | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/skps/best_NME_epoch_113.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/skps/20230524_074949.log) |

# 准备 DIV2K 数据集

```bibtex
}
```

- 训练集:  [DIV2K dataset](https://data.vision.ee.ethz.ch/cvl/DIV2K/).
- 验证集:  [Set5](https://drive.google.com/drive/folders/1B3DJGQKB6eNdwuQIhdskA64qUuVKLZ9u) 和 [Set14](https://drive.google.com/drive/folders/1B3DJGQKB6eNdwuQIhdskA64qUuVKLZ9u).

请注意，我们将原始的验证集（文件名 0801 到 0900）合并进了原始的训练集（文件名 0001 到 0800）。文件目录结构应如下所示：

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── DIV2K
│   │   │   ├── ...
│   │   │   ├── ...

│   │   │   ├── ...
│   │   │   ├── ...
│   │   │   ├── X2
│   │   │   ├── X3
│   │   │   ├── X4
│   │   │   ├── X2
│   │   │   ├── X3
│   │   │   ├── X4
│   ├── Set5
│   ├── Set14
```

## 裁剪子图

为了加快 IO，建议将 DIV2K 中的图片裁剪为一系列子图，为此，我们提供了一个脚本：

```shell
```

生成的数据保存在 `DIV2K` 目录下，其文件结构如下所示，其中 `_sub` 表示子图:

```text
mmagic
├── mmagic
├── tools
├── data
│   ├── DIV2K
│   │   │   ├── X2
│   │   │   ├── X3
│   │   │   ├── X4

├── tools
├── data
│   ├── DIV2K
│   │   │   ├── X2
│   │   │   ├── X3
│   │   │   ├── X4
│   │   ├── ...
...
```

## 准备标注列表文件

如果您想使用`标注模式`来处理数据集，需要先准备一个 `txt` 格式的标注文件。

标注文件中的每一行包含了图片名以及图片尺寸（这些通常是 ground-truth 图片），这两个字段用空格间隔开。

标注文件示例:

```text
```

请注意，`preprocess_div2k_dataset` 脚本默认生成一份标注文件。

## 准备 LMDB 格式的 DIV2K 数据集

如果您想使用 `LMDB` 以获得更快的 IO 速度，可以通过以下脚本来构建 LMDB 文件

```shell
```

# wandb记录特征图可视化

MMSegmentation 1.x 提供了 Weights & Biases 的后端支持，方便对项目代码结果的可视化和管理。

## Wandb的配置

安装 Weights & Biases 的过程可以参考 [官方安装指南](https://docs.wandb.ai/quickstart)，具体的步骤如下:

```shell
```

在 `vis_backend` 中添加 `WandbVisBackend`。

```python
```

## 测试数据和结果及特征图的可视化

`SegLocalVisualizer` 是继承自 MMEngine 中 `Visualizer` 类的子类，适用于 MMSegmentation 可视化，有关 `Visualizer` 的详细信息请参考在 MMEngine 中的[可视化教程](https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/visualization.html) 。

</div>

```shell

</div>

```shell

```

```python

        pass

vis_backends=[dict(type='WandbVisBackend')],

if args.gt_mask:

    main()

```

将上述代码保存为 feature_map_visual.py，在终端执行如下代码

```shell
python feature_map_visual.py ${图像} ${配置文件} ${检查点文件} [可选参数]
```

样例

```shell
```

可视化后的图像结果和它的对应的 feature map图像会出现在wandb账户中

</div>

# Segmenter

> [Segmenter: Transformer for Semantic Segmentation](https://arxiv.org/abs/2105.05633)

<a href="https://github.com/rstrudel/segmenter">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.21.0/mmseg/models/decode_heads/segmenter_mask_head.py#L15">Code Snippet</a>

Image segmentation is often ambiguous at the level of individual image patches and requires contextual information to reach label consensus. In this paper we introduce Segmenter, a transformer model for semantic segmentation. In contrast to convolution-based methods, our approach allows to model global context already at the first layer and throughout the network. We build on the recent Vision Transformer (ViT) and extend it to semantic segmentation. To do so, we rely on the output embeddings corresponding to image patches and obtain class labels from these embeddings with a point-wise linear decoder or a mask transformer decoder. We leverage models pre-trained for image classification and show that we can fine-tune them on moderate sized datasets available for semantic segmentation. The linear decoder allows to obtain excellent results already, but the performance can be further improved by a mask transformer generating class masks. We conduct an extensive ablation study to show the impact of the different parameters, in particular the performance is better for large models and small patch sizes. Segmenter attains excellent results for semantic segmentation. It outperforms the state of the art on both ADE20K and Pascal Context datasets and is competitive on Cityscapes.

<img src="https://user-images.githubusercontent.com/24582831/148507554-87eb80bd-02c7-4c31-b102-c6141e231ec8.png" width="70%"/>
</div>

## Usage

We have provided pretrained models converted from [ViT-AugReg](https://github.com/rwightman/pytorch-image-models/blob/f55c22bebf9d8afc449d317a723231ef72e0d662/timm/models/vision_transformer.py#L54-L106).

If you want to convert keys on your own to use the pre-trained ViT model from [Segmenter](https://github.com/rstrudel/segmenter), we also provide a script [`vitjax2mmseg.py`](../../tools/model_converters/vitjax2mmseg.py) in the tools directory to convert the key of models from [ViT-AugReg](https://github.com/rwightman/pytorch-image-models/blob/f55c22bebf9d8afc449d317a723231ef72e0d662/timm/models/vision_transformer.py#L54-L106) to MMSegmentation style.

```shell
python tools/model_converters/vitjax2mmseg.py ${PRETRAIN_PATH} ${STORE_PATH}
```

E.g.

```shell
python tools/model_converters/vitjax2mmseg.py \

```shell
```

E.g.

```shell
Ti_16-i21k-300ep-lr_0.001-aug_none-wd_0.03-do_0.0-sd_0.0--imagenet2012-steps_20k-lr_0.03-res_384.npz \
pretrain/vit_tiny_p16_384.pth
```

In our default setting, pretrained models and their corresponding [ViT-AugReg](https://github.com/rwightman/pytorch-image-models/blob/f55c22bebf9d8afc449d317a723231ef72e0d662/timm/models/vision_transformer.py#L54-L106) models could be defined below:

| pretrained models     | original models                                                                                                                                                                 |

| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| vit_tiny_p16_384.pth  | [vit_tiny_patch16_384](https://storage.googleapis.com/vit_models/augreg/Ti_16-i21k-300ep-lr_0.001-aug_none-wd_0.03-do_0.0-sd_0.0--imagenet2012-steps_20k-lr_0.03-res_384.npz)   |
| vit_small_p16_384.pth | [vit_small_patch16_384](https://storage.googleapis.com/vit_models/augreg/S_16-i21k-300ep-lr_0.001-aug_light1-wd_0.03-do_0.0-sd_0.0--imagenet2012-steps_20k-lr_0.03-res_384.npz) |
| vit_base_p16_384.pth  | [vit_base_patch16_384](https://storage.googleapis.com/vit_models/augreg/B_16-i21k-300ep-lr_0.001-aug_medium1-wd_0.1-do_0.0-sd_0.0--imagenet2012-steps_20k-lr_0.01-res_384.npz)  |

| vit_large_p16_384.pth | [vit_large_patch16_384](https://storage.googleapis.com/vit_models/augreg/L_16-i21k-300ep-lr_0.001-aug_medium1-wd_0.1-do_0.1-sd_0.1--imagenet2012-steps_20k-lr_0.01-res_384.npz) |

### ADE20K

| Method           | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device | mIoU  | mIoU(ms+flip) | config                                                                                                                              | download                                                                                                                                                                                                                                                                                                                                                                                       |

| Segmenter Mask   | ViT-T_16 | 512x512   | 160000  | 1.21     | 27.98          | V100   | 39.99 | 40.83         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segmenter/segmenter_vit-t_mask_8xb1-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segmenter/segmenter_vit-t_mask_8x1_512x512_160k_ade20k/segmenter_vit-t_mask_8x1_512x512_160k_ade20k_20220105_151706-ffcf7509.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segmenter/segmenter_vit-t_mask_8x1_512x512_160k_ade20k/segmenter_vit-t_mask_8x1_512x512_160k_ade20k_20220105_151706.log.json)         |

| Segmenter Linear | ViT-S_16 | 512x512   | 160000  | 1.78     | 28.07          | V100   | 45.75 | 46.82         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segmenter/segmenter_vit-s_fcn_8xb1-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segmenter/segmenter_vit-s_linear_8x1_512x512_160k_ade20k/segmenter_vit-s_linear_8x1_512x512_160k_ade20k_20220105_151713-39658c46.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segmenter/segmenter_vit-s_linear_8x1_512x512_160k_ade20k/segmenter_vit-s_linear_8x1_512x512_160k_ade20k_20220105_151713.log.json) |

| Segmenter Mask   | ViT-S_16 | 512x512   | 160000  | 2.03     | 24.80          | V100   | 46.19 | 47.85         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segmenter/segmenter_vit-s_mask_8xb1-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segmenter/segmenter_vit-s_mask_8x1_512x512_160k_ade20k/segmenter_vit-s_mask_8x1_512x512_160k_ade20k_20220105_151706-511bb103.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segmenter/segmenter_vit-s_mask_8x1_512x512_160k_ade20k/segmenter_vit-s_mask_8x1_512x512_160k_ade20k_20220105_151706.log.json)         |

| Segmenter Mask   | ViT-B_16 | 512x512   | 160000  | 4.20     | 13.20          | V100   | 49.60 | 51.07         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segmenter/segmenter_vit-b_mask_8xb1-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segmenter/segmenter_vit-b_mask_8x1_512x512_160k_ade20k/segmenter_vit-b_mask_8x1_512x512_160k_ade20k_20220105_151706-bc533b08.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segmenter/segmenter_vit-b_mask_8x1_512x512_160k_ade20k/segmenter_vit-b_mask_8x1_512x512_160k_ade20k_20220105_151706.log.json)         |

| Segmenter Mask   | ViT-L_16 | 640x640   | 160000  | 16.56    | 2.62           | V100   | 52.16 | 53.65         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segmenter/segmenter_vit-l_mask_8xb1-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segmenter/segmenter_vit-l_mask_8x1_512x512_160k_ade20k/segmenter_vit-l_mask_8x1_512x512_160k_ade20k_20220105_162750-7ef345be.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segmenter/segmenter_vit-l_mask_8x1_512x512_160k_ade20k/segmenter_vit-l_mask_8x1_512x512_160k_ade20k_20220105_162750.log.json)         |

```bibtex
@inproceedings{strudel2021segmenter,
  title={Segmenter: Transformer for semantic segmentation},
  author={Strudel, Robin and Garcia, Ricardo and Laptev, Ivan and Schmid, Cordelia},
  pages={7262--7272},
}
```

> [EDVR: Video Restoration with Enhanced Deformable Convolutional Networks](https://arxiv.org/abs/1905.02716?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%253A+arxiv%252FQSXk+%2528ExcitingAds%2521+cs+updates+on+arXiv.org%2529)

Video restoration tasks, including super-resolution, deblurring, etc, are drawing increasing attention in the computer vision community. A challenging benchmark named REDS is released in the NTIRE19 Challenge. This new benchmark challenges existing methods from two aspects: (1) how to align multiple frames given large motions, and (2) how to effectively fuse different frames with diverse motion and blur. In this work, we propose a novel Video Restoration framework with Enhanced Deformable networks, termed EDVR, to address these challenges. First, to handle large motions, we devise a Pyramid, Cascading and Deformable (PCD) alignment module, in which frame alignment is done at the feature level using deformable convolutions in a coarse-to-fine manner. Second, we propose a Temporal and Spatial Attention (TSA) fusion module, in which attention is applied both temporally and spatially, so as to emphasize important features for subsequent restoration. Thanks to these modules, our EDVR wins the champions and outperforms the second place by a large margin in all four tracks in the NTIRE19 video restoration and enhancement challenges. EDVR also demonstrates superior performance to state-of-the-art published methods on video super-resolution and deblurring.

 <img src="https://user-images.githubusercontent.com/7676947/144018263-6a1f74a4-d011-47fd-906b-290dd77eed64.png" width="400"/>
</div >

|                                    Model                                     | Dataset |  PSNR   |  SSIM  |    Training Resources    |                                     Download                                     |
| :--------------------------------------------------------------------------: | :-----: | :-----: | :----: | :----------------------: | :------------------------------------------------------------------------------: |

|       [edvrm_wotsa_x4_8x4_600k_reds](./edvrm_wotsa_8xb4-600k_reds.py)        |  REDS   | 30.3430 | 0.8664 |            8             | [model](https://download.openmmlab.com/mmediting/restorers/edvr/edvrm_wotsa_x4_8x4_600k_reds_20200522-0570e567.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/edvr/edvrm_wotsa_x4_8x4_600k_reds_20200522_141644.log.json) |
|             [edvrm_x4_8x4_600k_reds](./edvrm_8xb4-600k_reds.py)              |  REDS   | 30.4194 | 0.8684 |            8             | [model](https://download.openmmlab.com/mmediting/restorers/edvr/edvrm_x4_8x4_600k_reds_20210625-e29b71b5.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/edvr/edvrm_x4_8x4_600k_reds_20200622_102544.log.json) |

| [edvrl_wotsa_c128b40_8x8_lr2e-4_600k_reds4](./edvrl_wotsa-c128b40_8xb8-lr2e-4-600k_reds4.py) |  REDS   | 31.0010 | 0.8784 | 8 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/edvr/edvrl_wotsa_c128b40_8x8_lr2e-4_600k_reds4_20211228-d895a769.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/edvr/edvrl_wotsa_c128b40_8x8_lr2e-4_600k_reds4_20211228_144658.log.json) |
| [edvrl_c128b40_8x8_lr2e-4_600k_reds4](./edvrl_c128b40_8xb8-lr2e-4-600k_reds4.py) |  REDS   | 31.0467 | 0.8793 | 8 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/edvr/edvrl_c128b40_8x8_lr2e-4_600k_reds4_20220104-4509865f.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/edvr/edvrl_c128b40_8x8_lr2e-4_600k_reds4_20220104_171823.log.json) |

**Train**

<details>

```shell
# cpu train

**Train**

<details>

```shell
# cpu train

```

**Test**

<details>

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py python tools/test.py configs/edvr/edvrm_8xb4-600k_reds.py https://download.openmmlab.com/mmediting/restorers/edvr/edvrm_x4_8x4_600k_reds_20210625-e29b71b5.pth

```

```bibtex
}
```

<details>

```bibtex
}
```

<details>

```bibtex

}
```

| [pose_hrnetv2_w18](/configs/face_2d_keypoint/topdown_heatmap/wflw/td-hm_hrnetv2-w18_8xb64-60e_wflw-256x256.py) |  256x256   |         4.06         |         6.97         |             3.99             |           4.83            |         4.58         |          3.94          |            4.33            | [ckpt](https://download.openmmlab.com/mmpose/face/hrnetv2/hrnetv2_w18_wflw_256x256-2bf032a6_20210125.pth) | [log](https://download.openmmlab.com/mmpose/face/hrnetv2/hrnetv2_w18_wflw_256x256_20210125.log.json) |

# 在 mmsegmentation projects 中贡献一个标准格式的数据集

- 在开始您的贡献流程前，请先阅读[《OpenMMLab 贡献代码指南》](https://mmcv.readthedocs.io/zh_CN/latest/community/contributing.html)，以详细的了解 OpenMMLab 代码库的代码贡献流程。
- 该教程以 [Gaofen Image Dataset (GID)](https://www.sciencedirect.com/science/article/pii/S0034425719303414) 高分 2 号卫星所拍摄的遥感图像语义分割数据集作为样例，来演示在 mmsegmentation 中的数据集贡献流程。

## 步骤 1： 配置 mmsegmentation 开发所需必要环境

- 开发所必需的环境安装请参考[中文快速入门指南](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/zh_cn/get_started.md)或[英文 get_started](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/en/get_started.md)。

- 如果您已安装了最新版的 pytorch、mmcv、mmengine，那么您可以跳过步骤 1 至[步骤 2](<#[步骤-2](#%E6%AD%A5%E9%AA%A4-2%E4%BB%A3%E7%A0%81%E8%B4%A1%E7%8C%AE%E5%89%8D%E7%9A%84%E5%87%86%E5%A4%87%E5%B7%A5%E4%BD%9C)>)。

- **注：** 在此处无需安装 mmsegmentation，只需安装开发 mmsegmentation 所必需的 pytorch、mmcv、mmengine 等即可。

**新建虚拟环境（如已有合适的开发环境，可跳过）**

- 从[官方网站](https://docs.conda.io/en/latest/miniconda.html)下载并安装 Miniconda
- 创建一个 conda 环境，并激活

```shell

```shell
```

**安装 pytorch （如环境下已安装 pytorch，可跳过）**

- 参考 [official instructions](https://pytorch.org/get-started/locally/) 安装 **PyTorch**

**使用 mim 安装 mmcv、mmengine**

- 使用 [MIM](https://github.com/open-mmlab/mim) 安装 [MMCV](https://github.com/open-mmlab/mmcv)

```shell
```

## 步骤 2：代码贡献前的准备工作

### 2.1 Fork mmsegmentation 仓库

- 通过浏览器打开[mmsegmentation 官方仓库](https://github.com/open-mmlab/mmsegmentation/tree/main)。
- 登录您的 GitHub 账户，以下步骤均需在 GitHub 登录的情况下进行。
- Fork mmsegmentation 仓库
  ![image](https://user-images.githubusercontent.com/50650583/233825567-b8bf273c-38f5-4487-b4c6-75ede1e283ee.png)
- Fork 之后，mmsegmentation 仓库将会出现在您的个人仓库中。

### 2.2 在您的代码编写软件中 git clone mmsegmentation

这里以 VSCODE 为例

- 打开 VSCODE，新建终端窗口并激活您在[步骤 1 ](#%E6%AD%A5%E9%AA%A4-1-%E9%85%8D%E7%BD%AE-mmsegmentation-%E5%BC%80%E5%8F%91%E6%89%80%E9%9C%80%E5%BF%85%E8%A6%81%E7%8E%AF%E5%A2%83)中所安装的虚拟环境。
- 在您 GitHub 的个人仓库中找到您 Fork 的 mmsegmentation 仓库，复制其链接。
  ![image](https://github.com/AI-Tianlong/OpenMMLabCamp/assets/50650583/92ad555b-c5b2-4a7f-a800-ebee1e405ab6)
- 在终端中执行命令
  ```bash
  git clone {您所复制的个人仓库的链接}
  ```
  ![image](https://github.com/AI-Tianlong/OpenMMLabCamp/assets/50650583/23ba2636-e66f-4ea5-9077-9dd6b69deb1d)
  **注：** 如提示以下信息，请在 GitHub 中添加 [SSH 秘钥](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent)
  ![image](https://github.com/AI-Tianlong/OpenMMLabCamp/assets/50650583/6fcab213-0739-483c-b345-c59656027377)
- 进入 mmsegmentation 目录（之后的操作均在 mmsegmentation 目录下）。
  ```bash
  cd mmsegmentation
  ```
- 在终端中执行以下命令，添加官方仓库为上游仓库。

  ```bash
  ```
  ```bash
  git remote add upstream git@github.com:open-mmlab/mmsegmentation.git
  ```
- 使用以下命令检查 remote 是否添加成功。
  ```bash
  git remote -v
  ```
  ![image](https://github.com/AI-Tianlong/OpenMMLabCamp/assets/50650583/beec7e5e-2b00-4e49-ab38-f0c79e346594)

### 2.3 切换目录至 mmsegmentation 并从源码安装mmsegmentation

在`mmsegmentation`目录下执行`pip install -v -e .`，通过源码构建方式安装 mmsegmentaion 库。
安装完成后，您将能看到如下图所示的文件树。
<img src="https://user-images.githubusercontent.com/50650583/233826064-4b111358-8f97-44dd-955c-df3204410b8b.png" alt="image" style="zoom:67%;" />

### 2.4 切换分支为 dev-1.x

正如您在[ mmsegmentation 官网](https://github.com/open-mmlab/mmsegmentation/tree/main)所见，该仓库有许多分支，默认分支`main`为稳定的发行版本，以及用于贡献者进行开发的`dev-1.x`分支。`dev-1.x`分支是贡献者们用来提交创意和 PR 的分支，`dev-1.x`分支的内容会被周期性的合入到`main`分支。
![image](https://user-images.githubusercontent.com/50650583/233826225-f4b7299d-de23-47db-900d-dfb01ba0efc3.png)

回到 VSCODE 中，在终端执行命令

```bash
git checkout dev-1.x
```

### 2.5 创新属于自己的新分支

在基于`dev-1.x`分支下，使用如下命令，创建属于您自己的分支。

```bash
# git checkout -b 您的GitHubID/您的分支想要实现的功能的名字
# git checkout -b AI-Tianlong/support_GID_dataset
git checkout -b {您的GitHubID/您的分支想要实现的功能的名字}
```

### 2.6 配置 pre-commit

OpenMMLab 仓库对代码质量有着较高的要求，所有提交的 PR 必须要通过代码格式检查。pre-commit 详细配置参阅[配置 pre-commit](https://mmcv.readthedocs.io/zh_CN/latest/community/contributing.html#pre-commit)。

## 步骤 3：在`mmsegmentation/projects`下贡献您的代码

**先对 GID 数据集进行分析**

这里以贡献高分 2 号遥感图像语义分割数据集 GID 为例，GID 数据集是由我国自主研发的高分 2 号卫星所拍摄的光学遥感图像所创建，经图像预处理后共提供了 150 张 6800x7200 像素的 RGB 三通道遥感图像。并提供了两种不同类别数的数据标注，一种是包含 5 类有效物体的 RGB 标签，另一种是包含 15 类有效物体的 RGB 标签。本教程将针对 5 类标签进行数据集贡献流程讲解。

GID 的 5 类有效标签分别为：0-背景-\[0,0,0\](mask 标签值-标签名称-RGB 标签值)、1-建筑-\[255,0,0\]、2-农田-\[0,255,0\]、3-森林-\[0,0,255\]、4-草地-\[255,255,0\]、5-水-\[0,0,255\]。在语义分割任务中，标签是与原图尺寸一致的单通道图像，标签图像中的像素值为真实样本图像中对应像素所包含的物体的类别。GID 数据集提供的是具有 RGB 三通道的彩色标签，为了模型的训练需要将 RGB 标签转换为 mask 标签。并且由于图像尺寸为 6800x7200 像素，对于神经网络的训练来有些过大，所以将每张图像裁切成了没有重叠的 512x512 的图像以便进行训练。
<img align='center' src="https://user-images.githubusercontent.com/50650583/234192183-83ee4209-e181-4a18-90ca-4d71757cd2c7.png" alt="image" style="zoom:67%;" />

### 3.1 在`mmsegmentation/projects`下创建新的项目文件夹

在`mmsegmentation/projects`下创建文件夹`gid_dataset`
![image](https://user-images.githubusercontent.com/50650583/233829687-8f2b6600-bc9d-48ff-a865-d462af54d55a.png)

### 3.2 贡献您的数据集代码

为了最终能将您在 projects 中贡献的代码更加顺畅的移入核心库中（对代码要求质量更高），非常建议按照核心库的目录来编辑您的数据集文件。

关于数据集有 4 个必要的文件：

- **1**  `mmseg/datasets/gid.py` 定义了数据集的尾缀、CLASSES、PALETTE、reduce_zero_label等
- **2** `configs/_base_/gid.py` GID 数据集的配置文件，定义了数据集的`dataset_type`（数据集类型，`mmseg/datasets/gid.py`中注册的数据集的类名）、`data_root`(数据集所在的根目录，建议将数据集通过软连接的方式将数据集放至`mmsegmentation/data`)、`train_pipline`(训练的数据流)、`test_pipline`(测试和验证时的数据流)、`img_rations`(多尺度预测时的多尺度配置)、`tta_pipeline`（多尺度预测）、`train_dataloader`(训练集的数据加载器)、`val_dataloader`(验证集的数据加载器)、`test_dataloader`(测试集的数据加载器)、`val_evaluator`(验证集的评估器)、`test_evaluator`(测试集的评估器)。
- **3** 使用了 GID 数据集的模型训练配置文件
  这个是可选的，但是强烈建议您添加。在核心库中，所贡献的数据集需要和参考文献中所提出的结果精度对齐，为了后期将您贡献的代码合并入核心库。如您的算力充足，最好能提供对应的模型配置文件在您贡献的数据集上所验证的结果以及相应的权重文件，并撰写较为详细的README.md文档。[示例参考结果](https://github.com/open-mmlab/mmsegmentation/tree/main/configs/deeplabv3plus#mapillary-vistas-v12)
  ![image](https://user-images.githubusercontent.com/50650583/233877682-eabe8723-bce9-40e4-a303-08c8385cb6b5.png)

![image](https://user-images.githubusercontent.com/50650583/233877682-eabe8723-bce9-40e4-a303-08c8385cb6b5.png)
- **4** 使用如下命令格式： 撰写`docs/zh_cn/user_guides/2_dataset_prepare.md`来添加您的数据集介绍，包括但不限于数据集的下载方式，数据集目录结构、数据集生成等一些必要性的文字性描述和运行命令。以更好地帮助用户能更快的实现数据集的准备工作。

### 3.3 贡献`tools/dataset_converters/gid.py`

由于 GID 数据集是由未经过切分的 6800x7200 图像所构成的数据集，并且没有划分训练集、验证集与测试集。以及其标签为 RGB 彩色标签，需要将标签转换为单通道的 mask label。为了方便训练，首先将 GID 数据集进行裁切和标签转换，并进行数据集划分，构建为 mmsegmentation 所支持的格式。

```python
# tools/dataset_converters/gid.py
import glob
import math
import os

from mmengine.utils import ProgressBar, mkdir_or_exist

        description='Convert GID dataset to mmsegmentation format')
    parser.add_argument('dataset_img_path', help='GID images folder path')
    parser.add_argument('dataset_label_path', help='GID labels folder path')

parser.add_argument('dataset_img_path', help='GID images folder path')
    parser.add_argument('--tmp_dir', help='path of the temporary directory')
    parser.add_argument('-o', '--out_dir', help='output path', default='data/gid')
        '--clip_size',
        type=int,
        help='clipped size of image after preparation',
        default=256)
        '--stride_size',
        type=int,
        help='stride of clipping original images',

GID_COLORMAP = dict(
    Background=(0, 0, 0), #0-背景-黑色
    Building=(255, 0, 0), #1-建筑-红色
    Farmland=(0, 255, 0), #2-农田-绿色
    Forest=(0, 0, 255), #3-森林-蓝色
    Meadow=(255, 255, 0),#4-草地-黄色
    Water=(0, 0, 255)#5-水-蓝色
)
palette = list(GID_COLORMAP.values())
classes = list(GID_COLORMAP.keys())

#############用列表来存一个 RGB 和一个类别的对应################

Meadow=(255, 255, 0),#4-草地-黄色
)

def colormap2label(palette):
    colormap2label_list = np.zeros(256**3, dtype = np.longlong)
    for i, colormap in enumerate(palette):
        colormap2label_list[(colormap[0] * 256 + colormap[1])*256+colormap[2]] = i
    return colormap2label_list

#############给定那个列表，和vis_png然后生成masks_png################
def label_indices(RGB_label, colormap2label_list):
    RGB_label = RGB_label.astype('int32')
    idx = (RGB_label[:, :, 0] * 256 + RGB_label[:, :, 1]) * 256 + RGB_label[:, :, 2]
    # print(idx.shape)
    return colormap2label_list[idx]

def RGB2mask(RGB_label, colormap2label_list):
    # RGB_label = np.array(Image.open(RGB_label).convert('RGB')) #打开RGB_png
    mask_label = label_indices(RGB_label, colormap2label_list) # .numpy()
    return mask_label

# RGB_label = np.array(Image.open(RGB_label).convert('RGB')) #打开RGB_png

colormap2label_list = colormap2label(palette)

def clip_big_image(image_path, clip_save_dir, args, to_label=False):
    """
    Original image of GID dataset is very large, thus pre-processing
    of them is adopted. Given fixed clip size and stride size to generate
    clipped image, the intersection　of width and height is determined.
    For example, given one 6800 x 7200 original image, the clip size is
    256 and stride size is 256, thus it would generate 29 x 27 = 783 images
    whose size are all 256 x 256.

    """

    image = mmcv.imread(image_path, channel_order='rgb')
    # image = mmcv.bgr2gray(image)

    h, w, c = image.shape
    clip_size = args.clip_size
    stride_size = args.stride_size

    num_rows = math.ceil((h - clip_size) / stride_size) if math.ceil(
        (h - clip_size) /

h, w, c = image.shape

        stride_size) * stride_size + clip_size >= h else math.ceil(
            (h - clip_size) / stride_size) + 1
    num_cols = math.ceil((w - clip_size) / stride_size) if math.ceil(
        (w - clip_size) /
        stride_size) * stride_size + clip_size >= w else math.ceil(
            (w - clip_size) / stride_size) + 1

    x, y = np.meshgrid(np.arange(num_cols + 1), np.arange(num_rows + 1))
    xmin = x * clip_size
    ymin = y * clip_size

    xmin = xmin.ravel()
    ymin = ymin.ravel()
    xmin_offset = np.where(xmin + clip_size > w, w - xmin - clip_size,
                           np.zeros_like(xmin))
    ymin_offset = np.where(ymin + clip_size > h, h - ymin - clip_size,
                           np.zeros_like(ymin))
    boxes = np.stack([
        xmin + xmin_offset, ymin + ymin_offset,

ymin_offset = np.where(ymin + clip_size > h, h - ymin - clip_size,
        np.minimum(xmin + clip_size, w),
        np.minimum(ymin + clip_size, h)
    ], axis=1)

    if to_label:
        image = RGB2mask(image, colormap2label_list) #这里得改一下

    for count, box in enumerate(boxes):
        start_x, start_y, end_x, end_y = box
        clipped_image = image[start_y:end_y,
                              start_x:end_x] if to_label else image[
                                  start_y:end_y, start_x:end_x, :]
        img_name = osp.basename(image_path).replace('.tif', '')
        img_name = img_name.replace('_label', '')
        if count % 3 == 0:
            mmcv.imwrite(
                clipped_image.astype(np.uint8),
                osp.join(
                    clip_save_dir.replace('train', 'val'),

if count % 3 == 0:
                osp.join(
                    f'{img_name}_{start_x}_{start_y}_{end_x}_{end_y}.png'))
        else:
                osp.join(
                    clip_save_dir,
        count += 1

    """
    According to this paper: https://ieeexplore.ieee.org/document/9343296/
    select 15 images contained in GID, , which cover the whole six
    categories, to generate train set and validation set.

    According to Paper: https://ieeexplore.ieee.org/document/9343296/

    """

    if args.out_dir is None:
        out_dir = osp.join('data', 'gid')
    else:
        out_dir = args.out_dir

    print('Making directories...')

"""

    else:

    mkdir_or_exist(osp.join(out_dir, 'img_dir', 'train'))
    mkdir_or_exist(osp.join(out_dir, 'img_dir', 'val'))
    mkdir_or_exist(osp.join(out_dir, 'ann_dir', 'train'))
    mkdir_or_exist(osp.join(out_dir, 'ann_dir', 'val'))

    src_path_list = glob.glob(os.path.join(args.dataset_img_path, '*.tif'))
    print(f'Find {len(src_path_list)} pictures')

    prog_bar = ProgressBar(len(src_path_list))

    dst_img_dir = osp.join(out_dir, 'img_dir', 'train')
    dst_label_dir = osp.join(out_dir, 'ann_dir', 'train')

    for i, img_path in enumerate(src_path_list):
        label_path = osp.join(args.dataset_label_path, osp.basename(img_path.replace('.tif', '_label.tif')))

        clip_big_image(img_path, dst_img_dir, args, to_label=False)
        clip_big_image(label_path, dst_label_dir, args, to_label=True)
        prog_bar.update()

clip_big_image(img_path, dst_img_dir, args, to_label=False)

    print('Done!')

    main()
```

### 3.4 贡献`mmseg/datasets/gid.py`

可参考[`projects/mapillary_dataset/mmseg/datasets/mapillary.py`](https://github.com/open-mmlab/mmsegmentation/blob/main/projects/mapillary_dataset/mmseg/datasets/mapillary.py)并在此基础上修改相应变量以适配您的数据集。

```python
# mmseg/datasets/gid.py
from mmseg.datasets.basesegdataset import BaseSegDataset
from mmseg.registry import DATASETS

# 注册数据集类
class GID_Dataset(BaseSegDataset):
    """Gaofen Image Dataset (GID)

    Dataset paper link:
    https://www.sciencedirect.com/science/article/pii/S0034425719303414
    https://x-ytong.github.io/project/GID.html

    GID  6 classes: background(others), built-up, farmland, forest, meadow, water

https://x-ytong.github.io/project/GID.html

    In This example, select 10 images from GID dataset as training set,
    and select 5 images as validation set.
    The selected images are listed as follows:

    GF2_PMS1__L1A0000647767-MSS1
    GF2_PMS1__L1A0001064454-MSS1
    GF2_PMS1__L1A0001348919-MSS1
    GF2_PMS1__L1A0001680851-MSS1
    GF2_PMS1__L1A0001680853-MSS1
    GF2_PMS1__L1A0001680857-MSS1
    GF2_PMS1__L1A0001757429-MSS1
    GF2_PMS2__L1A0000607681-MSS2
    GF2_PMS2__L1A0000635115-MSS2
    GF2_PMS2__L1A0000658637-MSS2
    GF2_PMS2__L1A0001206072-MSS2
    GF2_PMS2__L1A0001471436-MSS2
    GF2_PMS2__L1A0001642620-MSS2
    GF2_PMS2__L1A0001787089-MSS2
    GF2_PMS2__L1A0001838560-MSS2

    The ``img_suffix`` is fixed to '.tif' and ``seg_map_suffix`` is
    fixed to '.tif' for GID.
    """
        classes=('Others', 'Built-up', 'Farmland', 'Forest',

The ``img_suffix`` is fixed to '.tif' and ``seg_map_suffix`` is
    """
                 'Meadow', 'Water'),

        palette=[[0, 0, 0], [255, 0, 0], [0, 255, 0], [0, 255, 255],
                 [255, 255, 0], [0, 0, 255]])

                 img_suffix='.png',
                 seg_map_suffix='.png',
                 reduce_zero_label=None,
            img_suffix=img_suffix,
            seg_map_suffix=seg_map_suffix,
            reduce_zero_label=reduce_zero_label,
            **kwargs)
```

### 3.5 贡献使用 GID 的训练 config file

```python
_base_ = [
    '../../../configs/_base_/models/deeplabv3plus_r50-d8.py',
    './_base_/datasets/gid.py',
    '../../../configs/_base_/default_runtime.py',
    '../../../configs/_base_/schedules/schedule_240k.py'
]

'./_base_/datasets/gid.py',
]
    imports=['projects.gid_dataset.mmseg.datasets.gid'])

crop_size = (256, 256)
    pretrained='open-mmlab://resnet101_v1c',
    backbone=dict(depth=101),
    decode_head=dict(num_classes=6),
    auxiliary_head=dict(num_classes=6))

```

### 3.6 撰写`docs/zh_cn/user_guides/2_dataset_prepare.md`

**Gaofen Image Dataset (GID)**

- GID 数据集可在[此处](https://x-ytong.github.io/project/Five-Billion-Pixels.html)进行下载。
- 此处选择 15 张图像生成训练集和验证集，该 15 张图像包含了所有六类信息。所选的图像名称如下：

```None

GF2_PMS1__L1A0001064454-MSS1
```

执行以下命令进行裁切及标签的转换，需要修改为您所存储 15 张图像及标签的路径。

```
python projects/gid_dataset/tools/dataset_converters/gid.py [15 张图像的路径] [15 张标签的路径]
```

```none
├── mmseg
├── tools
├── data
│   ├── gid

```

### 3.7 贡献的代码及文档通过`pre-commit`检查

使用命令

```bash
git add .
git commit -m "添加描述"
git push
```

### 3.8 在 GitHub 中向 mmsegmentation 提交 PR

```

使用命令

```bash
git add .
git push
```

具体步骤可见[《OpenMMLab 贡献代码指南》](https://mmcv.readthedocs.io/zh_CN/latest/community/contributing.html)

# HRNet

> [Deep High-Resolution Representation Learning for Human Pose Estimation](https://arxiv.org/abs/1908.07919)

<a href="https://github.com/HRNet/HRNet-Semantic-Segmentation">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/backbones/hrnet.py#L218">Code Snippet</a>

High-resolution representations are essential for position-sensitive vision problems, such as human pose estimation, semantic segmentation, and object detection. Existing state-of-the-art frameworks first encode the input image as a low-resolution representation through a subnetwork that is formed by connecting high-to-low resolution convolutions \\emph{in series} (e.g., ResNet, VGGNet), and then recover the high-resolution representation from the encoded low-resolution representation. Instead, our proposed network, named as High-Resolution Network (HRNet), maintains high-resolution representations through the whole process. There are two key characteristics: (i) Connect the high-to-low resolution convolution streams \\emph{in parallel}; (ii) Repeatedly exchange the information across resolutions. The benefit is that the resulting representation is semantically richer and spatially more precise. We show the superiority of the proposed HRNet in a wide range of applications, including human pose estimation, semantic segmentation, and object detection, suggesting that the HRNet is a stronger backbone for computer vision problems. All the codes are available at [this https URL](https://github.com/HRNet).

<img src="https://user-images.githubusercontent.com/24582831/142901680-64c285bc-669f-4924-b054-46a2f07c5427.png" width="80%"/>
</div>

| Method | Backbone           | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                    | download                                                                                                                                                                                                                                                                                                                               |

| FCN    | HRNetV2p-W18-Small | 512x1024  |   40000 | 1.7      | 23.74          | V100   | 73.86 |         75.91 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18s_4xb2-40k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_512x1024_40k_cityscapes/fcn_hr18s_512x1024_40k_cityscapes_20200601_014216-93db27d0.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_512x1024_40k_cityscapes/fcn_hr18s_512x1024_40k_cityscapes_20200601_014216.log.json)     |

| FCN    | HRNetV2p-W18       | 512x1024  |   40000 | 2.9      | 12.97          | V100   | 77.19 |         78.92 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18_4xb2-40k_cityscapes-512x1024.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_512x1024_40k_cityscapes/fcn_hr18_512x1024_40k_cityscapes_20200601_014216-f196fb4e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_512x1024_40k_cityscapes/fcn_hr18_512x1024_40k_cityscapes_20200601_014216.log.json)         |

| FCN    | HRNetV2p-W48       | 512x1024  |   40000 | 6.2      | 6.42           | V100   | 78.48 |         79.69 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr48_4xb2-40k_cityscapes-512x1024.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_512x1024_40k_cityscapes/fcn_hr48_512x1024_40k_cityscapes_20200601_014240-a989b146.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_512x1024_40k_cityscapes/fcn_hr48_512x1024_40k_cityscapes_20200601_014240.log.json)         |

| FCN    | HRNetV2p-W18-Small | 512x1024  |   80000 | -        | -              | V100   | 75.31 |         77.48 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18s_4xb2-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_512x1024_80k_cityscapes/fcn_hr18s_512x1024_80k_cityscapes_20200601_202700-1462b75d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_512x1024_80k_cityscapes/fcn_hr18s_512x1024_80k_cityscapes_20200601_202700.log.json)     |

| FCN    | HRNetV2p-W18       | 512x1024  |   80000 | -        | -              | V100   | 78.65 |         80.35 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18_4xb2-80k_cityscapes-512x1024.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_512x1024_80k_cityscapes/fcn_hr18_512x1024_80k_cityscapes_20200601_223255-4e7b345e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_512x1024_80k_cityscapes/fcn_hr18_512x1024_80k_cityscapes_20200601_223255.log.json)         |

| FCN    | HRNetV2p-W48       | 512x1024  |   80000 | -        | -              | V100   | 79.93 |         80.72 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr48_4xb2-80k_cityscapes-512x1024.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_512x1024_80k_cityscapes/fcn_hr48_512x1024_80k_cityscapes_20200601_202606-58ea95d6.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_512x1024_80k_cityscapes/fcn_hr48_512x1024_80k_cityscapes_20200601_202606.log.json)         |

| FCN    | HRNetV2p-W18-Small | 512x1024  |  160000 | -        | -              | V100   | 76.31 |         78.31 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18s_4xb2-160k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_512x1024_160k_cityscapes/fcn_hr18s_512x1024_160k_cityscapes_20200602_190901-4a0797ea.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_512x1024_160k_cityscapes/fcn_hr18s_512x1024_160k_cityscapes_20200602_190901.log.json) |

| FCN    | HRNetV2p-W18       | 512x1024  |  160000 | -        | -              | V100   | 78.80 |         80.74 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18_4xb2-160k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_512x1024_160k_cityscapes/fcn_hr18_512x1024_160k_cityscapes_20200602_190822-221e4a4f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_512x1024_160k_cityscapes/fcn_hr18_512x1024_160k_cityscapes_20200602_190822.log.json)     |

| FCN    | HRNetV2p-W48       | 512x1024  |  160000 | -        | -              | V100   | 80.65 |         81.92 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr48_4xb2-160k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_512x1024_160k_cityscapes/fcn_hr48_512x1024_160k_cityscapes_20200602_190946-59b7973e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_512x1024_160k_cityscapes/fcn_hr48_512x1024_160k_cityscapes_20200602_190946.log.json)     |

### ADE20K

### ADE20K

| Method | Backbone           | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                               | download                                                                                                                                                                                                                                                                                                           |

| FCN    | HRNetV2p-W18-Small | 512x512   |   80000 | 3.8      | 38.66          | V100   | 31.38 |         32.45 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18s_4xb4-80k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_512x512_80k_ade20k/fcn_hr18s_512x512_80k_ade20k_20200614_144345-77fc814a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_512x512_80k_ade20k/fcn_hr18s_512x512_80k_ade20k_20200614_144345.log.json)     |

| FCN    | HRNetV2p-W18       | 512x512   |   80000 | 4.9      | 22.57          | V100   | 36.27 |         37.28 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18_4xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_512x512_80k_ade20k/fcn_hr18_512x512_80k_ade20k_20210827_114910-6c9382c0.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_512x512_80k_ade20k/fcn_hr18_512x512_80k_ade20k_20210827_114910.log.json)         |

| FCN    | HRNetV2p-W48       | 512x512   |   80000 | 8.2      | 21.23          | V100   | 41.90 |         43.27 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr48_4xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_512x512_80k_ade20k/fcn_hr48_512x512_80k_ade20k_20200614_193946-7ba5258d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_512x512_80k_ade20k/fcn_hr48_512x512_80k_ade20k_20200614_193946.log.json)         |

| FCN    | HRNetV2p-W18-Small | 512x512   |  160000 | -        | -              | V100   | 33.07 |         34.56 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18s_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_512x512_160k_ade20k/fcn_hr18s_512x512_160k_ade20k_20210829_174739-f1e7c2e7.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_512x512_160k_ade20k/fcn_hr18s_512x512_160k_ade20k_20210829_174739.log.json) |

| FCN    | HRNetV2p-W18       | 512x512   |  160000 | -        | -              | V100   | 36.79 |         38.58 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18_4xb4-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_512x512_160k_ade20k/fcn_hr18_512x512_160k_ade20k_20200614_214426-ca961836.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_512x512_160k_ade20k/fcn_hr18_512x512_160k_ade20k_20200614_214426.log.json)     |

| FCN    | HRNetV2p-W48       | 512x512   |  160000 | -        | -              | V100   | 42.02 |         43.86 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr48_4xb4-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_512x512_160k_ade20k/fcn_hr48_512x512_160k_ade20k_20200614_214407-a52fc02c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_512x512_160k_ade20k/fcn_hr48_512x512_160k_ade20k_20200614_214407.log.json)     |

| Method | Backbone           | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                | download                                                                                                                                                                                                                                                                                                               |

| FCN    | HRNetV2p-W18-Small | 512x512   |   20000 | 1.8      | 43.36          | V100   |  65.5 |         68.89 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18s_4xb4-20k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_512x512_20k_voc12aug/fcn_hr18s_512x512_20k_voc12aug_20210829_174910-0aceadb4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_512x512_20k_voc12aug/fcn_hr18s_512x512_20k_voc12aug_20210829_174910.log.json) |

| FCN    | HRNetV2p-W18       | 512x512   |   20000 | 2.9      | 23.48          | V100   | 72.30 |         74.71 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18_4xb4-20k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_512x512_20k_voc12aug/fcn_hr18_512x512_20k_voc12aug_20200617_224503-488d45f7.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_512x512_20k_voc12aug/fcn_hr18_512x512_20k_voc12aug_20200617_224503.log.json)     |

| FCN    | HRNetV2p-W48       | 512x512   |   20000 | 6.2      | 22.05          | V100   | 75.87 |         78.58 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr48_4xb4-20k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_512x512_20k_voc12aug/fcn_hr48_512x512_20k_voc12aug_20200617_224419-89de05cd.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_512x512_20k_voc12aug/fcn_hr48_512x512_20k_voc12aug_20200617_224419.log.json)     |

| FCN    | HRNetV2p-W18-Small | 512x512   |   40000 | -        | -              | V100   | 66.61 |         70.00 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18s_4xb4-40k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_512x512_40k_voc12aug/fcn_hr18s_512x512_40k_voc12aug_20200614_000648-4f8d6e7f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_512x512_40k_voc12aug/fcn_hr18s_512x512_40k_voc12aug_20200614_000648.log.json) |

| FCN    | HRNetV2p-W18       | 512x512   |   40000 | -        | -              | V100   | 72.90 |         75.59 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18_4xb4-40k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_512x512_40k_voc12aug/fcn_hr18_512x512_40k_voc12aug_20200613_224401-1b4b76cd.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_512x512_40k_voc12aug/fcn_hr18_512x512_40k_voc12aug_20200613_224401.log.json)     |

| FCN    | HRNetV2p-W48       | 512x512   |   40000 | -        | -              | V100   | 76.24 |         78.49 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr48_4xb4-40k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_512x512_40k_voc12aug/fcn_hr48_512x512_40k_voc12aug_20200613_222111-1b0f18bc.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_512x512_40k_voc12aug/fcn_hr48_512x512_40k_voc12aug_20200613_222111.log.json)     |

| Method | Backbone     | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                     | download                                                                                                                                                                                                                                                                                                                                   |

| FCN    | HRNetV2p-W48 | 480x480   |   40000 | 6.1      | 8.86           | V100   | 45.14 |         47.42 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr48_4xb4-40k_pascal-context-480x480.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_480x480_40k_pascal_context/fcn_hr48_480x480_40k_pascal_context_20200911_164852-667d00b0.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_480x480_40k_pascal_context/fcn_hr48_480x480_40k_pascal_context-20200911_164852.log.json) |

| FCN    | HRNetV2p-W48 | 480x480   |   80000 | -        | -              | V100   | 45.84 |         47.84 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr48_4xb4-80k_pascal-context-480x480.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_480x480_80k_pascal_context/fcn_hr48_480x480_80k_pascal_context_20200911_155322-847a6711.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_480x480_80k_pascal_context/fcn_hr48_480x480_80k_pascal_context-20200911_155322.log.json) |

| Method | Backbone     | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                        | download                                                                                                                                                                                                                                                                                                                                               |

| FCN    | HRNetV2p-W48 | 480x480   |   40000 | -        | -              | V100   | 50.33 |         52.83 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr48_4xb4-40k_pascal-context-59-480x480.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_480x480_40k_pascal_context_59/fcn_hr48_480x480_40k_pascal_context_59_20210410_122738-b808b8b2.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_480x480_40k_pascal_context_59/fcn_hr48_480x480_40k_pascal_context_59-20210410_122738.log.json) |

| FCN    | HRNetV2p-W48 | 480x480   |   80000 | -        | -              | V100   | 51.12 |         53.56 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr48_4xb4-80k_pascal-context-59-480x480.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_480x480_80k_pascal_context_59/fcn_hr48_480x480_80k_pascal_context_59_20210411_003240-3ae7081e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_480x480_80k_pascal_context_59/fcn_hr48_480x480_80k_pascal_context_59-20210411_003240.log.json) |

### LoveDA

### LoveDA

| Method | Backbone           | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                               | download                                                                                                                                                                                                                                                                                                       |

| FCN    | HRNetV2p-W18-Small | 512x512   |   80000 | 1.59     | 24.87          | V100   | 49.28 |         49.42 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18s_4xb4-80k_loveda-512x512.pyy) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_512x512_80k_loveda/fcn_hr18s_512x512_80k_loveda_20211210_203228-60a86a7a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_512x512_80k_loveda/fcn_hr18s_512x512_80k_loveda_20211210_203228.log.json) |

| FCN    | HRNetV2p-W18       | 512x512   |   80000 | 2.76     | 12.92          | V100   | 50.81 |         50.95 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18_4xb4-80k_loveda-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_512x512_80k_loveda/fcn_hr18_512x512_80k_loveda_20211210_203952-93d9c3b3.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_512x512_80k_loveda/fcn_hr18_512x512_80k_loveda_20211210_203952.log.json)     |

| FCN    | HRNetV2p-W48       | 512x512   |   80000 | 6.20     | 9.61           | V100   | 51.42 |         51.64 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr48_4xb4-80k_loveda-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_512x512_80k_loveda/fcn_hr48_512x512_80k_loveda_20211211_044756-67072f55.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_512x512_80k_loveda/fcn_hr48_512x512_80k_loveda_20211211_044756.log.json)     |

| FCN    | HRNetV2p-W18-Small | 512x512   |   80000 | 1.58     | 36.00          | V100   | 77.64 |          78.8 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18s_4xb4-80k_potsdam-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_512x512_80k_potsdam/fcn_hr18s_512x512_80k_potsdam_20211218_205517-ba32af63.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_512x512_80k_potsdam/fcn_hr18s_512x512_80k_potsdam_20211218_205517.log.json) |

| FCN    | HRNetV2p-W18       | 512x512   |   80000 | 2.76     | 19.25          | V100   | 78.26 |         79.24 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18_4xb4-80k_potsdam-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_512x512_80k_potsdam/fcn_hr18_512x512_80k_potsdam_20211218_205517-5d0387ad.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_512x512_80k_potsdam/fcn_hr18_512x512_80k_potsdam_20211218_205517.log.json)     |

| FCN    | HRNetV2p-W48       | 512x512   |   80000 | 6.20     | 16.42          | V100   | 78.39 |         79.34 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr48_4xb4-80k_potsdam-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_512x512_80k_potsdam/fcn_hr48_512x512_80k_potsdam_20211219_020601-97434c78.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_512x512_80k_potsdam/fcn_hr48_512x512_80k_potsdam_20211219_020601.log.json)     |

| Method | Backbone           | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                 | download                                                                                                                                                                                                                                                                                                                                   |

| FCN    | HRNetV2p-W18-Small | 512x512   |   80000 | 1.58     | 38.11          | V100   | 71.81 |          73.1 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18s_4xb4-80k_vaihingen-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_4x4_512x512_80k_vaihingen/fcn_hr18s_4x4_512x512_80k_vaihingen_20211231_230909-b23aae02.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_4x4_512x512_80k_vaihingen/fcn_hr18s_4x4_512x512_80k_vaihingen_20211231_230909.log.json) |

| FCN    | HRNetV2p-W18       | 512x512   |   80000 | 2.76     | 19.55          | V100   | 72.57 |         74.09 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18_4xb4-80k_vaihingen-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_4x4_512x512_80k_vaihingen/fcn_hr18_4x4_512x512_80k_vaihingen_20211231_231216-2ec3ae8a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_4x4_512x512_80k_vaihingen/fcn_hr18_4x4_512x512_80k_vaihingen_20211231_231216.log.json)     |

| FCN    | HRNetV2p-W48       | 512x512   |   80000 | 6.20     | 17.25          | V100   | 72.50 |         73.52 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr48_4xb4-80k_vaihingen-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_4x4_512x512_80k_vaihingen/fcn_hr48_4x4_512x512_80k_vaihingen_20211231_231244-7133cb22.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_4x4_512x512_80k_vaihingen/fcn_hr48_4x4_512x512_80k_vaihingen_20211231_231244.log.json)     |

### iSAID

### iSAID

| Method | Backbone           | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                             | download                                                                                                                                                                                                                                                                                                                   |

| FCN    | HRNetV2p-W18-Small | 896x896   |   80000 | 4.95     | 13.84          | V100   | 62.30 |         62.97 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18s_4xb4-80k_isaid-896x896.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_4x4_896x896_80k_isaid/fcn_hr18s_4x4_896x896_80k_isaid_20220118_001603-3cc0769b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18s_4x4_896x896_80k_isaid/fcn_hr18s_4x4_896x896_80k_isaid_20220118_001603.log.json) |

| FCN    | HRNetV2p-W18       | 896x896   |   80000 | 8.30     | 7.71           | V100   | 65.06 |         65.60 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr18_4xb4-80k_isaid-896x896.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_4x4_896x896_80k_isaid/fcn_hr18_4x4_896x896_80k_isaid_20220110_182230-49bf752e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr18_4x4_896x896_80k_isaid/fcn_hr18_4x4_896x896_80k_isaid_20220110_182230.log.json)     |

| FCN    | HRNetV2p-W48       | 896x896   |   80000 | 16.89    | 7.34           | V100   | 67.80 |         68.53 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/hrnet/fcn_hr48_4xb4-80k_isaid-896x896.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_4x4_896x896_80k_isaid/fcn_hr48_4x4_896x896_80k_isaid_20220114_174643-547fc420.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/hrnet/fcn_hr48_4x4_896x896_80k_isaid/fcn_hr48_4x4_896x896_80k_isaid_20220114_174643.log.json)     |

Note:

}
```

# Quantize model

## Why quantization ?

The fixed-point model has many advantages over the fp32 model:

- Smaller size, 8-bit model reduces file size by 75%
- Benefit from the smaller model, the Cache hit rate is improved and inference would be faster
- Chips tend to have corresponding fixed-point acceleration instructions which are faster and less energy consumed (int8 on a common CPU requires only about 10% of energy)

APK file size and heat generation are key indicators while evaluating mobile APP;
On server side, quantization means that you can increase model size in exchange for precision and keep the same QPS.

## Post training quantization scheme

Taking ncnn backend as an example, the complete workflow is as follows:

  <img src="../_static/image/quant_model.png"/>
</div>

mmdeploy generates quantization table based on static graph (onnx) and uses backend tools to convert fp32 model to fixed point.

mmdeploy currently supports ncnn with PTQ.

</div>

## How to convert model

[After mmdeploy installation](../01-how-to-build/build_from_source.md), install ppq

```bash
git clone https://github.com/openppl-public/ppq.git
cd ppq
```

Back in mmdeploy, enable quantization with the option 'tools/deploy.py --quant'.

```bash

export MODEL_CONFIG=/home/rg/konghuanjun/mmpretrain/configs/resnet/resnet18_8xb32_in1k.py

# get some imagenet sample images
git clone https://github.com/nihui/imagenet-sample-images --depth=1

# quantize

# quantize
python3 tools/deploy.py  configs/mmpretrain/classification_ncnn-int8_static.py  ${MODEL_CONFIG}  ${MODEL_PATH}   /path/to/self-test.png   --work-dir work_dir --device cpu --quant --quant-image-dir /path/to/imagenet-sample-images
...
```

Description

|     Parameter     |                             Meaning                              |
| :---------------: | :--------------------------------------------------------------: |
|      --quant      |         Enable quantization, the default value is False          |
| --quant-image-dir | Calibrate dataset, use Validation Set in MODEL_CONFIG by default |

## Custom calibration dataset

Calibration set is used to calculate quantization layer parameters. Some DFQ (Data Free Quantization) methods do not even require a dataset.

- Create a folder, just put in some images (no directory structure, no negative example, no special filename format)
- The image needs to be the data comes from real scenario otherwise the accuracy would be drop
- You can not quantize model with test dataset
  | Type  | Train dataset | Validation dataset | Test dataset  | Calibration dataset |
  | ----- | ------------- | ------------------ | ------------- | ------------------- |
  | Usage | QAT           | PTQ                | Test accuracy | PTQ                 |

It is highly recommended that [verifying model precision](profile_model.md) after quantization. [Here](../03-benchmark/quantization.md) is some quantization model test result.

# Pose2seg: Detection free human instance segmentation

<details>

```bibtex
}
```

}
```

<details>

```bibtex
}
```

<details>

```bibtex

```bibtex
}
```

# BoxInst

> [BoxInst: High-Performance Instance Segmentation with Box Annotations](https://arxiv.org/pdf/2012.02310.pdf)

We present a high-performance method that can achieve mask-level instance segmentation with only bounding-box annotations for training. While this setting has been studied in the literature, here we show significantly stronger performance with a simple design (e.g., dramatically improving previous best reported mask AP of 21.1% to 31.6% on the COCO dataset). Our core idea is to redesign the loss

of learning masks in instance segmentation, with no modification to the segmentation network itself. The new loss functions can supervise the mask training without relying on mask annotations. This is made possible with two loss terms, namely, 1) a surrogate term that minimizes the discrepancy between the projections of the ground-truth box and the predicted mask; 2) a pairwise loss that can exploit the prior that proximal pixels with similar colors are very likely to have the same category label. Experiments demonstrate that the redesigned mask loss can yield surprisingly high-quality instance masks with only box annotations. For example, without using any mask annotations, with a ResNet-101 backbone and 3× training schedule, we achieve 33.2% mask AP on COCO test-dev split (vs. 39.1% of the fully supervised counterpart). Our excellent experiment results on COCO and Pascal VOC indicate that our method dramatically narrows the performance gap between weakly and fully supervised instance segmentation.

<img src="https://user-images.githubusercontent.com/57584090/209087723-756b76d7-5061-4000-a93c-df1194a439a0.png"/>
</div>

| Backbone |  Style  | MS train | Lr schd | bbox AP | mask AP |                   Config                    |                                                                                                                                                  Download                                                                                                                                                   |
| :------: | :-----: | :------: | :-----: | :-----: | :-----: | :-----------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

|   R-50   | pytorch |    Y     |   1x    |  39.6   |  31.1   | [config](./boxinst_r50_fpn_ms-90k_coco.py)  |  [model](https://download.openmmlab.com/mmdetection/v3.0/boxinst/boxinst_r50_fpn_ms-90k_coco/boxinst_r50_fpn_ms-90k_coco_20221228_163052-6add751a.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/boxinst/boxinst_r50_fpn_ms-90k_coco/boxinst_r50_fpn_ms-90k_coco_20221228_163052.log.json)   |
|  R-101   | pytorch |    Y     |   1x    |  41.8   |  32.7   | [config](./boxinst_r101_fpn_ms-90k_coco.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/boxinst/boxinst_r101_fpn_ms-90k_coco/boxinst_r101_fpn_ms-90k_coco_20221229_145106-facf375b.pth) \|[log](https://download.openmmlab.com/mmdetection/v3.0/boxinst/boxinst_r101_fpn_ms-90k_coco/boxinst_r101_fpn_ms-90k_coco_20221229_145106.log.json) |

```latex
@inproceedings{tian2020boxinst,
  title     =  {{BoxInst}: High-Performance Instance Segmentation with Box Annotations},

```latex
  author    =  {Tian, Zhi and Shen, Chunhua and Wang, Xinlong and Chen, Hao},
  year      =  {2021}
}
```

# 任务运行和监控

## 评测任务发起

评测任务的程序入口为 `run.py`，使用方法如下：

```shell
```

启动方式：

- 本地机器运行: `run.py $Config`，$Config 中不包含 `eval` 和 `infer` 字段。
- srun运行: `run.py $Config --slurm -p $PARTITION_name`。
- dlc运行： `run.py $Config --dlc --aliyun-cfg $AliYun_Cfg`， 后续会有教程。
- 定制化启动: `run.py $Config` $Config 中包含 `eval` 和 `infer` 字段，参考 [评估文档](./evaluation.md)。

参数解释如下：

- `-p`: 指定 slurm 分区；
- `-q`: 指定 slurm quotatype（默认为 None），可选 reserved, auto, spot。该参数可能仅适用于部分 slurm 的变体；
- `--debug`: 开启时，推理和评测任务会以单进程模式运行，且输出会实时回显，便于调试；
- `-m`: 运行模式，默认为 `all`。可以指定为 `infer` 则仅运行推理，获得输出结果；如果在 `{WORKDIR}` 中已经有模型输出，则指定为 `eval` 仅运行评测，获得评测结果；如果在 `results/` 中已有单项评测结果，则指定为 `viz` 仅运行可视化；指定为 `all` 则同时运行推理和评测。
- `-r`: 重用已有的推理结果。如果后面跟有时间戳，则会复用工作路径下该时间戳的结果；否则则复用指定工作路径下的最新结果。
- `-w`: 指定工作路径，默认为 `./outputs/default`
- `-l`: 打开飞书机器人状态上报。
- `--dry-run`: 开启时，推理和评测任务仅会分发但不会真正运行，便于调试；

以运行模式`-m all`为例，整体运行流如下：

1. 读取配置文件，解析出模型、数据集、评估器等配置信息
2. 评测任务主要分为推理 `infer`、评测 `eval` 和可视化 `viz` 三个阶段，其中推理和评测经过 Partitioner 进行任务切分后，交由 Runner 负责并行执行。单个推理和评测任务则被抽象成 `OpenICLInferTask` 和 `OpenICLEvalTask`。
3. 两阶段分别结束后，可视化阶段会读取 `results/` 中的评测结果，生成可视化报告。

## 任务监控：飞书机器人

配置方式:

1. 打开 `configs/lark.py` 文件，并在文件中加入以下行：

   ```python
   ```

   通常， Webhook URL 格式如 https://open.feishu.cn/open-apis/bot/v2/hook/xxxxxxxxxxxxxxxxx 。

2. 在完整的评测配置中继承该文件：

   ```python

   ```

3. 为了避免机器人频繁发消息形成骚扰，默认运行时状态不会自动上报。有需要时，可以通过 `-l` 或 `--lark` 启动状态上报：

   ```bash

   ```

   ```bash
   ```

## 运行结果

所有运行结果会默认放在`outputs/default/`目录下，目录结构如下所示：

```text
├── ...
│   ├── configs
│   ├── logs
│   │   ├── eval
│   ├── predictions
│   │   └── MODEL1
│   └── results
```

其中，每一个时间戳中存在以下内容：

- configs 文件夹，用于存放以这个时间戳为输出目录的每次运行对应的配置文件；
- logs 文件夹，用于存放推理和评测两个阶段的输出日志文件，各个文件夹内会以模型为子文件夹存放日志；
- predicitions 文件夹，用于存放推理 json 结果，以模型为子文件夹；
- results 文件夹，用于存放评测 json 结果，以模型为子文件夹

另外，所有指定-r 但是没有指定对应时间戳将会按照排序选择最新的文件夹作为输出目录。

## Summerizer介绍 （待更新）

# MILAN

> [MILAN: Masked Image Pretraining on Language Assisted Representation](https://arxiv.org/pdf/2208.06049)

Self-attention based transformer models have been dominating many computer
vision tasks in the past few years. Their superb model qualities heavily depend
on the excessively large labeled image datasets. In order to reduce the reliance
on large labeled datasets, reconstruction based masked autoencoders are gaining
popularity, which learn high quality transferable representations from unlabeled
images. For the same purpose, recent weakly supervised image pretraining methods
explore language supervision from text captions accompanying the images. In this
work, we propose masked image pretraining on language assisted representation,
dubbed as MILAN. Instead of predicting raw pixels or low level features, our
pretraining objective is to reconstruct the image features with substantial semantic

signals that are obtained using caption supervision. Moreover, to accommodate our
reconstruction target, we propose a more efficient prompting decoder architecture
and a semantic aware mask sampling mechanism, which further advance the
transfer performance of the pretrained model. Experimental results demonstrate
that MILAN delivers higher accuracy than the previous works. When the masked
autoencoder is pretrained and finetuned on ImageNet-1K dataset with an input
resolution of 224×224, MILAN achieves a top-1 accuracy of 85.4% on ViTB/16, surpassing previous state-of-the-arts by 1%. In the downstream semantic
segmentation task, MILAN achieves 52.7 mIoU using ViT-B/16 backbone on
ADE20K dataset, outperforming previous masked pretraining results by 4 points.

<img src="https://user-images.githubusercontent.com/30762564/205210369-41a65c4c-bcd4-4147-91ea-c6c9061ab455.png" width="80%"/>
</div>

```python

predict = inference_model('vit-base-p16_milan-pre_8xb128-coslr-100e_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('milan_vit-base-p16_16xb256-amp-coslr-400e_in1k', pretrained=True)
```

```

Train:

```shell
python tools/train.py configs/milan/milan_vit-base-p16_16xb256-amp-coslr-400e_in1k.py
```

Test:

```shell
python tools/test.py configs/milan/benchmarks/vit-base-p16_8xb128-coslr-100e_in1k.py https://download.openmmlab.com/mmselfsup/1.x/milan/milan_vit-base-p16_16xb256-amp-coslr-400e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k-milan_20221129-74ac94fa.pth
```

```

| Model                                            | Params (M) | Flops (G) |                           Config                            |                                  Download                                  |
| :----------------------------------------------- | :--------: | :-------: | :---------------------------------------------------------: | :------------------------------------------------------------------------: |

| `milan_vit-base-p16_16xb256-amp-coslr-400e_in1k` |   111.91   |   17.58   | [config](milan_vit-base-p16_16xb256-amp-coslr-400e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/milan/milan_vit-base-p16_16xb256-amp-coslr-400e_in1k/milan_vit-base-p16_16xb256-amp-coslr-400e_in1k_20221129-180922e8.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/milan/milan_vit-base-p16_16xb256-amp-coslr-400e_in1k/milan_vit-base-p16_16xb256-amp-coslr-400e_in1k_20221129-180922e8.json) |

| `vit-base-p16_milan-pre_8xb128-coslr-100e_in1k` | [MILAN](https://download.openmmlab.com/mmselfsup/1.x/milan/milan_vit-base-p16_16xb256-amp-coslr-400e_in1k/milan_vit-base-p16_16xb256-amp-coslr-400e_in1k_20221129-180922e8.pth) |   86.57    |   17.58   |   85.30   | [config](benchmarks/vit-base-p16_8xb128-coslr-100e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/milan/milan_vit-base-p16_16xb256-amp-coslr-400e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k-milan_20221129-74ac94fa.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/milan/milan_vit-base-p16_16xb256-amp-coslr-400e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k-milan_20221129-74ac94fa.json) |

| `vit-base-p16_milan-pre_8xb2048-linear-coslr-100e_in1k` | [MILAN](https://download.openmmlab.com/mmselfsup/1.x/milan/milan_vit-base-p16_16xb256-amp-coslr-400e_in1k/milan_vit-base-p16_16xb256-amp-coslr-400e_in1k_20221129-180922e8.pth) |   86.57    |   17.58   |   78.90   | [config](benchmarks/vit-base-p16_8xb2048-linear-coslr-100e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/milan/milan_vit-base-p16_16xb256-amp-coslr-400e_in1k/vit-base-p16_linear-8xb2048-coslr-100e_in1k/vit-base-p16_linear-8xb2048-coslr-100e_in1k_20221129-03f26f85.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/milan/milan_vit-base-p16_16xb256-amp-coslr-400e_in1k/vit-base-p16_linear-8xb2048-coslr-100e_in1k/vit-base-p16_linear-8xb2048-coslr-100e_in1k_20221129-03f26f85.json) |

```bibtex
@article{Hou2022MILANMI,
  title={MILAN: Masked Image Pretraining on Language Assisted Representation},
  author={Zejiang Hou and Fei Sun and Yen-Kuang Chen and Yuan Xie and S. Y. Kung},

```bibtex
}
```

We present a new method for efficient high-quality image segmentation of objects and scenes. By analogizing classical computer graphics methods for efficient rendering with over- and undersampling challenges faced in pixel labeling tasks, we develop a unique perspective of image segmentation as a rendering problem. From this vantage, we present the PointRend (Point-based Rendering) neural network module: a module that performs point-based segmentation predictions at adaptively selected locations based on an iterative subdivision algorithm. PointRend can be flexibly applied to both instance and semantic segmentation tasks by building on top of existing state-of-the-art models. While many concrete implementations of the general idea are possible, we show that a simple design already achieves excellent results. Qualitatively, PointRend outputs crisp object boundaries in regions that are over-smoothed by previous methods. Quantitatively, PointRend yields significant gains on COCO and Cityscapes, for both instance and semantic segmentation. PointRend's efficiency enables output resolutions that are otherwise impractical in terms of memory or computation compared to existing approaches.

<img src="https://user-images.githubusercontent.com/40661020/143970097-d38b6801-d3c8-468f-b8b0-639be3689907.png"/>
</div>

| Backbone | Style | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                       Config                       |                                                                                                                                                                      Download                                                                                                                                                                      |

| :------: | :---: | :-----: | :------: | :------------: | :----: | :-----: | :------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| R-50-FPN | caffe |   1x    |   4.6    |                |  38.4  |  36.3   | [config](./point-rend_r50-caffe_fpn_ms-1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/point_rend/point_rend_r50_caffe_fpn_mstrain_1x_coco/point_rend_r50_caffe_fpn_mstrain_1x_coco-1bcb5fb4.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/point_rend/point_rend_r50_caffe_fpn_mstrain_1x_coco/point_rend_r50_caffe_fpn_mstrain_1x_coco_20200612_161407.log.json) |

| R-50-FPN | caffe |   3x    |   4.6    |                |  41.0  |  38.0   | [config](./point-rend_r50-caffe_fpn_ms-3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/point_rend/point_rend_r50_caffe_fpn_mstrain_3x_coco/point_rend_r50_caffe_fpn_mstrain_3x_coco-e0ebb6b7.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/point_rend/point_rend_r50_caffe_fpn_mstrain_3x_coco/point_rend_r50_caffe_fpn_mstrain_3x_coco_20200614_002632.log.json) |

Note: All models are trained with multi-scale, the input image shorter side is randomly scaled to one of (640, 672, 704, 736, 768, 800).

```latex
@InProceedings{kirillov2019pointrend,
  title={{PointRend}: Image Segmentation as Rendering},
  author={Alexander Kirillov and Yuxin Wu and Kaiming He and Ross Girshick},
  journal={ArXiv:1912.08193},
}
```

# ResNet for Audio

[Audiovisual SlowFast Networks for Video Recognition](https://arxiv.org/abs/2001.08740)

We present Audiovisual SlowFast Networks, an architecture for integrated audiovisual perception. AVSlowFast has Slow and Fast visual pathways that are deeply inte- grated with a Faster Audio pathway to model vision and sound in a unified representation. We fuse audio and vi- sual features at multiple layers, enabling audio to con- tribute to the formation of hierarchical audiovisual con- cepts. To overcome training difficulties that arise from dif- ferent learning dynamics for audio and visual modalities, we introduce DropPathway, which randomly drops the Au- dio pathway during training as an effective regularization technique. Inspired by prior studies in neuroscience, we perform hierarchical audiovisual synchronization to learn joint audiovisual features. We report state-of-the-art results on six video action classification and detection datasets, perform detailed ablation studies, and show the gener- alization of AVSlowFast to learn self-supervised audiovi- sual features. Code will be made available at: https: //github.com/facebookresearch/SlowFast.

<img src="https://user-images.githubusercontent.com/30782254/147050415-a30ad32a-ce52-452d-ac3d-91058c8d0cc9.png" width="800"/>
</div>

| frame sampling strategy | n_fft | gpus | backbone | pretrain | top1 acc | top5 acc | testing protocol | gpu_mem(M) |                 config                 |                 ckpt                 |                 log                  |
| :---------------------: | :---: | :--: | :------: | :------: | :------: | :------: | :--------------: | :--------: | :------------------------------------: | :----------------------------------: | :----------------------------------: |

|         64x1x1          | 1024  |  8   | Resnet18 |   None   |   19.7   |  35.75   |     10 clips     |    1897    | [config](/configs/recognition_audio/resnet/tsn_r18_8xb320-64x1x1-100e_kinetics400-audio-feature.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition_audio/resnet/tsn_r18_8xb320-64x1x1-100e_kinetics400-audio-feature/tsn_r18_8xb320-64x1x1-100e_kinetics400-audio-feature_20201012-bf34df6c.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition_audio/resnet/tsn_r18_8xb320-64x1x1-100e_kinetics400-audio-feature/tsn_r18_8xb320-64x1x1-100e_kinetics400-audio-feature.log) |

1. The **gpus** indicates the number of gpus we used to get the checkpoint. It is noteworthy that the configs we provide are used for 8 gpus as default.

For more details on data preparation, you can refer to `Prepare audio` in [Data Preparation Tutorial](/docs/en/user_guides/prepare_dataset.md).

## Train

## Train

```shell
```

Example: train ResNet model on Kinetics-400 audio dataset in a deterministic option with periodic validation.

```shell
python tools/train.py configs/recognition_audio/resnet/tsn_r18_8xb320-64x1x1-100e_kinetics400-audio-feature.py \
```

## Test

```shell
```

Example: test ResNet model on Kinetics-400 audio dataset and dump the result to a pkl file.

```shell

```shell
```

```shell
python tools/test.py configs/recognition_audio/resnet/tsn_r18_8xb320-64x1x1-100e_kinetics400-audio-feature.py \
```

```BibTeX
@article{xiao2020audiovisual,
  title={Audiovisual SlowFast Networks for Video Recognition},
  author={Xiao, Fanyi and Lee, Yong Jae and Grauman, Kristen and Malik, Jitendra and Feichtenhofer, Christoph},
  journal={arXiv preprint arXiv:2001.08740},
}
```

}
```

|                  Config                   | Input Size | PCK@0.2<sup><br>(COCO-Wholebody-Hand) | PCK@0.2<sup><br>(Hand5) | AUC<sup><br>(Hand5) | EPE<sup><br>(Hand5) | FLOPS(G) |                  Download                   |

| :---------------------------------------: | :--------: | :-----------------------------------: | :---------------------: | :-----------------: | :-----------------: | :------: | :-----------------------------------------: |
| [RTMPose-m\*<sup><br>(alpha version)](./rtmpose/hand_2d_keypoint/rtmpose-m_8xb32-210e_coco-wholebody-hand-256x256.py) |  256x256   |                 81.5                  |          96.4           |        83.9         |        5.06         |  2.581   | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-hand5_pt-aic-coco_210e-256x256-74fb594_20230320.pth) |

# Migration of Distributed Training Settings

We have merged [MMGeneration 1.x](https://github.com/open-mmlab/mmgeneration/tree/1.x) into MMagic. Here is migration of Distributed Training Settings about MMGeneration.

In 0.x version, MMGeneration uses `DDPWrapper` and `DynamicRunner` to train static and dynamic model (e.g., PGGAN and StyleGANv2) respectively. In 1.x version, we use `MMSeparateDistributedDataParallel` provided by MMEngine to implement distributed training.

The configuration differences are shown below:

<thead>
  <tr>
<tbody>
<tr>

```python
# Use DDPWrapper

```

</td>

```python

is_dynamic_ddp=False)
```

</td>

```python
```

</td>

</tr>
</thead>
</table>

<thead>
  <tr>
    <th> Dynamic Model in 0.x Version </th>
    <th> Dynamic Model in 1.x Version </th>
<tbody>
<tr>

```python

# Use DynamicRunner
```

</td>

```python
    find_unused_parameters=True) # set `find_unused_parameters` for dynamic models
```

</td>

</tr>
</thead>
</table>

<details>
<summary align="right">IconVSR (CVPR'2021)</summary>

```bibtex
}
```

<br/>

对于 REDS4，我们对 RGB 通道进行评估。对于其他数据集，我们对 Y 通道进行评估。我们使用 `PSNR` 和 `SSIM` 作为指标。
IconVSR 组件的预训练权重可以在这里找到：[SPyNet](https://download.openmmlab.com/mmediting/restorers/basicvsr/spynet_20210409-c6c1bd09.pth)，[用于 REDS 的 EDVR-M](https://download.openmmlab.com/mmediting/restorers/iconvsr/edvrm_reds_20210413-3867262f.pth)，以及 [用于 Vimeo-90K 的 EDVR-M](https://download.openmmlab.com/mmediting/restorers/iconvsr/edvrm_vimeo90k_20210413-e40e99a8.pth)。

| 算法 | REDS4 (BIx4)<br>PSNR/SSIM (RGB) | Vimeo-90K-T (BIx4)<br>PSNR/SSIM (Y) | Vid4 (BIx4)<br>PSNR/SSIM (Y) | UDM10 (BDx4)<br>PSNR/SSIM (Y) | Vimeo-90K-T (BDx4)<br>PSNR/SSIM (Y) | Vid4 (BDx4)<br>PSNR/SSIM (Y) | GPU 信息 | 下载 |
| :-: | :-----------------------------: | :---------------------------------: | :--------------------------: | :---------------------------: | :---------------------------------: | :--------------------------: | :-----: | :--: |
| [iconvsr_reds4](./iconvsr_2xb4_reds4.py) |       **31.6926/0.8951**        |           36.4983/0.9416            |      **27.4809/0.8354**      |        35.3377/0.9471         |           34.4299/0.9287            |        25.2110/0.7732        | 2 (Tesla V100-PCIE-32GB) | [模型](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_reds4_20210413-9e09d621.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_reds4_20210413_222735.log.json) |

| [iconvsr_vimeo90k_bi](./iconvsr_2xb4_vimeo90k-bi.py) |         30.3452/0.8659          |         **37.3729/0.9467**          |        27.4238/0.8297        |        34.2595/0.9398         |           34.5548/0.9295            |        24.6666/0.7491        | 2 (Tesla V100-PCIE-32GB) | [模型](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bi_20210413-7c7418dc.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bi_20210413_222757.log.json) |

| [iconvsr_vimeo90k_bd](./iconvsr_2xb4_vimeo90k-bd.py) |         29.0150/0.8465          |           34.6780/0.9339            |        26.3109/0.8028        |      **40.0640/0.9697**       |         **37.7573/0.9517**          |      **28.2464/0.8612**      | 2 (Tesla V100-PCIE-32GB) | [模型](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bd_20210414-5f38cb34.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/iconvsr/iconvsr_vimeo90k_bd_20210414_084128.log.json) |

## 快速开始

**训练**

<details>

```shell
# CPU上训练

# 单个GPU上训练

# 多个GPU上训练
```

**测试**

<details>

```

**测试**

<details>

```shell
# CPU上测试

# 单个GPU上测试

# 多个GPU上测试
```

# Migration from MMSegmentation 0.x

This guide describes the fundamental differences between MMSegmentation 0.x and MMSegmentation 1.x in terms of behaviors and the APIs, and how these all relate to your migration journey.

MMSegmentation 1.x depends on some new packages, you can prepare a new clean environment and install again according to the [installation tutorial](../get_started.md).

Or install the below packages manually.

1. [MMEngine](https://github.com/open-mmlab/mmengine): MMEngine is the core the OpenMMLab 2.0 architecture, and we splited many compentents unrelated to computer vision from MMCV to MMEngine.

2. [MMCV](https://github.com/open-mmlab/mmcv): The computer vision package of OpenMMLab. This is not a new dependency, but you need to upgrade it to **2.0.0** version or above.

3. [MMClassification](https://github.com/open-mmlab/mmclassification)(Optional): The image classification toolbox and benchmark of OpenMMLab. This is not a new dependency, but you need to upgrade it to **1.0.0rc6** version.

4. [MMDetection](https://github.com/open-mmlab/mmdetection)(Optional): The object detection toolbox and benchmark of OpenMMLab. This is not a new dependency, but you need to upgrade it to **3.0.0** version or above.

## Train launch

The main improvement of OpenMMLab 2.0 is releasing MMEngine which provides universal and powerful runner for unified interfaces to launch training jobs.

Compared with MMSeg0.x, MMSeg1.x provides fewer command line arguments in `tools/train.py`

<tr>
<td>Function</td>
</tr>
<tr>

<tr>
</tr>
<tr>
<td>Loading pre-trained checkpoint</td>
</tr>
<tr>
<td>Resuming Train from specific checkpoint</td>
</tr>
<tr>
<td>Resuming Train from the latest checkpoint</td>
</tr>
<tr>
<td>Whether not to evaluate the checkpoint during training</td>
</tr>
<tr>
<td>Training device assignment</td>
</tr>
<tr>
<td>Whether or not set different seeds for different ranks</td>
</tr>

</tr>
<tr>
</tr>
<td>Whether to set deterministic options for CUDNN backend</td>
</table>

## Test launch

Similar to training launch, there are only common arguments in tools/test.py of MMSegmentation 1.x.
Below is the difference in test scripts,
please refer to [this documentation](../user_guides/4_train_test.md) for more details about test launch.

<tr>
</tr>
<tr>
<td>Evaluation metrics</td>
</tr>
<tr>
<td>Whether to use test time augmentation</td>
</tr>
<tr>
<td>Whether save the output results without perform evaluation</td>

</tr>
<tr>
</tr>
<tr>
</tr>
</table>

## Configuration file

No changes in `model.backbone`, `model.neck`, `model.decode_head` and `model.losses` fields.

Add `model.data_preprocessor` field to configure the `DataPreProcessor`, including:

- `mean` (Sequence, optional): The pixel mean of R, G, B channels. Defaults to None.

- `std` (Sequence, optional): The pixel standard deviation of R, G, B channels. Defaults to None.

- `size` (Sequence, optional): Fixed padding size.

- `size_divisor` (int, optional): The divisor of padded size.

- `seg_pad_val` (float, optional): Padding value of segmentation map. Default: 255.

- `padding_mode` (str): Type of padding. Default: 'constant'.

  - constant: pads with a constant value, this value is specified with pad_val.

- `bgr_to_rgb` (bool): whether to convert image from BGR to RGB.Defaults to False.

- `rgb_to_bgr` (bool): whether to convert image from RGB to BGR. Defaults to False.

**Note:**
Please refer [models documentation](../advanced_guides/models.md) for more details.

Changes in **data**:

The original `data` field is split to `train_dataloader`, `val_dataloader` and `test_dataloader`. This allows us to configure them in fine-grained. For example, you can specify different sampler and batch size during training and test.
The `samples_per_gpu` is renamed to `batch_size`.
The `workers_per_gpu` is renamed to `num_workers`.

<tr>
<td>

```python

<tr>
<td>

```python
)
```

</td>
<tr>
<td>

```python
)

)

```

</td>
</tr>
</table>

Changes in **pipeline**

- The original formatting transforms **`ToTensor`**、**`ImageToTensor`**、**`Collect`** are combined as [`PackSegInputs`](mmseg.datasets.transforms.PackSegInputs)
- We don't recommend to do **`Normalize`** and **Pad** in the dataset pipeline. Please remove it from pipelines and set it in the `data_preprocessor` field.

- The original **`Resize`** in MMSeg 1.x has been changed to **`RandomResize`** and the input arguments `img_scale` is renamed to `scale`, and the default value of `keep_ratio` is modified to False.
- The original `test_pipeline` combines single-scale test and multi-scale test together, in MMSeg 1.x we separate it into `test_pipeline` and `tta_pipeline`.

**Note:**
We move some work of data transforms to the data preprocessor, like normalization, see [the documentation](package.md) for more details.

train_pipeline

<tr>
<td>

```python

]
```

</td>
<tr>
<td>

```python
    dict(
]
```

</td>
</tr>
</table>

test_pipeline

]
```

</td>
</tr>
</table>

<tr>
<td>

```python
    dict(
        ])
]
```

</td>
<tr>
<td>

```python
]

dict(type='Resize', scale=(2560, 640), keep_ratio=True),
]
    dict(
            [
            ],
            [
        ])
]
```

</td>
</tr>
</table>

- The **`evaluation`** field is split to `val_evaluator` and `test_evaluator`. And it won't support `interval` and `save_best` arguments.

]
```

</td>
</tr>
</table>

  The `interval` is moved to `train_cfg.val_interval`, and the `save_best` is moved to `default_hooks.checkpoint.save_best`. `pre_eval` has been removed.
- `'mIoU'` has been changed to `'IoUMetric'`.

<tr>
<td>

```python
```

</td>
<tr>
<td>

```python
```

</td>
</tr>
</table>

### Optimizer and Schedule settings

- `paramwise_cfg` is also a sub field of `optim_wrapper`, instead of `optimizer`.

<tr>
<td>

```python
```

</td>
<tr>
<td>

```python
```

</td>
</tr>
</table>

clip_grad=dict(max_norm=1, norm_type=2))
```

</td>
</tr>
</table>

- The `warmup` related arguments are removed, since we use schedulers combination to implement this functionality.

The new schedulers combination mechanism is very flexible, and you can use it to design many kinds of learning rate / momentum curves. See [the tutorial](TODO) for more details.

<tr>
<td>

```python
```

</td>
<tr>
<td>

```python
    dict(
    dict(
    )
]

dict(
    )
]
```

</td>
</tr>
</table>

Most configuration in the original `runner` field is moved to `train_cfg`, `val_cfg` and `test_cfg`, which configure the loop in training, validation and test.

<tr>
<td>

```python
```

</td>
<tr>
<td>

```python
val_cfg = dict(type='ValLoop') # Use the default validation loop.
test_cfg = dict(type='TestLoop') # Use the default test loop.
```

</td>
</tr>
</table>

```

</td>
</tr>
</table>

In fact, in OpenMMLab 2.0, we introduced `Loop` to control the behaviors in training, validation and test. The functionalities of `Runner` are also changed. You can find more details of [runner tutorial](https://github.com/open-mmlab/mmengine/blob/main/docs/en/design/runner.md) in [MMEngine](https://github.com/open-mmlab/mmengine/).

```python

    # print log every 50 iterations.

```python

    # save checkpoint every 2000 iterations.

    # validation results visualization.
```

In addition, we split the original logger to logger and visualizer. The logger is used to record information and the visualizer is used to show the logger in different backends, like terminal and TensorBoard.

<tr>
<td>

```python
    hooks=[

<tr>
<td>

```python
    hooks=[
    ])
```

</td>
<tr>
<td>

```python
    ...
)
```

</td>
</tr>
</table>

  - If `resume=True` and `load_from` is **not None**, resume training from the checkpoint in `load_from`.
  - If `resume=True` and `load_from` is **None**, try to resume from the latest checkpoint in the work directory.

- If `resume=True` and `load_from` is **None**, try to resume from the latest checkpoint in the work directory.
  - If `resume=False` and `load_from` is **not None**, only load the checkpoint, not resume training.
  - If `resume=False` and `load_from` is **None**, do not load nor resume.

Changes in **`dist_params`**: The `dist_params` field is a sub field of `env_cfg` now. And there are some new configurations in the `env_cfg`.

```python

)
```

# set distributed parameters
)
```

New field **`visualizer`**: The visualizer is a new design in OpenMMLab 2.0 architecture. We use a visualizer instance in the runner to handle results & log visualization and save to different backends. See the [visualization tutorial](../user_guides/visualization.md) for more details.

New field **`default_scope`**: The start point to search module for all registries. The `default_scope` in MMSegmentation is `mmseg`. See [the registry tutorial](https://github.com/open-mmlab/mmengine/blob/main/docs/en/advanced_tutorials/registry.md) for more details.

# Preparing Classic5 Dataset

```bibtex
}
```

```text
mmagic
├── mmagic
├── tools
├── data
```

## 使用方法

首先，请参考[链接](https://mmcv.readthedocs.io/zh_CN/latest/get_started/build.html#npu-mmcv-full)安装带有 NPU 支持的 MMCV 和[链接](https://mmengine.readthedocs.io/en/latest/get_started/installation.html#build-from-source)安装 MMEngine。

使用如下命令，可以利用 8 个 NPU 在机器上训练模型（以 ResNet 为例）：

```shell
bash tools/dist_train.sh configs/cspnet/resnet50_8xb32_in1k.py 8
```

或者，使用如下命令，在一个 NPU 上训练模型（以 ResNet 为例）：

```shell
python tools/train.py configs/cspnet/resnet50_8xb32_in1k.py
```

## 经过验证的模型

| [ResNet-50](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/resnet/README.md) |   76.40   |   93.21   | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/resnet/resnet50_8xb32_in1k.py) | [model](<>) \| [log](https://download.openmmlab.com/mmclassification/v1/device/npu/resnet50_8xb32_in1k.log) |
| [ResNetXt-32x4d-50](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/resnext/README.md) |   77.48   |   93.75   | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/resnext/resnext50-32x4d_8xb32_in1k.py) | [model](<>) \| [log](https://download.openmmlab.com/mmclassification/v1/device/npu/resnext50-32x4d_8xb32_in1k.log) |

| [HRNet-W18](https://github.com/open-mmlab/mmclassification/blob/master/configs/hrnet/README.md) |   77.06   |   93.57   | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/hrnet/hrnet-w18_4xb32_in1k.py) | [model](<>) \| [log](https://download.openmmlab.com/mmclassification/v1/device/npu/hrnet-w18_4xb32_in1k.log) |
| [ResNetV1D-152](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/resnet/README.md) |   79.41   |   94.48   | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/resnet/resnetv1d152_8xb32_in1k.py) | [model](<>) \| [log](https://download.openmmlab.com/mmclassification/v1/device/npu/resnetv1d152_8xb32_in1k.log) |

| [SE-ResNet-50](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/seresnet/README.md) |   77.65   |   93.74   | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/seresnet/seresnet50_8xb32_in1k.py) | [model](<>) \|[log](https://download.openmmlab.com/mmclassification/v1/device/npu/seresnet50_8xb32_in1k.log) |
| [ShuffleNetV2 1.0x](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/shufflenet_v2/README.md) |   69.52   |   88.79   | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/shufflenet_v2/shufflenet-v2-1x_16xb64_in1k.py) | [model](<>) \| [log](https://download.openmmlab.com/mmclassification/v1/device/npu/shufflenet-v2-1x_16xb64_in1k.log) |

| [MobileNetV2](https://github.com/open-mmlab/mmclassification/tree/1.x/configs/mobilenet_v2) |   71.74   |   90.28   | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/mobilenet_v2/mobilenet-v2_8xb32_in1k.py) | [model](<>) \| [log](https://download.openmmlab.com/mmclassification/v1/device/npu/mobilenet-v2_8xb32_in1k.log) |
| [MobileNetV3-Small](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/mobilenet_v3/README.md) |   67.09   |   87.17   | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/mobilenet_v3/mobilenet-v3-small_8xb128_in1k.py) | [model](<>) \| [log](https://download.openmmlab.com/mmclassification/v1/device/npu/mobilenet-v3-small.log) |

| [\*CSPResNeXt50](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/cspnet/README.md) |   77.25   |   93.46   | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/cspnet/cspresnext50_8xb32_in1k.py) | [model](<>) \| [log](https://download.openmmlab.com/mmclassification/v1/device/npu/cspresnext50_8xb32_in1k.log) |
| [\*EfficientNet-B4](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/efficientnet/README.md) |   75.73   |  92.9100  | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/efficientnet/efficientnet-b4_8xb32_in1k.py) | [model](<>) \|[log](https://download.openmmlab.com/mmclassification/v1/device/npu/efficientnet-b4_8xb32_in1k.log) |

| [\*\*DenseNet121](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/densenet/README.md) |   72.53   |   90.85   | [config](https://github.com/open-mmlab/mmclassification/blob/1.x/configs/densenet/densenet121_4xb256_in1k.py) | [model](<>) \| [log](https://download.openmmlab.com/mmclassification/v1/device/npu/densenet121_4xb256_in1k.log) |

**注意:**

- 如果没有特别标记，NPU 上的结果与使用 FP32 的 GPU 上的结果结果相同。
- (\*) 这些模型的训练结果低于相应模型中自述文件上的结果，主要是因为自述文件上的结果直接是 timm 训练得出的权重，而这边的结果是根据 mmcls 的配置重新训练得到的结果。GPU 上的配置训练结果与 NPU 的结果相同。
- (\*\*)这个模型的精度略低，因为 config 是 4 张卡的配置，我们使用 8 张卡来运行，用户可以调整超参数以获得最佳精度结果。

# Migration of Visualization

In 0.x, MMEditing use `VisualizationHook` to visualize results in training process. In 1.x version, we unify the function of those hooks into `BasicVisualizationHook` / `VisualizationHook`. Additionally, follow the design of MMEngine, we implement `ConcatImageVisualizer` / `Visualizer` and a group of `VisBackend` to draw and save the visualization results.

<thead>
  <tr>
    <th> 0.x version </th>
<tbody>
<tr>

```python
)
```

</td>

```python

```

</td>

</tr>
</thead>
</table>

To learn more about the visualization function, please refers to [this tutorial](../user_guides/visualization.md).

# Style-Based Global Appearance Flow for Virtual Try-On (CVPR 2022)

Awesome try-on desplays are like this:

![image1](examples/000010_0.png)

```
Author: @FerryHuang.

This is an implementation of https://github.com/SenHe/Flow-Style-VTON adapting to mmediting. Only inference is supported so far.
```

## Usage

Please refer to [Get Started](https://mmediting.readthedocs.io/en/latest/get_started/I.html) to install
MMEditing.

```shell
```

Please check the [official repo](https://github.com/SenHe/Flow-Style-VTON) and download test-set and pretrained checkpoints and put them under the folder projects/flow_style_vton

```bash
cd projects/flow_style_vton

```bash
python inference.py
```

Expectedly, two folders will be made im_gar_flow_wg and our_t_results, containing the
try-on procedures and the final results, respectively.

```bibtex
@inproceedings{he2022fs_vton,
  title={Style-Based Global Appearance Flow for Virtual Try-On},
  author={He, Sen and Song, Yi-Zhe and Xiang, Tao},
}
```

to MMediting projects.

    <!-- The code's design shall follow existing interfaces and convention. For example, each model component should be registered into `mmedit.registry.MODELS` and configurable via a config file. -->

    <!-- Unit tests for the major module are required. [Example](https://github.com/open-mmlab/mmediting/blob/main/tests/test_models/test_backbones/test_vision_transformer.py) -->

    <!-- It will used for mmediting to acquire your models. [Example](https://github.com/open-mmlab/mmediting/blob/main/configs/mvit/metafile.yml). In particular, you may have to refactor this README into a standard one. [Example](https://github.com/open-mmlab/mmediting/blob/main/configs/swin_transformer/README.md) -->

# 教程 7：实用工具（待更新）

我们在 `tools/` 目录下提供了很多有用的工具。

### 获取 FLOP 和参数量（实验性）

我们提供了一个改编自 [flops-counter.pytorch](https://github.com/sovrasov/flops-counter.pytorch) 的脚本来计算模型的 FLOP 和参数量。

```shell
```

例如，

```shell
python tools/get_flops.py configs/resotorer/srresnet.py --shape 40 40
```

你会得到以下的结果。

```
```

**注**：此工具仍处于实验阶段，我们不保证数字正确。 您可以将结果用于简单的比较，但在技术报告或论文中采用它之前，请仔细检查它。

(1) FLOPs 与输入形状有关，而参数量与输入形状无关。默认输入形状为 (1, 3, 250, 250)。
(2) 一些运算符不计入 FLOP，如 GN 和自定义运算符。
你可以通过修改 [`mmcv/cnn/utils/flops_counter.py`](https://github.com/open-mmlab/mmcv/blob/master/mmcv/cnn/utils/flops_counter.py) 来添加对新运算符的支持。

### 发布模型

在将模型上传到 AWS 之前，您可能需要
(1) 将模型权重转换为 CPU tensors, (2) 删除优化器状态，和
(3) 计算模型权重文件的哈希并将哈希 ID 附加到文件名。

```shell
```

例如，

```shell

### 发布模型

```shell
```

例如，

```shell
python tools/publish_model.py work_dirs/example_exp/latest.pth example_model_20200202.pth
```

最终输出文件名将是 `example_model_20200202-{hash id}.pth`.

### 转换为 ONNX（实验性）

我们提供了一个脚本将模型转换为 [ONNX](https://github.com/onnx/onnx) 格式。 转换后的模型可以通过 [Netron](https://github.com/lutzroeder/netron) 等工具进行可视化。此外，我们还支持比较 Pytorch 和 ONNX 模型之间的输出结果。

```bash
python tools/pytorch2onnx.py
    ${CHECKPOINT_PATH} \
    ${MODEL_TYPE} \
    ${IMAGE_PATH} \
    --trimap-path ${TRIMAP_PATH} \
    --output-file ${OUTPUT_ONNX} \
    --show \
    --verify \
    --dynamic-export
```

参数说明：

- `checkpoint` : 模型模型权重文件的路径。
- `model_type` : 配置文件的模型类型，选项： `inpainting`, `mattor`, `restorer`, `synthesizer`。
- `image_path` : 输入图像文件的路径。
- `--trimap-path` : 输入三元图文件的路径，用于 mattor 模型。

- `--output-file`: 输出 ONNX 模型的路径。默认为 `tmp.onnx`。
- `--opset-version` : ONNX opset 版本。默认为 11。
- `--show`: 确定是否打印导出模型的架构。默认为 `False`。
- `--verify`: 确定是否验证导出模型的正确性。默认为 `False`。
- `--dynamic-export`: 确定是否导出具有动态输入和输出形状的 ONNX 模型。默认为 `False`。

**注**：此工具仍处于试验阶段。目前不支持某些自定义运算符。我们现在只支持 `mattor` 和 `restorer`。

#### 支持导出到 ONNX 的模型列表

下表列出了保证可导出到 ONNX 并可在 ONNX Runtime 中运行的模型。

|   模型   |                                                                               配置                                                                               | 动态形状 | 批量推理 | 备注 |
| :------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------: | :------: | :------: | :--: |

|  ESRGAN  |       [esrgan_x4c64b23g32_g1_400k_div2k.py](https://github.com/open-mmlab/mmagic/blob/master/configs/restorers/esrgan/esrgan_x4c64b23g32_g1_400k_div2k.py)       |    Y     |    Y     |      |
|  ESRGAN  | [esrgan_psnr_x4c64b23g32_g1_1000k_div2k.py](https://github.com/open-mmlab/mmagic/blob/master/configs/restorers/esrgan/esrgan_psnr_x4c64b23g32_g1_1000k_div2k.py) |    Y     |    Y     |      |
|  SRCNN   |            [srcnn_x4k915_g1_1000k_div2k.py](https://github.com/open-mmlab/mmagic/blob/master/configs/restorers/srcnn/srcnn_x4k915_g1_1000k_div2k.py)             |    Y     |    Y     |      |
|   DIM    |          [dim_stage3_v16_pln_1x1_1000k_comp1k.py](https://github.com/open-mmlab/mmagic/blob/master/configs/dim/dim_stage3_v16_pln_1x1_1000k_comp1k.py)           |    Y     |    Y     |      |

|   GCA    |                     [gca_r34_4x10_200k_comp1k.py](https://github.com/open-mmlab/mmagic/blob/master/configs/gca/gca_r34_4x10_200k_comp1k.py)                      |    N     |    Y     |      |
| IndexNet |             [indexnet_mobv2_1x16_78k_comp1k.py](https://github.com/open-mmlab/mmagic/blob/master/configs/indexnet/indexnet_mobv2_1x16_78k_comp1k.py)             |    Y     |    Y     |      |

**注**：

- *以上所有模型均使用 Pytorch==1.6.0 和 onnxruntime==1.5.1*
- 如果您遇到上面列出的模型的任何问题，请创建一个 issue，我们会尽快处理。对于列表中未包含的型号，请尝试自行解决。
- 由于此功能是实验性的并且可能会快速更改，请始终尝试使用最新的 `mmcv` 和 `mmagic`。

### 将 ONNX 转换为 TensorRT（实验性）

我们还提供了将 [ONNX](https://github.com/onnx/onnx) 模型转换为 [TensorRT](https://github.com/NVIDIA/TensorRT) 格式的脚本。 此外，我们支持比较 ONNX 和 TensorRT 模型之间的输出结果。

```bash
python tools/onnx2tensorrt.py
    ${INPUT_ONNX} \
    --trt-file ${OUT_TENSORRT} \
    --max-shape INT INT INT INT \
    --min-shape INT INT INT INT \
    --workspace-size INT \

${MODEL_TYPE} \
    --fp16 \
    --show \
    --verify \
    --verbose
```

参数说明：

- `model_type` :配置文件的模型类型，选项： `inpainting`, `mattor`, `restorer`, `synthesizer`。
- `img_path` : 输入图像文件的路径。
- `onnx_file` : 输入 ONNX 文件的路径。
- `--trt-file` : 输出 TensorRT 模型的路径。默认为 `tmp.trt`。
- `--max-shape` : 模型输入的最大形状。
- `--min-shape` : 模型输入的最小形状。
- `--workspace-size`: 以 GiB 为单位的最大工作空间大小。默认为 1 GiB。
- `--fp16`: 确定是否以 fp16 模式导出 TensorRT。默认为 `False`。
- `--show`: 确定是否显示 ONNX 和 TensorRT 的输出。默认为 `False`。
- `--verbose`: 确定在创建 TensorRT 引擎时是否详细记录日志消息。默认为 `False`。

**注**：此工具仍处于试验阶段。 目前不支持某些自定义运算符。 我们现在只支持 `restorer`。 在生成 SRCNN 的 ONNX 文件时，将 SCRNN 模型中的 'bicubic' 替换为 'bilinear' \[此处\](https://github.com/open-mmlab/mmagic/blob/764e6065e315b7d0033762038fcbf0bb1c570d4d/mmagic.bones/modelsrnn py#L40）。 因为 TensorRT 目前不支持 bicubic 插值，最终性能将下降约 4%。

#### 支持导出到 TensorRT 的模型列表

下表列出了保证可导出到 TensorRT 引擎并可在 TensorRT 中运行的模型。

|  模型  |                                                                     配置                                                                      | 动态形状 | 批量推理 |                 备注                  |
| :----: | :-------------------------------------------------------------------------------------------------------------------------------------------: | :------: | :------: | :-----------------------------------: |

| ESRGAN | [esrgan_x4c64b23g32_g1_400k_div2k.py](https://github.com/open-mmlab/mmagic/blob/master/configs/restorers/esrgan/esrgan_x4c64b23g32_g1_400k_div2k.py) |    Y     |    Y     |                                       |
| ESRGAN | [esrgan_psnr_x4c64b23g32_g1_1000k_div2k.py](https://github.com/open-mmlab/mmagic/blob/master/configs/restorers/esrgan/esrgan_psnr_x4c64b23g32_g1_1000k_div2k.py) |    Y     |    Y     |                                       |
| SRCNN  |   [srcnn_x4k915_g1_1000k_div2k.py](https://github.com/open-mmlab/mmagic/blob/master/configs/restorers/srcnn/srcnn_x4k915_g1_1000k_div2k.py)   |    Y     |    Y     | 'bicubic' 上采样必须替换为 'bilinear' |

**注**：

- *以上所有模型均使用 Pytorch==1.8.1、onnxruntime==1.7.0 和 tensorrt==7.2.3.4 进行测试*
- 如果您遇到上面列出的模型的任何问题，请创建一个问题，我们会尽快处理。 对于列表中未包含的型号，请尝试自行解决。
- 由于此功能是实验性的并且可能会快速更改，因此请始终尝试使用最新的 `mmcv` 和 `mmagic`。

### 评估 ONNX 和 TensorRT 模型（实验性）

我们在 `tools/deploy_test.py` 中提供了评估 TensorRT 和 ONNX 模型的方法。

#### 先决条件

#### 先决条件

要评估 ONNX 和 TensorRT 模型，应先安装 onnx、onnxruntime 和 TensorRT。遵循 [mmcv 中的 ONNXRuntime](https://mmcv.readthedocs.io/en/latest/onnxruntime_op.html) 和 \[mmcv 中的 TensorRT 插件\](https://github.com/open-mmlab/mmcv/blob/master/docs/tensorrt_plugin.md%EF%BC%89%E4%BD%BF%E7%94%A8 ONNXRuntime 自定义操作和 TensorRT 插件安装 `mmcv-full`。

#### 用法

```bash
    ${MODEL_PATH} \
    --save-path ${SAVE_PATH} \
    ----cfg-options ${CFG_OPTIONS} \
```

#### 参数说明：

- `config`: 模型配置文件的路径。
- `model`: TensorRT 或 ONNX 模型文件的路径。
- `backend`: 用于测试的后端，选择 tensorrt 或 onnxruntime。
- `--out`: pickle 格式的输出结果文件的路径。
- `--save-path`: 存储图像的路径，如果没有给出，则不会保存图像。
- `--cfg-options`: 覆盖使用的配置文件中的一些设置，`xxx=yyy` 格式的键值对将被合并到配置文件中。

#### 结果和模型

<table border="1" class="docutils">
	<tr>

#### 结果和模型

	<tr>
	    <th align="center">Model</th>
	    <th align="center">Config</th>
	    <th align="center">Dataset</th>
	    <th align="center">Metric</th>
	    <th align="center">PyTorch</th>
	    <th align="center">ONNX Runtime</th>
      <th align="center">TensorRT FP32</th>
      <th align="center">TensorRT FP16</th>
	</tr>
    <tr>
	    <td align="center" rowspan="6">ESRGAN</td>
	    <td align="center" rowspan="6">
            <code>esrgan_x4c64b23g32_g1_400k_div2k.py</code>
        </td>
	    <td align="center" rowspan="2">Set5</td>
        <td align="center">PSNR</td>
        <td align="center">28.2700</td>
        <td align="center">28.2619</td>
        <td align="center">28.2616</td>
    </tr>
    <tr>
        <td align="center">SSIM</td>

<td align="center">28.2619</td>
    </tr>
    <tr>
        <td align="center">0.7778</td>
        <td align="center">0.7784</td>
        <td align="center">0.7783</td>
    </tr>
    <tr>
        <td align="center" rowspan="2">Set14</td>
        <td align="center">24.6328</td>
        <td align="center">24.6290</td>
        <td align="center">24.6274</td>
    </tr>
    <tr>
        <td align="center">0.6491</td>
        <td align="center">0.6494</td>
    </tr>
    <tr>
        <td align="center" rowspan="2">DIV2K</td>
        <td align="center">26.6531</td>
        <td align="center">26.6532</td>

</tr>
    <tr>
    </tr>
    <tr>
        <td align="center">0.7340</td>
    </tr>
    <tr>
            <code>esrgan_psnr_x4c64b23g32_g1_1000k_div2k.py</code>
        </td>
        <td align="center">30.6428</td>
        <td align="center">30.6307</td>
        <td align="center">30.6305</td>
    </tr>
    <tr>
        <td align="center">0.8559</td>

<td align="center">30.6307</td>
    </tr>
    <tr>
        <td align="center">0.8565</td>
        <td align="center">0.8566</td>
    </tr>
    <tr>
        <td align="center">27.0543</td>
        <td align="center">27.0422</td>
        <td align="center">27.0411</td>
    </tr>
    <tr>
        <td align="center">0.7447</td>
        <td align="center">0.7450</td>
        <td align="center">0.7449</td>
    </tr>
    <tr>
        <td align="center">29.3354</td>

<td align="center">PSNR</td>
        <td align="center">29.3339</td>
    </tr>
    <tr>
        <td align="center">0.8263</td>
    </tr>
    <tr>
	    <td align="center" rowspan="6">SRCNN</td>
            <code>srcnn_x4k915_g1_1000k_div2k.py</code>
        </td>
        <td align="center">28.4316</td>
        <td align="center">28.4120</td>
        <td align="center">27.2144</td>
        <td align="center">27.2127</td>
    </tr>
    <tr>
        <td align="center">0.8099</td>
        <td align="center">0.8106</td>
        <td align="center">0.7782</td>

</tr>
    <tr>
        <td align="center">0.7781</td>
    </tr>
    <tr>
        <td align="center">25.6486</td>
        <td align="center">25.6367</td>
        <td align="center">24.8613</td>
        <td align="center">24.8599</td>
    </tr>
    <tr>
        <td align="center">0.7014</td>
        <td align="center">0.7015</td>
        <td align="center">0.6674</td>
        <td align="center">0.6673</td>
    </tr>
    <tr>
        <td align="center">27.7460</td>
        <td align="center">26.9891</td>
        <td align="center">26.9862</td>
    </tr>
    <tr>

<td align="center">27.7460</td>
    </tr>
    <tr>
        <td align="center">0.7854</td>
        <td align="center">0.78543</td>
        <td align="center">0.7605</td>
        <td align="center">0.7604</td>
    </tr>
</table>

**注**：

- 所有 ONNX 和 TensorRT 模型都使用数据集上的动态形状进行评估，图像根据原始配置文件进行预处理。
- 此工具仍处于试验阶段，我们目前仅支持 `restorer`。

# 包结构更改

本节包含您对 MMSeg 0.x 和 1.x 之间的变化可能感到好奇的内容。

<table>
<tr>
</tr>
<tr>
</tr>
<tr>
</tr>
<tr>
</tr>
<tr>
</tr>
<tr>
</tr>
<tr>
</tr>
<tr>
<td></td>
</tr>
<tr>
<td></td>
<tr>
</table>

## 已删除的包

在 OpenMMLab 2.0 中，`core` 包已被删除。`core` 的 `hooks` 和 `optimizers` 被移动到了 `mmseg.engine` 中，而 `core` 中的 `evaluation` 目前是 mmseg.evaluation。

`ops` 包含 `encoding` 和 `wrappers`，它们被移到了 `mmseg.models.utils` 中。

## 增加的包

OpenMMLab 2.0 增加了一个新的深度学习训练基础库 MMEngine。它是所有 OpenMMLab 代码库的训练引擎。

## 增加的包

mmseg 的 `engine` 包是一些用于语义分割任务的定制模块，如 `SegVisualizationHook` 用于可视化分割掩膜。

在 OpenMMLab 2.0 中，我们为计算机视觉任务设计了数据结构，在 mmseg 中，我们在 `structure` 包中实现了 `SegDataSample`。

我们将所有评估指标都移动到了 `mmseg.evaluation` 中。

我们将 MMSegmentation 中所有类型模块的注册实现移动到 `mmseg.registry` 中。

## 修改的包

OpenMMLab 2.0 尝试支持计算机视觉的多任务统一接口，并发布了更强的 [`Runner`](https://github.com/open-mmlab/mmengine/blob/main/docs/zh_cn/design/runner.md)，因此 MMSeg 1.x 删除了 `train.py` 和 `test.py` 中的模块，并将 `init_segmentor` 重命名为 `init_model`，将 `inference_segmentor` 重命名为 `inference_model`。

以下是 `mmseg.apis` 的更改：

|         函数          | 变化                                           |
| :-------------------: | :--------------------------------------------- |

|         函数          | 变化                                           |
|   `init_segmentor`    | 重命名为 `init_model`                          |
| `inference_segmentor` | 重命名为 `inference_model`                     |
| `show_result_pyplot`  | 基于 `SegLocalVisualizer` 实现                 |
|     `train_model`     | 删除，使用 `runner.train` 训练。               |
|   `multi_gpu_test`    | 删除，使用 `runner.test` 测试。                |
|   `single_gpu_test`   | 删除，使用 `runner.test` 测试。                |
|   `set_random_seed`   | 删除，使用 `mmengine.runner.set_random_seed`。 |
|  `init_random_seed`   | 删除，使用 `mmengine.dist.sync_random_seed`。  |

OpenMMLab 2.0 将 `BaseDataset` 定义为数据集的函数和接口，MMSegmentation 1.x 也遵循此协议，并定义了从 `BaseDataset` 继承的 `BaseSegDataset`。MMCV 2.x 收集多种任务的通用数据转换，例如分类、检测、分割，因此 MMSegmentation 1.x 使用这些数据转换并将其从 mmseg.dataset 中删除。

|        包/模块        | 更改                                                                                 |
| :-------------------: | :----------------------------------------------------------------------------------- |
|   `mmseg.pipelines`   | 移动到 `mmcv.transforms` 中                                                          |
|    `mmseg.sampler`    | 移动到 `mmengine.dataset.sampler` 中                                                 |
|    `CustomDataset`    | 重命名为 `BaseSegDataset` 并从 MMEngine 中的 `BaseDataset` 继承                      |
| `DefaultFormatBundle` | 替换为 `PackSegInputs`                                                               |
|  `LoadImageFromFile`  | 移动到 `mmcv.transforms.LoadImageFromFile` 中                                        |

|   `LoadAnnotations`   | 移动到 `mmcv.transforms.LoadAnnotations` 中                                          |
|       `Resize`        | 移动到 `mmcv.transforms` 中并拆分为 `Resize`，`RandomResize` 和 `RandomChoiceResize` |
|     `RandomFlip`      | 移动到 `mmcv.transforms.RandomFlip` 中                                               |
|         `Pad`         | 移动到 `mmcv.transforms.Pad` 中                                                      |
|      `Normalize`      | 移动到 `mmcv.transforms.Normalize` 中                                                |
|       `Compose`       | 移动到 `mmcv.transforms.Compose` 中                                                  |
|    `ImageToTensor`    | 移动到 `mmcv.transforms.ImageToTensor` 中                                            |

`models` 没有太大变化，只是从以前的 `mmseg.ops` 添加了 `encoding` 和 `wrappers`

# 默认约定

如果你想把 MMDetection 修改为自己的项目，请遵循下面的约定。

## 关于图片 shape 顺序的说明

在OpenMMLab 2.0中， 为了与 OpenCV 的输入参数相一致，图片处理 pipeline 中关于图像 shape 的输入参数总是以 `(width, height)` 的顺序排列。
相反，为了计算方便，经过 pipeline 和 model 的字段的顺序是 `(height, width)`。具体来说在每个数据 pipeline 处理的结果中，字段和它们的值含义如下：

- img_shape: (height, width)
- ori_shape: (height, width)
- pad_shape: (height, width)
- batch_input_shape: (height, width)

以 `Mosaic` 为例，其初始化参数如下所示：

```python
class Mosaic(BaseTransform):
                img_scale: Tuple[int, int] = (640, 640),
                center_ratio_range: Tuple[float, float] = (0.5, 1.5),
                bbox_clip_border: bool = True,
                pad_val: float = 114.0,
                prob: float = 1.0) -> None:
       ...

       # img_scale 顺序应该是 (width, height)
       self.img_scale = img_scale

        ...

        results['img'] = mosaic_img
        # (height, width)

# img_scale 顺序应该是 (width, height)

        ...

        results['img_shape'] = mosaic_img.shape[:2]
```

## 损失

在 MMDetection 中，`model(**data)` 的返回值是一个字典，包含着所有的损失和评价指标，他们将会由 `model(**data)` 返回。

例如，在 bbox head 中，

```python
class BBoxHead(nn.Module):
    ...
    def loss(self, ...):
        # 分类损失
        losses['loss_cls'] = self.loss_cls(...)
        # 分类准确率
        losses['acc'] = accuracy(...)
        # 边界框损失
        losses['loss_bbox'] = self.loss_bbox(...)
```

`'bbox_head.loss()'` 在模型 forward 阶段会被调用。返回的字典中包含了 `'loss_bbox'`,`'loss_cls'`,`'acc'`。只有 `'loss_bbox'`, `'loss_cls'` 会被用于反向传播，`'acc'` 只会被作为评价指标来监控训练过程。

我们默认，只有那些键的名称中包含 `'loss'` 的值会被用于反向传播。这个行为可以通过修改 `BaseDetector.train_step()` 来改变。

## 空 proposals

在 MMDetection 中，我们为两阶段方法中空 proposals 的情况增加了特殊处理和单元测试。我们同时需要处理整个 batch 和单一图片中空 proposals 的情况。例如，在 CascadeRoIHead 中，

```python
# 简单的测试
...

# 在整个 batch中 都没有 proposals
if rois.shape[0] == 0:
    bbox_results = [[
        np.zeros((0, 5), dtype=np.float32)
        for _ in range(self.bbox_head[-1].num_classes)
    ]] * num_imgs
    if self.with_mask:
        mask_classes = self.mask_head[-1].num_classes
        segm_results = [[[] for _ in range(mask_classes)]
                        for _ in range(num_imgs)]
        results = list(zip(bbox_results, segm_results))
    else:
        results = bbox_results
    return results
...

# 在单张图片中没有 proposals
for i in range(self.num_stages):
    ...
    if i < self.num_stages - 1:
          for j in range(num_imgs):
                # 处理空 proposals
                if rois[j].shape[0] > 0:

    ...
                    bbox_label = cls_score[j][:, :-1].argmax(dim=1)
                    refine_roi = self.bbox_head[i].regress_by_class(
                         rois[j], bbox_label[j], bbox_pred[j], img_metas[j])
                    refine_roi_list.append(refine_roi)
```

如果你有自定义的 `RoIHead`, 你可以参考上面的方法来处理空 proposals 的情况。

## 全景分割数据集

在 MMDetection 中，我们支持了 COCO 全景分割数据集 `CocoPanopticDataset`。对于它的实现，我们在这里声明一些默认约定。

1. 在 mmdet\<=2.16.0 时，语义分割标注中的前景和背景标签范围与 MMDetection 中的默认规定有所不同。标签 `0` 代表 `VOID` 标签。
   从 mmdet=2.17.0 开始，为了和框的类别标注保持一致，语义分割标注的类别标签也改为从 `0` 开始，标签 `255` 代表 `VOID` 类。
   为了达成这一目标，我们在流程 `Pad` 里支持了设置 `seg` 的填充值的功能。
2. 在评估中，全景分割结果必须是一个与原图大小相同的图。结果图中每个像素的值有如此形式：`instance_id * INSTANCE_OFFSET + category_id`。

# 概览

MMOCR 是一个基于 [PyTorch](https://pytorch.org/) 和 [MMDetection](https://github.com/open-mmlab/mmdetection) 的开源工具箱，支持众多 OCR 相关的模型，涵盖了文本检测、文本识别以及关键信息提取等多个主要方向。它还支持了大多数流行的学术数据集，并提供了许多实用工具帮助用户对数据集和模型进行多方面的探索和调试，助力优质模型的产出和落地。它具有以下特点：

- **全流程，多模型**：支持了全流程的 OCR 任务，包括文本检测、文本识别及关键信息提取的各种最新模型。
- **模块化设计**：MMOCR 的模块化设计使用户可以按需定义及复用模型中的各个模块。
- **实用工具众多**：MMOCR 提供了全面的可视化工具、验证工具和性能评测工具，帮助用户对模型进行排错、调优或客观比较。
- **由 [OpenMMLab](https://openmmlab.com/) 强力驱动**：与家族内的其它算法库一样，MMOCR 遵循着 OpenMMLab 严谨的开发准则和接口约定，极大地降低了用户切换各算法库时的学习成本。同时，MMOCR 也可以非常便捷地与家族内其他算法库跨库联动，从而满足用户跨领域研究和落地的需求。

随着 OpenMMLab 家族架构的整体升级， MMOCR 也相应地进行了大幅度的升级和修改。在这个大版本的更新中，MMOCR 中大量的冗余代码和重复实现被移除，多个关键方法的运行效率得到了提升，且整体框架设计上变得更为统一。考虑到该版本相较于 0.x 存在一些后向不兼容的修改，我们准备了一份详细的[迁移指南](../migration/overview.md)，并在里面列出了新版本所作出的所有改动和迁移所需的步骤，力求帮助熟悉旧版框架的用户尽快完成升级。尽管这可能需要一定时间，但我们相信由 MMOCR 和 OpenMMLab 生态系统整体带来的新特性会让这一切变得尤为值得。😊

接下来，请根据实际需求选择你需要阅读的章节。

- 我们推荐初学者通过【[快速运行](quick_run.md)】来熟悉 MMOCR 的基本用法，并从【用户指南】提供的案例中逐步掌握 MMOCR 的用法。
- 中高级开发者则可以从【基础概念】中了解各个组件的背景、约定和推荐实现。

- 请阅读 [FAQ](faq.md) 来查找常见问题的答案。
- 同时，如果你在文档中未能找到需要的答案，欢迎通过 [issue](https://github.com/open-mmlab/mmocr/issues) 进行反馈。
- 我们也欢迎每一位用户成为贡献者！请阅读 [贡献指南](../notes/contribution_guide.md) 来了解如何为 MMOCR 做出贡献。

# ResNeXt

> [Aggregated Residual Transformations for Deep Neural Networks](https://openaccess.thecvf.com/content_cvpr_2017/html/Xie_Aggregated_Residual_Transformations_CVPR_2017_paper.html)

<img src="https://user-images.githubusercontent.com/26739999/142574479-21fb00a2-e63e-4bc6-a9f2-989cd6e15528.png" width="70%"/>
</div>

```python

predict = inference_model('resnext50-32x4d_8xb32_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('resnext50-32x4d_8xb32_in1k', pretrained=True)
```

Train:

```shell
python tools/train.py configs/resnext/resnext50-32x4d_8xb32_in1k.py
```

Test:

```shell

Train:

```shell
```

Test:

```shell
python tools/test.py configs/resnext/resnext50-32x4d_8xb32_in1k.py https://download.openmmlab.com/mmclassification/v0/resnext/resnext50_32x4d_b32x8_imagenet_20210429-56066e27.pth
```

| Model                         |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                  Config                  |                                      Download                                      |
| :---------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :--------------------------------------: | :--------------------------------------------------------------------------------: |

| `resnext50-32x4d_8xb32_in1k`  | From scratch |   25.03    |   4.27    |   77.90   |   93.66   | [config](resnext50-32x4d_8xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/resnext/resnext50_32x4d_b32x8_imagenet_20210429-56066e27.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnext/resnext50_32x4d_b32x8_imagenet_20210429-56066e27.json) |
| `resnext101-32x4d_8xb32_in1k` | From scratch |   44.18    |   8.03    |   78.61   |   94.17   | [config](resnext101-32x4d_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/resnext/resnext101_32x4d_b32x8_imagenet_20210506-e0fa3dd5.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnext/resnext101_32x4d_b32x8_imagenet_20210506-e0fa3dd5.json) |

| `resnext101-32x8d_8xb32_in1k` | From scratch |   88.79    |   16.50   |   79.27   |   94.58   | [config](resnext101-32x8d_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/resnext/resnext101_32x8d_b32x8_imagenet_20210506-23a247d5.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnext/resnext101_32x8d_b32x8_imagenet_20210506-23a247d5.json) |
| `resnext152-32x4d_8xb32_in1k` | From scratch |   59.95    |   11.80   |   78.88   |   94.33   | [config](resnext152-32x4d_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/resnext/resnext152_32x4d_b32x8_imagenet_20210524-927787be.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnext/resnext152_32x4d_b32x8_imagenet_20210524-927787be.json) |

```bibtex

```bibtex
}
```

# Group-Free 3D Object Detection via Transformers

> [Group-Free 3D Object Detection via Transformers](https://arxiv.org/abs/2104.00678)

Recently, directly detecting 3D objects from 3D point clouds has received increasing attention. To extract object representation from an irregular point cloud, existing methods usually take a point grouping step to assign the points to an object candidate so that a PointNet-like network could be used to derive object features from the grouped points. However, the inaccurate point assignments caused by the hand-crafted grouping scheme decrease the performance of 3D object detection. In this paper, we present a simple yet effective method for directly detecting 3D objects from the 3D point cloud. Instead of grouping local points to each object candidate, our method computes the feature of an object from all the points in the point cloud with the help of an attention mechanism in the Transformers, where the contribution of each point is automatically learned in the network training. With an improved attention stacking scheme, our method fuses object features in different stages and generates more accurate object detection results. With few bells and whistles, the proposed method achieves state-of-the-art 3D object detection performance on two widely used benchmarks, ScanNet V2 and SUN RGB-D.

<img src="https://user-images.githubusercontent.com/36950400/143868101-09787c2a-9e0b-4013-8800-b4e315d535f0.png" width="800"/>
</div>

We implement Group-Free-3D and provide the result and checkpoints on ScanNet datasets.

|                              Method                              |   Backbone    | Lr schd | Mem (GB) | Inf time (fps) |     AP@0.25     |     AP@0.5      |                                                                                                                                                                                                        Download                                                                                                                                                                                                        |

|    [L6, O256](./groupfree3d_head-L6-O256_4xb8_scannet-seg.py)    |  PointNet++   |   3x    |   6.7    |                | 66.17 (65.67\*) | 48.47 (47.74\*) |           [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/groupfree3d/groupfree3d_8x4_scannet-3d-18class-L6-O256/groupfree3d_8x4_scannet-3d-18class-L6-O256_20210702_145347-3499eb55.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/groupfree3d/groupfree3d_8x4_scannet-3d-18class-L6-O256/groupfree3d_8x4_scannet-3d-18class-L6-O256_20210702_145347.log.json)           |

|   [L12, O256](./groupfree3d_head-L12-O256_4xb8_scannet-seg.py)   |  PointNet++   |   3x    |   9.4    |                | 66.57 (66.22\*) | 48.21 (48.95\*) |         [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/groupfree3d/groupfree3d_8x4_scannet-3d-18class-L12-O256/groupfree3d_8x4_scannet-3d-18class-L12-O256_20210702_150907-1c5551ad.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/groupfree3d/groupfree3d_8x4_scannet-3d-18class-L12-O256/groupfree3d_8x4_scannet-3d-18class-L12-O256_20210702_150907.log.json)         |

| [L12, O256](./groupfree3d_w2x-head-L12-O256_4xb8_scannet-seg.py) | PointNet++w2x |   3x    |   13.3   |                | 68.20 (67.30\*) | 51.02 (50.44\*) | [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/groupfree3d/groupfree3d_8x4_scannet-3d-18class-w2x-L12-O256/groupfree3d_8x4_scannet-3d-18class-w2x-L12-O256_20210702_200301-944f0ac0.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/groupfree3d/groupfree3d_8x4_scannet-3d-18class-w2x-L12-O256/groupfree3d_8x4_scannet-3d-18class-w2x-L12-O256_20210702_200301.log.json) |

| [L12, O512](./groupfree3d_w2x-head-L12-O512_4xb8_scannet-seg.py) | PointNet++w2x |   3x    |   18.8   |                | 68.22 (68.20\*) | 52.61 (51.31\*) | [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/groupfree3d/groupfree3d_8x4_scannet-3d-18class-w2x-L12-O512/groupfree3d_8x4_scannet-3d-18class-w2x-L12-O512_20210702_220204-187b71c7.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/groupfree3d/groupfree3d_8x4_scannet-3d-18class-w2x-L12-O512/groupfree3d_8x4_scannet-3d-18class-w2x-L12-O512_20210702_220204.log.json) |

- We defined L6-O256 represent num_layers=6 and num_proposals=256. And w2x means that the model backbone weight is twice the original.

- We report the best results (AP@0.50) on validation set during each training. * means the evaluation method in the paper: we train each setting 5 times and test each training trial 5 times, then the average performance of these 25 trials is reported to account for algorithm randomness.
- We use 4 GPUs for training by default as the original code.

```latex
@article{liu2021,
  title={Group-Free 3D Object Detection via Transformers},
  author={Liu, Ze and Zhang, Zheng and Cao, Yue and Hu, Han and Tong, Xin},
}
```

# Unconditional GANs Datasets

**Data preparation for unconditional model** is simple. What you need to do is downloading the images and put them into a directory. Next, you should set a symlink in the `data` directory. For standard unconditional gans with static architectures, like DCGAN and StyleGAN2, `UnconditionalImageDataset` is designed to train such unconditional models. Here is an example config for FFHQ dataset:

```python
dataset_type = 'BasicImageDataset'

    dict(type='LoadImageFromFile', key='img'),
    dict(type='Flip', keys=['img'], direction='horizontal'),
    dict(type='PackInputs', keys=['img'], meta_keys=['img_path'])
]

# `batch_size` and `data_root` need to be set.
    sampler=dict(type='InfiniteSampler', shuffle=True),
        data_root=None,  # set by user
```

sampler=dict(type='InfiniteSampler', shuffle=True),
```

Here, we adopt `InfinitySampler` to avoid frequent dataloader reloading, which will accelerate the training procedure. As shown in the example, `pipeline` provides important data pipeline to process images, including loading from file system, resizing, cropping, transferring to `torch.Tensor` and packing to `DataSample`. All of supported data pipelines can be found in `mmagic/datasets/transforms`.

For unconditional GANs with dynamic architectures like PGGAN and StyleGANv1, `GrowScaleImgDataset` is recommended to use for training. Since such dynamic architectures need real images in different scales, directly adopting `UnconditionalImageDataset` will bring heavy I/O cost for loading multiple high-resolution images. Here is an example we use for training PGGAN in CelebA-HQ dataset:

```python

```python
dataset_type = 'GrowScaleImgDataset'

]

# `samples_per_gpu` and `imgs_root` need to be set.
        type='GrowScaleImgDataset',
        data_roots={
            '1024': './data/ffhq/images',
            '256': './data/ffhq/ffhq_imgs/ffhq_256',
            '64': './data/ffhq/ffhq_imgs/ffhq_64'
        },
        gpu_samples_base=4,
        # note that this should be changed with total gpu number
        gpu_samples_per_scale={
            '4': 64,
            '8': 32,
            '16': 16,
            '32': 8,
            '64': 4,
            '128': 4,
            '256': 4,
            '512': 4,
            '1024': 4
        },
        len_per_stage=300000,
        pipeline=pipeline),
    sampler=dict(type='InfiniteSampler', shuffle=True))
```

'256': 4,
            '512': 4,
            '1024': 4
        },
```

In this dataset, you should provide a dictionary of image paths to the `data_roots`. Thus, you should resize the images in the dataset in advance.
For the resizing methods in the data pre-processing, we adopt bilinear interpolation methods in all of the experiments studied in MMagic.

Note that this dataset should be used with `PGGANFetchDataHook`. In this config file, this hook should be added in the customized hooks, as shown below.

```python
    dict(
        # vis ema and orig at the same time
            name='fake_img',
            sample_model='ema/orig',
            target_keys=['ema', 'orig'])),
    dict(type='PGGANFetchDataHook')

]
```

This fetching data hook helps the dataloader update the status of dataset to change the data source and batch size during training.

Here, we provide several download links of datasets frequently used in unconditional models: [LSUN](http://dl.yf.io/lsun/), [CelebA](http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html), [CelebA-HQ](https://drive.google.com/drive/folders/11Vz0fqHS2rXDb5pprgTjpD7S2BAJhi1P), [FFHQ](https://drive.google.com/drive/folders/1u2xu7bSrWxrbUxk-dT-UvEJq8IjdmNTP).

# NAS-FPN

> [NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection](https://arxiv.org/abs/1904.07392)

Current state-of-the-art convolutional architectures for object detection are manually designed. Here we aim to learn a better architecture of feature pyramid network for object detection. We adopt Neural Architecture Search and discover a new feature pyramid architecture in a novel scalable search space covering all cross-scale connections. The discovered architecture, named NAS-FPN, consists of a combination of top-down and bottom-up connections to fuse features across scales. NAS-FPN, combined with various backbone models in the RetinaNet framework, achieves better accuracy and latency tradeoff compared to state-of-the-art object detection models. NAS-FPN improves mobile detection accuracy by 2 AP compared to state-of-the-art SSDLite with MobileNetV2 model in \[32\] and achieves 48.3 AP which surpasses Mask R-CNN \[10\] detection accuracy with less computation time.

<img src="https://user-images.githubusercontent.com/40661020/143968037-cedd76e9-1ae7-4869-bd34-c9d8611d630c.png"/>
</div>

We benchmark the new training schedule (crop training, large batch, unfrozen BN, 50 epochs) introduced in NAS-FPN. RetinaNet is used in the paper.

|  Backbone   | Lr schd | Mem (GB) | Inf time (fps) | box AP |                        Config                        |                                                                                                                                                             Download                                                                                                                                                             |

| :---------: | :-----: | :------: | :------------: | :----: | :--------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|  R-50-FPN   |   50e   |   12.9   |      22.9      |  37.9  |  [config](./retinanet_r50_fpn_crop640-50e_coco.py)   |       [model](https://download.openmmlab.com/mmdetection/v2.0/nas_fpn/retinanet_r50_fpn_crop640_50e_coco/retinanet_r50_fpn_crop640_50e_coco-9b953d76.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/nas_fpn/retinanet_r50_fpn_crop640_50e_coco/retinanet_r50_fpn_crop640_50e_coco_20200529_095329.log.json)       |

| R-50-NASFPN |   50e   |   13.2   |      23.0      |  40.5  | [config](./retinanet_r50_nasfpn_crop640-50e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/nas_fpn/retinanet_r50_nasfpn_crop640_50e_coco/retinanet_r50_nasfpn_crop640_50e_coco-0ad1f644.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/nas_fpn/retinanet_r50_nasfpn_crop640_50e_coco/retinanet_r50_nasfpn_crop640_50e_coco_20200528_230008.log.json) |

**Note**: We find that it is unstable to train NAS-FPN and there is a small chance that results can be 3% mAP lower.

```latex
@inproceedings{ghiasi2019fpn,
  title={Nas-fpn: Learning scalable feature pyramid architecture for object detection},
  author={Ghiasi, Golnaz and Lin, Tsung-Yi and Le, Quoc V},
  pages={7036--7045},
}
```

3. The documentation has been modified accordingly, like docstring or example tutorials.

# 学习配置文件

我们使用 Python 文件作为配置文件，将模块化和继承设计融入我们的配置系统中，这方便进行各种实验。
您可以在 `$MMAction2/configs` 目录下找到所有提供的配置文件。如果您想要查看配置文件，
您可以运行 `python tools/analysis_tools/print_config.py /PATH/TO/CONFIG` 来查看完整的配置文件。

- [学习配置文件](#学习配置文件)
  - [通过脚本参数修改配置](#通过脚本参数修改配置)
  - [配置文件结构](#配置文件结构)
  - [配置文件命名约定](#配置文件命名约定)
    - [动作识别的配置系统](#动作识别的配置系统)
    - [时空动作检测的配置系统](#时空动作检测的配置系统)
    - [动作定位的配置系统](#动作定位的配置系统)

在使用 `tools/train.py` 或 `tools/test.py` 提交作业时，您可以通过指定 `--cfg-options` 来原地修改配置。

- 更新字典的配置键。

  可以按照原始配置中字典键的顺序来指定配置选项。
  例如，`--cfg-options model.backbone.norm_eval=False` 将模型骨干中的所有 BN 模块更改为 `train` 模式。

- 更新配置列表中的键。

  一些配置字典在配置文件中以列表形式组成。例如，训练流程 `train_pipeline` 通常是一个列表，
  例如 `[dict(type='SampleFrames'), ...]`。如果您想要在流程中将 `'SampleFrames'` 更改为 `'DenseSampleFrames'`，
  您可以指定 `--cfg-options train_pipeline.0.type=DenseSampleFrames`。

  如果要更新的值是列表或元组。例如，配置文件通常设置 `model.data_preprocessor.mean=[123.675, 116.28, 103.53]`。如果您想要

您可以指定 `--cfg-options train_pipeline.0.type=DenseSampleFrames`。

  更改此键，您可以指定 `--cfg-options model.data_preprocessor.mean="[128,128,128]"`。请注意，引号 " 是支持列表/元组数据类型的必需内容。

## 配置文件结构

`configs/_base_` 下有 3 种基本组件类型，即 models、schedules 和 default_runtime。
许多方法只需要一个模型、一个训练计划和一个默认运行时组件就可以轻松构建，如 TSN、I3D、SlowOnly 等。
由 `_base_` 组件组成的配置文件被称为 _primitive_。

对于同一文件夹下的所有配置文件，建议只有**一个** _primitive_ 配置文件。其他所有配置文件都应该继承自 _primitive_ 配置文件。这样，继承级别的最大值为 3。

为了方便理解，我们建议贡献者继承现有方法。
例如，如果基于 TSN 进行了一些修改，用户可以首先通过指定 `_base_ = ../tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py` 来继承基本的 TSN 结构，然后在配置文件中修改必要的字段。

如果您正在构建一个与任何现有方法的结构不共享的全新方法，可以在 `configs/TASK` 下创建一个文件夹。

请参考 [mmengine](https://mmengine.readthedocs.io/en/latest/tutorials/config.html) 获取详细文档。

## 配置文件命名约定

我们遵循以下样式来命名配置文件。建议贡献者遵循相同的样式。配置文件名分为几个部分，不同部分逻辑上用下划线 `'_'` 连接，同一部分的设置用破折号 `'-'` 连接。

```
{算法信息}_{模块信息}_{训练信息}_{数据信息}.py
```

```
```

`{xxx}` 是必填字段，`[yyy]` 是可选字段。

- `{算法信息}`:
  - `{模型}`: 模型类型，例如 `tsn`、`i3d`、`swin`、`vit` 等。
  - `[模型设置]`: 某些模型的特定设置，例如 `base`、`p16`、`w877` 等。
- `{模块信息}`:
  - `[预训练信息]`: 预训练信息，例如 `kinetics400-pretrained`、`in1k-pre` 等。
  - `{骨干网络}`: 骨干网络类型，例如 `r50`（ResNet-50）等。
  - `[骨干网络设置]`: 某些骨干网络的特定设置，例如 `nl-dot-product`、`bnfrozen`、`nopool` 等。
- `{训练信息}`:
  - `{gpu x batch_per_gpu]}`: GPU 和每个 GPU 上的样本数。
  - `{pipeline设置}`: 帧采样设置，例如 `dense`、`{clip_len}x{frame_interval}x{num_clips}`、`u48` 等。
  - `{schedule}`: 训练计划，例如 `coslr-20e`。
- `{数据信息}`:
  - `{数据集}`: 数据集名称，例如 `kinetics400`、`mmit` 等。
  - `{模态}`: 数据模态，例如 `rgb`、`flow`、`keypoint-2d` 等。

### 动作识别的配置系统

我们将模块化设计融入我们的配置系统中，
这方便进行各种实验。

- TSN 的示例

  为了帮助用户对完整的配置结构和动作识别系统中的模块有一个基本的了解，
  我们对 TSN 的配置进行简要注释如下。有关每个模块中每个参数的更详细用法和替代方法，请参阅 API 文档。

  ```python
  # 模型设置
  model = dict(  # 模型的配置
      type='Recognizer2D',  # 识别器的类名

- TSN 的示例

  ```python
  # 模型设置
      backbone=dict(  # 骨干网络的配置
          type='ResNet',  # 骨干网络的名称
          pretrained='torchvision://resnet50',  # 预训练模型的 URL/网站
          depth=50,  # ResNet 模型的深度
          norm_eval=False),  # 是否在训练时将 BN 层设置为评估模式
      cls_head=dict(  # 分类头的配置
          type='TSNHead',  # 分类头的名称
          num_classes=400,  # 要分类的类别数量。
          in_channels=2048,  # 分类头的输入通道数。
          spatial_type='avg',  # 空间维度池化的类型
          consensus=dict(type='AvgConsensus', dim=1),  # 一致性模块的配置
          dropout_ratio=0.4,  # dropout 层中的概率
          init_std=0.01, # 线性层初始化的标准差值
          average_clips='prob'),  # 平均多个剪辑结果的方法
      data_preprocessor=dict(  # 数据预处理器的配置
          type='ActionDataPreprocessor',  # 数据预处理器的名称
          mean=[123.675, 116.28, 103.53],  # 不同通道的均值用于归一化

data_preprocessor=dict(  # 数据预处理器的配置
          std=[58.395, 57.12, 57.375],  # 不同通道的标准差用于归一化
          format_shape='NCHW'),  # 最终图像形状的格式
      # 模型训练和测试设置
      train_cfg=None,  # TSN 的训练超参数的配置
      test_cfg=None)  # TSN 的测试超参数的配置

  # 数据集设置
  dataset_type = 'RawframeDataset'  # 用于训练、验证和测试的数据集类型
  data_root = 'data/kinetics400/rawframes_train/'  # 用于训练的数据的根路径
  data_root_val = 'data/kinetics400/rawframes_val/'  # 用于验证和测试的数据的根路径
  ann_file_train = 'data/kinetics400/kinetics400_train_list_rawframes.txt'  # 用于训练的注释文件的路径
  ann_file_val = 'data/kinetics400/kinetics400_val_list_rawframes.txt'  # 用于验证的注释文件的路径
  ann_file_test = 'data/kinetics400/kinetics400_val_list_rawframes.txt'  # 用于测试的注释文件的路径

  train_pipeline = [  # 训练数据处理流程
      dict(  # SampleFrames 的配置
          type='SampleFrames',  # 采样帧的流程，从视频中采样帧
          clip_len=1,  # 每个采样输出剪辑的帧数

          frame_interval=1,  # 相邻采样帧的时间间隔
          num_clips=3),  # 要采样的剪辑数
      dict(  # RawFrameDecode 的配置
          type='RawFrameDecode'),  # 加载和解码帧的流程，选择给定索引的原始帧
      dict(  # Resize 的配置
          type='Resize',  # 调整大小的流程
          scale=(-1, 256)),  # 要调整图像的比例
      dict(  # MultiScaleCrop 的配置
          type='MultiScaleCrop',  # 多尺度裁剪的流程，根据随机选择的尺度列表裁剪图像
          input_size=224,  # 网络的输入大小
          scales=(1, 0.875, 0.75, 0.66),  # 要选择的宽度和高度的尺度
          random_crop=False,  # 是否随机采样裁剪框
          max_wh_scale_gap=1),  # 宽度和高度尺度级别的最大差距
          scale=(224, 224),  # 要调整图像的比例
          keep_ratio=False),  # 是否保持纵横比进行调整大小
      dict(  # Flip 的配置
          type='Flip',  # 翻转的流程
          flip_ratio=0.5),  # 实施翻转的概率
      dict(  # FormatShape 的配置

keep_ratio=False),  # 是否保持纵横比进行调整大小
          type='FormatShape',  # 格式化形状的流程，将最终图像形状格式化为给定的 input_format
          input_format='NCHW'),  # 最终图像形状的格式
      dict(type='PackActionInputs')  # PackActionInputs 的配置
  ]
  val_pipeline = [  # 验证数据处理流程
          num_clips=3,  # 要采样的剪辑数
          test_mode=True),  # 是否在采样时设置为测试模式
      dict(  # CenterCrop 的配置
          type='CenterCrop',  # 中心裁剪的流程，从图像中裁剪中心区域
          crop_size=224),  # 要裁剪的图像大小

dict(  # CenterCrop 的配置
          flip_ratio=0),  # 实施翻转的概率
  ]
  test_pipeline = [  # 测试数据处理流程
          num_clips=25,  # 要采样的剪辑数
      dict(  # TenCrop 的配置
          type='TenCrop',  # 十次裁剪的流程，从图像中裁剪十个区域

type='Resize',  # 调整大小的流程
  ]

  train_dataloader = dict(  # 训练数据加载器的配置
      batch_size=32,  # 训练时每个单个 GPU 的批量大小
      num_workers=8,  # 训练时每个单个 GPU 的数据预取进程数
      persistent_workers=True,  # 如果为 `True`，则数据加载器在一个 epoch 结束后不会关闭工作进程，这可以加速训练速度
          type='DefaultSampler',  # 支持分布式和非分布式训练的 DefaultSampler。参考 https://github.com/open-mmlab/mmengine/blob/main/mmengine/dataset/sampler.py
          shuffle=True),  # 每个 epoch 随机打乱训练数据
      dataset=dict(  # 训练数据集的配置

shuffle=True),  # 每个 epoch 随机打乱训练数据
          ann_file=ann_file_train,  # 注释文件的路径
          data_prefix=dict(img=data_root),  # 帧路径的前缀
  val_dataloader = dict(  # 验证数据加载器的配置
      batch_size=1,  # 验证时每个单个 GPU 的批量大小
      num_workers=8,  # 验证时每个单个 GPU 的数据预取进程数
      persistent_workers=True,  # 如果为 `True`，则数据加载器在一个 epoch 结束后不会关闭工作进程
          shuffle=False),  # 验证和测试时不进行随机打乱
      dataset=dict(  # 验证数据集的配置
          ann_file=ann_file_val,  # 注释文件的路径
          data_prefix=dict(img=data_root_val),  # 帧路径的前缀
  test_dataloader = dict(  # 测试数据加载器的配置
      batch_size=32,  # 测试时每个单个 GPU 的批量大小
      num_workers=8,  # 测试时每个单个 GPU 的数据预取进程数

batch_size=32,  # 测试时每个单个 GPU 的批量大小
      dataset=dict(  # 测试数据集的配置

  # 评估设置
  val_evaluator = dict(type='AccMetric')  # 验证评估器的配置
  test_evaluator = val_evaluator  # 测试评估器的配置

  train_cfg = dict(  # 训练循环的配置
      type='EpochBasedTrainLoop',  # 训练循环的名称
      max_epochs=100,  # 总的训练周期数
      val_begin=1,  # 开始验证的训练周期
      val_interval=1)  # 验证间隔
  val_cfg = dict(  # 验证循环的配置
      type='ValLoop')  # 验证循环的名称
  test_cfg = dict( # 测试循环的配置
      type='TestLoop')  # 测试循环的名称

  # 学习策略
  param_scheduler = [  # 更新优化器参数的学习率测率，支持字典或列表
      dict(type='MultiStepLR',  # 达到一个里程碑时衰减学习率

type='ValLoop')  # 验证循环的名称

  # 学习策略
          begin=0,  # 开始更新学习率的步骤
          end=100,  # 结束更新学习率的步骤
          by_epoch=True,  # 是否按 epoch 更新学习率
          milestones=[40, 80],  # 衰减学习率的步骤
          gamma=0.1)]  # 学习率衰减的乘法因子

  # 优化器
  optim_wrapper = dict(  # 优化器包装器的配置
      type='OptimWrapper',  # 优化器包装器的名称，切换到 AmpOptimWrapper 可以启用混合精度训练
      optimizer=dict(  # 优化器的配置。支持 PyTorch 中的各种优化器。参考 https://pytorch.org/docs/stable/optim.html#algorithms
          type='SGD',  # 优化器的名称
          lr=0.01,  # 学习率
          momentum=0.9,  # 动量因子
          weight_decay=0.0001),  # 权重衰减
      clip_grad=dict(max_norm=40, norm_type=2))  # 梯度裁剪的配置

  # 运行时设置
  default_scope = 'mmaction'  # 用于查找模块的默认注册表作用域。参考 https://mmengine.readthedocs.io/en/latest/tutorials/registry.html
  default_hooks = dict(  # 执行默认操作的钩子，如更新模型参数和保存权重。

# 运行时设置
      runtime_info=dict(type='RuntimeInfoHook'),  # 将运行时信息更新到消息中心的钩子
      timer=dict(type='IterTimerHook'),  # 用于记录迭代过程中花费的时间的日志记录器
          type='LoggerHook',  # 用于记录训练/验证/测试阶段的日志记录器
          interval=20,  # 打印日志的间隔
          ignore_last=False), # 忽略每个 epoch 中最后几个迭代的日志
      param_scheduler=dict(type='ParamSchedulerHook'),  # 更新优化器中某些超参数的钩子
          type='CheckpointHook',  # 定期保存权重的钩子
          interval=3,  # 保存的周期
          save_best='auto',  # 用于评估最佳权重的指标
          max_keep_ckpts=3),  # 保留的最大权重文件数量
      sampler_seed=dict(type='DistSamplerSeedHook'),  # 用于分布式训练的数据加载采样器
      sync_buffers=dict(type='SyncBuffersHook'))  # 在每个 epoch 结束时同步模型缓冲区

  env_cfg = dict(  # 设置环境的字典
      cudnn_benchmark=False,  # 是否启用 cudnn benchmark

sync_buffers=dict(type='SyncBuffersHook'))  # 在每个 epoch 结束时同步模型缓冲区

      mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0), # 设置多进程的参数
      dist_cfg=dict(backend='nccl')) # 设置分布式环境的参数，也可以设置端口号

      type='LogProcessor',  # 用于格式化日志信息的日志处理器
      window_size=20,  # 默认的平滑间隔
      by_epoch=True)  # 是否使用 epoch 类型格式化日志
  vis_backends = [  # 可视化后端的列表
      dict(type='LocalVisBackend')]  # 本地可视化后端
  visualizer = dict(  # 可视化器的配置
      type='ActionVisualizer',  # 可视化器的名称
  log_level = 'INFO'  # 日志记录的级别
  load_from = None  # 从给定路径加载模型权重作为预训练模型。这不会恢复训练。
  resume = False  # 是否从 `load_from` 中定义的权重恢复。如果 `load_from` 为 None，则会从 `work_dir` 中恢复最新的权重。
  ```

### 时空动作检测的配置系统

我们将模块化设计融入我们的配置系统中，这方便进行各种实验。

- FastRCNN 的示例

  为了帮助用户对完整的配置结构和时空动作检测系统中的模块有一个基本的了解，
  我们对 FastRCNN 的配置进行简要注释如下。有关每个模块中每个参数的更详细用法和替代方法，请参阅 API 文档。

  ```python
  # 模型设置

```

  ```python
  # 模型设置
      type='FastRCNN',  # 检测器的类名
      _scope_='mmdet',  # 当前配置的范围
          type='ResNet3dSlowOnly',  # 骨干网络的名称
          depth=50, # ResNet 模型的深度
          pretrained=None,   # 预训练模型的 URL/网站
          pretrained2d=False, # 如果预训练模型是 2D 的
          lateral=False,  # 如果骨干网络带有横向连接
          num_stages=4, # ResNet 模型的阶段数
          conv1_kernel=(1, 7, 7), # Conv1 的卷积核大小
          conv1_stride_t=1, # Conv1 的时间步长
          pool1_stride_t=1, # Pool1 的时间步长
          spatial_strides=(1, 2, 2, 1)),  # 每个 ResNet 阶段的空间步长
      roi_head=dict(  # roi_head 的配置
          type='AVARoIHead',  # roi_head 的名称
          bbox_roi_extractor=dict(  # bbox_roi_extractor 的配置
              type='SingleRoIExtractor3D',  # bbox_roi_extractor 的名称

type='AVARoIHead',  # roi_head 的名称
              roi_layer_type='RoIAlign',  # RoI 操作的类型
              output_size=8,  # RoI 操作的输出特征大小
              with_temporal_pool=True), # 是否进行时间维度的池化
          bbox_head=dict( # bbox_head 的配置
              type='BBoxHeadAVA', # bbox_head 的名称
              in_channels=2048, # 输入特征的通道数
              num_classes=81, # 动作类别数 + 1
              multilabel=True,  # 数据集是否为多标签
              dropout_ratio=0.5),  # 使用的 dropout 比例
          format_shape='NCHW'))  # 最终图像形状的格式
      train_cfg=dict(
          rcnn=dict(
              assigner=dict(
                  type='MaxIoUAssignerAVA',  # 分配器的名称

format_shape='NCHW'))  # 最终图像形状的格式
                  pos_iou_thr=0.9,  # 正样本的 IoU 阈值，> pos_iou_thr -> 正样本
                  neg_iou_thr=0.9,  # 负样本的 IoU 阈值，< neg_iou_thr -> 负样本
                  min_pos_iou=0.9),  # 正样本的最小可接受 IoU
              sampler=dict(
                  type='RandomSampler',  # 采样器的名称
                  num=32,  # 采样器的批处理大小
                  pos_fraction=1,  # 采样器的正样本比例
                  neg_pos_ub=-1,  # 负样本与正样本数量比率的上限
                  add_gt_as_proposals=True),  # 将 gt 边界框添加到 proposals 中
              pos_weight=1.0)),  # 正样本的损失权重
      test_cfg=dict(rcnn=None))  # 测试的配置

  # 数据集设置
  dataset_type = 'AVADataset'  # 训练、验证和测试的数据集类型
  data_root = 'data/ava/rawframes'  # 数据的根目录
  anno_root = 'data/ava/annotations'  # 注释的根目录

  ann_file_train = f'{anno_root}/ava_train_v2.1.csv'  # 训练注释文件的路径

data_root = 'data/ava/rawframes'  # 数据的根目录

  ann_file_val = f'{anno_root}/ava_val_v2.1.csv'  # 验证注释文件的路径

  exclude_file_train = f'{anno_root}/ava_train_excluded_timestamps_v2.1.csv'  # 训练排除注释文件的路径
  exclude_file_val = f'{anno_root}/ava_val_excluded_timestamps_v2.1.csv'  # 验证排除注释文件的路径

  label_file = f'{anno_root}/ava_action_list_v2.1_for_activitynet_2018.pbtxt'  # 标签文件的路径

  proposal_file_train = f'{anno_root}/ava_dense_proposals_train.FAIR.recall_93.9.pkl'  # 训练示例的人体检测 proposals 文件的路径
  proposal_file_val = f'{anno_root}/ava_dense_proposals_val.FAIR.recall_93.9.pkl'  # 验证示例的人体检测 proposals 文件的路径

      dict(
          type='AVASampleFrames',  # 从视频中采样帧的管道
          clip_len=4,  # 每个采样输出的帧数
          frame_interval=16),  # 相邻采样帧之间的时间间隔
      dict(
          type='RawFrameDecode'),  # 加载和解码帧的管道，使用给定的索引选择原始帧
      dict(

clip_len=4,  # 每个采样输出的帧数
      dict(
      dict(
          type='RandomRescale',  # 随机缩放短边
          scale_range=(256, 320)),  # 随机缩放的短边尺寸范围
      dict(
          type='RandomCrop',  # 随机裁剪给定大小的补丁
          size=256),  # 裁剪补丁的大小
      dict(
          type='Flip',  # 翻转管道
          flip_ratio=0.5),  # 翻转的概率
      dict(
          type='FormatShape',  # 格式化形状的管道，将最终图像形状格式化为给定的输入格式
          input_format='NCTHW',  # 最终图像形状的格式
          collapse=True),  # 如果 N == 1，则减少维度 N
      dict(type='PackActionInputs')  # 打包输入数据
  ]

      dict(
      dict(
      dict(
          type='Resize',  # 调整大小的管道
          scale=(-1, 256)),  # 调整图像的尺度
      dict(

dict(
      dict(
      dict(
  ]

      batch_size=32,  # 每个单 GPU 训练的批处理大小
      num_workers=8,  # 每个单 GPU 训练时预取数据的 worker 数量
      persistent_workers=True,  # 如果为 `True`，则数据加载器在一个 epoch 结束后不会关闭 worker 进程，这可以加快训练速度
          type='DefaultSampler',  # 默认采样器，支持分布式和非分布式训练。参考 https://github.com/open-mmlab/mmengine/blob/main/mmengine/dataset/sampler.py
          shuffle=True),  # 在每个 epoch 中随机打乱训练数据
          exclude_file=exclude_file_train,  # 排除注释文件的路径
          label_file=label_file,  # 标签文件的路径

          proposal_file=proposal_file_train,  # 人体检测 proposals 的路径
          pipeline=train_pipeline)
  )
      batch_size=1,  # 每个单 GPU 评估的批处理大小
      num_workers=8,  # 每个单 GPU 评估时预取数据的 worker 数量
      persistent_workers=True,  # 如果为 `True`，则数据加载器在一个 epoch 结束后不会关闭 worker 进程
          shuffle=False),  # 在验证和测试时不打乱数据
          exclude_file=exclude_file_val,  # 排除注释文件的路径
          proposal_file=proposal_file_val,  # 人体检测 proposals 的路径

label_file=label_file,  # 标签文件的路径
          test_mode=True)
  )
  test_dataloader = val_dataloader  # 测试数据加载器的配置

  # 评估设置

  train_cfg = dict(
      max_epochs=20,  # 总的训练 epoch 数量
      val_begin=1,  # 开始验证的 epoch
      val_interval=1)  # 验证的间隔
  val_cfg = dict(
  test_cfg = dict(

  # 学习策略
      dict(
          type='LinearLR',  # 线性减少每个参数组的学习率
          start_factor=0.1,  # 第一个 epoch 中学习率的乘法因子
          end=5),  # 停止更新学习率的步骤

start_factor=0.1,  # 第一个 epoch 中学习率的乘法因子
      dict(
          type='MultiStepLR',  # 当 epoch 数达到里程碑时，减少学习率
          end=20,  # 停止更新学习率的步骤
          milestones=[10, 15],  # 学习率衰减的步骤
          gamma=0.1)  # 学习率衰减的乘法因子
  ]

  # 优化器
      type='OptimWrapper',  # 优化器包装器的名称，切换到 AmpOptimWrapper 以启用混合精度训练
      optimizer=dict(
          lr=0.2,  # 学习率
      clip_grad=dict(max_norm=40, norm_type=2))  # 梯度剪裁的配置

  # 运行时设置
  default_scope = 'mmaction'  # 默认注册表范围，用于查找模块。参考 https://mmengine.readthedocs.io/en/latest/tutorials/registry.html
  default_hooks = dict(

          type='LoggerHook',  # 用于记录训练/验证/测试阶段的日志的日志记录器
          ignore_last=False),  # 忽略每个 epoch 中最后几次迭代的日志
      param_scheduler=dict(type='ParamSchedulerHook'),  # 更新优化器中的某些超参数的钩子
          interval=3,  # 保存周期
          save_best='auto',  # 在评估过程中测量最佳权重的指标
      sync_buffers=dict(type='SyncBuffersHook'))  # 在每个 epoch 结束时同步模型缓冲区的钩子
  env_cfg = dict(
      cudnn_benchmark=False,  # 是否启用 cudnn 的基准测试
      mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),  # 设置多进程的参数
      dist_cfg=dict(backend='nccl'))  # 设置分布式环境的参数，也可以设置端口

mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),  # 设置多进程的参数

      window_size=20,  # 默认平滑间隔
      dict(type='LocalVisBackend')]  # 可视化后端的列表
  visualizer = dict(
  log_level = 'INFO'  # 日志级别
              'slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb/'
              'slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb_20220901-e7b65fad.pth')  # 从给定路径加载模型权重作为预训练模型。这不会恢复训练。
  resume = False  # 是否从 `load_from` 中定义的权重恢复训练。如果 `load_from` 为 None，则会从 `work_dir` 中恢复最新的权重。
  ```

### 动作定位的配置系统

我们将模块化设计引入了配置系统中，方便进行各种实验。

- BMN 的示例

resume = False  # 是否从 `load_from` 中定义的权重恢复训练。如果 `load_from` 为 None，则会从 `work_dir` 中恢复最新的权重。
  ```

- BMN 的示例

  为了帮助用户对完整的配置结构和动作定位系统中的模块有一个基本的了解，我们对 BMN 的配置进行了简要注释，具体如下所示。有关每个模块中每个参数的更详细用法和替代方法，请参阅 [API 文档](https://mmaction2.readthedocs.io/en/latest/api.html)。

  ```python
  # 模型设置
      type='BMN',  # 定位器的类名
      temporal_dim=100,  # 每个视频选取的总帧数
      boundary_ratio=0.5,  # 确定视频边界的比率
      num_samples=32,  # 每个 proposal 的采样数量
      num_samples_per_bin=3,  # 每个采样的 bin 的采样数量
      feat_dim=400,  # 特征的维度
      soft_nms_alpha=0.4,  # Soft NMS 的 alpha 值
      soft_nms_low_threshold=0.5,  # Soft NMS 的低阈值
      soft_nms_high_threshold=0.9,  # Soft NMS 的高阈值
      post_process_top_k=100)  # 后处理中的 top-k proposal 数量

  # 数据集设置
  dataset_type = 'ActivityNetDataset'  # 用于训练、验证和测试的数据集类型
  data_root = 'data/activitynet_feature_cuhk/csv_mean_100/'  # 用于训练的数据的根目录

post_process_top_k=100)  # 后处理中的 top-k proposal 数量

  # 数据集设置
  data_root_val = 'data/activitynet_feature_cuhk/csv_mean_100/'  # 用于验证和测试的数据的根目录
  ann_file_train = 'data/ActivityNet/anet_anno_train.json'  # 用于训练的注释文件的路径
  ann_file_val = 'data/ActivityNet/anet_anno_val.json'  # 用于验证的注释文件的路径
  ann_file_test = 'data/ActivityNet/anet_anno_test.json'  # 用于测试的注释文件的路径

      dict(type='LoadLocalizationFeature'),  # 加载定位特征的管道
      dict(type='GenerateLocalizationLabels'),  # 生成定位标签的管道
      dict(
          type='PackLocalizationInputs',  # 打包定位数据
          keys=('gt_bbox'),  # 输入的键
          meta_keys=('video_name'))]  # 输入的元键
      dict(

      dict(
          keys=('gt_bbox'),   # 输入的键
                     'annotations', 'feature_frame'))]  # 输入的元键
      dict(
      batch_size=8,  # 每个单 GPU 训练的批处理大小

batch_size=8,  # 每个单 GPU 训练的批处理大小
          data_prefix=dict(video=data_root),  # 视频路径的前缀
  )

sampler=dict(
          data_prefix=dict(video=data_root_val),  # 视频路径的前缀
  )
      batch_size=1,  # 每个单 GPU 测试的批处理大小
      num_workers=8,  # 每个单 GPU 测试时预取数据的 worker 数量
  )

  # 评估设置
  work_dir = './work_dirs/bmn_400x100_2x8_9e_activitynet_feature/'  # 保存当前实验的模型权重和日志的目录
      type='ANetMetric',
      metric_type='AR@AN',

)

  # 评估设置
      dump_config=dict(
          out=f'{work_dir}/results.json',  # 输出文件的路径
          output_format='json'))  # 输出文件的格式
  test_evaluator = val_evaluator  # 将 test_evaluator 设置为 val_evaluator

  max_epochs = 9  # 训练模型的总 epoch 数量
      max_epochs=max_epochs,  # 总的训练 epoch 数量

  # 学习策略
      dict(
          end=max_epochs,  # 停止更新学习率的步骤
          milestones=[7, ],  # 学习率衰减的步骤
  ]

  # 优化器

end=max_epochs,  # 停止更新学习率的步骤
  ]

  # 优化器
          type='Adam',  # 优化器的名称
          lr=0.001,  # 学习率

  # 运行时设置

interval=20,  # 打印日志的间隔

by_epoch=True)  # 是否使用 epoch 类型格式化日志
  ```

# MMPretrain 模型部署

- [MMPretrain 模型部署](#mmpretrain-模型部署)
    - [安装 mmpretrain](#安装-mmpretrain)

[MMPretrain](https://github.com/open-mmlab/mmpretrain.git) 是基于 Python 的的图像分类工具，属于 [OpenMMLab](https://openmmlab.com)。

## 安装

### 安装 mmpretrain

请参考[官网安装指南](https://github.com/open-mmlab/mmpretrain/tree/main#installation).

```shell

```shell
```

## 模型转换

你可以使用 [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/tree/main/tools/deploy.py) 把 mmpretrain 模型一键式转换为推理后端模型。

以下，我们将演示如何把 `resnet18` 转换为 onnx 模型。

```shell

    --show \
```

转换的关键之一是使用正确的配置文件。项目中已内置了各后端部署[配置文件](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmpretrain)。
文件的命名模式是：

```
```

其中：

在上例中，你也可以把 `resnet18` 转为其他后端模型。比如使用`classification_tensorrt-fp16_dynamic-224x224-224x224.py`，把模型转为 tensorrt-fp16 模型。

```{tip}
```

```{tip}
```

## 模型规范

上例中的`mmdeploy_models/mmpretrain/ort`，结构如下：

```
```

重要的是：

## 模型推理

### 后端模型推理

```python

```

```python
import cv2

```

```python
import cv2

```

## 模型支持列表

# Contributing to MMRotate (To be updated)

- [Workflow](#workflow)
- [Code style](#code-style)
  - [Python](#python)
  - [C++ and CUDA](#c-and-cuda)

1. fork and pull the latest MMRotate repository (MMRotate)
2. checkout a new branch (do not use master branch for PRs)

```{note}
```

### Python

Style configurations can be found in [setup.cfg](https://github.com/open-mmlab/mmediting/blob/1.x/setup.cfg).

The config for a pre-commit hook is stored in [.pre-commit-config](https://github.com/open-mmlab/mmediting/blob/1.x/.pre-commit-config.yaml).

```shell
```

```shell
```

Before you create a PR, make sure that your code lints and is formatted by yapf.
```

# YOLACT

> [YOLACT: Real-time Instance Segmentation](https://arxiv.org/abs/1904.02689)

We present a simple, fully-convolutional model for real-time instance segmentation that achieves 29.8 mAP on MS COCO at 33.5 fps evaluated on a single Titan Xp, which is significantly faster than any previous competitive approach. Moreover, we obtain this result after training on only one GPU. We accomplish this by breaking instance segmentation into two parallel subtasks: (1) generating a set of prototype masks and (2) predicting per-instance mask coefficients. Then we produce instance masks by linearly combining the prototypes with the mask coefficients. We find that because this process doesn't depend on repooling, this approach produces very high-quality masks and exhibits temporal stability for free. Furthermore, we analyze the emergent behavior of our prototypes and show they learn to localize instances on their own in a translation variant manner, despite being fully-convolutional. Finally, we also propose Fast NMS, a drop-in 12 ms faster replacement for standard NMS that only has a marginal performance penalty.

<img src="https://user-images.githubusercontent.com/40661020/144001225-8c62cad7-a330-4f60-873f-61aa17e99223.png"/>
</div>

A simple, fully convolutional model for real-time instance segmentation. This is the code for our paper:

- [YOLACT: Real-time Instance Segmentation](https://arxiv.org/abs/1904.02689)

<!-- - [YOLACT++: Better Real-time Instance Segmentation](https://arxiv.org/abs/1912.06218) -->

For a real-time demo, check out our ICCV video:
[![IMAGE ALT TEXT HERE](https://img.youtube.com/vi/0pMfmo8qfpQ/0.jpg)](https://www.youtube.com/watch?v=0pMfmo8qfpQ)

Here are our YOLACT models along with their FPS on a Titan Xp and mAP on COCO's `val`:

| Image Size | GPU x BS |   Backbone    | \*FPS | mAP  | Weights |                 Configs                  |                                                            Download                                                             |

| :--------: | :------: | :-----------: | :---: | :--: | :-----: | :--------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------: |
|    550     |   1x8    | Resnet50-FPN  | 42.5  | 29.0 |         | [config](./yolact_r50_1xb8-55e_coco.py)  |  [model](https://download.openmmlab.com/mmdetection/v2.0/yolact/yolact_r50_1x8_coco/yolact_r50_1x8_coco_20200908-f38d58df.pth)  |
|    550     |   8x8    | Resnet50-FPN  | 42.5  | 28.4 |         | [config](./yolact_r50_8xb8-55e_coco.py)  |  [model](https://download.openmmlab.com/mmdetection/v2.0/yolact/yolact_r50_8x8_coco/yolact_r50_8x8_coco_20200908-ca34f5db.pth)  |
|    550     |   1x8    | Resnet101-FPN | 33.5  | 30.4 |         | [config](./yolact_r101_1xb8-55e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/yolact/yolact_r101_1x8_coco/yolact_r101_1x8_coco_20200908-4cbe9101.pth) |

\*Note: The FPS is evaluated by the [original implementation](https://github.com/dbolya/yolact). When calculating FPS, only the model inference time is taken into account. Data loading and post-processing operations such as converting masks to RLE code, generating COCO JSON results, image rendering are not included.

All the aforementioned models are trained with a single GPU. It typically takes ~12GB VRAM when using resnet-101 as the backbone. If you want to try multiple GPUs training, you may have to modify the configuration files accordingly, such as adjusting the training schedule and freezing batch norm.

```Shell
# Trains using the resnet-101 backbone with a batch size of 8 on a single GPU.
./tools/dist_train.sh configs/yolact/yolact_r101.py 1
```

## Testing

Please refer to [mmdetection/docs/getting_started.md](https://mmdetection.readthedocs.io/en/latest/1_exist_data_model.html#test-existing-models).

```

## Testing

If you use YOLACT or this code base in your work, please cite

```latex
@inproceedings{yolact-iccv2019,
  author    = {Daniel Bolya and Chong Zhou and Fanyi Xiao and Yong Jae Lee},
  title     = {YOLACT: {Real-time} Instance Segmentation},
  booktitle = {ICCV},
}
```

<!-- For YOLACT++, please cite

```latex
@misc{yolact-plus-arxiv2019,
  title         = {YOLACT++: Better Real-time Instance Segmentation},
  author        = {Daniel Bolya and Chong Zhou and Fanyi Xiao and Yong Jae Lee},
  year          = {2019},
  eprint        = {1912.06218},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV}
}
``` -->

# Convention in MMPretrain

## Model Naming Convention

We follow the below convention to name models. Contributors are advised to follow the same style. The model names are divided into five parts: algorithm info, module information, pretrain information, training information and data information. Logically, different parts are concatenated by underscores `'_'`, and words in the same part are concatenated by dashes `'-'`.

```text
{algorithm info}_{module info}_{pretrain info}_{training info}_{data info}
```

- `algorithm info` (optional): The main algorithm information, it's includes the main training algorithms like MAE, BEiT, etc.
- `module info`:  The module information, it usually includes the backbone name, such as resnet, vit, etc.
- `pretrain info`: (optional): The pretrain model information, such as the pretrain model is trained on ImageNet-21k.
- `training info`: The training information, some training schedule, including batch size, lr schedule, data augment and the like.

- `data info`: The data information, it usually includes the dataset name, input size and so on, such as imagenet, cifar, etc.

### Algorithm information

The main algorithm name to train the model. For example:

- `simclr`
- `mocov2`
- `eva-mae-style`

The model trained by supervised image classification can omit this field.

### Module information

The modules of the model, usually, the backbone must be included in this field, and the neck and head
information can be omitted. For example:

- `resnet50`
- `vit-base-p16`
- `swin-base`

### Pretrain information

If the model is a fine-tuned model from a pre-trained model, we need to record some information of the
pre-trained model. For example:

- The source of the pre-trained model: `fb`, `openai`, etc.
- The method to train the pre-trained model: `clip`, `mae`, `distill`, etc.

- The dataset used for pre-training: `in21k`, `laion2b`, etc. (`in1k` can be omitted.)
- The training duration: `300e`, `1600e`, etc.

Not all information is necessary, only select the necessary information to distinguish different pre-trained
models.

At the end of this field, use a `-pre` as an identifier, like `mae-in21k-pre`.

### Training information

Training schedule, including training type, `batch size`, `lr schedule`, data augment, special loss functions and so on:

- format `{gpu x batch_per_gpu}`, such as `8xb32`

Training type (mainly seen in the transformer network, such as the `ViT` algorithm, which is usually divided into two training type: pre-training and fine-tuning):

- `ft` : configuration file for fine-tuning
- `pt` : configuration file for pretraining

Training recipe. Usually, only the part that is different from the original paper will be marked. These methods will be arranged in the order `{pipeline aug}-{train aug}-{loss trick}-{scheduler}-{epochs}`.

- `coslr-200e` : use cosine scheduler to train 200 epochs
- `autoaug-mixup-lbs-coslr-50e` : use `autoaug`, `mixup`, `label smooth`, `cosine scheduler` to train 50 epochs

If the model is converted from a third-party repository like the official repository, the training information
can be omitted and use a `3rdparty` as an identifier.

### Data information

- `in1k` : `ImageNet1k` dataset, default to use the input image size of 224x224;
- `in21k` : `ImageNet21k` dataset, also called `ImageNet22k` dataset, default to use the input image size of 224x224;
- `in1k-384px` : Indicates that the input image size is 384x384;
- `cifar100`

### Model Name Example

```text

```text
vit-base-p32_clip-openai-pre_3rdparty_in1k
```

- `vit-base-p32`: The module information
- `clip-openai-pre`: The pre-train information.
  - `clip`: The pre-train method is clip.
  - `openai`: The pre-trained model is come from OpenAI.
  - `pre`: The pre-train identifier.
- `3rdparty`: The model is converted from a third-party repository.
- `in1k`: Dataset information. The model is trained from ImageNet-1k dataset and the input size is `224x224`.

```text
beit_beit-base-p16_8xb256-amp-coslr-300e_in1k
```

- `beit`: The algorithm information
- `beit-base`: The module information, since the backbone is a modified ViT from BEiT, the backbone name is
  also `beit`.
- `8xb256-amp-coslr-300e`: The training information.
  - `8xb256`: Use 8 GPUs and the batch size on each GPU is 256.
  - `amp`: Use automatic-mixed-precision training.

also `beit`.
  - `coslr`: Use cosine annealing learning rate scheduler.
  - `300e`: To train 300 epochs.

The naming of the config file is almost the same with the model name, with several difference:

- The training information is necessary, and cannot be `3rdparty`.
- If the config file only includes backbone settings, without neither head settings nor dataset settings. We
  will name it as `{module info}_headless.py`. This kind of config files are usually used for third-party
  pre-trained models on large datasets.

## Checkpoint Naming Convention

The naming of the weight mainly includes the model name, date and hash value.

```text
{model_name}_{date}-{hash}.pth
```

# Test and Train on Standard Datasets

### Test existing models on standard datasets

- CPU

You can use the following commands to test a dataset.

```shell
python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--cfg-options test_evaluator.pklfile_prefix=${RESULT_FILE}]  [--show] [--show-dir ${SHOW_DIR}]

# CPU: disable GPUs and run single-gpu testing script (experimental)

# multi-gpu testing
./tools/dist_test.sh ${CONFIG_FILE} ${CHECKPOINT_FILE} ${GPU_NUM} [--cfg-options test_evaluator.pklfile_prefix=${RESULT_FILE}]  [--show] [--show-dir ${SHOW_DIR}]
```

**Note**:

For now, CPU testing is only supported for SMOKE.

```

**Note**:

- `--show`: If specified, detection results will be plotted in the silient mode. It is only applicable to single GPU testing and used for debugging and visualization. This should be used with `--show-dir`.
- `--show-dir`: If specified, detection results will be plotted on the `***_points.obj` and `***_pred.obj` files in the specified directory. It is only applicable to single GPU testing and used for debugging and visualization. You do NOT need a GUI available in your environment for using this option.

All evaluation related arguments are set in the `test_evaluator` in corresponding dataset configuration. such as
`test_evaluator = dict(type='KittiMetric', ann_file=data_root + 'kitti_infos_val.pkl', pklfile_prefix=None, submission_prefix=None)`

The arguments:

- `type`: The name of the corresponding metric, usually associated with the dataset.
- `ann_file`: The path of annotation file.

- `pklfile_prefix`: An optional argument. The filename of the output results in pickle format. If not specified, the results will not be saved to a file.
- `submission_prefix`: An optional argument. The results will be saved to a file then you can upload it to do the official evaluation.

Examples:

Assume that you have already downloaded the checkpoints to the directory `checkpoints/`.

1. Test VoteNet on ScanNet and save the points and prediction visualization results.

   ```shell
   python tools/test.py configs/votenet/votenet_8xb8_scannet-3d.py \
       checkpoints/votenet_8x8_scannet-3d-18class_20200620_230238-2cea9c3a.pth \
       --show --show-dir ./data/scannet/show_results
   ```

2. Test VoteNet on ScanNet, save the points, prediction, groundtruth visualization results, and evaluate the mAP.

   ```shell

--show --show-dir ./data/scannet/show_results
   ```

   ```shell
   ```

3. Test VoteNet on ScanNet (without saving the test results) and evaluate the mAP.

   ```shell
       checkpoints/votenet_8x8_scannet-3d-18class_20200620_230238-2cea9c3a.pth
   ```

4. Test SECOND on KITTI with 8 GPUs, and evaluate the mAP.

   ```shell
   ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/second/second_hv_secfpn_8xb6-80e_kitti-3d-3class.py \
       checkpoints/hv_second_secfpn_6x8_80e_kitti-3d-3class_20200620_230238-9208083a.pth
   ```

checkpoints/hv_second_secfpn_6x8_80e_kitti-3d-3class_20200620_230238-9208083a.pth
   ```

5. Test PointPillars on nuScenes with 8 GPUs, and generate the json file to be submit to the official evaluation server.

   ```shell
   ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/pointpillars_hv_secfpn_sbn-all_8xb4-2x_nus-3d.py \
       checkpoints/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d_20200620_230405-2fa62f3d.pth \
      --cfg-options 'test_evaluator.jsonfile_prefix=./pointpillars_nuscenes_results'
   ```

   The generated results be under `./pointpillars_nuscenes_results` directory.

6. Test SECOND on KITTI with 8 GPUs, and generate the pkl files and submission data to be submit to the official evaluation server.

   ```shell
       checkpoints/hv_second_secfpn_6x8_80e_kitti-3d-3class_20200620_230238-9208083a.pth \

checkpoints/hv_second_secfpn_6x8_80e_kitti-3d-3class_20200620_230238-9208083a.pth \
       --cfg-options 'test_evaluator.pklfile_prefix=./second_kitti_results' 'submission_prefix=./second_kitti_results'
   ```

   The generated results be under `./second_kitti_results` directory.

7. Test PointPillars on Lyft with 8 GPUs, generate the pkl files and make a submission to the leaderboard.

   ```shell
   ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/hv_pointpillars_fpn_sbn-2x8_2x_lyft-3d.py \
       checkpoints/hv_pointpillars_fpn_sbn-2x8_2x_lyft-3d_latest.pth \
       --cfg-options 'test_evaluator.jsonfile_prefix=results/pp_lyft/results_challenge' \
       'test_evaluator.csv_savepath=results/pp_lyft/results_challenge.csv' \
       'test_evaluator.pklfile_prefix=results/pp_lyft/results_challenge.pkl'
   ```

'test_evaluator.csv_savepath=results/pp_lyft/results_challenge.csv' \
   ```

   **Notice**: To generate submissions on Lyft, `csv_savepath` must be given in the `--cfg-options`. After generating the csv file, you can make a submission with kaggle commands given on the [website](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/submit).

   Note that in the [config of Lyft dataset](../../configs/_base_/datasets/lyft-3d.py), the value of `ann_file` keyword in `test` is `'lyft_infos_test.pkl'`, which is the official test set of Lyft without annotation. To test on the validation set, please change this to `'lyft_infos_val.pkl'`.

8. Test PointPillars on waymo with 8 GPUs, and evaluate the mAP with waymo metrics.

   ```shell
   ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymo-3d-car.py  \

```shell
       checkpoints/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car_latest.pth \
       --cfg-options 'test_evaluator.pklfile_prefix=results/waymo-car/kitti_results' \
       'test_evaluator.submission_prefix=results/waymo-car/kitti_results'
   ```

**Notice**: For evaluation on waymo, please follow the [instruction](https://github.com/waymo-research/waymo-open-dataset/blob/master/docs/quick_start.md/) to build the binary file `compute_detection_metrics_main` for metrics computation and put it into `mmdet3d/core/evaluation/waymo_utils/`.(Sometimes when using bazel to build `compute_detection_metrics_main`, an error `'round' is not a member of 'std'` may appear. We just need to remove the `std::` before `round` in that file.) `pklfile_prefix` should be given in the `--eval-options` for the bin file generation. For metrics, `waymo` is the recommended official evaluation prototype. Currently, evaluating with choice `kitti` is adapted from KITTI and the results for each difficulty are not exactly the same as the definition of KITTI. Instead, most of objects are marked with difficulty 0 currently, which will be fixed in the future. The reasons of its instability include the large computation for evaluation, the lack of occlusion and truncation in the converted data, different definition of difficulty and different methods of computing average precision.

9. Test PointPillars on waymo with 8 GPUs, generate the bin files and make a submission to the leaderboard.

   ```shell
   ```

   **Notice**: After generating the bin file, you can simply build the binary file `create_submission` and use them to create a submission file by following the [instruction](https://github.com/waymo-research/waymo-open-dataset/blob/master/docs/quick_start.md/). For evaluation on the validation set with the eval server, you can also use the same way to generate a submission.

## Train predefined models on standard datasets

MMDetection3D implements distributed training and non-distributed training,
which uses `MMDistributedDataParallel` and `MMDataParallel` respectively.

All outputs (log files and checkpoints) will be saved to the working directory,

By default we evaluate the model on the validation set after each epoch, you can change the evaluation interval by adding the interval argument in the training config.

```python
train_cfg = dict(type='EpochBasedTrainLoop', val_interval=1)  # This evaluate the model per 12 epoch.
```

**Important**: The default learning rate in config files is for 8 GPUs and the exact batch size is marked by the config's file name, e.g. '2xb8' means 2 samples per GPU using 8 GPUs.

```

According to the [Linear Scaling Rule](https://arxiv.org/abs/1706.02677), you need to set the learning rate proportional to the batch size if you use different GPUs or images per GPU, e.g., lr=0.01 for 4 GPUs * 2 img/gpu and lr=0.08 for 16 GPUs * 4 img/gpu. However, since most of the models in this repo use ADAM rather than SGD for optimization, the rule may not hold and users need to tune the learning rate by themselves.

```shell
```

If you want to specify the working directory in the command, you can add an argument `--work-dir ${YOUR_WORK_DIR}`.

### Training with CPU (experimental)

```

```shell
```

And then run the script of train with a single GPU.

**Note**:

For now, most of the point cloud related algorithms rely on 3D CUDA op, which can not be trained on CPU. Some monocular 3D object detection algorithms, like FCOS3D and SMOKE can be trained on CPU. We do not recommend users to use CPU for training because it is too slow. We support this feature to allow users to debug certain models on machines without GPU for convenience.

```shell
```

- `--cfg-options 'Key=value'`: Override some settings in the used config.

```shell
```

If you run MMDetection3D on a cluster managed with [slurm](https://slurm.schedmd.com/), you can use the script `slurm_train.sh`. (This script also supports single machine training.)

```shell
```

Here is an example of using 16 GPUs to train Mask R-CNN on the dev partition.

```shell
GPUS=16 ./tools/slurm_train.sh dev pp_kitti_3class configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py /nfs/xxxx/pp_kitti_3class
```

You can check [slurm_train.sh](https://github.com/open-mmlab/mmdetection/blob/master/tools/slurm_train.sh) for full arguments and environment variables.

```

```shell
NNODES=2 NODE_RANK=0 PORT=$MASTER_PORT MASTER_ADDR=$MASTER_ADDR ./tools/dist_train.sh $CONFIG $GPUS
```

```shell
NNODES=2 NODE_RANK=1 PORT=$MASTER_PORT MASTER_ADDR=$MASTER_ADDR ./tools/dist_train.sh $CONFIG $GPUS
```

```shell

```shell
```

If you use launch training jobs with Slurm, there are two ways to specify the ports.

   ```shell
   CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config1.py ${WORK_DIR} --cfg-options 'env_cfg.dist_cfg.port=29500'
   CUDA_VISIBLE_DEVICES=4,5,6,7 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config2.py ${WORK_DIR} --cfg-options 'env_cfg.dist_cfg.port=29501'
   ```

CUDA_VISIBLE_DEVICES=4,5,6,7 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config2.py ${WORK_DIR} --cfg-options 'env_cfg.dist_cfg.port=29501'
   ```

2. Modify the config files (usually the 6th line from the bottom in config files) to set different communication ports.

   In `config1.py`,

   ```python
   env_cfg = dict(
       dist_cfg=dict(backend='nccl', port=29500)
   )
   ```

   In `config2.py`,

   ```python
       dist_cfg=dict(backend='nccl', port=29501)
   )
   ```

   ```shell
   ```

# InstructBLIP

```sh
git clone https://github.com/salesforce/LAVIS.git
cd ./LAVIS
```

### Modify the config

Modify the config of InstructBlip, like model path of LLM and Qformer.

Then update `tasks.py` like the following code snippet.

```python

    from .instructblip.instructblip_mmbench import (instruct_blip_dataloader,
                                                    instruct_blip_evaluator,
                                                    instruct_blip_load_from,
                                                    instruct_blip_model)

models = [instruct_blip_model]
datasets = [instruct_blip_dataloader]
evaluators = [instruct_blip_evaluator]
load_froms = [instruct_blip_load_from]
num_gpus = 8
num_procs = 8
launcher = 'pytorch'  # or 'slurm'
```

#### Slurm

```sh
cd $root
```

```

#### Slurm

```sh
cd $root
```

```sh
cd $root
python run.py configs/multimodal/tasks.py --mm-eval 
```

# 准备 Jester

## 简介

```BibTeX
}
```

用户可以参照数据集 [官网](https://developer.qualcomm.com/software/ai-datasets/jester)，获取数据集相关的基本信息。
在准备数据集前，请确保命令行当前路径为 `$MMACTION2/tools/data/jester/`。

首先，用户需要在 [官网](https://developer.qualcomm.com/software/ai-datasets/jester) 完成注册，才能下载标注文件。下载好的标注文件需要放在 `$MMACTION2/data/jester/annotations` 文件夹下。

[jester 官网](https://developer.qualcomm.com/software/ai-datasets/jester) 并未提供原始视频文件，只提供了对原视频文件进行抽取得到的 RGB 帧，用户可在 [jester 官网](https://developer.qualcomm.com/software/ai-datasets/jester) 直接下载。

将下载好的压缩文件放在 `$MMACTION2/data/jester/` 文件夹下，并使用以下脚本进行解压。

```shell

```shell
```

```python
    val=dict(
```

如果用户只想使用 RGB 帧训练，则该部分是 **可选项**。

ann_file=ann_file_test,
```

```shell
```

```shell
```

```shell
```

```shell
```

在完成所有 Jester 数据集准备流程后，

```shell
```

在整个 MMAction2 文件夹下，Jester 的文件结构如下：

```
mmaction2
├── tools
├── data
│   |   |   ├──...
│   |   |   ├── 1
│   |   |   ├── 2
│   |   |   ├── ...

```

关于对 jester 进行训练和验证，请参考 [训练和测试教程](/docs/en/user_guides/train_test.md)。

# 2D Hand Pose Estimation

2D hand pose estimation is defined as the task of detecting the poses (or keypoints) of the hand from an input image.

Normally, the input images are cropped hand images, where the hand locates at the center;

Please follow [DATA Preparation](/docs/en/dataset_zoo/2d_hand_keypoint.md) to prepare data.

## Demo

Please follow [Demo](/demo/docs/en/2d_hand_demo.md) to run demos.

<img src="https://user-images.githubusercontent.com/11788150/109098558-8c54db00-775c-11eb-8966-85df96b23dc5.gif" width="600px" alt><br>

**KIE**

```json
{
    [
      {
        "label": 1
      },
      {
      }
    ], //...
}

// Open Set
{
    [
      {
        "edge": 1
      },
      {
        "edge": 1
      }
    ]
}
```

# CCNet

> [CCNet: Criss-Cross Attention for Semantic Segmentation](https://arxiv.org/abs/1811.11721)

<a href="https://github.com/speedinghzl/CCNet">Official Repo</a>

Contextual information is vital in visual understanding problems, such as semantic segmentation and object detection. We propose a Criss-Cross Network (CCNet) for obtaining full-image contextual information in a very effective and efficient way. Concretely, for each pixel, a novel criss-cross attention module harvests the contextual information of all the pixels on its criss-cross path. By taking a further recurrent operation, each pixel can finally capture the full-image dependencies. Besides, a category consistent loss is proposed to enforce the criss-cross attention module to produce more discriminative features. Overall, CCNet is with the following merits: 1) GPU memory friendly. Compared with the non-local block, the proposed recurrent criss-cross attention module requires 11x less GPU memory usage. 2) High computational efficiency. The recurrent criss-cross attention significantly reduces FLOPs by about 85% of the non-local block. 3) The state-of-the-art performance. We conduct extensive experiments on semantic segmentation benchmarks including Cityscapes, ADE20K, human parsing benchmark LIP, instance segmentation benchmark COCO, video segmentation benchmark CamVid. In particular, our CCNet achieves the mIoU scores of 81.9%, 45.76% and 55.47% on the Cityscapes test set, the ADE20K validation set and the LIP validation set respectively, which are the new state-of-the-art results. The source codes are available at [this https URL](https://github.com/speedinghzl/CCNet).

<img src="https://user-images.githubusercontent.com/24582831/142899159-b329c12a-0fde-44df-8718-def6cfb004e4.png" width="70%"/>
</div>

| CCNet  | R-50-D8  | 512x1024  |   40000 | 6        | 3.32           | V100   | 77.76 |         78.87 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ccnet/ccnet_r50-d8_4xb2-40k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r50-d8_512x1024_40k_cityscapes/ccnet_r50-d8_512x1024_40k_cityscapes_20200616_142517-4123f401.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r50-d8_512x1024_40k_cityscapes/ccnet_r50-d8_512x1024_40k_cityscapes_20200616_142517.log.json)     |

| CCNet  | R-101-D8 | 512x1024  |   40000 | 9.5      | 2.31           | V100   | 76.35 |         78.19 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ccnet/ccnet_r101-d8_4xb2-40k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r101-d8_512x1024_40k_cityscapes/ccnet_r101-d8_512x1024_40k_cityscapes_20200616_142540-a3b84ba6.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r101-d8_512x1024_40k_cityscapes/ccnet_r101-d8_512x1024_40k_cityscapes_20200616_142540.log.json) |

| CCNet  | R-50-D8  | 769x769   |   40000 | 6.8      | 1.43           | V100   | 78.46 |         79.93 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ccnet/ccnet_r50-d8_4xb2-40k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r50-d8_769x769_40k_cityscapes/ccnet_r50-d8_769x769_40k_cityscapes_20200616_145125-76d11884.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r50-d8_769x769_40k_cityscapes/ccnet_r50-d8_769x769_40k_cityscapes_20200616_145125.log.json)         |

| CCNet  | R-101-D8 | 769x769   |   40000 | 10.7     | 1.01           | V100   | 76.94 |         78.62 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ccnet/ccnet_r101-d8_4xb2-40k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r101-d8_769x769_40k_cityscapes/ccnet_r101-d8_769x769_40k_cityscapes_20200617_101428-4f57c8d0.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r101-d8_769x769_40k_cityscapes/ccnet_r101-d8_769x769_40k_cityscapes_20200617_101428.log.json)     |

| CCNet  | R-50-D8  | 512x1024  |   80000 | -        | -              | V100   | 79.03 |         80.16 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ccnet/ccnet_r50-d8_4xb2-80k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r50-d8_512x1024_80k_cityscapes/ccnet_r50-d8_512x1024_80k_cityscapes_20200617_010421-869a3423.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r50-d8_512x1024_80k_cityscapes/ccnet_r50-d8_512x1024_80k_cityscapes_20200617_010421.log.json)     |

| CCNet  | R-101-D8 | 512x1024  |   80000 | -        | -              | V100   | 78.87 |         79.90 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ccnet/ccnet_r101-d8_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r101-d8_512x1024_80k_cityscapes/ccnet_r101-d8_512x1024_80k_cityscapes_20200617_203935-ffae8917.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r101-d8_512x1024_80k_cityscapes/ccnet_r101-d8_512x1024_80k_cityscapes_20200617_203935.log.json) |

| CCNet  | R-50-D8  | 769x769   |   80000 | -        | -              | V100   | 79.29 |         81.08 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ccnet/ccnet_r50-d8_4xb2-80k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r50-d8_769x769_80k_cityscapes/ccnet_r50-d8_769x769_80k_cityscapes_20200617_010421-73eed8ca.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r50-d8_769x769_80k_cityscapes/ccnet_r50-d8_769x769_80k_cityscapes_20200617_010421.log.json)         |

| CCNet  | R-101-D8 | 769x769   |   80000 | -        | -              | V100   | 79.45 |         80.66 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ccnet/ccnet_r101-d8_4xb2-80k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r101-d8_769x769_80k_cityscapes/ccnet_r101-d8_769x769_80k_cityscapes_20200618_011502-ad3cd481.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r101-d8_769x769_80k_cityscapes/ccnet_r101-d8_769x769_80k_cityscapes_20200618_011502.log.json)     |

### ADE20K

| CCNet  | R-50-D8  | 512x512   |   80000 | 8.8      | 20.89          | V100   | 41.78 |         42.98 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ccnet/ccnet_r50-d8_4xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r50-d8_512x512_80k_ade20k/ccnet_r50-d8_512x512_80k_ade20k_20200615_014848-aa37f61e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r50-d8_512x512_80k_ade20k/ccnet_r50-d8_512x512_80k_ade20k_20200615_014848.log.json)         |

| CCNet  | R-101-D8 | 512x512   |   80000 | 12.2     | 14.11          | V100   | 43.97 |         45.13 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ccnet/ccnet_r101-d8_4xb4-80k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r101-d8_512x512_80k_ade20k/ccnet_r101-d8_512x512_80k_ade20k_20200615_014848-1f4929a3.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r101-d8_512x512_80k_ade20k/ccnet_r101-d8_512x512_80k_ade20k_20200615_014848.log.json)     |

| CCNet  | R-50-D8  | 512x512   |  160000 | -        | -              | V100   | 42.08 |         43.13 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ccnet/ccnet_r50-d8_4xb4-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r50-d8_512x512_160k_ade20k/ccnet_r50-d8_512x512_160k_ade20k_20200616_084435-7c97193b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r50-d8_512x512_160k_ade20k/ccnet_r50-d8_512x512_160k_ade20k_20200616_084435.log.json)     |

| CCNet  | R-101-D8 | 512x512   |  160000 | -        | -              | V100   | 43.71 |         45.04 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ccnet/ccnet_r101-d8_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r101-d8_512x512_160k_ade20k/ccnet_r101-d8_512x512_160k_ade20k_20200616_000644-e849e007.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r101-d8_512x512_160k_ade20k/ccnet_r101-d8_512x512_160k_ade20k_20200616_000644.log.json) |

| CCNet  | R-50-D8  | 512x512   |   20000 | 6        | 20.45          | V100   | 76.17 |         77.51 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ccnet/ccnet_r50-d8_4xb4-20k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r50-d8_512x512_20k_voc12aug/ccnet_r50-d8_512x512_20k_voc12aug_20200617_193212-fad81784.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r50-d8_512x512_20k_voc12aug/ccnet_r50-d8_512x512_20k_voc12aug_20200617_193212.log.json)     |

| CCNet  | R-101-D8 | 512x512   |   20000 | 9.5      | 13.64          | V100   | 77.27 |         79.02 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ccnet/ccnet_r101-d8_4xb4-20k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r101-d8_512x512_20k_voc12aug/ccnet_r101-d8_512x512_20k_voc12aug_20200617_193212-0007b61d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r101-d8_512x512_20k_voc12aug/ccnet_r101-d8_512x512_20k_voc12aug_20200617_193212.log.json) |

| CCNet  | R-50-D8  | 512x512   |   40000 | -        | -              | V100   | 75.96 |         77.04 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ccnet/ccnet_r50-d8_4xb4-40k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r50-d8_512x512_40k_voc12aug/ccnet_r50-d8_512x512_40k_voc12aug_20200613_232127-c2a15f02.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r50-d8_512x512_40k_voc12aug/ccnet_r50-d8_512x512_40k_voc12aug_20200613_232127.log.json)     |

| CCNet  | R-101-D8 | 512x512   |   40000 | -        | -              | V100   | 77.87 |         78.90 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/ccnet/ccnet_r101-d8_4xb4-40k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r101-d8_512x512_40k_voc12aug/ccnet_r101-d8_512x512_40k_voc12aug_20200613_232127-c30da577.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/ccnet/ccnet_r101-d8_512x512_40k_voc12aug/ccnet_r101-d8_512x512_40k_voc12aug_20200613_232127.log.json) |

```bibtex
@article{huang2018ccnet,
    title={CCNet: Criss-Cross Attention for Semantic Segmentation},
    author={Huang, Zilong and Wang, Xinggang and Huang, Lichao and Huang, Chang and Wei, Yunchao and Liu, Wenyu},
}
```

# TVM 特性支持

MMDeploy 已经将 TVM 集成到模型转换工具以及 SDK 当中。可用的特性包括：

- AutoTVM 调优器
- Ansor 调优器
- Graph Executor 运行时
- Virtual Machine 运行时

# OpenVINO Support

This tutorial is based on Linux systems like Ubuntu-18.04.

It is recommended to create a virtual environment for the project.

### Install python package

Install [OpenVINO](https://docs.openvino.ai/2022.3/get_started.html). It is recommended to use the installer or install using pip.
Installation example using [pip](https://pypi.org/project/openvino-dev/):

```bash
pip install openvino-dev[onnx]==2022.3.0
```

### Download OpenVINO runtime for SDK (Optional)

If you want to use OpenVINO in SDK, you need install OpenVINO with [install_guides](https://docs.openvino.ai/2022.3/openvino_docs_install_guides_installing_openvino_from_archive_linux.html#installing-openvino-runtime).
Take `openvino==2022.3.0` as example:

```bash
wget https://storage.openvinotoolkit.org/repositories/openvino/packages/2022.3/linux/l_openvino_toolkit_ubuntu20_2022.3.0.9052.9752fafe8eb_x86_64.tgz
tar xzf ./l_openvino_toolkit*.tgz
cd l_openvino*

export InferenceEngine_DIR=$pwd/runtime/cmake
bash ./install_dependencies/install_openvino_dependencies.sh
```

### Build mmdeploy SDK with OpenVINO (Optional)

Install MMDeploy following the [instructions](../01-how-to-build/build_from_source.md).

```bash
cmake -DMMDEPLOY_TARGET_DEVICES='cpu' -DMMDEPLOY_TARGET_BACKENDS=openvino -DInferenceEngine_DIR=${InferenceEngine_DIR} ..
```

To work with models from [MMDetection](https://mmdetection.readthedocs.io/en/3.x/get_started.html), you may need to install it additionally.

## Usage

You could follow the instructions of tutorial [How to convert model](../02-how-to-run/convert_model.md)

Example:

```bash

## Usage

Example:

```bash
    configs/mmdet/detection/detection_openvino_static-300x300.py \
    /mmdetection_dir/mmdetection/configs/ssd/ssd300_coco.py \
    /tmp/snapshots/ssd300_coco_20210803_015428-d231a06e.pth \
```

## List of supported models exportable to OpenVINO from MMDetection

The table below lists the models that are guaranteed to be exportable to OpenVINO from MMDetection.

|     Model name     |                                  Config                                   | Dynamic Shape |
| :----------------: | :-----------------------------------------------------------------------: | :-----------: |
|        ATSS        |                  `configs/atss/atss_r50_fpn_1x_coco.py`                   |       Y       |

| Cascade Mask R-CNN |        `configs/cascade_rcnn/cascade_mask_rcnn_r50_fpn_1x_coco.py`        |       Y       |
|   Cascade R-CNN    |          `configs/cascade_rcnn/cascade_rcnn_r50_fpn_1x_coco.py`           |       Y       |
|    Faster R-CNN    |           `configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py`            |       Y       |
|        FCOS        | `configs/fcos/fcos_x101_64x4d_fpn_gn-head_mstrain_640-800_4x2_2x_coco.py` |       Y       |
|      FoveaBox      |             `configs/foveabox/fovea_r50_fpn_4x4_1x_coco.py `              |       Y       |
|        FSAF        |                  `configs/fsaf/fsaf_r50_fpn_1x_coco.py`                   |       Y       |
|     Mask R-CNN     |             `configs/mask_rcnn/mask_rcnn_r50_fpn_1x_coco.py`              |       Y       |

|     RetinaNet      |             `configs/retinanet/retinanet_r50_fpn_1x_coco.py`              |       Y       |
|        SSD         |                       `configs/ssd/ssd300_coco.py`                        |       Y       |
|       YOLOv3       |            `configs/yolo/yolov3_d53_mstrain-608_273e_coco.py`             |       Y       |
|       YOLOX        |                `configs/yolox/yolox_tiny_8x8_300e_coco.py`                |       Y       |
| Faster R-CNN + DCN |         `configs/dcn/faster_rcnn_r50_fpn_dconv_c3-c5_1x_coco.py`          |       Y       |
|       VFNet        |                 `configs/vfnet/vfnet_r50_fpn_1x_coco.py`                  |       Y       |

Notes:

- Custom operations from OpenVINO use the domain `org.openvinotoolkit`.
- For faster work in OpenVINO in the Faster-RCNN, Mask-RCNN, Cascade-RCNN, Cascade-Mask-RCNN models

Notes:

  the RoiAlign operation is replaced with the [ExperimentalDetectronROIFeatureExtractor](https://docs.openvino.ai/2022.3/openvino_docs_ops_detection_ExperimentalDetectronROIFeatureExtractor_6.html) operation in the ONNX graph.
- Models "VFNet" and "Faster R-CNN + DCN" use the custom "DeformableConv2D" operation.

## Deployment config

With the deployment config, you can specify additional options for the Model Optimizer.
To do this, add the necessary parameters to the `backend_config.mo_options` in the fields `args` (for parameters with values) and `flags` (for flags).

Example:

```python
    mo_options=dict(
        args=dict({
            '--mean_values': [0, 0, 0],
            '--scale_values': [255, 255, 255],
            '--data_type': 'FP32',
        }),

        }),
        flags=['--disable_fusing'],
    )
)
```

Information about the possible parameters for the Model Optimizer can be found in the [documentation](https://docs.openvino.ai/latest/openvino_docs_MO_DG_prepare_model_convert_model_Converting_Model.html).

- ImportError: libpython3.7m.so.1.0: cannot open shared object file: No such file or directory

  To resolve missing external dependency on Ubuntu\*, execute the following command:

  ```bash
  sudo apt-get install libpython3.7
  ```

Before reading this tutorial, it is recommended to read MMEngine's [Visualization](https://github.com/open-mmlab/mmengine/blob/main/docs/en/advanced_tutorials/visualization.md) documentation to get a first glimpse of the `Visualizer` definition and usage.

Based on MMEngine's Visualizer, MMDet comes with a variety of pre-built visualization tools that can be used by the user by simply modifying the following configuration files.

- The `tools/analysis_tools/browse_dataset.py` script provides a dataset visualization function that draws images and corresponding annotations after Data Transforms, as described in [`browse_dataset.py`](useful_tools.md#Visualization).

- The `VisualizerHook` is implemented in MMDet, which uses the `Visualizer` to visualize or store the prediction results of the validation or prediction phase into the backend set by the `Visualizer`, so by modifying the `Visualizer` backend in the configuration file, for example, to ` TensorBoardVISBackend` or `WandbVISBackend`, you can implement storing the predicted images to `TensorBoard` or `Wandb`.

Thanks to the use of the registration mechanism, in MMDet we can set the behavior of the `Visualizer` by modifying the configuration file. Usually, we define the default configuration for the visualizer in `configs/_base_/default_runtime.py`, see [configuration tutorial](config.md) for details.

```Python
```

- Users can directly use `DetLocalVisualizer` to visualize labels or predictions for support tasks.
- MMDet sets the visualization backend `vis_backend` to the local visualization backend `LocalVisBackend` by default, saving all visualization results and other training information in a local folder.

## Storage

## Storage

MMDet uses the local visualization backend [`LocalVisBackend`](mmengine.visualization.LocalVisBackend) by default, and the model loss, learning rate, model evaluation accuracy and visualization The information stored in `VisualizerHook` and `LoggerHook`, including loss, learning rate, evaluation accuracy will be saved to the `{work_dir}/{config_name}/{time}/{vis_data}` folder by default. In addition, MMDet also supports other common visualization backends, such as `TensorboardVisBackend` and `WandbVisBackend`, and you only need to change the `vis_backends` type in the configuration file to the corresponding visualization backend. For example, you can store data to `TensorBoard` and `Wandb` by simply inserting the following code block into the configuration file.

```Python

```Python
```

## Plot

MMDet mainly uses [`DetVisualizationHook`](mmdet.engine.hooks.DetVisualizationHook) to plot the prediction results of validation and test, by default `DetVisualizationHook` is off, and the default configuration is as follows.

```Python
```

The following table shows the parameters supported by `DetVisualizationHook`.

|    draw    |      The DetVisualizationHook is turned on and off by the enable parameter, which is the default state.       |

If you want to enable `DetVisualizationHook` related functions and configurations during training or testing, you only need to modify the configuration, take `configs/rtmdet/rtmdet_tiny_8xb32-300e_coco.py` as an example, draw annotations and predictions at the same time, and display the images, the configuration can be modified as follows

```Python
```

```Python
```

</div>

```Shell

```

```

</div>

我们在 `tools/` 文件夹路径下提供了许多有用的工具。

## 日志分析

给定一个训练的日志文件，您可以绘制出 loss/mAP 曲线。首先需要运行 `pip install seaborn` 安装依赖包。

![loss曲线图](../../../resources/loss_curve.png)

```shell
python tools/analysis_tools/analyze_logs.py plot_curve [--keys ${KEYS}] [--title ${TITLE}] [--legend ${LEGEND}] [--backend ${BACKEND}] [--style ${STYLE}] [--out ${OUT_FILE}] [--mode ${MODE}] [--interval ${INTERVAL}]
```

**注意**: 如果您想绘制的指标是在验证阶段计算得到的，您需要添加一个标志 `--mode eval` ，如果您每经过一个 `${INTERVAL}` 的间隔进行评估，您需要增加一个参数 `--interval ${INTERVAL}`。

示例：

- 绘制出某次运行的分类 loss。

  ```shell
  ```

- 绘制出某次运行的分类和回归 loss，并且保存图片为 pdf 格式。

  ```shell
  ```

- 在同一张图片中比较两次运行的 bbox mAP。

  ```shell
  # 根据 Car_3D_moderate_strict 在 KITTI 上评估 PartA2 和 second。

```

  ```shell
  python tools/analysis_tools/analyze_logs.py plot_curve tools/logs/PartA2.log.json tools/logs/second.log.json --keys KITTI/Car_3D_moderate_strict --legend PartA2 second --mode eval --interval 1
  # 根据 Car_3D_moderate_strict 在 KITTI 上分别对车和 3 类评估 PointPillars。
  python tools/analysis_tools/analyze_logs.py plot_curve tools/logs/pp-3class.log.json tools/logs/pp.log.json --keys KITTI/Car_3D_moderate_strict --legend pp-3class pp --mode eval --interval 2
  ```

您也能计算平均训练速度。

```shell
python tools/analysis_tools/analyze_logs.py cal_train_time log.json [--include-outliers]
```

预期输出应该如下所示。

```
-----Analyze train time of work_dirs/some_exp/20190611_192040.log.json-----
slowest epoch 11, average time is 1.2024
fastest epoch 1, average time is 1.1909
time std over epochs is 0.0028
average iter time: 1.1959 s/iter
```

&#8195;

## 模型部署

```

&#8195;

## 模型部署

**注意**：此工具仍然处于试验阶段，目前只有 SECOND 支持用 [`TorchServe`](https://pytorch.org/serve/) 部署，我们将会在未来支持更多的模型。

为了使用 [`TorchServe`](https://pytorch.org/serve/) 部署 `MMDetection3D` 模型，您可以遵循以下步骤：

### 1. 将模型从 MMDetection3D 转换到 TorchServe

```shell
python tools/deployment/mmdet3d2torchserve.py ${CONFIG_FILE} ${CHECKPOINT_FILE} \
```

**Note**: ${MODEL_STORE} 需要为文件夹的绝对路径。

### 2. 构建 `mmdet3d-serve` 镜像

```shell
docker build -t mmdet3d-serve:latest docker/serve/
```

### 3. 运行 `mmdet3d-serve`

查看官网文档来 [使用 docker 运行 TorchServe](https://github.com/pytorch/serve/blob/master/docker/README.md#running-torchserve-in-a-production-docker-environment)。

```

为了在 GPU 上运行，您需要安装 [nvidia-docker](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html)。您可以忽略 `--gpus` 参数，从而在 CPU 上运行。

例子：

```shell
--cpus 8 \
mmdet3d-serve:latest
```

[阅读文档](https://github.com/pytorch/serve/blob/072f5d088cce9bb64b2a18af065886c9b01b317b/docs/rest_api.md/) 关于 Inference (8080), Management (8081) and Metrics (8082) 接口。

### 4. 测试部署

您可以使用 `test_torchserver.py` 进行部署， 同时比较 torchserver 和 pytorch 的结果。

```shell
```

例子:

```

例子:

```shell
python tools/deployment/test_torchserver.py demo/data/kitti/kitti_000008.bin configs/second/hv_second_secfpn_6x8_80e_kitti-3d-car.py checkpoints/hv_second_secfpn_6x8_80e_kitti-3d-car_20200620_230238-393f000c.pth second
```

&#8195;

# 模型复杂度

您可以使用 MMDetection 中的 `tools/analysis_tools/get_flops.py` 这个脚本文件，基于 [flops-counter.pytorch](https://github.com/sovrasov/flops-counter.pytorch) 计算一个给定模型的计算量 (FLOPS) 和参数量 (params)。

```shell
```

您将会得到如下的结果：

```text
Input shape: (4000, 4)
Flops: 5.78 GFLOPs
Params: 953.83 k
```

**注意**：此工具仍然处于试验阶段，我们不能保证数值是绝对正确的。您可以将结果用于简单的比较，但在写技术文档报告或者论文之前您需要再次确认一下。

```

1. 计算量 (FLOPs) 和输入形状有关，但是参数量 (params) 则和输入形状无关。默认的输入形状为 (1, 40000, 4)。
2. 一些运算操作不计入计算量 (FLOPs)，比如说像GN和定制的运算操作，详细细节请参考 [`mmcv.cnn.get_model_complexity_info()`](https://github.com/open-mmlab/mmcv/blob/master/mmcv/cnn/utils/flops_counter.py)。
3. 我们现在仅仅支持单模态输入（点云或者图片）的单阶段模型的计算量 (FLOPs) 计算，我们将会在未来支持两阶段和多模态模型的计算。

&#8195;

## 模型转换

### RegNet 模型转换到 MMDetection

`tools/model_converters/regnet2mmdet.py` 将 pycls 预训练 RegNet 模型中的键转换为 MMDetection 风格。

```shell
```

### Detectron ResNet 转换到 Pytorch

MMDetection 中的 `tools/detectron2pytorch.py` 能够把原始的 detectron 中预训练的 ResNet 模型的键转换为 PyTorch 风格。

```shell
python tools/detectron2pytorch.py ${SRC} ${DST} ${DEPTH} [-h]
```

### 准备要发布的模型

```shell
```

`tools/model_converters/publish_model.py` 帮助用户准备他们用于发布的模型。

在您上传一个模型到云服务器 (AWS) 之前，您需要做以下几步：

1. 将模型权重转换为 CPU 张量
2. 删除记录优化器状态 (optimizer states) 的相关信息
3. 计算检查点 (checkpoint) 文件的哈希编码 (hash id) 并且把哈希编码加到文件名里

```shell
```

例如，

```shell
```

最终的输出文件名将会是 `faster_rcnn_r50_fpn_1x_20190801-{hash id}.pth`。

&#8195;

# 数据集转换

`tools/dataset_converters/` 包含转换数据集为其他格式的一些工具。其中大多数转换数据集为基于 pickle 的信息文件，比如 KITTI，nuscense 和 lyft。Waymo 转换器被用来重新组织 waymo 原始数据为 KITTI 风格。用户能够参考它们了解我们转换数据格式的方法。将它们修改为 nuImages 转换器等脚本也很方便。

为了转换 nuImages 数据集为 COCO 格式，请使用下面的指令：

```shell

```shell
```

- `--data-root`: 数据集的根目录，默认为 `./data/nuimages`。
- `--version`: 数据集的版本，默认为 `v1.0-mini`。要获取完整数据集，请使用 `--version v1.0-train v1.0-val v1.0-mini`。
- `--out-dir`: 注释和语义掩码的输出目录，默认为 `./data/nuimages/annotations/`。
- `--nproc`: 数据准备的进程数，默认为 `4`。由于图片是并行处理的，更大的进程数目能够减少准备时间。
- `--extra-tag`: 注释的额外标签，默认为 `nuimages`。这可用于将不同时间处理的不同注释分开以供研究。

更多的数据准备细节参考 [doc](https://mmdetection3d.readthedocs.io/zh_CN/latest/data_preparation.html)，nuImages 数据集的细节参考 [README](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/nuimages/README.md/)。

&#8195;

# 其他内容

## 打印完整的配置文件

`tools/misc/print_config.py` 逐字打印整个配置文件，展开所有的导入。

```shell
```

# 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks

> [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://arxiv.org/abs/1904.08755)

In many robotics and VR/AR applications, 3D-videos are readily-available sources of input (a continuous sequence of depth images, or LIDAR scans). However, those 3D-videos are processed frame-by-frame either through 2D convnets or 3D perception algorithms. In this work, we propose 4-dimensional convolutional neural networks for spatio-temporal perception that can directly process such 3D-videos using high-dimensional convolutions. For this, we adopt sparse tensors and propose the generalized sparse convolution that encompasses all discrete convolutions. To implement the generalized sparse convolution, we create an open-source auto-differentiation library for sparse tensors that provides extensive functions for high-dimensional convolutional neural networks. We create 4D spatio-temporal convolutional neural networks using the library and validate them on various 3D semantic segmentation benchmarks and proposed 4D datasets for 3D-video perception. To overcome challenges in the 4D space, we propose the hybrid kernel, a special case of the generalized sparse convolution, and the trilateral-stationary conditional random field that enforces spatio-temporal consistency in the 7D space-time-chroma space. Experimentally, we show that convolutional neural networks with only generalized 3D sparse convolutions can outperform 2D or 2D-3D hybrid methods by a large margin. Also, we show that on 3D-videos, 4D spatio-temporal convolutional neural networks are robust to noise, outperform 3D convolutional neural networks and are faster than the 3D counterpart in some cases.

<img src="https://user-images.githubusercontent.com/72679458/225243534-cd0ed738-4224-4e7c-bcac-4f4c8d89f3a9.png" width="800"/>
</div>

We implement MinkUNet with [TorchSparse](https://github.com/mit-han-lab/torchsparse) / [Minkowski Engine](https://github.com/NVIDIA/MinkowskiEngine) / [Spconv](https://github.com/traveller59/spconv) backend and provide the result and checkpoints on SemanticKITTI datasets.

|                                            Method                                             |     Backend      | Lr schd | Amp | Laser-Polar Mix | Mem (GB) | Training Time (hours) |  FPS   | mIoU |                                                                                                                                                                          Download                                                                                                                                                                           |

|         [MinkUNet18-W16](./minkunet18_w16_torchsparse_8xb2-amp-15e_semantickitti.py)          |   torchsparse    |   15e   |  ✔  |        ✗        |   3.4    |           -           |   -    | 60.3 | [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/minkunet/minkunet_w16_8xb2-15e_semantickitti/minkunet_w16_8xb2-15e_semantickitti_20230309_160737-0d8ec25b.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/minkunet/minkunet_w16_8xb2-15e_semantickitti/minkunet_w16_8xb2-15e_semantickitti_20230309_160737.log) |

|         [MinkUNet18-W20](./minkunet18_w20_torchsparse_8xb2-amp-15e_semantickitti.py)          |   torchsparse    |   15e   |  ✔  |        ✗        |   3.7    |           -           |   -    | 61.6 | [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/minkunet/minkunet_w20_8xb2-15e_semantickitti/minkunet_w20_8xb2-15e_semantickitti_20230309_160718-c3b92e6e.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/minkunet/minkunet_w20_8xb2-15e_semantickitti/minkunet_w20_8xb2-15e_semantickitti_20230309_160718.log) |

|         [MinkUNet18-W32](./minkunet18_w32_torchsparse_8xb2-amp-15e_semantickitti.py)          |   torchsparse    |   15e   |  ✔  |        ✗        |   4.9    |           -           |   -    | 63.1 | [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/minkunet/minkunet_w32_8xb2-15e_semantickitti/minkunet_w32_8xb2-15e_semantickitti_20230309_160710-7fa0a6f1.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/minkunet/minkunet_w32_8xb2-15e_semantickitti/minkunet_w32_8xb2-15e_semantickitti_20230309_160710.log) |

|     [MinkUNet34-W32](./minkunet34_w32_minkowski_8xb2-laser-polar-mix-3x_semantickitti.py)     | minkowski engine |   3x    |  ✗  |        ✔        |   11.5   |          6.5          |  12.2  | 69.2 |          [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/minkunet/minkunet34_w32_minkowski_8xb2-laser-polar-mix-3x_semantickitti_20230514_202236-839847a8.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/minkunet/minkunet34_w32_minkowski_8xb2-laser-polar-mix-3x_semantickitti_20230514_202236.log)          |

|    [MinkUNet34-W32](./minkunet34_w32_spconv_8xb2-amp-laser-polar-mix-3x_semantickitti.py)     |      spconv      |   3x    |  ✔  |        ✔        |   6.7    |           2           | 14.6\* | 68.3 |         [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/minkunet/minkunet34_w32_spconv_8xb2-amp-laser-polar-mix-3x_semantickitti_20230512_233152-e0698a0f.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/minkunet/minkunet34_w32_spconv_8xb2-amp-laser-polar-mix-3x_semantickitti_20230512_233152.log)         |

|      [MinkUNet34-W32](./minkunet34_w32_spconv_8xb2-laser-polar-mix-3x_semantickitti.py)       |      spconv      |   3x    |  ✗  |        ✔        |   10.5   |           6           |  14.5  | 69.3 |             [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/minkunet/minkunet34_w32_spconv_8xb2-laser-polar-mix-3x_semantickitti_20230512_233817-72b200d8.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/minkunet/minkunet34_w32_spconv_8xb2-laser-polar-mix-3x_semantickitti_20230512_233817.log)             |

|  [MinkUNet34-W32](./minkunet34_w32_torchsparse_8xb2-amp-laser-polar-mix-3x_semantickitti.py)  |   torchsparse    |   3x    |  ✔  |        ✔        |   6.6    |           3           |  12.8  | 69.3 |    [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/minkunet/minkunet34_w32_torchsparse_8xb2-amp-laser-polar-mix-3x_semantickitti_20230512_233511-bef6cad0.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/minkunet/minkunet34_w32_torchsparse_8xb2-amp-laser-polar-mix-3x_semantickitti_20230512_233511.log)    |

|    [MinkUNet34-W32](./minkunet34_w32_torchsparse_8xb2-laser-polar-mix-3x_semantickitti.py)    |   torchsparse    |   3x    |  ✗  |        ✔        |   11.8   |          5.5          |  15.9  | 68.7 |        [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/minkunet/minkunet34_w32_torchsparse_8xb2-laser-polar-mix-3x_semantickitti_20230512_233601-2b61b0ab.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/minkunet/minkunet34_w32_torchsparse_8xb2-laser-polar-mix-3x_semantickitti_20230512_233601.log)        |

| [MinkUNet34v2-W32](minkunet34v2_w32_torchsparse_8xb2-amp-laser-polar-mix-3x_semantickitti.py) |   torchsparse    |   3x    |  ✔  |        ✔        |   8.9    |           -           |   -    | 70.3 |  [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/minkunet/minkunet34v2_w32_torchsparse_8xb2-amp-laser-polar-mix-3x_semantickitti_20230510_221853-b14a68b3.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/minkunet/minkunet34v2_w32_torchsparse_8xb2-amp-laser-polar-mix-3x_semantickitti_20230510_221853.log)  |

**Note:** Referring to [PCSeg](https://github.com/PJLab-ADG/PCSeg), MinkUNet34v2 is modified based on MinkUNet34.

**Note\*:** Training Time and FPS are measured on NVIDIA A100. The versions of Torchsparse, Minkowski Engine and Spconv are 0.5.4, 1.4.0 and 2.3.6 respectively. Since spconv 2.3.6 has a bug with fp16 on in the inference stage, the actual FPS measurement using fp32.

```latex
@inproceedings{choy20194d,
  title={4d spatio-temporal convnets: Minkowski convolutional neural networks},
  author={Choy, Christopher and Gwak, JunYoung and Savarese, Silvio},
  pages={3075--3084},
}
```

# Objects365 Dataset

> [Objects365 Dataset](https://openaccess.thecvf.com/content_ICCV_2019/papers/Shao_Objects365_A_Large-Scale_High-Quality_Dataset_for_Object_Detection_ICCV_2019_paper.pdf)

#### Objects365 Dataset V1

[Objects365 Dataset V1](http://www.objects365.org/overview.html) is a brand new dataset,
designed to spur object detection research with a focus on diverse objects in the Wild.
It has 365 object categories over 600K training images. More than 10 million, high-quality bounding boxes are manually labeled through a three-step, carefully designed annotation pipeline. It is the largest object detection dataset (with full annotation) so far and establishes a more challenging benchmark for the community. Objects365 can serve as a better feature learning dataset for localization-sensitive tasks like object detection
and semantic segmentation.

<img src="https://user-images.githubusercontent.com/48282753/208368046-b7573022-06c9-4a99-af17-a6ac7407e3d8.png" height="400"/>
</div>

#### Objects365 Dataset V2

[Objects365 Dataset V2](http://www.objects365.org/overview.html) is based on the V1 release of the Objects365 dataset.
Objects 365 annotated 365 object classes on more than 1800k images, with more than 29 million bounding boxes in the training set, surpassing PASCAL VOC, ImageNet, and COCO datasets.
Objects 365 includes 11 categories of people, clothing, living room, bathroom, kitchen, office/medical, electrical appliances, transportation, food, animals, sports/musical instruments, and each category has dozens of subcategories.

```
@inproceedings{shao2019objects365,
  title={Objects365: A large-scale, high-quality dataset for object detection},

```
  author={Shao, Shuai and Li, Zeming and Zhang, Tianyuan and Peng, Chao and Yu, Gang and Zhang, Xiangyu and Li, Jing and Sun, Jian},
  pages={8430--8439},
}
```

1. You need to download and extract Objects365 dataset. Users can download Objects365 V2 by using `tools/misc/download_dataset.py`.

   **Usage**

   ```shell
   python tools/misc/download_dataset.py --dataset-name objects365v2 \
   --save-dir ${SAVING PATH} \
   --unzip \
   --delete  # Optional, delete the download zip file
   ```

   **Note:** There is no download link for Objects365 V1 right now. If you would like to download Objects365-V1, please visit [official website](http://www.objects365.org/) to concat the author.

2. The directory should be like this:

   ```none

   ```none
   ├── mmdet
   ├── tools
   ├── data
   │   ├── Objects365
   │   │   ├── Obj365_v1
   │   │   │   ├── annotations
   │   │   │   │   ├── objects365_train.json
   │   │   │   │   ├── objects365_val.json
   │   │   │   ├── train        # training images
   │   │   │   ├── val          # validation images
   │   │   ├── Obj365_v2
   │   │   │   │   ├── zhiyuan_objv2_train.json
   │   │   │   │   ├── zhiyuan_objv2_val.json
   │   │   │   │   ├── patch0
   │   │   │   │   ├── patch1
   │   │   │   │   ├── ...
   ```

### Objects365 V1

│   │   │   ├── val          # validation images
   ```

| Architecture | Backbone |  Style  | Lr schd | Mem (GB) | box AP |                                                              Config                                                               |                                                                                                                                                                                Download                                                                                                                                                                                |

| Faster R-CNN |   R-50   | pytorch |   1x    |    -     |  19.6  |   [config](https://github.com/open-mmlab/mmdetection/tree/main/configs/objects365/faster-rcnn_r50_fpn_16xb4-1x_objects365v1.py)   |           [model](https://download.openmmlab.com/mmdetection/v2.0/objects365/faster_rcnn_r50_fpn_16x4_1x_obj365v1/faster_rcnn_r50_fpn_16x4_1x_obj365v1_20221219_181226-9ff10f95.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/objects365/faster_rcnn_r50_fpn_16x4_1x_obj365v1/faster_rcnn_r50_fpn_16x4_1x_obj365v1_20221219_181226.log.json)           |

| Faster R-CNN |   R-50   | pytorch |  1350K  |    -     |  22.3  | [config](https://github.com/open-mmlab/mmdetection/tree/main/configs/objects365/faster-rcnn_r50-syncbn_fpn_1350k_objects365v1.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/objects365/faster_rcnn_r50_fpn_syncbn_1350k_obj365v1/faster_rcnn_r50_fpn_syncbn_1350k_obj365v1_20220510_142457-337d8965.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/objects365/faster_rcnn_r50_fpn_syncbn_1350k_obj365v1/faster_rcnn_r50_fpn_syncbn_1350k_obj365v1_20220510_142457.log.json) |

|  Retinanet   |   R-50   | pytorch |   1x    |    -     |  14.8  |       [config](https://github.com/open-mmlab/mmdetection/tree/main/configs/objects365/retinanet_r50_fpn_1x_objects365v1.py)       |                         [model](https://download.openmmlab.com/mmdetection/v2.0/objects365/retinanet_r50_fpn_1x_obj365v1/retinanet_r50_fpn_1x_obj365v1_20221219_181859-ba3e3dd5.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/objects365/retinanet_r50_fpn_1x_obj365v1/retinanet_r50_fpn_1x_obj365v1_20221219_181859.log.json)                         |

|  Retinanet   |   R-50   | pytorch |  1350K  |    -     |  18.0  |  [config](https://github.com/open-mmlab/mmdetection/tree/main/configs/objects365/retinanet_r50-syncbn_fpn_1350k_objects365v1.py)  |     [model](https://download.openmmlab.com/mmdetection/v2.0/objects365/retinanet_r50_fpn_syncbn_1350k_obj365v1/retinanet_r50_fpn_syncbn_1350k_obj365v1_20220513_111237-7517c576.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/objects365/retinanet_r50_fpn_syncbn_1350k_obj365v1/retinanet_r50_fpn_syncbn_1350k_obj365v1_20220513_111237.log.json)     |

### Objects365 V2

| Architecture | Backbone |  Style  | Lr schd | Mem (GB) | box AP |                                                            Config                                                             |                                                                                                                                                                      Download                                                                                                                                                                      |

| Faster R-CNN |   R-50   | pytorch |   1x    |    -     |  19.8  | [config](https://github.com/open-mmlab/mmdetection/tree/main/configs/objects365/faster-rcnn_r50_fpn_16xb4-1x_objects365v2.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/objects365/faster_rcnn_r50_fpn_16x4_1x_obj365v2/faster_rcnn_r50_fpn_16x4_1x_obj365v2_20221220_175040-5910b015.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/objects365/faster_rcnn_r50_fpn_16x4_1x_obj365v2/faster_rcnn_r50_fpn_16x4_1x_obj365v2_20221220_175040.log.json) |

|  Retinanet   |   R-50   | pytorch |   1x    |    -     |  16.7  |     [config](https://github.com/open-mmlab/mmdetection/tree/main/configs/objects365/retinanet_r50_fpn_1x_objects365v2.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/objects365/retinanet_r50_fpn_1x_obj365v2/retinanet_r50_fpn_1x_obj365v2_20221223_122105-d9b191f1.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/objects365/retinanet_r50_fpn_1x_obj365v2/retinanet_r50_fpn_1x_obj365v2_20221223_122105.log.json)               |

# DeepPose: Human pose estimation via deep neural networks

<details>

```bibtex
}
```

}
```

We propose a method for human pose estimation based on Deep Neural Networks (DNNs). The pose estimation is formulated as a DNN-based regression problem towards body joints. We present a cascade of such DNN regressors which results in high precision pose estimates. The approach has the advantage of reasoning about pose in a holistic fashion and has a simple but yet powerful formulation which capitalizes on recent advances in Deep Learning. We present a detailed empirical analysis with state-of-art or better performance on four academic benchmarks of diverse real-world images.

</div>

# Get Started

MMDetection3D works on Linux, Windows (experimental support) and macOS. It requires Python 3.7+, CUDA 10.0+, and PyTorch 1.8+.

```{note}
```

```shell
```

```shell
```

```shell
```

```shell
```

```shell
```

We recommend that users follow our best practices to install MMDetection3D. However, the whole process is highly customizable. See [Customize Installation](#customize-installation) section for more information.

**Step 0.** Install [MMEngine](https://github.com/open-mmlab/mmengine), [MMCV](https://github.com/open-mmlab/mmcv) and [MMDetection](https://github.com/open-mmlab/mmdetection) using [MIM](https://github.com/open-mmlab/mim).

```shell
```

**Note**: In MMCV-v2.x, `mmcv-full` is renamed to `mmcv`, if you want to install `mmcv` without CUDA ops, you can use `mim install "mmcv-lite>=2.0.0rc4"` to install the lite version.

**Step 1.** Install MMDetection3D.

Case a: If you develop and run mmdet3d directly, install it from source:

```shell
# "-e" means installing a project in edtiable mode,
```

Case b: If you use mmdet3d as a dependency or third-party package, install it with MIM:

```shell
mim install "mmdet3d>=1.1.0"
```

Note:

1. If you would like to use `opencv-python-headless` instead of `opencv-python`,
   you can install it before installing MMCV.

```shell
```

Note:

2. Some dependencies are optional. Simply running `pip install -v -e .` will only install the minimum runtime requirements. To use optional dependencies like `albumentations` and `imagecorruptions` either install them manually with `pip install -r requirements/optional.txt` or specify desired extras when calling `pip` (e.g. `pip install -v -e .[optional]`). Valid keys for the extras field are: `all`, `tests`, `build`, and `optional`.

   We have supported `spconv 2.0`. If the user has installed `spconv 2.0`, the code will use `spconv 2.0` first, which will take up less GPU memory than using the default `mmcv spconv`. Users can use the following commands to install `spconv 2.0`:

   ```shell
   ```

   Where `xxx` is the CUDA version in the environment.

```shell
   ```

   For example, using CUDA 10.2, the command will be `pip install cumm-cu102 && pip install spconv-cu102`.

   Supported CUDA versions include 10.2, 11.1, 11.3, and 11.4. Users can also install it by building from the source. For more details please refer to [spconv v2.x](https://github.com/traveller59/spconv).

   We also support `Minkowski Engine` as a sparse convolution backend. If necessary please follow original [installation guide](https://github.com/NVIDIA/MinkowskiEngine#installation) or use `pip` to install it:

   ```shell

export CPLUS_INCLUDE_PATH=CPLUS_INCLUDE_PATH:${YOUR_CONDA_ENVS_DIR}/include
   ```

   We also support `Torchsparse` as a sparse convolution backend. If necessary please follow original [installation guide](https://github.com/mit-han-lab/torchsparse#installation) or use `pip` to install it:

   ```shell
   sudo apt-get install libsparsehash-dev
   ```

   or omit sudo install by following command:

   ```shell

```

   ```shell
   ```

3. The code can not be built for CPU only environment (where CUDA isn't available) for now.

### Verify the Installation

To verify whether MMDetection3D is installed correctly, we provide some sample codes to run an inference demo.

```shell
```

```shell
```

The downloading will take several seconds or more, depending on your network environment. When it is done, you will find two files `pointpillars_hv_secfpn_8xb6-160e_kitti-3d-car.py` and `hv_pointpillars_secfpn_6x8_160e_kitti-3d-car_20220331_134606-d42d15ed.pth` in your current folder.

Case a: If you install MMDetection3D from source, just run the following command.

```shell
```

You will see a visualizer interface with point cloud, where bounding boxes are plotted on cars.

**Note**:

```

**Note**:

If you want to input a `.ply` file, you can use the following function and convert it to `.bin` format. Then you can use the converted `.bin` file to run demo.
Note that you need to install `pandas` and `plyfile` before using this script. This function can also be used for data preprocessing for training `ply data`.

```python

    plydata = PlyData.read(input_path)  # read file
    data = plydata.elements[0].data  # read data
    data_pd = pd.DataFrame(data)  # convert to DataFrame
    data_np = np.zeros(data_pd.shape, dtype=np.float)  # initialize array to store data
    property_names = data[0].dtype.names  # read names of properties
            property_names):  # read data by property

property_names = data[0].dtype.names  # read names of properties
```

Examples:

```python
```

If you have point clouds in other format (`.off`, `.obj`, etc.), you can use `trimesh` to convert them into `.ply`.

```python

    mesh = trimesh.load(input_path, file_type=original_type)  # read file
    mesh.export(output_path, file_type='ply')  # convert to ply
```

Examples:

```python
```

Case b: If you install MMDetection3D with MIM, open your python interpreter and copy&paste the following codes.

```python

```python

```

You will see a list of `Det3DDataSample`, and the predictions are in the `pred_instances_3d`, indicating the detected bounding boxes, labels, and scores.

#### CUDA Versions

```{note}
```

```

For example, you can install MMEngine by the following command:

```shell
```

For example, the following command install MMCV built for PyTorch 1.12.x and CUDA 11.6:

```shell
```

```shell
```

[Google Colab](https://colab.research.google.com/) usually has PyTorch installed, thus we only need to install MMEngine, MMCV, MMDetection, and MMDetection3D with the following commands.

**Step 1.** Install [MMEngine](https://github.com/open-mmlab/mmengine), [MMCV](https://github.com/open-mmlab/mmcv) and [MMDetection](https://github.com/open-mmlab/mmdetection) using [MIM](https://github.com/open-mmlab/mim).

```shell
```

**Step 2.** Install MMDetection3D from source.

```shell
```

```python
# Example output: 1.1.0, or an another version.
```

```{note}

```

```python
```

```{note}
```

#### Using MMDetection3D with Docker

We provide a [Dockerfile](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/docker/Dockerfile) to build an image. Ensure that your [docker version](https://docs.docker.com/engine/install/) >= 19.03.

```shell
```

Run it with:

```shell
```

```

```shell
```

You may [open an issue](https://github.com/open-mmlab/mmdetection3d/issues/new/choose) on GitHub if no solution is found.

### Use Multiple Versions of MMDetection3D in Development

Training and testing scripts have already been modified in `PYTHONPATH` in order to make sure the scripts are using their own versions of MMDetection3D.

To install the default version of MMDetection3D in your environment, you can exclude the following code in the related scripts:

```shell
```

# Key Information Extraction

```{note}
```

The structure of the key information extraction dataset directory is organized as follows.

```text
  ├── class_list.txt
  ├── dict.txt
  ├── image_files
  ├── openset_train.txt
  ├── openset_test.txt
  ├── test.txt
  └── train.txt
```

## Preparation Steps

- Just download and extract [wildreceipt.tar](https://download.openmmlab.com/mmocr/data/wildreceipt.tar).

- Step0: have [WildReceipt](#WildReceipt) prepared.
- Step1: Convert annotation files to OpenSet format:

```bash
# You may find more available arguments by running
# python tools/data/kie/closeset_to_openset.py -h
python tools/data/kie/closeset_to_openset.py data/wildreceipt/train.txt data/wildreceipt/openset_train.txt

python tools/data/kie/closeset_to_openset.py data/wildreceipt/test.txt data/wildreceipt/openset_test.txt
```

```{note}
You can learn more about the key differences between CloseSet and OpenSet annotations in our [tutorial](../tutorials/kie_closeset_openset.md).
```

# 提交测试结果

## 全景分割测试结果提交

下面几节介绍如何在 COCO 测试开发集上生成泛视分割模型的预测结果，并将预测提交到 [COCO评估服务器](https://competitions.codalab.org/competitions/19507)

### 前提条件

- 下载 [COCO测试数据集图像](http://images.cocodataset.org/zips/test2017.zip)，[测试图像信息](http://images.cocodataset.org/annotations/image_info_test2017.zip)，和[全景训练/相关注释](http://images.cocodataset.org/annotations/panoptic_annotations_trainval2017.zip)，然后解压缩它们，把 `test2017` 放到 `data/coco/`，把 json 文件和注释文件放到 `data/coco/annotations/` 。

```shell
# 假设 data/coco/ 不存在
# 下载 test2017
# 解压缩它们
# 删除 zip 文件(可选)

# 解压缩它们
```

- 运行以下代码更新测试图像信息中的类别信息。由于 `image_info_test-dev2017.json` 的类别信息中缺少属性 `isthing` ，我们需要用 `panoptic_val2017.json` 中的类别信息更新它。

```shell
```

在完成上述准备之后，你的 `data` 目录结构应该是这样:

```text
data
`-- coco
```

### coco 测试开发的推理

|   |-- panoptic_train2017.json
```

要在 coco test-dev 上进行推断，我们应该首先更新 `test_dataloder` 和 `test_evaluator` 的设置。有两种方法可以做到这一点:1. 在配置文件中更新它们;2. 在命令行中更新它们。

#### 在配置文件中更新它们

相关的设置在 `configs/_base_/datasets/ coco_panoptical .py` 的末尾，如下所示。

```python

```

以下任何一种方法都可以用于更新 coco test-dev 集上的推理设置

情况1:直接取消注释 `configs/_base_/datasets/ coco_panoptical .py` 中的设置。

情况2:将以下设置复制到您现在使用的配置文件中。

```python
```

然后通过以下命令对 coco test-dev et 进行推断。

```shell
```

#### 在命令行中更新它们

coco test-dev 上更新相关设置和推理的命令如下所示。

```shell
# 用一个 gpu 测试

```

```shell
# 用一个 gpu 测试
# 用四个 gpu 测试

test_dataloader.dataset.ann_file=annotations/panoptic_image_info_test-dev2017.json \
# 用 slurm 测试
```

例子:假设我们使用预先训练的带有 ResNet-50 骨干网的 MaskFormer 对 `test2017` 执行推断。

```shell
# 单 gpu 测试

test_evaluator.outfile_prefix=${WORK_DIR}/results
```

```shell
# 单 gpu 测试
```

### 重命名文件并压缩结果

推理之后，全景分割结果(一个 json 文件和一个存储掩码的目录)将在 `WORK_DIR` 中。我们应该按照 [COCO's Website](https://cocodataset.org/#upload)上的命名约定重新命名它们。最后，我们需要将 json 和存储掩码的目录压缩到 zip 文件中，并根据命名约定重命名该 zip 文件。注意， zip 文件应该**直接**包含上述两个文件。

重命名文件和压缩结果的命令:

```shell

```shell
# 在 WORK_DIR 中，我们有 panoptic 分割结果: 'panoptic' 和 'results. panoptical .json'。
# 将 '[algorithm_name]' 替换为您使用的算法名称
```

# CSRA

> [Residual Attention: A Simple but Effective Method for Multi-Label Recognition](https://arxiv.org/abs/2108.02456)

Multi-label image recognition is a challenging computer vision task of practical use. Progresses in this area, however, are often characterized by complicated methods, heavy computations, and lack of intuitive explanations. To effectively capture different spatial regions occupied by objects from different categories, we propose an embarrassingly simple module, named class-specific residual attention (CSRA). CSRA generates class-specific features for every category by proposing a simple spatial attention score, and then combines it with the class-agnostic average pooling feature. CSRA achieves state-of-the-art results on multilabel recognition, and at the same time is much simpler than them. Furthermore, with only 4 lines of code, CSRA also leads to consistent improvement across many diverse pretrained models and datasets without any extra training. CSRA is both easy to implement and light in computations, which also enjoys intuitive explanations and visualizations.

<img src="https://user-images.githubusercontent.com/84259897/176982245-3ffcff56-a4ea-4474-9967-bc2b612bbaa3.png" width="80%"/>
</div>

```python

model = get_model('resnet101-csra_1xb16_voc07-448px', pretrained=True)
```

Train:

```shell
python tools/train.py configs/csra/resnet101-csra_1xb16_voc07-448px.py
```

Test:

```shell
python tools/test.py configs/csra/resnet101-csra_1xb16_voc07-448px.py https://download.openmmlab.com/mmclassification/v0/csra/resnet101-csra_1xb16_voc07-448px_20220722-29efb40a.pth
```

```

### Multi-Label Classification on PASCAL VOC 2007

| Model                              |   Pretrain   | Params (M) | Flops (G) |  CF1  |  OF1  |  mAP  |                    Config                     |                                  Download                                   |
| :--------------------------------- | :----------: | :--------: | :-------: | :---: | :---: | :---: | :-------------------------------------------: | :-------------------------------------------------------------------------: |
| `resnet101-csra_1xb16_voc07-448px` | From scratch |   23.55    |   4.12    | 89.16 | 90.80 | 94.98 | [config](resnet101-csra_1xb16_voc07-448px.py) | [model](https://download.openmmlab.com/mmclassification/v0/csra/resnet101-csra_1xb16_voc07-448px_20220722-29efb40a.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/csra/resnet101-csra_1xb16_voc07-448px_20220722-29efb40a.json) |

```bibtex

```bibtex
@misc{https://doi.org/10.48550/arxiv.2108.02456,
  doi = {10.48550/ARXIV.2108.02456},
  url = {https://arxiv.org/abs/2108.02456},
  author = {Zhu, Ke and Wu, Jianxin},
  title = {Residual Attention: A Simple but Effective Method for Multi-Label Recognition},
  copyright = {arXiv.org perpetual, non-exclusive license}
}
```

# 准备 VideoLQ 数据集

```bibtex
}
```

数据集可以从 [Dropbox](https://www.dropbox.com/sh/hc06f1livdhutbo/AAAMPy92EOqVjRN8waT0ie8ja?dl=0) / [Google Drive](https://drive.google.com/drive/folders/1-1iJRNdqdFZWOnoUU4xG1Z1QhwsGwMDy?usp=sharing) / [OneDrive](https://entuedu-my.sharepoint.com/:f:/g/personal/chan0899_e_ntu_edu_sg/ErSugvUBxoBMlvSAHhqT5BEB9-4ZaqxzJIcc9uvVa8JGHg?e=WpHJTc) 下载。

```text
mmagic
├── mmagic
├── tools
├── data
│   │   ├── 000
│   │   │   ├── ...
│   │   ├── 001
│   │   ├── 002
│   │   ├── ...
```

<details>

```bibtex
}
```

<details>

```bibtex

```bibtex
}
```

| [pose_hrnet_w32](/configs/body_2d_keypoint/topdown_heatmap/aic/td-hm_hrnet-w32_8xb64-210e_aic-256x192.py) |  256x192   | 0.323 |      0.761      |      0.218      | 0.366 |      0.789      | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_aic_256x192-30a4e465_20200826.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_aic_256x192_20200826.log.json) |

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.azarlive.android.png) Azar-视频交友与聊天 June 20, 2018

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.cyberlink.youcammakeup.png) 玩美彩妆 - 自拍美颜 & 智能美妆相机 June 21, 2018

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.fotoable.makeup.png) You Makeup Photo Camera 2.1.5

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.fotoable.cartoon.cam.png) 滤镜相机 Cartoon Camera- Paintlab January 24, 2018

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.pipcamera.activity.png) 画中画相机 January 30, 2018

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.hefe.pro.editor.png) Photo Editor Pro 1.1.4.1029

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.apus.camera.id.png) Air Camera 1.7.3.1002

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.fotoable.fotobeauty.png) 美丽拍－懂你的自拍美颜相机 February 1, 2018

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.perfectcorp.ycf.png) 玩美Fun-特效动图自拍滤镜&分享相片！ May 15, 2018

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.ufotosoft.justshot.png) Sweet Snap - 生活贴纸&图像编辑器,实时滤镜,录制视频和有趣表情包,美容效果 June 22, 2018

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.wantu.activity.png) 玩图 - 美图相机 March 29, 2018

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.meitu.meiyancamera.png) 美颜相机 7.6.95

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.lyrebirdstudio.colorizer.lite.png) 自拍相机 - 照片编辑器和过滤器和贴纸 April 27, 2018

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.apusapps.fulakora.png) APUS Camera 1.7.2.1001

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/video.like.png) LIKE短视频 — 魔法视频自拍神器 2.2.4

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.qiyi.video.png) 爱奇艺 9.6.0

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.eg.android.AlipayGphone.png) 支付宝 10.1.25.752

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.perfectcorp.beautycircle.png) YouCam Shop - World's First AR Makeup Shopping App 3.4.0

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.lyrebirdstudio.beauty.png) 美容化妆自拍相机和自拍照片编辑器 1.4.8

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.jingdong.app.mall.png) 京东-挑好物，上京东 7.0.8

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.versa.png) Versa 2.9.2

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.tencent.weishi.png) 微视 4.3.1.88

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.smile.gifmaker.png) 快手短视频—国民短视频平台 5.4.2.5360

![](https://github.com/nihui/ncnn-assets/raw/master/20180626/com.sdu.didi.psnger.png) 滴滴出行 5.3.0

# Simple baselines for human pose estimation and tracking

<details>

```bibtex
}
```

}
```

There has been significant progress on pose estimation and increasing interests on pose tracking in recent years. At the same time, the overall algorithm and system complexity increases as well, making the algorithm analysis and comparison more difficult. This work provides simple and effective baseline methods. They are helpful for inspiring and evaluating new ideas for the field. State-of-the-art results are achieved on challenging benchmarks.

<img src="https://user-images.githubusercontent.com/15977946/146522977-5f355832-e9c1-442f-a34f-9d24fb0aefa8.png">
</div>

# TRB: A Novel Triplet Representation for Understanding 2D Human Body

<details>

```bibtex
}
```

# 模型服务

## 部署 [LLaMA-2](https://github.com/facebookresearch/llama) 服务

请从[这里](https://huggingface.co/meta-llama) 下载 llama2 模型，参考如下命令部署服务：

```shell
```

```shell
```

```shell
```

## 部署 [LLaMA](https://github.com/facebookresearch/llama) 服务

请填写[这张表](https://docs.google.com/forms/d/e/1FAIpQLSfqNECQnMkycAp2jP4Z9TFX0cGR4uf7b_fBxjY_OjhJILlKGA/viewform)，获取 LLaMA 模型权重

```shell

```shell
```

```shell
```

```shell
```

```shell

```

```shell
```

### 部署 [Vicuna](https://lmsys.org/blog/2023-03-30-vicuna/) 服务

```shell

```

```shell

```

# ResNeSt: Split-Attention Networks

<details>

```bibtex
}
```

}
```

<img src="https://user-images.githubusercontent.com/15977946/146538001-387b40dc-886c-48c5-8621-1dc287fa2ae3.png">
</div>

# How to Contribute to MMPose

Welcome to join the MMPose community, we are committed to building cutting-edge computer vision foundational library. All kinds of contributions are welcomed, including but not limited to:

- **Fix bugs**
  1. If the modification involves significant changes, it's recommended to create an issue first that describes the error information and how to trigger the bug. Other developers will discuss it with you and propose a proper solution.
  2. Fix the bug and add the corresponding unit test, submit the PR.
- **Add new features or components**
  1. If the new feature or module involves a large amount of code changes, we suggest you to submit an issue first, and we will confirm the necessity of the function with you.
  2. Implement the new feature and add unit tests, submit the PR.
- **Improve documentation or translation**
  - If you find errors or incomplete documentation, please submit a PR directly.

```{note}

2. Implement the new feature and add unit tests, submit the PR.

```{note}
- If you hope to contribute to MMPose 1.0, please create a new branch from dev-1.x and submit a PR to the dev-1.x branch.
- If you are the author of papers in this field and would like to include your work to MMPose, please contact us. We will much appreciate your contribution.
- If you hope to share your MMPose-based projects with the community at once, consider creating a PR to `Projects` directory, which will simplify the review process and bring in the projects as soon as possible. Checkout our [example project](/projects/example_project)
- If you wish to join the MMPose developers, please feel free to contact us and we will invite you to join the MMPose developers group.
```

```

The commands for processing pull requests are implemented using Git, and this chapter details Git Configuration and associated GitHub.

### Git Configuration

First, you need to install Git and configure your Git username and email.

```shell
# view the Git version
```

Second, check your Git config and ensure that `user.name` and `user.email` are properly configured.

```shell
# view the Git config
# configure the user name and email
git config --global user.name "Change your user name here"
git config --global user.email "Change your user email here"
```

## Pull Request Workflow

```

If you’re not familiar with Pull Request, don’t worry! The following guidance will tell you how to create a Pull Request step by step. If you want to dive into the development mode of Pull Request, you can refer to the [official documents](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests).

### 1. Fork and Clone

Then you need to clone the forked repository to your local machine.

```shell

```shell
# clone the forked repository

# Add official repository as upstream remote
cd mmpose
```

Enter the following command in the terminal to see if the remote repository was successfully added.

```shell
```

If the following message appears, you have successfully added a remote repository.

```Shell
```

```{note}

```

```{note}
Here’s a brief introduction to the origin and upstream. When we use “git clone”, we create an “origin” remote by default, which points to the repository cloned from. As for “upstream”, we add it ourselves to point to the target repository. Of course, if you don’t like the name “upstream”, you could name it as you wish. Usually, we’ll push the code to “origin”. If the pushed code conflicts with the latest code in official(“upstream”), we should pull the latest code from upstream to resolve the conflicts, and then push to “origin” again. The posted Pull Request will be updated automatically.
```

### 2. Configure pre-commit

```

You should configure pre-commit in the local development environment to make sure the code style matches that of OpenMMLab. Note: The following code should be executed under the MMPOSE directory.

```Shell
```

```Shell
```

```{note}
Chinese users may fail to download the pre-commit hooks due to the network issue. In this case, you could download these hooks from:

or:

```

If the installation process is interrupted, you can repeatedly run `pre-commit run ...` to continue the installation.

or:

```

If the code does not conform to the code style specification, pre-commit will raise a warning and fixes some of the errors automatically.

### 3. Create a development branch

After configuring the pre-commit, we should create a branch based on the dev branch to develop the new feature or fix the bug. The proposed branch name is `username/pr_name`.

```Shell
```

In subsequent development, if the dev branch of the local repository lags behind the dev branch of the official repository, you need to pull the upstream dev branch first and then rebase it to the local development branch.

```Shell

```Shell
```

When rebasing, if a conflict arises, you need to resolve the conflict manually, then execute the `git add` command, and then execute the `git rebase --continue` command until the rebase is complete.

### 4. Commit the code and pass the unit test

After the local development is done, we need to pass the unit tests locally and then commit the code.

```shell
# run unit test

# commit the code
git add .
```

### 5. Push the code to the remote repository

After the local development is done, we need to push the code to the remote repository.

```Shell
```

### 6. Create a Pull Request

#### (1) Create a Pull Request on GitHub

#### (2) Fill in the Pull Request template

## Code Style

### Python

We adopt [PEP8](https://www.python.org/dev/peps/pep-0008/) as the preferred code style, and use the following tools for linting and formatting:

Style configurations of yapf and isort can be found in [setup.cfg](/setup.cfg).

The config for a pre-commit hook is stored in [.pre-commit-config](/.pre-commit-config.yaml).

```{note}

```{note}
```

# Preparing UCF101-24

```BibTeX
}
```

For basic dataset information, you can refer to the dataset [website](http://www.thumos.info/download.html).
Before we start, please make sure that the directory is located at `$MMACTION2/tools/data/ucf101_24/`.

## Download and Extract

You can download the RGB frames, optical flow and ground truth annotations from [google drive](https://drive.google.com/drive/folders/1BvGywlAGrACEqRyfYbz3wzlVV3cDFkct).
The data are provided from [MOC](https://github.com/MCG-NJU/MOC-Detector/blob/master/readme/Dataset.md), which is adapted from [act-detector](https://github.com/vkalogeiton/caffe/tree/act-detector) and [corrected-UCF101-Annots](https://github.com/gurkirt/corrected-UCF101-Annots).

:::{note}

:::{note}
The annotation of this UCF101-24 is from [here](https://github.com/gurkirt/corrected-UCF101-Annots), which is more correct.
:::

After downloading the `UCF101_v2.tar.gz` file and put it in `$MMACTION2/tools/data/ucf101_24/`, you can run the following command to uncompress.

```shell
```

## Check Directory Structure

After uncompressing, you will get the `rgb-images` directory, `brox-images` directory and `UCF101v2-GT.pkl` for UCF101-24.

In the context of the whole project (for UCF101-24 only), the folder structure will look like:

```
mmaction2
├── tools
├── data
│   |   |   ├── ...

│   |   |   ├── ...
│   |   |   ├── ...

```

:::{note}
The `UCF101v2-GT.pkl` exists as a cache, it contains 6 items as follows:
:::

1. `labels` (list): List of the 24 labels.
2. `gttubes` (dict): Dictionary that contains the ground truth tubes for each video.

:::

   A **gttube** is dictionary that associates with each index of label and a list of tubes.
   A **tube** is a numpy array with `nframes` rows and 5 columns, each col is in format like `<frame index> <x1> <y1> <x2> <y2>`.
3. `nframes` (dict): Dictionary that contains the number of frames for each video, like `'HorseRiding/v_HorseRiding_g05_c02': 151`.
4. `train_videos` (list): A list with `nsplits=1` elements, each one containing the list of training videos.
5. `test_videos` (list): A list with `nsplits=1` elements, each one containing the list of testing videos.
6. `resolution` (dict): Dictionary that outputs a tuple (h,w) of the resolution for each video, like `'FloorGymnastics/v_FloorGymnastics_g09_c03': (240, 320)`.

# Preparing Video Retrieval Datasets

```BibTeX
}
```

Before we start, please make sure that the directory is located at `$MMACTION2/tools/data/video_retrieval/`.

## Preparing MSRVTT dataset

For basic dataset information, you can refer to the MSRVTT dataset [website](https://www.microsoft.com/en-us/research/publication/msr-vtt-a-large-video-description-dataset-for-bridging-video-and-language/). Run the following command to prepare the MSRVTT dataset:

```shell
```

After preparation, the folder structure will look like:

```
mmaction2
├── tools
├── data

```
mmaction2
├── tools
├── data
│   │           ├── ...
```

# 在标注数据集上测试和训练

### 在标准数据集上测试已有模型

- 单显卡
- CPU
- 单节点多显卡
- 多节点

你可以通过以下命令来测试数据集：

```shell
# 单块显卡测试
python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [--eval ${EVAL_METRICS}] [--show] [--show-dir ${SHOW_DIR}]

# CPU：禁用显卡并运行单块 CPU 测试脚本（实验性）

# 多块显卡测试
./tools/dist_test.sh ${CONFIG_FILE} ${CHECKPOINT_FILE} ${GPU_NUM} [--out ${RESULT_FILE}] [--eval ${EVAL_METRICS}]
```

**注意**:

目前我们只支持 SMOKE 的 CPU 推理测试。

可选参数：

- `--show`：如果被指定，检测结果会在静默模式下被保存，用于调试和可视化，但只在单块 GPU 测试的情况下生效，和 `--show-dir` 搭配使用。
- `--show-dir`：如果被指定，检测结果会被保存在指定文件夹下的 `***_points.obj` 和 `***_pred.obj` 文件中，用于调试和可视化，但只在单块 GPU 测试的情况下生效，对于这个选项，图形化界面在你的环境中不是必需的。

所有和评估相关的参数在相应的数据集配置的 `test_evaluator` 中设置。例如 `test_evaluator = dict(type='KittiMetric', ann_file=data_root + 'kitti_infos_val.pkl', pklfile_prefix=None, submission_prefix=None)`

参数：

参数：

- `type`：相对应的评价指标名，通常和数据集相关联。
- `ann_file`：标注文件路径。
- `pklfile_prefix`：可选参数。输出结果保存成 pickle 格式的文件名。如果没有指定，结果将不会保存成文件。
- `submission_prefix`：可选参数。结果将被保存到文件中，然后你可以将它上传到官方评估服务器中。

示例：

假定你已经把模型权重文件下载到 `checkpoints/` 文件夹下，

1. 在 ScanNet 数据集上测试 VoteNet，保存模型，可视化预测结果

   ```shell
   ```

2. 在 ScanNet 数据集上测试 VoteNet，保存模型，可视化预测结果，可视化真实标签，计算 mAP

   ```shell
   ```

3. 在 ScanNet 数据集上测试 VoteNet（不保存测试结果），计算 mAP

checkpoints/votenet_8x8_scannet-3d-18class_20200620_230238-2cea9c3a.pth \
   ```

   ```shell
   ```

4. 使用 8 块显卡在 KITTI 数据集上测试 SECOND，计算 mAP

   ```shell
   ```

5. 使用 8 块显卡在 nuScenes 数据集上测试 PointPillars，生成提交给官方评测服务器的 json 文件

   ```shell
   ```

checkpoints/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d_20200620_230405-2fa62f3d.pth \
   ```

   生成的结果会保存在 `./pointpillars_nuscenes_results` 目录。

6. 使用 8 块显卡在 KITTI 数据集上测试 SECOND，生成提交给官方评测服务器的 txt 文件

   ```shell
   ```

   生成的结果会保存在 `./second_kitti_results` 目录。

7. 使用 8 块显卡在 Lyft 数据集上测试 PointPillars，生成提交给排行榜的 pkl 文件

   ```shell

```shell
   ```

   **注意**：为了生成 Lyft 数据集的提交结果，`--eval-options` 必须指定 `csv_savepath`。生成 csv 文件后，你可以使用[网站](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/submit)上给出的 kaggle 命令提交结果。

   注意在 [Lyft 数据集的配置文件](../../configs/_base_/datasets/lyft-3d.py)，`test` 中的 `ann_file` 值为 `lyft_infos_test.pkl`，是没有标注的 Lyft 官方测试集。要在验证数据集上测试，请把它改为 `lyft_infos_val.pkl`。

8. 使用 8 块显卡在 waymo 数据集上测试 PointPillars，使用 waymo 度量方法计算 mAP

   ```shell

   ```shell
   ```

   **注意**：对于 waymo 数据集上的评估，请根据[说明](https://github.com/waymo-research/waymo-open-dataset/blob/master/docs/quick_start.md/)构建二进制文件 `compute_detection_metrics_main` 来做度量计算，并把它放在 `mmdet3d/core/evaluation/waymo_utils/`。（在使用 bazel 构建  `compute_detection_metrics_main` 时，有时会出现 `'round' is not a member of 'std'` 的错误，我们只需要把那个文件中 `round` 前的 `std::` 去掉。）二进制文件生成时需要在 `--eval-options` 中给定 `pklfile_prefix`。对于度量方法，`waymo` 是推荐的官方评估策略，目前 `kitti` 评估是依照 KITTI 而来的，每个难度的结果和 KITTI 的定义并不完全一致。目前大多数物体都被标记为0难度，会在未来修复。它的不稳定原因包括评估的计算大、转换后的数据缺乏遮挡和截断、难度的定义不同以及平均精度的计算方法不同。

9. 使用 8 块显卡在 waymo 数据集上测试 PointPillars，生成 bin 文件并提交到排行榜

   ```shell
   ```

   **注意**：生成 bin 文件后，你可以简单地构建二进制文件  `create_submission`，并根据[说明](https://github.com/waymo-research/waymo-open-dataset/blob/master/docs/quick_start.md/)创建提交的文件。要在验证服务器上评测验证数据集，你也可以用同样的方式生成提交的文件。

## 在标准数据集上训练预定义模型

MMDetection3D 分别用 `MMDistributedDataParallel` and `MMDataParallel` 实现了分布式训练和非分布式训练。

所有的输出（日志文件和模型权重文件）都会被保存到工作目录下，通过配置文件里的 `work_dir` 指定。

默认我们每过一个周期都在验证数据集上评测模型，你可以通过在训练配置里添加间隔参数来改变评测的时间间隔：

```python
train_cfg = dict(type='EpochBasedTrainLoop', val_interval=1)  # 每12个周期评估一次模型
```

```python
```

**重要**：配置文件中的默认学习率对应 8 块显卡，配置文件名里有具体的批量大小，比如 '2xb8' 表示一共 8 块显卡，每块显卡 2 个样本。
根据 [Linear Scaling Rule](https://arxiv.org/abs/1706.02677)，当你使用不同数量的显卡或每块显卡有不同数量的图像时，需要依批量大小按比例调整学习率。如果用 4 块显卡、每块显卡 2 幅图像时学习率为 0.01，那么用 16 块显卡、每块显卡 4 幅图像时学习率应设为 0.08。然而，由于大多数模型使用 ADAM 而不是 SGD 进行优化，上述规则可能并不适用，用户需要自己调整学习率。

### 使用单块显卡进行训练

```shell
```

如果你想在命令中指定工作目录，添加参数 `--work-dir ${YOUR_WORK_DIR}`。

### 使用 CPU 进行训练 (实验性)

在 CPU 上训练的过程与单 GPU 训练一致。 我们只需要在训练过程之前禁用显卡。

```shell
```

之后运行单显卡训练脚本即可。

**注意**：

目前，大多数点云相关算法都依赖于 3D CUDA 算子，无法在 CPU 上进行训练。 一些单目 3D 物体检测算法，例如 FCOS3D、SMOKE 可以在 CPU 上进行训练。我们不推荐用户使用 CPU 进行训练，这太过缓慢。我们支持这个功能是为了方便用户在没有显卡的机器上调试某些特定的方法。

### 使用多块显卡进行训练

```shell

**注意**：

```shell
```

可选参数：

- `--cfg-options 'Key=value'`：覆盖使用的配置中的一些设定。

### 使用多个机器进行训练

如果要在 [slurm](https://slurm.schedmd.com/) 管理的集群上运行 MMDectection3D，你可以使用 `slurm_train.sh` 脚本（该脚本也支持单机训练）

```shell
```

下面是一个使用 16 块显卡在 dev 分区上训练 Mask R-CNN 的示例：

```shell
```

你可以查看 [slurm_train.sh](https://github.com/open-mmlab/mmdetection/blob/master/tools/slurm_train.sh) 来获取所有的参数和环境变量。

在第一台机器上:

```shell

```

在第一台机器上:

```shell
```

在第二台机器上:

```shell
```

### 在单个机器上启动多个任务

如果你在单个机器上启动多个任务，比如，在具有8块显卡的机器上进行2个4块显卡训练的任务，你需要为每个任务指定不同的端口（默认为29500）以避免通信冲突。

如果你使用 `dist_train.sh` 启动训练任务，可以在命令中设置端口：

```shell
```

如果你使用 Slurm 启动训练任务，有两种方式指定端口：

1. 通过 `--cfg-options` 设置端口，这是更推荐的，因为它不改变原来的配置

   ```shell

```shell
   ```

2. 修改配置文件（通常在配置文件的倒数第6行）来设置不同的通信端口

   在 `config1.py` 中，

   ```python
   )
   ```

   在 `config2.py` 中，

   ```python
   )
   ```

   然后，你可以使用 `config1.py` and `config2.py` 启动两个任务

   ```shell
   ```

<details>

```bibtex
}
```

<details>

```bibtex

| [pose_mobilenetv2](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_mobilenetv2_8xb64-210e_mpii-256x256.py) |  256x256   | 0.854 |  0.234   | [ckpt](https://download.openmmlab.com/mmpose/top_down/mobilenetv2/mobilenetv2_mpii_256x256-e068afa7_20200812.pth) | [log](https://download.openmmlab.com/mmpose/top_down/mobilenetv2/mobilenetv2_mpii_256x256_20200812.log.json) |

# Preparing DOTA Dataset

```bibtex
}
```

## download dota dataset

The dota dataset can be downloaded from [here](https://captain-whu.github.io/DOTA/dataset.html).

```none
mmrotate
├── tools
├── data
│   ├── DOTA
│   │   ├── val
│   │   ├── test
```

## split dota dataset

Please crop the original images into 1024×1024 patches with an overlap of 200 by run

```shell
python tools/data/dota/split/img_split.py --base-json \
  tools/data/dota/split/split_configs/ss_trainval.json

```shell

  tools/data/dota/split/split_configs/ss_test.json
```

If you want to get a multiple scale dataset, you can run the following command.

```shell
  tools/data/dota/split/split_configs/ms_trainval.json

  tools/data/dota/split/split_configs/ms_test.json
```

Please update the `img_dirs` and `ann_dirs` in json.

The new data structure is as follows:

```none
mmrotate
├── tools
├── data
│   ├── split_ss_dota
│   │   ├── trainval
│   │   │   ├── annfiles
│   │   ├── test
```

Please change `data_root` in `configs/_base_/datasets/dota.py` to `data/split_ss_dota`.

## Generating COCO style annotations

│   │   ├── test
```

Please convert the annotations from txt to json by run

```shell
python tools/data/dota/dota2coco.py \
  data/split_ss_dota/trainval/ \
  data/split_ss_dota/trainval.json

  data/split_ss_dota/test/ \
  data/split_ss_dota/test.json
```

```none
mmrotate
├── tools
├── data
│   │   │   ├── trainval.json
│   │   ├── test
│   │   │   ├── test.json
```

Please change `data_root` in `configs/_base_/datasets/dota_coco.py` to `data/split_ss_dota`.

# R2plus1D

[A closer look at spatiotemporal convolutions for action recognition](https://openaccess.thecvf.com/content_cvpr_2018/html/Tran_A_Closer_Look_CVPR_2018_paper.html)

In this paper we discuss several forms of spatiotemporal convolutions for video analysis and study their effects on action recognition. Our motivation stems from the observation that 2D CNNs applied to individual frames of the video have remained solid performers in action recognition. In this work we empirically demonstrate the accuracy advantages of 3D CNNs over 2D CNNs within the framework of residual learning. Furthermore, we show that factorizing the 3D convolutional filters into separate spatial and temporal components yields significantly advantages in accuracy. Our empirical study leads to the design of a new spatiotemporal convolutional block "R(2+1)D" which gives rise to CNNs that achieve results comparable or superior to the state-of-the-art on Sports-1M, Kinetics, UCF101 and HMDB51.

<img src="https://user-images.githubusercontent.com/34324155/143043885-3d00413c-b556-445e-9673-f5805c08c195.png" width="800"/>
</div>

|          8x8x1          |  224x224   |  8   | ResNet34 |   None   |  69.76   |  88.41   | 10 clips x 3 crop | 53.1G | 63.8M  | [config](/configs/recognition/r2plus1d/r2plus1d_r34_8xb8-8x8x1-180e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/r2plus1d/r2plus1d_r34_8xb8-8x8x1-180e_kinetics400-rgb/r2plus1d_r34_8xb8-8x8x1-180e_kinetics400-rgb_20220812-47cfe041.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/r2plus1d/r2plus1d_r34_8xb8-8x8x1-180e_kinetics400-rgb/r2plus1d_r34_8xb8-8x8x1-180e_kinetics400-rgb.log) |

|         32x2x1          |  224x224   |  8   | ResNet34 |   None   |  75.46   |  92.28   | 10 clips x 3 crop | 213G  | 63.8M  | [config](/configs/recognition/r2plus1d/r2plus1d_r34_8xb8-32x2x1-180e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/r2plus1d/r2plus1d_r34_8xb8-32x2x1-180e_kinetics400-rgb/r2plus1d_r34_8xb8-32x2x1-180e_kinetics400-rgb_20220812-4270588c.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/r2plus1d/r2plus1d_r34_8xb8-32x2x1-180e_kinetics400-rgb/r2plus1d_r34_8xb8-32x2x1-180e_kinetics400-rgb.log) |

## Train

```shell
```

Example: train R(2+1)D model on Kinetics-400 dataset in a deterministic option.

```shell
python tools/train.py configs/recognition/r2plus1d/r2plus1d_r34_8xb8-8x8x1-180e_kinetics400-rgb.py \

```

```shell
```

## Test

```shell
```

Example: test R(2+1)D model on Kinetics-400 dataset and dump the result to a pkl file.

```shell
python tools/test.py configs/recognition/r2plus1d/r2plus1d_r34_8xb8-8x8x1-180e_kinetics400-rgb.py \
```

```BibTeX
@inproceedings{tran2018closer,

```

```BibTeX
  title={A closer look at spatiotemporal convolutions for action recognition},
  author={Tran, Du and Wang, Heng and Torresani, Lorenzo and Ray, Jamie and LeCun, Yann and Paluri, Manohar},
  pages={6450--6459},
}
```

<details>

```bibtex
}
```

<details>

```bibtex

| [pose_scnet_50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_scnet50_8xb64-210e_coco-256x192.py) |  256x192   | 0.728 |      0.899      |      0.807      | 0.784 |      0.938      | [ckpt](https://download.openmmlab.com/mmpose/top_down/scnet/scnet50_coco_256x192-6920f829_20200709.pth) | [log](https://download.openmmlab.com/mmpose/top_down/scnet/scnet50_coco_256x192_20200709.log.json) |
| [pose_scnet_50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_scnet50_8xb32-210e_coco-384x288.py) |  384x288   | 0.751 |      0.906      |      0.818      | 0.802 |      0.942      | [ckpt](https://download.openmmlab.com/mmpose/top_down/scnet/scnet50_coco_384x288-9cacd0ea_20200709.pth) | [log](https://download.openmmlab.com/mmpose/top_down/scnet/scnet50_coco_384x288_20200709.log.json) |

| [pose_scnet_101](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_scnet101_8xb32-210e_coco-256x192.py) |  256x192   | 0.733 |      0.902      |      0.811      | 0.789 |      0.940      | [ckpt](https://download.openmmlab.com/mmpose/top_down/scnet/scnet101_coco_256x192-6d348ef9_20200709.pth) | [log](https://download.openmmlab.com/mmpose/top_down/scnet/scnet101_coco_256x192_20200709.log.json) |
| [pose_scnet_101](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_scnet101_8xb48-210e_coco-384x288.py) |  384x288   | 0.752 |      0.906      |      0.823      | 0.804 |      0.943      | [ckpt](https://download.openmmlab.com/mmpose/top_down/scnet/scnet101_coco_384x288-0b6e631b_20200709.pth) | [log](https://download.openmmlab.com/mmpose/top_down/scnet/scnet101_coco_384x288_20200709.log.json) |

# SemanticKITTI Dataset

This page provides specific tutorials about the usage of MMDetection3D for SemanticKITTI dataset.

You can download SemanticKITTI dataset [HERE](http://semantic-kitti.org/dataset.html#download) and unzip all zip files.

```
├── tools
├── data
│   │   │   ├── 00
│   │   │   ├── 01
│   │   │   ├── ..
│   │   │   ├── 22
```

├── data
│   │   │   ├── 00
│   │   │   ├── 01
│   │   │   ├── ..
│   │   │   ├── 22
```

SemanticKITTI dataset contains 23 sequences, where \[0-7\], \[9-10\] are used as training set (about 19k training samples), sequence 8 as validation set (about 4k validation samples) and \[11-22\] as test set (about 20k test samples). Each sequence contains velodyne and labels folders for LIDAR point cloud data and segmentation annotations (where the high 16 bits store the instance segmentation annotations and the low 16 bits store the semantic segmentation annotations), respectively.

### Create SemanticKITTI Dataset

We support scripts that generate dataset information for training and testing. Create `.pkl` info by running:

```bash
```

```bash
```

```
├── tools
├── data
│   │   │   ├── 00
│   │   │   ├── 01
│   │   │   ├── ..
│   │   │   ├── 22
```

- `semantickitti_infos_train.pkl`: training dataset, a dict contains two keys: `metainfo` and `data_list`.
  `metainfo` contains the basic information for the dataset itself, while `data_list` is a list of dict, each dict (hereinafter referred to as `info`) contains all the detailed information of single sample as follows:
  - info\['sample_id'\]: The index of this sample in the whole dataset.

- info\['sample_id'\]: The index of this sample in the whole dataset.
  - info\['pts_semantic_mask_pth'\]: The path of 3D semantic segmentation annotation file.

Please refer to [semantickitti_converter.py](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/semantickitti_converter.py) and [update_infos_to_v2.py ](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/update_infos_to_v2.py) for more details.

A typical train pipeline of 3D segmentation on SemanticKITTI is as below:

```python
    dict(

```python
    dict(
    dict(
    dict(
    dict(
    ),
]
```

),
]
```

  - `GlobalRotScaleTrans`: rotate/scale/transform input point cloud.

An example to evaluate MinkUNet with 8 GPUs with semantickitti metrics is as follows:

```shell
```

## Metrics

Typically mean intersection over union (mIoU) is used for evaluation on Semantickitti. In detail, we first compute IoU for multiple classes and then average them to get mIoU, please refer to [seg_eval.py](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/evaluation/functional/seg_eval.py).

An example of printed evaluation results is as follows:

# LVIS

> [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://arxiv.org/abs/1908.03195)

Progress on object detection is enabled by datasets that focus the research community's attention on open challenges. This process led us from simple images to complex scenes and from bounding boxes to segmentation masks. In this work, we introduce LVIS (pronounced \`el-vis'): a new dataset for Large Vocabulary Instance Segmentation. We plan to collect ~2 million high-quality instance segmentation masks for over 1000 entry-level object categories in 164k images. Due to the Zipfian distribution of categories in natural images, LVIS naturally has a long tail of categories with few training samples. Given that state-of-the-art deep learning methods for object detection perform poorly in the low-sample regime, we believe that our dataset poses an important and exciting new scientific challenge.

<img src="https://user-images.githubusercontent.com/40661020/143967423-85b9b705-05ea-4bbc-9a41-eccc14240c7a.png" height="300"/>
</div>

## Common Setting

- Please follow [install guide](../../docs/get_started.md#install-mmdetection) to install open-mmlab forked cocoapi first.

- Run following scripts to install our forked lvis-api.

  ```shell
  ```

- All experiments use oversample strategy [here](../../docs/tutorials/customize_dataset.md#class-balanced-dataset) with oversample threshold `1e-3`.

- The size of LVIS v0.5 is half of COCO, so schedule `2x` in LVIS is roughly the same iterations as `1x` in COCO.

## Results and models of LVIS v0.5

|    Backbone     |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                               Config                               |                                                                                                                                                                                      Download                                                                                                                                                                                      |

|    R-50-FPN     | pytorch |   2x    |    -     |       -        |  26.1  |  25.9   |    [config](./mask-rcnn_r50_fpn_sample1e-3_ms-2x_lvis-v0.5.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/lvis/mask_rcnn_r50_fpn_sample1e-3_mstrain_2x_lvis/mask_rcnn_r50_fpn_sample1e-3_mstrain_2x_lvis-dbd06831.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/lvis/mask_rcnn_r50_fpn_sample1e-3_mstrain_2x_lvis/mask_rcnn_r50_fpn_sample1e-3_mstrain_2x_lvis_20200531_160435.log.json)               |

|    R-101-FPN    | pytorch |   2x    |    -     |       -        |  27.1  |  27.0   |    [config](./mask-rcnn_r101_fpn_sample1e-3_ms-2x_lvis-v0.5.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/lvis/mask_rcnn_r101_fpn_sample1e-3_mstrain_2x_lvis/mask_rcnn_r101_fpn_sample1e-3_mstrain_2x_lvis-54582ee2.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/lvis/mask_rcnn_r101_fpn_sample1e-3_mstrain_2x_lvis/mask_rcnn_r101_fpn_sample1e-3_mstrain_2x_lvis_20200601_134748.log.json)             |

| X-101-32x4d-FPN | pytorch |   2x    |    -     |       -        |  26.7  |  26.9   | [config](./mask-rcnn_x101-32x4d_fpn_sample1e-3_ms-2x_lvis-v0.5.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/lvis/mask_rcnn_x101_32x4d_fpn_sample1e-3_mstrain_2x_lvis/mask_rcnn_x101_32x4d_fpn_sample1e-3_mstrain_2x_lvis-3cf55ea2.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/lvis/mask_rcnn_x101_32x4d_fpn_sample1e-3_mstrain_2x_lvis/mask_rcnn_x101_32x4d_fpn_sample1e-3_mstrain_2x_lvis_20200531_221749.log.json) |

| X-101-64x4d-FPN | pytorch |   2x    |    -     |       -        |  26.4  |  26.0   | [config](./mask-rcnn_x101-64x4d_fpn_sample1e-3_ms-2x_lvis-v0.5.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/lvis/mask_rcnn_x101_64x4d_fpn_sample1e-3_mstrain_2x_lvis/mask_rcnn_x101_64x4d_fpn_sample1e-3_mstrain_2x_lvis-1c99a5ad.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/lvis/mask_rcnn_x101_64x4d_fpn_sample1e-3_mstrain_2x_lvis/mask_rcnn_x101_64x4d_fpn_sample1e-3_mstrain_2x_lvis_20200601_194651.log.json) |

## Results and models of LVIS v1

|    Backbone     |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                              Config                              |                                                                                                                                                                                            Download                                                                                                                                                                                            |

|    R-50-FPN     | pytorch |   1x    |   9.1    |       -        |  22.5  |  21.7   |    [config](./mask-rcnn_r50_fpn_sample1e-3_ms-1x_lvis-v1.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/lvis/mask_rcnn_r50_fpn_sample1e-3_mstrain_1x_lvis_v1/mask_rcnn_r50_fpn_sample1e-3_mstrain_1x_lvis_v1-aa78ac3d.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/lvis/mask_rcnn_r50_fpn_sample1e-3_mstrain_1x_lvis_v1/mask_rcnn_r50_fpn_sample1e-3_mstrain_1x_lvis_v1-20200829_061305.log.json)               |

|    R-101-FPN    | pytorch |   1x    |   10.8   |       -        |  24.6  |  23.6   |    [config](./mask-rcnn_r101_fpn_sample1e-3_ms-1x_lvis-v1.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/lvis/mask_rcnn_r101_fpn_sample1e-3_mstrain_1x_lvis_v1/mask_rcnn_r101_fpn_sample1e-3_mstrain_1x_lvis_v1-ec55ce32.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/lvis/mask_rcnn_r101_fpn_sample1e-3_mstrain_1x_lvis_v1/mask_rcnn_r101_fpn_sample1e-3_mstrain_1x_lvis_v1-20200829_070959.log.json)             |

| X-101-32x4d-FPN | pytorch |   1x    |   11.8   |       -        |  26.7  |  25.5   | [config](./mask-rcnn_x101-32x4d_fpn_sample1e-3_ms-1x_lvis-v1.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/lvis/mask_rcnn_x101_32x4d_fpn_sample1e-3_mstrain_1x_lvis_v1/mask_rcnn_x101_32x4d_fpn_sample1e-3_mstrain_1x_lvis_v1-ebbc5c81.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/lvis/mask_rcnn_x101_32x4d_fpn_sample1e-3_mstrain_1x_lvis_v1/mask_rcnn_x101_32x4d_fpn_sample1e-3_mstrain_1x_lvis_v1-20200829_071317.log.json) |

| X-101-64x4d-FPN | pytorch |   1x    |   14.6   |       -        |  27.2  |  25.8   | [config](./mask-rcnn_x101-64x4d_fpn_sample1e-3_ms-1x_lvis-v1.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/lvis/mask_rcnn_x101_64x4d_fpn_sample1e-3_mstrain_1x_lvis_v1/mask_rcnn_x101_64x4d_fpn_sample1e-3_mstrain_1x_lvis_v1-43d9edfe.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/lvis/mask_rcnn_x101_64x4d_fpn_sample1e-3_mstrain_1x_lvis_v1/mask_rcnn_x101_64x4d_fpn_sample1e-3_mstrain_1x_lvis_v1-20200830_060206.log.json) |

```latex
@inproceedings{gupta2019lvis,
  title={{LVIS}: A Dataset for Large Vocabulary Instance Segmentation},
  author={Gupta, Agrim and Dollar, Piotr and Girshick, Ross},
}
```

Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are available. In this paper, we systematically study model scaling and identify that carefully balancing network depth, width, and resolution can lead to better performance. Based on this observation, we propose a new scaling method that uniformly scales all dimensions of depth/width/resolution using a simple yet highly effective compound coefficient. We demonstrate the effectiveness of this method on scaling up MobileNets and ResNet.

To go even further, we use neural architecture search to design a new baseline network and scale it up to obtain a family of models, called EfficientNets, which achieve much better accuracy and efficiency than previous ConvNets. In particular, our EfficientNet-B7 achieves state-of-the-art 84.3% top-1 accuracy on ImageNet, while being 8.4x smaller and 6.1x faster on inference than the best existing ConvNet. Our EfficientNets also transfer well and achieve state-of-the-art accuracy on CIFAR-100 (91.7%), Flowers (98.8%), and 3 other transfer learning datasets, with an order of magnitude fewer parameters.

|    Backbone     |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                         Config                          |                                                                                                                                                                              Download                                                                                                                                                                              |

| :-------------: | :-----: | :-----: | :------: | :------------: | :----: | :-----------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| Efficientnet-b3 | pytorch |   1x    |    -     |       -        |  40.5  | [config](./retinanet_effb3_fpn_8xb4-crop896-1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/efficientnet/retinanet_effb3_fpn_crop896_8x4_1x_coco/retinanet_effb3_fpn_crop896_8x4_1x_coco_20220322_234806-615a0dda.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/efficientnet/retinanet_effb3_fpn_crop896_8x4_1x_coco/retinanet_effb3_fpn_crop896_8x4_1x_coco_20220322_234806.log.json) |

```latex
@article{tan2019efficientnet,
  author={Tan, Mingxing and Le, Quoc V},
  journal={arXiv preprint arXiv:1905.11946},
}
```

# 3D Hand Pose Estimation

3D hand pose estimation is defined as the task of detecting the poses (or keypoints) of the hand from an input image.

Please follow [DATA Preparation](/docs/en/dataset_zoo/3d_hand_keypoint.md) to prepare data.

# Swin transformer: Hierarchical vision transformer using shifted windows

<details>

```bibtex
}
```

This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with Shifted windows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it compatible with a broad range of vision tasks, including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO testdev) and semantic segmentation (53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-theart by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones. The hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures.

<img src="https://user-images.githubusercontent.com/40661020/143999551-6a527048-de38-485c-a1b6-3133ffa5bfaa.png">
</div>

**Text Detection**

```text

377,117,463,117,465,130,378,130,Genaxis Theatre
493,115,519,115,519,131,493,131,[06]
374,155,409,155,409,170,374,170,###
```

```text

word_1.png, "Genaxis Theatre"
word_2.png, "[06]"
word_3.png, "62-03"
```

# Pretrained Model Migration

Due to the extensive refactoring and fixing of the model structure in the new version, MMOCR 1.x does not support load weights trained by the old version. We have updated the pre-training weights and logs of all models on our website.

In addition, we are working on the development of a weight migration tool for text detection tasks and plan to release it in the near future. Since the text recognition and key information extraction models are too much modified and the migration is lossy, we do not plan to support them accordingly for the time being. If you have specific requirements, please feel free to raise an [Issue](https://github.com/open-mmlab/mmocr/issues).

# GLIP

> [Grounded Language-Image Pre-training](https://arxiv.org/abs/2112.03857)

<img src="https://github.com/microsoft/GLIP/blob/main/docs/lead.png" width="70%"/>
</div>

```python
model = get_model('swin-t_glip-pre_3rdparty', pretrained=True)
```

The pre-trained models are used to fine-tune, and therefore don't have evaluation results.

| Model                                       |          Pretrain          | resolution |                                                       Download                                                        |

| :------------------------------------------ | :------------------------: | :--------: | :-------------------------------------------------------------------------------------------------------------------: |
| GLIP-T (`swin-t_glip-pre_3rdparty`)\*       |    O365,GoldG,CC3M,SBU     |  224x224   |    [model](https://download.openmmlab.com/mmclassification/v1/glip/swin-t_glip-pre_3rdparty_20230413-d85813b5.pth)    |
| GLIP-L (`swin-l_glip-pre_3rdparty_384px`)\* | FourODs,GoldG,CC3M+12M,SBU |  384x384   | [model](https://download.openmmlab.com/mmclassification/v1/glip/swin-l_glip-pre_3rdparty_384px_20230413-04b198e8.pth) |

*Models with * are converted from the [official repo](https://github.com/microsoft/GLIP).*

```bibtex
@inproceedings{li2021grounded,
      title={Grounded Language-Image Pre-training},

```bibtex
      author={Liunian Harold Li* and Pengchuan Zhang* and Haotian Zhang* and Jianwei Yang and Chunyuan Li and Yiwu Zhong and Lijuan Wang and Lu Yuan and Lei Zhang and Jenq-Neng Hwang and Kai-Wei Chang and Jianfeng Gao},
}
```

# ISNet

[ISNet: Integrate Image-Level and Semantic-Level Context for Semantic Segmentation](https://arxiv.org/pdf/2108.12382.pdf)

This is an implementation of [ISNet](https://arxiv.org/pdf/2108.12382.pdf).
[Official Repo](https://github.com/SegmentationBLWX/sssegmentation)

## Usage

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `isnet/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
```

```shell
mim train mmsegmentation configs/isnet_r50-d8_8xb2-160k_cityscapes-512x1024.py --work-dir work_dirs/isnet
```

```shell
```

```shell
```

```shell
mim train mmsegmentation configs/isnet_r50-d8_8xb2-160k_cityscapes-512x1024.py --work-dir work_dirs/isnet --launcher pytorch --gpus 8
```

```shell
mim test mmsegmentation configs/isnet_r50-d8_8xb2-160k_cityscapes-512x1024.py --work-dir work_dirs/isnet --checkpoint ${CHECKPOINT_PATH}
```

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) |  mIoU | mIoU(ms+flip) | config                                                          | download                                                                                                                 |

| ------ | -------- | --------- | ------: | -------- | -------------- | ----: | ------------: | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| ISNet  | R-50-D8  | 512x1024  |       - | -        | -              | 79.32 |         80.88 | [config](configs/isnet_r50-d8_8xb2-160k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/isnet/isnet_r50-d8_cityscapes-512x1024_20230104-a7a8ccf2.pth) |

```bibtex
@article{Jin2021ISNetII,
  title={ISNet: Integrate Image-Level and Semantic-Level Context for Semantic Segmentation},
  author={Zhenchao Jin and B. Liu and Qi Chu and Nenghai Yu},
  journal={2021 IEEE/CVF International Conference on Computer Vision (ICCV)},
  pages={7169-7178}
}
```

The progress of ISNet.

journal={2021 IEEE/CVF International Conference on Computer Vision (ICCV)},
}
```

<!-- The PIC (person in charge) or contributors of this project should check all the items that they believe have been finished, which will further be verified by codebase maintainers via a PR.

  <!-- The code's design shall follow existing interfaces and convention. For example, each model component should be registered into `mmseg.registry.MODELS` and configurable via a config file. -->

  <!-- Each major object should contain a docstring, describing its functionality and arguments. If you have adapted the code from other open-source projects, don't forget to cite the source project in docstring and make sure your behavior is not against its license. Typically, we do not accept any code snippet under GPL license. [A Short Guide to Open Source Licenses](https://medium.com/nationwide-technology/a-short-guide-to-open-source-licenses-cf5b1c329edd) -->

  <!-- If you are reproducing the result from a paper, make sure your model's inference-time performance matches that in the original paper. The weights usually could be obtained by simply renaming the keys in the official pre-trained weights. This test could be skipped though, if you are able to prove the training-time correctness and check the second milestone. -->

  <!-- As this template does. -->

  <!-- If you are reproducing the result from a paper, checking this item means that you should have trained your model from scratch based on the original paper's specification and verified that the final result matches the report within a minor error range. -->

  <!-- Ideally *all* the methods should have [type hints](https://www.pythontutorial.net/python-basics/python-type-hints/) and [docstrings](https://google.github.io/styleguide/pyguide.html#381-docstrings). [Example](https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/utils/io.py#L9) -->

  <!-- Unit tests for each module are required. [Example](https://github.com/open-mmlab/mmsegmentation/blob/main/tests/test_utils/test_io.py#L14) -->

  <!-- Refactor your code according to reviewer's comment. -->

  <!-- It will be parsed by MIM and Inferencer. [Example](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/fcn.yml) -->

  <!-- In particular, you may have to refactor this README into a standard one. [Example](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/fcn/README.md) -->

# Data structure in MMagic \[Coming Soon!\]

<details>
<summary align="right"><a href="https://arxiv.org/abs/2111.12704">RealBasicVSR (CVPR'2022)</a></summary>

```bibtex
}
```

<br/>

在 Y 通道上评估。 计算 NRQM、NIQE 和 PI 的代码可以在[这里](https://github.com/roimehrez/PIRM2018)找到。我们使用 MATLAB 官方代码计算 BRISQUE。

|                                 算法                                 | NRQM (Y) | NIQE (Y) | PI (Y) | BRISQUE (Y) |         GPU 信息         |                                  Download                                   |

| :------------------------------------------------------------------: | :------: | :------: | :----: | :---------: | :----------------------: | :-------------------------------------------------------------------------: |
| [realbasicvsr_c64b20_1x30x8_lr5e-5_150k_reds](./realbasicvsr_c64b20-1x30x8_8xb1-lr5e-5-150k_reds.py) |  6.0477  |  3.7662  | 3.8593 |   29.030    | 8 (Tesla V100-SXM2-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/real_basicvsr/realbasicvsr_c64b20_1x30x8_lr5e-5_150k_reds_20211104-52f77c2c.pth)/[log](https://download.openmmlab.com/mmediting/restorers/real_basicvsr/realbasicvsr_c64b20_1x30x8_lr5e-5_150k_reds_20211104_183640.log.json) |

| [realbasicvsr_wogan-c64b20-2x30x8_8xb2-lr1e-4-300k_reds](./realbasicvsr_wogan-c64b20-2x30x8_8xb2-lr1e-4-300k_reds.py) |    -     |    -     |   -    |      -      | 8 (Tesla V100-SXM2-32GB) | [model](http://download.openmmlab.com/mmediting/restorers/real_basicvsr/realbasicvsr_wogan_c64b20_2x30x8_lr1e-4_300k_reds_20211027-0e2ff207.pth)/[log](http://download.openmmlab.com/mmediting/restorers/real_basicvsr/realbasicvsr_wogan_c64b20_2x30x8_lr1e-4_300k_reds_20211027_114039.log.json) |

## 训练

训练分为两个阶段：

1. 使用 [realbasicvsr_wogan-c64b20-2x30x8_8xb2-lr1e-4-300k_reds.py](realbasicvsr_wogan-c64b20-2x30x8_8xb2-lr1e-4-300k_reds.py) 训练一个没有感知损失和对抗性损失的模型。
2. 使用感知损失和对抗性损失 [realbasicvsr_c64b20-1x30x8_8xb1-lr5e-5-150k_reds.py ](realbasicvsr_c64b20-1x30x8_8xb1-lr5e-5-150k_reds.py) 微调模型。

**注:**

1. 您可能希望将图像裁剪为子图像以加快 IO。请参阅[此处](../../tools/dataset_converters/reds/preprocess_reds_dataset.py)了解更多详情。

## 快速开始

**训练**

<details>

```shell
# CPU上训练

**注:**

## 快速开始

**训练**

<details>

```shell
# CPU上训练

# 单个GPU上训练

# 多个GPU上训练
```

**测试**

<details>

```shell
# CPU上测试

# 单个GPU上测试

# 单个GPU上测试
python tools/test.py configs/real_basicvsr/realbasicvsr_c64b20-1x30x8_8xb1-lr5e-5-150k_reds.py https://download.openmmlab.com/mmediting/restorers/real_basicvsr/realbasicvsr_c64b20_1x30x8_lr5e-5_150k_reds_20211104-52f77c2c.pth

# 多个GPU上测试
```

# How to support new backends

MMDeploy supports a number of backend engines. We welcome the contribution of new backends. In this tutorial, we will introduce the general procedures to support a new backend in MMDeploy.

Before contributing the codes, there are some requirements for the new backend that need to be checked:

- The backend must support ONNX as IR.
- If the backend requires model files or weight files other than a ".onnx" file, a conversion tool that converts the ".onnx" file to model files and weight files is required. The tool can be a Python API, a script, or an executable program.
- It is highly recommended that the backend provides a Python interface to load the backend files and inference for validation.

## Support backend conversion

The backends in MMDeploy must support the ONNX. The backend loads the ".onnx" file directly, or converts the ".onnx" to its own format using the conversion tool. In this section, we will introduce the steps to support backend conversion.

1. Add backend constant in `mmdeploy/utils/constants.py` that denotes the name of the backend.

   **Example**:

   ```Python

       # Take TensorRT as an example
   ```

2. Add a corresponding package (a folder with `__init__.py`) in `mmdeploy/backend/`. For example, `mmdeploy/backend/tensorrt`. In the `__init__.py`, there must be a function named `is_available` which checks if users have installed the backend library. If the check is passed, then the remaining files of the package will be loaded.

**Example**:

   ```Python

       __all__ = [
       ]
   ```

3. Create a config file in `configs/_base_/backends` (e.g., `configs/_base_/backends/tensorrt.py`).  If the backend just takes the '.onnx' file as input, the new config can be simple. The config of the backend only consists of one field denoting the name of the backend (which should be same as the name in `mmdeploy/utils/constants.py`).

   ```python
   backend_config = dict(type='onnxruntime')
   ```

   If the backend requires other files, then the arguments for the conversion from ".onnx" file to backend files should be included in the config file.

   **Example:**

   ```Python

**Example:**

   ```Python

   ```

   After possessing a base backend config file, you can easily construct a complete deploy config through inheritance. Please refer to our [config tutorial](../02-how-to-run/write_config.md) for more details. Here is an example:

   ```Python
   _base_ = ['../_base_/backends/onnxruntime.py']

   ```

```Python

   ```

4. If the backend requires model files or weight files other than a ".onnx" file, create a `onnx2backend.py` file in the corresponding folder (e.g., create `mmdeploy/backend/tensorrt/onnx2tensorrt.py`). Then add a conversion function `onnx2backend` in the file. The function should convert a given ".onnx" file to the required backend files in a given work directory. There are no requirements on other parameters of the function and the implementation details. You can use any tools for conversion. Here are some examples:

   **Use Python script:**

   ```Python

output_names: List[str], onnx_path: str, work_dir: str):

   ```

   **Use executable program:**

   ```Python
   ```

5. Define APIs in a new package in  `mmdeploy/apis`.

save_param, save_bin = get_output_model_file(onnx_path, work_dir)
   ```

   ```Python

   ```

   Create a backend manager class which derive from `BaseBackendManager`, implement its `to_backend` static method.

   ```Python
    @classmethod
                ir_files: Sequence[str],
                deploy_cfg: Any,
                work_dir: str,
                log_level: int = logging.INFO,
                device: str = 'cpu',
                **kwargs) -> Sequence[str]:

deploy_cfg: Any,
   ```

6. Convert the models of OpenMMLab to backends (if necessary) and inference on backend engine. If you find some incompatible operators when testing, you can try to rewrite the original model for the backend following the [rewriter tutorial](support_new_model.md) or add custom operators.

7. Add docstring and unit tests for new code :).

## Support backend inference

Although the backend engines are usually implemented in C/C++, it is convenient for testing and debugging if the backend provides Python inference interface. We encourage the contributors to support backend inference in the Python interface of MMDeploy. In this section we will introduce the steps to support backend inference.

1. Add a file named `wrapper.py` to corresponding folder in `mmdeploy/backend/{backend}`. For example, `mmdeploy/backend/tensorrt/wrapper.py`. This module should implement and register a wrapper class that inherits the base class `BaseWrapper` in `mmdeploy/backend/base/base_wrapper.py`.

   ```Python

   ```

2. The wrapper class can initialize the engine in `__init__` function and inference in `forward` function. Note that the `__init__` function must take a parameter `output_names` and pass it to base class to determine the orders of output tensors. The input and output variables of `forward` should be dictionaries denoting the name and value of the tensors.

3. For the convenience of performance testing, the class should define a "execute" function that only calls the inference interface of the backend engine. The `forward` function should call the "execute" function after preprocessing the data.

   ```Python

           # ...

           # ...

torch.Tensor]) -> Dict[str, torch.Tensor]:
           # ...

           # ...

   ```

4. Create a backend manager class which derive from `BaseBackendManager`, implement its `build_wrapper` static method.

   ```Python

input_names: Optional[Sequence[str]] = None,
   ```

5. Add docstring and unit tests for new code :).

## Support new backends using MMDeploy as a third party

Previous parts show how to add a new backend in MMDeploy, which requires changing its source codes. However, if we treat MMDeploy as a third party, the methods above are no longer efficient. To this end, adding a new backend requires us pre-install another package named `aenum`. We can install it directly through `pip install aenum`.

After installing `aenum` successfully, we can use it to add a new backend through:

```python

try:
```

We can run the codes above before we use the rewrite logic of MMDeploy.

# 准备 Kinetics-\[400/600/700\]

## 简介

```BibTeX
}
```

请参照 [官方网站](https://deepmind.com/research/open-source/open-source-datasets/kinetics/) 以获取数据集基本信息。

:::{note}
由于部分 YouTube 链接失效，爬取的 Kinetics 数据集大小可能与原版不同。以下是我们所使用 Kinetics 数据集的大小：

|   数据集    | 训练视频 | 验证集视频 |
| :---------: | :------: | :--------: |
| Kinetics400 |  240436  |   19796    |
| Kinetics600 |  383393  |   27910    |
| Kinetics700 |  542357  |   34824    |
|     :::     |          |            |

:::{note}
MMAction2 代码仓库中提供的 Kinetics 实验性能，都是基于这个版本的数据得到的。我们建议用户使用这个版本的 Kinetics 数据集进行实验。
:::

# MIM 支持下载 Kinetics-400/600/700 数据集。用户可以通过一行命令，从 OpenDataLab 进行下载，并进行预处理。
```Bash

:::

```Bash
odl login
# 通过 MIM 进行 Kinetics-400 数据集下载，预处理。注意这将花费较长时间
# 通过 MIM 进行 Kinetics-600 数据集下载，预处理。注意这将花费较长时间
# 通过 MIM 进行 Kinetics-700 数据集下载，预处理。注意这将花费较长时间
```

````

此脚本用于准备数据集 kinetics400，kinetics600，kinetics700。为准备 kinetics 数据集的不同版本，用户需将脚本中的 `${DATASET}` 赋值为数据集对应版本名称，可选项为 `kinetics400`，`kinetics600`， `kinetics700`。
在开始之前，用户需确保当前目录为 `$MMACTION2/tools/data/${DATASET}/`。
首先，用户可以使用如下脚本从 [Kinetics 数据集官网](https://deepmind.com/research/open-source/open-source-datasets/kinetics/)下载标注文件并进行预处理：

```shell
```

由于部分视频的 URL 不可用，当前官方标注中所含视频数量可能小于初始版本。所以 MMAction2 提供了另一种方式以获取初始版本标注作为参考。

```shell
```

在这其中，Kinetics400 和 Kinetics600 的标注文件来自 [官方爬虫](https://github.com/activitynet/ActivityNet/tree/199c9358907928a47cdfc81de4db788fddc2f91d/Crawler/Kinetics/data)，
Kinetics700 的标注文件于 05/02/2021 下载自 [网站](https://deepmind.com/research/open-source/open-source-datasets/kinetics/)。

```shell
```

## 2. 准备视频

用户可以使用以下脚本准备视频，视频准备代码修改自 [官方爬虫](https://github.com/activitynet/ActivityNet/tree/master/Crawler/Kinetics)。注意这一步骤将花费较长时间。

```shell
```

**重要提示**：如果在此之前已下载好 Kinetics 数据集的视频，还需使用重命名脚本来替换掉类名中的空格：

```shell
```

为提升解码速度，用户可以使用以下脚本将原始视频缩放至更小的分辨率（利用稠密编码）：

```bash
```

```

```bash
```

也可以从 [Academic Torrents](https://academictorrents.com/) 中下载短边长度为 256 的 [kinetics400](https://academictorrents.com/details/184d11318372f70018cf9a72ef867e2fb9ce1d26) 和 [kinetics700](https://academictorrents.com/details/49f203189fb69ae96fb40a6d0e129949e1dfec98)，或从 Common Visual Data Foundation 维护的 [cvdfoundation/kinetics-dataset](https://github.com/cvdfoundation/kinetics-dataset) 中下载 Kinetics400/Kinetics600/Kinetics-700-2020。

```shell

```shell
```

如果用户只使用 RGB 帧（由于光流提取非常耗时），可以考虑执行以下脚本，仅用 denseflow 提取 RGB 帧：

```shell
```

如果用户未安装 denseflow，以下脚本可以使用 OpenCV 进行 RGB 帧的提取，但视频原分辨率大小会被保留：

```shell
```

```shell
```

以上的命令生成短边长度为 256 的 RGB 帧和光流帧。如果用户需要生成短边长度为 320 的帧 (320p)，或是固定分辨率为 340 x 256 的帧，可改变参数 `--new-short 256` 为 `--new-short 320` 或 `--new-width 340 --new-height 256`。
更多细节可以参考 [数据准备](/docs/zh_cn/user_guides/prepare_dataset.md)。

```shell

```shell
```

````
`````

### 目录结构

在完整完成 Kinetics 的数据处理后，将得到帧文件夹（RGB 帧和光流帧），视频以及标注文件。

在整个项目目录下（仅针对 Kinetics），*最简* 目录结构如下所示：

```
mmaction2
├── tools
├── data
│   │   │   ├── ...
│   │   │   ├── ...
│   │   ├── rawframes_train（可选）
│   │   ├── rawframes_val（可选）

```

关于 Kinetics 数据集上的训练与测试，请参照 [训练教程](/docs/zh_cn/user_guides/train_test.md)。

# TANet

[TAM: Temporal Adaptive Module for Video Recognition](https://openaccess.thecvf.com/content/ICCV2021/html/Liu_TAM_Temporal_Adaptive_Module_for_Video_Recognition_ICCV_2021_paper.html)

Video data is with complex temporal dynamics due to various factors such as camera motion, speed variation, and different activities. To effectively capture this diverse motion pattern, this paper presents a new temporal adaptive module ({\\bf TAM}) to generate video-specific temporal kernels based on its own feature map. TAM proposes a unique two-level adaptive modeling scheme by decoupling the dynamic kernel into a location sensitive importance map and a location invariant aggregation weight. The importance map is learned in a local temporal window to capture short-term information, while the aggregation weight is generated from a global view with a focus on long-term structure. TAM is a modular block and could be integrated into 2D CNNs to yield a powerful video architecture (TANet) with a very small extra computational cost. The extensive experiments on Kinetics-400 and Something-Something datasets demonstrate that our TAM outperforms other temporal modeling methods consistently, and achieves the state-of-the-art performance under the similar complexity.

<img src="https://user-images.githubusercontent.com/34324155/143018253-c3e1ba5b-ac35-4c55-be28-0134b76888e8.png" width="800"/>
</div>

| frame sampling strategy | resolution | gpus | backbone | pretrain | top1 acc | top5 acc |      reference top1 acc       |      reference top5 acc       | testing protocol | FLOPs | params |      config       |      ckpt       |      log       |
| :---------------------: | :--------: | :--: | :------: | :------: | :------: | :------: | :---------------------------: | :---------------------------: | :--------------: | :---: | :----: | :---------------: | :-------------: | :------------: |

|       dense-1x1x8       |  224x224   |  8   | ResNet50 | ImageNet |  76.25   |  92.41   | [76.22](https://github.com/liu-zhy/temporal-adaptive-module/blob/master/scripts/test_tam_kinetics_rgb_8f.sh) | [92.53](https://github.com/liu-zhy/temporal-adaptive-module/blob/master/scripts/test_tam_kinetics_rgb_8f.sh) | 8 clips x 3 crop | 43.0G | 25.6M  | [config](/configs/recognition/tanet/tanet_imagenet-pretrained-r50_8xb8-dense-1x1x8-100e_kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tanet/tanet_imagenet-pretrained-r50_8xb8-dense-1x1x8-100e_kinetics400-rgb/tanet_imagenet-pretrained-r50_8xb8-dense-1x1x8-100e_kinetics400-rgb_20220919-a34346bc.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tanet/tanet_imagenet-pretrained-r50_8xb8-dense-1x1x8-100e_kinetics400-rgb/tanet_imagenet-pretrained-r50_8xb8-dense-1x1x8-100e_kinetics400-rgb.log) |

| frame sampling strategy | resolution | gpus | backbone | pretrain |  top1 acc   |  top5 acc   | testing protocol  | FLOPs | params |               config               |               ckpt               |               log               |
| :---------------------: | :--------: | :--: | :------: | :------: | :---------: | :---------: | :---------------: | :---: | :----: | :--------------------------------: | :------------------------------: | :-----------------------------: |

|          1x1x8          |  224x224   |  8   | ResNet50 | ImageNet | 46.98/49.71 | 75.75/77.43 | 16 clips x 3 crop | 43.1G | 25.1M  | [config](/configs/recognition/tanet/tanet_imagenet-pretrained-r50_8xb8-1x1x8-50e_sthv1-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tanet/tanet_imagenet-pretrained-r50_8xb8-1x1x8-50e_sthv1-rgb/tanet_imagenet-pretrained-r50_8xb8-1x1x8-50e_sthv1-rgb_20220906-de50e4ef.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tanet/tanet_imagenet-pretrained-r50_8xb8-1x1x8-50e_sthv1-rgb/tanet_imagenet-pretrained-r50_8xb8-1x1x8-50e_sthv1-rgb.log) |

|         1x1x16          |  224x224   |  8   | ResNet50 | ImageNet | 48.24/50.95 | 78.16/79.28 | 16 clips x 3 crop | 86.1G | 25.1M  | [config](/configs/recognition/tanet/tanet_imagenet-pretrained-r50_8xb6-1x1x16-50e_sthv1-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tanet/tanet_imagenet-pretrained-r50_8xb6-1x1x16-50e_sthv1-rgb/tanet_imagenet-pretrained-r50_8xb6-1x1x16-50e_sthv1-rgb_20220919-cc37e9b8.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tanet/tanet_imagenet-pretrained-r50_8xb6-1x1x16-50e_sthv1-rgb/tanet_imagenet-pretrained-r50_8xb6-1x1x16-50e_sthv1-rgb.log) |

2. The values in columns named after "reference" are the results got by testing on our dataset, using the checkpoints provided by the author with same model settings. The checkpoints for reference repo can be downloaded [here](https://drive.google.com/drive/folders/1sFfmP3yrfc7IzRshEELOby7-aEoymIFL?usp=sharing).

- [Kinetics400](/tools/data/kinetics/README.md)

## Train

```shell
```

Example: train TANet model on Kinetics-400 dataset in a deterministic option with periodic validation.

```shell
python tools/train.py configs/recognition/tanet/tanet_imagenet-pretrained-r50_8xb8-dense-1x1x8-100e_kinetics400-rgb.py \
```

## Test

```shell
```

Example: test TANet model on Kinetics-400 dataset and dump the result to a pkl file.

```shell

```shell
```

```shell
python tools/test.py configs/recognition/tanet/tanet_imagenet-pretrained-r50_8xb8-dense-1x1x8-100e_kinetics400-rgb.py \
```

```BibTeX
@article{liu2020tam,
  title={TAM: Temporal Adaptive Module for Video Recognition},
  author={Liu, Zhaoyang and Wang, Limin and Wu, Wayne and Qian, Chen and Lu, Tong},
  journal={arXiv preprint arXiv:2005.06803},
}
```

# GCNet

<a href="https://github.com/xvjiarui/GCNet">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/decode_heads/gc_head.py#L10">Code Snippet</a>

The Non-Local Network (NLNet) presents a pioneering approach for capturing long-range dependencies, via aggregating query-specific global context to each query position. However, through a rigorous empirical analysis, we have found that the global contexts modeled by non-local network are almost the same for different query positions within an image. In this paper, we take advantage of this finding to create a simplified network based on a query-independent formulation, which maintains the accuracy of NLNet but with significantly less computation. We further observe that this simplified design shares similar structure with Squeeze-Excitation Network (SENet). Hence we unify them into a three-step general framework for global context modeling. Within the general framework, we design a better instantiation, called the global context (GC) block, which is lightweight and can effectively model the global context. The lightweight property allows us to apply it for multiple layers in a backbone network to construct a global context network (GCNet), which generally outperforms both simplified NLNet and SENet on major benchmarks for various recognition tasks. The code and configurations are released at [this https URL](https://github.com/xvjiarui/GCNet).

<img src="https://user-images.githubusercontent.com/24582831/142901601-ad17922e-2538-4b48-9f51-84a57d44b12b.png" width="80%"/>
</div>

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                        | download                                                                                                                                                                                                                                                                                                                                           |

| GCNet  | R-50-D8  | 512x1024  |   40000 | 5.8      | 3.93           | V100   | 77.69 |         78.56 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/gcnet/gcnet_r50-d8_4xb2-40k_cityscapes-512x1024.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r50-d8_512x1024_40k_cityscapes/gcnet_r50-d8_512x1024_40k_cityscapes_20200618_074436-4b0fd17b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r50-d8_512x1024_40k_cityscapes/gcnet_r50-d8_512x1024_40k_cityscapes_20200618_074436.log.json)     |

| GCNet  | R-101-D8 | 512x1024  |   40000 | 9.2      | 2.61           | V100   | 78.28 |         79.34 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/gcnet/gcnet_r101-d8_4xb2-40k_cityscapes-512x1024.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r101-d8_512x1024_40k_cityscapes/gcnet_r101-d8_512x1024_40k_cityscapes_20200618_074436-5e62567f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r101-d8_512x1024_40k_cityscapes/gcnet_r101-d8_512x1024_40k_cityscapes_20200618_074436.log.json) |

| GCNet  | R-50-D8  | 769x769   |   40000 | 6.5      | 1.67           | V100   | 78.12 |         80.09 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/gcnet/gcnet_r50-d8_4xb2-40k_cityscapes-769x769.py)    | [model](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r50-d8_769x769_40k_cityscapes/gcnet_r50-d8_769x769_40k_cityscapes_20200618_182814-a26f4471.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r50-d8_769x769_40k_cityscapes/gcnet_r50-d8_769x769_40k_cityscapes_20200618_182814.log.json)         |

| GCNet  | R-101-D8 | 769x769   |   40000 | 10.5     | 1.13           | V100   | 78.95 |         80.71 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/gcnet/gcnet_r101-d8_4xb2-40k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r101-d8_769x769_40k_cityscapes/gcnet_r101-d8_769x769_40k_cityscapes_20200619_092550-ca4f0a84.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r101-d8_769x769_40k_cityscapes/gcnet_r101-d8_769x769_40k_cityscapes_20200619_092550.log.json)     |

| GCNet  | R-50-D8  | 512x1024  |   80000 | -        | -              | V100   | 78.48 |         80.01 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/gcnet/gcnet_r50-d8_4xb2-80k_cityscapes-512x1024.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r50-d8_512x1024_80k_cityscapes/gcnet_r50-d8_512x1024_80k_cityscapes_20200618_074450-ef8f069b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r50-d8_512x1024_80k_cityscapes/gcnet_r50-d8_512x1024_80k_cityscapes_20200618_074450.log.json)     |

| GCNet  | R-101-D8 | 512x1024  |   80000 | -        | -              | V100   | 79.03 |         79.84 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/gcnet/gcnet_r101-d8_4xb2-80k_cityscapes-512x1024.pyy) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r101-d8_512x1024_80k_cityscapes/gcnet_r101-d8_512x1024_80k_cityscapes_20200618_074450-778ebf69.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r101-d8_512x1024_80k_cityscapes/gcnet_r101-d8_512x1024_80k_cityscapes_20200618_074450.log.json) |

| GCNet  | R-50-D8  | 769x769   |   80000 | -        | -              | V100   | 78.68 |         80.66 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/gcnet/gcnet_r50-d8_4xb2-80k_cityscapes-769x769.py)    | [model](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r50-d8_769x769_80k_cityscapes/gcnet_r50-d8_769x769_80k_cityscapes_20200619_092516-4839565b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r50-d8_769x769_80k_cityscapes/gcnet_r50-d8_769x769_80k_cityscapes_20200619_092516.log.json)         |

| GCNet  | R-101-D8 | 769x769   |   80000 | -        | -              | V100   | 79.18 |         80.71 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/gcnet/gcnet_r101-d8_4xb2-80k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r101-d8_769x769_80k_cityscapes/gcnet_r101-d8_769x769_80k_cityscapes_20200619_092628-8e043423.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r101-d8_769x769_80k_cityscapes/gcnet_r101-d8_769x769_80k_cityscapes_20200619_092628.log.json)     |

### ADE20K

| GCNet  | R-50-D8  | 512x512   |   80000 | 8.5      | 23.38          | V100   | 41.47 |         42.85 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/gcnet/gcnet_r50-d8_4xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r50-d8_512x512_80k_ade20k/gcnet_r50-d8_512x512_80k_ade20k_20200614_185146-91a6da41.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r50-d8_512x512_80k_ade20k/gcnet_r50-d8_512x512_80k_ade20k_20200614_185146.log.json)         |

| GCNet  | R-101-D8 | 512x512   |   80000 | 12       | 15.20          | V100   | 42.82 |         44.54 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/gcnet/gcnet_r101-d8_4xb4-80k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r101-d8_512x512_80k_ade20k/gcnet_r101-d8_512x512_80k_ade20k_20200615_020811-c3fcb6dd.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r101-d8_512x512_80k_ade20k/gcnet_r101-d8_512x512_80k_ade20k_20200615_020811.log.json)     |

| GCNet  | R-50-D8  | 512x512   |  160000 | -        | -              | V100   | 42.37 |         43.52 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/gcnet/gcnet_r50-d8_4xb4-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r50-d8_512x512_160k_ade20k/gcnet_r50-d8_512x512_160k_ade20k_20200615_224122-d95f3e1f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r50-d8_512x512_160k_ade20k/gcnet_r50-d8_512x512_160k_ade20k_20200615_224122.log.json)     |

| GCNet  | R-101-D8 | 512x512   |  160000 | -        | -              | V100   | 43.69 |         45.21 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/gcnet/gcnet_r101-d8_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r101-d8_512x512_160k_ade20k/gcnet_r101-d8_512x512_160k_ade20k_20200615_225406-615528d7.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r101-d8_512x512_160k_ade20k/gcnet_r101-d8_512x512_160k_ade20k_20200615_225406.log.json) |

| GCNet  | R-50-D8  | 512x512   |   20000 | 5.8      | 23.35          | V100   | 76.42 |         77.51 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/gcnet/gcnet_r50-d8_4xb4-20k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r50-d8_512x512_20k_voc12aug/gcnet_r50-d8_512x512_20k_voc12aug_20200617_165701-3cbfdab1.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r50-d8_512x512_20k_voc12aug/gcnet_r50-d8_512x512_20k_voc12aug_20200617_165701.log.json)     |

| GCNet  | R-101-D8 | 512x512   |   20000 | 9.2      | 14.80          | V100   | 77.41 |         78.56 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/gcnet/gcnet_r101-d8_4xb4-20k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r101-d8_512x512_20k_voc12aug/gcnet_r101-d8_512x512_20k_voc12aug_20200617_165713-6c720aa9.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r101-d8_512x512_20k_voc12aug/gcnet_r101-d8_512x512_20k_voc12aug_20200617_165713.log.json) |

| GCNet  | R-50-D8  | 512x512   |   40000 | -        | -              | V100   | 76.24 |         77.63 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/gcnet/gcnet_r50-d8_4xb4-40k_voc12aug-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r50-d8_512x512_40k_voc12aug/gcnet_r50-d8_512x512_40k_voc12aug_20200613_195105-9797336d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r50-d8_512x512_40k_voc12aug/gcnet_r50-d8_512x512_40k_voc12aug_20200613_195105.log.json)     |

| GCNet  | R-101-D8 | 512x512   |   40000 | -        | -              | V100   | 77.84 |         78.59 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/gcnet/gcnet_r101-d8_4xb4-40k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r101-d8_512x512_40k_voc12aug/gcnet_r101-d8_512x512_40k_voc12aug_20200613_185806-1e38208d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/gcnet/gcnet_r101-d8_512x512_40k_voc12aug/gcnet_r101-d8_512x512_40k_voc12aug_20200613_185806.log.json) |

```bibtex
@inproceedings{cao2019gcnet,
  title={Gcnet: Non-local networks meet squeeze-excitation networks and beyond},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision Workshops},
}
```

In the Supervised Fine-Tuning (SFT) process of Language Model Learning (LLM), we often inject some predefined strings into the conversation according to actual requirements, in order to prompt the model to output content according to certain guidelines. For example, in some `chat` model fine-tuning, we may add system-level instructions at the beginning of each dialogue, and establish a format to represent the conversation between the user and the model. In a conversation, the model may expect the text format to be as follows:

```bash
```

During evaluation, we also need to enter questions according to the agreed format for the model to perform its best.

```

In addition, similar situations exist in API models. General API dialogue models allow users to pass in historical dialogues when calling, and some models also allow the input of SYSTEM level instructions. To better evaluate the ability of API models, we hope to make the data as close as possible to the multi-round dialogue template of the API model itself during the evaluation, rather than stuffing all the content into an instruction.

Therefore, we need to specify different parsing templates for different models. In OpenCompass, we call this set of parsing templates **Meta Template**. Meta Template is tied to the model's configuration and is combined with the dialogue template of the dataset during runtime to ultimately generate the most suitable prompt for the current model.

```python

```python
# When specifying, just pass the meta_template field into the model
models = [
    dict(
        meta_template = ...,  # meta template
    )
]
```

Next, we will introduce how to configure Meta Template on two types of models.
You are recommended to read [here](./prompt_template.md#dialogue-prompt) for the basic syntax of the dialogue template before reading this chapter.

```{note}
In some cases (such as testing the base station), we don't need to inject any instructions into the normal dialogue, in which case we can leave the meta template empty. In this case, the prompt received by the model is defined only by the dataset configuration and is a regular string. If the dataset configuration uses a dialogue template, speeches from different roles will be concatenated with \n.
```

## Application on Language Models

```

The following figure shows several situations where the data is built into a prompt through the prompt template and meta template from the dataset in the case of 2-shot learning. Readers can use this figure as a reference to help understand the following sections.

We will explain how to define the meta template with several examples.

Suppose that according to the dialogue template of the dataset, the following dialogue was produced:

```python
])
```

```python
])
```

We want to pass this dialogue to a model that has already gone through SFT. The model's agreed dialogue begins with the speech of different roles with `<Role Name>:` and ends with a special token and \\n. Here is the complete string the model expects to receive:

```Plain
```

In the meta template, we only need to abstract the format of each round of dialogue into the following configuration:

```python
    round=[
    ],
 )
```

Some datasets may introduce SYSTEM-level roles:

```python

],
 )
```

```python
])
```

Assuming the model also accepts the SYSTEM role, and expects the input to be:

```
```

We can put the definition of the SYSTEM role into `reserved_roles`. Roles in `reserved_roles` will not appear in regular conversations, but they allow the dialogue template of the dataset configuration to call them in `begin` or `end`.

```python
    round=[

```python
    round=[
    ],
 ),
```

If the model does not accept the SYSTEM role, it is not necessary to configure this item, and it can still run normally. In this case, the string received by the model becomes:

```
```

This is because in the predefined datasets in OpenCompass, each `SYSTEM` speech has a `fallback_role='HUMAN'`, that is, if the `SYSTEM` role in the meta template does not exist, the speaker will be switched to the `HUMAN` role.

Some models may need to consider embedding other strings at the beginning or end of the conversation, such as system instructions:

```
```

In this case, we can specify these strings by specifying the begin and end parameters.

```python
    round=[
    ],
    end="end of conversation",
 ),
```

begin="Meta instruction: You are now a helpful and harmless AI assistant.",
 ),
```

In **generative** task evaluation, we will not directly input the answer to the model, but by truncating the prompt, while retaining the previous text, we leave the answer output by the model blank.

```
<BOT>:
```

We only need to set the `generate` field in BOT's configuration to True, and OpenCompass will automatically leave the last utterance of BOT blank:

```python
    round=[
    ],

    round=[
    ],
 ),
```

Note that `generate` only affects generative inference. When performing discriminative inference, the prompt received by the model is still complete.

### Full Definition

```bash
models = [
            round=[

round=[
                    dict(role='THOUGHTS', begin='THOUGHTS: ', end='<eot>\n', prompt='None'), # Here we can set the default prompt, which may be overridden by the specific dataset
            ],
         ),
     )
]
```

The `meta_template` is a dictionary that can contain the following fields:

- `begin`, `end`: (str, optional) The beginning and ending of the prompt, typically some system-level instructions.

- `round`: (list) The template format of each round of dialogue. The content of the prompt for each round of dialogue is controlled by the dialogue template configured in the dataset.

- `reserved_roles`: (list, optional) Specify roles that do not appear in `round` but may be used in the dataset configuration, such as the `SYSTEM` role.

- `eos_token_id`: (int, optional): Specifies the ID of the model's eos token. If not set, it defaults to the eos token id in the tokenizer. Its main role is to trim the output of the model in generative tasks, so it should generally be set to the first token id of the end corresponding to the item with generate=True.

The `round` of the `meta_template` specifies the format of each role's speech in a round of dialogue. It accepts a list of dictionaries, each dictionary's keys are as follows:

- `role` (str): The name of the role participating in the dialogue. This string does not affect the actual prompt.

- `begin`, `end` (str): Specifies the fixed beginning or end when this role speaks.

- `prompt` (str): The role's prompt. It is allowed to leave it blank in the meta template, but in this case, it must be specified in the prompt of the dataset configuration.

- `generate` (bool): When specified as True, this role is the one the model plays. In generation tasks, the prompt received by the model will be cut off at the `begin` of this role, and the remaining content will be filled by the model.

## Application to API Models

The meta template of the API model is similar to the meta template of the general model, but the configuration is simpler. Users can, as per their requirements, directly use one of the two configurations below to evaluate the API model in a multi-turn dialogue manner:

```bash
# If the API model does not support system instructions

```bash
    round=[
    ],
)

# If the API model supports system instructions
    round=[
    ],
    ],
)
```

### Principle

Even though different API models accept different data structures, there are commonalities overall. Interfaces that accept dialogue history generally allow users to pass in prompts from the following three roles:

- User
- Robot
- System (optional)

- User
- Robot

In this regard, OpenCompass has preset three `api_role` values for API models: `HUMAN`, `BOT`, `SYSTEM`, and stipulates that in addition to regular strings, the input accepted by API models includes a middle format of dialogue represented by `PromptList`. The API model will repackage the dialogue in a multi-turn dialogue format and send it to the backend. However, to activate this feature, users need to map the roles `role` in the dataset prompt template to the corresponding `api_role` in the above meta template. The following figure illustrates the relationship between the input accepted by the API model and the Prompt Template and Meta Template.

## Debugging

If you need to debug the prompt, it is recommended to use the `tools/prompt_viewer.py` script to preview the actual prompt received by the model after preparing the configuration file. Read [here](../tools.md#prompt-viewer) for more.

# RTMPose

|       Model        | Input Size |  AP   |  AR   |           Details and Download            |
| :----------------: | :--------: | :---: | :---: | :---------------------------------------: |
|     RTMPose-t      |  256x192   | 0.682 | 0.736 | [rtmpose_coco.md](./coco/rtmpose_coco.md) |
|     RTMPose-s      |  256x192   | 0.716 | 0.768 | [rtmpose_coco.md](./coco/rtmpose_coco.md) |
|     RTMPose-m      |  256x192   | 0.746 | 0.795 | [rtmpose_coco.md](./coco/rtmpose_coco.md) |
|     RTMPose-l      |  256x192   | 0.758 | 0.806 | [rtmpose_coco.md](./coco/rtmpose_coco.md) |
| RTMPose-t-aic-coco |  256x192   | 0.685 | 0.738 | [rtmpose_coco.md](./coco/rtmpose_coco.md) |

| RTMPose-s-aic-coco |  256x192   | 0.722 | 0.772 | [rtmpose_coco.md](./coco/rtmpose_coco.md) |
| RTMPose-m-aic-coco |  256x192   | 0.758 | 0.806 | [rtmpose_coco.md](./coco/rtmpose_coco.md) |
| RTMPose-l-aic-coco |  256x192   | 0.765 | 0.813 | [rtmpose_coco.md](./coco/rtmpose_coco.md) |
| RTMPose-m-aic-coco |  384x288   | 0.770 | 0.816 | [rtmpose_coco.md](./coco/rtmpose_coco.md) |
| RTMPose-l-aic-coco |  384x288   | 0.773 | 0.819 | [rtmpose_coco.md](./coco/rtmpose_coco.md) |

|   Model   | Input Size | PCKh@0.5 | PCKh@0.1 |           Details and Download            |
| :-------: | :--------: | :------: | :------: | :---------------------------------------: |
| RTMPose-m |  256x256   |  0.907   |  0.348   | [rtmpose_mpii.md](./mpii/rtmpose_mpii.md) |

|   Model   | Input Size |  AP   |  AR   |                   Details and Download                   |
| :-------: | :--------: | :---: | :---: | :------------------------------------------------------: |
| RTMPose-m |  256x192   | 0.706 | 0.788 | [rtmpose_crowdpose.md](./crowdpose/rtmpose_crowdpose.md) |

| RTMPose-s |  256x192   | 0.311 | 0.381 | [rtmpose_humanart.md](./humanart/rtmpose_humanart.md) |

| RTMPose-m |  256x192   | 0.355 | 0.417 | [rtmpose_humanart.md](./humanart/rtmpose_humanart.md) |
| RTMPose-l |  256x192   | 0.378 | 0.442 | [rtmpose_humanart.md](./humanart/rtmpose_humanart.md) |

| RTMPose-s |  256x192   | 0.698 | 0.732 | [rtmpose_humanart.md](./humanart/rtmpose_humanart.md) |
| RTMPose-m |  256x192   | 0.728 | 0.759 | [rtmpose_humanart.md](./humanart/rtmpose_humanart.md) |
| RTMPose-l |  256x192   | 0.753 | 0.783 | [rtmpose_humanart.md](./humanart/rtmpose_humanart.md) |

Following typical conventions, we use `Dataset` and `DataLoader` for data loading
with multiple workers. `Dataset` returns a dict of data items corresponding
the arguments of models' forward method.
Since the data in object detection may not be the same size (point number, gt bbox size, etc.),
we introduce a new `DataContainer` type in MMCV to help collect and distribute
data of different size.
See [here](https://github.com/open-mmlab/mmcv/blob/master/mmcv/parallel/data_container.py) for more details.

The data preparation pipeline and the dataset is decomposed. Usually a dataset
defines how to process the annotations and a data pipeline defines all the steps to prepare a data dict.

We present a classical pipeline in the following figure. The blue blocks are pipeline operations. With the pipeline going on, each operator can add new keys (marked as green) to the result dict or update the existing keys (marked as orange).

Here is an pipeline example for PointPillars.

```python
    dict(
    dict(
    dict(

backend_args=backend_args),
    dict(
]
    dict(
    dict(
    dict(

    dict(
    dict(
            dict(
            dict(
            dict(
        ])
]
```

For each operation, we list the related dict fields that are added/updated/removed.

dict(type='Collect3D', keys=['points'])
        ])
]
```

- add: points

- update: points

- add: gt_bboxes_3d, gt_labels_3d, gt_bboxes, gt_labels, pts_instance_mask, pts_semantic_mask, bbox3d_fields, pts_mask_fields, pts_seg_fields

- add: pcd_trans, pcd_rotation, pcd_scale_factor
- update: points, \*bbox3d_fields

- add: flip, pcd_horizontal_flip, pcd_vertical_flip

- update: gt_bboxes_3d, gt_labels_3d

- update: points, gt_bboxes_3d, gt_labels_3d, gt_bboxes, gt_labels

- add: img_meta (the keys of img_meta is specified by `meta_keys`)
- remove: all other keys except for those specified by `keys`

- update: scale, pcd_scale_factor, flip, flip_direction, pcd_horizontal_flip, pcd_vertical_flip with list of augmented data with these specific parameters

1. Write a new pipeline in any file, e.g., `my_pipeline.py`. It takes a dict as input and return a dict.

   ```python

   ```

2. Import the new class.

   ```python
   ```

results['dummy'] = True
   ```

   ```python
   ```

   ```python
       dict(
       dict(
       dict(

dict(type='PointsRangeFilter', point_cloud_range=point_cloud_range),
   ]
   ```

# Model Complexity Analysis

## Get the FLOPs and params (experimental)

We provide a script adapted from [MMEngine](https://github.com/open-mmlab/mmengine/blob/main/mmengine/analysis/complexity_analysis.py) to compute the FLOPs and params of a given model.

```shell
```

- `--shape`: Input size, support single value or double value parameter, such as `--shape 256` or `--shape 224 256`. If not set, default to be `224 224`.

Example:

```shell
```

You will get the final result like this.

```text
```

Also, you will get the detailed complexity information of each layer like this:

```text

```

```text

```

This tool is still experimental and we do not guarantee that the number is correct. You may well use the result for simple comparisons, but double-check it before you adopt it in technical reports or papers.
- FLOPs are related to the input shape while parameters are not. The default input shape is (1, 3, 224, 224).

- Some operators are not counted into FLOPs like custom operators. Refer to [`mmengine.analysis.complexity_analysis._DEFAULT_SUPPORTED_FLOP_OPS`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/analysis/complexity_analysis.py) for details.
```

# Customize Training Schedule

In our codebase, [default training schedules](https://github.com/open-mmlab/mmpretrain/blob/main/configs/_base_/schedules) have been provided for common datasets such as CIFAR, ImageNet, etc. If we attempt to experiment on these datasets for higher accuracy or on different new methods and datasets, we might possibly need to modify the strategies.

In this tutorial, we will introduce how to modify configs to construct optimizers, use parameter-wise finely configuration, gradient clipping, gradient accumulation as well as customize learning rate and momentum schedules. Furthermore, introduce a template to customize self-implemented optimizationmethods for the project.

## Customize optimization

We use the `optim_wrapper` field to configure the strategies of optimization, which includes choices of optimizer, choices of automatic mixed precision training, parameter-wise configurations, gradient clipping and accumulation. Details are seen below.

We support all the optimizers implemented by PyTorch, and to use them, please change the `optimizer` field of config files.

For example, if you want to use [`SGD`](torch.optim.SGD), the modification in config file could be as the following. Notice that optimization related settings should all wrapped inside the `optim_wrapper`.

```python
)
```

```{note}
`type` in optimizer is not a constructor but a optimizer name in PyTorch.
Refers to {external+torch:ref}`List of optimizers supported by PyTorch <optim:algorithms>` for more choices.
```

To modify the learning rate of the model, just modify the `lr` in the config of optimizer.
You can also directly set other arguments according to the [API doc](torch.optim) of PyTorch.

```

For example, if you want to use [`Adam`](torch.optim.Adam) with settings like `torch.optim.Adam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False)` in PyTorch. You could use the config below:

```python
        lr=0.001,
)
```

The default type of `optim_wrapper` field is [`OptimWrapper`](mmengine.optim.OptimWrapper), therefore, you can
omit the type field usually, like:

```python
        lr=0.001,
```
````

```python
        lr=0.001,
```
````

### Use AMP training

If we want to use the automatic mixed precision training, we can simply change the type of `optim_wrapper` to `AmpOptimWrapper` in config files.

```python
```

Alternatively, for conveniency, we can set `--amp` parameter to turn on the AMP option directly in the `tools/train.py` script. Refers to [Training tutorial](../user_guides/train.md) for details of starting a training.

  ```python
  ```

  - `norm_decay_mult`: Multiplier for weight decay of weight and bias of normalization layers. Defaults to 1.

- `norm_decay_mult`: Multiplier for weight decay of weight and bias of normalization layers. Defaults to 1.
  - `flat_decay_mult`: Multiplier for weight decay of all one-dimensional parameters. Defaults to 1.

  MMPretrain can use `custom_keys` in `paramwise_cfg` to specify different parameters to use different learning rates or weight decay.

  For example, to set all learning rates and weight decays of `backbone.layer0` to 0, the rest of `backbone` remains the same as optimizer and the learning rate of `head` to 0.001, use the configs below.

  ```python

```python
          }))
  ```

```python
```

```

```python
```

```python
```

```python
```

In training, the optimzation parameters such as learing rate, momentum, are usually not fixed but changing through iterations or epochs. PyTorch supports several learning rate schedulers, which are not sufficient for complex strategies. In MMPretrain, we provide `param_scheduler` for better controls of different parameter schedules.

All available learning rate scheduler can be found {external+mmengine:doc}`here <api/optim>`, and the

  ```python
  ```

  ```python
  ```

```python
  ```

  In MMPretrain, simply combines desired schedules in `param_scheduler` as a list can achieve the warmup strategy.

  ```python

),
    ]
  ```

  Notice that, we use `begin` and `end` arguments here to assign the valid range, which is [`begin`, `end`) for this schedule. And the range unit is defined by `by_epoch` argument. If not specified, the `begin` is 0 and the `end` is the max epochs or iterations.

  ```{tip}
  To check that the learning rate curve is as expected, after completing your configuration file，you could use [optimizer parameter visualization tool](../useful_tools/scheduler_visualization.md) to draw the corresponding learning rate adjustment curve.
  ```

```

```python
         begin=0,
         end=1000)
]
```

```{note}
This part will modify the MMPretrain source code or add code to the MMPretrain framework, beginners can skip it.
```

]
```

```{note}
```

In academic research and industrial practice, it may be necessary to use optimization methods not implemented by MMPretrain, and you can add them through the following methods.

1. Implement a New Optimizer

   Assume you want to add an optimizer named `MyOptimizer`, which has arguments `a`, `b`, and `c`.
   You need to create a new file under `mmpretrain/engine/optimizers`, and implement the new optimizer in the file, for example, in `mmpretrain/engine/optimizers/my_optimizer.py`:

   ```python

           ...

           ...
   ```

2. Import the Optimizer

           ...

           ...
   ```

   To find the above module defined above, this module should be imported during the running.

   Import it in the `mmpretrain/engine/optimizers/__init__.py` to add it into the `mmpretrain.engine` package.

   ```python
   # In mmpretrain/engine/optimizers/__init__.py
   ...
   from .my_optimizer import MyOptimizer # MyOptimizer maybe other class name

   ```

   During running, we will automatically import the `mmpretrain.engine` package and register the `MyOptimizer` at the same time.

3. Specify the Optimizer in Config

   Then you can use `MyOptimizer` in the `optim_wrapper.optimizer` field of config files.

   ```python
   ```

```python
   ```

```python

```python
# In mmpretrain/engine/optimizers/my_optim_constructor.py

        ...

        ...
```

Here is a specific example of [OptimWrapperConstructor](mmpretrain.engine.optimizers.LearningRateDecayOptimWrapperConstructor).

1. Import it in the `mmpretrain/engine/optimizers/__init__.py` to add it into the `mmpretrain.engine` package.

   ```python
   ...

   ```

```python
   ...

   ```

   ```python
   )
   ```

本文将主要介绍如何使用自定义数据集来进行模型的训练和测试，以 Waymo 数据集作为示例来说明整个流程。

基本步骤如下所示：

1. 准备自定义数据集；
2. 准备配置文件；
3. 在自定义数据集上进行模型的训练、测试和推理。

在 MMDetection3D 中有三种方式来自定义一个新的数据集：

1. 将新数据集的数据格式重新组织成已支持的数据集格式；
2. 将新数据集的数据格式重新组织成已支持的一种中间格式；
3. 从头开始创建一个新的数据集。

由于前两种方式比第三种方式更加容易，我们更加建议采用前两种方式来自定义数据集。

在本文中，我们给出示例将数据转换成 KITTI 数据集的数据格式，你可以参考此处将你的数据集重新组织成 KITTI 格式。关于标准格式的数据集，你可以参考[自定义数据集文档](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/docs/zh_cn/advanced_guides/customize_dataset.md)。

**注意**：考虑到 Waymo 数据集的格式与现有的其他数据集的格式的差别较大，因此本文以该数据集为例来讲解如何自定义数据集，从而方便理解数据集自定义的过程。若需要创建的新数据集与现有的数据集的组织格式较为相似，如 Lyft 数据集和 nuScenes 数据集，采用对数据集的中间格式进行转换的方式（第二种方式）相比于采用对数据格式进行转换的方式（第一种方式）会更加简单易行。

### KITTI 数据集格式

应用于 3D 目标检测的 KITTI 原始数据集的组织方式通常如下所示，其中 `ImageSets` 包含数据集划分文件，用以划分训练集/验证集/测试集，`calib` 包含对于每个数据样本的标定信息，`image_2` 和 `velodyne` 分别包含图像数据和点云数据，`label_2` 包含与 3D 目标检测相关的标注文件。

```
├── tools
├── data
│   ├── kitti

```
├── tools
├── data
│   ├── kitti
```

KITTI 官方提供的目标检测开发[工具包](https://s3.eu-central-1.amazonaws.com/avg-kitti/devkit_object.zip)详细描述了 KITTI 数据集的标注格式，例如，KITTI 标注格式包含了以下的标注信息：

```
#  值    名称      描述
   1    类型      描述检测目标的类型：'Car'，'Van'，'Truck'，
                  'Pedestrian'，'Person_sitting'，'Cyclist'，'Tram'，
                  'Misc' 或 'DontCare'
   1    截断程度　 从 0（非截断）到 1（截断）的浮点数，其中截断指的是离开检测图像边界的检测目标
   1    遮挡程度　 用来表示遮挡状态的四种整数（0，1，2，3）:
                  0 = 可见，1 = 部分遮挡
                  2 = 大面积遮挡，3 = 未知
   1    观测角    观测目标的角度，取值范围为 [-pi..pi]
   4    标注框    检测目标在图像中的二维标注框（以0为初始下标）：包括每个检测目标的左上角和右下角的坐标
   3    维度　    检测目标的三维维度：高度、宽度、长度（以米为单位）

2 = 大面积遮挡，3 = 未知
   3    位置　    相机坐标系下的三维位置 x，y，z（以米为单位）
   1    y 旋转　  相机坐标系下检测目标绕着Y轴的旋转角，取值范围为 [-pi..pi]
   1    得分　    仅在计算结果时使用，检测中表示置信度的浮点数，用于生成 p/r 曲线，在p/r 图中，越高的曲线表示结果越好。
```

假定我们使用 Waymo 数据集。

在下载好数据集后，我们需要实现一个函数用来将输入数据和标注文件转换成 KITTI 风格。然后我们可以通过继承 `KittiDataset` 实现 `WaymoDataset`，用来加载数据以及训练模型，通过继承 `KittiMetric` 实现 `WaymoMetric` 来做模型的评估。

```

具体来说，首先使用[数据转换器](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/waymo_converter.py)将 Waymo 数据集转换成 KITTI 数据集的格式，并定义 [Waymo 类](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/datasets/waymo_dataset.py)对转换的数据进行处理。此外需要添加 waymo [评估类](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/evaluation/metrics/waymo_metric.py)来评估结果。因为我们将 Waymo 原始数据集进行预处理并重新组织成 KITTI 数据集的格式，因此可以比较容易通过继承 KittiDataset 类来实现 WaymoDataset 类。需要注意的是，由于 Waymo 数据集有相应的官方评估方法，我们需要进一步实现新的 Waymo 评估方法，更多关于评估方法参考[评估文档](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/metric_and_evaluator.md)。最后，用户可以成功地转换数据并使用 `WaymoDataset` 训练以及 `WaymoMetric` 评估模型。

更多关于 Waymo 数据集预处理的中间结果的细节，请参照对应的[说明文档](https://mmdetection3d.readthedocs.io/zh_CN/latest/datasets/waymo_det.html)。

## 准备配置文件

## 准备配置文件

第二步是准备配置文件来帮助数据集的读取和使用，另外，为了在 3D 检测中获得不错的性能，调整超参数通常是必要的。

## 准备配置文件

假设我们想要使用 PointPillars 模型在 Waymo 数据集上实现三类的 3D 目标检测：vehicle、cyclist、pedestrian，参照 KITTI 数据集[配置文件](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/datasets/kitti-3d-3class.py)、模型[配置文件](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/models/pointpillars_hv_secfpn_kitti.py)和[整体配置文件](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py)，我们需要准备[数据集配置文件](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/datasets/waymoD5-3d-3class.py)、[模型配置文件](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/models/pointpillars_hv_secfpn_waymo.py)，并将这两种文件进行结合得到[整体配置文件](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/pointpillars/pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymoD5-3d-3class.py)。

为了使用一个新的配置文件来训练模型，可以通过下面的命令来实现：

```shell

```shell
```

更多的使用细节，请参考[案例 1](https://mmdetection3d.readthedocs.io/zh_CN/latest/1_exist_data_model.html)。

## 测试和推理

为了测试已经训练好的模型的性能，可以通过下面的命令来实现：

```shell
```

**注意**：为了使用 Waymo 数据集的评估方法，需要参考[说明文档](https://mmdetection3d.readthedocs.io/zh_CN/latest/datasets/waymo_det.html)并按照官方指导来准备与评估相关联的文件。

更多有关测试和推理的使用细节，请参考[案例 1](https://mmdetection3d.readthedocs.io/zh_CN/latest/1_exist_data_model.html) 。

# StrongSORT: Make DeepSORT Great Again

Existing Multi-Object Tracking (MOT) methods can be roughly classified as tracking-by-detection and joint-detection-association paradigms. Although the latter has elicited more attention and demonstrates comparable performance relative to the former, we claim that the tracking-by-detection paradigm is still the optimal solution in terms of tracking accuracy. In this paper, we revisit the classic tracker DeepSORT and upgrade it from various aspects, i.e., detection, embedding and association. The resulting tracker, called StrongSORT, sets new HOTA and IDF1 records on MOT17 and MOT20. We also present two lightweight and plug-and-play algorithms to further refine the tracking results. Firstly, an appearance-free link model (AFLink) is proposed to associate short tracklets into complete trajectories. To the best of our knowledge, this is the first global link model without appearance information. Secondly, we propose Gaussian-smoothed interpolation (GSI) to compensate for missing detections. Instead of ignoring motion information like linear interpolation, GSI is based on the Gaussian process regression algorithm and can achieve more accurate localizations. Moreover, AFLink and GSI can be plugged into various trackers with a negligible extra computational cost (591.9 and 140.9 Hz, respectively, on MOT17). By integrating StrongSORT with the two algorithms, the final tracker StrongSORT++ ranks first on MOT17 and MOT20 in terms of HOTA and IDF1 metrics and surpasses the second-place one by 1.3 - 2.2. Code will be released soon.

  <img src="https://user-images.githubusercontent.com/99722489/185282811-ec82bdf6-8889-4f01-9c4d-a8e104f775b7.png"/>
</div>

```latex
@article{du2022strongsort,
  title={Strongsort: Make deepsort great again},
  author={Du, Yunhao and Song, Yang and Yang, Bo and Zhao, Yanyun},
  journal={arXiv preprint arXiv:2202.13514},
}
```

author={Du, Yunhao and Song, Yang and Yang, Bo and Zhao, Yanyun},
}
```

|    Method    | Detector | ReID |           Train Set           |    Test Set    | Public | Inf time (fps) | HOTA | MOTA | IDF1 |  FP   |  FN   | IDSw. |                                        Config                                        |                                                                                                                                                                                   Download                                                                                                                                                                                    |

| StrongSORT++ | YOLOX-X  | R50  | CrowdHuman + MOT17-half-train | MOT17-half-val |   N    |       -        | 70.9 | 78.4 | 83.3 | 15237 | 19035 |  582  | [config](strongsort_yolox_x_8xb4-80e_crowdhuman-mot17halftrain_test-mot17halfval.py) | [detector](https://download.openmmlab.com/mmtracking/mot/strongsort/mot_dataset/yolox_x_crowdhuman_mot17-private-half_20220812_192036-b6c9ce9a.pth) [reid](https://download.openmmlab.com/mmtracking/mot/reid/reid_r50_6e_mot17-4bf6b63d.pth) [AFLink](https://download.openmmlab.com/mmtracking/mot/strongsort/mot_dataset/aflink_motchallenge_20220812_190310-a7578ad3.pth) |

|    Method    | Detector | ReID |        Train Set         |  Test Set  | Public | Inf time (fps) | HOTA | MOTA | IDF1 |  FP   |  FN   | IDSw. |                                    Config                                     |                                                                                                                                                                                         Download                                                                                                                                                                                         |

| StrongSORT++ | YOLOX-X  | R50  | CrowdHuman + MOT20-train | MOT20-test |   N    |       -        | 62.9 | 75.5 | 77.3 | 29043 | 96155 | 1640  | [config](strongsort_yolox_x_8xb4-80e_crowdhuman-mot20train_test-mot20test.py) | [detector](https://download.openmmlab.com/mmtracking/mot/strongsort/mot_dataset/yolox_x_crowdhuman_mot20-private_20220812_192123-77c014de.pth) [reid](https://download.openmmlab.com/mmtracking/mot/reid/reid_r50_6e_mot20_20210803_212426-c83b1c01.pth) [AFLink](https://download.openmmlab.com/mmtracking/mot/strongsort/mot_dataset/aflink_motchallenge_20220812_190310-a7578ad3.pth) |

We implement StrongSORT with independent detector and ReID models.

# Training YOLOX-X on crowdhuman and mot17-half-train dataset with following command.
bash tools/dist_train.sh configs/det/yolox_x_8xb4-80e_crowdhuman-mot17halftrain_test-mot17halfval.py 8
```

And you can train the ReID model as follows.

# Training ReID model on mot17-train80 dataset with following command.

```

**2.1 Example on MOTxx-halfval dataset**

bash tools/dist_test_tracking.sh configs/strongsort/strongsort_yolox_x_8xb4-80e_crowdhuman-mot17halftrain_test-mot17halfval.py 8 --detector ${CHECKPOINT_PATH} --reid ${CHECKPOINT_PATH}
```

**2.2 Example on MOTxx-test dataset**

```

It will be stored in `./mot_20_test_res`, you can modify the saved path in `test_evaluator` of the config.

bash tools/dist_test_tracking.sh configs/strongsort/strongsort_yolox_x_8xb4-80e_crowdhuman-mot20train_test-mot20test.py 8 --detector ${CHECKPOINT_PATH} --reid ${CHECKPOINT_PATH}
```

### 3.Inference

```shell

```shell
python demo/mot_demo.py demo/demo_mot.mp4 configs/strongsort/strongsort_yolox_x_8xb4-80e_crowdhuman-mot17halftrain_test-mot17halfval.py --detector ${CHECKPOINT_FILE} --reid ${CHECKPOINT_PATH} --out mot.mp4
```

# 模型部署 (待更新)

MMRotate 1.x 完全基于 [MMDeploy](https://mmdeploy.readthedocs.io/) 來部署模型。 我们将在下一个版本完善这个文档。

# Acknowledgements

- Thanks to bug1989 [https://github.com/bug1989] for contributing the initial quantized int8 inference code and a large variety of device benchmark
- Thanks to zhiliu6 [https://github.com/zhiliu6] for contributing the darknet conversion tool, operators and YOLO examples
- Thanks to Tijmen Verhulsdonck [https://github.com/Timen] for contributing the massive AVX optimization for x86 platform

# Prompt Overview

The prompt is the input to the Language Model (LLM), used to guide the model to generate text or calculate perplexity (PPL). The selection of prompts can significantly impact the accuracy of the evaluated model. The process of converting the dataset into a series of prompts is defined by templates.

In OpenCompass, we split the template into two parts: the data-side template and the model-side template. When evaluating a model, the data will pass through both the data-side template and the model-side template, ultimately transforming into the input required by the model.

The data-side template is referred to as [prompt_template](./prompt_template.md), which represents the process of converting the fields in the dataset into prompts.

The model-side template is referred to as [meta_template](./meta_template.md), which represents how the model transforms these prompts into its expected input.

We also offer some prompting examples regarding [Chain of Thought](./chain_of_thought.md).

# OVERVIEW (To be updated)

This chapter introduces the basic conception of rotated object detection and the framework of MMRotate, and provides
links to detailed tutorials about MMRotate.

## What is rotated object detection

### Problem definition

Benefiting from the vigorous development of general object detection, most current
rotated object detection models are based on classic general object detector.
With the development of detection tasks, horizontal boxes have been unable to
meet the needs of researchers in some subdivisions. We call it rotating object
detection by redefining the object representation and increasing the number of
regression degrees of freedom to achieve rotated rectangle, quadrilateral, and
even arbitrary shape detection. Performing high-precision rotated object detection
more efficiently has become a current research hotspot. The following areas are
where rotated object detection has been applied or has great potential: face

recognition, scene text, remote sensing, self-driving, medical,
robotic grasping, etc.

### What is rotated box

The most notable difference between rotated object detection and generic detection is
the replacement of horizontal box annotations with rotated box annotations.
They are defined as follows:

- Horizontal box: A rectangle with the `width` along the `x-axis` and `height` along
  the `y-axis`. Usually, it can be represented by the coordinates of 2 diagonal
  vertices `(x_i, y_i)`  (i = 1, 2), or it can be represented by the coordinates
  of the center point and the `width` and `height`, `(x_center, y_center, width, height)`.
- Rotated box: It is obtained by rotating the horizontal box around the center
  point by an `angle`, and the definition method of its rotated box is

  obtained by adding a radian parameter `(x_center, y_center, width, height, theta)`,
  where `theta = angle * pi / 180`. The unit of `theta` is `rad`. When the rotation
  angle is a multiple of 90°, the rotated box degenerates into a horizontal box.
  The rotated box annotations exported by the annotation software are usually polygons,
  which need to be converted to the rotated box definition method before training.

```{note}
In MMRotate, angle parameters are in radians.
```

### Rotation direction

A rotated box can be obtained by rotating a horizontal box clockwise or
counterclockwise around its center point. The rotation direction is closely
related to the choice of the coordinate system. The image space adopts the
right-handed coordinate system `(y, x)`, where y is `up->down` and x is `left->right`.

There are two opposite directions of rotation:

- Clockwise（CW）

Schematic of `CW`

```
|  |             |
|  |     box     h
v

```

Rotation matrix of `CW`

```{math}
```

Rotation transformation of `CW`

```{math}
P_A=
=
=
\\
```

- Counterclockwise（CCW）

Schematic of `CCW`

```

\\
```

```
|  |             |
|  |     box     h
v

```

Rotation matrix of `CCW`

```{math}
```

Rotation transformation of `CCW`

```{math}
P_A=
=
=
\\
```

The operators that can set the rotation direction in MMCV are:

- box_iou_rotated (Defaults to `CW`)
- nms_rotated (Defaults to `CW`)

\\
```

- RoIAlignRotated (Defaults to `CCW`)
- RiRoIAlignRotated (Defaults to `CCW`).

```{note}
In MMRotate, the rotation direction of the rotated boxes is `CW`.
```

### Definition of rotated box

Due to the difference in the definition range of `theta`, the following three
definitions of the rotated box gradually emerge in rotated object detection:

- {math}`D_{oc^{\prime}}`: OpenCV Definition, `angle∈(0, 90°]`, `theta∈(0, pi / 2]`,
  The angle between the `width` of the rectangle and the positive semi-axis of x is
  a positive acute angle. This definition comes from the `cv2.minAreaRect` function
  in OpenCV, which returns an angle in the range `(0, 90°]`.
- {math}`D_{le135}`: Long Edge Definition (135°)，`angle∈[-45°, 135°)`,
  `theta∈[-pi / 4, 3 * pi / 4)` and `width > height`.

in OpenCV, which returns an angle in the range `(0, 90°]`.
- {math}`D_{le90}`: Long Edge Definition (90°)，`angle∈[-90°, 90°)`,
  `theta∈[-pi / 2, pi / 2)` and `width > height`.

</div>

The conversion relationship between the three definitions is not involved in
MMRotate, so we will not introduce it much more. Refer to the below
[blog](https://zhuanlan.zhihu.com/p/459018810) to dive deeper.

```{note}
MMRotate supports the above three definitions of rotated box simultaneously,
which can be flexibly switched through the configuration file.
```

It should be noted that if the OpenCV version is less than 4.5.1, the angle range
of `cv2.minAreaRect` is between `[-90°, 0°)`. [Reference](https://github.com/opencv/opencv/issues/19749)

```

In order to facilitate the distinction, the old version of the OpenCV definition
is denoted as {math}`D_{oc}`.

- {math}`D_{oc^{\prime}}` : OpenCV definition, `opencv>=4.5.1`, `angle∈(0, 90°]`, `theta∈(0, pi / 2]`.
- {math}`D_{oc}` : Old OpenCV definition, `opencv<4.5.1`, `angle∈[-90°, 0°)`, `theta∈[-pi / 2, 0)`.

</div>

The conversion relationship between the two OpenCV definitions is as follows:

```{math}
\\

D_{oc}\left( h_{oc},w_{oc},\theta _{oc}+\pi /2 \right) , otherwise\\
\\
```

```{note}
Regardless of the OpenCV version you are using, MMRotate will convert the theta
of the OpenCV definition to (0, pi / 2].
```

The code for evaluating mAP involves the calculation of IoU. We can directly
calculate the IoU of the rotated boxes or convert the rotated boxes to a polygons
and then calculate the polygons IoU (DOTA online evaluation uses the calculation
of polygons IoU).

## What is MMRotate

MMRotate is a toolbox that provides a framework for unified implementation and evaluation of rotated object detection,
and below is its whole framework:

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/mmrotate-arch.png" width=80%/>
</div>

MMRotate consists of 4 main parts, `datasets`, `models`, `core` and `apis`.

- `datasets` is for data loading and data augmentation. In this part,
  we support various datasets for rotated object detection algorithms,
  useful data augmentation transforms in `pipelines` for pre-processing image.

- `models` contains models and loss functions.

- `core` provides evaluation tools for model training and evaluation.

- `apis` provides high-level APIs for models training, testing, and inference.

The module design of MMRotate is as follows:

</div>

The following points need to be noted due to different definitions of rotated box:

- Loading annotations
- Data augmentation
- Assigning samples
- Evaluation

Here is a detailed step-by-step guide to learn more about MMRotate:

1. For installation instructions, please see [install](install.md).

2. [get_started](get_started.md) is for the basic usage of MMRotate.

- [Config](tutorials/customize_config.md)
- [Customize Dataset](tutorials/customize_dataset.md)
- [Customize Model](tutorials/customize_models.md)
- [Customize Runtime](tutorials/customize_runtime.md)

# CLIP

> [Learning Transferable Visual Models From Natural Language Supervision](https://arxiv.org/abs/2103.00020)

State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained model weights at this https URL.

<img src="https://raw.githubusercontent.com/Scarecrow0/figures_cache/main/clip_main_fig.png" width="100%"/>
</div>

```python

predict = inference_model('vit-base-p32_clip-laion2b-in12k-pre_3rdparty_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('vit-base-p32_clip-laion2b-in12k-pre_3rdparty_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/clip/vit-base-p32_pt-64xb64_in1k.py https://download.openmmlab.com/mmclassification/v0/clip/clip-vit-base-p32_laion2b-in12k-pre_3rdparty_in1k_20221220-b384e830.pth
```

| Model                                        |         Pretrain          | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                     Config                     |                     Download                     |
| :------------------------------------------- | :-----------------------: | :--------: | :-------: | :-------: | :-------: | :--------------------------------------------: | :----------------------------------------------: |

| `vit-base-p32_clip-laion2b-in12k-pre_3rdparty_in1k`\* | CLIP LAION2B ImageNet-12k |   88.22    |   4.36    |   83.06   |   96.49   |    [config](vit-base-p32_pt-64xb64_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/clip/clip-vit-base-p32_laion2b-in12k-pre_3rdparty_in1k_20221220-b384e830.pth) |
| `vit-base-p32_clip-laion2b-pre_3rdparty_in1k`\* |       CLIP LAION2B        |   88.22    |   4.36    |   82.46   |   96.12   |    [config](vit-base-p32_pt-64xb64_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/clip/clip-vit-base-p32_laion2b-pre_3rdparty_in1k_20221220-194df57f.pth) |
| `vit-base-p32_clip-openai-pre_3rdparty_in1k`\* |        CLIP OPENAI        |   88.22    |   4.36    |   81.77   |   95.89   |    [config](vit-base-p32_pt-64xb64_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/clip/clip-vit-base-p32_openai-pre_3rdparty_in1k_20221220-a0182ba9.pth) |

| `vit-base-p32_clip-laion2b-in12k-pre_3rdparty_in1k-384px`\* | CLIP LAION2B ImageNet-12k |   88.22    |   12.66   |   85.39   |   97.67   | [config](vit-base-p32_pt-64xb64_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/clip/clip-vit-base-p32_laion2b-in12k-pre_3rdparty_in1k-384px_20221220-c7757552.pth) |
| `vit-base-p32_clip-openai-in12k-pre_3rdparty_in1k-384px`\* | CLIP OPENAI ImageNet-12k  |   88.22    |   12.66   |   85.13   |   97.42   | [config](vit-base-p32_pt-64xb64_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/clip/clip-vit-base-p32_openai-in12k-pre_3rdparty_in1k-384px_20221220-dc2e49ea.pth) |
| `vit-base-p16_clip-laion2b-in12k-pre_3rdparty_in1k`\* | CLIP LAION2B ImageNet-12k |   86.57    |   16.86   |   86.02   |   97.76   |    [config](vit-base-p16_pt-64xb64_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/clip/clip-vit-base-p16_laion2b-in12k-pre_3rdparty_in1k_20221220-a5e31f8c.pth) |

| `vit-base-p16_clip-laion2b-pre_3rdparty_in1k`\* |       CLIP LAION2B        |   86.57    |   16.86   |   85.49   |   97.59   |    [config](vit-base-p16_pt-64xb64_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/clip/clip-vit-base-p16_laion2b-pre_3rdparty_in1k_20221220-5e24ff58.pth) |
| `vit-base-p16_clip-openai-in12k-pre_3rdparty_in1k`\* | CLIP OPENAI ImageNet-12k  |   86.57    |   16.86   |   85.99   |   97.72   |    [config](vit-base-p16_pt-64xb64_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/clip/clip-vit-base-p16_openai-in12k-pre_3rdparty_in1k_20221220-90d930a8.pth) |
| `vit-base-p16_clip-openai-pre_3rdparty_in1k`\* |        CLIP OPENAI        |   86.57    |   16.86   |   85.30   |   97.50   |    [config](vit-base-p16_pt-64xb64_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/clip/clip-vit-base-p16_openai-pre_3rdparty_in1k_20221220-c7d9c899.pth) |

| `vit-base-p32_clip-laion2b-in12k-pre_3rdparty_in1k-448px`\* | CLIP LAION2B ImageNet-12k |   88.22    |   17.20   |   85.76   |   97.63   | [config](vit-base-p32_pt-64xb64_in1k-448px.py) | [model](https://download.openmmlab.com/mmclassification/v0/clip/clip-vit-base-p32_laion2b-in12k-pre_3rdparty_in1k-448px_20221220-ca404a7d.pth) |
| `vit-base-p16_clip-laion2b-in12k-pre_3rdparty_in1k-384px`\* | CLIP LAION2B ImageNet-12k |   86.57    |   49.37   |   87.17   |   98.02   | [config](vit-base-p16_pt-64xb64_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/clip/clip-vit-base-p16_laion2b-in12k-pre_3rdparty_in1k-384px_20221220-84ed0cc0.pth) |
| `vit-base-p16_clip-laion2b-pre_3rdparty_in1k-384px`\* |       CLIP LAION2B        |   86.57    |   49.37   |   86.52   |   97.97   | [config](vit-base-p16_pt-64xb64_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/clip/clip-vit-base-p16_laion2b-pre_3rdparty_in1k-384px_20221220-558ed826.pth) |

| `vit-base-p16_clip-openai-in12k-pre_3rdparty_in1k-384px`\* | CLIP OPENAI ImageNet-12k  |   86.57    |   49.37   |   86.87   |   98.05   | [config](vit-base-p16_pt-64xb64_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/clip/clip-vit-base-p16_openai-in12k-pre_3rdparty_in1k-384px_20221220-8df86b74.pth) |
| `vit-base-p16_clip-openai-pre_3rdparty_in1k-384px`\* |        CLIP OPENAI        |   86.57    |   49.37   |   86.25   |   97.90   | [config](vit-base-p16_pt-64xb64_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/clip/clip-vit-base-p16_openai-pre_3rdparty_in1k-384px_20221220-eb012e87.pth) |

*Models with * are converted from the [timm](https://github.com/rwightman/pytorch-image-models). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@InProceedings{pmlr-v139-radford21a,
title = {Learning Transferable Visual Models From Natural Language Supervision},

```bibtex
author = {Radford, Alec and Kim, Jong Wook and Hallacy, Chris and Ramesh, Aditya and Goh, Gabriel and Agarwal, Sandhini and Sastry, Girish and Askell, Amanda and Mishkin, Pamela and Clark, Jack and Krueger, Gretchen and Sutskever, Ilya},
booktitle = {Proceedings of the 38th International Conference on Machine Learning},
series = {Proceedings of Machine Learning Research},
publisher = {PMLR},
}
```

# 参与贡献 MMACTION2

欢迎各种形式的贡献，包括但不限于以下内容。

- 修改拼写错误或代码错误
- 新功能和组件
- 添加文档或将文档翻译成其他语言
- 添加关于视频理解算法的新项目（推荐），具体细节请参考[这里](../projectzoo.md)

## 工作流程

1. Fork 并拉取最新的 mmaction2
2. 创建一个有意义的新分支（不要使用主分支进行 PR）
3. 提交你的更改
4. 创建一个 PR

```{note}
- 如果你计划添加一些涉及大规模更改的新功能，请首先打开一个 issue 进行讨论。
- 如果你是论文的作者，并希望将你的方法包含在 mmaction2 中，请与我们联系。我们将非常感谢您的贡献。
```

## 代码风格

### Python

我们采用 [PEP8](https://www.python.org/dev/peps/pep-0008/) 作为首选代码风格。

我们使用以下工具进行代码检查和格式化：

- [flake8](http://flake8.pycqa.org/en/latest/)：检查器
- [yapf](https://github.com/google/yapf)：格式化器
- [isort](https://github.com/timothycrosley/isort)：排序导入
- [codespell](https://github.com/codespell-project/codespell)：一个用于修复文本文件中常见拼写错误的 Python 工具。
- [mdformat](https://github.com/executablebooks/mdformat)：Mdformat 是一个自由裁量的 Markdown 格式化工具，可用于强制执行一致的 Markdown 文件样式。
- [docformatter](https://github.com/myint/docformatter)：一个格式化工具，用于格式化文档字符串。

yapf 和 isort 的样式配置可以在 [setup.cfg](https://github.com/open-mmlab/mmaction2/blob/main/setup.cfg) 中找到。

我们使用 [pre-commit hook](https://pre-commit.com/) 来保证每次提交时自动进行代码检查和格式化，启用的功能包括 `flake8`, `yapf`, `isort`, `trailing whitespaces`, `markdown files`, 修复 `end-of-files`, `double-quoted-strings`, `python-encoding-pragma`, `mixed-line-ending`, 对 `requirments.txt`的排序等。
预提交钩子的配置存储在 [.pre-commit-config](https://github.com/open-mmlab/mmaction2/blob/main/.pre-commit-config.yaml) 中。

在克隆仓库后，你需要安装初始化的预提交钩子。

```shell
```

从仓库文件夹中

```shell
```

在此之后，每次提交，代码规范检查和格式化工具都将被强制执行。

```{note}
在创建 PR 之前，请确保你的代码通过了 lint 检查并由 yapf 进行了格式化。
```

### C++ 和 CUDA

我们遵循 [Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html)。

## BDD100K

  ```bash
  ```

```none
├── mmseg
├── tools
├── data
│   │   │   └── 10k

```

# 准备 RealBlur 数据集

```bibtex
}
```

测试数据集 RealBlurR 可以从 [此处](https://drive.google.com/file/d/1glgeWXCy7Y0qWDc0MXBTUlZYJf8984hS/) 下载。
测试数据集 RealBlurJ 可以从 [此处](https://drive.google.com/file/d/1Rb1DhhXmX7IXfilQ-zL9aGjQfAAvQTrW/) 下载。

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── RealBlur_R
|   ├── RealBlur_J
```

# EfficientNetV2

> [EfficientNetV2: Smaller Models and Faster Training](https://arxiv.org/abs/2104.00298)

This paper introduces EfficientNetV2, a new family of convolutional networks that have faster training speed and better parameter efficiency than previous models. To develop this family of models, we use a combination of training-aware neural architecture search and scaling, to jointly optimize training speed and parameter efficiency. The models were searched from the search space enriched with new ops such as Fused-MBConv. Our experiments show that EfficientNetV2 models train much faster than state-of-the-art models while being up to 6.8x smaller.   Our training can be further sped up by progressively increasing the image size during training, but it often causes a drop in accuracy. To compensate for this accuracy drop, we propose to adaptively adjust regularization (e.g., dropout and data augmentation) as well, such that we can achieve both fast training and good accuracy.   With progressive learning, our EfficientNetV2 significantly outperforms previous models on ImageNet and CIFAR/Cars/Flowers datasets. By pretraining on the same ImageNet21k, our EfficientNetV2 achieves 87.3% top-1 accuracy on ImageNet ILSVRC2012, outperforming the recent ViT by 2.0% accuracy while training 5x-11x faster using the same computing resources. Code will be available at https://github.com/google/automl/tree/master/efficientnetv2.

<img src="https://user-images.githubusercontent.com/18586273/208616931-0c5107f1-f08c-48d3-8694-7a6eaf227dc2.png" width="50%"/>
</div>

```python

predict = inference_model('efficientnetv2-b0_3rdparty_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('efficientnetv2-b0_3rdparty_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/efficientnet_v2/efficientnetv2-b0_8xb32_in1k.py https://download.openmmlab.com/mmclassification/v0/efficientnetv2/efficientnetv2-b0_3rdparty_in1k_20221221-9ef6e736.pth
```

| Model                                | Params (M) | Flops (G) |                   Config                   |                                                Download                                                 |
| :----------------------------------- | :--------: | :-------: | :----------------------------------------: | :-----------------------------------------------------------------------------------------------------: |

| `efficientnetv2-s_3rdparty_in21k`\*  |   48.16    |   3.31    | [config](efficientnetv2-s_8xb32_in21k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/efficientnetv2/efficientnetv2-s_3rdparty_in21k_20221220-c0572b56.pth) |
| `efficientnetv2-m_3rdparty_in21k`\*  |   80.84    |   5.86    | [config](efficientnetv2-m_8xb32_in21k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/efficientnetv2/efficientnetv2-m_3rdparty_in21k_20221220-073e944c.pth) |
| `efficientnetv2-l_3rdparty_in21k`\*  |   145.22   |   13.11   | [config](efficientnetv2-l_8xb32_in21k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/efficientnetv2/efficientnetv2-l_3rdparty_in21k_20221220-f28f91e1.pth) |
| `efficientnetv2-xl_3rdparty_in21k`\* |   234.82   |   18.86   | [config](efficientnetv2-xl_8xb32_in21k.py) | [model](https://download.openmmlab.com/mmclassification/v0/efficientnetv2/efficientnetv2-xl_3rdparty_in21k_20221220-b2c9329c.pth) |

*Models with * are converted from the [timm](https://github.com/rwightman/pytorch-image-models/blob/main/timm/models/efficientnet.py). The config files of these models are only for inference. We haven't reproduce the training results.*

| Model                                         |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                     Config                      |                          Download                           |
| :-------------------------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :---------------------------------------------: | :---------------------------------------------------------: |

| `efficientnetv2-b0_3rdparty_in1k`\*           | From scratch |    7.14    |   0.92    |   78.52   |   94.44   |    [config](efficientnetv2-b0_8xb32_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/efficientnetv2/efficientnetv2-b0_3rdparty_in1k_20221221-9ef6e736.pth) |
| `efficientnetv2-b1_3rdparty_in1k`\*           | From scratch |    8.14    |   1.44    |   79.80   |   94.89   |    [config](efficientnetv2-b1_8xb32_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/efficientnetv2/efficientnetv2-b1_3rdparty_in1k_20221221-6955d9ce.pth) |
| `efficientnetv2-b2_3rdparty_in1k`\*           | From scratch |   10.10    |   1.99    |   80.63   |   95.30   |    [config](efficientnetv2-b2_8xb32_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/efficientnetv2/efficientnetv2-b2_3rdparty_in1k_20221221-74f7d493.pth) |

| `efficientnetv2-b3_3rdparty_in1k`\*           | From scratch |   14.36    |   3.50    |   82.03   |   95.88   |    [config](efficientnetv2-b3_8xb32_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/efficientnetv2/efficientnetv2-b3_3rdparty_in1k_20221221-b6f07a36.pth) |
| `efficientnetv2-s_3rdparty_in1k`\*            | From scratch |   21.46    |   9.72    |   83.82   |   96.67   | [config](efficientnetv2-s_8xb32_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/efficientnetv2/efficientnetv2-s_3rdparty_in1k_20221220-f0eaff9d.pth) |
| `efficientnetv2-m_3rdparty_in1k`\*            | From scratch |   54.14    |   26.88   |   85.01   |   97.26   | [config](efficientnetv2-m_8xb32_in1k-480px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/efficientnetv2/efficientnetv2-m_3rdparty_in1k_20221220-9dc0c729.pth) |

| `efficientnetv2-l_3rdparty_in1k`\*            | From scratch |   118.52   |   60.14   |   85.43   |   97.31   | [config](efficientnetv2-l_8xb32_in1k-480px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/efficientnetv2/efficientnetv2-l_3rdparty_in1k_20221220-5c3bac0f.pth) |
| `efficientnetv2-s_in21k-pre_3rdparty_in1k`\*  | ImageNet-21k |   21.46    |   9.72    |   84.29   |   97.26   | [config](efficientnetv2-s_8xb32_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/efficientnetv2/efficientnetv2-s_in21k-pre-3rdparty_in1k_20221220-7a7c8475.pth) |
| `efficientnetv2-m_in21k-pre_3rdparty_in1k`\*  | ImageNet-21k |   54.14    |   26.88   |   85.47   |   97.76   | [config](efficientnetv2-m_8xb32_in1k-480px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/efficientnetv2/efficientnetv2-m_in21k-pre-3rdparty_in1k_20221220-a1013a04.pth) |

| `efficientnetv2-l_in21k-pre_3rdparty_in1k`\*  | ImageNet-21k |   118.52   |   60.14   |   86.31   |   97.99   | [config](efficientnetv2-l_8xb32_in1k-480px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/efficientnetv2/efficientnetv2-l_in21k-pre-3rdparty_in1k_20221220-63df0efd.pth) |
| `efficientnetv2-xl_in21k-pre_3rdparty_in1k`\* | ImageNet-21k |   208.12   |   98.34   |   86.39   |   97.83   | [config](efficientnetv2-xl_8xb32_in1k-512px.py) | [model](https://download.openmmlab.com/mmclassification/v0/efficientnetv2/efficientnetv2-xl_in21k-pre-3rdparty_in1k_20221220-583ac18b.pth) |

```bibtex
@inproceedings{tan2021efficientnetv2,
  title={Efficientnetv2: Smaller models and faster training},

```bibtex
  pages={10096--10106},
}
```

# 优化器参数策略可视化

该工具旨在帮助用户检查优化器的超参数调度器（无需训练），支持学习率（learning rate）和动量（momentum）。

## 工具简介

```bash
```

- **`-n, --ngpus`**: 使用 GPU 的数量，默认为 1。
- `--window-size`: 可视化窗口大小，如果没有指定，默认为 `12*7`。如果需要指定，按照格式 `'W*H'`。

```{note}
```

```{note}
```

## 如何在开始训练前可视化学习率曲线

你可以使用如下命令来绘制配置文件 `configs/swin_transformer/swin-base_16xb64_in1k.py` 将会使用的变化率曲线：

```bash
```

We appreciate all contributions to improve MMOCR. Please read [Contribution Guide](/docs/en/notes/contribution_guide.md) for step-by-step instructions to make a contribution to MMOCR, and [CONTRIBUTING.md](https://github.com/open-mmlab/mmcv/blob/master/CONTRIBUTING.md) in MMCV for more details about the contributing guideline.

# Preparing Multi-Moments in Time

```BibTeX
@misc{monfort2019multimoments,
    title={Multi-Moments in Time: Learning and Interpreting Models for Multi-Action Video Understanding},
    author={Mathew Monfort and Kandan Ramakrishnan and Alex Andonian and Barry A McNamara and Alex Lascelles, Bowen Pan, Quanfu Fan, Dan Gutfreund, Rogerio Feris, Aude Oliva},
    eprint={1911.00232},
}
```

For basic dataset information, you can refer to the dataset [website](http://moments.csail.mit.edu).
Before we start, please make sure that the directory is located at `$MMACTION2/tools/data/mmit/`.

## Step 1. Prepare Annotations and Videos

First of all, you have to visit the official [website](http://moments.csail.mit.edu/), fill in an application form for downloading the dataset. Then you will get the download link. You can use `bash preprocess_data.sh` to prepare annotations and videos. However, the download command is missing in that script. Remember to download the dataset to the proper place follow the comment in this script.

```
python ../resize_videos.py ../../../data/mmit/videos/ ../../../data/mmit/videos_256p_dense_cache --dense --level 2
```

## Step 2. Extract RGB and Flow

```

First, you can run the following script to soft link SSD.

```shell
mkdir /mnt/SSD/mmit_extracted/
ln -s /mnt/SSD/mmit_extracted/ ../../../data/mmit/rawframes
```

```shell
```

```shell
```

```shell
```

If both are required, run the following script to extract frames using "tvl1" algorithm.

```shell
```

## Step 3. Generate File List

you can run the follow script to generate file list in the format of rawframes or videos.

```shell
```

## Step 4. Check Directory Structure

After the whole data process for Multi-Moments in Time preparation,
you will get the rawframes (RGB + Flow), videos and annotation files for Multi-Moments in Time.

In the context of the whole project (for Multi-Moments in Time only), the folder structure will look like:

```
mmaction2/
└── data
    └── mmit
        ├── annotations
        │   ├── moments_categories.txt
        │   ├── trainingSet.txt
        │   └── validationSet.txt
        ├── mmit_train_rawframes.txt
        ├── mmit_train_videos.txt
        ├── mmit_val_rawframes.txt
        ├── mmit_val_videos.txt

│   └── validationSet.txt
        ├── rawframes
        │   ├── 0-3-6-2-9-1-2-6-14603629126_5
        │   │   ├── flow_x_00001.jpg
        │   │   ├── flow_x_00002.jpg
        │   │   ├── ...
        │   │   ├── flow_y_00001.jpg
        │   │   ├── flow_y_00002.jpg
        │   │   ├── ...
        │   │   ├── img_00001.jpg
        │   │   └── img_00002.jpg
        │   │   ├── ...
        │   └── yt-zxQfALnTdfc_56
        │   │   ├── ...
        │   └── ...

        └── videos
            └── adult+female+singing
                ├── 0-3-6-2-9-1-2-6-14603629126_5.mp4
                └── yt-zxQfALnTdfc_56.mp4
            └── ...
```

For training and evaluating on Multi-Moments in Time, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).

# DBNet

> [Real-time Scene Text Detection with Differentiable Binarization](https://arxiv.org/abs/1911.08947)

Recently, segmentation-based methods are quite popular in scene text detection, as the segmentation results can more accurately describe scene text of various shapes such as curve text. However, the post-processing of binarization is essential for segmentation-based detection, which converts probability maps produced by a segmentation method into bounding boxes/regions of text. In this paper, we propose a module named Differentiable Binarization (DB), which can perform the binarization process in a segmentation network. Optimized along with a DB module, a segmentation network can adaptively set the thresholds for binarization, which not only simplifies the post-processing but also enhances the performance of text detection. Based on a simple segmentation network, we validate the performance improvements of DB on five benchmark datasets, which consistently achieves state-of-the-art results, in terms of both detection accuracy and speed. In particular, with a light-weight backbone, the performance improvements by DB are significant so that we can look for an ideal tradeoff between detection accuracy and efficiency. Specifically, with a backbone of ResNet-18, our detector achieves an F-measure of 82.8, running at 62 FPS, on the MSRA-TD500 dataset.

<img src="https://user-images.githubusercontent.com/22607038/142791306-0da6db2a-20a6-4a68-b228-64ff275f67b3.png"/>
</div>

### SynthText

|                                  Method                                   | Backbone | Training set | #iters  |                                               Download                                               |
| :-----------------------------------------------------------------------: | :------: | :----------: | :-----: | :--------------------------------------------------------------------------------------------------: |
| [DBNet_r18](/configs/textdet/dbnet/dbnet_resnet18_fpnc_100k_synthtext.py) | ResNet18 |  SynthText   | 100,000 | [model](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet18_fpnc_100k_synthtext/dbnet_resnet18_fpnc_100k_synthtext-2e9bf392.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet18_fpnc_100k_synthtext/20221214_150351.log) |

|             Method             |             Backbone             |             Pretrained Model             |  Training set   |    Test set    | #epochs | Test size | Precision | Recall | Hmean  |             Download             |
| :----------------------------: | :------------------------------: | :--------------------------------------: | :-------------: | :------------: | :-----: | :-------: | :-------: | :----: | :----: | :------------------------------: |

| [DBNet_r18](/configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py) |             ResNet18             |                    -                     | ICDAR2015 Train | ICDAR2015 Test |  1200   |    736    |  0.8853   | 0.7583 | 0.8169 | [model](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015/dbnet_resnet18_fpnc_1200e_icdar2015_20220825_221614-7c0e94f2.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015/20220825_221614.log) |

| [DBNet_r50](/configs/textdet/dbnet/dbnet_resnet50_1200e_icdar2015.py) |             ResNet50             |                    -                     | ICDAR2015 Train | ICDAR2015 Test |  1200   |   1024    |  0.8744   | 0.8276 | 0.8504 | [model](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50_1200e_icdar2015/dbnet_resnet50_1200e_icdar2015_20221102_115917-54f50589.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50_1200e_icdar2015/20221102_115917.log) |

| [DBNet_r50dcn](/configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py) |           ResNet50-DCN           | [Synthtext](https://download.openmmlab.com/mmocr/textdet/dbnet/tmp_1.0_pretrain/dbnet_r50dcnv2_fpnc_sbn_2e_synthtext_20210325-ed322016.pth) | ICDAR2015 Train | ICDAR2015 Test |  1200   |   1024    |  0.8784   | 0.8315 | 0.8543 | [model](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015_20220828_124917-452c443c.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015/20220828_124917.log) |

| [DBNet_r50-oclip](/configs/textdet/dbnet/dbnet_resnet50-oclip_1200e_icdar2015.py) | [ResNet50-oCLIP](https://download.openmmlab.com/mmocr/backbone/resnet50-oclip-7ba0c533.pth) |                    -                     | ICDAR2015 Train | ICDAR2015 Test |  1200   |   1024    |  0.9052   | 0.8272 | 0.8644 | [model](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50-oclip_1200e_icdar2015/dbnet_resnet50-oclip_1200e_icdar2015_20221102_115917-bde8c87a.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50-oclip_1200e_icdar2015/20221102_115917.log) |

|                         Method                         | Backbone | Pretrained Model |  Training set   |    Test set    | #epochs | Test size | Precision | Recall | Hmean  |                         Download                         |

| :----------------------------------------------------: | :------: | :--------------: | :-------------: | :------------: | :-----: | :-------: | :-------: | :----: | :----: | :------------------------------------------------------: |
| [DBNet_r18](/configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_totaltext.py) | ResNet18 |        -         | Totaltext Train | Totaltext Test |  1200   |    736    |  0.8640   | 0.7770 | 0.8182 | [model](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet18_fpnc_1200e_totaltext/dbnet_resnet18_fpnc_1200e_totaltext-3ed3233c.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet18_fpnc_1200e_totaltext/20221219_201038.log) |

```bibtex
@article{Liao_Wan_Yao_Chen_Bai_2020,
    title={Real-Time Scene Text Detection with Differentiable Binarization},
    journal={Proceedings of the AAAI Conference on Artificial Intelligence},
    author={Liao, Minghui and Wan, Zhaoyi and Yao, Cong and Chen, Kai and Bai, Xiang},

journal={Proceedings of the AAAI Conference on Artificial Intelligence},
    pages={11474-11481}}
```

# 基准测试

这里我们对 MMDetection3D 和其他开源 3D 目标检测代码库中模型的训练速度和测试速度进行了基准测试。

## 配置

- 硬件：8 NVIDIA Tesla V100 (32G) GPUs, Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
- 软件：Python 3.7, CUDA 10.1, cuDNN 7.6.5, PyTorch 1.3, numba 0.48.0.
- 模型：由于不同代码库所实现的模型种类有所不同，在基准测试中我们选择了 SECOND、PointPillars、Part-A2 和 VoteNet 几种模型，分别与其他代码库中的相应模型实现进行了对比。
- 度量方法：我们使用整个训练过程中的平均吞吐量作为度量方法，并跳过每个 epoch 的前 50 次迭代以消除训练预热的影响。

## 主要结果

对于模型的训练速度（样本/秒），我们将 MMDetection3D 与其他实现了相同模型的代码库进行了对比。结果如下所示，表格内的数字越大，代表模型的训练速度越快。代码库中不支持的模型使用 `×` 进行标识。

|        模型         | MMDetection3D | OpenPCDet | votenet | Det3D |
| :-----------------: | :-----------: | :-------: | :-----: | :---: |
|       VoteNet       |      358      |     ×     |   77    |   ×   |
|  PointPillars-car   |      141      |     ×     |    ×    |  140  |
| PointPillars-3class |      107      |    44     |    ×    |   ×   |
|       SECOND        |      40       |    30     |    ×    |   ×   |
|       Part-A2       |      17       |    14     |    ×    |   ×   |

## 测试细节

## 测试细节

### 为了计算速度所做的修改

- __MMDetection3D__：我们尝试使用与其他代码库中尽可能相同的配置，具体配置细节见 [基准测试配置](https://github.com/open-mmlab/MMDetection3D/blob/main/configs/benchmark)。

- __Det3D__：为了与 Det3D 进行比较，我们使用了 commit [519251e](https://github.com/poodarchu/Det3D/tree/519251e72a5c1fdd58972eabeac67808676b9bb7) 所对应的代码版本。

- __OpenPCDet__：为了与 OpenPCDet 进行比较，我们使用了 commit [b32fbddb](https://github.com/open-mmlab/OpenPCDet/tree/b32fbddbe06183507bad433ed99b407cbc2175c2) 所对应的代码版本。

  为了计算训练速度，我们在 `./tools/train_utils/train_utils.py` 文件中添加了用于记录运行时间的代码。我们对每个 epoch 的训练速度进行计算，并报告所有 epoch 的平均速度。

  <details>
    <summary>
    （为了使用相同方法进行测试所做的具体修改 - 点击展开）
    </summary>

  ```diff
  diff --git a/tools/train_utils/train_utils.py b/tools/train_utils/train_utils.py
  index 91f21dd..021359d 100644
  --- a/tools/train_utils/train_utils.py
  +++ b/tools/train_utils/train_utils.py

diff --git a/tools/train_utils/train_utils.py b/tools/train_utils/train_utils.py
  @@ -2,6 +2,7 @@ import torch
   import os
   import glob
   import tqdm
  +import datetime
   from torch.nn.utils import clip_grad_norm_

  @@ -13,7 +14,10 @@ def train_one_epoch(model, optimizer, train_loader, model_func, lr_scheduler, ac
       if rank == 0:
           pbar = tqdm.tqdm(total=total_it_each_epoch, leave=leave_pbar, desc='train', dynamic_ncols=True)

  +    start_time = None
       for cur_it in range(total_it_each_epoch):
  +        if cur_it > 49 and start_time is None:
  +            start_time = datetime.datetime.now()
           try:
               batch = next(dataloader_iter)
           except StopIteration:
  @@ -55,9 +59,11 @@ def train_one_epoch(model, optimizer, train_loader, model_func, lr_scheduler, ac

try:
                   tb_log.add_scalar('learning_rate', cur_lr, accumulated_iter)
                   for key, val in tb_dict.items():
                       tb_log.add_scalar('train_' + key, val, accumulated_iter)
  +    endtime = datetime.datetime.now()
  +    speed = (endtime - start_time).seconds / (total_it_each_epoch - 50)
           pbar.close()
  -    return accumulated_iter
  +    return accumulated_iter, speed

   def train_model(model, optimizer, train_loader, model_func, lr_scheduler, optim_cfg,
  @@ -65,6 +71,7 @@ def train_model(model, optimizer, train_loader, model_func, lr_scheduler, optim_
                   lr_warmup_scheduler=None, ckpt_save_interval=1, max_ckpt_save_num=50,
                   merge_all_iters_to_one_epoch=False):

lr_warmup_scheduler=None, ckpt_save_interval=1, max_ckpt_save_num=50,
       accumulated_iter = start_iter
  +    speeds = []
       with tqdm.trange(start_epoch, total_epochs, desc='epochs', dynamic_ncols=True, leave=(rank == 0)) as tbar:
           total_it_each_epoch = len(train_loader)
           if merge_all_iters_to_one_epoch:
  @@ -82,7 +89,7 @@ def train_model(model, optimizer, train_loader, model_func, lr_scheduler, optim_
                   cur_scheduler = lr_warmup_scheduler
               else:
                   cur_scheduler = lr_scheduler
  -            accumulated_iter = train_one_epoch(
  +            accumulated_iter, speed = train_one_epoch(
                   model, optimizer, train_loader, model_func,
                   lr_scheduler=cur_scheduler,
                   accumulated_iter=accumulated_iter, optim_cfg=optim_cfg,

model, optimizer, train_loader, model_func,
  @@ -91,7 +98,7 @@ def train_model(model, optimizer, train_loader, model_func, lr_scheduler, optim_
                   total_it_each_epoch=total_it_each_epoch,
                   dataloader_iter=dataloader_iter
               )
  -
  +            speeds.append(speed)
               # save trained model
               trained_epoch = cur_epoch + 1
               if trained_epoch % ckpt_save_interval == 0 and rank == 0:
  @@ -107,6 +114,8 @@ def train_model(model, optimizer, train_loader, model_func, lr_scheduler, optim_
                   save_checkpoint(
                       checkpoint_state(model, optimizer, trained_epoch, accumulated_iter), filename=ckpt_name,
                   )
  +            print(speed)
  +    print(f'*******{sum(speeds) / len(speeds)}******')

)

   def model_state_to_cpu(model_state):
  ```

- __MMDetection3D__：在 v0.1.0 版本下, 执行如下命令：

  ```bash
  ./tools/dist_train.sh configs/votenet/votenet_8xb16_sunrgbd-3d.py 8 --no-validate
  ```

- __votenet__：在 commit [2f6d6d3](https://github.com/facebookresearch/votenet/tree/2f6d6d36ff98d96901182e935afe48ccee82d566) 版本下，执行如下命令：

  ```bash
  python train.py --dataset sunrgbd --batch_size 16
  ```

  然后执行如下命令，对测试速度进行评估：

  ```bash
  python eval.py --dataset sunrgbd --checkpoint_path log_sunrgbd/checkpoint.tar --batch_size 1 --dump_dir eval_sunrgbd --cluster_sampling seed_fps --use_3d_nms --use_cls_nms --per_class_proposal
  ```

  注意，为了计算推理速度，我们对 `eval.py` 进行了修改。

  <details>
  <summary>
  （为了对相同模型进行测试所做的具体修改 - 点击展开）
  </summary>

  ```diff
  diff --git a/eval.py b/eval.py
    index c0b2886..04921e9 100644
    --- a/eval.py
    +++ b/eval.py

<details>
  <summary>

  ```diff
    @@ -10,6 +10,7 @@ import os
     import sys
     import numpy as np
     from datetime import datetime
    +import time
     import argparse
     import importlib
     import torch
    @@ -28,7 +29,7 @@ parser.add_argument('--checkpoint_path', default=None, help='Model checkpoint pa
     parser.add_argument('--dump_dir', default=None, help='Dump dir to save sample outputs [default: None]')
     parser.add_argument('--num_point', type=int, default=20000, help='Point Number [default: 20000]')
     parser.add_argument('--num_target', type=int, default=256, help='Point Number [default: 256]')
    -parser.add_argument('--batch_size', type=int, default=8, help='Batch Size during training [default: 8]')

-parser.add_argument('--batch_size', type=int, default=8, help='Batch Size during training [default: 8]')
    +parser.add_argument('--batch_size', type=int, default=1, help='Batch Size during training [default: 8]')
     parser.add_argument('--vote_factor', type=int, default=1, help='Number of votes generated from each seed [default: 1]')
     parser.add_argument('--cluster_sampling', default='vote_fps', help='Sampling strategy for vote clusters: vote_fps, seed_fps, random [default: vote_fps]')
     parser.add_argument('--ap_iou_thresholds', default='0.25,0.5', help='A list of AP IoU thresholds [default: 0.25,0.5]')
    @@ -132,6 +133,7 @@ CONFIG_DICT = {'remove_empty_box': (not FLAGS.faster_eval), 'use_3d_nms': FLAGS.
     # ------------------------------------------------------------------------- GLOBAL CONFIG END

     def evaluate_one_epoch():
    +    time_list = list()
         stat_dict = {}
         ap_calculator_list = [APCalculator(iou_thresh, DATASET_CONFIG.class2type) \

def evaluate_one_epoch():
             for iou_thresh in AP_IOU_THRESHOLDS]
    @@ -144,6 +146,8 @@ def evaluate_one_epoch():

             # Forward pass
             inputs = {'point_clouds': batch_data_label['point_clouds']}
    +        torch.cuda.synchronize()
    +        start_time = time.perf_counter()
             with torch.no_grad():
                 end_points = net(inputs)

    @@ -161,6 +165,12 @@ def evaluate_one_epoch():

             batch_pred_map_cls = parse_predictions(end_points, CONFIG_DICT)
             batch_gt_map_cls = parse_groundtruths(end_points, CONFIG_DICT)
    +        elapsed = time.perf_counter() - start_time
    +        time_list.append(elapsed)
    +
    +        if len(time_list==200):

+        torch.cuda.synchronize()
    +
    +            print("average inference time: %4f"%(sum(time_list[5:])/len(time_list[5:])))
             for ap_calculator in ap_calculator_list:
                 ap_calculator.step(batch_pred_map_cls, batch_gt_map_cls)

  ```

### PointPillars-car

  ```bash
  ./tools/dist_train.sh configs/benchmark/hv_pointpillars_secfpn_3x8_100e_det3d_kitti-3d-car.py 8 --no-validate
  ```

- __Det3D__：在 commit [519251e](https://github.com/poodarchu/Det3D/tree/519251e72a5c1fdd58972eabeac67808676b9bb7) 版本下，使用 `kitti_point_pillars_mghead_syncbn.py` 并执行如下命令：

  ```bash
  ./tools/scripts/train.sh --launcher=slurm --gpus=8
  ```

  注意，为了训练 PointPillars，我们对 `train.sh` 进行了修改。

  <details>
  <summary>

  ```diff

```bash
  ```

  <details>
  <summary>

  ```diff
  diff --git a/tools/scripts/train.sh b/tools/scripts/train.sh
  index 3a93f95..461e0ea 100755
  --- a/tools/scripts/train.sh
  +++ b/tools/scripts/train.sh
  @@ -16,9 +16,9 @@ then
   fi

   # Voxelnet
  -python -m torch.distributed.launch --nproc_per_node=8 ./tools/train.py examples/second/configs/  kitti_car_vfev3_spmiddlefhd_rpn1_mghead_syncbn.py --work_dir=$SECOND_WORK_DIR
  +# python -m torch.distributed.launch --nproc_per_node=8 ./tools/train.py examples/second/configs/  kitti_car_vfev3_spmiddlefhd_rpn1_mghead_syncbn.py --work_dir=$SECOND_WORK_DIR
   # python -m torch.distributed.launch --nproc_per_node=8 ./tools/train.py examples/cbgs/configs/  nusc_all_vfev3_spmiddleresnetfhd_rpn2_mghead_syncbn.py --work_dir=$NUSC_CBGS_WORK_DIR

# python -m torch.distributed.launch --nproc_per_node=8 ./tools/train.py examples/cbgs/configs/  nusc_all_vfev3_spmiddleresnetfhd_rpn2_mghead_syncbn.py --work_dir=$NUSC_CBGS_WORK_DIR
   # python -m torch.distributed.launch --nproc_per_node=8 ./tools/train.py examples/second/configs/  lyft_all_vfev3_spmiddleresnetfhd_rpn2_mghead_syncbn.py --work_dir=$LYFT_CBGS_WORK_DIR

   # PointPillars
  -# python -m torch.distributed.launch --nproc_per_node=8 ./tools/train.py ./examples/point_pillars/configs/  original_pp_mghead_syncbn_kitti.py --work_dir=$PP_WORK_DIR
  +python -m torch.distributed.launch --nproc_per_node=8 ./tools/train.py ./examples/point_pillars/configs/  kitti_point_pillars_mghead_syncbn.py
  ```

### PointPillars-3class

  ```bash
  ./tools/dist_train.sh configs/benchmark/hv_pointpillars_secfpn_4x8_80e_pcdet_kitti-3d-3class.py 8 --no-validate
  ```

  ```bash
  ```

- __OpenPCDet__：在 commit [b32fbddb](https://github.com/open-mmlab/OpenPCDet/tree/b32fbddbe06183507bad433ed99b407cbc2175c2) 版本下，执行如下命令：

  ```bash
  cd tools
  sh scripts/slurm_train.sh ${PARTITION} ${JOB_NAME} 8  --cfg_file ./cfgs/kitti_models/pointpillar.yaml --batch_size 32  --workers 32 --epochs 80
  ```

### SECOND

基准测试中的 SECOND 指在 [second.Pytorch](https://github.com/traveller59/second.pytorch) 首次被实现的 [SECONDv1.5](https://github.com/traveller59/second.pytorch/blob/master/second/configs/all.fhd.config)。Det3D 实现的 SECOND 中，使用了自己实现的 Multi-Group Head，因此无法将它的速度与其他代码库进行对比。

  ```bash
  ./tools/dist_train.sh configs/benchmark/hv_second_secfpn_4x8_80e_pcdet_kitti-3d-3class.py 8 --no-validate
  ```

  ```bash
  ```

  ```bash
  cd tools
  sh ./scripts/slurm_train.sh ${PARTITION} ${JOB_NAME} 8  --cfg_file ./cfgs/kitti_models/second.yaml --batch_size 32  --workers 32 --epochs 80
  ```

  ```bash
  ./tools/dist_train.sh configs/benchmark/hv_PartA2_secfpn_4x8_cyclic_80e_pcdet_kitti-3d-3class.py 8 --no-validate
  ```

- __OpenPCDet__：在 commit [b32fbddb](https://github.com/open-mmlab/OpenPCDet/tree/b32fbddbe06183507bad433ed99b407cbc2175c2) 版本下，执行如下命令以进行模型训练：

  ```bash
  cd tools
  sh ./scripts/slurm_train.sh ${PARTITION} ${JOB_NAME} 8  --cfg_file ./cfgs/kitti_models/PartA2.yaml --batch_size 32 --workers 32 --epochs 80
  ```

# 准备 Places365 数据集

```bibtex
 }

```

请从 [Places365](http://places2.csail.mit.edu/download.html) 下载并准备数据。

```text
mmagic
├── mmagic
├── tools
├── data
|   |   ├── meta

```

# 自定义数据集

MMPose 目前已支持了多个任务和相应的数据集。您可以在 [数据集](https://mmpose.readthedocs.io/zh_CN/latest/dataset_zoo.html) 找到它们。请按照相应的指南准备数据。

- [自定义数据集-将数据组织为 COCO 格式](#自定义数据集-将数据组织为-coco-格式)
- [创建自定义数据集的元信息文件](#创建自定义数据集的元信息文件)
- [创建自定义数据集类](#创建自定义数据集类)
- [创建自定义配置文件](#创建自定义配置文件)
- [数据集封装](#数据集封装)

## 将数据组织为 COCO 格式

最简单的使用自定义数据集的方法是将您的注释格式转换为 COCO 数据集格式。

COCO 格式的注释 JSON 文件具有以下必要键：

```python
    {
        'id': 1268
    },
    ...
],
    {
            ...
            424.34,
            223.3]],
            0,0,0,
            0,0,0,
            0,0,0,
            0,0,0,
            0,0,0,

449,232,2,
            0,0,0,
            0,0,0,
            0,0,0,
            0,0,0,
    },
    ...
],
 ]
```

JSON 标注文件中有三个关键词是必需的：

- `images`：包含所有图像信息的列表，每个图像都有一个 `file_name`、`height`、`width` 和 `id` 键。
- `annotations`：包含所有实例标注信息的列表，每个实例都有一个 `segmentation`、`keypoints`、`num_keypoints`、`area`、`iscrowd`、`image_id`、`bbox`、`category_id` 和 `id` 键。
- `categories`：包含所有类别信息的列表，每个类别都有一个 `id` 和 `name` 键。以人体姿态估计为例，`id` 为 1，`name` 为 `person`。

如果您的数据集已经是 COCO 格式的，那么您可以直接使用 `CocoDataset` 类来读取该数据集。

## 创建自定义数据集的元信息文件

对于一个新的数据集而言，您需要创建一个新的数据集元信息文件。该文件包含了数据集的基本信息，如关键点个数、排列顺序、可视化颜色、骨架连接关系等。元信息文件通常存放在 `config/_base_/datasets/` 目录下，例如：

```
config/_base_/datasets/custom.py
```

元信息文件中需要包含以下信息：

- `keypoint_info`：每个关键点的信息：
  1. `name`: 关键点名称，必须是唯一的，例如 `nose`、`left_eye` 等。
  2. `id`: 关键点 ID，必须是唯一的，从 0 开始。
  3. `color`: 关键点可视化时的颜色，以 (\[B, G, R\]) 格式组织起来，用于可视化。
  4. `type`: 关键点类型，可以是 `upper`、`lower` 或 \`\`，用于数据增强。
  5. `swap`: 关键点交换关系，用于水平翻转数据增强。
- `skeleton_info`：骨架连接关系，用于可视化。
- `joint_weights`：每个关键点的权重，用于损失函数计算。
- `sigma`：标准差，用于计算 OKS 分数，详细信息请参考 [keypoints-eval](https://cocodataset.org/#keypoints-eval)。

下面是一个简化版本的元信息文件（[完整版](/configs/_base_/datasets/coco.py)）：

```python

```python
    ),
        0:
        1:
        dict(
            id=1,
        ...
        16:
        dict(
            id=16,
    },
        0:

dict(
            id=16,
    },
        0:
        ...
        18:
        dict(
    },
        1.5
    ],
    sigmas=[
    ])
```

## 创建自定义数据集类

如果标注信息不是用 COCO 格式存储的，那么您需要创建一个新的数据集类。数据集类需要继承自 `BaseDataset` 类，并且需要按照以下步骤实现：

1. 在 `mmpose/datasets/datasets` 目录下找到该数据集符合的 package，如果没有符合的，则创建一个新的 package。

2. 在该 package 下创建一个新的数据集类，在对应的注册器中进行注册：

   ```python

   ```python

   ```

   如果未注册，你会在运行时遇到 `KeyError: 'XXXXX is not in the dataset registry'`。
   关于 `mmengine.BaseDataset` 的更多信息，请参考 [这个文档](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/basedataset.html)。

3. 确保你在 package 的 `__init__.py` 中导入了该数据集类。

4. 确保你在 `mmpose/datasets/__init__.py` 中导入了该 package。

## 创建自定义配置文件

在配置文件中，你需要修改跟数据集有关的部分，例如：

```python
...
# 自定义数据集类

        ...),
    )

data_prefix=dict(img='path/to/your/train/img'),
        ...),
    )

        ...),
    )

        ...),
    )
...
```

请确保所有的路径都是正确的。

## 数据集封装

目前 [MMEngine](https://github.com/open-mmlab/mmengine) 支持以下数据集封装：

- [ConcatDataset](https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/basedataset.html#concatdataset)

## 数据集封装

- [RepeatDataset](https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/basedataset.html#repeatdataset)

MMPose 提供了一个 `CombinedDataset` 类，它可以将多个数据集封装成一个数据集。它的使用方法如下：

```python
        # 使用转换器将标注信息统一为需要的格式
    ])

    ])

]

    ])

]

)
```

- **合并数据集的元信息** 决定了标注格式，可以是子数据集的元信息，也可以是自定义的元信息。如果要自定义元信息，可以参考 [创建自定义数据集的元信息文件](#创建自定义数据集的元信息文件)。
- **KeypointConverter** 用于将不同的标注格式转换成统一的格式。比如将关键点个数不同、关键点排列顺序不同的数据集进行合并。
- 更详细的说明请前往[混合数据集训练](../user_guides/mixed_datasets.md)。

}
```

<details>

```bibtex
}
```

| [pose_resnet_50](/configs/hand_2d_keypoint/topdown_heatmap/coco_wholebody_hand/td-hm_res50_8xb32-210e_coco-wholebody-hand-256x256.py) |  256x256   |  0.800  | 0.833 | 4.64 | [ckpt](https://download.openmmlab.com/mmpose/hand/resnet/res50_coco_wholebody_hand_256x256-8dbc750c_20210908.pth) | [log](https://download.openmmlab.com/mmpose/hand/resnet/res50_coco_wholebody_hand_256x256_20210908.log.json) |

# Preparing GoPro Dataset

```bibtex
}
```

The train datasets can be download from [here](https://drive.google.com/file/d/1zgALzrLCC_tcXKu_iHQTHukKUVT1aodI/). The test datasets can be download from [here](https://drive.google.com/file/d/1k6DTSHu4saUgrGTYkkZXTptILyG9RRll/).

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── GoPro
|   |   ├── test
```

MMDetection3D provides a `Det3DLocalVisualizer` to visualize and store the state of the model during training and testing, as well as results, with the following features.

1. Support the basic drawing interface for multi-modality data and multi-task.
2. Support multiple backends such as local, TensorBoard, to write training status such as `loss`, `lr`, or performance evaluation metrics and to a specified single or multiple backends.
3. Support ground truth visualization on multimodal data, and cross-modal visualization of 3D detection results.

## Basic Drawing Interface

Inherited from `DetLocalVisualizer`, `Det3DLocalVisualizer` provides an interface for drawing common objects on 2D images, such as drawing detection boxes, points, text, lines, circles, polygons, and binary masks. More details about 2D drawing can refer to the [visualization documentation](https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/visualization.html) in MMDetection. Here we introduce the 3D drawing interface:

### Drawing point cloud on the image

We support drawing point cloud on the image by using `draw_points_on_image`.

```python

```

### Drawing 3D Boxes on Point Cloud

We support drawing 3D boxes on point cloud by using `draw_bboxes_3d`.

```python

```

### Drawing Projected 3D Boxes on Image

We support drawing projected 3D boxes on image by using `draw_proj_bboxes_3d`.

```python

```

### Drawing BEV Boxes

We support drawing BEV boxes by using `draw_bev_bboxes`.

```python

```

### Drawing 3D Semantic Mask

We support draw segmentation mask via per-point colorization by using `draw_seg_mask`.

```python

```

## Results

To see the prediction results of trained models, you can run the following command:

```bash
```

After running this command, plotted results including input data and the output of networks visualized on the input will be saved in `${SHOW_DIR}`.

```

After running this command, you will obtain the input data, the output of networks and ground-truth labels visualized on the input (e.g. `***_gt.png` and `***_pred.png` in multi-modality detection task and vision-based detection task) in `${SHOW_DIR}`. When `show` is enabled, [Open3D](http://www.open3d.org/) will be used to visualize the results online. If you are running test in remote server without GUI, the online visualization is not supported. You can download the `results.pkl` from the remote server, and visualize the prediction results offline in your local machine.

To visualize the results with `Open3D` backend offline, you can run the following command:

```bash
```

```bash
```

This allows the inference and results generation to be done in remote server and the users can open them on their host with GUI.

## Dataset

We also provide scripts to visualize the dataset without inference. You can use `tools/misc/browse_dataset.py` to show loaded data and ground-truth online and save them on the disk. Currently we support single-modality 3D detection and 3D segmentation on all the datasets, multi-modality 3D detection on KITTI and SUN RGB-D, as well as monocular 3D detection on nuScenes. To browse the KITTI dataset, you can run the following command:

```shell
```

```shell
```

**Notice**: Once specifying `--output-dir`, the images of views specified by users will be saved when pressing `_ESC_` in open3d window. If you want to zoom out/in the point clouds to inspect more details, you could specify `--show-interval=0` in the command.

To verify the data consistency and the effect of data augmentation, you can also add `--aug` flag to visualize the data after data augmentation using the command as below:

```shell
python tools/misc/browse_dataset.py configs/_base_/datasets/kitti-3d-3class.py --task lidar_det --aug --output-dir ${OUTPUT_DIR}
```

If you also want to show 2D images with 3D bounding boxes projected onto them, you need to find a config that supports multi-modality data loading, and then change the `--task` args to `multi-modality_det`. An example is showed below:

```shell

```shell
```

You can simply browse different datasets using different configs, e.g. visualizing the ScanNet dataset in 3D semantic segmentation task:

```shell
```

And browsing the nuScenes dataset in monocular 3D detection task:

```shell
```

# Contributing to MMagic

1. fork and pull the latest MMagic repository (MMagic)

```{note}
```

### Python

Style configurations can be found in [setup.cfg](/setup.cfg).

```shell
```

```shell
```

```

# build sdk

1. installed opencv (you can skip this step if you have installed it)

   `./install_opencv.sh`

2. set environment variable and path

   `source ./set_env.sh`

3. build sdk

   `./build_sdk.sh` \
   (if you installed opencv by ./install_opencv.sh)

   or

   `./build_sdk.sh "path/to/folder/of/OpenCVConfig.cmake"` \

   the executable will be generated in: `bin/`

# QueryInst

> [Instances as Queries](https://openaccess.thecvf.com/content/ICCV2021/html/Fang_Instances_As_Queries_ICCV_2021_paper.html)

We present QueryInst, a new perspective for instance segmentation. QueryInst is a multi-stage end-to-end system that treats instances of interest as learnable queries, enabling query based object detectors, e.g., Sparse R-CNN, to have strong instance segmentation performance. The attributes of instances such as categories, bounding boxes, instance masks, and instance association embeddings are represented by queries in a unified manner. In QueryInst, a query is shared by both detection and segmentation via dynamic convolutions and driven by parallelly-supervised multi-stage learning. We conduct extensive experiments on three challenging benchmarks, i.e., COCO, CityScapes, and YouTube-VIS to evaluate the effectiveness of QueryInst in object detection, instance segmentation, and video instance segmentation tasks. For the first time, we demonstrate that a simple end-to-end query based framework can achieve the state-of-the-art performance in various instance-level recognition tasks.

<img src="https://user-images.githubusercontent.com/40661020/143971527-c1b7ff78-e95f-4edb-9d5e-3d6d7d902999.png"/>
</div>

|   Model   | Backbone  |  Style  | Lr schd | Number of Proposals | Multi-Scale | RandomCrop | box AP | mask AP |                                 Config                                  |                                                                                                                                                                                                                       Download                                                                                                                                                                                                                       |

| QueryInst | R-50-FPN  | pytorch |   1x    |         100         |    False    |   False    |  42.0  |  37.5   |                [config](./queryinst_r50_fpn_1x_coco.py)                 |                                                                         [model](https://download.openmmlab.com/mmdetection/v2.0/queryinst/queryinst_r50_fpn_1x_coco/queryinst_r50_fpn_1x_coco_20210907_084916-5a8f1998.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/queryinst/queryinst_r50_fpn_1x_coco/queryinst_r50_fpn_1x_coco_20210907_084916.log.json)                                                                         |

| QueryInst | R-50-FPN  | pytorch |   3x    |         100         |    True     |   False    |  44.8  |  39.8   |           [config](./queryinst_r50_fpn_ms-480-800-3x_coco.py)           |                                         [model](https://download.openmmlab.com/mmdetection/v2.0/queryinst/queryinst_r50_fpn_mstrain_480-800_3x_coco/queryinst_r50_fpn_mstrain_480-800_3x_coco_20210901_103643-7837af86.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/queryinst/queryinst_r50_fpn_mstrain_480-800_3x_coco/queryinst_r50_fpn_mstrain_480-800_3x_coco_20210901_103643.log.json)                                         |

| QueryInst | R-50-FPN  | pytorch |   3x    |         300         |    True     |    True    |  47.5  |  41.7   | [config](./queryinst_r50_fpn_300-proposals_crop-ms-480-800-3x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/queryinst/queryinst_r50_fpn_300_proposals_crop_mstrain_480-800_3x_coco/queryinst_r50_fpn_300_proposals_crop_mstrain_480-800_3x_coco_20210904_101802-85cffbd8.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/queryinst/queryinst_r50_fpn_300_proposals_crop_mstrain_480-800_3x_coco/queryinst_r50_fpn_300_proposals_crop_mstrain_480-800_3x_coco_20210904_101802.log.json)   |

| QueryInst | R-101-FPN | pytorch |   3x    |         100         |    True     |   False    |  46.4  |  41.0   |          [config](./queryinst_r101_fpn_ms-480-800-3x_coco.py)           |                                       [model](https://download.openmmlab.com/mmdetection/v2.0/queryinst/queryinst_r101_fpn_mstrain_480-800_3x_coco/queryinst_r101_fpn_mstrain_480-800_3x_coco_20210904_104048-91f9995b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/queryinst/queryinst_r101_fpn_mstrain_480-800_3x_coco/queryinst_r101_fpn_mstrain_480-800_3x_coco_20210904_104048.log.json)                                       |

| QueryInst | R-101-FPN | pytorch |   3x    |         300         |    True     |    True    |  49.0  |  42.9   | [config](./queryinst_r101_fpn_300-proposals_crop-ms-480-800-3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/queryinst/queryinst_r101_fpn_300_proposals_crop_mstrain_480-800_3x_coco/queryinst_r101_fpn_300_proposals_crop_mstrain_480-800_3x_coco_20210904_153621-76cce59f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/queryinst/queryinst_r101_fpn_300_proposals_crop_mstrain_480-800_3x_coco/queryinst_r101_fpn_300_proposals_crop_mstrain_480-800_3x_coco_20210904_153621.log.json) |

```latex
@InProceedings{Fang_2021_ICCV,
    author    = {Fang, Yuxin and Yang, Shusheng and Wang, Xinggang and Li, Yu and Fang, Chen and Shan, Ying and Feng, Bin and Liu, Wenyu},
    title     = {Instances As Queries},
    month     = {October},

title     = {Instances As Queries},
    pages     = {6910-6919}
}
```

# ubuntu 交叉编译 aarch64

mmdeploy 选 ncnn 作为 aarch64 嵌入式 linux 设备的推理后端。 完整的部署分为两部分：

Host

- 模型转换
- 交叉编译嵌入式设备所需 SDK 和 bin

Device

- 运行编译结果

## 1. Host 模型转换

参照文档安装 [mmdeploy](../01-how-to-build/) 和 [mmpretrain](https://github.com/open-mmlab/mmpretrain)，转换 resnet18 对应模型包

```bash

# 模型转换
```

## 2. Host 交叉编译

建议直接用脚本编译

```bash
```

以下是脚本对应的手动过程

a) 安装 aarch64 交叉编译工具

```bash
```

b) 交叉编译 opencv 安装到 tmp 目录

```bash

```

```bash
```

```bash
..
```

c) 交叉编译 ncnn 安装到 tmp 目录

```bash
cmake .. \
..
```

d) 交叉编译 mmdeploy，install/bin 目录是可执行文件

```bash
cmake .. \

```bash
cmake .. \
..
```

## 3. Device 执行

确认转换模型用了 `--dump-info`，这样 `resnet18` 目录才有 `pipeline.json` 等 SDK 所需文件。

把 dump 好的模型目录(resnet18)、可执行文件(image_classification)、测试图片(tests/data/tiger.jpeg)、交叉编译的 OpenCV(/tmp/ocv-aarch64) 拷贝到设备中

```bash
..
```

# Kvasir-Sessile Dataset (Kvasir SEG)

This project supports **`Kvasir-Sessile Dataset (Kvasir SEG) `**, which can be downloaded from [here](https://opendatalab.com/Kvasir-Sessile_dataset).

## Dataset Overview

The Kvasir-SEG dataset contains polyp images and their corresponding ground truth from the Kvasir Dataset v2. The resolution of the images contained in Kvasir-SEG varies from 332x487 to 1920x1072 pixels.

| Dataset Name                                                  | Anatomical Region | Task Type    | Modality  | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License                                                   |

| ------------------------------------------------------------- | ----------------- | ------------ | --------- | ------------ | --------------------- | ---------------------- | ------------ | --------------------------------------------------------- |
| [Kvarsir-SEG](https://opendatalab.com/Kvasir-Sessile_dataset) | abdomen           | segmentation | endoscopy | 2            | 196/-/-               | yes/-/-                | 2020         | [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/) |

| background |    196     |   92.31    |    -     |    -     |     -     |     -     |
|   polyp    |    196     |    7.69    |    -     |    -     |     -     |     -     |

Note:

Note:

![kvasir-seg](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/endoscopy_images/kvasir_seg/kvasir_seg_dataset.png?raw=true)

```
 }
```

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `kvasir_seg/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
```

- download dataset from [here](https://opendatalab.com/Kvasir-Sessile_dataset) and decompress data to path `'data/'`.

```none
  ├── mmseg
  │   │   │   │   ├── kvasir_seg

```

| background |    156     |   92.28    |    40    |  92.41   |     -     |     -     |
|   polyp    |    156     |    7.72    |    40    |   7.59   |     -     |     -     |

```shell
mim train mmseg .configs/${CONFIG_FILE}
```

```shell
```

```shell
```

# 2-PM Vessel Dataset

This project supports **`2-PM Vessel Dataset`**, which can be downloaded from [here](https://opendatalab.org.cn/2-PM_Vessel_Dataset).

An open-source volumetric brain vasculature dataset obtained with two-photon microscopy at Focused Ultrasound Lab, at Sunnybrook Research Institute (affiliated with University of Toronto by Dr. Alison Burgess, Charissa Poon and Marc Santos).

The dataset contains a total of 12 volumetric stacks consisting images of mouse brain vasculature and tumor vasculature.

| Dataset Name                                                 | Anatomical Region | Task Type    | Modality          | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License                                                       |

| ------------------------------------------------------------ | ----------------- | ------------ | ----------------- | ------------ | --------------------- | ---------------------- | ------------ | ------------------------------------------------------------- |
| [2pm_vessel](https://opendatalab.org.cn/2-PM_Vessel_Dataset) | vessel            | segmentation | microscopy_images | 2            | 216/-/-               | yes/-/-                | 2021         | [CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/) |

| background |    216     |   85.78    |    -     |    -     |     -     |     -     |
|   vessel   |    180     |   14.22    |    -     |    -     |     -     |     -     |

Note:

Note:

![2pmv](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/histopathology/2pm_vessel/2pm_vessel_dataset.png?raw=true)

```
@article{teikari2016deep,
	title={Deep learning convolutional networks for multiphoton microscopy vasculature segmentation},
	author={Teikari, Petteri and Santos, Marc and Poon, Charissa and Hynynen, Kullervo},
	journal={arXiv preprint arXiv:1606.02382},
	year={2016}
}
```

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `2pm_vessel/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
```

- download dataset from [here](https://opendatalab.org.cn/2-PM_Vessel_Dataset) and decompress data to path `'data/'`.

```shell
odl get    2-PM_Vessel_Dataset
cd ..
```

```none
  ├── mmseg
  │   │   │   │   ├── 2pm_vessel

```

| background |    172     |   85.88    |    44    |   85.4   |     -     |     -     |
|   vessel   |    142     |   14.12    |    38    |   14.6   |     -     |     -     |

```shell
```

```shell
```

# Removing the Bias of Integral Pose Regression

<details>

```bibtex
  }
```

Heatmap-based detection methods are dominant for 2D human pose estimation even though regression is more intuitive. The introduction of the integral regression method, which, architecture-wise uses an implicit heatmap, brings the two approaches even closer together. This begs the question -- does detection really outperform regression? In this paper, we investigate the difference in supervision between the heatmap-based detection and integral regression, as this is the key remaining difference between the two approaches. In the process, we discover an underlying bias behind integral pose regression that arises from taking the expectation after the softmax function. To counter the bias, we present a compensation method which we find to improve integral regression accuracy on all 2D pose estimation benchmarks. We further propose a simple combined detection and bias-compensated regression method that considerably outperforms state-of-the-art baselines with few added components.

<img src="https://user-images.githubusercontent.com/13503330/189810184-159432bb-32a1-403c-8150-e90edce1a5bb.png">
</div>

# Rotated RetinaNet

> [Focal loss for dense object detection](https://arxiv.org/pdf/1708.02002.pdf)

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/retina.png" width="800"/>
</div>

The highest accuracy object detectors to date are based on a two-stage approach popularized by R-CNN, where a classifier is applied to a sparse set of candidate object locations. In contrast, one-stage detectors that are applied over a regular, dense sampling of possible object locations have the potential to be faster and simpler, but have trailed the accuracy of two-stage detectors thus far. In this paper, we investigate why this is the case. We discover that the extreme foreground-background class imbalance encountered during training of dense detectors is the central cause. We propose to address this class imbalance by reshaping the standard cross entropy loss such that it down-weights the loss assigned to well-classified examples. Our novel Focal Loss focuses training on a sparse set of hard examples and prevents the vast number of easy negatives from overwhelming the detector during training. To evaluate the effectiveness of our loss, we design and train a simple dense detector we call RetinaNet. Our results show that when trained with the focal loss, RetinaNet is able to match the speed of previous one-stage detectors while surpassing the accuracy of all existing state-of-the-art two-stage detectors.

DOTA1.0

|         Backbone         |  mAP  | Angle | lr schd | Mem (GB) | Inf Time (fps) |  Aug  | Batch Size |                                                   Configs                                                   |                                                                                                                                                                                            Download                                                                                                                                                                                            |

| ResNet50 (1024,1024,200) | 64.55 |  oc   |   1x    |   3.38   |      15.7      |   -   |     2      |         [rotated-retinanet-hbox-oc_r50_fpn_1x_dota](./rotated-retinanet-hbox-oc_r50_fpn_1x_dota.py)         |                 [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_hbb_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_r50_fpn_1x_dota_oc-e8a7c7df.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_hbb_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_r50_fpn_1x_dota_oc_20220121_095315.log.json)                 |

| ResNet50 (1024,1024,200) | 68.42 | le90  |   1x    |   3.38   |      16.9      |   -   |     2      |       [rotated-retinanet-rbox-le90_r50_fpn_1x_dota](./rotated-retinanet-rbox-le90_r50_fpn_1x_dota.py)       |             [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le90/rotated_retinanet_obb_r50_fpn_1x_dota_le90-c0097bc4.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le90/rotated_retinanet_obb_r50_fpn_1x_dota_le90_20220128_130740.log.json)             |

| ResNet50 (1024,1024,200) | 68.79 | le90  |   1x    |   2.36   |      22.4      |   -   |     2      |   [rotated-retinanet-rbox-le90_r50_fpn_amp-1x_dota](./rotated-retinanet-rbox-le90_r50_fpn_amp-1x_dota.py)   |   [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_fp16_1x_dota_le90/rotated_retinanet_obb_r50_fpn_fp16_1x_dota_le90-01de71b5.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_fp16_1x_dota_le90/rotated_retinanet_obb_r50_fpn_fp16_1x_dota_le90_20220303_183714.log.json)   |

| ResNet50 (1024,1024,200) | 69.79 | le135 |   1x    |   3.38   |      17.2      |   -   |     2      |      [rotated-retinanet-rbox-le135_r50_fpn_1x_dota](./rotated-retinanet-rbox-le135_r50_fpn_1x_dota.py)      |           [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le135/rotated_retinanet_obb_r50_fpn_1x_dota_le135-e4131166.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le135/rotated_retinanet_obb_r50_fpn_1x_dota_le135_20220128_130755.log.json)           |

| ResNet50 (1024,1024,500) | 76.50 | le90  |   1x    |          |      17.5      | MS+RR |     2      | [rotated-retinanet-rbox-le90_r50_fpn_rr-1x_dota-ms](./rotated-retinanet-rbox-le90_r50_fpn_rr-1x_dota-ms.py) | [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_ms_rr_le90/rotated_retinanet_obb_r50_fpn_1x_dota_ms_rr_le90-1da1ec9c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_ms_rr_le90/rotated_retinanet_obb_r50_fpn_1x_dota_ms_rr_le90_20220210_114843.log.json) |

HRSC

HRSC

|      Backbone      |  mAP  | AP50  | AP75  | Angle | lr schd | Mem (GB) | Inf Time (fps) | Aug | Batch Size |                                                Configs                                                |                                                                                                                                                                                      Download                                                                                                                                                                                      |

| ResNet50 (800,512) | 45.09 | 79.30 | 46.90 |  oc   |   6x    |   1.56   |      39.2      | RR  |     2      |   [rotated-retinanet-hbox-oc_r50_fpn_rr-6x_hrsc](./rotated-retinanet-hbox-oc_r50_fpn_rr-6x_hrsc.py)   |     [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_hbb_r50_fpn_6x_hrsc_rr_oc/rotated_retinanet_hbb_r50_fpn_6x_hrsc_rr_oc-f37eada6.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_hbb_r50_fpn_6x_hrsc_rr_oc/rotated_retinanet_hbb_r50_fpn_6x_hrsc_rr_oc_20220412_103639.log.json)     |

| ResNet50 (800,512) | 52.06 | 84.80 | 58.10 | le90  |   6x    |   1.56   |      38.2      | RR  |     2      | [rotated-retinanet-rbox-le90_r50_fpn_rr-6x_hrsc](./rotated-retinanet-rbox-le90_r50_fpn_rr-6x_hrsc.py) | [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_6x_hrsc_rr_le90/rotated_retinanet_obb_r50_fpn_6x_hrsc_rr_le90-ee4f18af.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_6x_hrsc_rr_le90/rotated_retinanet_obb_r50_fpn_6x_hrsc_rr_le90_20220412_110739.log.json) |

DIOR

DIOR

|      Backbone      |  mAP  | Angle | lr schd | Mem (GB) | Inf Time (fps) | Aug | Batch Size |                                             Configs                                             |                                                                                                                                                                              Download                                                                                                                                                                              |

| ResNet50 (800,800) | 50.26 |  oc   |   1x    |   3.66   |                |  -  |     2      |   [rotated-retinanet-rbox-oc_r50_fpn_1x_dior](./rotated-retinanet-rbox-oc_r50_fpn_1x_dior.py)   |     [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_retinanet/rotated-retinanet-rbox-oc_r50_fpn_1x_dior/rotated-retinanet-rbox-oc_r50_fpn_1x_dior-dbdbc2f8.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_retinanet/rotated-retinanet-rbox-oc_r50_fpn_1x_dior/rotated-retinanet-rbox-oc_r50_fpn_1x_dior_20221124_173852.json)     |

| ResNet50 (800,800) | 53.94 | le90  |   1x    |   3.62   |                |  -  |     2      | [rotated-retinanet-rbox-le90_r50_fpn_1x_dior](./rotated-retinanet-rbox-le90_r50_fpn_1x_dior.py) | [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_retinanet/rotated-retinanet-rbox-le90_r50_fpn_1x_dior/rotated-retinanet-rbox-le90_r50_fpn_1x_dior-caf9143c.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_retinanet/rotated-retinanet-rbox-le90_r50_fpn_1x_dior/rotated-retinanet-rbox-le90_r50_fpn_1x_dior_20221124_230602.json) |

Notes:

```
@inproceedings{lin2017focal,
  title={Focal loss for dense object detection},

```
  author={Lin, Tsung-Yi and Goyal, Priya and Girshick, Ross and He, Kaiming and Doll{\'a}r, Piotr},
}
```

# 准备 UDM10 数据集

```bibtex
@inproceedings{PFNL,
    title={Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations},
    author={Yi, Peng and Wang, Zhongyuan and Jiang, Kui and Jiang, Junjun and Ma, Jiayi},
    booktitle={IEEE International Conference on Computer Vision (ICCV)},
    pages={3106-3115},
}
```

数据集可以从 [此处](https://drive.google.com/file/d/1G4V4KZZhhfzUlqHiSBBuWyqLyIOvOs0W/) 下载。

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── UDM10
|   |   ├── GT
|   |   ├── BIx4
|   |   ├── BDx4
```

}
```

<details>

```bibtex
}
```

institution={arXiv:1705.01389},
}
```

| Arch                                                       | Input Size | PCKh@0.7 |  AUC  | EPE  |                            ckpt                            |                            log                            |
| :--------------------------------------------------------- | :--------: | :------: | :---: | :--: | :--------------------------------------------------------: | :-------------------------------------------------------: |

| [pose_hrnetv2_w18_udp](/configs/hand_2d_keypoint/topdown_heatmap/rhd2d/td-hm_hrnetv2-w18_udp-8xb64-210e_rhd2d-256x256.py) |  256x256   |  0.992   | 0.902 | 2.19 | [ckpt](https://download.openmmlab.com/mmpose/hand/udp/hrnetv2_w18_rhd2d_256x256_udp-63ba6007_20210330.pth) | [log](https://download.openmmlab.com/mmpose/hand/udp/hrnetv2_w18_rhd2d_256x256_udp_20210330.log.json) |

<!-- Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers. -->

<!-- Please describe the motivation of this PR and the goal you want to achieve through this PR. -->

<!-- Please briefly describe what modification is made in this PR. -->

<!-- Does the modification introduce changes that break the backward compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR. -->

<!-- If this PR introduces a new feature, it is better to list some use cases here and update the documentation. -->

- [ ] I have read and followed the workflow indicated in the [CONTRIBUTING.md](https://github.com/open-mmlab/mmpose/blob/master/.github/CONTRIBUTING.md) to create this PR.
- [ ] Pre-commit or linting tools indicated in [CONTRIBUTING.md](https://github.com/open-mmlab/mmpose/blob/master/.github/CONTRIBUTING.md) are used to fix the potential lint issues.
- [ ] New functionalities are covered by complete unit tests. If not, please add more unit tests to ensure correctness.

# Backends Support

We support different file client backends: Disk, Ceph and LMDB, etc. Here is an example of how to modify configs for Ceph-based data loading and saving.

## Load data and annotations from Ceph

We support loading data and generated annotation info files (pkl and json) from Ceph:

```python
# set file client backends as Ceph
backend_args = dict(
    backend='petrel',
    path_mapping=dict({
        './data/nuscenes/':
        's3://openmmlab/datasets/detection3d/nuscenes/', # replace the path with your data path on Ceph
        'data/nuscenes/':
        's3://openmmlab/datasets/detection3d/nuscenes/' # replace the path with your data path on Ceph
    }))

    rate=1.0,
    prepare=dict(filter_by_difficulty=[-1], filter_by_min_points=dict(Car=5)),
    sample_groups=dict(Car=15),
    # set file client for points loader to load training data

prepare=dict(filter_by_difficulty=[-1], filter_by_min_points=dict(Car=5)),
    # set file client for data base sampler to load db info file

    # set file client for loading training data
    dict(type='LoadPointsFromFile', coord_type='LIDAR', load_dim=4, use_dim=4, backend_args=backend_args),
    # set file client for loading training data annotations
    dict(type='LoadAnnotations3D', with_bbox_3d=True, with_label_3d=True, backend_args=backend_args),
    dict(
        translation_std=[0.25, 0.25, 0.25],

    dict(
        rot_range=[-0.15707963267, 0.15707963267]),
    dict(
]
    # set file client for loading validation/testing data
    dict(

dict(type='LoadPointsFromFile', coord_type='LIDAR', load_dim=4, use_dim=4, backend_args=backend_args),
    dict(
            dict(
            dict(
            dict(
        ])
]

    # set file client for loading training info files (.pkl)
        times=2,

])
]

        times=2,
        dataset=dict(pipeline=train_pipeline, classes=class_names, backend_args=backend_args)),
    # set file client for loading validation info files (.pkl)
    val=dict(pipeline=test_pipeline, classes=class_names,backend_args=backend_args),
    # set file client for loading testing info files (.pkl)
    test=dict(pipeline=test_pipeline, classes=class_names, backend_args=backend_args))
```

## Load pretrained model from Ceph

```python
    pts_backbone=dict(
        type='NoStemRegNet',
        arch='regnetx_1.6gf',
            type='Pretrained', checkpoint='s3://openmmlab/checkpoints/mmdetection3d/regnetx_1.6gf'), # replace the path with your pretrained model path on Ceph
        ...
```

## Load checkpoint from Ceph

```python

...
```

```python
# replace the path with your checkpoint path on Ceph
load_from = 's3://openmmlab/checkpoints/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-car/hv_pointpillars_secfpn_6x8_160e_kitti-3d-car_20200620_230614-77663cd6.pth.pth'
```

## Save checkpoint into Ceph

```python
# checkpoint saving
# replace the path with your checkpoint saving path on Ceph
checkpoint_config = dict(interval=1, max_keep_ckpts=2, out_dir='s3://openmmlab/mmdetection3d')
```

## EvalHook saves the best checkpoint into Ceph

```python
evaluation = dict(interval=1, save_best='bbox', out_dir='s3://openmmlab/mmdetection3d')
```

## Save the training log into Ceph

The training log will be backed up to the specified Ceph path after training.

```python
    hooks=[

```

```python
    hooks=[
        dict(type='TextLoggerHook', out_dir='s3://openmmlab/mmdetection3d'),
    ])
```

You can also delete the local training log after backing up to the specified Ceph path by setting `keep_local = False`.

```python
    hooks=[
        dict(type='TextLoggerHook', out_dir='s3://openmmlab/mmdetection3d', keep_local=False),
    ])
```

```txt
13_15_0_par.jpg WYNDHAM
13_15_1_par.jpg HOTEL
12_16_0_par.jpg UNITED
```

MMPose provides a wide variety of pre-trained models for pose estimation, which can be found in the [Model Zoo](https://mmpose.readthedocs.io/en/latest/model_zoo.html).
This guide will demonstrate **how to perform inference**, or running pose estimation on provided images or videos using trained models.

For instructions on testing existing models on standard datasets, refer to this [guide](./train_and_test.md#test).

In MMPose, a model is defined by a configuration file, while its pre-existing parameters are stored in a checkpoint file. You can find the model configuration files and corresponding checkpoint URLs in the [Model Zoo](https://mmpose.readthedocs.io/en/latest/modelzoo.html). We recommend starting with the HRNet model, using [this configuration file](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192.py) and [this checkpoint file](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192-81c58e40_20220909.pth).

## Inferencer: a Unified Inference Interface

MMPose offers a comprehensive API for inference, known as `MMPoseInferencer`. This API enables users to perform inference on both images and videos using all the models supported by MMPose. Furthermore, the API provides automatic visualization of inference results and allows for the convenient saving of predictions.

The `MMPoseInferencer` can be used in any Python program to perform pose estimation. Below is an example of inference on a given image using the pre-trained human pose estimator within the Python shell.

```python

img_path = 'tests/data/coco/000000000785.jpg'   # replace this with your own image path

# instantiate the inferencer using the model alias

# The MMPoseInferencer API employs a lazy inference approach,
# creating a prediction generator when given input

```

If everything works fine, you will see the following image in a new window:

The `result` variable is a dictionary comprising two keys, `'visualization'` and `'predictions'`.

- `'visualization'` holds a list which:

  - contains visualization results, such as the input image, markers of the estimated poses, and optional predicted heatmaps.
  - remains empty if the `return_vis` argument is not specified.

- `'predictions'` stores:

  - a list of estimated keypoints for each identified instance.

The structure of the `result` dictionary is as follows:

```python
result = {
        # number of elements: batch_size (defaults to 1)

```python
result = {
        ...
    ],
        # pose estimation result of each image
        [
            # pose information of each detected instance
            # number of elements: number of detected instances
            {'keypoints': ...,  # instance 1
            ...
            },
            {'keypoints': ...,  # instance 2
            ...
            },
        ]
    ...
    ]
}

```

A **command-line interface (CLI)** tool for the inferencer is also available: `demo/inferencer_demo.py`. This tool allows users to perform inference using the same model and inputs with the following command:

```bash

```bash
```

The predictions will be save in `predictions/000000000785.json`. The argument names correspond with the `MMPoseInferencer`, which serves as an API.

The inferencer is capable of processing a range of input types, which includes the following:

- A path to an image
- A path to a video
- A path to a folder (which will cause all images in that folder to be inferred)
- An image array (NA for CLI tool)
- A list of image arrays (NA for CLI tool)
- A webcam (in which case the `input` parameter should be set to either `'webcam'` or `'webcam:{CAMERA_ID}'`)

Please note that when the input corresponds to multiple images, such as when the input is a video or a folder path, the inference process needs to iterate over the results generator in order to perform inference on all the frames or images within the folder. Here's an example in Python:

```python

```python

```

In this example, the `inferencer` takes the `folder_path` as input and returns a generator object (`result_generator`) that produces inference results. By iterating over the `result_generator` and storing each result in the `results` list, you can obtain the inference results for all the frames or images within the folder.

### Custom Pose Estimation Models

The inferencer provides several methods that can be used to customize the models employed:

```python

# build the inferencer with model alias

# build the inferencer with model config name

# build the inferencer with model config path and checkpoint path/URL

)
```

The complere list of model alias can be found in the [Model Alias](#model-alias) section.

**Custom Object Detector for Top-down Pose Estimation Models**

In addition, top-down pose estimators also require an object detection model. The inferencer is capable of inferring the instance type for models trained with datasets supported in MMPose, and subsequently constructing the necessary object detection model. Alternatively, users may also manually specify the detection model using the following methods:

```python

# specify detection model by alias
# the available aliases include 'human', 'hand', 'face', 'animal',

```python

# as well as any additional aliases defined in mmdet
    # suppose the pose estimator is trained on custom dataset
)

# specify detection model with model config name
    det_cat_ids=[0],  # the category id of 'human' class
)

# specify detection model with config path and checkpoint path/URL

det_weights='https://download.openmmlab.com/mmdetection/v2.0/' \
)
```

To perform top-down pose estimation on cropped images containing a single object, users can set `det_model='whole_image'`. This bypasses the object detector initialization, creating a bounding box that matches the input image size and directly sending the entire image to the top-down pose estimator.

### Dump Results

After performing pose estimation, you might want to save the results for further analysis or processing. This section will guide you through saving the predicted keypoints and visualizations to your local machine.

To save the predictions in a JSON file, use the `pred_out_dir` argument when running the inferencer:

```python

```python
```

The predictions will be saved in the `predictions/` folder in JSON format, with each file named after the corresponding input image or video.

For more advanced scenarios, you can also access the predictions directly from the `result` dictionary returned by the inferencer. The key `'predictions'` contains a list of predicted keypoints for each individual instance in the input image or video. You can then manipulate or store these results using your preferred method.

Keep in mind that if you want to save both the visualization images and the prediction files in a single folder, you can use the `out_dir` argument:

```python
```

```python
```

In this case, the visualization images will be saved in the `output/visualization/` folder, while the predictions will be stored in the `output/predictions/` folder.

The inferencer can automatically draw predictions on input images or videos. Visualization results can be displayed in a new window and saved locally.

To view the visualization results in a new window, use the following code:

```python
```

Notice that:

- If the input video comes from a webcam, displaying the visualization results in a new window will be enabled by default, allowing users to see the inputs.
- If there is no GUI on the platform, this step may become stuck.

To save the visualization results locally, specify the `vis_out_dir` argument like this:

```python

```python
```

The input images or videos with predicted poses will be saved in the `vis_results/` folder.

As seen in the above image, the visualization of estimated poses consists of keypoints (depicted by solid circles) and skeletons (represented by lines). The default size of these visual elements might not produce satisfactory results. Users can adjust the circle size and line thickness using the `radius` and `thickness` arguments, as shown below:

```python
```

### Arguments of Inferencer

```python
```

The `MMPoseInferencer` offers a variety of arguments for customizing pose estimation, visualization, and saving predictions. Below is a list of the arguments available when initializing the inferencer and their descriptions:

| Argument         | Description                                                                                                      |
| ---------------- | ---------------------------------------------------------------------------------------------------------------- |
| `pose2d`         | Specifies the model alias, configuration file name, or configuration file path for the 2D pose estimation model. |
| `pose2d_weights` | Specifies the URL or local path to the 2D pose estimation model's checkpoint file.                               |

| `pose3d`         | Specifies the model alias, configuration file name, or configuration file path for the 3D pose estimation model. |
| `pose3d_weights` | Specifies the URL or local path to the 3D pose estimation model's checkpoint file.                               |
| `det_model`      | Specifies the model alias, configuration file name, or configuration file path for the object detection model.   |
| `det_weights`    | Specifies the URL or local path to the object detection model's checkpoint file.                                 |
| `det_cat_ids`    | Specifies the list of category IDs corresponding to the object classes to be detected.                           |
| `device`         | The device to perform the inference. If left `None`, the Inferencer will select the most suitable one.           |

| `scope`          | The namespace where the model modules are defined.                                                               |

The inferencer is designed for both visualization and saving predictions. The table below presents the list of arguments available when using the `MMPoseInferencer` for inference, along with their compatibility with 2D and 3D inferencing:

| Argument                 | Description                                                                                                                                                       | 2D  | 3D  |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | --- |

| `show`                   | Controls the display of the image or video in a pop-up window.                                                                                                    | ✔️  | ✔️  |
| `radius`                 | Sets the visualization keypoint radius.                                                                                                                           | ✔️  | ✔️  |
| `thickness`              | Determines the link thickness for visualization.                                                                                                                  | ✔️  | ✔️  |
| `kpt_thr`                | Sets the keypoint score threshold. Keypoints with scores exceeding this threshold will be displayed.                                                              | ✔️  | ✔️  |

| `draw_bbox`              | Decides whether to display the bounding boxes of instances.                                                                                                       | ✔️  | ✔️  |
| `draw_heatmap`           | Decides if the predicted heatmaps should be drawn.                                                                                                                | ✔️  | ❌  |
| `black_background`       | Decides whether the estimated poses should be displayed on a black background.                                                                                    | ✔️  | ❌  |
| `skeleton_style`         | Sets the skeleton style. Options include 'mmpose' (default) and 'openpose'.                                                                                       | ✔️  | ❌  |

| `use_oks_tracking`       | Decides whether to use OKS as a similarity measure in tracking.                                                                                                   | ❌  | ✔️  |
| `tracking_thr`           | Sets the similarity threshold for tracking.                                                                                                                       | ❌  | ✔️  |
| `norm_pose_2d`           | Decides whether to scale the bounding box to the dataset's average bounding box scale and relocate the bounding box to the dataset's average bounding box center. | ❌  | ✔️  |
| `rebase_keypoint_height` | Decides whether to set the lowest keypoint with height 0.                                                                                                         | ❌  | ✔️  |

| `return_vis`             | Decides whether to include visualization images in the results.                                                                                                   | ✔️  | ✔️  |
| `vis_out_dir`            | Defines the folder path to save the visualization images. If unset, the visualization images will not be saved.                                                   | ✔️  | ✔️  |
| `return_datasample`      | Determines if the prediction should be returned in the `PoseDataSample` format.                                                                                   | ✔️  | ✔️  |
| `pred_out_dir`           | Specifies the folder path to save the predictions. If unset, the predictions will not be saved.                                                                   | ✔️  | ✔️  |

| `out_dir`                | If `vis_out_dir` or `pred_out_dir` is unset, these will be set to `f'{out_dir}/visualization'` or `f'{out_dir}/predictions'`, respectively.                       | ✔️  | ✔️  |

### Model Alias

The MMPose library has predefined aliases for several frequently used models. These aliases can be utilized as a shortcut when initializing the `MMPoseInferencer`, as an alternative to providing the full model configuration name. Here are the available 2D model aliases and their corresponding configuration names:

| Alias     | Configuration Name                                 | Task                            | Pose Estimator | Detector            |
| --------- | -------------------------------------------------- | ------------------------------- | -------------- | ------------------- |
| animal    | rtmpose-m_8xb64-210e_ap10k-256x256                 | Animal pose estimation          | RTMPose-m      | RTMDet-m            |

| human     | rtmpose-m_8xb256-420e_aic-coco-256x192             | Human pose estimation           | RTMPose-m      | RTMDet-m            |
| face      | rtmpose-m_8xb64-60e_wflw-256x256                   | Face keypoint detection         | RTMPose-m      | yolox-s             |
| hand      | rtmpose-m_8xb32-210e_coco-wholebody-hand-256x256   | Hand keypoint detection         | RTMPose-m      | ssdlite_mobilenetv2 |
| wholebody | rtmpose-m_8xb64-270e_coco-wholebody-256x192        | Human wholebody pose estimation | RTMPose-m      | RTMDet-m            |
| vitpose   | td-hm_ViTPose-base-simple_8xb64-210e_coco-256x192  | Human pose estimation           | ViTPose-base   | RTMDet-m            |
| vitpose-s | td-hm_ViTPose-small-simple_8xb64-210e_coco-256x192 | Human pose estimation           | ViTPose-small  | RTMDet-m            |

| vitpose-b | td-hm_ViTPose-base-simple_8xb64-210e_coco-256x192  | Human pose estimation           | ViTPose-base   | RTMDet-m            |
| vitpose-l | td-hm_ViTPose-large-simple_8xb64-210e_coco-256x192 | Human pose estimation           | ViTPose-large  | RTMDet-m            |
| vitpose-h | td-hm_ViTPose-huge-simple_8xb64-210e_coco-256x192  | Human pose estimation           | ViTPose-huge   | RTMDet-m            |

The following table lists the available 3D model aliases and their corresponding configuration names:

| Alias   | Configuration Name                                        | Task                     | 3D Pose Estimator | 2D Pose Estimator | Detector |
| ------- | --------------------------------------------------------- | ------------------------ | ----------------- | ----------------- | -------- |

| human3d | pose-lift_videopose3d-243frm-supv-cpn-ft_8xb128-200e_h36m | Human 3D pose estimation | VideoPose3D       | RTMPose-m         | RTMDet-m |

In addition, users can utilize the CLI tool to display all available aliases with the following command:

```shell
```

# 自定义模型 (待更新)

我们大致将模型组件分为了 5 种类型。

- 主干网络 (Backbone): 通常是一个全卷积网络 (FCN)，用来提取特征图，比如残差网络 (ResNet)。也可以是基于视觉 Transformer 的网络，比如 Swin Transformer 等。
- Neck: 主干网络和任务头 (Head) 之间的连接组件，比如 FPN, ReFPN。
- 任务头 (Head): 用于某种具体任务（比如边界框预测）的组件。
- 区域特征提取器 (Roi Extractor): 用于从特征图上提取区域特征的组件，比如 RoI Align Rotated。
- 损失 (loss): 任务头上用于计算损失函数的组件，比如 FocalLoss, GWDLoss, and KFIoULoss。

## 开发新的组件

### 添加新的主干网络

新建文件 `mmrotate/models/backbones/mobilenet.py`。

```python

        pass

        pass
```

#### 2. 导入模块

你可以将下面的代码添加到 `mmrotate/models/backbones/__init__.py` 中：

```python
```

或者添加如下代码

```python

```python
```

或者添加如下代码

```python
```

到配置文件中以避免修改原始代码。

```python
    ...
        arg1=xxx,
    ...
```

新建文件 `mmrotate/models/necks/pafpn.py`。

```python

                num_outs,
        pass

        pass
```

你可以添加下述代码到 `mmrotate/models/necks/__init__.py` 中

```python

pass

        pass
```

```python
```

或者添加

```python
```

```python
```

这里，我们以 [Double Head R-CNN](https://arxiv.org/abs/1904.06493) 为例来展示如何添加一个新的 Head。

首先，添加一个新的 bbox head 到 `mmrotate/models/roi_heads/bbox_heads/double_bbox_head.py`。

```python

                                      /-> cls
                                      \-> reg
                                      /-> cls
                                      \-> reg

```

```

然后，如有必要，我们需要实现一个新的 RoI Head。我们打算从 `StandardRoIHead` 继承出新的 `DoubleHeadRoIHead`。我们发现 `StandardRoIHead` 已经实现了下述函数。

```python

    """

"""

                      x,

                    x,

```

Double Head 的修改主要在 \_bbox_forward 的逻辑中，且它从 `StandardRoIHead` 中继承了其他逻辑。
在 `mmrotate/models/roi_heads/double_roi_head.py` 中, 我们实现如下的新的 RoI Head:

```python

"""Test without augmentation."""

```

```python

    """

            rois,

bbox_reg_feats = self.bbox_roi_extractor(
            rois,

```

最后，用户需要把这个新模块添加到 `mmrotate/models/bbox_heads/__init__.py` 以及 `mmrotate/models/roi_heads/__init__.py` 中。 这样，注册机制就能找到并加载它们。

另外，用户也可以添加

```python
```

到配置文件中来实现同样的目的。

### 添加新的损失

为了添加一个新的损失函数，用户需要在 `mmrotate/models/losses/my_loss.py` 中实现。

```

### 添加新的损失

```python

                pred,
                target,

weight=None,
```

然后，用户需要把下面的代码加到 `mmrotate/models/losses/__init__.py` 中。

```python

```

或者，你可以添加：

```python
```

因为 MyLoss 是用于回归的，你需要在 Head 中修改 `loss_bbox` 字段：

```python
```

## Visualization Tools

### Dataset Visualization Tool

MMOCR provides a dataset visualization tool `tools/visualizations/browse_datasets.py` to help users troubleshoot possible dataset-related problems. You just need to specify the path to the training config (usually stored in `configs/textdet/dbnet/xxx.py`) or the dataset config (usually stored in `configs/textdet/_base_/datasets/xxx.py`), and the tool will automatically plots the transformed (or original) images and labels.

#### Usage

```bash
```

[-t, --task ${DATASET_TASK}] \
```

| ARGS                | Type                                  | Description                                                                                                                                      |
| config              | str                                   | (required) Path to the config.                                                                                                                   |

| -o, --output-dir    | str                                   | If GUI is not available, specifying an output path to save the visualization results.                                                            |
| -p, --phase         | str                                   | Phase of dataset to visualize. Use "train", "test" or "val" if you just want to visualize the default split. It's also possible to be a dataset variable name, which might be useful when a dataset split has multiple variants in the config. |
| -m, --mode          | `original`, `transformed`, `pipeline` | Display mode: display original pictures or transformed pictures or comparison pictures.`original` only visualizes the original dataset & annotations; `transformed` shows the resulting images processed through all the transforms; `pipeline` shows all the intermediate images. Defaults to "transformed". |

| -t, --task          | `auto`, `textdet`, `textrecog`        | Specify the task type of the dataset. If `auto`, the task type will be inferred from the config. If the script is unable to infer the task type, you need to specify it manually. Defaults to `auto`. |
| -n, --show-number   | int                                   | The number of samples to visualized. If not specified, display all images in the dataset.                                                        |
| -i, --show-interval | float                                 | Interval of visualization (s), defaults to 2.                                                                                                    |
| --cfg-options       | float                                 | Override configs.[Example](./config.md#command-line-modification)                                                                                |

#### Examples

The following example demonstrates how to use the tool to visualize the training data used by the "DBNet_R50_icdar2015" model.

```Bash
# Example: Visualizing the training data used by dbnet_r50dcn_v2_fpnc_1200e_icadr2015 model
```

By default, the visualization mode is "transformed", and you will see the images & annotations being transformed by the pipeline:

</center>

If you just want to visualize the original dataset, simply set the mode to "original":

</center>

```Bash
```

Or, to visualize the entire pipeline:

```Bash
```

In addition, users can also visualize the original images and their corresponding labels of the dataset by specifying the path to the dataset config file, for example:

```Bash

```Bash
```

Some datasets might have multiple variants. For example, the test split of `icdar2015` textrecog dataset has two variants, which the [base dataset config](/configs/textrecog/_base_/datasets/icdar2015.py) defines as follows:

```python
    # ...
    )

    # ...
)
```

In this case, you can specify the variant name to visualize the corresponding dataset:

```Bash
```

```Bash
```

Based on this tool, users can easily verify if the annotation of a custom dataset is correct.

### Hyper-parameter Scheduler Visualization

#### Introduce the scheduler visualization tool

```bash
```

[--title ${TITLE}] \
```

```{note}
```

#### How to plot the learning rate curve without training

You can use the following command to plot the step learning rate schedule used in the config `configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py`:

```bash
```

<div align=center><img src="https://user-images.githubusercontent.com/43344034/232757392-b29b8e3a-77af-451c-8786-d3b4259ab388.png" style=" width: auto; height: 40%; "></div>

## Analysis Tools

### Offline Evaluation Tool

For saved prediction results, we provide an offline evaluation script `tools/analysis_tools/offline_eval.py`. The following example demonstrates how to use this tool to evaluate the output of the "PSENet" model offline.

```Bash
# When running the test script for the first time, you can save the output of the model by specifying the --save-preds parameter
# Example: Testing on PSENet

# Then, using the saved outputs for offline evaluation
# Example: Offline evaluation of saved PSENet results
```

```

`-save-preds` saves the output to `work_dir/CONFIG_NAME/MODEL_NAME_predictions.pkl` by default

In addition, based on this tool, users can also convert predictions obtained from other libraries into MMOCR-supported formats, then use MMOCR's built-in metrics to evaluate them.

| ARGS          | Type  | Description                                                       |
| ------------- | ----- | ----------------------------------------------------------------- |
| config        | str   | (required) Path to the config.                                    |
| pkl_results   | str   | (required) The saved predictions.                                 |
| --cfg-options | float | Override configs.[Example](./config.md#command-line-modification) |

### Calculate FLOPs and the Number of Parameters

We provide a method to calculate the FLOPs and the number of parameters, first we install the dependencies using the following command.

```shell
```

The usage of the script to calculate FLOPs and the number of parameters is as follows.

```shell
```

| ARGS    | Type | Description                                                                               |
| ------- | ---- | ----------------------------------------------------------------------------------------- |
| config  | str  | (required) Path to the config.                                                            |
| --shape | int  | Image size to use when calculating FLOPs, such as `--shape 320 320`. Default is `640 640` |

For example, you can run the following command to get FLOPs and the number of parameters of `dbnet_resnet18_fpnc_100k_synthtext.py`:

```shell
```

The output is as follows:

```shell

```

# Double Heads

> [Rethinking Classification and Localization for Object Detection](https://arxiv.org/abs/1904.06493)

Two head structures (i.e. fully connected head and convolution head) have been widely used in R-CNN based detectors for classification and localization tasks. However, there is a lack of understanding of how does these two head structures work for these two tasks. To address this issue, we perform a thorough analysis and find an interesting fact that the two head structures have opposite preferences towards the two tasks. Specifically, the fully connected head (fc-head) is more suitable for the classification task, while the convolution head (conv-head) is more suitable for the localization task. Furthermore, we examine the output feature maps of both heads and find that fc-head has more spatial sensitivity than conv-head. Thus, fc-head has more capability to distinguish a complete object from part of an object, but is not robust to regress the whole object. Based upon these findings, we propose a Double-Head method, which has a fully connected head focusing on classification and a convolution head for bounding box regression. Without bells and whistles, our method gains +3.5 and +2.8 AP on MS COCO dataset from Feature Pyramid Network (FPN) baselines with ResNet-50 and ResNet-101 backbones, respectively.

<img src="https://user-images.githubusercontent.com/40661020/143879010-e30f654b-f93e-44b2-a186-c251fdca5bda.png"/>
</div>

| Backbone |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                    Config                     |                                                                                                                                                        Download                                                                                                                                                         |

| :------: | :-----: | :-----: | :------: | :------------: | :----: | :-------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| R-50-FPN | pytorch |   1x    |   6.8    |      9.5       |  40.0  | [config](./dh-faster-rcnn_r50_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/double_heads/dh_faster_rcnn_r50_fpn_1x_coco/dh_faster_rcnn_r50_fpn_1x_coco_20200130-586b67df.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/double_heads/dh_faster_rcnn_r50_fpn_1x_coco/dh_faster_rcnn_r50_fpn_1x_coco_20200130_220238.log.json) |

```latex
@article{wu2019rethinking,
    title={Rethinking Classification and Localization for Object Detection},

```latex
    author={Yue Wu and Yinpeng Chen and Lu Yuan and Zicheng Liu and Lijuan Wang and Hongzhi Li and Yun Fu},
    eprint={1904.06493},
}
```

# Second: Sparsely embedded convolutional detection

> [SECOND: Sparsely Embedded Convolutional Detection](https://www.mdpi.com/1424-8220/18/10/3337)

LiDAR-based or RGB-D-based object detection is used in numerous applications, ranging from autonomous driving to robot vision. Voxel-based 3D convolutional networks have been used for some time to enhance the retention of information when processing point cloud LiDAR data. However, problems remain, including a slow inference speed and low orientation estimation performance. We therefore investigate an improved sparse convolution method for such networks, which significantly increases the speed of both training and inference. We also introduce a new form of angle loss regression to improve the orientation estimation performance and a new data augmentation approach that can enhance the convergence speed and performance. The proposed network produces state-of-the-art results on the KITTI 3D object detection benchmarks while maintaining a fast inference speed.

<img src="https://user-images.githubusercontent.com/79644370/143889364-10be11c3-838e-4fc9-9613-184f0cd08907.png" width="800"/>
</div>

We implement SECOND and provide the results and checkpoints on KITTI dataset.

### KITTI

|                              Backbone                               |  Class  |  Lr schd   | Mem (GB) | Inf time (fps) |  mAP  |                                                                                                                                                                                             Download                                                                                                                                                                                             |

|        [SECFPN](./second_hv_secfpn_8xb6-80e_kitti-3d-car.py)        |   Car   | cyclic 80e |   5.4    |                | 78.2  |                       [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/second/second_hv_secfpn_8xb6-80e_kitti-3d-car/second_hv_secfpn_8xb6-80e_kitti-3d-car-75d9305e.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/second/second_hv_secfpn_8xb6-80e_kitti-3d-car/second_hv_secfpn_8xb6-80e_kitti-3d-car-20230420_191750.log)                        |

|  [SECFPN (FP16)](./second_hv_secfpn_8xb6-amp-80e_kitti-3d-car.py)   |   Car   | cyclic 80e |   2.9    |                | 78.72 |       [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/fp16/hv_second_secfpn_fp16_6x8_80e_kitti-3d-car/hv_second_secfpn_fp16_6x8_80e_kitti-3d-car_20200924_211301-1f5ad833.pth)\| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/fp16/hv_second_secfpn_fp16_6x8_80e_kitti-3d-car/hv_second_secfpn_fp16_6x8_80e_kitti-3d-car_20200924_211301.log.json)        |

|      [SECFPN](./second_hv_secfpn_8xb6-80e_kitti-3d-3class.py)       | 3 Class | cyclic 80e |   5.4    |                | 65.3  |                 [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/second/second_hv_secfpn_8xb6-80e_kitti-3d-3class/second_hv_secfpn_8xb6-80e_kitti-3d-3class-b086d0a3.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/second/second_hv_secfpn_8xb6-80e_kitti-3d-3class/second_hv_secfpn_8xb6-80e_kitti-3d-3class-20230420_221130.log)                  |

| [SECFPN (FP16)](./second_hv_secfpn_8xb6-amp-80e_kitti-3d-3class.py) | 3 Class | cyclic 80e |   2.9    |                | 67.4  | [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/fp16/hv_second_secfpn_fp16_6x8_80e_kitti-3d-3class/hv_second_secfpn_fp16_6x8_80e_kitti-3d-3class_20200925_110059-05f67bdf.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/fp16/hv_second_secfpn_fp16_6x8_80e_kitti-3d-3class/hv_second_secfpn_fp16_6x8_80e_kitti-3d-3class_20200925_110059.log.json) |

### Waymo

|                              Backbone                              | Load Interval |  Class  | Lr schd | Mem (GB) | Inf time (fps) | mAP@L1 | mAPH@L1 | mAP@L2 | **mAPH@L2** |                                                                                           Download                                                                                            |

| :----------------------------------------------------------------: | :-----------: | :-----: | :-----: | :------: | :------------: | :----: | :-----: | :----: | :---------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| [SECFPN](./second_hv_secfpn_sbn-all_16xb2-2x_waymoD5-3d-3class.py) |       5       | 3 Class |   2x    |   8.12   |                |  65.3  |  61.7   |  58.9  |    55.7     | [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/second/hv_second_secfpn_sbn_4x8_2x_waymoD5-3d-3class/hv_second_secfpn_sbn_4x8_2x_waymoD5-3d-3class_20201115_112448.log.json) |

|                            above @ Car                             |               |         |   2x    |   8.12   |                |  67.1  |  66.6   |  58.7  |    58.2     |                                                                                                                                                                                               |
|                         above @ Pedestrian                         |               |         |   2x    |   8.12   |                |  68.1  |  59.1   |  59.5  |    51.5     |                                                                                                                                                                                               |

|                          above @ Cyclist                           |               |         |   2x    |   8.12   |                |  60.7  |  59.5   |  58.4  |    57.3     |                                                                                                                                                                                               |

Note:

- See more details about metrics and data split on Waymo [HERE](https://github.com/open-mmlab/mmdetection3d/tree/main/configs/pointpillars). For implementation details, we basically follow the original settings. All of these results are achieved without bells-and-whistles, e.g. ensemble, multi-scale training and test augmentation.

```latex
@article{yan2018second,
  title={Second: Sparsely embedded convolutional detection},
  author={Yan, Yan and Mao, Yuxing and Li, Bo},
  journal={Sensors},

```latex
}
```

# DSVT: Dynamic Sparse Voxel Transformer with Rotated Sets

> [DSVT: Dynamic Sparse Voxel Transformer with Rotated Sets](https://arxiv.org/abs/2301.06051)

Designing an efficient yet deployment-friendly 3D backbone to handle sparse point clouds is a fundamental problem
in 3D perception. Compared with the customized sparse
convolution, the attention mechanism in Transformers is
more appropriate for flexibly modeling long-range relationships and is easier to be deployed in real-world applications.
However, due to the sparse characteristics of point clouds,
it is non-trivial to apply a standard transformer on sparse
points. In this paper, we present Dynamic Sparse Voxel
Transformer (DSVT), a single-stride window-based voxel
Transformer backbone for outdoor 3D perception. In order
to efficiently process sparse points in parallel, we propose
Dynamic Sparse Window Attention, which partitions a series
of local regions in each window according to its sparsity

and then computes the features of all regions in a fully parallel manner. To allow the cross-set connection, we design
a rotated set partitioning strategy that alternates between
two partitioning configurations in consecutive self-attention
layers. To support effective downsampling and better encode geometric information, we also propose an attentionstyle 3D pooling module on sparse points, which is powerful
and deployment-friendly without utilizing any customized
CUDA operations. Our model achieves state-of-the-art performance with a broad range of 3D perception tasks. More
importantly, DSVT can be easily deployed by TensorRT with
real-time inference speed (27Hz). Code will be available at
https://github.com/Haiyang-W/DSVT.

<img src="https://github-production-user-asset-6210df.s3.amazonaws.com/34888372/245692705-e61be20c-2a7d-4ab9-85e3-b36f662c1bdf.png" width="800"/>
</div>

We implement DSVT and provide the results on Waymo dataset.

## Usage

```shell
pip install torch_scatter==2.0.9
python projects/DSVT/setup.py develop # compile `ingroup_inds_op` cuda operation
```

```bash

```

```bash
python tools/test.py projects/DSVT/configs/dsvt_voxel032_res-second_secfpn_8xb1-cyclic-12e_waymoD5-3d-3class.py ${CHECKPOINT_PATH}
```

The support of training DSVT is on the way.

### Waymo

|                                     Middle Encoder                                     |                                          Backbone                                           | Load Interval | Voxel type (voxel size) | Multi-Class NMS | Multi-frames | Mem (GB) | Inf time (fps) | mAP@L1 | mAPH@L1 | mAP@L2 | **mAPH@L2** | Download |

| :------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------: | :-----------: | :---------------------: | :-------------: | :----------: | :------: | :------------: | :----: | :-----: | :----: | :---------: | :------: |
| [DSVT](./configs/dsvt_voxel032_res-second_secfpn_8xb1-cyclic-12e_waymoD5-3d-3class.py) | [ResSECOND](./configs/dsvt_voxel032_res-second_secfpn_8xb1-cyclic-12e_waymoD5-3d-3class.py) |       5       |      voxel (0.32)       |        ✓        |      ×       |          |                |  75.2  |  72.2   |  68.9  |    66.1     |          |

**Note** that `ResSECOND` denotes the base block in SECOND has residual layers.

```latex
@inproceedings{wang2023dsvt,
    title={DSVT: Dynamic Sparse Voxel Transformer with Rotated Sets},
    author={Haiyang Wang, Chen Shi, Shaoshuai Shi, Meng Lei, Sen Wang, Di He, Bernt Schiele and Liwei Wang},

title={DSVT: Dynamic Sparse Voxel Transformer with Rotated Sets},
}
```

# 如何量化模型

## 为什么要量化

相对于 fp32 模型，定点模型有诸多优点：

- 体积更小，8-bit 模型可降低 75% 文件大小
- 由于模型变小，Cache 命中率提升，速度更快
- 芯片往往有对应的定点加速指令，这些指令更快、能耗更低（常见 CPU 上 int8 大约只需要 10% 能量）

安装包体积、发热都是移动端评价 APP 的关键指标；而在服务端，“加速”意味着可以维持相同 QPS、增大模型换取精度提升。

## mmdeploy 离线量化方案

以 ncnn backend 为例，完整的工作流如下：

</div>

mmdeploy 基于静态图（onnx）生成推理框架所需的量化表，再用后端工具把浮点模型转为定点。

目前 mmdeploy 支持 ncnn PTQ。

## 模型怎么转定点

[mmdeploy 安装](../01-how-to-build/build_from_source.md)完成后，加载 ppq 并安装

```bash
cd ppq
```

回到 mmdeploy, 使用 `tools/deploy.py --quant` 选项开启量化。

```bash
export MODEL_CONFIG=/path/to/mmpretrain/configs/resnet/resnet18_8xb16_cifar10.py
export MODEL_PATH=https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_b16x8_cifar10_20210528-bd6371c8.pth

# 找一些 imagenet 样例图

# 量化模型
...
```

参数说明

|       参数        |                       含义                       |
| :---------------: | :----------------------------------------------: |
|      --quant      |            是否开启量化，默认为 False            |
| --quant-image-dir | 校准数据集，默认使用 MODEL_CONFIG 中的**验证集** |

## 自建校准数据集

校准集是用来计算量化层参数的，某些 DFQ（Data Free Quantization）方法甚至不需要校准集

- 新建文件夹，直接放入图片即可（不需要目录结构、不要负例、没有命名要求）
- 图片需为真实业务场景中的数据，相差过远会导致精度下降
- 不能直接拿测试集做量化，否则是过拟合
  | 类型 | 训练集 | 验证集 | 测试集   | 校准集 |
  | ---- | ------ | ------ | -------- | ------ |

  | 用法 | QAT    | PTQ    | 测试精度 | PTQ    |

**强烈建议**量化结束后，[按此文档](profile_model.md) 验证模型精度。[这里](../03-benchmark/quantization.md) 是一些量化模型测试结果。

}
```

|                                    Config                                    | Input Size | NME<sup><br>(LaPa) | FLOPS<sup><br>(G) |                                    Download                                     |

| :--------------------------------------------------------------------------: | :--------: | :----------------: | :---------------: | :-----------------------------------------------------------------------------: |
| [RTMPose-t\*](./rtmpose/face_2d_keypoint/rtmpose-t_8xb256-120e_lapa-256x256.py) |  256x256   |        1.67        |       0.652       | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-t_simcc-face6_pt-in1k_120e-256x256-df79d9a5_20230529.pth) |
| [RTMPose-s\*](./rtmpose/face_2d_keypoint/rtmpose-m_8xb256-120e_lapa-256x256.py) |  256x256   |        1.59        |       1.119       | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_simcc-face6_pt-in1k_120e-256x256-d779fdef_20230529.pth) |

| [RTMPose-m\*](./rtmpose/face_2d_keypoint/rtmpose-m_8xb256-120e_lapa-256x256.py) |  256x256   |        1.44        |       2.852       | [Model](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-face6_pt-in1k_120e-256x256-72a37400_20230529.pth) |

# VIG

> [Vision GNN: An Image is Worth Graph of Nodes](https://arxiv.org/abs/2206.00272)

Network architecture plays a key role in the deep learning-based computer vision system. The widely-used convolutional neural network and transformer treat the image as a grid or sequence structure, which is not flexible to capture irregular and complex objects. In this paper, we propose to represent the image as a graph structure and introduce a new Vision GNN (ViG) architecture to extract graph-level feature for visual tasks. We first split the image to a number of patches which are viewed as nodes, and construct a graph by connecting the nearest neighbors. Based on the graph representation of images, we build our ViG model to transform and exchange information among all the nodes. ViG consists of two basic modules: Grapher module with graph convolution for aggregating and updating graph information, and FFN module with two linear layers for node feature transformation. Both isotropic and pyramid architectures of ViG are built with different model sizes. Extensive experiments on image recognition and object detection tasks demonstrate the superiority of our ViG architecture. We hope this pioneering study of GNN on general visual tasks will provide useful inspiration and experience for future research.

<img src="https://user-images.githubusercontent.com/26739999/212789461-f085e4da-9ce9-435f-93c0-e1b84d10b79f.png" width="50%"/>
</div>

```python

predict = inference_model('vig-tiny_3rdparty_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('vig-tiny_3rdparty_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/vig/vig-tiny_8xb128_in1k.py https://download.openmmlab.com/mmclassification/v0/vig/vig-tiny_3rdparty_in1k_20230117-6414c684.pth
```

| `vig-tiny_3rdparty_in1k`\*    | From scratch |    7.18    |   1.31    |   74.40   |   92.34   |  [config](vig-tiny_8xb128_in1k.py)   | [model](https://download.openmmlab.com/mmclassification/v0/vig/vig-tiny_3rdparty_in1k_20230117-6414c684.pth) |
| `vig-small_3rdparty_in1k`\*   | From scratch |   22.75    |   4.54    |   80.61   |   95.28   |  [config](vig-small_8xb128_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/vig/vig-small_3rdparty_in1k_20230117-5338bf3b.pth) |
| `vig-base_3rdparty_in1k`\*    | From scratch |   20.68    |   17.68   |   82.62   |   96.04   |  [config](vig-base_8xb128_in1k.py)   | [model](https://download.openmmlab.com/mmclassification/v0/vig/vig-base_3rdparty_in1k_20230117-92f6f12f.pth) |
| `pvig-tiny_3rdparty_in1k`\*   | From scratch |    9.46    |   1.71    |   78.38   |   94.38   |  [config](pvig-tiny_8xb128_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/vig/pvig-tiny_3rdparty_in1k_20230117-eb77347d.pth) |

| `pvig-small_3rdparty_in1k`\*  | From scratch |   29.02    |   4.57    |   82.00   |   95.97   | [config](pvig-small_8xb128_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/vig/pvig-small_3rdparty_in1k_20230117-9433dc96.pth) |
| `pvig-medium_3rdparty_in1k`\* | From scratch |   51.68    |   8.89    |   83.12   |   96.35   | [config](pvig-medium_8xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/vig/pvig-medium_3rdparty_in1k_20230117-21057a6d.pth) |
| `pvig-base_3rdparty_in1k`\*   | From scratch |   95.21    |   16.86   |   83.59   |   96.52   |  [config](pvig-base_8xb128_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/vig/pvig-base_3rdparty_in1k_20230117-dbab3c85.pth) |

*Models with * are converted from the [official repo](https://github.com/huawei-noah/Efficient-AI-Backbones/tree/master/vig_pytorch). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@inproceedings{han2022vig,
  title={Vision GNN: An Image is Worth Graph of Nodes},
  author={Kai Han and Yunhe Wang and Jianyuan Guo and Yehui Tang and Enhua Wu},
}
```

MMDeploy provides useful tools for deploying OpenMMLab models to various platforms and devices.

With the help of them, you can not only do model deployment using our pre-defined pipelines but also customize your own deployment pipeline.

In MMDeploy, the deployment pipeline can be illustrated by a sequential modules, i.e., Model Converter, MMDeploy Model and Inference SDK.

![deploy-pipeline](https://user-images.githubusercontent.com/4560679/172306700-31b4c922-2f04-42ed-a1d6-c360f2f3048c.png)

### Model Converter

Model Converter aims at converting training models from OpenMMLab into backend models that can be run on target devices.
It is able to transform PyTorch model into IR model, i.e., ONNX, TorchScript, as well as convert IR model to backend model. By combining them together, we can achieve one-click **end-to-end** model deployment.

### MMDeploy Model

MMDeploy Model is the result package exported by Model Converter.

Beside the backend models, it also includes the model meta info, which will be used by Inference SDK.

### Inference SDK

Inference SDK is developed by C/C++, wrapping the preprocessing, model forward and postprocessing modules in model inference.
It supports FFI such as C, C++, Python, C#, Java and so on.

In order to do an end-to-end model deployment, MMDeploy requires Python 3.6+ and PyTorch 1.8+.

```shell
conda create --name mmdeploy python=3.8 -y
```

```shell

```

```shell
conda install pytorch=={pytorch_version} torchvision=={torchvision_version} cudatoolkit={cudatoolkit_version} -c pytorch -c conda-forge
```

```shell
conda install pytorch=={pytorch_version} torchvision=={torchvision_version} cpuonly -c pytorch
```

```{note}
On GPU platform, please ensure that {cudatoolkit_version} matches your host CUDA toolkit version. Otherwise, it probably brings in conflicts when deploying model with TensorRT.
```

We recommend that users follow our best practices installing MMDeploy.

**Step 0.** Install [MMCV](https://github.com/open-mmlab/mmcv).

```shell
```

**Step 1.** Install MMDeploy and inference engine

```shell
```

We recommend using MMDeploy precompiled package as our best practice. Currently, we support model converter and sdk inference pypi package, and the sdk c/cpp library is provided [here](https://github.com/open-mmlab/mmdeploy/releases). You can download them according to your target platform and device.

The supported platform and device matrix is presented as following:

<table>
<thead>
  <tr>
    <th>OS-Arch</th>
    <th>Device</th>
    <th>ONNX Runtime</th>
    <th>TensorRT</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td rowspan="2">Linux-x86_64</td>
    <td>CPU</td>
    <td>Y</td>
  </tr>
  <tr>
    <td>CUDA</td>
  </tr>
  <tr>
    <td rowspan="2">Windows-x86_64</td>
  </tr>
  <tr>
  </tr>
</tbody>
</table>

</tr>
  <tr>
  </tr>
  <tr>
  </tr>
</tbody>
</table>

**Note: if MMDeploy prebuilt package doesn't meet your target platforms or devices, please [build MMDeploy from source](01-how-to-build/build_from_source.md)**

Take the latest precompiled package as example, you can install it as follows:

<summary><b>Linux-x86_64</b></summary>

```shell
# 1. install MMDeploy model converter
pip install mmdeploy==1.2.0

# 2. install MMDeploy sdk inference
# you can install one to install according whether you need gpu inference
# 2.1 support onnxruntime
pip install mmdeploy-runtime==1.2.0
# 2.2 support onnxruntime-gpu, tensorrt
pip install mmdeploy-runtime-gpu==1.2.0

# 3. install inference engine
# 3.1 install TensorRT
# !!! If you want to convert a tensorrt model or inference with tensorrt,

# download TensorRT-8.2.3.0 CUDA 11.x tar package from NVIDIA, and extract it to the current directory
pip install TensorRT-8.2.3.0/python/tensorrt-8.2.3.0-cp38-none-linux_x86_64.whl
export LD_LIBRARY_PATH=${TENSORRT_DIR}/lib:$LD_LIBRARY_PATH
# !!! Moreover, download cuDNN 8.2.1 CUDA 11.x tar package from NVIDIA, and extract it to the current directory

# 3.2 install ONNX Runtime
# 3.2.1 onnxruntime

# 3.2.2 onnxruntime-gpu
```

<summary><b>Windows-x86_64</b></summary>

Please learn its prebuilt package from [this](02-how-to-run/prebuilt_package_windows.md) guide.

## Convert Model

After the installation, you can enjoy the model deployment journey starting from converting PyTorch model to backend model by running `tools/deploy.py`.

Based on the above settings, we provide an example to convert the Faster R-CNN in [MMDetection](https://github.com/open-mmlab/mmdetection) to TensorRT as below:

```shell
# clone mmdeploy to get the deployment config. `--recursive` is not necessary
git clone -b main https://github.com/open-mmlab/mmdeploy.git

# clone mmdetection repo. We have to use the config file to build PyTorch nn module
cd ..

# download Faster R-CNN checkpoint
wget -P checkpoints https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth

    mmdeploy/configs/mmdet/detection/detection_tensorrt_dynamic-320x320-1344x1344.py \
    mmdetection/configs/faster_rcnn/faster-rcnn_r50_fpn_1x_coco.py \
    checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
    mmdetection/demo/demo.jpg \
    --work-dir mmdeploy_model/faster-rcnn \
```

The converted model and its meta info will be found in the path specified by `--work-dir`.
And they make up of MMDeploy Model that can be fed to MMDeploy SDK to do model inference.

For more details about model conversion, you can read [how_to_convert_model](02-how-to-run/convert_model.md). If you want to customize the conversion pipeline, you can edit the config file by following [this](02-how-to-run/write_config.md) tutorial.

```{tip}
You can convert the above model to onnx model and perform ONNX Runtime inference

```{tip}
just by changing 'detection_tensorrt_dynamic-320x320-1344x1344.py' to 'detection_onnxruntime_dynamic.py' and making '--device' as 'cpu'.
```

## Inference Model

After model conversion, we can perform inference not only by Model Converter but also by Inference SDK.

### Inference by Model Converter

Model Converter provides a unified API named as `inference_model` to do the job, making all inference backends API transparent to users.
Take the previous converted Faster R-CNN tensorrt model for example,

```python
  model_cfg='mmdetection/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py',
  deploy_cfg='mmdeploy/configs/mmdet/detection/detection_tensorrt_dynamic-320x320-1344x1344.py',
  backend_files=['mmdeploy_model/faster-rcnn/end2end.engine'],
  img='mmdetection/demo/demo.jpg',
```

```{note}

backend_files=['mmdeploy_model/faster-rcnn/end2end.engine'],
```

```{note}
'backend_files' in this API refers to backend engine file path, which MUST be put in a list, since some inference engines like OpenVINO and ncnn separate the network structure and its weights into two files.
```

### Inference by SDK

You can directly run MMDeploy demo programs in the precompiled package to get inference results.

```shell
wget https://github.com/open-mmlab/mmdeploy/releases/download/v1.2.0/mmdeploy-1.2.0-linux-x86_64-cuda11.3.tar.gz
tar xf mmdeploy-1.2.0-linux-x86_64-cuda11.3
cd mmdeploy-1.2.0-linux-x86_64-cuda11.3
# run python demo
python example/python/object_detection.py cuda ../mmdeploy_model/faster-rcnn ../mmdetection/demo/demo.jpg
# run C/C++ demo
# build the demo according to the README.md in the folder.
./bin/object_detection cuda ../mmdeploy_model/faster-rcnn ../mmdetection/demo/demo.jpg
```

```{note}

```

```{note}
In the above command, the input model is SDK Model path. It is NOT engine file path but actually the path passed to --work-dir. It not only includes engine files but also meta information like 'deploy.json' and 'pipeline.json'.
```

In the next section, we will provide examples of deploying the converted Faster R-CNN model talked above with SDK different FFI (Foreign Function Interface).

```python
import cv2

img = cv2.imread('mmdetection/demo/demo.jpg')

detector = Detector(model_path='mmdeploy_models/faster-rcnn', device_name='cuda', device_id=0)

      continue

```

You can find more examples from [here](https://github.com/open-mmlab/mmdeploy/tree/main/demo/python).

Using SDK C++ API should follow next pattern,

![image](https://user-images.githubusercontent.com/4560679/182554739-7fff57fc-5c84-44ed-b139-4749fae27404.png)

Now let's apply this procedure on the above Faster R-CNN model.

```C++

  std::string model_path = "mmdeploy_model/faster-rcnn";

  std::string image_path = "mmdetection/demo/demo.jpg";

    if (dets[i].score < 0.3) {
      continue;
    }

if (dets[i].score < 0.3) {
      continue;
    }
  }
  return 0;
}
```

When you build this example, try to add MMDeploy package in your CMake project as following. Then pass `-DMMDeploy_DIR` to cmake, which indicates the path where `MMDeployConfig.cmake` locates. You can find it in the prebuilt package.

```Makefile
```

For more SDK C++ API usages, please read these [samples](https://github.com/open-mmlab/mmdeploy/tree/main/demo/csrc/cpp).

```

For the rest C, C# and Java API usages, please read [C demos](https://github.com/open-mmlab/mmdeploy/tree/main/demo/csrc/c), [C# demos](https://github.com/open-mmlab/mmdeploy/tree/main/demo/csharp) and [Java demos](https://github.com/open-mmlab/mmdeploy/tree/main/demo/java) respectively.
We'll talk about them more in our next release.

#### Accelerate preprocessing（Experimental）

If you want to fuse preprocess for acceleration，please refer to this [doc](./02-how-to-run/fuse_transform.md)

## Evaluate Model

You can test the performance of deployed model using `tool/test.py`. For example,

```shell
python ${MMDEPLOY_DIR}/tools/test.py \
    ${MMDEPLOY_DIR}/configs/detection/detection_tensorrt_dynamic-320x320-1344x1344.py \
    ${MMDET_DIR}/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py \

    --model ${BACKEND_MODEL_FILES} \
    --metrics ${METRICS} \
```

```{note}
Regarding the --model option, it represents the converted engine files path when using Model Converter to do performance test. But when you try to test the metrics by Inference SDK, this option refers to the directory path of MMDeploy Model.
```

You can read [how to evaluate a model](02-how-to-run/profile_model.md) for more details.

# 日志分析工具

## 日志分析

### 日志分析工具介绍

`tools/analysis_tools/analyze_logs.py` 脚本绘制指定键值的变化曲线。

```shell
```

**所有参数的说明：**

- `json_logs` : 模型配置文件的路径（可同时传入多个，使用空格分开）。
- `--keys` : 分析日志的关键字段，数量为 `len(${JSON_LOGS}) * len(${KEYS})` 默认为 ‘loss’。
- `--title` : 分析日志的图片名称，默认使用配置文件名， 默认为空。
- `--legend` : 图例的名称，其数目必须与相等`len(${JSON_LOGS}) * len(${KEYS})`。     默认使用 `"${JSON_LOG}-${KEYS}"`.
- `--backend` : matplotlib 的绘图后端，默认由 matplotlib 自动选择。
- `--style` : 绘图配色风格，默认为 `whitegrid`。
- `--out` : 保存分析图片的路径，如不指定则不保存。
- `--window-size`: 可视化窗口大小，如果没有指定，默认为  `'12*7'`。如果需要指定，需按照格式 `'W*H'`。

```{note}

```{note}
```

### 如何或绘制损失/精度曲线

我们将给出一些示例，来展示如何使用 `tools/analysis_tools/analyze_logs.py`脚本绘制精度曲线的损失曲线

#### 绘制某日志文件对应的损失曲线图

```shell
```

#### 绘制某日志文件对应的 top-1 和 top-5 准确率曲线图，并将曲线图导出为 results.jpg 文件。

```shell
```

#### 在同一图像内绘制两份日志文件对应的 top-1 准确率曲线图。

```shell
```

### 如何统计训练时间

`tools/analysis_tools/analyze_logs.py` 也可以根据日志文件统计训练耗时。

```shell

```

```shell
```

- `json_logs`：模型配置文件的路径（可同时传入多个，使用空格分开）。
- `--include-outliers`：如果指定，将不会排除每个轮次中第一个时间记录（有时第一轮迭代会耗时较长）。

**示例：**

```shell
```

预计输出结果如下所示：

```text
```

## 结果分析

利用 `tools/test.py` 的`--out`，我们可以将所有的样本的推理结果保存到输出文件中。利用这一文件，我们可以进行进一步的分析。

### 如何进行离线度量评估

我们提供了 `tools/analysis_tools/eval_metric.py` 脚本，使用户能够根据预测文件评估模型。

```shell
```

**所有参数说明**：

```shell
```

- `result`：`tools/test.py` 输出的结果文件。
- `--metric`：用于评估结果的指标，请至少指定一个指标，并且你可以通过指定多个 `--metric` 来同时计算多个指标。

请参考[评估文档](mmpretrain.evaluation)选择可用的评估指标和对应的选项。

```{note}
在 `tools/test.py` 中，我们支持使用 `--out-item` 选项来选择保存何种结果至输出文件。
请确保没有额外指定 `--out-item`，或指定了 `--out-item=pred`。
```

**示例**:

```shell
# 获取结果文件

# 计算 top-1 和 top-5 准确率

# 计算总体准确率，各个类别上的精确度、召回率、F1-score
```

```

### 如何绘制测试结果的混淆矩阵

我们提供 `tools/analysis_tools/confusion_matrix.py`，帮助用户能够从测试输出文件中绘制混淆矩阵。

```shell
      [--show] \
      [--cmap] \
```

- `config`：配置文件的路径。
- `result`：`tools/test.py`的输出结果文件，或是模型权重文件。
- `--out`：将混淆矩阵保存到指定路径下的 pickle 文件中。
- `--show`：是否可视化混淆矩阵图。
- `--show-path`：将可视化混淆矩阵图保存到指定路径下。
- `--include-values`：是否在可视化混淆矩阵图中显示具体值。
- `--cmap`：用以可视化混淆矩阵的颜色配置。

```{note}
```

```{note}
```

```shell
# 获取结果文件

# 将混淆矩阵计算结果保存至 cm.pkl 中

# 可视化混淆矩阵图，并在图形窗口显示
```

### 如何将预测结果可视化

我们可以使用脚本 `tools/analysis_tools/analyze_results.py` 来保存预测成功或失败时得分最高的图像。

```shell
```

**所有参数说明：**:

${RESULT} \
```

- `result`：`tools/test.py`的输出结果文件。
- `--out_dir`：保存结果分析的文件夹路径。
- `--topk`：分别保存多少张预测成功/失败的图像。如果不指定，默认为 `20`。
- `--rescale-factor`：图像的缩放系数，如果样本图像过大或过小时可以使用（过小的图像可能导致结果标签非常模糊）。

```{note}
```

**示例**:

```shell
# 获取预测结果文件

# 保存预测成功/失败的图像中，得分最高的前 10 张，并在可视化时将输出图像放大 10 倍。
       --topk 10 \

       --topk 10 \
```

# \[WIP\] Useful Tools

### Plot training logs

`tools/analyze_logs.py` plots loss/mIoU curves given a training log file. `pip install seaborn` first to install the dependency.

```shell
python tools/analysis_tools/analyze_logs.py xxx.json [--keys ${KEYS}] [--legend ${LEGEND}] [--backend ${BACKEND}] [--style ${STYLE}] [--out ${OUT_FILE}]
```

Examples:

- Plot the mIoU, mAcc, aAcc metrics.

  ```shell
  python tools/analysis_tools/analyze_logs.py log.json --keys mIoU mAcc aAcc --legend mIoU mAcc aAcc
  ```

- Plot loss metric.

  ```shell
  python tools/analysis_tools/analyze_logs.py log.json --keys loss --legend loss
  ```

### Confusion Matrix (experimental)

In order to generate and plot a `nxn` confusion matrix where `n` is the number of classes, you can follow the steps:

#### 1.Generate a prediction result in pkl format using `test.py`

```shell

```shell
python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${PATH_TO_RESULT_FILE}]
```

Example:

```shell
configs/fcn/fcn_r50-d8_4xb2-40k_cityscapes-512x1024.py \
--out result/pred_result.pkl
```

#### 2. Use `confusion_matrix.py` to generate and plot a confusion matrix

```shell
python tools/confusion_matrix.py ${CONFIG_FILE} ${PATH_TO_RESULT_FILE} ${SAVE_DIR} --show
```

- `config`: Path to the test config file.
- `prediction_path`: Path to the prediction .pkl result.
- `save_dir`: Directory where confusion matrix will be saved.
- `--show`: Enable result visualize.
- `--color-theme`: Theme of the matrix color map.

- `--cfg_options`: Custom options to replace the config file.

Example:

```shell
python tools/confusion_matrix.py \
result/pred_result.pkl \
result/confusion_matrix \
--show
```

### Get the FLOPs and params (experimental)

```shell
```

```none
```

:::{note}

```

```none
```

:::{note}
This tool is still experimental and we do not guarantee that the number is correct. You may well use the result for simple comparisons, but double check it before you adopt it in technical reports or papers.
:::

(1) FLOPs are related to the input shape while parameters are not. The default input shape is (1, 3, 1280, 800).
(2) Some operators are not counted into FLOPs like GN and custom operators.

### Publish a model

(1) convert model weights to CPU tensors, (2) delete the optimizer states and

```shell
python tools/misc/publish_model.py ${INPUT_FILENAME} ${OUTPUT_FILENAME}
```

E.g.,

```shell

```shell
```

E.g.,

```shell
python tools/publish_model.py work_dirs/pspnet/latest.pth psp_r50_512x1024_40k_cityscapes.pth
```

The final output filename will be `psp_r50_512x1024_40k_cityscapes-{hash id}.pth`.

imports.

```shell
python tools/misc/print_config.py \
  --graph \
```

- `--graph` : Determines whether to print the models graph.
- `--cfg-options`: Custom options to replace the config file.

`tools/model_converters/` provide several scripts to convert pretrain models released by other repos to MMSegmentation style.

### ViT Swin MiT Transformer Models

- ViT

  `tools/model_converters/vit2mmseg.py` convert keys in timm pretrained vit models to MMSegmentation style.

  ```shell
  python tools/model_converters/vit2mmseg.py ${SRC} ${DST}
  ```

- Swin

  `tools/model_converters/swin2mmseg.py` convert keys in official pretrained swin models to MMSegmentation style.

  ```shell
  ```

  `tools/model_converters/mit2mmseg.py` convert keys in official pretrained mit models to MMSegmentation style.

  ```shell
  ```

In order to serve an `MMSegmentation` model with [`TorchServe`](https://pytorch.org/serve/), you can follow the steps:

### 1. Convert model from MMSegmentation to TorchServe

```shell

```shell
python tools/torchserve/mmseg2torchserve.py ${CONFIG_FILE} ${CHECKPOINT_FILE} \
```

:::{note}
:::

### 2. Build `mmseg-serve` docker image

```shell
```

### 3. Run `mmseg-serve`

Example:

```shell
--cpus 8 \

Example:

```shell
--cpus 8 \
```

[Read the docs](https://github.com/pytorch/serve/blob/072f5d088cce9bb64b2a18af065886c9b01b317b/docs/rest_api.md) about the Inference (8080), Management (8081) and Metrics (8082) APIs

```shell
```

The response will be a ".png" mask.

You can visualize the output as follows:

```python
```

You should see something similar to:

And you can use `test_torchserve.py` to compare result of torchserve and pytorch, and visualize them.

```

```shell
```

Example:

```shell
fcn
```

# Control Net (2023)

> [Adding Conditional Control to Text-to-Image Diffusion Models](https://arxiv.org/abs/2302.05543)

We present a neural network structure, ControlNet, to control pretrained large diffusion models to support additional input conditions. The ControlNet learns task-specific conditions in an end-to-end way, and the learning is robust even when the training dataset is small (\< 50k). Moreover, training a ControlNet is as fast as fine-tuning a diffusion model, and the model can be trained on a personal devices. Alternatively, if powerful computation clusters are available, the model can scale to large amounts (millions to billions) of data. We report that large diffusion models like Stable Diffusion can be augmented with ControlNets to enable conditional inputs like edge maps, segmentation maps, keypoints, etc. This may enrich the methods to control large diffusion models and further facilitate related applications.

<img src="https://user-images.githubusercontent.com/28132635/230302421-a9107d03-92d3-44b1-91b4-fde4ad2725d4.png">
</div>

We use ControlNet's weights provided by HuggingFace Diffusers. You do not have to download the weights manually. If you use Diffusers wrapper, the weights will be downloaded automatically.

This model has several weights including vae, unet and clip. You should download the weights from [stable-diffusion-1.5](https://huggingface.co/runwayml/stable-diffusion-v1-5) and change the 'pretrained_model_path' in config to the weights dir.

|                      Model                      | Dataset |                                             Download                                             |
| :---------------------------------------------: | :-----: | :----------------------------------------------------------------------------------------------: |

|    [ControlNet-Canny](./controlnet-canny.py)    |    -    |  [model](https://huggingface.co/lllyasviel/ControlNet/blob/main/models/control_sd15_canny.pth)   |
| [ControlNet-Segmentation](./controlnet-seg.py)  |    -    |   [model](https://huggingface.co/lllyasviel/ControlNet/blob/main/models/control_sd15_seg.pth)    |
|     [ControlNet-Pose](./controlnet-pose.py)     |    -    | [model](https://huggingface.co/lllyasviel/ControlNet/blob/main/models/control_sd15_openpose.pth) |
| [ControlNet-Demo](./controlnet-1xb1-fill50k.py) |    -    |                                                -                                                 |

Noted that, [ControlNet-Demo](./controlnet-1xb1-demo_dataset.py) is a demo config to train ControlNet with toy dataset named Fill50K.

Besides above configs, ControlNet have weight with other condition inputs, such as [depth](https://huggingface.co/lllyasviel/ControlNet/blob/main/models/control_sd15_depth.pth), [hed](https://huggingface.co/lllyasviel/ControlNet/blob/main/models/control_sd15_hed.pth), [mlsd](https://huggingface.co/lllyasviel/ControlNet/blob/main/models/control_sd15_mlsd.pth), [normal](https://huggingface.co/lllyasviel/ControlNet/blob/main/models/control_sd15_normal.pth), [scribble](https://huggingface.co/lllyasviel/ControlNet/blob/main/models/control_sd15_scribble.pth). You can simple change the `from_pretrained` field of ControlNet to use these weights. For example:

```python
# Switch from canny....
controlnet=dict(
    type='ControlNetModel',
    from_pretrained='lllyasviel/sd-controlnet-canny')

# To normal....

```python

    from_pretrained='lllyasviel/sd-controlnet-normal')
```

```python
import cv2

cfg = Config.fromfile('configs/controlnet/controlnet-canny.py')
controlnet = MODELS.build(cfg.model).cuda()

prompt = 'Room with blue walls and a yellow ceiling.'
control_url = 'https://user-images.githubusercontent.com/28132635/230288866-99603172-04cb-47b3-8adb-d1aa532d1d2c.jpg'
control_img = mmcv.imread(control_url)
control = cv2.Canny(control_img, 100, 200)
control = control[:, :, None]
control = np.concatenate([control] * 3, axis=2)

control = Image.fromarray(control)

output_dict = controlnet.infer(prompt, control=control)
samples = output_dict['samples']
for idx, sample in enumerate(samples):
    sample.save(f'sample_{idx}.png')
controls = output_dict['controls']
for idx, control in enumerate(controls):
    control.save(f'control_{idx}.png')
```

<thead>
  <tr>
    <td>
  <img src="https://user-images.githubusercontent.com/28132635/230297033-4f5c32df-365c-4cf4-8e4f-1b76a4cbb0b7.png" width="400"/>
  <br/>
  <b>'control_0.png'</b>
    <td>
  <img src="https://user-images.githubusercontent.com/28132635/230298159-a25695f8-fee4-40b2-aec0-01566ab25a97.png" width="400"/>
  <br/>
  <b>'sample_0.png'</b>
    <td>
</thead>
</table>

<img src="https://user-images.githubusercontent.com/28132635/230298159-a25695f8-fee4-40b2-aec0-01566ab25a97.png" width="400"/>
  <br/>
    <td>
</thead>
</table>

If you want to pretrained weights rather than original Stable-Diffusion v1.5, you can refers to the following codes.

```python

cfg = Config.fromfile('configs/controlnet/controlnet-pose.py')
# convert ControlNet's weight from SD-v1.5 to Counterfeit-v2.5
cfg.model.unet.from_pretrained = 'gsdf/Counterfeit-V2.5'
cfg.model.vae.from_pretrained = 'gsdf/Counterfeit-V2.5'
cfg.model.init_cfg['type'] = 'convert_from_unet'

# call init_weights manually to convert weight
controlnet.init_weights()

prompt = 'masterpiece, best quality, sky, black hair, skirt, sailor collar, looking at viewer, short hair, building, bangs, neckerchief, long sleeves, cloudy sky, power lines, shirt, cityscape, pleated skirt, scenery, blunt bangs, city, night, black sailor collar, closed mouth'

control_url = 'https://user-images.githubusercontent.com/28132635/230380893-2eae68af-d610-4f7f-aa68-c2f22c2abf7e.png'
control = Image.fromarray(control_img)
control.save('control.png')

output_dict = controlnet.infer(prompt, control=control, width=512, height=512, guidance_scale=7.5)
```

```

<thead>
  <tr>
    <td>
  <img src="https://user-images.githubusercontent.com/28132635/230385313-92b20696-8bb5-4666-aa93-a8df6b19dee7.png" width="400"/>
  <br/>
    <td>
  <img src="https://user-images.githubusercontent.com/28132635/230385320-129493b6-aa1f-406f-9fd3-b08c8c04bd89.png" width="400"/>
  <br/>
    <td>
</thead>
</table>

## Train your own ControlNet!

You can start training your own ControlNet with the toy dataset [Fill50K](https://huggingface.co/lllyasviel/ControlNet/blob/main/training/fill50k.zip) with the following command:

```bash
bash tools/dist_train.sh configs/controlnet/controlnet-1xb1-demo_dataset 1
```

```bash
```

If you want use gradient accumulation, you can add `accumulative_counts` field to the optimizer's config as follow:

```python
# From...
optim_wrapper = dict(controlnet=dict(optimizer=dict(type='AdamW', lr=1e-5)))
# To...
    controlnet=dict(accumulative_counts=4, optimizer=dict(type='AdamW', lr=1e-5)))
```

We support **[tomesd](https://github.com/dbolya/tomesd)** now! It is developed for stable-diffusion-based models referring to [ToMe](https://github.com/facebookresearch/ToMe), an efficient ViT speed-up tool based on token merging. To work on with **tomesd** in `mmagic`, you just need to add `tomesd_cfg` to `model` in [ControlNet-Canny](./controlnet-canny.py). The only requirement is `torch >= 1.12.1` in order to properly support `torch.Tensor.scatter_reduce()` functionality. Please do check it before running the demo.

```python
    type='ControlStableDiffusion',
    ...
    ...
    init_cfg=dict(type='init_from_unet'))
```

Our codebase for the stable diffusion models builds heavily on [diffusers codebase](https://github.com/huggingface/diffusers) and the model weights are from [stable-diffusion-1.5](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_controlnet.py) and [ControlNet](https://huggingface.co/lllyasviel/ControlNet/tree/main/models).

```bibtex
      title={Adding Conditional Control to Text-to-Image Diffusion Models},
      author={Lvmin Zhang and Maneesh Agrawala},
      eprint={2302.05543},

}
```

# Deep residual learning for image recognition

<details>

```bibtex
}
```

}
```

Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from

considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC
& COCO 2015 competitions1 , where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

<img src="https://user-images.githubusercontent.com/15977946/146538539-d635ee57-200c-40f5-93cf-a34a5b416feb.png">
</div>

</div>

|       Method       |  Backbone  |  Pretrain   | Lr schd | Multi-scale crop | FP16 | Mem (GB) | box AP | mask AP |                                         Config                                          |                                                                                                                                                                                                                                            Download                                                                                                                                                                                                                                             |

|     Mask R-CNN     | ConvNeXt-T | ImageNet-1K |   3x    |       yes        | yes  |   7.3    |  46.2  |  41.7   |            [config](./mask-rcnn_convnext-t-p4-w7_fpn_amp-ms-crop-3x_coco.py)            |                                           [model](https://download.openmmlab.com/mmdetection/v2.0/convnext/mask_rcnn_convnext-t_p4_w7_fpn_fp16_ms-crop_3x_coco/mask_rcnn_convnext-t_p4_w7_fpn_fp16_ms-crop_3x_coco_20220426_154953-050731f4.pth)  \| [log](https://download.openmmlab.com/mmdetection/v2.0/convnext/mask_rcnn_convnext-t_p4_w7_fpn_fp16_ms-crop_3x_coco/mask_rcnn_convnext-t_p4_w7_fpn_fp16_ms-crop_3x_coco_20220426_154953.log.json)                                           |

| Cascade Mask R-CNN | ConvNeXt-T | ImageNet-1K |   3x    |       yes        | yes  |   9.0    |  50.3  |  43.6   | [config](./cascade-mask-rcnn_convnext-t-p4-w7_fpn_4conv1fc-giou_amp-ms-crop-3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/convnext/cascade_mask_rcnn_convnext-t_p4_w7_fpn_giou_4conv1f_fp16_ms-crop_3x_coco/cascade_mask_rcnn_convnext-t_p4_w7_fpn_giou_4conv1f_fp16_ms-crop_3x_coco_20220509_204200-8f07c40b.pth)  \| [log](https://download.openmmlab.com/mmdetection/v2.0/convnext/cascade_mask_rcnn_convnext-t_p4_w7_fpn_giou_4conv1f_fp16_ms-crop_3x_coco/cascade_mask_rcnn_convnext-t_p4_w7_fpn_giou_4conv1f_fp16_ms-crop_3x_coco_20220509_204200.log.json) |

| Cascade Mask R-CNN | ConvNeXt-S | ImageNet-1K |   3x    |       yes        | yes  |   12.3   |  51.8  |  44.8   | [config](./cascade-mask-rcnn_convnext-s-p4-w7_fpn_4conv1fc-giou_amp-ms-crop-3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/convnext/cascade_mask_rcnn_convnext-s_p4_w7_fpn_giou_4conv1f_fp16_ms-crop_3x_coco/cascade_mask_rcnn_convnext-s_p4_w7_fpn_giou_4conv1f_fp16_ms-crop_3x_coco_20220510_201004-3d24f5a4.pth)  \| [log](https://download.openmmlab.com/mmdetection/v2.0/convnext/cascade_mask_rcnn_convnext-s_p4_w7_fpn_giou_4conv1f_fp16_ms-crop_3x_coco/cascade_mask_rcnn_convnext-s_p4_w7_fpn_giou_4conv1f_fp16_ms-crop_3x_coco_20220510_201004.log.json) |

**Note**:

- ConvNeXt backbone needs to install [MMPreTrain](https://github.com/open-mmlab/mmpretrain) first, which has abundant backbones for downstream tasks.

```shell
```

- The performance is unstable. `Cascade Mask R-CNN` may fluctuate about 0.2 mAP.

```bibtex

```shell
```

```bibtex
}
```

除了训练和测试脚本，我们还在 `tools/` 目录下提供了许多有用的工具。

## 日志分析

`tools/analysis_tools/analyze_logs.py` 可利用指定的训练 log 文件绘制 loss/mAP 曲线图，
第一次运行前请先运行 `pip install seaborn` 安装必要依赖.

```shell
```

样例:

- 绘制分类损失曲线图

  ```shell
  ```

- 绘制分类损失、回归损失曲线图，保存图片为对应的 pdf 文件

  ```shell
  ```

- 在相同图像中比较两次运行结果的 bbox mAP

  ```shell
  ```

- 计算平均训练速度

  ```shell

```

  ```shell
  ```

- 计算平均训练速度

  ```shell
  ```

  输出以如下形式展示

  ```text
  ```

## 结果分析

使用 `tools/analysis_tools/analyze_results.py` 可计算每个图像 mAP，随后根据真实标注框与预测框的比较结果，展示或保存最高与最低 top-k 得分的预测图像。

**使用方法**

```shell
      [--show] \
```

各个参数选项的作用:

- `config`: model config 文件的路径。

[--wait-time ${WAIT_TIME}] \
```

- `prediction_path`:  使用 `tools/test.py` 输出的 pickle 格式结果文件。
- `show_dir`: 绘制真实标注框与预测框的图像存放目录。
- `--show`：决定是否展示绘制 box 后的图片，默认值为 `False`。
- `--wait-time`: show 时间的间隔，若为 0 表示持续显示。
- `--topk`: 根据最高或最低 `topk` 概率排序保存的图片数量，若不指定，默认设置为 `20`。
- `--show-score-thr`: 能够展示的概率阈值，默认为 `0`。
- `--cfg-options`: 如果指定，可根据指定键值对覆盖更新配置文件的对应选项

**样例**:
假设你已经通过 `tools/test.py` 得到了 pickle 格式的结果文件，路径为 './result.pkl'。

1. 测试 Faster R-CNN 并可视化结果，保存图片至 `results/`

```shell
       results \
       --show
```

2. 测试 Faster R-CNN 并指定 top-k 参数为 50，保存结果图片至 `results/`

```shell

```

```shell
       results \
       --topk 50
```

3. 如果你想过滤低概率的预测结果，指定 `show-score-thr` 参数

```shell
       results \
```

## 可视化

### 可视化数据集

`tools/analysis_tools/browse_dataset.py` 可帮助使用者检查所使用的检测数据集（包括图像和标注），或保存图像至指定目录。

```shell
```

### 可视化模型

在可视化之前，需要先转换模型至 ONNX 格式，[可参考此处](#convert-mmdetection-model-to-onnx-experimental)。
注意，现在只支持 RetinaNet，之后的版本将会支持其他模型
转换后的模型可以被其他工具可视化[Netron](https://github.com/lutzroeder/netron)。

如果你想要一个轻量 GUI 可视化检测结果，你可以参考 [DetVisGUI project](https://github.com/Chien-Hung/DetVisGUI/tree/mmdetection)。

## 误差分析

`tools/analysis_tools/coco_error_analysis.py` 使用不同标准分析每个类别的 COCO 评估结果。同时将一些有帮助的信息体现在图表上。

```shell
```

样例:

假设你已经把 [Mask R-CNN checkpoint file](https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_fpn_1x_coco/mask_rcnn_r50_fpn_1x_coco_20200205-d4b0c5d6.pth) 放置在文件夹 'checkpoint' 中（其他模型请在 [model zoo](./model_zoo.md) 中获取）。

为了保存 bbox 结果信息，我们需要用下列方式修改 `test_evaluator` :

1. 查找当前 config 文件相对应的  'configs/base/datasets' 数据集信息。
2. 用当前数据集 config 中的 test_evaluator 以及 test_dataloader 替换原始文件的 test_evaluator 以及 test_dataloader。

3. 使用以下命令得到 bbox 或 segmentation 的 json 格式文件。

```shell
```

1. 得到每一类的 COCO bbox 误差结果，并保存分析结果图像至指定目录。（在 [config](../../../configs/_base_/datasets/coco_instance.py) 中默认目录是 './work_dirs/coco_instance/test')

```shell
       results \
```

2. 得到每一类的 COCO 分割误差结果，并保存分析结果图像至指定目录。

```shell
       results \
```

## 模型服务部署

       results \
```

## 模型服务部署

如果你想使用 [`TorchServe`](https://pytorch.org/serve/) 搭建一个 `MMDetection` 模型服务，可以参考以下步骤：

### 1. 安装 TorchServe

假设你已经成功安装了包含 `PyTorch` 和 `MMDetection` 的 `Python` 环境，那么你可以运行以下命令来安装 `TorchServe` 及其依赖项。有关更多其他安装选项，请参考[快速入门](https://github.com/pytorch/serve/blob/master/README.md#serve-a-model)。

```shell
```

**注意**: 如果你想在 docker 中使用`TorchServe`，请参考[torchserve docker](https://github.com/pytorch/serve/blob/master/docker/README.md)。

### 2. 把 MMDetection 模型转换至 TorchServe

```shell
```

### 3. 启动 `TorchServe`

```shell

```

```shell
```

### 4. 测试部署效果

```shell
```

你可以得到下列 json 信息：

```json
[
  {
    "bbox": [
    ],
  },
  {
    "bbox": [
    ],
  },
  {
    "bbox": [
    ],

},
  {
    "bbox": [
    ],
  }
]
```

#### 结果对比

你也可以使用 `test_torchserver.py` 来比较 `TorchServe` 和 `PyTorch` 的结果，并可视化：

```shell
```

样例:

```shell
yolov3 \
```

### 5. 停止 `TorchServe`

```shell
```

## 模型复杂度

`tools/analysis_tools/get_flops.py` 工具可用于计算指定模型的 FLOPs、参数量大小（改编自 [flops-counter.pytorch](https://github.com/sovrasov/flops-counter.pytorch) ）。

```shell

```shell
```

## 模型复杂度

```shell
```

获得的结果如下：

```text
```

**注意**：这个工具还只是实验性质，我们不保证这个数值是绝对正确的。你可以将他用于简单的比较，但如果用于科技论文报告需要再三检查确认。

1. FLOPs 与输入的形状大小相关，参数量没有这个关系，默认的输入形状大小为 (1, 3, 1280, 800) 。
2. 一些算子并不计入 FLOPs，比如 GN 或其他自定义的算子。你可以参考 [`mmcv.cnn.get_model_complexity_info()`](https://github.com/open-mmlab/mmcv/blob/2.x/mmcv/cnn/utils/flops_counter.py) 查看更详细的说明。
3. 两阶段检测的 FLOPs 大小取决于 proposal 的数量。

## 模型转换

### MMDetection 模型转换至 ONNX 格式

我们提供了一个脚本用于转换模型至 [ONNX](https://github.com/onnx/onnx) 格式。同时还支持比较 Pytorch 与 ONNX 模型的输出结果以便对照。更详细的内容可以参考 [mmdeploy](https://github.com/open-mmlab/mmdeploy)。

## 模型转换

### MMDetection 1.x 模型转换至 MMDetection 2.x 模型

`tools/model_converters/upgrade_model_version.py` 可将旧版本的 MMDetection checkpoints 转换至新版本。但要注意此脚本不保证在新版本加入非兼容更新后还能正常转换，建议您直接使用新版本的 checkpoints。

```shell
```

### RegNet 模型转换至 MMDetection 模型

`tools/model_converters/regnet2mmdet.py` 将 pycls 编码的预训练 RegNet 模型转换为 MMDetection 风格。

```shell
```

### Detectron ResNet 模型转换至 Pytorch 模型

`tools/model_converters/detectron2pytorch.py` 将 detectron 的原始预训练 RegNet 模型转换为 MMDetection 风格。

```shell
```

### 制作发布用模型

```shell
```

`tools/model_converters/publish_model.py` 可用来制作一个发布用的模型。

在发布模型至 AWS 之前，你可能需要：

1. 将模型转换至 CPU 张量
2. 删除优化器状态
3. 计算 checkpoint 文件的 hash 值，并将 hash 号码记录至文件名。

```shell
```

样例：

```shell
```

最后输出的文件名如下所示： `faster_rcnn_r50_fpn_1x_20190801-{hash id}.pth`.

## 数据集转换

`tools/data_converters/` 提供了将 Cityscapes 数据集与 Pascal VOC 数据集转换至 COCO 数据集格式的工具

```shell
```

## 数据集下载

```

## 数据集下载

`tools/misc/download_dataset.py` 可以下载各类形如 COCO， VOC， LVIS 数据集。

```shell
```

对于中国境内的用户，我们也推荐使用开源数据平台 [OpenDataLab](https://opendatalab.com/?source=OpenMMLab%20GitHub) 来获取这些数据集，以获得更好的下载体验:

## 基准测试

### 鲁棒性测试基准

## 基准测试

`tools/analysis_tools/test_robustness.py` 及 `tools/analysis_tools/robustness_eval.py` 帮助使用者衡量模型的鲁棒性。其核心思想来源于 [Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming](https://arxiv.org/abs/1907.07484)。如果你想了解如何在污损图像上评估模型的效果，以及参考该基准的一组标准模型，请参照 [robustness_benchmarking.md](robustness_benchmarking.md)。

### FPS 测试基准

`tools/analysis_tools/benchmark.py` 可帮助使用者计算 FPS，FPS 计算包括了模型向前传播与后处理过程。为了得到更精确的计算值，现在的分布式计算模式只支持一个 GPU。

```shell
```

```

样例：假设你已经下载了 `Faster R-CNN` 模型 checkpoint 并放置在 `checkpoints/` 目录下。

```shell
```

## 更多工具

### 以某个评估标准进行评估

`tools/analysis_tools/eval_metric.py` 根据配置文件中的评估方式对 pkl 结果文件进行评估。

```shell
```

### 打印全部 config

`tools/misc/print_config.py` 可将所有配置继承关系展开，完全打印相应的配置文件。

```shell

[--eval-options ${EVAL_OPTIONS [EVAL_OPTIONS ...]}]
```

```shell
```

## 超参数优化

### YOLO Anchor 优化

`tools/analysis_tools/optimize_anchors.py` 提供了两种方法优化 YOLO 的 anchors。

其中一种方法使用 K 均值 anchor 聚类（k-means anchor cluster），源自 [darknet](https://github.com/AlexeyAB/darknet/blob/master/src/detector.c#L1421)。

```shell
```

另一种方法使用差分进化算法优化 anchors。

```shell
```

样例：

```shell

```shell
```

样例：

```shell
```

你可能会看到如下结果：

```

......

......

```

## 混淆矩阵

混淆矩阵是对检测结果的概览。
`tools/analysis_tools/confusion_matrix.py` 可对预测结果进行分析，绘制成混淆矩阵表。
首先，运行 `tools/test.py` 保存 `.pkl` 预测结果。
之后再运行：

```
```

最后你可以得到如图的混淆矩阵：

## COCO 分离和遮挡实例分割性能评估

对于最先进的目标检测器来说，检测被遮挡的物体仍然是一个挑战。

我们实现了论文 [A Tri-Layer Plugin to Improve Occluded Detection](https://arxiv.org/abs/2210.10046) 中提出的指标来计算分离和遮挡目标的召回率。

使用此评价指标有两种方法：

### 离线评测

我们提供了一个脚本对存储后的检测结果文件计算指标。

首先，使用 `tools/test.py` 脚本存储检测结果：

```shell
```

然后，运行 `tools/analysis_tools/coco_occluded_separated_recall.py` 脚本来计算分离和遮挡目标的掩码的召回率:

```shell
```

输出如下：

```

```

### 在线评测

我们实现继承自 `CocoMetic` 的 `CocoOccludedSeparatedMetric`。

```

### 在线评测

要在训练期间评估分离和遮挡掩码的召回率，只需在配置中将 evaluator 类型替换为 `CocoOccludedSeparatedMetric`：

```python
    type='CocoOccludedSeparatedMetric',  # 修改此处
```

如果您使用了此指标，请引用论文：

```latex
}
```

# Frequently Asked Questions (FAQ)

We list some common troubles faced by many users and their corresponding solutions here. Feel free to enrich the list if you find any frequent issues and have ways to help others to solve them. If the contents here do not cover your issue, please create an issue using the [provided templates](https://github.com/open-mmlab/mmsegmentation/blob/dev-1.x/.github/ISSUE_TEMPLATE/error-report.md/) and make sure you fill in all required information in the template.

The compatible MMSegmentation, MMCV and MMEngine versions are as below. Please install the correct versions of them to avoid installation issues.

| MMSegmentation version |          MMCV version          | MMEngine version  | MMClassification (optional) version | MMDetection (optional) version |
| :--------------------: | :----------------------------: | :---------------: | :---------------------------------: | :----------------------------: |

|     dev-1.x branch     |         mmcv >= 2.0.0          | MMEngine >= 0.7.4 |        mmpretrain>=1.0.0rc7         |         mmdet >= 3.0.0         |
|      main branch       |         mmcv >= 2.0.0          | MMEngine >= 0.7.4 |        mmpretrain>=1.0.0rc7         |         mmdet >= 3.0.0         |
|         1.1.1          |         mmcv >= 2.0.0          | MMEngine >= 0.7.4 |        mmpretrain>=1.0.0rc7         |         mmdet >= 3.0.0         |
|         1.1.0          |         mmcv >= 2.0.0          | MMEngine >= 0.7.4 |        mmpretrain>=1.0.0rc7         |         mmdet >= 3.0.0         |
|         1.0.0          |        mmcv >= 2.0.0rc4        | MMEngine >= 0.7.1 |           mmcls==1.0.0rc6           |         mmdet >= 3.0.0         |

|        1.0.0rc6        |        mmcv >= 2.0.0rc4        | MMEngine >= 0.5.0 |           mmcls>=1.0.0rc0           |       mmdet >= 3.0.0rc6        |
|        1.0.0rc5        |        mmcv >= 2.0.0rc4        | MMEngine >= 0.2.0 |           mmcls>=1.0.0rc0           |        mmdet>=3.0.0rc6         |
|        1.0.0rc4        |        mmcv == 2.0.0rc3        | MMEngine >= 0.1.0 |           mmcls>=1.0.0rc0           |  mmdet>=3.0.0rc4, \<=3.0.0rc5  |
|        1.0.0rc3        |        mmcv == 2.0.0rc3        | MMEngine >= 0.1.0 |           mmcls>=1.0.0rc0           |  mmdet>=3.0.0rc4, \<=3.0.0rc5  |
|        1.0.0rc2        |        mmcv == 2.0.0rc3        | MMEngine >= 0.1.0 |           mmcls>=1.0.0rc0           |  mmdet>=3.0.0rc4, \<=3.0.0rc5  |

|        1.0.0rc1        | mmcv >= 2.0.0rc1, \<=2.0.0rc3> | MMEngine >= 0.1.0 |           mmcls>=1.0.0rc0           |          Not required          |
|        1.0.0rc0        | mmcv >= 2.0.0rc1, \<=2.0.0rc3> | MMEngine >= 0.1.0 |           mmcls>=1.0.0rc0           |          Not required          |

Notes:

- MMClassification and MMDetatction are optional for MMSegmentation. If you didn't install them, `ConvNeXt` (required MMClassification) and MaskFormer, Mask2Former (required MMDetection) cannot be used. We recommend to install them with source code. Please refer to [MMClasssication](https://github.com/open-mmlab/mmclassification) and [MMDetection](https://github.com/open-mmlab/mmdetection) for more details about their installation.

- To install MMSegmentation 0.x and master branch, please refer to [the faq 0.x document](https://mmsegmentation.readthedocs.io/en/latest/faq.html#installation) to check compatible versions of MMCV.

- If you have installed an incompatible version of mmcv, please run `pip uninstall mmcv` to uninstall the installed mmcv first. If you have previously installed mmcv-full (which exists in OpenMMLab 1.x), please run `pip uninstall mmcv-full` to uninstall it.

- If "No module named 'mmcv'" appears, please follow the steps below;

  1. Use `pip uninstall mmcv` to uninstall the existing mmcv in the environment.
  2. Install the corresponding mmcv according to the [installation instructions](https://mmsegmentation.readthedocs.io/en/dev-1.x/get_started.html#best-practices).

## How to know the number of GPUs needed to train the model

- Infer from the name of the config file of the model. You can refer to the `Config Name Style` part of [Learn about Configs](../user_guides/1_config.md). For example, for config file with name `segformer_mit-b0_8xb1-160k_cityscapes-1024x1024.py`, `8xb1` means training the model corresponding to it needs 8 GPUs, and the batch size of each GPU is 1.
- Infer from the log file. Open the log file of the model and search `nGPU` in the file. The number of figures following `nGPU` is the number of GPUs needed to train the model. For instance, searching for `nGPU` in the log file yields the record `nGPU 0,1,2,3,4,5,6,7`, which indicates that eight GPUs are needed to train the model.

## What does the auxiliary head mean

Briefly, it is a deep supervision trick to improve the accuracy. In the training phase, `decode_head` is for decoding semantic segmentation output, `auxiliary_head` is just adding an auxiliary loss, the segmentation result produced by it has no impact to your model's result, it just works in training. You may read this [paper](https://arxiv.org/pdf/1612.01105.pdf) for more information.

## How to output the segmentation mask image when running the test script

In the test script, we provide `--out` argument to control whether output the painted images. Users might run the following command:

```shell
```

## How to handle binary segmentation task

```shell
```

MMSegmentation uses `num_classes` and `out_channels` to control output of last layer `self.conv_seg`. More details could be found [here](https://github.com/open-mmlab/mmsegmentation/blob/dev-1.x/mmseg/models/decode_heads/decode_head.py).

`num_classes` should be the same as number of types of labels, in binary segmentation task, dataset only has two types of labels: foreground and background, so `num_classes=2`. `out_channels` controls the output channel of last layer of model, it usually equals to `num_classes`.
But in binary segmentation task, there are two solutions:

- Set `out_channels=2`, using Cross Entropy Loss in training, using `F.softmax()` and `argmax()` to get prediction of each pixel in inference.

- Set `out_channels=1`, using Binary Cross Entropy Loss in training, using `F.sigmoid()` and `threshold` to get prediction of each pixel in inference. `threshold` is set 0.3 as default.

In summary, to implement binary segmentation methods users should modify below parameters in the `decode_head` and `auxiliary_head` configs. Here is a modification example of [pspnet_unet_s5-d16.py](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/_base_/models/pspnet_unet_s5-d16.py):

- (1) `num_classes=2`, `out_channels=2` and `use_sigmoid=False` in `CrossEntropyLoss`.

```python
    type='PSPHead',
    in_channels=64,
    in_index=4,
    num_classes=2,
    out_channels=2,
    loss_decode=dict(
        type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0)),
auxiliary_head=dict(
    type='FCNHead',
    in_channels=128,

num_classes=2,
    in_index=3,
        type='CrossEntropyLoss', use_sigmoid=False, loss_weight=0.4)),
```

- (2) `num_classes=2`, `out_channels=1` and `use_sigmoid=True` in `CrossEntropyLoss`.

```python
    out_channels=1,
        type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0)),
        type='CrossEntropyLoss', use_sigmoid=True, loss_weight=0.4)),
```

## Functionality of `reduce_zero_label`

in_index=3,
```

The parameter type of `reduce_zero_label` in dataset is Boolean, which is default to False. It is used to ignore the dataset label 0. The specific method is to change label 0 to 255, and subtract 1 from the corresponding number of all the remaining labels. At the same time, set 255 as ignore index in the decode head, which means that it will not participate in the loss calculation.

Following is the specific implementation logic of `reduce_zero_label`:

```python
if self.reduce_zero_label:
    # avoid using underflow conversion
    gt_semantic_seg[gt_semantic_seg == 0] = 255
    gt_semantic_seg = gt_semantic_seg - 1
    gt_semantic_seg[gt_semantic_seg == 254] = 255
```

Whether your dataset needs to use `reduce_zero_label`, there are two types of situations:

- On [Potsdam](https://github.com/open-mmlab/mmsegmentation/blob/1.x/docs/en/user_guides/2_dataset_prepare.md#isprs-potsdam) dataset, there are six classes: 0-Impervious surfaces, 1-Building, 2-Low vegetation, 3-Tree, 4-Car, 5-Clutter/background. However, this dataset provides two types of RGB labels, one with black pixels at the edges of the images, and the other without. For labels with black edges, in [dataset_converters.py](https://github.com/open-mmlab/mmsegmentation/blob/dev-1.x/tools/dataset_converters/potsdam.py), it converts the black edges to label 0, and the other labels are 1-Impervious surfaces, 2-Building, 3-Low vegetation, 4-Tree, 5-Car, 6-Clutter/background. Therefore, in the dataset config [potsdam.py](https://github.com/open-mmlab/mmsegmentation/blob/ff95416c3b5ce8d62b9289f743531398efce534f/mmseg/datasets/potsdam.py#L23) `reduce_zero_label=True`。 If you are using labels without black edges, then there are only class 0-5 in the mask label. At this point, you should use `reduce_zero_label=False`. `reduce_zero_label` usage needs to be considered with your actual situation.

- On a dataset with class 0 as the background class, if you need to separate the background from the rest of your classes ultimately then you do not need to use `reduce_zero_label`, which in the dataset config settings should be `reduce_zero_label=False`

**Note:** Please confirm the number of original classes in the dataset. If there are only two classes, you should not use `reduce_zero_label` which is `reduce_zero_label=False`.

# ConvNeXt-V2

> [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](http://arxiv.org/abs/2301.00808)

Driven by improved architectures and better representation learning frameworks, the field of visual recognition has enjoyed rapid modernization and performance boost in the early 2020s. For example, modern ConvNets, represented by ConvNeXt \[52\], have demonstrated strong performance in various scenarios. While these models were originally designed for supervised learning with ImageNet labels, they can also potentially benefit from self-supervised learning techniques such as masked autoencoders (MAE) . However, we found that simply combining these two approaches leads to subpar performance. In this paper, we propose a fully convolutional masked autoencoder framework and a new Global Response Normalization (GRN) layer that can be added to the ConvNeXt architecture to enhance inter-channel feature competition. This co-design of self-supervised learning techniques and architectural improvement results in a new model family called ConvNeXt V2, which significantly improves the performance of pure ConvNets on various recognition benchmarks, including ImageNet classification, COCO detection, and ADE20K segmentation. We also provide pre-trained ConvNeXt V2 models of various sizes, ranging from an efficient 3.7Mparameter Atto model with 76.7% top-1 accuracy on Im-ageNet, to a 650M Huge model that achieves a state-of-theart 88.9% accuracy using only public training data.

<img src="https://user-images.githubusercontent.com/12907710/212588579-02d621d8-5796-4f0d-b4d2-758fe9c2f395.png" width="50%"/>
</div>

|   Method   |   Backbone    | Pretrain | Lr schd | Augmentation | Mem (GB) | box AP | mask AP |                            Config                            |                                                                                                                                                                                        Download                                                                                                                                                                                         |

| Mask R-CNN | ConvNeXt-V2-B |  FCMAE   |   3x    |     LSJ      |   22.5   |  52.9  |  46.4   | [config](./mask-rcnn_convnext-v2-b_fpn_lsj-3x-fcmae_coco.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/convnextv2/mask-rcnn_convnext-v2-b_fpn_lsj-3x-fcmae_coco/mask-rcnn_convnext-v2-b_fpn_lsj-3x-fcmae_coco_20230113_110947-757ee2dd.pth)  \| [log](https://download.openmmlab.com/mmdetection/v3.0/convnextv2/mask-rcnn_convnext-v2-b_fpn_lsj-3x-fcmae_coco/mask-rcnn_convnext-v2-b_fpn_lsj-3x-fcmae_coco_20230113_110947.log.json) |

**Note**:

- This is a pre-release version of ConvNeXt-V2 object detection. The official finetuning setting of ConvNeXt-V2 has not been released yet.
- ConvNeXt backbone needs to install [MMPretrain](https://github.com/open-mmlab/mmpretrain/) first, which has abundant backbones for downstream tasks.

```shell
```

```bibtex

```shell
```

```bibtex
}
```

}
```

<details>

```bibtex
}
```

| [pose_hrnetv2_w18_dark](/configs/face_2d_keypoint/topdown_heatmap/coco_wholebody_face/td-hm_hrnetv2-w18_dark-8xb32-60e_coco-wholebody-face-256x256.py) |  256x256   | 0.0513 | [ckpt](https://download.openmmlab.com/mmpose/face/darkpose/hrnetv2_w18_coco_wholebody_face_256x256_dark-3d9a334e_20210909.pth) | [log](https://download.openmmlab.com/mmpose/face/darkpose/hrnetv2_w18_coco_wholebody_face_256x256_dark_20210909.log.json) |

<details>

```bibtex
}
```

<details>

```bibtex

journal={Computational Visual Media},
}
```

<details>

```bibtex
}
```

| [pose_pvt-s](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_pvt-s_8xb64-210e_coco-256x192.py) |  256x192   | 0.714 |      0.896      |      0.794      | 0.773 |      0.936      | [ckpt](https://download.openmmlab.com/mmpose/top_down/pvt/pvt_small_coco_256x192-4324a49d_20220501.pth) | [log](https://download.openmmlab.com/mmpose/top_down/pvt/pvt_small_coco_256x192_20220501.log.json) |

| [pose_pvtv2-b2](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_pvtv2-b2_8xb64-210e_coco-256x192.py) |  256x192   | 0.737 |      0.905      |      0.812      | 0.791 |      0.942      | [ckpt](https://download.openmmlab.com/mmpose/top_down/pvt/pvtv2_b2_coco_256x192-b4212737_20220501.pth) | [log](https://download.openmmlab.com/mmpose/top_down/pvt/pvtv2_b2_coco_256x192_20220501.log.json) |

# RetinaNet

> [Focal Loss for Dense Object Detection](https://arxiv.org/abs/1708.02002)

<img src="https://user-images.githubusercontent.com/40661020/143973551-2b8e766a-1677-4f6d-953d-2e6d2a3c67b5.png" height="300"/>
</div>

|    Backbone     |  Style  |   Lr schd    | Mem (GB) | Inf time (fps) | box AP |                     Config                      |                                                                                                                                                         Download                                                                                                                                                          |

| :-------------: | :-----: | :----------: | :------: | :------------: | :----: | :---------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|    R-18-FPN     | pytorch |      1x      |   1.7    |                |  31.7  |    [config](./retinanet_r18_fpn_1x_coco.py)     |           [model](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r18_fpn_1x_coco/retinanet_r18_fpn_1x_coco_20220407_171055-614fd399.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r18_fpn_1x_coco/retinanet_r18_fpn_1x_coco_20220407_171055.log.json)            |

|    R-18-FPN     | pytorch | 1x(1 x 8 BS) |   5.0    |                |  31.7  |  [config](./retinanet_r18_fpn_1xb8-1x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r18_fpn_1x8_1x_coco/retinanet_r18_fpn_1x8_1x_coco_20220407_171255-4ea310d7.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r18_fpn_1x8_1x_coco/retinanet_r18_fpn_1x8_1x_coco_20220407_171255.log.json)    |
|    R-50-FPN     |  caffe  |      1x      |   3.5    |      18.6      |  36.3  | [config](./retinanet_r50-caffe_fpn_1x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r50_caffe_fpn_1x_coco/retinanet_r50_caffe_fpn_1x_coco_20200531-f11027c5.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r50_caffe_fpn_1x_coco/retinanet_r50_caffe_fpn_1x_coco_20200531_012518.log.json)   |

|    R-50-FPN     | pytorch |      1x      |   3.8    |      19.0      |  36.5  |    [config](./retinanet_r50_fpn_1x_coco.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r50_fpn_1x_coco/retinanet_r50_fpn_1x_coco_20200130-c2398f9e.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r50_fpn_1x_coco/retinanet_r50_fpn_1x_coco_20200130_002941.log.json)               |
| R-50-FPN (FP16) | pytorch |      1x      |   2.8    |      31.6      |  36.4  |  [config](./retinanet_r50_fpn_amp-1x_coco.py)   |          [model](https://download.openmmlab.com/mmdetection/v2.0/fp16/retinanet_r50_fpn_fp16_1x_coco/retinanet_r50_fpn_fp16_1x_coco_20200702-0dbfb212.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/fp16/retinanet_r50_fpn_fp16_1x_coco/retinanet_r50_fpn_fp16_1x_coco_20200702_020127.log.json)          |

|    R-50-FPN     | pytorch |      2x      |    -     |       -        |  37.4  |    [config](./retinanet_r50_fpn_2x_coco.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r50_fpn_2x_coco/retinanet_r50_fpn_2x_coco_20200131-fdb43119.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r50_fpn_2x_coco/retinanet_r50_fpn_2x_coco_20200131_114738.log.json)               |
|    R-101-FPN    |  caffe  |      1x      |   5.5    |      14.7      |  38.5  | [config](./retinanet_r101-caffe_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r101_caffe_fpn_1x_coco/retinanet_r101_caffe_fpn_1x_coco_20200531-b428fa0f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r101_caffe_fpn_1x_coco/retinanet_r101_caffe_fpn_1x_coco_20200531_012536.log.json) |

|    R-101-FPN    | pytorch |      1x      |   5.7    |      15.0      |  38.5  |    [config](./retinanet_r101_fpn_1x_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r101_fpn_1x_coco/retinanet_r101_fpn_1x_coco_20200130-7a93545f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r101_fpn_1x_coco/retinanet_r101_fpn_1x_coco_20200130_003055.log.json)             |
|    R-101-FPN    | pytorch |      2x      |    -     |       -        |  38.9  |    [config](./retinanet_r101_fpn_2x_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r101_fpn_2x_coco/retinanet_r101_fpn_2x_coco_20200131-5560aee8.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r101_fpn_2x_coco/retinanet_r101_fpn_2x_coco_20200131_114859.log.json)             |

| X-101-32x4d-FPN | pytorch |      1x      |   7.0    |      12.1      |  39.9  | [config](./retinanet_x101-32x4d_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_x101_32x4d_fpn_1x_coco/retinanet_x101_32x4d_fpn_1x_coco_20200130-5c8b7ec4.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_x101_32x4d_fpn_1x_coco/retinanet_x101_32x4d_fpn_1x_coco_20200130_003004.log.json) |
| X-101-32x4d-FPN | pytorch |      2x      |    -     |       -        |  40.1  | [config](./retinanet_x101-32x4d_fpn_2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_x101_32x4d_fpn_2x_coco/retinanet_x101_32x4d_fpn_2x_coco_20200131-237fc5e1.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_x101_32x4d_fpn_2x_coco/retinanet_x101_32x4d_fpn_2x_coco_20200131_114812.log.json) |

| X-101-64x4d-FPN | pytorch |      1x      |   10.0   |      8.7       |  41.0  | [config](./retinanet_x101-64x4d_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_x101_64x4d_fpn_1x_coco/retinanet_x101_64x4d_fpn_1x_coco_20200130-366f5af1.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_x101_64x4d_fpn_1x_coco/retinanet_x101_64x4d_fpn_1x_coco_20200130_003008.log.json) |
| X-101-64x4d-FPN | pytorch |      2x      |    -     |       -        |  40.8  | [config](./retinanet_x101-64x4d_fpn_2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_x101_64x4d_fpn_2x_coco/retinanet_x101_64x4d_fpn_2x_coco_20200131-bca068ab.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_x101_64x4d_fpn_2x_coco/retinanet_x101_64x4d_fpn_2x_coco_20200131_114833.log.json) |

|    Backbone     |  Style  | Lr schd | Mem (GB) | box AP |                           Config                           |                                                                                                                                                                                 Download                                                                                                                                                                                  |

| :-------------: | :-----: | :-----: | :------: | :----: | :--------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|    R-50-FPN     | pytorch |   3x    |   3.5    |  39.5  |    [config](./retinanet_r50_fpn_ms-640-800-3x_coco.py)     |               [model](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r50_fpn_mstrain_3x_coco/retinanet_r50_fpn_mstrain_3x_coco_20210718_220633-88476508.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r50_fpn_mstrain_3x_coco/retinanet_r50_fpn_mstrain_3x_coco_20210718_220633-88476508.log.json)               |

|    R-101-FPN    |  caffe  |   3x    |   5.4    |  40.7  |     [config](./retinanet_r101-caffe_fpn_ms-3x_coco.py)     | [model](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r101_caffe_fpn_mstrain_3x_coco/retinanet_r101_caffe_fpn_mstrain_3x_coco_20210721_063439-88a8a944.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r101_caffe_fpn_mstrain_3x_coco/retinanet_r101_caffe_fpn_mstrain_3x_coco_20210721_063439-88a8a944.log.json) |
|    R-101-FPN    | pytorch |   3x    |   5.4    |   41   |    [config](./retinanet_r101_fpn_ms-640-800-3x_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r101_fpn_mstrain_3x_coco/retinanet_r101_fpn_mstrain_3x_coco_20210720_214650-7ee888e0.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r101_fpn_mstrain_3x_coco/retinanet_r101_fpn_mstrain_3x_coco_20210720_214650-7ee888e0.log.json)             |

| X-101-64x4d-FPN | pytorch |   3x    |   9.8    |  41.6  | [config](./retinanet_x101-64x4d_fpn_ms-640-800-3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_x101_64x4d_fpn_mstrain_3x_coco/retinanet_x101_64x4d_fpn_mstrain_3x_coco_20210719_051838-022c2187.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_x101_64x4d_fpn_mstrain_3x_coco/retinanet_x101_64x4d_fpn_mstrain_3x_coco_20210719_051838-022c2187.log.json) |

```latex
}
```

# Simple online and realtime tracking with a deep association metric

Simple Online and Realtime Tracking (SORT) is a pragmatic approach to multiple object tracking with a focus on simple, effective algorithms. In this paper, we integrate appearance information to improve the performance of SORT. Due to this extension we are able to track objects through longer periods of occlusions, effectively reducing the number of identity switches. In spirit of the original framework we place much of the computational complexity into an offline pre-training stage where we learn a deep association metric on a largescale person re-identification dataset. During online application, we establish measurement-to-track associations using nearest neighbor queries in visual appearance space. Experimental evaluation shows that our extensions reduce the number of identity switches by 45%, achieving overall competitive performance at high frame rates.

  <img src="https://user-images.githubusercontent.com/26813582/145542023-22950508-b35f-41b6-bc78-33d6a82bc3c3.png"/>
</div>

Currently we do not support training ReID models for DeepSORT.
We directly use the ReID model from [Tracktor](https://github.com/phil-bergmann/tracking_wo_bnw). These missed features will be supported in the future.

|  Method  |      Detector      | ReID | Train Set  | Test Set | Public | Inf time (fps) | HOTA | MOTA | IDF1 |  FP   |  FN   | IDSw. |                                       Config                                       |                                                                                                         Download                                                                                                         |

| :------: | :----------------: | :--: | :--------: | :------: | :----: | :------------: | :--: | :--: | :--: | :---: | :---: | :---: | :--------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| DeepSORT | R50-FasterRCNN-FPN | R50  | half-train | half-val |   N    |      13.8      | 57.0 | 63.7 | 69.5 | 15063 | 40323 | 3276  | [config](deepsort_faster-rcnn_r50_fpn_8xb2-4e_mot17halftrain_test-mot17halfval.py) | [detector](https://download.openmmlab.com/mmtracking/mot/faster_rcnn/faster-rcnn_r50_fpn_4e_mot17-half-64ee2ed4.pth) [reid](https://download.openmmlab.com/mmtracking/mot/reid/tracktor_reid_r50_iter25245-a452f51f.pth) |

We implement DeepSORT with independent detector and ReID models.

```

```

**4.1.1 use separate trained detector and reid model to evaluating and testing**

```shell
bash tools/dist_test_tracking.sh configs/deepsort/deepsort_faster-rcnn_r50_fpn_8xb2-4e_mot17halftrain_test-mot17halfval.py 8 --detector ${DETECTOR_CHECKPOINT_PATH} --reid ${REID_CHECKPOINT_PATH}
```

```
```

bash tools/dist_test_tracking.sh configs/deepsort/deepsort_faster-rcnn_r50_fpn_8xb2-4e_mot17train_test-mot17test 8 --detector ${DETECTOR_CHECKPOINT_PATH} --reid ${REID_CHECKPOINT_PATH}
```

```

```shell
python demo/mot_demo.py demo/demo_mot.mp4 configs/deepsort/deepsort_faster-rcnn_r50_fpn_8xb2-4e_mot17train_test-mot17test --detector ${DETECTOR_CHECKPOINT_PATH} --reid ${REID_CHECKPOINT_PATH} --out mot.mp4
```

```latex
@inproceedings{wojke2017simple,
  title={Simple online and realtime tracking with a deep association metric},
  author={Wojke, Nicolai and Bewley, Alex and Paulus, Dietrich},
  booktitle={2017 IEEE international conference on image processing (ICIP)},

author={Wojke, Nicolai and Bewley, Alex and Paulus, Dietrich},
  pages={3645--3649},
}
```

# Twins

> [Twins: Revisiting the Design of Spatial Attention in Vision Transformers](http://arxiv-export-lb.library.cornell.edu/abs/2104.13840)

Very recently, a variety of vision transformer architectures for dense prediction tasks have been proposed and they show that the design of spatial attention is critical to their success in these tasks. In this work, we revisit the design of the spatial attention and demonstrate that a carefully-devised yet simple spatial attention mechanism performs favourably against the state-of-the-art schemes. As a result, we propose two vision transformer architectures, namely, Twins-PCPVT and Twins-SVT. Our proposed architectures are highly-efficient and easy to implement, only involving matrix multiplications that are highly optimized in modern deep learning frameworks. More importantly, the proposed architectures achieve excellent performance on a wide range of visual tasks, including image level classification as well as dense detection and segmentation. The simplicity and strong performance suggest that our proposed architectures may serve as stronger backbones for many vision tasks. Our code is released at [this https URL](https://github.com/Meituan-AutoML/Twins).

<img src="https://user-images.githubusercontent.com/24582831/145021310-57826cf5-5e03-4c7c-9081-ffa744bdae27.png" width="80%"/>
</div>

```python

predict = inference_model('twins-pcpvt-small_3rdparty_8xb128_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('twins-pcpvt-small_3rdparty_8xb128_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/twins/twins-pcpvt-small_8xb128_in1k.py https://download.openmmlab.com/mmclassification/v0/twins/twins-pcpvt-small_3rdparty_8xb128_in1k_20220126-ef23c132.pth
```

| `twins-pcpvt-small_3rdparty_8xb128_in1k`\* | From scratch |   24.11    |   3.67    |   81.14   |   95.69   | [config](twins-pcpvt-small_8xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/twins/twins-pcpvt-small_3rdparty_8xb128_in1k_20220126-ef23c132.pth) |
| `twins-pcpvt-base_3rdparty_8xb128_in1k`\*  | From scratch |   43.83    |   6.45    |   82.66   |   96.26   | [config](twins-pcpvt-base_8xb128_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/twins/twins-pcpvt-base_3rdparty_8xb128_in1k_20220126-f8c4b0d5.pth) |
| `twins-pcpvt-large_3rdparty_16xb64_in1k`\* | From scratch |   60.99    |   9.51    |   83.09   |   96.59   | [config](twins-pcpvt-large_16xb64_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/twins/twins-pcpvt-large_3rdparty_16xb64_in1k_20220126-c1ef8d80.pth) |

| `twins-svt-small_3rdparty_8xb128_in1k`\*   | From scratch |   24.06    |   2.82    |   81.77   |   95.57   |  [config](twins-svt-small_8xb128_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/twins/twins-svt-small_3rdparty_8xb128_in1k_20220126-8fe5205b.pth) |
| `twins-svt-base_8xb128_3rdparty_in1k`\*    | From scratch |   56.07    |   8.35    |   83.13   |   96.29   |  [config](twins-svt-base_8xb128_in1k.py)   | [model](https://download.openmmlab.com/mmclassification/v0/twins/twins-svt-base_3rdparty_8xb128_in1k_20220126-e31cc8e9.pth) |
| `twins-svt-large_3rdparty_16xb64_in1k`\*   | From scratch |   99.27    |   14.82   |   83.60   |   96.50   |  [config](twins-svt-large_16xb64_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/twins/twins-svt-large_3rdparty_16xb64_in1k_20220126-4817645f.pth) |

*Models with * are converted from the [timm](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/twins.py). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@article{chu2021twins,
  title={Twins: Revisiting spatial attention design in vision transformers},
  author={Chu, Xiangxiang and Tian, Zhi and Wang, Yuqing and Zhang, Bo and Ren, Haibing and Wei, Xiaolin and Xia, Huaxia and Shen, Chunhua},
  journal={arXiv preprint arXiv:2104.13840},
  year={2021}altgvt
}
```

# GFL

> [Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection](https://arxiv.org/abs/2006.04388)

One-stage detector basically formulates object detection as dense classification and localization. The classification is usually optimized by Focal Loss and the box location is commonly learned under Dirac delta distribution. A recent trend for one-stage detectors is to introduce an individual prediction branch to estimate the quality of localization, where the predicted quality facilitates the classification to improve detection performance. This paper delves into the representations of the above three fundamental elements: quality estimation, classification and localization. Two problems are discovered in existing practices, including (1) the inconsistent usage of the quality estimation and classification between training and inference and (2) the inflexible Dirac delta distribution for localization when there is ambiguity and uncertainty in complex scenes. To address the problems, we design new representations for these elements. Specifically, we merge the quality estimation into the class prediction vector to form a joint representation of localization quality and classification, and use a vector to represent arbitrary distribution of box locations. The improved representations eliminate the inconsistency risk and accurately depict the flexible distribution in real data, but contain continuous labels, which is beyond the scope of Focal Loss. We then propose Generalized Focal Loss (GFL) that generalizes Focal Loss from its discrete form to the continuous version for successful optimization. On COCO test-dev, GFL achieves 45.0% AP using ResNet-101 backbone, surpassing state-of-the-art SAPD (43.5%) and ATSS (43.6%) with higher or comparable inference speed, under the same backbone and training settings. Notably, our best model can achieve a single-model single-scale AP of 48.2%, at 10 FPS on a single 2080Ti GPU.

<img src="https://user-images.githubusercontent.com/40661020/143887865-44dc384d-ba0d-44e8-b3d7-d5fa837838cf.png"/>
</div>

|     Backbone      |  Style  | Lr schd | Multi-scale Training | Inf time (fps) | box AP |                          Config                          |                                                                                                                                                                                   Download                                                                                                                                                                                   |

|       R-50        | pytorch |   1x    |          No          |      19.5      |  40.2  |            [config](./gfl_r50_fpn_1x_coco.py)            |                                                       [model](https://download.openmmlab.com/mmdetection/v2.0/gfl/gfl_r50_fpn_1x_coco/gfl_r50_fpn_1x_coco_20200629_121244-25944287.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gfl/gfl_r50_fpn_1x_coco/gfl_r50_fpn_1x_coco_20200629_121244.log.json)                                                       |

|       R-50        | pytorch |   2x    |         Yes          |      19.5      |  42.9  |          [config](./gfl_r50_fpn_ms-2x_coco.py)           |                                       [model](https://download.openmmlab.com/mmdetection/v2.0/gfl/gfl_r50_fpn_mstrain_2x_coco/gfl_r50_fpn_mstrain_2x_coco_20200629_213802-37bb1edc.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gfl/gfl_r50_fpn_mstrain_2x_coco/gfl_r50_fpn_mstrain_2x_coco_20200629_213802.log.json)                                       |

|       R-101       | pytorch |   2x    |         Yes          |      14.7      |  44.7  |          [config](./gfl_r101_fpn_ms-2x_coco.py)          |                                     [model](https://download.openmmlab.com/mmdetection/v2.0/gfl/gfl_r101_fpn_mstrain_2x_coco/gfl_r101_fpn_mstrain_2x_coco_20200629_200126-dd12f847.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gfl/gfl_r101_fpn_mstrain_2x_coco/gfl_r101_fpn_mstrain_2x_coco_20200629_200126.log.json)                                     |

|    R-101-dcnv2    | pytorch |   2x    |         Yes          |      12.9      |  47.1  |    [config](./gfl_r101-dconv-c3-c5_fpn_ms-2x_coco.py)    |             [model](https://download.openmmlab.com/mmdetection/v2.0/gfl/gfl_r101_fpn_dconv_c3-c5_mstrain_2x_coco/gfl_r101_fpn_dconv_c3-c5_mstrain_2x_coco_20200630_102002-134b07df.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gfl/gfl_r101_fpn_dconv_c3-c5_mstrain_2x_coco/gfl_r101_fpn_dconv_c3-c5_mstrain_2x_coco_20200630_102002.log.json)             |

|    X-101-32x4d    | pytorch |   2x    |         Yes          |      12.1      |  45.9  |       [config](./gfl_x101-32x4d_fpn_ms-2x_coco.py)       |                         [model](https://download.openmmlab.com/mmdetection/v2.0/gfl/gfl_x101_32x4d_fpn_mstrain_2x_coco/gfl_x101_32x4d_fpn_mstrain_2x_coco_20200630_102002-50c1ffdb.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gfl/gfl_x101_32x4d_fpn_mstrain_2x_coco/gfl_x101_32x4d_fpn_mstrain_2x_coco_20200630_102002.log.json)                         |

| X-101-32x4d-dcnv2 | pytorch |   2x    |         Yes          |      10.7      |  48.1  | [config](./gfl_x101-32x4d-dconv-c4-c5_fpn_ms-2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/gfl/gfl_x101_32x4d_fpn_dconv_c4-c5_mstrain_2x_coco/gfl_x101_32x4d_fpn_dconv_c4-c5_mstrain_2x_coco_20200630_102002-14a2bf25.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/gfl/gfl_x101_32x4d_fpn_dconv_c4-c5_mstrain_2x_coco/gfl_x101_32x4d_fpn_dconv_c4-c5_mstrain_2x_coco_20200630_102002.log.json) |

\[3\] *`dcnv2` denotes deformable convolutional networks v2.* \
\[4\] *FPS is tested with a single GeForce RTX 2080Ti GPU, using a batch size of 1.*

We provide config files to reproduce the object detection results in the paper [Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection](https://arxiv.org/abs/2006.04388)

```latex
@article{li2020generalized,
  title={Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection},
  author={Li, Xiang and Wang, Wenhai and Wu, Lijun and Chen, Shuo and Hu, Xiaolin and Li, Jun and Tang, Jinhui and Yang, Jian},
  journal={arXiv preprint arXiv:2006.04388},
}
```

### 2D Human Whole-Body Pose Top-Down Image Demo

```shell
```

The pre-trained hand pose estimation models can be downloaded from [model zoo](https://mmpose.readthedocs.io/en/latest/model_zoo/2d_wholebody_keypoint.html).
Take [coco-wholebody_vipnas_res50_dark](https://download.openmmlab.com/mmpose/top_down/vipnas/vipnas_res50_wholebody_256x192_dark-67c0ce35_20211112.pth) model as an example:

```shell

```

```shell
```

```shell
```

Examples:

```shell

https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_coco_wholebody_384x288_dark-f5726563_20200918.pth \
```

### 2D Human Whole-Body Pose Top-Down Video Demo

The above demo script can also take video as input, and run mmdet for human detection, and mmpose for pose estimation.

Examples:

```shell

configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_hrnet-w48_dark-8xb32-210e_coco-wholebody-384x288.py \
```

### 2D Human Whole-Body Pose Estimation with Inferencer

```shell

```shell
```

This command infers all images located in `tests/data/crowdpose` and saves the visualization results in the `vis_results/crowdpose` directory.

1. set `model.test_cfg.flip_test=False` in [pose_hrnet_w48_dark+](/configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_hrnet-w48_dark-8xb32-210e_coco-wholebody-384x288.py#L90).

# Customize Runtime Settings (To be updated)

We already support to use all the optimizers implemented by PyTorch, and the only modification is to change the `optimizer` field of config files.
For example, if you want to use `ADAM` (note that the performance could drop a lot), the modification could be as the following.

```python
```

To modify the learning rate of the model, the users only need to modify the `lr` in the config of `optimizer`. The users can directly set arguments following the [API doc](https://pytorch.org/docs/stable/optim.html?highlight=optim#module-torch.optim) of PyTorch.

You need to create a new directory named `mmrotate/core/optimizer`.
And then implement the new optimizer in a file, e.g., in `mmrotate/core/optimizer/my_optimizer.py`:

```python

```

- Modify `mmrotate/core/optimizer/__init__.py` to import it.

  The newly defined module should be imported in `mmrotate/core/optimizer/__init__.py` so that the registry will
  find the new module and add it:

```python
```

The newly defined module should be imported in `mmrotate/core/optimizer/__init__.py` so that the registry will

```python
```

```python
```

The module `mmrotate.core.optimizer.my_optimizer` will be imported at the beginning of the program and the class `MyOptimizer` is then automatically registered.
`mmrotate.core.optimizer.my_optimizer.MyOptimizer` **cannot** be imported directly.

Then you can use `MyOptimizer` in `optimizer` field of config files.

In the configs, the optimizers are defined by the field `optimizer` like the following:

```python
```

```python
```

### Customize optimizer constructor

The users can do those fine-grained parameter tuning through customizing optimizer constructor.

```python

```

The default optimizer constructor is implemented [here](https://github.com/open-mmlab/mmcv/blob/9ecd6b0d5ff9d2172c49a182eaa669e9f27bb8e7/mmcv/runner/optimizer/default_constructor.py#L11), which could also serve as a template for new optimizer constructor.

Tricks not implemented by the optimizer should be implemented through optimizer constructor (e.g., set parameter-wise learning rates) or hooks. We list some common settings that could stabilize the training or accelerate the training. Feel free to create PR, issue for more settings.

  ```python
  ```

  If your config inherits the base config which already sets the `optimizer_config`, you might need `_delete_=True` to override the unnecessary settings. See the [config documentation](https://mmdetection.readthedocs.io/en/latest/tutorials/config.html) for more details.

  Momentum scheduler is usually used with LR scheduler, for example, the following config is used in 3D detection to accelerate convergence.

Momentum scheduler is usually used with LR scheduler, for example, the following config is used in 3D detection to accelerate convergence.
  For more details, please refer to the implementation of [CyclicLrUpdater](https://github.com/open-mmlab/mmcv/blob/f48241a65aebfe07db122e9db320c31b685dc674/mmcv/runner/hooks/lr_updater.py#L327) and [CyclicMomentumUpdater](https://github.com/open-mmlab/mmcv/blob/f48241a65aebfe07db122e9db320c31b685dc674/mmcv/runner/hooks/momentum_updater.py#L130).

  ```python
  )
  )
  ```

)
  )
  ```

By default we use step learning rate with 1x schedule, this calls [`StepLRHook`](https://github.com/open-mmlab/mmcv/blob/f48241a65aebfe07db122e9db320c31b685dc674/mmcv/runner/hooks/lr_updater.py#L153) in MMCV.
We support many other learning rate schedule [here](https://github.com/open-mmlab/mmcv/blob/master/mmcv/runner/hooks/lr_updater.py), such as `CosineAnnealing` and `Poly` schedule. Here are some examples

  ```python
  ```

  ```python
  ```

## Customize workflow

```python
  ```

Workflow is a list of (phase, epochs) to specify the running order and epochs.
By default it is set to be

```python
```

which means running 1 epoch for training.
Sometimes user may want to check some metrics (e.g. loss, accuracy) about the model on the validate set.
In such case, we can set the workflow as

```python
```

so that 1 epoch for training and 1 epoch for validation will be run iteratively.

**Note**:

1. The parameters of model will not be updated during val epoch.
2. Keyword `total_epochs` in the config only controls the number of training epochs and will not affect the validation workflow.

3. Workflows `[('train', 1), ('val', 1)]` and `[('train', 1)]` will not change the behavior of `EvalHook` because `EvalHook` is called by `after_train_epoch` and validation workflow only affect hooks that are called through `after_val_epoch`. Therefore, the only difference between `[('train', 1), ('val', 1)]` and `[('train', 1)]` is that the runner will calculate losses on validation set after each training epoch.

There are some occasions when the users might need to implement a new hook. MMRotate supports customized hooks in training. Thus the users could implement a hook directly in mmrotate or their mmdet-based codebases and use the hook by only modifying the config in training.

Here we give an example of creating a new hook in mmrotate and using it in training.

```python

        pass

        pass

        pass

        pass

        pass

        pass

        pass
```

Depending on the functionality of the hook, the users need to specify what the hook will do at each stage of the training in `before_run`, `after_run`, `before_epoch`, `after_epoch`, `before_iter`, and `after_iter`.

Then we need to make `MyHook` imported. Assuming the file is in `mmrotate/core/utils/my_hook.py` there are two ways to do that:

- Modify `mmrotate/core/utils/__init__.py` to import it.

  The newly defined module should be imported in `mmrotate/core/utils/__init__.py` so that the registry will

```python
```

```python
```

```python
]
```

```python
]
```

### Use hooks implemented in MMCV

]
```

If the hook is already implemented in MMCV, you can directly modify the config to use the hook as below

#### 4. Example: `NumClassCheckHook`

We implement a customized hook named  [NumClassCheckHook](https://github.com/open-mmlab/mmdetection/blob/master/mmdet/datasets/utils.py) to check whether the `num_classes` in head matches the length of `CLASSSES` in `dataset`.

We set it in [default_runtime.py](https://github.com/open-mmlab/mmdetection/blob/master/configs/_base_/default_runtime.py).

```python
```

There are some common hooks that are not registered through `custom_hooks`, they are

In those hooks, only the logger hook has the `VERY_LOW` priority, others' priority are `NORMAL`.
The above-mentioned tutorials already covers how to modify `optimizer_config`, `momentum_config`, and `lr_config`.
Here we reveals how what we can do with `log_config`, `checkpoint_config`, and `evaluation`.

The MMCV runner will use `checkpoint_config` to initialize [`CheckpointHook`](https://github.com/open-mmlab/mmcv/blob/9ecd6b0d5ff9d2172c49a182eaa669e9f27bb8e7/mmcv/runner/hooks/checkpoint.py#L9).

```python
```

```python
```

The users could set `max_keep_ckpts` to only save only small number of checkpoints or decide whether to store state dict of optimizer by `save_optimizer`. More details of the arguments are [here](https://mmcv.readthedocs.io/en/latest/api.html#mmcv.runner.CheckpointHook)

The `log_config` wraps multiple logger hooks and enables to set intervals. Now MMCV supports `WandbLoggerHook`, `MlflowLoggerHook`, and `TensorboardLoggerHook`.
The detail usages can be found in the [doc](https://mmcv.readthedocs.io/en/latest/api.html#mmcv.runner.LoggerHook).

```python
    hooks=[
    ])
```

The config of `evaluation` will be used to initialize the [`EvalHook`](https://github.com/open-mmlab/mmdetection/blob/7a404a2c000620d52156774a5025070d9e00d918/mmdet/core/evaluation/eval_hooks.py#L8).

Except the key `interval`, other arguments such as `metric` will be passed to the `dataset.evaluate()`

```python
```

# 融合预处理（实验性功能）

MMDeploy提供了一些Transform融合的能力，当使用SDK进行推理时，可以通过修改pipeline.json来开启融合选项，在某些Transform的组合下可以对预处理进行加速。

若要在MMDeploy的SDK中加入融合能力，可参考CVFusion的使用。

## 一、使用CVFusion

有两种选择，一种是在编译mmdeploy的时候，使用我们提供的融合kernel代码，一种是自己使用CVFusion生成融合kernel的代码。

A）使用提供的kernel代码

1. 从这里下载代码，并解压，将csrc文件夹拷贝到mmdeploy的根目录。

   [elena_kernel-20220823.tar.gz](https://github.com/open-mmlab/mmdeploy/files/9399795/elena_kernel-20220823.tar.gz)

2. 编译mmdeploy的时候，增加选项`-DMMDEPLOY_ELENA_FUSION=ON`

B) 使用CVFusion生成kernel

1. 编译CVFusion

   ```bash
   $ git clone --recursive https://github.com/OpenComputeLab/CVFusion.git
   $ cd CVFusion
   $ bash build.sh

   # add OpFuse to PATH
   $ export PATH=`pwd`/build/examples/MMDeploy:$PATH
   ```

2. 下载各个算法codebase

   ```bash
   $ tree -L 1 .
   ├── mmdeploy
   ├── mmpretrain
   ├── mmdetection
   ├── mmsegmentation
   ├── ...
   ```

3. 生成融合kernel

   ```bash
   python tools/elena/extract_transform.py ..
   # 生成的代码会保存在csrc/preprocess/elena/{cpu_kernel}/{cuda_kernel}
   ```

   ├── ...
   ```

   ```bash
   ```

4. 编译mmdeploy的时候，增加选项`-DMMDEPLOY_ELENA_FUSION=ON`

## 二、模型转换

模型转换时通过`--dump-info`生成SDK所需文件。

```bash

```

## 三、模型推理

若当前pipeline的预处理模块支持融合，`pipeline.json`中会有`fuse_transform`字段，表示融合开关，默认为`false`。当启用融合算法时，需要把`false`改为`true`

## 背景

CoT（思维链）是帮助大型语言模型解决如数学问题和关系推理问题等复杂问题的有效方式，在OpenCompass中，我们支持多种类型的CoT方法。

## 1. 零样本思维链

可以通过在数据集配置中简单地添加 “Let's think step by step"，来更改数据集配置的 PromptTemplate，从而实现 零样本 CoT prompt 以进行评估：

```python
    ),
)
```

## 2. 小样本思维链

小样本思维链可以使大型语言模型更容易跟随预设的指示并得到更好的答案。对于小样本思维链，按照以下配置将思维链模板添加到 `PromptTemplate` 中，可以创建一个 one-shot prompt：

```python
        template=

'''),
)
```

SC (Self-Consistency) 方法是在 [此文章](https://arxiv.org/abs/2203.11171) 中提出的，该方法会为问题生成多条不同的推理路径，并对生成的答案进行众数投票。这种方法在复杂推理任务中表现出了显著的能力，但由于需要推理多次来采样多条推理链，所以可能会消耗很多的时间和资源。在 OpenCompass 中，您可以通过在数据集配置中将 `GenInferencer` 替换为 `SCInferencer` 并设置相应的参数参数来简单地实现 SC 方法，例如：

```python
# 此SC版gsm8k测试配置可以在： opencompass.configs.datasets.gsm8k.gsm8k_gen_a3e34a.py 中找到。
        type=SCInferencer, # 替换 GenInferencer 为 SCInferencer

```python
        generation_kwargs=dict(do_sample=True, temperature=0.7, top_k=40),  # 设置采样参数以确保模型生成不同的输出，目前仅适用于从HuggingFace加载的模型。
    )
)
```

```{note}
注意，OpenCompass 默认使用 argmax 的方式采样下一个 token，因此若不指定采样参数，模型每次的推理结果将会是完全一致的，多轮评测将会失效。
```

其中 `SAMPLE_SIZE` 是推理路径的数量，较高的值通常会带来更高的性能。SC方法的原论文中展示了不同推理任务间推理路径数量与性能之间的关系：

从图中可以看出，在不同的推理任务中，随着推理路径数量的增加，性能呈现出增长的趋势。但是，对于某些任务，增加推理路径的数量可能达到一个极限，进一步增加推理路径的数量可能不会带来更多的性能提升。因此，需要在具体任务中进行实验和调整，找到最适合任务的推理路径数量。

相比一般的CoT方法采样一条推理路径，ToT(Tree-of-Thoughts)允许语言模型同时考虑多种不同的推理路径，通过对推理过程进行自我评估，以及在必要时进行前瞻或回溯以做出全局选择。具体的，分为下面四个阶段：

**1. 问题分解 (Thought Decomposition)**

根据问题的特点，将问题分解成多个中间步骤。每个步骤可以是短语、算式或写作计划，这取决于问题的性质。

**2. 推理过程生成 (Thought Generation)**

假设解决问题需要k个步骤，有两种方法生成推理内容：

- 独立采样：对于每个状态，模型会独立地从CoT提示中完整抽取k个推理内容，不依赖于其他的推理内容。
- 顺序生成：顺序地使用“提示”来逐步引导推理内容生成，每个推理内容都可能依赖于前一个推理内容。

**3. 启发式评估 (Heuristic Evaluation)**

使用启发式方法评估每个生成的推理内容对问题解决的贡献，这种自我评估基于语言模型的自我反馈，如设计Prompt让模型对多个生成结果进行打分。

**4. 选择搜索算法 (Search Algorithm)**

根据生成和评估推理内容的方法，选择适当的搜索算法。例如，可以使用广度优先搜索（BFS）或深度优先搜索（DFS）等算法来系统地探索思考树，并进行前瞻和回溯。

在OpenCompass中，需要根据需要设置ToT参数，以下是[ToT论文](https://arxiv.org/pdf/2305.10601.pdf)中24点游戏的样例配置，目前支持Huggingface模型进行ToT推理：

```python
# 此 ToT Game24 配置可以在以下路径找到：opencompass/configs/datasets/game24/game24_gen_8dfde3.py。

        template='{input}'), #直接传入input内容，因为Prompt需要分段指定
    inferencer=dict(type=ToTInferencer, # 替换GenInferencer为ToTInferencer
                    method_generate='propose',  # 生成推理内容的方法，可以是独立采样（sample）或顺序生成（propose）
                    method_evaluate='value', # 评估推理内容的方法，可以是投票 （vote）或打分（value）
                    method_select='greedy', # 选择推理内容的方法，可以是贪心（greedy）或随机（sample）
                    task_wrapper=dict(type=Game24PromptWrapper) # 该Wrapper类包含每个步骤的Prompt和推理内容的生成及评估方法，需要根据任务进行自定义
                    ))

```

n_select_sample=5,
                    ))

```

如果要在自定义的数据集上使用ToT方法，相比普通评测方式，需要在`opencompass.datasets.YourDataConfig.py`中额外设置`YourDataPromptWrapper`类，以进行ToT中的推理生成和启发式评估。对于类似游戏24点的推理任务，具体可以参考`opencompass/datasets/game24.py`。

</div>

首先你应该从 https://huggingface.co/datasets/VISION-Workshop/VISION-Datasets 下载数据集，并将其组织为如下格式：

```text
├── mmdet
├── tools
├── data
```

然后你可以使用将以下命令保存为 `vision_unzip.sh` 文件，并将其放置于 `mmdetection` 根目录下，然后 `bash vision_unzip.sh` 运行脚本进行解压处理

```shell

done
```

最终的文件组织格式如下所示：

```shell

done
```

```text
├── mmdet
├── tools
├── data
|   │   |    |── val

|   │   |    |── val
...
```

TODO

```latex
}
```

We provide scripts for multi-modality/single-modality (LiDAR-based/vision-based), indoor/outdoor 3D detection and 3D semantic segmentation demos. The pre-trained models can be downloaded from [model zoo](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/docs/en/model_zoo.md). We provide pre-processed sample data from KITTI, SUN RGB-D, nuScenes and ScanNet dataset. You can use any other data following our pre-processing steps.

## Testing

### 3D Detection

#### Point cloud demo

To test a 3D detector on point cloud data, simply run:

```shell
```

```shell
```

The visualization results including a point cloud and predicted 3D bounding boxes will be saved in `${OUT_DIR}/PCD_NAME`, which you can open using [MeshLab](http://www.meshlab.net/). Note that if you set the flag `--show`, the prediction result will be displayed online using [Open3D](http://www.open3d.org/).

Example on KITTI data using [PointPillars model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-car/hv_pointpillars_secfpn_6x8_160e_kitti-3d-car_20220331_134606-d42d15ed.pth):

```shell
```

```shell
```

Example on SUN RGB-D data using [VoteNet model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/votenet/votenet_16x8_sunrgbd-3d-10class/votenet_16x8_sunrgbd-3d-10class_20210820_162823-bf11f014.pth):

```shell
```

#### Monocular 3D demo

To test a monocular 3D detector on image data, simply run:

```shell
python demo/mono_det_demo.py ${IMAGE_FILE} ${ANNOTATION_FILE} ${CONFIG_FILE} ${CHECKPOINT_FILE} [--device ${GPU_ID}] [--cam-type ${CAM_TYPE}] [--score-thr ${SCORE-THR}] [--out-dir ${OUT_DIR}] [--show]
```

```

where the `ANNOTATION_FILE` should provide the 3D to 2D projection matrix (camera intrinsic matrix), and `CAM_TYPE` should be specified according to dataset. For example, if you want to inference on the front camera image, the `CAM_TYPE` should be set as `CAM_2` for KITTI, and `CAM_FRONT` for nuScenes. By specifying `CAM_TYPE`, you can even infer on any camera images for datasets with multi-view cameras, such as nuScenes and Waymo. `SCORE-THR` is the 3D bbox threshold while visualization. The visualization results including an image and its predicted 3D bounding boxes projected on the image will be saved in `${OUT_DIR}/IMG_NAME`.

Example on KITTI data using [PGD model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pgd/pgd_r101_caffe_fpn_gn-head_3x4_4x_kitti-mono3d/pgd_r101_caffe_fpn_gn-head_3x4_4x_kitti-mono3d_20211022_102608-8a97533b.pth):

```shell

```shell
```

**Note**: For PGD, the prediction score is not among (0, 1).

Example on nuScenes data using [FCOS3D model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_finetune/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_finetune_20210717_095645-8d806dc2.pth):

```shell
```

```

**Note** that when visualizing results of monocular 3D detection for flipped images, the camera intrinsic matrix should also be modified accordingly. See more details and examples in PR [#744](https://github.com/open-mmlab/mmdetection3d/pull/744).

#### Multi-modality demo

To test a 3D detector on multi-modality data (typically point cloud and image), simply run:

```shell
```

where the `ANNOTATION_FILE` should provide the 3D to 2D projection matrix. The visualization results including a point cloud, an image, predicted 3D bounding boxes and their projection on the image will be saved in `${OUT_DIR}/PCD_NAME`.

Example on KITTI data using [MVX-Net model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/mvxnet/mvxnet_fpn_dv_second_secfpn_8xb2-80e_kitti-3d-3class/mvxnet_fpn_dv_second_secfpn_8xb2-80e_kitti-3d-3class-8963258a.pth):

```shell
```

Example on SUN RGB-D data using [ImVoteNet model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/imvotenet/imvotenet_stage2_16x8_sunrgbd-3d-10class/imvotenet_stage2_16x8_sunrgbd-3d-10class_20210819_192851-1bcd1b97.pth):

```shell
```

```

Example on NuScenes data using [BEVFusion model](https://drive.google.com/file/d/1QkvbYDk4G2d6SZoeJqish13qSyXA4lp3/view?usp=share_link):

```shell
```

### 3D Segmentation

To test a 3D segmentor on point cloud data, simply run:

```shell
python demo/pcd_seg_demo.py ${PCD_FILE} ${CONFIG_FILE} ${CHECKPOINT_FILE} [--device ${GPU_ID}] [--out-dir ${OUT_DIR}] [--show]
```

The visualization results including a point cloud and its predicted 3D segmentation mask will be saved in `${OUT_DIR}/PCD_NAME`.

```

Example on ScanNet data using [PointNet++ (SSG) model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/pointnet2/pointnet2_ssg_16x2_cosine_200e_scannet_seg-3d-20class/pointnet2_ssg_16x2_cosine_200e_scannet_seg-3d-20class_20210514_143644-ee73704a.pth):

```shell
```

# SUN RGB-D 数据集

## 数据集的准备

对于数据集准备的整体流程，请参考 SUN RGB-D 的[指南](https://github.com/open-mmlab/mmdetection3d/blob/master/data/sunrgbd/README.md)。

### 下载 SUN RGB-D 数据与工具包

在[这里](http://rgbd.cs.princeton.edu/data/)下载 SUN RGB-D 的数据。接下来，将 `SUNRGBD.zip`、`SUNRGBDMeta2DBB_v2.mat`、`SUNRGBDMeta3DBB_v2.mat` 和 `SUNRGBDtoolbox.zip` 移动到 `OFFICIAL_SUNRGBD` 文件夹，并解压文件。

下载完成后，数据处理之前的文件目录结构如下：

```
sunrgbd
├── matlab
```

### 从原始数据中提取 3D 检测所需数据与标注

通过运行如下指令从原始文件中提取出 SUN RGB-D 的标注（这需要您的机器中安装了 MATLAB）：

```bash
```

主要的步骤包括：

- 提取出训练集和验证集的索引文件；
- 从原始数据中提取出 3D 检测所需要的数据；
- 从原始的标注数据中提取并组织检测任务使用的标注数据。

```

主要的步骤包括：

用于从深度图中提取点云数据的 `extract_rgbd_data_v2.m` 的主要部分如下：

```matlab

% 从深度图获取点云

% MAT 文件比 TXT 文件小三倍。在 Python 中我们可以使用
% scipy.io.loadmat('xxx.mat')['points3d_rgb'] 来加载数据
% 保存点云数据
```

用于提取并组织检测任务标注的 `extract_rgbd_data_v1.m` 的主要部分如下：

```matlab
% 输出 2D 和 3D 包围框

% 保存点云数据
```

```matlab
    centroid = data.groundtruth3DBB(j).centroid;  % 3D 包围框中心
    classname = data.groundtruth3DBB(j).classname;  % 类名
    orientation = data.groundtruth3DBB(j).orientation;  % 3D 包围框方向
    coeffs = abs(data.groundtruth3DBB(j).coeffs);  % 3D 包围框大小
    box2d = data2d.groundtruth2DBB(j).gtBb2D;  % 2D 包围框
end
```

上面的两个脚本调用了 SUN RGB-D 提供的[工具包](https://rgbd.cs.princeton.edu/data/SUNRGBDtoolbox.zip)中的一些函数，如 `read3dPoints`。

使用上述脚本提取数据后，文件目录结构应如下：

```
sunrgbd
├── matlab

end
```

```
sunrgbd
├── matlab
│   ├── calib
│   ├── depth
│   ├── image
│   ├── label
```

在如下每个文件夹下，都有总计 5285 个训练集样本和 5050 个验证集样本：

- `calib`：`.txt` 后缀的相机标定文件。
- `depth`：`.mat` 后缀的点云文件，包含 xyz 坐标和 rgb 色彩值。
- `image`：`.jpg` 后缀的二维图像文件。
- `label`：`.txt` 后缀的用于检测任务的标注数据（版本二）。
- `label_v1`：`.txt` 后缀的用于检测任务的标注数据（版本一）。
- `seg_label`：`.txt` 后缀的用于分割任务的标注数据。

目前，我们使用版本一的数据用于训练与测试，因此版本二的标注并未使用。

### 创建数据集

请运行如下指令创建数据集：

```shell

### 创建数据集

```shell
```

或者，如果使用 slurm，可以使用如下指令替代：

```
```

之前提到的点云数据就会被处理并以 `.bin` 格式重新存储。与此同时，`.pkl` 文件也被生成，用于存储数据标注和元信息。

```
sunrgbd
├── matlab
│   ├── ...
│   ├── ...
│   ├── ...
├── points
```

- `points/xxxxxx.bin`：降采样后的点云数据。
- `sunrgbd_infos_train.pkl`：训练集数据信息（标注与元信息），每个场景所含数据信息具体如下：
  - info\['lidar_points'\]：字典包含了与激光雷达点相关的信息。
  - info\['images'\]：字典包含了与图像数据相关的信息。
    - info\['images'\]\['CAM0'\]\['img_path'\]：图像的文件名。
    - info\['images'\]\['CAM0'\]\['depth2img'\]：深度到图像的变换矩阵，形状为 (4, 4)。

- info\['images'\]：字典包含了与图像数据相关的信息。
    - info\['images'\]\['CAM0'\]\['height'\]：图像的高。
    - info\['images'\]\['CAM0'\]\['width'\]：图像的宽。
  - info\['instances'\]：由字典组成的列表，包含了该帧的所有标注信息。每个字典与单个实例的标注相关。对于其中的第 i 个实例，我们有：
    - info\['instances'\]\[i\]\['bbox_3d'\]：长度为 7 的列表，表示深度坐标系下的 3D 边界框。
    - info\['instances'\]\[i\]\['bbox'\]：长度为 4 的列表，以 (x1, y1, x2, y2) 的顺序表示实例的 2D 边界框。
    - info\['instances'\]\[i\]\['bbox_label_3d'\]：整数表示实例的 3D 标签，-1 表示忽略该类别。
    - info\['instances'\]\[i\]\['bbox_label'\]：整数表示实例的 2D 标签，-1 表示忽略该类别。
- `sunrgbd_infos_val.pkl`：验证集上的数据信息，与 `sunrgbd_infos_train.pkl` 格式完全一致。

## 训练流程

SUN RGB-D 上纯点云 3D 物体检测的典型流程如下：

```python
    dict(
    dict(

dict(
    dict(
    ),
    dict(
    dict(
]
```

点云上的数据增强

- `RandomFlip3D`：随机左右或前后翻转输入点云。
- `GlobalRotScaleTrans`：旋转输入点云，对于 SUN RGB-D 角度通常落入 \[-30, 30\]（度）的范围；并放缩输入点云，对于 SUN RGB-D 比例通常落入 \[0.85, 1.15\] 的范围；最后平移输入点云，对于 SUN RGB-D 通常位移量为 0（即不做位移）。
- `PointSample`：降采样输入点云。

SUN RGB-D 上多模态（点云和图像）3D 物体检测的典型流程如下：

```python
    dict(

```python
    dict(
    dict(
    ),
    dict(
    dict(
]
```

图像上的数据增强

- `Resize`：改变输入图像的大小，`keep_ratio=True` 意味着图像的比例不改变。

type='Pack3DDetInputs',
]
```

图像上的数据增强

- `RandomFlip`：随机地翻折图像。

图像增强的实现取自 [MMDetection](https://github.com/open-mmlab/mmdetection/tree/dev-3.x/mmdet/datasets/transforms)。

## 度量指标

与 ScanNet 一样，通常使用 mAP（全类平均精度）来评估 SUN RGB-D 的检测任务的性能，比如 `mAP@0.25` 和 `mAP@0.5`。具体来说，评估时调用一个通用的计算 3D 物体检测多个类别的精度和召回率的函数。更多细节请参考 [`indoor_eval.py`](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/evaluation/functional/indoor_eval.py)。

因为 SUN RGB-D 包含有图像数据，所以图像上的物体检测也是可行的。举个例子，在 ImVoteNet 中，我们首先训练了一个图像检测器，并且也使用 mAP 指标，如 `mAP@0.5`，来评估其表现。我们使用 [MMDetection](https://github.com/open-mmlab/mmdetection) 库中的 `eval_map` 函数来计算 mAP。

| [pose_hrnet_w32](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192.py) |  256x192   | 0.749 |      0.906      |      0.821      | 0.804 |      0.945      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192-81c58e40_20220909.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192_20220909.log) |
| [pose_hrnet_w32](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-384x288.py) |  384x288   | 0.761 |      0.908      |      0.826      | 0.811 |      0.944      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-384x288-ca5956af_20220909.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-384x288_20220909.log) |

| [pose_hrnet_w48](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_8xb32-210e_coco-256x192.py) |  256x192   | 0.756 |      0.908      |      0.826      | 0.809 |      0.945      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_8xb32-210e_coco-256x192-0e67c616_20220913.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_8xb32-210e_coco-256x192_20220913.log) |
| [pose_hrnet_w48](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_8xb32-210e_coco-384x288.py) |  384x288   | 0.767 |      0.911      |      0.832      | 0.817 |      0.947      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_8xb32-210e_coco-384x288-c161b7de_20220915.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_8xb32-210e_coco-384x288_20220915.log) |

# 如何 Profile 模型

模型转换结束后，MMDeploy 提供了 `tools/test.py` 做为单测工具。

## 依赖

需要参照 [安装说明](../01-how-to-build/build_from_source.md) 完成依赖安装
，按照 [转换说明](../02-how-to-run/convert_model.md) 转出模型。

## 用法

```shell
[--log2file ${LOG_RESULT_TO_FILE}]
```

## 参数详解

| 参数         | 说明                      |
| ------------ | ------------------------- |
| deploy_cfg   | 部署配置文件              |
| model_cfg    | codebase 中的模型配置文件 |
| log2file     | 保存日志和运行文件的路径  |
| speed-test   | 是否做速度测试            |
| warm-up      | 执行前是否 warm-up        |
| log-interval | 日志打印间隔              |

## 使用样例

执行模型推理

```shell
```

profile 速度测试

```shell

{MMPRETRAIN_DIR}/configs/resnet/resnet50_b32x8_imagenet.py \
```

```shell
    --speed-test \
```

# BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation

> [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://arxiv.org/abs/2205.13542)

Multi-sensor fusion is essential for an accurate and reliable autonomous driving system. Recent approaches are based on point-level fusion: augmenting the LiDAR point cloud with camera features. However, the camera-to-LiDAR projection throws away the semantic density of camera features, hindering the effectiveness of such methods, especially for semantic-oriented tasks (such as 3D scene segmentation). In this paper, we break this deeply-rooted convention with BEVFusion, an efficient and generic multi-task multi-sensor fusion framework. It unifies multi-modal features in the shared bird's-eye view (BEV) representation space, which nicely preserves both geometric and semantic information. To achieve this, we diagnose and lift key efficiency bottlenecks in the view transformation with optimized BEV pooling, reducing latency by more than 40x. BEVFusion is fundamentally task-agnostic and seamlessly supports different 3D perception tasks with almost no architectural changes. It establishes the new state of the art on nuScenes, achieving 1.3% higher mAP and NDS on 3D object detection and 13.6% higher mIoU on BEV map segmentation, with 1.9x lower computation cost. Code to reproduce our

results is available at https://github.com/mit-han-lab/bevfusion.

<img src="https://user-images.githubusercontent.com/34888372/215313913-4b43f8a1-e2e2-49ba-b631-992155351922.png" width="800"/>
</div>

We implement BEVFusion and support training and testing on NuScenes dataset.

## Usage

### Compiling operations on CUDA

**Note** that the voxelization OP in the original implementation of `BEVFusion` is different from the implementation in MMCV. If you want to use the original pretrained model [here](https://github.com/mit-han-lab/bevfusion/blob/main/README.md), you need to use the original implementation of voxelization OP.

```python
python projects/BEVFusion/setup.py develop
```

### Demo

```python
```

### Demo

Run a demo on NuScenes data using [BEVFusion model](https://drive.google.com/file/d/1QkvbYDk4G2d6SZoeJqish13qSyXA4lp3/view?usp=share_link):

```shell
python demo/multi_modality_demo.py demo/data/nuscenes/n015-2018-07-24-11-22-45+0800__LIDAR_TOP__1532402927647951.pcd.bin demo/data/nuscenes/ demo/data/nuscenes/n015-2018-07-24-11-22-45+0800.pkl projects/BEVFusion/configs/bevfusion_lidar-cam_voxel0075_second_secfpn_8xb4-cyclic-20e_nus-3d.py ${CHECKPOINT_FILE} --cam-type all --score-thr 0.2 --show
```

1. You should train the lidar-only detector first:

```bash
bash tools/dist_train.py projects/BEVFusion/configs/bevfusion_lidar_voxel0075_second_secfpn_8xb4-cyclic-20e_nus-3d.py 8
```

```bash
```

2. Download the [Swin pre-trained model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/bevfusion/swint-nuimages-pretrained.pth). Given the image pre-trained backbone and the lidar-only pre-trained detector, you could train the lidar-camera fusion model:

```bash
bash tools/dist_train.sh projects/BEVFusion/configs/bevfusion_lidar-cam_voxel0075_second_secfpn_8xb4-cyclic-20e_nus-3d.py 8 --cfg-options load_from=${LIDAR_PRETRAINED_CHECKPOINT} model.img_backbone.init_cfg.checkpoint=${IMAGE_PRETRAINED_BACKBONE}
```

**Note** that if you want to reduce CUDA memory usage and computational overhead, you could directly add `--amp` on the tail of the above commands. The model under this setting will be trained in fp16 mode.

```bash
bash tools/dist_test.sh projects/BEVFusion/configs/bevfusion_lidar-cam_voxel0075_second_secfpn_8xb4-cyclic-20e_nus-3d.py ${CHECKPOINT_PATH} 8
```

|                                           Modality                                           | Voxel type (voxel size) | NMS | Mem (GB) | Inf time (fps) | NDS  | mAP  |                                                                                                                                                             Download                                                                                                                                                              |

| :------------------------------------------------------------------------------------------: | :---------------------: | :-: | :------: | :------------: | :--: | :--: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|     [lidar](./configs/bevfusion_lidar_voxel0075_second_secfpn_8xb4-cyclic-20e_nus-3d.py)     |      voxel (0.075)      |  ×  |    -     |       -        | 69.6 | 64.9 |     [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/bevfusion/bevfusion_lidar_voxel0075_second_secfpn_8xb4-cyclic-20e_nus-3d-2628f933.pth) [logs](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/bevfusion/bevfusion_lidar_voxel0075_second_secfpn_8xb4-cyclic-20e_nus-3d_20230322_053447.log)     |

| [lidar-cam](./configs/bevfusion_lidar-cam_voxel0075_second_secfpn_8xb4-cyclic-20e_nus-3d.py) |      voxel (0.075)      |  ×  |    -     |       -        | 71.4 | 68.6 | [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/bevfusion/bevfusion_lidar-cam_voxel0075_second_secfpn_8xb4-cyclic-20e_nus-3d-5239b1af.pth) [logs](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/bevfusion/bevfusion_lidar-cam_voxel0075_second_secfpn_8xb4-cyclic-20e_nus-3d_20230524_001539.log) |

```latex
@inproceedings{liu2022bevfusion,
  title={BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation},
  author={Liu, Zhijian and Tang, Haotian and Amini, Alexander and Yang, Xingyu and Mao, Huizi and Rus, Daniela and Han, Song},
  booktitle={IEEE International Conference on Robotics and Automation (ICRA)},
}
```

booktitle={IEEE International Conference on Robotics and Automation (ICRA)},
}
```

# 支持的 RKNN 特征

目前, MMDeploy 只在 rk3588 和 rv1126 的 linux 平台上测试过.

以下特性需要手动在 MMDeploy 自行配置，如[这里](https://github.com/open-mmlab/mmdeploy/tree/main/configs/_base_/backends/rknn.py).

- target_platform ！= default
- optimization level ！= 1

### Prepare ScanNet Data for Indoor Detection or Segmentation Task

We follow the procedure in [votenet](https://github.com/facebookresearch/votenet/).

1. Download ScanNet v2 data [HERE](https://github.com/ScanNet/ScanNet). Link or move the 'scans' folder to this level of directory. If you are performing segmentation tasks and want to upload the results to its official [benchmark](http://kaldir.vc.in.tum.de/scannet_benchmark/), please also link or move the 'scans_test' folder to this directory.

2. In this directory, extract point clouds and annotations by running `python batch_load_scannet_data.py`. Add the `--max_num_point 50000` flag if you only use the ScanNet data for the detection task. It will downsample the scenes to less points.

3. In this directory, extract RGB image with poses by running `python extract_posed_images.py`. This step is optional. Skip it if you don't plan to use multi-view RGB images. Add `--max-images-per-scene -1` to disable limiting number of images per scene. ScanNet scenes contain up to 5000+ frames per each. After extraction, all the .jpg images require 2 Tb disk space. The recommended 300 images per scene require less then 100 Gb. For example multi-view 3d detector ImVoxelNet samples 50 and 100 images per training and test scene.

4. Enter the project root directory, generate training data by running

```bash
python tools/create_data.py scannet --root-path ./data/scannet --out-dir ./data/scannet --extra-tag scannet
```

```bash
cd ../..
```

```bash
cd ../..
```

```
scannet
├── scans
├── points

```

# Probabilistic and Geometric Depth: Detecting Objects in Perspective

> [Probabilistic and Geometric Depth: Detecting Objects in Perspective](https://arxiv.org/abs/2107.14160)

3D object detection is an important capability needed in various practical applications such as driver assistance systems. Monocular 3D detection, as a representative general setting among image-based approaches, provides a more economical solution than conventional settings relying on LiDARs but still yields unsatisfactory results. This paper first presents a systematic study on this problem. We observe that the current monocular 3D detection can be simplified as an instance depth estimation problem: The inaccurate instance depth blocks all the other 3D attribute predictions from improving the overall detection performance. Moreover, recent methods directly estimate the depth based on isolated instances or pixels while ignoring the geometric relations across different objects. To this end, we construct geometric relation graphs across predicted objects and use the graph to facilitate depth estimation. As the preliminary depth estimation of each instance is usually inaccurate in this ill-posed setting, we incorporate a probabilistic representation to capture the uncertainty. It provides an important indicator to identify confident predictions and further guide the depth propagation. Despite the simplicity of the basic idea, our method, PGD, obtains significant improvements on KITTI and nuScenes benchmarks, achieving 1st place out of all monocular vision-only methods while still maintaining real-time efficiency. Code and models will be released at [this https URL](https://github.com/open-mmlab/mmdetection3d).

<img src="https://user-images.githubusercontent.com/79644370/143884065-d1a19fdf-bcc0-4249-84cf-b7a85fa1eb2f.png" width="800"/>
</div>

PGD, also can be regarded as FCOS3D++, is a simple yet effective monocular 3D detector. It enhances the FCOS3D baseline by involving local geometric constraints and improving instance depth estimation.

We release the code and model for both KITTI and nuScenes benchmark, which is a good supplement for the original FCOS3D baseline (only supported on nuScenes).

For clean implementation, our preliminary release supports base models with proposed local geometric constraints and the probabilistic depth representation. We will involve the geometric graph part in the future.

A more extensive study based on FCOS3D and PGD is on-going. Please stay tuned.

### KITTI

### KITTI

|                             Backbone                              | Lr schd | Mem (GB) | Inf time (fps) | mAP_11 / mAP_40 |                                                                                                                                                                                              Download                                                                                                                                                                                              |

| [ResNet101](./pgd_r101-caffe_fpn_head-gn_4xb3-4x_kitti-mono3d.py) |   4x    |   9.07   |                |  18.33 / 13.23  | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pgd/pgd_r101_caffe_fpn_gn-head_3x4_4x_kitti-mono3d/pgd_r101_caffe_fpn_gn-head_3x4_4x_kitti-mono3d_20211022_102608-8a97533b.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pgd/pgd_r101_caffe_fpn_gn-head_3x4_4x_kitti-mono3d/pgd_r101_caffe_fpn_gn-head_3x4_4x_kitti-mono3d_20211022_102608.log.json) |

| Car (AP11) | 24.09 / 30.11 | 18.33 / 23.46 | 16.90 / 19.33 |
| Car (AP40) | 19.27 / 26.60 | 13.23 / 18.23 | 10.65 / 15.00 |

Note: mAP represents Car moderate 3D strict AP11 / AP40 results. Because of the limited data for pedestrians and cyclists, the detection performance for these two classes is usually unstable. Therefore, we only list car detection results here. In addition, AP40 is a more recommended metric for reference due to its much better stability.

|                                     Backbone                                      | Lr schd | Mem (GB) | mAP  | NDS  |                                                                                                                                                                                                              Download                                                                                                                                                                                                              |

|      [ResNet101 w/ DCN](./pgd_r101-caffe_fpn_head-gn_16xb2-1x_nus-mono3d.py)      |   1x    |   9.20   | 31.7 | 39.3 |                   [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pgd/pgd_r101_caffe_fpn_gn-head_2x16_1x_nus-mono3d/pgd_r101_caffe_fpn_gn-head_2x16_1x_nus-mono3d_20211116_195350-f4b5eec2.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pgd/pgd_r101_caffe_fpn_gn-head_2x16_1x_nus-mono3d/pgd_r101_caffe_fpn_gn-head_2x16_1x_nus-mono3d_20211116_195350.log.json)                   |

| [above w/ finetune](./pgd_r101-caffe_fpn_head-gn_16xb2-1x_nus-mono3d_finetune.py) |   1x    |   9.20   | 34.6 | 41.1 | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pgd/pgd_r101_caffe_fpn_gn-head_2x16_1x_nus-mono3d_finetune/pgd_r101_caffe_fpn_gn-head_2x16_1x_nus-mono3d_finetune_20211118_093245-fd419681.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pgd/pgd_r101_caffe_fpn_gn-head_2x16_1x_nus-mono3d_finetune/pgd_r101_caffe_fpn_gn-head_2x16_1x_nus-mono3d_finetune_20211118_093245.log.json) |

|                                   above w/ tta                                    |   1x    |   9.20   | 35.5 | 41.8 |                                                                                                                                                                                                                                                                                                                                                                                                                                    |

|      [ResNet101 w/ DCN](./pgd_r101-caffe_fpn_head-gn_16xb2-2x_nus-mono3d.py)      |   2x    |   9.20   | 33.6 | 40.9 |                   [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pgd/pgd_r101_caffe_fpn_gn-head_2x16_2x_nus-mono3d/pgd_r101_caffe_fpn_gn-head_2x16_2x_nus-mono3d_20211112_125314-cb677266.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pgd/pgd_r101_caffe_fpn_gn-head_2x16_2x_nus-mono3d/pgd_r101_caffe_fpn_gn-head_2x16_2x_nus-mono3d_20211112_125314.log.json)                   |

| [above w/ finetune](./pgd_r101-caffe_fpn_head-gn_16xb2-2x_nus-mono3d_finetune.py) |   2x    |   9.20   | 35.8 | 42.5 | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pgd/pgd_r101_caffe_fpn_gn-head_2x16_2x_nus-mono3d_finetune/pgd_r101_caffe_fpn_gn-head_2x16_2x_nus-mono3d_finetune_20211114_162135-5ec7c1cd.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pgd/pgd_r101_caffe_fpn_gn-head_2x16_2x_nus-mono3d_finetune/pgd_r101_caffe_fpn_gn-head_2x16_2x_nus-mono3d_finetune_20211114_162135.log.json) |

|                                   above w/ tta                                    |   2x    |   9.20   | 36.8 | 43.1 |                                                                                                                                                                                                                                                                                                                                                                                                                                    |

```latex
@inproceedings{wang2021pgd,
    title={{Probabilistic and Geometric Depth: Detecting} Objects in Perspective},
    author={Wang, Tai and Zhu, Xinge and Pang, Jiangmiao and Lin, Dahua},
    booktitle={Conference on Robot Learning (CoRL) 2021},
}
# For the baseline version
    title={{FCOS3D: Fully} Convolutional One-Stage Monocular 3D Object Detection},

}
    booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops},
}
```

# PIDNet

> [PIDNet: A Real-time Semantic Segmentation Network Inspired from PID Controller](https://arxiv.org/pdf/2206.02066.pdf)

<a href="https://github.com/XuJiacong/PIDNet">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/models/backbones/pidnet.py">Code Snippet</a>

Two-branch network architecture has shown its efficiency and effectiveness for real-time semantic segmentation tasks. However, direct fusion of low-level details and high-level semantics will lead to a phenomenon that the detailed features are easily overwhelmed by surrounding contextual information, namely overshoot in this paper, which limits the improvement of the accuracy of existed two-branch models. In this paper, we bridge a connection between Convolutional Neural Network (CNN) and Proportional-IntegralDerivative (PID) controller and reveal that the two-branch network is nothing but a Proportional-Integral (PI) controller, which inherently suffers from the similar overshoot issue. To alleviate this issue, we propose a novel threebranch network architecture: PIDNet, which possesses three branches to parse the detailed, context and boundary information (derivative of semantics), respectively, and employs boundary attention to guide the fusion of detailed and context branches in final stage. The family of PIDNets achieve the best trade-off between inference speed and accuracy and their test accuracy surpasses all the existed models with similar inference speed on Cityscapes, CamVid and COCO-Stuff datasets. Especially, PIDNet-S achieves 78.6% mIOU with inference speed of 93.2 FPS on Cityscapes test set and 80.1% mIOU with speed of 153.7 FPS on CamVid test set.

<img src="https://raw.githubusercontent.com/XuJiacong/PIDNet/main/figs/pidnet.jpg" width="800"/>
</div>

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device | mIoU  | mIoU(ms+flip) | config                                                                                                                     | download                                                                                                                                                                                                                                                                                                                                                 |

| PIDNet | PIDNet-S | 1024x1024 | 120000  | 3.38     | 80.82          | A100   | 78.74 | 80.87         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pidnet/pidnet-s_2xb6-120k_1024x1024-cityscapes.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pidnet/pidnet-s_2xb6-120k_1024x1024-cityscapes/pidnet-s_2xb6-120k_1024x1024-cityscapes_20230302_191700-bb8e3bcc.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pidnet/pidnet-s_2xb6-120k_1024x1024-cityscapes/pidnet-s_2xb6-120k_1024x1024-cityscapes_20230302_191700.json) |

| PIDNet | PIDNet-M | 1024x1024 | 120000  | 5.14     | 71.98          | A100   | 80.22 | 82.05         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pidnet/pidnet-m_2xb6-120k_1024x1024-cityscapes.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pidnet/pidnet-m_2xb6-120k_1024x1024-cityscapes/pidnet-m_2xb6-120k_1024x1024-cityscapes_20230301_143452-f9bcdbf3.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pidnet/pidnet-m_2xb6-120k_1024x1024-cityscapes/pidnet-m_2xb6-120k_1024x1024-cityscapes_20230301_143452.json) |

| PIDNet | PIDNet-L | 1024x1024 | 120000  | 5.83     | 60.06          | A100   | 80.89 | 82.37         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/pidnet/pidnet-l_2xb6-120k_1024x1024-cityscapes.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/pidnet/pidnet-l_2xb6-120k_1024x1024-cityscapes/pidnet-l_2xb6-120k_1024x1024-cityscapes_20230303_114514-0783ca6b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/pidnet/pidnet-l_2xb6-120k_1024x1024-cityscapes/pidnet-l_2xb6-120k_1024x1024-cityscapes_20230303_114514.json) |

## Notes

The pretrained weights in config files are converted from [the official repo](https://github.com/XuJiacong/PIDNet#models).

```bibtex
@misc{xu2022pidnet,
      title={PIDNet: A Real-time Semantic Segmentation Network Inspired from PID Controller},
      author={Jiacong Xu and Zixiang Xiong and Shankar P. Bhattacharyya},
      eprint={2206.02066},

author={Jiacong Xu and Zixiang Xiong and Shankar P. Bhattacharyya},
}
```

# MiniGPT4

> [MiniGPT-4: Enhancing Vision-language Understanding with Advanced Large Language Models](https://arxiv.org/abs/2304.10592)

The recent GPT-4 has demonstrated extraordinary multi-modal abilities, such as directly generating websites from handwritten text and identifying humorous elements within images. These features are rarely observed in previous vision-language models. We believe the primary reason for GPT-4's advanced multi-modal generation capabilities lies in the utilization of a more advanced large language model (LLM). To examine this phenomenon, we present MiniGPT-4, which aligns a frozen visual encoder with a frozen LLM, Vicuna, using just one projection layer. Our findings reveal that MiniGPT-4 possesses many capabilities similar to those exhibited by GPT-4 like detailed image description generation and website creation from hand-written drafts. Furthermore, we also observe other emerging capabilities in MiniGPT-4, including writing stories and poems inspired by given images, providing solutions to problems shown in images, teaching users how to cook based on food photos, etc. In our experiment, we found that only performing the pretraining on raw image-text pairs could produce unnatural language outputs that lack coherency including repetition and fragmented sentences. To address this problem, we curate a high-quality, well-aligned dataset in the second stage to finetune our model using a conversational template. This step proved crucial for augmenting the model's generation reliability and overall usability. Notably, our model is highly computationally efficient, as we only train a projection layer utilizing approximately 5 million aligned image-text pairs. Our code, pre-trained model, and collected dataset are available at https://minigpt-4.github.io/.

<img src="https://github.com/open-mmlab/mmpretrain/assets/36138628/1d8f328d-6c91-493e-8992-29e84a0fc3c8" width="80%"/>
</div>

```python

result = inference_model('minigpt-4_vicuna-7b_caption', 'demo/cat-dog.png')
# {'pred_caption': 'This image shows a small dog and a kitten sitting on a blanket in a field of flowers. The dog is looking up at the kitten with a playful expression on its face. The background is a colorful striped blanket, and there are flowers all around them. The image is well composed with the two animals sitting in the center of the frame, surrounded by the flowers and blanket.'}
```

For Vicuna model, please refer to [MiniGPT-4 page](https://github.com/Vision-CAIR/MiniGPT-4) for preparation guidelines.

```

| Model                           | Params (M) | Flops (G) |                  Config                  |                                                    Download                                                    |
| :------------------------------ | :--------: | :-------: | :--------------------------------------: | :------------------------------------------------------------------------------------------------------------: |
| `minigpt-4_vicuna-7b_caption`\* |  8121.32   |    N/A    | [config](minigpt-4_vicuna-7b_caption.py) | [model](https://download.openmmlab.com/mmpretrain/v1.0/minigpt4/minigpt-4_linear-projection_20230615-714b5f52.pth) |

*Models with * are converted from the [official repo](https://github.com/Vision-CAIR/MiniGPT-4/tree/main). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@article{zhu2023minigpt,
  title={MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models},
  author={Zhu, Deyao and Chen, Jun and Shen, Xiaoqian and Li, Xiang and Elhoseiny, Mohamed},
  journal={arXiv preprint arXiv:2304.10592},
}
```

# 安装

## 环境依赖

- PyTorch 1.6 或更高版本
- CUDA 10.1
- NCCL 2
- GCC 5.4.0 或更高版本

## 准备环境

```{note}
如果你已经在本地安装了 PyTorch，请直接跳转到[安装步骤](#安装步骤)。
```

**第一步** 下载并安装 [Miniconda](https://docs.conda.io/en/latest/miniconda.html).

**第二步** 创建并激活一个 conda 环境：

```shell
```

**第三步** 依照[官方指南](https://pytorch.org/get-started/locally/)，安装 PyTorch。

```{code-tab} shell GPU 平台
```

```{code-tab} shell CPU 平台
```

````

## 安装步骤

我们建议大多数用户采用我们的推荐方式安装 MMOCR。倘若你需要更灵活的安装过程，则可以参考[自定义安装](#自定义安装)一节。

### 推荐步骤

**第一步** 使用 [MIM](https://github.com/open-mmlab/mim) 安装 [MMEngine](https://github.com/open-mmlab/mmengine)， [MMCV](https://github.com/open-mmlab/mmcv) 和 [MMDetection](https://github.com/open-mmlab/mmdetection)。

```shell

```shell
```

**第二步** 安装 MMOCR.

若你需要直接运行 MMOCR 或在其基础上进行开发，则通过源码安装（推荐）。

如果你将 MMOCR 作为一个外置依赖库使用，则可以通过 MIM 安装。

````{group-tab} 源码安装

```shell
cd mmocr
# "-v" 会让安装过程产生更详细的输出
# "-e" 会以可编辑的方式安装该代码库，你对该代码库所作的任何更改都会立即生效
```

````

````{group-tab} MIM 安装

```shell

```

````

`````

**第三步（可选）** 如果你需要使用与 `albumentations` 有关的变换（如 ABINet 数据流水线中的 `Albu`），或需要构建文档、运行单元测试的依赖，请使用以下命令安装依赖：

```shell
# 安装 albu
# 安装文档、测试等依赖
```

````

```shell
```

````

`````

```{note}

```

````

```shell
```

````

`````

```{note}

我们建议在安装 `albumentations` 之后检查当前环境，确保 `opencv-python` 和 `opencv-python-headless` 没有同时被安装，否则有可能会产生一些无法预知的错误。如果它们不巧同时存在于环境当中，请卸载 `opencv-python-headless` 以确保 MMOCR 的可视化工具可以正常运行。

查看 [`albumentations` 的官方文档](https://albumentations.ai/docs/getting_started/installation/#note-on-opencv-dependencies)以获知详情。

```

### 检验

你可以通过运行一个简单的推理任务来检验 MMOCR 的安装是否成功。

在 Python 中运行以下代码：

```python
```
````

如果你是通过源码安装的 MMOCR，你可以在 MMOCR 的根目录下运行以下命令：

```shell
```
````

`````

若 MMOCR 的安装无误，你在这一节完成后应当能看到以图片和文字形式表示的识别结果：

```shell
```
````

`````

</div>
<br/ >

```bash
# 识别结果
[...]}]}
```

```{note}
```

## 自定义安装

### CUDA 版本

```{note}
```

要使用 pip 而不是 MIM 来安装 MMCV，请遵照 [MMCV 安装指南](https://mmcv.readthedocs.io/zh_CN/latest/get_started/installation.html)。
它需要你用指定 url 的形式手动指定对应的 PyTorch 和 CUDA 版本。

举个例子，如下命令将会安装基于 PyTorch 1.10.x 和 CUDA 11.3 编译的 mmcv-full。

```shell

```shell
pip install 'mmcv>=2.0.0rc1' -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10/index.html
```

MMOCR 可以仅在 CPU 环境中安装，在 CPU 模式下，你可以完成训练（需要 MMCV 版本 >= 1.4.4）、测试和模型推理等所有操作。

在 CPU 模式下，MMCV 中的以下算子将不可用：

如果你尝试使用用到了以上算子的模型进行训练、测试或推理，程序将会报错。以下为可能受到影响的模型列表：

|                          算子                           |                          模型                           |

### 通过 Docker 使用 MMOCR

我们提供了一个 [Dockerfile](https://github.com/open-mmlab/mmocr/blob/master/docker/Dockerfile) 文件以建立 docker 镜像 。

```shell
```

使用以下命令运行。

```shell
docker run --gpus all --shm-size=8g -it -v {实际数据目录}:/mmocr/data mmocr
```

## 对 MMEngine、MMCV 和 MMDetection 的版本依赖

为了确保代码实现的正确性，MMOCR 每个版本都有可能改变对 MMEngine、MMCV 和 MMDetection 版本的依赖。请根据以下表格确保版本之间的相互匹配。

}

```

<details>

```bibtex
}
```

<details>

```bibtex

| [rtmpose-m](/configs/wholebody_2d_keypoint/rtmpose/coco-wholebody/rtmpose-m_8xb64-270e_coco-wholebody-256x192.py) |  256x192   |  0.673  |  0.750  |  0.615  |  0.752  |  0.813  |  0.871  |  0.475  |  0.589  |  0.582   |  0.674   | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-coco-wholebody_pt-aic-coco_270e-256x192-cd5e845c_20230123.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-coco-wholebody_pt-aic-coco_270e-256x192-cd5e845c_20230123.json) |

| [rtmpose-l](/configs/wholebody_2d_keypoint/rtmpose/coco-wholebody/rtmpose-l_8xb64-270e_coco-wholebody-256x192.py) |  256x192   |  0.695  |  0.769  |  0.658  |  0.785  |  0.833  |  0.887  |  0.519  |  0.628  |  0.611   |  0.700   | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-coco-wholebody_pt-aic-coco_270e-256x192-6f206314_20230124.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-coco-wholebody_pt-aic-coco_270e-256x192-6f206314_20230124.json) |

| [rtmpose-l](/configs/wholebody_2d_keypoint/rtmpose/coco-wholebody/rtmpose-l_8xb32-270e_coco-wholebody-384x288.py) |  384x288   |  0.712  |  0.781  |  0.693  |  0.811  |  0.882  |  0.919  |  0.579  |  0.677  |  0.648   |  0.730   | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-coco-wholebody_pt-aic-coco_270e-384x288-eaeb96c8_20230125.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-coco-wholebody_pt-aic-coco_270e-384x288-eaeb96c8_20230125.json) |

# SEResNet

> [Squeeze-and-Excitation Networks](https://openaccess.thecvf.com/content_cvpr_2018/html/Hu_Squeeze-and-Excitation_Networks_CVPR_2018_paper.html)

The central building block of convolutional neural networks (CNNs) is the convolution operator, which enables networks to construct informative features by fusing both spatial and channel-wise information within local receptive fields at each layer. A broad range of prior research has investigated the spatial component of this relationship, seeking to strengthen the representational power of a CNN by enhancing the quality of spatial encodings throughout its feature hierarchy. In this work, we focus instead on the channel relationship and propose a novel architectural unit, which we term the "Squeeze-and-Excitation" (SE) block, that adaptively recalibrates channel-wise feature responses by explicitly modelling interdependencies between channels. We show that these blocks can be stacked together to form SENet architectures that generalise extremely effectively across different datasets. We further demonstrate that SE blocks bring significant improvements in performance for existing state-of-the-art CNNs at slight additional computational cost. Squeeze-and-Excitation Networks formed the foundation of our ILSVRC 2017 classification submission which won first place and reduced the top-5 error to 2.251%, surpassing the winning entry of 2016 by a relative improvement of ~25%.

<img src="https://user-images.githubusercontent.com/26739999/142574668-3464d087-b962-48ba-ad1d-5d6b33c3ba0b.png" width="50%"/>
</div>

```python

predict = inference_model('seresnet50_8xb32_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('seresnet50_8xb32_in1k', pretrained=True)
```

Train:

```shell
python tools/train.py configs/seresnet/seresnet50_8xb32_in1k.py
```

Test:

```shell

Train:

```shell
```

Test:

```shell
python tools/test.py configs/seresnet/seresnet50_8xb32_in1k.py https://download.openmmlab.com/mmclassification/v0/se-resnet/se-resnet50_batch256_imagenet_20200804-ae206104.pth
```

| Model                    |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |               Config                |                                           Download                                           |
| :----------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :---------------------------------: | :------------------------------------------------------------------------------------------: |

| `seresnet50_8xb32_in1k`  | From scratch |   28.09    |   4.13    |   77.74   |   93.84   | [config](seresnet50_8xb32_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/se-resnet/se-resnet50_batch256_imagenet_20200804-ae206104.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/se-resnet/se-resnet50_batch256_imagenet_20200708-657b3c36.log.json) |
| `seresnet101_8xb32_in1k` | From scratch |   49.33    |   7.86    |   78.26   |   94.07   | [config](seresnet101_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/se-resnet/se-resnet101_batch256_imagenet_20200804-ba5b51d4.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/se-resnet/se-resnet101_batch256_imagenet_20200708-038a4d04.log.json) |

```bibtex

```bibtex
}
```

# Preparing CelebA-HQ Dataset

```bibtex
}
```

Follow the instructions [here](https://github.com/tkarras/progressive_growing_of_gans#preparing-datasets-for-training) to prepare the dataset.

```text
mmagic
├── mmagic
├── tools
├── data

```

# Welcome to Projects of MMPose

Hey there! This is the place for you to contribute your awesome keypoint detection techniques to MMPose!

We know the unit tests in core package can be a bit intimidating, so we've made it easier and more efficient for you to implement your algorithms here.

And the **best part**?

- Projects in this folder are designed to be **easier to merge**!

- Projects in this folder are **NOT** strictly required for **writing unit tests**!

- We want to make it **as painless as possible** for you to contribute and make MMPose even greater.

If you're not sure where to start, check out our [example project](./example_project) to see how to add your algorithms easily. And if you have any questions, take a look at our [FAQ](./faq.md).

We also provide some documentation listed below to help you get started:

- [New Model Guide](https://mmpose.readthedocs.io/en/latest/guide_to_framework.html#step3-model)

  A guide to help you add new models to MMPose.

- [Contribution Guide](https://mmpose.readthedocs.io/en/latest/contribution_guide.html)

  A guide for new contributors on how to add their projects to MMPose.

- [Discussions](https://github.com/open-mmlab/mmpose/discussions)

  We encourage you to start a discussion and share your ideas!

## Project List

- **[:zap:RTMPose](./rtmpose)**: Real-Time Multi-Person Pose Estimation toolkit based on MMPose

  <img src="https://github.com/open-mmlab/mmpose/assets/13503330/5b637d76-41dd-4376-9a7f-854cd120799d" width=800 height=200 />

- **[:art:MMPose4AIGC](./mmpose4aigc)**: Guide AI image generation with MMPose

  <img src="https://user-images.githubusercontent.com/13503330/222403836-c65ba905-4bdd-4a44-834c-ff8d5959649d.png" width="800"/>

- **[:bulb:YOLOX-Pose](./yolox-pose)**: Enhancing YOLO for Multi Person Pose Estimation Using Object Keypoint Similarity Loss

  <img src="https://user-images.githubusercontent.com/26127467/226655503-3cee746e-6e42-40be-82ae-6e7cae2a4c7e.jpg" width="800" style="width: 800px; height: 200px; object-fit: cover"/>

- **[📖Awesome MMPose](./awesome-mmpose/)**: A list of Tutorials, Papers, Datasets related to MMPose

  <img src="https://user-images.githubusercontent.com/13503330/231416285-5467d313-0732-4ada-97e1-12be6ec69a28.png" width="800"/>

- **What's next? Join the rank of <span style="color:blue"> *MMPose contributors* </span> by creating a new  project**!

# Swin Transformer

> [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://arxiv.org/abs/2103.14030)

<a href="https://github.com/microsoft/Swin-Transformer">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/backbones/swin.py#L524">Code Snippet</a>

This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with Shifted windows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it compatible with a broad range of vision tasks, including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation (53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-the-art by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones. The hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures. The code and models are publicly available at [this https URL](https://github.com/microsoft/Swin-Transformer).

<img src="https://user-images.githubusercontent.com/24582831/142902882-3fb9014c-11b6-47e9-aa14-500dfe7cbb1c.png" width="80%"/>
</div>

## Usage

We have provided pretrained models converted from [official repo](https://github.com/microsoft/Swin-Transformer)．

If you want to convert keys on your own to use official repositories' pre-trained models, we also provide a script [`swin2mmseg.py`](../../tools/model_converters/swin2mmseg.py) in the tools directory to convert the key of models from [the official repo](https://github.com/SwinTransformer/Swin-Transformer-Semantic-Segmentation) to MMSegmentation style.

```shell
python tools/model_converters/swin2mmseg.py ${PRETRAIN_PATH} ${STORE_PATH}
```

E.g.

```shell
python tools/model_converters/swin2mmseg.py https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224.pth pretrain/swin_base_patch4_window7_224.pth
```

```shell
```

In our default setting, pretrained models and their corresponding [original models](https://github.com/microsoft/Swin-Transforme) models could be defined below:

| pretrained models                              | original models                                                                                                                                    |
| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |

| pretrain/swin_tiny_patch4_window7_224.pth      | [swin_tiny_patch4_window7_224.pth](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_tiny_patch4_window7_224.pth)           |
| pretrain/swin_small_patch4_window7_224.pth     | [swin_small_patch4_window7_224.pth](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_small_patch4_window7_224.pth)         |
| pretrain/swin_base_patch4_window7_224.pth      | [swin_base_patch4_window7_224.pth](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224.pth)           |
| pretrain/swin_base_patch4_window7_224_22k.pth  | [swin_base_patch4_window7_224_22k.pth](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224_22k.pth)   |

| pretrain/swin_base_patch4_window12_384.pth     | [swin_base_patch4_window12_384.pth](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384.pth)         |
| pretrain/swin_base_patch4_window12_384_22k.pth | [swin_base_patch4_window12_384_22k.pth](https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384_22k.pth) |

### ADE20K

### ADE20K

| Method  | Backbone | Crop Size | pretrain     | pretrain img size | Batch Size | Lr schd | Mem (GB) | Inf time (fps) | Device | mIoU  | mIoU(ms+flip) | config                                                                                                                                                          | download                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |

| UPerNet | Swin-T   | 512x512   | ImageNet-1K  | 224x224           | 16         | 160000  | 5.02     | 21.06          | V100   | 44.41 |         45.79 | [config](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/swin/swin-tiny-patch4-window7-in1k-pre_upernet_8xb2-160k_ade20k-512x512.py)           | [model](https://download.openmmlab.com/mmsegmentation/v0.5/swin/upernet_swin_tiny_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K/upernet_swin_tiny_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K_20210531_112542-e380ad3e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/swin/upernet_swin_tiny_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K/upernet_swin_tiny_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K_20210531_112542.log.json)         |

| UPerNet | Swin-S   | 512x512   | ImageNet-1K  | 224x224           | 16         | 160000  | 6.17     | 14.72          | V100   | 47.72 |         49.24 | [config](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/swin/swin-small-patch4-window7-in1k-pre_upernet_8xb2-160k_ade20k-512x512.py)          | [model](https://download.openmmlab.com/mmsegmentation/v0.5/swin/upernet_swin_small_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K/upernet_swin_small_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K_20210526_192015-ee2fff1c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/swin/upernet_swin_small_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K/upernet_swin_small_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K_20210526_192015.log.json)     |

| UPerNet | Swin-B   | 512x512   | ImageNet-1K  | 224x224           | 16         | 160000  | 7.61     | 12.65          | V100   | 47.99 |         49.57 | [config](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/swin/swin-base-patch4-window7-in1k-pre_upernet_8xb2-160k_ade20k-512x512.py)           | [model](https://download.openmmlab.com/mmsegmentation/v0.5/swin/upernet_swin_base_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K/upernet_swin_base_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K_20210526_192340-593b0e13.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/swin/upernet_swin_base_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K/upernet_swin_base_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K_20210526_192340.log.json)         |

| UPerNet | Swin-B   | 512x512   | ImageNet-22K | 224x224           | 16         | 160000  | -        | -              | V100   | 50.13 |          51.9 | [config](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/swin/swin-base-patch4-window7-in22k-pre_upernet_8xb2-160k_ade20k-512x512.py)          | [model](https://download.openmmlab.com/mmsegmentation/v0.5/swin/upernet_swin_base_patch4_window7_512x512_160k_ade20k_pretrain_224x224_22K/upernet_swin_base_patch4_window7_512x512_160k_ade20k_pretrain_224x224_22K_20210526_211650-762e2178.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/swin/upernet_swin_base_patch4_window7_512x512_160k_ade20k_pretrain_224x224_22K/upernet_swin_base_patch4_window7_512x512_160k_ade20k_pretrain_224x224_22K_20210526_211650.log.json)     |

| UPerNet | Swin-B   | 512x512   | ImageNet-1K  | 384x384           | 16         | 160000  | 8.52     | 12.10          | V100   | 48.35 |         49.65 | [config](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/swin/swin-base-patch4-window12-in1k-384x384-pre_upernet_8xb2-160k_ade20k-512x512.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/swin/upernet_swin_base_patch4_window12_512x512_160k_ade20k_pretrain_384x384_1K/upernet_swin_base_patch4_window12_512x512_160k_ade20k_pretrain_384x384_1K_20210531_132020-05b22ea4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/swin/upernet_swin_base_patch4_window12_512x512_160k_ade20k_pretrain_384x384_1K/upernet_swin_base_patch4_window12_512x512_160k_ade20k_pretrain_384x384_1K_20210531_132020.log.json)     |

| UPerNet | Swin-B   | 512x512   | ImageNet-22K | 384x384           | 16         | 160000  | -        | -              | V100   | 50.76 |          52.4 | [config](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/swin/swin-base-patch4-window12-in22k-384x384-pre_upernet_8xb2-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/swin/upernet_swin_base_patch4_window12_512x512_160k_ade20k_pretrain_384x384_22K/upernet_swin_base_patch4_window12_512x512_160k_ade20k_pretrain_384x384_22K_20210531_125459-429057bf.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/swin/upernet_swin_base_patch4_window12_512x512_160k_ade20k_pretrain_384x384_22K/upernet_swin_base_patch4_window12_512x512_160k_ade20k_pretrain_384x384_22K_20210531_125459.log.json) |

```bibtex
@article{liu2021Swin,
  title={Swin Transformer: Hierarchical Vision Transformer using Shifted Windows},

```bibtex
  journal={arXiv preprint arXiv:2103.14030},
}
```

# Evaluation with LMDeploy

We now support evaluation of models accelerated by the [LMDeploy](https://github.com/InternLM/lmdeploy). LMDeploy is a toolkit designed for compressing, deploying, and serving LLM. **TurboMind** is an efficient inference engine proposed by LMDeploy. OpenCompass is compatible with TurboMind. We now illustrate how to evaluate a model with the support of TurboMind in OpenCompass.

## Setup

### Install OpenCompass

Please follow the [instructions](https://opencompass.readthedocs.io/en/latest/get_started.html) to install the OpenCompass and prepare the evaluation datasets.

### Install LMDeploy

Install lmdeploy via pip (python 3.8+)

```shell
```

We take the InternLM as example.

### Step-1: Get InternLM model

```shell

```shell

```

### Step-2: Verify the Converted Model

```shell
```

### Step-3: Evaluate the Converted Model

In the home folder of OpenCompass

```shell
```

You are expected to get the evaluation results after the inference and evaluation.

# Preparing HVU

```BibTeX
}
```

For basic dataset information, please refer to the official [project](https://github.com/holistic-video-understanding/HVU-Dataset/) and the [paper](https://arxiv.org/abs/1904.11451).
Before we start, please make sure that the directory is located at `$MMACTION2/tools/data/hvu/`.

```shell
```

Besides, you need to run the following command to parse the tag list of HVU.

```shell
```

```shell
```

```shell
```

You can use the following script to extract both RGB and Flow frames.

```shell
```

By default, we generate frames with short edge resized to 256.
More details can be found in [prepare_dataset](/docs/en/user_guides/prepare_dataset.md)

You can run the follow scripts to generate file list in the format of videos and rawframes, respectively.

```shell

```shell
```

## Step 5. Generate File List for Each Individual Tag Categories

This part is **optional** if you don't want to train models on HVU for a specific tag category.

The file list generated in step 4 contains labels of different categories. These file lists can only be
handled with HVUDataset and used for multi-task learning of different tag categories. The component
`LoadHVULabel` is needed to load the multi-category tags, and the `HVULoss` should be used to train
the model.

If you only want to train video recognition models for a specific tag category, i.e. you want to train
a recognition model on HVU which only handles tags in the category `action`, we recommend you to use

the following command to generate file lists for the specific tag category. The new list, which only
contains tags of a specific category, can be handled with `VideoDataset` or `RawframeDataset`. The
recognition models can be trained with `BCELossWithLogits`.

The following command generates file list for the tag category ${category}, note that the tag category you
specified should be in the 6 tag categories available in HVU: \['action', 'attribute', 'concept', 'event',
'object', 'scene'\].

```shell
```

The filename of the generated file list for ${category} is generated by replacing `hvu` in the original
filename with `hvu_${category}`. For example, if the original filename is `hvu_train.json`, the filename
of the file list for action is `hvu_action_train.json`.

After the whole data pipeline for HVU preparation.
you can get the rawframes (RGB + Flow), videos and annotation files for HVU.

In the context of the whole project (for HVU only), the full folder structure will look like:

```
mmaction2
├── tools
├── data
│   ├── hvu
│   │   │   ├── ...

```

For training and evaluating on HVU, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).

# RTMDet-R

|  Backbone   | pretrain |  Aug  | mmAP  | mAP50 | mAP75 | Params(M) | FLOPS(G) | TRT-FP16-Latency(ms) |                          Config                          |                                                                                                                                                                       Download                                                                                                                                                                       |

| RTMDet-tiny |    IN    |  RR   | 47.37 | 75.36 | 50.64 |   4.88    |  20.45   |         4.40         |        [config](./rotated_rtmdet_tiny-3x-dota.py)        |                             [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_tiny-3x-dota/rotated_rtmdet_tiny-3x-dota-9d821076.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_tiny-3x-dota/rotated_rtmdet_tiny-3x-dota_20221201_120814.json)                             |

| RTMDet-tiny |    IN    | MS+RR | 53.59 | 79.82 | 58.87 |   4.88    |  20.45   |         4.40         |      [config](./rotated_rtmdet_tiny-3x-dota_ms.py)       |                       [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_tiny-3x-dota_ms/rotated_rtmdet_tiny-3x-dota_ms-f12286ff.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_tiny-3x-dota_ms/rotated_rtmdet_tiny-3x-dota_ms_20221113_201235.log)                        |

|  RTMDet-s   |    IN    |  RR   | 48.16 | 76.93 | 50.59 |   8.86    |  37.62   |         4.86         |         [config](./rotated_rtmdet_s-3x-dota.py)          |                                   [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_s-3x-dota/rotated_rtmdet_s-3x-dota-11f6ccf5.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_s-3x-dota/rotated_rtmdet_s-3x-dota_20221124_081442.json)                                   |

|  RTMDet-s   |    IN    | MS+RR | 54.43 | 79.98 | 60.07 |   8.86    |  37.62   |         4.86         |        [config](./rotated_rtmdet_s-3x-dota_ms.py)        |                             [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_s-3x-dota_ms/rotated_rtmdet_s-3x-dota_ms-20ead048.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_s-3x-dota_ms/rotated_rtmdet_s-3x-dota_ms_20221113_201055.json)                             |

|  RTMDet-m   |    IN    |  RR   | 50.56 | 78.24 | 54.47 |   24.67   |  99.76   |         7.82         |         [config](./rotated_rtmdet_m-3x-dota.py)          |                                   [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_m-3x-dota/rotated_rtmdet_m-3x-dota-beeadda6.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_m-3x-dota/rotated_rtmdet_m-3x-dota_20221122_011234.json)                                   |

|  RTMDet-m   |    IN    | MS+RR | 55.00 | 80.26 | 61.26 |   24.67   |  99.76   |         7.82         |        [config](./rotated_rtmdet_m-3x-dota_ms.py)        |                             [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_m-3x-dota_ms/rotated_rtmdet_m-3x-dota_ms-c71eb375.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_m-3x-dota_ms/rotated_rtmdet_m-3x-dota_ms_20221122_011234.json)                             |

|  RTMDet-l   |    IN    |  RR   | 51.01 | 78.85 | 55.21 |   52.27   |  204.21  |        10.82         |         [config](./rotated_rtmdet_l-3x-dota.py)          |                                   [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_l-3x-dota/rotated_rtmdet_l-3x-dota-23992372.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_l-3x-dota/rotated_rtmdet_l-3x-dota_20221122_011241.json)                                   |

|  RTMDet-l   |    IN    | MS+RR | 55.52 | 80.54 | 61.47 |   52.27   |  204.21  |        10.82         |        [config](./rotated_rtmdet_l-3x-dota_ms.py)        |                             [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_l-3x-dota_ms/rotated_rtmdet_l-3x-dota_ms-2738da34.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_l-3x-dota_ms/rotated_rtmdet_l-3x-dota_ms_20221122_011241.json)                             |

|  RTMDet-l   |   COCO   | MS+RR | 56.74 | 81.33 | 63.45 |   52.27   |  204.21  |        10.82         | [config](./rotated_rtmdet_l-coco_pretrain-3x-dota_ms.py) | [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_l-coco_pretrain-3x-dota_ms/rotated_rtmdet_l-coco_pretrain-3x-dota_ms-06d248a2.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_l-coco_pretrain-3x-dota_ms/rotated_rtmdet_l-coco_pretrain-3x-dota_ms_20221113_202010.json) |

- By default, DOTA-v1.0 dataset trained with 3x schedule and image size 1024\*1024.

### HRSC

|  Backbone   | pretrain | Aug | mAP 07 | mAP 12 | Params(M) | FLOPS(G) |                   Config                   |                                                                                                                                           Download                                                                                                                                           |

| :---------: | :------: | :-: | :----: | :----: | :-------: | :------: | :----------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| RTMDet-tiny |    IN    | RR  |  90.6  |  97.1  |   4.88    |  12.54   | [config](./rotated_rtmdet_tiny-9x-hrsc.py) | [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_tiny-9x-hrsc/rotated_rtmdet_tiny-9x-hrsc-9f2e3ca6.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_tiny-9x-hrsc/rotated_rtmdet_tiny-9x-hrsc_20221125_145920.json) |

- By default, HRSC dataset trained with 9x schedule and image size 800\*800.

### Stronger augmentation

We also provide configs with Mixup, Mosaic and RandomRotate with longer schedule. Training time is less than MS.

DOTA:

| Backbone | pretrain | schedule |       Aug       | mmAP  | mAP50 | mAP75 |                    Config                     |                                                                                                                                                 Download                                                                                                                                                 |

| :------: | :------: | :------: | :-------------: | :---: | :---: | :---: | :-------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| RTMDet-l |    IN    |   100e   | Mixup+Mosaic+RR | 54.59 | 80.16 | 61.16 | [config](./rotated_rtmdet_l-100e-aug-dota.py) | [model](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_l-100e-aug-dota/rotated_rtmdet_l-100e-aug-dota-bc59fd88.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/rotated_rtmdet/rotated_rtmdet_l-100e-aug-dota/rotated_rtmdet_l-100e-aug-dota_20221124_224135.json) |

**Note**:

1. We follow the latest metrics from the DOTA evaluation server, original voc format mAP is now mAP50.

**Note**:

2. `IN` means ImageNet pretrain, `COCO` means COCO pretrain.
3. Different from the report, the inference speed here is measured on an NVIDIA 2080Ti GPU with TensorRT 8.4.3, cuDNN 8.2.0, FP16, batch size=1, and with NMS.

```
}
```

# Lyft Dataset

This page provides specific tutorials about the usage of MMDetection3D for Lyft dataset.

You can download Lyft 3D detection data [HERE](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/data) and unzip all zip files.

Like the general way to prepare a dataset, it is recommended to symlink the dataset root to `$MMDETECTION3D/data`.

```
├── tools
├── data
│   ├── lyft
```

```

Here `v1.01-train` and `v1.01-test` contain the metafiles which are similar to those of nuScenes. `.txt` files contain the data split information.
Lyft does not have an official split for training and validation set, so we provide a split considering the number of objects from different categories in different scenes.
`sample_submission.csv` is the base file for submission on the Kaggle evaluation server.
Note that we follow the original folder names for clear organization. Please rename the raw folders as shown above.

The way to organize Lyft dataset is similar to nuScenes. We also generate the `.pkl` files which share almost the same structure.

Next, we will mainly focus on the difference between these two datasets. For a more detailed explanation of the info structure, please refer to [nuScenes tutorial](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/docs/en/advanced_guides/datasets/nuscenes_det.md).

To prepare info files for Lyft, run the following commands:

```bash
```

Note that the second command serves the purpose of fixing a corrupted lidar data file. Please refer to the discussion [here](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/discussion/110000) for more details.

```
├── tools
├── data
│   ├── lyft
```

- `lyft_infos_train.pkl`: training dataset, a dict contains two keys: `metainfo` and `data_list`.

```

    - info\['lidar_points'\]\['ego2global'\]: The transformation matrix from the ego vehicle to global coordinates. (4x4 list)
  - info\['lidar_sweeps'\]: A list contains sweeps information (The intermediate lidar frames without annotations).
    - info\['lidar_sweeps'\]\[i\]\['lidar_points'\]\['lidar2ego'\]: The transformation matrix from this lidar sensor to ego vehicle in i-th sweep timestamp
    - info\['lidar_sweeps'\]\[i\]\['lidar_points'\]\['ego2global'\]: The transformation matrix from the ego vehicle in i-th sweep timestamp to global coordinates. (4x4 list)

- info\['lidar_sweeps'\]\[i\]\['lidar_points'\]\['ego2global'\]: The transformation matrix from the ego vehicle in i-th sweep timestamp to global coordinates. (4x4 list)
    - info\['lidar_sweeps'\]\[i\]\['lidar2sensor'\]: The transformation matrix from the keyframe lidar to the i-th frame lidar. (4x4 list)

- info\['images'\]\['CAM_XXX'\]\['cam2img'\]: The transformation matrix recording the intrinsic parameters when projecting 3D points to each image plane. (3x3 list)
    - info\['instances'\]\[i\]\['bbox_3d'\]: List of 7 numbers representing the 3D bounding box in lidar coordinate system of the instance, in (x, y, z, l, w, h, yaw) order.

- info\['instances'\]\[i\]\['bbox_3d'\]: List of 7 numbers representing the 3D bounding box in lidar coordinate system of the instance, in (x, y, z, l, w, h, yaw) order.
    - info\['instances'\]\[i\]\['bbox_label_3d'\]: A int starting from 0 indicates the label of instance, while the -1 indicates ignore class.

Next, we will elaborate on the difference compared to nuScenes in terms of the details recorded in these info files.

- Without `lyft_database/xxxxx.bin`: This folder and `.bin` files are not extracted on the Lyft dataset due to the negligible effect of ground-truth sampling in the experiments.

- `lyft_infos_train.pkl`:

  - Without info\['instances'\]\[i\]\['velocity'\]: There is no velocity measurement on Lyft.

  - Without info\['instances'\]\[i\]\['num_lidar_pts'\] and info\['instances'\]\['num_radar_pts'\]

Here we only explain the data recorded in the training info files. The same applies to the validation set and test set (without instances).

Please refer to [lyft_converter.py](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/lyft_converter.py) for more details about the structure of `lyft_infos_xxx.pkl`.

A typical training pipeline of LiDAR-based 3D detection (including multi-modality methods) on Lyft is almost the same as nuScenes as below.

```python
    dict(
    dict(

dict(
    dict(
    dict(
    dict(
]
```

Similar to nuScenes, models on Lyft also need the `'LoadPointsFromMultiSweeps'` pipeline to load point clouds from consecutive frames.

]
```

In addition, considering the intensity of LiDAR points collected by Lyft is invalid, we also set the `use_dim` in `'LoadPointsFromMultiSweeps'` to `[0, 1, 2, 4]` by default,
where the first 3 dimensions refer to point coordinates, and the last refers to timestamp differences.

An example to evaluate PointPillars with 8 GPUs with Lyft metrics is as follows:

```shell
```

## Metrics

Lyft proposes a more strict metric for evaluating the predicted 3D bounding boxes.
The basic criteria to judge whether a predicted box is positive or not is the same as KITTI, i.e. the 3D Intersection over Union (IoU).

However, it adopts a way similar to COCO to compute the mean average precision (mAP) -- compute the average precision under different thresholds of 3D IoU from 0.5-0.95.
Actually, overlap more than 0.7 3D IoU is a quite strict criterion for 3D detection methods, so the overall performance seems a little low.
The imbalance of annotations for different categories is another important reason for the finally lower results compared to other datasets.
Please refer to its [official website](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/overview/evaluation) for more details about the definition of this metric.

We employ this official method for evaluation on Lyft. An example of printed evaluation results is as follows:

```

```
```

An example to test PointPillars on Lyft with 8 GPUs and generate a submission to the leaderboard is as follows.

```shell
```

```

After generating the `work_dirs/pp-lyft/results_challenge.csv`, you can submit it to the Kaggle evaluation server. Please refer to the [official website](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles) for more information.

# 数据增强变化

在本教程中，我们将介绍 MMSegmentation 中数据增强变化流程的设计。

本指南的结构如下：

- [数据增强变化](#数据增强变化)
  - [数据增强变化流程设计](#数据增强变化流程设计)
    - [数据加载](#数据加载)
    - [预处理](#预处理)
    - [格式修改](#格式修改)

## 数据增强变化流程设计

按照惯例，我们使用 `Dataset` 和 `DataLoader` 多进程地加载数据。`Dataset` 返回与模型 forward 方法的参数相对应的数据项的字典。由于语义分割中的数据可能大小不同，我们在 MMCV 中引入了一种新的 `DataContainer` 类型，以帮助收集和分发不同大小的数据。参见[此处](https://github.com/open-mmlab/mmcv/blob/master/mmcv/parallel/data_container.py)了解更多详情。

在 MMSegmentation 的 1.x 版本中，所有数据转换都继承自 [`BaseTransform`](https://github.com/open-mmlab/mmcv/blob/2.x/mmcv/transforms/base.py#L6).

转换的输入和输出类型都是字典。一个简单的示例如下：

```python

```

数据准备流程和数据集是解耦的。通常，数据集定义如何处理标注，数据流程定义准备数据字典的所有步骤。流程由一系列操作组成。每个操作都将字典作为输入，并为接下来的转换输出字典。

操作分为数据加载、预处理、格式修改和测试数据增强。

这里是 PSPNet 的流程示例：

```python
    dict(
]

dict(type='RandomCrop', crop_size=crop_size, cat_max_ratio=0.75),
]
]
```

对于每个操作，我们列出了 `添加`/`更新`/`删除` 相关的字典字段。在流程前，我们可以从数据集直接获得的信息是 `img_path` 和 `seg_map_path`。

### 数据加载

`LoadImageFromFile`：从文件加载图像。

- 添加：`img`，`img_shape`，`ori_shape`

`LoadAnnotations`：加载数据集提供的语义分割图。

- 添加：`seg_fields`，`gt_seg_map`

### 预处理

`RandomResize`：随机调整图像和分割图大小。

-添加：`scale`，`scale_factor`，`keep_ratio`
-更新：`img`，`img_shape`，`gt_seg_map`

`Resize`：调整图像和分割图的大小。

-更新：`img`，`gt_seg_map`，`img_shape`

`RandomCrop`：随机裁剪图像和分割图。

`RandomFlip`：翻转图像和分割图。

-添加：`flip`，`flip_direction`
-更新：`img`，`gt_seg_map`

`PhotoMetricDistortion`：按顺序对图像应用光度失真，每个变换的应用概率为 0.5。随机对比度的位置是第二或倒数第二（分别为下面的模式 0 或 1）。

```
1.随机亮度
2.随机对比度（模式 0）
3.将颜色从 BGR 转换为 HSV
4.随机饱和度
5.随机色调
6.将颜色从 HSV 转换为 BGR
7.随机对比度（模式 1）
```

- 更新：`img`

### 格式修改

`PackSegInputs`：为语义分段打包输入数据。

- 添加：`inputs`，`data_sample`
- 删除：由 `meta_keys` 指定的 keys（合并到 data_sample 的 metainfo 中），所有其他 keys

# MoCoV2

> [Improved Baselines with Momentum Contrastive Learning](https://arxiv.org/abs/2003.04297)

Contrastive unsupervised learning has recently shown encouraging progress, e.g., in Momentum Contrast (MoCo) and SimCLR. In this note, we verify the effectiveness of two of SimCLR’s design improvements by implementing them in the MoCo framework. With simple modifications to MoCo—namely, using an MLP projection head and more data augmentation—we establish stronger baselines that outperform SimCLR and do not require large training batches. We hope this will make state-of-the-art unsupervised learning research more accessible.

<img  src="https://user-images.githubusercontent.com/36138628/149720067-b65e5736-d425-45b3-93ed-6f2427fc6217.png" width="500" />
</div>

```python

</div>

```python

predict = inference_model('resnet50_mocov2-pre_8xb32-linear-steplr-100e_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('mocov2_resnet50_8xb32-coslr-200e_in1k', pretrained=True)
```

Train:

```shell
python tools/train.py configs/mocov2/mocov2_resnet50_8xb32-coslr-200e_in1k.py
```

Test:

```shell

Train:

```shell
```

Test:

```shell
python tools/test.py configs/mocov2/benchmarks/resnet50_8xb32-linear-steplr-100e_in1k.py https://download.openmmlab.com/mmselfsup/1.x/mocov2/mocov2_resnet50_8xb32-coslr-200e_in1k/resnet50_linear-8xb32-steplr-100e_in1k/resnet50_linear-8xb32-steplr-100e_in1k_20220825-994c4128.pth
```

| `mocov2_resnet50_8xb32-coslr-200e_in1k` |   55.93    |   4.11    | [config](mocov2_resnet50_8xb32-coslr-200e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/mocov2/mocov2_resnet50_8xb32-coslr-200e_in1k/mocov2_resnet50_8xb32-coslr-200e_in1k_20220825-b6d23c86.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mocov2/mocov2_resnet50_8xb32-coslr-200e_in1k/mocov2_resnet50_8xb32-coslr-200e_in1k_20220825-b6d23c86.json) |

| `resnet50_mocov2-pre_8xb32-linear-steplr-100e_in1k` | [MOCOV2](https://download.openmmlab.com/mmselfsup/1.x/mocov2/mocov2_resnet50_8xb32-coslr-200e_in1k/mocov2_resnet50_8xb32-coslr-200e_in1k_20220825-b6d23c86.pth) |   25.56    |   4.11    |   67.50   | [config](benchmarks/resnet50_8xb32-linear-steplr-100e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/mocov2/mocov2_resnet50_8xb32-coslr-200e_in1k/resnet50_linear-8xb32-steplr-100e_in1k/resnet50_linear-8xb32-steplr-100e_in1k_20220825-994c4128.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/mocov2/mocov2_resnet50_8xb32-coslr-200e_in1k/resnet50_linear-8xb32-steplr-100e_in1k/resnet50_linear-8xb32-steplr-100e_in1k_20220825-994c4128.json) |

```bibtex
@article{chen2020improved,
  title={Improved baselines with momentum contrastive learning},
  author={Chen, Xinlei and Fan, Haoqi and Girshick, Ross and He, Kaiming},
  journal={arXiv preprint arXiv:2003.04297},
}
```

# 自定义日志

MMAction2 在运行过程中会产生大量的日志，如损失、迭代时间、学习率等。在这一部分，我们将向你介绍如何输出自定义日志。有关日志系统的更多详细信息，请参考 [MMEngine 教程](https://mmengine.readthedocs.io/zh_CN/latest/advanced_tutorials/logging.html)。

- [自定义日志](#自定义日志)
  - [灵活的日志系统](#灵活的日志系统)
  - [定制日志](#定制日志)
  - [导出调试日志](#导出调试日志)

## 灵活的日志系统

默认情况下，MMAction2 的日志系统由 [default_runtime](/configs/_base_/default_runtime.py) 中的 `LogProcessor` 配置：

```python
```

默认情况下，`LogProcessor` 捕获 `model.forward` 返回的所有以 `loss` 开头的字段。例如，在以下模型中，`loss1` 和 `loss2` 将在没有任何额外配置的情况下自动记录到日志。

```python

```

输出日志遵循以下格式：

```

feat = self.linear(img)
```

```
```

`LogProcessor` 将按以下格式输出日志：

- 日志的前缀：
  - epoch 模式(`by_epoch=True`)：`Epoch(train) [{current_epoch}/{current_iteration}]/{dataloader_length}`
  - iteration 模式(`by_epoch=False`)：`Iter(train) [{current_iteration}/{max_iteration}]`
- 学习率 (`lr`)：最后一次迭代的学习率。
- 时间：
  - `time`：过去 `window_size` 次迭代的推理平均时间。
  - `data_time`：过去 `window_size` 次迭代的数据加载平均时间。
  - `eta`：完成训练的预计到达时间。
- 损失：过去 `window_size` 次迭代中模型输出的平均损失。

- 时间：

默认情况下，log_processor 输出基于 epoch 的日志(`by_epoch=True`)。要得到与 `train_cfg` 匹配的预期日志，我们应在 `train_cfg` 和 `log_processor` 中设置相同的 `by_epoch` 值。
```

根据以上规则，代码片段将每20次迭代计算 loss1 和 loss2 的平均值。更多类型的统计方法，请参考 [mmengine.runner.LogProcessor](mmengine.runner.LogProcessor)。

## 定制日志

日志系统不仅可以记录 `loss`，`lr` 等，还可以收集和输出自定义日志。例如，如果我们想要统计中间损失：

`ToyModel` 在 forward 中计算 `loss_tmp`，但不将其保存到返回字典中。

```python

        # 在消息中心更新中间的 `loss_tmp`

```

将 `loss_tmp` 添加到配置中：

```python
        # 使用平均值统计 loss_tmp
            dict(
        ])
```

`loss_tmp` 将被添加到输出日志中：

```
```

## 导出调试日志

要将调试日志导出到 `work_dir`，你可以在配置文件中设置日志级别如下：

```

```

## 导出调试日志

```
```

```
...
```

此外，如果你正在使用共享存储训练你的模型，那么在 `debug` 模式下，不同排名的日志将被保存。日志的层级结构如下：

```text
./tmp
...
```

在具有独立存储的多台机器上的日志：

```text
# 设备：0：

...
```

```text
# 设备：0：
work_dir/
    ...

# 设备：7：
work_dir/
    ...
```

# Res2Net

<img src="https://user-images.githubusercontent.com/40661020/143972411-8d08113f-9fce-4d24-a138-4fadf2c54f9a.png" height="300"/>
</div>

We propose a novel building block for CNNs, namely Res2Net, by constructing hierarchical residual-like connections within one single residual block. The Res2Net represents multi-scale features at a granular level and increases the range of receptive fields for each network layer.

|     Backbone      | Params. | GFLOPs | top-1 err. | top-5 err. |
| :---------------: | :-----: | :----: | :--------: | :--------: |
|    ResNet-101     | 44.6 M  |  7.8   |   22.63    |    6.44    |
| ResNeXt-101-64x4d |  83.5M  |  15.5  |   20.40    |     -      |
|   HRNetV2p-W48    |  77.5M  |  16.1  |   20.70    |    5.50    |
|    Res2Net-101    |  45.2M  |  8.3   |   18.77    |    4.64    |

Compared with other backbone networks, Res2Net requires fewer parameters and FLOPs.

**Note:**

**Note:**

- GFLOPs for classification are calculated with image size (224x224).

|  Backbone  |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                       Config                       |                                                                                                                                               Download                                                                                                                                               |

| :--------: | :-----: | :-----: | :------: | :------------: | :----: | :------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| R2-101-FPN | pytorch |   2x    |   7.4    |       -        |  43.0  | [config](./faster-rcnn_res2net-101_fpn_2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/res2net/faster_rcnn_r2_101_fpn_2x_coco/faster_rcnn_r2_101_fpn_2x_coco-175f1da6.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/res2net/faster_rcnn_r2_101_fpn_2x_coco/faster_rcnn_r2_101_fpn_2x_coco_20200514_231734.log.json) |

|  Backbone  |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                      Config                      |                                                                                                                                           Download                                                                                                                                           |
| :--------: | :-----: | :-----: | :------: | :------------: | :----: | :-----: | :----------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

| R2-101-FPN | pytorch |   2x    |   7.9    |       -        |  43.6  |  38.7   | [config](./mask-rcnn_res2net-101_fpn_2x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/res2net/mask_rcnn_r2_101_fpn_2x_coco/mask_rcnn_r2_101_fpn_2x_coco-17f061e8.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/res2net/mask_rcnn_r2_101_fpn_2x_coco/mask_rcnn_r2_101_fpn_2x_coco_20200515_002413.log.json) |

|  Backbone  |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                        Config                        |                                                                                                                                                   Download                                                                                                                                                   |

| :--------: | :-----: | :-----: | :------: | :------------: | :----: | :--------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| R2-101-FPN | pytorch |   20e   |   7.8    |       -        |  45.7  | [config](./cascade-rcnn_res2net-101_fpn_20e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/res2net/cascade_rcnn_r2_101_fpn_20e_coco/cascade_rcnn_r2_101_fpn_20e_coco-f4b7b7db.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/res2net/cascade_rcnn_r2_101_fpn_20e_coco/cascade_rcnn_r2_101_fpn_20e_coco_20200515_091644.log.json) |

|  Backbone  |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                          Config                           |                                                                                                                                                             Download                                                                                                                                                             |
| :--------: | :-----: | :-----: | :------: | :------------: | :----: | :-----: | :-------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

| R2-101-FPN | pytorch |   20e   |   9.5    |       -        |  46.4  |  40.0   | [config](./cascade-mask-rcnn_res2net-101_fpn_20e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/res2net/cascade_mask_rcnn_r2_101_fpn_20e_coco/cascade_mask_rcnn_r2_101_fpn_20e_coco-8a7b41e1.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/res2net/cascade_mask_rcnn_r2_101_fpn_20e_coco/cascade_mask_rcnn_r2_101_fpn_20e_coco_20200515_091645.log.json) |

|  Backbone  |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                   Config                    |                                                                                                                                 Download                                                                                                                                 |

| :--------: | :-----: | :-----: | :------: | :------------: | :----: | :-----: | :-----------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| R2-101-FPN | pytorch |   20e   |    -     |       -        |  47.5  |  41.6   | [config](./htc_res2net-101_fpn_20e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/res2net/htc_r2_101_fpn_20e_coco/htc_r2_101_fpn_20e_coco-3a8d2112.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/res2net/htc_r2_101_fpn_20e_coco/htc_r2_101_fpn_20e_coco_20200515_150029.log.json) |

- Res2Net ImageNet pretrained models are in [Res2Net-PretrainedModels](https://github.com/Res2Net/Res2Net-PretrainedModels).
- More applications of Res2Net are in [Res2Net-Github](https://github.com/Res2Net/).

```latex
}
```

# 配置文件

MMOCR 主要使用 Python 文件作为配置文件。其配置文件系统的设计整合了模块化与继承的思想，方便用户进行各种实验。

## 常见用法

```{note}
本小节建议结合 {external+mmengine:doc}`MMEngine: 配置(Config) <tutorials/config>` 中的初级用法共同阅读。
```

MMOCR 最常用的操作为三种：配置文件的继承，对 `_base_` 变量的引用以及对 `_base_` 变量的修改。对于 `_base_` 的继承与修改, MMEngine.Config 提供了两种语法，一种是针对 Python，Json， Yaml 均可使用的操作；另一种则仅适用于 Python 配置文件。在 MMOCR 中，我们**更推荐使用只针对Python的语法**，因此下文将以此为基础作进一步介绍。

这里以 `configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py` 为例，说明常用的三种用法。

```Python
_base_ = [
]

```

配置文件存在继承的机制，即一个配置文件 A 可以将另一个配置文件 B 作为自己的基础并直接继承其中的所有字段，从而避免了大量的复制粘贴。

在 dbnet_resnet18_fpnc_1200e_icdar2015.py 中可以看到：

```Python
_base_ = [
]
```

上述语句会读取列表中的所有基础配置文件，它们中的所有字段都会被载入到 dbnet_resnet18_fpnc_1200e_icdar2015.py 中。我们可以通过在 Python 解释中运行以下语句，了解配置文件被解析后的结构：

```Python

```Python
```

可以发现，被解析的配置包含了所有base配置中的字段和信息。

```{note}
请注意：各 _base_ 配置文件中不能存在同名变量。
```

### `_base_` 变量的引用

有时，我们可能需要直接引用 `_base_` 配置中的某些字段，以避免重复定义。假设我们想要获取 `_base_` 配置中的变量 `pseudo`，就可以直接通过 `_base_.pseudo` 获得 `_base_` 配置中的变量。

该语法已广泛用于 MMOCR 的配置中。MMOCR 中各个模型的数据集和管道（pipeline）配置都引用于*基本*配置。如在

```Python
# ...
    # ...
```

### `_base_` 变量的修改

在 MMOCR 中不同算法在不同数据集通常有不同的数据流水线（pipeline)，因此经常会会存在修改数据集中 `pipeline` 的场景。同时还存在很多场景需要修改 `_base_` 配置中的变量，例如想修改某个算法的训练策略，某个模型的某些算法模块（更换 backbone 等）。用户可以直接利用 Python 的语法直接修改引用的 `_base_` 变量。针对 dict，我们也提供了与类属性修改类似的方法，可以直接修改类属性修改字典内的内容。

1. 字典

   这里以修改数据集中的 `pipeline` 为例：

   可以利用 Python 语法修改字典：

   ```python
   # 获取 _base_ 中的数据集

1. 字典

   ```python
   # 可以直接利用 Python 的 update 修改变量
   ```

   也可以使用类属性的方法进行修改：

   ```Python
   # 类属性方法修改
   ```

2. 列表

   假设 `_base_` 配置中的变量 `pseudo = [1, 2, 3]`， 需要修改为 `[1, 2, 4]`:

   ```Python
   ```

   可以直接重写：

   ```Python
   ```

   或者利用 Python 语法修改列表：

   ```Python
   ```

### 命令行修改配置

有时候我们只希望修部分配置，而不想修改配置文件本身。例如实验过程中想更换学习率，但是又不想重新写一个配置文件，可以通过命令行传入参数来覆盖相关配置。

我们可以在命令行里传入 `--cfg-options`，并在其之后的参数直接修改对应字段，例如我们想在运行 train 的时候修改学习率，只需要在命令行执行：

```Shell

```

```Shell
```

更多详细用法参考 {external+mmengine:ref}`MMEngine: 命令行修改配置 <命令行修改配置>`.

## 配置内容

通过配置文件与注册器的配合，MMOCR 可以在不侵入代码的前提下修改训练参数以及模型配置。具体而言，用户可以在配置文件中对如下模块进行自定义修改：环境配置、Hook 配置、日志配置、训练策略配置、数据相关配置、模型相关配置、评测配置、可视化配置。

本文档将以文字检测算法 `DBNet` 和文字识别算法 `CRNN` 为例来详细介绍 Config 中的内容。

### 环境配置

```Python
```

主要包含三个部分：

- 设置所有注册器的默认 `scope` 为 `mmocr`， 保证所有的模块首先从 `MMOCR` 代码库中进行搜索。若果该模块不存在，则继续从上游算法库 `MMEngine` 和 `MMCV` 中进行搜索，详见 {external+mmengine:doc}`MMEngine: 注册器 <tutorials/registry>`。

```

主要包含三个部分：

- `env_cfg` 设置分布式环境配置， 更多配置可以详见 {external+mmengine:doc}`MMEngine: Runner <tutorials/runner>`。

- `randomness` 设置 numpy， torch，cudnn 等随机种子，更多配置详见 {external+mmengine:doc}`MMEngine: Runner <tutorials/runner>`。

### Hook 配置

Hook 主要分为两个部分，默认 hook 以及自定义 hook。默认 hook 为所有任务想要运行所必须的配置，自定义 hook 一般服务于特定的算法或某些特定任务（目前为止 MMOCR 中没有自定义的 Hook）。

```Python
    timer=dict(type='IterTimerHook'), # 时间记录，包括数据增强时间以及模型推理时间
    logger=dict(type='LoggerHook', interval=1), # 日志打印间隔
    param_scheduler=dict(type='ParamSchedulerHook'), # 更新学习率等超参
    checkpoint=dict(type='CheckpointHook', interval=1),# 保存 checkpoint， interval控制保存间隔
    sampler_seed=dict(type='DistSamplerSeedHook'), # 多机情况下设置种子
    sync_buffer=dict(type='SyncBuffersHook'), # 多卡情况下，同步buffer

sampler_seed=dict(type='DistSamplerSeedHook'), # 多机情况下设置种子
    visualization=dict( # 可视化val 和 test 的结果
```

这里简单介绍几个经常可能会变动的 hook，通用的修改方法参考[修改配置](#base-变量的修改)。

- `LoggerHook`：用于配置日志记录器的行为。例如，通过修改 `interval` 可以控制日志打印的间隔，每 `interval` 次迭代 (iteration) 打印一次日志，更多设置可参考 [LoggerHook API](mmengine.hooks.LoggerHook)。

- `CheckpointHook`：用于配置模型断点保存相关的行为，如保存最优权重，保存最新权重等。同样可以修改 `interval` 控制保存 checkpoint 的间隔。更多设置可参考 [CheckpointHook API](mmengine.hooks.CheckpointHook)

- `VisualizationHook`：用于配置可视化相关行为，例如在验证或测试时可视化预测结果，**默认为关**。同时该 Hook 依赖[可视化配置](#可视化配置)。想要了解详细功能可以参考 [Visualizer](visualization.md)。更多配置可以参考 [VisualizationHook API](mmocr.engine.hooks.VisualizationHook)。

如果想进一步了解默认 hook 的配置以及功能，可以参考 {external+mmengine:doc}`MMEngine: 钩子(Hook) <tutorials/hook>`。

### 日志配置

此部分主要用来配置日志配置等级以及日志处理器。

```Python
```

- 日志配置等级与 {external+python:doc}`Python: logging <library/logging>` 的配置一致，

- 日志处理器主要用来控制输出的格式，详细功能可参考 {external+mmengine:doc}`MMEngine: 记录日志 <advanced_tutorials/logging>`：

  - `by_epoch=True` 表示按照epoch输出日志，日志格式需要和 `train_cfg` 中的 `type='EpochBasedTrainLoop'` 参数保持一致。例如想按迭代次数输出日志，就需要令  `log_processor` 中的 ` by_epoch=False` 的同时 `train_cfg` 中的 `type = 'IterBasedTrainLoop'`。

  - `window_size` 表示损失的平滑窗口，即最近 `window_size` 次迭代的各种损失的均值。logger 中最终打印的 loss 值为各种损失的平均值。

<div id="schedule_config"></div>

### 训练策略配置

此部分主要包含优化器设置、学习率策略和 `Loop` 设置。

对不同算法任务(文字检测，文字识别，关键信息提取)，通常有自己任务常用的调参策略。这里列出了文字识别中的 `CRNN` 所用涉及的相应配置。

```Python
# 优化器

### 训练策略配置

```Python
# 优化器
                    max_epochs=5, # 训练轮数
                    val_interval=1) # 评测间隔
```

- `optim_wrapper` : 主要包含两个部分，优化器封装 (OptimWrapper) 以及优化器 (Optimizer)。详情使用信息可见 {external+mmengine:doc}`MMEngine: 优化器封装 <tutorials/optim_wrapper>`

  - 优化器封装支持不同的训练策略，包括混合精度训练(AMP)、梯度累加和梯度截断。

  - 优化器设置中支持了 PyTorch 所有的优化器，所有支持的优化器见 {external+torch:ref}`PyTorch 优化器列表 <optim:algorithms>`。

- 优化器封装支持不同的训练策略，包括混合精度训练(AMP)、梯度累加和梯度截断。

- `param_scheduler` : 学习率调整策略，支持大部分 PyTorch 中的学习率调度器，例如 `ExponentialLR`，`LinearLR`，`StepLR`，`MultiStepLR` 等，使用方式也基本一致，所有支持的调度器见[调度器接口文档](mmengine.optim.scheduler), 更多功能可以参考 {external+mmengine:doc}`MMEngine: 优化器参数调整策略 <tutorials/param_scheduler>`。

- `train/test/val_cfg` : 任务的执行流程，MMEngine 提供了四种流程：`EpochBasedTrainLoop`, `IterBasedTrainLoop`, `ValLoop`, `TestLoop` 更多可以参考 {external+mmengine:doc}`MMEngine: 循环控制器 <design/runner>`。

### 数据相关配置

#### 数据集配置

主要用于配置两个方向：

- 数据集的图像与标注文件的位置。

- 数据增强相关的配置。在 OCR 领域中，数据增强通常与模型强相关。

更多参数配置可以参考[数据基类](#TODO)。

数据集字段的命名规则在 MMOCR 中为：

```Python
{数据集名称缩写}_{算法任务}_{训练/测试/验证} = dict(...)
```

- 数据集缩写：见 [数据集名称对应表](#TODO)

- 算法任务：文本检测-det，文字识别-rec，关键信息提取-kie

- 训练/测试/验证：数据集用于训练，测试还是验证

以识别为例，使用 Syn90k 作为训练集，以 icdar2013 和 icdar2015 作为测试集配置如下：

```Python
# 识别数据集配置

```

```Python
# 识别数据集配置

```

#### 数据流水线配置

data_prefix=dict(img_path='ch4_test_word_images_gt/'),
```

MMOCR 中，数据集的构建与数据准备是相互解耦的。也就是说，`OCRDataset` 等数据集构建类负责完成标注文件的读取与解析功能；而数据变换方法（Data Transforms）则进一步实现了数据读取、数据增强、数据格式化等相关功能。

同时一般情况下训练和测试会存在不同的增强策略，因此一般会存在训练流水线（train_pipeline）和测试流水线（test_pipeline）。更多信息可以参考[数据流水线](../basic_concepts/transforms.md)

- 训练流水线的数据增强流程通常为：数据读取(LoadImageFromFile)->标注信息读取(LoadXXXAnntation)->数据增强->数据格式化(PackXXXInputs)。

- 测试流水线的数据增强流程通常为：数据读取(LoadImageFromFile)->数据增强->标注信息读取(LoadXXXAnntation)->数据格式化(PackXXXInputs)。

由于 OCR 任务的特殊性，一般情况下不同模型有不同数据增强的方式，相同模型在不同数据集一般也会有不同的数据增强方式。以 CRNN 为例：

```Python
# 数据增强
    dict(
    dict(

ignore_empty=True,
    dict(
]
    dict(
    dict(
    dict(
]
```

<div id="dataloader_config"></div>

#### Dataloader 配置

主要为构造数据集加载器(dataloader)所需的配置信息，更多教程看参考 {external+torch:doc}`PyTorch 数据加载器 <data>`。

```Python
# Dataloader 部分

```Python
```

### 模型相关配置

#### 网络配置

用于配置模型的网络结构，不同的算法任务有不同的网络结构。更多信息可以参考[网络结构](../basic_concepts/structures.md)

##### 文本检测

文本检测主要包含几个部分:

- `data_preprocessor`: [数据处理器](mmocr.models.textdet.data_preprocessors.TextDetDataPreprocessor)
- `backbone`: 特征提取网络
- `neck`: 颈网络配置
- `det_head`: 检测头网络配置
  - `module_loss`: 模型损失函数配置

  - `postprocessor`: 模型预测结果后处理配置

我们以 DBNet 为例，介绍文字检测中模型配置：

```Python
        depth=18,

style='caffe'),
```

##### 文本识别

文本识别主要包含：

- `data_processor`: [数据预处理配置](mmocr.models.textrec.data_processors.TextRecDataPreprocessor)
- `preprocessor`: 网络预处理配置，如TPS等
- `backbone`:特征提取配置
- `encoder`: 编码器配置
- `decoder`: 解码器配置
  - `module_loss`: 解码器损失
  - `postprocessor`: 解码器后处理
  - `dictionary`: 字典配置

以 CRNN 为例：

```Python
# 模型部分

backbone=dict(type='VeryDeepVgg', leaky_relu=False, input_channels=1),
```

#### 权重加载配置

可以通过 `load_from` 参数加载检查点（checkpoint）文件中的模型权重，只需要将 `load_from` 参数设置为检查点文件的路径即可。

用户也可通过设置 `resume=True` ，加载检查点中的训练状态信息来恢复训练。当 `load_from` 和 `resume=True` 同时被设置时，执行器将加载 `load_from` 路径对应的检查点文件中的训练状态。

如果仅设置 `resume=True`，执行器将会尝试从 `work_dir` 文件夹中寻找并读取最新的检查点文件

```Python
load_from = None # 加载checkpoint的路径
resume = False # 是否 resume
```

更多可以参考 {external+mmengine:ref}`MMEngine: 加载权重或恢复训练 <加载权重或恢复训练>` 与 [OCR 进阶技巧-断点恢复训练](train_test.md#从断点恢复训练)。

### 评测配置

```

### 评测配置

在模型验证和模型测试中，通常需要对模型精度做定量评测。MMOCR 通过评测指标(Metric)和评测器(Evaluator)来完成这一功能。更多可以参考{external+mmengine:doc}`MMEngine: 评测指标(Metric)和评测器(Evaluator) <tutorials/evaluation>` 和 [评测器](../basic_concepts/evaluation.md)

评测部分包含两个部分，评测器和评测指标。接下来我们分部分展开讲解。

#### 评测器

评测器主要用来管理多个数据集以及多个 `Metric`。针对单数据集与多数据集情况，评测器分为了单数据集评测器与多数据集评测器，这两种评测器均可管理多个 `Metric`.

单数据集评测器配置如下：

```Python
# 单个数据集 单个 Metric 情况

# 单个数据集 多个 Metric 情况
```

在实现中默认为单数据集评测器，因此对单数据集评测情况下，一般情况下只需配置评测器，即为

```Python
val_evaluator = dict()

val_evaluator = [...]
```

type='Evaluator',
```

```Python

```

多数据集评测与单数据集评测存在两个位置上的不同：评测器类别与前缀。评测器类别必须为`MultiDatasetsEvaluator`且不能省略，前缀主要用来区分不同数据集在相同评测指标下的结果，请参考[多数据集评测](../basic_concepts/evaluation.md)。

假设我们需要在 IC13 和 IC15 情况下测试精度，则配置如下：

```Python
# 多个数据集，单个 Metric 情况

# 多个数据集，多个 Metric 情况
```

#### 评测指标

评测指标指不同度量精度的方法，同时可以多个评测指标共同使用，更多评测指标原理参考 {external+mmengine:doc}`MMEngine: 评测指标 <tutorials/evaluation>`，在 MMOCR 中不同算法任务有不同的评测指标。 更多 OCR 相关的评测指标可以参考 [评测指标](../basic_concepts/evaluation.md)。

文字检测: [`HmeanIOUMetric`](mmocr.evaluation.metrics.HmeanIOUMetric)

文字识别: [`WordMetric`](mmocr.evaluation.metrics.WordMetric)，[`CharMetric`](mmocr.evaluation.metrics.CharMetric)， [`OneMinusNEDMetric`](mmocr.evaluation.metrics.OneMinusNEDMetric)

关键信息提取: [`F1Metric`](mmocr.evaluation.metrics.F1Metric)

以文本检测为例说明，在单数据集评测情况下，使用单个 `Metric`：

```Python
```

以文本识别为例，对多个数据集(IC13 和 IC15)用多个 `Metric` (`WordMetric` 和 `CharMetric`)进行评测：

```Python
# 评测部分
    metrics=[
        dict(
    ],
```

### 可视化配置

每个任务配置该任务对应的可视化器。可视化器主要用于用户模型中间结果的可视化或存储，及 val 和 test 预测结果的可视化。同时可视化的结果可以通过可视化后端储存到不同的后端，比如 WandB，TensorBoard 等。常用修改操作可见[可视化](visualization.md)。

文本检测的可视化默认配置如下：

```Python

### 可视化配置

```Python
    type='TextDetLocalVisualizer',  # 不同任务有不同的可视化器
```

## 目录结构

`MMOCR` 所有配置文件都放置在 `configs` 文件夹下。为了避免配置文件过长，同时提高配置文件的可复用性以及清晰性，MMOCR 利用 Config 文件的继承特性，将配置内容的八个部分做了拆分。因为每部分均与算法任务相关，因此 MMOCR 对每个任务在 Config 中提供了一个任务文件夹，即 `textdet` (文字检测任务)、`textrecog` (文字识别任务)、`kie` (关键信息提取)。同时各个任务算法配置文件夹下进一步划分为两个部分：`_base_` 文件夹与诸多算法文件夹：

1. `_base_` 文件夹下主要存放与具体算法无关的一些通用配置文件，各部分依目录分为常用的数据集、常用的训练策略以及通用的运行配置。

2. 算法配置文件夹中存放与算法强相关的配置项。算法配置文件夹主要分为两部分：

   1. 算法的模型与数据流水线：OCR 领域中一般情况下数据增强策略与算法强相关，因此模型与数据流水线通常置于统一位置。

   2. 算法在制定数据集上的特定配置：用于训练和测试的配置，将分散在不同位置的 *base* 配置汇总。同时可能会修改一些`_base_`中的变量，如batch size, 数据流水线，训练策略等

最后的将配置内容中的各个模块分布在不同配置文件中，最终各配置文件内容如下:

2. 算法在制定数据集上的特定配置：用于训练和测试的配置，将分散在不同位置的 *base* 配置汇总。同时可能会修改一些`_base_`中的变量，如batch size, 数据流水线，训练策略等

</style>

<thead>
  <tr>

</style>

<thead>
  <tr>
    <td class="tg-0pky"><a href="#dataset_config">数据集配置</a></td>
  </tr>
  <tr>
    <td class="tg-0pky"><a href="#schedule_config">训练策略配置</a></td>
  </tr>
  <tr>
    <td class="tg-0pky"><a href="#env_config">环境配置</a><br><a href="#hook_config">默认hook配置</a><br><a href="#log_config">日志配置</a> <br><a href="#weight_config">权重加载配置</a> <br><a href="#eval_config">评测配置</a> <br><a href="#vis_config">可视化配置</a></td>
  </tr>
  <tr>

</tr>
  <tr>
    <td class="tg-0pky"><a href="#model_config">网络配置</a> <br><a href="#pipeline_config">数据流水线</a></td>
  </tr>
  <tr>
    <td class="tg-0pky"><a href="#dataloader_config">Dataloader 配置</a> <br><a href="#pipeline_config">数据流水线(Optional)</a></td>
  </tr>
</thead>
</table>

最终目录结构如下：

```Python
configs
│   └── dbnet

│   └── dbnet
│   └── crnn
└── kie
    ├── _base_
    └── sgdmr
```

## 配置文件以及权重命名规则

MMOCR 按照以下风格进行配置文件命名，代码库的贡献者需要遵循相同的命名规则。文件名总体分为四部分：算法信息，模块信息，训练信息和数据信息。逻辑上属于不同部分的单词之间用下划线 `'_'` 连接，同一部分有多个单词用短横线 `'-'` 连接。

```Python
```

- 算法信息(algorithm info)：算法名称，如 dbnet, crnn 等

- 模块信息(module info)：按照数据流的顺序列举一些中间的模块，其内容依赖于算法任务，同时为了避免Config过长，会省略一些与模型强相关的模块。下面举例说明：

```Python
```

  - 对于文字检测任务和关键信息提取任务:

    ```Python
    {{算法信息}}_{{backbone}}_{{neck}}_{{head}}_{{训练信息}}_{{数据信息}}.py
    ```

    一般情况下 head 位置一般为算法专有的 head，因此一般省略。

  - 对于文本识别任务：

    ```Python
    {{算法信息}}_{{backbone}}_{{encoder}}_{{decoder}}_{{训练信息}}_{{数据信息}}.py
    ```

    一般情况下 encoder 和 decoder 位置一般为算法专有，因此一般省略。

- 训练信息(training info)：训练策略的一些设置，包括 batch size，schedule 等

- 数据信息(data info)：数据集名称、模态、输入尺寸等，如 icdar2015，synthtext 等

# BigGAN (ICLR'2019)

> [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://openreview.net/forum?id=B1xsqj09Fm)

Despite recent progress in generative image modeling, successfully generating high-resolution, diverse samples from complex datasets such as ImageNet remains an elusive goal. To this end, we train Generative Adversarial Networks at the largest scale yet attempted, and study the instabilities specific to such scale. We find that applying orthogonal regularization to the generator renders it amenable to a simple "truncation trick," allowing fine control over the trade-off between sample fidelity and variety by reducing the variance of the Generator's input. Our modifications lead to models which set the new state of the art in class-conditional image synthesis. When trained on ImageNet at 128x128 resolution, our models (BigGANs) achieve an Inception Score (IS) of 166.5 and Frechet Inception Distance (FID) of 7.4, improving over the previous best IS of 52.52 and FID of 18.6.

<img src="https://user-images.githubusercontent.com/28132635/143154280-4cb22e16-92c8-4b34-9e2c-6357ed0bdac8.png"/>
</div>

The `BigGAN/BigGAN-Deep` is a conditional generation model that can generate both high-resolution and high-quality images by scaling up the batch size and the number of model parameters.

We have finished training `BigGAN` in `Cifar10` (32x32) and are aligning training performance in `ImageNet1k` (128x128). Some sampled results are shown below for your reference.

  <b> Results from our BigGAN trained in CIFAR10</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/22982797/126476913-3ce8e2c8-f189-4caa-90ed-b44e279cb669.png" width="800"/>
</div>

  <b> Results from our BigGAN trained in ImageNet</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/22982797/127615534-6278ce1b-5cff-4189-83c6-9ecc8de08dfc.png" width="800"/>
</div>

<b> Results from our BigGAN trained in ImageNet</b>
  <br/>
</div>

Evaluation of our trained BigGAN.

|                                    Model                                    |  Dataset   |    FID (Iter)     |      IS (Iter)      |                                    Download                                     |
| :-------------------------------------------------------------------------: | :--------: | :---------------: | :-----------------: | :-----------------------------------------------------------------------------: |

|          [BigGAN 32x32](./biggan_2xb25-500kiters_cifar10-32x32.py)          |  CIFAR10   |   9.78(390000)    |    8.70(390000)     | [model](https://download.openmmlab.com/mmediting/biggan/biggan_cifar10_32x32_b25x2_500k_20210728_110906-08b61a44.pth)\|[log](https://download.openmmlab.com/mmediting/biggan/biggan_cifar10_32_b25x2_500k_20210706_171051.log.json) |
| [BigGAN 128x128 Best FID](./biggan_ajbrock-sn_8xb32-1500kiters_imagenet1k-128x128.py) | ImageNet1k | **8.69**(1232000) |   101.15(1232000)   | [model](https://download.openmmlab.com/mmediting/biggan/biggan_imagenet1k_128x128_b32x8_best_fid_iter_1232000_20211111_122548-5315b13d.pth)\|[log](https://download.openmmlab.com/mmediting/biggan/biggan_imagenet1k_128x128_b32x8_1500k_20211111_122548-5315b13d.log.json) |

| [BigGAN 128x128 Best IS](./biggan_ajbrock-sn_8xb32-1500kiters_imagenet1k-128x128.py) | ImageNet1k |  13.51(1328000)   | **129.07**(1328000) | [model](https://download.openmmlab.com/mmediting/biggan/biggan_imagenet1k_128x128_b32x8_best_is_iter_1328000_20211111_122911-28c688bc.pth)\|[log](https://download.openmmlab.com/mmediting/biggan/biggan_imagenet1k_128x128_b32x8_1500k_20211111_122548-5315b13d.log.json) |

### Note on reproducibility

`BigGAN 128x128` model is trained with V100 GPUs and CUDA 10.1 and can hardly reproduce the result with A100 and CUDA 11.3. If you have any idea about the reproducibility, please feel free to contact with us.

## Converted weights

Since we haven't finished training our models, we provide you with several pre-trained weights which have been evaluated. Here, we refer to [BigGAN-PyTorch](https://github.com/ajbrock/BigGAN-PyTorch) and [pytorch-pretrained-BigGAN](https://github.com/huggingface/pytorch-pretrained-BigGAN).

Evaluation results and download links are provided below.

|                        Model                         |  Dataset   |   FID   |   IS    |                        Download                         |                        Original Download link                         |
| :--------------------------------------------------: | :--------: | :-----: | :-----: | :-----------------------------------------------------: | :-------------------------------------------------------------------: |
| [BigGAN 128x128](./biggan_cvt-BigGAN-PyTorch-rgb_imagenet1k-128x128.py) | ImageNet1k | 10.1414 | 96.728  | [model](https://download.openmmlab.com/mmediting/biggan/biggan_imagenet1k_128x128_cvt_BigGAN-PyTorch_rgb_20210730_125223-3e353fef.pth) | [link](https://drive.google.com/open?id=1nAle7FCVFZdix2--ks0r5JBkFnKw8ctW) |

| [BigGAN-Deep 128x128](./biggan-deep_cvt-hugging-face-rgb_imagenet1k-128x128.py) | ImageNet1k | 5.9471  | 107.161 | [model](https://download.openmmlab.com/mmediting/biggan/biggan-deep_imagenet1k_128x128_cvt_hugging-face_rgb_20210728_111659-099e96f9.pth) | [link](https://s3.amazonaws.com/models.huggingface.co/biggan/biggan-deep-128-pytorch_model.bin) |
| [BigGAN-Deep 256x256](./biggan-deep_cvt-hugging-face_rgb_imagenet1k-256x256.py) | ImageNet1k | 11.3151 | 135.107 | [model](https://download.openmmlab.com/mmediting/biggan/biggan-deep_imagenet1k_256x256_cvt_hugging-face_rgb_20210728_111735-28651569.pth) | [link](https://s3.amazonaws.com/models.huggingface.co/biggan/biggan-deep-256-pytorch_model.bin) |

| [BigGAN-Deep 512x512](./biggan-deep_cvt-hugging-face_rgb_imagenet1k-512x512.py) | ImageNet1k | 16.8728 | 124.368 | [model](https://download.openmmlab.com/mmediting/biggan/biggan-deep_imagenet1k_512x512_cvt_hugging-face_rgb_20210728_112346-a42585f2.pth) | [link](https://s3.amazonaws.com/models.huggingface.co/biggan/biggan-deep-512-pytorch_model.bin) |

Sampling results are shown below.

  <b> Results from our BigGAN-Deep with Pre-trained weights in ImageNet 128x128 with truncation factor 0.4</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/22982797/126481730-8da7180b-7b1b-42f0-9bec-78d879b6265b.png" width="800"/>
</div>

  <b> Results from our BigGAN-Deep with Pre-trained weights in ImageNet 256x256 with truncation factor 0.4</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/22982797/126486040-64effa29-959e-4e43-bcae-15925a2e0599.png" width="800"/>
</div>

<br/>
</div>

  <b> Results from our BigGAN-Deep with Pre-trained weights in ImageNet 512x512 truncation factor 0.4</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/22982797/126487428-50101454-59cb-469d-a1f1-36ffb6291582.png" width="800"/>
</div>
Sampling with truncation trick above can be performed by command below.

```bash
python demo/conditional_demo.py CONFIG_PATH CKPT_PATH --sample-cfg truncation=0.4 # set truncation value as you want
```

For converted weights, we provide model configs under `configs/_base_/models` listed as follows:

```bash
# biggan_cvt-BigGAN-PyTorch-rgb_imagenet1k-128x128.py
# biggan-deep_cvt-hugging-face-rgb_imagenet1k-128x128.py
# biggan-deep_cvt-hugging-face_rgb_imagenet1k-256x256.py
# biggan-deep_cvt-hugging-face_rgb_imagenet1k-512x512.py
```

```

To perform image Interpolation on BigGAN(or other conditional models), run

```bash
python apps/conditional_interpolate.py CONFIG_PATH  CKPT_PATH  --samples-path SAMPLES_PATH
```

  <b> Image interpolating Results of our BigGAN-Deep</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/22982797/126580403-2baa987b-ff55-4fb5-a53a-b08e8a6a72a2.png" width="800"/>
</div>

To perform image Interpolation on BigGAN with fixed noise, run

```bash
python apps/conditional_interpolate.py CONFIG_PATH  CKPT_PATH  --samples-path SAMPLES_PATH --fix-z
```

  <b> Image interpolating Results of our BigGAN-Deep with fixed noise</b>
  <br/>

```

  <br/>
  <img src="https://user-images.githubusercontent.com/22982797/128123804-6df1dfca-1057-4b96-8428-787a86f81ef1.png" width="800"/>
</div>
To perform image Interpolation on BigGAN with fixed label, run

```bash
python apps/conditional_interpolate.py CONFIG_PATH  CKPT_PATH  --samples-path SAMPLES_PATH --fix-y
```

  <b> Image interpolating Results of our BigGAN-Deep with fixed label</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/22982797/128124596-421396f1-3f23-4098-b629-b00d29d710a9.png" width="800"/>
</div>

```latex
    brock2018large,
    title={Large Scale {GAN} Training for High Fidelity Natural Image Synthesis},
    author={Andrew Brock and Jeff Donahue and Karen Simonyan},

author={Andrew Brock and Jeff Donahue and Karen Simonyan},
    url={https://openreview.net/forum?id=B1xsqj09Fm},
}
```

# ATSS

> [Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection](https://arxiv.org/abs/1912.02424)

<img src="https://user-images.githubusercontent.com/40661020/143870776-c81168f5-e8b2-44ee-978b-509e4372c5c9.png"/>
</div>

| Backbone |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                Config                |                                                                                                                            Download                                                                                                                             |
| :------: | :-----: | :-----: | :------: | :------------: | :----: | :----------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

|   R-50   | pytorch |   1x    |   3.7    |      19.7      |  39.4  | [config](./atss_r50_fpn_1x_coco.py)  | [model](https://download.openmmlab.com/mmdetection/v2.0/atss/atss_r50_fpn_1x_coco/atss_r50_fpn_1x_coco_20200209-985f7bd0.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/atss/atss_r50_fpn_1x_coco/atss_r50_fpn_1x_coco_20200209_102539.log.json) |
|  R-101   | pytorch |   1x    |   5.6    |      12.3      |  41.5  | [config](./atss_r101_fpn_1x_coco.py) |   [model](https://download.openmmlab.com/mmdetection/v2.0/atss/atss_r101_fpn_1x_coco/atss_r101_fpn_1x_20200825-dfcadd6f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/atss/atss_r101_fpn_1x_coco/atss_r101_fpn_1x_20200825-dfcadd6f.log.json)   |

```latex
@article{zhang2019bridging,
  title   =  {Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection},
  author  =  {Zhang, Shifeng and Chi, Cheng and Yao, Yongqiang and Lei, Zhen and Li, Stan Z.},

author  =  {Zhang, Shifeng and Chi, Cheng and Yao, Yongqiang and Lei, Zhen and Li, Stan Z.},
  journal =  {arXiv preprint arXiv:1912.02424},
}
```

# Ubuntu18.04 交叉编译 NDK snpe 推理服务

mmdeploy 已提供预编译包，如果你想自己编译、或需要对 .proto 接口做修改，可参考此文档。

注意 gRPC 官方文档并没有对 NDK 的完整支持。

## 一、环境说明

| 项目     | 版本           | 备注                                  |
| -------- | -------------- | ------------------------------------- |
| snpe     | 1.59           | 1.60 使用 clang-8.0，可能导致兼容问题 |
| host OS  | ubuntu18.04    | snpe1.59 指定版本                     |
| NDK      | r17c           | snpe1.59 指定版本                     |
| gRPC     | commit 6f698b5 | -                                     |
| 硬件设备 | qcom888        | 需要 qcom 芯片                        |

## 二、NDK 交叉编译 gRPC

1. 拉取 gRPC repo,  在 host 上编译出 `protoc` 和 `grpc_cpp_plugin`

```bash
# 安装依赖
# 编译

$ cmake \
  ../..
# 需要安装到 host 环境
$ make -j

$ cmake \
  ../..
$ make -j
```

2. 下载 NDK，交叉编译 android aarch64 所需静态库

```bash

# 设置环境变量

# 编译

$ make -j
```

3. 此时 `/tmp/android_grpc_install` 应有完整的安装文件

```bash
$ tree -L 1
.
├── bin
├── lib

$ make -j
```

```bash
$ tree -L 1
.
├── bin
├── lib
└── share
```

## 三、【可跳过】自测 NDK gRPC 是否正常

1. 编译 gRPC 自带的 helloworld

```bash

$ make -j
```

2. 打开手机调试模式，push 编译结果到 `/data/local/tmp` 目录

```

tips：对于国产手机，设置 - 版本号，点击 7 次可进入开发者模式，然后才能打开 USB 调试

```bash
```

3. `adb shell` 进手机，执行 client/server

```bash
```

## 四、交叉编译 snpe 推理服务

1. 打开 [snpe tools 官网](https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk/tools)，下载 1.59 版本。 解压并设置环境变量

**注意 snpe >= 1.60 开始使用 `clang-8.0`，可能导致旧设备与 `libc++_shared.so` 不兼容。**

```bash
```

2. 打开 mmdeploy  snpe server 目录，使用交叉编译 gRPC 时的选项

```bash

$ cmake .. \
 -DCMAKE_TOOLCHAIN_FILE=${ANDROID_NDK_ROOT}/build/cmake/android.toolchain.cmake \

$ cmake .. \

 $ make -j
```

最终可得到 `infernece_server`，`adb push` 到设备上即可执行。

## 五、重新生成 proto 接口

如果改过 `inference.proto`，需要重新生成 .cpp 和 .py 通信接口

```Shell

```

```Shell

```

## 参考文档

This tutorial provides instructions for users to use the pre-trained models
to finetune them on other datasets, so that better performance can be achieved.

- [Finetuning Models](#finetuning-models)
  - [Choose Template Config](#choose-template-config)
  - [Modify Head](#modify-head)
  - [Modify Dataset](#modify-dataset)
  - [Modify Training Schedule](#modify-training-schedule)
  - [Use Pre-Trained Model](#use-pre-trained-model)
  - [Start Training](#start-training)

## Outline

1. Add support for the new dataset. See [Prepare Dataset](prepare_dataset.md) and [Customize Dataset](../advanced_guides/customize_dataset.md).
2. Modify the configs. This will be discussed in this tutorial.

## Choose Template Config

Here, we would like to take `configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py` as an example. We first copy this config file to the same folder and rename it to `tsn_ucf101.py`, then four parts in the config need attention, specifically, add new keys for non-existing keys and modify the original keys for existing keys.

## Modify Head

The `num_classes` in the `cls_head` need to be changed to the class number of the new dataset.
The weights of the pre-trained models are reused except for the final prediction layer.
So it is safe to change the class number.
In our case, UCF101 has 101 classes.
So we change it from 400 (class number of Kinetics-400) to 101.

```python
        num_classes=101  # change from 400 to 101
        ))
```

## Modify Dataset

```python
        ))
```

MMAction2 supports UCF101, Kinetics-400, Moments in Time, Multi-Moments in Time, THUMOS14,
Something-Something V1&V2, ActivityNet Dataset.
The users may need to adapt one of the above datasets to fit their special datasets.
You could refer to [Prepare Dataset](prepare_dataset.md) and [Customize Datast](../advanced_guides/customize_dataset.md) for more details.
In our case, UCF101 is already supported by various dataset types, like `VideoDataset`,
so we change the config as follows.

```python
```

## Modify Training Schedule

```

Finetuning usually requires a smaller learning rate and fewer training epochs.

```python
    max_epochs=50,  # change from 100 to 50

    dict(
        begin=0,
        end=50,  # change from 100 to 50
        milestones=[20, 40],  # change milestones
]

        lr=0.005, # change from 0.01 to 0.005
```

## Use Pre-Trained Model

type='SGD',
```

To use the pre-trained model for the whole network, the new config adds the link of pre-trained models in the `load_from`.
We set `load_from=None` as default in `configs/_base_/default_runtime.py` and owing to [inheritance design](config.md), users can directly change it by setting `load_from` in their configs.

```python
load_from = 'https://download.openmmlab.com/mmaction/v1.0/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb_20220906-cd10898e.pth'  # model path can be found in model zoo
```

Now, we have finished the fine-tuning config file as follows:

```python
_base_ = [

```

```python
_base_ = [
]

        ))

    dict(

dict(type='Resize', scale=(-1, 256)),
    dict(
]
    dict(
]
    dict(

]
    dict(
]

sampler=dict(type='DefaultSampler', shuffle=False),

    dict(
        begin=0,
]

type='MultiStepLR',
        begin=0,
]

```

An easier way is to inherit the kinetics400 config and only specify the modified keys. Please make sure that the custom config is in the same folder with `configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py`.

```python
_base_ = [
    'tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py'  # inherit template config
]

        num_classes=101))  # change from 400 to 101

]

    dict(
        begin=0,

val_begin=1,

    dict(
        begin=0,
]

```

You can use the following command to finetune a model on your dataset.

```shell
```

```

```shell
```

Example: train the TSN model on Kinetics-400 dataset in a deterministic option.

```shell
```

For more details, you can refer to the **Training** part in the [Training and Test Tutorial](train_test.md).

## v1.0.0 (04/06/2023)

We are excited to announce the first official release of MMOCR 1.0, with numerous enhancements, bug fixes, and the introduction of new dataset support!

### 🌟 Highlights

- Support for SCUT-CTW1500, SynthText, and MJSynth datasets
- Updated FAQ and documentation
- Deprecation of file_client_args in favor of backend_args
- Added a new MMOCR tutorial notebook

### 🆕 New Features & Enhancement

- Add SCUT-CTW1500 by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/1677
- Cherry Pick #1205 by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1774
- Make lanms-neo optional by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1772
- SynthText by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1779
- Deprecate file_client_args and use backend_args instead by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1765
- MJSynth by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1791

- Add MMOCR tutorial notebook by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1771
- decouple batch_size to det_batch_size, rec_batch_size and kie_batch_size in MMOCRInferencer by @hugotong6425 in https://github.com/open-mmlab/mmocr/pull/1801
- Accepts local-rank in train.py and test.py by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1806
- update stitch_boxes_into_lines by @cherryjm in https://github.com/open-mmlab/mmocr/pull/1824
- Add tests for pytorch 2.0 by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1836

### 📝 Docs

- FAQ by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1773
- Remove LoadImageFromLMDB from docs by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1767

### 📝 Docs

- Mark projects in docs by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1766
- add opendatalab download link by @jorie-peng in https://github.com/open-mmlab/mmocr/pull/1753
- Fix some deadlinks in the docs by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1469
- Fix quick run by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1775
- Dataset by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1782
- Update faq by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1817
- more social network links by @fengshiwest in https://github.com/open-mmlab/mmocr/pull/1818
- Update docs after branch switching by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1834

### 🛠️ Bug Fixes:

- Place dicts to .mim by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1781
- Test svtr_small instead of svtr_tiny by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1786
- Add pse weight to metafile by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1787
- Synthtext metafile by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1788
- Clear up some unused scripts by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1798
- if dst not exists, when move a single file may raise a file not exists error. by @KevinNuNu in https://github.com/open-mmlab/mmocr/pull/1803
- CTW1500 by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1814
- MJSynth & SynthText Dataset Preparer config by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1805

- Use poly_intersection instead of poly.intersection to avoid sup… by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1811
- Abinet: fix ValueError: Blur limit must be odd when centered=True. Got: (3, 6) by @hugotong6425 in https://github.com/open-mmlab/mmocr/pull/1821
- Bug generated during kie inference visualization by @Yangget in https://github.com/open-mmlab/mmocr/pull/1830
- Revert sync bn in inferencer by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1832
- Fix mmdet digit version by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1840

### 🎉 New Contributors

- @jorie-peng made their first contribution in https://github.com/open-mmlab/mmocr/pull/1753
- @hugotong6425 made their first contribution in https://github.com/open-mmlab/mmocr/pull/1801

- @fengshiwest made their first contribution in https://github.com/open-mmlab/mmocr/pull/1818
- @cherryjm made their first contribution in https://github.com/open-mmlab/mmocr/pull/1824
- @Yangget made their first contribution in https://github.com/open-mmlab/mmocr/pull/1830

Thank you to all the contributors for making this release possible! We're excited about the new features and enhancements in this version, and we're looking forward to your feedback and continued support. Happy coding! 🚀

**Full Changelog**: https://github.com/open-mmlab/mmocr/compare/v1.0.0rc6...v1.0.0

## v1.0.0rc6 (03/07/2023)

1. Two new models, ABCNet v2 (inference only) and SPTS are added to `projects/` folder.

2. Announcing `Inferencer`, a unified inference interface in OpenMMLab for everyone's easy access and quick inference with all the pre-trained weights. [Docs](https://mmocr.readthedocs.io/en/dev-1.x/user_guides/inference.html)
3. Users can use test-time augmentation for text recognition tasks. [Docs](https://mmocr.readthedocs.io/en/dev-1.x/user_guides/train_test.html#test-time-augmentation)
4. Support [batch augmentation](https://openaccess.thecvf.com/content_CVPR_2020/papers/Hoffer_Augment_Your_Batch_Improving_Generalization_Through_Instance_Repetition_CVPR_2020_paper.pdf) through [`BatchAugSampler`](https://github.com/open-mmlab/mmocr/pull/1757), which is a technique used in SPTS.

5. Dataset Preparer has been refactored to allow more flexible configurations. Besides, users are now able to prepare text recognition datasets in LMDB formats. [Docs](https://mmocr.readthedocs.io/en/dev-1.x/user_guides/data_prepare/dataset_preparer.html#lmdb-format)
6. Some textspotting datasets have been revised to enhance the correctness and consistency with the common practice.
7. Potential spurious warnings from `shapely` have been eliminated.

### Dependency

This version requires MMEngine >= 0.6.0, MMCV >= 2.0.0rc4 and MMDet >= 3.0.0rc5.

- Discard deprecated lmdb dataset format and only support img+label now by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1681
- abcnetv2 inference by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1657
- Add RepeatAugSampler by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1678
- SPTS by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1696

- Refactor Inferencers by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1608
- Dynamic return type for rescale_polygons by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1702
- Revise upstream version limit by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1703
- TextRecogCropConverter add crop with opencv warpPersepective function by @KevinNuNu in https://github.com/open-mmlab/mmocr/pull/1667
- change cudnn benchmark to false by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1705
- Add ST-pretrained DB-series models and logs by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1635
- Only keep meta and state_dict when publish model by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1729
- Rec TTA by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1401

- Speedup formatting by replacing np.transpose with torch… by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1719
- Support auto import modules from registry. by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1731
- Support batch visualization & dumping in Inferencer by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1722
- add a new argument font_properties to set a specific font file in order to draw Chinese characters properly by @KevinNuNu in https://github.com/open-mmlab/mmocr/pull/1709
- Refactor data converter and gather by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1707
- Support batch augmentation through BatchAugSampler by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1757

- Put all registry into registry.py by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1760
- train by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1756
- configs for regression benchmark by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1755
- Support lmdb format in Dataset Preparer by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1762

### Docs

- update the link of DBNet by @AllentDan in https://github.com/open-mmlab/mmocr/pull/1672
- Add notice for default branch switching by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1693
- docs: Add twitter discord medium youtube link by @vansin in https://github.com/open-mmlab/mmocr/pull/1724
- Remove unsupported datasets in docs by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1670

- Update dockerfile by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1671
- Explicitly create np object array for compatibility by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1691
- Fix a minor error in docstring by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/1685
- Fix lint by @triple-Mu in https://github.com/open-mmlab/mmocr/pull/1694
- Fix LoadOCRAnnotation ut by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1695
- Fix isort pre-commit error by @KevinNuNu in https://github.com/open-mmlab/mmocr/pull/1697
- Update owners by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1699
- Detect intersection before using shapley.intersection to eliminate spurious warnings by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1710

- Fix some inferencer bugs by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1706
- Fix textocr ignore flag by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1712
- Add missing softmax in ASTER forward_test by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/1718
- Fix head in readme by @vansin in https://github.com/open-mmlab/mmocr/pull/1727
- Fix some browse dataset script bugs and draw textdet gt instance with ignore flags by @KevinNuNu in https://github.com/open-mmlab/mmocr/pull/1701
- icdar textrecog ann parser skip data with ignore flag by @KevinNuNu in https://github.com/open-mmlab/mmocr/pull/1708
- bezier_to_polygon ->  bezier2polygon by @double22a in https://github.com/open-mmlab/mmocr/pull/1739

- Fix docs recog CharMetric P/R error definition by @KevinNuNu in https://github.com/open-mmlab/mmocr/pull/1740
- Remove outdated resources in demo/ by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1747
- Fix wrong ic13 textspotting split data; add lexicons to ic13, ic15 and totaltext by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1758
- SPTS readme by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1761

- @triple-Mu made their first contribution in https://github.com/open-mmlab/mmocr/pull/1694
- @double22a made their first contribution in https://github.com/open-mmlab/mmocr/pull/1739

**Full Changelog**: https://github.com/open-mmlab/mmocr/compare/v1.0.0rc5...v1.0.0rc6

## v1.0.0rc5 (01/06/2023)

1. Two models, Aster and SVTR, are added to our model zoo. The full implementation of ABCNet is also available now.

2. Dataset Preparer supports 5 more datasets: CocoTextV2, FUNSD, TextOCR, NAF, SROIE.
3. We have 4 more text recognition transforms, and two helper transforms. See https://github.com/open-mmlab/mmocr/pull/1646 https://github.com/open-mmlab/mmocr/pull/1632 https://github.com/open-mmlab/mmocr/pull/1645 for details.
4. The transform, `FixInvalidPolygon`, is getting smarter at dealing with invalid polygons, and now capable of handling more weird annotations. As a result, a complete training cycle on TotalText dataset can be performed bug-free. The weights of DBNet and FCENet pretrained on TotalText are also released.

- Update ic15 det config according to DataPrepare by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1617

- Refactor icdardataset metainfo to lowercase. by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1620
- Add ASTER Encoder by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/1239
- Add ASTER decoder by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/1625
- Add ASTER config by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/1238
- Update ASTER config by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/1629
- Support browse_dataset.py to visualize original dataset by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1503
- Add CocoTextv2 to dataset preparer by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1514
- Add Funsd to dataset preparer by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1550

- Add TextOCR to Dataset Preparer by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1543
- Refine example projects and readme by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1628
- Enhance FixInvalidPolygon, add RemoveIgnored transform by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1632
- ConditionApply by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1646
- Add NAF to dataset preparer by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/1609
- Add SROIE to dataset preparer by @FerryHuang in https://github.com/open-mmlab/mmocr/pull/1639
- Add svtr decoder by @willpat1213 in https://github.com/open-mmlab/mmocr/pull/1448
- Add missing unit tests by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/1651

- Add svtr encoder by @willpat1213 in https://github.com/open-mmlab/mmocr/pull/1483
- ABCNet train by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1610
- Totaltext cfgs for DB and FCE by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1633
- Add Aliases to models by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1611
- SVTR transforms by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1645
- Add SVTR framework and configs by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1621
- Issue Template by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1663

### Docs

- Add Chinese translation for browse_dataset.py by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1647
- updata abcnet doc by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1658

- update the dbnetpp\`s readme file by @zhuyue66 in https://github.com/open-mmlab/mmocr/pull/1626
- Inferencer docs by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1744

- nn.SmoothL1Loss beta can not be zero in PyTorch 1.13 version by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1616
- ctc loss bug if target is empty by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1618
- Add torch 1.13 by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1619
- Remove outdated tutorial link by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1627
- Dev 1.x  some doc mistakes by @KevinNuNu in https://github.com/open-mmlab/mmocr/pull/1630

- Support custom font to visualize some languages (e.g. Korean) by @ProtossDragoon in https://github.com/open-mmlab/mmocr/pull/1567
- db_module_loss，negative number encountered in sqrt by @KevinNuNu in https://github.com/open-mmlab/mmocr/pull/1640
- Use int instead of np.int by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1636
- Remove support for py3.6 by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1660

- @zhuyue66 made their first contribution in https://github.com/open-mmlab/mmocr/pull/1626
- @KevinNuNu made their first contribution in https://github.com/open-mmlab/mmocr/pull/1630
- @FerryHuang made their first contribution in https://github.com/open-mmlab/mmocr/pull/1639

- @willpat1213 made their first contribution in https://github.com/open-mmlab/mmocr/pull/1448

**Full Changelog**: https://github.com/open-mmlab/mmocr/compare/v1.0.0rc4...v1.0.0rc5

## v1.0.0rc4 (12/06/2022)

1. Dataset Preparer can automatically generate base dataset configs at the end of the preparation process, and supports 6 more datasets: IIIT5k, CUTE80, ICDAR2013, ICDAR2015, SVT, SVTP.

2. Introducing our `projects/` folder - implementing new models and features into OpenMMLab's algorithm libraries has long been complained to be troublesome due to the rigorous requirements on code quality, which could hinder the fast iteration of SOTA models and might discourage community members from sharing their latest outcome here. We now introduce `projects/` folder, where some experimental features, frameworks and models can be placed, only needed to satisfy the minimum requirement on the code quality. Everyone is welcome to post their implementation of any great ideas in this folder! We also add the first [example project](https://github.com/open-mmlab/mmocr/tree/dev-1.x/projects/example_project) to illustrate what we expect a good project to have (check out the raw content of README.md for more info!).

3. Inside the `projects/` folder, we are releasing the preview version of ABCNet, which is the first implementation of text spotting models in MMOCR. It's inference-only now, but the full implementation will be available very soon.

- Add SVT to dataset preparer by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1521
- Polish bbox2poly by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1532
- Add SVTP to dataset preparer by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1523
- Iiit5k converter by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1530
- Add cute80 to dataset preparer by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1522
- Add IC13 preparer by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1531
- Add 'Projects/' folder, and the first example project by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1524

- Rename to {dataset-name}\_task_train/test by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1541
- Add print_config.py to the tools by @IncludeMathH in https://github.com/open-mmlab/mmocr/pull/1547
- Add get_md5 by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1553
- Add config generator by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1552
- Support IC15_1811  by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1556
- Update CT80 config by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1555
- Add config generators to all textdet and textrecog configs by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1560
- Refactor TPS by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/1240
- Add TextSpottingConfigGenerator by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1561

- Add common typing by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1596
- Update textrecog config and readme  by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1597
- Support head loss or postprocessor is None for only infer by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1594
- Textspotting datasample by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1593
- Simplify mono_gather by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1588
- ABCNet v1 infer by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1598

### Docs

- Add Chinese Guidance on How to Add New Datasets to Dataset Preparer by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1506
- Update the qq group link by @vansin in https://github.com/open-mmlab/mmocr/pull/1569

- Collapse some sections; update logo url by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1571
- Update dataset preparer (CN) by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1591

- Fix two bugs in dataset preparer by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1513
- Register bug of CLIPResNet by @jyshee in https://github.com/open-mmlab/mmocr/pull/1517
- Being more conservative on Dataset Preparer by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1520
- python -m pip upgrade in windows by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1525
- Fix wildreceipt metafile by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1528
- Fix Dataset Preparer Extract by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1527
- Fix ICDARTxtParser by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1529

- Fix Dataset Zoo Script by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1533
- Fix crop without padding and recog metainfo delete unuse info by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1526
- Automatically create nonexistent directory for base  configs by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1535
- Change mmcv.dump to mmengine.dump by @ProtossDragoon in https://github.com/open-mmlab/mmocr/pull/1540
- mmocr.utils.typing -> mmocr.utils.typing_utils by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1538
- Wildreceipt tests by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1546
- Fix judge exist dir by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1542

- Fix IC13 textdet config by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1563
- Fix IC13 textrecog annotations by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1568
- Auto scale lr by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1584
- Fix icdar data parse for text containing separator by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1587
- Fix textspotting ut by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1599
- Fix TextSpottingConfigGenerator and TextSpottingDataConverter by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1604
- Keep E2E Inferencer output simple by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1559

- @jyshee made their first contribution in https://github.com/open-mmlab/mmocr/pull/1517

- @ProtossDragoon made their first contribution in https://github.com/open-mmlab/mmocr/pull/1540
- @IncludeMathH made their first contribution in https://github.com/open-mmlab/mmocr/pull/1547

**Full Changelog**: https://github.com/open-mmlab/mmocr/compare/v1.0.0rc3...v1.0.0rc4

## v1.0.0rc3 (11/03/2022)

1. We release several pretrained models using [oCLIP-ResNet](https://github.com/open-mmlab/mmocr/blob/1.x/configs/backbone/oclip/README.md) as the backbone, which is a ResNet variant trained with [oCLIP](https://www.ecva.net/papers/eccv_2022/papers_ECCV/papers/136880282.pdf) and can significantly boost the performance of text detection models.

2. Preparing datasets is troublesome and tedious, especially in OCR domain where multiple datasets are usually required. In order to free our users from laborious work, we designed a [Dataset Preparer](https://mmocr.readthedocs.io/en/dev-1.x/user_guides/data_prepare/dataset_preparer.html) to help you get a bunch of datasets ready for use, with only **one line of command**!  Dataset Preparer is also crafted to consist of a series of reusable modules, each responsible for handling one of the standardized phases throughout the preparation process, shortening the development cycle on supporting new datasets.

- Add Dataset Preparer by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1484

* support modified resnet structure used in oCLIP by @HannibalAPE in https://github.com/open-mmlab/mmocr/pull/1458
* Add oCLIP configs by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1509

### Docs

### Docs

- Update install.md by @rogachevai in https://github.com/open-mmlab/mmocr/pull/1494
- Refine some docs by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1455
- Update some dataset preparer related docs by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1502
- oclip readme by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1505

- Fix offline_eval error caused by new data flow by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1500

- @rogachevai made their first contribution in https://github.com/open-mmlab/mmocr/pull/1494
- @HannibalAPE made their first contribution in https://github.com/open-mmlab/mmocr/pull/1458

**Full Changelog**: https://github.com/open-mmlab/mmocr/compare/v1.0.0rc2...v1.0.0rc3

## v1.0.0rc2 (10/14/2022)

This release relaxes the version requirement of `MMEngine` to `>=0.1.0, < 1.0.0`.

## v1.0.0rc1 (10/09/2022)

This release fixes a severe bug leading to inaccurate metric report in multi-GPU training.
We release the weights for all the text recognition models in MMOCR 1.0 architecture. The inference shorthand for them are also added back to `ocr.py`. Besides, more documentation chapters are available now.

- Simplify the Mask R-CNN config by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1391
- auto scale lr by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1326
- Update paths to pretrain weights by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1416
- Streamline duplicated split_result in pan_postprocessor by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1418

- Update model links in ocr.py and inference.md by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1431
- Update rec configs by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1417
- Visualizer refine by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1411
- Support get flops and parameters in dev-1.x by @vansin in https://github.com/open-mmlab/mmocr/pull/1414

### Docs

- intersphinx and api by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1367
- Fix quickrun by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1374
- Fix some docs issues by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1385
- Add Documents for DataElements by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1381
- config english by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1372

- Metrics by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1399
- Add version switcher to menu by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1407
- Data Transforms by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1392
- Fix inference docs by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1415
- Fix some docs by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1410
- Add maintenance plan to migration guide by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1413
- Update Recog Models by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1402

- clear metric.results only done in main process by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1379

- Fix a bug in MMDetWrapper by @xinke-wang in https://github.com/open-mmlab/mmocr/pull/1393
- Fix browse_dataset.py by @Mountchicken in https://github.com/open-mmlab/mmocr/pull/1398
- ImgAugWrapper: Do not cilp polygons if not applicable by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1231
- Fix CI by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1365
- Fix merge stage test by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1370
- Del CI support for torch 1.5.1 by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1371
- Test windows cu111 by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1373
- Fix windows CI by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1387
- Upgrade pre commit hooks by @Harold-lkk in https://github.com/open-mmlab/mmocr/pull/1429

- Skip invalid augmented polygons in ImgAugWrapper by @gaotongxiao in https://github.com/open-mmlab/mmocr/pull/1434

- @vansin made their first contribution in https://github.com/open-mmlab/mmocr/pull/1414

**Full Changelog**: https://github.com/open-mmlab/mmocr/compare/v1.0.0rc0...v1.0.0rc1

## v1.0.0rc0 (09/01/2022)

We are excited to announce the release of MMOCR 1.0.0rc0.
MMOCR 1.0.0rc0 is the first version of MMOCR 1.x, a part of the OpenMMLab 2.0 projects.
Built upon the new [training engine](https://github.com/open-mmlab/mmengine),
MMOCR 1.x unifies the interfaces of dataset, models, evaluation, and visualization with faster training and testing speed.

5. **More documentation and tutorials**. We add a bunch of documentation and tutorials to help users get started more smoothly. Read it [here](https://mmocr.readthedocs.io/en/dev-1.x/).

- MMOCR 1.x relies on MMEngine to run. MMEngine is a new foundational library for training deep learning models in OpenMMLab 2.0 models. The dependencies of file IO and training are migrated from MMCV 1.x to MMEngine.
- MMOCR 1.x relies on MMCV>=2.0.0rc0. Although MMCV no longer maintains the training functionalities since 2.0.0rc0, MMOCR 1.x relies on the data transforms, CUDA operators, and image processing interfaces in MMCV. Note that the package `mmcv` is the version that provide pre-built CUDA operators and `mmcv-lite` does not since MMCV 2.0.0rc0, while `mmcv-full` has been deprecated.

- MMOCR 1.x uses Runner in [MMEngine](https://github.com/open-mmlab/mmengine) rather than that in MMCV. The new Runner implements and unifies the building logic of dataset, model, evaluation, and visualizer. Therefore, MMOCR 1.x no longer maintains the building logics of those modules in `mmocr.train.apis` and `tools/train.py`. Those code have been migrated into [MMEngine](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/runner.py). Please refer to the [migration guide of Runner in MMEngine](https://mmengine.readthedocs.io/en/latest/migration/runner.html) for more details.

- The [Runner in MMEngine](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/runner.py) uses a different config structures to ease the understanding of the components in runner. Users can read the [config example of MMOCR](../user_guides/config.md) or refer to the [migration guide in MMEngine](https://mmengine.readthedocs.io/en/latest/migration/runner.html) for migration details.
- The file names of configs and models are also refactored to follow the new rules unified across OpenMMLab 2.0 projects. Please refer to the [user guides of config](../user_guides/config.md) for more details.

The Dataset classes implemented in MMOCR 1.x all inherits from the `BaseDetDataset`, which inherits from the [BaseDataset in MMEngine](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/basedataset.html). There are several changes of Dataset in MMOCR 1.x.

- The interfaces are changed accordingly.

The data transforms in MMOCR 1.x all inherits from those in MMCV>=2.0.0rc0, which follows a new convention in OpenMMLab 2.0 projects.

- The interfaces are also changed. Please refer to the [API Reference](https://mmocr.readthedocs.io/en/dev-1.x/)
- The functionality of some data transforms (e.g., `Resize`) are decomposed into several transforms.
- The same data transforms in different OpenMMLab 2.0 libraries have the same augmentation implementation and the logic of the same arguments, i.e., `Resize` in MMDet 3.x and MMOCR 1.x will resize the image in the exact same manner given the same arguments.

#### Model

#### Model

The models in MMOCR 1.x all inherits from `BaseModel` in MMEngine, which defines a new convention of models in OpenMMLab 2.0 projects. Users can refer to the [tutorial of model](https://mmengine.readthedocs.io/en/latest/tutorials/model.html) in MMengine for more details. Accordingly, there are several changes as the following:

- The model interfaces, including the input and output formats, are significantly simplified and unified following the new convention in MMOCR 1.x. Specifically, all the input data in training and testing are packed into `inputs` and `data_samples`, where `inputs` contains model inputs like a list of image tensors, and `data_samples` contains other information of the current data sample such as ground truths and model predictions. In this way, different tasks in MMOCR 1.x can share the same input arguments, which makes the models more general and suitable for multi-task learning.

- The model has a data preprocessor module, which is used to pre-process the input data of model. In MMOCR 1.x, the data preprocessor usually does necessary steps to form the input images into a batch, such as padding. It can also serve as a place for some special data augmentations or more efficient data transformations like normalization.
- The internal logic of model have been changed. In MMOCR 0.x, model used `forward_train` and `simple_test` to deal with different model forward logics. In MMOCR 1.x and OpenMMLab 2.0, the forward function has three modes: `loss`, `predict`, and `tensor` for training, inference, and tracing or other purposes, respectively. The forward function calls `self.loss()`, `self.predict()`, and `self._forward()` given the modes `loss`, `predict`, and `tensor`, respectively.

MMOCR 1.x mainly implements corresponding metrics for each task, which are manipulated by [Evaluator](https://mmengine.readthedocs.io/en/latest/design/evaluator.html) to complete the evaluation.
In addition, users can build evaluator in MMOCR 1.x to conduct offline evaluation, i.e., evaluate predictions that may not produced by MMOCR, prediction follows our dataset conventions. More details can be find in the [Evaluation Tutorial](https://mmengine.readthedocs.io/en/latest/tutorials/evaluation.html) in MMEngine.

The functions of visualization in MMOCR 1.x are removed. Instead, in OpenMMLab 2.0 projects, we use [Visualizer](https://mmengine.readthedocs.io/en/latest/design/visualization.html) to visualize data. MMOCR 1.x implements `TextDetLocalVisualizer`, `TextRecogLocalVisualizer`, and `KIELocalVisualizer` to allow visualization of ground truths, model predictions, and feature maps, etc., at any place, for the three tasks supported in MMOCR. It also supports to dump the visualization data to any external visualization backends such as Tensorboard and Wandb. Check our [Visualization Document](https://mmocr.readthedocs.io/en/dev-1.x/user_guides/visualization.html) for more details.

- Most models enjoy a performance improvement from the new framework and refactor of data transforms. For example, in MMOCR 1.x, DBNet-R50 achieves **0.854** hmean score on ICDAR 2015, while the counterpart can only get **0.840** hmean score in MMOCR 0.x.

- Support mixed precision training of most of the models. However, the [rest models](https://mmocr.readthedocs.io/en/dev-1.x/user_guides/train_test.html#mixed-precision-training) are not supported yet because the operators they used might not be representable in fp16. We will update the documentation and list the results of mixed precision training.

1. Test-time augmentation: which was supported in MMOCR 0.x, is not implemented yet in this version due to limited time slot. We will support it in the following releases with a new and simplified design.

4. Documentation: we will add more design docs, tutorials, and migration guidance so that the community can deep dive into our new design, participate the future development, and smoothly migrate downstream libraries to MMOCR 1.x.

# Coordinate System

MMDetection3D uses three different coordinate systems. The existence of different coordinate systems in the society of 3D object detection is necessary, because for various 3D data collection devices, such as LiDAR, depth camera, etc., the coordinate systems are not consistent, and different 3D datasets also follow different data formats. Early works, such as SECOND, VoteNet, convert the raw data to another format, forming conventions that some later works also follow, making the conversion between coordinate systems even more complicated.

Despite the variety of datasets and equipment, by summarizing the line of works on 3D object detection we can roughly categorize coordinate systems into three:

- Camera coordinate system -- the coordinate system of most cameras, in which the positive direction of the y-axis points to the ground, the positive direction of the x-axis points to the right, and the positive direction of the z-axis points to the front.

```
             up  z front
              |    ^
              |   /
              |  /
              | /
              |/
  left ------ 0 ------> x right
              |
              |
              |
              |
              v
            y down
  ```

- LiDAR coordinate system -- the coordinate system of many LiDARs, in which the negative direction of the z-axis points to the ground, the positive direction of the x-axis points to the front, and the positive direction of the y-axis points to the left.

  ```
               z up  x front
                 ^    ^
                 |   /
                 |  /
                 | /
                 |/
  y left <------ 0 ------ right
  ```

- Depth coordinate system -- the coordinate system used by VoteNet, H3DNet, etc., in which the negative direction of the z-axis points to the ground, the positive direction of the x-axis points to the right, and the positive direction of the y-axis points to the front.

  ```

```
            z up  y front
              ^    ^
              |   /
              |  /
              | /
              |/
  ```

The definition of coordinate systems in this tutorial is actually **more than just defining the three axes**. For a box in the form of $(x, y, z, dx, dy, dz, r)$, our coordinate systems also define how to interpret the box dimensions $(dx, dy, dz)$ and the yaw angle $r$.

The illustration of the three coordinate systems is shown below:

![](https://raw.githubusercontent.com/open-mmlab/mmdetection3d/master/resources/coord_sys_all.png)

The three figures above are the 3D coordinate systems while the three figures below are the bird's eye view.

We will stick to the three coordinate systems defined in this tutorial in the future.

## Definition of the yaw angle

Please refer to [wikipedia](https://en.wikipedia.org/wiki/Euler_angles#Tait%E2%80%93Bryan_angles) for the standard definition of the yaw angle. In object detection, we choose an axis as the gravity axis, and a reference direction on the plane $\\Pi$ perpendicular to the gravity axis, then the reference direction has a yaw angle of 0, and other directions on $\\Pi$ have non-zero yaw angles depending on its angle with the reference direction.

Currently, for all supported datasets, annotations do not include pitch angle and roll angle, which means we need only consider the yaw angle when predicting boxes and calculating overlap between boxes.

In MMDetection3D, all three coordinate systems are right-handed coordinate systems, which means the ascending direction of the yaw angle is counter-clockwise if viewed from the negative direction of the gravity axis (the axis is pointing at one's eyes).

The figure below shows that, in this right-handed coordinate system, if we set the positive direction of the x-axis as a reference direction, then the positive direction of the y-axis has a yaw angle of $\\frac{\\pi}{2}$.

```
                     z up  y front (yaw=0.5*pi)
                      ^    ^
                      |   /
                      |  /
                      | /
                      |/
left (yaw=pi)  ------ 0 ------> x right (yaw=0)
```

^    ^
                      |   /
                      |  /
                      | /
                      |/
```

For a box, the value of its yaw angle equals its direction minus a reference direction. In all three coordinate systems in MMDetection3D, the reference direction is always the positive direction of the x-axis, while the direction of a box is defined to be parallel with the x-axis if its yaw angle is 0. The definition of the yaw angle of a box is illustrated in the figure below.

```
y front
  ^      box direction (yaw=0.5*pi)
 /|\        ^
  |        /|\
  |     ____|____
  |    |    |    |
  |    |    |    |
__|____|____|____|______\ x right
  |    |    |    |      /
  |    |    |    |
  |    |____|____|
  |
```

## Definition of the box dimensions

|     ____|____
  |    |    |    |
  |    |    |    |
  |    |    |    |      /
  |    |    |    |
  |
```

The definition of the box dimensions cannot be disentangled with the definition of the yaw angle. In the previous section, we said that the direction of a box is defined to be parallel with the x-axis if its yaw angle is 0. Then naturally, the dimension of a box which corresponds to the x-axis should be $dx$. However, this is not always the case in some datasets (we will address that later).

The following figures show the meaning of the correspondence between the x-axis and $dx$, and between the y-axis and $dy$.

```
y front
 /|\        ^
  |        /|\
  |    |    |    |
  |    |    |    | dx
  |    |    |    |      /
  |    |    |    |
  |         dy
```

|        /|\
  |    |    |    |
  |    |    |    | dx
  |    |    |    |      /
  |    |    |    |
  |         dy
```

Note that the box direction is always parallel with the edge $dx$.

```
y front
  ^     _________
 /|\   |    |    |
  |    |    |    |
  |    |    |    | dy
  |    |____|____|____\  box direction (yaw=0)
  |    |    |    |    /
__|____|____|____|_________\ x right
  |    |    |    |         /
  |         dx
  |
```

## Relation with raw coordinate systems of supported datasets

### KITTI

  |    |    |    |         /
  |         dx
  |
```

### KITTI

The raw annotation of KITTI is under camera coordinate system, see [get_label_anno](https://github.com/open-mmlab/mmdetection3d/blob/master/tools/dataset_converters/kitti_data_utils.py). In MMDetection3D, to train LiDAR-based models on KITTI, the data is first converted from camera coordinate system to LiDAR coordinate system, see [get_ann_info](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/datasets/kitti_dataset.py). For training vision-based models, the data is kept in the camera coordinate system.

In SECOND, the LiDAR coordinate system for a box is defined as follows (a bird's eye view):

![](https://raw.githubusercontent.com/traveller59/second.pytorch/master/images/kittibox.png)

For each box, the dimensions are $(w, l, h)$, and the reference direction for the yaw angle is the positive direction of the y axis. For more details, refer to the [repo](https://github.com/traveller59/second.pytorch#concepts).

Our LiDAR coordinate system has two changes:

- The yaw angle is defined to be right-handed instead of left-handed for consistency;
- The box dimensions are $(l, w, h)$ instead of $(w, l, h)$, since $w$ corresponds to $dy$ and $l$ corresponds to $dx$ in KITTI.

### Waymo

We use the KITTI-format data of Waymo dataset. Therefore, KITTI and Waymo also share the same coordinate system in our implementation.

### Waymo

NuScenes provides a toolkit for evaluation, in which each box is wrapped into a `Box` instance. The coordinate system of `Box` is different from our LiDAR coordinate system in that the first two elements of the box dimension correspond to $(dy, dx)$, or $(w, l)$, respectively, instead of the reverse. For more details, please refer to the NuScenes [tutorial](https://github.com/open-mmlab/mmdetection3d/blob/master/docs/en/datasets/nuscenes_det.md#notes).

Readers may refer to the [NuScenes development kit](https://github.com/nutonomy/nuscenes-devkit/tree/master/python-sdk/nuscenes/eval/detection) for the definition of a [NuScenes box](https://github.com/nutonomy/nuscenes-devkit/blob/2c6a752319f23910d5f55cc995abc547a9e54142/python-sdk/nuscenes/utils/data_classes.py#L457) and implementation of [NuScenes evaluation](https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/nuscenes/eval/detection/evaluate.py).

### Lyft

Lyft shares the same data format with NuScenes as far as coordinate system is involved.

Please refer to the [official website](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/data) for more information.

The raw data of ScanNet is not point cloud but mesh. The sampled point cloud data is under our depth coordinate system. For ScanNet detection task, the box annotations are axis-aligned, and the yaw angle is always zero. Therefore the direction of the yaw angle in our depth coordinate system makes no difference regarding ScanNet.

The raw data of SUN RGB-D is not point cloud but RGB-D image. By back projection, we obtain the corresponding point cloud for each image, which is under our Depth coordinate system. However, the annotation is not under our system and thus needs conversion.

For the conversion from raw annotation to annotation under our Depth coordinate system, please refer to [sunrgbd_data_utils.py](https://github.com/open-mmlab/mmdetection3d/blob/master/tools/dataset_converters/sunrgbd_data_utils.py).

### S3DIS

### S3DIS

S3DIS shares the same coordinate system as ScanNet in our implementation. However, S3DIS is a segmentation-task-only dataset, and thus no annotation is coordinate system sensitive.

### Box conversion (between different coordinate systems)

Take the conversion between our Camera coordinate system and LiDAR coordinate system as an example:

First, for points and box centers, the coordinates before and after the conversion satisfy the following relationship:

- $x\_{LiDAR}=z\_{camera}$
- $y\_{LiDAR}=-x\_{camera}$
- $z\_{LiDAR}=-y\_{camera}$

Then, the box dimensions before and after the conversion satisfy the following relationship:

- $dx\_{LiDAR}=dx\_{camera}$
- $dy\_{LiDAR}=dz\_{camera}$
- $dz\_{LiDAR}=dy\_{camera}$

Finally, the yaw angle should also be converted:

- $r\_{LiDAR}=-\\frac{\\pi}{2}-r\_{camera}$

See the code [here](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/core/bbox/structures/box_3d_mode.py) for more details.

### Bird's Eye View

The BEV of a camera coordinate system box is $(x, z, dx, dz, -r)$ if the 3D box is $(x, y, z, dx, dy, dz, r)$. The inversion of the sign of the yaw angle is because the positive direction of the gravity axis of the Camera coordinate system points to the ground.

See the code [here](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/core/bbox/structures/cam_box3d.py) for more details.

### Rotation of boxes

We set the rotation of all kinds of boxes to be counter-clockwise about the gravity axis. Therefore, to rotate a 3D box we first calculate the new box center, and then we add the rotation angle to the yaw angle.

## Common FAQ

#### Q1: Are the box related ops universal to all coordinate system types?

No. For example, [RoI-Aware Pooling ops](https://github.com/open-mmlab/mmcv/blob/master/mmcv/ops/roiaware_pool3d.py) is applicable to boxes under Depth or LiDAR coordinate system only. The evaluation functions for KITTI dataset [here](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/core/evaluation/kitti_utils) are only applicable to boxes under Camera coordinate system since the rotation is clockwise if viewed from above.

For each box related op, we have marked the type of boxes to which we can apply the op.

#### Q2: In every coordinate system, do the three axes point exactly to the right, the front, and the ground, respectively?

No. For example, in KITTI, we need a calibration matrix when converting from Camera coordinate system to LiDAR coordinate system.

#### Q3: How does a phase difference of $2\\pi$ in the yaw angle of a box affect evaluation?

For IoU calculation, a phase difference of $2\\pi$ in the yaw angle will result in the same box, thus not affecting evaluation.

For angle prediction evaluation such as the NDS metric in NuScenes and the AOS metric in KITTI, the angle of predicted boxes will be first standardized, so the phase difference of $2\\pi$ will not change the result.

#### Q4: How does a phase difference of $\\pi$ in the yaw angle of a box affect evaluation?

For IoU calculation, a phase difference of $\\pi$ in the yaw angle will result in the same box, thus not affecting evaluation.

However, for angle prediction evaluation, this will result in the exact opposite direction.

Just think about a car. The yaw angle is the angle between the direction of the car front and the positive direction of the x-axis. If we add $\\pi$ to this angle, the car front will become the car rear.

For categories such as barrier, the front and the rear have no difference, therefore a phase difference of $\\pi$ will not affect the angle prediction score.

# Detecting Twenty-thousand Classes using Image-level Supervision

**Detic**: A **Det**ector with **i**mage **c**lasses that can use image-level labels to easily train detectors.

<p align="center"> <img src='https://github.com/facebookresearch/Detic/blob/main/docs/teaser.jpeg?raw=true' align="center" height="300px"> </p>

> [**Detecting Twenty-thousand Classes using Image-level Supervision**](http://arxiv.org/abs/2201.02605),
> Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, Ishan Misra,
> *ECCV 2022 ([arXiv 2201.02605](http://arxiv.org/abs/2201.02605))*

## Usage

Detic requires to install CLIP.

```shell
pip install git+https://github.com/openai/CLIP.git
```

### Demo

#### Inference with existing dataset vocabulary embeddings

```shell
```

### Demo

First, go to the Detic project folder.

```shell
cd projects/Detic
```

Then, download the pre-computed CLIP embeddings from [dataset metainfo](https://github.com/facebookresearch/Detic/tree/main/datasets/metadata) to the `datasets/metadata` folder.
The CLIP embeddings will be loaded to the zero-shot classifier during inference.
For example, you can download LVIS's class name embeddings with the following command:

```shell
wget -P datasets/metadata https://raw.githubusercontent.com/facebookresearch/Detic/main/datasets/metadata/lvis_v1_clip_a%2Bcname.npy
```

You can run demo like this:

```shell
python demo.py \
  ${IMAGE_PATH} \
  ${CONFIG_PATH} \
  ${MODEL_PATH} \
  --show \
  --score-thr 0.5 \
  --dataset lvis
```

```

```shell
  --show \
```

![image](https://user-images.githubusercontent.com/12907710/213624759-f0a2ba0c-0f5c-4424-a350-5ba5349e5842.png)

### Inference with custom vocabularies

- Detic can detects any class given class names by using CLIP.

You can detect custom classes with `--class-name` command:

```
  --show \
  --score-thr 0.3 \
  --class-name headphone webcam paper coffe
```

![image](https://user-images.githubusercontent.com/12907710/213624637-e9e8a313-9821-4782-a18a-4408c876852b.png)

Note that `headphone`, `paper` and `coffe` (typo intended) are not LVIS classes. Despite the misspelled class name, Detic can produce a reasonable detection for `coffe`.

## Results

## Results

Here we only provide the Detic Swin-B model for the open vocabulary demo. Multi-dataset training and open-vocabulary testing will be supported in the future.

To find more variants, please visit the [official model zoo](https://github.com/facebookresearch/Detic/blob/main/docs/MODEL_ZOO.md).

| Backbone |       Training data        |                                Config                                 |                                                                                      Download                                                                                      |

| :------: | :------------------------: | :-------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|  Swin-B  | ImageNet-21K & LVIS & COCO | [config](./configs/detic_centernet2_swin-b_fpn_4x_lvis-coco-in21k.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/detic/detic_centernet2_swin-b_fpn_4x_lvis-coco-in21k/detic_centernet2_swin-b_fpn_4x_lvis-coco-in21k_20230120-0d301978.pth) |

If you find Detic is useful in your research or applications, please consider giving a star 🌟 to the [official repository](https://github.com/facebookresearch/Detic) and citing Detic by the following BibTeX entry.

```BibTeX
@inproceedings{zhou2022detecting,
  title={Detecting Twenty-thousand Classes using Image-level Supervision},

```BibTeX
  author={Zhou, Xingyi and Girdhar, Rohit and Joulin, Armand and Kr{\"a}henb{\"u}hl, Philipp and Misra, Ishan},
}

```

# 后端支持

我们支持不同的文件客户端后端：磁盘、Ceph 和 LMDB 等。下面是修改配置使之从 Ceph 加载和保存数据的示例。

## 从 Ceph 读取数据和标注文件

我们支持从 Ceph 加载数据和生成的标注信息文件（pkl 和 json）：

```python
    }))

    rate=1.0,

points_loader=dict(

    dict(
    dict(

dict(type='RandomFlip3D', flip_ratio_bev_horizontal=0.5),
    dict(
]
    dict(
            dict(

flip=False,
            dict(
            dict(
            dict(
        ])
]

        times=2,

# set file client for loading validation info files (.pkl)
```

## 从 Ceph 读取预训练模型

```python
        ...
```

## 从 Ceph 读取模型权重文件

```python
load_from = 's3://openmmlab/checkpoints/mmdetection3d/v0.1.0_models/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-car/hv_pointpillars_secfpn_6x8_160e_kitti-3d-car_20200620_230614-77663cd6.pth'
```

## 保存模型权重文件至 Ceph

```

```python
```

## EvalHook 保存最优模型权重文件至 Ceph

```python
```

## 训练日志保存至 Ceph

训练后的训练日志会备份到指定的 Ceph 路径。

```python
    hooks=[
    ])
```

您还可以通过设置 `keep_local = False` 备份到指定的 Ceph 路径后删除本地训练日志。

```python
    hooks=[
    ])
```

<details>

```bibtex
}
```

<details>

```bibtex

| [pose_hourglass_52](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hourglass52_8xb32-210e_coco-256x256.py) |  256x256   | 0.726 |      0.896      |      0.799      | 0.780 |      0.934      | [ckpt](https://download.openmmlab.com/mmpose/top_down/hourglass/hourglass52_coco_256x256-4ec713ba_20200709.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hourglass/hourglass52_coco_256x256_20200709.log.json) |
| [pose_hourglass_52](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hourglass52_8xb32-210e_coco-384x384.py) |  384x384   | 0.746 |      0.900      |      0.812      | 0.797 |      0.939      | [ckpt](https://download.openmmlab.com/mmpose/top_down/hourglass/hourglass52_coco_384x384-be91ba2b_20200812.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hourglass/hourglass52_coco_384x384_20200812.log.json) |

# 快速上手

在这一章里，我们将带领你走过MMPose工作流程中关键的七个步骤，帮助你快速上手：

1. 使用预训练模型进行推理
2. 准备数据集
3. 准备配置文件
4. 可视化训练图片
5. 训练
6. 测试
7. 可视化

## 安装

请查看[安装指南](./installation.md)，以了解完整步骤。

## 快速开始

### 使用预训练模型进行推理

你可以通过以下命令来使用预训练模型对单张图片进行识别：

```Bash
    configs/body_2d_keypoint/topdown_regression/coco/td-reg_res50_rle-8xb64-210e_coco-256x192.py\
    https://download.openmmlab.com/mmpose/top_down/deeppose/deeppose_res50_coco_256x192_rle-2ea9bb4a_20220616.pth
```

该命令中用到了测试图片、完整的配置文件、预训练模型，如果MMPose安装无误，将会弹出一个新窗口，对检测结果进行可视化显示：

![inference_demo](https://user-images.githubusercontent.com/13503330/187112344-0c5062f2-689c-445c-a259-d5d4311e2497.png)

更多演示脚本的详细参数说明可以在 [模型推理](./user_guides/inference.md) 中找到。

### 准备数据集

MMPose支持各种不同的任务，我们提供了对应的数据集准备教程。

- [2D人体关键点](./dataset_zoo/2d_body_keypoint.md)

- [3D人体关键点](./dataset_zoo/3d_body_keypoint.md)

- [2D人手关键点](./dataset_zoo/2d_hand_keypoint.md)

- [3D人手关键点](./dataset_zoo/3d_hand_keypoint.md)

- [2D人脸关键点](./dataset_zoo/2d_face_keypoint.md)

- [2D全身人体关键点](./dataset_zoo/2d_wholebody_keypoint.md)

- [2D服饰关键点](./dataset_zoo/2d_fashion_landmark.md)

- [2D动物关键点](./dataset_zoo/2d_animal_keypoint.md)

你可以在【2D人体关键点数据集】>【COCO】下找到COCO数据集的准备教程，并按照教程完成数据集的下载和整理。

```{note}
在MMPose中，我们建议将COCO数据集存放到新建的 `$MMPOSE/data` 目录下。
```

### 准备配置文件

MMPose拥有一套强大的配置系统，用于管理训练所需的一系列必要参数：

- **通用**：环境、Hook、Checkpoint、Logger、Timer等

- **数据**：Dataset、Dataloader、数据增强等

- **模型**：Backbone、Neck、Head、损失函数等

- **评测**：Metrics

在`$MMPOSE/configs`目录下，我们提供了大量前沿论文方法的配置文件，可供直接使用和参考。

要在COCO数据集上训练基于ResNet50的RLE模型时，所需的配置文件为：

```Bash
$MMPOSE/configs/body_2d_keypoint/topdown_regression/coco/td-reg_res50_rle-8xb64-210e_coco-256x192.py
```

我们需要将配置文件中的 data_root 变量修改为COCO数据集存放路径：

```Python

```Bash
```

```Python
data_root = 'data/coco'
```

```{note}
感兴趣的读者也可以查阅 [配置文件](./user_guides/configs.md) 来进一步学习MMPose所使用的配置系统。
```

### 可视化训练图片

在开始训练之前，我们还可以对训练图片进行可视化，检查训练图片是否正确进行了数据增强。

我们提供了相应的可视化脚本：

```Bash
python tools/misc/browse_dastaset.py \
    configs/body_2d_keypoint/topdown_regression/coco/td-reg_res50_rle-8xb64-210e_coco-256x192.py \
    --mode transformed
```

![transformed_training_img](https://user-images.githubusercontent.com/13503330/187112376-e604edcb-46cc-4995-807b-e8f204f991b0.png)

### 训练

确定数据无误后，运行以下命令启动训练：

```Bash
python tools/train.py configs/body_2d_keypoint/topdown_regression/coco/td-reg_res50_rle-8xb64-210e_coco-256x192.py
```

```{note}
MMPose中集成了大量实用训练trick和功能：

- 学习率warmup和scheduling

- ImageNet预训练权重

- 自动学习率缩放、自动batch size缩放

- CPU训练、多机多卡训练、集群训练

- HardDisk、LMDB、Petrel、HTTP等不同数据后端

- 混合精度浮点训练

- TensorBoard

```

```{note}

- 混合精度浮点训练

```

### 测试

在不指定额外参数时，训练的权重和日志信息会默认存储到`$MMPOSE/work_dirs`目录下，最优的模型权重存放在`$MMPOSE/work_dir/best_coco`目录下。

我们可以通过如下指令测试模型在COCO验证集上的精度：

```Bash
    work_dir/best_coco/AP_epoch_20.pth
```

在COCO验证集上评测结果样例如下：

```Bash
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] =  0.704
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets= 20 ] =  0.883
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets= 20 ] =  0.777
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] =  0.667
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] =  0.769

Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] =  0.667
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] =  0.751
 Average Recall     (AR) @[ IoU=0.50      | area=   all | maxDets= 20 ] =  0.920
 Average Recall     (AR) @[ IoU=0.75      | area=   all | maxDets= 20 ] =  0.815
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] =  0.709
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] =  0.811
08/23 12:04:42 - mmengine - INFO - Epoch(test) [3254/3254]  coco/AP: 0.704168  coco/AP .5: 0.883134  coco/AP .75: 0.777015  coco/AP (M): 0.667207  coco/AP (L): 0.768644  coco/AR: 0.750913  coco/AR .5: 0.919710  coco/AR .75: 0.815334  coco/AR (M): 0.709232  coco/AR (L): 0.811334
```

```{note}
如果需要测试模型在其他数据集上的表现，可以前往 [训练与测试](./user_guides/train_and_test.md) 查看。
```

### 可视化

```

```{note}
```

### 可视化

除了对关键点骨架的可视化以外，我们还支持对热度图进行可视化，你只需要在配置文件中设置`output_heatmap=True`：

```Python
    ## 内容省略
    test_cfg = dict(
        ## 内容省略
        output_heatmaps=True
    )
)
```

或在命令行中添加`--cfg-options='model.test_cfg.output_heatmaps=True'`。

可视化效果如下：

```{note}
如果你希望深入地学习MMPose，将其应用到自己的项目当中，我们准备了一份详细的 [迁移指南](./migration.md) 。
```

# Preparing Moments in Time

```BibTeX
}
```

For basic dataset information, you can refer to the dataset [website](http://moments.csail.mit.edu/).
Before we start, please make sure that the directory is located at `$MMACTION2/tools/data/mit/`.

```shell
```

```

```shell
```

```shell
```

```shell

```

```shell
```

```shell
```

```shell
```

After the whole data process for Moments in Time preparation,
you will get the rawframes (RGB + Flow), videos and annotation files for Moments in Time.

In the context of the whole project (for Moments in Time only), the folder structure will look like:

```
mmaction2
├── data
│   └── mit

```
mmaction2
├── data
│   └── mit

│       │   │       ├── ...
│           │   │   ├── ...
│           │       ├── ...
│           │   ├── ...
└── ...

```

For training and evaluating on Moments in Time, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).

# ResNeSt

It is well known that featuremap attention and multi-path representation are important for visual recognition. In this paper, we present a modularized architecture, which applies the channel-wise attention on different network branches to leverage their success in capturing cross-feature interactions and learning diverse representations. Our design results in a simple and unified computation block, which can be parameterized using only a few variables. Our model, named ResNeSt, outperforms EfficientNet in accuracy and latency trade-off on image classification. In addition, ResNeSt has achieved superior transfer learning results on several public benchmarks serving as the backbone, and has been adopted by the winning entries of COCO-LVIS challenge.

<img src="https://user-images.githubusercontent.com/40661020/143973475-b5b33b15-ed04-4fc6-890a-521f1a62bc52.png"/>
</div>

| Backbone  |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                                  Config                                   |                                                                                                                                                                                                                             Download                                                                                                                                                                                                                             |

| S-50-FPN  | pytorch |   1x    |   4.8    |       -        |  42.0  | [config](./faster-rcnn_s50_fpn_syncbn-backbone+head_ms-range-1x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/resnest/faster_rcnn_s50_fpn_syncbn-backbone%2Bhead_mstrain-range_1x_coco/faster_rcnn_s50_fpn_syncbn-backbone%2Bhead_mstrain-range_1x_coco_20200926_125502-20289c16.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/resnest/faster_rcnn_s50_fpn_syncbn-backbone%2Bhead_mstrain-range_1x_coco/faster_rcnn_s50_fpn_syncbn-backbone%2Bhead_mstrain-range_1x_coco-20200926_125502.log.json)   |

| S-101-FPN | pytorch |   1x    |   7.1    |       -        |  44.5  | [config](./faster-rcnn_s101_fpn_syncbn-backbone+head_ms-range-1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/resnest/faster_rcnn_s101_fpn_syncbn-backbone%2Bhead_mstrain-range_1x_coco/faster_rcnn_s101_fpn_syncbn-backbone%2Bhead_mstrain-range_1x_coco_20201006_021058-421517f1.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/resnest/faster_rcnn_s101_fpn_syncbn-backbone%2Bhead_mstrain-range_1x_coco/faster_rcnn_s101_fpn_syncbn-backbone%2Bhead_mstrain-range_1x_coco-20201006_021058.log.json) |

| Backbone  |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                              Config                               |                                                                                                                                                                                                             Download                                                                                                                                                                                                             |

| S-50-FPN  | pytorch |   1x    |   5.5    |       -        |  42.6  |  38.1   | [config](./mask-rcnn_s50_fpn_syncbn-backbone+head_ms-1x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/resnest/mask_rcnn_s50_fpn_syncbn-backbone%2Bhead_mstrain_1x_coco/mask_rcnn_s50_fpn_syncbn-backbone%2Bhead_mstrain_1x_coco_20200926_125503-8a2c3d47.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/resnest/mask_rcnn_s50_fpn_syncbn-backbone%2Bhead_mstrain_1x_coco/mask_rcnn_s50_fpn_syncbn-backbone%2Bhead_mstrain_1x_coco-20200926_125503.log.json)   |

| S-101-FPN | pytorch |   1x    |   7.8    |       -        |  45.2  |  40.2   | [config](./mask-rcnn_s101_fpn_syncbn-backbone+head_ms-1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/resnest/mask_rcnn_s101_fpn_syncbn-backbone%2Bhead_mstrain_1x_coco/mask_rcnn_s101_fpn_syncbn-backbone%2Bhead_mstrain_1x_coco_20201005_215831-af60cdf9.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/resnest/mask_rcnn_s101_fpn_syncbn-backbone%2Bhead_mstrain_1x_coco/mask_rcnn_s101_fpn_syncbn-backbone%2Bhead_mstrain_1x_coco-20201005_215831.log.json) |

| Backbone  |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP |                                   Config                                   |                                                                                                                                                                                                                              Download                                                                                                                                                                                                                              |

| S-50-FPN  | pytorch |   1x    |    -     |       -        |  44.5  | [config](./cascade-rcnn_s50_fpn_syncbn-backbone+head_ms-range-1x_coco.py)  | [model](https://download.openmmlab.com/mmdetection/v2.0/resnest/cascade_rcnn_s50_fpn_syncbn-backbone%2Bhead_mstrain-range_1x_coco/cascade_rcnn_s50_fpn_syncbn-backbone%2Bhead_mstrain-range_1x_coco_20201122_213640-763cc7b5.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/resnest/cascade_rcnn_s101_fpn_syncbn-backbone%2Bhead_mstrain-range_1x_coco/cascade_rcnn_s101_fpn_syncbn-backbone%2Bhead_mstrain-range_1x_coco-20201005_113242.log.json) |

| S-101-FPN | pytorch |   1x    |   8.4    |       -        |  46.8  | [config](./cascade-rcnn_s101_fpn_syncbn-backbone+head_ms-range-1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/resnest/cascade_rcnn_s101_fpn_syncbn-backbone%2Bhead_mstrain-range_1x_coco/cascade_rcnn_s101_fpn_syncbn-backbone%2Bhead_mstrain-range_1x_coco_20201005_113242-b9459f8f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/resnest/cascade_rcnn_s50_fpn_syncbn-backbone%2Bhead_mstrain-range_1x_coco/cascade_rcnn_s50_fpn_syncbn-backbone%2Bhead_mstrain-range_1x_coco-20201122_213640.log.json) |

| Backbone  |  Style  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                                  Config                                   |                                                                                                                                                                                                                             Download                                                                                                                                                                                                                             |

| S-50-FPN  | pytorch |   1x    |    -     |       -        |  45.4  |  39.5   | [config](./cascade-mask-rcnn_s50_fpn_syncbn-backbone+head_ms-1x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/resnest/cascade_mask_rcnn_s50_fpn_syncbn-backbone%2Bhead_mstrain_1x_coco/cascade_mask_rcnn_s50_fpn_syncbn-backbone%2Bhead_mstrain_1x_coco_20201122_104428-99eca4c7.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/resnest/cascade_mask_rcnn_s50_fpn_syncbn-backbone%2Bhead_mstrain_1x_coco/cascade_mask_rcnn_s50_fpn_syncbn-backbone%2Bhead_mstrain_1x_coco-20201122_104428.log.json)   |

| S-101-FPN | pytorch |   1x    |   10.5   |       -        |  47.7  |  41.4   | [config](./cascade-mask-rcnn_s101_fpn_syncbn-backbone+head_ms-1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/resnest/cascade_mask_rcnn_s101_fpn_syncbn-backbone%2Bhead_mstrain_1x_coco/cascade_mask_rcnn_s101_fpn_syncbn-backbone%2Bhead_mstrain_1x_coco_20201005_113243-42607475.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/resnest/cascade_mask_rcnn_s101_fpn_syncbn-backbone%2Bhead_mstrain_1x_coco/cascade_mask_rcnn_s101_fpn_syncbn-backbone%2Bhead_mstrain_1x_coco-20201005_113243.log.json) |

```latex
}
```

# DeepFashion

> [DeepFashion: Powering Robust Clothes Recognition and Retrieval With Rich Annotations](https://openaccess.thecvf.com/content_cvpr_2016/html/Liu_DeepFashion_Powering_Robust_CVPR_2016_paper.html)

Recent advances in clothes recognition have been driven by the construction of clothes datasets. Existing datasets are limited in the amount of annotations and are difficult to cope with the various challenges in real-world applications. In this work, we introduce DeepFashion, a large-scale clothes dataset with comprehensive annotations. It contains over 800,000 images, which are richly annotated with massive attributes, clothing landmarks, and correspondence of images taken under different scenarios including store, street snapshot, and consumer. Such rich annotations enable the development of powerful algorithms in clothes recognition and facilitating future researches. To demonstrate the advantages of DeepFashion, we propose a new deep model, namely FashionNet, which learns clothing features by jointly predicting clothing attributes and landmarks. The estimated landmarks are then employed to pool or gate the learned features. It is optimized in an iterative manner. Extensive experiments demonstrate the effectiveness of FashionNet and the usefulness of DeepFashion.

<img src="https://user-images.githubusercontent.com/40661020/143876310-08470a6a-ea3a-4ec1-a6f2-8ec5df36a8a0.png"/>
</div>

[MMFashion](https://github.com/open-mmlab/mmfashion) develops "fashion parsing and segmentation" module
based on the dataset
[DeepFashion-Inshop](https://drive.google.com/drive/folders/0B7EVK8r0v71pVDZFQXRsMDZCX1E?usp=sharing).
Its annotation follows COCO style.
To use it, you need to first download the data. Note that we only use "img_highres" in this task.
The file tree should be like this:

```sh
├── mmdet
├── tools
├── data
│   ├── DeepFashion
│   │   ├── In-shop
|   │   │   ├── Anno
|   │   │   │   ├── segmentation
|   │   │   │   |   ├── DeepFashion_segmentation_train.json
|   │   │   │   |   ├── DeepFashion_segmentation_query.json
|   │   │   │   |   ├── DeepFashion_segmentation_gallery.json
|   │   │   │   ├── list_bbox_inshop.txt
|   │   │   │   ├── list_description_inshop.json

|   │   │   │   ├── list_item_inshop.txt
|   │   │   │   └── list_landmarks_inshop.txt
|   │   │   ├── Eval
|   │   │   │   └── list_eval_partition.txt
|   │   │   ├── Img
|   │   │   │   ├── img
|   │   │   │   │   ├──XXX.jpg
|   │   │   │   ├── img_highres
|   │   │   │   └── ├──XXX.jpg

```

After that you can train the Mask RCNN r50 on DeepFashion-In-shop dataset by launching training with the `mask_rcnn_r50_fpn_1x.py` config
or creating your own config file.

```

| Backbone | Model type |       Dataset       | bbox detection Average Precision | segmentation Average Precision |                      Config                      |                                                                                                                                       Download (Google)                                                                                                                                       |

| :------: | :--------: | :-----------------: | :------------------------------: | :----------------------------: | :----------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ResNet50 | Mask RCNN  | DeepFashion-In-shop |              0.599               |             0.584              | [config](./mask-rcnn_r50_fpn_15e_deepfashion.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/deepfashion/mask_rcnn_r50_fpn_15e_deepfashion/mask_rcnn_r50_fpn_15e_deepfashion_20200329_192752.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/deepfashion/mask_rcnn_r50_fpn_15e_deepfashion/20200329_192752.log.json) |

```latex

```latex
   author = {Liu, Ziwei and Luo, Ping and Qiu, Shi and Wang, Xiaogang and Tang, Xiaoou},
   title = {DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations},
   booktitle = {Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
   month = {June},
   year = {2016}
}
```

# 基准和模型库 (待更新)

- [Rotated RetinaNet-OBB/HBB](https://github.com/open-mmlab/mmrotate/tree/main/configs/rotated_retinanet/README.md) (
  ICCV'2017)
- [Rotated FasterRCNN-OBB](https://github.com/open-mmlab/mmrotate/tree/main/configs/rotated_faster_rcnn/README.md) (
  TPAMI'2017)
- [Rotated RepPoints-OBB](https://github.com/open-mmlab/mmrotate/tree/main/configs/rotated_reppoints/README.md) (
  ICCV'2019)

## DOTA v1.0 数据集上的结果

|          骨干网络          |  mAP  | 角度编码方式 | 训练策略 | 显存占用 (GB) | 推理时间 (fps) | 增强方法 | 批量大小 |                                                                       配置                                                                        |                                                                                                                                                                                                        下载                                                                                                                                                                                                        |

|  ResNet50 (1024,1024,200)  | 59.44 |      oc      |    1x    |     3.45      |      15.6      |    -     |    2     |                [rotated-reppoints-qbox_r50_fpn_1x_dota](../../configs/rotated_reppoints/rotated-reppoints-qbox_r50_fpn_1x_dota.py)                |                                   [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_reppoints/rotated_reppoints_r50_fpn_1x_dota_oc/rotated_reppoints_r50_fpn_1x_dota_oc-d38ce217.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_reppoints/rotated_reppoints_r50_fpn_1x_dota_oc/rotated_reppoints_r50_fpn_1x_dota_oc_20220205_145010.log.json)                                   |

|  ResNet50 (1024,1024,200)  | 64.55 |      oc      |    1x    |     3.38      |      15.7      |    -     |    2     |             [rotated-retinanet-hbox-oc_r50_fpn_1x_dota](../../configs/rotated_retinanet/rotated-retinanet-hbox-oc_r50_fpn_1x_dota.py)             |                           [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_hbb_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_r50_fpn_1x_dota_oc-e8a7c7df.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_hbb_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_r50_fpn_1x_dota_oc_20220121_095315.log.json)                           |

|  ResNet50 (1024,1024,200)  | 65.59 |      oc      |    1x    |     3.12      |      18.5      |    -     |    2     |                     [rotated_atss_hbb_r50_fpn_1x_dota_oc](../../configs/rotated_atss/rotated_atss_hbb_r50_fpn_1x_dota_oc.py)                      |                                          [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_atss/rotated_atss_hbb_r50_fpn_1x_dota_oc/rotated_atss_hbb_r50_fpn_1x_dota_oc-eaa94033.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_atss/rotated_atss_hbb_r50_fpn_1x_dota_oc/rotated_atss_hbb_r50_fpn_1x_dota_oc_20220402_002230.log.json)                                          |

|  ResNet50 (1024,1024,200)  | 66.45 |      oc      |    1x    |     3.53      |      15.3      |    -     |    2     |                         [sasm-reppoints-qbox_r50_fpn_1x_dota](../../configs/sasm/sasm-reppoints-qbox_r50_fpn_1x_dota.py)                          |                                                      [model](https://download.openmmlab.com/mmrotate/v0.1.0/sasm/sasm_reppoints_r50_fpn_1x_dota_oc/sasm_reppoints_r50_fpn_1x_dota_oc-6d9edded.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/sasm/sasm_reppoints_r50_fpn_1x_dota_oc/sasm_reppoints_r50_fpn_1x_dota_oc_20220205_144938.log.json)                                                      |

|  ResNet50 (1024,1024,200)  | 68.42 |     le90     |    1x    |     3.38      |      16.9      |    -     |    2     |           [rotated-retinanet-rbox-le90_r50_fpn_1x_dota](../../configs/rotated_retinanet/rotated-retinanet-rbox-le90_r50_fpn_1x_dota.py)           |                       [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le90/rotated_retinanet_obb_r50_fpn_1x_dota_le90-c0097bc4.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le90/rotated_retinanet_obb_r50_fpn_1x_dota_le90_20220128_130740.log.json)                       |

|  ResNet50 (1024,1024,200)  | 68.79 |     le90     |    1x    |     2.36      |      22.4      |    -     |    2     |                [rotated-retinanet-rbox-le90_r50_fpn_amp-1x_dota](../../configs/rotated-retinanet-rbox-le90_r50_fpn_amp-1x_dota.py)                |             [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_fp16_1x_dota_le90/rotated_retinanet_obb_r50_fpn_fp16_1x_dota_le90-01de71b5.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_fp16_1x_dota_le90/rotated_retinanet_obb_r50_fpn_fp16_1x_dota_le90_20220303_183714.log.json)             |

|  ResNet50 (1024,1024,200)  | 69.49 |    le135     |    1x    |     4.05      |      8.6       |    -     |    2     |                        [g_reppoints_r50_fpn_1x_dota_le135](../../configs/g_reppoints/g_reppoints_r50_fpn_1x_dota_le135.py)                        |                                               [model](https://download.openmmlab.com/mmrotate/v0.1.0/g_reppoints/g_reppoints_r50_fpn_1x_dota_le135/g_reppoints_r50_fpn_1x_dota_le135-b840eed7.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/g_reppoints/g_reppoints_r50_fpn_1x_dota_le135/g_reppoints_r50_fpn_1x_dota_le135_20220202_233631.log.json)                                               |

|  ResNet50 (1024,1024,200)  | 69.51 |     le90     |    1x    |     4.40      |      24.0      |    -     |    2     | [rotated-retinanet-rbox-le90_r50_fpn_csl-gaussian_amp-1x_dota](../../configs/csl/rotated-retinanet-rbox-le90_r50_fpn_csl-gaussian_amp-1x_dota.py) | [model](https://download.openmmlab.com/mmrotate/v0.1.0/csl/rotated_retinanet_obb_csl_gaussian_r50_fpn_fp16_1x_dota_le90/rotated_retinanet_obb_csl_gaussian_r50_fpn_fp16_1x_dota_le90-b4271aed.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/csl/rotated_retinanet_obb_csl_gaussian_r50_fpn_fp16_1x_dota_le90/rotated_retinanet_obb_csl_gaussian_r50_fpn_fp16_1x_dota_le90_20220321_010033.log.json) |

|  ResNet50 (1024,1024,200)  | 69.55 |      oc      |    1x    |     3.39      |      15.5      |    -     |    2     |                [rotated-retinanet-hbox-oc_r50_fpn_gwd_1x_dota](../../configs/gwd/rotated-retinanet-hbox-oc_r50_fpn_gwd_1x_dota.py)                |                                 [model](https://download.openmmlab.com/mmrotate/v0.1.0/gwd/rotated_retinanet_hbb_gwd_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_gwd_r50_fpn_1x_dota_oc-41fd7805.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/gwd/rotated_retinanet_hbb_gwd_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_gwd_r50_fpn_1x_dota_oc_20220120_152421.log.json)                                 |

|  ResNet50 (1024,1024,200)  | 69.60 |     le90     |    1x    |     3.38      |      15.1      |    -     |    2     |           [rotated-retinanet-hbox-le90_r50_fpn_kfiou_1x_dota](../../configs/kfiou/rotated-retinanet-hbox-le90_r50_fpn_kfiou_1x_dota.py)           |                       [model](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le90/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le90-03e02f75.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le90/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le90_20220209_173225.log.json)                       |

|  ResNet50 (1024,1024,200)  | 69.63 |    le135     |    1x    |     3.45      |      16.1      |    -     |    2     |                                     [cfa-qbox_r50_fpn_1x_dota](../../configs/cfa/cfa-qbox_r50_fpn_1x_dota.py)                                     |                                                                       [model](https://download.openmmlab.com/mmrotate/v0.1.0/cfa/cfa_r50_fpn_1x_dota_le135/cfa_r50_fpn_1x_dota_le135-aed1cbc6.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/cfa/cfa_r50_fpn_1x_dota_le135/cfa_r50_fpn_1x_dota_le135_20220205_144859.log.json)                                                                       |

|  ResNet50 (1024,1024,200)  | 69.76 |      oc      |    1x    |     3.39      |      15.6      |    -     |    2     |             [rotated-retinanet-hbox-oc_r50_fpn_kfiou_1x_dota](../../configs/kfiou/rotated-retinanet-hbox-oc_r50_fpn_kfiou_1x_dota.py)             |                           [model](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_oc-c00be030.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_oc_20220126_081643.log.json)                           |

|  ResNet50 (1024,1024,200)  | 69.77 |    le135     |    1x    |     3.38      |      15.3      |    -     |    2     |          [rotated-retinanet-hbox-le135_r50_fpn_kfiou_1x_dota](../../configs/kfiou/rotated-retinanet-hbox-le135_r50_fpn_kfiou_1x_dota.py)          |                     [model](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le135/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le135-0eaa4156.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le135/rotated_retinanet_hbb_kfiou_r50_fpn_1x_dota_le135_20220209_173257.log.json)                     |

|  ResNet50 (1024,1024,200)  | 69.79 |    le135     |    1x    |     3.38      |      17.2      |    -     |    2     |          [rotated-retinanet-rbox-le135_r50_fpn_1x_dota](../../configs/rotated_retinanet/rotated-retinanet-rbox-le135_r50_fpn_1x_dota.py)          |                     [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le135/rotated_retinanet_obb_r50_fpn_1x_dota_le135-e4131166.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le135/rotated_retinanet_obb_r50_fpn_1x_dota_le135_20220128_130755.log.json)                     |

|  ResNet50 (1024,1024,200)  | 69.80 |      oc      |    1x    |     3.54      |      12.4      |    -     |    2     |                                    [r3det-oc_r50_fpn_1x_dota](../../configs/r3det/r3det-oc_r50_fpn_1x_dota.py)                                    |                                                                       [model](https://download.openmmlab.com/mmrotate/v0.1.0/r3det/r3det_r50_fpn_1x_dota_oc/r3det_r50_fpn_1x_dota_oc-b1fb045c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/r3det/r3det_r50_fpn_1x_dota_oc/r3det_r50_fpn_1x_dota_oc_20220126_191226.log.json)                                                                       |

|  ResNet50 (1024,1024,200)  | 69.94 |      oc      |    1x    |     3.39      |      15.6      |    -     |    2     |                [rotated-retinanet-hbox-oc_r50_fpn_kld_1x_dota](../../configs/kld/rotated-retinanet-hbox-oc_r50_fpn_kld_1x_dota.py)                |                                 [model](https://download.openmmlab.com/mmrotate/v0.1.0/kld/rotated_retinanet_hbb_kld_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_kld_r50_fpn_1x_dota_oc-49c1f937.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kld/rotated_retinanet_hbb_kld_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_kld_r50_fpn_1x_dota_oc_20220125_201832.log.json)                                 |

|  ResNet50 (1024,1024,200)  | 70.18 |      oc      |    1x    |     3.23      |      15.6      |    -     |    2     |                               [r3det-tiny-oc_r50_fpn_1x_dota](../../configs/r3det/r3det-tiny-oc_r50_fpn_1x_dota.py)                               |                                                             [model](https://download.openmmlab.com/mmrotate/v0.1.0/r3det/r3det_tiny_r50_fpn_1x_dota_oc/r3det_tiny_r50_fpn_1x_dota_oc-c98a616c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/r3det/r3det_tiny_r50_fpn_1x_dota_oc/r3det_tiny_r50_fpn_1x_dota_oc_20220209_171624.log.json)                                                             |

|  ResNet50 (1024,1024,200)  | 70.64 |     le90     |    1x    |     3.12      |      18.2      |    -     |    2     |                   [rotated_atss_obb_r50_fpn_1x_dota_le90](../../configs/rotated_atss/rotated_atss_obb_r50_fpn_1x_dota_le90.py)                    |                                      [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_atss/rotated_atss_obb_r50_fpn_1x_dota_le90/rotated_atss_obb_r50_fpn_1x_dota_le90-e029ca06.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_atss/rotated_atss_obb_r50_fpn_1x_dota_le90/rotated_atss_obb_r50_fpn_1x_dota_le90_20220402_002048.log.json)                                      |

|  ResNet50 (1024,1024,200)  | 70.70 |     le90     |    1x    |     4.18      |                |    -     |    2     |                  [rotated-fcos-hbox-le90_r50_fpn_1x_dota](../../configs/rotated_fcos/rotated-fcos-hbox-le90_r50_fpn_1x_dota.py)                   |                          [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_sep_angle_r50_fpn_1x_dota_le90/rotated_fcos_sep_angle_r50_fpn_1x_dota_le90-0be71a0c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_sep_angle_r50_fpn_1x_dota_le90/rotated_fcos_sep_angle_r50_fpn_1x_dota_le90_20220409_023250.log.json)                          |

|  ResNet50 (1024,1024,200)  | 71.28 |     le90     |    1x    |     4.18      |                |    -     |    2     |                       [rotated-fcos-le90_r50_fpn_1x_dota](../../configs/rotated_fcos/rotated-fcos-le90_r50_fpn_1x_dota.py)                        |                                              [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_r50_fpn_1x_dota_le90/rotated_fcos_r50_fpn_1x_dota_le90-d87568ed.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_r50_fpn_1x_dota_le90/rotated_fcos_r50_fpn_1x_dota_le90_20220413_163526.log.json)                                              |

|  ResNet50 (1024,1024,200)  | 71.76 |     le90     |    1x    |     4.23      |                |    -     |    2     |     [rotated-fcos-hbox-le90_r50_fpn_csl-gaussian_1x_dota](../../configs/rotated_fcos/rotated-fcos-hbox-le90_r50_fpn_csl-gaussian_1x_dota.py)      |                    [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_csl_gaussian_r50_fpn_1x_dota_le90/rotated_fcos_csl_gaussian_r50_fpn_1x_dota_le90-4e044ad2.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_csl_gaussian_r50_fpn_1x_dota_le90/rotated_fcos_csl_gaussian_r50_fpn_1x_dota_le90_20220409_080616.log.json)                    |

|  ResNet50 (1024,1024,200)  | 71.83 |      oc      |    1x    |     3.54      |      12.4      |    -     |    2     |                          [r3det-oc_r50_fpn_kld-stable_1x_dota](../../configs/kld/r3det-oc_r50_fpn_kld-stable_1x_dota.py)                          |                                                                 [model](https://download.openmmlab.com/mmrotate/v0.1.0/kld/r3det_kld_r50_fpn_1x_dota_oc/r3det_kld_r50_fpn_1x_dota_oc-31866226.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kld/r3det_kld_r50_fpn_1x_dota_oc/r3det_kld_r50_fpn_1x_dota_oc_20220210_114049.log.json)                                                                 |

|  ResNet50 (1024,1024,200)  | 71.89 |     le90     |    1x    |     4.18      |                |    -     |    2     |                   [rotated-fcos-le90_r50_fpn_kld_1x_dota](../../configs/rotated_fcos/rotated-fcos-le90_r50_fpn_kld_1x_dota.py)                    |                                      [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_kld_r50_fpn_1x_dota_le90/rotated_fcos_kld_r50_fpn_1x_dota_le90-ecafdb2b.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_fcos/rotated_fcos_kld_r50_fpn_1x_dota_le90/rotated_fcos_kld_r50_fpn_1x_dota_le90_20220409_202939.log.json)                                      |

|  ResNet50 (1024,1024,200)  | 71.94 |    le135     |    1x    |     3.45      |      16.1      |    -     |    2     |              [oriented-reppoints-qbox_r50_fpn_1x_dota](../../configs/oriented_reppoints/oriented-reppoints-qbox_r50_fpn_1x_dota.py)               |                           [model](https://download.openmmlab.com/mmrotate/v0.1.0/orientedreppoints/oriented_reppoints_r50_fpn_1x_dota_le135/oriented_reppoints_r50_fpn_1x_dota_le135-ef072de9.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/orientedreppoints/oriented_reppoints_r50_fpn_1x_dota_le135/oriented_reppoints_r50_fpn_1x_dota_le135_20220505_092853.log.json)                           |

|  ResNet50 (1024,1024,200)  | 72.29 |    le135     |    1x    |     3.19      |      18.8      |    -     |    2     |                  [rotated_atss_obb_r50_fpn_1x_dota_le135](../../configs/rotated_atss/rotated_atss_obb_r50_fpn_1x_dota_le135.py)                   |                                    [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_atss/rotated_atss_obb_r50_fpn_1x_dota_le135/rotated_atss_obb_r50_fpn_1x_dota_le135-eab7bc12.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_atss/rotated_atss_obb_r50_fpn_1x_dota_le135/rotated_atss_obb_r50_fpn_1x_dota_le135_20220402_002138.log.json)                                    |

|  ResNet50 (1024,1024,200)  | 72.68 |      oc      |    1x    |     3.62      |      12.2      |    -     |    2     |                           [r3det-oc_r50_fpn_kfiou-ln_1x_dota](../../configs/kfiou/r3det-oc_r50_fpn_kfiou-ln_1x_dota.py)                           |                                                     [model](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/r3det_kfiou_ln_r50_fpn_1x_dota_oc/r3det_kfiou_ln_r50_fpn_1x_dota_oc-8e7f049d.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kfiou/r3det_kfiou_ln_r50_fpn_1x_dota_oc/r3det_kfiou_ln_r50_fpn_1x_dota_oc_20220123_074507.log.json)                                                     |

|  ResNet50 (1024,1024,200)  | 72.76 |      oc      |    1x    |     3.44      |      14.0      |    -     |    2     |                            [r3det-tiny-oc_r50_fpn_kld_1x_dota](../../configs/kld/r3det-tiny-oc_r50_fpn_kld_1x_dota.py)                            |                                                       [model](https://download.openmmlab.com/mmrotate/v0.1.0/kld/r3det_tiny_kld_r50_fpn_1x_dota_oc/r3det_tiny_kld_r50_fpn_1x_dota_oc-589e142a.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/kld/r3det_tiny_kld_r50_fpn_1x_dota_oc/r3det_tiny_kld_r50_fpn_1x_dota_oc_20220209_172917.log.json)                                                       |

|  ResNet50 (1024,1024,200)  | 73.23 |     le90     |    1x    |     8.45      |      16.4      |    -     |    2     |                    [gliding-vertex-rbox_r50_fpn_1x_dota](../../configs/gliding_vertex/gliding-vertex-rbox_r50_fpn_1x_dota.py)                     |                                        [model](https://download.openmmlab.com/mmrotate/v0.1.0/gliding_vertex/gliding_vertex_r50_fpn_1x_dota_le90/gliding_vertex_r50_fpn_1x_dota_le90-12e7423c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/gliding_vertex/gliding_vertex_r50_fpn_1x_dota_le90/gliding_vertex_r50_fpn_1x_dota_le90_20220129_085529.log.json)                                        |

|  ResNet50 (1024,1024,200)  | 73.40 |     le90     |    1x    |     8.46      |      16.5      |    -     |    2     |             [rotated-faster-rcnn-le90_r50_fpn_1x_dota](../../configs/rotated_faster_rcnn/rotated-faster-rcnn-le90_r50_fpn_1x_dota.py)             |                         [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_faster_rcnn/rotated_faster_rcnn_r50_fpn_1x_dota_le90/rotated_faster_rcnn_r50_fpn_1x_dota_le90-0393aa5c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_faster_rcnn/rotated_faster_rcnn_r50_fpn_1x_dota_le90/rotated_faster_rcnn_r50_fpn_1x_dota_le90_20220131_082156.log.json)                         |

|  ResNet50 (1024,1024,200)  | 73.45 |      oc      |   40e    |     3.45      |      16.1      |    -     |    2     |                                     [cfa-qbox_r50_fpn_40e_dota](../../configs/cfa/cfa_r50_fpn_40e_dota_oc.py)                                     |                                                                           [model](https://download.openmmlab.com/mmrotate/v0.1.0/cfa/cfa_r50_fpn_40e_dota_oc/cfa_r50_fpn_40e_dota_oc-2f387232.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/cfa/cfa_r50_fpn_40e_dota_oc/cfa_r50_fpn_40e_dota_oc_20220209_171237.log.json)                                                                           |

|  ResNet50 (1024,1024,200)  | 73.91 |    le135     |    1x    |     3.14      |      15.5      |    -     |    2     |                               [s2anet-le135_r50_fpn_1x_dota](../../configs/s2anet/s2anet-le135_r50_fpn_1x_dota.py)                                |                                                              [model](https://download.openmmlab.com/mmrotate/v0.1.0/s2anet/s2anet_r50_fpn_1x_dota_le135/s2anet_r50_fpn_1x_dota_le135-5dfcf396.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/s2anet/s2anet_r50_fpn_1x_dota_le135/s2anet_r50_fpn_1x_dota_le135_20220124_163529.log.json)                                                              |

|  ResNet50 (1024,1024,200)  | 74.19 |    le135     |    1x    |     2.17      |      17.4      |    -     |    2     |                               [s2anet-le135_r50_fpn_amp-1x_dota](../../configs/s2anet-le135_r50_fpn_amp-1x_dota.py)                               |                                                    [model](https://download.openmmlab.com/mmrotate/v0.1.0/s2anet/s2anet_r50_fpn_fp16_1x_dota_le135/s2anet_r50_fpn_fp16_1x_dota_le135-5cac515c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/s2anet/s2anet_r50_fpn_fp16_1x_dota_le135/s2anet_r50_fpn_fp16_1x_dota_le135_20220303_194910.log.json)                                                    |

|  ResNet50 (1024,1024,200)  | 75.63 |     le90     |    1x    |     7.37      |      21.2      |    -     |    2     |                         [oriented-rcnn-le90_r50_fpn_amp-1x_dota](../../configs/oriented-rcnn-le90_r50_fpn_amp-1x_dota.py)                         |                                 [model](https://download.openmmlab.com/mmrotate/v0.1.0/oriented_rcnn/oriented_rcnn_r50_fpn_fp16_1x_dota_le90/oriented_rcnn_r50_fpn_fp16_1x_dota_le90-57c88621.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/oriented_rcnn/oriented_rcnn_r50_fpn_fp16_1x_dota_le90/oriented_rcnn_r50_fpn_fp16_1x_dota_le90_20220303_195049.log.json)                                 |

|  ResNet50 (1024,1024,200)  | 75.69 |     le90     |    1x    |     8.46      |      16.2      |    -     |    2     |                      [oriented-rcnn-le90_r50_fpn_1x_dota](../../configs/oriented_rcnn/oriented-rcnn-le90_r50_fpn_1x_dota.py)                      |                                           [model](https://download.openmmlab.com/mmrotate/v0.1.0/oriented_rcnn/oriented_rcnn_r50_fpn_1x_dota_le90/oriented_rcnn_r50_fpn_1x_dota_le90-6d2b2ce0.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/oriented_rcnn/oriented_rcnn_r50_fpn_1x_dota_le90/oriented_rcnn_r50_fpn_1x_dota_le90_20220127_100150.log.json)                                           |

|  ResNet50 (1024,1024,200)  | 75.75 |     le90     |    1x    |     7.56      |      19.3      |    -     |    2     |                             [roi-trans-le90_r50_fpn_amp-1x_dota](../../configs/roi-trans-le90_r50_fpn_amp-1x_dota.py)                             |                                             [model](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_r50_fpn_fp16_1x_dota_le90/roi_trans_r50_fpn_fp16_1x_dota_le90-62eb88b1.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_r50_fpn_fp16_1x_dota_le90/roi_trans_r50_fpn_fp16_1x_dota_le90_20220303_193513.log.json)                                             |

| ReResNet50 (1024,1024,200) | 75.99 |     le90     |    1x    |     7.71      |      13.3      |    -     |    2     |                              [redet-le90_re50_refpn_amp-1x_dota](../../configs/redet-le90_re50_refpn_amp-1x_dota.py)                              |                                                   [model](https://download.openmmlab.com/mmrotate/v0.1.0/redet/redet_re50_refpn_fp16_1x_dota_le90/redet_re50_refpn_fp16_1x_dota_le90-1e34da2d.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/redet/redet_re50_refpn_fp16_1x_dota_le90/redet_re50_refpn_fp16_1x_dota_le90_20220303_194725.log.json)                                                   |

|  ResNet50 (1024,1024,200)  | 76.08 |     le90     |    1x    |     8.67      |      14.4      |    -     |    2     |                            [roi-trans-le90_r50_fpn_1x_dota](../../configs/roi_trans/roi-trans-le90_r50_fpn_1x_dota.py)                            |                                                       [model](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_r50_fpn_1x_dota_le90/roi_trans_r50_fpn_1x_dota_le90-d1f0b77a.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_r50_fpn_1x_dota_le90/roi_trans_r50_fpn_1x_dota_le90_20220130_132727.log.json)                                                       |

|  ResNet50 (1024,1024,500)  | 76.50 |     le90     |    1x    |               |      17.5      |  MS+RR   |    2     |     [rotated-retinanet-rbox-le90_r50_fpn_rr-1x_dota-ms](../../configs/rotated_retinanet/rotated-retinanet-rbox-le90_r50_fpn_rr-1x_dota-ms.py)     |           [model](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_ms_rr_le90/rotated_retinanet_obb_r50_fpn_1x_dota_ms_rr_le90-1da1ec9c.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_ms_rr_le90/rotated_retinanet_obb_r50_fpn_1x_dota_ms_rr_le90_20220210_114843.log.json)           |

| ReResNet50 (1024,1024,200) | 76.68 |     le90     |    1x    |     9.32      |      10.9      |    -     |    2     |                               [redet-le90_re50_refpn_1x_dota](../../configs/redet/redet-le90_re50_refpn_1x_dota.py)                               |                                                                 [model](https://download.openmmlab.com/mmrotate/v0.1.0/redet/redet_re50_fpn_1x_dota_le90/redet_re50_fpn_1x_dota_le90-724ab2da.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/redet/redet_re50_fpn_1x_dota_le90/redet_re50_fpn_1x_dota_le90_20220130_132751.log.json)                                                                 |

| Swin-tiny (1024,1024,200)  | 77.51 |     le90     |    1x    |               |      10.9      |    -     |    2     |                      [roi-trans-le90_swin-tiny_fpn_1x_dota](../../configs/roi_trans/roi-trans-le90_swin-tiny_fpn_1x_dota.py)                      |                                           [model](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_swin_tiny_fpn_1x_dota_le90/roi_trans_swin_tiny_fpn_1x_dota_le90-ddeee9ae.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_swin_tiny_fpn_1x_dota_le90/roi_trans_swin_tiny_fpn_1x_dota_le90_20220131_083622.log.json)                                           |

|  ResNet50 (1024,1024,500)  | 79.66 |     le90     |    1x    |               |      14.4      |  MS+RR   |    2     |                      [roi_trans_r50_fpn_1x_dota_ms_rr_le90](../../configs/roi_trans/roi_trans_r50_fpn_1x_dota_ms_rr_le90.py)                      |                                           [model](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_r50_fpn_1x_dota_ms_rr_le90/roi_trans_r50_fpn_1x_dota_ms_rr_le90-fa99496f.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/roi_trans/roi_trans_r50_fpn_1x_dota_ms_rr_le90/roi_trans_r50_fpn_1x_dota_ms_rr_le90_20220205_171729.log.json)                                           |

| ReResNet50 (1024,1024,500) | 79.87 |     le90     |    1x    |               |      10.9      |  MS+RR   |    2     |                         [redet-le90_re50_refpn_rr-1x_dota-ms](../../configs/redet/redet-le90_re50_refpn_rr-1x_dota-ms.py)                         |                                                     [model](https://download.openmmlab.com/mmrotate/v0.1.0/redet/redet_re50_fpn_1x_dota_ms_rr_le90/redet_re50_fpn_1x_dota_ms_rr_le90-fc9217b5.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/redet/redet_re50_fpn_1x_dota_ms_rr_le90/redet_re50_fpn_1x_dota_ms_rr_le90_20220206_105343.log.json)                                                     |

|  ResNet50 (1024,1024,200)  | 68.75 |     le90     |    1x    |     6.25      |                |    -     |    2     |                          [h2rbox-le90_r50_fpn_adamw-1x_dota](../../configs/h2rbox/h2rbox-le90_r50_fpn_adamw-1x_dota.py)                           |                                                        [model](https://download.openmmlab.com/mmrotate/v1.0/h2rbox/h2rbox-le90_r50_fpn_adamw-1x_dota/h2rbox-le90_r50_fpn_adamw-1x_dota-d02c933a.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/h2rbox/h2rbox-le90_r50_fpn_adamw-1x_dota/h2rbox-le90_r50_fpn_adamw-1x_dota-20221124_153420.json)                                                        |

- `MS` 表示多尺度图像增强。
- `RR` 表示随机旋转增强。

上述模型都是使用 1 * 1080ti/2080ti 训练得到的，并且在 1 * 2080ti 上进行推理测试。

# YOLOX

> [YOLOX: Exceeding YOLO Series in 2021](https://arxiv.org/abs/2107.08430)

In this report, we present some experienced improvements to YOLO series, forming a new high-performance detector -- YOLOX. We switch the YOLO detector to an anchor-free manner and conduct other advanced detection techniques, i.e., a decoupled head and the leading label assignment strategy SimOTA to achieve state-of-the-art results across a large scale range of models: For YOLO-Nano with only 0.91M parameters and 1.08G FLOPs, we get 25.3% AP on COCO, surpassing NanoDet by 1.8% AP; for YOLOv3, one of the most widely used detectors in industry, we boost it to 47.3% AP on COCO, outperforming the current best practice by 3.0% AP; for YOLOX-L with roughly the same amount of parameters as YOLOv4-CSP, YOLOv5-L, we achieve 50.0% AP on COCO at a speed of 68.9 FPS on Tesla V100, exceeding YOLOv5-L by 1.8% AP. Further, we won the 1st Place on Streaming Perception Challenge (Workshop on Autonomous Driving at CVPR 2021) using a single YOLOX-L model. We hope this report can provide useful experience for developers and researchers in practical scenes, and we also provide deploy versions with ONNX, TensorRT, NCNN, and Openvino supported.

<img src="https://user-images.githubusercontent.com/40661020/144001736-9fb303dd-eac7-46b0-ad45-214cfa51e928.png"/>
</div>

|  Backbone  | size | Mem (GB) | box AP |                  Config                  |                                                                                                                                         Download                                                                                                                                         |
| :--------: | :--: | :------: | :----: | :--------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

| YOLOX-tiny | 416  |   3.5    |  32.0  | [config](./yolox_tiny_8xb8-300e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/yolox/yolox_tiny_8x8_300e_coco/yolox_tiny_8x8_300e_coco_20211124_171234-b4047906.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/yolox/yolox_tiny_8x8_300e_coco/yolox_tiny_8x8_300e_coco_20211124_171234.log.json) |
|  YOLOX-s   | 640  |   7.6    |  40.5  |  [config](./yolox_s_8xb8-300e_coco.py)   |       [model](https://download.openmmlab.com/mmdetection/v2.0/yolox/yolox_s_8x8_300e_coco/yolox_s_8x8_300e_coco_20211121_095711-4592a793.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/yolox/yolox_s_8x8_300e_coco/yolox_s_8x8_300e_coco_20211121_095711.log.json)       |

|  YOLOX-l   | 640  |   19.9   |  49.4  |  [config](./yolox_l_8xb8-300e_coco.py)   |       [model](https://download.openmmlab.com/mmdetection/v2.0/yolox/yolox_l_8x8_300e_coco/yolox_l_8x8_300e_coco_20211126_140236-d3bd2b23.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/yolox/yolox_l_8x8_300e_coco/yolox_l_8x8_300e_coco_20211126_140236.log.json)       |
|  YOLOX-x   | 640  |   28.1   |  50.9  |  [config](./yolox_x_8xb8-300e_coco.py)   |       [model](https://download.openmmlab.com/mmdetection/v2.0/yolox/yolox_x_8x8_300e_coco/yolox_x_8x8_300e_coco_20211126_140254-1ef88d67.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/yolox/yolox_x_8x8_300e_coco/yolox_x_8x8_300e_coco_20211126_140254.log.json)       |

**Note**:

1. The test score threshold is 0.001, and the box AP indicates the best AP.

**Note**:

2. Due to the need for pre-training weights, we cannot reproduce the performance of the `yolox-nano` model. Please refer to https://github.com/Megvii-BaseDetection/YOLOX/issues/674 for more information.
3. We also trained the model by the official release of YOLOX based on [Megvii-BaseDetection/YOLOX#735](https://github.com/Megvii-BaseDetection/YOLOX/issues/735) with commit ID [38c633](https://github.com/Megvii-BaseDetection/YOLOX/tree/38c633bf176462ee42b110c70e4ffe17b5753208). We found that the best AP of `YOLOX-tiny`, `YOLOX-s`, `YOLOX-l`, and `YOLOX-x` is 31.8, 40.3, 49.2, and 50.9, respectively. The performance is consistent with that of our re-implementation (see Table above) but still has a gap (0.3~0.8 AP) in comparison with the reported performance in their [README](https://github.com/Megvii-BaseDetection/YOLOX/blob/38c633bf176462ee42b110c70e4ffe17b5753208/README.md#benchmark).

```latex
}
```

# 准备 ActivityNet

## 简介

```BibTeX
}
```

用户可参考该数据集的 [官网](http://activity-net.org/)，以获取数据集相关的基本信息。
对于时序动作检测任务，用户可以使用这个 [代码库](https://github.com/wzmsltw/BSN-boundary-sensitive-network#code-and-data-preparation) 提供的缩放过（rescaled）的 ActivityNet 特征，
或者使用 MMAction2 进行特征提取（这将具有更高的精度）。MMAction2 同时提供了以上所述的两种数据使用流程。
在数据集准备前，请确保命令行当前路径为 `$MMACTION2/tools/data/activitynet/`。

## 选项 1：用户可以使用这个 [代码库](https://github.com/wzmsltw/BSN-boundary-sensitive-network#code-and-data-preparation) 提供的特征

### 步骤 1. 下载标注文件

首先，用户可以使用以下命令下载标注文件。

```shell
```

### 步骤 2. 准备视频特征

之后，用户可以使用以下命令下载 ActivityNet 特征。

```shell

```shell
```

```shell
```

### 步骤 3. 处理标注文件

之后，用户可以使用以下命令处理下载的标注文件，以便于训练和测试。
该脚本会首先合并两个标注文件，然后再将其分为 `train`, `val` 和 `test` 三个部分。

```shell
```

## 选项 2：使用 MMAction2 对 [官网](http://activity-net.org/) 提供的视频进行特征抽取

```shell
```

### 步骤 2. 准备视频

之后，用户可以使用以下脚本准备视频数据。
该代码参考自 [官方爬虫](https://github.com/activitynet/ActivityNet/tree/master/Crawler/Kinetics)，该过程将会耗费较多时间。

```shell
```

由于 ActivityNet 数据集中的一些视频已经在 YouTube 失效，[官网](http://activity-net.org/) 在谷歌网盘和百度网盘提供了完整的数据集数据。
如果用户想要获取失效的数据集，则需要填写 [下载页面](http://activity-net.org/download.html) 中提供的 [需求表格](https://docs.google.com/forms/d/e/1FAIpQLSeKaFq9ZfcmZ7W0B0PbEhfbTHY41GeEgwsa7WobJgGUhn4DTQ/viewform) 以获取 7 天的下载权限。

MMAction2 同时也提供了 [BSN 代码库](https://github.com/wzmsltw/BSN-boundary-sensitive-network#code-and-data-preparation) 的标注文件的下载步骤。

```shell
```

对于这种情况，该下载脚本将在下载后更新此标注文件，以确保每个视频都存在。

### 步骤 3. 抽取 RGB 帧和光流

可使用以下命令抽取视频帧和光流。

```shell
```

以上脚本将会生成短边 256 分辨率的视频。如果用户想生成短边 320 分辨率的视频（即 320p），或者 340x256 的固定分辨率，用户可以通过改变参数由 `--new-short 256` 至 `--new-short 320`，或者 `--new-width 340 --new-height 256` 进行设置
更多细节可参考 [数据准备指南](/docs/zh_cn/user_guides/prepare_dataset.md)

### 步骤 4. 生成用于 ActivityNet 微调的文件列表

根据抽取的帧，用户可以生成视频级别（video-level）或者片段级别（clip-level）的文件列表，其可用于微调 ActivityNet。

```shell
```

```shell
```

### 步骤 5. 在 ActivityNet 上微调 TSN 模型

用户可使用 `configs/recognition/tsn` 目录中的 ActivityNet 配置文件进行 TSN 模型微调。
用户需要使用 Kinetics 相关模型（同时支持 RGB 模型与光流模型）进行预训练。

### 步骤 6. 使用预训练模型进行 ActivityNet 特征抽取

在 ActivityNet 上微调 TSN 模型之后，用户可以使用该模型进行 RGB 特征和光流特征的提取。

```shell

```

在提取完特征后，用户可以使用后处理脚本整合 RGB 特征和光流特征，生成 `100-t X 400-d` 维度的特征用于时序动作检测。

```shell
```

## 最后一步：检查文件夹结构

```

在完成所有 ActivityNet 数据集准备流程后，用户可以获得对应的特征文件，RGB + 光流文件，视频文件以及标注文件。

在整个 MMAction2 文件夹下，ActivityNet 的文件结构如下：

```
mmaction2
├── tools
├── data

(若根据选项 1 进行数据处理)
│   │   │   |   ├── ..

(若根据选项 2 进行数据处理)
│   │   │   ├── ..

│   │   │   ├── ..
│   │   │   │   ├── ..
│   │   │   ├── ..

```

关于对 ActivityNet 进行训练和验证，可以参考 [训练教程](/docs/zh_cn/user_guides/train_test.md).

# 模型库

## 镜像地址

从 MMDetection V2.0 起，我们只通过阿里云维护模型库。V1.x 版本的模型已经弃用。

## 共同设置

- 所有模型都是在 `coco_2017_train` 上训练，在 `coco_2017_val` 上测试。
- 我们使用分布式训练。
- 所有 pytorch-style 的 ImageNet 预训练主干网络来自 PyTorch 的模型库，caffe-style 的预训练主干网络来自 detectron2 最新开源的模型。
- 为了与其他代码库公平比较，文档中所写的 GPU 内存是8个 GPU 的 `torch.cuda.max_memory_allocated()` 的最大值，此值通常小于 nvidia-smi 显示的值。
- 我们以网络 forward 和后处理的时间加和作为推理时间，不包含数据加载时间。所有结果通过 [benchmark.py](https://github.com/open-mmlab/mmdetection/blob/main/tools/analysis_tools/benchmark.py) 脚本计算所得。该脚本会计算推理 2000 张图像的平均时间。

## ImageNet 预训练模型

通过 ImageNet 分类任务预训练的主干网络进行初始化是很常见的操作。所有预训练模型的链接都可以在 [open_mmlab](https://github.com/open-mmlab/mmcv/blob/master/mmcv/model_zoo/open_mmlab.json) 中找到。根据 `img_norm_cfg` 和原始权重，我们可以将所有 ImageNet 预训练模型分为以下几种情况：

- TorchVision：torchvision 模型权重，包含 ResNet50, ResNet101。`img_norm_cfg` 为 `dict(mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)`。

- Pycls：[pycls](https://github.com/facebookresearch/pycls) 模型权重，包含 RegNetX。`img_norm_cfg` 为 `dict(   mean=[103.530, 116.280, 123.675], std=[57.375, 57.12, 58.395], to_rgb=False)`。
- MSRA styles：[MSRA](https://github.com/KaimingHe/deep-residual-networks) 模型权重，包含 ResNet50_Caffe，ResNet101_Caffe。`img_norm_cfg` 为 `dict(   mean=[103.530, 116.280, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False)`。
- Caffe2 styles：现阶段只包含 ResNext101_32x8d。`img_norm_cfg` 为 `dict(mean=[103.530, 116.280, 123.675], std=[57.375, 57.120, 58.395], to_rgb=False)`。
- Other styles: SSD 的 `img_norm_cfg` 为 `dict(mean=[123.675, 116.28, 103.53], std=[1, 1, 1], to_rgb=True)`，YOLOv3 的 `img_norm_cfg` 为 `dict(mean=[0, 0, 0], std=[255., 255., 255.], to_rgb=True)`。

MMdetection 常用到的主干网络细节如下表所示：

| 模型             | 来源        | 链接                                                                                                                                                                                                | 描述                                                                                                                                                                                                                       |
| ---------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

| ResNet50         | TorchVision | [torchvision 中的 ResNet-50](https://download.pytorch.org/models/resnet50-19c8e357.pth)                                                                                                             | 来自 [torchvision 中的 ResNet-50](https://download.pytorch.org/models/resnet50-19c8e357.pth)。                                                                                                                             |
| ResNet101        | TorchVision | [torchvision 中的 ResNet-101](https://download.pytorch.org/models/resnet101-5d3b4d8f.pth)                                                                                                           | 来自 [torchvision 中的 ResNet-101](https://download.pytorch.org/models/resnet101-5d3b4d8f.pth)。                                                                                                                           |

| RegNetX          | Pycls       | [RegNetX_3.2gf](https://download.openmmlab.com/pretrain/third_party/regnetx_3.2gf-c2599b0f.pth)，[RegNetX_800mf](https://download.openmmlab.com/pretrain/third_party/regnetx_800mf-1f4be4c7.pth) 等 | 来自 [pycls](https://github.com/facebookresearch/pycls)。                                                                                                                                                                  |
| ResNet50_Caffe   | MSRA        | [MSRA 中的 ResNet-50](https://download.openmmlab.com/pretrain/third_party/resnet50_caffe-788b5fa3.pth)                                                                                              | 由 [Detectron2 中的 R-50.pkl](https://dl.fbaipublicfiles.com/detectron2/ImageNetPretrained/MSRA/R-50.pkl) 转化的副本。原始权重文件来自 [MSRA 中的原始 ResNet-50](https://github.com/KaimingHe/deep-residual-networks)。    |

| ResNet101_Caffe  | MSRA        | [MSRA 中的 ResNet-101](https://download.openmmlab.com/pretrain/third_party/resnet101_caffe-3ad79236.pth)                                                                                            | 由 [Detectron2 中的 R-101.pkl](https://dl.fbaipublicfiles.com/detectron2/ImageNetPretrained/MSRA/R-101.pkl) 转化的副本。原始权重文件来自 [MSRA 中的原始 ResNet-101](https://github.com/KaimingHe/deep-residual-networks)。 |
| ResNext101_32x8d | Caffe2      | [Caffe2 ResNext101_32x8d](https://download.openmmlab.com/pretrain/third_party/resnext101_32x8d-1516f1aa.pth)                                                                                        | 由 [Detectron2 中的 X-101-32x8d.pkl](https://dl.fbaipublicfiles.com/detectron2/ImageNetPretrained/FAIR/X-101-32x8d.pkl) 转化的副本。原始 ResNeXt-101-32x8d 由 FB 使用 Caffe2 训练。                                        |

### RPN

请参考 [RPN](https://github.com/open-mmlab/mmdetection/blob/main/configs/rpn)。

### RPN

请参考 [Faster R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/faster_rcnn)。

请参考 [Mask R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/mask_rcnn)。

### Fast R-CNN (使用提前计算的 proposals)

请参考 [Fast R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/fast_rcnn)。

请参考 [RetinaNet](https://github.com/open-mmlab/mmdetection/blob/main/configs/retinanet)。

请参考 [Cascade R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/cascade_rcnn)。

请参考 [HTC](https://github.com/open-mmlab/mmdetection/blob/main/configs/htc)。

### SSD

请参考 [SSD](https://github.com/open-mmlab/mmdetection/blob/main/configs/ssd)。

### SSD

请参考 [Group Normalization](https://github.com/open-mmlab/mmdetection/blob/main/configs/gn)。

请参考 [Weight Standardization](https://github.com/open-mmlab/mmdetection/blob/main/configs/gn+ws)。

请参考 [Deformable Convolutional Networks](https://github.com/open-mmlab/mmdetection/blob/main/configs/dcn)。

请参考 [CARAFE](https://github.com/open-mmlab/mmdetection/blob/main/configs/carafe)。

请参考 [Instaboost](https://github.com/open-mmlab/mmdetection/blob/main/configs/instaboost)。

请参考 [Libra R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/libra_rcnn)。

请参考 [Guided Anchoring](https://github.com/open-mmlab/mmdetection/blob/main/configs/guided_anchoring)。

### FCOS

请参考 [FCOS](https://github.com/open-mmlab/mmdetection/blob/main/configs/fcos)。

请参考 [FoveaBox](https://github.com/open-mmlab/mmdetection/blob/main/configs/foveabox)。

请参考 [RepPoints](https://github.com/open-mmlab/mmdetection/blob/main/configs/reppoints)。

请参考 [FreeAnchor](https://github.com/open-mmlab/mmdetection/blob/main/configs/free_anchor)。

请参考 [Grid R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/grid_rcnn)。

### GHM

请参考 [GHM](https://github.com/open-mmlab/mmdetection/blob/main/configs/ghm)。

### GCNet

请参考 [GCNet](https://github.com/open-mmlab/mmdetection/blob/main/configs/gcnet)。

### HRNet

### GHM

### GCNet

### HRNet

请参考 [HRNet](https://github.com/open-mmlab/mmdetection/blob/main/configs/hrnet)。

请参考 [Mask Scoring R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/ms_rcnn)。

请参考 [Rethinking ImageNet Pre-training](https://github.com/open-mmlab/mmdetection/blob/main/configs/scratch)。

请参考 [NAS-FPN](https://github.com/open-mmlab/mmdetection/blob/main/configs/nas_fpn)。

### ATSS

请参考 [ATSS](https://github.com/open-mmlab/mmdetection/blob/main/configs/atss)。

### FSAF

请参考 [FSAF](https://github.com/open-mmlab/mmdetection/blob/main/configs/fsaf)。

请参考 [RegNet](https://github.com/open-mmlab/mmdetection/blob/main/configs/regnet)。

请参考 [Res2Net](https://github.com/open-mmlab/mmdetection/blob/main/configs/res2net)。

### GRoIE

请参考 [GRoIE](https://github.com/open-mmlab/mmdetection/blob/main/configs/groie)。

请参考 [Dynamic R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/dynamic_rcnn)。

请参考 [PointRend](https://github.com/open-mmlab/mmdetection/blob/main/configs/point_rend)。

请参考 [DetectoRS](https://github.com/open-mmlab/mmdetection/blob/main/configs/detectors)。

请参考 [Generalized Focal Loss](https://github.com/open-mmlab/mmdetection/blob/main/configs/gfl)。

请参考 [CornerNet](https://github.com/open-mmlab/mmdetection/blob/main/configs/cornernet)。

### YOLOv3

请参考 [YOLOv3](https://github.com/open-mmlab/mmdetection/blob/main/configs/yolo)。

### PAA

### YOLOv3

### PAA

请参考 [PAA](https://github.com/open-mmlab/mmdetection/blob/main/configs/paa)。

### SABL

请参考 [SABL](https://github.com/open-mmlab/mmdetection/blob/main/configs/sabl)。

请参考 [CentripetalNet](https://github.com/open-mmlab/mmdetection/blob/main/configs/centripetalnet)。

请参考 [ResNeSt](https://github.com/open-mmlab/mmdetection/blob/main/configs/resnest)。

### DETR

请参考 [DETR](https://github.com/open-mmlab/mmdetection/blob/main/configs/detr)。

请参考 [Deformable DETR](https://github.com/open-mmlab/mmdetection/blob/main/configs/deformable_detr)。

请参考 [AutoAssign](https://github.com/open-mmlab/mmdetection/blob/main/configs/autoassign)。

### YOLOF

请参考 [YOLOF](https://github.com/open-mmlab/mmdetection/blob/main/configs/yolof)。

### YOLOF

请参考 [Seesaw Loss](https://github.com/open-mmlab/mmdetection/blob/main/configs/seesaw_loss)。

请参考 [CenterNet](https://github.com/open-mmlab/mmdetection/blob/main/configs/centernet)。

### YOLOX

请参考 [YOLOX](https://github.com/open-mmlab/mmdetection/blob/main/configs/yolox)。

### PVT

请参考 [PVT](https://github.com/open-mmlab/mmdetection/blob/main/configs/pvt)。

### SOLO

请参考 [SOLO](https://github.com/open-mmlab/mmdetection/blob/main/configs/solo)。

请参考 [QueryInst](https://github.com/open-mmlab/mmdetection/blob/main/configs/queryinst)。

我们还在 [PASCAL VOC](https://github.com/open-mmlab/mmdetection/blob/main/configs/pascal_voc)，[Cityscapes](https://github.com/open-mmlab/mmdetection/blob/main/configs/cityscapes) 和 [WIDER FACE](https://github.com/open-mmlab/mmdetection/blob/main/configs/wider_face) 上对一些方法进行了基准测试。

我们还通过多尺度训练和更长的训练策略来训练用 ResNet-50 和 [RegNetX-3.2G](https://github.com/open-mmlab/mmdetection/blob/main/configs/regnet) 作为主干网络的 [Faster R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/faster_rcnn) 和 [Mask R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/mask_rcnn)。这些模型可以作为下游任务的预训练模型。

## 速度基准

### 训练速度基准

我们提供 [analyze_logs.py](https://github.com/open-mmlab/mmdetection/blob/main/tools/analysis_tools/analyze_logs.py) 来得到训练中每一次迭代的平均时间。示例请参考 [Log Analysis](https://mmdetection.readthedocs.io/en/latest/useful_tools.html#log-analysis)。

我们与其他流行框架的 Mask R-CNN 训练速度进行比较（数据是从 [detectron2](https://github.com/facebookresearch/detectron2/blob/main/docs/notes/benchmarks.md/) 复制而来）。在 mmdetection 中，我们使用 [mask-rcnn_r50-caffe_fpn_poly-1x_coco_v1.py](https://github.com/open-mmlab/mmdetection/blob/main/configs/mask_rcnn/mask-rcnn_r50-caffe_fpn_poly-1x_coco_v1.py) 进行基准测试。它与 detectron2 的 [mask_rcnn_R_50_FPN_noaug_1x.yaml](https://github.com/facebookresearch/detectron2/blob/main/configs/Detectron1-Comparisons/mask_rcnn_R_50_FPN_noaug_1x.yaml) 设置完全一样。同时，我们还提供了[模型权重](https://download.openmmlab.com/mmdetection/v2.0/benchmark/mask_rcnn_r50_caffe_fpn_poly_1x_coco_no_aug/mask_rcnn_r50_caffe_fpn_poly_1x_coco_no_aug_compare_20200518-10127928.pth)和[训练 log](https://download.openmmlab.com/mmdetection/v2.0/benchmark/mask_rcnn_r50_caffe_fpn_poly_1x_coco_no_aug/mask_rcnn_r50_caffe_fpn_poly_1x_coco_no_aug_20200518_105755.log.json) 作为参考。为了跳过 GPU 预热时间，吞吐量按照100-500次迭代之间的平均吞吐量来计算。

| 框架                                                                                   | 吞吐量 (img/s) |
| -------------------------------------------------------------------------------------- | -------------- |
| [Detectron2](https://github.com/facebookresearch/detectron2)                           | 62             |
| [MMDetection](https://github.com/open-mmlab/mmdetection)                               | 61             |
| [maskrcnn-benchmark](https://github.com/facebookresearch/maskrcnn-benchmark/)          | 53             |
| [tensorpack](https://github.com/tensorpack/tensorpack/tree/master/examples/FasterRCNN) | 50             |
| [simpledet](https://github.com/TuSimple/simpledet/)                                    | 39             |
| [Detectron](https://github.com/facebookresearch/Detectron)                             | 19             |
| [matterport/Mask_RCNN](https://github.com/matterport/Mask_RCNN/)                       | 14             |

### 推理时间基准

### 推理时间基准

我们提供 [benchmark.py](https://github.com/open-mmlab/mmdetection/blob/main/tools/analysis_tools/benchmark.py) 对推理时间进行基准测试。此脚本将推理 2000 张图片并计算忽略前 5 次推理的平均推理时间。可以通过设置 `LOG-INTERVAL` 来改变 log 输出间隔（默认为 50）。

```shell
```

模型库中，所有模型在基准测量推理时间时都没设置 `fuse-conv-bn`, 此设置可以使推理时间更短。

## 与 Detectron2 对比

我们在速度和精度方面对 mmdetection 和 [Detectron2](https://github.com/facebookresearch/detectron2.git) 进行对比。对比所使用的 detectron2 的 commit id 为 [185c27e](https://github.com/facebookresearch/detectron2/tree/185c27e4b4d2d4c68b5627b3765420c6d7f5a659)(30/4/2020)。
为了公平对比，我们所有的实验都在同一机器下进行。

### 硬件

### 软件环境

- CUDA 10.1

### 精度

### 硬件

### 软件环境

- CUDA 10.1

### 精度

| 模型                                                                                                                            | 训练策略 | Detectron2                                                                                                                             | mmdetection | 下载                                                                                                                                                                                                                                                                                                                                                             |

| [Faster R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/faster_rcnn/faster-rcnn_r50-caffe_fpn_ms-1x_coco.py) | 1x       | [37.9](https://github.com/facebookresearch/detectron2/blob/main/configs/COCO-Detection/faster_rcnn_R_50_FPN_1x.yaml)                   | 38.0        | [model](https://download.openmmlab.com/mmdetection/v2.0/benchmark/faster_rcnn_r50_caffe_fpn_mstrain_1x_coco/faster_rcnn_r50_caffe_fpn_mstrain_1x_coco-5324cff8.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/benchmark/faster_rcnn_r50_caffe_fpn_mstrain_1x_coco/faster_rcnn_r50_caffe_fpn_mstrain_1x_coco_20200429_234554.log.json)             |

| [Mask R-CNN](https://github.com/open-mmlab/mmdetection/blob/main/configs/mask_rcnn/mask-rcnn_r50-caffe_fpn_ms-poly-1x_coco.py)  | 1x       | [38.6 & 35.2](https://github.com/facebookresearch/detectron2/blob/master/configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_1x.yaml) | 38.8 & 35.4 | [model](https://download.openmmlab.com/mmdetection/v2.0/benchmark/mask_rcnn_r50_caffe_fpn_mstrain-poly_1x_coco/mask_rcnn_r50_caffe_fpn_mstrain-poly_1x_coco-dbecf295.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/benchmark/mask_rcnn_r50_caffe_fpn_mstrain-poly_1x_coco/mask_rcnn_r50_caffe_fpn_mstrain-poly_1x_coco_20200430_054239.log.json) |

| [Retinanet](https://github.com/open-mmlab/mmdetection/blob/main/configs/retinanet/retinanet_r50-caffe_fpn_ms-1x_coco.py)        | 1x       | [36.5](https://github.com/facebookresearch/detectron2/blob/master/configs/COCO-Detection/retinanet_R_50_FPN_1x.yaml)                   | 37.0        | [model](https://download.openmmlab.com/mmdetection/v2.0/benchmark/retinanet_r50_caffe_fpn_mstrain_1x_coco/retinanet_r50_caffe_fpn_mstrain_1x_coco-586977a0.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/benchmark/retinanet_r50_caffe_fpn_mstrain_1x_coco/retinanet_r50_caffe_fpn_mstrain_1x_coco_20200430_014748.log.json)                     |

### 训练速度

训练速度使用 s/iter 来度量。结果越低越好。

| 模型         | Detectron2 | mmdetection |

### 推理速度

推理速度通过单张 GPU 下的 fps(img/s) 来度量，越高越好。
为了与 Detectron2 保持一致，我们所写的推理时间除去了数据加载时间。

### 推理速度

对于 Mask RCNN，我们去除了后处理中 RLE 编码的时间。
我们在括号中给出了官方给出的速度。由于硬件差异，官方给出的速度会比我们所测试得到的速度快一些。

| 模型         | Detectron2  | mmdetection |

### 训练内存

# EMANet

> [Expectation-Maximization Attention Networks for Semantic Segmentation](https://arxiv.org/abs/1907.13426)

<a href="https://xialipku.github.io/EMANet">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/decode_heads/ema_head.py#L80">Code Snippet</a>

Self-attention mechanism has been widely used for various tasks. It is designed to compute the representation of each position by a weighted sum of the features at all positions. Thus, it can capture long-range relations for computer vision tasks. However, it is computationally consuming. Since the attention maps are computed w.r.t all other positions. In this paper, we formulate the attention mechanism into an expectation-maximization manner and iteratively estimate a much more compact set of bases upon which the attention maps are computed. By a weighted summation upon these bases, the resulting representation is low-rank and deprecates noisy information from the input. The proposed Expectation-Maximization Attention (EMA) module is robust to the variance of input and is also friendly in memory and computation. Moreover, we set up the bases maintenance and normalization methods to stabilize its training procedure. We conduct extensive experiments on popular semantic segmentation benchmarks including PASCAL VOC, PASCAL Context and COCO Stuff, on which we set new records.

<img src="https://user-images.githubusercontent.com/24582831/142901186-7bfe15e2-805a-420e-81b0-74f214f20a36.png" width="80%"/>
</div>

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                           | download                                                                                                                                                                                                                                                                                                                                                 |

| EMANet | R-50-D8  | 512x1024  |   80000 |      5.4 | 4.58           | V100   | 77.59 | 79.44         | [config](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/emanet/eemanet_r50-d8_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/emanet/emanet_r50-d8_512x1024_80k_cityscapes/emanet_r50-d8_512x1024_80k_cityscapes_20200901_100301-c43fcef1.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/emanet/emanet_r50-d8_512x1024_80k_cityscapes/emanet_r50-d8_512x1024_80k_cityscapes-20200901_100301.log.json)     |

| EMANet | R-101-D8 | 512x1024  |   80000 |      6.2 | 2.87           | V100   | 79.10 | 81.21         | [config](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/emanet/emanet_r101-d8_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/emanet/emanet_r101-d8_512x1024_80k_cityscapes/emanet_r101-d8_512x1024_80k_cityscapes_20200901_100301-2d970745.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/emanet/emanet_r101-d8_512x1024_80k_cityscapes/emanet_r101-d8_512x1024_80k_cityscapes-20200901_100301.log.json) |

| EMANet | R-50-D8  | 769x769   |   80000 |      8.9 | 1.97           | V100   | 79.33 | 80.49         | [config](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/emanet/emanet_r50-d8_4xb2-80k_cityscapes-769x769.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/emanet/emanet_r50-d8_769x769_80k_cityscapes/emanet_r50-d8_769x769_80k_cityscapes_20200901_100301-16f8de52.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/emanet/emanet_r50-d8_769x769_80k_cityscapes/emanet_r50-d8_769x769_80k_cityscapes-20200901_100301.log.json)         |

| EMANet | R-101-D8 | 769x769   |   80000 |     10.1 | 1.22           | V100   | 79.62 | 81.00         | [config](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/emanet/emanet_r101-d8_4xb2-80k_cityscapes-769x769.py)  | [model](https://download.openmmlab.com/mmsegmentation/v0.5/emanet/emanet_r101-d8_769x769_80k_cityscapes/emanet_r101-d8_769x769_80k_cityscapes_20200901_100301-47a324ce.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/emanet/emanet_r101-d8_769x769_80k_cityscapes/emanet_r101-d8_769x769_80k_cityscapes-20200901_100301.log.json)     |

```bibtex
@inproceedings{li2019expectation,
  title={Expectation-maximization attention networks for semantic segmentation},
  author={Li, Xia and Zhong, Zhisheng and Wu, Jianlong and Yang, Yibo and Lin, Zhouchen and Liu, Hong},
  pages={9167--9176},
}
```

# Preparing UDM10 Dataset

```bibtex
}
```

The datasets can be downloaded from [here](https://drive.google.com/file/d/1G4V4KZZhhfzUlqHiSBBuWyqLyIOvOs0W/).

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── UDM10
|   |   ├── GT
|   |   ├── BIx4
|   |   ├── BDx4
```

# 操作概述

MMDeploy 提供了一系列工具，帮助您更轻松的将 OpenMMLab 下的算法部署到各种设备与平台上。

您可以使用我们设计的流程一“部”到位，也可以定制您自己的转换流程。

## 流程简介

MMDeploy 定义的模型部署流程，如下图所示：

### 模型转换（Model Converter）

模型转换的主要功能是把输入的模型格式，转换为目标设备的推理引擎所要求的模型格式。

目前，MMDeploy 可以把 PyTorch 模型转换为 ONNX、TorchScript 等和设备无关的 IR 模型。也可以将 ONNX 模型转换为推理后端模型。两者相结合，可实现端到端的模型转换，也就是从训练端到生产端的一键式部署。

### MMDeploy 模型（MMDeploy Model）

也称 SDK Model。它是模型转换结果的集合。不仅包括后端模型，还包括模型的元信息。这些信息将用于推理 SDK 中。

### 推理 SDK（Inference SDK）

封装了模型的前处理、网络推理和后处理过程。对外提供多语言的模型推理接口。

## 准备工作

对于端到端的模型转换和推理，MMDeploy 依赖 Python 3.6+ 以及 PyTorch 1.8+。

**第一步**：从[官网](https://docs.conda.io/en/latest/miniconda.html)下载并安装 Miniconda

**第二步**：创建并激活 conda 环境

```shell
```

**第三步**: 参考[官方文档](https://pytorch.org/get-started/locally/)并安装 PyTorch

在 GPU 环境下：

```shell

```shell
```

在 GPU 环境下：

```shell
```

在 CPU 环境下：

```shell
```

```{note}
在 GPU 环境下，请务必保证 {cudatoolkit_version} 和主机的 CUDA Toolkit 版本一致，避免在使用 TensorRT 时，可能引起的版本冲突问题。
```

**第一步**：通过 [MIM](https://github.com/open-mmlab/mim) 安装 [MMCV](https://github.com/open-mmlab/mmcv)

```shell
```

**第二步**: 安装 MMDeploy 和 推理引擎

我们推荐用户使用预编译包安装和体验 MMDeploy 功能。目前提供模型转换(trt/ort)以及 SDK 推理的 pypi 预编译包，SDK 的 c/cpp 库可从[这里](https://github.com/open-mmlab/mmdeploy/releases) 选择最新版本下载并安装。

目前，MMDeploy 的预编译包支持的平台和设备矩阵如下：

<table>
<thead>
  <tr>

<table>
<thead>
  <tr>
  </tr>
</thead>
<tbody>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>
  </tr>
  <tr>
  </tr>
</tbody>
</table>

**注：对于不在上述表格中的软硬件平台，请参考[源码安装文档](01-how-to-build/build_from_source.md)，正确安装和配置 MMDeploy。**

以最新的预编译包为例，你可以参考以下命令安装：

```shell
# 1. 安装 MMDeploy 模型转换工具（含trt/ort自定义算子）

# 2. 安装 MMDeploy SDK推理工具
# 根据是否需要GPU推理可任选其一进行下载安装
# 2.1 支持 onnxruntime 推理
# 2.2 支持 onnxruntime-gpu tensorrt 推理

# 3. 安装推理引擎
# 3.1 安装推理引擎 TensorRT

# 3. 安装推理引擎
# !!! 若要进行 TensorRT 模型的转换以及推理，从 NVIDIA 官网下载 TensorRT-8.2.3.0 CUDA 11.x 安装包并解压到当前目录。
# !!! 另外还需要从 NVIDIA 官网下载 cuDNN 8.2.1 CUDA 11.x 安装包并解压到当前目录

# 3.2 安装推理引擎 ONNX Runtime

```

请阅读 [这里](02-how-to-run/prebuilt_package_windows.md)，了解 MMDeploy 预编译包在 Windows 平台下的使用方法。

## 模型转换

在准备工作就绪后，我们可以使用 MMDeploy 中的工具 `tools/deploy.py`，将 OpenMMLab 的 PyTorch 模型转换成推理后端支持的格式。
对于`tools/deploy.py` 的使用细节，请参考 [如何转换模型](02-how-to-run/convert_model.md)。

以 [MMDetection](https://github.com/open-mmlab/mmdetection) 中的 `Faster R-CNN` 为例，我们可以使用如下命令，将 PyTorch 模型转换为 TenorRT 模型，从而部署到 NVIDIA GPU 上.

```shell

```shell
# 克隆 mmdeploy 仓库。转换时，需要使用 mmdeploy 仓库中的配置文件，建立转换流水线, `--recursive` 不是必须的

# 安装 mmdetection。转换时，需要使用 mmdetection 仓库中的模型配置文件，构建 PyTorch nn module
cd ..

# 下载 Faster R-CNN 模型权重

# 执行转换命令，实现端到端的转换

mmdetection/configs/faster_rcnn/faster-rcnn_r50_fpn_1x_coco.py \
```

转换结果被保存在 `--work-dir` 指向的文件夹中。**该文件夹中不仅包含推理后端模型，还包括推理元信息。这些内容的整体被定义为 SDK Model。推理 SDK 将用它进行模型推理。**

```{tip}
把上述转换命令中的detection_tensorrt_dynamic-320x320-1344x1344.py 换成 detection_onnxruntime_dynamic.py，并修改 --device 为 cpu，
即可以转出 onnx 模型，并用 ONNXRuntime 进行推理
```

## 模型推理

在转换完成后，你既可以使用 Model Converter 进行推理，也可以使用 Inference SDK。

### 使用 Model Converter 的推理 API

Model Converter 屏蔽了推理后端接口的差异，对其推理 API 进行了统一封装，接口名称为 `inference_model`。

以上文中 Faster R-CNN 的 TensorRT 模型为例，你可以使用如下方式进行模型推理工作：

```python

```

```{note}
接口中的 model_path 指的是推理引擎文件的路径，比如例子当中end2end.engine文件的路径。路径必须放在 list 中，因为有的推理引擎模型结构和权重是分开存储的。
```

### 使用推理 SDK

你可以直接运行预编译包中的 demo 程序，输入 SDK Model 和图像，进行推理，并查看推理结果。

```shell

# 运行 python demo
# 运行 C/C++ demo
# 根据文件夹内的 README.md 进行编译
```

```{note}

```

```{note}
以上述命令中，输入模型是 SDK Model 的路径（也就是 Model Converter 中 --work-dir 参数），而不是推理引擎文件的路径。
因为 SDK 不仅要获取推理引擎文件，还需要推理元信息（deploy.json, pipeline.json）。它们合在一起，构成 SDK Model，存储在 --work-dir 下
```

除了 demo 程序，预编译包还提供了 SDK 多语言接口。你可以根据自己的项目需求，选择合适的语言接口，
把 MMDeploy SDK 集成到自己的项目中，进行二次开发。

对于检测功能，你也可以参考如下代码，集成 MMDeploy SDK Python API 到自己的项目中：

```python
import cv2

# 读取图片

# 创建检测器
# 执行推理
# 使用阈值过滤推理结果，并绘制到原图中
      continue

      continue

```

更多示例，请查阅[这里](https://github.com/open-mmlab/mmdeploy/tree/main/demo/python)。

使用 C++ API 进行模型推理的流程符合下面的模式：
![image](https://user-images.githubusercontent.com/4560679/182554486-2bf0ff80-9e82-4a0f-bccc-5e1860444302.png)

以下是具体过程：

```C++

  // mmdeploy SDK model，以上文中转出的 faster r-cnn 模型为例

  // 1. 读取模型
  // 2. 创建预测器
  // 3. 读取图像

// 1. 读取模型
  // 2. 创建预测器
  // 3. 读取图像
  // 4. 应用预测器推理
  // 5. 处理推理结果: 此处我们选择可视化推理结果
      continue;
    }
  }
  return 0;
}
```

在您的项目CMakeLists中，增加：

```

}
  return 0;
}
```

```

编译时，使用 -DMMDeploy_DIR，传入MMDeloyConfig.cmake所在的路径。它在预编译包中的sdk/lib/cmake/MMDeloy下。
更多示例，请查阅[此处](https://github.com/open-mmlab/mmdeploy/tree/main/demo/csrc/cpp)。

对于 C API、C# API、Java API 的使用方法，请分别阅读代码[C demos](https://github.com/open-mmlab/mmdeploy/tree/main/demo/csrc/c)， [C# demos](https://github.com/open-mmlab/mmdeploy/tree/main/demo/csharp) 和 [Java demos](https://github.com/open-mmlab/mmdeploy/tree/main/demo/java)。
我们将在后续版本中详细讲述它们的用法。

#### 加速预处理（实验性功能）

若要对预处理进行加速，请查阅[此处](./02-how-to-run/fuse_transform.md)

## 模型精度评估

为了测试部署模型的精度，推理效率，我们提供了 `tools/test.py` 来帮助完成相关工作。以上文中的部署模型为例：

```bash
python mmdeploy/tools/test.py \
    mmdeploy/configs/detection/detection_tensorrt_dynamic-320x320-1344x1344.py \
    mmdetection/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py \

```bash
    --model mmdeploy_model/faster-rcnn/end2end.engine \
```

```{note}
关于 --model 选项，当使用 Model Converter 进行推理时，它代表转换后的推理后端模型的文件路径。而当使用 SDK 测试模型精度时，该选项表示 MMDeploy Model 的路径.
```

请阅读 [如何进行模型评估](02-how-to-run/profile_model.md) 了解关于 `tools/test.py` 的使用细节。

# Build lmdeploy on windows

- [CMake 3.17+](https://github.com/Kitware/CMake/releases)
- [Visual Studio 2019+](https://visualstudio.microsoft.com/downloads/)
- [CUDA Toolkit 11.8+](https://developer.nvidia.com/cuda-toolkit-archive)

cd build
..\builder\windows\generate.ps1
cd ..
rm build -Force -Recurse
python setup.py bdist_wheel -d build\wheel
```

# 基于视觉的 3D 检测

基于视觉的 3D 检测是指基于纯视觉输入的 3D 检测方法，例如基于单目、双目和多视图图像的 3D 检测。目前，我们只支持单目和多视图的 3D 检测方法。其他方法也应该与我们的框架兼容，并在将来得到支持。

它期望给定的模型以任意数量的图像作为输入，并为每一个感兴趣的目标预测 3D 框及类别标签。以 nuScenes 数据集 FCOS3D 为例，我们将展示如何准备数据，在标准的 3D 检测基准上训练并测试模型，以及可视化并验证结果。

## 数据准备

首先，我们需要下载原始数据并按照[数据准备文档](https://mmdetection3d.readthedocs.io/zh_CN/latest/data_preparation.html)中提供的标准方式重新组织数据。

由于不同数据集的原始数据有不同的组织方式，我们通常需要用 pkl 或 json 文件收集有用的数据信息。因此，在准备好所有的原始数据之后，我们需要运行 `create_data.py` 中提供的脚本来为不同的数据集生成数据信息。例如，对于 nuScenes，我们需要运行如下命令：

```
```

```
├── tools
├── data
│   │   ├── maps

│   │   ├── nuscenes_infos_train_mono3d.coco.json
│   │   ├── nuscenes_infos_trainval_mono3d.coco.json
│   │   ├── nuscenes_infos_val_mono3d.coco.json
│   │   ├── nuscenes_infos_test_mono3d.coco.json
```

注意，此处的 pkl 文件主要用于使用 LiDAR 数据的方法，json 文件用于 2D 检测/纯视觉的 3D 检测。在 v0.13.0 支持单目 3D 检测之前，json 文件只包含 2D 检测的信息，因此如果你需要最新的信息，请切换到 v0.13.0 之后的分支。

## 训练

接着，我们将使用提供的配置文件训练 FCOS3D。基本的脚本与其他模型一样。当你使用不同的 GPU 设置进行训练时，你基本上可以按照这个[教程](https://mmdetection3d.readthedocs.io/zh_CN/latest/1_exist_data_model.html#inference-with-existing-models)的示例。假设我们在一台具有 8 块 GPU 的机器上使用分布式训练：

```
./tools/dist_train.sh configs/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d.py 8
```

```
```

注意，配置文件名中的 `2x8` 是指训练时用了 8 块 GPU，每块 GPU 上有 2 个数据样本。如果你的自定义设置不同于此，那么有时候你需要相应的调整学习率。基本规则可以参考[此处](https://arxiv.org/abs/1706.02677)。

我们也可以通过运行以下命令微调 FCOS3D，从而达到更好的性能：

```
./tools/dist_train.sh fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_finetune.py 8
```

通过先前的脚本训练好一个基准模型后，请记得相应的修改[此处](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_finetune.py#L8)的路径。

## 定量评估

在训练期间，模型权重文件将会根据配置文件中的 `evaluation = dict(interval=xxx)` 设置被周期性地评估。

我们支持不同数据集的官方评估方案。由于输出格式与基于其他模态的 3D 检测相同，因此评估方法也是一样的。

对于 nuScenes，将使用基于距离的平均精度（mAP）以及 nuScenes 检测分数（NDS）分别对 10 个类别进行评估。评估结果将会被打印到终端中，如下所示：

```

```

```

此外，在训练完成后你也可以评估特定的模型权重文件。你可以简单地执行以下脚本：

```
./tools/dist_test.sh configs/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d.py \
    work_dirs/fcos3d/latest.pth --eval mAP
```

## 测试与提交

```

```
```

## 测试与提交

如果你只想在在线基准上进行推理或测试模型性能，你需要将之前评估脚本中的 `--eval mAP` 替换成 `--format-only`，并在需要的情况下指定 `jsonfile_prefix`，例如，添加选项 `--eval-options jsonfile_prefix=work_dirs/fcos3d/test_submission`。请确保配置文件中的[测试信息](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/_base_/datasets/nus-mono3d.py#L93)由验证集相应地改为测试集。

在生成结果后，你可以压缩文件夹并上传至 nuScenes 3D 检测挑战的 evalAI 评估服务器上。

## 定性评估

MMDetection3D 还提供了通用的可视化工具，以便于我们可以对训练好的模型预测的检测结果有一个直观的感受。你也可以在评估阶段通过设置 `--eval-options 'show=True' 'out_dir=${SHOW_DIR}'` 来在线可视化检测结果，或者使用 `tools/misc/visualize_results.py` 来离线地进行可视化。

此外，我们还提供了脚本 `tools/misc/browse_dataset.py` 用于可视化数据集而不做推理。更多的细节请参考[可视化文档](https://mmdetection3d.readthedocs.io/zh_CN/latest/useful_tools.html#visualization)。

注意，目前我们仅支持纯视觉方法在图像上的可视化。将来我们将集成在前景图以及鸟瞰图（BEV）中的可视化。

# CRNN

> [An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition](https://arxiv.org/abs/1507.05717)

Image-based sequence recognition has been a long-standing research topic in computer vision. In this paper, we investigate the problem of scene text recognition, which is among the most important and challenging tasks in image-based sequence recognition. A novel neural network architecture, which integrates feature extraction, sequence modeling and transcription into a unified framework, is proposed. Compared with previous systems for scene text recognition, the proposed architecture possesses four distinctive properties: (1) It is end-to-end trainable, in contrast to most of the existing algorithms whose components are separately trained and tuned. (2) It naturally handles sequences in arbitrary lengths, involving no character segmentation or horizontal scale normalization. (3) It is not confined to any predefined lexicon and achieves remarkable performances in both lexicon-free and lexicon-based scene text recognition tasks. (4) It generates an effective yet much smaller model, which is more practical for real-world application scenarios. The experiments on standard benchmarks, including the IIIT-5K, Street View Text and ICDAR datasets, demonstrate the superiority of the proposed algorithm over the prior arts. Moreover, the proposed algorithm performs well in the task of image-based music score recognition, which evidently verifies the generality of it.

<img src="https://user-images.githubusercontent.com/22607038/142797788-6b1cd78d-1dd6-4e02-be32-3dbd257c4992.png"/>
</div>

## Dataset

| trainset | instance_num | repeat_num | note  |
| :------: | :----------: | :--------: | :---: |
|  Syn90k  |   8919273    |     1      | synth |

|                          methods                           |        | Regular Text |           |     |           | Irregular Text |        |                                      download                                       |

| :--------------------------------------------------------: | :----: | :----------: | :-------: | :-: | :-------: | :------------: | :----: | :---------------------------------------------------------------------------------: |
|                          methods                           | IIIT5K |     SVT      | IC13-1015 |     | IC15-2077 |      SVTP      |  CT80  |                                                                                     |
|   [CRNN](/configs/textrecog/crnn/crnn_mini-vgg_5e_mj.py)   | 0.8053 |    0.7991    |  0.8739   |     |  0.5571   |     0.6093     | 0.5694 | [model](https://download.openmmlab.com/mmocr/textrecog/crnn/crnn_mini-vgg_5e_mj/crnn_mini-vgg_5e_mj_20220826_224120-8afbedbb.pth) \| [log](https://download.openmmlab.com/mmocr/textrecog/crnn/crnn_mini-vgg_5e_mj/20220826_224120.log) |

| [CRNN-TTA](/configs/textrecog/crnn/crnn_mini-vgg_5e_mj.py) | 0.8013 |    0.7975    |  0.8631   |     |  0.5763   |     0.6093     | 0.5764 | [model](https://download.openmmlab.com/mmocr/textrecog/crnn/crnn_mini-vgg_5e_mj/crnn_mini-vgg_5e_mj_20220826_224120-8afbedbb.pth) \| [log](https://download.openmmlab.com/mmocr/textrecog/crnn/crnn_mini-vgg_5e_mj/20220826_224120.log) |

```bibtex
@article{shi2016end,
  title={An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition},
  author={Shi, Baoguang and Bai, Xiang and Yao, Cong},
}
```

# MMRotate 模型部署

- [MMRotate 模型部署](#mmrotate-模型部署)
    - [安装 mmrotate](#安装-mmrotate)

[MMRotate](https://github.com/open-mmlab/mmrotate) 是一个基于 PyTorch 的旋转物体检测的开源工具箱，也是 [OpenMMLab](https://openmmlab.com/) 项目的一部分。

## 安装

### 安装 mmrotate

请参考[官网安装指南](https://mmrotate.readthedocs.io/zh_CN/latest/get_started.html)。

```shell

```shell
```

**说明**:

## 模型转换

你可以使用 [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/blob/main/tools/deploy.py) 把 mmrotate 模型一键式转换为推理后端模型。
该工具的详细使用说明请参考[这里](https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/02-how-to-run/convert_model.md#usage).

以下，我们将演示如何把 `rotated-faster-rcnn` 转换为 onnx 模型。

```shell

```shell

    --show \
```

转换的关键之一是使用正确的配置文件。项目中已内置了各后端部署[配置文件](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmrotate)。
文件的命名模式是：

```
```

其中：

文件的命名模式是：

```
```

其中：

在上例中，你也可以把 `rotated-faster-rcnn` 转为其他后端模型。比如使用`rotated-detection_tensorrt-fp16_dynamic-320x320-1024x1024.py`，把模型转为 tensorrt-fp16 模型。

```{tip}
```

## 模型规范

上例中的`mmdeploy_models/mmrotate/ort`，结构如下：

```
```

重要的是：

## 模型推理

```

重要的是：

## 模型推理

### 后端模型推理

```python

```

```python
import cv2

```

## 模型支持列表

```

## 模型支持列表

# Stable Diffusion (2022)

> [Stable Diffusion](https://github.com/CompVis/stable-diffusion)

Stable Diffusion is a latent diffusion model conditioned on the text embeddings of a CLIP text encoder, which allows you to create images from text inputs. This model builds upon the CVPR'22 work [High-Resolution Image Synthesis with Latent Diffusion Models](https://ommer-lab.com/research/latent-diffusion-models/). The official code was released at [stable-diffusion](https://github.com/CompVis/stable-diffusion) and also implemented at [diffusers](https://github.com/huggingface/diffusers). We support this algorithm here to facilitate the community to learn together and compare it with other text2image methods.

<thead>
  <tr>
    <td>

<thead>
  <tr>
    <td>
  <img src="https://user-images.githubusercontent.com/12782558/209609229-8221c7cc-d5c9-44d5-a1af-c254b5a95fae.png" width="400"/>
  <br/>
  <b>A mecha robot in a favela in expressionist style</b>
    <td>
  <img src="https://user-images.githubusercontent.com/12782558/210951970-a81e80c3-822e-4782-901e-db52e34b85a3.png" width="400"/>
  <br/>
  <b>A Chinese palace is beside a beautiful lake</b>
    <td>
  <img src="https://user-images.githubusercontent.com/12782558/210952108-df82e5ad-6eb6-4948-8d22-3802299d1131.png" width="400"/>
  <br/>
  <b>A panda is having dinner at KFC</b>
  </tr>
</thead>
</table>

|                                        Model                                         | Dataset | Download |

  </tr>
</thead>
</table>

| :----------------------------------------------------------------------------------: | :-----: | :------: |
|          [stable_diffusion_v1.5](./stable-diffusion_ddim_denoisingunet.py)           |    -    |    -     |
| [stable_diffusion_v1.5_tomesd](./stable-diffusion_ddim_denoisingunet-tomesd_5e-1.py) |    -    |    -     |

We use stable diffusion v1.5 weights. This model has several weights including vae, unet and clip.

You may download the weights from [stable-diffusion-1.5](https://huggingface.co/runwayml/stable-diffusion-v1-5) and change the 'from_pretrained' in config to the weights dir.

Download with git:

```shell
git clone https://huggingface.co/runwayml/stable-diffusion-v1-5
```

```python

```shell
```

```python
from mmengine import MODELS, Config
from torchvision import utils

config = 'configs/stable_diffusion/stable-diffusion_ddim_denoisingunet.py'
config = Config.fromfile(config).copy()
# change the 'pretrained_model_path' if you have downloaded the weights manually
# config.model.unet.from_pretrained = '/path/to/your/stable-diffusion-v1-5'
# config.model.vae.from_pretrained = '/path/to/your/stable-diffusion-v1-5'

StableDiffuser = MODELS.build(config.model)
prompt = 'A mecha robot in a favela in expressionist style'
StableDiffuser = StableDiffuser.to('cuda')

image = StableDiffuser.infer(prompt)['samples'][0]
image.save('robot.png')
```

## Use ToMe to accelerate your stable diffusion model

```

We support **[tomesd](https://github.com/dbolya/tomesd)** now! It is developed based on [ToMe](https://github.com/facebookresearch/ToMe), an efficient ViT speed-up tool based on token merging. To work on with **tomesd** in `mmagic`, you just need to add `tomesd_cfg` to `model` as shown in [stable_diffusion_v1.5_tomesd](stable-diffusion_ddim_denoisingunet-tomesd_5e-1.py). The only requirement is `torch >= 1.12.1` in order to properly support `torch.Tensor.scatter_reduce()` functionality. Please do check it before running the demo.

```python
...
    type='StableDiffusion',
    unet=unet,
    vae=vae,
    enable_xformers=False,
    text_encoder=dict(
        type='ClipWrapper',
        clip_type='huggingface',
        pretrained_model_name_or_path=stable_diffusion_v15_url,

vae=vae,
        subfolder='text_encoder'),
    tokenizer=stable_diffusion_v15_url,
    scheduler=diffusion_scheduler,
    test_scheduler=diffusion_scheduler,
    tomesd_cfg=dict(
        ratio=0.5))
```

The detailed settings for `tomesd_cfg` are as follows:

- `ratio (float)`: The ratio of tokens to merge. For example, 0.4 would reduce the total number of tokens by 40%.The maximum value for this is 1-(1/(`sx` * `sy`)). **By default, the max ratio is 0.75, usually \<= 0.5 is recommended.** Higher values result in more speed-up, but with more visual quality loss.
- `max_downsample (int)`: Apply ToMe to layers with at most this amount of downsampling. E.g., 1 only applies to layers with no downsampling, while 8 applies to all layers. Should be chosen from 1, 2, 4, 8. **1, 2 are recommended.**

- `sx, sy (int, int)`: The stride for computing dst sets. A higher stride means you can merge more tokens, **default setting of (2, 2) works well in most cases**. `sx` and `sy` do not need to divide image size.
- `use_rand (bool)`: Whether or not to allow random perturbations when computing dst sets. By default: True, but if you're having weird artifacts you can try turning this off.
- `merge_attn (bool)`: Whether or not to merge tokens for attention **(recommended)**.
- `merge_crossattn (bool)`: Whether or not to merge tokens for cross attention **(not recommended)**.
- `merge_mlp (bool)`: Whether or not to merge tokens for the mlp layers **(especially not recommended)**.

For more details about the **tomesd** setting, please refer to [Token Merging for Stable Diffusion](https://arxiv.org/abs/2303.17604).

Then following the code below, you can evaluate the speed-up performance on stable diffusion models or stable-diffusion-based models ([DreamBooth](../dreambooth/README.md), [ControlNet](../controlnet/README.md)).

```python
import time

_device = 0
work_dir = '/path/to/your/work_dir'
config = 'configs/stable_diffusion/stable-diffusion_ddim_denoisingunet-tomesd_5e-1.py'
# # change the 'pretrained_model_path' if you have downloaded the weights manually

# w/o tomesd
config.model.tomesd_cfg = None

StableDiffuser = MODELS.build(config.model).to(f'cuda:{_device}')

# inference time evaluation params
size = 512
ratios = [0.5, 0.75]
samples_perprompt = 5

t = time.time()
for i in range(100//samples_perprompt):
    image = StableDiffuser.infer(prompt, height=size, width=size, num_images_per_prompt=samples_perprompt)['samples'][0]
    if i == 0:
        image.save(f"{work_dir}/wo_tomesd.png")
print(f"Generating 100 images with {samples_perprompt} images per prompt, without ToMe speed-up, time used : {time.time() - t}s")

for ratio in ratios:
    # w/ tomesd
    config.model.tomesd_cfg = dict(ratio=ratio)
    sd_model = MODELS.build(config.model).to(f'cuda:{_device}')

    t = time.time()
    for i in range(100//samples_perprompt):

# w/ tomesd

        image = sd_model.infer(prompt, height=size, width=size, num_images_per_prompt=samples_perprompt)['samples'][0]
        if i == 0:
            image.save(f"{work_dir}/w_tomesd_ratio_{ratio}.png")

    print(f"Generating 100 images with {samples_perprompt} images per prompt, merging ratio {ratio}, time used : {time.time() - t}s")
```

print(f"Generating 100 images with {samples_perprompt} images per prompt, merging ratio {ratio}, time used : {time.time() - t}s")
```

Here are some inference performance comparisons running on **single RTX 3090** with `torch 2.0.0+cu118` as backends. The results are reasonable, when enabling `xformers`, the speed-up ratio is a little bit lower. But `tomesd` still effectively reduces the inference time. It is especially recommended that enable `tomesd` when the `image_size` and `num_images_per_prompt` are large, since the number of similar tokens are larger and `tomesd` can achieve better performance.

|                                   Model                                    | Dataset | Download | xformer |            Ratio            | Size / Num images per prompt |                     Time (s)                      |

| :------------------------------------------------------------------------: | :-----: | :------: | :-----: | :-------------------------: | :--------------------------: | :-----------------------------------------------: |
| [stable_diffusion_v1.5-tomesd](./stable-diffusion_ddim_denoisingunet-tomesd_5e-1.py) |    -    |    -     |   w/o   | w/o tome <br> 0.5 <br> 0.75 |         512  /    5          | 542.20 <br> 427.65 (↓21.1%) <br>  393.05 (↓27.5%) |
| [stable_diffusion_v1.5-tomesd](./stable-diffusion_ddim_denoisingunet-tomesd_5e-1.py) |    -    |    -     |   w/    | w/o tome <br> 0.5 <br> 0.75 |         512  /    5          | 541.64 <br> 428.53 (↓20.9%) <br>  396.38 (↓26.8%) |

<thead>
  <tr>
    <td>
  <img src="https://user-images.githubusercontent.com/49406546/234613951-43b94470-89ff-4edc-a2e2-bea9e7a8a566.png" width="400"/>
  <br/>
  <b> w/o ToMe </b>
    <td>

<img src="https://user-images.githubusercontent.com/49406546/234613951-43b94470-89ff-4edc-a2e2-bea9e7a8a566.png" width="400"/>
  <br/>
    <td>
  <img src="https://user-images.githubusercontent.com/49406546/234613969-213e3436-b73c-4b8e-82ce-b91492b44db3.png" width="400"/>
  <br/>
  <b> w/ ToMe Speed-up (token merge ratio=0.5) </b>
    <td>
  <img src="https://user-images.githubusercontent.com/49406546/234613983-82fee9a3-05f7-4b1d-85dc-a507e85ecb31.png" width="400"/>
  <br/>
  <b> w/ ToMe Speed-up (token merge ratio=0.75) </b>
  </tr>
</thead>
</table>

Our codebase for the stable diffusion models builds heavily on [diffusers codebase](https://github.com/huggingface/diffusers) and the model weights are from [stable-diffusion-1.5](https://huggingface.co/runwayml/stable-diffusion-v1-5).

```bibtex

```bibtex
@misc{rombach2021highresolution,
      title={High-Resolution Image Synthesis with Latent Diffusion Models},
      author={Robin Rombach and Andreas Blattmann and Dominik Lorenz and Patrick Esser and Björn Ommer},
      eprint={2112.10752},
}

@article{bolya2023tomesd,
  title={Token Merging for Fast Stable Diffusion},
  author={Bolya, Daniel and Hoffman, Judy},
}

@inproceedings{bolya2023tome,
  title={Token Merging: Your {ViT} but Faster},
  author={Bolya, Daniel and Fu, Cheng-Yang and Dai, Xiaoliang and Zhang, Peizhao and Feichtenhofer, Christoph and Hoffman, Judy},
}
```

<details>

```bibtex
}
```

<details>

```bibtex

| [pose_resnetv1d_50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnetv1d50_8xb64-210e_coco-256x192.py) |  256x192   | 0.722 |      0.897      |      0.796      | 0.777 |      0.936      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnetv1d50_8xb64-210e_coco-256x192-27545d63_20221020.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnetv1d50_8xb64-210e_coco-256x192_20221020.log) |
| [pose_resnetv1d_50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnetv1d50_8xb64-210e_coco-384x288.py) |  384x288   | 0.730 |      0.899      |      0.800      | 0.782 |      0.935      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnetv1d50_8xb64-210e_coco-384x288-0646b46e_20221020.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnetv1d50_8xb64-210e_coco-384x288_20221020.log) |

| [pose_resnetv1d_101](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnetv1d101_8xb64-210e_coco-256x192.py) |  256x192   | 0.732 |      0.901      |      0.808      | 0.785 |      0.940      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnetv1d101_8xb64-210e_coco-256x192-ee9e7212_20221021.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnetv1d101_8xb64-210e_coco-256x192_20221021.log) |
| [pose_resnetv1d_101](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnetv1d101_8xb32-210e_coco-384x288.py) |  384x288   | 0.748 |      0.906      |      0.817      | 0.798 |      0.941      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnetv1d101_8xb32-210e_coco-384x288-d0b5875f_20221028.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnetv1d101_8xb32-210e_coco-384x288_20221028.log) |

| [pose_resnetv1d_152](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnetv1d152_8xb32-210e_coco-256x192.py) |  256x192   | 0.737 |      0.904      |      0.814      | 0.790 |      0.940      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnetv1d152_8xb32-210e_coco-256x192-fd49f947_20221021.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnetv1d152_8xb32-210e_coco-256x192_20221021.log) |
| [pose_resnetv1d_152](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnetv1d152_8xb48-210e_coco-384x288.py) |  384x288   | 0.751 |      0.907      |      0.821      | 0.801 |      0.942      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnetv1d152_8xb48-210e_coco-384x288-b9a99602_20221022.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_resnetv1d152_8xb48-210e_coco-384x288_20221022.log) |

# 安装

## 前置条件

MMAction2 适用于 Linux、Windows 和 MacOS。它需要 Python 3.7+，CUDA 10.2+ 和 PyTorch 1.8+。

```{note}
如果您熟悉 PyTorch 并且已经安装了它，可以跳过这部分内容，直接转到[下一节](#installation)。否则，您可以按照以下步骤进行准备工作。
```

**第一步。** 从[官方网站](https://docs.conda.io/en/latest/miniconda.html)下载并安装 Miniconda。

**第二步。** 创建一个 conda 环境并激活它。

```shell
```

**第三步。** 安装 PyTorch，按照[官方说明](https://pytorch.org/get-started/locally/)进行操作，例如：

在 GPU 平台上：

```shell
```

此命令将自动安装最新版本的 PyTorch 和 cudatoolkit，请确保它们与您的环境匹配。
```

在 CPU 平台上：

```shell
```

## 最佳实践

我们建议用户遵循我们的最佳实践来安装 MMAction2。然而，整个过程是高度可定制的。更多信息请参见[自定义安装](#customize-installation)部分。

```

在 CPU 平台上：

```shell
```

## 最佳实践

**第一步。** 使用 [MIM](https://github.com/open-mmlab/mim) 安装 [MMEngine](https://github.com/open-mmlab/mmengine)、[MMCV](https://github.com/open-mmlab/mmcv)、[MMDetection](https://github.com/open-mmlab/mmdetection)（可选）和 [MMPose](https://github.com/open-mmlab/mmpose)（可选）。

```shell
mim install mmpose
```

**第二步。** 安装 MMAction2。

根据您的需求，我们支持两种安装模式：

- [从源代码构建 MMAction2（推荐）](#build-mmaction2-from-source)：您想在 MMAction2 框架上开发自己的动作识别任务或新功能。例如，添加新的数据集或新的模型。因此，您可以使用我们提供的所有工具。
- [安装为 Python 包](#install-as-a-python-package)：您只想在项目中调用 MMAction2 的 API 或导入 MMAction2 的模块。

### 从源代码构建 MMAction2

在这种情况下，从源代码安装 mmaction2：

```shell

```shell
# "-e" 表示以可编辑形式安装，这样可以在不重新安装的情况下，让本地修改直接生效。
```

可选地，如果您希望为 MMAction2 做出贡献或体验实验功能，请切换到 `dev-1.x` 分支：

```shell
```

### 安装为 Python 包

只需使用 pip 安装即可。

```shell
pip install mmaction2
```

## 验证安装

为了验证 MMAction2 是否安装正确，我们提供了一些示例代码来运行推理演示。

**第一步。** 下载配置文件和权重文件。

```shell
mim download mmaction2 --config tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb --dest .
```

**第二步。** 验证推理演示。

选项（a）。如果您是从源代码安装的 mmaction2，可以运行以下命令：

```shell
# demo.mp4 和 label_map_k400.txt 都来自于 Kinetics-400
python demo/demo.py tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb.py \
    tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb_20220906-2692d16c.pth \
    demo/demo.mp4 tools/data/kinetics/label_map_k400.txt
```

您将在终端看到前5个标签及其对应的分数。

tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb_20220906-2692d16c.pth \
```

选项（b）。如果您将 mmaction2 安装为一个 Python 包，可以在 Python 解释器中运行以下代码，这将进行类似的验证：

```python
from operator import itemgetter
from mmaction.apis import init_recognizer, inference_recognizer

config_file = 'tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb.py'
checkpoint_file = 'tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb_20220906-2692d16c.pth'
video_file = 'demo/demo.mp4'
label_file = 'tools/data/kinetics/label_map_k400.txt'
model = init_recognizer(config_file, checkpoint_file, device='cpu')  # or device='cuda:0'
pred_result = inference_recognizer(model, video_file)

pred_scores = pred_result.pred_scores.item.tolist()
score_tuples = tuple(zip(range(len(pred_scores)), pred_scores))
score_sorted = sorted(score_tuples, key=itemgetter(1), reverse=True)
top5_label = score_sorted[:5]

labels = open(label_file).readlines()
labels = [x.strip() for x in labels]
results = [(labels[k[0]], k[1]) for k in top5_label]

print('The top-5 labels with corresponding scores are:')
for result in results:
    print(f'{result[0]}: ', result[1])
```

## 自定义安装

### CUDA 版本

在安装 PyTorch 时，您可能需要指定 CUDA 的版本。如果您不确定选择哪个版本，请遵循我们的建议：

请确保 GPU 驱动程序满足最低版本要求。有关更多信息，请参见[此表格](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cuda-major-component-versions__table-cuda-toolkit-driver-versions)。

```{note}

```{note}
如果按照我们的最佳实践进行安装，仅安装 CUDA 运行时库就足够了，因为不会在本地编译任何 CUDA 代码。然而，如果您希望从源代码编译 MMCV 或开发其他 CUDA 运算符，您需要从 NVIDIA 的[网站](https://developer.nvidia.com/cuda-downloads)安装完整的 CUDA 工具包，并且其版本应与 PyTorch 的 CUDA 版本匹配，即 `conda install` 命令中指定的 cudatoolkit 的版本。
```

MMCV 包含 C++ 和 CUDA 扩展，因此它与 PyTorch 的关系比较复杂。MIM 可以自动解决这些依赖关系，使安装变得更加容易。但这不是必须的。

如果您希望使用 pip 而不是 MIM 安装 MMCV，请参考[MMCV 安装指南](https://mmcv.readthedocs.io/en/latest/get_started/installation.html)。这需要手动指定基于 PyTorch 版本和其 CUDA 版本的 find-url。

例如，以下命令安装了为 PyTorch 1.10.x 和 CUDA 11.3 构建的 mmcv。

```shell
pip install mmcv -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10/index.html
```

MMAction2 可以仅在 CPU 环境中安装。在 CPU 模式下，你可以完成训练、测试和模型推理等所有操作。

在 CPU 模式下，MMCV 的部分功能将不可用，通常是一些 GPU 编译的算子。不过不用担心， MMAction2 中几乎所有的模型都不会依赖这些算子。

```

### 通过 Docker 使用 MMAction2

我们提供了一个[Dockerfile](https://github.com/open-mmlab/mmaction2/blob/main/docker/Dockerfile)来构建镜像。确保您的[docker 版本](https://docs.docker.com/engine/install/) >=19.03。

```shell
# 构建一个基于 PyTorch 1.6.0、CUDA 10.1 和 CUDNN 7 的镜像。
# 如果您喜欢其他版本，请修改 Dockerfile。
docker build -f ./docker/Dockerfile --rm -t mmaction2 .
```

使用以下命令运行它：

```shell
# 例如构建PyTorch 1.6.0, CUDA 10.1, CUDNN 7的镜像
# 如果你喜欢其他版本,只要修改Dockerfile
docker run --gpus all --shm-size=8g -it -v {DATA_DIR}:/mmaction2/data mmaction2
```

## 故障排除

1. 当从旧版本 `0.x` 迁移到新版本 `1.x` 时，您可能会遇到依赖库版本不匹配的问题。下面是在按照上述安装过程执行后，通过 `pip list` 命令显示的每个依赖库的版本。请确保在终端中显示的每个依赖库版本都大于或等于（即 `>=`）下面每个依赖库的版本。

```shell
mmaction2                1.0.0
mmcv                     2.0.0
mmdet                    3.0.0
mmengine                 0.7.2
mmpose                   1.0.0
```

## natural assembly
* no register dependency, no penalty
```
ld1     {v0.4s}, [r0], #16
fmla    v10.4s, v16.4s, v24.s[0]
fmla    v11.4s, v16.4s, v24.s[1]
fmla    v12.4s, v16.4s, v24.s[2]
fmla    v13.4s, v16.4s, v24.s[3]
```

## A53
* 128bit vector load cannot be dual issued with fmla, wait 2 cycles
* 64bit vector load cannot be dual issued with fmla, wait 1 cycle
* 64bit integer load can be dual issued with fmla, no penalty
* pointer update can be dual issued with fmla, no penalty
* 64bit vector load and 64bit vector insert can be dual issued, no penalty
* any vector load cannot be issued on the 4th cycle of each fmla (enters the accumulator pipeline)

### practical guide
* use 64bit vector load only
* issue vector load every three fmla
* 1 cycle to load 64bit, dual issue with the previous interleaved 64bit insert
* load the remaining 64bit into integer register, dual issue with fmla
* update pointer, dual issue with fmla

* insert 64bit into vector from integer register, dual issue with the next interleaved 64bit load
* add nop every three fmla if no load, seems to be faster
```
ldr     d0, [r0] // 1 cycle, v0 first 64bit
fmla
ldr     x23, [r0, #8] // 0 cycle, v0 second 64bit to temp register
fmla
add     r0, r0, #16 // 0 cycle, update pointer
fmla
ldr     d1, [r0] // 1 cycle, v1 first 64bit
ins     v0.d[1], x23 // 0 cycle, v0 second 64bit complete
fmla
ldr     x23, [r0, #8] // 0 cycle, v1 second 64bit to temp register
fmla
fmla
ins     v1.d[1], x23 // 1 cycle, v1 second 64bit complete
nop
fmla
fmla
fmla
nop
nop
fmla
fmla
fmla
```

## A55
* 64bit vector load can be dual issued with fmla, no penalty

nop
fmla
fmla
fmla
nop
nop
fmla
fmla
fmla
```

## A55
* 64bit vector insert can be dual issued with fmla, no penalty

* load 64bit, dual issue with fmla
* insert 64bit into vector from integer register, dual issue with fmla
* interleaved load loose register dependency
* nop trick is not needed
```
ldr     d0, [r0] // 0 cycle, v0 first 64bit
fmla
fmla
fmla
ldr     d1, [r0] // 0 cycle, v1 first 64bit
fmla
fmla

fmla
fmla
fmla
fmla
fmla
fmla
ins     v1.d[1], x23 // 0 cycle, v1 second 64bit complete
fmla
```

# 开始你的第一步

## 依赖

本节中，我们将演示如何用 PyTorch 准备一个环境。

MMDetection 支持在 Linux，Windows 和 macOS 上运行。它需要 Python 3.7 以上，CUDA 9.2 以上和 PyTorch 1.8 及其以上。

```{note}
如果你对 PyTorch 有经验并且已经安装了它，你可以直接跳转到[下一小节](#安装流程)。否则，你可以按照下述步骤进行准备。
```

```shell
```

**步骤 2.** 基于 [PyTorch 官方说明](https://pytorch.org/get-started/locally/)安装 PyTorch。

在 GPU 平台上：

```shell
```

在 CPU 平台上：

```shell
```

## 安装流程

我们推荐用户参照我们的最佳实践安装 MMDetection。不过，整个过程也是可定制化的，更多信息请参考[自定义安装](#自定义安装)章节。

### 最佳实践

**步骤 0.** 使用 [MIM](https://github.com/open-mmlab/mim) 安装 [MMEngine](https://github.com/open-mmlab/mmengine) 和 [MMCV](https://github.com/open-mmlab/mmcv)。

```shell
```

```shell
```

**注意：** 在 MMCV-v2.x 中，`mmcv-full` 改名为 `mmcv`，如果你想安装不包含 CUDA 算子精简版，可以通过 `mim install "mmcv-lite>=2.0.0rc1"` 来安装。

**步骤 1.** 安装 MMDetection。

方案 a：如果你开发并直接运行 mmdet，从源码安装它：

```shell
```

方案 b：如果你将 mmdet 作为依赖或第三方 Python 包，使用 MIM 安装：

```shell
```

## 验证安装

为了验证 MMDetection 是否安装正确，我们提供了一些示例代码来执行模型推理。

```shell
```

下载将需要几秒钟或更长时间，这取决于你的网络环境。完成后，你会在当前文件夹中发现两个文件 `rtmdet_tiny_8xb32-300e_coco.py` 和 `rtmdet_tiny_8xb32-300e_coco_20220902_112414-78e30dcc.pth`。

方案 a：如果你通过源码安装的 MMDetection，那么直接运行以下命令进行验证：

```shell

```shell
```

你会在当前文件夹中的 `outputs/vis` 文件夹中看到一个新的图像 `demo.jpg`，图像中包含有网络预测的检测框。

方案 b：如果你通过 MIM 安装的 MMDetection，那么可以打开你的 Python 解析器，复制并粘贴以下代码：

```python

```

你将会看到一个包含 `DetDataSample` 的列表，预测结果在 `pred_instance` 里，包含有检测框，类别和得分。

### 自定义安装

在安装 PyTorch 时，你需要指定 CUDA 的版本。如果你不清楚应该选择哪一个，请遵循我们的建议：

请确保你的 GPU 驱动版本满足最低的版本需求，参阅 NVIDIA 官方的 [CUDA 工具箱和相应的驱动版本关系表](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cuda-major-component-versions__table-cuda-toolkit-driver-versions)。

```{note}
如果按照我们的最佳实践，安装 CUDA 运行时库就足够了，这是因为不需要在本地编译 CUDA 代码。但如果你希望从源码编译 MMCV，或是开发其他 CUDA 算子，那么就必须安装完整的 CUDA 工具链，参见 [NVIDIA 官网](https://developer.nvidia.com/cuda-downloads)，另外还需要确保该 CUDA 工具链的版本与 PyTorch 安装时的配置相匹配（如用 `conda install` 安装 PyTorch 时指定的 cudatoolkit 版本）。
```

#### 不使用 MIM 安装 MMEngine

要使用 pip 而不是 MIM 来安装 MMEngine，请遵照 [MMEngine 安装指南](https://mmengine.readthedocs.io/zh_CN/latest/get_started/installation.html)。

例如，你可以通过以下命令安装 MMEngine。

```shell
```

MMCV 包含 C++ 和 CUDA 扩展，因此其对 PyTorch 的依赖比较复杂。MIM 会自动解析这些依赖，选择合适的 MMCV 预编译包，使安装更简单，但它并不是必需的。

```shell
```

要使用 pip 而不是 MIM 来安装 MMCV，请遵照 [MMCV 安装指南](https://mmcv.readthedocs.io/zh_CN/2.x/get_started/installation.html)。它需要您用指定 url 的形式手动指定对应的 PyTorch 和 CUDA 版本。

例如，下述命令将会安装基于 PyTorch 1.12.x 和 CUDA 11.6 编译的 MMCV。

```shell
```

#### 在 CPU 环境中安装

MMDetection 可以在 CPU 环境中构建。在 CPU 模式下，可以进行模型训练（需要 MMCV 版本 >= 2.0.0rc1）、测试或者推理。

但是，以下功能在该模式下不能使用：

- CARAFE
- nms_cuda

因此，如果尝试训练/测试/推理包含上述算子的模型，将会报错。下表列出了将会受影响的相关算法。

- CARAFE
- nms_cuda

|                          操作                           |                                           模型                                           |

[Google Colab](https://colab.research.google.com/) 通常已经包含了 PyTorch 环境，因此我们只需要安装 MMEngine，MMCV 和 MMDetection 即可，命令如下：

**步骤 1.** 使用 [MIM](https://github.com/open-mmlab/mim) 安装 [MMEngine](https://github.com/open-mmlab/mmengine) 和 [MMCV](https://github.com/open-mmlab/mmcv)。

```shell
```

**步骤 2.** 使用源码安装 MMDetection。

```shell
```

```python
# 预期输出：3.0.0 或其他版本号
```

```{note}

```

```python
```

```{note}
```

#### 通过 Docker 使用 MMDetection

我们提供了一个 [Dockerfile](../../docker/Dockerfile) 来构建一个镜像。请确保你的 [docker 版本](https://docs.docker.com/engine/install/) >=19.03。

```shell
# 如果你想要其他版本，只需要修改 Dockerfile
```

```shell
```

### 排除故障

如果你在安装过程中遇到一些问题，请先查看 [FAQ](notes/faq.md) 页面。如果没有找到解决方案，你也可以在 GitHub 上[提出一个问题](https://github.com/open-mmlab/mmdetection/issues/new/choose)。

### 使用多个 MMDetection 版本进行开发

训练和测试的脚本已经在 `PYTHONPATH` 中进行了修改，以确保脚本使用当前目录中的 MMDetection。

要使环境中安装默认版本的 MMDetection 而不是当前正在使用的，可以删除出现在相关脚本中的代码：

```shell
```

# 将单阶段检测器作为 RPN

候选区域网络 (Region Proposal Network, RPN) 作为 [Faster R-CNN](https://arxiv.org/abs/1506.01497) 的一个子模块，将为 Faster R-CNN 的第二阶段产生候选区域。在 MMDetection 里大多数的二阶段检测器使用 [`RPNHead`](../../../mmdet/models/dense_heads/rpn_head.py)作为候选区域网络来产生候选区域。然而，任何的单阶段检测器都可以作为候选区域网络，是因为他们对边界框的预测可以被视为是一种候选区域，并且因此能够在 R-CNN 中得到改进。因此在 MMDetection v3.0 中会支持将单阶段检测器作为 RPN 使用。

接下来我们通过一个例子，即如何在 [Faster R-CNN](../../../configs/faster_rcnn/faster-rcnn_r50_fpn_fcos-rpn_1x_coco.py) 中使用一个无锚框的单阶段的检测器模型 [FCOS](../../../configs/fcos/fcos_r50-caffe_fpn_gn-head_1x_coco.py) 作为 RPN ，详细阐述具体的全部流程。

主要流程如下:

1. 在 Faster R-CNN 中使用 `FCOSHead` 作为 `RPNHead`
2. 评估候选区域
3. 用预先训练的 FCOS 训练定制的 Faster R-CNN

## 在 Faster R-CNN 中使用 `FCOSHead` 作为` RPNHead`

主要流程如下:

2. 评估候选区域

为了在 Faster R-CNN 中使用 `FCOSHead` 作为 `RPNHead` ，我们应该创建一个名为 `configs/faster_rcnn/faster-rcnn_r50_fpn_fcos-rpn_1x_coco.py` 的配置文件，并且在 `configs/faster_rcnn/faster-rcnn_r50_fpn_fcos-rpn_1x_coco.py` 中将 `rpn_head` 的设置替换为 `bbox_head` 的设置，此外我们仍然使用 FCOS 的瓶颈设置，步幅为`[8,16,32,64,128]`，并且更新 `bbox_roi_extractor` 的 `featmap_stride` 为 ` [8,16,32,64,128]`。为了避免损失变慢，我们在前1000次迭代而不是前500次迭代中应用预热，这意味着 lr 增长得更慢。相关配置如下:

```python
_base_ = [
]
    # 从 configs/fcos/fcos_r50-caffe_fpn_gn-head_1x_coco.py 复制
        add_extra_convs='on_output',  # 使用 P5
        _delete_=True,  # 忽略未使用的旧设置

        num_classes=1,  # 对于 rpn, num_classes = 1，如果 num_classes > 1，它将在 TwoStageDetector 中自动设置为1
    roi_head=dict(  # featmap_strides 的更新取决于于颈部的步伐
# 学习率
    dict(

bbox_roi_extractor=dict(featmap_strides=[8, 16, 32, 64, 128])))
# 学习率
    dict(
        end=1000),  # 慢慢增加 lr，否则损失变成 NAN
    dict(
        begin=0,
        end=12,
]
```

然后，我们可以使用下面的命令来训练我们的定制模型。更多训练命令，请参考[这里](train.md)。

```python
# 使用8个 GPU 进行训练
bash
tools/dist_train.sh
configs/faster_rcnn/faster-rcnn_r50_fpn_fcos-rpn_1x_coco.py
--work-dir  /work_dirs/faster-rcnn_r50_fpn_fcos-rpn_1x_coco
```

## 评估候选区域

候选区域的质量对检测器的性能有重要影响，因此，我们也提供了一种评估候选区域的方法。和上面一样创建一个新的名为 `configs/rpn/fcos-rpn_r50_fpn_1x_coco.py` 的配置文件，并且在 `configs/rpn/fcos-rpn_r50_fpn_1x_coco.py` 中将 `rpn_head` 的设置替换为 `bbox_head` 的设置。

```python
_base_ = [
]

```python
_base_ = [
]
        num_classes=1,  # 对于 rpn, num_classes = 1，如果 num_classes >为1，它将在 rpn 中自动设置为1

```

假设我们在训练之后有检查点 `./work_dirs/faster-rcnn_r50_fpn_fcos-rpn_1x_coco/epoch_12.pth` ，然后，我们可以使用下面的命令来评估建议的质量。

```python
# 使用8个 GPU 进行测试
bash
tools/dist_test.sh
configs/rpn/fcos-rpn_r50_fpn_1x_coco.py
--work_dirs /faster-rcnn_r50_fpn_fcos-rpn_1x_coco/epoch_12.pth
```

## 用预先训练的 FCOS 训练定制的 Faster R-CNN

```python
bash
```

预训练不仅加快了训练的收敛速度，而且提高了检测器的性能。因此，我们在这里给出一个例子来说明如何使用预先训练的 FCOS 作为 RPN 来加速训练和提高精度。假设我们想在 Faster R-CNN 中使用 `FCOSHead` 作为 `rpn_head`，并加载预先训练权重来进行训练 [`fcos_r50-caffe_fpn_gn-head_1x_coco`](https://download.openmmlab.com/mmdetection/v2.0/fcos/fcos_r50_caffe_fpn_gn-head_1x_coco/fcos_r50_caffe_fpn_gn-head_1x_coco-821213aa.pth)。 配置文件 `configs/faster_rcnn/faster-rcnn_r50-caffe_fpn_fcos- rpn_1x_copy .py` 的内容如下所示。注意，`fcos_r50-caffe_fpn_gn-head_1x_coco` 使用 ResNet50 的 caffe 版本，因此需要更新 `data_preprocessor` 中的像素平均值和 std。

```python
_base_ = [
]

]

strides=[8, 16, 32, 64, 128],
```

训练命令如下。

```python
bash
configs/faster_rcnn/faster-rcnn_r50-caffe_fpn_fcos-rpn_1x_coco.py  \
--work-dir /work_dirs/faster-rcnn_r50-caffe_fpn_fcos-rpn_1x_coco
```

# PyTorch 2.0 Compatibility and Benchmarks

MMPose 1.0.0 is now compatible with PyTorch 2.0, ensuring that users can leverage the latest features and performance improvements offered by the PyTorch 2.0 framework when using MMPose. With the integration of inductor, users can expect faster model speeds. The table below shows several example models:

| Model     |     Training Speed      |    Memory     |
| :-------- | :---------------------: | :-----------: |
| ViTPose-B | 29.6% ↑ (0.931 → 0.655) | 10586 → 10663 |
| ViTPose-S | 33.7% ↑ (0.563 → 0.373) |  6091 → 6170  |
| HRNet-w32 | 12.8% ↑ (0.553 → 0.482) | 9849 → 10145  |
| HRNet-w48 | 37.1% ↑ (0.437 → 0.275) |  7319 → 7394  |
| RTMPose-t | 6.3% ↑ (1.533 → 1.437)  |  6292 → 6489  |
| RTMPose-s | 13.1% ↑ (1.645 → 1.430) |  9013 → 9208  |

- Pytorch 2.0 test, add projects doc and refactor by @LareinaM in [PR#2136](https://github.com/open-mmlab/mmpose/pull/2136)

# Deformable DETR

> [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://arxiv.org/abs/2010.04159)

DETR has been recently proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance. However, it suffers from slow convergence and limited feature spatial resolution, due to the limitation of Transformer attention modules in processing image feature maps. To mitigate these issues, we proposed Deformable DETR, whose attention modules only attend to a small set of key sampling points around a reference. Deformable DETR can achieve better performance than DETR (especially on small objects) with 10 times less training epochs. Extensive experiments on the COCO benchmark demonstrate the effectiveness of our approach.

<img src="https://user-images.githubusercontent.com/40661020/143877617-ad9b24fd-77ce-46aa-9689-1a44b5594132.png"/>
</div>

</div>

| Backbone |                Model                | Lr schd | box AP |                              Config                               |                                                                                                                                                                                                       Download                                                                                                                                                                                                       |

|   R-50   |           Deformable DETR           |   50e   |  44.3  |         [config](./deformable-detr_r50_16xb2-50e_coco.py)         |                                 [model](https://download.openmmlab.com/mmdetection/v3.0/deformable_detr/deformable-detr_r50_16xb2-50e_coco/deformable-detr_r50_16xb2-50e_coco_20221029_210934-6bc7d21b.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/deformable_detr/deformable-detr_r50_16xb2-50e_coco/deformable-detr_r50_16xb2-50e_coco_20221029_210934.log.json)                                 |

|   R-50   | + iterative bounding box refinement |   50e   |  46.2  |     [config](./deformable-detr-refine_r50_16xb2-50e_coco.py)      |                   [model](https://download.openmmlab.com/mmdetection/v3.0/deformable_detr/deformable-detr-refine_r50_16xb2-50e_coco/deformable-detr-refine_r50_16xb2-50e_coco_20221022_225303-844e0f93.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/deformable_detr/deformable-detr-refine_r50_16xb2-50e_coco/deformable-detr-refine_r50_16xb2-50e_coco_20221022_225303.log.json)                   |

|   R-50   |    ++ two-stage Deformable DETR     |   50e   |  47.0  | [config](./deformable-detr-refine-twostage_r50_16xb2-50e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v3.0/deformable_detr/deformable-detr-refine-twostage_r50_16xb2-50e_coco/deformable-detr-refine-twostage_r50_16xb2-50e_coco_20221021_184714-acc8a5ff.pth) \| [log](https://download.openmmlab.com/mmdetection/v3.0/deformable_detr/deformable-detr-refine-twostage_r50_16xb2-50e_coco/deformable-detr-refine-twostage_r50_16xb2-50e_coco_20221021_184714.log.json) |

### NOTE

1. All models are trained with batch size 32.
2. The performance is unstable. `Deformable DETR` and `iterative bounding box refinement` may fluctuate about 0.3 mAP. `two-stage Deformable DETR` may fluctuate about 0.2 mAP.

We provide the config files for Deformable DETR: [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://arxiv.org/abs/2010.04159).

```latex
zhu2021deformable,

```latex
title={Deformable DETR: Deformable Transformers for End-to-End Object Detection},
author={Xizhou Zhu and Weijie Su and Lewei Lu and Bin Li and Xiaogang Wang and Jifeng Dai},
booktitle={International Conference on Learning Representations},
url={https://openreview.net/forum?id=gZ9hCDWe6ke}
}
```

<details>

```bibtex
}
```

<details>

```bibtex

<details>

```bibtex
}
```

<details>

```bibtex

| [pose_hrnet_w32_fp16](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_fp16-8xb64-210e_coco-256x192.py) |  256x192   | 0.749 |      0.907      |      0.822      | 0.802 |      0.946      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_fp16-8xb64-210e_coco-256x192-f1e84e3b_20220914.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_fp16-8xb64-210e_coco-256x192_20220914.log) |

# Animated Drawings (SIGGRAPH'2023)

> [A Method for Animating Children's Drawings of The Human Figure](https://arxiv.org/abs/2303.12741)

> **Task**: Drawing

Children’s drawings have a wonderful inventiveness, creativity, and variety to them. We present a system that automatically animates children’s drawings of the human figure, is robust to the variance inherent in these depictions, and is simple and straightforward enough for anyone to use. We demonstrate the value and broad appeal of our approach by building and releasing the Animated Drawings Demo, a freely available public website that has been used by millions of people around the world. We present a set of experiments exploring the amount of training data needed for fine-tuning, as well as a perceptual study demonstrating the appeal of a novel twisted perspective retargeting technique. Finally, we introduce the Amateur Drawings Dataset, a first-of-its-kind annotated dataset, collected via the public demo, containing over 178,000 amateur drawings and corresponding user-accepted character bounding boxes, segmentation masks, and joint location annotations.

 <img src="https://user-images.githubusercontent.com/6675724/219223438-2c93f9cb-d4b5-45e9-a433-149ed76affa6.gif" width="800"/>
</div >

### 1. Install Animated Drawings

```shell
cd  mmagic/projects/animated_drawings
pip install -e git+https://github.com/facebookresearch/AnimatedDrawings.git#egg=animated_drawings
```

### 2. Download resources

```shell
mkdir resources
# download image
wget -O sakuragi.png https://user-images.githubusercontent.com/12782558/236157945-452fb9d0-e02e-4f36-8338-34f0ca0fe962.png
# download mask image
wget -O sakuragi_mask.png https://user-images.githubusercontent.com/12782558/236157965-539a5467-edae-40d0-a9da-7bb5906bcdc4.png
# download background image
wget -O basketball_playground.png https://user-images.githubusercontent.com/12782558/236190727-0e456482-2ae3-4304-9e6c-6ba7d319ea71.png
```

### 3. Generate character

```

By running following codes, you will get texture and mask images for animated rendering in characters/slamdunk directory.

```shell
cd mmagic/projects/animated_drawings
python tools/generate_character.py
```

### 3. Generate video

By running following codes, you will get a Sakuragi moving like a zombie.

```shell
python tools/generate_video.py
```

The output video will be saved at resources dir, and it looks like this:

 <img src="https://user-images.githubusercontent.com/12782558/236162056-c9a65baa-89c4-4cb3-84da-7777f5f21757.gif" width="512"/>
</div >

```bibtex
@misc{smith2023method,
      title={A Method for Animating Children's Drawings of the Human Figure},

</div >

```bibtex
      author={Harrison Jesse Smith and Qingyuan Zheng and Yifei Li and Somya Jain and Jessica K. Hodgins},
      eprint={2303.12741},
}
```

# TorchScript support

```bash

<details>

```bibtex
}
```

<details>

```bibtex

| [pose_resnetv1d_50](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_resnetv1d50_8xb64-210e_mpii-256x256.py) |  256x256   | 0.881 |  0.290   | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnetv1d/resnetv1d50_mpii_256x256-2337a92e_20200812.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnetv1d/resnetv1d50_mpii_256x256_20200812.log.json) |
| [pose_resnetv1d_101](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_resnetv1d101_8xb64-210e_mpii-256x256.py) |  256x256   | 0.883 |  0.295   | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnetv1d/resnetv1d101_mpii_256x256-2851d710_20200812.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnetv1d/resnetv1d101_mpii_256x256_20200812.log.json) |

| [pose_resnetv1d_152](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_resnetv1d152_8xb64-210e_mpii-256x256.py) |  256x256   | 0.888 |  0.300   | [ckpt](https://download.openmmlab.com/mmpose/top_down/resnetv1d/resnetv1d152_mpii_256x256-8b10a87c_20200812.pth) | [log](https://download.openmmlab.com/mmpose/top_down/resnetv1d/resnetv1d152_mpii_256x256_20200812.log.json) |

# Structures

To unify input and output interfaces  between different models and modules, OpenMMLab 2.0 MMEngine defines an abstract data structure,
it has implemented basic functions of `Create`, `Read`, `Update`, `Delete`, supported data transferring among different types of devices
and tensor-like or dictionary-like operations such as `.cpu()`, `.cuda()`, `.get()` and `.detach()`.
More details can be found [here](https://github.com/open-mmlab/mmengine/blob/main/docs/en/advanced_tutorials/data_element.md).

MMSegmentation also follows this interface protocol and defines `SegDataSample` which is used to encapsulate the data of semantic segmentation task.

## Semantic Segmentation Data SegDataSample

[SegDataSample](mmseg.structures.SegDataSample) includes three main fields `gt_sem_seg`, `pred_sem_seg` and `seg_logits`, which are used to store the annotation information and prediction results respectively.

| Field          | Type                      | Description                                |
| -------------- | ------------------------- | ------------------------------------------ |
| gt_sem_seg     | [`PixelData`](#pixeldata) | Annotation information.                    |
| pred_instances | [`PixelData`](#pixeldata) | The predicted result.                      |
| seg_logits     | [`PixelData`](#pixeldata) | The raw (non-normalized) predicted result. |

The following sample code demonstrates the use of `SegDataSample`.

```python

img_meta = dict(img_shape=(4, 4, 3),
                 pad_shape=(4, 4, 3))
# defining gt_segmentations for encapsulate the ground truth data
gt_segmentations = PixelData(metainfo=img_meta)
gt_segmentations.data = torch.randint(0, 2, (1, 4, 4))

# add and process property in SegDataSample
data_sample.gt_sem_seg = gt_segmentations

assert 'gt_sem_seg' in data_sample
assert 'sem_seg' in data_sample.gt_sem_seg
assert 'img_shape' in data_sample.gt_sem_seg.metainfo_keys()
print(data_sample.gt_sem_seg.shape)
'''
(4, 4)
'''
print(data_sample)
'''
<SegDataSample(

    gt_sem_seg: <PixelData(

            img_shape: (4, 4, 3)
            pad_shape: (4, 4, 3)

            data: tensor([[[1, 1, 1, 0],
                         [1, 0, 1, 1],
                         [1, 1, 1, 1],
                         [0, 1, 0, 1]]])
        ) at 0x1c2b4156460>
) at 0x1c2aae44d60>
'''

# delete and change property in SegDataSample

'''

data_sample.gt_sem_seg.set_metainfo(dict(img_shape=(4,4,9), pad_shape=(4,4,9)))
del data_sample.gt_sem_seg.img_shape

# Tensor-like operations
cuda_gt_segmentations = gt_segmentations.cuda()
cuda_gt_segmentations = gt_segmentations.to('cuda:0')
cpu_gt_segmentations = cuda_gt_segmentations.cpu()
cpu_gt_segmentations = cuda_gt_segmentations.to('cpu')
```

## Customize New Property in SegDataSample

If you want to customize new property in `SegDataSample`, you may follow [SegDataSample](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/structures/seg_data_sample.py) below:

```python
class SegDataSample(BaseDataElement):

```python
    ...

    @property
    def xxx_property(self) -> xxxData:
        return self._xxx_property

    @xxx_property.setter
    def xxx_property(self, value: xxxData) -> None:
        self.set_field(value, '_xxx_property', dtype=xxxData)

    @xxx_property.deleter
    def xxx_property(self) -> None:
        del self._xxx_property
```

Then a new property would be added to `SegDataSample`.

It is recommended to symlink the dataset root to `$MMSEGMENTATION/data`.
For users in China, we also recommend you get the dsdl dataset from our opensource platform [OpenDataLab](https://opendatalab.com/), for better download and use experience，here is an example: [DSDLReadme](../../../configs/dsdl/README.md)， welcome to try.

```none
├── mmseg
├── tools
├── data
│   │   │   ├── val
│   │   │   ├── val

│   ├── ade

│   ├── DRIVE
│   ├── HRF
│   ├── STARE
|   │   ├── gps
|   │   │   ├── val
|   │   ├── gt
|   │   │   └── val

|   │   ├── gps
|   │   │   ├── val
|   │   ├── gt
|   │   │   └── val
|   │   |   ├── val
│   │   │   ├── val
│   │   │   ├── val
│   │   │   ├── val
│   │   │   ├── val
│   │   │   ├── val

│   │   │   ├── val
│   │   │   ├── val
│   │   │   ├── val
│   ├── iSAID
│   │   │   ├── val
│   │   │   ├── val
│   │   │   ├── val
│   │   │   ├── val

│   │   │   └── 10k

```

The data could be found [here](https://www.cityscapes-dataset.com/downloads/) after registration.

By convention, `**labelTrainIds.png` are used for cityscapes training.
We provided a [script](https://github.com/open-mmlab/mmsegmentation/blob/1.x/tools/dataset_converters/cityscapes.py) based on [cityscapesscripts](https://github.com/mcordts/cityscapesScripts)to generate `**labelTrainIds.png`.

```shell
# --nproc means 8 process for conversion, which could be omitted as well.
```

Pascal VOC 2012 could be downloaded from [here](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar).

```

Beside, most recent works on Pascal VOC dataset usually exploit extra augmentation data, which could be found [here](http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/semantic_contours/benchmark.tgz).

If you would like to use augmented VOC dataset, please run following command to convert augmentation annotations into proper format.

```shell
```

Please refer to [concat dataset](../advanced_guides/add_datasets.md#concatenate-dataset) and [voc_aug config example](../../../configs/_base_/datasets/pascal_voc12_aug.py) for details about how to concatenate them and train them together.

## ADE20K

## ADE20K

The training and validation set of ADE20K could be download from this [link](http://data.csail.mit.edu/places/ADEchallenge/ADEChallengeData2016.zip).
We may also download test set from [here](http://data.csail.mit.edu/places/ADEchallenge/release_test.zip).

The training and validation set of Pascal Context could be download from [here](http://host.robots.ox.ac.uk/pascal/VOC/voc2010/VOCtrainval_03-May-2010.tar). You may also download test set from [here](http://host.robots.ox.ac.uk:8080/eval/downloads/VOC2010test.tar) after registration.

To split the training and validation set from original dataset, you may download trainval_merged.json from [here](https://codalabuser.blob.core.windows.net/public/trainval_merged.json).

If you would like to use Pascal Context dataset, please install [Detail](https://github.com/zhanghang1989/detail-api) and then run the following command to convert annotations into proper format.

```shell

```shell
```

The data could be downloaded [here](http://calvin.inf.ed.ac.uk/wp-content/uploads/data/cocostuffdataset/cocostuff-10k-v1.1.zip) by wget.

For COCO Stuff 10k dataset, please run the following commands to download and convert the dataset.

```shell
# download

# unzip

```

By convention, mask labels in `/path/to/coco_stuff164k/annotations/*2014/*_labelTrainIds.png` are used for COCO Stuff 10k training and testing.

```

For COCO Stuff 164k dataset, please run the following commands to download and convert the augmented dataset.

```shell
# download

# unzip

```

By convention, mask labels in `/path/to/coco_stuff164k/annotations/*2017/*_labelTrainIds.png` are used for COCO Stuff 164k training and testing.

```

The details of this dataset could be found at [here](https://github.com/nightrome/cocostuff#downloads).

The training and validation set of CHASE DB1 could be download from [here](https://staffnet.kingston.ac.uk/~ku15565/CHASE_DB1/assets/CHASEDB1.zip).

To convert CHASE DB1 dataset to MMSegmentation format, you should run the following command:

```shell
```

The script will make directory structure automatically.

## DRIVE

The training and validation set of DRIVE could be download from [here](https://drive.grand-challenge.org/). Before that, you should register an account. Currently '1st_manual' is not provided officially.

To convert DRIVE dataset to MMSegmentation format, you should run the following command:

```shell

```shell
```

## HRF

First, download [healthy.zip](https://www5.cs.fau.de/fileadmin/research/datasets/fundus-images/healthy.zip), [glaucoma.zip](https://www5.cs.fau.de/fileadmin/research/datasets/fundus-images/glaucoma.zip), [diabetic_retinopathy.zip](https://www5.cs.fau.de/fileadmin/research/datasets/fundus-images/diabetic_retinopathy.zip), [healthy_manualsegm.zip](https://www5.cs.fau.de/fileadmin/research/datasets/fundus-images/healthy_manualsegm.zip), [glaucoma_manualsegm.zip](https://www5.cs.fau.de/fileadmin/research/datasets/fundus-images/glaucoma_manualsegm.zip) and [diabetic_retinopathy_manualsegm.zip](https://www5.cs.fau.de/fileadmin/research/datasets/fundus-images/diabetic_retinopathy_manualsegm.zip).

To convert HRF dataset to MMSegmentation format, you should run the following command:

```shell
```

## STARE

First, download [stare-images.tar](http://cecas.clemson.edu/~ahoover/stare/probing/stare-images.tar), [labels-ah.tar](http://cecas.clemson.edu/~ahoover/stare/probing/labels-ah.tar) and [labels-vk.tar](http://cecas.clemson.edu/~ahoover/stare/probing/labels-vk.tar).

To convert STARE dataset to MMSegmentation format, you should run the following command:

```shell
```

```shell
```

Since we only support test models on this dataset, you may only download [the validation set](https://data.vision.ee.ethz.ch/csakarid/shared/GCMA_UIoU/Dark_Zurich_val_anon.zip).

Since we only support test models on this dataset, you may only download [the test set](http://data.vision.ee.ethz.ch/daid/NighttimeDriving/NighttimeDrivingTest.zip).

## LoveDA

The data could be downloaded from Google Drive [here](https://drive.google.com/drive/folders/1ibYV0qwn4yuuh068Rnc-w4tPi0U0c-ti?usp=sharing).

Or it can be downloaded from [zenodo](https://zenodo.org/record/5706578#.YZvN7SYRXdF), you should run the following command:

```shell
# Download Train.zip
# Download Val.zip

```shell
# Download Test.zip
```

For LoveDA dataset, please run the following command to re-organize the dataset.

```shell
```

Using trained model to predict test set of LoveDA and submit it to server can be found [here](https://codalab.lisn.upsaclay.fr/competitions/421).

More details about LoveDA can be found [here](https://github.com/Junjue-Wang/LoveDA).

The [Potsdam](https://www.isprs.org/education/benchmarks/UrbanSemLab/2d-sem-label-potsdam.aspx) dataset is for urban semantic segmentation used in the 2D Semantic Labeling Contest - Potsdam.

The dataset can be requested at the challenge [homepage](https://www.isprs.org/education/benchmarks/UrbanSemLab/default.aspx).

Or download on [BaiduNetdisk](https://pan.baidu.com/s/1K-cLVZnd1X7d8c26FQ-nGg?pwd=mseg)，password：mseg, [Google Drive](https://drive.google.com/drive/folders/1w3EJuyUGet6_qmLwGAWZ9vw5ogeG0zLz?usp=sharing) and [OpenDataLab](https://opendatalab.com/ISPRS_Potsdam/download).
The '2_Ortho_RGB.zip' and '5_Labels_all_noBoundary.zip' are required.

For Potsdam dataset, please run the following command to re-organize the dataset.

```shell
```

In our default setting, it will generate 3456 images for training and 2016 images for validation.

The [Vaihingen](https://www2.isprs.org/commissions/comm2/wg4/benchmark/2d-sem-label-vaihingen/) dataset is for urban semantic segmentation used in the 2D Semantic Labeling Contest - Vaihingen.

The dataset can be requested at the challenge [homepage](https://www2.isprs.org/commissions/comm2/wg4/benchmark/data-request-form/).
Or [BaiduNetdisk](https://pan.baidu.com/s/109D3WLrLafsuYtLeerLiiA?pwd=mseg)，password：mseg, [Google Drive](https://drive.google.com/drive/folders/1w3NhvLVA2myVZqOn2pbiDXngNC7NTP_t?usp=sharing).
The 'ISPRS_semantic_labeling_Vaihingen.zip' and 'ISPRS_semantic_labeling_Vaihingen_ground_truth_eroded_COMPLETE.zip' are required.

For Vaihingen dataset, please run the following command to re-organize the dataset.

```shell
```

In our default setting (`clip_size`=512, `stride_size`=256), it will generate 344 images for training and 398 images for validation.

## iSAID

```

## iSAID

The data images could be download from [DOTA-v1.0](https://captain-whu.github.io/DOTA/dataset.html) (train/val/test)

The data annotations could be download from [iSAID](https://captain-whu.github.io/iSAID/dataset.html) (train/val)

The dataset is a Large-scale Dataset for Instance Segmentation (also have semantic segmentation) in Aerial Images.

You may need to follow the following structure for dataset preparation after downloading iSAID dataset.

```none
├── data
│   ├── iSAID
│   │   ├── val
│   │   ├── test

│   │   ├── val
│   │   ├── test
```

```shell
```

In our default setting (`patch_width`=896, `patch_height`=896, `overlap_area`=384), it will generate 33978 images for training and 11644 images for validation.

This dataset could be download from [this page](https://lip.sysuhcp.com/overview.php).

Please run the following commands to unzip dataset.

```shell
cd LIP
cd ..
```

The contents of LIP datasets include:

```none
├── data
│   ├── LIP

cd ..
```

```none
├── data
│   ├── LIP
│   │   │   ├── ...
│   │   │   ├── ...
│   │   │   ├── ...
│   │   │   ├── ...
```

This dataset could be download from [this page](https://www.synapse.org/#!Synapse:syn3193805/wiki/).

To follow the data preparation setting of [TransUNet](https://arxiv.org/abs/2102.04306), which splits original training set (30 scans) into new training (18 scans) and validation set (12 scans). Please run the following command to prepare the dataset.

```shell
```

Then create `train.txt` and `val.txt` to split dataset.

```shell
```

According to TransUnet, the following is the data set division.

train.txt

```none
```

val.txt

```none
```

The contents of synapse datasets include:

```none
│   ├── img
│   │   ├── ...
│   ├── label
│   │   ├── ...
```

Then, use this command to convert synapse dataset.

│   │   ├── ...
│   ├── label
│   │   ├── ...
```

```shell
```

Noted that MMSegmentation default evaluation metric (such as mean dice value) is calculated on 2D slice image, which is not comparable to results of 3D scan in some paper such as [TransUNet](https://arxiv.org/abs/2102.04306).

## REFUGE

Register in [REFUGE Challenge](https://refuge.grand-challenge.org) and download [REFUGE dataset](https://refuge.grand-challenge.org/REFUGE2Download).

Then, unzip `REFUGE2.zip` and the contents of original datasets include:

```none

```

Please run the following command to convert REFUGE dataset:

```shell
```

The script will make directory structure below:

```none
```

It includes 400 images for training, 400 images for validation and 400 images for testing which is the same as REFUGE 2018 dataset.

- The dataset could be download [here](https://www.mapillary.com/dataset/vistas) after registration.

- Mapillary Vistas Dataset use 8-bit with color-palette to store labels. No conversion operation is required.

- Assumption you have put the dataset zip file in `mmsegmentation/data/mapillary`

- Please run the following commands to unzip dataset.

  ```bash
  ```

- After unzip, you will get Mapillary Vistas Dataset like this structure. Semantic segmentation mask labels in `labels` folder.

  ```none
  ├── mmseg
  ├── tools
  ├── data

  ```

- You could set Datasets version with `MapillaryDataset_v1` and `MapillaryDataset_v2` in your configs.
  View the Mapillary Vistas Datasets config file here [V1.2](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/_base_/datasets/mapillary_v1.py) and [V2.0](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/_base_/datasets/mapillary_v2.py)

[LEVIR-CD](https://justchenhao.github.io/LEVIR/) Large-scale Remote Sensing Change Detection Dataset for Building.

Download the dataset from [here](https://justchenhao.github.io/LEVIR/).

The supplement version of the dataset can be requested on the [homepage](https://github.com/S2Looking/Dataset)

Please download the supplement version of the dataset, then unzip `LEVIR-CD+.zip`, the contents of original datasets include:

```none
│   │   │   ├── A
│   │   │   ├── B
│   │   ├── test
│   │   │   ├── A
│   │   │   ├── B
```

For LEVIR-CD dataset, please run the following command to crop images without overlap:

```shell
```

The size of cropped image is 256x256, which is consistent with the original paper.

## BDD100K

```

## BDD100K

- You could download BDD100k datasets from  [here](https://bdd-data.berkeley.edu/) after  registration.

- You can download images and masks by clicking  `10K Images` button and `Segmentation` button.

- After download, unzip by the following instructions:

  ```bash
  ```

- And get

```none
├── mmseg
├── tools
├── data
│   │   │   └── 10k

```

# DCN

> [Deformable Convolutional Networks](https://arxiv.org/abs/1703.06211)

Convolutional neural networks (CNNs) are inherently limited to model geometric transformations due to the fixed geometric structures in its building modules. In this work, we introduce two new modules to enhance the transformation modeling capacity of CNNs, namely, deformable convolution and deformable RoI pooling. Both are based on the idea of augmenting the spatial sampling locations in the modules with additional offsets and learning the offsets from target tasks, without additional supervision. The new modules can readily replace their plain counterparts in existing CNNs and can be easily trained end-to-end by standard back-propagation, giving rise to deformable convolutional networks. Extensive experiments validate the effectiveness of our approach on sophisticated vision tasks of object detection and semantic segmentation.

<img src="https://user-images.githubusercontent.com/40661020/143876246-c4985e25-e286-4511-9b7c-97af2857461e.png"/>
</div>

|    Backbone     |    Model     |  Style  |     Conv     | Pool  | Lr schd | Mem (GB) | Inf time (fps) | box AP | mask AP |                               Config                                |                                                                                                                                                                                       Download                                                                                                                                                                                       |

|    R-50-FPN     |    Faster    | pytorch | dconv(c3-c5) |   -   |   1x    |   4.0    |      17.8      |  41.3  |         |       [config](./faster-rcnn_r50-dconv-c3-c5_fpn_1x_coco.py)        |                      [model](https://download.openmmlab.com/mmdetection/v2.0/dcn/faster_rcnn_r50_fpn_dconv_c3-c5_1x_coco/faster_rcnn_r50_fpn_dconv_c3-c5_1x_coco_20200130-d68aed1e.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/dcn/faster_rcnn_r50_fpn_dconv_c3-c5_1x_coco/faster_rcnn_r50_fpn_dconv_c3-c5_1x_coco_20200130_212941.log.json)                       |

|    R-50-FPN     |    Faster    | pytorch |      -       | dpool |   1x    |   5.0    |      17.2      |  38.9  |         |          [config](./faster-rcnn_r50_fpn_dpool_1x_coco.py)           |                                  [model](https://download.openmmlab.com/mmdetection/v2.0/dcn/faster_rcnn_r50_fpn_dpool_1x_coco/faster_rcnn_r50_fpn_dpool_1x_coco_20200307-90d3c01d.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/dcn/faster_rcnn_r50_fpn_dpool_1x_coco/faster_rcnn_r50_fpn_dpool_1x_coco_20200307_203250.log.json)                                   |

|    R-101-FPN    |    Faster    | pytorch | dconv(c3-c5) |   -   |   1x    |   6.0    |      12.5      |  42.7  |         |       [config](./faster-rcnn_r101-dconv-c3-c5_fpn_1x_coco.py)       |                    [model](https://download.openmmlab.com/mmdetection/v2.0/dcn/faster_rcnn_r101_fpn_dconv_c3-c5_1x_coco/faster_rcnn_r101_fpn_dconv_c3-c5_1x_coco_20200203-1377f13d.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/dcn/faster_rcnn_r101_fpn_dconv_c3-c5_1x_coco/faster_rcnn_r101_fpn_dconv_c3-c5_1x_coco_20200203_230019.log.json)                     |

| X-101-32x4d-FPN |    Faster    | pytorch | dconv(c3-c5) |   -   |   1x    |   7.3    |      10.0      |  44.5  |         |    [config](./faster-rcnn_x101-32x4d-dconv-c3-c5_fpn_1x_coco.py)    |        [model](https://download.openmmlab.com/mmdetection/v2.0/dcn/faster_rcnn_x101_32x4d_fpn_dconv_c3-c5_1x_coco/faster_rcnn_x101_32x4d_fpn_dconv_c3-c5_1x_coco_20200203-4f85c69c.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/dcn/faster_rcnn_x101_32x4d_fpn_dconv_c3-c5_1x_coco/faster_rcnn_x101_32x4d_fpn_dconv_c3-c5_1x_coco_20200203_001325.log.json)         |

|    R-50-FPN     |     Mask     | pytorch | dconv(c3-c5) |   -   |   1x    |   4.5    |      15.4      |  41.8  |  37.4   |        [config](./mask-rcnn_r50-dconv-c3-c5_fpn_1x_coco.py)         |                          [model](https://download.openmmlab.com/mmdetection/v2.0/dcn/mask_rcnn_r50_fpn_dconv_c3-c5_1x_coco/mask_rcnn_r50_fpn_dconv_c3-c5_1x_coco_20200203-4d9ad43b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/dcn/mask_rcnn_r50_fpn_dconv_c3-c5_1x_coco/mask_rcnn_r50_fpn_dconv_c3-c5_1x_coco_20200203_061339.log.json)                           |

|    R-101-FPN    |     Mask     | pytorch | dconv(c3-c5) |   -   |   1x    |   6.5    |      11.7      |  43.5  |  38.9   |        [config](./mask-rcnn_r101-dconv-c3-c5_fpn_1x_coco.py)        |                        [model](https://download.openmmlab.com/mmdetection/v2.0/dcn/mask_rcnn_r101_fpn_dconv_c3-c5_1x_coco/mask_rcnn_r101_fpn_dconv_c3-c5_1x_coco_20200216-a71f5bce.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/dcn/mask_rcnn_r101_fpn_dconv_c3-c5_1x_coco/mask_rcnn_r101_fpn_dconv_c3-c5_1x_coco_20200216_191601.log.json)                         |

|    R-50-FPN     |   Cascade    | pytorch | dconv(c3-c5) |   -   |   1x    |   4.5    |      14.6      |  43.8  |         |       [config](./cascade-rcnn_r50-dconv-c3-c5_fpn_1x_coco.py)       |                    [model](https://download.openmmlab.com/mmdetection/v2.0/dcn/cascade_rcnn_r50_fpn_dconv_c3-c5_1x_coco/cascade_rcnn_r50_fpn_dconv_c3-c5_1x_coco_20200130-2f1fca44.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/dcn/cascade_rcnn_r50_fpn_dconv_c3-c5_1x_coco/cascade_rcnn_r50_fpn_dconv_c3-c5_1x_coco_20200130_220843.log.json)                     |

|    R-101-FPN    |   Cascade    | pytorch | dconv(c3-c5) |   -   |   1x    |   6.4    |      11.0      |  45.0  |         |      [config](./cascade-rcnn_r101-dconv-c3-c5_fpn_1x_coco.py)       |                  [model](https://download.openmmlab.com/mmdetection/v2.0/dcn/cascade_rcnn_r101_fpn_dconv_c3-c5_1x_coco/cascade_rcnn_r101_fpn_dconv_c3-c5_1x_coco_20200203-3b2f0594.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/dcn/cascade_rcnn_r101_fpn_dconv_c3-c5_1x_coco/cascade_rcnn_r101_fpn_dconv_c3-c5_1x_coco_20200203_224829.log.json)                   |

|    R-50-FPN     | Cascade Mask | pytorch | dconv(c3-c5) |   -   |   1x    |   6.0    |      10.0      |  44.4  |  38.6   |    [config](./cascade-mask-rcnn_r50-dconv-c3-c5_fpn_1x_coco.py)     |          [model](https://download.openmmlab.com/mmdetection/v2.0/dcn/cascade_mask_rcnn_r50_fpn_dconv_c3-c5_1x_coco/cascade_mask_rcnn_r50_fpn_dconv_c3-c5_1x_coco_20200202-42e767a2.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/dcn/cascade_mask_rcnn_r50_fpn_dconv_c3-c5_1x_coco/cascade_mask_rcnn_r50_fpn_dconv_c3-c5_1x_coco_20200202_010309.log.json)           |

|    R-101-FPN    | Cascade Mask | pytorch | dconv(c3-c5) |   -   |   1x    |   8.0    |      8.6       |  45.8  |  39.7   |    [config](./cascade-mask-rcnn_r101-dconv-c3-c5_fpn_1x_coco.py)    |        [model](https://download.openmmlab.com/mmdetection/v2.0/dcn/cascade_mask_rcnn_r101_fpn_dconv_c3-c5_1x_coco/cascade_mask_rcnn_r101_fpn_dconv_c3-c5_1x_coco_20200204-df0c5f10.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/dcn/cascade_mask_rcnn_r101_fpn_dconv_c3-c5_1x_coco/cascade_mask_rcnn_r101_fpn_dconv_c3-c5_1x_coco_20200204_134006.log.json)         |

| X-101-32x4d-FPN | Cascade Mask | pytorch | dconv(c3-c5) |   -   |   1x    |   9.2    |                |  47.3  |  41.1   | [config](./cascade-mask-rcnn_x101-32x4d-dconv-c3-c5_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/dcn/cascade_mask_rcnn_x101_32x4d_fpn_dconv_c3-c5_1x_coco/cascade_mask_rcnn_x101_32x4d_fpn_dconv_c3-c5_1x_coco-e75f90c8.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/dcn/cascade_mask_rcnn_x101_32x4d_fpn_dconv_c3-c5_1x_coco/cascade_mask_rcnn_x101_32x4d_fpn_dconv_c3-c5_1x_coco-20200606_183737.log.json) |

| R-50-FPN (FP16) |     Mask     | pytorch | dconv(c3-c5) |   -   |   1x    |   3.0    |                |  41.9  |  37.5   |      [config](./mask-rcnn_r50-dconv-c3-c5_fpn_amp-1x_coco.py)       |            [model](https://download.openmmlab.com/mmdetection/v2.0/fp16/mask_rcnn_r50_fpn_fp16_dconv_c3-c5_1x_coco/mask_rcnn_r50_fpn_fp16_dconv_c3-c5_1x_coco_20210520_180247-c06429d2.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/fp16/mask_rcnn_r50_fpn_fp16_dconv_c3-c5_1x_coco/mask_rcnn_r50_fpn_fp16_dconv_c3-c5_1x_coco_20210520_180247.log.json)            |

- `dconv` denotes deformable convolution, `c3-c5` means adding dconv in resnet stage 3 to 5. `dpool` denotes deformable roi pooling.

```latex
@inproceedings{dai2017deformable,
  title={Deformable Convolutional Networks},
  author={Dai, Jifeng and Qi, Haozhi and Xiong, Yuwen and Li, Yi and Zhang, Guodong and Hu, Han and Wei, Yichen},
}
```

# MMagic Deployment

- [MMagic Deployment](#mmagic-deployment)
    - [Install mmagic](#install-mmagic)
    - [Convert super resolution model](#convert-super-resolution-model)

[MMagic](https://github.com/open-mmlab/mmagic/tree/main) aka `mmagic` is an open-source image and video editing toolbox based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com/) project.

### Install mmagic

Please follow the [installation guide](https://github.com/open-mmlab/mmagic/tree/main#installation) to install mmagic.

```shell

```shell
```

You can use [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/tree/main/tools/deploy.py) to convert mmagic models to the specified backend models. Its detailed usage can be learned from [here](https://github.com/open-mmlab/mmdeploy/tree/main/docs/en/02-how-to-run/convert_model.md#usage).

When using `tools/deploy.py`, it is crucial to specify the correct deployment config. We've already provided builtin deployment config [files](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmagic) of all supported backends for mmagic, under which the config file path follows the pattern:

```
```

  MMDeploy supports models of one task in mmagic, i.e., `super resolution`. Please refer to chapter [supported models](#supported-models) for task-model organization.

### Convert super resolution model

The command below shows an example about converting `ESRGAN` model to onnx model that can be inferred by ONNX Runtime.

```shell
  --show \
```

esrgan_psnr_x4c64b23g32_1x16_1000k_div2k_20200420-bf5c993c.pth \
  --show \
```

You can also convert the above model to other backend models by changing the deployment config file `*_onnxruntime_dynamic.py` to [others](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmagic), e.g., converting to tensorrt model by `super-resolution/super-resolution_tensorrt-_dynamic-32x32-512x512.py`.

```{tip}
```

The converted model locates in the working directory like `mmdeploy_models/mmagic/ort` in the previous example. It includes:

```
```

```
```

in which,

The whole package **mmdeploy_models/mmagic/ort** is defined as **mmdeploy SDK model**, i.e., **mmdeploy SDK model** includes both backend model and inference meta information.

```python

```

```python
import cv2

```

```python
import cv2

```

# SPTS: Single-Point Text Spotting

<div>
<a href="https://arxiv.org/abs/2112.07917">[arXiv paper]</a>
</div>

This is an implementation of [SPTS](https://github.com/shannanyinxiang/SPTS) based on [MMOCR](https://github.com/open-mmlab/mmocr/tree/dev-1.x), [MMCV](https://github.com/open-mmlab/mmcv), and [MMEngine](https://github.com/open-mmlab/mmengine).

Existing scene text spotting (i.e., end-to-end text detection and recognition) methods rely on costly bounding box annotations (e.g., text-line, word-level, or character-level bounding boxes). For the first time, we demonstrate that training scene text spotting models can be achieved with an extremely low-cost annotation of a single-point for each instance. We propose an end-to-end scene text spotting method that tackles scene text spotting as a sequence prediction task. Given an image as input, we formulate the desired detection and recognition results as a sequence of discrete tokens and use an auto-regressive Transformer to predict the sequence. The proposed method is simple yet effective, which can achieve state-of-the-art results on widely used benchmarks. Most significantly, we show that the performance is not very sensitive to the positions of the point annotation, meaning that it can be much easier to be annotated or even be automatically generated than the bounding box that requires precise positions. We believe that such a pioneer attempt indicates a significant opportunity for scene text spotting applications of a much larger scale than previously possible.

<center>
<img src="https://user-images.githubusercontent.com/22607038/215685203-fbf2d00c-39d3-48bb-9d05-4fd28c56431c.png">
</center>

## Usage

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `SPTS/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
# Linux
```

```shell
# Linux
```

As of now, the implementation uses datasets provided by SPTS for **pre-training**, and uses MMOCR's datasets for **fine-tuning and testing**. It's because the test split of SPTS's datasets does not contain enough information for e2e evaluation; and MMOCR's dataset preparer has not yet supported all the datasets used in SPTS. *We are working on this issue, and they will be available in MMOCR's dataset preparer very soon.*

Please follow these steps to prepare the datasets:

- Download and extract all the SPTS datasets into `spts-data/` following [SPTS official guide](https://github.com/shannanyinxiang/SPTS#dataset).

- Use [Dataset Preparer](https://mmocr.readthedocs.io/en/dev-1.x/user_guides/data_prepare/dataset_preparer.html) to prepare `icdar2013`, `icdar2015` and `totaltext` for `textspotting` task.

  ```shell
  # Run in MMOCR's root directory

```shell
  python tools/dataset_converters/prepare_dataset.py icdar2013 icdar2015 totaltext  --task textspotting
  ```

  Then create a soft link to `data/` directory in the project root directory:

  ```shell
  ln -s ../../data/ .
  ```

#### Pretrain

```bash
mim train mmocr config/spts/spts_resnet50_8xb8-150e_pretrain-spts.py --work-dir work_dirs/ --amp
```

```bash
mim train mmocr config/spts/spts_resnet50_8xb8-150e_pretrain-spts.py --work-dir work_dirs/ --launcher pytorch --gpus 8 --amp
```

#### Finetune

Similarly, run the following command to finetune the model on a dataset (e.g. icdar2013):

```bash
mim train mmocr config/spts/spts_resnet50_8xb8-200e_icdar2013.py --work-dir work_dirs/ --cfg-options "load_from={CHECKPOINT_PATH}" --amp
```

```bash
```

To finetune on multiple GPUs, e.g. 8 GPUs, run the following command:

```bash
mim train mmocr config/spts/spts_resnet50_8xb8-200e_icdar2013.py --work-dir work_dirs/ --launcher pytorch --gpus 8 --cfg-options "load_from={CHECKPOINT_PATH}" --amp
```

In the current directory, run the following command to test the model on a dataset (e.g. icdar2013):

```bash
mim test mmocr config/spts/spts_resnet50_8xb8-200e_icdar2013.py --work-dir work_dirs/ --checkpoint ${CHECKPOINT_PATH}
```

## Convert Weights from Official Repo

Users may download the weights from [SPTS](https://github.com/shannanyinxiang/SPTS#inference) and use the conversion script to convert them into MMOCR format.

```bash
python tools/ckpt_adapter.py [SPTS_WEIGHTS_PATH] [MMOCR_WEIGHTS_PATH]
```

## Results

```bash
```

## Results

All the models are trained on 8x A100 GPUs with AMP on (`--amp`). The overall batch size is 64.

| Name       | Pretrained                                                                              | Generic | Weak  | Strong | Download                                                                              |
| ---------- | --------------------------------------------------------------------------------------- | ------- | ----- | ------ | ------------------------------------------------------------------------------------- |

| ICDAR 2013 | [model](https://download.openmmlab.com/mmocr/textspotting/spts/spts_resnet50_150e_pretrain-spts/spts_resnet50_150e_pretrain-spts-c9fe4c78.pth) / [log](https://download.openmmlab.com/mmocr/textspotting/spts/spts_resnet50_150e_pretrain-spts/20230223_194550.log) | 87.10   | 91.46 | 93.41  | [model](https://download.openmmlab.com/mmocr/textspotting/spts/spts_resnet50_200e_icdar2013/spts_resnet50_200e_icdar2013-64cb4d31.pth) / [log](https://download.openmmlab.com/mmocr/textspotting/spts/spts_resnet50_200e_icdar2013/20230303_140316.log) |

| ICDAR 2015 | [model](https://download.openmmlab.com/mmocr/textspotting/spts/spts_resnet50_150e_pretrain-spts/spts_resnet50_150e_pretrain-spts-c9fe4c78.pth) / [log](https://download.openmmlab.com/mmocr/textspotting/spts/spts_resnet50_150e_pretrain-spts/20230223_194550.log) | 69.09   | 73.45 | 79.19  | [model](https://download.openmmlab.com/mmocr/textspotting/spts/spts_resnet50_200e_icdar2015/spts_resnet50_200e_icdar2015-d6e8621c.pth) / [log](https://download.openmmlab.com/mmocr/textspotting/spts/spts_resnet50_200e_icdar2015/20230302_230026.log) |

|   Name    | Pretrained                                                                             | None-Hmean | Full-Hmean | Download                                                                              |
| :-------: | -------------------------------------------------------------------------------------- | :--------: | :--------: | ------------------------------------------------------------------------------------- |

| Totaltext | [model](https://download.openmmlab.com/mmocr/textspotting/spts/spts_resnet50_150e_pretrain-spts/spts_resnet50_150e_pretrain-spts-c9fe4c78.pth) / [log](https://download.openmmlab.com/mmocr/textspotting/spts/spts_resnet50_150e_pretrain-spts/20230223_194550.log) |   73.99    |   82.34    | [model](https://download.openmmlab.com/mmocr/textspotting/spts/spts_resnet50_200e_totaltext/spts_resnet50_200e_totaltext-e3521af6.pth) / [log](https://download.openmmlab.com/mmocr/textspotting/spts/spts_resnet50_200e_totaltext/20230303_103040.log) |

If you find SPTS useful in your research or applications, please cite SPTS with the following BibTeX entry.

```BibTeX
@inproceedings{peng2022spts,
  title={SPTS: Single-Point Text Spotting},
  author={Peng, Dezhi and Wang, Xinyu and Liu, Yuliang and Zhang, Jiaxin and Huang, Mingxin and Lai, Songxuan and Zhu, Shenggao and Li, Jing and Lin, Dahua and Shen, Chunhua and Bai, Xiang and Jin, Lianwen},

booktitle={Proceedings of the 30th ACM International Conference on Multimedia},
}
```

# HorNet

> [HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions](https://arxiv.org/abs/2207.14284)

Recent progress in vision Transformers exhibits great success in various tasks driven by the new spatial modeling mechanism based on dot-product self-attention. In this paper, we show that the key ingredients behind the vision Transformers, namely input-adaptive, long-range and high-order spatial interactions, can also be efficiently implemented with a convolution-based framework. We present the Recursive Gated Convolution (g nConv) that performs high-order spatial interactions with gated convolutions and recursive designs. The new operation is highly flexible and customizable, which is compatible with various variants of convolution and extends the two-order interactions in self-attention to arbitrary orders without introducing significant extra computation. g nConv can serve as a plug-and-play module to improve various vision Transformers and convolution-based models. Based on the operation, we construct a new family of generic vision backbones named HorNet. Extensive experiments on ImageNet classification, COCO object detection and ADE20K semantic segmentation show HorNet outperform Swin Transformers and ConvNeXt by a significant margin with similar overall architecture and training configurations. HorNet also shows favorable scalability to more training data and a larger model size. Apart from the effectiveness in visual encoders, we also show g nConv can be applied to task-specific decoders and consistently improve dense prediction performance with less computation. Our results demonstrate that g nConv can be a new basic module for visual modeling that effectively combines the merits of both vision Transformers and CNNs. Code is available at https://github.com/raoyongming/HorNet.

<img src="https://user-images.githubusercontent.com/24734142/188356236-b8e3db94-eaa6-48e9-b323-15e5ba7f2991.png" width="80%"/>
</div>

```python

predict = inference_model('hornet-tiny_3rdparty_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('hornet-tiny_3rdparty_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/hornet/hornet-tiny_8xb128_in1k.py https://download.openmmlab.com/mmclassification/v0/hornet/hornet-tiny_3rdparty_in1k_20220915-0e8eedff.pth
```

| Model                             |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                 Config                  |                                    Download                                     |
| :-------------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :-------------------------------------: | :-----------------------------------------------------------------------------: |

| `hornet-tiny_3rdparty_in1k`\*     | From scratch |   22.41    |   3.98    |   82.84   |   96.24   |  [config](hornet-tiny_8xb128_in1k.py)   | [model](https://download.openmmlab.com/mmclassification/v0/hornet/hornet-tiny_3rdparty_in1k_20220915-0e8eedff.pth) |
| `hornet-tiny-gf_3rdparty_in1k`\*  | From scratch |   22.99    |   3.90    |   82.98   |   96.38   | [config](hornet-tiny-gf_8xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/hornet/hornet-tiny-gf_3rdparty_in1k_20220915-4c35a66b.pth) |
| `hornet-small_3rdparty_in1k`\*    | From scratch |   49.53    |   8.83    |   83.79   |   96.75   |  [config](hornet-small_8xb64_in1k.py)   | [model](https://download.openmmlab.com/mmclassification/v0/hornet/hornet-small_3rdparty_in1k_20220915-5935f60f.pth) |

| `hornet-small-gf_3rdparty_in1k`\* | From scratch |   50.40    |   8.71    |   83.98   |   96.77   | [config](hornet-small-gf_8xb64_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/hornet/hornet-small-gf_3rdparty_in1k_20220915-649ca492.pth) |
| `hornet-base_3rdparty_in1k`\*     | From scratch |   87.26    |   15.58   |   84.24   |   96.94   |   [config](hornet-base_8xb64_in1k.py)   | [model](https://download.openmmlab.com/mmclassification/v0/hornet/hornet-base_3rdparty_in1k_20220915-a06176bb.pth) |
| `hornet-base-gf_3rdparty_in1k`\*  | From scratch |   88.42    |   15.42   |   84.32   |   96.95   | [config](hornet-base-gf_8xb64_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/hornet/hornet-base-gf_3rdparty_in1k_20220915-82c06fa7.pth) |

*Models with * are converted from the [official repo](https://github.com/raoyongming/HorNet). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@article{rao2022hornet,
  title={HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions},
  author={Rao, Yongming and Zhao, Wenliang and Tang, Yansong and Zhou, Jie and Lim, Ser-Lam and Lu, Jiwen},
  journal={arXiv preprint arXiv:2207.14284},
}
```

# ActionCLIP Project

[ActionCLIP: A New Paradigm for Video Action Recognition](https://arxiv.org/abs/2109.08472)

The canonical approach to video action recognition dictates a neural model to do a classic and standard 1-of-N majority vote task. They are trained to predict a fixed set of predefined categories, limiting their transferable ability on new datasets with unseen concepts. In this paper, we provide a new perspective on action recognition by attaching importance to the semantic information of label texts rather than simply mapping them into numbers. Specifically, we model this task as a video-text matching problem within a multimodal learning framework, which strengthens the video representation with more semantic language supervision and enables our model to do zero-shot action recognition without any further labeled data or parameters requirements. Moreover, to handle the deficiency of label texts and make use of tremendous web data, we propose a new paradigm based on this multimodal learning framework for action recognition, which we dub "pre-train, prompt and fine-tune". This paradigm first learns powerful representations from pre-training on a large amount of web image-text or video-text data. Then it makes the action recognition task to act more like pre-training problems via prompt engineering. Finally, it end-to-end fine-tunes on target datasets to obtain strong performance. We give an instantiation of the new paradigm, ActionCLIP, which not only has superior and flexible zero-shot/few-shot transfer ability but also reaches a top performance on general action recognition task, achieving 83.8% top-1 accuracy on Kinetics-400 with a ViT-B/16 as the backbone.

<img src="https://github-production-user-asset-6210df.s3.amazonaws.com/58767402/237413093-75d76018-0521-4642-af68-32141fb4fed1.png" width="800"/>
</div>

## Usage

Please refer to [Installation](https://mmaction2.readthedocs.io/en/latest/get_started/installation.html) to install MMAction2. Run the following command to install `clip`.

```shell
```

Assume that you are located at `$MMACTION2/projects/actionclip`.

```shell
```

```

```shell
```

```bash
mim test mmaction configs/actionclip_vit-base-p32-res224-clip-pre_1x1x8_k400-rgb.py --checkpoint $CHECKPOINT
```

```bash
mim test mmaction configs/actionclip_vit-base-p32-res224-clip-pre_1x1x8_k400-rgb.py --checkpoint $CHECKPOINT --launcher pytorch --gpus 8
```

```bash
mim test mmaction configs/actionclip_vit-base-p32-res224-clip-pre_1x1x8_k400-rgb.py --checkpoint $CHECKPOINT --launcher slurm \

```

```bash
```

## Results

### Kinetics400

| frame sampling strategy | backbone | top1 acc | top5 acc |  testing protocol  |                                config                                |                                ckpt                                 |
| :---------------------: | :------: | :------: | :------: | :----------------: | :------------------------------------------------------------------: | :-----------------------------------------------------------------: |

|          1x1x8          | ViT-B/32 |   77.6   |   93.8   | 8 clips  x 1 crop  | [config](./configs/actionclip_vit-base-p32-res224-clip-pre_1x1x8_k400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/projects/actionclip/actionclip_vit-base-p32-res224-clip-pre_1x1x8_k400-rgb/vit-b-32-8f.pth)\[1\] |
|          1x1x8          | ViT-B/16 |   80.3   |   95.2   | 8 clips  x 1 crop  | [config](./configs/actionclip_vit-base-p16-res224-clip-pre_1x1x8_k400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/projects/actionclip/actionclip_vit-base-p16-res224-clip-pre_1x1x8_k400-rgb/vit-b-16-8f.pth)\[1\] |
|         1x1x16          | ViT-B/16 |   81.1   |   95.6   | 16 clips  x 1 crop | [config](./configs/actionclip_vit-base-p16-res224-clip-pre_1x1x16_k400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/projects/actionclip/actionclip_vit-base-p16-res224-clip-pre_1x1x16_k400-rgb/vit-b-16-16f.pth)\[1\] |

|         1x1x32          | ViT-B/16 |   81.3   |   95.8   | 32 clips  x 1 crop | [config](./configs/actionclip_vit-base-p16-res224-clip-pre_1x1x32_k400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/projects/actionclip/actionclip_vit-base-p16-res224-clip-pre_1x1x32_k400-rgb/vit-b-16-32f.pth)\[1\] |

\[1\] The models are ported from the repo [ActionCLIP](https://github.com/sallymmx/ActionCLIP) and tested on our data. Currently, we only support the testing of ActionCLIP models. Due to the variation in testing data, our reported test accuracy differs from that of the original repository (on average, it is lower by one point). Please refer to this [issue](https://github.com/sallymmx/ActionCLIP/issues/14) for more details.

## Zero-Shot Prediction

We offer two methods for zero-shot prediction as follows. The `test.mp4` can be downloaded from [here](https://github-production-user-asset-6210df.s3.amazonaws.com/58767402/237333525-89ebee9a-573e-4e27-9047-0ad6422fa82f.mp4).

### Using Naive Pytorch

```python
import clip
from models.load import init_actionclip

register_all_modules(True)

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = init_actionclip('ViT-B/32-8', device=device)

video_anno = dict(filename='test.mp4', start_index=0)
video = preprocess(video_anno).unsqueeze(0).to(device)

template = 'The woman is {}'
labels = ['singing', 'dancing', 'performing']
text = clip.tokenize([template.format(label) for label in labels]).to(device)

    video_features = model.encode_video(video)
    text_features = model.encode_text(text)

video_features /= video_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
similarity = (100 * video_features @ text_features.T).softmax(dim=-1)
probs = similarity.cpu().numpy()

print("Label probs:", probs)  # [[9.995e-01 5.364e-07 6.666e-04]]
```

### Using MMAction2 APIs

```

```python

config_path = 'configs/actionclip_vit-base-p32-res224-clip-pre_1x1x8_k400-rgb.py'
checkpoint_path = 'https://download.openmmlab.com/mmaction/v1.0/projects/actionclip/actionclip_vit-base-p32-res224-clip-pre_1x1x8_k400-rgb/vit-b-32-8f.pth'

# Update the labels, the default is the label list of K400.
config = mmengine.Config.fromfile(config_path)
config.model.labels_or_label_file = labels
config.model.template = template

model = init_recognizer(config=config, checkpoint=checkpoint_path, device=device)

pred_result = inference_recognizer(model, 'test.mp4')
probs = pred_result.pred_scores.item.cpu().numpy()
print("Label probs:", probs)  # [9.995e-01 5.364e-07 6.666e-04]
```

```bibtex
@article{wang2021actionclip,
  title={Actionclip: A new paradigm for video action recognition},
  author={Wang, Mengmeng and Xing, Jiazheng and Liu, Yong},
  journal={arXiv preprint arXiv:2109.08472},
}
```

# Brain CT Images with Intracranial Hemorrhage Masks (Cranium)

This project supports **`Brain CT Images with Intracranial Hemorrhage Masks (Cranium)`**, which can be downloaded from [here](https://www.kaggle.com/datasets/vbookshelf/computed-tomography-ct-images).

This dataset consists of head CT (Computed Thomography) images in jpg format. There are 2500 brain window images and 2500 bone window images, for 82 patients. There are approximately 30 image slices per patient. 318 images have associated intracranial image masks. Also included are csv files containing hemorrhage diagnosis data and patient data.
This is version 1.0.0 of this dataset. A full description of this dataset as well as updated versions can be found here:
https://physionet.org/content/ct-ich/1.0.0/

| Dataset Name                                                                        | Anatomical Region | Task Type    | Modality | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License                                                   |
| ----------------------------------------------------------------------------------- | ----------------- | ------------ | -------- | ------------ | --------------------- | ---------------------- | ------------ | --------------------------------------------------------- |

| [Cranium](https://www.kaggle.com/datasets/vbookshelf/computed-tomography-ct-images) | head_and_neck     | segmentation | ct       | 2            | 2501/-/-              | yes/-/-                | 2020         | [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/) |

| background |    2501    |   99.93    |    -     |    -     |     -     |     -     |
| hemorrhage |    318     |    0.07    |    -     |    -     |     -     |     -     |

Note:

![cranium](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/ct/cranium/cranium_dataset.png?raw=true)

```
@article{hssayeni2020computed,

```
	title={Computed tomography images for intracranial hemorrhage detection and segmentation},
	author={Hssayeni, Murtadha and Croock, MS and Salman, AD and Al-khafaji, HF and Yahya, ZA and Ghoraani, B},
	journal={Intracranial Hemorrhage Segmentation Using A Deep Convolutional Model. Data},
	volume={5},
	pages={179},
}
```

All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `cranium/` root directory, run the following line to add the current directory to `PYTHONPATH`:

```shell
```

- download dataset from [here](https://www.kaggle.com/datasets/vbookshelf/computed-tomography-ct-images) and decompress data to path `'data/'`.

```none
  ├── mmseg
  │   │   │   ├── ct
  │   │   │   │   ├── cranium

```

| background |    2000    |   99.93    |   501    |  99.92   |     -     |     -     |
| hemorrhage |    260     |    0.07    |   260    |   0.08   |     -     |     -     |

```shell
```

```shell
```

```shell
```

The code under this folder is from the official [ActivityNet repo](https://github.com/activitynet/ActivityNet).
Some unused codes are removed to minimize the length of codes added.

# Benchmarks

Here we benchmark the training and testing speed of models in MMDetection3D,
with some other open source 3D detection codebases.

## Settings

- Hardwares: 8 NVIDIA Tesla V100 (32G) GPUs, Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
- Software: Python 3.7, CUDA 10.1, cuDNN 7.6.5, PyTorch 1.3, numba 0.48.0.
- Model: Since all the other codebases implements different models, we compare the corresponding models including SECOND, PointPillars, Part-A2, and VoteNet with them separately.
- Metrics: We use the average throughput in iterations of the entire training run and skip the first 50 iterations of each epoch to skip GPU warmup time.

## Main Results

We compare the training speed (samples/s) with other codebases if they implement the similar models. The results are as below, the greater the numbers in the table, the faster of the training process. The models that are not supported by other codebases are marked by `×`.

|       Methods       | MMDetection3D | OpenPCDet | votenet | Det3D |

## Details of Comparison

### Modification for Calculating Speed

- __MMDetection3D__: We try to use as similar settings as those of other codebases as possible using [benchmark configs](https://github.com/open-mmlab/MMDetection3D/blob/main/configs/benchmark).

- __Det3D__: For comparison with Det3D, we use the commit [519251e](https://github.com/poodarchu/Det3D/tree/519251e72a5c1fdd58972eabeac67808676b9bb7).

- __OpenPCDet__: For comparison with OpenPCDet, we use the commit [b32fbddb](https://github.com/open-mmlab/OpenPCDet/tree/b32fbddbe06183507bad433ed99b407cbc2175c2).

  For training speed, we add code to record the running time in the file `./tools/train_utils/train_utils.py`. We calculate the speed of each epoch, and report the average speed of all the epochs.

  <details>
    <summary>
    (diff to make it use the same method for benchmarking speed - click to expand)

  ```diff
   import os

+++ b/tools/train_utils/train_utils.py
   import os

           try:

for key, val in tb_dict.items():

total_it_each_epoch = len(train_loader)
               else:
               )
  -

dataloader_iter=dataloader_iter
               )
  -
                   )

  ```

- __MMDetection3D__: With release v0.1.0, run

  ```bash
  ```

- __votenet__: At commit [2f6d6d3](https://github.com/facebookresearch/votenet/tree/2f6d6d36ff98d96901182e935afe48ccee82d566), run

  ```bash

```

  ```bash
  ```

  Then benchmark the test speed by running

  ```bash
  ```

  Note that eval.py is modified to compute inference time.

  <details>
  <summary>
  (diff to benchmark the similar models - click to expand)

  ```diff
     import sys

@@ -10,6 +10,7 @@ import os
     import sys

parser.add_argument('--vote_factor', type=int, default=1, help='Number of votes generated from each seed [default: 1]')

@@ -144,6 +146,8 @@ def evaluate_one_epoch():

    +

  ```

ap_calculator.step(batch_pred_map_cls, batch_gt_map_cls)

  ```

  ```bash
  ```

- __Det3D__: At commit [519251e](https://github.com/poodarchu/Det3D/tree/519251e72a5c1fdd58972eabeac67808676b9bb7), use `kitti_point_pillars_mghead_syncbn.py` and run

  ```bash
  ```

  Note that the config in train.sh is modified to train point pillars.

  <details>
  <summary>

  ```diff
   fi

   # Voxelnet

index 3a93f95..461e0ea 100755
   fi

   # Voxelnet

# PointPillars
  ```

  ```bash
  ```

- __OpenPCDet__: At commit [b32fbddb](https://github.com/open-mmlab/OpenPCDet/tree/b32fbddbe06183507bad433ed99b407cbc2175c2), run

  ```bash
  cd tools
  ```

### SECOND

```bash
  cd tools
  ```

### SECOND

For SECOND, we mean the [SECONDv1.5](https://github.com/traveller59/second.pytorch/blob/master/second/configs/all.fhd.config) that was first implemented in [second.Pytorch](https://github.com/traveller59/second.pytorch). Det3D's implementation of SECOND uses its self-implemented Multi-Group Head, so its speed is not compatible with other codebases.

  ```bash
  ```

  ```bash
  cd tools

```

  ```bash
  cd tools
  ```

  ```bash
  ```

- __OpenPCDet__: At commit [b32fbddb](https://github.com/open-mmlab/OpenPCDet/tree/b32fbddbe06183507bad433ed99b407cbc2175c2), train the model by running

  ```bash
  cd tools
  ```

# Preparing UCF-101

```BibTeX
}
```

For basic dataset information, you can refer to the dataset [website](https://www.crcv.ucf.edu/research/data-sets/ucf101/).
Before we start, please make sure that the directory is located at `$MMACTION2/tools/data/ucf101/`.

```shell
```

```shell
```

```

```shell
```

```
```

If you have plenty of SSD space, then we recommend extracting frames there for better I/O performance. The extracted frames (RGB + Flow) will take up about 100GB.

```shell
```

```shell
```

```shell
```

```shell
```

If Optical Flow is also required, run the following script to extract flow using "tvl1" algorithm.

```shell
```

```shell
```

```shell
```

After the whole data process for UCF-101 preparation,
you will get the rawframes (RGB + Flow), videos and annotation files for UCF-101.

In the context of the whole project (for UCF-101 only), the folder structure will look like:

```
mmaction2
├── tools
├── data

│   │   │   ├── ...

```

For training and evaluating on UCF-101, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).

# 关键信息提取

```{note}
我们正努力往 [Dataset Preparer](./dataset_preparer.md) 中增加更多数据集。对于 [Dataset Preparer](./dataset_preparer.md) 暂未能完整支持的数据集，本页提供了一系列手动下载的步骤，供有需要的用户使用。
```

## 概览

关键信息提取任务的数据集，文件目录应按如下配置：

```text
```

## 准备步骤

- 下载并解压 [wildreceipt.tar](https://download.openmmlab.com/mmocr/data/wildreceipt.tar)

- 准备好 [WildReceipt](#WildReceipt)。
- 转换 WildReceipt 成 OpenSet 格式:

```bash
# 你可以运行以下命令以获取更多可用参数：
```

```{note}
[这篇教程](../tutorials/kie_closeset_openset.md)里讲述了更多 CloseSet 和 OpenSet 数据格式之间的区别。
```

# 常见问题解答（FAQ）

我们在这里列出了使用时的一些常见问题及其相应的解决方案。 如果您发现有一些问题被遗漏，请随时提 PR 丰富这个列表。 如果您无法在此获得帮助，请使用 [issue 模板](https://github.com/open-mmlab/mmsegmentation/blob/dev-1.x/.github/ISSUE_TEMPLATE/error-report.md/)创建问题，但是请在模板中填写所有必填信息，这有助于我们更快定位问题。

## 安装

兼容的 MMSegmentation 和 MMCV 版本如下。请安装正确版本的 MMCV 以避免安装问题。

如果您已经安装了版本不合适的 mmcv，请先运行`pip uninstall mmcv`卸载已安装的 mmcv，如您先前安装的为 mmcv-full（存在于 OpenMMLab 1.x），请运行`pip uninstall mmcv-full`进行卸载。

- 如出现 "No module named 'mmcv'"
  1. 使用`pip uninstall mmcv`卸载环境中现有的 mmcv
  2. 按照[安装说明](../get_started.md)安装对应的 mmcv

## 如何获知模型训练时需要的显卡数量

- 看模型的 config 文件命名。可以参考[了解配置文件](../user_guides/1_config.md)中的`配置文件命名风格`部分。比如，对于名字为`segformer_mit-b0_8xb1-160k_cityscapes-1024x1024.py`的 config 文件，`8xb1`代表训练其对应的模型需要的卡数为 8，每张卡中的 batch size 为 1。
- 看模型的 log 文件。点开该模型的 log 文件，并在其中搜索`nGPU`，在`nGPU`后的数字个数即训练时所需的卡数。比如，在 log 文件中搜索`nGPU`得到`nGPU 0,1,2,3,4,5,6,7`的记录，则说明训练该模型需要使用八张卡。

## auxiliary head 是什么

简单来说，这是一个提高准确率的深度监督技术。在训练阶段，`decode_head`用于输出语义分割的结果，`auxiliary_head` 只是增加了一个辅助损失，其产生的分割结果对你的模型结果没有影响，仅在在训练中起作用。您可以阅读这篇[论文](https://arxiv.org/pdf/1612.01105.pdf)了解更多信息。

## 运行测试脚本时如何输出绘制分割掩膜的图像

在测试脚本中，我们提供了`--out`参数来控制是否输出保存预测的分割掩膜图像。您可以运行以下命令输出测试结果：

```shell
```

更多用例细节可查阅[文档](https://github.com/open-mmlab/mmsegmentation/blob/dev-1.x/docs/zh_cn/user_guides/4_train_test.md#%E6%B5%8B%E8%AF%95%E5%B9%B6%E4%BF%9D%E5%AD%98%E5%88%86%E5%89%B2%E7%BB%93%E6%9E%9C)，[PR #2712](https://github.com/open-mmlab/mmsegmentation/pull/2712) 以及[迁移文档](https://github.com/open-mmlab/mmsegmentation/blob/dev-1.x/docs/zh_cn/migration/interface.md#%E6%B5%8B%E8%AF%95%E5%90%AF%E5%8A%A8)了解相关说明。

## 如何处理二值分割任务?

MMSegmentation 使用 `num_classes` 和 `out_channels` 来控制模型最后一层 `self.conv_seg` 的输出。更多细节可以参考 [这里](https://github.com/open-mmlab/mmsegmentation/blob/dev-1.x/mmseg/models/decode_heads/decode_head.py)。

`num_classes` 应该和数据集本身类别个数一致，当是二值分割时，数据集只有前景和背景两类，所以 `num_classes` 为 2. `out_channels` 控制模型最后一层的输出的通道数，通常和 `num_classes` 相等，但当二值分割时候，可以有两种处理方法, 分别是：

- 设置 `out_channels=2`，在训练时以 Cross Entropy Loss 作为损失函数，在推理时使用 `F.softmax()` 归一化 logits 值，然后通过 `argmax()` 得到每个像素的预测结果。

- 设置 `out_channels=1`，在训练时以 Binary Cross Entropy Loss 作为损失函数，在推理时使用 `F.sigmoid()` 和 `threshold` 得到预测结果，`threshold` 默认为 0.3。

对于实现上述两种计算二值分割的方法，需要在 `decode_head` 和 `auxiliary_head` 的配置里修改。下面是对样例 [pspnet_unet_s5-d16.py](https://github.com/open-mmlab/mmsegmentation/blob/dev-1.x/configs/_base_/models/pspnet_unet_s5-d16.py) 做出的对应修改。

- (1) `num_classes=2`, `out_channels=2` 并在 `CrossEntropyLoss` 里面设置 `use_sigmoid=False`。

```python

```python
```

- (2) `num_classes=2`, `out_channels=1` 并在 `CrossEntropyLoss` 里面设置 `use_sigmoid=True`.

```python

```

## `reduce_zero_label` 的作用

数据集中 `reduce_zero_label` 参数类型为布尔类型，默认为 False，它的功能是为了忽略数据集 label 0。具体做法是将 label 0 改为 255，其余 label 相应编号减 1，同时 decode head 里将 255 设为 ignore index，即不参与 loss 计算。
以下是 `reduce_zero_label` 具体实现逻辑:

```python
```

关于您的数据集是否需要使用 reduce_zero_label，有以下两类情况：

```

- 例如在 [Potsdam](https://github.com/open-mmlab/mmsegmentation/blob/1.x/docs/en/user_guides/2_dataset_prepare.md#isprs-potsdam) 数据集上，有 0-不透水面、1-建筑、2-低矮植被、3-树、4-汽车、5-杂乱，六类。但该数据集提供了两种 RGB 标签，一种为图像边缘处有黑色像素的标签，另一种是没有黑色边缘的标签。对于有黑色边缘的标签，在 [dataset_converters.py](https://github.com/open-mmlab/mmsegmentation/blob/dev-1.x/tools/dataset_converters/potsdam.py)中，其将黑色边缘转换为 label 0，其余标签分别为 1-不透水面、2-建筑、3-低矮植被、4-树、5-汽车、6-杂乱，那么此时，就应该在数据集 [potsdam.py](https://github.com/open-mmlab/mmsegmentation/blob/ff95416c3b5ce8d62b9289f743531398efce534f/mmseg/datasets/potsdam.py#L23) 中将`reduce_zero_label=True`。如果使用的是没有黑色边缘的标签，那么 mask label 中只有 0-5，此时就应该使`reduce_zero_label=False`。需要结合您的实际情况来使用。
- 例如在第 0 类为 background 类别的数据集上，如果您最终是需要将背景和您的其余类别分开时，是不需要使用`reduce_zero_label`的，此时在数据集中应该将其设置为`reduce_zero_label=False`

**注意:** 使用 `reduce_zero_label` 请确认数据集原始类别个数，如果只有两类，需要关闭 `reduce_zero_label` 即设置 `reduce_zero_label=False`。

# 300 faces in-the-wild challenge: Database and results

<details>

```bibtex
}
```

# 基于 MMDetection 的项目

有许多开源项目都是基于 MMDetection 搭建的，我们在这里列举一部分作为样例，展示如何基于 MMDetection 搭建您自己的项目。
由于这个页面列举的项目并不完全，我们欢迎社区提交 Pull Request 来更新这个文档。

## MMDetection 的拓展项目

一些项目拓展了 MMDetection 的边界，如将 MMDetection 拓展支持 3D 检测或者将 MMDetection 用于部署。
它们展示了 MMDetection 的许多可能性，所以我们在这里也列举一些。

## 研究项目

同样有许多研究论文是基于 MMDetection 进行的。许多论文都发表在了顶级的会议或期刊上，或者对社区产生了深远的影响。
为了向社区提供一个可以参考的论文列表，帮助大家开发或者比较新的前沿算法，我们在这里也遵循会议的时间顺序列举了一些论文。
MMDetection 中已经支持的算法不在此列。

# Colorectal Nuclear Segmentation and Phenotypes (CoNSeP) Dataset

This project supports **`Colorectal Nuclear Segmentation and Phenotypes (CoNSeP) Dataset`**, which can be downloaded from [here](https://warwick.ac.uk/fac/cross_fac/tia/data/hovernet/).

The CoNSeP (Colon Segmentation and Phenotyping) dataset consists of 41 H&E stained image tiles, each with a size of 1,000×1,000 pixels and a magnification of 40x. These images were extracted from 16 colorectal adenocarcinoma (CRA) whole slide images (WSI), each of which belonged to a separate patient and was scanned using an Omnyx VL120 scanner at the Pathology Department of the University Hospitals Coventry and Warwickshire NHS Trust, UK. This dataset was first used in  paper named, "HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images".

| Dataset name                                             | Anatomical region | Task type    | Modality       | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License |
| -------------------------------------------------------- | ----------------- | ------------ | -------------- | ------------ | --------------------- | ---------------------- | ------------ | ------- |
| [CoNIC202](https://conic-challenge.grand-challenge.org/) | abdomen           | segmentation | histopathology | 7            | 4981/-/-              | yes/-/-                | 2022         | -       |

|           Class Name            | Num. Train | Pct. Train | Num. Val | Pct. Val | Num. Test | Pct. Test |
| :-----------------------------: | :--------: | :--------: | :------: | :------: | :-------: | :-------: |
|           background            |     27     |   83.61    |    14    |   80.4   |     -     |     -     |

|              other              |     17     |    0.17    |    9     |   0.52   |     -     |     -     |
|          inflammatory           |     25     |    2.66    |    14    |   2.14   |     -     |     -     |
|       healthy epithelial        |     3      |    1.47    |    2     |   1.58   |     -     |     -     |
| dysplastic/malignant epithelial |     10     |    7.17    |    8     |   9.16   |     -     |     -     |
|           fibroblast            |     23     |    3.84    |    14    |   4.63   |     -     |     -     |
|             muscle              |     8      |    1.05    |    3     |   1.42   |     -     |     -     |
|           endothelial           |     7      |    0.02    |    4     |   0.15   |     -     |     -     |

Note:

Note:

![bac](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/histopathology/consep/consep_dataset.png)

```shell
```

- download dataset from [here](https://opendatalab.com/CoNSeP) and decompress data to path `'data/'`.

```shell
```

```none
  ├── mmseg
  │   │   │   │   ├── consep

```

```shell
```

```shell
```

```

```
@article{graham2019hover,
  title={Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images},
  author={Graham, Simon and Vu, Quoc Dang and Raza, Shan E Ahmed and Azam, Ayesha and Tsang, Yee Wah and Kwak, Jin Tae and Rajpoot, Nasir},
  journal={Medical Image Analysis},
  volume={58},
  pages={101563},
}
```

volume={58},
}
```

# 量化测试结果

目前 mmdeploy 支持 ncnn 量化

## ncnn 量化

### 分类任务

|                                                        model                                                         |   dataset   | fp32 top-1 (%) | int8 top-1 (%) |
| :------------------------------------------------------------------------------------------------------------------: | :---------: | :------------: | :------------: |
|       [ResNet-18](https://github.com/open-mmlab/mmpretrain/blob/main/configs/resnet/resnet18_8xb16_cifar10.py)       |   Cifar10   |     94.82      |     94.83      |
| [ResNeXt-32x4d-50](https://github.com/open-mmlab/mmpretrain/blob/main/configs/resnext/resnext50-32x4d_8xb32_in1k.py) | ImageNet-1k |     77.90      |    78.20\*     |
|  [MobileNet V2](https://github.com/open-mmlab/mmpretrain/blob/main/configs/mobilenet_v2/mobilenet-v2_8xb32_in1k.py)  | ImageNet-1k |     71.86      |    71.43\*     |

|       [HRNet-W18\*](https://github.com/open-mmlab/mmpretrain/blob/main/configs/hrnet/hrnet-w18_4xb32_in1k.py)        | ImageNet-1k |     76.75      |    76.25\*     |

备注：

- 因为 imagenet-1k 数据量很大、ncnn 未正式发布 Vulkan int8 版本，考虑到 CPU 运行时间，仅用部分测试集（4000/50000）
- 量化后精度会有差异，分类模型涨点 1% 以内是正常情况

### OCR 检测任务

|                                                               model                                                               |  dataset  | fp32 hmean |   int8 hmean   |
| :-------------------------------------------------------------------------------------------------------------------------------: | :-------: | :--------: | :------------: |
|      [PANet](https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/panet/panet_resnet18_fpem-ffm_600e_icdar2015.py)       | ICDAR2015 |   0.795    | 0.792 @thr=0.9 |

| [TextSnake](https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/textsnake/textsnake_resnet50_fpn-unet_1200e_ctw1500.py) |  CTW1500  |   0.817    |     0.818      |

备注：[mmocr](https://github.com/open-mmlab/mmocr) 使用 `shapely` 计算 IoU，实现方法会导致轻微的精度差异

### 姿态检测任务

|                                                                         model                                                                          | dataset  | fp32 AP | int8 AP |
| :----------------------------------------------------------------------------------------------------------------------------------------------------: | :------: | :-----: | :-----: |

| [Hourglass](https://github.com/open-mmlab/mmpose/blob/main/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hourglass52_8xb32-210e_coco-256x256.py) | COCO2017 |  0.717  |  0.713  |

备注：测试转换后的模型精度时，对于 mmpose 模型，在模型配置文件中 `flip_test` 需设置为 `False`。

# Preparing Vimeo90K-triplet Dataset

```bibtex
}
```

The training and test datasets can be download from [here](http://toflow.csail.mit.edu/).

The Vimeo90K-triplet  dataset has a `clip/sequence/img` folder structure:

```text
mmagic
├── mmagic
├── tools
├── data
│   │   │   ├── ...
```

# BLIP

> [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://arxiv.org/abs/2201.12086)

Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to video-language tasks in a zero-shot manner.

<img src="https://user-images.githubusercontent.com/26739999/236374275-94d2f94b-d9a7-4f12-b694-f15a2be00be6.png" width="90%"/>
</div>

```python

result = inference_model('blip-base_3rdparty_caption', 'demo/cat-dog.png')
# {'pred_caption': 'a puppy and a cat sitting on a blanket'}
```

Test:

```shell
python tools/test.py configs/blip/blip-base_8xb32_caption.py https://download.openmmlab.com/mmclassification/v1/blip/blip-base_3rdparty_coco-caption_20230419-a5b71af3.pth
```

```

| Model                          | Params (M) | BLEU-4 | CIDER  |                 Config                 |                                                    Download                                                    |
| :----------------------------- | :--------: | :----: | :----: | :------------------------------------: | :------------------------------------------------------------------------------------------------------------: |
| `blip-base_3rdparty_caption`\* |   223.97   | 40.12  | 132.82 | [config](./blip-base_8xb32_caption.py) | [model](https://download.openmmlab.com/mmclassification/v1/blip/blip-base_3rdparty_coco-caption_20230419-a5b71af3.pth) |

### Image Caption on NoCaps

| Model                          | Params (M) | SPICE | CIDER  |                Config                 |                                                     Download                                                     |

| :----------------------------- | :--------: | :---: | :----: | :-----------------------------------: | :--------------------------------------------------------------------------------------------------------------: |
| `blip-base_3rdparty_caption`\* |   223.97   | 14.69 | 109.12 | [config](./blip-base_8xb32_nocaps.py) | [model](https://download.openmmlab.com/mmclassification/v1/blip/blip-base_3rdparty_coco-caption_20230419-a5b71af3.pth) |

### Image Caption on Flickr30k

| Model                          | Params (M) | SPICE | CIDER |                      Config                      |                                                Download                                                |
| :----------------------------- | :--------: | :---: | :---: | :----------------------------------------------: | :----------------------------------------------------------------------------------------------------: |

| `blip-base_3rdparty_caption`\* |   223.97   | 15.58 | 68.89 | [config](./blip-base_8xb32_caption_flickr30k.py) | [model](https://download.openmmlab.com/mmclassification/v1/blip/blip-base_3rdparty_coco-caption_20230419-a5b71af3.pth) |

| Model                     | Params (M) | Accuracy (testA) | Accuracy (testB) |                Config                |                                             Download                                              |
| :------------------------ | :--------: | :--------------: | :--------------: | :----------------------------------: | :-----------------------------------------------------------------------------------------------: |

| `blip-base_8xb16_refcoco` |   498.49   |      86.14       |      77.33       | [config](blip-base_8xb16_refcoco.py) | [model](https://download.openmmlab.com/mmclassification/v1/blip/blip-base_8xb16_refcoco_20230508-d2d10f4c.pth) \| [log](https://download.openmmlab.com/mmclassification/v1/blip/blip-base_8xb16_refcoco_20230508-d2d10f4c.json) |

| Model                      | Params (M) | Accuracy |               Config               |                                                       Download                                                        |
| :------------------------- | :--------: | :------: | :--------------------------------: | :-------------------------------------------------------------------------------------------------------------------: |

| `blip-base_3rdparty_vqa`\* |   361.48   |  78.20   | [config](./blip-base_8xb32_vqa.py) | [model](https://download.openmmlab.com/mmclassification/v1/blip/blip-base_3rdparty-capflit_vqa_20230505-81488941.pth) |

### Visual Question Answering on OK-VQA

| Model                      | Params (M) | Accuracy |                Config                |                                                       Download                                                        |
| :------------------------- | :--------: | :------: | :----------------------------------: | :-------------------------------------------------------------------------------------------------------------------: |
| `blip-base_3rdparty_vqa`\* |   361.48   |  40.59#  | [config](./blip-base_8xb32_okvqa.py) | [model](https://download.openmmlab.com/mmclassification/v1/blip/blip-base_3rdparty-capflit_vqa_20230505-81488941.pth) |

### Visual Question Answering on OCR-VQA

| Model                      | Params (M) | Accuracy |                Config                 |                                                       Download                                                        |
| :------------------------- | :--------: | :------: | :-----------------------------------: | :-------------------------------------------------------------------------------------------------------------------: |
| `blip-base_3rdparty_vqa`\* |   361.48   |  28.30#  | [config](./blip-base_8xb32_ocrvqa.py) | [model](https://download.openmmlab.com/mmclassification/v1/blip/blip-base_3rdparty-capflit_vqa_20230505-81488941.pth) |

| Model                            | Params (M) | Recall@1 | Recall@5 |                  Config                  |                                                Download                                                |

| :------------------------------- | :--------: | :------: | :------: | :--------------------------------------: | :----------------------------------------------------------------------------------------------------: |
| `blip-base_3rdparty_retrieval`\* |   447.49   |  82.52   |  95.34   | [config](./blip-base_8xb32_retrieval.py) | [model](https://download.openmmlab.com/mmclassification/v1/blip/blip-base_3rdparty_coco-retrieval_20230419-a1804d2c.pth) |

### Text-To-Image Retrieval on COCO

| `blip-base_3rdparty_retrieval`\* |   447.49   |  64.82   |  86.28   | [config](./blip-base_8xb32_retrieval.py) | [model](https://download.openmmlab.com/mmclassification/v1/blip/blip-base_3rdparty_coco-retrieval_20230419-a1804d2c.pth) |

### Image-To-Text Retrieval on Flickr30k

| Model                            | Params (M) | Recall@1 | Recall@5 |                       Config                       |                                           Download                                           |
| :------------------------------- | :--------: | :------: | :------: | :------------------------------------------------: | :------------------------------------------------------------------------------------------: |
| `blip-base_3rdparty_retrieval`\* |   447.49   |  95.10#  |  99.60#  | [config](./blip-base_8xb32_retrieval_flickr30k.py) | [model](https://download.openmmlab.com/mmclassification/v1/blip/blip-base_3rdparty_coco-retrieval_20230419-a1804d2c.pth) |

### Text-To-Image Retrieval on Flickr30k

| `blip-base_3rdparty_retrieval`\* |   447.49   |  85.26#  |  96.58#  | [config](./blip-base_8xb32_retrieval_flickr30k.py) | [model](https://download.openmmlab.com/mmclassification/v1/blip/blip-base_3rdparty_coco-retrieval_20230419-a1804d2c.pth) |

### NLVR on NLVR2

| Model                       | Params (M) | Top-1 (%) |               Config                |                                                    Download                                                    |

| :-------------------------- | :--------: | :-------: | :---------------------------------: | :------------------------------------------------------------------------------------------------------------: |
| `blip-base_3rdparty_nlvr`\* |   259.37   |   82.33   | [config](./blip-base_8xb32_nlvr.py) | [model](https://download.openmmlab.com/mmclassification/v1/blip/blip-base_3rdparty_nlvr_20230427-3b14d33f.pth) |

*Results with # denote zero-shot evaluation. The corresponding model hasn't been finetuned on that dataset.*

```bibtex
@inproceedings{li2022blip,
      title={BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation},
      author={Junnan Li and Dongxu Li and Caiming Xiong and Steven Hoi},

author={Junnan Li and Dongxu Li and Caiming Xiong and Steven Hoi},
      booktitle={ICML},
}
```

# ISANet

> [Interlaced Sparse Self-Attention for Semantic Segmentation](https://arxiv.org/abs/1907.12273)

<a href="https://github.com/openseg-group/openseg.pytorch">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.18.0/mmseg/models/decode_heads/isa_head.py#L58">Code Snippet</a>

In this paper, we present a so-called interlaced sparse self-attention approach to improve the efficiency of the \\emph{self-attention} mechanism for semantic segmentation. The main idea is that we factorize the dense affinity matrix as the product of two sparse affinity matrices. There are two successive attention modules each estimating a sparse affinity matrix. The first attention module is used to estimate the affinities within a subset of positions that have long spatial interval distances and the second attention module is used to estimate the affinities within a subset of positions that have short spatial interval distances. These two attention modules are designed so that each position is able to receive the information from all the other positions. In contrast to the original self-attention module, our approach decreases the computation and memory complexity substantially especially when processing high-resolution feature maps. We empirically verify the effectiveness of our approach on six challenging semantic segmentation benchmarks.

<img src="https://user-images.githubusercontent.com/24582831/142901868-03d80da4-b9c0-4df9-8509-5f684ba9dadc.png" width="80%"/>
</div>

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device | mIoU  | mIoU(ms+flip) |                                                                                                                         config | download                                                                                                                                                                                                                                                                                                                                                 |

| ISANet | R-50-D8  | 512x1024  | 40000   |    5.869 | 2.91           | V100   | 78.49 |         79.44 |  [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/isanet/isanet_r50-d8_4xb2-40k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r50-d8_512x1024_40k_cityscapes/isanet_r50-d8_512x1024_40k_cityscapes_20210901_054739-981bd763.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r50-d8_512x1024_40k_cityscapes/isanet_r50-d8_512x1024_40k_cityscapes_20210901_054739.log.json)     |

| ISANet | R-50-D8  | 512x1024  | 80000   |    5.869 | 2.91           | V100   | 78.68 |         80.25 |  [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/isanet/isanet_r50-d8_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r50-d8_512x1024_80k_cityscapes/isanet_r50-d8_512x1024_80k_cityscapes_20210901_074202-89384497.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r50-d8_512x1024_80k_cityscapes/isanet_r50-d8_512x1024_80k_cityscapes_20210901_074202.log.json)     |

| ISANet | R-50-D8  | 769x769   | 40000   |    6.759 | 1.54           | V100   | 78.70 |         80.28 |   [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/isanet/isanet_r50-d8_4xb2-40k_cityscapes-769x769.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r50-d8_769x769_40k_cityscapes/isanet_r50-d8_769x769_40k_cityscapes_20210903_050200-4ae7e65b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r50-d8_769x769_40k_cityscapes/isanet_r50-d8_769x769_40k_cityscapes_20210903_050200.log.json)         |

| ISANet | R-50-D8  | 769x769   | 80000   |    6.759 | 1.54           | V100   | 79.29 |         80.53 |   [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/isanet/isanet_r50-d8_4xb2-80k_cityscapes-769x769.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r50-d8_769x769_80k_cityscapes/isanet_r50-d8_769x769_80k_cityscapes_20210903_101126-99b54519.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r50-d8_769x769_80k_cityscapes/isanet_r50-d8_769x769_80k_cityscapes_20210903_101126.log.json)         |

| ISANet | R-101-D8 | 512x1024  | 40000   |    9.425 | 2.35           | V100   | 79.58 |         81.05 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/isanet/isanet_r101-d8_4xb2-40k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r101-d8_512x1024_40k_cityscapes/isanet_r101-d8_512x1024_40k_cityscapes_20210901_145553-293e6bd6.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r101-d8_512x1024_40k_cityscapes/isanet_r101-d8_512x1024_40k_cityscapes_20210901_145553.log.json) |

| ISANet | R-101-D8 | 512x1024  | 80000   |    9.425 | 2.35           | V100   | 80.32 |         81.58 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/isanet/isanet_r101-d8_4xb2-80k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r101-d8_512x1024_80k_cityscapes/isanet_r101-d8_512x1024_80k_cityscapes_20210901_145243-5b99c9b2.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r101-d8_512x1024_80k_cityscapes/isanet_r101-d8_512x1024_80k_cityscapes_20210901_145243.log.json) |

| ISANet | R-101-D8 | 769x769   | 40000   |   10.815 | 0.92           | V100   | 79.68 |         80.95 |  [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/isanet/isanet_r101-d8_4xb2-40k_cityscapes-769x769.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r101-d8_769x769_40k_cityscapes/isanet_r101-d8_769x769_40k_cityscapes_20210903_111320-509e7224.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r101-d8_769x769_40k_cityscapes/isanet_r101-d8_769x769_40k_cityscapes_20210903_111320.log.json)     |

| ISANet | R-101-D8 | 769x769   | 80000   |   10.815 | 0.92           | V100   | 80.61 |         81.59 |  [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/isanet/isanet_r101-d8_4xb2-80k_cityscapes-769x769.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r101-d8_769x769_80k_cityscapes/isanet_r101-d8_769x769_80k_cityscapes_20210903_111319-24f71dfa.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r101-d8_769x769_80k_cityscapes/isanet_r101-d8_769x769_80k_cityscapes_20210903_111319.log.json)     |

### ADE20K

### ADE20K

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device | mIoU  | mIoU(ms+flip) |                                                                                                                     config | download                                                                                                                                                                                                                                                                                                                                 |

| ISANet | R-50-D8  | 512x512   | 80000   |      9.0 | 22.55          | V100   | 41.12 |         42.35 |   [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/isanet/isanet_r50-d8_4xb4-80k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r50-d8_512x512_80k_ade20k/isanet_r50-d8_512x512_80k_ade20k_20210903_124557-6ed83a0c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r50-d8_512x512_80k_ade20k/isanet_r50-d8_512x512_80k_ade20k_20210903_124557.log.json)         |

| ISANet | R-50-D8  | 512x512   | 160000  |      9.0 | 22.55          | V100   | 42.59 |         43.07 |  [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/isanet/isanet_r50-d8_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r50-d8_512x512_160k_ade20k/isanet_r50-d8_512x512_160k_ade20k_20210903_104850-f752d0a3.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r50-d8_512x512_160k_ade20k/isanet_r50-d8_512x512_160k_ade20k_20210903_104850.log.json)     |

| ISANet | R-101-D8 | 512x512   | 80000   |   12.562 | 10.56          | V100   | 43.51 |         44.38 |  [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/isanet/isanet_r101-d8_4xb4-80k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r101-d8_512x512_80k_ade20k/isanet_r101-d8_512x512_80k_ade20k_20210903_162056-68b235c2.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r101-d8_512x512_80k_ade20k/isanet_r101-d8_512x512_80k_ade20k_20210903_162056.log.json)     |

| ISANet | R-101-D8 | 512x512   | 160000  |   12.562 | 10.56          | V100   | 43.80 |          45.4 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/isanet/isanet_r101-d8_4xb4-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r101-d8_512x512_160k_ade20k/isanet_r101-d8_512x512_160k_ade20k_20210903_211431-a7879dcd.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r101-d8_512x512_160k_ade20k/isanet_r101-d8_512x512_160k_ade20k_20210903_211431.log.json) |

| Method | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device | mIoU  | mIoU(ms+flip) |                                                                                                                      config | download                                                                                                                                                                                                                                                                                                                                     |

| ISANet | R-50-D8  | 512x512   | 20000   |      5.9 | 23.08          | V100   | 76.78 |         77.79 |  [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/isanet/isanet_r50-d8_4xb4-20k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r50-d8_512x512_20k_voc12aug/isanet_r50-d8_512x512_20k_voc12aug_20210901_164838-79d59b80.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r50-d8_512x512_20k_voc12aug/isanet_r50-d8_512x512_20k_voc12aug_20210901_164838.log.json)     |

| ISANet | R-50-D8  | 512x512   | 40000   |      5.9 | 23.08          | V100   | 76.20 |         77.22 |  [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/isanet/isanet_r50-d8_4xb4-40k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r50-d8_512x512_40k_voc12aug/isanet_r50-d8_512x512_40k_voc12aug_20210901_151349-7d08a54e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r50-d8_512x512_40k_voc12aug/isanet_r50-d8_512x512_40k_voc12aug_20210901_151349.log.json)     |

| ISANet | R-101-D8 | 512x512   | 20000   |    9.465 | 7.42           | V100   | 78.46 |         79.16 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/isanet/isanet_r101-d8_4xb4-20k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r101-d8_512x512_20k_voc12aug/isanet_r101-d8_512x512_20k_voc12aug_20210901_115805-3ccbf355.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r101-d8_512x512_20k_voc12aug/isanet_r101-d8_512x512_20k_voc12aug_20210901_115805.log.json) |

| ISANet | R-101-D8 | 512x512   | 40000   |    9.465 | 7.42           | V100   | 78.12 |         79.04 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/isanet/isanet_r101-d8_4xb4-40k_voc12aug-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r101-d8_512x512_40k_voc12aug/isanet_r101-d8_512x512_40k_voc12aug_20210901_145814-bc71233b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/isanet/isanet_r101-d8_512x512_40k_voc12aug/isanet_r101-d8_512x512_40k_voc12aug_20210901_145814.log.json) |

```bibetex
@article{huang2019isa,
  title={Interlaced Sparse Self-Attention for Semantic Segmentation},
  author={Huang, Lang and Yuan, Yuhui and Guo, Jianyuan and Zhang, Chao and Chen, Xilin and Wang, Jingdong},
  journal={arXiv preprint arXiv:1907.12273},
}
```

The technical report above is also presented at:

@article{yuan2021ocnet,

journal={arXiv preprint arXiv:1907.12273},
}
```

  title={OCNet: Object Context for Semantic Segmentation},
  author={Yuan, Yuhui and Huang, Lang and Guo, Jianyuan and Zhang, Chao and Chen, Xilin and Wang, Jingdong},
  pages={1--24},
}
```

# 自定义模型

在我们的设计中，我们定义一个完整的模型为顶层模块，根据功能的不同，基本几种不同类型的模型组件组成。

- 模型：顶层模块定义了具体的任务类型，例如 `ImageClassifier` 用在图像分类任务中， `MAE` 用在自监督学习中， `ImageToImageRetriever` 用在图像检索中。
- 主干网络：通常是一个特征提取网络，涵盖了模型之间绝大多数的差异，例如 `ResNet`、`MobileNet`。
- 颈部：用于连接主干网络和头部的组件，例如 `GlobalAveragePooling`。
- 头部：用于执行特定任务的组件，例如 `ClsHead`、 `ContrastiveHead`。
- 损失函数：在头部用于计算损失函数的组件，例如 `CrossEntropyLoss`、`LabelSmoothLoss`。
- 目标生成器: 用于自监督学习任务的组件，例如 `VQKD`、 `HOGGenerator`。

## 添加新的顶层模型

通常来说，对于图像分类和图像检索任务来说，模型顶层模型流程基本一致。但是不同的自监督学习算法却用不同的计算流程，像 `MAE` 和 `BEiT` 就大不相同。 所以在这个部分，我们将简单介绍如何添加一个新的自监督学习算法。

### 添加新的自监督学习算法

1. 创建新文件 `mmpretrain/models/selfsup/new_algorithm.py` 以及实现 `NewAlgorithm`

   ```python

           pass

           pass

           pass

           pass

   ```

2. 在 `mmpretrain/models/selfsup/__init__.py` 中导入对应的新算法

   ```python
   ...

   __all__ = [
       ...,
       ...
   ]
   ```

3. 在配置文件中使用新算法

   ```python
       neck=...,
       head=...,
       ...
   )
   ```

## 添加新的主干网络

这里，我们以 `ResNet_CIFAR` 为例，展示了如何开发一个新的主干网络组件。

```python
       neck=...,
       head=...,
       ...
   )
   ```

`ResNet_CIFAR` 针对 CIFAR 32x32 的图像输入，远小于大多数模型使用的ImageNet默认的224x224输入配置，所以我们将骨干网络中 `kernel_size=7,stride=2`
的设置替换为 `kernel_size=3, stride=1`，并移除了 stem 层之后的
`MaxPooling`，以避免传递过小的特征图到残差块中。

最简单的方式就是继承自 `ResNet` 并只修改 stem 层。

1. 创建一个新文件 `mmpretrain/models/backbones/resnet_cifar.py`。

   ```python

       （对这个主干网络的简短描述）

       Args:
           ...
           （参数文档）
       """

       def __init__(self, depth, deep_stem=False, **kwargs):
           # 调用基类 ResNet 的初始化函数

...
           （参数文档）
       """

           super(ResNet_CIFAR, self).__init__(depth, deep_stem=deep_stem **kwargs)
           # 其他特殊的初始化流程

           # 重载基类的方法，以实现对网络结构的修改
               stride=1,

           # 如果需要的话，可以自定义forward方法

self.relu = nn.ReLU(inplace=True)

           outs = []
           # 输出值需要是一个包含不同层多尺度输出的元组
           # 如果不需要多尺度特征，可以直接在最终输出上包一层元组

           # 如果需要的话，可以自定义权重初始化的方法

           # 如果有预训练模型，则不需要进行权重初始化
               return

           # 通常来说，我们建议用`init_cfg`去列举不同层权重初始化方法
           # 包括卷积层，线性层，归一化层等等
           # 如果有特殊需要，可以在这里进行额外的初始化操作
           ...
   ```

```{note}

return

           ...
   ```

```{note}
在 OpenMMLab 2.0 的设计中，将原有的`BACKBONES`、`NECKS`、`HEADS`、`LOSSES`等注册名统一为`MODELS`.
```

2. 在 `mmpretrain/models/backbones/__init__.py` 中导入新模块

   ```python
   ...

   __all__ = [
   ]
   ```

3. 在配置文件中使用新的主干网络

   ```python
       ...
           depth=18,
           other_arg=xxx),
       ...
   ```

### 为自监督学习添加新的主干网络

对于一部分自监督学习算法，主干网络做了一定修改，例如 `MAE`、`BEiT` 等。 这些主干网络需要处理 `mask` 相关的逻辑，以此从可见的图像块中提取对应的特征信息。

以 [MAEViT](mmpretrain.models.selfsup.MAEViT) 作为例子，我们需要重写 `forward` 函数，进行基于 `mask` 的计算。我们实现了 `init_weights` 进行特定权重的初始化和 `random_masking` 函数来生成 `MAE` 预训练所需要的 `mask`。

```python

```

## 添加新的颈部组件

这里我们以 `GlobalAveragePooling` 为例。这是一个非常简单的颈部组件，没有任何参数。

要添加新的颈部组件，我们主要需要实现 `forward` 函数，该函数对主干网络的输出进行
一些操作并将结果传递到头部。

1. 创建一个新文件 `mmpretrain/models/necks/gap.py`

   ```python

   ```python

           # 简单起见，我们默认输入是一个张量
   ```

2. 在 `mmpretrain/models/necks/__init__.py` 中导入新模块

   ```python
   ...

   __all__ = [
   ]
   ```

3. 修改配置文件以使用新的颈部组件

   ```python
   )
   ```

## 添加新的头部组件

### 基于分类头

在此，我们以一个简化的 `VisionTransformerClsHead` 为例，说明如何开发新的头部组件。

要添加一个新的头部组件，基本上我们需要实现 `pre_logits` 函数用于进入最后的分类头之前需要的处理，
以及 `forward` 函数。

1. 创建一个文件 `mmpretrain/models/heads/vit_head.py`.

   ```python

   ```python

   class LinearClsHead(ClsHead):

           # 骨干网络的输出通常包含多尺度信息的元组
           # 对于分类任务来说，我们只需要关注最后的输出

           # VisionTransformer的最终输出是一个包含patch tokens和cls tokens的元组
           # 这里我们只需要cls tokens

           # 完成除了最后的线性分类头以外的操作

feat = feats[-1]

           # 完成最后的分类头
           cls_score = self.fc(pre_logits)
   ```

2. 在 `mmpretrain/models/heads/__init__.py` 中导入这个模块

   ```python
   ...

   __all__ = [
   ]
   ```

3. 修改配置文件以使用新的头部组件。

   ```python
           ...,
       ))
   ```

### 基于 BaseModule 类

这是一个基于 MMEngine 中的 `BaseModule` 进行开发例子，`MAEPretrainHead`，主要是为了 `MAE` 掩码学习。我们需要实现 `loss` 函数来计算损失吗，不过其它的函数均为可选项。

```python

        return x

```

完成实现后，之后的步骤和 [基于分类头](#基于分类头) 中的步骤 2 和步骤 3 一致。

## 添加新的损失函数

target = self.construct_target(target)

```

要添加新的损失函数，我们主要需要在损失函数模块中 `forward` 函数。这里需要注意的是，损失模块也应该注册到`MODELS`中。另外，利用装饰器 `weighted_loss` 可以方便的实现对每个元素的损失进行加权平均。

假设我们要模拟从另一个分类模型生成的概率分布，需要添加 `L1loss` 来实现该目的。

1. 创建一个新文件 `mmpretrain/models/losses/l1_loss.py`

   ```python

                   pred,

super(L1Loss, self).__init__()

                   pred,
                   target,
   ```

2. 在文件 `mmpretrain/models/losses/__init__.py` 中导入这个模块

   ```python
   ...

   __all__ = [
   ]
   ```

3. 修改配置文件中的 `loss` 字段以使用新的损失函数

   ```python
       ))
   ```

__all__ = [
   ]
   ```

   ```python
       ))
   ```

最后我们可以在配置文件中结合所有新增的模型组件来使用新的模型。由于`ResNet_CIFAR` 不是一个基于ViT的骨干网络，这里我们不用`VisionTransformerClsHead`的配置。

```python
        depth=18,
    ))

```

```{tip}
为了方便，相同的模型组件可以直接从已有的config文件里继承，更多细节可以参考[学习配置文件](../user_guides/config.md)。
```

# Use Pure Point Cloud Dataset

## Data Pre-Processing

### Convert Point cloud format

Currently, we only support bin format point cloud training and inference, before training on your own datasets, you need to transform your point cloud format to bin file. The common point cloud data formats include pcd and las, we provide some open-source tools for reference.

1. Convert pcd to bin: https://github.com/leofansq/Tools_RosBag2KITTI
2. Convert las to bin: The common conversion path is las -> pcd -> bin, and the conversion from las -> pcd can be achieved through [this tool](https://github.com/Hitachi-Automotive-And-Industry-Lab/semantic-segmentation-editor).

### Point cloud annotation

MMDetection3D does not support point cloud annotation. Some open-source annotation tools are offered for reference:

Besides, we improved [LATTE](https://github.com/bernwang/latte) for better usage. More details can be found [here](https://arxiv.org/abs/2011.10174).

To support a new data format, you can either convert them to existing formats or directly convert them to the middle format. You could also choose to convert them offline (before training by a script) or online (implement a new dataset and do the conversion at training).

Once your datasets only contain point cloud file and 3D Bounding box annotations, without calib file. We recommend converting it into the basic formats, the annotations files in basic format has the following necessary keys:

```python

[
    {'sample_idx':
     'lidar_points': {'lidar_path': velodyne_path,
                      ....
                     },

```python

[
                      ....
                     },
     'annos': {'box_type_3d':  (str)  'LiDAR/Camera/Depth'
               'gt_bboxes_3d':  <np.ndarray> (n, 7)
               'gt_names':  [list]
               ....
            }
     'calib': { .....}
     'images': { .....}
    }
]

```

In MMDetection3D, for the data that is inconvenient to read directly online, we recommend converting it into into basic format as above and do the conversion offline, thus you only need to modify the config's data annotation paths and classes after the conversion.
To use data that share a similar format as the existing datasets, e.g., Lyft has a similar format as the nuScenes dataset, we recommend directly implementing a new data converter and a dataset class to convert the data and load the data, respectively. In this procedure, the code can inherit from the existing dataset classes to reuse the code.

There is also a way if users do not want to convert the annotation format to existing formats.
Actually, we convert all the supported datasets into pickle files, which summarize useful information for model training and inference.

The annotation of a dataset is a list of dict, each dict corresponds to a frame.
A basic example (used in KITTI) is as follows. A frame consists of several keys, like `image`, `point_cloud`, `calib` and `annos`.
As long as we could directly read data according to these information, the organization of raw data could also be different from existing ones.
With this design, we provide an alternative choice for customizing datasets.

```python

[
    {'image': {'image_idx': 0, 'image_path': 'training/image_2/000000.png', 'image_shape': array([ 370, 1224], dtype=int32)},
     'point_cloud': {'num_features': 4, 'velodyne_path': 'training/velodyne/000000.bin'},

'point_cloud': {'num_features': 4, 'velodyne_path': 'training/velodyne/000000.bin'},
     'calib': {'P0': array([[707.0493,   0.    , 604.0814,   0.    ],
       [  0.    , 707.0493, 180.5066,   0.    ],
       [  0.    ,   0.    ,   1.    ,   0.    ],
       [  0.    ,   0.    ,   0.    ,   1.    ]]),
       'P1': array([[ 707.0493,    0.    ,  604.0814, -379.7842],
       [   0.    ,  707.0493,  180.5066,    0.    ],
       [   0.    ,    0.    ,    1.    ,    0.    ],
       [   0.    ,    0.    ,    0.    ,    1.    ]]),
       'P2': array([[ 7.070493e+02,  0.000000e+00,  6.040814e+02,  4.575831e+01],
       [ 0.000000e+00,  7.070493e+02,  1.805066e+02, -3.454157e-01],
       [ 0.000000e+00,  0.000000e+00,  1.000000e+00,  4.981016e-03],
       [ 0.000000e+00,  0.000000e+00,  0.000000e+00,  1.000000e+00]]),
       'P3': array([[ 7.070493e+02,  0.000000e+00,  6.040814e+02, -3.341081e+02],
       [ 0.000000e+00,  7.070493e+02,  1.805066e+02,  2.330660e+00],

'P3': array([[ 7.070493e+02,  0.000000e+00,  6.040814e+02, -3.341081e+02],
       [ 0.000000e+00,  0.000000e+00,  1.000000e+00,  3.201153e-03],
       'R0_rect': array([[ 0.9999128 ,  0.01009263, -0.00851193,  0.        ],
       [-0.01012729,  0.9999406 , -0.00403767,  0.        ],
       [ 0.00847068,  0.00412352,  0.9999556 ,  0.        ],
       [ 0.        ,  0.        ,  0.        ,  1.        ]]),
       'Tr_velo_to_cam': array([[ 0.00692796, -0.9999722 , -0.00275783, -0.02457729],
       [-0.00116298,  0.00274984, -0.9999955 , -0.06127237],
       [ 0.9999753 ,  0.00693114, -0.0011439 , -0.3321029 ],
       'Tr_imu_to_velo': array([[ 9.999976e-01,  7.553071e-04, -2.035826e-03, -8.086759e-01],
       [-7.854027e-04,  9.998898e-01, -1.482298e-02,  3.195559e-01],

'Tr_imu_to_velo': array([[ 9.999976e-01,  7.553071e-04, -2.035826e-03, -8.086759e-01],
       [ 2.024406e-03,  1.482454e-02,  9.998881e-01, -7.997231e-01],
       [ 0.000000e+00,  0.000000e+00,  0.000000e+00,  1.000000e+00]])},
     'annos': {'name': array(['Pedestrian'], dtype='<U10'), 'truncated': array([0.]), 'occluded': array([0]), 'alpha': array([-0.2]), 'bbox': array([[712.4 , 143.  , 810.73, 307.92]]), 'dimensions': array([[1.2 , 1.89, 0.48]]), 'location': array([[1.84, 1.47, 8.41]]), 'rotation_y': array([0.01]), 'score': array([0.]), 'index': array([0], dtype=int32), 'group_ids': array([0], dtype=int32), 'difficulty': array([0], dtype=int32), 'num_points_in_gt': array([377], dtype=int32)}}
    ...
]
```

On top of this you can write a new Dataset class inherited from `Custom3DDataset`, and overwrite related methods,

...
]
```

like [KittiDataset](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/datasets/kitti_dataset.py) and [ScanNetDataset](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/datasets/scannet_dataset.py).

Here we provide an example of customized dataset.

Assume the annotation has been reorganized into a list of dict in pickle files like basic format.
The bounding boxes annotations are stored in `annotation.pkl` as the following

```
{'sample_idx': 120,
 'lidar_points': {'lidar_path': 'training/000004.bin'},
 'annos': {'bbox_type_3d': 'LiDAR',
           'gt_bboxes_3d': array([[1.48129511,  3.52074146,  1.85652947, 1.74445975, 0.23195696, 0.57235193, -0.25525],
           [ 2.90395617, -3.48033905,  1.52682471,[0.66077662, 0.17072392, 0.67153597, 2.23145]]),

[ 2.90395617, -3.48033905,  1.52682471,[0.66077662, 0.17072392, 0.67153597, 2.23145]]),
           'gt_names': ['car', 'pedestrian']
          }
}
```

If the pkl only contains the necessary keys, you can directly use the `Custom3DDataset` for training:

Then in the config, to use `Custom3DDataset` you can modify the config as the following

```python
    type='Custom3DDataset',
    ann_file = 'annotation.pkl',
)
```

otherwise you need to create a new dataset in `mmdet3d/datasets/my_dataset.py` to load the data and rewrite the `get_ann_info` method.

```python
from os import path as osp

from mmdet3d.core import show_result
from mmdet3d.core.bbox import DepthInstance3DBoxes
from mmdet.datasets import DATASETS
from .custom_3d import Custom3DDataset

class MyDataset(Custom3DDataset):
    classes = ('cabinet', 'bed', 'chair', 'sofa', 'table', 'door', 'window',

               'bookshelf', 'picture', 'counter', 'desk', 'curtain',
               'refrigerator', 'showercurtrain', 'toilet', 'sink', 'bathtub',
               'garbagebin')

                 data_root,
                 ann_file,
                 pipeline=None,
                 classes=None,
                 modality=None,
                 box_type_3d='Depth',
                 filter_empty_gt=True,
                 test_mode=False):
            ann_file=ann_file,
            pipeline=pipeline,
            classes=classes,
            modality=modality,
            box_type_3d=box_type_3d,
            filter_empty_gt=filter_empty_gt,
            test_mode=test_mode)

    def get_ann_info(self, index):

modality=modality,

        # Use index to get the annos, thus the evalhook could also use this api
        info = self.data_infos[index]
        if info['annos']['gt_num'] != 0:
            gt_bboxes_3d = info['annos']['gt_boxes_upright_depth'].astype(
                np.float32)  # k, 6
            gt_labels_3d = info['annos']['class'].astype(np.int64)
        else:
            gt_bboxes_3d = np.zeros((0, 6), dtype=np.float32)
            gt_labels_3d = np.zeros((0, ), dtype=np.int64)

        # to target box structure
        gt_bboxes_3d = DepthInstance3DBoxes(
            gt_bboxes_3d,
            box_dim=gt_bboxes_3d.shape[-1],
            with_yaw=False,
            origin=(0.5, 0.5, 0.5)).convert_to(self.box_mode_3d)

        pts_instance_mask_path = osp.join(self.data_root,

box_dim=gt_bboxes_3d.shape[-1],

                                          info['pts_instance_mask_path'])
        pts_semantic_mask_path = osp.join(self.data_root,
                                          info['pts_semantic_mask_path'])

        anns_results = dict(
            gt_bboxes_3d=gt_bboxes_3d,
            gt_labels_3d=gt_labels_3d,
            pts_instance_mask_path=pts_instance_mask_path,
            pts_semantic_mask_path=pts_semantic_mask_path)
        return anns_results

```

```python
)
```

```python
)
```

MMDetection3D also supports many dataset wrappers to mix the dataset or modify the dataset distribution for training like MMDetection.

```python
        times=N,
            ...
        )
    )
```

dataset=dict(  # This is the original config of Dataset_A
            ...
        )
    )
```

```python
            ...
        )
    )
```

You may refer to [source code](https://github.com/open-mmlab/mmdetection/blob/master/mmdet/datasets/dataset_wrappers.py) for details.

   ```python
   )
   ```

   ```python
   )
   ```

   ```python

```python

       train = [
       ],
       )
   ```

   ```python

       val=dict(
   ```

**Note:**

```python
    times=N,
        ...
    )
)
    ...
)
    ...
)
    times=M,
        ...
    )
)
    train = [
    ],
)

```

With existing dataset types, we can modify the class names of them to train subset of the annotations.

```python

```python
    train=dict(classes=classes),
    val=dict(classes=classes),
    test=dict(classes=classes))
```

MMDetection V2.0 also supports to read the classes from a file, which is common in real applications.
For example, assume the `classes.txt` contains the name of classes as the following.

```
person
bicycle
car
```

Users can set the classes as a file path, the dataset will load it and convert it to a list automatically.

```python
classes = 'path/to/classes.txt'
```

## Loading Point Clouds Adjustment

```python
```

Generally speaking, the most basic bin data contains (x, y, z) information, and some also include intensity, elongation (point cloud elongation), timestamp, and the point cloud dimension ranges from 3 to 6. In MMDetection3D, you need to adjust the some settings in config while customized dataset training:

```python
    dict(
        # adjust accordingly according to the dimension
        # of the point cloud of your own dataset
        load_dim=3,
        # actually used dimension，you can also specify the
        # specific dimension in list format
        use_dim=3),
```

## Training Setting Adjustment

load_dim=3,
```

In order to avoid some problems in the training process and improve the performance of the model on the custom dataset, some training settings need to be adjusted according to the dataset.

### Adjust Point Cloud Range and Annotations in Config

For example, we can adjust `point_cloud_range` in config file to change training point cloud range. In KITTI dataset, the `point_cloud_range` is set to be `[0, -39.68, -3, 69.12, 39.68, 1]`.
By setting point cloud range, the `PointsRangeFilter` is used to filter point cloud and its mask (semantic and instance), and `ObjectRangeFilter` is used to filter 3D bounding boxes.

```python
```

### Adjust Voxel Size in Config

```python
```

Here you can refer to the setting of the existing datasets. theoretically, `voxel_size` is linked to the setting of `point_cloud_range`. Setting a smaller `voxel_size` will increase the voxel num and the corresponding memory consumption. In addition, the following issues need to be noted:

if the `point_cloud_range` and `voxel_size` are set to be `[0, -40, -3, 70.4, 40, 1]` and `[0.05, 0.05, 0.1]` respectively, then the shape of intermediate feature map should be `[(1-(-3))/0.1+1, (40-(-40))/0.05, (70.4-0)/0.05]=[41, 1600, 1408]`. More details refers to this [issue](https://github.com/open-mmlab/mmdetection3d/issues/382).

### Adjust Anchor Range and Size in Config

```python
anchor_generator=dict(
    type='Anchor3DRangeGenerator',
    ranges=[
        [0, -40.0, -0.6, 70.4, 40.0, -0.6],

```python
    ranges=[
        [0, -40.0, -1.78, 70.4, 40.0, -1.78],
    ],
    sizes=[[0.8, 0.6, 1.73], [1.76, 0.6, 1.73], [3.9, 1.6, 1.56]],
    rotations=[0, 1.57],
    reshape_out=False),
```

Regarding the setting of `anchor_size`, it is usually necessary to count the average length, width and height of the entire training dataset as `anchor_size` to obtain the best results.

**Note** (related to MMDetection):

- Before MMDetection v2.5.0, the dataset will filter out the empty GT images automatically if the classes are set and there is no way to disable that through config. This is an undesirable behavior and introduces confusion because if the classes are not set, the dataset only filters the empty GT images when `filter_empty_gt=True` and `test_mode=False`. After MMDetection v2.5.0, we decouple the image filtering process and the classes modification, i.e., the dataset will only filter empty GT images when `filter_empty_gt=True` and `test_mode=False`, no matter whether the classes are set. Thus, setting the classes only influences the annotations of classes used for training and users could decide whether to filter empty GT images by themselves.
- Since the middle format only has box labels and does not contain the class names, when using `CustomDataset`, users cannot filter out the empty GT images through configs but only do this offline.

- The features for setting dataset classes and dataset filtering will be refactored to be more user-friendly in the future (depends on the progress).

# GLIDE (Arxiv'2021)

> [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://papers.nips.cc/paper/2021/file/49ad23d1ec9fa4bd8d77d02681df5cfa-Paper.pdf)

> **Task**: Text2Image, diffusion

Diffusion models have recently been shown to generate high-quality synthetic images, especially when paired with a guidance technique to trade off diversity for fidelity. We explore diffusion models for the problem of text-conditional image synthesis and compare two different guidance strategies: CLIP guidance and classifier-free guidance. We find that the latter is preferred by human evaluators for both photorealism and caption similarity, and often produces photorealistic samples. Samples from a 3.5 billion parameter text-conditional diffusion model using classifierfree guidance are favored by human evaluators to those from DALL-E, even when the latter uses expensive CLIP reranking. Additionally, we find that our models can be fine-tuned to perform image inpainting, enabling powerful text-driven image editing. We train a smaller model on a filtered dataset and release the code and weights at https://github.com/openai/glide-text2im.

 <img src="https://user-images.githubusercontent.com/22982797/209770463-31f3083d-b939-4ed6-b504-6a5baf7365b5.png" width="400"/>
</div >

  <b>an oil painting of a corgi</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/22982797/210042533-1df54b2d-d8a8-42b1-974c-06861e3e6ef6.png" width="400"/>
</div>

  <b>an cartoon painting of a cat</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/22982797/210042530-ada31a01-7c9d-452b-bc72-56ae0182ef2f.png" width="400"/>
</div>

**Laion**

| Method | Resolution | Config                                                                     | Weights                                                                             |
| ------ | ---------- | -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |

| Glide  | 64x64      | [config](projects/glide/configs/glide_ddim-classifier-free_laion-64x64.py) | [model](https://download.openmmlab.com/mmagic/glide/glide_laion-64x64-02afff47.pth) |

You can run glide as follows:

```python
from mmagic.apis import init_model
from projects.glide.models import *

config = 'projects/glide/configs/glide_ddim-classifier-free_laion-64x64.py'
ckpt = 'https://download.openmmlab.com/mmagic/glide/glide_laion-64x64-02afff47.pth'
model = init_model(config, ckpt).cuda().eval()
prompt = "an oil painting of a corgi"

    samples = model.infer(init_image=None,
                prompt=prompt,
                batch_size=16,

                guidance_scale=3.,
                num_inference_steps=100,
                labels=None,
                classifier_scale=0.0,
                show_progress=True)['samples']
```

```bibtex
@article{2021GLIDE,
  title={GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models},
  author={ Nichol, A.  and  Dhariwal, P.  and  Ramesh, A.  and  Shyam, P.  and  Mishkin, P.  and  Mcgrew, B.  and  Sutskever, I.  and  Chen, M. },
}
```

# Use Docker Image

This document guides how to install mmdeploy with [Docker](https://docs.docker.com/get-docker/).

## Get prebuilt docker images

MMDeploy provides prebuilt docker images for the convenience of its users on [Docker Hub](https://hub.docker.com/r/openmmlab/mmdeploy). The docker images are built on
the latest and released versions. For instance, the image with tag `openmmlab/mmdeploy:ubuntu20.04-cuda11.3-mmdeploy` is built on the latest mmdeploy and the image with tag `openmmlab/mmdeploy:ubuntu20.04-cuda11.3-mmdeploy1.2.0` is for `mmdeploy==1.2.0`.
The specifications of the Docker Image are shown below.

You can select a [tag](https://hub.docker.com/r/openmmlab/mmdeploy/tags) and run `docker pull` to get the docker image:

```shell
```

## Build docker images (optional)

If the prebuilt docker images do not meet your requirements,
then you can build your own image by running the following script.
The docker file is `docker/Release/Dockerfile`and its building argument is `MMDEPLOY_VERSION`,
which can be a [tag](https://github.com/open-mmlab/mmdeploy/tags) or a branch from [mmdeploy](https://github.com/open-mmlab/mmdeploy).

```shell

```shell
```

## Run docker container

After pulling or building the docker image, you can use `docker run` to launch the docker service:

```shell
```

## FAQs

   As described [here](https://forums.developer.nvidia.com/t/cuda-error-the-provided-ptx-was-compiled-with-an-unsupported-toolchain/185754), update the GPU driver to the latest one for your GPU.

   ```shell

```shell

   ```

# Conventions

Please check the following conventions if you would like to modify MMDetection as your own project.

## About the order of image shape

In OpenMMLab 2.0, to be consistent with the input argument of OpenCV, the argument about image shape in the data transformation pipeline is always in the `(width, height)` order. On the contrary, for computation convenience, the order of the field going through the data pipeline and the model is `(height, width)`. Specifically, in the results processed by each data transform pipeline, the fields and their value meaning is as below:

As an example, the initialization arguments of `Mosaic` are as below:

```python

```python
       ...

       # img_scale order should be (width, height)

        ...

```

## Loss

In MMDetection, a `dict` containing losses and metrics will be returned by `model(**data)`.

For example, in bbox head,

```python
    ...
        # classification loss
        # classification accuracy

    ...
        # bbox regression loss
```

`bbox_head.loss()` will be called during model forward.
The returned dict contains `'loss_bbox'`, `'loss_cls'`, `'acc'` .
Only `'loss_bbox'`, `'loss_cls'` will be used during back propagation,
`'acc'` will only be used as a metric to monitor training process.

By default, only values whose keys contain `'loss'` will be back propagated.
This behavior could be changed by modifying `BaseDetector.train_step()`.

## Empty Proposals

In MMDetection, We have added special handling and unit test for empty proposals of two-stage. We need to deal with the empty proposals of the entire batch and single image at the same time. For example, in CascadeRoIHead,

```python
# simple_test method
...
# There is no proposal in the whole batch
    else:
...

# There is no proposal in the single image
    ...
                # Handle empty proposal
                         rois[j], bbox_label, bbox_pred[j], img_metas[j])

refine_roi = self.bbox_head[i].regress_by_class(
```

If you have customized `RoIHead`, you can refer to the above method to deal with empty proposals.

## Coco Panoptic Dataset

In MMDetection, we have supported COCO Panoptic dataset. We clarify a few conventions about the implementation of `CocoPanopticDataset` here.

1. For mmdet\<=2.16.0, the range of foreground and background labels in semantic segmentation are different from the default setting of MMDetection. The label `0` stands for `VOID` label and the category labels start from `1`.
   Since mmdet=2.17.0, the category labels of semantic segmentation start from `0` and label `255` stands for `VOID` for consistency with labels of bounding boxes.
   To achieve that, the `Pad` pipeline supports setting the padding value for `seg`.

To achieve that, the `Pad` pipeline supports setting the padding value for `seg`.
2. In the evaluation, the panoptic result is a map with the same shape as the original image. Each value in the result map has the format of `instance_id * INSTANCE_OFFSET + category_id`.

<details>

```bibtex
}
```

<details>

```bibtex

| [pose_scnet_50](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_scnet50_8xb64-210e_mpii-256x256.py) |  256x256   | 0.888 |  0.290   | [ckpt](https://download.openmmlab.com/mmpose/top_down/scnet/scnet50_mpii_256x256-a54b6af5_20200812.pth) | [log](https://download.openmmlab.com/mmpose/top_down/scnet/scnet50_mpii_256x256_20200812.log.json) |
| [pose_scnet_101](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_scnet101_8xb64-210e_mpii-256x256.py) |  256x256   | 0.887 |  0.293   | [ckpt](https://download.openmmlab.com/mmpose/top_down/scnet/scnet101_mpii_256x256-b4c2d184_20200812.pth) | [log](https://download.openmmlab.com/mmpose/top_down/scnet/scnet101_mpii_256x256_20200812.log.json) |

本节我们继续演示如何使用 demo 脚本进行 2D 脸部关键点的识别。同样的，用户仍要确保开发环境已经安装了 3.0 版本以上的 [MMdetection](https://github.com/open-mmlab/mmdetection) 。

我们在 [mmdet model zoo](/demo/docs/zh_cn/mmdet_modelzoo.md#脸部-bounding-box-检测模型) 提供了一个预训练好的脸部 Bounding Box 预测模型，用户可以前往下载。

### 2D 脸部图片关键点识别推理

```shell
```

用户可以在 [model zoo](https://mmpose.readthedocs.io/en/dev-1.x/model_zoo/face_2d_keypoint.html) 获取预训练好的脸部关键点识别模型。

这里我们用 [aflw model](https://download.openmmlab.com/mmpose/face/hrnetv2/hrnetv2_w18_aflw_256x256-f2bbc62b_20210125.pth) 来进行演示：

```shell

```shell
```

可视化结果如下图所示：

```shell

```shell
```

### 2D 脸部视频关键点识别推理

```shell

https://download.openmmlab.com/mmpose/mmdet_pretrained/yolo-x_8xb8-300e_coco-face_13274d7c.pth \
```

这段视频可以在 [Google Drive](https://drive.google.com/file/d/1kQt80t6w802b_vgVcmiV_QfcSJ3RWzmb/view?usp=sharing) 下载。

### 使用 Inferencer 进行 2D 脸部关键点识别推理

```shell
```

```shell
```

该命令会对输入的 `tests/data/wflw` 下所有的图片进行推理并且把可视化结果都存入 `vis_results/wflw` 文件夹下。

除此之外， Inferencer 也支持保存预测的姿态结果。具体信息可在 [Inferencer 文档](https://mmpose.readthedocs.io/en/dev-1.x/user_guides/inference.html#inferencer-a-unified-inference-interface) 查看。

### 加速推理

对于 2D 脸部关键点预测模型，用户可以通过修改配置文件中的 `model.test_cfg.flip_test=False` 来加速，例如 [aflw_hrnetv2](../../../configs/face_2d_keypoint/topdown_heatmap/aflw/td-hm_hrnetv2-w18_8xb64-60e_aflw-256x256.py) 中的第 90 行。

# MMDetection Deployment

- [MMDetection Deployment](#mmdetection-deployment)
    - [Install mmdet](#install-mmdet)

[MMDetection](https://github.com/open-mmlab/mmdetection) aka `mmdet` is an open source object detection toolbox based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com/) project.

### Install mmdet

Please follow the [installation guide](https://mmdetection.readthedocs.io/en/3.x/get_started.html) to install mmdet.

```shell

```shell
```

You can use [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/tree/main/tools/deploy.py) to convert mmdet models to the specified backend models. Its detailed usage can be learned from [here](../02-how-to-run/convert_model.md).

The command below shows an example about converting `Faster R-CNN` model to onnx model that can be inferred by ONNX Runtime.

```shell

```shell
    --show \
```

It is crucial to specify the correct deployment config during model conversion. We've already provided builtin deployment config [files](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmdet) of all supported backends for mmdetection, under which the config file path follows the pattern:

```
```

```
```

  mmdet models like `RetinaNet`, `Faster R-CNN` and `DETR` and so on belongs to `detection` task. While `Mask R-CNN` is one of `instance-seg` models. You can find more of them in chapter [Supported models](#supported-models).

Therefore, in the above example, you can also convert `faster r-cnn` to other backend models by changing the deployment config file `detection_onnxruntime_dynamic.py` to [others](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmdet/detection), e.g., converting to tensorrt-fp16 model by `detection_tensorrt-fp16_dynamic-320x320-1344x1344.py`.

```{tip}
```

The converted model locates in the working directory like `mmdeploy_models/mmdet/ort` in the previous example. It includes:

```

```
```

in which,

The whole package **mmdeploy_models/mmdet/ort** is defined as **mmdeploy SDK model**, i.e., **mmdeploy SDK model** includes both backend model and inference meta information.

```python

```

```python
import cv2

```

```python
import cv2

    continue

```

- For transformer based models, strongly suggest use `TensorRT>=8.4`.
- <i id="mask2former">Mask2Former</i> should use `TensorRT>=8.6.1` for dynamic shape inference.

<details>
<summary align="right">SRGAN (CVPR'2016)</summary>

```bibtex
}
```

<br/>

| [msrresnet_x4c64b16_1x16_300k_div2k](./msrresnet_x4c64b16_1xb16-1000k_div2k.py) | 30.2252 / 0.8491  | 26.7762 / 0.7369 | 28.9748 / 0.8178 |    1     | [模型](https://download.openmmlab.com/mmediting/restorers/srresnet_srgan/msrresnet_x4c64b16_1x16_300k_div2k_20200521-61556be5.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/srresnet_srgan/msrresnet_x4c64b16_1x16_300k_div2k_20200521_110246.log.json) |

| [srgan_x4c64b16_1x16_1000k_div2k](./srgan_x4c64b16_1xb16-1000k_div2k.py) | 27.9499 /  0.7846 | 24.7383 / 0.6491 | 26.5697 / 0.7365 |    1     | [模型](https://download.openmmlab.com/mmediting/restorers/srresnet_srgan/srgan_x4c64b16_1x16_1000k_div2k_20200606-a1f0810e.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/srresnet_srgan/srgan_x4c64b16_1x16_1000k_div2k_20200506_191442.log.json) |

## 快速开始

**训练**

<details>

```shell
# CPU上训练

# 单个GPU上训练

# 多个GPU上训练
```

**测试**

<details>

```shell
# CPU上测试

```

**测试**

<details>

```shell
# CPU上测试

# 单个GPU上测试

# 多个GPU上测试
```

# MobileNetV3

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/backbones/mobilenet_v3.py#L15">Code Snippet</a>

<img src="https://user-images.githubusercontent.com/24582831/142902036-3dc2e0c0-d475-4816-b1ac-961836b41f5c.png" width="60%"/>
</div>

| Method | Backbone           | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                                  | download                                                                                                                                                                                                                                                                                                                                                                                                     |

| LRASPP | M-V3-D8            | 512x1024  |  320000 |      8.9 | 15.22          | V100   | 69.54 | 70.89         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mobilenet_v3/mobilenet-v3-d8_lraspp_4xb4-320k_cityscapes-512x1024.py)           | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v3/lraspp_m-v3-d8_512x1024_320k_cityscapes/lraspp_m-v3-d8_512x1024_320k_cityscapes_20201224_220337-cfe8fb07.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v3/lraspp_m-v3-d8_512x1024_320k_cityscapes/lraspp_m-v3-d8_512x1024_320k_cityscapes-20201224_220337.log.json)                                     |

| LRASPP | M-V3-D8 (scratch)  | 512x1024  |  320000 |      8.9 | 14.77          | V100   | 67.87 | 69.78         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mobilenet_v3/mobilenet-v3-d8-scratch_lraspp_4xb4-320k_cityscapes-512x1024.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v3/lraspp_m-v3-d8_scratch_512x1024_320k_cityscapes/lraspp_m-v3-d8_scratch_512x1024_320k_cityscapes_20201224_220337-9f29cd72.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v3/lraspp_m-v3-d8_scratch_512x1024_320k_cityscapes/lraspp_m-v3-d8_scratch_512x1024_320k_cityscapes-20201224_220337.log.json)     |

| LRASPP | M-V3s-D8           | 512x1024  |  320000 |      5.3 | 23.64          | V100   | 64.11 | 66.42         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mobilenet_v3/mobilenet-v3-d8-s_lraspp_4xb4-320k_cityscapes-512x1024.py)         | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v3/lraspp_m-v3s-d8_512x1024_320k_cityscapes/lraspp_m-v3s-d8_512x1024_320k_cityscapes_20201224_223935-61565b34.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v3/lraspp_m-v3s-d8_512x1024_320k_cityscapes/lraspp_m-v3s-d8_512x1024_320k_cityscapes-20201224_223935.log.json)                                 |

| LRASPP | M-V3s-D8 (scratch) | 512x1024  |  320000 |      5.3 | 24.50          | V100   | 62.74 | 65.01         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/mobilenet_v3/mobilenet-v3-d8-scratch-s_lraspp_4xb4-320k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v3/lraspp_m-v3s-d8_scratch_512x1024_320k_cityscapes/lraspp_m-v3s-d8_scratch_512x1024_320k_cityscapes_20201224_223935-03daeabb.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/mobilenet_v3/lraspp_m-v3s-d8_scratch_512x1024_320k_cityscapes/lraspp_m-v3s-d8_scratch_512x1024_320k_cityscapes-20201224_223935.log.json) |

```bibtex
  title={Searching for MobileNetV3},
  author={Howard, Andrew and Sandler, Mark and Chu, Grace and Chen, Liang-Chieh and Chen, Bo and Tan, Mingxing and Wang, Weijun and Zhu, Yukun and Pang, Ruoming and Vasudevan, Vijay and Le, Quoc V. and Adam, Hartwig},

booktitle={The IEEE International Conference on Computer Vision (ICCV)},
  pages={1314-1324},
  month={October},
  doi={10.1109/ICCV.2019.00140}}
}
```

# mmdeploy 各目录功能

本文主要介绍 mmdeploy 各目录功能，以及从模型到具体推理框架是怎么工作的。

## 一、大致看下目录结构

整个 mmdeploy 可以看成比较独立的两部分：模型转换 和 SDK。

我们介绍整个 repo 目录结构和功能，不必细究源码、有个印象即可。

外围目录功能：

```bash
$ tree -L 1
.
├── CMakeLists.txt    # 编译模型转换自定义算子和 SDK 的 cmake 配置
├── configs                   # 模型转换要用的算法库配置
├── csrc                          # SDK 和自定义算子
├── demo                      # 各语言的 ffi 接口应用实例，如 csharp、java、python 等
├── docker                   #  docker build
├── mmdeploy           # 用于模型转换的 python 包
├── requirements      # python 包安装依赖
├── service                    # 有些小板子不能跑 python，模型转换用的 C/S 模式。这个目录放 Server
├── tests                         # 单元测试
├── third_party           # SDK 和 ffi 要的第三方依赖
└── tools                        # 工具，也是一切功能的入口。除了 deploy.py 还有 onnx2xx.py、profiler.py 和 test.py
```

这样大致应该清楚了

- 模型转换主要看 tools + mmdeploy + 小部分 csrc 目录；
- 而 SDK 的本体在 csrc + third_party + demo 三个目录。

## 二、模型转换

模型以 mmpretrain 的 ViT 为例，推理框架就用 ncnn 举例。其他模型、推理都是类似的。

```

这样大致应该清楚了

## 二、模型转换

我们看下 mmdeploy/mmdeploy 目录结构，有个印象即可：

```bash
.
├── apis                             # tools 工具用的 api，都是这里实现的，如 onnx2ncnn.py
│   ├── calibration.py          # trt 专用收集量化数据
│   ├── core                              # 软件脚手架
│   ├── extract_model.py  # onnx 模型只想导出一部分，切 onnx 用的
│   ├── inference.py             # 抽象函数，实际会调 torch/ncnn 具体的 inference
│   ├── ncnn                            # 引用 backend/ncnn 的函数，只是包了一下
│   └── visualize.py              # 还是抽象函数，实际会调用 torch/ncnn 具体的 inference 和 visualize
..
├── backend                  # 具体的 backend 包装
│   ├── base                            # 因为有多个 backend，所以得有个 base 类的 OO 设计
│   ├── ncnn                           # 这里为模型转换调用 ncnn python 接口
│   │   ├── init_plugins.py           # 找 ncnn 自定义算子和 ncnn 工具的路径
│   │   ├── onnx2ncnn.py            # 把 `mmdeploy_onnx2ncnn` 封装成 python 接口

│   │   ├── quant.py                       # 封装 `ncnn2int8` 工具为 python 接口
│   │   └── wrapper.py                  # 封装 pyncnn forward 接口
..
├── codebase                #  mm 系列算法 forward 重写
│   ├── base                          # 有多个算法，需要点 OO 设计
│   ├── mmpretrain                      #  mmpretrain 相关模型重写
│   │   ├── deploy                       # mmpretrain 对 base 抽象 task/model/codebase 的实现
│   │   └── models                      # 开始真正的模型重写
│   │       ├── backbones                 # 骨干网络部分的重写，例如  multiheadattention
│   │       ├── heads                           # 例如  MultiLabelClsHead
│   │       ├── necks                            # 例如 GlobalAveragePooling
│..
├── core                         # 软件脚手架，重写机制怎么实现的
├── mmcv                     # mmcv 有的 opr 也需要重写
├── pytorch                 #  针对 ncnn 重写 torch 的 opr，例如 Gemm
..
```

│..
..
```

上面的每一行是需要读的，请勿跳过。

当敲下`tools/deploy.py` 转换 ViT，核心是这 3 件事：

1. mmpretrain ViT forward 过程的重写
2. ncnn 不支持 gather opr，自定义一下、和 libncnn.so 一起加载
3. 真实跑一遍，渲染结果，确保正确

### 1. forward 重写

因为 onnx 会生成稀碎的算子、ncnn 也不是完美支持 onnx，所以 mmdeploy 的方案是劫持有问题的 forward 代码、改成适合 ncnn 的 onnx 结果。

例如把 `conv -> shape -> concat_const -> reshape` 过程改成 `conv -> reshape`，削掉多余的 `shape` 和 `concat` 算子。

所有的 mmpretrain 算法重写都在 `mmdeploy/codebase/mmpretrain/models`目录。

### 2. 自定义算子

针对 ncnn 自定义的算子都在 `csrc/mmdeploy/backend_ops/ncnn/`目录，编译后和 libncnn.so 一起加载。本质是在 hotfix ncnn，目前实现了

- topk
- shape
- gather
- expand

### 3. 转换和测试

ncnn 的兼容性较好，转换用的是修改后的 `mmdeploy_onnx2ncnn`，推理封装了 `pyncnn`+ 自定义 ops。

遇到 snpe 这种不支持 python 的框架，则使用 C/S 模式：用 gRPC 等协议封装一个 server，转发真实的推理结果。

渲染使用上游算法框架的渲染 API，mmdeploy 自身不做绘制。

## 三、SDK

## 三、SDK

模型转换完成后，可用 C++ 编译的 SDK 执行在不同平台上。

我们看下 csrc/mmdeploy 目录结构：

```bash
.
├── apis           # Csharp、java、go、Rust 等 ffi 接口
├── backend_ops    # 各推理框架的自定义算子
├── codebase       # 各 mm 算法框架偏好的结果类型，例如检测任务多用 bbox
├── core           # 脚手架，对图、算子、设备的抽象
├── device         # CPU/GPU device 抽象的实现
├── execution      # 对 exec 抽象的实现
├── graph          # 对图抽象的实现
├── model          # 实现 zip 压缩和非压缩两种工作目录
├── net            # net 的具体实现，例如封装了 ncnn forward C 接口
├── preprocess     # 预处理的实现
└── utils          # OCV 工具类
```

SDK 本质是设计了一套计算图的抽象，把**多个模型**的

- 预处理
- 推理
- 后处理

调度起来，同时提供多种语言的 ffi。

- [ ] I have read and followed the workflow indicated in the [CONTRIBUTING.md](https://github.com/open-mmlab/mmagic/blob/main/.github/CONTRIBUTING.md) to create this PR.
- [ ] Pre-commit or linting tools indicated in [CONTRIBUTING.md](https://github.com/open-mmlab/mmagic/blob/main/.github/CONTRIBUTING.md) are used to fix the potential lint issues.

## Usage

Please refer to the [building documentation of MMCV](https://mmcv.readthedocs.io/en/latest/get_started/build.html#build-mmcv-full-on-ascend-npu-machine) to install MMCV and [mmengine](https://mmengine.readthedocs.io/en/latest/get_started/installation.html#build-from-source) on NPU devices.

```shell
```

```shell
```

# 自定义数据处理流程

## 数据流的设计

在[新数据集教程](./datasets.md)中，我们知道数据集类使用 `load_data_list` 方法来初始化整个数据集，我们将每个样本的信息保存到一个 dict 中。

通常，为了节省内存，我们只加载 `load_data_list` 中的图片路径和标签，使用时加载完整的图片内容。此外，我们可能希望在训练时选择样本时进行一些随机数据扩充。几乎所有的数据加载、预处理和格式化操作都可以通过**数据管道**在 MMPretrain 中进行配置。

数据管道意味着在从数据集中索引样本时如何处理样本字典，它由一系列数据变换组成。每个数据变换都将一个字典作为输入，对其进行处理，并为下一个数据变换输出一个字典。

这是 ImageNet 上 ResNet-50 训练的数据管道示例。

```python
]
```

MMPretrain 中所有可用的数据变换都可以在 [数据变换文档](mmpretrain.datasets.transforms) 中找到。

## 修改训练/测试管道

MMPretrain 中的数据管道非常灵活。您几乎可以从配置文件中控制数据预处理的每一步，但另一方面，面对如此多的选项，您可能会感到困惑。

这是图像分类任务的常见做法和指南。

### 读取

在数据管道的开始，我们通常需要从文件路径加载图像数据。
[`LoadImageFromFile`](mmcv.transforms.LoadImageFromFile) 通常用于执行此任务。

```python
    ...
]
```

### 读取

```python
    ...
]
```

如果您想从具有特殊格式或特殊位置的文件中加载数据，您可以 [实施新的加载变换](#添加新的数据变换) 并将其添加到数据管道的开头。

### 增强和其它处理

在训练过程中，我们通常需要做数据增强来避免过拟合。在测试过程中，我们还需要做一些数据处理，比如调整大小和裁剪。这些数据变换将放置在加载过程之后。

这是一个简单的数据扩充方案示例。它会将输入图像随机调整大小并裁剪到指定比例，并随机水平翻转图像。

```python
    ...
    ...
]
```

这是 [Swin-Transformer](../papers/swin_transformer.md) 训练中使用的大量数据增强配方示例。 为了与官方实施保持一致，它指定 `pillow` 作为调整大小后端，`bicubic` 作为调整大小算法。 此外，它添加了 [`RandAugment`](mmpretrain.datasets.transforms.RandAugment) 和 [`RandomErasing`](mmpretrain.datasets.transforms.RandomErasing) 作为额外的数据增强方法。

此配置指定了数据扩充的每个细节，您只需将其复制到您自己的配置文件中即可应用 Swin-Transformer 的数据扩充。

```python

    ...

```python

    ...
    dict(
    dict(
    ...
]
```

```{note}

erase_prob=0.25,
    ...
]
```

```{note}
通常，数据管道中的数据增强部分仅处理图像方面的变换，而不处理图像归一化或混合/剪切混合等变换。 因为我们可以对 batch data 做 image normalization 和 mixup/cutmix 来加速。要配置图像归一化和 mixup/cutmix，请使用 [数据预处理器](mmpretrain.models.utils.data_preprocessor)。
```

### 格式化

格式化是从数据信息字典中收集训练数据，并将这些数据转换为模型友好的格式。

在大多数情况下，您可以简单地使用 [`PackInputs`](mmpretrain.datasets.transforms.PackInputs)，它将 NumPy 数组格式的图像转换为 PyTorch 张量，并将 ground truth 类别信息和其他元信息打包为 [`DataSample`](mmpretrain.structures.DataSample)。

```python
    ...
]
```

## 添加新的数据变换

1. 在任何文件中写入一个新的数据转换，例如 `my_transform.py`，并将其放在文件夹 `mmpretrain/datasets/transforms/` 中。 数据变换类需要继承 [`mmcv.transforms.BaseTransform`](mmcv.transforms.BaseTransform) 类并覆盖以字典作为输入并返回字典的 `transform` 方法。

   ```python

```python

   ```

2. 在 `mmpretrain/datasets/transforms/__init__.py` 中导入新的变换

   ```python
   ...

   __all__ = [
   ]
   ```

3. 在配置文件中使用

   ```python
       ...
       ...
   ]
   ```

## 数据管道可视化

数据流水线设计完成后，可以使用 [可视化工具](../useful_tools/dataset_visualization.md) 查看效果。

# 准备 DF2K_OST 数据集

```bibtex
}
```

- DIV2K 数据集可以在 [这里](https://data.vision.ee.ethz.ch/cvl/DIV2K/) 下载 (我们只使用训练集)。
- Flickr2K 数据集可以在 [这里](https://cv.snu.ac.kr/research/EDSR/Flickr2K.tar) 下载 (我们只使用训练集)。
- OST 数据集可以在 [这里](https://openmmlab.oss-cn-hangzhou.aliyuncs.com/datasets/OST_dataset.zip) 下载 (我们只使用训练集)。

请先将所有图片放入 `GT` 文件夹（命名不需要按顺序）：

```text
mmagic
├── mmagic
├── tools
├── data
│   │   ├── GT
│   │   │   ├── ...
...
```

## 裁剪子图像

为了更快的 IO，我们建议将图像裁剪为子图像。 我们提供了这样一个脚本：

```shell

│   │   │   ├── ...
...
```

## 裁剪子图像

```shell
```

生成的数据存放在 `df2k_ost` 下，数据结构如下，其中 `_sub` 表示子图像。

```text
mmagic
├── mmagic
├── tools
├── data
│   │   ├── GT
...
```

标注文件示例:

```text
```

请注意，`preprocess_df2k_ost_dataset.py` 脚本默认生成一份标注文件。

如果你想使用 LMDB 数据集来获得更快的 IO 速度，你可以通过以下方式制作 LMDB 文件：

```shell
```

# Build for RKNN

This tutorial is based on Ubuntu-18.04 and Rockchip NPU `rk3588`. For different NPU devices, you may have to use different rknn packages.
Below is a table describing the relationship:

| Device               | Python Package                                                   | c/c++ SDK                                          |
| -------------------- | ---------------------------------------------------------------- | -------------------------------------------------- |
| RK1808/RK1806        | [rknn-toolkit](https://github.com/rockchip-linux/rknn-toolkit)   | [rknpu](https://github.com/rockchip-linux/rknpu)   |
| RV1109/RV1126        | [rknn-toolkit](https://github.com/rockchip-linux/rknn-toolkit)   | [rknpu](https://github.com/rockchip-linux/rknpu)   |
| RK3566/RK3568/RK3588 | [rknn-toolkit2](https://github.com/rockchip-linux/rknn-toolkit2) | [rknpu2](https://github.com/rockchip-linux/rknpu2) |

| RV1103/RV1106        | [rknn-toolkit2](https://github.com/rockchip-linux/rknn-toolkit2) | [rknpu2](https://github.com/rockchip-linux/rknpu2) |

1. Get RKNN-Toolkit2 or RKNN-Toolkit through git. RKNN-Toolkit2 for example:

   ```
   git clone git@github.com:rockchip-linux/rknn-toolkit2.git
   ```

2. Install RKNN python package following [rknn-toolkit2 doc](https://github.com/rockchip-linux/rknn-toolkit2/tree/master/doc) or [rknn-toolkit doc](https://github.com/rockchip-linux/rknn-toolkit/tree/master/doc). When installing rknn python package, it is better to append `--no-deps` after the commands to avoid dependency conflicts. RKNN-Toolkit2 package for example:

   ```
   pip install packages/rknn_toolkit2-1.4.0_22dcfef4-cp36-cp36m-linux_x86_64.whl --no-deps

```
   ```

3. Install ONNX==1.8.0 before reinstall MMDeploy from source following the [instructions](../01-how-to-build/build_from_source.md). Note that there are conflicts between the pip dependencies of MMDeploy and RKNN. Here is the suggested packages versions for python 3.6:

   ```
   protobuf==3.19.4
   onnx==1.8.0
   onnxruntime==1.8.0
   torch==1.8.0
   torchvision==0.9.0
   ```

4. Install torch and torchvision using conda. For example:

```
```

To work with models from [MMPretrain](https://mmpretrain.readthedocs.io/en/latest/get_started.html), you may need to install it additionally.

## Usage

Example:

```bash
    configs/mmpretrain/classification_rknn-fp16_static-224x224.py \
    /mmpretrain_dir/configs/resnet/resnet50_8xb32_in1k.py \

## Usage

Example:

```bash
    https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_batch256_imagenet_20200708-cfb998bf.pth \
    /mmpretrain_dir/demo/demo.JPEG \
    --work-dir ../resnet50 \
```

With the deployment config, you can modify the `backend_config` for your preference. An example `backend_config` of mmpretrain is shown as below:

```python
    type='rknn',
        mean_values=None,
        std_values=None,
        target_platform='rk3588',
        optimization_level=3),
    quantization_config=dict(do_quantization=False, dataset=None),
    input_size_list=[[3, 224, 224]])

```

std_values=None,

```

The contents of `common_config` are for `rknn.config()`. The contents of `quantization_config` are used to control `rknn.build()`. You may have to modify `target_platform` for your own preference.

## Build SDK with Rockchip NPU

### Build SDK with RKNPU2

1. Get rknpu2 through git:

   ```
   git clone git@github.com:rockchip-linux/rknpu2.git
   ```

   ```
   ```

2. For linux, download gcc cross compiler. The download link of the compiler from the official user guide of `rknpu2` was deprecated. You may use another verified [link](https://github.com/Caesar-github/gcc-buildroot-9.3.0-2020.03-x86_64_aarch64-rockchip-linux-gnu). After download and unzip the compiler, you may open the terminal, set `RKNN_TOOL_CHAIN` and `RKNPU2_DEVICE_DIR` by `export RKNN_TOOL_CHAIN=/path/to/gcc/usr;export RKNPU2_DEVICE_DIR=/path/to/rknpu2/runtime/RK3588`.

3. after the above preparition, run the following commands:

```shell
mkdir -p build && rm -rf build/CM* && cd build
export LD_LIBRARY_PATH=$RKNN_TOOL_CHAIN/lib64:$LD_LIBRARY_PATH
cmake \
    -DCMAKE_TOOLCHAIN_FILE=/path/to/mmdeploy/cmake/toolchains/rknpu2-linux-gnu.cmake \
    -DCMAKE_BUILD_TYPE=Debug \

cmake \
    -DOpenCV_DIR=${RKNPU2_DEVICE_DIR}/../../examples/3rdparty/opencv/opencv-linux-aarch64/share/OpenCV \
    -DMMDEPLOY_TARGET_DEVICES="cpu" \
    -DMMDEPLOY_TARGET_BACKENDS="rknn" \
    -DMMDEPLOY_BUILD_TEST=ON \
    ..
make && make install
```

## Run the demo with SDK

First make sure that`--dump-info`is used during convert model, so that the working directory has the files required by the SDK such as `pipeline.json`.

```bash
adb push resnet50  /data/local/tmp/resnet50
adb push /mmpretrain_dir/demo/demo.JPEG /data/local/tmp/resnet50/demo.JPEG
cd build
adb push lib /data/local/tmp/lib

```bash
cd build
adb push bin/image_classification /data/local/tmp/image_classification
```

```bash
adb shell
cd /data/local/tmp
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/data/local/tmp/lib
./image_classification cpu ./resnet50  ./resnet50/demo.JPEG
..
label: 65, score: 0.95
```

- MMDet models.

  YOLOV3 & YOLOX: you may paste the following partition configuration into [detection_rknn_static-320x320.py](https://github.com/open-mmlab/mmdeploy/blob/main/configs/mmdet/detection/detection_rknn-int8_static-320x320.py):

  ```python
  # yolov3, yolox for rknn-toolkit and rknn-toolkit2
  partition_config = dict(
      type='rknn',  # the partition policy name
      apply_marks=True,  # should always be set to True
      partition_cfg=[

          dict(
              save_file='model.onnx',  # name to save the partitioned onnx
              start=['detector_forward:input'],  # [mark_name:input, ...]
              end=['yolo_head:input'],  # [mark_name:output, ...]
              output_names=[f'pred_maps.{i}' for i in range(3)]) # output names
      ])
  ```

  RTMDet: you may paste the following partition configuration into [detection_rknn-int8_static-640x640.py](https://github.com/open-mmlab/mmdeploy/blob/main/configs/mmdet/detection/detection_rknn-int8_static-640x640.py):

  ```python
  # rtmdet for rknn-toolkit and rknn-toolkit2
          dict(

          dict(
              end=['rtmdet_head:output'],  # [mark_name:output, ...]
              output_names=[f'pred_maps.{i}' for i in range(6)]) # output names
      ])
  ```

  RetinaNet & SSD & FSAF with rknn-toolkit2, you may paste the following partition configuration into [detection_rknn_static-320x320.py](https://github.com/open-mmlab/mmdeploy/blob/main/configs/mmdet/detection/detection_rknn-int8_static-320x320.py). Users with rknn-toolkit can directly use default config.

  ```python
  # retinanet, ssd for rknn-toolkit2
      apply_marks=True,
          dict(

```python
          dict(
              save_file='model.onnx',
              start='detector_forward:input',
              end=['BaseDenseHead:output'],
              output_names=[f'BaseDenseHead.cls.{i}' for i in range(5)] +
              [f'BaseDenseHead.loc.{i}' for i in range(5)])
      ])
  ```

- SDK only supports int8 rknn model, which require `do_quantization=True` when converting models.

- Latency problem.

  For devices running RKNPU like rv1126, please set `pre_compile=True` in `quantization_config` when converting models.
  Or the latency may not suit your need.

You can plot loss/mAP curves given a training log file. Run `pip install seaborn` first to install the dependency.

```shell
```

**Notice**: If the metric you want to plot is calculated in the eval stage, you need to add the flag `--mode eval`. If you perform evaluation with an interval of `${INTERVAL}`, you need to add the args `--interval ${INTERVAL}`.

Examples:

  ```shell
  ```

  ```shell

```

  ```shell
  ```

  ```shell
  # evaluate PartA2 and second on KITTI according to Car_3D_moderate_strict
  # evaluate PointPillars for car and 3 classes on KITTI according to Car_3D_moderate_strict
  ```

You can also compute the average training speed.

```shell

```

```shell
```

The output is expected to be like the following.

```
```

&#8195;

**Note**: This tool is still experimental now, only SECOND is supported to be served with [`TorchServe`](https://pytorch.org/serve/). We'll support more models in the future.

In order to serve an `MMDetection3D` model with [`TorchServe`](https://pytorch.org/serve/), you can follow the steps:

### 1. Convert the model from MMDetection3D to TorchServe

```shell
```

```shell
```

### 2. Build `mmdet3d-serve` docker image

```shell
```

### 3. Run `mmdet3d-serve`

In order to run it on the GPU, you need to install [nvidia-docker](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html). You can omit the `--gpus` argument in order to run on the CPU.

Example:

```shell
--cpus 8 \
```

```shell
--cpus 8 \
```

You can use `test_torchserver.py` to compare result of torchserver and pytorch.

```shell
```

Example:

```shell
```

&#8195;

```

&#8195;

You can use `tools/analysis_tools/get_flops.py` in MMDetection3D, a script adapted from [flops-counter.pytorch](https://github.com/sovrasov/flops-counter.pytorch), to compute the FLOPs and params of a given model.

```shell
```

```text
Input shape: (40000, 4)
```

   input shape is (1, 40000, 4).

2. Some operators are not counted into FLOPs like GN and custom operators. Refer to [`mmcv.cnn.get_model_complexity_info()`](https://github.com/open-mmlab/mmcv/blob/master/mmcv/cnn/utils/flops_counter.py) for details.
3. We currently only support FLOPs calculation of single-stage models with single-modality input (point cloud or image). We will support two-stage and multi-modality models in the future.

&#8195;

```shell
```

`tools/detectron2pytorch.py` in MMDetection could convert keys in the original detectron pretrained

```

```shell
```

   filename.

```shell
```

E.g.,

```shell
```

&#8195;

```

&#8195;

`tools/dataset_converters/` contains tools for converting datasets to other formats. Most of them convert datasets to pickle based info files, like kitti, nuscenes and lyft. Waymo converter is used to reorganize waymo raw data like KITTI style. Users could refer to them for our approach to converting data format. It is also convenient to modify them to use as scripts like nuImages converter.

To convert the nuImages dataset into COCO format, please use the command below:

```shell
```

--out-dir ${OUT_DIR} --nproc ${NUM_WORKERS} --extra-tag ${TAG}
```

More details could be referred to the [doc](https://mmdetection3d.readthedocs.io/en/latest/data_preparation.html) for dataset preparation and [README](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/nuimages/README.md/) for nuImages dataset.

&#8195;

imports.

```shell
```

# 2D Wholebody Keypoint Datasets

- [COCO-WholeBody](#coco-wholebody) \[ [Homepage](https://github.com/jin-s13/COCO-WholeBody/) \]
- [Halpe](#halpe) \[ [Homepage](https://github.com/Fang-Haoshu/Halpe-FullBody/) \]

## COCO-WholeBody

<details>

```bibtex
}
```

}
```

  <img src="https://user-images.githubusercontent.com/100993824/227770977-c8f00355-c43a-467e-8444-d307789cf4b2.png" height="300px">
</div>

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── coco
        │   │-- ...
            │-- ...

```

Please also install the latest version of [Extended COCO API](https://github.com/jin-s13/xtcocoapi) (version>=1.5) to support COCO-WholeBody evaluation:

## Halpe

<details>

```bibtex

## Halpe

<details>

```bibtex
}
```

  <img src="https://user-images.githubusercontent.com/100993824/227771087-b839ea5b-4461-4ba7-8a9a-823b78e2ca44.png" height="300px">
</div>

For [Halpe](https://github.com/Fang-Haoshu/Halpe-FullBody/) dataset, please download images and annotations from [Halpe download](https://github.com/Fang-Haoshu/Halpe-FullBody).

</div>

The images of the training set are from [HICO-Det](https://drive.google.com/open?id=1QZcJmGVlF9f4h-XLWe9Gkmnmj2z1gSnk) and those of the validation set are from [COCO](http://images.cocodataset.org/zips/val2017.zip).

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
    │── halpe
        │   │-- halpe_train_v1.json
        │   |-- halpe_val_v1.json

        │-- hico_20160224_det
        │   │-- anno_bbox.mat
        │   │-- anno.mat
        │   │-- README
        │   │   │-- train2015
        │   │   │   │-- HICO_train2015_00000001.jpg
        │   │   │   │-- HICO_train2015_00000002.jpg
        │   │   │   │-- HICO_train2015_00000003.jpg
        │   │   │   │-- ...
        │   │   │-- test2015
        │   │-- tools
        │   │-- ...
            │-- ...

```

Please also install the latest version of [Extended COCO API](https://github.com/jin-s13/xtcocoapi) (version>=1.5) to support Halpe evaluation:

# 测试模型重写

模型 [rewriter](support_new_model.md) 完成后，还需完成对应测试用例，以验证重写是否生效。通常我们需要对比原始模型和重写后的输出。原始模型输出可以调用模型的 forward 函数直接获取，而生成重写模型输出的方法取决于重写的复杂性。

## 测试简单的重写

如果对模型的更改很小（例如，仅更改一个或两个变量且无副作用），则可为重写函数/模块构造输入，在`RewriteContext`中运行推理并检查结果。

```python
        else:

```

在示例中，我们仅更改 forward 函数。我们可以通过编写以下函数来测试这个重写：

```python

return self.simple_test(img, {})
```

```python

            pass

```

在这个测试函数中，我们构造派生类 `BaseClassifier` 来测试重写能否工作。通过直接调用`model(input)`来获得原始输出，并通过在`RewriteContext`中调用`model(input)`来获取重写的输出。最后断检查输出。

## 测试复杂重写

assert backend_output == 'simple_test'
```

## 测试复杂重写

有时我们可能会对原始模型函数进行重大更改（例如，消除分支语句以生成正确的计算图）。即使运行在Python中的重写模型的输出是正确的，我们也不能保证重写的模型可以在后端按预期工作。因此，我们需要在后端测试重写的模型。

```python

```

此重写函数的行为很复杂，我们应该按如下方式测试它：

```python

img_shape = [int(val) for val in img_shape]

```

```python

    # ...

    }

rewrite_outputs, is_backend_output = get_rewrite_outputs(
    else:
```

我们已经提供了一些使用函数做测试，例如可以先 build 模型，用  `get_model_outputs` 获取原始输出；然后用`WrapModel` 包装重写函数，使用`get_rewrite_outputs` 获取结果。这个例子里会返回输出内容和是否来自后端两个结果。

因为我们也不确定用户是否正确安装后端，所以得检查结果来自 Python 还是真实后端推理结果。单元测试必须涵盖这两种结果，最后用`torch.allclose` 对比两种结果的差异。

API 文档中有测试用例完整用法。

# How to prepare your own datasets

In this document, we will introduce the design of each datasets in MMagic and how users can design their own dataset.

- [How to prepare your own datasets](#how-to-prepare-your-own-datasets)
  - [Supported Data Format](#supported-data-format)
    - [BasicImageDataset](#basicimagedataset)
    - [BasicFramesDataset](#basicframesdataset)
    - [BasicConditonalDataset](#basicconditonaldataset)
      - [1. Annotation file read by line (e.g., txt)](#1-annotation-file-read-by-line-eg-txt)
      - [2. Dict-based annotation file (e.g., json):](#2-dict-based-annotation-file-eg-json)
      - [3. Folder-based annotation (no annotation file need):](#3-folder-based-annotation-no-annotation-file-need)
    - [ImageNet Dataset and CIFAR10 Dataset](#imagenet-dataset-and-cifar10-dataset)
    - [AdobeComp1kDataset](#adobecomp1kdataset)
    - [GrowScaleImgDataset](#growscaleimgdataset)
    - [SinGANDataset](#singandataset)
    - [PairedImageDataset](#pairedimagedataset)

- [AdobeComp1kDataset](#adobecomp1kdataset)
    - [UnpairedImageDataset](#unpairedimagedataset)
  - [Design a new dataset](#design-a-new-dataset)

## Supported Data Format

In MMagic, all datasets are inherited from `BaseDataset`.
Each dataset load the list of data info (e.g., data path) by `load_data_list`.
In `__getitem__`, `prepare_data` is called to get the preprocessed data.
In `prepare_data`, data loading pipeline consists of the following steps:

1. fetch the data info by passed index, implemented by `get_data_info`
2. apply data transforms to the data, implemented by `pipeline`

### BasicImageDataset

**BasicImageDataset** `mmagic.datasets.BasicImageDataset`

General image dataset designed for low-level vision tasks with image, such as image super-resolution, inpainting and unconditional image generation. The annotation file is optional.

If use annotation file, the annotation format can be shown as follows.

```bash
   Case 1 (CelebA-HQ):

       000001.png
       000002.png

   Case 2 (DIV2K):

       0001_s001.png (480,480,3)
       0001_s002.png (480,480,3)
       0001_s003.png (480,480,3)
       0002_s001.png (480,480,3)
       0002_s002.png (480,480,3)

   Case 3 (Vimeo90k):

       00001/0266 (256, 448, 3)
       00001/0268 (256, 448, 3)
```

Here we give several examples showing how to use `BasicImageDataset`. Assume the file structure as the following:

```md
mmagic (root)
├── mmagic
├── tools
├── data
│   ├── DIV2K
│   │   │   ├── image.png

```md
├── mmagic
├── tools
├── data
│   ├── DIV2K
│   │   │   ├── X2
│   │   │   ├── X3
│   │   │   ├── X4
│   │   │   │   ├── image_x4.png
│   │   │   ├── X2
│   │   │   ├── X3
│   │   │   ├── X4
│   ├── places
│   │   ├── test_set
│   │   ├── train_set
|   |   ├── meta
|   |   |    ├── Places365_train.txt
|   |   |    ├── Places365_val.txt
|   ├── celebahq
│   │   ├── imgs_1024

```

Case 1: Loading DIV2K dataset for training a SISR model.

```python
   dataset = BasicImageDataset(
       ann_file='',
       metainfo=dict(
           dataset_type='div2k',
           task_name='sisr'),
       data_root='data/DIV2K',
       data_prefix=dict(
           gt='DIV2K_train_HR', img='DIV2K_train_LR_bicubic/X4'),
       filename_tmpl=dict(img='{}_x4', gt='{}'),
       pipeline=[])
```

data_root='data/DIV2K',
```

Case 2: Loading places dataset for training an inpainting model.

```python
       ann_file='meta/Places365_train.txt',
           dataset_type='places365',
           task_name='inpainting'),
       data_root='data/places',
       data_prefix=dict(gt='train_set'),
```

Case 3: Loading CelebA-HQ dataset for training an PGGAN.

```python
dataset = BasicImageDataset(
        pipeline=[],
        data_root='./data/celebahq/imgs_1024')
```

### BasicFramesDataset

**BasicFramesDataset** `mmagic.datasets.BasicFramesDataset`
General frames dataset designed for low-level vision tasks with frames, such as video super-resolution and video frame interpolation. The annotation file is optional.

```bash
Case 1 (Vid4):

   calendar 41
   city 34
   foliage 49
   walk 47

Case 2 (REDS):

   000/00000000.png (720, 1280, 3)
   000/00000001.png (720, 1280, 3)

Case 3 (Vimeo90k):

   00001/0266 (256, 448, 3)
   00001/0268 (256, 448, 3)
```

Assume the file structure as the following:

```bash
├── mmagic
├── tools
├── data
│   ├── Vid4
│   │   ├── BIx4
│   │   │   │   ├── img1.png
│   │   ├── GT
|   |   |   ├── 00001
│   │   │   │   ├── 0389
│   │   │   │   │   ├── img1.png
│   │   │   │   │   ├── img2.png
│   │   │   │   │   ├── img3.png
```

```

Case 1: Loading Vid4 dataset for training a VSR model.

```python
dataset = BasicFramesDataset(
    ann_file='meta_info_Vid4_GT.txt',
    metainfo=dict(dataset_type='vid4', task_name='vsr'),
    data_root='data/Vid4',
    data_prefix=dict(img='BIx4', gt='GT'),
    pipeline=[],
    depth=2,
    num_input_frames=5)
```

Case 2: Loading Vimeo90k dataset for training a VFI model.

```python
    ann_file='tri_trainlist.txt',
    metainfo=dict(dataset_type='vimeo90k', task_name='vfi'),
    data_root='data/vimeo-triplet',
    data_prefix=dict(img='sequences', gt='sequences'),
    depth=2,
    load_frames_list=dict(
        img=['img1.png', 'img3.png'], gt=['img2.png']))
```

### BasicConditonalDataset

data_prefix=dict(img='sequences', gt='sequences'),
    depth=2,
```

**BasicConditonalDataset** `mmagic.datasets.BasicConditonalDataset` is designed for conditional GANs (e.g., SAGAN, BigGAN). This dataset support load label for the annotation file. `BasicConditonalDataset` support three kinds of annotation as follow:

#### 1. Annotation file read by line (e.g., txt)

Sample files structure:

```
    data_prefix/
    ├── folder_1
    │   ├── xxx.png
    │   ├── xxy.png
    │   └── ...
    └── folder_2
        ├── 123.png
        ├── nsdf3.png
        └── ...
```

Sample annotation file (the first column is the image path and the second column is the index of category):

```
    folder_1/xxx.png 0
    folder_1/xxy.png 1
    folder_2/123.png 5
    folder_2/nsdf3.png 3
    ...
```

Config example for ImageNet dataset:

```python

```
    ...
```

```python
    type='BasicConditionalDataset,
    data_root='./data/imagenet/',
    ann_file='meta/train.txt',
    data_prefix='train',
    pipeline=train_pipeline),
```

#### 2. Dict-based annotation file (e.g., json):

```
    │   └── ...
        └── ...
```

Sample annotation file (the key is the image path and the value column
is the label):

```
    {
        "folder_1/xxx.png": [1, 2, 3, 4],
        "folder_1/xxy.png": [2, 4, 1, 0],
        "folder_2/123.png": [0, 9, 8, 1],
        "folder_2/nsdf3.png", [1, 0, 0, 2],
        ...
    }
```

Config example for EG3D (shapenet-car) dataset:

```python
    type='BasicConditionalDataset',

"folder_2/nsdf3.png", [1, 0, 0, 2],
        ...
    }
```

```python
    data_root='./data/eg3d/shapenet-car',
    ann_file='annotation.json',
```

In this kind of annotation, labels can be any type and not restricted to an index.

#### 3. Folder-based annotation (no annotation file need):

```
    ├── class_x
    │   └── ...
    │       └── xxz.png
    └── class_y
        ├── ...
        └── asd932_.png
```

If the annotation file is specified, the dataset will be generated by the first two ways, otherwise, try the third way.

### ImageNet Dataset and CIFAR10 Dataset

└── asd932_.png
```

**ImageNet Dataset** `mmagic.datasets.ImageNet` and **CIFAR10 Dataset**`mmagic.datasets.CIFAR10` are datasets specific designed for ImageNet and CIFAR10 datasets. Both two datasets are encapsulation of `BasicConditionalDataset`. You can used them to load data from ImageNet dataset and CIFAR10 dataset easily.

Config example for ImageNet:

```python
    dict(type='RandomCropLongEdge', keys=['img']),
    dict(type='Resize', scale=(128, 128), keys=['img'], backend='pillow'),
    dict(type='Flip', keys=['img'], flip_ratio=0.5, direction='horizontal'),
]

    type='ImageNet',
    pipeline=pipeline),
```

dict(type='PackInputs')
]

```

Config example for CIFAR10:

```python
pipeline = [dict(type='PackInputs')]

    type='CIFAR10',
    data_root='./data',
    data_prefix='cifar10',
```

### AdobeComp1kDataset

**AdobeComp1kDataset** `mmagic.datasets.AdobeComp1kDataset`
Adobe composition-1k dataset.

The dataset loads (alpha, fg, bg) data and apply specified transforms to
the data. You could specify whether composite merged image online or load
composited merged image in pipeline.

Example for online comp-1k dataset:

```md
[
   {
       "alpha": 'alpha/000.png',
       "fg": 'fg/000.png',
       "bg": 'bg/000.png'
   },
   {
       "alpha": 'alpha/001.png',
       "fg": 'fg/001.png',
       "bg": 'bg/001.png'
   },
]
```

Example for offline comp-1k dataset:

```md
[
  {

"bg": 'bg/000.png'
   },
   {
   },
]
```

```md
[
  {
      "alpha": 'alpha/000.png',
      "merged": 'merged/000.png',
      "fg": 'fg/000.png',
      "bg": 'bg/000.png'
  },
  {
      "alpha": 'alpha/001.png',
      "merged": 'merged/001.png',
      "fg": 'fg/001.png',
      "bg": 'bg/001.png'
  },
]
```

### GrowScaleImgDataset

`GrowScaleImgDataset` is designed for dynamic GAN models (e.g., PGGAN and StyleGANv1).
In this dataset, we support switching the data root during training to load training images of different resolutions.
This procedure is implemented by `GrowScaleImgDataset.update_annotations` and is called by `PGGANFetchDataHook.before_train_iter` in the training process.

```python
def update_annotations(self, curr_scale):
    # determine if the data root needs to be updated
    if curr_scale == self._actual_curr_scale:
        return False

```python

    # fetch new data root by resolution (scale)
    for scale in self._img_scales:
        if curr_scale <= scale:
            self._curr_scale = scale
            break
        if scale == self._img_scales[-1]:
            assert RuntimeError(
                f'Cannot find a suitable scale for {curr_scale}')
    self._actual_curr_scale = curr_scale
    self.data_root = self.data_roots[str(self._curr_scale)]

    # reload the data list with new data root
    self.load_data_list()

    # print basic dataset information to check the validity
    print_log('Update Dataset: ' + repr(self), 'current')
    return True
```

### SinGANDataset

`SinGANDataset` is designed for SinGAN's training.
In SinGAN's training, we do not iterate the images in the dataset but return a consistent preprocessed image dict.

```

Therefore, we bypass the default data loading logic of `BaseDataset` because we do not need to load the corresponding image data based on the given index.

```python
def load_data_list(self, min_size, max_size, scale_factor_init):
    # load single image
    real = mmcv.imread(self.data_root)
    self.reals, self.scale_factor, self.stop_scale = create_real_pyramid(
        real, min_size, max_size, scale_factor_init)

    self.data_dict = {}

    # generate multi scale image
    for i, real in enumerate(self.reals):
        self.data_dict[f'real_scale{i}'] = real

    self.data_dict['input_sample'] = np.zeros_like(
        self.data_dict['real_scale0']).astype(np.float32)

def __getitem__(self, index):
    # directly return the transformed data dict
    return self.pipeline(self.data_dict)
```

self.data_dict['real_scale0']).astype(np.float32)

```

### PairedImageDataset

`PairedImageDataset` is designed for translation models that needs paired training data (e.g., Pix2Pix).
The directory structure is shown below. Each image files are the concatenation of the image pair.

```
./data/dataset_name/
├── test
│   └── XXX.jpg
└── train
    └── XXX.jpg
```

In `PairedImageDataset`, we scan the file list in `load_data_list` and save path in `pair_path` field to fit the `LoadPairedImageFromFile` transformation.

```python
def load_data_list(self):
    data_infos = []
    pair_paths = sorted(self.scan_folder(self.data_root))
    for pair_path in pair_paths:
        # save path in the specific field
        data_infos.append(dict(pair_path=pair_path))

    return data_infos
```

### UnpairedImageDataset

for pair_path in pair_paths:

```

`UnpairedImageDataset` is designed for translation models that do not need paired data (e.g., CycleGAN). The directory structure is shown below.

```
├── testA
├── testB
├── trainA
└── trainB

```

In this dataset, we overwrite `__getitem__` function to load random image pair in the training process.

```python
def __getitem__(self, idx):
    if not self.test_mode:
        return self.prepare_train_data(idx)

    return self.prepare_test_data(idx)

def prepare_train_data(self, idx):
    img_a_path = self.data_infos_a[idx % self.len_a]['path']
    idx_b = np.random.randint(0, self.len_b)
    img_b_path = self.data_infos_b[idx_b]['path']
    results = dict()
    results[f'img_{self.domain_a}_path'] = img_a_path

idx_b = np.random.randint(0, self.len_b)
    results[f'img_{self.domain_b}_path'] = img_b_path
    return self.pipeline(results)

def prepare_test_data(self, idx):
    img_b_path = self.data_infos_b[idx % self.len_b]['path']
```

## Design a new dataset

If you want to create a dataset for a new low level CV task (e.g. denoise, derain, defog, and de-reflection) or existing dataset format doesn't meet your need, you can reorganize new data formats to existing format.

Or create a new dataset in `mmagic/datasets` to load the data.

Inheriting from the base class of datasets such as `BasicImageDataset` and `BasicFramesDataset` will make it easier to create a new dataset.

And you can create a new dataset inherited from [BaseDataset](https://github.com/open-mmlab/mmengine/blob/main/mmengine/dataset/base_dataset.py) which is the base class of datasets in [MMEngine](https://github.com/open-mmlab/mmengine).

Here is an example of creating a dataset for video frame interpolation:

```python
from .basic_frames_dataset import BasicFramesDataset
from mmagic.registry import DATASETS

class NewVFIDataset(BasicFramesDataset):
    """Introduce the dataset

    Examples of file structure.

    Args:
        pipeline (list[dict | callable]): A sequence of data transformations.
        folder (str | :obj:`Path`): Path to the folder.
        ann_file (str | :obj:`Path`): Path to the annotation file.

pipeline (list[dict | callable]): A sequence of data transformations.
        test_mode (bool): Store `True` when building test dataset.
            Default: `False`.
    """

    def __init__(self, ann_file, metainfo, data_root, data_prefix,
                    pipeline, test_mode=False):
        super().__init__(ann_file, metainfo, data_root, data_prefix,
                            pipeline, test_mode)
        self.data_infos = self.load_annotations()

    def load_annotations(self):
        """Load annoations for the dataset.

        Returns:
            list[dict]: A list of dicts for paired paths and other information.
        """
        ...

```

Welcome to [submit new dataset classes to MMagic](https://github.com/open-mmlab/mmagic/compare).

"""
        ...

```

We use [RepeatDataset](https://github.com/open-mmlab/mmengine/blob/main/mmengine/dataset/dataset_wrapper.py) as wrapper to repeat the dataset.
For example, suppose the original dataset is Dataset_A, to repeat it, the config looks like the following

```python
        times=N,
            ...
        )
    )
```

You may refer to [tutorial in MMEngine](https://github.com/open-mmlab/mmengine/blob/main/docs/en/advanced_tutorials/basedataset.md).

# Preparing Skeleton Dataset

```BibTeX
}
```

We release the skeleton annotations used in [Revisiting Skeleton-based Action Recognition](https://arxiv.org/abs/2104.13586). By default, we use [Faster-RCNN](https://github.com/open-mmlab/mmdetection/blob/master/configs/faster_rcnn/faster_rcnn_r50_caffe_fpn_mstrain_1x_coco-person.py) with ResNet50 backbone for human detection and [HRNet-w32](https://github.com/open-mmlab/mmpose/blob/master/configs/body/2d_kpt_sview_rgb_img/topdown_heatmap/coco/hrnet_w32_coco_256x192.py) for single person pose estimation. For FineGYM, we use Ground-Truth bounding boxes for the athlete instead of detection bounding boxes.

## Prepare Annotations

We provide links to the pre-processed skeleton annotations, you can directly download them and use them for training & testing.

  - GYM 2D skeletons are extracted with ground-truth human bounding boxes, which can be downloaded with [link](https://download.openmmlab.com/mmaction/pyskl/data/gym/gym_gt_bboxes.pkl). Please cite [PoseConv3D](https://arxiv.org/abs/2104.13586) if you use it in your project.

- Kinetics400 \[2D Skeleton\]: https://download.openmmlab.com/mmaction/v1.0/skeleton/data/k400_2d.pkl (Table of contents only, no skeleton annotations)

For Kinetics400, since the skeleton annotations are large, we do not provide the direct download links on aliyun. Please use the following link to download the `kpfiles` and extract it under `$MMACTION2/data/k400` for Kinetics400 training & testing: https://mycuhk-my.sharepoint.com/:u:/g/personal/1155136485_link_cuhk_edu_hk/EeyDCVskqLtClMVVwqD53acBF2FEwkctp3vtRbkLfnKSTw?e=B3SZlM

If you want to generate 2D skeleton annotations of specified video, please install mmdetection and mmpose first, then use the following script to extract skeleton annotations of NTURGB+D video:

```python
```

please note that, due to the upgrade of mmpose, the inference results may have slight differences from the provided skeleton annotations.

## The Format of Annotations

Each pickle file corresponds to an action recognition dataset. The content of a pickle file is a dictionary with two fields: `split` and `annotations`

2. Annotations: The value of the `annotations` field is a list of skeleton annotations, each skeleton annotation is a dictionary, containing the following fields:

   1. `frame_dir` (str): The identifier of the corresponding video.
   2. `total_frames` (int): The number of frames in this video.
   3. `img_shape` (tuple\[int\]): The shape of a video frame, a tuple with two elements, in the format of (height, width). Only required for 2D skeletons.
   4. `original_shape` (tuple\[int\]): Same as `img_shape`.
   5. `label` (int): The action label.
   6. `keypoint` (np.ndarray, with shape \[M x T x V x C\]): The keypoint annotation. M: number of persons; T: number of frames (same as `total_frames`); V: number of keypoints (25 for NTURGB+D 3D skeleton, 17 for CoCo, 18 for OpenPose, etc. ); C: number of dimensions for keypoint coordinates (C=2 for 2D keypoint, C=3 for 3D keypoint).

7. `keypoint_score` (np.ndarray, with shape \[M x T x V\]): The confidence score of keypoints. Only required for 2D skeletons.

For skeleton data visualization, you need also to prepare the RGB videos. Please refer to \[visualize_heatmap_volume\] for detailed process. Here we provide some visualization examples from NTU-60 and FineGYM.

<table>
<thead>
  <tr>
    <td>
  <br/>
  <br/>
  <br/>
    <td>
  <br/>
  <br/>
  <br/>

<br/>
  <br/>
  <br/>
    <td>
  <br/>
  <br/>
  <br/>
  </tr>
</thead>
</table>

## Convert the NTU RGB+D raw skeleton data to our format (only applicable to GCN backbones)

Here we also provide the script for converting the NTU RGB+D raw skeleton data to our format.
First, download the raw skeleton data of NTU-RGBD 60 and NTU-RGBD 120 from https://github.com/shahroudy/NTURGB-D.

For NTU-RGBD 60, preprocess data and convert the data format with

```python
```

For NTU-RGBD 120, preprocess data and convert the data format with

```python
```

## Convert annotations from third-party projects

We provide scripts to convert skeleton annotations from third-party projects to MMAction2 formats:

**TODO**:

# SimMIM

> [SimMIM: A Simple Framework for Masked Image Modeling](https://arxiv.org/abs/2111.09886)

This paper presents SimMIM, a simple framework for masked image modeling. We simplify recently proposed related approaches without special designs such as blockwise masking and tokenization via discrete VAE or clustering. To study what let the masked image modeling task learn good representations, we systematically study the major components in our framework, and find that simple designs of each component have revealed very strong representation learning performance: 1) random masking of the input image with a moderately large masked patch size (e.g., 32) makes a strong pre-text task; 2) predicting raw pixels of RGB values by direct regression performs no worse than the patch classification approaches with complex designs; 3) the prediction head can be as light as a linear layer, with no worse performance than heavier ones. Using ViT-B, our approach achieves 83.8% top-1 fine-tuning accuracy on ImageNet-1K by pre-training also on this dataset, surpassing previous best approach by +0.6%. When applied on a larger model of about 650 million parameters, SwinV2H, it achieves 87.1% top-1 accuracy on ImageNet-1K using only ImageNet-1K data. We also leverage this approach to facilitate the training of a 3B model (SwinV2-G), that by 40× less data than that in previous practice, we achieve the state-of-the-art on four representative vision benchmarks. The code and models will be publicly available at https: //github.com/microsoft/SimMIM .

<img src="https://user-images.githubusercontent.com/30762564/159404597-ac6d3a44-ee59-4cdc-8f6f-506a7d1b18b6.png" width="70%"/>
</div>

```python

predict = inference_model('swin-base-w6_simmim-100e-pre_8xb256-coslr-100e_in1k-192px', 'demo/bird.JPEG')
```

```python

model = get_model('simmim_swin-base-w6_8xb256-amp-coslr-100e_in1k-192px', pretrained=True)
```

Train:

```shell

```

Train:

```shell
python tools/train.py configs/simmim/simmim_swin-base-w6_8xb256-amp-coslr-100e_in1k-192px.py
```

Test:

```shell
python tools/test.py configs/simmim/benchmarks/swin-base-w6_8xb256-coslr-100e_in1k-192px.py https://download.openmmlab.com/mmselfsup/1.x/simmim/simmim_swin-base_8xb256-amp-coslr-100e_in1k-192/swin-base_ft-8xb256-coslr-100e_in1k/swin-base_ft-8xb256-coslr-100e_in1k_20220829-9cf23aa1.pth
```

| Model                                                     | Params (M) | Flops (G) |                            Config                             |                            Download                             |

| :-------------------------------------------------------- | :--------: | :-------: | :-----------------------------------------------------------: | :-------------------------------------------------------------: |
| `simmim_swin-base-w6_8xb256-amp-coslr-100e_in1k-192px`    |   89.87    |   18.83   | [config](simmim_swin-base-w6_8xb256-amp-coslr-100e_in1k-192px.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/simmim/simmim_swin-base_8xb256-amp-coslr-100e_in1k-192/simmim_swin-base_8xb256-amp-coslr-100e_in1k-192_20220829-0e15782d.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/simmim/simmim_swin-base_8xb256-amp-coslr-100e_in1k-192/simmim_swin-base_8xb256-amp-coslr-100e_in1k-192_20220829-0e15782d.json) |

| `simmim_swin-base-w6_16xb128-amp-coslr-800e_in1k-192px`   |   89.87    |   18.83   | [config](simmim_swin-base-w6_16xb128-amp-coslr-800e_in1k-192px.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/simmim/simmim_swin-base_16xb128-amp-coslr-800e_in1k-192/simmim_swin-base_16xb128-amp-coslr-800e_in1k-192_20220916-a0e931ac.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/simmim/simmim_swin-base_16xb128-amp-coslr-800e_in1k-192/simmim_swin-base_16xb128-amp-coslr-800e_in1k-192_20220916-a0e931ac.json) |

| `simmim_swin-large-w12_16xb128-amp-coslr-800e_in1k-192px` |   199.92   |   55.85   | [config](simmim_swin-large-w12_16xb128-amp-coslr-800e_in1k-192px.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/simmim/simmim_swin-large_16xb128-amp-coslr-800e_in1k-192/simmim_swin-large_16xb128-amp-coslr-800e_in1k-192_20220916-4ad216d3.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/simmim/simmim_swin-large_16xb128-amp-coslr-800e_in1k-192/simmim_swin-large_16xb128-amp-coslr-800e_in1k-192_20220916-4ad216d3.json) |

| `swin-base-w6_simmim-100e-pre_8xb256-coslr-100e_in1k-192px` | [SIMMIM 100-Epochs](https://download.openmmlab.com/mmselfsup/1.x/simmim/simmim_swin-base_8xb256-amp-coslr-100e_in1k-192/simmim_swin-base_8xb256-amp-coslr-100e_in1k-192_20220829-0e15782d.pth) |   87.75    |   11.30   |   82.70   | [config](benchmarks/swin-base-w6_8xb256-coslr-100e_in1k-192px.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/simmim/simmim_swin-base_8xb256-amp-coslr-100e_in1k-192/swin-base_ft-8xb256-coslr-100e_in1k/swin-base_ft-8xb256-coslr-100e_in1k_20220829-9cf23aa1.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/simmim/simmim_swin-base_8xb256-amp-coslr-100e_in1k-192/swin-base_ft-8xb256-coslr-100e_in1k/swin-base_ft-8xb256-coslr-100e_in1k_20220829-9cf23aa1.json) |

| `swin-base-w7_simmim-100e-pre_8xb256-coslr-100e_in1k` | [SIMMIM 100-Epochs](https://download.openmmlab.com/mmselfsup/1.x/simmim/simmim_swin-base_8xb256-amp-coslr-100e_in1k-192/simmim_swin-base_8xb256-amp-coslr-100e_in1k-192_20220829-0e15782d.pth) |   87.77    |   15.47   |   83.50   | [config](benchmarks/swin-base-w7_8xb256-coslr-100e_in1k.py) |                      N/A                      |

| `swin-base-w6_simmim-800e-pre_8xb256-coslr-100e_in1k-192px` | [SIMMIM 800-Epochs](https://download.openmmlab.com/mmselfsup/1.x/simmim/simmim_swin-base_16xb128-amp-coslr-800e_in1k-192/simmim_swin-base_16xb128-amp-coslr-800e_in1k-192_20220916-a0e931ac.pth) |   87.77    |   15.47   |   83.80   | [config](benchmarks/swin-base-w7_8xb256-coslr-100e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/simmim/simmim_swin-base_16xb128-amp-coslr-800e_in1k-192/swin-base_ft-8xb256-coslr-100e_in1k-224/swin-base_ft-8xb256-coslr-100e_in1k-224_20221208-155cc6e6.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/simmim/simmim_swin-base_16xb128-amp-coslr-800e_in1k-192/swin-base_ft-8xb256-coslr-100e_in1k-224/swin-base_ft-8xb256-coslr-100e_in1k-224_20221208-155cc6e6.json) |

| `swin-large-w14_simmim-800e-pre_8xb256-coslr-100e_in1k` | [SIMMIM 800-Epochs](https://download.openmmlab.com/mmselfsup/1.x/simmim/simmim_swin-large_16xb128-amp-coslr-800e_in1k-192/simmim_swin-large_16xb128-amp-coslr-800e_in1k-192_20220916-4ad216d3.pth) |   196.85   |   38.85   |   84.80   | [config](benchmarks/swin-large-w14_8xb256-coslr-100e_in1k.py) | [model](https://download.openmmlab.com/mmselfsup/1.x/simmim/simmim_swin-large_16xb128-amp-coslr-800e_in1k-192/swin-large_ft-8xb256-coslr-ws14-100e_in1k-224/swin-large_ft-8xb256-coslr-ws14-100e_in1k-224_20220916-d4865790.pth) \| [log](https://download.openmmlab.com/mmselfsup/1.x/simmim/simmim_swin-large_16xb128-amp-coslr-800e_in1k-192/swin-large_ft-8xb256-coslr-ws14-100e_in1k-224/swin-large_ft-8xb256-coslr-ws14-100e_in1k-224_20220916-d4865790.json) |

```bibtex
@inproceedings{xie2021simmim,
  title={SimMIM: A Simple Framework for Masked Image Modeling},

```bibtex
  author={Xie, Zhenda and Zhang, Zheng and Cao, Yue and Lin, Yutong and Bao, Jianmin and Yao, Zhuliang and Dai, Qi and Hu, Han},
  booktitle={International Conference on Computer Vision and Pattern Recognition (CVPR)},
}
```

MMAction2 works on Linux, Windows and macOS. It requires Python 3.7+, CUDA 10.2+ and PyTorch 1.8+.

```{note}
```

```shell
```

```shell
```

```shell
```

```

```shell
```

We recommend that users follow our best practices to install MMAction2. However, the whole process is highly customizable. See [Customize Installation](#customize-installation) section for more information.

**Step 1.** Install [MMEngine](https://github.com/open-mmlab/mmengine), [MMCV](https://github.com/open-mmlab/mmcv), [MMDetection](https://github.com/open-mmlab/mmdetection) (optional) and [MMPose](https://github.com/open-mmlab/mmpose) (optional) using [MIM](https://github.com/open-mmlab/mim).

```shell
```

**Step 2.** Install MMAction2.

```shell
```

- [Install from source (Recommended)](#build-mmaction2-from-source): You want to develop your own action recognition task or new features on MMAction2 framework. For example, adding new dataset or new models. Thus, you can use all tools we provided.
- [Install as a Python package](#install-as-a-python-package): You just want to call MMAction2's APIs or import MMAction2's modules in your project.

### Build MMAction2 from source

In this case, install mmaction2 from source:

```shell
```

```

Optionally, if you want to contribute to MMAction2 or experience experimental functions, please checkout to the `dev-1.x` branch:

```shell
```

Just install with pip.

```shell
```

To verify whether MMAction2 is installed correctly, we provide some sample codes to run an inference demo.

**Step 1.** Download the config and checkpoint files.

```shell
```

Option (a). If you install mmaction2 from source, you can run the following command:

```shell

```shell
```

You will see the top-5 labels with corresponding scores in your terminal.

Option (b). If you install mmaction2 as a python package, you can run the following codes in your python interpreter, which will do the similar verification:

```python

```

When installing PyTorch, you may need to specify the version of CUDA. If you are

```

```{note}

```

MMCV contains C++ and CUDA extensions, so it depends on PyTorch in a complex
way. MIM solves such dependencies automatically and makes the installation
easier. However, it is not a must.

To install MMCV with pip instead of MIM, please follow
[MMCV installation guides](https://mmcv.readthedocs.io/en/latest/get_started/installation.html).

```shell
```

```shell
```

MMAction2 can be built for CPU-only environment. In CPU mode you can train, test or inference a model.

Some functionalities are gone in this mode, usually GPU-compiled ops. But don't
worry, almost all models in MMAction2 don't depend on these ops.

### Using MMAction2 with Docker

We provide a [Dockerfile](https://github.com/open-mmlab/mmaction2/blob/main/docker/Dockerfile)

```shell
# build an image with PyTorch 1.8.1, CUDA 10.2, CUDNN 7.
```

Run it with

```shell
```

```

Run it with

```shell
```

1. When migrating from the old version `0.x` to the new version `1.x`, you may encounter issues with mismatched versions of dependent libraries. Below is a display of the versions of each dependent library after following the aforementioned installation process, as shown by `pip list` command. Please ensure that the versions of each dependent library displayed in your terminal are greater than or equal to (i.e., `>=`) the versions shown below for each dependent library.

```shell
mmcv                     2.0.0
```

```bibtex
}
```

</div>

```text
mmpose
├── mmpose
├── docs
├── tests
├── tools
`── data
        |   |── all
        |
        `── images
        |   |── train
        |   |── val
        |   |── test
```

<details>

```bibtex

# 迁移文档

我们在 MMPretrain 1.x 版本中引入了一些修改，可能会产生兼容性问题。请按照本教程从 MMClassification 0.x 或是 MMSelfSup 0.x 迁移您的项目。

## 新的依赖

MMPretrain 1.x 版本依赖于一些新的代码包，您应该根据 [安装教程](./get_started.md) 来创建新的环境，尽管你可能已经拥有了一个可以正常运行 MMClassification 0.x 或 MMSelfSup 0.x 的环境。请参考[安装文档](./get_started.md) 对依赖库进行对应的安装。
```

1. [MMEngine](https://github.com/open-mmlab/mmengine)：MMEngine 是 OpenMMLab 2.0 架构的核心库，我们将许多与计算机视觉无关的组件从 MMCV 拆分到了 MMEngine。
2. [MMCV](https://github.com/open-mmlab/mmcv)：OpenMMLab 计算机视觉基础库，这不是一个新的依赖，但你需要将其升级到 `2.0.0rc1` 版本以上。
3. [rich](https://github.com/Textualize/rich)：一个命令行美化库，用以在命令行中呈现更美观的输出。

# 配置文件的通用改变

在这个部分，我们将介绍一些旧版本 (**MMClassification 0.x** 或 **MMSelfSup 0.x**) 和 **MMPretrain 1.x** 之间通用的变化规范。

## 训练策略设置

| MMCls or MMSelfSup 0.x | MMPretrain 1.x  | 备注                                                                                                     |

## 训练策略设置

| ---------------------- | --------------- | -------------------------------------------------------------------------------------------------------- |
| optimizer_config       | /               | `optimizer_config` 已经被**移除**。                                                                      |
| /                      | optim_wrapper   | `optim_wrapper` 提供了参数更新的相关字段。                                                               |
| lr_config              | param_scheduler | `param_scheduler` 是一个列表设置学习率或者是其它参数，这将比之前更加灵活。                               |
| runner                 | train_cfg       | `train_cfg` 中的循环设置（如 `EpochBasedTrainLoop`，`IterBasedTrainLoop`）将控制模型训练过程中的工作流。 |

**`optimizer`** 和 **`optimizer_config`** 字段的变化：

- 现在我们使用 `optim_wrapper` 字段指定与优化过程有关的所有配置。而 `optimizer` 字段是 `optim_wrapper` 的一个
  子字段。

  子字段。
- `paramwise_cfg` 字段不再是 `optimizer` 的子字段，而是 `optim_wrapper` 的子字段。
- `optimizer_config` 字段被移除，其配置项被移入 `optim_wrapper` 字段。
- `grad_clip` 被重命名为 `clip_grad`

<tr>
<td>

```python
    ))

```

</td>
<tr>
<td>

```python
    ),
)
```

</td>
</tr>
</table>

**`lr_config`** 字段的变化：

- `lr_config` 字段被移除，我们使用新的 `param_scheduler` 配置取代。
- `warmup` 相关的字段都被移除，因为学习率预热可以通过多个学习率规划器的组合来实现，因此不再单独实现。

clip_grad=dict(max_norm=1.0),
)
```

</td>
</tr>
</table>

新的优化器参数规划器组合机制非常灵活，你可以使用它来设计多种学习率、动量曲线，详见{external+mmengine:doc}`MMEngine 中的教程 <tutorials/param_scheduler>`。

<tr>
<td>

```python
    min_lr=0,
```

</td>
<tr>
<td>

```python
    # 学习率预热
    dict(
        end=5,
        # 每轮迭代都更新学习率，而不是每个 epoch
    # 主学习率规划器
]
```

</td>
</tr>
</table>

**`runner`** 字段的变化：

原 `runner` 字段被拆分为 `train_cfg`，`val_cfg` 和 `test_cfg` 三个字段，分别配置训练、验证和测试循环。

<tr>

]
```

</td>
</tr>
</table>

<tr>
<td>

```python
```

</td>
<tr>
<td>

```python
# `val_interval` 字段来自原配置中 `evaluation.interval` 字段
val_cfg = dict()   # 空字典表示使用默认验证配置
test_cfg = dict()  # 空字典表示使用默认测试配置
```

</td>
</tr>
</table>

在 OpenMMLab 2.0 中，我们引入了“循环控制器”来控制训练、验证和测试行为，而原先 `Runner` 功能也相应地发生了变化。详细介绍参见 MMEngine 中的{external+mmengine:doc}`执行器教程 <design/runner>`。

## 运行设置

**`checkpoint_config`** 和 **`log_config`** 字段的变化：

`checkpoint_config` 被移动至 `default_hooks.checkpoint`，`log_config` 被移动至 `default_hooks.logger`。同时，
我们将很多原先在训练脚本中隐式定义的钩子移动到了 `default_hooks` 字段。

```python
    # 记录每轮迭代的耗时

    # 每 100 轮迭代打印一次日志

```python

    # 启用优化器参数规划器

    # 每个 epoch 保存一次模型权重文件，并且自动保存最优权重文件

    # 在分布式环境中设置采样器种子

    # 可视化验证结果，将 `enable` 设为 True 来启用这一功能。
)
```

此外，我们将原来的日志功能拆分为日志记录和可视化器。日志记录负责按照指定间隔保存日志数据，以及进行数据平滑等处理，可视化器用于在不同的后端记录日志，如终端、TensorBoard 和 WandB。

<tr>
<td>

```python
    hooks=[
    ])
```

</td>
<tr>
<td>

```python
    ...
)

    ])
```

</td>
<tr>
<td>

```python
    ...
)

)
```

</td>
</tr>
</table>

**`load_from`** 和 **`resume_from`** 字段的变动：

- `resume_from` 字段被移除。我们现在使用 `resume` 和 `load_from` 字段实现以下功能：
  - 如 `resume=True` 且 `load_from` 不为 None，从 `load_from` 指定的权重文件恢复训练。
  - 如 `resume=True` 且 `load_from` 为 None，尝试从工作目录中最新的权重文件恢复训练。
  - 如 `resume=False` 且 `load_from` 不为 None，仅加载指定的权重文件，不恢复训练。
  - 如 `resume=False` 且 `load_from` 为 None，不进行任何操作。

**`dist_params`** 字段的变动：`dist_params` 字段被移动为 `env_cfg` 字段的一个子字段。以下为 `env_cfg` 字段的所
有配置项：

```python

    # 设置多进程相关参数

    # 设置分布式相关参数
)

)
```

**`workflow`** 字段的变动：`workflow` 相关的功能现已被移除。

新字段 **`visualizer`**：可视化器是 OpenMMLab 2.0 架构中的新设计，我们使用可视化器进行日志、结果的可视化与多后
端的存储。详见 MMEngine 中的{external+mmengine:doc}`可视化教程 <advanced_tutorials/visualization>`。

```python
        # 将下行取消注释，即可将日志和可视化结果保存至 TesnorBoard
    ]
)
```

新字段 **`default_scope`**：指定所有注册器进行模块搜索默认的起点。MMPretrain 中的 `default_scope` 字段为 `mmpretrain`，大部分情况下不需要修改。详见 MMengine 中的{external+mmengine:doc}`注册器教程 <advanced_tutorials/registry>`。

## 其他变动

我们将所有注册器的定义从各个包移动到了 `mmpretrain.registry`。

# 从 MMClassification 0.x 迁移

## 配置文件

在 MMPretrain 1.x 中，我们重构了配置文件的结构，绝大部分原来的配置文件无法直接使用。

## 其他变动

## 配置文件

在本节中，我们将介绍配置文件的所有变化。我们假设您已经对[配置文件](./user_guides/config.md)有所了解。

### 模型设置

`model.backbone`、`model.neck` 和 `model.head` 字段没有变化。

**`model.train_cfg`** 字段的变化：

- `BatchMixup` 被重命名为 [`Mixup`](mmpretrain.models.utils.batch_augments.Mixup)
- `BatchCutMix` 被重命名为 [`CutMix`](mmpretrain.models.utils.batch_augments.CutMix)
- `BatchResizeMix` 被重命名为 [`ResizeMix`](mmpretrain.models.utils.batch_augments.ResizeMix)
- 以上增强中的 `prob` 参数均被移除，现在在 `train_cfg` 中使用一个统一的 `probs` 字段指定每个增强的概率。如果没
  有指定 `probs` 字段，现在将均匀地随机选择一种增强。

<tr>
<td>

```python
    ...
    ]
)
```

</td>
<tr>
<td>

```python
    ...

dict(type='BatchCutMix', alpha=1.0, num_classes=1000, prob=0.5)
    ]
)
```

</td>
<tr>
<td>

```python
    ...
        dict(type='Mixup', alpha=0.8),
        dict(type='CutMix', alpha=1.0),
    ]
)
```

</td>
</tr>
</table>

### 数据设置

**`data`** 字段的变化：

- 原先的 `data` 字段被拆分为 `train_dataloader`，`val_dataloader` 和 `test_dataloader` 字段。这允许我们进行更
  加细粒度的配置。比如在训练和测试中指定不同的采样器、批次大小等。
- `samples_per_gpu` 字段被重命名为 `batch_size`
- `workers_per_gpu` 字段被重命名为 `num_workers`

<tr>
<td>

```python
)
```

</td>
<tr>
<td>

```python
    sampler=dict(type='DefaultSampler', shuffle=True)  # 必要的
)

)

    sampler=dict(type='DefaultSampler', shuffle=False)  # 必要的
)

```

</td>
</tr>
</table>

**`pipeline`** 字段的变化：

- 原先的 **`ToTensor`**、**`ImageToTensor`** 和 **`Collect`** 被合并为 [`PackInputs`](mmpretrain.datasets.transforms.PackInputs)
- 我们建议去除数据集流水线中的 **`Normalize`** 变换，转而使用 `data_preprocessor` 字段进行归一化预处理。
- [**`RandomFlip`**](mmcv.transforms.RandomFlip) 中的 `flip_prob` 参数被重命名为 `prob`
- [**`RandomCrop`**](mmpretrain.datasets.transforms.RandomCrop) 中的 `size` 参数被重命名为 `crop_size`
- [**`RandomResizedCrop`**](mmpretrain.datasets.transforms.RandomResizedCrop) 中的 `size` 参数被重命名为 `scale`
- [**`Resize`**](mmcv.transforms.Resize) 中的 `size` 参数被重命名为 `scale`。并且不再支持形如 `(256, -1)` 的尺寸，请使用 [`ResizeEdge`](mmpretrain.datasets.transforms.ResizeEdge)

- [**`AutoAugment`**](mmpretrain.datasets.transforms.AutoAugment) 和 [**`RandAugment`**](mmpretrain.datasets.transforms.RandAugment) 中的 `policies` 参数现在支持使用字符串来指定某些预设的策略集，`AutoAugment` 支持 "imagenet"，`RandAugment` 支持 "timm_increasing"
- **`RandomResizedCrop`** 和 **`CenterCrop`** 不再支持 `efficientnet_style` 参数，请使用 [`EfficientNetRandomCrop`](mmpretrain.datasets.transforms.EfficientNetRandomCrop) 和 [`EfficientNetCenterCrop`](mmpretrain.datasets.transforms.EfficientNetCenterCrop)

```{note}
我们将一些数据变换工作移至数据预处理器进行，如归一化，请参阅[文档](mmpretrain.models.utils.data_preprocessor)了解更多详细信息。
```

<tr>
<td>

```python

]
```

</td>
<tr>
<td>

```python

]
```

</td>
</tr>
</table>

**`evaluation`** 字段的变化：

- 原先的 **`evaluation`** 字段被拆分为 `val_evaluator` 和 `test_evaluator`，并且不再支持 `interval` 和 `save_best`

dict(type='PackInputs'),
]
```

</td>
</tr>
</table>

  参数。`interval` 参数被移动至 `train_cfg.val_interval` 字段，详见[训练策略配置](./user_guides/config.md#训练策略)。而 `save_best` 参数被移动至 `default_hooks.checkpoint.save_best` 字段，详见 [运行设置](./user_guides/config.md#运行设置)。
- 'accuracy' 指标被重命名为 [`Accuracy`](mmpretrain.evaluation.Accuracy)
- 'precision'，'recall'，'f1-score' 和 'support' 指标被组合为 [`SingleLabelMetric`](mmpretrain.evaluation.SingleLabelMetric)，并使用 `items` 参数指定具体计算哪些指标。
- 'mAP' 指标被重命名为 [`AveragePrecision`](mmpretrain.evaluation.AveragePrecision)
- 'CP'，'CR'，'CF1'，'OP'，'OR' 和 'OF1' 指标被组合为 [`MultiLabelMetric`](mmpretrain.evaluation.MultiLabelMetric)，并使用 `items` 和 `average` 参数指定具体计算哪些指标。

<tr>
<td>

```python
)
```

</td>
<tr>
<td>

<tr>
<td>

```python
)
```

</td>
<tr>
<td>

```python
```

</td>
</tr>
</table>
<tr>
<td>

```python
)
```

</td>
<tr>
<td>

```python
        thr=0.5),
]
```

</td>
</tr>
</table>

## 模块变动

详见[包文档](mmpretrain.apis)

|         函数         | 变动                                                                                                                              |

|         函数         | 变动                                                                                                                              |
| :------------------: | :-------------------------------------------------------------------------------------------------------------------------------- |
|     `init_model`     | 无变动                                                                                                                            |
|  `inference_model`   | 无变动，但我们推荐使用功能更强的 [`mmpretrain.ImageClassificationInferencer`](mmpretrain.apis.ImageClassificationInferencer)。    |
|    `train_model`     | 移除，直接使用 `runner.train` 进行训练。                                                                                          |
|   `multi_gpu_test`   | 移除，直接使用 `runner.test` 进行测试。                                                                                           |

|  `single_gpu_test`   | 移除，直接使用 `runner.test` 进行测试。                                                                                           |
| `show_result_pyplot` | 移除，使用 [`mmpretrain.ImageClassificationInferencer`](mmpretrain.apis.ImageClassificationInferencer) 进行模型推理和结果可视化。 |
|  `set_random_seed`   | 移除，使用 `mmengine.runner.set_random_seed`.                                                                                     |
|  `init_random_seed`  | 移除，使用 `mmengine.dist.sync_random_seed`.                                                                                      |

`mmpretrain.core` 包被重命名为 [`mmpretrain.engine`](mmpretrain.engine)

|      子包       | 变动                                                                                                                              |

|      子包       | 变动                                                                                                                              |
|  `evaluation`   | 移除，使用 [`mmpretrain.evaluation`](mmpretrain.evaluation)                                                                       |
|     `hook`      | 移动至 [`mmpretrain.engine.hooks`](mmpretrain.engine.hooks)                                                                       |
|  `optimizers`   | 移动至 [`mmpretrain.engine.optimizers`](mmpretrain.engine.optimizers)                                                             |
|     `utils`     | 移除，分布式环境相关的函数统一至 [`mmengine.dist`](api/dist) 包                                                                   |

| `visualization` | 移除，其中可视化相关的功能被移动至 [`mmpretrain.visualization.UniversalVisualizer`](mmpretrain.visualization.UniversalVisualizer) |

`hooks` 包中的 `MMClsWandbHook` 尚未实现。

`hooks` 包中的 `CosineAnnealingCooldownLrUpdaterHook` 被移除。我们现在支持使用学习率规划器的组合实现该功能。详见[自定义训练优化策略](./advanced_guides/schedule.md)。

详见[包文档](mmpretrain.datasets)

|                                         数据集类                                          | 变动                                                                     |
| :---------------------------------------------------------------------------------------: | :----------------------------------------------------------------------- |
|                   [`CustomDataset`](mmpretrain.datasets.CustomDataset)                    | 增加了 `data_root` 参数，作为 `data_prefix` 和 `ann_file` 的共同根路径。 |

|                        [`ImageNet`](mmpretrain.datasets.ImageNet)                         | 与 `CustomDataset` 相同。                                                |
|                     [`ImageNet21k`](mmpretrain.datasets.ImageNet21k)                      | 与 `CustomDataset` 相同。                                                |
|   [`CIFAR10`](mmpretrain.datasets.CIFAR10) & [`CIFAR100`](mmpretrain.datasets.CIFAR100)   | `test_mode` 参数目前是必要参数。                                         |
| [`MNIST`](mmpretrain.datasets.MNIST) & [`FashionMNIST`](mmpretrain.datasets.FashionMNIST) | `test_mode` 参数目前是必要参数。                                         |
|                             [`VOC`](mmpretrain.datasets.VOC)                              | 现在需要指定 `data_root`，`image_set_path` 和 `test_mode` 参数。         |

|                             [`CUB`](mmpretrain.datasets.CUB)                              | 现在需要指定 `data_root` 和 `test_mode` 参数。                           |

`mmpretrain.datasets.pipelines` 包被重命名为 `mmpretrain.datasets.transforms`

|           数据变换类            | 变动                                                                                                                                                                      |

|       `LoadImageFromFile`       | 移除，使用 [`mmcv.transforms.LoadImageFromFile`](mmcv.transforms.LoadImageFromFile)                                                                                       |
|          `RandomFlip`           | 移除，使用 [`mmcv.transforms.RandomFlip`](mmcv.transforms.RandomFlip)，其中 `flip_prob` 参数被重命名为 `prob`                                                             |
|          `RandomCrop`           | `size` 参数被重命名为 `crop_size`                                                                                                                                         |
|       `RandomResizedCrop`       | `size` 参数被重命名为 `scale`；`scale` 参数被重命名为 `crop_ratio_range`；不再支持 `efficientnet_style`，请使用 [`EfficientNetRandomCrop`](mmpretrain.datasets.transforms.EfficientNetRandomCrop) |

|          `CenterCrop`           | 移除，使用 [`mmcv.transforms.CenterCrop`](mmcv.transforms.CenterCrop)；不再支持 `efficientnet_style`，请使用 [`EfficientNetCenterCrop`](mmpretrain.datasets.transforms.EfficientNetCenterCrop) |
|            `Resize`             | 移除，使用 [`mmcv.transforms.Resize`](mmcv.transforms.Resize)；`size` 参数被重命名为 `scale`，且不再支持形如 `(256, -1)` 参数，使用 [`ResizeEdge`](mmpretrain.datasets.transforms.ResizeEdge) |
| `AutoAugment` & `RandomAugment` | `policies` 参数现在支持使用字符串指定预设的策略集。                                                                                                                       |
|            `Compose`            | 移除，使用 [`mmcv.transforms.Compose`](mmcv.transforms.Compose)                                                                                                           |

详见[包文档](mmpretrain.models)，**backbones**、**necks** 和 **losses** 的结构没有变动。

[`ImageClassifier`](mmpretrain.models.classifiers.ImageClassifier) 的变动：

|  分类器的方法   | 变动                                                                                                                    |
| :-------------: | :---------------------------------------------------------------------------------------------------------------------- |
| `extract_feat`  | 无变动                                                                                                                  |
|    `forward`    | 现在需要三个输入：`inputs`、`data_samples` 和 `mode`。详见[文档](mmpretrain.models.classifiers.ImageClassifier.forward) |
| `forward_train` | 变更为 `loss` 方法。                                                                                                    |
|  `simple_test`  | 变更为 `predict` 方法。                                                                                                 |

|  `train_step`   | `optimizer` 参数被修改为 `optim_wrapper`，接受 [`OptimWrapper`](mmengine.optim.OptimWrapper)                            |
|   `val_step`    | 原先的 `val_step` 与 `train_step` 一致，现在该方法将会调用 `predict`                                                    |
|   `test_step`   | 新方法，与 `val_step` 一致。                                                                                            |

[heads](mmpretrain.models.heads) 中的变动：

|  分类头的方法   | 变动                                                                                                                                     |
| :-------------: | :--------------------------------------------------------------------------------------------------------------------------------------- |

|  `pre_logits`   | 无变动                                                                                                                                   |
| `forward_train` | 变更为 `loss` 方法。                                                                                                                     |
|  `simple_test`  | 变更为 `predict` 方法。                                                                                                                  |
|     `loss`      | 现在接受 `data_samples` 参数，而不是 `gt_labels`，`data_samples` 参数应当接受 [ClsDataSample](mmpretrain.structures.DataSample) 的列表。 |
|    `forward`    | 新方法，它将返回分类头的输出，不会进行任何后处理（包括 softmax 或 sigmoid）。                                                            |

详见[包文档](mmpretrain.utils)

|             函数             | 变动                                                                                                          |
| :--------------------------: | :------------------------------------------------------------------------------------------------------------ |
|        `collect_env`         | 无变动                                                                                                        |
|      `get_root_logger`       | 移除，使用 [`mmengine.logging.MMLogger.get_current_instance`](mmengine.logging.MMLogger.get_current_instance) |
|       `load_json_log`        | 输出格式发生变化。                                                                                            |

|   `setup_multi_processes`    | 移除，使用 [`mmengine.utils.dl_utils.set_multi_processing`](mmengine.utils.dl_utils.set_multi_processing)     |
| `wrap_non_distributed_model` | 移除，现在 runner 会自动包装模型。                                                                            |
|   `wrap_distributed_model`   | 移除，现在 runner 会自动包装模型。                                                                            |
|     `auto_select_device`     | 移除，现在 runner 会自动选择设备。                                                                            |

# 从 MMSelfSup 0.x 迁移

## 配置文件

本章节将介绍 `_base_` 文件夹中的配置文件的变化，主要包含以下三个部分：

- 数据集：`configs/_base_/datasets`
- 模型：`configs/_base_/models`
- 优化器及调度：`configs/_base_/schedules`

### 数据集

在 **MMSelfSup 0.x** 中，我们使用字段 `data` 来整合数据相关信息, 例如 `samples_per_gpu`，`train`，`val` 等。

### 数据集

在 **MMPretrain 1.x** 中，我们分别使用字段  `train_dataloader`, `val_dataloader` 整理训练和验证的数据相关信息，并且 `data` 字段已经被 **移除**。

<tr>
<td>

```python
        ),
    ),
    val=...)
```

</td>

<tr>
<td>

```python

```

</td>

<tr>
<td>

```python
```

</td>
</tr>
</table>

另外，我们 **移除** 了字段 `data_source`，以此来保证我们项目和其它 OpenMMLab 项目数据流的一致性。请查阅 [Config](user_guides/config.md) 获取更详细的信息。

**`pipeline`** 中的变化：

以 MAE 的 `pipeline` 作为例子，新的写法如下：

```python
    dict(
        size=224,
        scale=(0.2, 1.0),
    dict(type='PackSelfSupInputs', meta_keys=['img_path'])
]
```

### 模型

scale=(0.2, 1.0),
]
```

### 模型

在模型的配置文件中，和 MMSeflSup 0.x 版本相比，主要有两点不同。

1. 有一个新的字段 `data_preprocessor`，主要负责对数据进行预处理，例如归一化，通道转换等。例子如下：

```python
    neck=...,
    head=...,
```

2. 在新版本的 `head` 字段中，我们新增加了 `loss`，主要负责损失函数的构建。例子如下：

```python
    neck=...,
```

## 模块变动

下列表格记录了代码模块、文件夹的主要改变。

```

## 模块变动

| apis                     | /                   | 目前 `apis` 文件夹已暂时被**移除**，在未来可能会再添加回来。                                                                                                  |

| core                     | engine              | `core` 文件夹重命名为 `engine`，包含了 `hooks`，`opimizers`。([API link](mmpretrain.engine))                                                                  |
| datasets                 | datasets            | 数据集相关类主要基于不同的数据集实现，例如 ImageNet，Places205。([API link](mmpretrain.datasets))                                                             |
| datasets/data_sources    | /                   | `data_sources` 已经被**移除**，并且现在 `datasets` 的逻辑和 OpenMMLab 其它项目保持一致。                                                                      |
| datasets/pipelines       | datasets/transforms | `pipelines` 文件夹已经重命名为 `transforms`。([API link](mmpretrain.datasets.transforms))                                                                     |

| /                        | evaluation          | `evaluation` 主要负责管理一些评测函数或者是类。([API link](mmpretrain.evaluation))                                                                            |
| models/algorithms        | selfsup             | 算法文件移动至 `selfsup` 文件夹。([API link](mmpretrain.models.selfsup))                                                                                      |
| models/backbones         | selfsup             | 自监督学习算法对应的，重新实现的主干网络移动到算法的 `.py` 文件中。([API link](mmpretrain.models.selfsup))                                                    |
| models/target_generators | selfsup             | 目标生成器的实现移动到算法的 `.py` 文件中。([API link](mmpretrain.models.selfsup))                                                                            |

| /                        | models/losses       | `losses` 文件夹提供了各种不同损失函数的实现。([API link](mmpretrain.models.losses))                                                                           |
| /                        | structures          | `structures` 文件夹提供了数据结构的实现。在 MMPretrain 中，我们实现了一种新的数据结构，`DataSample`，在训练/验证过程中来传输和接受数据信息。([API link](mmpretrain.structures)) |
| /                        | visualization       | `visualization` 文件夹包含了 visualizer，主要负责一些可视化的工作，例如数据增强的可视化。([API link](mmpretrain.visualization))                               |

# Changelog (MMPreTrain)

## v1.0.1(28/07/2023)

- Add init_cfg with type='pretrained' to downstream tasks ([#1717](https://github.com/open-mmlab/mmpretrain/pull/1717)
- Set 'is_init' in some multimodal methods ([#1718](https://github.com/open-mmlab/mmpretrain/pull/1718)
- Adapt test cases on Ascend NPU ([#1728](https://github.com/open-mmlab/mmpretrain/pull/1728)
- Add GPU Acceleration Apple silicon mac ([#1699](https://github.com/open-mmlab/mmpretrain/pull/1699)
- BEiT refactor ([#1705](https://github.com/open-mmlab/mmpretrain/pull/1705)

- Fix dict update in minigpt4. ([#1709](https://github.com/open-mmlab/mmpretrain/pull/1709)
- Fix nested predict for multi-task prediction ([#1716](https://github.com/open-mmlab/mmpretrain/pull/1716)
- Fix the issue #1711 "GaussianBlur doesn't work" ([#1722](https://github.com/open-mmlab/mmpretrain/pull/1722)
- Just to correct a typo of 'target' ([#1655](https://github.com/open-mmlab/mmpretrain/pull/1655)

- Fix freeze without cls_token in vit ([#1693](https://github.com/open-mmlab/mmpretrain/pull/1693)
- Fix RandomCrop bug ([#1706](https://github.com/open-mmlab/mmpretrain/pull/1706)

### Docs Update

- Fix spelling ([#1689](https://github.com/open-mmlab/mmpretrain/pull/1689)

## v1.0.0(04/07/2023)

- Support inference of more **multi-modal** algorithms, such as **LLaVA**, **MiniGPT-4**, **Otter**, etc.
- Support around **10 multi-modal datasets**!
- Add **iTPN**, **SparK** self-supervised learning algorithms.
- Provide examples of [New Config](https://github.com/open-mmlab/mmpretrain/tree/main/mmpretrain/configs/) and [DeepSpeed/FSDP](https://github.com/open-mmlab/mmpretrain/tree/main/configs/mae/benchmarks/).

- Transfer shape-bias tool from mmselfsup ([#1658](https://github.com/open-mmlab/mmpretrain/pull/1685))

- Download dataset by using MIM&OpenDataLab ([#1630](https://github.com/open-mmlab/mmpretrain/pull/1630))
- Support New Configs ([#1639](https://github.com/open-mmlab/mmpretrain/pull/1639), [#1647](https://github.com/open-mmlab/mmpretrain/pull/1647), [#1665](https://github.com/open-mmlab/mmpretrain/pull/1665))
- Support Flickr30k Retrieval dataset ([#1625](https://github.com/open-mmlab/mmpretrain/pull/1625))
- Support SparK ([#1531](https://github.com/open-mmlab/mmpretrain/pull/1531))
- Support LLaVA ([#1652](https://github.com/open-mmlab/mmpretrain/pull/1652))
- Support Otter ([#1651](https://github.com/open-mmlab/mmpretrain/pull/1651))
- Support MiniGPT-4 ([#1642](https://github.com/open-mmlab/mmpretrain/pull/1642))
- Add support for VizWiz dataset ([#1636](https://github.com/open-mmlab/mmpretrain/pull/1636))

- Add support for vsr dataset ([#1634](https://github.com/open-mmlab/mmpretrain/pull/1634))
- Add InternImage Classification project ([#1569](https://github.com/open-mmlab/mmpretrain/pull/1569))
- Support OCR-VQA dataset ([#1621](https://github.com/open-mmlab/mmpretrain/pull/1621))
- Support OK-VQA dataset ([#1615](https://github.com/open-mmlab/mmpretrain/pull/1615))
- Support TextVQA dataset ([#1569](https://github.com/open-mmlab/mmpretrain/pull/1569))
- Support iTPN and HiViT ([#1584](https://github.com/open-mmlab/mmpretrain/pull/1584))
- Add retrieval mAP metric ([#1552](https://github.com/open-mmlab/mmpretrain/pull/1552))
- Support NoCap dataset based on BLIP. ([#1582](https://github.com/open-mmlab/mmpretrain/pull/1582))
- Add GQA dataset ([#1585](https://github.com/open-mmlab/mmpretrain/pull/1585))

- Update fsdp vit-huge and vit-large config ([#1675](https://github.com/open-mmlab/mmpretrain/pull/1675))
- Support deepspeed with flexible runner ([#1673](https://github.com/open-mmlab/mmpretrain/pull/1673))
- Update Otter and LLaVA docs and config. ([#1653](https://github.com/open-mmlab/mmpretrain/pull/1653))
- Add image_only param of ScienceQA ([#1613](https://github.com/open-mmlab/mmpretrain/pull/1613))
- Support to use "split" to specify training set/validation ([#1535](https://github.com/open-mmlab/mmpretrain/pull/1535))

- Refactor \_prepare_pos_embed in ViT ([#1656](https://github.com/open-mmlab/mmpretrain/pull/1656)， [#1679](https://github.com/open-mmlab/mmpretrain/pull/1679))

- Freeze pre norm in vision transformer ([#1672](https://github.com/open-mmlab/mmpretrain/pull/1672))
- Fix bug loading IN1k dataset ([#1641](https://github.com/open-mmlab/mmpretrain/pull/1641))
- Fix sam bug ([#1633](https://github.com/open-mmlab/mmpretrain/pull/1633))
- Fixed circular import error for new transform ([#1609](https://github.com/open-mmlab/mmpretrain/pull/1609))
- Update torchvision transform wrapper ([#1595](https://github.com/open-mmlab/mmpretrain/pull/1595))
- Set default out_type in CAM visualization ([#1586](https://github.com/open-mmlab/mmpretrain/pull/1586))

- Fix spelling ([#1681](https://github.com/open-mmlab/mmpretrain/pull/1681))

- Fix doc typos ([#1671](https://github.com/open-mmlab/mmpretrain/pull/1671), [#1644](https://github.com/open-mmlab/mmpretrain/pull/1644), [#1629](https://github.com/open-mmlab/mmpretrain/pull/1629))
- Add t-SNE visualization doc ([#1555](https://github.com/open-mmlab/mmpretrain/pull/1555))

## v1.0.0rc8(22/05/2023)

- Support multiple multi-modal algorithms and inferencers. You can explore these features by the [gradio demo](https://github.com/open-mmlab/mmpretrain/tree/main/projects/gradio_demo)!
- Register torchvision transforms into MMPretrain, you can now easily integrate torchvision's data augmentations in MMPretrain.

- Support Chinese CLIP. ([#1576](https://github.com/open-mmlab/mmpretrain/pull/1576))
- Add ScienceQA Metrics ([#1577](https://github.com/open-mmlab/mmpretrain/pull/1577))
- Support multiple multi-modal algorithms and inferencers. ([#1561](https://github.com/open-mmlab/mmpretrain/pull/1561))
- add eva02 backbone ([#1450](https://github.com/open-mmlab/mmpretrain/pull/1450))
- Support dinov2 backbone ([#1522](https://github.com/open-mmlab/mmpretrain/pull/1522))
- Support some downstream classification datasets. ([#1467](https://github.com/open-mmlab/mmpretrain/pull/1467))
- Support GLIP ([#1308](https://github.com/open-mmlab/mmpretrain/pull/1308))
- Register torchvision transforms into mmpretrain ([#1265](https://github.com/open-mmlab/mmpretrain/pull/1265))

- Add ViT of SAM ([#1476](https://github.com/open-mmlab/mmpretrain/pull/1476))

- [Refactor] Support to freeze channel reduction and add layer decay function ([#1490](https://github.com/open-mmlab/mmpretrain/pull/1490))
- [Refactor] Support resizing pos_embed while loading ckpt and format output ([#1488](https://github.com/open-mmlab/mmpretrain/pull/1488))

- Fix scienceqa ([#1581](https://github.com/open-mmlab/mmpretrain/pull/1581))
- Fix config of beit ([#1528](https://github.com/open-mmlab/mmpretrain/pull/1528))
- Incorrect stage freeze on RIFormer Model ([#1573](https://github.com/open-mmlab/mmpretrain/pull/1573))
- Fix ddp bugs caused by `out_type`. ([#1570](https://github.com/open-mmlab/mmpretrain/pull/1570))

- Fix multi-task-head loss potential bug ([#1530](https://github.com/open-mmlab/mmpretrain/pull/1530))
- Support bce loss without batch augmentations ([#1525](https://github.com/open-mmlab/mmpretrain/pull/1525))
- Fix clip generator init bug ([#1518](https://github.com/open-mmlab/mmpretrain/pull/1518))
- Fix the bug in binary cross entropy loss ([#1499](https://github.com/open-mmlab/mmpretrain/pull/1499))

- Update PoolFormer citation to CVPR version ([#1505](https://github.com/open-mmlab/mmpretrain/pull/1505))
- Refine Inference Doc ([#1489](https://github.com/open-mmlab/mmpretrain/pull/1489))
- Add doc for usage of confusion matrix ([#1513](https://github.com/open-mmlab/mmpretrain/pull/1513))
- Update MMagic link ([#1517](https://github.com/open-mmlab/mmpretrain/pull/1517))
- Fix example_project README ([#1575](https://github.com/open-mmlab/mmpretrain/pull/1575))

- Add NPU support page ([#1481](https://github.com/open-mmlab/mmpretrain/pull/1481))
- train cfg: Removed old description ([#1473](https://github.com/open-mmlab/mmpretrain/pull/1473))
- Fix typo in MultiLabelDataset docstring ([#1483](https://github.com/open-mmlab/mmpretrain/pull/1483))

## v1.0.0rc7(07/04/2023)

- Support **LeViT**, **XCiT**, **ViG** and **ConvNeXt-V2** backbone.
- Add t-SNE visualization.
- Support confusion matrix calculation and plot.

- Support RIFormer. ([#1453](https://github.com/open-mmlab/mmpretrain/pull/1453))

- Support XCiT Backbone. ([#1305](https://github.com/open-mmlab/mmclassification/pull/1305))
- Support calculate confusion matrix and plot it. ([#1287](https://github.com/open-mmlab/mmclassification/pull/1287))
- Support RetrieverRecall metric & Add ArcFace config ([#1316](https://github.com/open-mmlab/mmclassification/pull/1316))
- Add `ImageClassificationInferencer`. ([#1261](https://github.com/open-mmlab/mmclassification/pull/1261))
- Support InShop Dataset (Image Retrieval). ([#1019](https://github.com/open-mmlab/mmclassification/pull/1019))
- Support LeViT backbone. ([#1238](https://github.com/open-mmlab/mmclassification/pull/1238))
- Support VIG Backbone. ([#1304](https://github.com/open-mmlab/mmclassification/pull/1304))

- Support ConvNeXt-V2 backbone. ([#1294](https://github.com/open-mmlab/mmclassification/pull/1294))

- Use PyTorch official `scaled_dot_product_attention` to accelerate `MultiheadAttention`. ([#1434](https://github.com/open-mmlab/mmpretrain/pull/1434))
- Add ln to vit avg_featmap output ([#1447](https://github.com/open-mmlab/mmpretrain/pull/1447))
- Update analysis tools and documentations. ([#1359](https://github.com/open-mmlab/mmclassification/pull/1359))
- Unify the `--out` and `--dump` in `tools/test.py`. ([#1307](https://github.com/open-mmlab/mmclassification/pull/1307))
- Enable to toggle whether Gem Pooling is trainable or not. ([#1246](https://github.com/open-mmlab/mmclassification/pull/1246))

- Update registries of mmcls. ([#1306](https://github.com/open-mmlab/mmclassification/pull/1306))
- Add metafile fill and validation tools. ([#1297](https://github.com/open-mmlab/mmclassification/pull/1297))
- Remove useless EfficientnetV2 config files. ([#1300](https://github.com/open-mmlab/mmclassification/pull/1300))

- Fix precise bn hook ([#1466](https://github.com/open-mmlab/mmpretrain/pull/1466))
- Fix retrieval multi gpu bug ([#1319](https://github.com/open-mmlab/mmclassification/pull/1319))
- Fix error repvgg-deploy base config path. ([#1357](https://github.com/open-mmlab/mmclassification/pull/1357))
- Fix bug in test tools. ([#1309](https://github.com/open-mmlab/mmclassification/pull/1309))

- Translate some tools tutorials to Chinese. ([#1321](https://github.com/open-mmlab/mmclassification/pull/1321))

- Add Chinese translation for runtime.md.  ([#1313](https://github.com/open-mmlab/mmclassification/pull/1313))

# Changelog (MMClassification)

## v1.0.0rc5(30/12/2022)

- Support EVA, RevViT, EfficientnetV2, CLIP, TinyViT and MixMIM backbones.
- Reproduce the training accuracy of ConvNeXt and RepVGG.
- Support multi-task training and testing.
- Support Test-time Augmentation.

- [Feature] Add EfficientnetV2 Backbone. ([#1253](https://github.com/open-mmlab/mmclassification/pull/1253))
- [Feature] Support TTA and add `--tta` in `tools/test.py`. ([#1161](https://github.com/open-mmlab/mmclassification/pull/1161))
- [Feature] Support Multi-task. ([#1229](https://github.com/open-mmlab/mmclassification/pull/1229))
- [Feature] Add clip backbone. ([#1258](https://github.com/open-mmlab/mmclassification/pull/1258))

- [Feature] Add mixmim backbone with checkpoints. ([#1224](https://github.com/open-mmlab/mmclassification/pull/1224))
- [Feature] Add TinyViT for dev-1.x. ([#1042](https://github.com/open-mmlab/mmclassification/pull/1042))
- [Feature] Add some scripts for development. ([#1257](https://github.com/open-mmlab/mmclassification/pull/1257))
- [Feature] Support EVA. ([#1239](https://github.com/open-mmlab/mmclassification/pull/1239))
- [Feature] Implementation of RevViT. ([#1127](https://github.com/open-mmlab/mmclassification/pull/1127))

- [Reproduce] Reproduce RepVGG  Training Accuracy. ([#1264](https://github.com/open-mmlab/mmclassification/pull/1264))
- [Enhance] Support ConvNeXt More Weights. ([#1240](https://github.com/open-mmlab/mmclassification/pull/1240))

- [Reproduce] Update ConvNeXt config files. ([#1256](https://github.com/open-mmlab/mmclassification/pull/1256))
- [CI] Update CI to test PyTorch 1.13.0. ([#1260](https://github.com/open-mmlab/mmclassification/pull/1260))
- [Project] Add ACCV workshop 1st Solution. ([#1245](https://github.com/open-mmlab/mmclassification/pull/1245))
- [Project] Add Example project. ([#1254](https://github.com/open-mmlab/mmclassification/pull/1254))

- [Fix] Fix imports in transforms. ([#1255](https://github.com/open-mmlab/mmclassification/pull/1255))
- [Fix] Fix CAM visualization. ([#1248](https://github.com/open-mmlab/mmclassification/pull/1248))
- [Fix] Fix the requirements and lazy register mmpretrain models. ([#1275](https://github.com/open-mmlab/mmclassification/pull/1275))

## v1.0.0rc4(06/12/2022)

- Upgrade API to get pre-defined models of MMClassification. See [#1236](https://github.com/open-mmlab/mmclassification/pull/1236) for more details.
- Refactor BEiT backbone and support v1/v2 inference. See [#1144](https://github.com/open-mmlab/mmclassification/pull/1144).

- Support getting model from the name defined in the model-index file. ([#1236](https://github.com/open-mmlab/mmclassification/pull/1236))

- Support evaluate on both EMA and non-EMA models. ([#1204](https://github.com/open-mmlab/mmclassification/pull/1204))
- Refactor BEiT backbone and support v1/v2 inference. ([#1144](https://github.com/open-mmlab/mmclassification/pull/1144))

- Fix `reparameterize_model.py` doesn't save meta info. ([#1221](https://github.com/open-mmlab/mmclassification/pull/1221))
- Fix dict update in BEiT. ([#1234](https://github.com/open-mmlab/mmclassification/pull/1234))

- Update install tutorial. ([#1223](https://github.com/open-mmlab/mmclassification/pull/1223))
- Update MobileNetv2 & MobileNetv3 readme. ([#1222](https://github.com/open-mmlab/mmclassification/pull/1222))
- Add version selection in the banner. ([#1217](https://github.com/open-mmlab/mmclassification/pull/1217))

## v1.0.0rc3(21/11/2022)

- Add **Switch Recipe** Hook, Now we can modify training pipeline, mixup and loss settings during training, see [#1101](https://github.com/open-mmlab/mmclassification/pull/1101).

- Add **TIMM and HuggingFace** wrappers. Now you can train/use models in TIMM/HuggingFace directly, see [#1102](https://github.com/open-mmlab/mmclassification/pull/1102).
- Support **retrieval tasks**, see [#1055](https://github.com/open-mmlab/mmclassification/pull/1055).
- Reproduce **mobileone** training accuracy. See [#1191](https://github.com/open-mmlab/mmclassification/pull/1191)

- Add checkpoints from EfficientNets NoisyStudent & L2. ([#1122](https://github.com/open-mmlab/mmclassification/pull/1122))
- Migrate CSRA head to 1.x. ([#1177](https://github.com/open-mmlab/mmclassification/pull/1177))
- Support RepLKnet backbone. ([#1129](https://github.com/open-mmlab/mmclassification/pull/1129))

- Add Switch Recipe Hook. ([#1101](https://github.com/open-mmlab/mmclassification/pull/1101))
- Add adan optimizer. ([#1180](https://github.com/open-mmlab/mmclassification/pull/1180))
- Support DaViT. ([#1105](https://github.com/open-mmlab/mmclassification/pull/1105))
- Support Activation Checkpointing for ConvNeXt. ([#1153](https://github.com/open-mmlab/mmclassification/pull/1153))
- Add TIMM and HuggingFace wrappers to build classifiers from them directly. ([#1102](https://github.com/open-mmlab/mmclassification/pull/1102))
- Add reduction for neck ([#978](https://github.com/open-mmlab/mmclassification/pull/978))
- Support HorNet Backbone for dev1.x. ([#1094](https://github.com/open-mmlab/mmclassification/pull/1094))

- Add arcface head. ([#926](https://github.com/open-mmlab/mmclassification/pull/926))
- Add Base Retriever and Image2Image Retriever for retrieval tasks. ([#1055](https://github.com/open-mmlab/mmclassification/pull/1055))
- Support MobileViT backbone. ([#1068](https://github.com/open-mmlab/mmclassification/pull/1068))

- [Enhance] Enhance ArcFaceClsHead. ([#1181](https://github.com/open-mmlab/mmclassification/pull/1181))
- [Refactor] Refactor to use new fileio API in MMEngine. ([#1176](https://github.com/open-mmlab/mmclassification/pull/1176))
- [Enhance] Reproduce mobileone training accuracy. ([#1191](https://github.com/open-mmlab/mmclassification/pull/1191))
- [Enhance] add deleting params info in swinv2. ([#1142](https://github.com/open-mmlab/mmclassification/pull/1142))

- [Enhance] Add more mobilenetv3 pretrains. ([#1154](https://github.com/open-mmlab/mmclassification/pull/1154))
- [Enhancement] RepVGG for YOLOX-PAI for dev-1.x. ([#1126](https://github.com/open-mmlab/mmclassification/pull/1126))
- [Improve] Speed up data preprocessor. ([#1064](https://github.com/open-mmlab/mmclassification/pull/1064))

- Fix the torchserve. ([#1143](https://github.com/open-mmlab/mmclassification/pull/1143))
- Fix configs due to api refactor of `num_classes`. ([#1184](https://github.com/open-mmlab/mmclassification/pull/1184))
- Update mmpretrain2torchserve. ([#1189](https://github.com/open-mmlab/mmclassification/pull/1189))
- Fix for `inference_model` cannot get classes information in checkpoint. ([#1093](https://github.com/open-mmlab/mmclassification/pull/1093))

- Add not-found page extension. ([#1207](https://github.com/open-mmlab/mmclassification/pull/1207))
- update visualization doc. ([#1160](https://github.com/open-mmlab/mmclassification/pull/1160))
- Support sort and search the Model Summary table. ([#1100](https://github.com/open-mmlab/mmclassification/pull/1100))
- Improve the ResNet model page. ([#1118](https://github.com/open-mmlab/mmclassification/pull/1118))
- update the readme of convnext. ([#1156](https://github.com/open-mmlab/mmclassification/pull/1156))
- Fix the installation docs link in README. ([#1164](https://github.com/open-mmlab/mmclassification/pull/1164))
- Improve ViT and MobileViT model pages. ([#1155](https://github.com/open-mmlab/mmclassification/pull/1155))

- Improve Swin Doc and Add Tabs enxtation. ([#1145](https://github.com/open-mmlab/mmclassification/pull/1145))
- Add MMEval projects link in README. ([#1162](https://github.com/open-mmlab/mmclassification/pull/1162))
- Add runtime configuration docs. ([#1128](https://github.com/open-mmlab/mmclassification/pull/1128))
- Add custom evaluation docs ([#1130](https://github.com/open-mmlab/mmclassification/pull/1130))
- Add custom pipeline docs. ([#1124](https://github.com/open-mmlab/mmclassification/pull/1124))
- Add MMYOLO projects link in MMCLS1.x. ([#1117](https://github.com/open-mmlab/mmclassification/pull/1117))

## v1.0.0rc2(12/10/2022)

- [Feature] Support DeiT3. ([#1065](https://github.com/open-mmlab/mmclassification/pull/1065))

- [Enhance] Update `analyze_results.py` for dev-1.x. ([#1071](https://github.com/open-mmlab/mmclassification/pull/1071))
- [Enhance] Get scores from inference api. ([#1070](https://github.com/open-mmlab/mmclassification/pull/1070))

- [Fix] Update requirements. ([#1083](https://github.com/open-mmlab/mmclassification/pull/1083))

- [Docs] Add 1x docs schedule. ([#1015](https://github.com/open-mmlab/mmclassification/pull/1015))

## v1.0.0rc1(30/9/2022)

- Support MViT for MMCLS 1.x ([#1023](https://github.com/open-mmlab/mmclassification/pull/1023))
- Add ViT huge architecture. ([#1049](https://github.com/open-mmlab/mmclassification/pull/1049))
- Support EdgeNeXt for dev-1.x. ([#1037](https://github.com/open-mmlab/mmclassification/pull/1037))

- Support Swin Transformer V2 for MMCLS 1.x. ([#1029](https://github.com/open-mmlab/mmclassification/pull/1029))
- Add efficientformer Backbone for MMCls 1.x. ([#1031](https://github.com/open-mmlab/mmclassification/pull/1031))
- Add MobileOne Backbone For MMCls 1.x.  ([#1030](https://github.com/open-mmlab/mmclassification/pull/1030))
- Support BEiT Transformer layer. ([#919](https://github.com/open-mmlab/mmclassification/pull/919))

- [Refactor] Fix visualization tools. ([#1045](https://github.com/open-mmlab/mmclassification/pull/1045))
- [Improve] Update benchmark scripts ([#1028](https://github.com/open-mmlab/mmclassification/pull/1028))

- [Improve] Update tools to enable `pin_memory` and `persistent_workers` by default. ([#1024](https://github.com/open-mmlab/mmclassification/pull/1024))
- [CI] Update circle-ci and github workflow. ([#1018](https://github.com/open-mmlab/mmclassification/pull/1018))

- Fix verify dataset tool in 1.x. ([#1062](https://github.com/open-mmlab/mmclassification/pull/1062))
- Fix `loss_weight` in `LabelSmoothLoss`. ([#1058](https://github.com/open-mmlab/mmclassification/pull/1058))
- Fix the output position of Swin-Transformer. ([#947](https://github.com/open-mmlab/mmclassification/pull/947))

- Auto generate model summary table.  ([#1010](https://github.com/open-mmlab/mmclassification/pull/1010))
- Refactor new modules tutorial. ([#998](https://github.com/open-mmlab/mmclassification/pull/998))

MMClassification 1.0.0rc0 is the first version of MMClassification 1.x, a part of the OpenMMLab 2.0 projects.

Built upon the new [training engine](https://github.com/open-mmlab/mmengine), MMClassification 1.x unifies the interfaces of dataset, models, evaluation, and visualization.

And there are some BC-breaking changes. Please check [the migration tutorial](https://mmclassification.readthedocs.io/en/1.x/migration.html) for more details.

## v0.23.1(2/6/2022)

- Dedicated MMClsWandbHook for MMClassification (Weights and Biases Integration) ([#764](https://github.com/open-mmlab/mmclassification/pull/764))

- Use mdformat instead of markdownlint to format markdown. ([#844](https://github.com/open-mmlab/mmclassification/pull/844))

- Fix wrong `--local_rank`.

- Update install tutorials. ([#854](https://github.com/open-mmlab/mmclassification/pull/854))
- Fix wrong link in README. ([#835](https://github.com/open-mmlab/mmclassification/pull/835))

## v0.23.0(1/5/2022)

- Support DenseNet. ([#750](https://github.com/open-mmlab/mmclassification/pull/750))
- Support VAN. ([#739](https://github.com/open-mmlab/mmclassification/pull/739))

- Support training on IPU and add fine-tuning configs of ViT. ([#723](https://github.com/open-mmlab/mmclassification/pull/723))

- New style API reference, and easier to use! Welcome [view it](https://mmclassification.readthedocs.io/en/master/api/models.html). ([#774](https://github.com/open-mmlab/mmclassification/pull/774))

## v0.22.1(15/4/2022)

- [Feature] Support resize relative position embedding in `SwinTransformer`. ([#749](https://github.com/open-mmlab/mmclassification/pull/749))
- [Feature] Add PoolFormer backbone and checkpoints. ([#746](https://github.com/open-mmlab/mmclassification/pull/746))

- [Enhance] Improve CPE performance by reduce memory copy. ([#762](https://github.com/open-mmlab/mmclassification/pull/762))
- [Enhance] Add extra dataloader settings in configs. ([#752](https://github.com/open-mmlab/mmclassification/pull/752))

## v0.22.0(30/3/2022)

- Support a series of CSP Network, such as CSP-ResNet, CSP-ResNeXt and CSP-DarkNet.
- A new `CustomDataset` class to help you build dataset of yourself!
- Support ConvMixer, RepMLP and new dataset - CUB dataset.

- [Feature] Add CSPNet and backbone and checkpoints ([#735](https://github.com/open-mmlab/mmclassification/pull/735))

- [Feature] Add `CustomDataset`. ([#738](https://github.com/open-mmlab/mmclassification/pull/738))
- [Feature] Add diff seeds to diff ranks.  ([#744](https://github.com/open-mmlab/mmclassification/pull/744))
- [Feature] Support ConvMixer. ([#716](https://github.com/open-mmlab/mmclassification/pull/716))
- [Feature] Our `dist_train` & `dist_test` tools support distributed training on multiple machines. ([#734](https://github.com/open-mmlab/mmclassification/pull/734))
- [Feature] Add RepMLP backbone and checkpoints. ([#709](https://github.com/open-mmlab/mmclassification/pull/709))
- [Feature] Support CUB dataset. ([#703](https://github.com/open-mmlab/mmclassification/pull/703))
- [Feature] Support ResizeMix. ([#676](https://github.com/open-mmlab/mmclassification/pull/676))

- [Enhance] Use `--a-b` instead of `--a_b` in arguments. ([#754](https://github.com/open-mmlab/mmclassification/pull/754))
- [Enhance] Add `get_cat_ids` and `get_gt_labels` to KFoldDataset. ([#721](https://github.com/open-mmlab/mmclassification/pull/721))
- [Enhance] Set torch seed in `worker_init_fn`. ([#733](https://github.com/open-mmlab/mmclassification/pull/733))

- [Fix] Fix the discontiguous output feature map of ConvNeXt. ([#743](https://github.com/open-mmlab/mmclassification/pull/743))

- [Docs] Add brief installation steps in README for copy&paste. ([#755](https://github.com/open-mmlab/mmclassification/pull/755))
- [Docs] fix logo url link from mmocr to mmpretrain. ([#732](https://github.com/open-mmlab/mmclassification/pull/732))

## v0.21.0(04/03/2022)

- Support ResNetV1c and Wide-ResNet, and provide pre-trained models.
- Support dynamic input shape for ViT-based algorithms. Now our ViT, DeiT, Swin-Transformer and T2T-ViT support forwarding with any input shape.
- Reproduce training results of DeiT. And our DeiT-T and DeiT-S have higher accuracy comparing with the official weights.

- Add ResNetV1c. ([#692](https://github.com/open-mmlab/mmclassification/pull/692))
- Support Wide-ResNet. ([#715](https://github.com/open-mmlab/mmclassification/pull/715))
- Support gem pooling ([#677](https://github.com/open-mmlab/mmclassification/pull/677))

- Reproduce training results of DeiT. ([#711](https://github.com/open-mmlab/mmclassification/pull/711))

- Add ConvNeXt pretrain models on ImageNet-1k. ([#707](https://github.com/open-mmlab/mmclassification/pull/707))
- Support dynamic input shape for ViT-based algorithms. ([#706](https://github.com/open-mmlab/mmclassification/pull/706))
- Add `evaluate` function for ConcatDataset. ([#650](https://github.com/open-mmlab/mmclassification/pull/650))
- Enhance vis-pipeline tool. ([#604](https://github.com/open-mmlab/mmclassification/pull/604))
- Return code 1 if scripts runs failed. ([#694](https://github.com/open-mmlab/mmclassification/pull/694))
- Use PyTorch official `one_hot` to implement `convert_to_one_hot`. ([#696](https://github.com/open-mmlab/mmclassification/pull/696))
- Add a new pre-commit-hook to automatically add a copyright. ([#710](https://github.com/open-mmlab/mmclassification/pull/710))

- Add deprecation message for deploy tools. ([#697](https://github.com/open-mmlab/mmclassification/pull/697))
- Upgrade isort pre-commit hooks. ([#687](https://github.com/open-mmlab/mmclassification/pull/687))
- Use `--gpu-id` instead of `--gpu-ids` in non-distributed multi-gpu training/testing. ([#688](https://github.com/open-mmlab/mmclassification/pull/688))
- Remove deprecation. ([#633](https://github.com/open-mmlab/mmclassification/pull/633))

- Fix Conformer forward with irregular input size. ([#686](https://github.com/open-mmlab/mmclassification/pull/686))
- Add `dist.barrier` to fix a bug in directory checking. ([#666](https://github.com/open-mmlab/mmclassification/pull/666))

## v0.20.1(07/02/2022)

- Fix the MMCV dependency version.

## v0.20.0(30/01/2022)

- Support K-fold cross-validation. The tutorial will be released later.
- Support HRNet, ConvNeXt, Twins and EfficientNet.
- Support model conversion from PyTorch to Core-ML by a tool.

- Support K-fold cross-validation. ([#563](https://github.com/open-mmlab/mmclassification/pull/563))
- Support HRNet and add pre-trained models. ([#660](https://github.com/open-mmlab/mmclassification/pull/660))
- Support ConvNeXt and add pre-trained models. ([#670](https://github.com/open-mmlab/mmclassification/pull/670))
- Support Twins and add pre-trained models. ([#642](https://github.com/open-mmlab/mmclassification/pull/642))
- Support EfficientNet and add pre-trained models.([#649](https://github.com/open-mmlab/mmclassification/pull/649))
- Support `features_only` option in `TIMMBackbone`. ([#668](https://github.com/open-mmlab/mmclassification/pull/668))

- Add conversion script from pytorch to Core-ML model. ([#597](https://github.com/open-mmlab/mmclassification/pull/597))

- New-style CPU training and inference. ([#674](https://github.com/open-mmlab/mmclassification/pull/674))
- Add setup multi-processing both in train and test. ([#671](https://github.com/open-mmlab/mmclassification/pull/671))
- Rewrite channel split operation in ShufflenetV2. ([#632](https://github.com/open-mmlab/mmclassification/pull/632))
- Deprecate the support for "python setup.py test". ([#646](https://github.com/open-mmlab/mmclassification/pull/646))
- Support single-label, softmax, custom eps by asymmetric loss. ([#609](https://github.com/open-mmlab/mmclassification/pull/609))
- Save class names in best checkpoint created by evaluation hook. ([#641](https://github.com/open-mmlab/mmclassification/pull/641))

- Fix potential unexcepted behaviors if `metric_options` is not specified in multi-label evaluation. ([#647](https://github.com/open-mmlab/mmclassification/pull/647))
- Fix API changes in  `pytorch-grad-cam&gt;=1.3.7`. ([#656](https://github.com/open-mmlab/mmclassification/pull/656))
- Fix bug which breaks `cal_train_time` in `analyze_logs.py`. ([#662](https://github.com/open-mmlab/mmclassification/pull/662))

- Update README in configs according to OpenMMLab standard. ([#672](https://github.com/open-mmlab/mmclassification/pull/672))
- Update installation guide and README. ([#624](https://github.com/open-mmlab/mmclassification/pull/624))

## v0.19.0(31/12/2021)

- The feature extraction function has been enhanced. See [#593](https://github.com/open-mmlab/mmclassification/pull/593) for more details.

- Provide the high-acc ResNet-50 training settings from [*ResNet strikes back*](https://arxiv.org/abs/2110.00476).
- Reproduce the training accuracy of T2T-ViT & RegNetX, and provide self-training checkpoints.
- Support DeiT & Conformer backbone and checkpoints.
- Provide a CAM visualization tool based on [pytorch-grad-cam](https://github.com/jacobgil/pytorch-grad-cam), and detailed [user guide](https://mmclassification.readthedocs.io/en/latest/tools/visualization.html#class-activation-map-visualization)!

- Support Precise BN. ([#401](https://github.com/open-mmlab/mmclassification/pull/401))
- Add CAM visualization tool. ([#577](https://github.com/open-mmlab/mmclassification/pull/577))
- Repeated Aug and Sampler Registry. ([#588](https://github.com/open-mmlab/mmclassification/pull/588))

- Add DeiT backbone and checkpoints. ([#576](https://github.com/open-mmlab/mmclassification/pull/576))
- Support LAMB optimizer. ([#591](https://github.com/open-mmlab/mmclassification/pull/591))
- Implement the conformer backbone. ([#494](https://github.com/open-mmlab/mmclassification/pull/494))
- Add the frozen function for Swin Transformer model. ([#574](https://github.com/open-mmlab/mmclassification/pull/574))
- Support using checkpoint in Swin Transformer to save memory. ([#557](https://github.com/open-mmlab/mmclassification/pull/557))

- [Reproduction] Reproduce RegNetX training accuracy. ([#587](https://github.com/open-mmlab/mmclassification/pull/587))

- [Reproduction] Reproduce training results of T2T-ViT. ([#610](https://github.com/open-mmlab/mmclassification/pull/610))
- [Enhance] Provide high-acc training settings of ResNet. ([#572](https://github.com/open-mmlab/mmclassification/pull/572))
- [Enhance] Set a random seed when the user does not set a seed. ([#554](https://github.com/open-mmlab/mmclassification/pull/554))
- [Enhance] Added `NumClassCheckHook` and unit tests. ([#559](https://github.com/open-mmlab/mmclassification/pull/559))
- [Enhance] Enhance feature extraction function. ([#593](https://github.com/open-mmlab/mmclassification/pull/593))
- [Enhance] Improve efficiency of precision, recall, f1_score and support. ([#595](https://github.com/open-mmlab/mmclassification/pull/595))

- [Enhance] Improve accuracy calculation performance. ([#592](https://github.com/open-mmlab/mmclassification/pull/592))
- [Refactor] Refactor `analysis_log.py`. ([#529](https://github.com/open-mmlab/mmclassification/pull/529))
- [Refactor] Use new API of matplotlib to handle blocking input in visualization. ([#568](https://github.com/open-mmlab/mmclassification/pull/568))
- [CI] Cancel previous runs that are not completed. ([#583](https://github.com/open-mmlab/mmclassification/pull/583))
- [CI] Skip build CI if only configs or docs modification. ([#575](https://github.com/open-mmlab/mmclassification/pull/575))

- Fix test sampler bug. ([#611](https://github.com/open-mmlab/mmclassification/pull/611))
- Try to create a symbolic link, otherwise copy. ([#580](https://github.com/open-mmlab/mmclassification/pull/580))

- Fix a bug for multiple output in swin transformer. ([#571](https://github.com/open-mmlab/mmclassification/pull/571))

- Update mmcv, torch, cuda version in Dockerfile and docs. ([#594](https://github.com/open-mmlab/mmclassification/pull/594))
- Add analysis&misc docs. ([#525](https://github.com/open-mmlab/mmclassification/pull/525))
- Fix docs build dependency. ([#584](https://github.com/open-mmlab/mmclassification/pull/584))

## v0.18.0(30/11/2021)

- Support MLP-Mixer backbone and provide pre-trained checkpoints.
- Add a tool to visualize the learning rate curve of the training phase. Welcome to use with the [tutorial](https://mmclassification.readthedocs.io/en/latest/tools/visualization.html#learning-rate-schedule-visualization)!

- Add MLP Mixer Backbone. ([#528](https://github.com/open-mmlab/mmclassification/pull/528), [#539](https://github.com/open-mmlab/mmclassification/pull/539))
- Support positive weights in BCE. ([#516](https://github.com/open-mmlab/mmclassification/pull/516))
- Add a tool to visualize learning rate in each iterations. ([#498](https://github.com/open-mmlab/mmclassification/pull/498))

- Use CircleCI to do unit tests. ([#567](https://github.com/open-mmlab/mmclassification/pull/567))
- Focal loss for single label tasks. ([#548](https://github.com/open-mmlab/mmclassification/pull/548))
- Remove useless `import_modules_from_string`. ([#544](https://github.com/open-mmlab/mmclassification/pull/544))
- Rename config files according to the config name standard. ([#508](https://github.com/open-mmlab/mmclassification/pull/508))
- Use `reset_classifier` to remove head of timm backbones. ([#534](https://github.com/open-mmlab/mmclassification/pull/534))

- Support passing arguments to loss from head. ([#523](https://github.com/open-mmlab/mmclassification/pull/523))
- Refactor `Resize` transform and add `Pad` transform. ([#506](https://github.com/open-mmlab/mmclassification/pull/506))
- Update mmcv dependency version. ([#509](https://github.com/open-mmlab/mmclassification/pull/509))

- Fix bug when using `ClassBalancedDataset`. ([#555](https://github.com/open-mmlab/mmclassification/pull/555))
- Fix a bug when using iter-based runner with 'val' workflow. ([#542](https://github.com/open-mmlab/mmclassification/pull/542))
- Fix interpolation method checking in `Resize`. ([#547](https://github.com/open-mmlab/mmclassification/pull/547))
- Fix a bug when load checkpoints in mulit-GPUs environment. ([#527](https://github.com/open-mmlab/mmclassification/pull/527))

- Fix an error on indexing scalar metrics in `analyze_result.py`. ([#518](https://github.com/open-mmlab/mmclassification/pull/518))
- Fix wrong condition judgment in `analyze_logs.py` and prevent empty curve. ([#510](https://github.com/open-mmlab/mmclassification/pull/510))

- Fix vit config and model broken links. ([#564](https://github.com/open-mmlab/mmclassification/pull/564))
- Add abstract and image for every paper. ([#546](https://github.com/open-mmlab/mmclassification/pull/546))
- Add mmflow and mim in banner and readme. ([#543](https://github.com/open-mmlab/mmclassification/pull/543))
- Add schedule and runtime tutorial docs. ([#499](https://github.com/open-mmlab/mmclassification/pull/499))
- Add the top-5 acc in ResNet-CIFAR README. ([#531](https://github.com/open-mmlab/mmclassification/pull/531))

- Fix TOC of `visualization.md` and add example images. ([#513](https://github.com/open-mmlab/mmclassification/pull/513))
- Use docs link of other projects and add MMCV docs. ([#511](https://github.com/open-mmlab/mmclassification/pull/511))

## v0.17.0(29/10/2021)

- Support Tokens-to-Token ViT backbone and Res2Net backbone. Welcome to use!
- Support ImageNet21k dataset.
- Add a pipeline visualization tool. Try it with the [tutorials](https://mmclassification.readthedocs.io/en/latest/tools/visualization.html#pipeline-visualization)!

- Add Tokens-to-Token ViT backbone and converted checkpoints. ([#467](https://github.com/open-mmlab/mmclassification/pull/467))
- Add Res2Net backbone and converted weights. ([#465](https://github.com/open-mmlab/mmclassification/pull/465))

- Support ImageNet21k dataset. ([#461](https://github.com/open-mmlab/mmclassification/pull/461))
- Support seesaw loss. ([#500](https://github.com/open-mmlab/mmclassification/pull/500))
- Add a pipeline visualization tool. ([#406](https://github.com/open-mmlab/mmclassification/pull/406))
- Add a tool to find broken files. ([#482](https://github.com/open-mmlab/mmclassification/pull/482))
- Add a tool to test TorchServe. ([#468](https://github.com/open-mmlab/mmclassification/pull/468))

- Refator Vision Transformer. ([#395](https://github.com/open-mmlab/mmclassification/pull/395))
- Use context manager to reuse matplotlib figures. ([#432](https://github.com/open-mmlab/mmclassification/pull/432))

- Remove `DistSamplerSeedHook` if use `IterBasedRunner`. ([#501](https://github.com/open-mmlab/mmclassification/pull/501))

- Set the priority of `EvalHook` to "LOW" to avoid a bug when using `IterBasedRunner`. ([#488](https://github.com/open-mmlab/mmclassification/pull/488))
- Fix a wrong parameter of `get_root_logger` in `apis/train.py`. ([#486](https://github.com/open-mmlab/mmclassification/pull/486))
- Fix version check in dataset builder. ([#474](https://github.com/open-mmlab/mmclassification/pull/474))

- Add English Colab tutorials and update Chinese Colab tutorials. ([#483](https://github.com/open-mmlab/mmclassification/pull/483), [#497](https://github.com/open-mmlab/mmclassification/pull/497))
- Add tutuorial for config files. ([#487](https://github.com/open-mmlab/mmclassification/pull/487))
- Add model-pages in Model Zoo. ([#480](https://github.com/open-mmlab/mmclassification/pull/480))

- Add code-spell pre-commit hook and fix a large mount of typos. ([#470](https://github.com/open-mmlab/mmclassification/pull/470))

## v0.16.0(30/9/2021)

- We have improved compatibility with downstream repositories like MMDetection and MMSegmentation. We will add some examples about how to use our backbones in MMDetection.
- Add RepVGG backbone and checkpoints. Welcome to use it!
- Add timm backbones wrapper, now you can simply use backbones of pytorch-image-models in MMClassification!

- Add RepVGG backbone and checkpoints. ([#414](https://github.com/open-mmlab/mmclassification/pull/414))
- Add timm backbones wrapper. ([#427](https://github.com/open-mmlab/mmclassification/pull/427))

- Fix TnT compatibility and verbose warning. ([#436](https://github.com/open-mmlab/mmclassification/pull/436))
- Support setting `--out-items` in `tools/test.py`.  ([#437](https://github.com/open-mmlab/mmclassification/pull/437))
- Add datetime info and saving model using torch\<1.6 format. ([#439](https://github.com/open-mmlab/mmclassification/pull/439))
- Improve downstream repositories compatibility. ([#421](https://github.com/open-mmlab/mmclassification/pull/421))
- Rename the option `--options` to `--cfg-options` in some tools. ([#425](https://github.com/open-mmlab/mmclassification/pull/425))
- Add PyTorch 1.9 and Python 3.9 build workflow, and remove some CI. ([#422](https://github.com/open-mmlab/mmclassification/pull/422))

- Fix format error in `test.py` when metric returns `np.ndarray`. ([#441](https://github.com/open-mmlab/mmclassification/pull/441))
- Fix `publish_model` bug if no parent of `out_file`. ([#463](https://github.com/open-mmlab/mmclassification/pull/463))
- Fix num_classes bug in pytorch2onnx.py. ([#458](https://github.com/open-mmlab/mmclassification/pull/458))
- Fix missing runtime requirement `packaging`. ([#459](https://github.com/open-mmlab/mmclassification/pull/459))
- Fix saving simplified model bug in ONNX export tool. ([#438](https://github.com/open-mmlab/mmclassification/pull/438))

- Update `getting_started.md` and `install.md`. And rewrite `finetune.md`. ([#466](https://github.com/open-mmlab/mmclassification/pull/466))

- Use PyTorch style docs theme. ([#457](https://github.com/open-mmlab/mmclassification/pull/457))
- Update metafile and Readme. ([#435](https://github.com/open-mmlab/mmclassification/pull/435))
- Add `CITATION.cff`. ([#428](https://github.com/open-mmlab/mmclassification/pull/428))

## v0.15.0(31/8/2021)

- Support `hparams` argument in `AutoAugment` and `RandAugment` to provide hyperparameters for sub-policies.
- Support custom squeeze channels in `SELayer`.
- Support classwise weight in losses.

- Add `hparams` argument in `AutoAugment` and `RandAugment` and some other improvement. ([#398](https://github.com/open-mmlab/mmclassification/pull/398))
- Support classwise weight in losses. ([#388](https://github.com/open-mmlab/mmclassification/pull/388))

- Enhance `SELayer` to support custom squeeze channels. ([#417](https://github.com/open-mmlab/mmclassification/pull/417))

### Code Refactor

- Better result visualization. ([#419](https://github.com/open-mmlab/mmclassification/pull/419))
- Use `post_process` function to handle pred result processing. ([#390](https://github.com/open-mmlab/mmclassification/pull/390))
- Update `digit_version` function. ([#402](https://github.com/open-mmlab/mmclassification/pull/402))
- Avoid albumentations to install both opencv and opencv-headless. ([#397](https://github.com/open-mmlab/mmclassification/pull/397))
- Avoid unnecessary listdir when building ImageNet. ([#396](https://github.com/open-mmlab/mmclassification/pull/396))
- Use dynamic mmcv download link in TorchServe dockerfile. ([#387](https://github.com/open-mmlab/mmclassification/pull/387))

### Docs Improvement

- Add readme of some algorithms and update meta yml. ([#418](https://github.com/open-mmlab/mmclassification/pull/418))
- Add Copyright information. ([#413](https://github.com/open-mmlab/mmclassification/pull/413))
- Fix typo 'metirc'. ([#411](https://github.com/open-mmlab/mmclassification/pull/411))
- Update QQ group QR code. ([#393](https://github.com/open-mmlab/mmclassification/pull/393))
- Add PR template and modify issue template. ([#380](https://github.com/open-mmlab/mmclassification/pull/380))

## v0.14.0(4/8/2021)

- Add transformer-in-transformer backbone and pretrain checkpoints, refers to [the paper](https://arxiv.org/abs/2103.00112).
- Add Chinese colab tutorial.
- Provide dockerfile to build mmpretrain dev docker image.

- Add transformer in transformer backbone and pretrain checkpoints. ([#339](https://github.com/open-mmlab/mmclassification/pull/339))
- Support mim, welcome to use mim to manage your mmpretrain project. ([#376](https://github.com/open-mmlab/mmclassification/pull/376))
- Add Dockerfile. ([#365](https://github.com/open-mmlab/mmclassification/pull/365))
- Add ResNeSt configs. ([#332](https://github.com/open-mmlab/mmclassification/pull/332))

- Use the `presistent_works` option if available, to accelerate training. ([#349](https://github.com/open-mmlab/mmclassification/pull/349))
- Add Chinese ipynb tutorial. ([#306](https://github.com/open-mmlab/mmclassification/pull/306))
- Refactor unit tests. ([#321](https://github.com/open-mmlab/mmclassification/pull/321))

- Support to test mmdet inference with mmpretrain backbone. ([#343](https://github.com/open-mmlab/mmclassification/pull/343))
- Use zero as default value of `thrs` in metrics. ([#341](https://github.com/open-mmlab/mmclassification/pull/341))

- Fix ImageNet dataset annotation file parse bug. ([#370](https://github.com/open-mmlab/mmclassification/pull/370))
- Fix docstring typo and init bug in ShuffleNetV1. ([#374](https://github.com/open-mmlab/mmclassification/pull/374))
- Use local ATTENTION registry to avoid conflict with other repositories. ([#376](https://github.com/open-mmlab/mmclassification/pull/375))
- Fix swin transformer config bug. ([#355](https://github.com/open-mmlab/mmclassification/pull/355))

- Fix `patch_cfg` argument bug in SwinTransformer. ([#368](https://github.com/open-mmlab/mmclassification/pull/368))
- Fix duplicate `init_weights` call in ViT init function. ([#373](https://github.com/open-mmlab/mmclassification/pull/373))
- Fix broken `_base_` link in a resnet config. ([#361](https://github.com/open-mmlab/mmclassification/pull/361))
- Fix vgg-19 model link missing. ([#363](https://github.com/open-mmlab/mmclassification/pull/363))

## v0.13.0(3/7/2021)

- Support Swin-Transformer backbone and add training configs for Swin-Transformer on ImageNet.

- Support Swin-Transformer backbone and add training configs for Swin-Transformer on ImageNet. (#271)
- Add pretained model of RegNetX. (#269)
- Support adding custom hooks in config file. (#305)
- Improve and add Chinese translation of `CONTRIBUTING.md` and all tools tutorials. (#320)

- Dump config before training. (#282)
- Add torchscript and torchserve deployment tools. (#279, #284)

- Improve test tools and add some new tools. (#322)
- Correct MobilenetV3 backbone structure and add pretained models. (#291)
- Refactor `PatchEmbed` and `HybridEmbed` as independent components. (#330)
- Refactor mixup and cutmix as `Augments` to support more functions. (#278)
- Refactor weights initialization method. (#270, #318, #319)
- Refactor `LabelSmoothLoss` to support multiple calculation formulas. (#285)

- Fix bug for CPU training. (#286)
- Fix missing test data when `num_imgs` can not be evenly divided by `num_gpus`. (#299)
- Fix build compatible with pytorch v1.3-1.5. (#301)
- Fix `magnitude_std` bug in `RandAugment`. (#309)

- Fix bug when `samples_per_gpu` is 1. (#311)

## v0.12.0(3/6/2021)

- Finish adding Chinese tutorials and build Chinese documentation on readthedocs.
- Update ResNeXt checkpoints and ResNet checkpoints on CIFAR.

- Improve and add Chinese translation of `data_pipeline.md` and `new_modules.md`. (#265)
- Build Chinese translation on readthedocs. (#267)
- Add an argument efficientnet_style to `RandomResizedCrop` and `CenterCrop`. (#268)

- Only allow directory operation when rank==0 when testing. (#258)
- Fix typo in `base_head`. (#274)
- Update ResNeXt checkpoints. (#283)

- Add attribute `data.test` in MNIST configs. (#264)
- Download CIFAR/MNIST dataset only on rank 0. (#273)
- Fix MMCV version compatibility. (#276)

- Fix CIFAR color channels bug and update checkpoints in model zoo. (#280)

## v0.11.1(21/5/2021)

- Refine `new_dataset.md` and add Chinese translation of `finture.md`, `new_dataset.md`.

- Add `dim` argument for `GlobalAveragePooling`. (#236)
- Add random noise to `RandAugment` magnitude. (#240)
- Refine `new_dataset.md` and add Chinese translation of `finture.md`, `new_dataset.md`. (#243)

- Refactor arguments passing for Heads. (#239)
- Allow more flexible `magnitude_range` in `RandAugment`. (#249)
- Inherits MMCV registry so that in the future OpenMMLab repos like MMDet and MMSeg could directly use the backbones supported in MMCls. (#252)

- Fix typo in `analyze_results.py`. (#237)
- Fix typo in unittests. (#238)

- Check if specified tmpdir exists when testing to avoid deleting existing data. (#242 & #258)
- Add missing config files in `MANIFEST.in`. (#250 & #255)
- Use temporary directory under shared directory to collect results to avoid unavailability of temporary directory for multi-node testing. (#251)

## v0.11.0(1/5/2021)

- Support cutmix trick.
- Support random augmentation.
- Add `tools/deployment/test.py` as a ONNX runtime test tool.
- Support ViT backbone and add training configs for ViT on ImageNet.
- Add Chinese `README.md` and some Chinese tutorials.

- Support cutmix trick. (#198)
- Add `simplify` option in `pytorch2onnx.py`. (#200)
- Support random augmentation. (#201)
- Add config and checkpoint for training ResNet on CIFAR-100. (#208)
- Add `tools/deployment/test.py` as a ONNX runtime test tool. (#212)

- Support ViT backbone and add training configs for ViT on ImageNet. (#214)
- Add finetuning configs for ViT on ImageNet. (#217)
- Add `device` option to support training on CPU. (#219)
- Add Chinese `README.md` and some Chinese tutorials. (#221)
- Add `metafile.yml` in configs to support interaction with paper with code(PWC) and MMCLI. (#225)
- Upload configs and converted checkpoints for ViT fintuning on ImageNet. (#230)

- Fix `LabelSmoothLoss` so that label smoothing and mixup could be enabled at the same time. (#203)
- Add `cal_acc` option in `ClsHead`. (#206)
- Check `CLASSES` in checkpoint to avoid unexpected key error. (#207)
- Check mmcv version when importing mmpretrain to ensure compatibility. (#209)
- Update `CONTRIBUTING.md` to align with that in MMCV. (#210)

- Change tags to html comments in configs README.md. (#226)
- Clean codes in ViT backbone. (#227)
- Reformat `pytorch2onnx.md` tutorial. (#229)
- Update `setup.py` to support MMCLI. (#232)

- Fix missing `cutmix_prob` in ViT configs. (#220)
- Fix backend for resize in ResNeXt configs. (#222)

## v0.10.0(1/4/2021)

- Support AutoAugmentation
- Add tutorials for installation and usage.

- Add `Rotate` pipeline for data augmentation. (#167)
- Add `Invert` pipeline for data augmentation. (#168)
- Add `Color` pipeline for data augmentation. (#171)
- Add `Solarize` and `Posterize` pipeline for data augmentation. (#172)
- Support fp16 training. (#178)
- Add tutorials for installation and basic usage of MMClassification.(#176)

- Support `AutoAugmentation`, `AutoContrast`, `Equalize`, `Contrast`, `Brightness` and `Sharpness` pipelines for data augmentation. (#179)

- Support dynamic shape export to onnx. (#175)
- Release training configs and update model zoo for fp16 (#184)
- Use MMCV's EvalHook in MMClassification (#182)

- Fix wrong naming in vgg config (#181)

## v0.9.0(1/3/2021)

- Implement mixup trick.
- Add a new tool to create TensorRT engine from ONNX, run inference and verify outputs in Python.

- Implement mixup and provide configs of training ResNet50 using mixup. (#160)
- Add `Shear` pipeline for data augmentation. (#163)
- Add `Translate` pipeline for data augmentation. (#165)

- Add `tools/onnx2tensorrt.py` as a tool to create TensorRT engine from ONNX, run inference and verify outputs in Python. (#153)

- Add `--eval-options` in `tools/test.py` to support eval options override, matching the behavior of other open-mmlab projects. (#158)
- Support showing and saving painted results in `mmpretrain.apis.test` and `tools/test.py`, matching the behavior of other open-mmlab projects. (#162)

- Fix configs for VGG, replace checkpoints converted from other repos with the ones trained by ourselves and upload the missing logs in the model zoo. (#161)

## v0.8.0(31/1/2021)

- Support multi-label task.
- Support more flexible metrics settings.
- Fix bugs.

- Fix bugs.

- Add evaluation metrics: mAP, CP, CR, CF1, OP, OR, OF1 for multi-label task. (#123)
- Add BCE loss for multi-label task. (#130)
- Add focal loss for multi-label task. (#131)
- Support PASCAL VOC 2007 dataset for multi-label task. (#134)
- Add asymmetric loss for multi-label task. (#132)
- Add analyze_results.py to select images for success/fail demonstration. (#142)
- Support new metric that calculates the total number of occurrences of each label. (#143)
- Support class-wise evaluation results. (#143)
- Add thresholds in eval_metrics. (#146)
- Add heads and a baseline config for multilabel task. (#145)

- Remove the models with 0 checkpoint and ignore the repeated papers when counting papers to gain more accurate model statistics. (#135)
- Add tags in README.md. (#137)
- Fix optional issues in docstring. (#138)

- Update stat.py to classify papers. (#139)
- Fix mismatched columns in README.md. (#150)
- Fix test.py to support more evaluation metrics. (#155)

- Fix bug in VGG weight_init. (#140)
- Fix bug in 2 ResNet configs in which outdated heads were used. (#147)
- Fix bug of misordered height and width in `RandomCrop` and `RandomResizedCrop`. (#151)
- Fix missing `meta_keys` in `Collect`. (#149 & #152)

## v0.7.0(31/12/2020)

- Add more evaluation metrics.
- Fix bugs.

- Remove installation of MMCV from requirements. (#90)
- Add 3 evaluation metrics: precision, recall and F-1 score. (#93)
- Allow config override during testing and inference with `--options`. (#91 & #96)

- Use `build_runner` to make runners more flexible. (#54)
- Support to get category ids in `BaseDataset`. (#72)
- Allow `CLASSES` override during `BaseDateset` initialization. (#85)

- Allow input image as ndarray during inference. (#87)
- Optimize MNIST config. (#98)
- Add config links in model zoo documentation. (#99)
- Use functions from MMCV to collect environment. (#103)
- Refactor config files so that they are now categorized by methods. (#116)
- Add README in config directory. (#117)
- Add model statistics. (#119)
- Refactor documentation in consistency with other MM repositories. (#126)

- Add missing `CLASSES` argument to dataset wrappers. (#66)
- Fix slurm evaluation error during training. (#69)
- Resolve error caused by shape in `Accuracy`. (#104)
- Fix bug caused by extremely insufficient data in distributed sampler.(#108)
- Fix bug in `gpu_ids` in distributed training. (#107)

- Fix bug caused by extremely insufficient data in collect results during testing (#114)

## v0.6.0(11/10/2020)

- Support new method: ResNeSt and VGG.
- Support new dataset: CIFAR10.
- Provide new tools to do model inference, model conversion from pytorch to onnx.

- Add model inference. (#16)
- Add pytorch2onnx. (#20)
- Add PIL backend for transform `Resize`. (#21)
- Add ResNeSt. (#25)
- Add VGG and its pretained models. (#27)
- Add CIFAR10 configs and models. (#38)
- Add albumentations transforms. (#45)
- Visualize results on image demo. (#58)

- Replace urlretrieve with urlopen in dataset.utils. (#13)
- Resize image according to its short edge. (#22)
- Update ShuffleNet config. (#31)

- Update pre-trained models for shufflenet_v2, shufflenet_v1, se-resnet50, se-resnet101. (#33)

- Fix init_weights in `shufflenet_v2.py`. (#29)
- Fix the parameter `size` in test_pipeline. (#30)
- Fix the parameter in cosine lr schedule. (#32)
- Fix the convert tools for mobilenet_v2. (#34)
- Fix crash in CenterCrop transform when image is greyscale (#40)
- Fix outdated configs. (#53)

# PPLNN 支持情况

## Usage

Example:

```bash
```

# 第一章：模型部署简介

OpenMMLab 的算法如何部署？这是很多社区用户的困惑。而[模型部署工具箱 MMDeploy](https://zhuanlan.zhihu.com/p/450342651) 的开源，强势打通了从算法模型到应用程序这“最后一公里”！今天我们将开启模型部署入门系列教程，在模型部署开源库 MMDeploy 的辅助下，介绍以下内容：

- 中间表示 ONNX 的定义标准。
- PyTorch 模型转换到 ONNX 模型的方法。
- 推理引擎 ONNX Runtime、TensorRT 的使用方法。
- 部署流水线 PyTorch - ONNX - ONNX Runtime/TensorRT 的示例及常见部署问题的解决方法。
- MMDeploy C/C++ 推理 SDK。

希望通过本系列教程，带领大家学会如何把自己的 PyTorch 模型部署到 ONNX Runtime/TensorRT 上，并学会如何把 OpenMMLab 开源体系中各个计算机视觉任务的模型用 MMDeploy 部署到各个推理引擎上。

我们默认大家熟悉 Python 语言，并对 PyTorch 框架有基本的认识，除此之外不需要了解任何模型部署的知识。

在第一篇文章中，我们将部署一个简单的超分辨率模型，认识中间表示、推理引擎等模型部署中的概念。

## 初识模型部署

在软件工程中，部署指把开发完毕的软件投入使用的过程，包括环境配置、软件安装等步骤。类似地，对于深度学习模型来说，模型部署指让训练好的模型在特定环境中运行的过程。相比于软件部署，模型部署会面临更多的难题：

1. 运行模型所需的环境难以配置。深度学习模型通常是由一些框架编写，比如 PyTorch、TensorFlow。由于框架规模、依赖环境的限制，这些框架不适合在手机、开发板等生产环境中安装。
2. 深度学习模型的结构通常比较庞大，需要大量的算力才能满足实时运行的需求。模型的运行效率需要优化。

因为这些难题的存在，模型部署不能靠简单的环境配置与安装完成。经过工业界和学术界数年的探索，模型部署有了一条流行的流水线：

![pipeline](https://user-images.githubusercontent.com/4560679/156556619-3da7a572-876b-4909-b26f-04e81190c546.png)

为了让模型最终能够部署到某一环境上，开发者们可以使用任意一种深度学习框架来定义网络结构，并通过训练确定网络中的参数。之后，模型的结构和参数会被转换成一种只描述网络结构的中间表示，一些针对网络结构的优化会在中间表示上进行。最后，用面向硬件的高性能编程框架（如 CUDA，OpenCL）编写，能高效执行深度学习网络中算子的推理引擎会把中间表示转换成特定的文件格式，并在对应硬件平台上高效运行模型。

这一条流水线解决了模型部署中的两大问题：使用对接深度学习框架和推理引擎的中间表示，开发者不必担心如何在新环境中运行各个复杂的框架；通过中间表示的网络结构优化和推理引擎对运算的底层优化，模型的运算效率大幅提升。

现在，让我们从一个模型部署的“Hello World”项目入手，见识一下模型部署各方面的知识吧！

## 部署第一个模型

### 创建 PyTorch 模型

仿照 PyTorch 的官方[部署教程](https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html)，让我们用 PyTorch 实现一个超分辨率模型，并把模型部署到 ONNX Runtime 这个推理引擎上。

首先，我们需要创建一个有 PyTorch 库的 Python 编程环境。如果你的 PyTorch 环境还没有装好，可以参考官方的[入门教程](https://pytorch.org/get-started/locally/)。我们强烈推荐使用 conda 来管理 Python 库。使用 conda 可以靠如下的命令初始化一个 PyTorch 环境：

```bash
# 创建预安装 Python 3.7 的名叫 deploy 虚拟环境

```bash
conda create -n deploy python=3.7 -y
# 进入虚拟环境
conda activate deploy
# 安装 cpu 版本的 PyTorch
```

如果你的设备支持 cuda 编程，我们建议你在配置 cuda 环境后使用 gpu 上的 PyTorch。比如将上面安装 PyTorch 的命令改成：

```bash
# 安装 cuda 11.3 的 PyTorch
# 如果你用的是其他版本的 cuda，请参考上面 PyTorch 的官方安装教程选择安装命令
conda install pytorch torchvision cudatoolkit=11.3 -c pytorch
```

本教程会用到其他一些第三方库。你可以用以下命令来安装这些库：

```bash
# 安装 ONNX Runtime, ONNX, OpenCV
pip install onnxruntime onnx opencv-python
```

在一切都配置完毕后，用下面的代码来创建一个超分辨率模型。

```python
import os

import cv2

self.upscale_factor = upscale_factor

        self.conv1 = nn.Conv2d(3,64,kernel_size=9,padding=4)
        self.conv2 = nn.Conv2d(64,32,kernel_size=1,padding=0)
        self.conv3 = nn.Conv2d(32,3,kernel_size=5,padding=2)

        x = self.img_upsampler(x)
        return out

        'https://raw.githubusercontent.com/open-mmlab/mmagic/master/tests/data/face/000001.png']

'https://raw.githubusercontent.com/open-mmlab/mmagic/master/tests/data/face/000001.png']

torch_output = model(torch.from_numpy(input_img)).detach().numpy()

```

在这份代码中，我们创建了一个经典的超分辨率网络 [SRCNN](https://arxiv.org/abs/1501.00092)。SRCNN 先把图像上采样到对应分辨率，再用 3 个卷积层处理图像。为了方便起见，我们跳过训练网络的步骤，直接下载模型权重（由于 MMagic 中 SRCNN 的权重结构和我们定义的模型不太一样，我们修改了权重字典的 key 来适配我们定义的模型），同时下载好输入图片。为了让模型输出成正确的图片格式，我们把模型的输出转换成 HWC 格式，并保证每一通道的颜色值都在 0~255 之间。如果脚本正常运行的话，一幅超分辨率的人脸照片会保存在 `face_torch.png` 中。

在 PyTorch 模型测试正确后，我们来正式开始部署这个模型。我们下一步的任务是把 PyTorch 模型转换成用中间表示 ONNX 描述的模型。

![face_torch](https://user-images.githubusercontent.com/4560679/156558692-e5b82284-22d1-434b-aace-b565ac223e73.png)

### 中间表示 —— ONNX

在介绍 ONNX 之前，我们先从本质上来认识一下神经网络的结构。神经网络实际上只是描述了数据计算的过程，其结构可以用计算图表示。比如 `a+b` 可以用下面的计算图来表示：

![a+b](https://user-images.githubusercontent.com/4560679/156558717-96bbe544-4dc7-4460-8850-3cb1790e39ec.png)

为了加速计算，一些框架会使用对神经网络“先编译，后执行”的静态图来描述网络。静态图的缺点是难以描述控制流（比如 if-else 分支语句和 for 循环语句），直接对其引入控制语句会导致产生不同的计算图。比如循环执行 n 次 `a=a+b`，对于不同的 n，会生成不同的计算图：

![n=2](https://user-images.githubusercontent.com/4560679/156558606-6ff18e19-f3b1-463f-8f83-60bf6f7ef64b.png)

ONNX（Open Neural Network Exchange）是 Facebook 和微软在 2017 年共同发布的，用于标准描述计算图的一种格式。目前，在数家机构的共同维护下，ONNX 已经对接了多种深度学习框架和多种推理引擎。因此，ONNX 被当成了深度学习框架到推理引擎的桥梁，就像编译器的中间语言一样。由于各框架兼容性不一，我们通常只用 ONNX 表示更容易部署的静态图。

让我们用下面的代码来把 PyTorch 的模型转换成 ONNX 格式的模型：

```python

        model,
        x,
        "srcnn.onnx",
        input_names=['input'],
        output_names=['output'])
```

        model,
        x,
```

其中，**torch.onnx.export** 是 PyTorch 自带的把模型转换成 ONNX 格式的函数。让我们先看一下前三个必选参数：前三个参数分别是要转换的模型、模型的任意一组输入、导出的 ONNX 文件的文件名。转换模型时，需要原模型和输出文件名是很容易理解的，但为什么需要为模型提供一组输入呢？这就涉及到 ONNX 转换的原理了。从 PyTorch 的模型到 ONNX 的模型，本质上是一种语言上的翻译。直觉上的想法是像编译器一样彻底解析原模型的代码，记录所有控制流。但前面也讲到，我们通常只用 ONNX 记录不考虑控制流的静态图。因此，PyTorch 提供了一种叫做追踪（trace）的模型转换方法：给定一组输入，再实际执行一遍模型，即把这组输入对应的计算图记录下来，保存为 ONNX 格式。**export** 函数用的就是追踪导出方法，需要给任意一组输入，让模型跑起来。我们的测试图片是三通道，256x256 大小的，这里也构造一个同样形状的随机张量。

剩下的参数中，`opset_version` 表示 ONNX 算子集的版本。深度学习的发展会不断诞生新算子，为了支持这些新增的算子，ONNX 会经常发布新的算子集，目前已经更新 15 个版本。 我们令 `opset_version=11`，即使用第 11 个 ONNX 算子集，是因为 SRCNN 中的 bicubic（双三次插值）在 opset11 中才得到支持。剩下的两个参数 `input_names`, `output_names` 是输入、输出 tensor 的名称，我们稍后会用到这些名称。

如果上述代码运行成功，目录下会新增一个 `srcnn.onnx` 的 ONNX 模型文件。我们可以用下面的脚本来验证一下模型文件是否正确。

```python

```python

onnx_model = onnx.load("srcnn.onnx")
try:
    onnx.checker.check_model(onnx_model)
    print("Model incorrect")
else:
    print("Model correct")
```

其中，**onnx.load** 函数用于读取一个 ONNX 模型。**onnx.checker.check_model** 用于检查模型格式是否正确，如果有错误的话该函数会直接报错。我们的模型是正确的，控制台中应该会打印出 "Model correct"。

接下来，让我们来看一看 ONNX 模型具体的结构是怎么样的。我们可以使用 Netron（开源的模型可视化工具）来可视化 ONNX 模型。把 `srcnn.onnx` 文件从本地的文件系统拖入网站，即可看到如下的可视化结果：

![model](https://user-images.githubusercontent.com/4560679/156558675-df96e7f8-0c90-4b52-81db-f80e21e522a1.png)

点击 input 或者 output，可以查看 ONNX 模型的基本信息，包括模型的版本信息，以及模型输入、输出的名称和数据类型。

![model_property](https://user-images.githubusercontent.com/4560679/156558624-0d77bf2c-bd01-40e3-a89c-1b0f69329576.png)

点击某一个算子节点，可以看到算子的具体信息。比如点击第一个 Conv 可以看到：

![node_property](https://user-images.githubusercontent.com/4560679/156558668-867ea202-9ac2-4a04-b836-91ced4f2e5ea.png)

每个算子记录了算子属性、图结构、权重三类信息。

- 算子属性信息即图中 attributes 里的信息，对于卷积来说，算子属性包括了卷积核大小（kernel_shape）、卷积步长（strides）等内容。这些算子属性最终会用来生成一个具体的算子。
- 图结构信息指算子节点在计算图中的名称、邻边的信息。对于图中的卷积来说，该算子节点叫做 Conv_2，输入数据叫做 11，输出数据叫做 12。根据每个算子节点的图结构信息，就能完整地复原出网络的计算图。
- 权重信息指的是网络经过训练后，算子存储的权重信息。对于卷积来说，权重信息包括卷积核的权重值和卷积后的偏差值。点击图中 conv1.weight, conv1.bias 后面的加号即可看到权重信息的具体内容。

现在，我们有了 SRCNN 的 ONNX 模型。让我们看看最后该如何把这个模型运行起来。

### 推理引擎 —— ONNX Runtime

**ONNX Runtime** 是由微软维护的一个跨平台机器学习推理加速器，也就是我们前面提到的“推理引擎”。ONNX Runtime 是直接对接 ONNX 的，即 ONNX Runtime 可以直接读取并运行 `.onnx` 文件，而不需要再把 `.onnx` 格式的文件转换成其他格式的文件。也就是说，对于 `PyTorch - ONNX - ONNX Runtime` 这条部署流水线，只要在目标设备中得到 `.onnx` 文件，并在 ONNX Runtime 上运行模型，模型部署就算大功告成了。

通过刚刚的操作，我们把 PyTorch 编写的模型转换成了 ONNX 模型，并通过可视化检查了模型的正确性。最后，让我们用 ONNX Runtime 运行一下模型，完成模型部署的最后一步。

ONNX Runtime 提供了 Python 接口。接着刚才的脚本，我们可以添加如下代码运行模型：

```python

```python

ort_session = onnxruntime.InferenceSession("srcnn.onnx")
ort_inputs = {'input': input_img}
ort_output = ort_session.run(['output'], ort_inputs)[0]

cv2.imwrite("face_ort.png", ort_output)
```

这段代码中，除去后处理操作外，和 ONNX Runtime 相关的代码只有三行。让我们简单解析一下这三行代码。**onnxruntime.InferenceSession** 用于获取一个 ONNX Runtime 推理器，其参数是用于推理的 ONNX 模型文件。推理器的 run 方法用于模型推理，其第一个参数为输出张量名的列表，第二个参数为输入值的字典。其中输入值字典的 key 为张量名，value 为 numpy 类型的张量值。输入输出张量的名称需要和 **torch.onnx.export** 中设置的输入输出名对应。

如果代码正常运行的话，另一幅超分辨率照片会保存在 `face_ort.png` 中。这幅图片和刚刚得到的 `face_torch.png` 是一模一样的。这说明 ONNX Runtime 成功运行了 SRCNN 模型，模型部署完成了！以后有用户想实现超分辨率的操作，我们只需要提供一个 `srcnn.onnx` 文件，并帮助用户配置好 ONNX Runtime 的 Python 环境，用几行代码就可以运行模型了。或者还有更简便的方法，我们可以利用 ONNX Runtime 编译出一个可以直接执行模型的应用程序。我们只需要给用户提供 ONNX 模型文件，并让用户在应用程序选择要执行的 ONNX 模型文件名就可以运行模型了。

## 总结

在这篇教程里，我们利用成熟的模型部署工具，轻松部署了一个初始版本的超分辨率模型 SRCNN。但在实际应用场景中，随着模型结构的复杂度不断加深，碰到的困难的也会越来越多。在下一篇教程里，我们将“升级”一下这个超分辨率模型，让它支持动态的输入。

看完这篇教程，是不是感觉知识太多一下消化不过来？没关系，模型部署本身有非常多的东西要学。为了举例的方便，这篇教程包含了许多未来才会讲到的知识点。事实上，读完这篇教程后，记下以下知识点就够了：

- 模型部署，指把训练好的模型在特定环境中运行的过程。模型部署要解决模型框架兼容性差和模型运行速度慢这两大问题。
- 模型部署的常见流水线是“深度学习框架-中间表示-推理引擎”。其中比较常用的一个中间表示是 ONNX。
- 深度学习模型实际上就是一个计算图。模型部署时通常把模型转换成静态的计算图，即没有控制流（分支语句、循环语句）的计算图。
- PyTorch 框架自带对 ONNX 的支持，只需要构造一组随机的输入，并对模型调用 **torch.onnx.export** 即可完成 PyTorch 到 ONNX 的转换。
- 推理引擎 ONNX Runtime 对 ONNX 模型有原生的支持。给定一个 `.onnx` 文件，只需要简单使用 ONNX Runtime 的 Python API 就可以完成模型推理。

为了实现深度学习算法的落地，充满挑战的模型部署是一个逃不开的步骤。MMDeploy 实现了 OpenMMLab 中目标检测、图像分割、超分辨率等多个视觉任务模型的部署，支持 ONNX Runtime，TensorRT，ncnn，openppl，OpenVINO 等多个推理引擎。

在后续的模型部署教程中，我们将在介绍模型部署技术的同时，介绍这些技术是如何运用在 MMDeploy 中的。

### Prepare SUN RGB-D Data

1. Download SUNRGBD data [HERE](http://rgbd.cs.princeton.edu/data/). Then, move SUNRGBD.zip, SUNRGBDMeta2DBB_v2.mat, SUNRGBDMeta3DBB_v2.mat and SUNRGBDtoolbox.zip to the OFFICIAL_SUNRGBD folder, unzip the zip files.

2. Enter the `matlab` folder, Extract point clouds and annotations by running `extract_split.m`, `extract_rgbd_data_v2.m` and `extract_rgbd_data_v1.m`.

3. Enter the project root directory, Generate training data by running

```bash
python tools/create_data.py sunrgbd --root-path ./data/sunrgbd --out-dir ./data/sunrgbd --extra-tag sunrgbd
```

```bash
cd matlab
cd ../../..

cd matlab
python tools/create_data.py sunrgbd --root-path ./data/sunrgbd  --out-dir ./data/sunrgbd --extra-tag sunrgbd
```

NOTE: SUNRGBDtoolbox.zip should have MD5 hash `18d22e1761d36352f37232cba102f91f` (you can check the hash with `md5 SUNRGBDtoolbox.zip` on Mac OS or `md5sum SUNRGBDtoolbox.zip` on Linux)

NOTE: If you would like to play around with [ImVoteNet](../../configs/imvotenet/README.md), the image data (`./data/sunrgbd/sunrgbd_trainval/image`) are required. If you pre-processed the data before mmdet3d version 0.12.0, please pre-process the data again due to some updates in data pre-processing

```bash
```

```
sunrgbd

```bash
```

```
sunrgbd
├── matlab
│   ├── calib
│   ├── depth
│   ├── image
│   ├── label
├── points

```

# Omnisource

<!-- TODO: add links to the tech report -->

We propose to train a recognizer that can classify images and videos. The recognizer is jointly trained on image and video datasets. Compared with pre-training on the same image dataset, this method can significantly improve the video recognition performance.

<!-- [IMAGE]

</div>
-->

| frame sampling strategy |   scheduler   | resolution | gpus | backbone | joint-training | top1 acc | top5 acc | testing protocol  | FLOPs  | params |            config             |            ckpt             |             log             |

| :---------------------: | :-----------: | :--------: | :--: | :------: | :------------: | :------: | :------: | :---------------: | :----: | :----: | :---------------------------: | :-------------------------: | :-------------------------: |
|          8x8x1          | Linear+Cosine |  224x224   |  8   | ResNet50 |    ImageNet    |  77.30   |  93.23   | 10 clips x 3 crop | 54.75G | 32.45M | [config](/configs/recognition/omnisource/slowonly_r50_8xb16-8x8x1-256e_imagenet-kinetics400-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/omnisource/slowonly_r50_8xb16-8x8x1-256e_imagenet-kinetics400-rgb_20230208-61c4be0d.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/omnisource/slowonly_r50_8xb16-8x8x1-256e_imagenet-kinetics400-rgb.log) |

## Train

```shell
```

```shell
python tools/train.py configs/recognition/omnisource/slowonly_r50_8xb16-8x8x1-256e_imagenet-kinetics400-rgb.py \
```

We found that the training of this Omnisource model could crash for unknown reasons. If this happens, you can resume training by adding the `--cfg-options resume=True` to the training script.

## Test

```shell
```

## Test

```shell
```

```shell
python tools/test.py configs/recognition/omnisource/slowonly_r50_8xb16-8x8x1-256e_imagenet-kinetics400-rgb.py \
```

```BibTeX
}
```

```BibTeX

pages={6202--6211},
}
```

```BibTeX
}
```

```shell
```

<details>

```bibtex
}
```

<details>

```bibtex

```bibtex
}
```

| [ViTPose-S](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-small_8xb64-210e_coco-256x192.py) |  256x192   | 0.739 |      0.903      |      0.816      | 0.792 |      0.942      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-small_8xb64-210e_coco-256x192-62d7a712_20230314.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-small_8xb64-210e_coco-256x192-62d7a712_20230314.json) |

| [ViTPose-B](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-base_8xb64-210e_coco-256x192.py) |  256x192   | 0.757 |      0.905      |      0.829      | 0.810 |      0.946      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-base_8xb64-210e_coco-256x192-216eae50_20230314.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-base_8xb64-210e_coco-256x192-216eae50_20230314.json) |

| [ViTPose-L](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-large_8xb64-210e_coco-256x192.py) |  256x192   | 0.782 |      0.914      |      0.850      | 0.834 |      0.952      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-large_8xb64-210e_coco-256x192-53609f55_20230314.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-large_8xb64-210e_coco-256x192-53609f55_20230314.json) |

| [ViTPose-H](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-huge_8xb64-210e_coco-256x192.py) |  256x192   | 0.788 |      0.917      |      0.855      | 0.839 |      0.954      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-huge_8xb64-210e_coco-256x192-e32adcd4_20230314.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-huge_8xb64-210e_coco-256x192-e32adcd4_20230314.json) |
| [ViTPose-H\*](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-huge_8xb64-210e_coco-256x192.py) |  256x192   | 0.790 |      0.916      |      0.857      | 0.840 |      0.953      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-huge_3rdparty_coco-256x192-5b738c8e_20230314) |                       -                       |

*Models with * are converted from the [official repo](https://github.com/ViTAE-Transformer/ViTPose).  The config files of these models are only for validation.*

> With simple decoder

| [ViTPose-S](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-small-simple_8xb64-210e_coco-256x192.py) |  256x192   | 0.736 |      0.900      |      0.811      | 0.790 |      0.940      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-small-simple_8xb64-210e_coco-256x192-4c101a76_20230314.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-small-simple_8xb64-210e_coco-256x192-4c101a76_20230314.json) |

| [ViTPose-B](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-base-simple_8xb64-210e_coco-256x192.py) |  256x192   | 0.756 |      0.906      |      0.826      | 0.809 |      0.946      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-base-simple_8xb64-210e_coco-256x192-0b8234ea_20230407.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-base-simple_8xb64-210e_coco-256x192-0b8234ea_20230407.json) |

| [ViTPose-L](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-large-simple_8xb64-210e_coco-256x192.py) |  256x192   | 0.780 |      0.914      |      0.851      | 0.833 |      0.952      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-large-simple_8xb64-210e_coco-256x192-3a7ee9e1_20230314.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-large-simple_8xb64-210e_coco-256x192-3a7ee9e1_20230314.json) |

| [ViTPose-H](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-huge-simple_8xb64-210e_coco-256x192.py) |  256x192   | 0.789 |      0.916      |      0.856      | 0.839 |      0.953      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-huge-simple_8xb64-210e_coco-256x192-ffd48c05_20230314.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-huge-simple_8xb64-210e_coco-256x192-ffd48c05_20230314.json) |

| [rtmpose-t](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-t_8xb256-420e_coco-256x192.py) |  256x192   | 0.682 |      0.883      |      0.759      | 0.736 |      0.920      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-tiny_simcc-coco_pt-aic-coco_420e-256x192-e613ba3f_20230127.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-tiny_simcc-coco_pt-aic-coco_420e-256x192-e613ba3f_20230127.json) |
| [rtmpose-s](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-s_8xb256-420e_coco-256x192.py) |  256x192   | 0.716 |      0.892      |      0.789      | 0.768 |      0.929      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_simcc-coco_pt-aic-coco_420e-256x192-8edcf0d7_20230127.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_simcc-coco_pt-aic-coco_420e-256x192-8edcf0d7_20230127.json) |

| [rtmpose-m](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-m_8xb256-420e_coco-256x192.py) |  256x192   | 0.746 |      0.899      |      0.817      | 0.795 |      0.935      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-coco_pt-aic-coco_420e-256x192-d8dd5ca4_20230127.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-coco_pt-aic-coco_420e-256x192-d8dd5ca4_20230127.json) |
| [rtmpose-l](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-l_8xb256-420e_coco-256x192.py) |  256x192   | 0.758 |      0.906      |      0.826      | 0.806 |      0.942      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-coco_pt-aic-coco_420e-256x192-1352a4d2_20230127.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-coco_pt-aic-coco_420e-256x192-1352a4d2_20230127.json) |

| [rtmpose-t-aic-coco](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-t_8xb256-420e_aic-coco-256x192.py) |  256x192   | 0.685 |      0.880      |      0.761      | 0.738 |      0.918      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-tiny_simcc-aic-coco_pt-aic-coco_420e-256x192-cfc8f33d_20230126.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-tiny_simcc-aic-coco_pt-aic-coco_420e-256x192-cfc8f33d_20230126.json) |
| [rtmpose-s-aic-coco](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-s_8xb256-420e_aic-coco-256x192.py) |  256x192   | 0.722 |      0.892      |      0.794      | 0.772 |      0.929      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_simcc-aic-coco_pt-aic-coco_420e-256x192-fcb2599b_20230126.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-s_simcc-aic-coco_pt-aic-coco_420e-256x192-fcb2599b_20230126.json) |

| [rtmpose-m-aic-coco](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-m_8xb256-420e_aic-coco-256x192.py) |  256x192   | 0.758 |      0.903      |      0.826      | 0.806 |      0.940      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-aic-coco_pt-aic-coco_420e-256x192-63eb25f7_20230126.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-aic-coco_pt-aic-coco_420e-256x192-63eb25f7_20230126.json) |
| [rtmpose-l-aic-coco](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-l_8xb256-420e_aic-coco-256x192.py) |  256x192   | 0.765 |      0.906      |      0.835      | 0.813 |      0.942      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-aic-coco_pt-aic-coco_420e-256x192-f016ffe0_20230126.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-aic-coco_pt-aic-coco_420e-256x192-f016ffe0_20230126.json) |

| [rtmpose-m-aic-coco](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-m_8xb256-420e_aic-coco-384x288.py) |  384x288   | 0.770 |      0.908      |      0.833      | 0.816 |      0.943      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-aic-coco_pt-aic-coco_420e-384x288-a62a0b32_20230228.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-m_simcc-aic-coco_pt-aic-coco_420e-384x288-a62a0b32_20230228.json) |
| [rtmpose-l-aic-coco](/configs/body_2d_keypoint/rtmpose/coco/rtmpose-l_8xb256-420e_aic-coco-384x288.py) |  384x288   | 0.773 |      0.907      |      0.835      | 0.819 |      0.942      | [ckpt](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-aic-coco_pt-aic-coco_420e-384x288-97d6cb0f_20230228.pth) | [log](https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/rtmpose-l_simcc-aic-coco_pt-aic-coco_420e-384x288-97d6cb0f_20230228.json) |

# TVM feature support

MMDeploy has integrated TVM for model conversion and SDK. Features include:

- AutoTVM tuner
- Ansor tuner
- Graph Executor runtime
- Virtual machine runtime

# Build for RISC-V

MMDeploy chooses ncnn as the inference backend on RISC-V platform. The deployment process consists of two steps:

Model conversion: Convert the PyTorch model to the ncnn model on the host side, and then upload the converted model to the device.

Model deployment: Compile ncnn and MMDeploy in cross-compilation mode on the host side, and then upload the executable for inference.

## 1. Model conversion

a) Install MMDeploy

You can refer to [Build document](./linux-x86_64.md) to install ncnn inference engine and MMDeploy

b) Convert model

```bash

```

## 2. Model deployment

a) Download the compiler toolchain and set environment

```bash
# download Xuantie-900-gcc-linux-5.10.4-glibc-x86_64-V2.2.6-20220516.tar.gz
```

b) Compile ncnn & opencv

```bash
# ncnn
# refer to https://github.com/Tencent/ncnn/wiki/how-to-build#build-for-allwinner-d1

# opencv
cmake .. \

cmake .. \
```

c) Compile mmdeploy SDK & demo

```bash
cmake .. \

```

After `make install`, the examples will locate in `install\bin`

```
.

```

```
.
├── ocr
```

### 4) Run the demo

First make sure that `--dump-info` is used during convert model, so that the `resnet18` directory has the files required by the SDK such as `pipeline.json`.

Copy the model folder(resnet18), executable(image_classification) file and test image(tests/data/tiger.jpeg) to the device.

```bash
```

# 自定义数据流水线

在本教程中，我们将介绍如何为你的任务构建数据流水线（即，数据转换）的一些方法。

- [自定义数据流水线](#自定义数据流水线)
  - [数据流水线设计](#数据流水线设计)
  - [修改训练/测试数据流水线](#修改训练/测试数据流水线)
    - [加载](#加载)
    - [采样帧和其他处理](#采样帧和其他处理)
    - [格式化](#格式化)
  - [添加新的数据转换](#添加新的数据转换)

## 数据流水线设计

数据流水线指的是从数据集索引样本时处理数据样本字典的过程，它包括一系列的数据转换。每个数据转换接受一个 `dict` 作为输入，对其进行处理，并产生一个 `dict` 作为输出，供序列中的后续数据转换使用。

以下是一个例子，用于使用 `VideoDataset` 在 Kinetics 上训练 SlowFast 的数据流水线。这个数据流水线首先使用 [`decord`](https://github.com/dmlc/decord) 读取原始视频并随机采样一个视频剪辑，该剪辑包含 `32` 帧，帧间隔为 `2`。然后，它对所有帧应用随机大小调整的裁剪和随机水平翻转，然后将数据形状格式化为 `NCTHW`，在这个例子中，它是 `(1, 3, 32, 224, 224)`。

```python

dict(type='RandomResizedCrop'),
]
```

MMAction2 中所有可用的数据转换的详细列表可以在 [mmaction.datasets.transforms](mmaction.datasets.transforms) 中找到。

## 修改训练/测试数据流水线

MMAction2 的数据流水线非常灵活，因为几乎每一步的数据预处理都可以从配置文件中进行配置。然而，对于一些用户来说，这种多样性可能会让人感到不知所措。

以下是一些用于构建动作识别任务数据流水线的一般实践和指南。

### 加载

在数据流水线的开始，通常是加载视频。然而，如果帧已经被提取出来，你应该使用 `RawFrameDecode` 并修改数据集类型为 `RawframeDataset`。

```python
]
```

]
```

如果你需要从具有不同格式（例如，`pkl`，`bin`等）的文件或从特定位置加载数据，你可以创建一个新的加载转换并将其包含在数据流水线的开始。有关更多详细信息，请参阅[添加新的数据转换](#添加新的数据转换)。

### 采样帧和其他处理

在训练和测试过程中，我们可能会有从视频中采样帧的不同策略。

例如，当测试 SlowFast 时，我们会均匀地采样多个剪辑，如下所示：

```python
    ...
    dict(
    ...
]
```

在上述例子中，每个视频将均匀地采样10个视频剪辑，每个剪辑包含32帧。 `test_mode=True` 用于实现这一点，与训练期间的随机采样相反。

另一个例子涉及 `TSN/TSM` 模型，它们从视频中采样多个片段：

```python
    ...
    ...
]
```

```python
    ...
    ...
]
```

通常，数据流水线中的数据增强只处理视频级的转换，例如调整大小或裁剪，而不处理像视频标准化或 mixup/cutmix 这样的转换。这是因为我们可以在批量视频数据上进行视频标准化和 mixup/cutmix，以使用 GPU 加速处理。要配置视频标准化和 mixup/cutmix，请使用 [mmaction.models.utils.data_preprocessor](mmaction.models.utils.data_preprocessor)。

### 格式化

格式化涉及从数据信息字典中收集训练数据，并将其转换为与模型兼容的格式。

在大多数情况下，你可以简单地使用 [`PackActionInputs`](mmaction.datasets.transforms.PackActionInputs)，它将以 `NumPy Array` 格式的图像转换为 `PyTorch Tensor`，并将地面真实类别信息和其他元信息打包为一个类似字典的对象 [`ActionDataSample`](mmaction.structures.ActionDataSample)。

```python
    ...
]
```

## 添加新的数据转换

```python
    ...
]
```

1. 要创建一个新的数据转换，编写一个新的转换类在一个 Python 文件中，例如，名为 `my_transforms.py`。数据转换类必须继承 [`mmcv.transforms.BaseTransform`](mmcv.transforms.BaseTransform) 类，并重写 `transform` 方法，该方法接受一个 `dict` 作为输入并返回一个 `dict`。最后，将 `my_transforms.py` 放在 `mmaction/datasets/transforms/` 文件夹中。

   ```python

           # 修改数据信息字典 `results`。
   ```

2. 在 `mmaction/datasets/transforms/__init__.py` 中导入新类。

   ```python
   ...

   __all__ = [
   ]
   ```

   ```python
       ...

```python
   ...

   __all__ = [
   ]
   ```

   ```python
       ...
       ...
   ]
   ```

# FAQ

We list some potential troubles encountered by users and developers, along with their corresponding solutions. Feel free to enrich the list if you find any frequent issues and contribute your solutions to solve them. If you have any trouble with environment configuration, model training, etc, please create an issue using the [provided templates](https://github.com/open-mmlab/mmdetection3d/blob/master/.github/ISSUE_TEMPLATE/error-report.md) and fill in all required information in the template.

## MMEngine/MMCV/MMDet/MMDet3D Installation

- Compatibility issue between MMEngine, MMCV, MMDetection and MMDetection3D; "ConvWS is already registered in conv layer"; "AssertionError: MMCV==xxx is used but incompatible. Please install mmcv>=xxx, \<=xxx."

- The required versions of MMEngine, MMCV and MMDetection for different versions of MMDetection3D are as below. Please install the correct version of MMEngine, MMCV and MMDetection to avoid installation issues.

| MMDetection3D version |     MMEngine version     |      MMCV version       |  MMDetection version  |
  | --------------------- | :----------------------: | :---------------------: | :-------------------: |
  | dev-1.x               | mmengine>=0.8.0, \<1.0.0 | mmcv>=2.0.0rc4, \<2.1.0 | mmdet>=3.0.0, \<3.2.0 |
  | main                  | mmengine>=0.8.0, \<1.0.0 | mmcv>=2.0.0rc4, \<2.1.0 | mmdet>=3.0.0, \<3.2.0 |
  | v1.2.0                | mmengine>=0.8.0, \<1.0.0 | mmcv>=2.0.0rc4, \<2.1.0 | mmdet>=3.0.0, \<3.2.0 |
  | v1.1.1                | mmengine>=0.7.1, \<1.0.0 | mmcv>=2.0.0rc4, \<2.1.0 | mmdet>=3.0.0, \<3.1.0 |

  **Note:** If you want to install mmdet3d-v1.0.0rcx, the compatible MMDetection, MMSegmentation and MMCV versions table can be found at [here](https://mmdetection3d.readthedocs.io/en/latest/faq.html#mmcv-mmdet-mmdet3d-installation). Please choose the correct version of MMCV, MMDetection and MMSegmentation to avoid installation issues.

- If you faced the error shown below when importing open3d:

  please downgrade open3d to 0.9.0.0, because the latest open3d needs the support of file 'GLIBC_2.27', which only exists in Ubuntu 18.04, not in Ubuntu 16.04.

- If you faced the error when importing pycocotools, this is because nuscenes-devkit installs pycocotools but mmdet relies on mmpycocotools. The current workaround is as below. We will migrate to use pycocotools in the future.

  ```shell
  ```

  **NOTE**: We have migrated to use pycocotools in mmdet3d >= 0.13.0.

- If you face the error shown below when importing pycocotools:

  please downgrade pycocotools to 2.0.1 because of the incompatibility between the newest pycocotools and numpy \< 1.20.0. Or you can compile and install the latest pycocotools from source as below:

  or

- If you face some errors about numba in cuda-9.0 environment, you should check the version of numba. In cuda-9.0 environment, the high version of numba is not supported and we suggest you could install numba==0.53.0.

## How to annotate point cloud?

MMDetection3D does not support point cloud annotation. Some open-source annotation tool are offered for reference:

Besides, we improved [LATTE](https://github.com/bernwang/latte) for better use. More details can be found [here](https://arxiv.org/abs/2011.10174).

<details>

```bibtex
}
```

<details>

```bibtex

| [pose_alexnet](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_alexnet_8xb64-210e_coco-256x192.py) |  256x192   | 0.448 |      0.767      |      0.461      | 0.521 |      0.829      | [ckpt](https://download.openmmlab.com/mmpose/top_down/alexnet/alexnet_coco_256x192-a7b1fd15_20200727.pth) | [log](https://download.openmmlab.com/mmpose/top_down/alexnet/alexnet_coco_256x192_20200727.log.json) |

# KV Cache 量化和测试结果

对于最大长度是 2048 的 LLaMa-7B fp16 模型，服务端每创建 1 个并发，都需要大约 1030MB 显存保存 kv_cache，即便是 A100 80G，能服务的用户也非常有限。

为了降低运行时显存，我们实现了 kv cache PTQ 量化，使用的公式如下：

```bash
```

## 如何开启 KV Cache INT8

### **第一步**

把 huggingface 格式的模型，转成 turbomind 推理格式，得到一个 workspace 目录

```bash
```

如果已经有 workspace 目录，可以跳过这步。

### **第二步**

获取量化参数

```bash
  --work_dir /path/to/internlm-chat-7b  \             # huggingface 模型目录
  --turbomind_dir workspace/trition_models/weights/ \ # 保存量化参数的目录
  --kv_sym False \                                    # 对称量化或非对称量化，默认为 False
  --num_tp 1  \                                       # Tensor 并行使用的 GPU 数，和 deploy.py 保持一致
```

`kv_qparams` 会在 `weights` 目录生成 fp32 缩放系数，文件格式是 `numpy.tofile` 产生的二进制。

--num_tp 1  \                                       # Tensor 并行使用的 GPU 数，和 deploy.py 保持一致
```

也可以先把 `turbomind_dir` 设成私有目录，再把缩放系数拷贝进 `workspace/trition_models/weights/`。

### **第三步**

修改 `workspace/trition_models/weights/config.ini`：

- use_context_fmha 改为 0，表示关闭 flashattention
- quant_policy 设置为 4。表示打开 kv_cache int8

这是因为 flashattention 有 v1、v2 两个版本，kv cache int8 曾经也实现过对称版本。

排列组合需要实现 4 套 kernel，在算法不确定的时候过早优化，对软件来说是场灾难。

### **第四步**

测试聊天效果

```bash
```

## 显存测试

测试对象为 [internlm-chat-7b](https://huggingface.co/internlm/internlm-chat-7b) 模型。
测试方法：

1. 使用 `deploy.py` 转换模型，修改 `workspace` 配置中的最大并发数；调整 `llama_config.ini` 中的请求数
2. 编译执行 `bin/llama_triton_example`，获取 fp16 版本在不同 batch_size 的显存情况
3. 开启量化，重新执行 `bin/llama_triton_example`，获取 int8 版本在不同 batch_size 显存情况

以下是两个版本的显存对比：

相对于直接量化 Weight（如 [GPTQ-for-LLaMa](https://github.com/qwopqwop200/GPTQ-for-LLaMa/)），我们做了两种方案在 7B 模型中的内存增长对比预估，部分数据来自 [llama.cpp](https://github.com/ggerganov/llama.cpp)。

可以看到，fp16 版本每个并发需要 1030MB 显存，因此量化 kv_cache 能显著降低运行时的显存增长速度。

## 精度测试

测试对象为 [internlm-chat-7b](https://huggingface.co/internlm/internlm-chat-7b) 指令模型。

以下是 `kCacheKVInt8` 方法仅从 c4 数据集，随机选择 128 条数据 PTQ 量化。量化前后均使用 [opencompass](https://github.com/InternLM/opencompass) 测试精度。

需要注意的是，`kCacheKVInt8` 和 `WeightInt4` 两种方案可以同时开启。

# 常见问题解答

我们在这里列出了使用时的一些常见问题及其相应的解决方案。 如果您发现有一些问题被遗漏，请随时提 PR 丰富这个列表。 如果您无法在此获得帮助，请使用 [issue模板](https://github.com/open-mmlab/mmdetection/blob/main/.github/ISSUE_TEMPLATE/error-report.md/)创建问题，但是请在模板中填写所有必填信息，这有助于我们更快定位问题。

## PyTorch 2.0 支持

MMDetection 目前绝大部分算法已经支持了 PyTorch 2.0 及其 `torch.compile` 功能, 用户只需要安装 MMDetection 3.0.0rc7 及其以上版本即可。如果你在使用中发现有不支持的算法，欢迎给我们反馈。我们也非常欢迎社区贡献者来 benchmark 对比 `torch.compile` 功能所带来的速度提升。

如果你想启动 `torch.compile` 功能，只需要在 `train.py` 或者 `test.py` 后面加上 `--cfg-options compile=True`。 以 RTMDet 为例，你可以使用以下命令启动 `torch.compile` 功能：

```shell
# 单卡

# 单机 8 卡

# 单机 8 卡 + AMP 混合精度训练
```

```

需要特别注意的是，PyTorch 2.0 对于动态 shape 支持不是非常完善，目标检测算法中大部分不仅输入 shape 是动态的，而且 loss 计算和后处理过程中也是动态的，这会导致在开启 `torch.compile` 功能后训练速度会变慢。基于此，如果你想启动 `torch.compile` 功能，则应该遵循如下原则：

1. 输入到网络的图片是固定 shape 的，而非多尺度的
2. 设置 `torch._dynamo.config.cache_size_limit` 参数。TorchDynamo 会将 Python 字节码转换并缓存，已编译的函数会被存入缓存中。当下一次检查发现需要重新编译时，该函数会被重新编译并缓存。但是如果重编译次数超过预设的最大值（64），则该函数将不再被缓存或重新编译。前面说过目标检测算法中的 loss 计算和后处理部分也是动态计算的，这些函数需要在每次迭代中重新编译。因此将 `torch._dynamo.config.cache_size_limit` 参数设置得更小一些可以有效减少编译时间

在 MMDetection 中可以通过环境变量 `DYNAMO_CACHE_SIZE_LIMIT` 设置 `torch._dynamo.config.cache_size_limit` 参数，以 RTMDet 为例，命令如下所示：

```shell
# 单卡

# 单机 8 卡

# 单机 8 卡
```

关于 PyTorch 2.0 的 dynamo 常见问题，可以参考 [这里](https://pytorch.org/docs/stable/dynamo/faq.html)

## 安装

- MMCV 与 MMDetection 的兼容问题: "ConvWS is already registered in conv layer"; "AssertionError: MMCV==xxx is used but incompatible. Please install mmcv>=xxx, \<=xxx."

  MMDetection，MMEngine 和 MMCV 的版本兼容关系如下。请选择合适的版本避免安装错误 。

  | MMDetection 版本 |        MMCV 版本        |      MMEngine 版本       |
  | :--------------: | :---------------------: | :----------------------: |
  |       main       |  mmcv>=2.0.0, \<2.1.0   | mmengine>=0.7.1, \<1.0.0 |
  |      3.1.0       |  mmcv>=2.0.0, \<2.1.0   | mmengine>=0.7.1, \<1.0.0 |
  |      3.0.0       |  mmcv>=2.0.0, \<2.1.0   | mmengine>=0.7.1, \<1.0.0 |
  |     3.0.0rc6     | mmcv>=2.0.0rc4, \<2.1.0 | mmengine>=0.6.0, \<1.0.0 |
  |     3.0.0rc5     | mmcv>=2.0.0rc1, \<2.1.0 | mmengine>=0.3.0, \<1.0.0 |

|     3.0.0rc6     | mmcv>=2.0.0rc4, \<2.1.0 | mmengine>=0.6.0, \<1.0.0 |
  |     3.0.0rc4     | mmcv>=2.0.0rc1, \<2.1.0 | mmengine>=0.3.0, \<1.0.0 |
  |     3.0.0rc3     | mmcv>=2.0.0rc1, \<2.1.0 | mmengine>=0.3.0, \<1.0.0 |
  |     3.0.0rc2     | mmcv>=2.0.0rc1, \<2.1.0 | mmengine>=0.1.0, \<1.0.0 |
  |     3.0.0rc1     | mmcv>=2.0.0rc1, \<2.1.0 | mmengine>=0.1.0, \<1.0.0 |
  |     3.0.0rc0     | mmcv>=2.0.0rc1, \<2.1.0 | mmengine>=0.1.0, \<1.0.0 |

  **注意：**

  1. 如果你希望安装 mmdet-v2.x, MMDetection 和 MMCV 版本兼容表可以在 [这里](https://mmdetection.readthedocs.io/en/stable/faq.html#installation) 找到，请选择合适的版本避免安装错误。
  2. 在 MMCV-v2.x 中，`mmcv-full` 改名为 `mmcv`，如果你想安装不包含 CUDA 算子的版本，可以选择安装 MMCV 精简版 `mmcv-lite`。

  原因是安装了 `mmcv-lite` 而不是 `mmcv`。

  1. `pip uninstall mmcv-lite` 卸载安装的 `mmcv-lite`

  2. 安装 `mmcv` 根据 [安装说明](https://mmcv.readthedocs.io/zh_CN/2.x/get_started/installation.html)。

- 在 Windows 环境下安装过程中遇到 "Microsoft Visual C++ 14.0 or graeter is required" error .

  这个错误发生在 pycotools 的 'pycocotools.\_mask' 扩展构建过程，其原因是缺少了对应 C++ 环境依赖。你需要到微软官方下载[对应工具](https://visualstudio.microsoft.com/zh-hans/visual-cpp-build-tools/)，选择“使用 C++ 的桌面开发”选项安装最小依赖，随后重新安装 pycocotools。

- 使用 albumentations

如果你希望使用 `albumentations`，我们建议使用 `pip install -r requirements/albu.txt`
或者 `pip install -U albumentations --no-binary qudida,albumentations` 进行安装。
如果简单地使用 `pip install albumentations>=0.3.2` 进行安装，
则会同时安装 `opencv-python-headless`（即便已经安装了 `opencv-python` 也会再次安装）。
我们建议在安装 `albumentations` 后检查环境，以确保没有同时安装 `opencv-python` 和 `opencv-python-headless`，

因为同时安装可能会导致一些问题。更多细节请参考[官方文档](https://albumentations.ai/docs/getting_started/installation/#note-on-opencv-dependencies) 。

- 在某些算法中出现 ModuleNotFoundError 错误

一些算法或者数据需要额外的依赖，例如 Instaboost、 Panoptic Segmentation、 LVIS dataset 等。请注意错误信息并安装相应的包，例如：

```shell
# 安装 instaboost 依赖
# 安装 panoptic segmentation 依赖
pip install git+https://github.com/cocodataset/panopticapi.git
# 安装 LVIS dataset 依赖
pip install git+https://github.com/lvis-dataset/lvis-api.git
```

## 代码

- 修改一些代码后是否需要重新安装 mmdet

如果你遵循最佳实践，即使用 `pip install -v -e .` 安装的 mmdet，则对本地代码所作的任何修改都会生效，无需重新安装

- 如何使用多个 MMDetection 版本进行开发

你可以拥有多个文件夹，例如 mmdet-3.0，mmdet-3.1。

要使环境中安装默认的 MMDetection 而不是当前正在在使用的，可以删除出现在相关脚本中的代码：

```shell
```

## PyTorch/CUDA 环境相关

```shell
```

  1. 临时解决方案为使用命令 `MMCV_WITH_OPS=1 MMCV_CUDA_ARGS='-gencode=arch=compute_80,code=sm_80' pip install -e .` 进行编译。 常见报错信息为 `nvcc fatal : Unsupported gpu architecture 'compute_86'` 意思是你的编译器不支持 sm_86 架构(包括英伟达 30 系列的显卡)的优化，至 CUDA toolkit 11.0 依旧未支持. 这个命令是通过增加宏 `MMCV_CUDA_ARGS='-gencode=arch=compute_80,code=sm_80` 让 nvcc 编译器为英伟达 30 系列显卡进行 `sm_80` 的优化，虽然这有可能会无法发挥出显卡所有性能。

  2. 有开发者已经在 [pytorch/pytorch#47585](https://github.com/pytorch/pytorch/pull/47585) 更新了 PyTorch 默认的编译 flag， 但是我们对此并没有进行测试。

- "invalid device function" 或者 "no kernel image is available for execution".

  1. 检查您正常安装了 CUDA runtime (一般在`/usr/local/`)，或者使用 `nvcc --version` 检查本地版本，有时安装 PyTorch 会顺带安装一个 CUDA runtime，并且实际优先使用 conda 环境中的版本，你可以使用 `conda list cudatoolkit` 查看其版本。

1. 检查您正常安装了 CUDA runtime (一般在`/usr/local/`)，或者使用 `nvcc --version` 检查本地版本，有时安装 PyTorch 会顺带安装一个 CUDA runtime，并且实际优先使用 conda 环境中的版本，你可以使用 `conda list cudatoolkit` 查看其版本。

  2. 编译 extension 的 CUDA Toolkit 版本与运行时的 CUDA Toolkit 版本是否相符，

     - 如果您从源码自己编译的，使用 `python mmdet/utils/collect_env.py` 检查编译编译 extension 的 CUDA Toolkit 版本，然后使用 `conda list cudatoolkit` 检查当前 conda 环境是否有 CUDA Toolkit，若有检查版本是否匹配， 如不匹配，更换 conda 环境的 CUDA Toolkit，或者使用匹配的 CUDA Toolkit 中的 nvcc 编译即可，如环境中无 CUDA Toolkit，可以使用 `nvcc -V`。

       等命令查看当前使用的 CUDA runtime。

     - 如果您是通过 pip 下载的预编译好的版本，请确保与当前 CUDA runtime 一致。

  3. 运行 `python mmdet/utils/collect_env.py` 检查是否为正确的 GPU 架构编译的 PyTorch， torchvision， 与 MMCV。 你或许需要设置 `TORCH_CUDA_ARCH_LIST` 来重新安装 MMCV，可以参考 [GPU 架构表](https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#gpu-feature-list),
     例如， 运行 `TORCH_CUDA_ARCH_LIST=7.0 pip install mmcv` 为 Volta GPU 编译 MMCV。这种架构不匹配的问题一般会出现在使用一些旧型号的 GPU 时候出现， 例如， Tesla K80。

- "undefined symbol" 或者 "cannot open xxx.so".

例如， 运行 `TORCH_CUDA_ARCH_LIST=7.0 pip install mmcv` 为 Volta GPU 编译 MMCV。这种架构不匹配的问题一般会出现在使用一些旧型号的 GPU 时候出现， 例如， Tesla K80。

  1. 如果这些 symbol 属于 CUDA/C++ (如 libcudart.so 或者 GLIBCXX)，使用 `python mmdet/utils/collect_env.py`检查 CUDA/GCC runtime 与编译 MMCV 的 CUDA 版本是否相同。
  2. 如果这些 symbols 属于 PyTorch，(例如, symbols containing caffe, aten, and TH), 检查当前 Pytorch 版本是否与编译 MMCV 的版本一致。
  3. 运行 `python mmdet/utils/collect_env.py` 检查 PyTorch， torchvision， MMCV 等的编译环境与运行环境一致。

  1. 如果你在使用 miniconda 而不是 anaconda，检查是否正确的安装了 Cython 如 [#3379](https://github.com/open-mmlab/mmdetection/issues/3379).
  2. 检查环境中的 `setuptools`, `Cython`, and `PyTorch` 相互之间版本是否匹配。

2. 检查环境中的 `setuptools`, `Cython`, and `PyTorch` 相互之间版本是否匹配。

  1. 检查 GCC 的版本，通常是因为 PyTorch 版本与 GCC 版本不匹配 （例如 GCC \< 4.9 )，我们推荐用户使用 GCC 5.4，我们也不推荐使用 GCC 5.5， 因为有反馈 GCC 5.5 会导致 "segmentation fault" 并且切换到 GCC 5.4 就可以解决问题。

  2. 检查是否正确安装了 CUDA 版本的 PyTorch 。

     ```shell
     ```

     是否返回True。

  3. 如果 `torch` 的安装是正确的，检查是否正确编译了 MMCV。

     ```shell
     ```

  4. 如果 MMCV 与 PyTorch 都被正确安装了，则使用 `ipdb`, `pdb` 设置断点，直接查找哪一部分的代码导致了 `segmentation fault`。

## Training 相关

  1. 检查数据的标注是否正常， 长或宽为 0 的框可能会导致回归 loss 变为 nan，一些小尺寸（宽度或高度小于 1）的框在数据增强（例如，instaboost）后也会导致此问题。 因此，可以检查标注并过滤掉那些特别小甚至面积为 0 的框，并关闭一些可能会导致 0 面积框出现数据增强。
  2. 降低学习率：由于某些原因，例如 batch size 大小的变化， 导致当前学习率可能太大。 您可以降低为可以稳定训练模型的值。
  3. 延长 warm up 的时间：一些模型在训练初始时对学习率很敏感，您可以把 `warmup_iters` 从 500 更改为 1000 或 2000。

2. 降低学习率：由于某些原因，例如 batch size 大小的变化， 导致当前学习率可能太大。 您可以降低为可以稳定训练模型的值。
  4. 添加 gradient clipping: 一些模型需要梯度裁剪来稳定训练过程。 默认的 `grad_clip` 是 `None`,  你可以在 config 设置 `optimizer_config=dict(_delete_=True, grad_clip=dict(max_norm=35, norm_type=2))`  如果你的 config 没有继承任何包含 `optimizer_config=dict(grad_clip=None)`,  你可以直接设置`optimizer_config=dict(grad_clip=dict(max_norm=35, norm_type=2))`.

  1. 存在大量 ground truth boxes 或者大量 anchor 的场景，可能在 assigner 会 OOM。 您可以在 assigner 的配置中设置 `gpu_assign_thr=N`，这样当超过 N 个 GT boxes 时，assigner 会通过 CPU 计算 IOU。

  2. 在 backbone 中设置 `with_cp=True`。 这使用 PyTorch 中的 `sublinear strategy` 来降低 backbone 占用的 GPU 显存。

  3. 使用 `config/fp16` 中的示例尝试混合精度训练。`loss_scale` 可能需要针对不同模型进行调整。

  4. 你也可以尝试使用 `AvoidCUDAOOM` 来避免该问题。首先它将尝试调用 `torch.cuda.empty_cache()`。如果失败，将会尝试把输入类型转换到 FP16。如果仍然失败，将会把输入从 GPUs 转换到 CPUs 进行计算。这里提供了两个使用的例子：

     ```python

4. 你也可以尝试使用 `AvoidCUDAOOM` 来避免该问题。首先它将尝试调用 `torch.cuda.empty_cache()`。如果失败，将会尝试把输入类型转换到 FP16。如果仍然失败，将会把输入从 GPUs 转换到 CPUs 进行计算。这里提供了两个使用的例子：

     ```python

     ```

     你也可也使用 `AvoidCUDAOOM` 作为装饰器让代码遇到 OOM 的时候继续运行：

     ```python

         ...
         return xxx
     ```

  1. 这个错误出现在存在参数没有在 forward 中使用，容易在 DDP 中运行不同分支时发生。
  2. 你可以在 config 设置 `find_unused_parameters = True` 进行训练 (会降低训练速度)。
  3. 你也可以通过在 config 中的 `optimizer_config` 里设置 `detect_anomalous_params=True` 查找哪些参数没有用到，但是需要 MMCV 的版本 >= 1.4.1。

- 训练中保存最好模型

2. 你可以在 config 设置 `find_unused_parameters = True` 进行训练 (会降低训练速度)。

  可以通过配置 `default_hooks = dict(checkpoint=dict(type='CheckpointHook', interval=1, save_best='auto')`开启。在 `auto` 参数情况下会根据返回的验证结果中的第一个 key 作为选择最优模型的依据，你也可以直接设置评估结果中的 key 来手动设置，例如 `save_best='coco/bbox_mAP'`。

- 在 Resume 训练中使用 `ExpMomentumEMAHook`

  如果在训练中使用了 `ExpMomentumEMAHook`，那么 resume 时候不能仅仅通过命令行参数 `--resume-from` 或 `--cfg-options resume_from` 实现恢复模型参数功能例如 `python tools/train.py configs/yolox/yolox_s_8x8_300e_coco.py --resume-from ./work_dir/yolox_s_8x8_300e_coco/epoch_x.pth`。以 `yolox_s` 算法为例，由于 `ExpMomentumEMAHook` 需要重新加载权重，你可以通过如下做法实现：

  ```python
  # 直接打开 configs/yolox/yolox_s_8x8_300e_coco.py 修改所有 resume_from 字段
  resume_from=./work_dir/yolox_s_8x8_300e_coco/epoch_x.pth
  custom_hooks=[...
      dict(
          type='ExpMomentumEMAHook',

# 直接打开 configs/yolox/yolox_s_8x8_300e_coco.py 修改所有 resume_from 字段
      dict(
          resume_from=./work_dir/yolox_s_8x8_300e_coco/epoch_x.pth,
          momentum=0.0001,
          priority=49)
      ]
  ```

## Evaluation 相关

- 使用 COCO Dataset 的测评接口时, 测评结果中 AP 或者 AR = -1
  1. 根据COCO数据集的定义，一张图像中的中等物体与小物体面积的阈值分别为 9216（96\*96）与 1024（32\*32）。
  2. 如果在某个区间没有检测框 AP 与 AR 认定为 -1.

## Model 相关

- **ResNet style 参数说明**

  ResNet style 可选参数允许 `pytorch` 和 `caffe`，其差别在于 Bottleneck 模块。Bottleneck 是 `1x1-3x3-1x1` 堆叠结构，在 `caffe` 模式模式下 stride=2 参数放置在第一个 `1x1` 卷积处，而 `pyorch` 模式下 stride=2 放在第二个 `3x3` 卷积处。一个简单示例如下：

  ```python
  else:
  ```

- **ResNeXt 参数说明**

if self.style == 'pytorch':
  else:
  ```

  ResNeXt 来自论文 [`Aggregated Residual Transformations for Deep Neural Networks`](https://arxiv.org/abs/1611.05431). 其引入分组卷积，并且通过变量基数来控制组的数量达到精度和复杂度的平衡，其有两个超参 `baseWidth` 和 `cardinality `来控制内部 Bottleneck 模块的基本宽度和分组数参数。以 MMDetection 中配置名为 `mask_rcnn_x101_64x4d_fpn_mstrain-poly_3x_coco.py` 为例，其中 `mask_rcnn` 代表算法采用 Mask R-CNN，`x101` 代表骨架网络采用 ResNeXt-101，`64x4d`代表 Bottleneck 一共分成 64 组，每组的基本宽度是 4。

- **骨架网络 eval 模式说明**

  因为检测模型通常比较大且输入图片分辨率很高，这会导致检测模型的 batch 很小，通常是 2，这会使得 BatchNorm 在训练过程计算的统计量方差非常大，不如主干网络预训练时得到的统计量稳定，因此在训练是一般都会使用 `norm_eval=True` 模式，直接使用预训练主干网络中的 BatchNorm 统计量，少数使用大 batch 的算法是 `norm_eval=False` 模式，例如 NASFPN。对于没有 ImageNet 预训练的骨架网络，如果 batch 比较小，可以考虑使用 `SyncBN`。

# H3DNet: 3D Object Detection Using Hybrid Geometric Primitives

> [H3DNet: 3D Object Detection Using Hybrid Geometric Primitives](https://arxiv.org/abs/2006.05682)

We introduce H3DNet, which takes a colorless 3D point cloud as input and outputs a collection of oriented object bounding boxes (or BB) and their semantic labels. The critical idea of H3DNet is to predict a hybrid set of geometric primitives, i.e., BB centers, BB face centers, and BB edge centers. We show how to convert the predicted geometric primitives into object proposals by defining a distance function between an object and the geometric primitives. This distance function enables continuous optimization of object proposals, and its local minimums provide high-fidelity object proposals. H3DNet then utilizes a matching and refinement module to classify object proposals into detected objects and fine-tune the geometric parameters of the detected objects. The hybrid set of geometric primitives not only provides more accurate signals for object detection than using a single type of geometric primitives, but it also provides an overcomplete set of constraints on the resulting 3D layout. Therefore, H3DNet can tolerate outliers in predicted geometric primitives. Our model achieves state-of-the-art 3D detection results on two large datasets with real 3D scans, ScanNet and SUN RGB-D.

<img src="https://user-images.githubusercontent.com/36950400/143868884-26f7fc63-93fd-48cb-a469-e2f55fda5550.png" width="800"/>
</div>

We implement H3DNet and provide the result and checkpoints on ScanNet datasets.

|                   Backbone                    | Lr schd | Mem (GB) | Inf time (fps) | AP@0.25 | AP@0.5 |                                                                                                                                                             Download                                                                                                                                                             |

| :-------------------------------------------: | :-----: | :------: | :------------: | :-----: | :----: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| [MultiBackbone](./h3dnet_8xb3_scannet-seg.py) |   3x    |   7.9    |                |  66.07  | 47.68  | [model](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/h3dnet/h3dnet_3x8_scannet-3d-18class/h3dnet_3x8_scannet-3d-18class_20210824_003149-414bd304.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.0.0_models/h3dnet/h3dnet_scannet-3d-18class/h3dnet_3x8_scannet-3d-18class_20210824_003149.log.json) |

**Notice**: If your current mmdetection3d version >= 0.6.0, and you are using the checkpoints downloaded from the above links or using checkpoints trained with mmdetection3d version \< 0.6.0, the checkpoints have to be first converted via [tools/model_converters/convert_h3dnet_checkpoints.py](../../tools/model_converters/convert_h3dnet_checkpoints.py):

```
python ./tools/model_converters/convert_h3dnet_checkpoints.py ${ORIGINAL_CHECKPOINT_PATH} --out=${NEW_CHECKPOINT_PATH}
```

```latex
@inproceedings{zhang2020h3dnet,
    author = {Zhang, Zaiwei and Sun, Bo and Yang, Haitao and Huang, Qixing},
    title = {H3DNet: 3D Object Detection Using Hybrid Geometric Primitives},
    booktitle = {Proceedings of the European Conference on Computer Vision},
}
```

<details>

```bibtex
}
```

<details>

```bibtex

| [pose_shufflenetv1](/configs/body_2d_keypoint/topdown_heatmap/mpii/td-hm_shufflenetv1_8xb64-210e_mpii-256x256.py) |  256x256   | 0.824 |  0.195   | [ckpt](https://download.openmmlab.com/mmpose/top_down/shufflenetv1/shufflenetv1_mpii_256x256-dcc1c896_20200925.pth) | [log](https://download.openmmlab.com/mmpose/top_down/shufflenetv1/shufflenetv1_mpii_256x256_20200925.log.json) |

# Prepare Models

To support the evaluation of new models in OpenCompass, there are several ways:

1. HuggingFace-based models
2. API-based models
3. Custom models

## HuggingFace-based Models

In OpenCompass, we support constructing evaluation models directly from HuggingFace's
`AutoModel.from_pretrained` and `AutoModelForCausalLM.from_pretrained` interfaces. If the model to be
evaluated follows the typical generation interface of HuggingFace models, there is no need to write code. You
can simply specify the relevant configurations in the configuration file.

Here is an example configuration file for a HuggingFace-based model:

```python
# Use `HuggingFace` to evaluate models supported by AutoModel.
# Use `HuggingFaceCausalLM` to evaluate models supported by AutoModelForCausalLM.

models = [
    dict(
        # Parameters for `HuggingFaceCausalLM` initialization.

models = [
    dict(
        # Common parameters shared by various models, not specific to `HuggingFaceCausalLM` initialization.
        abbr='llama-7b',            # Model abbreviation used for result display.
        max_out_len=100,            # Maximum number of generated tokens.
        batch_size=16,              # The size of a batch during inference.
        run_cfg=dict(num_gpus=1),   # Run configuration to specify resource requirements.
    )
]
```

Explanation of some of the parameters:

- `batch_padding=False`: If set to False, each sample in a batch will be inferred individually. If set to True,

)
]
```

  a batch of samples will be padded and inferred together. For some models, such padding may lead to
  unexpected results. If the model being evaluated supports sample padding, you can set this parameter to True
  to speed up inference.
- `padding_side='left'`: Perform padding on the left side. Not all models support padding, and padding on the
  right side may interfere with the model's output.
- `truncation_side='left'`: Perform truncation on the left side. The input prompt for evaluation usually
  consists of both the in-context examples prompt and the input prompt. If the right side of the input prompt
  is truncated, it may cause the input of the generation model to be inconsistent with the expected format.
  Therefore, if necessary, truncation should be performed on the left side.

is truncated, it may cause the input of the generation model to be inconsistent with the expected format.

During evaluation, OpenCompass will instantiate the evaluation model based on the `type` and the
initialization parameters specified in the configuration file. Other parameters are used for inference,
summarization, and other processes related to the model. For example, in the above configuration file, we will
instantiate the model as follows during evaluation:

```python
)
```

## API-based Models

Currently, OpenCompass supports API-based model inference for the following:

- OpenAI (`opencompass.models.OpenAI`)
- More coming soon

max_seq_len=2048,
)
```

Let's take the OpenAI configuration file as an example to see how API-based models are used in the
configuration file.

```python

models = [
    dict(
        type=OpenAI,                             # Using the OpenAI model
        # Parameters for `OpenAI` initialization
        path='gpt-4',                            # Specify the model type
        max_seq_len=2048,                        # The max input number of tokens
        # Common parameters shared by various models, not specific to `OpenAI` initialization.
        abbr='GPT-4',                            # Model abbreviation used for result display.
        max_out_len=512,                         # Maximum number of generated tokens.

abbr='GPT-4',                            # Model abbreviation used for result display.
        batch_size=1,                            # The size of a batch during inference.
        run_cfg=dict(num_gpus=0),                # Resource requirements (no GPU needed)
    ),
]
```

## Custom Models

If the above methods do not support your model evaluation requirements, you can refer to
[Supporting New Models](../advanced_guides/new_model.md) to add support for new models in OpenCompass.

# BasicVSR (CVPR'2021)

<details>
<summary align="right">BasicVSR (CVPR'2021)</summary>

```bibtex
}
```

<br/>

| [basicvsr_reds4](./basicvsr_2xb4_reds4.py) |       **31.4170/0.8909**        |           36.2848/0.9395            |        27.2694/0.8318        |        33.4478/0.9306         |           34.4700/0.9286            |        24.4541/0.7455        | 2 (Tesla V100-PCIE-32GB) | [模型](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_reds4_20120409-0e599677.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_reds4_20210409_092646.log.json) |

| [basicvsr_vimeo90k_bi](./basicvsr_2xb4_vimeo90k-bi.py) |         30.3128/0.8660          |         **37.2026/0.9451**          |      **27.2755/0.8248**      |        34.5554/0.9434         |           34.8097/0.9316            |        25.0517/0.7636        | 2 (Tesla V100-PCIE-32GB) | [模型](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bi_20210409-d2d8f760.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bi_20210409_132702.log.json) |

| [basicvsr_vimeo90k_bd](./basicvsr_2xb4_vimeo90k-bd.py) |         29.0376/0.8481          |           34.6427/0.9335            |        26.2708/0.8022        |      **39.9953/0.9695**       |         **37.5501/0.9499**          |      **27.9791/0.8556**      | 2 (Tesla V100-PCIE-32GB) | [模型](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bd_20210409-0154dd64.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bd_20210409_132740.log.json) |

## 快速开始

**训练**

<details>

```shell
# CPU上训练
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/basicvsr/basicvsr_2xb4_reds4.py

# 单个GPU上训练
python tools/train.py configs/basicvsr/basicvsr_2xb4_reds4.py

# 多个GPU上训练
./tools/dist_train.sh configs/basicvsr/basicvsr_2xb4_reds4.py 8
```

**测试**

<details>

```

**测试**

<details>

```shell
# CPU上测试
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/basicvsr/basicvsr_2xb4_reds4.py https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_reds4_20120409-0e599677.pth

# 单个GPU上测试
python tools/test.py configs/basicvsr/basicvsr_2xb4_reds4.py https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_reds4_20120409-0e599677.pth

# 多个GPU上测试
./tools/dist_test.sh configs/basicvsr/basicvsr_2xb4_reds4.py https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_reds4_20120409-0e599677.pth 8
```

# 准备 Kinetics-710

## 介绍

```BibTeX
}
```

关于基本数据集信息，请参考 [论文](https://arxiv.org/pdf/2211.09552.pdf)。这些脚本可以用于准备 kinetics-710。MMAction2 以 Concat Daataset 的形式支持了 Kinetics-710 数据集，我们只提供一个注释文件列表，并利用 Kinetics-400/600/700 数据集的原始数据。你可以参考 [配置](/configs/recognition/uniformerv2/uniformerv2-base-p16-res224_clip_u8_kinetics710-rgb.py) 了解详情，它也提供了一个模板配置，说明了如何在 MMAction2 中使用 Concat Dataset。
在我们开始之前，请确保目录位于 `$MMACTION2`。

## 第一步：下载 Kinetics 400/600/700

Kinetics-710 是基于 Kinetics-400/600/700 的视频数据集，它合并了这些 Kinetics 数据集的训练集，并根据 Youtube ID 删除了重复的视频。MMAction2 提供了一个基于 Kinetics-400/600/700 的 OpenDataLab 版本的标注文件，你可以通过 [MIM](https://github.com/open-mmlab/mim) 从 OpenDataLab 下载。

```shell
odl login
# 下载 Kinetics-400/600/700，注意这可能需要很长时间。

```

## 第二步：下载 Kinetics-710 标注文件

我们提供了与 OpenDataLab 版本 Kinetics 相对应的 Kinetics-710 标注文件列表，你可以从阿里云下载它，并将其解压到 `$MMACTION2/data/`

```shell

```

## 第三步：文件夹结构

完成 Kinetics 准备的整个数据流程后。
你可以得到 Kinetics-710 的视频和注释文件。

在整个项目目录下（仅针对 Kinetics），*最小*的文件夹结构如下：
（*最小*意味着一些数据是不必要的：例如，你可能想要使用原始视频格式评估 kinetics。）

```
mmaction2

```

```
mmaction2
├── tools
├── data
│   │   │   ├── ...
│   │   │   ├── ...
│   │   │   ├── ...
```

```

关于在 Kinetics 上进行训练和评估，请参考 [训练和测试教程](/docs/en/user_guides/train_test.md)。

# 支持新模型

目前我们已经支持的模型有 HF 模型、部分模型 API 、部分第三方模型。

## 新增API模型

新增基于API的模型，需要在 `opencompass/models` 下新建 `mymodel_api.py` 文件，继承 `BaseAPIModel`，并实现 `generate` 方法来进行推理，以及 `get_token_len` 方法来计算 token 的长度。在定义好之后修改对应配置文件名称即可。

```python

                 path: str,
        ...

        self,
        inputs,
        pass

self,
        inputs,
        pass

        pass
```

## 新增第三方模型

新增基于API的模型，需要在 `opencompass/models` 下新建 `mymodel.py` 文件，继承 `BaseModel`，并实现  `generate` 方法来进行生成式推理， `get_ppl` 方法来进行判别式推理，以及 `get_token_len` 方法来计算 token 的长度。在定义好之后修改对应配置文件名称即可。

```python

        ...

        pass

def get_token_len(self, prompt: str) -> int:
        pass

        pass

        pass
```

# MASTER

> [MASTER: Multi-aspect non-local network for scene text recognition](https://arxiv.org/abs/1910.02562)

Attention-based scene text recognizers have gained huge success, which leverages a more compact intermediate representation to learn 1d- or 2d- attention by a RNN-based encoder-decoder architecture. However, such methods suffer from attention-drift problem because high similarity among encoded features leads to attention confusion under the RNN-based local attention mechanism. Moreover, RNN-based methods have low efficiency due to poor parallelization. To overcome these problems, we propose the MASTER, a self-attention based scene text recognizer that (1) not only encodes the input-output attention but also learns self-attention which encodes feature-feature and target-target relationships inside the encoder and decoder and (2) learns a more powerful and robust intermediate representation to spatial distortion, and (3) owns a great training efficiency because of high training parallelization and a high-speed inference because of an efficient memory-cache mechanism. Extensive experiments on various benchmarks demonstrate the superior performance of our MASTER on both regular and irregular scene text.

<img src="https://user-images.githubusercontent.com/65173622/164642001-037f81b7-37dd-4808-a6a9-09ff6f6a17ea.JPG">
</div>

## Dataset

| SynthAdd  |   1216889    |     1      | synth  |

|                             Methods                             |   Backbone    |        | Regular Text |           |     |           | Irregular Text |        |                             download                              |
| :-------------------------------------------------------------: | :-----------: | :----: | :----------: | :-------: | :-: | :-------: | :------------: | :----: | :---------------------------------------------------------------: |
|                                                                 |               | IIIT5K |     SVT      | IC13-1015 |     | IC15-2077 |      SVTP      |  CT80  |                                                                   |

| [MASTER](/configs/textrecog/master/master_resnet31_12e_st_mj_sa.py) | R31-GCAModule | 0.9490 |    0.8887    |  0.9517   |     |  0.7650   |     0.8465     | 0.8889 | [model](https://download.openmmlab.com/mmocr/textrecog/master/master_resnet31_12e_st_mj_sa/master_resnet31_12e_st_mj_sa_20220915_152443-f4a5cabc.pth) \| [log](https://download.openmmlab.com/mmocr/textrecog/master/master_resnet31_12e_st_mj_sa/20220915_152443.log) |
| [MASTER-TTA](/configs/textrecog/master/master_resnet31_12e_st_mj_sa.py) | R31-GCAModule | 0.9450 |    0.8887    |  0.9478   |     |  0.7906   |     0.8481     | 0.8958 |                                                                   |

```bibtex
@article{Lu2021MASTER,
  title={MASTER: Multi-Aspect Non-local Network for Scene Text Recognition},
  author={Ning Lu and Wenwen Yu and Xianbiao Qi and Yihao Chen and Ping Gong and Rong Xiao and Xiang Bai},
  journal={Pattern Recognition},
}
```

  ```bash
  ```

  ```none
  ├── mmseg
  ├── tools
  ├── data

  ```

  View the Mapillary Vistas Datasets config file here [V1.2](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/_base_/datasets/mapillary_v1.py) and  [V2.0](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/_base_/datasets/mapillary_v2.py)

- **View datasets labels index and palette**

- **Mapillary Vistas Datasets labels information**
  **v1.2 information**

  ```none
  There are 66 labels classes in v1.2
  0--Bird--[165, 42, 42],
  1--Ground Animal--[0, 192, 0],
  2--Curb--[196, 196, 196],
  3--Fence--[190, 153, 153],
  4--Guard Rail--[180, 165, 180],
  5--Barrier--[90, 120, 150],
  6--Wall--[102, 102, 156],
  7--Bike Lane--[128, 64, 255],
  8--Crosswalk - Plain--[140, 140, 200],
  9--Curb Cut--[170, 170, 170],
  10--Parking--[250, 170, 160],
  11--Pedestrian Area--[96, 96, 96],
  12--Rail Track--[230, 150, 140],
  13--Road--[128, 64, 128],
  14--Service Lane--[110, 110, 110],
  15--Sidewalk--[244, 35, 232],
  16--Bridge--[150, 100, 100],
  17--Building--[70, 70, 70],
  18--Tunnel--[150, 120, 90],
  19--Person--[220, 20, 60],
  20--Bicyclist--[255, 0, 0],
  21--Motorcyclist--[255, 0, 100],
  22--Other Rider--[255, 0, 200],
  23--Lane Marking - Crosswalk--[200, 128, 128],
  24--Lane Marking - General--[255, 255, 255],

20--Bicyclist--[255, 0, 0],
  25--Mountain--[64, 170, 64],
  26--Sand--[230, 160, 50],
  27--Sky--[70, 130, 180],
  28--Snow--[190, 255, 255],
  29--Terrain--[152, 251, 152],
  30--Vegetation--[107, 142, 35],
  31--Water--[0, 170, 30],
  32--Banner--[255, 255, 128],
  33--Bench--[250, 0, 30],
  34--Bike Rack--[100, 140, 180],
  35--Billboard--[220, 220, 220],
  36--Catch Basin--[220, 128, 128],
  37--CCTV Camera--[222, 40, 40],
  38--Fire Hydrant--[100, 170, 30],
  39--Junction Box--[40, 40, 40],
  40--Mailbox--[33, 33, 33],
  41--Manhole--[100, 128, 160],
  42--Phone Booth--[142, 0, 0],
  43--Pothole--[70, 100, 150],
  44--Street Light--[210, 170, 100],
  45--Pole--[153, 153, 153],
  46--Traffic Sign Frame--[128, 128, 128],
  47--Utility Pole--[0, 0, 80],
  48--Traffic Light--[250, 170, 30],

44--Street Light--[210, 170, 100],
  49--Traffic Sign (Back)--[192, 192, 192],
  50--Traffic Sign (Front)--[220, 220, 0],
  51--Trash Can--[140, 140, 20],
  52--Bicycle--[119, 11, 32],
  53--Boat--[150, 0, 255],
  54--Bus--[0, 60, 100],
  55--Car--[0, 0, 142],
  56--Caravan--[0, 0, 90],
  57--Motorcycle--[0, 0, 230],
  58--On Rails--[0, 80, 100],
  59--Other Vehicle--[128, 64, 64],
  60--Trailer--[0, 0, 110],
  61--Truck--[0, 0, 70],
  62--Wheeled Slow--[0, 0, 192],
  63--Car Mount--[32, 32, 32],
  64--Ego Vehicle--[120, 10, 10],
  65--Unlabeled--[0, 0, 0]
  ```

  **v2.0 information**

  ```none
  There are 124 labels classes in v2.0
  2--Ambiguous Barrier--[250, 170, 31],
  3--Concrete Block--[250, 170, 32],
  4--Curb--[196, 196, 196],
  5--Fence--[190, 153, 153],

0--Bird--[165, 42, 42],
  6--Guard Rail--[180, 165, 180],
  7--Barrier--[90, 120, 150],
  8--Road Median--[250, 170, 33],
  9--Road Side--[250, 170, 34],
  10--Lane Separator--[128, 128, 128],
  11--Temporary Barrier--[250, 170, 35],
  12--Wall--[102, 102, 156],
  13--Bike Lane--[128, 64, 255],
  14--Crosswalk - Plain--[140, 140, 200],
  15--Curb Cut--[170, 170, 170],
  16--Driveway--[250, 170, 36],
  17--Parking--[250, 170, 160],
  18--Parking Aisle--[250, 170, 37],
  19--Pedestrian Area--[96, 96, 96],
  20--Rail Track--[230, 150, 140],
  21--Road--[128, 64, 128],
  22--Road Shoulder--[110, 110, 110],
  23--Service Lane--[110, 110, 110],
  24--Sidewalk--[244, 35, 232],
  25--Traffic Island--[128, 196, 128],
  26--Bridge--[150, 100, 100],
  27--Building--[70, 70, 70],
  28--Garage--[150, 150, 150],

23--Service Lane--[110, 110, 110],
  29--Tunnel--[150, 120, 90],
  30--Person--[220, 20, 60],
  31--Person Group--[220, 20, 60],
  32--Bicyclist--[255, 0, 0],
  33--Motorcyclist--[255, 0, 100],
  34--Other Rider--[255, 0, 200],
  35--Lane Marking - Dashed Line--[255, 255, 255],
  36--Lane Marking - Straight Line--[255, 255, 255],
  37--Lane Marking - Zigzag Line--[250, 170, 29],
  38--Lane Marking - Ambiguous--[250, 170, 28],
  39--Lane Marking - Arrow (Left)--[250, 170, 26],
  40--Lane Marking - Arrow (Other)--[250, 170, 25],
  41--Lane Marking - Arrow (Right)--[250, 170, 24],
  42--Lane Marking - Arrow (Split Left or Straight)--[250, 170, 22],
  43--Lane Marking - Arrow (Split Right or Straight)--[250, 170, 21],
  44--Lane Marking - Arrow (Straight)--[250, 170, 20],
  45--Lane Marking - Crosswalk--[255, 255, 255],

43--Lane Marking - Arrow (Split Right or Straight)--[250, 170, 21],
  46--Lane Marking - Give Way (Row)--[250, 170, 19],
  47--Lane Marking - Give Way (Single)--[250, 170, 18],
  48--Lane Marking - Hatched (Chevron)--[250, 170, 12],
  49--Lane Marking - Hatched (Diagonal)--[250, 170, 11],
  50--Lane Marking - Other--[255, 255, 255],
  51--Lane Marking - Stop Line--[255, 255, 255],
  52--Lane Marking - Symbol (Bicycle)--[250, 170, 16],
  53--Lane Marking - Symbol (Other)--[250, 170, 15],
  54--Lane Marking - Text--[250, 170, 15],
  55--Lane Marking (only) - Dashed Line--[255, 255, 255],
  56--Lane Marking (only) - Crosswalk--[255, 255, 255],
  57--Lane Marking (only) - Other--[255, 255, 255],
  58--Lane Marking (only) - Test--[255, 255, 255],
  59--Mountain--[64, 170, 64],
  60--Sand--[230, 160, 50],
  61--Sky--[70, 130, 180],
  62--Snow--[190, 255, 255],

58--Lane Marking (only) - Test--[255, 255, 255],
  63--Terrain--[152, 251, 152],
  64--Vegetation--[107, 142, 35],
  65--Water--[0, 170, 30],
  66--Banner--[255, 255, 128],
  67--Bench--[250, 0, 30],
  68--Bike Rack--[100, 140, 180],
  69--Catch Basin--[220, 128, 128],
  70--CCTV Camera--[222, 40, 40],
  71--Fire Hydrant--[100, 170, 30],
  72--Junction Box--[40, 40, 40],
  73--Mailbox--[33, 33, 33],
  74--Manhole--[100, 128, 160],
  75--Parking Meter--[20, 20, 255],
  76--Phone Booth--[142, 0, 0],
  77--Pothole--[70, 100, 150],
  78--Signage - Advertisement--[250, 171, 30],
  79--Signage - Ambiguous--[250, 172, 30],
  80--Signage - Back--[250, 173, 30],
  81--Signage - Information--[250, 174, 30],
  82--Signage - Other--[250, 175, 30],
  83--Signage - Store--[250, 176, 30],
  84--Street Light--[210, 170, 100],
  85--Pole--[153, 153, 153],

81--Signage - Information--[250, 174, 30],
  86--Pole Group--[153, 153, 153],
  87--Traffic Sign Frame--[128, 128, 128],
  88--Utility Pole--[0, 0, 80],
  89--Traffic Cone--[210, 60, 60],
  90--Traffic Light - General (Single)--[250, 170, 30],
  91--Traffic Light - Pedestrians--[250, 170, 30],
  92--Traffic Light - General (Upright)--[250, 170, 30],
  93--Traffic Light - General (Horizontal)--[250, 170, 30],
  94--Traffic Light - Cyclists--[250, 170, 30],
  95--Traffic Light - Other--[250, 170, 30],
  96--Traffic Sign - Ambiguous--[192, 192, 192],
  97--Traffic Sign (Back)--[192, 192, 192],
  98--Traffic Sign - Direction (Back)--[192, 192, 192],
  99--Traffic Sign - Direction (Front)--[220, 220, 0],
  100--Traffic Sign (Front)--[220, 220, 0],
  101--Traffic Sign - Parking--[0, 0, 196],
  102--Traffic Sign - Temporary (Back)--[192, 192, 192],

99--Traffic Sign - Direction (Front)--[220, 220, 0],
  103--Traffic Sign - Temporary (Front)--[220, 220, 0],
  104--Trash Can--[140, 140, 20],
  105--Bicycle--[119, 11, 32],
  106--Boat--[150, 0, 255],
  107--Bus--[0, 60, 100],
  108--Car--[0, 0, 142],
  109--Caravan--[0, 0, 90],
  110--Motorcycle--[0, 0, 230],
  111--On Rails--[0, 80, 100],
  112--Other Vehicle--[128, 64, 64],
  113--Trailer--[0, 0, 110],
  114--Truck--[0, 0, 70],
  115--Vehicle Group--[0, 0, 142],
  116--Wheeled Slow--[0, 0, 192],
  117--Water Valve--[170, 170, 170],
  118--Car Mount--[32, 32, 32],
  119--Dynamic--[111, 74, 0],
  120--Ego Vehicle--[120, 10, 10],
  121--Ground--[81, 0, 81],
  122--Static--[111, 111, 0],
  123--Unlabeled--[0, 0, 0]
  ```

# SATRN

> [On Recognizing Texts of Arbitrary Shapes with 2D Self-Attention](https://arxiv.org/abs/1910.04396)

Scene text recognition (STR) is the task of recognizing character sequences in natural scenes. While there have been great advances in STR methods, current methods still fail to recognize texts in arbitrary shapes, such as heavily curved or rotated texts, which are abundant in daily life (e.g. restaurant signs, product labels, company logos, etc). This paper introduces a novel architecture to recognizing texts of arbitrary shapes, named Self-Attention Text Recognition Network (SATRN), which is inspired by the Transformer. SATRN utilizes the self-attention mechanism to describe two-dimensional (2D) spatial dependencies of characters in a scene text image. Exploiting the full-graph propagation of self-attention, SATRN can recognize texts with arbitrary arrangements and large inter-character spacing. As a result, SATRN outperforms existing STR models by a large margin of 5.7 pp on average in "irregular text" benchmarks. We provide empirical analyses that illustrate the inner mechanisms and the extent to which the model is applicable (e.g. rotated and multi-line text). We will open-source the code.

<img src="https://user-images.githubusercontent.com/22607038/142798828-cc4ded5d-3fb8-478c-9f3e-74edbcf41982.png"/>
</div>

## Dataset

|                                Methods                                 |        | Regular Text |           |     |           | Irregular Text |        |                                download                                 |

| :--------------------------------------------------------------------: | :----: | :----------: | :-------: | :-: | :-------: | :------------: | :----: | :---------------------------------------------------------------------: |
|                                                                        | IIIT5K |     SVT      | IC13-1015 |     | IC15-2077 |      SVTP      |  CT80  |                                                                         |
|      [Satrn](/configs/textrecog/satrn/satrn_shallow_5e_st_mj.py)       | 0.9600 |    0.9181    |  0.9606   |     |  0.8045   |     0.8837     | 0.8993 | [model](https://download.openmmlab.com/mmocr/textrecog/satrn/satrn_shallow_5e_st_mj/satrn_shallow_5e_st_mj_20220915_152443-5fd04a4c.pth) \| [log](https://download.openmmlab.com/mmocr/textrecog/satrn/satrn_shallow_5e_st_mj/20220915_152443.log) |

|    [Satrn-TTA](/configs/textrecog/satrn/satrn_shallow_5e_st_mj.py)     | 0.9530 |    0.9181    |  0.9527   |     |  0.8276   |     0.8884     | 0.9028 |                                                                         |
| [Satrn_small](/configs/textrecog/satrn/satrn_shallow-small_5e_st_mj.py) | 0.9423 |    0.9011    |  0.9567   |     |  0.7886   |     0.8574     | 0.8472 | [model](https://download.openmmlab.com/mmocr/textrecog/satrn/satrn_shallow-small_5e_st_mj/satrn_shallow-small_5e_st_mj_20220915_152442-5591bf27.pth) \| [log](https://download.openmmlab.com/mmocr/textrecog/satrn/satrn_shallow-small_5e_st_mj/20220915_152442.log) |
| [Satrn_small-TTA](/configs/textrecog/satrn/satrn_shallow-small_5e_st_mj.py) | 0.9380 |    0.8995    |  0.9488   |     |  0.8122   |     0.8620     | 0.8507 |                                                                         |

```bibtex
@article{junyeop2019recognizing,

```bibtex
  title={On Recognizing Texts of Arbitrary Shapes with 2D Self-Attention},
  author={Junyeop Lee, Sungrae Park, Jeonghun Baek, Seong Joon Oh, Seonghyeon Kim, Hwalsuk Lee},
}
```

# 参与贡献 OpenMMLab

欢迎任何类型的贡献，包括但不限于

- 添加新功能和新组件

## 工作流程

1. fork 并 pull 最新的 OpenMMLab 仓库 (MMPreTrain)
2. 签出到一个新分支（不要使用 master 分支提交 PR）
3. 进行修改并提交至 fork 出的自己的远程仓库
4. 在我们的仓库中创建一个 PR

```{note}
如果你计划添加一些新的功能，并引入大量改动，请尽量首先创建一个 issue 来进行讨论。
```

## 代码风格

### Python

我们采用 [PEP8](https://www.python.org/dev/peps/pep-0008/) 作为统一的代码风格。

我们使用下列工具来进行代码风格检查与格式化：

- [yapf](https://github.com/google/yapf): 一个 Python 文件的格式化工具。
- [docformatter](https://github.com/myint/docformatter): 一个 docstring 格式化工具。

yapf 和 isort 的格式设置位于 [setup.cfg](https://github.com/open-mmlab/mmpretrain/blob/main/setup.cfg)

我们使用 [pre-commit hook](https://pre-commit.com/) 来保证每次提交时自动进行代

码检查和格式化，启用的功能包括 `flake8`, `yapf`, `isort`, `trailing whitespaces`, `markdown files`, 修复 `end-of-files`, `double-quoted-strings`,
`python-encoding-pragma`, `mixed-line-ending`, 对 `requirments.txt`的排序等。
pre-commit hook 的配置文件位于 [.pre-commit-config](https://github.com/open-mmlab/mmpretrain/blob/main/.pre-commit-config.yaml)

在你克隆仓库后，你需要按照如下步骤安装并初始化 pre-commit hook。

```shell
```

在仓库文件夹中执行

```shell
```

在创建 PR 之前，请确保你的代码完成了代码规范检查，并经过了 yapf 的格式化。
```

> **Task**: Colorization

</div >

|                                              Model                                              | Dataset |                                              Download                                              |
| :---------------------------------------------------------------------------------------------: | :-----: | :------------------------------------------------------------------------------------------------: |
| [instance_aware_colorization_officiial](./inst-colorizatioon_full_official_cocostuff-256x256.py) | MS-COCO | [model](https://download.openmmlab.com/mmediting/inst_colorization/inst-colorizatioon_full_official_cocostuff-256x256-5b9d4eee.pth) |

<details>
<summary>Colorization demo</summary>

You can use the following commands to colorize an image.

```shell

<details>

```shell

```

For more demos, you can refer to [Tutorial 3: inference with pre-trained models](https://mmagic.readthedocs.io/en/latest/user_guides/3_inference.html).

<details>

```bibtex
}
```

# Learn About Config

OpenCompass uses the OpenMMLab modern style configuration files. If you are familiar with the OpenMMLab style
configuration files, you can directly refer to
[A Pure Python style Configuration File (Beta)](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#a-pure-python-style-configuration-file-beta)
to understand the differences between the new-style and original configuration files. If you have not
encountered OpenMMLab style configuration files before, I will explain the usage of configuration files using
a simple example. Make sure you have installed the latest version of MMEngine to support the
new-style configuration files.

## Basic Format

OpenCompass configuration files are in Python format, following basic Python syntax. Each configuration item
is specified by defining variables. For example, when defining a model, we use the following configuration:

```python

```python

models = [
    dict(
    )
]
```

When reading the configuration file, use `Config.fromfile` from MMEngine for parsing:

```python
```

## Inheritance Mechanism

```

OpenCompass configuration files use Python's import mechanism for file inheritance. Note that when inheriting
configuration files, we need to use the `read_base` context manager.

```python

    from .model_cfg import models  # Inherits the 'models' from model_cfg.py
```

Parse the configuration file using `Config.fromfile`:

```python
```

## Evaluation Configuration Example

```python

```

```python

# Concatenate the datasets to be evaluated into the datasets field

models = [
    dict(
        # Initialization parameters for `HuggingFaceCausalLM`

        # Common parameters for all models, not specific to HuggingFaceCausalLM's initialization parameters
        abbr='llama-7b',            # Model abbreviation for result display
        max_out_len=100,            # Maximum number of generated tokens
        batch_size=16,
        run_cfg=dict(num_gpus=1),   # Run configuration for specifying resource requirements
    )
]
```

## Dataset Configuration File Example

In the above example configuration file, we directly inherit the dataset-related configurations. Next, we will
use the PIQA dataset configuration file as an example to demonstrate the meanings of each field in the dataset
configuration file. If you do not intend to modify the prompt for model testing or add new datasets, you can
skip this section.

The PIQA dataset [configuration file](https://github.com/InternLM/opencompass/blob/main/configs/datasets/piqa/piqa_ppl_1cf9f0.py) is as follows.
It is a configuration for evaluating based on perplexity (PPL) and does not use In-Context Learning.

```python

# Reading configurations
# The loaded dataset is usually organized as dictionaries, specifying the input fields used to form the prompt
# and the output field used as the answer in each sample
)

)

# Inference configurations
    # Prompt generation configuration
        # Prompt template, the template format matches the inferencer type specified later
        # Here, to calculate PPL, we need to specify the prompt template for each answer
        }),
    # In-Context example configuration, specifying `ZeroRetriever` here, which means not using in-context example.
    # Inference method configuration
    #   - PPLInferencer uses perplexity (PPL) to obtain answers

    #   - GenInferencer uses the model's generated results to obtain answers

# Metric configuration, using Accuracy as the evaluation metric

# Dataset configuration, where all the above variables are parameters for this configuration
# It is a list used to specify the configurations of different evaluation subsets of a dataset.
    dict(
```

For detailed configuration of the **Prompt generation configuration**, you can refer to the [Prompt Template](../prompt/prompt_template.md).

## Advanced Evaluation Configuration

```

In OpenCompass, we support configuration options such as task partitioner and runner for more flexible and
efficient utilization of computational resources.

By default, we use size-based partitioning for inference tasks. You can specify the sample number threshold
for task partitioning using `--max-partition-size` when starting the task. Additionally, we use local
resources for inference and evaluation tasks by default. If you want to use Slurm cluster resources, you can
use the `--slurm` parameter and the `--partition` parameter to specify the Slurm runner backend when starting
the task.

Furthermore, if the above functionalities do not meet your requirements for task partitioning and runner
backend configuration, you can provide more detailed configurations in the configuration file. Please refer to

[Efficient Evaluation](./evaluation.md) for more information.

# 准备 Multi-Moments in Time

## 简介

```BibTeX
}
```

在准备数据集前，请确保命令行当前路径为 `$MMACTION2/tools/data/mmit/`。

## 步骤 1. Prepare Annotations and Videos

```
```

```
```

```shell
```

```shell
```

```shell
```

```shell
```

```shell
```

```shell
```

```shell
```

在完成 Multi-Moments in Time 数据集准备流程后，用户可以得到 Multi-Moments in Time 的 RGB 帧 + 光流文件，视频文件以及标注文件。

在整个 MMAction2 文件夹下，Multi-Moments in Time 的文件结构如下：

```
└── data
    └── mmit
        │   │   ├── ...
        │   │   ├── ...
        │   │   ├── ...

│   │   ├── flow_y_00001.jpg
        │   │   ├── ...
        │   │   ├── ...
        │   │   ├── ...
        │   └── ...

        └── videos
            └── ...
```

关于对 Multi-Moments in Time 进行训练和验证，可以参照 [训练教程](/docs/zh_cn/user_guides/train_test.md)。

[provided templates](https://github.com/open-mmlab/mmpretrain/issues/new/choose)

- Compatibility issue between MMEngine, MMCV and MMPretrain

  Compatible MMPretrain and MMEngine, MMCV versions are shown as below. Please
  choose the correct version of MMEngine and MMCV to avoid installation issues.

  | MMPretrain version | MMEngine version  |   MMCV version   |
  | :----------------: | :---------------: | :--------------: |
  |    1.0.1 (main)    | mmengine >= 0.8.3 |  mmcv >= 2.0.0   |
  |       1.0.0        | mmengine >= 0.8.0 |  mmcv >= 2.0.0   |

| :----------------: | :---------------: | :--------------: |
  |      1.0.0rc8      | mmengine >= 0.7.1 | mmcv >= 2.0.0rc4 |
  |      1.0.0rc7      | mmengine >= 0.5.0 | mmcv >= 2.0.0rc4 |

  ```{note}
  Since the `dev` branch is under frequent development, the MMEngine and MMCV
  version dependency may be inaccurate. If you encounter problems when using
  the `dev` branch, please try to update MMEngine and MMCV to the latest version.
  ```

  If you simply use `pip install albumentations>=0.3.2`, it will install `opencv-python-headless` simultaneously
  (even though you have already installed `opencv-python`). Please refer to the

If you simply use `pip install albumentations>=0.3.2`, it will install `opencv-python-headless` simultaneously
  [official documentation](https://albumentations.ai/docs/getting_started/installation/#note-on-opencv-dependencies)
  for details.

## General Questions

### Do I need to reinstall mmpretrain after some code modifications?

If you follow [the best practice](../get_started.md#best-practices) and install mmpretrain from source,
any local modifications made to the code will take effect without
reinstallation.

### How to develop with multiple MMPretrain versions?

Generally speaking, we recommend to use different virtual environments to
manage MMPretrain in different working directories. However, you
can also use the same environment to develop MMPretrain in different
folders, like mmpretrain-0.21, mmpretrain-0.23. When you run the train or test shell script,

it will adopt the mmpretrain package in the current folder. And when you run other Python
script, you can also add `` PYTHONPATH=`pwd`  `` at the beginning of your command
to use the package in the current folder.

Conversely, to use the default MMPretrain installed in the environment
rather than the one you are working with, you can remove the following line
in those shell scripts:

```shell
```

### What's the relationship between the `load_from` and the `init_cfg`?

- `load_from`: If `resume=False`, only imports model weights, which is mainly used to load trained models;
  If `resume=True`, load all of the model weights, optimizer state, and other training information, which is
  mainly used to resume interrupted training.

If `resume=True`, load all of the model weights, optimizer state, and other training information, which is

- `init_cfg`: You can also specify `init=dict(type="Pretrained", checkpoint=xxx)` to load checkpoint, it
  means load the weights during model weights initialization. That is, it will be only done at the
  beginning of the training. It's mainly used to fine-tune a pre-trained model, and you can set it in
  the backbone config and use `prefix` field to only load backbone weights, for example:

```python
      type='ResNet',
      depth=50,
      init_cfg=dict(type='Pretrained', checkpoints=xxx, prefix='backbone'),
  )
  ...
)
```

See the [Fine-tune Models](./finetune_custom_dataset.md) for more details about fine-tuning.

### What's the difference between `default_hooks` and `custom_hooks`?

)
  ...
)
```

Almost no difference. Usually, the `default_hooks` field is used to specify the hooks that will be used in almost
all experiments, and the `custom_hooks` field is used in only some experiments.

Another difference is the `default_hooks` is a dict while the `custom_hooks` is a list, please don't be
confused.

### During training, I got no training log, what's the reason?

If your training dataset is small while the batch size is large, our default log interval may be too large to
record your training log.

You can shrink the log interval and try again, like:

```python
    ...
    ...
)
```

### How to train with other datasets, like my own dataset or COCO?

```python
    ...
    ...
)
```

We provide [specific examples](./pretrain_custom_dataset.md) to show how to train with other datasets.

# Bottom-up Human Pose Estimation via Disentangled Keypoint Regression

<details>

```bibtex
}
```

In this paper, we are interested in the bottom-up paradigm of estimating human poses from an image. We study the dense keypoint regression framework that is previously inferior to the keypoint detection and grouping framework. Our motivation is that regressing keypoint positions accurately needs to learn representations that focus on the keypoint regions.

We present a simple yet effective approach, named disentangled keypoint regression (DEKR). We adopt adaptive convolutions through pixel-wise spatial transformer to activate the pixels in the keypoint regions and accordingly learn representations from them. We use a multi-branch structure for separate regression: each branch learns a representation with dedicated adaptive convolutions and regresses one keypoint. The resulting disentangled representations are able to attend to the keypoint regions, respectively, and thus the keypoint regression is spatially more accurate. We empirically show that the proposed direct regression method outperforms keypoint detection and grouping methods and achieves superior bottom-up pose estimation results on two benchmark datasets, COCO and CrowdPose. The code and models are available at [this https URL](https://github.com/HRNet/DEKR).

<img src="https://user-images.githubusercontent.com/26127467/192512411-1f8e66c1-4ba2-4db6-a66a-31c0f7c13882.png">
</div>

## <a href='https://mmdetection.readthedocs.io/en/latest/'>English</a>

## <a href='https://mmdetection.readthedocs.io/zh_CN/latest/'>简体中文</a>

## 形状偏差(Shape Bias)工具用法

形状偏差(shape bias)衡量模型与纹理相比，如何依赖形状来感知图像中的语义。关于更多细节，我们向感兴趣的读者推荐这篇[论文](https://arxiv.org/abs/2106.07411) 。MMPretrain提供现成的工具箱来获得分类模型的形状偏差。您可以按照以下步骤操作：

### 准备数据集

首先你应该下载[cue-conflict](https://github.com/bethgelab/model-vs-human/releases/download/v0.1/cue-conflict.tar.gz) 到`data`文件夹，然后解压缩这个数据集。之后，你的`data`文件夹应具有一下结构：

```text
data
|      |──bear
|      ...
|      |── truck
```

### 修改分类配置

我们在使用MAE预训练的ViT-base模型上运行形状偏移工具。它的配置文件为`configs/mae/benchmarks/vit-base-p16_8xb128-coslr-100e_in1k.py`，它的检查点可从[此链接](https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-1600e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k/vit-base-p16_ft-8xb128-coslr-100e_in1k_20220825-cf70aa21.pth) 下载。将原始配置中的test_pipeline, test_dataloader和test_evaluation替换为以下配置：

```python
    dict(

```python
    dict(
]
```

请注意，你应该对上面的`csv_dir`和`model_name`进行自定义修改。我把修改后的示例配置文件重命名为`configs/mae/benchmarks/`文件夹中的`vit-base-p16_8xb128-coslr-100e_in1k_shape-bias.py`文件。

### 用上面修改后的配置文件在你的模型上做推断

然后，你应该使用修改后的配置文件在`cue-conflict`数据集上推断你的模型。

```shell

```shell
```

- `$CONFIG`: 修改后的配置文件的路径。
- `$CHECKPOINT`: 检查点文件的路径或链接。

```shell
# Example
```

之后，你应该在`csv_dir`文件夹中获得一个名为`cue-conflict_model-name_session-1.csv`的csv文件。除了这个文件以外，你还应该下载这些[csv文件](https://github.com/bethgelab/model-vs-human/tree/master/raw-data/cue-conflict) 到`csv_dir`。

### 绘制形状偏差图

然后我们可以开始绘制形状偏差图：

```shell
```

**所有参数的说明**:

- `--csv-dir $CSV_DIR`, 与保存这些csv文件的目录相同。

```

- `--result-dir $RESULT_DIR`, 输出名为`cue-conflict_shape-bias_matrixplot.pdf`的结果的目录。
- `--colors $RGB`, 应该是RGB值，格式为R G B，例如100 100 100，如果你想绘制几个模型的形状偏差，可以是多个RGB值。
- `--plotting-names $YOUR_MODEL_NAME`, 形状偏移图中图例的名称，您可以将其设置为模型名称。如果要绘制多个模型，plotting_names可以是多个值。
- `model-names $YOUR_MODEL_NAME`, 应与配置中指定的名称相同，如果要绘制多个模型的形状偏差，则可以是多个名称。

请注意，`--colors`的每三个值对应于`--model-names`的一个值。完成以上所有步骤后，你将获得下图。

</div>

# Supported ncnn feature

The current use of the ncnn feature is as follows:

The following features cannot be automatically enabled by mmdeploy and you need to manually modify the ncnn build options or adjust the running parameters in the SDK

- Profiling per layer
- Turn off NCNN_STRING to reduce .so file size
- Set thread number and CPU affinity

# AGCN

[Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition](https://openaccess.thecvf.com/content_CVPR_2019/html/Shi_Two-Stream_Adaptive_Graph_Convolutional_Networks_for_Skeleton-Based_Action_Recognition_CVPR_2019_paper.html)

In skeleton-based action recognition, graph convolutional networks (GCNs), which model the human body skeletons as spatiotemporal graphs, have achieved remarkable performance. However, in existing GCN-based methods, the topology of the graph is set manually, and it is fixed over all layers and input samples. This may not be optimal for the hierarchical GCN and diverse samples in action recognition tasks. In addition, the second-order information (the lengths and directions of bones) of the skeleton data, which is naturally more informative and discriminative for action recognition, is rarely investigated in existing methods. In this work, we propose a novel two-stream adaptive graph convolutional network (2s-AGCN) for skeleton-based action recognition. The topology of the graph in our model can be either uniformly or individually learned by the BP algorithm in an end-to-end manner. This data-driven method increases the flexibility of the model for graph construction and brings more generality to adapt to various data samples. Moreover, a two-stream framework is proposed to model both the first-order and the second-order information simultaneously, which shows notable improvement for the recognition accuracy. Extensive experiments on the two large-scale datasets, NTU-RGBD and Kinetics-Skeleton, demonstrate that the performance of our model exceeds the state-of-the-art with a significant margin.

<img src="https://user-images.githubusercontent.com/30782254/143212681-a676d7a0-e92b-4a8a-ad8c-c5826eb58019.png" width="800"/>
</div>

|       uniform 100       |    joint     |  8   |   AGCN   |  88.60   |     10 clips     | 4.4G  |  3.5M  | [config](/configs/skeleton/2s-agcn/2s-agcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/2s-agcn/2s-agcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d/2s-agcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d_20221222-4c0ed77e.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/2s-agcn/2s-agcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d/2s-agcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.log) |

|       uniform 100       |     bone     |  8   |   AGCN   |  91.59   |     10 clips     | 4.4G  |  3.5M  | [config](/configs/skeleton/2s-agcn/2s-agcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-2d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/2s-agcn/2s-agcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-2d/2s-agcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-2d_20221222-293878b5.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/2s-agcn/2s-agcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-2d/2s-agcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-2d.log) |

|       uniform 100       | joint-motion |  8   |   AGCN   |  88.02   |     10 clips     | 4.4G  |  3.5M  | [config](/configs/skeleton/2s-agcn/2s-agcn_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-2d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/2s-agcn/2s-agcn_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-2d/2s-agcn_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-2d_20221222-0c86e3a1.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/2s-agcn/2s-agcn_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-2d/2s-agcn_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-2d.log) |

|       uniform 100       | bone-motion  |  8   |   AGCN   |  88.82   |     10 clips     | 4.4G  |  3.5M  | [config](/configs/skeleton/2s-agcn/2s-agcn_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-2d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/2s-agcn/2s-agcn_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-2d/2s-agcn_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-2d_20221222-87996f0d.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/2s-agcn/2s-agcn_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-2d/2s-agcn_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-2d.log) |
|                         |  two-stream  |      |          |  91.95   |                  |       |        |                                           |                                         |                                        |

|       uniform 100       |    joint     |  8   |   AGCN   |  88.26   |     10 clips     | 6.5G  |  3.5M  | [config](/configs/skeleton/2s-agcn/2s-agcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/2s-agcn/2s-agcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d/2s-agcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d_20221222-24dabf78.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/2s-agcn/2s-agcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d/2s-agcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-3d.log) |

|       uniform 100       |     bone     |  8   |   AGCN   |  89.22   |     10 clips     | 6.5G  |  3.5M  | [config](/configs/skeleton/2s-agcn/2s-agcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-3d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/2s-agcn/2s-agcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-3d/2s-agcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-3d_20221222-abe70a7f.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/2s-agcn/2s-agcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-3d/2s-agcn_8xb16-bone-u100-80e_ntu60-xsub-keypoint-3d.log) |

|       uniform 100       | joint-motion |  8   |   AGCN   |  86.73   |     10 clips     | 6.5G  |  3.5M  | [config](/configs/skeleton/2s-agcn/2s-agcn_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-3d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/2s-agcn/2s-agcn_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-3d/2s-agcn_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-3d_20221222-923cd3c3.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/2s-agcn/2s-agcn_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-3d/2s-agcn_8xb16-joint-motion-u100-80e_ntu60-xsub-keypoint-3d.log) |

|       uniform 100       | bone-motion  |  8   |   AGCN   |  86.41   |     10 clips     | 6.5G  |  3.5M  | [config](/configs/skeleton/2s-agcn/2s-agcn_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-3d.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/skeleton/2s-agcn/2s-agcn_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-3d/2s-agcn_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-3d_20221222-3d8f6f43.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/skeleton/2s-agcn/2s-agcn_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-3d/2s-agcn_8xb16-bone-motion-u100-80e_ntu60-xsub-keypoint-3d.log) |
|                         |  two-stream  |      |          |  90.27   |                  |       |        |                                           |                                         |                                        |

|                         | four-stream  |      |          |  90.89   |                  |       |        |                                           |                                         |                                        |

## Train

```shell
```

## Train

```shell
```

```shell
python tools/train.py configs/skeleton/2s-agcn/2s-agcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py \
```

## Test

```shell
```

Example: test AGCN model on NTU60-2D dataset and dump the result to a pickle file.

```shell
python tools/test.py configs/skeleton/2s-agcn/2s-agcn_8xb16-joint-u100-80e_ntu60-xsub-keypoint-2d.py \
```

```shell
```

```BibTeX
@inproceedings{shi2019two,
  title={Two-stream adaptive graph convolutional networks for skeleton-based action recognition},
  author={Shi, Lei and Zhang, Yifan and Cheng, Jian and Lu, Hanqing},
  pages={12026--12035},
}
```

# ResNet

> [Deep Residual Learning for Image Recognition](https://openaccess.thecvf.com/content_cvpr_2016/html/He_Deep_Residual_Learning_CVPR_2016_paper.html)

**Residual Networks**, or **ResNets**, learn residual functions with reference to the layer inputs, instead of
learning unreferenced functions. In the mainstream previous works, like VGG, the neural networks are a stack
of layers and every layer attempts to fit a desired underlying mapping. In ResNets, a few stacked layers are
grouped as a block, and the layers in a block attempts to learn a residual mapping.

Formally, denoting the desired underlying mapping of a block as $\mathcal{H}(x)$, split the underlying mapping
into the sum of the identity and the residual mapping as $\mathcal{H}(x) = x + \mathcal{F}(x)$, and let the
stacked non-linear layers fit the residual mapping $\mathcal{F}(x)$.

Many works proved this method makes deep neural networks easier to optimize, and can gain accuracy from
considerably increased depth. Recently, the residual structure is widely used in various models.

<img src="https://user-images.githubusercontent.com/26739999/142574068-60cfdeea-c4ec-4c49-abb2-5dc2facafc3b.png" width="40%"/>
</div>

<details>

<br>

</div>

<details>

<br>
Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers---8x deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers.

The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.
</br>

```python

predict = inference_model('resnet18_8xb16_cifar10', 'demo/bird.JPEG')
```

```python

model = get_model('resnet18_8xb16_cifar10', pretrained=True)
```

```

Train:

```shell
python tools/train.py configs/resnet/resnet18_8xb16_cifar10.py
```

Test:

```shell
python tools/test.py configs/resnet/resnet18_8xb16_cifar10.py https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_b16x8_cifar10_20210528-bd6371c8.pth
```

| Model                              |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                    Config                     |                                 Download                                 |

| :--------------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :-------------------------------------------: | :----------------------------------------------------------------------: |
| `resnet18_8xb32_in1k`              | From scratch |   11.69    |   1.82    |   69.90   |   89.43   |       [config](resnet18_8xb32_in1k.py)        | [model](https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_8xb32_in1k_20210831-fbbb1da6.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_8xb32_in1k_20210831-fbbb1da6.json) |
| `resnet34_8xb32_in1k`              | From scratch |    2.18    |   3.68    |   73.62   |   91.59   |       [config](resnet34_8xb32_in1k.py)        | [model](https://download.openmmlab.com/mmclassification/v0/resnet/resnet34_8xb32_in1k_20210831-f257d4e6.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnet/resnet34_8xb32_in1k_20210831-f257d4e6.json) |

| `resnet50_8xb32_in1k`              | From scratch |   25.56    |   4.12    |   76.55   |   93.06   |       [config](resnet50_8xb32_in1k.py)        | [model](https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_8xb32_in1k_20210831-ea4938fc.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_8xb32_in1k_20210831-ea4938fc.json) |
| `resnet101_8xb32_in1k`             | From scratch |   44.55    |   7.85    |   77.97   |   94.06   |       [config](resnet101_8xb32_in1k.py)       | [model](https://download.openmmlab.com/mmclassification/v0/resnet/resnet101_8xb32_in1k_20210831-539c63f8.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnet/resnet101_8xb32_in1k_20210831-539c63f8.json) |

| `resnet152_8xb32_in1k`             | From scratch |   60.19    |   11.58   |   78.48   |   94.13   |       [config](resnet152_8xb32_in1k.py)       | [model](https://download.openmmlab.com/mmclassification/v0/resnet/resnet152_8xb32_in1k_20210901-4d7582fa.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnet/resnet152_8xb32_in1k_20210901-4d7582fa.json) |
| `resnetv1d50_8xb32_in1k`           | From scratch |   25.58    |   4.36    |   77.54   |   93.57   |      [config](resnetv1d50_8xb32_in1k.py)      | [model](https://download.openmmlab.com/mmclassification/v0/resnet/resnetv1d50_b32x8_imagenet_20210531-db14775a.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnet/resnetv1d50_b32x8_imagenet_20210531-db14775a.json) |

| `resnetv1d101_8xb32_in1k`          | From scratch |   44.57    |   8.09    |   78.93   |   94.48   |     [config](resnetv1d101_8xb32_in1k.py)      | [model](https://download.openmmlab.com/mmclassification/v0/resnet/resnetv1d101_b32x8_imagenet_20210531-6e13bcd3.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnet/resnetv1d101_b32x8_imagenet_20210531-6e13bcd3.json) |
| `resnetv1d152_8xb32_in1k`          | From scratch |   60.21    |   11.82   |   79.41   |   94.70   |     [config](resnetv1d152_8xb32_in1k.py)      | [model](https://download.openmmlab.com/mmclassification/v0/resnet/resnetv1d152_b32x8_imagenet_20210531-278cf22a.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnet/resnetv1d152_b32x8_imagenet_20210531-278cf22a.json) |

| `resnet50_8xb32-fp16_in1k`         | From scratch |   25.56    |   4.12    |   76.30   |   93.07   |     [config](resnet50_8xb32-fp16_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/fp16/resnet50_batch256_fp16_imagenet_20210320-b3964210.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/fp16/resnet50_batch256_fp16_imagenet_20210320-b3964210.json) |
| `resnet50_8xb256-rsb-a1-600e_in1k` | From scratch |   25.56    |   4.12    |   80.12   |   94.78   | [config](resnet50_8xb256-rsb-a1-600e_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_8xb256-rsb-a1-600e_in1k_20211228-20e21305.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_8xb256-rsb-a1-600e_in1k_20211228-20e21305.json) |

| `resnet50_8xb256-rsb-a2-300e_in1k` | From scratch |   25.56    |   4.12    |   79.55   |   94.37   | [config](resnet50_8xb256-rsb-a2-300e_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_8xb256-rsb-a2-300e_in1k_20211228-0fd8be6e.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_8xb256-rsb-a2-300e_in1k_20211228-0fd8be6e.json) |
| `resnet50_8xb256-rsb-a3-100e_in1k` | From scratch |   25.56    |   4.12    |   78.30   |   93.80   | [config](resnet50_8xb256-rsb-a3-100e_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_8xb256-rsb-a3-100e_in1k_20211228-3493673c.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_8xb256-rsb-a3-100e_in1k_20211228-3493673c.json) |

| `resnetv1c50_8xb32_in1k`           | From scratch |   25.58    |   4.36    |   77.01   |   93.58   |      [config](resnetv1c50_8xb32_in1k.py)      | [model](https://download.openmmlab.com/mmclassification/v0/resnet/resnetv1c50_8xb32_in1k_20220214-3343eccd.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnet/resnetv1c50_8xb32_in1k_20220214-3343eccd.json) |
| `resnetv1c101_8xb32_in1k`          | From scratch |   44.57    |   8.09    |   78.30   |   94.27   |     [config](resnetv1c101_8xb32_in1k.py)      | [model](https://download.openmmlab.com/mmclassification/v0/resnet/resnetv1c101_8xb32_in1k_20220214-434fe45f.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnet/resnetv1c101_8xb32_in1k_20220214-434fe45f.json) |

| `resnetv1c152_8xb32_in1k`          | From scratch |   60.21    |   11.82   |   78.76   |   94.41   |     [config](resnetv1c152_8xb32_in1k.py)      | [model](https://download.openmmlab.com/mmclassification/v0/resnet/resnetv1c152_8xb32_in1k_20220214-c013291f.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnet/resnetv1c152_8xb32_in1k_20220214-c013291f.json) |

### Image Classification on CIFAR-10

| Model                     |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) |                Config                |                                              Download                                               |
| :------------------------ | :----------: | :--------: | :-------: | :-------: | :----------------------------------: | :-------------------------------------------------------------------------------------------------: |

| `resnet18_8xb16_cifar10`  | From scratch |   11.17    |   0.56    |   94.82   | [config](resnet18_8xb16_cifar10.py)  | [model](https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_b16x8_cifar10_20210528-bd6371c8.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_b16x8_cifar10_20210528-bd6371c8.json) |
| `resnet34_8xb16_cifar10`  | From scratch |   21.28    |   1.16    |   95.34   | [config](resnet34_8xb16_cifar10.py)  | [model](https://download.openmmlab.com/mmclassification/v0/resnet/resnet34_b16x8_cifar10_20210528-a8aa36a6.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnet/resnet34_b16x8_cifar10_20210528-a8aa36a6.json) |

| `resnet50_8xb16_cifar10`  | From scratch |   23.52    |   1.31    |   95.55   | [config](resnet50_8xb16_cifar10.py)  | [model](https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_b16x8_cifar10_20210528-f54bfad9.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_b16x8_cifar10_20210528-f54bfad9.json) |
| `resnet101_8xb16_cifar10` | From scratch |   42.51    |   2.52    |   95.58   | [config](resnet101_8xb16_cifar10.py) | [model](https://download.openmmlab.com/mmclassification/v0/resnet/resnet101_b16x8_cifar10_20210528-2d29e936.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnet/resnet101_b16x8_cifar10_20210528-2d29e936.json) |

| `resnet152_8xb16_cifar10` | From scratch |   58.16    |   3.74    |   95.76   | [config](resnet152_8xb16_cifar10.py) | [model](https://download.openmmlab.com/mmclassification/v0/resnet/resnet152_b16x8_cifar10_20210528-3e8e9178.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnet/resnet152_b16x8_cifar10_20210528-3e8e9178.json) |

### Image Classification on CIFAR-100

| `resnet50_8xb16_cifar100` | From scratch |   23.71    |   1.31    |   79.90   |   95.19   | [config](resnet50_8xb16_cifar100.py) | [model](https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_b16x8_cifar100_20210528-67b58a1b.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_b16x8_cifar100_20210528-67b58a1b.json) |

### Image Classification on CUB-200-2011

| Model               |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) |             Config             |                                                    Download                                                     |
| :------------------ | :----------: | :--------: | :-------: | :-------: | :----------------------------: | :-------------------------------------------------------------------------------------------------------------: |

| `resnet50_8xb8_cub` | From scratch |   23.92    |   16.48   |   88.45   | [config](resnet50_8xb8_cub.py) | [model](https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_8xb8_cub_20220307-57840e60.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_8xb8_cub_20220307-57840e60.json) |

```bibtex
}
```

    <sup>
      </a>
    </sup>
    <sup>
      </a>
    </sup>
  </div>

[![docs](https://img.shields.io/badge/docs-latest-blue)](https://mmsegmentation.readthedocs.io/zh_CN/latest/)

文档: <https://mmsegmentation.readthedocs.io/zh_CN/latest>

</div>

</div>

</div>

## 简介

MMSegmentation 是一个基于 PyTorch 的语义分割开源工具箱。它是 OpenMMLab 项目的一部分。

[main](https://github.com/open-mmlab/mmsegmentation/tree/main) 分支代码目前支持 PyTorch 1.6 以上的版本。

### 🎉 MMSegmentation v1.0.0 简介 🎉

## 简介

我们非常高兴地宣布 MMSegmentation 最新版本的正式发布！在这个新版本中，主要分支是 [main](https://github.com/open-mmlab/mmsegmentation/tree/main) 分支，开发分支是 [dev-1.x](https://github.com/open-mmlab/mmsegmentation/tree/dev-1.x)。而之前版本的稳定分支保留为 [0.x](https://github.com/open-mmlab/mmsegmentation/tree/0.x) 分支。请注意，[master](https://github.com/open-mmlab/mmsegmentation/tree/master) 分支将只在有限的时间内维护，然后将被删除。我们鼓励您在使用过程中注意分支选择和更新。感谢您一如既往的支持和热情，让我们共同努力，使 MMSegmentation 变得更加健壮和强大！💪

MMSegmentation v1.x 在 0.x 版本的基础上有了显著的提升，提供了更加灵活和功能丰富的体验。为了更好使用 v1.x 中的新功能，我们诚挚邀请您查阅我们详细的 [📚 迁移指南](https://mmsegmentation.readthedocs.io/zh_CN/latest/migration/interface.html)，以帮助您无缝地过渡您的项目。您的支持对我们来说非常宝贵，我们热切期待您的反馈！

![示例图片](resources/seg_demo.gif)

### 主要特性

- **统一的基准平台**

  我们将各种各样的语义分割算法集成到了一个统一的工具箱，进行基准测试。

### 主要特性

  MMSegmentation 将分割框架解耦成不同的模块组件，通过组合不同的模块组件，用户可以便捷地构建自定义的分割模型。

- **丰富的即插即用的算法和模型**

  MMSegmentation 支持了众多主流的和最新的检测算法，例如 PSPNet，DeepLabV3，PSANet，DeepLabV3+ 等.

- **速度快**

  训练速度比其他语义分割代码库更快或者相当。

## 更新日志

最新版本 v1.1.1 在 2023.07.24 发布。
如果想了解更多版本更新细节和历史信息，请阅读[更新日志](docs/en/notes/changelog.md)。

## 安装

请参考[快速入门文档](docs/zh_cn/get_started.md#installation)进行安装，参考[数据集准备](docs/zh_cn/user_guides/2_dataset_prepare.md)处理数据。

## 快速入门

请参考[概述](docs/zh_cn/overview.md)对 MMSegmetation 进行初步了解

请参考[用户指南](https://mmsegmentation.readthedocs.io/zh_CN/latest/user_guides/index.html)了解 mmseg 的基本使用，以及[进阶指南](https://mmsegmentation.readthedocs.io/zh_CN/latest/advanced_guides/index.html)深入了解 mmseg 设计和代码实现。

同时，我们提供了 Colab 教程。你可以在[这里](demo/MMSegmentation_Tutorial.ipynb)浏览教程，或者直接在 Colab 上[运行](https://colab.research.google.com/github/open-mmlab/mmsegmentation/blob/main/demo/MMSegmentation_Tutorial.ipynb)。

若需要将 0.x 版本的代码迁移至新版，请参考[迁移文档](docs/zh_cn/migration)。

## 教程文档

<details>
<summary>开启 MMSeg 之旅</summary>

- [MMSeg 概述](docs/zh_cn/overview.md)
- [安装和验证](docs/zh_cn/get_started.md)
- [常见问题解答](docs/zh_cn/notes/faq.md)

<details>
<summary>MMSeg 快速入门教程</summary>

- [教程1：了解配置文件](docs/zh_cn/user_guides/1_config.md)
- [教程2：准备数据集](docs/zh_cn/user_guides/2_dataset_prepare.md)
- [教程3：使用预训练模型推理](docs/zh_cn/user_guides/3_inference.md)
- [教程4：使用现有模型进行训练和测试](docs/zh_cn/user_guides/4_train_test.md)
- [教程5：模型部署](docs/zh_cn/user_guides/5_deployment.md)
- [常用工具](docs/zh_cn/user_guides/useful_tools.md)
- [特征图可视化](docs/zh_cn/user_guides/visualization_feature_map.md)
- [可视化](docs/zh_cn/user_guides/visualization.md)

<details>
<summary>MMSeg 细节介绍</summary>

<details>

- [MMSeg 数据集介绍](docs/zh_cn/advanced_guides/datasets.md)
- [MMSeg 模型介绍](docs/zh_cn/advanced_guides/models.md)
- [MMSeg 数据结构介绍](docs/zh_cn/advanced_guides/structures.md)
- [MMSeg 数据增强介绍](docs/zh_cn/advanced_guides/transforms.md)
- [MMSeg 数据流介绍](docs/zh_cn/advanced_guides/data_flow.md)
- [MMSeg 训练引擎介绍](docs/zh_cn/advanced_guides/engine.md)
- [MMSeg 模型评测介绍](docs/zh_cn/advanced_guides/evaluation.md)

<details>
<summary>MMSeg 开发教程</summary>

- [新增自定义数据集](docs/zh_cn/advanced_guides/add_datasets.md)
- [新增评测指标](docs/zh_cn/advanced_guides/add_metrics.md)
- [新增自定义模型](docs/zh_cn/advanced_guides/add_models.md)
- [新增自定义数据增强](docs/zh_cn/advanced_guides/add_transforms.md)
- [自定义运行设定](docs/zh_cn/advanced_guides/customize_runtime.md)
- [训练技巧](docs/zh_cn/advanced_guides/training_tricks.md)

- [如何给 MMSeg贡献代码](.github/CONTRIBUTING.md)
- [在 projects 给 MMSeg 贡献一个标准数据集](docs/zh_cn/advanced_guides/contribute_dataset.md)
- [NPU (华为 昇腾)](docs/zh_cn/device/npu.md)
- [0.x → 1.x 迁移文档](docs/zh_cn/migration/interface.md)，[0.x → 1.x 库变更文档](docs/zh_cn/migration/package.md)

<summary>已支持的骨干网络：</summary>

<summary>已支持的算法：</summary>

<summary>已支持的数据集：</summary>

- [x] [Cityscapes](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/zh_cn/user_guides/2_dataset_prepare.md#cityscapes)
- [x] [PASCAL VOC](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/zh_cn/user_guides/2_dataset_prepare.md#pascal-voc)
- [x] [ADE20K](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/zh_cn/user_guides/2_dataset_prepare.md#ade20k)

- [x] [Pascal Context](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/zh_cn/user_guides/2_dataset_prepare.md#pascal-context)
- [x] [COCO-Stuff 10k](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/zh_cn/user_guides/2_dataset_prepare.md#coco-stuff-10k)
- [x] [COCO-Stuff 164k](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/zh_cn/user_guides/2_dataset_prepare.md#coco-stuff-164k)
- [x] [CHASE_DB1](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/zh_cn/user_guides/2_dataset_prepare.md#chase-db1)
- [x] [DRIVE](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/zh_cn/user_guides/2_dataset_prepare.md#drive)
- [x] [HRF](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/zh_cn/user_guides/2_dataset_prepare.md#hrf)

- [x] [Vaihingen](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/zh_cn/user_guides/2_dataset_prepare.md#isprs-vaihingen)
- [x] [iSAID](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/zh_cn/user_guides/2_dataset_prepare.md#isaid)
- [x] [Mapillary Vistas](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/zh_cn/user_guides/2_dataset_prepare.md#mapillary-vistas-datasets)
- [x] [LEVIR-CD](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/zh_cn/user_guides/2_dataset_prepare.md#levir-cd)
- [x] [BDD100K](https://github.com/open-mmlab/mmsegmentation/blob/main/docs/zh_cn/user_guides/2_dataset_prepare.md#bdd100K)

如果遇到问题，请参考 [常见问题解答](docs/zh_cn/notes/faq.md)。

## 社区项目

[这里](projects/README.md)有一些由社区用户支持和维护的基于 MMSegmentation 的 SOTA 模型和解决方案的实现。这些项目展示了基于 MMSegmentation 的研究和产品开发的最佳实践。

## 贡献指南

我们感谢所有的贡献者为改进和提升 MMSegmentation 所作出的努力。请参考[贡献指南](.github/CONTRIBUTING.md)来了解参与项目贡献的相关指引。

## 致谢

## 贡献指南

## 致谢

MMSegmentation 是一个由来自不同高校和企业的研发人员共同参与贡献的开源项目。我们感谢所有为项目提供算法复现和新功能支持的贡献者，以及提供宝贵反馈的用户。我们希望这个工具箱和基准测试可以为社区提供灵活的代码工具，供用户复现已有算法并开发自己的新模型，从而不断为开源社区提供贡献。

## 引用

如果你觉得本项目对你的研究工作有所帮助，请参考如下 bibtex 引用 MMSegmentation。

```bibtex
}
```

## 开源许可证

- [MIM](https://github.com/open-mmlab/mim): OpenMMLab 项目、算法、模型的统一入口

扫描下方的二维码可关注 OpenMMLab 团队的 [知乎官方账号](https://www.zhihu.com/people/openmmlab)，加入 [OpenMMLab 团队](https://jq.qq.com/?_wv=1027&k=aCvMxdr3) 以及 [MMSegmentation](https://jq.qq.com/?_wv=1027&k=ukevz6Ie) 的 QQ 群。

 <img src="docs/zh_cn/imgs/zhihu_qrcode.jpg" height="400" />  <img src="docs/zh_cn/imgs/qq_group_qrcode.jpg" height="400" />  <img src="docs/zh_cn/imgs/seggroup_qrcode.jpg" height="400" />
 </div>

# 训练与测试

中文内容建设中，暂时请查阅[英文版文档](../../en/user_guides/train_and_test.md)

# 从 MMDetection 2.x 迁移至 3.x

MMDetection 3.x 版本是一个重大更新，包含了许多 API 和配置文件的变化。本文档旨在帮助用户从 MMDetection 2.x 版本迁移到 3.x 版本。
我们将迁移指南分为以下几个部分：

- [配置文件迁移](./config_migration.md)
- [API 和 Registry 迁移](./api_and_registry_migration.md)
- [数据集迁移](./dataset_migration.md)
- [模型迁移](./model_migration.md)
- [常见问题](./migration_faq.md)

如果您在迁移过程中遇到任何问题，欢迎在 issue 中提出。我们也欢迎您为本文档做出贡献。

# Migration of Model Settings

We update model settings in MMagic 1.x. Important modifications are as following.

- Remove `pretrained` fields.
- Add `train_cfg` and `test_cfg` fields in model settings.
- Add `data_preprocessor` fields. Normalization and color space transforms operations are moved from datasets transforms pipelines to data_preprocessor. We will introduce data_preprocessor later.

<thead>
  <tr>
<tbody>
<tr>

```python
    type='BasicRestorer',  # Name of the model
        type='EDSR',  # Type of the generator

mid_channels=64,  # Channel number of intermediate features
    pretrained=None,
    pixel_loss=dict(type='L1Loss', loss_weight=1.0, reduction='mean'))  # Config for pixel loss model training and testing settings
```

</td>

```python

out_channels=3,  # Channel number of outputs
                                                             255.]))
```

</td>

</tr>
</thead>
</table>

We refactor models in MMagic 1.x. Important modifications are as following.

255.]))
```

</td>

</tr>
</thead>
</table>

- Add `data_preprocessor` module in `models`. Normalization and color space transforms operations are moved from datasets transforms pipelines to data_preprocessor. The data out from the data pipeline is transformed by this module and then fed into the model.

More details of models are shown in [model guides](../howto/models.md).

# Quick Run

This chapter will take you through the basic functions of MMOCR. And we assume you [installed MMOCR from source](install.md#best-practices). You may check out the [tutorial notebook](https://colab.research.google.com/github/open-mmlab/mmocr/blob/dev-1.x/demo/tutorial.ipynb) for how to perform inference, training and testing interactively.

Run the following in MMOCR's root directory:

```shell
```

You should be able to see a pop-up image and the inference result printed out in the console.

</div>
<br />

```bash

[...]}]}
```

```{note}
```

A detailed description of MMOCR's inference interface can be found [here](../user_guides/inference.md)

In addition to using our well-provided pre-trained models, you can also train models on your own datasets. In the next section, we will take you through the basic functions of MMOCR by training DBNet on the mini [ICDAR 2015](https://rrc.cvc.uab.es/?ch=4&com=downloads) dataset as an example.

Since the variety of OCR dataset formats are not conducive to either switching or joint training of multiple datasets, MMOCR proposes a uniform [data format](../user_guides/dataset_prepare.md), and provides [dataset preparer](../user_guides/data_prepare/dataset_preparer.md) for commonly used OCR datasets. Usually, to use those datasets in MMOCR, you just need to follow the steps to get them ready for use.

```{note}
```

Here, we have prepared a lite version of ICDAR 2015 dataset for demonstration purposes. Download our pre-prepared [zip](https://download.openmmlab.com/mmocr/data/icdar2015/mini_icdar2015.tar.gz) and extract it to the `data/` directory under mmocr to get our prepared image and annotation file.

```Bash
```

```Bash
```

Once the dataset is prepared, we will then specify the location of the training set and the training parameters by modifying the config file.

In this example, we will train a DBNet using resnet18 as its backbone. Since MMOCR already has a config file for the full ICDAR 2015 dataset (`configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py`), we just need to make some modifications on top of it.

We first need to modify the path to the dataset. In this config, most of the key config files are imported in `_base_`, such as the database configuration from `configs/textdet/_base_/datasets/icdar2015.py`. Open that file and replace the path pointed to by `icdar2015_textdet_data_root` in the first line with:

```Python
```

```Python
```

Also, because of the reduced dataset size, we have to reduce the number of training epochs to 400 accordingly, shorten the validation interval as well as the weight storage interval to 10 rounds, and drop the learning rate decay strategy. The following lines of configuration can be directly put into `configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py` to take effect.

```Python
# Save checkpoints every 10 epochs, and only keep the latest checkpoint
    ))
# Set the maximum number of epochs to 400, and validate the model every 10 epochs
# Fix learning rate as a constant
    dict(type='ConstantLR', factor=1.0),
]
```

]
```

Here, we have rewritten the corresponding parameters in the base configuration directly through the inheritance ({external+mmengine:doc}`MMEngine: Config <advanced_tutorials/config>`) mechanism of the config. The original fields are distributed in `configs/textdet/_base_/schedules/schedule_sgd_1200e.py` and `configs/textdet/_base_/default_runtime.py`.

```{note}
For a more detailed description of config, please refer to [here](../user_guides/config.md).
```

Before we start the training, we can also visualize the image processed by training-time [data transforms](../basic_concepts/transforms.md). It's quite simple: pass the config file we need to visualize into the [browse_dataset.py](/tools/analysis_tools/browse_dataset.py) script.

```Bash

```Bash
```

The transformed images and annotations will be displayed one by one in a pop-up window.

</center>

```{note}
```

```{tip}
```

```

```Bash
```

Depending on the system environment, MMOCR will automatically use the best device for training. If a GPU is available, a single GPU training will be started by default. When you start to see the output of the losses, you have successfully started the training.

```Bash

```

Without extra configurations, model weights will be saved to `work_dirs/dbnet_resnet18_fpnc_1200e_icdar2015/`, while the logs will be stored in `work_dirs/dbnet_resnet18_fpnc_1200e_icdar2015/TIMESTAMP/`. Next, we just need to wait with some patience for training to finish.

```{note}
```

## Testing

After 400 epochs, we observe that DBNet performs best in the last epoch, with `hmean` reaching 60.86 (You may see a different result):

```Bash

```

## Testing

```Bash
```

```{note}
It may not have been trained to be optimal, but it is sufficient for a demo.
```

However, this value only reflects the performance of DBNet on the mini ICDAR 2015 dataset. For a comprehensive evaluation, we also need to see how it performs on out-of-distribution datasets. For example, `tests/data/det_toy_dataset` is a very small real dataset that we can use to verify the actual performance of DBNet.

Before testing, we also need to make some changes to the location of the dataset. Open `configs/textdet/_base_/datasets/icdar2015.py` and change `data_root` of `icdar2015_textdet_test` to `tests/data/det_toy_dataset`:

```Python
# ...

```Python
# ...
    #  ...
    )
```

```Bash
```

```Bash

```

The model achieves an hmean of 0.6667 on this dataset.

```{note}
```

## Visualize the Outputs

```

We can also visualize its prediction output in `test.py`. You can open a pop-up visualization window with the `show` parameter; and can also specify the directory where the prediction result images are exported with the `show-dir` parameter.

```Bash
```

The true labels and predicted values are displayed in a tiled fashion in the visualization results. The green boxes in the left panel indicate the true labels and the red boxes in the right panel indicate the predicted values.

</div>

```{note}

</div>

```{note}
For a description of more visualization features, see [here](../user_guides/visualization.md).
```

# Human Body 2D Pose Estimation

Multi-person human pose estimation is defined as the task of detecting the poses (or keypoints) of all people from an input image.

Top-down methods (e.g. DeepPose) divide the task into two stages: human detection and pose estimation. They perform human detection first, followed by single-person pose estimation given human bounding boxes.

Bottom-up approaches (e.g. Associative Embedding) first detect all the keypoints and then group/associate them into person instances.

Please follow [DATA Preparation](/docs/en/dataset_zoo/2d_body_keypoint.md) to prepare data.

## Demo

Please follow [Demo](/demo/docs/en/2d_human_pose_demo.md#2d-human-pose-demo) to run demos.

</div>

# PGGAN (ICLR'2018)

> [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://arxiv.org/abs/1710.10196)

We describe a new training methodology for generative adversarial networks. The key idea is to grow both the generator and discriminator progressively: starting from a low resolution, we add new layers that model increasingly fine details as training progresses. This both speeds the training up and greatly stabilizes it, allowing us to produce images of unprecedented quality, e.g., CelebA images at 1024^2. We also propose a simple way to increase the variation in generated images, and achieve a record inception score of 8.80 in unsupervised CIFAR10. Additionally, we describe several implementation details that are important for discouraging unhealthy competition between the generator and discriminator. Finally, we suggest a new metric for evaluating GAN results, both in terms of image quality and variation. As an additional contribution, we construct a higher-quality version of the CelebA dataset.

<img src="https://user-images.githubusercontent.com/28132635/143053374-c03894c3-6def-49c2-94ed-80c4accee726.JPG" />
</div>

  <b> Results (compressed) from our PGGAN trained in CelebA-HQ@1024</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/12726765/114009864-1df45400-9896-11eb-9d25-da9eabfe02ce.png" width="800"/>
</div>

|                              Model                              |    Dataset     | MS-SSIM |     SWD(xx,xx,xx,xx/avg)     |                                         Download                                         |
| :-------------------------------------------------------------: | :------------: | :-----: | :--------------------------: | :--------------------------------------------------------------------------------------: |

| [pggan_128x128](./pggan_8xb4-12Mimgs_celeba-cropped-128x128.py) | celeba-cropped | 0.3023  | 3.42, 4.04, 4.78, 20.38/8.15 | [model](https://download.openmmlab.com/mmediting/pggan/pggan_celeba-cropped_128_g8_20210408_181931-85a2e72c.pth) |
|  [pggan_128x128](./pggan_8xb4-12Mimgs_lsun-bedroom-128x128.py)  |  lsun-bedroom  | 0.0602  |  3.5, 2.96, 2.76, 9.65/4.72  | [model](https://download.openmmlab.com/mmediting/pggan/pggan_lsun-bedroom_128x128_g8_20210408_182033-5e59f45d.pth) |
|  [pggan_1024x1024](./pggan_8xb4-12Mimg_celeba-hq-1024x1024.py)  |   celeba-hq    | 0.3379  | 8.93, 3.98, 3.07, 2.64/4.655 | [model](https://download.openmmlab.com/mmediting/pggan/pggan_celeba-hq_1024_g8_20210408_181911-f1ef51c3.pth) |

<summary align="right"><a href="https://arxiv.org/abs/1710.10196">PGGAN (arXiv'2017)</a></summary>

```latex

```latex
  url={https://arxiv.org/abs/1710.10196},
}
```

<details>

```bibtex
}
```

# spdlog

Very fast, header-only/compiled, C++ logging library. [![Build Status](https://app.travis-ci.com/gabime/spdlog.svg?branch=v1.x)](https://app.travis-ci.com/gabime/spdlog)&nbsp; [![Build status](https://ci.appveyor.com/api/projects/status/d2jnxclg20vd0o50?svg=true&branch=v1.x)](https://ci.appveyor.com/project/gabime/spdlog) [![Release](https://img.shields.io/github/release/gabime/spdlog.svg)](https://github.com/gabime/spdlog/releases/latest)

## Install 
#### Header only version
Copy the include [folder](https://github.com/gabime/spdlog/tree/v1.x/include/spdlog) to your build tree and use a C++11 compiler.

#### Static lib version (recommended - much faster compile times)
```console
$ git clone https://github.com/gabime/spdlog.git
$ cd spdlog && mkdir build && cd build
$ cmake .. && make -j
```
      
   see example [CMakeLists.txt](https://github.com/gabime/spdlog/blob/v1.x/example/CMakeLists.txt) on how to use.

## Platforms
 * Linux, FreeBSD, OpenBSD, Solaris, AIX

```
      
 * Windows (msvc 2013+, cygwin)
 * macOS (clang 3.5+)
 * Android

## Package managers:
* Debian: `sudo apt install libspdlog-dev`
* Homebrew: `brew install spdlog`
* MacPorts: `sudo port install spdlog`
* FreeBSD:  `pkg install spdlog`
* Fedora: `dnf install spdlog`
* Gentoo: `emerge dev-libs/spdlog`
* Arch Linux: `pacman -S spdlog`
* vcpkg: `vcpkg install spdlog`
* conan: `spdlog/[>=1.4.1]`
* conda: `conda install -c conda-forge spdlog`
* build2: ```depends: spdlog ^1.8.2```

* Very fast (see [benchmarks](#benchmarks) below).
* Headers only or compiled
* Feature rich formatting, using the excellent [fmt](https://github.com/fmtlib/fmt) library.
* Asynchronous mode (optional)
* [Custom](https://github.com/gabime/spdlog/wiki/3.-Custom-formatting) formatting.
* Multi/Single threaded loggers.

* Various log targets:
    * Rotating log files.
    * Daily log files.
    * Console logging (colors supported).
    * syslog.
    * Windows event log.
    * Windows debugger (```OutputDebugString(..)```).
    * Easily [extendable](https://github.com/gabime/spdlog/wiki/4.-Sinks#implementing-your-own-sink) with custom log targets.
* Log filtering - log levels can be modified in runtime as well as in compile time.
* Support for loading log levels from argv or from environment var.
* [Backtrace](#backtrace-support) support - store debug messages in a ring buffer and display later on demand.
 
## Usage samples

#### Basic usage
```c++
#include "spdlog/spdlog.h"

int main() 
{
    spdlog::info("Welcome to spdlog!");
    spdlog::error("Some error message with arg: {}", 1);
    
    spdlog::warn("Easy padding in numbers like {:08d}", 12);

int main() 
{
    
    spdlog::critical("Support for int: {0:d};  hex: {0:x};  oct: {0:o}; bin: {0:b}", 42);
    spdlog::info("Support for floats {:03.2f}", 1.23456);
    spdlog::info("Positional args are {1} {0}..", "too", "supported");
    spdlog::info("{:<30}", "left aligned");
    
    spdlog::set_level(spdlog::level::debug); // Set global log level to debug
    spdlog::debug("This message should be displayed..");    
    
    // change log pattern
    spdlog::set_pattern("[%H:%M:%S %z] [%n] [%^---%L---%$] [thread %t] %v");
    
    // Compile time log levels
    // define SPDLOG_ACTIVE_LEVEL to desired level
    SPDLOG_TRACE("Some trace message with param {}", 42);
    SPDLOG_DEBUG("Some debug message");
}

```
---
#### Create stdout/stderr logger object
```c++

SPDLOG_TRACE("Some trace message with param {}", 42);
}

```
---
```c++
#include "spdlog/sinks/stdout_color_sinks.h"
void stdout_example()
{
    // create color multi threaded logger
    auto console = spdlog::stdout_color_mt("console");    
    auto err_logger = spdlog::stderr_color_mt("stderr");    
    spdlog::get("console")->info("loggers can be retrieved from a global registry using the spdlog::get(logger_name)");
}
```

---
#### Basic file logger
```c++
#include "spdlog/sinks/basic_file_sink.h"
void basic_logfile_example()
{
    try 
    {
        auto logger = spdlog::basic_logger_mt("basic_logger", "logs/basic-log.txt");
    }
    catch (const spdlog::spdlog_ex &ex)
    {
        std::cout << "Log init failed: " << ex.what() << std::endl;
    }
}
```
---
#### Rotating files
```c++
#include "spdlog/sinks/rotating_file_sink.h"
void rotating_example()
{

{
    }
}
```
---
```c++
{
    // Create a file rotating logger with 5mb size max and 3 rotated files
    auto max_size = 1048576 * 5;
    auto max_files = 3;
    auto logger = spdlog::rotating_logger_mt("some_logger_name", "logs/rotating.txt", max_size, max_files);
}
```

---
#### Daily files
```c++

#include "spdlog/sinks/daily_file_sink.h"
void daily_example()
{
    // Create a daily logger - a new file is created every day on 2:30am
    auto logger = spdlog::daily_logger_mt("daily_logger", "logs/daily.txt", 2, 30);
}

```

---
#### Backtrace support
```c++
// Debug messages can be stored in a ring buffer instead of being logged immediately.
// This is useful in order to display debug logs only when really needed (e.g. when error happens).
// When needed, call dump_backtrace() to see them.

spdlog::enable_backtrace(32); // Store the latest 32 messages in a buffer. Older messages will be dropped.
// or my_logger->enable_backtrace(32)..
for(int i = 0; i < 100; i++)
{
  spdlog::debug("Backtrace message {}", i); // not logged yet..
}
// e.g. if some error happened:
spdlog::dump_backtrace(); // log them now! show the last 32 messages

// or my_logger->dump_backtrace(32)..
```

---
#### Periodic flush
```c++
// periodically flush all *registered* loggers every 3 seconds:
// warning: only use if all your loggers are thread safe ("_mt" loggers)
spdlog::flush_every(std::chrono::seconds(3));

```

---
#### Stopwatch
```c++
// Stopwatch support for spdlog
#include "spdlog/stopwatch.h"
void stopwatch_example()
{
    spdlog::stopwatch sw;    
    spdlog::debug("Elapsed {}", sw);
    spdlog::debug("Elapsed {:.3}", sw);       
}

```

{
}

```

---
#### Log binary data in hex
```c++
// many types of std::container<char> types can be used.
// ranges are supported too.
// format flags:
// {:X} - print in uppercase.
// {:s} - don't separate each byte with space.
// {:p} - don't print the position on each line start.
// {:n} - don't split the output to lines.
// {:a} - show ASCII if :n is not set.

#include "spdlog/fmt/bin_to_hex.h"

void binary_example()
{
    auto console = spdlog::get("console");
    std::array<char, 80> buf;
    console->info("Binary example: {}", spdlog::to_hex(buf));
    console->info("Another binary example:{:n}", spdlog::to_hex(std::begin(buf), std::begin(buf) + 10));
    // more examples:
    // logger->info("uppercase: {:X}", spdlog::to_hex(buf));
    // logger->info("uppercase, no delimiters: {:Xs}", spdlog::to_hex(buf));

// more examples:
    // logger->info("uppercase, no delimiters, no position info: {:Xsp}", spdlog::to_hex(buf));
}

```

---
#### Logger with multi sinks - each with different format and log level
```c++

// create logger with 2 targets with different log levels and formats.
// the console will show only warnings or errors, while the file will log all.
void multi_sink_example()
{
    auto console_sink = std::make_shared<spdlog::sinks::stdout_color_sink_mt>();
    console_sink->set_level(spdlog::level::warn);
    console_sink->set_pattern("[multi_sink_example] [%^%l%$] %v");

    auto file_sink = std::make_shared<spdlog::sinks::basic_file_sink_mt>("logs/multisink.txt", true);
    file_sink->set_level(spdlog::level::trace);

    spdlog::logger logger("multi_sink", {console_sink, file_sink});
    logger.set_level(spdlog::level::debug);

file_sink->set_level(spdlog::level::trace);

    logger.warn("this should appear in both console and file");
    logger.info("this message should not appear in the console, only in the file");
}
```

---
#### Asynchronous logging
```c++
#include "spdlog/async.h"
void async_example()
{
    // default thread pool settings can be modified *before* creating the async logger:
    // spdlog::init_thread_pool(8192, 1); // queue with 8k items and 1 backing thread.
    auto async_file = spdlog::basic_logger_mt<spdlog::async_factory>("async_file_logger", "logs/async_log.txt");
    // alternatively:
    // auto async_file = spdlog::create_async<spdlog::sinks::basic_file_sink_mt>("async_file_logger", "logs/async_log.txt");   
}

```

---
#### Asynchronous logger with multi sinks  
```c++

}

```

---
```c++

void multi_sink_example2()
{
    spdlog::init_thread_pool(8192, 1);
    auto stdout_sink = std::make_shared<spdlog::sinks::stdout_color_sink_mt >();
    auto rotating_sink = std::make_shared<spdlog::sinks::rotating_file_sink_mt>("mylog.txt", 1024*1024*10, 3);
    std::vector<spdlog::sink_ptr> sinks {stdout_sink, rotating_sink};
    auto logger = std::make_shared<spdlog::async_logger>("loggername", sinks.begin(), sinks.end(), spdlog::thread_pool(), spdlog::async_overflow_policy::block);
    spdlog::register_logger(logger);
}
```
 
---
#### User defined types
```c++
// user defined types logging by implementing operator<<
#include "spdlog/fmt/ostr.h" // must be included
struct my_type
{
    int i;
    template<typename OStream>
    friend OStream &operator<<(OStream &os, const my_type &c)
    {

{
    int i;
    {
        return os << "[my_type i=" << c.i << "]";
    }
};

void user_defined_example()
{
    spdlog::get("console")->info("user defined type: {}", my_type{14});
}

```

---
#### User defined flags in the log pattern
```c++ 
// Log patterns can contain custom flags.
// the following example will add new flag '%*' - which will be bound to a <my_formatter_flag> instance.
#include "spdlog/pattern_formatter.h"
class my_formatter_flag : public spdlog::custom_flag_formatter
{
public:
    void format(const spdlog::details::log_msg &, const std::tm &, spdlog::memory_buf_t &dest) override
    {
        std::string some_txt = "custom-flag";
        dest.append(some_txt.data(), some_txt.data() + some_txt.size());
    }

    std::unique_ptr<custom_flag_formatter> clone() const override
    {

std::string some_txt = "custom-flag";
    }

    {
        return spdlog::details::make_unique<my_formatter_flag>();
    }
};

void custom_flags_example()
{    
    auto formatter = std::make_unique<spdlog::pattern_formatter>();
    formatter->add_flag<my_formatter_flag>('*').set_pattern("[%n] [%*] [%^%l%$] %v");
    spdlog::set_formatter(std::move(formatter));
}

```

---
#### Custom error handler
```c++
void err_handler_example()
{
    // can be set globally or per logger(logger->set_error_handler(..))
    spdlog::set_error_handler([](const std::string &msg) { spdlog::get("console")->error("*** LOGGER ERROR ***: {}", msg); });
    spdlog::get("console")->info("some invalid message to trigger an error {}{}{}{}", 3);
}

```

---
#### syslog 
```c++
#include "spdlog/sinks/syslog_sink.h"
void syslog_example()
{

spdlog::get("console")->info("some invalid message to trigger an error {}{}{}{}", 3);
}

```

---
```c++
{
    std::string ident = "spdlog-example";
    auto syslog_logger = spdlog::syslog_logger_mt("syslog", ident, LOG_PID);
    syslog_logger->warn("This is warning that will end up in syslog.");
}
```
---
#### Android example 
```c++
#include "spdlog/sinks/android_sink.h"
void android_example()
{
    std::string tag = "spdlog-android";
    auto android_logger = spdlog::android_logger_mt("android", tag);
    android_logger->critical("Use \"adb shell logcat\" to view this message.");
}
```

---
#### Load log levels from env variable or from argv

```c++
#include "spdlog/cfg/env.h"
int main (int argc, char *argv[])
{
    spdlog::cfg::load_env_levels();
    // or from command line:
    // ./example SPDLOG_LEVEL=info,mylogger=trace
    // #include "spdlog/cfg/argv.h" // for loading levels from argv

{
    // spdlog::cfg::load_argv_levels(argc, argv);
}
```
So then you can:

$ export SPDLOG_LEVEL=info,mylogger=trace
$ ./example
```

---
#### Log file open/close event handlers
```c++
// You can get callbacks from spdlog before/after log file has been opened or closed. 
// This is useful for cleanup procedures or for adding someting the start/end of the log files.
void file_events_example()
{
    // pass the spdlog::file_event_handlers to file sinks for open/close log file notifications
    spdlog::file_event_handlers handlers;
    handlers.before_open = [](spdlog::filename_t filename) { spdlog::info("Before opening {}", filename); };
    handlers.after_open = [](spdlog::filename_t filename, std::FILE *fstream) { fputs("After opening\n", fstream); };

handlers.after_open = [](spdlog::filename_t filename, std::FILE *fstream) { fputs("After opening\n", fstream); };
    handlers.before_close = [](spdlog::filename_t filename, std::FILE *fstream) { fputs("Before closing\n", fstream); };
    handlers.after_close = [](spdlog::filename_t filename) { spdlog::info("After closing {}", filename); };
    auto my_logger = spdlog::basic_logger_st("some_logger", "logs/events-sample.txt", true, handlers);        
}
```

---
#### Replace the Default Logger
```c++
void replace_default_logger_example()
{
    auto new_logger = spdlog::basic_logger_mt("new_default_logger", "logs/new-default-log.txt", true);
    spdlog::set_default_logger(new_logger);
    spdlog::info("new logger log message");
}
```

---
## Benchmarks

Below are some [benchmarks](https://github.com/gabime/spdlog/blob/v1.x/bench/bench.cpp) done in Ubuntu 64 bit, Intel i7-4770 CPU @ 3.40GHz

#### Synchronous mode
```
[info] **************************************************************

```
[info] Single thread, 1,000,000 iterations
[info] basic_st         Elapsed: 0.17 secs        5,777,626/sec
[info] rotating_st      Elapsed: 0.18 secs        5,475,894/sec
[info] daily_st         Elapsed: 0.20 secs        5,062,659/sec
[info] empty_logger     Elapsed: 0.07 secs       14,127,300/sec
[info] C-string (400 bytes). Single thread, 1,000,000 iterations
[info] basic_st         Elapsed: 0.41 secs        2,412,483/sec
[info] rotating_st      Elapsed: 0.72 secs        1,389,196/sec
[info] daily_st         Elapsed: 0.42 secs        2,393,298/sec
[info] null_st          Elapsed: 0.04 secs       27,446,957/sec

[info] 10 threads, competing over the same logger object, 1,000,000 iterations
[info] basic_mt         Elapsed: 0.60 secs        1,659,613/sec
[info] rotating_mt      Elapsed: 0.62 secs        1,612,493/sec
[info] daily_mt         Elapsed: 0.61 secs        1,638,305/sec
[info] null_mt          Elapsed: 0.16 secs        6,272,758/sec
```
#### Asynchronous mode
```
[info] -------------------------------------------------
[info] Messages     : 1,000,000
[info] Threads      : 10
[info] Queue        : 8,192 slots
[info] Queue memory : 8,192 x 272 = 2,176 KB 
[info] 
[info] *********************************
[info] Queue Overflow Policy: block

[info] 
[info] Elapsed: 1.70784 secs     585,535/sec
[info] Elapsed: 1.69805 secs     588,910/sec
[info] Elapsed: 1.7026 secs      587,337/sec
[info] 
[info] Queue Overflow Policy: overrun
[info] Elapsed: 0.372816 secs    2,682,285/sec
[info] Elapsed: 0.379758 secs    2,633,255/sec
[info] Elapsed: 0.373532 secs    2,677,147/sec

```

## Documentation
Documentation can be found in the [wiki](https://github.com/gabime/spdlog/wiki/1.-QuickStart) pages.

---

```

---

Thanks to [JetBrains](https://www.jetbrains.com/?from=spdlog) for donating product licenses to help develop **spdlog** <a href="https://www.jetbrains.com/?from=spdlog"><img src="logos/jetbrains-variant-4.svg" width="94" align="center" /></a>

<details>

```bibtex
}
```

<details>

```bibtex

author={Koestinger, Martin and Wohlhart, Paul and Roth, Peter M and Bischof, Horst},
}
```

| [pose_hrnetv2_w18](/configs/face_2d_keypoint/topdown_heatmap/aflw/td-hm_hrnetv2-w18_8xb64-60e_aflw-256x256.py) |  256x256   |         1.41         |          1.27           | [ckpt](https://download.openmmlab.com/mmpose/face/hrnetv2/hrnetv2_w18_aflw_256x256-f2bbc62b_20210125.pth) | [log](https://download.openmmlab.com/mmpose/face/hrnetv2/hrnetv2_w18_aflw_256x256_20210125.log.json) |

# GWD

> [Rethinking Rotated Object Detection with Gaussian Wasserstein Distance Loss](https://arxiv.org/pdf/2101.11952.pdf)

<img src="https://raw.githubusercontent.com/zytx121/image-host/main/imgs/gwd.png" width="800"/>
</div>

</div>

Boundary discontinuity and its inconsistency to the final detection metric have been the bottleneck for rotating detection regression loss design. In this paper, we propose a novel regression loss based on Gaussian Wasserstein distance as a fundamental approach to solve the problem. Specifically, the rotated bounding box is converted to a 2- D Gaussian distribution, which enables to approximate the indifferentiable rotational IoU induced loss by the Gaussian Wasserstein distance (GWD) which can be learned efficiently by gradient back-propagation. GWD can still be informative for learning even there is no overlapping between two rotating bounding boxes which is often the case for small object detection. Thanks to its three unique properties, GWD can also elegantly solve the boundary discontinuity and square-like problem regardless how the bounding box is defined. Experiments on five datasets using different detectors show the effectiveness of our approach.

| ResNet50 (1024,1024,200) | 69.55 |  oc   |   1x    |   3.39   |      15.5      |  -  |     2      |      [rotated-retinanet-hbox-oc_r50_fpn_gwd_1x_dota](./rotated-retinanet-hbox-oc_r50_fpn_gwd_1x_dota.py)       |       [model](https://download.openmmlab.com/mmrotate/v0.1.0/gwd/rotated_retinanet_hbb_gwd_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_gwd_r50_fpn_1x_dota_oc-41fd7805.pth) \| [log](https://download.openmmlab.com/mmrotate/v0.1.0/gwd/rotated_retinanet_hbb_gwd_r50_fpn_1x_dota_oc/rotated_retinanet_hbb_gwd_r50_fpn_1x_dota_oc_20220120_152421.log.json)       |

```
@inproceedings{yang2021rethinking,
    title={Rethinking Rotated Object Detection with Gaussian Wasserstein Distance Loss},
    author={Yang, Xue and Yan, Junchi and Qi, Ming and Wang, Wentao and Xiaopeng, Zhang and Qi, Tian},
    booktitle={International Conference on Machine Learning},
}
```

DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries

> [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://arxiv.org/abs/2110.06922)

We introduce a framework for multi-camera 3D object detection. In
contrast to existing works, which estimate 3D bounding boxes directly from
monocular images or use depth prediction networks to generate input for 3D object
detection from 2D information, our method manipulates predictions directly
in 3D space. Our architecture extracts 2D features from multiple camera images
and then uses a sparse set of 3D object queries to index into these 2D features,
linking 3D positions to multi-view images using camera transformation matrices.
Finally, our model makes a bounding box prediction per object query, using a
set-to-set loss to measure the discrepancy between the ground-truth and the prediction.
This top-down approach outperforms its bottom-up counterpart in which

object bounding box prediction follows per-pixel depth estimation, since it does
not suffer from the compounding error introduced by a depth prediction model.
Moreover, our method does not require post-processing such as non-maximum
suppression, dramatically improving inference speed. We achieve state-of-the-art
performance on the nuScenes autonomous driving benchmark.

<img src="https://user-images.githubusercontent.com/67246790/209751755-3d0f0ad5-6a39-4d14-a1c7-346b5c228a1b.png" width="800"/>
</div>

This directory contains the implementations of DETR3D (https://arxiv.org/abs/2110.06922). Our implementations are built on top of MMdetection3D.

</div>

We have updated DETR3D to be compatible with latest mmdet3d-dev1.x. The codebase and config files have all changed to adapt to the new mmdet3d version. All previous pretrained models are verified with the result listed below. However, newly trained models are yet to be uploaded.

We require the version of mmdet \<= V3.0.0rc5. The mmdet later than V3.0.0rc5 has refactored DETR-series and its config file, but our configs and code are yet to be updated.

## Train

1. Downloads the [pretrained backbone weights](https://drive.google.com/drive/folders/1h5bDg7Oh9hKvkFL-dRhu5-ahrEp2lRNN?usp=sharing) to pretrained/

2. For example, to train DETR3D on 8 GPUs, please use

```bash
bash tools/dist_train.sh projects/DETR3D/configs/detr3d_res101_gridmask.py 8 --cfg-options load_from=pretrained/fcos3d.pth
```

```bash
```

## Evaluation using pretrained models

1. Download the newly trained weights accordingly.

   |                                                Backbone                                                 | mAP  | NDS  |                                                                                                                 Download                                                                                                                 |

| :-----------------------------------------------------------------------------------------------------: | :--: | :--: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
   |          [DETR3D, ResNet101 w/ DCN, evaluation on val set](./configs/detr3d_r101_gridmask.py)           | 35.5 | 42.8 |                 [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/detr3d/detr3d_r101_gridmask.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/detr3d/detr3d_r101_gridmask.log)                 |

|             [above, + CBGS, evaluation on val set](./configs/detr3d_r101_gridmask_cbgs.py)              | 35.2 | 42.7 |            [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/detr3d/detr3d_r101_gridmask_cbgs.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/detr3d/detr3d_r101_gridmask_cbgs.log)            |
   | [DETR3D, VoVNet on trainval, evaluation on test set](./configs/detr3d_vovnet_gridmask_trainval_cbgs.py) | 41.4 | 48.1 | [model](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/detr3d/detr3d_vovnet_gridmask_trainval_cbgs.pth) \| [log](https://download.openmmlab.com/mmdetection3d/v1.1.0_models/detr3d/detr3d_vovnet_gridmask_trainval_cbgs.log) |

2. Testing

   To test, use:

   ```bash
    bash tools/dist_test.sh projects/DETR3D/configs/detr3d_res101_gridmask.py ${CHECKPOINT_PATH} 8
   ```

## Converting old models (Optional)

For old models please refer to [Object DGCNN & DETR3D](https://github.com/WangYueFt/detr3d)

```

From v0.17.3 to v1.0.0, mmdet3d has changed its bbox representation. Given that Box(x,y,z,θ), we have x_new = y_old, y_new = x_old, θ_new = -θ_old - π/2.

Old models are trained on v0.17.3. Our regression branch outputs (cx,cy,w,l,cz,h,sin(θ),cos(θ),vx,vy). For a previous model which outputs y=\[y0,y1,y2,y3,y4,y5,y6,y7,y8,y9\], we get y_new = \[...,y3,y2,...,-y7,-y6, ...\]. So we should change the final Linear layer's weight accordingly.

To convert the old weights, please use

```bash
python projects/DETR3D/detr3d/old_detr3d_converter.py ${CHECKPOINT_DIR}/detr3d_resnet101.pth ${CHECKPOINT_DIR}/detr3d_r101_v1.0.0.pth --code_size 10
```

If you find this repo useful for your research, please consider citing the papers

```
   detr3d,
   title={DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries},

```
   detr3d,
   author={Wang, Yue and Guizilini, Vitor and Zhang, Tianyuan and Wang, Yilun and Zhao, Hang and and Solomon, Justin M.},
   booktitle={The Conference on Robot Learning ({CoRL})},
   year={2021}
}
```

# Preparing RealBlur Dataset

```bibtex
}
```

The test datasets RealBlurR can be download from [here](https://drive.google.com/file/d/1glgeWXCy7Y0qWDc0MXBTUlZYJf8984hS/).
The test datasets RealBlurJ can be download from [here](https://drive.google.com/file/d/1Rb1DhhXmX7IXfilQ-zL9aGjQfAAvQTrW/).

```text
mmagic
├── mmagic
├── tools
├── data
```

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily got feedback.
If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Does the modification introduces changes that break the back-compatibility of this repo?
If so, please describe how it breaks the compatibility and how users should modify their codes to keep compatibility with this PR.

1. Pre-commit or other linting tools should be used to fix the potential lint issues.

2. The modification should be covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
3. If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMCls.
4. The documentation should be modified accordingly, like docstring or example tutorials.

# Build Java API

- OpenJDK >= 10

**Step 1.** Download OpenJDK. Using OpenJDK-18 as example:

```bash
wget https://download.java.net/java/GA/jdk18/43f95e8614114aeaa8e8a5fcf20a682d/36/GPL/openjdk-18_linux-x64_bin.tar.gz
tar xvf openjdk-18_linux-x64_bin.tar.gz
```

**Step 2.** Setting environment variables:

```bash
export JAVA_HOME=${PWD}/jdk-18
export PATH=${JAVA_HOME}/bin:${PATH}
```

**Step 3.** Switch default Java version(optional):

```bash
sudo update-alternatives --config java
sudo update-alternatives --config javac
```

If you have multiple Java versions, you should select the version you will use.

For using Java apis, you should build Java class and C++ SDK.

**Step 1.** Build Java class.

Build Java `.class` files.

```bash
cd csrc/mmdeploy/apis/java
javac mmdeploy/*.java
cd ../../../..
```

**Step 2.** Build SDK.

```bash
```

Build MMDeploy SDK. Please follow this [tutorial](../../../../docs/en/01-how-to-build/linux-x86_64.md)/[教程](../../../../docs/zh_cn/01-how-to-build/linux-x86_64.md) to build SDK. Remember to set the MMDEPLOY_BUILD_SDK_JAVA_API option to ON.

<details>

```bibtex
}
```

<details>

```bibtex
}
```

}
```

| [pose_hrnetv2_w18](/configs/hand_2d_keypoint/topdown_heatmap/rhd2d/td-hm_hrnetv2-w18_8xb64-210e_rhd2d-256x256.py) |  256x256   |  0.992  | 0.902 | 2.21 | [ckpt](https://download.openmmlab.com/mmpose/hand/hrnetv2/hrnetv2_w18_rhd2d_256x256-95b20dd8_20210330.pth) | [log](https://download.openmmlab.com/mmpose/hand/hrnetv2/hrnetv2_w18_rhd2d_256x256_20210330.log.json) |

# MMOCR 1.x 更新汇总

此处列出了 MMOCR 1.x 相对于 0.x 版本的重大更新。

1. 架构升级：MMOCR 1.x 是基于 [MMEngine](https://github.com/open-mmlab/mmengine)，提供了一个通用的、强大的执行器，允许更灵活的定制，提供了统一的训练和测试入口。

2. 统一接口：MMOCR 1.x 统一了数据集、模型、评估和可视化的接口和内部逻辑。支持更强的扩展性。

3. 跨项目调用：受益于统一的设计，你可以使用其他OpenMMLab项目中实现的模型，如MMDet。 我们提供了一个例子，说明如何通过MMDetWrapper使用MMDetection的Mask R-CNN。查看我们的文档以了解更多细节。更多的包装器将在未来发布。

4. 更强的可视化：我们提供了一系列可视化工具， 用户现在可以更方便可视化数据。

5. 更多的文档和教程：我们增加了更多的教程，降低用户的学习门槛。

6. 一站式数据准备：准备数据集已经不再是难事。使用我们的 [Dataset Preparer](https://mmocr.readthedocs.io/zh_CN/dev-1.x/user_guides/data_prepare/dataset_preparer.html)，一行命令即可让多个数据集准备就绪。

7. 拥抱更多 `projects/`: 我们推出了 `projects/` 文件夹，用于存放一些实验性的新特性、框架和模型。我们对这个文件夹下的代码规范不作过多要求，力求让社区的所有想法第一时间得到实现和展示。请查看我们的[样例 project](https://github.com/open-mmlab/mmocr/blob/dev-1.x/projects/example_project/) 以了解更多。

8. 更多新模型：MMOCR 1.0 支持了更多模型和模型种类。

</div>

> [FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation](https://arxiv.org/abs/2012.08512.pdf)

|                                      算法                                       | scale | Vimeo90k-triplet  |      GPU 信息       |                                       下载                                       |
| :-----------------------------------------------------------------------------: | :---: | :---------------: | :-----------------: | :------------------------------------------------------------------------------: |

| [flavr_in4out1_g8b4_vimeo90k_septuplet](./flavr_in4out1_8xb4_vimeo90k-septuplet.py) |  x2   | 36.3340 / 0.96015 | 8 (Tesla PG503-216) | [模型](https://download.openmmlab.com/mmediting/video_interpolators/flavr/flavr_in4out1_g8b4_vimeo90k_septuplet_20220509-c2468995.pth) \| [日志](https://download.openmmlab.com/mmediting/video_interpolators/flavr/flavr_in4out1_g8b4_vimeo90k_septuplet_20220509-c2468995.log.json) |

注：FLAVR 中的 8 倍视频插帧算法将会在未来版本中支持。

## 快速开始

**训练**

<details>

```shell
# CPU上训练

# 单个GPU上训练

# 多个GPU上训练
```

**测试**

<details>

```shell
# CPU上测试

```

**测试**

<details>

```shell
# CPU上测试

# 单个GPU上测试

# 多个GPU上测试
```

```bibtex

```

```bibtex
}
```

```bibtex
}
```

<details>

```bibtex
}
```

}
```

| [HRNet-w32](/configs/body_2d_keypoint/dekr/coco/dekr_hrnet-w32_8xb10-140e_coco-512x512.py) |  512x512   | 0.686 |      0.868      |      0.750      | 0.735 |      0.898      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/dekr/coco/dekr_hrnet-w32_8xb10-140e_coco-512x512_ac7c17bf-20221228.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/dekr/coco/dekr_hrnet-w32_8xb10-140e_coco-512x512_20221228.json) |
| [HRNet-w48](/configs/body_2d_keypoint/dekr/coco/dekr_hrnet-w48_8xb10-140e_coco-640x640.py) |  640x640   | 0.714 |      0.883      |      0.777      | 0.762 |      0.915      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/dekr/coco/dekr_hrnet-w48_8xb10-140e_coco-640x640_74796c32-20230124.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/dekr/coco/dekr_hrnet-w48_8xb10-140e_coco-640x640_20230124.json) |

# TVM 测试

## 支持模型列表

表中仅列出已测试模型，未列出的模型可能同样支持，可以自行尝试转换。

## Test

- tvm 0.9.0

\*: 由于暂时不支持动态转换，因此仅提供 SSD 的精度测试结果。

# XCiT

> [XCiT: Cross-Covariance Image Transformers](https://arxiv.org/abs/2106.09681)

Following their success in natural language processing, transformers have recently shown much promise for computer vision. The self-attention operation underlying transformers yields global interactions between all tokens ,i.e. words or image patches, and enables flexible modelling of image data beyond the local interactions of convolutions. This flexibility, however, comes with a quadratic complexity in time and memory, hindering application to long sequences and high-resolution images. We propose a "transposed" version of self-attention that operates across feature channels rather than tokens, where the interactions are based on the cross-covariance matrix between keys and queries. The resulting cross-covariance attention (XCA) has linear complexity in the number of tokens, and allows efficient processing of high-resolution images. Our cross-covariance image transformer (XCiT) is built upon XCA. It combines the accuracy of conventional transformers with the scalability of convolutional architectures. We validate the effectiveness and generality of XCiT by reporting excellent results on multiple vision benchmarks, including image classification and self-supervised feature learning on ImageNet-1k, object detection and instance segmentation on COCO, and semantic segmentation on ADE20k.

<img src="https://user-images.githubusercontent.com/26739999/218900814-64a44606-150b-4757-aec8-7015c77a9fd1.png" width="60%"/>
</div>

```python

model = get_model('xcit-nano-12-p16_3rdparty_in1k', pretrained=True)
```

Test:

```shell
python tools/test.py configs/xcit/xcit-nano-12-p16_8xb128_in1k.py https://download.openmmlab.com/mmclassification/v0/xcit/xcit-nano-12-p16_3rdparty_in1k_20230213-ed776c38.pth
```

```

| Model                                           | Params (M) | Flops (G) |                      Config                       |                                       Download                                        |
| :---------------------------------------------- | :--------: | :-------: | :-----------------------------------------------: | :-----------------------------------------------------------------------------------: |
| `xcit-nano-12-p16_3rdparty_in1k`\*              |    3.05    |   0.56    |     [config](xcit-nano-12-p16_8xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-nano-12-p16_3rdparty_in1k_20230213-ed776c38.pth) |

| `xcit-nano-12-p16_3rdparty-dist_in1k`\*         |    3.05    |   0.56    |     [config](xcit-nano-12-p16_8xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-nano-12-p16_3rdparty-dist_in1k_20230213-fb247f7b.pth) |
| `xcit-tiny-12-p16_3rdparty_in1k`\*              |    6.72    |   1.24    |     [config](xcit-tiny-12-p16_8xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-tiny-12-p16_3rdparty_in1k_20230213-82c547ca.pth) |
| `xcit-tiny-12-p16_3rdparty-dist_in1k`\*         |    6.72    |   1.24    |     [config](xcit-tiny-12-p16_8xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-tiny-12-p16_3rdparty-dist_in1k_20230213-d5fde0a3.pth) |

| `xcit-nano-12-p16_3rdparty-dist_in1k-384px`\*   |    3.05    |   1.64    |  [config](xcit-nano-12-p16_8xb128_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-nano-12-p16_3rdparty-dist_in1k-384px_20230213-712db4d4.pth) |
| `xcit-nano-12-p8_3rdparty_in1k`\*               |    3.05    |   2.16    |     [config](xcit-nano-12-p8_8xb128_in1k.py)      | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-nano-12-p8_3rdparty_in1k_20230213-3370c293.pth) |
| `xcit-nano-12-p8_3rdparty-dist_in1k`\*          |    3.05    |   2.16    |     [config](xcit-nano-12-p8_8xb128_in1k.py)      | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-nano-12-p8_3rdparty-dist_in1k_20230213-2f87d2b3.pth) |

| `xcit-tiny-24-p16_3rdparty_in1k`\*              |   12.12    |   2.34    |     [config](xcit-tiny-24-p16_8xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-tiny-24-p16_3rdparty_in1k_20230213-366c1cd0.pth) |
| `xcit-tiny-24-p16_3rdparty-dist_in1k`\*         |   12.12    |   2.34    |     [config](xcit-tiny-24-p16_8xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-tiny-24-p16_3rdparty-dist_in1k_20230213-b472e80a.pth) |
| `xcit-tiny-12-p16_3rdparty-dist_in1k-384px`\*   |    6.72    |   3.64    |  [config](xcit-tiny-12-p16_8xb128_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-tiny-12-p16_3rdparty-dist_in1k-384px_20230213-00a20023.pth) |

| `xcit-tiny-12-p8_3rdparty_in1k`\*               |    6.71    |   4.81    |     [config](xcit-tiny-12-p8_8xb128_in1k.py)      | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-tiny-12-p8_3rdparty_in1k_20230213-8b02f8f5.pth) |
| `xcit-tiny-12-p8_3rdparty-dist_in1k`\*          |    6.71    |   4.81    |     [config](xcit-tiny-12-p8_8xb128_in1k.py)      | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-tiny-12-p8_3rdparty-dist_in1k_20230213-f3f9b44f.pth) |
| `xcit-small-12-p16_3rdparty_in1k`\*             |   26.25    |   4.81    |    [config](xcit-small-12-p16_8xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-small-12-p16_3rdparty_in1k_20230213-d36779d2.pth) |

| `xcit-small-12-p16_3rdparty-dist_in1k`\*        |   26.25    |   4.81    |    [config](xcit-small-12-p16_8xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-small-12-p16_3rdparty-dist_in1k_20230213-c95bbae1.pth) |
| `xcit-nano-12-p8_3rdparty-dist_in1k-384px`\*    |    3.05    |   6.34    |  [config](xcit-nano-12-p8_8xb128_in1k-384px.py)   | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-nano-12-p8_3rdparty-dist_in1k-384px_20230213-09d925ef.pth) |
| `xcit-tiny-24-p16_3rdparty-dist_in1k-384px`\*   |   12.12    |   6.87    |  [config](xcit-tiny-24-p16_8xb128_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-tiny-24-p16_3rdparty-dist_in1k-384px_20230213-20e13917.pth) |

| `xcit-small-24-p16_3rdparty_in1k`\*             |   47.67    |   9.10    |    [config](xcit-small-24-p16_8xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-small-24-p16_3rdparty_in1k_20230213-40febe38.pth) |
| `xcit-small-24-p16_3rdparty-dist_in1k`\*        |   47.67    |   9.10    |    [config](xcit-small-24-p16_8xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-small-24-p16_3rdparty-dist_in1k_20230213-130d7262.pth) |
| `xcit-tiny-24-p8_3rdparty_in1k`\*               |   12.11    |   9.21    |     [config](xcit-tiny-24-p8_8xb128_in1k.py)      | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-tiny-24-p8_3rdparty_in1k_20230213-4b9ba392.pth) |

| `xcit-tiny-24-p8_3rdparty-dist_in1k`\*          |   12.11    |   9.21    |     [config](xcit-tiny-24-p8_8xb128_in1k.py)      | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-tiny-24-p8_3rdparty-dist_in1k_20230213-ad9c44b0.pth) |
| `xcit-tiny-12-p8_3rdparty-dist_in1k-384px`\*    |    6.71    |   14.13   |  [config](xcit-tiny-12-p8_8xb128_in1k-384px.py)   | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-tiny-12-p8_3rdparty-dist_in1k-384px_20230213-a072174a.pth) |
| `xcit-small-12-p16_3rdparty-dist_in1k-384px`\*  |   26.25    |   14.14   | [config](xcit-small-12-p16_8xb128_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-small-12-p16_3rdparty-dist_in1k-384px_20230213-ba36c982.pth) |

| `xcit-medium-24-p16_3rdparty_in1k`\*            |   84.40    |   16.13   |    [config](xcit-medium-24-p16_8xb128_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-medium-24-p16_3rdparty_in1k_20230213-ad0aa92e.pth) |
| `xcit-medium-24-p16_3rdparty-dist_in1k`\*       |   84.40    |   16.13   |    [config](xcit-medium-24-p16_8xb128_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-medium-24-p16_3rdparty-dist_in1k_20230213-aca5cd0c.pth) |
| `xcit-small-12-p8_3rdparty_in1k`\*              |   26.21    |   18.69   |     [config](xcit-small-12-p8_8xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-small-12-p8_3rdparty_in1k_20230213-9e364ce3.pth) |

| `xcit-small-12-p8_3rdparty-dist_in1k`\*         |   26.21    |   18.69   |     [config](xcit-small-12-p8_8xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-small-12-p8_3rdparty-dist_in1k_20230213-71886580.pth) |
| `xcit-small-24-p16_3rdparty-dist_in1k-384px`\*  |   47.67    |   26.72   | [config](xcit-small-24-p16_8xb128_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-small-24-p16_3rdparty-dist_in1k-384px_20230213-28fa2d0e.pth) |
| `xcit-tiny-24-p8_3rdparty-dist_in1k-384px`\*    |   12.11    |   27.05   |  [config](xcit-tiny-24-p8_8xb128_in1k-384px.py)   | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-tiny-24-p8_3rdparty-dist_in1k-384px_20230213-30d5e5ec.pth) |

| `xcit-small-24-p8_3rdparty_in1k`\*              |   47.63    |   35.81   |     [config](xcit-small-24-p8_8xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-small-24-p8_3rdparty_in1k_20230213-280ebcc7.pth) |
| `xcit-small-24-p8_3rdparty-dist_in1k`\*         |   47.63    |   35.81   |     [config](xcit-small-24-p8_8xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-small-24-p8_3rdparty-dist_in1k_20230213-f2773c78.pth) |
| `xcit-large-24-p16_3rdparty_in1k`\*             |   189.10   |   35.86   |    [config](xcit-large-24-p16_8xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-large-24-p16_3rdparty_in1k_20230214-d29d2529.pth) |

| `xcit-large-24-p16_3rdparty-dist_in1k`\*        |   189.10   |   35.86   |    [config](xcit-large-24-p16_8xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-large-24-p16_3rdparty-dist_in1k_20230214-4fea599c.pth) |
| `xcit-medium-24-p16_3rdparty-dist_in1k-384px`\* |   84.40    |   47.39   | [config](xcit-medium-24-p16_8xb128_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-medium-24-p16_3rdparty-dist_in1k-384px_20230214-6c23a201.pth) |
| `xcit-small-12-p8_3rdparty-dist_in1k-384px`\*   |   26.21    |   54.92   |  [config](xcit-small-12-p8_8xb128_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-small-12-p8_3rdparty-dist_in1k-384px_20230214-9f2178bc.pth) |

| `xcit-medium-24-p8_3rdparty_in1k`\*             |   84.32    |   63.52   |    [config](xcit-medium-24-p8_8xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-medium-24-p8_3rdparty_in1k_20230214-c362850b.pth) |
| `xcit-medium-24-p8_3rdparty-dist_in1k`\*        |   84.32    |   63.52   |    [config](xcit-medium-24-p8_8xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-medium-24-p8_3rdparty-dist_in1k_20230214-625c953b.pth) |
| `xcit-small-24-p8_3rdparty-dist_in1k-384px`\*   |   47.63    |  105.24   |  [config](xcit-small-24-p8_8xb128_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-small-24-p8_3rdparty-dist_in1k-384px_20230214-57298eca.pth) |

| `xcit-large-24-p16_3rdparty-dist_in1k-384px`\*  |   189.10   |  105.35   | [config](xcit-large-24-p16_8xb128_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-large-24-p16_3rdparty-dist_in1k-384px_20230214-bd515a34.pth) |
| `xcit-large-24-p8_3rdparty_in1k`\*              |   188.93   |  141.23   |     [config](xcit-large-24-p8_8xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-large-24-p8_3rdparty_in1k_20230214-08f2f664.pth) |
| `xcit-large-24-p8_3rdparty-dist_in1k`\*         |   188.93   |  141.23   |     [config](xcit-large-24-p8_8xb128_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-large-24-p8_3rdparty-dist_in1k_20230214-8c092b34.pth) |

| `xcit-medium-24-p8_3rdparty-dist_in1k-384px`\*  |   84.32    |  186.67   | [config](xcit-medium-24-p8_8xb128_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-medium-24-p8_3rdparty-dist_in1k-384px_20230214-5db925e0.pth) |
| `xcit-large-24-p8_3rdparty-dist_in1k-384px`\*   |   188.93   |  415.00   |  [config](xcit-large-24-p8_8xb128_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/xcit/xcit-large-24-p8_3rdparty-dist_in1k-384px_20230214-9f718b1a.pth) |

*Models with * are converted from the [official repo](https://github.com/facebookresearch/xcit). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@article{el2021xcit,
  title={XCiT: Cross-Covariance Image Transformers},

```bibtex
  author={El-Nouby, Alaaeldin and Touvron, Hugo and Caron, Mathilde and Bojanowski, Piotr and Douze, Matthijs and Joulin, Armand and Laptev, Ivan and Neverova, Natalia and Synnaeve, Gabriel and Verbeek, Jakob and others},
  journal={arXiv preprint arXiv:2106.09681},
}
```

# S3DIS Dataset

For the overall process, please refer to the [README](https://github.com/open-mmlab/mmdetection3d/blob/master/data/s3dis/README.md/) page for S3DIS.

### Export S3DIS data

By exporting S3DIS data, we load the raw point cloud data and generate the relevant annotations including semantic labels and instance labels.

The directory structure before exporting should be as below:

```
├── tools
├── data
│   ├── s3dis
```

```

Under folder `Stanford3dDataset_v1.2_Aligned_Version`, the rooms are spilted into 6 areas. We use 5 areas for training and 1 for evaluation (typically `Area_5`). Under the directory of each area, there are folders in which raw point cloud data and relevant annotations are saved. For instance, under folder `Area_1/office_1` the files are as below:

- `office_1.txt`: A txt file storing coordinates and colors of each point in the raw point cloud data.

- `Annotations/`: This folder contains txt files for different object instances. Each txt file represents one instance, e.g.

  - `chair_1.txt`: A txt file storing raw point cloud data of one chair in this room.

  If we concat all the txt files under `Annotations/`, we will get the same point cloud as denoted by `office_1.txt`.

If we concat all the txt files under `Annotations/`, we will get the same point cloud as denoted by `office_1.txt`.

Export S3DIS data by running `python collect_indoor3d_data.py`. The main steps include:

- Export original txt files to point cloud, instance label and semantic label.

And the core function `export` in `indoor3d_util.py` is as follows:

```python
    """Convert original dataset files to points, instance mask and semantic
    mask files. We aggregated all the points from each instance in the room.

    Args:
        anno_path (str): path to annotations. e.g. Area_1/office_2/Annotations/
        out_filename (str): path to save collected points and labels.
        file_format (str): txt or numpy, determines what file format to save.

    Note:
        the points are shifted before save, the most negative point is now
            at origin.
    """

Note:
            at origin.
    """
    ins_idx = 1  # instance ids should be indexed from 1, so 0 is unannotated

    # an example of `anno_path`: Area_1/office_1/Annotations
    # which contains all object instances in this room as txt files
        if one_class not in class_names:  # some rooms have 'staris' class

    # align point cloud to the origin

points_list.append(np.concatenate([points, labels, ins_labels], 1))

```

where we load and concatenate all the point cloud instances under `Annotations/` to form raw point cloud and generate semantic/instance labels. After exporting each room, the point cloud data, semantic labels and instance labels should be saved in `.npy` files.

```shell
```

```shell
```

The above exported point cloud files, semantic label files and instance label files are further saved in `.bin` format. Meanwhile `.pkl` info files are also generated for each area.

```
s3dis
├── points

```

- `points/xxxxx.bin`: The exported point cloud data.
- `instance_mask/xxxxx.bin`: The instance label for each point, value range: \[0, ${NUM_INSTANCES}\], 0: unannotated.
- `semantic_mask/xxxxx.bin`: The semantic label for each point, value range: \[0, 12\].
- `s3dis_infos_Area_1.pkl`: Area 1 data infos, the detailed info of each room is as follows:
  - info\['pts_path'\]: The path of `points/xxxxx.bin`.
  - info\['pts_instance_mask_path'\]: The path of `instance_mask/xxxxx.bin`.

- info\['pts_path'\]: The path of `points/xxxxx.bin`.
  - info\['pts_semantic_mask_path'\]: The path of `semantic_mask/xxxxx.bin`.
  - `Area_1_label_weight.npy`: Weighting factor for each semantic class. Since the number of points in different classes varies greatly, it's a common practice to use label re-weighting to get a better performance.
  - `Area_1_resampled_scene_idxs.npy`: Re-sampling index for each scene. Different rooms will be sampled multiple times according to their number of points to balance training data.

A typical training pipeline of S3DIS for 3D semantic segmentation is as below.

```python

```python
    dict(
    dict(
    dict(
    dict(
        eps=0.0),
    dict(

        eps=0.0),
    dict(
    dict(
]
```

- `PointSegClassMapping`: Only the valid category ids will be mapped to class label ids like \[0, 13) during training. Other class ids will be converted to `ignore_index` which equals to `13`.
- `IndoorPatchPointSample`: Crop a patch containing a fixed number of points from input point cloud. `block_size` indicates the size of the cropped block, typically `1.0` for S3DIS.

  - `GlobalRotScaleTrans`: randomly rotate and scale input point cloud.
  - `RandomJitterPoints`: randomly jitter point cloud by adding different noise vector to each point.
  - `RandomDropPointsColor`: set the colors of point cloud to all zeros by a probability `drop_ratio`.

## Metrics

Typically mean intersection over union (mIoU) is used for evaluation on S3DIS. In detail, we first compute IoU for multiple classes and then average them to get mIoU, please refer to [seg_eval.py](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/evaluation/functional/seg_eval.py).

As introduced in section `Export S3DIS data`, S3DIS trains on 5 areas and evaluates on the remaining 1 area. But there are also other area split schemes in different papers.
To enable flexible combination of train-val splits, we use sub-dataset to represent one area, and concatenate them to form a larger training set. An example of training on area 1, 2, 3, 4, 6 and evaluating on area 5 is shown as below:

```python

        ],

```

where we specify the areas used for training/validation by setting `ann_files` and `scene_idxs` with lists that include corresponding paths. The train-val split can be simply modified via changing the `train_area` and `test_area` variables.

<details>

```bibtex
}
```

<details>

```bibtex
}
```

| [S-ViPNAS-MobileNetV3](/configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_vipnas-mbv3_8xb64-210e_coco-wholebody-256x192.py) |  256x192   |  0.619  |  0.700  |  0.477  |  0.608  |  0.585  |  0.689  |  0.386  |  0.505  |  0.473   |  0.578   | [ckpt](https://download.openmmlab.com/mmpose/top_down/vipnas/vipnas_mbv3_coco_wholebody_256x192-0fee581a_20211205.pth) | [log](https://download.openmmlab.com/mmpose/top_down/vipnas/vipnas_mbv3_coco_wholebody_256x192_20211205.log.json) |
| [S-ViPNAS-Res50](/configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_vipnas-res50_8xb64-210e_coco-wholebody-256x192.py) |  256x192   |  0.643  |  0.726  |  0.553  |  0.694  |  0.587  |  0.698  |  0.410  |  0.529  |  0.495   |  0.607   | [ckpt](https://download.openmmlab.com/mmpose/top_down/vipnas/vipnas_res50_wholebody_256x192-49e1c3a4_20211112.pth) | [log](https://download.openmmlab.com/mmpose/top_down/vipnas/vipnas_res50_wholebody_256x192_20211112.log.json) |

# Prompt 模板

## 背景

在语言模型的评测中，我们常会将原始数据集以一定的规则构造成 prompt，以便模型能够按照要求回答问题。

通常，我们会在 prompt 开头放入指令，几个 in-context example（上下文样例），再在最后放入题目。例如：

```text
1+1=?
2
3+9=?
12
5+6=?
```

大量的实验表明，即便测试的原始题目相同，对于 prompt 的不同构造方式会对模型的表现产生影响。可能影响的因素包括：

- Prompt 本身的构成方式，包括指令、in-context example、题目的写法；
- in-context example 的选择，包括了选择的数量和方式；
- 对 prompt 的使用方式。是让模型基于 prompt 进行补全，还是从候选的 prompt 中选择一个最好的作为答案？

OpenCompass 将 prompt 的构建策略定义在了数据集配置中的 `infer_cfg` 部分。一个典型的 `infer_cfg` 如下所示:

```python
    ice_template=dict(  # 用于构造 In Context Example (ice) 的模板
    ),
    prompt_template=dict(  # 用于构造主干 prompt 的模板
    ),
    retriever=dict(type=FixKRetriever),  # 定义 in context example 的获取方式
    inferencer=dict(type=GenInferencer, fix_id_list=[0, 1]),  # 使用何种方式推理得到 prediction

ice_token="</E>"
    ),
)
```

本文档中，我们将会主要介绍 `ice_template`、`prompt_template`、`inferencer` 的定义方法。对于 `retriever` 的介绍请参考其他章节。

我们首先介绍 prompt 的基本语法。

## 字符串式 prompt

字符串式的模板是比较经典的模板形式，考虑下面的模板：

```python
)
```

运行时，花括号`{}`内的字段会被替换成数据样本内的对应字段。如果数据样本中没有对应的字段，则会保持原样输出。

例如我们有一个数据 example 如下:

```python
example = {
    'answer': '2',  # 假设 answer 被写在了 reader_cfg.output_column 中
}
```

则填入模板后的结果为：

```text
Answer:
```

可以看到，问题的实际答案 `answer` 并没有出现在生成的结果中。这是因为 OpenCompass 会遮盖被写在 `reader_cfg.output_column` 中的字段，避免答案泄露。关于 `reader_cfg` 的详细说明，请参考介绍数据集配置的相关文档。

## 对话式 prompt

```text
Answer:
```

在实际的测试中，简单的补全式测试并不能很好地测试出对话式的模型的性能，因此我们更希望 prompt 能以对话的格式输入到模型中。另外，不同的模型对对话的格式定义也不一样，因此我们也需要数据集侧产生的 prompt 更加通用，在测试时再结合具体模型生成符合需求的提示词。

因此，OpenCompass 在字符串式模板之上，增加了对对话式模板的支持。对话式模板更加灵活，它可以结合模型侧不同的 [meta_template](./meta_template.md) 生成不同对话形式的提示词，同时适用于基座和对话模型，但定义也相对复杂。

现在，让我们假设有一个数据样本如下：

```python
example = {
}
```

接下来，我们来展示几个例子：

````{tab} 普通对话
```python
        round=[
        ]
    )
)
```

OpenCompass 把数据填入模板后得到的中间结果为：

```python

dict(role="HUMAN", prompt="Question: {question}"),
        ]
    )
)
```

```python
])
```

````

````{tab} 多轮对话
```python
        round=[
        ]
    )
)
```

```python

```

```python
])
```
````

````{tab} 带 SYSTEM 的对话

```python
        begin=[
        ],
        round=[
        ]
    )
)
```

```python
])
```

])
```

在具体的 meta template 中处理时，如果定义中存在 SYSTEM 角色，则会调用 SYSTEM 的模板进行处理。否则，会调用 fallback_role 角色的模板进行处理，也就是这个例子中的 HUMAN 角色。

````

`````

可以见到，在对话式的模板中，prompt 是以不同角色 `role` 的对话为形式进行组织的。在当前 OpenCompass 的预定义数据集配置中，一个 prompt 中常有的角色有：

- `HUMAN`：人类，通常为提问的一方
- `BOT`：语言模型，通常为回答的一方
- `SYSTEM`：系统，通常用在提示词的开头，负责下达指令。

另外与字符串式的模板不同，经过对话式模板所生成的 prompt 从固定的字符串变成了一个中间结构 PromptList。这个结构会进一步与模型侧的 [meta template](./meta_template.md) 相结合，拼装完成得到最终的提示词。如果不指定 meta template，PromptList 中各项的 prompt 则会直接按行拼接成字符串。

```{note}
上面例子中 PromptList 中的内容并非模型最终的输入，而取决于 meta template 的处理。一个容易产生误解的地方是，在生成式的评测中，最后一个 `BOT` 角色的 prompt `Answer: ` **不会**实际输入到模型。这是由于 API 模型通常并无法自定义模型回复的开头，因此这一设定保持了语言模型与 API 模型在评测上行为的一致。更多信息可以参考 [meta template](./meta_template.md) 的文档。
```

<details>
<summary>点击查看完整参数介绍</summary>

```

<details>

- `begin`，`end` ：(list，可选) prompt 的开头和结尾，通常是一些系统级别的指令。里面的每一项**允许是一个字典或字符串**。

- `round`：(list) 对话的模板格式。列表的每一项**只允许是一个字典**。

每一个字典的参数如下：

- `role`（str）: 参与对话的角色名，用于与 `meta_template` 中的名称进行关联，不会影响实际生成的 prompt。

- `fallback_role` (str) : 缺省角色名，假设 `meta_template` 中找不到 `role`，则会尝试使用 `fallback_role` 进行关联。默认为 `None`

- `prompt` (str) : 角色的对话内容。

## Prompt 模板 与 `inferencer`

在明白了 prompt 模板的基础定义方式后，我们还要根据 `inferencer` 的类型组织 prompt 模板。

OpenCompass 中主要支持了两种 Infernecer：`GenInferencer` 和 `PPLInferencer`，它们对应着两种不同的推理方式。

`GenInferencer` 对应生成式的推理。在推理时，模型被要求以输入的提示词为基准，继续往下续写。此时，`template` 则单一地表示这一句话对应的模板，例如:

````{group-tab} 字符串式模板
```python
)
```
````

````{group-tab} 对话式模板
```python
        begin=[

template='Solve the following questions.\n{question}\n{answer}'
)
```
````

```python
        begin=[
        ],
        round=[
        ]
    )
)
```
````

`````

则模型的推理结果将会是往下续写的字符串。

而 `PPLInferencer` 对应判别式推理。在推理时，模型被要求计算多个输入字符串各自的混淆度 (PerPLexity / ppl)，并将其中 ppl 最小的项作为模型的推理结果。此时 `template` 是一个 `dict`，表示每一句话所对应的模板，例如:

```python

    )
)
```
````

```python
        "A": dict(
            round=[
            ]
        ),
        "B": dict(
            round=[
            ]
        ),
        "C": dict(
            round=[
            ]

round=[
            ]
        ),
            round=[
            ]
        ),
    )
)
```
````

`````

此时模型的推理结果将会是 `template` 的四个 key 之一 ("A" / "B" / "C" / "UNK")

## `ice_template` 与 `prompt_template`

在 OpenCompass 中，对于 0-shot 的评测，我们通常只需要定义 `prompt_template` 字段，即可完成 prompt 的构造。但对于 few shot 的评测，我们还需要定义 `ice_template` 字段，管理上下文学习中样例所对应的 prompt 模板。

`ice_template` 和 `prompt_template` 两者遵循的语法和规则一致，完整 prompt 的构造流程可以使用如下的伪代码进行表示：

```python
```

```python
```

现在，让我们假设有两个训练数据 (ex1, ex2) 和一个测试数据 (ex3):

```python
ex1 = {
}
ex2 = {
}
ex3 = {
}
```

接下来，我们看一下不同的 prompt 构造方法对应的实际效果：

模板配置如下：

```python
    ),
    )
)
```

会得到以下字符串：

```text
2+2=?
4
3+3=?
6

template='Solve the following questions.\n</E>{question}\n{answer}'
    )
)
```

会得到以下字符串：

```text
2+2=?
4
3+3=?
6
1+1=?

```

````

模板配置如下：

```python
            round=[
            ]
        )
    ),
            begin=[
                '</E>',
            ],
            round=[
            ],
        ),
    )
)
```

```python

            ],
        ),
    )
)
```

```python
])
```
````

`````

### 省略式使用方法

值得一提的是，为了简便配置文件，`prompt_template` 这一字段是可被省略的。当 `prompt_template` 字段被省略时，`ice_template` 会同时被作为 `prompt_template`，用于拼装得到完整的 prompt。以下两份 `infer_cfg` 是等价的：

  <thead>
  <tr>
      <th>完整写法</th>
      <th>省略写法</th>
  <tbody>
  <tr>
  <td>

```python
    ),

<td>

```python
    ),
    ),
    # ...
)
```

</td>
  <td>

```python
    ),
    # ...
)
```

</td>
  </tr>
  </thead>
  </table>

更一般地，即便在 0-shot learning 的情况下（即 `retriever` 为 `ZeroRetriver`）时，这一机制依然生效。因此以下配置也是合法的：

```python
    dict(
            ),
        )
    ),
]
```

## 使用建议

template="Q: {question}\nA: {answer}",
            ),
        )
    ),
]
```

## 使用建议

建议使用 [Prompt Viewer](../tools.md) 工具对完成拼装后的 prompt 进行可视化，确认模板是否正确，结果是否符合预期。

# MLP-Mixer

> [MLP-Mixer: An all-MLP Architecture for Vision](https://arxiv.org/abs/2105.01601)

Convolutional Neural Networks (CNNs) are the go-to model for computer vision. Recently, attention-based networks, such as the Vision Transformer, have also become popular. In this paper we show that while convolutions and attention are both sufficient for good performance, neither of them are necessary. We present MLP-Mixer, an architecture based exclusively on multi-layer perceptrons (MLPs). MLP-Mixer contains two types of layers: one with MLPs applied independently to image patches (i.e. "mixing" the per-location features), and one with MLPs applied across patches (i.e. "mixing" spatial information). When trained on large datasets, or with modern regularization schemes, MLP-Mixer attains competitive scores on image classification benchmarks, with pre-training and inference cost comparable to state-of-the-art models. We hope that these results spark further research beyond the realms of well established CNNs and Transformers.

<img src="https://user-images.githubusercontent.com/26739999/143178327-7118b48a-5f5f-4844-a614-a571917384ca.png" width="90%"/>
</div>

```python

predict = inference_model('mlp-mixer-base-p16_3rdparty_64xb64_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('mlp-mixer-base-p16_3rdparty_64xb64_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/mlp_mixer/mlp-mixer-base-p16_64xb64_in1k.py https://download.openmmlab.com/mmclassification/v0/mlp-mixer/mixer-base-p16_3rdparty_64xb64_in1k_20211124-1377e3e0.pth
```

| Model                                        |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                    Config                    |                            Download                             |
| :------------------------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :------------------------------------------: | :-------------------------------------------------------------: |

| `mlp-mixer-base-p16_3rdparty_64xb64_in1k`\*  | From scratch |   59.88    |   12.61   |   76.68   |   92.25   | [config](mlp-mixer-base-p16_64xb64_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/mlp-mixer/mixer-base-p16_3rdparty_64xb64_in1k_20211124-1377e3e0.pth) |
| `mlp-mixer-large-p16_3rdparty_64xb64_in1k`\* | From scratch |   208.20   |   44.57   |   72.34   |   88.02   | [config](mlp-mixer-large-p16_64xb64_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mlp-mixer/mixer-large-p16_3rdparty_64xb64_in1k_20211124-5a2519d2.pth) |

*Models with * are converted from the [timm](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/mlp_mixer.py). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@misc{tolstikhin2021mlpmixer,
      title={MLP-Mixer: An all-MLP Architecture for Vision},

```bibtex
      author={Ilya Tolstikhin and Neil Houlsby and Alexander Kolesnikov and Lucas Beyer and Xiaohua Zhai and Thomas Unterthiner and Jessica Yung and Andreas Steiner and Daniel Keysers and Jakob Uszkoreit and Mario Lucic and Alexey Dosovitskiy},
      eprint={2105.01601},
}
```

# CMake Build Option Spec

<thead>
  <tr>
    <th>VALUE</th>
    <th>DEFAULT</th>
    <th>REMARK</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>Switch to build shared libs</td>
  </tr>
  <tr>
    <td>Switch to build MMDeploy SDK</td>
  </tr>

<tr>
  <td>Build single lib</td>
  </tr>

<tr>
  <td>Build unittest</td>
  </tr>

<tr>
    <td>Switch to build MMDeploy SDK python package</td>
  </tr>

<tr>
    <td>Build C# SDK API</td>
  </tr>

<tr>

</tr>

<tr>
  </tr>

<tr>
    <td>Build Java SDK API</td>
  </tr>
  <tr>
    <td>MMDEPLOY_BUILD_TEST</td>
    <td>Switch to build MMDeploy SDK unittest cases</td>
  </tr>

<tr>
    <td>Build with spdlog installation package that comes with the system</td>
  </tr>

<tr>
    <td>Enable SDK with zip format</td>
  </tr>

<tr>
    <td>Build for cplus code coverage report</td>
  </tr>

<tr>
    <td>Enable target device. You can enable more by

<td>Build for cplus code coverage report</td>
  </tr>

<tr>
   passing a semicolon separated list of device names to <code>MMDEPLOY_TARGET_DEVICES</code> variable, e.g. <code>-DMMDEPLOY_TARGET_DEVICES="cpu;cuda"</code> </td>
  </tr>
  <tr>
    <td>{"trt", "ort", "pplnn", "ncnn", "openvino", "torchscript", "snpe", "tvm"}</td>
    <td>Enabling inference engine. <b>By default, no target inference engine is set, since it highly depends on the use case.</b> When more than one engine are specified, it has to be set with a semicolon separated list of inference backend names, e.g. <pre><code>-DMMDEPLOY_TARGET_BACKENDS="trt;ort;pplnn;ncnn;openvino"</code></pre>
    After specifying the inference engine, it's package path has to be passed to cmake as follows, <br>

After specifying the inference engine, it's package path has to be passed to cmake as follows, <br>
    1. <b>trt</b>: TensorRT. <code>TENSORRT_DIR</code> and <code>CUDNN_DIR</code> are needed.
-DTENSORRT_DIR=${TENSORRT_DIR}
-DCUDNN_DIR=${CUDNN_DIR}
    2. <b>ort</b>: ONNXRuntime. <code>ONNXRUNTIME_DIR</code> is needed.
<pre><code>-DONNXRUNTIME_DIR=${ONNXRUNTIME_DIR}</code></pre>
    3. <b>pplnn</b>: PPL.NN. <code>pplnn_DIR</code> is needed.
<pre><code>-Dpplnn_DIR=${PPLNN_DIR}</code></pre>
    4. <b>ncnn</b>: ncnn. <code>ncnn_DIR</code> is needed.
<pre><code>-Dncnn_DIR=${NCNN_DIR}/build/install/lib/cmake/ncnn</code></pre>
    5. <b>openvino</b>: OpenVINO. <code>InferenceEngine_DIR</code> is needed.
<pre><code>-DInferenceEngine_DIR=${INTEL_OPENVINO_DIR}/deployment_tools/inference_engine/share</code></pre>
    6. <b>torchscript</b>: TorchScript. <code>Torch_DIR</code> is needed.
<pre><code>-DTorch_DIR=${Torch_DIR}</code></pre>

6. <b>torchscript</b>: TorchScript. <code>Torch_DIR</code> is needed.
    7. <b>snpe</b>: qcom snpe. <code>SNPE_ROOT</code> must existed in the environment variable because of C/S mode.<br>
    8. <b>coreml</b>: CoreML. <code>Torch_DIR</code> is required.
    9. <b>TVM</b>: TVM. <code>TVM_DIR</code> is required. <pre><code>-DTVM_DIR=${TVM_DIR}</code></pre>
   </td>
  </tr>
  <tr>
    <td>Enable codebase's postprocess modules. You can provide a semicolon separated list of codebase names to enable them, e.g., <code>-DMMDEPLOY_CODEBASES="mmpretrain;mmdet"</code>. Or you can pass <code>all</code> to enable them all, i.e., <code>-DMMDEPLOY_CODEBASES=all</code></td>
  </tr>

</tbody>
</table>

# LeViT

> [LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference](https://arxiv.org/abs/2104.01136)

We design a family of image classification architectures that optimize the trade-off between accuracy and efficiency in a high-speed regime. Our work exploits recent findings in attention-based architectures, which are competitive on highly parallel processing hardware. We revisit principles from the extensive literature on convolutional neural networks to apply them to transformers, in particular activation maps with decreasing resolutions. We also introduce the attention bias, a new way to integrate positional information in vision transformers. As a result, we propose LeVIT: a hybrid neural network for fast inference image classification. We consider different measures of efficiency on different hardware platforms, so as to best reflect a wide range of application scenarios. Our extensive experiments empirically validate our technical choices and show they are suitable to most architectures. Overall, LeViT significantly outperforms existing convnets and vision transformers with respect to the speed/accuracy tradeoff. For example, at 80% ImageNet top-1 accuracy, LeViT is 5 times faster than EfficientNet on CPU.

<img src="https://raw.githubusercontent.com/facebookresearch/LeViT/main/.github/levit.png" width="90%"/>
</div>

```python

predict = inference_model('levit-128s_3rdparty_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('levit-128s_3rdparty_in1k', pretrained=True)
```

Test:

```shell

```

Test:

```shell
python tools/test.py configs/levit/levit-128s_8xb256_in1k.py https://download.openmmlab.com/mmclassification/v0/levit/levit-128s_3rdparty_in1k_20230117-e9fbd209.pth
```

| Model                        |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |               Config                |                                         Download                                         |
| :--------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :---------------------------------: | :--------------------------------------------------------------------------------------: |

| `levit-128s_3rdparty_in1k`\* | From scratch |    7.39    |   0.31    |   76.51   |   92.90   | [config](levit-128s_8xb256_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/levit/levit-128s_3rdparty_in1k_20230117-e9fbd209.pth) |
| `levit-128_3rdparty_in1k`\*  | From scratch |    8.83    |   0.41    |   78.58   |   93.95   | [config](levit-128_8xb256_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/levit/levit-128_3rdparty_in1k_20230117-3be02a02.pth) |
| `levit-192_3rdparty_in1k`\*  | From scratch |   10.56    |   0.67    |   79.86   |   94.75   | [config](levit-192_8xb256_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/levit/levit-192_3rdparty_in1k_20230117-8217a0f9.pth) |
| `levit-256_3rdparty_in1k`\*  | From scratch |   18.38    |   1.14    |   81.59   |   95.46   | [config](levit-256_8xb256_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/levit/levit-256_3rdparty_in1k_20230117-5ae2ce7d.pth) |

| `levit-384_3rdparty_in1k`\*  | From scratch |   38.36    |   2.37    |   82.59   |   95.95   | [config](levit-384_8xb256_in1k.py)  | [model](https://download.openmmlab.com/mmclassification/v0/levit/levit-384_3rdparty_in1k_20230117-f3539cce.pth) |

*Models with * are converted from the [official repo](https://github.com/facebookresearch/LeViT). The config files of these models are only for inference. We haven't reproduce the training results.*

```bibtex
@InProceedings{Graham_2021_ICCV,
    author    = {Graham, Benjamin and El-Nouby, Alaaeldin and Touvron, Hugo and Stock, Pierre and Joulin, Armand and Jegou, Herve and Douze, Matthijs},
    title     = {LeViT: A Vision Transformer in ConvNet's Clothing for Faster Inference},
    pages     = {12259-12269}
}
```

# Semi-supervised Object Detection

Semi-supervised object detection uses both labeled data and unlabeled data for training. It not only reduces the annotation burden for training high-performance object detectors but also further improves the object detector by using a large number of unlabeled data.

A typical procedure to train a semi-supervised object detector is as below:

- [Prepare and split dataset](#Prepare-and-split-dataset)
- [Configure multi-branch pipeline](#Configure-multi-branch-pipeline)
- [Configure semi-supervised dataloader](#Configure-semi-supervised-dataloader)
- [Configure semi-supervised model](#Configure-semi-supervised-model)
- [Configure MeanTeacherHook](#Configure-MeanTeacherHook)
- [Configure TeacherStudentValLoop](#Configure-TeacherStudentValLoop)

## Prepare and split dataset

We provide a dataset download script, which downloads the coco2017 dataset by default and decompresses it automatically.

```shell
```

```shell
```

The decompressed dataset directory structure is as below:

```plain
├── data
│   ├── coco
```

There are two common experimental settings for semi-supervised object detection on the coco2017 dataset:

```

(1) Split `train2017` according to a fixed percentage (1%, 2%, 5% and 10%) as a labeled dataset, and the rest of `train2017` as an unlabeled dataset. Because the different splits of `train2017` as labeled datasets will cause significant fluctuation on the accuracy of the semi-supervised detectors, five-fold cross-validation is used in practice to evaluate the algorithm. We provide the dataset split script:

```shell
```

By default, the script will split `train2017` according to the labeled data ratio  1%, 2%, 5% and 10%, and each split will be randomly repeated 5 times for cross-validation. The generated semi-supervised annotation file name format is as below:

- the name format of labeled dataset: `instances_train2017.{fold}@{percent}.json`

- the name format of unlabeled dataset: `instances_train2017.{fold}@{percent}-unlabeled.json`

Here, `fold` is used for cross-validation, and `percent` represents the ratio of labeled data. The directory structure of the divided dataset is as below:

```plain
├── data
│   ├── coco

```

(2) Use `train2017` as the labeled dataset and `unlabeled2017` as the unlabeled dataset. Since `image_info_unlabeled2017.json` does not contain `categories` information, the `CocoDataset` cannot be initialized, so you need to write the `categories` of `instances_train2017.json` into `image_info_unlabeled2017.json` and save it as `instances_unlabeled2017.json`, the relevant script is as below:

```python

```

```

The processed dataset directory is as below:

```plain
├── data
│   ├── coco
```

## Configure multi-branch pipeline

There are two main approaches to semi-supervised learning,
[consistency regularization](https://research.nvidia.com/sites/default/files/publications/laine2017iclr_paper.pdf)

and [pseudo label](https://www.researchgate.net/profile/Dong-Hyun-Lee/publication/280581078_Pseudo-Label_The_Simple_and_Efficient_Semi-Supervised_Learning_Method_for_Deep_Neural_Networks/links/55bc4ada08ae092e9660b776/Pseudo-Label-The-Simple-and-Efficient-Semi-Supervised-Learning-Method-for-Deep-Neural-Networks.pdf).
Consistency regularization often requires some careful design, while pseudo label have a simpler form and are easier to extend to downstream tasks.
We adopt a teacher-student joint training semi-supervised object detection framework based on pseudo label, so labeled data and unlabeled data need to configure different data pipeline:

(1) Pipeline for labeled data：

```python

```python
    dict(type='LoadImageFromFile', backend_args=backend_args),
]
```

(2) Pipeline for unlabeled data：

```python
    dict(

    dict(
]

    dict(
        ]),
    dict(

dict(
]

    dict(
    )
]
```

## Configure semi-supervised dataloader

(1) Build a semi-supervised dataset. Use `ConcatDataset` to concatenate labeled and unlabeled datasets.

```python

data_prefix=dict(img='train2017/'),

```

batch_size=batch_size,
```

(2) Use multi-source dataset sampler. Use `GroupMultiSourceSampler` to sample data form batches from `labeled_dataset` and `labeled_dataset`, `source_ratio` controls the proportion of labeled data and unlabeled data in the batch. `GroupMultiSourceSampler` also ensures that the images in the same batch have similar aspect ratios. If you don't need to guarantee the aspect ratio of the images in the batch, you can use `MultiSourceSampler`. The sampling diagram of `GroupMultiSourceSampler` is as below:

</div>

</div>

`sup=1000` indicates that the scale of the labeled dataset is 1000, `sup_h=200` indicates that the scale of the images with an aspect ratio greater than or equal to 1 in the labeled dataset is 200, and `sup_w=800` indicates that the scale of the images with an aspect ratio less than 1 in the labeled dataset is 800,
`unsup=9000` indicates that the scale of the unlabeled dataset is 9000, `unsup_h=1800` indicates that the scale of the images with an aspect ratio greater than or equal to 1 in the unlabeled dataset is 1800, and `unsup_w=7200` indicates the scale of the images with an aspect ratio less than 1 in the unlabeled dataset is 7200.

`GroupMultiSourceSampler` randomly selects a group according to the overall aspect ratio distribution of the images in the labeled dataset and the unlabeled dataset, and then sample data to form batches from the two datasets according to `source_ratio`, so labeled datasets and unlabeled datasets have different repetitions.

## Configure semi-supervised model

We choose `Faster R-CNN` as `detector` for semi-supervised training. Take the semi-supervised object detection algorithm `SoftTeacher` as an example,
the model configuration can be inherited from `_base_/models/faster-rcnn_r50_fpn.py`, replacing the backbone network of the detector with `caffe` style.
Note that unlike the supervised training configs, `Faster R-CNN` as `detector` is an attribute of `model`, not `model` .
In addition, `data_preprocessor` needs to be set to `MultiBranchDataPreprocessor`, which is used to pad and normalize images from different pipelines.

Finally, parameters required for semi-supervised training and testing can be configured via `semi_train_cfg` and `semi_test_cfg`.

```python
_base_ = [
]

    depth=50,

norm_eval=True,

```

In addition, we also support semi-supervised training for other detection models, such as `RetinaNet` and `Cascade R-CNN`. Since `SoftTeacher` only supports `Faster R-CNN`, it needs to be replaced with `SemiBaseDetector`, example is as below:

```python
_base_ = [

```python
_base_ = [
]

```

Following the semi-supervised training configuration of `SoftTeacher`, change `batch_size` to 2 and `source_ratio` to `[1, 1]`, the experimental results of supervised and semi-supervised training of `RetinaNet`, `Faster R-CNN`, `Cascade R-CNN` and `SoftTeacher` on the 10% coco `train2017` are as below:

## Configure MeanTeacherHook

Usually, the teacher model is updated by Exponential Moving Average (EMA) the student model, and then the teacher model is optimized with the optimization of the student model, which can be achieved by configuring `custom_hooks`:

```python
```

## Configure TeacherStudentValLoop

```python
```

Since there are two models in the teacher-student joint training framework, we can replace `ValLoop` with `TeacherStudentValLoop` to test the accuracy of both models during the training process.

```python
```

}
```

<details>

```bibtex
```

| [pose_hrnet_w32-coco](configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192.py) |  256x192   | 0.252 |      0.397      |      0.255      | 0.321 |      0.485      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192-81c58e40_20220909.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192_20220909.log) |

| [pose_hrnet_w32-humanart-coco](configs/body_2d_keypoint/topdown_heatmap/humanart/td-hm_hrnet-w32_8xb64-210e_humanart-256x192.py) |  256x192   | 0.399 |      0.545      |      0.420      | 0.466 |      0.613      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_hrnet-w32_8xb64-210e_humanart-256x192-0773ef0b_20230614.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_hrnet-w32_8xb64-210e_humanart-256x192-0773ef0b_20230614.json) |

| [pose_hrnet_w48-coco](configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_8xb32-210e_coco-256x192.py) |  256x192   | 0.271 |      0.413      |      0.277      | 0.339 |      0.499      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_8xb32-210e_coco-256x192-0e67c616_20220913.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_8xb32-210e_coco-256x192_20220913.log) |

| [pose_hrnet_w48-humanart-coco](configs/body_2d_keypoint/topdown_heatmap/humanart/td-hm_hrnet-w48_8xb32-210e_humanart-256x192.py) |  256x192   | 0.417 |      0.553      |      0.442      | 0.481 |      0.617      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_hrnet-w48_8xb32-210e_humanart-256x192-05178983_20230614.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_hrnet-w48_8xb32-210e_humanart-256x192-05178983_20230614.json) |

| [pose_hrnet_w32-coco](configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192.py) |  256x192   | 0.533 |      0.771      |      0.562      | 0.574 |      0.792      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192-81c58e40_20220909.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192_20220909.log) |

| [pose_hrnet_w32-humanart-coco](configs/body_2d_keypoint/topdown_heatmap/humanart/td-hm_hrnet-w32_8xb64-210e_humanart-256x192.py) |  256x192   | 0.754 |      0.906      |      0.812      | 0.783 |      0.916      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_hrnet-w32_8xb64-210e_humanart-256x192-0773ef0b_20230614.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_hrnet-w32_8xb64-210e_humanart-256x192-0773ef0b_20230614.json) |

| [pose_hrnet_w48-coco](configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_8xb32-210e_coco-256x192.py) |  256x192   | 0.557 |      0.782      |      0.593      | 0.595 |      0.804      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_8xb32-210e_coco-256x192-0e67c616_20220913.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_8xb32-210e_coco-256x192_20220913.log) |

| [pose_hrnet_w48-humanart-coco](configs/body_2d_keypoint/topdown_heatmap/humanart/td-hm_hrnet-w48_8xb32-210e_humanart-256x192.py) |  256x192   | 0.769 |      0.906      |      0.825      | 0.796 |      0.919      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_hrnet-w48_8xb32-210e_humanart-256x192-05178983_20230614.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_hrnet-w48_8xb32-210e_humanart-256x192-05178983_20230614.json) |

| [pose_hrnet_w32-coco](configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192.py) |  256x192   | 0.749 |      0.906      |      0.821      | 0.804 |      0.945      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192-81c58e40_20220909.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w32_8xb64-210e_coco-256x192_20220909.log) |

| [pose_hrnet_w32-humanart-coco](configs/body_2d_keypoint/topdown_heatmap/humanart/td-hm_hrnet-w32_8xb64-210e_humanart-256x192.py) |  256x192   | 0.741 |      0.902      |      0.814      | 0.795 |      0.941      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_hrnet-w32_8xb64-210e_humanart-256x192-0773ef0b_20230614.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_hrnet-w32_8xb64-210e_humanart-256x192-0773ef0b_20230614.json) |

| [pose_hrnet_w48-coco](configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_8xb32-210e_coco-256x192.py) |  256x192   | 0.756 |      0.908      |      0.826      | 0.809 |      0.945      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_8xb32-210e_coco-256x192-0e67c616_20220913.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_hrnet-w48_8xb32-210e_coco-256x192_20220913.log) |

| [pose_hrnet_w48-humanart-coco](configs/body_2d_keypoint/topdown_heatmap/humanart/td-hm_hrnet-w48_8xb32-210e_humanart-256x192.py) |  256x192   | 0.751 |      0.905      |      0.822      | 0.805 |      0.943      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_hrnet-w48_8xb32-210e_humanart-256x192-05178983_20230614.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/human_art/td-hm_hrnet-w48_8xb32-210e_humanart-256x192-05178983_20230614.json) |

# 源码手动安装

如果网络良好，我们建议使用 [docker](build_from_docker.md) 或 [一键式脚本](build_from_script.md) 方式。

## 下载

```bash
```

### FAQ

- 如果由于网络等原因导致拉取仓库子模块失败，可以尝试通过如下指令手动再次安装子模块:

  ```bash

  # 返回至 third_party 目录, 克隆 pybind11
  cd ..

  cd ..
  cd spdlog
  ```

- 如果以 `SSH` 方式 `git clone` 代码失败，您可以尝试使用 `HTTPS` 协议下载代码：

  ```bash
  git clone -b main https://github.com/open-mmlab/mmdeploy.git MMDeploy
  cd MMDeploy
  git submodule update --init --recursive
  ```

## 编译

根据您的目标平台，点击如下对应的链接，按照说明编译 MMDeploy

```

## 编译

- [Qcom SNPE](snpe.md)

# 准备 Vimeo90K 数据集

```bibtex
}
```

将数据集路径 `vimeo_septuplet/sequences` 重命名为 `vimeo90k/GT`。Vimeo90K 数据集包含了如下所示的 `clip/sequence/img` 目录结构：

```text
vimeo90k
├── GT
│   ├── 00001
│   │   ├── 0001
│   │   │   ├── ...
│   │   ├── 0002
│   │   ├── 0003
│   │   ├── ...
│   ├── 00002
│   ├── ...
```

为了生成下采样图像BIx4和BDx4，以及准备所需的标注文件，需要执行如下命令：

```shell
```

```text
mmagic
├── mmagic
├── tools

```shell
```

```text
mmagic
├── mmagic
├── tools
├── data
│   │   ├── GT
│   │   │   ├── ...
│   │   ├── BIx4
│   │   ├── BDx4
```

## 准备 LMDB 格式的 Vimeo90K 数据集

```shell
```

MMPose 1.x unifies the interfaces of dataset, models, evaluation, and visualization with faster training and testing speed.
It also provide a general semi-supervised object detection framework, and more strong baselines.

# Quick Run

This page provides a basic tutorial about the usage of MMPose.

We will walk you through the 7 key steps of a typical MMPose workflow by training a top-down residual log-likelihood algorithm based on resnet50 on COCO dataset:

1. Inference with a pretrained model
2. Prepare the dataset
4. Browse the transformed images
5. Training
6. Testing
7. Visualization

For installation instructions, please refer to [Installation](./installation.md).

### Inference with a pretrained model

We provide a useful script to perform pose estimation with a pretrained model:

```Bash
```

https://download.openmmlab.com/mmpose/top_down/deeppose/deeppose_res50_coco_256x192_rle-2ea9bb4a_20220616.pth
```

If MMPose is properly installed, you will get the visualized result as follows:

```{note}
More demo and full instructions can be found in [Inference](./user_guides/inference.md).
```

### Prepare the dataset

MMPose supports multiple tasks. We provide the corresponding guidelines for data preparation.

- [2D Body Keypoint Detection](./dataset_zoo/2d_body_keypoint.md)

- [3D Body Keypoint Detection](./dataset_zoo/3d_body_keypoint.md)

- [2D Hand Keypoint Detection](./dataset_zoo/2d_hand_keypoint.md)

- [3D Hand Keypoint Detection](./dataset_zoo/3d_hand_keypoint.md)

- [2D Face Keypoint Detection](./dataset_zoo/2d_face_keypoint.md)

- [2D WholeBody Keypoint Detection](./dataset_zoo/2d_wholebody_keypoint.md)

- [2D Fashion Landmark Detection](./dataset_zoo/2d_fashion_landmark.md)

- [2D Animal Keypoint Detection](./dataset_zoo/2d_animal_keypoint.md)

You can refer to \[2D Body Keypoint Detection\] > \[COCO\] for COCO dataset preparation.

```{note}
In MMPose, we suggest placing the data under `$MMPOSE/data`.
```

MMPose is equipped with a powerful config system to conduct various experiments conveniently. A config file organizes the settings of:

We provide a bunch of well-prepared configs under `$MMPOSE/configs` so that you can directly use or modify.

Going back to our example, we  will use the prepared config:

```Bash
```

You can set the path of the COCO dataset by modifying `data_root` in the config：

```Python
```

```{note}
If you wish to learn more about our config system, please refer to [Configs](./user_guides/configs.md).
```

### Browse the transformed images

Before training, we can browse the transformed training data to check if the images are augmented properly:

```Bash

```

```Bash
```

Use the following command to train with a single GPU:

```Bash
```

```{note}
MMPose automates many useful training tricks and functions including:

- Learning rate warmup and scheduling

- ImageNet pretrained models

- Automatic learning rate scaling

- Multi-GPU and Multi-Node training support

- Various Data backend support, e.g. HardDisk, LMDB, Petrel, HTTP etc.

- Mixed precision training support

```

### Testing

Checkpoints and logs will be saved under `$MMPOSE/work_dirs` by default. The best model is under `$MMPOSE/work_dir/best_coco`.

Use the following command to evaluate the model on COCO dataset:

```Bash
```

Here is an example of evaluation results：

```Bash

Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets= 20 ] =  0.777
```

```{note}

```

```{note}
If you want to perform evaluation on other datasets, please refer to [Train & Test](./user_guides/train_and_test.md).
```

In addition to the visualization of the keypoint skeleton, MMPose also supports the visualization of Heatmaps by setting `output_heatmap=True` in confg:

```Python
    ## omitted
        ## omitted
    )
)
```

or add `--cfg-options='model.test_cfg.output_heatmaps=True'` at the end of your command.

Visualization result (top: decoded keypoints; bottom: predicted heatmap):

```{note}
If you wish to apply MMPose to your own projects, we have prepared a detailed [Migration guide](./migration.md).
```

<details>

```bibtex
}
```

<details>

```bibtex

author={Sagonas, Christos and Antonakos, Epameinondas and Tzimiropoulos, Georgios and Zafeiriou, Stefanos and Pantic, Maja},
}
```

The model is trained on 300W train.

| Arch                               | Input Size | NME<sub>*common*</sub> | NME<sub>*challenge*</sub> | NME<sub>*full*</sub> | NME<sub>*test*</sub> |                ckpt                 |                log                 |
| :--------------------------------- | :--------: | :--------------------: | :-----------------------: | :------------------: | :------------------: | :---------------------------------: | :--------------------------------: |

| [pose_hrnetv2_w18](/configs/face_2d_keypoint/topdown_heatmap/300w/td-hm_hrnetv2-w18_8xb64-60e_300w-256x256.py) |  256x256   |          2.92          |           5.64            |         3.45         |         4.10         | [ckpt](https://download.openmmlab.com/mmpose/face/hrnetv2/hrnetv2_w18_300w_256x256-eea53406_20211019.pth) | [log](https://download.openmmlab.com/mmpose/face/hrnetv2/hrnetv2_w18_300w_256x256_20211019.log.json) |

## BDD100K

  ```bash
  ```

```none
├── mmseg
├── tools
├── data
│   │   │   └── 10k

# 数据结构

为了统一模型和各功能模块之间的输入和输出的接口, 在 OpenMMLab 2.0 MMEngine 中定义了一套抽象数据结构, 实现了基础的增/删/查/改功能, 支持不同设备间的数据迁移, 也支持了如
`.cpu()`, `.cuda()`, `.get()` 和 `.detach()` 的类字典和张量的操作。具体可以参考 [MMEngine 文档](https://github.com/open-mmlab/mmengine/blob/main/docs/en/advanced_tutorials/data_element.md)。

同样的, MMSegmentation 亦遵循了 OpenMMLab 2.0 各模块间的接口协议, 定义了 `SegDataSample` 用来封装语义分割任务所需要的数据。

## 语义分割数据 SegDataSample

[SegDataSample](mmseg.structures.SegDataSample) 包括了三个主要数据字段 `gt_sem_seg`, `pred_sem_seg` 和 `seg_logits`, 分别用来存放标注信息, 预测结果和预测的未归一化前的 logits 值。

| 字段           | 类型                      | 描述                            |
| -------------- | ------------------------- | ------------------------------- |
| gt_sem_seg     | [`PixelData`](#pixeldata) | 图像标注信息.                   |
| pred_instances | [`PixelData`](#pixeldata) | 图像预测结果.                   |
| seg_logits     | [`PixelData`](#pixeldata) | 模型预测未归一化前的 logits 值. |

以下示例代码展示了 `SegDataSample` 的使用方法：

```python

```python

# 定义 gt_segmentations 用于封装模型的输出信息

# 增加和处理 SegDataSample　中的属性
assert 'data' in data_sample.gt_sem_seg
'''
(4, 4)
'''
'''

gt_sem_seg: <PixelData(

'''

# 删除和修改 SegDataSample　中的属性

# 类张量的操作

```

## 在 SegDataSample 中自定义新的属性

如果你想在 `SegDataSample` 中自定义新的属性，你可以参考下面的 [SegDataSample](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/structures/seg_data_sample.py) 示例:

```python
    ...

    @property

```

这样一个新的属性 `xxx_property` 就将被增加到 `SegDataSample` 里面了。

# 模型库统计数据

- 论文数量: 47

- 模型数量: 612

  - \[ALGORITHM\] [ANN](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/ann) (16 ckpts)

  - \[ALGORITHM\] [APCNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/apcnet) (12 ckpts)

  - \[BACKBONE\] [BEiT](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/beit) (2 ckpts)

  - \[ALGORITHM\] [BiSeNetV1](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/bisenetv1) (11 ckpts)

  - \[ALGORITHM\] [BiSeNetV2](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/bisenetv2) (4 ckpts)

  - \[ALGORITHM\] [CCNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/ccnet) (16 ckpts)

  - \[ALGORITHM\] [CGNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/cgnet) (2 ckpts)

  - \[BACKBONE\] [ConvNeXt](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/convnext) (6 ckpts)

- \[BACKBONE\] [ConvNeXt](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/convnext) (6 ckpts)

  - \[ALGORITHM\] [DANet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/danet) (16 ckpts)

  - \[ALGORITHM\] [DeepLabV3](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/deeplabv3) (41 ckpts)

  - \[ALGORITHM\] [DeepLabV3+](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/deeplabv3plus) (42 ckpts)

  - \[ALGORITHM\] [DMNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/dmnet) (12 ckpts)

  - \[ALGORITHM\] [DNLNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/dnlnet) (12 ckpts)

  - \[ALGORITHM\] [DPT](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/dpt) (1 ckpts)

  - \[ALGORITHM\] [EMANet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/emanet) (4 ckpts)

- \[ALGORITHM\] [EMANet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/emanet) (4 ckpts)

  - \[ALGORITHM\] [EncNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/encnet) (12 ckpts)

  - \[ALGORITHM\] [ERFNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/erfnet) (1 ckpts)

  - \[ALGORITHM\] [FastFCN](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/fastfcn) (12 ckpts)

  - \[ALGORITHM\] [Fast-SCNN](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/fastscnn) (1 ckpts)

  - \[ALGORITHM\] [FCN](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/fcn) (41 ckpts)

  - \[ALGORITHM\] [GCNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/gcnet) (16 ckpts)

  - \[BACKBONE\] [HRNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/hrnet) (37 ckpts)

  - \[ALGORITHM\] [ICNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/icnet) (12 ckpts)

- \[ALGORITHM\] [ICNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/icnet) (12 ckpts)

  - \[ALGORITHM\] [ISANet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/isanet) (16 ckpts)

  - \[ALGORITHM\] [K-Net](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/knet) (7 ckpts)

  - \[BACKBONE\] [MAE](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/mae) (1 ckpts)

  - \[ALGORITHM\] [Mask2Former](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/mask2former) (13 ckpts)

  - \[ALGORITHM\] [MaskFormer](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/maskformer) (4 ckpts)

  - \[BACKBONE\] [MobileNetV2](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/mobilenet_v2) (8 ckpts)

  - \[BACKBONE\] [MobileNetV3](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/mobilenet_v3) (4 ckpts)

- \[BACKBONE\] [MobileNetV3](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/mobilenet_v3) (4 ckpts)

  - \[ALGORITHM\] [NonLocal Net](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/nonlocal_net) (16 ckpts)

  - \[ALGORITHM\] [OCRNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/ocrnet) (24 ckpts)

  - \[ALGORITHM\] [PointRend](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/point_rend) (4 ckpts)

  - \[BACKBONE\] [PoolFormer](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/poolformer) (5 ckpts)

  - \[ALGORITHM\] [PSANet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/psanet) (16 ckpts)

  - \[ALGORITHM\] [PSPNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/pspnet) (54 ckpts)

  - \[BACKBONE\] [ResNeSt](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/resnest) (8 ckpts)

- \[BACKBONE\] [ResNeSt](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/resnest) (8 ckpts)

  - \[ALGORITHM\] [SegFormer](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/segformer) (13 ckpts)

  - \[ALGORITHM\] [Segmenter](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/segmenter) (5 ckpts)

  - \[ALGORITHM\] [Semantic FPN](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/sem_fpn) (4 ckpts)

  - \[ALGORITHM\] [SETR](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/setr) (7 ckpts)

  - \[ALGORITHM\] [STDC](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/stdc) (4 ckpts)

  - \[BACKBONE\] [Swin Transformer](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/swin) (6 ckpts)

  - \[BACKBONE\] [Twins](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/twins) (12 ckpts)

- \[BACKBONE\] [Twins](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/twins) (12 ckpts)

  - \[ALGORITHM\] [UNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/unet) (25 ckpts)

  - \[ALGORITHM\] [UPerNet](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/upernet) (16 ckpts)

  - \[BACKBONE\] [Vision Transformer](https://github.com/open-mmlab/mmsegmentation/blob/master/configs/vit) (11 ckpts)

This is an example README for community `projects/`. We have provided detailed explanations for each field in the form of html comments, which are visible when you read the source of this README file. If you wish to submit your project to our main repository, then all the fields in this README are mandatory for others to understand what you have achieved in this implementation. For more details, read our [contribution guide](https://mmrotate.readthedocs.io/en/1.x/notes/contribution_guide.html) or approach us in [Discussions](https://github.com/open-mmlab/mmrotate/discussions).

## Usage

## Usage

In MMRotate's root directory, run the following command to train the model:

```bash
python tools/train.py projects/example_project/configs/r3det-oc_dummy-resnet_fpn_1x_dota.py
```

In MMRotate's root directory, run the following command to test the model:

```bash
python tools/test.py projects/example_project/confsigs/r3det-oc_dummy-resnet_fpn_1x_dota.py ${CHECKPOINT_PATH}
```

## Results

```bash
```

## Results

<!-- List the results as usually done in other model's README. [Example](https://github.com/open-mmlab/mmrotate/blob/1.x/configs/r3det/README.md#results-and-models)

|         Backbone         |  mAP  | Angle | lr schd | Mem (GB) | Inf Time (fps) | Aug | Batch Size |                                      Configs                                       |         Download         |
| :----------------------: | :---: | :---: | :-----: | :------: | :------------: | :-: | :--------: | :--------------------------------------------------------------------------------: | :----------------------: |

| ResNet50 (1024,1024,200) | 69.80 |  oc   |   1x    |   3.54   |      12.4      |  -  |     2      | [r3det-oc_dummy-resnet_fpn_1x_dota](confsigs/r3det-oc_dummy-resnet_fpn_1x_dota.py) | [model](<>) \| [log](<>) |

```bibtex
}
```

Zhang, Wenwei and Chen, Kai},
}
```

    <!-- The code's design shall follow existing interfaces and convention. For example, each model component should be registered into `mmrotate.registry.MODELS` and configurable via a config file. -->

    <!-- Ideally *all* the methods should have [type hints](https://www.pythontutorial.net/python-basics/python-type-hints/) and [docstrings](https://google.github.io/styleguide/pyguide.html#381-docstrings). [Example](https://github.com/open-mmlab/mmrotate/blob/766185ed317f99379cb14035a6f9e5cf8a5340ad/mmrotate/structures/bbox/box_converters.py#L61-L78) -->

    <!-- Unit tests for each module are required. [Example](https://github.com/open-mmlab/mmrotate/blob/766185ed317f99379cb14035a6f9e5cf8a5340ad/tests/test_structures/test_bbox/test_box_converters.py#L43-L52) -->

    <!-- It will be parsed by MIM and Inferencer. [Example](https://github.com/open-mmlab/mmrotate/blob/1.x/configs/r3det/metafile.yml) -->

  <!-- In particular, you may have to refactor this README into a standard one. [Example](https://github.com/open-mmlab/mmrotate/blob/1.x/configs/r3det/README.md) -->

# MMAction2 模型部署

- [MMAction2 模型部署](#mmaction2-模型部署)
    - [安装 mmaction2](#安装-mmaction2)
    - [视频分类任务模型转换](#视频分类任务模型转换)
      - [视频分类 SDK 模型推理](#视频分类-sdk-模型推理)

[MMAction2](https://github.com/open-mmlab/mmaction2)是一款基于 PyTorch 的视频理解开源工具箱，是[OpenMMLab](https://openmmlab.com)项目的成员之一。

## 安装

### 安装 mmaction2

请参考[官网安装指南](https://github.com/open-mmlab/mmaction2/tree/main#installation).

```shell

```shell
```

## 模型转换

你可以使用 [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/tree/main/tools/deploy.py) 把 mmaction2 模型一键式转换为推理后端模型。

转换的关键之一是使用正确的配置文件。项目中已内置了各后端部署[配置文件](https://github.com/open-mmlab/mmdeploy/tree/main/configs/mmaction)。
文件的命名模式是：

```
```

文件的命名模式是：

```
```

其中：

- **{task}:** mmaction2 中的任务
- **{2d/3d}:** 表示模型的类别

以下，我们将演示如何把视频分类任务中 `tsn` 模型转换为 onnx 模型。

### 视频分类任务模型转换

```shell

    --show \
```

## 模型规范

上例中的`mmdeploy_models/mmaction/tsn/ort`，结构如下：

```
```

重要的是：

## 模型推理

### 后端模型推理

```python

## 模型推理

### 后端模型推理

```python

```

#### 视频分类 SDK 模型推理

```python
import cv2

#  ...

```

```

> mmaction2 的 C#，Java接口待开发

## 模型支持列表

# LSGAN (ICCV'2017)

> [Least Squares Generative Adversarial Networks](https://openaccess.thecvf.com/content_iccv_2017/html/Mao_Least_Squares_Generative_ICCV_2017_paper.html)

Unsupervised learning with generative adversarial networks (GANs) has proven hugely successful. Regular GANs hypothesize the discriminator as a classifier with the sigmoid cross entropy loss function. However, we found that this loss function may lead to the vanishing gradients problem during the learning process. To overcome such a problem, we propose in this paper the Least Squares Generative Adversarial Networks (LSGANs) which adopt the least squares loss function for the discriminator. We show that minimizing the objective function of LSGAN yields minimizing the Pearson χ2 divergence. There are two benefits of LSGANs over regular GANs. First, LSGANs are able to generate higher quality images than regular GANs. Second, LSGANs perform more stable during the learning process. We evaluate LSGANs on five scene datasets and the experimental results show that the images generated by LSGANs are of better quality than the ones generated by regular GANs. We also conduct two comparison experiments between LSGANs and regular GANs to illustrate the stability of LSGANs.

<img src="https://user-images.githubusercontent.com/28132635/143052264-afd97b91-5fd1-4134-ad4d-529e364fdcc8.JPG"/>
</div>

  <b> LSGAN 64x64, CelebA-Cropped</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/22982797/116498716-f4e74200-a8dc-11eb-9c28-5549d96e20a6.png" width="800"/>
</div>

| [LSGAN 64x64](./lsgan_dcgan-archi_lr1e-3-1xb128-12Mimgs_celeba-cropped-64x64.py) | CelebA-Cropped |     6.16, 6.83, 37.64/16.87     | 0.3216  | 11.9258 | [model](https://download.openmmlab.com/mmediting/lsgan/lsgan_celeba-cropped_dcgan-archi_lr-1e-3_64_b128x1_12m_20210429_144001-92ca1d0d.pth)\| [log](https://download.openmmlab.com/mmediting/lsgan/lsgan_celeba-cropped_dcgan-archi_lr-1e-3_64_b128x1_12m_20210422_131925.log.json) |
| [LSGAN 64x64](./lsgan_dcgan-archi_lr1e-4-1xb128-12Mimgs_lsun-bedroom-64x64.py) |  LSUN-Bedroom  |      5.66, 9.0, 18.6/11.09      | 0.0671  | 30.7390 | [model](https://download.openmmlab.com/mmediting/lsgan/lsgan_lsun-bedroom_dcgan-archi_lr-1e-4_64_b128x1_12m_20210429_144602-ec4ec6bb.pth)\| [log](https://download.openmmlab.com/mmediting/lsgan/lsgan_lsun-bedroom_dcgan-archi_lr-1e-4_64_b128x1_12m_20210423_005020.log.json) |

| [LSGAN 128x128](./lsgan_dcgan-archi_lr1e-4-1xb64-10Mimgs_celeba-cropped-128x128.py) | CelebA-Cropped | 21.66, 9.83, 16.06, 70.76/29.58 | 0.3691  | 38.3752 | [model](https://download.openmmlab.com/mmediting/lsgan/lsgan_celeba-cropped_dcgan-archi_lr-1e-4_128_b64x1_10m_20210429_144229-01ba67dc.pth)\| [log](https://download.openmmlab.com/mmediting/lsgan/lsgan_celeba-cropped_dcgan-archi_lr-1e-4_128_b64x1_10m_20210423_132126.log.json) |
| [LSGAN 128x128](./lsgan_lsgan-archi_lr1e-4-1xb64-10Mimgs_lsun-bedroom-128x128.py) |  LSUN-Bedroom  |  19.52, 9.99, 7.48, 14.3/12.82  | 0.0612  | 51.5500 | [model](https://download.openmmlab.com/mmediting/lsgan/lsgan_lsun-bedroom_lsgan-archi_lr-1e-4_128_b64x1_10m_20210429_155605-cf78c0a8.pth)\| [log](https://download.openmmlab.com/mmediting/lsgan/lsgan_lsun-bedroom_lsgan-archi_lr-1e-4_128_b64x1_10m_20210429_142302.log.json) |

```latex
@inproceedings{mao2017least,
  title={Least squares generative adversarial networks},

```latex
  author={Mao, Xudong and Li, Qing and Xie, Haoran and Lau, Raymond YK and Wang, Zhen and Paul Smolley, Stephen},
  pages={2794--2802},
  url={https://openaccess.thecvf.com/content_iccv_2017/html/Mao_Least_Squares_Generative_ICCV_2017_paper.html},
}
```

# Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision

<details>
<summary align="right"><a href="https://ieeexplore.ieee.org/abstract/document/8374605/">MPI-INF-3DHP (3DV'2017)</a></summary>

```bibtex
  author = {Mehta, Dushyant and Rhodin, Helge and Casas, Dan and Fua, Pascal and Sotnychenko, Oleksandr and Xu, Weipeng and Theobalt, Christian},
  title = {Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision},
  booktitle = {3D Vision (3DV), 2017 Fifth International Conference on},
  url = {http://gvv.mpi-inf.mpg.de/3dhp_dataset},
  organization={IEEE},
  doi={10.1109/3dv.2017.00064},
}
```

## ncnn Ops

### Expand

<dl>

# 瑞芯微 NPU 部署

- [瑞芯微 NPU 部署](#瑞芯微-npu-部署)
    - [安装环境](#安装环境)
    - [分类模型转换](#分类模型转换)
    - [检测模型转换](#检测模型转换)
    - [部署 config 说明](#部署-config-说明)
    - [问题说明](#问题说明)
    - [Host 交叉编译](#host-交叉编译)
    - [Device 执行推理](#device-执行推理)

MMDeploy 支持把模型部署到瑞芯微设备上。已支持的芯片：RV1126、RK3588。

完整的部署过程包含两个步骤：

1. 模型转换

   - 在主机上，将 PyTorch 模型转换为 RKNN 模型

2. 模型推理

   - 在主机上， 使用交叉编译工具得到设备所需的 SDK 和 bin
   - 把转好的模型和编好的 SDK、bin，传到设备，进行推理

## 模型转换

### 安装环境

1. 请参考[快速入门](../get_started.md)，创建 conda 虚拟环境，并安装 PyTorch、mmcv-full

2. 安装 RKNN Toolkit

   如下表所示，瑞芯微提供了 2 套 RKNN Toolkit，对应于不同的芯片型号

   <table>
    <thead>
      <tr>
        <th>Device</th>
        <th>RKNN-Toolkit</th>
        <th>Installation Guide</th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td>RK1808 / RK1806 / RV1109 / RV1126</td>
        <td><code>git clone https://github.com/rockchip-linux/rknn-toolkit</code></td>

</tr>
    </thead>
    <tbody>
      <tr>
        <td><a href="https://github.com/rockchip-linux/rknn-toolkit2/tree/master/doc">安装指南</a></td>
      </tr>
      <tr>
        <td>RK3566 / RK3568 / RK3588 / RV1103 / RV1106</td>
        <td><code>git clone https://github.com/rockchip-linux/rknn-toolkit2</code></td>
        <td><a href="https://github.com/rockchip-linux/rknn-toolkit/tree/master/doc">安装指南</a></td>
      </tr>
    </tbody>
    </table>

   2.1 通过 `git clone` 下载和设备匹配的 RKNN Toolkit

   2.2 参考表中的安装指南，安装 RKNN python 安装包。建议在安装时，使用选项 `--no-deps`，以避免依赖包的冲突。以 rknn-toolkit2 为例:

   ```
   ```

2.2 参考表中的安装指南，安装 RKNN python 安装包。建议在安装时，使用选项 `--no-deps`，以避免依赖包的冲突。以 rknn-toolkit2 为例:

   ```
   ```

   2.3 先安装onnx==1.8.0,跟着 [instructions](../01-how-to-build/build_from_source.md)，源码安装 MMDeploy。 需要注意的是， MMDeploy 和 RKNN 依赖的安装包间有冲突的内容. 这里提供建议在 python 3.6 环境中使用的安装包版本:

   ```
   ```

### 分类模型转换

以 mmpretrain 中的 resnet50 为例，模型转换命令如下：

```shell
# 安装 mmpretrain
git clone https://github.com/open-mmlab/mmpretrain

# 执行转换命令
cd /the/path/of/mmdeploy
    /the/path/of/mmpretrain/configs/resnet/resnet50_8xb32_in1k.py \
    /the/path/of/mmpretrain/demo/demo.JPEG \

https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_batch256_imagenet_20200708-cfb998bf.pth \
    --work-dir mmdeploy_models/mmpretrain/resnet50 \
```

```{note}
若转换过程中，遇到 NoModuleFoundError 的问题，使用 pip install 对应的包
```

### 检测模型转换

- YOLOV3 & YOLOX

将下面的模型拆分配置写入到 [detection_rknn_static.py](https://github.com/open-mmlab/mmdeploy/blob/main/configs/mmdet/detection/detection_rknn-int8_static-320x320.py)

```python
# yolov3, yolox for rknn-toolkit and rknn-toolkit2
  type='rknn',  # the partition policy name
  apply_marks=True,  # should always be set to True
  partition_cfg=[
      dict(
          save_file='model.onnx',  # name to save the partitioned onnx
          start=['detector_forward:input'],  # [mark_name:input, ...]
          end=['yolo_head:input'],  # [mark_name:output, ...]
          output_names=[f'pred_maps.{i}' for i in range(3)]) # output names
  ])

end=['yolo_head:input'],  # [mark_name:output, ...]
  ])
```

执行命令：

```shell
# 安装 mmdet
pip install mmdet

# 执行转换命令
    configs/mmdet/detection/detection_rknn-int8_static-320x320.py \
    /the/path/of/mmdet/configs/yolov3/yolov3_mobilenetv2_320_300e_coco.py \
    https://download.openmmlab.com/mmdetection/v2.0/yolo/yolov3_mobilenetv2_320_300e_coco/yolov3_mobilenetv2_320_300e_coco_20210719_215349-d18dff72.pth \
    /the/path/of/mmdet/demo/demo.jpg \
    --work-dir mmdeploy_models/mmdet/yolov3 \
```

- RTMDet

将下面的模型拆分配置写入到 [detection_rknn-int8_static-640x640.py](https://github.com/open-mmlab/mmdeploy/blob/main/configs/mmdet/detection/detection_rknn-int8_static-640x640.py)

```python
# rtmdet for rknn-toolkit and rknn-toolkit2

```python
    type='rknn',  # the partition policy name
    apply_marks=True,  # should always be set to True
        dict(
            save_file='model.onnx',  # name to save the partitioned onnx
            start=['detector_forward:input'],  # [mark_name:input, ...]
            end=['rtmdet_head:output'],  # [mark_name:output, ...]
            output_names=[f'pred_maps.{i}' for i in range(6)]) # output names
    ])
```

- RetinaNet & SSD & FSAF with rknn-toolkit2

将下面的模型拆分配置写入到 [detection_rknn_static.py](https://github.com/open-mmlab/mmdeploy/blob/main/configs/mmdet/detection/detection_rknn-int8_static-320x320.py)。使用 rknn-toolkit 的用户则不用。

```python
# retinanet, ssd and fsaf for rknn-toolkit2
    apply_marks=True,
        dict(
            save_file='model.onnx',
            start='detector_forward:input',

type='rknn',  # the partition policy name
        dict(
            end=['BaseDenseHead:output'],
            output_names=[f'BaseDenseHead.cls.{i}' for i in range(5)] +
            [f'BaseDenseHead.loc.{i}' for i in range(5)])
    ])
```

### 部署 config 说明

部署 config，你可以根据需要修改 `backend_config` 字段. 一个 mmpretrain 的 `backend_config`例子如下:

```python

```

`common_config` 的内容服务于 `rknn.config()`. `quantization_config` 的内容服务于 `rknn.build()`。

### 问题说明

- SDK 只支持 int8 的 rknn 模型，这需要在转换模型时设置 `do_quantization=True`。

input_size_list=[[3, 224, 224]])

```

### 问题说明

- 模型速度问题：如果使用的设备运行的是 RKNPU，比如 rv1126 设备，请记得在 `quantization_config` 中设置 `pre_compile=True`。

## 模型推理

### Host 交叉编译

若 host 是 Ubuntu 18.04 及以上版本，推荐脚本编译：

```shell
bash tools/scripts/ubuntu_cross_build_rknn.sh <model>
```

命令中的参数 model 表示瑞芯微芯片的型号，目前支持 rv1126，rk3588。

以下是对脚本中编译过程的说明。

如下表所示，瑞芯微提供了 2 套 RKNN API 工具包，对应于不同的芯片型号。而每套 RKNN API 工具包又分别对应不同的 gcc 交叉编译工具。

| Device                                     | RKNN API                                           |
| ------------------------------------------ | -------------------------------------------------- |
| RK1808 / RK1806 / RV1109 / RV1126          | [rknpu](https://github.com/rockchip-linux/rknpu)   |
| RK3566 / RK3568 / RK3588 / RV1103 / RV1106 | [rknpu2](https://github.com/rockchip-linux/rknpu2) |

以支持的 rv1126 和 rk3588 为例，mmdeploy 在 ubuntu18.04 上的交叉编译过程如下：

- **rv11126**

1. 下载 RKNN API 包

```shell
git clone https://github.com/rockchip-linux/rknpu
export RKNPU_DIR=$(pwd)/rknpu
```

2. 准备 gcc 交叉编译工具

```shell
sudo apt-get install gcc-arm-linux-gnueabihf
sudo apt-get install g++-arm-linux-gnueabihf
```

3. 源码安装 OpenCV

```shell
git clone https://github.com/opencv/opencv --depth=1 --branch=4.6.0 --recursive
cd opencv
mkdir -p build_arm_gnueabi && cd build_arm_gnueabi
cmake .. -DCMAKE_INSTALL_PREFIX=install \
    -DCMAKE_TOOLCHAIN_FILE=../platforms/linux/arm-gnueabi.toolchain.cmake \
    -DBUILD_PERF_TESTS=OFF -DBUILD_SHARED_LIBS=OFF -DBUILD_TESTS=OFF -DCMAKE_BUILD_TYPE=Release
make -j $(nproc) && make install
export OpenCV_ARM_INSTALL_DIR=$(pwd)/install
```

-DBUILD_PERF_TESTS=OFF -DBUILD_SHARED_LIBS=OFF -DBUILD_TESTS=OFF -DCMAKE_BUILD_TYPE=Release
```

4. 编译 mmdeploy SDK

```shell
cmake .. \
-DCMAKE_TOOLCHAIN_FILE=../cmake/toolchains/arm-linux-gnueabihf.cmake \
-DMMDEPLOY_BUILD_SDK=ON \
-DMMDEPLOY_TARGET_BACKENDS="rknn" \
-DRKNPU_DEVICE_DIR=${RKNPU_DIR}/rknn/rknn_api/librknn_api \
-DOpenCV_DIR=${OpenCV_ARM_INSTALL_DIR}/lib/cmake/opencv4
```

- **rk3588**

```shell
git clone https://github.com/rockchip-linux/rknpu2
export RKNPU2_DEVICE_DIR=$(pwd)/rknpu2/runtime/RK3588
```

```shell
git clone https://github.com/Caesar-github/gcc-buildroot-9.3.0-2020.03-x86_64_aarch64-rockchip-linux-gnu
export RKNN_TOOL_CHAIN=$(pwd)/gcc-buildroot-9.3.0-2020.03-x86_64_aarch64-rockchip-linux-gnu

```

3. 下载 opencv 预编译包

```shell
cd opencv
mkdir -p build_aarch64 && cd build_aarch64
cmake .. -DCMAKE_INSTALL_PREFIX=install
    -DCMAKE_TOOLCHAIN_FILE=../platforms/linux/aarch64-gnu.toolchain.cmake \
export OpenCV_AARCH64_INSTALL_DIR=$(pwd)/install
```

```shell
cmake ..\
    -DCMAKE_TOOLCHAIN_FILE=../cmake/toolchains/rknpu2-linux-gnu.cmake \

cmake ..\
    -DOpenCV_DIR=${OpenCV_AARCH64_INSTALL_DIR}/lib/cmake/opencv4
```

### Device 执行推理

首先，确保`--dump-info`在转模型的时候调用了, 这样工作目录下包含 SDK 需要的配置文件 `pipeline.json`。

使用 `adb push` 将转好的模型、编好的 SDK 和 bin 文件推到设备上。

```bash
cd {/the/path/to/mmdeploy}
adb push mmdeploy_models/mmpretrain/resnet50  /root/resnet50
adb push {/the/path/of/mmpretrain}/demo/demo.JPEG /root/demo.JPEG
adb push build/install /root/mmdeploy_sdk
```

通过 adb shell，打开设备终端，设置环境变量，运行例子。

```bash
adb shell
cd /root/mmdeploy_sdk
export LD_LIBRARY_PATH=$(pwd)/lib:${LD_LIBRARY_PATH}
./bin/image_classification cpu ../resnet50  ../demo.JPEG
```

结果显示：

```shell
```

# 在标准数据集上训练自定义模型（待更新）

在本文中，你将知道如何在标准数据集上训练、测试和推理自定义模型。我们将在 cityscapes 数据集上以自定义 Cascade Mask R-CNN R50 模型为例演示整个过程，为了方便说明，我们将 neck 模块中的 `FPN` 替换为 `AugFPN`，并且在训练中的自动增强类中增加 `Rotate` 或 `TranslateX`。

基本步骤如下所示：

1. 准备标准数据集
2. 准备你的自定义模型
3. 准备配置文件
4. 在标准数据集上对模型进行训练、测试和推理

## 准备标准数据集

在本文中，我们使用 cityscapes 标准数据集为例进行说明。

推荐将数据集根路径采用符号链接方式链接到 `$MMDETECTION/data`。

如果你的文件结构不同，你可能需要在配置文件中进行相应的路径更改。标准的文件组织格式如下所示：

```none
├── mmdet
├── tools
├── data
│   ├── coco
│   │   │   ├── val
│   │   │   ├── val
```

你也可以通过如下方式设定数据集根路径

```bash
```

我们将会使用环境便变量 `$MMDET_DATASETS` 作为数据集的根目录，因此你无需再修改相应配置文件的路径信息。

```

```bash
```

你需要使用脚本 `tools/dataset_converters/cityscapes.py` 将 cityscapes 标注转化为 coco 标注格式。

```shell
```

目前在 `cityscapes `文件夹中的配置文件所对应模型是采用 COCO 预训练权重进行初始化的。

如果你的网络不可用或者比较慢，建议你先手动下载对应的预训练权重，否则可能在训练开始时候出现错误。

## 准备你的自定义模型

第二步是准备你的自定义模型或者训练相关配置。假设你想在已有的  Cascade Mask R-CNN R50 检测模型基础上，新增一个新的 neck 模块 `AugFPN` 去代替默认的 `FPN`，以下是具体实现：

### 1 定义新的 neck (例如 AugFPN)

首先创建新文件  `mmdet/models/necks/augfpn.py`.

```python

                num_outs,

                num_outs,
        pass

        pass
```

### 2 导入模块

你可以采用两种方式导入模块，第一种是在  `mmdet/models/necks/__init__.py` 中添加如下内容

```python
```

第二种是增加如下代码到对应配置中，这种方式的好处是不需要改动代码

```python
```

### 3 修改配置

```python
```

关于自定义模型其余相关细节例如实现新的骨架网络，头部网络、损失函数，以及运行时训练配置例如定义新的优化器、使用梯度裁剪、定制训练调度策略和钩子等，请参考文档 [自定义模型](tutorials/customize_models.md) 和 [自定义运行时训练配置](tutorials/customize_runtime.md)。

## 准备配置文件

num_outs=5)
```

## 准备配置文件

第三步是准备训练配置所需要的配置文件。假设你打算基于 cityscapes 数据集，在 Cascade Mask R-CNN R50 中新增 `AugFPN` 模块，同时增加 `Rotate` 或者 `Translate` 数据增强策略，假设你的配置文件位于 `configs/cityscapes/` 目录下，并且取名为 `cascade-mask-rcnn_r50_augfpn_autoaug-10e_cityscapes.py`，则配置信息如下：

```python
# 继承 base 配置，然后进行针对性修改
_base_ = [
]

    # 设置 `init_cfg` 为 None，表示不加载 ImageNet 预训练权重，
    # 后续可以设置 `load_from` 参数用来加载 COCO 预训练权重
    # 使用新增的 `AugFPN` 模块代替默认的 `FPN`
    # 我们也需要将 num_classes 从 80 修改为 8 来匹配 cityscapes 数据集标注

    # 这个修改包括 `bbox_head` 和 `mask_head`.
            dict(
                # 将 COCO 类别修改为 cityscapes 类别

use_sigmoid=False,
            dict(
            dict(

loss_bbox=dict(type='SmoothL1Loss', beta=1.0,
            dict(
        ],

],
            # 将 COCO 类别修改为 cityscapes 类别

# 覆写 `train_pipeline`，然后新增 `AutoAugment` 训练配置
    dict(
            [dict(
                 level=5,
                 prob=0.5)
            ],
             dict(
                 level=5,
                 prob=0.5,
            ],
        ]),
    dict(

dict(
                 level=5,
                 prob=0.5,
            ],
        ]),
    dict(
]

# 设置每张显卡的批处理大小，同时设置新的训练 pipeline
    samples_per_gpu=1,
    workers_per_gpu=3,
    train=dict(dataset=dict(pipeline=train_pipeline)))

# 设置优化器

# 设置定制的学习率策略
    dict(
    dict(
        begin=0,
        end=10,
]

# 训练，验证，测试配置

begin=0,
        end=10,
]

# 我们采用 COCO 预训练过的 Cascade Mask R-CNN R50 模型权重作为初始化权重，可以得到更加稳定的性能
```

## 训练新模型

为了能够使用新增配置来训练模型，你可以运行如下命令：

```shell
```

如果想了解更多用法，可以参考 [例子1](1_exist_data_model.md)。

## 测试和推理

为了能够测试训练好的模型，你可以运行如下命令：

```shell
```

 <img src="https://user-images.githubusercontent.com/7676947/144011085-fdded077-24de-468b-826e-5f82716219a5.png" width="400"/>
</div >

|                         Model                          |      Dataset       | PSNR (RGB)  | SSIM (RGB) |  PSNR (Y)   |  SSIM (Y)  |    Training Resources    |                            Download                            |
| :----------------------------------------------------: | :----------------: | :---------: | :--------: | :---------: | :--------: | :----------------------: | :------------------------------------------------------------: |

|       [basicvsr_reds4](./basicvsr_2xb4_reds4.py)       |    REDS4 (BIx4)    | **31.4170** | **0.8909** |      -      |     -      | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_reds4_20120409-0e599677.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_reds4_20210409_092646.log.json) |
|       [basicvsr_reds4](./basicvsr_2xb4_reds4.py)       |    UDM10 (BDx4)    |      -      |     -      |   33.4478   |   0.9306   | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_reds4_20120409-0e599677.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_reds4_20210409_092646.log.json) |

|       [basicvsr_reds4](./basicvsr_2xb4_reds4.py)       | Vimeo-90K-T (BIx4) |      -      |     -      |   36.2848   |   0.9395   | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_reds4_20120409-0e599677.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_reds4_20210409_092646.log.json) |
|       [basicvsr_reds4](./basicvsr_2xb4_reds4.py)       | Vimeo-90K-T (BDx4) |      -      |     -      |   34.4700   |   0.9286   | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_reds4_20120409-0e599677.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_reds4_20210409_092646.log.json) |

|       [basicvsr_reds4](./basicvsr_2xb4_reds4.py)       |    Vid4 (BIx4)     |      -      |     -      |   27.2694   |   0.8318   | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_reds4_20120409-0e599677.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_reds4_20210409_092646.log.json) |
|       [basicvsr_reds4](./basicvsr_2xb4_reds4.py)       |    Vid4 (BDx4)     |      -      |     -      |   24.4541   |   0.7455   | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_reds4_20120409-0e599677.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_reds4_20210409_092646.log.json) |

| [basicvsr_vimeo90k_bi](./basicvsr_2xb4_vimeo90k-bi.py) |    REDS4 (BIx4)    |   30.3128   |   0.8660   |      -      |     -      | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bi_20210409-d2d8f760.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bi_20210409_132702.log.json) |
| [basicvsr_vimeo90k_bi](./basicvsr_2xb4_vimeo90k-bi.py) |    UDM10 (BDx4)    |      -      |     -      |   34.5554   | **0.9451** | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bi_20210409-d2d8f760.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr/) |

| [basicvsr_vimeo90k_bi](./basicvsr_2xb4_vimeo90k-bi.py) | Vimeo-90K-T (BIx4) |      -      |     -      | **37.2026** |   0.9434   | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bi_20210409-d2d8f760.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr/) |
| [basicvsr_vimeo90k_bi](./basicvsr_2xb4_vimeo90k-bi.py) | Vimeo-90K-T (BDx4) |      -      |     -      |   34.8097   |   0.9316   | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bi_20210409-d2d8f760.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr/) |

| [basicvsr_vimeo90k_bi](./basicvsr_2xb4_vimeo90k-bi.py) |    Vid4 (BIx4)     |      -      |     -      | **27.2755** | **0.8248** | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bi_20210409-d2d8f760.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr/) |
| [basicvsr_vimeo90k_bi](./basicvsr_2xb4_vimeo90k-bi.py) |    Vid4 (BDx4)     |      -      |     -      |   25.0517   |   0.7636   | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bi_20210409-d2d8f760.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr/) |

| [basicvsr_vimeo90k_bd](./basicvsr_2xb4_vimeo90k-bd.py) |    REDS4 (BIx4)    |   29.0376   |   0.8481   |      -      |     -      | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bd_20210409-0154dd64.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bd_20210409_132740.log.json) |
| [basicvsr_vimeo90k_bd](./basicvsr_2xb4_vimeo90k-bd.py) |    UDM10 (BDx4)    |      -      |     -      | **39.9953** | **0.9695** | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bd_20210409-0154dd64.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bd_20210409_132740.log.json) |

| [basicvsr_vimeo90k_bd](./basicvsr_2xb4_vimeo90k-bd.py) | Vimeo-90K-T (BIx4) |      -      |     -      |   34.6427   |   0.9335   | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bd_20210409-0154dd64.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bd_20210409_132740.log.json) |
| [basicvsr_vimeo90k_bd](./basicvsr_2xb4_vimeo90k-bd.py) | Vimeo-90K-T (BDx4) |      -      |     -      | **37.5501** | **0.9499** | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bd_20210409-0154dd64.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bd_20210409_132740.log.json) |

| [basicvsr_vimeo90k_bd](./basicvsr_2xb4_vimeo90k-bd.py) |    Vid4 (BIx4)     |      -      |     -      |   26.2708   |   0.8022   | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bd_20210409-0154dd64.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bd_20210409_132740.log.json) |
| [basicvsr_vimeo90k_bd](./basicvsr_2xb4_vimeo90k-bd.py) |    Vid4 (BDx4)     |      -      |     -      | **27.9791** | **0.8556** | 2 (Tesla V100-PCIE-32GB) | [model](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bd_20210409-0154dd64.pth) \| [log](https://download.openmmlab.com/mmediting/restorers/basicvsr/basicvsr_vimeo90k_bd_20210409_132740.log.json) |

**Train**

<details>

```shell
# cpu train

**Train**

<details>

```shell
# cpu train

```

**Test**

<details>

```shell
# cpu test

```

```bibtex

}
```

<details>

```bibtex
}
```

<details>

```bibtex

| [deeppose_resnet_50_rle](/configs/body_2d_keypoint/topdown_regression/coco/td-reg_res50_rle-8xb64-210e_coco-256x192.py) |  256x192   | 0.706 |      0.888      |      0.776      | 0.753 |      0.924      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_regression/coco/td-reg_res50_rle-8xb64-210e_coco-256x192-d37efd64_20220913.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_regression/coco/td-reg_res50_rle-8xb64-210e_coco-256x192-d37efd64_20220913.log.json) |

| [deeppose_resnet_50_rle_pretrained](/configs/body_2d_keypoint/topdown_regression/coco/td-reg_res50_rle-pretrained-8xb64-210e_coco-256x192.py) |  256x192   | 0.719 |      0.891      |      0.788      | 0.764 |      0.925      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_regression/coco/td-reg_res50_rle-pretrained-8xb64-210e_coco-256x192-2cb494ee_20220913.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_regression/coco/td-reg_res50_rle-pretrained-8xb64-210e_coco-256x192-2cb494ee_20220913.log.json) |

| [deeppose_resnet_101_rle](/configs/body_2d_keypoint/topdown_regression/coco/td-reg_res101_rle-8xb64-210e_coco-256x192.py) |  256x192   | 0.722 |      0.894      |      0.794      | 0.768 |      0.930      | [ckpt](https://download.openmmlab.com/mmpose/top_down/deeppose/deeppose_res101_coco_256x192_rle-16c3d461_20220615.pth) | [log](https://download.openmmlab.com/mmpose/top_down/deeppose/deeppose_res101_coco_256x192_rle_20220615.log.json) |
| [deeppose_resnet_152_rle](/configs/body_2d_keypoint/topdown_regression/coco/td-reg_res152_rle-8xb64-210e_coco-256x192.py) |  256x192   | 0.731 |      0.897      |      0.805      | 0.777 |      0.933      | [ckpt](https://download.openmmlab.com/mmpose/top_down/deeppose/deeppose_res152_coco_256x192_rle-c05bdccf_20220615.pth) | [log](https://download.openmmlab.com/mmpose/top_down/deeppose/deeppose_res152_coco_256x192_rle_20220615.log.json) |

| [deeppose_resnet_152_rle](/configs/body_2d_keypoint/topdown_regression/coco/td-reg_res152_rle-8xb64-210e_coco-384x288.py) |  384x288   | 0.749 |      0.901      |      0.815      | 0.793 |      0.935      | [ckpt](https://download.openmmlab.com/mmpose/top_down/deeppose/deeppose_res152_coco_384x288_rle-b77c4c37_20220624.pth) | [log](https://download.openmmlab.com/mmpose/top_down/deeppose/deeppose_res152_coco_384x288_rle_20220624.log.json) |

# Branch Migration

At an earlier stage, MMOCR had three branches: `main`, `1.x`, and `dev-1.x`. Some of these branches have been renamed together with the official MMOCR 1.0.0 release, and here is the changelog.

- `main` branch housed the code for MMOCR 0.x (e.g., v0.6.3). Now it has been renamed to `0.x`.
- `1.x` contained the code for MMOCR 1.x (e.g., 1.0.0rc6). Now it is an alias of `main`, and will be removed in mid 2023.
- `dev-1.x` was the development branch for MMOCR 1.x. Now it remains unchanged.

For more information about the branches, check out [branches](../notes/branches.md).

## Resolving Conflicts When Upgrading the `main` branch

For users who wish to upgrade from the old `main` branch that has the code for MMOCR 0.x, the non-fast-forwarded-able nature of the upgrade may cause conflicts. To resolve these conflicts, follow the steps below:

1. Commit all the changes you have on `main` if you have any. Backup your current `main` branch by creating a copy.

   ```bash

   ```bash
   ```

2. Fetch the latest changes from the remote repository.

   ```bash
   ```

3. Reset the `main` branch to the latest `main` branch on the remote repository by running `git reset --hard openmmlab/main`.

   ```bash
   ```

By following these steps, you can successfully upgrade your `main` branch.

<details>

```bibtex
}
```

<details>
<summary align="right"><a href="https://sutdcv.github.io/Animal-Kingdom/">AnimalKingdom (CVPR'2022)</a></summary>

```bibtex
@InProceedings{
    Ng_2022_CVPR,

title     = {Animal Kingdom: A Large and Diverse Dataset for Animal Behavior Understanding},
 }
```

Results on AnimalKingdom validation set

| Arch                                                   | Input Size | PCK(0.05) | Official Repo | Paper  | ckpt                                                   | log                                                    |
| ------------------------------------------------------ | ---------- | --------- | ------------- | ------ | ------------------------------------------------------ | ------------------------------------------------------ |

| [P1_hrnet_w32](configs/animal_2d_keypoint/topdown_heatmap/ak/td-hm_hrnet-w32_8xb32-300e_animalkingdom_P1-256x256.py) | 256x256    | 0.6323    | 0.6342        | 0.6606 | [ckpt](https://download.openmmlab.com/mmpose/v1/animal_2d_keypoint/topdown_heatmap/animal_kingdom/td-hm_hrnet-w32_8xb32-300e_animalkingdom_P1-256x256-08bf96cb_20230519.pth) | [log](https://download.openmmlab.com/mmpose/v1/animal_2d_keypoint/topdown_heatmap/animal_kingdom/td-hm_hrnet-w32_8xb32-300e_animalkingdom_P1-256x256-08bf96cb_20230519.json) |

| [P2_hrnet_w32](configs/animal_2d_keypoint/topdown_heatmap/ak/td-hm_hrnet-w32_8xb32-300e_animalkingdom_P2-256x256.py) | 256x256    | 0.3741    | 0.3726        | 0.393  | [ckpt](https://download.openmmlab.com/mmpose/v1/animal_2d_keypoint/topdown_heatmap/animal_kingdom/td-hm_hrnet-w32_8xb32-300e_animalkingdom_P2-256x256-2396cc58_20230519.pth) | [log](https://download.openmmlab.com/mmpose/v1/animal_2d_keypoint/topdown_heatmap/animal_kingdom/td-hm_hrnet-w32_8xb32-300e_animalkingdom_P2-256x256-2396cc58_20230519.json) |

| [P3_mammals_hrnet_w32](configs/animal_2d_keypoint/topdown_heatmap/ak/td-hm_hrnet-w32_8xb32-300e_animalkingdom_P3_mammal-256x256.py) | 256x256    | 0.571     | 0.5719        | 0.6159 | [ckpt](https://download.openmmlab.com/mmpose/v1/animal_2d_keypoint/topdown_heatmap/animal_kingdom/td-hm_hrnet-w32_8xb32-300e_animalkingdom_P3_mammal-256x256-e8aadf02_20230519.pth) | [log](https://download.openmmlab.com/mmpose/v1/animal_2d_keypoint/topdown_heatmap/animal_kingdom/td-hm_hrnet-w32_8xb32-300e_animalkingdom_P3_mammal-256x256-e8aadf02_20230519.json) |

| [P3_amphibians_hrnet_w32](configs/animal_2d_keypoint/topdown_heatmap/ak/td-hm_hrnet-w32_8xb32-300e_animalkingdom_P3_amphibian-256x256.py) | 256x256    | 0.5358    | 0.5432        | 0.5674 | [ckpt](https://download.openmmlab.com/mmpose/v1/animal_2d_keypoint/topdown_heatmap/animal_kingdom/td-hm_hrnet-w32_8xb32-300e_animalkingdom_P3_amphibian-256x256-845085f9_20230519.pth) | [log](https://download.openmmlab.com/mmpose/v1/animal_2d_keypoint/topdown_heatmap/animal_kingdom/td-hm_hrnet-w32_8xb32-300e_animalkingdom_P3_amphibian-256x256-845085f9_20230519.json) |

| [P3_reptiles_hrnet_w32](configs/animal_2d_keypoint/topdown_heatmap/ak/td-hm_hrnet-w32_8xb32-300e_animalkingdom_P3_reptile-256x256.py) | 256x256    | 0.51      | 0.5           | 0.5606 | [ckpt](https://download.openmmlab.com/mmpose/v1/animal_2d_keypoint/topdown_heatmap/animal_kingdom/td-hm_hrnet-w32_8xb32-300e_animalkingdom_P3_reptile-256x256-e8440c16_20230519.pth) | [log](https://download.openmmlab.com/mmpose/v1/animal_2d_keypoint/topdown_heatmap/animal_kingdom/td-hm_hrnet-w32_8xb32-300e_animalkingdom_P3_reptile-256x256-e8440c16_20230519.json) |

| [P3_birds_hrnet_w32](configs/animal_2d_keypoint/topdown_heatmap/ak/td-hm_hrnet-w32_8xb32-300e_animalkingdom_P3_bird-256x256.py) | 256x256    | 0.7671    | 0.7636        | 0.7735 | [ckpt](https://download.openmmlab.com/mmpose/v1/animal_2d_keypoint/topdown_heatmap/animal_kingdom/td-hm_hrnet-w32_8xb32-300e_animalkingdom_P3_bird-256x256-566feff5_20230519.pth) | [log](https://download.openmmlab.com/mmpose/v1/animal_2d_keypoint/topdown_heatmap/animal_kingdom/td-hm_hrnet-w32_8xb32-300e_animalkingdom_P3_bird-256x256-566feff5_20230519.json) |

| [P3_fishes_hrnet_w32](configs/animal_2d_keypoint/topdown_heatmap/ak/td-hm_hrnet-w32_8xb32-300e_animalkingdom_P3_fish-256x256.py) | 256x256    | 0.6406    | 0.636         | 0.6825 | [ckpt](https://download.openmmlab.com/mmpose/v1/animal_2d_keypoint/topdown_heatmap/animal_kingdom/td-hm_hrnet-w32_8xb32-300e_animalkingdom_P3_fish-256x256-76c3999f_20230519.pth) | [log](https://download.openmmlab.com/mmpose/v1/animal_2d_keypoint/topdown_heatmap/animal_kingdom/td-hm_hrnet-w32_8xb32-300e_animalkingdom_P3_fish-256x256-76c3999f_20230519.json) |

# AutoAssign

> [AutoAssign: Differentiable Label Assignment for Dense Object Detection](https://arxiv.org/abs/2007.03496)

Determining positive/negative samples for object detection is known as label assignment. Here we present an anchor-free detector named AutoAssign. It requires little human knowledge and achieves appearance-aware through a fully differentiable weighting mechanism. During training, to both satisfy the prior distribution of data and adapt to category characteristics, we present Center Weighting to adjust the category-specific prior distributions. To adapt to object appearances, Confidence Weighting is proposed to adjust the specific assign strategy of each instance. The two weighting modules are then combined to generate positive and negative weights to adjust each location's confidence. Extensive experiments on the MS COCO show that our method steadily surpasses other best sampling strategies by large margins with various backbones. Moreover, our best model achieves 52.1% AP, outperforming all existing one-stage detectors. Besides, experiments on other datasets, e.g., PASCAL VOC, Objects365, and WiderFace, demonstrate the broad applicability of AutoAssign.

<img src="https://user-images.githubusercontent.com/40661020/143870875-33567e44-0584-4470-9a90-0df0fb6c1fe2.png"/>
</div>

| Backbone | Style | Lr schd | Mem (GB) | box AP |                     Config                      |                                                                                                                                                        Download                                                                                                                                                         |
| :------: | :---: | :-----: | :------: | :----: | :---------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

|   R-50   | caffe |   1x    |   4.08   |  40.4  | [config](./autoassign_r50-caffe_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/autoassign/auto_assign_r50_fpn_1x_coco/auto_assign_r50_fpn_1x_coco_20210413_115540-5e17991f.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/autoassign/auto_assign_r50_fpn_1x_coco/auto_assign_r50_fpn_1x_coco_20210413_115540-5e17991f.log.json) |

**Note**:

1. We find that the performance is unstable with 1x setting and may fluctuate by about 0.3 mAP. mAP 40.3 ~ 40.6 is acceptable. Such fluctuation can also be found in the original implementation.
2. You can get a more stable results ~ mAP 40.6 with a schedule total 13 epoch, and learning rate is divided by 10 at 10th and 13th epoch.

```latex
@article{zhu2020autoassign,
  title={AutoAssign: Differentiable Label Assignment for Dense Object Detection},

```latex
  author={Zhu, Benjin and Wang, Jianfeng and Jiang, Zhengkai and Zong, Fuhang and Liu, Songtao and Li, Zeming and Sun, Jian},
  journal={arXiv preprint arXiv:2007.03496},
}
```

# ncnn GLSL 扩展

## 理由
不同的 GPU 支持不同的功能，有的支持 fp16 作为缓冲存储类型，有的支持 fp16 作为操作数变量，有的老 GPU 只支持 fp32。

当 GPU 支持 `VK_KHR_16bit_storage` 扩展时，为了尽量减少 GPU 的内存带宽消耗，我们会优先使用 fp16 作为存储类型。否则，我们使用 `packHalf2x16` 和 `unpackHalf2x16` 在 GLSL 4.2 中将 2 个 fp32 压缩为 uint，从而减少读写带宽。

同样，当 GPU 支持 `VK_KHR_shader_float16_int8` 扩展时，为了加快计算效率，我们会优先使用 fp16 作为运算操作数，这通常会使速度翻倍。否则，我们使用 fp32。

为了确保最广泛的兼容性，将编写以下用于声明描述符绑定和加载数据的代码

```c
#if NCNN_fp16_storage // GPU支持 16bit storage
#elif NCNN_fp16_packed // GPU支持 GLSL 4.2
#else // GPU仅支持 fp32
#endif

{

#if NCNN_fp16_storage && NCNN_fp16_arithmetic // GPU支持 16bit storage 和 shader float16
#elif NCNN_fp16_storage // GPU支持 16bit storage 但不包含 shader float16

f16vec4 x = blob_data[i];
#elif NCNN_fp16_packed && NCNN_fp16_arithmetic // GPU支持 GLSL 4.2 和 shader float16
#endif
}
```

如您所见，仅声明缓冲区类型并读取值会消耗大量代码行，这是项目维护的噩梦。因此，ncnn 增加了更灵活的数据类型和辅助函数，以减小代码的大小并提高可读性，并且会根据 GPU 支持的功能级别自动扩展到最高效的实现。

上面的代码，通过使用 ncnn GLSL 扩展，可以简化为

```c

{

}
```

ncnn GLSL 扩展为存储、计算、共享内存以及缓冲区和图像的加载、存储、转换函数提供了必要的数据类型。我们还提供了一些缓冲区和图像复制函数，以防止在使用 fp16 作为中间数据类型时丢失精度，并避免不必要的 `unpackHalf2x16` 和 `packHalf2x16` 配对。

# 编译GLSL的入口点

}
```

ncnn库中的 gpu.h 头文件公开了3个用于将 GLSL 代码编译为 Spir-V 二进制的API函数，它们支持 ncnn GLSL 扩展，这3个函数接受 opt switch 来控制 ncnn GLSL 扩展形式。前两个函数接受原始 GLSL 代码字符串作为参数，最后一个函数用于创建 ncnn 的已存在的内置着色器。

```cpp

// 在线 Spir-V 编译器

```

## 直接编译ncnn扩展GLSL代码

您可以使用 ncnn GLSL 扩展编写着色器代码，使用 ncnn 函数编译为 Spir-V。编译后的产品是符合标准的 Spir-V 二进制文件，可以直接用于在 Vulkan API 中创建流水线对象

```cpp

```cpp

#endif
#endif

{

    v = v + 123;

}
)";

 // 您可以控制Vulkan扩展行为
 // 当GPU支持16位存储的话

// 稍后再创建管道对象

```

## ncnn内置着色器

ncnn内部的着色器索引在标头中公开，如果需要可以使用 `layer_shader_type.h`

```cpp

```

# 数据类型

## 存储类型(storage type)

在描述符绑定中声明缓冲区数据布局

```c
```

|存储类型|fp32|fp16p|fp16s|

## 算术类型(arithmetic type)

在 GLSL 代码中声明局部变量

```c
{
}
```

|算术类型|fp32|fp16a|

```c
{
}
```

## 本地类型(local type)

在共享本地内存中声明变量

```c
```

## 图像格式类型(image format type)和精度类型(precision hint type)

在描述符绑定中声明图像格式

```c
```

|格式类型|fp32|fp16p|fp16s|

|精度类型|fp32|fp16p|fp16s|

# 缓冲区函数(buffer functions)

- 从 src[offset] 加载已经确定类型的值

```c

```c
```

- 将已确定类型的值存储到 dst[偏移量]

```c
```

- 从已确定类型 src[src_offset] 的值拷贝到 dst[dst_offset]

```c
```

- 从 src[src_offsets[0],src_offsets[1],...] 的值拷贝并打包到 dst[dst_offset]

```c

```

```c
```

- 从 src[src_offset] 的值拷贝并解包到 dst[dst_offsets[0],dst_offsets[1],...]

```c
```

# 图像函数

- 根据 sampler?D 图像(透过 src 和 pos) 来加载数据

```c

```c
```

- 存储确定类型的值到 image?D (透过 dst 和 pos 参数)

```c
```

- 把 sampler?D 的值的内容(透过 src 和 src_pos 参数) 拷贝到 image?D (透过 dst 和 dst_pos 参数)

```

```c
```

注意：由于图像是不透明的数据结构，因此不提供复制和打包/解包功能。要实现此操作，您需要先加载，然后再存储。

# 本地数据转换函数

- 存储缓冲区转换到本地内存

```c

```

# 本地数据转换函数

```c
```

- 本地内存转换到局部变量

```c
```

注意：本地内存的常见用法是先从全局内存中读取，存储在本地内存中，然后再从本地内存中读取局部变量以供后续使用。因此，此处仅提供存储类型到本地类型和本地类型到算术类型的转换函数。

# 杂项函数

- 更推荐使用专业化常量(specialization constants)，而不是推动常量(push constants)

```c
T psc(T x)
```

在 `专用常量` 和 `推送常量` 部分中声明相同的变量，然后在专用常量给定非零时 `psc(x)` 将成为编译时常量，否则将通过推送常量动态。这通常用于张量形状特化。我们通常可以解析所有形状信息，并使它们成为编译时常量，以实现让着色器得到更积极的优化。

```c

{
    int size;
} p;

{
}
```

# 平台宏定义

判断当前平台是否为 moltenvk，以启用对于某些特定于平台的解决方法

```c
// 启用moltenvk的解决方法
#endif
```

# 条件宏定义与option的关系

仅当用户启用某些选项时才启用 GLSL 扩展

```c

```c
#endif
```

```c
#endif
#endif
```

声明图像或缓冲区的描述符绑定

```c
#else
#endif
```

|宏定义|option中所定义的变量|
|---|---|

}
```

<details>

```bibtex
}
```

}
```

<details>
<summary align="right">DeepFillv1 (CVPR'2018)</summary>

```bibtex
}
```

<br/>

|                       算法                        |  掩膜类型   | 分辨率  | 训练集容量 |    测试集     | l1 损失 |  PSNR  | SSIM  | GPU 信息 |                                         下载                                         |
| :-----------------------------------------------: | :---------: | :-----: | :--------: | :-----------: | :-----: | :----: | :---: | :------: | :----------------------------------------------------------------------------------: |

| [DeepFillv1](./deepfillv1_8xb2_places-256x256.py) | square bbox | 256x256 |   3500k    | Places365-val | 11.019  | 23.429 | 0.862 |    8     | [模型](https://download.openmmlab.com/mmediting/inpainting/deepfillv1/deepfillv1_256x256_8x2_places_20200619-c00a0e21.pth) \| [日志](https://download.openmmlab.com/mmediting/inpainting/deepfillv1/deepfillv1_256x256_8x2_places_20200619-c00a0e21.log.json) |

|                       算法                        |  掩膜类型   | 分辨率  | 训练集容量 |   测试集   | l1 损失 |  PSNR  | SSIM  | GPU 信息 |                                          下载                                           |
| :-----------------------------------------------: | :---------: | :-----: | :--------: | :--------: | :-----: | :----: | :---: | :------: | :-------------------------------------------------------------------------------------: |

| [DeepFillv1](./deepfillv1_4xb4_celeba-256x256.py) | square bbox | 256x256 |   1500k    | CelebA-val |  6.677  | 26.878 | 0.911 |    4     | [模型](https://download.openmmlab.com/mmediting/inpainting/deepfillv1/deepfillv1_256x256_4x4_celeba_20200619-dd51a855.pth) \| [日志](https://download.openmmlab.com/mmediting/inpainting/deepfillv1/deepfillv1_256x256_4x4_celeba_20200619-dd51a855.log.json) |

## 快速开始

**训练**

<details>

```shell
# CPU上训练

# 单个GPU上训练

# 多个GPU上训练
```

**测试**

<details>

```shell
# CPU上测试

```

**测试**

<details>

```shell
# CPU上测试

# 单个GPU上测试

# 多个GPU上测试
```

# 常见问题解答 (待更新)

我们在这里列出了使用时的一些常见问题及其相应的解决方案。 如果您发现有一些问题被遗漏，请随时提 PR 丰富这个列表。如果您无法在此获得帮助，请使用 [issue模板](https://github.com/open-mmlab/mmdetection/blob/master/.github/ISSUE_TEMPLATE/error-report.md/) 创建问题，但是请在模板中填写所有必填信息，这有助于我们更快定位问题。

## MMCV 安装相关

MMRotate 和 MMCV, MMDet 版本兼容性如下所示，需要安装正确的版本以避免安装出现问题。

**注意：**

**注意：**

1. 如果你希望安装 mmrotate-v0.x, MMRotate 和 MMCV 版本兼容表可以在 [这里](https://mmrotate.readthedocs.io/en/stable/faq.html#installation) 找到，请选择合适的版本避免安装错误。
2. 在 MMCV-v2.x 中，`mmcv-full` 改名为 `mmcv`，如果你想安装不包含 CUDA 算子的版本，可以选择安装 MMCV 精简版 `mmcv-lite`。

  1. 常见报错信息为 `nvcc fatal: Unsupported gpu architecture 'compute_86'` 意思是你的编译器应该为 sm_86 进行优化，例如， 英伟达 30 系列的显卡，但这样的优化 CUDA toolkit 11.0 并不支持。

     此解决方案通过添加 `MMCV_WITH_OPS=1 MMCV_CUDA_ARGS='-gencode=arch=compute_80,code=sm_80' pip install -e .` 来修改编译标志，这告诉编译器 `nvcc` 为 **sm_80** 进行优化，例如 Nvidia A100，尽管 A100 不同于 30 系列的显卡，但他们使用相似的图灵架构。这种解决方案可能会丧失一些性能但的确有效。
  2. PyTorch 开发者已经在 [pytorch/pytorch#47585](https://github.com/pytorch/pytorch/pull/47585) 更新了 PyTorch 默认的编译标志，所以使用 PyTorch-nightly 可能也能解决这个问题，但是我们对此并没有验证这种方式是否有效。

  1. 检查您的 cuda 运行时版本(一般在 `/usr/local/`)、指令 `nvcc --version` 显示的版本以及 `conda list cudatoolkit` 指令显式的版本是否匹配。

  2. 通过运行 `python mmdet/utils/collect_env.py` 来检查是否为当前的GPU架构编译了正确的 PyTorch、torchvision 和 MMCV，你可能需要设置 `TORCH_CUDA_ARCH_LIST` 来重新安装 MMCV。可以参考 [GPU 架构表](https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#gpu-feature-list)，即通过运行 `TORCH_CUDA_ARCH_LIST=7.0 pip install mmcv-full` 为 Volta GPU 编译 MMCV。这种架构不匹配的问题一般会出现在使用一些旧型号的 GPU 时候，例如，Tesla K80。
  3. 检查运行环境是否与 mmcv/mmdet 编译时相同，例如，您可能使用 CUDA 10.0 编译 MMCV，但在 CUDA 9.0 环境中运行它。

  1. 如果这些 symbols 属于 CUDA/C++ (例如，libcudart.so 或者 GLIBCXX)，检查 CUDA/GCC 运行时环境是否与编译 MMCV 的一致。例如使用 `python mmdet/utils/collect_env.py` 检查 `"MMCV Compiler"`/`"MMCV CUDA Compiler"` 是否和 `"GCC"`/`"CUDA_HOME"` 一致。
  2. 如果这些 symbols 属于 PyTorch，(例如，symbols containing caffe、aten 和 TH)， 检查当前 PyTorch 版本是否与编译 MMCV 的版本一致。

2. 如果这些 symbols 属于 PyTorch，(例如，symbols containing caffe、aten 和 TH)， 检查当前 PyTorch 版本是否与编译 MMCV 的版本一致。
  3. 运行 `python mmdet/utils/collect_env.py` 检查 PyTorch、torchvision、MMCV 等的编译环境与运行环境一致。

  1. 如果你在使用 miniconda 而不是 anaconda，检查是否正确的安装了 Cython 如 [#3379](https://github.com/open-mmlab/mmdetection/issues/3379)。您需要先手动安装 Cpython 然后运命令 `pip install -r requirements.txt`。
  2. 检查环境中的 `setuptools`、`Cython` 和 `PyTorch` 相互之间版本是否匹配。

  1. 检查 GCC 的版本并使用 GCC 5.4，通常是因为 PyTorch 版本与 GCC 版本不匹配 （例如，对于 Pytorch GCC \< 4.9)，我们推荐用户使用 GCC 5.4，我们也不推荐使用 GCC 5.5， 因为有反馈 GCC 5.5 会导致 "segmentation fault" 并且切换到 GCC 5.4 就可以解决问题。

  2. 检查是是否 PyTorch 被正确的安装并可以使用 CUDA 算子，例如在终端中键入如下的指令。

     ```shell
     ```

     并判断是否是否返回 True。

     ```shell

```shell
     ```

     ```shell
     ```

     如果 MMCV 被正确的安装了，那么上面的两条指令不会有问题。

  4. 如果 MMCV 与 PyTorch 都被正确安装了，则使用 `ipdb`、`pdb` 设置断点，直接查找哪一部分的代码导致了 `segmentation fault`。

## E2CNN

  1. 这是因为 `container_abcs` 在 PyTorch 1.9 之后被移除.

  2. 将文件 `python3.7/site-packages/e2cnn/nn/modules/module_list.py` 中的

     ```shell
     ```

     替换成

     ```shell
     else:
     ```

  3. 或者降低 Pytorch 的版本。

from torch.six import container_abcs
     else:
     ```

  1. 检查数据的标注是否正常，长或宽为 0 的框可能会导致回归 loss 变为 nan，一些小尺寸（宽度或高度小于 1）的框在数据增强（例如，instaboost）后也会导致此问题。因此，可以检查标注并过滤掉那些特别小甚至面积为 0 的框，并关闭一些可能会导致 0 面积框出现数据增强。
  2. 降低学习率：由于某些原因，例如 batch size 大小的变化，导致当前学习率可能太大。您可以降低为可以稳定训练模型的值。
  4. 添加 gradient clipping: 一些模型需要梯度裁剪来稳定训练过程。默认的 `grad_clip` 是 `None`，你可以在 config 设置 `optimizer_config=dict(_delete_=True, grad_clip=dict(max_norm=35, norm_type=2))`。 如果你的 config 没有继承任何包含 `optimizer_config=dict(grad_clip=None)`，你可以直接设置 `optimizer_config=dict(grad_clip=dict(max_norm=35, norm_type=2))`。

  1. 存在大量 ground truth boxes 或者大量 anchor 的场景，可能在 assigner 会 OOM。您可以在 assigner 的配置中设置 `gpu_assign_thr=N`，这样当超过 N 个 GT boxes 时，assigner 会通过 CPU 计算 IoU。

  2. 在 backbone 中设置 `with_cp=True`。这使用 PyTorch 中的 `sublinear strategy` 来降低 backbone 占用的 GPU 显存。
  3. 通过在配置文件中设置 `fp16 = dict(loss_scale='dynamic')` 来尝试混合精度训练。

  1. 错误表明，您的模块有没用于产生损失的参数，这种现象可能是由于在 DDP 模式下运行代码中的不同分支造成的。
  2. 您可以在配置中设置 `find_unused_parameters = True` 来解决上述问题，或者手动查找那些未使用的参数。

- 使用 COCO Dataset 的测评接口时，测评结果中 AP 或者 AR = -1。
  1. 根据 COCO 数据集的定义，一张图像中的中等物体与小物体面积的阈值分别为 9216（96\*96）与 1024（32\*32）。
  2. 如果在某个区间没有物体即 GT，AP 与 AR 将被设置为 -1。

# SegFormer

> [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://arxiv.org/abs/2105.15203)

<a href="https://github.com/NVlabs/SegFormer">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/backbones/mit.py#L246">Code Snippet</a>

We present SegFormer, a simple, efficient yet powerful semantic segmentation framework which unifies Transformers with lightweight multilayer perception (MLP) decoders. SegFormer has two appealing features: 1) SegFormer comprises a novel hierarchically structured Transformer encoder which outputs multiscale features. It does not need positional encoding, thereby avoiding the interpolation of positional codes which leads to decreased performance when the testing resolution differs from training. 2) SegFormer avoids complex decoders. The proposed MLP decoder aggregates information from different layers, and thus combining both local attention and global attention to render powerful representations. We show that this simple and lightweight design is the key to efficient segmentation on Transformers. We scale our approach up to obtain a series of models from SegFormer-B0 to SegFormer-B5, reaching significantly better performance and efficiency than previous counterparts. For example, SegFormer-B4 achieves 50.3% mIoU on ADE20K with 64M parameters, being 5x smaller and 2.2% better than the previous best method. Our best model, SegFormer-B5, achieves 84.0% mIoU on Cityscapes validation set and shows excellent zero-shot robustness on Cityscapes-C. Code will be released at: [this http URL](https://github.com/NVlabs/SegFormer).

<img src="https://user-images.githubusercontent.com/24582831/142902600-e188073e-5744-4ba9-8dbf-9316e55c74aa.png" width="70%"/>
</div>

## Usage

We provide a script [`mit2mmseg.py`](../../tools/model_converters/mit2mmseg.py) in the tools directory to convert the key of models from [the official repo](https://github.com/NVlabs/SegFormer) to MMSegmentation style.

```shell
python tools/model_converters/mit2mmseg.py ${PRETRAIN_PATH} ${STORE_PATH}
```

### ADE20K

```

### ADE20K

| Method    | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device   |  mIoU | mIoU(ms+flip) | config                                                                                                                          | download                                                                                                                                                                                                                                                                                                                                               |

| Segformer | MIT-B0   | 512x512   |  160000 |      2.1 | 51.32          | 1080 Ti  | 37.41 | 38.34         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segformer/segformer_mit-b0_8xb2-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b0_512x512_160k_ade20k/segformer_mit-b0_512x512_160k_ade20k_20210726_101530-8ffa8fda.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b0_512x512_160k_ade20k/segformer_mit-b0_512x512_160k_ade20k_20210726_101530.log.json) |

| Segformer | MIT-B1   | 512x512   |  160000 |      2.6 | 47.66          | TITAN Xp | 40.97 | 42.54         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segformer/segformer_mit-b1_8xb2-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b1_512x512_160k_ade20k/segformer_mit-b1_512x512_160k_ade20k_20210726_112106-d70e859d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b1_512x512_160k_ade20k/segformer_mit-b1_512x512_160k_ade20k_20210726_112106.log.json) |

| Segformer | MIT-B2   | 512x512   |  160000 |      3.6 | 30.88          | TITAN Xp | 45.58 | 47.03         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segformer/segformer_mit-b2_8xb2-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b2_512x512_160k_ade20k/segformer_mit-b2_512x512_160k_ade20k_20210726_112103-cbd414ac.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b2_512x512_160k_ade20k/segformer_mit-b2_512x512_160k_ade20k_20210726_112103.log.json) |

| Segformer | MIT-B3   | 512x512   |  160000 |      4.8 | 22.11          | V100     | 47.82 | 48.81         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segformer/segformer_mit-b3_8xb2-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b3_512x512_160k_ade20k/segformer_mit-b3_512x512_160k_ade20k_20210726_081410-962b98d2.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b3_512x512_160k_ade20k/segformer_mit-b3_512x512_160k_ade20k_20210726_081410.log.json) |

| Segformer | MIT-B4   | 512x512   |  160000 |      6.1 | 15.45          | V100     | 48.46 | 49.76         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segformer/segformer_mit-b4_8xb2-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b4_512x512_160k_ade20k/segformer_mit-b4_512x512_160k_ade20k_20210728_183055-7f509d7d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b4_512x512_160k_ade20k/segformer_mit-b4_512x512_160k_ade20k_20210728_183055.log.json) |

| Segformer | MIT-B5   | 512x512   |  160000 |      7.2 | 11.89          | V100     | 49.13 | 50.22         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segformer/segformer_mit-b5_8xb2-160k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b5_512x512_160k_ade20k/segformer_mit-b5_512x512_160k_ade20k_20210726_145235-94cedf59.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b5_512x512_160k_ade20k/segformer_mit-b5_512x512_160k_ade20k_20210726_145235.log.json) |

| Segformer | MIT-B5   | 640x640   |  160000 |     11.5 | 11.30          | V100     | 49.62 | 50.36         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segformer/segformer_mit-b5_8xb2-160k_ade20k-640x640.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b5_640x640_160k_ade20k/segformer_mit-b5_640x640_160k_ade20k_20210801_121243-41d2845b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b5_640x640_160k_ade20k/segformer_mit-b5_640x640_160k_ade20k_20210801_121243.log.json) |

Evaluation with AlignedResize:

| Method    | Backbone | Crop Size | Lr schd |  mIoU | mIoU(ms+flip) |
| --------- | -------- | --------- | ------: | ----: | ------------- |
| Segformer | MIT-B0   | 512x512   |  160000 |  38.1 | 38.57         |
| Segformer | MIT-B1   | 512x512   |  160000 | 41.64 | 42.76         |
| Segformer | MIT-B2   | 512x512   |  160000 | 46.53 | 47.49         |

| Segformer | MIT-B3   | 512x512   |  160000 | 48.46 | 49.14         |
| Segformer | MIT-B4   | 512x512   |  160000 | 49.34 | 50.29         |
| Segformer | MIT-B5   | 512x512   |  160000 | 50.08 | 50.72         |
| Segformer | MIT-B5   | 640x640   |  160000 | 50.58 | 50.8          |

We replace `AlignedResize` in original implementatiuon to `Resize + ResizeToMultiple`. If you want to test by
using `AlignedResize`, you can change the dataset pipeline like this:

```python
    dict(type='Resize', scale=(2048, 512), keep_ratio=True),
    # resize image to multiple of 32, improve SegFormer by 0.5-1.0 mIoU.
    dict(type='ResizeToMultiple', size_divisor=32),

dict(type='ResizeToMultiple', size_divisor=32),
]
```

The lower fps result is caused by the sliding window inference scheme (window size:1024x1024).

dict(type='PackSegInputs')
]
```

| Method    | Backbone | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                                | download                                                                                                                                                                                                                                                                                                                                                                                       |

| Segformer | MIT-B0   | 1024x1024 |  160000 |     3.64 | 4.74           | V100   | 76.54 | 78.22         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segformer/segformer_mit-b0_8xb1-160k_cityscapes-1024x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b0_8x1_1024x1024_160k_cityscapes/segformer_mit-b0_8x1_1024x1024_160k_cityscapes_20211208_101857-e7f88502.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b0_8x1_1024x1024_160k_cityscapes/segformer_mit-b0_8x1_1024x1024_160k_cityscapes_20211208_101857.log.json) |

| Segformer | MIT-B1   | 1024x1024 |  160000 |     4.49 | 4.3            | V100   | 78.56 | 79.73         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segformer/segformer_mit-b1_8xb1-160k_cityscapes-1024x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b1_8x1_1024x1024_160k_cityscapes/segformer_mit-b1_8x1_1024x1024_160k_cityscapes_20211208_064213-655c7b3f.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b1_8x1_1024x1024_160k_cityscapes/segformer_mit-b1_8x1_1024x1024_160k_cityscapes_20211208_064213.log.json) |

| Segformer | MIT-B2   | 1024x1024 |  160000 |     7.42 | 3.36           | V100   | 81.08 | 82.18         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segformer/segformer_mit-b2_8xb1-160k_cityscapes-1024x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b2_8x1_1024x1024_160k_cityscapes/segformer_mit-b2_8x1_1024x1024_160k_cityscapes_20211207_134205-6096669a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b2_8x1_1024x1024_160k_cityscapes/segformer_mit-b2_8x1_1024x1024_160k_cityscapes_20211207_134205.log.json) |

| Segformer | MIT-B3   | 1024x1024 |  160000 |    10.86 | 2.53           | V100   | 81.94 | 83.14         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segformer/segformer_mit-b3_8xb1-160k_cityscapes-1024x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b3_8x1_1024x1024_160k_cityscapes/segformer_mit-b3_8x1_1024x1024_160k_cityscapes_20211206_224823-a8f8a177.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b3_8x1_1024x1024_160k_cityscapes/segformer_mit-b3_8x1_1024x1024_160k_cityscapes_20211206_224823.log.json) |

| Segformer | MIT-B4   | 1024x1024 |  160000 |    15.07 | 1.88           | V100   | 81.89 | 83.38         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segformer/segformer_mit-b4_8xb1-160k_cityscapes-1024x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b4_8x1_1024x1024_160k_cityscapes/segformer_mit-b4_8x1_1024x1024_160k_cityscapes_20211207_080709-07f6c333.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b4_8x1_1024x1024_160k_cityscapes/segformer_mit-b4_8x1_1024x1024_160k_cityscapes_20211207_080709.log.json) |

| Segformer | MIT-B5   | 1024x1024 |  160000 |    18.00 | 1.39           | V100   | 82.25 | 83.48         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/segformer/segformer_mit-b5_8xb1-160k_cityscapes-1024x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b5_8x1_1024x1024_160k_cityscapes/segformer_mit-b5_8x1_1024x1024_160k_cityscapes_20211206_072934-87a052ec.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/segformer/segformer_mit-b5_8x1_1024x1024_160k_cityscapes/segformer_mit-b5_8x1_1024x1024_160k_cityscapes_20211206_072934.log.json) |

```bibtex
@article{xie2021segformer,
  title={SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers},
  author={Xie, Enze and Wang, Wenhai and Yu, Zhiding and Anandkumar, Anima and Alvarez, Jose M and Luo, Ping},
  journal={arXiv preprint arXiv:2105.15203},
}
```

# Migration of AMP Training

In 0.x, MMEditing do not support AMP training for the entire forward process.
Instead, users must use `auto_fp16` decorator to warp the specific submodule and convert the parameter of submodule to fp16.
This allows for fine-grained control of the model parameters, but is more cumbersome to use.
In addition, users need to handle operations such as scaling of the loss function during the training process by themselves.

MMagic 1.x use `AmpOptimWrapper` provided by MMEngine.
In `AmpOptimWrapper.update_params`, gradient scaling and `GradScaler` updating is automatically performed.
And in `optim_context` context manager, `auto_cast` is applied to the entire forward process.

Specifically, the difference between the 0.x and 1.x is as follows:

<thead>
  <tr>
<tbody>
<tr>

```python
# config
```

```python
# code

<thead>
  <tr>
<tbody>
<tr>

```python
# config
```

```python
# code

        # get data from data_batch

running_status=None):

            # add support for fp16
        else:

        else:
```

</td>

```python
# config

else:
```

</td>

```python
# config
        type='AmpOptimWrapper',  # use amp wrapper
```

```python
# code

        # use parse_loss provide by `BaseModel`

```

</td>

</tr>
</thead>
</table>

To avoid user modifications to the configuration file, MMagic provides the `--amp` option in `train.py`, which allows the user to start AMP training without modifying the configuration file.
Users can start AMP training by following command:

```bash

```bash

# for slurm users
```

# Swin-Transformer

**Swin Transformer** (the name **Swin** stands for Shifted window) is initially described in [the paper](https://arxiv.org/pdf/2103.14030.pdf), which capably serves as a general-purpose backbone for computer vision. It is basically a hierarchical Transformer whose representation is computed with shifted windows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection.

Swin Transformer achieves strong performance on COCO object detection (58.7 box AP and 51.1 mask AP on test-dev) and ADE20K semantic segmentation (53.5 mIoU on val), surpassing previous models by a large margin.

<img src="https://user-images.githubusercontent.com/26739999/142576715-14668c6b-5cb8-4de8-ac51-419fae773c90.png" width="90%"/>
</div>

<details>

<br>

This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with **Shifted windows**. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it compatible with a broad range of vision tasks, including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation (53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-the-art by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones. The hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures.

</br>

```python

predict = inference_model('swin-tiny_16xb64_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('swin-tiny_16xb64_in1k', pretrained=True)
```

Train:

```shell
python tools/train.py configs/swin_transformer/swin-tiny_16xb64_in1k.py
```

Test:

```shell

Train:

```shell
```

Test:

```shell
python tools/test.py configs/swin_transformer/swin-tiny_16xb64_in1k.py https://download.openmmlab.com/mmclassification/v0/swin-transformer/swin_tiny_224_b16x64_300e_imagenet_20210616_090925-66df6be6.pth
```

| Model                                      |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) | Top-5 (%) |                  Config                   |                               Download                               |
| :----------------------------------------- | :----------: | :--------: | :-------: | :-------: | :-------: | :---------------------------------------: | :------------------------------------------------------------------: |

| `swin-tiny_16xb64_in1k`                    | From scratch |   28.29    |   4.36    |   81.18   |   95.61   |    [config](swin-tiny_16xb64_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/swin-transformer/swin_tiny_224_b16x64_300e_imagenet_20210616_090925-66df6be6.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/swin-transformer/swin_tiny_224_b16x64_300e_imagenet_20210616_090925.json) |
| `swin-small_16xb64_in1k`                   | From scratch |   49.61    |   8.52    |   83.02   |   96.29   |    [config](swin-small_16xb64_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/swin-transformer/swin_small_224_b16x64_300e_imagenet_20210615_110219-7f9d988b.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/swin-transformer/swin_small_224_b16x64_300e_imagenet_20210615_110219.json) |

| `swin-base_16xb64_in1k`                    | From scratch |   87.77    |   15.14   |   83.36   |   96.44   |    [config](swin-base_16xb64_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/swin-transformer/swin_base_224_b16x64_300e_imagenet_20210616_190742-93230b0d.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/swin-transformer/swin_base_224_b16x64_300e_imagenet_20210616_190742.json) |
| `swin-tiny_3rdparty_in1k`\*                | From scratch |   28.29    |   4.36    |   81.18   |   95.52   |    [config](swin-tiny_16xb64_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/swin-transformer/convert/swin_tiny_patch4_window7_224-160bb0a5.pth) |
| `swin-small_3rdparty_in1k`\*               | From scratch |   49.61    |   8.52    |   83.21   |   96.25   |    [config](swin-small_16xb64_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/swin-transformer/convert/swin_small_patch4_window7_224-cc7a01c9.pth) |

| `swin-base_3rdparty_in1k`\*                | From scratch |   87.77    |   15.14   |   83.42   |   96.44   |    [config](swin-base_16xb64_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/swin-transformer/convert/swin_base_patch4_window7_224-4670dd19.pth) |
| `swin-base_3rdparty_in1k-384`\*            | From scratch |   87.90    |   44.49   |   84.49   |   96.95   | [config](swin-base_16xb64_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/swin-transformer/convert/swin_base_patch4_window12_384-02c598a4.pth) |
| `swin-base_in21k-pre-3rdparty_in1k`\*      | From scratch |   87.77    |   15.14   |   85.16   |   97.50   |    [config](swin-base_16xb64_in1k.py)     | [model](https://download.openmmlab.com/mmclassification/v0/swin-transformer/convert/swin_base_patch4_window7_224_22kto1k-f967f799.pth) |

| `swin-base_in21k-pre-3rdparty_in1k-384`\*  | From scratch |   87.90    |   44.49   |   86.44   |   98.05   | [config](swin-base_16xb64_in1k-384px.py)  | [model](https://download.openmmlab.com/mmclassification/v0/swin-transformer/convert/swin_base_patch4_window12_384_22kto1k-d59b0d1d.pth) |
| `swin-large_in21k-pre-3rdparty_in1k`\*     | From scratch |   196.53   |   34.04   |   86.24   |   97.88   |    [config](swin-large_16xb64_in1k.py)    | [model](https://download.openmmlab.com/mmclassification/v0/swin-transformer/convert/swin_large_patch4_window7_224_22kto1k-5f0996db.pth) |
| `swin-large_in21k-pre-3rdparty_in1k-384`\* | From scratch |   196.74   |  100.04   |   87.25   |   98.25   | [config](swin-large_16xb64_in1k-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/swin-transformer/convert/swin_large_patch4_window12_384_22kto1k-0a40944b.pth) |

*Models with * are converted from the [official repo](https://github.com/microsoft/Swin-Transformer/blob/777f6c66604bb5579086c4447efe3620344d95a9/models/swin_transformer.py#L458). The config files of these models are only for inference. We haven't reproduce the training results.*

| Model                       |   Pretrain   | Params (M) | Flops (G) | Top-1 (%) |                 Config                 |                                            Download                                             |
| :-------------------------- | :----------: | :--------: | :-------: | :-------: | :------------------------------------: | :---------------------------------------------------------------------------------------------: |

| `swin-large_8xb8_cub-384px` | From scratch |   195.51   |  100.04   |   91.87   | [config](swin-large_8xb8_cub-384px.py) | [model](https://download.openmmlab.com/mmclassification/v0/swin-transformer/swin-large_8xb8_cub_384px_20220307-1bbaee6a.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/swin-transformer/swin-large_8xb8_cub_384px_20220307-1bbaee6a.json) |

```bibtex
}
```

# MobileOne

> [An Improved One millisecond Mobile Backbone](https://arxiv.org/abs/2206.04040)

Mobileone is proposed by apple and based on reparameterization. On the apple chips, the accuracy of the model is close to 0.76 on the ImageNet dataset when the latency is less than 1ms. Its main improvements based on [RepVGG](../repvgg) are fllowing:

- Reparameterization using Depthwise convolution and Pointwise convolution instead of normal convolution.
- Removal of the residual structure which is not friendly to access memory.

<img src="https://user-images.githubusercontent.com/18586273/183552452-74657532-f461-48f7-9aa7-c23f006cdb07.png" width="40%"/>
</div>

<details>

<br>

Efficient neural network backbones for mobile devices are often optimized for metrics such as FLOPs or parameter count. However, these metrics may not correlate well with latency of the network when deployed on a mobile device. Therefore, we perform extensive analysis of different metrics by deploying several mobile-friendly networks on a mobile device. We identify and analyze architectural and optimization bottlenecks in recent efficient neural networks and provide ways to mitigate these bottlenecks. To this end, we design an efficient backbone MobileOne, with variants achieving an inference time under 1 ms on an iPhone12 with 75.9% top-1 accuracy on ImageNet. We show that MobileOne achieves state-of-the-art performance within the efficient architectures while being many times faster on mobile. Our best model obtains similar performance on ImageNet as MobileFormer while being 38x faster. Our model obtains 2.3% better top-1 accuracy on ImageNet than EfficientNet at similar latency. Furthermore, we show that our model generalizes to multiple tasks - image classification, object detection, and semantic segmentation with significant improvements in latency and accuracy as compared to existing efficient architectures when deployed on a mobile device.

</br>

```python

predict = inference_model('mobileone-s0_8xb32_in1k', 'demo/bird.JPEG')
```

```python

model = get_model('mobileone-s0_8xb32_in1k', pretrained=True)
```

Train:

```shell
python tools/train.py configs/mobileone/mobileone-s0_8xb32_in1k.py
```

Test:

```shell
python tools/test.py configs/mobileone/mobileone-s0_8xb32_in1k.py https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s0_8xb32_in1k_20221110-0bc94952.pth

```

Test:

```shell
```

| `mobileone-s0_8xb32_in1k` | From scratch |    2.08    |   0.27    |   71.34   |   89.87   | [config](mobileone-s0_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s0_8xb32_in1k_20221110-0bc94952.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s0_8xb32_in1k_20221110-0bc94952.json) |
| `mobileone-s1_8xb32_in1k` | From scratch |    4.76    |   0.82    |   75.72   |   92.54   | [config](mobileone-s1_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s1_8xb32_in1k_20221110-ceeef467.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s1_8xb32_in1k_20221110-ceeef467.json) |

| `mobileone-s2_8xb32_in1k` | From scratch |    7.81    |   1.30    |   77.37   |   93.34   | [config](mobileone-s2_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s2_8xb32_in1k_20221110-9c7ecb97.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s2_8xb32_in1k_20221110-9c7ecb97.json) |
| `mobileone-s3_8xb32_in1k` | From scratch |   10.08    |   1.89    |   78.06   |   93.83   | [config](mobileone-s3_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s3_8xb32_in1k_20221110-c95eb3bf.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s3_8xb32_in1k_20221110-c95eb3bf.json) |

| `mobileone-s4_8xb32_in1k` | From scratch |   14.84    |   2.98    |   79.69   |   94.46   | [config](mobileone-s4_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s4_8xb32_in1k_20221110-28d888cb.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s4_8xb32_in1k_20221110-28d888cb.json) |

```bibtex
@article{mobileone2022,
  title={An Improved One millisecond Mobile Backbone},
  author={Vasu, Pavan Kumar Anasosalu and Gabriel, James and Zhu, Jeff and Tuzel, Oncel and Ranjan, Anurag},
  journal={arXiv preprint arXiv:2206.04040},
}
```

<details>

```bibtex
}
```

<details>

```bibtex

| [rsn_18](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_rsn18_8xb32-210e_coco-256x192.py) |  256x192   | 0.704 |      0.887      |      0.781      | 0.773 |      0.927      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_rsn18_8xb32-210e_coco-256x192-9049ed09_20221013.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_rsn18_8xb32-210e_coco-256x192_20221013.log) |
| [rsn_50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_rsn50_8xb32-210e_coco-256x192.py) |  256x192   | 0.724 |      0.894      |      0.799      | 0.790 |      0.935      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_rsn50_8xb32-210e_coco-256x192-c35901d5_20221013.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_rsn50_8xb32-210e_coco-256x192_20221013.log) |

| [2xrsn_50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_2xrsn50_8xb32-210e_coco-256x192.py) |  256x192   | 0.748 |      0.900      |      0.821      | 0.810 |      0.939      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_2xrsn50_8xb32-210e_coco-256x192-9ede341e_20221013.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_2xrsn50_8xb32-210e_coco-256x192_20221013.log) |
| [3xrsn_50](/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_3xrsn50_8xb32-210e_coco-256x192.py) |  256x192   | 0.750 |      0.900      |      0.824      | 0.814 |      0.941      | [ckpt](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_3xrsn50_8xb32-210e_coco-256x192-c3e3c4fe_20221013.pth) | [log](https://download.openmmlab.com/mmpose/v1/body_2d_keypoint/topdown_heatmap/coco/td-hm_3xrsn50_8xb32-210e_coco-256x192_20221013.log) |

In terms of model deployment, most ML models require some preprocessing steps on the input data and postprocessing steps on the output to get structured output. MMDeploy sdk provides a lot of pre-processing and post-processing process. When you convert and deploy a model, you can enjoy the convenience brought by mmdeploy sdk.

You can refer to [convert model](../02-how-to-run/convert_model.md) for more details.

After model conversion with `--dump-info`, the structure of model directory (tensorrt model) is as follows. If you convert to other backend, the structure will be slightly different. The two images are for quick conversion validation.

```bash
```

The files related to sdk are:

- deploy.json    // model information.
- pipeline.json  // inference information.

```

- end2end.engine // model file for tensort, will be different for other backends.

SDK can read the model directory directly or you can pack the related files to zip archive for better distribution or encryption. To read the zip file, the sdk should build with `-DMMDEPLOY_ZIP_MODEL=ON`

## SDK Inference

Generally speaking, there are three steps to inference a model.

- Create a pipeline
- Load the data
- Model inference

We use `classifier` as an example to show these three steps.

### Create a pipeline

#### Load model from disk

```cpp

```

#### Load model from memory

```cpp

```

### Load the data

```cpp
```

### Model inference

```cpp
```

# H2RBox

> [H2RBox: Horizontal Box Annotation is All You Need for Oriented Object Detection](https://arxiv.org/abs/2210.06742)

<img src="https://github.com/yangxue0827/h2rbox-mmrotate/blob/main/configs/h2rbox/pipeline.png" width="800"/>
</div>

Oriented object detection emerges in many applications from aerial images to autonomous driving, while many existing detection benchmarks are annotated with horizontal bounding box only which is also less costive than fine-grained rotated box, leading to a gap between the readily available training corpus and the rising demand for oriented object detection.  This paper proposes a simple yet effective oriented object detection approach called H2RBox merely using horizontal box annotation for weakly-supervised training, which closes the above gap and shows competitive performance even against those trained with rotated boxes.  The cores of our method are weakly- and self-supervised learning, which predicts the angle of the object by learning the consistency of two different views. To our best knowledge, H2RBox is the first horizontal box annotation-based oriented object detector. Compared to an alternative i.e. horizontal box-supervised instance segmentation with our post adaption to oriented object detection, our approach is not susceptible to the prediction quality of mask and can perform more robustly in complex scenes containing a large number of dense objects and outliers. Experimental results show that H2RBox has significant performance and speed advantages over horizontal box-supervised instance segmentation methods, as well as lower memory requirements. While compared to rotated box-supervised oriented object detectors, our method shows very close performance and speed, and even surpasses them in some cases.

DOTA1.0

|         Backbone         |  mAP  | Angle | lr schd | Mem (GB) | Inf Time (fps) | MS  | Batch Size |                                      Configs                                      |                                                                                                                                                     Download                                                                                                                                                     |

| :----------------------: | :---: | :---: | :-----: | :------: | :------------: | :-: | :--------: | :-------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ResNet50 (1024,1024,200) | 68.75 | le90  |   1x    |   6.25   |                |  -  |     2      |    [h2rbox-le90_r50_fpn_adamw-1x_dota](./h2rbox-le90_r50_fpn_adamw-1x_dota.py)    |       [model](https://download.openmmlab.com/mmrotate/v1.0/h2rbox/h2rbox-le90_r50_fpn_adamw-1x_dota/h2rbox-le90_r50_fpn_adamw-1x_dota-d02c933a.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/h2rbox/h2rbox-le90_r50_fpn_adamw-1x_dota/h2rbox-le90_r50_fpn_adamw-1x_dota-20221124_153420.json)       |

| ResNet50 (1024,1024,200) | 71.31 | le90  |   3x    |   6.64   |                |  -  |     2      |    [h2rbox-le90_r50_fpn_adamw-3x_dota](./h2rbox-le90_r50_fpn_adamw-3x_dota.py)    |       [model](https://download.openmmlab.com/mmrotate/v1.0/h2rbox/h2rbox-le90_r50_fpn_adamw-3x_dota/h2rbox-le90_r50_fpn_adamw-3x_dota-8bca2d7f.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/h2rbox/h2rbox-le90_r50_fpn_adamw-3x_dota/h2rbox-le90_r50_fpn_adamw-3x_dota-20221124_180458.json)       |
| ResNet50 (1024,1024,200) | 74.43 | le90  |   1x    |   6.12   |       -        |  √  |     2      | [h2rbox-le90_r50_fpn_adamw-1x_dota-ms](./h2rbox-le90_r50_fpn_adamw-1x_dota-ms.py) | [model](https://download.openmmlab.com/mmrotate/v1.0/h2rbox/h2rbox-le90_r50_fpn_adamw-1x_dota-ms/h2rbox-le90_r50_fpn_adamw-1x_dota-ms-30dcdc68.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/h2rbox/h2rbox-le90_r50_fpn_adamw-1x_dota-ms/h2rbox-le90_r50_fpn_adamw-1x_dota-ms-20221124_224240.json) |

DOTA1.5

DOTA1.5

|         Backbone         |  mAP  | Angle | lr schd | Mem (GB) | Inf Time (fps) | MS  | Batch Size |                                          Configs                                          |                                                                                                                                                     Download                                                                                                                                                     |

| ResNet50 (1024,1024,200) | 60.33 | le90  |   1x    |   6.83   |                |  -  |     2      | [h2rbox-le90_r50_fpn_adamw-1x_dotav15](./dotav15/h2rbox-le90_r50_fpn_adamw-1x_dotav15.py) | [model](https://download.openmmlab.com/mmrotate/v1.0/h2rbox/h2rbox-le90_r50_fpn_adamw-1x_dotav15/h2rbox-le90_r50_fpn_adamw-1x_dotav15-5f2178e6.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/h2rbox/h2rbox-le90_r50_fpn_adamw-1x_dotav15/h2rbox-le90_r50_fpn_adamw-1x_dotav15-20221125_173828.json) |

| ResNet50 (1024,1024,200) | 62.65 | le90  |   1x    |   6.34   |                |  -  |     2      | [h2rbox-le90_r50_fpn_adamw-3x_dotav15](./dotav15/h2rbox-le90_r50_fpn_adamw-3x_dotav15.py) | [model](https://download.openmmlab.com/mmrotate/v1.0/h2rbox/h2rbox-le90_r50_fpn_adamw-3x_dotav15/h2rbox-le90_r50_fpn_adamw-3x_dotav15-d8337c8e.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/h2rbox/h2rbox-le90_r50_fpn_adamw-3x_dotav15/h2rbox-le90_r50_fpn_adamw-3x_dotav15-20221126_135218.json) |

DOTA2.0

DOTA2.0

|         Backbone         |  mAP  | Angle | lr schd | Mem (GB) | Inf Time (fps) | MS  | Batch Size |                                        Configs                                         |                                                                                                                                                   Download                                                                                                                                                   |
| :----------------------: | :---: | :---: | :-----: | :------: | :------------: | :-: | :--------: | :------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

| ResNet50 (1024,1024,200) | 47.08 | le90  |   1x    |   7.58   |                |  -  |     2      | [h2rbox-le90_r50_fpn_adamw-1x_dotav2](./dotav2/h2rbox-le90_r50_fpn_adamw-1x_dotav2.py) | [model](https://download.openmmlab.com/mmrotate/v1.0/h2rbox/h2rbox-le90_r50_fpn_adamw-1x_dotav2/h2rbox-le90_r50_fpn_adamw-1x_dotav2-366ced3d.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/h2rbox/h2rbox-le90_r50_fpn_adamw-1x_dotav2/h2rbox-le90_r50_fpn_adamw-1x_dotav2-20221126_141507.json) |
| ResNet50 (1024,1024,200) | 50.20 | le90  |   1x    |   7.74   |                |  -  |     2      | [h2rbox-le90_r50_fpn_adamw-3x_dotav2](./dotav2/h2rbox-le90_r50_fpn_adamw-3x_dotav2.py) | [model](https://download.openmmlab.com/mmrotate/v1.0/h2rbox/h2rbox-le90_r50_fpn_adamw-3x_dotav2/h2rbox-le90_r50_fpn_adamw-3x_dotav2-85bf9bfa.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/h2rbox/h2rbox-le90_r50_fpn_adamw-3x_dotav2/h2rbox-le90_r50_fpn_adamw-3x_dotav2-20221126_225316.json) |

DIOR

DIOR

|         Backbone         | AP50:95 | AP50  | AP75  | Angle | lr schd | Mem (GB) | Inf Time (fps) | MS  | Batch Size |                                     Configs                                      |                                                                                                                                               Download                                                                                                                                               |

| :----------------------: | :-----: | :---: | :---: | :---: | :-----: | :------: | :------------: | :-: | :--------: | :------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ResNet50 (1024,1024,200) |  33.06  | 57.40 | 32.50 | le90  |   1x    |   3.83   |                |  -  |     2      | [h2rbox-le90_r50_fpn_adamw-1x_dior](./dior/h2rbox-le90_r50_fpn_adamw-1x_dior.py) | [model](https://download.openmmlab.com/mmrotate/v1.0/h2rbox/h2rbox-le90_r50_fpn_adamw-1x_dior/h2rbox-le90_r50_fpn_adamw-1x_dior-949b0e4c.pth) \| [log](https://download.openmmlab.com/mmrotate/v1.0/h2rbox/h2rbox-le90_r50_fpn_adamw-1x_dior/h2rbox-le90_r50_fpn_adamw-1x_dior-20221130_204038.json) |

- `Inf Time` was tested on a single RTX3090.
- [Original PyTorch Implementation (Paper) for H2RBox](https://github.com/yangxue0827/h2rbox-mmrotate)
- [Jittor Implementation for H2RBox](https://github.com/yangxue0827/h2rbox-jittor)
- [JDet Implementation for H2RBox](https://github.com/Jittor/JDet)
- The memory log of MMRotate 1.x is dynamic, so the last iteration is recorded here.

```
@article{yang2022h2rbox,
  title={H2RBox: Horizontal Box Annotation is All You Need for Oriented Object Detection},
  author={Yang, Xue and Zhang, Gefan and Li, Wentong and Wang, Xuehui and Zhou, Yue and Yan, Junchi},
  journal={arXiv preprint arXiv:2210.06742},
}

```

# SVTR

> [SVTR: Scene Text Recognition with a Single Visual Model](https://arxiv.org/abs/2205.00159)

Dominant scene text recognition models commonly contain two building blocks, a visual model for feature extraction and a sequence model for text transcription. This hybrid architecture, although accurate, is complex and less efficient. In this study, we propose a Single Visual model for Scene Text recognition within the patch-wise image tokenization framework, which dispenses with the sequential modeling entirely. The method, termed SVTR, firstly decomposes an image text into small patches named character components. Afterward, hierarchical stages are recurrently carried out by component-level mixing, merging and/or combining. Global and local mixing blocks are devised to perceive the inter-character and intra-character patterns, leading to a multi-grained character component perception. Thus, characters are recognized by a simple linear prediction. Experimental results on both English and Chinese scene text recognition tasks demonstrate the effectiveness of SVTR. SVTR-L (Large) achieves highly competitive accuracy in English and outperforms existing methods by a large margin in Chinese, while running faster. In addition, SVTR-T (Tiny) is an effective and much smaller model, which shows appealing speed at inference.

<img src="https://user-images.githubusercontent.com/22607038/210541576-025df5d5-f4d2-4037-82e0-246cf8cd3c25.png"/>
</div>

## Dataset

|                              Methods                              |        | Regular Text |           |     |           | Irregular Text |        |                                   download                                   |

| :---------------------------------------------------------------: | :----: | :----------: | :-------: | :-: | :-------: | :------------: | :----: | :--------------------------------------------------------------------------: |
|                                                                   | IIIT5K |     SVT      | IC13-1015 |     | IC15-2077 |      SVTP      |  CT80  |                                                                              |
|    [SVTR-tiny](/configs/textrecog/svtr/svtr-tiny_20e_st_mj.py)    |   -    |      -       |     -     |     |     -     |       -        |   -    |                                      -                                       |

|   [SVTR-small](/configs/textrecog/svtr/svtr-small_20e_st_mj.py)   | 0.8553 |    0.9026    |  0.9448   |     |  0.7496   |     0.8496     | 0.8854 | [model](https://download.openmmlab.com/mmocr/textrecog/svtr/svtr-small_20e_st_mj/svtr-small_20e_st_mj-35d800d6.pth) \| [log](https://download.openmmlab.com/mmocr/textrecog/svtr/svtr-small_20e_st_mj/20230105_184454.log) |
| [SVTR-small-TTA](/configs/textrecog/svtr/svtr-small_20e_st_mj.py) | 0.8397 |    0.8964    |  0.9241   |     |  0.7597   |     0.8124     | 0.8646 |                                                                              |
|    [SVTR-base](/configs/textrecog/svtr/svtr-base_20e_st_mj.py)    | 0.8570 |    0.9181    |  0.9438   |     |  0.7448   |     0.8388     | 0.9028 | [model](https://download.openmmlab.com/mmocr/textrecog/svtr/svtr-base_20e_st_mj/svtr-base_20e_st_mj-ea500101.pth) \| [log](https://download.openmmlab.com/mmocr/textrecog/svtr/svtr-base_20e_st_mj/20221227_175415.log) |

|  [SVTR-base-TTA](/configs/textrecog/svtr/svtr-base_20e_st_mj.py)  | 0.8517 |    0.9011    |  0.9379   |     |  0.7569   |     0.8279     | 0.8819 |                                                                              |
|   [SVTR-large](/configs/textrecog/svtr/svtr-large_20e_st_mj.py)   |   -    |      -       |     -     |     |     -     |       -        |   -    |                                      -                                       |

```{note}
The implementation and configuration follow the original code and paper, but there is still a gap between the reproduced results and the official ones. We appreciate any suggestions to improve its performance.
```

```bibtex
@inproceedings{ijcai2022p124,
  title     = {SVTR: Scene Text Recognition with a Single Visual Model},
  author    = {Du, Yongkun and Chen, Zhineng and Jia, Caiyan and Yin, Xiaoting and Zheng, Tianlun and Li, Chenxia and Du, Yuning and Jiang, Yu-Gang},

author    = {Du, Yongkun and Chen, Zhineng and Jia, Caiyan and Yin, Xiaoting and Zheng, Tianlun and Li, Chenxia and Du, Yuning and Jiang, Yu-Gang},
  booktitle = {Proceedings of the Thirty-First International Joint Conference on
               Artificial Intelligence, {IJCAI-22}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Lud De Raedt},
  pages     = {884--890},
  year      = {2022},
  month     = {7},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2022/124},
  url       = {https://doi.org/10.24963/ijcai.2022/124},
}

```

# Data Transforms (Need to update)

## Design of Data transforms pipeline

The data transforms pipeline and the dataset is decomposed. Usually a dataset
defines how to process the annotations and a data transforms pipeline defines all the steps to prepare a data dict.
A pipeline consists of a sequence of data transforms. Each operation takes a dict as input and also output a dict for the next transform.

Here is a pipeline example for Faster R-CNN.

```python

```python
    dict(
        with_bbox=True),  # Whether to use bounding box, True for detection
    dict(
        type='Resize',  # Pipeline that resize the images and their annotations
        scale=(1333, 800),  # The largest scale of image
        ),
    dict(
        type='RandomFlip',  # Augmentation pipeline that flip the images and their annotations
]

prob=0.5),  # The probability to flip
]
    dict(type='Resize', scale=(1333, 800), keep_ratio=True),  # Pipeline that resize the images
    dict(
]
```

# 编解码器

在关键点检测任务中，根据算法的不同，需要利用标注信息，生成不同格式的训练目标，比如归一化的坐标值、一维向量、高斯热图等。同样的，对于模型输出的结果，也需要经过处理转换成标注信息格式。我们一般将标注信息到训练目标的处理过程称为编码，模型输出到标注信息的处理过程称为解码。

编码和解码是一对紧密相关的互逆处理过程。在 MMPose 早期版本中，编码和解码过程往往分散在不同模块里，使其不够直观和统一，增加了学习和维护成本。

MMPose 1.0 中引入了新模块 **编解码器（Codec）** ，将关键点数据的编码和解码过程进行集成，以增加代码的友好度和复用性。

编解码器在工作流程中所处的位置如下所示：

![codec-cn](https://user-images.githubusercontent.com/13503330/187829784-4d5939de-97d7-43cc-b934-c6d17c02d589.png)

一个编解码器主要包含两个部分：

- 编码器
- 解码器

### 编码器

编码器主要负责将处于输入图片尺度的坐标值，编码为模型训练所需要的目标格式，主要包括：

- 归一化的坐标值：用于 Regression-based 方法
- 一维向量：用于 SimCC-based 方法
- 高斯热图：用于 Heatmap-based 方法

以 Regression-based 方法的编码器为例：

```Python

    Args:
            (N, K)

Args:
            (N, K)

    Returns:
        dict:
            (N, K)
    """

```

keypoint_weights = np.where(valid, 1., 0.).astype(np.float32)

```

编码后的数据会在 `PackPoseInputs` 中被转换为 Tensor 格式，并封装到 `data_sample.gt_instance_labels` 中供模型调用，一般主要用于 loss 计算，下面以 `RegressionHead` 中的 `loss()` 为例：

```Python

    ])

])

    ### 后续内容省略 ###
```

### 解码器

解码器主要负责将模型的输出解码为输入图片尺度的坐标值，处理过程与编码器相反。

以 Regression-based 方法的解码器为例：

```Python
    space.

    Args:

    Returns:
        tuple:

    """

"""

    else:

```

默认情况下，`decode()` 方法只提供单个目标数据的解码过程，你也可以通过 `batch_decode()` 来实现批量解码提升执行效率。

## 常见用法

在 MMPose 配置文件中，主要有三处涉及编解码器：

- 定义编解码器
- 生成训练目标
- 模型头部

### 定义编解码器

以回归方法生成归一化的坐标值为例，在配置文件中，我们通过如下方式定义编解码器：

```Python
```

### 生成训练目标

在数据处理阶段生成训练目标时，需要传入编解码器用于编码：

```Python

- 生成训练目标
- 模型头部

### 定义编解码器

```Python
```

### 生成训练目标

```Python
```

### 模型头部

在 MMPose 中，我们在模型头部对模型的输出进行解码，需要传入编解码器用于解码：

```Python
)
```

它们在配置文件中的具体位置如下:

```Python

codec = dict(type='RegressionLabel', input_size=(192, 256))                     ## 定义 ##

        depth=50,
    ),

bgr_to_rgb=True),
        depth=50,
    ),
        decoder=codec),                                                         ## 模型头部 ##
    ))

dict(type='RandomFlip', direction='horizontal'),
    dict(type='GenerateTarget', encoder=codec),   ## 生成训练目标 ##
]
]
```

# PNNX
PyTorch Neural Network eXchange(PNNX) is an open standard for PyTorch model interoperability. PNNX provides an open model format for PyTorch. It defines computation graph as well as high level operators strictly matches PyTorch.

# Rationale
PyTorch is currently one of the most popular machine learning frameworks. We need to deploy the trained AI model to various hardware and environments more conveniently and easily.

Before PNNX, we had the following methods:

1. export to ONNX, and deploy with ONNX-runtime
2. export to ONNX, and convert onnx to inference-framework specific format, and deploy with TensorRT/OpenVINO/ncnn/etc.
3. export to TorchScript, and deploy with libtorch

As far as we know, ONNX has the ability to express the PyTorch model and it is an open standard. People usually use ONNX as an intermediate representation between PyTorch and the inference platform. However, ONNX still has the following fatal problems, which makes the birth of PNNX necessary:

1. ONNX does not have a human-readable and editable file representation, making it difficult for users to easily modify the computation graph or add custom operators.
2. The operator definition of ONNX is not completely in accordance with PyTorch. When exporting some PyTorch operators, glue operators are often added passively by ONNX, which makes the computation graph inconsistent with PyTorch and may impact the inference efficiency.
3. There are a large number of additional parameters designed to be compatible with various ML frameworks in the operator definition in ONNX. These parameters increase the burden of inference implementation on hardware and software.

PNNX tries to define a set of operators and a simple and easy-to-use format that are completely contrasted with the python api of PyTorch, so that the conversion and interoperability of PyTorch models are more convenient.

# Features

1. [Human readable and editable format](#the-pnnxparam-format)

# Features

2. [Plain model binary in storage zip](#the-pnnxbin-format)
3. [One-to-one mapping of PNNX operators and PyTorch python api](#pnnx-operator)
4. [Preserve math expression as one operator](#pnnx-expression-operator)
5. [Preserve torch function as one operator](#pnnx-torch-function-operator)
6. [Preserve miscellaneous module as one operator](#pnnx-module-operator)
7. [Inference via exported PyTorch python code](#pnnx-python-inference)
8. [Tensor shape propagation](#pnnx-shape-propagation)
9. [Model optimization](#pnnx-model-optimization)
10. [Custom operator support](#pnnx-custom-operator)

# Build TorchScript to PNNX converter

1. Install PyTorch and TorchVision c++ library
2. Build PNNX with cmake

# Usage

1. Export your model to TorchScript

```python
import torchvision.models as models

net = models.resnet18(pretrained=True)
net = net.eval()

# Usage

```python

# You could try disabling checking when tracing raises error
# mod = torch.jit.trace(net, x, check_trace=False)
mod = torch.jit.trace(net, x)

mod.save("resnet18.pt")
```

2. Convert TorchScript to PNNX

```shell
pnnx resnet18.pt inputshape=[1,3,224,224]
```

Normally, you will get seven files

```resnet18.pnnx.param``` PNNX graph definition

```resnet18.pnnx.bin``` PNNX model weight

```resnet18_pnnx.py``` PyTorch script for inference, the python code for model construction and weight initialization

```resnet18.pnnx.onnx``` PNNX model in onnx format

```resnet18.ncnn.param``` ncnn graph definition

```resnet18.ncnn.bin``` ncnn model weight

```resnet18_ncnn.py``` pyncnn script for inference

3. Visualize PNNX with Netron

Open https://netron.app/ in browser, and drag resnet18.pnnx.param into it.

4. PNNX command line options

```
Usage: pnnx [model.pt] [(key=value)...]
  pnnxparam=model.pnnx.param
  pnnxbin=model.pnnx.bin
  pnnxpy=model_pnnx.py
  pnnxonnx=model.pnnx.onnx
  ncnnparam=model.ncnn.param
  ncnnbin=model.ncnn.bin
  ncnnpy=model_ncnn.py
  fp16=1
  optlevel=2
  device=cpu/gpu
  inputshape=[1,3,224,224],...
  inputshape2=[1,3,320,320],...
  customop=/home/nihui/.cache/torch_extensions/fused/fused.so,...
  moduleop=models.common.Focus,models.yolo.Detect,...
Sample usage: pnnx mobilenet_v2.pt inputshape=[1,3,224,224]
              pnnx yolov5s.pt inputshape=[1,3,640,640] inputshape2=[1,3,320,320] device=gpu moduleop=models.common.Focus,models.yolo.Detect
```

pnnx yolov5s.pt inputshape=[1,3,640,640] inputshape2=[1,3,320,320] device=gpu moduleop=models.common.Focus,models.yolo.Detect
```

`pnnxparam` (default="*.pnnx.param", * is the model name): PNNX graph definition file

`pnnxbin` (default="*.pnnx.bin"): PNNX model weight

`pnnxpy` (default="*_pnnx.py"): PyTorch script for inference, including model construction and weight initialization code

`pnnxonnx` (default="*.pnnx.onnx"): PNNX model in onnx format

`ncnnparam` (default="*.ncnn.param"): ncnn graph definition

`ncnnbin` (default="*.ncnn.bin"): ncnn model weight

`ncnnpy` (default="*_ncnn.py"): pyncnn script for inference

`fp16` (default=1): save ncnn weight and onnx in fp16 data type

`optlevel` (default=2): graph optimization level 

| Option | Optimization level              |
|--------|---------------------------------|
|   0    | do not apply optimization       |
|   1    | optimization for inference      |
|   2    | optimization more for inference |

`device` (default="cpu"): device type for the input in TorchScript model, cpu or gpu

`inputshape` (Optional): shapes of model inputs. It is used to resolve tensor shapes in model graph. for example, `[1,3,224,224]` for the model with only 1 input, `[1,3,224,224],[1,3,224,224]` for the model that have 2 inputs.

`inputshape2` (Optional): shapes of alternative model inputs, the format is identical to `inputshape`. Usually, it is used with `inputshape` to resolve dynamic shape (-1) in model graph.

`customop` (Optional): list of Torch extensions (dynamic library) for custom operators, separated by ",". For example, `/home/nihui/.cache/torch_extensions/fused/fused.so,...`

`moduleop` (Optional): list of modules to keep as one big operator, separated by ",". for example, `models.common.Focus,models.yolo.Detect`

# The pnnx.param format

```
7767517
4 3
pnnx.Input      input       0 1 0
nn.Conv2d       conv_0      1 1 0 1 bias=1 dilation=(1,1) groups=1 in_channels=12 kernel_size=(3,3) out_channels=16 padding=(0,0) stride=(1,1) @bias=(16)f32 @weight=(16,12,3,3)f32
nn.Conv2d       conv_1      1 1 1 2 bias=1 dilation=(1,1) groups=1 in_channels=16 kernel_size=(2,2) out_channels=20 padding=(2,2) stride=(2,2) @bias=(20)f32 @weight=(20,16,2,2)f32
pnnx.Output     output      1 0 2
```
```
[magic]
```
```
[operator count] [operand count]
```
* operator count : count of the operator line follows
* operand count : count of all operands
### operator line
```
[type] [name] [input count] [output count] [input operands] [output operands] [operator params]
```
* type : type name, such as Conv2d ReLU etc

```
```
* name : name of this operator
* input count : count of the operands this operator needs as input
* output count : count of the operands this operator produces as output
* input operands : name list of all the input blob names, separated by space
* output operands : name list of all the output blob names, separated by space
* operator params : key=value pair list, separated by space, operator weights are prefixed by ```@``` symbol, tensor shapes are prefixed by ```#``` symbol, input parameter keys are prefixed by ```$```

# The pnnx.bin format

pnnx.bin file is a zip file with store-only mode(no compression)

weight binary file has its name composed by operator name and weight name

For example, ```nn.Conv2d       conv_0      1 1 0 1 bias=1 dilation=(1,1) groups=1 in_channels=12 kernel_size=(3,3) out_channels=16 padding=(0,0) stride=(1,1) @bias=(16) @weight=(16,12,3,3)``` would pull conv_0.weight and conv_0.bias into pnnx.bin zip archive.

weight binaries can be listed or modified with any archive application eg. 7zip

![pnnx.bin](https://raw.githubusercontent.com/nihui/ncnn-assets/master/pnnx/pnnx.bin.png)

# PNNX operator
PNNX always preserve operators from what PyTorch python api provides.

Here is the netron visualization comparison among ONNX, TorchScript and PNNX with the original PyTorch python code shown.

```python

class Model(nn.Module):
        super(Model, self).__init__()

```python

        self.attention = nn.MultiheadAttention(embed_dim=256, num_heads=32)

        x, _ = self.attention(x, x, x)
        return x
```

|ONNX|TorchScript|PNNX|
|----|---|---|
|![MultiheadAttention.onnx](https://raw.githubusercontent.com/nihui/ncnn-assets/master/pnnx/MultiheadAttention.onnx.png)|![MultiheadAttention.pt](https://raw.githubusercontent.com/nihui/ncnn-assets/master/pnnx/MultiheadAttention.pt.png)|![MultiheadAttention.pnnx](https://raw.githubusercontent.com/nihui/ncnn-assets/master/pnnx/MultiheadAttention.pnnx.png)|

# PNNX expression operator
PNNX trys to preserve expression from what PyTorch python code writes.

```python

def foo(x, y):
    return torch.sqrt((2 * x + y) / 12)
```

```python

```

|![math.onnx](https://raw.githubusercontent.com/nihui/ncnn-assets/master/pnnx/math.onnx.png)|![math.pt](https://raw.githubusercontent.com/nihui/ncnn-assets/master/pnnx/math.pt.png)|![math.pnnx](https://raw.githubusercontent.com/nihui/ncnn-assets/master/pnnx/math.pnnx.png)|

# PNNX torch function operator
PNNX trys to preserve torch functions and Tensor member functions as one operator from what PyTorch python api provides.

```python

        x = F.normalize(x, eps=1e-3)
        return x
```

        return x
```

|![function.onnx](https://raw.githubusercontent.com/nihui/ncnn-assets/master/pnnx/function.onnx.png)|![function.pt](https://raw.githubusercontent.com/nihui/ncnn-assets/master/pnnx/function.pt.png)|![function.pnnx](https://raw.githubusercontent.com/nihui/ncnn-assets/master/pnnx/function.pnnx.png)|

# PNNX module operator
Users could ask PNNX to keep module as one big operator when it has complex logic.

The process is optional and could be enabled via moduleop command line option.

After pass_level0, all modules will be presented in terminal output, then you can pick the interesting ones as module operators.
```
############# pass_level0
inline module = models.common.Bottleneck
inline module = models.common.C3
inline module = models.common.Concat
inline module = models.common.Conv

```
inline module = models.common.Focus
inline module = models.common.SPP
inline module = models.yolo.Detect
inline module = utils.activations.SiLU
```

```bash
pnnx yolov5s.pt inputshape=[1,3,640,640] moduleop=models.common.Focus,models.yolo.Detect
```

```python

class Focus(nn.Module):
    # Focus wh information into c-space
    def __init__(self, c1, c2, k=1, s=1, p=None, g=1, act=True):  # ch_in, ch_out, kernel, stride, padding, groups
        self.conv = Conv(c1 * 4, c2, k, s, p, g, act)

    def forward(self, x):  # x(b,c,w,h) -> y(b,4c,w/2,h/2)

super().__init__()

        return self.conv(torch.cat([x[..., ::2, ::2], x[..., 1::2, ::2], x[..., ::2, 1::2], x[..., 1::2, 1::2]], 1))
```

|ONNX|TorchScript|PNNX|PNNX with module operator|
|![focus.onnx](https://raw.githubusercontent.com/nihui/ncnn-assets/master/pnnx/focus.onnx.png)|![focus.pt](https://raw.githubusercontent.com/nihui/ncnn-assets/master/pnnx/focus.pt.png)|![focus.pnnx](https://raw.githubusercontent.com/nihui/ncnn-assets/master/pnnx/focus.pnnx.png)|![focus.pnnx2](https://raw.githubusercontent.com/nihui/ncnn-assets/master/pnnx/focus.pnnx2.png)|

# PNNX python inference

A python script will be generated by default when converting TorchScript to pnnx.

This script is the python code representation of PNNX and can be used for model inference.

There are some utility functions for loading weight binary from pnnx.bin.

You can even export the model TorchScript AGAIN from this generated code!

```python

        self.linear_0 = nn.Linear(in_features=128, out_features=256, bias=True)
        self.linear_1 = nn.Linear(in_features=256, out_features=4, bias=True)

        x = self.linear_0(x)
        x = F.leaky_relu(x, 0.15)
        x = self.linear_1(x)
        return x
```

```python
import os
import tempfile, zipfile

        self.linear_0 = nn.Linear(bias=True, in_features=128, out_features=256)
        self.linear_1 = nn.Linear(bias=True, in_features=256, out_features=4)

        archive = zipfile.ZipFile('../../function.pnnx.bin', 'r')
        self.linear_0.bias = self.load_pnnx_bin_as_parameter(archive, 'linear_0.bias', (256), 'float32')
        self.linear_0.weight = self.load_pnnx_bin_as_parameter(archive, 'linear_0.weight', (256,128), 'float32')
        self.linear_1.bias = self.load_pnnx_bin_as_parameter(archive, 'linear_1.bias', (4), 'float32')
        self.linear_1.weight = self.load_pnnx_bin_as_parameter(archive, 'linear_1.weight', (4,256), 'float32')
        archive.close()

    def load_pnnx_bin_as_parameter(self, archive, key, shape, dtype):

archive.close()

        return nn.Parameter(self.load_pnnx_bin_as_tensor(archive, key, shape, dtype))

    def load_pnnx_bin_as_tensor(self, archive, key, shape, dtype):
        fd, tmppath = tempfile.mkstemp()
        with os.fdopen(fd, 'wb') as tmpf, archive.open(key) as keyfile:
            tmpf.write(keyfile.read())
        m = np.memmap(tmppath, dtype=dtype, mode='r', shape=shape).copy()
        os.remove(tmppath)
        return torch.from_numpy(m)

    def forward(self, v_x_1):
        v_7 = self.linear_0(v_x_1)
        v_input_1 = F.leaky_relu(input=v_7, negative_slope=0.150000)
        v_12 = self.linear_1(v_input_1)
        return v_12
```

# PNNX shape propagation
Users could ask PNNX to resolve all tensor shapes in model graph and constify some common expressions involved when tensor shapes are known.

The process is optional and could be enabled via inputshape command line option.

```bash

```bash
pnnx shufflenet_v2_x1_0.pt inputshape=[1,3,224,224]
```

```python
def channel_shuffle(x: Tensor, groups: int) -> Tensor:
    batchsize, num_channels, height, width = x.size()
    channels_per_group = num_channels // groups

    # reshape
    x = x.view(batchsize, groups, channels_per_group, height, width)

    x = torch.transpose(x, 1, 2).contiguous()

    # flatten
    x = x.view(batchsize, -1, height, width)

    return x
```

|without shape propagation|with shape propagation|
|---|---|
|![noshapeinfer](https://raw.githubusercontent.com/nihui/ncnn-assets/master/pnnx/noshapeinfer.png)|![shapeinfer](https://raw.githubusercontent.com/nihui/ncnn-assets/master/pnnx/shapeinfer.pnnx.png)|

# PNNX model optimization

|ONNX|TorchScript|PNNX without optimization|PNNX with optimization|

|![optlessonnx](https://raw.githubusercontent.com/nihui/ncnn-assets/master/pnnx/optless.onnx.png)|![optlesspt](https://raw.githubusercontent.com/nihui/ncnn-assets/master/pnnx/optless.pt.png)|![optless](https://raw.githubusercontent.com/nihui/ncnn-assets/master/pnnx/optless.pnnx.png)|![opt](https://raw.githubusercontent.com/nihui/ncnn-assets/master/pnnx/opt.pnnx.png)|

# PNNX custom operator

```python
import os

from torch.autograd import Function
from torch.utils.cpp_extension import load, _import_module_from_library

module_path = os.path.dirname(__file__)
upfirdn2d_op = load(
    'upfirdn2d',
    sources=[
        os.path.join(module_path, 'upfirdn2d.cpp'),
        os.path.join(module_path, 'upfirdn2d_kernel.cu'),
    ],
    is_python_module=False
)

def upfirdn2d(input, kernel, up=1, down=1, pad=(0, 0)):
    pad_x0 = pad[0]
    pad_x1 = pad[1]

os.path.join(module_path, 'upfirdn2d_kernel.cu'),
    ],
)

    pad_y0 = pad[0]
    pad_y1 = pad[1]

    kernel_h, kernel_w = kernel.shape
    batch, channel, in_h, in_w = input.shape

    input = input.reshape(-1, in_h, in_w, 1)

    out_h = (in_h * up + pad_y0 + pad_y1 - kernel_h) // down + 1
    out_w = (in_w * up + pad_x0 + pad_x1 - kernel_w) // down + 1

    out = torch.ops.upfirdn2d_op.upfirdn2d(input, kernel, up, up, down, down, pad_x0, pad_x1, pad_y0, pad_y1)

    out = out.view(-1, channel, out_h, out_w)

    return out
```

```cpp
#include <torch/extension.h>

torch::Tensor upfirdn2d(const torch::Tensor& input, const torch::Tensor& kernel,
                        int64_t up_x, int64_t up_y, int64_t down_x, int64_t down_y,
                        int64_t pad_x0, int64_t pad_x1, int64_t pad_y0, int64_t pad_y1) {
    // operator body
}

int64_t up_x, int64_t up_y, int64_t down_x, int64_t down_y,
}

TORCH_LIBRARY(upfirdn2d_op, m) {
    m.def("upfirdn2d", upfirdn2d);
}
```

<img src="https://raw.githubusercontent.com/nihui/ncnn-assets/master/pnnx/customop.pnnx.png" width="400" />

# Supported PyTorch operator status

| torch.nn        | Is Supported | Export to ncnn |
|---------------------------|----|---|
|nn.AdaptiveAvgPool1d       | :heavy_check_mark: | :heavy_check_mark: |
|nn.AdaptiveAvgPool2d       | :heavy_check_mark: | :heavy_check_mark: |
|nn.AdaptiveAvgPool3d       | :heavy_check_mark: | :heavy_check_mark: |
|nn.AdaptiveMaxPool1d       | :heavy_check_mark: | :heavy_check_mark: |
|nn.AdaptiveMaxPool2d       | :heavy_check_mark: | :heavy_check_mark: |
|nn.AdaptiveMaxPool3d       | :heavy_check_mark: | :heavy_check_mark: |

|nn.AlphaDropout            | :heavy_check_mark: | :heavy_check_mark: |
|nn.AvgPool1d               | :heavy_check_mark: | :heavy_check_mark:* |
|nn.AvgPool2d               | :heavy_check_mark: | :heavy_check_mark:* |
|nn.AvgPool3d               | :heavy_check_mark: | :heavy_check_mark:* |
|nn.BatchNorm1d             | :heavy_check_mark: | :heavy_check_mark: |
|nn.BatchNorm2d             | :heavy_check_mark: | :heavy_check_mark: |
|nn.BatchNorm3d             | :heavy_check_mark: | :heavy_check_mark: |
|nn.Bilinear                |   |
|nn.CELU                    | :heavy_check_mark: |
|nn.ChannelShuffle          | :heavy_check_mark: | :heavy_check_mark: |
|nn.ConstantPad1d           | :heavy_check_mark: | :heavy_check_mark: |
|nn.ConstantPad2d           | :heavy_check_mark: | :heavy_check_mark: |

|nn.ConstantPad3d           | :heavy_check_mark: | :heavy_check_mark: |
|nn.Conv1d                  | :heavy_check_mark: | :heavy_check_mark: |
|nn.Conv2d                  | :heavy_check_mark: | :heavy_check_mark: |
|nn.Conv3d                  | :heavy_check_mark: | :heavy_check_mark: |
|nn.ConvTranspose1d         | :heavy_check_mark: | :heavy_check_mark: |
|nn.ConvTranspose2d         | :heavy_check_mark: | :heavy_check_mark: |
|nn.ConvTranspose3d         | :heavy_check_mark: | :heavy_check_mark: |
|nn.CosineSimilarity        |   |
|nn.Dropout                 | :heavy_check_mark: | :heavy_check_mark: |
|nn.Dropout2d               | :heavy_check_mark: | :heavy_check_mark: |
|nn.Dropout3d               | :heavy_check_mark: | :heavy_check_mark: |
|nn.ELU                     | :heavy_check_mark: | :heavy_check_mark: |

|nn.Embedding               | :heavy_check_mark: | :heavy_check_mark: |
|nn.EmbeddingBag            |   |
|nn.Flatten                 | :heavy_check_mark: |
|nn.Fold                    | :heavy_check_mark: | :heavy_check_mark: |
|nn.FractionalMaxPool2d     |   |
|nn.FractionalMaxPool3d     |   |
|nn.GELU                    | :heavy_check_mark: | :heavy_check_mark: |
|nn.GLU                     | :heavy_check_mark: | :heavy_check_mark: |
|nn.GroupNorm               | :heavy_check_mark: | :heavy_check_mark: |
|nn.GRU                     | :heavy_check_mark: | :heavy_check_mark: |
|nn.GRUCell                 |   |
|nn.Hardshrink              | :heavy_check_mark: |
|nn.Hardsigmoid             | :heavy_check_mark: | :heavy_check_mark: |
|nn.Hardswish               | :heavy_check_mark: | :heavy_check_mark: |

|nn.Hardtanh                | :heavy_check_mark: | :heavy_check_mark: |
|nn.Identity                |   |
|nn.InstanceNorm1d          | :heavy_check_mark: |
|nn.InstanceNorm2d          | :heavy_check_mark: | :heavy_check_mark: |
|nn.InstanceNorm3d          | :heavy_check_mark: |
|nn.LayerNorm               | :heavy_check_mark: | :heavy_check_mark: |
|nn.LazyBatchNorm1d         |   |
|nn.LazyBatchNorm2d         |   |
|nn.LazyBatchNorm3d         |   |
|nn.LazyConv1d              |   |
|nn.LazyConv2d              |   |
|nn.LazyConv3d              |   |
|nn.LazyConvTranspose1d     |   |
|nn.LazyConvTranspose2d     |   |
|nn.LazyConvTranspose3d     |   |
|nn.LazyLinear              |   |
|nn.LeakyReLU               | :heavy_check_mark: | :heavy_check_mark: |

|nn.Linear                  | :heavy_check_mark: | :heavy_check_mark: |
|nn.LocalResponseNorm       | :heavy_check_mark: | :heavy_check_mark: |
|nn.LogSigmoid              | :heavy_check_mark: | :heavy_check_mark: |
|nn.LogSoftmax              | :heavy_check_mark: | :heavy_check_mark: |
|nn.LPPool1d                | :heavy_check_mark: |
|nn.LPPool2d                | :heavy_check_mark: |
|nn.LSTM                    | :heavy_check_mark: | :heavy_check_mark: |
|nn.LSTMCell                |   |
|nn.MaxPool1d               | :heavy_check_mark: | :heavy_check_mark: |
|nn.MaxPool2d               | :heavy_check_mark: | :heavy_check_mark: |
|nn.MaxPool3d               | :heavy_check_mark: | :heavy_check_mark: |
|nn.MaxUnpool1d             |   |
|nn.MaxUnpool2d             |   |
|nn.MaxUnpool3d             |   |

|nn.Mish                    | :heavy_check_mark: | :heavy_check_mark: |
|nn.MultiheadAttention      | :heavy_check_mark: | :heavy_check_mark:* |
|nn.PairwiseDistance        |   |
|nn.PixelShuffle            | :heavy_check_mark: | :heavy_check_mark: |
|nn.PixelUnshuffle          | :heavy_check_mark: | :heavy_check_mark: |
|nn.PReLU                   | :heavy_check_mark: | :heavy_check_mark: |
|nn.ReflectionPad1d         | :heavy_check_mark: | :heavy_check_mark: |
|nn.ReflectionPad2d         | :heavy_check_mark: | :heavy_check_mark: |
|nn.ReLU                    | :heavy_check_mark: | :heavy_check_mark: |
|nn.ReLU6                   | :heavy_check_mark: | :heavy_check_mark: |
|nn.ReplicationPad1d        | :heavy_check_mark: | :heavy_check_mark: |
|nn.ReplicationPad2d        | :heavy_check_mark: | :heavy_check_mark: |

|nn.ReplicationPad3d        | :heavy_check_mark: |
|nn.RNN                     | :heavy_check_mark: | :heavy_check_mark:* |
|nn.RNNBase                 |   |
|nn.RNNCell                 |   |
|nn.RReLU                   | :heavy_check_mark: |
|nn.SELU                    | :heavy_check_mark: | :heavy_check_mark: |
|nn.Sigmoid                 | :heavy_check_mark: | :heavy_check_mark: |
|nn.SiLU                    | :heavy_check_mark: | :heavy_check_mark: |
|nn.Softmax                 | :heavy_check_mark: | :heavy_check_mark: |
|nn.Softmax2d               | :heavy_check_mark: | :heavy_check_mark: |
|nn.Softmin                 | :heavy_check_mark: |
|nn.Softplus                | :heavy_check_mark: |
|nn.Softshrink              | :heavy_check_mark: |
|nn.Softsign                | :heavy_check_mark: |
|nn.SyncBatchNorm           |   |

|nn.Tanh                    | :heavy_check_mark: | :heavy_check_mark: |
|nn.Tanhshrink              | :heavy_check_mark: |
|nn.Threshold               | :heavy_check_mark: |
|nn.Transformer             |   |
|nn.TransformerDecoder      |   |
|nn.TransformerDecoderLayer |   |
|nn.TransformerEncoder      |   |
|nn.TransformerEncoderLayer |   |
|nn.Unflatten               |   |
|nn.Unfold                  | :heavy_check_mark: | :heavy_check_mark: |
|nn.Upsample                | :heavy_check_mark: | :heavy_check_mark: |
|nn.UpsamplingBilinear2d    | :heavy_check_mark: | :heavy_check_mark: |
|nn.UpsamplingNearest2d     | :heavy_check_mark: | :heavy_check_mark: |
|nn.ZeroPad2d               | :heavy_check_mark: | :heavy_check_mark: |

| torch.nn.functional | Is Supported | Export to ncnn |

|---------------------------|----|----|
|F.adaptive_avg_pool1d      | :heavy_check_mark: | :heavy_check_mark: |
|F.adaptive_avg_pool2d      | :heavy_check_mark: | :heavy_check_mark: |
|F.adaptive_avg_pool3d      | :heavy_check_mark: | :heavy_check_mark: |
|F.adaptive_max_pool1d      | :heavy_check_mark: | :heavy_check_mark: |
|F.adaptive_max_pool2d      | :heavy_check_mark: | :heavy_check_mark: |
|F.adaptive_max_pool3d      | :heavy_check_mark: | :heavy_check_mark: |
|F.affine_grid              | :heavy_check_mark: |
|F.alpha_dropout            | :heavy_check_mark: | :heavy_check_mark: |
|F.avg_pool1d               | :heavy_check_mark: | :heavy_check_mark:* |
|F.avg_pool2d               | :heavy_check_mark: | :heavy_check_mark:* |
|F.avg_pool3d               | :heavy_check_mark: | :heavy_check_mark:* |

|F.batch_norm               | :heavy_check_mark: | :heavy_check_mark: |
|F.bilinear                 |  |
|F.celu                     | :heavy_check_mark: |
|F.conv1d                   | :heavy_check_mark: | :heavy_check_mark: |
|F.conv2d                   | :heavy_check_mark: | :heavy_check_mark: |
|F.conv3d                   | :heavy_check_mark: | :heavy_check_mark: |
|F.conv_transpose1d         | :heavy_check_mark: | :heavy_check_mark: |
|F.conv_transpose2d         | :heavy_check_mark: | :heavy_check_mark: |
|F.conv_transpose3d         | :heavy_check_mark: | :heavy_check_mark: |
|F.cosine_similarity        |  |
|F.dropout                  | :heavy_check_mark: | :heavy_check_mark: |
|F.dropout2d                | :heavy_check_mark: | :heavy_check_mark: |
|F.dropout3d                | :heavy_check_mark: | :heavy_check_mark: |

|F.elu                      | :heavy_check_mark: | :heavy_check_mark: |
|F.elu_                     | :heavy_check_mark: | :heavy_check_mark: |
|F.embedding                | :heavy_check_mark: | :heavy_check_mark: |
|F.embedding_bag            |  |
|F.feature_alpha_dropout    | :heavy_check_mark: | :heavy_check_mark: |
|F.fold                     | :heavy_check_mark: | :heavy_check_mark: |
|F.fractional_max_pool2d    |  |
|F.fractional_max_pool3d    |  |
|F.gelu                     | :heavy_check_mark: | :heavy_check_mark: |
|F.glu                      | :heavy_check_mark: | :heavy_check_mark: |
|F.grid_sample              | :heavy_check_mark: | :heavy_check_mark: |
|F.group_norm               | :heavy_check_mark: | :heavy_check_mark: |
|F.gumbel_softmax           |  |
|F.hardshrink               | :heavy_check_mark: |

|F.hardsigmoid              | :heavy_check_mark: | :heavy_check_mark: |
|F.hardswish                | :heavy_check_mark: | :heavy_check_mark: |
|F.hardtanh                 | :heavy_check_mark: | :heavy_check_mark: |
|F.hardtanh_                | :heavy_check_mark: | :heavy_check_mark: |
|F.instance_norm            | :heavy_check_mark: | :heavy_check_mark: |
|F.interpolate              | :heavy_check_mark: | :heavy_check_mark: |
|F.layer_norm               | :heavy_check_mark: | :heavy_check_mark: |
|F.leaky_relu               | :heavy_check_mark: | :heavy_check_mark: |
|F.leaky_relu_              | :heavy_check_mark: | :heavy_check_mark: |
|F.linear                   | :heavy_check_mark: | :heavy_check_mark:* |
|F.local_response_norm      | :heavy_check_mark: | :heavy_check_mark: |

|F.logsigmoid               | :heavy_check_mark: | :heavy_check_mark: |
|F.log_softmax              | :heavy_check_mark: | :heavy_check_mark: |
|F.lp_pool1d                | :heavy_check_mark: |
|F.lp_pool2d                | :heavy_check_mark: |
|F.max_pool1d               | :heavy_check_mark: | :heavy_check_mark: |
|F.max_pool2d               | :heavy_check_mark: | :heavy_check_mark: |
|F.max_pool3d               | :heavy_check_mark: | :heavy_check_mark: |
|F.max_unpool1d             |  |
|F.max_unpool2d             |  |
|F.max_unpool3d             |  |
|F.mish                     | :heavy_check_mark: | :heavy_check_mark: |
|F.normalize                | :heavy_check_mark: | :heavy_check_mark: |
|F.one_hot                  |  |
|F.pad                      | :heavy_check_mark: | :heavy_check_mark: |
|F.pairwise_distance        |  |

|F.pdist                    |  |
|F.pixel_shuffle            | :heavy_check_mark: | :heavy_check_mark: |
|F.pixel_unshuffle          | :heavy_check_mark: | :heavy_check_mark: |
|F.prelu                    | :heavy_check_mark: | :heavy_check_mark: |
|F.relu                     | :heavy_check_mark: | :heavy_check_mark: |
|F.relu_                    | :heavy_check_mark: | :heavy_check_mark: |
|F.relu6                    | :heavy_check_mark: | :heavy_check_mark: |
|F.rrelu                    | :heavy_check_mark: |
|F.rrelu_                   | :heavy_check_mark: |
|F.scaled_dot_product_attention | :heavy_check_mark: |                |
|F.selu                     | :heavy_check_mark: | :heavy_check_mark: |
|F.sigmoid                  | :heavy_check_mark: | :heavy_check_mark: |
|F.silu                     | :heavy_check_mark: | :heavy_check_mark: |

|F.softmax                  | :heavy_check_mark: | :heavy_check_mark: |
|F.softmin                  | :heavy_check_mark: |
|F.softplus                 | :heavy_check_mark: |
|F.softshrink               | :heavy_check_mark: |
|F.softsign                 | :heavy_check_mark: |
|F.tanh                     | :heavy_check_mark: | :heavy_check_mark: |
|F.tanhshrink               | :heavy_check_mark: |
|F.threshold                | :heavy_check_mark: |
|F.threshold_               | :heavy_check_mark: |
|F.unfold                   | :heavy_check_mark: | :heavy_check_mark: |
|F.upsample                 | :heavy_check_mark: | :heavy_check_mark: |
|F.upsample_bilinear        | :heavy_check_mark: | :heavy_check_mark: |
|F.upsample_nearest         | :heavy_check_mark: | :heavy_check_mark: |

# Cascade RPN

> [Cascade RPN: Delving into High-Quality Region Proposal Network with Adaptive Convolution](https://arxiv.org/abs/1909.06720)

This paper considers an architecture referred to as Cascade Region Proposal Network (Cascade RPN) for improving the region-proposal quality and detection performance by systematically addressing the limitation of the conventional RPN that heuristically defines the anchors and aligns the features to the anchors. First, instead of using multiple anchors with predefined scales and aspect ratios, Cascade RPN relies on a single anchor per location and performs multi-stage refinement. Each stage is progressively more stringent in defining positive samples by starting out with an anchor-free metric followed by anchor-based metrics in the ensuing stages. Second, to attain alignment between the features and the anchors throughout the stages, adaptive convolution is proposed that takes the anchors in addition to the image features as its input and learns the sampled features guided by the anchors. A simple implementation of a two-stage Cascade RPN achieves AR 13.4 points higher than that of the conventional RPN, surpassing any existing region proposal methods. When adopting to Fast R-CNN and Faster R-CNN, Cascade RPN can improve the detection mAP by 3.1 and 3.5 points, respectively.

<img src="https://user-images.githubusercontent.com/40661020/143872368-1580193a-d19c-4723-a579-c7ed2d5da4d1.png"/>
</div>

### Region proposal performance

| Method | Backbone | Style | Mem (GB) | Train time (s/iter) | Inf time (fps) | AR 1000 |                      Config                      |                                                                    Download                                                                    |
| :----: | :------: | :---: | :------: | :-----------------: | :------------: | :-----: | :----------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------: |

|  CRPN  | R-50-FPN | caffe |    -     |          -          |       -        |  72.0   | [config](./cascade-rpn_r50-caffe_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rpn/crpn_r50_caffe_fpn_1x_coco/cascade_rpn_r50_caffe_fpn_1x_coco-7aa93cef.pth) |

### Detection performance

|    Method    |  Proposal   | Backbone | Style | Schedule | Mem (GB) | Train time (s/iter) | Inf time (fps) | box AP |                            Config                            |                                                                            Download                                                                             |

| :----------: | :---------: | :------: | :---: | :------: | :------: | :-----------------: | :------------: | :----: | :----------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|  Fast R-CNN  | Cascade RPN | R-50-FPN | caffe |    1x    |    -     |          -          |       -        |  39.9  |  [config](./cascade-rpn_fast-rcnn_r50-caffe_fpn_1x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rpn/crpn_fast_rcnn_r50_caffe_fpn_1x_coco/crpn_fast_rcnn_r50_caffe_fpn_1x_coco-cb486e66.pth)   |

| Faster R-CNN | Cascade RPN | R-50-FPN | caffe |    1x    |    -     |          -          |       -        |  40.4  | [config](./cascade-rpn_faster-rcnn_r50-caffe_fpn_1x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rpn/crpn_faster_rcnn_r50_caffe_fpn_1x_coco/crpn_faster_rcnn_r50_caffe_fpn_1x_coco-c8283cca.pth) |

We provide the code for reproducing experiment results of [Cascade RPN](https://arxiv.org/abs/1909.06720).

```latex
  title={Cascade RPN: Delving into High-Quality Region Proposal Network with Adaptive Convolution},
  author={Vu, Thang and Jang, Hyunjun and Pham, Trung X and Yoo, Chang D},
  booktitle={Conference on Neural Information Processing Systems (NeurIPS)},
}
```

# MMagic Viewer

- [Introduction](#introduction)
- [Major features](#major-features)
- [Prerequisites](#prerequisites)
- [Getting Started](#getting-started)
- [Examples](#examples)
- [Contributing](#contributing)

**MMagic Viewer** is a qualitative comparison tools to facilitate your research.

## Major features

- **Patch-based comparison**
  - Crop a patch on multiple images to compare
  - Batch comparison
  - Flexible settings on number of columns and size of images.
  - Save your comparison result
- **Before/After slider comparison**
  - Support both videos and images comparison
  - Record and save comparison results as a video clip

MMagic Viewer works on Linux, Windows and macOS. It requires:

- Python >= 3.6
- PyQt5
- opencv-python (headless version)

Install PyQt5.

```shell
pip install PyQt5
```

Install and check opencv-python version.
If your meet following errors:

```

```shell
```

```
QObject::moveToThread: Current thread is not the object's thread.
Available platform plugins are: xcb... .
```

Please install opencv-python-headless version.

```shell
pip install opencv-python-headless
```

Install MMagic.

```shell
```

Run

```shell
python tools/gui/gui.py
```

**1. Patch-based comparison: batch images**

https://user-images.githubusercontent.com/49083766/199232588-7a07a3d9-725d-48be-89bf-1ffb45bd5d74.mp4

**2. Patch-based comparison: single image**

https://user-images.githubusercontent.com/49083766/199232606-f8539191-4bda-4b2c-975a-59020927abae.mp4

**3. Before/After slider comparison: images**

https://user-images.githubusercontent.com/49083766/199232615-2c56dcf1-0b42-41a5-884c-16a8f28a2647.mp4

**4. Before/After slider comparison: input video frames**

https://user-images.githubusercontent.com/49083766/199232617-e03a06dc-727b-43bb-8110-049d0fff28ba.mp4

**5. Before/After slider comparison: input Mp4 video**

https://user-images.githubusercontent.com/49083766/199232651-87d8064e-cbaf-4d30-b90b-94ee0af7d497.mp4

**6. Before/After slider comparison: record**

https://user-images.githubusercontent.com/49083766/199232634-eca70d28-8437-400a-8ab9-d2fe396b6ea9.mp4

We appreciate all contributions to improve MMagic Viewer. You can create your issue to report bugs or request new features. Welcome to give us suggestions or contribute your codes.

# MMPretrain 中的约定

## 模型命名规则

MMPretrain 按照以下风格进行模型命名，代码库的贡献者需要遵循相同的命名规则。模型名总体分为五个部分：算法信息，模块信息，预训练信息，训练信息和数据信息。逻辑上属于不同部分的单词之间用下划线 `'_'` 连接，同一部分有多个单词用短横线 `'-'` 连接。

```text
```

- `algorithm info`（可选）：算法信息，表示用以训练该模型的主要算法，如 MAE、BEiT 等
- `module info`：模块信息，主要包含模型的主干网络名称，如 resnet、vit 等
- `pretrain info`（可选）：预训练信息，比如预训练模型是在 ImageNet-21k 数据集上训练的等
- `training info`：训练信息，训练策略设置，包括 batch size，schedule 以及数据增强等；
- `data info`：数据信息，数据集名称、模态、输入尺寸等，如 imagenet, cifar 等；

### 算法信息

指用以训练该模型的算法名称，例如：

- `simclr`
- `mocov2`

使用监督图像分类任务训练的模型可以省略这个字段。

### 模块信息

指模型的结构信息，一般主要包含模型的主干网络结构，`neck` 和 `head` 信息一般被省略。例如：

### 预训练信息

如果该模型是在预训练模型基础上，通过微调获得的，我们需要记录预训练模型的一些信息。例如：

- 预训练模型的来源：`fb`、`openai`等。
- 训练预训练模型的方法：`clip`、`mae`、`distill` 等。
- 用于预训练的数据集：`in21k`、`laion2b`等（`in1k`可以省略）
- 训练时长：`300e`、`1600e` 等。

并非所有信息都是必要的，只需要选择用以区分不同的预训练模型的信息即可。

在此字段的末尾，使用 `-pre` 作为标识符，例如 `mae-in21k-pre`。

### 训练信息

训练策略的一些设置，包括训练类型、 `batch size`、 `lr schedule`、 数据增强以及特殊的损失函数等等,比如:
Batch size 信息：

- 格式为`{gpu x batch_per_gpu}`, 如 `8xb32`

训练类型(主要见于 transformer 网络，如 `ViT` 算法，这类算法通常分为预训练和微调两种模式):

- `ft` : Finetune config，用于微调的配置文件
- `pt` : Pretrain config，用于预训练的配置文件

训练策略信息，训练策略以复现配置文件为基础，此基础不必标注训练策略。但如果在此基础上进行改进，则需注明训练策略，按照应用点位顺序排列，如：`{pipeline aug}-{train aug}-{loss trick}-{scheduler}-{epochs}`

- `coslr-200e` : 使用 cosine scheduler, 训练 200 个 epoch
- `autoaug-mixup-lbs-coslr-50e` : 使用了 `autoaug`、`mixup`、`label smooth`、`cosine scheduler`, 训练了 50 个轮次

如果模型是从官方仓库等第三方仓库转换过来的，训练信息可以省略，使用 `3rdparty` 作为标识符。

### 数据信息

- `in1k` : `ImageNet1k` 数据集，默认使用 `224x224` 大小的图片
- `in21k` : `ImageNet21k` 数据集，有些地方也称为 `ImageNet22k` 数据集，默认使用 `224x224` 大小的图片
- `in1k-384px` : 表示训练的输出图片大小为 `384x384`

### 数据信息

### 模型命名案例

```text
```

- `vit-base-p32`: 模块信息
- `clip-openai-pre`：预训练信息
  - `clip`：预训练方法是 clip
  - `openai`：预训练模型来自 OpenAI
  - `pre`：预训练标识符
- `3rdparty`：模型是从第三方仓库转换而来的
- `in1k`：数据集信息。该模型是从 ImageNet-1k 数据集训练而来的，输入大小为 `224x224`

```text
```

- `beit`: 算法信息
- `beit-base`：模块信息，由于主干网络来自 BEiT 中提出的修改版 ViT，主干网络名称也是 `beit`
- `8xb256-amp-coslr-300e`：训练信息
  - `8xb256`：使用 8 个 GPU，每个 GPU 的批量大小为 256
  - `amp`：使用自动混合精度训练
  - `coslr`：使用余弦退火学习率调度器
  - `300e`：训练 300 个 epoch

配置文件的命名与模型名称几乎相同，有几点不同：

- 训练信息是必要的，不能是 `3rdparty`
- 如果配置文件只包含主干网络设置，既没有头部设置也没有数据集设置，我们将其命名为`{module info}_headless.py`。这种配置文件通常用于大型数据集上的第三方预训练模型。

### 权重命名规则

权重的命名主要包括模型名称，日期和哈希值。

### 权重命名规则

```text
```

## current model load api
### Cons
#### two functions
#### deal float32 float16 quantized-u8
#### deal alignment size
```cpp
int Convolution::load_model(FILE* binfp)
{
    int nread;

    union
    {
        struct
        {
            unsigned char f0;
            unsigned char f1;
            unsigned char f2;
            unsigned char f3;
        };
        unsigned int tag;
    } flag_struct;

    nread = fread(&flag_struct, sizeof(flag_struct), 1, binfp);
    if (nread != 1)
    {
        fprintf(stderr, "Convolution read flag_struct failed %d\n", nread);
        return -1;
    }

    unsigned int flag = flag_struct.f0 + flag_struct.f1 + flag_struct.f2 + flag_struct.f3;

    weight_data.create(weight_data_size);
    if (weight_data.empty())

    if (flag_struct.tag == 0x01306B47)
    {
        // half-precision weight data
        int align_weight_data_size = alignSize(weight_data_size * sizeof(unsigned short), 4);

return -100;

    {
        std::vector<unsigned short> float16_weights;
        float16_weights.resize(align_weight_data_size);
        nread = fread(float16_weights.data(), align_weight_data_size, 1, binfp);
        if (nread != 1)
        {
            fprintf(stderr, "Convolution read float16_weights failed %d\n", nread);
            return -1;
        }

        weight_data = Mat::from_float16(float16_weights.data(), weight_data_size);
        if (weight_data.empty())
            return -100;
    }
    else if (flag != 0)
    {
        // quantized weight data
        float quantization_value[256];
        nread = fread(quantization_value, 256 * sizeof(float), 1, binfp);
        {
            fprintf(stderr, "Convolution read quantization_value failed %d\n", nread);

nread = fread(quantization_value, 256 * sizeof(float), 1, binfp);
        {
            return -1;
        }

        int align_weight_data_size = alignSize(weight_data_size * sizeof(unsigned char), 4);
        std::vector<unsigned char> index_array;
        index_array.resize(align_weight_data_size);
        nread = fread(index_array.data(), align_weight_data_size, 1, binfp);
        {
            fprintf(stderr, "Convolution read index_array failed %d\n", nread);
            return -1;
        }

        float* weight_data_ptr = weight_data;
        for (int i = 0; i < weight_data_size; i++)
        {
            weight_data_ptr[i] = quantization_value[ index_array[i] ];
        }
    }
    else if (flag_struct.f0 == 0)
    {
        // raw weight data
        nread = fread(weight_data, weight_data_size * sizeof(float), 1, binfp);

}
    }
    {
        {
            fprintf(stderr, "Convolution read weight_data failed %d\n", nread);
            return -1;
        }
    }

    if (bias_term)
    {
        bias_data.create(num_output);
        if (bias_data.empty())
        nread = fread(bias_data, num_output * sizeof(float), 1, binfp);
        {
            fprintf(stderr, "Convolution read bias_data failed %d\n", nread);
            return -1;
        }
    }

    return 0;
}

int Convolution::load_model(const unsigned char*& mem)
{
    union
    {
        struct
        {
        };

{
        };

    memcpy(&flag_struct, mem, sizeof(flag_struct));
    mem += sizeof(flag_struct);

    {
        weight_data = Mat::from_float16((unsigned short*)mem, weight_data_size);
        mem += alignSize(weight_data_size * sizeof(unsigned short), 4);
    }
    {
        const float* quantization_value = (const float*)mem;
        mem += 256 * sizeof(float);

        const unsigned char* index_array = (const unsigned char*)mem;
        mem += alignSize(weight_data_size * sizeof(unsigned char), 4);

mem += 256 * sizeof(float);

        weight_data.create(weight_data_size);
        {
        }
    }
    {
        weight_data = Mat(weight_data_size, (float*)mem);
        mem += weight_data_size * sizeof(float);
    }

    {
        bias_data = Mat(num_output, (float*)mem);
        mem += num_output * sizeof(float);
    }

    return 0;
}
```

## new model load api proposed
### Pros
#### element type detection
```cpp
int Convolution::load_model(const ModelBin& mb)
{
    // auto detect element type

}
```

### Pros
```cpp
{
    weight_data = mb.load(weight_data_size, 0);

    {
        // certain type specified
        bias_data = mb.load(num_output, 1);
    }

    return 0;
}
```

# 自定义数据集

为了支持新的数据格式，可以选择将数据转换成现成的格式（COCO 或者 PASCAL）或将其转换成中间格式。当然也可以选择以离线的形式（在训练之前使用脚本转换）或者在线的形式（实现一个新的 dataset 在训练中进行转换）来转换数据。

在 MMDetection 中，建议将数据转换成 COCO 格式并以离线的方式进行，因此在完成数据转换后只需修改配置文件中的标注数据的路径和类别即可。

### 将新的数据格式转换为现有的数据格式

最简单的方法就是将你的数据集转换成现有的数据格式（COCO 或者 PASCAL VOC）

COCO 格式的 JSON 标注文件有如下必要的字段：

```python
    {
        'id': 1268
    },
    ...
],

    {
            247.09,
            ...
            219.03,
            249.06]],  # 如果有 mask 标签且为多边形 XY 点坐标格式，则需要保证至少包括 3 个点坐标，否则为无效多边形
    },
    ...
],

 ]
```

在 JSON 文件中有三个必要的键：

'bbox': [192.81, 224.8, 74.73, 33.43],
    },
    ...
],

 ]
```

- `images`: 包含多个图片以及它们的信息的数组，例如 `file_name`、`height`、`width` 和 `id`。
- `annotations`: 包含多个实例标注信息的数组。
- `categories`: 包含多个类别名字和 ID 的数组。

在数据预处理之后，使用现有的数据格式来训练自定义的新数据集有如下两步（以 COCO 为例）：

1. 为自定义数据集修改配置文件。

这里我们举一个例子来展示上面的两个步骤，这个例子使用包括 5 个类别的 COCO 格式的数据集来训练一个现有的 Cascade Mask R-CNN R50-FPN 检测器

#### 1. 为自定义数据集修改配置文件

配置文件的修改涉及两个方面：

1. `dataloaer` 部分。需要在 `train_dataloader.dataset`、`val_dataloader.dataset` 和 `test_dataloader.dataset` 中添加 `metainfo=dict(classes=classes)`, 其中 classes 必须是 tuple 类型。
2. `model` 部分中的 `num_classes`。需要将默认值（COCO 数据集中为 80）修改为自定义数据集中的类别数。

`configs/my_custom_config.py` 内容如下：

```python

# 新的配置来自基础的配置以更好地说明需要修改的地方

# 1. 数据集设定

```python

# 1. 数据集设定

        # 将类别名字添加至 `metainfo` 字段中
        )
    )

    )

)

        )
    )

# 2. 模型设置

# 将所有的 `num_classes` 默认值修改为 5（原来为80）
            dict(
                # 将所有的 `num_classes` 默认值修改为 5（原来为 80）
            dict(
            dict(
    # 将所有的 `num_classes` 默认值修改为 5（原来为 80）
```

#### 2. 检查自定义数据集的标注

# 将所有的 `num_classes` 默认值修改为 5（原来为 80）
```

假设你自己的数据集是 COCO 格式，那么需要保证数据的标注没有问题：

1. 标注文件中 `categories` 的长度要与配置中的 `classes` 元组长度相匹配，它们都表示有几类。（如例子中有 5 个类别）
2. 配置文件中 `classes` 字段应与标注文件里 `categories` 下的 `name` 有相同的元素且顺序一致。MMDetection 会自动将 `categories` 中不连续的 `id` 映射成连续的索引，因此 `categories` 下的 `name`的字符串顺序会影响标签的索引。同时，配置文件中的 `classes` 的字符串顺序也会影响到预测框可视化时的标签。
3. `annotations` 中的 `category_id` 必须是有效的值。比如所有 `category_id` 的值都应该属于 `categories` 中的 `id`。

下面是一个有效标注的例子：

```python

    {
            247.09,
            ...
            219.03,
            249.06]],  # 如果有 mask 标签。
    },
    ...
],

'area': 1035.749,
    },
    ...
],

# MMDetection 会自动将 `categories` 中不连续的 `id` 映射成连续的索引。
 ]
```

我们使用这种方式来支持 CityScapes 数据集。脚本在 [cityscapes.py](https://github.com/open-mmlab/mmdetection/blob/main/tools/dataset_converters/cityscapes.py) 并且我们提供了微调的 [configs](https://github.com/open-mmlab/mmdetection/blob/main/configs/cityscapes).

**注意**

1. 对于实例分割数据集, **MMDetection 目前只支持评估 COCO 格式的 mask AP**.
2. 推荐训练之前进行离线转换，这样就可以继续使用 `CocoDataset` 且只需修改标注文件的路径以及训练的种类。

### 调整新的数据格式为中间格式

如果不想将标注格式转换为 COCO 或者 PASCAL 格式也是可行的。实际上，我们在 MMEngine 的 [BaseDataset](https://github.com/open-mmlab/mmengine/blob/main/mmengine/dataset/base_dataset.py#L116) 中定义了一种简单的标注格式并且与所有现有的数据格式兼容，也能进行离线或者在线转换。

数据集的标注必须为 `json` 或 `yaml`，`yml` 或 `pickle`，`pkl` 格式；标注文件中存储的字典必须包含 `metainfo` 和 `data_list` 两个字段。其中 `metainfo` 是一个字典，里面包含数据集的元信息，例如类别信息；`data_list` 是一个列表，列表中每个元素是一个字典，该字典定义了一个原始数据（raw data），每个原始数据包含一个或若干个训练/测试样本。

以下是一个 JSON 标注文件的例子:

```json
{
        {
            ...
        },
        [
            {
                [
                  {
                  },
                  {

"bbox_label": 1,
                  },
                  {
                  }
                ]
              },
            {
                [
                  {
                    "ignore_flag": 1
                  }
                ]
              },
            ...
        ]
}
```

有些数据集可能会提供如：crowd/difficult/ignored bboxes 标注，那么我们使用 `ignore_flag`来包含它们。

在得到上述标准的数据标注格式后，可以直接在配置中使用 MMDetection 的 [BaseDetDataset](https://github.com/open-mmlab/mmdetection/blob/main/mmdet/datasets/base_det_dataset.py#L13) ，而无需进行转换。

### 自定义数据集例子

假设文本文件中表示的是一种全新的标注格式。边界框的标注信息保存在 `annotation.txt` 中，内容如下：

```
#

```
#
1280 720
2
10 20 40 60 1
20 40 50 60 2
#
1280 720
3
50 20 40 60 2
20 40 30 45 2
30 40 50 60 3
```

我们可以在 `mmdet/datasets/my_dataset.py` 中创建一个新的 dataset 用以加载数据。

```python

    }

                continue

img_shape = ann_list[i + 2].split(' ')

                dict(
                    img_id=i,
                ))

```

配置文件中，可以使用 `MyDataset` 进行如下修改

```python
)
```

## 使用 dataset 包装器自定义数据集

MMEngine 也支持非常多的数据集包装器（wrapper）来混合数据集或在训练时修改数据集的分布，其支持如下三种数据集包装：

type='MyDataset',
)
```

- `RepeatDataset`：将整个数据集简单地重复。
- `ClassBalancedDataset`：以类别均衡的方式重复数据集。
- `ConcatDataset`：合并数据集。

具体使用方式见 [MMEngine 数据集包装器](#TODO)。

## 修改数据集的类别

根据现有数据集的类型，我们可以修改它们的类别名称来训练其标注的子集。
例如，如果只想训练当前数据集中的三个类别，那么就可以修改数据集的 `metainfo` 字典，数据集就会自动屏蔽掉其他类别的真实框。

```python
    )
    )
    )
```

**注意**

)
    )
    )
```

**注意**

- 在 MMDetection v2.5.0 之前，如果类别为集合时数据集将自动过滤掉不包含 GT 的图片，且没办法通过修改配置将其关闭。这是一种不可取的行为而且会引起混淆，因为当类别不是集合时数据集时，只有在 `filter_empty_gt=True` 以及 `test_mode=False` 的情况下才会过滤掉不包含 GT 的图片。在 MMDetection v2.5.0 之后，我们将图片的过滤以及类别的修改进行解耦，数据集只有在 `filter_cfg=dict(filter_empty_gt=True)` 和 `test_mode=False` 的情况下才会过滤掉不包含 GT 的图片，无论类别是否为集合。设置类别只会影响用于训练的标注类别，用户可以自行决定是否过滤不包含 GT 的图片。
- 直接使用 MMEngine 中的 `BaseDataset` 或者 MMDetection 中的 `BaseDetDataset` 时用户不能通过修改配置来过滤不含 GT 的图片，但是可以通过离线的方式来解决。
- 当设置数据集中的 `classes` 时，记得修改 `num_classes`。从 v2.9.0 (PR#4508) 之后，我们实现了 [NumClassCheckHook](https://github.com/open-mmlab/mmdetection/blob/main/mmdet/engine/hooks/num_class_check_hook.py) 来检查类别数是否一致。

## COCO 全景分割数据集

现在我们也支持 COCO Panoptic Dataset，全景注释的格式与 COCO 格式不同，其前景和背景都将存在于注释文件中。COCO Panoptic 格式的注释 JSON 文件具有以下必要的键：

```python

```python
    {
        'id': 1268
    },
    ...
]

    {
            {
            },
            ...
        ]
    },
    ...
]

    ...
 ]
```

此外，`seg` 必须设置为全景注释图像的路径。

```python

{'id': 0, 'name': 'person'},
    ...
 ]
```

```python

    )
)
    )
)
    )
)
```

}
```

<details>

```bibtex
}
```

| [deeppose_resnet_50](/configs/hand_2d_keypoint/topdown_regression/rhd2d/td-reg_res50_8xb64-210e_rhd2d-256x256.py) |  256x256   |  0.988  | 0.865 | 3.32 | [ckpt](https://download.openmmlab.com/mmpose/hand/deeppose/deeppose_res50_rhd2d_256x256-37f1c4d3_20210330.pth) | [log](https://download.openmmlab.com/mmpose/hand/deeppose/deeppose_res50_rhd2d_256x256_20210330.log.json) |

# Swin

This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with Shifted windows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it compatible with a broad range of vision tasks, including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation (53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-the-art by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones. The hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures.

<img src="https://user-images.githubusercontent.com/40661020/143999551-6a527048-de38-485c-a1b6-3133ffa5bfaa.png"/>
</div>

| Backbone |  Pretrain   | Lr schd | Multi-scale crop | FP16 | Mem (GB) | Inf time (fps) | box AP | mask AP |                            Config                             |                                                                                                                                                                                      Download                                                                                                                                                                                       |

|  Swin-T  | ImageNet-1K |   1x    |        no        |  no  |   7.6    |                |  42.7  |  39.3   |       [config](./mask-rcnn_swin-t-p4-w7_fpn_1x_coco.py)       |                           [model](https://download.openmmlab.com/mmdetection/v2.0/swin/mask_rcnn_swin-t-p4-w7_fpn_1x_coco/mask_rcnn_swin-t-p4-w7_fpn_1x_coco_20210902_120937-9d6b7cfa.pth)  \| [log](https://download.openmmlab.com/mmdetection/v2.0/swin/mask_rcnn_swin-t-p4-w7_fpn_1x_coco/mask_rcnn_swin-t-p4-w7_fpn_1x_coco_20210902_120937.log.json)                           |

|  Swin-T  | ImageNet-1K |   3x    |       yes        |  no  |   10.2   |                |  46.0  |  41.6   |   [config](./mask-rcnn_swin-t-p4-w7_fpn_ms-crop-3x_coco.py)   |           [model](https://download.openmmlab.com/mmdetection/v2.0/swin/mask_rcnn_swin-t-p4-w7_fpn_ms-crop-3x_coco/mask_rcnn_swin-t-p4-w7_fpn_ms-crop-3x_coco_20210906_131725-bacf6f7b.pth)  \| [log](https://download.openmmlab.com/mmdetection/v2.0/swin/mask_rcnn_swin-t-p4-w7_fpn_ms-crop-3x_coco/mask_rcnn_swin-t-p4-w7_fpn_ms-crop-3x_coco_20210906_131725.log.json)           |

|  Swin-T  | ImageNet-1K |   3x    |       yes        | yes  |   7.8    |                |  46.0  |  41.7   | [config](./mask-rcnn_swin-t-p4-w7_fpn_amp-ms-crop-3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/swin/mask_rcnn_swin-t-p4-w7_fpn_fp16_ms-crop-3x_coco/mask_rcnn_swin-t-p4-w7_fpn_fp16_ms-crop-3x_coco_20210908_165006-90a4008c.pth)  \| [log](https://download.openmmlab.com/mmdetection/v2.0/swin/mask_rcnn_swin-t-p4-w7_fpn_fp16_ms-crop-3x_coco/mask_rcnn_swin-t-p4-w7_fpn_fp16_ms-crop-3x_coco_20210908_165006.log.json) |

|  Swin-S  | ImageNet-1K |   3x    |       yes        | yes  |   11.9   |                |  48.2  |  43.2   | [config](./mask-rcnn_swin-s-p4-w7_fpn_amp-ms-crop-3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/swin/mask_rcnn_swin-s-p4-w7_fpn_fp16_ms-crop-3x_coco/mask_rcnn_swin-s-p4-w7_fpn_fp16_ms-crop-3x_coco_20210903_104808-b92c91f1.pth)  \| [log](https://download.openmmlab.com/mmdetection/v2.0/swin/mask_rcnn_swin-s-p4-w7_fpn_fp16_ms-crop-3x_coco/mask_rcnn_swin-s-p4-w7_fpn_fp16_ms-crop-3x_coco_20210903_104808.log.json) |

### Notice

Please follow the example
of `retinanet_swin-t-p4-w7_fpn_1x_coco.py` when you want to combine Swin Transformer with
the one-stage detector. Because there is a layer norm at the outs of Swin Transformer, you must set `start_level` as 0 in FPN, so we have to set the `out_indices` of backbone as `[1,2,3]`.

```latex
    title={Swin Transformer: Hierarchical Vision Transformer using Shifted Windows},

```latex
    author={Liu, Ze and Lin, Yutong and Cao, Yue and Hu, Han and Wei, Yixuan and Zhang, Zheng and Lin, Stephen and Guo, Baining},
    journal={arXiv preprint arXiv:2103.14030},
}
```

}
```

<details>

```bibtex
}
```

| [pose_hrnetv2_w18_dark](/configs/hand_2d_keypoint/topdown_heatmap/rhd2d/td-hm_hrnetv2-w18_dark-8xb64-210e_rhd2d-256x256.py) |  256x256   |  0.992  | 0.903 | 2.18 | [ckpt](https://download.openmmlab.com/mmpose/hand/dark/hrnetv2_w18_rhd2d_256x256_dark-4df3a347_20210330.pth) | [log](https://download.openmmlab.com/mmpose/hand/dark/hrnetv2_w18_rhd2d_256x256_dark_20210330.log.json) |

# SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videos

<details>
<summary align="right"><a href="https://arxiv.org/abs/2112.13715">SmoothNet (arXiv'2021)</a></summary>

```bibtex
@article{zeng2021smoothnet,
  title={SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videos},
  author={Zeng, Ailing and Yang, Lei and Ju, Xuan and Li, Jiefeng and Wang, Jianyi and Xu, Qiang},
  journal={arXiv preprint arXiv:2112.13715},
}
```

When analyzing human motion videos, the output jitters from existing pose estimators are highly-unbalanced. Most frames only suffer from slight jitters, while significant jitters occur in those frames with occlusion or poor image quality. Such complex poses often persist in videos, leading to consecutive frames with poor estimation results and large jitters. Existing pose smoothing solutions based on temporal convolutional networks, recurrent neural networks, or low-pass filters cannot deal with such a long-term jitter problem without considering the significant and persistent errors within the jittering video segment. Motivated by the above observation, we propose a novel plug-and-play refinement network, namely SMOOTHNET, which can be attached to any existing pose estimators to improve its temporal smoothness and enhance its per-frame precision simultaneously. Especially, SMOOTHNET is a simple yet effective data-driven fully-connected network with large receptive fields, effectively mitigating the impact of long-term jitters with unreliable estimation results. We conduct extensive experiments on twelve backbone networks with seven datasets across 2D and 3D pose estimation, body recovery, and downstream tasks. Our results demonstrate that the proposed SMOOTHNET consistently outperforms existing solutions, especially on those clips with high errors and long-term jitters.

<img src="https://user-images.githubusercontent.com/15977946/161272519-0165c0e2-f0e8-45ad-88dd-ddb49fc81bda.png">
</div>

# NAFNet (ECCV'2022)

> [Simple Baselines for Image Restoration](https://arxiv.org/abs/2204.04676)

> **Task**: Image Restoration

Although there have been significant advances in the field of image restoration recently, the system complexity of the state-of-the-art (SOTA) methods is increasing as well, which may hinder the convenient analysis and comparison of methods. In this paper, we propose a simple baseline that exceeds the SOTA methods and is computationally efficient. To further simplify the baseline, we reveal that the nonlinear activation functions, e.g. Sigmoid, ReLU, GELU, Softmax, etc. are not necessary: they could be replaced by multiplication or removed. Thus, we derive a Nonlinear Activation Free Network, namely NAFNet, from the baseline. SOTA results are achieved on various challenging benchmarks, e.g. 33.69 dB PSNR on GoPro (for image deblurring), exceeding the previous SOTA 0.38 dB with only 8.4% of its computational costs; 40.30 dB PSNR on SIDD (for image denoising), exceeding the previous SOTA 0.28 dB with less than half of its computational costs.

 <img src="https://user-images.githubusercontent.com/43229734/199919292-81d307d9-144b-4d07-9f26-0c09d86e84a5.jpg" width="400"/>
</div >

|                                  Model                                  | Dataset | image size |       PSNR       |      SSIM      | GPU Info |                                  Download                                  |
| :---------------------------------------------------------------------: | :-----: | :--------: | :--------------: | :------------: | :------: | :------------------------------------------------------------------------: |
| [nafnet_c64eb2248mb12db2222_8xb8-lr1e-3-400k_sidd](./nafnet_c64eb2248mb12db2222_8xb8-lr1e-3-400k_sidd.py) |  SIDD   |  256X256   | 40.3045(40.3045) | 0.9253(0.9614) | 1 (A100) | [model](https://download.openmmlab.com/mmediting/nafnet/NAFNet-SIDD-midc64.pth) \| log(coming soon) |

| [nafnet_c64eb11128mb1db1111_8xb8-lr1e-3-400k_gopro](./nafnet_c64eb11128mb1db1111_8xb8-lr1e-3-400k_gopro.py) |  GoPro  |  1280x720  | 33.7246(33.7103) | 0.9479(0.9668) | 1 (A100) | [model](https://download.openmmlab.com/mmediting/nafnet/NAFNet-GoPro-midc64.pth) \| log(coming soon) |

Note:

- a(b) where a denotes the value run by MMagic, b denotes the value copied from the original paper.
- PSNR is evaluated on RGB channels.
- SSIM is evaluated by averaging SSIMs on RGB channels, however the original paper uses the 3D SSIM kernel.

**Train**

<details>

```shell
# cpu train

```

For more details, you can refer to **Train a model** part in [train_test.md](../../docs/en/user_guides/train_test.md).

**Test**

<details>

```shell
# cpu test

```

Pretrained checkpoints will come soon.

```

For more details, you can refer to **Test a pre-trained model** part in [train_test.md](../../docs/en/user_guides/train_test.md).

```bibtex
}
```

# UNet

> [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://arxiv.org/abs/1505.04597)

<a href="http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.17.0/mmseg/models/backbones/unet.py#L225">Code Snippet</a>

There is large consent that successful training of deep networks requires many thousand annotated training samples. In this paper, we present a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently. The architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. We show that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks. Using the same network trained on transmitted light microscopy images (phase contrast and DIC) we won the ISBI cell tracking challenge 2015 in these categories by a large margin. Moreover, the network is fast. Segmentation of a 512x512 image takes less than a second on a recent GPU. The full implementation (based on Caffe) and the trained networks are available at [this http URL](https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/).

<img src="https://user-images.githubusercontent.com/24582831/142902977-20fe689d-a147-4d92-9690-dbfde8b68dbe.png" width="70%"/>
</div>

| Method     | Backbone    | Loss          | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device |  mIoU | mIoU(ms+flip) | config                                                                                                                         | download                                                                                                                                                                                                                                                                                                                                                                     |

| UNet + FCN | UNet-S5-D16 | Cross Entropy | 512x1024  |  160000 | 17.91    | 3.05           | V100   | 69.10 |         71.05 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_fcn_4xb4-160k_cityscapes-512x1024.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/fcn_unet_s5-d16_4x4_512x1024_160k_cityscapes/fcn_unet_s5-d16_4x4_512x1024_160k_cityscapes_20211210_145204-6860854e.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/fcn_unet_s5-d16_4x4_512x1024_160k_cityscapes/fcn_unet_s5-d16_4x4_512x1024_160k_cityscapes_20211210_145204.log.json) |

### DRIVE

### DRIVE

| Method           | Backbone    | Loss                 | Image Size | Crop Size | Stride | Lr schd | Mem (GB) | Inf time (fps) | Device | mDice |  Dice | config                                                                                                                                      | download                                                                                                                                                                                                                                                                                                                                                                                                         |

| UNet + FCN       | UNet-S5-D16 | Cross Entropy        | 584x565    | 64x64     |  42x42 | 40000   | 0.680    |              - | V100   | 88.38 | 78.67 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_fcn_4xb4-40k_drive-64x64.py)                       | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/fcn_unet_s5-d16_64x64_40k_drive/fcn_unet_s5-d16_64x64_40k_drive_20201223_191051-5daf6d3b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/unet_s5-d16_64x64_40k_drive/unet_s5-d16_64x64_40k_drive-20201223_191051.log.json)                                                                                                 |

| UNet + FCN       | UNet-S5-D16 | Cross Entropy + Dice | 584x565    | 64x64     |  42x42 | 40000   | 0.582    |              - | V100   | 88.71 | 79.32 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_fcn_4xb4-ce-1.0-dice-3.0-40k_drive-64x64.py)       | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/fcn_unet_s5-d16_ce-1.0-dice-3.0_64x64_40k_drive/fcn_unet_s5-d16_ce-1.0-dice-3.0_64x64_40k_drive_20211210_201820-785de5c2.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/fcn_unet_s5-d16_ce-1.0-dice-3.0_64x64_40k_drive/fcn_unet_s5-d16_ce-1.0-dice-3.0_64x64_40k_drive_20211210_201820.log.json)                         |

| UNet + PSPNet    | UNet-S5-D16 | Cross Entropy        | 584x565    | 64x64     |  42x42 | 40000   | 0.599    |              - | V100   | 88.35 | 78.62 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_pspnet_4xb4-40k_drive-64x64.py)                    | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/pspnet_unet_s5-d16_64x64_40k_drive/pspnet_unet_s5-d16_64x64_40k_drive_20201227_181818-aac73387.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/pspnet_unet_s5-d16_64x64_40k_drive/pspnet_unet_s5-d16_64x64_40k_drive-20201227_181818.log.json)                                                                             |

| UNet + PSPNet    | UNet-S5-D16 | Cross Entropy + Dice | 584x565    | 64x64     |  42x42 | 40000   | 0.585    |              - | V100   | 88.76 | 79.42 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_pspnet_4xb4-ce-1.0-dice-3.0-40k_drive-64x64.py)    | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/pspnet_unet_s5-d16_ce-1.0-dice-3.0_64x64_40k_drive/pspnet_unet_s5-d16_ce-1.0-dice-3.0_64x64_40k_drive_20211210_201821-22b3e3ba.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/pspnet_unet_s5-d16_ce-1.0-dice-3.0_64x64_40k_drive/pspnet_unet_s5-d16_ce-1.0-dice-3.0_64x64_40k_drive_20211210_201821.log.json)             |

| UNet + DeepLabV3 | UNet-S5-D16 | Cross Entropy        | 584x565    | 64x64     |  42x42 | 40000   | 0.596    |              - | V100   | 88.38 | 78.69 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_deeplabv3_4xb4-40k_drive-64x64.py)                 | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/deeplabv3_unet_s5-d16_64x64_40k_drive/deeplabv3_unet_s5-d16_64x64_40k_drive_20201226_094047-0671ff20.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/deeplabv3_unet_s5-d16_64x64_40k_drive/deeplabv3_unet_s5-d16_64x64_40k_drive-20201226_094047.log.json)                                                                 |

| UNet + DeepLabV3 | UNet-S5-D16 | Cross Entropy + Dice | 584x565    | 64x64     |  42x42 | 40000   | 0.582    |              - | V100   | 88.84 | 79.56 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_deeplabv3_4xb4-ce-1.0-dice-3.0-40k_drive-64x64.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/deeplabv3_unet_s5-d16_ce-1.0-dice-3.0_64x64_40k_drive/deeplabv3_unet_s5-d16_ce-1.0-dice-3.0_64x64_40k_drive_20211210_201825-6bf0efd7.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/deeplabv3_unet_s5-d16_ce-1.0-dice-3.0_64x64_40k_drive/deeplabv3_unet_s5-d16_ce-1.0-dice-3.0_64x64_40k_drive_20211210_201825.log.json) |

### STARE

### STARE

| Method           | Backbone    | Loss                 | Image Size | Crop Size | Stride | Lr schd | Mem (GB) | Inf time (fps) | Device | mDice |  Dice | config                                                                                                                                        | download                                                                                                                                                                                                                                                                                                                                                                                                                 |

| UNet + FCN       | UNet-S5-D16 | Cross Entropy        | 605x700    | 128x128   |  85x85 | 40000   | 0.968    |              - | V100   | 89.78 | 81.02 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_fcn_4xb4-40k_stare-128x128.py)                       | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/fcn_unet_s5-d16_128x128_40k_stare/fcn_unet_s5-d16_128x128_40k_stare_20201223_191051-7d77e78b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/unet_s5-d16_128x128_40k_stare/unet_s5-d16_128x128_40k_stare-20201223_191051.log.json)                                                                                                 |

| UNet + FCN       | UNet-S5-D16 | Cross Entropy + Dice | 605x700    | 128x128   |  85x85 | 40000   | 0.986    |              - | V100   | 90.65 | 82.70 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_fcn_4xb4-ce-1.0-dice-3.0-40k_stare-128x128.py)       | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/fcn_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_stare/fcn_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_stare_20211210_201821-f75705a9.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/fcn_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_stare/fcn_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_stare_20211210_201821.log.json)                         |

| UNet + PSPNet    | UNet-S5-D16 | Cross Entropy        | 605x700    | 128x128   |  85x85 | 40000   | 0.982    |              - | V100   | 89.89 | 81.22 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_pspnet_4xb4-40k_stare-128x128.py)                    | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/pspnet_unet_s5-d16_128x128_40k_stare/pspnet_unet_s5-d16_128x128_40k_stare_20201227_181818-3c2923c4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/pspnet_unet_s5-d16_128x128_40k_stare/pspnet_unet_s5-d16_128x128_40k_stare-20201227_181818.log.json)                                                                             |

| UNet + PSPNet    | UNet-S5-D16 | Cross Entropy + Dice | 605x700    | 128x128   |  85x85 | 40000   | 1.028    |              - | V100   | 90.72 | 82.84 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_pspnet_4xb4-ce-1.0-dice-3.0-40k_stare-128x128.py)    | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/pspnet_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_stare/pspnet_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_stare_20211210_201823-f1063ef7.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/pspnet_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_stare/pspnet_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_stare_20211210_201823.log.json)             |

| UNet + DeepLabV3 | UNet-S5-D16 | Cross Entropy        | 605x700    | 128x128   |  85x85 | 40000   | 0.999    |              - | V100   | 89.73 | 80.93 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_deeplabv3_4xb4-40k_stare-128x128.py)                 | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/deeplabv3_unet_s5-d16_128x128_40k_stare/deeplabv3_unet_s5-d16_128x128_40k_stare_20201226_094047-93dcb93c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/deeplabv3_unet_s5-d16_128x128_40k_stare/deeplabv3_unet_s5-d16_128x128_40k_stare-20201226_094047.log.json)                                                                 |

| UNet + DeepLabV3 | UNet-S5-D16 | Cross Entropy + Dice | 605x700    | 128x128   |  85x85 | 40000   | 1.010    |              - | V100   | 90.65 | 82.71 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_deeplabv3_4xb4-ce-1.0-dice-3.0-40k_stare-128x128.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/deeplabv3_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_stare/deeplabv3_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_stare_20211210_201825-21db614c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/deeplabv3_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_stare/deeplabv3_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_stare_20211210_201825.log.json) |

### CHASE_DB1

| Method           | Backbone    | Loss                 | Image Size | Crop Size | Stride | Lr schd | Mem (GB) | Inf time (fps) | Device | mDice |  Dice | config                                                                                                                                            | download                                                                                                                                                                                                                                                                                                                                                                                                                                 |

| UNet + FCN       | UNet-S5-D16 | Cross Entropy        | 960x999    | 128x128   |  85x85 | 40000   | 0.968    |              - | V100   | 89.46 | 80.24 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_fcn_4xb4-40k_chase-db1-128x128.py)                       | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/fcn_unet_s5-d16_128x128_40k_chase_db1/fcn_unet_s5-d16_128x128_40k_chase_db1_20201223_191051-11543527.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/unet_s5-d16_128x128_40k_chase_db1/unet_s5-d16_128x128_40k_chase_db1-20201223_191051.log.json)                                                                                                 |

| UNet + FCN       | UNet-S5-D16 | Cross Entropy + Dice | 960x999    | 128x128   |  85x85 | 40000   | 0.986    |              - | V100   | 89.52 | 80.40 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_fcn_4xb4-ce-1.0-dice-3.0-40k_chase-db1-128x128.py)       | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/fcn_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_chase-db1/fcn_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_chase-db1_20211210_201821-1c4eb7cf.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/fcn_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_chase-db1/fcn_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_chase-db1_20211210_201821.log.json)                         |

| UNet + PSPNet    | UNet-S5-D16 | Cross Entropy        | 960x999    | 128x128   |  85x85 | 40000   | 0.982    |              - | V100   | 89.52 | 80.36 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_pspnet_4xb4-40k_chase-db1-128x128.py)                    | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/pspnet_unet_s5-d16_128x128_40k_chase_db1/pspnet_unet_s5-d16_128x128_40k_chase_db1_20201227_181818-68d4e609.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/pspnet_unet_s5-d16_128x128_40k_chase_db1/pspnet_unet_s5-d16_128x128_40k_chase_db1-20201227_181818.log.json)                                                                             |

| UNet + PSPNet    | UNet-S5-D16 | Cross Entropy + Dice | 960x999    | 128x128   |  85x85 | 40000   | 1.028    |              - | V100   | 89.45 | 80.28 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_pspnet_4xb4-ce-1.0-dice-3.0-40k_chase-db1-128x128.py)    | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/pspnet_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_chase-db1/pspnet_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_chase-db1_20211210_201823-c0802c4d.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/pspnet_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_chase-db1/pspnet_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_chase-db1_20211210_201823.log.json)             |

| UNet + DeepLabV3 | UNet-S5-D16 | Cross Entropy        | 960x999    | 128x128   |  85x85 | 40000   | 0.999    |              - | V100   | 89.57 | 80.47 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet_s5-d16_deeplabv3_4xb4-40k_chase-db1-128x128.py)                 | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/deeplabv3_unet_s5-d16_128x128_40k_chase_db1/deeplabv3_unet_s5-d16_128x128_40k_chase_db1_20201226_094047-4c5aefa3.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/deeplabv3_unet_s5-d16_128x128_40k_chase_db1/deeplabv3_unet_s5-d16_128x128_40k_chase_db1-20201226_094047.log.json)                                                                 |

| UNet + DeepLabV3 | UNet-S5-D16 | Cross Entropy + Dice | 960x999    | 128x128   |  85x85 | 40000   | 1.010    |              - | V100   | 89.49 | 80.37 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_deeplabv3_4xb4-ce-1.0-dice-3.0-40k_chase-db1-128x128.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/deeplabv3_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_chase-db1/deeplabv3_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_chase-db1_20211210_201825-4ef29df5.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/deeplabv3_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_chase-db1/deeplabv3_unet_s5-d16_ce-1.0-dice-3.0_128x128_40k_chase-db1_20211210_201825.log.json) |

### HRF

### HRF

| Method           | Backbone    | Loss                 | Image Size | Crop Size |  Stride | Lr schd | Mem (GB) | Inf time (fps) | Device | mDice |  Dice | config                                                                                                                                      | download                                                                                                                                                                                                                                                                                                                                                                                                         |

| UNet + FCN       | UNet-S5-D16 | Cross Entropy        | 2336x3504  | 256x256   | 170x170 | 40000   | 2.525    |              - | V100   | 88.92 | 79.45 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_fcn_4xb4-40k_hrf-256x256.py)                       | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/fcn_unet_s5-d16_256x256_40k_hrf/fcn_unet_s5-d16_256x256_40k_hrf_20201223_173724-d89cf1ed.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/unet_s5-d16_256x256_40k_hrf/unet_s5-d16_256x256_40k_hrf-20201223_173724.log.json)                                                                                                 |

| UNet + FCN       | UNet-S5-D16 | Cross Entropy + Dice | 2336x3504  | 256x256   | 170x170 | 40000   | 2.623    |              - | V100   | 89.64 | 80.87 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_fcn_4xb4-ce-1.0-dice-3.0-40k_hrf-256x256.py)       | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/fcn_unet_s5-d16_ce-1.0-dice-3.0_256x256_40k_hrf/fcn_unet_s5-d16_ce-1.0-dice-3.0_256x256_40k_hrf_20211210_201821-c314da8a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/fcn_unet_s5-d16_ce-1.0-dice-3.0_256x256_40k_hrf/fcn_unet_s5-d16_ce-1.0-dice-3.0_256x256_40k_hrf_20211210_201821.log.json)                         |

| UNet + PSPNet    | UNet-S5-D16 | Cross Entropy        | 2336x3504  | 256x256   | 170x170 | 40000   | 2.588    |              - | V100   | 89.24 | 80.07 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_pspnet_4xb4-40k_hrf-256x256.py)                    | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/pspnet_unet_s5-d16_256x256_40k_hrf/pspnet_unet_s5-d16_256x256_40k_hrf_20201227_181818-fdb7e29b.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/pspnet_unet_s5-d16_256x256_40k_hrf/pspnet_unet_s5-d16_256x256_40k_hrf-20201227_181818.log.json)                                                                             |

| UNet + PSPNet    | UNet-S5-D16 | Cross Entropy + Dice | 2336x3504  | 256x256   | 170x170 | 40000   | 2.798    |              - | V100   | 89.69 | 80.96 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_pspnet_4xb4-ce-1.0-dice-3.0-40k_hrf-256x256.py)    | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/pspnet_unet_s5-d16_ce-1.0-dice-3.0_256x256_40k_hrf/pspnet_unet_s5-d16_ce-1.0-dice-3.0_256x256_40k_hrf_20211210_201823-53d492fa.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/pspnet_unet_s5-d16_ce-1.0-dice-3.0_256x256_40k_hrf/pspnet_unet_s5-d16_ce-1.0-dice-3.0_256x256_40k_hrf_20211210_201823.log.json)             |

| UNet + DeepLabV3 | UNet-S5-D16 | Cross Entropy        | 2336x3504  | 256x256   | 170x170 | 40000   | 2.604    |              - | V100   | 89.32 | 80.21 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_deeplabv3_4xb4-40k_hrf-256x256.py)                 | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/deeplabv3_unet_s5-d16_256x256_40k_hrf/deeplabv3_unet_s5-d16_256x256_40k_hrf_20201226_094047-3a1fdf85.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/deeplabv3_unet_s5-d16_256x256_40k_hrf/deeplabv3_unet_s5-d16_256x256_40k_hrf-20201226_094047.log.json)                                                                 |

| UNet + DeepLabV3 | UNet-S5-D16 | Cross Entropy + Dice | 2336x3504  | 256x256   | 170x170 | 40000   | 2.607    |              - | V100   | 89.56 | 80.71 | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/unet/unet-s5-d16_deeplabv3_4xb4-ce-1.0-dice-3.0-40k_hrf-256x256.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/unet/deeplabv3_unet_s5-d16_ce-1.0-dice-3.0_256x256_40k_hrf/deeplabv3_unet_s5-d16_ce-1.0-dice-3.0_256x256_40k_hrf_20211210_202032-59daf7a4.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/unet/deeplabv3_unet_s5-d16_ce-1.0-dice-3.0_256x256_40k_hrf/deeplabv3_unet_s5-d16_ce-1.0-dice-3.0_256x256_40k_hrf_20211210_202032.log.json) |

Note:

- In  `DRIVE`, `STARE`, `CHASE_DB1`, and `HRF` dataset, `mDice` is mean dice of background and vessel, while `Dice` is dice metric of vessel(foreground) only.

```bibtex
@inproceedings{ronneberger2015u,
  title={U-net: Convolutional networks for biomedical image segmentation},

```bibtex
  author={Ronneberger, Olaf and Fischer, Philipp and Brox, Thomas},
  booktitle={International Conference on Medical image computing and computer-assisted intervention},
  pages={234--241},
}
```

<details>
<summary align="right">GLEAN (CVPR'2021)</summary>

```bibtex
}
```

<br/>

有关训练和测试中使用的元信息，请参阅[此处](https://github.com/ckkelvinchan/GLEAN)。 结果在 RGB 通道上进行评估。

|                                         算法                                          | PSNR  |         GPU 信息         |                                          下载                                          |
| :-----------------------------------------------------------------------------------: | :---: | :----------------------: | :------------------------------------------------------------------------------------: |

|                        [glean_cat_8x](./glean_x8_2xb8_cat.py)                         | 23.98 | 2 (Tesla V100-PCIE-32GB) | [模型](https://download.openmmlab.com/mmediting/restorers/glean/glean_cat_8x_20210614-d3ac8683.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/glean/glean_cat_8x_20210614_145540.log.json) |
|                      [glean_ffhq_16x](./glean_x16_2xb8_ffhq.py)                       | 26.91 | 2 (Tesla V100-PCIE-32GB) | [模型](https://download.openmmlab.com/mmediting/restorers/glean/glean_ffhq_16x_20210527-61a3afad.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/glean/glean_ffhq_16x_20210527_194536.log.json) |
|                       [glean_cat_16x](./glean_x16_2xb8_cat.py)                        | 20.88 | 2 (Tesla V100-PCIE-32GB) | [模型](https://download.openmmlab.com/mmediting/restorers/glean/glean_cat_16x_20210527-68912543.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/glean/glean_cat_16x_20210527_103708.log.json) |

| [glean_in128out1024_4x2_300k_ffhq_celebahq](./glean_in128out1024_4xb2-300k_ffhq-celeba-hq.py) | 27.94 | 4 (Tesla V100-SXM3-32GB) | [模型](https://download.openmmlab.com/mmediting/restorers/glean/glean_in128out1024_4x2_300k_ffhq_celebahq_20210812-acbcb04f.pth) \| [日志](https://download.openmmlab.com/mmediting/restorers/glean/glean_in128out1024_4x2_300k_ffhq_celebahq_20210812_100549.log.json) |
|                   [glean_fp16_cat_8x](./glean_x8-fp16_2xb8_cat.py)                    |   -   |            -             |                                           -                                            |
|                 [glean_fp16_ffhq_16x](./glean_x16-fp16_2xb8_ffhq.py)                  |   -   |            -             |                                           -                                            |

| [glean_fp16_in128out1024_4x2_300k_ffhq_celebahq](./glean_in128out1024-fp16_4xb2-300k_ffhq-celeba-hq.py) |   -   |            -             |                                           -                                            |

## 快速开始

**训练**

<details>

```shell
# CPU上训练

# 单个GPU上训练

# 多个GPU上训练
```

**测试**

<details>

```shell
# CPU上测试

# 单个GPU上测试

# CPU上测试

# 单个GPU上测试

# 多个GPU上测试
```

# Model Deployment (To be updated)

MMRotate 1.x fully relies on [MMDeploy](https://mmdeploy.readthedocs.io/) to deploy models.

# 准备 GLEAN 数据集

```bibtex
}
```

## 准备 cat_train 数据集

1. 从[LSUN 主页](https://www.yf.io/p/lsun)下载[cat 数据集](http://dl.yf.io/lsun/objects/cat.zip)。

2. 从[GLEAN 主页](https://github.com/ckkelvinchan/GLEAN)下载[cat_train/meta_info_LSUNcat_GT.txt](https://github.com/ckkelvinchan/GLEAN/blob/main/data/cat_train/meta_info_LSUNcat_GT.txt)。

3. 导出图像并下采样

从 lmdb 文件中导出图像，并下采样到所需尺寸。为此，我们提供了一个脚本：

```shell
```

生成的数据存储在 `cat_train` 目录下，目录结构应如下所示：

```text
mmagic
├── mmagic
├── tools

```

```text
mmagic
├── mmagic
├── tools
├── data
│   │   ├── GT
...
```

## 准备 cat_test 数据集

1. 从数据集[主页](https://archive.org/details/CAT_DATASET)下载[CAT 数据集](https://archive.org/download/CAT_DATASET/CAT_DATASET_02.zip)。

2. 从[GLEAN 主页](https://github.com/ckkelvinchan/GLEAN)下载[cat_test/meta_info_Cat100_GT.txt](https://github.com/ckkelvinchan/GLEAN/blob/main/data/cat_test/meta_info_Cat100_GT.txt)。

3. 下采样

将图像下采样到所需尺寸。为此，我们提供了一个脚本：

```shell
```

生成的数据存储在 `cat_test` 目录下，目录结构应如下所示：

```text
mmagic
├── mmagic
├── tools
├── data
│   │   ├── GT
...
```

```text
mmagic
├── mmagic
├── tools
├── data
│   │   ├── GT
...
```

## 准备 FFHQ 数据集

1. 从数据集[主页](https://github.com/NVlabs/ffhq-dataset)下载[FFHQ 数据集 (images1024x1024)](https://drive.google.com/drive/folders/1tZUcXDBeOibC6jcMCtgRRz67pzrAHeHL)。

将文件目录重构为如下所示：

```text
ffhq
├── images
|   ├── ...
```

2. 从[GLEAN 主页](https://github.com/ckkelvinchan/GLEAN)下载[ffhq/meta_info_FFHQ_GT.txt](https://github.com/ckkelvinchan/GLEAN/blob/main/data/FFHQ/meta_info_FFHQ_GT.txt)。

3. 下采样

```shell
```

生成的数据存储在 `ffhq` 目录下，目录结构应如下所示：

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── ffhq
...
```

```text
mmagic
├── mmagic
├── tools
├── data
|   ├── ffhq
...
```

## 准备 CelebA-HQ 数据集

1. 根据数据集[主页](https://github.com/tkarras/progressive_growing_of_gans)文档准备数据集。

```text
CelebA-HQ
├── GT
|   ├── ...
```

2. 从[GLEAN 主页](https://github.com/ckkelvinchan/GLEAN)下载[CelebA-HQ/meta_info_CelebAHQ_val100_GT.txt](https://github.com/ckkelvinchan/GLEAN/blob/main/data/CelebA-HQ/meta_info_CelebAHQ_val100_GT.txt)。

3. 下采样

```shell
```

生成的数据存储在 `CelebA-HQ` 目录下，目录结构应如下所示：

```text
mmagic
├── mmagic
├── tools
├── data
|   |   ├── GT
...
```

## 准备 FFHQ_CelebAHQ 数据集

├── tools
├── data
|   |   ├── GT
...
```

将 FFHQ(`ffhq/images`) 和 CelebA-HQ(`CelebA-HQ/GT`) 合并，生成 FFHQ_CelebAHQ 数据集。

文件目录重构应如下所示：

```text
mmagic
├── mmagic
├── tools
├── data
|   |   ├── GT
...
```

## List of examples

### 1. RTMPose-Deploy

RTMPose-Deploy is a C++ code example for RTMPose localized deployment.

- [ONNXRuntime-CPU](https://github.com/HW140701/RTMPose-Deploy)
- [TensorRT](https://github.com/Dominic23331/rtmpose_tensorrt)

### 2. RTMPose inference with ONNXRuntime (Python)

### 3. PoseTracker Android Demo

PoseTracker Android Demo Prototype based on mmdeploy.

- [Original Repository](https://github.com/hanrui1sensetime/PoseTracker-Android-Prototype)

# FAQ

  ```python
        dict(
    ])
  ```

  ```python
  ```

- 编译时enable_language(CUDA) 报错

  ```

  ```python
  ```

  ```

  ```

  **原因：** CUDA Toolkit 11.1安装在Visual Studio之前，造成VS的插件没有安装。或者VS的版本过新，使得CUDA Toolkit的安装的时候跳过了VS插件的安装

  **解决方法：** 可以通过手工拷贝插件的方式来解决这个问题。比如将`C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\extras\visual_studio_integration\MSBuildExtensions`中的四个文件拷贝到`C:\Software\Microsoft Visual Studio\2022\Community\Msbuild\Microsoft\VC\v170\BuildCustomizations` 目录下。具体路径根据实际情况进行更改。

- Windows系统下，转模型可视化时以及SDK推理时遇到
  ```
  ```
  **原因：** 在较新的windows系统中，系统路径下下有两个`onnxruntime.dll`，且会优先加载，造成冲突。
  ```

```
  ```
  ```
  **解决方法：** 以下两个方案任选其一
  1. 将下载的onnxruntime中的lib目录下的dll拷贝到mmdeploy_onnxruntime_ops.dll的同级目录（推荐使用Everything 进行查找）
  2. 将系统路径下的这两个dll改名，使其加载不到，可能涉及到修改文件权限的操作

### Pip

  ```bash
  $ which pip
  ```

# 如何支持新的模型

我们提供了多种工具来支持模型转换

## 函数的重写器

PyTorch 神经网络是用 python 编写的，可以简化算法的开发。但与此同时 Python 的流程控制和第三方库会使得网络导出为中间语言的过程变得困难。为此我们提供了一个“MonKey path”工具将不支持的功能重写为另一个可支持中间语言导出的功能。下述是一个具体的使用例子：

```python
    else:
```

使用函数重写器是十分容易的，只需添加一个带参数的装饰器即可：

- `func_name`是需要被重载的函数，它可以是其他PyTorch 的函数或者是自定义的函数。模块中的方法也可以通过工具进行重载。
- `backend`是推理引擎。当模型被导入到引擎的时候，函数会被重载。如果没有给出，重载默认的参数就是重载的参数。如果后端的重载的参数不存在，将会按照预设的默认模式进行重载。
  当参数与原始的参数相同时，除了把上下文信息`ctx` 作为第一的参数外，上下文也提供了一些有用的信息，例如:部署的配置`ctx.cfg` 和原始的函数（已经被重载）`ctx.origin_func`。

可参照[这些样例代码](https://github.com/open-mmlab/mmdeploy/blob/main/mmdeploy/codebase/mmpretrain/models/backbones/shufflenet_v2.py)。

## 模型重载器

如果您想用另一个模块替换整个模块，我们还有另一个重载器，如下所示：

```python
                 module,
                 cfg,

module.img_upsampler = nn.Upsample(
```

就像函数重载器一样，可添加一个带参数的装饰器：

- `module_type` 要重载的模块类。
- `backend` 是推理引擎。当模型被导入到引擎的时候，函数会被重载。如果没有给出，重载默认的参数就是重载的参数。如果后端的重载的参数不存在，将会按照预设的默认模式进行重载。

网络中模块的所有实例都将替换为这个新类的实例。原始模块和部署配置将作为前两个参数进行传递。

## 符号函数重写

PyTorch 和 ONNX 之间的映射是通过 PyTorch 中的符号函数进行定义的。自定义符号函数可以帮助我们绕过一些推理引擎不支持的 ONNX 节点。

```python
        dims = []
    else:

        dims = []
    else:
```

装饰器的参数

- `func_name`要添加符号的函数名称。如果是自定义的，请使用完整路径`torch.autograd.Function`。或者如果它是 PyTorch 内置函数，则只用写一个名称即可。
- 如果函数是 PyTorch 内置函数，则为True。
- `arg_descriptors` 符号函数参数的描述符，将被传递给`torch.onnx.symbolic_helper._parse_arg`。

就像函数重载器的`ctx`一样，第一个参数会提供上下文信息。上下文中了一些有用的信息，例如部署配置ctx.cfg和原始功能（已被重载）`ctx.origin_func`。请注意， `ctx.origin_func`只能在`is_pytorch==False`时使用。

[这里](https://github.com/open-mmlab/mmdeploy/tree/6420e2044515ff2052960c0f8bb9e351e6a7f2c2/mmdeploy/pytorch/symbolics)有很多实现可参考。

# 坐标系

## 概述

MMDetection3D 使用 3 种不同的坐标系。3D 目标检测领域中不同坐标系的存在是非常有必要的，因为对于各种 3D 数据采集设备来说，如激光雷达、深度相机等，使用的坐标系是不一致的，不同的 3D 数据集也遵循不同的数据格式。早期的工作，比如 SECOND、VoteNet 将原始数据转换为另一种格式，形成了一些后续工作也遵循的约定，使得不同坐标系之间的转换变得更加复杂。

尽管数据集和采集设备多种多样，但是通过总结 3D 目标检测的工作线，我们可以将坐标系大致分为三类：

- 相机坐标系 -- 大多数相机的坐标系，在该坐标系中 y 轴正方向指向地面，x 轴正方向指向右侧，z 轴正方向指向前方。

  ```
              上  z 前
              |    ^
              |   /
              |  /
              | /
              |/
  左   ------ 0 ------> x 右
              |
              |
              |
              |
              v
            y 下
  ```

- 激光雷达坐标系 -- 众多激光雷达的坐标系，在该坐标系中 z 轴负方向指向地面，x 轴正方向指向前方，y 轴正方向指向左侧。

  ```
                z 上   x 前
                 ^    ^
                 |   /
                 |  /
                 | /
                 |/
  y 左   <------ 0 ------ 右
  ```

- 深度坐标系 -- VoteNet、H3DNet 等模型使用的坐标系，在该坐标系中 z 轴负方向指向地面，x 轴正方向指向右侧，y 轴正方向指向前方。

  ```
             z 上   y 前
              ^    ^
              |   /
              |  /
              | /

```

  ```
             z 上   y 前
              ^    ^
              |   /
              |  /
              | /
              |/
  ```

该教程中的坐标系定义实际上**不仅仅是定义三个轴**。对于形如 $(x, y, z, dx, dy, dz, r)$ 的框来说，我们的坐标系也定义了如何解释框的尺寸 $(dx, dy, dz)$ 和转向角 (yaw) 角度 $r$。

三个坐标系的图示如下：

上面三张图是 3D 坐标系，下面三张图是鸟瞰图。

以后我们将坚持使用本教程中定义的三个坐标系。

## 转向角 (yaw) 的定义

请参考[维基百科](https://en.wikipedia.org/wiki/Euler_angles#Tait%E2%80%93Bryan_angles)了解转向角的标准定义。在目标检测中，我们选择一个轴作为重力轴，并在垂直于重力轴的平面 $\\Pi$ 上选取一个参考方向，那么参考方向的转向角为 0，在 $\\Pi$ 上的其他方向有非零的转向角，其角度取决于其与参考方向的角度。

目前，对于所有支持的数据集，标注不包括俯仰角 (pitch) 和滚动角 (roll)，这意味着我们在预测框和计算框之间的重叠时只需考虑转向角 (yaw)。

在 MMDetection3D 中，所有坐标系都是右手坐标系，这意味着如果从重力轴的负方向（轴的正方向指向人眼）看，转向角 (yaw) 沿着逆时针方向增加。

下图显示，在右手坐标系中，如果我们设定 x 轴正方向为参考方向，那么 y 轴正方向的转向角 (yaw) 为 $\\frac{\\pi}{2}$。

```

```
                     z 上  y 前 (yaw=0.5*pi)
                      ^    ^
                      |   /
                      |  /
                      | /
                      |/
左 (yaw=pi)    ------ 0 ------> x 右 (yaw=0)
```

对于一个框来说，其转向角 (yaw) 的值等于其方向减去一个参考方向。在 MMDetection3D 的所有三个坐标系中，参考方向总是 x 轴的正方向，而如果一个框的转向角 (yaw) 为 0，则其方向被定义为与 x 轴平行。框的转向角 (yaw) 的定义如下图所示。

```
  y 前
  ^      框的方向 (yaw=0.5*pi)
 /|\        ^
  |        /|\
  |    |    |    |
  |    |    |    |
__|____|____|____|______\ x 右
  |    |    |    |      /
  |    |    |    |
  |
```

## 框尺寸的定义

框尺寸的定义与转向角 (yaw) 的定义是分不开的。在上一节中，我们提到如果一个框的转向角 (yaw) 为 0，它的方向就被定义为与 x 轴平行。那么自然地，一个框对应于 x 轴的尺寸应该是 $dx$。但是，这在某些数据集中并非总是如此（我们稍后会解决这个问题）。

下图展示了 x 轴和 $dx$，y 轴和 $dy$ 对应的含义。

```
y 前
 /|\        ^
  |        /|\

```
y 前
 /|\        ^
  |        /|\
  |    |    |    |
  |    |    |    | dx
  |    |    |    |      /
  |    |    |    |
  |         dy
```

注意框的方向总是和 $dx$ 边平行。

```
y 前
 /|\   |    |    |
  |    |    |    |
  |    |    |    | dy
  |    |____|____|____\  框的方向 (yaw=0)
  |    |    |    |    /
__|____|____|____|_________\ x 右
  |    |    |    |         /
  |         dx
  |
```

## 与支持的数据集的原始坐标系的关系

### KITTI

KITTI 数据集的原始标注是在相机坐标系下的，详见 [get_label_anno](https://github.com/open-mmlab/mmdetection3d/blob/master/tools/data_converter/kitti_data_utils.py)。在 MMDetection3D 中，为了在 KITTI 数据集上训练基于激光雷达的模型，首先将数据从相机坐标系转换到激光雷达坐标，详见 [get_ann_info](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/datasets/kitti_dataset.py)。对于训练基于视觉的模型，数据保持在相机坐标系不变。

在 SECOND 中，框的激光雷达坐标系定义如下（鸟瞰图）：

对于每个框来说，尺寸为 $(w, l, h)$，转向角 (yaw) 的参考方向为 y 轴正方向。更多细节请参考[代码库](https://github.com/traveller59/second.pytorch#concepts)。

我们的激光雷达坐标系有两处改变：

- 转向角 (yaw) 被定义为右手而非左手，从而保持一致性；
- 框的尺寸为 $(l, w, h)$ 而非 $(w, l, h)$，由于在 KITTI 数据集中 $w$ 对应 $dy$，$l$ 对应 $dx$。

### Waymo

我们使用 Waymo 数据集的 KITTI 格式数据。因此，在我们的实现中 KITTI 和 Waymo 也共用相同的坐标系。

NuScenes 提供了一个评估工具包，其中每个框都被包装成一个 `Box` 实例。`Box` 的坐标系不同于我们的激光雷达坐标系，在 `Box` 坐标系中，前两个表示框尺寸的元素分别对应 $(dy, dx)$ 或者 $(w, l)$，和我们的表示方法相反。更多细节请参考 NuScenes [教程](https://github.com/open-mmlab/mmdetection3d/blob/master/docs/zh_cn/datasets/nuscenes_det.md#notes)。

读者可以参考 [NuScenes 开发工具](https://github.com/nutonomy/nuscenes-devkit/tree/master/python-sdk/nuscenes/eval/detection)，了解 [NuScenes 框](https://github.com/nutonomy/nuscenes-devkit/blob/2c6a752319f23910d5f55cc995abc547a9e54142/python-sdk/nuscenes/utils/data_classes.py#L457) 的定义和 [NuScenes 评估](https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/nuscenes/eval/detection/evaluate.py)的过程。

### Lyft

就涉及坐标系而言，Lyft 和 NuScenes 共用相同的数据格式。

请参考[官方网站](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/data)获取更多信息。

ScanNet 的原始数据不是点云而是网格，需要在我们的深度坐标系下进行采样得到点云数据。对于 ScanNet 检测任务，框的标注是轴对齐的，并且转向角 (yaw) 始终是 0。因此，我们的深度坐标系中转向角 (yaw) 的方向对 ScanNet 没有影响。

SUN RGB-D 的原始数据不是点云而是 RGB-D 图像。我们通过反投影，可以得到每张图像对应的点云，其在我们的深度坐标系下。但是，数据集的标注并不在我们的系统中，所以需要进行转换。

将原始标注转换为我们的深度坐标系下的标注的转换过程请参考 [sunrgbd_data_utils.py](https://github.com/open-mmlab/mmdetection3d/blob/master/tools/data_converter/sunrgbd_data_utils.py)。

### S3DIS

### S3DIS

在我们的实现中，S3DIS 与 ScanNet 共用相同的坐标系。然而 S3DIS 是一个仅限于分割任务的数据集，因此没有标注是坐标系敏感的。

## 例子

### 框（在不同坐标系间）的转换

以相机坐标系和激光雷达坐标系间的转换为例：

首先，对于点和框的中心点，坐标转换前后满足下列关系：

然后，框的尺寸转换前后满足下列关系：

最后，转向角 (yaw) 也应该被转换：

详见[此处](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/core/bbox/structures/box_3d_mode.py)代码了解更多细节。

### 鸟瞰图

如果 3D 框是 $(x, y, z, dx, dy, dz, r)$，相机坐标系下框的鸟瞰图是 $(x, z, dx, dz, -r)$。转向角 (yaw) 符号取反是因为相机坐标系重力轴的正方向指向地面。

详见[此处](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/core/bbox/structures/cam_box3d.py)代码了解更多细节。

### 框的旋转

### 框的旋转

我们将各种框的旋转设定为绕着重力轴逆时针旋转。因此，为了旋转一个 3D 框，我们首先需要计算新的框的中心，然后将旋转角度添加到转向角 (yaw)。

## 常见问题

#### Q1: 与框相关的算子是否适用于所有坐标系类型？

否。例如，[用于 RoI-Aware Pooling 的算子](https://github.com/open-mmlab/mmcv/blob/master/mmcv/ops/roiaware_pool3d.py)只适用于深度坐标系和激光雷达坐标系下的框。由于如果从上方看，旋转是顺时针的，所以 KITTI 数据集[这里](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/core/evaluation/kitti_utils.py)的评估函数仅适用于相机坐标系下的框。

对于每个和框相关的算子，我们注明了其所适用的框类型。

#### Q2: 在每个坐标系中，三个轴是否分别准确地指向右侧、前方和地面？

否。例如在 KITTI 中，从相机坐标系转换为激光雷达坐标系时，我们需要一个校准矩阵。

#### Q3: 框中转向角 (yaw) $2\\pi$ 的相位差如何影响评估？

对于交并比 (IoU) 计算，转向角 (yaw) 有 $2\\pi$ 的相位差的两个框是相同的，所以不会影响评估。

对于角度预测评估，例如 NuScenes 中的 NDS 指标和 KITTI 中的 AOS 指标，会先对预测框的角度进行标准化，因此 $2\\pi$ 的相位差不会改变结果。

#### Q4: 框中转向角 (yaw) $\\pi$ 的相位差如何影响评估？

对于交并比 (IoU) 计算，转向角 (yaw) 有 $\\pi$ 的相位差的两个框是相同的，所以不会影响评估。

然而，对于角度预测评估，这会导致完全相反的方向。

考虑一辆汽车，转向角 (yaw) 是汽车前部方向与 x 轴正方向之间的夹角。如果我们将该角度增加 $\\pi$，车前部将变成车后部。

对于某些类别，例如障碍物，前后没有区别，因此 $\\pi$ 的相位差不会对角度预测分数产生影响。

# 教程 1: 了解配置文件（待更新）

mmagic 采用基于 python 文件的配置系统，您可以在 `$MMagic/configs` 下查看预置的配置文件。

配置文件按照下面的风格命名。我们建议社区贡献者使用同样的风格。

```bash
{model}_[model setting]_{backbone}_[refiner]_[norm setting]_[misc]_[gpu x batch_per_gpu]_{schedule}_{dataset}
```

`{xxx}` 是必填字段，`[yyy]` 是可选的。

- `{model}`: 模型种类，例如 `srcnn`, `dim` 等等。
- `[model setting]`: 特定设置一些模型，例如，输入图像 `resolution` , 训练 `stage name`。
- `{backbone}`: 主干网络种类，例如 `r50` (ResNet-50)、`x101` (ResNeXt-101)。
- `{refiner}`: 精炼器种类，例如 `pln` 简单精炼器模型
- `[norm_setting]`: 指定归一化设置，默认为批归一化，其他归一化可以设为: `bn`(批归一化), `gn` (组归一化), `syncbn` (同步批归一化)。
- `[misc]`: 模型中各式各样的设置/插件，例如 `dconv`, `gcb`, `attention`, `mstrain`。
- `[gpu x batch_per_gpu]`: GPU数目 和每个 GPU 的样本数， 默认为 `8x2 `。
- `{schedule}`: 训练策略，如 `20k`, `100k` 等，意思是 `20k` 或 `100k` 迭代轮数。
- `{dataset}`: 数据集，如 `places`（图像补全）、`comp1k`（抠图）、`div2k`（图像恢复）和 `paired`（图像生成）。

## 配置文件 - 生成

与 [MMDetection](https://github.com/open-mmlab/mmdetection) 一样，我们将模块化和继承设计融入我们的配置系统，以方便进行各种实验。

## 示例 - pix2pix

## 配置文件 - 生成

为了帮助用户对完整的配置和生成系统中的模块有一个基本的了解，我们对 pix2pix 的配置做如下简要说明。
更详细的用法和各个模块对应的替代方案，请参考 API 文档。

```python
# 模型设置
    type='Pix2Pix',  # 合成器名称
        type='UnetGenerator',  # 生成器名称
        in_channels=3,  # 生成器的输入通道数
        out_channels=3,  # 生成器的输出通道数
        num_down=8,  # # 生成器中下采样的次数
        base_channels=64,  # 生成器最后卷积层的通道数
        norm_cfg=dict(type='BN'),  # 归一化层的配置
        use_dropout=True,  # 是否在生成器中使用 dropout
        init_cfg=dict(type='normal', gain=0.02)),  # 初始化配置
        type='PatchDiscriminator',  # 判别器的名称
        in_channels=6,  # 判别器的输入通道数
        base_channels=64,  # 判别器第一卷积层的通道数
        num_conv=3,  # 判别器中堆叠的中间卷积层（不包括输入和输出卷积层）的数量
    gan_loss=dict(
        type='GANLoss',  # GAN 损失的名称

norm_cfg=dict(type='BN'),  # 归一化层的配置
        gan_type='vanilla',  # GAN 损失的类型
        real_label_val=1.0,  # GAN 损失函数中真实标签的值
        fake_label_val=0.0,  # GAN 损失函数中伪造标签的值
        loss_weight=1.0),  # GAN 损失函数的权重
    pixel_loss=dict(type='L1Loss', loss_weight=100.0, reduction='mean'))
# 模型训练和测试设置
    direction='b2a')  # pix2pix 的图像到图像的转换方向 (模型训练的方向，和测试方向一致)。模型默认: a2b
test_cfg = dict(
    direction='b2a',   # pix2pix 的图像到图像的转换方向 (模型测试的方向，和训练方向一致)。模型默认: a2b
    show_input=True)  # 保存 pix2pix 的测试图像时是否显示输入的真实图像

# 数据设置
train_dataset_type = 'GenerationPairedDataset'  # 训练数据集的类型
val_dataset_type = 'GenerationPairedDataset'  # 验证/测试数据集类型
img_norm_cfg = dict(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])  # 输入图像归一化配置
    dict(
        type='LoadPairedImageFromFile',  # 从文件路径加载图像对
        io_backend='disk',  # 存储图像的 IO 后端

    dict(
        key='pair',  # 查找对应路径的关键词
        flag='color'),  # 加载图像标志
    dict(
        type='Resize',  # 图像大小调整
        keys=['img_a', 'img_b'],  # 要调整大小的图像的关键词
        scale=(286, 286),  # 调整图像大小的比例
        interpolation='bicubic'),  # 调整图像大小时用于插值的算法
    dict(
        type='FixedCrop',  # 固定裁剪，在特定位置将配对图像裁剪为特定大小以训练 pix2pix
        keys=['img_a', 'img_b'],  # 要裁剪的图像的关键词
        crop_size=(256, 256)),  # 裁剪图像的大小
    dict(
        type='Flip',  # 翻转图像
        keys=['img_a', 'img_b'],  # 要翻转的图像的关键词
        direction='horizontal'),  # 水平或垂直翻转图像
    dict(
        type='RescaleToZeroOne',  # 将图像从 [0, 255] 缩放到 [0, 1]
        keys=['img_a', 'img_b']),  # 要重新缩放的图像的关键词
    dict(
        type='Normalize',  # 图像归一化
        keys=['img_a', 'img_b'],  # 要归一化的图像的关键词

keys=['img_a', 'img_b']),  # 要重新缩放的图像的关键词
    dict(
        to_rgb=True,  # 是否将图像通道从 BGR 转换为 RGB
        **img_norm_cfg),  # 图像归一化配置（`img_norm_cfg` 的定义见上文）
    dict(
       type='ToTensor',  # 将图像转化为 Tensor
       keys=['img_a', 'img_b']),  # 要从图像转换为 Tensor 的图像的关键词
    dict(
        type='Collect',  # 决定数据中哪些键应该传递给合成器
        keys=['img_a', 'img_b'],  # 图像的关键词
        meta_keys=['img_a_path', 'img_b_path'])  # 图片的元关键词
]
    dict(
    dict(
        scale=(256, 256),  # 调整图像大小的比例
    dict(

scale=(256, 256),  # 调整图像大小的比例
    dict(
    dict(
    dict(
    dict(
]
data_root = 'data/pix2pix/facades'  # 数据的根路径
    samples_per_gpu=1,  # 单个 GPU 的批量大小
    workers_per_gpu=4,  # 为每个 GPU 预取数据的 Worker 数
    drop_last=True,  # 是否丢弃训练中的最后一批数据
    val_samples_per_gpu=1,  # 验证中单个 GPU 的批量大小
    val_workers_per_gpu=0,  # 在验证中为每个 GPU 预取数据的 Worker 数

drop_last=True,  # 是否丢弃训练中的最后一批数据
        type=train_dataset_type,
        dataroot=data_root,
        test_mode=False),
    val=dict(  # 验证数据集配置
        type=val_dataset_type,
    test=dict(  # 测试数据集配置

# 优化器
optimizers = dict(  # 用于构建优化器的配置，支持 PyTorch 中所有优化器，且参数与 PyTorch 中对应优化器相同
    generator=dict(type='Adam', lr=2e-4, betas=(0.5, 0.999)),
    discriminator=dict(type='Adam', lr=2e-4, betas=(0.5, 0.999)))

# 学习策略
lr_config = dict(policy='Fixed', by_epoch=False)  # 用于注册 LrUpdater 钩子的学习率调度程序配置

# 检查点保存

discriminator=dict(type='Adam', lr=2e-4, betas=(0.5, 0.999)))

# 学习策略

# 检查点保存
checkpoint_config = dict(interval=4000, save_optimizer=True, by_epoch=False)  # 配置检查点钩子，实现参考 https://github.com/open-mmlab/mmcv/blob/master/mmcv/runner/hooks/checkpoint.py
evaluation = dict(  # 构建验证钩子的配置
    interval=4000,  # 验证区间
    save_image=True)  # 是否保存图片
log_config = dict(  # 配置注册记录器钩子
    interval=100,  # 打印日志的时间间隔
    hooks=[
        dict(type='TextLoggerHook', by_epoch=False),  # 用于记录训练过程的记录器
        # dict(type='TensorboardLoggerHook')  # 还支持 Tensorboard 记录器
    ])
visual_config = None  # 构建可视化钩子的配置

# 运行设置
total_iters = 80000  # 训练模型的总迭代次数
cudnn_benchmark = True  # 设置 cudnn_benchmark
dist_params = dict(backend='nccl')  # 设置分布式训练的参数，端口也可以设置
log_level = 'INFO'  # 日志级别
load_from = None  # 从给定路径加载模型作为预训练模型。 这不会恢复训练
resume_from = None  # 从给定路径恢复检查点，当检查点被保存时，训练将从该 epoch 恢复

workflow = [('train', 1)]  # runner 的工作流程。 [('train', 1)] 表示只有一个工作流程，名为 'train' 的工作流程执行一次。 训练当前生成模型时保持不变
exp_name = 'pix2pix_facades'  # 实验名称
work_dir = f'./work_dirs/{exp_name}'  # 保存当前实验的模型检查点和日志的目录
```

## 配置文件 - 补全

## 配置名称样式

## 配置字段说明

为了帮助用户对完整的配置和修复系统中的模块有一个基本的了解，我们对 Global&Local 的配置作如下简要说明。更详细的用法和各个模块对应的替代方案，请参考 API 文档。

```python
    type='GLInpaintor', # 补全器的名称
        type='GLEncoderDecoder', # 编码器-解码器的名称
        encoder=dict(type='GLEncoder', norm_cfg=dict(type='SyncBN')), # 编码器的配置
        decoder=dict(type='GLDecoder', norm_cfg=dict(type='SyncBN')), # 解码器的配置
            type='GLDilationNeck', norm_cfg=dict(type='SyncBN'))), # 扩颈的配置

decoder=dict(type='GLDecoder', norm_cfg=dict(type='SyncBN')), # 解码器的配置
        type='GLDiscs', # 判别器的名称
            in_channels=3, # 判别器的输入通道数
            max_channels=512, # 判别器中的最大通道数
            fc_in_channels=512 * 4 * 4, # 最后一个全连接层的输入通道
            fc_out_channels=1024, # 最后一个全连接层的输出通道
            num_convs=6, # 判别器中使用的卷积数量
            norm_cfg=dict(type='SyncBN') # 归一化层的配置
        ),
            num_convs=5, # 判别器中使用的卷积数量
        ),
    ),
        type='GANLoss', # GAN 损失的名称
        gan_type='vanilla', # GAN 损失的类型

norm_cfg=dict(type='SyncBN') # 归一化层的配置
        ),
    ),
        loss_weight=0.001 # GAN 损失函数的权重
    ),
        type='L1Loss', # L1 损失的类型
        loss_weight=1.0 # L1 损失函数的权重
    ),
    pretrained=None) # 预训练权重的路径

    disc_step=1, # 训练生成器之前训练判别器的迭代次数
    iter_tc=90000, # 预热生成器的迭代次数
    iter_td=100000, # 预热判别器的迭代次数
    start_iter=0, # 开始的迭代
    local_size=(128, 128)) # 图像块的大小
test_cfg = dict(metrics=['l1']) # 测试的配置

dataset_type = 'ImgInpaintingDataset' # 数据集类型
input_shape = (256, 256) # 输入图像的形状

    dict(type='LoadImageFromFile', key='gt_img'), # 加载图片的配置
    dict(
        type='LoadMask', # 加载掩码
        mask_mode='bbox', # 掩码的类型
            max_bbox_shape=(128, 128), # 检测框的形状
            max_bbox_delta=40, # 检测框高宽的变化
            min_margin=20,  # 检测框到图片边界的最小距离

mask_mode='bbox', # 掩码的类型
            img_shape=input_shape)),  # 输入图像的形状
    dict(
        type='Crop', # 裁剪
        keys=['gt_img'],  # 要裁剪的图像的关键词
        crop_size=(384, 384),  # 裁剪图像块的大小
        random_crop=True,  # 是否使用随机裁剪
    ),
    dict(
        keys=['gt_img'],  # 要调整大小的图像的关键词
        scale=input_shape,  # 调整图像大小的比例
        keep_ratio=False,  # 调整大小时是否保持比例
    ),
    dict(
        keys=['gt_img'],  # 要归一化的图像的关键词
        mean=[127.5] * 3,  # 归一化中使用的均值
        std=[127.5] * 3,  # 归一化中使用的标准差
        to_rgb=False),  # 是否将图像通道从 BGR 转换为 RGB
    dict(type='GetMaskedImage'),  # 获取被掩盖的图像
    dict(
        keys=['gt_img', 'masked_img', 'mask', 'mask_bbox'],  # 要收集的数据的关键词

dict(type='GetMaskedImage'),  # 获取被掩盖的图像
    dict(
        meta_keys=['gt_img_path']),  # 要收集的数据的元关键词
    dict(type='ToTensor', keys=['gt_img', 'masked_img', 'mask']),  # 将图像转化为 Tensor
    dict(type='ToTensor', keys=['mask_bbox'])  # 转化为 Tensor
]

test_pipeline = train_pipeline  # 构建测试/验证流程

data_root = 'data/places365'  # 数据根目录

    samples_per_gpu=12,  # 单个 GPU 的批量大小
    workers_per_gpu=8,  # 为每个 GPU 预取数据的 Worker 数
    val_workers_per_gpu=8,  # 在验证中为每个 GPU 预取数据的 Worker 数
        ann_file=f'{data_root}/train_places_img_list_total.txt',

data_prefix=data_root,
        ann_file=f'{data_root}/val_places_img_list.txt',

    generator=dict(type='Adam', lr=0.0004), disc=dict(type='Adam', lr=0.0004))

checkpoint_config = dict(by_epoch=False, interval=50000)  # 配置检查点钩子，实现参考 https://github.com/open-mmlab/mmcv/blob/master/mmcv/runner/hooks/checkpoint.py
    hooks=[
        dict(type='TextLoggerHook', by_epoch=False),
        # dict(type='TensorboardLoggerHook'),  # 支持 Tensorboard 记录器
        # dict(type='PaviLoggerHook', init_kwargs=dict(project='mmagic'))
    ])  # 用于记录训练过程的记录器

# dict(type='TensorboardLoggerHook'),  # 支持 Tensorboard 记录器

visual_config = dict(  # 构建可视化钩子的配置
    res_name_list=[
        'gt_img', 'masked_img', 'fake_res', 'fake_img', 'fake_gt_local'
    ],
)  # 用于可视化训练过程的记录器。

evaluation = dict(interval=50000)  # 构建验证钩子的配置

total_iters = 500002
work_dir = None  # 保存当前实验的模型检查点和日志的目录
workflow = [('train', 10000)]  # runner 的工作流程。 [('train', 1)] 表示只有一个工作流程，名为 'train' 的工作流程执行一次。 训练当前生成模型时保持不变
exp_name = 'gl_places'  # 实验名称
find_unused_parameters = False  # 是否在分布式训练中查找未使用的参数
```

## 配置文件 - 抠图

```

## 配置文件 - 抠图

## 例子 - Deep Image Matting Model

为了帮助用户对一个完整的配置有一个基本的了解，我们对我们实现的原始 DIM 模型的配置做一个简短的评论，如下所示。 更详细的用法和各个模块对应的替代方案，请参考 API 文档。

```python
# 模型配置
    type='DIM',  # 模型的名称（我们称之为抠图器）
    backbone=dict(  # 主干网络的配置
        type='SimpleEncoderDecoder',  # 主干网络的类型
        encoder=dict(  # 编码器的配置
            type='VGG16'),  # 编码器的类型
        decoder=dict(  # 解码器的配置
            type='PlainDecoder')),  # 解码器的类型
    pretrained='./weights/vgg_state_dict.pth',  # 编码器的预训练权重
    loss_alpha=dict(  # alpha 损失的配置
        type='CharbonnierLoss',  # 预测的 alpha 遮罩的损失类型
        loss_weight=0.5),  # alpha 损失的权重
    loss_comp=dict(  # 组合损失的配置
        type='CharbonnierCompLoss',  # 组合损失的类型
        loss_weight=0.5))  # 组合损失的权重
train_cfg = dict(  # 训练 DIM 模型的配置

loss_weight=0.5),  # alpha 损失的权重
    train_backbone=True,  # 在 DIM stage 1 中，会对主干网络进行训练
    train_refiner=False)  # 在 DIM stage 1 中，不会对精炼器进行训练
test_cfg = dict(  # 测试 DIM 模型的配置
    refine=False,  # 是否使用精炼器输出作为输出，在 stage 1 中，我们不使用它
    metrics=['SAD', 'MSE', 'GRAD', 'CONN'])  # 测试时使用的指标

# 数据配置
dataset_type = 'AdobeComp1kDataset'  # 数据集类型，这将用于定义数据集
data_root = 'data/adobe_composition-1k'  # 数据的根目录
img_norm_cfg = dict(  # 归一化输入图像的配置
    mean=[0.485, 0.456, 0.406],  # 归一化中使用的均值
    std=[0.229, 0.224, 0.225],  # 归一化中使用的标准差
    to_rgb=True)  # 是否将图像通道从 BGR 转换为 RGB
    dict(
        type='LoadImageFromFile',  # 从文件加载 alpha 遮罩
        key='alpha',  # 注释文件中 alpha 遮罩的键关键词。流程将从路径 “alpha_path” 中读取 alpha 遮罩
        flag='grayscale'),  # 加载灰度图像，形状为（高度、宽度）
    dict(
        type='LoadImageFromFile',  # 从文件中加载图像

key='alpha',  # 注释文件中 alpha 遮罩的键关键词。流程将从路径 “alpha_path” 中读取 alpha 遮罩
    dict(
        key='fg'),  # 要加载的图像的关键词。流程将从路径 “fg_path” 读取 fg
    dict(
        key='bg'),  # 要加载的图像的关键词。流程将从路径 “bg_path” 读取 bg
    dict(
        key='merged'),  # 要加载的图像的关键词。流程将从路径 “merged_path” 读取并合并
    dict(
        type='CropAroundUnknown',  # 在未知区域（半透明区域）周围裁剪图像
        keys=['alpha', 'merged', 'ori_merged', 'fg', 'bg'],  # 要裁剪的图像
        crop_sizes=[320, 480, 640]),  # 裁剪大小
    dict(
        keys=['alpha', 'merged', 'ori_merged', 'fg', 'bg']),  # 要翻转的图像
    dict(
        keys=['alpha', 'merged', 'ori_merged', 'fg', 'bg'],  # 图像调整大小的图像
        scale=(320, 320),  # 目标大小
        keep_ratio=False),  # 是否保持高宽比例
    dict(

dict(
    dict(
        type='GenerateTrimap',  # 从 alpha 遮罩生成三元图。
        kernel_size=(1, 30)),  # 腐蚀/扩张内核大小的范围
    dict(
        type='RescaleToZeroOne', # 将图像从 [0, 255] 缩放到 [0, 1]
        keys=['merged', 'alpha', 'ori_merged', 'fg', 'bg']),  # 要重新缩放的图像
    dict(
        keys=['merged'],  # 要归一化的图像
    dict(
        keys=['merged', 'alpha', 'trimap', 'ori_merged', 'fg', 'bg'],  # 图像的关键词
        meta_keys=[]),  # 图片的元关键词，这里不需要元信息。
    dict(
        type='ToTensor',  # 将图像转化为 Tensor
        keys=['merged', 'alpha', 'trimap', 'ori_merged', 'fg', 'bg']),  # 要转换为 Tensor 的图像
]
    dict(
        type='LoadImageFromFile', # 从文件加载 alpha 遮罩

keys=['merged', 'alpha', 'trimap', 'ori_merged', 'fg', 'bg']),  # 要转换为 Tensor 的图像
]
    dict(
        flag='grayscale',
    dict(
        key='trimap',  # 要加载的图像的关键词。流程将从路径 “trimap_path” 读取三元图
        flag='grayscale', # 加载灰度图像，形状为（高度、宽度）
        save_original_img=True), # 保存三元图用于计算指标。 它将与 “ori_trimap” 一起保存
    dict(
    dict(
        type='Pad',  # 填充图像以与模型的下采样因子对齐
        keys=['trimap', 'merged'],  # 要填充的图像
        mode='reflect'),  # 填充模式
    dict(
        type='RescaleToZeroOne',  # 与 train_pipeline 相同
        keys=['merged', 'ori_alpha']),  # 要缩放的图像
    dict(
        type='Normalize',  # 与 train_pipeline 相同
        keys=['merged'],

dict(
    dict(
        **img_norm_cfg),
    dict(
        type='Collect',  # 与 train_pipeline 相同
        keys=['merged', 'trimap'],
        meta_keys=[
            'merged_path', 'pad', 'merged_ori_shape', 'ori_alpha',
            'ori_trimap'
        ]),
    dict(
        type='ToTensor',  # 与 train_pipeline 相同
        keys=['merged', 'trimap']),
]
    samples_per_gpu=1,  #单个 GPU 的批量大小
        type=dataset_type,  # 数据集的类型
        ann_file=f'{data_root}/training_list.json',  # 注解文件路径
        data_prefix=data_root,  # 图像路径的前缀
        pipeline=train_pipeline),  # 见上文 train_pipeline
        ann_file=f'{data_root}/test_list.json',

data_prefix=data_root,  # 图像路径的前缀
        pipeline=test_pipeline),  # 见上文 test_pipeline
        pipeline=test_pipeline))  # 见上文 test_pipeline

# 优化器
optimizers = dict(type='Adam', lr=0.00001)  # 用于构建优化器的配置，支持 PyTorch 中所有优化器，且参数与 PyTorch 中对应优化器相同
# 学习策略
lr_config = dict(  # 用于注册 LrUpdater 钩子的学习率调度程序配置
    policy='Fixed')  # 调度器的策略，支持 CosineAnnealing、Cyclic 等。支持的 LrUpdater 详情请参考 https://github.com/open-mmlab/mmcv/blob/master/mmcv/runner/hooks/lr_updater.py#L9。

# 检查点保存
checkpoint_config = dict(  # 配置检查点钩子，实现参考 https://github.com/open-mmlab/mmcv/blob/master/mmcv/runner/hooks/checkpoint.py
    interval=40000,  # 保存间隔为 40000 次迭代
    by_epoch=False)  # 按迭代计数

# 检查点保存
evaluation = dict(  # # 构建验证钩子的配置
    interval=40000)  # 验证区间
    interval=10,  # 打印日志的时间间隔
    hooks=[
        # dict(type='TensorboardLoggerHook')  # 支持 Tensorboard 记录器
    ])

total_iters = 1000000  # 训练模型的总迭代次数
work_dir = './work_dirs/dim_stage1'  # 保存当前实验的模型检查点和日志的目录
workflow = [('train', 1)]  # runner 的工作流程。 [('train', 1)] 表示只有一个工作流程，名为 'train' 的工作流程执行一次。 训练当前抠图模型时保持不变
```

## 配置文件 - 复原

## 示例-EDSR

为了帮助用户理解 mmagic 的配置文件结构，这里以 EDSR 为例，给出其配置文件的注释。对于每个模块的详细用法以及对应参数的选择，请参照 API 文档。

```

## 配置文件 - 复原

## 示例-EDSR

```python
exp_name = 'edsr_x2c64b16_1x16_300k_div2k'  # 实验名称

scale = 2  # 上采样放大因子

# 模型设置
    type='BasicRestorer',  # 图像恢复模型类型
    generator=dict(  # 生成器配置
        type='EDSR',  # 生成器类型
        in_channels=3,  # 输入通道数
        out_channels=3,  # 输出通道数
        mid_channels=64,  # 中间特征通道数
        num_blocks=16,  # 残差块数目
        upscale_factor=scale, # 上采样因子
        res_scale=1,  # 残差缩放因子
        rgb_mean=(0.4488, 0.4371, 0.4040),  # 输入图像 RGB 通道的平均值
        rgb_std=(1.0, 1.0, 1.0)),  # 输入图像 RGB 通道的方差
    pixel_loss=dict(type='L1Loss', loss_weight=1.0, reduction='mean'))  # 像素损失函数的配置

train_cfg = None  # 训练的配置
test_cfg = dict(  # 测试的配置
    metrics=['PSNR'],  # 测试时使用的评价指标
    crop_border=scale)  # 测试时裁剪的边界尺寸

# 数据集设置
train_dataset_type = 'SRAnnotationDataset'  # 用于训练的数据集类型
val_dataset_type = 'SRFolderDataset'  #  用于验证的数据集类型

metrics=['PSNR'],  # 测试时使用的评价指标

# 数据集设置
train_pipeline = [  # 训练数据前处理流水线步骤组成的列表
    dict(type='LoadImageFromFile',  # 从文件加载图像
        io_backend='disk',  # 读取图像时使用的 io 类型
        key='lq',  # 设置LR图像的键来找到相应的路径
        flag='unchanged'),  # 读取图像的标识
        io_backend='disk',  # 读取图像时使用的io类型
        key='gt',  # 设置HR图像的键来找到相应的路径
    dict(type='RescaleToZeroOne', keys=['lq', 'gt']),  # 将图像从[0，255]重缩放到[0，1]
    dict(type='Normalize',  # 正则化图像
        keys=['lq', 'gt'],  # 执行正则化图像的键
        mean=[0, 0, 0],  # 平均值
        std=[1, 1, 1],  # 标准差
        to_rgb=True),  # 更改为 RGB 通道
    dict(type='PairedRandomCrop', gt_patch_size=96),  # LR 和 HR 成对随机裁剪
    dict(type='Flip',  # 图像翻转
        keys=['lq', 'gt'],  # 执行翻转图像的键
        flip_ratio=0.5,  # 执行翻转的几率

dict(type='PairedRandomCrop', gt_patch_size=96),  # LR 和 HR 成对随机裁剪
        direction='horizontal'),  # 翻转方向
        flip_ratio=0.5,  # 执行翻转几率
        direction='vertical'),  # 翻转方向
    dict(type='RandomTransposeHW',  # 图像的随机的转置
        keys=['lq', 'gt'],  # 执行转置图像的键
        transpose_ratio=0.5  # 执行转置的几率
        ),
    dict(type='Collect',  # Collect 类决定哪些键会被传递到生成器中
        keys=['lq', 'gt'],  # 传入模型的键
        meta_keys=['lq_path', 'gt_path']), # 元信息键。在训练中，不需要元信息
    dict(type='ToTensor',  # 将图像转换为张量
        keys=['lq', 'gt'])  # 执行图像转换为张量的键
]
test_pipeline = [  # 测试数据前处理流水线步骤组成的列表
    dict(
        type='LoadImageFromFile',  # 从文件加载图像
    dict(

io_backend='disk',  # 读取图像时使用的io类型
    dict(
    dict(
        type='Normalize',  # 正则化图像
        to_rgb=True),  # 更改为RGB通道
    dict(type='Collect',  # Collect类决定哪些键会被传递到生成器中
        meta_keys=['lq_path', 'gt_path']),  # 元信息键
]

    # 训练
    samples_per_gpu=16,  # 单个 GPU 的批大小
    workers_per_gpu=6,  # 单个 GPU 的 dataloader 的进程
    drop_last=True,  # 在训练过程中丢弃最后一个批次
    train=dict(  # 训练数据集的设置
        type='RepeatDataset',  # 基于迭代的重复数据集

samples_per_gpu=16,  # 单个 GPU 的批大小
        times=1000,  # 重复数据集的重复次数
            type=train_dataset_type,  # 数据集类型
            lq_folder='data/DIV2K/DIV2K_train_LR_bicubic/X2_sub',  # lq文件夹的路径
            gt_folder='data/DIV2K/DIV2K_train_HR_sub',  # gt文件夹的路径
            ann_file='data/DIV2K/meta_info_DIV2K800sub_GT.txt',  # 批注文件的路径
            pipeline=train_pipeline,  # 训练流水线，如上所示
            scale=scale)),  # 上采样放大因子

    # 验证
    val_samples_per_gpu=1,  # 验证时单个 GPU 的批大小
    val_workers_per_gpu=1,  # 验证时单个 GPU 的 dataloader 的进程
    val=dict(
        type=val_dataset_type,  # 数据集类型
        lq_folder='data/val_set5/Set5_bicLRx2',  # lq 文件夹的路径
        gt_folder='data/val_set5/Set5_mod12',  # gt 文件夹的路径
        pipeline=test_pipeline,  # 测试流水线，如上所示
        scale=scale,  # 上采样放大因子

lq_folder='data/val_set5/Set5_bicLRx2',  # lq 文件夹的路径
        filename_tmpl='{}'),  # 文件名模板

    # 测试
        filename_tmpl='{}'))  # 文件名模板

# 优化器设置
optimizers = dict(generator=dict(type='Adam', lr=1e-4, betas=(0.9, 0.999)))  # 用于构建优化器的设置，支持PyTorch中所有参数与PyTorch中参数相同的优化器

# 学习策略
total_iters = 300000  # 训练模型的总迭代数
lr_config = dict( # 用于注册LrUpdater钩子的学习率调度程序配置
    policy='Step', by_epoch=False, step=[200000], gamma=0.5)  # 调度器的策略，还支持余弦、循环等

checkpoint_config = dict(  # 模型权重钩子设置，更多细节可参考 https://github.com/open-mmlab/mmcv/blob/master/mmcv/runner/hooks/checkpoint.py

    interval=5000,  # 模型权重文件保存间隔为5000次迭代
    save_optimizer=True,  # 保存优化器
    by_epoch=False)  # 按迭代次数计数
    interval=5000,  # 执行验证的间隔为5000次迭代
    save_image=True,  # 验证期间保存图像
    gpu_collect=True)  # 使用gpu收集
log_config = dict(  # 注册日志钩子的设置
    interval=100,  # 打印日志间隔
    hooks=[
        dict(type='TextLoggerHook', by_epoch=False),  # 记录训练过程信息的日志
        dict(type='TensorboardLoggerHook'),  # 同时支持 Tensorboard 日志
    ])
visual_config = None  # 可视化的设置

# 运行设置
dist_params = dict(backend='nccl')  # 建立分布式训练的设置，其中端口号也可以设置
work_dir = f'./work_dirs/{exp_name}'  # 记录当前实验日志和模型权重文件的文件夹
load_from = None # 从给定路径加载模型作为预训练模型. 这个选项不会用于断点恢复训练
resume_from = None # 加载给定路径的模型权重文件作为断点续连的模型, 训练将从该时间点保存的周期点继续进行
workflow = [('train', 1)]  # runner 的执行流. [('train', 1)] 代表只有一个执行流，并且这个名为 train 的执行流只执行一次
```

## Webcam Demo

The original Webcam API has been deprecated starting from version v1.1.0. Users now have the option to utilize either the Inferencer or the demo script for conducting pose estimation using webcam input.

### Webcam Demo with Inferencer

Users can utilize the MMPose Inferencer to estimate human poses in webcam inputs by executing the following command:

```shell
```

For additional information about the arguments of Inferencer, please refer to the [Inferencer Documentation](/docs/en/user_guides/inference.md).

### Webcam Demo with Demo Script

All of the demo scripts, except for `demo/image_demo.py`, support webcam input.

Take `demo/topdown_demo_with_mmdet.py` as example, users can utilize this script with webcam input by specifying **`--input webcam`** in the command:

```shell

```shell
    --show
```

# Legacy Configs in MMDetection V1.x

Configs in this directory implement the legacy configs used by MMDetection V1.x and its model zoos.

To help users convert their models from V1.x to MMDetection V2.0, we provide v1.x configs to inference the converted v1.x models.
Due to the BC-breaking changes in MMDetection V2.0 from MMDetection V1.x, running inference with the same model weights in these two version will produce different results. The difference will cause within 1% AP absolute difference as can be found in the following table.

## Usage

To upgrade the model version, the users need to do the following steps.

### 1. Convert model weights

There are three main difference in the model weights between V1.x and V2.0 codebases.

1. Since the class order in all the detector's classification branch is reordered, all the legacy model weights need to go through the conversion process.

2. The regression and segmentation head no longer contain the background channel. Weights in these background channels should be removed to fix in the current codebase.
3. For two-stage detectors, their wegihts need to be upgraded since MMDetection V2.0 refactors all the two-stage detectors with `RoIHead`.

The users can do the same modification as mentioned above for the self-implemented
detectors. We provide a scripts `tools/model_converters/upgrade_model_version.py` to convert the model weights in the V1.x model zoo.

```bash
python tools/model_converters/upgrade_model_version.py ${OLD_MODEL_PATH} ${NEW_MODEL_PATH} --num-classes ${NUM_CLASSES}

```

- OLD_MODEL_PATH: the path to load the model weights in 1.x version.
- NEW_MODEL_PATH: the path to save the converted model weights in 2.0 version.

```

- NUM_CLASSES: number of classes of the original model weights. Usually it is 81 for COCO dataset, 21 for VOC dataset.
  The number of classes in V2.0 models should be equal to that in V1.x models - 1.

### 2. Use configs with legacy settings

After converting the model weights, checkout to the v1.2 release to find the corresponding config file that uses the legacy settings.
The V1.x models usually need these three legacy modules: `LegacyAnchorGenerator`, `LegacyDeltaXYWHBBoxCoder`, and `RoIAlign(align=False)`.
For models using ResNet Caffe backbones, they also need to change the pretrain name and the corresponding `img_norm_cfg`.
An example is in [`retinanet_r50-caffe_fpn_1x_coco_v1.py`](retinanet_r50-caffe_fpn_1x_coco_v1.py)

Then use the config to test the model weights. For most models, the obtained results should be close to that in V1.x.
We provide configs of some common structures in this directory.

The performance change after converting the models in this directory are listed as the following.

|           Method            |  Style  | Lr schd | V1.x box AP | V1.x mask AP | V2.0 box AP | V2.0 mask AP |                       Config                        |                                                             Download                                                              |
| :-------------------------: | :-----: | :-----: | :---------: | :----------: | :---------: | :----------: | :-------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------: |

|     Mask R-CNN R-50-FPN     | pytorch |   1x    |    37.3     |     34.2     |    36.8     |     33.9     |     [config](./mask-rcnn_r50_fpn_1x_coco_v1.py)     |     [model](https://s3.ap-northeast-2.amazonaws.com/open-mmlab/mmdetection/models/mask_rcnn_r50_fpn_1x_20181010-069fa190.pth)     |
|     RetinaNet R-50-FPN      |  caffe  |   1x    |    35.8     |      -       |    35.4     |      -       |  [config](./retinanet_r50-caffe_fpn_1x_coco_v1.py)  |                                                                                                                                   |
|     RetinaNet R-50-FPN      | pytorch |   1x    |    35.6     |      -       |    35.2     |      -       |     [config](./retinanet_r50_fpn_1x_coco_v1.py)     |     [model](https://s3.ap-northeast-2.amazonaws.com/open-mmlab/mmdetection/models/retinanet_r50_fpn_1x_20181125-7b0c2548.pth)     |

| Cascade Mask R-CNN R-50-FPN | pytorch |   1x    |    41.2     |     35.7     |    40.8     |     35.6     | [config](./cascade-mask-rcnn_r50_fpn_1x_coco_v1.py) | [model](https://s3.ap-northeast-2.amazonaws.com/open-mmlab/mmdetection/models/cascade_mask_rcnn_r50_fpn_1x_20181123-88b170c9.pth) |
|        SSD300-VGG16         |  caffe  |  120e   |    25.7     |      -       |    25.4     |      -       |            [config](./ssd300_coco_v1.py)            | [model](https://s3.ap-northeast-2.amazonaws.com/open-mmlab/mmdetection/models/ssd300_coco_vgg16_caffe_120e_20181221-84d7110b.pth) |

# Panoptic FPN

> [Panoptic feature pyramid networks](https://arxiv.org/abs/1901.02446)

<img src="https://user-images.githubusercontent.com/40661020/143968979-a1593758-c9d7-44a6-a3b8-d9686ef19ce8.png" height="300"/>
</div>

## Dataset

PanopticFPN requires COCO and [COCO-panoptic](http://images.cocodataset.org/annotations/panoptic_annotations_trainval2017.zip) dataset for training and evaluation. You need to download and extract it in the COCO dataset path.

```none
├── mmdet
├── tools
├── data
│   ├── coco
```

```

| Backbone  |  style  | Lr schd | Mem (GB) | Inf time (fps) |  PQ  |  SQ  |  RQ  | PQ_th | SQ_th | RQ_th | PQ_st | SQ_st | RQ_st |                     Config                      |                                                                                                                                                                          Download                                                                                                                                                                          |

| R-50-FPN  | pytorch |   1x    |   4.7    |                | 40.2 | 77.8 | 49.3 | 47.8  | 80.9  | 57.5  | 28.9  | 73.1  | 37.0  |   [config](./panoptic-fpn_r50_fpn_1x_coco.py)   |                   [model](https://download.openmmlab.com/mmdetection/v2.0/panoptic_fpn/panoptic_fpn_r50_fpn_1x_coco/panoptic_fpn_r50_fpn_1x_coco_20210821_101153-9668fd13.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/panoptic_fpn/panoptic_fpn_r50_fpn_1x_coco/panoptic_fpn_r50_fpn_1x_coco_20210821_101153.log.json)                   |

| R-50-FPN  | pytorch |   3x    |    -     |       -        | 42.5 | 78.1 | 51.7 | 50.3  | 81.5  | 60.3  | 30.7  | 73.0  | 38.8  | [config](./panoptic-fpn_r50_fpn_ms-3x_coco.py)  |   [model](https://download.openmmlab.com/mmdetection/v2.0/panoptic_fpn/panoptic_fpn_r50_fpn_mstrain_3x_coco/panoptic_fpn_r50_fpn_mstrain_3x_coco_20210824_171155-5650f98b.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/panoptic_fpn/panoptic_fpn_r50_fpn_mstrain_3x_coco/panoptic_fpn_r50_fpn_mstrain_3x_coco_20210824_171155.log.json)   |

| R-101-FPN | pytorch |   1x    |   6.7    |                | 42.2 | 78.3 | 51.4 | 50.1  | 81.4  | 59.9  | 30.3  | 73.6  | 38.5  |  [config](./panoptic-fpn_r101_fpn_1x_coco.py)   |                 [model](https://download.openmmlab.com/mmdetection/v2.0/panoptic_fpn/panoptic_fpn_r101_fpn_1x_coco/panoptic_fpn_r101_fpn_1x_coco_20210820_193950-ab9157a2.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/panoptic_fpn/panoptic_fpn_r101_fpn_1x_coco/panoptic_fpn_r101_fpn_1x_coco_20210820_193950.log.json)                 |

| R-101-FPN | pytorch |   3x    |    -     |       -        | 44.1 | 78.9 | 53.6 | 52.1  | 81.7  | 62.3  | 32.0  | 74.6  | 40.3  | [config](./panoptic-fpn_r101_fpn_ms-3x_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/panoptic_fpn/panoptic_fpn_r101_fpn_mstrain_3x_coco/panoptic_fpn_r101_fpn_mstrain_3x_coco_20210823_114712-9c99acc4.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/panoptic_fpn/panoptic_fpn_r101_fpn_mstrain_3x_coco/panoptic_fpn_r101_fpn_mstrain_3x_coco_20210823_114712.log.json) |

The base method for panoptic segmentation task.

```latex
@inproceedings{kirillov2018panopticfpn,
  author = {
    Alexander Kirillov,
    Ross Girshick,
    Kaiming He,
    Piotr Dollar,
  },
  title = {Panoptic Feature Pyramid Networks},
}
```

# 准备 Something-Something V2

## 简介

```BibTeX
}
```

用户可参考该数据集的 [官网](https://developer.qualcomm.com/software/ai-datasets/something-something)，以获取数据集相关的基本信息。

# MIM 支持下载 Something-Something V2 数据集。用户可以通过一行命令，从 OpenDataLab 进行下载，并进行预处理。
```Bash
odl login
```

````

```Bash
odl login
```

````

首先，用户需要在 [官网](https://developer.qualcomm.com/software/ai-datasets/something-something) 完成注册，才能下载标注文件。下载好的标注文件需要放在 `$MMACTION2/data/sthv2/annotations` 文件夹下。
用户可以使用以下命令下载标注文件。在数据集准备前，请确保命令行当前路径为 `$MMACTION2/tools/data/sthv2/`。

之后，用户可将下载好的压缩文件放在 `$MMACTION2/data/sthv2/` 文件夹下，并且使用以下指令进行解压。

```shell
```

## 步骤 3. 抽取 RGB 帧和光流

```shell

```shell
```

```shell
```

```shell
```

```shell
```

```shell
```

````
`````

在完成所有 Something-Something V2 数据集准备流程后，

在整个 MMAction2 文件夹下，Something-Something V2 的文件结构如下：

```
mmaction2
├── tools
├── data
│   ├── sthv2

```
mmaction2
├── tools
├── data
│   ├── sthv2
│   │   ├── sthv2_{train,val}_list_rawframes.txt（可选）
│   |   |   ├──...
│   |   |   ├── 1
│   |   |   ├── 2
│   |   |   ├── ...

```

关于对 Something-Something V2 进行训练和验证，请参考 [训练和测试教程](/docs/en/user_guides/train_test.md)。

# SSD

> [SSD: Single Shot MultiBox Detector](https://arxiv.org/abs/1512.02325)

We present a method for detecting objects in images using a single deep neural network. Our approach, named SSD, discretizes the output space of bounding boxes into a set of default boxes over different aspect ratios and scales per feature map location. At prediction time, the network generates scores for the presence of each object category in each default box and produces adjustments to the box to better match the object shape. Additionally, the network combines predictions from multiple feature maps with different resolutions to naturally handle objects of various sizes. Our SSD model is simple relative to methods that require object proposals because it completely eliminates proposal generation and subsequent pixel or feature resampling stage and encapsulates all computation in a single network. This makes SSD easy to train and straightforward to integrate into systems that require a detection component. Experimental results on the PASCAL VOC, MS COCO, and ILSVRC datasets confirm that SSD has comparable accuracy to methods that utilize an additional object proposal step and is much faster, while providing a unified framework for both training and inference. Compared to other single stage methods, SSD has much better accuracy, even with a smaller input image size. For 300×300 input, SSD achieves 72.1% mAP on VOC2007 test at 58 FPS on a Nvidia Titan X and for 500×500 input, SSD achieves 75.1% mAP, outperforming a comparable state of the art Faster R-CNN model.

<img src="https://user-images.githubusercontent.com/40661020/143998553-4e12f681-6025-46b4-8410-9e2e1e53a8ec.png"/>
</div>

## Results and models of SSD

| Backbone | Size | Style | Lr schd | Mem (GB) | Inf time (fps) | box AP |           Config           |                                                                                                             Download                                                                                                             |
| :------: | :--: | :---: | :-----: | :------: | :------------: | :----: | :------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |

|  VGG16   | 300  | caffe |  120e   |   9.9    |      43.7      |  25.5  | [config](./ssd300_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/ssd/ssd300_coco/ssd300_coco_20210803_015428-d231a06e.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/ssd/ssd300_coco/ssd300_coco_20210803_015428.log.json) |
|  VGG16   | 512  | caffe |  120e   |   19.4   |      30.7      |  29.5  | [config](./ssd512_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/ssd/ssd512_coco/ssd512_coco_20210803_022849-0a47a1ca.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/ssd/ssd512_coco/ssd512_coco_20210803_022849.log.json) |

## Results and models of SSD-Lite

|  Backbone   | Size | Training from scratch | Lr schd | Mem (GB) | Inf time (fps) | box AP |                           Config                           |                                                                                                                                                                 Download                                                                                                                                                                 |

| :---------: | :--: | :-------------------: | :-----: | :------: | :------------: | :----: | :--------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| MobileNetV2 | 320  |          yes          |  600e   |   4.0    |      69.9      |  21.3  | [config](./ssdlite_mobilenetv2-scratch_8xb24-600e_coco.py) | [model](https://download.openmmlab.com/mmdetection/v2.0/ssd/ssdlite_mobilenetv2_scratch_600e_coco/ssdlite_mobilenetv2_scratch_600e_coco_20210629_110627-974d9307.pth) \| [log](https://download.openmmlab.com/mmdetection/v2.0/ssd/ssdlite_mobilenetv2_scratch_600e_coco/ssdlite_mobilenetv2_scratch_600e_coco_20210629_110627.log.json) |

## Notice

## Notice

### Compatibility

In v2.14.0, [PR5291](https://github.com/open-mmlab/mmdetection/pull/5291) refactored SSD neck and head for more
flexible usage. If users want to use the SSD checkpoint trained in the older versions, we provide a scripts
`tools/model_converters/upgrade_ssd_version.py` to convert the model weights.

```bash

```

### SSD-Lite training settings

There are some differences between our implementation of MobileNetV2 SSD-Lite and the one in [TensorFlow 1.x detection model zoo](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf1_detection_zoo.md) .

1. Use 320x320 as input size instead of 300x300.
2. The anchor sizes are different.

3. The C4 feature map is taken from the last layer of stage 4 instead of the middle of the block.
4. The model in TensorFlow1.x is trained on coco 2014 and validated on coco minival2014, but we trained and validated the model on coco 2017. The mAP on val2017 is usually a little lower than minival2014 (refer to the results in TensorFlow Object Detection API, e.g., MobileNetV2 SSD gets 22 mAP on minival2014 but 20.2 mAP on val2017).

```latex
@article{Liu_2016,
   title={SSD: Single Shot MultiBox Detector},
   journal={ECCV},
   author={Liu, Wei and Anguelov, Dragomir and Erhan, Dumitru and Szegedy, Christian and Reed, Scott and Fu, Cheng-Yang and Berg, Alexander C.},
   year={2016},
}
```

```text
# train split
# x1 y1 x2 y2 "transcript"

# x1, y1, x2, y2, "transcript"

38, 43, 920, 215, "Tiredness"
275, 264, 665, 450, "kills"
0, 699, 77, 830, "A"
```

```text

word_1.png, "PROPER"
word_2.png, "FOOD"
word_3.png, "PRONTO"
```

# ATRW: A Benchmark for Amur Tiger Re-identification in the Wild

<details>

```bibtex
}
```

> **任务**: 条件生成对抗网络

</div>

`BigGAN/BigGAN-Deep`是一个条件生成模型，通过扩大批次大小和模型参数的数量，可以生成高分辨率和高质量的图像。

我们已经在`Cifar10`（32x32）中完成了`BigGAN`的训练，并在`ImageNet1k`（128x128）上对齐了训练性能。下面是一些抽样的结果，供你参考。

  <b> 我们在 CIFAR10 上训练的 BigGAN 的结果</b>
  <br/>
</div>

  <b> 我们在 ImageNet 上训练的 BigGAN 的结果</b>
  <br/>
</div>

对我们训练的 BigGAN 进行评估.

|                                     算法                                      |   数据集   |    FID (Iter)     |      IS (Iter)      |                                     下载                                      |

| :---------------------------------------------------------------------------: | :--------: | :---------------: | :-----------------: | :---------------------------------------------------------------------------: |
|           [BigGAN 32x32](./biggan_2xb25-500kiters_cifar10-32x32.py)           |  CIFAR10   |   9.78(390000)    |    8.70(390000)     | [model](https://download.openmmlab.com/mmediting/biggan/biggan_cifar10_32x32_b25x2_500k_20210728_110906-08b61a44.pth)\|[log](https://download.openmmlab.com/mmediting/biggan/biggan_cifar10_32_b25x2_500k_20210706_171051.log.json) |

### 关于可复现性的说明

`BigGAN 128x128`模型是用 V100 GPU 和 CUDA 10.1 训练的，用 A100 和 CUDA 11.3 很难再现结果。如果你对复现有任何想法，请随时与我们联系。

## 转换后的权重

由于我们还没有完成对模型的训练，我们为您提供了几个已经评估过的预训练权重。这里，我们指的是[BigGAN-PyTorch](https://github.com/ajbrock/BigGAN-PyTorch)和[pytorch-pretrained-BigGAN](https://github.com/huggingface/pytorch-pretrained-BigGAN)。

下面提供了评估结果和下载链接

|                           模型                           |   数据集   |   FID   |   IS    |                           下载                            |                        原始权重下载链接                         |

| :------------------------------------------------------: | :--------: | :-----: | :-----: | :-------------------------------------------------------: | :-------------------------------------------------------------: |

采样结果如下。

  <b> BigGAN-Deep 在 ImageNet 128x128 中使用预训练权重的结果，截断因子为 0.4 </b>
  <br/>
</div>

<br/>
</div>

  <b> BigGAN-Deep 在 ImageNet 256x256 中使用预训练权重的结果，截断因子为 0.4 </b>
  <br/>
</div>

  <b> BigGAN-Deep 在 ImageNet 512x512 中使用预训练权重的结果，截断因子为 0.4 </b>
  <br/>
</div>
上面的截断取样技巧可以通过下面的命令进行。

```bash
```

对于转换后的权重，我们在`configs/_base_/models`下提供模型配置，列举如下。

```bash

```bash
```

要在 BigGAN（或其他条件模型）上执行图像插值，请运行

```bash
```

  <b> 我们的 BigGAN-Deep 的图像插值结果</b>
  <br/>
</div>

要在 BigGAN 上进行具有固定噪声的图像插值，请运行

```bash
```

  <b> 我们的 BigGAN-Deep 在固定噪音下的图像插值结果 </b>
  <br/>
</div>
要在 BigGAN 上执行具有固定标签的图像插值，请运行

```bash

<br/>
</div>

```bash
```

  <b> 我们的 BigGAN-Deep 带有固定标签的图像插值结果</b>
  <br/>
</div>

```latex
}
```

# SinGAN (ICCV'2019)

> [Singan: Learning a Generative Model from a Single Natural Image](https://openaccess.thecvf.com/content_ICCV_2019/html/Shaham_SinGAN_Learning_a_Generative_Model_From_a_Single_Natural_Image_ICCV_2019_paper.html)

> **Task**: Internal Learning

We introduce SinGAN, an unconditional generative model that can be learned from a single natural image. Our model is trained to capture the internal distribution of patches within the image, and is then able to generate high quality, diverse samples that carry the same visual content as the image. SinGAN contains a pyramid of fully convolutional GANs, each responsible for learning the patch distribution at a different scale of the image. This allows generating new samples of arbitrary size and aspect ratio, that have significant variability, yet maintain both the global structure and the fine textures of the training image. In contrast to previous single image GAN schemes, our approach is not limited to texture images, and is not conditional (i.e. it generates samples from noise). User studies confirm that the generated samples are commonly confused to be real images. We illustrate the utility of SinGAN in a wide range of image manipulation tasks.

<img src="https://user-images.githubusercontent.com/28132635/143054395-000ceec1-3be9-4447-b4b9-effc9de94c62.JPG"/>
</div>

  <b> SinGAN balloons</b>
  <br/>
  <img src="https://user-images.githubusercontent.com/12726765/113702715-7861a900-970c-11eb-9dd8-0743cc30701f.png" width="800"/>
</div>

|             Model              |                                     Dataset                                     | Num Scales |                                     Download                                      |
| :----------------------------: | :-----------------------------------------------------------------------------: | :--------: | :-------------------------------------------------------------------------------: |

| [SinGAN](./singan_balloons.py) | [balloons.png](https://download.openmmlab.com/mmediting/dataset/singan/balloons.png) |     8      | [ckpt](https://download.openmmlab.com/mmediting/singan/singan_balloons_20210406_191047-8fcd94cf.pth) \| [pkl](https://download.openmmlab.com/mmediting/singan/singan_balloons_20210406_191047-8fcd94cf.pkl) |
|   [SinGAN](./singan_fish.py)   | [fish.jpg](https://download.openmmlab.com/mmediting/dataset/singan/fish-crop.jpg) |     10     | [ckpt](https://download.openmmlab.com/mmediting/singan/singan_fis_20210406_201006-860d91b6.pth) \| [pkl](https://download.openmmlab.com/mmediting/singan/singan_fis_20210406_201006-860d91b6.pkl) |

| [SinGAN](./singan_bohemian.py) | [bohemian.png](https://download.openmmlab.com/mmediting/dataset/singan/bohemian.png) |     10     | [ckpt](https://download.openmmlab.com/mmediting/singan/singan_bohemian_20210406_175439-f964ee38.pth) \| [pkl](https://download.openmmlab.com/mmediting/singan/singan_bohemian_20210406_175439-f964ee38.pkl) |

## Notes for using SinGAN

When training SinGAN models, users may obtain the number of scales (stages) in advance via the following commands. This number is important for constructing config file, which is related to the generator, discriminator, the training iterations and so on.

```shell
>>> from mmgen.datasets.singan_dataset import create_real_pyramid
>>> import mmcv
>>> real = mmcv.imread('real_img_path')
>>> _, _, num_scales = create_real_pyramid(real, min_size=25, max_size=300, scale_factor_init=0.75)
```

```

When testing SinGAN models, users have to modify the config file to add the `test_cfg`. As shown in `configs/singan/singan_balloons.py`, the only thing you need to do is add the path for `pkl` data. There are some important data containing in the pickle files which you can download from our website.

```python
    _delete_ = True
    pkl_data = 'path to pkl data'
)
```

```latex
@inproceedings{shaham2019singan,
  title={Singan: Learning a generative model from a single natural image},
  author={Shaham, Tamar Rott and Dekel, Tali and Michaeli, Tomer},
  pages={4570--4580},

author={Shaham, Tamar Rott and Dekel, Tali and Michaeli, Tomer},
  url={https://openaccess.thecvf.com/content_ICCV_2019/html/Shaham_SinGAN_Learning_a_Generative_Model_From_a_Single_Natural_Image_ICCV_2019_paper.html},
}
```

# Customize Evaluation Metrics

## Use metrics in MMPretrain

In MMPretrain, we have provided multiple metrics for both single-label classification and multi-label
classification:

**Single-label Classification**:

- [`SingleLabelMetric`](mmpretrain.evaluation.SingleLabelMetric), including precision, recall, f1-score and
  support.

**Multi-label Classification**:

- [`AveragePrecision`](mmpretrain.evaluation.AveragePrecision), or AP (mAP).
- [`MultiLabelMetric`](mmpretrain.evaluation.MultiLabelMetric), including precision, recall, f1-score and
  support.

To use these metrics during validation and testing, we need to modify the `val_evaluator` and `test_evaluator`
fields in the config file.

Here is several examples:

1. Calculate top-1 and top-5 accuracy during both validation and test.

   ```python
   ```

   ```python
   ```

2. Calculate top-1 accuracy, top-5 accuracy, precision and recall during both validation and test.

   ```python
   ]
   ```

3. Calculate mAP (mean AveragePrecision), CP (Class-wise mean Precision), CR (Class-wise mean Recall), CF
   (Class-wise mean F1-score), OP (Overall mean Precision), OR (Overall mean Recall) and OF1 (Overall mean
   F1-score).

   ```python
   ]
   ```

## Add new metrics

dict(type='MultiLabelMetric', average='micro'),  # overall mean
   ]
   ```

MMPretrain supports the implementation of customized evaluation metrics for users who pursue higher customization.

You need to create a new file under `mmpretrain/evaluation/metrics`, and implement the new metric in the file, for example, in `mmpretrain/evaluation/metrics/my_metric.py`. And create a customized evaluation metric class `MyMetric` which inherits [`BaseMetric in MMEngine`](mmengine.evaluator.BaseMetric).

The data format processing method `process` and the metric calculation method `compute_metrics` need to be overwritten respectively. Add it to the `METRICS` registry to implement any customized evaluation metric.

```python

```python

    """
        ...

    """
        ...
```

Then, import it in the `mmpretrain/evaluation/metrics/__init__.py` to add it into the `mmpretrain.evaluation` package.

```python
...

```

```python
...

```

Finally, use `MyMetric` in the `val_evaluator` and `test_evaluator` field of config files.

```python
```

```{note}
More details can be found in {external+mmengine:doc}`MMEngine Documentation: Evaluation <design/evaluation>`.
```

# 基于激光雷达的 3D 语义分割

基于激光雷达的 3D 语义分割是 MMDetection3D 支持的最基础的任务之一。它期望给定的模型以激光雷达采集的任意数量的特征点为输入，并预测每个输入点的语义标签。接下来，我们以 ScanNet 数据集上的 PointNet++ (SSG) 为例，展示如何准备数据，在标准的 3D 语义分割基准上训练并测试模型，以及可视化并验证结果。

## 数据准备

首先，我们需要从 ScanNet [官方网站](http://kaldir.vc.in.tum.de/scannet_benchmark/documentation)下载原始数据。

由于不同数据集的原始数据有不同的组织方式，我们通常需要用 pkl 或 json 文件收集有用的数据信息。

因此，在准备好所有的原始数据之后，我们可以遵循 [ScanNet 文档](https://github.com/open-mmlab/mmdetection3d/blob/master/data/scannet/README.md/)中的说明生成数据信息。

```
├── tools
├── data

```

## 训练

接着，我们将使用提供的配置文件训练 PointNet++ (SSG) 模型。当你使用不同的 GPU 设置进行训练时，你基本上可以按照这个[教程](https://mmdetection3d.readthedocs.io/zh_CN/latest/1_exist_data_model.html#inference-with-existing-models)的示例脚本。假设我们在一台具有 2 块 GPU 的机器上使用分布式训练：

```
```

注意，配置文件名中的 `16x2` 是指训练时用了 2 块 GPU，每块 GPU 上有 16 个样本。如果你的自定义设置不同于此，那么有时候你需要相应的调整学习率。基本规则可以参考[此处](https://arxiv.org/abs/1706.02677)。

## 定量评估

在训练期间，模型权重将会根据配置文件中的 `train_cfg = dict(val_interval=xxx)` 设置被周期性地评估。我们支持不同数据集的官方评估方案。对于 ScanNet，将使用 20 个类别的平均交并比 (mIoU) 对模型进行评估。评估结果将会被打印到终端中，如下所示：

```

## 定量评估

```

```

```
```

## 测试与提交

如果你只想在在线基准上进行推理或测试模型性能，你需要在配置文件中的 `test_evalutor` 字段增加 `submission_prefix`， 例如配置文件增加 `test_evaluator = dict(type='SegMetric',submission_prefix=work_dirs/pointnet2_ssg/test_submission`)。

```

## 测试与提交

并将 ScanNet 数据集[配置文件](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/_base_/datasets/scannet-seg.py#L129)中的 `ann_file=scannet_infos_val.pkl` 变成 `ann_file=scannet_infos_test.pkl`。在生成结果后，你可以压缩文件夹并上传至 [ScanNet 评估服务器](http://kaldir.vc.in.tum.de/scannet_benchmark/semantic_label_3d)上。

## 定性评估

MMDetection3D 还提供了通用的可视化工具，以便于我们可以对训练好的模型预测的分割结果有一个直观的感受。你也可以在评估阶段通过设置 `--eval-options 'show=True' 'out_dir=${SHOW_DIR}'` 来在线可视化分割结果，或者使用 `tools/misc/visualize_results.py` 来离线地进行可视化。此外，我们还提供了脚本 `tools/misc/browse_dataset.py` 用于可视化数据集而不做推理。更多的细节请参考[可视化文档](https://mmdetection3d.readthedocs.io/zh_CN/latest/useful_tools.html#visualization)。

# TIN

[Temporal Interlacing Network](https://ojs.aaai.org/index.php/AAAI/article/view/6872)

For a long time, the vision community tries to learn the spatio-temporal representation by combining convolutional neural network together with various temporal models, such as the families of Markov chain, optical flow, RNN and temporal convolution. However, these pipelines consume enormous computing resources due to the alternately learning process for spatial and temporal information. One natural question is whether we can embed the temporal information into the spatial one so the information in the two domains can be jointly learned once-only. In this work, we answer this question by presenting a simple yet powerful operator -- temporal interlacing network (TIN). Instead of learning the temporal features, TIN fuses the two kinds of information by interlacing spatial representations from the past to the future, and vice versa. A differentiable interlacing target can be learned to control the interlacing process. In this way, a heavy temporal model is replaced by a simple interlacing operator. We theoretically prove that with a learnable interlacing target, TIN performs equivalently to the regularized temporal convolution network (r-TCN), but gains 4% more accuracy with 6x less latency on 6 challenging benchmarks. These results push the state-of-the-art performances of video understanding by a considerable margin. Not surprising, the ensemble model of the proposed TIN won the 1st place in the ICCV19 - Multi Moments in Time challenge.

<img src="https://user-images.githubusercontent.com/34324155/143018602-d32bd546-e4f5-442c-9173-e4303676efb3.png" width="800"/>
</div>

|          1x1x8          | height 100 | 8x4  | ResNet50 | ImageNet |  39.68   |  68.55   |       44.04        |       72.72        | 8 clips x 1 crop |            x            |    6181    | [config](/configs/recognition/tin/tin_imagenet-pretrained-r50_8xb6-1x1x8-40e_sthv1-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tin/tin_imagenet-pretrained-r50_8xb6-1x1x8-40e_sthv1-rgb/tin_imagenet-pretrained-r50_8xb6-1x1x8-40e_sthv1-rgb_20220913-9b7804d6.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tin/tin_imagenet-pretrained-r50_8xb6-1x1x8-40e_sthv1-rgb/tin_imagenet-pretrained-r50_8xb6-1x1x8-40e_sthv1-rgb.log) |

|          1x1x8          | height 240 | 8x4  | ResNet50 | ImageNet |  54.78   |  82.18   |       56.48        |       83.45        | 8 clips x 1 crop |            x            |    6185    | [config](/configs/recognition/tin/tin_imagenet-pretrained-r50_8xb6-1x1x8-40e_sthv2-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tin/tin_imagenet-pretrained-r50_8xb6-1x1x8-40e_sthv2-rgb/tin_imagenet-pretrained-r50_8xb6-1x1x8-40e_sthv2-rgb_20220913-84f9b4b0.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tin/tin_imagenet-pretrained-r50_8xb6-1x1x8-40e_sthv2-rgb/tin_imagenet-pretrained-r50_8xb6-1x1x8-40e_sthv2-rgb.log) |

| frame sampling strategy |   resolution   | gpus | backbone |    pretrain     | top1 acc | top5 acc | testing protocol | inference time(video/s) | gpu_mem(M) |          config           |          ckpt           |           log           |
| :---------------------: | :------------: | :--: | :------: | :-------------: | :------: | :------: | :--------------: | :---------------------: | :--------: | :-----------------------: | :---------------------: | :---------------------: |

|          1x1x8          | short-side 256 | 8x4  | ResNet50 | TSM-Kinetics400 |  71.77   |  90.36   | 8 clips x 1 crop |            x            |    6185    | [config](/configs/recognition/tin/tin_imagenet-pretrained-r50_8xb6-1x1x8-40e_sthv2-rgb.py) | [ckpt](https://download.openmmlab.com/mmaction/v1.0/recognition/tin/tin_kinetics400-pretrained-tsm-r50_1x1x8-50e_kinetics400-rgb/tin_kinetics400-pretrained-tsm-r50_1x1x8-50e_kinetics400-rgb_20220913-7f10d0c0.pth) | [log](https://download.openmmlab.com/mmaction/v1.0/recognition/tin/tin_kinetics400-pretrained-tsm-r50_1x1x8-50e_kinetics400-rgb/tin_kinetics400-pretrained-tsm-r50_1x1x8-50e_kinetics400-rgb.log) |

Here, we use `finetune` to indicate that we use [TSM model](https://download.openmmlab.com/mmaction/v1.0/v1.0/recognition/tsm/tsm_imagenet-pretrained-r50_8xb16-1x1x8-50e_kinetics400-rgb/tsm_imagenet-pretrained-r50_8xb16-1x1x8-50e_kinetics400-rgb_20220831-64d69186.pth) trained on Kinetics-400 to finetune the TIN model on Kinetics-400.

:::{note}

1. The **reference topk acc** are got by training the [original repo #1aacd0c](https://github.com/deepcs233/TIN/tree/1aacd0c4c30d5e1d334bf023e55b855b59f158db) with no [AverageMeter issue](https://github.com/deepcs233/TIN/issues/4).
   The [AverageMeter issue](https://github.com/deepcs233/TIN/issues/4) will lead to incorrect performance, so we fix it before running.
2. The **gpus** indicates the number of gpu we used to get the checkpoint. It is noteworthy that the configs we provide are used for 8 gpus as default.

3. The values in columns named after "reference" are the results got by training on the original repo, using the same model settings.

4. The validation set of Kinetics400 we used consists of 19796 videos. These videos are available at [Kinetics400-Validation](https://mycuhk-my.sharepoint.com/:u:/g/personal/1155136485_link_cuhk_edu_hk/EbXw2WX94J1Hunyt3MWNDJUBz-nHvQYhO9pvKqm6g39PMA?e=a9QldB). The corresponding [data list](https://download.openmmlab.com/mmaction/v1.0/dataset/k400_val/kinetics_val_list.txt) (each line is of the format 'video_id, num_frames, label_index') and the [label map](https://download.openmmlab.com/mmaction/v1.0/dataset/k400_val/kinetics_class2ind.txt) are also available.

:::

## Train

```shell

## Train

```shell
```

Example: train TIN model on Something-Something V1 dataset in a deterministic option with periodic validation.

```shell
python tools/train.py configs/recognition/tin/tin_imagenet-pretrained-r50_8xb6-1x1x8-40e_sthv1-rgb.py \
    --work-dir work_dirs/tin_imagenet-pretrained-r50_8xb6-1x1x8-40e_sthv1-rgb randomness.seed=0 randomness.deterministic=True
```

## Test

```shell
```

Example: test TIN model on Something-Something V1 dataset and dump the result to a json file.

```shell

```shell
```

```shell
python tools/test.py configs/recognition/tin/tin_imagenet-pretrained-r50_8xb6-1x1x8-40e_sthv1-rgb.py \
    checkpoints/SOME_CHECKPOINT.pth --dump result.json
```

```BibTeX
@article{shao2020temporal,
    title={Temporal Interlacing Network},
    author={Hao Shao and Shengju Qian and Yu Liu},
    journal={AAAI},
}
```

# Single-view 3D Human Body Pose Estimation

## Video-based Single-view 3D Human Body Pose Estimation

Video-based 3D pose estimation is the detection and analysis of X, Y, Z coordinates of human body joints from a sequence of RGB images.

For single-person 3D pose estimation from a monocular camera, existing works can be classified into three categories:

(1) from 2D poses to 3D poses (2D-to-3D pose lifting)

(2) jointly learning 2D and 3D poses, and

(3) directly regressing 3D poses from images.

#### Human3.6m Dataset

| Arch | Receptive Field | MPJPE | P-MPJPE | N-MPJPE | ckpt | log |

| :------------------------------------------------------ | :-------------: | :---: | :-----: | :-----: | :------------------------------------------------------: | :-----------------------------------------------------: |

| [VideoPose3D-supervised](/configs/body_3d_keypoint/pose_lift/h36m/pose-lift_videopose3d-27frm-supv_8xb128-80e_h36m.py) | 27 | 40.1 | 30.1 | / | [ckpt](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_27frames_fullconv_supervised-fe8fbba9_20210527.pth) | [log](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_27frames_fullconv_supervised_20210527.log.json) |

| [VideoPose3D-supervised](/configs/body_3d_keypoint/pose_lift/h36m/pose-lift_videopose3d-81frm-supv_8xb128-80e_h36m.py) | 81 | 39.1 | 29.3 | / | [ckpt](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_81frames_fullconv_supervised-1f2d1104_20210527.pth) | [log](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_81frames_fullconv_supervised_20210527.log.json) |

| [VideoPose3D-supervised](/configs/body_3d_keypoint/pose_lift/h36m/pose-lift_videopose3d-243frm-supv_8xb128-80e_h36m.py) | 243 | | | / | [ckpt](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_243frames_fullconv_supervised-880bea25_20210527.pth) | [log](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_243frames_fullconv_supervised_20210527.log.json) |

| [VideoPose3D-supervised-CPN](/configs/body_3d_keypoint/pose_lift/h36m/pose-lift_videopose3d-1frm-supv-cpn-ft_8xb128-80e_h36m.py) | 1 | 53.0 | 41.3 | / | [ckpt](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_1frame_fullconv_supervised_cpn_ft-5c3afaed_20210527.pth) | [log](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_1frame_fullconv_supervised_cpn_ft_20210527.log.json) |

| [VideoPose3D-supervised-CPN](/configs/body_3d_keypoint/pose_lift/h36m/pose-lift_videopose3d-243frm-supv-cpn-ft_8xb128-200e_h36m.py) | 243 | | | / | [ckpt](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_243frames_fullconv_supervised_cpn_ft-88f5abbb_20210527.pth) | [log](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_243frames_fullconv_supervised_cpn_ft_20210527.log.json) |

| [VideoPose3D-semi-supervised](/configs/body_3d_keypoint/pose_lift/h36m/pose-lift_videopose3d-27frm-semi-supv_8xb64-200e_h36m.py) | 27 | 57.2 | 42.4 | 54.2 | [ckpt](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_27frames_fullconv_semi-supervised-54aef83b_20210527.pth) | [log](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_27frames_fullconv_semi-supervised_20210527.log.json) |

| [VideoPose3D-semi-supervised-CPN](/configs/body_3d_keypoint/pose_lift/h36m/pose-lift_videopose3d-27frm-semi-supv-cpn-ft_8xb64-200e_h36m.py) | 27 | 67.3 | 50.4 | 63.6 | [ckpt](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_27frames_fullconv_semi-supervised_cpn_ft-71be9cde_20210527.pth) | [log](https://download.openmmlab.com/mmpose/body3d/videopose/videopose_h36m_27frames_fullconv_semi-supervised_cpn_ft_20210527.log.json) |

## Image-based Single-view 3D Human Body Pose Estimation

3D pose estimation is the detection and analysis of X, Y, Z coordinates of human body joints from an RGB image.

| Arch | MPJPE | P-MPJPE | N-MPJPE | ckpt | log |

| [SimpleBaseline3D-tcn](/configs/body_3d_keypoint/pose_lift/h36m/pose-lift_simplebaseline3d_8xb64-200e_h36m.py) | 43.4 | 34.3 | /|[ckpt](https://download.openmmlab.com/mmpose/body3d/simple_baseline/simple3Dbaseline_h36m-f0ad73a4_20210419.pth) | [log](https://download.openmmlab.com/mmpose/body3d/simple_baseline/20210415_065056.log.json) |

# DBNetpp

> [Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion](https://arxiv.org/abs/2202.10304)

Recently, segmentation-based scene text detection methods have drawn extensive attention in the scene text detection field, because of their superiority in detecting the text instances of arbitrary shapes and extreme aspect ratios, profiting from the pixel-level descriptions. However, the vast majority of the existing segmentation-based approaches are limited to their complex post-processing algorithms and the scale robustness of their segmentation models, where the post-processing algorithms are not only isolated to the model optimization but also time-consuming and the scale robustness is usually strengthened by fusing multi-scale feature maps directly. In this paper, we propose a Differentiable Binarization (DB) module that integrates the binarization process, one of the most important steps in the post-processing procedure, into a segmentation network. Optimized along with the proposed DB module, the segmentation network can produce more accurate results, which enhances the accuracy of text detection with a simple pipeline. Furthermore, an efficient Adaptive Scale Fusion (ASF) module is proposed to improve the scale robustness by fusing features of different scales adaptively. By incorporating the proposed DB and ASF with the segmentation network, our proposed scene text detector consistently achieves state-of-the-art results, in terms of both detection accuracy and speed, on five standard benchmarks.

<img src="https://user-images.githubusercontent.com/45810070/166850828-f1e48c25-4a0f-429d-ae54-6997ed25c062.png"/>
</div>

|                                       Method                                       |    BackBone    | Training set | #iters  |                                       Download                                        |
| :--------------------------------------------------------------------------------: | :------------: | :----------: | :-----: | :-----------------------------------------------------------------------------------: |

| [DBNetpp_r50dcn](/configs/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_100k_synthtext.py) | ResNet50-dcnv2 |  SynthText   | 100,000 | [model](https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_100k_synthtext/dbnetpp_resnet50-dcnv2_fpnc_100k_synthtext-00f0a80b.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_100k_synthtext/20221215_013531.log) |

|             Method             |             BackBone             |             Pretrained Model             |  Training set   |    Test set    | #epochs | Test size | Precision | Recall | Hmean  |             Download             |

| [DBNetpp_r50](/configs/textdet/dbnetpp/dbnetpp_resnet50_fpnc_1200e_icdar2015.py) |             ResNet50             |                    -                     | ICDAR2015 Train | ICDAR2015 Test |  1200   |   1024    |  0.9079   | 0.8209 | 0.8622 | [model](https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50_fpnc_1200e_icdar2015/dbnetpp_resnet50_fpnc_1200e_icdar2015_20221025_185550-013730aa.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50_fpnc_1200e_icdar2015/20221025_185550.log) |

| [DBNetpp_r50dcn](/configs/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015.py) |          ResNet50-dcnv2          | [Synthtext](/configs/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_100k_synthtext.py) ([model](https://download.openmmlab.com/mmocr/textdet/dbnetpp/tmp_1.0_pretrain/dbnetpp_r50dcnv2_fpnc_100k_iter_synthtext-20220502-352fec8a.pth)) | ICDAR2015 Train | ICDAR2015 Test |  1200   |   1024    |  0.9116   | 0.8291 | 0.8684 | [model](https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015_20220829_230108-f289bd20.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015/20220829_230108.log) |

| [DBNetpp_r50-oclip](/configs/textdet/dbnetpp/dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015.py) | [ResNet50-oCLIP](https://download.openmmlab.com/mmocr/backbone/resnet50-oclip-7ba0c533.pth) |                    -                     | ICDAR2015 Train | ICDAR2015 Test |  1200   |   1024    |  0.9174   | 0.8609 | 0.8882 | [model](https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015/dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015_20221101_124139-4ecb39ac.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015/20221101_124139.log) |

```bibtex
@article{liao2022real,
    title={Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion},
    author={Liao, Minghui and Zou, Zhisheng and Wan, Zhaoyi and Yao, Cong and Bai, Xiang},
    publisher={IEEE}
}
```

# 第三章：PyTorch 转 ONNX 详解

ONNX 是目前模型部署中最重要的中间表示之一。学懂了 ONNX 的技术细节，就能规避大量的模型部署问题。从这篇文章开始，在接下来的三篇文章里，我们将由浅入深地介绍 ONNX 相关的知识。在第一篇文章里，我们会介绍更多 PyTorch 转 ONNX 的细节，让大家完全掌握把简单的 PyTorch 模型转成 ONNX 模型的方法；在第二篇文章里，我们将介绍如何在 PyTorch 中支持更多的 ONNX 算子，让大家能彻底走通 PyTorch 到 ONNX 这条部署路线；第三篇文章里，我们讲介绍 ONNX 本身的知识，以及修改、调试 ONNX 模型的常用方法，使大家能自行解决大部分和 ONNX 有关的部署问题。

在把 PyTorch 模型转换成 ONNX 模型时，我们往往只需要轻松地调用一句`torch.onnx.export`就行了。这个函数的接口看上去简单，但它在使用上还有着诸多的“潜规则”。在这篇教程中，我们会详细介绍 PyTorch 模型转 ONNX 模型的原理及注意事项。除此之外，我们还会介绍 PyTorch 与 ONNX 的算子对应关系，以教会大家如何处理 PyTorch 模型转换时可能会遇到的算子支持问题。

## `torch.onnx.export` 细解

在这一节里，我们将详细介绍 PyTorch 到 ONNX 的转换函数—— torch.onnx.export。我们希望大家能够更加灵活地使用这个模型转换接口，并通过了解它的实现原理来更好地应对该函数的报错（由于模型部署的兼容性问题，部署复杂模型时该函数时常会报错）。

### 计算图导出方法

[TorchScript](https://pytorch.org/docs/stable/jit.html) 是一种序列化和优化 PyTorch 模型的格式，在优化过程中，一个`torch.nn.Module`模型会被转换成 TorchScript 的`torch.jit.ScriptModule`模型。现在， TorchScript 也被常当成一种中间表示使用。我们在[其他文章](https://zhuanlan.zhihu.com/p/486914187)中对 TorchScript 有详细的介绍，这里介绍 TorchScript 仅用于说明 PyTorch 模型转 ONNX的原理。
`torch.onnx.export`中需要的模型实际上是一个`torch.jit.ScriptModule`。而要把普通 PyTorch 模型转一个这样的 TorchScript 模型，有跟踪（trace）和脚本化（script）两种导出计算图的方法。如果给`torch.onnx.export`传入了一个普通 PyTorch 模型（`torch.nn.Module`)，那么这个模型会默认使用跟踪的方法导出。这一过程如下图所示：

![image](https://user-images.githubusercontent.com/47652064/163531613-9eb3c851-933e-4b0d-913a-bf92ac36e80b.png)

回忆一下我们[第一篇教程](01_introduction_to_model_deployment.md) 知识：跟踪法只能通过实际运行一遍模型的方法导出模型的静态图，即无法识别出模型中的控制流（如循环）；脚本化则能通过解析模型来正确记录所有的控制流。我们以下面这段代码为例来看一看这两种转换方法的区别：

```python

```python

    def __init__(self, n):
        self.n = n
        self.conv = torch.nn.Conv2d(3, 3, 3)

        for i in range(self.n):
            x = self.conv(x)
        return x

models = [Model(2), Model(3)]
model_names = ['model_2', 'model_3']

for model, model_name in zip(models, model_names):
    dummy_input = torch.rand(1, 3, 10, 10)
    dummy_output = model(dummy_input)
    model_trace = torch.jit.trace(model, dummy_input)
    model_script = torch.jit.script(model)

    # 跟踪法与直接 torch.onnx.export(model, ...)等价
    torch.onnx.export(model_trace, dummy_input, f'{model_name}_trace.onnx', example_outputs=dummy_output)
    # 脚本化必须先调用 torch.jit.sciprt

# 跟踪法与直接 torch.onnx.export(model, ...)等价
    torch.onnx.export(model_script, dummy_input, f'{model_name}_script.onnx', example_outputs=dummy_output)
```

在这段代码里，我们定义了一个带循环的模型，模型通过参数`n`来控制输入张量被卷积的次数。之后，我们各创建了一个`n=2`和`n=3`的模型。我们把这两个模型分别用跟踪和脚本化的方法进行导出。
值得一提的是，由于这里的两个模型（`model_trace`, `model_script`)是 TorchScript 模型，`export`函数已经不需要再运行一遍模型了。（如果模型是用跟踪法得到的，那么在执行`torch.jit.trace`的时候就运行过一遍了；而用脚本化导出时，模型不需要实际运行）参数中的`dummy_input`和`dummy_output`仅仅是为了获取输入和输出张量的类型和形状。
运行上面的代码，我们把得到的4个 onnx 文件用 Netron 可视化：

![image](https://user-images.githubusercontent.com/47652064/163531637-994ffa0a-847d-4c0d-a9e3-0ecd78c9a3aa.png)

首先看跟踪法得到的 ONNX 模型结构。可以看出来，对于不同的 `n`,ONNX 模型的结构是不一样的。

![image](https://user-images.githubusercontent.com/47652064/163531659-b06e5df2-6e18-462e-82ff-b16d95b9765c.png)

而用脚本化的话，最终的 ONNX 模型用 `Loop` 节点来表示循环。这样哪怕对于不同的 `n`，ONNX 模型也有同样的结构。

由于推理引擎对静态图的支持更好，通常我们在模型部署时不需要显式地把 PyTorch 模型转成 TorchScript 模型，直接把 PyTorch 模型用 `torch.onnx.export` 跟踪导出即可。了解这部分的知识主要是为了在模型转换报错时能够更好地定位问题是否发生在 PyTorch 转 TorchScript 阶段。

### 参数讲解

了解完转换函数的原理后，我们来详细介绍一下该函数的主要参数的作用。我们主要会从应用的角度来介绍每个参数在不同的模型部署场景中应该如何设置，而不会去列出每个参数的所有设置方法。该函数详细的 API 文档可参考 [torch.onnx ‒ PyTorch 1.11.0 documentation](https://pytorch.org/docs/stable/onnx.html#functions)

`torch.onnx.export` 在 `torch.onnx.__init__.py`文件中的定义如下：

```python
def export(model, args, f, export_params=True, verbose=False, training=TrainingMode.EVAL,
           input_names=None, output_names=None, aten=False, export_raw_ir=False,
           operator_export_type=None, opset_version=None, _retain_param_name=True,
           do_constant_folding=True, example_outputs=None, strip_doc_string=True,

operator_export_type=None, opset_version=None, _retain_param_name=True,
           dynamic_axes=None, keep_initializers_as_inputs=None, custom_opsets=None,
           enable_onnx_checker=True, use_external_data_format=False):
```

前三个必选参数为模型、模型输入、导出的 onnx 文件名，我们对这几个参数已经很熟悉了。我们来着重看一下后面的一些常用可选参数。

#### export_params

模型中是否存储模型权重。一般中间表示包含两大类信息：模型结构和模型权重，这两类信息可以在同一个文件里存储，也可以分文件存储。ONNX 是用同一个文件表示记录模型的结构和权重的。
我们部署时一般都默认这个参数为 True。如果 onnx 文件是用来在不同框架间传递模型（比如 PyTorch 到 Tensorflow）而不是用于部署，则可以令这个参数为 False。

#### input_names, output_names

设置输入和输出张量的名称。如果不设置的话，会自动分配一些简单的名字（如数字）。
ONNX 模型的每个输入和输出张量都有一个名字。很多推理引擎在运行 ONNX 文件时，都需要以“名称-张量值”的数据对来输入数据，并根据输出张量的名称来获取输出数据。在进行跟张量有关的设置（比如添加动态维度）时，也需要知道张量的名字。
在实际的部署流水线中，我们都需要设置输入和输出张量的名称，并保证 ONNX 和推理引擎中使用同一套名称。

#### opset_version

转换时参考哪个 ONNX 算子集版本，默认为9。后文会详细介绍 PyTorch 与 ONNX 的算子对应关系。

#### dynamic_axes

指定输入输出张量的哪些维度是动态的。

为了追求效率，ONNX 默认所有参与运算的张量都是静态的（张量的形状不发生改变）。但在实际应用中，我们又希望模型的输入张量是动态的，尤其是本来就没有形状限制的全卷积模型。因此，我们需要显式地指明输入输出张量的哪几个维度的大小是可变的。
我们来看一个`dynamic_axes`的设置例子：

```python

        x = self.conv(x)
        return x

dummy_input = torch.rand(1, 3, 10, 10)
model_names = ['model_static.onnx',
'model_dynamic_0.onnx',
'model_dynamic_23.onnx']

dynamic_axes_0 = {
    'in' : [0],
    'out' : [0]
}
dynamic_axes_23 = {
    'in' : [2, 3],
    'out' : [2, 3]
}

torch.onnx.export(model, dummy_input, model_names[0],
input_names=['in'], output_names=['out'])
torch.onnx.export(model, dummy_input, model_names[1],

'in' : [2, 3],
}

input_names=['in'], output_names=['out'], dynamic_axes=dynamic_axes_0)
torch.onnx.export(model, dummy_input, model_names[2],
input_names=['in'], output_names=['out'], dynamic_axes=dynamic_axes_23)
```

首先，我们导出3个 ONNX 模型，分别为没有动态维度、第0维动态、第2第3维动态的模型。
在这份代码里，我们是用列表的方式表示动态维度，例如：

    'in' : [0],
    'out' : [0]
}
```

由于 ONNX 要求每个动态维度都有一个名字，这样写的话会引出一条 UserWarning，警告我们通过列表的方式设置动态维度的话系统会自动为它们分配名字。一种显式添加动态维度名字的方法如下：
```python
    'in' : {0: 'batch'},
    'out' : {0: 'batch'}
}
````

由于在这份代码里我们没有更多的对动态维度的操作，因此简单地用列表指定动态维度即可。
之后，我们用下面的代码来看一看动态维度的作用：

```python

origin_tensor = np.random.rand(1, 3, 10, 10).astype(np.float32)
mult_batch_tensor = np.random.rand(2, 3, 10, 10).astype(np.float32)

```python

big_tensor = np.random.rand(1, 3, 20, 20).astype(np.float32)

inputs = [origin_tensor, mult_batch_tensor, big_tensor]
exceptions = dict()

for model_name in model_names:
    for i, input in enumerate(inputs):
        try:
            ort_session = onnxruntime.InferenceSession(model_name)
            ort_inputs = {'in': input}
            ort_session.run(['out'], ort_inputs)
        except Exception as e:
            exceptions[(i, model_name)] = e
            print(f'Input[{i}] on model {model_name} error.')
        else:
            print(f'Input[{i}] on model {model_name} succeed.')
```

我们在模型导出计算图时用的是一个形状为`(1, 3, 10, 10)`的张量。现在，我们来尝试以形状分别是`(1, 3, 10, 10), (2, 3, 10, 10), (1, 3, 20, 20)`为输入，用ONNX Runtime运行一下这几个模型，看看哪些情况下会报错，并保存对应的报错信息。得到的输出信息应该如下：

```python
Input[0] on model model_static.onnx succeed.

```python
Input[1] on model model_static.onnx error.
Input[2] on model model_static.onnx error.
Input[0] on model model_dynamic_0.onnx succeed.
Input[1] on model model_dynamic_0.onnx succeed.
Input[2] on model model_dynamic_0.onnx error.
Input[0] on model model_dynamic_23.onnx succeed.
Input[1] on model model_dynamic_23.onnx error.
Input[2] on model model_dynamic_23.onnx succeed.
```

可以看出，形状相同的`(1, 3, 10, 10)`的输入在所有模型上都没有出错。而对于batch（第0维）或者长宽（第2、3维）不同的输入，只有在设置了对应的动态维度后才不会出错。我们可以错误信息中找出是哪些维度出了问题。比如我们可以用以下代码查看`input[1]`在`model_static.onnx`中的报错信息：

```python
print(exceptions[(1, 'model_static.onnx')])

# output
# [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Got invalid dimensions for input: in for the following indices index: 0 Got: 2 Expected: 1 Please fix either the inputs or the model.
```

这段报错告诉我们名字叫`in`的输入的第0维不匹配。本来该维的长度应该为1，但我们的输入是2。实际部署中，如果我们碰到了类似的报错，就可以通过设置动态维度来解决问题。

### 使用技巧

```

### 使用技巧

通过学习之前的知识，我们基本掌握了 `torch.onnx.export` 函数的部分实现原理和参数设置方法，足以完成简单模型的转换了。但在实际应用中，使用该函数还会踩很多坑。这里我们模型部署团队把在实战中积累的一些经验分享给大家。

#### 使模型在 ONNX 转换时有不同的行为

有些时候，我们希望模型在直接用 PyTorch 推理时有一套逻辑，而在导出的ONNX模型中有另一套逻辑。比如，我们可以把一些后处理的逻辑放在模型里，以简化除运行模型之外的其他代码。`torch.onnx.is_in_onnx_export()`可以实现这一任务，该函数仅在执行 `torch.onnx.export()`时为真。以下是一个例子：

```python

        if torch.onnx.is_in_onnx_export():
            x = torch.clip(x, 0, 1)
        return x
```

这里，我们仅在模型导出时把输出张量的数值限制在[0, 1]之间。使用 `is_in_onnx_export` 确实能让我们方便地在代码中添加和模型部署相关的逻辑。但是，这些代码对只关心模型训练的开发者和用户来说很不友好，突兀的部署逻辑会降低代码整体的可读性。同时，`is_in_onnx_export` 只能在每个需要添加部署逻辑的地方都“打补丁”，难以进行统一的管理。我们之后会介绍如何使用 MMDeploy 的重写机制来规避这些问题。

#### 利用中断张量跟踪的操作

PyTorch 转 ONNX 的跟踪导出法是不是万能的。如果我们在模型中做了一些很“出格”的操作，跟踪法会把某些取决于输入的中间结果变成常量，从而使导出的ONNX模型和原来的模型有出入。以下是一个会造成这种“跟踪中断”的例子：

```python

        x = x * x[0].item()
        return x, torch.Tensor([i for i in x])

dummy_input = torch.rand(10)
torch.onnx.export(model, dummy_input, 'a.onnx')
```

如果你尝试去导出这个模型，会得到一大堆 warning，告诉你转换出来的模型可能不正确。这也难怪，我们在这个模型里使用了`.item()`把 torch 中的张量转换成了普通的 Python 变量，还尝试遍历 torch 张量，并用一个列表新建一个 torch 张量。这些涉及张量与普通变量转换的逻辑都会导致最终的 ONNX 模型不太正确。
另一方面，我们也可以利用这个性质，在保证正确性的前提下令模型的中间结果变成常量。这个技巧常常用于模型的静态化上，即令模型中所有的张量形状都变成常量。在未来的教程中，我们会在部署实例中详细介绍这些“高级”操作。

#### 使用张量为输入（PyTorch版本 \< 1.9.0）

正如我们第一篇教程所展示的，在较旧(\< 1.9.0)的 PyTorch 中把 Python 数值作为 `torch.onnx.export()`的模型输入时会报错。出于兼容性的考虑，我们还是推荐以张量为模型转换时的模型输入。

## PyTorch 对 ONNX 的算子支持

在确保`torch.onnx.export()`的调用方法无误后，PyTorch 转 ONNX 时最容易出现的问题就是算子不兼容了。这里我们会介绍如何判断某个 PyTorch 算子在 ONNX 中是否兼容，以助大家在碰到报错时能更好地把错误归类。而具体添加算子的方法我们会在之后的文章里介绍。
在转换普通的`torch.nn.Module`模型时，PyTorch 一方面会用跟踪法执行前向推理，把遇到的算子整合成计算图；另一方面，PyTorch 还会把遇到的每个算子翻译成 ONNX 中定义的算子。在这个翻译过程中，可能会碰到以下情况：

- 该算子可以一对一地翻译成一个 ONNX 算子。
- 该算子在 ONNX 中没有直接对应的算子，会翻译成一至多个 ONNX 算子。
- 该算子没有定义翻译成 ONNX 的规则，报错。

那么，该如何查看 PyTorch 算子与 ONNX 算子的对应情况呢？由于 PyTorch 算子是向 ONNX 对齐的，这里我们先看一下 ONNX 算子的定义情况，再看一下 PyTorch 定义的算子映射关系。

### ONNX 算子文档

ONNX 算子的定义情况，都可以在官方的[算子文档](https://github.com/onnx/onnx/blob/main/docs/Operators.md)中查看。这份文档十分重要，我们碰到任何和 ONNX 算子有关的问题都得来”请教“这份文档。

![image](https://user-images.githubusercontent.com/47652064/163531682-306991b9-1ffe-49fe-8aee-be27b618b096.png)

这份文档中最重要的开头的这个算子变更表格。表格的第一列是算子名，第二列是该算子发生变动的算子集版本号，也就是我们之前在`torch.onnx.export`中提到的`opset_version`表示的算子集版本号。通过查看算子第一次发生变动的版本号，我们可以知道某个算子是从哪个版本开始支持的；通过查看某算子小于等于`opset_version`的第一个改动记录，我们可以知道当前算子集版本中该算子的定义规则。

![image](https://user-images.githubusercontent.com/47652064/163531690-2d70e6d2-728b-4f7f-8f5a-efaaf620ff02.png)

通过点击表格中的链接，我们可以查看某个算子的输入、输出参数规定及使用示例。比如上图是Relu在 ONNX 中的定义规则，这份定义表明 Relu 应该有一个输入和一个输入，输入输出的类型相同，均为 tensor。

### PyTorch 对 ONNX 算子的映射

在 PyTorch 中，和 ONNX 有关的定义全部放在 [torch.onnx 目录](https://github.com/pytorch/pytorch/tree/main/torch/onnx)中，如下图所示：

![image](https://user-images.githubusercontent.com/47652064/163531700-ddf994e5-6989-483c-a1a3-f1b50dfd84f0.png)

其中，`symbloic_opset{n}.py`（符号表文件）即表示 PyTorch 在支持第 n 版 ONNX 算子集时新加入的内容。我们之前讲过， bicubic 插值是在第 11 个版本开始支持的。我们以它为例来看看如何查找算子的映射情况。
首先，使用搜索功能，在`torch/onnx`文件夹搜索"bicubic"，可以发现这个这个插值在第 11 个版本的定义文件中：

![image](https://user-images.githubusercontent.com/47652064/163531714-7cf9b784-5b7f-4438-ba01-8cff4c7c9ddc.png)

之后，我们按照代码的调用逻辑，逐步跳转直到最底层的 ONNX 映射函数：

```python
upsample_bicubic2d = _interpolate("upsample_bicubic2d", 4, "cubic")

->

def _interpolate(name, dim, interpolate_mode):
    return sym_help._interpolate_helper(name, dim, interpolate_mode)

->

def _interpolate_helper(name, dim, interpolate_mode):
    def symbolic_fn(g, input, output_size, *args):
        ...

    return symbolic_fn
```

最后，在`symbolic_fn`中，我们可以看到插值算子是怎么样被映射成多个 ONNX 算子的。其中，每一个`g.op`就是一个 ONNX 的定义。比如其中的 `Resize` 算子就是这样写的：

```python
    return g.op("Resize",
                input,
                empty_roi,
                empty_scales,
                output_size,

```python
                input,
                coordinate_transformation_mode_s=coordinate_transformation_mode,
                cubic_coeff_a_f=-0.75,  # only valid when mode="cubic"
                mode_s=interpolate_mode,  # nearest, linear, or cubic
                nearest_mode_s="floor")  # only valid when mode="nearest"
```

通过在前面提到的 ONNX 算子文档中查找 [Resize 算子的定义](https://github.com/onnx/onnx/blob/main/docs/Operators.md#resize)，我们就可以知道这每一个参数的含义了。用类似的方法，我们可以去查询其他 ONNX 算子的参数含义，进而知道 PyTorch 中的参数是怎样一步一步传入到每个 ONNX 算子中的。
掌握了如何查询 PyTorch 映射到 ONNX 的关系后，我们在实际应用时就可以在 `torch.onnx.export()`的`opset_version`中先预设一个版本号，碰到了问题就去对应的 PyTorch 符号表文件里去查。如果某算子确实不存在，或者算子的映射关系不满足我们的要求，我们就可能得用其他的算子绕过去，或者自定义算子了。

## 总结

## 总结

在这篇教程中，我们系统地介绍了 PyTorch 转 ONNX 的原理。我们先是着重讲解了使用最频繁的 `torch.onnx.export`函数，又给出了查询 PyTorch 对 ONNX 算子支持情况的方法。通过本文，我们希望大家能够成功转换出大部分不需要添加新算子的 ONNX 模型，并在碰到算子问题时能够有效定位问题原因。具体而言，大家读完本文后应该了解以下的知识：

- 跟踪法和脚本化在导出带控制语句的计算图时有什么区别。
- `torch.onnx.export()`中该如何设置 `input_names, output_names, dynamic_axes`。
- 使用 `torch.onnx.is_in_onnx_export()`来使模型在转换到 ONNX 时有不同的行为。
- 如何查询 [ONNX 算子文档](https://github.com/onnx/onnx/blob/main/docs/Operators.md)。
- 如何查询 PyTorch 对某个 ONNX 版本的新特性支持情况。
- 如何判断 PyTorch 对某个 ONNX 算子是否支持，支持的方法是怎样的。

这期介绍的知识比较抽象，大家会不会觉得有点“水”？没关系，下一篇教程中，我们将以给出代码实例的形式，介绍多种为 PyTorch 转 ONNX 添加算子支持的方法，为大家在 PyTorch 转 ONNX 这条路上扫除更多的障碍。

## 练习

1. Asinh 算子出现于第 9 个 ONNX 算子集。PyTorch 在 9 号版本的符号表文件中是怎样支持这个算子的？
2. BitShift 算子出现于第11个 ONNX 算子集。PyTorch 在 11 号版本的符号表文件中是怎样支持这个算子的？

## 练习

3. 在 [第一篇教程](01_introduction_to_model_deployment.md) 中，我们讲过 PyTorch （截至第 11 号算子集）不支持在插值中设置动态的放缩系数。这个系数对应 `torch.onnx.symbolic_helper._interpolate_helper`的symbolic_fn的Resize算子映射关系中的哪个参数？我们是如何修改这一参数的？

练习的答案会在下期教程中揭晓。

# Twins

> [Twins: Revisiting the Design of Spatial Attention in Vision Transformers](https://arxiv.org/pdf/2104.13840.pdf)

<a href = "https://github.com/Meituan-AutoML/Twins">Official Repo</a>

<a href="https://github.com/open-mmlab/mmsegmentation/blob/v0.20.0/mmseg/models/backbones/twins.py#L352">Code Snippet</a>

</div>

## Usage

We have provided pretrained models converted from [official repo](https://github.com/Meituan-AutoML/Twins).

If you want to convert keys on your own to use official repositories' pre-trained models, we also provide a script [`twins2mmseg.py`](../../tools/model_converters/twins2mmseg.py) in the tools directory to convert the key of models from [the official repo](https://github.com/Meituan-AutoML/Twins) to MMSegmentation style.

```shell
python tools/model_converters/twins2mmseg.py ${PRETRAIN_PATH} ${STORE_PATH} ${MODEL_TYPE}
```

This script convert `pcpvt` or `svt` pretrained model from `PRETRAIN_PATH` and store the converted model in `STORE_PATH`.

```shell
python tools/model_converters/twins2mmseg.py ./alt_gvt_base.pth ./pretrained/alt_gvt_base.pth svt
```

```shell
```

### ADE20K

| Method  | Backbone            | Crop Size | Lr schd | Mem (GB) | Inf time (fps) | Device | mIoU  | mIoU(ms+flip) | config                                                                                                                              | download                                                                                                                                                                                                                                                                                                                                                                                       |

| FPN     | Twins-PCPVT-S       | 512x512   | 80000   | 6.60     | 27.15          | V100   | 43.26 | 44.11         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/twins/twins_pcpvt-s_fpn_fpnhead_8xb4-80k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_pcpvt-s_fpn_fpnhead_8x4_512x512_80k_ade20k/twins_pcpvt-s_fpn_fpnhead_8x4_512x512_80k_ade20k_20211201_204132-41acd132.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_pcpvt-s_fpn_fpnhead_8x4_512x512_80k_ade20k/twins_pcpvt-s_fpn_fpnhead_8x4_512x512_80k_ade20k_20211201_204132.log.json) |

| UPerNet | Twins-PCPVT-S       | 512x512   | 160000  | 9.67     | 14.24          | V100   | 46.04 | 46.92         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/twins/twins_pcpvt-s_uperhead_8xb4-160k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_pcpvt-s_uperhead_8x4_512x512_160k_ade20k/twins_pcpvt-s_uperhead_8x4_512x512_160k_ade20k_20211201_233537-8e99c07a.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_pcpvt-s_uperhead_8x4_512x512_160k_ade20k/twins_pcpvt-s_uperhead_8x4_512x512_160k_ade20k_20211201_233537.log.json)         |

| FPN     | Twins-PCPVT-B       | 512x512   | 80000   | 8.41     | 19.67          | V100   | 45.66 | 46.48         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/twins/twins_pcpvt-b_fpn_fpnhead_8xb4-80k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_pcpvt-b_fpn_fpnhead_8x4_512x512_80k_ade20k/twins_pcpvt-b_fpn_fpnhead_8x4_512x512_80k_ade20k_20211130_141019-d396db72.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_pcpvt-b_fpn_fpnhead_8x4_512x512_80k_ade20k/twins_pcpvt-b_fpn_fpnhead_8x4_512x512_80k_ade20k_20211130_141019.log.json) |

| UPerNet | Twins-PCPVT-B (8x2) | 512x512   | 160000  | 6.46     | 12.04          | V100   | 47.91 | 48.64         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/twins/twins_pcpvt-b_uperhead_8xb2-160k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_pcpvt-b_uperhead_8x2_512x512_160k_ade20k/twins_pcpvt-b_uperhead_8x2_512x512_160k_ade20k_20211130_141020-02094ea5.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_pcpvt-b_uperhead_8x2_512x512_160k_ade20k/twins_pcpvt-b_uperhead_8x2_512x512_160k_ade20k_20211130_141020.log.json)         |

| FPN     | Twins-PCPVT-L       | 512x512   | 80000   | 10.78    | 14.32          | V100   | 45.94 | 46.70         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/twins/twins_pcpvt-l_fpn_fpnhead_8xb4-80k_ade20k-512x512.py) | [model](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_pcpvt-l_fpn_fpnhead_8x4_512x512_80k_ade20k/twins_pcpvt-l_fpn_fpnhead_8x4_512x512_80k_ade20k_20211201_105226-bc6d61dc.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_pcpvt-l_fpn_fpnhead_8x4_512x512_80k_ade20k/twins_pcpvt-l_fpn_fpnhead_8x4_512x512_80k_ade20k_20211201_105226.log.json) |

| UPerNet | Twins-PCPVT-L (8x2) | 512x512   | 160000  | 7.82     | 10.70          | V100   | 49.35 | 50.08         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/twins/twins_pcpvt-l_uperhead_8xb2-160k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_pcpvt-l_uperhead_8x2_512x512_160k_ade20k/twins_pcpvt-l_uperhead_8x2_512x512_160k_ade20k_20211201_075053-c6095c07.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_pcpvt-l_uperhead_8x2_512x512_160k_ade20k/twins_pcpvt-l_uperhead_8x2_512x512_160k_ade20k_20211201_075053.log.json)         |

| FPN     | Twins-SVT-S         | 512x512   | 80000   | 5.80     | 29.79          | V100   | 44.47 | 45.42         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/twins/twins_svt-s_fpn_fpnhead_8xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_svt-s_fpn_fpnhead_8x4_512x512_80k_ade20k/twins_svt-s_fpn_fpnhead_8x4_512x512_80k_ade20k_20211130_141006-0a0d3317.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_svt-s_fpn_fpnhead_8x4_512x512_80k_ade20k/twins_svt-s_fpn_fpnhead_8x4_512x512_80k_ade20k_20211130_141006.log.json)         |

| UPerNet | SVT-S  (8x2)        | 512x512   | 160000  | 4.93     | 15.09          | V100   | 46.08 | 46.96         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/twins/twins_svt-s_uperhead_8xb2-160k_ade20k-512x512.py)     | [model](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_svt-s_uperhead_8x2_512x512_160k_ade20k/twins_svt-s_uperhead_8x2_512x512_160k_ade20k_20211130_141005-e48a2d94.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_svt-s_uperhead_8x2_512x512_160k_ade20k/twins_svt-s_uperhead_8x2_512x512_160k_ade20k_20211130_141005.log.json)                 |

| FPN     | Twins-SVT-B         | 512x512   | 80000   | 8.75     | 21.10          | V100   | 46.77 | 47.47         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/twins/twins_svt-b_fpn_fpnhead_8xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_svt-b_fpn_fpnhead_8x4_512x512_80k_ade20k/twins_svt-b_fpn_fpnhead_8x4_512x512_80k_ade20k_20211201_113849-88b2907c.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_svt-b_fpn_fpnhead_8x4_512x512_80k_ade20k/twins_svt-b_fpn_fpnhead_8x4_512x512_80k_ade20k_20211201_113849.log.json)         |

| UPerNet | Twins-SVT-B  (8x2)  | 512x512   | 160000  | 6.77     | 12.66          | V100   | 48.04 | 48.87         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/twins/twins_svt-b_uperhead_8xb2-160k_ade20k-512x512.py)     | [model](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_svt-b_uperhead_8x2_512x512_160k_ade20k/twins_svt-b_uperhead_8x2_512x512_160k_ade20k_20211202_040826-0943a1f1.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_svt-b_uperhead_8x2_512x512_160k_ade20k/twins_svt-b_uperhead_8x2_512x512_160k_ade20k_20211202_040826.log.json)                 |

| FPN     | Twins-SVT-L         | 512x512   | 80000   | 11.20    | 17.80          | V100   | 46.55 | 47.74         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/twins/twins_svt-l_fpn_fpnhead_8xb4-80k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_svt-l_fpn_fpnhead_8x4_512x512_80k_ade20k/twins_svt-l_fpn_fpnhead_8x4_512x512_80k_ade20k_20211130_141005-1d59bee2.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_svt-l_fpn_fpnhead_8x4_512x512_80k_ade20k/twins_svt-l_fpn_fpnhead_8x4_512x512_80k_ade20k_20211130_141005.log.json)         |

| UPerNet | Twins-SVT-L  (8x2)  | 512x512   | 160000  | 8.41     | 10.73          | V100   | 49.65 | 50.63         | [config](https://github.com/open-mmlab/mmsegmentation/blob/main/configs/twins/twins_pcpvt-l_uperhead_8xb2-160k_ade20k-512x512.py)   | [model](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_svt-l_uperhead_8x2_512x512_160k_ade20k/twins_svt-l_uperhead_8x2_512x512_160k_ade20k_20211130_141005-3e2cae61.pth) \| [log](https://download.openmmlab.com/mmsegmentation/v0.5/twins/twins_svt-l_uperhead_8x2_512x512_160k_ade20k/twins_svt-l_uperhead_8x2_512x512_160k_ade20k_20211130_141005.log.json)                 |

Note:

- `8x2` means 8 GPUs with 2 samples per GPU in training. Default setting of Twins on ADE20K is 8 GPUs with 4 samples per GPU in training.

Note:

- `UPerNet` and `FPN` are decoder heads utilized in corresponding Twins model, which is `UPerHead` and `FPNHead`, respectively. Specifically, models in [official repo](https://github.com/Meituan-AutoML/Twins) all use `UPerHead`.

```bibtex
}
```

```bibtex
}
```

<details>

```bibtex
}
```

| [pose_hrnet_w32_dark](/configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_hrnet-w32_dark-8xb64-210e_coco-wholebody-256x192.py) |  256x192   |  0.693  |  0.764  |  0.564  |  0.674  |  0.737  |  0.809  |  0.503  |  0.602  |  0.582   |  0.671   | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_coco_wholebody_256x192_dark-469327ef_20200922.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_coco_wholebody_256x192_dark_20200922.log.json) |

| [pose_hrnet_w48_dark+](/configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_hrnet-w48_dark-8xb32-210e_coco-wholebody-384x288.py) |  384x288   |  0.742  |  0.807  |  0.707  |  0.806  |  0.841  |  0.892  |  0.602  |  0.694  |  0.661   |  0.743   | [ckpt](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_coco_wholebody_384x288_dark-f5726563_20200918.pth) | [log](https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_coco_wholebody_384x288_dark_20200918.log.json) |

Note: `+` means the model is first pre-trained on original COCO dataset, and then fine-tuned on COCO-WholeBody dataset. We find this will lead to better performance.

> [Deep Image Matting](https://arxiv.org/abs/1703.03872)

Image matting is a fundamental computer vision problem and has many applications. Previous algorithms have poor performance when an image has similar foreground and background colors or complicated textures. The main reasons are prior methods 1) only use low-level features and 2) lack high-level context. In this paper, we propose a novel deep learning based algorithm that can tackle both these problems. Our deep model has two parts. The first part is a deep convolutional encoder-decoder network that takes an image and the corresponding trimap as inputs and predict the alpha matte of the image. The second part is a small convolutional network that refines the alpha matte predictions of the first network to have more accurate alpha values and sharper edges. In addition, we also create a large-scale image matting dataset including 49300 training images and 1000 testing images. We evaluate our algorithm on the image matting benchmark, our testing set, and a wide variety of real images. Experimental results clearly demonstrate the superiority of our algorithm over previous methods.

 <img src="https://user-images.githubusercontent.com/12726765/144175771-05b4d8f5-1abc-48ee-a5f1-8cc89a156e27.png" width="400"/>
</div >

|                                 Model                                  |    Dataset     | SAD  |  MSE  |   GRAD   |   CONN   | Training Resources |                                 Download                                  |
| :--------------------------------------------------------------------: | :------------: | :--: | :---: | :------: | :------: | :----------------: | :-----------------------------------------------------------------------: |

|         [stage1 (our)](./dim_stage1-v16_1xb1-1000k_comp1k.py)          | Composition-1k | 53.8 | 0.017 |   32.7   |   54.5   |         1          | [model](https://download.openmmlab.com/mmediting/mattors/dim/dim_stage1_v16_1x1_1000k_comp1k_SAD-53.8_20200605_140257-979a420f.pth) \| [log](https://download.openmmlab.com/mmediting/mattors/dim/dim_stage1_v16_1x1_1000k_comp1k_20200605_140257.log.json) |
|       [stage2 (our)](./dim_stage2-v16-pln_1xb1-1000k_comp1k.py)        | Composition-1k | 52.3 | 0.016 |   29.4   |   52.4   |         1          | [model](https://download.openmmlab.com/mmediting/mattors/dim/dim_stage2_v16_pln_1x1_1000k_comp1k_SAD-52.3_20200607_171909-d83c4775.pth) \| [log](https://download.openmmlab.com/mmediting/mattors/dim/dim_stage2_v16_pln_1x1_1000k_comp1k_20200607_171909.log.json) |

|       [stage3 (our)](./dim_stage3-v16-pln_1xb1-1000k_comp1k.py)        | Composition-1k | 50.6 | 0.015 | **29.0** | **50.7** |         1          | [model](https://download.openmmlab.com/mmediting/mattors/dim/dim_stage3_v16_pln_1x1_1000k_comp1k_SAD-50.6_20200609_111851-647f24b6.pth) \| [log](https://download.openmmlab.com/mmediting/mattors/dim/dim_stage3_v16_pln_1x1_1000k_comp1k_20200609_111851.log.json) |
| [stage1 (online merge)](./dim_stage1-v16_1xb1-1000k_comp1k_online-merge.py) | Composition-1k |  -   |   -   |    -     |    -     |         -          |                                     -                                     |

<!-- |     stage1 (paper)         | Composition-1k |  54.6   |   0.017   |   36.7   |   55.3   |         -          |                                      -                                       |

|                              stage3 (paper)   |Composition-1k | **50.4** | **0.014** |   31.0   |   50.8   |         -          |                                      -                                       | -->

**NOTE**

- stage1: train the encoder-decoder part without the refinement part.
- stage2: fix the encoder-decoder part and train the refinement part.
- stage3: fine-tune the whole network.

> The performance of the model is not stable during the training. Thus, the reported performance is not from the last checkpoint. Instead, it is the best performance of all validations during training.

**Train**

<details>

DIM is trained with three stages.

**Stage 1**: train the encoder-decoder part without the refinement part.

```shell
# cpu train

```

**Stage 2**: fix the encoder-decoder part and train the refinement part.

```shell
# cpu train

```shell
# cpu train

```

**Stage 3**: fine-tune the whole network.

```shell
# cpu train

```

**Test**

<details>

```shell
# cpu test

**Test**

<details>

```shell
# cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/dim/dim_stage3-v16-pln_1xb1-1000k_comp1k.py https://download.openmmlab.com/mmediting/mattors/dim/dim_stage3_v16_pln_1x1_1000k_comp1k_SAD-50.6_20200609_111851-647f24b6.pth

python tools/test.py configs/dim/dim_stage3-v16-pln_1xb1-1000k_comp1k.py https://download.openmmlab.com/mmediting/mattors/dim/dim_stage3_v16_pln_1x1_1000k_comp1k_SAD-50.6_20200609_111851-647f24b6.pth

./tools/dist_test.sh configs/dim/dim_stage3-v16-pln_1xb1-1000k_comp1k.py https://download.openmmlab.com/mmediting/mattors/dim/dim_stage3_v16_pln_1x1_1000k_comp1k_SAD-50.6_20200609_111851-647f24b6.pth 8
```

```

```bibtex
}
```

# Quantization test result

Currently mmdeploy support ncnn quantization

## Quantize with ncnn

### mmpretrain

Note:

- Because of the large amount of imagenet-1k data and ncnn has not released Vulkan int8 version, only part of the test set (4000/50000) is used.
- The accuracy will vary after quantization, and it is normal for the classification model to increase by less than 1%.

### OCR detection

Note:  [mmocr](https://github.com/open-mmlab/mmocr)  Uses 'shapely' to compute IoU, which results in a slight difference in accuracy

### Pose detection

Note: MMPose models are tested with `flip_test` explicitly set to `False` in model configs.

## v1.1.1(07/24/2023)

### Features

- Add bdd100K datasets ([#3158](https://github.com/open-mmlab/mmsegmentation/pull/3158))
- Remove batch inference assertion ([#3210](https://github.com/open-mmlab/mmsegmentation/pull/3210))

- Fix train map path for coco-stuff164k.py ([#3187](https://github.com/open-mmlab/mmsegmentation/pull/3187))
- Fix mim search error ([#3194](https://github.com/open-mmlab/mmsegmentation/pull/3194))
- Fix SegTTAModel with no attribute '\_gt_sem_seg' error ([#3152](https://github.com/open-mmlab/mmsegmentation/pull/3152))
- Fix Albumentations default key mapping mismatch ([#3195](https://github.com/open-mmlab/mmsegmentation/pull/3195))

- @OliverGrace made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/3187
- @ZiAn-Su made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/3152

- @CastleDream made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/3158
- @coding-famer made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/3174
- @Alias-z made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/3195

## v1.1.0(06/28/2023)

## What's Changed

- Support albu transform ([#2943](https://github.com/open-mmlab/mmsegmentation/pull/2943))
- Support DDRNet ([#2855](https://github.com/open-mmlab/mmsegmentation/pull/2855))
- Add GDAL backend and Support LEVIR-CD Dataset ([#2903](https://github.com/open-mmlab/mmsegmentation/pull/2903))
- Support DSDL Dataset ([#2925](https://github.com/open-mmlab/mmsegmentation/pull/2925))
- huasdorff distance loss ([#2820](https://github.com/open-mmlab/mmsegmentation/pull/2820))

### New Projects

- Support SAM inferencer ([#2897](https://github.com/open-mmlab/mmsegmentation/pull/2897))
- Added a supported for Visual Attention Network (VAN) ([#2987](https://github.com/open-mmlab/mmsegmentation/pull/2987))
- add GID dataset ([#3038](https://github.com/open-mmlab/mmsegmentation/pull/3038))
- add Medical semantic seg dataset: Bactteria ([#2568](https://github.com/open-mmlab/mmsegmentation/pull/2568))
- add Medical semantic seg dataset: Vampire ([#2633](https://github.com/open-mmlab/mmsegmentation/pull/2633))
- add Medical semantic seg dataset: Ravir ([#2635](https://github.com/open-mmlab/mmsegmentation/pull/2635))
- add Medical semantic seg dataset: Cranium ([#2675](https://github.com/open-mmlab/mmsegmentation/pull/2675))

- add Medical semantic seg dataset: bccs ([#2861](https://github.com/open-mmlab/mmsegmentation/pull/2861))
- add Medical semantic seg dataset: Gamma Task3 dataset ([#2695](https://github.com/open-mmlab/mmsegmentation/pull/2695))
- add Medical semantic seg dataset: consep ([#2724](https://github.com/open-mmlab/mmsegmentation/pull/2724))
- add Medical semantic seg dataset: breast_cancer_cell_seg dataset ([#2726](https://github.com/open-mmlab/mmsegmentation/pull/2726))
- add Medical semantic seg dataset: chest_image_pneum dataset ([#2727](https://github.com/open-mmlab/mmsegmentation/pull/2727))
- add Medical semantic seg dataset: conic2022 ([#2725](https://github.com/open-mmlab/mmsegmentation/pull/2725))
- add Medical semantic seg dataset: dr_hagis ([#2729](https://github.com/open-mmlab/mmsegmentation/pull/2729))

- add Medical semantic seg dataset: orvs ([#2728](https://github.com/open-mmlab/mmsegmentation/pull/2728))
- add Medical semantic seg dataset: ISIC-2016 Task1 ([#2708](https://github.com/open-mmlab/mmsegmentation/pull/2708))
- add Medical semantic seg dataset: ISIC-2017 Task1 ([#2709](https://github.com/open-mmlab/mmsegmentation/pull/2709))
- add Medical semantic seg dataset: Kvasir seg ([#2677](https://github.com/open-mmlab/mmsegmentation/pull/2677))
- add Medical semantic seg dataset: Kvasir seg aliyun ([#2678](https://github.com/open-mmlab/mmsegmentation/pull/2678))
- add Medical semantic seg dataset: Rite ([#2680](https://github.com/open-mmlab/mmsegmentation/pull/2680))
- add Medical semantic seg dataset: Fusc2021 ([#2682](https://github.com/open-mmlab/mmsegmentation/pull/2682))

- add Medical semantic seg dataset: 2pm vessel ([#2685](https://github.com/open-mmlab/mmsegmentation/pull/2685))
- add Medical semantic seg dataset: Pcam ([#2684](https://github.com/open-mmlab/mmsegmentation/pull/2684))
- add Medical semantic seg dataset: Pannuke ([#2683](https://github.com/open-mmlab/mmsegmentation/pull/2683))
- add Medical semantic seg dataset: Covid 19 ct cxr ([#2688](https://github.com/open-mmlab/mmsegmentation/pull/2688))
- add Medical semantic seg dataset: Crass ([#2690](https://github.com/open-mmlab/mmsegmentation/pull/2690))
- add Medical semantic seg dataset: Chest x ray images with pneumothorax masks ([#2687](https://github.com/open-mmlab/mmsegmentation/pull/2687))

### Enhancement

- Robust mapping from image path to seg map path ([#3091](https://github.com/open-mmlab/mmsegmentation/pull/3091))

- Change assertion logic inference cfg.model.test_cfg ([#3012](https://github.com/open-mmlab/mmsegmentation/pull/3012))
- Refactor dice loss ([#3002](https://github.com/open-mmlab/mmsegmentation/pull/3002))
- Update Dockerfile libgl1-mesa-dev ([#3095](https://github.com/open-mmlab/mmsegmentation/pull/3095))
- Prevent passed `ann_file` from silently failing to load ([#2966](https://github.com/open-mmlab/mmsegmentation/pull/2966))
- Update the translation of models documentation ([#2833](https://github.com/open-mmlab/mmsegmentation/pull/2833))
- Add docs contents at README.md ([#3083](https://github.com/open-mmlab/mmsegmentation/pull/3083))
- Enhance swin pretrained model loading ([#3097](https://github.com/open-mmlab/mmsegmentation/pull/3097))

- Handle case where device is neither CPU nor CUDA in HamHead ([#2868](https://github.com/open-mmlab/mmsegmentation/pull/2868))
- Fix bugs when out_channels==1 ([#2911](https://github.com/open-mmlab/mmsegmentation/pull/2911))
- Fix binary C=1 focal loss & dataset fileio ([#2935](https://github.com/open-mmlab/mmsegmentation/pull/2935))
- Fix isaid dataset pre-processing tool ([#3010](https://github.com/open-mmlab/mmsegmentation/pull/3010))
- Fix bug cannot use both '--tta' and '--out' while testing ([#3067](https://github.com/open-mmlab/mmsegmentation/pull/3067))
- Fix inferencer ut ([#3117](https://github.com/open-mmlab/mmsegmentation/pull/3117))

- Fix document ([#2863](https://github.com/open-mmlab/mmsegmentation/pull/2863), [#2896](https://github.com/open-mmlab/mmsegmentation/pull/2896), [#2919](https://github.com/open-mmlab/mmsegmentation/pull/2919), [#2951](https://github.com/open-mmlab/mmsegmentation/pull/2951), [#2970](https://github.com/open-mmlab/mmsegmentation/pull/2970), [#2961](https://github.com/open-mmlab/mmsegmentation/pull/2961), [#3042](https://github.com/open-mmlab/mmsegmentation/pull/3042), )
- Fix squeeze error when N=1 and C=1 ([#2933](https://github.com/open-mmlab/mmsegmentation/pull/2933))

- @liu-mengyang made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2896
- @likyoo made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2911
- @1qh made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2902

- @JoshuaChou2018 made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2951
- @jts250 made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2833
- @MGAMZ made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2970
- @tianbinli made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2568
- @Provable0816 made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2633
- @Zoulinx made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2903
- @wufan-tb made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2925
- @haruishi43 made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2966

- @Masaaki-75 made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2675
- @tang576225574 made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2987
- @Kedreamix made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/3010
- @nightrain01 made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/3067
- @shigengtian made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/3095
- @SheffieldCao made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/3097
- @wangruohui made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/3091
- @LHamnett made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/3012

## v1.0.0(04/06/2023)

- Add Mapillary Vistas Datasets support to MMSegmentation Core Package ([#2576](https://github.com/open-mmlab/mmsegmentation/pull/2576))
- Support PIDNet ([#2609](https://github.com/open-mmlab/mmsegmentation/pull/2609))
- Support SegNeXt ([#2654](https://github.com/open-mmlab/mmsegmentation/pull/2654))

- Support calculating FLOPs of segmentors ([#2706](https://github.com/open-mmlab/mmsegmentation/pull/2706))
- Support multi-band image for Mosaic ([#2748](https://github.com/open-mmlab/mmsegmentation/pull/2748))
- Support dump segment prediction ([#2712](https://github.com/open-mmlab/mmsegmentation/pull/2712))

### Bug fix

- Fix format_result and fix prefix param in cityscape metric, and rename CitysMetric to CityscapesMetric ([#2660](https://github.com/open-mmlab/mmsegmentation/pull/2660))

### Bug fix

- Support input gt seg map is not 2D ([#2739](https://github.com/open-mmlab/mmsegmentation/pull/2739))
- Fix accepting an unexpected argument `local-rank` in PyTorch 2.0 ([#2812](https://github.com/open-mmlab/mmsegmentation/pull/2812))

### Documentation

- Add Chinese version of various documentation ([#2673](https://github.com/open-mmlab/mmsegmentation/pull/2673), [#2702](https://github.com/open-mmlab/mmsegmentation/pull/2702), [#2703](https://github.com/open-mmlab/mmsegmentation/pull/2703), [#2701](https://github.com/open-mmlab/mmsegmentation/pull/2701), [#2722](https://github.com/open-mmlab/mmsegmentation/pull/2722), [#2733](https://github.com/open-mmlab/mmsegmentation/pull/2733), [#2769](https://github.com/open-mmlab/mmsegmentation/pull/2769), [#2790](https://github.com/open-mmlab/mmsegmentation/pull/2790), [#2798](https://github.com/open-mmlab/mmsegmentation/pull/2798))

- Update and refine various English documentation ([#2715](https://github.com/open-mmlab/mmsegmentation/pull/2715), [#2755](https://github.com/open-mmlab/mmsegmentation/pull/2755), [#2745](https://github.com/open-mmlab/mmsegmentation/pull/2745), [#2797](https://github.com/open-mmlab/mmsegmentation/pull/2797), [#2799](https://github.com/open-mmlab/mmsegmentation/pull/2799), [#2821](https://github.com/open-mmlab/mmsegmentation/pull/2821), [#2827](https://github.com/open-mmlab/mmsegmentation/pull/2827), [#2831](https://github.com/open-mmlab/mmsegmentation/pull/2831))
- Add deeplabv3 model structure documentation ([#2426](https://github.com/open-mmlab/mmsegmentation/pull/2426))
- Add custom metrics documentation ([#2799](https://github.com/open-mmlab/mmsegmentation/pull/2799))
- Add faq in dev-1.x branch ([#2765](https://github.com/open-mmlab/mmsegmentation/pull/2765))

- @liuruiqiang made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2554
- @wangjiangben-hw made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2569
- @jinxianwei made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2557
- @KKIEEK made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2747
- @Renzhihan made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2765

## v1.0.0rc6(03/03/2023)

- Support MMSegInferencer ([#2413](https://github.com/open-mmlab/mmsegmentation/pull/2413), [#2658](https://github.com/open-mmlab/mmsegmentation/pull/2658))
- Support REFUGE dataset ([#2554](https://github.com/open-mmlab/mmsegmentation/pull/2554))

- Support auto import modules from registry ([#2481](https://github.com/open-mmlab/mmsegmentation/pull/2481))
- Replace numpy ascontiguousarray with torch contiguous to speed-up ([#2604](https://github.com/open-mmlab/mmsegmentation/pull/2604))
- Add browse_dataset.py tool ([#2649](https://github.com/open-mmlab/mmsegmentation/pull/2649))

### Bug fix

- Rename and Fix bug of projects HieraSeg ([#2565](https://github.com/open-mmlab/mmsegmentation/pull/2565))
- Add out_channels  in `CascadeEncoderDecoder` and update OCRNet and MobileNet v2 results ([#2656](https://github.com/open-mmlab/mmsegmentation/pull/2656))

- Add dataflow documentation of Chinese version ([#2652](https://github.com/open-mmlab/mmsegmentation/pull/2652))
- Add custmized runtime documentation of English version ([#2533](https://github.com/open-mmlab/mmsegmentation/pull/2533))

- Add documentation for visualizing feature map using wandb backend ([#2557](https://github.com/open-mmlab/mmsegmentation/pull/2557))
- Add documentation for benchmark results on NPU (HUAWEI Ascend) ([#2569](https://github.com/open-mmlab/mmsegmentation/pull/2569), [#2596](https://github.com/open-mmlab/mmsegmentation/pull/2596), [#2610](https://github.com/open-mmlab/mmsegmentation/pull/2610))
- Fix api name error in the migration doc ([#2601](https://github.com/open-mmlab/mmsegmentation/pull/2601))
- Refine projects documentation ([#2586](https://github.com/open-mmlab/mmsegmentation/pull/2586))
- Refine MMSegmentation documentation ([#2668](https://github.com/open-mmlab/mmsegmentation/pull/2668), [#2659](https://github.com/open-mmlab/mmsegmentation/pull/2659))

- @zccjjj made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2548

## v1.0.0rc5(02/01/2023)

### Bug fix

- Fix MaskFormer and Mask2Former when install mmdet from source ([#2532](https://github.com/open-mmlab/mmsegmentation/pull/2532))
- Support new fileio interface in `MMCV>=2.0.0rc4` ([#2543](https://github.com/open-mmlab/mmsegmentation/pull/2543))
- Fix ERFNet URL in dev-1.x branch ([#2537](https://github.com/open-mmlab/mmsegmentation/pull/2537))

- Fix misleading `List[Tensor]` types ([#2546](https://github.com/open-mmlab/mmsegmentation/pull/2546))
- Rename typing.py to typing_utils.py ([#2548](https://github.com/open-mmlab/mmsegmentation/pull/2548))

## v1.0.0rc4(01/30/2023)

- Support ISNet (ICCV'2021) in projects ([#2400](https://github.com/open-mmlab/mmsegmentation/pull/2400))
- Support HSSN (CVPR'2022) in projects ([#2444](https://github.com/open-mmlab/mmsegmentation/pull/2444))

- Add Gaussian Noise and Blur for biomedical data ([#2373](https://github.com/open-mmlab/mmsegmentation/pull/2373))
- Add BioMedicalRandomGamma ([#2406](https://github.com/open-mmlab/mmsegmentation/pull/2406))
- Add BioMedical3DPad ([#2383](https://github.com/open-mmlab/mmsegmentation/pull/2383))
- Add BioMedical3DRandomFlip ([#2404](https://github.com/open-mmlab/mmsegmentation/pull/2404))

- Add `gt_edge_map` field to SegDataSample ([#2466](https://github.com/open-mmlab/mmsegmentation/pull/2466))
- Support synapse dataset ([#2432](https://github.com/open-mmlab/mmsegmentation/pull/2432), [#2465](https://github.com/open-mmlab/mmsegmentation/pull/2465))
- Support Mapillary Vistas Dataset in projects ([#2484](https://github.com/open-mmlab/mmsegmentation/pull/2484))
- Switch order of `reduce_zero_label` and applying `label_map` ([#2517](https://github.com/open-mmlab/mmsegmentation/pull/2517))

- Add ZN Customized_runtime Doc ([#2502](https://github.com/open-mmlab/mmsegmentation/pull/2502))
- Add EN datasets.md ([#2464](https://github.com/open-mmlab/mmsegmentation/pull/2464))

- Fix minor typo in migration `package.md` ([#2518](https://github.com/open-mmlab/mmsegmentation/pull/2518))

### Bug fix

- Fix incorrect `img_shape` value assignment in RandomCrop ([#2469](https://github.com/open-mmlab/mmsegmentation/pull/2469))
- Fix inference api and support setting palette to SegLocalVisualizer ([#2475](https://github.com/open-mmlab/mmsegmentation/pull/2475))
- Unfinished label conversion from `-1` to `255` ([#2516](https://github.com/open-mmlab/mmsegmentation/pull/2516))

- @blueyo0 made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2373
- @Fivethousand5k made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2406

- @suyanzhou626 made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2383
- @unrealMJ made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2400
- @Dominic23331 made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2432
- @AI-Tianlong made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2444
- @morkovka1337 made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2492
- @Leeinsn made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2404
- @siddancha made their first contribution in https://github.com/open-mmlab/mmsegmentation/pull/2516

## v1.0.0rc3(31/12/2022)

- Support test time augmentation ([#2184](https://github.com/open-mmlab/mmsegmentation/pull/2184))

- Add 'Projects/' folder and the first example project ([#2412](https://github.com/open-mmlab/mmsegmentation/pull/2412))

- Add Biomedical 3D array random crop transform ([#2378](https://github.com/open-mmlab/mmsegmentation/pull/2378))

- Add Chinese version of config tutorial ([#2371](https://github.com/open-mmlab/mmsegmentation/pull/2371))
- Add Chinese version of train & test tutorial  ([#2355](https://github.com/open-mmlab/mmsegmentation/pull/2355))
- Add Chinese version of overview ([(#2397)](https://github.com/open-mmlab/mmsegmentation/pull/2397)))
- Add Chinese version of get_started ([#2417](https://github.com/open-mmlab/mmsegmentation/pull/2417))
- Add datasets in Chinese ([#2387](https://github.com/open-mmlab/mmsegmentation/pull/2387))

- Add dataflow document ([#2403](https://github.com/open-mmlab/mmsegmentation/pull/2403))
- Add pspnet model structure graph ([#2437](https://github.com/open-mmlab/mmsegmentation/pull/2437))
- Update some content of engine Chinese documentation ([#2341](https://github.com/open-mmlab/mmsegmentation/pull/2341))
- Update TTA to migration documentation ([#2335](https://github.com/open-mmlab/mmsegmentation/pull/2335))

### Bug fix

- Remove dependency mmdet when do not use MaskFormerHead and MMDET_Mask2FormerHead ([#2448](https://github.com/open-mmlab/mmsegmentation/pull/2448))

- Add torch1.13 checking in CI ([#2402](https://github.com/open-mmlab/mmsegmentation/pull/2402))
- Fix pytorch version for merge stage test  ([#2449](https://github.com/open-mmlab/mmsegmentation/pull/2449))

## v1.0.0rc2(6/12/2022)

- Support MaskFormer ([#2215](https://github.com/open-mmlab/mmsegmentation/pull/2215))
- Support Mask2Former ([#2255](https://github.com/open-mmlab/mmsegmentation/pull/2255))

- Add ResizeShortestEdge transform ([#2339](https://github.com/open-mmlab/mmsegmentation/pull/2339))
- Support padding in data pre-processor for model testing([#2290](https://github.com/open-mmlab/mmsegmentation/pull/2290))
- Fix the problem of post-processing not removing padding ([#2367](https://github.com/open-mmlab/mmsegmentation/pull/2367))

### Bug fix

- Fix links in README ([#2024](https://github.com/open-mmlab/mmsegmentation/pull/2024))
- Fix swin load state_dict ([#2304](https://github.com/open-mmlab/mmsegmentation/pull/2304))
- Fix typo of BaseSegDataset docstring ([#2322](https://github.com/open-mmlab/mmsegmentation/pull/2322))

- Fix the bug in the visualization step ([#2326](https://github.com/open-mmlab/mmsegmentation/pull/2326))
- Fix ignore class id from -1 to 255 in BaseSegDataset ([#2332](https://github.com/open-mmlab/mmsegmentation/pull/2332))
- Fix KNet IterativeDecodeHead bug ([#2334](https://github.com/open-mmlab/mmsegmentation/pull/2334))
- Add input argument for datasets ([#2379](https://github.com/open-mmlab/mmsegmentation/pull/2379))
- Fix typo in warning on binary classification ([#2382](https://github.com/open-mmlab/mmsegmentation/pull/2382))

- Fix ci for 1.x ([#2011](https://github.com/open-mmlab/mmsegmentation/pull/2011), [#2019](https://github.com/open-mmlab/mmsegmentation/pull/2019))

- Fix lint and pre-commit hook ([#2308](https://github.com/open-mmlab/mmsegmentation/pull/2308))
- Add `data` string in .gitignore file in dev-1.x branch ([#2336](https://github.com/open-mmlab/mmsegmentation/pull/2336))
- Make scipy as a default dependency in runtime ([#2362](https://github.com/open-mmlab/mmsegmentation/pull/2362))
- Delete mmcls in runtime.txt ([#2368](https://github.com/open-mmlab/mmsegmentation/pull/2368))

- Update configuration documentation ([#2048](https://github.com/open-mmlab/mmsegmentation/pull/2048))
- Update inference documentation ([#2052](https://github.com/open-mmlab/mmsegmentation/pull/2052))
- Update train test documentation ([#2061](https://github.com/open-mmlab/mmsegmentation/pull/2061))

- Update get started documentatin ([#2148](https://github.com/open-mmlab/mmsegmentation/pull/2148))
- Update transforms documentation ([#2088](https://github.com/open-mmlab/mmsegmentation/pull/2088))
- Add MMEval projects like in README ([#2259](https://github.com/open-mmlab/mmsegmentation/pull/2259))
- Translate the visualization.md ([#2298](https://github.com/open-mmlab/mmsegmentation/pull/2298))

## v1.0.0rc1 (2/11/2022)

- Support PoolFormer ([#2191](https://github.com/open-mmlab/mmsegmentation/pull/2191))
- Add Decathlon dataset ([#2227](https://github.com/open-mmlab/mmsegmentation/pull/2227))

- Add BioMedical data loading ([#2176](https://github.com/open-mmlab/mmsegmentation/pull/2176))

- Add LIP dataset ([#2251](https://github.com/open-mmlab/mmsegmentation/pull/2251))
- Add `GenerateEdge` data transform ([#2210](https://github.com/open-mmlab/mmsegmentation/pull/2210))

### Bug fix

- Fix segmenter-vit-s_fcn config ([#2037](https://github.com/open-mmlab/mmsegmentation/pull/2037))
- Fix binary segmentation ([#2101](https://github.com/open-mmlab/mmsegmentation/pull/2101))
- Fix MMSegmentation colab demo ([#2089](https://github.com/open-mmlab/mmsegmentation/pull/2089))
- Fix ResizeToMultiple transform ([#2185](https://github.com/open-mmlab/mmsegmentation/pull/2185))
- Use SyncBN in mobilenet_v2 ([#2198](https://github.com/open-mmlab/mmsegmentation/pull/2198))
- Fix typo in installation ([#2175](https://github.com/open-mmlab/mmsegmentation/pull/2175))

- Fix typo in visualization.md ([#2116](https://github.com/open-mmlab/mmsegmentation/pull/2116))

- Add mim extras_requires in setup.py ([#2012](https://github.com/open-mmlab/mmsegmentation/pull/2012))
- Fix CI ([#2029](https://github.com/open-mmlab/mmsegmentation/pull/2029))
- Remove ops module ([#2063](https://github.com/open-mmlab/mmsegmentation/pull/2063))
- Add pyupgrade pre-commit hook ([#2078](https://github.com/open-mmlab/mmsegmentation/pull/2078))
- Add `out_file` in `add_datasample` of `SegLocalVisualizer` to directly save image ([#2090](https://github.com/open-mmlab/mmsegmentation/pull/2090))
- Upgrade pre commit hooks ([#2154](https://github.com/open-mmlab/mmsegmentation/pull/2154))

- Ignore test timm in CI when torch\<1.7 ([#2158](https://github.com/open-mmlab/mmsegmentation/pull/2158))
- Update requirements ([#2186](https://github.com/open-mmlab/mmsegmentation/pull/2186))
- Fix Windows platform CI ([#2202](https://github.com/open-mmlab/mmsegmentation/pull/2202))

- Add `Overview` documentation ([#2042](https://github.com/open-mmlab/mmsegmentation/pull/2042))
- Add `Evaluation` documentation ([#2077](https://github.com/open-mmlab/mmsegmentation/pull/2077))
- Add `Migration` documentation ([#2066](https://github.com/open-mmlab/mmsegmentation/pull/2066))
- Add `Structures` documentation ([#2070](https://github.com/open-mmlab/mmsegmentation/pull/2070))
- Add `Structures` ZN documentation ([#2129](https://github.com/open-mmlab/mmsegmentation/pull/2129))
- Add `Engine` ZN documentation ([#2157](https://github.com/open-mmlab/mmsegmentation/pull/2157))

- Update `Prepare datasets` and `Visualization` doc ([#2054](https://github.com/open-mmlab/mmsegmentation/pull/2054))
- Update `Models` documentation ([#2160](https://github.com/open-mmlab/mmsegmentation/pull/2160))
- Update `Add New Modules` documentation ([#2067](https://github.com/open-mmlab/mmsegmentation/pull/2067))
- Fix the installation commands in get_started.md ([#2174](https://github.com/open-mmlab/mmsegmentation/pull/2174))
- Add MMYOLO to README.md ([#2220](https://github.com/open-mmlab/mmsegmentation/pull/2220))

## v1.0.0rc0 (31/8/2022)

We are excited to announce the release of MMSegmentation 1.0.0rc0.
MMSeg 1.0.0rc0 is the first version of MMSegmentation 1.x, a part of the OpenMMLab 2.0 projects.

MMSeg 1.x unifies the interfaces of dataset, models, evaluation, and visualization with faster training and testing speed.

1. **New engines** MMSeg 1.x is based on [MMEngine](https://github.com/open-mmlab/mmengine), which provides a general and powerful runner that allows more flexible customizations and significantly simplifies the entrypoints of high-level interfaces.

2. **Unified interfaces** As a part of the OpenMMLab 2.0 projects, MMSeg 1.x unifies and refactors the interfaces and internal logics of train, testing, datasets, models, evaluation, and visualization. All the OpenMMLab 2.0 projects share the same design in those interfaces and logics to allow the emergence of multi-task/modality algorithms.

3. **Faster speed** We optimize the training and inference speed for common models.

4. **New features**:

   - Support TverskyLoss function

5. **More documentation and tutorials**. We add a bunch of documentation and tutorials to help users get started more smoothly. Read it [here](https://mmsegmentation.readthedocs.io/en/1.x/).

- MMSeg 1.x runs on PyTorch>=1.6. We have deprecated the support of PyTorch 1.5 to embrace the mixed precision training and other new features since PyTorch 1.6. Some models can still run on PyTorch 1.5, but the full functionality of MMSeg 1.x is not guaranteed.

- MMSeg 1.x uses Runner in [MMEngine](https://github.com/open-mmlab/mmengine) rather than that in MMCV. The new Runner implements and unifies the building logic of dataset, model, evaluation, and visualizer. Therefore, MMSeg 1.x no longer maintains the building logics of those modules in `mmseg.train.apis` and `tools/train.py`. Those code have been migrated into [MMEngine](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/runner.py). Please refer to the [migration guide of Runner in MMEngine](https://mmengine.readthedocs.io/en/latest/migration/runner.html) for more details.

- The [Runner in MMEngine](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/runner.py) uses a different config structures to ease the understanding of the components in runner. Users can read the [config example of mmseg](../user_guides/config.md) or refer to the [migration guide in MMEngine](https://mmengine.readthedocs.io/en/latest/migration/runner.html) for migration details.
- The file names of configs and models are also refactored to follow the new rules unified across OpenMMLab 2.0 projects. Please refer to the [user guides of config](../user_guides/1_config.md) for more details.

#### Components

- Dataset
- Model
- Visualization

- Support mixed precision training of all the models. However, some models may got Nan results due to some numerical issues. We will update the documentation and list their results (accuracy of failure) of mixed precision training.

- Fix several config file errors [#1994](https://github.com/open-mmlab/mmsegmentation/pull/1994)

1. Support data structures and encapsulating `seg_logits` in data samples, which can be return from models to support more common evaluation metrics.

1. Test-time augmentation: which is supported in MMSeg 0.x is not implemented in this version due to limited time slot. We will support it in the following releases with a new and simplified design.

4. Documentation: we will add more design docs, tutorials, and migration guidance so that the community can deep dive into our new design, participate the future development, and smoothly migrate downstream libraries to MMSeg 1.x.

# 概述

本章节向您介绍 MMSegmentation 框架以及语义分割相关的基本概念。我们还提供了关于 MMSegmentation 的详细教程链接。

## 什么是语义分割？

语义分割是将图像中属于同一目标类别的部分聚类在一起的任务。它也是一种像素级预测任务，因为图像中的每一个像素都将根据类别进行分类。该任务的一些示例基准有 [Cityscapes](https://www.cityscapes-dataset.com/benchmarks/), [PASCAL VOC](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/) 和 [ADE20K](https://groups.csail.mit.edu/vision/datasets/ADE20K/) 。通常用平均交并比 (Mean IoU) 和像素准确率 (Pixel Accuracy) 这两个指标来评估模型。

## 什么是 MMSegmentation?

MMSegmentation 是一个工具箱，它为语义分割任务的统一实现和模型评估提供了一个框架，并且高质量实现了常用的语义分割方法和数据集。

MMSeg 主要包含了 apis, structures, datasets, models, engine, evaluation 和 visualization 这七个主要部分。

- **apis** 提供了模型推理的高级api

- **structures** 提供了分割任务的数据结构 `SegDataSample`

- **datasets** 支持用于语义分割的多种数据集

  - **transforms** 包含多种数据增强变换

- **models** 是分割器最重要的部分，包含了分割器的不同组件

  - **segmentors** 定义了所有分割模型类
  - **backbones** 包含各种骨干网络，可将图像映射为特征图
  - **necks** 包含各种模型颈部组件，用于连接分割头和骨干网络
  - **decode_heads** 包含各种分割头，将特征图作为输入，并预测分割结果

- **data_preprocessors** 用于预处理模型的输入数据

- **engine** 是运行时组件的一部分，扩展了 [MMEngine](https://github.com/open-mmlab/mmengine) 的功能

  - **optimizers** 提供了优化器和优化器封装
  - **hooks** 提供了 runner 的各种钩子

- **evaluation** 提供了评估模型性能的不同指标

- **visualization** 分割结果的可视化工具

## 如何使用本指南？

以下是详细步骤，将带您一步步学习如何使用 MMSegmentation :

1. 有关安装说明，请参阅 [开始你的第一步](get_started.md)。

2. 对于初学者来说，MMSegmentation 是开始语义分割之旅的最好选择，因为这里实现了许多 SOTA 模型以及经典的模型 [model](model_zoo.md) 。另外，将各类组件和高级 API 結合使用，可以更便捷的执行分割任务。关于 MMSegmentation 的基本用法，请参考下面的教程：

   - [配置](user_guides/1_config.md)
   - [数据预处理](user_guides/2_dataset_prepare.md)
   - [推理](user_guides/3_inference.md)
   - [训练和测试](user_guides/4_train_test.md)

3. 如果你想了解 MMSegmentation 工作的基本类和功能，请参考下面的教程来深入研究：

   - [数据流](advanced_guides/data_flow.md)
   - [结构](advanced_guides/structures.md)
   - [模型](advanced_guides/models.md)

   - [数据集](advanced_guides/datasets.md)
   - [评估](advanced_guides/evaluation.md)

4. MMSegmentation 也为用户自定义和一些前沿的研究提供了教程，请参考下面的教程来建立你自己的分割项目：

   - [添加新的模型](advanced_guides/add_models.md)
   - [添加新的数据集](advanced_guides/add_datasets.md)
   - [添加新的 transform](advanced_guides/add_transforms.md)
   - [自定义 runtime](advanced_guides/customize_runtime.md)

5. 如果您更熟悉 MMSegmentation v0.x , 以下是 MMSegmentation v0.x 迁移到 v1.x 的文档

   - [迁移](migration/index.rst)

## 参考来源

- https://paperswithcode.com/task/semantic-segmentation/codeless#task-home

MMSegmentation 1.x provides convenient ways for monitoring training status or visualizing data and model predictions.

## Training status Monitor

MMSegmentation 1.x uses TensorBoard to monitor training status.

### TensorBoard Configuration

Install TensorBoard following [official instructions](https://www.tensorflow.org/install) e.g.

```shell
```

Add `TensorboardVisBackend` in `vis_backend` of `visualizer` in `default_runtime.py` config file:

```python
```

### Examining scalars in TensorBoard

Launch training experiment e.g.

```shell
```

```

```shell
```

Find the `vis_data` path of `work_dir` after starting training, for example, the vis_data path of this particular test is as follows:

```shell
```

The scalar file in vis_data path includes learning rate, losses and data_time etc, also record metrics results and you can refer [logging tutorial](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/logging.html) in MMEngine to log custom data. The tensorboard visualization results are executed with the following command:

```shell
```

## Data and Results visualization

### Visualizer Data Samples during Model Testing or Validation

```shell
```

MMSegmentation provides `SegVisualizationHook` which is a [hook](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/hook.md) working to visualize ground truth and prediction of segmentation during model testing and evaluation. Its configuration is in `default_hooks`, please see [Runner tutorial](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/runner.md) for more details.

For example, In `_base_/schedules/schedule_20k.py`, modify the `SegVisualizationHook` configuration, set `draw` to `True` to enable the storage of network inference results, `interval` indicates the sampling interval of the prediction results, and when set to 1, each inference result of the network will be saved. `interval` is set to 50 by default:

```python

```python

```

After launch training experiment, visualization results will be stored in the local folder in validation loop,
or when launch evaluation a model on one dataset, the prediction results will be store in the local.
The stored results of the local visualization are kept in `vis_image` under `$WORK_DIRS/vis_data`, e.g.:

```shell
```

In addition, if `TensorboardVisBackend` is add in `vis_backends`, like [above](#tensorboard-configuration),
we can also run the following command to view them in TensorBoard:

```shell

```

```shell
```

### Visualize a Single Data Sample

If you want to visualize a single data sample, we suggest to use `SegLocalVisualizer`.

</div>

```shell

</div>

```shell
```

Then you can find their local path and use the scripts below to visualize:

```python
# `PixelData` is data structure for pixel-level annotations or predictions defined in MMEngine.
# Please refer to below tutorial file of data structures in MMEngine:
# https://github.com/open-mmlab/mmengine/tree/main/docs/en/advanced_tutorials/data_element.md

# `SegDataSample` is data structure interface between different components
# defined in MMSegmentation, it includes ground truth, prediction and
# predicted logits of semantic segmentation.
# Please refer to below tutorial file of `SegDataSample` for more details:

    osp.join(
    ),
    'color')
    osp.join(
    ),

    osp.join(
    ),

# The meta information of dataset usually includes `classes` for class names and
# `palette` for visualization color of each foreground.
# All class names and palettes are defined in the file:

classes=('road', 'sidewalk', 'building', 'wall', 'fence',
# When `show=True`, the results would be shown directly,
# else if `show=False`, the results would be saved in local directory folder.
```

```

Then the visualization result of image with its corresponding ground truth could be found in `./work_dirs/vis_data/vis_image/` whose name is `out_file_cityscapes_0.png`:

</div>

If you would like to know more visualization usage, you can refer to [visualization tutorial](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/visualization.html) in MMEngine.